2024-04-26T00:06:27.520 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-26T00:06:27.520 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673875 --description rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/snaps-few-objects} --name teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi -- /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673875/orig.config.yaml 2024-04-26T00:06:27.558 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T00:06:27.653 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673875 branch: squid description: rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/snaps-few-objects} email: ceph-qa@ceph.com first_in_suite: false job_id: '7673875' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: squid ceph: conf: client: debug monc: 20 debug ms: 1 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr1: false ms bind msgr2: true ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: mon ms inject internal delays: 0.002 ms inject socket failures: 2500 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 paxos service trim min: 5 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon client ping interval: 4 mon client ping timeout: 12 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(POOL_APP_NOT_ENABLED\) sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a owner: scheduled_teuthology@teuthology priority: 101 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 9886 sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 sleep_before_teardown: 0 subset: 30270/100000 suite: rados suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a targets: smithi037.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPp8mLUzu4RpgeN5gOjjGGS5Ns9rCRCmBaFTAT1N6MWLXIlWueulz8y9M+5G9DsvjWX9Jh/KSGOEvCVJZybRHSM= smithi045.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH2wtlUjTvH0ATp/pPDHlgyHB93Z7OB7GkiyYgQbJo6GGZhwJfUr66AQRpqRgV5wArlXOkHWapqvvNtH2Qt2IDc= tasks: - install: null - ceph: null - mon_thrash: freeze_mon_duration: 20 freeze_mon_probability: 10 revive_delay: 20 thrash_delay: 1 thrash_many: true - rados: clients: - client.0 objects: 50 op_weights: copy_from: 50 delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-25_21:00:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-26T00:06:27.653 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa; will attempt to use it 2024-04-26T00:06:27.654 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks 2024-04-26T00:06:27.654 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-26T00:06:27.655 INFO:teuthology.task.internal:Checking packages... 2024-04-26T00:06:27.679 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'd2c45d977142e024a59b68a8f1dce8f0b9840551' 2024-04-26T00:06:27.680 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-26T00:06:27.680 INFO:teuthology.packaging:ref: None 2024-04-26T00:06:27.680 INFO:teuthology.packaging:tag: None 2024-04-26T00:06:27.680 INFO:teuthology.packaging:branch: squid 2024-04-26T00:06:27.680 INFO:teuthology.packaging:sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-26T00:06:27.680 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=squid 2024-04-26T00:06:27.895 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2447-gb22e2ebd-1jammy 2024-04-26T00:06:27.898 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-26T00:06:27.904 INFO:teuthology.task.internal:no buildpackages task found 2024-04-26T00:06:27.904 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-26T00:06:27.931 INFO:teuthology.task.internal:Saving configuration 2024-04-26T00:06:27.945 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-26T00:06:27.956 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T00:06:27.982 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi037.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673875', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 00:00:48.419879', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFWui+9qfjLFUp9n0sfHW2w5ZXWh4ooBAHW8C4J1iDF5'} 2024-04-26T00:06:28.004 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi045.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673875', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 00:00:48.421742', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPo34A6TQPTAO1/y7HSqZXMXqKgOsMtxI27b+E1qDURb'} 2024-04-26T00:06:28.004 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-26T00:06:28.010 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-04-26T00:06:28.011 INFO:teuthology.task.internal:roles: ubuntu@smithi045.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-04-26T00:06:28.011 INFO:teuthology.run_tasks:Running task console_log... 2024-04-26T00:06:28.107 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8e7e174280>, signals=[15]) 2024-04-26T00:06:28.108 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-26T00:06:28.114 INFO:teuthology.task.internal:Opening connections... 2024-04-26T00:06:28.114 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi037.front.sepia.ceph.com 2024-04-26T00:06:28.116 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T00:06:28.205 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi045.front.sepia.ceph.com 2024-04-26T00:06:28.206 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T00:06:28.287 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-26T00:06:28.294 DEBUG:teuthology.orchestra.run.smithi037:> uname -m 2024-04-26T00:06:28.299 INFO:teuthology.orchestra.run.smithi037.stdout:x86_64 2024-04-26T00:06:28.300 DEBUG:teuthology.orchestra.run.smithi037:> cat /etc/os-release 2024-04-26T00:06:28.346 INFO:teuthology.orchestra.run.smithi037.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-26T00:06:28.346 INFO:teuthology.orchestra.run.smithi037.stdout:NAME="Ubuntu" 2024-04-26T00:06:28.346 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION_ID="22.04" 2024-04-26T00:06:28.346 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-26T00:06:28.346 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION_CODENAME=jammy 2024-04-26T00:06:28.346 INFO:teuthology.orchestra.run.smithi037.stdout:ID=ubuntu 2024-04-26T00:06:28.346 INFO:teuthology.orchestra.run.smithi037.stdout:ID_LIKE=debian 2024-04-26T00:06:28.346 INFO:teuthology.orchestra.run.smithi037.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-26T00:06:28.346 INFO:teuthology.orchestra.run.smithi037.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-26T00:06:28.347 INFO:teuthology.orchestra.run.smithi037.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-26T00:06:28.347 INFO:teuthology.orchestra.run.smithi037.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-26T00:06:28.347 INFO:teuthology.orchestra.run.smithi037.stdout:UBUNTU_CODENAME=jammy 2024-04-26T00:06:28.347 INFO:teuthology.lock.ops:Updating smithi037.front.sepia.ceph.com on lock server 2024-04-26T00:06:28.370 DEBUG:teuthology.orchestra.run.smithi045:> uname -m 2024-04-26T00:06:28.376 INFO:teuthology.orchestra.run.smithi045.stdout:x86_64 2024-04-26T00:06:28.377 DEBUG:teuthology.orchestra.run.smithi045:> cat /etc/os-release 2024-04-26T00:06:28.425 INFO:teuthology.orchestra.run.smithi045.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-26T00:06:28.425 INFO:teuthology.orchestra.run.smithi045.stdout:NAME="Ubuntu" 2024-04-26T00:06:28.425 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION_ID="22.04" 2024-04-26T00:06:28.425 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-26T00:06:28.426 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION_CODENAME=jammy 2024-04-26T00:06:28.426 INFO:teuthology.orchestra.run.smithi045.stdout:ID=ubuntu 2024-04-26T00:06:28.426 INFO:teuthology.orchestra.run.smithi045.stdout:ID_LIKE=debian 2024-04-26T00:06:28.426 INFO:teuthology.orchestra.run.smithi045.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-26T00:06:28.426 INFO:teuthology.orchestra.run.smithi045.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-26T00:06:28.426 INFO:teuthology.orchestra.run.smithi045.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-26T00:06:28.426 INFO:teuthology.orchestra.run.smithi045.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-26T00:06:28.426 INFO:teuthology.orchestra.run.smithi045.stdout:UBUNTU_CODENAME=jammy 2024-04-26T00:06:28.426 INFO:teuthology.lock.ops:Updating smithi045.front.sepia.ceph.com on lock server 2024-04-26T00:06:28.453 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-26T00:06:28.462 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-26T00:06:28.492 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-26T00:06:28.492 DEBUG:teuthology.orchestra.run.smithi037:> test '!' -e /home/ubuntu/cephtest 2024-04-26T00:06:28.495 DEBUG:teuthology.orchestra.run.smithi045:> test '!' -e /home/ubuntu/cephtest 2024-04-26T00:06:28.499 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-26T00:06:28.523 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-26T00:06:28.523 DEBUG:teuthology.orchestra.run.smithi037:> test -z $(ls -A /var/lib/ceph) 2024-04-26T00:06:28.547 DEBUG:teuthology.orchestra.run.smithi045:> test -z $(ls -A /var/lib/ceph) 2024-04-26T00:06:28.561 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-26T00:06:28.614 INFO:teuthology.run_tasks:Running task kernel... 2024-04-26T00:06:28.629 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-26T00:06:28.629 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-26T00:06:28.630 DEBUG:teuthology.orchestra.run.smithi037:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T00:06:28.630 DEBUG:teuthology.orchestra.run.smithi045:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T00:06:28.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T00:06:28.635 DEBUG:teuthology.orchestra.run.smithi045:> uname -r 2024-04-26T00:06:28.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T00:06:28.636 DEBUG:teuthology.orchestra.run.smithi037:> uname -r 2024-04-26T00:06:28.682 INFO:teuthology.orchestra.run.smithi045.stdout:5.15.0-56-generic 2024-04-26T00:06:28.682 INFO:teuthology.task.kernel:Running kernel on smithi045: 5.15.0-56-generic 2024-04-26T00:06:28.683 DEBUG:teuthology.orchestra.run.smithi045:> sudo apt-get clean 2024-04-26T00:06:28.684 INFO:teuthology.orchestra.run.smithi037.stdout:5.15.0-56-generic 2024-04-26T00:06:28.684 INFO:teuthology.task.kernel:Running kernel on smithi037: 5.15.0-56-generic 2024-04-26T00:06:28.684 DEBUG:teuthology.orchestra.run.smithi037:> sudo apt-get clean 2024-04-26T00:06:28.802 DEBUG:teuthology.orchestra.run.smithi037:> sudo apt-get update 2024-04-26T00:06:28.810 DEBUG:teuthology.orchestra.run.smithi045:> sudo apt-get update 2024-04-26T00:06:28.942 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T00:06:28.944 INFO:teuthology.orchestra.run.smithi037.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-26T00:06:28.945 INFO:teuthology.orchestra.run.smithi037.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-26T00:06:28.957 INFO:teuthology.orchestra.run.smithi045.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-26T00:06:29.032 INFO:teuthology.orchestra.run.smithi037.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-26T00:06:29.066 INFO:teuthology.orchestra.run.smithi045.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T00:06:29.147 INFO:teuthology.orchestra.run.smithi045.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-26T00:06:29.199 INFO:teuthology.orchestra.run.smithi037.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-26T00:06:29.274 INFO:teuthology.orchestra.run.smithi037.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,610 kB] 2024-04-26T00:06:29.281 INFO:teuthology.orchestra.run.smithi037.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,394 kB] 2024-04-26T00:06:29.308 INFO:teuthology.orchestra.run.smithi045.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-26T00:06:29.325 INFO:teuthology.orchestra.run.smithi037.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-26T00:06:29.328 INFO:teuthology.orchestra.run.smithi037.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-26T00:06:29.328 INFO:teuthology.orchestra.run.smithi037.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,773 kB] 2024-04-26T00:06:29.354 INFO:teuthology.orchestra.run.smithi037.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-26T00:06:29.354 INFO:teuthology.orchestra.run.smithi037.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [300 kB] 2024-04-26T00:06:29.361 INFO:teuthology.orchestra.run.smithi037.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-26T00:06:29.361 INFO:teuthology.orchestra.run.smithi037.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [601 kB] 2024-04-26T00:06:29.367 INFO:teuthology.orchestra.run.smithi037.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-26T00:06:29.375 INFO:teuthology.orchestra.run.smithi037.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-26T00:06:29.376 INFO:teuthology.orchestra.run.smithi037.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-26T00:06:29.377 INFO:teuthology.orchestra.run.smithi037.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-26T00:06:29.377 INFO:teuthology.orchestra.run.smithi037.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-26T00:06:29.395 INFO:teuthology.orchestra.run.smithi037.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [618 kB] 2024-04-26T00:06:29.398 INFO:teuthology.orchestra.run.smithi037.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-26T00:06:29.398 INFO:teuthology.orchestra.run.smithi037.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-26T00:06:29.400 INFO:teuthology.orchestra.run.smithi037.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-26T00:06:29.403 INFO:teuthology.orchestra.run.smithi037.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-26T00:06:29.403 INFO:teuthology.orchestra.run.smithi037.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-04-26T00:06:29.408 INFO:teuthology.orchestra.run.smithi045.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,394 kB] 2024-04-26T00:06:29.421 INFO:teuthology.orchestra.run.smithi037.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-26T00:06:29.421 INFO:teuthology.orchestra.run.smithi037.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-04-26T00:06:29.424 INFO:teuthology.orchestra.run.smithi037.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-26T00:06:29.425 INFO:teuthology.orchestra.run.smithi037.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [700 kB] 2024-04-26T00:06:29.431 INFO:teuthology.orchestra.run.smithi037.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,071 kB] 2024-04-26T00:06:29.442 INFO:teuthology.orchestra.run.smithi037.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-26T00:06:29.446 INFO:teuthology.orchestra.run.smithi037.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-26T00:06:29.446 INFO:teuthology.orchestra.run.smithi037.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-26T00:06:29.448 INFO:teuthology.orchestra.run.smithi037.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-26T00:06:29.448 INFO:teuthology.orchestra.run.smithi037.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-26T00:06:29.449 INFO:teuthology.orchestra.run.smithi037.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-26T00:06:29.449 INFO:teuthology.orchestra.run.smithi037.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-04-26T00:06:29.449 INFO:teuthology.orchestra.run.smithi037.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-26T00:06:29.466 INFO:teuthology.orchestra.run.smithi045.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-26T00:06:29.468 INFO:teuthology.orchestra.run.smithi037.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-26T00:06:29.468 INFO:teuthology.orchestra.run.smithi037.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-26T00:06:29.468 INFO:teuthology.orchestra.run.smithi037.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB] 2024-04-26T00:06:29.469 INFO:teuthology.orchestra.run.smithi037.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-04-26T00:06:29.470 INFO:teuthology.orchestra.run.smithi045.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-26T00:06:29.471 INFO:teuthology.orchestra.run.smithi045.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-26T00:06:29.471 INFO:teuthology.orchestra.run.smithi045.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,773 kB] 2024-04-26T00:06:29.488 INFO:teuthology.orchestra.run.smithi037.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-26T00:06:29.488 INFO:teuthology.orchestra.run.smithi037.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-26T00:06:29.498 INFO:teuthology.orchestra.run.smithi045.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [300 kB] 2024-04-26T00:06:29.498 INFO:teuthology.orchestra.run.smithi045.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-26T00:06:29.499 INFO:teuthology.orchestra.run.smithi045.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-26T00:06:29.509 INFO:teuthology.orchestra.run.smithi045.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [601 kB] 2024-04-26T00:06:29.509 INFO:teuthology.orchestra.run.smithi045.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-26T00:06:29.515 INFO:teuthology.orchestra.run.smithi045.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-26T00:06:29.517 INFO:teuthology.orchestra.run.smithi045.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-26T00:06:29.518 INFO:teuthology.orchestra.run.smithi045.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-26T00:06:29.518 INFO:teuthology.orchestra.run.smithi045.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-26T00:06:29.518 INFO:teuthology.orchestra.run.smithi045.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-26T00:06:29.559 INFO:teuthology.orchestra.run.smithi045.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-26T00:06:29.738 INFO:teuthology.orchestra.run.smithi045.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,610 kB] 2024-04-26T00:06:31.788 INFO:teuthology.orchestra.run.smithi045.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [618 kB] 2024-04-26T00:06:32.153 INFO:teuthology.orchestra.run.smithi045.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-26T00:06:32.310 INFO:teuthology.orchestra.run.smithi045.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-26T00:06:32.319 INFO:teuthology.orchestra.run.smithi045.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-26T00:06:32.337 INFO:teuthology.orchestra.run.smithi045.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-04-26T00:06:33.234 INFO:teuthology.orchestra.run.smithi045.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-04-26T00:06:33.344 INFO:teuthology.orchestra.run.smithi045.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-26T00:06:33.515 INFO:teuthology.orchestra.run.smithi045.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,071 kB] 2024-04-26T00:06:34.158 INFO:teuthology.orchestra.run.smithi045.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [700 kB] 2024-04-26T00:06:34.196 INFO:teuthology.orchestra.run.smithi045.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-26T00:06:34.209 INFO:teuthology.orchestra.run.smithi045.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-26T00:06:34.211 INFO:teuthology.orchestra.run.smithi045.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-26T00:06:34.213 INFO:teuthology.orchestra.run.smithi045.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-26T00:06:34.213 INFO:teuthology.orchestra.run.smithi045.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-26T00:06:34.214 INFO:teuthology.orchestra.run.smithi045.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-26T00:06:34.214 INFO:teuthology.orchestra.run.smithi045.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-26T00:06:34.240 INFO:teuthology.orchestra.run.smithi045.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-04-26T00:06:34.244 INFO:teuthology.orchestra.run.smithi045.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-26T00:06:34.411 INFO:teuthology.orchestra.run.smithi045.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-26T00:06:34.495 INFO:teuthology.orchestra.run.smithi045.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-04-26T00:06:34.578 INFO:teuthology.orchestra.run.smithi045.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB] 2024-04-26T00:06:34.661 INFO:teuthology.orchestra.run.smithi045.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-26T00:06:34.692 INFO:teuthology.orchestra.run.smithi045.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-26T00:06:35.301 INFO:teuthology.orchestra.run.smithi037.stdout:Fetched 13.2 MB in 2s (6,912 kB/s) 2024-04-26T00:06:36.713 INFO:teuthology.orchestra.run.smithi037.stdout:Reading package lists... 2024-04-26T00:06:36.731 DEBUG:teuthology.orchestra.run.smithi037:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-26T00:06:36.799 INFO:teuthology.orchestra.run.smithi037.stdout:Reading package lists... 2024-04-26T00:06:37.003 INFO:teuthology.orchestra.run.smithi037.stdout:Building dependency tree... 2024-04-26T00:06:37.004 INFO:teuthology.orchestra.run.smithi037.stdout:Reading state information... 2024-04-26T00:06:37.190 INFO:teuthology.orchestra.run.smithi037.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T00:06:37.190 INFO:teuthology.orchestra.run.smithi037.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T00:06:37.190 INFO:teuthology.orchestra.run.smithi037.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T00:06:37.191 INFO:teuthology.orchestra.run.smithi037.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T00:06:37.191 INFO:teuthology.orchestra.run.smithi037.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-26T00:06:37.191 INFO:teuthology.orchestra.run.smithi037.stdout: python2.7-minimal 2024-04-26T00:06:37.191 INFO:teuthology.orchestra.run.smithi037.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T00:06:37.192 INFO:teuthology.orchestra.run.smithi037.stdout:The following additional packages will be installed: 2024-04-26T00:06:37.192 INFO:teuthology.orchestra.run.smithi037.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-26T00:06:37.192 INFO:teuthology.orchestra.run.smithi037.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-26T00:06:37.192 INFO:teuthology.orchestra.run.smithi037.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-26T00:06:37.194 INFO:teuthology.orchestra.run.smithi037.stdout:Suggested packages: 2024-04-26T00:06:37.194 INFO:teuthology.orchestra.run.smithi037.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-26T00:06:37.194 INFO:teuthology.orchestra.run.smithi037.stdout:Recommended packages: 2024-04-26T00:06:37.194 INFO:teuthology.orchestra.run.smithi037.stdout: thermald 2024-04-26T00:06:37.251 INFO:teuthology.orchestra.run.smithi037.stdout:The following NEW packages will be installed: 2024-04-26T00:06:37.252 INFO:teuthology.orchestra.run.smithi037.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-26T00:06:37.252 INFO:teuthology.orchestra.run.smithi037.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-26T00:06:37.252 INFO:teuthology.orchestra.run.smithi037.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-26T00:06:37.253 INFO:teuthology.orchestra.run.smithi037.stdout:The following packages will be upgraded: 2024-04-26T00:06:37.253 INFO:teuthology.orchestra.run.smithi037.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-26T00:06:37.311 INFO:teuthology.orchestra.run.smithi037.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-26T00:06:37.311 INFO:teuthology.orchestra.run.smithi037.stdout:Need to get 113 MB of archives. 2024-04-26T00:06:37.311 INFO:teuthology.orchestra.run.smithi037.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-26T00:06:37.312 INFO:teuthology.orchestra.run.smithi037.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-26T00:06:37.964 INFO:teuthology.orchestra.run.smithi037.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-26T00:06:38.133 INFO:teuthology.orchestra.run.smithi037.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-26T00:06:39.312 INFO:teuthology.orchestra.run.smithi045.stdout:Fetched 13.2 MB in 6s (2,269 kB/s) 2024-04-26T00:06:39.343 INFO:teuthology.orchestra.run.smithi037.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-26T00:06:39.343 INFO:teuthology.orchestra.run.smithi037.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-26T00:06:39.343 INFO:teuthology.orchestra.run.smithi037.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-26T00:06:39.578 INFO:teuthology.orchestra.run.smithi037.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-26T00:06:39.614 INFO:teuthology.orchestra.run.smithi037.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-26T00:06:40.026 INFO:teuthology.orchestra.run.smithi037.stdout:Fetched 113 MB in 2s (48.1 MB/s) 2024-04-26T00:06:40.212 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-26T00:06:40.915 INFO:teuthology.orchestra.run.smithi045.stdout:Reading package lists... 2024-04-26T00:06:40.933 DEBUG:teuthology.orchestra.run.smithi045:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-26T00:06:40.998 INFO:teuthology.orchestra.run.smithi045.stdout:Reading package lists... 2024-04-26T00:06:41.203 INFO:teuthology.orchestra.run.smithi045.stdout:Building dependency tree... 2024-04-26T00:06:41.204 INFO:teuthology.orchestra.run.smithi045.stdout:Reading state information... 2024-04-26T00:06:41.388 INFO:teuthology.orchestra.run.smithi045.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T00:06:41.388 INFO:teuthology.orchestra.run.smithi045.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T00:06:41.388 INFO:teuthology.orchestra.run.smithi045.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T00:06:41.388 INFO:teuthology.orchestra.run.smithi045.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T00:06:41.389 INFO:teuthology.orchestra.run.smithi045.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-26T00:06:41.389 INFO:teuthology.orchestra.run.smithi045.stdout: python2.7-minimal 2024-04-26T00:06:41.389 INFO:teuthology.orchestra.run.smithi045.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T00:06:41.390 INFO:teuthology.orchestra.run.smithi045.stdout:The following additional packages will be installed: 2024-04-26T00:06:41.390 INFO:teuthology.orchestra.run.smithi045.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-26T00:06:41.390 INFO:teuthology.orchestra.run.smithi045.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-26T00:06:41.391 INFO:teuthology.orchestra.run.smithi045.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-26T00:06:41.392 INFO:teuthology.orchestra.run.smithi045.stdout:Suggested packages: 2024-04-26T00:06:41.392 INFO:teuthology.orchestra.run.smithi045.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-26T00:06:41.392 INFO:teuthology.orchestra.run.smithi045.stdout:Recommended packages: 2024-04-26T00:06:41.392 INFO:teuthology.orchestra.run.smithi045.stdout: thermald 2024-04-26T00:06:41.449 INFO:teuthology.orchestra.run.smithi045.stdout:The following NEW packages will be installed: 2024-04-26T00:06:41.449 INFO:teuthology.orchestra.run.smithi045.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-26T00:06:41.449 INFO:teuthology.orchestra.run.smithi045.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-26T00:06:41.449 INFO:teuthology.orchestra.run.smithi045.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-26T00:06:41.450 INFO:teuthology.orchestra.run.smithi045.stdout:The following packages will be upgraded: 2024-04-26T00:06:41.451 INFO:teuthology.orchestra.run.smithi045.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-26T00:06:41.509 INFO:teuthology.orchestra.run.smithi045.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-26T00:06:41.509 INFO:teuthology.orchestra.run.smithi045.stdout:Need to get 113 MB of archives. 2024-04-26T00:06:41.509 INFO:teuthology.orchestra.run.smithi045.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-26T00:06:41.510 INFO:teuthology.orchestra.run.smithi045.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-26T00:06:41.962 INFO:teuthology.orchestra.run.smithi045.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-26T00:06:42.072 INFO:teuthology.orchestra.run.smithi045.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-26T00:06:42.185 INFO:teuthology.orchestra.run.smithi037.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-26T00:06:42.189 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T00:06:42.258 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:06:43.636 INFO:teuthology.orchestra.run.smithi045.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-26T00:06:43.636 INFO:teuthology.orchestra.run.smithi045.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-26T00:06:43.636 INFO:teuthology.orchestra.run.smithi045.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-26T00:06:43.935 INFO:teuthology.orchestra.run.smithi045.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-26T00:06:43.987 INFO:teuthology.orchestra.run.smithi045.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-26T00:06:44.399 INFO:teuthology.orchestra.run.smithi045.stdout:Fetched 113 MB in 3s (44.8 MB/s) 2024-04-26T00:06:44.562 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-26T00:06:44.767 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-26T00:06:44.788 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T00:06:44.873 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:06:45.295 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-26T00:06:45.316 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T00:06:45.353 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:06:46.619 INFO:teuthology.orchestra.run.smithi045.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-26T00:06:46.623 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T00:06:47.075 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:06:49.253 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-26T00:06:49.274 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T00:06:49.355 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:06:49.814 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-26T00:06:49.835 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T00:06:49.872 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:06:51.811 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T00:06:51.911 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T00:06:52.219 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T00:06:52.358 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T00:06:52.669 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-26T00:06:52.692 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-26T00:06:52.735 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-26T00:06:56.701 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T00:06:56.850 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T00:06:57.169 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T00:06:57.290 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T00:06:57.593 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-26T00:06:57.616 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-26T00:06:57.660 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-26T00:06:57.894 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-26T00:06:57.931 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T00:06:57.961 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:07:01.146 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T00:07:01.262 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T00:07:01.579 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-26T00:07:01.700 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:07:01.826 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-26T00:07:01.955 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:07:03.301 INFO:teuthology.orchestra.run.smithi037.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-26T00:07:03.302 INFO:teuthology.orchestra.run.smithi037.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-26T00:07:03.302 INFO:teuthology.orchestra.run.smithi037.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-26T00:07:03.302 INFO:teuthology.orchestra.run.smithi037.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-26T00:07:03.466 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:07:03.823 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-26T00:07:03.860 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T00:07:03.890 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:07:05.186 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:07:06.636 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-26T00:07:06.762 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-26T00:07:06.888 INFO:teuthology.orchestra.run.smithi037.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:07:06.955 INFO:teuthology.orchestra.run.smithi037.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-26T00:07:06.955 INFO:teuthology.orchestra.run.smithi037.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-26T00:07:07.371 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T00:07:07.495 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T00:07:07.871 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-26T00:07:08.012 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:07:08.135 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-26T00:07:08.263 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:07:09.678 INFO:teuthology.orchestra.run.smithi045.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-26T00:07:09.678 INFO:teuthology.orchestra.run.smithi045.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-26T00:07:09.678 INFO:teuthology.orchestra.run.smithi045.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-26T00:07:09.678 INFO:teuthology.orchestra.run.smithi045.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-26T00:07:09.849 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:07:11.385 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:07:12.794 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-26T00:07:12.920 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-26T00:07:13.071 INFO:teuthology.orchestra.run.smithi045.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T00:07:13.132 INFO:teuthology.orchestra.run.smithi045.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-26T00:07:13.133 INFO:teuthology.orchestra.run.smithi045.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-26T00:07:21.855 INFO:teuthology.orchestra.run.smithi037.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-26T00:07:21.856 INFO:teuthology.orchestra.run.smithi037.stdout:Sourcing file `/etc/default/grub' 2024-04-26T00:07:21.864 INFO:teuthology.orchestra.run.smithi037.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-26T00:07:21.880 INFO:teuthology.orchestra.run.smithi037.stdout:Generating grub configuration file ... 2024-04-26T00:07:22.168 INFO:teuthology.orchestra.run.smithi037.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-26T00:07:22.190 INFO:teuthology.orchestra.run.smithi037.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-26T00:07:22.460 INFO:teuthology.orchestra.run.smithi037.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-26T00:07:22.465 INFO:teuthology.orchestra.run.smithi037.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-26T00:07:22.533 INFO:teuthology.orchestra.run.smithi037.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-26T00:07:22.539 INFO:teuthology.orchestra.run.smithi037.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-26T00:07:22.696 INFO:teuthology.orchestra.run.smithi037.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-26T00:07:22.696 INFO:teuthology.orchestra.run.smithi037.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-26T00:07:22.696 INFO:teuthology.orchestra.run.smithi037.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-26T00:07:22.740 INFO:teuthology.orchestra.run.smithi037.stdout:done 2024-04-26T00:07:23.237 DEBUG:teuthology.orchestra.run.smithi037:> dpkg -s linux-image-generic 2024-04-26T00:07:23.266 INFO:teuthology.orchestra.run.smithi037.stdout:Package: linux-image-generic 2024-04-26T00:07:23.267 INFO:teuthology.orchestra.run.smithi037.stdout:Status: install ok installed 2024-04-26T00:07:23.267 INFO:teuthology.orchestra.run.smithi037.stdout:Priority: optional 2024-04-26T00:07:23.267 INFO:teuthology.orchestra.run.smithi037.stdout:Section: kernel 2024-04-26T00:07:23.267 INFO:teuthology.orchestra.run.smithi037.stdout:Installed-Size: 21 2024-04-26T00:07:23.267 INFO:teuthology.orchestra.run.smithi037.stdout:Maintainer: Ubuntu Kernel Team 2024-04-26T00:07:23.267 INFO:teuthology.orchestra.run.smithi037.stdout:Architecture: amd64 2024-04-26T00:07:23.267 INFO:teuthology.orchestra.run.smithi037.stdout:Source: linux-meta 2024-04-26T00:07:23.267 INFO:teuthology.orchestra.run.smithi037.stdout:Version: 5.15.0.105.102 2024-04-26T00:07:23.267 INFO:teuthology.orchestra.run.smithi037.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-26T00:07:23.267 INFO:teuthology.orchestra.run.smithi037.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-26T00:07:23.267 INFO:teuthology.orchestra.run.smithi037.stdout:Recommends: thermald 2024-04-26T00:07:23.267 INFO:teuthology.orchestra.run.smithi037.stdout:Description: Generic Linux kernel image 2024-04-26T00:07:23.268 INFO:teuthology.orchestra.run.smithi037.stdout: This package will always depend on the latest generic kernel image 2024-04-26T00:07:23.268 INFO:teuthology.orchestra.run.smithi037.stdout: available. 2024-04-26T00:07:23.268 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-26T00:07:23.268 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-26T00:07:23.268 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-26T00:07:23.268 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-26T00:07:23.268 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi037.front.sepia.ceph.com, path=None, version=distro) 2024-04-26T00:07:23.269 DEBUG:teuthology.orchestra.run.smithi037:> sudo apt-get clean 2024-04-26T00:07:23.424 DEBUG:teuthology.orchestra.run.smithi037:> sudo apt-get update 2024-04-26T00:07:23.581 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-26T00:07:23.681 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T00:07:23.768 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-26T00:07:23.844 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-26T00:07:24.986 INFO:teuthology.orchestra.run.smithi037.stdout:Reading package lists... 2024-04-26T00:07:25.004 DEBUG:teuthology.orchestra.run.smithi037:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-26T00:07:25.066 INFO:teuthology.orchestra.run.smithi037.stdout:Reading package lists... 2024-04-26T00:07:25.268 INFO:teuthology.orchestra.run.smithi037.stdout:Building dependency tree... 2024-04-26T00:07:25.269 INFO:teuthology.orchestra.run.smithi037.stdout:Reading state information... 2024-04-26T00:07:25.452 INFO:teuthology.orchestra.run.smithi037.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-26T00:07:25.452 INFO:teuthology.orchestra.run.smithi037.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T00:07:25.452 INFO:teuthology.orchestra.run.smithi037.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T00:07:25.452 INFO:teuthology.orchestra.run.smithi037.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T00:07:25.452 INFO:teuthology.orchestra.run.smithi037.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T00:07:25.452 INFO:teuthology.orchestra.run.smithi037.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-26T00:07:25.452 INFO:teuthology.orchestra.run.smithi037.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-26T00:07:25.452 INFO:teuthology.orchestra.run.smithi037.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T00:07:25.495 INFO:teuthology.orchestra.run.smithi037.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-26T00:07:25.497 DEBUG:teuthology.orchestra.run.smithi037:> dpkg -s linux-image-generic 2024-04-26T00:07:25.514 INFO:teuthology.orchestra.run.smithi037.stdout:Package: linux-image-generic 2024-04-26T00:07:25.514 INFO:teuthology.orchestra.run.smithi037.stdout:Status: install ok installed 2024-04-26T00:07:25.514 INFO:teuthology.orchestra.run.smithi037.stdout:Priority: optional 2024-04-26T00:07:25.514 INFO:teuthology.orchestra.run.smithi037.stdout:Section: kernel 2024-04-26T00:07:25.514 INFO:teuthology.orchestra.run.smithi037.stdout:Installed-Size: 21 2024-04-26T00:07:25.514 INFO:teuthology.orchestra.run.smithi037.stdout:Maintainer: Ubuntu Kernel Team 2024-04-26T00:07:25.514 INFO:teuthology.orchestra.run.smithi037.stdout:Architecture: amd64 2024-04-26T00:07:25.514 INFO:teuthology.orchestra.run.smithi037.stdout:Source: linux-meta 2024-04-26T00:07:25.515 INFO:teuthology.orchestra.run.smithi037.stdout:Version: 5.15.0.105.102 2024-04-26T00:07:25.515 INFO:teuthology.orchestra.run.smithi037.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-26T00:07:25.515 INFO:teuthology.orchestra.run.smithi037.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-26T00:07:25.515 INFO:teuthology.orchestra.run.smithi037.stdout:Recommends: thermald 2024-04-26T00:07:25.515 INFO:teuthology.orchestra.run.smithi037.stdout:Description: Generic Linux kernel image 2024-04-26T00:07:25.515 INFO:teuthology.orchestra.run.smithi037.stdout: This package will always depend on the latest generic kernel image 2024-04-26T00:07:25.515 INFO:teuthology.orchestra.run.smithi037.stdout: available. 2024-04-26T00:07:25.515 DEBUG:teuthology.orchestra.run.smithi037:> mktemp 2024-04-26T00:07:25.562 INFO:teuthology.orchestra.run.smithi037.stdout:/tmp/tmp.S0OuV5laq4 2024-04-26T00:07:25.562 DEBUG:teuthology.orchestra.run.smithi037:> sudo cp /boot/grub/grub.cfg /tmp/tmp.S0OuV5laq4 2024-04-26T00:07:25.619 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 0666 /tmp/tmp.S0OuV5laq4 2024-04-26T00:07:25.734 DEBUG:teuthology.orchestra.remote:smithi037:/tmp/tmp.S0OuV5laq4 is 10KB 2024-04-26T00:07:25.787 DEBUG:teuthology.orchestra.run.smithi037:> rm -fr /tmp/tmp.S0OuV5laq4 2024-04-26T00:07:25.793 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-26T00:07:25.847 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T00:07:25.847 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-26T00:07:25.847 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-26T00:07:25.927 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-26T00:07:25.927 DEBUG:teuthology.orchestra.run.smithi037:> sudo update-grub 2024-04-26T00:07:27.080 INFO:teuthology.orchestra.run.smithi037.stderr:Sourcing file `/etc/default/grub' 2024-04-26T00:07:27.081 INFO:teuthology.orchestra.run.smithi037.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-26T00:07:27.083 INFO:teuthology.orchestra.run.smithi037.stderr:Generating grub configuration file ... 2024-04-26T00:07:27.274 INFO:teuthology.orchestra.run.smithi037.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-26T00:07:27.280 INFO:teuthology.orchestra.run.smithi037.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-26T00:07:27.499 INFO:teuthology.orchestra.run.smithi037.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-26T00:07:27.504 INFO:teuthology.orchestra.run.smithi037.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-26T00:07:27.545 INFO:teuthology.orchestra.run.smithi037.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-26T00:07:27.552 INFO:teuthology.orchestra.run.smithi037.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-26T00:07:27.701 INFO:teuthology.orchestra.run.smithi037.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-26T00:07:27.702 INFO:teuthology.orchestra.run.smithi037.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-26T00:07:27.702 INFO:teuthology.orchestra.run.smithi037.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-26T00:07:27.720 INFO:teuthology.orchestra.run.smithi037.stderr:done 2024-04-26T00:07:27.729 DEBUG:teuthology.orchestra.run.smithi037:> sudo shutdown -r now 2024-04-26T00:07:28.031 INFO:teuthology.orchestra.run.smithi045.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-26T00:07:28.031 INFO:teuthology.orchestra.run.smithi045.stdout:Sourcing file `/etc/default/grub' 2024-04-26T00:07:28.061 INFO:teuthology.orchestra.run.smithi045.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-26T00:07:28.081 INFO:teuthology.orchestra.run.smithi045.stdout:Generating grub configuration file ... 2024-04-26T00:07:28.367 INFO:teuthology.orchestra.run.smithi045.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-26T00:07:28.384 INFO:teuthology.orchestra.run.smithi045.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-26T00:07:28.745 INFO:teuthology.orchestra.run.smithi045.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-26T00:07:28.751 INFO:teuthology.orchestra.run.smithi045.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-26T00:07:28.821 INFO:teuthology.orchestra.run.smithi045.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-26T00:07:28.827 INFO:teuthology.orchestra.run.smithi045.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-26T00:07:28.994 INFO:teuthology.orchestra.run.smithi045.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-26T00:07:28.994 INFO:teuthology.orchestra.run.smithi045.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-26T00:07:28.994 INFO:teuthology.orchestra.run.smithi045.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-26T00:07:29.046 INFO:teuthology.orchestra.run.smithi045.stdout:done 2024-04-26T00:07:29.587 DEBUG:teuthology.orchestra.run.smithi045:> dpkg -s linux-image-generic 2024-04-26T00:07:29.654 INFO:teuthology.orchestra.run.smithi045.stdout:Package: linux-image-generic 2024-04-26T00:07:29.654 INFO:teuthology.orchestra.run.smithi045.stdout:Status: install ok installed 2024-04-26T00:07:29.654 INFO:teuthology.orchestra.run.smithi045.stdout:Priority: optional 2024-04-26T00:07:29.654 INFO:teuthology.orchestra.run.smithi045.stdout:Section: kernel 2024-04-26T00:07:29.654 INFO:teuthology.orchestra.run.smithi045.stdout:Installed-Size: 21 2024-04-26T00:07:29.654 INFO:teuthology.orchestra.run.smithi045.stdout:Maintainer: Ubuntu Kernel Team 2024-04-26T00:07:29.655 INFO:teuthology.orchestra.run.smithi045.stdout:Architecture: amd64 2024-04-26T00:07:29.655 INFO:teuthology.orchestra.run.smithi045.stdout:Source: linux-meta 2024-04-26T00:07:29.655 INFO:teuthology.orchestra.run.smithi045.stdout:Version: 5.15.0.105.102 2024-04-26T00:07:29.655 INFO:teuthology.orchestra.run.smithi045.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-26T00:07:29.655 INFO:teuthology.orchestra.run.smithi045.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-26T00:07:29.655 INFO:teuthology.orchestra.run.smithi045.stdout:Recommends: thermald 2024-04-26T00:07:29.655 INFO:teuthology.orchestra.run.smithi045.stdout:Description: Generic Linux kernel image 2024-04-26T00:07:29.655 INFO:teuthology.orchestra.run.smithi045.stdout: This package will always depend on the latest generic kernel image 2024-04-26T00:07:29.655 INFO:teuthology.orchestra.run.smithi045.stdout: available. 2024-04-26T00:07:29.656 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-26T00:07:29.656 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-26T00:07:29.656 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-26T00:07:29.656 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-26T00:07:29.657 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi045.front.sepia.ceph.com, path=None, version=distro) 2024-04-26T00:07:29.657 DEBUG:teuthology.orchestra.run.smithi045:> sudo apt-get clean 2024-04-26T00:07:29.820 DEBUG:teuthology.orchestra.run.smithi045:> sudo apt-get update 2024-04-26T00:07:29.943 INFO:teuthology.orchestra.run.smithi045.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-26T00:07:29.943 INFO:teuthology.orchestra.run.smithi045.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T00:07:29.955 INFO:teuthology.orchestra.run.smithi045.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-26T00:07:29.976 INFO:teuthology.orchestra.run.smithi045.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-26T00:07:31.470 INFO:teuthology.orchestra.run.smithi045.stdout:Reading package lists... 2024-04-26T00:07:31.488 DEBUG:teuthology.orchestra.run.smithi045:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-26T00:07:31.543 INFO:teuthology.orchestra.run.smithi045.stdout:Reading package lists... 2024-04-26T00:07:31.724 INFO:teuthology.orchestra.run.smithi045.stdout:Building dependency tree... 2024-04-26T00:07:31.725 INFO:teuthology.orchestra.run.smithi045.stdout:Reading state information... 2024-04-26T00:07:31.885 INFO:teuthology.orchestra.run.smithi045.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-26T00:07:31.885 INFO:teuthology.orchestra.run.smithi045.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T00:07:31.886 INFO:teuthology.orchestra.run.smithi045.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T00:07:31.886 INFO:teuthology.orchestra.run.smithi045.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T00:07:31.886 INFO:teuthology.orchestra.run.smithi045.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T00:07:31.886 INFO:teuthology.orchestra.run.smithi045.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-26T00:07:31.886 INFO:teuthology.orchestra.run.smithi045.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-26T00:07:31.886 INFO:teuthology.orchestra.run.smithi045.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T00:07:31.925 INFO:teuthology.orchestra.run.smithi045.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-26T00:07:31.927 DEBUG:teuthology.orchestra.run.smithi045:> dpkg -s linux-image-generic 2024-04-26T00:07:31.984 INFO:teuthology.orchestra.run.smithi045.stdout:Package: linux-image-generic 2024-04-26T00:07:31.984 INFO:teuthology.orchestra.run.smithi045.stdout:Status: install ok installed 2024-04-26T00:07:31.984 INFO:teuthology.orchestra.run.smithi045.stdout:Priority: optional 2024-04-26T00:07:31.984 INFO:teuthology.orchestra.run.smithi045.stdout:Section: kernel 2024-04-26T00:07:31.985 INFO:teuthology.orchestra.run.smithi045.stdout:Installed-Size: 21 2024-04-26T00:07:31.985 INFO:teuthology.orchestra.run.smithi045.stdout:Maintainer: Ubuntu Kernel Team 2024-04-26T00:07:31.985 INFO:teuthology.orchestra.run.smithi045.stdout:Architecture: amd64 2024-04-26T00:07:31.985 INFO:teuthology.orchestra.run.smithi045.stdout:Source: linux-meta 2024-04-26T00:07:31.985 INFO:teuthology.orchestra.run.smithi045.stdout:Version: 5.15.0.105.102 2024-04-26T00:07:31.985 INFO:teuthology.orchestra.run.smithi045.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-26T00:07:31.985 INFO:teuthology.orchestra.run.smithi045.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-26T00:07:31.985 INFO:teuthology.orchestra.run.smithi045.stdout:Recommends: thermald 2024-04-26T00:07:31.985 INFO:teuthology.orchestra.run.smithi045.stdout:Description: Generic Linux kernel image 2024-04-26T00:07:31.985 INFO:teuthology.orchestra.run.smithi045.stdout: This package will always depend on the latest generic kernel image 2024-04-26T00:07:31.985 INFO:teuthology.orchestra.run.smithi045.stdout: available. 2024-04-26T00:07:31.986 DEBUG:teuthology.orchestra.run.smithi045:> mktemp 2024-04-26T00:07:32.032 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/tmp.tFV6tZC1KG 2024-04-26T00:07:32.033 DEBUG:teuthology.orchestra.run.smithi045:> sudo cp /boot/grub/grub.cfg /tmp/tmp.tFV6tZC1KG 2024-04-26T00:07:32.089 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0666 /tmp/tmp.tFV6tZC1KG 2024-04-26T00:07:32.213 DEBUG:teuthology.orchestra.remote:smithi045:/tmp/tmp.tFV6tZC1KG is 10KB 2024-04-26T00:07:32.226 DEBUG:teuthology.orchestra.run.smithi045:> rm -fr /tmp/tmp.tFV6tZC1KG 2024-04-26T00:07:32.231 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-26T00:07:32.284 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-26T00:07:32.284 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-26T00:07:32.284 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-26T00:07:32.374 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-26T00:07:32.374 DEBUG:teuthology.orchestra.run.smithi045:> sudo update-grub 2024-04-26T00:07:33.658 INFO:teuthology.orchestra.run.smithi045.stderr:Sourcing file `/etc/default/grub' 2024-04-26T00:07:33.658 INFO:teuthology.orchestra.run.smithi045.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-26T00:07:33.660 INFO:teuthology.orchestra.run.smithi045.stderr:Generating grub configuration file ... 2024-04-26T00:07:33.887 INFO:teuthology.orchestra.run.smithi045.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-26T00:07:33.893 INFO:teuthology.orchestra.run.smithi045.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-26T00:07:34.167 INFO:teuthology.orchestra.run.smithi045.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-26T00:07:34.173 INFO:teuthology.orchestra.run.smithi045.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-26T00:07:34.217 INFO:teuthology.orchestra.run.smithi045.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-26T00:07:34.223 INFO:teuthology.orchestra.run.smithi045.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-26T00:07:34.354 INFO:teuthology.orchestra.run.smithi045.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-26T00:07:34.354 INFO:teuthology.orchestra.run.smithi045.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-26T00:07:34.355 INFO:teuthology.orchestra.run.smithi045.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-26T00:07:34.380 INFO:teuthology.orchestra.run.smithi045.stderr:done 2024-04-26T00:07:34.381 DEBUG:teuthology.orchestra.run.smithi045:> sudo shutdown -r now 2024-04-26T00:07:57.747 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-26T00:07:57.748 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-26T00:07:57.749 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T00:08:04.391 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-26T00:08:04.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-04-26T00:08:04.392 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T00:08:16.159 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-26T00:08:22.814 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.45 2024-04-26T00:08:25.162 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-26T00:08:25.162 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T00:08:31.818 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-04-26T00:08:31.819 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T00:08:43.550 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-26T00:08:55.562 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-26T00:08:55.563 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T00:08:58.626 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-26T00:09:13.642 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-26T00:09:13.643 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T00:09:13.931 DEBUG:teuthology.orchestra.run.smithi037:> true 2024-04-26T00:09:14.542 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-26T00:09:14.542 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-105-generic"... 2024-04-26T00:09:14.543 DEBUG:teuthology.orchestra.run.smithi037:> uname -r 2024-04-26T00:09:14.590 INFO:teuthology.orchestra.run.smithi037.stdout:5.15.0-105-generic 2024-04-26T00:09:14.591 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-26T00:09:14.591 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-26T00:09:14.591 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-26T00:09:15.591 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-26T00:09:15.592 DEBUG:teuthology.orchestra.run.smithi037:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-26T00:09:15.673 INFO:teuthology.orchestra.run.smithi037.stdout:ttyS1 2024-04-26T00:09:15.702 DEBUG:teuthology.parallel:result is None 2024-04-26T00:09:31.821 DEBUG:teuthology.orchestra.remote:timed out 2024-04-26T00:09:43.826 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-04-26T00:09:43.827 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T00:09:44.069 DEBUG:teuthology.orchestra.run.smithi045:> true 2024-04-26T00:09:44.638 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-04-26T00:09:44.638 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-105-generic"... 2024-04-26T00:09:44.639 DEBUG:teuthology.orchestra.run.smithi045:> uname -r 2024-04-26T00:09:44.687 INFO:teuthology.orchestra.run.smithi045.stdout:5.15.0-105-generic 2024-04-26T00:09:44.687 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-26T00:09:44.687 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-26T00:09:44.688 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-26T00:09:45.689 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-26T00:09:45.689 DEBUG:teuthology.orchestra.run.smithi045:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-26T00:09:45.762 INFO:teuthology.orchestra.run.smithi045.stdout:ttyS1 2024-04-26T00:09:45.792 DEBUG:teuthology.parallel:result is None 2024-04-26T00:09:45.792 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-26T00:09:45.801 INFO:teuthology.task.internal:Creating test directory... 2024-04-26T00:09:45.802 DEBUG:teuthology.orchestra.run.smithi037:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T00:09:45.804 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T00:09:45.810 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-26T00:09:45.846 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-26T00:09:45.896 INFO:teuthology.task.internal:Creating archive directory... 2024-04-26T00:09:45.896 DEBUG:teuthology.orchestra.run.smithi037:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T00:09:45.899 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T00:09:45.918 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-26T00:09:45.947 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-26T00:09:45.947 DEBUG:teuthology.orchestra.run.smithi037:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T00:09:45.951 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T00:09:45.971 INFO:teuthology.orchestra.run.smithi037.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T00:09:45.977 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T00:09:45.980 INFO:teuthology.orchestra.run.smithi037.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T00:09:45.985 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T00:09:45.987 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-26T00:09:45.994 INFO:teuthology.task.internal:Configuring sudo... 2024-04-26T00:09:45.995 DEBUG:teuthology.orchestra.run.smithi037:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T00:09:46.028 DEBUG:teuthology.orchestra.run.smithi045:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T00:09:46.046 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-26T00:09:46.057 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-26T00:09:46.057 DEBUG:teuthology.orchestra.run.smithi037:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T00:09:46.086 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T00:09:46.094 DEBUG:teuthology.orchestra.run.smithi037:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T00:09:46.139 DEBUG:teuthology.orchestra.run.smithi037:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T00:09:46.187 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T00:09:46.187 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T00:09:46.255 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T00:09:46.262 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T00:09:46.311 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-26T00:09:46.311 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T00:09:46.386 DEBUG:teuthology.orchestra.run.smithi037:> sudo service rsyslog restart 2024-04-26T00:09:46.388 DEBUG:teuthology.orchestra.run.smithi045:> sudo service rsyslog restart 2024-04-26T00:09:46.460 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-26T00:09:46.470 INFO:teuthology.task.internal:Starting timer... 2024-04-26T00:09:46.470 INFO:teuthology.run_tasks:Running task pcp... 2024-04-26T00:09:46.516 INFO:teuthology.run_tasks:Running task selinux... 2024-04-26T00:09:46.589 DEBUG:teuthology.task.selinux:Excluding smithi037: OS 'ubuntu' does not support SELinux 2024-04-26T00:09:46.589 DEBUG:teuthology.task.selinux:Excluding smithi045: OS 'ubuntu' does not support SELinux 2024-04-26T00:09:46.589 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-26T00:09:46.589 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-04-26T00:09:46.589 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-26T00:09:46.590 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-26T00:09:46.626 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-26T00:09:46.632 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-26T00:09:46.754 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-26T00:09:46.775 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-26T00:09:46.777 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi037.front.sepia.ceph.com,smithi045.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-26T00:16:58.505 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi037.front.sepia.ceph.com'), Remote(name='ubuntu@smithi045.front.sepia.ceph.com')] 2024-04-26T00:16:58.507 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-26T00:16:58.508 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T00:16:58.595 DEBUG:teuthology.orchestra.run.smithi037:> true 2024-04-26T00:16:58.682 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-26T00:16:58.683 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-04-26T00:16:58.683 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T00:16:58.810 DEBUG:teuthology.orchestra.run.smithi045:> true 2024-04-26T00:16:58.890 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-04-26T00:16:58.891 INFO:teuthology.run_tasks:Running task clock... 2024-04-26T00:16:58.900 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-26T00:16:58.900 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T00:16:58.901 DEBUG:teuthology.orchestra.run.smithi037:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T00:16:58.906 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T00:16:58.906 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T00:16:58.930 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 00:16:58 ntpd[17129]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-26T00:16:58.930 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 00:16:58 ntpd[17129]: Command line: ntpd -gq 2024-04-26T00:16:58.930 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 00:16:58 ntpd[17129]: ---------------------------------------------------- 2024-04-26T00:16:58.931 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 00:16:58 ntpd[17129]: ntp-4 is maintained by Network Time Foundation, 2024-04-26T00:16:58.931 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 00:16:58 ntpd[17129]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-26T00:16:58.931 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 00:16:58 ntpd[17129]: corporation. Support and training for ntp-4 are 2024-04-26T00:16:58.931 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 00:16:58 ntpd[17129]: available at https://www.nwtime.org/support 2024-04-26T00:16:58.931 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 00:16:58 ntpd[17129]: ---------------------------------------------------- 2024-04-26T00:16:58.931 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 00:16:58 ntpd[17129]: proto: precision = 0.047 usec (-24) 2024-04-26T00:16:58.931 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 00:16:58 ntpd[17129]: basedate set to 2022-02-04 2024-04-26T00:16:58.931 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 00:16:58 ntpd[17129]: gps base set to 2022-02-06 (week 2196) 2024-04-26T00:16:58.931 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 00:16:58 ntpd[17129]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-26T00:16:58.932 INFO:teuthology.orchestra.run.smithi037.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-26T00:16:58.932 INFO:teuthology.orchestra.run.smithi037.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-26T00:16:58.933 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 00:16:58 ntpd[17129]: restrict ::: KOD does nothing without LIMITED. 2024-04-26T00:16:58.933 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 00:16:58 ntpd[17129]: Listen and drop on 0 v6wildcard [::]:123 2024-04-26T00:16:58.933 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 00:16:58 ntpd[17129]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-26T00:16:58.933 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 00:16:58 ntpd[17129]: Listen normally on 2 lo 127.0.0.1:123 2024-04-26T00:16:58.933 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 00:16:58 ntpd[17129]: Listen normally on 3 ens1f0 172.21.15.37:123 2024-04-26T00:16:58.933 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 00:16:58 ntpd[17129]: Listen normally on 4 lo [::1]:123 2024-04-26T00:16:58.933 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 00:16:58 ntpd[17129]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:fe88:7794%4]:123 2024-04-26T00:16:58.933 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 00:16:58 ntpd[17129]: Listening on routing socket on fd #22 for interface updates 2024-04-26T00:16:58.962 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 00:16:58 ntpd[17168]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-26T00:16:58.962 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 00:16:58 ntpd[17168]: Command line: ntpd -gq 2024-04-26T00:16:58.962 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 00:16:58 ntpd[17168]: ---------------------------------------------------- 2024-04-26T00:16:58.962 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 00:16:58 ntpd[17168]: ntp-4 is maintained by Network Time Foundation, 2024-04-26T00:16:58.962 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 00:16:58 ntpd[17168]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-26T00:16:58.963 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 00:16:58 ntpd[17168]: corporation. Support and training for ntp-4 are 2024-04-26T00:16:58.963 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 00:16:58 ntpd[17168]: available at https://www.nwtime.org/support 2024-04-26T00:16:58.963 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 00:16:58 ntpd[17168]: ---------------------------------------------------- 2024-04-26T00:16:58.963 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 00:16:58 ntpd[17168]: proto: precision = 0.047 usec (-24) 2024-04-26T00:16:58.963 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 00:16:58 ntpd[17168]: basedate set to 2022-02-04 2024-04-26T00:16:58.963 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 00:16:58 ntpd[17168]: gps base set to 2022-02-06 (week 2196) 2024-04-26T00:16:58.963 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 00:16:58 ntpd[17168]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-26T00:16:58.964 INFO:teuthology.orchestra.run.smithi045.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-26T00:16:58.964 INFO:teuthology.orchestra.run.smithi045.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-26T00:16:58.966 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 00:16:58 ntpd[17168]: restrict ::: KOD does nothing without LIMITED. 2024-04-26T00:16:58.966 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 00:16:58 ntpd[17168]: Listen and drop on 0 v6wildcard [::]:123 2024-04-26T00:16:58.966 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 00:16:58 ntpd[17168]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-26T00:16:58.967 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 00:16:58 ntpd[17168]: Listen normally on 2 lo 127.0.0.1:123 2024-04-26T00:16:58.967 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 00:16:58 ntpd[17168]: Listen normally on 3 ens1f0 172.21.15.45:123 2024-04-26T00:16:58.967 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 00:16:58 ntpd[17168]: Listen normally on 4 lo [::1]:123 2024-04-26T00:16:58.967 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 00:16:58 ntpd[17168]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:12be%4]:123 2024-04-26T00:16:58.967 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 00:16:58 ntpd[17168]: Listening on routing socket on fd #22 for interface updates 2024-04-26T00:16:59.930 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:16:59 ntpd[17129]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T00:16:59.930 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:16:59 ntpd[17129]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T00:16:59.930 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:16:59 ntpd[17129]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T00:16:59.930 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:16:59 ntpd[17129]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T00:16:59.931 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:16:59 ntpd[17129]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T00:16:59.931 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:16:59 ntpd[17129]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T00:16:59.961 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:16:59 ntpd[17168]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T00:16:59.961 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:16:59 ntpd[17168]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T00:16:59.961 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:16:59 ntpd[17168]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T00:16:59.961 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:16:59 ntpd[17168]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T00:16:59.962 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:16:59 ntpd[17168]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T00:16:59.962 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:16:59 ntpd[17168]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T00:17:01.931 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:01 ntpd[17129]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T00:17:01.931 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:01 ntpd[17129]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T00:17:01.931 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:01 ntpd[17129]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T00:17:01.931 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:01 ntpd[17129]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T00:17:01.931 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:01 ntpd[17129]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T00:17:01.931 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:01 ntpd[17129]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T00:17:01.931 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:01 ntpd[17129]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T00:17:01.931 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:01 ntpd[17129]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T00:17:01.931 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:01 ntpd[17129]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T00:17:01.932 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:01 ntpd[17129]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T00:17:01.932 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:01 ntpd[17129]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T00:17:01.932 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:01 ntpd[17129]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T00:17:01.961 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:01 ntpd[17168]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T00:17:01.961 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:01 ntpd[17168]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T00:17:01.962 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:01 ntpd[17168]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T00:17:01.962 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:01 ntpd[17168]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T00:17:01.962 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:01 ntpd[17168]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T00:17:01.962 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:01 ntpd[17168]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T00:17:01.962 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:01 ntpd[17168]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T00:17:01.962 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:01 ntpd[17168]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T00:17:02.930 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:02 ntpd[17129]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T00:17:02.931 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:02 ntpd[17129]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T00:17:02.931 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:02 ntpd[17129]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T00:17:02.931 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:02 ntpd[17129]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T00:17:02.931 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:02 ntpd[17129]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T00:17:02.931 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:02 ntpd[17129]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T00:17:02.931 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:02 ntpd[17129]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T00:17:02.931 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:02 ntpd[17129]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T00:17:02.961 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:02 ntpd[17168]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T00:17:02.961 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:02 ntpd[17168]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T00:17:02.962 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:02 ntpd[17168]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T00:17:02.962 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:02 ntpd[17168]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T00:17:02.962 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:02 ntpd[17168]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T00:17:02.962 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:02 ntpd[17168]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T00:17:03.931 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:03 ntpd[17129]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T00:17:03.931 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:03 ntpd[17129]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T00:17:03.931 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:03 ntpd[17129]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T00:17:03.931 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:03 ntpd[17129]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T00:17:03.931 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:03 ntpd[17129]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T00:17:03.931 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:03 ntpd[17129]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T00:17:03.931 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:03 ntpd[17129]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T00:17:03.931 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:03 ntpd[17129]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T00:17:03.931 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:03 ntpd[17129]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T00:17:03.932 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:03 ntpd[17129]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T00:17:03.932 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:03 ntpd[17129]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T00:17:03.932 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:03 ntpd[17129]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T00:17:03.961 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:03 ntpd[17168]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T00:17:03.962 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:03 ntpd[17168]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T00:17:03.962 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:03 ntpd[17168]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T00:17:03.962 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:03 ntpd[17168]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T00:17:03.962 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:03 ntpd[17168]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T00:17:03.962 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:03 ntpd[17168]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T00:17:03.962 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:03 ntpd[17168]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T00:17:03.962 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:03 ntpd[17168]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T00:17:03.962 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:03 ntpd[17168]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T00:17:03.962 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:03 ntpd[17168]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T00:17:03.962 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:03 ntpd[17168]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T00:17:03.962 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:03 ntpd[17168]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T00:17:04.930 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:04 ntpd[17129]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T00:17:04.931 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:04 ntpd[17129]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T00:17:04.931 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:04 ntpd[17129]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T00:17:04.931 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:04 ntpd[17129]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T00:17:04.931 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:04 ntpd[17129]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T00:17:04.931 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:04 ntpd[17129]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T00:17:04.961 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:04 ntpd[17168]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T00:17:04.961 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:04 ntpd[17168]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T00:17:04.961 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:04 ntpd[17168]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T00:17:04.961 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:04 ntpd[17168]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T00:17:04.962 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:04 ntpd[17168]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T00:17:04.962 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:04 ntpd[17168]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T00:17:05.931 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:05 ntpd[17129]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T00:17:05.931 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:05 ntpd[17129]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T00:17:05.931 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:05 ntpd[17129]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T00:17:05.931 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:05 ntpd[17129]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T00:17:05.931 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:05 ntpd[17129]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T00:17:05.931 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:05 ntpd[17129]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T00:17:05.931 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:05 ntpd[17129]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T00:17:05.932 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:05 ntpd[17129]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T00:17:05.932 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:05 ntpd[17129]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T00:17:05.932 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:05 ntpd[17129]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T00:17:05.932 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:05 ntpd[17129]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T00:17:05.932 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:05 ntpd[17129]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T00:17:05.932 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 00:17:05 ntpd[17129]: ntpd: time slew -0.001571 s 2024-04-26T00:17:05.932 INFO:teuthology.orchestra.run.smithi037.stdout:ntpd: time slew -0.001571s 2024-04-26T00:17:05.933 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:05 ntpd[17129]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-26T00:17:05.933 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 00:17:05 ntpd[17129]: can't open /var/log/ntpstats/loopstats.20240426: Permission denied 2024-04-26T00:17:05.961 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:05 ntpd[17168]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T00:17:05.962 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:05 ntpd[17168]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T00:17:05.962 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:05 ntpd[17168]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T00:17:05.962 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:05 ntpd[17168]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T00:17:05.962 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:05 ntpd[17168]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T00:17:05.962 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:05 ntpd[17168]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T00:17:05.962 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:05 ntpd[17168]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T00:17:05.962 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:05 ntpd[17168]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T00:17:05.962 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 00:17:05 ntpd[17168]: ntpd: time slew -0.000342 s 2024-04-26T00:17:05.963 INFO:teuthology.orchestra.run.smithi045.stdout:ntpd: time slew -0.000342s 2024-04-26T00:17:05.963 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:05 ntpd[17168]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-26T00:17:05.963 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 00:17:05 ntpd[17168]: can't open /var/log/ntpstats/loopstats.20240426: Permission denied 2024-04-26T00:17:05.983 INFO:teuthology.orchestra.run.smithi037.stdout: remote refid st t when poll reach delay offset jitter 2024-04-26T00:17:05.983 INFO:teuthology.orchestra.run.smithi037.stdout:============================================================================== 2024-04-26T00:17:05.983 INFO:teuthology.orchestra.run.smithi037.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T00:17:05.983 INFO:teuthology.orchestra.run.smithi037.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T00:17:05.983 INFO:teuthology.orchestra.run.smithi037.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T00:17:05.983 INFO:teuthology.orchestra.run.smithi037.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T00:17:06.015 INFO:teuthology.orchestra.run.smithi045.stdout: remote refid st t when poll reach delay offset jitter 2024-04-26T00:17:06.015 INFO:teuthology.orchestra.run.smithi045.stdout:============================================================================== 2024-04-26T00:17:06.015 INFO:teuthology.orchestra.run.smithi045.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T00:17:06.015 INFO:teuthology.orchestra.run.smithi045.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T00:17:06.015 INFO:teuthology.orchestra.run.smithi045.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T00:17:06.015 INFO:teuthology.orchestra.run.smithi045.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T00:17:06.016 INFO:teuthology.run_tasks:Running task install... 2024-04-26T00:17:06.025 DEBUG:teuthology.task.install:project ceph 2024-04-26T00:17:06.026 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}} 2024-04-26T00:17:06.026 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'} 2024-04-26T00:17:06.026 INFO:teuthology.task.install:Using flavor: default 2024-04-26T00:17:06.032 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-26T00:17:06.032 INFO:teuthology.task.install:extra packages: [] 2024-04-26T00:17:06.032 DEBUG:teuthology.orchestra.run.smithi037:> sudo apt-key list | grep Ceph 2024-04-26T00:17:06.033 DEBUG:teuthology.orchestra.run.smithi045:> sudo apt-key list | grep Ceph 2024-04-26T00:17:06.091 INFO:teuthology.orchestra.run.smithi037.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-26T00:17:06.118 INFO:teuthology.orchestra.run.smithi045.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-26T00:17:06.118 INFO:teuthology.orchestra.run.smithi037.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-26T00:17:06.119 INFO:teuthology.orchestra.run.smithi037.stdout:uid [ unknown] Ceph.com (release key) 2024-04-26T00:17:06.120 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-26T00:17:06.120 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-26T00:17:06.146 INFO:teuthology.orchestra.run.smithi045.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-26T00:17:06.146 INFO:teuthology.orchestra.run.smithi045.stdout:uid [ unknown] Ceph.com (release key) 2024-04-26T00:17:06.146 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-26T00:17:06.146 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-26T00:17:06.346 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default/ 2024-04-26T00:17:06.346 INFO:teuthology.task.install.deb:Package version is 19.0.0-2445-gd2c45d97-1jammy 2024-04-26T00:17:06.359 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default/ 2024-04-26T00:17:06.360 INFO:teuthology.task.install.deb:Package version is 19.0.0-2445-gd2c45d97-1jammy 2024-04-26T00:17:06.521 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T00:17:06.521 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-26T00:17:06.536 DEBUG:teuthology.orchestra.run.smithi037:> sudo apt-get update 2024-04-26T00:17:06.768 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-26T00:17:06.768 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-26T00:17:06.783 DEBUG:teuthology.orchestra.run.smithi045:> sudo apt-get update 2024-04-26T00:17:06.825 INFO:teuthology.orchestra.run.smithi037.stdout:Ign:1 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy InRelease 2024-04-26T00:17:06.862 INFO:teuthology.orchestra.run.smithi037.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-26T00:17:06.900 INFO:teuthology.orchestra.run.smithi037.stdout:Ign:3 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-26T00:17:06.936 INFO:teuthology.orchestra.run.smithi037.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,416 B] 2024-04-26T00:17:06.984 INFO:teuthology.orchestra.run.smithi037.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-04-26T00:17:07.071 INFO:teuthology.orchestra.run.smithi045.stdout:Ign:1 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy InRelease 2024-04-26T00:17:07.110 INFO:teuthology.orchestra.run.smithi045.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-26T00:17:07.151 INFO:teuthology.orchestra.run.smithi045.stdout:Ign:3 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-26T00:17:07.260 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T00:17:07.271 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:7 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-26T00:17:07.293 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-26T00:17:07.430 INFO:teuthology.orchestra.run.smithi045.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-04-26T00:17:07.515 INFO:teuthology.orchestra.run.smithi045.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,416 B] 2024-04-26T00:17:08.946 INFO:teuthology.orchestra.run.smithi045.stdout:Hit:6 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-26T00:17:09.334 INFO:teuthology.orchestra.run.smithi045.stdout:Hit:7 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T00:17:09.355 INFO:teuthology.orchestra.run.smithi045.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-26T00:17:09.375 INFO:teuthology.orchestra.run.smithi045.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-26T00:17:09.663 INFO:teuthology.orchestra.run.smithi045.stdout:Fetched 28.9 kB in 3s (10.7 kB/s) 2024-04-26T00:17:09.883 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:9 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-26T00:17:10.059 INFO:teuthology.orchestra.run.smithi037.stdout:Fetched 28.9 kB in 3s (8,698 B/s) 2024-04-26T00:17:10.637 INFO:teuthology.orchestra.run.smithi045.stdout:Reading package lists... 2024-04-26T00:17:10.653 DEBUG:teuthology.orchestra.run.smithi045:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2445-gd2c45d97-1jammy cephadm=19.0.0-2445-gd2c45d97-1jammy ceph-mds=19.0.0-2445-gd2c45d97-1jammy ceph-mgr=19.0.0-2445-gd2c45d97-1jammy ceph-common=19.0.0-2445-gd2c45d97-1jammy ceph-fuse=19.0.0-2445-gd2c45d97-1jammy ceph-test=19.0.0-2445-gd2c45d97-1jammy ceph-volume=19.0.0-2445-gd2c45d97-1jammy radosgw=19.0.0-2445-gd2c45d97-1jammy python3-rados=19.0.0-2445-gd2c45d97-1jammy python3-rgw=19.0.0-2445-gd2c45d97-1jammy python3-cephfs=19.0.0-2445-gd2c45d97-1jammy python3-rbd=19.0.0-2445-gd2c45d97-1jammy libcephfs2=19.0.0-2445-gd2c45d97-1jammy libcephfs-dev=19.0.0-2445-gd2c45d97-1jammy librados2=19.0.0-2445-gd2c45d97-1jammy librbd1=19.0.0-2445-gd2c45d97-1jammy rbd-fuse=19.0.0-2445-gd2c45d97-1jammy 2024-04-26T00:17:10.716 INFO:teuthology.orchestra.run.smithi045.stdout:Reading package lists... 2024-04-26T00:17:10.927 INFO:teuthology.orchestra.run.smithi045.stdout:Building dependency tree... 2024-04-26T00:17:10.928 INFO:teuthology.orchestra.run.smithi045.stdout:Reading state information... 2024-04-26T00:17:11.053 INFO:teuthology.orchestra.run.smithi037.stdout:Reading package lists... 2024-04-26T00:17:11.070 DEBUG:teuthology.orchestra.run.smithi037:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2445-gd2c45d97-1jammy cephadm=19.0.0-2445-gd2c45d97-1jammy ceph-mds=19.0.0-2445-gd2c45d97-1jammy ceph-mgr=19.0.0-2445-gd2c45d97-1jammy ceph-common=19.0.0-2445-gd2c45d97-1jammy ceph-fuse=19.0.0-2445-gd2c45d97-1jammy ceph-test=19.0.0-2445-gd2c45d97-1jammy ceph-volume=19.0.0-2445-gd2c45d97-1jammy radosgw=19.0.0-2445-gd2c45d97-1jammy python3-rados=19.0.0-2445-gd2c45d97-1jammy python3-rgw=19.0.0-2445-gd2c45d97-1jammy python3-cephfs=19.0.0-2445-gd2c45d97-1jammy python3-rbd=19.0.0-2445-gd2c45d97-1jammy libcephfs2=19.0.0-2445-gd2c45d97-1jammy libcephfs-dev=19.0.0-2445-gd2c45d97-1jammy librados2=19.0.0-2445-gd2c45d97-1jammy librbd1=19.0.0-2445-gd2c45d97-1jammy rbd-fuse=19.0.0-2445-gd2c45d97-1jammy 2024-04-26T00:17:11.134 INFO:teuthology.orchestra.run.smithi037.stdout:Reading package lists... 2024-04-26T00:17:11.138 INFO:teuthology.orchestra.run.smithi045.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T00:17:11.138 INFO:teuthology.orchestra.run.smithi045.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T00:17:11.138 INFO:teuthology.orchestra.run.smithi045.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T00:17:11.139 INFO:teuthology.orchestra.run.smithi045.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T00:17:11.139 INFO:teuthology.orchestra.run.smithi045.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-26T00:17:11.139 INFO:teuthology.orchestra.run.smithi045.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-26T00:17:11.139 INFO:teuthology.orchestra.run.smithi045.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T00:17:11.140 INFO:teuthology.orchestra.run.smithi045.stdout:The following additional packages will be installed: 2024-04-26T00:17:11.141 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-26T00:17:11.141 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-26T00:17:11.141 INFO:teuthology.orchestra.run.smithi045.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-26T00:17:11.141 INFO:teuthology.orchestra.run.smithi045.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-26T00:17:11.142 INFO:teuthology.orchestra.run.smithi045.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-26T00:17:11.142 INFO:teuthology.orchestra.run.smithi045.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-26T00:17:11.142 INFO:teuthology.orchestra.run.smithi045.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-26T00:17:11.142 INFO:teuthology.orchestra.run.smithi045.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-26T00:17:11.142 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-26T00:17:11.142 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-26T00:17:11.142 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-26T00:17:11.143 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-26T00:17:11.143 INFO:teuthology.orchestra.run.smithi045.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-26T00:17:11.143 INFO:teuthology.orchestra.run.smithi045.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-26T00:17:11.143 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-26T00:17:11.143 INFO:teuthology.orchestra.run.smithi045.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-26T00:17:11.143 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-26T00:17:11.143 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-26T00:17:11.143 INFO:teuthology.orchestra.run.smithi045.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-26T00:17:11.143 INFO:teuthology.orchestra.run.smithi045.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-26T00:17:11.143 INFO:teuthology.orchestra.run.smithi045.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-26T00:17:11.143 INFO:teuthology.orchestra.run.smithi045.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-26T00:17:11.143 INFO:teuthology.orchestra.run.smithi045.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-26T00:17:11.143 INFO:teuthology.orchestra.run.smithi045.stdout: xmlstarlet zip 2024-04-26T00:17:11.144 INFO:teuthology.orchestra.run.smithi045.stdout:Suggested packages: 2024-04-26T00:17:11.144 INFO:teuthology.orchestra.run.smithi045.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-26T00:17:11.144 INFO:teuthology.orchestra.run.smithi045.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-26T00:17:11.144 INFO:teuthology.orchestra.run.smithi045.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-26T00:17:11.144 INFO:teuthology.orchestra.run.smithi045.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-26T00:17:11.144 INFO:teuthology.orchestra.run.smithi045.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-26T00:17:11.144 INFO:teuthology.orchestra.run.smithi045.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-26T00:17:11.144 INFO:teuthology.orchestra.run.smithi045.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-26T00:17:11.144 INFO:teuthology.orchestra.run.smithi045.stdout:Recommended packages: 2024-04-26T00:17:11.144 INFO:teuthology.orchestra.run.smithi045.stdout: btrfs-tools 2024-04-26T00:17:11.251 INFO:teuthology.orchestra.run.smithi045.stdout:The following NEW packages will be installed: 2024-04-26T00:17:11.251 INFO:teuthology.orchestra.run.smithi045.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-26T00:17:11.252 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-26T00:17:11.252 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-26T00:17:11.252 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-26T00:17:11.252 INFO:teuthology.orchestra.run.smithi045.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-26T00:17:11.252 INFO:teuthology.orchestra.run.smithi045.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-26T00:17:11.253 INFO:teuthology.orchestra.run.smithi045.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-26T00:17:11.253 INFO:teuthology.orchestra.run.smithi045.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-26T00:17:11.253 INFO:teuthology.orchestra.run.smithi045.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-26T00:17:11.253 INFO:teuthology.orchestra.run.smithi045.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-26T00:17:11.253 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-26T00:17:11.253 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-26T00:17:11.253 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-26T00:17:11.253 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-26T00:17:11.253 INFO:teuthology.orchestra.run.smithi045.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-26T00:17:11.254 INFO:teuthology.orchestra.run.smithi045.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-26T00:17:11.254 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-26T00:17:11.254 INFO:teuthology.orchestra.run.smithi045.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-26T00:17:11.254 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-26T00:17:11.254 INFO:teuthology.orchestra.run.smithi045.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-26T00:17:11.254 INFO:teuthology.orchestra.run.smithi045.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-26T00:17:11.254 INFO:teuthology.orchestra.run.smithi045.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-26T00:17:11.254 INFO:teuthology.orchestra.run.smithi045.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-26T00:17:11.254 INFO:teuthology.orchestra.run.smithi045.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-26T00:17:11.254 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-26T00:17:11.254 INFO:teuthology.orchestra.run.smithi045.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-26T00:17:11.334 INFO:teuthology.orchestra.run.smithi037.stdout:Building dependency tree... 2024-04-26T00:17:11.334 INFO:teuthology.orchestra.run.smithi037.stdout:Reading state information... 2024-04-26T00:17:11.397 INFO:teuthology.orchestra.run.smithi045.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-26T00:17:11.397 INFO:teuthology.orchestra.run.smithi045.stdout:Need to get 179 MB of archives. 2024-04-26T00:17:11.397 INFO:teuthology.orchestra.run.smithi045.stdout:After this operation, 822 MB of additional disk space will be used. 2024-04-26T00:17:11.398 INFO:teuthology.orchestra.run.smithi045.stdout:Get:1 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2445-gd2c45d97-1jammy [3,541 kB] 2024-04-26T00:17:11.444 INFO:teuthology.orchestra.run.smithi045.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-26T00:17:11.539 INFO:teuthology.orchestra.run.smithi037.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T00:17:11.539 INFO:teuthology.orchestra.run.smithi037.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T00:17:11.540 INFO:teuthology.orchestra.run.smithi037.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T00:17:11.540 INFO:teuthology.orchestra.run.smithi037.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T00:17:11.540 INFO:teuthology.orchestra.run.smithi037.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-26T00:17:11.540 INFO:teuthology.orchestra.run.smithi037.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-26T00:17:11.540 INFO:teuthology.orchestra.run.smithi037.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T00:17:11.541 INFO:teuthology.orchestra.run.smithi037.stdout:The following additional packages will be installed: 2024-04-26T00:17:11.541 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-26T00:17:11.541 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-26T00:17:11.542 INFO:teuthology.orchestra.run.smithi037.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-26T00:17:11.542 INFO:teuthology.orchestra.run.smithi037.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-26T00:17:11.543 INFO:teuthology.orchestra.run.smithi037.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-26T00:17:11.543 INFO:teuthology.orchestra.run.smithi037.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-26T00:17:11.543 INFO:teuthology.orchestra.run.smithi037.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-26T00:17:11.543 INFO:teuthology.orchestra.run.smithi037.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-26T00:17:11.543 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-26T00:17:11.543 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-26T00:17:11.543 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-26T00:17:11.543 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-26T00:17:11.543 INFO:teuthology.orchestra.run.smithi037.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-26T00:17:11.543 INFO:teuthology.orchestra.run.smithi037.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-26T00:17:11.544 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-26T00:17:11.544 INFO:teuthology.orchestra.run.smithi037.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-26T00:17:11.544 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-26T00:17:11.544 INFO:teuthology.orchestra.run.smithi037.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-26T00:17:11.544 INFO:teuthology.orchestra.run.smithi037.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-26T00:17:11.544 INFO:teuthology.orchestra.run.smithi037.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-26T00:17:11.544 INFO:teuthology.orchestra.run.smithi037.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-26T00:17:11.544 INFO:teuthology.orchestra.run.smithi037.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-26T00:17:11.544 INFO:teuthology.orchestra.run.smithi037.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-26T00:17:11.544 INFO:teuthology.orchestra.run.smithi037.stdout: xmlstarlet zip 2024-04-26T00:17:11.545 INFO:teuthology.orchestra.run.smithi037.stdout:Suggested packages: 2024-04-26T00:17:11.545 INFO:teuthology.orchestra.run.smithi037.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-26T00:17:11.545 INFO:teuthology.orchestra.run.smithi037.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-26T00:17:11.545 INFO:teuthology.orchestra.run.smithi037.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-26T00:17:11.545 INFO:teuthology.orchestra.run.smithi037.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-26T00:17:11.545 INFO:teuthology.orchestra.run.smithi037.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-26T00:17:11.545 INFO:teuthology.orchestra.run.smithi037.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-26T00:17:11.545 INFO:teuthology.orchestra.run.smithi037.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-26T00:17:11.545 INFO:teuthology.orchestra.run.smithi037.stdout:Recommended packages: 2024-04-26T00:17:11.545 INFO:teuthology.orchestra.run.smithi037.stdout: btrfs-tools 2024-04-26T00:17:11.607 INFO:teuthology.orchestra.run.smithi045.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2445-gd2c45d97-1jammy [3,218 kB] 2024-04-26T00:17:11.652 INFO:teuthology.orchestra.run.smithi037.stdout:The following NEW packages will be installed: 2024-04-26T00:17:11.652 INFO:teuthology.orchestra.run.smithi037.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-26T00:17:11.652 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-26T00:17:11.652 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-26T00:17:11.653 INFO:teuthology.orchestra.run.smithi037.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-26T00:17:11.653 INFO:teuthology.orchestra.run.smithi037.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-26T00:17:11.653 INFO:teuthology.orchestra.run.smithi037.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-26T00:17:11.654 INFO:teuthology.orchestra.run.smithi037.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-26T00:17:11.654 INFO:teuthology.orchestra.run.smithi037.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-26T00:17:11.654 INFO:teuthology.orchestra.run.smithi037.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-26T00:17:11.654 INFO:teuthology.orchestra.run.smithi037.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-26T00:17:11.654 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-26T00:17:11.654 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-26T00:17:11.654 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-26T00:17:11.654 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-26T00:17:11.655 INFO:teuthology.orchestra.run.smithi037.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-26T00:17:11.655 INFO:teuthology.orchestra.run.smithi037.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-26T00:17:11.655 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-26T00:17:11.655 INFO:teuthology.orchestra.run.smithi037.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-26T00:17:11.655 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-26T00:17:11.655 INFO:teuthology.orchestra.run.smithi037.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-26T00:17:11.655 INFO:teuthology.orchestra.run.smithi037.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-26T00:17:11.655 INFO:teuthology.orchestra.run.smithi037.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-26T00:17:11.655 INFO:teuthology.orchestra.run.smithi037.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-26T00:17:11.655 INFO:teuthology.orchestra.run.smithi037.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-26T00:17:11.655 INFO:teuthology.orchestra.run.smithi037.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-26T00:17:11.655 INFO:teuthology.orchestra.run.smithi037.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-26T00:17:11.704 INFO:teuthology.orchestra.run.smithi045.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2445-gd2c45d97-1jammy [711 kB] 2024-04-26T00:17:11.722 INFO:teuthology.orchestra.run.smithi037.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-26T00:17:11.722 INFO:teuthology.orchestra.run.smithi037.stdout:Need to get 179 MB of archives. 2024-04-26T00:17:11.722 INFO:teuthology.orchestra.run.smithi037.stdout:After this operation, 822 MB of additional disk space will be used. 2024-04-26T00:17:11.722 INFO:teuthology.orchestra.run.smithi037.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-26T00:17:11.727 INFO:teuthology.orchestra.run.smithi045.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2445-gd2c45d97-1jammy [356 kB] 2024-04-26T00:17:11.738 INFO:teuthology.orchestra.run.smithi045.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2445-gd2c45d97-1jammy [32.8 kB] 2024-04-26T00:17:11.739 INFO:teuthology.orchestra.run.smithi045.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2445-gd2c45d97-1jammy [180 kB] 2024-04-26T00:17:11.745 INFO:teuthology.orchestra.run.smithi045.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2445-gd2c45d97-1jammy [65.8 kB] 2024-04-26T00:17:11.747 INFO:teuthology.orchestra.run.smithi045.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2445-gd2c45d97-1jammy [330 kB] 2024-04-26T00:17:11.758 INFO:teuthology.orchestra.run.smithi045.stdout:Get:10 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2445-gd2c45d97-1jammy [6,775 kB] 2024-04-26T00:17:11.829 INFO:teuthology.orchestra.run.smithi037.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-26T00:17:11.832 INFO:teuthology.orchestra.run.smithi037.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-26T00:17:11.853 INFO:teuthology.orchestra.run.smithi037.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-26T00:17:11.920 INFO:teuthology.orchestra.run.smithi037.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-26T00:17:11.924 INFO:teuthology.orchestra.run.smithi037.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-26T00:17:11.937 INFO:teuthology.orchestra.run.smithi037.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-26T00:17:11.942 INFO:teuthology.orchestra.run.smithi037.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-26T00:17:11.942 INFO:teuthology.orchestra.run.smithi037.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-26T00:17:11.943 INFO:teuthology.orchestra.run.smithi037.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-26T00:17:11.952 INFO:teuthology.orchestra.run.smithi037.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-26T00:17:11.955 INFO:teuthology.orchestra.run.smithi037.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-26T00:17:11.957 INFO:teuthology.orchestra.run.smithi037.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-26T00:17:11.958 INFO:teuthology.orchestra.run.smithi037.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-26T00:17:11.965 INFO:teuthology.orchestra.run.smithi037.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-26T00:17:11.967 INFO:teuthology.orchestra.run.smithi037.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-26T00:17:11.970 INFO:teuthology.orchestra.run.smithi037.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-26T00:17:11.983 INFO:teuthology.orchestra.run.smithi045.stdout:Get:11 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2445-gd2c45d97-1jammy [112 kB] 2024-04-26T00:17:11.986 INFO:teuthology.orchestra.run.smithi045.stdout:Get:12 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2445-gd2c45d97-1jammy [467 kB] 2024-04-26T00:17:11.986 INFO:teuthology.orchestra.run.smithi037.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-26T00:17:11.987 INFO:teuthology.orchestra.run.smithi037.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-26T00:17:11.987 INFO:teuthology.orchestra.run.smithi037.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-26T00:17:11.988 INFO:teuthology.orchestra.run.smithi037.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-26T00:17:11.988 INFO:teuthology.orchestra.run.smithi037.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-26T00:17:11.989 INFO:teuthology.orchestra.run.smithi037.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-26T00:17:11.989 INFO:teuthology.orchestra.run.smithi037.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-26T00:17:12.000 INFO:teuthology.orchestra.run.smithi045.stdout:Get:13 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2445-gd2c45d97-1jammy [25.6 MB] 2024-04-26T00:17:12.006 INFO:teuthology.orchestra.run.smithi037.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-26T00:17:12.008 INFO:teuthology.orchestra.run.smithi037.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-26T00:17:12.008 INFO:teuthology.orchestra.run.smithi037.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-26T00:17:12.027 INFO:teuthology.orchestra.run.smithi037.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-26T00:17:12.028 INFO:teuthology.orchestra.run.smithi037.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-26T00:17:12.028 INFO:teuthology.orchestra.run.smithi037.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-26T00:17:12.029 INFO:teuthology.orchestra.run.smithi037.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-26T00:17:12.030 INFO:teuthology.orchestra.run.smithi037.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-26T00:17:12.030 INFO:teuthology.orchestra.run.smithi037.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-26T00:17:12.030 INFO:teuthology.orchestra.run.smithi037.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-26T00:17:12.047 INFO:teuthology.orchestra.run.smithi037.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-26T00:17:12.048 INFO:teuthology.orchestra.run.smithi037.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-26T00:17:12.049 INFO:teuthology.orchestra.run.smithi037.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-26T00:17:12.068 INFO:teuthology.orchestra.run.smithi037.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-26T00:17:12.068 INFO:teuthology.orchestra.run.smithi037.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-26T00:17:12.069 INFO:teuthology.orchestra.run.smithi037.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-26T00:17:12.070 INFO:teuthology.orchestra.run.smithi037.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-26T00:17:12.071 INFO:teuthology.orchestra.run.smithi037.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-26T00:17:12.073 INFO:teuthology.orchestra.run.smithi037.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-26T00:17:12.074 INFO:teuthology.orchestra.run.smithi037.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-26T00:17:12.089 INFO:teuthology.orchestra.run.smithi037.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-26T00:17:12.090 INFO:teuthology.orchestra.run.smithi037.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-26T00:17:12.092 INFO:teuthology.orchestra.run.smithi037.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-26T00:17:12.109 INFO:teuthology.orchestra.run.smithi037.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-26T00:17:12.110 INFO:teuthology.orchestra.run.smithi037.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-26T00:17:12.112 INFO:teuthology.orchestra.run.smithi037.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-26T00:17:12.169 INFO:teuthology.orchestra.run.smithi037.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-26T00:17:12.169 INFO:teuthology.orchestra.run.smithi037.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-26T00:17:12.171 INFO:teuthology.orchestra.run.smithi037.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-26T00:17:12.171 INFO:teuthology.orchestra.run.smithi037.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-26T00:17:12.172 INFO:teuthology.orchestra.run.smithi037.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-26T00:17:12.175 INFO:teuthology.orchestra.run.smithi045.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-26T00:17:12.191 INFO:teuthology.orchestra.run.smithi037.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-26T00:17:12.193 INFO:teuthology.orchestra.run.smithi037.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-26T00:17:12.193 INFO:teuthology.orchestra.run.smithi037.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-26T00:17:12.218 INFO:teuthology.orchestra.run.smithi037.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-26T00:17:12.218 INFO:teuthology.orchestra.run.smithi037.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-26T00:17:12.230 INFO:teuthology.orchestra.run.smithi037.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-26T00:17:12.230 INFO:teuthology.orchestra.run.smithi037.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-26T00:17:12.230 INFO:teuthology.orchestra.run.smithi037.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-26T00:17:12.230 INFO:teuthology.orchestra.run.smithi037.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-26T00:17:12.230 INFO:teuthology.orchestra.run.smithi037.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-26T00:17:12.230 INFO:teuthology.orchestra.run.smithi037.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-26T00:17:12.232 INFO:teuthology.orchestra.run.smithi037.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-26T00:17:12.237 INFO:teuthology.orchestra.run.smithi045.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-26T00:17:12.251 INFO:teuthology.orchestra.run.smithi037.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-26T00:17:12.253 INFO:teuthology.orchestra.run.smithi037.stdout:Get:69 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2445-gd2c45d97-1jammy [3,541 kB] 2024-04-26T00:17:12.254 INFO:teuthology.orchestra.run.smithi037.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-26T00:17:12.255 INFO:teuthology.orchestra.run.smithi037.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-26T00:17:12.272 INFO:teuthology.orchestra.run.smithi037.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-26T00:17:12.274 INFO:teuthology.orchestra.run.smithi037.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-26T00:17:12.275 INFO:teuthology.orchestra.run.smithi037.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-26T00:17:12.276 INFO:teuthology.orchestra.run.smithi037.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-26T00:17:12.280 INFO:teuthology.orchestra.run.smithi037.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-26T00:17:12.281 INFO:teuthology.orchestra.run.smithi037.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-26T00:17:12.284 INFO:teuthology.orchestra.run.smithi037.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-26T00:17:12.292 INFO:teuthology.orchestra.run.smithi037.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-26T00:17:12.293 INFO:teuthology.orchestra.run.smithi037.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-26T00:17:12.293 INFO:teuthology.orchestra.run.smithi037.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-26T00:17:12.313 INFO:teuthology.orchestra.run.smithi037.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-26T00:17:12.314 INFO:teuthology.orchestra.run.smithi037.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-26T00:17:12.321 INFO:teuthology.orchestra.run.smithi037.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-26T00:17:12.321 INFO:teuthology.orchestra.run.smithi037.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-26T00:17:12.321 INFO:teuthology.orchestra.run.smithi037.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-26T00:17:12.323 INFO:teuthology.orchestra.run.smithi037.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-26T00:17:12.323 INFO:teuthology.orchestra.run.smithi037.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-26T00:17:12.694 INFO:teuthology.orchestra.run.smithi045.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-26T00:17:12.920 INFO:teuthology.orchestra.run.smithi045.stdout:Get:17 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2445-gd2c45d97-1jammy [5,084 kB] 2024-04-26T00:17:13.023 INFO:teuthology.orchestra.run.smithi045.stdout:Get:18 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2445-gd2c45d97-1jammy [244 kB] 2024-04-26T00:17:13.030 INFO:teuthology.orchestra.run.smithi045.stdout:Get:19 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2445-gd2c45d97-1jammy [124 kB] 2024-04-26T00:17:13.034 INFO:teuthology.orchestra.run.smithi045.stdout:Get:20 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2445-gd2c45d97-1jammy [1,526 kB] 2024-04-26T00:17:13.085 INFO:teuthology.orchestra.run.smithi045.stdout:Get:21 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2445-gd2c45d97-1jammy [6,245 kB] 2024-04-26T00:17:13.295 INFO:teuthology.orchestra.run.smithi045.stdout:Get:22 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2445-gd2c45d97-1jammy [22.7 MB] 2024-04-26T00:17:14.242 INFO:teuthology.orchestra.run.smithi045.stdout:Get:23 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2445-gd2c45d97-1jammy [14.0 kB] 2024-04-26T00:17:14.243 INFO:teuthology.orchestra.run.smithi045.stdout:Get:24 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2445-gd2c45d97-1jammy [869 kB] 2024-04-26T00:17:14.260 INFO:teuthology.orchestra.run.smithi045.stdout:Get:25 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2445-gd2c45d97-1jammy [2,437 kB] 2024-04-26T00:17:14.304 INFO:teuthology.orchestra.run.smithi045.stdout:Get:26 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2445-gd2c45d97-1jammy [791 kB] 2024-04-26T00:17:14.314 INFO:teuthology.orchestra.run.smithi045.stdout:Get:27 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2445-gd2c45d97-1jammy [147 kB] 2024-04-26T00:17:14.316 INFO:teuthology.orchestra.run.smithi045.stdout:Get:28 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2445-gd2c45d97-1jammy [2,293 kB] 2024-04-26T00:17:14.373 INFO:teuthology.orchestra.run.smithi045.stdout:Get:29 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2445-gd2c45d97-1jammy [8,625 kB] 2024-04-26T00:17:14.743 INFO:teuthology.orchestra.run.smithi045.stdout:Get:30 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2445-gd2c45d97-1jammy [14.2 kB] 2024-04-26T00:17:14.743 INFO:teuthology.orchestra.run.smithi045.stdout:Get:31 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2445-gd2c45d97-1jammy [50.7 MB] 2024-04-26T00:17:17.249 INFO:teuthology.orchestra.run.smithi045.stdout:Get:32 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2445-gd2c45d97-1jammy [121 kB] 2024-04-26T00:17:17.252 INFO:teuthology.orchestra.run.smithi045.stdout:Get:33 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2445-gd2c45d97-1jammy [38.9 kB] 2024-04-26T00:17:17.253 INFO:teuthology.orchestra.run.smithi045.stdout:Get:34 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2445-gd2c45d97-1jammy [13.4 MB] 2024-04-26T00:17:17.697 INFO:teuthology.orchestra.run.smithi045.stdout:Get:35 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2445-gd2c45d97-1jammy [91.4 kB] 2024-04-26T00:17:17.710 INFO:teuthology.orchestra.run.smithi037.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2445-gd2c45d97-1jammy [3,218 kB] 2024-04-26T00:17:18.733 INFO:teuthology.orchestra.run.smithi045.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-26T00:17:19.002 INFO:teuthology.orchestra.run.smithi037.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2445-gd2c45d97-1jammy [711 kB] 2024-04-26T00:17:19.249 INFO:teuthology.orchestra.run.smithi037.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2445-gd2c45d97-1jammy [356 kB] 2024-04-26T00:17:19.314 INFO:teuthology.orchestra.run.smithi045.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-26T00:17:19.357 INFO:teuthology.orchestra.run.smithi037.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2445-gd2c45d97-1jammy [32.8 kB] 2024-04-26T00:17:19.358 INFO:teuthology.orchestra.run.smithi037.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2445-gd2c45d97-1jammy [180 kB] 2024-04-26T00:17:19.427 INFO:teuthology.orchestra.run.smithi037.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2445-gd2c45d97-1jammy [65.8 kB] 2024-04-26T00:17:19.430 INFO:teuthology.orchestra.run.smithi037.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2445-gd2c45d97-1jammy [330 kB] 2024-04-26T00:17:19.536 INFO:teuthology.orchestra.run.smithi037.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2445-gd2c45d97-1jammy [6,775 kB] 2024-04-26T00:17:21.388 INFO:teuthology.orchestra.run.smithi037.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2445-gd2c45d97-1jammy [112 kB] 2024-04-26T00:17:21.390 INFO:teuthology.orchestra.run.smithi037.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2445-gd2c45d97-1jammy [467 kB] 2024-04-26T00:17:21.498 INFO:teuthology.orchestra.run.smithi037.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2445-gd2c45d97-1jammy [25.6 MB] 2024-04-26T00:17:21.534 INFO:teuthology.orchestra.run.smithi045.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-26T00:17:22.193 INFO:teuthology.orchestra.run.smithi045.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-26T00:17:22.216 INFO:teuthology.orchestra.run.smithi045.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-26T00:17:22.246 INFO:teuthology.orchestra.run.smithi045.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-26T00:17:24.054 INFO:teuthology.orchestra.run.smithi045.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-26T00:17:24.683 INFO:teuthology.orchestra.run.smithi045.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-26T00:17:24.904 INFO:teuthology.orchestra.run.smithi037.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2445-gd2c45d97-1jammy [5,084 kB] 2024-04-26T00:17:25.115 INFO:teuthology.orchestra.run.smithi045.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-26T00:17:25.157 INFO:teuthology.orchestra.run.smithi045.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-26T00:17:25.159 INFO:teuthology.orchestra.run.smithi045.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-26T00:17:25.232 INFO:teuthology.orchestra.run.smithi037.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2445-gd2c45d97-1jammy [244 kB] 2024-04-26T00:17:25.238 INFO:teuthology.orchestra.run.smithi037.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2445-gd2c45d97-1jammy [124 kB] 2024-04-26T00:17:25.263 INFO:teuthology.orchestra.run.smithi037.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2445-gd2c45d97-1jammy [1,526 kB] 2024-04-26T00:17:25.369 INFO:teuthology.orchestra.run.smithi037.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2445-gd2c45d97-1jammy [6,245 kB] 2024-04-26T00:17:25.455 INFO:teuthology.orchestra.run.smithi045.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-26T00:17:25.648 INFO:teuthology.orchestra.run.smithi045.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-26T00:17:25.753 INFO:teuthology.orchestra.run.smithi045.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-26T00:17:25.768 INFO:teuthology.orchestra.run.smithi037.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2445-gd2c45d97-1jammy [22.7 MB] 2024-04-26T00:17:25.857 INFO:teuthology.orchestra.run.smithi045.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-26T00:17:25.878 INFO:teuthology.orchestra.run.smithi045.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-26T00:17:26.002 INFO:teuthology.orchestra.run.smithi045.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-26T00:17:26.003 INFO:teuthology.orchestra.run.smithi045.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-26T00:17:26.185 INFO:teuthology.orchestra.run.smithi045.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-26T00:17:26.268 INFO:teuthology.orchestra.run.smithi045.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-26T00:17:26.299 INFO:teuthology.orchestra.run.smithi045.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-26T00:17:26.317 INFO:teuthology.orchestra.run.smithi045.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-26T00:17:26.359 INFO:teuthology.orchestra.run.smithi045.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-26T00:17:26.579 INFO:teuthology.orchestra.run.smithi045.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-26T00:17:26.613 INFO:teuthology.orchestra.run.smithi045.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-26T00:17:26.636 INFO:teuthology.orchestra.run.smithi045.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-26T00:17:26.659 INFO:teuthology.orchestra.run.smithi045.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-26T00:17:26.709 INFO:teuthology.orchestra.run.smithi045.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-26T00:17:26.723 INFO:teuthology.orchestra.run.smithi045.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-26T00:17:26.914 INFO:teuthology.orchestra.run.smithi045.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-26T00:17:27.026 INFO:teuthology.orchestra.run.smithi037.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2445-gd2c45d97-1jammy [14.0 kB] 2024-04-26T00:17:27.027 INFO:teuthology.orchestra.run.smithi037.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2445-gd2c45d97-1jammy [869 kB] 2024-04-26T00:17:27.044 INFO:teuthology.orchestra.run.smithi037.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2445-gd2c45d97-1jammy [2,437 kB] 2024-04-26T00:17:27.149 INFO:teuthology.orchestra.run.smithi037.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2445-gd2c45d97-1jammy [791 kB] 2024-04-26T00:17:27.188 INFO:teuthology.orchestra.run.smithi037.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2445-gd2c45d97-1jammy [147 kB] 2024-04-26T00:17:27.191 INFO:teuthology.orchestra.run.smithi037.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2445-gd2c45d97-1jammy [2,293 kB] 2024-04-26T00:17:27.248 INFO:teuthology.orchestra.run.smithi045.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-26T00:17:27.295 INFO:teuthology.orchestra.run.smithi037.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2445-gd2c45d97-1jammy [8,625 kB] 2024-04-26T00:17:27.299 INFO:teuthology.orchestra.run.smithi045.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-26T00:17:27.315 INFO:teuthology.orchestra.run.smithi045.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-26T00:17:27.639 INFO:teuthology.orchestra.run.smithi037.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2445-gd2c45d97-1jammy [14.2 kB] 2024-04-26T00:17:27.640 INFO:teuthology.orchestra.run.smithi037.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2445-gd2c45d97-1jammy [50.7 MB] 2024-04-26T00:17:28.371 INFO:teuthology.orchestra.run.smithi045.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-26T00:17:28.382 INFO:teuthology.orchestra.run.smithi045.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-26T00:17:28.463 INFO:teuthology.orchestra.run.smithi045.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-26T00:17:28.526 INFO:teuthology.orchestra.run.smithi045.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-26T00:17:28.740 INFO:teuthology.orchestra.run.smithi045.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-26T00:17:29.065 INFO:teuthology.orchestra.run.smithi045.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-26T00:17:29.132 INFO:teuthology.orchestra.run.smithi045.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-26T00:17:29.527 INFO:teuthology.orchestra.run.smithi045.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-26T00:17:29.578 INFO:teuthology.orchestra.run.smithi037.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2445-gd2c45d97-1jammy [121 kB] 2024-04-26T00:17:29.580 INFO:teuthology.orchestra.run.smithi037.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2445-gd2c45d97-1jammy [38.9 kB] 2024-04-26T00:17:29.582 INFO:teuthology.orchestra.run.smithi037.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2445-gd2c45d97-1jammy [13.4 MB] 2024-04-26T00:17:29.792 INFO:teuthology.orchestra.run.smithi045.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-26T00:17:29.972 INFO:teuthology.orchestra.run.smithi045.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-26T00:17:29.977 INFO:teuthology.orchestra.run.smithi037.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2445-gd2c45d97-1jammy [91.4 kB] 2024-04-26T00:17:29.985 INFO:teuthology.orchestra.run.smithi045.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-26T00:17:30.006 INFO:teuthology.orchestra.run.smithi045.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-26T00:17:30.071 INFO:teuthology.orchestra.run.smithi045.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-26T00:17:30.370 INFO:teuthology.orchestra.run.smithi045.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-26T00:17:30.371 INFO:teuthology.orchestra.run.smithi045.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-26T00:17:30.374 INFO:teuthology.orchestra.run.smithi045.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-26T00:17:30.374 INFO:teuthology.orchestra.run.smithi045.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-26T00:17:30.377 INFO:teuthology.orchestra.run.smithi045.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-26T00:17:30.531 INFO:teuthology.orchestra.run.smithi037.stdout:Fetched 179 MB in 18s (9,800 kB/s) 2024-04-26T00:17:30.596 INFO:teuthology.orchestra.run.smithi045.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-26T00:17:30.685 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-26T00:17:30.780 INFO:teuthology.orchestra.run.smithi037.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-26T00:17:30.785 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-26T00:17:30.867 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-26T00:17:31.050 INFO:teuthology.orchestra.run.smithi045.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-26T00:17:31.067 INFO:teuthology.orchestra.run.smithi045.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-26T00:17:31.212 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-26T00:17:31.239 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-26T00:17:31.279 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-26T00:17:31.574 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-26T00:17:31.600 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-26T00:17:31.640 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-26T00:17:32.052 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-26T00:17:32.079 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T00:17:32.127 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T00:17:32.497 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-26T00:17:32.523 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T00:17:32.572 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T00:17:32.828 INFO:teuthology.orchestra.run.smithi045.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-26T00:17:32.831 INFO:teuthology.orchestra.run.smithi045.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-26T00:17:32.889 INFO:teuthology.orchestra.run.smithi045.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-26T00:17:32.908 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-26T00:17:32.935 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T00:17:32.945 INFO:teuthology.orchestra.run.smithi045.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-26T00:17:32.975 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T00:17:32.995 INFO:teuthology.orchestra.run.smithi045.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-26T00:17:33.048 INFO:teuthology.orchestra.run.smithi045.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-26T00:17:33.073 INFO:teuthology.orchestra.run.smithi045.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-26T00:17:33.093 INFO:teuthology.orchestra.run.smithi045.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-26T00:17:33.328 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-26T00:17:33.355 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-26T00:17:33.394 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-26T00:17:33.824 INFO:teuthology.orchestra.run.smithi045.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-26T00:17:33.848 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package librados2. 2024-04-26T00:17:33.874 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../007-librados2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:17:33.914 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking librados2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:17:34.242 INFO:teuthology.orchestra.run.smithi045.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-26T00:17:34.289 INFO:teuthology.orchestra.run.smithi045.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-26T00:17:34.308 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package librbd1. 2024-04-26T00:17:34.323 INFO:teuthology.orchestra.run.smithi045.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-26T00:17:34.324 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../008-librbd1_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:17:34.358 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking librbd1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:17:34.420 INFO:teuthology.orchestra.run.smithi045.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-26T00:17:34.462 INFO:teuthology.orchestra.run.smithi045.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-26T00:17:34.475 INFO:teuthology.orchestra.run.smithi045.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-26T00:17:34.480 INFO:teuthology.orchestra.run.smithi045.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-26T00:17:34.527 INFO:teuthology.orchestra.run.smithi045.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-26T00:17:34.530 INFO:teuthology.orchestra.run.smithi045.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-26T00:17:34.551 INFO:teuthology.orchestra.run.smithi045.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-26T00:17:34.721 INFO:teuthology.orchestra.run.smithi045.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-26T00:17:34.770 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libcephfs2. 2024-04-26T00:17:34.796 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:17:34.836 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libcephfs2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:17:35.000 INFO:teuthology.orchestra.run.smithi045.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-26T00:17:35.019 INFO:teuthology.orchestra.run.smithi045.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-26T00:17:35.122 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-rados. 2024-04-26T00:17:35.138 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:17:35.180 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-rados (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:17:35.313 INFO:teuthology.orchestra.run.smithi045.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-26T00:17:35.495 INFO:teuthology.orchestra.run.smithi045.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-26T00:17:35.525 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-26T00:17:35.552 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T00:17:35.591 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-ceph-argparse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:17:35.819 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-cephfs. 2024-04-26T00:17:35.836 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:17:35.869 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-cephfs (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:17:36.147 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-26T00:17:36.174 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T00:17:36.213 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:17:36.558 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-26T00:17:36.585 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-26T00:17:36.625 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-26T00:17:36.689 INFO:teuthology.orchestra.run.smithi045.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-26T00:17:36.766 INFO:teuthology.orchestra.run.smithi045.stdout:Get:115 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-26T00:17:36.799 INFO:teuthology.orchestra.run.smithi045.stdout:Get:116 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-26T00:17:36.919 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-prettytable. 2024-04-26T00:17:36.946 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-26T00:17:36.985 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-26T00:17:37.213 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-rbd. 2024-04-26T00:17:37.229 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:17:37.263 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-rbd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:17:37.272 INFO:teuthology.orchestra.run.smithi045.stdout:Get:117 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-26T00:17:37.423 INFO:teuthology.orchestra.run.smithi045.stdout:Get:118 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-26T00:17:37.574 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-26T00:17:37.601 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-26T00:17:37.641 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-26T00:17:37.910 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-26T00:17:37.937 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-26T00:17:37.977 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-26T00:17:38.280 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-26T00:17:38.295 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-26T00:17:38.338 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-26T00:17:38.616 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package lua5.1. 2024-04-26T00:17:38.643 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-26T00:17:38.674 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-26T00:17:38.952 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package lua-any. 2024-04-26T00:17:38.969 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-26T00:17:39.011 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-26T00:17:39.255 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package zip. 2024-04-26T00:17:39.270 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-26T00:17:39.313 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking zip (3.0-12build2) ... 2024-04-26T00:17:39.641 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package unzip. 2024-04-26T00:17:39.668 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-26T00:17:39.708 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-26T00:17:40.087 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package luarocks. 2024-04-26T00:17:40.114 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-26T00:17:40.153 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-26T00:17:40.561 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package librgw2. 2024-04-26T00:17:40.589 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../025-librgw2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:17:40.628 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking librgw2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:17:41.222 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-rgw. 2024-04-26T00:17:41.238 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:17:41.280 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-rgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:17:41.567 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-26T00:17:41.594 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-26T00:17:41.625 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-26T00:17:41.919 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libradosstriper1. 2024-04-26T00:17:41.946 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:17:41.986 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libradosstriper1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:17:42.406 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-common. 2024-04-26T00:17:42.433 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:17:42.472 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:17:42.831 INFO:teuthology.orchestra.run.smithi045.stdout:Fetched 179 MB in 31s (5,795 kB/s) 2024-04-26T00:17:42.991 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-26T00:17:43.093 INFO:teuthology.orchestra.run.smithi045.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-26T00:17:43.099 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-26T00:17:43.189 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-26T00:17:43.534 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-26T00:17:43.561 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-26T00:17:43.600 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-26T00:17:43.844 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-base. 2024-04-26T00:17:43.871 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:17:43.921 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-base (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:17:44.053 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-26T00:17:44.080 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-26T00:17:44.120 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-26T00:17:44.492 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-26T00:17:44.520 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-26T00:17:44.523 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-26T00:17:44.550 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T00:17:44.559 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-26T00:17:44.598 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T00:17:44.836 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-cheroot. 2024-04-26T00:17:44.853 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-26T00:17:44.886 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-26T00:17:45.009 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-26T00:17:45.036 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T00:17:45.076 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T00:17:45.173 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-26T00:17:45.188 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-26T00:17:45.223 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-26T00:17:45.446 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-26T00:17:45.467 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-26T00:17:45.473 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T00:17:45.483 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-26T00:17:45.512 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T00:17:45.525 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-26T00:17:45.823 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-26T00:17:45.837 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-26T00:17:45.850 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-26T00:17:45.864 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-26T00:17:45.890 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-26T00:17:45.903 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-26T00:17:46.176 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package librados2. 2024-04-26T00:17:46.203 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../007-librados2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:17:46.206 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-tempora. 2024-04-26T00:17:46.233 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-26T00:17:46.234 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking librados2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:17:46.273 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-26T00:17:46.534 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-portend. 2024-04-26T00:17:46.550 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-26T00:17:46.584 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-26T00:17:46.679 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package librbd1. 2024-04-26T00:17:46.694 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../008-librbd1_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:17:46.737 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking librbd1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:17:46.862 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-26T00:17:46.889 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-26T00:17:46.928 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-26T00:17:47.181 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libcephfs2. 2024-04-26T00:17:47.208 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:17:47.231 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-26T00:17:47.248 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libcephfs2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:17:47.258 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-26T00:17:47.298 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-26T00:17:47.525 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-rados. 2024-04-26T00:17:47.541 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:17:47.743 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-rados (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:17:47.918 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-natsort. 2024-04-26T00:17:47.945 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-26T00:17:48.010 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-26T00:17:48.020 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-26T00:17:48.047 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T00:17:48.086 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-ceph-argparse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:17:48.313 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-logutils. 2024-04-26T00:17:48.339 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-cephfs. 2024-04-26T00:17:48.339 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-26T00:17:48.365 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:17:48.379 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-26T00:17:48.405 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-cephfs (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:17:48.632 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-26T00:17:48.659 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-26T00:17:48.690 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-26T00:17:48.707 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-26T00:17:48.734 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T00:17:48.774 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:17:48.993 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-mako. 2024-04-26T00:17:49.018 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-26T00:17:49.060 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-26T00:17:49.119 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-26T00:17:49.145 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-26T00:17:49.185 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-26T00:17:49.380 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-26T00:17:49.407 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-26T00:17:49.446 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-26T00:17:49.513 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-prettytable. 2024-04-26T00:17:49.540 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-26T00:17:49.571 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-26T00:17:49.749 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-26T00:17:49.776 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-26T00:17:49.807 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-rbd. 2024-04-26T00:17:49.816 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-26T00:17:49.834 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:17:49.873 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-rbd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:17:50.102 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-webob. 2024-04-26T00:17:50.129 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-26T00:17:50.169 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-26T00:17:50.226 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-26T00:17:50.253 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-26T00:17:50.293 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-26T00:17:50.480 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-waitress. 2024-04-26T00:17:50.507 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-26T00:17:50.542 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-26T00:17:50.570 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-26T00:17:50.586 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-26T00:17:50.628 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-26T00:17:50.799 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-tempita. 2024-04-26T00:17:50.815 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-26T00:17:50.849 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-26T00:17:50.915 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-26T00:17:50.942 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-26T00:17:50.973 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-26T00:17:51.161 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-paste. 2024-04-26T00:17:51.188 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-26T00:17:51.227 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-26T00:17:51.284 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package lua5.1. 2024-04-26T00:17:51.311 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-26T00:17:51.351 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-26T00:17:51.497 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-26T00:17:51.513 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-26T00:17:51.547 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-26T00:17:51.679 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package lua-any. 2024-04-26T00:17:51.694 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-26T00:17:51.737 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-26T00:17:51.875 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-26T00:17:51.902 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-26T00:17:51.941 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-26T00:17:52.031 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package zip. 2024-04-26T00:17:52.058 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-26T00:17:52.097 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking zip (3.0-12build2) ... 2024-04-26T00:17:52.244 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-webtest. 2024-04-26T00:17:52.271 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-26T00:17:52.328 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-26T00:17:52.367 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package unzip. 2024-04-26T00:17:52.394 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-26T00:17:52.425 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-26T00:17:52.639 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pecan. 2024-04-26T00:17:52.666 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-26T00:17:52.705 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-26T00:17:52.929 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package luarocks. 2024-04-26T00:17:52.956 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-26T00:17:52.996 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-26T00:17:53.052 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-certifi. 2024-04-26T00:17:53.079 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-26T00:17:53.118 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-26T00:17:53.378 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package librgw2. 2024-04-26T00:17:53.405 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../025-librgw2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:17:53.405 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-idna. 2024-04-26T00:17:53.432 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-26T00:17:53.444 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking librgw2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:17:53.471 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-26T00:17:53.774 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-urllib3. 2024-04-26T00:17:53.801 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-26T00:17:53.841 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-26T00:17:53.988 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-rgw. 2024-04-26T00:17:54.016 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:17:54.055 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-rgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:17:54.161 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-requests. 2024-04-26T00:17:54.188 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-26T00:17:54.227 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-26T00:17:54.333 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-26T00:17:54.360 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-26T00:17:54.391 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-26T00:17:54.539 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-26T00:17:54.566 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-26T00:17:54.605 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-26T00:17:54.710 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libradosstriper1. 2024-04-26T00:17:54.737 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:17:54.777 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libradosstriper1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:17:54.891 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-26T00:17:54.918 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T00:17:54.958 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:17:55.205 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-common. 2024-04-26T00:17:55.232 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:17:55.271 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:17:55.304 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-26T00:17:55.321 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:17:55.355 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:17:55.691 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-26T00:17:55.718 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-26T00:17:55.757 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-26T00:17:56.052 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-openssl. 2024-04-26T00:17:56.079 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-26T00:17:56.118 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-26T00:17:56.421 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-mgr. 2024-04-26T00:17:56.437 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:17:56.471 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-mgr (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:17:56.606 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-base. 2024-04-26T00:17:56.629 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:17:56.669 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-base (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:17:56.816 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-mon. 2024-04-26T00:17:56.843 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:17:56.882 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-mon (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:17:57.190 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-26T00:17:57.216 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-26T00:17:57.247 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-26T00:17:57.478 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-osd. 2024-04-26T00:17:57.508 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:17:57.544 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-osd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:17:57.567 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-cheroot. 2024-04-26T00:17:57.593 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-26T00:17:57.633 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-26T00:17:58.111 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-26T00:17:58.138 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-26T00:17:58.177 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-26T00:17:58.463 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-26T00:17:58.491 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-26T00:17:58.530 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-26T00:17:58.622 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph. 2024-04-26T00:17:58.638 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../066-ceph_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:17:58.672 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:17:58.774 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-26T00:17:58.798 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-26T00:17:58.832 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-26T00:17:58.967 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-fuse. 2024-04-26T00:17:58.994 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:17:59.033 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:17:59.135 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-tempora. 2024-04-26T00:17:59.162 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-26T00:17:59.201 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-26T00:17:59.395 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-mds. 2024-04-26T00:17:59.422 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:17:59.461 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-mds (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:17:59.521 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-portend. 2024-04-26T00:17:59.537 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-26T00:17:59.579 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-26T00:17:59.806 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package cephadm. 2024-04-26T00:17:59.822 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../069-cephadm_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:17:59.857 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-26T00:17:59.864 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:17:59.873 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-26T00:17:59.907 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-26T00:18:00.134 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-26T00:18:00.150 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-26T00:18:00.150 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-26T00:18:00.177 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-26T00:18:00.184 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-26T00:18:00.217 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-26T00:18:00.519 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-26T00:18:00.523 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-natsort. 2024-04-26T00:18:00.536 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-26T00:18:00.550 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-26T00:18:00.569 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-26T00:18:00.589 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-26T00:18:00.851 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-logutils. 2024-04-26T00:18:00.878 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-26T00:18:00.909 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-26T00:18:01.212 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-26T00:18:01.229 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-26T00:18:01.270 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-26T00:18:01.573 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-mako. 2024-04-26T00:18:01.591 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-babel. 2024-04-26T00:18:01.600 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-26T00:18:01.619 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-26T00:18:01.657 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-26T00:18:01.660 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-26T00:18:02.001 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-26T00:18:02.002 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-jinja2. 2024-04-26T00:18:02.028 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-26T00:18:02.029 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-26T00:18:02.067 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-26T00:18:02.068 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-26T00:18:02.328 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-26T00:18:02.345 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-26T00:18:02.378 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-26T00:18:02.438 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-26T00:18:02.466 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T00:18:02.505 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:02.672 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-webob. 2024-04-26T00:18:02.699 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-26T00:18:02.739 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-26T00:18:02.858 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-26T00:18:02.886 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-26T00:18:02.925 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-26T00:18:03.108 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-waitress. 2024-04-26T00:18:03.135 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-26T00:18:03.178 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-26T00:18:03.603 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-routes. 2024-04-26T00:18:03.614 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-tempita. 2024-04-26T00:18:03.631 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-26T00:18:03.642 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-26T00:18:03.661 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-26T00:18:03.686 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-26T00:18:03.948 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-26T00:18:03.964 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T00:18:03.989 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-paste. 2024-04-26T00:18:03.997 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:04.016 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-26T00:18:04.055 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-26T00:18:04.392 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-26T00:18:04.419 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-26T00:18:04.450 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-26T00:18:04.727 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-26T00:18:04.754 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-26T00:18:04.794 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-26T00:18:05.105 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-webtest. 2024-04-26T00:18:05.133 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-26T00:18:05.172 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-26T00:18:05.219 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-26T00:18:05.247 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-26T00:18:05.286 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-26T00:18:05.449 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-pecan. 2024-04-26T00:18:05.465 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-26T00:18:05.499 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-26T00:18:05.781 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-joblib. 2024-04-26T00:18:05.808 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-26T00:18:05.837 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-certifi. 2024-04-26T00:18:05.847 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-26T00:18:05.853 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-26T00:18:05.887 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-26T00:18:06.159 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-26T00:18:06.187 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-26T00:18:06.206 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-idna. 2024-04-26T00:18:06.225 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-26T00:18:06.233 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-26T00:18:06.273 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-26T00:18:06.529 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-sklearn. 2024-04-26T00:18:06.556 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-26T00:18:06.568 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-urllib3. 2024-04-26T00:18:06.594 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-26T00:18:06.595 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-26T00:18:06.634 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-26T00:18:06.945 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-requests. 2024-04-26T00:18:06.972 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-26T00:18:07.011 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-26T00:18:07.281 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-26T00:18:07.308 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-26T00:18:07.337 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-26T00:18:07.339 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-26T00:18:07.365 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T00:18:07.489 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:07.625 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-26T00:18:07.641 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T00:18:07.675 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:08.232 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-26T00:18:08.259 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:18:08.296 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:08.359 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-cachetools. 2024-04-26T00:18:08.387 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-26T00:18:08.417 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-26T00:18:08.557 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-26T00:18:08.573 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-26T00:18:08.616 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-26T00:18:08.729 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-26T00:18:08.756 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-26T00:18:08.795 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-26T00:18:08.918 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-openssl. 2024-04-26T00:18:08.945 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-26T00:18:08.985 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-26T00:18:09.107 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-26T00:18:09.134 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-26T00:18:09.173 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-26T00:18:09.312 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-mgr. 2024-04-26T00:18:09.339 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:18:09.379 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-mgr (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:09.484 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-rsa. 2024-04-26T00:18:09.512 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-26T00:18:09.551 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-26T00:18:09.749 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-mon. 2024-04-26T00:18:09.778 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:18:09.815 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-mon (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:09.913 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-google-auth. 2024-04-26T00:18:09.929 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-26T00:18:09.963 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-26T00:18:10.283 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-26T00:18:10.311 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-26T00:18:10.349 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-26T00:18:10.402 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-osd. 2024-04-26T00:18:10.429 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:18:10.468 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-osd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:10.619 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-websocket. 2024-04-26T00:18:10.647 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-26T00:18:10.686 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-26T00:18:10.989 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-26T00:18:11.005 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-26T00:18:11.064 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-26T00:18:11.522 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph. 2024-04-26T00:18:11.549 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../066-ceph_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:18:11.588 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:11.728 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-26T00:18:11.756 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T00:18:11.786 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:11.916 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-fuse. 2024-04-26T00:18:11.943 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:18:11.982 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:12.081 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-26T00:18:12.108 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-26T00:18:12.147 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-26T00:18:12.352 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-mds. 2024-04-26T00:18:12.379 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:18:12.418 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-mds (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:12.483 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-26T00:18:12.511 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-26T00:18:12.550 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-26T00:18:12.780 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package cephadm. 2024-04-26T00:18:12.806 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../069-cephadm_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:18:12.836 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package jq. 2024-04-26T00:18:12.837 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:12.863 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-26T00:18:12.911 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-26T00:18:13.115 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-26T00:18:13.143 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-26T00:18:13.164 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package socat. 2024-04-26T00:18:13.191 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-26T00:18:13.230 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-26T00:18:13.315 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-26T00:18:13.533 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package xmlstarlet. 2024-04-26T00:18:13.550 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-26T00:18:13.591 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-26T00:18:13.651 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-26T00:18:13.667 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-26T00:18:13.709 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-26T00:18:13.946 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-test. 2024-04-26T00:18:13.971 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:18:14.012 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-test (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:14.714 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-babel. 2024-04-26T00:18:14.730 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-26T00:18:14.763 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-26T00:18:15.050 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-jinja2. 2024-04-26T00:18:15.066 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-26T00:18:15.109 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-26T00:18:15.436 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-26T00:18:15.463 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T00:18:15.502 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:15.805 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-26T00:18:15.818 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-26T00:18:15.855 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-26T00:18:16.158 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-routes. 2024-04-26T00:18:16.185 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-26T00:18:16.224 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-26T00:18:16.400 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-volume. 2024-04-26T00:18:16.428 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T00:18:16.467 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-volume (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:16.561 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-26T00:18:16.589 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T00:18:16.627 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:16.771 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-26T00:18:16.799 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:18:16.837 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libcephfs-dev (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:17.157 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-26T00:18:17.184 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-26T00:18:17.223 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-26T00:18:17.553 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-26T00:18:17.581 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-26T00:18:17.619 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-26T00:18:17.940 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-26T00:18:17.956 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-26T00:18:17.990 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package nvme-cli. 2024-04-26T00:18:17.998 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-26T00:18:18.018 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-26T00:18:18.056 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-26T00:18:18.470 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package pkg-config. 2024-04-26T00:18:18.498 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-26T00:18:18.536 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-26T00:18:18.677 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-joblib. 2024-04-26T00:18:18.704 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-26T00:18:18.743 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-26T00:18:19.031 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-26T00:18:19.059 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-26T00:18:19.097 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-26T00:18:19.130 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-26T00:18:19.157 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-26T00:18:19.221 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-26T00:18:19.516 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-sklearn. 2024-04-26T00:18:19.517 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-26T00:18:19.543 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-26T00:18:19.545 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-26T00:18:19.582 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-26T00:18:19.583 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-26T00:18:19.853 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pastescript. 2024-04-26T00:18:19.869 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-26T00:18:19.911 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-26T00:18:20.232 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pluggy. 2024-04-26T00:18:20.260 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-26T00:18:20.265 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-26T00:18:20.294 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T00:18:20.298 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-26T00:18:20.417 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:20.609 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-psutil. 2024-04-26T00:18:20.639 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-26T00:18:20.676 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-26T00:18:21.037 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-py. 2024-04-26T00:18:21.065 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-26T00:18:21.112 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-26T00:18:21.395 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-cachetools. 2024-04-26T00:18:21.423 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-26T00:18:21.432 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pygments. 2024-04-26T00:18:21.460 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-26T00:18:21.461 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-26T00:18:21.498 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-26T00:18:21.764 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-26T00:18:21.792 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-26T00:18:21.831 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-26T00:18:21.944 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-26T00:18:21.972 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-26T00:18:22.002 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-26T00:18:22.167 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-26T00:18:22.195 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-26T00:18:22.234 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-26T00:18:22.307 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-toml. 2024-04-26T00:18:22.335 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-26T00:18:22.374 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-26T00:18:22.512 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-rsa. 2024-04-26T00:18:22.528 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-26T00:18:22.561 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-26T00:18:22.701 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pytest. 2024-04-26T00:18:22.729 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-26T00:18:22.784 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-26T00:18:22.865 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-google-auth. 2024-04-26T00:18:22.893 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-26T00:18:22.931 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-26T00:18:23.138 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-simplejson. 2024-04-26T00:18:23.166 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-26T00:18:23.204 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-26T00:18:23.251 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-26T00:18:23.279 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-26T00:18:23.317 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-26T00:18:23.482 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-26T00:18:23.498 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-26T00:18:23.540 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-26T00:18:23.620 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-websocket. 2024-04-26T00:18:23.648 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-26T00:18:23.687 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-26T00:18:23.998 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-26T00:18:24.026 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-26T00:18:24.065 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-26T00:18:24.202 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package radosgw. 2024-04-26T00:18:24.231 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../116-radosgw_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:18:24.260 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking radosgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:24.737 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-26T00:18:24.754 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T00:18:24.795 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:24.980 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package rbd-fuse. 2024-04-26T00:18:24.996 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:18:25.038 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking rbd-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:25.089 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-26T00:18:25.117 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-26T00:18:25.156 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-26T00:18:25.335 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-26T00:18:25.484 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-26T00:18:25.512 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-26T00:18:25.550 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-26T00:18:25.593 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-26T00:18:25.702 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-26T00:18:25.819 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package jq. 2024-04-26T00:18:25.847 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-26T00:18:25.886 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-26T00:18:25.954 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-26T00:18:26.130 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package socat. 2024-04-26T00:18:26.158 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-26T00:18:26.196 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-26T00:18:26.284 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-26T00:18:26.499 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package xmlstarlet. 2024-04-26T00:18:26.516 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-26T00:18:26.549 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-26T00:18:26.758 INFO:teuthology.orchestra.run.smithi037.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-26T00:18:26.837 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:26.887 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-test. 2024-04-26T00:18:26.915 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:18:26.953 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-test (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:27.388 INFO:teuthology.orchestra.run.smithi037.stdout:Adding system user cephadm....done 2024-04-26T00:18:27.544 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-26T00:18:27.829 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-26T00:18:28.096 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-26T00:18:28.231 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-26T00:18:28.490 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-26T00:18:28.742 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-26T00:18:28.859 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-26T00:18:29.186 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-26T00:18:29.341 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-volume. 2024-04-26T00:18:29.357 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T00:18:29.391 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-volume (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:29.529 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-26T00:18:29.694 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-26T00:18:29.710 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:18:29.743 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libcephfs-dev (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:29.797 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-26T00:18:30.046 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-26T00:18:30.073 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-26T00:18:30.075 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-26T00:18:30.113 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-26T00:18:30.350 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-26T00:18:30.484 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-26T00:18:30.512 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-26T00:18:30.550 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-26T00:18:30.618 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-26T00:18:30.877 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-26T00:18:30.912 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package nvme-cli. 2024-04-26T00:18:30.941 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-26T00:18:30.978 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-26T00:18:31.128 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-ceph-argparse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:31.379 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-26T00:18:31.400 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package pkg-config. 2024-04-26T00:18:31.429 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-26T00:18:31.468 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-26T00:18:31.505 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-26T00:18:31.631 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-26T00:18:31.728 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-26T00:18:31.745 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-26T00:18:31.757 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-26T00:18:31.786 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-26T00:18:32.008 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-26T00:18:32.239 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-26T00:18:32.267 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-26T00:18:32.306 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-26T00:18:32.411 INFO:teuthology.orchestra.run.smithi037.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-26T00:18:32.447 INFO:teuthology.orchestra.run.smithi037.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-26T00:18:32.495 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-26T00:18:32.595 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-26T00:18:32.608 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-pastescript. 2024-04-26T00:18:32.637 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-26T00:18:32.674 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-26T00:18:32.947 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-26T00:18:33.003 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-pluggy. 2024-04-26T00:18:33.031 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-26T00:18:33.069 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-26T00:18:33.072 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-26T00:18:33.334 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-26T00:18:33.339 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-psutil. 2024-04-26T00:18:33.367 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-26T00:18:33.396 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-26T00:18:33.592 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-26T00:18:33.717 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-py. 2024-04-26T00:18:33.745 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-26T00:18:33.774 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-26T00:18:33.893 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up zip (3.0-12build2) ... 2024-04-26T00:18:34.002 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-26T00:18:34.052 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-pygments. 2024-04-26T00:18:34.069 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-26T00:18:34.102 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-26T00:18:34.574 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-26T00:18:34.602 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-26T00:18:34.639 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-26T00:18:34.729 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-26T00:18:34.959 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-toml. 2024-04-26T00:18:34.987 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-26T00:18:34.997 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-26T00:18:35.025 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-26T00:18:35.123 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-26T00:18:35.249 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-26T00:18:35.294 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-pytest. 2024-04-26T00:18:35.323 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-26T00:18:35.361 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-26T00:18:35.567 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-26T00:18:35.706 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-simplejson. 2024-04-26T00:18:35.734 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-26T00:18:35.772 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-26T00:18:35.959 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-26T00:18:36.050 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-26T00:18:36.075 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-26T00:18:36.116 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-26T00:18:36.210 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-26T00:18:36.579 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-26T00:18:36.787 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package radosgw. 2024-04-26T00:18:36.815 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../116-radosgw_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:18:36.844 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking radosgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:36.871 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-26T00:18:37.236 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-26T00:18:37.466 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-26T00:18:37.592 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:37.673 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package rbd-fuse. 2024-04-26T00:18:37.702 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T00:18:37.739 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking rbd-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:37.914 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-idna (3.3-1) ... 2024-04-26T00:18:38.052 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-26T00:18:38.261 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-26T00:18:38.369 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-26T00:18:38.494 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-26T00:18:38.784 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-26T00:18:39.104 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-26T00:18:39.450 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-26T00:18:39.521 INFO:teuthology.orchestra.run.smithi045.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-26T00:18:39.596 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:39.703 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T00:18:39.836 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-26T00:18:40.095 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-26T00:18:40.103 INFO:teuthology.orchestra.run.smithi045.stdout:Adding system user cephadm....done 2024-04-26T00:18:40.204 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-26T00:18:40.235 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-26T00:18:40.339 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-26T00:18:40.511 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-26T00:18:40.632 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-26T00:18:40.762 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-26T00:18:40.872 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-26T00:18:40.891 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-26T00:18:41.116 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-26T00:18:41.134 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T00:18:41.260 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-26T00:18:41.368 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-26T00:18:41.494 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-26T00:18:41.553 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-26T00:18:41.796 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-26T00:18:41.804 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-26T00:18:42.064 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-26T00:18:42.173 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-26T00:18:42.323 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-26T00:18:42.424 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-26T00:18:42.566 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-26T00:18:42.709 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-26T00:18:42.825 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-26T00:18:42.951 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-26T00:18:42.976 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-26T00:18:43.227 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-26T00:18:43.236 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-26T00:18:43.328 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-26T00:18:43.504 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-26T00:18:43.612 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-26T00:18:43.754 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-ceph-argparse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:43.922 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-26T00:18:44.030 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-26T00:18:44.131 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-26T00:18:44.173 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-26T00:18:44.258 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-26T00:18:44.274 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-26T00:18:44.375 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-26T00:18:44.383 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-26T00:18:44.465 INFO:teuthology.orchestra.run.smithi037.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-26T00:18:44.643 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-26T00:18:44.760 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-26T00:18:44.886 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-26T00:18:45.004 INFO:teuthology.orchestra.run.smithi045.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-26T00:18:45.033 INFO:teuthology.orchestra.run.smithi045.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-26T00:18:45.088 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-26T00:18:45.214 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-26T00:18:45.296 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-26T00:18:45.547 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-26T00:18:45.623 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-26T00:18:45.706 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-26T00:18:45.758 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-26T00:18:45.949 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T00:18:46.026 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-26T00:18:46.075 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-26T00:18:46.293 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-26T00:18:46.401 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-26T00:18:46.603 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up zip (3.0-12build2) ... 2024-04-26T00:18:46.720 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-26T00:18:46.746 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-26T00:18:47.012 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-26T00:18:47.138 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-26T00:18:47.414 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-26T00:18:47.431 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-26T00:18:47.673 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-26T00:18:47.707 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-26T00:18:47.800 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-26T00:18:47.926 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-26T00:18:48.051 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-26T00:18:48.235 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-26T00:18:48.428 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-26T00:18:48.628 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-26T00:18:48.738 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-26T00:18:48.879 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-26T00:18:49.073 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-26T00:18:49.200 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up librados2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:49.247 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-26T00:18:49.326 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:49.477 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-26T00:18:49.540 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-26T00:18:49.746 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libcephfs2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:49.881 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libradosstriper1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:49.900 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-26T00:18:50.007 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-26T00:18:50.159 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-26T00:18:50.285 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:50.300 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up librbd1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:50.426 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-mgr-modules-core (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:50.554 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:50.595 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-idna (3.3-1) ... 2024-04-26T00:18:50.772 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-26T00:18:50.772 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-26T00:18:50.913 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-26T00:18:51.288 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libcephfs-dev (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:51.387 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-rados (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:51.505 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up librgw2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:51.631 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-26T00:18:52.068 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-26T00:18:52.320 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T00:18:52.454 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-26T00:18:52.705 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-26T00:18:52.831 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-26T00:18:52.926 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-rbd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:52.966 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-26T00:18:53.052 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up rbd-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:53.178 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-rgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:53.275 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-26T00:18:53.321 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-cephfs (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:53.447 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:53.535 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-26T00:18:53.794 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T00:18:53.904 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-26T00:18:53.965 INFO:teuthology.orchestra.run.smithi037.stdout:Adding group ceph....done 2024-04-26T00:18:54.188 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-26T00:18:54.333 INFO:teuthology.orchestra.run.smithi037.stdout:Adding system user ceph....done 2024-04-26T00:18:54.439 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-26T00:18:54.501 INFO:teuthology.orchestra.run.smithi037.stdout:Setting system user ceph properties....done 2024-04-26T00:18:54.521 INFO:teuthology.orchestra.run.smithi037.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-26T00:18:54.670 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-26T00:18:54.716 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-26T00:18:54.975 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-26T00:18:55.039 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-26T00:18:55.234 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-26T00:18:55.502 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-26T00:18:55.540 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-test (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:55.620 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-26T00:18:55.666 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up radosgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:55.888 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-26T00:18:55.989 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-26T00:18:56.148 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-26T00:18:56.148 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-26T00:18:56.273 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-26T00:18:56.641 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-26T00:18:56.660 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-base (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:56.909 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-26T00:18:57.035 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-26T00:18:57.161 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-26T00:18:57.250 INFO:teuthology.orchestra.run.smithi045.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-26T00:18:57.321 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-26T00:18:57.529 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-26T00:18:57.630 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-26T00:18:57.820 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-mds (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:58.006 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-26T00:18:58.052 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-26T00:18:58.052 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-26T00:18:58.291 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-26T00:18:58.417 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-26T00:18:58.555 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-mgr (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:58.676 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T00:18:58.791 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-26T00:18:58.791 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-26T00:18:58.802 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-26T00:18:59.137 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-26T00:18:59.444 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-osd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:18:59.563 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-26T00:18:59.730 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-26T00:18:59.730 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-26T00:18:59.831 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-26T00:18:59.957 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-26T00:19:00.233 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-26T00:19:00.254 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:19:00.380 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:19:00.509 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-26T00:19:00.815 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-mon (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:19:00.862 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-26T00:19:01.027 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-26T00:19:01.027 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-26T00:19:01.255 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-26T00:19:01.540 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-26T00:19:01.542 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-mgr-cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:19:01.675 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:19:01.776 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-mgr-dashboard (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:19:01.893 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-26T00:19:01.902 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-volume (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:19:02.019 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up librados2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:19:02.145 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:19:02.239 INFO:teuthology.orchestra.run.smithi037.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-26T00:19:02.254 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-26T00:19:02.348 INFO:teuthology.orchestra.run.smithi037.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-26T00:19:02.509 INFO:teuthology.orchestra.run.smithi037.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-26T00:19:02.557 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libcephfs2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:19:02.683 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libradosstriper1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:19:02.810 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-26T00:19:03.128 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up librbd1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:19:03.254 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-mgr-modules-core (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:19:03.388 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:19:03.592 INFO:teuthology.orchestra.run.smithi037.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-26T00:19:03.615 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-26T00:19:03.615 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-26T00:19:04.123 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libcephfs-dev (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:19:04.249 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-rados (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:19:04.250 INFO:teuthology.orchestra.run.smithi037.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-26T00:19:04.256 DEBUG:teuthology.parallel:result is None 2024-04-26T00:19:04.375 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up librgw2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:19:04.501 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-26T00:19:05.812 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-rbd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:19:05.964 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up rbd-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:19:06.115 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-rgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:19:06.267 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-cephfs (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:19:06.392 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:19:06.944 INFO:teuthology.orchestra.run.smithi045.stdout:Adding group ceph....done 2024-04-26T00:19:07.328 INFO:teuthology.orchestra.run.smithi045.stdout:Adding system user ceph....done 2024-04-26T00:19:07.500 INFO:teuthology.orchestra.run.smithi045.stdout:Setting system user ceph properties....done 2024-04-26T00:19:07.516 INFO:teuthology.orchestra.run.smithi045.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-26T00:19:07.661 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-26T00:19:08.009 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-26T00:19:08.469 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-test (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:19:08.569 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up radosgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:19:09.020 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-26T00:19:09.020 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-26T00:19:09.471 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-base (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:19:09.894 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-26T00:19:10.350 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-mds (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:19:10.564 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-26T00:19:10.565 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-26T00:19:11.052 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-mgr (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:19:11.348 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-26T00:19:11.348 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-26T00:19:11.863 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-osd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:19:12.208 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-26T00:19:12.209 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-26T00:19:12.755 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:19:12.906 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:19:13.083 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-mon (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:19:13.306 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-26T00:19:13.307 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-26T00:19:13.751 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-mgr-cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:19:13.893 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:19:13.994 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-mgr-dashboard (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:19:14.138 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-volume (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T00:19:14.507 INFO:teuthology.orchestra.run.smithi045.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-26T00:19:14.654 INFO:teuthology.orchestra.run.smithi045.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-26T00:19:14.896 INFO:teuthology.orchestra.run.smithi045.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-26T00:19:15.612 INFO:teuthology.orchestra.run.smithi045.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-26T00:19:16.275 INFO:teuthology.orchestra.run.smithi045.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-26T00:19:16.280 DEBUG:teuthology.parallel:result is None 2024-04-26T00:19:16.281 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-26T00:19:16.440 DEBUG:teuthology.orchestra.run.smithi037:> dpkg-query -W -f '${Version}' ceph 2024-04-26T00:19:16.460 INFO:teuthology.orchestra.run.smithi037.stdout:19.0.0-2445-gd2c45d97-1jammy 2024-04-26T00:19:16.460 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445-gd2c45d97-1jammy 2024-04-26T00:19:16.460 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445-gd2c45d97-1jammy is installed. 2024-04-26T00:19:16.462 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-26T00:19:16.689 DEBUG:teuthology.orchestra.run.smithi045:> dpkg-query -W -f '${Version}' ceph 2024-04-26T00:19:16.710 INFO:teuthology.orchestra.run.smithi045.stdout:19.0.0-2445-gd2c45d97-1jammy 2024-04-26T00:19:16.710 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445-gd2c45d97-1jammy 2024-04-26T00:19:16.710 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445-gd2c45d97-1jammy is installed. 2024-04-26T00:19:16.712 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-26T00:19:16.712 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T00:19:16.712 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T00:19:16.727 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-26T00:19:16.727 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T00:19:16.765 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-26T00:19:16.766 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T00:19:16.766 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T00:19:16.781 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T00:19:16.837 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-26T00:19:16.837 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T00:19:16.851 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T00:19:16.905 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-26T00:19:16.905 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T00:19:16.905 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T00:19:16.919 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T00:19:16.972 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-26T00:19:16.973 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T00:19:16.986 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T00:19:17.040 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-26T00:19:17.041 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T00:19:17.041 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T00:19:17.055 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T00:19:17.108 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-26T00:19:17.109 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T00:19:17.123 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T00:19:17.177 INFO:teuthology.run_tasks:Running task ceph... 2024-04-26T00:19:17.314 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-26T00:19:17.315 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 777 /var/log/ceph 2024-04-26T00:19:17.317 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 777 /var/log/ceph 2024-04-26T00:19:17.330 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-26T00:19:17.331 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-26T00:19:17.374 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-26T00:19:17.386 INFO:tasks.ceph:Creating extra log directories... 2024-04-26T00:19:17.386 DEBUG:teuthology.orchestra.run.smithi037:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-26T00:19:17.430 DEBUG:teuthology.orchestra.run.smithi045:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-26T00:19:17.490 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-26T00:19:17.490 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client ping interval': 4, 'mon client ping timeout': 12, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-26T00:19:17.490 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673875', 'branch': 'squid', 'description': 'rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/snaps-few-objects}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7673875', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client ping interval': 4, 'mon client ping timeout': 12, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'b22e2ebdeb24376882b7bda2a7329c8cccc2276a'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 101, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 9886, 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'sleep_before_teardown': 0, 'subset': '30270/100000', 'suite': 'rados', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'b22e2ebdeb24376882b7bda2a7329c8cccc2276a', 'targets': {'smithi037.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPp8mLUzu4RpgeN5gOjjGGS5Ns9rCRCmBaFTAT1N6MWLXIlWueulz8y9M+5G9DsvjWX9Jh/KSGOEvCVJZybRHSM=', 'smithi045.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH2wtlUjTvH0ATp/pPDHlgyHB93Z7OB7GkiyYgQbJo6GGZhwJfUr66AQRpqRgV5wArlXOkHWapqvvNtH2Qt2IDc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'freeze_mon_duration': 20, 'freeze_mon_probability': 10, 'revive_delay': 20, 'thrash_delay': 1, 'thrash_many': True}}, {'rados': {'clients': ['client.0'], 'objects': 50, 'op_weights': {'copy_from': 50, 'delete': 50, 'read': 100, 'rollback': 50, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-25_21:00:03', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-26T00:19:17.491 DEBUG:teuthology.orchestra.run.smithi037:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-26T00:19:17.493 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-26T00:19:17.499 DEBUG:teuthology.orchestra.run.smithi037:> sudo install -d -m0777 -- /var/run/ceph 2024-04-26T00:19:17.542 DEBUG:teuthology.orchestra.run.smithi045:> sudo install -d -m0777 -- /var/run/ceph 2024-04-26T00:19:17.557 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T00:19:17.557 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T00:19:17.601 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T00:19:17.601 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_1 2024-04-26T00:19:17.649 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T00:19:17.650 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T00:19:17.650 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 5h/5d Inode: 870 Links: 1 2024-04-26T00:19:17.650 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T00:19:17.650 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-04-26 00:13:38.930389069 +0000 2024-04-26T00:19:17.650 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-04-26 00:13:38.678391319 +0000 2024-04-26T00:19:17.650 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-04-26 00:13:38.678391319 +0000 2024-04-26T00:19:17.650 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: - 2024-04-26T00:19:17.650 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T00:19:17.703 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2024-04-26T00:19:17.704 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2024-04-26T00:19:17.704 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000436297 s, 1.2 MB/s 2024-04-26T00:19:17.705 DEBUG:teuthology.orchestra.run.smithi037:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T00:19:17.755 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_2 2024-04-26T00:19:17.802 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T00:19:17.802 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T00:19:17.802 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 5h/5d Inode: 889 Links: 1 2024-04-26T00:19:17.802 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T00:19:17.802 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-04-26 00:13:39.414384745 +0000 2024-04-26T00:19:17.802 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-04-26 00:13:39.138387211 +0000 2024-04-26T00:19:17.802 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-04-26 00:13:39.138387211 +0000 2024-04-26T00:19:17.803 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: - 2024-04-26T00:19:17.803 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T00:19:17.855 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2024-04-26T00:19:17.856 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2024-04-26T00:19:17.856 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000439033 s, 1.2 MB/s 2024-04-26T00:19:17.857 DEBUG:teuthology.orchestra.run.smithi037:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T00:19:17.906 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_3 2024-04-26T00:19:17.956 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T00:19:17.957 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T00:19:17.957 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 5h/5d Inode: 903 Links: 1 2024-04-26T00:19:17.958 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T00:19:17.958 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-04-26 00:13:39.898380422 +0000 2024-04-26T00:19:17.958 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-04-26 00:13:39.638382744 +0000 2024-04-26T00:19:17.958 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-04-26 00:13:39.638382744 +0000 2024-04-26T00:19:17.959 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: - 2024-04-26T00:19:17.959 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T00:19:18.012 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2024-04-26T00:19:18.012 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2024-04-26T00:19:18.012 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000410069 s, 1.2 MB/s 2024-04-26T00:19:18.013 DEBUG:teuthology.orchestra.run.smithi037:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T00:19:18.062 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_4 2024-04-26T00:19:18.109 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T00:19:18.109 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T00:19:18.109 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 5h/5d Inode: 920 Links: 1 2024-04-26T00:19:18.109 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T00:19:18.109 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-04-26 00:13:40.426375706 +0000 2024-04-26T00:19:18.109 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-04-26 00:13:40.146378207 +0000 2024-04-26T00:19:18.109 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-04-26 00:13:40.146378207 +0000 2024-04-26T00:19:18.110 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: - 2024-04-26T00:19:18.110 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T00:19:18.164 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2024-04-26T00:19:18.164 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2024-04-26T00:19:18.164 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000345542 s, 1.5 MB/s 2024-04-26T00:19:18.165 DEBUG:teuthology.orchestra.run.smithi037:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T00:19:18.214 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-26T00:19:18.214 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-26T00:19:18.214 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T00:19:18.220 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T00:19:18.220 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_1 2024-04-26T00:19:18.269 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T00:19:18.270 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T00:19:18.270 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 877 Links: 1 2024-04-26T00:19:18.270 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T00:19:18.270 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-04-26 00:13:36.673404527 +0000 2024-04-26T00:19:18.270 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-04-26 00:13:36.425406740 +0000 2024-04-26T00:19:18.270 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-04-26 00:13:36.425406740 +0000 2024-04-26T00:19:18.270 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: - 2024-04-26T00:19:18.270 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T00:19:18.324 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-26T00:19:18.324 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-26T00:19:18.324 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000307889 s, 1.7 MB/s 2024-04-26T00:19:18.326 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T00:19:18.376 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_2 2024-04-26T00:19:18.425 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T00:19:18.425 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T00:19:18.425 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 893 Links: 1 2024-04-26T00:19:18.425 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T00:19:18.425 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-04-26 00:13:37.161400174 +0000 2024-04-26T00:19:18.425 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-04-26 00:13:36.893402564 +0000 2024-04-26T00:19:18.425 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-04-26 00:13:36.893402564 +0000 2024-04-26T00:19:18.426 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: - 2024-04-26T00:19:18.426 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T00:19:18.479 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-26T00:19:18.480 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-26T00:19:18.480 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000311555 s, 1.6 MB/s 2024-04-26T00:19:18.481 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T00:19:18.530 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_3 2024-04-26T00:19:18.577 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T00:19:18.577 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T00:19:18.577 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 911 Links: 1 2024-04-26T00:19:18.577 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T00:19:18.578 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-04-26 00:13:37.649395820 +0000 2024-04-26T00:19:18.578 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-04-26 00:13:37.413397925 +0000 2024-04-26T00:19:18.578 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-04-26 00:13:37.413397925 +0000 2024-04-26T00:19:18.578 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: - 2024-04-26T00:19:18.578 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T00:19:18.632 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-26T00:19:18.632 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-26T00:19:18.632 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000307651 s, 1.7 MB/s 2024-04-26T00:19:18.633 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T00:19:18.686 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_4 2024-04-26T00:19:18.733 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T00:19:18.734 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T00:19:18.734 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 925 Links: 1 2024-04-26T00:19:18.734 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T00:19:18.734 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-04-26 00:13:38.137391467 +0000 2024-04-26T00:19:18.734 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-04-26 00:13:37.889393680 +0000 2024-04-26T00:19:18.734 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-04-26 00:13:37.889393680 +0000 2024-04-26T00:19:18.734 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: - 2024-04-26T00:19:18.734 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T00:19:18.788 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-26T00:19:18.788 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-26T00:19:18.788 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000294991 s, 1.7 MB/s 2024-04-26T00:19:18.789 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T00:19:18.843 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-04-26T00:19:18.844 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi037.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi045.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-04-26T00:19:18.844 INFO:tasks.ceph:Generating config... 2024-04-26T00:19:18.846 INFO:tasks.ceph:[client] debug monc = 20 2024-04-26T00:19:18.846 INFO:tasks.ceph:[client] debug ms = 1 2024-04-26T00:19:18.846 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-04-26T00:19:18.846 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-04-26T00:19:18.847 INFO:tasks.ceph:[global] ms bind msgr1 = False 2024-04-26T00:19:18.847 INFO:tasks.ceph:[global] ms bind msgr2 = True 2024-04-26T00:19:18.847 INFO:tasks.ceph:[global] ms inject delay max = 1 2024-04-26T00:19:18.847 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2024-04-26T00:19:18.848 INFO:tasks.ceph:[global] ms inject delay type = mon 2024-04-26T00:19:18.848 INFO:tasks.ceph:[global] ms inject internal delays = 0.002 2024-04-26T00:19:18.848 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2024-04-26T00:19:18.848 INFO:tasks.ceph:[global] ms type = async 2024-04-26T00:19:18.848 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-26T00:19:18.848 INFO:tasks.ceph:[mgr] debug monc = 10 2024-04-26T00:19:18.848 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-26T00:19:18.848 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-26T00:19:18.848 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-26T00:19:18.848 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-26T00:19:18.848 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-04-26T00:19:18.848 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-04-26T00:19:18.848 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-04-26T00:19:18.848 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-04-26T00:19:18.849 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-04-26T00:19:18.849 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-04-26T00:19:18.849 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-04-26T00:19:18.849 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-04-26T00:19:18.849 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-26T00:19:18.849 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-04-26T00:19:18.849 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-04-26T00:19:18.849 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-26T00:19:18.849 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-04-26T00:19:18.849 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-26T00:19:18.849 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-26T00:19:18.849 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-26T00:19:18.849 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-26T00:19:18.850 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-26T00:19:18.850 INFO:tasks.ceph:[osd] mon client ping interval = 4 2024-04-26T00:19:18.850 INFO:tasks.ceph:[osd] mon client ping timeout = 12 2024-04-26T00:19:18.850 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-26T00:19:18.850 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-26T00:19:18.850 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-26T00:19:18.850 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-04-26T00:19:18.850 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-04-26T00:19:18.850 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-26T00:19:18.850 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-04-26T00:19:18.850 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-04-26T00:19:18.850 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-26T00:19:18.851 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-04-26T00:19:18.851 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-04-26T00:19:18.851 INFO:tasks.ceph:Setting up mon.a... 2024-04-26T00:19:18.851 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-26T00:19:18.885 INFO:teuthology.orchestra.run.smithi037.stdout:creating /etc/ceph/ceph.keyring 2024-04-26T00:19:18.901 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-26T00:19:18.983 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-26T00:19:19.037 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.37'), ('mon.c', '[v2:172.21.15.37:3301,v1:172.21.15.37:6790]'), ('mon.b', '172.21.15.45')] 2024-04-26T00:19:19.038 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.37,[v2:172.21.15.37:3301,v1:172.21.15.37:6790],172.21.15.45', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client ping interval': 4, 'mon client ping timeout': 12, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 10}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-04-26T00:19:19.040 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T00:19:19.040 DEBUG:teuthology.orchestra.run.smithi037:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-26T00:19:19.085 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.37 --addv c '[v2:172.21.15.37:3301,v1:172.21.15.37:6790]' --add b 172.21.15.45 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-26T00:19:19.172 INFO:teuthology.orchestra.run.smithi037.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-26T00:19:19.172 INFO:teuthology.orchestra.run.smithi037.stdout:monmaptool: generated fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:19.172 INFO:teuthology.orchestra.run.smithi037.stdout:setting min_mon_release = quincy 2024-04-26T00:19:19.172 INFO:teuthology.orchestra.run.smithi037.stdout:epoch 0 2024-04-26T00:19:19.172 INFO:teuthology.orchestra.run.smithi037.stdout:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:19.172 INFO:teuthology.orchestra.run.smithi037.stdout:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:19.172 INFO:teuthology.orchestra.run.smithi037.stdout:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:19.172 INFO:teuthology.orchestra.run.smithi037.stdout:min_mon_release 17 (quincy) 2024-04-26T00:19:19.173 INFO:teuthology.orchestra.run.smithi037.stdout:election_strategy: 3 2024-04-26T00:19:19.173 INFO:teuthology.orchestra.run.smithi037.stdout:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:19.173 INFO:teuthology.orchestra.run.smithi037.stdout:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:19.173 INFO:teuthology.orchestra.run.smithi037.stdout:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:19.173 INFO:teuthology.orchestra.run.smithi037.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-26T00:19:19.174 DEBUG:teuthology.orchestra.run.smithi037:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-26T00:19:19.222 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 3cf2f905-fe97-4d94-8eba-4a21c519e869... 2024-04-26T00:19:19.223 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-26T00:19:19.266 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-26T00:19:19.295 INFO:teuthology.orchestra.run.smithi037.stdout:[global] 2024-04-26T00:19:19.295 INFO:teuthology.orchestra.run.smithi037.stdout: chdir = "" 2024-04-26T00:19:19.296 INFO:teuthology.orchestra.run.smithi037.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-26T00:19:19.296 INFO:teuthology.orchestra.run.smithi037.stdout: auth supported = cephx 2024-04-26T00:19:19.296 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:19:19.296 INFO:teuthology.orchestra.run.smithi037.stdout: filestore xattr use omap = true 2024-04-26T00:19:19.296 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:19:19.296 INFO:teuthology.orchestra.run.smithi037.stdout: mon clock drift allowed = 1.000 2024-04-26T00:19:19.296 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:19:19.296 INFO:teuthology.orchestra.run.smithi037.stdout: osd crush chooseleaf type = 0 2024-04-26T00:19:19.296 INFO:teuthology.orchestra.run.smithi037.stdout: auth debug = true 2024-04-26T00:19:19.296 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:19:19.296 INFO:teuthology.orchestra.run.smithi037.stdout: ms die on old message = true 2024-04-26T00:19:19.296 INFO:teuthology.orchestra.run.smithi037.stdout: ms die on bug = true 2024-04-26T00:19:19.296 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:19:19.296 INFO:teuthology.orchestra.run.smithi037.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-26T00:19:19.297 INFO:teuthology.orchestra.run.smithi037.stdout: mon pg warn max object skew = 0 2024-04-26T00:19:19.297 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:19:19.297 INFO:teuthology.orchestra.run.smithi037.stdout: # disable pg_autoscaler by default for new pools 2024-04-26T00:19:19.297 INFO:teuthology.orchestra.run.smithi037.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-26T00:19:19.297 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:19:19.297 INFO:teuthology.orchestra.run.smithi037.stdout: osd pool default size = 2 2024-04-26T00:19:19.297 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:19:19.297 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd allow primary affinity = true 2024-04-26T00:19:19.297 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd allow pg remap = true 2024-04-26T00:19:19.297 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on legacy crush tunables = false 2024-04-26T00:19:19.297 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on crush straw calc version zero = false 2024-04-26T00:19:19.297 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on no sortbitwise = false 2024-04-26T00:19:19.297 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on osd down out interval zero = false 2024-04-26T00:19:19.297 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on too few osds = false 2024-04-26T00:19:19.298 INFO:teuthology.orchestra.run.smithi037.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-26T00:19:19.298 INFO:teuthology.orchestra.run.smithi037.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-26T00:19:19.298 INFO:teuthology.orchestra.run.smithi037.stdout: mon_allow_pool_size_one = true 2024-04-26T00:19:19.298 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:19:19.298 INFO:teuthology.orchestra.run.smithi037.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-26T00:19:19.298 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:19:19.298 INFO:teuthology.orchestra.run.smithi037.stdout: osd default data pool replay window = 5 2024-04-26T00:19:19.298 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:19:19.298 INFO:teuthology.orchestra.run.smithi037.stdout: mon allow pool delete = true 2024-04-26T00:19:19.298 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:19:19.298 INFO:teuthology.orchestra.run.smithi037.stdout: mon cluster log file level = debug 2024-04-26T00:19:19.298 INFO:teuthology.orchestra.run.smithi037.stdout: debug asserts on shutdown = true 2024-04-26T00:19:19.298 INFO:teuthology.orchestra.run.smithi037.stdout: mon health detail to clog = false 2024-04-26T00:19:19.298 INFO:teuthology.orchestra.run.smithi037.stdout: mon host = "172.21.15.37,[v2:172.21.15.37:3301,v1:172.21.15.37:6790],172.21.15.45" 2024-04-26T00:19:19.299 INFO:teuthology.orchestra.run.smithi037.stdout: mon client directed command retry = 5 2024-04-26T00:19:19.299 INFO:teuthology.orchestra.run.smithi037.stdout: mon election default strategy = 3 2024-04-26T00:19:19.299 INFO:teuthology.orchestra.run.smithi037.stdout: ms bind msgr1 = False 2024-04-26T00:19:19.299 INFO:teuthology.orchestra.run.smithi037.stdout: ms bind msgr2 = True 2024-04-26T00:19:19.299 INFO:teuthology.orchestra.run.smithi037.stdout: ms inject delay max = 1 2024-04-26T00:19:19.299 INFO:teuthology.orchestra.run.smithi037.stdout: ms inject delay probability = 0.005 2024-04-26T00:19:19.299 INFO:teuthology.orchestra.run.smithi037.stdout: ms inject delay type = mon 2024-04-26T00:19:19.299 INFO:teuthology.orchestra.run.smithi037.stdout: ms inject internal delays = 0.002 2024-04-26T00:19:19.299 INFO:teuthology.orchestra.run.smithi037.stdout: ms inject socket failures = 2500 2024-04-26T00:19:19.299 INFO:teuthology.orchestra.run.smithi037.stdout: ms type = async 2024-04-26T00:19:19.299 INFO:teuthology.orchestra.run.smithi037.stdout: fsid = 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:19.299 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:19:19.299 INFO:teuthology.orchestra.run.smithi037.stdout:[osd] 2024-04-26T00:19:19.300 INFO:teuthology.orchestra.run.smithi037.stdout: osd journal size = 100 2024-04-26T00:19:19.300 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:19:19.300 INFO:teuthology.orchestra.run.smithi037.stdout: osd scrub load threshold = 5.0 2024-04-26T00:19:19.300 INFO:teuthology.orchestra.run.smithi037.stdout: osd scrub max interval = 600 2024-04-26T00:19:19.300 INFO:teuthology.orchestra.run.smithi037.stdout: osd mclock profile = high_recovery_ops 2024-04-26T00:19:19.300 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:19:19.300 INFO:teuthology.orchestra.run.smithi037.stdout: osd recover clone overlap = true 2024-04-26T00:19:19.300 INFO:teuthology.orchestra.run.smithi037.stdout: osd recovery max chunk = 1048576 2024-04-26T00:19:19.300 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:19:19.300 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug shutdown = true 2024-04-26T00:19:19.300 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug op order = true 2024-04-26T00:19:19.300 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug verify stray on activate = true 2024-04-26T00:19:19.300 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug trim objects = true 2024-04-26T00:19:19.300 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:19:19.300 INFO:teuthology.orchestra.run.smithi037.stdout: osd open classes on start = true 2024-04-26T00:19:19.301 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug pg log writeout = true 2024-04-26T00:19:19.301 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:19:19.301 INFO:teuthology.orchestra.run.smithi037.stdout: osd deep scrub update digest min age = 30 2024-04-26T00:19:19.301 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:19:19.301 INFO:teuthology.orchestra.run.smithi037.stdout: osd map max advance = 10 2024-04-26T00:19:19.301 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:19:19.301 INFO:teuthology.orchestra.run.smithi037.stdout: journal zero on create = true 2024-04-26T00:19:19.301 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:19:19.301 INFO:teuthology.orchestra.run.smithi037.stdout: filestore ondisk finisher threads = 3 2024-04-26T00:19:19.301 INFO:teuthology.orchestra.run.smithi037.stdout: filestore apply finisher threads = 3 2024-04-26T00:19:19.301 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:19:19.301 INFO:teuthology.orchestra.run.smithi037.stdout: bdev debug aio = true 2024-04-26T00:19:19.301 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug misdirected ops = true 2024-04-26T00:19:19.302 INFO:teuthology.orchestra.run.smithi037.stdout: bluestore block size = 96636764160 2024-04-26T00:19:19.302 INFO:teuthology.orchestra.run.smithi037.stdout: bluestore compression algorithm = snappy 2024-04-26T00:19:19.302 INFO:teuthology.orchestra.run.smithi037.stdout: bluestore compression mode = aggressive 2024-04-26T00:19:19.302 INFO:teuthology.orchestra.run.smithi037.stdout: bluestore fsck on mount = True 2024-04-26T00:19:19.302 INFO:teuthology.orchestra.run.smithi037.stdout: bluestore zero block detection = True 2024-04-26T00:19:19.302 INFO:teuthology.orchestra.run.smithi037.stdout: debug bluefs = 20 2024-04-26T00:19:19.302 INFO:teuthology.orchestra.run.smithi037.stdout: debug bluestore = 20 2024-04-26T00:19:19.302 INFO:teuthology.orchestra.run.smithi037.stdout: debug ms = 1 2024-04-26T00:19:19.302 INFO:teuthology.orchestra.run.smithi037.stdout: debug osd = 20 2024-04-26T00:19:19.302 INFO:teuthology.orchestra.run.smithi037.stdout: debug rocksdb = 10 2024-04-26T00:19:19.302 INFO:teuthology.orchestra.run.smithi037.stdout: mon client ping interval = 4 2024-04-26T00:19:19.302 INFO:teuthology.orchestra.run.smithi037.stdout: mon client ping timeout = 12 2024-04-26T00:19:19.302 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd backfillfull_ratio = 0.85 2024-04-26T00:19:19.302 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd full ratio = 0.9 2024-04-26T00:19:19.303 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd nearfull ratio = 0.8 2024-04-26T00:19:19.303 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug verify cached snaps = True 2024-04-26T00:19:19.303 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug verify missing on start = True 2024-04-26T00:19:19.303 INFO:teuthology.orchestra.run.smithi037.stdout: osd failsafe full ratio = 0.95 2024-04-26T00:19:19.303 INFO:teuthology.orchestra.run.smithi037.stdout: osd mclock override recovery settings = True 2024-04-26T00:19:19.303 INFO:teuthology.orchestra.run.smithi037.stdout: osd objectstore = bluestore 2024-04-26T00:19:19.303 INFO:teuthology.orchestra.run.smithi037.stdout: osd op queue = debug_random 2024-04-26T00:19:19.303 INFO:teuthology.orchestra.run.smithi037.stdout: osd op queue cut off = debug_random 2024-04-26T00:19:19.303 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:19:19.303 INFO:teuthology.orchestra.run.smithi037.stdout:[mgr] 2024-04-26T00:19:19.303 INFO:teuthology.orchestra.run.smithi037.stdout: debug ms = 1 2024-04-26T00:19:19.303 INFO:teuthology.orchestra.run.smithi037.stdout: debug mgr = 20 2024-04-26T00:19:19.304 INFO:teuthology.orchestra.run.smithi037.stdout: debug mon = 20 2024-04-26T00:19:19.304 INFO:teuthology.orchestra.run.smithi037.stdout: debug auth = 20 2024-04-26T00:19:19.304 INFO:teuthology.orchestra.run.smithi037.stdout: mon reweight min pgs per osd = 4 2024-04-26T00:19:19.304 INFO:teuthology.orchestra.run.smithi037.stdout: mon reweight min bytes per osd = 10 2024-04-26T00:19:19.304 INFO:teuthology.orchestra.run.smithi037.stdout: mgr/telemetry/nag = false 2024-04-26T00:19:19.304 INFO:teuthology.orchestra.run.smithi037.stdout: debug monc = 10 2024-04-26T00:19:19.304 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:19:19.304 INFO:teuthology.orchestra.run.smithi037.stdout:[mon] 2024-04-26T00:19:19.304 INFO:teuthology.orchestra.run.smithi037.stdout: debug ms = 1 2024-04-26T00:19:19.304 INFO:teuthology.orchestra.run.smithi037.stdout: debug mon = 20 2024-04-26T00:19:19.304 INFO:teuthology.orchestra.run.smithi037.stdout: debug paxos = 20 2024-04-26T00:19:19.304 INFO:teuthology.orchestra.run.smithi037.stdout: debug auth = 20 2024-04-26T00:19:19.304 INFO:teuthology.orchestra.run.smithi037.stdout: mon data avail warn = 5 2024-04-26T00:19:19.304 INFO:teuthology.orchestra.run.smithi037.stdout: mon mgr mkfs grace = 240 2024-04-26T00:19:19.304 INFO:teuthology.orchestra.run.smithi037.stdout: mon reweight min pgs per osd = 4 2024-04-26T00:19:19.305 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd reporter subtree level = osd 2024-04-26T00:19:19.305 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd prime pg temp = true 2024-04-26T00:19:19.305 INFO:teuthology.orchestra.run.smithi037.stdout: mon reweight min bytes per osd = 10 2024-04-26T00:19:19.305 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:19:19.305 INFO:teuthology.orchestra.run.smithi037.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-26T00:19:19.305 INFO:teuthology.orchestra.run.smithi037.stdout: auth mon ticket ttl = 660 # 11m 2024-04-26T00:19:19.305 INFO:teuthology.orchestra.run.smithi037.stdout: auth service ticket ttl = 240 # 4m 2024-04-26T00:19:19.305 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:19:19.305 INFO:teuthology.orchestra.run.smithi037.stdout: # don't complain about insecure global_id in the test suite 2024-04-26T00:19:19.305 INFO:teuthology.orchestra.run.smithi037.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-26T00:19:19.305 INFO:teuthology.orchestra.run.smithi037.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-26T00:19:19.305 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:19:19.305 INFO:teuthology.orchestra.run.smithi037.stdout: # 1m isn't quite enough 2024-04-26T00:19:19.306 INFO:teuthology.orchestra.run.smithi037.stdout: mon_down_mkfs_grace = 2m 2024-04-26T00:19:19.306 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:19:19.306 INFO:teuthology.orchestra.run.smithi037.stdout: mon_warn_on_filestore_osds = false 2024-04-26T00:19:19.306 INFO:teuthology.orchestra.run.smithi037.stdout: mon min osdmap epochs = 25 2024-04-26T00:19:19.306 INFO:teuthology.orchestra.run.smithi037.stdout: mon osdmap full prune interval = 2 2024-04-26T00:19:19.306 INFO:teuthology.orchestra.run.smithi037.stdout: mon osdmap full prune min = 15 2024-04-26T00:19:19.306 INFO:teuthology.orchestra.run.smithi037.stdout: mon osdmap full prune txsize = 2 2024-04-26T00:19:19.306 INFO:teuthology.orchestra.run.smithi037.stdout: mon scrub inject crc mismatch = 0.01 2024-04-26T00:19:19.306 INFO:teuthology.orchestra.run.smithi037.stdout: mon scrub inject missing keys = 0.05 2024-04-26T00:19:19.306 INFO:teuthology.orchestra.run.smithi037.stdout: mon scrub interval = 300 2024-04-26T00:19:19.306 INFO:teuthology.orchestra.run.smithi037.stdout: paxos service trim min = 5 2024-04-26T00:19:19.306 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:19:19.306 INFO:teuthology.orchestra.run.smithi037.stdout:[client] 2024-04-26T00:19:19.306 INFO:teuthology.orchestra.run.smithi037.stdout: rgw cache enabled = true 2024-04-26T00:19:19.306 INFO:teuthology.orchestra.run.smithi037.stdout: rgw enable ops log = true 2024-04-26T00:19:19.307 INFO:teuthology.orchestra.run.smithi037.stdout: rgw enable usage log = true 2024-04-26T00:19:19.307 INFO:teuthology.orchestra.run.smithi037.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-26T00:19:19.307 INFO:teuthology.orchestra.run.smithi037.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-26T00:19:19.307 INFO:teuthology.orchestra.run.smithi037.stdout: debug monc = 20 2024-04-26T00:19:19.307 INFO:teuthology.orchestra.run.smithi037.stdout: debug ms = 1 2024-04-26T00:19:19.307 INFO:teuthology.orchestra.run.smithi037.stdout:[mon.a] 2024-04-26T00:19:19.307 INFO:teuthology.orchestra.run.smithi037.stdout:[mon.c] 2024-04-26T00:19:19.307 INFO:teuthology.orchestra.run.smithi037.stdout:[mon.b] 2024-04-26T00:19:19.308 INFO:teuthology.orchestra.run.smithi045.stdout:[global] 2024-04-26T00:19:19.308 INFO:teuthology.orchestra.run.smithi045.stdout: chdir = "" 2024-04-26T00:19:19.308 INFO:teuthology.orchestra.run.smithi045.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-26T00:19:19.308 INFO:teuthology.orchestra.run.smithi045.stdout: auth supported = cephx 2024-04-26T00:19:19.308 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T00:19:19.308 INFO:teuthology.orchestra.run.smithi045.stdout: filestore xattr use omap = true 2024-04-26T00:19:19.308 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T00:19:19.308 INFO:teuthology.orchestra.run.smithi045.stdout: mon clock drift allowed = 1.000 2024-04-26T00:19:19.308 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T00:19:19.308 INFO:teuthology.orchestra.run.smithi045.stdout: osd crush chooseleaf type = 0 2024-04-26T00:19:19.309 INFO:teuthology.orchestra.run.smithi045.stdout: auth debug = true 2024-04-26T00:19:19.309 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T00:19:19.309 INFO:teuthology.orchestra.run.smithi045.stdout: ms die on old message = true 2024-04-26T00:19:19.309 INFO:teuthology.orchestra.run.smithi045.stdout: ms die on bug = true 2024-04-26T00:19:19.309 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T00:19:19.309 INFO:teuthology.orchestra.run.smithi045.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-26T00:19:19.309 INFO:teuthology.orchestra.run.smithi045.stdout: mon pg warn max object skew = 0 2024-04-26T00:19:19.309 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T00:19:19.310 INFO:teuthology.orchestra.run.smithi045.stdout: # disable pg_autoscaler by default for new pools 2024-04-26T00:19:19.310 INFO:teuthology.orchestra.run.smithi045.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-26T00:19:19.310 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T00:19:19.310 INFO:teuthology.orchestra.run.smithi045.stdout: osd pool default size = 2 2024-04-26T00:19:19.310 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T00:19:19.310 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd allow primary affinity = true 2024-04-26T00:19:19.310 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd allow pg remap = true 2024-04-26T00:19:19.310 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on legacy crush tunables = false 2024-04-26T00:19:19.310 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on crush straw calc version zero = false 2024-04-26T00:19:19.310 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on no sortbitwise = false 2024-04-26T00:19:19.310 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on osd down out interval zero = false 2024-04-26T00:19:19.311 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on too few osds = false 2024-04-26T00:19:19.311 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-26T00:19:19.311 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-26T00:19:19.311 INFO:teuthology.orchestra.run.smithi045.stdout: mon_allow_pool_size_one = true 2024-04-26T00:19:19.311 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T00:19:19.311 INFO:teuthology.orchestra.run.smithi045.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-26T00:19:19.311 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T00:19:19.311 INFO:teuthology.orchestra.run.smithi045.stdout: osd default data pool replay window = 5 2024-04-26T00:19:19.311 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T00:19:19.311 INFO:teuthology.orchestra.run.smithi045.stdout: mon allow pool delete = true 2024-04-26T00:19:19.311 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T00:19:19.311 INFO:teuthology.orchestra.run.smithi045.stdout: mon cluster log file level = debug 2024-04-26T00:19:19.311 INFO:teuthology.orchestra.run.smithi045.stdout: debug asserts on shutdown = true 2024-04-26T00:19:19.311 INFO:teuthology.orchestra.run.smithi045.stdout: mon health detail to clog = false 2024-04-26T00:19:19.311 INFO:teuthology.orchestra.run.smithi045.stdout: mon host = "172.21.15.37,[v2:172.21.15.37:3301,v1:172.21.15.37:6790],172.21.15.45" 2024-04-26T00:19:19.312 INFO:teuthology.orchestra.run.smithi045.stdout: mon client directed command retry = 5 2024-04-26T00:19:19.312 INFO:teuthology.orchestra.run.smithi045.stdout: mon election default strategy = 3 2024-04-26T00:19:19.312 INFO:teuthology.orchestra.run.smithi045.stdout: ms bind msgr1 = False 2024-04-26T00:19:19.312 INFO:teuthology.orchestra.run.smithi045.stdout: ms bind msgr2 = True 2024-04-26T00:19:19.312 INFO:teuthology.orchestra.run.smithi045.stdout: ms inject delay max = 1 2024-04-26T00:19:19.312 INFO:teuthology.orchestra.run.smithi045.stdout: ms inject delay probability = 0.005 2024-04-26T00:19:19.312 INFO:teuthology.orchestra.run.smithi045.stdout: ms inject delay type = mon 2024-04-26T00:19:19.312 INFO:teuthology.orchestra.run.smithi045.stdout: ms inject internal delays = 0.002 2024-04-26T00:19:19.312 INFO:teuthology.orchestra.run.smithi045.stdout: ms inject socket failures = 2500 2024-04-26T00:19:19.312 INFO:teuthology.orchestra.run.smithi045.stdout: ms type = async 2024-04-26T00:19:19.312 INFO:teuthology.orchestra.run.smithi045.stdout: fsid = 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:19.312 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T00:19:19.312 INFO:teuthology.orchestra.run.smithi045.stdout:[osd] 2024-04-26T00:19:19.312 INFO:teuthology.orchestra.run.smithi045.stdout: osd journal size = 100 2024-04-26T00:19:19.313 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T00:19:19.313 INFO:teuthology.orchestra.run.smithi045.stdout: osd scrub load threshold = 5.0 2024-04-26T00:19:19.313 INFO:teuthology.orchestra.run.smithi045.stdout: osd scrub max interval = 600 2024-04-26T00:19:19.313 INFO:teuthology.orchestra.run.smithi045.stdout: osd mclock profile = high_recovery_ops 2024-04-26T00:19:19.313 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T00:19:19.313 INFO:teuthology.orchestra.run.smithi045.stdout: osd recover clone overlap = true 2024-04-26T00:19:19.313 INFO:teuthology.orchestra.run.smithi045.stdout: osd recovery max chunk = 1048576 2024-04-26T00:19:19.313 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T00:19:19.313 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug shutdown = true 2024-04-26T00:19:19.313 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug op order = true 2024-04-26T00:19:19.313 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug verify stray on activate = true 2024-04-26T00:19:19.313 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug trim objects = true 2024-04-26T00:19:19.313 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T00:19:19.314 INFO:teuthology.orchestra.run.smithi045.stdout: osd open classes on start = true 2024-04-26T00:19:19.314 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug pg log writeout = true 2024-04-26T00:19:19.314 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T00:19:19.314 INFO:teuthology.orchestra.run.smithi045.stdout: osd deep scrub update digest min age = 30 2024-04-26T00:19:19.314 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T00:19:19.314 INFO:teuthology.orchestra.run.smithi045.stdout: osd map max advance = 10 2024-04-26T00:19:19.314 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T00:19:19.314 INFO:teuthology.orchestra.run.smithi045.stdout: journal zero on create = true 2024-04-26T00:19:19.314 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T00:19:19.314 INFO:teuthology.orchestra.run.smithi045.stdout: filestore ondisk finisher threads = 3 2024-04-26T00:19:19.314 INFO:teuthology.orchestra.run.smithi045.stdout: filestore apply finisher threads = 3 2024-04-26T00:19:19.314 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T00:19:19.314 INFO:teuthology.orchestra.run.smithi045.stdout: bdev debug aio = true 2024-04-26T00:19:19.314 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug misdirected ops = true 2024-04-26T00:19:19.314 INFO:teuthology.orchestra.run.smithi045.stdout: bluestore block size = 96636764160 2024-04-26T00:19:19.315 INFO:teuthology.orchestra.run.smithi045.stdout: bluestore compression algorithm = snappy 2024-04-26T00:19:19.315 INFO:teuthology.orchestra.run.smithi045.stdout: bluestore compression mode = aggressive 2024-04-26T00:19:19.315 INFO:teuthology.orchestra.run.smithi045.stdout: bluestore fsck on mount = True 2024-04-26T00:19:19.315 INFO:teuthology.orchestra.run.smithi045.stdout: bluestore zero block detection = True 2024-04-26T00:19:19.315 INFO:teuthology.orchestra.run.smithi045.stdout: debug bluefs = 20 2024-04-26T00:19:19.315 INFO:teuthology.orchestra.run.smithi045.stdout: debug bluestore = 20 2024-04-26T00:19:19.315 INFO:teuthology.orchestra.run.smithi045.stdout: debug ms = 1 2024-04-26T00:19:19.315 INFO:teuthology.orchestra.run.smithi045.stdout: debug osd = 20 2024-04-26T00:19:19.315 INFO:teuthology.orchestra.run.smithi045.stdout: debug rocksdb = 10 2024-04-26T00:19:19.315 INFO:teuthology.orchestra.run.smithi045.stdout: mon client ping interval = 4 2024-04-26T00:19:19.315 INFO:teuthology.orchestra.run.smithi045.stdout: mon client ping timeout = 12 2024-04-26T00:19:19.316 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd backfillfull_ratio = 0.85 2024-04-26T00:19:19.316 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd full ratio = 0.9 2024-04-26T00:19:19.316 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd nearfull ratio = 0.8 2024-04-26T00:19:19.316 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug verify cached snaps = True 2024-04-26T00:19:19.316 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug verify missing on start = True 2024-04-26T00:19:19.316 INFO:teuthology.orchestra.run.smithi045.stdout: osd failsafe full ratio = 0.95 2024-04-26T00:19:19.316 INFO:teuthology.orchestra.run.smithi045.stdout: osd mclock override recovery settings = True 2024-04-26T00:19:19.316 INFO:teuthology.orchestra.run.smithi045.stdout: osd objectstore = bluestore 2024-04-26T00:19:19.316 INFO:teuthology.orchestra.run.smithi045.stdout: osd op queue = debug_random 2024-04-26T00:19:19.316 INFO:teuthology.orchestra.run.smithi045.stdout: osd op queue cut off = debug_random 2024-04-26T00:19:19.316 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T00:19:19.316 INFO:teuthology.orchestra.run.smithi045.stdout:[mgr] 2024-04-26T00:19:19.316 INFO:teuthology.orchestra.run.smithi045.stdout: debug ms = 1 2024-04-26T00:19:19.316 INFO:teuthology.orchestra.run.smithi045.stdout: debug mgr = 20 2024-04-26T00:19:19.316 INFO:teuthology.orchestra.run.smithi045.stdout: debug mon = 20 2024-04-26T00:19:19.317 INFO:teuthology.orchestra.run.smithi045.stdout: debug auth = 20 2024-04-26T00:19:19.317 INFO:teuthology.orchestra.run.smithi045.stdout: mon reweight min pgs per osd = 4 2024-04-26T00:19:19.317 INFO:teuthology.orchestra.run.smithi045.stdout: mon reweight min bytes per osd = 10 2024-04-26T00:19:19.317 INFO:teuthology.orchestra.run.smithi045.stdout: mgr/telemetry/nag = false 2024-04-26T00:19:19.317 INFO:teuthology.orchestra.run.smithi045.stdout: debug monc = 10 2024-04-26T00:19:19.317 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T00:19:19.317 INFO:teuthology.orchestra.run.smithi045.stdout:[mon] 2024-04-26T00:19:19.317 INFO:teuthology.orchestra.run.smithi045.stdout: debug ms = 1 2024-04-26T00:19:19.317 INFO:teuthology.orchestra.run.smithi045.stdout: debug mon = 20 2024-04-26T00:19:19.317 INFO:teuthology.orchestra.run.smithi045.stdout: debug paxos = 20 2024-04-26T00:19:19.317 INFO:teuthology.orchestra.run.smithi045.stdout: debug auth = 20 2024-04-26T00:19:19.317 INFO:teuthology.orchestra.run.smithi045.stdout: mon data avail warn = 5 2024-04-26T00:19:19.317 INFO:teuthology.orchestra.run.smithi045.stdout: mon mgr mkfs grace = 240 2024-04-26T00:19:19.317 INFO:teuthology.orchestra.run.smithi045.stdout: mon reweight min pgs per osd = 4 2024-04-26T00:19:19.317 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd reporter subtree level = osd 2024-04-26T00:19:19.318 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd prime pg temp = true 2024-04-26T00:19:19.318 INFO:teuthology.orchestra.run.smithi045.stdout: mon reweight min bytes per osd = 10 2024-04-26T00:19:19.318 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T00:19:19.318 INFO:teuthology.orchestra.run.smithi045.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-26T00:19:19.318 INFO:teuthology.orchestra.run.smithi045.stdout: auth mon ticket ttl = 660 # 11m 2024-04-26T00:19:19.318 INFO:teuthology.orchestra.run.smithi045.stdout: auth service ticket ttl = 240 # 4m 2024-04-26T00:19:19.318 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T00:19:19.318 INFO:teuthology.orchestra.run.smithi045.stdout: # don't complain about insecure global_id in the test suite 2024-04-26T00:19:19.318 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-26T00:19:19.318 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-26T00:19:19.318 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T00:19:19.318 INFO:teuthology.orchestra.run.smithi045.stdout: # 1m isn't quite enough 2024-04-26T00:19:19.318 INFO:teuthology.orchestra.run.smithi045.stdout: mon_down_mkfs_grace = 2m 2024-04-26T00:19:19.318 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T00:19:19.319 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_filestore_osds = false 2024-04-26T00:19:19.319 INFO:teuthology.orchestra.run.smithi045.stdout: mon min osdmap epochs = 25 2024-04-26T00:19:19.319 INFO:teuthology.orchestra.run.smithi045.stdout: mon osdmap full prune interval = 2 2024-04-26T00:19:19.319 INFO:teuthology.orchestra.run.smithi045.stdout: mon osdmap full prune min = 15 2024-04-26T00:19:19.319 INFO:teuthology.orchestra.run.smithi045.stdout: mon osdmap full prune txsize = 2 2024-04-26T00:19:19.319 INFO:teuthology.orchestra.run.smithi045.stdout: mon scrub inject crc mismatch = 0.01 2024-04-26T00:19:19.319 INFO:teuthology.orchestra.run.smithi045.stdout: mon scrub inject missing keys = 0.05 2024-04-26T00:19:19.319 INFO:teuthology.orchestra.run.smithi045.stdout: mon scrub interval = 300 2024-04-26T00:19:19.319 INFO:teuthology.orchestra.run.smithi045.stdout: paxos service trim min = 5 2024-04-26T00:19:19.319 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T00:19:19.319 INFO:teuthology.orchestra.run.smithi045.stdout:[client] 2024-04-26T00:19:19.319 INFO:teuthology.orchestra.run.smithi045.stdout: rgw cache enabled = true 2024-04-26T00:19:19.319 INFO:teuthology.orchestra.run.smithi045.stdout: rgw enable ops log = true 2024-04-26T00:19:19.319 INFO:teuthology.orchestra.run.smithi045.stdout: rgw enable usage log = true 2024-04-26T00:19:19.319 INFO:teuthology.orchestra.run.smithi045.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-26T00:19:19.320 INFO:teuthology.orchestra.run.smithi045.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-26T00:19:19.320 INFO:teuthology.orchestra.run.smithi045.stdout: debug monc = 20 2024-04-26T00:19:19.320 INFO:teuthology.orchestra.run.smithi045.stdout: debug ms = 1 2024-04-26T00:19:19.320 INFO:teuthology.orchestra.run.smithi045.stdout:[mon.a] 2024-04-26T00:19:19.320 INFO:teuthology.orchestra.run.smithi045.stdout:[mon.c] 2024-04-26T00:19:19.320 INFO:teuthology.orchestra.run.smithi045.stdout:[mon.b] 2024-04-26T00:19:19.321 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-26T00:19:19.322 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-26T00:19:19.415 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-26T00:19:19.415 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T00:19:19.415 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-26T00:19:19.421 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T00:19:19.422 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-26T00:19:19.469 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi037.front.sepia.ceph.com 2024-04-26T00:19:19.470 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T00:19:19.470 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-26T00:19:19.470 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-26T00:19:19.533 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T00:19:19.533 DEBUG:teuthology.orchestra.run.smithi037:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-26T00:19:19.583 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi045.front.sepia.ceph.com 2024-04-26T00:19:19.584 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-26T00:19:19.584 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-26T00:19:19.584 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-26T00:19:19.610 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-26T00:19:19.610 DEBUG:teuthology.orchestra.run.smithi045:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-26T00:19:19.702 INFO:tasks.ceph:Setting up mon nodes... 2024-04-26T00:19:19.702 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-26T00:19:19.702 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-26T00:19:19.790 INFO:teuthology.orchestra.run.smithi045.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-26T00:19:19.803 INFO:tasks.ceph:Setting up mds nodes... 2024-04-26T00:19:19.804 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-26T00:19:19.804 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-26T00:19:19.838 INFO:teuthology.orchestra.run.smithi045.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-26T00:19:19.850 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-26T00:19:19.851 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi037.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi045.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-04-26T00:19:19.851 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-26T00:19:19.866 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-26T00:19:19.866 INFO:tasks.ceph:role: osd.0 2024-04-26T00:19:19.866 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi037.front.sepia.ceph.com 2024-04-26T00:19:19.866 DEBUG:teuthology.orchestra.run.smithi037:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-26T00:19:19.929 INFO:teuthology.orchestra.run.smithi037.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T00:19:19.929 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T00:19:19.929 INFO:teuthology.orchestra.run.smithi037.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T00:19:19.929 INFO:teuthology.orchestra.run.smithi037.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T00:19:19.929 INFO:teuthology.orchestra.run.smithi037.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T00:19:19.930 INFO:teuthology.orchestra.run.smithi037.stdout: = sunit=0 swidth=0 blks 2024-04-26T00:19:19.930 INFO:teuthology.orchestra.run.smithi037.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T00:19:19.930 INFO:teuthology.orchestra.run.smithi037.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T00:19:19.930 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T00:19:19.930 INFO:teuthology.orchestra.run.smithi037.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T00:19:20.402 INFO:teuthology.orchestra.run.smithi037.stdout:Discarding blocks...Done. 2024-04-26T00:19:20.404 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi037.front.sepia.ceph.com -o noatime 2024-04-26T00:19:20.404 DEBUG:teuthology.orchestra.run.smithi037:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-26T00:19:20.557 DEBUG:teuthology.orchestra.run.smithi037:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-26T00:19:20.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T00:19:20.569 INFO:teuthology.orchestra.run.smithi037.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T00:19:20.569 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-26T00:19:20.625 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-26T00:19:20.625 INFO:tasks.ceph:role: osd.1 2024-04-26T00:19:20.625 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi037.front.sepia.ceph.com 2024-04-26T00:19:20.625 DEBUG:teuthology.orchestra.run.smithi037:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-26T00:19:20.689 INFO:teuthology.orchestra.run.smithi037.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T00:19:20.690 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T00:19:20.690 INFO:teuthology.orchestra.run.smithi037.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T00:19:20.690 INFO:teuthology.orchestra.run.smithi037.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T00:19:20.690 INFO:teuthology.orchestra.run.smithi037.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T00:19:20.690 INFO:teuthology.orchestra.run.smithi037.stdout: = sunit=0 swidth=0 blks 2024-04-26T00:19:20.690 INFO:teuthology.orchestra.run.smithi037.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T00:19:20.690 INFO:teuthology.orchestra.run.smithi037.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T00:19:20.690 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T00:19:20.690 INFO:teuthology.orchestra.run.smithi037.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T00:19:21.127 INFO:teuthology.orchestra.run.smithi037.stdout:Discarding blocks...Done. 2024-04-26T00:19:21.129 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi037.front.sepia.ceph.com -o noatime 2024-04-26T00:19:21.129 DEBUG:teuthology.orchestra.run.smithi037:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-26T00:19:21.218 DEBUG:teuthology.orchestra.run.smithi037:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-26T00:19:21.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T00:19:21.229 INFO:teuthology.orchestra.run.smithi037.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T00:19:21.229 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-26T00:19:21.283 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-26T00:19:21.283 INFO:tasks.ceph:role: osd.2 2024-04-26T00:19:21.283 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi037.front.sepia.ceph.com 2024-04-26T00:19:21.283 DEBUG:teuthology.orchestra.run.smithi037:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-26T00:19:21.345 INFO:teuthology.orchestra.run.smithi037.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T00:19:21.345 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T00:19:21.345 INFO:teuthology.orchestra.run.smithi037.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T00:19:21.345 INFO:teuthology.orchestra.run.smithi037.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T00:19:21.345 INFO:teuthology.orchestra.run.smithi037.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T00:19:21.345 INFO:teuthology.orchestra.run.smithi037.stdout: = sunit=0 swidth=0 blks 2024-04-26T00:19:21.345 INFO:teuthology.orchestra.run.smithi037.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T00:19:21.345 INFO:teuthology.orchestra.run.smithi037.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T00:19:21.345 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T00:19:21.345 INFO:teuthology.orchestra.run.smithi037.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T00:19:21.799 INFO:teuthology.orchestra.run.smithi037.stdout:Discarding blocks...Done. 2024-04-26T00:19:21.802 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi037.front.sepia.ceph.com -o noatime 2024-04-26T00:19:21.802 DEBUG:teuthology.orchestra.run.smithi037:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-26T00:19:21.905 DEBUG:teuthology.orchestra.run.smithi037:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-26T00:19:21.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T00:19:21.916 INFO:teuthology.orchestra.run.smithi037.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T00:19:21.916 DEBUG:teuthology.orchestra.run.smithi037:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T00:19:22.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:22.072+0000 7f4df77017c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-26T00:19:22.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:22.072+0000 7f4df77017c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-26T00:19:22.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:22.072+0000 7f4df77017c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-26T00:19:22.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:22.072+0000 7f4df77017c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-26T00:19:22.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:22.072+0000 7f4df77017c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-26T00:19:22.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:22.084+0000 7f4df77017c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-26T00:19:25.045 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-26T00:19:25.059 DEBUG:teuthology.orchestra.run.smithi037:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T00:19:25.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:25.156+0000 7f17542647c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-26T00:19:25.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:25.156+0000 7f17542647c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-26T00:19:25.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:25.156+0000 7f17542647c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-26T00:19:25.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:25.156+0000 7f17542647c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-26T00:19:25.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:25.156+0000 7f17542647c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-26T00:19:25.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:25.160+0000 7f17542647c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-26T00:19:28.123 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-26T00:19:28.179 DEBUG:teuthology.orchestra.run.smithi037:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T00:19:28.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:28.280+0000 7ff8f668b7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-26T00:19:28.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:28.280+0000 7ff8f668b7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-26T00:19:28.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:28.280+0000 7ff8f668b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-26T00:19:28.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:28.280+0000 7ff8f668b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-26T00:19:28.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:28.280+0000 7ff8f668b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-26T00:19:28.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:28.280+0000 7ff8f668b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-26T00:19:31.227 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-26T00:19:31.285 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-26T00:19:31.301 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-04-26T00:19:31.301 INFO:tasks.ceph:role: osd.3 2024-04-26T00:19:31.301 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi045.front.sepia.ceph.com 2024-04-26T00:19:31.301 DEBUG:teuthology.orchestra.run.smithi045:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-26T00:19:31.357 INFO:teuthology.orchestra.run.smithi045.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T00:19:31.357 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T00:19:31.357 INFO:teuthology.orchestra.run.smithi045.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T00:19:31.357 INFO:teuthology.orchestra.run.smithi045.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T00:19:31.357 INFO:teuthology.orchestra.run.smithi045.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T00:19:31.357 INFO:teuthology.orchestra.run.smithi045.stdout: = sunit=0 swidth=0 blks 2024-04-26T00:19:31.357 INFO:teuthology.orchestra.run.smithi045.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T00:19:31.358 INFO:teuthology.orchestra.run.smithi045.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T00:19:31.358 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T00:19:31.358 INFO:teuthology.orchestra.run.smithi045.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T00:19:31.380 INFO:teuthology.orchestra.run.smithi045.stdout:Discarding blocks...Done. 2024-04-26T00:19:31.382 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi045.front.sepia.ceph.com -o noatime 2024-04-26T00:19:31.382 DEBUG:teuthology.orchestra.run.smithi045:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-04-26T00:19:31.441 DEBUG:teuthology.orchestra.run.smithi045:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-26T00:19:31.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T00:19:31.494 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T00:19:31.494 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-26T00:19:31.546 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-04-26T00:19:31.546 INFO:tasks.ceph:role: osd.4 2024-04-26T00:19:31.546 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi045.front.sepia.ceph.com 2024-04-26T00:19:31.547 DEBUG:teuthology.orchestra.run.smithi045:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-26T00:19:31.600 INFO:teuthology.orchestra.run.smithi045.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T00:19:31.600 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T00:19:31.600 INFO:teuthology.orchestra.run.smithi045.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T00:19:31.600 INFO:teuthology.orchestra.run.smithi045.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T00:19:31.601 INFO:teuthology.orchestra.run.smithi045.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T00:19:31.601 INFO:teuthology.orchestra.run.smithi045.stdout: = sunit=0 swidth=0 blks 2024-04-26T00:19:31.601 INFO:teuthology.orchestra.run.smithi045.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T00:19:31.601 INFO:teuthology.orchestra.run.smithi045.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T00:19:31.601 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T00:19:31.601 INFO:teuthology.orchestra.run.smithi045.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T00:19:31.624 INFO:teuthology.orchestra.run.smithi045.stdout:Discarding blocks...Done. 2024-04-26T00:19:31.626 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi045.front.sepia.ceph.com -o noatime 2024-04-26T00:19:31.626 DEBUG:teuthology.orchestra.run.smithi045:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-04-26T00:19:31.683 DEBUG:teuthology.orchestra.run.smithi045:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-26T00:19:31.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T00:19:31.734 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T00:19:31.734 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-26T00:19:31.786 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-04-26T00:19:31.786 INFO:tasks.ceph:role: osd.5 2024-04-26T00:19:31.786 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi045.front.sepia.ceph.com 2024-04-26T00:19:31.787 DEBUG:teuthology.orchestra.run.smithi045:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-26T00:19:31.841 INFO:teuthology.orchestra.run.smithi045.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T00:19:31.841 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T00:19:31.841 INFO:teuthology.orchestra.run.smithi045.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T00:19:31.841 INFO:teuthology.orchestra.run.smithi045.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T00:19:31.841 INFO:teuthology.orchestra.run.smithi045.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T00:19:31.841 INFO:teuthology.orchestra.run.smithi045.stdout: = sunit=0 swidth=0 blks 2024-04-26T00:19:31.841 INFO:teuthology.orchestra.run.smithi045.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T00:19:31.841 INFO:teuthology.orchestra.run.smithi045.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T00:19:31.841 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T00:19:31.841 INFO:teuthology.orchestra.run.smithi045.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T00:19:31.864 INFO:teuthology.orchestra.run.smithi045.stdout:Discarding blocks...Done. 2024-04-26T00:19:31.866 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi045.front.sepia.ceph.com -o noatime 2024-04-26T00:19:31.866 DEBUG:teuthology.orchestra.run.smithi045:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-04-26T00:19:31.924 DEBUG:teuthology.orchestra.run.smithi045:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-26T00:19:31.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T00:19:31.977 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T00:19:31.978 DEBUG:teuthology.orchestra.run.smithi045:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T00:19:32.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:32.102+0000 7fc2e6eda7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-26T00:19:32.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:32.102+0000 7fc2e6eda7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-26T00:19:32.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:32.102+0000 7fc2e6eda7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-26T00:19:32.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:32.102+0000 7fc2e6eda7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-26T00:19:32.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:32.102+0000 7fc2e6eda7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-26T00:19:32.084 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:32.102+0000 7fc2e6eda7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-26T00:19:35.049 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-26T00:19:35.062 DEBUG:teuthology.orchestra.run.smithi045:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T00:19:35.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:35.162+0000 7f2f881527c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-26T00:19:35.144 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:35.162+0000 7f2f881527c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-26T00:19:35.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:35.162+0000 7f2f881527c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-26T00:19:35.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:35.162+0000 7f2f881527c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-26T00:19:35.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:35.162+0000 7f2f881527c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-26T00:19:35.145 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:35.166+0000 7f2f881527c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-26T00:19:38.122 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-26T00:19:38.182 DEBUG:teuthology.orchestra.run.smithi045:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T00:19:38.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:38.282+0000 7facff1387c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-26T00:19:38.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:38.286+0000 7facff1387c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-26T00:19:38.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:38.286+0000 7facff1387c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-26T00:19:38.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:38.286+0000 7facff1387c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-26T00:19:38.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:38.286+0000 7facff1387c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-26T00:19:38.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:38.286+0000 7facff1387c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-26T00:19:41.245 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-26T00:19:41.302 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-26T00:19:41.302 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T00:19:41.302 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-26T00:19:41.317 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T00:19:41.317 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-26T00:19:41.372 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T00:19:41.372 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-26T00:19:41.429 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-26T00:19:41.429 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-26T00:19:41.444 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-26T00:19:41.444 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-26T00:19:41.500 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-26T00:19:41.500 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-26T00:19:41.556 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-26T00:19:41.556 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-26T00:19:41.612 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-26T00:19:41.612 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-26T00:19:41.661 INFO:tasks.ceph:Adding keys to all mons... 2024-04-26T00:19:41.661 DEBUG:teuthology.orchestra.run.smithi037:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-26T00:19:41.664 DEBUG:teuthology.orchestra.run.smithi045:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-26T00:19:41.706 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.0] 2024-04-26T00:19:41.707 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQAK8ypmisiXBBAAZdD84ttk4Au+ZquPU+aZAA== 2024-04-26T00:19:41.707 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.1] 2024-04-26T00:19:41.707 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQAN8ypmOte2CRAA2bm1T1V69VeF8ala7GEC5A== 2024-04-26T00:19:41.707 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.2] 2024-04-26T00:19:41.707 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQAQ8ypmg83yEBAAq9cqx2sy/ae7820GiRZXSQ== 2024-04-26T00:19:41.707 INFO:teuthology.orchestra.run.smithi037.stdout:[mgr.x] 2024-04-26T00:19:41.707 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQAH8ypmIx6ZMBAAWYA4kCHum44nBVzhW6BGIw== 2024-04-26T00:19:41.707 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.3] 2024-04-26T00:19:41.707 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQAU8ypmpT95BhAAZHSX3NyDMRGANgANAwMIzg== 2024-04-26T00:19:41.707 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.4] 2024-04-26T00:19:41.707 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQAX8ypmFVIZChAABCuamS3Hf/yMlIJ47B7RZg== 2024-04-26T00:19:41.707 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.5] 2024-04-26T00:19:41.707 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQAa8ypm8kxJERAA3aMSOHKp4zpMKBxF1PKa4Q== 2024-04-26T00:19:41.708 INFO:teuthology.orchestra.run.smithi037.stdout:[client.0] 2024-04-26T00:19:41.708 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQAH8ypmuzeCMxAALt8PqORZJo2a2PIsFgGIOA== 2024-04-26T00:19:41.715 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.0] 2024-04-26T00:19:41.715 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQAK8ypmisiXBBAAZdD84ttk4Au+ZquPU+aZAA== 2024-04-26T00:19:41.715 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.1] 2024-04-26T00:19:41.715 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQAN8ypmOte2CRAA2bm1T1V69VeF8ala7GEC5A== 2024-04-26T00:19:41.715 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.2] 2024-04-26T00:19:41.715 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQAQ8ypmg83yEBAAq9cqx2sy/ae7820GiRZXSQ== 2024-04-26T00:19:41.715 INFO:teuthology.orchestra.run.smithi045.stdout:[mgr.x] 2024-04-26T00:19:41.715 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQAH8ypmIx6ZMBAAWYA4kCHum44nBVzhW6BGIw== 2024-04-26T00:19:41.715 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.3] 2024-04-26T00:19:41.715 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQAU8ypmpT95BhAAZHSX3NyDMRGANgANAwMIzg== 2024-04-26T00:19:41.715 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.4] 2024-04-26T00:19:41.715 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQAX8ypmFVIZChAABCuamS3Hf/yMlIJ47B7RZg== 2024-04-26T00:19:41.716 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.5] 2024-04-26T00:19:41.716 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQAa8ypm8kxJERAA3aMSOHKp4zpMKBxF1PKa4Q== 2024-04-26T00:19:41.716 INFO:teuthology.orchestra.run.smithi045.stdout:[client.0] 2024-04-26T00:19:41.716 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQAH8ypmuzeCMxAALt8PqORZJo2a2PIsFgGIOA== 2024-04-26T00:19:41.717 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T00:19:41.754 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T00:19:41.806 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T00:19:41.809 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T00:19:41.851 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T00:19:41.893 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T00:19:41.935 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-26T00:19:41.978 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-26T00:19:42.024 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T00:19:42.062 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T00:19:42.113 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T00:19:42.146 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T00:19:42.187 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T00:19:42.230 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T00:19:42.279 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-26T00:19:42.314 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-26T00:19:42.361 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-26T00:19:42.361 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-26T00:19:42.373 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-26T00:19:42.463 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-26T00:19:42.473 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-26T00:19:42.526 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-26T00:19:42.612 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-26T00:19:42.624 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-26T00:19:42.639 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-26T00:19:42.733 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-26T00:19:42.744 DEBUG:teuthology.orchestra.run.smithi037:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-26T00:19:42.746 DEBUG:teuthology.orchestra.run.smithi045:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-26T00:19:42.793 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-26T00:19:42.793 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-26T00:19:42.794 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-26T00:19:42.796 INFO:tasks.ceph.mon.a:Started 2024-04-26T00:19:42.796 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-26T00:19:42.796 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-26T00:19:42.799 INFO:tasks.ceph.mon.c:Started 2024-04-26T00:19:42.799 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-26T00:19:42.799 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-26T00:19:42.838 INFO:tasks.ceph.mon.b:Started 2024-04-26T00:19:42.838 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-26T00:19:42.838 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-26T00:19:42.838 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-26T00:19:42.840 INFO:tasks.ceph.mgr.x:Started 2024-04-26T00:19:42.841 INFO:tasks.ceph:Setting crush tunables to default 2024-04-26T00:19:42.841 DEBUG:teuthology.orchestra.run.smithi037:> sudo ceph --cluster ceph osd crush tunables default 2024-04-26T00:19:42.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.984+0000 7f13866d5640 10 monclient: get_monmap_and_config 2024-04-26T00:19:42.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.984+0000 7f13866d5640 10 monclient: build_initial_monmap 2024-04-26T00:19:42.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.984+0000 7f13866d5640 10 monclient: monmap: 2024-04-26T00:19:42.965 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:19:42.965 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:42.965 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:42.988157+0000 2024-04-26T00:19:42.965 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:42.988157+0000 2024-04-26T00:19:42.965 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:19:42.965 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:19:42.965 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:19:42.965 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:19:42.965 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:19:42.965 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:19:42.965 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:19:42.965 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:19:42.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.984+0000 7f13866d5640 1 Processor -- start 2024-04-26T00:19:42.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.984+0000 7f13866d5640 1 -- start start 2024-04-26T00:19:42.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.984+0000 7f13866d5640 10 monclient: init 2024-04-26T00:19:42.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.984+0000 7f13866d5640 10 monclient: _reopen_session rank -1 2024-04-26T00:19:42.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.984+0000 7f13866d5640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-04-26T00:19:42.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.984+0000 7f13866d5640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13800a3d90 0x7f138009bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:42.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.984+0000 7f13866d5640 10 monclient(hunting): picked mon.noname-b con 0x7f13800a3d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:19:42.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.984+0000 7f13866d5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f13800a4170 addr v1:172.21.15.45:6789/0 2024-04-26T00:19:42.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.984+0000 7f13866d5640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f138013c3e0 0x7f138013e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:42.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.984+0000 7f13866d5640 10 monclient(hunting): picked mon.noname-a con 0x7f138013c3e0 addr v2:172.21.15.37:3300/0 2024-04-26T00:19:42.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.984+0000 7f13866d5640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13800a5150 con 0x7f13800a4170 2024-04-26T00:19:42.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.984+0000 7f13866d5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:42.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.984+0000 7f13866d5640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f138009b280 con 0x7f138013c3e0 2024-04-26T00:19:42.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.984+0000 7f13866d5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:42.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.984+0000 7f13866d5640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f138009b560 con 0x7f13800a3d90 2024-04-26T00:19:42.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.984+0000 7f13866d5640 10 monclient(hunting): _renew_subs 2024-04-26T00:19:42.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.984+0000 7f13866d5640 10 monclient(hunting): authenticate will time out at 964.293550s 2024-04-26T00:19:42.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.984+0000 7f137ffff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13800a3d90 0x7f138009bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:42.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.984+0000 7f137ffff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13800a3d90 0x7f138009bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:44580/0 (socket says 172.21.15.37:44580) 2024-04-26T00:19:42.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.984+0000 7f137ffff640 1 -- 172.21.15.37:0/2286232982 learned_addr learned my addr 172.21.15.37:0/2286232982 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:19:42.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.984+0000 7f137ffff640 10 monclient(hunting): get_auth_request con 0x7f13800a3d90 auth_method 0 2024-04-26T00:19:42.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.984+0000 7f137ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:42.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.984+0000 7f137ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:42.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.984+0000 7f137ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:42.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.984+0000 7f1384c72640 1 --2- 172.21.15.37:0/2286232982 >> v2:172.21.15.37:3300/0 conn(0x7f138013c3e0 0x7f138013e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:42.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.984+0000 7f1384c72640 10 monclient(hunting): get_auth_request con 0x7f138013c3e0 auth_method 0 2024-04-26T00:19:42.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.984+0000 7f1384c72640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:42.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.984+0000 7f1384c72640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:42.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.984+0000 7f1384c72640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:42.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.984+0000 7f137ffff640 1 -- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13800a3d90 msgr2=0x7f138009bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:19:42.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.984+0000 7f137ffff640 1 -- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13800a3d90 msgr2=0x7f138009bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:19:42.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.984+0000 7f137ffff640 1 --2- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13800a3d90 0x7f138009bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:19:42.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.984+0000 7f137ffff640 1 --2- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13800a3d90 0x7f138009bf40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:19:42.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.984+0000 7f1384c72640 1 -- 172.21.15.37:0/2286232982 >> v2:172.21.15.37:3300/0 conn(0x7f138013c3e0 msgr2=0x7f138013e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:19:42.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.984+0000 7f1384c72640 1 -- 172.21.15.37:0/2286232982 >> v2:172.21.15.37:3300/0 conn(0x7f138013c3e0 msgr2=0x7f138013e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:19:42.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.984+0000 7f1384c72640 1 --2- 172.21.15.37:0/2286232982 >> v2:172.21.15.37:3300/0 conn(0x7f138013c3e0 0x7f138013e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:19:42.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.984+0000 7f1384c72640 1 --2- 172.21.15.37:0/2286232982 >> v2:172.21.15.37:3300/0 conn(0x7f138013c3e0 0x7f138013e7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:19:42.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:42.988+0000 7f137f7fe640 1 -- 172.21.15.37:0/2286232982 >> v1:172.21.15.45:6789/0 conn(0x7f13800a4170 legacy=0x7f13800a45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:19:43.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:43.188+0000 7f137ffff640 1 --2- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13800a3d90 0x7f138009bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:43.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:43.188+0000 7f137ffff640 10 monclient(hunting): get_auth_request con 0x7f13800a3d90 auth_method 0 2024-04-26T00:19:43.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:43.188+0000 7f137ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:43.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:43.188+0000 7f137ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:43.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:43.188+0000 7f137ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:19:43.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:43.188+0000 7f1384c72640 1 --2- 172.21.15.37:0/2286232982 >> v2:172.21.15.37:3300/0 conn(0x7f138013c3e0 0x7f138013e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:43.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:43.188+0000 7f1384c72640 10 monclient(hunting): get_auth_request con 0x7f138013c3e0 auth_method 0 2024-04-26T00:19:43.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:43.188+0000 7f1384c72640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:43.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:43.188+0000 7f1384c72640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:43.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:43.188+0000 7f1384c72640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:19:43.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:43.188+0000 7f137ffff640 1 -- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13800a3d90 msgr2=0x7f138009bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:19:43.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:43.188+0000 7f137ffff640 1 -- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13800a3d90 msgr2=0x7f138009bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:19:43.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:43.188+0000 7f137ffff640 1 --2- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13800a3d90 0x7f138009bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:19:43.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:43.188+0000 7f137ffff640 1 --2- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13800a3d90 0x7f138009bf40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:19:43.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:43.188+0000 7f1384c72640 1 -- 172.21.15.37:0/2286232982 >> v2:172.21.15.37:3300/0 conn(0x7f138013c3e0 msgr2=0x7f138013e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:19:43.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:43.188+0000 7f1384c72640 1 -- 172.21.15.37:0/2286232982 >> v2:172.21.15.37:3300/0 conn(0x7f138013c3e0 msgr2=0x7f138013e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:19:43.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:43.188+0000 7f1384c72640 1 --2- 172.21.15.37:0/2286232982 >> v2:172.21.15.37:3300/0 conn(0x7f138013c3e0 0x7f138013e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:19:43.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:43.188+0000 7f1384c72640 1 --2- 172.21.15.37:0/2286232982 >> v2:172.21.15.37:3300/0 conn(0x7f138013c3e0 0x7f138013e7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:19:43.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:43.592+0000 7f137ffff640 1 --2- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13800a3d90 0x7f138009bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:43.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:43.592+0000 7f1384c72640 1 --2- 172.21.15.37:0/2286232982 >> v2:172.21.15.37:3300/0 conn(0x7f138013c3e0 0x7f138013e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:43.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:43.592+0000 7f137ffff640 10 monclient(hunting): get_auth_request con 0x7f13800a3d90 auth_method 0 2024-04-26T00:19:43.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:43.592+0000 7f137ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:43.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:43.592+0000 7f137ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:43.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:43.592+0000 7f137ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:19:43.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:43.592+0000 7f1384c72640 10 monclient(hunting): get_auth_request con 0x7f138013c3e0 auth_method 0 2024-04-26T00:19:43.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:43.592+0000 7f1384c72640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:43.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:43.592+0000 7f1384c72640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:43.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:43.592+0000 7f1384c72640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:19:43.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:43.592+0000 7f1384c72640 1 -- 172.21.15.37:0/2286232982 >> v2:172.21.15.37:3300/0 conn(0x7f138013c3e0 msgr2=0x7f138013e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:19:43.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:43.592+0000 7f1384c72640 1 -- 172.21.15.37:0/2286232982 >> v2:172.21.15.37:3300/0 conn(0x7f138013c3e0 msgr2=0x7f138013e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:19:43.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:43.592+0000 7f1384c72640 1 --2- 172.21.15.37:0/2286232982 >> v2:172.21.15.37:3300/0 conn(0x7f138013c3e0 0x7f138013e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:19:43.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:43.592+0000 7f137ffff640 1 -- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13800a3d90 msgr2=0x7f138009bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:19:43.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:43.592+0000 7f137ffff640 1 -- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13800a3d90 msgr2=0x7f138009bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:19:43.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:43.592+0000 7f137ffff640 1 --2- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13800a3d90 0x7f138009bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:19:43.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:43.592+0000 7f1384c72640 1 --2- 172.21.15.37:0/2286232982 >> v2:172.21.15.37:3300/0 conn(0x7f138013c3e0 0x7f138013e7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:19:43.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:43.592+0000 7f137ffff640 1 --2- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13800a3d90 0x7f138009bf40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:19:44.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:44.396+0000 7f137ffff640 1 --2- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13800a3d90 0x7f138009bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:44.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:44.396+0000 7f1384c72640 1 --2- 172.21.15.37:0/2286232982 >> v2:172.21.15.37:3300/0 conn(0x7f138013c3e0 0x7f138013e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:44.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:44.396+0000 7f137ffff640 10 monclient(hunting): get_auth_request con 0x7f13800a3d90 auth_method 0 2024-04-26T00:19:44.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:44.396+0000 7f137ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:44.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:44.396+0000 7f137ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:44.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:44.396+0000 7f137ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:19:44.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:44.396+0000 7f1384c72640 10 monclient(hunting): get_auth_request con 0x7f138013c3e0 auth_method 0 2024-04-26T00:19:44.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:44.396+0000 7f1384c72640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:44.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:44.396+0000 7f1384c72640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:44.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:44.396+0000 7f1384c72640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:19:44.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:44.396+0000 7f137ffff640 1 -- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13800a3d90 msgr2=0x7f138009bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:19:44.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:44.396+0000 7f137ffff640 1 -- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13800a3d90 msgr2=0x7f138009bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:19:44.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:44.396+0000 7f137ffff640 1 --2- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13800a3d90 0x7f138009bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:19:44.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:44.396+0000 7f1384c72640 1 -- 172.21.15.37:0/2286232982 >> v2:172.21.15.37:3300/0 conn(0x7f138013c3e0 msgr2=0x7f138013e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:19:44.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:44.396+0000 7f1384c72640 1 -- 172.21.15.37:0/2286232982 >> v2:172.21.15.37:3300/0 conn(0x7f138013c3e0 msgr2=0x7f138013e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:19:44.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:44.396+0000 7f1384c72640 1 --2- 172.21.15.37:0/2286232982 >> v2:172.21.15.37:3300/0 conn(0x7f138013c3e0 0x7f138013e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:19:44.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:44.396+0000 7f1384c72640 1 --2- 172.21.15.37:0/2286232982 >> v2:172.21.15.37:3300/0 conn(0x7f138013c3e0 0x7f138013e7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:19:44.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:44.396+0000 7f137ffff640 1 --2- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13800a3d90 0x7f138009bf40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:19:45.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:45.984+0000 7f137dffb640 10 monclient(hunting): tick 2024-04-26T00:19:45.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:45.984+0000 7f137dffb640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:19:45.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:45.984+0000 7f137dffb640 1 monclient(hunting): continuing hunt 2024-04-26T00:19:45.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:45.984+0000 7f137dffb640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:19:45.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:45.984+0000 7f137dffb640 1 -- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13800a3d90 msgr2=0x7f138009bf40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:19:45.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:45.984+0000 7f137dffb640 1 --2- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13800a3d90 0x7f138009bf40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:45.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:45.984+0000 7f137dffb640 1 -- 172.21.15.37:0/2286232982 >> v2:172.21.15.37:3300/0 conn(0x7f138013c3e0 msgr2=0x7f138013e7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:19:45.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:45.984+0000 7f137dffb640 1 --2- 172.21.15.37:0/2286232982 >> v2:172.21.15.37:3300/0 conn(0x7f138013c3e0 0x7f138013e7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:45.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:45.984+0000 7f137dffb640 1 -- 172.21.15.37:0/2286232982 >> v1:172.21.15.45:6789/0 conn(0x7f13800a4170 legacy=0x7f13800a45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:45.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:45.984+0000 7f137dffb640 10 monclient: _add_conns ranks=[0,4] 2024-04-26T00:19:45.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:45.984+0000 7f137dffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f136c003730 addr v1:172.21.15.37:6789/0 2024-04-26T00:19:45.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:45.984+0000 7f137dffb640 1 --2- 172.21.15.37:0/2286232982 >> v2:172.21.15.45:3300/0 conn(0x7f136c007480 0x7f136c009940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:45.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:45.984+0000 7f137dffb640 10 monclient(hunting): picked mon.noname-c con 0x7f136c007480 addr v2:172.21.15.45:3300/0 2024-04-26T00:19:45.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:45.984+0000 7f137dffb640 1 -- 172.21.15.37:0/2286232982 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f138006dc60 con 0x7f136c003730 2024-04-26T00:19:45.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:45.984+0000 7f137dffb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:45.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:45.984+0000 7f137dffb640 1 -- 172.21.15.37:0/2286232982 --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f138009b280 con 0x7f136c007480 2024-04-26T00:19:45.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:45.984+0000 7f137dffb640 10 monclient(hunting): _renew_subs 2024-04-26T00:19:45.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:45.984+0000 7f137f7fe640 1 --2- 172.21.15.37:0/2286232982 >> v2:172.21.15.45:3300/0 conn(0x7f136c007480 0x7f136c009940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:45.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:45.984+0000 7f137f7fe640 10 monclient(hunting): get_auth_request con 0x7f136c007480 auth_method 0 2024-04-26T00:19:45.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:45.984+0000 7f137f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:45.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:45.984+0000 7f137f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:45.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:45.984+0000 7f137f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:45.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:45.984+0000 7f137f7fe640 1 -- 172.21.15.37:0/2286232982 >> v2:172.21.15.45:3300/0 conn(0x7f136c007480 msgr2=0x7f136c009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:19:45.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:45.984+0000 7f137f7fe640 1 -- 172.21.15.37:0/2286232982 >> v2:172.21.15.45:3300/0 conn(0x7f136c007480 msgr2=0x7f136c009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:19:45.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:45.984+0000 7f137f7fe640 1 --2- 172.21.15.37:0/2286232982 >> v2:172.21.15.45:3300/0 conn(0x7f136c007480 0x7f136c009940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:19:45.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:45.984+0000 7f137f7fe640 1 --2- 172.21.15.37:0/2286232982 >> v2:172.21.15.45:3300/0 conn(0x7f136c007480 0x7f136c009940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:19:45.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:45.988+0000 7f137ffff640 1 -- 172.21.15.37:0/2286232982 >> v1:172.21.15.37:6789/0 conn(0x7f136c003730 legacy=0x7f136c005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:19:46.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:46.188+0000 7f137f7fe640 1 --2- 172.21.15.37:0/2286232982 >> v2:172.21.15.45:3300/0 conn(0x7f136c007480 0x7f136c009940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:46.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:46.188+0000 7f137f7fe640 10 monclient(hunting): get_auth_request con 0x7f136c007480 auth_method 0 2024-04-26T00:19:46.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:46.188+0000 7f137f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:46.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:46.188+0000 7f137f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:46.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:46.188+0000 7f137f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:19:46.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:46.188+0000 7f137f7fe640 1 -- 172.21.15.37:0/2286232982 >> v2:172.21.15.45:3300/0 conn(0x7f136c007480 msgr2=0x7f136c009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:19:46.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:46.188+0000 7f137f7fe640 1 -- 172.21.15.37:0/2286232982 >> v2:172.21.15.45:3300/0 conn(0x7f136c007480 msgr2=0x7f136c009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:19:46.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:46.188+0000 7f137f7fe640 1 --2- 172.21.15.37:0/2286232982 >> v2:172.21.15.45:3300/0 conn(0x7f136c007480 0x7f136c009940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:19:46.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:46.188+0000 7f137f7fe640 1 --2- 172.21.15.37:0/2286232982 >> v2:172.21.15.45:3300/0 conn(0x7f136c007480 0x7f136c009940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:19:46.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:46.592+0000 7f137f7fe640 1 --2- 172.21.15.37:0/2286232982 >> v2:172.21.15.45:3300/0 conn(0x7f136c007480 0x7f136c009940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:46.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:46.592+0000 7f137f7fe640 10 monclient(hunting): get_auth_request con 0x7f136c007480 auth_method 0 2024-04-26T00:19:46.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:46.592+0000 7f137f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:46.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:46.592+0000 7f137f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:46.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:46.592+0000 7f137f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:19:46.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:46.592+0000 7f137f7fe640 1 -- 172.21.15.37:0/2286232982 >> v2:172.21.15.45:3300/0 conn(0x7f136c007480 msgr2=0x7f136c009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:19:46.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:46.592+0000 7f137f7fe640 1 -- 172.21.15.37:0/2286232982 >> v2:172.21.15.45:3300/0 conn(0x7f136c007480 msgr2=0x7f136c009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:19:46.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:46.592+0000 7f137f7fe640 1 --2- 172.21.15.37:0/2286232982 >> v2:172.21.15.45:3300/0 conn(0x7f136c007480 0x7f136c009940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:19:46.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:46.592+0000 7f137f7fe640 1 --2- 172.21.15.37:0/2286232982 >> v2:172.21.15.45:3300/0 conn(0x7f136c007480 0x7f136c009940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:19:47.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:47.396+0000 7f137f7fe640 1 --2- 172.21.15.37:0/2286232982 >> v2:172.21.15.45:3300/0 conn(0x7f136c007480 0x7f136c009940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:47.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:47.396+0000 7f137f7fe640 10 monclient(hunting): get_auth_request con 0x7f136c007480 auth_method 0 2024-04-26T00:19:47.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:47.396+0000 7f137f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:47.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:47.396+0000 7f137f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:47.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:47.396+0000 7f137f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:19:47.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:47.396+0000 7f137f7fe640 1 -- 172.21.15.37:0/2286232982 >> v2:172.21.15.45:3300/0 conn(0x7f136c007480 msgr2=0x7f136c009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:19:47.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:47.396+0000 7f137f7fe640 1 -- 172.21.15.37:0/2286232982 >> v2:172.21.15.45:3300/0 conn(0x7f136c007480 msgr2=0x7f136c009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:19:47.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:47.396+0000 7f137f7fe640 1 --2- 172.21.15.37:0/2286232982 >> v2:172.21.15.45:3300/0 conn(0x7f136c007480 0x7f136c009940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:19:47.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:47.396+0000 7f137f7fe640 1 --2- 172.21.15.37:0/2286232982 >> v2:172.21.15.45:3300/0 conn(0x7f136c007480 0x7f136c009940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:19:48.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:48.984+0000 7f137dffb640 10 monclient(hunting): tick 2024-04-26T00:19:48.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:48.984+0000 7f137dffb640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:19:48.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:48.984+0000 7f137dffb640 1 monclient(hunting): continuing hunt 2024-04-26T00:19:48.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:48.984+0000 7f137dffb640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:19:48.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:48.984+0000 7f137dffb640 1 -- 172.21.15.37:0/2286232982 >> v2:172.21.15.45:3300/0 conn(0x7f136c007480 msgr2=0x7f136c009940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:19:48.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:48.984+0000 7f137dffb640 1 --2- 172.21.15.37:0/2286232982 >> v2:172.21.15.45:3300/0 conn(0x7f136c007480 0x7f136c009940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:48.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:48.984+0000 7f137dffb640 1 -- 172.21.15.37:0/2286232982 >> v1:172.21.15.37:6789/0 conn(0x7f136c003730 legacy=0x7f136c005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:48.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:48.984+0000 7f1384c72640 1 -- 172.21.15.37:0/2286232982 reap_dead start 2024-04-26T00:19:48.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:48.984+0000 7f137dffb640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-04-26T00:19:48.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:48.984+0000 7f137dffb640 1 --2- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f136c00a190 0x7f136c009940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:48.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:48.984+0000 7f137dffb640 10 monclient(hunting): picked mon.noname-b con 0x7f136c00a190 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:19:48.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:48.984+0000 7f137dffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f136c00a6b0 addr v1:172.21.15.37:6789/0 2024-04-26T00:19:48.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:48.984+0000 7f137dffb640 1 --2- 172.21.15.37:0/2286232982 >> v2:172.21.15.37:3300/0 conn(0x7f136c00ac00 0x7f136c005c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:48.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:48.984+0000 7f137dffb640 10 monclient(hunting): picked mon.noname-a con 0x7f136c00ac00 addr v2:172.21.15.37:3300/0 2024-04-26T00:19:48.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:48.984+0000 7f137dffb640 1 -- 172.21.15.37:0/2286232982 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f136c00b050 con 0x7f136c00a6b0 2024-04-26T00:19:48.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:48.984+0000 7f137dffb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:48.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:48.984+0000 7f137dffb640 1 -- 172.21.15.37:0/2286232982 --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f138009b280 con 0x7f136c00ac00 2024-04-26T00:19:48.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:48.984+0000 7f137dffb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:48.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:48.984+0000 7f137dffb640 1 -- 172.21.15.37:0/2286232982 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f138009b560 con 0x7f136c00a190 2024-04-26T00:19:48.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:48.984+0000 7f137dffb640 10 monclient(hunting): _renew_subs 2024-04-26T00:19:48.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:48.984+0000 7f137ffff640 1 --2- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f136c00a190 0x7f136c009940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:48.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:48.984+0000 7f1384c72640 1 --2- 172.21.15.37:0/2286232982 >> v2:172.21.15.37:3300/0 conn(0x7f136c00ac00 0x7f136c005c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:48.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:48.984+0000 7f137ffff640 10 monclient(hunting): get_auth_request con 0x7f136c00a190 auth_method 0 2024-04-26T00:19:48.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:48.984+0000 7f137ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:48.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:48.984+0000 7f137ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:48.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:48.984+0000 7f137ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:48.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:48.984+0000 7f1384c72640 10 monclient(hunting): get_auth_request con 0x7f136c00ac00 auth_method 0 2024-04-26T00:19:48.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:48.984+0000 7f1384c72640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:48.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:48.984+0000 7f1384c72640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:48.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:48.984+0000 7f1384c72640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:48.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:48.984+0000 7f137ffff640 1 -- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f136c00a190 msgr2=0x7f136c009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:19:48.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:48.984+0000 7f137ffff640 1 -- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f136c00a190 msgr2=0x7f136c009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:19:48.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:48.984+0000 7f137ffff640 1 --2- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f136c00a190 0x7f136c009940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:19:48.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:48.984+0000 7f1384c72640 1 -- 172.21.15.37:0/2286232982 >> v2:172.21.15.37:3300/0 conn(0x7f136c00ac00 msgr2=0x7f136c005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:19:48.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:48.984+0000 7f1384c72640 1 -- 172.21.15.37:0/2286232982 >> v2:172.21.15.37:3300/0 conn(0x7f136c00ac00 msgr2=0x7f136c005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:19:48.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:48.984+0000 7f1384c72640 1 --2- 172.21.15.37:0/2286232982 >> v2:172.21.15.37:3300/0 conn(0x7f136c00ac00 0x7f136c005c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:19:48.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:48.984+0000 7f1384c72640 1 --2- 172.21.15.37:0/2286232982 >> v2:172.21.15.37:3300/0 conn(0x7f136c00ac00 0x7f136c005c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:19:48.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:48.984+0000 7f137ffff640 1 --2- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f136c00a190 0x7f136c009940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:19:48.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:48.988+0000 7f137f7fe640 1 -- 172.21.15.37:0/2286232982 >> v1:172.21.15.37:6789/0 conn(0x7f136c00a6b0 legacy=0x7f136c00d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:19:49.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:49.188+0000 7f137ffff640 1 --2- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f136c00a190 0x7f136c009940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:49.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:49.188+0000 7f1384c72640 1 --2- 172.21.15.37:0/2286232982 >> v2:172.21.15.37:3300/0 conn(0x7f136c00ac00 0x7f136c005c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:49.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:49.188+0000 7f137ffff640 10 monclient(hunting): get_auth_request con 0x7f136c00a190 auth_method 0 2024-04-26T00:19:49.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:49.188+0000 7f137ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:49.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:49.188+0000 7f137ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:49.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:49.188+0000 7f137ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:19:49.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:49.188+0000 7f1384c72640 10 monclient(hunting): get_auth_request con 0x7f136c00ac00 auth_method 0 2024-04-26T00:19:49.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:49.188+0000 7f1384c72640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:49.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:49.188+0000 7f1384c72640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:49.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:49.188+0000 7f1384c72640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:19:49.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:49.188+0000 7f137ffff640 1 -- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f136c00a190 msgr2=0x7f136c009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:19:49.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:49.188+0000 7f1384c72640 1 -- 172.21.15.37:0/2286232982 >> v2:172.21.15.37:3300/0 conn(0x7f136c00ac00 msgr2=0x7f136c005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:19:49.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:49.188+0000 7f1384c72640 1 -- 172.21.15.37:0/2286232982 >> v2:172.21.15.37:3300/0 conn(0x7f136c00ac00 msgr2=0x7f136c005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:19:49.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:49.188+0000 7f137ffff640 1 -- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f136c00a190 msgr2=0x7f136c009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:19:49.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:49.188+0000 7f1384c72640 1 --2- 172.21.15.37:0/2286232982 >> v2:172.21.15.37:3300/0 conn(0x7f136c00ac00 0x7f136c005c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:19:49.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:49.188+0000 7f137ffff640 1 --2- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f136c00a190 0x7f136c009940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:19:49.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:49.188+0000 7f1384c72640 1 --2- 172.21.15.37:0/2286232982 >> v2:172.21.15.37:3300/0 conn(0x7f136c00ac00 0x7f136c005c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:19:49.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:49.188+0000 7f137ffff640 1 --2- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f136c00a190 0x7f136c009940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:19:49.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:49.592+0000 7f137ffff640 1 --2- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f136c00a190 0x7f136c009940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:49.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:49.592+0000 7f1384c72640 1 --2- 172.21.15.37:0/2286232982 >> v2:172.21.15.37:3300/0 conn(0x7f136c00ac00 0x7f136c005c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:49.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:49.592+0000 7f137ffff640 10 monclient(hunting): get_auth_request con 0x7f136c00a190 auth_method 0 2024-04-26T00:19:49.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:49.592+0000 7f137ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:49.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:49.592+0000 7f137ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:49.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:49.592+0000 7f137ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:19:49.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:49.592+0000 7f1384c72640 10 monclient(hunting): get_auth_request con 0x7f136c00ac00 auth_method 0 2024-04-26T00:19:49.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:49.592+0000 7f1384c72640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:49.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:49.592+0000 7f1384c72640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:49.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:49.592+0000 7f1384c72640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:19:49.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:49.592+0000 7f137ffff640 1 -- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f136c00a190 msgr2=0x7f136c009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:19:49.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:49.592+0000 7f137ffff640 1 -- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f136c00a190 msgr2=0x7f136c009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:19:49.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:49.592+0000 7f137ffff640 1 --2- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f136c00a190 0x7f136c009940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:19:49.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:49.592+0000 7f1384c72640 1 -- 172.21.15.37:0/2286232982 >> v2:172.21.15.37:3300/0 conn(0x7f136c00ac00 msgr2=0x7f136c005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:19:49.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:49.592+0000 7f1384c72640 1 -- 172.21.15.37:0/2286232982 >> v2:172.21.15.37:3300/0 conn(0x7f136c00ac00 msgr2=0x7f136c005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:19:49.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:49.592+0000 7f1384c72640 1 --2- 172.21.15.37:0/2286232982 >> v2:172.21.15.37:3300/0 conn(0x7f136c00ac00 0x7f136c005c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:19:49.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:49.592+0000 7f137ffff640 1 --2- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f136c00a190 0x7f136c009940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:19:49.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:49.592+0000 7f1384c72640 1 --2- 172.21.15.37:0/2286232982 >> v2:172.21.15.37:3300/0 conn(0x7f136c00ac00 0x7f136c005c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:19:50.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 1 -- 172.21.15.37:0/2286232982 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3281381714 0 0) 0x7f136c00b050 con 0x7f136c00a6b0 2024-04-26T00:19:50.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 10 monclient(hunting): _negotiate 2024-04-26T00:19:50.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:50.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:50.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 10 monclient(hunting): my global_id is 4098 2024-04-26T00:19:50.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 1 -- 172.21.15.37:0/2286232982 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1368003530 con 0x7f136c00a6b0 2024-04-26T00:19:50.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 1 -- 172.21.15.37:0/2286232982 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3481662127 0 0) 0x7f1368003530 con 0x7f136c00a6b0 2024-04-26T00:19:50.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 1 -- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f136c00a190 msgr2=0x7f136c009940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:19:50.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 1 --2- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f136c00a190 0x7f136c009940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:50.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 1 -- 172.21.15.37:0/2286232982 >> v2:172.21.15.37:3300/0 conn(0x7f136c00ac00 msgr2=0x7f136c005c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:19:50.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 1 --2- 172.21.15.37:0/2286232982 >> v2:172.21.15.37:3300/0 conn(0x7f136c00ac00 0x7f136c005c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:50.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 10 monclient: _finish_hunting 0 2024-04-26T00:19:50.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 1 monclient: found mon.noname-a-legacy 2024-04-26T00:19:50.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:19:50.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.37:6789/0 2024-04-26T00:19:50.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 1 -- 172.21.15.37:0/2286232982 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f136c00ebd0 con 0x7f136c00a6b0 2024-04-26T00:19:50.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 10 monclient: _finish_auth 0 2024-04-26T00:19:50.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 10 monclient: _check_auth_tickets 2024-04-26T00:19:50.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:19:50.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 1 -- 172.21.15.37:0/2286232982 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 866326386 0 0) 0x7f1374007c30 con 0x7f136c00a6b0 2024-04-26T00:19:50.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f13866d5640 5 monclient: authenticate success, global_id 4098 2024-04-26T00:19:50.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f13866d5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:19:50.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:50.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-26T00:19:50.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 10 monclient: dump: 2024-04-26T00:19:50.044 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:19:50.045 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:50.045 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:50.045 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:50.045 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:50.045 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:19:50.045 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:50.045 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:50.045 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:50.045 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:19:50.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] but i'm connected to v1:172.21.15.37:6789/0, reconnecting 2024-04-26T00:19:50.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 10 monclient: _reopen_session rank -1 2024-04-26T00:19:50.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 1 -- 172.21.15.37:0/2286232982 >> v1:172.21.15.37:6789/0 conn(0x7f136c00a6b0 legacy=0x7f136c00d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:50.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:19:50.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 1 --2- 172.21.15.37:0/2286232982 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13680039c0 0x7f1368003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:50.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 10 monclient(hunting): picked mon.b con 0x7f13680039c0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:19:50.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 1 --2- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1368004810 0x7f1368004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:50.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 10 monclient(hunting): picked mon.c con 0x7f1368004810 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:19:50.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 1 --2- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f13680051d0 0x7f136800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:50.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 10 monclient(hunting): picked mon.a con 0x7f13680051d0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:19:50.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:50.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 1 -- 172.21.15.37:0/2286232982 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f138009b280 con 0x7f13680051d0 2024-04-26T00:19:50.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:50.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 1 -- 172.21.15.37:0/2286232982 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f138009b560 con 0x7f13680039c0 2024-04-26T00:19:50.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:50.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 1 -- 172.21.15.37:0/2286232982 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f136800c0f0 con 0x7f1368004810 2024-04-26T00:19:50.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 10 monclient(hunting): _renew_subs 2024-04-26T00:19:50.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137effd640 10 monclient(hunting): _finish_auth 0 2024-04-26T00:19:50.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137f7fe640 1 --2- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1368004810 0x7f1368004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:50.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f13866d5640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-26T00:19:50.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137f7fe640 10 monclient(hunting): get_auth_request con 0x7f1368004810 auth_method 0 2024-04-26T00:19:50.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:50.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:50.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:19:50.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f1384c72640 1 --2- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f13680051d0 0x7f136800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:50.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f1384c72640 10 monclient(hunting): get_auth_request con 0x7f13680051d0 auth_method 0 2024-04-26T00:19:50.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f1384c72640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:50.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f1384c72640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:50.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f1384c72640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:19:50.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f1384c72640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:19:50.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f1384c72640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:19:50.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f1384c72640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:19:50.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:19:50.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:19:50.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:19:50.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f1384c72640 10 monclient(hunting): handle_auth_done global_id 4098 payload 293 2024-04-26T00:19:50.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f1384c72640 1 -- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1368004810 msgr2=0x7f1368004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:19:50.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f1384c72640 1 --2- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1368004810 0x7f1368004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:50.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137ffff640 1 --2- 172.21.15.37:0/2286232982 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13680039c0 0x7f1368003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:50.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f1384c72640 1 -- 172.21.15.37:0/2286232982 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13680039c0 msgr2=0x7f1368003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:19:50.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f1384c72640 1 --2- 172.21.15.37:0/2286232982 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13680039c0 0x7f1368003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:50.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f1384c72640 10 monclient: _finish_hunting 0 2024-04-26T00:19:50.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f1384c72640 1 monclient: found mon.a 2024-04-26T00:19:50.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f1384c72640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:19:50.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f1384c72640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:19:50.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f1384c72640 1 -- 172.21.15.37:0/2286232982 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1368003530 con 0x7f13680051d0 2024-04-26T00:19:50.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f1384c72640 10 monclient: _finish_auth 0 2024-04-26T00:19:50.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f1384c72640 10 monclient: _check_auth_tickets 2024-04-26T00:19:50.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f1384c72640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:19:50.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137ffff640 10 monclient: get_auth_request con 0x7f13680039c0 auth_method 0 2024-04-26T00:19:50.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.060+0000 7f137ffff640 1 --2- 172.21.15.37:0/2286232982 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13680039c0 0x7f1368003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:19:50.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.064+0000 7f1384c72640 1 -- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f13680051d0 msgr2=0x7f136800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:19:50.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.064+0000 7f1384c72640 1 --2- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f13680051d0 0x7f136800b9d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f13800a5420 tx=0x7f137801e280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dca0d1c411f2903 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:19:50.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.064+0000 7f137effd640 1 -- 172.21.15.37:0/2286232982 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13780b4070 con 0x7f13680051d0 2024-04-26T00:19:50.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.064+0000 7f137effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:50.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.068+0000 7f137effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:19:50.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.068+0000 7f137effd640 10 monclient: dump: 2024-04-26T00:19:50.052 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:19:50.052 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:50.052 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:50.052 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:50.052 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:50.052 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:19:50.052 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:50.052 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:50.052 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:50.052 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:19:50.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.068+0000 7f137effd640 1 -- 172.21.15.37:0/2286232982 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13780af410 con 0x7f13680051d0 2024-04-26T00:19:50.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.068+0000 7f137effd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:19:50.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.068+0000 7f137effd640 1 -- 172.21.15.37:0/2286232982 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1378009a40 con 0x7f13680051d0 2024-04-26T00:19:50.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.068+0000 7f137effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:50.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.068+0000 7f137effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:19:50.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.068+0000 7f137effd640 10 monclient: dump: 2024-04-26T00:19:50.053 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:19:50.053 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:50.053 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:50.053 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:50.053 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:50.053 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:19:50.053 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:50.053 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:50.053 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:50.053 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:19:50.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.068+0000 7f13866d5640 10 monclient: get_monmap_and_config success 2024-04-26T00:19:50.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.068+0000 7f13866d5640 4 set_mon_vals no callback set 2024-04-26T00:19:50.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.068+0000 7f13866d5640 10 monclient: shutdown 2024-04-26T00:19:50.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.068+0000 7f13866d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:19:50.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.068+0000 7f13866d5640 1 -- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f13680051d0 msgr2=0x7f136800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:50.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.068+0000 7f13866d5640 1 --2- 172.21.15.37:0/2286232982 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f13680051d0 0x7f136800b9d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f13800a5420 tx=0x7f137801e280 comp rx=0 tx=0).stop 2024-04-26T00:19:50.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.068+0000 7f1384c72640 1 -- 172.21.15.37:0/2286232982 reap_dead start 2024-04-26T00:19:50.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.068+0000 7f13866d5640 1 -- 172.21.15.37:0/2286232982 shutdown_connections 2024-04-26T00:19:50.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.068+0000 7f13866d5640 1 -- 172.21.15.37:0/2286232982 >> 172.21.15.37:0/2286232982 conn(0x7f13800aa0a0 msgr2=0x7f13800a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:19:50.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.068+0000 7f13866d5640 1 -- 172.21.15.37:0/2286232982 shutdown_connections 2024-04-26T00:19:50.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.068+0000 7f13866d5640 1 -- 172.21.15.37:0/2286232982 wait complete. 2024-04-26T00:19:50.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.068+0000 7f13866d5640 10 monclient: build_initial_monmap 2024-04-26T00:19:50.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.068+0000 7f13866d5640 10 monclient: monmap: 2024-04-26T00:19:50.054 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:19:50.054 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:19:50.055 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:19:50.055 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:19:50.055 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:19:50.055 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:19:50.055 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:19:50.055 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:19:50.055 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:19:50.055 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:19:50.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.068+0000 7f13866d5640 1 Processor -- start 2024-04-26T00:19:50.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.068+0000 7f13866d5640 1 -- start start 2024-04-26T00:19:50.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.068+0000 7f13866d5640 10 monclient: init 2024-04-26T00:19:50.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f13866d5640 10 monclient: _reopen_session rank -1 2024-04-26T00:19:50.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f13866d5640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:19:50.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f13866d5640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f13801cf810 0x7f13801cfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:50.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f13866d5640 10 monclient(hunting): picked mon.noname-a con 0x7f13801cf810 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:19:50.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f13866d5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13801d0130 0x7f13801363d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:50.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f13866d5640 10 monclient(hunting): picked mon.noname-b con 0x7f13801d0130 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:19:50.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f13866d5640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13801223f0 0x7f1380136910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:50.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f13866d5640 10 monclient(hunting): picked mon.noname-c con 0x7f13801223f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:19:50.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f13866d5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:50.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f13866d5640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f1380133ca0 con 0x7f13801cf810 2024-04-26T00:19:50.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f13866d5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:50.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f13866d5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1380133b20 con 0x7f13801d0130 2024-04-26T00:19:50.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f13866d5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:50.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f13866d5640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f1380133e20 con 0x7f13801223f0 2024-04-26T00:19:50.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f13866d5640 10 monclient(hunting): _renew_subs 2024-04-26T00:19:50.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f13866d5640 10 monclient(hunting): authenticate will time out at 971.380778s 2024-04-26T00:19:50.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f137ffff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f13801cf810 0x7f13801cfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:50.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f137ffff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f13801cf810 0x7f13801cfbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:48980/0 (socket says 172.21.15.37:48980) 2024-04-26T00:19:50.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f1384c72640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13801223f0 0x7f1380136910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:50.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f137ffff640 1 -- 172.21.15.37:0/1428856293 learned_addr learned my addr 172.21.15.37:0/1428856293 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:19:50.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f137ffff640 10 monclient(hunting): get_auth_request con 0x7f13801cf810 auth_method 0 2024-04-26T00:19:50.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f137ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:50.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f137ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:50.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f137ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:50.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f1384c72640 10 monclient(hunting): get_auth_request con 0x7f13801223f0 auth_method 0 2024-04-26T00:19:50.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f1384c72640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:50.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f1384c72640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:50.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f1384c72640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:50.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f137f7fe640 1 --2- 172.21.15.37:0/1428856293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13801d0130 0x7f13801363d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:50.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f137ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:19:50.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f137ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:19:50.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f137ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:19:50.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f137f7fe640 10 monclient(hunting): get_auth_request con 0x7f13801d0130 auth_method 0 2024-04-26T00:19:50.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f137f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:50.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f137f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:50.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f137f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:50.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f1384c72640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:19:50.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f1384c72640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:19:50.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f1384c72640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:19:50.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f137f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:19:50.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f137f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:19:50.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f137f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:19:50.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f1384c72640 10 monclient(hunting): handle_auth_done global_id 4100 payload 824 2024-04-26T00:19:50.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f1384c72640 1 -- 172.21.15.37:0/1428856293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13801d0130 msgr2=0x7f13801363d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:50.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f1384c72640 1 --2- 172.21.15.37:0/1428856293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13801d0130 0x7f13801363d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:50.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f1384c72640 1 -- 172.21.15.37:0/1428856293 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f13801cf810 msgr2=0x7f13801cfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:50.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f1384c72640 1 --2- 172.21.15.37:0/1428856293 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f13801cf810 0x7f13801cfbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:50.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f1384c72640 10 monclient: _finish_hunting 0 2024-04-26T00:19:50.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f1384c72640 1 monclient: found mon.noname-c 2024-04-26T00:19:50.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f1384c72640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:19:50.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f1384c72640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:19:50.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f1384c72640 1 -- 172.21.15.37:0/1428856293 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1380120180 con 0x7f13801223f0 2024-04-26T00:19:50.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f1384c72640 1 -- 172.21.15.37:0/1428856293 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13801223f0 msgr2=0x7f1380136910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:19:50.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f1384c72640 1 --2- 172.21.15.37:0/1428856293 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13801223f0 0x7f1380136910 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f13780b1000 tx=0x7f1368005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:19:50.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.072+0000 7f137cff9640 1 -- 172.21.15.37:0/1428856293 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13780b4040 con 0x7f13801223f0 2024-04-26T00:19:50.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.076+0000 7f137cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:50.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.076+0000 7f137cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:19:50.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.076+0000 7f137cff9640 10 monclient: dump: 2024-04-26T00:19:50.062 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:19:50.062 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:50.062 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:50.062 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:50.063 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:50.063 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:19:50.063 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:50.063 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:50.063 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:50.063 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:19:50.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.076+0000 7f137cff9640 10 monclient: _finish_auth 0 2024-04-26T00:19:50.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.076+0000 7f137cff9640 10 monclient: _check_auth_tickets 2024-04-26T00:19:50.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.076+0000 7f137cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:19:50.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.076+0000 7f13866d5640 5 monclient: authenticate success, global_id 4100 2024-04-26T00:19:50.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.076+0000 7f137cff9640 1 -- 172.21.15.37:0/1428856293 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13780c1070 con 0x7f13801223f0 2024-04-26T00:19:50.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.076+0000 7f137cff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:19:50.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.076+0000 7f13866d5640 10 monclient: _renew_subs 2024-04-26T00:19:50.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.076+0000 7f137cff9640 1 -- 172.21.15.37:0/1428856293 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1378009a40 con 0x7f13801223f0 2024-04-26T00:19:50.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.076+0000 7f13866d5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:19:50.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.076+0000 7f13866d5640 1 -- 172.21.15.37:0/1428856293 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f138009ada0 con 0x7f13801223f0 2024-04-26T00:19:50.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.076+0000 7f137cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:50.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.076+0000 7f137cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:19:50.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.076+0000 7f137cff9640 10 monclient: dump: 2024-04-26T00:19:50.064 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:19:50.064 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:50.064 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:50.064 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:50.064 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:50.064 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:19:50.064 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:50.064 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:50.064 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:50.065 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:19:50.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.076+0000 7f137e7fc640 4 set_mon_vals no callback set 2024-04-26T00:19:50.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.076+0000 7f13866d5640 10 monclient: _renew_subs 2024-04-26T00:19:50.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.076+0000 7f13866d5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:19:50.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.076+0000 7f13866d5640 1 -- 172.21.15.37:0/1428856293 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1380121cf0 con 0x7f13801223f0 2024-04-26T00:19:50.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.076+0000 7f137cff9640 1 -- 172.21.15.37:0/1428856293 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f13780b7070 con 0x7f13801223f0 2024-04-26T00:19:50.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.076+0000 7f137cff9640 1 -- 172.21.15.37:0/1428856293 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f13780b9c80 con 0x7f13801223f0 2024-04-26T00:19:50.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.076+0000 7f13866d5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:19:50.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.076+0000 7f13866d5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:19:50.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.076+0000 7f13866d5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:19:50.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.076+0000 7f13866d5640 1 -- 172.21.15.37:0/1428856293 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f138004f510 con 0x7f13801223f0 2024-04-26T00:19:50.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.088+0000 7f137cff9640 1 -- 172.21.15.37:0/1428856293 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f137800bc30 con 0x7f13801223f0 2024-04-26T00:19:50.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.088+0000 7f137cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:19:50.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.088+0000 7f137cff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:19:50.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.152+0000 7f13866d5640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-04-26T00:19:50.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.152+0000 7f13866d5640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-04-26T00:19:50.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.152+0000 7f13866d5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:19:50.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.152+0000 7f13866d5640 1 -- 172.21.15.37:0/1428856293 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f1380075dc0 con 0x7f13801223f0 2024-04-26T00:19:50.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.156+0000 7f137cff9640 1 -- 172.21.15.37:0/1428856293 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f13780b9460 con 0x7f13801223f0 2024-04-26T00:19:50.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.156+0000 7f137cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-04-26T00:19:50.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.156+0000 7f137cff9640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-04-26T00:19:50.139 INFO:teuthology.orchestra.run.smithi037.stderr:adjusted tunables profile to default 2024-04-26T00:19:50.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.156+0000 7f13866d5640 10 monclient: shutdown 2024-04-26T00:19:50.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.156+0000 7f13866d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:19:50.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.156+0000 7f13866d5640 1 -- 172.21.15.37:0/1428856293 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13801223f0 msgr2=0x7f1380136910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:50.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.156+0000 7f13866d5640 1 --2- 172.21.15.37:0/1428856293 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13801223f0 0x7f1380136910 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f13780b1000 tx=0x7f1368005840 comp rx=0 tx=0).stop 2024-04-26T00:19:50.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.156+0000 7f13866d5640 1 -- 172.21.15.37:0/1428856293 shutdown_connections 2024-04-26T00:19:50.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.156+0000 7f13866d5640 1 --2- 172.21.15.37:0/1428856293 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13801223f0 0x7f1380136910 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:50.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.156+0000 7f13866d5640 1 --2- 172.21.15.37:0/1428856293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13801d0130 0x7f13801363d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:50.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.156+0000 7f13866d5640 1 --2- 172.21.15.37:0/1428856293 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f13801cf810 0x7f13801cfbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:50.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.160+0000 7f13866d5640 1 -- 172.21.15.37:0/1428856293 >> 172.21.15.37:0/1428856293 conn(0x7f13800aa0a0 msgr2=0x7f1380133270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:19:50.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.160+0000 7f13866d5640 1 -- 172.21.15.37:0/1428856293 shutdown_connections 2024-04-26T00:19:50.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:50.160+0000 7f13866d5640 1 -- 172.21.15.37:0/1428856293 wait complete. 2024-04-26T00:19:50.157 INFO:tasks.ceph:check_enable_crimson: False 2024-04-26T00:19:50.157 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-26T00:19:50.157 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T00:19:50.158 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-26T00:19:50.170 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T00:19:50.170 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-26T00:19:50.223 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T00:19:50.223 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-26T00:19:50.275 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-26T00:19:50.275 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-26T00:19:50.289 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-26T00:19:50.289 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-26T00:19:50.338 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-04-26T00:19:50.358+0000 7fc2fd1e0180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T00:19:50.344 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-26T00:19:50.344 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-26T00:19:50.401 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph --cluster ceph osd new a5d4b9d3-5499-491f-a123-85d73de452af 0 2024-04-26T00:19:50.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.590+0000 7fb2a0a48640 10 monclient: get_monmap_and_config 2024-04-26T00:19:50.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.590+0000 7fb2a0a48640 10 monclient: build_initial_monmap 2024-04-26T00:19:50.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.590+0000 7fb2a0a48640 10 monclient: monmap: 2024-04-26T00:19:50.571 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-04-26T00:19:50.571 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:50.571 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-04-26T00:19:50.596071+0000 2024-04-26T00:19:50.572 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-04-26T00:19:50.596071+0000 2024-04-26T00:19:50.572 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-04-26T00:19:50.572 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 3 2024-04-26T00:19:50.572 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:19:50.572 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:19:50.572 INFO:teuthology.orchestra.run.smithi045.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:19:50.572 INFO:teuthology.orchestra.run.smithi045.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:19:50.572 INFO:teuthology.orchestra.run.smithi045.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:19:50.572 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-26T00:19:50.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.590+0000 7fb2a0a48640 1 Processor -- start 2024-04-26T00:19:50.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.590+0000 7fb2a0a48640 1 -- start start 2024-04-26T00:19:50.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.590+0000 7fb2a0a48640 10 monclient: init 2024-04-26T00:19:50.574 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.590+0000 7fb2a0a48640 10 monclient: _reopen_session rank -1 2024-04-26T00:19:50.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.590+0000 7fb2a0a48640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-04-26T00:19:50.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.590+0000 7fb2a0a48640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb29c0a2fe0 addr v1:172.21.15.37:6789/0 2024-04-26T00:19:50.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.590+0000 7fb2a0a48640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb29c071dc0 0x7fb29c0a1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:50.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.590+0000 7fb2a0a48640 10 monclient(hunting): picked mon.noname-c con 0x7fb29c071dc0 addr v2:172.21.15.45:3300/0 2024-04-26T00:19:50.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.590+0000 7fb2a0a48640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb29c1391c0 addr v1:172.21.15.45:6789/0 2024-04-26T00:19:50.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.590+0000 7fb2a0a48640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb29c072d70 con 0x7fb29c0a2fe0 2024-04-26T00:19:50.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.590+0000 7fb2a0a48640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb29c070b20 con 0x7fb29c1391c0 2024-04-26T00:19:50.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.590+0000 7fb2a0a48640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:50.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.590+0000 7fb2a0a48640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fb29c0710e0 con 0x7fb29c071dc0 2024-04-26T00:19:50.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.590+0000 7fb2a0a48640 10 monclient(hunting): _renew_subs 2024-04-26T00:19:50.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.590+0000 7fb2a0a48640 10 monclient(hunting): authenticate will time out at 956.435753s 2024-04-26T00:19:50.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.590+0000 7fb299d9b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb29c071dc0 0x7fb29c0a1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:50.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.590+0000 7fb29ad9d640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fb29c1391c0 0x7fb29c13b5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:38446/0 (socket says 172.21.15.45:38446) 2024-04-26T00:19:50.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.590+0000 7fb29ad9d640 1 -- 172.21.15.45:0/3270562159 learned_addr learned my addr 172.21.15.45:0/3270562159 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-26T00:19:50.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.590+0000 7fb299d9b640 10 monclient(hunting): get_auth_request con 0x7fb29c071dc0 auth_method 0 2024-04-26T00:19:50.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.590+0000 7fb299d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:50.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.590+0000 7fb299d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:50.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.590+0000 7fb299d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:50.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.594+0000 7fb299d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:19:50.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.594+0000 7fb299d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:19:50.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.594+0000 7fb299d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:19:50.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.594+0000 7fb299d9b640 10 monclient(hunting): handle_auth_done global_id 4105 payload 274 2024-04-26T00:19:50.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.594+0000 7fb299d9b640 1 -- 172.21.15.45:0/3270562159 >> v1:172.21.15.45:6789/0 conn(0x7fb29c1391c0 legacy=0x7fb29c13b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:19:50.576 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.594+0000 7fb299d9b640 1 -- 172.21.15.45:0/3270562159 >> v1:172.21.15.37:6789/0 conn(0x7fb29c0a2fe0 legacy=0x7fb29c0a13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:19:50.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.594+0000 7fb299d9b640 10 monclient: _finish_hunting 0 2024-04-26T00:19:50.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.594+0000 7fb299d9b640 1 monclient: found mon.noname-c 2024-04-26T00:19:50.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.594+0000 7fb299d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:19:50.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.594+0000 7fb299d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:19:50.577 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.594+0000 7fb299d9b640 1 -- 172.21.15.45:0/3270562159 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb29c070e00 con 0x7fb29c071dc0 2024-04-26T00:19:50.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb299d9b640 1 -- 172.21.15.45:0/3270562159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb29c071dc0 msgr2=0x7fb29c0a1ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:19:50.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb299d9b640 1 --2- 172.21.15.45:0/3270562159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb29c071dc0 0x7fb29c0a1ae0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb29c071260 tx=0x7fb28c01c2a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=148670b03a873b3a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:19:50.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb299d9b640 1 -- 172.21.15.45:0/3270562159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb29c071dc0 msgr2=0x7fb29c0a1ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-26T00:19:50.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb299d9b640 1 -- 172.21.15.45:0/3270562159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb29c071dc0 msgr2=0x7fb29c0a1ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:19:50.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb299d9b640 1 --2- 172.21.15.45:0/3270562159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb29c071dc0 0x7fb29c0a1ae0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb29c071260 tx=0x7fb28c01c2a0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:19:50.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb299d9b640 1 --2- 172.21.15.45:0/3270562159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb29c071dc0 0x7fb29c0a1ae0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb29c071260 tx=0x7fb28c01c2a0 comp rx=0 tx=0).stop 2024-04-26T00:19:50.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb29959a640 10 monclient: ms_handle_reset current mon [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:19:50.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb29959a640 10 monclient: _reopen_session rank -1 2024-04-26T00:19:50.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb29959a640 1 -- 172.21.15.45:0/3270562159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb29c071dc0 msgr2=0x7fb29c0a1ae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:19:50.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb29959a640 1 --2- 172.21.15.45:0/3270562159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb29c071dc0 0x7fb29c0a1ae0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:50.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb29959a640 10 monclient: _add_conns ranks=[2,3] 2024-04-26T00:19:50.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb29959a640 1 --2- 172.21.15.45:0/3270562159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb284003510 0x7fb284005960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:50.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb29959a640 10 monclient(hunting): picked mon.noname-b con 0x7fb284003510 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:19:50.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb29959a640 1 --2- 172.21.15.45:0/3270562159 >> v2:172.21.15.37:3300/0 conn(0x7fb284006280 0x7fb284008740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:50.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb29959a640 10 monclient(hunting): picked mon.noname-a con 0x7fb284006280 addr v2:172.21.15.37:3300/0 2024-04-26T00:19:50.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb29959a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:50.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb29959a640 1 -- 172.21.15.45:0/3270562159 --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fb284009010 con 0x7fb284006280 2024-04-26T00:19:50.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb29959a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:50.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb29959a640 1 -- 172.21.15.45:0/3270562159 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb2840091c0 con 0x7fb284003510 2024-04-26T00:19:50.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb29959a640 10 monclient(hunting): _renew_subs 2024-04-26T00:19:50.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb29959a640 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:19:50.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb299d9b640 1 --2- 172.21.15.45:0/3270562159 >> v2:172.21.15.37:3300/0 conn(0x7fb284006280 0x7fb284008740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:50.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb29a59c640 1 --2- 172.21.15.45:0/3270562159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb284003510 0x7fb284005960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:50.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb299d9b640 10 monclient(hunting): get_auth_request con 0x7fb284006280 auth_method 0 2024-04-26T00:19:50.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb299d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:50.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb299d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:50.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb299d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:19:50.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb29a59c640 10 monclient(hunting): get_auth_request con 0x7fb284003510 auth_method 0 2024-04-26T00:19:50.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb29a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:50.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb29a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:50.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb29a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:19:50.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb299d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:19:50.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb299d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:19:50.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb299d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:19:50.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb29a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:19:50.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb29a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:19:50.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb29a59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:19:50.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb299d9b640 10 monclient(hunting): handle_auth_done global_id 4105 payload 293 2024-04-26T00:19:50.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb299d9b640 1 -- 172.21.15.45:0/3270562159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb284003510 msgr2=0x7fb284005960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:19:50.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb299d9b640 1 --2- 172.21.15.45:0/3270562159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb284003510 0x7fb284005960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:50.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb299d9b640 10 monclient: _finish_hunting 0 2024-04-26T00:19:50.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb299d9b640 1 monclient: found mon.noname-a 2024-04-26T00:19:50.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb299d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:19:50.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb299d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:19:50.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb299d9b640 1 -- 172.21.15.45:0/3270562159 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb284009480 con 0x7fb284006280 2024-04-26T00:19:50.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.598+0000 7fb29a59c640 1 --2- 172.21.15.45:0/3270562159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb284003510 0x7fb284005960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:19:50.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb299d9b640 1 -- 172.21.15.45:0/3270562159 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb284006280 msgr2=0x7fb284008740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:19:50.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb299d9b640 1 --2- 172.21.15.45:0/3270562159 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb284006280 0x7fb284008740 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb29c06f600 tx=0x7fb28c009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9a3464ad62dd05a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:19:50.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb29959a640 1 -- 172.21.15.45:0/3270562159 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb28c0b0070 con 0x7fb284006280 2024-04-26T00:19:50.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb29959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:50.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb29959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:19:50.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb29959a640 10 monclient: dump: 2024-04-26T00:19:50.593 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-04-26T00:19:50.593 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:50.593 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:50.593 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:50.593 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:50.593 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 3 2024-04-26T00:19:50.594 INFO:teuthology.orchestra.run.smithi045.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:50.594 INFO:teuthology.orchestra.run.smithi045.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:50.594 INFO:teuthology.orchestra.run.smithi045.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:50.594 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-26T00:19:50.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb29959a640 10 monclient: _finish_auth 0 2024-04-26T00:19:50.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb29959a640 10 monclient: _check_auth_tickets 2024-04-26T00:19:50.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb29959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:19:50.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb29959a640 1 -- 172.21.15.45:0/3270562159 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb29c0a2230 con 0x7fb284006280 2024-04-26T00:19:50.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb29959a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:19:50.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb29959a640 1 -- 172.21.15.45:0/3270562159 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb28c003a20 con 0x7fb284006280 2024-04-26T00:19:50.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb29959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:50.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb29959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:19:50.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb29959a640 10 monclient: dump: 2024-04-26T00:19:50.595 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-04-26T00:19:50.595 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:50.595 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:50.595 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:50.595 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:50.595 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 3 2024-04-26T00:19:50.595 INFO:teuthology.orchestra.run.smithi045.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:50.595 INFO:teuthology.orchestra.run.smithi045.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:50.596 INFO:teuthology.orchestra.run.smithi045.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:50.596 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-26T00:19:50.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb2a0a48640 5 monclient: authenticate success, global_id 4105 2024-04-26T00:19:50.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb2a0a48640 10 monclient: get_monmap_and_config success 2024-04-26T00:19:50.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb2a0a48640 4 set_mon_vals no callback set 2024-04-26T00:19:50.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb2a0a48640 10 monclient: shutdown 2024-04-26T00:19:50.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb2a0a48640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:19:50.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb2a0a48640 1 -- 172.21.15.45:0/3270562159 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb284006280 msgr2=0x7fb284008740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:50.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb2a0a48640 1 --2- 172.21.15.45:0/3270562159 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb284006280 0x7fb284008740 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb29c06f600 tx=0x7fb28c009190 comp rx=0 tx=0).stop 2024-04-26T00:19:50.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb29ad9d640 1 -- 172.21.15.45:0/3270562159 reap_dead start 2024-04-26T00:19:50.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb2a0a48640 1 -- 172.21.15.45:0/3270562159 shutdown_connections 2024-04-26T00:19:50.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb2a0a48640 1 --2- 172.21.15.45:0/3270562159 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb284006280 0x7fb284008740 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:50.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb2a0a48640 1 --2- 172.21.15.45:0/3270562159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb29c071dc0 0x7fb29c0a1ae0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:50.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb2a0a48640 1 -- 172.21.15.45:0/3270562159 >> 172.21.15.45:0/3270562159 conn(0x7fb29c0a24c0 msgr2=0x7fb29c0a28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:19:50.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb2a0a48640 1 -- 172.21.15.45:0/3270562159 shutdown_connections 2024-04-26T00:19:50.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb2a0a48640 1 -- 172.21.15.45:0/3270562159 wait complete. 2024-04-26T00:19:50.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb2a0a48640 10 monclient: build_initial_monmap 2024-04-26T00:19:50.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb2a0a48640 10 monclient: monmap: 2024-04-26T00:19:50.597 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-04-26T00:19:50.597 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:19:50.597 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-04-26T00:19:50.597 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-04-26T00:19:50.597 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-04-26T00:19:50.597 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-04-26T00:19:50.598 INFO:teuthology.orchestra.run.smithi045.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:19:50.598 INFO:teuthology.orchestra.run.smithi045.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:19:50.598 INFO:teuthology.orchestra.run.smithi045.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:19:50.598 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-26T00:19:50.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb2a0a48640 1 Processor -- start 2024-04-26T00:19:50.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb2a0a48640 1 -- start start 2024-04-26T00:19:50.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb2a0a48640 10 monclient: init 2024-04-26T00:19:50.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb2a0a48640 10 monclient: _reopen_session rank -1 2024-04-26T00:19:50.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb2a0a48640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:19:50.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb2a0a48640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb29c071dc0 0x7fb29c1ccf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:50.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb2a0a48640 10 monclient(hunting): picked mon.noname-a con 0x7fb29c071dc0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:19:50.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb2a0a48640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb284006280 0x7fb29c138530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:50.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb2a0a48640 10 monclient(hunting): picked mon.noname-b con 0x7fb284006280 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:19:50.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb2a0a48640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb29c138a70 0x7fb29c11ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:50.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb2a0a48640 10 monclient(hunting): picked mon.noname-c con 0x7fb29c138a70 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:19:50.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb2a0a48640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:50.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb2a0a48640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb29c13f880 con 0x7fb29c071dc0 2024-04-26T00:19:50.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb2a0a48640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:50.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb2a0a48640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb29c13f700 con 0x7fb284006280 2024-04-26T00:19:50.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb2a0a48640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:50.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb2a0a48640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb29c13fa00 con 0x7fb29c138a70 2024-04-26T00:19:50.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb2a0a48640 10 monclient(hunting): _renew_subs 2024-04-26T00:19:50.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb2a0a48640 10 monclient(hunting): authenticate will time out at 956.447192s 2024-04-26T00:19:50.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb299d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb284006280 0x7fb29c138530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:50.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb299d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb284006280 0x7fb29c138530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50148/0 (socket says 172.21.15.45:50148) 2024-04-26T00:19:50.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb299d9b640 1 -- 172.21.15.45:0/1381395757 learned_addr learned my addr 172.21.15.45:0/1381395757 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-26T00:19:50.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb299d9b640 10 monclient(hunting): get_auth_request con 0x7fb284006280 auth_method 0 2024-04-26T00:19:50.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb299d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:50.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb299d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:50.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb299d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:50.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb299d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:19:50.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb299d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:19:50.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb299d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:19:50.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb29a59c640 1 --2- 172.21.15.45:0/1381395757 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb29c071dc0 0x7fb29c1ccf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:50.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb29ad9d640 1 -- 172.21.15.45:0/1381395757 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb29c138a70 msgr2=0x7fb29c11ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:19:50.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb29ad9d640 1 -- 172.21.15.45:0/1381395757 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb29c138a70 msgr2=0x7fb29c11ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:19:50.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb29ad9d640 1 --2- 172.21.15.45:0/1381395757 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb29c138a70 0x7fb29c11ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2024-04-26T00:19:50.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb29a59c640 10 monclient(hunting): get_auth_request con 0x7fb29c071dc0 auth_method 0 2024-04-26T00:19:50.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb29a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:50.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb29a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:50.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb29a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:50.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb29ad9d640 1 --2- 172.21.15.45:0/1381395757 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb29c138a70 0x7fb29c11ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:19:50.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb299d9b640 10 monclient(hunting): handle_auth_done global_id 4108 payload 824 2024-04-26T00:19:50.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb299d9b640 1 -- 172.21.15.45:0/1381395757 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb29c138a70 msgr2=0x7fb29c11ef50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:19:50.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb299d9b640 1 --2- 172.21.15.45:0/1381395757 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb29c138a70 0x7fb29c11ef50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:50.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb299d9b640 1 -- 172.21.15.45:0/1381395757 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb29c071dc0 msgr2=0x7fb29c1ccf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:50.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb299d9b640 1 --2- 172.21.15.45:0/1381395757 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb29c071dc0 0x7fb29c1ccf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:50.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb299d9b640 10 monclient: _finish_hunting 0 2024-04-26T00:19:50.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb299d9b640 1 monclient: found mon.noname-b 2024-04-26T00:19:50.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb299d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:19:50.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb299d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.45:3300/0 2024-04-26T00:19:50.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.602+0000 7fb299d9b640 1 -- 172.21.15.45:0/1381395757 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb29c122c00 con 0x7fb284006280 2024-04-26T00:19:50.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.606+0000 7fb299d9b640 1 -- 172.21.15.45:0/1381395757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb284006280 msgr2=0x7fb29c138530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:19:50.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.606+0000 7fb299d9b640 1 --2- 172.21.15.45:0/1381395757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb284006280 0x7fb29c138530 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb28c01c780 tx=0x7fb28c019040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:19:50.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.606+0000 7fb28affd640 1 -- 172.21.15.45:0/1381395757 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb28c0b0070 con 0x7fb284006280 2024-04-26T00:19:50.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.606+0000 7fb28affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:50.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.606+0000 7fb28affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:19:50.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.606+0000 7fb28affd640 10 monclient: dump: 2024-04-26T00:19:50.603 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-04-26T00:19:50.603 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:50.603 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:50.603 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:50.603 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:50.603 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 3 2024-04-26T00:19:50.603 INFO:teuthology.orchestra.run.smithi045.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:50.603 INFO:teuthology.orchestra.run.smithi045.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:50.603 INFO:teuthology.orchestra.run.smithi045.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:50.603 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-26T00:19:50.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.606+0000 7fb28affd640 10 monclient: _finish_auth 0 2024-04-26T00:19:50.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.606+0000 7fb28affd640 10 monclient: _check_auth_tickets 2024-04-26T00:19:50.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.606+0000 7fb28affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:19:50.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.606+0000 7fb2a0a48640 5 monclient: authenticate success, global_id 4108 2024-04-26T00:19:50.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.606+0000 7fb2a0a48640 10 monclient: _renew_subs 2024-04-26T00:19:50.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.606+0000 7fb28affd640 1 -- 172.21.15.45:0/1381395757 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb28c003a30 con 0x7fb284006280 2024-04-26T00:19:50.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.606+0000 7fb2a0a48640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.45:3300/0 2024-04-26T00:19:50.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.606+0000 7fb2a0a48640 1 -- 172.21.15.45:0/1381395757 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb29c122ee0 con 0x7fb284006280 2024-04-26T00:19:50.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.606+0000 7fb28affd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:19:50.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.606+0000 7fb28affd640 1 -- 172.21.15.45:0/1381395757 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb28c003de0 con 0x7fb284006280 2024-04-26T00:19:50.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.606+0000 7fb28affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:50.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.606+0000 7fb28affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:19:50.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.606+0000 7fb298d99640 4 set_mon_vals no callback set 2024-04-26T00:19:50.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.606+0000 7fb28affd640 10 monclient: dump: 2024-04-26T00:19:50.605 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-04-26T00:19:50.605 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:50.605 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:50.605 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:50.605 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:50.605 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 3 2024-04-26T00:19:50.605 INFO:teuthology.orchestra.run.smithi045.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:50.605 INFO:teuthology.orchestra.run.smithi045.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:50.605 INFO:teuthology.orchestra.run.smithi045.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:50.606 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-26T00:19:50.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.606+0000 7fb2a0a48640 10 monclient: _renew_subs 2024-04-26T00:19:50.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.606+0000 7fb2a0a48640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.45:3300/0 2024-04-26T00:19:50.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.606+0000 7fb2a0a48640 1 -- 172.21.15.45:0/1381395757 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb29c0a4e40 con 0x7fb284006280 2024-04-26T00:19:50.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.606+0000 7fb28affd640 1 -- 172.21.15.45:0/1381395757 <== mon.1 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb28c0c8800 con 0x7fb284006280 2024-04-26T00:19:50.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.606+0000 7fb28affd640 1 -- 172.21.15.45:0/1381395757 <== mon.1 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7fb28c0be890 con 0x7fb284006280 2024-04-26T00:19:50.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.606+0000 7fb2a0a48640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:19:50.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.606+0000 7fb2a0a48640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:19:50.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.606+0000 7fb2a0a48640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.45:3300/0 2024-04-26T00:19:50.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.606+0000 7fb2a0a48640 1 -- 172.21.15.45:0/1381395757 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb29c04f510 con 0x7fb284006280 2024-04-26T00:19:50.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.614+0000 7fb28affd640 1 -- 172.21.15.45:0/1381395757 <== mon.1 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fb28c0ba030 con 0x7fb284006280 2024-04-26T00:19:50.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.614+0000 7fb28affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:19:50.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.614+0000 7fb28affd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:19:50.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.686+0000 7fb2a0a48640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a5d4b9d3-5499-491f-a123-85d73de452af", "id": 0}] 2024-04-26T00:19:50.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.686+0000 7fb2a0a48640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a5d4b9d3-5499-491f-a123-85d73de452af", "id": 0}] 2024-04-26T00:19:50.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.686+0000 7fb2a0a48640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.45:3300/0 2024-04-26T00:19:50.665 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.686+0000 7fb2a0a48640 1 -- 172.21.15.45:0/1381395757 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a5d4b9d3-5499-491f-a123-85d73de452af", "id": 0} v 0) -- 0x7fb29c04f510 con 0x7fb284006280 2024-04-26T00:19:50.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.690+0000 7fb28affd640 1 -- 172.21.15.45:0/1381395757 <== mon.1 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a5d4b9d3-5499-491f-a123-85d73de452af", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7fb28c015fb0 con 0x7fb284006280 2024-04-26T00:19:50.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.690+0000 7fb28affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a5d4b9d3-5499-491f-a123-85d73de452af", "id": 0}] 2024-04-26T00:19:50.671 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.690+0000 7fb28affd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:19:50.671 INFO:teuthology.orchestra.run.smithi045.stdout:0 2024-04-26T00:19:50.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.694+0000 7fb2a0a48640 10 monclient: shutdown 2024-04-26T00:19:50.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.694+0000 7fb2a0a48640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:19:50.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.694+0000 7fb2a0a48640 1 -- 172.21.15.45:0/1381395757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb284006280 msgr2=0x7fb29c138530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:50.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.694+0000 7fb2a0a48640 1 --2- 172.21.15.45:0/1381395757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb284006280 0x7fb29c138530 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb28c01c780 tx=0x7fb28c019040 comp rx=0 tx=0).stop 2024-04-26T00:19:50.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.694+0000 7fb2a0a48640 1 -- 172.21.15.45:0/1381395757 shutdown_connections 2024-04-26T00:19:50.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.694+0000 7fb2a0a48640 1 --2- 172.21.15.45:0/1381395757 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb29c138a70 0x7fb29c11ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:50.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.694+0000 7fb2a0a48640 1 --2- 172.21.15.45:0/1381395757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb284006280 0x7fb29c138530 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:50.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.694+0000 7fb2a0a48640 1 --2- 172.21.15.45:0/1381395757 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb29c071dc0 0x7fb29c1ccf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:50.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.694+0000 7fb2a0a48640 1 -- 172.21.15.45:0/1381395757 >> 172.21.15.45:0/1381395757 conn(0x7fb29c0a24c0 msgr2=0x7fb29c0a50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:19:50.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.694+0000 7fb2a0a48640 1 -- 172.21.15.45:0/1381395757 shutdown_connections 2024-04-26T00:19:50.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.694+0000 7fb2a0a48640 1 -- 172.21.15.45:0/1381395757 wait complete. 2024-04-26T00:19:50.690 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph --cluster ceph osd new cb4583fd-a0fa-4559-9f99-5a3e02b5818c 1 2024-04-26T00:19:50.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.830+0000 7f268d6dd640 10 monclient: get_monmap_and_config 2024-04-26T00:19:50.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.830+0000 7f268d6dd640 10 monclient: build_initial_monmap 2024-04-26T00:19:50.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.830+0000 7f268d6dd640 10 monclient: monmap: 2024-04-26T00:19:50.810 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-04-26T00:19:50.814 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:50.814 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-04-26T00:19:50.835849+0000 2024-04-26T00:19:50.814 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-04-26T00:19:50.835849+0000 2024-04-26T00:19:50.814 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-04-26T00:19:50.815 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 3 2024-04-26T00:19:50.815 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:19:50.815 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:19:50.816 INFO:teuthology.orchestra.run.smithi045.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:19:50.816 INFO:teuthology.orchestra.run.smithi045.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:19:50.816 INFO:teuthology.orchestra.run.smithi045.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:19:50.816 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-26T00:19:50.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.830+0000 7f268d6dd640 1 Processor -- start 2024-04-26T00:19:50.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.830+0000 7f268d6dd640 1 -- start start 2024-04-26T00:19:50.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.830+0000 7f268d6dd640 10 monclient: init 2024-04-26T00:19:50.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.830+0000 7f268d6dd640 10 monclient: _reopen_session rank -1 2024-04-26T00:19:50.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.830+0000 7f268d6dd640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-04-26T00:19:50.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.830+0000 7f268d6dd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f26880a1b60 addr v1:172.21.15.37:6789/0 2024-04-26T00:19:50.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.830+0000 7f268d6dd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f26881379e0 0x7f268813c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:50.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.830+0000 7f268d6dd640 10 monclient(hunting): picked mon.noname-a con 0x7f26881379e0 addr v2:172.21.15.37:3300/0 2024-04-26T00:19:50.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.830+0000 7f268d6dd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2688137e30 addr v1:172.21.15.45:6789/0 2024-04-26T00:19:50.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.830+0000 7f268d6dd640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f268809d7d0 con 0x7f26880a1b60 2024-04-26T00:19:50.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.830+0000 7f268d6dd640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f268809dcc0 con 0x7f2688137e30 2024-04-26T00:19:50.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.830+0000 7f268d6dd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:50.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.830+0000 7f268d6dd640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f268806c590 con 0x7f26881379e0 2024-04-26T00:19:50.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.830+0000 7f268d6dd640 10 monclient(hunting): _renew_subs 2024-04-26T00:19:50.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.830+0000 7f268d6dd640 10 monclient(hunting): authenticate will time out at 956.675471s 2024-04-26T00:19:50.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.830+0000 7f26877fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f2688137e30 0x7f268813ec70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:38452/0 (socket says 172.21.15.45:38452) 2024-04-26T00:19:50.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.830+0000 7f26877fe640 1 -- 172.21.15.45:0/2711089594 learned_addr learned my addr 172.21.15.45:0/2711089594 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-26T00:19:50.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.830+0000 7f26867fc640 1 --2- 172.21.15.45:0/2711089594 >> v2:172.21.15.37:3300/0 conn(0x7f26881379e0 0x7f268813c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:50.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.830+0000 7f26867fc640 10 monclient(hunting): get_auth_request con 0x7f26881379e0 auth_method 0 2024-04-26T00:19:50.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.830+0000 7f26867fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:50.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.830+0000 7f26867fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:50.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.830+0000 7f26867fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:50.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.834+0000 7f26867fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:19:50.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.834+0000 7f26867fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:19:50.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.834+0000 7f26867fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:19:50.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.834+0000 7f26867fc640 10 monclient(hunting): handle_auth_done global_id 4113 payload 274 2024-04-26T00:19:50.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.834+0000 7f26867fc640 1 -- 172.21.15.45:0/2711089594 >> v1:172.21.15.45:6789/0 conn(0x7f2688137e30 legacy=0x7f268813ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:19:50.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.834+0000 7f26867fc640 1 -- 172.21.15.45:0/2711089594 >> v1:172.21.15.37:6789/0 conn(0x7f26880a1b60 legacy=0x7f26881372d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:19:50.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.834+0000 7f26867fc640 10 monclient: _finish_hunting 0 2024-04-26T00:19:50.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.834+0000 7f26867fc640 1 monclient: found mon.noname-a 2024-04-26T00:19:50.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.834+0000 7f26867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:19:50.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.834+0000 7f26867fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:19:50.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.834+0000 7f26867fc640 1 -- 172.21.15.45:0/2711089594 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26880a29b0 con 0x7f26881379e0 2024-04-26T00:19:50.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.838+0000 7f26867fc640 1 -- 172.21.15.45:0/2711089594 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26881379e0 msgr2=0x7f268813c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:19:50.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.838+0000 7f26867fc640 1 --2- 172.21.15.45:0/2711089594 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26881379e0 0x7f268813c720 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f267c0097f0 tx=0x7f267c01c190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36f29849060e1bc3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:19:50.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.838+0000 7f2685ffb640 1 -- 172.21.15.45:0/2711089594 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267c0b0070 con 0x7f26881379e0 2024-04-26T00:19:50.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.838+0000 7f2685ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:50.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.838+0000 7f2685ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:19:50.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.838+0000 7f2685ffb640 10 monclient: dump: 2024-04-26T00:19:50.824 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-04-26T00:19:50.824 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:50.824 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:50.824 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:50.824 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:50.824 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 3 2024-04-26T00:19:50.825 INFO:teuthology.orchestra.run.smithi045.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:50.825 INFO:teuthology.orchestra.run.smithi045.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:50.825 INFO:teuthology.orchestra.run.smithi045.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:50.825 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-26T00:19:50.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.838+0000 7f2685ffb640 10 monclient: _finish_auth 0 2024-04-26T00:19:50.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.838+0000 7f2685ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:19:50.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.838+0000 7f2685ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:19:50.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.838+0000 7f2685ffb640 1 -- 172.21.15.45:0/2711089594 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f267c0ab600 con 0x7f26881379e0 2024-04-26T00:19:50.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.838+0000 7f2685ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:19:50.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.838+0000 7f2685ffb640 1 -- 172.21.15.45:0/2711089594 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267c0ab9d0 con 0x7f26881379e0 2024-04-26T00:19:50.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.838+0000 7f2685ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:50.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.838+0000 7f2685ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:19:50.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.838+0000 7f2685ffb640 10 monclient: dump: 2024-04-26T00:19:50.826 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-04-26T00:19:50.826 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:50.826 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:50.826 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:50.826 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:50.826 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 3 2024-04-26T00:19:50.826 INFO:teuthology.orchestra.run.smithi045.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:50.826 INFO:teuthology.orchestra.run.smithi045.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:50.826 INFO:teuthology.orchestra.run.smithi045.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:50.827 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-26T00:19:50.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.838+0000 7f268d6dd640 5 monclient: authenticate success, global_id 4113 2024-04-26T00:19:50.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.838+0000 7f268d6dd640 10 monclient: get_monmap_and_config success 2024-04-26T00:19:50.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.838+0000 7f268d6dd640 4 set_mon_vals no callback set 2024-04-26T00:19:50.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.838+0000 7f268d6dd640 10 monclient: shutdown 2024-04-26T00:19:50.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.838+0000 7f268d6dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:19:50.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.838+0000 7f268d6dd640 1 -- 172.21.15.45:0/2711089594 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26881379e0 msgr2=0x7f268813c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:50.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.838+0000 7f268d6dd640 1 --2- 172.21.15.45:0/2711089594 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26881379e0 0x7f268813c720 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f267c0097f0 tx=0x7f267c01c190 comp rx=0 tx=0).stop 2024-04-26T00:19:50.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.838+0000 7f268d6dd640 1 -- 172.21.15.45:0/2711089594 shutdown_connections 2024-04-26T00:19:50.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.838+0000 7f268d6dd640 1 --2- 172.21.15.45:0/2711089594 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26881379e0 0x7f268813c720 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:50.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.838+0000 7f268d6dd640 1 -- 172.21.15.45:0/2711089594 >> 172.21.15.45:0/2711089594 conn(0x7f26880a2e20 msgr2=0x7f26880a3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:19:50.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.838+0000 7f268d6dd640 1 -- 172.21.15.45:0/2711089594 shutdown_connections 2024-04-26T00:19:50.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.838+0000 7f268d6dd640 1 -- 172.21.15.45:0/2711089594 wait complete. 2024-04-26T00:19:50.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.838+0000 7f268d6dd640 10 monclient: build_initial_monmap 2024-04-26T00:19:50.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.838+0000 7f268d6dd640 10 monclient: monmap: 2024-04-26T00:19:50.828 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-04-26T00:19:50.828 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:19:50.828 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-04-26T00:19:50.828 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-04-26T00:19:50.828 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-04-26T00:19:50.828 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-04-26T00:19:50.828 INFO:teuthology.orchestra.run.smithi045.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:19:50.828 INFO:teuthology.orchestra.run.smithi045.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:19:50.828 INFO:teuthology.orchestra.run.smithi045.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:19:50.828 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-26T00:19:50.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.838+0000 7f268d6dd640 1 Processor -- start 2024-04-26T00:19:50.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.838+0000 7f268d6dd640 1 -- start start 2024-04-26T00:19:50.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.838+0000 7f268d6dd640 10 monclient: init 2024-04-26T00:19:50.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f268d6dd640 10 monclient: _reopen_session rank -1 2024-04-26T00:19:50.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f268d6dd640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:19:50.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f268d6dd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26880a1b60 0x7f268813a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:50.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f268d6dd640 10 monclient(hunting): picked mon.noname-c con 0x7f26880a1b60 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:19:50.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f268d6dd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26881379e0 0x7f268813ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:50.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f268d6dd640 10 monclient(hunting): picked mon.noname-b con 0x7f26881379e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:19:50.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f268d6dd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2688137e30 0x7f268813b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:50.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f268d6dd640 10 monclient(hunting): picked mon.noname-a con 0x7f2688137e30 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:19:50.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f268d6dd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:50.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f268d6dd640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f26880a1f40 con 0x7f2688137e30 2024-04-26T00:19:50.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f268d6dd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:50.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f268d6dd640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f268809e030 con 0x7f26881379e0 2024-04-26T00:19:50.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f268d6dd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:50.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f268d6dd640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2688142dc0 con 0x7f26880a1b60 2024-04-26T00:19:50.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f268d6dd640 10 monclient(hunting): _renew_subs 2024-04-26T00:19:50.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f268d6dd640 10 monclient(hunting): authenticate will time out at 956.684436s 2024-04-26T00:19:50.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f26867fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26881379e0 0x7f268813ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:50.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f26867fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26881379e0 0x7f268813ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50160/0 (socket says 172.21.15.45:50160) 2024-04-26T00:19:50.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f26867fc640 1 -- 172.21.15.45:0/3528076838 learned_addr learned my addr 172.21.15.45:0/3528076838 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-26T00:19:50.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f26867fc640 10 monclient(hunting): get_auth_request con 0x7f26881379e0 auth_method 0 2024-04-26T00:19:50.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f26867fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:50.830 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f26867fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:50.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f26867fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:50.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f26867fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:19:50.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f26867fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:19:50.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f26867fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:19:50.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f2686ffd640 1 --2- 172.21.15.45:0/3528076838 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26880a1b60 0x7f268813a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:50.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f2686ffd640 10 monclient(hunting): get_auth_request con 0x7f26880a1b60 auth_method 0 2024-04-26T00:19:50.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f2686ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:50.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f2686ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:50.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f2686ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:50.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f26877fe640 1 --2- 172.21.15.45:0/3528076838 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2688137e30 0x7f268813b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:50.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f26867fc640 10 monclient(hunting): handle_auth_done global_id 4111 payload 824 2024-04-26T00:19:50.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f26867fc640 1 -- 172.21.15.45:0/3528076838 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26880a1b60 msgr2=0x7f268813a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:50.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f26867fc640 1 --2- 172.21.15.45:0/3528076838 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26880a1b60 0x7f268813a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:50.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f26867fc640 1 -- 172.21.15.45:0/3528076838 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2688137e30 msgr2=0x7f268813b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:50.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f26867fc640 1 --2- 172.21.15.45:0/3528076838 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2688137e30 0x7f268813b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:50.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f26867fc640 10 monclient: _finish_hunting 0 2024-04-26T00:19:50.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f26867fc640 1 monclient: found mon.noname-b 2024-04-26T00:19:50.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f26867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:19:50.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f26867fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.45:3300/0 2024-04-26T00:19:50.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f26867fc640 1 -- 172.21.15.45:0/3528076838 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26880a2680 con 0x7f26881379e0 2024-04-26T00:19:50.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f26877fe640 10 monclient: get_auth_request con 0x7f2688137e30 auth_method 0 2024-04-26T00:19:50.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f26877fe640 1 --2- 172.21.15.45:0/3528076838 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2688137e30 0x7f268813b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:19:50.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f26867fc640 1 -- 172.21.15.45:0/3528076838 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26881379e0 msgr2=0x7f268813ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:19:50.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f26867fc640 1 --2- 172.21.15.45:0/3528076838 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26881379e0 0x7f268813ab60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f267c0ae040 tx=0x7f267c01a9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:19:50.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f266f7fe640 1 -- 172.21.15.45:0/3528076838 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267c0b0050 con 0x7f26881379e0 2024-04-26T00:19:50.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f266f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:50.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f266f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:19:50.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f266f7fe640 10 monclient: dump: 2024-04-26T00:19:50.834 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-04-26T00:19:50.834 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:50.834 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:50.834 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:50.834 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:50.834 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 3 2024-04-26T00:19:50.835 INFO:teuthology.orchestra.run.smithi045.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:50.835 INFO:teuthology.orchestra.run.smithi045.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:50.835 INFO:teuthology.orchestra.run.smithi045.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:50.835 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-26T00:19:50.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f266f7fe640 10 monclient: _finish_auth 0 2024-04-26T00:19:50.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f266f7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:19:50.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f266f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:19:50.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f266f7fe640 1 -- 172.21.15.45:0/3528076838 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f267c0bc070 con 0x7f26881379e0 2024-04-26T00:19:50.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f266f7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:19:50.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f266f7fe640 1 -- 172.21.15.45:0/3528076838 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267c0b5c20 con 0x7f26881379e0 2024-04-26T00:19:50.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f266f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:50.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f266f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:19:50.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f266f7fe640 10 monclient: dump: 2024-04-26T00:19:50.836 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-04-26T00:19:50.836 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:50.836 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:50.836 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:50.836 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:50.836 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 3 2024-04-26T00:19:50.836 INFO:teuthology.orchestra.run.smithi045.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:50.836 INFO:teuthology.orchestra.run.smithi045.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:50.836 INFO:teuthology.orchestra.run.smithi045.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:50.836 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-26T00:19:50.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f268d6dd640 5 monclient: authenticate success, global_id 4111 2024-04-26T00:19:50.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f268d6dd640 10 monclient: _renew_subs 2024-04-26T00:19:50.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f268d6dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.45:3300/0 2024-04-26T00:19:50.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f268d6dd640 1 -- 172.21.15.45:0/3528076838 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f268813c1c0 con 0x7f26881379e0 2024-04-26T00:19:50.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f2684ff9640 4 set_mon_vals no callback set 2024-04-26T00:19:50.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f268d6dd640 10 monclient: _renew_subs 2024-04-26T00:19:50.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f268d6dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.45:3300/0 2024-04-26T00:19:50.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f268d6dd640 1 -- 172.21.15.45:0/3528076838 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26881400a0 con 0x7f26881379e0 2024-04-26T00:19:50.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f266f7fe640 1 -- 172.21.15.45:0/3528076838 <== mon.1 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f267c0c1460 con 0x7f26881379e0 2024-04-26T00:19:50.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.842+0000 7f266f7fe640 1 -- 172.21.15.45:0/3528076838 <== mon.1 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f267c0b7800 con 0x7f26881379e0 2024-04-26T00:19:50.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.846+0000 7f268d6dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:19:50.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.846+0000 7f268d6dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:19:50.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.846+0000 7f268d6dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.45:3300/0 2024-04-26T00:19:50.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.846+0000 7f268d6dd640 1 -- 172.21.15.45:0/3528076838 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f268804f510 con 0x7f26881379e0 2024-04-26T00:19:50.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.846+0000 7f266f7fe640 1 -- 172.21.15.45:0/3528076838 <== mon.1 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f268809d7d0 con 0x7f26881379e0 2024-04-26T00:19:50.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.846+0000 7f266f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:19:50.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.846+0000 7f266f7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:19:50.843 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-04-26T00:19:50.862+0000 7fc2fd1e0180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T00:19:50.895 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.914+0000 7f268d6dd640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "cb4583fd-a0fa-4559-9f99-5a3e02b5818c", "id": 1}] 2024-04-26T00:19:50.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.914+0000 7f268d6dd640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "cb4583fd-a0fa-4559-9f99-5a3e02b5818c", "id": 1}] 2024-04-26T00:19:50.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.914+0000 7f268d6dd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.45:3300/0 2024-04-26T00:19:50.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.914+0000 7f268d6dd640 1 -- 172.21.15.45:0/3528076838 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd new", "uuid": "cb4583fd-a0fa-4559-9f99-5a3e02b5818c", "id": 1} v 0) -- 0x7f268806ca10 con 0x7f26881379e0 2024-04-26T00:19:50.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.974+0000 7f266f7fe640 1 -- 172.21.15.45:0/3528076838 <== mon.1 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cb4583fd-a0fa-4559-9f99-5a3e02b5818c", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f267c0b5e10 con 0x7f26881379e0 2024-04-26T00:19:50.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.974+0000 7f266f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "cb4583fd-a0fa-4559-9f99-5a3e02b5818c", "id": 1}] 2024-04-26T00:19:50.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.974+0000 7f266f7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:19:50.956 INFO:teuthology.orchestra.run.smithi045.stdout:1 2024-04-26T00:19:50.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.978+0000 7f268d6dd640 10 monclient: shutdown 2024-04-26T00:19:50.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.978+0000 7f268d6dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:19:50.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.978+0000 7f268d6dd640 1 -- 172.21.15.45:0/3528076838 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26881379e0 msgr2=0x7f268813ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:50.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.978+0000 7f268d6dd640 1 --2- 172.21.15.45:0/3528076838 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26881379e0 0x7f268813ab60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f267c0ae040 tx=0x7f267c01a9d0 comp rx=0 tx=0).stop 2024-04-26T00:19:50.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.978+0000 7f268d6dd640 1 -- 172.21.15.45:0/3528076838 shutdown_connections 2024-04-26T00:19:50.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.978+0000 7f268d6dd640 1 --2- 172.21.15.45:0/3528076838 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2688137e30 0x7f268813b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:50.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.978+0000 7f268d6dd640 1 --2- 172.21.15.45:0/3528076838 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26881379e0 0x7f268813ab60 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:50.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.978+0000 7f268d6dd640 1 --2- 172.21.15.45:0/3528076838 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26880a1b60 0x7f268813a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:50.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.978+0000 7f268d6dd640 1 -- 172.21.15.45:0/3528076838 >> 172.21.15.45:0/3528076838 conn(0x7f26880a2e20 msgr2=0x7f268809ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:19:50.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.978+0000 7f268d6dd640 1 -- 172.21.15.45:0/3528076838 shutdown_connections 2024-04-26T00:19:50.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:50.978+0000 7f268d6dd640 1 -- 172.21.15.45:0/3528076838 wait complete. 2024-04-26T00:19:50.974 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph --cluster ceph osd new 78fc9fe6-2163-4cb3-9944-3773bd488df8 2 2024-04-26T00:19:51.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.118+0000 7fbeb842c640 10 monclient: get_monmap_and_config 2024-04-26T00:19:51.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.118+0000 7fbeb842c640 10 monclient: build_initial_monmap 2024-04-26T00:19:51.098 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.118+0000 7fbeb842c640 10 monclient: monmap: 2024-04-26T00:19:51.098 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-04-26T00:19:51.098 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:51.098 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-04-26T00:19:51.123850+0000 2024-04-26T00:19:51.098 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-04-26T00:19:51.123850+0000 2024-04-26T00:19:51.099 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-04-26T00:19:51.099 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 3 2024-04-26T00:19:51.099 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:19:51.099 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:19:51.099 INFO:teuthology.orchestra.run.smithi045.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:19:51.099 INFO:teuthology.orchestra.run.smithi045.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:19:51.099 INFO:teuthology.orchestra.run.smithi045.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:19:51.099 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-26T00:19:51.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.118+0000 7fbeb842c640 1 Processor -- start 2024-04-26T00:19:51.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.118+0000 7fbeb842c640 1 -- start start 2024-04-26T00:19:51.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.118+0000 7fbeb842c640 10 monclient: init 2024-04-26T00:19:51.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.118+0000 7fbeb842c640 10 monclient: _reopen_session rank -1 2024-04-26T00:19:51.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.118+0000 7fbeb842c640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-04-26T00:19:51.101 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.118+0000 7fbeb842c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbeb00a38e0 addr v1:172.21.15.45:6789/0 2024-04-26T00:19:51.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.118+0000 7fbeb842c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbeb00a2410 0x7fbeb013c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:51.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.118+0000 7fbeb842c640 10 monclient(hunting): picked mon.noname-b con 0x7fbeb00a2410 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:19:51.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.118+0000 7fbeb842c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbeb00a27f0 addr v1:172.21.15.37:6789/0 2024-04-26T00:19:51.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.118+0000 7fbeb842c640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbeb009a9d0 con 0x7fbeb00a27f0 2024-04-26T00:19:51.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.118+0000 7fbeb842c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbeb009acb0 con 0x7fbeb00a38e0 2024-04-26T00:19:51.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.118+0000 7fbeb842c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:51.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.118+0000 7fbeb842c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fbeb009bc80 con 0x7fbeb00a2410 2024-04-26T00:19:51.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.118+0000 7fbeb842c640 10 monclient(hunting): _renew_subs 2024-04-26T00:19:51.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.118+0000 7fbeb842c640 10 monclient(hunting): authenticate will time out at 956.963431s 2024-04-26T00:19:51.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.118+0000 7fbeb61c8640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fbeb00a38e0 0x7fbeb00a1d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:38466/0 (socket says 172.21.15.45:38466) 2024-04-26T00:19:51.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.118+0000 7fbeb61c8640 1 -- 172.21.15.45:0/1212309947 learned_addr learned my addr 172.21.15.45:0/1212309947 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-26T00:19:51.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.118+0000 7fbeb59c7640 1 --2- 172.21.15.45:0/1212309947 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbeb00a2410 0x7fbeb013c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:51.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.118+0000 7fbeb59c7640 10 monclient(hunting): get_auth_request con 0x7fbeb00a2410 auth_method 0 2024-04-26T00:19:51.102 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.118+0000 7fbeb59c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:51.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.118+0000 7fbeb59c7640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:51.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.118+0000 7fbeb59c7640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:51.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.122+0000 7fbeb59c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:19:51.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.122+0000 7fbeb59c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:19:51.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.122+0000 7fbeb59c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:19:51.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.122+0000 7fbeb59c7640 10 monclient(hunting): handle_auth_done global_id 4109 payload 274 2024-04-26T00:19:51.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.122+0000 7fbeb59c7640 1 -- 172.21.15.45:0/1212309947 >> v1:172.21.15.45:6789/0 conn(0x7fbeb00a38e0 legacy=0x7fbeb00a1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:19:51.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.122+0000 7fbeb59c7640 1 -- 172.21.15.45:0/1212309947 >> v1:172.21.15.37:6789/0 conn(0x7fbeb00a27f0 legacy=0x7fbeb013ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:19:51.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.122+0000 7fbeb59c7640 10 monclient: _finish_hunting 0 2024-04-26T00:19:51.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.122+0000 7fbeb59c7640 1 monclient: found mon.noname-b 2024-04-26T00:19:51.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.122+0000 7fbeb59c7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:19:51.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.122+0000 7fbeb59c7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:19:51.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.122+0000 7fbeb59c7640 1 -- 172.21.15.45:0/1212309947 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeb009af90 con 0x7fbeb00a2410 2024-04-26T00:19:51.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.126+0000 7fbeb59c7640 1 -- 172.21.15.45:0/1212309947 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbeb00a2410 msgr2=0x7fbeb013c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:19:51.105 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.126+0000 7fbeb59c7640 1 --2- 172.21.15.45:0/1212309947 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbeb00a2410 0x7fbeb013c6b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbeac0097f0 tx=0x7fbeac01cd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3b3c67290874dd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:19:51.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.126+0000 7fbeb51c6640 1 -- 172.21.15.45:0/1212309947 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeac0b0070 con 0x7fbeb00a2410 2024-04-26T00:19:51.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.126+0000 7fbeb51c6640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:51.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.126+0000 7fbeb51c6640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:19:51.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.126+0000 7fbeb51c6640 10 monclient: dump: 2024-04-26T00:19:51.106 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-04-26T00:19:51.106 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:51.106 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:51.106 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:51.106 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:51.106 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 3 2024-04-26T00:19:51.107 INFO:teuthology.orchestra.run.smithi045.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:51.107 INFO:teuthology.orchestra.run.smithi045.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:51.107 INFO:teuthology.orchestra.run.smithi045.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:51.107 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-26T00:19:51.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.126+0000 7fbeb51c6640 10 monclient: _finish_auth 0 2024-04-26T00:19:51.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.126+0000 7fbeb51c6640 10 monclient: _check_auth_tickets 2024-04-26T00:19:51.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.126+0000 7fbeb51c6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:19:51.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.126+0000 7fbeb51c6640 1 -- 172.21.15.45:0/1212309947 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbeac0ab600 con 0x7fbeb00a2410 2024-04-26T00:19:51.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.126+0000 7fbeb51c6640 10 monclient: handle_config config(0 keys) 2024-04-26T00:19:51.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.126+0000 7fbeb51c6640 1 -- 172.21.15.45:0/1212309947 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeac0ab9d0 con 0x7fbeb00a2410 2024-04-26T00:19:51.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.126+0000 7fbeb51c6640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:51.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.126+0000 7fbeb51c6640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:19:51.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.126+0000 7fbeb51c6640 10 monclient: dump: 2024-04-26T00:19:51.107 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-04-26T00:19:51.107 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:51.108 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:51.108 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:51.108 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:51.108 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 3 2024-04-26T00:19:51.108 INFO:teuthology.orchestra.run.smithi045.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:51.108 INFO:teuthology.orchestra.run.smithi045.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:51.108 INFO:teuthology.orchestra.run.smithi045.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:51.108 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-26T00:19:51.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.126+0000 7fbeb842c640 5 monclient: authenticate success, global_id 4109 2024-04-26T00:19:51.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.126+0000 7fbeb842c640 10 monclient: get_monmap_and_config success 2024-04-26T00:19:51.108 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.126+0000 7fbeb842c640 4 set_mon_vals no callback set 2024-04-26T00:19:51.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.126+0000 7fbeb842c640 10 monclient: shutdown 2024-04-26T00:19:51.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.126+0000 7fbeb842c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:19:51.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.126+0000 7fbeb842c640 1 -- 172.21.15.45:0/1212309947 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbeb00a2410 msgr2=0x7fbeb013c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:51.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.126+0000 7fbeb842c640 1 --2- 172.21.15.45:0/1212309947 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbeb00a2410 0x7fbeb013c6b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbeac0097f0 tx=0x7fbeac01cd80 comp rx=0 tx=0).stop 2024-04-26T00:19:51.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.126+0000 7fbeb842c640 1 -- 172.21.15.45:0/1212309947 shutdown_connections 2024-04-26T00:19:51.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.126+0000 7fbeb842c640 1 --2- 172.21.15.45:0/1212309947 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbeb00a2410 0x7fbeb013c6b0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:51.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.126+0000 7fbeb842c640 1 -- 172.21.15.45:0/1212309947 >> 172.21.15.45:0/1212309947 conn(0x7fbeb00a2df0 msgr2=0x7fbeb00a31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:19:51.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.126+0000 7fbeb842c640 1 -- 172.21.15.45:0/1212309947 shutdown_connections 2024-04-26T00:19:51.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.126+0000 7fbeb842c640 1 -- 172.21.15.45:0/1212309947 wait complete. 2024-04-26T00:19:51.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.126+0000 7fbeb842c640 10 monclient: build_initial_monmap 2024-04-26T00:19:51.112 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.126+0000 7fbeb842c640 10 monclient: monmap: 2024-04-26T00:19:51.112 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-04-26T00:19:51.112 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:19:51.112 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-04-26T00:19:51.112 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-04-26T00:19:51.112 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-04-26T00:19:51.112 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-04-26T00:19:51.112 INFO:teuthology.orchestra.run.smithi045.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:19:51.112 INFO:teuthology.orchestra.run.smithi045.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:19:51.113 INFO:teuthology.orchestra.run.smithi045.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:19:51.113 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-26T00:19:51.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.126+0000 7fbeb842c640 1 Processor -- start 2024-04-26T00:19:51.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb842c640 1 -- start start 2024-04-26T00:19:51.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb842c640 10 monclient: init 2024-04-26T00:19:51.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb842c640 10 monclient: _reopen_session rank -1 2024-04-26T00:19:51.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb842c640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:19:51.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb842c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbeb00a2410 0x7fbeb01cf2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:51.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb842c640 10 monclient(hunting): picked mon.noname-a con 0x7fbeb00a2410 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:19:51.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb842c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbeb00a27f0 0x7fbeb0098470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:51.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb842c640 10 monclient(hunting): picked mon.noname-c con 0x7fbeb00a27f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:19:51.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb842c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbeb00a38e0 0x7fbeb00989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:51.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb842c640 10 monclient(hunting): picked mon.noname-b con 0x7fbeb00a38e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:19:51.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb842c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:51.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb842c640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb0142cc0 con 0x7fbeb00a2410 2024-04-26T00:19:51.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb842c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:51.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb842c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb0142b40 con 0x7fbeb00a38e0 2024-04-26T00:19:51.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb842c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:51.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb842c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fbeb0142e40 con 0x7fbeb00a27f0 2024-04-26T00:19:51.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb842c640 10 monclient(hunting): _renew_subs 2024-04-26T00:19:51.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb842c640 10 monclient(hunting): authenticate will time out at 956.972806s 2024-04-26T00:19:51.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb69c9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbeb00a38e0 0x7fbeb00989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:51.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb69c9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbeb00a38e0 0x7fbeb00989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50164/0 (socket says 172.21.15.45:50164) 2024-04-26T00:19:51.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb69c9640 1 -- 172.21.15.45:0/1747925848 learned_addr learned my addr 172.21.15.45:0/1747925848 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-26T00:19:51.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb69c9640 10 monclient(hunting): get_auth_request con 0x7fbeb00a38e0 auth_method 0 2024-04-26T00:19:51.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb69c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:51.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb69c9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:51.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb69c9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:51.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb69c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:19:51.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb69c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:19:51.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb69c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:19:51.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb61c8640 1 --2- 172.21.15.45:0/1747925848 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbeb00a2410 0x7fbeb01cf2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:51.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb59c7640 1 --2- 172.21.15.45:0/1747925848 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbeb00a27f0 0x7fbeb0098470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:51.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb61c8640 10 monclient(hunting): get_auth_request con 0x7fbeb00a2410 auth_method 0 2024-04-26T00:19:51.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb61c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:51.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb61c8640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:51.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb61c8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:51.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb59c7640 10 monclient(hunting): get_auth_request con 0x7fbeb00a27f0 auth_method 0 2024-04-26T00:19:51.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb59c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:51.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb59c7640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:51.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb59c7640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:51.115 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb69c9640 10 monclient(hunting): handle_auth_done global_id 4114 payload 824 2024-04-26T00:19:51.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb69c9640 1 -- 172.21.15.45:0/1747925848 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbeb00a27f0 msgr2=0x7fbeb0098470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:51.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb69c9640 1 --2- 172.21.15.45:0/1747925848 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbeb00a27f0 0x7fbeb0098470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:51.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb69c9640 1 -- 172.21.15.45:0/1747925848 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbeb00a2410 msgr2=0x7fbeb01cf2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:51.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb69c9640 1 --2- 172.21.15.45:0/1747925848 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbeb00a2410 0x7fbeb01cf2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:51.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb69c9640 10 monclient: _finish_hunting 0 2024-04-26T00:19:51.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb69c9640 1 monclient: found mon.noname-b 2024-04-26T00:19:51.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb69c9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:19:51.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb69c9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.45:3300/0 2024-04-26T00:19:51.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb69c9640 1 -- 172.21.15.45:0/1747925848 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeb013d590 con 0x7fbeb00a38e0 2024-04-26T00:19:51.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb69c9640 1 -- 172.21.15.45:0/1747925848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbeb00a38e0 msgr2=0x7fbeb00989b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:19:51.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb69c9640 1 --2- 172.21.15.45:0/1747925848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbeb00a38e0 0x7fbeb00989b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fbea000c410 tx=0x7fbea000c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:19:51.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbea6ffd640 1 -- 172.21.15.45:0/1747925848 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea0017070 con 0x7fbeb00a38e0 2024-04-26T00:19:51.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbea6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:51.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbea6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:19:51.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbea6ffd640 10 monclient: dump: 2024-04-26T00:19:51.118 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-04-26T00:19:51.118 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:51.118 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:51.118 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:51.118 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:51.119 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 3 2024-04-26T00:19:51.119 INFO:teuthology.orchestra.run.smithi045.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:51.119 INFO:teuthology.orchestra.run.smithi045.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:51.119 INFO:teuthology.orchestra.run.smithi045.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:51.119 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-26T00:19:51.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbea6ffd640 10 monclient: _finish_auth 0 2024-04-26T00:19:51.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbea6ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:19:51.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbea6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:19:51.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbea6ffd640 1 -- 172.21.15.45:0/1747925848 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbea0005230 con 0x7fbeb00a38e0 2024-04-26T00:19:51.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbea6ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:19:51.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbea6ffd640 1 -- 172.21.15.45:0/1747925848 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea00117a0 con 0x7fbeb00a38e0 2024-04-26T00:19:51.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb842c640 5 monclient: authenticate success, global_id 4114 2024-04-26T00:19:51.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbea6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:51.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbea6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:19:51.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbea6ffd640 10 monclient: dump: 2024-04-26T00:19:51.120 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-04-26T00:19:51.120 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:51.120 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:51.120 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:51.120 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:51.120 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 3 2024-04-26T00:19:51.120 INFO:teuthology.orchestra.run.smithi045.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:51.120 INFO:teuthology.orchestra.run.smithi045.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:51.120 INFO:teuthology.orchestra.run.smithi045.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:51.120 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-26T00:19:51.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb842c640 10 monclient: _renew_subs 2024-04-26T00:19:51.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbea7fff640 4 set_mon_vals no callback set 2024-04-26T00:19:51.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb842c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.45:3300/0 2024-04-26T00:19:51.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb842c640 1 -- 172.21.15.45:0/1747925848 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbeb013f100 con 0x7fbeb00a38e0 2024-04-26T00:19:51.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb842c640 10 monclient: _renew_subs 2024-04-26T00:19:51.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb842c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.45:3300/0 2024-04-26T00:19:51.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.130+0000 7fbeb842c640 1 -- 172.21.15.45:0/1747925848 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbeb013d0f0 con 0x7fbeb00a38e0 2024-04-26T00:19:51.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.134+0000 7fbea6ffd640 1 -- 172.21.15.45:0/1747925848 <== mon.1 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fbea001a070 con 0x7fbeb00a38e0 2024-04-26T00:19:51.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.134+0000 7fbea6ffd640 1 -- 172.21.15.45:0/1747925848 <== mon.1 v2:172.21.15.45:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7fbea0006c00 con 0x7fbeb00a38e0 2024-04-26T00:19:51.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.134+0000 7fbeb842c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:19:51.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.134+0000 7fbeb842c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:19:51.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.134+0000 7fbeb842c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.45:3300/0 2024-04-26T00:19:51.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.134+0000 7fbeb842c640 1 -- 172.21.15.45:0/1747925848 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbeb01cf860 con 0x7fbeb00a38e0 2024-04-26T00:19:51.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.134+0000 7fbea6ffd640 1 -- 172.21.15.45:0/1747925848 <== mon.1 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fbea00067e0 con 0x7fbeb00a38e0 2024-04-26T00:19:51.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.134+0000 7fbea6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:19:51.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.134+0000 7fbea6ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:19:51.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.202+0000 7fbeb842c640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "78fc9fe6-2163-4cb3-9944-3773bd488df8", "id": 2}] 2024-04-26T00:19:51.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.202+0000 7fbeb842c640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "78fc9fe6-2163-4cb3-9944-3773bd488df8", "id": 2}] 2024-04-26T00:19:51.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.202+0000 7fbeb842c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.45:3300/0 2024-04-26T00:19:51.184 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.202+0000 7fbeb842c640 1 -- 172.21.15.45:0/1747925848 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd new", "uuid": "78fc9fe6-2163-4cb3-9944-3773bd488df8", "id": 2} v 0) -- 0x7fbeb013eca0 con 0x7fbeb00a38e0 2024-04-26T00:19:51.305 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-04-26T00:19:51.326+0000 7fc2fd1e0180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T00:19:51.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.466+0000 7fbea6ffd640 1 -- 172.21.15.45:0/1747925848 <== mon.1 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "78fc9fe6-2163-4cb3-9944-3773bd488df8", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7fbea0011b00 con 0x7fbeb00a38e0 2024-04-26T00:19:51.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.466+0000 7fbea6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "78fc9fe6-2163-4cb3-9944-3773bd488df8", "id": 2}] 2024-04-26T00:19:51.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.466+0000 7fbea6ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:19:51.448 INFO:teuthology.orchestra.run.smithi045.stdout:2 2024-04-26T00:19:51.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.470+0000 7fbeb842c640 10 monclient: shutdown 2024-04-26T00:19:51.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.470+0000 7fbeb842c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:19:51.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.470+0000 7fbeb842c640 1 -- 172.21.15.45:0/1747925848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbeb00a38e0 msgr2=0x7fbeb00989b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:51.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.470+0000 7fbeb842c640 1 --2- 172.21.15.45:0/1747925848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbeb00a38e0 0x7fbeb00989b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fbea000c410 tx=0x7fbea000c8d0 comp rx=0 tx=0).stop 2024-04-26T00:19:51.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.470+0000 7fbeb842c640 1 -- 172.21.15.45:0/1747925848 shutdown_connections 2024-04-26T00:19:51.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.470+0000 7fbeb842c640 1 --2- 172.21.15.45:0/1747925848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbeb00a38e0 0x7fbeb00989b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:51.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.470+0000 7fbeb842c640 1 --2- 172.21.15.45:0/1747925848 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbeb00a27f0 0x7fbeb0098470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:51.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.470+0000 7fbeb842c640 1 --2- 172.21.15.45:0/1747925848 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbeb00a2410 0x7fbeb01cf2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:51.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.470+0000 7fbeb842c640 1 -- 172.21.15.45:0/1747925848 >> 172.21.15.45:0/1747925848 conn(0x7fbeb00a2df0 msgr2=0x7fbeb00a40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:19:51.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.470+0000 7fbeb842c640 1 -- 172.21.15.45:0/1747925848 shutdown_connections 2024-04-26T00:19:51.449 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.470+0000 7fbeb842c640 1 -- 172.21.15.45:0/1747925848 wait complete. 2024-04-26T00:19:51.464 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph --cluster ceph osd new 338bb632-eca1-422b-b196-fd6cadb8307c 3 2024-04-26T00:19:51.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.606+0000 7fdac1728640 10 monclient: get_monmap_and_config 2024-04-26T00:19:51.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.606+0000 7fdac1728640 10 monclient: build_initial_monmap 2024-04-26T00:19:51.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.606+0000 7fdac1728640 10 monclient: monmap: 2024-04-26T00:19:51.586 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-04-26T00:19:51.586 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:51.586 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-04-26T00:19:51.611462+0000 2024-04-26T00:19:51.586 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-04-26T00:19:51.611462+0000 2024-04-26T00:19:51.586 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-04-26T00:19:51.586 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 3 2024-04-26T00:19:51.586 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:19:51.586 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:19:51.586 INFO:teuthology.orchestra.run.smithi045.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:19:51.586 INFO:teuthology.orchestra.run.smithi045.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:19:51.586 INFO:teuthology.orchestra.run.smithi045.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:19:51.586 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-26T00:19:51.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.606+0000 7fdac1728640 1 Processor -- start 2024-04-26T00:19:51.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.606+0000 7fdac1728640 1 -- start start 2024-04-26T00:19:51.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.606+0000 7fdac1728640 10 monclient: init 2024-04-26T00:19:51.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.606+0000 7fdac1728640 10 monclient: _reopen_session rank -1 2024-04-26T00:19:51.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.606+0000 7fdac1728640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-04-26T00:19:51.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.606+0000 7fdac1728640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fdabc0a3d90 0x7fdabc09bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:51.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.606+0000 7fdac1728640 10 monclient(hunting): picked mon.noname-a con 0x7fdabc0a3d90 addr v2:172.21.15.37:3300/0 2024-04-26T00:19:51.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.606+0000 7fdac1728640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdabc0a4170 addr v1:172.21.15.37:6789/0 2024-04-26T00:19:51.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.606+0000 7fdac1728640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdabc13c3e0 addr v1:172.21.15.45:6789/0 2024-04-26T00:19:51.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.606+0000 7fdac1728640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdabc0a5150 con 0x7fdabc0a4170 2024-04-26T00:19:51.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.606+0000 7fdac1728640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdabc09b280 con 0x7fdabc13c3e0 2024-04-26T00:19:51.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.606+0000 7fdac1728640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:51.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.606+0000 7fdac1728640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fdabc09b560 con 0x7fdabc0a3d90 2024-04-26T00:19:51.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.606+0000 7fdac1728640 10 monclient(hunting): _renew_subs 2024-04-26T00:19:51.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.606+0000 7fdac1728640 10 monclient(hunting): authenticate will time out at 957.451666s 2024-04-26T00:19:51.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.606+0000 7fdabb7fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fdabc13c3e0 0x7fdabc13e7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:38476/0 (socket says 172.21.15.45:38476) 2024-04-26T00:19:51.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.606+0000 7fdabb7fe640 1 -- 172.21.15.45:0/228579875 learned_addr learned my addr 172.21.15.45:0/228579875 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-26T00:19:51.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.610+0000 7fdabaffd640 1 --2- 172.21.15.45:0/228579875 >> v2:172.21.15.37:3300/0 conn(0x7fdabc0a3d90 0x7fdabc09bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:51.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.610+0000 7fdabaffd640 10 monclient(hunting): get_auth_request con 0x7fdabc0a3d90 auth_method 0 2024-04-26T00:19:51.590 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.610+0000 7fdabaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:51.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.610+0000 7fdabaffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:51.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.610+0000 7fdabaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:51.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.610+0000 7fdabaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:19:51.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.610+0000 7fdabaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:19:51.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.610+0000 7fdabaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:19:51.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.610+0000 7fdabaffd640 10 monclient(hunting): handle_auth_done global_id 4119 payload 274 2024-04-26T00:19:51.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.610+0000 7fdabaffd640 1 -- 172.21.15.45:0/228579875 >> v1:172.21.15.45:6789/0 conn(0x7fdabc13c3e0 legacy=0x7fdabc13e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:19:51.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.610+0000 7fdabaffd640 1 -- 172.21.15.45:0/228579875 >> v1:172.21.15.37:6789/0 conn(0x7fdabc0a4170 legacy=0x7fdabc0a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:19:51.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.610+0000 7fdabaffd640 10 monclient: _finish_hunting 0 2024-04-26T00:19:51.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.610+0000 7fdabaffd640 1 monclient: found mon.noname-a 2024-04-26T00:19:51.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.610+0000 7fdabaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:19:51.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.610+0000 7fdabaffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:19:51.591 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.610+0000 7fdabaffd640 1 -- 172.21.15.45:0/228579875 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdabc06dc60 con 0x7fdabc0a3d90 2024-04-26T00:19:51.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.614+0000 7fdabaffd640 1 -- 172.21.15.45:0/228579875 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdabc0a3d90 msgr2=0x7fdabc09bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:19:51.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.614+0000 7fdabaffd640 1 --2- 172.21.15.45:0/228579875 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdabc0a3d90 0x7fdabc09bf40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fdaac0097c0 tx=0x7fdaac01c2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e06137a49bab7f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:19:51.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.614+0000 7fdaba7fc640 1 -- 172.21.15.45:0/228579875 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaac0b0070 con 0x7fdabc0a3d90 2024-04-26T00:19:51.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.614+0000 7fdaba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:51.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.614+0000 7fdaba7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:19:51.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.614+0000 7fdaba7fc640 10 monclient: dump: 2024-04-26T00:19:51.594 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-04-26T00:19:51.594 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:51.594 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:51.594 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:51.594 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:51.594 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 3 2024-04-26T00:19:51.594 INFO:teuthology.orchestra.run.smithi045.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:51.594 INFO:teuthology.orchestra.run.smithi045.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:51.594 INFO:teuthology.orchestra.run.smithi045.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:51.594 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-26T00:19:51.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.614+0000 7fdaba7fc640 10 monclient: _finish_auth 0 2024-04-26T00:19:51.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.614+0000 7fdaba7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:19:51.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.614+0000 7fdaba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:19:51.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.614+0000 7fdaba7fc640 1 -- 172.21.15.45:0/228579875 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaac0ab600 con 0x7fdabc0a3d90 2024-04-26T00:19:51.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.614+0000 7fdac1728640 5 monclient: authenticate success, global_id 4119 2024-04-26T00:19:51.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.614+0000 7fdac1728640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:19:51.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.614+0000 7fdaba7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:19:51.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.614+0000 7fdac1728640 10 monclient: get_monmap_and_config success 2024-04-26T00:19:51.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.614+0000 7fdaba7fc640 1 -- 172.21.15.45:0/228579875 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaac0ab9d0 con 0x7fdabc0a3d90 2024-04-26T00:19:51.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.614+0000 7fdac1728640 4 set_mon_vals no callback set 2024-04-26T00:19:51.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.614+0000 7fdaba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:51.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.614+0000 7fdaba7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:19:51.595 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.614+0000 7fdaba7fc640 10 monclient: dump: 2024-04-26T00:19:51.595 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-04-26T00:19:51.595 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:51.595 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:51.596 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:51.596 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:51.596 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 3 2024-04-26T00:19:51.596 INFO:teuthology.orchestra.run.smithi045.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:51.596 INFO:teuthology.orchestra.run.smithi045.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:51.596 INFO:teuthology.orchestra.run.smithi045.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:51.596 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-26T00:19:51.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.614+0000 7fdac1728640 10 monclient: shutdown 2024-04-26T00:19:51.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.614+0000 7fdac1728640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:19:51.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.614+0000 7fdac1728640 1 -- 172.21.15.45:0/228579875 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdabc0a3d90 msgr2=0x7fdabc09bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:51.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.614+0000 7fdac1728640 1 --2- 172.21.15.45:0/228579875 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdabc0a3d90 0x7fdabc09bf40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fdaac0097c0 tx=0x7fdaac01c2e0 comp rx=0 tx=0).stop 2024-04-26T00:19:51.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.614+0000 7fdac1728640 1 -- 172.21.15.45:0/228579875 shutdown_connections 2024-04-26T00:19:51.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.614+0000 7fdac1728640 1 --2- 172.21.15.45:0/228579875 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdabc0a3d90 0x7fdabc09bf40 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:51.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.614+0000 7fdac1728640 1 -- 172.21.15.45:0/228579875 >> 172.21.15.45:0/228579875 conn(0x7fdabc0aa0a0 msgr2=0x7fdabc0a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:19:51.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.614+0000 7fdac1728640 1 -- 172.21.15.45:0/228579875 shutdown_connections 2024-04-26T00:19:51.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.614+0000 7fdac1728640 1 -- 172.21.15.45:0/228579875 wait complete. 2024-04-26T00:19:51.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.614+0000 7fdac1728640 10 monclient: build_initial_monmap 2024-04-26T00:19:51.601 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.614+0000 7fdac1728640 10 monclient: monmap: 2024-04-26T00:19:51.601 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-04-26T00:19:51.601 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:19:51.601 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-04-26T00:19:51.601 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-04-26T00:19:51.602 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-04-26T00:19:51.602 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-04-26T00:19:51.602 INFO:teuthology.orchestra.run.smithi045.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:19:51.602 INFO:teuthology.orchestra.run.smithi045.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:19:51.602 INFO:teuthology.orchestra.run.smithi045.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:19:51.602 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-26T00:19:51.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.614+0000 7fdac1728640 1 Processor -- start 2024-04-26T00:19:51.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.614+0000 7fdac1728640 1 -- start start 2024-04-26T00:19:51.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.614+0000 7fdac1728640 10 monclient: init 2024-04-26T00:19:51.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdac1728640 10 monclient: _reopen_session rank -1 2024-04-26T00:19:51.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdac1728640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:19:51.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdac1728640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdabc0a3d90 0x7fdabc1cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:51.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdac1728640 10 monclient(hunting): picked mon.noname-b con 0x7fdabc0a3d90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:19:51.602 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdac1728640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdabc0a4170 0x7fdabc1cf490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:51.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdac1728640 10 monclient(hunting): picked mon.noname-a con 0x7fdabc0a4170 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:19:51.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdac1728640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdabc13c3e0 0x7fdabc138fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:51.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdac1728640 10 monclient(hunting): picked mon.noname-c con 0x7fdabc13c3e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:19:51.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdac1728640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:51.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdac1728640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fdabc142920 con 0x7fdabc0a4170 2024-04-26T00:19:51.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdac1728640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:51.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdac1728640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdabc09c690 con 0x7fdabc0a3d90 2024-04-26T00:19:51.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdac1728640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:51.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdac1728640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fdabc142aa0 con 0x7fdabc13c3e0 2024-04-26T00:19:51.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdac1728640 10 monclient(hunting): _renew_subs 2024-04-26T00:19:51.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdac1728640 10 monclient(hunting): authenticate will time out at 957.460409s 2024-04-26T00:19:51.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdabaffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdabc0a3d90 0x7fdabc1cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:51.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdabaffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdabc0a3d90 0x7fdabc1cecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50180/0 (socket says 172.21.15.45:50180) 2024-04-26T00:19:51.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdabaffd640 1 -- 172.21.15.45:0/3260311642 learned_addr learned my addr 172.21.15.45:0/3260311642 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-26T00:19:51.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdabaffd640 10 monclient(hunting): get_auth_request con 0x7fdabc0a3d90 auth_method 0 2024-04-26T00:19:51.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdabaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:51.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdabaffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:51.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdabaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:51.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdabaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:19:51.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdabaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:19:51.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdabaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:19:51.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdabb7fe640 1 --2- 172.21.15.45:0/3260311642 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdabc13c3e0 0x7fdabc138fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:51.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdabaffd640 10 monclient(hunting): handle_auth_done global_id 4117 payload 824 2024-04-26T00:19:51.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdab3fff640 1 --2- 172.21.15.45:0/3260311642 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdabc0a4170 0x7fdabc1cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:51.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdabaffd640 1 -- 172.21.15.45:0/3260311642 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdabc13c3e0 msgr2=0x7fdabc138fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:51.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdabaffd640 1 --2- 172.21.15.45:0/3260311642 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdabc13c3e0 0x7fdabc138fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:51.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdabaffd640 1 -- 172.21.15.45:0/3260311642 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdabc0a4170 msgr2=0x7fdabc1cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:51.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdabaffd640 1 --2- 172.21.15.45:0/3260311642 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdabc0a4170 0x7fdabc1cf490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:51.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdabaffd640 10 monclient: _finish_hunting 0 2024-04-26T00:19:51.604 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdabaffd640 1 monclient: found mon.noname-b 2024-04-26T00:19:51.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdabaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:19:51.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdabaffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.45:3300/0 2024-04-26T00:19:51.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdabaffd640 1 -- 172.21.15.45:0/3260311642 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdabc13ac10 con 0x7fdabc0a3d90 2024-04-26T00:19:51.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdabb7fe640 10 monclient: get_auth_request con 0x7fdabc13c3e0 auth_method 0 2024-04-26T00:19:51.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdab3fff640 10 monclient: get_auth_request con 0x7fdabc0a4170 auth_method 0 2024-04-26T00:19:51.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdabb7fe640 1 --2- 172.21.15.45:0/3260311642 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdabc13c3e0 0x7fdabc138fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:19:51.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdab3fff640 1 --2- 172.21.15.45:0/3260311642 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdabc0a4170 0x7fdabc1cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:19:51.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdabaffd640 1 -- 172.21.15.45:0/3260311642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdabc0a3d90 msgr2=0x7fdabc1cecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:19:51.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdabaffd640 1 --2- 172.21.15.45:0/3260311642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdabc0a3d90 0x7fdabc1cecc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fdaac0ae040 tx=0x7fdaac01ab20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:19:51.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdab37fe640 1 -- 172.21.15.45:0/3260311642 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaac0b0050 con 0x7fdabc0a3d90 2024-04-26T00:19:51.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdab37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:51.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdab37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:19:51.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdab37fe640 10 monclient: dump: 2024-04-26T00:19:51.605 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-04-26T00:19:51.606 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:51.606 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:51.606 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:51.606 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:51.606 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 3 2024-04-26T00:19:51.606 INFO:teuthology.orchestra.run.smithi045.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:51.606 INFO:teuthology.orchestra.run.smithi045.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:51.606 INFO:teuthology.orchestra.run.smithi045.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:51.606 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-26T00:19:51.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdab37fe640 10 monclient: _finish_auth 0 2024-04-26T00:19:51.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdab37fe640 10 monclient: _check_auth_tickets 2024-04-26T00:19:51.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdab37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:19:51.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdab37fe640 1 -- 172.21.15.45:0/3260311642 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaac0bc070 con 0x7fdabc0a3d90 2024-04-26T00:19:51.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdac1728640 5 monclient: authenticate success, global_id 4117 2024-04-26T00:19:51.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdab37fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:19:51.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdab37fe640 1 -- 172.21.15.45:0/3260311642 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaac0b5d60 con 0x7fdabc0a3d90 2024-04-26T00:19:51.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdab37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:51.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdab37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:19:51.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdab37fe640 10 monclient: dump: 2024-04-26T00:19:51.607 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-04-26T00:19:51.607 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:51.607 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:51.607 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:51.607 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:51.607 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 3 2024-04-26T00:19:51.607 INFO:teuthology.orchestra.run.smithi045.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:51.607 INFO:teuthology.orchestra.run.smithi045.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:51.607 INFO:teuthology.orchestra.run.smithi045.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:51.607 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-26T00:19:51.607 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdac1728640 10 monclient: _renew_subs 2024-04-26T00:19:51.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdac1728640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.45:3300/0 2024-04-26T00:19:51.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdab9ffb640 4 set_mon_vals no callback set 2024-04-26T00:19:51.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdac1728640 1 -- 172.21.15.45:0/3260311642 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdabc120830 con 0x7fdabc0a3d90 2024-04-26T00:19:51.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdac1728640 10 monclient: _renew_subs 2024-04-26T00:19:51.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdac1728640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.45:3300/0 2024-04-26T00:19:51.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdac1728640 1 -- 172.21.15.45:0/3260311642 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdabc1374b0 con 0x7fdabc0a3d90 2024-04-26T00:19:51.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdab37fe640 1 -- 172.21.15.45:0/3260311642 <== mon.1 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fdaac0c1460 con 0x7fdabc0a3d90 2024-04-26T00:19:51.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.618+0000 7fdab37fe640 1 -- 172.21.15.45:0/3260311642 <== mon.1 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7fdaac0c1be0 con 0x7fdabc0a3d90 2024-04-26T00:19:51.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.622+0000 7fdac1728640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:19:51.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.622+0000 7fdac1728640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:19:51.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.622+0000 7fdac1728640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.45:3300/0 2024-04-26T00:19:51.608 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.622+0000 7fdac1728640 1 -- 172.21.15.45:0/3260311642 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdabc04f510 con 0x7fdabc0a3d90 2024-04-26T00:19:51.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.622+0000 7fdab37fe640 1 -- 172.21.15.45:0/3260311642 <== mon.1 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fdaac015ff0 con 0x7fdabc0a3d90 2024-04-26T00:19:51.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.622+0000 7fdab37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:19:51.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.622+0000 7fdab37fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:19:51.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.702+0000 7fdac1728640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "338bb632-eca1-422b-b196-fd6cadb8307c", "id": 3}] 2024-04-26T00:19:51.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.702+0000 7fdac1728640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "338bb632-eca1-422b-b196-fd6cadb8307c", "id": 3}] 2024-04-26T00:19:51.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.702+0000 7fdac1728640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.45:3300/0 2024-04-26T00:19:51.684 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.702+0000 7fdac1728640 1 -- 172.21.15.45:0/3260311642 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd new", "uuid": "338bb632-eca1-422b-b196-fd6cadb8307c", "id": 3} v 0) -- 0x7fdabc075dc0 con 0x7fdabc0a3d90 2024-04-26T00:19:51.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.710+0000 7fdab37fe640 1 -- 172.21.15.45:0/3260311642 <== mon.1 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "338bb632-eca1-422b-b196-fd6cadb8307c", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7fdaac0c1750 con 0x7fdabc0a3d90 2024-04-26T00:19:51.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.710+0000 7fdab37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "338bb632-eca1-422b-b196-fd6cadb8307c", "id": 3}] 2024-04-26T00:19:51.691 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.710+0000 7fdab37fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:19:51.691 INFO:teuthology.orchestra.run.smithi045.stdout:3 2024-04-26T00:19:51.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.710+0000 7fdac1728640 10 monclient: shutdown 2024-04-26T00:19:51.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.710+0000 7fdac1728640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:19:51.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.710+0000 7fdac1728640 1 -- 172.21.15.45:0/3260311642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdabc0a3d90 msgr2=0x7fdabc1cecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:51.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.710+0000 7fdac1728640 1 --2- 172.21.15.45:0/3260311642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdabc0a3d90 0x7fdabc1cecc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fdaac0ae040 tx=0x7fdaac01ab20 comp rx=0 tx=0).stop 2024-04-26T00:19:51.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.710+0000 7fdac1728640 1 -- 172.21.15.45:0/3260311642 shutdown_connections 2024-04-26T00:19:51.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.710+0000 7fdac1728640 1 --2- 172.21.15.45:0/3260311642 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdabc13c3e0 0x7fdabc138fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:51.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.710+0000 7fdac1728640 1 --2- 172.21.15.45:0/3260311642 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdabc0a4170 0x7fdabc1cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:51.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.710+0000 7fdac1728640 1 --2- 172.21.15.45:0/3260311642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdabc0a3d90 0x7fdabc1cecc0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:51.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.710+0000 7fdac1728640 1 -- 172.21.15.45:0/3260311642 >> 172.21.15.45:0/3260311642 conn(0x7fdabc0aa0a0 msgr2=0x7fdabc0a24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:19:51.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.710+0000 7fdac1728640 1 -- 172.21.15.45:0/3260311642 shutdown_connections 2024-04-26T00:19:51.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.710+0000 7fdac1728640 1 -- 172.21.15.45:0/3260311642 wait complete. 2024-04-26T00:19:51.707 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph --cluster ceph osd new ce2e0db7-d486-4cb1-9913-6a4a011e5c8c 4 2024-04-26T00:19:51.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.838+0000 7f31a60c8640 10 monclient: get_monmap_and_config 2024-04-26T00:19:51.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.838+0000 7f31a60c8640 10 monclient: build_initial_monmap 2024-04-26T00:19:51.819 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.838+0000 7f31a60c8640 10 monclient: monmap: 2024-04-26T00:19:51.819 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-04-26T00:19:51.819 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:51.819 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-04-26T00:19:51.844576+0000 2024-04-26T00:19:51.819 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-04-26T00:19:51.844576+0000 2024-04-26T00:19:51.819 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-04-26T00:19:51.820 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 3 2024-04-26T00:19:51.820 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:19:51.820 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:19:51.820 INFO:teuthology.orchestra.run.smithi045.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:19:51.820 INFO:teuthology.orchestra.run.smithi045.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:19:51.820 INFO:teuthology.orchestra.run.smithi045.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:19:51.820 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-26T00:19:51.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.838+0000 7f31a60c8640 1 Processor -- start 2024-04-26T00:19:51.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.838+0000 7f31a60c8640 1 -- start start 2024-04-26T00:19:51.820 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.838+0000 7f31a60c8640 10 monclient: init 2024-04-26T00:19:51.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.838+0000 7f31a60c8640 10 monclient: _reopen_session rank -1 2024-04-26T00:19:51.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.838+0000 7f31a60c8640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-04-26T00:19:51.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.838+0000 7f31a60c8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f31a00a3d90 addr v1:172.21.15.45:6789/0 2024-04-26T00:19:51.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.838+0000 7f31a60c8640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f31a00a4170 0x7f31a00a45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:51.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.838+0000 7f31a60c8640 10 monclient(hunting): picked mon.noname-a con 0x7f31a00a4170 addr v2:172.21.15.37:3300/0 2024-04-26T00:19:51.822 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.842+0000 7f31a60c8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f31a013c3e0 addr v1:172.21.15.37:6789/0 2024-04-26T00:19:51.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.842+0000 7f31a60c8640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31a00a5150 con 0x7f31a013c3e0 2024-04-26T00:19:51.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.842+0000 7f31a60c8640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31a009b280 con 0x7f31a00a3d90 2024-04-26T00:19:51.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.842+0000 7f31a60c8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:51.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.842+0000 7f31a60c8640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f31a009c650 con 0x7f31a00a4170 2024-04-26T00:19:51.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.842+0000 7f31a60c8640 10 monclient(hunting): _renew_subs 2024-04-26T00:19:51.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.842+0000 7f31a60c8640 10 monclient(hunting): authenticate will time out at 957.684070s 2024-04-26T00:19:51.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.842+0000 7f319f7fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f31a00a3d90 0x7f31a009bf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:38484/0 (socket says 172.21.15.45:38484) 2024-04-26T00:19:51.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.842+0000 7f319f7fe640 1 -- 172.21.15.45:0/1590675063 learned_addr learned my addr 172.21.15.45:0/1590675063 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-26T00:19:51.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.842+0000 7f319effd640 1 --2- 172.21.15.45:0/1590675063 >> v2:172.21.15.37:3300/0 conn(0x7f31a00a4170 0x7f31a00a45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:51.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.842+0000 7f319effd640 10 monclient(hunting): get_auth_request con 0x7f31a00a4170 auth_method 0 2024-04-26T00:19:51.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.842+0000 7f319effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:51.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.842+0000 7f319effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:51.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.842+0000 7f319effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:51.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.842+0000 7f319effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:19:51.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.842+0000 7f319effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:19:51.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.842+0000 7f319effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:19:51.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.842+0000 7f319effd640 10 monclient(hunting): handle_auth_done global_id 4122 payload 274 2024-04-26T00:19:51.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.842+0000 7f319effd640 1 -- 172.21.15.45:0/1590675063 >> v1:172.21.15.45:6789/0 conn(0x7f31a00a3d90 legacy=0x7f31a009bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:19:51.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.842+0000 7f319effd640 1 -- 172.21.15.45:0/1590675063 >> v1:172.21.15.37:6789/0 conn(0x7f31a013c3e0 legacy=0x7f31a013e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:19:51.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.842+0000 7f319effd640 10 monclient: _finish_hunting 0 2024-04-26T00:19:51.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.842+0000 7f319effd640 1 monclient: found mon.noname-a 2024-04-26T00:19:51.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.842+0000 7f319effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:19:51.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.842+0000 7f319effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:19:51.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.842+0000 7f319effd640 1 -- 172.21.15.45:0/1590675063 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31a009b560 con 0x7f31a00a4170 2024-04-26T00:19:51.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.846+0000 7f319effd640 1 -- 172.21.15.45:0/1590675063 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31a00a4170 msgr2=0x7f31a00a45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:19:51.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.846+0000 7f319effd640 1 --2- 172.21.15.45:0/1590675063 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31a00a4170 0x7f31a00a45e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f31940097c0 tx=0x7f319401c2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=846ce56280de095a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:19:51.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.846+0000 7f319e7fc640 1 -- 172.21.15.45:0/1590675063 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31940b0070 con 0x7f31a00a4170 2024-04-26T00:19:51.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.846+0000 7f319e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:51.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.846+0000 7f319e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:19:51.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.846+0000 7f319e7fc640 10 monclient: dump: 2024-04-26T00:19:51.827 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-04-26T00:19:51.827 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:51.827 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:51.827 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:51.827 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:51.827 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 3 2024-04-26T00:19:51.827 INFO:teuthology.orchestra.run.smithi045.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:51.827 INFO:teuthology.orchestra.run.smithi045.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:51.827 INFO:teuthology.orchestra.run.smithi045.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:51.827 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-26T00:19:51.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.846+0000 7f319e7fc640 10 monclient: _finish_auth 0 2024-04-26T00:19:51.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.846+0000 7f319e7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:19:51.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.846+0000 7f319e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:19:51.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.846+0000 7f319e7fc640 1 -- 172.21.15.45:0/1590675063 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31940ab600 con 0x7f31a00a4170 2024-04-26T00:19:51.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.846+0000 7f319e7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:19:51.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.846+0000 7f319e7fc640 1 -- 172.21.15.45:0/1590675063 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31940ab9d0 con 0x7f31a00a4170 2024-04-26T00:19:51.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.846+0000 7f319e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:51.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.846+0000 7f319e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:19:51.828 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.846+0000 7f319e7fc640 10 monclient: dump: 2024-04-26T00:19:51.828 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-04-26T00:19:51.828 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:51.828 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:51.828 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:51.828 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:51.828 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 3 2024-04-26T00:19:51.829 INFO:teuthology.orchestra.run.smithi045.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:51.829 INFO:teuthology.orchestra.run.smithi045.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:51.829 INFO:teuthology.orchestra.run.smithi045.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:51.829 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-26T00:19:51.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.846+0000 7f31a60c8640 5 monclient: authenticate success, global_id 4122 2024-04-26T00:19:51.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.846+0000 7f31a60c8640 10 monclient: get_monmap_and_config success 2024-04-26T00:19:51.829 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.846+0000 7f31a60c8640 4 set_mon_vals no callback set 2024-04-26T00:19:51.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.846+0000 7f31a60c8640 10 monclient: shutdown 2024-04-26T00:19:51.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.846+0000 7f31a60c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:19:51.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.846+0000 7f31a60c8640 1 -- 172.21.15.45:0/1590675063 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31a00a4170 msgr2=0x7f31a00a45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:51.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.846+0000 7f31a60c8640 1 --2- 172.21.15.45:0/1590675063 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31a00a4170 0x7f31a00a45e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f31940097c0 tx=0x7f319401c2e0 comp rx=0 tx=0).stop 2024-04-26T00:19:51.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.846+0000 7f31a60c8640 1 -- 172.21.15.45:0/1590675063 shutdown_connections 2024-04-26T00:19:51.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.846+0000 7f31a60c8640 1 --2- 172.21.15.45:0/1590675063 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31a00a4170 0x7f31a00a45e0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:51.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.846+0000 7f31a60c8640 1 -- 172.21.15.45:0/1590675063 >> 172.21.15.45:0/1590675063 conn(0x7f31a00aa0a0 msgr2=0x7f31a00a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:19:51.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.846+0000 7f31a60c8640 1 -- 172.21.15.45:0/1590675063 shutdown_connections 2024-04-26T00:19:51.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.846+0000 7f31a60c8640 1 -- 172.21.15.45:0/1590675063 wait complete. 2024-04-26T00:19:51.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f31a60c8640 10 monclient: build_initial_monmap 2024-04-26T00:19:51.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f31a60c8640 10 monclient: monmap: 2024-04-26T00:19:51.834 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-04-26T00:19:51.834 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:19:51.834 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-04-26T00:19:51.834 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-04-26T00:19:51.834 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-04-26T00:19:51.834 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-04-26T00:19:51.834 INFO:teuthology.orchestra.run.smithi045.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:19:51.834 INFO:teuthology.orchestra.run.smithi045.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:19:51.834 INFO:teuthology.orchestra.run.smithi045.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:19:51.835 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-26T00:19:51.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f31a60c8640 1 Processor -- start 2024-04-26T00:19:51.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f31a60c8640 1 -- start start 2024-04-26T00:19:51.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f31a60c8640 10 monclient: init 2024-04-26T00:19:51.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f31a60c8640 10 monclient: _reopen_session rank -1 2024-04-26T00:19:51.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f31a60c8640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:19:51.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f31a60c8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31a00a3d90 0x7f31a01cf000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:51.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f31a60c8640 10 monclient(hunting): picked mon.noname-b con 0x7f31a00a3d90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:19:51.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f31a60c8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f31a00a4170 0x7f31a01cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:51.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f31a60c8640 10 monclient(hunting): picked mon.noname-c con 0x7f31a00a4170 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:19:51.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f31a60c8640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31a013c3e0 0x7f31a0138d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:51.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f31a60c8640 10 monclient(hunting): picked mon.noname-a con 0x7f31a013c3e0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:19:51.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f31a60c8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:51.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f31a60c8640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f31a0142aa0 con 0x7f31a013c3e0 2024-04-26T00:19:51.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f31a60c8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:51.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f31a60c8640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f31a0142920 con 0x7f31a00a3d90 2024-04-26T00:19:51.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f31a60c8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:51.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f31a60c8640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f31a0142c20 con 0x7f31a00a4170 2024-04-26T00:19:51.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f31a60c8640 10 monclient(hunting): _renew_subs 2024-04-26T00:19:51.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f31a60c8640 10 monclient(hunting): authenticate will time out at 957.693326s 2024-04-26T00:19:51.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f319f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31a00a3d90 0x7f31a01cf000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:51.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f319f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31a00a3d90 0x7f31a01cf000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50188/0 (socket says 172.21.15.45:50188) 2024-04-26T00:19:51.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f319f7fe640 1 -- 172.21.15.45:0/3170108730 learned_addr learned my addr 172.21.15.45:0/3170108730 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-26T00:19:51.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f319f7fe640 10 monclient(hunting): get_auth_request con 0x7f31a00a3d90 auth_method 0 2024-04-26T00:19:51.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f319f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:51.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f319f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:51.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f319f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:51.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f319f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:19:51.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f319f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:19:51.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f319f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:19:51.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f319ffff640 1 --2- 172.21.15.45:0/3170108730 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31a013c3e0 0x7f31a0138d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:51.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f319effd640 1 --2- 172.21.15.45:0/3170108730 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f31a00a4170 0x7f31a01cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:51.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f319ffff640 10 monclient(hunting): get_auth_request con 0x7f31a013c3e0 auth_method 0 2024-04-26T00:19:51.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f319ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:51.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f319ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:51.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f319ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:51.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f319f7fe640 10 monclient(hunting): handle_auth_done global_id 4120 payload 824 2024-04-26T00:19:51.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f319f7fe640 1 -- 172.21.15.45:0/3170108730 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f31a00a4170 msgr2=0x7f31a01cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:51.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f319f7fe640 1 --2- 172.21.15.45:0/3170108730 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f31a00a4170 0x7f31a01cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:51.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f319f7fe640 1 -- 172.21.15.45:0/3170108730 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31a013c3e0 msgr2=0x7f31a0138d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:51.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f319f7fe640 1 --2- 172.21.15.45:0/3170108730 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31a013c3e0 0x7f31a0138d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:51.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f319f7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:19:51.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f319f7fe640 1 monclient: found mon.noname-b 2024-04-26T00:19:51.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f319f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:19:51.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f319f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.45:3300/0 2024-04-26T00:19:51.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f319f7fe640 1 -- 172.21.15.45:0/3170108730 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31a009ada0 con 0x7f31a00a3d90 2024-04-26T00:19:51.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f319effd640 10 monclient: get_auth_request con 0x7f31a00a4170 auth_method 0 2024-04-26T00:19:51.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.850+0000 7f319effd640 1 --2- 172.21.15.45:0/3170108730 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f31a00a4170 0x7f31a01cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:19:51.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.854+0000 7f319f7fe640 1 -- 172.21.15.45:0/3170108730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31a00a3d90 msgr2=0x7f31a01cf000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:19:51.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.854+0000 7f319f7fe640 1 --2- 172.21.15.45:0/3170108730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31a00a3d90 0x7f31a01cf000 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f319000c470 tx=0x7f319000c930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:19:51.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.854+0000 7f3187fff640 1 -- 172.21.15.45:0/3170108730 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3190007070 con 0x7f31a00a3d90 2024-04-26T00:19:51.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.854+0000 7f3187fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:51.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.854+0000 7f3187fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:19:51.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.854+0000 7f3187fff640 10 monclient: dump: 2024-04-26T00:19:51.840 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-04-26T00:19:51.840 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:51.840 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:51.840 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:51.840 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:51.840 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 3 2024-04-26T00:19:51.840 INFO:teuthology.orchestra.run.smithi045.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:51.840 INFO:teuthology.orchestra.run.smithi045.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:51.840 INFO:teuthology.orchestra.run.smithi045.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:51.840 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-26T00:19:51.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.854+0000 7f3187fff640 10 monclient: _finish_auth 0 2024-04-26T00:19:51.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.854+0000 7f3187fff640 10 monclient: _check_auth_tickets 2024-04-26T00:19:51.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.854+0000 7f3187fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:19:51.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.854+0000 7f31a60c8640 5 monclient: authenticate success, global_id 4120 2024-04-26T00:19:51.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.854+0000 7f3187fff640 1 -- 172.21.15.45:0/3170108730 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3190007210 con 0x7f31a00a3d90 2024-04-26T00:19:51.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.854+0000 7f31a60c8640 10 monclient: _renew_subs 2024-04-26T00:19:51.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.854+0000 7f31a60c8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.45:3300/0 2024-04-26T00:19:51.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.854+0000 7f31a60c8640 1 -- 172.21.15.45:0/3170108730 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31a013b2a0 con 0x7f31a00a3d90 2024-04-26T00:19:51.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.854+0000 7f3187fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:19:51.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.854+0000 7f3187fff640 1 -- 172.21.15.45:0/3170108730 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3190007600 con 0x7f31a00a3d90 2024-04-26T00:19:51.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.854+0000 7f3187fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:51.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.854+0000 7f3187fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:19:51.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.854+0000 7f3187fff640 10 monclient: dump: 2024-04-26T00:19:51.841 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-04-26T00:19:51.841 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:51.841 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:51.841 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:51.842 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:51.842 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 3 2024-04-26T00:19:51.842 INFO:teuthology.orchestra.run.smithi045.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:51.842 INFO:teuthology.orchestra.run.smithi045.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:51.842 INFO:teuthology.orchestra.run.smithi045.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:51.842 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-26T00:19:51.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.854+0000 7f31a60c8640 10 monclient: _renew_subs 2024-04-26T00:19:51.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.854+0000 7f31a60c8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.45:3300/0 2024-04-26T00:19:51.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.854+0000 7f319dffb640 4 set_mon_vals no callback set 2024-04-26T00:19:51.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.854+0000 7f31a60c8640 1 -- 172.21.15.45:0/3170108730 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31a0137450 con 0x7f31a00a3d90 2024-04-26T00:19:51.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.854+0000 7f3187fff640 1 -- 172.21.15.45:0/3170108730 <== mon.1 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f319001b400 con 0x7f31a00a3d90 2024-04-26T00:19:51.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.854+0000 7f3187fff640 1 -- 172.21.15.45:0/3170108730 <== mon.1 v2:172.21.15.45:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f3190012c70 con 0x7f31a00a3d90 2024-04-26T00:19:51.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.854+0000 7f31a60c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:19:51.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.854+0000 7f31a60c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:19:51.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.854+0000 7f31a60c8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.45:3300/0 2024-04-26T00:19:51.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.854+0000 7f31a60c8640 1 -- 172.21.15.45:0/3170108730 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31a004f510 con 0x7f31a00a3d90 2024-04-26T00:19:51.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.854+0000 7f3187fff640 1 -- 172.21.15.45:0/3170108730 <== mon.1 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f3190012720 con 0x7f31a00a3d90 2024-04-26T00:19:51.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.854+0000 7f3187fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:19:51.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.854+0000 7f3187fff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:19:51.899 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-04-26T00:19:51.918+0000 7fc2fd1e0180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T00:19:51.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.922+0000 7f31a60c8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ce2e0db7-d486-4cb1-9913-6a4a011e5c8c", "id": 4}] 2024-04-26T00:19:51.901 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.922+0000 7f31a60c8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ce2e0db7-d486-4cb1-9913-6a4a011e5c8c", "id": 4}] 2024-04-26T00:19:51.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.922+0000 7f31a60c8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.45:3300/0 2024-04-26T00:19:51.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.922+0000 7f31a60c8640 1 -- 172.21.15.45:0/3170108730 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ce2e0db7-d486-4cb1-9913-6a4a011e5c8c", "id": 4} v 0) -- 0x7f31a013bf40 con 0x7f31a00a3d90 2024-04-26T00:19:51.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.926+0000 7f3187fff640 1 -- 172.21.15.45:0/3170108730 <== mon.1 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ce2e0db7-d486-4cb1-9913-6a4a011e5c8c", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f3190012900 con 0x7f31a00a3d90 2024-04-26T00:19:51.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.926+0000 7f3187fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ce2e0db7-d486-4cb1-9913-6a4a011e5c8c", "id": 4}] 2024-04-26T00:19:51.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.926+0000 7f3187fff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:19:51.907 INFO:teuthology.orchestra.run.smithi045.stdout:4 2024-04-26T00:19:51.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.926+0000 7f31a60c8640 10 monclient: shutdown 2024-04-26T00:19:51.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.926+0000 7f31a60c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:19:51.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.926+0000 7f31a60c8640 1 -- 172.21.15.45:0/3170108730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31a00a3d90 msgr2=0x7f31a01cf000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:51.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.926+0000 7f31a60c8640 1 --2- 172.21.15.45:0/3170108730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31a00a3d90 0x7f31a01cf000 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f319000c470 tx=0x7f319000c930 comp rx=0 tx=0).stop 2024-04-26T00:19:51.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.926+0000 7f31a60c8640 1 -- 172.21.15.45:0/3170108730 shutdown_connections 2024-04-26T00:19:51.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.926+0000 7f31a60c8640 1 --2- 172.21.15.45:0/3170108730 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31a013c3e0 0x7f31a0138d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:51.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.926+0000 7f31a60c8640 1 --2- 172.21.15.45:0/3170108730 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f31a00a4170 0x7f31a01cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:51.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.926+0000 7f31a60c8640 1 --2- 172.21.15.45:0/3170108730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31a00a3d90 0x7f31a01cf000 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:51.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.930+0000 7f31a60c8640 1 -- 172.21.15.45:0/3170108730 >> 172.21.15.45:0/3170108730 conn(0x7f31a00aa0a0 msgr2=0x7f31a00a3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:19:51.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.930+0000 7f31a60c8640 1 -- 172.21.15.45:0/3170108730 shutdown_connections 2024-04-26T00:19:51.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:51.930+0000 7f31a60c8640 1 -- 172.21.15.45:0/3170108730 wait complete. 2024-04-26T00:19:51.923 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph --cluster ceph osd new a61739cc-41d3-48b1-a471-4092b2edbf1d 5 2024-04-26T00:19:51.959 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-04-26T00:19:51.978+0000 7fc2fd1e0180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T00:19:52.015 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-04-26T00:19:52.034+0000 7fc2fd1e0180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T00:19:52.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.050+0000 7f9a1a0c3640 10 monclient: get_monmap_and_config 2024-04-26T00:19:52.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.050+0000 7f9a1a0c3640 10 monclient: build_initial_monmap 2024-04-26T00:19:52.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.050+0000 7f9a1a0c3640 10 monclient: monmap: 2024-04-26T00:19:52.031 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-04-26T00:19:52.031 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:52.031 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-04-26T00:19:52.056842+0000 2024-04-26T00:19:52.032 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-04-26T00:19:52.056842+0000 2024-04-26T00:19:52.032 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-04-26T00:19:52.032 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 3 2024-04-26T00:19:52.032 INFO:teuthology.orchestra.run.smithi045.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:19:52.032 INFO:teuthology.orchestra.run.smithi045.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:19:52.032 INFO:teuthology.orchestra.run.smithi045.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:19:52.032 INFO:teuthology.orchestra.run.smithi045.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:19:52.032 INFO:teuthology.orchestra.run.smithi045.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:19:52.032 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-26T00:19:52.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.050+0000 7f9a1a0c3640 1 Processor -- start 2024-04-26T00:19:52.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.050+0000 7f9a1a0c3640 1 -- start start 2024-04-26T00:19:52.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.050+0000 7f9a1a0c3640 10 monclient: init 2024-04-26T00:19:52.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a1a0c3640 10 monclient: _reopen_session rank -1 2024-04-26T00:19:52.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a1a0c3640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-04-26T00:19:52.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a1a0c3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9a1406b1d0 addr v1:172.21.15.45:6789/0 2024-04-26T00:19:52.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a1a0c3640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9a1406b5b0 0x7f9a1406b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:52.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a1a0c3640 10 monclient(hunting): picked mon.noname-a con 0x7f9a1406b5b0 addr v2:172.21.15.37:3300/0 2024-04-26T00:19:52.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a1a0c3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a1413c720 0x7f9a1413eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:52.035 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a1a0c3640 10 monclient(hunting): picked mon.noname-b con 0x7f9a1413c720 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:19:52.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a1a0c3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a1409e720 con 0x7f9a1406b1d0 2024-04-26T00:19:52.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a1a0c3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:52.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a1a0c3640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f9a140a29e0 con 0x7f9a1406b5b0 2024-04-26T00:19:52.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a1a0c3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:52.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a1a0c3640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9a1406d730 con 0x7f9a1413c720 2024-04-26T00:19:52.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a1a0c3640 10 monclient(hunting): _renew_subs 2024-04-26T00:19:52.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a1a0c3640 10 monclient(hunting): authenticate will time out at 957.896318s 2024-04-26T00:19:52.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a137fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f9a1406b1d0 0x7f9a1409ebb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.45:38488/0 (socket says 172.21.15.45:38488) 2024-04-26T00:19:52.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a137fe640 1 -- 172.21.15.45:0/2938336359 learned_addr learned my addr 172.21.15.45:0/2938336359 (peer_addr_for_me v1:172.21.15.45:0/0) 2024-04-26T00:19:52.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a13fff640 1 --2- 172.21.15.45:0/2938336359 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a1413c720 0x7f9a1413eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:52.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a12ffd640 1 --2- 172.21.15.45:0/2938336359 >> v2:172.21.15.37:3300/0 conn(0x7f9a1406b5b0 0x7f9a1406b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:52.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a12ffd640 10 monclient(hunting): get_auth_request con 0x7f9a1406b5b0 auth_method 0 2024-04-26T00:19:52.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a12ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:52.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a12ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:52.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:52.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a13fff640 10 monclient(hunting): get_auth_request con 0x7f9a1413c720 auth_method 0 2024-04-26T00:19:52.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:52.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a13fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:52.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a13fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:52.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a12ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:19:52.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a12ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:19:52.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a12ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:19:52.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a13fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:19:52.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a13fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:19:52.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a13fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:19:52.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a12ffd640 10 monclient(hunting): handle_auth_done global_id 4128 payload 274 2024-04-26T00:19:52.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a12ffd640 1 -- 172.21.15.45:0/2938336359 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a1413c720 msgr2=0x7f9a1413eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:19:52.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a12ffd640 1 --2- 172.21.15.45:0/2938336359 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a1413c720 0x7f9a1413eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:52.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a12ffd640 1 -- 172.21.15.45:0/2938336359 >> v1:172.21.15.45:6789/0 conn(0x7f9a1406b1d0 legacy=0x7f9a1409ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:19:52.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a12ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:19:52.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a12ffd640 1 monclient: found mon.noname-a 2024-04-26T00:19:52.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:19:52.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a12ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:19:52.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.054+0000 7f9a12ffd640 1 -- 172.21.15.45:0/2938336359 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a1409de80 con 0x7f9a1406b5b0 2024-04-26T00:19:52.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.058+0000 7f9a12ffd640 1 -- 172.21.15.45:0/2938336359 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9a1406b5b0 msgr2=0x7f9a1406b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:19:52.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.058+0000 7f9a12ffd640 1 --2- 172.21.15.45:0/2938336359 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9a1406b5b0 0x7f9a1406b990 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9a080097c0 tx=0x7f9a0801c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fad962efc081a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:19:52.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.058+0000 7f9a127fc640 1 -- 172.21.15.45:0/2938336359 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a080b0070 con 0x7f9a1406b5b0 2024-04-26T00:19:52.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.058+0000 7f9a127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:52.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.058+0000 7f9a127fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:19:52.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.058+0000 7f9a127fc640 10 monclient: dump: 2024-04-26T00:19:52.042 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-04-26T00:19:52.042 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:52.043 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:52.043 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:52.043 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:52.043 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 3 2024-04-26T00:19:52.043 INFO:teuthology.orchestra.run.smithi045.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:52.043 INFO:teuthology.orchestra.run.smithi045.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:52.043 INFO:teuthology.orchestra.run.smithi045.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:52.043 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-26T00:19:52.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.058+0000 7f9a127fc640 10 monclient: _finish_auth 0 2024-04-26T00:19:52.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.058+0000 7f9a127fc640 10 monclient: _check_auth_tickets 2024-04-26T00:19:52.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.058+0000 7f9a127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:19:52.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.058+0000 7f9a127fc640 1 -- 172.21.15.45:0/2938336359 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a080ab600 con 0x7f9a1406b5b0 2024-04-26T00:19:52.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.058+0000 7f9a1a0c3640 5 monclient: authenticate success, global_id 4128 2024-04-26T00:19:52.043 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.058+0000 7f9a1a0c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:19:52.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.058+0000 7f9a127fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:19:52.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.058+0000 7f9a1a0c3640 10 monclient: get_monmap_and_config success 2024-04-26T00:19:52.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.058+0000 7f9a127fc640 1 -- 172.21.15.45:0/2938336359 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a080ab9d0 con 0x7f9a1406b5b0 2024-04-26T00:19:52.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.058+0000 7f9a1a0c3640 4 set_mon_vals no callback set 2024-04-26T00:19:52.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.058+0000 7f9a127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:52.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.058+0000 7f9a127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:19:52.044 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.058+0000 7f9a127fc640 10 monclient: dump: 2024-04-26T00:19:52.044 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-04-26T00:19:52.044 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:52.044 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:52.044 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:52.044 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:52.044 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 3 2024-04-26T00:19:52.044 INFO:teuthology.orchestra.run.smithi045.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:52.044 INFO:teuthology.orchestra.run.smithi045.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:52.045 INFO:teuthology.orchestra.run.smithi045.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:52.045 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-26T00:19:52.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.058+0000 7f9a1a0c3640 10 monclient: shutdown 2024-04-26T00:19:52.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.058+0000 7f9a1a0c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:19:52.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.058+0000 7f9a1a0c3640 1 -- 172.21.15.45:0/2938336359 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9a1406b5b0 msgr2=0x7f9a1406b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:52.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.058+0000 7f9a1a0c3640 1 --2- 172.21.15.45:0/2938336359 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9a1406b5b0 0x7f9a1406b990 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9a080097c0 tx=0x7f9a0801c320 comp rx=0 tx=0).stop 2024-04-26T00:19:52.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.058+0000 7f9a1a0c3640 1 -- 172.21.15.45:0/2938336359 shutdown_connections 2024-04-26T00:19:52.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.058+0000 7f9a1a0c3640 1 --2- 172.21.15.45:0/2938336359 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a1413c720 0x7f9a1413eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:52.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.058+0000 7f9a1a0c3640 1 --2- 172.21.15.45:0/2938336359 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9a1406b5b0 0x7f9a1406b990 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:52.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.058+0000 7f9a1a0c3640 1 -- 172.21.15.45:0/2938336359 >> 172.21.15.45:0/2938336359 conn(0x7f9a140a2c50 msgr2=0x7f9a140a3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:19:52.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.058+0000 7f9a1a0c3640 1 -- 172.21.15.45:0/2938336359 shutdown_connections 2024-04-26T00:19:52.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.058+0000 7f9a1a0c3640 1 -- 172.21.15.45:0/2938336359 wait complete. 2024-04-26T00:19:52.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a1a0c3640 10 monclient: build_initial_monmap 2024-04-26T00:19:52.050 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a1a0c3640 10 monclient: monmap: 2024-04-26T00:19:52.050 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 0 2024-04-26T00:19:52.051 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:19:52.051 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 0.000000 2024-04-26T00:19:52.051 INFO:teuthology.orchestra.run.smithi045.stderr:created 0.000000 2024-04-26T00:19:52.051 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 0 (unknown) 2024-04-26T00:19:52.051 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 1 2024-04-26T00:19:52.051 INFO:teuthology.orchestra.run.smithi045.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:19:52.051 INFO:teuthology.orchestra.run.smithi045.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:19:52.051 INFO:teuthology.orchestra.run.smithi045.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:19:52.051 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-26T00:19:52.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a1a0c3640 1 Processor -- start 2024-04-26T00:19:52.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a1a0c3640 1 -- start start 2024-04-26T00:19:52.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a1a0c3640 10 monclient: init 2024-04-26T00:19:52.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a1a0c3640 10 monclient: _reopen_session rank -1 2024-04-26T00:19:52.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a1a0c3640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:19:52.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a1a0c3640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9a1406b1d0 0x7f9a14070400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:52.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a1a0c3640 10 monclient(hunting): picked mon.noname-a con 0x7f9a1406b1d0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:19:52.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a1a0c3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a1406b5b0 0x7f9a14070940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:52.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a1a0c3640 10 monclient(hunting): picked mon.noname-c con 0x7f9a1406b5b0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:19:52.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a1a0c3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a1413c720 0x7f9a14070e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:52.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a1a0c3640 10 monclient(hunting): picked mon.noname-b con 0x7f9a1413c720 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:19:52.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a1a0c3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:52.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a1a0c3640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9a141419f0 con 0x7f9a1406b1d0 2024-04-26T00:19:52.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a1a0c3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:52.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a1a0c3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9a14141870 con 0x7f9a1413c720 2024-04-26T00:19:52.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a1a0c3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:52.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a1a0c3640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9a14141b70 con 0x7f9a1406b5b0 2024-04-26T00:19:52.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a1a0c3640 10 monclient(hunting): _renew_subs 2024-04-26T00:19:52.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a1a0c3640 10 monclient(hunting): authenticate will time out at 957.905177s 2024-04-26T00:19:52.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a13fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a1413c720 0x7f9a14070e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:52.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a13fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a1413c720 0x7f9a14070e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50198/0 (socket says 172.21.15.45:50198) 2024-04-26T00:19:52.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a13fff640 1 -- 172.21.15.45:0/1987487512 learned_addr learned my addr 172.21.15.45:0/1987487512 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-04-26T00:19:52.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a13fff640 10 monclient(hunting): get_auth_request con 0x7f9a1413c720 auth_method 0 2024-04-26T00:19:52.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:52.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a13fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:52.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a13fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:52.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a13fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:19:52.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a13fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:19:52.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a13fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:19:52.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a137fe640 1 --2- 172.21.15.45:0/1987487512 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9a1406b1d0 0x7f9a14070400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:52.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a12ffd640 1 --2- 172.21.15.45:0/1987487512 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a1406b5b0 0x7f9a14070940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:52.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a137fe640 10 monclient(hunting): get_auth_request con 0x7f9a1406b1d0 auth_method 0 2024-04-26T00:19:52.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:52.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a137fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:52.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a137fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:52.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a12ffd640 10 monclient(hunting): get_auth_request con 0x7f9a1406b5b0 auth_method 0 2024-04-26T00:19:52.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a12ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:52.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a12ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:52.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:52.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a13fff640 10 monclient(hunting): handle_auth_done global_id 4123 payload 824 2024-04-26T00:19:52.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a13fff640 1 -- 172.21.15.45:0/1987487512 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a1406b5b0 msgr2=0x7f9a14070940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:52.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a13fff640 1 --2- 172.21.15.45:0/1987487512 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a1406b5b0 0x7f9a14070940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:52.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a13fff640 1 -- 172.21.15.45:0/1987487512 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9a1406b1d0 msgr2=0x7f9a14070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:52.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a13fff640 1 --2- 172.21.15.45:0/1987487512 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9a1406b1d0 0x7f9a14070400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:52.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a13fff640 10 monclient: _finish_hunting 0 2024-04-26T00:19:52.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a13fff640 1 monclient: found mon.noname-b 2024-04-26T00:19:52.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:19:52.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a13fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.45:3300/0 2024-04-26T00:19:52.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.062+0000 7f9a13fff640 1 -- 172.21.15.45:0/1987487512 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a1409d770 con 0x7f9a1413c720 2024-04-26T00:19:52.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.066+0000 7f9a13fff640 1 -- 172.21.15.45:0/1987487512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a1413c720 msgr2=0x7f9a14070e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:19:52.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.066+0000 7f9a13fff640 1 --2- 172.21.15.45:0/1987487512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a1413c720 0x7f9a14070e80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c00ef90 tx=0x7f9a0c00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:19:52.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.066+0000 7f99fbfff640 1 -- 172.21.15.45:0/1987487512 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a0c018020 con 0x7f9a1413c720 2024-04-26T00:19:52.055 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.066+0000 7f99fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:52.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.066+0000 7f99fbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:19:52.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.066+0000 7f99fbfff640 10 monclient: dump: 2024-04-26T00:19:52.056 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-04-26T00:19:52.056 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:52.056 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:52.056 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:52.056 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:52.056 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 3 2024-04-26T00:19:52.056 INFO:teuthology.orchestra.run.smithi045.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:52.056 INFO:teuthology.orchestra.run.smithi045.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:52.056 INFO:teuthology.orchestra.run.smithi045.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:52.056 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-26T00:19:52.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.066+0000 7f99fbfff640 10 monclient: _finish_auth 0 2024-04-26T00:19:52.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.066+0000 7f99fbfff640 10 monclient: _check_auth_tickets 2024-04-26T00:19:52.056 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.066+0000 7f99fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:19:52.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.066+0000 7f99fbfff640 1 -- 172.21.15.45:0/1987487512 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a0c0146a0 con 0x7f9a1413c720 2024-04-26T00:19:52.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.066+0000 7f99fbfff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:19:52.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.066+0000 7f99fbfff640 1 -- 172.21.15.45:0/1987487512 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a0c014a90 con 0x7f9a1413c720 2024-04-26T00:19:52.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.066+0000 7f99fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:52.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.066+0000 7f99fbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:19:52.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.066+0000 7f99fbfff640 10 monclient: dump: 2024-04-26T00:19:52.057 INFO:teuthology.orchestra.run.smithi045.stderr:epoch 1 2024-04-26T00:19:52.057 INFO:teuthology.orchestra.run.smithi045.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:52.057 INFO:teuthology.orchestra.run.smithi045.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:52.057 INFO:teuthology.orchestra.run.smithi045.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:52.057 INFO:teuthology.orchestra.run.smithi045.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:52.057 INFO:teuthology.orchestra.run.smithi045.stderr:election_strategy: 3 2024-04-26T00:19:52.057 INFO:teuthology.orchestra.run.smithi045.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:52.057 INFO:teuthology.orchestra.run.smithi045.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:52.058 INFO:teuthology.orchestra.run.smithi045.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:52.058 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-26T00:19:52.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.066+0000 7f9a117fa640 4 set_mon_vals no callback set 2024-04-26T00:19:52.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.066+0000 7f9a1a0c3640 5 monclient: authenticate success, global_id 4123 2024-04-26T00:19:52.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.066+0000 7f9a1a0c3640 10 monclient: _renew_subs 2024-04-26T00:19:52.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.066+0000 7f9a1a0c3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.45:3300/0 2024-04-26T00:19:52.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.066+0000 7f9a1a0c3640 1 -- 172.21.15.45:0/1987487512 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a14090490 con 0x7f9a1413c720 2024-04-26T00:19:52.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.066+0000 7f9a1a0c3640 10 monclient: _renew_subs 2024-04-26T00:19:52.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.066+0000 7f9a1a0c3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.45:3300/0 2024-04-26T00:19:52.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.066+0000 7f9a1a0c3640 1 -- 172.21.15.45:0/1987487512 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a140724e0 con 0x7f9a1413c720 2024-04-26T00:19:52.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.066+0000 7f99fbfff640 1 -- 172.21.15.45:0/1987487512 <== mon.1 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9a0c01b020 con 0x7f9a1413c720 2024-04-26T00:19:52.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.066+0000 7f99fbfff640 1 -- 172.21.15.45:0/1987487512 <== mon.1 v2:172.21.15.45:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f9a0c013a00 con 0x7f9a1413c720 2024-04-26T00:19:52.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.066+0000 7f9a1a0c3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:19:52.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.066+0000 7f9a1a0c3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:19:52.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.066+0000 7f9a1a0c3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.45:3300/0 2024-04-26T00:19:52.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.066+0000 7f9a1a0c3640 1 -- 172.21.15.45:0/1987487512 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a1404f510 con 0x7f9a1413c720 2024-04-26T00:19:52.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.066+0000 7f99fbfff640 1 -- 172.21.15.45:0/1987487512 <== mon.1 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f9a0c022030 con 0x7f9a1413c720 2024-04-26T00:19:52.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.066+0000 7f99fbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:19:52.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.066+0000 7f99fbfff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:19:52.099 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-04-26T00:19:52.118+0000 7fc2fd1e0180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T00:19:52.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.134+0000 7f9a1a0c3640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a61739cc-41d3-48b1-a471-4092b2edbf1d", "id": 5}] 2024-04-26T00:19:52.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.134+0000 7f9a1a0c3640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a61739cc-41d3-48b1-a471-4092b2edbf1d", "id": 5}] 2024-04-26T00:19:52.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.134+0000 7f9a1a0c3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.45:3300/0 2024-04-26T00:19:52.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.134+0000 7f9a1a0c3640 1 -- 172.21.15.45:0/1987487512 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a61739cc-41d3-48b1-a471-4092b2edbf1d", "id": 5} v 0) -- 0x7f9a14072240 con 0x7f9a1413c720 2024-04-26T00:19:52.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.138+0000 7f99fbfff640 1 -- 172.21.15.45:0/1987487512 <== mon.1 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a61739cc-41d3-48b1-a471-4092b2edbf1d", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f9a1409d770 con 0x7f9a1413c720 2024-04-26T00:19:52.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.138+0000 7f99fbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a61739cc-41d3-48b1-a471-4092b2edbf1d", "id": 5}] 2024-04-26T00:19:52.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.138+0000 7f99fbfff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:19:52.118 INFO:teuthology.orchestra.run.smithi045.stdout:5 2024-04-26T00:19:52.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.138+0000 7f9a1a0c3640 10 monclient: shutdown 2024-04-26T00:19:52.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.138+0000 7f9a1a0c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:19:52.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.138+0000 7f9a1a0c3640 1 -- 172.21.15.45:0/1987487512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a1413c720 msgr2=0x7f9a14070e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:52.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.138+0000 7f9a1a0c3640 1 --2- 172.21.15.45:0/1987487512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a1413c720 0x7f9a14070e80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c00ef90 tx=0x7f9a0c00c490 comp rx=0 tx=0).stop 2024-04-26T00:19:52.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.138+0000 7f9a1a0c3640 1 -- 172.21.15.45:0/1987487512 shutdown_connections 2024-04-26T00:19:52.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.138+0000 7f9a1a0c3640 1 --2- 172.21.15.45:0/1987487512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a1413c720 0x7f9a14070e80 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:52.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.138+0000 7f9a1a0c3640 1 --2- 172.21.15.45:0/1987487512 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a1406b5b0 0x7f9a14070940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:52.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.138+0000 7f9a1a0c3640 1 --2- 172.21.15.45:0/1987487512 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9a1406b1d0 0x7f9a14070400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:52.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.138+0000 7f9a1a0c3640 1 -- 172.21.15.45:0/1987487512 >> 172.21.15.45:0/1987487512 conn(0x7f9a140a2c50 msgr2=0x7f9a1409f3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:19:52.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.138+0000 7f9a1a0c3640 1 -- 172.21.15.45:0/1987487512 shutdown_connections 2024-04-26T00:19:52.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T00:19:52.138+0000 7f9a1a0c3640 1 -- 172.21.15.45:0/1987487512 wait complete. 2024-04-26T00:19:52.134 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-26T00:19:52.134 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-26T00:19:52.137 INFO:tasks.ceph.osd.0:Started 2024-04-26T00:19:52.137 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-26T00:19:52.137 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-26T00:19:52.140 INFO:tasks.ceph.osd.1:Started 2024-04-26T00:19:52.140 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-26T00:19:52.140 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-26T00:19:52.142 INFO:tasks.ceph.osd.2:Started 2024-04-26T00:19:52.142 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-26T00:19:52.143 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-26T00:19:52.155 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-04-26T00:19:52.174+0000 7fc2fd1e0180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T00:19:52.157 INFO:tasks.ceph.osd.3:Started 2024-04-26T00:19:52.157 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-26T00:19:52.157 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-26T00:19:52.159 INFO:tasks.ceph.osd.4:Started 2024-04-26T00:19:52.159 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-26T00:19:52.159 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-26T00:19:52.161 INFO:tasks.ceph.osd.5:Started 2024-04-26T00:19:52.161 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T00:19:52.243 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-04-26T00:19:52.262+0000 7fc2fd1e0180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T00:19:52.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f968e931640 10 monclient: get_monmap_and_config 2024-04-26T00:19:52.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f968e931640 10 monclient: build_initial_monmap 2024-04-26T00:19:52.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f968e931640 10 monclient: monmap: 2024-04-26T00:19:52.278 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:19:52.278 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:52.278 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:52.301672+0000 2024-04-26T00:19:52.278 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:52.301672+0000 2024-04-26T00:19:52.278 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:19:52.278 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:19:52.278 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:19:52.279 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:19:52.279 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:19:52.279 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:19:52.279 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:19:52.279 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:19:52.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f968e931640 1 Processor -- start 2024-04-26T00:19:52.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f968e931640 1 -- start start 2024-04-26T00:19:52.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f968e931640 10 monclient: init 2024-04-26T00:19:52.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f968e931640 10 monclient: _reopen_session rank -1 2024-04-26T00:19:52.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f968e931640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-04-26T00:19:52.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f968e931640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9688136da0 0x7f968809bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:52.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f968e931640 10 monclient(hunting): picked mon.noname-c con 0x7f9688136da0 addr v2:172.21.15.45:3300/0 2024-04-26T00:19:52.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f968e931640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f968809c410 addr v1:172.21.15.45:6789/0 2024-04-26T00:19:52.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f968e931640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9688137890 0x7f968813e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:52.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f968e931640 10 monclient(hunting): picked mon.noname-b con 0x7f9688137890 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:19:52.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f968e931640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f968809b4f0 con 0x7f968809c410 2024-04-26T00:19:52.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f968e931640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:52.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f968e931640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f968809bb40 con 0x7f9688136da0 2024-04-26T00:19:52.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f968e931640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:52.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f968e931640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f968809b210 con 0x7f9688137890 2024-04-26T00:19:52.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f968e931640 10 monclient(hunting): _renew_subs 2024-04-26T00:19:52.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f968e931640 10 monclient(hunting): authenticate will time out at 973.606997s 2024-04-26T00:19:52.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f968cece640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9688137890 0x7f968813e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:52.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f968cece640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9688137890 0x7f968813e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:36070/0 (socket says 172.21.15.37:36070) 2024-04-26T00:19:52.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f968cece640 1 -- 172.21.15.37:0/3040653591 learned_addr learned my addr 172.21.15.37:0/3040653591 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:19:52.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f968cece640 10 monclient(hunting): get_auth_request con 0x7f9688137890 auth_method 0 2024-04-26T00:19:52.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f9687fff640 1 --2- 172.21.15.37:0/3040653591 >> v2:172.21.15.45:3300/0 conn(0x7f9688136da0 0x7f968809bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:52.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f968cece640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:52.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f968cece640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:52.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f968cece640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:52.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f9687fff640 10 monclient(hunting): get_auth_request con 0x7f9688136da0 auth_method 0 2024-04-26T00:19:52.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f9687fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:52.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f9687fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:52.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f9687fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:52.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f968cece640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:19:52.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f968cece640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:19:52.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f968cece640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:19:52.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f9687fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:19:52.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f9687fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:19:52.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f9687fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:19:52.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f968cece640 10 monclient(hunting): handle_auth_done global_id 4127 payload 274 2024-04-26T00:19:52.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f968cece640 1 -- 172.21.15.37:0/3040653591 >> v2:172.21.15.45:3300/0 conn(0x7f9688136da0 msgr2=0x7f968809bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:19:52.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f968cece640 1 --2- 172.21.15.37:0/3040653591 >> v2:172.21.15.45:3300/0 conn(0x7f9688136da0 0x7f968809bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:52.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f968cece640 1 -- 172.21.15.37:0/3040653591 >> v1:172.21.15.45:6789/0 conn(0x7f968809c410 legacy=0x7f9688137180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:19:52.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f968cece640 10 monclient: _finish_hunting 0 2024-04-26T00:19:52.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f968cece640 1 monclient: found mon.noname-b 2024-04-26T00:19:52.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f968cece640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:19:52.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f968cece640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:19:52.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.296+0000 7f968cece640 1 -- 172.21.15.37:0/3040653591 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f968809b7d0 con 0x7f9688137890 2024-04-26T00:19:52.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.300+0000 7f968cece640 1 -- 172.21.15.37:0/3040653591 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9688137890 msgr2=0x7f968813e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:19:52.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.300+0000 7f968cece640 1 --2- 172.21.15.37:0/3040653591 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9688137890 0x7f968813e2e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9680009e50 tx=0x7f968001c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8c62b0d5156bc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:19:52.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.300+0000 7f9686ffd640 1 -- 172.21.15.37:0/3040653591 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96800a5750 con 0x7f9688137890 2024-04-26T00:19:52.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.300+0000 7f9686ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:52.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.300+0000 7f9686ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:19:52.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.300+0000 7f9686ffd640 10 monclient: dump: 2024-04-26T00:19:52.288 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:19:52.288 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:52.288 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:52.288 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:52.288 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:52.288 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:19:52.288 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:52.288 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:52.288 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:52.289 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:19:52.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.300+0000 7f9686ffd640 10 monclient: _finish_auth 0 2024-04-26T00:19:52.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.300+0000 7f9686ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:19:52.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.300+0000 7f9686ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:19:52.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f9686ffd640 1 -- 172.21.15.37:0/3040653591 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96800a58f0 con 0x7f9688137890 2024-04-26T00:19:52.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968e931640 5 monclient: authenticate success, global_id 4127 2024-04-26T00:19:52.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968e931640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:19:52.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f9686ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:19:52.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968e931640 10 monclient: get_monmap_and_config success 2024-04-26T00:19:52.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f9686ffd640 1 -- 172.21.15.37:0/3040653591 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96800a5cc0 con 0x7f9688137890 2024-04-26T00:19:52.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968e931640 4 set_mon_vals no callback set 2024-04-26T00:19:52.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f9686ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:52.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f9686ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:19:52.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f9686ffd640 10 monclient: dump: 2024-04-26T00:19:52.289 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:19:52.290 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:52.290 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:52.290 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:52.290 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:52.290 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:19:52.290 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:52.290 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:52.290 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:52.290 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:19:52.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968e931640 10 monclient: shutdown 2024-04-26T00:19:52.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968e931640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:19:52.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968e931640 1 -- 172.21.15.37:0/3040653591 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9688137890 msgr2=0x7f968813e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:52.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968e931640 1 --2- 172.21.15.37:0/3040653591 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9688137890 0x7f968813e2e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9680009e50 tx=0x7f968001c8b0 comp rx=0 tx=0).stop 2024-04-26T00:19:52.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968e931640 1 -- 172.21.15.37:0/3040653591 shutdown_connections 2024-04-26T00:19:52.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968e931640 1 --2- 172.21.15.37:0/3040653591 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9688137890 0x7f968813e2e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:52.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968e931640 1 --2- 172.21.15.37:0/3040653591 >> v2:172.21.15.45:3300/0 conn(0x7f9688136da0 0x7f968809bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:52.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968e931640 1 -- 172.21.15.37:0/3040653591 >> 172.21.15.37:0/3040653591 conn(0x7f96880a5f00 msgr2=0x7f96880a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:19:52.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968e931640 1 -- 172.21.15.37:0/3040653591 shutdown_connections 2024-04-26T00:19:52.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968e931640 1 -- 172.21.15.37:0/3040653591 wait complete. 2024-04-26T00:19:52.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968e931640 10 monclient: build_initial_monmap 2024-04-26T00:19:52.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968e931640 10 monclient: monmap: 2024-04-26T00:19:52.291 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:19:52.291 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:19:52.291 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:19:52.291 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:19:52.291 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:19:52.291 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:19:52.291 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:19:52.291 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:19:52.292 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:19:52.292 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:19:52.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968e931640 1 Processor -- start 2024-04-26T00:19:52.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968e931640 1 -- start start 2024-04-26T00:19:52.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968e931640 10 monclient: init 2024-04-26T00:19:52.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968e931640 10 monclient: _reopen_session rank -1 2024-04-26T00:19:52.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968e931640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:19:52.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968e931640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f968809c410 0x7f96881cea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:52.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968e931640 10 monclient(hunting): picked mon.noname-b con 0x7f968809c410 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:19:52.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968e931640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9688136da0 0x7f9688135ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:52.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968e931640 10 monclient(hunting): picked mon.noname-a con 0x7f9688136da0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:19:52.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968e931640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9688137890 0x7f968813ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:52.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968e931640 10 monclient(hunting): picked mon.noname-c con 0x7f9688137890 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:19:52.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968e931640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:52.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968e931640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f96881414b0 con 0x7f9688136da0 2024-04-26T00:19:52.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968e931640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:52.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968e931640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9688141330 con 0x7f968809c410 2024-04-26T00:19:52.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968e931640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:52.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968e931640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9688141630 con 0x7f9688137890 2024-04-26T00:19:52.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968e931640 10 monclient(hunting): _renew_subs 2024-04-26T00:19:52.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968e931640 10 monclient(hunting): authenticate will time out at 973.614266s 2024-04-26T00:19:52.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968cece640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9688137890 0x7f968813ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:52.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968cece640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9688137890 0x7f968813ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:36078/0 (socket says 172.21.15.37:36078) 2024-04-26T00:19:52.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968cece640 1 -- 172.21.15.37:0/1669470654 learned_addr learned my addr 172.21.15.37:0/1669470654 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:19:52.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968cece640 10 monclient(hunting): get_auth_request con 0x7f9688137890 auth_method 0 2024-04-26T00:19:52.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968cece640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:52.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968cece640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:52.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968cece640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:52.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f96877fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9688136da0 0x7f9688135ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:52.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f96877fe640 10 monclient(hunting): get_auth_request con 0x7f9688136da0 auth_method 0 2024-04-26T00:19:52.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f96877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:52.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f96877fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:52.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f96877fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:52.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f9687fff640 1 --2- 172.21.15.37:0/1669470654 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f968809c410 0x7f96881cea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:52.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968cece640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:19:52.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968cece640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:19:52.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968cece640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:19:52.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f9687fff640 10 monclient(hunting): get_auth_request con 0x7f968809c410 auth_method 0 2024-04-26T00:19:52.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f9687fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:52.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f9687fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:52.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f9687fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:52.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f96877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:19:52.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f96877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:19:52.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f96877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:19:52.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f9687fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:19:52.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f9687fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:19:52.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f9687fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:19:52.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968cece640 10 monclient(hunting): handle_auth_done global_id 4130 payload 824 2024-04-26T00:19:52.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968cece640 1 -- 172.21.15.37:0/1669470654 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f968809c410 msgr2=0x7f96881cea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:52.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968cece640 1 --2- 172.21.15.37:0/1669470654 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f968809c410 0x7f96881cea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:52.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968cece640 1 -- 172.21.15.37:0/1669470654 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9688136da0 msgr2=0x7f9688135ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:52.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968cece640 1 --2- 172.21.15.37:0/1669470654 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9688136da0 0x7f9688135ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:52.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968cece640 10 monclient: _finish_hunting 0 2024-04-26T00:19:52.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968cece640 1 monclient: found mon.noname-c 2024-04-26T00:19:52.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968cece640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:19:52.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968cece640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:19:52.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.304+0000 7f968cece640 1 -- 172.21.15.37:0/1669470654 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9688120830 con 0x7f9688137890 2024-04-26T00:19:52.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.308+0000 7f968cece640 1 -- 172.21.15.37:0/1669470654 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9688137890 msgr2=0x7f968813ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:19:52.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.308+0000 7f968cece640 1 --2- 172.21.15.37:0/1669470654 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9688137890 0x7f968813ad60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f96800af040 tx=0x7f9680019650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:19:52.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.308+0000 7f9684ff9640 1 -- 172.21.15.37:0/1669470654 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96800bc020 con 0x7f9688137890 2024-04-26T00:19:52.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.308+0000 7f9684ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:52.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.308+0000 7f9684ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:19:52.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.308+0000 7f9684ff9640 10 monclient: dump: 2024-04-26T00:19:52.298 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:19:52.298 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:52.298 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:52.298 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:52.298 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:52.298 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:19:52.298 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:52.298 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:52.299 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:52.299 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:19:52.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.308+0000 7f9684ff9640 10 monclient: _finish_auth 0 2024-04-26T00:19:52.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.308+0000 7f9684ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:19:52.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.308+0000 7f9684ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:19:52.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.308+0000 7f9684ff9640 1 -- 172.21.15.37:0/1669470654 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96800b8a90 con 0x7f9688137890 2024-04-26T00:19:52.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.308+0000 7f9684ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:19:52.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.308+0000 7f9684ff9640 1 -- 172.21.15.37:0/1669470654 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96800a5750 con 0x7f9688137890 2024-04-26T00:19:52.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.308+0000 7f9684ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:52.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.308+0000 7f9684ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:19:52.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.308+0000 7f9684ff9640 10 monclient: dump: 2024-04-26T00:19:52.299 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:19:52.299 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:52.299 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:52.299 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:52.300 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:52.300 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:19:52.300 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:52.300 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:52.300 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:52.300 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:19:52.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.308+0000 7f96867fc640 4 set_mon_vals no callback set 2024-04-26T00:19:52.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.308+0000 7f968e931640 5 monclient: authenticate success, global_id 4130 2024-04-26T00:19:52.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.308+0000 7f968e931640 10 monclient: _renew_subs 2024-04-26T00:19:52.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.308+0000 7f968e931640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:19:52.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.308+0000 7f968e931640 1 -- 172.21.15.37:0/1669470654 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f968809aca0 con 0x7f9688137890 2024-04-26T00:19:52.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.308+0000 7f968e931640 10 monclient: _renew_subs 2024-04-26T00:19:52.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.308+0000 7f968e931640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:19:52.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.308+0000 7f968e931640 1 -- 172.21.15.37:0/1669470654 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9688120550 con 0x7f9688137890 2024-04-26T00:19:52.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.308+0000 7f9684ff9640 1 -- 172.21.15.37:0/1669470654 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f96800bf020 con 0x7f9688137890 2024-04-26T00:19:52.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.308+0000 7f9684ff9640 1 -- 172.21.15.37:0/1669470654 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f96800b6b20 con 0x7f9688137890 2024-04-26T00:19:52.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.308+0000 7f968e931640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:19:52.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.308+0000 7f968e931640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:19:52.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.308+0000 7f968e931640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:19:52.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.308+0000 7f968e931640 1 -- 172.21.15.37:0/1669470654 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9688066f40 con 0x7f9688137890 2024-04-26T00:19:52.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.312+0000 7f9684ff9640 1 -- 172.21.15.37:0/1669470654 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f96800c6030 con 0x7f9688137890 2024-04-26T00:19:52.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.312+0000 7f9684ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:19:52.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.312+0000 7f9684ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:19:52.301 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-04-26T00:19:52.318+0000 7fc2fd1e0180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T00:19:52.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.368+0000 7f968e931640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-26T00:19:52.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.368+0000 7f968e931640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-26T00:19:52.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.368+0000 7f968e931640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:19:52.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.368+0000 7f968e931640 1 -- 172.21.15.37:0/1669470654 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f968813d2e0 con 0x7f9688137890 2024-04-26T00:19:52.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.368+0000 7f9684ff9640 1 -- 172.21.15.37:0/1669470654 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (secure 0 0 0) 0x7f96800165c0 con 0x7f9688137890 2024-04-26T00:19:52.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.368+0000 7f9684ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-26T00:19:52.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.368+0000 7f9684ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:19:52.351 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:19:52.351 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":8,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","created":"2024-04-26T00:19:50.025527+0000","modified":"2024-04-26T00:19:52.138726+0000","last_up_change":"0.000000","last_in_change":"2024-04-26T00:19:52.138726+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a5d4b9d3-5499-491f-a123-85d73de452af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cb4583fd-a0fa-4559-9f99-5a3e02b5818c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78fc9fe6-2163-4cb3-9944-3773bd488df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"338bb632-eca1-422b-b196-fd6cadb8307c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ce2e0db7-d486-4cb1-9913-6a4a011e5c8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a61739cc-41d3-48b1-a471-4092b2edbf1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T00:19:52.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.372+0000 7f968e931640 10 monclient: shutdown 2024-04-26T00:19:52.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.372+0000 7f968e931640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:19:52.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.372+0000 7f968e931640 1 -- 172.21.15.37:0/1669470654 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9688137890 msgr2=0x7f968813ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:52.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.372+0000 7f968e931640 1 --2- 172.21.15.37:0/1669470654 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9688137890 0x7f968813ad60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f96800af040 tx=0x7f9680019650 comp rx=0 tx=0).stop 2024-04-26T00:19:52.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.372+0000 7f968e931640 1 -- 172.21.15.37:0/1669470654 shutdown_connections 2024-04-26T00:19:52.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.372+0000 7f968e931640 1 --2- 172.21.15.37:0/1669470654 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9688137890 0x7f968813ad60 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:52.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.372+0000 7f968e931640 1 --2- 172.21.15.37:0/1669470654 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9688136da0 0x7f9688135ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:52.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.372+0000 7f968e931640 1 --2- 172.21.15.37:0/1669470654 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f968809c410 0x7f96881cea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:52.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.372+0000 7f968e931640 1 -- 172.21.15.37:0/1669470654 >> 172.21.15.37:0/1669470654 conn(0x7f96880a5f00 msgr2=0x7f96880a7200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:19:52.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.372+0000 7f968e931640 1 -- 172.21.15.37:0/1669470654 shutdown_connections 2024-04-26T00:19:52.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:19:52.372+0000 7f968e931640 1 -- 172.21.15.37:0/1669470654 wait complete. 2024-04-26T00:19:52.368 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-26T00:19:52.369 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-26T00:19:52.457 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-04-26T00:19:52.478+0000 7fc2fd1e0180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T00:19:52.522 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-04-26T00:19:52.542+0000 7fc2fd1e0180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T00:19:52.761 INFO:tasks.ceph.osd.2.smithi037.stderr:2024-04-26T00:19:52.780+0000 7f24a4f927c0 -1 Falling back to public interface 2024-04-26T00:19:52.777 INFO:tasks.ceph.osd.1.smithi037.stderr:2024-04-26T00:19:52.796+0000 7f10e57957c0 -1 Falling back to public interface 2024-04-26T00:19:52.778 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T00:19:52.796+0000 7f0f99b4c7c0 -1 Falling back to public interface 2024-04-26T00:19:52.789 INFO:tasks.ceph.osd.3.smithi045.stderr:2024-04-26T00:19:52.810+0000 7f7d840157c0 -1 Falling back to public interface 2024-04-26T00:19:52.801 INFO:tasks.ceph.osd.4.smithi045.stderr:2024-04-26T00:19:52.822+0000 7fb9570547c0 -1 Falling back to public interface 2024-04-26T00:19:52.805 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T00:19:52.826+0000 7f55ad81e7c0 -1 Falling back to public interface 2024-04-26T00:19:52.973 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-26T00:19:53.092 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e4006a640 10 monclient: get_monmap_and_config 2024-04-26T00:19:53.092 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e4006a640 10 monclient: build_initial_monmap 2024-04-26T00:19:53.093 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e4006a640 10 monclient: monmap: 2024-04-26T00:19:53.093 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-04-26T00:19:53.093 INFO:teuthology.misc.health.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:53.093 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-04-26T00:19:53.116035+0000 2024-04-26T00:19:53.093 INFO:teuthology.misc.health.smithi037.stderr:created 2024-04-26T00:19:53.116035+0000 2024-04-26T00:19:53.093 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:19:53.093 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-04-26T00:19:53.093 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:19:53.093 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:19:53.093 INFO:teuthology.misc.health.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:19:53.093 INFO:teuthology.misc.health.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:19:53.093 INFO:teuthology.misc.health.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:19:53.093 INFO:teuthology.misc.health.smithi037.stderr: 2024-04-26T00:19:53.093 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e4006a640 1 Processor -- start 2024-04-26T00:19:53.093 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e4006a640 1 -- start start 2024-04-26T00:19:53.094 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e4006a640 10 monclient: init 2024-04-26T00:19:53.096 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e4006a640 10 monclient: _reopen_session rank -1 2024-04-26T00:19:53.096 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e4006a640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-04-26T00:19:53.096 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e4006a640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7e3809f410 0x7f7e3809f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:53.096 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e4006a640 10 monclient(hunting): picked mon.noname-c con 0x7f7e3809f410 addr v2:172.21.15.45:3300/0 2024-04-26T00:19:53.096 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e4006a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7e380a0310 addr v1:172.21.15.37:6789/0 2024-04-26T00:19:53.096 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e4006a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7e380a6380 0x7f7e381376d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:53.096 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e4006a640 10 monclient(hunting): picked mon.noname-b con 0x7f7e380a6380 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:19:53.097 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e4006a640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e380a7080 con 0x7f7e380a0310 2024-04-26T00:19:53.097 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e4006a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:53.097 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e4006a640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f7e38067ce0 con 0x7f7e3809f410 2024-04-26T00:19:53.097 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e4006a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:53.097 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e4006a640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f7e380a0cc0 con 0x7f7e380a6380 2024-04-26T00:19:53.097 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e4006a640 10 monclient(hunting): _renew_subs 2024-04-26T00:19:53.097 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e4006a640 10 monclient(hunting): authenticate will time out at 974.421450s 2024-04-26T00:19:53.097 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e3d605640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f7e380a0310 0x7f7e38136fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:34098/0 (socket says 172.21.15.37:34098) 2024-04-26T00:19:53.097 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e3e607640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7e380a6380 0x7f7e381376d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:53.097 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e3d605640 1 -- 172.21.15.37:0/829608059 learned_addr learned my addr 172.21.15.37:0/829608059 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:19:53.097 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e3e607640 10 monclient(hunting): get_auth_request con 0x7f7e380a6380 auth_method 0 2024-04-26T00:19:53.097 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e3e607640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:53.097 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e3e607640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:53.097 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e3e607640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:53.098 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e3e607640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:19:53.098 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e3e607640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:19:53.098 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e3e607640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:19:53.098 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e3de06640 1 --2- 172.21.15.37:0/829608059 >> v2:172.21.15.45:3300/0 conn(0x7f7e3809f410 0x7f7e3809f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:53.098 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e3de06640 10 monclient(hunting): get_auth_request con 0x7f7e3809f410 auth_method 0 2024-04-26T00:19:53.098 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e3de06640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:53.098 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e3de06640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:53.098 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e3de06640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:53.098 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e3e607640 10 monclient(hunting): handle_auth_done global_id 4133 payload 274 2024-04-26T00:19:53.098 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e3e607640 1 -- 172.21.15.37:0/829608059 >> v2:172.21.15.45:3300/0 conn(0x7f7e3809f410 msgr2=0x7f7e3809f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:19:53.098 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e3e607640 1 --2- 172.21.15.37:0/829608059 >> v2:172.21.15.45:3300/0 conn(0x7f7e3809f410 0x7f7e3809f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:53.098 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e3e607640 1 -- 172.21.15.37:0/829608059 >> v1:172.21.15.37:6789/0 conn(0x7f7e380a0310 legacy=0x7f7e38136fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:19:53.098 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e3e607640 10 monclient: _finish_hunting 0 2024-04-26T00:19:53.098 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e3e607640 1 monclient: found mon.noname-b 2024-04-26T00:19:53.098 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e3e607640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:19:53.099 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e3e607640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:19:53.099 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.112+0000 7f7e3e607640 1 -- 172.21.15.37:0/829608059 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e3809fd30 con 0x7f7e380a6380 2024-04-26T00:19:53.104 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.116+0000 7f7e3e607640 1 -- 172.21.15.37:0/829608059 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7e380a6380 msgr2=0x7f7e381376d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:19:53.105 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.116+0000 7f7e3e607640 1 --2- 172.21.15.37:0/829608059 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7e380a6380 0x7f7e381376d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7e30009d90 tx=0x7f7e3001c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=102f7fd7fc26ceb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:19:53.105 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.116+0000 7f7e3ce04640 1 -- 172.21.15.37:0/829608059 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e300b2070 con 0x7f7e380a6380 2024-04-26T00:19:53.105 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.116+0000 7f7e3ce04640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:53.105 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.116+0000 7f7e3ce04640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:19:53.105 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.116+0000 7f7e3ce04640 10 monclient: dump: 2024-04-26T00:19:53.105 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-04-26T00:19:53.105 INFO:teuthology.misc.health.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:53.105 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:53.105 INFO:teuthology.misc.health.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:53.105 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:53.105 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-04-26T00:19:53.105 INFO:teuthology.misc.health.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:53.105 INFO:teuthology.misc.health.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:53.105 INFO:teuthology.misc.health.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:53.106 INFO:teuthology.misc.health.smithi037.stderr: 2024-04-26T00:19:53.106 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.116+0000 7f7e3ce04640 10 monclient: _finish_auth 0 2024-04-26T00:19:53.106 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.116+0000 7f7e3ce04640 10 monclient: _check_auth_tickets 2024-04-26T00:19:53.106 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.116+0000 7f7e3ce04640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:19:53.106 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.116+0000 7f7e3ce04640 1 -- 172.21.15.37:0/829608059 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e300a5810 con 0x7f7e380a6380 2024-04-26T00:19:53.106 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.116+0000 7f7e4006a640 5 monclient: authenticate success, global_id 4133 2024-04-26T00:19:53.106 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.116+0000 7f7e4006a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:19:53.106 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.116+0000 7f7e3ce04640 10 monclient: handle_config config(0 keys) 2024-04-26T00:19:53.106 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.116+0000 7f7e4006a640 10 monclient: get_monmap_and_config success 2024-04-26T00:19:53.106 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.116+0000 7f7e4006a640 4 set_mon_vals no callback set 2024-04-26T00:19:53.106 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.116+0000 7f7e3ce04640 1 -- 172.21.15.37:0/829608059 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e300a5be0 con 0x7f7e380a6380 2024-04-26T00:19:53.106 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.116+0000 7f7e3ce04640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:53.106 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.116+0000 7f7e3ce04640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:19:53.106 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.116+0000 7f7e3ce04640 10 monclient: dump: 2024-04-26T00:19:53.107 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-04-26T00:19:53.107 INFO:teuthology.misc.health.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:53.107 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:53.107 INFO:teuthology.misc.health.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:53.107 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:53.107 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-04-26T00:19:53.107 INFO:teuthology.misc.health.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:53.107 INFO:teuthology.misc.health.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:53.107 INFO:teuthology.misc.health.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:53.107 INFO:teuthology.misc.health.smithi037.stderr: 2024-04-26T00:19:53.107 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.116+0000 7f7e4006a640 10 monclient: shutdown 2024-04-26T00:19:53.107 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.116+0000 7f7e4006a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:19:53.107 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.116+0000 7f7e4006a640 1 -- 172.21.15.37:0/829608059 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7e380a6380 msgr2=0x7f7e381376d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:53.107 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.116+0000 7f7e4006a640 1 --2- 172.21.15.37:0/829608059 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7e380a6380 0x7f7e381376d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7e30009d90 tx=0x7f7e3001c880 comp rx=0 tx=0).stop 2024-04-26T00:19:53.108 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.116+0000 7f7e4006a640 1 -- 172.21.15.37:0/829608059 shutdown_connections 2024-04-26T00:19:53.108 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.116+0000 7f7e4006a640 1 --2- 172.21.15.37:0/829608059 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7e380a6380 0x7f7e381376d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:53.108 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.116+0000 7f7e4006a640 1 --2- 172.21.15.37:0/829608059 >> v2:172.21.15.45:3300/0 conn(0x7f7e3809f410 0x7f7e3809f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:53.108 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.116+0000 7f7e4006a640 1 -- 172.21.15.37:0/829608059 >> 172.21.15.37:0/829608059 conn(0x7f7e380a7590 msgr2=0x7f7e380a79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:19:53.108 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.116+0000 7f7e4006a640 1 -- 172.21.15.37:0/829608059 shutdown_connections 2024-04-26T00:19:53.108 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.116+0000 7f7e4006a640 1 -- 172.21.15.37:0/829608059 wait complete. 2024-04-26T00:19:53.108 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.116+0000 7f7e4006a640 10 monclient: build_initial_monmap 2024-04-26T00:19:53.108 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.116+0000 7f7e4006a640 10 monclient: monmap: 2024-04-26T00:19:53.108 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-04-26T00:19:53.108 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:19:53.108 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-04-26T00:19:53.108 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-04-26T00:19:53.108 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:19:53.108 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-04-26T00:19:53.108 INFO:teuthology.misc.health.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:19:53.109 INFO:teuthology.misc.health.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:19:53.109 INFO:teuthology.misc.health.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:19:53.109 INFO:teuthology.misc.health.smithi037.stderr: 2024-04-26T00:19:53.109 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e4006a640 1 Processor -- start 2024-04-26T00:19:53.109 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e4006a640 1 -- start start 2024-04-26T00:19:53.109 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e4006a640 10 monclient: init 2024-04-26T00:19:53.109 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e4006a640 10 monclient: _reopen_session rank -1 2024-04-26T00:19:53.109 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e4006a640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:19:53.109 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e4006a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e3809f410 0x7f7e3813e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:53.109 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e4006a640 10 monclient(hunting): picked mon.noname-a con 0x7f7e3809f410 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:19:53.109 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e4006a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7e380a0310 0x7f7e3813e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:53.109 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e4006a640 10 monclient(hunting): picked mon.noname-c con 0x7f7e380a0310 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:19:53.109 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e4006a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e380a6380 0x7f7e3806fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:53.109 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e4006a640 10 monclient(hunting): picked mon.noname-b con 0x7f7e380a6380 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:19:53.109 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e4006a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:53.109 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e4006a640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f7e38141d30 con 0x7f7e3809f410 2024-04-26T00:19:53.110 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e4006a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:53.110 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e4006a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7e38137e20 con 0x7f7e380a6380 2024-04-26T00:19:53.110 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e4006a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:53.110 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e4006a640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f7e38141eb0 con 0x7f7e380a0310 2024-04-26T00:19:53.110 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e4006a640 10 monclient(hunting): _renew_subs 2024-04-26T00:19:53.110 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e4006a640 10 monclient(hunting): authenticate will time out at 974.428876s 2024-04-26T00:19:53.110 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e3de06640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e3809f410 0x7f7e3813e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:53.110 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e3d605640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7e380a0310 0x7f7e3813e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:53.111 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e3de06640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e3809f410 0x7f7e3813e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:49010/0 (socket says 172.21.15.37:49010) 2024-04-26T00:19:53.111 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e3de06640 1 -- 172.21.15.37:0/2443442596 learned_addr learned my addr 172.21.15.37:0/2443442596 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:19:53.111 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e3d605640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7e380a0310 0x7f7e3813e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:36096/0 (socket says 172.21.15.37:36096) 2024-04-26T00:19:53.111 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e3de06640 10 monclient(hunting): get_auth_request con 0x7f7e3809f410 auth_method 0 2024-04-26T00:19:53.111 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e3de06640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:53.111 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e3de06640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:53.111 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e3de06640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:53.111 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e3d605640 10 monclient(hunting): get_auth_request con 0x7f7e380a0310 auth_method 0 2024-04-26T00:19:53.111 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e3d605640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:53.111 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e3d605640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:53.112 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e3d605640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:53.112 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e3d605640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:19:53.112 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e3d605640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:19:53.112 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e3d605640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:19:53.112 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e3de06640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:19:53.112 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e3de06640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:19:53.112 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e3de06640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:19:53.112 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e3d605640 10 monclient(hunting): handle_auth_done global_id 4136 payload 824 2024-04-26T00:19:53.112 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e3e607640 1 --2- 172.21.15.37:0/2443442596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e380a6380 0x7f7e3806fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:53.112 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e3d605640 1 -- 172.21.15.37:0/2443442596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e380a6380 msgr2=0x7f7e3806fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:53.112 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e3d605640 1 --2- 172.21.15.37:0/2443442596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e380a6380 0x7f7e3806fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:53.112 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e3d605640 1 -- 172.21.15.37:0/2443442596 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e3809f410 msgr2=0x7f7e3813e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:53.112 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e3d605640 1 --2- 172.21.15.37:0/2443442596 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e3809f410 0x7f7e3813e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:53.112 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e3d605640 10 monclient: _finish_hunting 0 2024-04-26T00:19:53.113 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e3d605640 1 monclient: found mon.noname-c 2024-04-26T00:19:53.113 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e3d605640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:19:53.113 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e3d605640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:19:53.113 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e3d605640 1 -- 172.21.15.37:0/2443442596 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e38138950 con 0x7f7e380a0310 2024-04-26T00:19:53.113 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e3e607640 10 monclient: get_auth_request con 0x7f7e380a6380 auth_method 0 2024-04-26T00:19:53.113 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e3de06640 1 --2- 172.21.15.37:0/2443442596 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e3809f410 0x7f7e3813e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:19:53.113 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e3e607640 1 --2- 172.21.15.37:0/2443442596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e380a6380 0x7f7e3806fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:19:53.113 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e3d605640 1 -- 172.21.15.37:0/2443442596 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7e380a0310 msgr2=0x7f7e3813e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:19:53.113 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e3d605640 1 --2- 172.21.15.37:0/2443442596 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7e380a0310 0x7f7e3813e6e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7e2800c440 tx=0x7f7e2800c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:19:53.113 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e2e7fc640 1 -- 172.21.15.37:0/2443442596 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e28017070 con 0x7f7e380a0310 2024-04-26T00:19:53.113 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e2e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:53.113 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e2e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:19:53.113 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e2e7fc640 10 monclient: dump: 2024-04-26T00:19:53.113 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-04-26T00:19:53.113 INFO:teuthology.misc.health.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:53.114 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:53.114 INFO:teuthology.misc.health.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:53.114 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:53.114 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-04-26T00:19:53.114 INFO:teuthology.misc.health.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:53.114 INFO:teuthology.misc.health.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:53.114 INFO:teuthology.misc.health.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:53.114 INFO:teuthology.misc.health.smithi037.stderr: 2024-04-26T00:19:53.114 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e2e7fc640 10 monclient: _finish_auth 0 2024-04-26T00:19:53.114 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e2e7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:19:53.114 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e2e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:19:53.114 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e2e7fc640 1 -- 172.21.15.37:0/2443442596 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e280070d0 con 0x7f7e380a0310 2024-04-26T00:19:53.114 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e4006a640 5 monclient: authenticate success, global_id 4136 2024-04-26T00:19:53.114 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e4006a640 10 monclient: _renew_subs 2024-04-26T00:19:53.114 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e4006a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:19:53.115 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e4006a640 1 -- 172.21.15.37:0/2443442596 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e38138c30 con 0x7f7e380a0310 2024-04-26T00:19:53.115 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e2e7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:19:53.115 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e2e7fc640 1 -- 172.21.15.37:0/2443442596 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e280074c0 con 0x7f7e380a0310 2024-04-26T00:19:53.115 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e2e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:53.115 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e2e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:19:53.115 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e2e7fc640 10 monclient: dump: 2024-04-26T00:19:53.115 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-04-26T00:19:53.115 INFO:teuthology.misc.health.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:53.115 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:53.115 INFO:teuthology.misc.health.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:53.115 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:53.115 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-04-26T00:19:53.115 INFO:teuthology.misc.health.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:53.115 INFO:teuthology.misc.health.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:53.115 INFO:teuthology.misc.health.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:53.116 INFO:teuthology.misc.health.smithi037.stderr: 2024-04-26T00:19:53.116 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e2f7fe640 4 set_mon_vals no callback set 2024-04-26T00:19:53.116 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e4006a640 10 monclient: _renew_subs 2024-04-26T00:19:53.116 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e4006a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:19:53.116 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.120+0000 7f7e4006a640 1 -- 172.21.15.37:0/2443442596 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e380716b0 con 0x7f7e380a0310 2024-04-26T00:19:53.116 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.124+0000 7f7e2e7fc640 1 -- 172.21.15.37:0/2443442596 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7e2801a070 con 0x7f7e380a0310 2024-04-26T00:19:53.116 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.124+0000 7f7e2e7fc640 1 -- 172.21.15.37:0/2443442596 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f7e2801ca30 con 0x7f7e380a0310 2024-04-26T00:19:53.116 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.124+0000 7f7e4006a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:19:53.116 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.124+0000 7f7e4006a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:19:53.116 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.124+0000 7f7e4006a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:19:53.116 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.124+0000 7f7e4006a640 1 -- 172.21.15.37:0/2443442596 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e38062e50 con 0x7f7e380a0310 2024-04-26T00:19:53.117 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.124+0000 7f7e2e7fc640 1 -- 172.21.15.37:0/2443442596 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f7e28021020 con 0x7f7e380a0310 2024-04-26T00:19:53.117 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.124+0000 7f7e2e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:19:53.117 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.124+0000 7f7e2e7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:19:53.156 INFO:tasks.ceph.mgr.x.smithi045.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-26T00:19:53.157 INFO:tasks.ceph.mgr.x.smithi045.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-26T00:19:53.157 INFO:tasks.ceph.mgr.x.smithi045.stderr: from numpy import show_config as show_numpy_config 2024-04-26T00:19:53.165 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.184+0000 7f7e4006a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-26T00:19:53.165 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.184+0000 7f7e4006a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-26T00:19:53.165 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.184+0000 7f7e4006a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:19:53.165 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.184+0000 7f7e4006a640 1 -- 172.21.15.37:0/2443442596 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7e3813ec20 con 0x7f7e380a0310 2024-04-26T00:19:53.165 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.184+0000 7f7e2e7fc640 1 -- 172.21.15.37:0/2443442596 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (secure 0 0 0) 0x7f7e280076a0 con 0x7f7e380a0310 2024-04-26T00:19:53.166 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.184+0000 7f7e2e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-26T00:19:53.166 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.184+0000 7f7e2e7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:19:53.166 INFO:teuthology.misc.health.smithi037.stdout: 2024-04-26T00:19:53.166 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":8,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","created":"2024-04-26T00:19:50.025527+0000","modified":"2024-04-26T00:19:52.138726+0000","last_up_change":"0.000000","last_in_change":"2024-04-26T00:19:52.138726+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a5d4b9d3-5499-491f-a123-85d73de452af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cb4583fd-a0fa-4559-9f99-5a3e02b5818c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"78fc9fe6-2163-4cb3-9944-3773bd488df8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"338bb632-eca1-422b-b196-fd6cadb8307c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ce2e0db7-d486-4cb1-9913-6a4a011e5c8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a61739cc-41d3-48b1-a471-4092b2edbf1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T00:19:53.167 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.184+0000 7f7e4006a640 10 monclient: shutdown 2024-04-26T00:19:53.167 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.184+0000 7f7e4006a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:19:53.167 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.184+0000 7f7e4006a640 1 -- 172.21.15.37:0/2443442596 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7e380a0310 msgr2=0x7f7e3813e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:53.167 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.184+0000 7f7e4006a640 1 --2- 172.21.15.37:0/2443442596 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7e380a0310 0x7f7e3813e6e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7e2800c440 tx=0x7f7e2800c900 comp rx=0 tx=0).stop 2024-04-26T00:19:53.167 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.184+0000 7f7e4006a640 1 -- 172.21.15.37:0/2443442596 shutdown_connections 2024-04-26T00:19:53.167 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.184+0000 7f7e4006a640 1 --2- 172.21.15.37:0/2443442596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e380a6380 0x7f7e3806fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:53.167 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.184+0000 7f7e4006a640 1 --2- 172.21.15.37:0/2443442596 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7e380a0310 0x7f7e3813e6e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:53.167 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.184+0000 7f7e4006a640 1 --2- 172.21.15.37:0/2443442596 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e3809f410 0x7f7e3813e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:53.167 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.184+0000 7f7e4006a640 1 -- 172.21.15.37:0/2443442596 >> 172.21.15.37:0/2443442596 conn(0x7f7e380a7590 msgr2=0x7f7e380a3f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:19:53.168 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.184+0000 7f7e4006a640 1 -- 172.21.15.37:0/2443442596 shutdown_connections 2024-04-26T00:19:53.168 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:53.184+0000 7f7e4006a640 1 -- 172.21.15.37:0/2443442596 wait complete. 2024-04-26T00:19:53.182 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-04-26T00:19:53.219 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-04-26T00:19:53.238+0000 7fc2fd1e0180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T00:19:53.360 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-04-26T00:19:53.378+0000 7fc2fd1e0180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T00:19:53.527 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-04-26T00:19:53.546+0000 7fc2fd1e0180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T00:19:53.594 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-04-26T00:19:53.614+0000 7fc2fd1e0180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T00:19:53.734 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-04-26T00:19:53.754+0000 7fc2fd1e0180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T00:19:54.376 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-04-26T00:19:54.398+0000 7fc2fd1e0180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T00:19:54.455 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-04-26T00:19:54.474+0000 7fc2fd1e0180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T00:19:54.645 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-04-26T00:19:54.666+0000 7fc2fd1e0180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T00:19:54.723 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-04-26T00:19:54.742+0000 7fc2fd1e0180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T00:19:55.028 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-04-26T00:19:55.050+0000 7fc2fd1e0180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T00:19:55.093 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-04-26T00:19:55.114+0000 7fc2fd1e0180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T00:19:56.141 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-04-26T00:19:56.162+0000 7fc2a4629640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-26T00:19:56.141 INFO:tasks.ceph.mgr.x.smithi045.stderr:2024-04-26T00:19:56.162+0000 7fc2a4629640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-26T00:19:56.597 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T00:19:56.616+0000 7f0f99b4c7c0 -1 osd.0 0 log_to_monitors true 2024-04-26T00:19:56.609 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T00:19:56.630+0000 7f55ad81e7c0 -1 osd.5 0 log_to_monitors true 2024-04-26T00:19:56.616 INFO:tasks.ceph.osd.3.smithi045.stderr:2024-04-26T00:19:56.638+0000 7f7d840157c0 -1 osd.3 0 log_to_monitors true 2024-04-26T00:19:56.624 INFO:tasks.ceph.osd.1.smithi037.stderr:2024-04-26T00:19:56.644+0000 7f10e57957c0 -1 osd.1 0 log_to_monitors true 2024-04-26T00:19:56.639 INFO:tasks.ceph.osd.4.smithi045.stderr:2024-04-26T00:19:56.658+0000 7fb9570547c0 -1 osd.4 0 log_to_monitors true 2024-04-26T00:19:56.643 INFO:tasks.ceph.osd.2.smithi037.stderr:2024-04-26T00:19:56.660+0000 7f24a4f927c0 -1 osd.2 0 log_to_monitors true 2024-04-26T00:19:58.196 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T00:19:58.216+0000 7f0f95af5640 -1 osd.0 0 waiting for initial osdmap 2024-04-26T00:19:58.197 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T00:19:58.218+0000 7f55a97c7640 -1 osd.5 0 waiting for initial osdmap 2024-04-26T00:19:58.199 INFO:tasks.ceph.osd.3.smithi045.stderr:2024-04-26T00:19:58.218+0000 7f7d7ffbe640 -1 osd.3 0 waiting for initial osdmap 2024-04-26T00:19:58.202 INFO:tasks.ceph.osd.2.smithi037.stderr:2024-04-26T00:19:58.220+0000 7f24a0f3b640 -1 osd.2 0 waiting for initial osdmap 2024-04-26T00:19:58.204 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T00:19:58.226+0000 7f55a5604640 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T00:19:58.205 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T00:19:58.224+0000 7f0f91932640 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T00:19:58.211 INFO:tasks.ceph.osd.3.smithi045.stderr:2024-04-26T00:19:58.230+0000 7f7d7bdfb640 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T00:19:58.215 INFO:tasks.ceph.osd.2.smithi037.stderr:2024-04-26T00:19:58.232+0000 7f249cd78640 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T00:19:58.891 INFO:tasks.ceph.osd.1.smithi037.stderr:2024-04-26T00:19:58.908+0000 7f10e1f50640 -1 osd.1 0 waiting for initial osdmap 2024-04-26T00:19:58.892 INFO:tasks.ceph.osd.4.smithi045.stderr:2024-04-26T00:19:58.914+0000 7fb95380f640 -1 osd.4 0 waiting for initial osdmap 2024-04-26T00:19:58.896 INFO:tasks.ceph.osd.1.smithi037.stderr:2024-04-26T00:19:58.916+0000 7f10d9573640 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T00:19:58.904 INFO:tasks.ceph.osd.4.smithi045.stderr:2024-04-26T00:19:58.926+0000 7fb94ae32640 -1 osd.4 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T00:19:59.788 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-26T00:19:59.911 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.927+0000 7fa13dcac640 10 monclient: get_monmap_and_config 2024-04-26T00:19:59.912 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.927+0000 7fa13dcac640 10 monclient: build_initial_monmap 2024-04-26T00:19:59.912 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.927+0000 7fa13dcac640 10 monclient: monmap: 2024-04-26T00:19:59.912 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-04-26T00:19:59.912 INFO:teuthology.misc.health.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:59.912 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-04-26T00:19:59.935415+0000 2024-04-26T00:19:59.912 INFO:teuthology.misc.health.smithi037.stderr:created 2024-04-26T00:19:59.935415+0000 2024-04-26T00:19:59.912 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:19:59.912 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-04-26T00:19:59.912 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:19:59.912 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:19:59.912 INFO:teuthology.misc.health.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:19:59.912 INFO:teuthology.misc.health.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:19:59.912 INFO:teuthology.misc.health.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:19:59.913 INFO:teuthology.misc.health.smithi037.stderr: 2024-04-26T00:19:59.917 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa13dcac640 1 Processor -- start 2024-04-26T00:19:59.917 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa13dcac640 1 -- start start 2024-04-26T00:19:59.917 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa13dcac640 10 monclient: init 2024-04-26T00:19:59.917 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa13dcac640 10 monclient: _reopen_session rank -1 2024-04-26T00:19:59.918 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa13dcac640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-04-26T00:19:59.918 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa13dcac640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa1380a2250 addr v1:172.21.15.37:6789/0 2024-04-26T00:19:59.918 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa13dcac640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa138099d60 0x7fa13809e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:59.918 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa13dcac640 10 monclient(hunting): picked mon.noname-c con 0x7fa138099d60 addr v2:172.21.15.45:3300/0 2024-04-26T00:19:59.918 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa13dcac640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa138139510 0x7fa13813b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:59.918 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa13dcac640 10 monclient(hunting): picked mon.noname-b con 0x7fa138139510 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:19:59.918 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa13dcac640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1380a28a0 con 0x7fa1380a2250 2024-04-26T00:19:59.919 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa13dcac640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:59.919 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa13dcac640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fa1380a2630 con 0x7fa138099d60 2024-04-26T00:19:59.919 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa13dcac640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:59.919 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa13dcac640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa13809f760 con 0x7fa138139510 2024-04-26T00:19:59.919 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa13dcac640 10 monclient(hunting): _renew_subs 2024-04-26T00:19:59.919 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa13dcac640 10 monclient(hunting): authenticate will time out at 981.241027s 2024-04-26T00:19:59.920 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa137fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa138139510 0x7fa13813b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:59.920 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa137fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa138139510 0x7fa13813b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:57656/0 (socket says 172.21.15.37:57656) 2024-04-26T00:19:59.920 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa1377fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fa1380a2250 0x7fa13809d990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:55742/0 (socket says 172.21.15.37:55742) 2024-04-26T00:19:59.920 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa137fff640 1 -- 172.21.15.37:0/1796363784 learned_addr learned my addr 172.21.15.37:0/1796363784 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:19:59.920 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa137fff640 10 monclient(hunting): get_auth_request con 0x7fa138139510 auth_method 0 2024-04-26T00:19:59.920 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa137fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:59.920 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa137fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:59.920 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa137fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:59.920 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa137fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:19:59.920 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa137fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:19:59.920 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa137fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:19:59.920 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa136ffd640 1 --2- 172.21.15.37:0/1796363784 >> v2:172.21.15.45:3300/0 conn(0x7fa138099d60 0x7fa13809e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:59.921 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa136ffd640 10 monclient(hunting): get_auth_request con 0x7fa138099d60 auth_method 0 2024-04-26T00:19:59.921 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa136ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:59.921 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa136ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:59.921 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa136ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:59.921 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa137fff640 10 monclient(hunting): handle_auth_done global_id 4169 payload 274 2024-04-26T00:19:59.921 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa137fff640 1 -- 172.21.15.37:0/1796363784 >> v2:172.21.15.45:3300/0 conn(0x7fa138099d60 msgr2=0x7fa13809e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:19:59.921 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa137fff640 1 --2- 172.21.15.37:0/1796363784 >> v2:172.21.15.45:3300/0 conn(0x7fa138099d60 0x7fa13809e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:59.922 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa137fff640 1 -- 172.21.15.37:0/1796363784 >> v1:172.21.15.37:6789/0 conn(0x7fa1380a2250 legacy=0x7fa13809d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:19:59.922 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa137fff640 10 monclient: _finish_hunting 0 2024-04-26T00:19:59.922 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa137fff640 1 monclient: found mon.noname-b 2024-04-26T00:19:59.922 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa137fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:19:59.922 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa137fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:19:59.922 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.931+0000 7fa137fff640 1 -- 172.21.15.37:0/1796363784 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa138099380 con 0x7fa138139510 2024-04-26T00:19:59.929 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.935+0000 7fa137fff640 1 -- 172.21.15.37:0/1796363784 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa138139510 msgr2=0x7fa13813b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:19:59.929 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.935+0000 7fa137fff640 1 --2- 172.21.15.37:0/1796363784 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa138139510 0x7fa13813b900 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa130009870 tx=0x7fa13001c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=639763b2c1c8d77f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:19:59.929 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.935+0000 7fa1367fc640 1 -- 172.21.15.37:0/1796363784 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1300b0070 con 0x7fa138139510 2024-04-26T00:19:59.929 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.935+0000 7fa1367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:59.929 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.935+0000 7fa1367fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:19:59.929 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.935+0000 7fa1367fc640 10 monclient: dump: 2024-04-26T00:19:59.929 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-04-26T00:19:59.929 INFO:teuthology.misc.health.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:59.929 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:59.930 INFO:teuthology.misc.health.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:59.930 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:59.930 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-04-26T00:19:59.930 INFO:teuthology.misc.health.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:59.930 INFO:teuthology.misc.health.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:59.930 INFO:teuthology.misc.health.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:59.930 INFO:teuthology.misc.health.smithi037.stderr: 2024-04-26T00:19:59.931 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.935+0000 7fa1367fc640 10 monclient: _finish_auth 0 2024-04-26T00:19:59.931 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.935+0000 7fa1367fc640 10 monclient: _check_auth_tickets 2024-04-26T00:19:59.931 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.935+0000 7fa1367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:19:59.931 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.935+0000 7fa1367fc640 1 -- 172.21.15.37:0/1796363784 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1300ab600 con 0x7fa138139510 2024-04-26T00:19:59.931 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.935+0000 7fa1367fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:19:59.931 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.935+0000 7fa1367fc640 1 -- 172.21.15.37:0/1796363784 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1300ab9d0 con 0x7fa138139510 2024-04-26T00:19:59.931 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.935+0000 7fa13dcac640 5 monclient: authenticate success, global_id 4169 2024-04-26T00:19:59.931 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.935+0000 7fa13dcac640 10 monclient: get_monmap_and_config success 2024-04-26T00:19:59.931 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.935+0000 7fa13dcac640 4 set_mon_vals no callback set 2024-04-26T00:19:59.931 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.935+0000 7fa1367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:59.931 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.935+0000 7fa1367fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:19:59.931 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.935+0000 7fa1367fc640 10 monclient: dump: 2024-04-26T00:19:59.932 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-04-26T00:19:59.932 INFO:teuthology.misc.health.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:59.932 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:59.932 INFO:teuthology.misc.health.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:59.932 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:59.932 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-04-26T00:19:59.932 INFO:teuthology.misc.health.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:59.932 INFO:teuthology.misc.health.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:59.932 INFO:teuthology.misc.health.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:59.932 INFO:teuthology.misc.health.smithi037.stderr: 2024-04-26T00:19:59.932 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.935+0000 7fa13dcac640 10 monclient: shutdown 2024-04-26T00:19:59.932 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.935+0000 7fa13dcac640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:19:59.933 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.935+0000 7fa13dcac640 1 -- 172.21.15.37:0/1796363784 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa138139510 msgr2=0x7fa13813b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:59.933 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.935+0000 7fa13dcac640 1 --2- 172.21.15.37:0/1796363784 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa138139510 0x7fa13813b900 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa130009870 tx=0x7fa13001c470 comp rx=0 tx=0).stop 2024-04-26T00:19:59.933 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.935+0000 7fa13dcac640 1 -- 172.21.15.37:0/1796363784 shutdown_connections 2024-04-26T00:19:59.933 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.935+0000 7fa13dcac640 1 --2- 172.21.15.37:0/1796363784 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa138139510 0x7fa13813b900 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:59.933 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.935+0000 7fa13dcac640 1 --2- 172.21.15.37:0/1796363784 >> v2:172.21.15.45:3300/0 conn(0x7fa138099d60 0x7fa13809e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:59.933 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.935+0000 7fa13dcac640 1 -- 172.21.15.37:0/1796363784 >> 172.21.15.37:0/1796363784 conn(0x7fa1380a3680 msgr2=0x7fa1380a3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:19:59.933 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.935+0000 7fa13dcac640 1 -- 172.21.15.37:0/1796363784 shutdown_connections 2024-04-26T00:19:59.933 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.935+0000 7fa13dcac640 1 -- 172.21.15.37:0/1796363784 wait complete. 2024-04-26T00:19:59.933 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.935+0000 7fa13dcac640 10 monclient: build_initial_monmap 2024-04-26T00:19:59.933 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.935+0000 7fa13dcac640 10 monclient: monmap: 2024-04-26T00:19:59.933 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-04-26T00:19:59.934 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:19:59.934 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-04-26T00:19:59.934 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-04-26T00:19:59.934 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:19:59.934 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-04-26T00:19:59.934 INFO:teuthology.misc.health.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:19:59.934 INFO:teuthology.misc.health.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:19:59.934 INFO:teuthology.misc.health.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:19:59.934 INFO:teuthology.misc.health.smithi037.stderr: 2024-04-26T00:19:59.934 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.935+0000 7fa13dcac640 1 Processor -- start 2024-04-26T00:19:59.934 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.935+0000 7fa13dcac640 1 -- start start 2024-04-26T00:19:59.934 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa13dcac640 10 monclient: init 2024-04-26T00:19:59.934 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa13dcac640 10 monclient: _reopen_session rank -1 2024-04-26T00:19:59.935 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa13dcac640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:19:59.935 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa13dcac640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa138099d60 0x7fa138138e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:59.935 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa13dcac640 10 monclient(hunting): picked mon.noname-a con 0x7fa138099d60 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:19:59.935 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa13dcac640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1380a2250 0x7fa138098980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:59.935 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa13dcac640 10 monclient(hunting): picked mon.noname-c con 0x7fa1380a2250 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:19:59.935 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa13dcac640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa138139510 0x7fa13813ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:59.935 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa13dcac640 10 monclient(hunting): picked mon.noname-b con 0x7fa138139510 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:19:59.935 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa13dcac640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:59.935 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa13dcac640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa13813c650 con 0x7fa138099d60 2024-04-26T00:19:59.935 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa13dcac640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:59.935 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa13dcac640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa13809e7f0 con 0x7fa138139510 2024-04-26T00:19:59.936 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa13dcac640 10 monclient(hunting): start opening mon connection 2024-04-26T00:19:59.936 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa13dcac640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa13813c7d0 con 0x7fa1380a2250 2024-04-26T00:19:59.936 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa13dcac640 10 monclient(hunting): _renew_subs 2024-04-26T00:19:59.936 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa13dcac640 10 monclient(hunting): authenticate will time out at 981.248527s 2024-04-26T00:19:59.936 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa136ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1380a2250 0x7fa138098980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:59.936 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa1377fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa138099d60 0x7fa138138e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:59.936 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa1377fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa138099d60 0x7fa138138e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44402/0 (socket says 172.21.15.37:44402) 2024-04-26T00:19:59.936 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa1377fe640 1 -- 172.21.15.37:0/3268455315 learned_addr learned my addr 172.21.15.37:0/3268455315 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:19:59.936 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa1377fe640 10 monclient(hunting): get_auth_request con 0x7fa138099d60 auth_method 0 2024-04-26T00:19:59.936 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa1377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:59.936 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa1377fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:59.936 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa136ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1380a2250 0x7fa138098980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:57658/0 (socket says 172.21.15.37:57658) 2024-04-26T00:19:59.937 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa1377fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:59.937 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa136ffd640 10 monclient(hunting): get_auth_request con 0x7fa1380a2250 auth_method 0 2024-04-26T00:19:59.937 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa136ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:19:59.937 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa136ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:19:59.937 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa136ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:19:59.937 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa136ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:19:59.937 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa136ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:19:59.937 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa136ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:19:59.937 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa1377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:19:59.937 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa1377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:19:59.937 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa1377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:19:59.937 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa137fff640 1 --2- 172.21.15.37:0/3268455315 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa138139510 0x7fa13813ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:59.938 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa136ffd640 10 monclient(hunting): handle_auth_done global_id 4172 payload 824 2024-04-26T00:19:59.938 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa136ffd640 1 -- 172.21.15.37:0/3268455315 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa138139510 msgr2=0x7fa13813ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:59.938 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa136ffd640 1 --2- 172.21.15.37:0/3268455315 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa138139510 0x7fa13813ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:59.938 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa136ffd640 1 -- 172.21.15.37:0/3268455315 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa138099d60 msgr2=0x7fa138138e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:19:59.938 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa136ffd640 1 --2- 172.21.15.37:0/3268455315 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa138099d60 0x7fa138138e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:19:59.938 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa136ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:19:59.938 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa136ffd640 1 monclient: found mon.noname-c 2024-04-26T00:19:59.938 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa136ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:19:59.938 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa136ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:19:59.938 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa136ffd640 1 -- 172.21.15.37:0/3268455315 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa138091590 con 0x7fa1380a2250 2024-04-26T00:19:59.938 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa137fff640 10 monclient: get_auth_request con 0x7fa138139510 auth_method 0 2024-04-26T00:19:59.938 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa1377fe640 1 --2- 172.21.15.37:0/3268455315 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa138099d60 0x7fa138138e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:19:59.939 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa137fff640 1 --2- 172.21.15.37:0/3268455315 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa138139510 0x7fa13813ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:19:59.939 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa136ffd640 1 -- 172.21.15.37:0/3268455315 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1380a2250 msgr2=0x7fa138098980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:19:59.939 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa136ffd640 1 --2- 172.21.15.37:0/3268455315 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1380a2250 0x7fa138098980 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa12c007e90 tx=0x7fa12c00e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:19:59.939 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa11ffff640 1 -- 172.21.15.37:0/3268455315 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa12c014620 con 0x7fa1380a2250 2024-04-26T00:19:59.939 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa11ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:59.939 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa11ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:19:59.939 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa11ffff640 10 monclient: dump: 2024-04-26T00:19:59.940 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-04-26T00:19:59.940 INFO:teuthology.misc.health.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:59.940 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:59.940 INFO:teuthology.misc.health.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:59.940 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:59.940 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-04-26T00:19:59.940 INFO:teuthology.misc.health.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:59.941 INFO:teuthology.misc.health.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:59.941 INFO:teuthology.misc.health.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:59.941 INFO:teuthology.misc.health.smithi037.stderr: 2024-04-26T00:19:59.941 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa11ffff640 10 monclient: _finish_auth 0 2024-04-26T00:19:59.941 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa11ffff640 10 monclient: _check_auth_tickets 2024-04-26T00:19:59.941 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa11ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:19:59.941 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa11ffff640 1 -- 172.21.15.37:0/3268455315 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa12c019070 con 0x7fa1380a2250 2024-04-26T00:19:59.941 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa13dcac640 5 monclient: authenticate success, global_id 4172 2024-04-26T00:19:59.941 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa11ffff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:19:59.941 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa11ffff640 1 -- 172.21.15.37:0/3268455315 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa12c0149e0 con 0x7fa1380a2250 2024-04-26T00:19:59.941 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa11ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:19:59.941 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa11ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:19:59.942 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa11ffff640 10 monclient: dump: 2024-04-26T00:19:59.942 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-04-26T00:19:59.942 INFO:teuthology.misc.health.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:19:59.942 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:59.942 INFO:teuthology.misc.health.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:19:59.942 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:19:59.942 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-04-26T00:19:59.942 INFO:teuthology.misc.health.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:19:59.942 INFO:teuthology.misc.health.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:19:59.942 INFO:teuthology.misc.health.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:19:59.942 INFO:teuthology.misc.health.smithi037.stderr: 2024-04-26T00:19:59.942 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.939+0000 7fa135ffb640 4 set_mon_vals no callback set 2024-04-26T00:19:59.943 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.943+0000 7fa13dcac640 10 monclient: _renew_subs 2024-04-26T00:19:59.943 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.943+0000 7fa13dcac640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:19:59.943 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.943+0000 7fa13dcac640 1 -- 172.21.15.37:0/3268455315 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1380a1b20 con 0x7fa1380a2250 2024-04-26T00:19:59.943 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.943+0000 7fa13dcac640 10 monclient: _renew_subs 2024-04-26T00:19:59.943 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.943+0000 7fa13dcac640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:19:59.943 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.943+0000 7fa13dcac640 1 -- 172.21.15.37:0/3268455315 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa138092d90 con 0x7fa1380a2250 2024-04-26T00:19:59.943 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.943+0000 7fa13dcac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:19:59.943 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.943+0000 7fa13dcac640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:19:59.943 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.943+0000 7fa13dcac640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:19:59.943 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.943+0000 7fa11ffff640 1 -- 172.21.15.37:0/3268455315 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 4) ==== 43962+0+0 (secure 0 0 0) 0x7fa12c014b80 con 0x7fa1380a2250 2024-04-26T00:19:59.943 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.943+0000 7fa13dcac640 1 -- 172.21.15.37:0/3268455315 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa138062e50 con 0x7fa1380a2250 2024-04-26T00:19:59.944 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.943+0000 7fa11ffff640 1 --2- 172.21.15.37:0/3268455315 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa124035e40 0x7fa124038300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:19:59.944 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.943+0000 7fa11ffff640 1 -- 172.21.15.37:0/3268455315 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(11..11 src has 1..11) ==== 2767+0+0 (secure 0 0 0) 0x7fa12c048cd0 con 0x7fa1380a2250 2024-04-26T00:19:59.944 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.943+0000 7fa1377fe640 1 --2- 172.21.15.37:0/3268455315 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa124035e40 0x7fa124038300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:19:59.944 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.943+0000 7fa1377fe640 10 monclient: get_auth_request con 0x7fa124035e40 auth_method 0 2024-04-26T00:19:59.944 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.947+0000 7fa1377fe640 1 --2- 172.21.15.37:0/3268455315 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa124035e40 0x7fa124038300 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa128006ec0 tx=0x7fa128007f30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:19:59.944 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.947+0000 7fa11ffff640 1 -- 172.21.15.37:0/3268455315 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa12c047ec0 con 0x7fa1380a2250 2024-04-26T00:19:59.945 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.947+0000 7fa11ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:19:59.945 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:19:59.947+0000 7fa11ffff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:00.092 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:20:00.111+0000 7fa13dcac640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-26T00:20:00.092 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:20:00.111+0000 7fa13dcac640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-26T00:20:00.092 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:20:00.111+0000 7fa13dcac640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:00.092 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:20:00.111+0000 7fa13dcac640 1 -- 172.21.15.37:0/3268455315 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa138098ec0 con 0x7fa1380a2250 2024-04-26T00:20:00.093 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:20:00.111+0000 7fa11ffff640 1 -- 172.21.15.37:0/3268455315 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) ==== 74+0+7017 (secure 0 0 0) 0x7fa12c01e890 con 0x7fa1380a2250 2024-04-26T00:20:00.093 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:20:00.111+0000 7fa11ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-26T00:20:00.093 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:20:00.111+0000 7fa11ffff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:00.093 INFO:teuthology.misc.health.smithi037.stdout: 2024-04-26T00:20:00.093 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":11,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","created":"2024-04-26T00:19:50.025527+0000","modified":"2024-04-26T00:19:59.222603+0000","last_up_change":"2024-04-26T00:19:59.222603+0000","last_in_change":"2024-04-26T00:19:52.138726+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a5d4b9d3-5499-491f-a123-85d73de452af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":3204000569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":3204000569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6811","nonce":3204000569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":3204000569}]},"public_addr":"172.21.15.37:6806/3204000569","cluster_addr":"172.21.15.37:6808/3204000569","heartbeat_back_addr":"172.21.15.37:6811/3204000569","heartbeat_front_addr":"172.21.15.37:6810/3204000569","state":["exists","up"]},{"osd":1,"uuid":"cb4583fd-a0fa-4559-9f99-5a3e02b5818c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":1084527868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6805","nonce":1084527868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6809","nonce":1084527868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6807","nonce":1084527868}]},"public_addr":"172.21.15.37:6804/1084527868","cluster_addr":"172.21.15.37:6805/1084527868","heartbeat_back_addr":"172.21.15.37:6809/1084527868","heartbeat_front_addr":"172.21.15.37:6807/1084527868","state":["exists","up"]},{"osd":2,"uuid":"78fc9fe6-2163-4cb3-9944-3773bd488df8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":603175474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6801","nonce":603175474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6803","nonce":603175474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":603175474}]},"public_addr":"172.21.15.37:6800/603175474","cluster_addr":"172.21.15.37:6801/603175474","heartbeat_back_addr":"172.21.15.37:6803/603175474","heartbeat_front_addr":"172.21.15.37:6802/603175474","state":["exists","up"]},{"osd":3,"uuid":"338bb632-eca1-422b-b196-fd6cadb8307c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":2173737584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6801","nonce":2173737584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6803","nonce":2173737584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":2173737584}]},"public_addr":"172.21.15.45:6800/2173737584","cluster_addr":"172.21.15.45:6801/2173737584","heartbeat_back_addr":"172.21.15.45:6803/2173737584","heartbeat_front_addr":"172.21.15.45:6802/2173737584","state":["exists","up"]},{"osd":4,"uuid":"ce2e0db7-d486-4cb1-9913-6a4a011e5c8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":152605921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6805","nonce":152605921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6807","nonce":152605921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":152605921}]},"public_addr":"172.21.15.45:6804/152605921","cluster_addr":"172.21.15.45:6805/152605921","heartbeat_back_addr":"172.21.15.45:6807/152605921","heartbeat_front_addr":"172.21.15.45:6806/152605921","state":["exists","up"]},{"osd":5,"uuid":"a61739cc-41d3-48b1-a471-4092b2edbf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3915746722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6809","nonce":3915746722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6811","nonce":3915746722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3915746722}]},"public_addr":"172.21.15.45:6808/3915746722","cluster_addr":"172.21.15.45:6809/3915746722","heartbeat_back_addr":"172.21.15.45:6811/3915746722","heartbeat_front_addr":"172.21.15.45:6810/3915746722","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T00:20:00.096 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:20:00.111+0000 7fa13dcac640 1 -- 172.21.15.37:0/3268455315 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa124035e40 msgr2=0x7fa124038300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:00.096 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:20:00.111+0000 7fa13dcac640 1 --2- 172.21.15.37:0/3268455315 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa124035e40 0x7fa124038300 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa128006ec0 tx=0x7fa128007f30 comp rx=0 tx=0).stop 2024-04-26T00:20:00.096 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:20:00.111+0000 7fa13dcac640 10 monclient: shutdown 2024-04-26T00:20:00.096 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:20:00.111+0000 7fa13dcac640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:00.096 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:20:00.111+0000 7fa13dcac640 1 -- 172.21.15.37:0/3268455315 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1380a2250 msgr2=0x7fa138098980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:00.096 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:20:00.111+0000 7fa13dcac640 1 --2- 172.21.15.37:0/3268455315 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1380a2250 0x7fa138098980 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa12c007e90 tx=0x7fa12c00e4b0 comp rx=0 tx=0).stop 2024-04-26T00:20:00.096 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:20:00.111+0000 7fa13dcac640 1 -- 172.21.15.37:0/3268455315 shutdown_connections 2024-04-26T00:20:00.096 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:20:00.111+0000 7fa13dcac640 1 --2- 172.21.15.37:0/3268455315 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa124035e40 0x7fa124038300 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:00.096 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:20:00.111+0000 7fa13dcac640 1 --2- 172.21.15.37:0/3268455315 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa138139510 0x7fa13813ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:00.096 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:20:00.111+0000 7fa13dcac640 1 --2- 172.21.15.37:0/3268455315 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1380a2250 0x7fa138098980 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:00.096 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:20:00.111+0000 7fa13dcac640 1 --2- 172.21.15.37:0/3268455315 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa138099d60 0x7fa138138e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:00.097 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:20:00.111+0000 7fa13dcac640 1 -- 172.21.15.37:0/3268455315 >> 172.21.15.37:0/3268455315 conn(0x7fa1380a3680 msgr2=0x7fa13809d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:00.097 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:20:00.115+0000 7fa13dcac640 1 -- 172.21.15.37:0/3268455315 shutdown_connections 2024-04-26T00:20:00.097 INFO:teuthology.misc.health.smithi037.stderr:2024-04-26T00:20:00.115+0000 7fa13dcac640 1 -- 172.21.15.37:0/3268455315 wait complete. 2024-04-26T00:20:00.112 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-04-26T00:20:00.113 INFO:tasks.ceph:Creating RBD pool 2024-04-26T00:20:00.113 DEBUG:teuthology.orchestra.run.smithi037:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-26T00:20:00.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.247+0000 7fde66fbc640 10 monclient: get_monmap_and_config 2024-04-26T00:20:00.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.251+0000 7fde66fbc640 10 monclient: build_initial_monmap 2024-04-26T00:20:00.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.251+0000 7fde66fbc640 10 monclient: monmap: 2024-04-26T00:20:00.232 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:00.232 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:00.232 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:00.255937+0000 2024-04-26T00:20:00.232 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:00.255937+0000 2024-04-26T00:20:00.232 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:00.233 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:00.233 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:00.233 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:00.233 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:00.233 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:00.233 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:00.233 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:00.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.251+0000 7fde66fbc640 1 Processor -- start 2024-04-26T00:20:00.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.251+0000 7fde66fbc640 1 -- start start 2024-04-26T00:20:00.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.251+0000 7fde66fbc640 10 monclient: init 2024-04-26T00:20:00.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.251+0000 7fde66fbc640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:00.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.251+0000 7fde66fbc640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-04-26T00:20:00.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.251+0000 7fde66fbc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fde600a2fe0 addr v1:172.21.15.37:6789/0 2024-04-26T00:20:00.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.251+0000 7fde66fbc640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fde60071da0 0x7fde600a1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:00.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.251+0000 7fde66fbc640 10 monclient(hunting): picked mon.noname-a con 0x7fde60071da0 addr v2:172.21.15.37:3300/0 2024-04-26T00:20:00.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.251+0000 7fde66fbc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fde601391c0 addr v1:172.21.15.45:6789/0 2024-04-26T00:20:00.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.251+0000 7fde66fbc640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde60072d70 con 0x7fde600a2fe0 2024-04-26T00:20:00.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.251+0000 7fde66fbc640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde60070b20 con 0x7fde601391c0 2024-04-26T00:20:00.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.251+0000 7fde66fbc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:00.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.251+0000 7fde66fbc640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fde600710e0 con 0x7fde60071da0 2024-04-26T00:20:00.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.251+0000 7fde64d58640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fde600a2fe0 0x7fde600a13d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:55750/0 (socket says 172.21.15.37:55750) 2024-04-26T00:20:00.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.251+0000 7fde66fbc640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:00.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.251+0000 7fde64d58640 1 -- 172.21.15.37:0/3617548198 learned_addr learned my addr 172.21.15.37:0/3617548198 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:20:00.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.251+0000 7fde5ffff640 1 --2- 172.21.15.37:0/3617548198 >> v2:172.21.15.37:3300/0 conn(0x7fde60071da0 0x7fde600a1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:00.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.251+0000 7fde66fbc640 10 monclient(hunting): authenticate will time out at 981.561896s 2024-04-26T00:20:00.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.251+0000 7fde5ffff640 10 monclient(hunting): get_auth_request con 0x7fde60071da0 auth_method 0 2024-04-26T00:20:00.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.251+0000 7fde5ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:00.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.251+0000 7fde5ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:00.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.251+0000 7fde5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:00.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.251+0000 7fde5ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:00.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.251+0000 7fde5ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:00.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.251+0000 7fde5ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:00.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.251+0000 7fde5ffff640 10 monclient(hunting): handle_auth_done global_id 4185 payload 274 2024-04-26T00:20:00.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.251+0000 7fde5ffff640 1 -- 172.21.15.37:0/3617548198 >> v1:172.21.15.45:6789/0 conn(0x7fde601391c0 legacy=0x7fde6013b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:00.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.251+0000 7fde5ffff640 1 -- 172.21.15.37:0/3617548198 >> v1:172.21.15.37:6789/0 conn(0x7fde600a2fe0 legacy=0x7fde600a13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:00.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.251+0000 7fde5ffff640 10 monclient: _finish_hunting 0 2024-04-26T00:20:00.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.251+0000 7fde5ffff640 1 monclient: found mon.noname-a 2024-04-26T00:20:00.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.251+0000 7fde5ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:00.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.251+0000 7fde5ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:00.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.251+0000 7fde5ffff640 1 -- 172.21.15.37:0/3617548198 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde6006f770 con 0x7fde60071da0 2024-04-26T00:20:00.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.255+0000 7fde5ffff640 1 -- 172.21.15.37:0/3617548198 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde60071da0 msgr2=0x7fde600a1ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:00.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.255+0000 7fde5ffff640 1 --2- 172.21.15.37:0/3617548198 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde60071da0 0x7fde600a1ae0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fde60073040 tx=0x7fde4c01c2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bed674a6e7b1fe37 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:00.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.255+0000 7fde5f7fe640 1 -- 172.21.15.37:0/3617548198 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde4c0b0070 con 0x7fde60071da0 2024-04-26T00:20:00.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.255+0000 7fde5f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:00.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.255+0000 7fde5f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:00.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.255+0000 7fde5f7fe640 10 monclient: dump: 2024-04-26T00:20:00.239 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:00.239 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:00.239 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:00.239 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:00.240 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:00.240 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:00.240 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:00.240 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:00.240 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:00.240 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:00.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.255+0000 7fde5f7fe640 10 monclient: _finish_auth 0 2024-04-26T00:20:00.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.255+0000 7fde5f7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:20:00.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.255+0000 7fde5f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:00.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.255+0000 7fde66fbc640 5 monclient: authenticate success, global_id 4185 2024-04-26T00:20:00.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.255+0000 7fde66fbc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:20:00.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.255+0000 7fde5f7fe640 1 -- 172.21.15.37:0/3617548198 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde4c0ab600 con 0x7fde60071da0 2024-04-26T00:20:00.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.255+0000 7fde5f7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:00.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.255+0000 7fde66fbc640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:00.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.255+0000 7fde66fbc640 4 set_mon_vals no callback set 2024-04-26T00:20:00.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.255+0000 7fde5f7fe640 1 -- 172.21.15.37:0/3617548198 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde4c0ab9d0 con 0x7fde60071da0 2024-04-26T00:20:00.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.255+0000 7fde5f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:00.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.255+0000 7fde5f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:00.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.255+0000 7fde5f7fe640 10 monclient: dump: 2024-04-26T00:20:00.241 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:00.241 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:00.241 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:00.241 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:00.241 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:00.241 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:00.241 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:00.241 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:00.241 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:00.241 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:00.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.255+0000 7fde66fbc640 10 monclient: shutdown 2024-04-26T00:20:00.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.255+0000 7fde66fbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:00.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.255+0000 7fde66fbc640 1 -- 172.21.15.37:0/3617548198 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde60071da0 msgr2=0x7fde600a1ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:00.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.255+0000 7fde66fbc640 1 --2- 172.21.15.37:0/3617548198 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde60071da0 0x7fde600a1ae0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fde60073040 tx=0x7fde4c01c2c0 comp rx=0 tx=0).stop 2024-04-26T00:20:00.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.255+0000 7fde66fbc640 1 -- 172.21.15.37:0/3617548198 shutdown_connections 2024-04-26T00:20:00.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.255+0000 7fde66fbc640 1 --2- 172.21.15.37:0/3617548198 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde60071da0 0x7fde600a1ae0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:00.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.255+0000 7fde66fbc640 1 -- 172.21.15.37:0/3617548198 >> 172.21.15.37:0/3617548198 conn(0x7fde600a24c0 msgr2=0x7fde600a28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:00.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.255+0000 7fde66fbc640 1 -- 172.21.15.37:0/3617548198 shutdown_connections 2024-04-26T00:20:00.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.255+0000 7fde66fbc640 1 -- 172.21.15.37:0/3617548198 wait complete. 2024-04-26T00:20:00.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.255+0000 7fde66fbc640 10 monclient: build_initial_monmap 2024-04-26T00:20:00.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.255+0000 7fde66fbc640 10 monclient: monmap: 2024-04-26T00:20:00.246 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:00.246 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:00.246 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:00.247 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:00.247 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:00.247 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:00.247 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:00.247 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:00.247 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:00.247 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:00.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde66fbc640 1 Processor -- start 2024-04-26T00:20:00.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde66fbc640 1 -- start start 2024-04-26T00:20:00.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde66fbc640 10 monclient: init 2024-04-26T00:20:00.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde66fbc640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:00.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde66fbc640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:20:00.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde66fbc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde60071da0 0x7fde6009aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:00.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde66fbc640 10 monclient(hunting): picked mon.noname-c con 0x7fde60071da0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:00.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde66fbc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde600a2fe0 0x7fde6009b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:00.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde66fbc640 10 monclient(hunting): picked mon.noname-b con 0x7fde600a2fe0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:00.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde66fbc640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde601391c0 0x7fde601cd390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:00.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde66fbc640 10 monclient(hunting): picked mon.noname-a con 0x7fde601391c0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:00.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde66fbc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:00.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde66fbc640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fde6013fd20 con 0x7fde601391c0 2024-04-26T00:20:00.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde66fbc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:00.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde66fbc640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fde60070e00 con 0x7fde600a2fe0 2024-04-26T00:20:00.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde66fbc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:00.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde66fbc640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fde6013fea0 con 0x7fde60071da0 2024-04-26T00:20:00.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde66fbc640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:00.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde66fbc640 10 monclient(hunting): authenticate will time out at 981.568714s 2024-04-26T00:20:00.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde64d58640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde60071da0 0x7fde6009aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:00.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde65559640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde601391c0 0x7fde601cd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:00.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde64d58640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde60071da0 0x7fde6009aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:57674/0 (socket says 172.21.15.37:57674) 2024-04-26T00:20:00.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde64d58640 1 -- 172.21.15.37:0/942822393 learned_addr learned my addr 172.21.15.37:0/942822393 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:00.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde64d58640 10 monclient(hunting): get_auth_request con 0x7fde60071da0 auth_method 0 2024-04-26T00:20:00.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde64d58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:00.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde64d58640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:00.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde64d58640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:00.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde65559640 10 monclient(hunting): get_auth_request con 0x7fde601391c0 auth_method 0 2024-04-26T00:20:00.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde65559640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:00.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde65559640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:00.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde65559640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:00.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde65559640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:00.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde65559640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:00.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde65559640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:00.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde64d58640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:00.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde64d58640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:00.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde64d58640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:00.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde5ffff640 1 --2- 172.21.15.37:0/942822393 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde600a2fe0 0x7fde6009b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:00.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde5ffff640 10 monclient(hunting): get_auth_request con 0x7fde600a2fe0 auth_method 0 2024-04-26T00:20:00.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde5ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:00.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde5ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:00.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:00.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde65559640 10 monclient(hunting): handle_auth_done global_id 4188 payload 824 2024-04-26T00:20:00.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde65559640 1 -- 172.21.15.37:0/942822393 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde60071da0 msgr2=0x7fde6009aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:00.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde65559640 1 --2- 172.21.15.37:0/942822393 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde60071da0 0x7fde6009aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:00.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde65559640 1 -- 172.21.15.37:0/942822393 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde600a2fe0 msgr2=0x7fde6009b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:00.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde65559640 1 --2- 172.21.15.37:0/942822393 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde600a2fe0 0x7fde6009b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:00.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde65559640 10 monclient: _finish_hunting 0 2024-04-26T00:20:00.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde65559640 1 monclient: found mon.noname-a 2024-04-26T00:20:00.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde65559640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:00.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde65559640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:00.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde65559640 1 -- 172.21.15.37:0/942822393 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde6009d520 con 0x7fde601391c0 2024-04-26T00:20:00.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde64d58640 1 --2- 172.21.15.37:0/942822393 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde60071da0 0x7fde6009aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:00.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde65559640 1 -- 172.21.15.37:0/942822393 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde601391c0 msgr2=0x7fde601cd390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:00.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde65559640 1 --2- 172.21.15.37:0/942822393 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde601391c0 0x7fde601cd390 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fde5800c750 tx=0x7fde5800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:00.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde5d7fa640 1 -- 172.21.15.37:0/942822393 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde58016020 con 0x7fde601391c0 2024-04-26T00:20:00.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde5d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:00.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde5d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:00.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde5d7fa640 10 monclient: dump: 2024-04-26T00:20:00.251 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:00.251 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:00.251 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:00.251 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:00.251 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:00.251 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:00.251 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:00.251 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:00.252 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:00.252 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:00.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde5d7fa640 10 monclient: _finish_auth 0 2024-04-26T00:20:00.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde5d7fa640 10 monclient: _check_auth_tickets 2024-04-26T00:20:00.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde5d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:00.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde5d7fa640 1 -- 172.21.15.37:0/942822393 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde5800aa60 con 0x7fde601391c0 2024-04-26T00:20:00.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde5d7fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:00.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde5d7fa640 1 -- 172.21.15.37:0/942822393 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5800ae50 con 0x7fde601391c0 2024-04-26T00:20:00.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde66fbc640 5 monclient: authenticate success, global_id 4188 2024-04-26T00:20:00.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde66fbc640 10 monclient: _renew_subs 2024-04-26T00:20:00.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde66fbc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:00.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde66fbc640 1 -- 172.21.15.37:0/942822393 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde6009c650 con 0x7fde601391c0 2024-04-26T00:20:00.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde5effd640 4 set_mon_vals no callback set 2024-04-26T00:20:00.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde5d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:00.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde5d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:00.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde5d7fa640 10 monclient: dump: 2024-04-26T00:20:00.253 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:00.253 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:00.253 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:00.253 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:00.253 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:00.253 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:00.253 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:00.253 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:00.253 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:00.253 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:00.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde66fbc640 10 monclient: _renew_subs 2024-04-26T00:20:00.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde66fbc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:00.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.259+0000 7fde66fbc640 1 -- 172.21.15.37:0/942822393 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde6006ef70 con 0x7fde601391c0 2024-04-26T00:20:00.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.263+0000 7fde5d7fa640 1 -- 172.21.15.37:0/942822393 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 4) ==== 43962+0+0 (secure 0 0 0) 0x7fde58019020 con 0x7fde601391c0 2024-04-26T00:20:00.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.263+0000 7fde66fbc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:00.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.263+0000 7fde66fbc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:00.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.263+0000 7fde66fbc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:00.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.263+0000 7fde66fbc640 1 -- 172.21.15.37:0/942822393 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde6004f510 con 0x7fde601391c0 2024-04-26T00:20:00.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.263+0000 7fde5d7fa640 1 --2- 172.21.15.37:0/942822393 >> v2:172.21.15.45:6812/2379918623 conn(0x7fde44035f90 0x7fde44038450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:00.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.263+0000 7fde5d7fa640 1 -- 172.21.15.37:0/942822393 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 2767+0+0 (secure 0 0 0) 0x7fde58048450 con 0x7fde601391c0 2024-04-26T00:20:00.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.263+0000 7fde64d58640 1 --2- 172.21.15.37:0/942822393 >> v2:172.21.15.45:6812/2379918623 conn(0x7fde44035f90 0x7fde44038450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:00.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.263+0000 7fde64d58640 10 monclient: get_auth_request con 0x7fde44035f90 auth_method 0 2024-04-26T00:20:00.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.267+0000 7fde64d58640 1 --2- 172.21.15.37:0/942822393 >> v2:172.21.15.45:6812/2379918623 conn(0x7fde44035f90 0x7fde44038450 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fde54007d60 tx=0x7fde54007cb0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:00.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.267+0000 7fde5d7fa640 1 -- 172.21.15.37:0/942822393 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fde58020030 con 0x7fde601391c0 2024-04-26T00:20:00.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.267+0000 7fde5d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:00.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.267+0000 7fde5d7fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:00.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.431+0000 7fde66fbc640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-04-26T00:20:00.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.431+0000 7fde66fbc640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-04-26T00:20:00.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.431+0000 7fde66fbc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:00.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:00.431+0000 7fde66fbc640 1 -- 172.21.15.37:0/942822393 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7fde600008e0 con 0x7fde601391c0 2024-04-26T00:20:01.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.251+0000 7fde5d7fa640 1 -- 172.21.15.37:0/942822393 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v13) ==== 109+0+0 (secure 0 0 0) 0x7fde58047640 con 0x7fde601391c0 2024-04-26T00:20:01.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.251+0000 7fde5d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-04-26T00:20:01.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.251+0000 7fde5d7fa640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-04-26T00:20:01.234 INFO:teuthology.orchestra.run.smithi037.stderr:pool 'rbd' created 2024-04-26T00:20:01.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.255+0000 7fde66fbc640 1 -- 172.21.15.37:0/942822393 >> v2:172.21.15.45:6812/2379918623 conn(0x7fde44035f90 msgr2=0x7fde44038450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:01.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.255+0000 7fde66fbc640 1 --2- 172.21.15.37:0/942822393 >> v2:172.21.15.45:6812/2379918623 conn(0x7fde44035f90 0x7fde44038450 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fde54007d60 tx=0x7fde54007cb0 comp rx=0 tx=0).stop 2024-04-26T00:20:01.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.255+0000 7fde66fbc640 10 monclient: shutdown 2024-04-26T00:20:01.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.255+0000 7fde66fbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:01.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.255+0000 7fde66fbc640 1 -- 172.21.15.37:0/942822393 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde601391c0 msgr2=0x7fde601cd390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:01.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.255+0000 7fde66fbc640 1 --2- 172.21.15.37:0/942822393 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde601391c0 0x7fde601cd390 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fde5800c750 tx=0x7fde5800cc10 comp rx=0 tx=0).stop 2024-04-26T00:20:01.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.255+0000 7fde66fbc640 1 -- 172.21.15.37:0/942822393 shutdown_connections 2024-04-26T00:20:01.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.255+0000 7fde66fbc640 1 --2- 172.21.15.37:0/942822393 >> v2:172.21.15.45:6812/2379918623 conn(0x7fde44035f90 0x7fde44038450 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:01.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.255+0000 7fde66fbc640 1 --2- 172.21.15.37:0/942822393 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde601391c0 0x7fde601cd390 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:01.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.255+0000 7fde66fbc640 1 --2- 172.21.15.37:0/942822393 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde600a2fe0 0x7fde6009b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:01.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.255+0000 7fde66fbc640 1 --2- 172.21.15.37:0/942822393 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde60071da0 0x7fde6009aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:01.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.255+0000 7fde66fbc640 1 -- 172.21.15.37:0/942822393 >> 172.21.15.37:0/942822393 conn(0x7fde600a24c0 msgr2=0x7fde60130410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:01.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.255+0000 7fde66fbc640 1 -- 172.21.15.37:0/942822393 shutdown_connections 2024-04-26T00:20:01.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.255+0000 7fde66fbc640 1 -- 172.21.15.37:0/942822393 wait complete. 2024-04-26T00:20:01.259 DEBUG:teuthology.orchestra.run.smithi037:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-26T00:20:01.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.407+0000 7f34a9142640 10 monclient: get_monmap_and_config 2024-04-26T00:20:01.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.407+0000 7f34a9142640 10 monclient: build_initial_monmap 2024-04-26T00:20:01.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.407+0000 7f34a9142640 10 monclient: monmap: 2024-04-26T00:20:01.391 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:01.391 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:01.391 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:01.414245+0000 2024-04-26T00:20:01.391 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:01.414245+0000 2024-04-26T00:20:01.391 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:01.391 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:01.391 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:01.391 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:01.391 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:01.391 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:01.391 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:01.392 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:01.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.407+0000 7f34a9142640 1 Processor -- start 2024-04-26T00:20:01.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.407+0000 7f34a9142640 1 -- start start 2024-04-26T00:20:01.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.407+0000 7f34a9142640 10 monclient: init 2024-04-26T00:20:01.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.407+0000 7f34a9142640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:01.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.407+0000 7f34a9142640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-04-26T00:20:01.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.407+0000 7f34a9142640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f34a409bae0 0x7f34a409bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:01.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.407+0000 7f34a9142640 10 monclient(hunting): picked mon.noname-c con 0x7f34a409bae0 addr v2:172.21.15.45:3300/0 2024-04-26T00:20:01.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.407+0000 7f34a9142640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f34a4137150 0x7f34a4072270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:01.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.407+0000 7f34a9142640 10 monclient(hunting): picked mon.noname-a con 0x7f34a4137150 addr v2:172.21.15.37:3300/0 2024-04-26T00:20:01.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.407+0000 7f34a9142640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34a4137550 0x7f34a40727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:01.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.407+0000 7f34a9142640 10 monclient(hunting): picked mon.noname-b con 0x7f34a4137550 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:01.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.407+0000 7f34a9142640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:01.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.407+0000 7f34a9142640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f34a40a1c30 con 0x7f34a4137150 2024-04-26T00:20:01.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.407+0000 7f34a9142640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:01.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.407+0000 7f34a9142640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f34a409c400 con 0x7f34a409bae0 2024-04-26T00:20:01.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.407+0000 7f34a9142640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:01.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.407+0000 7f34a9142640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f34a40a14c0 con 0x7f34a4137550 2024-04-26T00:20:01.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.407+0000 7f34a9142640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:01.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.407+0000 7f34a9142640 10 monclient(hunting): authenticate will time out at 982.719970s 2024-04-26T00:20:01.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.411+0000 7f34a259c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f34a4137150 0x7f34a4072270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:01.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.411+0000 7f34a359e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34a4137550 0x7f34a40727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:01.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.411+0000 7f34a259c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f34a4137150 0x7f34a4072270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44430/0 (socket says 172.21.15.37:44430) 2024-04-26T00:20:01.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.411+0000 7f34a259c640 1 -- 172.21.15.37:0/372845606 learned_addr learned my addr 172.21.15.37:0/372845606 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:01.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.411+0000 7f34a359e640 10 monclient(hunting): get_auth_request con 0x7f34a4137550 auth_method 0 2024-04-26T00:20:01.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.411+0000 7f34a359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:01.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.411+0000 7f34a359e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:01.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.411+0000 7f34a359e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:01.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.411+0000 7f34a259c640 10 monclient(hunting): get_auth_request con 0x7f34a4137150 auth_method 0 2024-04-26T00:20:01.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.411+0000 7f34a259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:01.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.411+0000 7f34a259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:01.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.411+0000 7f34a259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:01.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.411+0000 7f34a259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:01.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.411+0000 7f34a259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:01.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.411+0000 7f34a259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:01.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.411+0000 7f34a359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:01.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.411+0000 7f34a359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:01.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.411+0000 7f34a359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:01.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.411+0000 7f34a2d9d640 1 --2- 172.21.15.37:0/372845606 >> v2:172.21.15.45:3300/0 conn(0x7f34a409bae0 0x7f34a409bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:01.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.411+0000 7f34a259c640 10 monclient(hunting): handle_auth_done global_id 4191 payload 274 2024-04-26T00:20:01.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.411+0000 7f34a259c640 1 -- 172.21.15.37:0/372845606 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34a4137550 msgr2=0x7f34a40727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:01.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.411+0000 7f34a259c640 1 --2- 172.21.15.37:0/372845606 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34a4137550 0x7f34a40727b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:01.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.411+0000 7f34a259c640 1 -- 172.21.15.37:0/372845606 >> v2:172.21.15.45:3300/0 conn(0x7f34a409bae0 msgr2=0x7f34a409bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:01.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.411+0000 7f34a259c640 1 --2- 172.21.15.37:0/372845606 >> v2:172.21.15.45:3300/0 conn(0x7f34a409bae0 0x7f34a409bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:01.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.411+0000 7f34a259c640 10 monclient: _finish_hunting 0 2024-04-26T00:20:01.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.411+0000 7f34a259c640 1 monclient: found mon.noname-a 2024-04-26T00:20:01.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.411+0000 7f34a259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:01.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.411+0000 7f34a259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:01.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.411+0000 7f34a259c640 1 -- 172.21.15.37:0/372845606 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34a4099d10 con 0x7f34a4137150 2024-04-26T00:20:01.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.411+0000 7f34a2d9d640 10 monclient: get_auth_request con 0x7f34a409bae0 auth_method 0 2024-04-26T00:20:01.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.411+0000 7f34a2d9d640 1 --2- 172.21.15.37:0/372845606 >> v2:172.21.15.45:3300/0 conn(0x7f34a409bae0 0x7f34a409bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:20:01.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a259c640 1 -- 172.21.15.37:0/372845606 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34a4137150 msgr2=0x7f34a4072270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:01.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a259c640 1 --2- 172.21.15.37:0/372845606 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34a4137150 0x7f34a4072270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f34a40a16d0 tx=0x7f348c01c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cc0ec8808ce5675 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:01.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a1d9b640 1 -- 172.21.15.37:0/372845606 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f348c0a5610 con 0x7f34a4137150 2024-04-26T00:20:01.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:01.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:01.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a1d9b640 10 monclient: dump: 2024-04-26T00:20:01.406 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:01.406 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:01.406 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:01.406 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:01.406 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:01.407 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:01.407 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:01.407 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:01.408 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:01.408 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:01.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a1d9b640 10 monclient: _finish_auth 0 2024-04-26T00:20:01.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a1d9b640 10 monclient: _check_auth_tickets 2024-04-26T00:20:01.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:01.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a9142640 5 monclient: authenticate success, global_id 4191 2024-04-26T00:20:01.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a9142640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:20:01.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a1d9b640 1 -- 172.21.15.37:0/372845606 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f348c0a57b0 con 0x7f34a4137150 2024-04-26T00:20:01.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a1d9b640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:01.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a1d9b640 1 -- 172.21.15.37:0/372845606 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f348c0a5b80 con 0x7f34a4137150 2024-04-26T00:20:01.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a9142640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:01.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a9142640 4 set_mon_vals no callback set 2024-04-26T00:20:01.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:01.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:01.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a1d9b640 10 monclient: dump: 2024-04-26T00:20:01.409 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:01.409 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:01.409 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:01.409 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:01.409 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:01.409 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:01.409 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:01.410 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:01.410 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:01.410 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:01.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a9142640 10 monclient: shutdown 2024-04-26T00:20:01.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a9142640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:01.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a9142640 1 -- 172.21.15.37:0/372845606 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34a4137150 msgr2=0x7f34a4072270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:01.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a9142640 1 --2- 172.21.15.37:0/372845606 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34a4137150 0x7f34a4072270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f34a40a16d0 tx=0x7f348c01c880 comp rx=0 tx=0).stop 2024-04-26T00:20:01.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a9142640 1 -- 172.21.15.37:0/372845606 shutdown_connections 2024-04-26T00:20:01.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a9142640 1 --2- 172.21.15.37:0/372845606 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34a4137550 0x7f34a40727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:01.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a9142640 1 --2- 172.21.15.37:0/372845606 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34a4137150 0x7f34a4072270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:01.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a9142640 1 --2- 172.21.15.37:0/372845606 >> v2:172.21.15.45:3300/0 conn(0x7f34a409bae0 0x7f34a409bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:01.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a9142640 1 -- 172.21.15.37:0/372845606 >> 172.21.15.37:0/372845606 conn(0x7f34a40a6080 msgr2=0x7f34a40a1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:01.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a9142640 1 -- 172.21.15.37:0/372845606 shutdown_connections 2024-04-26T00:20:01.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a9142640 1 -- 172.21.15.37:0/372845606 wait complete. 2024-04-26T00:20:01.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a9142640 10 monclient: build_initial_monmap 2024-04-26T00:20:01.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a9142640 10 monclient: monmap: 2024-04-26T00:20:01.411 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:01.411 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:01.411 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:01.411 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:01.411 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:01.411 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:01.411 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:01.411 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:01.411 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:01.412 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:01.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a9142640 1 Processor -- start 2024-04-26T00:20:01.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a9142640 1 -- start start 2024-04-26T00:20:01.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a9142640 10 monclient: init 2024-04-26T00:20:01.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a9142640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:01.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a9142640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:20:01.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a9142640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34a409bae0 0x7f34a413d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:01.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a9142640 10 monclient(hunting): picked mon.noname-b con 0x7f34a409bae0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:01.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a9142640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34a4137150 0x7f34a4098380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:01.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a9142640 10 monclient(hunting): picked mon.noname-a con 0x7f34a4137150 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:01.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a9142640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34a4137550 0x7f34a40988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:01.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a9142640 10 monclient(hunting): picked mon.noname-c con 0x7f34a4137550 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:01.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a9142640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:01.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a9142640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f34a4140a80 con 0x7f34a4137150 2024-04-26T00:20:01.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a9142640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:01.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a9142640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f34a4140900 con 0x7f34a409bae0 2024-04-26T00:20:01.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a9142640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:01.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a9142640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f34a4140c00 con 0x7f34a4137550 2024-04-26T00:20:01.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a9142640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:01.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a9142640 10 monclient(hunting): authenticate will time out at 982.727441s 2024-04-26T00:20:01.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a259c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34a4137150 0x7f34a4098380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:01.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a259c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34a4137150 0x7f34a4098380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44436/0 (socket says 172.21.15.37:44436) 2024-04-26T00:20:01.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a259c640 1 -- 172.21.15.37:0/966763891 learned_addr learned my addr 172.21.15.37:0/966763891 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:01.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a259c640 10 monclient(hunting): get_auth_request con 0x7f34a4137150 auth_method 0 2024-04-26T00:20:01.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:01.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:01.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:01.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a359e640 1 --2- 172.21.15.37:0/966763891 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34a4137550 0x7f34a40988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:01.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:01.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:01.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:01.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a359e640 10 monclient(hunting): get_auth_request con 0x7f34a4137550 auth_method 0 2024-04-26T00:20:01.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:01.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a359e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:01.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a359e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:01.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a2d9d640 1 --2- 172.21.15.37:0/966763891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34a409bae0 0x7f34a413d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:01.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a2d9d640 10 monclient(hunting): get_auth_request con 0x7f34a409bae0 auth_method 0 2024-04-26T00:20:01.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:01.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a2d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:01.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:01.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:01.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:01.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:01.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a259c640 10 monclient(hunting): handle_auth_done global_id 4194 payload 824 2024-04-26T00:20:01.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a259c640 1 -- 172.21.15.37:0/966763891 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34a4137550 msgr2=0x7f34a40988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:01.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a259c640 1 --2- 172.21.15.37:0/966763891 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34a4137550 0x7f34a40988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:01.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.415+0000 7f34a259c640 1 -- 172.21.15.37:0/966763891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34a409bae0 msgr2=0x7f34a413d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:01.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.419+0000 7f34a259c640 1 --2- 172.21.15.37:0/966763891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34a409bae0 0x7f34a413d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:01.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.419+0000 7f34a259c640 10 monclient: _finish_hunting 0 2024-04-26T00:20:01.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.419+0000 7f34a259c640 1 monclient: found mon.noname-a 2024-04-26T00:20:01.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.419+0000 7f34a259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:01.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.419+0000 7f34a259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:01.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.419+0000 7f34a259c640 1 -- 172.21.15.37:0/966763891 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34a413cc30 con 0x7f34a4137150 2024-04-26T00:20:01.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.419+0000 7f34a259c640 1 -- 172.21.15.37:0/966763891 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34a4137150 msgr2=0x7f34a4098380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:01.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.419+0000 7f34a259c640 1 --2- 172.21.15.37:0/966763891 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34a4137150 0x7f34a4098380 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f348c01af30 tx=0x7f348c019620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:01.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.419+0000 7f349b7fe640 1 -- 172.21.15.37:0/966763891 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f348c0bc020 con 0x7f34a4137150 2024-04-26T00:20:01.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.419+0000 7f349b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:01.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.419+0000 7f349b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:01.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.419+0000 7f349b7fe640 10 monclient: dump: 2024-04-26T00:20:01.416 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:01.416 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:01.416 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:01.416 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:01.416 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:01.417 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:01.417 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:01.417 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:01.417 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:01.417 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:01.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.419+0000 7f349b7fe640 10 monclient: _finish_auth 0 2024-04-26T00:20:01.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.419+0000 7f349b7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:20:01.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.419+0000 7f349b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:01.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.419+0000 7f349b7fe640 1 -- 172.21.15.37:0/966763891 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f348c0b8880 con 0x7f34a4137150 2024-04-26T00:20:01.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.419+0000 7f349b7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:01.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.419+0000 7f349b7fe640 1 -- 172.21.15.37:0/966763891 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f348c0a5610 con 0x7f34a4137150 2024-04-26T00:20:01.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.419+0000 7f349b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:01.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.419+0000 7f349b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:01.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.419+0000 7f349b7fe640 10 monclient: dump: 2024-04-26T00:20:01.418 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:01.418 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:01.418 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:01.418 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:01.418 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:01.418 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:01.418 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:01.418 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:01.419 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:01.419 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:01.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.419+0000 7f34a9142640 5 monclient: authenticate success, global_id 4194 2024-04-26T00:20:01.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.419+0000 7f34a9142640 10 monclient: _renew_subs 2024-04-26T00:20:01.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.419+0000 7f34a9142640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:01.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.419+0000 7f34a159a640 4 set_mon_vals no callback set 2024-04-26T00:20:01.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.419+0000 7f34a9142640 1 -- 172.21.15.37:0/966763891 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34a413cf10 con 0x7f34a4137150 2024-04-26T00:20:01.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.419+0000 7f34a9142640 10 monclient: _renew_subs 2024-04-26T00:20:01.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.419+0000 7f34a9142640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:01.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.419+0000 7f34a9142640 1 -- 172.21.15.37:0/966763891 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34a4098e00 con 0x7f34a4137150 2024-04-26T00:20:01.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.419+0000 7f34a9142640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:01.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.419+0000 7f34a9142640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:01.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.419+0000 7f34a9142640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:01.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.419+0000 7f34a9142640 1 -- 172.21.15.37:0/966763891 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34a413ca50 con 0x7f34a4137150 2024-04-26T00:20:01.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.419+0000 7f349b7fe640 1 -- 172.21.15.37:0/966763891 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 4) ==== 43962+0+0 (secure 0 0 0) 0x7f348c0bf020 con 0x7f34a4137150 2024-04-26T00:20:01.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.419+0000 7f349b7fe640 1 --2- 172.21.15.37:0/966763891 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3484035e40 0x7f3484038300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:01.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.423+0000 7f349b7fe640 1 -- 172.21.15.37:0/966763891 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 3484+0+0 (secure 0 0 0) 0x7f348c0e3450 con 0x7f34a4137150 2024-04-26T00:20:01.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.423+0000 7f34a2d9d640 1 --2- 172.21.15.37:0/966763891 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3484035e40 0x7f3484038300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:01.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.423+0000 7f34a2d9d640 10 monclient: get_auth_request con 0x7f3484035e40 auth_method 0 2024-04-26T00:20:01.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.423+0000 7f34a2d9d640 1 --2- 172.21.15.37:0/966763891 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3484035e40 0x7f3484038300 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f349400a9a0 tx=0x7f3494008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:01.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.427+0000 7f349b7fe640 1 -- 172.21.15.37:0/966763891 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f34a413ca50 con 0x7f34a4137150 2024-04-26T00:20:01.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.427+0000 7f349b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:01.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.427+0000 7f349b7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:01.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.587+0000 7f34a9142640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-04-26T00:20:01.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.587+0000 7f34a9142640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-04-26T00:20:01.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.587+0000 7f34a9142640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:01.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:01.587+0000 7f34a9142640 1 -- 172.21.15.37:0/966763891 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) -- 0x7f34a404f580 con 0x7f34a4137150 2024-04-26T00:20:02.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.255+0000 7f349b7fe640 1 -- 172.21.15.37:0/966763891 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v14) ==== 173+0+0 (secure 0 0 0) 0x7f348c0c6070 con 0x7f34a4137150 2024-04-26T00:20:02.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.255+0000 7f349b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-04-26T00:20:02.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.255+0000 7f349b7fe640 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2024-04-26T00:20:02.237 INFO:teuthology.orchestra.run.smithi037.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-26T00:20:02.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.259+0000 7f34a9142640 1 -- 172.21.15.37:0/966763891 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3484035e40 msgr2=0x7f3484038300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:02.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.259+0000 7f34a9142640 1 --2- 172.21.15.37:0/966763891 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3484035e40 0x7f3484038300 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f349400a9a0 tx=0x7f3494008040 comp rx=0 tx=0).stop 2024-04-26T00:20:02.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.259+0000 7f34a9142640 10 monclient: shutdown 2024-04-26T00:20:02.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.259+0000 7f34a9142640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:02.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.259+0000 7f34a9142640 1 -- 172.21.15.37:0/966763891 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34a4137150 msgr2=0x7f34a4098380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:02.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.259+0000 7f34a9142640 1 --2- 172.21.15.37:0/966763891 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34a4137150 0x7f34a4098380 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f348c01af30 tx=0x7f348c019620 comp rx=0 tx=0).stop 2024-04-26T00:20:02.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.259+0000 7f34a9142640 1 -- 172.21.15.37:0/966763891 shutdown_connections 2024-04-26T00:20:02.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.259+0000 7f34a9142640 1 --2- 172.21.15.37:0/966763891 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3484035e40 0x7f3484038300 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:02.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.259+0000 7f34a9142640 1 --2- 172.21.15.37:0/966763891 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34a4137550 0x7f34a40988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:02.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.259+0000 7f34a9142640 1 --2- 172.21.15.37:0/966763891 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34a4137150 0x7f34a4098380 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f348c01af30 tx=0x7f348c019620 comp rx=0 tx=0).stop 2024-04-26T00:20:02.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.259+0000 7f34a9142640 1 --2- 172.21.15.37:0/966763891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34a409bae0 0x7f34a413d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:02.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.259+0000 7f34a9142640 1 -- 172.21.15.37:0/966763891 >> 172.21.15.37:0/966763891 conn(0x7f34a40a6080 msgr2=0x7f34a4133810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:02.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.263+0000 7f34a9142640 1 -- 172.21.15.37:0/966763891 shutdown_connections 2024-04-26T00:20:02.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.263+0000 7f34a9142640 1 -- 172.21.15.37:0/966763891 wait complete. 2024-04-26T00:20:02.273 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-26T00:20:02.273 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-26T00:20:02.273 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-26T00:20:02.273 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-26T00:20:02.274 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-26T00:20:02.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.423+0000 7f83d8ede640 10 monclient: get_monmap_and_config 2024-04-26T00:20:02.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.423+0000 7f83d8ede640 10 monclient: build_initial_monmap 2024-04-26T00:20:02.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.423+0000 7f83d8ede640 10 monclient: monmap: 2024-04-26T00:20:02.407 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:02.407 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:02.407 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:02.430858+0000 2024-04-26T00:20:02.407 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:02.430858+0000 2024-04-26T00:20:02.408 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:02.408 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:02.408 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:02.408 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:02.408 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:02.408 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:02.408 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:02.408 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:02.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.423+0000 7f83d8ede640 1 Processor -- start 2024-04-26T00:20:02.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.423+0000 7f83d8ede640 1 -- start start 2024-04-26T00:20:02.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.423+0000 7f83d8ede640 10 monclient: init 2024-04-26T00:20:02.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.427+0000 7f83d8ede640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:02.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.427+0000 7f83d8ede640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-04-26T00:20:02.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.427+0000 7f83d8ede640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f83d4136e10 addr v1:172.21.15.45:6789/0 2024-04-26T00:20:02.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.427+0000 7f83d8ede640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f83d4137900 0x7f83d409c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:02.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.427+0000 7f83d8ede640 10 monclient(hunting): picked mon.noname-c con 0x7f83d4137900 addr v2:172.21.15.45:3300/0 2024-04-26T00:20:02.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.427+0000 7f83d8ede640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f83d413c340 addr v1:172.21.15.37:6789/0 2024-04-26T00:20:02.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.427+0000 7f83d8ede640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83d409b680 con 0x7f83d413c340 2024-04-26T00:20:02.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.427+0000 7f83d8ede640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83d409bbd0 con 0x7f83d4136e10 2024-04-26T00:20:02.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.427+0000 7f83d8ede640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:02.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.427+0000 7f83d8ede640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f83d409b8f0 con 0x7f83d4137900 2024-04-26T00:20:02.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.427+0000 7f83d8ede640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:02.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.427+0000 7f83d8ede640 10 monclient(hunting): authenticate will time out at 983.736591s 2024-04-26T00:20:02.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.427+0000 7f83d2d9d640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f83d413c340 0x7f83d413e730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:55764/0 (socket says 172.21.15.37:55764) 2024-04-26T00:20:02.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.427+0000 7f83d2d9d640 1 -- 172.21.15.37:0/3121254810 learned_addr learned my addr 172.21.15.37:0/3121254810 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:20:02.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.427+0000 7f83cbfff640 1 --2- 172.21.15.37:0/3121254810 >> v2:172.21.15.45:3300/0 conn(0x7f83d4137900 0x7f83d409c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:02.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.427+0000 7f83cbfff640 10 monclient(hunting): get_auth_request con 0x7f83d4137900 auth_method 0 2024-04-26T00:20:02.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.427+0000 7f83cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:02.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.427+0000 7f83cbfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:02.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.427+0000 7f83cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:02.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.427+0000 7f83cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:02.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.427+0000 7f83cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:02.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.427+0000 7f83cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:02.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.427+0000 7f83cbfff640 10 monclient(hunting): handle_auth_done global_id 4189 payload 274 2024-04-26T00:20:02.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.427+0000 7f83cbfff640 1 -- 172.21.15.37:0/3121254810 >> v1:172.21.15.45:6789/0 conn(0x7f83d4136e10 legacy=0x7f83d41371f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:02.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.427+0000 7f83cbfff640 1 -- 172.21.15.37:0/3121254810 >> v1:172.21.15.37:6789/0 conn(0x7f83d413c340 legacy=0x7f83d413e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:02.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.427+0000 7f83cbfff640 10 monclient: _finish_hunting 0 2024-04-26T00:20:02.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.427+0000 7f83cbfff640 1 monclient: found mon.noname-c 2024-04-26T00:20:02.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.427+0000 7f83cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:02.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.427+0000 7f83cbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:20:02.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.427+0000 7f83cbfff640 1 -- 172.21.15.37:0/3121254810 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83d40a5d90 con 0x7f83d4137900 2024-04-26T00:20:02.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.431+0000 7f83cbfff640 1 -- 172.21.15.37:0/3121254810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83d4137900 msgr2=0x7f83d409c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:02.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.431+0000 7f83cbfff640 1 --2- 172.21.15.37:0/3121254810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83d4137900 0x7f83d409c5b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f83bc0097c0 tx=0x7f83bc01c2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a3ca6de7eee7128 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:02.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.431+0000 7f83d1d9b640 1 -- 172.21.15.37:0/3121254810 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83bc0b0070 con 0x7f83d4137900 2024-04-26T00:20:02.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.431+0000 7f83d1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:02.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.431+0000 7f83d1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:20:02.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.431+0000 7f83d1d9b640 10 monclient: dump: 2024-04-26T00:20:02.416 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:02.416 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:02.416 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:02.416 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:02.416 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:02.416 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:02.416 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:02.416 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:02.416 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:02.416 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:02.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.431+0000 7f83d1d9b640 10 monclient: _finish_auth 0 2024-04-26T00:20:02.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.431+0000 7f83d1d9b640 10 monclient: _check_auth_tickets 2024-04-26T00:20:02.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.431+0000 7f83d1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:02.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.431+0000 7f83d8ede640 5 monclient: authenticate success, global_id 4189 2024-04-26T00:20:02.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.431+0000 7f83d8ede640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:20:02.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.431+0000 7f83d1d9b640 1 -- 172.21.15.37:0/3121254810 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83bc0ab600 con 0x7f83d4137900 2024-04-26T00:20:02.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.431+0000 7f83d1d9b640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:02.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.431+0000 7f83d8ede640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:02.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.431+0000 7f83d8ede640 4 set_mon_vals no callback set 2024-04-26T00:20:02.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.431+0000 7f83d1d9b640 1 -- 172.21.15.37:0/3121254810 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83bc0ab9d0 con 0x7f83d4137900 2024-04-26T00:20:02.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.431+0000 7f83d1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:02.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.431+0000 7f83d1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:20:02.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.431+0000 7f83d1d9b640 10 monclient: dump: 2024-04-26T00:20:02.417 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:02.417 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:02.417 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:02.417 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:02.417 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:02.417 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:02.418 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:02.418 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:02.418 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:02.418 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:02.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.431+0000 7f83d8ede640 10 monclient: shutdown 2024-04-26T00:20:02.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.431+0000 7f83d8ede640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:02.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.431+0000 7f83d8ede640 1 -- 172.21.15.37:0/3121254810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83d4137900 msgr2=0x7f83d409c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:02.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.431+0000 7f83d8ede640 1 --2- 172.21.15.37:0/3121254810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83d4137900 0x7f83d409c5b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f83bc0097c0 tx=0x7f83bc01c2e0 comp rx=0 tx=0).stop 2024-04-26T00:20:02.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.431+0000 7f83d8ede640 1 -- 172.21.15.37:0/3121254810 shutdown_connections 2024-04-26T00:20:02.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.431+0000 7f83d8ede640 1 --2- 172.21.15.37:0/3121254810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83d4137900 0x7f83d409c5b0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:02.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.431+0000 7f83d8ede640 1 -- 172.21.15.37:0/3121254810 >> 172.21.15.37:0/3121254810 conn(0x7f83d40a64e0 msgr2=0x7f83d40a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:02.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.431+0000 7f83d8ede640 1 -- 172.21.15.37:0/3121254810 shutdown_connections 2024-04-26T00:20:02.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.431+0000 7f83d8ede640 1 -- 172.21.15.37:0/3121254810 wait complete. 2024-04-26T00:20:02.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.431+0000 7f83d8ede640 10 monclient: build_initial_monmap 2024-04-26T00:20:02.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.431+0000 7f83d8ede640 10 monclient: monmap: 2024-04-26T00:20:02.419 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:02.419 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:02.419 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:02.419 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:02.419 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:02.419 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:02.419 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:02.419 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:02.419 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:02.419 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:02.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83d8ede640 1 Processor -- start 2024-04-26T00:20:02.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83d8ede640 1 -- start start 2024-04-26T00:20:02.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83d8ede640 10 monclient: init 2024-04-26T00:20:02.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83d8ede640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:02.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83d8ede640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:20:02.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83d8ede640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83d4136e10 0x7f83d41ced70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:02.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83d8ede640 10 monclient(hunting): picked mon.noname-b con 0x7f83d4136e10 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:02.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83d8ede640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f83d4137900 0x7f83d4135d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:02.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83d8ede640 10 monclient(hunting): picked mon.noname-c con 0x7f83d4137900 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:02.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83d8ede640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83d413c340 0x7f83d4137e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:02.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83d8ede640 10 monclient(hunting): picked mon.noname-a con 0x7f83d413c340 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:02.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83d8ede640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:02.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83d8ede640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f83d4142a00 con 0x7f83d413c340 2024-04-26T00:20:02.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83d8ede640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:02.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83d8ede640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f83d4142880 con 0x7f83d4136e10 2024-04-26T00:20:02.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83d8ede640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:02.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83d8ede640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f83d4142b80 con 0x7f83d4137900 2024-04-26T00:20:02.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83d8ede640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:02.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83d8ede640 10 monclient(hunting): authenticate will time out at 983.744887s 2024-04-26T00:20:02.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83cbfff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f83d4137900 0x7f83d4135d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:02.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83d2d9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83d413c340 0x7f83d4137e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:02.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83cbfff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f83d4137900 0x7f83d4135d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:57694/0 (socket says 172.21.15.37:57694) 2024-04-26T00:20:02.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83cbfff640 1 -- 172.21.15.37:0/158076198 learned_addr learned my addr 172.21.15.37:0/158076198 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:02.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83cbfff640 10 monclient(hunting): get_auth_request con 0x7f83d4137900 auth_method 0 2024-04-26T00:20:02.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:02.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83cbfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:02.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:02.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83d2d9d640 10 monclient(hunting): get_auth_request con 0x7f83d413c340 auth_method 0 2024-04-26T00:20:02.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83d2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:02.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83d2d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:02.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83d2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:02.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:02.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:02.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:02.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83d2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:02.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83d2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:02.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83d2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:02.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83d259c640 1 --2- 172.21.15.37:0/158076198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83d4136e10 0x7f83d41ced70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:02.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83d259c640 10 monclient(hunting): get_auth_request con 0x7f83d4136e10 auth_method 0 2024-04-26T00:20:02.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83d259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:02.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83d259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:02.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83d259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:02.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83cbfff640 10 monclient(hunting): handle_auth_done global_id 4184 payload 824 2024-04-26T00:20:02.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83cbfff640 1 -- 172.21.15.37:0/158076198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83d4136e10 msgr2=0x7f83d41ced70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:02.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83cbfff640 1 --2- 172.21.15.37:0/158076198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83d4136e10 0x7f83d41ced70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:02.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83cbfff640 1 -- 172.21.15.37:0/158076198 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83d413c340 msgr2=0x7f83d4137e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:02.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83cbfff640 1 --2- 172.21.15.37:0/158076198 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83d413c340 0x7f83d4137e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:02.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83cbfff640 10 monclient: _finish_hunting 0 2024-04-26T00:20:02.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83cbfff640 1 monclient: found mon.noname-c 2024-04-26T00:20:02.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:02.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83cbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:20:02.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83cbfff640 1 -- 172.21.15.37:0/158076198 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83d413b6e0 con 0x7f83d4137900 2024-04-26T00:20:02.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83d2d9d640 1 --2- 172.21.15.37:0/158076198 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83d413c340 0x7f83d4137e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:02.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83d259c640 1 --2- 172.21.15.37:0/158076198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83d4136e10 0x7f83d41ced70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:20:02.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83cbfff640 1 -- 172.21.15.37:0/158076198 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f83d4137900 msgr2=0x7f83d4135d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:02.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83cbfff640 1 --2- 172.21.15.37:0/158076198 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f83d4137900 0x7f83d4135d90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f83bc0ae040 tx=0x7f83bc01ab20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:02.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83caffd640 1 -- 172.21.15.37:0/158076198 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83bc0b0050 con 0x7f83d4137900 2024-04-26T00:20:02.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:02.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83caffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:20:02.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83caffd640 10 monclient: dump: 2024-04-26T00:20:02.428 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:02.428 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:02.428 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:02.428 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:02.428 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:02.428 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:02.428 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:02.428 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:02.428 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:02.428 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:02.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83caffd640 10 monclient: _finish_auth 0 2024-04-26T00:20:02.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83caffd640 10 monclient: _check_auth_tickets 2024-04-26T00:20:02.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:02.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83caffd640 1 -- 172.21.15.37:0/158076198 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83bc0bc070 con 0x7f83d4137900 2024-04-26T00:20:02.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83caffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:02.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83caffd640 1 -- 172.21.15.37:0/158076198 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83bc0aae50 con 0x7f83d4137900 2024-04-26T00:20:02.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:02.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83caffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:02.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83caffd640 10 monclient: dump: 2024-04-26T00:20:02.429 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:02.429 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:02.429 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:02.429 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:02.429 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:02.429 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:02.429 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:02.429 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:02.430 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:02.430 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:02.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.439+0000 7f83d8ede640 5 monclient: authenticate success, global_id 4184 2024-04-26T00:20:02.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.435+0000 7f83d159a640 4 set_mon_vals no callback set 2024-04-26T00:20:02.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.439+0000 7f83d8ede640 10 monclient: _renew_subs 2024-04-26T00:20:02.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.439+0000 7f83d8ede640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:02.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.439+0000 7f83d8ede640 1 -- 172.21.15.37:0/158076198 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83d409aca0 con 0x7f83d4137900 2024-04-26T00:20:02.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.439+0000 7f83d8ede640 10 monclient: _renew_subs 2024-04-26T00:20:02.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.439+0000 7f83d8ede640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:02.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.439+0000 7f83d8ede640 1 -- 172.21.15.37:0/158076198 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f83d41cfbf0 con 0x7f83d4137900 2024-04-26T00:20:02.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.439+0000 7f83caffd640 1 -- 172.21.15.37:0/158076198 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 4) ==== 43962+0+0 (secure 0 0 0) 0x7f83bc0c1460 con 0x7f83d4137900 2024-04-26T00:20:02.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.439+0000 7f83d8ede640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:02.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.439+0000 7f83d8ede640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:02.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.439+0000 7f83d8ede640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:02.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.439+0000 7f83d8ede640 1 -- 172.21.15.37:0/158076198 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83d4075910 con 0x7f83d4137900 2024-04-26T00:20:02.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.439+0000 7f83caffd640 1 --2- 172.21.15.37:0/158076198 >> v2:172.21.15.45:6812/2379918623 conn(0x7f83b4035fc0 0x7f83b4038480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:02.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.439+0000 7f83caffd640 1 -- 172.21.15.37:0/158076198 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(14..14 src has 1..14) ==== 3506+0+0 (secure 0 0 0) 0x7f83bc0e33a0 con 0x7f83d4137900 2024-04-26T00:20:02.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.439+0000 7f83d259c640 1 --2- 172.21.15.37:0/158076198 >> v2:172.21.15.45:6812/2379918623 conn(0x7f83b4035fc0 0x7f83b4038480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:02.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.439+0000 7f83d259c640 10 monclient: get_auth_request con 0x7f83b4035fc0 auth_method 0 2024-04-26T00:20:02.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.443+0000 7f83d259c640 1 --2- 172.21.15.37:0/158076198 >> v2:172.21.15.45:6812/2379918623 conn(0x7f83b4035fc0 0x7f83b4038480 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f83c4003ec0 tx=0x7f83c4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:02.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.443+0000 7f83caffd640 1 -- 172.21.15.37:0/158076198 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f83bc015ff0 con 0x7f83d4137900 2024-04-26T00:20:02.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.443+0000 7f83caffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:02.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.443+0000 7f83caffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:02.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.659+0000 7f83d8ede640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-04-26T00:20:02.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.659+0000 7f83d8ede640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-04-26T00:20:02.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.659+0000 7f83d8ede640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:02.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.659+0000 7f83d8ede640 1 -- 172.21.15.37:0/158076198 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f83d413b360 con 0x7f83d4137900 2024-04-26T00:20:02.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.663+0000 7f83caffd640 1 -- 172.21.15.37:0/158076198 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v4) ==== 74+0+84021 (secure 0 0 0) 0x7f83bc0e22a0 con 0x7f83d4137900 2024-04-26T00:20:02.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.663+0000 7f83caffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-04-26T00:20:02.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.663+0000 7f83caffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:02.648 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:20:02.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.667+0000 7f83d8ede640 1 -- 172.21.15.37:0/158076198 >> v2:172.21.15.45:6812/2379918623 conn(0x7f83b4035fc0 msgr2=0x7f83b4038480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:02.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.667+0000 7f83d8ede640 1 --2- 172.21.15.37:0/158076198 >> v2:172.21.15.45:6812/2379918623 conn(0x7f83b4035fc0 0x7f83b4038480 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f83c4003ec0 tx=0x7f83c4009040 comp rx=0 tx=0).stop 2024-04-26T00:20:02.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.667+0000 7f83d8ede640 10 monclient: shutdown 2024-04-26T00:20:02.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.667+0000 7f83d8ede640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:02.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.667+0000 7f83d8ede640 1 -- 172.21.15.37:0/158076198 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f83d4137900 msgr2=0x7f83d4135d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:02.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.667+0000 7f83d8ede640 1 --2- 172.21.15.37:0/158076198 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f83d4137900 0x7f83d4135d90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f83bc0ae040 tx=0x7f83bc01ab20 comp rx=0 tx=0).stop 2024-04-26T00:20:02.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.667+0000 7f83d8ede640 1 -- 172.21.15.37:0/158076198 shutdown_connections 2024-04-26T00:20:02.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.667+0000 7f83d8ede640 1 --2- 172.21.15.37:0/158076198 >> v2:172.21.15.45:6812/2379918623 conn(0x7f83b4035fc0 0x7f83b4038480 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:02.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.667+0000 7f83d8ede640 1 --2- 172.21.15.37:0/158076198 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83d413c340 0x7f83d4137e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:02.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.667+0000 7f83d8ede640 1 --2- 172.21.15.37:0/158076198 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f83d4137900 0x7f83d4135d90 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:02.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.667+0000 7f83d8ede640 1 --2- 172.21.15.37:0/158076198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83d4136e10 0x7f83d41ced70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:02.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.667+0000 7f83d8ede640 1 -- 172.21.15.37:0/158076198 >> 172.21.15.37:0/158076198 conn(0x7f83d40a64e0 msgr2=0x7f83d40a73e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:02.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.667+0000 7f83d8ede640 1 -- 172.21.15.37:0/158076198 shutdown_connections 2024-04-26T00:20:02.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.667+0000 7f83d8ede640 1 -- 172.21.15.37:0/158076198 wait complete. 2024-04-26T00:20:02.667 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":4,"flags":0,"active_gid":4107,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":2379918623}]},"active_addr":"172.21.15.45:6812/2379918623","active_change":"2024-04-26T00:19:55.137986+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.45:0","nonce":1207469177}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.45:0","nonce":1239105620}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.45:0","nonce":3592496103}]}]} 2024-04-26T00:20:02.669 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-26T00:20:02.669 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-26T00:20:02.669 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T00:20:02.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9293adc640 10 monclient: get_monmap_and_config 2024-04-26T00:20:02.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9293adc640 10 monclient: build_initial_monmap 2024-04-26T00:20:02.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9293adc640 10 monclient: monmap: 2024-04-26T00:20:02.785 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:02.785 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:02.785 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:02.808744+0000 2024-04-26T00:20:02.785 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:02.808744+0000 2024-04-26T00:20:02.785 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:02.785 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:02.785 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:02.785 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:02.785 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:02.785 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:02.786 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:02.786 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:02.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9293adc640 1 Processor -- start 2024-04-26T00:20:02.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9293adc640 1 -- start start 2024-04-26T00:20:02.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9293adc640 10 monclient: init 2024-04-26T00:20:02.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9293adc640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:02.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9293adc640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-04-26T00:20:02.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9293adc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f928c0725a0 addr v1:172.21.15.45:6789/0 2024-04-26T00:20:02.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9293adc640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f928c133ce0 0x7f928c072980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:02.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9293adc640 10 monclient(hunting): picked mon.noname-a con 0x7f928c133ce0 addr v2:172.21.15.37:3300/0 2024-04-26T00:20:02.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9293adc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f928c0a1bc0 0x7f928c13f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:02.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9293adc640 10 monclient(hunting): picked mon.noname-b con 0x7f928c0a1bc0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:02.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9293adc640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f928c06e860 con 0x7f928c0725a0 2024-04-26T00:20:02.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9293adc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:02.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9293adc640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f928c134140 con 0x7f928c133ce0 2024-04-26T00:20:02.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9293adc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:02.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9293adc640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f928c06eb40 con 0x7f928c0a1bc0 2024-04-26T00:20:02.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9293adc640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:02.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9293adc640 10 monclient(hunting): authenticate will time out at 984.114486s 2024-04-26T00:20:02.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9291077640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f928c133ce0 0x7f928c072980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:02.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9292079640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f928c0a1bc0 0x7f928c13f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:02.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9291077640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f928c133ce0 0x7f928c072980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44454/0 (socket says 172.21.15.37:44454) 2024-04-26T00:20:02.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9292079640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f928c0a1bc0 0x7f928c13f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:57704/0 (socket says 172.21.15.37:57704) 2024-04-26T00:20:02.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9291077640 1 -- 172.21.15.37:0/3223062084 learned_addr learned my addr 172.21.15.37:0/3223062084 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:02.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9291077640 10 monclient(hunting): get_auth_request con 0x7f928c133ce0 auth_method 0 2024-04-26T00:20:02.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9291077640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:02.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9291077640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:02.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9291077640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:02.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9292079640 10 monclient(hunting): get_auth_request con 0x7f928c0a1bc0 auth_method 0 2024-04-26T00:20:02.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9292079640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:02.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9292079640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:02.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9292079640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:02.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9291077640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:02.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9291077640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:02.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9291077640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:02.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9292079640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:02.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9292079640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:02.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9292079640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:02.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9291077640 10 monclient(hunting): handle_auth_done global_id 4206 payload 274 2024-04-26T00:20:02.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9291077640 1 -- 172.21.15.37:0/3223062084 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f928c0a1bc0 msgr2=0x7f928c13f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:02.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9291077640 1 --2- 172.21.15.37:0/3223062084 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f928c0a1bc0 0x7f928c13f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:02.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9291077640 1 -- 172.21.15.37:0/3223062084 >> v1:172.21.15.45:6789/0 conn(0x7f928c0725a0 legacy=0x7f928c0a14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:02.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9291077640 10 monclient: _finish_hunting 0 2024-04-26T00:20:02.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9291077640 1 monclient: found mon.noname-a 2024-04-26T00:20:02.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9291077640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:02.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9291077640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:02.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9291077640 1 -- 172.21.15.37:0/3223062084 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f928c06ee20 con 0x7f928c133ce0 2024-04-26T00:20:02.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.803+0000 7f9292079640 1 --2- 172.21.15.37:0/3223062084 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f928c0a1bc0 0x7f928c13f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:02.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.807+0000 7f9291077640 1 -- 172.21.15.37:0/3223062084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f928c133ce0 msgr2=0x7f928c072980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:02.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.807+0000 7f9291077640 1 --2- 172.21.15.37:0/3223062084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f928c133ce0 0x7f928c072980 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f927c009870 tx=0x7f927c01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dd8920093522e90 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:02.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.807+0000 7f9290876640 1 -- 172.21.15.37:0/3223062084 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f927c0b0070 con 0x7f928c133ce0 2024-04-26T00:20:02.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.807+0000 7f9290876640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:02.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.807+0000 7f9290876640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:02.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.807+0000 7f9290876640 10 monclient: dump: 2024-04-26T00:20:02.796 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:02.796 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:02.796 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:02.796 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:02.796 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:02.796 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:02.796 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:02.796 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:02.796 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:02.796 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:02.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.807+0000 7f9290876640 10 monclient: _finish_auth 0 2024-04-26T00:20:02.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.807+0000 7f9290876640 10 monclient: _check_auth_tickets 2024-04-26T00:20:02.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.807+0000 7f9290876640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:02.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.807+0000 7f9290876640 1 -- 172.21.15.37:0/3223062084 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f927c0ab600 con 0x7f928c133ce0 2024-04-26T00:20:02.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.807+0000 7f9290876640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:02.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.807+0000 7f9293adc640 5 monclient: authenticate success, global_id 4206 2024-04-26T00:20:02.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.807+0000 7f9290876640 1 -- 172.21.15.37:0/3223062084 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f927c0ab9d0 con 0x7f928c133ce0 2024-04-26T00:20:02.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.807+0000 7f9290876640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:02.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.807+0000 7f9290876640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:02.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.807+0000 7f9290876640 10 monclient: dump: 2024-04-26T00:20:02.797 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:02.797 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:02.797 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:02.797 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:02.797 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:02.797 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:02.797 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:02.797 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:02.798 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:02.798 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:02.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.807+0000 7f9293adc640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:02.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.807+0000 7f9293adc640 4 set_mon_vals no callback set 2024-04-26T00:20:02.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.807+0000 7f9293adc640 10 monclient: shutdown 2024-04-26T00:20:02.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.807+0000 7f9293adc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:02.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.807+0000 7f9293adc640 1 -- 172.21.15.37:0/3223062084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f928c133ce0 msgr2=0x7f928c072980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:02.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.807+0000 7f9293adc640 1 --2- 172.21.15.37:0/3223062084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f928c133ce0 0x7f928c072980 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f927c009870 tx=0x7f927c01c3e0 comp rx=0 tx=0).stop 2024-04-26T00:20:02.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9293adc640 1 -- 172.21.15.37:0/3223062084 shutdown_connections 2024-04-26T00:20:02.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9293adc640 1 --2- 172.21.15.37:0/3223062084 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f928c0a1bc0 0x7f928c13f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:02.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9293adc640 1 --2- 172.21.15.37:0/3223062084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f928c133ce0 0x7f928c072980 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:02.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9293adc640 1 -- 172.21.15.37:0/3223062084 >> 172.21.15.37:0/3223062084 conn(0x7f928c0a6250 msgr2=0x7f928c0a2610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:02.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9293adc640 1 -- 172.21.15.37:0/3223062084 shutdown_connections 2024-04-26T00:20:02.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9293adc640 1 -- 172.21.15.37:0/3223062084 wait complete. 2024-04-26T00:20:02.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9293adc640 10 monclient: build_initial_monmap 2024-04-26T00:20:02.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9293adc640 10 monclient: monmap: 2024-04-26T00:20:02.799 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:02.799 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:02.799 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:02.799 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:02.799 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:02.799 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:02.799 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:02.799 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:02.799 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:02.799 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:02.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9293adc640 1 Processor -- start 2024-04-26T00:20:02.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9293adc640 1 -- start start 2024-04-26T00:20:02.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9293adc640 10 monclient: init 2024-04-26T00:20:02.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9293adc640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:02.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9293adc640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:20:02.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9293adc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f928c0725a0 0x7f928c13dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:02.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9293adc640 10 monclient(hunting): picked mon.noname-b con 0x7f928c0725a0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:02.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9293adc640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f928c0a1bc0 0x7f928c1395f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:02.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9293adc640 10 monclient(hunting): picked mon.noname-a con 0x7f928c0a1bc0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:02.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9293adc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f928c133ce0 0x7f928c139b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:02.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9293adc640 10 monclient(hunting): picked mon.noname-c con 0x7f928c133ce0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:02.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9293adc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:02.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9293adc640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f928c1421a0 con 0x7f928c0a1bc0 2024-04-26T00:20:02.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9293adc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:02.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9293adc640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f928c142020 con 0x7f928c0725a0 2024-04-26T00:20:02.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9293adc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:02.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9293adc640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f928c142320 con 0x7f928c133ce0 2024-04-26T00:20:02.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9293adc640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:02.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9293adc640 10 monclient(hunting): authenticate will time out at 984.121681s 2024-04-26T00:20:02.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9291077640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f928c0a1bc0 0x7f928c1395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:02.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9292079640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f928c133ce0 0x7f928c139b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:02.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9292079640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f928c133ce0 0x7f928c139b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:57714/0 (socket says 172.21.15.37:57714) 2024-04-26T00:20:02.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9292079640 1 -- 172.21.15.37:0/886897774 learned_addr learned my addr 172.21.15.37:0/886897774 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:02.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9292079640 10 monclient(hunting): get_auth_request con 0x7f928c133ce0 auth_method 0 2024-04-26T00:20:02.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9292079640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:02.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9292079640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:02.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9292079640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:02.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9291077640 1 --2- 172.21.15.37:0/886897774 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f928c0a1bc0 0x7f928c1395f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44470/0 (socket says 172.21.15.37:44470) 2024-04-26T00:20:02.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9291077640 10 monclient(hunting): get_auth_request con 0x7f928c0a1bc0 auth_method 0 2024-04-26T00:20:02.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9291077640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:02.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9291077640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:02.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9291077640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:02.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9292079640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:02.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9292079640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:02.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9292079640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:02.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9291077640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:02.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9291077640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:02.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9291077640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:02.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9291878640 1 --2- 172.21.15.37:0/886897774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f928c0725a0 0x7f928c13dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:02.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9292079640 10 monclient(hunting): handle_auth_done global_id 4196 payload 824 2024-04-26T00:20:02.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9292079640 1 -- 172.21.15.37:0/886897774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f928c0725a0 msgr2=0x7f928c13dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:02.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9292079640 1 --2- 172.21.15.37:0/886897774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f928c0725a0 0x7f928c13dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:02.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9292079640 1 -- 172.21.15.37:0/886897774 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f928c0a1bc0 msgr2=0x7f928c1395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:02.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9292079640 1 --2- 172.21.15.37:0/886897774 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f928c0a1bc0 0x7f928c1395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:02.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9292079640 10 monclient: _finish_hunting 0 2024-04-26T00:20:02.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9292079640 1 monclient: found mon.noname-c 2024-04-26T00:20:02.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9292079640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:02.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9292079640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:20:02.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9292079640 1 -- 172.21.15.37:0/886897774 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f928c13d200 con 0x7f928c133ce0 2024-04-26T00:20:02.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9291878640 10 monclient: get_auth_request con 0x7f928c0725a0 auth_method 0 2024-04-26T00:20:02.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.811+0000 7f9291878640 1 --2- 172.21.15.37:0/886897774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f928c0725a0 0x7f928c13dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:20:02.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.815+0000 7f9292079640 1 -- 172.21.15.37:0/886897774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f928c133ce0 msgr2=0x7f928c139b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:02.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.815+0000 7f9292079640 1 --2- 172.21.15.37:0/886897774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f928c133ce0 0x7f928c139b30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f928400efc0 tx=0x7f9284009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:02.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.815+0000 7f92827fc640 1 -- 172.21.15.37:0/886897774 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9284018030 con 0x7f928c133ce0 2024-04-26T00:20:02.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.815+0000 7f92827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:02.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.815+0000 7f92827fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:20:02.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.815+0000 7f92827fc640 10 monclient: dump: 2024-04-26T00:20:02.806 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:02.806 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:02.806 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:02.806 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:02.806 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:02.806 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:02.806 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:02.806 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:02.806 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:02.806 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:02.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.815+0000 7f92827fc640 10 monclient: _finish_auth 0 2024-04-26T00:20:02.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.815+0000 7f92827fc640 10 monclient: _check_auth_tickets 2024-04-26T00:20:02.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.815+0000 7f92827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:02.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.815+0000 7f9293adc640 5 monclient: authenticate success, global_id 4196 2024-04-26T00:20:02.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.815+0000 7f92827fc640 1 -- 172.21.15.37:0/886897774 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92840039a0 con 0x7f928c133ce0 2024-04-26T00:20:02.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.815+0000 7f9293adc640 10 monclient: _renew_subs 2024-04-26T00:20:02.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.815+0000 7f9293adc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:02.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.815+0000 7f9293adc640 1 -- 172.21.15.37:0/886897774 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f928c13bad0 con 0x7f928c133ce0 2024-04-26T00:20:02.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.815+0000 7f92827fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:02.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.815+0000 7f92827fc640 1 -- 172.21.15.37:0/886897774 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9284003d70 con 0x7f928c133ce0 2024-04-26T00:20:02.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.815+0000 7f92827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:02.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.815+0000 7f92827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:02.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.815+0000 7f9283fff640 4 set_mon_vals no callback set 2024-04-26T00:20:02.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.815+0000 7f92827fc640 10 monclient: dump: 2024-04-26T00:20:02.808 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:02.808 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:02.808 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:02.808 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:02.808 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:02.808 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:02.808 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:02.808 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:02.808 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:02.808 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:02.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.815+0000 7f9293adc640 10 monclient: _renew_subs 2024-04-26T00:20:02.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.815+0000 7f9293adc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:02.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.815+0000 7f9293adc640 1 -- 172.21.15.37:0/886897774 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f928c0a58a0 con 0x7f928c133ce0 2024-04-26T00:20:02.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.815+0000 7f9293adc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:02.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.815+0000 7f9293adc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:02.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.815+0000 7f9293adc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:02.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.815+0000 7f9293adc640 1 -- 172.21.15.37:0/886897774 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f928c066f40 con 0x7f928c133ce0 2024-04-26T00:20:02.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.815+0000 7f92827fc640 1 -- 172.21.15.37:0/886897774 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 4) ==== 43962+0+0 (secure 0 0 0) 0x7f928401b070 con 0x7f928c133ce0 2024-04-26T00:20:02.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.815+0000 7f92827fc640 1 --2- 172.21.15.37:0/886897774 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9270035e20 0x7f92700382e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:02.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.815+0000 7f92827fc640 1 -- 172.21.15.37:0/886897774 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(14..14 src has 1..14) ==== 3506+0+0 (secure 0 0 0) 0x7f92840492b0 con 0x7f928c133ce0 2024-04-26T00:20:02.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.815+0000 7f9291878640 1 --2- 172.21.15.37:0/886897774 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9270035e20 0x7f92700382e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:02.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.815+0000 7f9291878640 10 monclient: get_auth_request con 0x7f9270035e20 auth_method 0 2024-04-26T00:20:02.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.819+0000 7f9291878640 1 --2- 172.21.15.37:0/886897774 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9270035e20 0x7f92700382e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9288003e40 tx=0x7f9288009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:02.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.819+0000 7f92827fc640 1 -- 172.21.15.37:0/886897774 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9284022070 con 0x7f928c133ce0 2024-04-26T00:20:02.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.819+0000 7f92827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:02.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.819+0000 7f92827fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:02.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.983+0000 7f9293adc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-26T00:20:02.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.983+0000 7f9293adc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-26T00:20:02.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.983+0000 7f9293adc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:02.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.983+0000 7f9293adc640 1 -- 172.21.15.37:0/886897774 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f928c06c1e0 con 0x7f928c133ce0 2024-04-26T00:20:02.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.987+0000 7f92827fc640 1 -- 172.21.15.37:0/886897774 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) ==== 74+0+10787 (secure 0 0 0) 0x7f9284010040 con 0x7f928c133ce0 2024-04-26T00:20:02.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.987+0000 7f92827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-26T00:20:02.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.987+0000 7f92827fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:02.969 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:20:02.969 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":14,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","created":"2024-04-26T00:19:50.025527+0000","modified":"2024-04-26T00:20:02.257391+0000","last_up_change":"2024-04-26T00:19:59.222603+0000","last_in_change":"2024-04-26T00:19:52.138726+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-26T00:20:00.477727+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-04-26T00:20:01.233750+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a5d4b9d3-5499-491f-a123-85d73de452af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":3204000569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":3204000569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6811","nonce":3204000569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":3204000569}]},"public_addr":"172.21.15.37:6806/3204000569","cluster_addr":"172.21.15.37:6808/3204000569","heartbeat_back_addr":"172.21.15.37:6811/3204000569","heartbeat_front_addr":"172.21.15.37:6810/3204000569","state":["exists","up"]},{"osd":1,"uuid":"cb4583fd-a0fa-4559-9f99-5a3e02b5818c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":1084527868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6805","nonce":1084527868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6809","nonce":1084527868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6807","nonce":1084527868}]},"public_addr":"172.21.15.37:6804/1084527868","cluster_addr":"172.21.15.37:6805/1084527868","heartbeat_back_addr":"172.21.15.37:6809/1084527868","heartbeat_front_addr":"172.21.15.37:6807/1084527868","state":["exists","up"]},{"osd":2,"uuid":"78fc9fe6-2163-4cb3-9944-3773bd488df8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":603175474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6801","nonce":603175474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6803","nonce":603175474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":603175474}]},"public_addr":"172.21.15.37:6800/603175474","cluster_addr":"172.21.15.37:6801/603175474","heartbeat_back_addr":"172.21.15.37:6803/603175474","heartbeat_front_addr":"172.21.15.37:6802/603175474","state":["exists","up"]},{"osd":3,"uuid":"338bb632-eca1-422b-b196-fd6cadb8307c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":2173737584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6801","nonce":2173737584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6803","nonce":2173737584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":2173737584}]},"public_addr":"172.21.15.45:6800/2173737584","cluster_addr":"172.21.15.45:6801/2173737584","heartbeat_back_addr":"172.21.15.45:6803/2173737584","heartbeat_front_addr":"172.21.15.45:6802/2173737584","state":["exists","up"]},{"osd":4,"uuid":"ce2e0db7-d486-4cb1-9913-6a4a011e5c8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":152605921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6805","nonce":152605921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6807","nonce":152605921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":152605921}]},"public_addr":"172.21.15.45:6804/152605921","cluster_addr":"172.21.15.45:6805/152605921","heartbeat_back_addr":"172.21.15.45:6807/152605921","heartbeat_front_addr":"172.21.15.45:6806/152605921","state":["exists","up"]},{"osd":5,"uuid":"a61739cc-41d3-48b1-a471-4092b2edbf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3915746722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6809","nonce":3915746722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6811","nonce":3915746722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3915746722}]},"public_addr":"172.21.15.45:6808/3915746722","cluster_addr":"172.21.15.45:6809/3915746722","heartbeat_back_addr":"172.21.15.45:6811/3915746722","heartbeat_front_addr":"172.21.15.45:6810/3915746722","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T00:19:57.580195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T00:19:57.630479+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T00:19:57.716834+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T00:19:57.665953+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T00:19:57.633161+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T00:19:57.663690+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T00:20:02.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.987+0000 7f9293adc640 1 -- 172.21.15.37:0/886897774 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9270035e20 msgr2=0x7f92700382e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:02.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.987+0000 7f9293adc640 1 --2- 172.21.15.37:0/886897774 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9270035e20 0x7f92700382e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9288003e40 tx=0x7f9288009040 comp rx=0 tx=0).stop 2024-04-26T00:20:02.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.987+0000 7f9293adc640 10 monclient: shutdown 2024-04-26T00:20:02.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.987+0000 7f9293adc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:02.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.987+0000 7f9293adc640 1 -- 172.21.15.37:0/886897774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f928c133ce0 msgr2=0x7f928c139b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:02.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.987+0000 7f9293adc640 1 --2- 172.21.15.37:0/886897774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f928c133ce0 0x7f928c139b30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f928400efc0 tx=0x7f9284009f90 comp rx=0 tx=0).stop 2024-04-26T00:20:02.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.987+0000 7f9293adc640 1 -- 172.21.15.37:0/886897774 shutdown_connections 2024-04-26T00:20:02.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.987+0000 7f9293adc640 1 --2- 172.21.15.37:0/886897774 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9270035e20 0x7f92700382e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:02.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.987+0000 7f9293adc640 1 --2- 172.21.15.37:0/886897774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f928c133ce0 0x7f928c139b30 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:02.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.987+0000 7f9293adc640 1 --2- 172.21.15.37:0/886897774 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f928c0a1bc0 0x7f928c1395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:02.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.987+0000 7f9293adc640 1 --2- 172.21.15.37:0/886897774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f928c0725a0 0x7f928c13dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:02.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.987+0000 7f9293adc640 1 -- 172.21.15.37:0/886897774 >> 172.21.15.37:0/886897774 conn(0x7f928c0a6250 msgr2=0x7f928c071430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:02.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.987+0000 7f9293adc640 1 -- 172.21.15.37:0/886897774 shutdown_connections 2024-04-26T00:20:02.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:02.987+0000 7f9293adc640 1 -- 172.21.15.37:0/886897774 wait complete. 2024-04-26T00:20:02.989 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-26T00:20:02.989 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T00:20:03.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.127+0000 7fab7f4ef640 10 monclient: get_monmap_and_config 2024-04-26T00:20:03.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.127+0000 7fab7f4ef640 10 monclient: build_initial_monmap 2024-04-26T00:20:03.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.127+0000 7fab7f4ef640 10 monclient: monmap: 2024-04-26T00:20:03.110 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:03.110 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.110 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:03.133266+0000 2024-04-26T00:20:03.110 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:03.133266+0000 2024-04-26T00:20:03.110 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:03.110 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.110 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:03.110 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:03.110 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:03.110 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:03.111 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:03.111 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.127+0000 7fab7f4ef640 1 Processor -- start 2024-04-26T00:20:03.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.127+0000 7fab7f4ef640 1 -- start start 2024-04-26T00:20:03.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.127+0000 7fab7f4ef640 10 monclient: init 2024-04-26T00:20:03.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.127+0000 7fab7f4ef640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:03.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.127+0000 7fab7f4ef640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-04-26T00:20:03.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.127+0000 7fab7f4ef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fab780a0260 addr v1:172.21.15.45:6789/0 2024-04-26T00:20:03.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.127+0000 7fab7f4ef640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab78137e40 0x7fab7813a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.127+0000 7fab7f4ef640 10 monclient(hunting): picked mon.noname-b con 0x7fab78137e40 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:03.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.127+0000 7fab7f4ef640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fab7813a770 0x7fab7813cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.127+0000 7fab7f4ef640 10 monclient(hunting): picked mon.noname-c con 0x7fab7813a770 addr v2:172.21.15.45:3300/0 2024-04-26T00:20:03.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.127+0000 7fab7f4ef640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab780a1130 con 0x7fab780a0260 2024-04-26T00:20:03.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.127+0000 7fab7f4ef640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.127+0000 7fab7f4ef640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fab780a0d50 con 0x7fab7813a770 2024-04-26T00:20:03.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.127+0000 7fab7f4ef640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.127+0000 7fab7f4ef640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fab7809ec10 con 0x7fab78137e40 2024-04-26T00:20:03.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.127+0000 7fab7f4ef640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:03.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.127+0000 7fab7f4ef640 10 monclient(hunting): authenticate will time out at 984.439399s 2024-04-26T00:20:03.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.127+0000 7fab7ca8a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab78137e40 0x7fab7813a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.127+0000 7fab7ca8a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab78137e40 0x7fab7813a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:57726/0 (socket says 172.21.15.37:57726) 2024-04-26T00:20:03.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.127+0000 7fab7ca8a640 1 -- 172.21.15.37:0/3404457334 learned_addr learned my addr 172.21.15.37:0/3404457334 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:03.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.127+0000 7fab7ca8a640 10 monclient(hunting): get_auth_request con 0x7fab78137e40 auth_method 0 2024-04-26T00:20:03.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.127+0000 7fab7ca8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.127+0000 7fab7ca8a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.127+0000 7fab7ca8a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.127+0000 7fab7da8c640 1 --2- 172.21.15.37:0/3404457334 >> v2:172.21.15.45:3300/0 conn(0x7fab7813a770 0x7fab7813cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.127+0000 7fab7da8c640 10 monclient(hunting): get_auth_request con 0x7fab7813a770 auth_method 0 2024-04-26T00:20:03.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.127+0000 7fab7da8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.127+0000 7fab7da8c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.127+0000 7fab7da8c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.127+0000 7fab7ca8a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.127+0000 7fab7ca8a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.127+0000 7fab7ca8a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.127+0000 7fab7da8c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.127+0000 7fab7da8c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.127+0000 7fab7da8c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.131+0000 7fab7ca8a640 10 monclient(hunting): handle_auth_done global_id 4199 payload 274 2024-04-26T00:20:03.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.131+0000 7fab7ca8a640 1 -- 172.21.15.37:0/3404457334 >> v2:172.21.15.45:3300/0 conn(0x7fab7813a770 msgr2=0x7fab7813cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:03.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.131+0000 7fab7ca8a640 1 --2- 172.21.15.37:0/3404457334 >> v2:172.21.15.45:3300/0 conn(0x7fab7813a770 0x7fab7813cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.131+0000 7fab7ca8a640 1 -- 172.21.15.37:0/3404457334 >> v1:172.21.15.45:6789/0 conn(0x7fab780a0260 legacy=0x7fab780a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:03.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.131+0000 7fab7ca8a640 10 monclient: _finish_hunting 0 2024-04-26T00:20:03.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.131+0000 7fab7ca8a640 1 monclient: found mon.noname-b 2024-04-26T00:20:03.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.131+0000 7fab7ca8a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:03.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.131+0000 7fab7ca8a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:20:03.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.131+0000 7fab7ca8a640 1 -- 172.21.15.37:0/3404457334 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab7806cff0 con 0x7fab78137e40 2024-04-26T00:20:03.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.131+0000 7fab7da8c640 1 --2- 172.21.15.37:0/3404457334 >> v2:172.21.15.45:3300/0 conn(0x7fab7813a770 0x7fab7813cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:03.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7ca8a640 1 -- 172.21.15.37:0/3404457334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab78137e40 msgr2=0x7fab7813a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:03.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7ca8a640 1 --2- 172.21.15.37:0/3404457334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab78137e40 0x7fab7813a230 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fab6c009e50 tx=0x7fab6c01c6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d073fdb59c79071 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab77fff640 1 -- 172.21.15.37:0/3404457334 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab6c0a9460 con 0x7fab78137e40 2024-04-26T00:20:03.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab77fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab77fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:20:03.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab77fff640 10 monclient: dump: 2024-04-26T00:20:03.121 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.121 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.121 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.121 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.121 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.121 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.121 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.122 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.122 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.122 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab77fff640 10 monclient: _finish_auth 0 2024-04-26T00:20:03.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab77fff640 10 monclient: _check_auth_tickets 2024-04-26T00:20:03.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab77fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:03.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7f4ef640 5 monclient: authenticate success, global_id 4199 2024-04-26T00:20:03.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7f4ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:20:03.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab77fff640 1 -- 172.21.15.37:0/3404457334 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab6c0ae070 con 0x7fab78137e40 2024-04-26T00:20:03.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab77fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:03.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab77fff640 1 -- 172.21.15.37:0/3404457334 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab6c0a9820 con 0x7fab78137e40 2024-04-26T00:20:03.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7f4ef640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:03.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7f4ef640 4 set_mon_vals no callback set 2024-04-26T00:20:03.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab77fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab77fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:03.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab77fff640 10 monclient: dump: 2024-04-26T00:20:03.123 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.123 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.123 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.123 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.123 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.123 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.123 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.123 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.123 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.123 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7f4ef640 10 monclient: shutdown 2024-04-26T00:20:03.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7f4ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:03.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7f4ef640 1 -- 172.21.15.37:0/3404457334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab78137e40 msgr2=0x7fab7813a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7f4ef640 1 --2- 172.21.15.37:0/3404457334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab78137e40 0x7fab7813a230 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fab6c009e50 tx=0x7fab6c01c6d0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7f4ef640 1 -- 172.21.15.37:0/3404457334 shutdown_connections 2024-04-26T00:20:03.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7f4ef640 1 --2- 172.21.15.37:0/3404457334 >> v2:172.21.15.45:3300/0 conn(0x7fab7813a770 0x7fab7813cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7f4ef640 1 --2- 172.21.15.37:0/3404457334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab78137e40 0x7fab7813a230 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7f4ef640 1 -- 172.21.15.37:0/3404457334 >> 172.21.15.37:0/3404457334 conn(0x7fab780a1ad0 msgr2=0x7fab780a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:03.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7f4ef640 1 -- 172.21.15.37:0/3404457334 shutdown_connections 2024-04-26T00:20:03.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7f4ef640 1 -- 172.21.15.37:0/3404457334 wait complete. 2024-04-26T00:20:03.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7f4ef640 10 monclient: build_initial_monmap 2024-04-26T00:20:03.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7f4ef640 10 monclient: monmap: 2024-04-26T00:20:03.124 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:03.124 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:03.124 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:03.124 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:03.124 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:03.125 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:03.125 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:03.125 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:03.125 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:03.125 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7f4ef640 1 Processor -- start 2024-04-26T00:20:03.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7f4ef640 1 -- start start 2024-04-26T00:20:03.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7f4ef640 10 monclient: init 2024-04-26T00:20:03.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7f4ef640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:03.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7f4ef640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:20:03.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7f4ef640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab780a0260 0x7fab781ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7f4ef640 10 monclient(hunting): picked mon.noname-b con 0x7fab780a0260 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:03.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7f4ef640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab78137e40 0x7fab78134e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7f4ef640 10 monclient(hunting): picked mon.noname-a con 0x7fab78137e40 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:03.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7f4ef640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab7813a770 0x7fab78135360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7f4ef640 10 monclient(hunting): picked mon.noname-c con 0x7fab7813a770 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:03.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7f4ef640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7f4ef640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fab7809f0f0 con 0x7fab78137e40 2024-04-26T00:20:03.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7f4ef640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7f4ef640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fab780a00e0 con 0x7fab780a0260 2024-04-26T00:20:03.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7f4ef640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7f4ef640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fab7809f4c0 con 0x7fab7813a770 2024-04-26T00:20:03.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7f4ef640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:03.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7f4ef640 10 monclient(hunting): authenticate will time out at 984.447082s 2024-04-26T00:20:03.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7da8c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab7813a770 0x7fab78135360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7ca8a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab78137e40 0x7fab78134e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7ca8a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab78137e40 0x7fab78134e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44486/0 (socket says 172.21.15.37:44486) 2024-04-26T00:20:03.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7da8c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab7813a770 0x7fab78135360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:57730/0 (socket says 172.21.15.37:57730) 2024-04-26T00:20:03.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7ca8a640 1 -- 172.21.15.37:0/3054224302 learned_addr learned my addr 172.21.15.37:0/3054224302 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:03.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7ca8a640 10 monclient(hunting): get_auth_request con 0x7fab78137e40 auth_method 0 2024-04-26T00:20:03.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7ca8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7ca8a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7ca8a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7da8c640 10 monclient(hunting): get_auth_request con 0x7fab7813a770 auth_method 0 2024-04-26T00:20:03.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7da8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7da8c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7da8c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7d28b640 1 --2- 172.21.15.37:0/3054224302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab780a0260 0x7fab781ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7da8c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7da8c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7da8c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7d28b640 10 monclient(hunting): get_auth_request con 0x7fab780a0260 auth_method 0 2024-04-26T00:20:03.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7d28b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7d28b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7d28b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7ca8a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7ca8a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7ca8a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7d28b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7d28b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7d28b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7da8c640 10 monclient(hunting): handle_auth_done global_id 4202 payload 824 2024-04-26T00:20:03.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7da8c640 1 -- 172.21.15.37:0/3054224302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab780a0260 msgr2=0x7fab781ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7da8c640 1 --2- 172.21.15.37:0/3054224302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab780a0260 0x7fab781ceb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7da8c640 1 -- 172.21.15.37:0/3054224302 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab78137e40 msgr2=0x7fab78134e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7da8c640 1 --2- 172.21.15.37:0/3054224302 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab78137e40 0x7fab78134e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7da8c640 10 monclient: _finish_hunting 0 2024-04-26T00:20:03.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7da8c640 1 monclient: found mon.noname-c 2024-04-26T00:20:03.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7da8c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:03.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7da8c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:20:03.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.135+0000 7fab7da8c640 1 -- 172.21.15.37:0/3054224302 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab7809e140 con 0x7fab7813a770 2024-04-26T00:20:03.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7da8c640 1 -- 172.21.15.37:0/3054224302 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab7813a770 msgr2=0x7fab78135360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:03.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7da8c640 1 --2- 172.21.15.37:0/3054224302 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab7813a770 0x7fab78135360 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fab7000e900 tx=0x7fab7000edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab75ffb640 1 -- 172.21.15.37:0/3054224302 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab70017070 con 0x7fab7813a770 2024-04-26T00:20:03.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab75ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab75ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:20:03.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab75ffb640 10 monclient: dump: 2024-04-26T00:20:03.133 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.133 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.134 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.134 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.134 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.134 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.134 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.134 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.134 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.134 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab75ffb640 10 monclient: _finish_auth 0 2024-04-26T00:20:03.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab75ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:20:03.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab75ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:03.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7f4ef640 5 monclient: authenticate success, global_id 4202 2024-04-26T00:20:03.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab75ffb640 1 -- 172.21.15.37:0/3054224302 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab7000cb50 con 0x7fab7813a770 2024-04-26T00:20:03.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab75ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:03.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7f4ef640 10 monclient: _renew_subs 2024-04-26T00:20:03.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab75ffb640 1 -- 172.21.15.37:0/3054224302 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab70004020 con 0x7fab7813a770 2024-04-26T00:20:03.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7f4ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:03.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7f4ef640 1 -- 172.21.15.37:0/3054224302 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab78120be0 con 0x7fab7813a770 2024-04-26T00:20:03.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab75ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab75ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:03.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab75ffb640 10 monclient: dump: 2024-04-26T00:20:03.136 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.136 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.136 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.136 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.136 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.137 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.137 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.137 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.137 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.137 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab777fe640 4 set_mon_vals no callback set 2024-04-26T00:20:03.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7f4ef640 10 monclient: _renew_subs 2024-04-26T00:20:03.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7f4ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:03.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7f4ef640 1 -- 172.21.15.37:0/3054224302 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fab78120040 con 0x7fab7813a770 2024-04-26T00:20:03.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7da8c640 1 -- 172.21.15.37:0/3054224302 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab7813a770 msgr2=0x7fab78135360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:20:03.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7da8c640 1 -- 172.21.15.37:0/3054224302 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab7813a770 msgr2=0x7fab78135360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:20:03.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7da8c640 1 --2- 172.21.15.37:0/3054224302 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab7813a770 0x7fab78135360 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fab7000e900 tx=0x7fab7000edc0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:20:03.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7da8c640 1 --2- 172.21.15.37:0/3054224302 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab7813a770 0x7fab78135360 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fab7000e900 tx=0x7fab7000edc0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab75ffb640 10 monclient: ms_handle_reset current mon [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:03.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab75ffb640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:03.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab75ffb640 1 -- 172.21.15.37:0/3054224302 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab7813a770 msgr2=0x7fab78135360 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:20:03.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab75ffb640 1 --2- 172.21.15.37:0/3054224302 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab7813a770 0x7fab78135360 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab75ffb640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:20:03.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab75ffb640 1 --2- 172.21.15.37:0/3054224302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab780a0260 0x7fab640058d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab75ffb640 10 monclient(hunting): picked mon.b con 0x7fab780a0260 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:03.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab75ffb640 1 --2- 172.21.15.37:0/3054224302 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab640062a0 0x7fab64008730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab75ffb640 10 monclient(hunting): picked mon.c con 0x7fab640062a0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:03.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab75ffb640 1 --2- 172.21.15.37:0/3054224302 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab78137e40 0x7fab6400b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab75ffb640 10 monclient(hunting): picked mon.a con 0x7fab78137e40 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:03.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab75ffb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab75ffb640 1 -- 172.21.15.37:0/3054224302 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fab6400bc60 con 0x7fab78137e40 2024-04-26T00:20:03.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab75ffb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab75ffb640 1 -- 172.21.15.37:0/3054224302 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fab6400bde0 con 0x7fab780a0260 2024-04-26T00:20:03.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab75ffb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab75ffb640 1 -- 172.21.15.37:0/3054224302 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fab6400bf60 con 0x7fab640062a0 2024-04-26T00:20:03.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab75ffb640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:03.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7ca8a640 1 --2- 172.21.15.37:0/3054224302 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab640062a0 0x7fab64008730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7ca8a640 10 monclient(hunting): get_auth_request con 0x7fab640062a0 auth_method 0 2024-04-26T00:20:03.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7ca8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7ca8a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7ca8a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:20:03.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7da8c640 1 --2- 172.21.15.37:0/3054224302 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab78137e40 0x7fab6400b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7f4ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:03.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7d28b640 1 --2- 172.21.15.37:0/3054224302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab780a0260 0x7fab640058d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7da8c640 10 monclient(hunting): get_auth_request con 0x7fab78137e40 auth_method 0 2024-04-26T00:20:03.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7da8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7da8c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7da8c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:20:03.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7d28b640 10 monclient(hunting): get_auth_request con 0x7fab780a0260 auth_method 0 2024-04-26T00:20:03.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7d28b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7d28b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7d28b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:20:03.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7f4ef640 10 monclient(hunting): _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:03.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7ca8a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7ca8a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7ca8a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:20:03.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7d28b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7d28b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7d28b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:20:03.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7da8c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7da8c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7da8c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:20:03.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7ca8a640 10 monclient(hunting): handle_auth_done global_id 4202 payload 293 2024-04-26T00:20:03.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7ca8a640 1 -- 172.21.15.37:0/3054224302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab780a0260 msgr2=0x7fab640058d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7ca8a640 1 --2- 172.21.15.37:0/3054224302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab780a0260 0x7fab640058d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7ca8a640 1 -- 172.21.15.37:0/3054224302 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab78137e40 msgr2=0x7fab6400b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7ca8a640 1 --2- 172.21.15.37:0/3054224302 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab78137e40 0x7fab6400b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7ca8a640 10 monclient: _finish_hunting 0 2024-04-26T00:20:03.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7ca8a640 1 monclient: found mon.c 2024-04-26T00:20:03.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7ca8a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:03.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7ca8a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:03.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7ca8a640 1 -- 172.21.15.37:0/3054224302 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7fab6400c290 con 0x7fab640062a0 2024-04-26T00:20:03.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7ca8a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:03.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7ca8a640 1 -- 172.21.15.37:0/3054224302 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab78066f40 con 0x7fab640062a0 2024-04-26T00:20:03.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7ca8a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:03.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7ca8a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:03.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7ca8a640 1 -- 172.21.15.37:0/3054224302 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab7806cff0 con 0x7fab640062a0 2024-04-26T00:20:03.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7ca8a640 10 monclient: _finish_auth 0 2024-04-26T00:20:03.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7ca8a640 10 monclient: _check_auth_tickets 2024-04-26T00:20:03.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.139+0000 7fab7ca8a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:03.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.143+0000 7fab7ca8a640 1 -- 172.21.15.37:0/3054224302 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab640062a0 msgr2=0x7fab64008730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:03.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.143+0000 7fab7ca8a640 1 --2- 172.21.15.37:0/3054224302 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab640062a0 0x7fab64008730 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fab6c0a9490 tx=0x7fab70009190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.143+0000 7fab75ffb640 1 -- 172.21.15.37:0/3054224302 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab6c009510 con 0x7fab640062a0 2024-04-26T00:20:03.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.143+0000 7fab75ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.143+0000 7fab75ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:03.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.143+0000 7fab75ffb640 10 monclient: dump: 2024-04-26T00:20:03.142 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.142 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.142 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.142 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.143 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.143 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.143 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.143 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.143 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.143 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.143+0000 7fab75ffb640 1 -- 172.21.15.37:0/3054224302 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab6c0c2020 con 0x7fab640062a0 2024-04-26T00:20:03.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.143+0000 7fab75ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:03.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.143+0000 7fab76ffd640 4 set_mon_vals no callback set 2024-04-26T00:20:03.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.143+0000 7fab75ffb640 1 -- 172.21.15.37:0/3054224302 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mgrmap(e 4) ==== 43962+0+0 (secure 0 0 0) 0x7fab6c0a94c0 con 0x7fab640062a0 2024-04-26T00:20:03.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.143+0000 7fab75ffb640 1 --2- 172.21.15.37:0/3054224302 >> v2:172.21.15.45:6812/2379918623 conn(0x7fab7813a770 0x7fab64040710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.143+0000 7fab75ffb640 1 -- 172.21.15.37:0/3054224302 <== mon.2 v2:172.21.15.37:3301/0 4 ==== osd_map(14..14 src has 1..14) ==== 3506+0+0 (secure 0 0 0) 0x7fab6c0b9070 con 0x7fab640062a0 2024-04-26T00:20:03.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.143+0000 7fab7d28b640 1 --2- 172.21.15.37:0/3054224302 >> v2:172.21.15.45:6812/2379918623 conn(0x7fab7813a770 0x7fab64040710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.143+0000 7fab7d28b640 10 monclient: get_auth_request con 0x7fab7813a770 auth_method 0 2024-04-26T00:20:03.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.147+0000 7fab7d28b640 1 --2- 172.21.15.37:0/3054224302 >> v2:172.21.15.45:6812/2379918623 conn(0x7fab7813a770 0x7fab64040710 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fab68009970 tx=0x7fab68009060 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.147+0000 7fab75ffb640 1 -- 172.21.15.37:0/3054224302 <== mon.2 v2:172.21.15.37:3301/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fab780a1130 con 0x7fab640062a0 2024-04-26T00:20:03.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.147+0000 7fab75ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:03.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.147+0000 7fab75ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:03.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.151+0000 7fab75ffb640 1 -- 172.21.15.37:0/3054224302 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fab6c0a6520 con 0x7fab640062a0 2024-04-26T00:20:03.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.151+0000 7fab75ffb640 10 monclient: handle_mon_command_ack 1 not found 2024-04-26T00:20:03.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.311+0000 7fab7f4ef640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-26T00:20:03.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.311+0000 7fab7f4ef640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-26T00:20:03.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.311+0000 7fab7f4ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:03.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.311+0000 7fab7f4ef640 1 -- 172.21.15.37:0/3054224302 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fab7806b9a0 con 0x7fab640062a0 2024-04-26T00:20:03.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.315+0000 7fab75ffb640 1 -- 172.21.15.37:0/3054224302 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) ==== 74+0+10761 (secure 0 0 0) 0x7fab6c0dfb70 con 0x7fab640062a0 2024-04-26T00:20:03.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.315+0000 7fab75ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-26T00:20:03.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.315+0000 7fab75ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:03.297 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:20:03.298 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":15,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","created":"2024-04-26T00:19:50.025527+0000","modified":"2024-04-26T00:20:03.259969+0000","last_up_change":"2024-04-26T00:19:59.222603+0000","last_in_change":"2024-04-26T00:19:52.138726+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-26T00:20:00.477727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-04-26T00:20:01.233750+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a5d4b9d3-5499-491f-a123-85d73de452af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":3204000569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":3204000569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6811","nonce":3204000569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":3204000569}]},"public_addr":"172.21.15.37:6806/3204000569","cluster_addr":"172.21.15.37:6808/3204000569","heartbeat_back_addr":"172.21.15.37:6811/3204000569","heartbeat_front_addr":"172.21.15.37:6810/3204000569","state":["exists","up"]},{"osd":1,"uuid":"cb4583fd-a0fa-4559-9f99-5a3e02b5818c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":1084527868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6805","nonce":1084527868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6809","nonce":1084527868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6807","nonce":1084527868}]},"public_addr":"172.21.15.37:6804/1084527868","cluster_addr":"172.21.15.37:6805/1084527868","heartbeat_back_addr":"172.21.15.37:6809/1084527868","heartbeat_front_addr":"172.21.15.37:6807/1084527868","state":["exists","up"]},{"osd":2,"uuid":"78fc9fe6-2163-4cb3-9944-3773bd488df8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":603175474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6801","nonce":603175474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6803","nonce":603175474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":603175474}]},"public_addr":"172.21.15.37:6800/603175474","cluster_addr":"172.21.15.37:6801/603175474","heartbeat_back_addr":"172.21.15.37:6803/603175474","heartbeat_front_addr":"172.21.15.37:6802/603175474","state":["exists","up"]},{"osd":3,"uuid":"338bb632-eca1-422b-b196-fd6cadb8307c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":2173737584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6801","nonce":2173737584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6803","nonce":2173737584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":2173737584}]},"public_addr":"172.21.15.45:6800/2173737584","cluster_addr":"172.21.15.45:6801/2173737584","heartbeat_back_addr":"172.21.15.45:6803/2173737584","heartbeat_front_addr":"172.21.15.45:6802/2173737584","state":["exists","up"]},{"osd":4,"uuid":"ce2e0db7-d486-4cb1-9913-6a4a011e5c8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":152605921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6805","nonce":152605921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6807","nonce":152605921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":152605921}]},"public_addr":"172.21.15.45:6804/152605921","cluster_addr":"172.21.15.45:6805/152605921","heartbeat_back_addr":"172.21.15.45:6807/152605921","heartbeat_front_addr":"172.21.15.45:6806/152605921","state":["exists","up"]},{"osd":5,"uuid":"a61739cc-41d3-48b1-a471-4092b2edbf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3915746722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6809","nonce":3915746722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6811","nonce":3915746722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3915746722}]},"public_addr":"172.21.15.45:6808/3915746722","cluster_addr":"172.21.15.45:6809/3915746722","heartbeat_back_addr":"172.21.15.45:6811/3915746722","heartbeat_front_addr":"172.21.15.45:6810/3915746722","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T00:19:57.580195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T00:19:57.630479+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T00:19:57.716834+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T00:19:57.665953+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T00:19:57.633161+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T00:19:57.663690+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T00:20:03.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.315+0000 7fab7f4ef640 1 -- 172.21.15.37:0/3054224302 >> v2:172.21.15.45:6812/2379918623 conn(0x7fab7813a770 msgr2=0x7fab64040710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.315+0000 7fab7f4ef640 1 --2- 172.21.15.37:0/3054224302 >> v2:172.21.15.45:6812/2379918623 conn(0x7fab7813a770 0x7fab64040710 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fab68009970 tx=0x7fab68009060 comp rx=0 tx=0).stop 2024-04-26T00:20:03.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.315+0000 7fab7f4ef640 10 monclient: shutdown 2024-04-26T00:20:03.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.315+0000 7fab7f4ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:03.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.315+0000 7fab7f4ef640 1 -- 172.21.15.37:0/3054224302 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab640062a0 msgr2=0x7fab64008730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.315+0000 7fab7f4ef640 1 --2- 172.21.15.37:0/3054224302 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab640062a0 0x7fab64008730 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fab6c0a9490 tx=0x7fab70009190 comp rx=0 tx=0).stop 2024-04-26T00:20:03.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.315+0000 7fab7f4ef640 1 -- 172.21.15.37:0/3054224302 shutdown_connections 2024-04-26T00:20:03.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.315+0000 7fab7f4ef640 1 --2- 172.21.15.37:0/3054224302 >> v2:172.21.15.45:6812/2379918623 conn(0x7fab7813a770 0x7fab64040710 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.315+0000 7fab7f4ef640 1 --2- 172.21.15.37:0/3054224302 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab78137e40 0x7fab6400b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.315+0000 7fab7f4ef640 1 --2- 172.21.15.37:0/3054224302 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab640062a0 0x7fab64008730 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.315+0000 7fab7f4ef640 1 --2- 172.21.15.37:0/3054224302 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab780a0260 0x7fab640058d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.315+0000 7fab7f4ef640 1 -- 172.21.15.37:0/3054224302 >> 172.21.15.37:0/3054224302 conn(0x7fab780a1ad0 msgr2=0x7fab781364a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:03.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.315+0000 7fab7f4ef640 1 -- 172.21.15.37:0/3054224302 shutdown_connections 2024-04-26T00:20:03.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.315+0000 7fab7f4ef640 1 -- 172.21.15.37:0/3054224302 wait complete. 2024-04-26T00:20:03.316 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-26T00:20:03.316 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-26T00:20:03.317 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-26T00:20:03.317 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-26T00:20:03.318 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-26T00:20:03.318 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-26T00:20:03.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.455+0000 7f4eddd7a640 10 monclient: get_monmap_and_config 2024-04-26T00:20:03.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.455+0000 7f4eddd7a640 10 monclient: build_initial_monmap 2024-04-26T00:20:03.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.455+0000 7f4eddd7a640 10 monclient: monmap: 2024-04-26T00:20:03.439 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:03.439 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.439 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:03.462404+0000 2024-04-26T00:20:03.439 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:03.462404+0000 2024-04-26T00:20:03.439 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:03.439 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.439 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:03.439 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:03.439 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:03.439 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:03.439 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:03.439 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.455+0000 7f4eddd7a640 1 Processor -- start 2024-04-26T00:20:03.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.455+0000 7f4eddd7a640 1 -- start start 2024-04-26T00:20:03.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.455+0000 7f4eddd7a640 10 monclient: init 2024-04-26T00:20:03.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.455+0000 7f4eddd7a640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:03.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.455+0000 7f4eddd7a640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-04-26T00:20:03.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.455+0000 7f4eddd7a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4ed80a3d90 0x7f4ed809bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.455+0000 7f4eddd7a640 10 monclient(hunting): picked mon.noname-b con 0x7f4ed80a3d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:03.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.455+0000 7f4eddd7a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4ed80a4170 addr v1:172.21.15.37:6789/0 2024-04-26T00:20:03.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.455+0000 7f4eddd7a640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f4ed813c3e0 0x7f4ed813e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.455+0000 7f4eddd7a640 10 monclient(hunting): picked mon.noname-a con 0x7f4ed813c3e0 addr v2:172.21.15.37:3300/0 2024-04-26T00:20:03.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.455+0000 7f4eddd7a640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ed80a5150 con 0x7f4ed80a4170 2024-04-26T00:20:03.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.455+0000 7f4eddd7a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.455+0000 7f4eddd7a640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f4ed809b280 con 0x7f4ed813c3e0 2024-04-26T00:20:03.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.455+0000 7f4eddd7a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.455+0000 7f4eddd7a640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4ed809b560 con 0x7f4ed80a3d90 2024-04-26T00:20:03.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.455+0000 7f4eddd7a640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:03.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.455+0000 7f4eddd7a640 10 monclient(hunting): authenticate will time out at 984.767995s 2024-04-26T00:20:03.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.459+0000 7f4ed6ffd640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f4ed80a4170 0x7f4ed80a45b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:55768/0 (socket says 172.21.15.37:55768) 2024-04-26T00:20:03.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.459+0000 7f4ed6ffd640 1 -- 172.21.15.37:0/576723245 learned_addr learned my addr 172.21.15.37:0/576723245 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:20:03.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.459+0000 7f4ed7fff640 1 --2- 172.21.15.37:0/576723245 >> v2:172.21.15.37:3300/0 conn(0x7f4ed813c3e0 0x7f4ed813e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.459+0000 7f4ed7fff640 10 monclient(hunting): get_auth_request con 0x7f4ed813c3e0 auth_method 0 2024-04-26T00:20:03.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.459+0000 7f4ed7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.459+0000 7f4ed7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.459+0000 7f4ed7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.459+0000 7f4ed77fe640 1 --2- 172.21.15.37:0/576723245 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4ed80a3d90 0x7f4ed809bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.459+0000 7f4ed7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.459+0000 7f4ed7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.459+0000 7f4ed7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.459+0000 7f4ed77fe640 10 monclient(hunting): get_auth_request con 0x7f4ed80a3d90 auth_method 0 2024-04-26T00:20:03.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.459+0000 7f4ed77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.459+0000 7f4ed77fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.459+0000 7f4ed77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.459+0000 7f4ed7fff640 10 monclient(hunting): handle_auth_done global_id 4218 payload 274 2024-04-26T00:20:03.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.459+0000 7f4ed7fff640 1 -- 172.21.15.37:0/576723245 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4ed80a3d90 msgr2=0x7f4ed809bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:03.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.459+0000 7f4ed7fff640 1 --2- 172.21.15.37:0/576723245 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4ed80a3d90 0x7f4ed809bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.459+0000 7f4ed7fff640 1 -- 172.21.15.37:0/576723245 >> v1:172.21.15.37:6789/0 conn(0x7f4ed80a4170 legacy=0x7f4ed80a45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:03.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.459+0000 7f4ed7fff640 10 monclient: _finish_hunting 0 2024-04-26T00:20:03.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.459+0000 7f4ed7fff640 1 monclient: found mon.noname-a 2024-04-26T00:20:03.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.459+0000 7f4ed7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:03.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.459+0000 7f4ed7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:03.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.459+0000 7f4ed7fff640 1 -- 172.21.15.37:0/576723245 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ed806dc60 con 0x7f4ed813c3e0 2024-04-26T00:20:03.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.459+0000 7f4ed77fe640 1 --2- 172.21.15.37:0/576723245 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4ed80a3d90 0x7f4ed809bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:20:03.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.463+0000 7f4ed7fff640 1 -- 172.21.15.37:0/576723245 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4ed813c3e0 msgr2=0x7f4ed813e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:03.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.463+0000 7f4ed7fff640 1 --2- 172.21.15.37:0/576723245 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4ed813c3e0 0x7f4ed813e7d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4ed0009d90 tx=0x7f4ed001c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=234b8b385c148db0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.463+0000 7f4ed67fc640 1 -- 172.21.15.37:0/576723245 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ed00b2070 con 0x7f4ed813c3e0 2024-04-26T00:20:03.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.463+0000 7f4ed67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.463+0000 7f4ed67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:03.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.463+0000 7f4ed67fc640 10 monclient: dump: 2024-04-26T00:20:03.446 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.446 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.446 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.446 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.446 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.446 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.447 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.447 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.447 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.447 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.463+0000 7f4ed67fc640 10 monclient: _finish_auth 0 2024-04-26T00:20:03.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.463+0000 7f4ed67fc640 10 monclient: _check_auth_tickets 2024-04-26T00:20:03.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.463+0000 7f4ed67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:03.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.463+0000 7f4ed67fc640 1 -- 172.21.15.37:0/576723245 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ed00a5780 con 0x7f4ed813c3e0 2024-04-26T00:20:03.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.463+0000 7f4ed67fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:03.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.463+0000 7f4ed67fc640 1 -- 172.21.15.37:0/576723245 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ed00a5b50 con 0x7f4ed813c3e0 2024-04-26T00:20:03.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.463+0000 7f4ed67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.463+0000 7f4ed67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:03.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.463+0000 7f4ed67fc640 10 monclient: dump: 2024-04-26T00:20:03.447 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.447 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.448 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.448 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.448 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.448 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.448 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.448 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.448 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.448 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.463+0000 7f4eddd7a640 5 monclient: authenticate success, global_id 4218 2024-04-26T00:20:03.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.463+0000 7f4eddd7a640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:03.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.463+0000 7f4eddd7a640 4 set_mon_vals no callback set 2024-04-26T00:20:03.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.463+0000 7f4eddd7a640 10 monclient: shutdown 2024-04-26T00:20:03.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.463+0000 7f4eddd7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:03.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.463+0000 7f4eddd7a640 1 -- 172.21.15.37:0/576723245 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4ed813c3e0 msgr2=0x7f4ed813e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.463+0000 7f4eddd7a640 1 --2- 172.21.15.37:0/576723245 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4ed813c3e0 0x7f4ed813e7d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4ed0009d90 tx=0x7f4ed001c810 comp rx=0 tx=0).stop 2024-04-26T00:20:03.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.463+0000 7f4eddd7a640 1 -- 172.21.15.37:0/576723245 shutdown_connections 2024-04-26T00:20:03.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.463+0000 7f4eddd7a640 1 --2- 172.21.15.37:0/576723245 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4ed813c3e0 0x7f4ed813e7d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.463+0000 7f4eddd7a640 1 --2- 172.21.15.37:0/576723245 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4ed80a3d90 0x7f4ed809bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.463+0000 7f4eddd7a640 1 -- 172.21.15.37:0/576723245 >> 172.21.15.37:0/576723245 conn(0x7f4ed80aa0a0 msgr2=0x7f4ed80a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:03.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.463+0000 7f4eddd7a640 1 -- 172.21.15.37:0/576723245 shutdown_connections 2024-04-26T00:20:03.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.463+0000 7f4eddd7a640 1 -- 172.21.15.37:0/576723245 wait complete. 2024-04-26T00:20:03.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4eddd7a640 10 monclient: build_initial_monmap 2024-04-26T00:20:03.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4eddd7a640 10 monclient: monmap: 2024-04-26T00:20:03.453 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:03.453 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:03.453 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:03.453 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:03.453 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:03.453 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:03.453 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:03.453 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:03.453 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:03.453 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4eddd7a640 1 Processor -- start 2024-04-26T00:20:03.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4eddd7a640 1 -- start start 2024-04-26T00:20:03.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4eddd7a640 10 monclient: init 2024-04-26T00:20:03.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4eddd7a640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:03.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4eddd7a640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:20:03.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4eddd7a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4ed80a3d90 0x7f4ed81cf000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4eddd7a640 10 monclient(hunting): picked mon.noname-a con 0x7f4ed80a3d90 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:03.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4eddd7a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ed80a4170 0x7f4ed813b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4eddd7a640 10 monclient(hunting): picked mon.noname-b con 0x7f4ed80a4170 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:03.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4eddd7a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4ed813c3e0 0x7f4ed813b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4eddd7a640 10 monclient(hunting): picked mon.noname-c con 0x7f4ed813c3e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:03.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4eddd7a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4eddd7a640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed8141740 con 0x7f4ed80a3d90 2024-04-26T00:20:03.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4eddd7a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4eddd7a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed809c690 con 0x7f4ed80a4170 2024-04-26T00:20:03.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4eddd7a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4eddd7a640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4ed81418c0 con 0x7f4ed813c3e0 2024-04-26T00:20:03.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4eddd7a640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:03.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4eddd7a640 10 monclient(hunting): authenticate will time out at 984.777810s 2024-04-26T00:20:03.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4ed7fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4ed813c3e0 0x7f4ed813b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4ed7fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4ed813c3e0 0x7f4ed813b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:57760/0 (socket says 172.21.15.37:57760) 2024-04-26T00:20:03.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4ed7fff640 1 -- 172.21.15.37:0/1813291053 learned_addr learned my addr 172.21.15.37:0/1813291053 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:03.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4ed7fff640 10 monclient(hunting): get_auth_request con 0x7f4ed813c3e0 auth_method 0 2024-04-26T00:20:03.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4ed7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4ed7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4ed7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4ed77fe640 1 --2- 172.21.15.37:0/1813291053 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4ed80a3d90 0x7f4ed81cf000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4ed77fe640 10 monclient(hunting): get_auth_request con 0x7f4ed80a3d90 auth_method 0 2024-04-26T00:20:03.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4ed77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4ed77fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4ed77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4ed7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4ed7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4ed6ffd640 1 --2- 172.21.15.37:0/1813291053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ed80a4170 0x7f4ed813b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4ed7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4ed6ffd640 10 monclient(hunting): get_auth_request con 0x7f4ed80a4170 auth_method 0 2024-04-26T00:20:03.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4ed6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4ed6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4ed6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4ed77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4ed77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4ed77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4ed6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4ed6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4ed6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4ed7fff640 10 monclient(hunting): handle_auth_done global_id 4208 payload 824 2024-04-26T00:20:03.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4ed7fff640 1 -- 172.21.15.37:0/1813291053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ed80a4170 msgr2=0x7f4ed813b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4ed7fff640 1 --2- 172.21.15.37:0/1813291053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ed80a4170 0x7f4ed813b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4ed7fff640 1 -- 172.21.15.37:0/1813291053 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4ed80a3d90 msgr2=0x7f4ed81cf000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4ed7fff640 1 --2- 172.21.15.37:0/1813291053 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4ed80a3d90 0x7f4ed81cf000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4ed7fff640 10 monclient: _finish_hunting 0 2024-04-26T00:20:03.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4ed7fff640 1 monclient: found mon.noname-c 2024-04-26T00:20:03.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4ed7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:03.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4ed7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:20:03.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4ed7fff640 1 -- 172.21.15.37:0/1813291053 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ed81378c0 con 0x7f4ed813c3e0 2024-04-26T00:20:03.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.467+0000 7f4ed77fe640 1 --2- 172.21.15.37:0/1813291053 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4ed80a3d90 0x7f4ed81cf000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:03.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.471+0000 7f4ed7fff640 1 -- 172.21.15.37:0/1813291053 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4ed813c3e0 msgr2=0x7f4ed813b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:03.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.471+0000 7f4ed7fff640 1 --2- 172.21.15.37:0/1813291053 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4ed813c3e0 0x7f4ed813b630 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4ed001d0b0 tx=0x7f4ed00195b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.471+0000 7f4ebffff640 1 -- 172.21.15.37:0/1813291053 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ed00b2050 con 0x7f4ed813c3e0 2024-04-26T00:20:03.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.471+0000 7f4ebffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.471+0000 7f4ebffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:20:03.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.471+0000 7f4ebffff640 10 monclient: dump: 2024-04-26T00:20:03.459 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.459 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.459 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.459 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.459 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.459 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.459 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.459 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.459 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.460 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.471+0000 7f4ebffff640 10 monclient: _finish_auth 0 2024-04-26T00:20:03.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.471+0000 7f4ebffff640 10 monclient: _check_auth_tickets 2024-04-26T00:20:03.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.471+0000 7f4ebffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:03.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.471+0000 7f4ebffff640 1 -- 172.21.15.37:0/1813291053 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ed00be070 con 0x7f4ed813c3e0 2024-04-26T00:20:03.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.471+0000 7f4ebffff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:03.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.471+0000 7f4ed5ffb640 4 set_mon_vals no callback set 2024-04-26T00:20:03.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.471+0000 7f5dd8e9b640 10 monclient: get_monmap_and_config 2024-04-26T00:20:03.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.471+0000 7f5dd8e9b640 10 monclient: build_initial_monmap 2024-04-26T00:20:03.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.471+0000 7f5dd8e9b640 10 monclient: monmap: 2024-04-26T00:20:03.461 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:03.461 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.461 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:03.477729+0000 2024-04-26T00:20:03.461 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:03.477729+0000 2024-04-26T00:20:03.461 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:03.461 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.461 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:03.461 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:03.461 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:03.462 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:03.462 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:03.462 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.471+0000 7f4ebffff640 1 -- 172.21.15.37:0/1813291053 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ed00addb0 con 0x7f4ed813c3e0 2024-04-26T00:20:03.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.471+0000 7f4ebffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.471+0000 7f4ebffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:03.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.471+0000 7f4ebffff640 10 monclient: dump: 2024-04-26T00:20:03.462 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.462 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.463 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.463 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.463 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.463 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.463 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.463 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.463 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.463 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.471+0000 7f4eddd7a640 5 monclient: authenticate success, global_id 4208 2024-04-26T00:20:03.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.471+0000 7f4eddd7a640 10 monclient: _renew_subs 2024-04-26T00:20:03.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.471+0000 7f4eddd7a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:03.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.471+0000 7f4eddd7a640 1 -- 172.21.15.37:0/1813291053 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ed81363a0 con 0x7f4ed813c3e0 2024-04-26T00:20:03.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.471+0000 7f4eddd7a640 10 monclient: _renew_subs 2024-04-26T00:20:03.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.471+0000 7f4eddd7a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:03.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.471+0000 7f4eddd7a640 1 -- 172.21.15.37:0/1813291053 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ed809ada0 con 0x7f4ed813c3e0 2024-04-26T00:20:03.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.471+0000 7f5dd8e9b640 1 Processor -- start 2024-04-26T00:20:03.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.471+0000 7f4ebdffb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:03.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.471+0000 7f4ebdffb640 1 -- 172.21.15.37:0/1813291053 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4eb0000f90 con 0x7f4ed813c3e0 2024-04-26T00:20:03.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.471+0000 7f5dd8e9b640 1 -- start start 2024-04-26T00:20:03.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.471+0000 7f5dd8e9b640 10 monclient: init 2024-04-26T00:20:03.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.471+0000 7f4ebffff640 1 -- 172.21.15.37:0/1813291053 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 4) ==== 43962+0+0 (secure 0 0 0) 0x7f4ed00c3460 con 0x7f4ed813c3e0 2024-04-26T00:20:03.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.471+0000 7f4ebffff640 1 --2- 172.21.15.37:0/1813291053 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4ec4035e40 0x7f4ec4038300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.471+0000 7f4ebffff640 1 -- 172.21.15.37:0/1813291053 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7f4ed00e3930 con 0x7f4ed813c3e0 2024-04-26T00:20:03.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f4ebffff640 1 --2- 172.21.15.37:0/1813291053 >> v2:172.21.15.45:6804/152605921 conn(0x7f4ec403b9d0 0x7f4ec403ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f5dd8e9b640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:03.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f5dd8e9b640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-04-26T00:20:03.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f5dd8e9b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5dd4131cd0 0x7f5dd41320b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f5dd8e9b640 10 monclient(hunting): picked mon.noname-c con 0x7f5dd4131cd0 addr v2:172.21.15.45:3300/0 2024-04-26T00:20:03.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f5dd8e9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5dd4073080 0x7f5dd4134c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f5dd8e9b640 10 monclient(hunting): picked mon.noname-b con 0x7f5dd4073080 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:03.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f5dd8e9b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5dd40a1b60 addr v1:172.21.15.45:6789/0 2024-04-26T00:20:03.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f5dd8e9b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5dd4071660 con 0x7f5dd40a1b60 2024-04-26T00:20:03.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f5dd8e9b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f5dd8e9b640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f5dd4071940 con 0x7f5dd4131cd0 2024-04-26T00:20:03.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f5dd8e9b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f5dd8e9b640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5dd4071c20 con 0x7f5dd4073080 2024-04-26T00:20:03.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f5dd8e9b640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:03.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f5dd8e9b640 10 monclient(hunting): authenticate will time out at 984.784147s 2024-04-26T00:20:03.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f4ed77fe640 1 --2- 172.21.15.37:0/1813291053 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4ec4035e40 0x7f4ec4038300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f4ebffff640 1 -- 172.21.15.37:0/1813291053 --> v2:172.21.15.45:6804/152605921 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4ed00a5780 con 0x7f4ec403b9d0 2024-04-26T00:20:03.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f4ebffff640 1 -- 172.21.15.37:0/1813291053 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7f4ed00e82a0 con 0x7f4ed813c3e0 2024-04-26T00:20:03.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f4ebffff640 10 monclient: handle_get_version_reply finishing 1 version 15 2024-04-26T00:20:03.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f4ed77fe640 10 monclient: get_auth_request con 0x7f4ec4035e40 auth_method 0 2024-04-26T00:20:03.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f5dd1d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5dd4073080 0x7f5dd4134c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f5dd1d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5dd4073080 0x7f5dd4134c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:57766/0 (socket says 172.21.15.37:57766) 2024-04-26T00:20:03.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f5dd1d9b640 1 -- 172.21.15.37:0/1587185900 learned_addr learned my addr 172.21.15.37:0/1587185900 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:03.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f5dd1d9b640 10 monclient(hunting): get_auth_request con 0x7f5dd4073080 auth_method 0 2024-04-26T00:20:03.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f5dd1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f5dd1d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f5dd1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f5dd259c640 1 --2- 172.21.15.37:0/1587185900 >> v2:172.21.15.45:3300/0 conn(0x7f5dd4131cd0 0x7f5dd41320b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f4ed6ffd640 1 --2- 172.21.15.37:0/1813291053 >> v2:172.21.15.45:6804/152605921 conn(0x7f4ec403b9d0 0x7f4ec403ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f5dd259c640 10 monclient(hunting): get_auth_request con 0x7f5dd4131cd0 auth_method 0 2024-04-26T00:20:03.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f5dd259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f5dd259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f4ed6ffd640 10 monclient: get_auth_request con 0x7f4ec403b9d0 auth_method 0 2024-04-26T00:20:03.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f5dd259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f5dd259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f5dd259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f5dd259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f5dd259c640 10 monclient(hunting): handle_auth_done global_id 4213 payload 274 2024-04-26T00:20:03.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f5dd259c640 1 -- 172.21.15.37:0/1587185900 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5dd4073080 msgr2=0x7f5dd4134c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:03.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f5dd259c640 1 --2- 172.21.15.37:0/1587185900 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5dd4073080 0x7f5dd4134c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f5dd259c640 1 -- 172.21.15.37:0/1587185900 >> v1:172.21.15.45:6789/0 conn(0x7f5dd40a1b60 legacy=0x7f5dd4135150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:03.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f5dd259c640 10 monclient: _finish_hunting 0 2024-04-26T00:20:03.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f5dd259c640 1 monclient: found mon.noname-c 2024-04-26T00:20:03.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f5dd259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:03.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f5dd259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:20:03.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f5dd259c640 1 -- 172.21.15.37:0/1587185900 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dd41325f0 con 0x7f5dd4131cd0 2024-04-26T00:20:03.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.475+0000 7f4ed77fe640 1 --2- 172.21.15.37:0/1813291053 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4ec4035e40 0x7f4ec4038300 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4ec8006e90 tx=0x7f4ec8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.479+0000 7f5dd259c640 1 -- 172.21.15.37:0/1587185900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5dd4131cd0 msgr2=0x7f5dd41320b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:03.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.479+0000 7f5dd259c640 1 --2- 172.21.15.37:0/1587185900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5dd4131cd0 0x7f5dd41320b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5dc40097c0 tx=0x7f5dc401c3d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f23f3ef0e3814f49 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.479+0000 7f4ed6ffd640 1 --2- 172.21.15.37:0/1813291053 >> v2:172.21.15.45:6804/152605921 conn(0x7f4ec403b9d0 0x7f4ec403ddf0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.479+0000 7f4ebffff640 1 -- 172.21.15.37:0/1813291053 <== osd.4 v2:172.21.15.45:6804/152605921 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4ed00a5780 con 0x7f4ec403b9d0 2024-04-26T00:20:03.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.495+0000 7f4ebdffb640 1 -- 172.21.15.37:0/1813291053 --> v2:172.21.15.45:6804/152605921 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4eb0002c80 con 0x7f4ec403b9d0 2024-04-26T00:20:03.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.495+0000 7f4ebffff640 1 -- 172.21.15.37:0/1813291053 <== osd.4 v2:172.21.15.45:6804/152605921 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f4eb0002c80 con 0x7f4ec403b9d0 2024-04-26T00:20:03.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.495+0000 7f4ebdffb640 1 -- 172.21.15.37:0/1813291053 >> v2:172.21.15.45:6804/152605921 conn(0x7f4ec403b9d0 msgr2=0x7f4ec403ddf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.495+0000 7f4ebdffb640 1 --2- 172.21.15.37:0/1813291053 >> v2:172.21.15.45:6804/152605921 conn(0x7f4ec403b9d0 0x7f4ec403ddf0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.495+0000 7f4ebdffb640 1 -- 172.21.15.37:0/1813291053 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4ec4035e40 msgr2=0x7f4ec4038300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.495+0000 7f4ebdffb640 1 --2- 172.21.15.37:0/1813291053 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4ec4035e40 0x7f4ec4038300 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4ec8006e90 tx=0x7f4ec8008040 comp rx=0 tx=0).stop 2024-04-26T00:20:03.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.495+0000 7f4ebdffb640 10 monclient: shutdown 2024-04-26T00:20:03.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.495+0000 7f4ebdffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:03.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.495+0000 7f4ebdffb640 1 -- 172.21.15.37:0/1813291053 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4ed813c3e0 msgr2=0x7f4ed813b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.495+0000 7f4ebdffb640 1 --2- 172.21.15.37:0/1813291053 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4ed813c3e0 0x7f4ed813b630 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4ed001d0b0 tx=0x7f4ed00195b0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.495+0000 7f4ed7fff640 1 -- 172.21.15.37:0/1813291053 reap_dead start 2024-04-26T00:20:03.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.495+0000 7f4ebdffb640 1 -- 172.21.15.37:0/1813291053 shutdown_connections 2024-04-26T00:20:03.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.495+0000 7f4ebdffb640 1 -- 172.21.15.37:0/1813291053 >> 172.21.15.37:0/1813291053 conn(0x7f4ed80aa0a0 msgr2=0x7f4ed813d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:03.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.495+0000 7f4ebdffb640 1 -- 172.21.15.37:0/1813291053 shutdown_connections 2024-04-26T00:20:03.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.495+0000 7f4ebdffb640 1 -- 172.21.15.37:0/1813291053 wait complete. 2024-04-26T00:20:03.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.503+0000 7feb06cd1640 10 monclient: get_monmap_and_config 2024-04-26T00:20:03.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.503+0000 7feb06cd1640 10 monclient: build_initial_monmap 2024-04-26T00:20:03.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.503+0000 7feb06cd1640 10 monclient: monmap: 2024-04-26T00:20:03.486 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:03.486 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.486 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:03.509874+0000 2024-04-26T00:20:03.486 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:03.509874+0000 2024-04-26T00:20:03.486 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:03.486 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.486 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:03.487 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:03.487 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:03.487 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:03.487 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:03.487 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.503+0000 7feb06cd1640 1 Processor -- start 2024-04-26T00:20:03.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.503+0000 7feb06cd1640 1 -- start start 2024-04-26T00:20:03.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.503+0000 7feb06cd1640 10 monclient: init 2024-04-26T00:20:03.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.503+0000 7feb06cd1640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:03.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.503+0000 7feb06cd1640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-04-26T00:20:03.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.503+0000 7feb06cd1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feb000a3d90 addr v1:172.21.15.45:6789/0 2024-04-26T00:20:03.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.503+0000 7feb06cd1640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7feb000a4170 0x7feb000a45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.503+0000 7feb06cd1640 10 monclient(hunting): picked mon.noname-a con 0x7feb000a4170 addr v2:172.21.15.37:3300/0 2024-04-26T00:20:03.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.503+0000 7feb06cd1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feb0013c3e0 addr v1:172.21.15.37:6789/0 2024-04-26T00:20:03.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.503+0000 7feb0526e640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7feb0013c3e0 0x7feb0013e7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:55780/0 (socket says 172.21.15.37:55780) 2024-04-26T00:20:03.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.503+0000 7feb0526e640 1 -- 172.21.15.37:0/3719297970 learned_addr learned my addr 172.21.15.37:0/3719297970 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:20:03.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.503+0000 7feafffff640 1 --2- 172.21.15.37:0/3719297970 >> v2:172.21.15.37:3300/0 conn(0x7feb000a4170 0x7feb000a45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.503+0000 7feb06cd1640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb000a5150 con 0x7feb0013c3e0 2024-04-26T00:20:03.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.503+0000 7feb06cd1640 1 -- 172.21.15.37:0/3719297970 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb0009b280 con 0x7feb000a3d90 2024-04-26T00:20:03.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.503+0000 7feb06cd1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.503+0000 7feb06cd1640 1 -- 172.21.15.37:0/3719297970 --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7feb0009c650 con 0x7feb000a4170 2024-04-26T00:20:03.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.503+0000 7feb06cd1640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:03.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.503+0000 7feb06cd1640 10 monclient(hunting): authenticate will time out at 984.815610s 2024-04-26T00:20:03.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.503+0000 7feafffff640 10 monclient(hunting): get_auth_request con 0x7feb000a4170 auth_method 0 2024-04-26T00:20:03.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.503+0000 7feafffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.503+0000 7feafffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.503+0000 7feafffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.503+0000 7feafffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.503+0000 7feafffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.503+0000 7feafffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.503+0000 7feafffff640 10 monclient(hunting): handle_auth_done global_id 4224 payload 274 2024-04-26T00:20:03.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.503+0000 7feafffff640 1 -- 172.21.15.37:0/3719297970 >> v1:172.21.15.45:6789/0 conn(0x7feb000a3d90 legacy=0x7feb0009bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:03.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.503+0000 7feafffff640 1 -- 172.21.15.37:0/3719297970 >> v1:172.21.15.37:6789/0 conn(0x7feb0013c3e0 legacy=0x7feb0013e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:03.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.503+0000 7feafffff640 10 monclient: _finish_hunting 0 2024-04-26T00:20:03.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.503+0000 7feafffff640 1 monclient: found mon.noname-a 2024-04-26T00:20:03.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.503+0000 7feafffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:03.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.503+0000 7feafffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:03.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.503+0000 7feafffff640 1 -- 172.21.15.37:0/3719297970 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb0009b560 con 0x7feb000a4170 2024-04-26T00:20:03.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.507+0000 7fca9be56640 10 monclient: get_monmap_and_config 2024-04-26T00:20:03.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.507+0000 7fca9be56640 10 monclient: build_initial_monmap 2024-04-26T00:20:03.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.507+0000 7fca9be56640 10 monclient: monmap: 2024-04-26T00:20:03.494 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:03.494 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.494 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:03.512887+0000 2024-04-26T00:20:03.494 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:03.512887+0000 2024-04-26T00:20:03.494 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:03.494 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.494 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:03.494 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:03.494 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:03.494 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:03.494 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:03.494 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.507+0000 7fca9be56640 1 Processor -- start 2024-04-26T00:20:03.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.507+0000 7fca9be56640 1 -- start start 2024-04-26T00:20:03.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.507+0000 7fca9be56640 10 monclient: init 2024-04-26T00:20:03.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.507+0000 7fca9be56640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:03.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.507+0000 7fca9be56640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-04-26T00:20:03.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.507+0000 7fca9be56640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fca94099420 addr v1:172.21.15.45:6789/0 2024-04-26T00:20:03.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.507+0000 7fca9be56640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fca94099800 0x7fca94138ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.507+0000 7fca9be56640 10 monclient(hunting): picked mon.noname-a con 0x7fca94099800 addr v2:172.21.15.37:3300/0 2024-04-26T00:20:03.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.507+0000 7fca9be56640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fca94099c10 addr v1:172.21.15.37:6789/0 2024-04-26T00:20:03.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.507+0000 7fca9be56640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca9409ed40 con 0x7fca94099c10 2024-04-26T00:20:03.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.507+0000 7fca9be56640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca9409f020 con 0x7fca94099420 2024-04-26T00:20:03.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.507+0000 7fca9be56640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.507+0000 7fca9be56640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fca940a2920 con 0x7fca94099800 2024-04-26T00:20:03.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.507+0000 7fca9be56640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:03.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.507+0000 7fca9be56640 10 monclient(hunting): authenticate will time out at 984.818507s 2024-04-26T00:20:03.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.507+0000 7fca993f1640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fca94099800 0x7fca94138ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.507+0000 7fca9a3f3640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fca94099c10 0x7fca9413b430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:55782/0 (socket says 172.21.15.37:55782) 2024-04-26T00:20:03.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.507+0000 7fca993f1640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fca94099800 0x7fca94138ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44544/0 (socket says 172.21.15.37:44544) 2024-04-26T00:20:03.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.507+0000 7fca9a3f3640 1 -- 172.21.15.37:0/4189056304 learned_addr learned my addr 172.21.15.37:0/4189056304 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:20:03.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.507+0000 7fca993f1640 10 monclient(hunting): get_auth_request con 0x7fca94099800 auth_method 0 2024-04-26T00:20:03.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.507+0000 7fca993f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.507+0000 7fca993f1640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.507+0000 7fca993f1640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.507+0000 7fca993f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.507+0000 7fca993f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.507+0000 7fca993f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.507+0000 7fca993f1640 10 monclient(hunting): handle_auth_done global_id 4227 payload 274 2024-04-26T00:20:03.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.507+0000 7fca993f1640 1 -- 172.21.15.37:0/4189056304 >> v1:172.21.15.45:6789/0 conn(0x7fca94099420 legacy=0x7fca9409ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:03.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.507+0000 7fca993f1640 1 -- 172.21.15.37:0/4189056304 >> v1:172.21.15.37:6789/0 conn(0x7fca94099c10 legacy=0x7fca9413b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:03.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.507+0000 7fca993f1640 10 monclient: _finish_hunting 0 2024-04-26T00:20:03.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.507+0000 7fca993f1640 1 monclient: found mon.noname-a 2024-04-26T00:20:03.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.507+0000 7fca993f1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:03.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.507+0000 7fca993f1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:03.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.507+0000 7fca993f1640 1 -- 172.21.15.37:0/4189056304 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca9409f300 con 0x7fca94099800 2024-04-26T00:20:03.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feafffff640 1 -- 172.21.15.37:0/3719297970 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb000a4170 msgr2=0x7feb000a45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:03.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feafffff640 1 --2- 172.21.15.37:0/3719297970 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb000a4170 0x7feb000a45e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7feaec0097c0 tx=0x7feaec01c2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88833be29d97953 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff412b22640 10 monclient: get_monmap_and_config 2024-04-26T00:20:03.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff412b22640 10 monclient: build_initial_monmap 2024-04-26T00:20:03.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7fd297671640 10 monclient: get_monmap_and_config 2024-04-26T00:20:03.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7fd297671640 10 monclient: build_initial_monmap 2024-04-26T00:20:03.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7fd297671640 10 monclient: monmap: 2024-04-26T00:20:03.499 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:03.499 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.499 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:03.516304+0000 2024-04-26T00:20:03.499 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:03.516304+0000 2024-04-26T00:20:03.499 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:03.499 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.499 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:03.499 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:03.499 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:03.499 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:03.499 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:03.500 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feaff7fe640 1 -- 172.21.15.37:0/3719297970 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaec0b0070 con 0x7feb000a4170 2024-04-26T00:20:03.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feaff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feaff7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:03.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feaff7fe640 10 monclient: dump: 2024-04-26T00:20:03.500 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.500 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.500 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.501 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.501 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.501 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.501 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.501 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.501 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.501 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feaff7fe640 10 monclient: _finish_auth 0 2024-04-26T00:20:03.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feaff7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:20:03.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feaff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:03.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feaff7fe640 1 -- 172.21.15.37:0/3719297970 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feaec0ab600 con 0x7feb000a4170 2024-04-26T00:20:03.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb06cd1640 5 monclient: authenticate success, global_id 4224 2024-04-26T00:20:03.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb06cd1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:20:03.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feaff7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:03.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb06cd1640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:03.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb06cd1640 4 set_mon_vals no callback set 2024-04-26T00:20:03.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feaff7fe640 1 -- 172.21.15.37:0/3719297970 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaec0ab9d0 con 0x7feb000a4170 2024-04-26T00:20:03.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feaff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feaff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:03.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feaff7fe640 10 monclient: dump: 2024-04-26T00:20:03.502 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.502 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.502 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.502 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.502 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.502 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.502 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.502 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.503 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.503 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff412b22640 10 monclient: monmap: 2024-04-26T00:20:03.503 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:03.503 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.503 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:03.516182+0000 2024-04-26T00:20:03.504 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:03.516182+0000 2024-04-26T00:20:03.504 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:03.504 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.504 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:03.504 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:03.504 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:03.504 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:03.504 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:03.504 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff412b22640 1 Processor -- start 2024-04-26T00:20:03.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff412b22640 1 -- start start 2024-04-26T00:20:03.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff412b22640 10 monclient: init 2024-04-26T00:20:03.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb06cd1640 10 monclient: shutdown 2024-04-26T00:20:03.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb06cd1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:03.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb06cd1640 1 -- 172.21.15.37:0/3719297970 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb000a4170 msgr2=0x7feb000a45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb06cd1640 1 --2- 172.21.15.37:0/3719297970 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb000a4170 0x7feb000a45e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7feaec0097c0 tx=0x7feaec01c2e0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb06cd1640 1 -- 172.21.15.37:0/3719297970 shutdown_connections 2024-04-26T00:20:03.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb06cd1640 1 --2- 172.21.15.37:0/3719297970 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb000a4170 0x7feb000a45e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb06cd1640 1 -- 172.21.15.37:0/3719297970 >> 172.21.15.37:0/3719297970 conn(0x7feb000aa0a0 msgr2=0x7feb000a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:03.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb06cd1640 1 -- 172.21.15.37:0/3719297970 shutdown_connections 2024-04-26T00:20:03.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff412b22640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:03.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb06cd1640 1 -- 172.21.15.37:0/3719297970 wait complete. 2024-04-26T00:20:03.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff412b22640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-04-26T00:20:03.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff412b22640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff40c0a3d90 0x7ff40c09bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff412b22640 10 monclient(hunting): picked mon.noname-a con 0x7ff40c0a3d90 addr v2:172.21.15.37:3300/0 2024-04-26T00:20:03.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff412b22640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff40c0a4170 0x7ff40c0a45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff412b22640 10 monclient(hunting): picked mon.noname-c con 0x7ff40c0a4170 addr v2:172.21.15.45:3300/0 2024-04-26T00:20:03.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff412b22640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff40c133b20 0x7ff40c133f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff412b22640 10 monclient(hunting): picked mon.noname-b con 0x7ff40c133b20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:03.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff412b22640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff412b22640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7ff40c0a5150 con 0x7ff40c0a3d90 2024-04-26T00:20:03.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff412b22640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff412b22640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7ff40c09b280 con 0x7ff40c0a4170 2024-04-26T00:20:03.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff412b22640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff412b22640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff40c09b560 con 0x7ff40c133b20 2024-04-26T00:20:03.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff412b22640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:03.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff412b22640 10 monclient(hunting): authenticate will time out at 984.821821s 2024-04-26T00:20:03.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff4110bf640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff40c133b20 0x7ff40c133f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff4110bf640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff40c133b20 0x7ff40c133f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:57768/0 (socket says 172.21.15.37:57768) 2024-04-26T00:20:03.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff4110bf640 1 -- 172.21.15.37:0/3241695312 learned_addr learned my addr 172.21.15.37:0/3241695312 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:03.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff4110bf640 10 monclient(hunting): get_auth_request con 0x7ff40c133b20 auth_method 0 2024-04-26T00:20:03.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff4110bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff4110bf640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff4110bf640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7fd297671640 1 Processor -- start 2024-04-26T00:20:03.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff40bfff640 1 --2- 172.21.15.37:0/3241695312 >> v2:172.21.15.45:3300/0 conn(0x7ff40c0a4170 0x7ff40c0a45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff40bfff640 10 monclient(hunting): get_auth_request con 0x7ff40c0a4170 auth_method 0 2024-04-26T00:20:03.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff40bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff40bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff40bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff4110bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff4110bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff4110bf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff40bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff40bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff40bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7fd297671640 1 -- start start 2024-04-26T00:20:03.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7fd297671640 10 monclient: init 2024-04-26T00:20:03.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb06cd1640 10 monclient: build_initial_monmap 2024-04-26T00:20:03.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb06cd1640 10 monclient: monmap: 2024-04-26T00:20:03.514 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:03.514 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:03.514 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:03.514 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:03.514 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:03.514 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:03.514 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:03.514 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:03.514 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:03.514 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff4110bf640 10 monclient(hunting): handle_auth_done global_id 4214 payload 274 2024-04-26T00:20:03.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff4110bf640 1 -- 172.21.15.37:0/3241695312 >> v2:172.21.15.45:3300/0 conn(0x7ff40c0a4170 msgr2=0x7ff40c0a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:03.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff4110bf640 1 --2- 172.21.15.37:0/3241695312 >> v2:172.21.15.45:3300/0 conn(0x7ff40c0a4170 0x7ff40c0a45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff4110bf640 1 -- 172.21.15.37:0/3241695312 >> v2:172.21.15.37:3300/0 conn(0x7ff40c0a3d90 msgr2=0x7ff40c09bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:03.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff4110bf640 1 --2- 172.21.15.37:0/3241695312 >> v2:172.21.15.37:3300/0 conn(0x7ff40c0a3d90 0x7ff40c09bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff4110bf640 10 monclient: _finish_hunting 0 2024-04-26T00:20:03.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff4110bf640 1 monclient: found mon.noname-b 2024-04-26T00:20:03.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff4110bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:03.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff4110bf640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:20:03.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7ff4110bf640 1 -- 172.21.15.37:0/3241695312 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff40c0a4b20 con 0x7ff40c133b20 2024-04-26T00:20:03.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7fd297671640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:03.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb06cd1640 1 Processor -- start 2024-04-26T00:20:03.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7fd297671640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-04-26T00:20:03.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7fd297671640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd2900a3ee0 addr v1:172.21.15.37:6789/0 2024-04-26T00:20:03.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7fd297671640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd2900a5b70 0x7fd29013c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7fd297671640 10 monclient(hunting): picked mon.noname-a con 0x7fd2900a5b70 addr v2:172.21.15.37:3300/0 2024-04-26T00:20:03.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7fd297671640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd29009c620 addr v1:172.21.15.45:6789/0 2024-04-26T00:20:03.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7fd297671640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd29009bc40 con 0x7fd2900a3ee0 2024-04-26T00:20:03.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7fd297671640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd29009b680 con 0x7fd29009c620 2024-04-26T00:20:03.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7fd297671640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7fd297671640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fd29009b960 con 0x7fd2900a5b70 2024-04-26T00:20:03.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7fd297671640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:03.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7fd297671640 10 monclient(hunting): authenticate will time out at 984.822943s 2024-04-26T00:20:03.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7fd295c0e640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fd29009c620 0x7fd29013e970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.37:60440/0 (socket says 172.21.15.37:60440) 2024-04-26T00:20:03.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7fd295c0e640 1 -- 172.21.15.37:0/2617952584 learned_addr learned my addr 172.21.15.37:0/2617952584 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:20:03.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb06cd1640 1 -- start start 2024-04-26T00:20:03.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb06cd1640 10 monclient: init 2024-04-26T00:20:03.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb06cd1640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:03.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb06cd1640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:20:03.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb06cd1640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb000a3d90 0x7feb001cedf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb06cd1640 10 monclient(hunting): picked mon.noname-a con 0x7feb000a3d90 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:03.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb06cd1640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb000a4170 0x7feb001cf5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb06cd1640 10 monclient(hunting): picked mon.noname-c con 0x7feb000a4170 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:03.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb06cd1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb0013c3e0 0x7feb00138d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb06cd1640 10 monclient(hunting): picked mon.noname-b con 0x7feb0013c3e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:03.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb06cd1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb06cd1640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7feb00142aa0 con 0x7feb000a3d90 2024-04-26T00:20:03.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb06cd1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb06cd1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7feb00142920 con 0x7feb0013c3e0 2024-04-26T00:20:03.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb06cd1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb06cd1640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7feb00142c20 con 0x7feb000a4170 2024-04-26T00:20:03.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb06cd1640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:03.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb06cd1640 10 monclient(hunting): authenticate will time out at 984.823385s 2024-04-26T00:20:03.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7fca993f1640 1 -- 172.21.15.37:0/4189056304 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fca94099800 msgr2=0x7fca94138ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:03.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7fca993f1640 1 --2- 172.21.15.37:0/4189056304 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fca94099800 0x7fca94138ee0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fca840097c0 tx=0x7fca8401c2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd948b4a903b1875 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb04a6d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb000a3d90 0x7feb001cedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb04a6d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb000a3d90 0x7feb001cedf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44566/0 (socket says 172.21.15.37:44566) 2024-04-26T00:20:03.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb04a6d640 1 -- 172.21.15.37:0/1346693862 learned_addr learned my addr 172.21.15.37:0/1346693862 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:03.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb04a6d640 10 monclient(hunting): get_auth_request con 0x7feb000a3d90 auth_method 0 2024-04-26T00:20:03.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb04a6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb04a6d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb04a6d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7fd294c0c640 1 --2- 172.21.15.37:0/2617952584 >> v2:172.21.15.37:3300/0 conn(0x7fd2900a5b70 0x7fd29013c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feafffff640 1 --2- 172.21.15.37:0/1346693862 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb000a4170 0x7feb001cf5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7fd294c0c640 10 monclient(hunting): get_auth_request con 0x7fd2900a5b70 auth_method 0 2024-04-26T00:20:03.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7fd294c0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feafffff640 10 monclient(hunting): get_auth_request con 0x7feb000a4170 auth_method 0 2024-04-26T00:20:03.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feafffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feafffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7fd294c0c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7fd294c0c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feafffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb04a6d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb04a6d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb04a6d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feafffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feafffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb0526e640 1 --2- 172.21.15.37:0/1346693862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb0013c3e0 0x7feb00138d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feafffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb0526e640 10 monclient(hunting): get_auth_request con 0x7feb0013c3e0 auth_method 0 2024-04-26T00:20:03.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb0526e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb0526e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.511+0000 7feb0526e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7feb0526e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7feb0526e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7feb0526e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7fd294c0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7fd294c0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7fd294c0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7fd294c0c640 10 monclient(hunting): handle_auth_done global_id 4233 payload 274 2024-04-26T00:20:03.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7fd294c0c640 1 -- 172.21.15.37:0/2617952584 >> v1:172.21.15.45:6789/0 conn(0x7fd29009c620 legacy=0x7fd29013e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:03.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7fd294c0c640 1 -- 172.21.15.37:0/2617952584 >> v1:172.21.15.37:6789/0 conn(0x7fd2900a3ee0 legacy=0x7fd2900a42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:03.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7fd294c0c640 10 monclient: _finish_hunting 0 2024-04-26T00:20:03.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7fd294c0c640 1 monclient: found mon.noname-a 2024-04-26T00:20:03.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7fd294c0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:03.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7fd294c0c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:03.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7fd294c0c640 1 -- 172.21.15.37:0/2617952584 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2900732b0 con 0x7fd2900a5b70 2024-04-26T00:20:03.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7feafffff640 10 monclient(hunting): handle_auth_done global_id 4217 payload 824 2024-04-26T00:20:03.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7feafffff640 1 -- 172.21.15.37:0/1346693862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb0013c3e0 msgr2=0x7feb00138d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7feafffff640 1 --2- 172.21.15.37:0/1346693862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb0013c3e0 0x7feb00138d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7feafffff640 1 -- 172.21.15.37:0/1346693862 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb000a3d90 msgr2=0x7feb001cedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7feafffff640 1 --2- 172.21.15.37:0/1346693862 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb000a3d90 0x7feb001cedf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7feafffff640 10 monclient: _finish_hunting 0 2024-04-26T00:20:03.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7feafffff640 1 monclient: found mon.noname-c 2024-04-26T00:20:03.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7feafffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:03.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7feafffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:20:03.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7feafffff640 1 -- 172.21.15.37:0/1346693862 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb0013a9d0 con 0x7feb000a4170 2024-04-26T00:20:03.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7feb04a6d640 1 --2- 172.21.15.37:0/1346693862 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb000a3d90 0x7feb001cedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:03.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7feb0526e640 1 --2- 172.21.15.37:0/1346693862 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb0013c3e0 0x7feb00138d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:03.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7f5dd159a640 1 -- 172.21.15.37:0/1587185900 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc40b1070 con 0x7f5dd4131cd0 2024-04-26T00:20:03.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7f5dd159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7f5dd159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:20:03.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7f5dd159a640 10 monclient: dump: 2024-04-26T00:20:03.528 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.528 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.528 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.528 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.528 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.529 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.529 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.529 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.529 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.529 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7f5dd159a640 10 monclient: _finish_auth 0 2024-04-26T00:20:03.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7f5dd159a640 10 monclient: _check_auth_tickets 2024-04-26T00:20:03.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7f5dd159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:03.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7f5dd8e9b640 5 monclient: authenticate success, global_id 4213 2024-04-26T00:20:03.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7f5dd8e9b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:20:03.530 INFO:teuthology.orchestra.run.smithi037.stdout:47244640258 2024-04-26T00:20:03.530 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-26T00:20:03.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7fca98bf0640 1 -- 172.21.15.37:0/4189056304 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca840b0070 con 0x7fca94099800 2024-04-26T00:20:03.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7fca98bf0640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7fca98bf0640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:03.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7fca98bf0640 10 monclient: dump: 2024-04-26T00:20:03.531 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.531 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.532 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.532 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.532 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.532 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.532 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.532 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.532 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.532 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7fca98bf0640 10 monclient: _finish_auth 0 2024-04-26T00:20:03.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7fca98bf0640 10 monclient: _check_auth_tickets 2024-04-26T00:20:03.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7fca98bf0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:03.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7fca98bf0640 1 -- 172.21.15.37:0/4189056304 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca840ab600 con 0x7fca94099800 2024-04-26T00:20:03.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7fca9be56640 5 monclient: authenticate success, global_id 4227 2024-04-26T00:20:03.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7fca9be56640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:20:03.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7fca98bf0640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:03.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7fca9be56640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:03.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7fca9be56640 4 set_mon_vals no callback set 2024-04-26T00:20:03.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7fca98bf0640 1 -- 172.21.15.37:0/4189056304 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca840ab9d0 con 0x7fca94099800 2024-04-26T00:20:03.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7fca98bf0640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7fca98bf0640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:03.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7fca98bf0640 10 monclient: dump: 2024-04-26T00:20:03.533 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.533 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.533 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.533 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.533 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.533 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.533 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.533 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.534 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.534 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7f5dd159a640 1 -- 172.21.15.37:0/1587185900 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dd4135a70 con 0x7f5dd4131cd0 2024-04-26T00:20:03.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7f5dd159a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:03.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7f5dd8e9b640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:03.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7f5dd8e9b640 4 set_mon_vals no callback set 2024-04-26T00:20:03.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7f5dd159a640 1 -- 172.21.15.37:0/1587185900 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc40ac610 con 0x7f5dd4131cd0 2024-04-26T00:20:03.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7f5dd159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7f5dd159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:20:03.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7f5dd159a640 10 monclient: dump: 2024-04-26T00:20:03.535 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.535 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.535 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.535 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.535 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.536 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.536 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.536 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.536 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.536 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7f5dd8e9b640 10 monclient: shutdown 2024-04-26T00:20:03.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7f5dd8e9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:03.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7f5dd8e9b640 1 -- 172.21.15.37:0/1587185900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5dd4131cd0 msgr2=0x7f5dd41320b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7f5dd8e9b640 1 --2- 172.21.15.37:0/1587185900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5dd4131cd0 0x7f5dd41320b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5dc40097c0 tx=0x7f5dc401c3d0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7f5dd8e9b640 1 -- 172.21.15.37:0/1587185900 shutdown_connections 2024-04-26T00:20:03.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7f5dd8e9b640 1 --2- 172.21.15.37:0/1587185900 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5dd4073080 0x7f5dd4134c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7f5dd8e9b640 1 --2- 172.21.15.37:0/1587185900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5dd4131cd0 0x7f5dd41320b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7f5dd8e9b640 1 -- 172.21.15.37:0/1587185900 >> 172.21.15.37:0/1587185900 conn(0x7f5dd40a7160 msgr2=0x7f5dd40a7540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:03.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7f5dd8e9b640 1 -- 172.21.15.37:0/1587185900 shutdown_connections 2024-04-26T00:20:03.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7f5dd8e9b640 1 -- 172.21.15.37:0/1587185900 wait complete. 2024-04-26T00:20:03.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7fca9be56640 10 monclient: shutdown 2024-04-26T00:20:03.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7fca9be56640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:03.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7fca9be56640 1 -- 172.21.15.37:0/4189056304 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fca94099800 msgr2=0x7fca94138ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7fca9be56640 1 --2- 172.21.15.37:0/4189056304 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fca94099800 0x7fca94138ee0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fca840097c0 tx=0x7fca8401c2e0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7feafffff640 1 -- 172.21.15.37:0/1346693862 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb000a4170 msgr2=0x7feb001cf5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:03.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7feafffff640 1 --2- 172.21.15.37:0/1346693862 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb000a4170 0x7feb001cf5c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7feaec0ad000 tx=0x7feaec019080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7fca9be56640 1 -- 172.21.15.37:0/4189056304 shutdown_connections 2024-04-26T00:20:03.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7fca9be56640 1 --2- 172.21.15.37:0/4189056304 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fca94099800 0x7fca94138ee0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7fca9be56640 1 -- 172.21.15.37:0/4189056304 >> 172.21.15.37:0/4189056304 conn(0x7fca940a2c50 msgr2=0x7fca940a3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:03.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7fca9be56640 1 -- 172.21.15.37:0/4189056304 shutdown_connections 2024-04-26T00:20:03.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7fca9be56640 1 -- 172.21.15.37:0/4189056304 wait complete. 2024-04-26T00:20:03.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7feafd7fa640 1 -- 172.21.15.37:0/1346693862 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaec0b9070 con 0x7feb000a4170 2024-04-26T00:20:03.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7feafd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7feafd7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:20:03.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7feafd7fa640 10 monclient: dump: 2024-04-26T00:20:03.540 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.540 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.540 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.540 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.540 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.540 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.540 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.540 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.540 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.540 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7feafd7fa640 10 monclient: _finish_auth 0 2024-04-26T00:20:03.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7feafd7fa640 10 monclient: _check_auth_tickets 2024-04-26T00:20:03.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7feafd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:03.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7feafd7fa640 1 -- 172.21.15.37:0/1346693862 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feaec0b5890 con 0x7feb000a4170 2024-04-26T00:20:03.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7feafd7fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:03.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7feafd7fa640 1 -- 172.21.15.37:0/1346693862 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaec0b0040 con 0x7feb000a4170 2024-04-26T00:20:03.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7feafd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7feafd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:03.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7feafd7fa640 10 monclient: dump: 2024-04-26T00:20:03.541 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.541 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.541 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.541 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.541 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.541 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.541 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.542 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.542 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.542 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7feafeffd640 4 set_mon_vals no callback set 2024-04-26T00:20:03.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7ff4110bf640 1 -- 172.21.15.37:0/3241695312 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff40c133b20 msgr2=0x7ff40c133f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:03.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7ff4110bf640 1 --2- 172.21.15.37:0/3241695312 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff40c133b20 0x7ff40c133f90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff40c09d140 tx=0x7ff40401c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79d5df15a8fc8394 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7feb06cd1640 5 monclient: authenticate success, global_id 4217 2024-04-26T00:20:03.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7feb06cd1640 10 monclient: _renew_subs 2024-04-26T00:20:03.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7feb06cd1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:03.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7feb06cd1640 1 -- 172.21.15.37:0/1346693862 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb00120460 con 0x7feb000a4170 2024-04-26T00:20:03.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7feb06cd1640 10 monclient: _renew_subs 2024-04-26T00:20:03.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7feb06cd1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:03.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7feb06cd1640 1 -- 172.21.15.37:0/1346693862 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7feb0011f9e0 con 0x7feb000a4170 2024-04-26T00:20:03.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7f5dd8e9b640 10 monclient: build_initial_monmap 2024-04-26T00:20:03.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7f5dd8e9b640 10 monclient: monmap: 2024-04-26T00:20:03.544 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:03.544 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:03.544 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:03.544 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:03.544 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:03.544 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:03.544 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:03.544 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:03.544 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:03.544 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7f5dd8e9b640 1 Processor -- start 2024-04-26T00:20:03.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7ff40b7fe640 1 -- 172.21.15.37:0/3241695312 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4040a5640 con 0x7ff40c133b20 2024-04-26T00:20:03.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7ff40b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7ff40b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:20:03.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7feb06cd1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:03.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7feb06cd1640 1 -- 172.21.15.37:0/1346693862 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7feb000a0890 con 0x7feb000a4170 2024-04-26T00:20:03.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7feafd7fa640 1 -- 172.21.15.37:0/1346693862 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 4) ==== 43962+0+0 (secure 0 0 0) 0x7feaec0bc070 con 0x7feb000a4170 2024-04-26T00:20:03.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7ff40b7fe640 10 monclient: dump: 2024-04-26T00:20:03.546 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.546 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.546 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.546 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.546 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.546 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.546 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.546 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.546 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.546 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7ff40b7fe640 10 monclient: _finish_auth 0 2024-04-26T00:20:03.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7ff40b7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:20:03.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7ff40b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:03.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7ff40b7fe640 1 -- 172.21.15.37:0/3241695312 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4040a57e0 con 0x7ff40c133b20 2024-04-26T00:20:03.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7ff40b7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:03.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7ff40b7fe640 1 -- 172.21.15.37:0/3241695312 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4040a5bb0 con 0x7ff40c133b20 2024-04-26T00:20:03.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7ff40b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7ff40b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:03.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7ff40b7fe640 10 monclient: dump: 2024-04-26T00:20:03.547 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.547 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.547 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.547 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.547 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.547 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.547 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.547 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.548 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.548 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7ff412b22640 5 monclient: authenticate success, global_id 4214 2024-04-26T00:20:03.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7ff412b22640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:03.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7ff412b22640 4 set_mon_vals no callback set 2024-04-26T00:20:03.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff412b22640 10 monclient: shutdown 2024-04-26T00:20:03.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff412b22640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:03.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff412b22640 1 -- 172.21.15.37:0/3241695312 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff40c133b20 msgr2=0x7ff40c133f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff412b22640 1 --2- 172.21.15.37:0/3241695312 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff40c133b20 0x7ff40c133f90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff40c09d140 tx=0x7ff40401c8b0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7f5dd8e9b640 1 -- start start 2024-04-26T00:20:03.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7f5dd8e9b640 10 monclient: init 2024-04-26T00:20:03.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd8e9b640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:03.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd8e9b640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:20:03.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd8e9b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5dd4073080 0x7f5dd4143010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd8e9b640 10 monclient(hunting): picked mon.noname-a con 0x7f5dd4073080 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:03.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd8e9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5dd40a1b60 0x7f5dd413ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd8e9b640 10 monclient(hunting): picked mon.noname-b con 0x7f5dd40a1b60 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:03.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd8e9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5dd4131cd0 0x7f5dd4140460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd8e9b640 10 monclient(hunting): picked mon.noname-c con 0x7f5dd4131cd0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:03.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd8e9b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd8e9b640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd413fb00 con 0x7f5dd4073080 2024-04-26T00:20:03.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd8e9b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd8e9b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd413f980 con 0x7f5dd40a1b60 2024-04-26T00:20:03.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd8e9b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd8e9b640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5dd413fc80 con 0x7f5dd4131cd0 2024-04-26T00:20:03.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd8e9b640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:03.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd8e9b640 10 monclient(hunting): authenticate will time out at 984.828151s 2024-04-26T00:20:03.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd1d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5dd40a1b60 0x7f5dd413ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd1d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5dd40a1b60 0x7f5dd413ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.37:55618/0 (socket says 172.21.15.37:55618) 2024-04-26T00:20:03.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd1d9b640 1 -- 172.21.15.37:0/3955027388 learned_addr learned my addr 172.21.15.37:0/3955027388 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:03.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd1d9b640 10 monclient(hunting): get_auth_request con 0x7f5dd40a1b60 auth_method 0 2024-04-26T00:20:03.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd1d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd259c640 1 --2- 172.21.15.37:0/3955027388 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5dd4073080 0x7f5dd4143010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7feafffff640 1 --2- 172.21.15.37:0/1346693862 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb000a4170 0x7feb001cf5c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7feaec0ad000 tx=0x7feaec019080 comp rx=0 tx=0).queue_received will delay after 1.71409e+09 on 0x7feaec0e72a0 mon_get_version_reply(handle=1 version=15) 2024-04-26T00:20:03.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd259c640 10 monclient(hunting): get_auth_request con 0x7f5dd4073080 auth_method 0 2024-04-26T00:20:03.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd294c0c640 1 -- 172.21.15.37:0/2617952584 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd2900a5b70 msgr2=0x7fd29013c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:03.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd294c0c640 1 --2- 172.21.15.37:0/2617952584 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd2900a5b70 0x7fd29013c420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd2800097c0 tx=0x7fd28001c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc55d3a5dc20f4f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7feafd7fa640 1 --2- 172.21.15.37:0/1346693862 >> v2:172.21.15.45:6812/2379918623 conn(0x7feae4035e40 0x7feae4038300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7feafd7fa640 1 -- 172.21.15.37:0/1346693862 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7feaec0e2970 con 0x7feb000a4170 2024-04-26T00:20:03.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff412b22640 1 -- 172.21.15.37:0/3241695312 shutdown_connections 2024-04-26T00:20:03.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff412b22640 1 --2- 172.21.15.37:0/3241695312 >> v2:172.21.15.45:3300/0 conn(0x7ff40c0a4170 0x7ff40c0a45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff412b22640 1 --2- 172.21.15.37:0/3241695312 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff40c133b20 0x7ff40c133f90 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff412b22640 1 --2- 172.21.15.37:0/3241695312 >> v2:172.21.15.37:3300/0 conn(0x7ff40c0a3d90 0x7ff40c09bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff412b22640 1 -- 172.21.15.37:0/3241695312 >> 172.21.15.37:0/3241695312 conn(0x7ff40c0aa0a0 msgr2=0x7ff40c0a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:03.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7fca9be56640 10 monclient: build_initial_monmap 2024-04-26T00:20:03.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7fca9be56640 10 monclient: monmap: 2024-04-26T00:20:03.554 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:03.554 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:03.554 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:03.554 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:03.554 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:03.554 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:03.554 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:03.554 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:03.554 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:03.554 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.515+0000 7fca9be56640 1 Processor -- start 2024-04-26T00:20:03.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca9be56640 1 -- start start 2024-04-26T00:20:03.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca9be56640 10 monclient: init 2024-04-26T00:20:03.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca9be56640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:03.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca9be56640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:20:03.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca9be56640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca94099420 0x7fca94072130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd259c640 10 monclient(hunting): handle_auth_done global_id 4236 payload 824 2024-04-26T00:20:03.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd259c640 1 -- 172.21.15.37:0/3955027388 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5dd4131cd0 msgr2=0x7f5dd4140460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd259c640 1 --2- 172.21.15.37:0/3955027388 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5dd4131cd0 0x7f5dd4140460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd259c640 1 -- 172.21.15.37:0/3955027388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5dd40a1b60 msgr2=0x7f5dd413ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd259c640 1 --2- 172.21.15.37:0/3955027388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5dd40a1b60 0x7f5dd413ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd259c640 10 monclient: _finish_hunting 0 2024-04-26T00:20:03.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd259c640 1 monclient: found mon.noname-a 2024-04-26T00:20:03.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca9be56640 10 monclient(hunting): picked mon.noname-b con 0x7fca94099420 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:03.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca9be56640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fca94099800 0x7fca94132400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca9be56640 10 monclient(hunting): picked mon.noname-c con 0x7fca94099800 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:03.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca9be56640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fca94099c10 0x7fca94132940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca9be56640 10 monclient(hunting): picked mon.noname-a con 0x7fca94099c10 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:03.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca9be56640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca9be56640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fca9413fb10 con 0x7fca94099c10 2024-04-26T00:20:03.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca9be56640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca9be56640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fca9413f990 con 0x7fca94099420 2024-04-26T00:20:03.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca9be56640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca9be56640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fca9413fc90 con 0x7fca94099800 2024-04-26T00:20:03.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:03.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:03.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd259c640 1 -- 172.21.15.37:0/3955027388 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dd4134b30 con 0x7f5dd4073080 2024-04-26T00:20:03.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca9be56640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:03.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca9be56640 10 monclient(hunting): authenticate will time out at 984.828769s 2024-04-26T00:20:03.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca9a3f3640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fca94099c10 0x7fca94132940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca9a3f3640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fca94099c10 0x7fca94132940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44574/0 (socket says 172.21.15.37:44574) 2024-04-26T00:20:03.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca9a3f3640 1 -- 172.21.15.37:0/2598006807 learned_addr learned my addr 172.21.15.37:0/2598006807 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:03.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca9a3f3640 10 monclient(hunting): get_auth_request con 0x7fca94099c10 auth_method 0 2024-04-26T00:20:03.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca9a3f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca9a3f3640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca9a3f3640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca99bf2640 1 --2- 172.21.15.37:0/2598006807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca94099420 0x7fca94072130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca99bf2640 10 monclient(hunting): get_auth_request con 0x7fca94099420 auth_method 0 2024-04-26T00:20:03.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca99bf2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca99bf2640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca99bf2640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca9a3f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca9a3f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7feafd7fa640 1 --2- 172.21.15.37:0/1346693862 >> v2:172.21.15.37:6800/603175474 conn(0x7feae403b950 0x7feae403dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7feb04a6d640 1 --2- 172.21.15.37:0/1346693862 >> v2:172.21.15.45:6812/2379918623 conn(0x7feae4035e40 0x7feae4038300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca9a3f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca99bf2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca99bf2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca99bf2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7feb04a6d640 10 monclient: get_auth_request con 0x7feae4035e40 auth_method 0 2024-04-26T00:20:03.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7feafd7fa640 1 -- 172.21.15.37:0/1346693862 --> v2:172.21.15.37:6800/603175474 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feaec007e50 con 0x7feae403b950 2024-04-26T00:20:03.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca9a3f3640 10 monclient(hunting): handle_auth_done global_id 4239 payload 824 2024-04-26T00:20:03.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca9a3f3640 1 -- 172.21.15.37:0/2598006807 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fca94099800 msgr2=0x7fca94132400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca9a3f3640 1 --2- 172.21.15.37:0/2598006807 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fca94099800 0x7fca94132400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca9a3f3640 1 -- 172.21.15.37:0/2598006807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca94099420 msgr2=0x7fca94072130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca9a3f3640 1 --2- 172.21.15.37:0/2598006807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fca94099420 0x7fca94072130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca9a3f3640 10 monclient: _finish_hunting 0 2024-04-26T00:20:03.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca9a3f3640 1 monclient: found mon.noname-a 2024-04-26T00:20:03.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff412b22640 1 -- 172.21.15.37:0/3241695312 shutdown_connections 2024-04-26T00:20:03.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff412b22640 1 -- 172.21.15.37:0/3241695312 wait complete. 2024-04-26T00:20:03.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd287fff640 1 -- 172.21.15.37:0/2617952584 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2800b0070 con 0x7fd2900a5b70 2024-04-26T00:20:03.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd287fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd287fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:03.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca9a3f3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:03.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca9a3f3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:03.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fca9a3f3640 1 -- 172.21.15.37:0/2598006807 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca94070c00 con 0x7fca94099c10 2024-04-26T00:20:03.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7feb0526e640 1 --2- 172.21.15.37:0/1346693862 >> v2:172.21.15.37:6800/603175474 conn(0x7feae403b950 0x7feae403dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd287fff640 10 monclient: dump: 2024-04-26T00:20:03.563 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.564 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.564 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.564 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.564 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.564 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.564 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.564 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.564 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.564 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd287fff640 10 monclient: _finish_auth 0 2024-04-26T00:20:03.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd287fff640 10 monclient: _check_auth_tickets 2024-04-26T00:20:03.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd287fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:03.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd297671640 5 monclient: authenticate success, global_id 4233 2024-04-26T00:20:03.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd287fff640 1 -- 172.21.15.37:0/2617952584 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2800a95f0 con 0x7fd2900a5b70 2024-04-26T00:20:03.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd297671640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:20:03.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd287fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:03.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd287fff640 1 -- 172.21.15.37:0/2617952584 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2800a99c0 con 0x7fd2900a5b70 2024-04-26T00:20:03.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd287fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd287fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:03.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd287fff640 10 monclient: dump: 2024-04-26T00:20:03.565 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.565 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.565 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.565 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.565 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.565 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.565 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.565 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.566 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.566 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd297671640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:03.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd297671640 4 set_mon_vals no callback set 2024-04-26T00:20:03.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7feb0526e640 10 monclient: get_auth_request con 0x7feae403b950 auth_method 0 2024-04-26T00:20:03.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff412b22640 10 monclient: build_initial_monmap 2024-04-26T00:20:03.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff412b22640 10 monclient: monmap: 2024-04-26T00:20:03.567 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:03.567 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:03.567 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:03.567 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:03.567 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:03.567 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:03.567 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:03.567 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:03.567 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:03.567 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd297671640 10 monclient: shutdown 2024-04-26T00:20:03.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd297671640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:03.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd297671640 1 -- 172.21.15.37:0/2617952584 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd2900a5b70 msgr2=0x7fd29013c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd297671640 1 --2- 172.21.15.37:0/2617952584 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd2900a5b70 0x7fd29013c420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd2800097c0 tx=0x7fd28001c370 comp rx=0 tx=0).stop 2024-04-26T00:20:03.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff412b22640 1 Processor -- start 2024-04-26T00:20:03.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd297671640 1 -- 172.21.15.37:0/2617952584 shutdown_connections 2024-04-26T00:20:03.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd297671640 1 --2- 172.21.15.37:0/2617952584 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd2900a5b70 0x7fd29013c420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd297671640 1 -- 172.21.15.37:0/2617952584 >> 172.21.15.37:0/2617952584 conn(0x7fd2900a5fe0 msgr2=0x7fd2900a63f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:03.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd297671640 1 -- 172.21.15.37:0/2617952584 shutdown_connections 2024-04-26T00:20:03.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff412b22640 1 -- start start 2024-04-26T00:20:03.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff412b22640 10 monclient: init 2024-04-26T00:20:03.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd297671640 1 -- 172.21.15.37:0/2617952584 wait complete. 2024-04-26T00:20:03.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff412b22640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:03.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff412b22640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:20:03.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff412b22640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff40c0a3d90 0x7ff40c1cef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff412b22640 10 monclient(hunting): picked mon.noname-b con 0x7ff40c0a3d90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:03.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff412b22640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff40c0a4170 0x7ff40c1cf460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff412b22640 10 monclient(hunting): picked mon.noname-a con 0x7ff40c0a4170 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:03.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff412b22640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff40c133b20 0x7ff40c1cf9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff412b22640 10 monclient(hunting): picked mon.noname-c con 0x7ff40c133b20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:03.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff412b22640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff412b22640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff40c140c10 con 0x7ff40c0a4170 2024-04-26T00:20:03.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff412b22640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff412b22640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff40c09c690 con 0x7ff40c0a3d90 2024-04-26T00:20:03.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff412b22640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff412b22640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff40c140d90 con 0x7ff40c133b20 2024-04-26T00:20:03.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff412b22640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:03.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff412b22640 10 monclient(hunting): authenticate will time out at 984.830912s 2024-04-26T00:20:03.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd297671640 10 monclient: build_initial_monmap 2024-04-26T00:20:03.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd297671640 10 monclient: monmap: 2024-04-26T00:20:03.572 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:03.572 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:03.572 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:03.572 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:03.572 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:03.572 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:03.572 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:03.572 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:03.572 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:03.572 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff4108be640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff40c0a3d90 0x7ff40c1cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff4108be640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff40c0a3d90 0x7ff40c1cef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.37:55636/0 (socket says 172.21.15.37:55636) 2024-04-26T00:20:03.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff4108be640 1 -- 172.21.15.37:0/3038845700 learned_addr learned my addr 172.21.15.37:0/3038845700 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:03.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff4108be640 10 monclient(hunting): get_auth_request con 0x7ff40c0a3d90 auth_method 0 2024-04-26T00:20:03.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff4108be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff4108be640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff4108be640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff4108be640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff4108be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff4108be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd297671640 1 Processor -- start 2024-04-26T00:20:03.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd297671640 1 -- start start 2024-04-26T00:20:03.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd297671640 10 monclient: init 2024-04-26T00:20:03.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff4108be640 10 monclient(hunting): handle_auth_done global_id 4228 payload 824 2024-04-26T00:20:03.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff4108be640 1 -- 172.21.15.37:0/3038845700 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff40c133b20 msgr2=0x7ff40c1cf9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff4108be640 1 --2- 172.21.15.37:0/3038845700 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff40c133b20 0x7ff40c1cf9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff4108be640 1 -- 172.21.15.37:0/3038845700 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff40c0a4170 msgr2=0x7ff40c1cf460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff4108be640 1 --2- 172.21.15.37:0/3038845700 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff40c0a4170 0x7ff40c1cf460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff4108be640 10 monclient: _finish_hunting 0 2024-04-26T00:20:03.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff4108be640 1 monclient: found mon.noname-b 2024-04-26T00:20:03.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff4108be640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:03.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff4108be640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.45:3300/0 2024-04-26T00:20:03.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7ff4108be640 1 -- 172.21.15.37:0/3038845700 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff40c13a6d0 con 0x7ff40c0a3d90 2024-04-26T00:20:03.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd297671640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:03.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd297671640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:20:03.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd297671640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd29009c620 0x7fd2901cf170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd297671640 10 monclient(hunting): picked mon.noname-a con 0x7fd29009c620 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:03.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd297671640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2900a3ee0 0x7fd29013f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd297671640 10 monclient(hunting): picked mon.noname-c con 0x7fd2900a3ee0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:03.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd297671640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2900a5b70 0x7fd29013f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd297671640 10 monclient(hunting): picked mon.noname-b con 0x7fd2900a5b70 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:03.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd297671640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd297671640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd290142cc0 con 0x7fd29009c620 2024-04-26T00:20:03.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd297671640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd297671640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd290142b40 con 0x7fd2900a5b70 2024-04-26T00:20:03.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd297671640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd297671640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd290142e40 con 0x7fd2900a3ee0 2024-04-26T00:20:03.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd297671640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:03.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7fd297671640 10 monclient(hunting): authenticate will time out at 984.831991s 2024-04-26T00:20:03.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd259c640 1 -- 172.21.15.37:0/3955027388 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5dd4073080 msgr2=0x7f5dd4143010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:03.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.519+0000 7f5dd259c640 1 --2- 172.21.15.37:0/3955027388 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5dd4073080 0x7f5dd4143010 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5dc40ac000 tx=0x7f5dc4019170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fd294c0c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2900a3ee0 0x7fd29013f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fd294c0c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2900a3ee0 0x7fd29013f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:57816/0 (socket says 172.21.15.37:57816) 2024-04-26T00:20:03.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fd294c0c640 1 -- 172.21.15.37:0/838721133 learned_addr learned my addr 172.21.15.37:0/838721133 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:03.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fd294c0c640 10 monclient(hunting): get_auth_request con 0x7fd2900a3ee0 auth_method 0 2024-04-26T00:20:03.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fd294c0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fd294c0c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fd294c0c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fd295c0e640 1 --2- 172.21.15.37:0/838721133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2900a5b70 0x7fd29013f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fd295c0e640 10 monclient(hunting): get_auth_request con 0x7fd2900a5b70 auth_method 0 2024-04-26T00:20:03.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fd295c0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fd295c0e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fd295c0e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fd294c0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fd294c0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7f5dc2ffd640 1 -- 172.21.15.37:0/3955027388 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc40b1040 con 0x7f5dd4073080 2024-04-26T00:20:03.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7f5dc2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7f5dc2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:03.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7f5dc2ffd640 10 monclient: dump: 2024-04-26T00:20:03.580 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.580 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.580 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.580 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.580 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.580 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.580 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.581 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.581 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.581 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7f5dc2ffd640 10 monclient: _finish_auth 0 2024-04-26T00:20:03.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7f5dc2ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:20:03.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7f5dc2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:03.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7f5dd8e9b640 5 monclient: authenticate success, global_id 4236 2024-04-26T00:20:03.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7f5dd8e9b640 10 monclient: _renew_subs 2024-04-26T00:20:03.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7f5dd8e9b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:03.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7f5dd8e9b640 1 -- 172.21.15.37:0/3955027388 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dd40a0d90 con 0x7f5dd4073080 2024-04-26T00:20:03.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7f5dd8e9b640 10 monclient: _renew_subs 2024-04-26T00:20:03.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7f5dd8e9b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:03.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7f5dd8e9b640 1 -- 172.21.15.37:0/3955027388 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dd41428c0 con 0x7f5dd4073080 2024-04-26T00:20:03.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fd294c0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fd295c0e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fd295c0e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fd295c0e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fd295c0e640 10 monclient(hunting): handle_auth_done global_id 4231 payload 824 2024-04-26T00:20:03.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fd295c0e640 1 -- 172.21.15.37:0/838721133 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2900a3ee0 msgr2=0x7fd29013f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fd295c0e640 1 --2- 172.21.15.37:0/838721133 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2900a3ee0 0x7fd29013f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fd295c0e640 1 -- 172.21.15.37:0/838721133 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd29009c620 msgr2=0x7fd2901cf170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fd295c0e640 1 --2- 172.21.15.37:0/838721133 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd29009c620 0x7fd2901cf170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fd295c0e640 10 monclient: _finish_hunting 0 2024-04-26T00:20:03.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fd295c0e640 1 monclient: found mon.noname-b 2024-04-26T00:20:03.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fd295c0e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:03.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fd295c0e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.45:3300/0 2024-04-26T00:20:03.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fd295c0e640 1 -- 172.21.15.37:0/838721133 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd29013a870 con 0x7fd2900a5b70 2024-04-26T00:20:03.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fca9a3f3640 1 -- 172.21.15.37:0/2598006807 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fca94099c10 msgr2=0x7fca94132940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:03.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fca9a3f3640 1 --2- 172.21.15.37:0/2598006807 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fca94099c10 0x7fca94132940 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fca8c00caf0 tx=0x7fca8c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7feb04a6d640 1 --2- 172.21.15.37:0/1346693862 >> v2:172.21.15.45:6812/2379918623 conn(0x7feae4035e40 0x7feae4038300 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7feaf4004470 tx=0x7feaf4003700 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7f5dc2ffd640 1 -- 172.21.15.37:0/3955027388 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dc4003970 con 0x7f5dd4073080 2024-04-26T00:20:03.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7f5dc2ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:03.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7f5dc2ffd640 1 -- 172.21.15.37:0/3955027388 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc4003d20 con 0x7f5dd4073080 2024-04-26T00:20:03.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7f5dc2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7f5dc2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:03.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7f5dc2ffd640 10 monclient: dump: 2024-04-26T00:20:03.586 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.586 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.586 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.586 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.586 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.586 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.586 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.587 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.587 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.587 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7f5dd0d99640 4 set_mon_vals no callback set 2024-04-26T00:20:03.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7f5dd8e9b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:03.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7f5dd8e9b640 1 -- 172.21.15.37:0/3955027388 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5dd4135a70 con 0x7f5dd4073080 2024-04-26T00:20:03.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fca8a7fc640 1 -- 172.21.15.37:0/2598006807 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca8c012400 con 0x7fca94099c10 2024-04-26T00:20:03.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fca8a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fca8a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:03.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fca8a7fc640 10 monclient: dump: 2024-04-26T00:20:03.588 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.588 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.589 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.589 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.589 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.589 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.589 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.589 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.589 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.589 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fca8a7fc640 10 monclient: _finish_auth 0 2024-04-26T00:20:03.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fca8a7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:20:03.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fca8a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:03.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fca9be56640 5 monclient: authenticate success, global_id 4239 2024-04-26T00:20:03.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fca9be56640 10 monclient: _renew_subs 2024-04-26T00:20:03.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fca9be56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:03.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fca9be56640 1 -- 172.21.15.37:0/2598006807 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca940a25f0 con 0x7fca94099c10 2024-04-26T00:20:03.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fca9be56640 10 monclient: _renew_subs 2024-04-26T00:20:03.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fca9be56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:03.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fca9be56640 1 -- 172.21.15.37:0/2598006807 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca94070640 con 0x7fca94099c10 2024-04-26T00:20:03.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fca8a7fc640 1 -- 172.21.15.37:0/2598006807 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca8c0125a0 con 0x7fca94099c10 2024-04-26T00:20:03.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fca8a7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:03.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fca8a7fc640 1 -- 172.21.15.37:0/2598006807 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca8c012950 con 0x7fca94099c10 2024-04-26T00:20:03.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fca8a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fca8a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:03.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fca8a7fc640 10 monclient: dump: 2024-04-26T00:20:03.590 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.590 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.590 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.590 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.590 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.591 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.591 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.591 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.591 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.591 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7ff4108be640 1 -- 172.21.15.37:0/3038845700 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff40c0a3d90 msgr2=0x7ff40c1cef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:03.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7ff4108be640 1 --2- 172.21.15.37:0/3038845700 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff40c0a3d90 0x7ff40c1cef20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff40000b550 tx=0x7ff40000ba10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7feb0526e640 1 --2- 172.21.15.37:0/1346693862 >> v2:172.21.15.37:6800/603175474 conn(0x7feae403b950 0x7feae403dd70 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7ff4097fa640 1 -- 172.21.15.37:0/3038845700 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff400015020 con 0x7ff40c0a3d90 2024-04-26T00:20:03.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7ff4097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7ff4097fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:20:03.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7ff4097fa640 10 monclient: dump: 2024-04-26T00:20:03.592 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.592 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.592 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.593 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.593 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.593 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.593 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.593 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.593 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.593 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7ff4097fa640 10 monclient: _finish_auth 0 2024-04-26T00:20:03.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7ff4097fa640 10 monclient: _check_auth_tickets 2024-04-26T00:20:03.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7ff4097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:03.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7ff4097fa640 1 -- 172.21.15.37:0/3038845700 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff400011ac0 con 0x7ff40c0a3d90 2024-04-26T00:20:03.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7ff4097fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:03.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7ff4097fa640 1 -- 172.21.15.37:0/3038845700 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4000103e0 con 0x7ff40c0a3d90 2024-04-26T00:20:03.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7ff4097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7ff4097fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:20:03.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7ff4097fa640 10 monclient: dump: 2024-04-26T00:20:03.594 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.594 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.594 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.594 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.594 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.594 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.594 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.594 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.594 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.594 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7ff40affd640 4 set_mon_vals no callback set 2024-04-26T00:20:03.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7ff412b22640 5 monclient: authenticate success, global_id 4228 2024-04-26T00:20:03.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7ff412b22640 10 monclient: _renew_subs 2024-04-26T00:20:03.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7ff412b22640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.45:3300/0 2024-04-26T00:20:03.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7ff412b22640 1 -- 172.21.15.37:0/3038845700 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff40c13c9c0 con 0x7ff40c0a3d90 2024-04-26T00:20:03.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7ff412b22640 10 monclient: _renew_subs 2024-04-26T00:20:03.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7ff412b22640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.45:3300/0 2024-04-26T00:20:03.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7ff412b22640 1 -- 172.21.15.37:0/3038845700 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff40c13d970 con 0x7ff40c0a3d90 2024-04-26T00:20:03.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7feafd7fa640 1 -- 172.21.15.37:0/1346693862 <== osd.2 v2:172.21.15.37:6800/603175474 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7feaec007e50 con 0x7feae403b950 2024-04-26T00:20:03.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fd295c0e640 1 -- 172.21.15.37:0/838721133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2900a5b70 msgr2=0x7fd29013f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:03.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fd295c0e640 1 --2- 172.21.15.37:0/838721133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2900a5b70 0x7fd29013f880 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd28800cac0 tx=0x7fd28800cec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7ff412b22640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.45:3300/0 2024-04-26T00:20:03.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7ff412b22640 1 -- 172.21.15.37:0/3038845700 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff40c09ada0 con 0x7ff40c0a3d90 2024-04-26T00:20:03.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fca8b7fe640 4 set_mon_vals no callback set 2024-04-26T00:20:03.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fca9be56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:03.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7fca9be56640 1 -- 172.21.15.37:0/2598006807 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fca9406f480 con 0x7fca94099c10 2024-04-26T00:20:03.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7f5dc2ffd640 1 -- 172.21.15.37:0/3955027388 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 4) ==== 43962+0+0 (secure 0 0 0) 0x7f5dc40c8430 con 0x7f5dd4073080 2024-04-26T00:20:03.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7f5dc2ffd640 1 --2- 172.21.15.37:0/3955027388 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5db4035df0 0x7f5db40382b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7f5dc2ffd640 1 -- 172.21.15.37:0/3955027388 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7f5dc40e2ca0 con 0x7f5dd4073080 2024-04-26T00:20:03.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7f5dc2ffd640 1 --2- 172.21.15.37:0/3955027388 >> v2:172.21.15.45:6808/3915746722 conn(0x7f5db403b890 0x7f5db403dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7f5dc2ffd640 1 -- 172.21.15.37:0/3955027388 --> v2:172.21.15.45:6808/3915746722 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5db403e320 con 0x7f5db403b890 2024-04-26T00:20:03.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7f5dc2ffd640 1 -- 172.21.15.37:0/3955027388 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7f5dc40e2f80 con 0x7f5dd4073080 2024-04-26T00:20:03.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.523+0000 7f5dc2ffd640 10 monclient: handle_get_version_reply finishing 1 version 15 2024-04-26T00:20:03.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.527+0000 7f5dd1d9b640 1 --2- 172.21.15.37:0/3955027388 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5db4035df0 0x7f5db40382b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.527+0000 7f5dd1d9b640 10 monclient: get_auth_request con 0x7f5db4035df0 auth_method 0 2024-04-26T00:20:03.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.527+0000 7ff4097fa640 1 -- 172.21.15.37:0/3038845700 <== mon.1 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 4) ==== 43962+0+0 (secure 0 0 0) 0x7ff400018020 con 0x7ff40c0a3d90 2024-04-26T00:20:03.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.527+0000 7f5dd2d9d640 1 --2- 172.21.15.37:0/3955027388 >> v2:172.21.15.45:6808/3915746722 conn(0x7f5db403b890 0x7f5db403dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.527+0000 7f5dd2d9d640 10 monclient: get_auth_request con 0x7f5db403b890 auth_method 0 2024-04-26T00:20:03.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.527+0000 7fca8a7fc640 1 -- 172.21.15.37:0/2598006807 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 4) ==== 43962+0+0 (secure 0 0 0) 0x7fca8c012af0 con 0x7fca94099c10 2024-04-26T00:20:03.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.527+0000 7ff4097fa640 1 --2- 172.21.15.37:0/3038845700 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff3f0035df0 0x7ff3f00382b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.527+0000 7ff4097fa640 1 -- 172.21.15.37:0/3038845700 <== mon.1 v2:172.21.15.45:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7ff400049160 con 0x7ff40c0a3d90 2024-04-26T00:20:03.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.527+0000 7ff4097fa640 1 --2- 172.21.15.37:0/3038845700 >> v2:172.21.15.45:6800/2173737584 conn(0x7ff3f003b900 0x7ff3f003dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.527+0000 7ff4097fa640 1 -- 172.21.15.37:0/3038845700 --> v2:172.21.15.45:6800/2173737584 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff3f003e390 con 0x7ff3f003b900 2024-04-26T00:20:03.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.527+0000 7ff4097fa640 1 -- 172.21.15.37:0/3038845700 <== mon.1 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7ff400049440 con 0x7ff40c0a3d90 2024-04-26T00:20:03.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.527+0000 7ff4097fa640 10 monclient: handle_get_version_reply finishing 1 version 15 2024-04-26T00:20:03.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.527+0000 7fca8a7fc640 1 --2- 172.21.15.37:0/2598006807 >> v2:172.21.15.45:6812/2379918623 conn(0x7fca78035df0 0x7fca780382b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.527+0000 7ff40bfff640 1 --2- 172.21.15.37:0/3038845700 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff3f0035df0 0x7ff3f00382b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.527+0000 7fca8a7fc640 1 -- 172.21.15.37:0/2598006807 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7fca8c048ec0 con 0x7fca94099c10 2024-04-26T00:20:03.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.527+0000 7ff40bfff640 10 monclient: get_auth_request con 0x7ff3f0035df0 auth_method 0 2024-04-26T00:20:03.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.527+0000 7ff4110bf640 1 --2- 172.21.15.37:0/3038845700 >> v2:172.21.15.45:6800/2173737584 conn(0x7ff3f003b900 0x7ff3f003dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.527+0000 7ff4110bf640 10 monclient: get_auth_request con 0x7ff3f003b900 auth_method 0 2024-04-26T00:20:03.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.527+0000 7fca8a7fc640 1 --2- 172.21.15.37:0/2598006807 >> v2:172.21.15.37:6804/1084527868 conn(0x7fca7803b890 0x7fca7803dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.527+0000 7fca8a7fc640 1 -- 172.21.15.37:0/2598006807 --> v2:172.21.15.37:6804/1084527868 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fca8c006e50 con 0x7fca7803b890 2024-04-26T00:20:03.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.527+0000 7fca8a7fc640 1 -- 172.21.15.37:0/2598006807 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7fca8c04b440 con 0x7fca94099c10 2024-04-26T00:20:03.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.527+0000 7fca8a7fc640 10 monclient: handle_get_version_reply finishing 1 version 15 2024-04-26T00:20:03.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.527+0000 7fca993f1640 1 --2- 172.21.15.37:0/2598006807 >> v2:172.21.15.37:6804/1084527868 conn(0x7fca7803b890 0x7fca7803dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.527+0000 7f5dd2d9d640 1 -- 172.21.15.37:0/3955027388 >> v2:172.21.15.45:6808/3915746722 conn(0x7f5db403b890 msgr2=0x7f5db403dcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:20:03.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.527+0000 7fca993f1640 10 monclient: get_auth_request con 0x7fca7803b890 auth_method 0 2024-04-26T00:20:03.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.527+0000 7f5dd2d9d640 1 -- 172.21.15.37:0/3955027388 >> v2:172.21.15.45:6808/3915746722 conn(0x7f5db403b890 msgr2=0x7f5db403dcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:20:03.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.527+0000 7f5dd2d9d640 1 --2- 172.21.15.37:0/3955027388 >> v2:172.21.15.45:6808/3915746722 conn(0x7f5db403b890 0x7f5db403dcb0 crc :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:20:03.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.527+0000 7f5dd2d9d640 1 --2- 172.21.15.37:0/3955027388 >> v2:172.21.15.45:6808/3915746722 conn(0x7f5db403b890 0x7f5db403dcb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:20:03.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.527+0000 7f5dd1d9b640 1 --2- 172.21.15.37:0/3955027388 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5db4035df0 0x7f5db40382b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5dd4131930 tx=0x7f5dc8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.527+0000 7ff40bfff640 1 --2- 172.21.15.37:0/3038845700 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff3f0035df0 0x7ff3f00382b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff40c13a0a0 tx=0x7ff3f8009520 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.531+0000 7fca993f1640 1 --2- 172.21.15.37:0/2598006807 >> v2:172.21.15.37:6804/1084527868 conn(0x7fca7803b890 0x7fca7803dcb0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.531+0000 7ff4110bf640 1 --2- 172.21.15.37:0/3038845700 >> v2:172.21.15.45:6800/2173737584 conn(0x7ff3f003b900 0x7ff3f003dd20 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.531+0000 7fd285ffb640 1 -- 172.21.15.37:0/838721133 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd28800ad80 con 0x7fd2900a5b70 2024-04-26T00:20:03.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.531+0000 7fd285ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.531+0000 7fd285ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:20:03.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.531+0000 7fd285ffb640 10 monclient: dump: 2024-04-26T00:20:03.605 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.605 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.605 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.605 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.605 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.605 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.605 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.605 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.605 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.605 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.531+0000 7fd285ffb640 10 monclient: _finish_auth 0 2024-04-26T00:20:03.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.531+0000 7fd285ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:20:03.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.531+0000 7fd285ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:03.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.531+0000 7fd285ffb640 1 -- 172.21.15.37:0/838721133 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd288012400 con 0x7fd2900a5b70 2024-04-26T00:20:03.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.531+0000 7fd285ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:03.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.531+0000 7fd285ffb640 1 -- 172.21.15.37:0/838721133 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2880127f0 con 0x7fd2900a5b70 2024-04-26T00:20:03.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.531+0000 7fd297671640 5 monclient: authenticate success, global_id 4231 2024-04-26T00:20:03.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.531+0000 7fd297671640 10 monclient: _renew_subs 2024-04-26T00:20:03.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.531+0000 7fd297671640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.45:3300/0 2024-04-26T00:20:03.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.531+0000 7fd297671640 1 -- 172.21.15.37:0/838721133 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd29009dd40 con 0x7fd2900a5b70 2024-04-26T00:20:03.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.531+0000 7fd297671640 10 monclient: _renew_subs 2024-04-26T00:20:03.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.531+0000 7fd297671640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.45:3300/0 2024-04-26T00:20:03.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.531+0000 7fd297671640 1 -- 172.21.15.37:0/838721133 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd29013c8f0 con 0x7fd2900a5b70 2024-04-26T00:20:03.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.531+0000 7fd285ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.531+0000 7fd2877fe640 4 set_mon_vals no callback set 2024-04-26T00:20:03.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.531+0000 7fd285ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:20:03.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.531+0000 7fd285ffb640 10 monclient: dump: 2024-04-26T00:20:03.607 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.607 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.607 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.607 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.607 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.607 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.607 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.607 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.607 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.607 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.531+0000 7fd297671640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.45:3300/0 2024-04-26T00:20:03.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.531+0000 7fd297671640 1 -- 172.21.15.37:0/838721133 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd270000f90 con 0x7fd2900a5b70 2024-04-26T00:20:03.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.531+0000 7fca8a7fc640 1 -- 172.21.15.37:0/2598006807 <== osd.1 v2:172.21.15.37:6804/1084527868 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fca8c006e50 con 0x7fca7803b890 2024-04-26T00:20:03.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.531+0000 7fd285ffb640 1 -- 172.21.15.37:0/838721133 <== mon.1 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 4) ==== 43962+0+0 (secure 0 0 0) 0x7fd288012a10 con 0x7fd2900a5b70 2024-04-26T00:20:03.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.531+0000 7ff4097fa640 1 -- 172.21.15.37:0/3038845700 <== osd.3 v2:172.21.15.45:6800/2173737584 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff3f003e390 con 0x7ff3f003b900 2024-04-26T00:20:03.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.531+0000 7fd285ffb640 1 --2- 172.21.15.37:0/838721133 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd274035e40 0x7fd274038300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.531+0000 7fd285ffb640 1 -- 172.21.15.37:0/838721133 <== mon.1 v2:172.21.15.45:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7fd288048dd0 con 0x7fd2900a5b70 2024-04-26T00:20:03.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.531+0000 7fd285ffb640 1 --2- 172.21.15.37:0/838721133 >> v2:172.21.15.37:6806/3204000569 conn(0x7fd27403b950 0x7fd27403dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.531+0000 7fd285ffb640 1 -- 172.21.15.37:0/838721133 --> v2:172.21.15.37:6806/3204000569 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd27403e3e0 con 0x7fd27403b950 2024-04-26T00:20:03.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.531+0000 7fd285ffb640 1 -- 172.21.15.37:0/838721133 <== mon.1 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7fd28801a770 con 0x7fd2900a5b70 2024-04-26T00:20:03.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.531+0000 7fd285ffb640 10 monclient: handle_get_version_reply finishing 1 version 15 2024-04-26T00:20:03.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.531+0000 7fd294c0c640 1 --2- 172.21.15.37:0/838721133 >> v2:172.21.15.37:6806/3204000569 conn(0x7fd27403b950 0x7fd27403dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.531+0000 7fd294c0c640 10 monclient: get_auth_request con 0x7fd27403b950 auth_method 0 2024-04-26T00:20:03.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.539+0000 7fd294c0c640 1 --2- 172.21.15.37:0/838721133 >> v2:172.21.15.37:6806/3204000569 conn(0x7fd27403b950 0x7fd27403dd70 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.539+0000 7fd294c0c640 1 -- 172.21.15.37:0/838721133 >> v2:172.21.15.37:6806/3204000569 conn(0x7fd27403b950 msgr2=0x7fd27403dd70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:20:03.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.539+0000 7fd294c0c640 1 -- 172.21.15.37:0/838721133 >> v2:172.21.15.37:6806/3204000569 conn(0x7fd27403b950 msgr2=0x7fd27403dd70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:20:03.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.539+0000 7fd294c0c640 1 --2- 172.21.15.37:0/838721133 >> v2:172.21.15.37:6806/3204000569 conn(0x7fd27403b950 0x7fd27403dd70 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:20:03.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.539+0000 7fd294c0c640 1 --2- 172.21.15.37:0/838721133 >> v2:172.21.15.37:6806/3204000569 conn(0x7fd27403b950 0x7fd27403dd70 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.539+0000 7fd285ffb640 1 -- 172.21.15.37:0/838721133 >> v2:172.21.15.37:6806/3204000569 conn(0x7fd27403b950 msgr2=0x7fd27403dd70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:20:03.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.539+0000 7fd285ffb640 1 --2- 172.21.15.37:0/838721133 >> v2:172.21.15.37:6806/3204000569 conn(0x7fd27403b950 0x7fd27403dd70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.539+0000 7fd285ffb640 1 --2- 172.21.15.37:0/838721133 >> v2:172.21.15.37:6806/3204000569 conn(0x7fd27403ffe0 0x7fd2740423f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.539+0000 7fd285ffb640 1 -- 172.21.15.37:0/838721133 --> v2:172.21.15.37:6806/3204000569 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd274042a50 con 0x7fd27403ffe0 2024-04-26T00:20:03.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.539+0000 7fd285ffb640 10 monclient: _renew_subs 2024-04-26T00:20:03.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.539+0000 7fd285ffb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.45:3300/0 2024-04-26T00:20:03.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.539+0000 7fd285ffb640 1 -- 172.21.15.37:0/838721133 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=16}) -- 0x7fd274042c30 con 0x7fd2900a5b70 2024-04-26T00:20:03.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.539+0000 7fd294c0c640 1 --2- 172.21.15.37:0/838721133 >> v2:172.21.15.37:6806/3204000569 conn(0x7fd27403ffe0 0x7fd2740423f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.539+0000 7fd294c0c640 10 monclient: get_auth_request con 0x7fd27403ffe0 auth_method 0 2024-04-26T00:20:03.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.543+0000 7fd294c0c640 1 --2- 172.21.15.37:0/838721133 >> v2:172.21.15.37:6806/3204000569 conn(0x7fd27403ffe0 0x7fd2740423f0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.543+0000 7fd285ffb640 1 -- 172.21.15.37:0/838721133 <== osd.0 v2:172.21.15.37:6806/3204000569 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd274042a50 con 0x7fd27403ffe0 2024-04-26T00:20:03.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.555+0000 7feb06cd1640 1 -- 172.21.15.37:0/1346693862 --> v2:172.21.15.37:6800/603175474 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7feb0004f510 con 0x7feae403b950 2024-04-26T00:20:03.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.555+0000 7feafd7fa640 1 -- 172.21.15.37:0/1346693862 <== osd.2 v2:172.21.15.37:6800/603175474 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7feb0004f510 con 0x7feae403b950 2024-04-26T00:20:03.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.555+0000 7feb06cd1640 1 -- 172.21.15.37:0/1346693862 >> v2:172.21.15.37:6800/603175474 conn(0x7feae403b950 msgr2=0x7feae403dd70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.555+0000 7feb06cd1640 1 --2- 172.21.15.37:0/1346693862 >> v2:172.21.15.37:6800/603175474 conn(0x7feae403b950 0x7feae403dd70 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.555+0000 7feb06cd1640 1 -- 172.21.15.37:0/1346693862 >> v2:172.21.15.45:6812/2379918623 conn(0x7feae4035e40 msgr2=0x7feae4038300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.555+0000 7feb06cd1640 1 --2- 172.21.15.37:0/1346693862 >> v2:172.21.15.45:6812/2379918623 conn(0x7feae4035e40 0x7feae4038300 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7feaf4004470 tx=0x7feaf4003700 comp rx=0 tx=0).stop 2024-04-26T00:20:03.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.555+0000 7feb06cd1640 10 monclient: shutdown 2024-04-26T00:20:03.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.555+0000 7feb06cd1640 20 monclient: shutdown canceling and discarding version request 1 2024-04-26T00:20:03.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.555+0000 7feb06cd1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:03.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.555+0000 7feb06cd1640 1 -- 172.21.15.37:0/1346693862 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb000a4170 msgr2=0x7feb001cf5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.555+0000 7feb06cd1640 1 --2- 172.21.15.37:0/1346693862 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb000a4170 0x7feb001cf5c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7feaec0ad000 tx=0x7feaec019080 comp rx=0 tx=0).stop 2024-04-26T00:20:03.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.555+0000 7feb0526e640 1 -- 172.21.15.37:0/1346693862 reap_dead start 2024-04-26T00:20:03.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.555+0000 7feb06cd1640 1 -- 172.21.15.37:0/1346693862 shutdown_connections 2024-04-26T00:20:03.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.555+0000 7feb06cd1640 1 -- 172.21.15.37:0/1346693862 >> 172.21.15.37:0/1346693862 conn(0x7feb000aa0a0 msgr2=0x7feb000a2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:03.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.555+0000 7feb06cd1640 1 -- 172.21.15.37:0/1346693862 shutdown_connections 2024-04-26T00:20:03.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.555+0000 7feb06cd1640 1 -- 172.21.15.37:0/1346693862 wait complete. 2024-04-26T00:20:03.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.559+0000 7fd297671640 1 -- 172.21.15.37:0/838721133 --> v2:172.21.15.37:6806/3204000569 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd270002c80 con 0x7fd27403ffe0 2024-04-26T00:20:03.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.559+0000 7fd285ffb640 1 -- 172.21.15.37:0/838721133 <== osd.0 v2:172.21.15.37:6806/3204000569 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd270002c80 con 0x7fd27403ffe0 2024-04-26T00:20:03.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.559+0000 7fd297671640 1 -- 172.21.15.37:0/838721133 >> v2:172.21.15.37:6806/3204000569 conn(0x7fd27403ffe0 msgr2=0x7fd2740423f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.559+0000 7fd297671640 1 --2- 172.21.15.37:0/838721133 >> v2:172.21.15.37:6806/3204000569 conn(0x7fd27403ffe0 0x7fd2740423f0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.559+0000 7fd297671640 1 -- 172.21.15.37:0/838721133 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd274035e40 msgr2=0x7fd274038300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.559+0000 7fd297671640 1 --2- 172.21.15.37:0/838721133 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd274035e40 0x7fd274038300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.559+0000 7fd297671640 10 monclient: shutdown 2024-04-26T00:20:03.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.559+0000 7fd297671640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:03.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.559+0000 7fd297671640 1 -- 172.21.15.37:0/838721133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2900a5b70 msgr2=0x7fd29013f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.559+0000 7fd297671640 1 --2- 172.21.15.37:0/838721133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2900a5b70 0x7fd29013f880 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd28800cac0 tx=0x7fd28800cec0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.559+0000 7fd295c0e640 1 -- 172.21.15.37:0/838721133 reap_dead start 2024-04-26T00:20:03.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.559+0000 7fd297671640 1 -- 172.21.15.37:0/838721133 shutdown_connections 2024-04-26T00:20:03.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.559+0000 7fd297671640 1 -- 172.21.15.37:0/838721133 >> 172.21.15.37:0/838721133 conn(0x7fd2900a5fe0 msgr2=0x7fd2900a28f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:03.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.559+0000 7fd297671640 1 -- 172.21.15.37:0/838721133 shutdown_connections 2024-04-26T00:20:03.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.559+0000 7fd297671640 1 -- 172.21.15.37:0/838721133 wait complete. 2024-04-26T00:20:03.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.559+0000 7fca9be56640 1 -- 172.21.15.37:0/2598006807 --> v2:172.21.15.37:6804/1084527868 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fca9404f510 con 0x7fca7803b890 2024-04-26T00:20:03.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.563+0000 7fca99bf2640 1 --2- 172.21.15.37:0/2598006807 >> v2:172.21.15.45:6812/2379918623 conn(0x7fca78035df0 0x7fca780382b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.563+0000 7fca99bf2640 10 monclient: get_auth_request con 0x7fca78035df0 auth_method 0 2024-04-26T00:20:03.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.563+0000 7fca8a7fc640 1 -- 172.21.15.37:0/2598006807 <== osd.1 v2:172.21.15.37:6804/1084527868 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fca9404f510 con 0x7fca7803b890 2024-04-26T00:20:03.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.563+0000 7fca9be56640 1 -- 172.21.15.37:0/2598006807 >> v2:172.21.15.37:6804/1084527868 conn(0x7fca7803b890 msgr2=0x7fca7803dcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.563+0000 7fca9be56640 1 --2- 172.21.15.37:0/2598006807 >> v2:172.21.15.37:6804/1084527868 conn(0x7fca7803b890 0x7fca7803dcb0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.563+0000 7ff412b22640 1 -- 172.21.15.37:0/3038845700 --> v2:172.21.15.45:6800/2173737584 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff40c0a0920 con 0x7ff3f003b900 2024-04-26T00:20:03.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.563+0000 7fca9be56640 1 -- 172.21.15.37:0/2598006807 >> v2:172.21.15.45:6812/2379918623 conn(0x7fca78035df0 msgr2=0x7fca780382b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.563+0000 7fca9be56640 1 --2- 172.21.15.37:0/2598006807 >> v2:172.21.15.45:6812/2379918623 conn(0x7fca78035df0 0x7fca780382b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7fca90003650 tx=0x7fca90007e20 comp rx=0 tx=0).stop 2024-04-26T00:20:03.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.563+0000 7fca9be56640 10 monclient: shutdown 2024-04-26T00:20:03.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.563+0000 7fca9be56640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:03.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.563+0000 7fca9be56640 1 -- 172.21.15.37:0/2598006807 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fca94099c10 msgr2=0x7fca94132940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.563+0000 7fca9be56640 1 --2- 172.21.15.37:0/2598006807 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fca94099c10 0x7fca94132940 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fca8c00caf0 tx=0x7fca8c00a060 comp rx=0 tx=0).stop 2024-04-26T00:20:03.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.563+0000 7fca9a3f3640 1 -- 172.21.15.37:0/2598006807 reap_dead start 2024-04-26T00:20:03.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.563+0000 7fca9be56640 1 -- 172.21.15.37:0/2598006807 shutdown_connections 2024-04-26T00:20:03.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.563+0000 7fca9be56640 1 -- 172.21.15.37:0/2598006807 >> 172.21.15.37:0/2598006807 conn(0x7fca940a2c50 msgr2=0x7fca940a4c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:03.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.563+0000 7fca9be56640 1 -- 172.21.15.37:0/2598006807 shutdown_connections 2024-04-26T00:20:03.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.563+0000 7fca9be56640 1 -- 172.21.15.37:0/2598006807 wait complete. 2024-04-26T00:20:03.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.563+0000 7ff4097fa640 1 -- 172.21.15.37:0/3038845700 <== osd.3 v2:172.21.15.45:6800/2173737584 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff40c0a0920 con 0x7ff3f003b900 2024-04-26T00:20:03.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.563+0000 7ff412b22640 1 -- 172.21.15.37:0/3038845700 >> v2:172.21.15.45:6800/2173737584 conn(0x7ff3f003b900 msgr2=0x7ff3f003dd20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.563+0000 7ff412b22640 1 --2- 172.21.15.37:0/3038845700 >> v2:172.21.15.45:6800/2173737584 conn(0x7ff3f003b900 0x7ff3f003dd20 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.563+0000 7ff412b22640 1 -- 172.21.15.37:0/3038845700 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff3f0035df0 msgr2=0x7ff3f00382b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.563+0000 7ff412b22640 1 --2- 172.21.15.37:0/3038845700 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff3f0035df0 0x7ff3f00382b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff40c13a0a0 tx=0x7ff3f8009520 comp rx=0 tx=0).stop 2024-04-26T00:20:03.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.563+0000 7ff412b22640 10 monclient: shutdown 2024-04-26T00:20:03.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.563+0000 7ff412b22640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:03.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.563+0000 7ff412b22640 1 -- 172.21.15.37:0/3038845700 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff40c0a3d90 msgr2=0x7ff40c1cef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.563+0000 7ff412b22640 1 --2- 172.21.15.37:0/3038845700 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff40c0a3d90 0x7ff40c1cef20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff40000b550 tx=0x7ff40000ba10 comp rx=0 tx=0).stop 2024-04-26T00:20:03.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.563+0000 7ff4110bf640 1 -- 172.21.15.37:0/3038845700 reap_dead start 2024-04-26T00:20:03.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.563+0000 7ff412b22640 1 -- 172.21.15.37:0/3038845700 shutdown_connections 2024-04-26T00:20:03.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.563+0000 7ff412b22640 1 -- 172.21.15.37:0/3038845700 >> 172.21.15.37:0/3038845700 conn(0x7ff40c0aa0a0 msgr2=0x7ff40c0a23e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:03.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.563+0000 7ff412b22640 1 -- 172.21.15.37:0/3038845700 shutdown_connections 2024-04-26T00:20:03.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.563+0000 7ff412b22640 1 -- 172.21.15.37:0/3038845700 wait complete. 2024-04-26T00:20:03.622 INFO:teuthology.orchestra.run.smithi037.stdout:47244640259 2024-04-26T00:20:03.623 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-26T00:20:03.624 INFO:teuthology.orchestra.run.smithi037.stdout:47244640258 2024-04-26T00:20:03.624 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-26T00:20:03.625 INFO:teuthology.orchestra.run.smithi037.stdout:47244640260 2024-04-26T00:20:03.625 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-26T00:20:03.626 INFO:teuthology.orchestra.run.smithi037.stdout:47244640259 2024-04-26T00:20:03.626 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-26T00:20:03.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.731+0000 7f5dd2d9d640 1 --2- 172.21.15.37:0/3955027388 >> v2:172.21.15.45:6808/3915746722 conn(0x7f5db403b890 0x7f5db403dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.731+0000 7f5dd2d9d640 10 monclient: get_auth_request con 0x7f5db403b890 auth_method 0 2024-04-26T00:20:03.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.735+0000 7f5dd2d9d640 1 --2- 172.21.15.37:0/3955027388 >> v2:172.21.15.45:6808/3915746722 conn(0x7f5db403b890 0x7f5db403dcb0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.735+0000 7f5dc2ffd640 1 -- 172.21.15.37:0/3955027388 <== osd.5 v2:172.21.15.45:6808/3915746722 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5db403e320 con 0x7f5db403b890 2024-04-26T00:20:03.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb54b1b8640 10 monclient: get_monmap_and_config 2024-04-26T00:20:03.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb54b1b8640 10 monclient: build_initial_monmap 2024-04-26T00:20:03.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb54b1b8640 10 monclient: monmap: 2024-04-26T00:20:03.745 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:03.746 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.746 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:03.769027+0000 2024-04-26T00:20:03.746 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:03.769027+0000 2024-04-26T00:20:03.746 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:03.746 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.746 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:03.746 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:03.746 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:03.746 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:03.746 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:03.746 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb54b1b8640 1 Processor -- start 2024-04-26T00:20:03.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb54b1b8640 1 -- start start 2024-04-26T00:20:03.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb54b1b8640 10 monclient: init 2024-04-26T00:20:03.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb54b1b8640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:03.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb54b1b8640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-04-26T00:20:03.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb54b1b8640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb5440a3d90 0x7fb54409bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb54b1b8640 10 monclient(hunting): picked mon.noname-a con 0x7fb5440a3d90 addr v2:172.21.15.37:3300/0 2024-04-26T00:20:03.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb54b1b8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5440a4170 0x7fb5440a45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb54b1b8640 10 monclient(hunting): picked mon.noname-b con 0x7fb5440a4170 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:03.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb54b1b8640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb544133b20 0x7fb544133f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb543fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5440a4170 0x7fb5440a45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb543fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5440a4170 0x7fb5440a45b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:57822/0 (socket says 172.21.15.37:57822) 2024-04-26T00:20:03.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb543fff640 1 -- 172.21.15.37:0/1987559422 learned_addr learned my addr 172.21.15.37:0/1987559422 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:03.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb548f54640 1 --2- 172.21.15.37:0/1987559422 >> v2:172.21.15.37:3300/0 conn(0x7fb5440a3d90 0x7fb54409bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb54b1b8640 10 monclient(hunting): picked mon.noname-c con 0x7fb544133b20 addr v2:172.21.15.45:3300/0 2024-04-26T00:20:03.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb54b1b8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb54b1b8640 1 -- 172.21.15.37:0/1987559422 --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fb5440a5150 con 0x7fb5440a3d90 2024-04-26T00:20:03.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb54b1b8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb54b1b8640 1 -- 172.21.15.37:0/1987559422 --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fb54409b280 con 0x7fb544133b20 2024-04-26T00:20:03.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb549755640 1 --2- 172.21.15.37:0/1987559422 >> v2:172.21.15.45:3300/0 conn(0x7fb544133b20 0x7fb544133f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb54b1b8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb54b1b8640 1 -- 172.21.15.37:0/1987559422 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb54409b560 con 0x7fb5440a4170 2024-04-26T00:20:03.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb54b1b8640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:03.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb54b1b8640 10 monclient(hunting): authenticate will time out at 985.075517s 2024-04-26T00:20:03.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb543fff640 10 monclient(hunting): get_auth_request con 0x7fb5440a4170 auth_method 0 2024-04-26T00:20:03.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb543fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb543fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb543fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb548f54640 10 monclient(hunting): get_auth_request con 0x7fb5440a3d90 auth_method 0 2024-04-26T00:20:03.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb548f54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb548f54640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb548f54640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb543fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb543fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb543fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb548f54640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb548f54640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb548f54640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb543fff640 10 monclient(hunting): handle_auth_done global_id 4226 payload 274 2024-04-26T00:20:03.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb543fff640 1 -- 172.21.15.37:0/1987559422 >> v2:172.21.15.45:3300/0 conn(0x7fb544133b20 msgr2=0x7fb544133f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:03.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb543fff640 1 --2- 172.21.15.37:0/1987559422 >> v2:172.21.15.45:3300/0 conn(0x7fb544133b20 0x7fb544133f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb543fff640 1 -- 172.21.15.37:0/1987559422 >> v2:172.21.15.37:3300/0 conn(0x7fb5440a3d90 msgr2=0x7fb54409bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:03.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb543fff640 1 --2- 172.21.15.37:0/1987559422 >> v2:172.21.15.37:3300/0 conn(0x7fb5440a3d90 0x7fb54409bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb543fff640 10 monclient: _finish_hunting 0 2024-04-26T00:20:03.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb543fff640 1 monclient: found mon.noname-b 2024-04-26T00:20:03.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb543fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:03.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb543fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:20:03.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.763+0000 7fb543fff640 1 -- 172.21.15.37:0/1987559422 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5440a4af0 con 0x7fb5440a4170 2024-04-26T00:20:03.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.767+0000 7fb549755640 10 monclient: get_auth_request con 0x7fb544133b20 auth_method 0 2024-04-26T00:20:03.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.767+0000 7fb549755640 1 --2- 172.21.15.37:0/1987559422 >> v2:172.21.15.45:3300/0 conn(0x7fb544133b20 0x7fb544133f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:20:03.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.767+0000 7f5dd8e9b640 1 -- 172.21.15.37:0/3955027388 --> v2:172.21.15.45:6808/3915746722 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5dd404f510 con 0x7f5db403b890 2024-04-26T00:20:03.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.767+0000 7f5dc2ffd640 1 -- 172.21.15.37:0/3955027388 <== osd.5 v2:172.21.15.45:6808/3915746722 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f5dd404f510 con 0x7f5db403b890 2024-04-26T00:20:03.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.767+0000 7f5dd8e9b640 1 -- 172.21.15.37:0/3955027388 >> v2:172.21.15.45:6808/3915746722 conn(0x7f5db403b890 msgr2=0x7f5db403dcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.767+0000 7f5dd8e9b640 1 --2- 172.21.15.37:0/3955027388 >> v2:172.21.15.45:6808/3915746722 conn(0x7f5db403b890 0x7f5db403dcb0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.767+0000 7f5dd8e9b640 1 -- 172.21.15.37:0/3955027388 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5db4035df0 msgr2=0x7f5db40382b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.767+0000 7f5dd8e9b640 1 --2- 172.21.15.37:0/3955027388 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5db4035df0 0x7f5db40382b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5dd4131930 tx=0x7f5dc8008040 comp rx=0 tx=0).stop 2024-04-26T00:20:03.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.767+0000 7f5dd8e9b640 10 monclient: shutdown 2024-04-26T00:20:03.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.767+0000 7f5dd8e9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:03.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.767+0000 7f5dd8e9b640 1 -- 172.21.15.37:0/3955027388 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5dd4073080 msgr2=0x7f5dd4143010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.767+0000 7f5dd8e9b640 1 --2- 172.21.15.37:0/3955027388 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5dd4073080 0x7f5dd4143010 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5dc40ac000 tx=0x7f5dc4019170 comp rx=0 tx=0).stop 2024-04-26T00:20:03.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.767+0000 7f5dd2d9d640 1 -- 172.21.15.37:0/3955027388 reap_dead start 2024-04-26T00:20:03.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.767+0000 7f5dd8e9b640 1 -- 172.21.15.37:0/3955027388 shutdown_connections 2024-04-26T00:20:03.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.767+0000 7f5dd8e9b640 1 -- 172.21.15.37:0/3955027388 >> 172.21.15.37:0/3955027388 conn(0x7f5dd40a7160 msgr2=0x7f5dd406d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:03.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.767+0000 7f5dd8e9b640 1 -- 172.21.15.37:0/3955027388 shutdown_connections 2024-04-26T00:20:03.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.767+0000 7f5dd8e9b640 1 -- 172.21.15.37:0/3955027388 wait complete. 2024-04-26T00:20:03.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.771+0000 7fb543fff640 1 -- 172.21.15.37:0/1987559422 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5440a4170 msgr2=0x7fb5440a45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:03.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.771+0000 7fb543fff640 1 --2- 172.21.15.37:0/1987559422 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5440a4170 0x7fb5440a45b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb544076100 tx=0x7fb53001c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8c620574bd5f11c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.771+0000 7fb5437fe640 1 -- 172.21.15.37:0/1987559422 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5300b0070 con 0x7fb5440a4170 2024-04-26T00:20:03.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.771+0000 7fb5437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.771+0000 7fb5437fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:20:03.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.771+0000 7fb5437fe640 10 monclient: dump: 2024-04-26T00:20:03.757 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.757 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.757 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.757 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.757 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.757 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.757 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.757 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.757 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.757 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.771+0000 7fb5437fe640 10 monclient: _finish_auth 0 2024-04-26T00:20:03.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.771+0000 7fb5437fe640 10 monclient: _check_auth_tickets 2024-04-26T00:20:03.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.771+0000 7fb5437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:03.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.771+0000 7fb5437fe640 1 -- 172.21.15.37:0/1987559422 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5300ab600 con 0x7fb5440a4170 2024-04-26T00:20:03.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.771+0000 7fb5437fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:03.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.771+0000 7fb5437fe640 1 -- 172.21.15.37:0/1987559422 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5300ab9d0 con 0x7fb5440a4170 2024-04-26T00:20:03.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.771+0000 7fb5437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.771+0000 7fb5437fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:03.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.771+0000 7fb5437fe640 10 monclient: dump: 2024-04-26T00:20:03.758 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.758 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.758 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.758 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.758 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.758 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.758 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.758 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.758 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.759 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.771+0000 7fb54b1b8640 5 monclient: authenticate success, global_id 4226 2024-04-26T00:20:03.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.771+0000 7fb54b1b8640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:03.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.771+0000 7fb54b1b8640 4 set_mon_vals no callback set 2024-04-26T00:20:03.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.771+0000 7fb54b1b8640 10 monclient: shutdown 2024-04-26T00:20:03.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.771+0000 7fb54b1b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:03.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.771+0000 7fb54b1b8640 1 -- 172.21.15.37:0/1987559422 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5440a4170 msgr2=0x7fb5440a45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.771+0000 7fb54b1b8640 1 --2- 172.21.15.37:0/1987559422 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5440a4170 0x7fb5440a45b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb544076100 tx=0x7fb53001c3e0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.771+0000 7fb54b1b8640 1 -- 172.21.15.37:0/1987559422 shutdown_connections 2024-04-26T00:20:03.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.771+0000 7fb54b1b8640 1 --2- 172.21.15.37:0/1987559422 >> v2:172.21.15.45:3300/0 conn(0x7fb544133b20 0x7fb544133f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.771+0000 7fb54b1b8640 1 --2- 172.21.15.37:0/1987559422 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5440a4170 0x7fb5440a45b0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.775+0000 7fb54b1b8640 1 --2- 172.21.15.37:0/1987559422 >> v2:172.21.15.37:3300/0 conn(0x7fb5440a3d90 0x7fb54409bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.775+0000 7fb54b1b8640 1 -- 172.21.15.37:0/1987559422 >> 172.21.15.37:0/1987559422 conn(0x7fb5440aa0a0 msgr2=0x7fb5440a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:03.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.775+0000 7fb54b1b8640 1 -- 172.21.15.37:0/1987559422 shutdown_connections 2024-04-26T00:20:03.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.775+0000 7fb54b1b8640 1 -- 172.21.15.37:0/1987559422 wait complete. 2024-04-26T00:20:03.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.775+0000 7fb54b1b8640 10 monclient: build_initial_monmap 2024-04-26T00:20:03.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.775+0000 7fb54b1b8640 10 monclient: monmap: 2024-04-26T00:20:03.763 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:03.763 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:03.763 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:03.763 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:03.763 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:03.763 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:03.763 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:03.763 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:03.763 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:03.763 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.775+0000 7fb54b1b8640 1 Processor -- start 2024-04-26T00:20:03.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.775+0000 7fb54b1b8640 1 -- start start 2024-04-26T00:20:03.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.775+0000 7fb54b1b8640 10 monclient: init 2024-04-26T00:20:03.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.775+0000 7fb54b1b8640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:03.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.775+0000 7fb54b1b8640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:20:03.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.775+0000 7fb54b1b8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5440a3d90 0x7fb5441cf040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb54b1b8640 10 monclient(hunting): picked mon.noname-b con 0x7fb5440a3d90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:03.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb54b1b8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5440a4170 0x7fb5441cf580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb54b1b8640 10 monclient(hunting): picked mon.noname-c con 0x7fb5440a4170 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:03.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb54b1b8640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb544133b20 0x7fb5441cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb548f54640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5440a3d90 0x7fb5441cf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb548f54640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5440a3d90 0x7fb5441cf040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.37:55656/0 (socket says 172.21.15.37:55656) 2024-04-26T00:20:03.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb543fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5440a4170 0x7fb5441cf580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb543fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5440a4170 0x7fb5441cf580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:57836/0 (socket says 172.21.15.37:57836) 2024-04-26T00:20:03.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb549755640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb544133b20 0x7fb5441cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb549755640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb544133b20 0x7fb5441cfac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44612/0 (socket says 172.21.15.37:44612) 2024-04-26T00:20:03.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb54b1b8640 10 monclient(hunting): picked mon.noname-a con 0x7fb544133b20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:03.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb548f54640 1 -- 172.21.15.37:0/4146518997 learned_addr learned my addr 172.21.15.37:0/4146518997 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:03.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb54b1b8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb54b1b8640 1 -- 172.21.15.37:0/4146518997 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb54409b8d0 con 0x7fb544133b20 2024-04-26T00:20:03.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb54b1b8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb54b1b8640 1 -- 172.21.15.37:0/4146518997 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb54409c690 con 0x7fb5440a3d90 2024-04-26T00:20:03.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb54b1b8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb54b1b8640 1 -- 172.21.15.37:0/4146518997 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb544140c10 con 0x7fb5440a4170 2024-04-26T00:20:03.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb54b1b8640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:03.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb54b1b8640 10 monclient(hunting): authenticate will time out at 985.089463s 2024-04-26T00:20:03.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb548f54640 10 monclient(hunting): get_auth_request con 0x7fb5440a3d90 auth_method 0 2024-04-26T00:20:03.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb548f54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb548f54640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb548f54640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb543fff640 10 monclient(hunting): get_auth_request con 0x7fb5440a4170 auth_method 0 2024-04-26T00:20:03.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb543fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb543fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb543fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb549755640 10 monclient(hunting): get_auth_request con 0x7fb544133b20 auth_method 0 2024-04-26T00:20:03.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb549755640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb549755640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb549755640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb543fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb543fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb543fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb549755640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb549755640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb549755640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb548f54640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb548f54640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb548f54640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb543fff640 10 monclient(hunting): handle_auth_done global_id 4229 payload 824 2024-04-26T00:20:03.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb543fff640 1 -- 172.21.15.37:0/4146518997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5440a3d90 msgr2=0x7fb5441cf040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb543fff640 1 --2- 172.21.15.37:0/4146518997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5440a3d90 0x7fb5441cf040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb543fff640 1 -- 172.21.15.37:0/4146518997 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb544133b20 msgr2=0x7fb5441cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb543fff640 1 --2- 172.21.15.37:0/4146518997 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb544133b20 0x7fb5441cfac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb543fff640 10 monclient: _finish_hunting 0 2024-04-26T00:20:03.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb543fff640 1 monclient: found mon.noname-c 2024-04-26T00:20:03.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb543fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:03.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb543fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:20:03.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb543fff640 1 -- 172.21.15.37:0/4146518997 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb54409ada0 con 0x7fb5440a4170 2024-04-26T00:20:03.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.779+0000 7fb549755640 1 --2- 172.21.15.37:0/4146518997 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb544133b20 0x7fb5441cfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:03.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.783+0000 7fb543fff640 1 -- 172.21.15.37:0/4146518997 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5440a4170 msgr2=0x7fb5441cf580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:03.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.783+0000 7fb543fff640 1 --2- 172.21.15.37:0/4146518997 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5440a4170 0x7fb5441cf580 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb5300ab000 tx=0x7fb530019180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.783+0000 7fb5417fa640 1 -- 172.21.15.37:0/4146518997 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5300b0040 con 0x7fb5440a4170 2024-04-26T00:20:03.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.783+0000 7fb5417fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.783+0000 7fb5417fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:20:03.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.783+0000 7fb5417fa640 10 monclient: dump: 2024-04-26T00:20:03.772 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.772 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.772 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.772 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.772 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.772 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.772 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.772 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.772 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.773 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.783+0000 7fb5417fa640 10 monclient: _finish_auth 0 2024-04-26T00:20:03.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.783+0000 7fb5417fa640 10 monclient: _check_auth_tickets 2024-04-26T00:20:03.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.783+0000 7fb5417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:03.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.783+0000 7fb5417fa640 1 -- 172.21.15.37:0/4146518997 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5300aa5b0 con 0x7fb5440a4170 2024-04-26T00:20:03.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.783+0000 7fb5417fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:03.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.783+0000 7fb5417fa640 1 -- 172.21.15.37:0/4146518997 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5300aa9a0 con 0x7fb5440a4170 2024-04-26T00:20:03.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.783+0000 7fb542ffd640 4 set_mon_vals no callback set 2024-04-26T00:20:03.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.783+0000 7fb5417fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.783+0000 7fb5417fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:03.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.783+0000 7fb5417fa640 10 monclient: dump: 2024-04-26T00:20:03.773 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.773 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.773 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.774 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.774 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.774 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.774 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.774 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.774 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.774 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.783+0000 7fb54b1b8640 5 monclient: authenticate success, global_id 4229 2024-04-26T00:20:03.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.783+0000 7fb54b1b8640 10 monclient: _renew_subs 2024-04-26T00:20:03.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.783+0000 7fb54b1b8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:03.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.783+0000 7fb54b1b8640 1 -- 172.21.15.37:0/4146518997 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5440a0890 con 0x7fb5440a4170 2024-04-26T00:20:03.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.783+0000 7fb54b1b8640 10 monclient: _renew_subs 2024-04-26T00:20:03.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.783+0000 7fb54b1b8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:03.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.783+0000 7fb54b1b8640 1 -- 172.21.15.37:0/4146518997 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb54413cd40 con 0x7fb5440a4170 2024-04-26T00:20:03.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.783+0000 7fb543fff640 1 --2- 172.21.15.37:0/4146518997 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5440a4170 0x7fb5441cf580 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb5300ab000 tx=0x7fb530019180 comp rx=0 tx=0).queue_received will delay after 1.71409e+09 on 0x7fb5300e1df0 osd_map(15..15 src has 1..15) 2024-04-26T00:20:03.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.783+0000 7fb5417fa640 1 -- 172.21.15.37:0/4146518997 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 4) ==== 43962+0+0 (secure 0 0 0) 0x7fb5300aab40 con 0x7fb5440a4170 2024-04-26T00:20:03.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.783+0000 7fb54b1b8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:03.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.783+0000 7fb54b1b8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:03.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.783+0000 7fb54b1b8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:03.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.783+0000 7fb54b1b8640 1 -- 172.21.15.37:0/4146518997 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb54404f510 con 0x7fb5440a4170 2024-04-26T00:20:03.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.787+0000 7fb5417fa640 1 --2- 172.21.15.37:0/4146518997 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb528035df0 0x7fb5280382b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.787+0000 7fb548f54640 1 --2- 172.21.15.37:0/4146518997 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb528035df0 0x7fb5280382b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.787+0000 7fb548f54640 10 monclient: get_auth_request con 0x7fb528035df0 auth_method 0 2024-04-26T00:20:03.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.791+0000 7fb548f54640 1 --2- 172.21.15.37:0/4146518997 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb528035df0 0x7fb5280382b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb54409c940 tx=0x7fb538009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.795+0000 7fb5417fa640 1 -- 172.21.15.37:0/4146518997 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7fb5300e1df0 con 0x7fb5440a4170 2024-04-26T00:20:03.778 INFO:teuthology.orchestra.run.smithi037.stdout:47244640258 2024-04-26T00:20:03.778 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-26T00:20:03.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa52147b640 10 monclient: get_monmap_and_config 2024-04-26T00:20:03.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa52147b640 10 monclient: build_initial_monmap 2024-04-26T00:20:03.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa52147b640 10 monclient: monmap: 2024-04-26T00:20:03.782 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:03.782 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.782 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:03.805676+0000 2024-04-26T00:20:03.782 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:03.805676+0000 2024-04-26T00:20:03.782 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:03.782 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.783 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:03.783 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:03.783 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:03.783 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:03.783 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:03.783 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa52147b640 1 Processor -- start 2024-04-26T00:20:03.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa52147b640 1 -- start start 2024-04-26T00:20:03.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa52147b640 10 monclient: init 2024-04-26T00:20:03.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa52147b640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:03.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa52147b640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-04-26T00:20:03.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa52147b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa51c131cd0 addr v1:172.21.15.37:6789/0 2024-04-26T00:20:03.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa52147b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa51c073080 0x7fa51c139f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa52147b640 10 monclient(hunting): picked mon.noname-b con 0x7fa51c073080 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:03.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa52147b640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa51c0a1b60 0x7fa51c13c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa52147b640 10 monclient(hunting): picked mon.noname-a con 0x7fa51c0a1b60 addr v2:172.21.15.37:3300/0 2024-04-26T00:20:03.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa52147b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa51c071660 con 0x7fa51c131cd0 2024-04-26T00:20:03.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa52147b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa52147b640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fa51c071940 con 0x7fa51c0a1b60 2024-04-26T00:20:03.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa52147b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa52147b640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa51c071c20 con 0x7fa51c073080 2024-04-26T00:20:03.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa52147b640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:03.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa52147b640 10 monclient(hunting): authenticate will time out at 985.111019s 2024-04-26T00:20:03.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa51a7fc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa51c073080 0x7fa51c139f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa51a7fc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa51c073080 0x7fa51c139f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:57852/0 (socket says 172.21.15.37:57852) 2024-04-26T00:20:03.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa51a7fc640 1 -- 172.21.15.37:0/1094436614 learned_addr learned my addr 172.21.15.37:0/1094436614 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:03.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa51a7fc640 10 monclient(hunting): get_auth_request con 0x7fa51c073080 auth_method 0 2024-04-26T00:20:03.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa51a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa51a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa51a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa51b7fe640 1 --2- 172.21.15.37:0/1094436614 >> v2:172.21.15.37:3300/0 conn(0x7fa51c0a1b60 0x7fa51c13c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa51b7fe640 10 monclient(hunting): get_auth_request con 0x7fa51c0a1b60 auth_method 0 2024-04-26T00:20:03.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa51b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa51b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa51b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa51a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa51a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa51a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa51b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa51b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa51b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa51a7fc640 10 monclient(hunting): handle_auth_done global_id 4232 payload 274 2024-04-26T00:20:03.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa51a7fc640 1 -- 172.21.15.37:0/1094436614 >> v2:172.21.15.37:3300/0 conn(0x7fa51c0a1b60 msgr2=0x7fa51c13c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:03.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa51a7fc640 1 --2- 172.21.15.37:0/1094436614 >> v2:172.21.15.37:3300/0 conn(0x7fa51c0a1b60 0x7fa51c13c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa51a7fc640 1 -- 172.21.15.37:0/1094436614 >> v1:172.21.15.37:6789/0 conn(0x7fa51c131cd0 legacy=0x7fa51c1320b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:20:03.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa51a7fc640 10 monclient: _finish_hunting 0 2024-04-26T00:20:03.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa51a7fc640 1 monclient: found mon.noname-b 2024-04-26T00:20:03.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa51a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:03.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa51a7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:20:03.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.799+0000 7fa51a7fc640 1 -- 172.21.15.37:0/1094436614 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa51c072680 con 0x7fa51c073080 2024-04-26T00:20:03.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.803+0000 7f25e6d6e640 10 monclient: get_monmap_and_config 2024-04-26T00:20:03.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.803+0000 7f25e6d6e640 10 monclient: build_initial_monmap 2024-04-26T00:20:03.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.803+0000 7f25e6d6e640 10 monclient: monmap: 2024-04-26T00:20:03.790 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:03.790 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.790 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:03.811198+0000 2024-04-26T00:20:03.790 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:03.811198+0000 2024-04-26T00:20:03.790 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:03.790 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.790 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:03.790 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:03.790 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:03.790 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:03.790 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:03.790 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.803+0000 7f25e6d6e640 1 Processor -- start 2024-04-26T00:20:03.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7f25e6d6e640 1 -- start start 2024-04-26T00:20:03.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7f25e6d6e640 10 monclient: init 2024-04-26T00:20:03.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7f25e6d6e640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:03.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7f25e6d6e640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-04-26T00:20:03.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7f25e6d6e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f25e00a3d90 0x7f25e009bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7f25e6d6e640 10 monclient(hunting): picked mon.noname-c con 0x7f25e00a3d90 addr v2:172.21.15.45:3300/0 2024-04-26T00:20:03.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7f25e6d6e640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f25e00a4170 0x7f25e00a45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7f25dffff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f25e00a4170 0x7f25e00a45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7f25dffff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f25e00a4170 0x7f25e00a45e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44638/0 (socket says 172.21.15.37:44638) 2024-04-26T00:20:03.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7f25dffff640 1 -- 172.21.15.37:0/2960953169 learned_addr learned my addr 172.21.15.37:0/2960953169 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:03.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7f25e6d6e640 10 monclient(hunting): picked mon.noname-a con 0x7f25e00a4170 addr v2:172.21.15.37:3300/0 2024-04-26T00:20:03.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7f25e4b0a640 1 --2- 172.21.15.37:0/2960953169 >> v2:172.21.15.45:3300/0 conn(0x7f25e00a3d90 0x7f25e009bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7fa51a7fc640 1 -- 172.21.15.37:0/1094436614 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa51c073080 msgr2=0x7fa51c139f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:03.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7fa51a7fc640 1 --2- 172.21.15.37:0/1094436614 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa51c073080 0x7fa51c139f30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa510009870 tx=0x7fa51001c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3d5890360d97d3b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7f25e6d6e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f25e0133b20 addr v1:172.21.15.45:6789/0 2024-04-26T00:20:03.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7f25e6d6e640 1 -- 172.21.15.37:0/2960953169 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25e00a5150 con 0x7f25e0133b20 2024-04-26T00:20:03.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7f25e6d6e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7f25e6d6e640 1 -- 172.21.15.37:0/2960953169 --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f25e009b280 con 0x7f25e00a4170 2024-04-26T00:20:03.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7f25e6d6e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7f25e6d6e640 1 -- 172.21.15.37:0/2960953169 --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f25e009b560 con 0x7f25e00a3d90 2024-04-26T00:20:03.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7f25e6d6e640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:03.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7f25e6d6e640 10 monclient(hunting): authenticate will time out at 985.117430s 2024-04-26T00:20:03.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7fa519ffb640 1 -- 172.21.15.37:0/1094436614 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5100b0070 con 0x7fa51c073080 2024-04-26T00:20:03.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7fa519ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7fa519ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:20:03.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7fa519ffb640 10 monclient: dump: 2024-04-26T00:20:03.794 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.794 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.794 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.794 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.794 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.794 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.794 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.794 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.794 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.794 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7fa519ffb640 10 monclient: _finish_auth 0 2024-04-26T00:20:03.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7fa519ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:20:03.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7fa519ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:03.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7fa52147b640 5 monclient: authenticate success, global_id 4232 2024-04-26T00:20:03.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7fa52147b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:20:03.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7fa519ffb640 1 -- 172.21.15.37:0/1094436614 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5100a9710 con 0x7fa51c073080 2024-04-26T00:20:03.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7fa519ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:03.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7fa519ffb640 1 -- 172.21.15.37:0/1094436614 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5100a9ae0 con 0x7fa51c073080 2024-04-26T00:20:03.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7fa519ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7fa519ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:03.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7fa519ffb640 10 monclient: dump: 2024-04-26T00:20:03.795 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.795 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.795 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.795 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.796 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.796 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.796 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.796 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.796 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.796 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7fa52147b640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:03.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7fa52147b640 4 set_mon_vals no callback set 2024-04-26T00:20:03.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7f25dffff640 10 monclient(hunting): get_auth_request con 0x7f25e00a4170 auth_method 0 2024-04-26T00:20:03.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7f25dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7f25dffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7f25dffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7f25dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7f25dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7f25dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7f25dffff640 10 monclient(hunting): handle_auth_done global_id 4251 payload 274 2024-04-26T00:20:03.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7f25dffff640 1 -- 172.21.15.37:0/2960953169 >> v2:172.21.15.45:3300/0 conn(0x7f25e00a3d90 msgr2=0x7f25e009bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:03.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7f25dffff640 1 --2- 172.21.15.37:0/2960953169 >> v2:172.21.15.45:3300/0 conn(0x7f25e00a3d90 0x7f25e009bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7f25dffff640 1 -- 172.21.15.37:0/2960953169 >> v1:172.21.15.45:6789/0 conn(0x7f25e0133b20 legacy=0x7f25e0133f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:03.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7f25dffff640 10 monclient: _finish_hunting 0 2024-04-26T00:20:03.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7f25dffff640 1 monclient: found mon.noname-a 2024-04-26T00:20:03.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7f25dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:03.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7f25dffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:03.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7f25dffff640 1 -- 172.21.15.37:0/2960953169 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25e00a4b20 con 0x7f25e00a4170 2024-04-26T00:20:03.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7f25e4b0a640 10 monclient: get_auth_request con 0x7f25e00a3d90 auth_method 0 2024-04-26T00:20:03.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7f25e4b0a640 1 --2- 172.21.15.37:0/2960953169 >> v2:172.21.15.45:3300/0 conn(0x7f25e00a3d90 0x7f25e009bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:20:03.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7fa52147b640 10 monclient: shutdown 2024-04-26T00:20:03.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7fa52147b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:03.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7fa52147b640 1 -- 172.21.15.37:0/1094436614 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa51c073080 msgr2=0x7fa51c139f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7fa52147b640 1 --2- 172.21.15.37:0/1094436614 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa51c073080 0x7fa51c139f30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa510009870 tx=0x7fa51001c470 comp rx=0 tx=0).stop 2024-04-26T00:20:03.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7fa52147b640 1 -- 172.21.15.37:0/1094436614 shutdown_connections 2024-04-26T00:20:03.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7fa52147b640 1 --2- 172.21.15.37:0/1094436614 >> v2:172.21.15.37:3300/0 conn(0x7fa51c0a1b60 0x7fa51c13c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7fa52147b640 1 --2- 172.21.15.37:0/1094436614 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa51c073080 0x7fa51c139f30 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7fa52147b640 1 -- 172.21.15.37:0/1094436614 >> 172.21.15.37:0/1094436614 conn(0x7fa51c0a7160 msgr2=0x7fa51c0a7540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:03.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7fa52147b640 1 -- 172.21.15.37:0/1094436614 shutdown_connections 2024-04-26T00:20:03.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7fa52147b640 1 -- 172.21.15.37:0/1094436614 wait complete. 2024-04-26T00:20:03.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7fa52147b640 10 monclient: build_initial_monmap 2024-04-26T00:20:03.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7fa52147b640 10 monclient: monmap: 2024-04-26T00:20:03.800 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:03.800 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:03.800 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:03.800 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:03.800 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:03.800 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:03.801 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:03.801 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:03.801 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:03.801 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7fa52147b640 1 Processor -- start 2024-04-26T00:20:03.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7fa52147b640 1 -- start start 2024-04-26T00:20:03.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.807+0000 7fa52147b640 10 monclient: init 2024-04-26T00:20:03.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa52147b640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:03.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa52147b640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:20:03.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa52147b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa51c073080 0x7fa51c13a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa52147b640 10 monclient(hunting): picked mon.noname-b con 0x7fa51c073080 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:03.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa52147b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa51c0a1b60 0x7fa51c13aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa52147b640 10 monclient(hunting): picked mon.noname-a con 0x7fa51c0a1b60 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:03.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa52147b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa51c131cd0 0x7fa51c13af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa52147b640 10 monclient(hunting): picked mon.noname-c con 0x7fa51c131cd0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:03.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa52147b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa52147b640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa51c13f570 con 0x7fa51c0a1b60 2024-04-26T00:20:03.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa52147b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa52147b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa51c13f3f0 con 0x7fa51c073080 2024-04-26T00:20:03.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa52147b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa52147b640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa51c13f6f0 con 0x7fa51c131cd0 2024-04-26T00:20:03.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa52147b640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:03.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa52147b640 10 monclient(hunting): authenticate will time out at 985.120129s 2024-04-26T00:20:03.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa51a7fc640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa51c0a1b60 0x7fa51c13aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa51a7fc640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa51c0a1b60 0x7fa51c13aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44652/0 (socket says 172.21.15.37:44652) 2024-04-26T00:20:03.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa51a7fc640 1 -- 172.21.15.37:0/3934835523 learned_addr learned my addr 172.21.15.37:0/3934835523 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:03.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa51a7fc640 10 monclient(hunting): get_auth_request con 0x7fa51c0a1b60 auth_method 0 2024-04-26T00:20:03.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa51a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa51a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa51a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa51b7fe640 1 --2- 172.21.15.37:0/3934835523 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa51c131cd0 0x7fa51c13af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa51b7fe640 10 monclient(hunting): get_auth_request con 0x7fa51c131cd0 auth_method 0 2024-04-26T00:20:03.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa51b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa51b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa51b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa51affd640 1 --2- 172.21.15.37:0/3934835523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa51c073080 0x7fa51c13a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa51a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa51a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa51a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa51b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa51b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa51b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa51affd640 10 monclient(hunting): get_auth_request con 0x7fa51c073080 auth_method 0 2024-04-26T00:20:03.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa51affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa51affd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa51affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa51affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa51affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa51affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa51a7fc640 10 monclient(hunting): handle_auth_done global_id 4254 payload 824 2024-04-26T00:20:03.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa51a7fc640 1 -- 172.21.15.37:0/3934835523 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa51c131cd0 msgr2=0x7fa51c13af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa51a7fc640 1 --2- 172.21.15.37:0/3934835523 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa51c131cd0 0x7fa51c13af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa51a7fc640 1 -- 172.21.15.37:0/3934835523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa51c073080 msgr2=0x7fa51c13a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa51a7fc640 1 --2- 172.21.15.37:0/3934835523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa51c073080 0x7fa51c13a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa51a7fc640 10 monclient: _finish_hunting 0 2024-04-26T00:20:03.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa51a7fc640 1 monclient: found mon.noname-a 2024-04-26T00:20:03.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa51a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:03.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa51a7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:03.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa51a7fc640 1 -- 172.21.15.37:0/3934835523 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa51c06c580 con 0x7fa51c0a1b60 2024-04-26T00:20:03.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa51b7fe640 1 --2- 172.21.15.37:0/3934835523 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa51c131cd0 0x7fa51c13af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:03.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7f25dffff640 1 -- 172.21.15.37:0/2960953169 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25e00a4170 msgr2=0x7f25e00a45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:03.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7f25dffff640 1 --2- 172.21.15.37:0/2960953169 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25e00a4170 0x7f25e00a45e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f25e00a57e0 tx=0x7f25cc01c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=713fb510561384ae server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7f25df7fe640 1 -- 172.21.15.37:0/2960953169 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25cc0b0070 con 0x7f25e00a4170 2024-04-26T00:20:03.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7f25df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7f25df7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:03.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7f25df7fe640 10 monclient: dump: 2024-04-26T00:20:03.806 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.806 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.806 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.806 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.806 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.807 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.807 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.807 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.807 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.807 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7f25df7fe640 10 monclient: _finish_auth 0 2024-04-26T00:20:03.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7f25df7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:20:03.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7f25df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:03.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7f25e6d6e640 5 monclient: authenticate success, global_id 4251 2024-04-26T00:20:03.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7f25e6d6e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:20:03.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7f25df7fe640 1 -- 172.21.15.37:0/2960953169 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25cc0ab600 con 0x7f25e00a4170 2024-04-26T00:20:03.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7f25df7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:03.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa51a7fc640 1 -- 172.21.15.37:0/3934835523 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa51c0a1b60 msgr2=0x7fa51c13aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:03.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa51a7fc640 1 --2- 172.21.15.37:0/3934835523 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa51c0a1b60 0x7fa51c13aa40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa51001cd10 tx=0x7fa510019210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7f25e6d6e640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:03.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7f25e6d6e640 4 set_mon_vals no callback set 2024-04-26T00:20:03.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7f25e6d6e640 10 monclient: shutdown 2024-04-26T00:20:03.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7f25e6d6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:03.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7f25e6d6e640 1 -- 172.21.15.37:0/2960953169 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25e00a4170 msgr2=0x7f25e00a45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7f25e6d6e640 1 --2- 172.21.15.37:0/2960953169 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25e00a4170 0x7f25e00a45e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f25e00a57e0 tx=0x7f25cc01c3a0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e6d6e640 1 -- 172.21.15.37:0/2960953169 shutdown_connections 2024-04-26T00:20:03.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e6d6e640 1 --2- 172.21.15.37:0/2960953169 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25e00a4170 0x7f25e00a45e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e6d6e640 1 --2- 172.21.15.37:0/2960953169 >> v2:172.21.15.45:3300/0 conn(0x7f25e00a3d90 0x7f25e009bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e6d6e640 1 -- 172.21.15.37:0/2960953169 >> 172.21.15.37:0/2960953169 conn(0x7f25e00aa0a0 msgr2=0x7f25e00a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:03.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e6d6e640 1 -- 172.21.15.37:0/2960953169 shutdown_connections 2024-04-26T00:20:03.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa5037fe640 1 -- 172.21.15.37:0/3934835523 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5100b0050 con 0x7fa51c0a1b60 2024-04-26T00:20:03.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa5037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa5037fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:03.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa5037fe640 10 monclient: dump: 2024-04-26T00:20:03.810 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.810 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.810 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.810 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.810 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.810 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.810 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.810 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.810 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.810 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa5037fe640 10 monclient: _finish_auth 0 2024-04-26T00:20:03.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa5037fe640 10 monclient: _check_auth_tickets 2024-04-26T00:20:03.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa5037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:03.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa5037fe640 1 -- 172.21.15.37:0/3934835523 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5100ba070 con 0x7fa51c0a1b60 2024-04-26T00:20:03.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa5037fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:03.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa5037fe640 1 -- 172.21.15.37:0/3934835523 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5100abb50 con 0x7fa51c0a1b60 2024-04-26T00:20:03.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa5037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa5037fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:03.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa5037fe640 10 monclient: dump: 2024-04-26T00:20:03.811 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.811 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.811 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.811 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.811 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.811 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.811 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.811 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.811 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.812 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa52147b640 5 monclient: authenticate success, global_id 4254 2024-04-26T00:20:03.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa52147b640 10 monclient: _renew_subs 2024-04-26T00:20:03.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa52147b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:03.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa52147b640 1 -- 172.21.15.37:0/3934835523 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa51c11db30 con 0x7fa51c0a1b60 2024-04-26T00:20:03.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa5197fa640 4 set_mon_vals no callback set 2024-04-26T00:20:03.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa52147b640 10 monclient: _renew_subs 2024-04-26T00:20:03.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa52147b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:03.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.811+0000 7fa52147b640 1 -- 172.21.15.37:0/3934835523 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa51c0708e0 con 0x7fa51c0a1b60 2024-04-26T00:20:03.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e6d6e640 1 -- 172.21.15.37:0/2960953169 wait complete. 2024-04-26T00:20:03.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7fa52147b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:03.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7fa52147b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:03.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7fa52147b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:03.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7fa52147b640 1 -- 172.21.15.37:0/3934835523 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa51c04f510 con 0x7fa51c0a1b60 2024-04-26T00:20:03.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e6d6e640 10 monclient: build_initial_monmap 2024-04-26T00:20:03.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e6d6e640 10 monclient: monmap: 2024-04-26T00:20:03.814 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:03.814 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:03.814 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:03.814 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:03.814 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:03.814 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:03.814 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:03.814 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:03.814 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:03.814 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e6d6e640 1 Processor -- start 2024-04-26T00:20:03.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7fa5037fe640 1 -- 172.21.15.37:0/3934835523 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 4) ==== 43962+0+0 (secure 0 0 0) 0x7fa5100abcf0 con 0x7fa51c0a1b60 2024-04-26T00:20:03.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7fa5037fe640 1 --2- 172.21.15.37:0/3934835523 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa508035e40 0x7fa508038300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7fa5037fe640 1 -- 172.21.15.37:0/3934835523 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7fa5100e28c0 con 0x7fa51c0a1b60 2024-04-26T00:20:03.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e3acc6640 10 monclient: get_monmap_and_config 2024-04-26T00:20:03.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e3acc6640 10 monclient: build_initial_monmap 2024-04-26T00:20:03.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e3acc6640 10 monclient: monmap: 2024-04-26T00:20:03.816 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:03.817 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.817 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:03.821564+0000 2024-04-26T00:20:03.817 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:03.821564+0000 2024-04-26T00:20:03.817 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:03.817 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.817 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:03.818 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:03.818 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:03.818 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:03.818 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:03.818 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e6d6e640 1 -- start start 2024-04-26T00:20:03.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e6d6e640 10 monclient: init 2024-04-26T00:20:03.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e6d6e640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:03.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e6d6e640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:20:03.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e6d6e640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25e00a3d90 0x7f25e006e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e6d6e640 10 monclient(hunting): picked mon.noname-a con 0x7f25e00a3d90 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:03.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e6d6e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f25e00a4170 0x7f25e006ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e6d6e640 10 monclient(hunting): picked mon.noname-c con 0x7f25e00a4170 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:03.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e6d6e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25e0133b20 0x7f25e006ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7fa51affd640 1 --2- 172.21.15.37:0/3934835523 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa508035e40 0x7fa508038300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7fa51affd640 10 monclient: get_auth_request con 0x7fa508035e40 auth_method 0 2024-04-26T00:20:03.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e6d6e640 10 monclient(hunting): picked mon.noname-b con 0x7f25e0133b20 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:03.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e6d6e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e6d6e640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f25e0141e30 con 0x7f25e00a3d90 2024-04-26T00:20:03.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e6d6e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e6d6e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f25e009c690 con 0x7f25e0133b20 2024-04-26T00:20:03.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e6d6e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e6d6e640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f25e0141fb0 con 0x7f25e00a4170 2024-04-26T00:20:03.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e6d6e640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:03.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e6d6e640 10 monclient(hunting): authenticate will time out at 985.126346s 2024-04-26T00:20:03.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e3acc6640 1 Processor -- start 2024-04-26T00:20:03.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e3acc6640 1 -- start start 2024-04-26T00:20:03.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e3acc6640 10 monclient: init 2024-04-26T00:20:03.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e4b0a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25e00a3d90 0x7f25e006e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25dffff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f25e00a4170 0x7f25e006ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e530b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25e0133b20 0x7f25e006ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e4b0a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25e00a3d90 0x7f25e006e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44666/0 (socket says 172.21.15.37:44666) 2024-04-26T00:20:03.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25dffff640 10 monclient(hunting): get_auth_request con 0x7f25e00a4170 auth_method 0 2024-04-26T00:20:03.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e4b0a640 1 -- 172.21.15.37:0/2527512055 learned_addr learned my addr 172.21.15.37:0/2527512055 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:03.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25dffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25dffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e4b0a640 10 monclient(hunting): get_auth_request con 0x7f25e00a3d90 auth_method 0 2024-04-26T00:20:03.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e4b0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e4b0a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e4b0a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e530b640 10 monclient(hunting): get_auth_request con 0x7f25e0133b20 auth_method 0 2024-04-26T00:20:03.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e530b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e3acc6640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:03.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e3acc6640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-04-26T00:20:03.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e530b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e530b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e4b0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e4b0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e4b0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e530b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e530b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e530b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e3acc6640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e2c0b8660 0x7f5e2c01e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25dffff640 10 monclient(hunting): handle_auth_done global_id 4238 payload 824 2024-04-26T00:20:03.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e3acc6640 10 monclient(hunting): picked mon.noname-b con 0x7f5e2c0b8660 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:03.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25dffff640 1 -- 172.21.15.37:0/2527512055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25e0133b20 msgr2=0x7f25e006ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25dffff640 1 --2- 172.21.15.37:0/2527512055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25e0133b20 0x7f25e006ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25dffff640 1 -- 172.21.15.37:0/2527512055 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25e00a3d90 msgr2=0x7f25e006e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e3acc6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e2c01eaf0 addr v1:172.21.15.45:6789/0 2024-04-26T00:20:03.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25dffff640 1 --2- 172.21.15.37:0/2527512055 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25e00a3d90 0x7f25e006e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25dffff640 10 monclient: _finish_hunting 0 2024-04-26T00:20:03.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25dffff640 1 monclient: found mon.noname-c 2024-04-26T00:20:03.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:03.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25dffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:20:03.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25dffff640 1 -- 172.21.15.37:0/2527512055 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25e009ada0 con 0x7f25e00a4170 2024-04-26T00:20:03.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e3acc6640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f5e2c022040 0x7f5e2c0bf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f25e4b0a640 1 --2- 172.21.15.37:0/2527512055 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25e00a3d90 0x7f25e006e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:03.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e3acc6640 10 monclient(hunting): picked mon.noname-a con 0x7f5e2c022040 addr v2:172.21.15.37:3300/0 2024-04-26T00:20:03.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e3acc6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e2c01dbd0 con 0x7f5e2c01eaf0 2024-04-26T00:20:03.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e3acc6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e3acc6640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f5e2c022440 con 0x7f5e2c022040 2024-04-26T00:20:03.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e3acc6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e39cc4640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e2c0b8660 0x7f5e2c01e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e3acc6640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5e2c01e190 con 0x7f5e2c0b8660 2024-04-26T00:20:03.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e3acc6640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:03.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e3acc6640 10 monclient(hunting): authenticate will time out at 985.127513s 2024-04-26T00:20:03.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e39cc4640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e2c0b8660 0x7f5e2c01e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:57884/0 (socket says 172.21.15.37:57884) 2024-04-26T00:20:03.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e39cc4640 1 -- 172.21.15.37:0/448459810 learned_addr learned my addr 172.21.15.37:0/448459810 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:03.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e39cc4640 10 monclient(hunting): get_auth_request con 0x7f5e2c0b8660 auth_method 0 2024-04-26T00:20:03.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e39cc4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e39cc4640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e39cc4640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e3a4c5640 1 --2- 172.21.15.37:0/448459810 >> v2:172.21.15.37:3300/0 conn(0x7f5e2c022040 0x7f5e2c0bf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e3a4c5640 10 monclient(hunting): get_auth_request con 0x7f5e2c022040 auth_method 0 2024-04-26T00:20:03.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e3a4c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e3a4c5640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e3a4c5640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e39cc4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e39cc4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e39cc4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e3a4c5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e3a4c5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e3a4c5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e39cc4640 10 monclient(hunting): handle_auth_done global_id 4241 payload 274 2024-04-26T00:20:03.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e39cc4640 1 -- 172.21.15.37:0/448459810 >> v2:172.21.15.37:3300/0 conn(0x7f5e2c022040 msgr2=0x7f5e2c0bf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:03.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e39cc4640 1 --2- 172.21.15.37:0/448459810 >> v2:172.21.15.37:3300/0 conn(0x7f5e2c022040 0x7f5e2c0bf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e39cc4640 1 -- 172.21.15.37:0/448459810 >> v1:172.21.15.45:6789/0 conn(0x7f5e2c01eaf0 legacy=0x7f5e2c0b8a40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:20:03.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e39cc4640 10 monclient: _finish_hunting 0 2024-04-26T00:20:03.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e39cc4640 1 monclient: found mon.noname-b 2024-04-26T00:20:03.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e39cc4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:03.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e39cc4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:20:03.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.815+0000 7f5e39cc4640 1 -- 172.21.15.37:0/448459810 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e2c023690 con 0x7f5e2c0b8660 2024-04-26T00:20:03.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.819+0000 7fa51affd640 1 --2- 172.21.15.37:0/3934835523 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa508035e40 0x7fa508038300 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa50c007fd0 tx=0x7fa50c007e40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.819+0000 7f25dffff640 1 -- 172.21.15.37:0/2527512055 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f25e00a4170 msgr2=0x7f25e006ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:03.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.819+0000 7f25dffff640 1 --2- 172.21.15.37:0/2527512055 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f25e00a4170 0x7f25e006ea30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f25e006e0e0 tx=0x7f25cc019140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.819+0000 7f25dd7fa640 1 -- 172.21.15.37:0/2527512055 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25cc0b0050 con 0x7f25e00a4170 2024-04-26T00:20:03.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.819+0000 7f25dd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.819+0000 7f25dd7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:20:03.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.819+0000 7f25dd7fa640 10 monclient: dump: 2024-04-26T00:20:03.835 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.835 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.835 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.835 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.835 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.835 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.835 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.835 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.835 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.835 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.819+0000 7f25dd7fa640 10 monclient: _finish_auth 0 2024-04-26T00:20:03.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.819+0000 7f25dd7fa640 10 monclient: _check_auth_tickets 2024-04-26T00:20:03.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.819+0000 7f25dd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:03.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.819+0000 7f25dd7fa640 1 -- 172.21.15.37:0/2527512055 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25cc0ba070 con 0x7f25e00a4170 2024-04-26T00:20:03.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.819+0000 7f25e6d6e640 5 monclient: authenticate success, global_id 4238 2024-04-26T00:20:03.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.819+0000 7f25e6d6e640 10 monclient: _renew_subs 2024-04-26T00:20:03.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.819+0000 7f25e6d6e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:03.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.819+0000 7f25e6d6e640 1 -- 172.21.15.37:0/2527512055 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25e00a0990 con 0x7f25e00a4170 2024-04-26T00:20:03.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.819+0000 7f25dd7fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:03.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.819+0000 7f25dd7fa640 1 -- 172.21.15.37:0/2527512055 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25cc0aade0 con 0x7f25e00a4170 2024-04-26T00:20:03.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.819+0000 7f25dd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.819+0000 7f25dd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:03.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.819+0000 7f25dd7fa640 10 monclient: dump: 2024-04-26T00:20:03.836 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.836 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.836 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.836 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.837 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.837 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.837 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.837 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.837 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.837 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.819+0000 7f25deffd640 4 set_mon_vals no callback set 2024-04-26T00:20:03.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.819+0000 7f25e6d6e640 10 monclient: _renew_subs 2024-04-26T00:20:03.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.819+0000 7f25e6d6e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:03.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.819+0000 7f25e6d6e640 1 -- 172.21.15.37:0/2527512055 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f25e0072140 con 0x7f25e00a4170 2024-04-26T00:20:03.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.819+0000 7f5b367a8640 10 monclient: get_monmap_and_config 2024-04-26T00:20:03.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.819+0000 7f5b367a8640 10 monclient: build_initial_monmap 2024-04-26T00:20:03.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.819+0000 7f5b367a8640 10 monclient: monmap: 2024-04-26T00:20:03.838 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:03.838 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.838 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:03.826708+0000 2024-04-26T00:20:03.838 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:03.826708+0000 2024-04-26T00:20:03.838 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:03.838 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.838 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:03.839 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:03.839 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:03.839 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:03.839 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:03.839 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.819+0000 7f5b367a8640 1 Processor -- start 2024-04-26T00:20:03.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.819+0000 7f5b367a8640 1 -- start start 2024-04-26T00:20:03.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.819+0000 7f5b367a8640 10 monclient: init 2024-04-26T00:20:03.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.819+0000 7f25e6d6e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:03.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.819+0000 7f25e6d6e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:03.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.819+0000 7f25e6d6e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:03.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.819+0000 7f25e6d6e640 1 -- 172.21.15.37:0/2527512055 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25e004f510 con 0x7f25e00a4170 2024-04-26T00:20:03.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.819+0000 7f5b367a8640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:03.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.819+0000 7f5b367a8640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-04-26T00:20:03.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.819+0000 7f5b367a8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5b300a3d90 addr v1:172.21.15.45:6789/0 2024-04-26T00:20:03.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.823+0000 7f5b367a8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b300a4170 0x7f5b300a45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.823+0000 7f5b367a8640 10 monclient(hunting): picked mon.noname-b con 0x7f5b300a4170 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:03.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.823+0000 7f5e39cc4640 1 -- 172.21.15.37:0/448459810 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e2c0b8660 msgr2=0x7f5e2c01e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:03.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.823+0000 7f5e39cc4640 1 --2- 172.21.15.37:0/448459810 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e2c0b8660 0x7f5e2c01e5b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5e3000dc90 tx=0x7f5e300206f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31cd2c4862f89cd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.823+0000 7f5b367a8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5b3013c3e0 addr v1:172.21.15.37:6789/0 2024-04-26T00:20:03.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.823+0000 7f5b2f7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b300a4170 0x7f5b300a45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.823+0000 7f5b367a8640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b300a5150 con 0x7f5b3013c3e0 2024-04-26T00:20:03.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.823+0000 7f5b2f7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b300a4170 0x7f5b300a45b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:57892/0 (socket says 172.21.15.37:57892) 2024-04-26T00:20:03.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.823+0000 7f5b2f7fe640 1 -- 172.21.15.37:0/113396297 learned_addr learned my addr 172.21.15.37:0/113396297 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:03.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.823+0000 7f5b367a8640 1 -- 172.21.15.37:0/113396297 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b3009b280 con 0x7f5b300a3d90 2024-04-26T00:20:03.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.823+0000 7f5b367a8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.823+0000 7f5b367a8640 1 -- 172.21.15.37:0/113396297 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5b3009c650 con 0x7f5b300a4170 2024-04-26T00:20:03.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.823+0000 7f5b367a8640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:03.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.823+0000 7f5b367a8640 10 monclient(hunting): authenticate will time out at 985.132474s 2024-04-26T00:20:03.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.823+0000 7f5b2f7fe640 10 monclient(hunting): get_auth_request con 0x7f5b300a4170 auth_method 0 2024-04-26T00:20:03.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.823+0000 7f5b2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.823+0000 7f5b2f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.823+0000 7f5b2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.823+0000 7f5b2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.823+0000 7f5b2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.823+0000 7f5b2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.823+0000 7f5b2f7fe640 10 monclient(hunting): handle_auth_done global_id 4244 payload 274 2024-04-26T00:20:03.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.823+0000 7f5b2f7fe640 1 -- 172.21.15.37:0/113396297 >> v1:172.21.15.45:6789/0 conn(0x7f5b300a3d90 legacy=0x7f5b3009bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:03.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.823+0000 7f5b2f7fe640 1 -- 172.21.15.37:0/113396297 >> v1:172.21.15.37:6789/0 conn(0x7f5b3013c3e0 legacy=0x7f5b3013e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:03.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.823+0000 7f5b2f7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:20:03.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.823+0000 7f5b2f7fe640 1 monclient: found mon.noname-b 2024-04-26T00:20:03.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.823+0000 7f5b2f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:03.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.823+0000 7f5b2f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:20:03.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.823+0000 7f5b2f7fe640 1 -- 172.21.15.37:0/113396297 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b3009b560 con 0x7f5b300a4170 2024-04-26T00:20:03.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.823+0000 7f25dd7fa640 1 -- 172.21.15.37:0/2527512055 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 4) ==== 43962+0+0 (secure 0 0 0) 0x7f25cc0ab5c0 con 0x7f25e00a4170 2024-04-26T00:20:03.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.823+0000 7f25dd7fa640 1 --2- 172.21.15.37:0/2527512055 >> v2:172.21.15.45:6812/2379918623 conn(0x7f25c4035e40 0x7f25c4038300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.823+0000 7f25dd7fa640 1 -- 172.21.15.37:0/2527512055 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7f25cc0e3610 con 0x7f25e00a4170 2024-04-26T00:20:03.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.823+0000 7f25e4b0a640 1 --2- 172.21.15.37:0/2527512055 >> v2:172.21.15.45:6812/2379918623 conn(0x7f25c4035e40 0x7f25c4038300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.823+0000 7f25e4b0a640 10 monclient: get_auth_request con 0x7f25c4035e40 auth_method 0 2024-04-26T00:20:03.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7fa5037fe640 1 -- 172.21.15.37:0/3934835523 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa5100160f0 con 0x7fa51c0a1b60 2024-04-26T00:20:03.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7fa5037fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:03.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7fa5037fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:03.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f25e4b0a640 1 --2- 172.21.15.37:0/2527512055 >> v2:172.21.15.45:6812/2379918623 conn(0x7f25c4035e40 0x7f25c4038300 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f25d4006fd0 tx=0x7f25d4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5e38cc2640 1 -- 172.21.15.37:0/448459810 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e3000b450 con 0x7f5e2c0b8660 2024-04-26T00:20:03.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5e38cc2640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5e38cc2640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:20:03.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5e38cc2640 10 monclient: dump: 2024-04-26T00:20:03.847 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.847 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.847 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.847 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.847 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.847 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.847 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.847 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.847 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.847 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5e38cc2640 10 monclient: _finish_auth 0 2024-04-26T00:20:03.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5e38cc2640 10 monclient: _check_auth_tickets 2024-04-26T00:20:03.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5e38cc2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:03.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5e38cc2640 1 -- 172.21.15.37:0/448459810 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e300b2070 con 0x7f5e2c0b8660 2024-04-26T00:20:03.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5e3acc6640 5 monclient: authenticate success, global_id 4241 2024-04-26T00:20:03.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5e3acc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:20:03.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5e38cc2640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:03.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5e38cc2640 1 -- 172.21.15.37:0/448459810 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e3000b810 con 0x7f5e2c0b8660 2024-04-26T00:20:03.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5e38cc2640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5e38cc2640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:03.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5e38cc2640 10 monclient: dump: 2024-04-26T00:20:03.848 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.848 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.848 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.848 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.848 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.849 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.849 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.849 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.849 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.849 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5e3acc6640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:03.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5e3acc6640 4 set_mon_vals no callback set 2024-04-26T00:20:03.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f25dd7fa640 1 -- 172.21.15.37:0/2527512055 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f25cc0160b0 con 0x7f25e00a4170 2024-04-26T00:20:03.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f25dd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:03.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f25dd7fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:03.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5e3acc6640 10 monclient: shutdown 2024-04-26T00:20:03.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5e3acc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:03.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5e3acc6640 1 -- 172.21.15.37:0/448459810 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e2c0b8660 msgr2=0x7f5e2c01e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5e3acc6640 1 --2- 172.21.15.37:0/448459810 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e2c0b8660 0x7f5e2c01e5b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5e3000dc90 tx=0x7f5e300206f0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5e3acc6640 1 -- 172.21.15.37:0/448459810 shutdown_connections 2024-04-26T00:20:03.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5e3acc6640 1 --2- 172.21.15.37:0/448459810 >> v2:172.21.15.37:3300/0 conn(0x7f5e2c022040 0x7f5e2c0bf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5e3acc6640 1 --2- 172.21.15.37:0/448459810 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e2c0b8660 0x7f5e2c01e5b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5e3acc6640 1 -- 172.21.15.37:0/448459810 >> 172.21.15.37:0/448459810 conn(0x7f5e2c0068d0 msgr2=0x7f5e2c006ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:03.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5e3acc6640 1 -- 172.21.15.37:0/448459810 shutdown_connections 2024-04-26T00:20:03.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5e3acc6640 1 -- 172.21.15.37:0/448459810 wait complete. 2024-04-26T00:20:03.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5e3acc6640 10 monclient: build_initial_monmap 2024-04-26T00:20:03.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5e3acc6640 10 monclient: monmap: 2024-04-26T00:20:03.851 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:03.851 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:03.851 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:03.851 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:03.851 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:03.851 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:03.852 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:03.852 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:03.852 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:03.852 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5e3acc6640 1 Processor -- start 2024-04-26T00:20:03.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5e3acc6640 1 -- start start 2024-04-26T00:20:03.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5e3acc6640 10 monclient: init 2024-04-26T00:20:03.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5b2f7fe640 1 -- 172.21.15.37:0/113396297 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b300a4170 msgr2=0x7f5b300a45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:03.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5b2f7fe640 1 --2- 172.21.15.37:0/113396297 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b300a4170 0x7f5b300a45b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5b240098a0 tx=0x7f5b2401c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5cb269021751d38 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5b2effd640 1 -- 172.21.15.37:0/113396297 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b240b0070 con 0x7f5b300a4170 2024-04-26T00:20:03.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5b2effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5b2effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:20:03.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5b2effd640 10 monclient: dump: 2024-04-26T00:20:03.853 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.853 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.853 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.853 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.853 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.853 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.853 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.853 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.853 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.853 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5b2effd640 10 monclient: _finish_auth 0 2024-04-26T00:20:03.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5b2effd640 10 monclient: _check_auth_tickets 2024-04-26T00:20:03.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5b2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:03.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5b2effd640 1 -- 172.21.15.37:0/113396297 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b240ab600 con 0x7f5b300a4170 2024-04-26T00:20:03.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5b2effd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:03.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5b2effd640 1 -- 172.21.15.37:0/113396297 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b240ab9d0 con 0x7f5b300a4170 2024-04-26T00:20:03.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5b2effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5b2effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:03.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5b2effd640 10 monclient: dump: 2024-04-26T00:20:03.854 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.854 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.854 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.854 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.854 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.854 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.854 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.854 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.855 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.855 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5b367a8640 5 monclient: authenticate success, global_id 4244 2024-04-26T00:20:03.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5b367a8640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:03.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5b367a8640 4 set_mon_vals no callback set 2024-04-26T00:20:03.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5b367a8640 10 monclient: shutdown 2024-04-26T00:20:03.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5b367a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:03.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5b367a8640 1 -- 172.21.15.37:0/113396297 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b300a4170 msgr2=0x7f5b300a45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5b367a8640 1 --2- 172.21.15.37:0/113396297 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b300a4170 0x7f5b300a45b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5b240098a0 tx=0x7f5b2401c3c0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5b367a8640 1 -- 172.21.15.37:0/113396297 shutdown_connections 2024-04-26T00:20:03.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5b367a8640 1 --2- 172.21.15.37:0/113396297 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b300a4170 0x7f5b300a45b0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.827+0000 7f5b367a8640 1 -- 172.21.15.37:0/113396297 >> 172.21.15.37:0/113396297 conn(0x7f5b300aa0a0 msgr2=0x7f5b300a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:03.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b367a8640 1 -- 172.21.15.37:0/113396297 shutdown_connections 2024-04-26T00:20:03.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e3acc6640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:03.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e3acc6640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:20:03.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b367a8640 1 -- 172.21.15.37:0/113396297 wait complete. 2024-04-26T00:20:03.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e3acc6640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e2c01eaf0 0x7f5e2c158f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e3acc6640 10 monclient(hunting): picked mon.noname-a con 0x7f5e2c01eaf0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:03.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e3acc6640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e2c022040 0x7f5e2c159480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e3acc6640 10 monclient(hunting): picked mon.noname-c con 0x7f5e2c022040 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:03.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e3acc6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e2c0b8660 0x7f5e2c0bcfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e3acc6640 10 monclient(hunting): picked mon.noname-b con 0x7f5e2c0b8660 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:03.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e3acc6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e3acc6640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f5e2c0c29c0 con 0x7f5e2c01eaf0 2024-04-26T00:20:03.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e3acc6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e3acc6640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5e2c0c2840 con 0x7f5e2c0b8660 2024-04-26T00:20:03.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e3acc6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e3acc6640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5e2c0c2b40 con 0x7f5e2c022040 2024-04-26T00:20:03.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e3acc6640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:03.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e3acc6640 10 monclient(hunting): authenticate will time out at 985.140443s 2024-04-26T00:20:03.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e39cc4640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e2c01eaf0 0x7f5e2c158f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e39cc4640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e2c01eaf0 0x7f5e2c158f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44692/0 (socket says 172.21.15.37:44692) 2024-04-26T00:20:03.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e39cc4640 1 -- 172.21.15.37:0/1391781735 learned_addr learned my addr 172.21.15.37:0/1391781735 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:03.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e39cc4640 10 monclient(hunting): get_auth_request con 0x7f5e2c01eaf0 auth_method 0 2024-04-26T00:20:03.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e39cc4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e39cc4640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e39cc4640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e394c3640 1 --2- 172.21.15.37:0/1391781735 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e2c022040 0x7f5e2c159480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e394c3640 10 monclient(hunting): get_auth_request con 0x7f5e2c022040 auth_method 0 2024-04-26T00:20:03.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e394c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e394c3640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e394c3640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e3a4c5640 1 --2- 172.21.15.37:0/1391781735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e2c0b8660 0x7f5e2c0bcfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e39cc4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e39cc4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e39cc4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e3a4c5640 10 monclient(hunting): get_auth_request con 0x7f5e2c0b8660 auth_method 0 2024-04-26T00:20:03.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e3a4c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e3a4c5640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e3a4c5640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b367a8640 10 monclient: build_initial_monmap 2024-04-26T00:20:03.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b367a8640 10 monclient: monmap: 2024-04-26T00:20:03.859 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:03.859 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:03.859 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:03.860 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:03.860 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:03.860 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:03.860 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:03.860 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:03.860 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:03.860 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b367a8640 1 Processor -- start 2024-04-26T00:20:03.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e3a4c5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e3a4c5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e3a4c5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e394c3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e394c3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e394c3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e39cc4640 10 monclient(hunting): handle_auth_done global_id 4263 payload 824 2024-04-26T00:20:03.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e39cc4640 1 -- 172.21.15.37:0/1391781735 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e2c022040 msgr2=0x7f5e2c159480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e39cc4640 1 --2- 172.21.15.37:0/1391781735 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e2c022040 0x7f5e2c159480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e39cc4640 1 -- 172.21.15.37:0/1391781735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e2c0b8660 msgr2=0x7f5e2c0bcfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e39cc4640 1 --2- 172.21.15.37:0/1391781735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e2c0b8660 0x7f5e2c0bcfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e39cc4640 10 monclient: _finish_hunting 0 2024-04-26T00:20:03.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e39cc4640 1 monclient: found mon.noname-a 2024-04-26T00:20:03.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e39cc4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:03.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e39cc4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:03.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e39cc4640 1 -- 172.21.15.37:0/1391781735 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e2c0aa060 con 0x7f5e2c01eaf0 2024-04-26T00:20:03.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b367a8640 1 -- start start 2024-04-26T00:20:03.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b367a8640 10 monclient: init 2024-04-26T00:20:03.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b367a8640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:03.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b367a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:20:03.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b367a8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b300a3d90 0x7f5b3009d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b367a8640 10 monclient(hunting): picked mon.noname-b con 0x7f5b300a3d90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:03.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b367a8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b300a4170 0x7f5b3009db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b367a8640 10 monclient(hunting): picked mon.noname-c con 0x7f5b300a4170 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:03.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b367a8640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b3013c3e0 0x7f5b3009e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b367a8640 10 monclient(hunting): picked mon.noname-a con 0x7f5b3013c3e0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:03.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b367a8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b367a8640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f5b3006e8c0 con 0x7f5b3013c3e0 2024-04-26T00:20:03.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b367a8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b367a8640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5b3006e740 con 0x7f5b300a3d90 2024-04-26T00:20:03.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b367a8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b367a8640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5b3006ea40 con 0x7f5b300a4170 2024-04-26T00:20:03.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b367a8640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:03.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b367a8640 10 monclient(hunting): authenticate will time out at 985.141718s 2024-04-26T00:20:03.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b34d45640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b3013c3e0 0x7f5b3009e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b34d45640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b3013c3e0 0x7f5b3009e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44708/0 (socket says 172.21.15.37:44708) 2024-04-26T00:20:03.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b34d45640 1 -- 172.21.15.37:0/2092412415 learned_addr learned my addr 172.21.15.37:0/2092412415 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:03.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b34d45640 10 monclient(hunting): get_auth_request con 0x7f5b3013c3e0 auth_method 0 2024-04-26T00:20:03.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b34d45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b34d45640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b34d45640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b2f7fe640 1 --2- 172.21.15.37:0/2092412415 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b300a4170 0x7f5b3009db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b2f7fe640 10 monclient(hunting): get_auth_request con 0x7f5b300a4170 auth_method 0 2024-04-26T00:20:03.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b2f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b34d45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b34d45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b34d45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b2ffff640 1 --2- 172.21.15.37:0/2092412415 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b300a3d90 0x7f5b3009d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b2ffff640 10 monclient(hunting): get_auth_request con 0x7f5b300a3d90 auth_method 0 2024-04-26T00:20:03.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b2ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b34d45640 10 monclient(hunting): handle_auth_done global_id 4266 payload 824 2024-04-26T00:20:03.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b34d45640 1 -- 172.21.15.37:0/2092412415 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b300a4170 msgr2=0x7f5b3009db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b34d45640 1 --2- 172.21.15.37:0/2092412415 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b300a4170 0x7f5b3009db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b34d45640 1 -- 172.21.15.37:0/2092412415 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b300a3d90 msgr2=0x7f5b3009d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b34d45640 1 --2- 172.21.15.37:0/2092412415 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b300a3d90 0x7f5b3009d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b34d45640 10 monclient: _finish_hunting 0 2024-04-26T00:20:03.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b34d45640 1 monclient: found mon.noname-a 2024-04-26T00:20:03.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b34d45640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:03.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b34d45640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:03.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5b34d45640 1 -- 172.21.15.37:0/2092412415 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b3013bf10 con 0x7f5b3013c3e0 2024-04-26T00:20:03.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e39cc4640 1 -- 172.21.15.37:0/1391781735 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e2c01eaf0 msgr2=0x7f5e2c158f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:03.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.831+0000 7f5e39cc4640 1 --2- 172.21.15.37:0/1391781735 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e2c01eaf0 0x7f5e2c158f40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5e3000d040 tx=0x7f5e3001ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5e2a7fc640 1 -- 172.21.15.37:0/1391781735 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e300bc030 con 0x7f5e2c01eaf0 2024-04-26T00:20:03.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5e2a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5e2a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:03.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5e2a7fc640 10 monclient: dump: 2024-04-26T00:20:03.868 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.868 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.868 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.868 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.868 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.868 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.868 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.868 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.868 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.869 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5e2a7fc640 10 monclient: _finish_auth 0 2024-04-26T00:20:03.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5e2a7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:20:03.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5e2a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:03.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5e2a7fc640 1 -- 172.21.15.37:0/1391781735 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e300b8440 con 0x7f5e2c01eaf0 2024-04-26T00:20:03.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5e2a7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:03.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5e2a7fc640 1 -- 172.21.15.37:0/1391781735 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e300b8850 con 0x7f5e2c01eaf0 2024-04-26T00:20:03.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5e2a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5e2a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:03.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5e2a7fc640 10 monclient: dump: 2024-04-26T00:20:03.870 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.870 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.870 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.870 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.870 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.870 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.870 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.870 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.870 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.870 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5e3acc6640 5 monclient: authenticate success, global_id 4263 2024-04-26T00:20:03.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5e3acc6640 10 monclient: _renew_subs 2024-04-26T00:20:03.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5e3acc6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:03.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5e3acc6640 1 -- 172.21.15.37:0/1391781735 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e2c0bf720 con 0x7f5e2c01eaf0 2024-04-26T00:20:03.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5e3acc6640 10 monclient: _renew_subs 2024-04-26T00:20:03.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5e3acc6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:03.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5e3acc6640 1 -- 172.21.15.37:0/1391781735 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e2c0aad70 con 0x7f5e2c01eaf0 2024-04-26T00:20:03.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5b34d45640 1 -- 172.21.15.37:0/2092412415 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b3013c3e0 msgr2=0x7f5b3009e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:03.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5b34d45640 1 --2- 172.21.15.37:0/2092412415 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b3013c3e0 0x7f5b3009e040 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5b2800c7b0 tx=0x7f5b2800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5e2bfff640 4 set_mon_vals no callback set 2024-04-26T00:20:03.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5b2cff9640 1 -- 172.21.15.37:0/2092412415 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b2800abe0 con 0x7f5b3013c3e0 2024-04-26T00:20:03.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5b2cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5b2cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:03.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5b2cff9640 10 monclient: dump: 2024-04-26T00:20:03.872 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.872 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.872 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.872 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.872 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.872 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.873 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.873 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.873 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.873 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5b2cff9640 10 monclient: _finish_auth 0 2024-04-26T00:20:03.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5b2cff9640 10 monclient: _check_auth_tickets 2024-04-26T00:20:03.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5b2cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:03.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5b2cff9640 1 -- 172.21.15.37:0/2092412415 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b2800ad80 con 0x7f5b3013c3e0 2024-04-26T00:20:03.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5b2cff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:03.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5b2cff9640 1 -- 172.21.15.37:0/2092412415 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b28004bb0 con 0x7f5b3013c3e0 2024-04-26T00:20:03.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5b2cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5b2cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:03.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5b2cff9640 10 monclient: dump: 2024-04-26T00:20:03.873 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.874 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.874 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.874 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.874 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.874 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.874 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.874 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.874 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.874 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5b2e7fc640 4 set_mon_vals no callback set 2024-04-26T00:20:03.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5e2a7fc640 1 -- 172.21.15.37:0/1391781735 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 4) ==== 43962+0+0 (secure 0 0 0) 0x7f5e300bf070 con 0x7f5e2c01eaf0 2024-04-26T00:20:03.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5b367a8640 5 monclient: authenticate success, global_id 4266 2024-04-26T00:20:03.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5b367a8640 10 monclient: _renew_subs 2024-04-26T00:20:03.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5b367a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:03.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5b367a8640 1 -- 172.21.15.37:0/2092412415 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b30137f30 con 0x7f5b3013c3e0 2024-04-26T00:20:03.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5e3acc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:03.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5e3acc6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:03.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5e3acc6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:03.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5b367a8640 10 monclient: _renew_subs 2024-04-26T00:20:03.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5b367a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:03.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5b367a8640 1 -- 172.21.15.37:0/2092412415 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b30071ef0 con 0x7f5b3013c3e0 2024-04-26T00:20:03.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5e3acc6640 1 -- 172.21.15.37:0/1391781735 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e1c005090 con 0x7f5e2c01eaf0 2024-04-26T00:20:03.877 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5e2a7fc640 1 --2- 172.21.15.37:0/1391781735 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5e18035df0 0x7f5e180382b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.877 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5e2a7fc640 1 -- 172.21.15.37:0/1391781735 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7f5e300e2630 con 0x7f5e2c01eaf0 2024-04-26T00:20:03.877 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5e394c3640 1 --2- 172.21.15.37:0/1391781735 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5e18035df0 0x7f5e180382b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.877 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5e394c3640 10 monclient: get_auth_request con 0x7f5e18035df0 auth_method 0 2024-04-26T00:20:03.877 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5b367a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:03.877 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5b367a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:03.877 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5b367a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:03.877 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.835+0000 7f5b367a8640 1 -- 172.21.15.37:0/2092412415 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b3004f510 con 0x7f5b3013c3e0 2024-04-26T00:20:03.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.839+0000 7f5e394c3640 1 --2- 172.21.15.37:0/1391781735 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5e18035df0 0x7f5e180382b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c0aa270 tx=0x7f5e24009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.839+0000 7f5b2cff9640 1 -- 172.21.15.37:0/2092412415 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 4) ==== 43962+0+0 (secure 0 0 0) 0x7f5b28005030 con 0x7f5b3013c3e0 2024-04-26T00:20:03.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.839+0000 7f5b2cff9640 1 --2- 172.21.15.37:0/2092412415 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5b1c035e40 0x7f5b1c038300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.839+0000 7f5b2ffff640 1 --2- 172.21.15.37:0/2092412415 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5b1c035e40 0x7f5b1c038300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.839+0000 7f5b2ffff640 10 monclient: get_auth_request con 0x7f5b1c035e40 auth_method 0 2024-04-26T00:20:03.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.843+0000 7f5b2ffff640 1 --2- 172.21.15.37:0/2092412415 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5b1c035e40 0x7f5b1c038300 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5b20004e30 tx=0x7f5b20009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.843+0000 7f5b2cff9640 1 -- 172.21.15.37:0/2092412415 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7f5b2801a560 con 0x7f5b3013c3e0 2024-04-26T00:20:03.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.843+0000 7f5e2a7fc640 1 -- 172.21.15.37:0/1391781735 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5e3001a400 con 0x7f5e2c01eaf0 2024-04-26T00:20:03.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.843+0000 7f5e2a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:03.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.843+0000 7f5e2a7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:03.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.851+0000 7f5b2cff9640 1 -- 172.21.15.37:0/2092412415 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5b2804f030 con 0x7f5b3013c3e0 2024-04-26T00:20:03.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.851+0000 7f5b2cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:03.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.851+0000 7f5b2cff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:03.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.931+0000 7f7272a3e640 10 monclient: get_monmap_and_config 2024-04-26T00:20:03.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.931+0000 7f7272a3e640 10 monclient: build_initial_monmap 2024-04-26T00:20:03.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.931+0000 7f7272a3e640 10 monclient: monmap: 2024-04-26T00:20:03.915 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:03.915 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.915 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:03.938780+0000 2024-04-26T00:20:03.915 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:03.938780+0000 2024-04-26T00:20:03.915 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:03.915 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.915 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:03.915 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:03.915 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:03.916 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:03.916 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:03.916 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.931+0000 7f7272a3e640 1 Processor -- start 2024-04-26T00:20:03.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.931+0000 7f7272a3e640 1 -- start start 2024-04-26T00:20:03.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.931+0000 7f7272a3e640 10 monclient: init 2024-04-26T00:20:03.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.931+0000 7f7272a3e640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:03.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.931+0000 7f7272a3e640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-04-26T00:20:03.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.931+0000 7f7272a3e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f726c0a3d90 0x7f726c09bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.931+0000 7f7272a3e640 10 monclient(hunting): picked mon.noname-c con 0x7f726c0a3d90 addr v2:172.21.15.45:3300/0 2024-04-26T00:20:03.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.935+0000 7f7272a3e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f726c0a4170 addr v1:172.21.15.45:6789/0 2024-04-26T00:20:03.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.935+0000 7f7272a3e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f726c13c3e0 addr v1:172.21.15.37:6789/0 2024-04-26T00:20:03.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.935+0000 7f7272a3e640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f726c0a5150 con 0x7f726c13c3e0 2024-04-26T00:20:03.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.935+0000 7f7272a3e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f726c09b280 con 0x7f726c0a4170 2024-04-26T00:20:03.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.935+0000 7f7272a3e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.935+0000 7f7272a3e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f726c09b560 con 0x7f726c0a3d90 2024-04-26T00:20:03.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.935+0000 7f7272a3e640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:03.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.935+0000 7f7272a3e640 10 monclient(hunting): authenticate will time out at 985.244166s 2024-04-26T00:20:03.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.935+0000 7f7270fdb640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f726c13c3e0 0x7f726c13e7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:55802/0 (socket says 172.21.15.37:55802) 2024-04-26T00:20:03.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.935+0000 7f7270fdb640 1 -- 172.21.15.37:0/2026401338 learned_addr learned my addr 172.21.15.37:0/2026401338 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:20:03.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.935+0000 7f726bfff640 1 --2- 172.21.15.37:0/2026401338 >> v2:172.21.15.45:3300/0 conn(0x7f726c0a3d90 0x7f726c09bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.935+0000 7f726bfff640 10 monclient(hunting): get_auth_request con 0x7f726c0a3d90 auth_method 0 2024-04-26T00:20:03.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.935+0000 7f726bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.935+0000 7f726bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.935+0000 7f726bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.935+0000 7f726bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.935+0000 7f726bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.935+0000 7f726bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.935+0000 7f726bfff640 10 monclient(hunting): handle_auth_done global_id 4249 payload 274 2024-04-26T00:20:03.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.935+0000 7f726bfff640 1 -- 172.21.15.37:0/2026401338 >> v1:172.21.15.45:6789/0 conn(0x7f726c0a4170 legacy=0x7f726c0a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:03.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.935+0000 7f726bfff640 1 -- 172.21.15.37:0/2026401338 >> v1:172.21.15.37:6789/0 conn(0x7f726c13c3e0 legacy=0x7f726c13e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:03.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.935+0000 7f726bfff640 10 monclient: _finish_hunting 0 2024-04-26T00:20:03.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.935+0000 7f726bfff640 1 monclient: found mon.noname-c 2024-04-26T00:20:03.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.935+0000 7f726bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:03.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.935+0000 7f726bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:20:03.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.935+0000 7f726bfff640 1 -- 172.21.15.37:0/2026401338 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f726c06dc60 con 0x7f726c0a3d90 2024-04-26T00:20:03.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.939+0000 7f726bfff640 1 -- 172.21.15.37:0/2026401338 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f726c0a3d90 msgr2=0x7f726c09bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:03.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.939+0000 7f726bfff640 1 --2- 172.21.15.37:0/2026401338 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f726c0a3d90 0x7f726c09bf40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f725c0097c0 tx=0x7f725c01c2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b68094818d2bfa53 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.939+0000 7f726affd640 1 -- 172.21.15.37:0/2026401338 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f725c0b0070 con 0x7f726c0a3d90 2024-04-26T00:20:03.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.939+0000 7f726affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.939+0000 7f726affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:20:03.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.939+0000 7f726affd640 10 monclient: dump: 2024-04-26T00:20:03.925 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.925 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.925 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.925 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.925 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.925 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.925 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.925 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.925 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.925 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.939+0000 7f726affd640 10 monclient: _finish_auth 0 2024-04-26T00:20:03.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.939+0000 7f726affd640 10 monclient: _check_auth_tickets 2024-04-26T00:20:03.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.939+0000 7f726affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:03.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.939+0000 7f7272a3e640 5 monclient: authenticate success, global_id 4249 2024-04-26T00:20:03.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.939+0000 7f7272a3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:20:03.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.939+0000 7f726affd640 1 -- 172.21.15.37:0/2026401338 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f725c0ab600 con 0x7f726c0a3d90 2024-04-26T00:20:03.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.939+0000 7f726affd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:03.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.939+0000 7f726affd640 1 -- 172.21.15.37:0/2026401338 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f725c0ab9d0 con 0x7f726c0a3d90 2024-04-26T00:20:03.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.939+0000 7f726affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.939+0000 7f726affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:20:03.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.939+0000 7f726affd640 10 monclient: dump: 2024-04-26T00:20:03.926 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.926 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.927 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.927 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.927 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.927 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.927 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.927 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.927 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.927 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.939+0000 7f7272a3e640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:03.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.939+0000 7f7272a3e640 4 set_mon_vals no callback set 2024-04-26T00:20:03.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.939+0000 7f7272a3e640 10 monclient: shutdown 2024-04-26T00:20:03.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.939+0000 7f7272a3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:03.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.939+0000 7f7272a3e640 1 -- 172.21.15.37:0/2026401338 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f726c0a3d90 msgr2=0x7f726c09bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.939+0000 7f7272a3e640 1 --2- 172.21.15.37:0/2026401338 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f726c0a3d90 0x7f726c09bf40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f725c0097c0 tx=0x7f725c01c2e0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.939+0000 7f7272a3e640 1 -- 172.21.15.37:0/2026401338 shutdown_connections 2024-04-26T00:20:03.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.939+0000 7f7272a3e640 1 --2- 172.21.15.37:0/2026401338 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f726c0a3d90 0x7f726c09bf40 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.939+0000 7f7272a3e640 1 -- 172.21.15.37:0/2026401338 >> 172.21.15.37:0/2026401338 conn(0x7f726c0aa0a0 msgr2=0x7f726c0a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:03.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.939+0000 7f7272a3e640 1 -- 172.21.15.37:0/2026401338 shutdown_connections 2024-04-26T00:20:03.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.939+0000 7f7272a3e640 1 -- 172.21.15.37:0/2026401338 wait complete. 2024-04-26T00:20:03.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f7272a3e640 10 monclient: build_initial_monmap 2024-04-26T00:20:03.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f7272a3e640 10 monclient: monmap: 2024-04-26T00:20:03.928 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:03.928 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:03.928 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:03.928 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:03.928 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:03.928 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:03.928 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:03.928 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:03.928 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:03.929 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f7272a3e640 1 Processor -- start 2024-04-26T00:20:03.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f7272a3e640 1 -- start start 2024-04-26T00:20:03.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f7272a3e640 10 monclient: init 2024-04-26T00:20:03.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f7272a3e640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:03.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f7272a3e640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:20:03.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f7272a3e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f726c0a3d90 0x7f726c1cebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f7272a3e640 10 monclient(hunting): picked mon.noname-b con 0x7f726c0a3d90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:03.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f7272a3e640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f726c0a4170 0x7f726c136e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f7272a3e640 10 monclient(hunting): picked mon.noname-a con 0x7f726c0a4170 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:03.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f7272a3e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f726c13c3e0 0x7f726c138f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f7272a3e640 10 monclient(hunting): picked mon.noname-c con 0x7f726c13c3e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:03.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f7272a3e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f7272a3e640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f726c142920 con 0x7f726c0a4170 2024-04-26T00:20:03.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f7272a3e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f7272a3e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f726c09c690 con 0x7f726c0a3d90 2024-04-26T00:20:03.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f7272a3e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:03.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f7272a3e640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f726c142aa0 con 0x7f726c13c3e0 2024-04-26T00:20:03.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f7272a3e640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:03.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f7272a3e640 10 monclient(hunting): authenticate will time out at 985.253366s 2024-04-26T00:20:03.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f726b7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f726c0a4170 0x7f726c136e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f726b7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f726c0a4170 0x7f726c136e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44722/0 (socket says 172.21.15.37:44722) 2024-04-26T00:20:03.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f726b7fe640 1 -- 172.21.15.37:0/3805953110 learned_addr learned my addr 172.21.15.37:0/3805953110 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:03.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f726b7fe640 10 monclient(hunting): get_auth_request con 0x7f726c0a4170 auth_method 0 2024-04-26T00:20:03.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f726b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f726b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f726b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f7270fdb640 1 --2- 172.21.15.37:0/3805953110 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f726c13c3e0 0x7f726c138f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f7270fdb640 10 monclient(hunting): get_auth_request con 0x7f726c13c3e0 auth_method 0 2024-04-26T00:20:03.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f7270fdb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f7270fdb640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f7270fdb640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f726b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f726b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f726b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f726bfff640 1 --2- 172.21.15.37:0/3805953110 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f726c0a3d90 0x7f726c1cebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f726bfff640 10 monclient(hunting): get_auth_request con 0x7f726c0a3d90 auth_method 0 2024-04-26T00:20:03.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f726bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:03.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f726bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:03.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f726bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:03.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f7270fdb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f7270fdb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f7270fdb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f726bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:03.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f726bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:03.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f726bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:03.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f7270fdb640 10 monclient(hunting): handle_auth_done global_id 4253 payload 824 2024-04-26T00:20:03.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f7270fdb640 1 -- 172.21.15.37:0/3805953110 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f726c0a3d90 msgr2=0x7f726c1cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f7270fdb640 1 --2- 172.21.15.37:0/3805953110 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f726c0a3d90 0x7f726c1cebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f7270fdb640 1 -- 172.21.15.37:0/3805953110 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f726c0a4170 msgr2=0x7f726c136e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f7270fdb640 1 --2- 172.21.15.37:0/3805953110 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f726c0a4170 0x7f726c136e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f7270fdb640 10 monclient: _finish_hunting 0 2024-04-26T00:20:03.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f7270fdb640 1 monclient: found mon.noname-c 2024-04-26T00:20:03.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f7270fdb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:03.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f7270fdb640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:20:03.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f7270fdb640 1 -- 172.21.15.37:0/3805953110 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f726c1204d0 con 0x7f726c13c3e0 2024-04-26T00:20:03.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.943+0000 7f726b7fe640 1 --2- 172.21.15.37:0/3805953110 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f726c0a4170 0x7f726c136e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:03.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.947+0000 7f7270fdb640 1 -- 172.21.15.37:0/3805953110 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f726c13c3e0 msgr2=0x7f726c138f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:03.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.947+0000 7f7270fdb640 1 --2- 172.21.15.37:0/3805953110 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f726c13c3e0 0x7f726c138f70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f726400cb20 tx=0x7f726400cf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.947+0000 7f7268ff9640 1 -- 172.21.15.37:0/3805953110 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7264017070 con 0x7f726c13c3e0 2024-04-26T00:20:03.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.947+0000 7f7268ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.947+0000 7f7268ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:20:03.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.947+0000 7f7268ff9640 10 monclient: dump: 2024-04-26T00:20:03.936 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.936 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.936 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.936 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.936 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.936 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.936 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.936 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.936 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.936 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.947+0000 7f7268ff9640 10 monclient: _finish_auth 0 2024-04-26T00:20:03.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.947+0000 7f7268ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:20:03.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.947+0000 7f7268ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:03.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.947+0000 7f7268ff9640 1 -- 172.21.15.37:0/3805953110 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f726400ade0 con 0x7f726c13c3e0 2024-04-26T00:20:03.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.947+0000 7f7272a3e640 5 monclient: authenticate success, global_id 4253 2024-04-26T00:20:03.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.947+0000 7f7272a3e640 10 monclient: _renew_subs 2024-04-26T00:20:03.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.947+0000 7f7272a3e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:03.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.947+0000 7f7272a3e640 1 -- 172.21.15.37:0/3805953110 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f726c13b630 con 0x7f726c13c3e0 2024-04-26T00:20:03.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.947+0000 7f7268ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:03.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.947+0000 7f7268ff9640 1 -- 172.21.15.37:0/3805953110 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7264012620 con 0x7f726c13c3e0 2024-04-26T00:20:03.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.947+0000 7f7268ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:03.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.947+0000 7f7268ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:03.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.947+0000 7f7268ff9640 10 monclient: dump: 2024-04-26T00:20:03.937 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:03.937 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:03.938 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.938 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:03.938 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:03.938 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:03.938 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:03.938 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:03.938 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:03.938 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:03.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.947+0000 7f7272a3e640 10 monclient: _renew_subs 2024-04-26T00:20:03.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.947+0000 7f7272a3e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:03.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.947+0000 7f7272a3e640 1 -- 172.21.15.37:0/3805953110 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f726c09ada0 con 0x7f726c13c3e0 2024-04-26T00:20:03.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.947+0000 7f726a7fc640 4 set_mon_vals no callback set 2024-04-26T00:20:03.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.947+0000 7f7268ff9640 1 -- 172.21.15.37:0/3805953110 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 4) ==== 43962+0+0 (secure 0 0 0) 0x7f726401a070 con 0x7f726c13c3e0 2024-04-26T00:20:03.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.947+0000 7f7272a3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:03.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.947+0000 7f7272a3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:03.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.947+0000 7f7272a3e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:03.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.947+0000 7f7272a3e640 1 -- 172.21.15.37:0/3805953110 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f726c04f510 con 0x7f726c13c3e0 2024-04-26T00:20:03.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.947+0000 7f7268ff9640 1 --2- 172.21.15.37:0/3805953110 >> v2:172.21.15.45:6812/2379918623 conn(0x7f725803e600 0x7f7258040ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:03.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.947+0000 7f7268ff9640 1 -- 172.21.15.37:0/3805953110 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7f7264048570 con 0x7f726c13c3e0 2024-04-26T00:20:03.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.947+0000 7f726bfff640 1 --2- 172.21.15.37:0/3805953110 >> v2:172.21.15.45:6812/2379918623 conn(0x7f725803e600 0x7f7258040ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:03.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.947+0000 7f726bfff640 10 monclient: get_auth_request con 0x7f725803e600 auth_method 0 2024-04-26T00:20:03.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.951+0000 7f726bfff640 1 --2- 172.21.15.37:0/3805953110 >> v2:172.21.15.45:6812/2379918623 conn(0x7f725803e600 0x7f7258040ac0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f726c0a6e00 tx=0x7f725c0a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:03.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.955+0000 7f7268ff9640 1 -- 172.21.15.37:0/3805953110 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7264047470 con 0x7f726c13c3e0 2024-04-26T00:20:03.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.955+0000 7f7268ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:03.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:03.955+0000 7f7268ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:03.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.003+0000 7fa52147b640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-26T00:20:03.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.003+0000 7fa52147b640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-26T00:20:03.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.003+0000 7fa52147b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:03.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.003+0000 7fa52147b640 1 -- 172.21.15.37:0/3934835523 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fa51c11d910 con 0x7fa51c0a1b60 2024-04-26T00:20:03.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.003+0000 7fa5037fe640 1 -- 172.21.15.37:0/3934835523 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa5100aa3a0 con 0x7fa51c0a1b60 2024-04-26T00:20:03.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.003+0000 7fa5037fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-26T00:20:03.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.003+0000 7fa5037fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:03.988 INFO:teuthology.orchestra.run.smithi037.stdout:47244640258 2024-04-26T00:20:03.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.007+0000 7fa52147b640 1 -- 172.21.15.37:0/3934835523 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa508035e40 msgr2=0x7fa508038300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.007+0000 7fa52147b640 1 --2- 172.21.15.37:0/3934835523 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa508035e40 0x7fa508038300 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa50c007fd0 tx=0x7fa50c007e40 comp rx=0 tx=0).stop 2024-04-26T00:20:03.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.007+0000 7fa52147b640 10 monclient: shutdown 2024-04-26T00:20:03.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.007+0000 7fa52147b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:03.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.007+0000 7fa52147b640 1 -- 172.21.15.37:0/3934835523 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa51c0a1b60 msgr2=0x7fa51c13aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:03.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.007+0000 7fa52147b640 1 --2- 172.21.15.37:0/3934835523 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa51c0a1b60 0x7fa51c13aa40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa51001cd10 tx=0x7fa510019210 comp rx=0 tx=0).stop 2024-04-26T00:20:03.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.007+0000 7fa52147b640 1 -- 172.21.15.37:0/3934835523 shutdown_connections 2024-04-26T00:20:03.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.007+0000 7fa52147b640 1 --2- 172.21.15.37:0/3934835523 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa508035e40 0x7fa508038300 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.007+0000 7fa52147b640 1 --2- 172.21.15.37:0/3934835523 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa51c131cd0 0x7fa51c13af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.007+0000 7fa52147b640 1 --2- 172.21.15.37:0/3934835523 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa51c0a1b60 0x7fa51c13aa40 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.007+0000 7fa52147b640 1 --2- 172.21.15.37:0/3934835523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa51c073080 0x7fa51c13a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:03.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.007+0000 7fa52147b640 1 -- 172.21.15.37:0/3934835523 >> 172.21.15.37:0/3934835523 conn(0x7fa51c0a7160 msgr2=0x7fa51c0a0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:03.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.007+0000 7fa52147b640 1 -- 172.21.15.37:0/3934835523 shutdown_connections 2024-04-26T00:20:03.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.007+0000 7fa52147b640 1 -- 172.21.15.37:0/3934835523 wait complete. 2024-04-26T00:20:04.005 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.3 2024-04-26T00:20:04.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.043+0000 7f5b2cff9640 1 -- 172.21.15.37:0/2092412415 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f5b28012420 con 0x7f5b3013c3e0 2024-04-26T00:20:04.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.043+0000 7f7268ff9640 1 -- 172.21.15.37:0/3805953110 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f72640074f0 con 0x7f726c13c3e0 2024-04-26T00:20:04.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.047+0000 7fb5417fa640 1 -- 172.21.15.37:0/4146518997 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fb54409ada0 con 0x7fb5440a4170 2024-04-26T00:20:04.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.047+0000 7fb5417fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:04.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.047+0000 7fb5417fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:04.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.047+0000 7f25dd7fa640 1 -- 172.21.15.37:0/2527512055 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f25cc0e1630 con 0x7f25e00a4170 2024-04-26T00:20:04.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.047+0000 7f5e2a7fc640 1 -- 172.21.15.37:0/1391781735 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f5e300c17e0 con 0x7f5e2c01eaf0 2024-04-26T00:20:04.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.059+0000 7f25e6d6e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-26T00:20:04.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.059+0000 7f25e6d6e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-26T00:20:04.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.059+0000 7f25e6d6e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:04.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.059+0000 7f25e6d6e640 1 -- 172.21.15.37:0/2527512055 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f25e0071b80 con 0x7f25e00a4170 2024-04-26T00:20:04.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.059+0000 7f25dd7fa640 1 -- 172.21.15.37:0/2527512055 <== mon.2 v2:172.21.15.37:3301/0 8 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f25cc0b3300 con 0x7f25e00a4170 2024-04-26T00:20:04.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.059+0000 7f25dd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-26T00:20:04.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.059+0000 7f25dd7fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:04.041 INFO:teuthology.orchestra.run.smithi037.stdout:47244640258 2024-04-26T00:20:04.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.059+0000 7f25e6d6e640 1 -- 172.21.15.37:0/2527512055 >> v2:172.21.15.45:6812/2379918623 conn(0x7f25c4035e40 msgr2=0x7f25c4038300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:04.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.059+0000 7f25e6d6e640 1 --2- 172.21.15.37:0/2527512055 >> v2:172.21.15.45:6812/2379918623 conn(0x7f25c4035e40 0x7f25c4038300 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f25d4006fd0 tx=0x7f25d4008040 comp rx=0 tx=0).stop 2024-04-26T00:20:04.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.059+0000 7f25e6d6e640 10 monclient: shutdown 2024-04-26T00:20:04.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.059+0000 7f25e6d6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:04.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.059+0000 7f25e6d6e640 1 -- 172.21.15.37:0/2527512055 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f25e00a4170 msgr2=0x7f25e006ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:04.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.059+0000 7f25e6d6e640 1 --2- 172.21.15.37:0/2527512055 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f25e00a4170 0x7f25e006ea30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f25e006e0e0 tx=0x7f25cc019140 comp rx=0 tx=0).stop 2024-04-26T00:20:04.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.059+0000 7f25e6d6e640 1 -- 172.21.15.37:0/2527512055 shutdown_connections 2024-04-26T00:20:04.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.059+0000 7f25e6d6e640 1 --2- 172.21.15.37:0/2527512055 >> v2:172.21.15.45:6812/2379918623 conn(0x7f25c4035e40 0x7f25c4038300 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:04.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.059+0000 7f25e6d6e640 1 --2- 172.21.15.37:0/2527512055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25e0133b20 0x7f25e006ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:04.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.059+0000 7f25e6d6e640 1 --2- 172.21.15.37:0/2527512055 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f25e00a4170 0x7f25e006ea30 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:04.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.059+0000 7f25e6d6e640 1 --2- 172.21.15.37:0/2527512055 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25e00a3d90 0x7f25e006e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:04.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.059+0000 7f25e6d6e640 1 -- 172.21.15.37:0/2527512055 >> 172.21.15.37:0/2527512055 conn(0x7f25e00aa0a0 msgr2=0x7f25e00a2950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:04.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.059+0000 7f25e6d6e640 1 -- 172.21.15.37:0/2527512055 shutdown_connections 2024-04-26T00:20:04.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.059+0000 7f25e6d6e640 1 -- 172.21.15.37:0/2527512055 wait complete. 2024-04-26T00:20:04.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.071+0000 7f5e3acc6640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-26T00:20:04.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.071+0000 7f5e3acc6640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-26T00:20:04.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.071+0000 7f5e3acc6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:04.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.071+0000 7f5e3acc6640 1 -- 172.21.15.37:0/1391781735 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f5e1c005380 con 0x7f5e2c01eaf0 2024-04-26T00:20:04.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.071+0000 7f5e2a7fc640 1 -- 172.21.15.37:0/1391781735 <== mon.0 v2:172.21.15.37:3300/0 8 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5e300e14a0 con 0x7f5e2c01eaf0 2024-04-26T00:20:04.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.071+0000 7f5e2a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-26T00:20:04.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.071+0000 7f5e2a7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:04.056 INFO:teuthology.orchestra.run.smithi037.stdout:47244640257 2024-04-26T00:20:04.059 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.0 2024-04-26T00:20:04.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.075+0000 7f5e3acc6640 1 -- 172.21.15.37:0/1391781735 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5e18035df0 msgr2=0x7f5e180382b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:04.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.075+0000 7f5e3acc6640 1 --2- 172.21.15.37:0/1391781735 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5e18035df0 0x7f5e180382b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c0aa270 tx=0x7f5e24009040 comp rx=0 tx=0).stop 2024-04-26T00:20:04.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.075+0000 7f5e3acc6640 10 monclient: shutdown 2024-04-26T00:20:04.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.075+0000 7f5e3acc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:04.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.075+0000 7f5e3acc6640 1 -- 172.21.15.37:0/1391781735 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e2c01eaf0 msgr2=0x7f5e2c158f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:04.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.075+0000 7f5e3acc6640 1 --2- 172.21.15.37:0/1391781735 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e2c01eaf0 0x7f5e2c158f40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5e3000d040 tx=0x7f5e3001ef30 comp rx=0 tx=0).stop 2024-04-26T00:20:04.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.075+0000 7f5e3acc6640 1 -- 172.21.15.37:0/1391781735 shutdown_connections 2024-04-26T00:20:04.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.075+0000 7f5e3acc6640 1 --2- 172.21.15.37:0/1391781735 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5e18035df0 0x7f5e180382b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:04.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.075+0000 7f5e3acc6640 1 --2- 172.21.15.37:0/1391781735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e2c0b8660 0x7f5e2c0bcfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:04.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.075+0000 7f5e3acc6640 1 --2- 172.21.15.37:0/1391781735 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e2c022040 0x7f5e2c159480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:04.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.075+0000 7f5e3acc6640 1 --2- 172.21.15.37:0/1391781735 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e2c01eaf0 0x7f5e2c158f40 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:04.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.075+0000 7f5e3acc6640 1 -- 172.21.15.37:0/1391781735 >> 172.21.15.37:0/1391781735 conn(0x7f5e2c0068d0 msgr2=0x7f5e2c01cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:04.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.075+0000 7f5e3acc6640 1 -- 172.21.15.37:0/1391781735 shutdown_connections 2024-04-26T00:20:04.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.075+0000 7f5e3acc6640 1 -- 172.21.15.37:0/1391781735 wait complete. 2024-04-26T00:20:04.081 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640257 for osd.1 2024-04-26T00:20:04.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.147+0000 7f5b367a8640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-26T00:20:04.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.147+0000 7f5b367a8640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-26T00:20:04.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.147+0000 7f5b367a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:04.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.147+0000 7f5b367a8640 1 -- 172.21.15.37:0/2092412415 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f5b30075dc0 con 0x7f5b3013c3e0 2024-04-26T00:20:04.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.147+0000 7f5b2cff9640 1 -- 172.21.15.37:0/2092412415 <== mon.0 v2:172.21.15.37:3300/0 8 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5b28004d90 con 0x7f5b3013c3e0 2024-04-26T00:20:04.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.147+0000 7f5b2cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-26T00:20:04.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.147+0000 7f5b2cff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:04.133 INFO:teuthology.orchestra.run.smithi037.stdout:47244640259 2024-04-26T00:20:04.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.151+0000 7f5b367a8640 1 -- 172.21.15.37:0/2092412415 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5b1c035e40 msgr2=0x7f5b1c038300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:04.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.151+0000 7f5b367a8640 1 --2- 172.21.15.37:0/2092412415 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5b1c035e40 0x7f5b1c038300 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5b20004e30 tx=0x7f5b20009040 comp rx=0 tx=0).stop 2024-04-26T00:20:04.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.151+0000 7f5b367a8640 10 monclient: shutdown 2024-04-26T00:20:04.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.151+0000 7f5b367a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:04.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.151+0000 7f5b367a8640 1 -- 172.21.15.37:0/2092412415 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b3013c3e0 msgr2=0x7f5b3009e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:04.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.151+0000 7f5b367a8640 1 --2- 172.21.15.37:0/2092412415 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b3013c3e0 0x7f5b3009e040 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5b2800c7b0 tx=0x7f5b2800cc70 comp rx=0 tx=0).stop 2024-04-26T00:20:04.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.151+0000 7f5b367a8640 1 -- 172.21.15.37:0/2092412415 shutdown_connections 2024-04-26T00:20:04.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.151+0000 7f5b367a8640 1 --2- 172.21.15.37:0/2092412415 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5b1c035e40 0x7f5b1c038300 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:04.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.151+0000 7f5b367a8640 1 --2- 172.21.15.37:0/2092412415 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b3013c3e0 0x7f5b3009e040 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:04.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.151+0000 7f5b367a8640 1 --2- 172.21.15.37:0/2092412415 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b300a4170 0x7f5b3009db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:04.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.151+0000 7f5b367a8640 1 --2- 172.21.15.37:0/2092412415 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b300a3d90 0x7f5b3009d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:04.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.151+0000 7f5b367a8640 1 -- 172.21.15.37:0/2092412415 >> 172.21.15.37:0/2092412415 conn(0x7f5b300aa0a0 msgr2=0x7f5b30070ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:04.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.151+0000 7f5b367a8640 1 -- 172.21.15.37:0/2092412415 shutdown_connections 2024-04-26T00:20:04.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.151+0000 7f5b367a8640 1 -- 172.21.15.37:0/2092412415 wait complete. 2024-04-26T00:20:04.148 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640259 for osd.2 2024-04-26T00:20:04.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.211+0000 7f7272a3e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-26T00:20:04.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.211+0000 7f7272a3e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-26T00:20:04.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.211+0000 7f7272a3e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:04.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.211+0000 7f7272a3e640 1 -- 172.21.15.37:0/3805953110 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f726c13b3b0 con 0x7f726c13c3e0 2024-04-26T00:20:04.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.211+0000 7f7268ff9640 1 -- 172.21.15.37:0/3805953110 <== mon.2 v2:172.21.15.37:3301/0 8 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7264007a10 con 0x7f726c13c3e0 2024-04-26T00:20:04.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.211+0000 7f7268ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-26T00:20:04.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.211+0000 7f7268ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:04.194 INFO:teuthology.orchestra.run.smithi037.stdout:47244640257 2024-04-26T00:20:04.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.215+0000 7f7272a3e640 1 -- 172.21.15.37:0/3805953110 >> v2:172.21.15.45:6812/2379918623 conn(0x7f725803e600 msgr2=0x7f7258040ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:04.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.215+0000 7f7272a3e640 1 --2- 172.21.15.37:0/3805953110 >> v2:172.21.15.45:6812/2379918623 conn(0x7f725803e600 0x7f7258040ac0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f726c0a6e00 tx=0x7f725c0a6000 comp rx=0 tx=0).stop 2024-04-26T00:20:04.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.215+0000 7f7272a3e640 10 monclient: shutdown 2024-04-26T00:20:04.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.215+0000 7f7272a3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:04.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.215+0000 7f7272a3e640 1 -- 172.21.15.37:0/3805953110 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f726c13c3e0 msgr2=0x7f726c138f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:04.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.215+0000 7f7272a3e640 1 --2- 172.21.15.37:0/3805953110 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f726c13c3e0 0x7f726c138f70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f726400cb20 tx=0x7f726400cf20 comp rx=0 tx=0).stop 2024-04-26T00:20:04.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.215+0000 7f7272a3e640 1 -- 172.21.15.37:0/3805953110 shutdown_connections 2024-04-26T00:20:04.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.215+0000 7f7272a3e640 1 --2- 172.21.15.37:0/3805953110 >> v2:172.21.15.45:6812/2379918623 conn(0x7f725803e600 0x7f7258040ac0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:04.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.215+0000 7f7272a3e640 1 --2- 172.21.15.37:0/3805953110 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f726c13c3e0 0x7f726c138f70 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:04.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.215+0000 7f7272a3e640 1 --2- 172.21.15.37:0/3805953110 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f726c0a4170 0x7f726c136e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:04.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.215+0000 7f7272a3e640 1 --2- 172.21.15.37:0/3805953110 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f726c0a3d90 0x7f726c1cebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:04.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.215+0000 7f7272a3e640 1 -- 172.21.15.37:0/3805953110 >> 172.21.15.37:0/3805953110 conn(0x7f726c0aa0a0 msgr2=0x7f726c0a25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:04.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.215+0000 7f7272a3e640 1 -- 172.21.15.37:0/3805953110 shutdown_connections 2024-04-26T00:20:04.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.215+0000 7f7272a3e640 1 -- 172.21.15.37:0/3805953110 wait complete. 2024-04-26T00:20:04.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.223+0000 7fb54b1b8640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-26T00:20:04.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.223+0000 7fb54b1b8640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-26T00:20:04.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.223+0000 7fb54b1b8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:04.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.223+0000 7fb54b1b8640 1 -- 172.21.15.37:0/4146518997 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fb54413dd10 con 0x7fb5440a4170 2024-04-26T00:20:04.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.223+0000 7fb5417fa640 1 -- 172.21.15.37:0/4146518997 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb5300b4940 con 0x7fb5440a4170 2024-04-26T00:20:04.213 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640257 for osd.5 2024-04-26T00:20:04.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.615+0000 7fb5417fa640 1 -- 172.21.15.37:0/4146518997 <== mon.2 v2:172.21.15.37:3301/0 8 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb5300160f0 con 0x7fb5440a4170 2024-04-26T00:20:04.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.615+0000 7fb5417fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-26T00:20:04.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.615+0000 7fb5417fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:04.593 INFO:teuthology.orchestra.run.smithi037.stdout:47244640257 2024-04-26T00:20:04.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.615+0000 7fb54b1b8640 1 -- 172.21.15.37:0/4146518997 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb528035df0 msgr2=0x7fb5280382b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:04.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.615+0000 7fb54b1b8640 1 --2- 172.21.15.37:0/4146518997 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb528035df0 0x7fb5280382b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb54409c940 tx=0x7fb538009190 comp rx=0 tx=0).stop 2024-04-26T00:20:04.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.615+0000 7fb54b1b8640 10 monclient: shutdown 2024-04-26T00:20:04.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.615+0000 7fb54b1b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:04.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.615+0000 7fb54b1b8640 1 -- 172.21.15.37:0/4146518997 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5440a4170 msgr2=0x7fb5441cf580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:04.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.615+0000 7fb54b1b8640 1 --2- 172.21.15.37:0/4146518997 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5440a4170 0x7fb5441cf580 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb5300ab000 tx=0x7fb530019180 comp rx=0 tx=0).stop 2024-04-26T00:20:04.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.615+0000 7fb54b1b8640 1 -- 172.21.15.37:0/4146518997 shutdown_connections 2024-04-26T00:20:04.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.615+0000 7fb54b1b8640 1 --2- 172.21.15.37:0/4146518997 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb528035df0 0x7fb5280382b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:04.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.615+0000 7fb54b1b8640 1 --2- 172.21.15.37:0/4146518997 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb544133b20 0x7fb5441cfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:04.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.615+0000 7fb54b1b8640 1 --2- 172.21.15.37:0/4146518997 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5440a4170 0x7fb5441cf580 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:04.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.615+0000 7fb54b1b8640 1 --2- 172.21.15.37:0/4146518997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5440a3d90 0x7fb5441cf040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:04.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.615+0000 7fb54b1b8640 1 -- 172.21.15.37:0/4146518997 >> 172.21.15.37:0/4146518997 conn(0x7fb5440aa0a0 msgr2=0x7fb5440a23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:04.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.615+0000 7fb54b1b8640 1 -- 172.21.15.37:0/4146518997 shutdown_connections 2024-04-26T00:20:04.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:04.615+0000 7fb54b1b8640 1 -- 172.21.15.37:0/4146518997 wait complete. 2024-04-26T00:20:04.617 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640257 for osd.4 2024-04-26T00:20:05.006 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-26T00:20:05.060 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-26T00:20:05.082 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-26T00:20:05.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.151+0000 7ff66b0b9640 10 monclient: get_monmap_and_config 2024-04-26T00:20:05.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.151+0000 7ff66b0b9640 10 monclient: build_initial_monmap 2024-04-26T00:20:05.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.151+0000 7ff66b0b9640 10 monclient: monmap: 2024-04-26T00:20:05.137 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:05.137 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:05.137 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:05.158433+0000 2024-04-26T00:20:05.137 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:05.158433+0000 2024-04-26T00:20:05.138 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:05.138 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:05.138 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:05.138 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:05.138 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:05.138 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:05.138 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:05.138 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:05.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.151+0000 7ff66b0b9640 1 Processor -- start 2024-04-26T00:20:05.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.151+0000 7ff66b0b9640 1 -- start start 2024-04-26T00:20:05.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.151+0000 7ff66b0b9640 10 monclient: init 2024-04-26T00:20:05.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.151+0000 7ff66b0b9640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:05.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.151+0000 7ff66b0b9640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-04-26T00:20:05.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.151+0000 7ff66b0b9640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff6640a3d90 0x7ff66409bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:05.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.151+0000 7ff66b0b9640 10 monclient(hunting): picked mon.noname-c con 0x7ff6640a3d90 addr v2:172.21.15.45:3300/0 2024-04-26T00:20:05.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.151+0000 7ff66b0b9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff6640a4170 addr v1:172.21.15.45:6789/0 2024-04-26T00:20:05.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.151+0000 7ff66b0b9640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff66413c3e0 0x7ff66413e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:05.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.151+0000 7ff66b0b9640 10 monclient(hunting): picked mon.noname-a con 0x7ff66413c3e0 addr v2:172.21.15.37:3300/0 2024-04-26T00:20:05.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.151+0000 7ff66b0b9640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6640a5150 con 0x7ff6640a4170 2024-04-26T00:20:05.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.151+0000 7ff66b0b9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:05.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.151+0000 7ff66b0b9640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7ff66409b280 con 0x7ff66413c3e0 2024-04-26T00:20:05.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.151+0000 7ff66b0b9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:05.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.151+0000 7ff66b0b9640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7ff66409b560 con 0x7ff6640a3d90 2024-04-26T00:20:05.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.151+0000 7ff66b0b9640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:05.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.151+0000 7ff66b0b9640 10 monclient(hunting): authenticate will time out at 986.463789s 2024-04-26T00:20:05.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.151+0000 7ff669656640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff66413c3e0 0x7ff66413e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:05.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.155+0000 7ff669656640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff66413c3e0 0x7ff66413e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44738/0 (socket says 172.21.15.37:44738) 2024-04-26T00:20:05.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.155+0000 7ff669656640 1 -- 172.21.15.37:0/2606122658 learned_addr learned my addr 172.21.15.37:0/2606122658 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:05.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.155+0000 7ff669656640 10 monclient(hunting): get_auth_request con 0x7ff66413c3e0 auth_method 0 2024-04-26T00:20:05.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.155+0000 7ff669656640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:05.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.155+0000 7ff669656640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:05.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.155+0000 7ff669656640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:05.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.155+0000 7ff669656640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:05.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.155+0000 7ff669656640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:05.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.155+0000 7ff669656640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:05.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.155+0000 7ff668e55640 1 --2- 172.21.15.37:0/2606122658 >> v2:172.21.15.45:3300/0 conn(0x7ff6640a3d90 0x7ff66409bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:05.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.155+0000 7ff668e55640 10 monclient(hunting): get_auth_request con 0x7ff6640a3d90 auth_method 0 2024-04-26T00:20:05.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.155+0000 7ff668e55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:05.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.155+0000 7ff668e55640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:05.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.155+0000 7ff668e55640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:05.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.155+0000 7ff669656640 10 monclient(hunting): handle_auth_done global_id 4272 payload 274 2024-04-26T00:20:05.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.155+0000 7ff669656640 1 -- 172.21.15.37:0/2606122658 >> v2:172.21.15.45:3300/0 conn(0x7ff6640a3d90 msgr2=0x7ff66409bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:05.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.155+0000 7ff669656640 1 --2- 172.21.15.37:0/2606122658 >> v2:172.21.15.45:3300/0 conn(0x7ff6640a3d90 0x7ff66409bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.155+0000 7ff669656640 1 -- 172.21.15.37:0/2606122658 >> v1:172.21.15.45:6789/0 conn(0x7ff6640a4170 legacy=0x7ff6640a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:05.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.155+0000 7ff669656640 10 monclient: _finish_hunting 0 2024-04-26T00:20:05.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.155+0000 7ff669656640 1 monclient: found mon.noname-a 2024-04-26T00:20:05.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.155+0000 7ff669656640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:05.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.155+0000 7ff669656640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:05.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.155+0000 7ff669656640 1 -- 172.21.15.37:0/2606122658 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff66406dc60 con 0x7ff66413c3e0 2024-04-26T00:20:05.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff669656640 1 -- 172.21.15.37:0/2606122658 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff66413c3e0 msgr2=0x7ff66413e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:05.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff669656640 1 --2- 172.21.15.37:0/2606122658 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff66413c3e0 0x7ff66413e7d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff6640a57e0 tx=0x7ff65401c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6574714de9a51936 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:05.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff6637fe640 1 -- 172.21.15.37:0/2606122658 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6540a5530 con 0x7ff66413c3e0 2024-04-26T00:20:05.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff6637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:05.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff6637fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:05.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff6637fe640 10 monclient: dump: 2024-04-26T00:20:05.145 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:05.145 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:05.145 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.145 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.145 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:05.145 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:05.145 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:05.145 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:05.145 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:05.145 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:05.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff6637fe640 10 monclient: _finish_auth 0 2024-04-26T00:20:05.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff6637fe640 10 monclient: _check_auth_tickets 2024-04-26T00:20:05.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff6637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:05.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff66b0b9640 5 monclient: authenticate success, global_id 4272 2024-04-26T00:20:05.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff66b0b9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:20:05.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff6637fe640 1 -- 172.21.15.37:0/2606122658 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6540a56d0 con 0x7ff66413c3e0 2024-04-26T00:20:05.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff6637fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:05.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff6637fe640 1 -- 172.21.15.37:0/2606122658 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6540a5aa0 con 0x7ff66413c3e0 2024-04-26T00:20:05.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff66b0b9640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:05.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff66b0b9640 4 set_mon_vals no callback set 2024-04-26T00:20:05.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff6637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:05.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff6637fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:05.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff6637fe640 10 monclient: dump: 2024-04-26T00:20:05.146 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:05.146 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:05.146 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.146 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.147 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:05.147 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:05.147 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:05.147 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:05.147 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:05.147 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:05.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff66b0b9640 10 monclient: shutdown 2024-04-26T00:20:05.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff66b0b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:05.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff66b0b9640 1 -- 172.21.15.37:0/2606122658 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff66413c3e0 msgr2=0x7ff66413e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:05.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff66b0b9640 1 --2- 172.21.15.37:0/2606122658 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff66413c3e0 0x7ff66413e7d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff6640a57e0 tx=0x7ff65401c870 comp rx=0 tx=0).stop 2024-04-26T00:20:05.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff66b0b9640 1 -- 172.21.15.37:0/2606122658 shutdown_connections 2024-04-26T00:20:05.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff66b0b9640 1 --2- 172.21.15.37:0/2606122658 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff66413c3e0 0x7ff66413e7d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff66b0b9640 1 --2- 172.21.15.37:0/2606122658 >> v2:172.21.15.45:3300/0 conn(0x7ff6640a3d90 0x7ff66409bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff66b0b9640 1 -- 172.21.15.37:0/2606122658 >> 172.21.15.37:0/2606122658 conn(0x7ff6640aa0a0 msgr2=0x7ff6640a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:05.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff66b0b9640 1 -- 172.21.15.37:0/2606122658 shutdown_connections 2024-04-26T00:20:05.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff66b0b9640 1 -- 172.21.15.37:0/2606122658 wait complete. 2024-04-26T00:20:05.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff66b0b9640 10 monclient: build_initial_monmap 2024-04-26T00:20:05.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff66b0b9640 10 monclient: monmap: 2024-04-26T00:20:05.148 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:05.148 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:05.148 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:05.148 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:05.148 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:05.148 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:05.148 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:05.148 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:05.148 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:05.148 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:05.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff66b0b9640 1 Processor -- start 2024-04-26T00:20:05.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff66b0b9640 1 -- start start 2024-04-26T00:20:05.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff66b0b9640 10 monclient: init 2024-04-26T00:20:05.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff66b0b9640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:05.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff66b0b9640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:20:05.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff66b0b9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6640a3d90 0x7ff6641cebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:05.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff66b0b9640 10 monclient(hunting): picked mon.noname-b con 0x7ff6640a3d90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:05.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff66b0b9640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff6640a4170 0x7ff66413ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:05.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff66b0b9640 10 monclient(hunting): picked mon.noname-a con 0x7ff6640a4170 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:05.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff66b0b9640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff66413c3e0 0x7ff6641218c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:05.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff66b0b9640 10 monclient(hunting): picked mon.noname-c con 0x7ff66413c3e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:05.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff66b0b9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:05.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff66b0b9640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff6641418c0 con 0x7ff6640a4170 2024-04-26T00:20:05.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff66b0b9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:05.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff66b0b9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff664141740 con 0x7ff6640a3d90 2024-04-26T00:20:05.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff66b0b9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:05.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff66b0b9640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff664141a40 con 0x7ff66413c3e0 2024-04-26T00:20:05.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff66b0b9640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:05.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff66b0b9640 10 monclient(hunting): authenticate will time out at 986.471154s 2024-04-26T00:20:05.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff669656640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff66413c3e0 0x7ff6641218c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:05.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff663fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff6640a4170 0x7ff66413ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:05.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff669656640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff66413c3e0 0x7ff6641218c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:57930/0 (socket says 172.21.15.37:57930) 2024-04-26T00:20:05.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff669656640 1 -- 172.21.15.37:0/2982058370 learned_addr learned my addr 172.21.15.37:0/2982058370 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:05.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff669656640 10 monclient(hunting): get_auth_request con 0x7ff66413c3e0 auth_method 0 2024-04-26T00:20:05.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff669656640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:05.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff669656640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:05.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff669656640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:05.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff663fff640 10 monclient(hunting): get_auth_request con 0x7ff6640a4170 auth_method 0 2024-04-26T00:20:05.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff663fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:05.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff663fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:05.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff663fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:05.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff669656640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:05.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff669656640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:05.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff669656640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:05.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff663fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:05.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff663fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:05.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff663fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:05.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff668e55640 1 --2- 172.21.15.37:0/2982058370 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6640a3d90 0x7ff6641cebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:05.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff668e55640 10 monclient(hunting): get_auth_request con 0x7ff6640a3d90 auth_method 0 2024-04-26T00:20:05.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff668e55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:05.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff668e55640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:05.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff668e55640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:05.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff663fff640 10 monclient(hunting): handle_auth_done global_id 4275 payload 824 2024-04-26T00:20:05.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff663fff640 1 -- 172.21.15.37:0/2982058370 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff66413c3e0 msgr2=0x7ff6641218c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:05.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff663fff640 1 --2- 172.21.15.37:0/2982058370 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff66413c3e0 0x7ff6641218c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff663fff640 1 -- 172.21.15.37:0/2982058370 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6640a3d90 msgr2=0x7ff6641cebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:05.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff663fff640 1 --2- 172.21.15.37:0/2982058370 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6640a3d90 0x7ff6641cebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff663fff640 10 monclient: _finish_hunting 0 2024-04-26T00:20:05.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff663fff640 1 monclient: found mon.noname-a 2024-04-26T00:20:05.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff663fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:05.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff663fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:05.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff663fff640 1 -- 172.21.15.37:0/2982058370 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff66409ada0 con 0x7ff6640a4170 2024-04-26T00:20:05.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff669656640 1 --2- 172.21.15.37:0/2982058370 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff66413c3e0 0x7ff6641218c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:05.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.159+0000 7ff668e55640 1 --2- 172.21.15.37:0/2982058370 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6640a3d90 0x7ff6641cebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:20:05.152 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-26T00:20:05.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.163+0000 7ff663fff640 1 -- 172.21.15.37:0/2982058370 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff6640a4170 msgr2=0x7ff66413ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:05.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.163+0000 7ff663fff640 1 --2- 172.21.15.37:0/2982058370 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff6640a4170 0x7ff66413ae20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff64c00c3e0 tx=0x7ff64c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:05.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.163+0000 7ff6617fa640 1 -- 172.21.15.37:0/2982058370 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff64c0119a0 con 0x7ff6640a4170 2024-04-26T00:20:05.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.163+0000 7ff6617fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:05.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.163+0000 7ff6617fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:05.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.163+0000 7ff6617fa640 10 monclient: dump: 2024-04-26T00:20:05.155 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:05.155 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:05.155 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.155 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.156 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:05.156 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:05.156 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:05.156 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:05.156 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:05.156 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:05.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.163+0000 7ff6617fa640 10 monclient: _finish_auth 0 2024-04-26T00:20:05.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.163+0000 7ff6617fa640 10 monclient: _check_auth_tickets 2024-04-26T00:20:05.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.163+0000 7ff6617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:05.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.163+0000 7ff6617fa640 1 -- 172.21.15.37:0/2982058370 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff64c017070 con 0x7ff6640a4170 2024-04-26T00:20:05.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.163+0000 7ff66b0b9640 5 monclient: authenticate success, global_id 4275 2024-04-26T00:20:05.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.163+0000 7ff66b0b9640 10 monclient: _renew_subs 2024-04-26T00:20:05.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.163+0000 7ff66b0b9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:05.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.163+0000 7ff66b0b9640 1 -- 172.21.15.37:0/2982058370 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff66413c970 con 0x7ff6640a4170 2024-04-26T00:20:05.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.163+0000 7ff6617fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:05.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.163+0000 7ff6617fa640 1 -- 172.21.15.37:0/2982058370 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff64c011dc0 con 0x7ff6640a4170 2024-04-26T00:20:05.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.163+0000 7ff6617fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:05.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.163+0000 7ff6617fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:05.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.163+0000 7ff6617fa640 10 monclient: dump: 2024-04-26T00:20:05.157 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:05.157 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:05.157 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.157 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.157 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:05.157 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:05.157 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:05.157 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:05.157 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:05.157 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:05.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.163+0000 7ff66b0b9640 10 monclient: _renew_subs 2024-04-26T00:20:05.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.163+0000 7ff66b0b9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:05.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.163+0000 7ff66b0b9640 1 -- 172.21.15.37:0/2982058370 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6641257b0 con 0x7ff6640a4170 2024-04-26T00:20:05.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.163+0000 7ff662ffd640 4 set_mon_vals no callback set 2024-04-26T00:20:05.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.163+0000 7ff65affd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:05.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.163+0000 7ff6617fa640 1 -- 172.21.15.37:0/2982058370 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff64c01c460 con 0x7ff6640a4170 2024-04-26T00:20:05.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.163+0000 7ff65affd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:05.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.163+0000 7ff65affd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:05.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.163+0000 7ff65affd640 1 -- 172.21.15.37:0/2982058370 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff63c005090 con 0x7ff6640a4170 2024-04-26T00:20:05.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.163+0000 7ff6617fa640 1 --2- 172.21.15.37:0/2982058370 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff644035e80 0x7ff644038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:05.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.163+0000 7ff6617fa640 1 -- 172.21.15.37:0/2982058370 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7ff64c048760 con 0x7ff6640a4170 2024-04-26T00:20:05.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.167+0000 7ff668e55640 1 --2- 172.21.15.37:0/2982058370 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff644035e80 0x7ff644038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:05.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.167+0000 7ff668e55640 10 monclient: get_auth_request con 0x7ff644035e80 auth_method 0 2024-04-26T00:20:05.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.167+0000 7ff668e55640 1 --2- 172.21.15.37:0/2982058370 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff644035e80 0x7ff644038340 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff65c007a70 tx=0x7ff65c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:05.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.171+0000 7ff6617fa640 1 -- 172.21.15.37:0/2982058370 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff64c047650 con 0x7ff6640a4170 2024-04-26T00:20:05.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.171+0000 7ff6617fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:05.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.171+0000 7ff6617fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:05.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.227+0000 7f97b2655640 10 monclient: get_monmap_and_config 2024-04-26T00:20:05.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.227+0000 7f97b2655640 10 monclient: build_initial_monmap 2024-04-26T00:20:05.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.227+0000 7f97b2655640 10 monclient: monmap: 2024-04-26T00:20:05.212 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:05.212 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:05.212 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:05.235351+0000 2024-04-26T00:20:05.212 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:05.235351+0000 2024-04-26T00:20:05.212 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:05.212 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:05.212 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:05.212 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:05.212 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:05.213 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:05.213 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:05.213 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:05.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97b2655640 1 Processor -- start 2024-04-26T00:20:05.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97b2655640 1 -- start start 2024-04-26T00:20:05.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97b2655640 10 monclient: init 2024-04-26T00:20:05.213 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-26T00:20:05.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97b2655640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:05.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97b2655640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-04-26T00:20:05.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97b2655640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97ac0a3d90 0x7f97ac09bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:05.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97b2655640 10 monclient(hunting): picked mon.noname-b con 0x7f97ac0a3d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:05.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97b2655640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f97ac0a4170 0x7f97ac0a45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:05.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97b2655640 10 monclient(hunting): picked mon.noname-a con 0x7f97ac0a4170 addr v2:172.21.15.37:3300/0 2024-04-26T00:20:05.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97b2655640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f97ac133b20 0x7f97ac133f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:05.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97b2655640 10 monclient(hunting): picked mon.noname-c con 0x7f97ac133b20 addr v2:172.21.15.45:3300/0 2024-04-26T00:20:05.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97b2655640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:05.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97b2655640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f97ac0a5150 con 0x7f97ac0a4170 2024-04-26T00:20:05.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97b2655640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:05.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97b2655640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f97ac09b280 con 0x7f97ac133b20 2024-04-26T00:20:05.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97b2655640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:05.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97b2655640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f97ac09b560 con 0x7f97ac0a3d90 2024-04-26T00:20:05.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97b2655640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:05.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97b2655640 10 monclient(hunting): authenticate will time out at 986.540680s 2024-04-26T00:20:05.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97ab7fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f97ac0a4170 0x7f97ac0a45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:05.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97abfff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97ac0a3d90 0x7f97ac09bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:05.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97ab7fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f97ac0a4170 0x7f97ac0a45b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44746/0 (socket says 172.21.15.37:44746) 2024-04-26T00:20:05.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97abfff640 10 monclient(hunting): get_auth_request con 0x7f97ac0a3d90 auth_method 0 2024-04-26T00:20:05.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97ab7fe640 1 -- 172.21.15.37:0/1945143909 learned_addr learned my addr 172.21.15.37:0/1945143909 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:05.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:05.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97abfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:05.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97abfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:05.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97ab7fe640 10 monclient(hunting): get_auth_request con 0x7f97ac0a4170 auth_method 0 2024-04-26T00:20:05.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:05.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97ab7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:05.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:05.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:05.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:05.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97b0bf2640 1 --2- 172.21.15.37:0/1945143909 >> v2:172.21.15.45:3300/0 conn(0x7f97ac133b20 0x7f97ac133f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:05.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:05.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97b0bf2640 10 monclient(hunting): get_auth_request con 0x7f97ac133b20 auth_method 0 2024-04-26T00:20:05.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97b0bf2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:05.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97b0bf2640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:05.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97b0bf2640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:05.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:05.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:05.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:05.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97abfff640 10 monclient(hunting): handle_auth_done global_id 4259 payload 274 2024-04-26T00:20:05.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97abfff640 1 -- 172.21.15.37:0/1945143909 >> v2:172.21.15.45:3300/0 conn(0x7f97ac133b20 msgr2=0x7f97ac133f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:05.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97abfff640 1 --2- 172.21.15.37:0/1945143909 >> v2:172.21.15.45:3300/0 conn(0x7f97ac133b20 0x7f97ac133f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97abfff640 1 -- 172.21.15.37:0/1945143909 >> v2:172.21.15.37:3300/0 conn(0x7f97ac0a4170 msgr2=0x7f97ac0a45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:05.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97abfff640 1 --2- 172.21.15.37:0/1945143909 >> v2:172.21.15.37:3300/0 conn(0x7f97ac0a4170 0x7f97ac0a45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97abfff640 10 monclient: _finish_hunting 0 2024-04-26T00:20:05.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97abfff640 1 monclient: found mon.noname-b 2024-04-26T00:20:05.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:05.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97abfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:20:05.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.231+0000 7f97abfff640 1 -- 172.21.15.37:0/1945143909 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97ac0a4af0 con 0x7f97ac0a3d90 2024-04-26T00:20:05.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.235+0000 7f97abfff640 1 -- 172.21.15.37:0/1945143909 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97ac0a3d90 msgr2=0x7f97ac09bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:05.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.235+0000 7f97abfff640 1 --2- 172.21.15.37:0/1945143909 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97ac0a3d90 0x7f97ac09bf40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f97ac076100 tx=0x7f979c024b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cf339d454e6d446 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:05.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.235+0000 7f97aaffd640 1 -- 172.21.15.37:0/1945143909 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f979c0ba070 con 0x7f97ac0a3d90 2024-04-26T00:20:05.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.235+0000 7f97aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:05.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.235+0000 7f97aaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:20:05.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.235+0000 7f97aaffd640 10 monclient: dump: 2024-04-26T00:20:05.221 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:05.221 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:05.221 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.221 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.221 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:05.221 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:05.221 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:05.221 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:05.221 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:05.221 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:05.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.235+0000 7f97aaffd640 10 monclient: _finish_auth 0 2024-04-26T00:20:05.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.235+0000 7f97aaffd640 10 monclient: _check_auth_tickets 2024-04-26T00:20:05.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.235+0000 7f97aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:05.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.235+0000 7f97aaffd640 1 -- 172.21.15.37:0/1945143909 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f979c0ad9e0 con 0x7f97ac0a3d90 2024-04-26T00:20:05.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.235+0000 7f97aaffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:05.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.235+0000 7f97b2655640 5 monclient: authenticate success, global_id 4259 2024-04-26T00:20:05.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.235+0000 7f97aaffd640 1 -- 172.21.15.37:0/1945143909 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f979c0addb0 con 0x7f97ac0a3d90 2024-04-26T00:20:05.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.235+0000 7f97b2655640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:05.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.235+0000 7f97b2655640 4 set_mon_vals no callback set 2024-04-26T00:20:05.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.235+0000 7f97aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:05.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.235+0000 7f97aaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:05.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.235+0000 7f97aaffd640 10 monclient: dump: 2024-04-26T00:20:05.222 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:05.222 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:05.222 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.222 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.222 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:05.223 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:05.223 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:05.223 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:05.223 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:05.223 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:05.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.235+0000 7f97b2655640 10 monclient: shutdown 2024-04-26T00:20:05.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.235+0000 7f97b2655640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:05.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.235+0000 7f97b2655640 1 -- 172.21.15.37:0/1945143909 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97ac0a3d90 msgr2=0x7f97ac09bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:05.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.235+0000 7f97b2655640 1 --2- 172.21.15.37:0/1945143909 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97ac0a3d90 0x7f97ac09bf40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f97ac076100 tx=0x7f979c024b50 comp rx=0 tx=0).stop 2024-04-26T00:20:05.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.235+0000 7f97b2655640 1 -- 172.21.15.37:0/1945143909 shutdown_connections 2024-04-26T00:20:05.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.235+0000 7f97b2655640 1 --2- 172.21.15.37:0/1945143909 >> v2:172.21.15.45:3300/0 conn(0x7f97ac133b20 0x7f97ac133f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.235+0000 7f97b2655640 1 --2- 172.21.15.37:0/1945143909 >> v2:172.21.15.37:3300/0 conn(0x7f97ac0a4170 0x7f97ac0a45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.235+0000 7f97b2655640 1 --2- 172.21.15.37:0/1945143909 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97ac0a3d90 0x7f97ac09bf40 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.235+0000 7f97b2655640 1 -- 172.21.15.37:0/1945143909 >> 172.21.15.37:0/1945143909 conn(0x7f97ac0aa0a0 msgr2=0x7f97ac0a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:05.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.235+0000 7f97b2655640 1 -- 172.21.15.37:0/1945143909 shutdown_connections 2024-04-26T00:20:05.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.235+0000 7f97b2655640 1 -- 172.21.15.37:0/1945143909 wait complete. 2024-04-26T00:20:05.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.235+0000 7f97b2655640 10 monclient: build_initial_monmap 2024-04-26T00:20:05.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.235+0000 7f97b2655640 10 monclient: monmap: 2024-04-26T00:20:05.224 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:05.224 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:05.224 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:05.224 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:05.224 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:05.224 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:05.224 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:05.224 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:05.224 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:05.224 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:05.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.235+0000 7f97b2655640 1 Processor -- start 2024-04-26T00:20:05.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97b2655640 1 -- start start 2024-04-26T00:20:05.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97b2655640 10 monclient: init 2024-04-26T00:20:05.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97b2655640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:05.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97b2655640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:20:05.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97b2655640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97ac0a3d90 0x7f97ac1377a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:05.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97b2655640 10 monclient(hunting): picked mon.noname-b con 0x7f97ac0a3d90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:05.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97b2655640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f97ac0a4170 0x7f97ac132a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:05.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97b2655640 10 monclient(hunting): picked mon.noname-a con 0x7f97ac0a4170 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:05.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97b2655640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97ac133b20 0x7f97ac132f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:05.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97b2655640 10 monclient(hunting): picked mon.noname-c con 0x7f97ac133b20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:05.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97b2655640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:05.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97b2655640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f97ac070d10 con 0x7f97ac0a4170 2024-04-26T00:20:05.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97b2655640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:05.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97b2655640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f97ac09c690 con 0x7f97ac0a3d90 2024-04-26T00:20:05.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97b2655640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:05.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97b2655640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f97ac070e90 con 0x7f97ac133b20 2024-04-26T00:20:05.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97b2655640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:05.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97ab7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f97ac0a4170 0x7f97ac132a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:05.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97ab7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f97ac0a4170 0x7f97ac132a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44752/0 (socket says 172.21.15.37:44752) 2024-04-26T00:20:05.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97ab7fe640 1 -- 172.21.15.37:0/3311461583 learned_addr learned my addr 172.21.15.37:0/3311461583 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:05.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97b2655640 10 monclient(hunting): authenticate will time out at 986.548978s 2024-04-26T00:20:05.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97ab7fe640 10 monclient(hunting): get_auth_request con 0x7f97ac0a4170 auth_method 0 2024-04-26T00:20:05.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:05.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97ab7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:05.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:05.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97b0bf2640 1 --2- 172.21.15.37:0/3311461583 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97ac133b20 0x7f97ac132f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:05.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:05.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:05.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:05.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97b0bf2640 10 monclient(hunting): get_auth_request con 0x7f97ac133b20 auth_method 0 2024-04-26T00:20:05.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97b0bf2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:05.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97b0bf2640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:05.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97b0bf2640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:05.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97b0bf2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:05.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97b0bf2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:05.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97b0bf2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:05.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97ab7fe640 10 monclient(hunting): handle_auth_done global_id 4281 payload 824 2024-04-26T00:20:05.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97ab7fe640 1 -- 172.21.15.37:0/3311461583 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97ac133b20 msgr2=0x7f97ac132f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:05.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97ab7fe640 1 --2- 172.21.15.37:0/3311461583 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97ac133b20 0x7f97ac132f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97ab7fe640 1 -- 172.21.15.37:0/3311461583 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97ac0a3d90 msgr2=0x7f97ac1377a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:05.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97ab7fe640 1 --2- 172.21.15.37:0/3311461583 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97ac0a3d90 0x7f97ac1377a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97ab7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:20:05.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97ab7fe640 1 monclient: found mon.noname-a 2024-04-26T00:20:05.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:05.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97ab7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:05.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97ab7fe640 1 -- 172.21.15.37:0/3311461583 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97ac123200 con 0x7f97ac0a4170 2024-04-26T00:20:05.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97ab7fe640 1 -- 172.21.15.37:0/3311461583 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f97ac0a4170 msgr2=0x7f97ac132a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:05.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.239+0000 7f97ab7fe640 1 --2- 172.21.15.37:0/3311461583 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f97ac0a4170 0x7f97ac132a40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f97a000e890 tx=0x7f97a000ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:05.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.243+0000 7f97a8ff9640 1 -- 172.21.15.37:0/3311461583 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a0017070 con 0x7f97ac0a4170 2024-04-26T00:20:05.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.243+0000 7f97a8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:05.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.243+0000 7f97a8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:05.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.243+0000 7f97a8ff9640 10 monclient: dump: 2024-04-26T00:20:05.231 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:05.231 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:05.231 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.231 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.231 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:05.231 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:05.231 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:05.231 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:05.231 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:05.231 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:05.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.243+0000 7f97a8ff9640 10 monclient: _finish_auth 0 2024-04-26T00:20:05.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.243+0000 7f97a8ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:20:05.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.243+0000 7f97a8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:05.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.243+0000 7f97a8ff9640 1 -- 172.21.15.37:0/3311461583 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97a000cb50 con 0x7f97ac0a4170 2024-04-26T00:20:05.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.243+0000 7f97b2655640 5 monclient: authenticate success, global_id 4281 2024-04-26T00:20:05.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.243+0000 7f97b2655640 10 monclient: _renew_subs 2024-04-26T00:20:05.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.243+0000 7f97b2655640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:05.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.243+0000 7f97b2655640 1 -- 172.21.15.37:0/3311461583 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97ac1213b0 con 0x7f97ac0a4170 2024-04-26T00:20:05.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.243+0000 7f97a8ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:05.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.243+0000 7f97a8ff9640 1 -- 172.21.15.37:0/3311461583 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a0013470 con 0x7f97ac0a4170 2024-04-26T00:20:05.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.243+0000 7f97a8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:05.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.243+0000 7f97a8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:05.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.243+0000 7f97a8ff9640 10 monclient: dump: 2024-04-26T00:20:05.232 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:05.232 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:05.232 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.233 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.233 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:05.233 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:05.233 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:05.233 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:05.233 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:05.233 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:05.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.243+0000 7f97b2655640 10 monclient: _renew_subs 2024-04-26T00:20:05.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.243+0000 7f97b2655640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:05.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.243+0000 7f97b2655640 1 -- 172.21.15.37:0/3311461583 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97ac0a7810 con 0x7f97ac0a4170 2024-04-26T00:20:05.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.243+0000 7f97aa7fc640 4 set_mon_vals no callback set 2024-04-26T00:20:05.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.243+0000 7f97b2655640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:05.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.243+0000 7f97b2655640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:05.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.243+0000 7f97b2655640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:05.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.243+0000 7f97b2655640 1 -- 172.21.15.37:0/3311461583 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97ac04f510 con 0x7f97ac0a4170 2024-04-26T00:20:05.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.243+0000 7f97a8ff9640 1 -- 172.21.15.37:0/3311461583 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f97a001a070 con 0x7f97ac0a4170 2024-04-26T00:20:05.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.243+0000 7f97a8ff9640 1 --2- 172.21.15.37:0/3311461583 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9798035ee0 0x7f97980383a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:05.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.243+0000 7f97a8ff9640 1 -- 172.21.15.37:0/3311461583 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7f97a0049040 con 0x7f97ac0a4170 2024-04-26T00:20:05.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.243+0000 7f97abfff640 1 --2- 172.21.15.37:0/3311461583 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9798035ee0 0x7f97980383a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:05.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.243+0000 7f97abfff640 10 monclient: get_auth_request con 0x7f9798035ee0 auth_method 0 2024-04-26T00:20:05.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.247+0000 7f97abfff640 1 --2- 172.21.15.37:0/3311461583 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9798035ee0 0x7f97980383a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f979c0b4000 tx=0x7f979c004740 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:05.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.247+0000 7f97a8ff9640 1 -- 172.21.15.37:0/3311461583 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f97a0013610 con 0x7f97ac0a4170 2024-04-26T00:20:05.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.247+0000 7f97a8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:05.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.247+0000 7f97a8ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:05.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.251+0000 7f2287227640 10 monclient: get_monmap_and_config 2024-04-26T00:20:05.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.251+0000 7f2287227640 10 monclient: build_initial_monmap 2024-04-26T00:20:05.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.251+0000 7f2287227640 10 monclient: monmap: 2024-04-26T00:20:05.235 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:05.236 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:05.236 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:05.259535+0000 2024-04-26T00:20:05.236 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:05.259535+0000 2024-04-26T00:20:05.236 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:05.236 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:05.236 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:05.236 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:05.236 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:05.236 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:05.236 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:05.236 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:05.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f2287227640 1 Processor -- start 2024-04-26T00:20:05.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f2287227640 1 -- start start 2024-04-26T00:20:05.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f2287227640 10 monclient: init 2024-04-26T00:20:05.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f2287227640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:05.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f2287227640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-04-26T00:20:05.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f2287227640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2280131cd0 0x7f22801320b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:05.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f2287227640 10 monclient(hunting): picked mon.noname-b con 0x7f2280131cd0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:05.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f2287227640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2280073080 0x7f2280134c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:05.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f2287227640 10 monclient(hunting): picked mon.noname-c con 0x7f2280073080 addr v2:172.21.15.45:3300/0 2024-04-26T00:20:05.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f2287227640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f22800a1b60 0x7f2280135150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:05.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f2287227640 10 monclient(hunting): picked mon.noname-a con 0x7f22800a1b60 addr v2:172.21.15.37:3300/0 2024-04-26T00:20:05.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f2284fc3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2280131cd0 0x7f22801320b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:05.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f2284fc3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2280131cd0 0x7f22801320b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:57956/0 (socket says 172.21.15.37:57956) 2024-04-26T00:20:05.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f2284fc3640 1 -- 172.21.15.37:0/39766210 learned_addr learned my addr 172.21.15.37:0/39766210 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:05.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f2287227640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:05.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f2287227640 1 -- 172.21.15.37:0/39766210 --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f2280071940 con 0x7f22800a1b60 2024-04-26T00:20:05.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f2287227640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:05.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f2287227640 1 -- 172.21.15.37:0/39766210 --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f2280071660 con 0x7f2280073080 2024-04-26T00:20:05.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f2287227640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:05.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f2287227640 1 -- 172.21.15.37:0/39766210 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2280071c20 con 0x7f2280131cd0 2024-04-26T00:20:05.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f2287227640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:05.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f2287227640 10 monclient(hunting): authenticate will time out at 986.565509s 2024-04-26T00:20:05.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f2284fc3640 10 monclient(hunting): get_auth_request con 0x7f2280131cd0 auth_method 0 2024-04-26T00:20:05.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f2284fc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:05.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f2284fc3640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:05.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f2284fc3640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:05.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f22857c4640 1 --2- 172.21.15.37:0/39766210 >> v2:172.21.15.37:3300/0 conn(0x7f22800a1b60 0x7f2280135150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:05.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f22857c4640 10 monclient(hunting): get_auth_request con 0x7f22800a1b60 auth_method 0 2024-04-26T00:20:05.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f22857c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:05.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f22857c4640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:05.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f22857c4640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:05.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f227ffff640 1 --2- 172.21.15.37:0/39766210 >> v2:172.21.15.45:3300/0 conn(0x7f2280073080 0x7f2280134c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:05.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f2284fc3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:05.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f2284fc3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:05.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f2284fc3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:05.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f227ffff640 10 monclient(hunting): get_auth_request con 0x7f2280073080 auth_method 0 2024-04-26T00:20:05.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f227ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:05.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f227ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:05.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f227ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:05.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f22857c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:05.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f22857c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:05.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f22857c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:05.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f227ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:05.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f227ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:05.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f227ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:05.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f22857c4640 10 monclient(hunting): handle_auth_done global_id 4284 payload 274 2024-04-26T00:20:05.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f22857c4640 1 -- 172.21.15.37:0/39766210 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2280131cd0 msgr2=0x7f22801320b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:05.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f22857c4640 1 --2- 172.21.15.37:0/39766210 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2280131cd0 0x7f22801320b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f22857c4640 1 -- 172.21.15.37:0/39766210 >> v2:172.21.15.45:3300/0 conn(0x7f2280073080 msgr2=0x7f2280134c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:05.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f22857c4640 1 --2- 172.21.15.37:0/39766210 >> v2:172.21.15.45:3300/0 conn(0x7f2280073080 0x7f2280134c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f22857c4640 10 monclient: _finish_hunting 0 2024-04-26T00:20:05.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f22857c4640 1 monclient: found mon.noname-a 2024-04-26T00:20:05.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f22857c4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:05.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f22857c4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:05.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f22857c4640 1 -- 172.21.15.37:0/39766210 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22801325f0 con 0x7f22800a1b60 2024-04-26T00:20:05.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.255+0000 7f2284fc3640 1 --2- 172.21.15.37:0/39766210 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2280131cd0 0x7f22801320b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:05.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.259+0000 7f22857c4640 1 -- 172.21.15.37:0/39766210 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f22800a1b60 msgr2=0x7f2280135150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:05.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.259+0000 7f22857c4640 1 --2- 172.21.15.37:0/39766210 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f22800a1b60 0x7f2280135150 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f22800a4cf0 tx=0x7f227801c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1062cb03ca720d9e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:05.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.259+0000 7f227f7fe640 1 -- 172.21.15.37:0/39766210 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22780b2070 con 0x7f22800a1b60 2024-04-26T00:20:05.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.259+0000 7f227f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:05.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.259+0000 7f227f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:05.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.259+0000 7f227f7fe640 10 monclient: dump: 2024-04-26T00:20:05.245 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:05.245 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:05.245 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.245 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.245 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:05.245 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:05.245 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:05.245 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:05.246 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:05.246 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:05.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.259+0000 7f227f7fe640 10 monclient: _finish_auth 0 2024-04-26T00:20:05.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.259+0000 7f227f7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:20:05.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.259+0000 7f227f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:05.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.259+0000 7f227f7fe640 1 -- 172.21.15.37:0/39766210 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22780a5830 con 0x7f22800a1b60 2024-04-26T00:20:05.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.259+0000 7f227f7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:05.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.259+0000 7f227f7fe640 1 -- 172.21.15.37:0/39766210 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22780a5c00 con 0x7f22800a1b60 2024-04-26T00:20:05.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.259+0000 7f227f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:05.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.259+0000 7f227f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:05.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.259+0000 7f227f7fe640 10 monclient: dump: 2024-04-26T00:20:05.246 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:05.246 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:05.246 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.247 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.247 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:05.247 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:05.247 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:05.247 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:05.247 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:05.247 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:05.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.259+0000 7f2287227640 5 monclient: authenticate success, global_id 4284 2024-04-26T00:20:05.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.259+0000 7f2287227640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:05.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.259+0000 7f2287227640 4 set_mon_vals no callback set 2024-04-26T00:20:05.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.259+0000 7f2287227640 10 monclient: shutdown 2024-04-26T00:20:05.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.259+0000 7f2287227640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:05.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.259+0000 7f2287227640 1 -- 172.21.15.37:0/39766210 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f22800a1b60 msgr2=0x7f2280135150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:05.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.259+0000 7f2287227640 1 --2- 172.21.15.37:0/39766210 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f22800a1b60 0x7f2280135150 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f22800a4cf0 tx=0x7f227801c8a0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.259+0000 7f2287227640 1 -- 172.21.15.37:0/39766210 shutdown_connections 2024-04-26T00:20:05.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.259+0000 7f2287227640 1 --2- 172.21.15.37:0/39766210 >> v2:172.21.15.45:3300/0 conn(0x7f2280073080 0x7f2280134c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.259+0000 7f2287227640 1 --2- 172.21.15.37:0/39766210 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f22800a1b60 0x7f2280135150 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.259+0000 7f2287227640 1 --2- 172.21.15.37:0/39766210 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2280131cd0 0x7f22801320b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.259+0000 7f2287227640 1 -- 172.21.15.37:0/39766210 >> 172.21.15.37:0/39766210 conn(0x7f22800a7160 msgr2=0x7f22800a7540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:05.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.259+0000 7f2287227640 1 -- 172.21.15.37:0/39766210 shutdown_connections 2024-04-26T00:20:05.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.259+0000 7f2287227640 1 -- 172.21.15.37:0/39766210 wait complete. 2024-04-26T00:20:05.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f2287227640 10 monclient: build_initial_monmap 2024-04-26T00:20:05.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f2287227640 10 monclient: monmap: 2024-04-26T00:20:05.248 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:05.248 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:05.248 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:05.248 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:05.248 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:05.248 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:05.248 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:05.249 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:05.249 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:05.249 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:05.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f2287227640 1 Processor -- start 2024-04-26T00:20:05.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f2287227640 1 -- start start 2024-04-26T00:20:05.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f2287227640 10 monclient: init 2024-04-26T00:20:05.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f2287227640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:05.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f2287227640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:20:05.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f2287227640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2280073080 0x7f2280130c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:05.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f2284fc3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2280073080 0x7f2280130c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:05.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f2284fc3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2280073080 0x7f2280130c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:57972/0 (socket says 172.21.15.37:57972) 2024-04-26T00:20:05.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f2287227640 10 monclient(hunting): picked mon.noname-c con 0x7f2280073080 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:05.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f2284fc3640 1 -- 172.21.15.37:0/303866246 learned_addr learned my addr 172.21.15.37:0/303866246 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:05.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f2287227640 1 --2- 172.21.15.37:0/303866246 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22800a1b60 0x7f2280131170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:05.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f2287227640 10 monclient(hunting): picked mon.noname-b con 0x7f22800a1b60 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:05.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f2287227640 1 --2- 172.21.15.37:0/303866246 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2280131cd0 0x7f22801cce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:05.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f2287227640 10 monclient(hunting): picked mon.noname-a con 0x7f2280131cd0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:05.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f2287227640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:05.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f2287227640 1 -- 172.21.15.37:0/303866246 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f228013eaf0 con 0x7f2280131cd0 2024-04-26T00:20:05.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f2287227640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:05.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f2287227640 1 -- 172.21.15.37:0/303866246 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f228013e970 con 0x7f22800a1b60 2024-04-26T00:20:05.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f2287227640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:05.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f2287227640 1 -- 172.21.15.37:0/303866246 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f228013ec70 con 0x7f2280073080 2024-04-26T00:20:05.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f2287227640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:05.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f22857c4640 1 --2- 172.21.15.37:0/303866246 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2280131cd0 0x7f22801cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:05.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f2287227640 10 monclient(hunting): authenticate will time out at 986.575706s 2024-04-26T00:20:05.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f2284fc3640 10 monclient(hunting): get_auth_request con 0x7f2280073080 auth_method 0 2024-04-26T00:20:05.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f2284fc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:05.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f2284fc3640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:05.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f2284fc3640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:05.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f22857c4640 10 monclient(hunting): get_auth_request con 0x7f2280131cd0 auth_method 0 2024-04-26T00:20:05.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f22857c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:05.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f22857c4640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:05.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f22857c4640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:05.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f227ffff640 1 --2- 172.21.15.37:0/303866246 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22800a1b60 0x7f2280131170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:05.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f22857c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:05.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f22857c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:05.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f22857c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:05.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f227ffff640 10 monclient(hunting): get_auth_request con 0x7f22800a1b60 auth_method 0 2024-04-26T00:20:05.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f227ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:05.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f227ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:05.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.263+0000 7f227ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:05.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.267+0000 7f2284fc3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:05.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.267+0000 7f2284fc3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:05.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.267+0000 7f2284fc3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:05.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.267+0000 7f227ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:05.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.267+0000 7f227ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:05.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.267+0000 7f227ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:05.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.267+0000 7f22857c4640 10 monclient(hunting): handle_auth_done global_id 4287 payload 824 2024-04-26T00:20:05.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.267+0000 7f22857c4640 1 -- 172.21.15.37:0/303866246 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2280073080 msgr2=0x7f2280130c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:05.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.267+0000 7f22857c4640 1 --2- 172.21.15.37:0/303866246 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2280073080 0x7f2280130c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.267+0000 7f22857c4640 1 -- 172.21.15.37:0/303866246 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22800a1b60 msgr2=0x7f2280131170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:05.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.267+0000 7f22857c4640 1 --2- 172.21.15.37:0/303866246 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22800a1b60 0x7f2280131170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.267+0000 7f22857c4640 10 monclient: _finish_hunting 0 2024-04-26T00:20:05.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.267+0000 7f22857c4640 1 monclient: found mon.noname-a 2024-04-26T00:20:05.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.267+0000 7f22857c4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:05.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.267+0000 7f22857c4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:05.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.267+0000 7f22857c4640 1 -- 172.21.15.37:0/303866246 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f228013a7a0 con 0x7f2280131cd0 2024-04-26T00:20:05.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.267+0000 7f2284fc3640 1 --2- 172.21.15.37:0/303866246 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2280073080 0x7f2280130c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:05.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.267+0000 7f22857c4640 1 -- 172.21.15.37:0/303866246 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2280131cd0 msgr2=0x7f22801cce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:05.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.267+0000 7f22857c4640 1 --2- 172.21.15.37:0/303866246 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2280131cd0 0x7f22801cce00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f22780af000 tx=0x7f2278019640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:05.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.267+0000 7f227d7fa640 1 -- 172.21.15.37:0/303866246 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22780bd020 con 0x7f2280131cd0 2024-04-26T00:20:05.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.267+0000 7f227d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:05.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.267+0000 7f227d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:05.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.267+0000 7f227d7fa640 10 monclient: dump: 2024-04-26T00:20:05.259 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:05.259 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:05.259 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.259 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.259 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:05.259 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:05.259 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:05.259 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:05.259 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:05.259 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:05.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.267+0000 7f227d7fa640 10 monclient: _finish_auth 0 2024-04-26T00:20:05.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.267+0000 7f227d7fa640 10 monclient: _check_auth_tickets 2024-04-26T00:20:05.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.267+0000 7f227d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:05.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.267+0000 7f227d7fa640 1 -- 172.21.15.37:0/303866246 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22780ae990 con 0x7f2280131cd0 2024-04-26T00:20:05.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.267+0000 7f227d7fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:05.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.267+0000 7f227d7fa640 1 -- 172.21.15.37:0/303866246 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22780b2040 con 0x7f2280131cd0 2024-04-26T00:20:05.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.267+0000 7f227d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:05.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.267+0000 7f227d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:05.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.267+0000 7f227d7fa640 10 monclient: dump: 2024-04-26T00:20:05.260 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:05.260 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:05.260 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.260 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.260 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:05.260 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:05.261 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:05.261 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:05.261 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:05.261 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:05.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.267+0000 7f227effd640 4 set_mon_vals no callback set 2024-04-26T00:20:05.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.267+0000 7f2287227640 5 monclient: authenticate success, global_id 4287 2024-04-26T00:20:05.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.267+0000 7f2287227640 10 monclient: _renew_subs 2024-04-26T00:20:05.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.267+0000 7f2287227640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:05.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.267+0000 7f2287227640 1 -- 172.21.15.37:0/303866246 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22800a6220 con 0x7f2280131cd0 2024-04-26T00:20:05.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.267+0000 7f2287227640 10 monclient: _renew_subs 2024-04-26T00:20:05.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.267+0000 7f2287227640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:05.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.267+0000 7f2287227640 1 -- 172.21.15.37:0/303866246 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f228013a4c0 con 0x7f2280131cd0 2024-04-26T00:20:05.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.271+0000 7f227d7fa640 1 -- 172.21.15.37:0/303866246 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f22780c0020 con 0x7f2280131cd0 2024-04-26T00:20:05.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.271+0000 7f2287227640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:05.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.271+0000 7f227d7fa640 1 --2- 172.21.15.37:0/303866246 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2264035ed0 0x7f2264038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:05.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.271+0000 7f2287227640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:05.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.271+0000 7f2287227640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:05.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.271+0000 7f2287227640 1 -- 172.21.15.37:0/303866246 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f228004f510 con 0x7f2280131cd0 2024-04-26T00:20:05.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.271+0000 7f227d7fa640 1 -- 172.21.15.37:0/303866246 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7f22780e8120 con 0x7f2280131cd0 2024-04-26T00:20:05.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.271+0000 7f2284fc3640 1 --2- 172.21.15.37:0/303866246 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2264035ed0 0x7f2264038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:05.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.275+0000 7f2284fc3640 10 monclient: get_auth_request con 0x7f2264035ed0 auth_method 0 2024-04-26T00:20:05.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.275+0000 7f227d7fa640 1 -- 172.21.15.37:0/303866246 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f22781155c0 con 0x7f2280131cd0 2024-04-26T00:20:05.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.275+0000 7f227d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:05.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.275+0000 7f227d7fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:05.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.287+0000 7f2284fc3640 1 --2- 172.21.15.37:0/303866246 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2264035ed0 0x7f2264038390 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2274009630 tx=0x7f2274009240 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:05.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.287+0000 7fed5f36f640 10 monclient: get_monmap_and_config 2024-04-26T00:20:05.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.287+0000 7fed5f36f640 10 monclient: build_initial_monmap 2024-04-26T00:20:05.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.287+0000 7fed5f36f640 10 monclient: monmap: 2024-04-26T00:20:05.270 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:05.270 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:05.271 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:05.294396+0000 2024-04-26T00:20:05.271 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:05.294396+0000 2024-04-26T00:20:05.271 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:05.271 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:05.271 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:05.271 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:05.271 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:05.271 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:05.271 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:05.271 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:05.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.287+0000 7fed5f36f640 1 Processor -- start 2024-04-26T00:20:05.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.287+0000 7fed5f36f640 1 -- start start 2024-04-26T00:20:05.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.287+0000 7fed5f36f640 10 monclient: init 2024-04-26T00:20:05.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.287+0000 7fed5f36f640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:05.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.287+0000 7fed5f36f640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-04-26T00:20:05.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.287+0000 7fed5f36f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fed580a3d90 addr v1:172.21.15.45:6789/0 2024-04-26T00:20:05.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.287+0000 7fed5f36f640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fed580a4170 0x7fed580a45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:05.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.287+0000 7fed5f36f640 10 monclient(hunting): picked mon.noname-c con 0x7fed580a4170 addr v2:172.21.15.45:3300/0 2024-04-26T00:20:05.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.287+0000 7fed5f36f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed5813c3e0 0x7fed5813e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:05.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.287+0000 7fed5f36f640 10 monclient(hunting): picked mon.noname-b con 0x7fed5813c3e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:05.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.287+0000 7fed5f36f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed580a5150 con 0x7fed580a3d90 2024-04-26T00:20:05.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.287+0000 7fed5f36f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:05.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.287+0000 7fed5f36f640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fed5809c650 con 0x7fed580a4170 2024-04-26T00:20:05.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.287+0000 7fed5f36f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:05.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.287+0000 7fed5f36f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fed5809b280 con 0x7fed5813c3e0 2024-04-26T00:20:05.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.287+0000 7fed5f36f640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:05.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.287+0000 7fed5f36f640 10 monclient(hunting): authenticate will time out at 986.599781s 2024-04-26T00:20:05.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.287+0000 7fed5d90c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed5813c3e0 0x7fed5813e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:05.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.287+0000 7fed5d90c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed5813c3e0 0x7fed5813e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:57974/0 (socket says 172.21.15.37:57974) 2024-04-26T00:20:05.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.287+0000 7fed5d90c640 1 -- 172.21.15.37:0/757176830 learned_addr learned my addr 172.21.15.37:0/757176830 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:05.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.287+0000 7fed5d90c640 10 monclient(hunting): get_auth_request con 0x7fed5813c3e0 auth_method 0 2024-04-26T00:20:05.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.291+0000 7fed5d90c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:05.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.291+0000 7fed5d90c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:05.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.291+0000 7fed5d90c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:05.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.291+0000 7fed5c90a640 1 --2- 172.21.15.37:0/757176830 >> v2:172.21.15.45:3300/0 conn(0x7fed580a4170 0x7fed580a45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:05.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.291+0000 7fed5d90c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:05.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.291+0000 7fed5d90c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:05.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.291+0000 7fed5d90c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:05.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.291+0000 7fed5c90a640 10 monclient(hunting): get_auth_request con 0x7fed580a4170 auth_method 0 2024-04-26T00:20:05.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.291+0000 7fed5c90a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:05.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.291+0000 7fed5c90a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:05.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.291+0000 7fed5c90a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:05.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.291+0000 7fed5d90c640 10 monclient(hunting): handle_auth_done global_id 4271 payload 274 2024-04-26T00:20:05.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.291+0000 7fed5d90c640 1 -- 172.21.15.37:0/757176830 >> v2:172.21.15.45:3300/0 conn(0x7fed580a4170 msgr2=0x7fed580a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:05.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.291+0000 7fed5d90c640 1 --2- 172.21.15.37:0/757176830 >> v2:172.21.15.45:3300/0 conn(0x7fed580a4170 0x7fed580a45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.291+0000 7fed5d90c640 1 -- 172.21.15.37:0/757176830 >> v1:172.21.15.45:6789/0 conn(0x7fed580a3d90 legacy=0x7fed5809bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:05.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.291+0000 7fed5d90c640 10 monclient: _finish_hunting 0 2024-04-26T00:20:05.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.291+0000 7fed5d90c640 1 monclient: found mon.noname-b 2024-04-26T00:20:05.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.291+0000 7fed5d90c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:05.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.291+0000 7fed5d90c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:20:05.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.291+0000 7fed5d90c640 1 -- 172.21.15.37:0/757176830 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed5809b560 con 0x7fed5813c3e0 2024-04-26T00:20:05.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5d90c640 1 -- 172.21.15.37:0/757176830 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed5813c3e0 msgr2=0x7fed5813e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:05.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5d90c640 1 --2- 172.21.15.37:0/757176830 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed5813c3e0 0x7fed5813e7d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fed50009e50 tx=0x7fed5001c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=209709a9b59b2c39 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:05.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed4ffff640 1 -- 172.21.15.37:0/757176830 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed500a5640 con 0x7fed5813c3e0 2024-04-26T00:20:05.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:05.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed4ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:20:05.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed4ffff640 10 monclient: dump: 2024-04-26T00:20:05.280 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:05.280 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:05.280 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.280 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.280 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:05.280 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:05.280 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:05.280 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:05.280 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:05.280 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:05.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed4ffff640 10 monclient: _finish_auth 0 2024-04-26T00:20:05.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed4ffff640 10 monclient: _check_auth_tickets 2024-04-26T00:20:05.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:05.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed4ffff640 1 -- 172.21.15.37:0/757176830 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed500a57e0 con 0x7fed5813c3e0 2024-04-26T00:20:05.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5f36f640 5 monclient: authenticate success, global_id 4271 2024-04-26T00:20:05.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5f36f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:20:05.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed4ffff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:05.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5f36f640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:05.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed4ffff640 1 -- 172.21.15.37:0/757176830 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed500a5bb0 con 0x7fed5813c3e0 2024-04-26T00:20:05.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5f36f640 4 set_mon_vals no callback set 2024-04-26T00:20:05.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:05.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed4ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:05.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed4ffff640 10 monclient: dump: 2024-04-26T00:20:05.281 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:05.282 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:05.282 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.282 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.282 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:05.282 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:05.282 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:05.282 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:05.282 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:05.282 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:05.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5f36f640 10 monclient: shutdown 2024-04-26T00:20:05.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5f36f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:05.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5f36f640 1 -- 172.21.15.37:0/757176830 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed5813c3e0 msgr2=0x7fed5813e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:05.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5f36f640 1 --2- 172.21.15.37:0/757176830 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed5813c3e0 0x7fed5813e7d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fed50009e50 tx=0x7fed5001c8b0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5f36f640 1 -- 172.21.15.37:0/757176830 shutdown_connections 2024-04-26T00:20:05.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5f36f640 1 --2- 172.21.15.37:0/757176830 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed5813c3e0 0x7fed5813e7d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5f36f640 1 --2- 172.21.15.37:0/757176830 >> v2:172.21.15.45:3300/0 conn(0x7fed580a4170 0x7fed580a45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5f36f640 1 -- 172.21.15.37:0/757176830 >> 172.21.15.37:0/757176830 conn(0x7fed580aa0a0 msgr2=0x7fed580a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:05.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5f36f640 1 -- 172.21.15.37:0/757176830 shutdown_connections 2024-04-26T00:20:05.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5f36f640 1 -- 172.21.15.37:0/757176830 wait complete. 2024-04-26T00:20:05.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5f36f640 10 monclient: build_initial_monmap 2024-04-26T00:20:05.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5f36f640 10 monclient: monmap: 2024-04-26T00:20:05.283 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:05.283 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:05.283 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:05.283 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:05.283 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:05.283 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:05.284 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:05.284 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:05.284 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:05.284 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:05.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5f36f640 1 Processor -- start 2024-04-26T00:20:05.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5f36f640 1 -- start start 2024-04-26T00:20:05.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5f36f640 10 monclient: init 2024-04-26T00:20:05.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5f36f640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:05.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5f36f640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:20:05.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5f36f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed580a3d90 0x7fed58139d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:05.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5f36f640 10 monclient(hunting): picked mon.noname-b con 0x7fed580a3d90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:05.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5f36f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed580a4170 0x7fed5813a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:05.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5f36f640 10 monclient(hunting): picked mon.noname-c con 0x7fed580a4170 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:05.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5f36f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed5813c3e0 0x7fed5813a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:05.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5f36f640 10 monclient(hunting): picked mon.noname-a con 0x7fed5813c3e0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:05.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5f36f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:05.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5f36f640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fed581418c0 con 0x7fed5813c3e0 2024-04-26T00:20:05.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5f36f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:05.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5f36f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fed58141740 con 0x7fed580a3d90 2024-04-26T00:20:05.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5f36f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:05.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5f36f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fed58141a40 con 0x7fed580a4170 2024-04-26T00:20:05.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5f36f640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:05.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5f36f640 10 monclient(hunting): authenticate will time out at 986.606998s 2024-04-26T00:20:05.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5d90c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed5813c3e0 0x7fed5813a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:05.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5c90a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed580a4170 0x7fed5813a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:05.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5d90c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed5813c3e0 0x7fed5813a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44780/0 (socket says 172.21.15.37:44780) 2024-04-26T00:20:05.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5d90c640 1 -- 172.21.15.37:0/2943998711 learned_addr learned my addr 172.21.15.37:0/2943998711 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:05.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5d90c640 10 monclient(hunting): get_auth_request con 0x7fed5813c3e0 auth_method 0 2024-04-26T00:20:05.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5d90c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:05.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5d90c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:05.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5d90c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:05.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5c90a640 10 monclient(hunting): get_auth_request con 0x7fed580a4170 auth_method 0 2024-04-26T00:20:05.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5c90a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:05.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5c90a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:05.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5c90a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:05.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5d10b640 1 --2- 172.21.15.37:0/2943998711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed580a3d90 0x7fed58139d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:05.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5c90a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:05.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5c90a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:05.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5c90a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:05.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5d10b640 10 monclient(hunting): get_auth_request con 0x7fed580a3d90 auth_method 0 2024-04-26T00:20:05.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5d10b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:05.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5d10b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:05.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5d10b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:05.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5d90c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:05.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5d90c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:05.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5d90c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:05.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5d10b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:05.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5d10b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:05.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5d10b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:05.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5c90a640 10 monclient(hunting): handle_auth_done global_id 4274 payload 824 2024-04-26T00:20:05.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5c90a640 1 -- 172.21.15.37:0/2943998711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed580a3d90 msgr2=0x7fed58139d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:05.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5c90a640 1 --2- 172.21.15.37:0/2943998711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed580a3d90 0x7fed58139d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5c90a640 1 -- 172.21.15.37:0/2943998711 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed5813c3e0 msgr2=0x7fed5813a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:05.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5c90a640 1 --2- 172.21.15.37:0/2943998711 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed5813c3e0 0x7fed5813a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5c90a640 10 monclient: _finish_hunting 0 2024-04-26T00:20:05.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5c90a640 1 monclient: found mon.noname-c 2024-04-26T00:20:05.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5c90a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:05.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5c90a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:20:05.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.295+0000 7fed5c90a640 1 -- 172.21.15.37:0/2943998711 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed58133670 con 0x7fed580a4170 2024-04-26T00:20:05.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.299+0000 7fed5c90a640 1 -- 172.21.15.37:0/2943998711 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed580a4170 msgr2=0x7fed5813a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:05.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.299+0000 7fed5c90a640 1 --2- 172.21.15.37:0/2943998711 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed580a4170 0x7fed5813a280 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fed4800d620 tx=0x7fed4800dae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:05.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.299+0000 7fed4dffb640 1 -- 172.21.15.37:0/2943998711 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed48018020 con 0x7fed580a4170 2024-04-26T00:20:05.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.299+0000 7fed4dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:05.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.299+0000 7fed4dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:20:05.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.299+0000 7fed4dffb640 10 monclient: dump: 2024-04-26T00:20:05.291 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:05.291 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:05.291 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.291 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.291 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:05.291 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:05.291 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:05.291 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:05.291 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:05.291 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:05.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.299+0000 7fed4dffb640 10 monclient: _finish_auth 0 2024-04-26T00:20:05.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.299+0000 7fed4dffb640 10 monclient: _check_auth_tickets 2024-04-26T00:20:05.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.299+0000 7fed4dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:05.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.299+0000 7fed4dffb640 1 -- 172.21.15.37:0/2943998711 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed48014830 con 0x7fed580a4170 2024-04-26T00:20:05.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.299+0000 7fed5f36f640 5 monclient: authenticate success, global_id 4274 2024-04-26T00:20:05.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.299+0000 7fed5f36f640 10 monclient: _renew_subs 2024-04-26T00:20:05.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.299+0000 7fed5f36f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:05.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.299+0000 7fed5f36f640 1 -- 172.21.15.37:0/2943998711 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed58132150 con 0x7fed580a4170 2024-04-26T00:20:05.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.299+0000 7fed4dffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:05.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.299+0000 7fed4dffb640 1 -- 172.21.15.37:0/2943998711 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed48014c20 con 0x7fed580a4170 2024-04-26T00:20:05.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.299+0000 7fed4f7fe640 4 set_mon_vals no callback set 2024-04-26T00:20:05.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.299+0000 7fed4dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:05.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.299+0000 7fed4dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:05.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.299+0000 7fed4dffb640 10 monclient: dump: 2024-04-26T00:20:05.292 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:05.292 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:05.293 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.293 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.293 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:05.293 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:05.293 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:05.293 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:05.293 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:05.293 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:05.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.299+0000 7fed5f36f640 10 monclient: _renew_subs 2024-04-26T00:20:05.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.299+0000 7fed5f36f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:05.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.299+0000 7fed5f36f640 1 -- 172.21.15.37:0/2943998711 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fed58137910 con 0x7fed580a4170 2024-04-26T00:20:05.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.299+0000 7fed4dffb640 1 -- 172.21.15.37:0/2943998711 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fed4801b020 con 0x7fed580a4170 2024-04-26T00:20:05.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.299+0000 7fed437fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:05.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.299+0000 7fed437fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:05.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.299+0000 7fed437fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:05.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.299+0000 7fed437fe640 1 -- 172.21.15.37:0/2943998711 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed30005090 con 0x7fed580a4170 2024-04-26T00:20:05.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.303+0000 7fed4dffb640 1 --2- 172.21.15.37:0/2943998711 >> v2:172.21.15.45:6812/2379918623 conn(0x7fed3c035ed0 0x7fed3c038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:05.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.303+0000 7fed4dffb640 1 -- 172.21.15.37:0/2943998711 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7fed480494b0 con 0x7fed580a4170 2024-04-26T00:20:05.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.303+0000 7fed5d10b640 1 --2- 172.21.15.37:0/2943998711 >> v2:172.21.15.45:6812/2379918623 conn(0x7fed3c035ed0 0x7fed3c038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:05.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.303+0000 7fed5d10b640 10 monclient: get_auth_request con 0x7fed3c035ed0 auth_method 0 2024-04-26T00:20:05.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.303+0000 7fed5d10b640 1 --2- 172.21.15.37:0/2943998711 >> v2:172.21.15.45:6812/2379918623 conn(0x7fed3c035ed0 0x7fed3c038390 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fed54007fd0 tx=0x7fed54009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:05.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.307+0000 7fed4dffb640 1 -- 172.21.15.37:0/2943998711 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fed48051020 con 0x7fed580a4170 2024-04-26T00:20:05.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.307+0000 7fed4dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:05.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.307+0000 7fed4dffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:05.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.395+0000 7f6fe639c640 10 monclient: get_monmap_and_config 2024-04-26T00:20:05.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.395+0000 7f6fe639c640 10 monclient: build_initial_monmap 2024-04-26T00:20:05.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.395+0000 7f6fe639c640 10 monclient: monmap: 2024-04-26T00:20:05.377 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:05.377 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:05.377 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:05.401013+0000 2024-04-26T00:20:05.378 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:05.401013+0000 2024-04-26T00:20:05.378 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:05.378 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:05.378 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:05.378 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:05.378 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:05.378 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:05.378 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:05.378 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:05.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.395+0000 7f6fe639c640 1 Processor -- start 2024-04-26T00:20:05.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.395+0000 7f6fe639c640 1 -- start start 2024-04-26T00:20:05.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.395+0000 7f6fe639c640 10 monclient: init 2024-04-26T00:20:05.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.395+0000 7f6fe639c640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:05.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.395+0000 7f6fe639c640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-04-26T00:20:05.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.395+0000 7f6fe639c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6fe00a3d90 addr v1:172.21.15.37:6789/0 2024-04-26T00:20:05.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.395+0000 7f6fe639c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6fe00a4170 addr v1:172.21.15.45:6789/0 2024-04-26T00:20:05.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.395+0000 7f6fe639c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6fe013d3f0 0x7f6fe013f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:05.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.395+0000 7f6fe639c640 10 monclient(hunting): picked mon.noname-a con 0x7f6fe013d3f0 addr v2:172.21.15.37:3300/0 2024-04-26T00:20:05.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.395+0000 7f6fe639c640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fe00a5150 con 0x7f6fe00a3d90 2024-04-26T00:20:05.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.395+0000 7f6fe639c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fe009b280 con 0x7f6fe00a4170 2024-04-26T00:20:05.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.395+0000 7f6fe639c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:05.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.395+0000 7f6fe639c640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f6fe009c650 con 0x7f6fe013d3f0 2024-04-26T00:20:05.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.395+0000 7f6fe639c640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:05.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.395+0000 7f6fe639c640 10 monclient(hunting): authenticate will time out at 986.706736s 2024-04-26T00:20:05.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.395+0000 7f6fdffff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f6fe00a3d90 0x7f6fe009bf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:55808/0 (socket says 172.21.15.37:55808) 2024-04-26T00:20:05.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.395+0000 7f6fdffff640 1 -- 172.21.15.37:0/540904283 learned_addr learned my addr 172.21.15.37:0/540904283 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:20:05.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.395+0000 7f6fe4939640 1 --2- 172.21.15.37:0/540904283 >> v2:172.21.15.37:3300/0 conn(0x7f6fe013d3f0 0x7f6fe013f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:05.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.395+0000 7f6fe4939640 10 monclient(hunting): get_auth_request con 0x7f6fe013d3f0 auth_method 0 2024-04-26T00:20:05.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.395+0000 7f6fe4939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:05.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.395+0000 7f6fe4939640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:05.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.395+0000 7f6fe4939640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:05.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.395+0000 7f6fe4939640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:05.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.395+0000 7f6fe4939640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:05.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.395+0000 7f6fe4939640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:05.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.395+0000 7f6fe4939640 10 monclient(hunting): handle_auth_done global_id 4293 payload 274 2024-04-26T00:20:05.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.395+0000 7f6fe4939640 1 -- 172.21.15.37:0/540904283 >> v1:172.21.15.45:6789/0 conn(0x7f6fe00a4170 legacy=0x7f6fe00a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:05.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.395+0000 7f6fe4939640 1 -- 172.21.15.37:0/540904283 >> v1:172.21.15.37:6789/0 conn(0x7f6fe00a3d90 legacy=0x7f6fe009bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:05.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.395+0000 7f6fe4939640 10 monclient: _finish_hunting 0 2024-04-26T00:20:05.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.395+0000 7f6fe4939640 1 monclient: found mon.noname-a 2024-04-26T00:20:05.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.395+0000 7f6fe4939640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:05.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.395+0000 7f6fe4939640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:05.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.395+0000 7f6fe4939640 1 -- 172.21.15.37:0/540904283 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fe009b560 con 0x7f6fe013d3f0 2024-04-26T00:20:05.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.399+0000 7f6fe4939640 1 -- 172.21.15.37:0/540904283 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6fe013d3f0 msgr2=0x7f6fe013f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:05.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.399+0000 7f6fe4939640 1 --2- 172.21.15.37:0/540904283 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6fe013d3f0 0x7f6fe013f7e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6fd8009d90 tx=0x7f6fd801c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=775000686d575a56 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:05.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.399+0000 7f6fdeffd640 1 -- 172.21.15.37:0/540904283 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd80b2070 con 0x7f6fe013d3f0 2024-04-26T00:20:05.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.399+0000 7f6fdeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:05.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.399+0000 7f6fdeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:05.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.399+0000 7f6fdeffd640 10 monclient: dump: 2024-04-26T00:20:05.383 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:05.383 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:05.384 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.384 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.384 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:05.384 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:05.384 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:05.384 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:05.384 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:05.384 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:05.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.399+0000 7f6fdeffd640 10 monclient: _finish_auth 0 2024-04-26T00:20:05.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.399+0000 7f6fdeffd640 10 monclient: _check_auth_tickets 2024-04-26T00:20:05.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.399+0000 7f6fdeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:05.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.399+0000 7f6fdeffd640 1 -- 172.21.15.37:0/540904283 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fd80a5860 con 0x7f6fe013d3f0 2024-04-26T00:20:05.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.399+0000 7f6fdeffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:05.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.399+0000 7f6fdeffd640 1 -- 172.21.15.37:0/540904283 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd80a5c30 con 0x7f6fe013d3f0 2024-04-26T00:20:05.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.399+0000 7f6fdeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:05.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.399+0000 7f6fdeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:05.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.399+0000 7f6fdeffd640 10 monclient: dump: 2024-04-26T00:20:05.385 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:05.385 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:05.385 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.385 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.385 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:05.385 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:05.385 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:05.385 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:05.385 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:05.385 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:05.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.399+0000 7f6fe639c640 5 monclient: authenticate success, global_id 4293 2024-04-26T00:20:05.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.399+0000 7f6fe639c640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:05.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.399+0000 7f6fe639c640 4 set_mon_vals no callback set 2024-04-26T00:20:05.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe639c640 10 monclient: shutdown 2024-04-26T00:20:05.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe639c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:05.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe639c640 1 -- 172.21.15.37:0/540904283 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6fe013d3f0 msgr2=0x7f6fe013f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:05.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe639c640 1 --2- 172.21.15.37:0/540904283 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6fe013d3f0 0x7f6fe013f7e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6fd8009d90 tx=0x7f6fd801c7b0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe639c640 1 -- 172.21.15.37:0/540904283 shutdown_connections 2024-04-26T00:20:05.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe639c640 1 --2- 172.21.15.37:0/540904283 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6fe013d3f0 0x7f6fe013f7e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe639c640 1 -- 172.21.15.37:0/540904283 >> 172.21.15.37:0/540904283 conn(0x7f6fe00aa0a0 msgr2=0x7f6fe00a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:05.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe639c640 1 -- 172.21.15.37:0/540904283 shutdown_connections 2024-04-26T00:20:05.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe639c640 1 -- 172.21.15.37:0/540904283 wait complete. 2024-04-26T00:20:05.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe639c640 10 monclient: build_initial_monmap 2024-04-26T00:20:05.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe639c640 10 monclient: monmap: 2024-04-26T00:20:05.389 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:05.390 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:05.390 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:05.390 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:05.390 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:05.390 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:05.390 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:05.390 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:05.390 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:05.390 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:05.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe639c640 1 Processor -- start 2024-04-26T00:20:05.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe639c640 1 -- start start 2024-04-26T00:20:05.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe639c640 10 monclient: init 2024-04-26T00:20:05.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe639c640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:05.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe639c640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:20:05.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe639c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6fe00a3d90 0x7f6fe01367b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:05.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe639c640 10 monclient(hunting): picked mon.noname-a con 0x7f6fe00a3d90 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:05.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe639c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fe00a4170 0x7f6fe0136cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:05.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe639c640 10 monclient(hunting): picked mon.noname-b con 0x7f6fe00a4170 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:05.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe639c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6fe013d3f0 0x7f6fe0139d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:05.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe639c640 10 monclient(hunting): picked mon.noname-c con 0x7f6fe013d3f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:05.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe639c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:05.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe639c640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe0142ef0 con 0x7f6fe00a3d90 2024-04-26T00:20:05.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe639c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:05.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe639c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe0142d70 con 0x7f6fe00a4170 2024-04-26T00:20:05.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe639c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:05.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe639c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f6fe0143070 con 0x7f6fe013d3f0 2024-04-26T00:20:05.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe639c640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:05.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe639c640 10 monclient(hunting): authenticate will time out at 986.713794s 2024-04-26T00:20:05.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fdffff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6fe00a3d90 0x7f6fe01367b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:05.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fdffff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6fe00a3d90 0x7f6fe01367b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44788/0 (socket says 172.21.15.37:44788) 2024-04-26T00:20:05.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fdffff640 1 -- 172.21.15.37:0/54874506 learned_addr learned my addr 172.21.15.37:0/54874506 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:05.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fdffff640 10 monclient(hunting): get_auth_request con 0x7f6fe00a3d90 auth_method 0 2024-04-26T00:20:05.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fdffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:05.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fdffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:05.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fdffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:05.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe4939640 1 --2- 172.21.15.37:0/54874506 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6fe013d3f0 0x7f6fe0139d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:05.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe4939640 10 monclient(hunting): get_auth_request con 0x7f6fe013d3f0 auth_method 0 2024-04-26T00:20:05.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe4939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:05.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe4939640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:05.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe4939640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:05.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fdffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:05.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fdffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:05.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fdf7fe640 1 --2- 172.21.15.37:0/54874506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fe00a4170 0x7f6fe0136cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:05.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fdffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:05.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe4939640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:05.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe4939640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:05.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe4939640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:05.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fdf7fe640 10 monclient(hunting): get_auth_request con 0x7f6fe00a4170 auth_method 0 2024-04-26T00:20:05.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fdf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:05.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fdf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:05.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:05.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe4939640 10 monclient(hunting): handle_auth_done global_id 4277 payload 824 2024-04-26T00:20:05.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe4939640 1 -- 172.21.15.37:0/54874506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fe00a4170 msgr2=0x7f6fe0136cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:05.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe4939640 1 --2- 172.21.15.37:0/54874506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fe00a4170 0x7f6fe0136cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe4939640 1 -- 172.21.15.37:0/54874506 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6fe00a3d90 msgr2=0x7f6fe01367b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:05.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe4939640 1 --2- 172.21.15.37:0/54874506 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6fe00a3d90 0x7f6fe01367b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe4939640 10 monclient: _finish_hunting 0 2024-04-26T00:20:05.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe4939640 1 monclient: found mon.noname-c 2024-04-26T00:20:05.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe4939640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:05.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe4939640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:20:05.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fe4939640 1 -- 172.21.15.37:0/54874506 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fe009ada0 con 0x7f6fe013d3f0 2024-04-26T00:20:05.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.403+0000 7f6fdffff640 1 --2- 172.21.15.37:0/54874506 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6fe00a3d90 0x7f6fe01367b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:05.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.407+0000 7f6fe4939640 1 -- 172.21.15.37:0/54874506 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6fe013d3f0 msgr2=0x7f6fe0139d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:05.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.407+0000 7f6fe4939640 1 --2- 172.21.15.37:0/54874506 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6fe013d3f0 0x7f6fe0139d20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6fd80af000 tx=0x7f6fd8019550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:05.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.407+0000 7f6fdcff9640 1 -- 172.21.15.37:0/54874506 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd80bd020 con 0x7f6fe013d3f0 2024-04-26T00:20:05.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.407+0000 7f6fdcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:05.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.407+0000 7f6fdcff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:20:05.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.407+0000 7f6fdcff9640 10 monclient: dump: 2024-04-26T00:20:05.395 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:05.395 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:05.395 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.395 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.395 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:05.395 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:05.395 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:05.395 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:05.395 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:05.395 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:05.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.407+0000 7f6fdcff9640 10 monclient: _finish_auth 0 2024-04-26T00:20:05.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.407+0000 7f6fdcff9640 10 monclient: _check_auth_tickets 2024-04-26T00:20:05.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.407+0000 7f6fdcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:05.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.407+0000 7f6fdcff9640 1 -- 172.21.15.37:0/54874506 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fd80b3040 con 0x7f6fe013d3f0 2024-04-26T00:20:05.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.407+0000 7f6fdcff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:05.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.407+0000 7f6fdcff9640 1 -- 172.21.15.37:0/54874506 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd80b2070 con 0x7f6fe013d3f0 2024-04-26T00:20:05.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.407+0000 7f6fdcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:05.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.407+0000 7f6fdcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:05.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.407+0000 7f6fdcff9640 10 monclient: dump: 2024-04-26T00:20:05.396 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:05.396 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:05.396 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.396 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.396 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:05.396 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:05.396 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:05.396 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:05.396 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:05.397 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:05.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.407+0000 7f6fde7fc640 4 set_mon_vals no callback set 2024-04-26T00:20:05.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.407+0000 7f6fe639c640 5 monclient: authenticate success, global_id 4277 2024-04-26T00:20:05.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.407+0000 7f6fe639c640 10 monclient: _renew_subs 2024-04-26T00:20:05.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.407+0000 7f6fe639c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:05.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.407+0000 7f6fe639c640 1 -- 172.21.15.37:0/54874506 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fe013d840 con 0x7f6fe013d3f0 2024-04-26T00:20:05.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.407+0000 7f6fe639c640 10 monclient: _renew_subs 2024-04-26T00:20:05.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.407+0000 7f6fe639c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:05.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.407+0000 7f6fe639c640 1 -- 172.21.15.37:0/54874506 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fe013c6a0 con 0x7f6fe013d3f0 2024-04-26T00:20:05.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.407+0000 7f6fdcff9640 1 -- 172.21.15.37:0/54874506 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f6fd80c0020 con 0x7f6fe013d3f0 2024-04-26T00:20:05.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.407+0000 7f6fdcff9640 1 --2- 172.21.15.37:0/54874506 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6fcc035f00 0x7f6fcc0383c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:05.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.407+0000 7f6fdcff9640 1 -- 172.21.15.37:0/54874506 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7f6fd80e3500 con 0x7f6fe013d3f0 2024-04-26T00:20:05.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.407+0000 7f6fe639c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:05.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.407+0000 7f6fe639c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:05.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.407+0000 7f6fe639c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:05.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.407+0000 7f6fe639c640 1 -- 172.21.15.37:0/54874506 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fe004f510 con 0x7f6fe013d3f0 2024-04-26T00:20:05.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.407+0000 7f6fdffff640 1 --2- 172.21.15.37:0/54874506 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6fcc035f00 0x7f6fcc0383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:05.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.407+0000 7f6fdffff640 10 monclient: get_auth_request con 0x7f6fcc035f00 auth_method 0 2024-04-26T00:20:05.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.411+0000 7f6fdffff640 1 --2- 172.21.15.37:0/54874506 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6fcc035f00 0x7f6fcc0383c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0007d60 tx=0x7f6fd0007cb0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:05.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.415+0000 7f6fdcff9640 1 -- 172.21.15.37:0/54874506 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6fd80c7070 con 0x7f6fe013d3f0 2024-04-26T00:20:05.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.415+0000 7f6fdcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:05.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.415+0000 7f6fdcff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:05.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.471+0000 7fed437fe640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-26T00:20:05.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.471+0000 7fed437fe640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-26T00:20:05.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.471+0000 7fed437fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:05.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.471+0000 7fed437fe640 1 -- 172.21.15.37:0/2943998711 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fed30005650 con 0x7fed580a4170 2024-04-26T00:20:05.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.471+0000 7fed4dffb640 1 -- 172.21.15.37:0/2943998711 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fed48009390 con 0x7fed580a4170 2024-04-26T00:20:05.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.471+0000 7fed4dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-26T00:20:05.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.471+0000 7fed4dffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:05.454 INFO:teuthology.orchestra.run.smithi037.stdout:47244640260 2024-04-26T00:20:05.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.471+0000 7fed437fe640 1 -- 172.21.15.37:0/2943998711 >> v2:172.21.15.45:6812/2379918623 conn(0x7fed3c035ed0 msgr2=0x7fed3c038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:05.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.471+0000 7fed437fe640 1 --2- 172.21.15.37:0/2943998711 >> v2:172.21.15.45:6812/2379918623 conn(0x7fed3c035ed0 0x7fed3c038390 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fed54007fd0 tx=0x7fed54009040 comp rx=0 tx=0).stop 2024-04-26T00:20:05.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.471+0000 7fed437fe640 10 monclient: shutdown 2024-04-26T00:20:05.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.471+0000 7fed437fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:05.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.471+0000 7fed437fe640 1 -- 172.21.15.37:0/2943998711 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed580a4170 msgr2=0x7fed5813a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:05.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.471+0000 7fed437fe640 1 --2- 172.21.15.37:0/2943998711 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed580a4170 0x7fed5813a280 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fed4800d620 tx=0x7fed4800dae0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.471+0000 7fed437fe640 1 -- 172.21.15.37:0/2943998711 shutdown_connections 2024-04-26T00:20:05.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.471+0000 7fed437fe640 1 --2- 172.21.15.37:0/2943998711 >> v2:172.21.15.45:6812/2379918623 conn(0x7fed3c035ed0 0x7fed3c038390 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.471+0000 7fed437fe640 1 --2- 172.21.15.37:0/2943998711 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed5813c3e0 0x7fed5813a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.471+0000 7fed437fe640 1 --2- 172.21.15.37:0/2943998711 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed580a4170 0x7fed5813a280 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.471+0000 7fed437fe640 1 --2- 172.21.15.37:0/2943998711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed580a3d90 0x7fed58139d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.471+0000 7fed437fe640 1 -- 172.21.15.37:0/2943998711 >> 172.21.15.37:0/2943998711 conn(0x7fed580aa0a0 msgr2=0x7fed580a27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:05.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.475+0000 7fed437fe640 1 -- 172.21.15.37:0/2943998711 shutdown_connections 2024-04-26T00:20:05.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.475+0000 7fed437fe640 1 -- 172.21.15.37:0/2943998711 wait complete. 2024-04-26T00:20:05.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.483+0000 7ff65affd640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-26T00:20:05.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.483+0000 7ff65affd640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-26T00:20:05.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.483+0000 7ff65affd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:05.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.483+0000 7ff65affd640 1 -- 172.21.15.37:0/2982058370 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7ff63c005380 con 0x7ff6640a4170 2024-04-26T00:20:05.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.483+0000 7ff6617fa640 1 -- 172.21.15.37:0/2982058370 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff64c01a760 con 0x7ff6640a4170 2024-04-26T00:20:05.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.483+0000 7ff6617fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-26T00:20:05.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.483+0000 7ff6617fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:05.467 INFO:teuthology.orchestra.run.smithi037.stdout:47244640259 2024-04-26T00:20:05.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.487+0000 7ff65affd640 1 -- 172.21.15.37:0/2982058370 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff644035e80 msgr2=0x7ff644038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:05.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.487+0000 7ff65affd640 1 --2- 172.21.15.37:0/2982058370 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff644035e80 0x7ff644038340 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff65c007a70 tx=0x7ff65c008040 comp rx=0 tx=0).stop 2024-04-26T00:20:05.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.487+0000 7ff65affd640 10 monclient: shutdown 2024-04-26T00:20:05.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.487+0000 7ff65affd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:05.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.487+0000 7ff65affd640 1 -- 172.21.15.37:0/2982058370 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff6640a4170 msgr2=0x7ff66413ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:05.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.487+0000 7ff65affd640 1 --2- 172.21.15.37:0/2982058370 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff6640a4170 0x7ff66413ae20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff64c00c3e0 tx=0x7ff64c00c8a0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.487+0000 7ff65affd640 1 -- 172.21.15.37:0/2982058370 shutdown_connections 2024-04-26T00:20:05.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.487+0000 7ff65affd640 1 --2- 172.21.15.37:0/2982058370 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff644035e80 0x7ff644038340 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.487+0000 7ff65affd640 1 --2- 172.21.15.37:0/2982058370 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff66413c3e0 0x7ff6641218c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.487+0000 7ff65affd640 1 --2- 172.21.15.37:0/2982058370 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff6640a4170 0x7ff66413ae20 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.487+0000 7ff65affd640 1 --2- 172.21.15.37:0/2982058370 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6640a3d90 0x7ff6641cebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.487+0000 7ff65affd640 1 -- 172.21.15.37:0/2982058370 >> 172.21.15.37:0/2982058370 conn(0x7ff6640aa0a0 msgr2=0x7ff66413cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:05.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.487+0000 7ff65affd640 1 -- 172.21.15.37:0/2982058370 shutdown_connections 2024-04-26T00:20:05.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.487+0000 7ff65affd640 1 -- 172.21.15.37:0/2982058370 wait complete. 2024-04-26T00:20:05.473 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.2 2024-04-26T00:20:05.473 DEBUG:teuthology.parallel:result is None 2024-04-26T00:20:05.491 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.3 2024-04-26T00:20:05.491 DEBUG:teuthology.parallel:result is None 2024-04-26T00:20:05.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.539+0000 7f97b2655640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-26T00:20:05.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.539+0000 7f97b2655640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-26T00:20:05.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.539+0000 7f97b2655640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:05.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.539+0000 7f97b2655640 1 -- 172.21.15.37:0/3311461583 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f97ac1bd260 con 0x7f97ac0a4170 2024-04-26T00:20:05.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.539+0000 7f97a8ff9640 1 -- 172.21.15.37:0/3311461583 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f97a0013d70 con 0x7f97ac0a4170 2024-04-26T00:20:05.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.539+0000 7f97a8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-26T00:20:05.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.539+0000 7f97a8ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:05.523 INFO:teuthology.orchestra.run.smithi037.stdout:47244640259 2024-04-26T00:20:05.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.543+0000 7f97b2655640 1 -- 172.21.15.37:0/3311461583 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9798035ee0 msgr2=0x7f97980383a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:05.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.543+0000 7f97b2655640 1 --2- 172.21.15.37:0/3311461583 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9798035ee0 0x7f97980383a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f979c0b4000 tx=0x7f979c004740 comp rx=0 tx=0).stop 2024-04-26T00:20:05.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.543+0000 7f97b2655640 10 monclient: shutdown 2024-04-26T00:20:05.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.543+0000 7f97b2655640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:05.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.543+0000 7f97b2655640 1 -- 172.21.15.37:0/3311461583 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f97ac0a4170 msgr2=0x7f97ac132a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:05.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.543+0000 7f97b2655640 1 --2- 172.21.15.37:0/3311461583 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f97ac0a4170 0x7f97ac132a40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f97a000e890 tx=0x7f97a000ed50 comp rx=0 tx=0).stop 2024-04-26T00:20:05.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.543+0000 7f97b2655640 1 -- 172.21.15.37:0/3311461583 shutdown_connections 2024-04-26T00:20:05.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.543+0000 7f97b2655640 1 --2- 172.21.15.37:0/3311461583 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9798035ee0 0x7f97980383a0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.543+0000 7f97b2655640 1 --2- 172.21.15.37:0/3311461583 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97ac133b20 0x7f97ac132f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.543+0000 7f97b2655640 1 --2- 172.21.15.37:0/3311461583 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f97ac0a4170 0x7f97ac132a40 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.543+0000 7f97b2655640 1 --2- 172.21.15.37:0/3311461583 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97ac0a3d90 0x7f97ac1377a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.543+0000 7f97b2655640 1 -- 172.21.15.37:0/3311461583 >> 172.21.15.37:0/3311461583 conn(0x7f97ac0aa0a0 msgr2=0x7f97ac0a82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:05.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.543+0000 7f97b2655640 1 -- 172.21.15.37:0/3311461583 shutdown_connections 2024-04-26T00:20:05.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.543+0000 7f97b2655640 1 -- 172.21.15.37:0/3311461583 wait complete. 2024-04-26T00:20:05.541 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.0 2024-04-26T00:20:05.542 DEBUG:teuthology.parallel:result is None 2024-04-26T00:20:05.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.559+0000 7f2287227640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-26T00:20:05.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.559+0000 7f2287227640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-26T00:20:05.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.559+0000 7f2287227640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:05.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.559+0000 7f2287227640 1 -- 172.21.15.37:0/303866246 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f228013bc10 con 0x7f2280131cd0 2024-04-26T00:20:05.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.563+0000 7f227d7fa640 1 -- 172.21.15.37:0/303866246 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2278016520 con 0x7f2280131cd0 2024-04-26T00:20:05.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.563+0000 7f227d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-26T00:20:05.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.563+0000 7f227d7fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:05.546 INFO:teuthology.orchestra.run.smithi037.stdout:47244640258 2024-04-26T00:20:05.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.563+0000 7f2287227640 1 -- 172.21.15.37:0/303866246 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2264035ed0 msgr2=0x7f2264038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:05.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.563+0000 7f2287227640 1 --2- 172.21.15.37:0/303866246 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2264035ed0 0x7f2264038390 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2274009630 tx=0x7f2274009240 comp rx=0 tx=0).stop 2024-04-26T00:20:05.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.563+0000 7f2287227640 10 monclient: shutdown 2024-04-26T00:20:05.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.563+0000 7f2287227640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:05.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.563+0000 7f2287227640 1 -- 172.21.15.37:0/303866246 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2280131cd0 msgr2=0x7f22801cce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:05.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.563+0000 7f2287227640 1 --2- 172.21.15.37:0/303866246 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2280131cd0 0x7f22801cce00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f22780af000 tx=0x7f2278019640 comp rx=0 tx=0).stop 2024-04-26T00:20:05.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.563+0000 7f2287227640 1 -- 172.21.15.37:0/303866246 shutdown_connections 2024-04-26T00:20:05.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.563+0000 7f2287227640 1 --2- 172.21.15.37:0/303866246 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2264035ed0 0x7f2264038390 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.563+0000 7f2287227640 1 --2- 172.21.15.37:0/303866246 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2280131cd0 0x7f22801cce00 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.563+0000 7f2287227640 1 --2- 172.21.15.37:0/303866246 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22800a1b60 0x7f2280131170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.563+0000 7f2287227640 1 --2- 172.21.15.37:0/303866246 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2280073080 0x7f2280130c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.563+0000 7f2287227640 1 -- 172.21.15.37:0/303866246 >> 172.21.15.37:0/303866246 conn(0x7f22800a7160 msgr2=0x7f22801338d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:05.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.563+0000 7f2287227640 1 -- 172.21.15.37:0/303866246 shutdown_connections 2024-04-26T00:20:05.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.563+0000 7f2287227640 1 -- 172.21.15.37:0/303866246 wait complete. 2024-04-26T00:20:05.561 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640258 for osd.1 2024-04-26T00:20:05.561 DEBUG:teuthology.parallel:result is None 2024-04-26T00:20:05.618 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-26T00:20:05.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.643+0000 7f6fe639c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-26T00:20:05.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.643+0000 7f6fe639c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-26T00:20:05.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.643+0000 7f6fe639c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:05.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.643+0000 7f6fe639c640 1 -- 172.21.15.37:0/54874506 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f6fe0075dc0 con 0x7f6fe013d3f0 2024-04-26T00:20:05.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.643+0000 7f6fdcff9640 1 -- 172.21.15.37:0/54874506 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6fd80164c0 con 0x7f6fe013d3f0 2024-04-26T00:20:05.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.643+0000 7f6fdcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-26T00:20:05.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.643+0000 7f6fdcff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:05.627 INFO:teuthology.orchestra.run.smithi037.stdout:47244640258 2024-04-26T00:20:05.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.647+0000 7f6fe639c640 1 -- 172.21.15.37:0/54874506 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6fcc035f00 msgr2=0x7f6fcc0383c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:05.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.647+0000 7f6fe639c640 1 --2- 172.21.15.37:0/54874506 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6fcc035f00 0x7f6fcc0383c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0007d60 tx=0x7f6fd0007cb0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.647+0000 7f6fe639c640 10 monclient: shutdown 2024-04-26T00:20:05.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.647+0000 7f6fe639c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:05.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.647+0000 7f6fe639c640 1 -- 172.21.15.37:0/54874506 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6fe013d3f0 msgr2=0x7f6fe0139d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:05.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.647+0000 7f6fe639c640 1 --2- 172.21.15.37:0/54874506 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6fe013d3f0 0x7f6fe0139d20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6fd80af000 tx=0x7f6fd8019550 comp rx=0 tx=0).stop 2024-04-26T00:20:05.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.647+0000 7f6fe639c640 1 -- 172.21.15.37:0/54874506 shutdown_connections 2024-04-26T00:20:05.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.647+0000 7f6fe639c640 1 --2- 172.21.15.37:0/54874506 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6fcc035f00 0x7f6fcc0383c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.647+0000 7f6fe639c640 1 --2- 172.21.15.37:0/54874506 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6fe013d3f0 0x7f6fe0139d20 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.647+0000 7f6fe639c640 1 --2- 172.21.15.37:0/54874506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fe00a4170 0x7f6fe0136cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.647+0000 7f6fe639c640 1 --2- 172.21.15.37:0/54874506 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6fe00a3d90 0x7f6fe01367b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.647+0000 7f6fe639c640 1 -- 172.21.15.37:0/54874506 >> 172.21.15.37:0/54874506 conn(0x7f6fe00aa0a0 msgr2=0x7f6fe00a8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:05.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.647+0000 7f6fe639c640 1 -- 172.21.15.37:0/54874506 shutdown_connections 2024-04-26T00:20:05.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.647+0000 7f6fe639c640 1 -- 172.21.15.37:0/54874506 wait complete. 2024-04-26T00:20:05.643 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640258 for osd.5 2024-04-26T00:20:05.643 DEBUG:teuthology.parallel:result is None 2024-04-26T00:20:05.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f1108709640 10 monclient: get_monmap_and_config 2024-04-26T00:20:05.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f1108709640 10 monclient: build_initial_monmap 2024-04-26T00:20:05.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f1108709640 10 monclient: monmap: 2024-04-26T00:20:05.721 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:05.721 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:05.721 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:05.744368+0000 2024-04-26T00:20:05.721 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:05.744368+0000 2024-04-26T00:20:05.721 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:05.721 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:05.721 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:05.721 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:05.721 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:05.721 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:05.721 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:05.721 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:05.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f1108709640 1 Processor -- start 2024-04-26T00:20:05.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f1108709640 1 -- start start 2024-04-26T00:20:05.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f1108709640 10 monclient: init 2024-04-26T00:20:05.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f1108709640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:05.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f1108709640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-04-26T00:20:05.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f1108709640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f11000a0300 0x7f110009fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:05.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f1108709640 10 monclient(hunting): picked mon.noname-b con 0x7f11000a0300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:05.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f1108709640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f11000a06e0 addr v1:172.21.15.45:6789/0 2024-04-26T00:20:05.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f1108709640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f11001352b0 0x7f110013cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:05.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f1108709640 10 monclient(hunting): picked mon.noname-c con 0x7f11001352b0 addr v2:172.21.15.45:3300/0 2024-04-26T00:20:05.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f1108709640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f110009c190 con 0x7f11000a06e0 2024-04-26T00:20:05.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f1108709640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:05.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f1108709640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f11000a13f0 con 0x7f11001352b0 2024-04-26T00:20:05.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f1108709640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:05.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f1108709640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f11000a1110 con 0x7f11000a0300 2024-04-26T00:20:05.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f1108709640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:05.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f1108709640 10 monclient(hunting): authenticate will time out at 987.049959s 2024-04-26T00:20:05.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f11064a5640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f11000a0300 0x7f110009fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:05.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f11064a5640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f11000a0300 0x7f110009fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:57994/0 (socket says 172.21.15.37:57994) 2024-04-26T00:20:05.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f11064a5640 1 -- 172.21.15.37:0/2661900879 learned_addr learned my addr 172.21.15.37:0/2661900879 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:05.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f11064a5640 10 monclient(hunting): get_auth_request con 0x7f11000a0300 auth_method 0 2024-04-26T00:20:05.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f11064a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:05.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f11064a5640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:05.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f11064a5640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:05.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f11064a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:05.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f11064a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:05.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f11064a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:05.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f1106ca6640 1 --2- 172.21.15.37:0/2661900879 >> v2:172.21.15.45:3300/0 conn(0x7f11001352b0 0x7f110013cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:05.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f1106ca6640 10 monclient(hunting): get_auth_request con 0x7f11001352b0 auth_method 0 2024-04-26T00:20:05.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f1106ca6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:05.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f1106ca6640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:05.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f1106ca6640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:05.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f11064a5640 10 monclient(hunting): handle_auth_done global_id 4280 payload 274 2024-04-26T00:20:05.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f11064a5640 1 -- 172.21.15.37:0/2661900879 >> v2:172.21.15.45:3300/0 conn(0x7f11001352b0 msgr2=0x7f110013cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:05.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f11064a5640 1 --2- 172.21.15.37:0/2661900879 >> v2:172.21.15.45:3300/0 conn(0x7f11001352b0 0x7f110013cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f11064a5640 1 -- 172.21.15.37:0/2661900879 >> v1:172.21.15.45:6789/0 conn(0x7f11000a06e0 legacy=0x7f1100134ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:05.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f11064a5640 10 monclient: _finish_hunting 0 2024-04-26T00:20:05.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f11064a5640 1 monclient: found mon.noname-b 2024-04-26T00:20:05.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f11064a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:05.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f11064a5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:20:05.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.739+0000 7f11064a5640 1 -- 172.21.15.37:0/2661900879 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11000a16d0 con 0x7f11000a0300 2024-04-26T00:20:05.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.743+0000 7f11064a5640 1 -- 172.21.15.37:0/2661900879 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f11000a0300 msgr2=0x7f110009fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:05.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.743+0000 7f11064a5640 1 --2- 172.21.15.37:0/2661900879 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f11000a0300 0x7f110009fc50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f10f0009870 tx=0x7f10f001c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=856ee7dcd2b72fa6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:05.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.743+0000 7f11054a3640 1 -- 172.21.15.37:0/2661900879 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f00b2070 con 0x7f11000a0300 2024-04-26T00:20:05.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.743+0000 7f11054a3640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:05.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.743+0000 7f11054a3640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:20:05.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.743+0000 7f11054a3640 10 monclient: dump: 2024-04-26T00:20:05.730 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:05.730 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:05.730 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.730 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.730 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:05.730 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:05.730 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:05.730 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:05.730 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:05.730 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:05.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.743+0000 7f11054a3640 10 monclient: _finish_auth 0 2024-04-26T00:20:05.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.743+0000 7f11054a3640 10 monclient: _check_auth_tickets 2024-04-26T00:20:05.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.743+0000 7f11054a3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:05.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.743+0000 7f11054a3640 1 -- 172.21.15.37:0/2661900879 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10f00ad680 con 0x7f11000a0300 2024-04-26T00:20:05.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.743+0000 7f11054a3640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:05.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.743+0000 7f11054a3640 1 -- 172.21.15.37:0/2661900879 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f00ada50 con 0x7f11000a0300 2024-04-26T00:20:05.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.743+0000 7f1108709640 5 monclient: authenticate success, global_id 4280 2024-04-26T00:20:05.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.743+0000 7f1108709640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:05.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.743+0000 7f1108709640 4 set_mon_vals no callback set 2024-04-26T00:20:05.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.743+0000 7f11054a3640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:05.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.743+0000 7f11054a3640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:05.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.743+0000 7f11054a3640 10 monclient: dump: 2024-04-26T00:20:05.731 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:05.731 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:05.731 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.731 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.731 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:05.731 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:05.732 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:05.732 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:05.732 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:05.732 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:05.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.743+0000 7f1108709640 10 monclient: shutdown 2024-04-26T00:20:05.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.743+0000 7f1108709640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:05.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.743+0000 7f1108709640 1 -- 172.21.15.37:0/2661900879 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f11000a0300 msgr2=0x7f110009fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:05.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.743+0000 7f1108709640 1 --2- 172.21.15.37:0/2661900879 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f11000a0300 0x7f110009fc50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f10f0009870 tx=0x7f10f001c3e0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.743+0000 7f1108709640 1 -- 172.21.15.37:0/2661900879 shutdown_connections 2024-04-26T00:20:05.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.743+0000 7f1108709640 1 --2- 172.21.15.37:0/2661900879 >> v2:172.21.15.45:3300/0 conn(0x7f11001352b0 0x7f110013cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.743+0000 7f1108709640 1 --2- 172.21.15.37:0/2661900879 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f11000a0300 0x7f110009fc50 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.743+0000 7f1108709640 1 -- 172.21.15.37:0/2661900879 >> 172.21.15.37:0/2661900879 conn(0x7f11000a1d50 msgr2=0x7f11000a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:05.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.743+0000 7f1108709640 1 -- 172.21.15.37:0/2661900879 shutdown_connections 2024-04-26T00:20:05.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.743+0000 7f1108709640 1 -- 172.21.15.37:0/2661900879 wait complete. 2024-04-26T00:20:05.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.743+0000 7f1108709640 10 monclient: build_initial_monmap 2024-04-26T00:20:05.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.743+0000 7f1108709640 10 monclient: monmap: 2024-04-26T00:20:05.733 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:05.733 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:05.733 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:05.733 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:05.733 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:05.733 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:05.733 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:05.733 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:05.733 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:05.733 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:05.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f1108709640 1 Processor -- start 2024-04-26T00:20:05.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f1108709640 1 -- start start 2024-04-26T00:20:05.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f1108709640 10 monclient: init 2024-04-26T00:20:05.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f1108709640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:05.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f1108709640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:20:05.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f1108709640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f11000a0300 0x7f11001cf2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:05.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f1108709640 10 monclient(hunting): picked mon.noname-c con 0x7f11000a0300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:05.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f1108709640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11000a06e0 0x7f1100133ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:05.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f1108709640 10 monclient(hunting): picked mon.noname-b con 0x7f11000a06e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:05.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f1108709640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11001352b0 0x7f11001340e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:05.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f1108709640 10 monclient(hunting): picked mon.noname-a con 0x7f11001352b0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:05.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f1108709640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:05.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f1108709640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f110013fc40 con 0x7f11001352b0 2024-04-26T00:20:05.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f1108709640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:05.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f1108709640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f110013fdc0 con 0x7f11000a06e0 2024-04-26T00:20:05.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f1108709640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:05.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f1108709640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f110013ff40 con 0x7f11000a0300 2024-04-26T00:20:05.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f1108709640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:05.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f1108709640 10 monclient(hunting): authenticate will time out at 987.056635s 2024-04-26T00:20:05.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f11064a5640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f11000a0300 0x7f11001cf2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:05.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f11064a5640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f11000a0300 0x7f11001cf2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:58000/0 (socket says 172.21.15.37:58000) 2024-04-26T00:20:05.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f11064a5640 1 -- 172.21.15.37:0/3952409888 learned_addr learned my addr 172.21.15.37:0/3952409888 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:05.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f11064a5640 10 monclient(hunting): get_auth_request con 0x7f11000a0300 auth_method 0 2024-04-26T00:20:05.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f11064a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:05.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f11064a5640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:05.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f11064a5640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:05.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f1106ca6640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11001352b0 0x7f11001340e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:05.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f11064a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:05.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f11064a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:05.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f11064a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:05.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f1106ca6640 10 monclient(hunting): get_auth_request con 0x7f11001352b0 auth_method 0 2024-04-26T00:20:05.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f1106ca6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:05.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f1106ca6640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:05.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f1106ca6640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:05.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f11064a5640 10 monclient(hunting): handle_auth_done global_id 4283 payload 824 2024-04-26T00:20:05.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f11064a5640 1 -- 172.21.15.37:0/3952409888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11000a06e0 msgr2=0x7f1100133ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:05.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f11064a5640 1 --2- 172.21.15.37:0/3952409888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11000a06e0 0x7f1100133ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f11064a5640 1 -- 172.21.15.37:0/3952409888 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11001352b0 msgr2=0x7f11001340e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:05.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f11064a5640 1 --2- 172.21.15.37:0/3952409888 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11001352b0 0x7f11001340e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f11064a5640 10 monclient: _finish_hunting 0 2024-04-26T00:20:05.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f11064a5640 1 monclient: found mon.noname-c 2024-04-26T00:20:05.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f11064a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:05.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f11064a5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:20:05.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f11064a5640 1 -- 172.21.15.37:0/3952409888 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f110011e260 con 0x7f11000a0300 2024-04-26T00:20:05.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f11064a5640 1 -- 172.21.15.37:0/3952409888 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f11000a0300 msgr2=0x7f11001cf2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:05.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f11064a5640 1 --2- 172.21.15.37:0/3952409888 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f11000a0300 0x7f11001cf2d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f10f001cc80 tx=0x7f10f001ac20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:05.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f10f6ffd640 1 -- 172.21.15.37:0/3952409888 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f00bc020 con 0x7f11000a0300 2024-04-26T00:20:05.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f10f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:05.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f10f6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:20:05.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f10f6ffd640 10 monclient: dump: 2024-04-26T00:20:05.739 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:05.739 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:05.739 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.739 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.739 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:05.739 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:05.739 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:05.739 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:05.739 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:05.739 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:05.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f10f6ffd640 10 monclient: _finish_auth 0 2024-04-26T00:20:05.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f10f6ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:20:05.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f10f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:05.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f10f6ffd640 1 -- 172.21.15.37:0/3952409888 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10f00ade20 con 0x7f11000a0300 2024-04-26T00:20:05.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f10f6ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:05.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f10f6ffd640 1 -- 172.21.15.37:0/3952409888 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f00b2040 con 0x7f11000a0300 2024-04-26T00:20:05.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f10f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:05.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f10f6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:05.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f10f6ffd640 10 monclient: dump: 2024-04-26T00:20:05.740 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:05.740 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:05.740 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.740 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:05.740 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:05.740 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:05.740 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:05.740 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:05.740 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:05.741 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:05.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f1108709640 5 monclient: authenticate success, global_id 4283 2024-04-26T00:20:05.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f1108709640 10 monclient: _renew_subs 2024-04-26T00:20:05.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f1108709640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:05.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f1108709640 1 -- 172.21.15.37:0/3952409888 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11001cf810 con 0x7f11000a0300 2024-04-26T00:20:05.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f1104ca2640 4 set_mon_vals no callback set 2024-04-26T00:20:05.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f1108709640 10 monclient: _renew_subs 2024-04-26T00:20:05.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f1108709640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:05.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.747+0000 7f1108709640 1 -- 172.21.15.37:0/3952409888 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1100120550 con 0x7f11000a0300 2024-04-26T00:20:05.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.751+0000 7f1108709640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:05.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.751+0000 7f1108709640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:05.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.751+0000 7f1108709640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:05.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.751+0000 7f10f6ffd640 1 -- 172.21.15.37:0/3952409888 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f10f00bf020 con 0x7f11000a0300 2024-04-26T00:20:05.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.751+0000 7f1108709640 1 -- 172.21.15.37:0/3952409888 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f110004f510 con 0x7f11000a0300 2024-04-26T00:20:05.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.751+0000 7f10f6ffd640 1 --2- 172.21.15.37:0/3952409888 >> v2:172.21.15.45:6812/2379918623 conn(0x7f10e4035ed0 0x7f10e4038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:05.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.751+0000 7f10f6ffd640 1 -- 172.21.15.37:0/3952409888 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7f10f00e2a60 con 0x7f11000a0300 2024-04-26T00:20:05.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.751+0000 7f1105ca4640 1 --2- 172.21.15.37:0/3952409888 >> v2:172.21.15.45:6812/2379918623 conn(0x7f10e4035ed0 0x7f10e4038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:05.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.751+0000 7f1105ca4640 10 monclient: get_auth_request con 0x7f10e4035ed0 auth_method 0 2024-04-26T00:20:05.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.751+0000 7f1105ca4640 1 --2- 172.21.15.37:0/3952409888 >> v2:172.21.15.45:6812/2379918623 conn(0x7f10e4035ed0 0x7f10e4038390 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f110011e5e0 tx=0x7f10fc0079c0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:05.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.755+0000 7f10f6ffd640 1 -- 172.21.15.37:0/3952409888 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f10f00e4e20 con 0x7f11000a0300 2024-04-26T00:20:05.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.755+0000 7f10f6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:05.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.755+0000 7f10f6ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:05.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.907+0000 7f1108709640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-26T00:20:05.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.907+0000 7f1108709640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-26T00:20:05.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.907+0000 7f1108709640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:05.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.907+0000 7f1108709640 1 -- 172.21.15.37:0/3952409888 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f110009eb60 con 0x7f11000a0300 2024-04-26T00:20:05.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.911+0000 7f10f6ffd640 1 -- 172.21.15.37:0/3952409888 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f10f00c6030 con 0x7f11000a0300 2024-04-26T00:20:05.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.911+0000 7f10f6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-26T00:20:05.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.911+0000 7f10f6ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:05.892 INFO:teuthology.orchestra.run.smithi037.stdout:47244640258 2024-04-26T00:20:05.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.911+0000 7f1108709640 1 -- 172.21.15.37:0/3952409888 >> v2:172.21.15.45:6812/2379918623 conn(0x7f10e4035ed0 msgr2=0x7f10e4038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:05.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.911+0000 7f1108709640 1 --2- 172.21.15.37:0/3952409888 >> v2:172.21.15.45:6812/2379918623 conn(0x7f10e4035ed0 0x7f10e4038390 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f110011e5e0 tx=0x7f10fc0079c0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.911+0000 7f1108709640 10 monclient: shutdown 2024-04-26T00:20:05.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.911+0000 7f1108709640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:05.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.911+0000 7f1108709640 1 -- 172.21.15.37:0/3952409888 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f11000a0300 msgr2=0x7f11001cf2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:05.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.911+0000 7f1108709640 1 --2- 172.21.15.37:0/3952409888 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f11000a0300 0x7f11001cf2d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f10f001cc80 tx=0x7f10f001ac20 comp rx=0 tx=0).stop 2024-04-26T00:20:05.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.911+0000 7f1108709640 1 -- 172.21.15.37:0/3952409888 shutdown_connections 2024-04-26T00:20:05.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.911+0000 7f1108709640 1 --2- 172.21.15.37:0/3952409888 >> v2:172.21.15.45:6812/2379918623 conn(0x7f10e4035ed0 0x7f10e4038390 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.911+0000 7f1108709640 1 --2- 172.21.15.37:0/3952409888 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11001352b0 0x7f11001340e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.911+0000 7f1108709640 1 --2- 172.21.15.37:0/3952409888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11000a06e0 0x7f1100133ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.911+0000 7f1108709640 1 --2- 172.21.15.37:0/3952409888 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f11000a0300 0x7f11001cf2d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:05.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.911+0000 7f1108709640 1 -- 172.21.15.37:0/3952409888 >> 172.21.15.37:0/3952409888 conn(0x7f11000a1d50 msgr2=0x7f11000a3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:05.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.911+0000 7f1108709640 1 -- 172.21.15.37:0/3952409888 shutdown_connections 2024-04-26T00:20:05.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:05.911+0000 7f1108709640 1 -- 172.21.15.37:0/3952409888 wait complete. 2024-04-26T00:20:05.909 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640258 for osd.4 2024-04-26T00:20:05.909 DEBUG:teuthology.parallel:result is None 2024-04-26T00:20:05.909 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-26T00:20:05.910 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T00:20:06.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.043+0000 7f856880b640 10 monclient: get_monmap_and_config 2024-04-26T00:20:06.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.043+0000 7f856880b640 10 monclient: build_initial_monmap 2024-04-26T00:20:06.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.043+0000 7f856880b640 10 monclient: monmap: 2024-04-26T00:20:06.028 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:06.028 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:06.028 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:06.051131+0000 2024-04-26T00:20:06.028 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:06.051131+0000 2024-04-26T00:20:06.028 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:06.028 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:06.028 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:06.028 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:06.028 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:06.028 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:06.028 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:06.029 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:06.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.047+0000 7f856880b640 1 Processor -- start 2024-04-26T00:20:06.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.047+0000 7f856880b640 1 -- start start 2024-04-26T00:20:06.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.047+0000 7f856880b640 10 monclient: init 2024-04-26T00:20:06.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.047+0000 7f856880b640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:06.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.047+0000 7f856880b640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-04-26T00:20:06.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.047+0000 7f856880b640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f8564072bf0 0x7f856409ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:06.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.047+0000 7f856880b640 10 monclient(hunting): picked mon.noname-a con 0x7f8564072bf0 addr v2:172.21.15.37:3300/0 2024-04-26T00:20:06.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.047+0000 7f856880b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8564099430 addr v1:172.21.15.37:6789/0 2024-04-26T00:20:06.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.047+0000 7f856880b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f85640a1550 addr v1:172.21.15.45:6789/0 2024-04-26T00:20:06.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.047+0000 7f856880b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8564072fd0 con 0x7f8564099430 2024-04-26T00:20:06.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.047+0000 7f856880b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8564099c60 con 0x7f85640a1550 2024-04-26T00:20:06.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.047+0000 7f856880b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:06.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.047+0000 7f856880b640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f856406b750 con 0x7f8564072bf0 2024-04-26T00:20:06.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.047+0000 7f856880b640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:06.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.047+0000 7f856880b640 10 monclient(hunting): authenticate will time out at 987.356813s 2024-04-26T00:20:06.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.047+0000 7f856259c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f8564072bf0 0x7f856409ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:06.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.047+0000 7f8561d9b640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f8564099430 0x7f85640a0e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:55818/0 (socket says 172.21.15.37:55818) 2024-04-26T00:20:06.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.047+0000 7f856259c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f8564072bf0 0x7f856409ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44814/0 (socket says 172.21.15.37:44814) 2024-04-26T00:20:06.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.047+0000 7f8561d9b640 1 -- 172.21.15.37:0/2255622322 learned_addr learned my addr 172.21.15.37:0/2255622322 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:20:06.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.047+0000 7f856259c640 10 monclient(hunting): get_auth_request con 0x7f8564072bf0 auth_method 0 2024-04-26T00:20:06.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.047+0000 7f856259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:06.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.047+0000 7f856259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:06.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.047+0000 7f856259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:06.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.047+0000 7f856259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:06.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.047+0000 7f856259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:06.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.047+0000 7f856259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:06.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.047+0000 7f856259c640 10 monclient(hunting): handle_auth_done global_id 4302 payload 274 2024-04-26T00:20:06.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.047+0000 7f856259c640 1 -- 172.21.15.37:0/2255622322 >> v1:172.21.15.45:6789/0 conn(0x7f85640a1550 legacy=0x7f856413f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:06.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.047+0000 7f856259c640 1 -- 172.21.15.37:0/2255622322 >> v1:172.21.15.37:6789/0 conn(0x7f8564099430 legacy=0x7f85640a0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:06.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.047+0000 7f856259c640 10 monclient: _finish_hunting 0 2024-04-26T00:20:06.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.047+0000 7f856259c640 1 monclient: found mon.noname-a 2024-04-26T00:20:06.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.047+0000 7f856259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:06.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.047+0000 7f856259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:06.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.047+0000 7f856259c640 1 -- 172.21.15.37:0/2255622322 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8564099f40 con 0x7f8564072bf0 2024-04-26T00:20:06.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.051+0000 7f856259c640 1 -- 172.21.15.37:0/2255622322 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8564072bf0 msgr2=0x7f856409ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:06.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.051+0000 7f856259c640 1 --2- 172.21.15.37:0/2255622322 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8564072bf0 0x7f856409ba30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f855c009dc0 tx=0x7f855c01c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2daaf7262fe49520 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:06.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.051+0000 7f856159a640 1 -- 172.21.15.37:0/2255622322 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f855c0a67d0 con 0x7f8564072bf0 2024-04-26T00:20:06.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.051+0000 7f856159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:06.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.051+0000 7f856159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:06.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.051+0000 7f856159a640 10 monclient: dump: 2024-04-26T00:20:06.035 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:06.035 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:06.035 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:06.035 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:06.035 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:06.035 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:06.035 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:06.035 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:06.035 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:06.035 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:06.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.051+0000 7f856159a640 10 monclient: _finish_auth 0 2024-04-26T00:20:06.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.051+0000 7f856159a640 10 monclient: _check_auth_tickets 2024-04-26T00:20:06.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.051+0000 7f856159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:06.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.051+0000 7f856880b640 5 monclient: authenticate success, global_id 4302 2024-04-26T00:20:06.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.051+0000 7f856880b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:20:06.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.051+0000 7f856159a640 1 -- 172.21.15.37:0/2255622322 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f855c0a6970 con 0x7f8564072bf0 2024-04-26T00:20:06.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.051+0000 7f856159a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:06.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.051+0000 7f856880b640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:06.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.051+0000 7f856880b640 4 set_mon_vals no callback set 2024-04-26T00:20:06.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.051+0000 7f856159a640 1 -- 172.21.15.37:0/2255622322 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f855c0a6d40 con 0x7f8564072bf0 2024-04-26T00:20:06.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.051+0000 7f856159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:06.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.051+0000 7f856159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:06.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.051+0000 7f856159a640 10 monclient: dump: 2024-04-26T00:20:06.036 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:06.036 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:06.037 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:06.037 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:06.037 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:06.037 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:06.037 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:06.037 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:06.037 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:06.037 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:06.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.051+0000 7f856880b640 10 monclient: shutdown 2024-04-26T00:20:06.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.051+0000 7f856880b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:06.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.051+0000 7f856880b640 1 -- 172.21.15.37:0/2255622322 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8564072bf0 msgr2=0x7f856409ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:06.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.051+0000 7f856880b640 1 --2- 172.21.15.37:0/2255622322 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8564072bf0 0x7f856409ba30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f855c009dc0 tx=0x7f855c01c790 comp rx=0 tx=0).stop 2024-04-26T00:20:06.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.051+0000 7f856880b640 1 -- 172.21.15.37:0/2255622322 shutdown_connections 2024-04-26T00:20:06.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.051+0000 7f856880b640 1 --2- 172.21.15.37:0/2255622322 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8564072bf0 0x7f856409ba30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:06.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.051+0000 7f856880b640 1 -- 172.21.15.37:0/2255622322 >> 172.21.15.37:0/2255622322 conn(0x7f85640a1f30 msgr2=0x7f85640a2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:06.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.051+0000 7f856880b640 1 -- 172.21.15.37:0/2255622322 shutdown_connections 2024-04-26T00:20:06.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.051+0000 7f856880b640 1 -- 172.21.15.37:0/2255622322 wait complete. 2024-04-26T00:20:06.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.051+0000 7f856880b640 10 monclient: build_initial_monmap 2024-04-26T00:20:06.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.051+0000 7f856880b640 10 monclient: monmap: 2024-04-26T00:20:06.042 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:06.043 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:06.043 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:06.043 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:06.043 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:06.043 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:06.043 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:06.043 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:06.043 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:06.043 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:06.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.051+0000 7f856880b640 1 Processor -- start 2024-04-26T00:20:06.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.051+0000 7f856880b640 1 -- start start 2024-04-26T00:20:06.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.051+0000 7f856880b640 10 monclient: init 2024-04-26T00:20:06.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f856880b640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:06.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f856880b640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:20:06.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f856880b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8564072bf0 0x7f85641cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:06.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f856880b640 10 monclient(hunting): picked mon.noname-c con 0x7f8564072bf0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:06.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f856880b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8564099430 0x7f85641305c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:06.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f856880b640 10 monclient(hunting): picked mon.noname-a con 0x7f8564099430 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:06.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f856880b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85640a1550 0x7f8564130b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:06.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f856880b640 10 monclient(hunting): picked mon.noname-b con 0x7f85640a1550 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:06.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f856880b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:06.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f856880b640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f856409c0f0 con 0x7f8564099430 2024-04-26T00:20:06.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f856880b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:06.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f856880b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f856409bf70 con 0x7f85640a1550 2024-04-26T00:20:06.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f856880b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:06.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f856880b640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f85641436c0 con 0x7f8564072bf0 2024-04-26T00:20:06.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f856880b640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:06.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f856880b640 10 monclient(hunting): authenticate will time out at 987.364408s 2024-04-26T00:20:06.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f8561d9b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8564099430 0x7f85641305c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:06.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f856259c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8564072bf0 0x7f85641cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:06.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f856259c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8564072bf0 0x7f85641cfb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:58002/0 (socket says 172.21.15.37:58002) 2024-04-26T00:20:06.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f8561d9b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8564099430 0x7f85641305c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44822/0 (socket says 172.21.15.37:44822) 2024-04-26T00:20:06.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f856259c640 1 -- 172.21.15.37:0/3933088497 learned_addr learned my addr 172.21.15.37:0/3933088497 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:06.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f8561d9b640 10 monclient(hunting): get_auth_request con 0x7f8564099430 auth_method 0 2024-04-26T00:20:06.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f8561d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:06.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f8561d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:06.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f8561d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:06.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f856259c640 10 monclient(hunting): get_auth_request con 0x7f8564072bf0 auth_method 0 2024-04-26T00:20:06.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f856259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:06.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f856259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:06.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f856259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:06.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f8561d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:06.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f8561d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:06.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f8561d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:06.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f856259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:06.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f856259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:06.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f856259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:06.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f8562d9d640 1 --2- 172.21.15.37:0/3933088497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85640a1550 0x7f8564130b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:06.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f8562d9d640 10 monclient(hunting): get_auth_request con 0x7f85640a1550 auth_method 0 2024-04-26T00:20:06.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f8562d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:06.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f8562d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:06.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f8562d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:06.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f8561d9b640 10 monclient(hunting): handle_auth_done global_id 4305 payload 824 2024-04-26T00:20:06.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f8561d9b640 1 -- 172.21.15.37:0/3933088497 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8564072bf0 msgr2=0x7f85641cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:06.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f8561d9b640 1 --2- 172.21.15.37:0/3933088497 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8564072bf0 0x7f85641cfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:06.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f8561d9b640 1 -- 172.21.15.37:0/3933088497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85640a1550 msgr2=0x7f8564130b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:06.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f8561d9b640 1 --2- 172.21.15.37:0/3933088497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85640a1550 0x7f8564130b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:06.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f8561d9b640 10 monclient: _finish_hunting 0 2024-04-26T00:20:06.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f8561d9b640 1 monclient: found mon.noname-a 2024-04-26T00:20:06.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f8561d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:06.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f8561d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:06.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f8561d9b640 1 -- 172.21.15.37:0/3933088497 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f856412f090 con 0x7f8564099430 2024-04-26T00:20:06.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f856259c640 1 --2- 172.21.15.37:0/3933088497 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8564072bf0 0x7f85641cfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:06.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f8561d9b640 1 -- 172.21.15.37:0/3933088497 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8564099430 msgr2=0x7f85641305c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:06.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f8561d9b640 1 --2- 172.21.15.37:0/3933088497 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8564099430 0x7f85641305c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8564099830 tx=0x7f854c00c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:06.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f855affd640 1 -- 172.21.15.37:0/3933088497 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f854c017070 con 0x7f8564099430 2024-04-26T00:20:06.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f855affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:06.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f855affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:06.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f855affd640 10 monclient: dump: 2024-04-26T00:20:06.048 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:06.048 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:06.048 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:06.048 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:06.048 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:06.048 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:06.048 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:06.048 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:06.048 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:06.048 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:06.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f855affd640 10 monclient: _finish_auth 0 2024-04-26T00:20:06.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f855affd640 10 monclient: _check_auth_tickets 2024-04-26T00:20:06.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f855affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:06.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f855affd640 1 -- 172.21.15.37:0/3933088497 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f854c0070d0 con 0x7f8564099430 2024-04-26T00:20:06.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f855affd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:06.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f855affd640 1 -- 172.21.15.37:0/3933088497 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f854c0074a0 con 0x7f8564099430 2024-04-26T00:20:06.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f855affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:06.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f855affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:06.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f855affd640 10 monclient: dump: 2024-04-26T00:20:06.049 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:06.049 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:06.049 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:06.049 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:06.049 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:06.049 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:06.049 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:06.049 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:06.049 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:06.050 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:06.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f856880b640 5 monclient: authenticate success, global_id 4305 2024-04-26T00:20:06.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f856880b640 10 monclient: _renew_subs 2024-04-26T00:20:06.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f856880b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:06.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f856880b640 1 -- 172.21.15.37:0/3933088497 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8564073580 con 0x7f8564099430 2024-04-26T00:20:06.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f8560d99640 4 set_mon_vals no callback set 2024-04-26T00:20:06.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f856880b640 10 monclient: _renew_subs 2024-04-26T00:20:06.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f856880b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:06.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.055+0000 7f856880b640 1 -- 172.21.15.37:0/3933088497 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f856412e7f0 con 0x7f8564099430 2024-04-26T00:20:06.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.059+0000 7f856880b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:06.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.059+0000 7f856880b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:06.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.059+0000 7f856880b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:06.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.059+0000 7f855affd640 1 -- 172.21.15.37:0/3933088497 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f854c01a070 con 0x7f8564099430 2024-04-26T00:20:06.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.059+0000 7f856880b640 1 -- 172.21.15.37:0/3933088497 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8564066f40 con 0x7f8564099430 2024-04-26T00:20:06.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.059+0000 7f855affd640 1 --2- 172.21.15.37:0/3933088497 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8544035ed0 0x7f8544038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:06.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.059+0000 7f855affd640 1 -- 172.21.15.37:0/3933088497 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7f854c047cc0 con 0x7f8564099430 2024-04-26T00:20:06.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.059+0000 7f856259c640 1 --2- 172.21.15.37:0/3933088497 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8544035ed0 0x7f8544038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:06.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.059+0000 7f856259c640 10 monclient: get_auth_request con 0x7f8544035ed0 auth_method 0 2024-04-26T00:20:06.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.059+0000 7f856259c640 1 --2- 172.21.15.37:0/3933088497 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8544035ed0 0x7f8544038390 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f855c009dc0 tx=0x7f855c0a7000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:06.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.063+0000 7f855affd640 1 -- 172.21.15.37:0/3933088497 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f854c046bb0 con 0x7f8564099430 2024-04-26T00:20:06.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.063+0000 7f855affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:06.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.063+0000 7f855affd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:06.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.219+0000 7f856880b640 1 -- 172.21.15.37:0/3933088497 --> v2:172.21.15.45:6812/2379918623 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f856406bf60 con 0x7f8544035ed0 2024-04-26T00:20:06.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.223+0000 7f855affd640 1 -- 172.21.15.37:0/3933088497 <== mgr.4107 v2:172.21.15.45:6812/2379918623 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33873 (secure 0 0 0) 0x7f856406bf60 con 0x7f8544035ed0 2024-04-26T00:20:06.206 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:20:06.207 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-04-26T00:20:06.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.227+0000 7f856880b640 1 -- 172.21.15.37:0/3933088497 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8544035ed0 msgr2=0x7f8544038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:06.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.227+0000 7f856880b640 1 --2- 172.21.15.37:0/3933088497 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8544035ed0 0x7f8544038390 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f855c009dc0 tx=0x7f855c0a7000 comp rx=0 tx=0).stop 2024-04-26T00:20:06.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.227+0000 7f856880b640 10 monclient: shutdown 2024-04-26T00:20:06.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.227+0000 7f856880b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:06.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.227+0000 7f856880b640 1 -- 172.21.15.37:0/3933088497 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8564099430 msgr2=0x7f85641305c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:06.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.227+0000 7f856880b640 1 --2- 172.21.15.37:0/3933088497 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8564099430 0x7f85641305c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8564099830 tx=0x7f854c00c870 comp rx=0 tx=0).stop 2024-04-26T00:20:06.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.227+0000 7f856880b640 1 -- 172.21.15.37:0/3933088497 shutdown_connections 2024-04-26T00:20:06.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.227+0000 7f856880b640 1 --2- 172.21.15.37:0/3933088497 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8544035ed0 0x7f8544038390 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:06.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.227+0000 7f856880b640 1 --2- 172.21.15.37:0/3933088497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85640a1550 0x7f8564130b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:06.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.227+0000 7f856880b640 1 --2- 172.21.15.37:0/3933088497 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8564099430 0x7f85641305c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:06.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.227+0000 7f856880b640 1 --2- 172.21.15.37:0/3933088497 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8564072bf0 0x7f85641cfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:06.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.227+0000 7f856880b640 1 -- 172.21.15.37:0/3933088497 >> 172.21.15.37:0/3933088497 conn(0x7f85640a1f30 msgr2=0x7f8564135860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:06.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.227+0000 7f856880b640 1 -- 172.21.15.37:0/3933088497 shutdown_connections 2024-04-26T00:20:06.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.227+0000 7f856880b640 1 -- 172.21.15.37:0/3933088497 wait complete. 2024-04-26T00:20:06.223 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-04-26T00:20:05.149627+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":145,"num_write_kb":2154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105,"ondisk_log_size":105,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":167156,"kb_used_data":820,"kb_used_omap":9,"kb_used_meta":166198,"kb_avail":566063884,"statfs":{"total":579820584960,"available":579649417216,"internally_reserved":0,"allocated":839680,"data_stored":1071264,"data_compressed":46459,"data_compressed_allocated":61440,"data_compressed_original":839680,"omap_allocated":9568,"internal_metadata":170187424},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-26T00:20:02.476315+0000","last_change":"2024-04-26T00:20:02.475146+0000","last_active":"2024-04-26T00:20:02.476315+0000","last_peered":"2024-04-26T00:20:02.476315+0000","last_clean":"2024-04-26T00:20:02.476315+0000","last_became_active":"2024-04-26T00:20:02.473928+0000","last_became_peered":"2024-04-26T00:20:02.473928+0000","last_unstale":"2024-04-26T00:20:02.476315+0000","last_undegraded":"2024-04-26T00:20:02.476315+0000","last_fullsized":"2024-04-26T00:20:02.476315+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:50:14.258009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T00:20:03.547413+0000","last_change":"2024-04-26T00:20:02.279987+0000","last_active":"2024-04-26T00:20:03.547413+0000","last_peered":"2024-04-26T00:20:03.547413+0000","last_clean":"2024-04-26T00:20:03.547413+0000","last_became_active":"2024-04-26T00:20:02.279183+0000","last_became_peered":"2024-04-26T00:20:02.279183+0000","last_unstale":"2024-04-26T00:20:03.547413+0000","last_undegraded":"2024-04-26T00:20:03.547413+0000","last_fullsized":"2024-04-26T00:20:03.547413+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:45:44.508952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T00:20:03.268034+0000","last_change":"2024-04-26T00:20:02.474779+0000","last_active":"2024-04-26T00:20:03.268034+0000","last_peered":"2024-04-26T00:20:03.268034+0000","last_clean":"2024-04-26T00:20:03.268034+0000","last_became_active":"2024-04-26T00:20:02.473888+0000","last_became_peered":"2024-04-26T00:20:02.473888+0000","last_unstale":"2024-04-26T00:20:03.268034+0000","last_undegraded":"2024-04-26T00:20:03.268034+0000","last_fullsized":"2024-04-26T00:20:03.268034+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:33:18.962907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T00:20:03.266877+0000","last_change":"2024-04-26T00:20:02.274249+0000","last_active":"2024-04-26T00:20:03.266877+0000","last_peered":"2024-04-26T00:20:03.266877+0000","last_clean":"2024-04-26T00:20:03.266877+0000","last_became_active":"2024-04-26T00:20:02.273745+0000","last_became_peered":"2024-04-26T00:20:02.273745+0000","last_unstale":"2024-04-26T00:20:03.266877+0000","last_undegraded":"2024-04-26T00:20:03.266877+0000","last_fullsized":"2024-04-26T00:20:03.266877+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:36:25.948539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"14'105","reported_seq":147,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T00:20:03.547490+0000","last_change":"2024-04-26T00:20:02.281782+0000","last_active":"2024-04-26T00:20:03.547490+0000","last_peered":"2024-04-26T00:20:03.547490+0000","last_clean":"2024-04-26T00:20:03.547490+0000","last_became_active":"2024-04-26T00:20:02.281000+0000","last_became_peered":"2024-04-26T00:20:02.281000+0000","last_unstale":"2024-04-26T00:20:03.547490+0000","last_undegraded":"2024-04-26T00:20:03.547490+0000","last_fullsized":"2024-04-26T00:20:03.547490+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:40:30.790500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":145,"num_write_kb":2154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T00:20:03.267137+0000","last_change":"2024-04-26T00:20:02.273965+0000","last_active":"2024-04-26T00:20:03.267137+0000","last_peered":"2024-04-26T00:20:03.267137+0000","last_clean":"2024-04-26T00:20:03.267137+0000","last_became_active":"2024-04-26T00:20:02.273213+0000","last_became_peered":"2024-04-26T00:20:02.273213+0000","last_unstale":"2024-04-26T00:20:03.267137+0000","last_undegraded":"2024-04-26T00:20:03.267137+0000","last_fullsized":"2024-04-26T00:20:03.267137+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:42:14.055403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T00:20:03.268104+0000","last_change":"2024-04-26T00:20:02.277311+0000","last_active":"2024-04-26T00:20:03.268104+0000","last_peered":"2024-04-26T00:20:03.268104+0000","last_clean":"2024-04-26T00:20:03.268104+0000","last_became_active":"2024-04-26T00:20:02.276409+0000","last_became_peered":"2024-04-26T00:20:02.276409+0000","last_unstale":"2024-04-26T00:20:03.268104+0000","last_undegraded":"2024-04-26T00:20:03.268104+0000","last_fullsized":"2024-04-26T00:20:03.268104+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:14:50.584720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-26T00:20:02.281720+0000","last_change":"2024-04-26T00:20:02.279805+0000","last_active":"2024-04-26T00:20:02.281720+0000","last_peered":"2024-04-26T00:20:02.281720+0000","last_clean":"2024-04-26T00:20:02.281720+0000","last_became_active":"2024-04-26T00:20:02.279053+0000","last_became_peered":"2024-04-26T00:20:02.279053+0000","last_unstale":"2024-04-26T00:20:02.281720+0000","last_undegraded":"2024-04-26T00:20:02.281720+0000","last_fullsized":"2024-04-26T00:20:02.281720+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:00:44.241406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T00:20:03.547492+0000","last_change":"2024-04-26T00:20:02.279764+0000","last_active":"2024-04-26T00:20:03.547492+0000","last_peered":"2024-04-26T00:20:03.547492+0000","last_clean":"2024-04-26T00:20:03.547492+0000","last_became_active":"2024-04-26T00:20:02.278959+0000","last_became_peered":"2024-04-26T00:20:02.278959+0000","last_unstale":"2024-04-26T00:20:03.547492+0000","last_undegraded":"2024-04-26T00:20:03.547492+0000","last_fullsized":"2024-04-26T00:20:03.547492+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:23:43.610174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":145,"num_write_kb":2154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":934976,"data_compressed":55314,"data_compressed_allocated":73728,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":105,"ondisk_log_size":105,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640258,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":122880,"data_stored":33587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640258,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31300,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340540,"statfs":{"total":96636764160,"available":96604712960,"internally_reserved":0,"allocated":122880,"data_stored":33587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":159744,"data_stored":431427,"data_compressed":18802,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":122880,"data_stored":33587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640258,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":188416,"data_stored":505489,"data_compressed":27657,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":122880,"data_stored":33587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":467488,"data_compressed":27657,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":467488,"data_compressed":27657,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T00:20:06.225 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T00:20:06.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.359+0000 7f0d7fdd4640 10 monclient: get_monmap_and_config 2024-04-26T00:20:06.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.359+0000 7f0d7fdd4640 10 monclient: build_initial_monmap 2024-04-26T00:20:06.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.359+0000 7f0d7fdd4640 10 monclient: monmap: 2024-04-26T00:20:06.343 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:06.343 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:06.343 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:06.366757+0000 2024-04-26T00:20:06.343 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:06.366757+0000 2024-04-26T00:20:06.343 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:06.343 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:06.344 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:06.344 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:06.344 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:06.344 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:06.344 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:06.344 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:06.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.359+0000 7f0d7fdd4640 1 Processor -- start 2024-04-26T00:20:06.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.359+0000 7f0d7fdd4640 1 -- start start 2024-04-26T00:20:06.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.359+0000 7f0d7fdd4640 10 monclient: init 2024-04-26T00:20:06.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.363+0000 7f0d7fdd4640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:06.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.363+0000 7f0d7fdd4640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-04-26T00:20:06.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.363+0000 7f0d7fdd4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0d78136eb0 addr v1:172.21.15.37:6789/0 2024-04-26T00:20:06.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.363+0000 7f0d7fdd4640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d781379a0 0x7f0d7813c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:06.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.363+0000 7f0d7fdd4640 10 monclient(hunting): picked mon.noname-b con 0x7f0d781379a0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:06.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.363+0000 7f0d7fdd4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0d7813c990 addr v1:172.21.15.45:6789/0 2024-04-26T00:20:06.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.363+0000 7f0d7fdd4640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d7809b8c0 con 0x7f0d78136eb0 2024-04-26T00:20:06.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.363+0000 7f0d7fdd4640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d7809b300 con 0x7f0d7813c990 2024-04-26T00:20:06.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.363+0000 7f0d7fdd4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:06.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.363+0000 7f0d7fdd4640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0d7809c7d0 con 0x7f0d781379a0 2024-04-26T00:20:06.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.363+0000 7f0d7fdd4640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:06.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.363+0000 7f0d7fdd4640 10 monclient(hunting): authenticate will time out at 987.672394s 2024-04-26T00:20:06.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.363+0000 7f0d7d36f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d781379a0 0x7f0d7813c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:06.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.363+0000 7f0d7d36f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d781379a0 0x7f0d7813c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:58016/0 (socket says 172.21.15.37:58016) 2024-04-26T00:20:06.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.363+0000 7f0d7d36f640 1 -- 172.21.15.37:0/1453136200 learned_addr learned my addr 172.21.15.37:0/1453136200 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:06.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.363+0000 7f0d7d36f640 10 monclient(hunting): get_auth_request con 0x7f0d781379a0 auth_method 0 2024-04-26T00:20:06.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.363+0000 7f0d7d36f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:06.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.363+0000 7f0d7d36f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:06.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.363+0000 7f0d7d36f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:06.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.363+0000 7f0d7db70640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f0d78136eb0 0x7f0d78137290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:55826/0 (socket says 172.21.15.37:55826) 2024-04-26T00:20:06.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.363+0000 7f0d7d36f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:06.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.363+0000 7f0d7d36f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:06.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.363+0000 7f0d7d36f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:06.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.363+0000 7f0d7d36f640 10 monclient(hunting): handle_auth_done global_id 4289 payload 274 2024-04-26T00:20:06.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.363+0000 7f0d7d36f640 1 -- 172.21.15.37:0/1453136200 >> v1:172.21.15.45:6789/0 conn(0x7f0d7813c990 legacy=0x7f0d7813ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:06.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.363+0000 7f0d7d36f640 1 -- 172.21.15.37:0/1453136200 >> v1:172.21.15.37:6789/0 conn(0x7f0d78136eb0 legacy=0x7f0d78137290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:06.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.363+0000 7f0d7d36f640 10 monclient: _finish_hunting 0 2024-04-26T00:20:06.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.363+0000 7f0d7d36f640 1 monclient: found mon.noname-b 2024-04-26T00:20:06.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.363+0000 7f0d7d36f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:06.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.363+0000 7f0d7d36f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:20:06.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.363+0000 7f0d7d36f640 1 -- 172.21.15.37:0/1453136200 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d7809b5e0 con 0x7f0d781379a0 2024-04-26T00:20:06.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7d36f640 1 -- 172.21.15.37:0/1453136200 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d781379a0 msgr2=0x7f0d7813c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:06.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7d36f640 1 --2- 172.21.15.37:0/1453136200 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d781379a0 0x7f0d7813c450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0d740098a0 tx=0x7f0d7401c1e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=270179ce07c24adb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:06.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7cb6e640 1 -- 172.21.15.37:0/1453136200 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d740b0070 con 0x7f0d781379a0 2024-04-26T00:20:06.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7cb6e640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:06.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7cb6e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:20:06.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7cb6e640 10 monclient: dump: 2024-04-26T00:20:06.353 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:06.353 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:06.353 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:06.353 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:06.353 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:06.353 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:06.353 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:06.353 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:06.353 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:06.353 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:06.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7cb6e640 10 monclient: _finish_auth 0 2024-04-26T00:20:06.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7cb6e640 10 monclient: _check_auth_tickets 2024-04-26T00:20:06.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7cb6e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:06.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7cb6e640 1 -- 172.21.15.37:0/1453136200 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d740ab600 con 0x7f0d781379a0 2024-04-26T00:20:06.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7cb6e640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:06.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7cb6e640 1 -- 172.21.15.37:0/1453136200 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d740ab9d0 con 0x7f0d781379a0 2024-04-26T00:20:06.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7cb6e640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:06.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7cb6e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:06.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7cb6e640 10 monclient: dump: 2024-04-26T00:20:06.354 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:06.354 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:06.354 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:06.354 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:06.354 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:06.354 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:06.354 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:06.355 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:06.355 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:06.355 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:06.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7fdd4640 5 monclient: authenticate success, global_id 4289 2024-04-26T00:20:06.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7fdd4640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:06.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7fdd4640 4 set_mon_vals no callback set 2024-04-26T00:20:06.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7fdd4640 10 monclient: shutdown 2024-04-26T00:20:06.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7fdd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:06.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7fdd4640 1 -- 172.21.15.37:0/1453136200 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d781379a0 msgr2=0x7f0d7813c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:06.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7fdd4640 1 --2- 172.21.15.37:0/1453136200 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d781379a0 0x7f0d7813c450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0d740098a0 tx=0x7f0d7401c1e0 comp rx=0 tx=0).stop 2024-04-26T00:20:06.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7fdd4640 1 -- 172.21.15.37:0/1453136200 shutdown_connections 2024-04-26T00:20:06.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7fdd4640 1 --2- 172.21.15.37:0/1453136200 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d781379a0 0x7f0d7813c450 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:06.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7fdd4640 1 -- 172.21.15.37:0/1453136200 >> 172.21.15.37:0/1453136200 conn(0x7f0d780a6300 msgr2=0x7f0d780a6710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:06.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7fdd4640 1 -- 172.21.15.37:0/1453136200 shutdown_connections 2024-04-26T00:20:06.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7fdd4640 1 -- 172.21.15.37:0/1453136200 wait complete. 2024-04-26T00:20:06.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7fdd4640 10 monclient: build_initial_monmap 2024-04-26T00:20:06.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7fdd4640 10 monclient: monmap: 2024-04-26T00:20:06.356 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:06.356 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:06.356 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:06.356 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:06.357 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:06.357 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:06.357 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:06.357 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:06.357 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:06.357 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:06.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7fdd4640 1 Processor -- start 2024-04-26T00:20:06.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7fdd4640 1 -- start start 2024-04-26T00:20:06.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7fdd4640 10 monclient: init 2024-04-26T00:20:06.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7fdd4640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:06.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7fdd4640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:20:06.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7fdd4640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d78136eb0 0x7f0d781cee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:06.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7fdd4640 10 monclient(hunting): picked mon.noname-a con 0x7f0d78136eb0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:06.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7fdd4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d781379a0 0x7f0d7813b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:06.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7fdd4640 10 monclient(hunting): picked mon.noname-b con 0x7f0d781379a0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:06.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7fdd4640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d7813c990 0x7f0d7813b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:06.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7fdd4640 10 monclient(hunting): picked mon.noname-c con 0x7f0d7813c990 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:06.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7fdd4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:06.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7fdd4640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f0d7809abf0 con 0x7f0d78136eb0 2024-04-26T00:20:06.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7fdd4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:06.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7fdd4640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0d78142cc0 con 0x7f0d781379a0 2024-04-26T00:20:06.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7fdd4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:06.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7fdd4640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0d7809ae10 con 0x7f0d7813c990 2024-04-26T00:20:06.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7fdd4640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:06.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7fdd4640 10 monclient(hunting): authenticate will time out at 987.679162s 2024-04-26T00:20:06.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7db70640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d78136eb0 0x7f0d781cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:06.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7db70640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d78136eb0 0x7f0d781cee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44834/0 (socket says 172.21.15.37:44834) 2024-04-26T00:20:06.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7db70640 1 -- 172.21.15.37:0/3567000392 learned_addr learned my addr 172.21.15.37:0/3567000392 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:06.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7db70640 10 monclient(hunting): get_auth_request con 0x7f0d78136eb0 auth_method 0 2024-04-26T00:20:06.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7db70640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:06.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7db70640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:06.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7db70640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:06.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7e371640 1 --2- 172.21.15.37:0/3567000392 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d7813c990 0x7f0d7813b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:06.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7e371640 10 monclient(hunting): get_auth_request con 0x7f0d7813c990 auth_method 0 2024-04-26T00:20:06.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7e371640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:06.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7e371640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:06.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7e371640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:06.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7db70640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:06.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7db70640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:06.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7db70640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:06.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7e371640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:06.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7e371640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:06.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7e371640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:06.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7db70640 10 monclient(hunting): handle_auth_done global_id 4308 payload 824 2024-04-26T00:20:06.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7d36f640 1 --2- 172.21.15.37:0/3567000392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d781379a0 0x7f0d7813b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:06.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7db70640 1 -- 172.21.15.37:0/3567000392 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d7813c990 msgr2=0x7f0d7813b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:06.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7db70640 1 --2- 172.21.15.37:0/3567000392 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d7813c990 0x7f0d7813b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:06.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7db70640 1 -- 172.21.15.37:0/3567000392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d781379a0 msgr2=0x7f0d7813b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:06.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7db70640 1 --2- 172.21.15.37:0/3567000392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d781379a0 0x7f0d7813b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:06.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7db70640 10 monclient: _finish_hunting 0 2024-04-26T00:20:06.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7db70640 1 monclient: found mon.noname-a 2024-04-26T00:20:06.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7db70640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:06.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7db70640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:06.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7db70640 1 -- 172.21.15.37:0/3567000392 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d780a33e0 con 0x7f0d78136eb0 2024-04-26T00:20:06.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.367+0000 7f0d7e371640 1 --2- 172.21.15.37:0/3567000392 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d7813c990 0x7f0d7813b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:06.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.371+0000 7f0d7db70640 1 -- 172.21.15.37:0/3567000392 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d78136eb0 msgr2=0x7f0d781cee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:06.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.371+0000 7f0d7db70640 1 --2- 172.21.15.37:0/3567000392 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d78136eb0 0x7f0d781cee80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c00c460 tx=0x7f0d6c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:06.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.371+0000 7f0d6a7fc640 1 -- 172.21.15.37:0/3567000392 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d6c017070 con 0x7f0d78136eb0 2024-04-26T00:20:06.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.371+0000 7f0d6a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:06.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.371+0000 7f0d6a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:06.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.371+0000 7f0d6a7fc640 10 monclient: dump: 2024-04-26T00:20:06.365 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:06.365 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:06.365 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:06.365 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:06.365 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:06.366 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:06.366 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:06.366 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:06.366 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:06.366 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:06.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.371+0000 7f0d6a7fc640 10 monclient: _finish_auth 0 2024-04-26T00:20:06.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.371+0000 7f0d6a7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:20:06.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.371+0000 7f0d6a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:06.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.371+0000 7f0d6a7fc640 1 -- 172.21.15.37:0/3567000392 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d6c005820 con 0x7f0d78136eb0 2024-04-26T00:20:06.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.371+0000 7f0d7fdd4640 5 monclient: authenticate success, global_id 4308 2024-04-26T00:20:06.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.371+0000 7f0d7fdd4640 10 monclient: _renew_subs 2024-04-26T00:20:06.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.371+0000 7f0d7fdd4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:06.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.371+0000 7f0d7fdd4640 1 -- 172.21.15.37:0/3567000392 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d7813c190 con 0x7f0d78136eb0 2024-04-26T00:20:06.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.371+0000 7f0d6a7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:06.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.371+0000 7f0d6a7fc640 1 -- 172.21.15.37:0/3567000392 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d6c005c10 con 0x7f0d78136eb0 2024-04-26T00:20:06.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.371+0000 7f0d6a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:06.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.371+0000 7f0d6a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:06.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.371+0000 7f0d6a7fc640 10 monclient: dump: 2024-04-26T00:20:06.367 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:06.367 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:06.367 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:06.368 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:06.368 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:06.368 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:06.368 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:06.368 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:06.368 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:06.368 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:06.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.371+0000 7f0d6b7fe640 4 set_mon_vals no callback set 2024-04-26T00:20:06.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.371+0000 7f0d7fdd4640 10 monclient: _renew_subs 2024-04-26T00:20:06.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.371+0000 7f0d7fdd4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:06.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.371+0000 7f0d7fdd4640 1 -- 172.21.15.37:0/3567000392 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d78139c60 con 0x7f0d78136eb0 2024-04-26T00:20:06.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.371+0000 7f0d7fdd4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:06.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.371+0000 7f0d7fdd4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:06.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.371+0000 7f0d7fdd4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:06.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.371+0000 7f0d7fdd4640 1 -- 172.21.15.37:0/3567000392 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d78066f40 con 0x7f0d78136eb0 2024-04-26T00:20:06.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.371+0000 7f0d6a7fc640 1 -- 172.21.15.37:0/3567000392 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f0d6c01a070 con 0x7f0d78136eb0 2024-04-26T00:20:06.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.371+0000 7f0d6a7fc640 1 --2- 172.21.15.37:0/3567000392 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0d5c035eb0 0x7f0d5c038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:06.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.371+0000 7f0d6a7fc640 1 -- 172.21.15.37:0/3567000392 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7f0d6c047d40 con 0x7f0d78136eb0 2024-04-26T00:20:06.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.371+0000 7f0d7d36f640 1 --2- 172.21.15.37:0/3567000392 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0d5c035eb0 0x7f0d5c038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:06.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.371+0000 7f0d7d36f640 10 monclient: get_auth_request con 0x7f0d5c035eb0 auth_method 0 2024-04-26T00:20:06.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.375+0000 7f0d7d36f640 1 --2- 172.21.15.37:0/3567000392 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0d5c035eb0 0x7f0d5c038370 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0d780a5990 tx=0x7f0d74008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:06.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.375+0000 7f0d6a7fc640 1 -- 172.21.15.37:0/3567000392 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f0d6c010300 con 0x7f0d78136eb0 2024-04-26T00:20:06.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.375+0000 7f0d6a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:06.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.375+0000 7f0d6a7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:06.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.527+0000 7f0d7fdd4640 1 -- 172.21.15.37:0/3567000392 --> v2:172.21.15.45:6812/2379918623 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0d78137690 con 0x7f0d5c035eb0 2024-04-26T00:20:06.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.531+0000 7f0d6a7fc640 1 -- 172.21.15.37:0/3567000392 <== mgr.4107 v2:172.21.15.45:6812/2379918623 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33873 (secure 0 0 0) 0x7f0d78137690 con 0x7f0d5c035eb0 2024-04-26T00:20:06.514 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:20:06.515 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-04-26T00:20:06.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.535+0000 7f0d7fdd4640 1 -- 172.21.15.37:0/3567000392 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0d5c035eb0 msgr2=0x7f0d5c038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:06.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.535+0000 7f0d7fdd4640 1 --2- 172.21.15.37:0/3567000392 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0d5c035eb0 0x7f0d5c038370 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0d780a5990 tx=0x7f0d74008040 comp rx=0 tx=0).stop 2024-04-26T00:20:06.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.535+0000 7f0d7fdd4640 10 monclient: shutdown 2024-04-26T00:20:06.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.535+0000 7f0d7fdd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:06.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.535+0000 7f0d7fdd4640 1 -- 172.21.15.37:0/3567000392 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d78136eb0 msgr2=0x7f0d781cee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:06.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.535+0000 7f0d7fdd4640 1 --2- 172.21.15.37:0/3567000392 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d78136eb0 0x7f0d781cee80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c00c460 tx=0x7f0d6c00c920 comp rx=0 tx=0).stop 2024-04-26T00:20:06.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.535+0000 7f0d7fdd4640 1 -- 172.21.15.37:0/3567000392 shutdown_connections 2024-04-26T00:20:06.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.535+0000 7f0d7fdd4640 1 --2- 172.21.15.37:0/3567000392 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0d5c035eb0 0x7f0d5c038370 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:06.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.535+0000 7f0d7fdd4640 1 --2- 172.21.15.37:0/3567000392 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d7813c990 0x7f0d7813b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:06.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.535+0000 7f0d7fdd4640 1 --2- 172.21.15.37:0/3567000392 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d781379a0 0x7f0d7813b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:06.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.535+0000 7f0d7fdd4640 1 --2- 172.21.15.37:0/3567000392 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d78136eb0 0x7f0d781cee80 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:06.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.535+0000 7f0d7fdd4640 1 -- 172.21.15.37:0/3567000392 >> 172.21.15.37:0/3567000392 conn(0x7f0d780a6300 msgr2=0x7f0d7809ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:06.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.535+0000 7f0d7fdd4640 1 -- 172.21.15.37:0/3567000392 shutdown_connections 2024-04-26T00:20:06.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.535+0000 7f0d7fdd4640 1 -- 172.21.15.37:0/3567000392 wait complete. 2024-04-26T00:20:06.531 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-04-26T00:20:05.149627+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":145,"num_write_kb":2154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105,"ondisk_log_size":105,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":167156,"kb_used_data":820,"kb_used_omap":9,"kb_used_meta":166198,"kb_avail":566063884,"statfs":{"total":579820584960,"available":579649417216,"internally_reserved":0,"allocated":839680,"data_stored":1071264,"data_compressed":46459,"data_compressed_allocated":61440,"data_compressed_original":839680,"omap_allocated":9568,"internal_metadata":170187424},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-26T00:20:02.476315+0000","last_change":"2024-04-26T00:20:02.475146+0000","last_active":"2024-04-26T00:20:02.476315+0000","last_peered":"2024-04-26T00:20:02.476315+0000","last_clean":"2024-04-26T00:20:02.476315+0000","last_became_active":"2024-04-26T00:20:02.473928+0000","last_became_peered":"2024-04-26T00:20:02.473928+0000","last_unstale":"2024-04-26T00:20:02.476315+0000","last_undegraded":"2024-04-26T00:20:02.476315+0000","last_fullsized":"2024-04-26T00:20:02.476315+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:50:14.258009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T00:20:03.547413+0000","last_change":"2024-04-26T00:20:02.279987+0000","last_active":"2024-04-26T00:20:03.547413+0000","last_peered":"2024-04-26T00:20:03.547413+0000","last_clean":"2024-04-26T00:20:03.547413+0000","last_became_active":"2024-04-26T00:20:02.279183+0000","last_became_peered":"2024-04-26T00:20:02.279183+0000","last_unstale":"2024-04-26T00:20:03.547413+0000","last_undegraded":"2024-04-26T00:20:03.547413+0000","last_fullsized":"2024-04-26T00:20:03.547413+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:45:44.508952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T00:20:03.268034+0000","last_change":"2024-04-26T00:20:02.474779+0000","last_active":"2024-04-26T00:20:03.268034+0000","last_peered":"2024-04-26T00:20:03.268034+0000","last_clean":"2024-04-26T00:20:03.268034+0000","last_became_active":"2024-04-26T00:20:02.473888+0000","last_became_peered":"2024-04-26T00:20:02.473888+0000","last_unstale":"2024-04-26T00:20:03.268034+0000","last_undegraded":"2024-04-26T00:20:03.268034+0000","last_fullsized":"2024-04-26T00:20:03.268034+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:33:18.962907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T00:20:03.266877+0000","last_change":"2024-04-26T00:20:02.274249+0000","last_active":"2024-04-26T00:20:03.266877+0000","last_peered":"2024-04-26T00:20:03.266877+0000","last_clean":"2024-04-26T00:20:03.266877+0000","last_became_active":"2024-04-26T00:20:02.273745+0000","last_became_peered":"2024-04-26T00:20:02.273745+0000","last_unstale":"2024-04-26T00:20:03.266877+0000","last_undegraded":"2024-04-26T00:20:03.266877+0000","last_fullsized":"2024-04-26T00:20:03.266877+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:36:25.948539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"14'105","reported_seq":147,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T00:20:03.547490+0000","last_change":"2024-04-26T00:20:02.281782+0000","last_active":"2024-04-26T00:20:03.547490+0000","last_peered":"2024-04-26T00:20:03.547490+0000","last_clean":"2024-04-26T00:20:03.547490+0000","last_became_active":"2024-04-26T00:20:02.281000+0000","last_became_peered":"2024-04-26T00:20:02.281000+0000","last_unstale":"2024-04-26T00:20:03.547490+0000","last_undegraded":"2024-04-26T00:20:03.547490+0000","last_fullsized":"2024-04-26T00:20:03.547490+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:40:30.790500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":145,"num_write_kb":2154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T00:20:03.267137+0000","last_change":"2024-04-26T00:20:02.273965+0000","last_active":"2024-04-26T00:20:03.267137+0000","last_peered":"2024-04-26T00:20:03.267137+0000","last_clean":"2024-04-26T00:20:03.267137+0000","last_became_active":"2024-04-26T00:20:02.273213+0000","last_became_peered":"2024-04-26T00:20:02.273213+0000","last_unstale":"2024-04-26T00:20:03.267137+0000","last_undegraded":"2024-04-26T00:20:03.267137+0000","last_fullsized":"2024-04-26T00:20:03.267137+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:42:14.055403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T00:20:03.268104+0000","last_change":"2024-04-26T00:20:02.277311+0000","last_active":"2024-04-26T00:20:03.268104+0000","last_peered":"2024-04-26T00:20:03.268104+0000","last_clean":"2024-04-26T00:20:03.268104+0000","last_became_active":"2024-04-26T00:20:02.276409+0000","last_became_peered":"2024-04-26T00:20:02.276409+0000","last_unstale":"2024-04-26T00:20:03.268104+0000","last_undegraded":"2024-04-26T00:20:03.268104+0000","last_fullsized":"2024-04-26T00:20:03.268104+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:14:50.584720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-26T00:20:02.281720+0000","last_change":"2024-04-26T00:20:02.279805+0000","last_active":"2024-04-26T00:20:02.281720+0000","last_peered":"2024-04-26T00:20:02.281720+0000","last_clean":"2024-04-26T00:20:02.281720+0000","last_became_active":"2024-04-26T00:20:02.279053+0000","last_became_peered":"2024-04-26T00:20:02.279053+0000","last_unstale":"2024-04-26T00:20:02.281720+0000","last_undegraded":"2024-04-26T00:20:02.281720+0000","last_fullsized":"2024-04-26T00:20:02.281720+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:00:44.241406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T00:20:03.547492+0000","last_change":"2024-04-26T00:20:02.279764+0000","last_active":"2024-04-26T00:20:03.547492+0000","last_peered":"2024-04-26T00:20:03.547492+0000","last_clean":"2024-04-26T00:20:03.547492+0000","last_became_active":"2024-04-26T00:20:02.278959+0000","last_became_peered":"2024-04-26T00:20:02.278959+0000","last_unstale":"2024-04-26T00:20:03.547492+0000","last_undegraded":"2024-04-26T00:20:03.547492+0000","last_fullsized":"2024-04-26T00:20:03.547492+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:23:43.610174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":145,"num_write_kb":2154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":934976,"data_compressed":55314,"data_compressed_allocated":73728,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":105,"ondisk_log_size":105,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640258,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":122880,"data_stored":33587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640258,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31300,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340540,"statfs":{"total":96636764160,"available":96604712960,"internally_reserved":0,"allocated":122880,"data_stored":33587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":159744,"data_stored":431427,"data_compressed":18802,"data_compressed_allocated":24576,"data_compressed_original":389120,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":122880,"data_stored":33587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640258,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":188416,"data_stored":505489,"data_compressed":27657,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":122880,"data_stored":33587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":467488,"data_compressed":27657,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":467488,"data_compressed":27657,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T00:20:06.532 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-26T00:20:06.532 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-26T00:20:06.532 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-26T00:20:06.533 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-26T00:20:06.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c4a488640 10 monclient: get_monmap_and_config 2024-04-26T00:20:06.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c4a488640 10 monclient: build_initial_monmap 2024-04-26T00:20:06.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c4a488640 10 monclient: monmap: 2024-04-26T00:20:06.642 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:06.642 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:06.642 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:06.665480+0000 2024-04-26T00:20:06.642 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:06.665480+0000 2024-04-26T00:20:06.642 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:06.642 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:06.642 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:06.642 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:06.642 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:06.642 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:06.643 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:06.643 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:06.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c4a488640 1 Processor -- start 2024-04-26T00:20:06.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c4a488640 1 -- start start 2024-04-26T00:20:06.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c4a488640 10 monclient: init 2024-04-26T00:20:06.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c4a488640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:06.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c4a488640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-04-26T00:20:06.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c4a488640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f1c44136e10 0x7f1c4409c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:06.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c4a488640 10 monclient(hunting): picked mon.noname-a con 0x7f1c44136e10 addr v2:172.21.15.37:3300/0 2024-04-26T00:20:06.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c4a488640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1c441371f0 0x7f1c44137600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:06.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c4a488640 10 monclient(hunting): picked mon.noname-c con 0x7f1c441371f0 addr v2:172.21.15.45:3300/0 2024-04-26T00:20:06.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c4a488640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1c44135d90 addr v1:172.21.15.37:6789/0 2024-04-26T00:20:06.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c4a488640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c4409b680 con 0x7f1c44135d90 2024-04-26T00:20:06.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c4a488640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:06.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c4a488640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f1c4409b8f0 con 0x7f1c44136e10 2024-04-26T00:20:06.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c4a488640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:06.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c4a488640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f1c4409bbd0 con 0x7f1c441371f0 2024-04-26T00:20:06.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c4a488640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:06.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c4a488640 10 monclient(hunting): authenticate will time out at 987.971114s 2024-04-26T00:20:06.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c43fff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f1c44136e10 0x7f1c4409c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:06.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c48a25640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f1c44135d90 0x7f1c44136220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:55838/0 (socket says 172.21.15.37:55838) 2024-04-26T00:20:06.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c48a25640 1 -- 172.21.15.37:0/3203121928 learned_addr learned my addr 172.21.15.37:0/3203121928 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:20:06.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c43fff640 10 monclient(hunting): get_auth_request con 0x7f1c44136e10 auth_method 0 2024-04-26T00:20:06.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:06.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c43fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:06.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c43fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:06.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c43fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:06.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c43fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:06.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c43fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:06.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c3bfff640 1 --2- 172.21.15.37:0/3203121928 >> v2:172.21.15.45:3300/0 conn(0x7f1c441371f0 0x7f1c44137600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:06.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c3bfff640 10 monclient(hunting): get_auth_request con 0x7f1c441371f0 auth_method 0 2024-04-26T00:20:06.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:06.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c3bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:06.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:06.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c43fff640 10 monclient(hunting): handle_auth_done global_id 4311 payload 274 2024-04-26T00:20:06.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c43fff640 1 -- 172.21.15.37:0/3203121928 >> v2:172.21.15.45:3300/0 conn(0x7f1c441371f0 msgr2=0x7f1c44137600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:06.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c43fff640 1 --2- 172.21.15.37:0/3203121928 >> v2:172.21.15.45:3300/0 conn(0x7f1c441371f0 0x7f1c44137600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:06.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c43fff640 1 -- 172.21.15.37:0/3203121928 >> v1:172.21.15.37:6789/0 conn(0x7f1c44135d90 legacy=0x7f1c44136220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:06.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c43fff640 10 monclient: _finish_hunting 0 2024-04-26T00:20:06.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c43fff640 1 monclient: found mon.noname-a 2024-04-26T00:20:06.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:06.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c43fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:06.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.659+0000 7f1c43fff640 1 -- 172.21.15.37:0/3203121928 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c44137b40 con 0x7f1c44136e10 2024-04-26T00:20:06.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c43fff640 1 -- 172.21.15.37:0/3203121928 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c44136e10 msgr2=0x7f1c4409c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:06.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c43fff640 1 --2- 172.21.15.37:0/3203121928 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c44136e10 0x7f1c4409c5b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1c44068e60 tx=0x7f1c2c01c300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12c9106ad185d4da server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:06.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c437fe640 1 -- 172.21.15.37:0/3203121928 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c2c0b0070 con 0x7f1c44136e10 2024-04-26T00:20:06.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:06.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c437fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:06.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c437fe640 10 monclient: dump: 2024-04-26T00:20:06.650 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:06.651 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:06.651 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:06.651 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:06.651 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:06.651 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:06.651 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:06.651 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:06.651 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:06.651 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:06.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c437fe640 10 monclient: _finish_auth 0 2024-04-26T00:20:06.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c437fe640 10 monclient: _check_auth_tickets 2024-04-26T00:20:06.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:06.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c4a488640 5 monclient: authenticate success, global_id 4311 2024-04-26T00:20:06.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c4a488640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:20:06.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c437fe640 1 -- 172.21.15.37:0/3203121928 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c2c0ab600 con 0x7f1c44136e10 2024-04-26T00:20:06.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c437fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:06.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c4a488640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:06.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c4a488640 4 set_mon_vals no callback set 2024-04-26T00:20:06.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c437fe640 1 -- 172.21.15.37:0/3203121928 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c2c0ab9d0 con 0x7f1c44136e10 2024-04-26T00:20:06.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:06.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c437fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:06.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c437fe640 10 monclient: dump: 2024-04-26T00:20:06.652 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:06.652 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:06.652 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:06.652 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:06.652 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:06.652 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:06.652 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:06.653 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:06.653 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:06.653 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:06.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c4a488640 10 monclient: shutdown 2024-04-26T00:20:06.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c4a488640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:06.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c4a488640 1 -- 172.21.15.37:0/3203121928 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c44136e10 msgr2=0x7f1c4409c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:06.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c4a488640 1 --2- 172.21.15.37:0/3203121928 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c44136e10 0x7f1c4409c5b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1c44068e60 tx=0x7f1c2c01c300 comp rx=0 tx=0).stop 2024-04-26T00:20:06.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c4a488640 1 -- 172.21.15.37:0/3203121928 shutdown_connections 2024-04-26T00:20:06.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c4a488640 1 --2- 172.21.15.37:0/3203121928 >> v2:172.21.15.45:3300/0 conn(0x7f1c441371f0 0x7f1c44137600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:06.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c4a488640 1 --2- 172.21.15.37:0/3203121928 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c44136e10 0x7f1c4409c5b0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:06.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c4a488640 1 -- 172.21.15.37:0/3203121928 >> 172.21.15.37:0/3203121928 conn(0x7f1c440a64e0 msgr2=0x7f1c440a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:06.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c4a488640 1 -- 172.21.15.37:0/3203121928 shutdown_connections 2024-04-26T00:20:06.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c4a488640 1 -- 172.21.15.37:0/3203121928 wait complete. 2024-04-26T00:20:06.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c4a488640 10 monclient: build_initial_monmap 2024-04-26T00:20:06.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c4a488640 10 monclient: monmap: 2024-04-26T00:20:06.654 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:06.654 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:06.654 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:06.654 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:06.654 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:06.654 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:06.654 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:06.654 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:06.654 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:06.654 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:06.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c4a488640 1 Processor -- start 2024-04-26T00:20:06.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c4a488640 1 -- start start 2024-04-26T00:20:06.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c4a488640 10 monclient: init 2024-04-26T00:20:06.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c4a488640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:06.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c4a488640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:20:06.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c4a488640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1c44135d90 0x7f1c441cf2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:06.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c4a488640 10 monclient(hunting): picked mon.noname-c con 0x7f1c44135d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:06.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c4a488640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1c44136e10 0x7f1c44134d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:06.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c4a488640 10 monclient(hunting): picked mon.noname-b con 0x7f1c44136e10 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:06.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c4a488640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c441371f0 0x7f1c4413a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:06.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c4a488640 10 monclient(hunting): picked mon.noname-a con 0x7f1c441371f0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:06.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c4a488640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:06.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c4a488640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f1c44141f10 con 0x7f1c441371f0 2024-04-26T00:20:06.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c4a488640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:06.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c4a488640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1c44141d90 con 0x7f1c44136e10 2024-04-26T00:20:06.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c4a488640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:06.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c4a488640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f1c44142090 con 0x7f1c44135d90 2024-04-26T00:20:06.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c4a488640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:06.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c4a488640 10 monclient(hunting): authenticate will time out at 987.979188s 2024-04-26T00:20:06.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c43fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1c44135d90 0x7f1c441cf2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:06.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c48a25640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c441371f0 0x7f1c4413a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:06.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c43fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1c44135d90 0x7f1c441cf2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:58034/0 (socket says 172.21.15.37:58034) 2024-04-26T00:20:06.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c43fff640 1 -- 172.21.15.37:0/2511697009 learned_addr learned my addr 172.21.15.37:0/2511697009 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:06.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c43fff640 10 monclient(hunting): get_auth_request con 0x7f1c44135d90 auth_method 0 2024-04-26T00:20:06.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:06.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c43fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:06.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c43fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:06.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c48a25640 10 monclient(hunting): get_auth_request con 0x7f1c441371f0 auth_method 0 2024-04-26T00:20:06.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c48a25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:06.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c48a25640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:06.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c48a25640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:06.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c43fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:06.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c43fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:06.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c43fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:06.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c48a25640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:06.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c48a25640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:06.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c48a25640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:06.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c3bfff640 1 --2- 172.21.15.37:0/2511697009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1c44136e10 0x7f1c44134d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:06.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c3bfff640 10 monclient(hunting): get_auth_request con 0x7f1c44136e10 auth_method 0 2024-04-26T00:20:06.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:06.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c3bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:06.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:06.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c43fff640 10 monclient(hunting): handle_auth_done global_id 4295 payload 824 2024-04-26T00:20:06.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c43fff640 1 -- 172.21.15.37:0/2511697009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1c44136e10 msgr2=0x7f1c44134d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:06.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c43fff640 1 --2- 172.21.15.37:0/2511697009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1c44136e10 0x7f1c44134d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:06.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c43fff640 1 -- 172.21.15.37:0/2511697009 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c441371f0 msgr2=0x7f1c4413a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:06.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c43fff640 1 --2- 172.21.15.37:0/2511697009 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c441371f0 0x7f1c4413a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:06.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c43fff640 10 monclient: _finish_hunting 0 2024-04-26T00:20:06.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c43fff640 1 monclient: found mon.noname-c 2024-04-26T00:20:06.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:06.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c43fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:20:06.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c43fff640 1 -- 172.21.15.37:0/2511697009 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c441d01b0 con 0x7f1c44135d90 2024-04-26T00:20:06.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.667+0000 7f1c48a25640 1 --2- 172.21.15.37:0/2511697009 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c441371f0 0x7f1c4413a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:06.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.671+0000 7f1c43fff640 1 -- 172.21.15.37:0/2511697009 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1c44135d90 msgr2=0x7f1c441cf2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:06.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.671+0000 7f1c43fff640 1 --2- 172.21.15.37:0/2511697009 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1c44135d90 0x7f1c441cf2c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c019f30 tx=0x7f1c2c0190a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:06.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.671+0000 7f1c417fa640 1 -- 172.21.15.37:0/2511697009 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c2c0b8020 con 0x7f1c44135d90 2024-04-26T00:20:06.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.671+0000 7f1c417fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:06.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.671+0000 7f1c417fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:20:06.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.671+0000 7f1c417fa640 10 monclient: dump: 2024-04-26T00:20:06.662 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:06.662 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:06.662 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:06.662 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:06.662 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:06.663 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:06.663 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:06.663 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:06.663 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:06.663 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:06.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.671+0000 7f1c417fa640 10 monclient: _finish_auth 0 2024-04-26T00:20:06.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.671+0000 7f1c417fa640 10 monclient: _check_auth_tickets 2024-04-26T00:20:06.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.671+0000 7f1c417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:06.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.671+0000 7f1c4a488640 5 monclient: authenticate success, global_id 4295 2024-04-26T00:20:06.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.671+0000 7f1c417fa640 1 -- 172.21.15.37:0/2511697009 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c2c0b4810 con 0x7f1c44135d90 2024-04-26T00:20:06.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.671+0000 7f1c417fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:06.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.671+0000 7f1c4a488640 10 monclient: _renew_subs 2024-04-26T00:20:06.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.671+0000 7f1c417fa640 1 -- 172.21.15.37:0/2511697009 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c2c0b0040 con 0x7f1c44135d90 2024-04-26T00:20:06.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.671+0000 7f1c4a488640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:06.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.671+0000 7f1c4a488640 1 -- 172.21.15.37:0/2511697009 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c4409aca0 con 0x7f1c44135d90 2024-04-26T00:20:06.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.671+0000 7f1c417fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:06.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.671+0000 7f1c427fc640 4 set_mon_vals no callback set 2024-04-26T00:20:06.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.671+0000 7f1c417fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:06.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.671+0000 7f1c417fa640 10 monclient: dump: 2024-04-26T00:20:06.664 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:06.664 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:06.664 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:06.664 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:06.664 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:06.664 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:06.664 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:06.664 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:06.664 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:06.664 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:06.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.671+0000 7f1c4a488640 10 monclient: _renew_subs 2024-04-26T00:20:06.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.671+0000 7f1c4a488640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:06.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.671+0000 7f1c4a488640 1 -- 172.21.15.37:0/2511697009 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c4413e950 con 0x7f1c44135d90 2024-04-26T00:20:06.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.671+0000 7f1c4a488640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:06.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.671+0000 7f1c4a488640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:06.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.671+0000 7f1c4a488640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:06.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.671+0000 7f1c4a488640 1 -- 172.21.15.37:0/2511697009 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c44075910 con 0x7f1c44135d90 2024-04-26T00:20:06.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.671+0000 7f1c417fa640 1 -- 172.21.15.37:0/2511697009 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f1c2c0bb020 con 0x7f1c44135d90 2024-04-26T00:20:06.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.671+0000 7f1c417fa640 1 --2- 172.21.15.37:0/2511697009 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1c24035e80 0x7f1c24038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:06.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.671+0000 7f1c417fa640 1 -- 172.21.15.37:0/2511697009 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7f1c2c0e25b0 con 0x7f1c44135d90 2024-04-26T00:20:06.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.671+0000 7f1c3bfff640 1 --2- 172.21.15.37:0/2511697009 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1c24035e80 0x7f1c24038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:06.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.671+0000 7f1c3bfff640 10 monclient: get_auth_request con 0x7f1c24035e80 auth_method 0 2024-04-26T00:20:06.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.675+0000 7f1c3bfff640 1 --2- 172.21.15.37:0/2511697009 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1c24035e80 0x7f1c24038340 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1c4413c140 tx=0x7f1c34008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:06.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.675+0000 7f1c417fa640 1 -- 172.21.15.37:0/2511697009 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f1c2c0c2030 con 0x7f1c44135d90 2024-04-26T00:20:06.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.675+0000 7f1c417fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:06.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.675+0000 7f1c417fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:06.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.867+0000 7f1c4a488640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-04-26T00:20:06.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.867+0000 7f1c4a488640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-04-26T00:20:06.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.867+0000 7f1c4a488640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:06.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.867+0000 7f1c4a488640 1 -- 172.21.15.37:0/2511697009 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f1c44066f40 con 0x7f1c44135d90 2024-04-26T00:20:06.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.867+0000 7f1c417fa640 1 -- 172.21.15.37:0/2511697009 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f1c2c016010 con 0x7f1c44135d90 2024-04-26T00:20:06.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.867+0000 7f1c417fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-04-26T00:20:06.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.867+0000 7f1c417fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:06.851 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:20:06.851 INFO:teuthology.orchestra.run.smithi037.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-26T00:20:06.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.871+0000 7f1c4a488640 1 -- 172.21.15.37:0/2511697009 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1c24035e80 msgr2=0x7f1c24038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:06.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.871+0000 7f1c4a488640 1 --2- 172.21.15.37:0/2511697009 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1c24035e80 0x7f1c24038340 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1c4413c140 tx=0x7f1c34008040 comp rx=0 tx=0).stop 2024-04-26T00:20:06.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.871+0000 7f1c4a488640 10 monclient: shutdown 2024-04-26T00:20:06.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.871+0000 7f1c4a488640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:06.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.871+0000 7f1c4a488640 1 -- 172.21.15.37:0/2511697009 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1c44135d90 msgr2=0x7f1c441cf2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:06.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.871+0000 7f1c4a488640 1 --2- 172.21.15.37:0/2511697009 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1c44135d90 0x7f1c441cf2c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c019f30 tx=0x7f1c2c0190a0 comp rx=0 tx=0).stop 2024-04-26T00:20:06.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.871+0000 7f1c4a488640 1 -- 172.21.15.37:0/2511697009 shutdown_connections 2024-04-26T00:20:06.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.871+0000 7f1c4a488640 1 --2- 172.21.15.37:0/2511697009 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1c24035e80 0x7f1c24038340 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:06.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.871+0000 7f1c4a488640 1 --2- 172.21.15.37:0/2511697009 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c441371f0 0x7f1c4413a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:06.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.871+0000 7f1c4a488640 1 --2- 172.21.15.37:0/2511697009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1c44136e10 0x7f1c44134d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:06.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.871+0000 7f1c4a488640 1 --2- 172.21.15.37:0/2511697009 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1c44135d90 0x7f1c441cf2c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:06.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.871+0000 7f1c4a488640 1 -- 172.21.15.37:0/2511697009 >> 172.21.15.37:0/2511697009 conn(0x7f1c440a64e0 msgr2=0x7f1c440a4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:06.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.871+0000 7f1c4a488640 1 -- 172.21.15.37:0/2511697009 shutdown_connections 2024-04-26T00:20:06.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:06.871+0000 7f1c4a488640 1 -- 172.21.15.37:0/2511697009 wait complete. 2024-04-26T00:20:06.867 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-26T00:20:06.868 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-04-26T00:20:06.880 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-04-26T00:20:06.881 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T00:20:06.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe34ddc2640 10 monclient: get_monmap_and_config 2024-04-26T00:20:06.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe34ddc2640 10 monclient: build_initial_monmap 2024-04-26T00:20:06.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe34ddc2640 10 monclient: monmap: 2024-04-26T00:20:06.989 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:06.989 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:06.990 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:07.013126+0000 2024-04-26T00:20:06.990 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:07.013126+0000 2024-04-26T00:20:06.990 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:06.990 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:06.990 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:06.990 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:06.990 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:06.990 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:06.990 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:06.990 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:06.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe34ddc2640 1 Processor -- start 2024-04-26T00:20:06.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe34ddc2640 1 -- start start 2024-04-26T00:20:06.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe34ddc2640 10 monclient: init 2024-04-26T00:20:06.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe34ddc2640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:06.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe34ddc2640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-04-26T00:20:06.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe34ddc2640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe348138330 0x7fe348138710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:06.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe34ddc2640 10 monclient(hunting): picked mon.noname-b con 0x7fe348138330 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:06.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe34ddc2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe34809de00 addr v1:172.21.15.45:6789/0 2024-04-26T00:20:06.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe34ddc2640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe3480a00e0 0x7fe3481434f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:06.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe34ddc2640 10 monclient(hunting): picked mon.noname-c con 0x7fe3480a00e0 addr v2:172.21.15.45:3300/0 2024-04-26T00:20:06.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe34ddc2640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe34809cb80 con 0x7fe34809de00 2024-04-26T00:20:06.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe34ddc2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:06.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe34ddc2640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fe34809d4b0 con 0x7fe3480a00e0 2024-04-26T00:20:06.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe34ddc2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:06.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe34ddc2640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe34809ce60 con 0x7fe348138330 2024-04-26T00:20:06.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe34ddc2640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:06.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe34ddc2640 10 monclient(hunting): authenticate will time out at 988.318766s 2024-04-26T00:20:06.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe3477fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe348138330 0x7fe348138710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:06.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe3477fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe348138330 0x7fe348138710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:58038/0 (socket says 172.21.15.37:58038) 2024-04-26T00:20:06.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe3477fe640 1 -- 172.21.15.37:0/1425174781 learned_addr learned my addr 172.21.15.37:0/1425174781 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:06.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe3477fe640 10 monclient(hunting): get_auth_request con 0x7fe348138330 auth_method 0 2024-04-26T00:20:06.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe3477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:06.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe3477fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:06.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe3477fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:06.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe3477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:06.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe3477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:06.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe3477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:06.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe347fff640 1 --2- 172.21.15.37:0/1425174781 >> v2:172.21.15.45:3300/0 conn(0x7fe3480a00e0 0x7fe3481434f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:06.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe347fff640 10 monclient(hunting): get_auth_request con 0x7fe3480a00e0 auth_method 0 2024-04-26T00:20:06.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe347fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:06.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe347fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:06.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe347fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:06.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe3477fe640 10 monclient(hunting): handle_auth_done global_id 4298 payload 274 2024-04-26T00:20:06.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe3477fe640 1 -- 172.21.15.37:0/1425174781 >> v2:172.21.15.45:3300/0 conn(0x7fe3480a00e0 msgr2=0x7fe3481434f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:06.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe3477fe640 1 --2- 172.21.15.37:0/1425174781 >> v2:172.21.15.45:3300/0 conn(0x7fe3480a00e0 0x7fe3481434f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:06.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe3477fe640 1 -- 172.21.15.37:0/1425174781 >> v1:172.21.15.45:6789/0 conn(0x7fe34809de00 legacy=0x7fe34809f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:06.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe3477fe640 10 monclient: _finish_hunting 0 2024-04-26T00:20:06.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe3477fe640 1 monclient: found mon.noname-b 2024-04-26T00:20:06.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe3477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:06.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe3477fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:20:06.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.007+0000 7fe3477fe640 1 -- 172.21.15.37:0/1425174781 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe34809d140 con 0x7fe348138330 2024-04-26T00:20:06.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe3477fe640 1 -- 172.21.15.37:0/1425174781 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe348138330 msgr2=0x7fe348138710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:06.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe3477fe640 1 --2- 172.21.15.37:0/1425174781 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe348138330 0x7fe348138710 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe338009870 tx=0x7fe33801c0c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c4e4dc820722f46 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:06.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe3467fc640 1 -- 172.21.15.37:0/1425174781 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3380b1070 con 0x7fe348138330 2024-04-26T00:20:06.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe3467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:06.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe3467fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:20:06.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe3467fc640 10 monclient: dump: 2024-04-26T00:20:06.997 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:06.997 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:06.998 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:06.998 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:06.998 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:06.998 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:06.998 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:06.998 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:06.998 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:06.998 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:06.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe3467fc640 10 monclient: _finish_auth 0 2024-04-26T00:20:06.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe3467fc640 10 monclient: _check_auth_tickets 2024-04-26T00:20:06.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe3467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:06.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe3467fc640 1 -- 172.21.15.37:0/1425174781 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3380a4830 con 0x7fe348138330 2024-04-26T00:20:06.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe34ddc2640 5 monclient: authenticate success, global_id 4298 2024-04-26T00:20:06.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe34ddc2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:20:06.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe3467fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:06.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe34ddc2640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:06.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe34ddc2640 4 set_mon_vals no callback set 2024-04-26T00:20:06.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe3467fc640 1 -- 172.21.15.37:0/1425174781 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3380a4c00 con 0x7fe348138330 2024-04-26T00:20:06.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe3467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:06.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe3467fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:06.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe3467fc640 10 monclient: dump: 2024-04-26T00:20:06.999 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:06.999 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:06.999 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:06.999 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:06.999 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:06.999 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:06.999 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:06.999 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:06.999 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:07.000 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:07.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe34ddc2640 10 monclient: shutdown 2024-04-26T00:20:07.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe34ddc2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:07.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe34ddc2640 1 -- 172.21.15.37:0/1425174781 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe348138330 msgr2=0x7fe348138710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:07.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe34ddc2640 1 --2- 172.21.15.37:0/1425174781 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe348138330 0x7fe348138710 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe338009870 tx=0x7fe33801c0c0 comp rx=0 tx=0).stop 2024-04-26T00:20:07.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe34ddc2640 1 -- 172.21.15.37:0/1425174781 shutdown_connections 2024-04-26T00:20:07.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe34ddc2640 1 --2- 172.21.15.37:0/1425174781 >> v2:172.21.15.45:3300/0 conn(0x7fe3480a00e0 0x7fe3481434f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:07.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe34ddc2640 1 --2- 172.21.15.37:0/1425174781 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe348138330 0x7fe348138710 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:07.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe34ddc2640 1 -- 172.21.15.37:0/1425174781 >> 172.21.15.37:0/1425174781 conn(0x7fe3480a1ad0 msgr2=0x7fe3480a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:07.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe34ddc2640 1 -- 172.21.15.37:0/1425174781 shutdown_connections 2024-04-26T00:20:07.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe34ddc2640 1 -- 172.21.15.37:0/1425174781 wait complete. 2024-04-26T00:20:07.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe34ddc2640 10 monclient: build_initial_monmap 2024-04-26T00:20:07.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe34ddc2640 10 monclient: monmap: 2024-04-26T00:20:07.004 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:07.004 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:07.004 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:07.004 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:07.004 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:07.004 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:07.004 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:07.004 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:07.004 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:07.004 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:07.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe34ddc2640 1 Processor -- start 2024-04-26T00:20:07.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe34ddc2640 1 -- start start 2024-04-26T00:20:07.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe34ddc2640 10 monclient: init 2024-04-26T00:20:07.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe34ddc2640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:07.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe34ddc2640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:20:07.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe34ddc2640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe34809de00 0x7fe3481d4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:07.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe34ddc2640 10 monclient(hunting): picked mon.noname-a con 0x7fe34809de00 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:07.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe34ddc2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3480a00e0 0x7fe348142400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:07.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe34ddc2640 10 monclient(hunting): picked mon.noname-b con 0x7fe3480a00e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:07.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe34ddc2640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe348138330 0x7fe34813cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:07.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe34ddc2640 10 monclient(hunting): picked mon.noname-c con 0x7fe348138330 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:07.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe34ddc2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:07.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe34ddc2640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe348146e00 con 0x7fe34809de00 2024-04-26T00:20:07.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe34ddc2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:07.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe34ddc2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe348146c80 con 0x7fe3480a00e0 2024-04-26T00:20:07.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe34ddc2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:07.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe34ddc2640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe348146f80 con 0x7fe348138330 2024-04-26T00:20:07.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe34ddc2640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:07.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe34ddc2640 10 monclient(hunting): authenticate will time out at 988.327106s 2024-04-26T00:20:07.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe3477fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe34809de00 0x7fe3481d4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:07.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe3477fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe34809de00 0x7fe3481d4190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44864/0 (socket says 172.21.15.37:44864) 2024-04-26T00:20:07.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe3477fe640 1 -- 172.21.15.37:0/579997488 learned_addr learned my addr 172.21.15.37:0/579997488 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:07.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe3477fe640 10 monclient(hunting): get_auth_request con 0x7fe34809de00 auth_method 0 2024-04-26T00:20:07.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe3477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:07.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe3477fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:07.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe3477fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:07.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe347fff640 1 --2- 172.21.15.37:0/579997488 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe348138330 0x7fe34813cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:07.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe347fff640 10 monclient(hunting): get_auth_request con 0x7fe348138330 auth_method 0 2024-04-26T00:20:07.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe347fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:07.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe347fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:07.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe347fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:07.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe3477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:07.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe3477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:07.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe3477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:07.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe347fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:07.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe347fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:07.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe347fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:07.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe346ffd640 1 --2- 172.21.15.37:0/579997488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3480a00e0 0x7fe348142400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:07.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe3477fe640 10 monclient(hunting): handle_auth_done global_id 4317 payload 824 2024-04-26T00:20:07.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe3477fe640 1 -- 172.21.15.37:0/579997488 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe348138330 msgr2=0x7fe34813cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:07.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe3477fe640 1 --2- 172.21.15.37:0/579997488 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe348138330 0x7fe34813cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:07.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe3477fe640 1 -- 172.21.15.37:0/579997488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3480a00e0 msgr2=0x7fe348142400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:07.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe3477fe640 1 --2- 172.21.15.37:0/579997488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3480a00e0 0x7fe348142400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:07.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe3477fe640 10 monclient: _finish_hunting 0 2024-04-26T00:20:07.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe3477fe640 1 monclient: found mon.noname-a 2024-04-26T00:20:07.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe3477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:07.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe3477fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:07.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe3477fe640 1 -- 172.21.15.37:0/579997488 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe348140ed0 con 0x7fe34809de00 2024-04-26T00:20:07.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe346ffd640 10 monclient: get_auth_request con 0x7fe3480a00e0 auth_method 0 2024-04-26T00:20:07.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.015+0000 7fe346ffd640 1 --2- 172.21.15.37:0/579997488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3480a00e0 0x7fe348142400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:20:07.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.019+0000 7fe3477fe640 1 -- 172.21.15.37:0/579997488 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe34809de00 msgr2=0x7fe3481d4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:07.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.019+0000 7fe3477fe640 1 --2- 172.21.15.37:0/579997488 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe34809de00 0x7fe3481d4190 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe3380af040 tx=0x7fe338019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:07.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.019+0000 7fe32ffff640 1 -- 172.21.15.37:0/579997488 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3380b1040 con 0x7fe34809de00 2024-04-26T00:20:07.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.019+0000 7fe32ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:07.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.019+0000 7fe32ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:07.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.019+0000 7fe32ffff640 10 monclient: dump: 2024-04-26T00:20:07.009 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:07.009 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:07.009 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:07.009 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:07.009 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:07.009 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:07.009 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:07.009 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:07.009 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:07.009 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:07.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.019+0000 7fe32ffff640 10 monclient: _finish_auth 0 2024-04-26T00:20:07.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.019+0000 7fe32ffff640 10 monclient: _check_auth_tickets 2024-04-26T00:20:07.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.019+0000 7fe32ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:07.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.019+0000 7fe32ffff640 1 -- 172.21.15.37:0/579997488 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3380ac660 con 0x7fe34809de00 2024-04-26T00:20:07.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.019+0000 7fe34ddc2640 5 monclient: authenticate success, global_id 4317 2024-04-26T00:20:07.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.019+0000 7fe34ddc2640 10 monclient: _renew_subs 2024-04-26T00:20:07.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.019+0000 7fe34ddc2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:07.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.019+0000 7fe34ddc2640 1 -- 172.21.15.37:0/579997488 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe34813f4c0 con 0x7fe34809de00 2024-04-26T00:20:07.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.019+0000 7fe32ffff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:07.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.019+0000 7fe32ffff640 1 -- 172.21.15.37:0/579997488 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3380aca50 con 0x7fe34809de00 2024-04-26T00:20:07.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.019+0000 7fe32ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:07.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.019+0000 7fe32ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:07.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.019+0000 7fe32ffff640 10 monclient: dump: 2024-04-26T00:20:07.010 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:07.011 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:07.011 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:07.011 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:07.011 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:07.011 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:07.011 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:07.011 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:07.011 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:07.011 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:07.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.019+0000 7fe34ddc2640 10 monclient: _renew_subs 2024-04-26T00:20:07.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.019+0000 7fe34ddc2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:07.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.019+0000 7fe34ddc2640 1 -- 172.21.15.37:0/579997488 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3481d48b0 con 0x7fe34809de00 2024-04-26T00:20:07.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.019+0000 7fe3457fa640 4 set_mon_vals no callback set 2024-04-26T00:20:07.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.019+0000 7fe34ddc2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:07.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.019+0000 7fe34ddc2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:07.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.019+0000 7fe34ddc2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:07.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.019+0000 7fe34ddc2640 1 -- 172.21.15.37:0/579997488 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe348066f40 con 0x7fe34809de00 2024-04-26T00:20:07.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.019+0000 7fe32ffff640 1 -- 172.21.15.37:0/579997488 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe3380bc060 con 0x7fe34809de00 2024-04-26T00:20:07.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.019+0000 7fe32ffff640 1 --2- 172.21.15.37:0/579997488 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe334035e80 0x7fe334038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:07.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.019+0000 7fe32ffff640 1 -- 172.21.15.37:0/579997488 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7fe3380e1190 con 0x7fe34809de00 2024-04-26T00:20:07.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.023+0000 7fe346ffd640 1 --2- 172.21.15.37:0/579997488 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe334035e80 0x7fe334038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:07.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.023+0000 7fe346ffd640 10 monclient: get_auth_request con 0x7fe334035e80 auth_method 0 2024-04-26T00:20:07.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.023+0000 7fe346ffd640 1 --2- 172.21.15.37:0/579997488 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe334035e80 0x7fe334038340 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe3481410e0 tx=0x7fe33c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:07.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.023+0000 7fe32ffff640 1 -- 172.21.15.37:0/579997488 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe3380160f0 con 0x7fe34809de00 2024-04-26T00:20:07.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.023+0000 7fe32ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:07.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.023+0000 7fe32ffff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:07.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.183+0000 7fe34ddc2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-26T00:20:07.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.183+0000 7fe34ddc2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-26T00:20:07.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.183+0000 7fe34ddc2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:07.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.183+0000 7fe34ddc2640 1 -- 172.21.15.37:0/579997488 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe3481430f0 con 0x7fe34809de00 2024-04-26T00:20:07.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.183+0000 7fe32ffff640 1 -- 172.21.15.37:0/579997488 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) ==== 74+0+10761 (secure 0 0 0) 0x7fe3380df0d0 con 0x7fe34809de00 2024-04-26T00:20:07.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.183+0000 7fe32ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-26T00:20:07.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.183+0000 7fe32ffff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:07.166 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:20:07.167 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":15,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","created":"2024-04-26T00:19:50.025527+0000","modified":"2024-04-26T00:20:03.259969+0000","last_up_change":"2024-04-26T00:19:59.222603+0000","last_in_change":"2024-04-26T00:19:52.138726+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-26T00:20:00.477727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-04-26T00:20:01.233750+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a5d4b9d3-5499-491f-a123-85d73de452af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":3204000569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":3204000569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6811","nonce":3204000569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":3204000569}]},"public_addr":"172.21.15.37:6806/3204000569","cluster_addr":"172.21.15.37:6808/3204000569","heartbeat_back_addr":"172.21.15.37:6811/3204000569","heartbeat_front_addr":"172.21.15.37:6810/3204000569","state":["exists","up"]},{"osd":1,"uuid":"cb4583fd-a0fa-4559-9f99-5a3e02b5818c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":1084527868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6805","nonce":1084527868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6809","nonce":1084527868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6807","nonce":1084527868}]},"public_addr":"172.21.15.37:6804/1084527868","cluster_addr":"172.21.15.37:6805/1084527868","heartbeat_back_addr":"172.21.15.37:6809/1084527868","heartbeat_front_addr":"172.21.15.37:6807/1084527868","state":["exists","up"]},{"osd":2,"uuid":"78fc9fe6-2163-4cb3-9944-3773bd488df8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":603175474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6801","nonce":603175474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6803","nonce":603175474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":603175474}]},"public_addr":"172.21.15.37:6800/603175474","cluster_addr":"172.21.15.37:6801/603175474","heartbeat_back_addr":"172.21.15.37:6803/603175474","heartbeat_front_addr":"172.21.15.37:6802/603175474","state":["exists","up"]},{"osd":3,"uuid":"338bb632-eca1-422b-b196-fd6cadb8307c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":2173737584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6801","nonce":2173737584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6803","nonce":2173737584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":2173737584}]},"public_addr":"172.21.15.45:6800/2173737584","cluster_addr":"172.21.15.45:6801/2173737584","heartbeat_back_addr":"172.21.15.45:6803/2173737584","heartbeat_front_addr":"172.21.15.45:6802/2173737584","state":["exists","up"]},{"osd":4,"uuid":"ce2e0db7-d486-4cb1-9913-6a4a011e5c8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":152605921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6805","nonce":152605921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6807","nonce":152605921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":152605921}]},"public_addr":"172.21.15.45:6804/152605921","cluster_addr":"172.21.15.45:6805/152605921","heartbeat_back_addr":"172.21.15.45:6807/152605921","heartbeat_front_addr":"172.21.15.45:6806/152605921","state":["exists","up"]},{"osd":5,"uuid":"a61739cc-41d3-48b1-a471-4092b2edbf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3915746722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6809","nonce":3915746722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6811","nonce":3915746722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3915746722}]},"public_addr":"172.21.15.45:6808/3915746722","cluster_addr":"172.21.15.45:6809/3915746722","heartbeat_back_addr":"172.21.15.45:6811/3915746722","heartbeat_front_addr":"172.21.15.45:6810/3915746722","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T00:19:57.580195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T00:19:57.630479+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T00:19:57.716834+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T00:19:57.665953+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T00:19:57.633161+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T00:19:57.663690+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T00:20:07.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.187+0000 7fe34ddc2640 1 -- 172.21.15.37:0/579997488 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe334035e80 msgr2=0x7fe334038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:07.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.187+0000 7fe34ddc2640 1 --2- 172.21.15.37:0/579997488 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe334035e80 0x7fe334038340 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe3481410e0 tx=0x7fe33c009040 comp rx=0 tx=0).stop 2024-04-26T00:20:07.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.187+0000 7fe34ddc2640 10 monclient: shutdown 2024-04-26T00:20:07.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.187+0000 7fe34ddc2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:07.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.187+0000 7fe34ddc2640 1 -- 172.21.15.37:0/579997488 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe34809de00 msgr2=0x7fe3481d4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:07.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.187+0000 7fe34ddc2640 1 --2- 172.21.15.37:0/579997488 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe34809de00 0x7fe3481d4190 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe3380af040 tx=0x7fe338019180 comp rx=0 tx=0).stop 2024-04-26T00:20:07.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.187+0000 7fe34ddc2640 1 -- 172.21.15.37:0/579997488 shutdown_connections 2024-04-26T00:20:07.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.187+0000 7fe34ddc2640 1 --2- 172.21.15.37:0/579997488 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe334035e80 0x7fe334038340 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:07.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.187+0000 7fe34ddc2640 1 --2- 172.21.15.37:0/579997488 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe348138330 0x7fe34813cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:07.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.187+0000 7fe34ddc2640 1 --2- 172.21.15.37:0/579997488 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3480a00e0 0x7fe348142400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:07.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.187+0000 7fe34ddc2640 1 --2- 172.21.15.37:0/579997488 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe34809de00 0x7fe3481d4190 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:07.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.187+0000 7fe34ddc2640 1 -- 172.21.15.37:0/579997488 >> 172.21.15.37:0/579997488 conn(0x7fe3480a1ad0 msgr2=0x7fe34806dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:07.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.187+0000 7fe34ddc2640 1 -- 172.21.15.37:0/579997488 shutdown_connections 2024-04-26T00:20:07.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.187+0000 7fe34ddc2640 1 -- 172.21.15.37:0/579997488 wait complete. 2024-04-26T00:20:07.184 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2024-04-26T00:20:00.477727+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2024-04-26T00:20:01.233750+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T00:20:07.184 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-04-26T00:20:07.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60cb644640 10 monclient: get_monmap_and_config 2024-04-26T00:20:07.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60cb644640 10 monclient: build_initial_monmap 2024-04-26T00:20:07.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60cb644640 10 monclient: monmap: 2024-04-26T00:20:07.312 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:07.313 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:07.313 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:07.336224+0000 2024-04-26T00:20:07.313 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:07.336224+0000 2024-04-26T00:20:07.313 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:07.313 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:07.313 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:07.313 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:07.313 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:07.313 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:07.313 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:07.313 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:07.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60cb644640 1 Processor -- start 2024-04-26T00:20:07.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60cb644640 1 -- start start 2024-04-26T00:20:07.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60cb644640 10 monclient: init 2024-04-26T00:20:07.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60cb644640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:07.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60cb644640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-04-26T00:20:07.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60cb644640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60c40a2c80 0x7f60c409ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:07.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60cb644640 10 monclient(hunting): picked mon.noname-b con 0x7f60c40a2c80 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:07.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60cb644640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f60c409f170 addr v1:172.21.15.37:6789/0 2024-04-26T00:20:07.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60cb644640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f60c4135430 0x7f60c413d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:07.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60cb644640 10 monclient(hunting): picked mon.noname-c con 0x7f60c4135430 addr v2:172.21.15.45:3300/0 2024-04-26T00:20:07.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60cb644640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60c409f910 con 0x7f60c409f170 2024-04-26T00:20:07.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60cb644640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:07.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60cb644640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f60c40a0f00 con 0x7f60c4135430 2024-04-26T00:20:07.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60cb644640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:07.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60cb644640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f60c40a3060 con 0x7f60c40a2c80 2024-04-26T00:20:07.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60cb644640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:07.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60cb644640 10 monclient(hunting): authenticate will time out at 988.642111s 2024-04-26T00:20:07.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60c93e0640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60c40a2c80 0x7f60c409ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:07.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60c8bdf640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f60c409f170 0x7f60c4134d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:55840/0 (socket says 172.21.15.37:55840) 2024-04-26T00:20:07.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60c8bdf640 1 -- 172.21.15.37:0/4180708413 learned_addr learned my addr 172.21.15.37:0/4180708413 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:20:07.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60c93e0640 10 monclient(hunting): get_auth_request con 0x7f60c40a2c80 auth_method 0 2024-04-26T00:20:07.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60c93e0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:07.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60c93e0640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:07.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60c93e0640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:07.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60c93e0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:07.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60c93e0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:07.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60c93e0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:07.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60c9be1640 1 --2- 172.21.15.37:0/4180708413 >> v2:172.21.15.45:3300/0 conn(0x7f60c4135430 0x7f60c413d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:07.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60c9be1640 10 monclient(hunting): get_auth_request con 0x7f60c4135430 auth_method 0 2024-04-26T00:20:07.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60c9be1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:07.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60c9be1640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:07.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60c9be1640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:07.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60c93e0640 10 monclient(hunting): handle_auth_done global_id 4304 payload 274 2024-04-26T00:20:07.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60c93e0640 1 -- 172.21.15.37:0/4180708413 >> v2:172.21.15.45:3300/0 conn(0x7f60c4135430 msgr2=0x7f60c413d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:07.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60c93e0640 1 --2- 172.21.15.37:0/4180708413 >> v2:172.21.15.45:3300/0 conn(0x7f60c4135430 0x7f60c413d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:07.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60c93e0640 1 -- 172.21.15.37:0/4180708413 >> v1:172.21.15.37:6789/0 conn(0x7f60c409f170 legacy=0x7f60c4134d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:07.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60c93e0640 10 monclient: _finish_hunting 0 2024-04-26T00:20:07.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60c93e0640 1 monclient: found mon.noname-b 2024-04-26T00:20:07.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60c93e0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:07.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60c93e0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:20:07.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.331+0000 7f60c93e0640 1 -- 172.21.15.37:0/4180708413 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60c40a0940 con 0x7f60c40a2c80 2024-04-26T00:20:07.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.335+0000 7f60c93e0640 1 -- 172.21.15.37:0/4180708413 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60c40a2c80 msgr2=0x7f60c409ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:07.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.335+0000 7f60c93e0640 1 --2- 172.21.15.37:0/4180708413 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60c40a2c80 0x7f60c409ec30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f60c00097c0 tx=0x7f60c001c320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd9e651f9d8c0f11 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:07.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.335+0000 7f60bbfff640 1 -- 172.21.15.37:0/4180708413 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c00b0070 con 0x7f60c40a2c80 2024-04-26T00:20:07.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.335+0000 7f60bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:07.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.335+0000 7f60bbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:20:07.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.335+0000 7f60bbfff640 10 monclient: dump: 2024-04-26T00:20:07.321 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:07.321 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:07.321 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:07.321 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:07.321 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:07.321 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:07.321 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:07.321 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:07.321 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:07.321 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:07.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.335+0000 7f60bbfff640 10 monclient: _finish_auth 0 2024-04-26T00:20:07.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.335+0000 7f60bbfff640 10 monclient: _check_auth_tickets 2024-04-26T00:20:07.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.335+0000 7f60bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:07.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.335+0000 7f60bbfff640 1 -- 172.21.15.37:0/4180708413 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60c00ab600 con 0x7f60c40a2c80 2024-04-26T00:20:07.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.335+0000 7f60bbfff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:07.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.335+0000 7f60bbfff640 1 -- 172.21.15.37:0/4180708413 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c00ab9d0 con 0x7f60c40a2c80 2024-04-26T00:20:07.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.335+0000 7f60cb644640 5 monclient: authenticate success, global_id 4304 2024-04-26T00:20:07.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.335+0000 7f60cb644640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:07.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.335+0000 7f60cb644640 4 set_mon_vals no callback set 2024-04-26T00:20:07.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.335+0000 7f60bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:07.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.335+0000 7f60bbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:07.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.335+0000 7f60bbfff640 10 monclient: dump: 2024-04-26T00:20:07.322 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:07.322 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:07.322 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:07.322 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:07.323 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:07.323 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:07.323 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:07.323 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:07.323 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:07.323 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:07.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.335+0000 7f60cb644640 10 monclient: shutdown 2024-04-26T00:20:07.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.335+0000 7f60cb644640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:07.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.335+0000 7f60cb644640 1 -- 172.21.15.37:0/4180708413 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60c40a2c80 msgr2=0x7f60c409ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:07.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.335+0000 7f60cb644640 1 --2- 172.21.15.37:0/4180708413 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60c40a2c80 0x7f60c409ec30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f60c00097c0 tx=0x7f60c001c320 comp rx=0 tx=0).stop 2024-04-26T00:20:07.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.335+0000 7f60cb644640 1 -- 172.21.15.37:0/4180708413 shutdown_connections 2024-04-26T00:20:07.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.335+0000 7f60cb644640 1 --2- 172.21.15.37:0/4180708413 >> v2:172.21.15.45:3300/0 conn(0x7f60c4135430 0x7f60c413d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:07.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.335+0000 7f60cb644640 1 --2- 172.21.15.37:0/4180708413 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60c40a2c80 0x7f60c409ec30 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:07.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.335+0000 7f60cb644640 1 -- 172.21.15.37:0/4180708413 >> 172.21.15.37:0/4180708413 conn(0x7f60c40a7de0 msgr2=0x7f60c40a3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:07.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.335+0000 7f60cb644640 1 -- 172.21.15.37:0/4180708413 shutdown_connections 2024-04-26T00:20:07.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.335+0000 7f60cb644640 1 -- 172.21.15.37:0/4180708413 wait complete. 2024-04-26T00:20:07.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60cb644640 10 monclient: build_initial_monmap 2024-04-26T00:20:07.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60cb644640 10 monclient: monmap: 2024-04-26T00:20:07.324 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:07.324 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:07.324 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:07.324 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:07.324 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:07.324 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:07.324 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:07.324 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:07.324 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:07.325 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:07.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60cb644640 1 Processor -- start 2024-04-26T00:20:07.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60cb644640 1 -- start start 2024-04-26T00:20:07.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60cb644640 10 monclient: init 2024-04-26T00:20:07.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60cb644640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:07.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60cb644640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:20:07.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60cb644640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60c409f170 0x7f60c41cf130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:07.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60cb644640 10 monclient(hunting): picked mon.noname-a con 0x7f60c409f170 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:07.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60cb644640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60c40a2c80 0x7f60c4133520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:07.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60cb644640 10 monclient(hunting): picked mon.noname-b con 0x7f60c40a2c80 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:07.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60cb644640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60c4135430 0x7f60c4133a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:07.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60cb644640 10 monclient(hunting): picked mon.noname-c con 0x7f60c4135430 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:07.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60cb644640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:07.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60cb644640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f60c409e050 con 0x7f60c409f170 2024-04-26T00:20:07.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60cb644640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:07.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60cb644640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f60c409ded0 con 0x7f60c40a2c80 2024-04-26T00:20:07.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60cb644640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:07.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60cb644640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f60c413fdc0 con 0x7f60c4135430 2024-04-26T00:20:07.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60cb644640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:07.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60cb644640 10 monclient(hunting): authenticate will time out at 988.649201s 2024-04-26T00:20:07.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60c9be1640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60c4135430 0x7f60c4133a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:07.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60c93e0640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60c409f170 0x7f60c41cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:07.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60c9be1640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60c4135430 0x7f60c4133a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:58070/0 (socket says 172.21.15.37:58070) 2024-04-26T00:20:07.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60c9be1640 1 -- 172.21.15.37:0/828777838 learned_addr learned my addr 172.21.15.37:0/828777838 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:07.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60c93e0640 10 monclient(hunting): get_auth_request con 0x7f60c409f170 auth_method 0 2024-04-26T00:20:07.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60c93e0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:07.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60c93e0640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:07.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60c93e0640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:07.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60c9be1640 10 monclient(hunting): get_auth_request con 0x7f60c4135430 auth_method 0 2024-04-26T00:20:07.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60c9be1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:07.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60c9be1640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:07.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60c9be1640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:07.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60c93e0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:07.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60c93e0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:07.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60c93e0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:07.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60c9be1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:07.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60c9be1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:07.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60c9be1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:07.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60c8bdf640 1 --2- 172.21.15.37:0/828777838 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60c40a2c80 0x7f60c4133520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:07.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60c93e0640 1 -- 172.21.15.37:0/828777838 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60c409f170 msgr2=0x7f60c41cf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:20:07.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60c93e0640 1 -- 172.21.15.37:0/828777838 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60c409f170 msgr2=0x7f60c41cf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:20:07.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60c93e0640 1 --2- 172.21.15.37:0/828777838 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60c409f170 0x7f60c41cf130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:20:07.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60c8bdf640 10 monclient(hunting): get_auth_request con 0x7f60c40a2c80 auth_method 0 2024-04-26T00:20:07.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60c8bdf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:07.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60c8bdf640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:07.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60c8bdf640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:07.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60c93e0640 1 --2- 172.21.15.37:0/828777838 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60c409f170 0x7f60c41cf130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:20:07.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60c9be1640 10 monclient(hunting): handle_auth_done global_id 4307 payload 824 2024-04-26T00:20:07.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60c9be1640 1 -- 172.21.15.37:0/828777838 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60c40a2c80 msgr2=0x7f60c4133520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:07.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60c9be1640 1 --2- 172.21.15.37:0/828777838 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60c40a2c80 0x7f60c4133520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:07.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60c9be1640 1 -- 172.21.15.37:0/828777838 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60c409f170 msgr2=0x7f60c41cf130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:20:07.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60c9be1640 1 --2- 172.21.15.37:0/828777838 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60c409f170 0x7f60c41cf130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:07.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60c9be1640 10 monclient: _finish_hunting 0 2024-04-26T00:20:07.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60c9be1640 1 monclient: found mon.noname-c 2024-04-26T00:20:07.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60c9be1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:07.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60c9be1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:20:07.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60c9be1640 1 -- 172.21.15.37:0/828777838 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60c411fef0 con 0x7f60c4135430 2024-04-26T00:20:07.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60c9be1640 1 -- 172.21.15.37:0/828777838 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60c4135430 msgr2=0x7f60c4133a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:07.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.339+0000 7f60c9be1640 1 --2- 172.21.15.37:0/828777838 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60c4135430 0x7f60c4133a60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f60bc007e40 tx=0x7f60bc00e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:07.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.343+0000 7f60b9ffb640 1 -- 172.21.15.37:0/828777838 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60bc013790 con 0x7f60c4135430 2024-04-26T00:20:07.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.343+0000 7f60b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:07.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.343+0000 7f60b9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:20:07.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.343+0000 7f60b9ffb640 10 monclient: dump: 2024-04-26T00:20:07.332 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:07.332 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:07.332 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:07.332 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:07.332 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:07.332 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:07.333 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:07.333 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:07.333 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:07.333 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:07.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.343+0000 7f60b9ffb640 10 monclient: _finish_auth 0 2024-04-26T00:20:07.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.343+0000 7f60b9ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:20:07.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.343+0000 7f60b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:07.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.343+0000 7f60cb644640 5 monclient: authenticate success, global_id 4307 2024-04-26T00:20:07.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.343+0000 7f60b9ffb640 1 -- 172.21.15.37:0/828777838 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60bc019070 con 0x7f60c4135430 2024-04-26T00:20:07.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.343+0000 7f60cb644640 10 monclient: _renew_subs 2024-04-26T00:20:07.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.343+0000 7f60cb644640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:07.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.343+0000 7f60cb644640 1 -- 172.21.15.37:0/828777838 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60c406ca50 con 0x7f60c4135430 2024-04-26T00:20:07.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.343+0000 7f60b9ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:07.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.343+0000 7f60b9ffb640 1 -- 172.21.15.37:0/828777838 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60bc013b90 con 0x7f60c4135430 2024-04-26T00:20:07.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.343+0000 7f60b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:07.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.343+0000 7f60b9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:07.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.343+0000 7f60b9ffb640 10 monclient: dump: 2024-04-26T00:20:07.334 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:07.334 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:07.334 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:07.334 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:07.334 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:07.334 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:07.334 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:07.334 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:07.334 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:07.334 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:07.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.343+0000 7f60cb644640 10 monclient: _renew_subs 2024-04-26T00:20:07.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.343+0000 7f60bb7fe640 4 set_mon_vals no callback set 2024-04-26T00:20:07.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.343+0000 7f60cb644640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:07.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.343+0000 7f60cb644640 1 -- 172.21.15.37:0/828777838 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f60c411e470 con 0x7f60c4135430 2024-04-26T00:20:07.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.343+0000 7f60b9ffb640 1 -- 172.21.15.37:0/828777838 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f60bc01e460 con 0x7f60c4135430 2024-04-26T00:20:07.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.343+0000 7f60cb644640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:07.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.343+0000 7f60cb644640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:07.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.343+0000 7f60cb644640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:07.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.343+0000 7f60cb644640 1 -- 172.21.15.37:0/828777838 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60c404f510 con 0x7f60c4135430 2024-04-26T00:20:07.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.343+0000 7f60b9ffb640 1 --2- 172.21.15.37:0/828777838 >> v2:172.21.15.45:6812/2379918623 conn(0x7f60a8035ed0 0x7f60a8038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:07.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.343+0000 7f60b9ffb640 1 -- 172.21.15.37:0/828777838 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7f60bc049290 con 0x7f60c4135430 2024-04-26T00:20:07.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.343+0000 7f60c93e0640 1 --2- 172.21.15.37:0/828777838 >> v2:172.21.15.45:6812/2379918623 conn(0x7f60a8035ed0 0x7f60a8038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:07.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.343+0000 7f60c93e0640 10 monclient: get_auth_request con 0x7f60a8035ed0 auth_method 0 2024-04-26T00:20:07.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.347+0000 7f60c93e0640 1 --2- 172.21.15.37:0/828777838 >> v2:172.21.15.45:6812/2379918623 conn(0x7f60a8035ed0 0x7f60a8038390 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f60c40a4b10 tx=0x7f60c00a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:07.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.347+0000 7f60b9ffb640 1 -- 172.21.15.37:0/828777838 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f60bc01e800 con 0x7f60c4135430 2024-04-26T00:20:07.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.347+0000 7f60b9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:07.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.347+0000 7f60b9ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:07.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.499+0000 7f60cb644640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-04-26T00:20:07.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.499+0000 7f60cb644640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-04-26T00:20:07.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.499+0000 7f60cb644640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:07.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.499+0000 7f60cb644640 1 -- 172.21.15.37:0/828777838 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) -- 0x7f60c4075ea0 con 0x7f60c4135430 2024-04-26T00:20:07.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.503+0000 7f60b9ffb640 1 -- 172.21.15.37:0/828777838 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v15) ==== 92+0+10 (secure 0 0 0) 0x7f60bc048180 con 0x7f60c4135430 2024-04-26T00:20:07.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.503+0000 7f60b9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-04-26T00:20:07.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.503+0000 7f60b9ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:07.484 INFO:teuthology.orchestra.run.smithi037.stdout:pg_num: 8 2024-04-26T00:20:07.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.503+0000 7f60cb644640 1 -- 172.21.15.37:0/828777838 >> v2:172.21.15.45:6812/2379918623 conn(0x7f60a8035ed0 msgr2=0x7f60a8038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:07.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.503+0000 7f60cb644640 1 --2- 172.21.15.37:0/828777838 >> v2:172.21.15.45:6812/2379918623 conn(0x7f60a8035ed0 0x7f60a8038390 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f60c40a4b10 tx=0x7f60c00a6000 comp rx=0 tx=0).stop 2024-04-26T00:20:07.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.503+0000 7f60cb644640 10 monclient: shutdown 2024-04-26T00:20:07.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.503+0000 7f60cb644640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:07.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.503+0000 7f60cb644640 1 -- 172.21.15.37:0/828777838 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60c4135430 msgr2=0x7f60c4133a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:07.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.503+0000 7f60cb644640 1 --2- 172.21.15.37:0/828777838 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60c4135430 0x7f60c4133a60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f60bc007e40 tx=0x7f60bc00e4b0 comp rx=0 tx=0).stop 2024-04-26T00:20:07.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.503+0000 7f60cb644640 1 -- 172.21.15.37:0/828777838 shutdown_connections 2024-04-26T00:20:07.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.503+0000 7f60cb644640 1 --2- 172.21.15.37:0/828777838 >> v2:172.21.15.45:6812/2379918623 conn(0x7f60a8035ed0 0x7f60a8038390 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:07.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.503+0000 7f60cb644640 1 --2- 172.21.15.37:0/828777838 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60c4135430 0x7f60c4133a60 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:07.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.503+0000 7f60cb644640 1 --2- 172.21.15.37:0/828777838 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60c40a2c80 0x7f60c4133520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:07.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.503+0000 7f60cb644640 1 --2- 172.21.15.37:0/828777838 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60c409f170 0x7f60c41cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:07.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.503+0000 7f60cb644640 1 -- 172.21.15.37:0/828777838 >> 172.21.15.37:0/828777838 conn(0x7f60c40a7de0 msgr2=0x7f60c413c150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:07.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.503+0000 7f60cb644640 1 -- 172.21.15.37:0/828777838 shutdown_connections 2024-04-26T00:20:07.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.503+0000 7f60cb644640 1 -- 172.21.15.37:0/828777838 wait complete. 2024-04-26T00:20:07.501 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T00:20:07.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.639+0000 7fdbb2e2c640 10 monclient: get_monmap_and_config 2024-04-26T00:20:07.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.639+0000 7fdbb2e2c640 10 monclient: build_initial_monmap 2024-04-26T00:20:07.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.639+0000 7fdbb2e2c640 10 monclient: monmap: 2024-04-26T00:20:07.623 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:07.623 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:07.624 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:07.646779+0000 2024-04-26T00:20:07.624 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:07.646779+0000 2024-04-26T00:20:07.624 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:07.624 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:07.624 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:07.624 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:07.624 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:07.625 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:07.625 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:07.625 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:07.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbb2e2c640 1 Processor -- start 2024-04-26T00:20:07.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbb2e2c640 1 -- start start 2024-04-26T00:20:07.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbb2e2c640 10 monclient: init 2024-04-26T00:20:07.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbb2e2c640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:07.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbb2e2c640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-04-26T00:20:07.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbb2e2c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdbac0a1c00 addr v1:172.21.15.37:6789/0 2024-04-26T00:20:07.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbb2e2c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fdbac073000 0x7fdbac138f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:07.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbb2e2c640 10 monclient(hunting): picked mon.noname-a con 0x7fdbac073000 addr v2:172.21.15.37:3300/0 2024-04-26T00:20:07.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbb2e2c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdbac09aec0 0x7fdbac13b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:07.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbb2e2c640 10 monclient(hunting): picked mon.noname-b con 0x7fdbac09aec0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:07.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbb2e2c640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbac0a02b0 con 0x7fdbac0a1c00 2024-04-26T00:20:07.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbb2e2c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:07.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbb2e2c640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fdbac0a25f0 con 0x7fdbac073000 2024-04-26T00:20:07.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbb2e2c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:07.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbb2e2c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fdbac09d0a0 con 0x7fdbac09aec0 2024-04-26T00:20:07.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbb2e2c640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:07.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbb2e2c640 10 monclient(hunting): authenticate will time out at 988.953284s 2024-04-26T00:20:07.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbb13c9640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdbac09aec0 0x7fdbac13b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:07.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbb0bc8640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fdbac0a1c00 0x7fdbac09fba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:55846/0 (socket says 172.21.15.37:55846) 2024-04-26T00:20:07.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbb0bc8640 1 -- 172.21.15.37:0/2695417921 learned_addr learned my addr 172.21.15.37:0/2695417921 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:20:07.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbabfff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fdbac073000 0x7fdbac138f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:07.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbb13c9640 10 monclient(hunting): get_auth_request con 0x7fdbac09aec0 auth_method 0 2024-04-26T00:20:07.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbb13c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:07.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbb13c9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:07.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbb13c9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:07.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbabfff640 10 monclient(hunting): get_auth_request con 0x7fdbac073000 auth_method 0 2024-04-26T00:20:07.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:07.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbabfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:07.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbabfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:07.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbb13c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:07.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbb13c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:07.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbb13c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:07.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbabfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:07.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbabfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:07.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbabfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:07.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbb13c9640 10 monclient(hunting): handle_auth_done global_id 4310 payload 274 2024-04-26T00:20:07.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbb13c9640 1 -- 172.21.15.37:0/2695417921 >> v2:172.21.15.37:3300/0 conn(0x7fdbac073000 msgr2=0x7fdbac138f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:07.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbb13c9640 1 --2- 172.21.15.37:0/2695417921 >> v2:172.21.15.37:3300/0 conn(0x7fdbac073000 0x7fdbac138f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:07.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbb13c9640 1 -- 172.21.15.37:0/2695417921 >> v1:172.21.15.37:6789/0 conn(0x7fdbac0a1c00 legacy=0x7fdbac09fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:07.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbb13c9640 10 monclient: _finish_hunting 0 2024-04-26T00:20:07.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbb13c9640 1 monclient: found mon.noname-b 2024-04-26T00:20:07.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbb13c9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:07.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbb13c9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:20:07.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.643+0000 7fdbb13c9640 1 -- 172.21.15.37:0/2695417921 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbac0a28a0 con 0x7fdbac09aec0 2024-04-26T00:20:07.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.647+0000 7fdbb13c9640 1 -- 172.21.15.37:0/2695417921 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdbac09aec0 msgr2=0x7fdbac13b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:07.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.647+0000 7fdbb13c9640 1 --2- 172.21.15.37:0/2695417921 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdbac09aec0 0x7fdbac13b4a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdba4009d90 tx=0x7fdba401c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f044cc7ea894dc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:07.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.647+0000 7fdbab7fe640 1 -- 172.21.15.37:0/2695417921 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba40b2070 con 0x7fdbac09aec0 2024-04-26T00:20:07.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.647+0000 7fdbab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:07.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.647+0000 7fdbab7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:20:07.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.647+0000 7fdbab7fe640 10 monclient: dump: 2024-04-26T00:20:07.632 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:07.632 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:07.632 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:07.632 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:07.632 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:07.632 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:07.633 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:07.633 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:07.633 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:07.633 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:07.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.647+0000 7fdbab7fe640 10 monclient: _finish_auth 0 2024-04-26T00:20:07.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.647+0000 7fdbab7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:20:07.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.647+0000 7fdbab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:07.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.647+0000 7fdbab7fe640 1 -- 172.21.15.37:0/2695417921 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdba40a5760 con 0x7fdbac09aec0 2024-04-26T00:20:07.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.647+0000 7fdbb2e2c640 5 monclient: authenticate success, global_id 4310 2024-04-26T00:20:07.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.647+0000 7fdbb2e2c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:20:07.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.647+0000 7fdbab7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:07.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.647+0000 7fdbb2e2c640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:07.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.647+0000 7fdbb2e2c640 4 set_mon_vals no callback set 2024-04-26T00:20:07.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.647+0000 7fdbab7fe640 1 -- 172.21.15.37:0/2695417921 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba40a5b30 con 0x7fdbac09aec0 2024-04-26T00:20:07.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.647+0000 7fdbab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:07.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.647+0000 7fdbab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:07.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.647+0000 7fdbab7fe640 10 monclient: dump: 2024-04-26T00:20:07.634 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:07.634 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:07.634 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:07.634 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:07.634 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:07.634 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:07.634 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:07.634 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:07.634 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:07.634 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:07.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.647+0000 7fdbb2e2c640 10 monclient: shutdown 2024-04-26T00:20:07.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.647+0000 7fdbb2e2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:07.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.647+0000 7fdbb2e2c640 1 -- 172.21.15.37:0/2695417921 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdbac09aec0 msgr2=0x7fdbac13b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:07.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.647+0000 7fdbb2e2c640 1 --2- 172.21.15.37:0/2695417921 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdbac09aec0 0x7fdbac13b4a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdba4009d90 tx=0x7fdba401c7f0 comp rx=0 tx=0).stop 2024-04-26T00:20:07.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.647+0000 7fdbb2e2c640 1 -- 172.21.15.37:0/2695417921 shutdown_connections 2024-04-26T00:20:07.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.647+0000 7fdbb2e2c640 1 --2- 172.21.15.37:0/2695417921 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdbac09aec0 0x7fdbac13b4a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:07.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.647+0000 7fdbb2e2c640 1 --2- 172.21.15.37:0/2695417921 >> v2:172.21.15.37:3300/0 conn(0x7fdbac073000 0x7fdbac138f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:07.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.647+0000 7fdbb2e2c640 1 -- 172.21.15.37:0/2695417921 >> 172.21.15.37:0/2695417921 conn(0x7fdbac0a6c90 msgr2=0x7fdbac0a2cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:07.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.647+0000 7fdbb2e2c640 1 -- 172.21.15.37:0/2695417921 shutdown_connections 2024-04-26T00:20:07.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.647+0000 7fdbb2e2c640 1 -- 172.21.15.37:0/2695417921 wait complete. 2024-04-26T00:20:07.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb2e2c640 10 monclient: build_initial_monmap 2024-04-26T00:20:07.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb2e2c640 10 monclient: monmap: 2024-04-26T00:20:07.635 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:07.635 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:07.635 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:07.635 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:07.635 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:07.636 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:07.636 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:07.636 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:07.636 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:07.636 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:07.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb2e2c640 1 Processor -- start 2024-04-26T00:20:07.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb2e2c640 1 -- start start 2024-04-26T00:20:07.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb2e2c640 10 monclient: init 2024-04-26T00:20:07.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb2e2c640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:07.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb2e2c640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:20:07.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb2e2c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdbac073000 0x7fdbac133200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:07.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb2e2c640 10 monclient(hunting): picked mon.noname-a con 0x7fdbac073000 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:07.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb2e2c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdbac09aec0 0x7fdbac133740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:07.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb2e2c640 10 monclient(hunting): picked mon.noname-b con 0x7fdbac09aec0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:07.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb2e2c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdbac0a1c00 0x7fdbac1cb930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:07.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb2e2c640 10 monclient(hunting): picked mon.noname-c con 0x7fdbac0a1c00 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:07.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb2e2c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:07.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb2e2c640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fdbac13e5d0 con 0x7fdbac073000 2024-04-26T00:20:07.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb2e2c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:07.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb2e2c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdbac13e450 con 0x7fdbac09aec0 2024-04-26T00:20:07.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb2e2c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:07.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb2e2c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fdbac13e750 con 0x7fdbac0a1c00 2024-04-26T00:20:07.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb2e2c640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:07.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb2e2c640 10 monclient(hunting): authenticate will time out at 988.961267s 2024-04-26T00:20:07.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb0bc8640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdbac073000 0x7fdbac133200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:07.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb0bc8640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdbac073000 0x7fdbac133200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44894/0 (socket says 172.21.15.37:44894) 2024-04-26T00:20:07.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb13c9640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdbac0a1c00 0x7fdbac1cb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:07.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb0bc8640 1 -- 172.21.15.37:0/2391952511 learned_addr learned my addr 172.21.15.37:0/2391952511 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:07.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb0bc8640 10 monclient(hunting): get_auth_request con 0x7fdbac073000 auth_method 0 2024-04-26T00:20:07.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb0bc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:07.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb0bc8640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:07.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb0bc8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:07.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb13c9640 10 monclient(hunting): get_auth_request con 0x7fdbac0a1c00 auth_method 0 2024-04-26T00:20:07.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb13c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:07.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb13c9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:07.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb13c9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:07.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb0bc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:07.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb0bc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:07.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb0bc8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:07.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb13c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:07.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb13c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:07.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb13c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:07.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb0bc8640 10 monclient(hunting): handle_auth_done global_id 4326 payload 824 2024-04-26T00:20:07.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbabfff640 1 --2- 172.21.15.37:0/2391952511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdbac09aec0 0x7fdbac133740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:07.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb0bc8640 1 -- 172.21.15.37:0/2391952511 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdbac0a1c00 msgr2=0x7fdbac1cb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:07.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb0bc8640 1 --2- 172.21.15.37:0/2391952511 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdbac0a1c00 0x7fdbac1cb930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:07.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb0bc8640 1 -- 172.21.15.37:0/2391952511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdbac09aec0 msgr2=0x7fdbac133740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:07.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb0bc8640 1 --2- 172.21.15.37:0/2391952511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdbac09aec0 0x7fdbac133740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:07.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb0bc8640 10 monclient: _finish_hunting 0 2024-04-26T00:20:07.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb0bc8640 1 monclient: found mon.noname-a 2024-04-26T00:20:07.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb0bc8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:07.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb0bc8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:07.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb0bc8640 1 -- 172.21.15.37:0/2391952511 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbac138e90 con 0x7fdbac073000 2024-04-26T00:20:07.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.651+0000 7fdbb13c9640 1 --2- 172.21.15.37:0/2391952511 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdbac0a1c00 0x7fdbac1cb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:07.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.655+0000 7fdbb0bc8640 1 -- 172.21.15.37:0/2391952511 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdbac073000 msgr2=0x7fdbac133200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:07.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.655+0000 7fdbb0bc8640 1 --2- 172.21.15.37:0/2391952511 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdbac073000 0x7fdbac133200 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdb9800c470 tx=0x7fdb9800c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:07.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.655+0000 7fdba97fa640 1 -- 172.21.15.37:0/2391952511 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb98007070 con 0x7fdbac073000 2024-04-26T00:20:07.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.655+0000 7fdba97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:07.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.655+0000 7fdba97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:07.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.655+0000 7fdba97fa640 10 monclient: dump: 2024-04-26T00:20:07.643 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:07.643 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:07.643 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:07.643 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:07.643 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:07.643 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:07.644 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:07.644 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:07.644 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:07.644 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:07.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.655+0000 7fdba97fa640 10 monclient: _finish_auth 0 2024-04-26T00:20:07.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.655+0000 7fdba97fa640 10 monclient: _check_auth_tickets 2024-04-26T00:20:07.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.655+0000 7fdba97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:07.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.655+0000 7fdba97fa640 1 -- 172.21.15.37:0/2391952511 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb98007210 con 0x7fdbac073000 2024-04-26T00:20:07.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.655+0000 7fdba97fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:07.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.655+0000 7fdba97fa640 1 -- 172.21.15.37:0/2391952511 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb98007600 con 0x7fdbac073000 2024-04-26T00:20:07.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.655+0000 7fdba97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:07.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.655+0000 7fdba97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:07.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.655+0000 7fdba97fa640 10 monclient: dump: 2024-04-26T00:20:07.644 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:07.644 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:07.645 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:07.645 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:07.645 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:07.645 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:07.645 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:07.645 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:07.645 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:07.645 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:07.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.655+0000 7fdbb2e2c640 5 monclient: authenticate success, global_id 4326 2024-04-26T00:20:07.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.655+0000 7fdbb2e2c640 10 monclient: _renew_subs 2024-04-26T00:20:07.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.655+0000 7fdbb2e2c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:07.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.655+0000 7fdbb2e2c640 1 -- 172.21.15.37:0/2391952511 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbac13afc0 con 0x7fdbac073000 2024-04-26T00:20:07.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.655+0000 7fdbaaffd640 4 set_mon_vals no callback set 2024-04-26T00:20:07.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.655+0000 7fdbb2e2c640 10 monclient: _renew_subs 2024-04-26T00:20:07.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.655+0000 7fdbb2e2c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:07.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.655+0000 7fdbb2e2c640 1 -- 172.21.15.37:0/2391952511 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbac13a340 con 0x7fdbac073000 2024-04-26T00:20:07.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.655+0000 7fdbb2e2c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:07.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.655+0000 7fdba97fa640 1 -- 172.21.15.37:0/2391952511 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fdb9801b400 con 0x7fdbac073000 2024-04-26T00:20:07.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.655+0000 7fdbb2e2c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:07.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.655+0000 7fdbb2e2c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:07.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.655+0000 7fdbb2e2c640 1 -- 172.21.15.37:0/2391952511 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbac066ed0 con 0x7fdbac073000 2024-04-26T00:20:07.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.655+0000 7fdba97fa640 1 --2- 172.21.15.37:0/2391952511 >> v2:172.21.15.45:6812/2379918623 conn(0x7fdb90035ed0 0x7fdb90038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:07.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.655+0000 7fdba97fa640 1 -- 172.21.15.37:0/2391952511 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7fdb98048680 con 0x7fdbac073000 2024-04-26T00:20:07.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.655+0000 7fdbabfff640 1 --2- 172.21.15.37:0/2391952511 >> v2:172.21.15.45:6812/2379918623 conn(0x7fdb90035ed0 0x7fdb90038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:07.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.655+0000 7fdbabfff640 10 monclient: get_auth_request con 0x7fdb90035ed0 auth_method 0 2024-04-26T00:20:07.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.659+0000 7fdbabfff640 1 --2- 172.21.15.37:0/2391952511 >> v2:172.21.15.45:6812/2379918623 conn(0x7fdb90035ed0 0x7fdb90038390 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdbac1386b0 tx=0x7fdba0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:07.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.659+0000 7fdba97fa640 1 -- 172.21.15.37:0/2391952511 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fdb98047570 con 0x7fdbac073000 2024-04-26T00:20:07.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.659+0000 7fdba97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:07.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.659+0000 7fdba97fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:07.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.811+0000 7fdbb2e2c640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-04-26T00:20:07.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.811+0000 7fdbb2e2c640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-04-26T00:20:07.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.811+0000 7fdbb2e2c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:07.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.811+0000 7fdbb2e2c640 1 -- 172.21.15.37:0/2391952511 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdbac09ea10 con 0x7fdbac073000 2024-04-26T00:20:07.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.811+0000 7fdba97fa640 1 -- 172.21.15.37:0/2391952511 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v15) ==== 93+0+10 (secure 0 0 0) 0x7fdb98012400 con 0x7fdbac073000 2024-04-26T00:20:07.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.811+0000 7fdba97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-04-26T00:20:07.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.811+0000 7fdba97fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:07.796 INFO:teuthology.orchestra.run.smithi037.stdout:pg_num: 1 2024-04-26T00:20:07.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.815+0000 7fdbb2e2c640 1 -- 172.21.15.37:0/2391952511 >> v2:172.21.15.45:6812/2379918623 conn(0x7fdb90035ed0 msgr2=0x7fdb90038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:07.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.815+0000 7fdbb2e2c640 1 --2- 172.21.15.37:0/2391952511 >> v2:172.21.15.45:6812/2379918623 conn(0x7fdb90035ed0 0x7fdb90038390 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdbac1386b0 tx=0x7fdba0008040 comp rx=0 tx=0).stop 2024-04-26T00:20:07.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.815+0000 7fdbb2e2c640 10 monclient: shutdown 2024-04-26T00:20:07.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.815+0000 7fdbb2e2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:07.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.815+0000 7fdbb2e2c640 1 -- 172.21.15.37:0/2391952511 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdbac073000 msgr2=0x7fdbac133200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:07.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.815+0000 7fdbb2e2c640 1 --2- 172.21.15.37:0/2391952511 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdbac073000 0x7fdbac133200 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdb9800c470 tx=0x7fdb9800c930 comp rx=0 tx=0).stop 2024-04-26T00:20:07.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.815+0000 7fdbb2e2c640 1 -- 172.21.15.37:0/2391952511 shutdown_connections 2024-04-26T00:20:07.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.815+0000 7fdbb2e2c640 1 --2- 172.21.15.37:0/2391952511 >> v2:172.21.15.45:6812/2379918623 conn(0x7fdb90035ed0 0x7fdb90038390 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:07.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.815+0000 7fdbb2e2c640 1 --2- 172.21.15.37:0/2391952511 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdbac0a1c00 0x7fdbac1cb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:07.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.815+0000 7fdbb2e2c640 1 --2- 172.21.15.37:0/2391952511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdbac09aec0 0x7fdbac133740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:07.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.815+0000 7fdbb2e2c640 1 --2- 172.21.15.37:0/2391952511 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdbac073000 0x7fdbac133200 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:07.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.815+0000 7fdbb2e2c640 1 -- 172.21.15.37:0/2391952511 >> 172.21.15.37:0/2391952511 conn(0x7fdbac0a6c90 msgr2=0x7fdbac06b1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:07.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.815+0000 7fdbb2e2c640 1 -- 172.21.15.37:0/2391952511 shutdown_connections 2024-04-26T00:20:07.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.815+0000 7fdbb2e2c640 1 -- 172.21.15.37:0/2391952511 wait complete. 2024-04-26T00:20:07.814 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-04-26T00:20:07.814 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T00:20:07.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f94e5f640 10 monclient: get_monmap_and_config 2024-04-26T00:20:07.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f94e5f640 10 monclient: build_initial_monmap 2024-04-26T00:20:07.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f94e5f640 10 monclient: monmap: 2024-04-26T00:20:07.929 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:07.929 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:07.929 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:07.952704+0000 2024-04-26T00:20:07.929 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:07.952704+0000 2024-04-26T00:20:07.929 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:07.929 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:07.929 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:07.929 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:07.929 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:07.929 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:07.929 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:07.930 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:07.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f94e5f640 1 Processor -- start 2024-04-26T00:20:07.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f94e5f640 1 -- start start 2024-04-26T00:20:07.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f94e5f640 10 monclient: init 2024-04-26T00:20:07.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f94e5f640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:07.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f94e5f640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-04-26T00:20:07.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f94e5f640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8f900a0260 0x7f8f900a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:07.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f94e5f640 10 monclient(hunting): picked mon.noname-c con 0x7f8f900a0260 addr v2:172.21.15.45:3300/0 2024-04-26T00:20:07.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f94e5f640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f8f90134b30 0x7f8f90134f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:07.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f94e5f640 10 monclient(hunting): picked mon.noname-a con 0x7f8f90134b30 addr v2:172.21.15.37:3300/0 2024-04-26T00:20:07.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f94e5f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8f90135490 addr v1:172.21.15.37:6789/0 2024-04-26T00:20:07.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f94e5f640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f900a1130 con 0x7f8f90135490 2024-04-26T00:20:07.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f94e5f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:07.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f94e5f640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f8f9009ec10 con 0x7f8f90134b30 2024-04-26T00:20:07.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f94e5f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:07.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f94e5f640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f8f9009f0f0 con 0x7f8f900a0260 2024-04-26T00:20:07.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f94e5f640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:07.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f94e5f640 10 monclient(hunting): authenticate will time out at 989.258333s 2024-04-26T00:20:07.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f8dd9b640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f8f90134b30 0x7f8f90134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:07.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f8ed9d640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f8f90135490 0x7f8f9013be60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:55852/0 (socket says 172.21.15.37:55852) 2024-04-26T00:20:07.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f8ed9d640 1 -- 172.21.15.37:0/1067547706 learned_addr learned my addr 172.21.15.37:0/1067547706 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:20:07.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f8dd9b640 10 monclient(hunting): get_auth_request con 0x7f8f90134b30 auth_method 0 2024-04-26T00:20:07.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f8dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:07.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f8dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:07.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f8dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:07.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f8dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:07.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f8dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:07.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f8e59c640 1 --2- 172.21.15.37:0/1067547706 >> v2:172.21.15.45:3300/0 conn(0x7f8f900a0260 0x7f8f900a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:07.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f8dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:07.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f8e59c640 10 monclient(hunting): get_auth_request con 0x7f8f900a0260 auth_method 0 2024-04-26T00:20:07.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f8e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:07.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f8e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:07.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f8e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:07.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f8dd9b640 10 monclient(hunting): handle_auth_done global_id 4329 payload 274 2024-04-26T00:20:07.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f8dd9b640 1 -- 172.21.15.37:0/1067547706 >> v2:172.21.15.45:3300/0 conn(0x7f8f900a0260 msgr2=0x7f8f900a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:07.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f8dd9b640 1 --2- 172.21.15.37:0/1067547706 >> v2:172.21.15.45:3300/0 conn(0x7f8f900a0260 0x7f8f900a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:07.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f8dd9b640 1 -- 172.21.15.37:0/1067547706 >> v1:172.21.15.37:6789/0 conn(0x7f8f90135490 legacy=0x7f8f9013be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:07.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f8dd9b640 10 monclient: _finish_hunting 0 2024-04-26T00:20:07.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f8dd9b640 1 monclient: found mon.noname-a 2024-04-26T00:20:07.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f8dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:07.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f8dd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:07.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f8dd9b640 1 -- 172.21.15.37:0/1067547706 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f9006cff0 con 0x7f8f90134b30 2024-04-26T00:20:07.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.947+0000 7f8f8e59c640 1 --2- 172.21.15.37:0/1067547706 >> v2:172.21.15.45:3300/0 conn(0x7f8f900a0260 0x7f8f900a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:20:07.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.951+0000 7f8f8dd9b640 1 -- 172.21.15.37:0/1067547706 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f90134b30 msgr2=0x7f8f90134f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:07.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.951+0000 7f8f8dd9b640 1 --2- 172.21.15.37:0/1067547706 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f90134b30 0x7f8f90134f50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8f900a2ad0 tx=0x7f8f8001c120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ca2344867a5ee22 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:07.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.951+0000 7f8f8d59a640 1 -- 172.21.15.37:0/1067547706 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f800b1070 con 0x7f8f90134b30 2024-04-26T00:20:07.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.951+0000 7f8f8d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:07.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.951+0000 7f8f8d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:07.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.951+0000 7f8f8d59a640 10 monclient: dump: 2024-04-26T00:20:07.937 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:07.937 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:07.937 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:07.937 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:07.937 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:07.937 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:07.937 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:07.937 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:07.937 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:07.937 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:07.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.951+0000 7f8f8d59a640 10 monclient: _finish_auth 0 2024-04-26T00:20:07.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.951+0000 7f8f8d59a640 10 monclient: _check_auth_tickets 2024-04-26T00:20:07.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.951+0000 7f8f8d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:07.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.951+0000 7f8f8d59a640 1 -- 172.21.15.37:0/1067547706 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f800ac440 con 0x7f8f90134b30 2024-04-26T00:20:07.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.951+0000 7f8f8d59a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:07.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.951+0000 7f8f8d59a640 1 -- 172.21.15.37:0/1067547706 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f800ac810 con 0x7f8f90134b30 2024-04-26T00:20:07.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.951+0000 7f8f94e5f640 5 monclient: authenticate success, global_id 4329 2024-04-26T00:20:07.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.951+0000 7f8f94e5f640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:07.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.951+0000 7f8f94e5f640 4 set_mon_vals no callback set 2024-04-26T00:20:07.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.951+0000 7f8f8d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:07.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.951+0000 7f8f8d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:07.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.951+0000 7f8f8d59a640 10 monclient: dump: 2024-04-26T00:20:07.938 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:07.938 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:07.938 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:07.938 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:07.939 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:07.939 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:07.939 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:07.939 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:07.939 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:07.939 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:07.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.951+0000 7f8f94e5f640 10 monclient: shutdown 2024-04-26T00:20:07.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.951+0000 7f8f94e5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:07.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.951+0000 7f8f94e5f640 1 -- 172.21.15.37:0/1067547706 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f90134b30 msgr2=0x7f8f90134f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:07.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.951+0000 7f8f94e5f640 1 --2- 172.21.15.37:0/1067547706 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f90134b30 0x7f8f90134f50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8f900a2ad0 tx=0x7f8f8001c120 comp rx=0 tx=0).stop 2024-04-26T00:20:07.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f94e5f640 1 -- 172.21.15.37:0/1067547706 shutdown_connections 2024-04-26T00:20:07.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f94e5f640 1 --2- 172.21.15.37:0/1067547706 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f90134b30 0x7f8f90134f50 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:07.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f94e5f640 1 --2- 172.21.15.37:0/1067547706 >> v2:172.21.15.45:3300/0 conn(0x7f8f900a0260 0x7f8f900a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:07.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f94e5f640 1 -- 172.21.15.37:0/1067547706 >> 172.21.15.37:0/1067547706 conn(0x7f8f900a1ad0 msgr2=0x7f8f900a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:07.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f94e5f640 1 -- 172.21.15.37:0/1067547706 shutdown_connections 2024-04-26T00:20:07.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f94e5f640 1 -- 172.21.15.37:0/1067547706 wait complete. 2024-04-26T00:20:07.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f94e5f640 10 monclient: build_initial_monmap 2024-04-26T00:20:07.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f94e5f640 10 monclient: monmap: 2024-04-26T00:20:07.940 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:07.940 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:07.940 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:07.940 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:07.940 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:07.940 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:07.940 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:07.940 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:07.940 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:07.940 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:07.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f94e5f640 1 Processor -- start 2024-04-26T00:20:07.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f94e5f640 1 -- start start 2024-04-26T00:20:07.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f94e5f640 10 monclient: init 2024-04-26T00:20:07.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f94e5f640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:07.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f94e5f640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:20:07.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f94e5f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8f900a0260 0x7f8f9013c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:07.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f94e5f640 10 monclient(hunting): picked mon.noname-c con 0x7f8f900a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:07.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f94e5f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f90134b30 0x7f8f9009c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:07.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f94e5f640 10 monclient(hunting): picked mon.noname-b con 0x7f8f90134b30 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:07.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f94e5f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f90135490 0x7f8f9009ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:07.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f94e5f640 10 monclient(hunting): picked mon.noname-a con 0x7f8f90135490 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:07.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f94e5f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:07.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f94e5f640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f8f9009f430 con 0x7f8f90135490 2024-04-26T00:20:07.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f94e5f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:07.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f94e5f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8f9009f2b0 con 0x7f8f90134b30 2024-04-26T00:20:07.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f94e5f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:07.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f94e5f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8f9009fe90 con 0x7f8f900a0260 2024-04-26T00:20:07.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f94e5f640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:07.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f94e5f640 10 monclient(hunting): authenticate will time out at 989.265504s 2024-04-26T00:20:07.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f8e59c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8f900a0260 0x7f8f9013c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:07.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f8ed9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f90135490 0x7f8f9009ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:07.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f8e59c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8f900a0260 0x7f8f9013c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:58094/0 (socket says 172.21.15.37:58094) 2024-04-26T00:20:07.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f8e59c640 1 -- 172.21.15.37:0/3329138043 learned_addr learned my addr 172.21.15.37:0/3329138043 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:07.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f8e59c640 10 monclient(hunting): get_auth_request con 0x7f8f900a0260 auth_method 0 2024-04-26T00:20:07.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f8e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:07.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f8e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:07.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f8e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:07.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f8ed9d640 10 monclient(hunting): get_auth_request con 0x7f8f90135490 auth_method 0 2024-04-26T00:20:07.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f8ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:07.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f8ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:07.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f8ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:07.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f8e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:07.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f8e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:07.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f8e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:07.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f8ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:07.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f8ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:07.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f8ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:07.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f8dd9b640 1 --2- 172.21.15.37:0/3329138043 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f90134b30 0x7f8f9009c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:07.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f8dd9b640 10 monclient(hunting): get_auth_request con 0x7f8f90134b30 auth_method 0 2024-04-26T00:20:07.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f8dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:07.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f8dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:07.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f8dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:07.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f8e59c640 10 monclient(hunting): handle_auth_done global_id 4316 payload 824 2024-04-26T00:20:07.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f8e59c640 1 -- 172.21.15.37:0/3329138043 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f90134b30 msgr2=0x7f8f9009c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:07.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f8e59c640 1 --2- 172.21.15.37:0/3329138043 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f90134b30 0x7f8f9009c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:07.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f8e59c640 1 -- 172.21.15.37:0/3329138043 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f90135490 msgr2=0x7f8f9009ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:07.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f8e59c640 1 --2- 172.21.15.37:0/3329138043 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f90135490 0x7f8f9009ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:07.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f8e59c640 10 monclient: _finish_hunting 0 2024-04-26T00:20:07.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f8e59c640 1 monclient: found mon.noname-c 2024-04-26T00:20:07.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f8e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:07.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f8e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:20:07.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.955+0000 7f8f8e59c640 1 -- 172.21.15.37:0/3329138043 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f9013b620 con 0x7f8f900a0260 2024-04-26T00:20:07.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.959+0000 7f8f8e59c640 1 -- 172.21.15.37:0/3329138043 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8f900a0260 msgr2=0x7f8f9013c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:07.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.959+0000 7f8f8e59c640 1 --2- 172.21.15.37:0/3329138043 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8f900a0260 0x7f8f9013c2c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8f7800e810 tx=0x7f8f7800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:07.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.959+0000 7f8f86ffd640 1 -- 172.21.15.37:0/3329138043 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f78017070 con 0x7f8f900a0260 2024-04-26T00:20:07.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.959+0000 7f8f86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:07.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.959+0000 7f8f86ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:20:07.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.959+0000 7f8f86ffd640 10 monclient: dump: 2024-04-26T00:20:07.947 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:07.947 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:07.947 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:07.947 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:07.947 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:07.948 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:07.948 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:07.948 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:07.948 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:07.948 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:07.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.959+0000 7f8f86ffd640 10 monclient: _finish_auth 0 2024-04-26T00:20:07.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.959+0000 7f8f86ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:20:07.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.959+0000 7f8f86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:07.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.959+0000 7f8f94e5f640 5 monclient: authenticate success, global_id 4316 2024-04-26T00:20:07.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.959+0000 7f8f94e5f640 10 monclient: _renew_subs 2024-04-26T00:20:07.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.959+0000 7f8f86ffd640 1 -- 172.21.15.37:0/3329138043 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f7800cae0 con 0x7f8f900a0260 2024-04-26T00:20:07.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.959+0000 7f8f94e5f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:07.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.959+0000 7f8f94e5f640 1 -- 172.21.15.37:0/3329138043 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f900729d0 con 0x7f8f900a0260 2024-04-26T00:20:07.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.959+0000 7f8f86ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:07.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.959+0000 7f8f86ffd640 1 -- 172.21.15.37:0/3329138043 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f7800f040 con 0x7f8f900a0260 2024-04-26T00:20:07.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.959+0000 7f8f86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:07.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.959+0000 7f8f86ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:07.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.959+0000 7f8f86ffd640 10 monclient: dump: 2024-04-26T00:20:07.949 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:07.949 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:07.949 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:07.949 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:07.949 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:07.949 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:07.949 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:07.949 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:07.949 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:07.949 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:07.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.959+0000 7f8f8cd99640 4 set_mon_vals no callback set 2024-04-26T00:20:07.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.959+0000 7f8f94e5f640 10 monclient: _renew_subs 2024-04-26T00:20:07.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.959+0000 7f8f94e5f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:07.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.959+0000 7f8f94e5f640 1 -- 172.21.15.37:0/3329138043 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f9013b340 con 0x7f8f900a0260 2024-04-26T00:20:07.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.959+0000 7f8f94e5f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:07.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.959+0000 7f8f94e5f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:07.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.959+0000 7f8f94e5f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:07.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.959+0000 7f8f94e5f640 1 -- 172.21.15.37:0/3329138043 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f90066f40 con 0x7f8f900a0260 2024-04-26T00:20:07.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.959+0000 7f8f86ffd640 1 -- 172.21.15.37:0/3329138043 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f8f7801a070 con 0x7f8f900a0260 2024-04-26T00:20:07.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.959+0000 7f8f86ffd640 1 --2- 172.21.15.37:0/3329138043 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8f70035ed0 0x7f8f70038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:07.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.959+0000 7f8f86ffd640 1 -- 172.21.15.37:0/3329138043 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7f8f78048580 con 0x7f8f900a0260 2024-04-26T00:20:07.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.959+0000 7f8f8dd9b640 1 --2- 172.21.15.37:0/3329138043 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8f70035ed0 0x7f8f70038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:07.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.959+0000 7f8f8dd9b640 10 monclient: get_auth_request con 0x7f8f70035ed0 auth_method 0 2024-04-26T00:20:07.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.963+0000 7f8f8dd9b640 1 --2- 172.21.15.37:0/3329138043 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8f70035ed0 0x7f8f70038390 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8f900a1a60 tx=0x7f8f80008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:07.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.963+0000 7f8f86ffd640 1 -- 172.21.15.37:0/3329138043 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f8f78024350 con 0x7f8f900a0260 2024-04-26T00:20:07.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.963+0000 7f8f86ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:07.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:07.963+0000 7f8f86ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:08.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.123+0000 7f8f94e5f640 1 -- 172.21.15.37:0/3329138043 --> v2:172.21.15.45:6812/2379918623 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8f90139c10 con 0x7f8f70035ed0 2024-04-26T00:20:08.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.127+0000 7f8f86ffd640 1 -- 172.21.15.37:0/3329138043 <== mgr.4107 v2:172.21.15.45:6812/2379918623 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33873 (secure 0 0 0) 0x7f8f90139c10 con 0x7f8f70035ed0 2024-04-26T00:20:08.109 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:20:08.110 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-04-26T00:20:08.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.127+0000 7f8f94e5f640 1 -- 172.21.15.37:0/3329138043 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8f70035ed0 msgr2=0x7f8f70038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:08.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.127+0000 7f8f94e5f640 1 --2- 172.21.15.37:0/3329138043 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8f70035ed0 0x7f8f70038390 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8f900a1a60 tx=0x7f8f80008040 comp rx=0 tx=0).stop 2024-04-26T00:20:08.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.127+0000 7f8f94e5f640 10 monclient: shutdown 2024-04-26T00:20:08.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.127+0000 7f8f94e5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:08.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.127+0000 7f8f94e5f640 1 -- 172.21.15.37:0/3329138043 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8f900a0260 msgr2=0x7f8f9013c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:08.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.127+0000 7f8f94e5f640 1 --2- 172.21.15.37:0/3329138043 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8f900a0260 0x7f8f9013c2c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8f7800e810 tx=0x7f8f7800ecd0 comp rx=0 tx=0).stop 2024-04-26T00:20:08.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.127+0000 7f8f94e5f640 1 -- 172.21.15.37:0/3329138043 shutdown_connections 2024-04-26T00:20:08.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.127+0000 7f8f94e5f640 1 --2- 172.21.15.37:0/3329138043 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8f70035ed0 0x7f8f70038390 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:08.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.127+0000 7f8f94e5f640 1 --2- 172.21.15.37:0/3329138043 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f90135490 0x7f8f9009ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:08.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.127+0000 7f8f94e5f640 1 --2- 172.21.15.37:0/3329138043 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f90134b30 0x7f8f9009c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:08.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.127+0000 7f8f94e5f640 1 --2- 172.21.15.37:0/3329138043 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8f900a0260 0x7f8f9013c2c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:08.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.127+0000 7f8f94e5f640 1 -- 172.21.15.37:0/3329138043 >> 172.21.15.37:0/3329138043 conn(0x7f8f900a1ad0 msgr2=0x7f8f900a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:08.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.127+0000 7f8f94e5f640 1 -- 172.21.15.37:0/3329138043 shutdown_connections 2024-04-26T00:20:08.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.127+0000 7f8f94e5f640 1 -- 172.21.15.37:0/3329138043 wait complete. 2024-04-26T00:20:08.127 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-04-26T00:20:07.150239+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":167200,"kb_used_data":888,"kb_used_omap":9,"kb_used_meta":166198,"kb_avail":566063840,"statfs":{"total":579820584960,"available":579649372160,"internally_reserved":0,"allocated":909312,"data_stored":1285862,"data_compressed":61234,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":170187424},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000612"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T00:20:04.049797+0000","last_change":"2024-04-26T00:20:02.475146+0000","last_active":"2024-04-26T00:20:04.049797+0000","last_peered":"2024-04-26T00:20:04.049797+0000","last_clean":"2024-04-26T00:20:04.049797+0000","last_became_active":"2024-04-26T00:20:02.473928+0000","last_became_peered":"2024-04-26T00:20:02.473928+0000","last_unstale":"2024-04-26T00:20:04.049797+0000","last_undegraded":"2024-04-26T00:20:04.049797+0000","last_fullsized":"2024-04-26T00:20:04.049797+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:25:12.904295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T00:20:03.547413+0000","last_change":"2024-04-26T00:20:02.279987+0000","last_active":"2024-04-26T00:20:03.547413+0000","last_peered":"2024-04-26T00:20:03.547413+0000","last_clean":"2024-04-26T00:20:03.547413+0000","last_became_active":"2024-04-26T00:20:02.279183+0000","last_became_peered":"2024-04-26T00:20:02.279183+0000","last_unstale":"2024-04-26T00:20:03.547413+0000","last_undegraded":"2024-04-26T00:20:03.547413+0000","last_fullsized":"2024-04-26T00:20:03.547413+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:45:44.508952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T00:20:03.268034+0000","last_change":"2024-04-26T00:20:02.474779+0000","last_active":"2024-04-26T00:20:03.268034+0000","last_peered":"2024-04-26T00:20:03.268034+0000","last_clean":"2024-04-26T00:20:03.268034+0000","last_became_active":"2024-04-26T00:20:02.473888+0000","last_became_peered":"2024-04-26T00:20:02.473888+0000","last_unstale":"2024-04-26T00:20:03.268034+0000","last_undegraded":"2024-04-26T00:20:03.268034+0000","last_fullsized":"2024-04-26T00:20:03.268034+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:33:18.962907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T00:20:03.266877+0000","last_change":"2024-04-26T00:20:02.274249+0000","last_active":"2024-04-26T00:20:03.266877+0000","last_peered":"2024-04-26T00:20:03.266877+0000","last_clean":"2024-04-26T00:20:03.266877+0000","last_became_active":"2024-04-26T00:20:02.273745+0000","last_became_peered":"2024-04-26T00:20:02.273745+0000","last_unstale":"2024-04-26T00:20:03.266877+0000","last_undegraded":"2024-04-26T00:20:03.266877+0000","last_fullsized":"2024-04-26T00:20:03.266877+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:36:25.948539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"15'117","reported_seq":162,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T00:20:03.649880+0000","last_change":"2024-04-26T00:20:02.281782+0000","last_active":"2024-04-26T00:20:03.649880+0000","last_peered":"2024-04-26T00:20:03.649880+0000","last_clean":"2024-04-26T00:20:03.649880+0000","last_became_active":"2024-04-26T00:20:02.281000+0000","last_became_peered":"2024-04-26T00:20:02.281000+0000","last_unstale":"2024-04-26T00:20:03.649880+0000","last_undegraded":"2024-04-26T00:20:03.649880+0000","last_fullsized":"2024-04-26T00:20:03.649880+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:40:30.790500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T00:20:03.267137+0000","last_change":"2024-04-26T00:20:02.273965+0000","last_active":"2024-04-26T00:20:03.267137+0000","last_peered":"2024-04-26T00:20:03.267137+0000","last_clean":"2024-04-26T00:20:03.267137+0000","last_became_active":"2024-04-26T00:20:02.273213+0000","last_became_peered":"2024-04-26T00:20:02.273213+0000","last_unstale":"2024-04-26T00:20:03.267137+0000","last_undegraded":"2024-04-26T00:20:03.267137+0000","last_fullsized":"2024-04-26T00:20:03.267137+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:42:14.055403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T00:20:03.268104+0000","last_change":"2024-04-26T00:20:02.277311+0000","last_active":"2024-04-26T00:20:03.268104+0000","last_peered":"2024-04-26T00:20:03.268104+0000","last_clean":"2024-04-26T00:20:03.268104+0000","last_became_active":"2024-04-26T00:20:02.276409+0000","last_became_peered":"2024-04-26T00:20:02.276409+0000","last_unstale":"2024-04-26T00:20:03.268104+0000","last_undegraded":"2024-04-26T00:20:03.268104+0000","last_fullsized":"2024-04-26T00:20:03.268104+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:14:50.584720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T00:20:04.049844+0000","last_change":"2024-04-26T00:20:02.279805+0000","last_active":"2024-04-26T00:20:04.049844+0000","last_peered":"2024-04-26T00:20:04.049844+0000","last_clean":"2024-04-26T00:20:04.049844+0000","last_became_active":"2024-04-26T00:20:02.279053+0000","last_became_peered":"2024-04-26T00:20:02.279053+0000","last_unstale":"2024-04-26T00:20:04.049844+0000","last_undegraded":"2024-04-26T00:20:04.049844+0000","last_fullsized":"2024-04-26T00:20:04.049844+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:11:26.730719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T00:20:03.547492+0000","last_change":"2024-04-26T00:20:02.279764+0000","last_active":"2024-04-26T00:20:03.547492+0000","last_peered":"2024-04-26T00:20:03.547492+0000","last_clean":"2024-04-26T00:20:03.547492+0000","last_became_active":"2024-04-26T00:20:02.278959+0000","last_became_peered":"2024-04-26T00:20:02.278959+0000","last_unstale":"2024-04-26T00:20:03.547492+0000","last_undegraded":"2024-04-26T00:20:03.547492+0000","last_fullsized":"2024-04-26T00:20:03.547492+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:23:43.610174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":61234,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":38001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31312,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340528,"statfs":{"total":96636764160,"available":96604700672,"internally_reserved":0,"allocated":131072,"data_stored":38001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":192512,"data_stored":566929,"data_compressed":30617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":38001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":192512,"data_stored":566929,"data_compressed":30617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":38001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T00:20:08.128 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T00:20:08.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.263+0000 7f3a0b176640 10 monclient: get_monmap_and_config 2024-04-26T00:20:08.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.263+0000 7f3a0b176640 10 monclient: build_initial_monmap 2024-04-26T00:20:08.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.263+0000 7f3a0b176640 10 monclient: monmap: 2024-04-26T00:20:08.246 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:08.247 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:08.247 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:08.270332+0000 2024-04-26T00:20:08.247 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:08.270332+0000 2024-04-26T00:20:08.247 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:08.247 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:08.247 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:08.247 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:08.247 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:08.247 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:08.247 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:08.247 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:08.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.263+0000 7f3a0b176640 1 Processor -- start 2024-04-26T00:20:08.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.263+0000 7f3a0b176640 1 -- start start 2024-04-26T00:20:08.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.263+0000 7f3a0b176640 10 monclient: init 2024-04-26T00:20:08.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.263+0000 7f3a0b176640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:08.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.263+0000 7f3a0b176640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-04-26T00:20:08.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.263+0000 7f3a0b176640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3a04071fc0 addr v1:172.21.15.45:6789/0 2024-04-26T00:20:08.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.263+0000 7f3a0b176640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3a040723a0 0x7f3a04072810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:08.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.263+0000 7f3a0b176640 10 monclient(hunting): picked mon.noname-c con 0x7f3a040723a0 addr v2:172.21.15.45:3300/0 2024-04-26T00:20:08.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.263+0000 7f3a0b176640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3a0413c340 addr v1:172.21.15.37:6789/0 2024-04-26T00:20:08.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.263+0000 7f3a0b176640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a04073200 con 0x7f3a0413c340 2024-04-26T00:20:08.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.267+0000 7f3a0b176640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a040a0c20 con 0x7f3a04071fc0 2024-04-26T00:20:08.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.267+0000 7f3a0b176640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:08.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.267+0000 7f3a0b176640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f3a040a0f00 con 0x7f3a040723a0 2024-04-26T00:20:08.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.267+0000 7f3a0b176640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:08.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.267+0000 7f3a0b176640 10 monclient(hunting): authenticate will time out at 989.576043s 2024-04-26T00:20:08.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.267+0000 7f3a09713640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f3a0413c340 0x7f3a0413e730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:55858/0 (socket says 172.21.15.37:55858) 2024-04-26T00:20:08.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.267+0000 7f3a09713640 1 -- 172.21.15.37:0/847739115 learned_addr learned my addr 172.21.15.37:0/847739115 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:20:08.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.267+0000 7f3a03fff640 1 --2- 172.21.15.37:0/847739115 >> v2:172.21.15.45:3300/0 conn(0x7f3a040723a0 0x7f3a04072810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:08.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.267+0000 7f3a03fff640 10 monclient(hunting): get_auth_request con 0x7f3a040723a0 auth_method 0 2024-04-26T00:20:08.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.267+0000 7f3a03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:08.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.267+0000 7f3a03fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:08.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.267+0000 7f3a03fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:08.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.267+0000 7f3a03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:08.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.267+0000 7f3a03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:08.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.267+0000 7f3a03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:08.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.267+0000 7f3a03fff640 10 monclient(hunting): handle_auth_done global_id 4306 payload 274 2024-04-26T00:20:08.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.267+0000 7f3a03fff640 1 -- 172.21.15.37:0/847739115 >> v1:172.21.15.45:6789/0 conn(0x7f3a04071fc0 legacy=0x7f3a04137240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:08.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.267+0000 7f3a03fff640 1 -- 172.21.15.37:0/847739115 >> v1:172.21.15.37:6789/0 conn(0x7f3a0413c340 legacy=0x7f3a0413e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:08.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.267+0000 7f3a03fff640 10 monclient: _finish_hunting 0 2024-04-26T00:20:08.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.267+0000 7f3a03fff640 1 monclient: found mon.noname-c 2024-04-26T00:20:08.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.267+0000 7f3a03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:08.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.267+0000 7f3a03fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:20:08.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.267+0000 7f3a03fff640 1 -- 172.21.15.37:0/847739115 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a04137950 con 0x7f3a040723a0 2024-04-26T00:20:08.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.271+0000 7f3a03fff640 1 -- 172.21.15.37:0/847739115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a040723a0 msgr2=0x7f3a04072810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:08.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.271+0000 7f3a03fff640 1 --2- 172.21.15.37:0/847739115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a040723a0 0x7f3a04072810 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f39f40097c0 tx=0x7f39f401c2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd1336a99c89ec57 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:08.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.271+0000 7f3a037fe640 1 -- 172.21.15.37:0/847739115 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39f40b0070 con 0x7f3a040723a0 2024-04-26T00:20:08.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.271+0000 7f3a037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:08.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.271+0000 7f3a037fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:20:08.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.271+0000 7f3a037fe640 10 monclient: dump: 2024-04-26T00:20:08.254 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:08.254 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:08.254 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:08.254 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:08.254 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:08.254 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:08.255 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:08.255 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:08.255 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:08.255 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:08.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.271+0000 7f3a037fe640 10 monclient: _finish_auth 0 2024-04-26T00:20:08.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.271+0000 7f3a037fe640 10 monclient: _check_auth_tickets 2024-04-26T00:20:08.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.271+0000 7f3a037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:08.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.271+0000 7f3a037fe640 1 -- 172.21.15.37:0/847739115 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39f40ab600 con 0x7f3a040723a0 2024-04-26T00:20:08.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.271+0000 7f3a037fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:08.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.271+0000 7f3a037fe640 1 -- 172.21.15.37:0/847739115 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39f40ab9d0 con 0x7f3a040723a0 2024-04-26T00:20:08.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.271+0000 7f3a0b176640 5 monclient: authenticate success, global_id 4306 2024-04-26T00:20:08.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.271+0000 7f3a0b176640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:08.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.271+0000 7f3a0b176640 4 set_mon_vals no callback set 2024-04-26T00:20:08.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.271+0000 7f3a037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:08.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.271+0000 7f3a037fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:20:08.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.271+0000 7f3a037fe640 10 monclient: dump: 2024-04-26T00:20:08.256 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:08.256 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:08.256 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:08.256 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:08.256 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:08.256 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:08.256 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:08.256 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:08.256 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:08.256 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:08.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.271+0000 7f3a0b176640 10 monclient: shutdown 2024-04-26T00:20:08.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.271+0000 7f3a0b176640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:08.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.271+0000 7f3a0b176640 1 -- 172.21.15.37:0/847739115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a040723a0 msgr2=0x7f3a04072810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:08.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.271+0000 7f3a0b176640 1 --2- 172.21.15.37:0/847739115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a040723a0 0x7f3a04072810 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f39f40097c0 tx=0x7f39f401c2e0 comp rx=0 tx=0).stop 2024-04-26T00:20:08.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.271+0000 7f3a0b176640 1 -- 172.21.15.37:0/847739115 shutdown_connections 2024-04-26T00:20:08.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.271+0000 7f3a0b176640 1 --2- 172.21.15.37:0/847739115 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a040723a0 0x7f3a04072810 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:08.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.271+0000 7f3a0b176640 1 -- 172.21.15.37:0/847739115 >> 172.21.15.37:0/847739115 conn(0x7f3a040a1ad0 msgr2=0x7f3a040a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:08.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.271+0000 7f3a0b176640 1 -- 172.21.15.37:0/847739115 shutdown_connections 2024-04-26T00:20:08.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.271+0000 7f3a0b176640 1 -- 172.21.15.37:0/847739115 wait complete. 2024-04-26T00:20:08.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.271+0000 7f3a0b176640 10 monclient: build_initial_monmap 2024-04-26T00:20:08.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.271+0000 7f3a0b176640 10 monclient: monmap: 2024-04-26T00:20:08.261 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:08.261 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:08.261 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:08.261 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:08.261 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:08.261 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:08.261 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:08.261 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:08.261 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:08.261 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:08.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a0b176640 1 Processor -- start 2024-04-26T00:20:08.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a0b176640 1 -- start start 2024-04-26T00:20:08.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a0b176640 10 monclient: init 2024-04-26T00:20:08.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a0b176640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:08.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a0b176640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:20:08.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a0b176640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a04071fc0 0x7f3a0413b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:08.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a0b176640 10 monclient(hunting): picked mon.noname-b con 0x7f3a04071fc0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:08.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a0b176640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a040723a0 0x7f3a04130bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:08.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a0b176640 10 monclient(hunting): picked mon.noname-a con 0x7f3a040723a0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:08.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a0b176640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3a0413c340 0x7f3a04131100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:08.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a0b176640 10 monclient(hunting): picked mon.noname-c con 0x7f3a0413c340 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:08.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a0b176640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:08.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a0b176640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f3a04142a40 con 0x7f3a040723a0 2024-04-26T00:20:08.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a0b176640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:08.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a0b176640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3a041428c0 con 0x7f3a04071fc0 2024-04-26T00:20:08.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a0b176640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:08.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a0b176640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f3a04142bc0 con 0x7f3a0413c340 2024-04-26T00:20:08.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a0b176640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:08.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a0b176640 10 monclient(hunting): authenticate will time out at 989.584904s 2024-04-26T00:20:08.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a03fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a040723a0 0x7f3a04130bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:08.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a09713640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3a0413c340 0x7f3a04131100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:08.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a03fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a040723a0 0x7f3a04130bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44926/0 (socket says 172.21.15.37:44926) 2024-04-26T00:20:08.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a03fff640 1 -- 172.21.15.37:0/3469682215 learned_addr learned my addr 172.21.15.37:0/3469682215 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:08.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a03fff640 10 monclient(hunting): get_auth_request con 0x7f3a040723a0 auth_method 0 2024-04-26T00:20:08.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:08.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a03fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:08.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a03fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:08.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a09713640 10 monclient(hunting): get_auth_request con 0x7f3a0413c340 auth_method 0 2024-04-26T00:20:08.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a09713640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:08.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a09713640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:08.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a09713640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:08.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a09713640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:08.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a09713640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:08.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a09713640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:08.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:08.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:08.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:08.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a08f12640 1 --2- 172.21.15.37:0/3469682215 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a04071fc0 0x7f3a0413b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:08.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a08f12640 10 monclient(hunting): get_auth_request con 0x7f3a04071fc0 auth_method 0 2024-04-26T00:20:08.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a08f12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:08.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a08f12640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:08.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a08f12640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:08.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a09713640 10 monclient(hunting): handle_auth_done global_id 4319 payload 824 2024-04-26T00:20:08.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a09713640 1 -- 172.21.15.37:0/3469682215 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a04071fc0 msgr2=0x7f3a0413b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:08.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a09713640 1 --2- 172.21.15.37:0/3469682215 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a04071fc0 0x7f3a0413b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:08.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a09713640 1 -- 172.21.15.37:0/3469682215 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a040723a0 msgr2=0x7f3a04130bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:08.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a09713640 1 --2- 172.21.15.37:0/3469682215 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a040723a0 0x7f3a04130bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:08.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a09713640 10 monclient: _finish_hunting 0 2024-04-26T00:20:08.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a09713640 1 monclient: found mon.noname-c 2024-04-26T00:20:08.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a09713640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:08.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a09713640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:20:08.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a09713640 1 -- 172.21.15.37:0/3469682215 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a0413a190 con 0x7f3a0413c340 2024-04-26T00:20:08.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a09713640 1 -- 172.21.15.37:0/3469682215 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3a0413c340 msgr2=0x7f3a04131100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:08.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a09713640 1 --2- 172.21.15.37:0/3469682215 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3a0413c340 0x7f3a04131100 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f39fc00cac0 tx=0x7f39fc00cec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:08.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a017fa640 1 -- 172.21.15.37:0/3469682215 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39fc00ad80 con 0x7f3a0413c340 2024-04-26T00:20:08.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:08.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a017fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:20:08.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a017fa640 10 monclient: dump: 2024-04-26T00:20:08.265 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:08.265 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:08.265 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:08.265 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:08.266 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:08.266 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:08.266 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:08.266 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:08.266 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:08.266 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:08.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a017fa640 10 monclient: _finish_auth 0 2024-04-26T00:20:08.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a017fa640 10 monclient: _check_auth_tickets 2024-04-26T00:20:08.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:08.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a0b176640 5 monclient: authenticate success, global_id 4319 2024-04-26T00:20:08.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a017fa640 1 -- 172.21.15.37:0/3469682215 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39fc012400 con 0x7f3a0413c340 2024-04-26T00:20:08.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a0b176640 10 monclient: _renew_subs 2024-04-26T00:20:08.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a0b176640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:08.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.275+0000 7f3a0b176640 1 -- 172.21.15.37:0/3469682215 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a0413a470 con 0x7f3a0413c340 2024-04-26T00:20:08.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.279+0000 7f3a017fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:08.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.279+0000 7f3a017fa640 1 -- 172.21.15.37:0/3469682215 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39fc0127f0 con 0x7f3a0413c340 2024-04-26T00:20:08.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.279+0000 7f3a017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:08.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.279+0000 7f3a017fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:08.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.279+0000 7f3a017fa640 10 monclient: dump: 2024-04-26T00:20:08.267 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:08.267 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:08.267 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:08.267 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:08.267 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:08.267 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:08.267 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:08.267 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:08.267 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:08.267 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:08.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.279+0000 7f3a02ffd640 4 set_mon_vals no callback set 2024-04-26T00:20:08.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.279+0000 7f3a0b176640 10 monclient: _renew_subs 2024-04-26T00:20:08.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.279+0000 7f3a0b176640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:08.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.279+0000 7f3a0b176640 1 -- 172.21.15.37:0/3469682215 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a04131640 con 0x7f3a0413c340 2024-04-26T00:20:08.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.279+0000 7f3a0b176640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:08.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.279+0000 7f3a0b176640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:08.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.279+0000 7f3a0b176640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:08.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.279+0000 7f3a0b176640 1 -- 172.21.15.37:0/3469682215 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a0413b9d0 con 0x7f3a0413c340 2024-04-26T00:20:08.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.279+0000 7f3a017fa640 1 -- 172.21.15.37:0/3469682215 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f39fc012a10 con 0x7f3a0413c340 2024-04-26T00:20:08.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.279+0000 7f3a017fa640 1 --2- 172.21.15.37:0/3469682215 >> v2:172.21.15.45:6812/2379918623 conn(0x7f39f0035e80 0x7f39f0038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:08.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.279+0000 7f3a017fa640 1 -- 172.21.15.37:0/3469682215 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7f39fc048f00 con 0x7f3a0413c340 2024-04-26T00:20:08.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.279+0000 7f3a08f12640 1 --2- 172.21.15.37:0/3469682215 >> v2:172.21.15.45:6812/2379918623 conn(0x7f39f0035e80 0x7f39f0038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:08.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.279+0000 7f3a08f12640 10 monclient: get_auth_request con 0x7f39f0035e80 auth_method 0 2024-04-26T00:20:08.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.283+0000 7f3a08f12640 1 --2- 172.21.15.37:0/3469682215 >> v2:172.21.15.45:6812/2379918623 conn(0x7f39f0035e80 0x7f39f0038340 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f39f8004e30 tx=0x7f39f8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:08.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.283+0000 7f3a017fa640 1 -- 172.21.15.37:0/3469682215 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f39fc01b4b0 con 0x7f3a0413c340 2024-04-26T00:20:08.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.283+0000 7f3a017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:08.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.283+0000 7f3a017fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:08.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.439+0000 7f3a0b176640 1 -- 172.21.15.37:0/3469682215 --> v2:172.21.15.45:6812/2379918623 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3a04066f40 con 0x7f39f0035e80 2024-04-26T00:20:08.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.439+0000 7f3a017fa640 1 -- 172.21.15.37:0/3469682215 <== mgr.4107 v2:172.21.15.45:6812/2379918623 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33873 (secure 0 0 0) 0x7f3a04066f40 con 0x7f39f0035e80 2024-04-26T00:20:08.424 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:20:08.424 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-04-26T00:20:08.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.443+0000 7f3a0b176640 1 -- 172.21.15.37:0/3469682215 >> v2:172.21.15.45:6812/2379918623 conn(0x7f39f0035e80 msgr2=0x7f39f0038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:08.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.443+0000 7f3a0b176640 1 --2- 172.21.15.37:0/3469682215 >> v2:172.21.15.45:6812/2379918623 conn(0x7f39f0035e80 0x7f39f0038340 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f39f8004e30 tx=0x7f39f8009040 comp rx=0 tx=0).stop 2024-04-26T00:20:08.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.443+0000 7f3a0b176640 10 monclient: shutdown 2024-04-26T00:20:08.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.443+0000 7f3a0b176640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:08.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.443+0000 7f3a0b176640 1 -- 172.21.15.37:0/3469682215 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3a0413c340 msgr2=0x7f3a04131100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:08.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.443+0000 7f3a0b176640 1 --2- 172.21.15.37:0/3469682215 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3a0413c340 0x7f3a04131100 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f39fc00cac0 tx=0x7f39fc00cec0 comp rx=0 tx=0).stop 2024-04-26T00:20:08.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.443+0000 7f3a0b176640 1 -- 172.21.15.37:0/3469682215 shutdown_connections 2024-04-26T00:20:08.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.443+0000 7f3a0b176640 1 --2- 172.21.15.37:0/3469682215 >> v2:172.21.15.45:6812/2379918623 conn(0x7f39f0035e80 0x7f39f0038340 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:08.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.443+0000 7f3a0b176640 1 --2- 172.21.15.37:0/3469682215 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3a0413c340 0x7f3a04131100 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:08.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.443+0000 7f3a0b176640 1 --2- 172.21.15.37:0/3469682215 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a040723a0 0x7f3a04130bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:08.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.443+0000 7f3a0b176640 1 --2- 172.21.15.37:0/3469682215 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a04071fc0 0x7f3a0413b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:08.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.443+0000 7f3a0b176640 1 -- 172.21.15.37:0/3469682215 >> 172.21.15.37:0/3469682215 conn(0x7f3a040a1ad0 msgr2=0x7f3a04134a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:08.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.443+0000 7f3a0b176640 1 -- 172.21.15.37:0/3469682215 shutdown_connections 2024-04-26T00:20:08.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.443+0000 7f3a0b176640 1 -- 172.21.15.37:0/3469682215 wait complete. 2024-04-26T00:20:08.441 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-04-26T00:20:07.150239+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":167200,"kb_used_data":888,"kb_used_omap":9,"kb_used_meta":166198,"kb_avail":566063840,"statfs":{"total":579820584960,"available":579649372160,"internally_reserved":0,"allocated":909312,"data_stored":1285862,"data_compressed":61234,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":170187424},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000612"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T00:20:04.049797+0000","last_change":"2024-04-26T00:20:02.475146+0000","last_active":"2024-04-26T00:20:04.049797+0000","last_peered":"2024-04-26T00:20:04.049797+0000","last_clean":"2024-04-26T00:20:04.049797+0000","last_became_active":"2024-04-26T00:20:02.473928+0000","last_became_peered":"2024-04-26T00:20:02.473928+0000","last_unstale":"2024-04-26T00:20:04.049797+0000","last_undegraded":"2024-04-26T00:20:04.049797+0000","last_fullsized":"2024-04-26T00:20:04.049797+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:25:12.904295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T00:20:03.547413+0000","last_change":"2024-04-26T00:20:02.279987+0000","last_active":"2024-04-26T00:20:03.547413+0000","last_peered":"2024-04-26T00:20:03.547413+0000","last_clean":"2024-04-26T00:20:03.547413+0000","last_became_active":"2024-04-26T00:20:02.279183+0000","last_became_peered":"2024-04-26T00:20:02.279183+0000","last_unstale":"2024-04-26T00:20:03.547413+0000","last_undegraded":"2024-04-26T00:20:03.547413+0000","last_fullsized":"2024-04-26T00:20:03.547413+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:45:44.508952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T00:20:03.268034+0000","last_change":"2024-04-26T00:20:02.474779+0000","last_active":"2024-04-26T00:20:03.268034+0000","last_peered":"2024-04-26T00:20:03.268034+0000","last_clean":"2024-04-26T00:20:03.268034+0000","last_became_active":"2024-04-26T00:20:02.473888+0000","last_became_peered":"2024-04-26T00:20:02.473888+0000","last_unstale":"2024-04-26T00:20:03.268034+0000","last_undegraded":"2024-04-26T00:20:03.268034+0000","last_fullsized":"2024-04-26T00:20:03.268034+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:33:18.962907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T00:20:03.266877+0000","last_change":"2024-04-26T00:20:02.274249+0000","last_active":"2024-04-26T00:20:03.266877+0000","last_peered":"2024-04-26T00:20:03.266877+0000","last_clean":"2024-04-26T00:20:03.266877+0000","last_became_active":"2024-04-26T00:20:02.273745+0000","last_became_peered":"2024-04-26T00:20:02.273745+0000","last_unstale":"2024-04-26T00:20:03.266877+0000","last_undegraded":"2024-04-26T00:20:03.266877+0000","last_fullsized":"2024-04-26T00:20:03.266877+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:36:25.948539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"15'117","reported_seq":162,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T00:20:03.649880+0000","last_change":"2024-04-26T00:20:02.281782+0000","last_active":"2024-04-26T00:20:03.649880+0000","last_peered":"2024-04-26T00:20:03.649880+0000","last_clean":"2024-04-26T00:20:03.649880+0000","last_became_active":"2024-04-26T00:20:02.281000+0000","last_became_peered":"2024-04-26T00:20:02.281000+0000","last_unstale":"2024-04-26T00:20:03.649880+0000","last_undegraded":"2024-04-26T00:20:03.649880+0000","last_fullsized":"2024-04-26T00:20:03.649880+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:40:30.790500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T00:20:03.267137+0000","last_change":"2024-04-26T00:20:02.273965+0000","last_active":"2024-04-26T00:20:03.267137+0000","last_peered":"2024-04-26T00:20:03.267137+0000","last_clean":"2024-04-26T00:20:03.267137+0000","last_became_active":"2024-04-26T00:20:02.273213+0000","last_became_peered":"2024-04-26T00:20:02.273213+0000","last_unstale":"2024-04-26T00:20:03.267137+0000","last_undegraded":"2024-04-26T00:20:03.267137+0000","last_fullsized":"2024-04-26T00:20:03.267137+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:42:14.055403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T00:20:03.268104+0000","last_change":"2024-04-26T00:20:02.277311+0000","last_active":"2024-04-26T00:20:03.268104+0000","last_peered":"2024-04-26T00:20:03.268104+0000","last_clean":"2024-04-26T00:20:03.268104+0000","last_became_active":"2024-04-26T00:20:02.276409+0000","last_became_peered":"2024-04-26T00:20:02.276409+0000","last_unstale":"2024-04-26T00:20:03.268104+0000","last_undegraded":"2024-04-26T00:20:03.268104+0000","last_fullsized":"2024-04-26T00:20:03.268104+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:14:50.584720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T00:20:04.049844+0000","last_change":"2024-04-26T00:20:02.279805+0000","last_active":"2024-04-26T00:20:04.049844+0000","last_peered":"2024-04-26T00:20:04.049844+0000","last_clean":"2024-04-26T00:20:04.049844+0000","last_became_active":"2024-04-26T00:20:02.279053+0000","last_became_peered":"2024-04-26T00:20:02.279053+0000","last_unstale":"2024-04-26T00:20:04.049844+0000","last_undegraded":"2024-04-26T00:20:04.049844+0000","last_fullsized":"2024-04-26T00:20:04.049844+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:11:26.730719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T00:20:03.547492+0000","last_change":"2024-04-26T00:20:02.279764+0000","last_active":"2024-04-26T00:20:03.547492+0000","last_peered":"2024-04-26T00:20:03.547492+0000","last_clean":"2024-04-26T00:20:03.547492+0000","last_became_active":"2024-04-26T00:20:02.278959+0000","last_became_peered":"2024-04-26T00:20:02.278959+0000","last_unstale":"2024-04-26T00:20:03.547492+0000","last_undegraded":"2024-04-26T00:20:03.547492+0000","last_fullsized":"2024-04-26T00:20:03.547492+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:23:43.610174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":61234,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":38001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31312,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340528,"statfs":{"total":96636764160,"available":96604700672,"internally_reserved":0,"allocated":131072,"data_stored":38001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":192512,"data_stored":566929,"data_compressed":30617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":38001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":192512,"data_stored":566929,"data_compressed":30617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":38001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T00:20:08.442 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-04-26T00:20:08.442 DEBUG:tasks.mon_thrash:Yielding 2024-04-26T00:20:08.442 INFO:teuthology.run_tasks:Running task rados... 2024-04-26T00:20:08.454 INFO:tasks.rados:Beginning rados... 2024-04-26T00:20:08.454 DEBUG:teuthology.run_tasks:Unwinding manager rados 2024-04-26T00:20:08.494 INFO:tasks.rados:joining rados 2024-04-26T00:20:08.494 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-04-26T00:20:08.494 INFO:tasks.mon_thrash.mon_thrasher:seed: 1714090808, revive delay: 20.0, thrash delay: 1.0 thrash many: True, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 20.0 2024-04-26T00:20:08.495 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:20:08.495 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:20:08.495 INFO:tasks.rados:clients are ['client.0'] 2024-04-26T00:20:08.495 INFO:tasks.rados:starting run 0 out of 1 2024-04-26T00:20:08.495 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-04-26T00:20:08.496 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2024-04-26T00:20:08.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.635+0000 7fdf1524c640 10 monclient: get_monmap_and_config 2024-04-26T00:20:08.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.635+0000 7fdf1524c640 10 monclient: build_initial_monmap 2024-04-26T00:20:08.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.635+0000 7fdf1524c640 10 monclient: monmap: 2024-04-26T00:20:08.620 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:08.620 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:08.620 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:08.643370+0000 2024-04-26T00:20:08.620 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:08.643370+0000 2024-04-26T00:20:08.620 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:08.620 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:08.620 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:08.620 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:08.620 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:08.620 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:08.621 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:08.621 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:08.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f07a15d4640 10 monclient: get_monmap_and_config 2024-04-26T00:20:08.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f07a15d4640 10 monclient: build_initial_monmap 2024-04-26T00:20:08.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f07a15d4640 10 monclient: monmap: 2024-04-26T00:20:08.622 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:08.622 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:08.622 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:08.643912+0000 2024-04-26T00:20:08.622 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:08.643912+0000 2024-04-26T00:20:08.622 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:08.622 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:08.622 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:08.622 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:08.622 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:08.622 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:08.622 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:08.623 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:08.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7fdf1524c640 1 Processor -- start 2024-04-26T00:20:08.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7fdf1524c640 1 -- start start 2024-04-26T00:20:08.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7fdf1524c640 10 monclient: init 2024-04-26T00:20:08.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f07a15d4640 1 Processor -- start 2024-04-26T00:20:08.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f07a15d4640 1 -- start start 2024-04-26T00:20:08.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f07a15d4640 10 monclient: init 2024-04-26T00:20:08.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7fdf1524c640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:08.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7fdf1524c640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-04-26T00:20:08.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7fdf1524c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdf100726b0 addr v1:172.21.15.37:6789/0 2024-04-26T00:20:08.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7fdf1524c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdf100a0110 0x7fdf1013c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:08.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7fdf1524c640 10 monclient(hunting): picked mon.noname-b con 0x7fdf100a0110 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:08.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7fdf1524c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdf1013c630 addr v1:172.21.15.45:6789/0 2024-04-26T00:20:08.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7fdf1524c640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf100a1470 con 0x7fdf100726b0 2024-04-26T00:20:08.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7fdf1524c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf100a0bd0 con 0x7fdf1013c630 2024-04-26T00:20:08.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7fdf1524c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:08.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7fdf1524c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fdf100a1bd0 con 0x7fdf100a0110 2024-04-26T00:20:08.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7fdf1524c640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:08.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7fdf1524c640 10 monclient(hunting): authenticate will time out at 989.949114s 2024-04-26T00:20:08.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7fdf0e59c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdf100a0110 0x7fdf1013c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:08.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7fdf0ed9d640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fdf100726b0 0x7fdf10072a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:55866/0 (socket says 172.21.15.37:55866) 2024-04-26T00:20:08.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7fdf0e59c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdf100a0110 0x7fdf1013c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:58100/0 (socket says 172.21.15.37:58100) 2024-04-26T00:20:08.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7fdf0ed9d640 1 -- 172.21.15.37:0/3551989839 learned_addr learned my addr 172.21.15.37:0/3551989839 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:20:08.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7fdf0e59c640 10 monclient(hunting): get_auth_request con 0x7fdf100a0110 auth_method 0 2024-04-26T00:20:08.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7fdf0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:08.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7fdf0e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:08.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7fdf0e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:08.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7fdf0e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:08.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7fdf0e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:08.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7fdf0e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:08.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f07a15d4640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:08.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f07a15d4640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-04-26T00:20:08.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f07a15d4640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f079c0a0260 0x7f079c0a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:08.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f07a15d4640 10 monclient(hunting): picked mon.noname-a con 0x7f079c0a0260 addr v2:172.21.15.37:3300/0 2024-04-26T00:20:08.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f07a15d4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f079c134b30 addr v1:172.21.15.45:6789/0 2024-04-26T00:20:08.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f07a15d4640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f079c135660 0x7f079c13ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:08.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f07a15d4640 10 monclient(hunting): picked mon.noname-b con 0x7f079c135660 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:08.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f07a15d4640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f079c0a1130 con 0x7f079c134b30 2024-04-26T00:20:08.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f07a15d4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:08.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f07a15d4640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f079c09ec10 con 0x7f079c0a0260 2024-04-26T00:20:08.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f07a15d4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:08.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f07a15d4640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f079c09f0f0 con 0x7f079c135660 2024-04-26T00:20:08.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f07a15d4640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:08.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f07a15d4640 10 monclient(hunting): authenticate will time out at 989.949651s 2024-04-26T00:20:08.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7fdf0e59c640 10 monclient(hunting): handle_auth_done global_id 4322 payload 274 2024-04-26T00:20:08.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7fdf0e59c640 1 -- 172.21.15.37:0/3551989839 >> v1:172.21.15.45:6789/0 conn(0x7fdf1013c630 legacy=0x7fdf1013ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:08.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f079affd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f079c0a0260 0x7f079c0a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:08.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f079b7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f079c135660 0x7f079c13ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:08.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f079affd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f079c0a0260 0x7f079c0a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44940/0 (socket says 172.21.15.37:44940) 2024-04-26T00:20:08.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f079affd640 1 -- 172.21.15.37:0/2836205680 learned_addr learned my addr 172.21.15.37:0/2836205680 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:08.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f079affd640 10 monclient(hunting): get_auth_request con 0x7f079c0a0260 auth_method 0 2024-04-26T00:20:08.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f079affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:08.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f079affd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:08.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f079affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:08.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f079b7fe640 10 monclient(hunting): get_auth_request con 0x7f079c135660 auth_method 0 2024-04-26T00:20:08.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f079b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:08.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f079b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:08.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f079b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:08.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7fdf0e59c640 1 -- 172.21.15.37:0/3551989839 >> v1:172.21.15.37:6789/0 conn(0x7fdf100726b0 legacy=0x7fdf10072a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:08.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7fdf0e59c640 10 monclient: _finish_hunting 0 2024-04-26T00:20:08.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7fdf0e59c640 1 monclient: found mon.noname-b 2024-04-26T00:20:08.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f079affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:08.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f079affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:08.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7fdf0e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:08.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7fdf0e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:20:08.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7fdf0e59c640 1 -- 172.21.15.37:0/3551989839 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf100a1190 con 0x7fdf100a0110 2024-04-26T00:20:08.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f079affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:08.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f079b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:08.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f079b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:08.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f079b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:08.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f079affd640 10 monclient(hunting): handle_auth_done global_id 4338 payload 274 2024-04-26T00:20:08.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f079affd640 1 -- 172.21.15.37:0/2836205680 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f079c135660 msgr2=0x7f079c13ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:08.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f079affd640 1 --2- 172.21.15.37:0/2836205680 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f079c135660 0x7f079c13ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:08.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f079affd640 1 -- 172.21.15.37:0/2836205680 >> v1:172.21.15.45:6789/0 conn(0x7f079c134b30 legacy=0x7f079c134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:08.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f079affd640 10 monclient: _finish_hunting 0 2024-04-26T00:20:08.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f079affd640 1 monclient: found mon.noname-a 2024-04-26T00:20:08.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f079affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:08.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f079affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:08.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f079affd640 1 -- 172.21.15.37:0/2836205680 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f079c06cff0 con 0x7f079c0a0260 2024-04-26T00:20:08.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.639+0000 7f079b7fe640 1 --2- 172.21.15.37:0/2836205680 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f079c135660 0x7f079c13ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:08.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7fdf0e59c640 1 -- 172.21.15.37:0/3551989839 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdf100a0110 msgr2=0x7fdf1013c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:08.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7fdf0e59c640 1 --2- 172.21.15.37:0/3551989839 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdf100a0110 0x7fdf1013c0f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdf040097f0 tx=0x7fdf0401c140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=430bcd696b8766b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:08.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7fdf0dd9b640 1 -- 172.21.15.37:0/3551989839 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf040b0070 con 0x7fdf100a0110 2024-04-26T00:20:08.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7fdf0dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:08.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7fdf0dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:20:08.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7fdf0dd9b640 10 monclient: dump: 2024-04-26T00:20:08.635 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:08.635 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:08.635 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:08.635 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:08.635 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:08.635 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:08.635 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:08.635 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:08.635 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:08.635 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:08.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7fdf0dd9b640 10 monclient: _finish_auth 0 2024-04-26T00:20:08.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7fdf0dd9b640 10 monclient: _check_auth_tickets 2024-04-26T00:20:08.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7fdf0dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:08.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7fdf0dd9b640 1 -- 172.21.15.37:0/3551989839 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf040ab600 con 0x7fdf100a0110 2024-04-26T00:20:08.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7fdf0dd9b640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:08.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7fdf0dd9b640 1 -- 172.21.15.37:0/3551989839 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf040ab9d0 con 0x7fdf100a0110 2024-04-26T00:20:08.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7fdf1524c640 5 monclient: authenticate success, global_id 4322 2024-04-26T00:20:08.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7fdf1524c640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:08.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7fdf1524c640 4 set_mon_vals no callback set 2024-04-26T00:20:08.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7fdf0dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:08.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7fdf0dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:08.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7fdf0dd9b640 10 monclient: dump: 2024-04-26T00:20:08.636 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:08.636 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:08.636 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:08.637 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:08.637 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:08.637 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:08.637 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:08.637 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:08.637 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:08.637 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:08.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7f079affd640 1 -- 172.21.15.37:0/2836205680 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f079c0a0260 msgr2=0x7f079c0a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:08.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7f079affd640 1 --2- 172.21.15.37:0/2836205680 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f079c0a0260 0x7f079c0a0640 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f078c009870 tx=0x7f078c01c200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e093db1f13e1ec75 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:08.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7f0799ffb640 1 -- 172.21.15.37:0/2836205680 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078c0b0070 con 0x7f079c0a0260 2024-04-26T00:20:08.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7f0799ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:08.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7f0799ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:08.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7f0799ffb640 10 monclient: dump: 2024-04-26T00:20:08.638 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:08.638 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:08.638 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:08.638 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:08.638 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:08.638 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:08.638 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:08.639 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:08.639 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:08.639 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:08.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7f0799ffb640 10 monclient: _finish_auth 0 2024-04-26T00:20:08.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7f0799ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:20:08.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7f0799ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:08.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7f0799ffb640 1 -- 172.21.15.37:0/2836205680 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f078c0ab400 con 0x7f079c0a0260 2024-04-26T00:20:08.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7f07a15d4640 5 monclient: authenticate success, global_id 4338 2024-04-26T00:20:08.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7f07a15d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:20:08.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7f0799ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:08.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7f07a15d4640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:08.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7f07a15d4640 4 set_mon_vals no callback set 2024-04-26T00:20:08.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7f0799ffb640 1 -- 172.21.15.37:0/2836205680 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078c0ab7d0 con 0x7f079c0a0260 2024-04-26T00:20:08.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7f0799ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:08.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7f0799ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:08.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7f0799ffb640 10 monclient: dump: 2024-04-26T00:20:08.640 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:08.640 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:08.640 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:08.640 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:08.640 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:08.640 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:08.640 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:08.640 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:08.640 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:08.640 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:08.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7fdf1524c640 10 monclient: shutdown 2024-04-26T00:20:08.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7fdf1524c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:08.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7fdf1524c640 1 -- 172.21.15.37:0/3551989839 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdf100a0110 msgr2=0x7fdf1013c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:08.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7fdf1524c640 1 --2- 172.21.15.37:0/3551989839 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdf100a0110 0x7fdf1013c0f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdf040097f0 tx=0x7fdf0401c140 comp rx=0 tx=0).stop 2024-04-26T00:20:08.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7fdf1524c640 1 -- 172.21.15.37:0/3551989839 shutdown_connections 2024-04-26T00:20:08.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7fdf1524c640 1 --2- 172.21.15.37:0/3551989839 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdf100a0110 0x7fdf1013c0f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:08.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7fdf1524c640 1 -- 172.21.15.37:0/3551989839 >> 172.21.15.37:0/3551989839 conn(0x7fdf100a1d50 msgr2=0x7fdf100a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:08.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7f07a15d4640 10 monclient: shutdown 2024-04-26T00:20:08.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7f07a15d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:08.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7f07a15d4640 1 -- 172.21.15.37:0/2836205680 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f079c0a0260 msgr2=0x7f079c0a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:08.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7f07a15d4640 1 --2- 172.21.15.37:0/2836205680 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f079c0a0260 0x7f079c0a0640 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f078c009870 tx=0x7f078c01c200 comp rx=0 tx=0).stop 2024-04-26T00:20:08.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7fdf1524c640 1 -- 172.21.15.37:0/3551989839 shutdown_connections 2024-04-26T00:20:08.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7fdf1524c640 1 -- 172.21.15.37:0/3551989839 wait complete. 2024-04-26T00:20:08.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7f07a15d4640 1 -- 172.21.15.37:0/2836205680 shutdown_connections 2024-04-26T00:20:08.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7f07a15d4640 1 --2- 172.21.15.37:0/2836205680 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f079c135660 0x7f079c13ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:08.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7f07a15d4640 1 --2- 172.21.15.37:0/2836205680 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f079c0a0260 0x7f079c0a0640 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:08.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7f07a15d4640 1 -- 172.21.15.37:0/2836205680 >> 172.21.15.37:0/2836205680 conn(0x7f079c0a1ad0 msgr2=0x7f079c0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:08.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7f07a15d4640 1 -- 172.21.15.37:0/2836205680 shutdown_connections 2024-04-26T00:20:08.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7f07a15d4640 1 -- 172.21.15.37:0/2836205680 wait complete. 2024-04-26T00:20:08.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7fdf1524c640 10 monclient: build_initial_monmap 2024-04-26T00:20:08.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7fdf1524c640 10 monclient: monmap: 2024-04-26T00:20:08.644 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:08.644 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:08.644 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:08.644 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:08.644 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:08.644 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:08.644 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:08.644 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:08.644 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:08.644 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:08.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7f07a15d4640 10 monclient: build_initial_monmap 2024-04-26T00:20:08.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.643+0000 7f07a15d4640 10 monclient: monmap: 2024-04-26T00:20:08.645 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:08.645 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:08.645 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:08.645 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:08.645 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:08.645 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:08.645 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:08.645 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:08.645 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:08.645 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:08.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f07a15d4640 1 Processor -- start 2024-04-26T00:20:08.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf1524c640 1 Processor -- start 2024-04-26T00:20:08.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f07a15d4640 1 -- start start 2024-04-26T00:20:08.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f07a15d4640 10 monclient: init 2024-04-26T00:20:08.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf1524c640 1 -- start start 2024-04-26T00:20:08.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf1524c640 10 monclient: init 2024-04-26T00:20:08.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f07a15d4640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:08.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f07a15d4640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:20:08.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f07a15d4640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f079c0a0260 0x7f079c1cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:08.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f07a15d4640 10 monclient(hunting): picked mon.noname-a con 0x7f079c0a0260 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:08.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f07a15d4640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f079c134b30 0x7f079c071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:08.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f07a15d4640 10 monclient(hunting): picked mon.noname-c con 0x7f079c134b30 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:08.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f07a15d4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f079c135660 0x7f079c0716b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:08.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f07a15d4640 10 monclient(hunting): picked mon.noname-b con 0x7f079c135660 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:08.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f07a15d4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:08.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f07a15d4640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f079c0a0010 con 0x7f079c0a0260 2024-04-26T00:20:08.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f07a15d4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:08.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f07a15d4640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f079c09fe90 con 0x7f079c135660 2024-04-26T00:20:08.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f07a15d4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:08.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f07a15d4640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f079c13dbc0 con 0x7f079c134b30 2024-04-26T00:20:08.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f07a15d4640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:08.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f07a15d4640 10 monclient(hunting): authenticate will time out at 989.956636s 2024-04-26T00:20:08.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf1524c640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:08.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf1524c640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:20:08.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf1524c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdf100726b0 0x7fdf101cf100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:08.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf1524c640 10 monclient(hunting): picked mon.noname-c con 0x7fdf100726b0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:08.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf1524c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdf100a0110 0x7fdf10135b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:08.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f079a7fc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f079c134b30 0x7f079c071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:08.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f079affd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f079c0a0260 0x7f079c1cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:08.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf0ed9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdf100726b0 0x7fdf101cf100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:08.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f079a7fc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f079c134b30 0x7f079c071170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:58118/0 (socket says 172.21.15.37:58118) 2024-04-26T00:20:08.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f079a7fc640 1 -- 172.21.15.37:0/436916017 learned_addr learned my addr 172.21.15.37:0/436916017 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:08.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f079a7fc640 10 monclient(hunting): get_auth_request con 0x7f079c134b30 auth_method 0 2024-04-26T00:20:08.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f079a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:08.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f079a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:08.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f079a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:08.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f079affd640 10 monclient(hunting): get_auth_request con 0x7f079c0a0260 auth_method 0 2024-04-26T00:20:08.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f079affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:08.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f079affd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:08.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f079affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:08.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf1524c640 10 monclient(hunting): picked mon.noname-a con 0x7fdf100a0110 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:08.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf0ed9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdf100726b0 0x7fdf101cf100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:58126/0 (socket says 172.21.15.37:58126) 2024-04-26T00:20:08.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf1524c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf1013c630 0x7fdf1011b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:08.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf1524c640 10 monclient(hunting): picked mon.noname-b con 0x7fdf1013c630 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:08.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf1524c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:08.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf1524c640 1 -- 172.21.15.37:0/852549648 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fdf10142cf0 con 0x7fdf100a0110 2024-04-26T00:20:08.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf0ed9d640 1 -- 172.21.15.37:0/852549648 learned_addr learned my addr 172.21.15.37:0/852549648 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:08.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf1524c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:08.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf1524c640 1 -- 172.21.15.37:0/852549648 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdf10142b70 con 0x7fdf1013c630 2024-04-26T00:20:08.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f079affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:08.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf1524c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:08.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf1524c640 1 -- 172.21.15.37:0/852549648 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fdf10142e70 con 0x7fdf100726b0 2024-04-26T00:20:08.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf1524c640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:08.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf1524c640 10 monclient(hunting): authenticate will time out at 989.956906s 2024-04-26T00:20:08.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f079affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:08.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f079affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:08.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf0e59c640 1 --2- 172.21.15.37:0/852549648 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdf100a0110 0x7fdf10135b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:08.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf0e59c640 10 monclient(hunting): get_auth_request con 0x7fdf100a0110 auth_method 0 2024-04-26T00:20:08.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:08.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf0e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:08.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f079a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:08.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f079a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:08.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f079a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:08.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf0e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:08.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf0ed9d640 10 monclient(hunting): get_auth_request con 0x7fdf100726b0 auth_method 0 2024-04-26T00:20:08.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf0ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:08.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf0ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:08.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf0ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:08.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f079affd640 10 monclient(hunting): handle_auth_done global_id 4341 payload 824 2024-04-26T00:20:08.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f079affd640 1 -- 172.21.15.37:0/436916017 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f079c134b30 msgr2=0x7f079c071170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:08.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f079affd640 1 --2- 172.21.15.37:0/436916017 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f079c134b30 0x7f079c071170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:08.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f079b7fe640 1 --2- 172.21.15.37:0/436916017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f079c135660 0x7f079c0716b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:08.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f079affd640 1 -- 172.21.15.37:0/436916017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f079c135660 msgr2=0x7f079c0716b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:08.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f079affd640 1 --2- 172.21.15.37:0/436916017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f079c135660 0x7f079c0716b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:08.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f079affd640 10 monclient: _finish_hunting 0 2024-04-26T00:20:08.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f079affd640 1 monclient: found mon.noname-a 2024-04-26T00:20:08.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f079affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:08.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f079affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:08.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f079affd640 1 -- 172.21.15.37:0/436916017 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f079c06f960 con 0x7f079c0a0260 2024-04-26T00:20:08.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f079b7fe640 10 monclient: get_auth_request con 0x7f079c135660 auth_method 0 2024-04-26T00:20:08.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f079b7fe640 1 --2- 172.21.15.37:0/436916017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f079c135660 0x7f079c0716b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:20:08.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf0e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:08.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf0e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:08.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf0f59e640 1 --2- 172.21.15.37:0/852549648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf1013c630 0x7fdf1011b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:08.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf0e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:08.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf0f59e640 10 monclient(hunting): get_auth_request con 0x7fdf1013c630 auth_method 0 2024-04-26T00:20:08.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf0f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:08.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf0f59e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:08.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf0f59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:08.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf0ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:08.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf0ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:08.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf0ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:08.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf0e59c640 10 monclient(hunting): handle_auth_done global_id 4344 payload 824 2024-04-26T00:20:08.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf0e59c640 1 -- 172.21.15.37:0/852549648 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdf100726b0 msgr2=0x7fdf101cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:08.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf0e59c640 1 --2- 172.21.15.37:0/852549648 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdf100726b0 0x7fdf101cf100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:08.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf0e59c640 1 -- 172.21.15.37:0/852549648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf1013c630 msgr2=0x7fdf1011b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:08.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf0e59c640 1 --2- 172.21.15.37:0/852549648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf1013c630 0x7fdf1011b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:08.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf0e59c640 10 monclient: _finish_hunting 0 2024-04-26T00:20:08.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf0e59c640 1 monclient: found mon.noname-a 2024-04-26T00:20:08.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf0e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:08.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf0e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:08.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf0e59c640 1 -- 172.21.15.37:0/852549648 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf1011f480 con 0x7fdf100a0110 2024-04-26T00:20:08.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf0f59e640 1 --2- 172.21.15.37:0/852549648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf1013c630 0x7fdf1011b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:20:08.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf0ed9d640 1 --2- 172.21.15.37:0/852549648 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdf100726b0 0x7fdf101cf100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:08.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f079affd640 1 -- 172.21.15.37:0/436916017 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f079c0a0260 msgr2=0x7f079c1cae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:08.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f079affd640 1 --2- 172.21.15.37:0/436916017 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f079c0a0260 0x7f079c1cae10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f078c0ab000 tx=0x7f078c018fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:08.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f07837fe640 1 -- 172.21.15.37:0/436916017 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078c0b0040 con 0x7f079c0a0260 2024-04-26T00:20:08.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f07837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:08.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f07837fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:08.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f07837fe640 10 monclient: dump: 2024-04-26T00:20:08.660 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:08.660 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:08.660 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:08.660 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:08.660 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:08.660 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:08.660 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:08.660 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:08.661 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:08.661 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:08.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f07837fe640 10 monclient: _finish_auth 0 2024-04-26T00:20:08.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f07837fe640 10 monclient: _check_auth_tickets 2024-04-26T00:20:08.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f07837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:08.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f07837fe640 1 -- 172.21.15.37:0/436916017 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f078c0abcb0 con 0x7f079c0a0260 2024-04-26T00:20:08.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f07a15d4640 5 monclient: authenticate success, global_id 4341 2024-04-26T00:20:08.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f07a15d4640 10 monclient: _renew_subs 2024-04-26T00:20:08.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f07a15d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:08.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f07a15d4640 1 -- 172.21.15.37:0/436916017 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f079c06e440 con 0x7f079c0a0260 2024-04-26T00:20:08.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f07837fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:08.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f07837fe640 1 -- 172.21.15.37:0/436916017 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078c0abe60 con 0x7f079c0a0260 2024-04-26T00:20:08.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f07837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:08.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f07837fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:08.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f07837fe640 10 monclient: dump: 2024-04-26T00:20:08.662 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:08.662 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:08.662 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:08.662 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:08.662 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:08.662 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:08.662 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:08.662 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:08.662 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:08.662 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:08.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f07a15d4640 10 monclient: _renew_subs 2024-04-26T00:20:08.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f07a15d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:08.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf0e59c640 1 -- 172.21.15.37:0/852549648 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdf100a0110 msgr2=0x7fdf10135b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:08.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f07a15d4640 1 -- 172.21.15.37:0/436916017 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f079c071bf0 con 0x7f079c0a0260 2024-04-26T00:20:08.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7f07997fa640 4 set_mon_vals no callback set 2024-04-26T00:20:08.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.647+0000 7fdf0e59c640 1 --2- 172.21.15.37:0/852549648 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdf100a0110 0x7fdf10135b00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdf040ad000 tx=0x7fdf04018ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:08.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.651+0000 7fdef77fe640 1 -- 172.21.15.37:0/852549648 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf040b9070 con 0x7fdf100a0110 2024-04-26T00:20:08.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.651+0000 7fdef77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:08.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.651+0000 7fdef77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:08.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.651+0000 7fdef77fe640 10 monclient: dump: 2024-04-26T00:20:08.664 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:08.664 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:08.664 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:08.664 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:08.664 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:08.664 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:08.664 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:08.665 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:08.665 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:08.665 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:08.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.651+0000 7fdef77fe640 10 monclient: _finish_auth 0 2024-04-26T00:20:08.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.651+0000 7fdef77fe640 10 monclient: _check_auth_tickets 2024-04-26T00:20:08.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.651+0000 7fdef77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:08.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.651+0000 7fdf1524c640 5 monclient: authenticate success, global_id 4344 2024-04-26T00:20:08.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.651+0000 7fdef77fe640 1 -- 172.21.15.37:0/852549648 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf040b5890 con 0x7fdf100a0110 2024-04-26T00:20:08.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.651+0000 7fdef77fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:08.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.651+0000 7fdef77fe640 1 -- 172.21.15.37:0/852549648 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf040b0040 con 0x7fdf100a0110 2024-04-26T00:20:08.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.651+0000 7fdef77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:08.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.651+0000 7fdef77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:08.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.651+0000 7fdef77fe640 10 monclient: dump: 2024-04-26T00:20:08.665 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:08.665 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:08.666 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:08.666 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:08.666 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:08.666 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:08.666 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:08.666 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:08.666 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:08.666 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:08.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.651+0000 7fdf1524c640 10 monclient: _renew_subs 2024-04-26T00:20:08.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.651+0000 7fdf0d59a640 4 set_mon_vals no callback set 2024-04-26T00:20:08.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.651+0000 7fdf1524c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:08.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.651+0000 7fdf1524c640 1 -- 172.21.15.37:0/852549648 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf10136a20 con 0x7fdf100a0110 2024-04-26T00:20:08.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.651+0000 7f07a15d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:08.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.651+0000 7f07a15d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:08.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.651+0000 7f07a15d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:08.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.651+0000 7f07a15d4640 1 -- 172.21.15.37:0/436916017 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f079c066f40 con 0x7f079c0a0260 2024-04-26T00:20:08.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.651+0000 7fdf1524c640 10 monclient: _renew_subs 2024-04-26T00:20:08.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.651+0000 7fdf1524c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:08.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.651+0000 7fdf1524c640 1 -- 172.21.15.37:0/852549648 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf1011d120 con 0x7fdf100a0110 2024-04-26T00:20:08.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.651+0000 7f07837fe640 1 -- 172.21.15.37:0/436916017 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f078c0bd3f0 con 0x7f079c0a0260 2024-04-26T00:20:08.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.651+0000 7f07837fe640 1 --2- 172.21.15.37:0/436916017 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0788035e80 0x7f0788038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:08.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.651+0000 7f07837fe640 1 -- 172.21.15.37:0/436916017 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7f078c0e1700 con 0x7f079c0a0260 2024-04-26T00:20:08.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.651+0000 7fdf1524c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:08.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.651+0000 7fdf1524c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:08.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.651+0000 7fdf1524c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:08.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.651+0000 7fdf1524c640 1 -- 172.21.15.37:0/852549648 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf1004f510 con 0x7fdf100a0110 2024-04-26T00:20:08.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.651+0000 7f079a7fc640 1 --2- 172.21.15.37:0/436916017 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0788035e80 0x7f0788038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:08.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.651+0000 7f079a7fc640 10 monclient: get_auth_request con 0x7f0788035e80 auth_method 0 2024-04-26T00:20:08.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.651+0000 7fdef77fe640 1 -- 172.21.15.37:0/852549648 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fdf040bc070 con 0x7fdf100a0110 2024-04-26T00:20:08.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.651+0000 7fdef77fe640 1 --2- 172.21.15.37:0/852549648 >> v2:172.21.15.45:6812/2379918623 conn(0x7fdefc035e80 0x7fdefc038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:08.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.651+0000 7fdf0ed9d640 1 --2- 172.21.15.37:0/852549648 >> v2:172.21.15.45:6812/2379918623 conn(0x7fdefc035e80 0x7fdefc038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:08.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.651+0000 7fdf0ed9d640 10 monclient: get_auth_request con 0x7fdefc035e80 auth_method 0 2024-04-26T00:20:08.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.655+0000 7f079a7fc640 1 --2- 172.21.15.37:0/436916017 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0788035e80 0x7f0788038340 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f079c1cbc70 tx=0x7f0790003700 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:08.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.655+0000 7fdf0ed9d640 1 --2- 172.21.15.37:0/852549648 >> v2:172.21.15.45:6812/2379918623 conn(0x7fdefc035e80 0x7fdefc038340 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdf00009eb0 tx=0x7fdf00009970 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:08.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.655+0000 7fdef77fe640 1 -- 172.21.15.37:0/852549648 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7fdf040aa560 con 0x7fdf100a0110 2024-04-26T00:20:08.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.655+0000 7f07837fe640 1 -- 172.21.15.37:0/436916017 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f079c0a1130 con 0x7f079c0a0260 2024-04-26T00:20:08.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.655+0000 7f07837fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:08.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.655+0000 7f07837fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:08.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.659+0000 7fdef77fe640 1 -- 172.21.15.37:0/852549648 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fdf100a1470 con 0x7fdf100a0110 2024-04-26T00:20:08.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.659+0000 7fdef77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:08.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.659+0000 7fdef77fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:08.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.887+0000 7fdf1524c640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}] 2024-04-26T00:20:08.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.887+0000 7fdf1524c640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}] 2024-04-26T00:20:08.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.887+0000 7fdf1524c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:08.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.887+0000 7fdf1524c640 1 -- 172.21.15.37:0/852549648 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} v 0) -- 0x7fdf1006ba50 con 0x7fdf100a0110 2024-04-26T00:20:08.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.931+0000 7f07a15d4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:20:08.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.931+0000 7f07a15d4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:20:08.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.931+0000 7f07a15d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:08.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.931+0000 7f07a15d4640 1 -- 172.21.15.37:0/436916017 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f079c09b560 con 0x7f079c0a0260 2024-04-26T00:20:08.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.935+0000 7f07837fe640 1 -- 172.21.15.37:0/436916017 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f078c0bddf0 con 0x7f079c0a0260 2024-04-26T00:20:08.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.935+0000 7f07837fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:20:08.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.935+0000 7f07837fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:08.916 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":18,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:20:08.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.935+0000 7f07a15d4640 1 -- 172.21.15.37:0/436916017 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0788035e80 msgr2=0x7f0788038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:08.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.935+0000 7f07a15d4640 1 --2- 172.21.15.37:0/436916017 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0788035e80 0x7f0788038340 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f079c1cbc70 tx=0x7f0790003700 comp rx=0 tx=0).stop 2024-04-26T00:20:08.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.935+0000 7f07a15d4640 10 monclient: shutdown 2024-04-26T00:20:08.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.935+0000 7f07a15d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:08.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.935+0000 7f07a15d4640 1 -- 172.21.15.37:0/436916017 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f079c0a0260 msgr2=0x7f079c1cae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:08.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.935+0000 7f07a15d4640 1 --2- 172.21.15.37:0/436916017 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f079c0a0260 0x7f079c1cae10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f078c0ab000 tx=0x7f078c018fa0 comp rx=0 tx=0).stop 2024-04-26T00:20:08.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.935+0000 7f07a15d4640 1 -- 172.21.15.37:0/436916017 shutdown_connections 2024-04-26T00:20:08.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.935+0000 7f07a15d4640 1 --2- 172.21.15.37:0/436916017 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0788035e80 0x7f0788038340 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:08.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.935+0000 7f07a15d4640 1 --2- 172.21.15.37:0/436916017 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f079c135660 0x7f079c0716b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:08.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.935+0000 7f07a15d4640 1 --2- 172.21.15.37:0/436916017 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f079c134b30 0x7f079c071170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:08.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.935+0000 7f07a15d4640 1 --2- 172.21.15.37:0/436916017 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f079c0a0260 0x7f079c1cae10 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:08.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.935+0000 7f07a15d4640 1 -- 172.21.15.37:0/436916017 >> 172.21.15.37:0/436916017 conn(0x7f079c0a1ad0 msgr2=0x7f079c0a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:08.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.935+0000 7f07a15d4640 1 -- 172.21.15.37:0/436916017 shutdown_connections 2024-04-26T00:20:08.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:08.935+0000 7f07a15d4640 1 -- 172.21.15.37:0/436916017 wait complete. 2024-04-26T00:20:08.933 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:20:08.933 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-26T00:20:08.933 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:20:08.933 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:20:09.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.063+0000 7fa704005640 10 monclient: get_monmap_and_config 2024-04-26T00:20:09.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.063+0000 7fa704005640 10 monclient: build_initial_monmap 2024-04-26T00:20:09.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.063+0000 7fa704005640 10 monclient: monmap: 2024-04-26T00:20:09.047 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:09.047 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:09.047 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:09.070866+0000 2024-04-26T00:20:09.047 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:09.070866+0000 2024-04-26T00:20:09.047 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:09.047 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:09.047 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:09.047 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:09.047 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:09.048 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:09.048 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:09.048 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:09.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.063+0000 7fa704005640 1 Processor -- start 2024-04-26T00:20:09.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.063+0000 7fa704005640 1 -- start start 2024-04-26T00:20:09.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.063+0000 7fa704005640 10 monclient: init 2024-04-26T00:20:09.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.067+0000 7fa704005640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:09.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.067+0000 7fa704005640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-04-26T00:20:09.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.067+0000 7fa704005640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa6fc0a3d90 addr v1:172.21.15.37:6789/0 2024-04-26T00:20:09.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.067+0000 7fa704005640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa6fc0a4170 addr v1:172.21.15.45:6789/0 2024-04-26T00:20:09.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.067+0000 7fa704005640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa6fc13d3f0 0x7fa6fc13f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:09.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.067+0000 7fa704005640 10 monclient(hunting): picked mon.noname-b con 0x7fa6fc13d3f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:09.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.067+0000 7fa704005640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6fc0a5150 con 0x7fa6fc0a3d90 2024-04-26T00:20:09.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.067+0000 7fa704005640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6fc09b280 con 0x7fa6fc0a4170 2024-04-26T00:20:09.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.067+0000 7fa704005640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:09.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.067+0000 7fa704005640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa6fc09c650 con 0x7fa6fc13d3f0 2024-04-26T00:20:09.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.067+0000 7fa704005640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:09.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.067+0000 7fa701da1640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fa6fc0a3d90 0x7fa6fc09bf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:55872/0 (socket says 172.21.15.37:55872) 2024-04-26T00:20:09.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.067+0000 7fa701da1640 1 -- 172.21.15.37:0/2743255141 learned_addr learned my addr 172.21.15.37:0/2743255141 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:20:09.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.067+0000 7fa7025a2640 1 --2- 172.21.15.37:0/2743255141 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa6fc13d3f0 0x7fa6fc13f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:09.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.067+0000 7fa704005640 10 monclient(hunting): authenticate will time out at 990.376754s 2024-04-26T00:20:09.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.067+0000 7fa7025a2640 10 monclient(hunting): get_auth_request con 0x7fa6fc13d3f0 auth_method 0 2024-04-26T00:20:09.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.067+0000 7fa7025a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:09.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.067+0000 7fa7025a2640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:09.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.067+0000 7fa7025a2640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:09.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.067+0000 7fa7025a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:09.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.067+0000 7fa7025a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:09.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.067+0000 7fa7025a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:09.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.067+0000 7fa7025a2640 10 monclient(hunting): handle_auth_done global_id 4334 payload 274 2024-04-26T00:20:09.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.067+0000 7fa7025a2640 1 -- 172.21.15.37:0/2743255141 >> v1:172.21.15.45:6789/0 conn(0x7fa6fc0a4170 legacy=0x7fa6fc0a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:09.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.067+0000 7fa7025a2640 1 -- 172.21.15.37:0/2743255141 >> v1:172.21.15.37:6789/0 conn(0x7fa6fc0a3d90 legacy=0x7fa6fc09bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:09.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.067+0000 7fa7025a2640 10 monclient: _finish_hunting 0 2024-04-26T00:20:09.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.067+0000 7fa7025a2640 1 monclient: found mon.noname-b 2024-04-26T00:20:09.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.067+0000 7fa7025a2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:09.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.067+0000 7fa7025a2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:20:09.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.067+0000 7fa7025a2640 1 -- 172.21.15.37:0/2743255141 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6fc09b560 con 0x7fa6fc13d3f0 2024-04-26T00:20:09.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa7025a2640 1 -- 172.21.15.37:0/2743255141 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa6fc13d3f0 msgr2=0x7fa6fc13f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:09.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa7025a2640 1 --2- 172.21.15.37:0/2743255141 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa6fc13d3f0 0x7fa6fc13f7e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa6f4009dc0 tx=0x7fa6f401c7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=126d4856bf9bc2b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:09.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa700d9f640 1 -- 172.21.15.37:0/2743255141 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f40b2070 con 0x7fa6fc13d3f0 2024-04-26T00:20:09.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa700d9f640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:09.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa700d9f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:20:09.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa700d9f640 10 monclient: dump: 2024-04-26T00:20:09.055 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:09.055 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:09.055 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.055 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.055 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:09.055 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:09.055 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:09.055 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:09.055 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:09.055 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:09.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa700d9f640 10 monclient: _finish_auth 0 2024-04-26T00:20:09.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa700d9f640 10 monclient: _check_auth_tickets 2024-04-26T00:20:09.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa700d9f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:09.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa700d9f640 1 -- 172.21.15.37:0/2743255141 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6f40a5840 con 0x7fa6fc13d3f0 2024-04-26T00:20:09.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa700d9f640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:09.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa700d9f640 1 -- 172.21.15.37:0/2743255141 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f40a5c10 con 0x7fa6fc13d3f0 2024-04-26T00:20:09.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa704005640 5 monclient: authenticate success, global_id 4334 2024-04-26T00:20:09.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa704005640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:09.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa704005640 4 set_mon_vals no callback set 2024-04-26T00:20:09.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa700d9f640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:09.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa700d9f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:09.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa700d9f640 10 monclient: dump: 2024-04-26T00:20:09.056 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:09.056 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:09.056 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.056 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.056 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:09.056 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:09.056 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:09.056 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:09.057 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:09.057 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:09.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa704005640 10 monclient: shutdown 2024-04-26T00:20:09.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa704005640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:09.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa704005640 1 -- 172.21.15.37:0/2743255141 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa6fc13d3f0 msgr2=0x7fa6fc13f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:09.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa704005640 1 --2- 172.21.15.37:0/2743255141 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa6fc13d3f0 0x7fa6fc13f7e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa6f4009dc0 tx=0x7fa6f401c7d0 comp rx=0 tx=0).stop 2024-04-26T00:20:09.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa704005640 1 -- 172.21.15.37:0/2743255141 shutdown_connections 2024-04-26T00:20:09.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa704005640 1 --2- 172.21.15.37:0/2743255141 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa6fc13d3f0 0x7fa6fc13f7e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:09.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa704005640 1 -- 172.21.15.37:0/2743255141 >> 172.21.15.37:0/2743255141 conn(0x7fa6fc0aa0a0 msgr2=0x7fa6fc0a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:09.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa704005640 1 -- 172.21.15.37:0/2743255141 shutdown_connections 2024-04-26T00:20:09.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa704005640 1 -- 172.21.15.37:0/2743255141 wait complete. 2024-04-26T00:20:09.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa704005640 10 monclient: build_initial_monmap 2024-04-26T00:20:09.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa704005640 10 monclient: monmap: 2024-04-26T00:20:09.057 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:09.057 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:09.058 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:09.058 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:09.058 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:09.058 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:09.058 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:09.058 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:09.058 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:09.058 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:09.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa704005640 1 Processor -- start 2024-04-26T00:20:09.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa704005640 1 -- start start 2024-04-26T00:20:09.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa704005640 10 monclient: init 2024-04-26T00:20:09.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa704005640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:09.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa704005640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:20:09.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa704005640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa6fc0a3d90 0x7fa6fc1cf190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:09.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa704005640 10 monclient(hunting): picked mon.noname-c con 0x7fa6fc0a3d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:09.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa704005640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6fc0a4170 0x7fa6fc136dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:09.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa704005640 10 monclient(hunting): picked mon.noname-a con 0x7fa6fc0a4170 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:09.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa704005640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6fc13d3f0 0x7fa6fc13a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:09.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa704005640 10 monclient(hunting): picked mon.noname-b con 0x7fa6fc13d3f0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:09.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa704005640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:09.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa704005640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa6fc142ef0 con 0x7fa6fc0a4170 2024-04-26T00:20:09.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa704005640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:09.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa704005640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa6fc142d70 con 0x7fa6fc13d3f0 2024-04-26T00:20:09.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa704005640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:09.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa704005640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa6fc143070 con 0x7fa6fc0a3d90 2024-04-26T00:20:09.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa704005640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:09.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa704005640 10 monclient(hunting): authenticate will time out at 990.383692s 2024-04-26T00:20:09.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa701da1640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa6fc0a3d90 0x7fa6fc1cf190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:09.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa701da1640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa6fc0a3d90 0x7fa6fc1cf190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:58144/0 (socket says 172.21.15.37:58144) 2024-04-26T00:20:09.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa701da1640 1 -- 172.21.15.37:0/3423290596 learned_addr learned my addr 172.21.15.37:0/3423290596 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:09.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa701da1640 10 monclient(hunting): get_auth_request con 0x7fa6fc0a3d90 auth_method 0 2024-04-26T00:20:09.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa701da1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:09.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa701da1640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:09.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa701da1640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:09.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa7015a0640 1 --2- 172.21.15.37:0/3423290596 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6fc0a4170 0x7fa6fc136dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:09.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa701da1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:09.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa701da1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:09.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa701da1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:09.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa7015a0640 10 monclient(hunting): get_auth_request con 0x7fa6fc0a4170 auth_method 0 2024-04-26T00:20:09.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa7015a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:09.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa7015a0640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:09.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.071+0000 7fa7015a0640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:09.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa7015a0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:09.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa7015a0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:09.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa7015a0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:09.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa701da1640 10 monclient(hunting): handle_auth_done global_id 4337 payload 824 2024-04-26T00:20:09.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa701da1640 1 -- 172.21.15.37:0/3423290596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6fc13d3f0 msgr2=0x7fa6fc13a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:09.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa701da1640 1 --2- 172.21.15.37:0/3423290596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6fc13d3f0 0x7fa6fc13a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:09.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa701da1640 1 -- 172.21.15.37:0/3423290596 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6fc0a4170 msgr2=0x7fa6fc136dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:09.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa701da1640 1 --2- 172.21.15.37:0/3423290596 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6fc0a4170 0x7fa6fc136dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:09.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa701da1640 10 monclient: _finish_hunting 0 2024-04-26T00:20:09.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa701da1640 1 monclient: found mon.noname-c 2024-04-26T00:20:09.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa701da1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:09.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa701da1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:20:09.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa701da1640 1 -- 172.21.15.37:0/3423290596 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6fc13d7d0 con 0x7fa6fc0a3d90 2024-04-26T00:20:09.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa701da1640 1 -- 172.21.15.37:0/3423290596 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa6fc0a3d90 msgr2=0x7fa6fc1cf190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:09.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa701da1640 1 --2- 172.21.15.37:0/3423290596 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa6fc0a3d90 0x7fa6fc1cf190 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa6f800c4a0 tx=0x7fa6f800c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:09.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa6f27fc640 1 -- 172.21.15.37:0/3423290596 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f8017070 con 0x7fa6fc0a3d90 2024-04-26T00:20:09.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa6f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:09.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa6f27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:20:09.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa6f27fc640 10 monclient: dump: 2024-04-26T00:20:09.068 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:09.068 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:09.068 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.068 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.068 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:09.068 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:09.068 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:09.069 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:09.069 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:09.069 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:09.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa6f27fc640 10 monclient: _finish_auth 0 2024-04-26T00:20:09.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa6f27fc640 10 monclient: _check_auth_tickets 2024-04-26T00:20:09.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa6f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:09.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa6f27fc640 1 -- 172.21.15.37:0/3423290596 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6f80070d0 con 0x7fa6fc0a3d90 2024-04-26T00:20:09.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa6f27fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:09.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa6f27fc640 1 -- 172.21.15.37:0/3423290596 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f80074c0 con 0x7fa6fc0a3d90 2024-04-26T00:20:09.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa704005640 5 monclient: authenticate success, global_id 4337 2024-04-26T00:20:09.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa704005640 10 monclient: _renew_subs 2024-04-26T00:20:09.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa704005640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:09.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa6f37fe640 4 set_mon_vals no callback set 2024-04-26T00:20:09.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa704005640 1 -- 172.21.15.37:0/3423290596 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6fc09ada0 con 0x7fa6fc0a3d90 2024-04-26T00:20:09.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa6f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:09.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa6f27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:09.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa6f27fc640 10 monclient: dump: 2024-04-26T00:20:09.070 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:09.070 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:09.070 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.070 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.070 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:09.070 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:09.070 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:09.070 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:09.070 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:09.070 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:09.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa704005640 10 monclient: _renew_subs 2024-04-26T00:20:09.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa704005640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:09.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa704005640 1 -- 172.21.15.37:0/3423290596 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6fc120740 con 0x7fa6fc0a3d90 2024-04-26T00:20:09.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa704005640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:09.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa704005640 1 --2- 172.21.15.37:0/3423290596 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6fc13a660 0x7fa6fc13aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:09.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa704005640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:09.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa704005640 1 -- 172.21.15.37:0/3423290596 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa6fc13af80 con 0x7fa6fc13a660 2024-04-26T00:20:09.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa7015a0640 1 --2- 172.21.15.37:0/3423290596 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6fc13a660 0x7fa6fc13aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:09.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa7015a0640 10 monclient: get_auth_request con 0x7fa6fc13a660 auth_method 0 2024-04-26T00:20:09.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa7015a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:09.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa7015a0640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:09.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa7015a0640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:09.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa7015a0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:09.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa7015a0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:09.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa6f27fc640 1 -- 172.21.15.37:0/3423290596 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa6f801a070 con 0x7fa6fc0a3d90 2024-04-26T00:20:09.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa7015a0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:09.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa7015a0640 10 monclient(hunting): handle_auth_done global_id 4350 payload 824 2024-04-26T00:20:09.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa7015a0640 1 -- 172.21.15.37:0/3423290596 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa6fc11f9e0 con 0x7fa6fc13a660 2024-04-26T00:20:09.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa6f27fc640 1 --2- 172.21.15.37:0/3423290596 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa6e0035ed0 0x7fa6e0038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:09.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.075+0000 7fa6f27fc640 1 -- 172.21.15.37:0/3423290596 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7fa6f8047df0 con 0x7fa6fc0a3d90 2024-04-26T00:20:09.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.079+0000 7fa7025a2640 1 --2- 172.21.15.37:0/3423290596 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa6e0035ed0 0x7fa6e0038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:09.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.079+0000 7fa7025a2640 10 monclient: get_auth_request con 0x7fa6e0035ed0 auth_method 0 2024-04-26T00:20:09.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.079+0000 7fa7015a0640 1 -- 172.21.15.37:0/3423290596 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6fc13a660 msgr2=0x7fa6fc13aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:09.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.079+0000 7fa7015a0640 1 --2- 172.21.15.37:0/3423290596 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6fc13a660 0x7fa6fc13aa40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc0a0070 tx=0x7fa6ec00fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:09.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.079+0000 7fa6f27fc640 1 -- 172.21.15.37:0/3423290596 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec003b80 con 0x7fa6fc13a660 2024-04-26T00:20:09.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.079+0000 7fa6f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:09.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.079+0000 7fa6f27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:09.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.079+0000 7fa6f27fc640 10 monclient: dump: 2024-04-26T00:20:09.072 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:09.072 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:09.072 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.073 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.073 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:09.073 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:09.073 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:09.073 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:09.073 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:09.073 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:09.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.079+0000 7fa6f27fc640 1 -- 172.21.15.37:0/3423290596 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa6ec019070 con 0x7fa6fc13a660 2024-04-26T00:20:09.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.079+0000 7fa6f27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:09.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.079+0000 7fa6f27fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:09.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.079+0000 7fa6f27fc640 1 -- 172.21.15.37:0/3423290596 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6fc13a660 msgr2=0x7fa6fc13aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:09.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.079+0000 7fa6f27fc640 1 --2- 172.21.15.37:0/3423290596 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6fc13a660 0x7fa6fc13aa40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc0a0070 tx=0x7fa6ec00fc00 comp rx=0 tx=0).stop 2024-04-26T00:20:09.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.079+0000 7fa6f27fc640 1 -- 172.21.15.37:0/3423290596 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6fc13a660 msgr2=0x7fa6fc13aa40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:20:09.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.079+0000 7fa6f27fc640 1 --2- 172.21.15.37:0/3423290596 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6fc13a660 0x7fa6fc13aa40 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc0a0070 tx=0x7fa6ec00fc00 comp rx=0 tx=0).stop 2024-04-26T00:20:09.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.079+0000 7fa7025a2640 1 --2- 172.21.15.37:0/3423290596 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa6e0035ed0 0x7fa6e0038390 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa6f40ac000 tx=0x7fa6f4004ac0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:09.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.087+0000 7fa704005640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:20:09.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.087+0000 7fa704005640 1 --2- 172.21.15.37:0/3423290596 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6fc09d070 0x7fa6fc09d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:09.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.087+0000 7fa704005640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:09.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.087+0000 7fa704005640 1 -- 172.21.15.37:0/3423290596 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa6fc04f510 con 0x7fa6fc09d070 2024-04-26T00:20:09.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.087+0000 7fa7015a0640 1 --2- 172.21.15.37:0/3423290596 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6fc09d070 0x7fa6fc09d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:09.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.087+0000 7fa7015a0640 10 monclient: get_auth_request con 0x7fa6fc09d070 auth_method 0 2024-04-26T00:20:09.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.087+0000 7fa7015a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:09.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.087+0000 7fa7015a0640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:09.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.087+0000 7fa7015a0640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:09.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.087+0000 7fa7015a0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:09.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.087+0000 7fa7015a0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:09.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.087+0000 7fa7015a0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:09.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.087+0000 7fa7015a0640 10 monclient(hunting): handle_auth_done global_id 4353 payload 824 2024-04-26T00:20:09.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.087+0000 7fa7015a0640 1 -- 172.21.15.37:0/3423290596 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa6fc13cfe0 con 0x7fa6fc09d070 2024-04-26T00:20:09.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.087+0000 7fa7015a0640 1 -- 172.21.15.37:0/3423290596 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6fc09d070 msgr2=0x7fa6fc09d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:09.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.087+0000 7fa7015a0640 1 --2- 172.21.15.37:0/3423290596 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6fc09d070 0x7fa6fc09d480 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc0a0070 tx=0x7fa6ec003770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:09.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.087+0000 7fa6f27fc640 1 -- 172.21.15.37:0/3423290596 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec006e00 con 0x7fa6fc09d070 2024-04-26T00:20:09.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.087+0000 7fa6f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:09.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.087+0000 7fa6f27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:09.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.087+0000 7fa6f27fc640 10 monclient: dump: 2024-04-26T00:20:09.077 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:09.077 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:09.077 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.077 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.077 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:09.077 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:09.077 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:09.077 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:09.077 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:09.077 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:09.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.087+0000 7fa6f27fc640 1 -- 172.21.15.37:0/3423290596 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4718 (secure 0 0 0) 0x7fa6ec007690 con 0x7fa6fc09d070 2024-04-26T00:20:09.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.087+0000 7fa6f27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:20:09.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.087+0000 7fa6f27fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:09.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.087+0000 7fa6f27fc640 1 -- 172.21.15.37:0/3423290596 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6fc09d070 msgr2=0x7fa6fc09d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:09.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.087+0000 7fa6f27fc640 1 --2- 172.21.15.37:0/3423290596 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6fc09d070 0x7fa6fc09d480 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc0a0070 tx=0x7fa6ec003770 comp rx=0 tx=0).stop 2024-04-26T00:20:09.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.087+0000 7fa6f27fc640 1 -- 172.21.15.37:0/3423290596 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6fc09d070 msgr2=0x7fa6fc09d480 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:20:09.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.087+0000 7fa6f27fc640 1 --2- 172.21.15.37:0/3423290596 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6fc09d070 0x7fa6fc09d480 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc0a0070 tx=0x7fa6ec003770 comp rx=0 tx=0).stop 2024-04-26T00:20:09.078 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:20:09.078 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:20:09.078 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:20:09.078 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:20:09.078 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 6, 2024-04-26T00:20:09.078 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 26191, 2024-04-26T00:20:09.078 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:20:09.079 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:20:09.079 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:20:09.079 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:20:09.079 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:09.079 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 19, 2024-04-26T00:20:09.079 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:20:09.079 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:20:09.079 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:20:09.079 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:20:09.079 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:20:09.079 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:20:09.079 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:20:09.079 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:20:09.079 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:20:09.080 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:20:09.080 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:20:09.080 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:20:09.080 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:20:09.080 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:20:09.080 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:09.080 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:20:09.080 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:20:09.080 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:20:09.080 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:20:09.080 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:20:09.080 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:20:09.080 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:20:09.080 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:20:09.080 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:20:09.081 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:20:09.081 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:20:09.081 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:20:09.081 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:20:09.081 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:09.081 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:09.081 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:20:09.081 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:20:09.081 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:20:09.081 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:20:09.081 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:20:09.081 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:20:09.081 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:20:09.081 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:20:09.081 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:20:09.082 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:20:09.082 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:20:09.082 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:20:09.082 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:20:09.082 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:20:09.082 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:20:09.082 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:20:09.082 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:20:09.082 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:20:09.082 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:20:09.082 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:20:09.082 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:20:09.082 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:20:09.082 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:20:09.082 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:20:09.083 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:20:09.083 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:20:09.083 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:20:09.083 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:20:09.083 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:09.083 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:20:09.083 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:09.083 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:20:09.083 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:09.083 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:20:09.083 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:20:09.083 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:20:09.083 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:20:09.083 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:09.083 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:20:09.084 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:20:09.084 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:09.084 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:09.084 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:09.084 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:20:09.084 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:20:09.084 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:09.084 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:09.084 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:09.084 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:09.084 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:20:09.084 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:20:09.084 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:20:09.084 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:20:09.085 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:20:09.085 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:09.085 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:09.085 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:20:09.085 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:20:09.085 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:20:09.085 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:20:09.085 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:09.085 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:20:09.085 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:20:09.085 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:09.085 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:09.085 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:09.085 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:20:09.085 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:20:09.086 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:09.086 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:09.086 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:09.086 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:09.086 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:20:09.086 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:20:09.086 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:20:09.086 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:20:09.086 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:20:09.086 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:09.086 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:09.086 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:20:09.086 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:20:09.086 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:20:09.087 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:20:09.087 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:09.087 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:20:09.087 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:20:09.087 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:09.087 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:09.087 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:09.087 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:20:09.087 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:20:09.087 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:09.087 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:09.087 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:09.087 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:09.087 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:20:09.087 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:20:09.088 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:20:09.088 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:20:09.088 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:20:09.088 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:09.088 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:09.088 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:09.088 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:20:09.088 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:20:09.088 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:09.088 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:20:09.088 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:20:09.088 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:20:09.088 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:09.088 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:09.088 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:20:09.089 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:09.089 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:20:09.089 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:20:09.089 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:20:09.089 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:09.089 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:09.089 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:20:09.089 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:09.089 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:20:09.089 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:20:09.089 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:20:09.089 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:09.089 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:09.089 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:09.089 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:20:09.090 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:20:09.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.087+0000 7fa704005640 1 -- 172.21.15.37:0/3423290596 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa6e0035ed0 msgr2=0x7fa6e0038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:09.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.087+0000 7fa704005640 1 --2- 172.21.15.37:0/3423290596 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa6e0035ed0 0x7fa6e0038390 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa6f40ac000 tx=0x7fa6f4004ac0 comp rx=0 tx=0).stop 2024-04-26T00:20:09.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.087+0000 7fa704005640 10 monclient: shutdown 2024-04-26T00:20:09.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.087+0000 7fa704005640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:09.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.087+0000 7fa704005640 1 -- 172.21.15.37:0/3423290596 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa6fc0a3d90 msgr2=0x7fa6fc1cf190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:09.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.087+0000 7fa704005640 1 --2- 172.21.15.37:0/3423290596 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa6fc0a3d90 0x7fa6fc1cf190 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa6f800c4a0 tx=0x7fa6f800c960 comp rx=0 tx=0).stop 2024-04-26T00:20:09.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.087+0000 7fa7025a2640 1 -- 172.21.15.37:0/3423290596 reap_dead start 2024-04-26T00:20:09.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.087+0000 7fa7025a2640 1 -- 172.21.15.37:0/3423290596 reap_dead start 2024-04-26T00:20:09.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.087+0000 7fa704005640 1 -- 172.21.15.37:0/3423290596 shutdown_connections 2024-04-26T00:20:09.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.087+0000 7fa704005640 1 -- 172.21.15.37:0/3423290596 >> 172.21.15.37:0/3423290596 conn(0x7fa6fc0aa0a0 msgr2=0x7fa6fc13de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:09.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.087+0000 7fa704005640 1 -- 172.21.15.37:0/3423290596 shutdown_connections 2024-04-26T00:20:09.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.087+0000 7fa704005640 1 -- 172.21.15.37:0/3423290596 wait complete. 2024-04-26T00:20:09.092 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:20:09.092 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:20:09.092 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:20:09.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.219+0000 7f7b7c094640 10 monclient: get_monmap_and_config 2024-04-26T00:20:09.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.219+0000 7f7b7c094640 10 monclient: build_initial_monmap 2024-04-26T00:20:09.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.219+0000 7f7b7c094640 10 monclient: monmap: 2024-04-26T00:20:09.203 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:09.203 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:09.203 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:09.226954+0000 2024-04-26T00:20:09.203 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:09.226954+0000 2024-04-26T00:20:09.203 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:09.203 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:09.203 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:09.204 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:09.204 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:09.204 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:09.204 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:09.204 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:09.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.219+0000 7f7b7c094640 1 Processor -- start 2024-04-26T00:20:09.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.219+0000 7f7b7c094640 1 -- start start 2024-04-26T00:20:09.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.219+0000 7f7b7c094640 10 monclient: init 2024-04-26T00:20:09.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7c094640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:09.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7c094640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-04-26T00:20:09.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7c094640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7b74136f20 0x7f7b7409bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:09.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7c094640 10 monclient(hunting): picked mon.noname-c con 0x7f7b74136f20 addr v2:172.21.15.45:3300/0 2024-04-26T00:20:09.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7c094640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f7b74137300 0x7f7b741377b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:09.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7c094640 10 monclient(hunting): picked mon.noname-a con 0x7f7b74137300 addr v2:172.21.15.37:3300/0 2024-04-26T00:20:09.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7c094640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7b74133c70 0x7f7b74134120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:09.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7c094640 10 monclient(hunting): picked mon.noname-b con 0x7f7b74133c70 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:09.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7c094640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:09.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7c094640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f7b740a50f0 con 0x7f7b74137300 2024-04-26T00:20:09.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7c094640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:09.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7c094640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f7b7409b330 con 0x7f7b74136f20 2024-04-26T00:20:09.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7c094640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:09.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7c094640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f7b7409b610 con 0x7f7b74133c70 2024-04-26T00:20:09.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7c094640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:09.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7c094640 10 monclient(hunting): authenticate will time out at 990.532656s 2024-04-26T00:20:09.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7a631640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7b74133c70 0x7f7b74134120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:09.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7962f640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f7b74137300 0x7f7b741377b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:09.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7a631640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7b74133c70 0x7f7b74134120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:58154/0 (socket says 172.21.15.37:58154) 2024-04-26T00:20:09.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7a631640 1 -- 172.21.15.37:0/3690566732 learned_addr learned my addr 172.21.15.37:0/3690566732 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:09.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7a631640 10 monclient(hunting): get_auth_request con 0x7f7b74133c70 auth_method 0 2024-04-26T00:20:09.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7a631640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:09.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7a631640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:09.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7a631640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:09.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7962f640 10 monclient(hunting): get_auth_request con 0x7f7b74137300 auth_method 0 2024-04-26T00:20:09.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7962f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:09.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7962f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:09.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7962f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:09.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7a631640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:09.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7a631640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:09.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7a631640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:09.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7962f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:09.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7962f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:09.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7962f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:09.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b79e30640 1 --2- 172.21.15.37:0/3690566732 >> v2:172.21.15.45:3300/0 conn(0x7f7b74136f20 0x7f7b7409bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:09.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b79e30640 10 monclient(hunting): get_auth_request con 0x7f7b74136f20 auth_method 0 2024-04-26T00:20:09.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b79e30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:09.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b79e30640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:09.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b79e30640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:09.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7a631640 10 monclient(hunting): handle_auth_done global_id 4340 payload 274 2024-04-26T00:20:09.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7a631640 1 -- 172.21.15.37:0/3690566732 >> v2:172.21.15.45:3300/0 conn(0x7f7b74136f20 msgr2=0x7f7b7409bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:09.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7a631640 1 --2- 172.21.15.37:0/3690566732 >> v2:172.21.15.45:3300/0 conn(0x7f7b74136f20 0x7f7b7409bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:09.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7a631640 1 -- 172.21.15.37:0/3690566732 >> v2:172.21.15.37:3300/0 conn(0x7f7b74137300 msgr2=0x7f7b741377b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:09.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7a631640 1 --2- 172.21.15.37:0/3690566732 >> v2:172.21.15.37:3300/0 conn(0x7f7b74137300 0x7f7b741377b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:09.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7a631640 10 monclient: _finish_hunting 0 2024-04-26T00:20:09.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7a631640 1 monclient: found mon.noname-b 2024-04-26T00:20:09.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7a631640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:09.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7a631640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:20:09.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.223+0000 7f7b7a631640 1 -- 172.21.15.37:0/3690566732 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b7409b8f0 con 0x7f7b74133c70 2024-04-26T00:20:09.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7a631640 1 -- 172.21.15.37:0/3690566732 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7b74133c70 msgr2=0x7f7b74134120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:09.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7a631640 1 --2- 172.21.15.37:0/3690566732 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7b74133c70 0x7f7b74134120 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7b74068940 tx=0x7f7b6c01c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb506402686d9143 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:09.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b78e2e640 1 -- 172.21.15.37:0/3690566732 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c0a5640 con 0x7f7b74133c70 2024-04-26T00:20:09.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b78e2e640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:09.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b78e2e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:20:09.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b78e2e640 10 monclient: dump: 2024-04-26T00:20:09.213 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:09.213 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:09.214 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.214 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.214 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:09.214 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:09.214 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:09.214 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:09.214 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:09.214 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:09.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b78e2e640 10 monclient: _finish_auth 0 2024-04-26T00:20:09.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b78e2e640 10 monclient: _check_auth_tickets 2024-04-26T00:20:09.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b78e2e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:09.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b78e2e640 1 -- 172.21.15.37:0/3690566732 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b6c0a57e0 con 0x7f7b74133c70 2024-04-26T00:20:09.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7c094640 5 monclient: authenticate success, global_id 4340 2024-04-26T00:20:09.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7c094640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:20:09.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b78e2e640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:09.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7c094640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:09.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b78e2e640 1 -- 172.21.15.37:0/3690566732 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c0a5bb0 con 0x7f7b74133c70 2024-04-26T00:20:09.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b78e2e640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:09.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7c094640 4 set_mon_vals no callback set 2024-04-26T00:20:09.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b78e2e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:09.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b78e2e640 10 monclient: dump: 2024-04-26T00:20:09.215 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:09.215 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:09.215 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.215 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.215 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:09.215 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:09.215 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:09.216 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:09.216 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:09.216 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:09.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7c094640 10 monclient: shutdown 2024-04-26T00:20:09.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7c094640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:09.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7c094640 1 -- 172.21.15.37:0/3690566732 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7b74133c70 msgr2=0x7f7b74134120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:09.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7c094640 1 --2- 172.21.15.37:0/3690566732 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7b74133c70 0x7f7b74134120 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7b74068940 tx=0x7f7b6c01c8b0 comp rx=0 tx=0).stop 2024-04-26T00:20:09.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7c094640 1 -- 172.21.15.37:0/3690566732 shutdown_connections 2024-04-26T00:20:09.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7c094640 1 --2- 172.21.15.37:0/3690566732 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7b74133c70 0x7f7b74134120 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:09.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7c094640 1 --2- 172.21.15.37:0/3690566732 >> v2:172.21.15.37:3300/0 conn(0x7f7b74137300 0x7f7b741377b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:09.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7c094640 1 --2- 172.21.15.37:0/3690566732 >> v2:172.21.15.45:3300/0 conn(0x7f7b74136f20 0x7f7b7409bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:09.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7c094640 1 -- 172.21.15.37:0/3690566732 >> 172.21.15.37:0/3690566732 conn(0x7f7b740a61b0 msgr2=0x7f7b740a65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:09.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7c094640 1 -- 172.21.15.37:0/3690566732 shutdown_connections 2024-04-26T00:20:09.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7c094640 1 -- 172.21.15.37:0/3690566732 wait complete. 2024-04-26T00:20:09.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7c094640 10 monclient: build_initial_monmap 2024-04-26T00:20:09.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7c094640 10 monclient: monmap: 2024-04-26T00:20:09.217 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:09.217 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:09.217 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:09.217 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:09.217 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:09.217 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:09.217 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:09.217 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:09.217 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:09.217 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:09.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7c094640 1 Processor -- start 2024-04-26T00:20:09.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7c094640 1 -- start start 2024-04-26T00:20:09.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7c094640 10 monclient: init 2024-04-26T00:20:09.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7c094640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:09.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7c094640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:20:09.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7c094640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7b74133c70 0x7f7b741330b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:09.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7c094640 10 monclient(hunting): picked mon.noname-a con 0x7f7b74133c70 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:09.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7c094640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7b74136f20 0x7f7b741335f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:09.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7c094640 10 monclient(hunting): picked mon.noname-c con 0x7f7b74136f20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:09.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7c094640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b74137300 0x7f7b741cee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:09.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7c094640 10 monclient(hunting): picked mon.noname-b con 0x7f7b74137300 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:09.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7c094640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:09.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7c094640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f7b7409ac60 con 0x7f7b74133c70 2024-04-26T00:20:09.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7c094640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:09.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7c094640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7b7409aae0 con 0x7f7b74137300 2024-04-26T00:20:09.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7c094640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:09.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7c094640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f7b74140c80 con 0x7f7b74136f20 2024-04-26T00:20:09.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7c094640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:09.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7c094640 10 monclient(hunting): authenticate will time out at 990.539619s 2024-04-26T00:20:09.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b79e30640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7b74133c70 0x7f7b741330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:09.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b79e30640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7b74133c70 0x7f7b741330b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44992/0 (socket says 172.21.15.37:44992) 2024-04-26T00:20:09.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b79e30640 1 -- 172.21.15.37:0/3275869685 learned_addr learned my addr 172.21.15.37:0/3275869685 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:09.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b79e30640 10 monclient(hunting): get_auth_request con 0x7f7b74133c70 auth_method 0 2024-04-26T00:20:09.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b79e30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:09.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b79e30640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:09.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b79e30640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:09.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7962f640 1 --2- 172.21.15.37:0/3275869685 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7b74136f20 0x7f7b741335f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:09.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7962f640 10 monclient(hunting): get_auth_request con 0x7f7b74136f20 auth_method 0 2024-04-26T00:20:09.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7962f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:09.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7962f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:09.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7962f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:09.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b79e30640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:09.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b79e30640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:09.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b79e30640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:09.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7962f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:09.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7962f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:09.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7962f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:09.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.227+0000 7f7b7a631640 1 --2- 172.21.15.37:0/3275869685 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b74137300 0x7f7b741cee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:09.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b7a631640 10 monclient(hunting): get_auth_request con 0x7f7b74137300 auth_method 0 2024-04-26T00:20:09.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b7a631640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:09.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b7a631640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:09.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b7a631640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:09.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b79e30640 10 monclient(hunting): handle_auth_done global_id 4359 payload 824 2024-04-26T00:20:09.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b79e30640 1 -- 172.21.15.37:0/3275869685 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7b74136f20 msgr2=0x7f7b741335f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:09.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b79e30640 1 --2- 172.21.15.37:0/3275869685 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7b74136f20 0x7f7b741335f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:09.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b79e30640 1 -- 172.21.15.37:0/3275869685 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b74137300 msgr2=0x7f7b741cee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:09.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b79e30640 1 --2- 172.21.15.37:0/3275869685 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7b74137300 0x7f7b741cee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:09.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b79e30640 10 monclient: _finish_hunting 0 2024-04-26T00:20:09.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b79e30640 1 monclient: found mon.noname-a 2024-04-26T00:20:09.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b79e30640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:09.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b79e30640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:09.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b79e30640 1 -- 172.21.15.37:0/3275869685 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b7413b010 con 0x7f7b74133c70 2024-04-26T00:20:09.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b79e30640 1 -- 172.21.15.37:0/3275869685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7b74133c70 msgr2=0x7f7b741330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:09.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b79e30640 1 --2- 172.21.15.37:0/3275869685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7b74133c70 0x7f7b741330b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7b7000d800 tx=0x7f7b7000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:09.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b6a7fc640 1 -- 172.21.15.37:0/3275869685 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7000b870 con 0x7f7b74133c70 2024-04-26T00:20:09.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b6a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:09.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b6a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:09.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b6a7fc640 10 monclient: dump: 2024-04-26T00:20:09.226 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:09.226 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:09.226 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.226 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.226 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:09.226 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:09.227 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:09.227 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:09.227 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:09.227 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:09.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b6a7fc640 10 monclient: _finish_auth 0 2024-04-26T00:20:09.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b6a7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:20:09.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b6a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:09.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b7c094640 5 monclient: authenticate success, global_id 4359 2024-04-26T00:20:09.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b7c094640 10 monclient: _renew_subs 2024-04-26T00:20:09.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b7c094640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:09.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b7c094640 1 -- 172.21.15.37:0/3275869685 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b7409ae30 con 0x7f7b74133c70 2024-04-26T00:20:09.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b6a7fc640 1 -- 172.21.15.37:0/3275869685 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b70019070 con 0x7f7b74133c70 2024-04-26T00:20:09.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b6a7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:09.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b6a7fc640 1 -- 172.21.15.37:0/3275869685 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7000bc90 con 0x7f7b74133c70 2024-04-26T00:20:09.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b6a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:09.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b6a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:09.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b6a7fc640 10 monclient: dump: 2024-04-26T00:20:09.228 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:09.228 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:09.228 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.228 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.228 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:09.228 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:09.228 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:09.228 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:09.228 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:09.228 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:09.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b7c094640 10 monclient: _renew_subs 2024-04-26T00:20:09.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b6bfff640 4 set_mon_vals no callback set 2024-04-26T00:20:09.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b7c094640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:09.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b7c094640 1 -- 172.21.15.37:0/3275869685 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b7413a730 con 0x7f7b74133c70 2024-04-26T00:20:09.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b7c094640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:09.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b7c094640 1 --2- 172.21.15.37:0/3275869685 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7b741cf370 0x7f7b741cf750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:09.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b7c094640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:09.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b7c094640 1 -- 172.21.15.37:0/3275869685 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f7b741cfc90 con 0x7f7b741cf370 2024-04-26T00:20:09.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b7962f640 1 --2- 172.21.15.37:0/3275869685 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7b741cf370 0x7f7b741cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:09.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b7962f640 10 monclient: get_auth_request con 0x7f7b741cf370 auth_method 0 2024-04-26T00:20:09.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b7962f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:09.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b7962f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:09.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b7962f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:09.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b7962f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:09.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b7962f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:09.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b7962f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:09.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b6a7fc640 1 -- 172.21.15.37:0/3275869685 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f7b7001e460 con 0x7f7b74133c70 2024-04-26T00:20:09.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b7962f640 10 monclient(hunting): handle_auth_done global_id 4346 payload 824 2024-04-26T00:20:09.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b7962f640 1 -- 172.21.15.37:0/3275869685 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7b7404f510 con 0x7f7b741cf370 2024-04-26T00:20:09.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.231+0000 7f7b6a7fc640 1 --2- 172.21.15.37:0/3275869685 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7b58035e80 0x7f7b58038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:09.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.235+0000 7f7b6a7fc640 1 -- 172.21.15.37:0/3275869685 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 3506+0+0 (secure 0 0 0) 0x7f7b70048e30 con 0x7f7b74133c70 2024-04-26T00:20:09.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.235+0000 7f7b7a631640 1 --2- 172.21.15.37:0/3275869685 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7b58035e80 0x7f7b58038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:09.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.235+0000 7f7b7a631640 10 monclient: get_auth_request con 0x7f7b58035e80 auth_method 0 2024-04-26T00:20:09.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.235+0000 7f7b7962f640 1 -- 172.21.15.37:0/3275869685 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7b741cf370 msgr2=0x7f7b741cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:09.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.235+0000 7f7b7962f640 1 --2- 172.21.15.37:0/3275869685 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7b741cf370 0x7f7b741cf750 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7b740a0520 tx=0x7f7b64011cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:09.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.235+0000 7f7b6a7fc640 1 -- 172.21.15.37:0/3275869685 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b64019070 con 0x7f7b741cf370 2024-04-26T00:20:09.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.235+0000 7f7b6a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:09.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.235+0000 7f7b6a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:09.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.235+0000 7f7b6a7fc640 10 monclient: dump: 2024-04-26T00:20:09.230 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:09.231 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:09.231 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.231 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.231 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:09.231 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:09.231 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:09.231 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:09.231 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:09.231 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:09.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.235+0000 7f7b6a7fc640 1 -- 172.21.15.37:0/3275869685 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7b6400f860 con 0x7f7b741cf370 2024-04-26T00:20:09.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.235+0000 7f7b6a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:09.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.235+0000 7f7b6a7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:09.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.235+0000 7f7b6a7fc640 1 -- 172.21.15.37:0/3275869685 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7b741cf370 msgr2=0x7f7b741cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:09.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.235+0000 7f7b6a7fc640 1 --2- 172.21.15.37:0/3275869685 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7b741cf370 0x7f7b741cf750 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7b740a0520 tx=0x7f7b64011cf0 comp rx=0 tx=0).stop 2024-04-26T00:20:09.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.235+0000 7f7b6a7fc640 1 -- 172.21.15.37:0/3275869685 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7b741cf370 msgr2=0x7f7b741cf750 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:20:09.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.235+0000 7f7b6a7fc640 1 --2- 172.21.15.37:0/3275869685 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7b741cf370 0x7f7b741cf750 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7b740a0520 tx=0x7f7b64011cf0 comp rx=0 tx=0).stop 2024-04-26T00:20:09.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.235+0000 7f7b7a631640 1 --2- 172.21.15.37:0/3275869685 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7b58035e80 0x7f7b58038340 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7b740a5ec0 tx=0x7f7b6c0a7070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:09.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.243+0000 7f7b7c094640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:20:09.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.243+0000 7f7b7c094640 1 --2- 172.21.15.37:0/3275869685 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7b74134c80 0x7f7b74135090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:09.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.243+0000 7f7b7c094640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:09.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.243+0000 7f7b7c094640 1 -- 172.21.15.37:0/3275869685 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f7b74075f10 con 0x7f7b74134c80 2024-04-26T00:20:09.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.243+0000 7f7b7962f640 1 --2- 172.21.15.37:0/3275869685 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7b74134c80 0x7f7b74135090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:09.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.243+0000 7f7b7962f640 10 monclient: get_auth_request con 0x7f7b74134c80 auth_method 0 2024-04-26T00:20:09.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.243+0000 7f7b7962f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:09.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.243+0000 7f7b7962f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:09.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.243+0000 7f7b7962f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:09.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.243+0000 7f7b7962f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:09.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.243+0000 7f7b7962f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:09.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.243+0000 7f7b7962f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:09.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.243+0000 7f7b7962f640 10 monclient(hunting): handle_auth_done global_id 4349 payload 824 2024-04-26T00:20:09.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.243+0000 7f7b7962f640 1 -- 172.21.15.37:0/3275869685 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7b74138a90 con 0x7f7b74134c80 2024-04-26T00:20:09.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.243+0000 7f7b7962f640 1 -- 172.21.15.37:0/3275869685 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7b74134c80 msgr2=0x7f7b74135090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:09.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.243+0000 7f7b7962f640 1 --2- 172.21.15.37:0/3275869685 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7b74134c80 0x7f7b74135090 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7b74075a60 tx=0x7f7b640046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:09.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.243+0000 7f7b6a7fc640 1 -- 172.21.15.37:0/3275869685 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b64019040 con 0x7f7b74134c80 2024-04-26T00:20:09.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.243+0000 7f7b6a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:09.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.243+0000 7f7b6a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:09.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.243+0000 7f7b6a7fc640 10 monclient: dump: 2024-04-26T00:20:09.234 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:09.234 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:09.234 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.234 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.234 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:09.234 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:09.234 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:09.234 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:09.234 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:09.234 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:09.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.243+0000 7f7b6a7fc640 1 -- 172.21.15.37:0/3275869685 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4716 (secure 0 0 0) 0x7f7b6401e450 con 0x7f7b74134c80 2024-04-26T00:20:09.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.243+0000 7f7b6a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:20:09.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.243+0000 7f7b6a7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:09.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.243+0000 7f7b6a7fc640 1 -- 172.21.15.37:0/3275869685 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7b74134c80 msgr2=0x7f7b74135090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:09.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.243+0000 7f7b6a7fc640 1 --2- 172.21.15.37:0/3275869685 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7b74134c80 0x7f7b74135090 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7b74075a60 tx=0x7f7b640046d0 comp rx=0 tx=0).stop 2024-04-26T00:20:09.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.243+0000 7f7b6a7fc640 1 -- 172.21.15.37:0/3275869685 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7b74134c80 msgr2=0x7f7b74135090 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:20:09.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.243+0000 7f7b6a7fc640 1 --2- 172.21.15.37:0/3275869685 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7b74134c80 0x7f7b74135090 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:09.235 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:20:09.235 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:20:09.235 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:20:09.236 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:20:09.236 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 6, 2024-04-26T00:20:09.236 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 26351, 2024-04-26T00:20:09.236 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:20:09.236 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:20:09.236 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:20:09.236 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:20:09.236 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:09.236 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 19, 2024-04-26T00:20:09.236 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:20:09.236 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:20:09.236 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:20:09.236 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:20:09.236 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:20:09.236 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:20:09.237 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:20:09.237 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:20:09.237 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:20:09.237 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:20:09.237 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:20:09.237 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:20:09.237 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:20:09.237 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:20:09.237 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:09.237 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:20:09.237 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:20:09.237 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:20:09.237 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:20:09.237 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:20:09.237 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:20:09.238 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:20:09.238 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:20:09.238 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:20:09.238 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:20:09.238 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:20:09.238 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:20:09.238 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:20:09.238 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:09.238 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:09.238 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:20:09.238 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:20:09.238 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:20:09.238 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:20:09.239 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:20:09.239 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:20:09.239 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:20:09.239 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:20:09.239 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:20:09.239 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:20:09.239 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:20:09.239 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:20:09.239 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:20:09.239 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:20:09.239 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:20:09.239 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:20:09.239 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:20:09.239 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:20:09.240 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:20:09.240 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:20:09.240 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:20:09.240 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:20:09.240 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:20:09.240 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:20:09.240 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:20:09.240 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:20:09.240 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:20:09.240 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:20:09.240 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:09.240 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:20:09.240 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:09.240 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:20:09.240 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:09.241 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:20:09.241 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:20:09.241 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:20:09.241 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:20:09.241 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:09.241 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:20:09.241 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:20:09.241 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:09.241 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:09.241 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:09.241 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:20:09.241 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:20:09.241 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:09.241 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:09.241 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:09.242 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:09.242 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:20:09.242 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:20:09.242 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:20:09.242 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:20:09.242 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:20:09.242 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:09.242 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:09.242 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:20:09.242 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:20:09.242 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:20:09.242 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:20:09.242 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:09.242 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:20:09.243 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:20:09.243 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:09.243 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:09.243 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:09.243 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:20:09.243 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:20:09.243 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:09.243 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:09.243 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:09.243 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:09.243 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:20:09.243 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:20:09.243 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:20:09.243 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:20:09.243 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:20:09.244 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:09.244 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:09.244 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:20:09.244 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:20:09.244 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:20:09.244 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:20:09.244 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:09.244 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:20:09.244 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:20:09.244 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:09.244 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:09.244 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:09.244 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:20:09.244 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:20:09.245 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:09.245 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:09.245 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:09.245 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:09.245 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:20:09.245 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:20:09.245 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:20:09.245 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:20:09.245 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:20:09.245 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:09.245 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:09.245 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:09.245 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:20:09.245 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:20:09.246 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:09.246 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:20:09.246 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:20:09.246 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:20:09.246 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:09.246 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:09.246 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:20:09.246 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:09.246 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:20:09.246 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:20:09.246 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:20:09.246 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:09.246 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:09.246 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:20:09.246 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:09.247 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:20:09.247 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:20:09.247 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:20:09.247 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:09.247 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:09.247 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:09.247 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:20:09.247 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:20:09.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.247+0000 7f7b7c094640 1 -- 172.21.15.37:0/3275869685 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7b58035e80 msgr2=0x7f7b58038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:09.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.247+0000 7f7b7c094640 1 --2- 172.21.15.37:0/3275869685 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7b58035e80 0x7f7b58038340 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7b740a5ec0 tx=0x7f7b6c0a7070 comp rx=0 tx=0).stop 2024-04-26T00:20:09.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.247+0000 7f7b7c094640 10 monclient: shutdown 2024-04-26T00:20:09.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.247+0000 7f7b7c094640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:09.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.247+0000 7f7b7c094640 1 -- 172.21.15.37:0/3275869685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7b74133c70 msgr2=0x7f7b741330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:09.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.247+0000 7f7b7c094640 1 --2- 172.21.15.37:0/3275869685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7b74133c70 0x7f7b741330b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7b7000d800 tx=0x7f7b7000dcc0 comp rx=0 tx=0).stop 2024-04-26T00:20:09.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.247+0000 7f7b7a631640 1 -- 172.21.15.37:0/3275869685 reap_dead start 2024-04-26T00:20:09.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.247+0000 7f7b7a631640 1 -- 172.21.15.37:0/3275869685 reap_dead start 2024-04-26T00:20:09.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.247+0000 7f7b7c094640 1 -- 172.21.15.37:0/3275869685 shutdown_connections 2024-04-26T00:20:09.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.247+0000 7f7b7c094640 1 -- 172.21.15.37:0/3275869685 >> 172.21.15.37:0/3275869685 conn(0x7f7b740a61b0 msgr2=0x7f7b740a7580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:09.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.247+0000 7f7b7c094640 1 -- 172.21.15.37:0/3275869685 shutdown_connections 2024-04-26T00:20:09.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.247+0000 7f7b7c094640 1 -- 172.21.15.37:0/3275869685 wait complete. 2024-04-26T00:20:09.249 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:20:09.249 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:20:09.250 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:20:09.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.299+0000 7fdef77fe640 1 -- 172.21.15.37:0/852549648 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]=0 pool 'unique_pool_0' created v16) ==== 130+0+0 (secure 0 0 0) 0x7fdf040aa7c0 con 0x7fdf100a0110 2024-04-26T00:20:09.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.299+0000 7fdef77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}] 2024-04-26T00:20:09.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.299+0000 7fdef77fe640 10 monclient: _finish_command 2 = system:0 pool 'unique_pool_0' created 2024-04-26T00:20:09.283 INFO:teuthology.orchestra.run.smithi037.stderr:pool 'unique_pool_0' created 2024-04-26T00:20:09.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.303+0000 7fdf1524c640 1 -- 172.21.15.37:0/852549648 >> v2:172.21.15.45:6812/2379918623 conn(0x7fdefc035e80 msgr2=0x7fdefc038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:09.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.303+0000 7fdf1524c640 1 --2- 172.21.15.37:0/852549648 >> v2:172.21.15.45:6812/2379918623 conn(0x7fdefc035e80 0x7fdefc038340 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdf00009eb0 tx=0x7fdf00009970 comp rx=0 tx=0).stop 2024-04-26T00:20:09.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.303+0000 7fdf1524c640 10 monclient: shutdown 2024-04-26T00:20:09.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.303+0000 7fdf1524c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:09.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.303+0000 7fdf1524c640 1 -- 172.21.15.37:0/852549648 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdf100a0110 msgr2=0x7fdf10135b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:09.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.303+0000 7fdf1524c640 1 --2- 172.21.15.37:0/852549648 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdf100a0110 0x7fdf10135b00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdf040ad000 tx=0x7fdf04018ee0 comp rx=0 tx=0).stop 2024-04-26T00:20:09.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.303+0000 7fdf1524c640 1 -- 172.21.15.37:0/852549648 shutdown_connections 2024-04-26T00:20:09.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.303+0000 7fdf1524c640 1 --2- 172.21.15.37:0/852549648 >> v2:172.21.15.45:6812/2379918623 conn(0x7fdefc035e80 0x7fdefc038340 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:09.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.303+0000 7fdf1524c640 1 --2- 172.21.15.37:0/852549648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf1013c630 0x7fdf1011b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:09.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.303+0000 7fdf1524c640 1 --2- 172.21.15.37:0/852549648 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdf100a0110 0x7fdf10135b00 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:09.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.303+0000 7fdf1524c640 1 --2- 172.21.15.37:0/852549648 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdf100726b0 0x7fdf101cf100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:09.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.303+0000 7fdf1524c640 1 -- 172.21.15.37:0/852549648 >> 172.21.15.37:0/852549648 conn(0x7fdf100a1d50 msgr2=0x7fdf1013ea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:09.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.303+0000 7fdf1524c640 1 -- 172.21.15.37:0/852549648 shutdown_connections 2024-04-26T00:20:09.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.303+0000 7fdf1524c640 1 -- 172.21.15.37:0/852549648 wait complete. 2024-04-26T00:20:09.305 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-04-26T00:20:09.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.375+0000 7f67f9d22640 10 monclient: get_monmap_and_config 2024-04-26T00:20:09.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.375+0000 7f67f9d22640 10 monclient: build_initial_monmap 2024-04-26T00:20:09.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.375+0000 7f67f9d22640 10 monclient: monmap: 2024-04-26T00:20:09.358 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:09.358 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:09.358 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:09.381920+0000 2024-04-26T00:20:09.358 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:09.381920+0000 2024-04-26T00:20:09.358 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:09.358 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:09.358 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:09.358 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:09.359 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:09.359 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:09.359 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:09.359 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:09.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.375+0000 7f67f9d22640 1 Processor -- start 2024-04-26T00:20:09.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.375+0000 7f67f9d22640 1 -- start start 2024-04-26T00:20:09.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.375+0000 7f67f9d22640 10 monclient: init 2024-04-26T00:20:09.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.375+0000 7f67f9d22640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:09.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.375+0000 7f67f9d22640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-04-26T00:20:09.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.375+0000 7f67f9d22640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f67f409f860 0x7f67f409c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:09.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.375+0000 7f67f9d22640 10 monclient(hunting): picked mon.noname-c con 0x7f67f409f860 addr v2:172.21.15.45:3300/0 2024-04-26T00:20:09.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.375+0000 7f67f9d22640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f67f409cd00 addr v1:172.21.15.37:6789/0 2024-04-26T00:20:09.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.375+0000 7f67f9d22640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f67f40a2270 0x7f67f413cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:09.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.375+0000 7f67f9d22640 10 monclient(hunting): picked mon.noname-b con 0x7f67f40a2270 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:09.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.375+0000 7f67f9d22640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67f406c0e0 con 0x7f67f409cd00 2024-04-26T00:20:09.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.375+0000 7f67f9d22640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:09.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.375+0000 7f67f9d22640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f67f409a280 con 0x7f67f409f860 2024-04-26T00:20:09.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.375+0000 7f67f9d22640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:09.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.375+0000 7f67f9d22640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f67f409fc40 con 0x7f67f40a2270 2024-04-26T00:20:09.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.375+0000 7f67f9d22640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:09.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.375+0000 7f67f9d22640 10 monclient(hunting): authenticate will time out at 990.687594s 2024-04-26T00:20:09.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.375+0000 7f67f3fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f67f40a2270 0x7f67f413cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:09.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.375+0000 7f67f3fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f67f40a2270 0x7f67f413cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:58212/0 (socket says 172.21.15.37:58212) 2024-04-26T00:20:09.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.375+0000 7f67f3fff640 1 -- 172.21.15.37:0/1181024228 learned_addr learned my addr 172.21.15.37:0/1181024228 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:09.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.375+0000 7f67f3fff640 10 monclient(hunting): get_auth_request con 0x7f67f40a2270 auth_method 0 2024-04-26T00:20:09.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.375+0000 7f67f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:09.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.375+0000 7f67f3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:09.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.375+0000 7f67f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:09.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.375+0000 7f67f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:09.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.375+0000 7f67f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:09.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.375+0000 7f67f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:09.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.379+0000 7f67f3fff640 10 monclient(hunting): handle_auth_done global_id 4352 payload 274 2024-04-26T00:20:09.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.379+0000 7f67f3fff640 1 -- 172.21.15.37:0/1181024228 >> v2:172.21.15.45:3300/0 conn(0x7f67f409f860 msgr2=0x7f67f409c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:09.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.379+0000 7f67f37fe640 1 --2- 172.21.15.37:0/1181024228 >> v2:172.21.15.45:3300/0 conn(0x7f67f409f860 0x7f67f409c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:09.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.379+0000 7f67f3fff640 1 --2- 172.21.15.37:0/1181024228 >> v2:172.21.15.45:3300/0 conn(0x7f67f409f860 0x7f67f409c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:09.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.379+0000 7f67f3fff640 1 -- 172.21.15.37:0/1181024228 >> v1:172.21.15.37:6789/0 conn(0x7f67f409cd00 legacy=0x7f67f40a1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:09.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.379+0000 7f67f3fff640 10 monclient: _finish_hunting 0 2024-04-26T00:20:09.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.379+0000 7f67f3fff640 1 monclient: found mon.noname-b 2024-04-26T00:20:09.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.379+0000 7f67f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:09.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.379+0000 7f67f3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:20:09.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.379+0000 7f67f3fff640 1 -- 172.21.15.37:0/1181024228 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67f41354d0 con 0x7f67f40a2270 2024-04-26T00:20:09.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f3fff640 1 -- 172.21.15.37:0/1181024228 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f67f40a2270 msgr2=0x7f67f413cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:09.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f3fff640 1 --2- 172.21.15.37:0/1181024228 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f67f40a2270 0x7f67f413cff0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f67f406f060 tx=0x7f67ec01c870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8c476d1f19ba003 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:09.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f27fc640 1 -- 172.21.15.37:0/1181024228 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ec0a55e0 con 0x7f67f40a2270 2024-04-26T00:20:09.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:09.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:20:09.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f27fc640 10 monclient: dump: 2024-04-26T00:20:09.366 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:09.366 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:09.366 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.366 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.366 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:09.366 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:09.366 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:09.366 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:09.367 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:09.367 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:09.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f27fc640 10 monclient: _finish_auth 0 2024-04-26T00:20:09.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f27fc640 10 monclient: _check_auth_tickets 2024-04-26T00:20:09.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:09.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f27fc640 1 -- 172.21.15.37:0/1181024228 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67ec0a5780 con 0x7f67f40a2270 2024-04-26T00:20:09.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f27fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:09.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f27fc640 1 -- 172.21.15.37:0/1181024228 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ec0a5b50 con 0x7f67f40a2270 2024-04-26T00:20:09.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f9d22640 5 monclient: authenticate success, global_id 4352 2024-04-26T00:20:09.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f9d22640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:09.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f9d22640 4 set_mon_vals no callback set 2024-04-26T00:20:09.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:09.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:09.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f27fc640 10 monclient: dump: 2024-04-26T00:20:09.367 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:09.368 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:09.368 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.368 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.368 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:09.368 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:09.368 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:09.368 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:09.368 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:09.368 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:09.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f9d22640 10 monclient: shutdown 2024-04-26T00:20:09.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f9d22640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:09.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f9d22640 1 -- 172.21.15.37:0/1181024228 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f67f40a2270 msgr2=0x7f67f413cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:09.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f9d22640 1 --2- 172.21.15.37:0/1181024228 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f67f40a2270 0x7f67f413cff0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f67f406f060 tx=0x7f67ec01c870 comp rx=0 tx=0).stop 2024-04-26T00:20:09.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f9d22640 1 -- 172.21.15.37:0/1181024228 shutdown_connections 2024-04-26T00:20:09.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f9d22640 1 --2- 172.21.15.37:0/1181024228 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f67f40a2270 0x7f67f413cff0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:09.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f9d22640 1 --2- 172.21.15.37:0/1181024228 >> v2:172.21.15.45:3300/0 conn(0x7f67f409f860 0x7f67f409c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:09.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f9d22640 1 -- 172.21.15.37:0/1181024228 >> 172.21.15.37:0/1181024228 conn(0x7f67f40a2c50 msgr2=0x7f67f40a3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:09.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f9d22640 1 -- 172.21.15.37:0/1181024228 shutdown_connections 2024-04-26T00:20:09.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f9d22640 1 -- 172.21.15.37:0/1181024228 wait complete. 2024-04-26T00:20:09.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f9d22640 10 monclient: build_initial_monmap 2024-04-26T00:20:09.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f9d22640 10 monclient: monmap: 2024-04-26T00:20:09.369 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:09.369 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:09.369 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:09.369 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:09.369 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:09.369 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:09.369 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:09.370 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:09.370 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:09.370 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:09.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f9d22640 1 Processor -- start 2024-04-26T00:20:09.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f9d22640 1 -- start start 2024-04-26T00:20:09.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f9d22640 10 monclient: init 2024-04-26T00:20:09.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f9d22640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:09.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f9d22640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:20:09.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f9d22640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67f409cd00 0x7f67f409d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:09.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f9d22640 10 monclient(hunting): picked mon.noname-b con 0x7f67f409cd00 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:09.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f9d22640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f67f409f860 0x7f67f409db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:09.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f9d22640 10 monclient(hunting): picked mon.noname-c con 0x7f67f409f860 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:09.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f9d22640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f67f40a2270 0x7f67f409e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:09.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f9d22640 10 monclient(hunting): picked mon.noname-a con 0x7f67f40a2270 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:09.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f9d22640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:09.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f9d22640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f67f40a0300 con 0x7f67f40a2270 2024-04-26T00:20:09.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f9d22640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:09.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f9d22640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f67f4135170 con 0x7f67f409cd00 2024-04-26T00:20:09.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f9d22640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:09.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f9d22640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f67f413af60 con 0x7f67f409f860 2024-04-26T00:20:09.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f9d22640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:09.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f9d22640 10 monclient(hunting): authenticate will time out at 990.694537s 2024-04-26T00:20:09.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f3fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f67f40a2270 0x7f67f409e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:09.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f2ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f67f409f860 0x7f67f409db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:09.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f3fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f67f40a2270 0x7f67f409e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:45004/0 (socket says 172.21.15.37:45004) 2024-04-26T00:20:09.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f3fff640 1 -- 172.21.15.37:0/110089496 learned_addr learned my addr 172.21.15.37:0/110089496 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:09.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f3fff640 10 monclient(hunting): get_auth_request con 0x7f67f40a2270 auth_method 0 2024-04-26T00:20:09.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:09.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:09.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:09.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f2ffd640 10 monclient(hunting): get_auth_request con 0x7f67f409f860 auth_method 0 2024-04-26T00:20:09.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:09.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:09.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:09.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:09.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:09.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:09.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:09.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:09.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:09.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f37fe640 1 --2- 172.21.15.37:0/110089496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67f409cd00 0x7f67f409d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:09.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f37fe640 10 monclient(hunting): get_auth_request con 0x7f67f409cd00 auth_method 0 2024-04-26T00:20:09.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:09.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:09.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:09.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f3fff640 10 monclient(hunting): handle_auth_done global_id 4362 payload 824 2024-04-26T00:20:09.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f3fff640 1 -- 172.21.15.37:0/110089496 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f67f409f860 msgr2=0x7f67f409db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:09.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f3fff640 1 --2- 172.21.15.37:0/110089496 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f67f409f860 0x7f67f409db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:09.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f3fff640 1 -- 172.21.15.37:0/110089496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67f409cd00 msgr2=0x7f67f409d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:09.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f3fff640 1 --2- 172.21.15.37:0/110089496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67f409cd00 0x7f67f409d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:09.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f3fff640 10 monclient: _finish_hunting 0 2024-04-26T00:20:09.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f3fff640 1 monclient: found mon.noname-a 2024-04-26T00:20:09.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:09.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:09.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f3fff640 1 -- 172.21.15.37:0/110089496 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67f406b490 con 0x7f67f40a2270 2024-04-26T00:20:09.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.383+0000 7f67f2ffd640 1 --2- 172.21.15.37:0/110089496 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f67f409f860 0x7f67f409db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:09.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67f3fff640 1 -- 172.21.15.37:0/110089496 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f67f40a2270 msgr2=0x7f67f409e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:09.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67f3fff640 1 --2- 172.21.15.37:0/110089496 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f67f40a2270 0x7f67f409e090 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f67ec01af20 tx=0x7f67ec019610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:09.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67dbfff640 1 -- 172.21.15.37:0/110089496 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ec0bc020 con 0x7f67f40a2270 2024-04-26T00:20:09.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:09.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67dbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:09.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67dbfff640 10 monclient: dump: 2024-04-26T00:20:09.379 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:09.379 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:09.379 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.379 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.379 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:09.379 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:09.379 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:09.379 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:09.379 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:09.379 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:09.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67dbfff640 10 monclient: _finish_auth 0 2024-04-26T00:20:09.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67dbfff640 10 monclient: _check_auth_tickets 2024-04-26T00:20:09.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:09.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67f9d22640 5 monclient: authenticate success, global_id 4362 2024-04-26T00:20:09.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67f9d22640 10 monclient: _renew_subs 2024-04-26T00:20:09.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67f9d22640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:09.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67f9d22640 1 -- 172.21.15.37:0/110089496 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67f41309c0 con 0x7f67f40a2270 2024-04-26T00:20:09.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67dbfff640 1 -- 172.21.15.37:0/110089496 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67ec0b8880 con 0x7f67f40a2270 2024-04-26T00:20:09.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67dbfff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:09.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67dbfff640 1 -- 172.21.15.37:0/110089496 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ec0a55e0 con 0x7f67f40a2270 2024-04-26T00:20:09.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:09.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67dbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:09.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67dbfff640 10 monclient: dump: 2024-04-26T00:20:09.380 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:09.380 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:09.380 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.380 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.380 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:09.381 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:09.381 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:09.381 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:09.381 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:09.381 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:09.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67f9d22640 10 monclient: _renew_subs 2024-04-26T00:20:09.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67f9d22640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:09.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67f9d22640 1 -- 172.21.15.37:0/110089496 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67f413bab0 con 0x7f67f40a2270 2024-04-26T00:20:09.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67f1ffb640 4 set_mon_vals no callback set 2024-04-26T00:20:09.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67f9d22640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:09.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67dbfff640 1 -- 172.21.15.37:0/110089496 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f67ec0bf020 con 0x7f67f40a2270 2024-04-26T00:20:09.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67f9d22640 1 --2- 172.21.15.37:0/110089496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67f41328a0 0x7f67f4132c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:09.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67f9d22640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:09.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67f9d22640 1 -- 172.21.15.37:0/110089496 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f67f413ca10 con 0x7f67f41328a0 2024-04-26T00:20:09.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67dbfff640 1 --2- 172.21.15.37:0/110089496 >> v2:172.21.15.45:6812/2379918623 conn(0x7f67e0035eb0 0x7f67e0038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:09.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67dbfff640 1 -- 172.21.15.37:0/110089496 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 3858+0+0 (secure 0 0 0) 0x7f67ec0e3a30 con 0x7f67f40a2270 2024-04-26T00:20:09.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67f37fe640 1 --2- 172.21.15.37:0/110089496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67f41328a0 0x7f67f4132c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:09.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67f37fe640 10 monclient: get_auth_request con 0x7f67f41328a0 auth_method 0 2024-04-26T00:20:09.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:09.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67f37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:09.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:09.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67f2ffd640 1 --2- 172.21.15.37:0/110089496 >> v2:172.21.15.45:6812/2379918623 conn(0x7f67e0035eb0 0x7f67e0038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:09.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67f2ffd640 10 monclient: get_auth_request con 0x7f67e0035eb0 auth_method 0 2024-04-26T00:20:09.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:09.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:09.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:09.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67f37fe640 10 monclient(hunting): handle_auth_done global_id 4324 payload 824 2024-04-26T00:20:09.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.387+0000 7f67f37fe640 1 -- 172.21.15.37:0/110089496 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f67f409e9c0 con 0x7f67f41328a0 2024-04-26T00:20:09.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.391+0000 7f67f2ffd640 1 --2- 172.21.15.37:0/110089496 >> v2:172.21.15.45:6812/2379918623 conn(0x7f67e0035eb0 0x7f67e0038370 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f67f409eda0 tx=0x7f67e8009060 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:09.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.391+0000 7f67f37fe640 1 -- 172.21.15.37:0/110089496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67f41328a0 msgr2=0x7f67f4132c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:09.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.391+0000 7f67f37fe640 1 --2- 172.21.15.37:0/110089496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67f41328a0 0x7f67f4132c80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f67e400e7a0 tx=0x7f67e400ec60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:09.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.391+0000 7f67dbfff640 1 -- 172.21.15.37:0/110089496 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e4019020 con 0x7f67f41328a0 2024-04-26T00:20:09.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.391+0000 7f67dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:09.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.391+0000 7f67dbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:20:09.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.391+0000 7f67dbfff640 10 monclient: dump: 2024-04-26T00:20:09.383 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:09.383 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:09.383 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.383 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.383 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:09.383 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:09.383 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:09.383 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:09.384 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:09.384 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:09.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.391+0000 7f67dbfff640 1 -- 172.21.15.37:0/110089496 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f67e400aba0 con 0x7f67f41328a0 2024-04-26T00:20:09.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.391+0000 7f67dbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:09.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.391+0000 7f67dbfff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:09.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.391+0000 7f67dbfff640 1 -- 172.21.15.37:0/110089496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67f41328a0 msgr2=0x7f67f4132c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:09.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.391+0000 7f67dbfff640 1 --2- 172.21.15.37:0/110089496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67f41328a0 0x7f67f4132c80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f67e400e7a0 tx=0x7f67e400ec60 comp rx=0 tx=0).stop 2024-04-26T00:20:09.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.391+0000 7f67dbfff640 1 -- 172.21.15.37:0/110089496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67f41328a0 msgr2=0x7f67f4132c80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:20:09.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.391+0000 7f67dbfff640 1 --2- 172.21.15.37:0/110089496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67f41328a0 0x7f67f4132c80 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f67e400e7a0 tx=0x7f67e400ec60 comp rx=0 tx=0).stop 2024-04-26T00:20:09.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.399+0000 7f67f9d22640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:20:09.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.399+0000 7f67f9d22640 1 --2- 172.21.15.37:0/110089496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67f41331c0 0x7f67f41d9d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:09.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.399+0000 7f67f9d22640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:09.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.399+0000 7f67f9d22640 1 -- 172.21.15.37:0/110089496 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f67f409e7f0 con 0x7f67f41331c0 2024-04-26T00:20:09.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.399+0000 7f67f37fe640 1 --2- 172.21.15.37:0/110089496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67f41331c0 0x7f67f41d9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:09.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.399+0000 7f67f37fe640 10 monclient: get_auth_request con 0x7f67f41331c0 auth_method 0 2024-04-26T00:20:09.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.399+0000 7f67f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:09.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.399+0000 7f67f37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:09.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.399+0000 7f67f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:09.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.399+0000 7f67f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:09.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.399+0000 7f67f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:09.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.399+0000 7f67f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:09.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.399+0000 7f67f37fe640 10 monclient(hunting): handle_auth_done global_id 4327 payload 824 2024-04-26T00:20:09.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.399+0000 7f67f37fe640 1 -- 172.21.15.37:0/110089496 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f67f413b110 con 0x7f67f41331c0 2024-04-26T00:20:09.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.403+0000 7f67f37fe640 1 -- 172.21.15.37:0/110089496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67f41331c0 msgr2=0x7f67f41d9d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:09.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.403+0000 7f67f37fe640 1 --2- 172.21.15.37:0/110089496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67f41331c0 0x7f67f41d9d00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f67e400e7a0 tx=0x7f67e40046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:09.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.403+0000 7f67dbfff640 1 -- 172.21.15.37:0/110089496 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e4019030 con 0x7f67f41331c0 2024-04-26T00:20:09.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.403+0000 7f67dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:09.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.403+0000 7f67dbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:20:09.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.403+0000 7f67dbfff640 10 monclient: dump: 2024-04-26T00:20:09.387 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:09.387 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:09.387 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.387 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.388 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:09.388 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:09.388 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:09.388 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:09.388 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:09.388 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:09.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.403+0000 7f67dbfff640 1 -- 172.21.15.37:0/110089496 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4886 (secure 0 0 0) 0x7f67e40202c0 con 0x7f67f41331c0 2024-04-26T00:20:09.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.403+0000 7f67dbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:20:09.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.403+0000 7f67dbfff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:09.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.403+0000 7f67dbfff640 1 -- 172.21.15.37:0/110089496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67f41331c0 msgr2=0x7f67f41d9d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:09.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.403+0000 7f67dbfff640 1 --2- 172.21.15.37:0/110089496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67f41331c0 0x7f67f41d9d00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f67e400e7a0 tx=0x7f67e40046d0 comp rx=0 tx=0).stop 2024-04-26T00:20:09.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.403+0000 7f67dbfff640 1 -- 172.21.15.37:0/110089496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67f41331c0 msgr2=0x7f67f41d9d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:20:09.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.403+0000 7f67dbfff640 1 --2- 172.21.15.37:0/110089496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67f41331c0 0x7f67f41d9d00 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:09.389 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:20:09.389 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:20:09.389 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:20:09.389 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:20:09.389 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 6, 2024-04-26T00:20:09.389 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 26471, 2024-04-26T00:20:09.389 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:20:09.389 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:20:09.389 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:20:09.389 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:20:09.389 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:09.389 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 19, 2024-04-26T00:20:09.390 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:20:09.390 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:20:09.390 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:20:09.390 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:20:09.390 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:20:09.390 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:20:09.390 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:20:09.390 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:20:09.390 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:20:09.390 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:20:09.390 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:20:09.390 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:20:09.390 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:20:09.390 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:20:09.391 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:09.391 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:20:09.391 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:20:09.391 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:20:09.391 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:20:09.391 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:20:09.391 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:20:09.391 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:20:09.391 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:20:09.391 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:20:09.391 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:20:09.391 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:20:09.391 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:20:09.391 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:20:09.392 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:09.392 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:09.392 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:20:09.392 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:20:09.392 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:20:09.392 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:20:09.392 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:20:09.392 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:20:09.392 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:20:09.392 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:20:09.392 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:20:09.392 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:20:09.392 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:20:09.392 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:20:09.393 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:20:09.393 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:20:09.393 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:20:09.393 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:20:09.393 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:20:09.393 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:20:09.393 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:20:09.393 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:20:09.393 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:20:09.393 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:20:09.393 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:20:09.393 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:20:09.393 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:20:09.393 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:20:09.394 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:20:09.394 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:20:09.394 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:09.394 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:20:09.394 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:09.394 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:20:09.394 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:09.394 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:20:09.394 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:20:09.394 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:20:09.394 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:20:09.394 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:09.394 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:20:09.394 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:20:09.395 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:09.395 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:09.395 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:09.395 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:20:09.395 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:20:09.395 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:09.395 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:09.395 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:09.395 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:09.395 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:20:09.395 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:20:09.395 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:20:09.395 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:20:09.395 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:20:09.396 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:09.396 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:09.396 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:20:09.396 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:20:09.396 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:20:09.396 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:20:09.396 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:09.396 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:20:09.396 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:20:09.397 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:09.397 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:09.397 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:09.397 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:20:09.397 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:20:09.397 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:09.397 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:09.397 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:09.397 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:09.397 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:20:09.397 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:20:09.397 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:20:09.397 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:20:09.397 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:20:09.398 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:09.398 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:09.398 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:20:09.398 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:20:09.398 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:20:09.398 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:20:09.398 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:09.398 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:20:09.398 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:20:09.398 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:09.398 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:09.398 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:09.398 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:20:09.398 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:20:09.399 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:09.399 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:09.399 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:09.399 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:09.399 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:20:09.399 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:20:09.399 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:20:09.399 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:20:09.399 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:20:09.399 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:09.399 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:09.399 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:09.399 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:20:09.399 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:20:09.400 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:09.400 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:20:09.400 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:20:09.400 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:20:09.400 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:09.400 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:09.400 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:20:09.400 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:09.400 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:20:09.400 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:20:09.400 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:20:09.400 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:09.400 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:09.400 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:20:09.400 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:09.401 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:20:09.401 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:20:09.401 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:20:09.401 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:09.401 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:09.401 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:20:09.401 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:09.401 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:20:09.401 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:20:09.401 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:20:09.401 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:09.401 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:09.401 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:09.401 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:20:09.401 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:20:09.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.403+0000 7f67f9d22640 1 -- 172.21.15.37:0/110089496 >> v2:172.21.15.45:6812/2379918623 conn(0x7f67e0035eb0 msgr2=0x7f67e0038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:09.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.403+0000 7f67f9d22640 1 --2- 172.21.15.37:0/110089496 >> v2:172.21.15.45:6812/2379918623 conn(0x7f67e0035eb0 0x7f67e0038370 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f67f409eda0 tx=0x7f67e8009060 comp rx=0 tx=0).stop 2024-04-26T00:20:09.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.403+0000 7f67f9d22640 10 monclient: shutdown 2024-04-26T00:20:09.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.403+0000 7f67f9d22640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:09.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.403+0000 7f67f9d22640 1 -- 172.21.15.37:0/110089496 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f67f40a2270 msgr2=0x7f67f409e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:09.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.403+0000 7f67f9d22640 1 --2- 172.21.15.37:0/110089496 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f67f40a2270 0x7f67f409e090 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f67ec01af20 tx=0x7f67ec019610 comp rx=0 tx=0).stop 2024-04-26T00:20:09.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.403+0000 7f67f3fff640 1 -- 172.21.15.37:0/110089496 reap_dead start 2024-04-26T00:20:09.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.403+0000 7f67f3fff640 1 -- 172.21.15.37:0/110089496 reap_dead start 2024-04-26T00:20:09.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.403+0000 7f67f9d22640 1 -- 172.21.15.37:0/110089496 shutdown_connections 2024-04-26T00:20:09.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.403+0000 7f67f9d22640 1 -- 172.21.15.37:0/110089496 >> 172.21.15.37:0/110089496 conn(0x7f67f40a2c50 msgr2=0x7f67f4096110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:09.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.403+0000 7f67f9d22640 1 -- 172.21.15.37:0/110089496 shutdown_connections 2024-04-26T00:20:09.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.403+0000 7f67f9d22640 1 -- 172.21.15.37:0/110089496 wait complete. 2024-04-26T00:20:09.404 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:20:09.405 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-04-26T00:20:09.405 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-26T00:20:09.405 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-04-26T00:20:09.405 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-04-26T00:20:09.405 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-26T00:20:09.405 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T00:20:09.431 INFO:tasks.ceph.mon.b:Stopped 2024-04-26T00:20:09.431 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-26T00:20:09.431 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:20:09.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.479+0000 7f6e983eb640 10 monclient: get_monmap_and_config 2024-04-26T00:20:09.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.479+0000 7f6e983eb640 10 monclient: build_initial_monmap 2024-04-26T00:20:09.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.479+0000 7f6e983eb640 10 monclient: monmap: 2024-04-26T00:20:09.463 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:09.463 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:09.464 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:09.487403+0000 2024-04-26T00:20:09.464 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:09.487403+0000 2024-04-26T00:20:09.464 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:09.464 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:09.464 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:09.464 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:09.464 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:09.464 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:09.464 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:09.464 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:09.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.483+0000 7f6e983eb640 1 Processor -- start 2024-04-26T00:20:09.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.483+0000 7f6e983eb640 1 -- start start 2024-04-26T00:20:09.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.483+0000 7f6e983eb640 10 monclient: init 2024-04-26T00:20:09.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.483+0000 7f6e983eb640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:09.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.483+0000 7f6e983eb640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-04-26T00:20:09.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.483+0000 7f6e983eb640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e900721b0 0x7f6e901339c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:09.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.483+0000 7f6e983eb640 10 monclient(hunting): picked mon.noname-b con 0x7f6e900721b0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:09.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.483+0000 7f6e983eb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6e90072590 addr v1:172.21.15.37:6789/0 2024-04-26T00:20:09.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.483+0000 7f6e983eb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6e90072990 addr v1:172.21.15.45:6789/0 2024-04-26T00:20:09.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.483+0000 7f6e983eb640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e9006f390 con 0x7f6e90072590 2024-04-26T00:20:09.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.483+0000 7f6e983eb640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e900a23b0 con 0x7f6e90072990 2024-04-26T00:20:09.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.483+0000 7f6e983eb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:09.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.483+0000 7f6e983eb640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f6e9006efb0 con 0x7f6e900721b0 2024-04-26T00:20:09.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.483+0000 7f6e983eb640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:09.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.483+0000 7f6e983eb640 10 monclient(hunting): authenticate will time out at 990.792744s 2024-04-26T00:20:09.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.483+0000 7f6e96187640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e900721b0 0x7f6e901339c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:09.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.483+0000 7f6e95986640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f6e90072590 0x7f6e900a1590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:55882/0 (socket says 172.21.15.37:55882) 2024-04-26T00:20:09.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.483+0000 7f6e95986640 1 -- 172.21.15.37:0/1765845390 learned_addr learned my addr 172.21.15.37:0/1765845390 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:20:09.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.483+0000 7f6e96988640 1 -- 172.21.15.37:0/1765845390 >> v1:172.21.15.45:6789/0 conn(0x7f6e90072990 legacy=0x7f6e900a1ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:20:09.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.483+0000 7f6e96187640 10 monclient(hunting): get_auth_request con 0x7f6e900721b0 auth_method 0 2024-04-26T00:20:09.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.483+0000 7f6e96187640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:09.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.483+0000 7f6e96187640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:09.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.483+0000 7f6e96187640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:09.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.483+0000 7f6e96187640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:09.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.483+0000 7f6e96187640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:09.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.483+0000 7f6e96187640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:09.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.483+0000 7f6e96187640 10 monclient(hunting): handle_auth_done global_id 4358 payload 274 2024-04-26T00:20:09.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.483+0000 7f6e96187640 1 -- 172.21.15.37:0/1765845390 >> v1:172.21.15.45:6789/0 conn(0x7f6e90072990 legacy=0x7f6e900a1ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:20:09.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.483+0000 7f6e96187640 1 -- 172.21.15.37:0/1765845390 >> v1:172.21.15.37:6789/0 conn(0x7f6e90072590 legacy=0x7f6e900a1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:09.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.483+0000 7f6e96187640 10 monclient: _finish_hunting 0 2024-04-26T00:20:09.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.483+0000 7f6e96187640 1 monclient: found mon.noname-b 2024-04-26T00:20:09.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.483+0000 7f6e96187640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:09.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.483+0000 7f6e96187640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:20:09.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.483+0000 7f6e96187640 1 -- 172.21.15.37:0/1765845390 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e900701b0 con 0x7f6e900721b0 2024-04-26T00:20:09.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e96187640 1 -- 172.21.15.37:0/1765845390 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e900721b0 msgr2=0x7f6e901339c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:09.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e96187640 1 --2- 172.21.15.37:0/1765845390 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e900721b0 0x7f6e901339c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c0097f0 tx=0x7f6e8c01d250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab92caa73624c674 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:09.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e95185640 1 -- 172.21.15.37:0/1765845390 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e8c0b1070 con 0x7f6e900721b0 2024-04-26T00:20:09.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e95185640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:09.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e95185640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:20:09.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e95185640 10 monclient: dump: 2024-04-26T00:20:09.471 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:09.471 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:09.471 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.471 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.471 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:09.471 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:09.471 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:09.471 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:09.471 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:09.471 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:09.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e95185640 10 monclient: _finish_auth 0 2024-04-26T00:20:09.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e95185640 10 monclient: _check_auth_tickets 2024-04-26T00:20:09.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e95185640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:09.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e95185640 1 -- 172.21.15.37:0/1765845390 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e8c0ac600 con 0x7f6e900721b0 2024-04-26T00:20:09.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e95185640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:09.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e95185640 1 -- 172.21.15.37:0/1765845390 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e8c0ac9d0 con 0x7f6e900721b0 2024-04-26T00:20:09.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e983eb640 5 monclient: authenticate success, global_id 4358 2024-04-26T00:20:09.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e983eb640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:09.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e983eb640 4 set_mon_vals no callback set 2024-04-26T00:20:09.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e95185640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:09.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e95185640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:09.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e95185640 10 monclient: dump: 2024-04-26T00:20:09.472 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:09.473 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:09.473 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.473 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.473 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:09.473 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:09.473 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:09.473 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:09.473 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:09.473 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:09.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e983eb640 10 monclient: shutdown 2024-04-26T00:20:09.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e983eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:09.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e983eb640 1 -- 172.21.15.37:0/1765845390 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e900721b0 msgr2=0x7f6e901339c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:09.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e983eb640 1 --2- 172.21.15.37:0/1765845390 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e900721b0 0x7f6e901339c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c0097f0 tx=0x7f6e8c01d250 comp rx=0 tx=0).stop 2024-04-26T00:20:09.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e983eb640 1 -- 172.21.15.37:0/1765845390 shutdown_connections 2024-04-26T00:20:09.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e983eb640 1 --2- 172.21.15.37:0/1765845390 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e900721b0 0x7f6e901339c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:09.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e983eb640 1 -- 172.21.15.37:0/1765845390 >> 172.21.15.37:0/1765845390 conn(0x7f6e900a6180 msgr2=0x7f6e900a2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:09.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e983eb640 1 -- 172.21.15.37:0/1765845390 shutdown_connections 2024-04-26T00:20:09.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e983eb640 1 -- 172.21.15.37:0/1765845390 wait complete. 2024-04-26T00:20:09.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e983eb640 10 monclient: build_initial_monmap 2024-04-26T00:20:09.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e983eb640 10 monclient: monmap: 2024-04-26T00:20:09.474 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:09.474 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:09.474 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:09.474 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:09.474 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:09.474 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:09.474 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:09.474 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:09.474 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:09.475 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:09.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e983eb640 1 Processor -- start 2024-04-26T00:20:09.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e983eb640 1 -- start start 2024-04-26T00:20:09.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e983eb640 10 monclient: init 2024-04-26T00:20:09.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e983eb640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:09.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e983eb640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:20:09.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e983eb640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e900721b0 0x7f6e90134390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:09.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e983eb640 10 monclient(hunting): picked mon.noname-c con 0x7f6e900721b0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:09.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e983eb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e90072590 0x7f6e901348d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:09.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e983eb640 10 monclient(hunting): picked mon.noname-b con 0x7f6e90072590 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:09.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e983eb640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6e90072990 0x7f6e90134e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:09.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e983eb640 10 monclient(hunting): picked mon.noname-a con 0x7f6e90072990 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:09.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e983eb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:09.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e983eb640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f6e9013e8d0 con 0x7f6e90072990 2024-04-26T00:20:09.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e983eb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:09.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e983eb640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6e9013e750 con 0x7f6e90072590 2024-04-26T00:20:09.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e983eb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:09.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e983eb640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f6e9013ea50 con 0x7f6e900721b0 2024-04-26T00:20:09.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e983eb640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:09.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e983eb640 10 monclient(hunting): authenticate will time out at 990.799525s 2024-04-26T00:20:09.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e96988640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6e90072990 0x7f6e90134e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:09.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e95986640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e90072590 msgr2=0x7f6e901348d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:20:09.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e96187640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e900721b0 0x7f6e90134390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:09.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e95986640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e90072590 0x7f6e901348d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:20:09.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e96988640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6e90072990 0x7f6e90134e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:45012/0 (socket says 172.21.15.37:45012) 2024-04-26T00:20:09.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e96988640 1 -- 172.21.15.37:0/743788398 learned_addr learned my addr 172.21.15.37:0/743788398 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:09.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e96988640 10 monclient(hunting): get_auth_request con 0x7f6e90072990 auth_method 0 2024-04-26T00:20:09.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e96988640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:09.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e96988640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:09.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e96988640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:09.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e96187640 10 monclient(hunting): get_auth_request con 0x7f6e900721b0 auth_method 0 2024-04-26T00:20:09.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e96187640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:09.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e96187640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:09.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e96187640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:09.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e96187640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:09.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e96187640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:09.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e96187640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:09.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e96988640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:09.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e96988640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:09.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.487+0000 7f6e96988640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:09.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e96988640 10 monclient(hunting): handle_auth_done global_id 4365 payload 824 2024-04-26T00:20:09.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e96988640 1 -- 172.21.15.37:0/743788398 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e900721b0 msgr2=0x7f6e90134390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:09.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e96988640 1 --2- 172.21.15.37:0/743788398 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e900721b0 0x7f6e90134390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:09.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e96988640 1 -- 172.21.15.37:0/743788398 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e90072590 msgr2=0x7f6e901348d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:20:09.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e96988640 1 --2- 172.21.15.37:0/743788398 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e90072590 0x7f6e901348d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:09.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e96988640 10 monclient: _finish_hunting 0 2024-04-26T00:20:09.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e96988640 1 monclient: found mon.noname-a 2024-04-26T00:20:09.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e96988640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:09.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e96988640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:09.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e96988640 1 -- 172.21.15.37:0/743788398 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e90135ea0 con 0x7f6e90072990 2024-04-26T00:20:09.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e96187640 1 --2- 172.21.15.37:0/743788398 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e900721b0 0x7f6e90134390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:09.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e96988640 1 -- 172.21.15.37:0/743788398 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6e90072990 msgr2=0x7f6e90134e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:09.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e96988640 1 --2- 172.21.15.37:0/743788398 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6e90072990 0x7f6e90134e10 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6e8800b690 tx=0x7f6e8800bb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:09.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e86ffd640 1 -- 172.21.15.37:0/743788398 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e8800cab0 con 0x7f6e90072990 2024-04-26T00:20:09.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:09.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e86ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:09.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e86ffd640 10 monclient: dump: 2024-04-26T00:20:09.482 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:09.482 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:09.482 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.482 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.482 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:09.482 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:09.482 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:09.482 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:09.482 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:09.483 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:09.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e86ffd640 10 monclient: _finish_auth 0 2024-04-26T00:20:09.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e86ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:20:09.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:09.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e86ffd640 1 -- 172.21.15.37:0/743788398 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e88010070 con 0x7f6e90072990 2024-04-26T00:20:09.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e86ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:09.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e86ffd640 1 -- 172.21.15.37:0/743788398 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e88015470 con 0x7f6e90072990 2024-04-26T00:20:09.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e983eb640 5 monclient: authenticate success, global_id 4365 2024-04-26T00:20:09.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e983eb640 10 monclient: _renew_subs 2024-04-26T00:20:09.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e983eb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:09.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e983eb640 1 -- 172.21.15.37:0/743788398 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e9009fe80 con 0x7f6e90072990 2024-04-26T00:20:09.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:09.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e86ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:09.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e94984640 4 set_mon_vals no callback set 2024-04-26T00:20:09.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e86ffd640 10 monclient: dump: 2024-04-26T00:20:09.484 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:09.484 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:09.484 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.484 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.484 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:09.484 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:09.484 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:09.484 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:09.484 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:09.484 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:09.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e983eb640 10 monclient: _renew_subs 2024-04-26T00:20:09.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e983eb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:09.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e983eb640 1 -- 172.21.15.37:0/743788398 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e90138200 con 0x7f6e90072990 2024-04-26T00:20:09.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e86ffd640 1 -- 172.21.15.37:0/743788398 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f6e88015690 con 0x7f6e90072990 2024-04-26T00:20:09.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e86ffd640 1 --2- 172.21.15.37:0/743788398 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6e74035e80 0x7f6e74038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:09.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e86ffd640 1 -- 172.21.15.37:0/743788398 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 3858+0+0 (secure 0 0 0) 0x7f6e880488b0 con 0x7f6e90072990 2024-04-26T00:20:09.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e983eb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:09.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e983eb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:09.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e983eb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:09.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.491+0000 7f6e983eb640 1 -- 172.21.15.37:0/743788398 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e9004f580 con 0x7f6e90072990 2024-04-26T00:20:09.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.495+0000 7f6e96187640 1 --2- 172.21.15.37:0/743788398 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6e74035e80 0x7f6e74038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:09.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.495+0000 7f6e96187640 10 monclient: get_auth_request con 0x7f6e74035e80 auth_method 0 2024-04-26T00:20:09.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.495+0000 7f6e96187640 1 --2- 172.21.15.37:0/743788398 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6e74035e80 0x7f6e74038340 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c0ac000 tx=0x7f6e8c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:09.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.499+0000 7f6e86ffd640 1 -- 172.21.15.37:0/743788398 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6e88014800 con 0x7f6e90072990 2024-04-26T00:20:09.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.499+0000 7f6e86ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:09.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.499+0000 7f6e86ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:09.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.575+0000 7fd77f1c6640 10 monclient: get_monmap_and_config 2024-04-26T00:20:09.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.575+0000 7fd77f1c6640 10 monclient: build_initial_monmap 2024-04-26T00:20:09.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.575+0000 7fd77f1c6640 10 monclient: monmap: 2024-04-26T00:20:09.558 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:09.558 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:09.558 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:09.581394+0000 2024-04-26T00:20:09.558 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:09.581394+0000 2024-04-26T00:20:09.558 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:09.558 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:09.558 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:09.558 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:09.558 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:09.558 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:09.558 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:09.558 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:09.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.575+0000 7fd77f1c6640 1 Processor -- start 2024-04-26T00:20:09.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.575+0000 7fd77f1c6640 1 -- start start 2024-04-26T00:20:09.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.575+0000 7fd77f1c6640 10 monclient: init 2024-04-26T00:20:09.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.575+0000 7fd77f1c6640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:09.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.575+0000 7fd77f1c6640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-04-26T00:20:09.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.575+0000 7fd77f1c6640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd778136e10 0x7fd77809c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:09.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.575+0000 7fd77f1c6640 10 monclient(hunting): picked mon.noname-b con 0x7fd778136e10 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:09.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.575+0000 7fd77f1c6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd7781371f0 addr v1:172.21.15.45:6789/0 2024-04-26T00:20:09.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.575+0000 7fd77f1c6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd77813c340 addr v1:172.21.15.37:6789/0 2024-04-26T00:20:09.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.575+0000 7fd77f1c6640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd77809b680 con 0x7fd77813c340 2024-04-26T00:20:09.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.575+0000 7fd77f1c6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd77809bbd0 con 0x7fd7781371f0 2024-04-26T00:20:09.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.575+0000 7fd77f1c6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:09.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.575+0000 7fd77f1c6640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd77809b8f0 con 0x7fd778136e10 2024-04-26T00:20:09.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.575+0000 7fd77f1c6640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:09.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.575+0000 7fd77f1c6640 10 monclient(hunting): authenticate will time out at 990.887260s 2024-04-26T00:20:09.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.575+0000 7fd77d763640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fd77813c340 0x7fd77813e730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:55888/0 (socket says 172.21.15.37:55888) 2024-04-26T00:20:09.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.575+0000 7fd777fff640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7fd7781371f0 legacy=0x7fd7781375d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:20:09.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.575+0000 7fd77d763640 1 -- 172.21.15.37:0/3098786855 learned_addr learned my addr 172.21.15.37:0/3098786855 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:20:09.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.575+0000 7fd77cf62640 1 --2- 172.21.15.37:0/3098786855 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd778136e10 0x7fd77809c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:09.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.575+0000 7fd77cf62640 10 monclient(hunting): get_auth_request con 0x7fd778136e10 auth_method 0 2024-04-26T00:20:09.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.575+0000 7fd77cf62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:09.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.575+0000 7fd77cf62640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:09.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.575+0000 7fd77cf62640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:09.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.575+0000 7fd77cf62640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:09.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.575+0000 7fd77cf62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:09.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.575+0000 7fd77cf62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:09.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.575+0000 7fd77cf62640 10 monclient(hunting): handle_auth_done global_id 4364 payload 274 2024-04-26T00:20:09.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.575+0000 7fd77cf62640 1 -- 172.21.15.37:0/3098786855 >> v1:172.21.15.45:6789/0 conn(0x7fd7781371f0 legacy=0x7fd7781375d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:20:09.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.575+0000 7fd77cf62640 1 -- 172.21.15.37:0/3098786855 >> v1:172.21.15.37:6789/0 conn(0x7fd77813c340 legacy=0x7fd77813e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:09.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.575+0000 7fd77cf62640 10 monclient: _finish_hunting 0 2024-04-26T00:20:09.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.575+0000 7fd77cf62640 1 monclient: found mon.noname-b 2024-04-26T00:20:09.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.575+0000 7fd77cf62640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:09.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.575+0000 7fd77cf62640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:20:09.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.575+0000 7fd77cf62640 1 -- 172.21.15.37:0/3098786855 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7780a5d90 con 0x7fd778136e10 2024-04-26T00:20:09.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.579+0000 7fd77cf62640 1 -- 172.21.15.37:0/3098786855 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd778136e10 msgr2=0x7fd77809c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:09.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.579+0000 7fd77cf62640 1 --2- 172.21.15.37:0/3098786855 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd778136e10 0x7fd77809c5b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd76c009820 tx=0x7fd76c01c300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7e2b9ee8cb28a6f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:09.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd7777fe640 1 -- 172.21.15.37:0/3098786855 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd76c0b0070 con 0x7fd778136e10 2024-04-26T00:20:09.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd7777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:09.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd7777fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:20:09.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd7777fe640 10 monclient: dump: 2024-04-26T00:20:09.565 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:09.565 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:09.565 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.565 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.565 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:09.565 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:09.565 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:09.565 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:09.565 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:09.565 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:09.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd7777fe640 10 monclient: _finish_auth 0 2024-04-26T00:20:09.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd7777fe640 10 monclient: _check_auth_tickets 2024-04-26T00:20:09.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd7777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:09.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd7777fe640 1 -- 172.21.15.37:0/3098786855 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd76c0ab600 con 0x7fd778136e10 2024-04-26T00:20:09.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77f1c6640 5 monclient: authenticate success, global_id 4364 2024-04-26T00:20:09.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd7777fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:09.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd7777fe640 1 -- 172.21.15.37:0/3098786855 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd76c0ab9d0 con 0x7fd778136e10 2024-04-26T00:20:09.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77f1c6640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:09.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77f1c6640 4 set_mon_vals no callback set 2024-04-26T00:20:09.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd7777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:09.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd7777fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:09.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd7777fe640 10 monclient: dump: 2024-04-26T00:20:09.566 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:09.566 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:09.566 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.566 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.567 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:09.567 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:09.567 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:09.567 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:09.567 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:09.567 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:09.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77f1c6640 10 monclient: shutdown 2024-04-26T00:20:09.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77f1c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:09.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77f1c6640 1 -- 172.21.15.37:0/3098786855 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd778136e10 msgr2=0x7fd77809c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:09.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77f1c6640 1 --2- 172.21.15.37:0/3098786855 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd778136e10 0x7fd77809c5b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd76c009820 tx=0x7fd76c01c300 comp rx=0 tx=0).stop 2024-04-26T00:20:09.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77f1c6640 1 -- 172.21.15.37:0/3098786855 shutdown_connections 2024-04-26T00:20:09.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77f1c6640 1 --2- 172.21.15.37:0/3098786855 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd778136e10 0x7fd77809c5b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:09.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77f1c6640 1 -- 172.21.15.37:0/3098786855 >> 172.21.15.37:0/3098786855 conn(0x7fd7780a64e0 msgr2=0x7fd7780a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:09.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77f1c6640 1 -- 172.21.15.37:0/3098786855 shutdown_connections 2024-04-26T00:20:09.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77f1c6640 1 -- 172.21.15.37:0/3098786855 wait complete. 2024-04-26T00:20:09.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77f1c6640 10 monclient: build_initial_monmap 2024-04-26T00:20:09.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77f1c6640 10 monclient: monmap: 2024-04-26T00:20:09.571 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:09.571 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:09.571 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:09.572 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:09.572 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:09.572 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:09.572 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:09.572 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:09.572 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:09.572 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:09.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77f1c6640 1 Processor -- start 2024-04-26T00:20:09.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77f1c6640 1 -- start start 2024-04-26T00:20:09.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77f1c6640 10 monclient: init 2024-04-26T00:20:09.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77f1c6640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:09.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77f1c6640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:20:09.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77f1c6640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd778136e10 0x7fd7781cef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:09.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77f1c6640 10 monclient(hunting): picked mon.noname-a con 0x7fd778136e10 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:09.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77f1c6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7781371f0 0x7fd778135e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:09.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77f1c6640 10 monclient(hunting): picked mon.noname-b con 0x7fd7781371f0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:09.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77f1c6640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd77813c340 0x7fd778137e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:09.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77f1c6640 10 monclient(hunting): picked mon.noname-c con 0x7fd77813c340 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:09.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77f1c6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:09.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77f1c6640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd778142a00 con 0x7fd778136e10 2024-04-26T00:20:09.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77f1c6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:09.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77f1c6640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd778142880 con 0x7fd7781371f0 2024-04-26T00:20:09.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77f1c6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:09.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77f1c6640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd778142b80 con 0x7fd77813c340 2024-04-26T00:20:09.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77f1c6640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:09.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77f1c6640 10 monclient(hunting): authenticate will time out at 990.894207s 2024-04-26T00:20:09.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77cf62640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd778136e10 0x7fd7781cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:09.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77d763640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd77813c340 0x7fd778137e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:09.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77cf62640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd778136e10 0x7fd7781cef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:45024/0 (socket says 172.21.15.37:45024) 2024-04-26T00:20:09.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77cf62640 1 -- 172.21.15.37:0/3118955386 learned_addr learned my addr 172.21.15.37:0/3118955386 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:09.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77cf62640 10 monclient(hunting): get_auth_request con 0x7fd778136e10 auth_method 0 2024-04-26T00:20:09.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77cf62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:09.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77cf62640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:09.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77cf62640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:09.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd777fff640 1 -- 172.21.15.37:0/3118955386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7781371f0 msgr2=0x7fd778135e20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:20:09.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77d763640 10 monclient(hunting): get_auth_request con 0x7fd77813c340 auth_method 0 2024-04-26T00:20:09.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd777fff640 1 --2- 172.21.15.37:0/3118955386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7781371f0 0x7fd778135e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:20:09.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77d763640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:09.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77d763640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:09.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77d763640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:09.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77cf62640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:09.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77cf62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:09.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77cf62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:09.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77d763640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:09.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77d763640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:09.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77d763640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:09.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77d763640 10 monclient(hunting): handle_auth_done global_id 4367 payload 824 2024-04-26T00:20:09.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77d763640 1 -- 172.21.15.37:0/3118955386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7781371f0 msgr2=0x7fd778135e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:20:09.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77d763640 1 --2- 172.21.15.37:0/3118955386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7781371f0 0x7fd778135e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:09.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77d763640 1 -- 172.21.15.37:0/3118955386 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd778136e10 msgr2=0x7fd7781cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:09.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77d763640 1 --2- 172.21.15.37:0/3118955386 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd778136e10 0x7fd7781cef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:09.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77d763640 10 monclient: _finish_hunting 0 2024-04-26T00:20:09.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77d763640 1 monclient: found mon.noname-c 2024-04-26T00:20:09.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77d763640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:09.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77d763640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:20:09.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77d763640 1 -- 172.21.15.37:0/3118955386 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd778139700 con 0x7fd77813c340 2024-04-26T00:20:09.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.583+0000 7fd77cf62640 1 --2- 172.21.15.37:0/3118955386 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd778136e10 0x7fd7781cef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:09.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.587+0000 7fd77d763640 1 -- 172.21.15.37:0/3118955386 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd77813c340 msgr2=0x7fd778137e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:09.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.587+0000 7fd77d763640 1 --2- 172.21.15.37:0/3118955386 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd77813c340 0x7fd778137e20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd7780a95b0 tx=0x7fd77000ce90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:09.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.587+0000 7fd7757fa640 1 -- 172.21.15.37:0/3118955386 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd77000ad50 con 0x7fd77813c340 2024-04-26T00:20:09.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.587+0000 7fd7757fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:09.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.587+0000 7fd7757fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:20:09.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.587+0000 7fd7757fa640 10 monclient: dump: 2024-04-26T00:20:09.576 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:09.576 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:09.576 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.576 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.576 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:09.576 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:09.576 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:09.576 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:09.576 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:09.577 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:09.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.587+0000 7fd7757fa640 10 monclient: _finish_auth 0 2024-04-26T00:20:09.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.587+0000 7fd7757fa640 10 monclient: _check_auth_tickets 2024-04-26T00:20:09.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.587+0000 7fd7757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:09.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.587+0000 7fd7757fa640 1 -- 172.21.15.37:0/3118955386 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd770012400 con 0x7fd77813c340 2024-04-26T00:20:09.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.587+0000 7fd77f1c6640 5 monclient: authenticate success, global_id 4367 2024-04-26T00:20:09.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.587+0000 7fd77f1c6640 10 monclient: _renew_subs 2024-04-26T00:20:09.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.587+0000 7fd77f1c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:09.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.587+0000 7fd77f1c6640 1 -- 172.21.15.37:0/3118955386 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7780a84a0 con 0x7fd77813c340 2024-04-26T00:20:09.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.587+0000 7fd7757fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:09.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.587+0000 7fd7757fa640 1 -- 172.21.15.37:0/3118955386 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7700127f0 con 0x7fd77813c340 2024-04-26T00:20:09.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.587+0000 7fd7757fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:09.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.587+0000 7fd7757fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:09.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.587+0000 7fd7757fa640 10 monclient: dump: 2024-04-26T00:20:09.577 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:09.578 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:09.578 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.578 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:09.578 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:09.578 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:09.578 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:09.578 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:09.578 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:09.578 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:09.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.587+0000 7fd77f1c6640 10 monclient: _renew_subs 2024-04-26T00:20:09.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.587+0000 7fd776ffd640 4 set_mon_vals no callback set 2024-04-26T00:20:09.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.587+0000 7fd77f1c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:09.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.587+0000 7fd77f1c6640 1 -- 172.21.15.37:0/3118955386 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd778136360 con 0x7fd77813c340 2024-04-26T00:20:09.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.587+0000 7fd7757fa640 1 -- 172.21.15.37:0/3118955386 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd7700129d0 con 0x7fd77813c340 2024-04-26T00:20:09.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.587+0000 7fd77f1c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:09.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.587+0000 7fd77f1c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:09.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.587+0000 7fd77f1c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:09.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.587+0000 7fd7757fa640 1 --2- 172.21.15.37:0/3118955386 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd75c035e60 0x7fd75c038320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:09.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.587+0000 7fd77f1c6640 1 -- 172.21.15.37:0/3118955386 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd778075910 con 0x7fd77813c340 2024-04-26T00:20:09.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.587+0000 7fd7757fa640 1 -- 172.21.15.37:0/3118955386 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 3858+0+0 (secure 0 0 0) 0x7fd770048f80 con 0x7fd77813c340 2024-04-26T00:20:09.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.587+0000 7fd77cf62640 1 --2- 172.21.15.37:0/3118955386 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd75c035e60 0x7fd75c038320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:09.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.587+0000 7fd77cf62640 10 monclient: get_auth_request con 0x7fd75c035e60 auth_method 0 2024-04-26T00:20:09.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.591+0000 7fd77cf62640 1 --2- 172.21.15.37:0/3118955386 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd75c035e60 0x7fd75c038320 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd76c01cba0 tx=0x7fd76c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:09.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.591+0000 7fd7757fa640 1 -- 172.21.15.37:0/3118955386 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fd770047d10 con 0x7fd77813c340 2024-04-26T00:20:09.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.591+0000 7fd7757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:09.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.591+0000 7fd7757fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:09.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.663+0000 7f6e983eb640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}] 2024-04-26T00:20:09.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.663+0000 7f6e983eb640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}] 2024-04-26T00:20:09.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.663+0000 7f6e983eb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:09.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.663+0000 7f6e983eb640 1 -- 172.21.15.37:0/743788398 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} v 0) -- 0x7f6e9006c3b0 con 0x7f6e90072990 2024-04-26T00:20:09.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.807+0000 7fd77f1c6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:20:09.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.807+0000 7fd77f1c6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:20:09.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.807+0000 7fd77f1c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:09.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.807+0000 7fd77f1c6640 1 -- 172.21.15.37:0/3118955386 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd778136530 con 0x7fd77813c340 2024-04-26T00:20:09.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.807+0000 7fd7757fa640 1 -- 172.21.15.37:0/3118955386 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fd77001b920 con 0x7fd77813c340 2024-04-26T00:20:09.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.807+0000 7fd7757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:20:09.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.807+0000 7fd7757fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:09.792 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:20:09.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.811+0000 7fd77f1c6640 1 -- 172.21.15.37:0/3118955386 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd75c035e60 msgr2=0x7fd75c038320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:09.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.811+0000 7fd77f1c6640 1 --2- 172.21.15.37:0/3118955386 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd75c035e60 0x7fd75c038320 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd76c01cba0 tx=0x7fd76c008040 comp rx=0 tx=0).stop 2024-04-26T00:20:09.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.811+0000 7fd77f1c6640 10 monclient: shutdown 2024-04-26T00:20:09.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.811+0000 7fd77f1c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:09.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.811+0000 7fd77f1c6640 1 -- 172.21.15.37:0/3118955386 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd77813c340 msgr2=0x7fd778137e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:09.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.811+0000 7fd77f1c6640 1 --2- 172.21.15.37:0/3118955386 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd77813c340 0x7fd778137e20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd7780a95b0 tx=0x7fd77000ce90 comp rx=0 tx=0).stop 2024-04-26T00:20:09.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.811+0000 7fd77f1c6640 1 -- 172.21.15.37:0/3118955386 shutdown_connections 2024-04-26T00:20:09.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.811+0000 7fd77f1c6640 1 --2- 172.21.15.37:0/3118955386 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd75c035e60 0x7fd75c038320 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:09.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.811+0000 7fd77f1c6640 1 --2- 172.21.15.37:0/3118955386 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd77813c340 0x7fd778137e20 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:09.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.811+0000 7fd77f1c6640 1 --2- 172.21.15.37:0/3118955386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7781371f0 0x7fd778135e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:09.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.811+0000 7fd77f1c6640 1 --2- 172.21.15.37:0/3118955386 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd778136e10 0x7fd7781cef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:09.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.811+0000 7fd77f1c6640 1 -- 172.21.15.37:0/3118955386 >> 172.21.15.37:0/3118955386 conn(0x7fd7780a64e0 msgr2=0x7fd7780a7490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:09.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.811+0000 7fd77f1c6640 1 -- 172.21.15.37:0/3118955386 shutdown_connections 2024-04-26T00:20:09.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:09.811+0000 7fd77f1c6640 1 -- 172.21.15.37:0/3118955386 wait complete. 2024-04-26T00:20:12.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.491+0000 7f6e85ffb640 10 monclient: tick 2024-04-26T00:20:12.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.491+0000 7f6e85ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:20:12.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.491+0000 7f6e85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:12.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.491+0000 7f6e85ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:12.809 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:20:12.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d87cc640 10 monclient: get_monmap_and_config 2024-04-26T00:20:12.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d87cc640 10 monclient: build_initial_monmap 2024-04-26T00:20:12.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d87cc640 10 monclient: monmap: 2024-04-26T00:20:12.942 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:12.943 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:12.943 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:12.966238+0000 2024-04-26T00:20:12.943 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:12.966238+0000 2024-04-26T00:20:12.943 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:12.943 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:12.943 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:12.943 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:12.943 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:12.943 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:12.943 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:12.943 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:12.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d87cc640 1 Processor -- start 2024-04-26T00:20:12.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d87cc640 1 -- start start 2024-04-26T00:20:12.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d87cc640 10 monclient: init 2024-04-26T00:20:12.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d87cc640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:12.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d87cc640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-04-26T00:20:12.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d87cc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc1d00a2fe0 0x7fc1d00724f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:12.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d87cc640 10 monclient(hunting): picked mon.noname-b con 0x7fc1d00a2fe0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:12.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d87cc640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc1d0072a30 0x7fc1d00a13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:12.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d87cc640 10 monclient(hunting): picked mon.noname-a con 0x7fc1d0072a30 addr v2:172.21.15.37:3300/0 2024-04-26T00:20:12.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d87cc640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc1d00a1910 0x7fc1d00a1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:12.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d87cc640 10 monclient(hunting): picked mon.noname-c con 0x7fc1d00a1910 addr v2:172.21.15.45:3300/0 2024-04-26T00:20:12.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d87cc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:12.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d87cc640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fc1d006efa0 con 0x7fc1d0072a30 2024-04-26T00:20:12.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d87cc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:12.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d87cc640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc1d00a33c0 con 0x7fc1d00a1910 2024-04-26T00:20:12.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d87cc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:12.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d87cc640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc1d0137480 con 0x7fc1d00a2fe0 2024-04-26T00:20:12.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d87cc640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:12.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d87cc640 10 monclient(hunting): authenticate will time out at 994.272111s 2024-04-26T00:20:12.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d6d69640 1 -- >> v2:172.21.15.45:3300/0 conn(0x7fc1d00a1910 msgr2=0x7fc1d00a1da0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:20:12.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d5d67640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc1d0072a30 0x7fc1d00a13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:12.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d6d69640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc1d00a1910 0x7fc1d00a1da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:20:12.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d6568640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc1d00a2fe0 0x7fc1d00724f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:12.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d5d67640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc1d0072a30 0x7fc1d00a13d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:56700/0 (socket says 172.21.15.37:56700) 2024-04-26T00:20:12.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d5d67640 1 -- 172.21.15.37:0/362893049 learned_addr learned my addr 172.21.15.37:0/362893049 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:12.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d5d67640 10 monclient(hunting): get_auth_request con 0x7fc1d0072a30 auth_method 0 2024-04-26T00:20:12.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d5d67640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:12.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d5d67640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:12.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d5d67640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:12.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d6568640 10 monclient(hunting): get_auth_request con 0x7fc1d00a2fe0 auth_method 0 2024-04-26T00:20:12.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d6568640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:12.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d6568640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:12.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d6568640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:12.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d5d67640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:12.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d5d67640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:12.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d5d67640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:12.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d6568640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:12.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d6568640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:12.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d6568640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:12.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d6568640 10 monclient(hunting): handle_auth_done global_id 4370 payload 274 2024-04-26T00:20:12.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d6568640 1 -- 172.21.15.37:0/362893049 >> v2:172.21.15.45:3300/0 conn(0x7fc1d00a1910 msgr2=0x7fc1d00a1da0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:20:12.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d6568640 1 --2- 172.21.15.37:0/362893049 >> v2:172.21.15.45:3300/0 conn(0x7fc1d00a1910 0x7fc1d00a1da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:12.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d6568640 1 -- 172.21.15.37:0/362893049 >> v2:172.21.15.37:3300/0 conn(0x7fc1d0072a30 msgr2=0x7fc1d00a13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:12.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d6568640 1 --2- 172.21.15.37:0/362893049 >> v2:172.21.15.37:3300/0 conn(0x7fc1d0072a30 0x7fc1d00a13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:12.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d6568640 10 monclient: _finish_hunting 0 2024-04-26T00:20:12.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d6568640 1 monclient: found mon.noname-b 2024-04-26T00:20:12.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d6568640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:12.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d6568640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:20:12.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d6568640 1 -- 172.21.15.37:0/362893049 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1d006e700 con 0x7fc1d00a2fe0 2024-04-26T00:20:12.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.963+0000 7fc1d5d67640 1 --2- 172.21.15.37:0/362893049 >> v2:172.21.15.37:3300/0 conn(0x7fc1d0072a30 0x7fc1d00a13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:12.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d6568640 1 -- 172.21.15.37:0/362893049 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc1d00a2fe0 msgr2=0x7fc1d00724f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:12.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d6568640 1 --2- 172.21.15.37:0/362893049 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc1d00a2fe0 0x7fc1d00724f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc1d0075c70 tx=0x7fc1cc01c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d32b90a3d59f2481 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:12.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d5566640 1 -- 172.21.15.37:0/362893049 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1cc0b0070 con 0x7fc1d00a2fe0 2024-04-26T00:20:12.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d5566640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:12.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d5566640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:20:12.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d5566640 10 monclient: dump: 2024-04-26T00:20:12.952 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:12.952 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:12.952 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:12.953 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:12.953 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:12.953 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:12.953 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:12.953 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:12.953 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:12.953 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:12.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d5566640 10 monclient: _finish_auth 0 2024-04-26T00:20:12.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d5566640 10 monclient: _check_auth_tickets 2024-04-26T00:20:12.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d5566640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:12.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d5566640 1 -- 172.21.15.37:0/362893049 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1cc0ab600 con 0x7fc1d00a2fe0 2024-04-26T00:20:12.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d5566640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:12.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d5566640 1 -- 172.21.15.37:0/362893049 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1cc0ab9d0 con 0x7fc1d00a2fe0 2024-04-26T00:20:12.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d87cc640 5 monclient: authenticate success, global_id 4370 2024-04-26T00:20:12.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d87cc640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:12.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d87cc640 4 set_mon_vals no callback set 2024-04-26T00:20:12.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d5566640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:12.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d5566640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:12.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d5566640 10 monclient: dump: 2024-04-26T00:20:12.954 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:12.954 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:12.954 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:12.954 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:12.954 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:12.954 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:12.954 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:12.954 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:12.954 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:12.955 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:12.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d87cc640 10 monclient: shutdown 2024-04-26T00:20:12.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d87cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:12.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d87cc640 1 -- 172.21.15.37:0/362893049 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc1d00a2fe0 msgr2=0x7fc1d00724f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:12.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d87cc640 1 --2- 172.21.15.37:0/362893049 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc1d00a2fe0 0x7fc1d00724f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc1d0075c70 tx=0x7fc1cc01c360 comp rx=0 tx=0).stop 2024-04-26T00:20:12.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d87cc640 1 -- 172.21.15.37:0/362893049 shutdown_connections 2024-04-26T00:20:12.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d87cc640 1 --2- 172.21.15.37:0/362893049 >> v2:172.21.15.45:3300/0 conn(0x7fc1d00a1910 0x7fc1d00a1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:12.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d87cc640 1 --2- 172.21.15.37:0/362893049 >> v2:172.21.15.37:3300/0 conn(0x7fc1d0072a30 0x7fc1d00a13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:12.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d87cc640 1 --2- 172.21.15.37:0/362893049 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc1d00a2fe0 0x7fc1d00724f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:12.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d87cc640 1 -- 172.21.15.37:0/362893049 >> 172.21.15.37:0/362893049 conn(0x7fc1d00a24c0 msgr2=0x7fc1d00a28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:12.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d87cc640 1 -- 172.21.15.37:0/362893049 shutdown_connections 2024-04-26T00:20:12.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d87cc640 1 -- 172.21.15.37:0/362893049 wait complete. 2024-04-26T00:20:12.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d87cc640 10 monclient: build_initial_monmap 2024-04-26T00:20:12.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d87cc640 10 monclient: monmap: 2024-04-26T00:20:12.956 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:12.956 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:12.956 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:12.956 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:12.956 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:12.956 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:12.956 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:12.956 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:12.956 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:12.956 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:12.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d87cc640 1 Processor -- start 2024-04-26T00:20:12.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d87cc640 1 -- start start 2024-04-26T00:20:12.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d87cc640 10 monclient: init 2024-04-26T00:20:12.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d87cc640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:12.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d87cc640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:20:12.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d87cc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc1d0072a30 0x7fc1d01ced00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:12.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d87cc640 10 monclient(hunting): picked mon.noname-c con 0x7fc1d0072a30 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:12.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d87cc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1d00a1910 0x7fc1d01305c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:12.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d87cc640 10 monclient(hunting): picked mon.noname-b con 0x7fc1d00a1910 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:12.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d87cc640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc1d00a2fe0 0x7fc1d0130b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:12.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d87cc640 10 monclient(hunting): picked mon.noname-a con 0x7fc1d00a2fe0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:12.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d87cc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:12.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d87cc640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d0072e10 con 0x7fc1d00a2fe0 2024-04-26T00:20:12.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d87cc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:12.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d87cc640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d01371e0 con 0x7fc1d00a1910 2024-04-26T00:20:12.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d87cc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:12.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d87cc640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc1d0072f90 con 0x7fc1d0072a30 2024-04-26T00:20:12.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d87cc640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:12.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d87cc640 10 monclient(hunting): authenticate will time out at 994.279268s 2024-04-26T00:20:12.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d6568640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc1d0072a30 0x7fc1d01ced00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:12.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d6568640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc1d0072a30 0x7fc1d01ced00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:34548/0 (socket says 172.21.15.37:34548) 2024-04-26T00:20:12.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d6568640 1 -- 172.21.15.37:0/1133812907 learned_addr learned my addr 172.21.15.37:0/1133812907 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:12.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d6568640 10 monclient(hunting): get_auth_request con 0x7fc1d0072a30 auth_method 0 2024-04-26T00:20:12.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d6568640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:12.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d6568640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:12.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d6568640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:12.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d6d69640 1 --2- 172.21.15.37:0/1133812907 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc1d00a2fe0 0x7fc1d0130b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:12.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d5d67640 1 -- 172.21.15.37:0/1133812907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1d00a1910 msgr2=0x7fc1d01305c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:20:12.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d6d69640 10 monclient(hunting): get_auth_request con 0x7fc1d00a2fe0 auth_method 0 2024-04-26T00:20:12.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d6d69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:12.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d6d69640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:12.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d6d69640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:12.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d5d67640 1 --2- 172.21.15.37:0/1133812907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1d00a1910 0x7fc1d01305c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:20:12.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d6568640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:12.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d6568640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:12.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d6568640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:12.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d6d69640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:12.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d6d69640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:12.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d6d69640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:12.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d6568640 10 monclient(hunting): handle_auth_done global_id 4373 payload 824 2024-04-26T00:20:12.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d6568640 1 -- 172.21.15.37:0/1133812907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1d00a1910 msgr2=0x7fc1d01305c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:20:12.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d6568640 1 --2- 172.21.15.37:0/1133812907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1d00a1910 0x7fc1d01305c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:12.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d6568640 1 -- 172.21.15.37:0/1133812907 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc1d00a2fe0 msgr2=0x7fc1d0130b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:12.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d6568640 1 --2- 172.21.15.37:0/1133812907 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc1d00a2fe0 0x7fc1d0130b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:12.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d6568640 10 monclient: _finish_hunting 0 2024-04-26T00:20:12.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d6568640 1 monclient: found mon.noname-c 2024-04-26T00:20:12.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d6568640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:12.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d6568640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:20:12.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.967+0000 7fc1d6568640 1 -- 172.21.15.37:0/1133812907 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1d013b6a0 con 0x7fc1d0072a30 2024-04-26T00:20:12.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.971+0000 7fc1d6568640 1 -- 172.21.15.37:0/1133812907 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc1d0072a30 msgr2=0x7fc1d01ced00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:12.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.971+0000 7fc1d6568640 1 --2- 172.21.15.37:0/1133812907 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc1d0072a30 0x7fc1d01ced00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc01cc00 tx=0x7fc1cc019100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:12.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.971+0000 7fc1c6ffd640 1 -- 172.21.15.37:0/1133812907 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1cc0bb070 con 0x7fc1d0072a30 2024-04-26T00:20:12.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.971+0000 7fc1c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:12.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.971+0000 7fc1c6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:20:12.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.971+0000 7fc1c6ffd640 10 monclient: dump: 2024-04-26T00:20:12.962 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:12.962 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:12.962 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:12.962 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:12.962 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:12.962 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:12.962 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:12.963 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:12.963 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:12.963 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:12.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.971+0000 7fc1c6ffd640 10 monclient: _finish_auth 0 2024-04-26T00:20:12.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.971+0000 7fc1c6ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:20:12.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.971+0000 7fc1c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:12.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.971+0000 7fc1c6ffd640 1 -- 172.21.15.37:0/1133812907 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1cc0b6650 con 0x7fc1d0072a30 2024-04-26T00:20:12.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.971+0000 7fc1d87cc640 5 monclient: authenticate success, global_id 4373 2024-04-26T00:20:12.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.971+0000 7fc1d87cc640 10 monclient: _renew_subs 2024-04-26T00:20:12.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.971+0000 7fc1d87cc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:12.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.971+0000 7fc1d87cc640 1 -- 172.21.15.37:0/1133812907 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1d013bf80 con 0x7fc1d0072a30 2024-04-26T00:20:12.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.971+0000 7fc1c6ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:12.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.971+0000 7fc1c6ffd640 1 -- 172.21.15.37:0/1133812907 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1cc0b0040 con 0x7fc1d0072a30 2024-04-26T00:20:12.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.971+0000 7fc1c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:12.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.971+0000 7fc1c6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:12.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.971+0000 7fc1c6ffd640 10 monclient: dump: 2024-04-26T00:20:12.964 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:12.964 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:12.964 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:12.964 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:12.964 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:12.964 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:12.964 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:12.964 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:12.964 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:12.964 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:12.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.971+0000 7fc1d4d65640 4 set_mon_vals no callback set 2024-04-26T00:20:12.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.971+0000 7fc1d87cc640 10 monclient: _renew_subs 2024-04-26T00:20:12.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.971+0000 7fc1d87cc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:12.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.971+0000 7fc1d87cc640 1 -- 172.21.15.37:0/1133812907 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1d013c420 con 0x7fc1d0072a30 2024-04-26T00:20:12.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.971+0000 7fc1d87cc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:12.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.971+0000 7fc1d87cc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:12.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.971+0000 7fc1d87cc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:12.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.971+0000 7fc1d87cc640 1 -- 172.21.15.37:0/1133812907 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1d0066f40 con 0x7fc1d0072a30 2024-04-26T00:20:12.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.971+0000 7fc1c6ffd640 1 -- 172.21.15.37:0/1133812907 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc1cc0be070 con 0x7fc1d0072a30 2024-04-26T00:20:12.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.971+0000 7fc1c6ffd640 1 --2- 172.21.15.37:0/1133812907 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc1b4035e80 0x7fc1b4038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:12.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.971+0000 7fc1c6ffd640 1 -- 172.21.15.37:0/1133812907 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 3858+0+0 (secure 0 0 0) 0x7fc1cc0e2d10 con 0x7fc1d0072a30 2024-04-26T00:20:12.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.971+0000 7fc1d5d67640 1 --2- 172.21.15.37:0/1133812907 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc1b4035e80 0x7fc1b4038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:12.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.971+0000 7fc1d5d67640 10 monclient: get_auth_request con 0x7fc1b4035e80 auth_method 0 2024-04-26T00:20:12.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.975+0000 7fc1d5d67640 1 --2- 172.21.15.37:0/1133812907 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc1b4035e80 0x7fc1b4038340 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc1d00a57d0 tx=0x7fc1c0009240 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:12.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.979+0000 7fc1c6ffd640 1 -- 172.21.15.37:0/1133812907 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc1cc0e9080 con 0x7fc1d0072a30 2024-04-26T00:20:12.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.979+0000 7fc1c6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:12.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:12.979+0000 7fc1c6ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:13.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:13.199+0000 7fc1d87cc640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:20:13.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:13.199+0000 7fc1d87cc640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:20:13.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:13.199+0000 7fc1d87cc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:13.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:13.199+0000 7fc1d87cc640 1 -- 172.21.15.37:0/1133812907 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc1d0132750 con 0x7fc1d0072a30 2024-04-26T00:20:13.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:13.199+0000 7fc1c6ffd640 1 -- 172.21.15.37:0/1133812907 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fc1cc016070 con 0x7fc1d0072a30 2024-04-26T00:20:13.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:13.199+0000 7fc1c6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:20:13.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:13.199+0000 7fc1c6ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:13.184 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":23,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:20:13.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:13.203+0000 7fc1d87cc640 1 -- 172.21.15.37:0/1133812907 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc1b4035e80 msgr2=0x7fc1b4038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:13.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:13.203+0000 7fc1d87cc640 1 --2- 172.21.15.37:0/1133812907 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc1b4035e80 0x7fc1b4038340 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc1d00a57d0 tx=0x7fc1c0009240 comp rx=0 tx=0).stop 2024-04-26T00:20:13.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:13.203+0000 7fc1d87cc640 10 monclient: shutdown 2024-04-26T00:20:13.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:13.203+0000 7fc1d87cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:13.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:13.203+0000 7fc1d87cc640 1 -- 172.21.15.37:0/1133812907 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc1d0072a30 msgr2=0x7fc1d01ced00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:13.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:13.203+0000 7fc1d87cc640 1 --2- 172.21.15.37:0/1133812907 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc1d0072a30 0x7fc1d01ced00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc01cc00 tx=0x7fc1cc019100 comp rx=0 tx=0).stop 2024-04-26T00:20:13.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:13.203+0000 7fc1d87cc640 1 -- 172.21.15.37:0/1133812907 shutdown_connections 2024-04-26T00:20:13.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:13.203+0000 7fc1d87cc640 1 --2- 172.21.15.37:0/1133812907 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc1b4035e80 0x7fc1b4038340 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:13.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:13.203+0000 7fc1d87cc640 1 --2- 172.21.15.37:0/1133812907 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc1d00a2fe0 0x7fc1d0130b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:13.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:13.203+0000 7fc1d87cc640 1 --2- 172.21.15.37:0/1133812907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1d00a1910 0x7fc1d01305c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:13.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:13.203+0000 7fc1d87cc640 1 --2- 172.21.15.37:0/1133812907 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc1d0072a30 0x7fc1d01ced00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:13.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:13.203+0000 7fc1d87cc640 1 -- 172.21.15.37:0/1133812907 >> 172.21.15.37:0/1133812907 conn(0x7fc1d00a24c0 msgr2=0x7fc1d00a42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:13.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:13.203+0000 7fc1d87cc640 1 -- 172.21.15.37:0/1133812907 shutdown_connections 2024-04-26T00:20:13.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:13.203+0000 7fc1d87cc640 1 -- 172.21.15.37:0/1133812907 wait complete. 2024-04-26T00:20:13.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:13.491+0000 7f6e85ffb640 10 monclient: tick 2024-04-26T00:20:13.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:13.491+0000 7f6e85ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:20:13.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:13.491+0000 7f6e85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:14.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:14.491+0000 7f6e85ffb640 10 monclient: tick 2024-04-26T00:20:14.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:14.491+0000 7f6e85ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:20:14.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:14.491+0000 7f6e85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:15.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:15.495+0000 7f6e85ffb640 10 monclient: tick 2024-04-26T00:20:15.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:15.495+0000 7f6e85ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:20:15.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:15.495+0000 7f6e85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:16.203 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:20:16.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b74cf640 10 monclient: get_monmap_and_config 2024-04-26T00:20:16.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b74cf640 10 monclient: build_initial_monmap 2024-04-26T00:20:16.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b74cf640 10 monclient: monmap: 2024-04-26T00:20:16.337 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:16.337 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:16.338 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:16.361195+0000 2024-04-26T00:20:16.338 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:16.361195+0000 2024-04-26T00:20:16.338 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:16.338 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:16.338 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:16.338 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:16.338 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:16.338 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:16.338 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:16.338 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:16.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b74cf640 1 Processor -- start 2024-04-26T00:20:16.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b74cf640 1 -- start start 2024-04-26T00:20:16.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b74cf640 10 monclient: init 2024-04-26T00:20:16.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b74cf640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:16.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b74cf640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-04-26T00:20:16.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b74cf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f11b0136e10 addr v1:172.21.15.37:6789/0 2024-04-26T00:20:16.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b74cf640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f11b0137900 0x7f11b009c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:16.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b74cf640 10 monclient(hunting): picked mon.noname-a con 0x7f11b0137900 addr v2:172.21.15.37:3300/0 2024-04-26T00:20:16.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b74cf640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f11b013c340 0x7f11b013e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:16.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b74cf640 10 monclient(hunting): picked mon.noname-c con 0x7f11b013c340 addr v2:172.21.15.45:3300/0 2024-04-26T00:20:16.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b74cf640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11b009b680 con 0x7f11b0136e10 2024-04-26T00:20:16.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b74cf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:16.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b74cf640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f11b009b8f0 con 0x7f11b0137900 2024-04-26T00:20:16.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b74cf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:16.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b74cf640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f11b009bbd0 con 0x7f11b013c340 2024-04-26T00:20:16.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b74cf640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:16.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b74cf640 10 monclient(hunting): authenticate will time out at 997.666950s 2024-04-26T00:20:16.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b5a6c640 1 -- >> v2:172.21.15.45:3300/0 conn(0x7f11b013c340 msgr2=0x7f11b013e730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:20:16.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b5a6c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f11b013c340 0x7f11b013e730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:20:16.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b526b640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f11b0136e10 0x7f11b01371f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:49554/0 (socket says 172.21.15.37:49554) 2024-04-26T00:20:16.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b526b640 1 -- 172.21.15.37:0/2959912364 learned_addr learned my addr 172.21.15.37:0/2959912364 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:20:16.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b4a6a640 1 --2- 172.21.15.37:0/2959912364 >> v2:172.21.15.37:3300/0 conn(0x7f11b0137900 0x7f11b009c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:16.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b4a6a640 10 monclient(hunting): get_auth_request con 0x7f11b0137900 auth_method 0 2024-04-26T00:20:16.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b4a6a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:16.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b4a6a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:16.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b4a6a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:16.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b4a6a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:16.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b4a6a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:16.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b4a6a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:16.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b4a6a640 10 monclient(hunting): handle_auth_done global_id 4377 payload 274 2024-04-26T00:20:16.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b4a6a640 1 -- 172.21.15.37:0/2959912364 >> v2:172.21.15.45:3300/0 conn(0x7f11b013c340 msgr2=0x7f11b013e730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:20:16.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b4a6a640 1 --2- 172.21.15.37:0/2959912364 >> v2:172.21.15.45:3300/0 conn(0x7f11b013c340 0x7f11b013e730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:16.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b4a6a640 1 -- 172.21.15.37:0/2959912364 >> v1:172.21.15.37:6789/0 conn(0x7f11b0136e10 legacy=0x7f11b01371f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:16.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b4a6a640 10 monclient: _finish_hunting 0 2024-04-26T00:20:16.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b4a6a640 1 monclient: found mon.noname-a 2024-04-26T00:20:16.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b4a6a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:16.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b4a6a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:16.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.359+0000 7f11b4a6a640 1 -- 172.21.15.37:0/2959912364 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11b00a5d90 con 0x7f11b0137900 2024-04-26T00:20:16.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.363+0000 7f11b4a6a640 1 -- 172.21.15.37:0/2959912364 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11b0137900 msgr2=0x7f11b009c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:16.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.363+0000 7f11b4a6a640 1 --2- 172.21.15.37:0/2959912364 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11b0137900 0x7f11b009c5b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f11b00a9590 tx=0x7f11ac01c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a1f32ea641409ea server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:16.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11a7fff640 1 -- 172.21.15.37:0/2959912364 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11ac0ac070 con 0x7f11b0137900 2024-04-26T00:20:16.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:16.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11a7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:16.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11a7fff640 10 monclient: dump: 2024-04-26T00:20:16.348 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:16.348 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:16.348 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:16.348 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:16.348 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:16.348 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:16.348 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:16.348 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:16.348 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:16.348 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:16.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11a7fff640 10 monclient: _finish_auth 0 2024-04-26T00:20:16.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11a7fff640 10 monclient: _check_auth_tickets 2024-04-26T00:20:16.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:16.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11a7fff640 1 -- 172.21.15.37:0/2959912364 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11ac0a6780 con 0x7f11b0137900 2024-04-26T00:20:16.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11a7fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:16.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11a7fff640 1 -- 172.21.15.37:0/2959912364 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11ac0a6b50 con 0x7f11b0137900 2024-04-26T00:20:16.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b74cf640 5 monclient: authenticate success, global_id 4377 2024-04-26T00:20:16.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b74cf640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:16.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b74cf640 4 set_mon_vals no callback set 2024-04-26T00:20:16.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:16.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11a7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:16.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11a7fff640 10 monclient: dump: 2024-04-26T00:20:16.349 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:16.349 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:16.349 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:16.349 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:16.350 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:16.350 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:16.350 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:16.350 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:16.350 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:16.350 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:16.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b74cf640 10 monclient: shutdown 2024-04-26T00:20:16.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b74cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:16.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b74cf640 1 -- 172.21.15.37:0/2959912364 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11b0137900 msgr2=0x7f11b009c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:16.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b74cf640 1 --2- 172.21.15.37:0/2959912364 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11b0137900 0x7f11b009c5b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f11b00a9590 tx=0x7f11ac01c7d0 comp rx=0 tx=0).stop 2024-04-26T00:20:16.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b74cf640 1 -- 172.21.15.37:0/2959912364 shutdown_connections 2024-04-26T00:20:16.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b74cf640 1 --2- 172.21.15.37:0/2959912364 >> v2:172.21.15.45:3300/0 conn(0x7f11b013c340 0x7f11b013e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:16.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b74cf640 1 --2- 172.21.15.37:0/2959912364 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11b0137900 0x7f11b009c5b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:16.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b74cf640 1 -- 172.21.15.37:0/2959912364 >> 172.21.15.37:0/2959912364 conn(0x7f11b00a64e0 msgr2=0x7f11b00a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:16.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b74cf640 1 -- 172.21.15.37:0/2959912364 shutdown_connections 2024-04-26T00:20:16.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b74cf640 1 -- 172.21.15.37:0/2959912364 wait complete. 2024-04-26T00:20:16.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b74cf640 10 monclient: build_initial_monmap 2024-04-26T00:20:16.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b74cf640 10 monclient: monmap: 2024-04-26T00:20:16.351 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:16.351 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:16.351 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:16.351 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:16.351 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:16.351 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:16.351 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:16.351 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:16.351 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:16.351 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:16.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b74cf640 1 Processor -- start 2024-04-26T00:20:16.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b74cf640 1 -- start start 2024-04-26T00:20:16.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b74cf640 10 monclient: init 2024-04-26T00:20:16.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b74cf640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:16.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b74cf640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:20:16.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b74cf640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f11b0136e10 0x7f11b011fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:16.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b74cf640 10 monclient(hunting): picked mon.noname-c con 0x7f11b0136e10 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:16.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b74cf640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11b0137900 0x7f11b0139570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:16.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b74cf640 10 monclient(hunting): picked mon.noname-b con 0x7f11b0137900 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:16.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b74cf640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11b013c340 0x7f11b01361a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:16.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b74cf640 10 monclient(hunting): picked mon.noname-a con 0x7f11b013c340 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:16.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b74cf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:16.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b74cf640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f11b0141820 con 0x7f11b013c340 2024-04-26T00:20:16.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b74cf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:16.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b74cf640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f11b01416a0 con 0x7f11b0137900 2024-04-26T00:20:16.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b74cf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:16.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b74cf640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f11b01419a0 con 0x7f11b0136e10 2024-04-26T00:20:16.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b74cf640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:16.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b74cf640 10 monclient(hunting): authenticate will time out at 997.674368s 2024-04-26T00:20:16.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b5a6c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11b013c340 0x7f11b01361a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:16.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b5a6c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11b013c340 0x7f11b01361a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:56720/0 (socket says 172.21.15.37:56720) 2024-04-26T00:20:16.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b5a6c640 1 -- 172.21.15.37:0/338494814 learned_addr learned my addr 172.21.15.37:0/338494814 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:16.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b5a6c640 10 monclient(hunting): get_auth_request con 0x7f11b013c340 auth_method 0 2024-04-26T00:20:16.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b5a6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:16.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b5a6c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:16.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b5a6c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:16.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b526b640 1 --2- 172.21.15.37:0/338494814 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f11b0136e10 0x7f11b011fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:16.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b526b640 10 monclient(hunting): get_auth_request con 0x7f11b0136e10 auth_method 0 2024-04-26T00:20:16.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b526b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:16.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b526b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:16.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b4a6a640 1 -- 172.21.15.37:0/338494814 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11b0137900 msgr2=0x7f11b0139570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:20:16.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b526b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:16.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b4a6a640 1 --2- 172.21.15.37:0/338494814 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11b0137900 0x7f11b0139570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:20:16.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b5a6c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:16.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b5a6c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:16.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b5a6c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:16.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b526b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:16.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b526b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:16.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b526b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:16.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b5a6c640 10 monclient(hunting): handle_auth_done global_id 4380 payload 824 2024-04-26T00:20:16.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b5a6c640 1 -- 172.21.15.37:0/338494814 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f11b0136e10 msgr2=0x7f11b011fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:16.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b5a6c640 1 --2- 172.21.15.37:0/338494814 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f11b0136e10 0x7f11b011fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:16.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b5a6c640 1 -- 172.21.15.37:0/338494814 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11b0137900 msgr2=0x7f11b0139570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:20:16.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b5a6c640 1 --2- 172.21.15.37:0/338494814 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11b0137900 0x7f11b0139570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:16.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b5a6c640 10 monclient: _finish_hunting 0 2024-04-26T00:20:16.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b5a6c640 1 monclient: found mon.noname-a 2024-04-26T00:20:16.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b5a6c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:16.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b5a6c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:16.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.367+0000 7f11b5a6c640 1 -- 172.21.15.37:0/338494814 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11b013ac90 con 0x7f11b013c340 2024-04-26T00:20:16.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.371+0000 7f11b5a6c640 1 -- 172.21.15.37:0/338494814 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11b013c340 msgr2=0x7f11b01361a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:16.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.371+0000 7f11b5a6c640 1 --2- 172.21.15.37:0/338494814 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11b013c340 0x7f11b01361a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f11a800b440 tx=0x7f11a800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:16.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.371+0000 7f11a5ffb640 1 -- 172.21.15.37:0/338494814 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11a8017070 con 0x7f11b013c340 2024-04-26T00:20:16.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.371+0000 7f11a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:16.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.371+0000 7f11a5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:16.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.371+0000 7f11a5ffb640 10 monclient: dump: 2024-04-26T00:20:16.357 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:16.357 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:16.357 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:16.357 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:16.357 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:16.357 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:16.358 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:16.358 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:16.358 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:16.358 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:16.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.371+0000 7f11a5ffb640 10 monclient: _finish_auth 0 2024-04-26T00:20:16.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.371+0000 7f11a5ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:20:16.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.371+0000 7f11a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:16.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.371+0000 7f11b74cf640 5 monclient: authenticate success, global_id 4380 2024-04-26T00:20:16.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.371+0000 7f11a5ffb640 1 -- 172.21.15.37:0/338494814 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11a80060d0 con 0x7f11b013c340 2024-04-26T00:20:16.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.371+0000 7f11a5ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:16.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.371+0000 7f11b74cf640 10 monclient: _renew_subs 2024-04-26T00:20:16.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.371+0000 7f11a5ffb640 1 -- 172.21.15.37:0/338494814 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11a80064c0 con 0x7f11b013c340 2024-04-26T00:20:16.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.371+0000 7f11b74cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:16.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.371+0000 7f11b74cf640 1 -- 172.21.15.37:0/338494814 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11b009aca0 con 0x7f11b013c340 2024-04-26T00:20:16.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.371+0000 7f11a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:16.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.371+0000 7f11a77fe640 4 set_mon_vals no callback set 2024-04-26T00:20:16.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.371+0000 7f11a5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:16.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.371+0000 7f11a5ffb640 10 monclient: dump: 2024-04-26T00:20:16.359 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:16.359 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:16.359 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:16.359 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:16.359 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:16.359 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:16.359 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:16.359 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:16.359 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:16.359 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:16.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.371+0000 7f11b74cf640 10 monclient: _renew_subs 2024-04-26T00:20:16.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.371+0000 7f11b74cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:16.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.371+0000 7f11b74cf640 1 -- 172.21.15.37:0/338494814 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11b013a480 con 0x7f11b013c340 2024-04-26T00:20:16.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.371+0000 7f11a5ffb640 1 -- 172.21.15.37:0/338494814 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f11a801a070 con 0x7f11b013c340 2024-04-26T00:20:16.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.371+0000 7f11b74cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:16.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.371+0000 7f11b74cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:16.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.371+0000 7f11b74cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:16.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.371+0000 7f11b74cf640 1 -- 172.21.15.37:0/338494814 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11b0075910 con 0x7f11b013c340 2024-04-26T00:20:16.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.371+0000 7f11a5ffb640 1 --2- 172.21.15.37:0/338494814 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1194035e30 0x7f11940382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:16.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.371+0000 7f11a5ffb640 1 -- 172.21.15.37:0/338494814 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 3858+0+0 (secure 0 0 0) 0x7f11a8047c80 con 0x7f11b013c340 2024-04-26T00:20:16.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.371+0000 7f11b526b640 1 --2- 172.21.15.37:0/338494814 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1194035e30 0x7f11940382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:16.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.371+0000 7f11b526b640 10 monclient: get_auth_request con 0x7f1194035e30 auth_method 0 2024-04-26T00:20:16.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.375+0000 7f11b526b640 1 --2- 172.21.15.37:0/338494814 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1194035e30 0x7f11940382f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f11a0007ca0 tx=0x7f11a0007bf0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:16.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.375+0000 7f11a5ffb640 1 -- 172.21.15.37:0/338494814 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f11a8010360 con 0x7f11b013c340 2024-04-26T00:20:16.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.375+0000 7f11a5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:16.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.375+0000 7f11a5ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:16.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.495+0000 7f6e85ffb640 10 monclient: tick 2024-04-26T00:20:16.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.495+0000 7f6e85ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:20:16.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.495+0000 7f6e85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:16.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.611+0000 7f11b74cf640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:20:16.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.611+0000 7f11b74cf640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:20:16.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.611+0000 7f11b74cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:16.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.611+0000 7f11b74cf640 1 -- 172.21.15.37:0/338494814 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f11b0066f40 con 0x7f11b013c340 2024-04-26T00:20:16.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.611+0000 7f11a5ffb640 1 -- 172.21.15.37:0/338494814 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f11a8010e80 con 0x7f11b013c340 2024-04-26T00:20:16.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.611+0000 7f11a5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:20:16.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.611+0000 7f11a5ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:16.591 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":26,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:20:16.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.615+0000 7f11b74cf640 1 -- 172.21.15.37:0/338494814 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1194035e30 msgr2=0x7f11940382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:16.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.615+0000 7f11b74cf640 1 --2- 172.21.15.37:0/338494814 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1194035e30 0x7f11940382f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f11a0007ca0 tx=0x7f11a0007bf0 comp rx=0 tx=0).stop 2024-04-26T00:20:16.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.615+0000 7f11b74cf640 10 monclient: shutdown 2024-04-26T00:20:16.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.615+0000 7f11b74cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:16.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.615+0000 7f11b74cf640 1 -- 172.21.15.37:0/338494814 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11b013c340 msgr2=0x7f11b01361a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:16.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.615+0000 7f11b74cf640 1 --2- 172.21.15.37:0/338494814 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11b013c340 0x7f11b01361a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f11a800b440 tx=0x7f11a800b900 comp rx=0 tx=0).stop 2024-04-26T00:20:16.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.615+0000 7f11b74cf640 1 -- 172.21.15.37:0/338494814 shutdown_connections 2024-04-26T00:20:16.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.615+0000 7f11b74cf640 1 --2- 172.21.15.37:0/338494814 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1194035e30 0x7f11940382f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:16.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.615+0000 7f11b74cf640 1 --2- 172.21.15.37:0/338494814 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11b013c340 0x7f11b01361a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:16.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.615+0000 7f11b74cf640 1 --2- 172.21.15.37:0/338494814 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11b0137900 0x7f11b0139570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:16.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.615+0000 7f11b74cf640 1 --2- 172.21.15.37:0/338494814 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f11b0136e10 0x7f11b011fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:16.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.615+0000 7f11b74cf640 1 -- 172.21.15.37:0/338494814 >> 172.21.15.37:0/338494814 conn(0x7f11b00a64e0 msgr2=0x7f11b013d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:16.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.615+0000 7f11b74cf640 1 -- 172.21.15.37:0/338494814 shutdown_connections 2024-04-26T00:20:16.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:16.615+0000 7f11b74cf640 1 -- 172.21.15.37:0/338494814 wait complete. 2024-04-26T00:20:17.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:17.495+0000 7f6e85ffb640 10 monclient: tick 2024-04-26T00:20:17.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:17.495+0000 7f6e85ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:20:17.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:17.495+0000 7f6e85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:18.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:18.495+0000 7f6e85ffb640 10 monclient: tick 2024-04-26T00:20:18.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:18.495+0000 7f6e85ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:20:18.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:18.495+0000 7f6e85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:19.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:19.495+0000 7f6e85ffb640 10 monclient: tick 2024-04-26T00:20:19.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:19.495+0000 7f6e85ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:20:19.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:19.495+0000 7f6e85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:19.611 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:20:19.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:19.763+0000 7f0ebcfb6640 10 monclient: get_monmap_and_config 2024-04-26T00:20:19.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:19.763+0000 7f0ebcfb6640 10 monclient: build_initial_monmap 2024-04-26T00:20:19.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:19.763+0000 7f0ebcfb6640 10 monclient: monmap: 2024-04-26T00:20:19.742 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:19.742 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:19.742 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:19.765612+0000 2024-04-26T00:20:19.742 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:19.765612+0000 2024-04-26T00:20:19.742 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:19.742 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:19.742 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:19.742 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:19.742 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:19.743 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:19.743 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:19.743 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:19.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:19.763+0000 7f0ebcfb6640 1 Processor -- start 2024-04-26T00:20:19.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:19.763+0000 7f0ebcfb6640 1 -- start start 2024-04-26T00:20:19.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:19.763+0000 7f0ebcfb6640 10 monclient: init 2024-04-26T00:20:19.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:19.763+0000 7f0ebcfb6640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:19.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:19.767+0000 7f0ebcfb6640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-04-26T00:20:19.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:19.767+0000 7f0ebcfb6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0eb8133b20 addr v1:172.21.15.45:6789/0 2024-04-26T00:20:19.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:19.767+0000 7f0ebcfb6640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0eb8133f00 0x7f0eb8134350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:19.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:19.767+0000 7f0ebcfb6640 10 monclient(hunting): picked mon.noname-c con 0x7f0eb8133f00 addr v2:172.21.15.45:3300/0 2024-04-26T00:20:19.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:19.767+0000 7f0ebcfb6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0eb8139050 addr v1:172.21.15.37:6789/0 2024-04-26T00:20:19.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:19.767+0000 7f0ebcfb6640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0eb809fe20 con 0x7f0eb8139050 2024-04-26T00:20:19.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:19.767+0000 7f0ebcfb6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0eb8072d00 con 0x7f0eb8133b20 2024-04-26T00:20:19.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:19.767+0000 7f0ebcfb6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:19.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:19.767+0000 7f0ebcfb6640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f0eb809ba60 con 0x7f0eb8133f00 2024-04-26T00:20:19.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:19.767+0000 7f0ebcfb6640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:19.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:19.767+0000 7f0ebcfb6640 10 monclient(hunting): authenticate will time out at 1001.072144s 2024-04-26T00:20:19.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:19.767+0000 7f0eaffff640 1 -- >> v2:172.21.15.45:3300/0 conn(0x7f0eb8133f00 msgr2=0x7f0eb8134350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:20:19.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:19.767+0000 7f0eb659c640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f0eb8133b20 legacy=0x7f0eb809e9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:20:19.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:19.767+0000 7f0eaffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0eb8133f00 0x7f0eb8134350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:20:19.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:19.767+0000 7f0eb6d9d640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f0eb8139050 0x7f0eb813b440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:41026/0 (socket says 172.21.15.37:41026) 2024-04-26T00:20:19.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:19.767+0000 7f0eb6d9d640 1 -- 172.21.15.37:0/2527334282 learned_addr learned my addr 172.21.15.37:0/2527334282 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:20:19.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:19.767+0000 7f0eb6d9d640 1 -- 172.21.15.37:0/2527334282 >> v1:172.21.15.37:6789/0 conn(0x7f0eb8139050 legacy=0x7f0eb813b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:19.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:19.971+0000 7f0eaffff640 1 -- 172.21.15.37:0/2527334282 >> v2:172.21.15.45:3300/0 conn(0x7f0eb8133f00 msgr2=0x7f0eb8134350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:20:19.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:19.971+0000 7f0eb659c640 1 -- 172.21.15.37:0/2527334282 >> v1:172.21.15.45:6789/0 conn(0x7f0eb8133b20 legacy=0x7f0eb809e9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:20:19.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:19.971+0000 7f0eaffff640 1 --2- 172.21.15.37:0/2527334282 >> v2:172.21.15.45:3300/0 conn(0x7f0eb8133f00 0x7f0eb8134350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:20:20.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:20.375+0000 7f0eaffff640 1 -- 172.21.15.37:0/2527334282 >> v2:172.21.15.45:3300/0 conn(0x7f0eb8133f00 msgr2=0x7f0eb8134350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:20:20.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:20.375+0000 7f0eb659c640 1 -- 172.21.15.37:0/2527334282 >> v1:172.21.15.45:6789/0 conn(0x7f0eb8133b20 legacy=0x7f0eb809e9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:20:20.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:20.375+0000 7f0eaffff640 1 --2- 172.21.15.37:0/2527334282 >> v2:172.21.15.45:3300/0 conn(0x7f0eb8133f00 0x7f0eb8134350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:20:20.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:20.495+0000 7f6e85ffb640 10 monclient: tick 2024-04-26T00:20:20.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:20.495+0000 7f6e85ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:20:20.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:20.495+0000 7f6e85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:21.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:21.179+0000 7f0eaffff640 1 -- 172.21.15.37:0/2527334282 >> v2:172.21.15.45:3300/0 conn(0x7f0eb8133f00 msgr2=0x7f0eb8134350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:20:21.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:21.179+0000 7f0eb659c640 1 -- 172.21.15.37:0/2527334282 >> v1:172.21.15.45:6789/0 conn(0x7f0eb8133b20 legacy=0x7f0eb809e9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:20:21.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:21.179+0000 7f0eaffff640 1 --2- 172.21.15.37:0/2527334282 >> v2:172.21.15.45:3300/0 conn(0x7f0eb8133f00 0x7f0eb8134350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:20:21.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:21.495+0000 7f6e85ffb640 10 monclient: tick 2024-04-26T00:20:21.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:21.495+0000 7f6e85ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:20:21.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:21.495+0000 7f6e85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:22.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.495+0000 7f6e85ffb640 10 monclient: tick 2024-04-26T00:20:22.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.495+0000 7f6e85ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:20:22.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.495+0000 7f6e85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:22.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.495+0000 7f6e85ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:22.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.767+0000 7f0eb4d99640 10 monclient(hunting): tick 2024-04-26T00:20:22.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.767+0000 7f0eb4d99640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:20:22.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.767+0000 7f0eb4d99640 1 monclient(hunting): continuing hunt 2024-04-26T00:20:22.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.767+0000 7f0eb4d99640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:20:22.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.767+0000 7f0eb4d99640 1 -- 172.21.15.37:0/2527334282 >> v2:172.21.15.45:3300/0 conn(0x7f0eb8133f00 msgr2=0x7f0eb8134350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:20:22.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.767+0000 7f0eb4d99640 1 --2- 172.21.15.37:0/2527334282 >> v2:172.21.15.45:3300/0 conn(0x7f0eb8133f00 0x7f0eb8134350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:22.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.767+0000 7f0eb4d99640 1 -- 172.21.15.37:0/2527334282 >> v1:172.21.15.45:6789/0 conn(0x7f0eb8133b20 legacy=0x7f0eb809e9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:20:22.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.767+0000 7f0eb4d99640 1 -- 172.21.15.37:0/2527334282 >> v1:172.21.15.37:6789/0 conn(0x7f0eb8139050 legacy=0x7f0eb813b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:22.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.767+0000 7f0eb4d99640 10 monclient: _add_conns ranks=[3,2] 2024-04-26T00:20:22.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.767+0000 7f0eb4d99640 1 --2- 172.21.15.37:0/2527334282 >> v2:172.21.15.37:3300/0 conn(0x7f0e98005960 0x7f0e98007de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:22.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.767+0000 7f0eb4d99640 10 monclient(hunting): picked mon.noname-a con 0x7f0e98005960 addr v2:172.21.15.37:3300/0 2024-04-26T00:20:22.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.767+0000 7f0eb4d99640 1 --2- 172.21.15.37:0/2527334282 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e98008470 0x7f0e9800a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:22.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.767+0000 7f0eb4d99640 10 monclient(hunting): picked mon.noname-b con 0x7f0e98008470 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:22.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.767+0000 7f0eb4d99640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:22.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.767+0000 7f0eb4d99640 1 -- 172.21.15.37:0/2527334282 --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f0eb809ba60 con 0x7f0e98005960 2024-04-26T00:20:22.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.767+0000 7f0eb4d99640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:22.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.767+0000 7f0eb4d99640 1 -- 172.21.15.37:0/2527334282 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0e9800b040 con 0x7f0e98008470 2024-04-26T00:20:22.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.767+0000 7f0eb4d99640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:22.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.767+0000 7f0eb659c640 1 --2- 172.21.15.37:0/2527334282 >> v2:172.21.15.37:3300/0 conn(0x7f0e98005960 0x7f0e98007de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:22.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.767+0000 7f0eaffff640 1 --2- 172.21.15.37:0/2527334282 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e98008470 0x7f0e9800a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:22.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.767+0000 7f0eb659c640 10 monclient(hunting): get_auth_request con 0x7f0e98005960 auth_method 0 2024-04-26T00:20:22.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.767+0000 7f0eb659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:22.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.767+0000 7f0eb659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:22.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.767+0000 7f0eb659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:22.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.767+0000 7f0eaffff640 10 monclient(hunting): get_auth_request con 0x7f0e98008470 auth_method 0 2024-04-26T00:20:22.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.767+0000 7f0eaffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:22.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.767+0000 7f0eaffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:22.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.767+0000 7f0eaffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:22.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.767+0000 7f0eb659c640 1 -- 172.21.15.37:0/2527334282 >> v2:172.21.15.37:3300/0 conn(0x7f0e98005960 msgr2=0x7f0e98007de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:20:22.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.767+0000 7f0eb659c640 1 -- 172.21.15.37:0/2527334282 >> v2:172.21.15.37:3300/0 conn(0x7f0e98005960 msgr2=0x7f0e98007de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:20:22.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.767+0000 7f0eaffff640 1 -- 172.21.15.37:0/2527334282 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e98008470 msgr2=0x7f0e9800a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:20:22.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.767+0000 7f0eaffff640 1 -- 172.21.15.37:0/2527334282 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e98008470 msgr2=0x7f0e9800a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:20:22.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.767+0000 7f0eb659c640 1 --2- 172.21.15.37:0/2527334282 >> v2:172.21.15.37:3300/0 conn(0x7f0e98005960 0x7f0e98007de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:20:22.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.767+0000 7f0eaffff640 1 --2- 172.21.15.37:0/2527334282 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e98008470 0x7f0e9800a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:20:22.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.767+0000 7f0eaffff640 1 --2- 172.21.15.37:0/2527334282 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e98008470 0x7f0e9800a930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:20:22.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.767+0000 7f0eb659c640 1 --2- 172.21.15.37:0/2527334282 >> v2:172.21.15.37:3300/0 conn(0x7f0e98005960 0x7f0e98007de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:20:22.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.971+0000 7f0eb659c640 1 --2- 172.21.15.37:0/2527334282 >> v2:172.21.15.37:3300/0 conn(0x7f0e98005960 0x7f0e98007de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:22.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.971+0000 7f0eaffff640 1 --2- 172.21.15.37:0/2527334282 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e98008470 0x7f0e9800a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:22.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.971+0000 7f0eb659c640 10 monclient(hunting): get_auth_request con 0x7f0e98005960 auth_method 0 2024-04-26T00:20:22.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.971+0000 7f0eb659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:22.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.971+0000 7f0eb659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:22.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.971+0000 7f0eb659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:20:22.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.971+0000 7f0eaffff640 10 monclient(hunting): get_auth_request con 0x7f0e98008470 auth_method 0 2024-04-26T00:20:22.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.971+0000 7f0eaffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:22.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.971+0000 7f0eaffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:22.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.971+0000 7f0eaffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:20:22.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.971+0000 7f0eb659c640 1 -- 172.21.15.37:0/2527334282 >> v2:172.21.15.37:3300/0 conn(0x7f0e98005960 msgr2=0x7f0e98007de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:20:22.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.971+0000 7f0eb659c640 1 -- 172.21.15.37:0/2527334282 >> v2:172.21.15.37:3300/0 conn(0x7f0e98005960 msgr2=0x7f0e98007de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:20:22.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.971+0000 7f0eb659c640 1 --2- 172.21.15.37:0/2527334282 >> v2:172.21.15.37:3300/0 conn(0x7f0e98005960 0x7f0e98007de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:20:22.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.971+0000 7f0eaffff640 1 -- 172.21.15.37:0/2527334282 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e98008470 msgr2=0x7f0e9800a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:20:22.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.971+0000 7f0eaffff640 1 -- 172.21.15.37:0/2527334282 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e98008470 msgr2=0x7f0e9800a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:20:22.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.971+0000 7f0eaffff640 1 --2- 172.21.15.37:0/2527334282 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e98008470 0x7f0e9800a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:20:22.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.971+0000 7f0eb659c640 1 --2- 172.21.15.37:0/2527334282 >> v2:172.21.15.37:3300/0 conn(0x7f0e98005960 0x7f0e98007de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:20:22.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:22.971+0000 7f0eaffff640 1 --2- 172.21.15.37:0/2527334282 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e98008470 0x7f0e9800a930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:20:23.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:23.375+0000 7f0eb659c640 1 --2- 172.21.15.37:0/2527334282 >> v2:172.21.15.37:3300/0 conn(0x7f0e98005960 0x7f0e98007de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:23.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:23.375+0000 7f0eaffff640 1 --2- 172.21.15.37:0/2527334282 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e98008470 0x7f0e9800a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:23.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:23.375+0000 7f0eb659c640 10 monclient(hunting): get_auth_request con 0x7f0e98005960 auth_method 0 2024-04-26T00:20:23.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:23.375+0000 7f0eb659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:23.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:23.375+0000 7f0eb659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:23.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:23.375+0000 7f0eb659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:20:23.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:23.375+0000 7f0eaffff640 10 monclient(hunting): get_auth_request con 0x7f0e98008470 auth_method 0 2024-04-26T00:20:23.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:23.375+0000 7f0eaffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:23.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:23.375+0000 7f0eaffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:23.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:23.375+0000 7f0eaffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:20:23.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:23.375+0000 7f0eb659c640 1 -- 172.21.15.37:0/2527334282 >> v2:172.21.15.37:3300/0 conn(0x7f0e98005960 msgr2=0x7f0e98007de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:20:23.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:23.375+0000 7f0eb659c640 1 -- 172.21.15.37:0/2527334282 >> v2:172.21.15.37:3300/0 conn(0x7f0e98005960 msgr2=0x7f0e98007de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:20:23.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:23.375+0000 7f0eb659c640 1 --2- 172.21.15.37:0/2527334282 >> v2:172.21.15.37:3300/0 conn(0x7f0e98005960 0x7f0e98007de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:20:23.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:23.375+0000 7f0eaffff640 1 -- 172.21.15.37:0/2527334282 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e98008470 msgr2=0x7f0e9800a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:20:23.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:23.375+0000 7f0eaffff640 1 -- 172.21.15.37:0/2527334282 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e98008470 msgr2=0x7f0e9800a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:20:23.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:23.375+0000 7f0eaffff640 1 --2- 172.21.15.37:0/2527334282 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e98008470 0x7f0e9800a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:20:23.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:23.375+0000 7f0eb659c640 1 --2- 172.21.15.37:0/2527334282 >> v2:172.21.15.37:3300/0 conn(0x7f0e98005960 0x7f0e98007de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:20:23.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:23.375+0000 7f0eaffff640 1 --2- 172.21.15.37:0/2527334282 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e98008470 0x7f0e9800a930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:20:23.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:23.495+0000 7f6e85ffb640 10 monclient: tick 2024-04-26T00:20:23.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:23.495+0000 7f6e85ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:20:23.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:23.495+0000 7f6e85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:24.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:24.179+0000 7f0eb659c640 1 --2- 172.21.15.37:0/2527334282 >> v2:172.21.15.37:3300/0 conn(0x7f0e98005960 0x7f0e98007de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:24.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:24.179+0000 7f0eaffff640 1 --2- 172.21.15.37:0/2527334282 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e98008470 0x7f0e9800a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:24.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:24.179+0000 7f0eb659c640 10 monclient(hunting): get_auth_request con 0x7f0e98005960 auth_method 0 2024-04-26T00:20:24.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:24.179+0000 7f0eb659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:24.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:24.179+0000 7f0eb659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:24.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:24.179+0000 7f0eb659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:20:24.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:24.179+0000 7f0eaffff640 10 monclient(hunting): get_auth_request con 0x7f0e98008470 auth_method 0 2024-04-26T00:20:24.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:24.179+0000 7f0eaffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:24.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:24.179+0000 7f0eaffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:24.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:24.179+0000 7f0eaffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:20:24.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:24.179+0000 7f0eb659c640 1 -- 172.21.15.37:0/2527334282 >> v2:172.21.15.37:3300/0 conn(0x7f0e98005960 msgr2=0x7f0e98007de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:20:24.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:24.179+0000 7f0eb659c640 1 -- 172.21.15.37:0/2527334282 >> v2:172.21.15.37:3300/0 conn(0x7f0e98005960 msgr2=0x7f0e98007de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:20:24.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:24.179+0000 7f0eb659c640 1 --2- 172.21.15.37:0/2527334282 >> v2:172.21.15.37:3300/0 conn(0x7f0e98005960 0x7f0e98007de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:20:24.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:24.179+0000 7f0eaffff640 1 -- 172.21.15.37:0/2527334282 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e98008470 msgr2=0x7f0e9800a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:20:24.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:24.179+0000 7f0eaffff640 1 -- 172.21.15.37:0/2527334282 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e98008470 msgr2=0x7f0e9800a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:20:24.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:24.179+0000 7f0eaffff640 1 --2- 172.21.15.37:0/2527334282 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e98008470 0x7f0e9800a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:20:24.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:24.179+0000 7f0eb659c640 1 --2- 172.21.15.37:0/2527334282 >> v2:172.21.15.37:3300/0 conn(0x7f0e98005960 0x7f0e98007de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:20:24.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:24.179+0000 7f0eaffff640 1 --2- 172.21.15.37:0/2527334282 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e98008470 0x7f0e9800a930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:20:24.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:24.495+0000 7f6e85ffb640 10 monclient: tick 2024-04-26T00:20:24.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:24.495+0000 7f6e85ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:20:24.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:24.495+0000 7f6e85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:25.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.495+0000 7f6e85ffb640 10 monclient: tick 2024-04-26T00:20:25.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.495+0000 7f6e85ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:20:25.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.495+0000 7f6e85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:25.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eb4d99640 10 monclient(hunting): tick 2024-04-26T00:20:25.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eb4d99640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:20:25.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eb4d99640 1 monclient(hunting): continuing hunt 2024-04-26T00:20:25.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eb4d99640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:20:25.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eb4d99640 1 -- 172.21.15.37:0/2527334282 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e98008470 msgr2=0x7f0e9800a930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:20:25.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eb4d99640 1 --2- 172.21.15.37:0/2527334282 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e98008470 0x7f0e9800a930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:25.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eb4d99640 1 -- 172.21.15.37:0/2527334282 >> v2:172.21.15.37:3300/0 conn(0x7f0e98005960 msgr2=0x7f0e98007de0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:20:25.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eb4d99640 1 --2- 172.21.15.37:0/2527334282 >> v2:172.21.15.37:3300/0 conn(0x7f0e98005960 0x7f0e98007de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:25.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eb6d9d640 1 -- 172.21.15.37:0/2527334282 reap_dead start 2024-04-26T00:20:25.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eb4d99640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-04-26T00:20:25.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eb4d99640 1 --2- 172.21.15.37:0/2527334282 >> v2:172.21.15.45:3300/0 conn(0x7f0e98005dd0 0x7f0e98008920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:25.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eb4d99640 10 monclient(hunting): picked mon.noname-c con 0x7f0e98005dd0 addr v2:172.21.15.45:3300/0 2024-04-26T00:20:25.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eb4d99640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0e98008e60 addr v1:172.21.15.45:6789/0 2024-04-26T00:20:25.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eb4d99640 1 --2- 172.21.15.37:0/2527334282 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e9800a960 0x7f0e9800f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:25.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eb4d99640 10 monclient(hunting): picked mon.noname-b con 0x7f0e9800a960 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:25.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eb4d99640 1 -- 172.21.15.37:0/2527334282 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0eb809dd10 con 0x7f0e98008e60 2024-04-26T00:20:25.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eb4d99640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:25.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eb4d99640 1 -- 172.21.15.37:0/2527334282 --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f0eb809ba60 con 0x7f0e98005dd0 2024-04-26T00:20:25.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eb4d99640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:25.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eb4d99640 1 -- 172.21.15.37:0/2527334282 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0e9800b040 con 0x7f0e9800a960 2024-04-26T00:20:25.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eb4d99640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:25.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eb659c640 1 -- 172.21.15.37:0/2527334282 >> v2:172.21.15.45:3300/0 conn(0x7f0e98005dd0 msgr2=0x7f0e98008920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:20:25.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eaffff640 1 -- 172.21.15.37:0/2527334282 >> v1:172.21.15.45:6789/0 conn(0x7f0e98008e60 legacy=0x7f0e98009240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:20:25.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eb659c640 1 --2- 172.21.15.37:0/2527334282 >> v2:172.21.15.45:3300/0 conn(0x7f0e98005dd0 0x7f0e98008920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:20:25.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eb6d9d640 1 --2- 172.21.15.37:0/2527334282 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e9800a960 0x7f0e9800f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:25.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eb6d9d640 10 monclient(hunting): get_auth_request con 0x7f0e9800a960 auth_method 0 2024-04-26T00:20:25.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eb6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:25.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eb6d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:25.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eb6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:25.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eb6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:25.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eb6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:25.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eb6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:25.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eb6d9d640 10 monclient(hunting): handle_auth_done global_id 4379 payload 274 2024-04-26T00:20:25.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eb6d9d640 1 -- 172.21.15.37:0/2527334282 >> v2:172.21.15.45:3300/0 conn(0x7f0e98005dd0 msgr2=0x7f0e98008920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:20:25.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eb6d9d640 1 --2- 172.21.15.37:0/2527334282 >> v2:172.21.15.45:3300/0 conn(0x7f0e98005dd0 0x7f0e98008920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:25.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eb6d9d640 1 -- 172.21.15.37:0/2527334282 >> v1:172.21.15.45:6789/0 conn(0x7f0e98008e60 legacy=0x7f0e98009240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:20:25.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eb6d9d640 10 monclient: _finish_hunting 0 2024-04-26T00:20:25.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eb6d9d640 1 monclient: found mon.noname-b 2024-04-26T00:20:25.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eb6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:25.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eb6d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:20:25.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.767+0000 7f0eb6d9d640 1 -- 172.21.15.37:0/2527334282 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eb8072d00 con 0x7f0e9800a960 2024-04-26T00:20:25.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.775+0000 7f0eb6d9d640 1 -- 172.21.15.37:0/2527334282 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e9800a960 msgr2=0x7f0e9800f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:25.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.775+0000 7f0eb6d9d640 1 --2- 172.21.15.37:0/2527334282 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e9800a960 0x7f0e9800f4c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0eb809f160 tx=0x7f0eb001c7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=227b399506168abd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:25.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.775+0000 7f0eb5d9b640 1 -- 172.21.15.37:0/2527334282 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb00af070 con 0x7f0e9800a960 2024-04-26T00:20:25.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.775+0000 7f0eb5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:25.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.775+0000 7f0eb5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:20:25.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.775+0000 7f0eb5d9b640 10 monclient: dump: 2024-04-26T00:20:25.755 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:25.755 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:25.755 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:25.755 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:25.755 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:25.755 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:25.755 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:25.755 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:25.755 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:25.755 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:25.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.775+0000 7f0eb5d9b640 10 monclient: _finish_auth 0 2024-04-26T00:20:25.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.775+0000 7f0eb5d9b640 10 monclient: _check_auth_tickets 2024-04-26T00:20:25.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.775+0000 7f0eb5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:25.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.775+0000 7f0eb5d9b640 1 -- 172.21.15.37:0/2527334282 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0eb0007350 con 0x7f0e9800a960 2024-04-26T00:20:25.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.775+0000 7f0eb5d9b640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:25.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.775+0000 7f0eb5d9b640 1 -- 172.21.15.37:0/2527334282 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb0007700 con 0x7f0e9800a960 2024-04-26T00:20:25.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.775+0000 7f0eb5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:25.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.775+0000 7f0eb5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:25.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.775+0000 7f0eb5d9b640 10 monclient: dump: 2024-04-26T00:20:25.756 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:25.756 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:25.756 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:25.756 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:25.756 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:25.757 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:25.757 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:25.757 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:25.757 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:25.757 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:25.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.775+0000 7f0ebcfb6640 5 monclient: authenticate success, global_id 4379 2024-04-26T00:20:25.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.775+0000 7f0ebcfb6640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:25.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.775+0000 7f0ebcfb6640 4 set_mon_vals no callback set 2024-04-26T00:20:25.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.775+0000 7f0ebcfb6640 10 monclient: shutdown 2024-04-26T00:20:25.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.775+0000 7f0ebcfb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:25.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.775+0000 7f0ebcfb6640 1 -- 172.21.15.37:0/2527334282 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e9800a960 msgr2=0x7f0e9800f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:25.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.775+0000 7f0ebcfb6640 1 --2- 172.21.15.37:0/2527334282 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e9800a960 0x7f0e9800f4c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0eb809f160 tx=0x7f0eb001c7d0 comp rx=0 tx=0).stop 2024-04-26T00:20:25.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0ebcfb6640 1 -- 172.21.15.37:0/2527334282 shutdown_connections 2024-04-26T00:20:25.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0ebcfb6640 1 --2- 172.21.15.37:0/2527334282 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e9800a960 0x7f0e9800f4c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:25.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0ebcfb6640 1 --2- 172.21.15.37:0/2527334282 >> v2:172.21.15.45:3300/0 conn(0x7f0e98005dd0 0x7f0e98008920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:25.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0ebcfb6640 1 -- 172.21.15.37:0/2527334282 >> 172.21.15.37:0/2527334282 conn(0x7f0eb80a1ad0 msgr2=0x7f0eb80a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:25.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0ebcfb6640 1 -- 172.21.15.37:0/2527334282 shutdown_connections 2024-04-26T00:20:25.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0ebcfb6640 1 -- 172.21.15.37:0/2527334282 wait complete. 2024-04-26T00:20:25.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0ebcfb6640 10 monclient: build_initial_monmap 2024-04-26T00:20:25.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0ebcfb6640 10 monclient: monmap: 2024-04-26T00:20:25.764 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:25.764 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:25.764 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:25.764 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:25.764 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:25.764 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:25.764 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:25.764 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:25.764 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:25.764 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:25.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0ebcfb6640 1 Processor -- start 2024-04-26T00:20:25.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0ebcfb6640 1 -- start start 2024-04-26T00:20:25.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0ebcfb6640 10 monclient: init 2024-04-26T00:20:25.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0ebcfb6640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:25.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0ebcfb6640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:20:25.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0ebcfb6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e98005dd0 0x7f0eb811b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:25.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0ebcfb6640 10 monclient(hunting): picked mon.noname-b con 0x7f0e98005dd0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:25.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0ebcfb6640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0e98008e60 0x7f0eb81cf970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:25.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0ebcfb6640 10 monclient(hunting): picked mon.noname-a con 0x7f0e98008e60 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:25.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0ebcfb6640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e9800a960 0x7f0eb81cfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:25.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0ebcfb6640 10 monclient(hunting): picked mon.noname-c con 0x7f0e9800a960 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:25.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0ebcfb6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:25.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0ebcfb6640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f0eb81308f0 con 0x7f0e98008e60 2024-04-26T00:20:25.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0ebcfb6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:25.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0ebcfb6640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0eb8130770 con 0x7f0e98005dd0 2024-04-26T00:20:25.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0ebcfb6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:25.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0ebcfb6640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0eb8130a70 con 0x7f0e9800a960 2024-04-26T00:20:25.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0ebcfb6640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:25.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0ebcfb6640 10 monclient(hunting): authenticate will time out at 1007.087053s 2024-04-26T00:20:25.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0eb659c640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e98005dd0 msgr2=0x7f0eb811b620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:20:25.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0eaffff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0e98008e60 0x7f0eb81cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:25.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0eb659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e98005dd0 0x7f0eb811b620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:20:25.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0eb6d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e9800a960 0x7f0eb81cfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:25.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0eaffff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0e98008e60 0x7f0eb81cf970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:43716/0 (socket says 172.21.15.37:43716) 2024-04-26T00:20:25.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0eaffff640 1 -- 172.21.15.37:0/2530480912 learned_addr learned my addr 172.21.15.37:0/2530480912 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:25.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0eaffff640 10 monclient(hunting): get_auth_request con 0x7f0e98008e60 auth_method 0 2024-04-26T00:20:25.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0eaffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:25.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0eaffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:25.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0eaffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:25.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0eb6d9d640 10 monclient(hunting): get_auth_request con 0x7f0e9800a960 auth_method 0 2024-04-26T00:20:25.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0eb6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:25.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0eb6d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:25.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0eb6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:25.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0eaffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:25.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0eaffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:25.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0eaffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:25.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0eb6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:25.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0eb6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:25.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0eb6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:25.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0eaffff640 10 monclient(hunting): handle_auth_done global_id 4383 payload 824 2024-04-26T00:20:25.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0eaffff640 1 -- 172.21.15.37:0/2530480912 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e9800a960 msgr2=0x7f0eb81cfeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:25.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0eaffff640 1 --2- 172.21.15.37:0/2530480912 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e9800a960 0x7f0eb81cfeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:25.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0eaffff640 1 -- 172.21.15.37:0/2530480912 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e98005dd0 msgr2=0x7f0eb811b620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:20:25.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0eaffff640 1 --2- 172.21.15.37:0/2530480912 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e98005dd0 0x7f0eb811b620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:25.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0eaffff640 10 monclient: _finish_hunting 0 2024-04-26T00:20:25.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0eaffff640 1 monclient: found mon.noname-a 2024-04-26T00:20:25.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0eaffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:25.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0eaffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:25.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0eaffff640 1 -- 172.21.15.37:0/2530480912 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eb811ca20 con 0x7f0e98008e60 2024-04-26T00:20:25.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.779+0000 7f0eb6d9d640 1 --2- 172.21.15.37:0/2530480912 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e9800a960 0x7f0eb81cfeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:25.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.783+0000 7f0eaffff640 1 -- 172.21.15.37:0/2530480912 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0e98008e60 msgr2=0x7f0eb81cf970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:25.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.783+0000 7f0eaffff640 1 --2- 172.21.15.37:0/2530480912 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0e98008e60 0x7f0eb81cf970 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0ea800bb70 tx=0x7f0ea8005480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:25.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.783+0000 7f0eaeffd640 1 -- 172.21.15.37:0/2530480912 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ea8006e60 con 0x7f0e98008e60 2024-04-26T00:20:25.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.783+0000 7f0eaeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:25.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.783+0000 7f0eaeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:25.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.783+0000 7f0eaeffd640 10 monclient: dump: 2024-04-26T00:20:25.769 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:25.769 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:25.769 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:25.769 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:25.769 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:25.769 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:25.769 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:25.769 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:25.769 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:25.769 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:25.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.783+0000 7f0eaeffd640 10 monclient: _finish_auth 0 2024-04-26T00:20:25.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.783+0000 7f0eaeffd640 10 monclient: _check_auth_tickets 2024-04-26T00:20:25.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.783+0000 7f0eaeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:25.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.783+0000 7f0eaeffd640 1 -- 172.21.15.37:0/2530480912 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ea8017600 con 0x7f0e98008e60 2024-04-26T00:20:25.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.783+0000 7f0eaeffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:25.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.783+0000 7f0eaeffd640 1 -- 172.21.15.37:0/2530480912 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ea80179f0 con 0x7f0e98008e60 2024-04-26T00:20:25.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.783+0000 7f0eaeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:25.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.783+0000 7f0eaeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:25.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.783+0000 7f0eaeffd640 10 monclient: dump: 2024-04-26T00:20:25.770 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:25.770 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:25.770 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:25.770 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:25.771 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:25.771 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:25.771 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:25.771 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:25.771 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:25.771 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:25.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.783+0000 7f0ebcfb6640 5 monclient: authenticate success, global_id 4383 2024-04-26T00:20:25.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.783+0000 7f0ebcfb6640 10 monclient: _renew_subs 2024-04-26T00:20:25.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.783+0000 7f0eb4d99640 4 set_mon_vals no callback set 2024-04-26T00:20:25.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.783+0000 7f0ebcfb6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:25.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.783+0000 7f0ebcfb6640 1 -- 172.21.15.37:0/2530480912 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0eb811cd00 con 0x7f0e98008e60 2024-04-26T00:20:25.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.783+0000 7f0ebcfb6640 10 monclient: _renew_subs 2024-04-26T00:20:25.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.783+0000 7f0ebcfb6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:25.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.783+0000 7f0ebcfb6640 1 -- 172.21.15.37:0/2530480912 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0eb81c1570 con 0x7f0e98008e60 2024-04-26T00:20:25.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.783+0000 7f0eaeffd640 1 -- 172.21.15.37:0/2530480912 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f0ea8017c10 con 0x7f0e98008e60 2024-04-26T00:20:25.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.783+0000 7f0ebcfb6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:25.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.783+0000 7f0ebcfb6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:25.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.783+0000 7f0ebcfb6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:25.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.783+0000 7f0ebcfb6640 1 -- 172.21.15.37:0/2530480912 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0eb8066f40 con 0x7f0e98008e60 2024-04-26T00:20:25.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.783+0000 7f0eaeffd640 1 --2- 172.21.15.37:0/2530480912 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0e98037b00 0x7f0e98039fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:25.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.783+0000 7f0eaeffd640 1 -- 172.21.15.37:0/2530480912 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 3858+0+0 (secure 0 0 0) 0x7f0ea8049a90 con 0x7f0e98008e60 2024-04-26T00:20:25.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.787+0000 7f0eb659c640 1 --2- 172.21.15.37:0/2530480912 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0e98037b00 0x7f0e98039fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:25.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.787+0000 7f0eb659c640 10 monclient: get_auth_request con 0x7f0e98037b00 auth_method 0 2024-04-26T00:20:25.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.787+0000 7f0eb659c640 1 --2- 172.21.15.37:0/2530480912 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0e98037b00 0x7f0e98039fc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0006d80 tx=0x7f0ea0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:25.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.791+0000 7f0eaeffd640 1 -- 172.21.15.37:0/2530480912 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f0ea8048820 con 0x7f0e98008e60 2024-04-26T00:20:25.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.791+0000 7f0eaeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:25.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.791+0000 7f0eaeffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:25.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.819+0000 7f6e86ffd640 1 -- 172.21.15.37:0/743788398 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]=0 enabled application 'rados' on pool 'unique_pool_0' v17) ==== 197+0+0 (secure 0 0 0) 0x7f6e88014a20 con 0x7f6e90072990 2024-04-26T00:20:25.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.819+0000 7f6e86ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}] 2024-04-26T00:20:25.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.819+0000 7f6e86ffd640 10 monclient: _finish_command 2 = system:0 enabled application 'rados' on pool 'unique_pool_0' 2024-04-26T00:20:25.798 INFO:teuthology.orchestra.run.smithi037.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-04-26T00:20:25.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.823+0000 7f6e983eb640 1 -- 172.21.15.37:0/743788398 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6e74035e80 msgr2=0x7f6e74038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:25.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.823+0000 7f6e983eb640 1 --2- 172.21.15.37:0/743788398 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6e74035e80 0x7f6e74038340 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c0ac000 tx=0x7f6e8c008040 comp rx=0 tx=0).stop 2024-04-26T00:20:25.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.823+0000 7f6e983eb640 10 monclient: shutdown 2024-04-26T00:20:25.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.823+0000 7f6e983eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:25.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.823+0000 7f6e983eb640 1 -- 172.21.15.37:0/743788398 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6e90072990 msgr2=0x7f6e90134e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:25.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.823+0000 7f6e983eb640 1 --2- 172.21.15.37:0/743788398 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6e90072990 0x7f6e90134e10 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6e8800b690 tx=0x7f6e8800bb50 comp rx=0 tx=0).stop 2024-04-26T00:20:25.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.823+0000 7f6e983eb640 1 -- 172.21.15.37:0/743788398 shutdown_connections 2024-04-26T00:20:25.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.823+0000 7f6e983eb640 1 --2- 172.21.15.37:0/743788398 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6e74035e80 0x7f6e74038340 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:25.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.823+0000 7f6e983eb640 1 --2- 172.21.15.37:0/743788398 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6e90072990 0x7f6e90134e10 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:25.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.823+0000 7f6e983eb640 1 --2- 172.21.15.37:0/743788398 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e90072590 0x7f6e901348d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:25.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.823+0000 7f6e983eb640 1 --2- 172.21.15.37:0/743788398 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e900721b0 0x7f6e90134390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:25.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.823+0000 7f6e983eb640 1 -- 172.21.15.37:0/743788398 >> 172.21.15.37:0/743788398 conn(0x7f6e900a6180 msgr2=0x7f6e900a0480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:25.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.827+0000 7f6e983eb640 1 -- 172.21.15.37:0/743788398 shutdown_connections 2024-04-26T00:20:25.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:25.827+0000 7f6e983eb640 1 -- 172.21.15.37:0/743788398 wait complete. 2024-04-26T00:20:26.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.027+0000 7f0ebcfb6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:20:26.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.027+0000 7f0ebcfb6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:20:26.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.027+0000 7f0ebcfb6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:26.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.027+0000 7f0ebcfb6640 1 -- 172.21.15.37:0/2530480912 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0eb809b1c0 con 0x7f0e98008e60 2024-04-26T00:20:26.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.027+0000 7f0eaeffd640 1 -- 172.21.15.37:0/2530480912 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7f0ea80205e0 con 0x7f0e98008e60 2024-04-26T00:20:26.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.027+0000 7f0eaeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:20:26.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.027+0000 7f0eaeffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:26.007 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":10,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:20:26.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.031+0000 7f0ebcfb6640 1 -- 172.21.15.37:0/2530480912 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0e98037b00 msgr2=0x7f0e98039fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:26.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.031+0000 7f0ebcfb6640 1 --2- 172.21.15.37:0/2530480912 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0e98037b00 0x7f0e98039fc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0006d80 tx=0x7f0ea0008040 comp rx=0 tx=0).stop 2024-04-26T00:20:26.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.031+0000 7f0ebcfb6640 10 monclient: shutdown 2024-04-26T00:20:26.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.031+0000 7f0ebcfb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:26.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.031+0000 7f0ebcfb6640 1 -- 172.21.15.37:0/2530480912 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0e98008e60 msgr2=0x7f0eb81cf970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:26.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.031+0000 7f0ebcfb6640 1 --2- 172.21.15.37:0/2530480912 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0e98008e60 0x7f0eb81cf970 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0ea800bb70 tx=0x7f0ea8005480 comp rx=0 tx=0).stop 2024-04-26T00:20:26.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.031+0000 7f0ebcfb6640 1 -- 172.21.15.37:0/2530480912 shutdown_connections 2024-04-26T00:20:26.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.031+0000 7f0ebcfb6640 1 --2- 172.21.15.37:0/2530480912 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0e98037b00 0x7f0e98039fc0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:26.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.031+0000 7f0ebcfb6640 1 --2- 172.21.15.37:0/2530480912 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e9800a960 0x7f0eb81cfeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:26.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.031+0000 7f0ebcfb6640 1 --2- 172.21.15.37:0/2530480912 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0e98008e60 0x7f0eb81cf970 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:26.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.031+0000 7f0ebcfb6640 1 --2- 172.21.15.37:0/2530480912 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e98005dd0 0x7f0eb811b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:26.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.031+0000 7f0ebcfb6640 1 -- 172.21.15.37:0/2530480912 >> 172.21.15.37:0/2530480912 conn(0x7f0eb80a1ad0 msgr2=0x7f0eb812fc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:26.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.031+0000 7f0ebcfb6640 1 -- 172.21.15.37:0/2530480912 shutdown_connections 2024-04-26T00:20:26.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.031+0000 7f0ebcfb6640 1 -- 172.21.15.37:0/2530480912 wait complete. 2024-04-26T00:20:26.025 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-26T00:20:26.026 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-04-26T00:20:26.026 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:20:26.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.163+0000 7fe79db6a640 10 monclient: get_monmap_and_config 2024-04-26T00:20:26.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.163+0000 7fe79db6a640 10 monclient: build_initial_monmap 2024-04-26T00:20:26.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.163+0000 7fe79db6a640 10 monclient: monmap: 2024-04-26T00:20:26.141 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:26.141 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:26.141 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:26.164755+0000 2024-04-26T00:20:26.141 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:26.164755+0000 2024-04-26T00:20:26.141 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:26.141 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:26.141 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:26.141 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:26.142 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:26.142 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:26.142 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:26.142 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:26.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.163+0000 7fe79db6a640 1 Processor -- start 2024-04-26T00:20:26.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.163+0000 7fe79db6a640 1 -- start start 2024-04-26T00:20:26.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.163+0000 7fe79db6a640 10 monclient: init 2024-04-26T00:20:26.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.163+0000 7fe79db6a640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:26.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.163+0000 7fe79db6a640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-04-26T00:20:26.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.163+0000 7fe79db6a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe798133f70 addr v1:172.21.15.37:6789/0 2024-04-26T00:20:26.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.163+0000 7fe79db6a640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe798134350 0x7fe79813c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:26.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.163+0000 7fe79db6a640 10 monclient(hunting): picked mon.noname-c con 0x7fe798134350 addr v2:172.21.15.45:3300/0 2024-04-26T00:20:26.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.163+0000 7fe79db6a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe798134790 addr v1:172.21.15.45:6789/0 2024-04-26T00:20:26.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.163+0000 7fe79db6a640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7980a4fa0 con 0x7fe798133f70 2024-04-26T00:20:26.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.163+0000 7fe79db6a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7980a51e0 con 0x7fe798134790 2024-04-26T00:20:26.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.163+0000 7fe79db6a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:26.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.163+0000 7fe79db6a640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fe79809b280 con 0x7fe798134350 2024-04-26T00:20:26.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.163+0000 7fe79db6a640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:26.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.163+0000 7fe79db6a640 10 monclient(hunting): authenticate will time out at 1007.470855s 2024-04-26T00:20:26.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.163+0000 7fe78effd640 1 -- >> v2:172.21.15.45:3300/0 conn(0x7fe798134350 msgr2=0x7fe79813c0c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:20:26.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.163+0000 7fe797fff640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7fe798134790 legacy=0x7fe79813e610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:20:26.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.163+0000 7fe78effd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe798134350 0x7fe79813c0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:20:26.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.163+0000 7fe7977fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fe798133f70 0x7fe79809c140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:41032/0 (socket says 172.21.15.37:41032) 2024-04-26T00:20:26.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.163+0000 7fe7977fe640 1 -- 172.21.15.37:0/3239588660 learned_addr learned my addr 172.21.15.37:0/3239588660 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:20:26.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe7977fe640 1 -- 172.21.15.37:0/3239588660 >> v1:172.21.15.37:6789/0 conn(0x7fe798133f70 legacy=0x7fe79809c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:26.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe796ffd640 1 -- 172.21.15.37:0/3239588660 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4063723501 0 0) 0x7fe7980a4fa0 con 0x7fe798133f70 2024-04-26T00:20:26.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe796ffd640 10 monclient(hunting): _negotiate 2024-04-26T00:20:26.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe796ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:26.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe796ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:26.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe796ffd640 10 monclient(hunting): my global_id is 4386 2024-04-26T00:20:26.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe796ffd640 1 -- 172.21.15.37:0/3239588660 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe784003530 con 0x7fe798133f70 2024-04-26T00:20:26.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe796ffd640 1 -- 172.21.15.37:0/3239588660 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 276721371 0 0) 0x7fe784003530 con 0x7fe798133f70 2024-04-26T00:20:26.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe796ffd640 1 -- 172.21.15.37:0/3239588660 >> v2:172.21.15.45:3300/0 conn(0x7fe798134350 msgr2=0x7fe79813c0c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:20:26.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe796ffd640 1 --2- 172.21.15.37:0/3239588660 >> v2:172.21.15.45:3300/0 conn(0x7fe798134350 0x7fe79813c0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:26.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe796ffd640 1 -- 172.21.15.37:0/3239588660 >> v1:172.21.15.45:6789/0 conn(0x7fe798134790 legacy=0x7fe79813e610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:20:26.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe796ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:20:26.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe796ffd640 1 monclient: found mon.noname-a-legacy 2024-04-26T00:20:26.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe796ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:26.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe796ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.37:6789/0 2024-04-26T00:20:26.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe796ffd640 1 -- 172.21.15.37:0/3239588660 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe79806dd00 con 0x7fe798133f70 2024-04-26T00:20:26.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe796ffd640 10 monclient: _finish_auth 0 2024-04-26T00:20:26.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe796ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:20:26.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe796ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:26.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe79db6a640 5 monclient: authenticate success, global_id 4386 2024-04-26T00:20:26.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe79db6a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:20:26.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe796ffd640 1 -- 172.21.15.37:0/3239588660 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 866326386 0 0) 0x7fe788001920 con 0x7fe798133f70 2024-04-26T00:20:26.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe796ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:26.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe796ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-26T00:20:26.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe796ffd640 10 monclient: dump: 2024-04-26T00:20:26.151 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:26.151 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:26.151 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:26.151 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:26.151 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:26.151 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:26.151 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:26.151 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:26.152 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:26.152 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:26.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe796ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] but i'm connected to v1:172.21.15.37:6789/0, reconnecting 2024-04-26T00:20:26.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe796ffd640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:26.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe796ffd640 1 -- 172.21.15.37:0/3239588660 >> v1:172.21.15.37:6789/0 conn(0x7fe798133f70 legacy=0x7fe79809c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:26.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe796ffd640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:20:26.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe796ffd640 1 --2- 172.21.15.37:0/3239588660 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7840039c0 0x7fe784003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:26.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe796ffd640 10 monclient(hunting): picked mon.b con 0x7fe7840039c0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:26.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe796ffd640 1 --2- 172.21.15.37:0/3239588660 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe784004810 0x7fe784004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:26.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe796ffd640 10 monclient(hunting): picked mon.c con 0x7fe784004810 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:26.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe796ffd640 1 --2- 172.21.15.37:0/3239588660 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7840053d0 0x7fe78400ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:26.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe796ffd640 10 monclient(hunting): picked mon.a con 0x7fe7840053d0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:26.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe796ffd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:26.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe796ffd640 1 -- 172.21.15.37:0/3239588660 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe79809b280 con 0x7fe7840053d0 2024-04-26T00:20:26.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe796ffd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:26.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe796ffd640 1 -- 172.21.15.37:0/3239588660 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe78400c310 con 0x7fe7840039c0 2024-04-26T00:20:26.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe796ffd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:26.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe796ffd640 1 -- 172.21.15.37:0/3239588660 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe78400c510 con 0x7fe784004810 2024-04-26T00:20:26.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe796ffd640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:26.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe796ffd640 10 monclient(hunting): _finish_auth 0 2024-04-26T00:20:26.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe79db6a640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-26T00:20:26.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe797fff640 1 --2- 172.21.15.37:0/3239588660 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7840053d0 0x7fe78400ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:26.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe797fff640 10 monclient(hunting): get_auth_request con 0x7fe7840053d0 auth_method 0 2024-04-26T00:20:26.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe78effd640 1 --2- 172.21.15.37:0/3239588660 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe784004810 0x7fe784004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:26.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe797fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:26.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe797fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:26.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe797fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:20:26.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe7977fe640 1 -- 172.21.15.37:0/3239588660 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7840039c0 msgr2=0x7fe784003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:20:26.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe78effd640 10 monclient(hunting): get_auth_request con 0x7fe784004810 auth_method 0 2024-04-26T00:20:26.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe78effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:26.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe78effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:26.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe78effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:20:26.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe7977fe640 1 --2- 172.21.15.37:0/3239588660 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7840039c0 0x7fe784003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:20:26.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe797fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:26.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe797fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:26.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe797fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:20:26.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe78effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:26.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe78effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:26.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe78effd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:20:26.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe797fff640 10 monclient(hunting): handle_auth_done global_id 4386 payload 293 2024-04-26T00:20:26.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe797fff640 1 -- 172.21.15.37:0/3239588660 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe784004810 msgr2=0x7fe784004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:26.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe797fff640 1 --2- 172.21.15.37:0/3239588660 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe784004810 0x7fe784004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:26.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe797fff640 1 -- 172.21.15.37:0/3239588660 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7840039c0 msgr2=0x7fe784003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:20:26.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe797fff640 1 --2- 172.21.15.37:0/3239588660 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7840039c0 0x7fe784003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:26.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe797fff640 10 monclient: _finish_hunting 0 2024-04-26T00:20:26.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe797fff640 1 monclient: found mon.a 2024-04-26T00:20:26.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe797fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:26.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe797fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:26.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe797fff640 1 -- 172.21.15.37:0/3239588660 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe784003530 con 0x7fe7840053d0 2024-04-26T00:20:26.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe797fff640 10 monclient: _finish_auth 0 2024-04-26T00:20:26.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe797fff640 10 monclient: _check_auth_tickets 2024-04-26T00:20:26.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.167+0000 7fe797fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:26.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.171+0000 7fe797fff640 1 -- 172.21.15.37:0/3239588660 reap_dead start 2024-04-26T00:20:26.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.171+0000 7fe797fff640 1 -- 172.21.15.37:0/3239588660 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7840053d0 msgr2=0x7fe78400ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:26.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.171+0000 7fe797fff640 1 --2- 172.21.15.37:0/3239588660 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7840053d0 0x7fe78400ba30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe790009e30 tx=0x7fe79001c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0d5f57bba7a7e7d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:26.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.171+0000 7fe796ffd640 1 -- 172.21.15.37:0/3239588660 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe790009a30 con 0x7fe7840053d0 2024-04-26T00:20:26.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.171+0000 7fe796ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:26.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.171+0000 7fe796ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:26.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.171+0000 7fe796ffd640 10 monclient: dump: 2024-04-26T00:20:26.162 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:26.163 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:26.163 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:26.163 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:26.163 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:26.163 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:26.163 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:26.163 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:26.163 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:26.163 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:26.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.171+0000 7fe796ffd640 1 -- 172.21.15.37:0/3239588660 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7900a6740 con 0x7fe7840053d0 2024-04-26T00:20:26.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.171+0000 7fe796ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:26.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.171+0000 7fe796ffd640 1 -- 172.21.15.37:0/3239588660 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7900a6b30 con 0x7fe7840053d0 2024-04-26T00:20:26.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.171+0000 7fe79db6a640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:26.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.171+0000 7fe79db6a640 4 set_mon_vals no callback set 2024-04-26T00:20:26.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.171+0000 7fe796ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:26.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.171+0000 7fe796ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:26.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.171+0000 7fe796ffd640 10 monclient: dump: 2024-04-26T00:20:26.164 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:26.164 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:26.164 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:26.164 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:26.164 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:26.164 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:26.164 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:26.164 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:26.164 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:26.164 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:26.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.171+0000 7fe79db6a640 10 monclient: shutdown 2024-04-26T00:20:26.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.171+0000 7fe79db6a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:26.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.171+0000 7fe79db6a640 1 -- 172.21.15.37:0/3239588660 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7840053d0 msgr2=0x7fe78400ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:26.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.171+0000 7fe79db6a640 1 --2- 172.21.15.37:0/3239588660 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7840053d0 0x7fe78400ba30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe790009e30 tx=0x7fe79001c910 comp rx=0 tx=0).stop 2024-04-26T00:20:26.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe79db6a640 1 -- 172.21.15.37:0/3239588660 shutdown_connections 2024-04-26T00:20:26.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe79db6a640 1 --2- 172.21.15.37:0/3239588660 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7840053d0 0x7fe78400ba30 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:26.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe79db6a640 1 -- 172.21.15.37:0/3239588660 >> 172.21.15.37:0/3239588660 conn(0x7fe7980a5f00 msgr2=0x7fe7980a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:26.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe79db6a640 1 -- 172.21.15.37:0/3239588660 shutdown_connections 2024-04-26T00:20:26.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe79db6a640 1 -- 172.21.15.37:0/3239588660 wait complete. 2024-04-26T00:20:26.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe79db6a640 10 monclient: build_initial_monmap 2024-04-26T00:20:26.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe79db6a640 10 monclient: monmap: 2024-04-26T00:20:26.165 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:26.165 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:26.165 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:26.165 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:26.165 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:26.166 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:26.166 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:26.166 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:26.166 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:26.166 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:26.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe79db6a640 1 Processor -- start 2024-04-26T00:20:26.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe79db6a640 1 -- start start 2024-04-26T00:20:26.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe79db6a640 10 monclient: init 2024-04-26T00:20:26.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe79db6a640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:26.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe79db6a640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:20:26.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe79db6a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7840053d0 0x7fe7981cef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:26.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe79db6a640 10 monclient(hunting): picked mon.noname-a con 0x7fe7840053d0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:26.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe79db6a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe79813d980 0x7fe79813de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:26.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe79db6a640 10 monclient(hunting): picked mon.noname-b con 0x7fe79813d980 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:26.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe79db6a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe7981374b0 0x7fe798137910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:26.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe79db6a640 10 monclient(hunting): picked mon.noname-c con 0x7fe7981374b0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:26.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe79db6a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:26.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe79db6a640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe79809c4d0 con 0x7fe7840053d0 2024-04-26T00:20:26.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe79db6a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:26.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe79db6a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe79809c350 con 0x7fe79813d980 2024-04-26T00:20:26.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe79db6a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:26.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe79db6a640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe79809c650 con 0x7fe7981374b0 2024-04-26T00:20:26.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe79db6a640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:26.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe79db6a640 10 monclient(hunting): authenticate will time out at 1007.481538s 2024-04-26T00:20:26.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe7977fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7840053d0 0x7fe7981cef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:26.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe797fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe7981374b0 0x7fe798137910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:26.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe78effd640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe79813d980 msgr2=0x7fe79813de00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:20:26.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe7977fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7840053d0 0x7fe7981cef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:43724/0 (socket says 172.21.15.37:43724) 2024-04-26T00:20:26.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe797fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe7981374b0 0x7fe798137910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:55390/0 (socket says 172.21.15.37:55390) 2024-04-26T00:20:26.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe7977fe640 1 -- 172.21.15.37:0/1995954407 learned_addr learned my addr 172.21.15.37:0/1995954407 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:26.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe7977fe640 10 monclient(hunting): get_auth_request con 0x7fe7840053d0 auth_method 0 2024-04-26T00:20:26.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe7977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:26.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe7977fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:26.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe7977fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:26.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe78effd640 1 --2- 172.21.15.37:0/1995954407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe79813d980 0x7fe79813de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:20:26.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe797fff640 10 monclient(hunting): get_auth_request con 0x7fe7981374b0 auth_method 0 2024-04-26T00:20:26.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe797fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:26.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe797fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:26.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe797fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:26.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe7977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:26.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe7977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:26.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe7977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:26.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe797fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:26.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe797fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:26.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe797fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:26.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe7977fe640 10 monclient(hunting): handle_auth_done global_id 4389 payload 824 2024-04-26T00:20:26.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe7977fe640 1 -- 172.21.15.37:0/1995954407 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe7981374b0 msgr2=0x7fe798137910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:26.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe7977fe640 1 --2- 172.21.15.37:0/1995954407 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe7981374b0 0x7fe798137910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:26.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe7977fe640 1 -- 172.21.15.37:0/1995954407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe79813d980 msgr2=0x7fe79813de00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:20:26.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe7977fe640 1 --2- 172.21.15.37:0/1995954407 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe79813d980 0x7fe79813de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:26.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe7977fe640 10 monclient: _finish_hunting 0 2024-04-26T00:20:26.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe7977fe640 1 monclient: found mon.noname-a 2024-04-26T00:20:26.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe7977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:26.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe7977fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:26.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe7977fe640 1 -- 172.21.15.37:0/1995954407 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe79813b440 con 0x7fe7840053d0 2024-04-26T00:20:26.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.175+0000 7fe797fff640 1 --2- 172.21.15.37:0/1995954407 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe7981374b0 0x7fe798137910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:26.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe7977fe640 1 -- 172.21.15.37:0/1995954407 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7840053d0 msgr2=0x7fe7981cef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:26.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe7977fe640 1 --2- 172.21.15.37:0/1995954407 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7840053d0 0x7fe7981cef40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe78800c730 tx=0x7fe78800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:26.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe7977fe640 1 --2- 172.21.15.37:0/1995954407 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7840053d0 0x7fe7981cef40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe78800c730 tx=0x7fe78800cbf0 comp rx=0 tx=0).queue_received will delay after 1.71409e+09 on 0x7fe788018070 mon_map magic: 0 2024-04-26T00:20:26.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe794ff9640 1 -- 172.21.15.37:0/1995954407 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe788018070 con 0x7fe7840053d0 2024-04-26T00:20:26.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe794ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:26.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe794ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:26.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe794ff9640 10 monclient: dump: 2024-04-26T00:20:26.170 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:26.170 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:26.170 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:26.170 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:26.171 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:26.171 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:26.171 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:26.171 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:26.171 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:26.171 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:26.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe794ff9640 10 monclient: _finish_auth 0 2024-04-26T00:20:26.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe794ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:20:26.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe794ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:26.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe794ff9640 1 -- 172.21.15.37:0/1995954407 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7880074e0 con 0x7fe7840053d0 2024-04-26T00:20:26.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe794ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:26.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe79db6a640 5 monclient: authenticate success, global_id 4389 2024-04-26T00:20:26.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe79db6a640 10 monclient: _renew_subs 2024-04-26T00:20:26.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe79db6a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:26.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe79db6a640 1 -- 172.21.15.37:0/1995954407 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe79813f250 con 0x7fe7840053d0 2024-04-26T00:20:26.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe7967fc640 4 set_mon_vals no callback set 2024-04-26T00:20:26.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe79db6a640 10 monclient: _renew_subs 2024-04-26T00:20:26.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe79db6a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:26.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe79db6a640 1 -- 172.21.15.37:0/1995954407 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe79813b160 con 0x7fe7840053d0 2024-04-26T00:20:26.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe79db6a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:26.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe79db6a640 1 --2- 172.21.15.37:0/1995954407 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe79802d180 0x7fe798137e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:26.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe79db6a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:26.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe79db6a640 1 -- 172.21.15.37:0/1995954407 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe79809a9e0 con 0x7fe79802d180 2024-04-26T00:20:26.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe78effd640 1 --2- 172.21.15.37:0/1995954407 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe79802d180 0x7fe798137e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:26.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe78effd640 10 monclient: get_auth_request con 0x7fe79802d180 auth_method 0 2024-04-26T00:20:26.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe78effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:26.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe78effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:26.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe78effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:26.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe794ff9640 1 -- 172.21.15.37:0/1995954407 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7880049f0 con 0x7fe7840053d0 2024-04-26T00:20:26.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe794ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:26.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe794ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:26.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe794ff9640 10 monclient: dump: 2024-04-26T00:20:26.173 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:26.173 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:26.173 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:26.173 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:26.173 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:26.173 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:26.173 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:26.173 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:26.173 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:26.173 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:26.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe78effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:26.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe78effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:26.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe794ff9640 1 -- 172.21.15.37:0/1995954407 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe78801b070 con 0x7fe7840053d0 2024-04-26T00:20:26.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe78effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:26.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe78effd640 10 monclient(hunting): handle_auth_done global_id 4392 payload 824 2024-04-26T00:20:26.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe78effd640 1 -- 172.21.15.37:0/1995954407 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe79802d560 con 0x7fe79802d180 2024-04-26T00:20:26.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe794ff9640 1 --2- 172.21.15.37:0/1995954407 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe778035e60 0x7fe778038320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:26.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe797fff640 1 --2- 172.21.15.37:0/1995954407 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe778035e60 0x7fe778038320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:26.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.179+0000 7fe797fff640 10 monclient: get_auth_request con 0x7fe778035e60 auth_method 0 2024-04-26T00:20:26.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.183+0000 7fe78effd640 1 -- 172.21.15.37:0/1995954407 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe79802d180 msgr2=0x7fe798137e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:26.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.183+0000 7fe78effd640 1 --2- 172.21.15.37:0/1995954407 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe79802d180 0x7fe798137e50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe7981cfdc0 tx=0x7fe78000e9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:26.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.183+0000 7fe794ff9640 1 -- 172.21.15.37:0/1995954407 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe780016020 con 0x7fe79802d180 2024-04-26T00:20:26.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.183+0000 7fe794ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:26.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.183+0000 7fe794ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:26.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.183+0000 7fe794ff9640 10 monclient: dump: 2024-04-26T00:20:26.175 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:26.175 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:26.175 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:26.175 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:26.175 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:26.175 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:26.175 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:26.175 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:26.175 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:26.175 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:26.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.183+0000 7fe794ff9640 1 -- 172.21.15.37:0/1995954407 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe7800050d0 con 0x7fe79802d180 2024-04-26T00:20:26.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.183+0000 7fe794ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:26.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.183+0000 7fe794ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:26.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.183+0000 7fe794ff9640 1 -- 172.21.15.37:0/1995954407 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe79802d180 msgr2=0x7fe798137e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:26.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.183+0000 7fe794ff9640 1 --2- 172.21.15.37:0/1995954407 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe79802d180 0x7fe798137e50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe7981cfdc0 tx=0x7fe78000e9d0 comp rx=0 tx=0).stop 2024-04-26T00:20:26.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.183+0000 7fe794ff9640 1 -- 172.21.15.37:0/1995954407 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe79802d180 msgr2=0x7fe798137e50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:20:26.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.183+0000 7fe794ff9640 1 --2- 172.21.15.37:0/1995954407 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe79802d180 0x7fe798137e50 secure :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe7981cfdc0 tx=0x7fe78000e9d0 comp rx=0 tx=0).stop 2024-04-26T00:20:26.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.183+0000 7fe797fff640 1 --2- 172.21.15.37:0/1995954407 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe778035e60 0x7fe778038320 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe79809c8d0 tx=0x7fe790005d40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:26.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.187+0000 7fe79db6a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:20:26.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.187+0000 7fe79db6a640 1 --2- 172.21.15.37:0/1995954407 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7980623f0 0x7fe798062800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:26.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.187+0000 7fe79db6a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:26.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.187+0000 7fe79db6a640 1 -- 172.21.15.37:0/1995954407 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe79804f510 con 0x7fe7980623f0 2024-04-26T00:20:26.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.187+0000 7fe78effd640 1 --2- 172.21.15.37:0/1995954407 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7980623f0 0x7fe798062800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:26.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.187+0000 7fe78effd640 10 monclient: get_auth_request con 0x7fe7980623f0 auth_method 0 2024-04-26T00:20:26.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.187+0000 7fe78effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:26.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.187+0000 7fe78effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:26.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.187+0000 7fe78effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:26.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.187+0000 7fe78effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:26.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.187+0000 7fe78effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:26.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.187+0000 7fe78effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:26.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.187+0000 7fe78effd640 10 monclient(hunting): handle_auth_done global_id 4395 payload 824 2024-04-26T00:20:26.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.187+0000 7fe78effd640 1 -- 172.21.15.37:0/1995954407 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe79809d0e0 con 0x7fe7980623f0 2024-04-26T00:20:26.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.191+0000 7fe78effd640 1 -- 172.21.15.37:0/1995954407 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7980623f0 msgr2=0x7fe798062800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:26.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.191+0000 7fe78effd640 1 --2- 172.21.15.37:0/1995954407 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7980623f0 0x7fe798062800 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe780017000 tx=0x7fe780005a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:26.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.191+0000 7fe794ff9640 1 -- 172.21.15.37:0/1995954407 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe780016070 con 0x7fe7980623f0 2024-04-26T00:20:26.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.191+0000 7fe794ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:26.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.191+0000 7fe794ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:26.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.191+0000 7fe794ff9640 10 monclient: dump: 2024-04-26T00:20:26.179 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:26.179 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:26.180 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:26.180 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:26.180 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:26.180 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:26.180 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:26.180 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:26.180 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:26.180 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:26.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.191+0000 7fe794ff9640 1 -- 172.21.15.37:0/1995954407 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4877 (secure 0 0 0) 0x7fe78001d020 con 0x7fe7980623f0 2024-04-26T00:20:26.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.191+0000 7fe794ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:20:26.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.191+0000 7fe794ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:26.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.191+0000 7fe794ff9640 1 -- 172.21.15.37:0/1995954407 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7980623f0 msgr2=0x7fe798062800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:26.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.191+0000 7fe794ff9640 1 --2- 172.21.15.37:0/1995954407 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7980623f0 0x7fe798062800 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe780017000 tx=0x7fe780005a90 comp rx=0 tx=0).stop 2024-04-26T00:20:26.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.191+0000 7fe794ff9640 1 -- 172.21.15.37:0/1995954407 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7980623f0 msgr2=0x7fe798062800 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:20:26.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.191+0000 7fe794ff9640 1 --2- 172.21.15.37:0/1995954407 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7980623f0 0x7fe798062800 secure :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe780017000 tx=0x7fe780005a90 comp rx=0 tx=0).stop 2024-04-26T00:20:26.181 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:20:26.181 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:20:26.181 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:20:26.181 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:20:26.181 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 10, 2024-04-26T00:20:26.181 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 43295, 2024-04-26T00:20:26.181 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:20:26.181 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:20:26.181 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:20:26.181 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:26.182 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 1, 2024-04-26T00:20:26.182 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:20:26.182 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:20:26.182 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:20:26.182 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:20:26.182 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:20:26.182 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:20:26.182 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:20:26.182 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:20:26.182 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:20:26.182 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:20:26.182 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:20:26.182 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:20:26.182 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:20:26.183 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:20:26.183 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:26.183 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:20:26.183 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:20:26.183 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:20:26.183 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:20:26.183 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:20:26.183 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:20:26.183 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:20:26.183 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:20:26.183 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:20:26.183 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:20:26.183 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:20:26.184 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:20:26.184 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:20:26.184 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:26.184 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:26.184 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:20:26.184 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:20:26.184 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:20:26.184 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:20:26.184 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:20:26.184 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:20:26.184 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:20:26.184 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:20:26.185 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:20:26.185 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:20:26.185 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:20:26.185 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:20:26.185 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:20:26.185 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:20:26.185 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:20:26.185 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:20:26.185 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:20:26.185 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:20:26.185 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:20:26.185 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:20:26.185 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:20:26.186 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:20:26.186 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:20:26.186 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:20:26.186 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:20:26.186 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:20:26.186 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:20:26.186 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:20:26.186 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:26.186 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:20:26.186 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:26.186 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:20:26.186 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:26.186 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:20:26.187 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:20:26.187 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:20:26.187 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:20:26.187 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:26.187 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:20:26.187 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:20:26.187 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:26.187 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:26.187 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:26.187 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:20:26.187 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:20:26.187 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:26.187 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:26.187 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:26.187 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:26.188 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:20:26.188 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:20:26.188 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:20:26.188 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:20:26.188 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:20:26.188 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:26.188 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:26.188 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:20:26.188 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:20:26.188 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:20:26.188 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:20:26.188 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:26.188 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:20:26.188 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:20:26.189 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:26.189 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:26.189 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:26.189 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:20:26.189 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:20:26.189 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:26.189 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:26.189 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:26.189 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:26.189 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:20:26.189 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:20:26.189 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:20:26.190 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:20:26.190 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:20:26.190 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:26.190 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:26.190 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:20:26.190 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:20:26.190 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:20:26.190 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:20:26.190 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:26.190 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:20:26.191 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:20:26.191 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:26.191 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:26.191 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:26.191 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:20:26.191 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:20:26.191 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:26.191 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:26.191 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:26.191 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:26.191 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:20:26.191 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:20:26.191 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:20:26.191 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:20:26.191 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:20:26.192 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:26.192 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:26.192 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:26.192 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:20:26.192 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:20:26.192 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:26.192 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:20:26.192 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:20:26.192 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:20:26.192 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:26.192 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:26.192 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:20:26.192 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:26.192 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:20:26.192 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:20:26.193 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:20:26.193 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:26.193 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:26.193 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:20:26.193 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:26.193 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:20:26.193 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:20:26.193 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:20:26.193 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:26.193 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:26.193 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:20:26.193 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:26.193 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:20:26.193 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:20:26.193 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:20:26.194 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:26.194 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:26.194 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:26.194 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:20:26.194 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:20:26.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.191+0000 7fe79db6a640 1 -- 172.21.15.37:0/1995954407 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe778035e60 msgr2=0x7fe778038320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:26.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.191+0000 7fe79db6a640 1 --2- 172.21.15.37:0/1995954407 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe778035e60 0x7fe778038320 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe79809c8d0 tx=0x7fe790005d40 comp rx=0 tx=0).stop 2024-04-26T00:20:26.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.191+0000 7fe79db6a640 10 monclient: shutdown 2024-04-26T00:20:26.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.191+0000 7fe79db6a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:26.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.191+0000 7fe79db6a640 1 -- 172.21.15.37:0/1995954407 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7840053d0 msgr2=0x7fe7981cef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:26.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.191+0000 7fe79db6a640 1 --2- 172.21.15.37:0/1995954407 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7840053d0 0x7fe7981cef40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe78800c730 tx=0x7fe78800cbf0 comp rx=0 tx=0).stop 2024-04-26T00:20:26.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.191+0000 7fe797fff640 1 -- 172.21.15.37:0/1995954407 reap_dead start 2024-04-26T00:20:26.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.191+0000 7fe794ff9640 1 -- 172.21.15.37:0/1995954407 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 3871+0+0 (unknown 0 0 0) 0x7fe788048020 con 0x7fe7840053d0 2024-04-26T00:20:26.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.191+0000 7fe79db6a640 1 -- 172.21.15.37:0/1995954407 shutdown_connections 2024-04-26T00:20:26.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.191+0000 7fe79db6a640 1 --2- 172.21.15.37:0/1995954407 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7840053d0 0x7fe7981cef40 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:26.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.191+0000 7fe79db6a640 1 -- 172.21.15.37:0/1995954407 >> 172.21.15.37:0/1995954407 conn(0x7fe7980a5f00 msgr2=0x7fe79809fad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:26.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.191+0000 7fe79db6a640 1 -- 172.21.15.37:0/1995954407 shutdown_connections 2024-04-26T00:20:26.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.191+0000 7fe79db6a640 1 -- 172.21.15.37:0/1995954407 wait complete. 2024-04-26T00:20:26.196 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:20:26.197 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-04-26T00:20:26.197 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:20:26.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.339+0000 7f68227a9640 10 monclient: get_monmap_and_config 2024-04-26T00:20:26.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.339+0000 7f68227a9640 10 monclient: build_initial_monmap 2024-04-26T00:20:26.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.339+0000 7f68227a9640 10 monclient: monmap: 2024-04-26T00:20:26.320 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:26.320 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:26.320 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:26.343830+0000 2024-04-26T00:20:26.321 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:26.343830+0000 2024-04-26T00:20:26.321 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:26.321 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:26.321 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:26.321 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:26.321 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:26.322 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:26.322 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:26.322 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:26.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f68227a9640 1 Processor -- start 2024-04-26T00:20:26.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f68227a9640 1 -- start start 2024-04-26T00:20:26.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f68227a9640 10 monclient: init 2024-04-26T00:20:26.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f68227a9640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:26.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f68227a9640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-04-26T00:20:26.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f68227a9640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f681c072180 0x7f681c072560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:26.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f68227a9640 10 monclient(hunting): picked mon.noname-b con 0x7f681c072180 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:26.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f68227a9640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f681c071670 0x7f681c0a13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:26.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f68227a9640 10 monclient(hunting): picked mon.noname-c con 0x7f681c071670 addr v2:172.21.15.45:3300/0 2024-04-26T00:20:26.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f68227a9640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f681c071a50 0x7f681c0a1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:26.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f68227a9640 10 monclient(hunting): picked mon.noname-a con 0x7f681c071a50 addr v2:172.21.15.37:3300/0 2024-04-26T00:20:26.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f68227a9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:26.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f68227a9640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f681c0702a0 con 0x7f681c071a50 2024-04-26T00:20:26.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f68227a9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:26.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f68227a9640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f681c070580 con 0x7f681c071670 2024-04-26T00:20:26.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f68227a9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:26.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f68227a9640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f681c070860 con 0x7f681c072180 2024-04-26T00:20:26.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f68227a9640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:26.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f68227a9640 10 monclient(hunting): authenticate will time out at 1007.649971s 2024-04-26T00:20:26.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f681b7fe640 1 -- >> v2:172.21.15.45:3300/0 conn(0x7f681c071670 msgr2=0x7f681c0a13d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:20:26.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f681b7fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f681c071670 0x7f681c0a13d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:20:26.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f6820d46640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f681c071a50 0x7f681c0a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:26.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f6820d46640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f681c071a50 0x7f681c0a1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:43744/0 (socket says 172.21.15.37:43744) 2024-04-26T00:20:26.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f6820d46640 1 -- 172.21.15.37:0/3531031838 learned_addr learned my addr 172.21.15.37:0/3531031838 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:26.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f6820d46640 10 monclient(hunting): get_auth_request con 0x7f681c071a50 auth_method 0 2024-04-26T00:20:26.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f6820d46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:26.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f6820d46640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:26.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f6820d46640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:26.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f681bfff640 1 --2- 172.21.15.37:0/3531031838 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f681c072180 0x7f681c072560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:26.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f681bfff640 10 monclient(hunting): get_auth_request con 0x7f681c072180 auth_method 0 2024-04-26T00:20:26.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f681bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:26.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f681bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:26.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f681bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:26.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f6820d46640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:26.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f6820d46640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:26.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f6820d46640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:26.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f681bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:26.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f681bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:26.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f681bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:26.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f6820d46640 10 monclient(hunting): handle_auth_done global_id 4398 payload 274 2024-04-26T00:20:26.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f6820d46640 1 -- 172.21.15.37:0/3531031838 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f681c072180 msgr2=0x7f681c072560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:26.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f6820d46640 1 --2- 172.21.15.37:0/3531031838 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f681c072180 0x7f681c072560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:26.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f6820d46640 1 -- 172.21.15.37:0/3531031838 >> v2:172.21.15.45:3300/0 conn(0x7f681c071670 msgr2=0x7f681c0a13d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:20:26.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f6820d46640 1 --2- 172.21.15.37:0/3531031838 >> v2:172.21.15.45:3300/0 conn(0x7f681c071670 0x7f681c0a13d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:26.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f6820d46640 10 monclient: _finish_hunting 0 2024-04-26T00:20:26.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f6820d46640 1 monclient: found mon.noname-a 2024-04-26T00:20:26.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f6820d46640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:26.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f6820d46640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:26.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.343+0000 7f6820d46640 1 -- 172.21.15.37:0/3531031838 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f681c072aa0 con 0x7f681c071a50 2024-04-26T00:20:26.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.347+0000 7f6820d46640 1 -- 172.21.15.37:0/3531031838 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f681c071a50 msgr2=0x7f681c0a1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:26.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.347+0000 7f6820d46640 1 --2- 172.21.15.37:0/3531031838 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f681c071a50 0x7f681c0a1910 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f681c0a3580 tx=0x7f681401ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5762c46dcd127208 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:26.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.347+0000 7f681affd640 1 -- 172.21.15.37:0/3531031838 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f681400b2d0 con 0x7f681c071a50 2024-04-26T00:20:26.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.347+0000 7f681affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:26.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.347+0000 7f681affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:26.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.347+0000 7f681affd640 10 monclient: dump: 2024-04-26T00:20:26.337 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:26.337 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:26.338 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:26.338 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:26.338 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:26.338 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:26.338 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:26.338 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:26.338 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:26.338 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:26.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.347+0000 7f681affd640 10 monclient: _finish_auth 0 2024-04-26T00:20:26.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.347+0000 7f681affd640 10 monclient: _check_auth_tickets 2024-04-26T00:20:26.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.347+0000 7f681affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:26.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.347+0000 7f681affd640 1 -- 172.21.15.37:0/3531031838 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f681400b470 con 0x7f681c071a50 2024-04-26T00:20:26.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.347+0000 7f681affd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:26.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.347+0000 7f681affd640 1 -- 172.21.15.37:0/3531031838 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68140a7b30 con 0x7f681c071a50 2024-04-26T00:20:26.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.347+0000 7f68227a9640 5 monclient: authenticate success, global_id 4398 2024-04-26T00:20:26.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.347+0000 7f68227a9640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:26.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.347+0000 7f68227a9640 4 set_mon_vals no callback set 2024-04-26T00:20:26.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.347+0000 7f681affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:26.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.347+0000 7f681affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:26.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.347+0000 7f681affd640 10 monclient: dump: 2024-04-26T00:20:26.339 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:26.339 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:26.339 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:26.339 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:26.339 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:26.340 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:26.340 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:26.340 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:26.340 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:26.340 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:26.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.347+0000 7f68227a9640 10 monclient: shutdown 2024-04-26T00:20:26.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.347+0000 7f68227a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:26.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.347+0000 7f68227a9640 1 -- 172.21.15.37:0/3531031838 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f681c071a50 msgr2=0x7f681c0a1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:26.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.347+0000 7f68227a9640 1 --2- 172.21.15.37:0/3531031838 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f681c071a50 0x7f681c0a1910 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f681c0a3580 tx=0x7f681401ebf0 comp rx=0 tx=0).stop 2024-04-26T00:20:26.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.347+0000 7f68227a9640 1 -- 172.21.15.37:0/3531031838 shutdown_connections 2024-04-26T00:20:26.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.347+0000 7f68227a9640 1 --2- 172.21.15.37:0/3531031838 >> v2:172.21.15.45:3300/0 conn(0x7f681c071670 0x7f681c0a13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:26.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.347+0000 7f68227a9640 1 --2- 172.21.15.37:0/3531031838 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f681c071a50 0x7f681c0a1910 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:26.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.347+0000 7f68227a9640 1 --2- 172.21.15.37:0/3531031838 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f681c072180 0x7f681c072560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:26.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.347+0000 7f68227a9640 1 -- 172.21.15.37:0/3531031838 >> 172.21.15.37:0/3531031838 conn(0x7f681c0a24c0 msgr2=0x7f681c0a28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:26.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.347+0000 7f68227a9640 1 -- 172.21.15.37:0/3531031838 shutdown_connections 2024-04-26T00:20:26.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.347+0000 7f68227a9640 1 -- 172.21.15.37:0/3531031838 wait complete. 2024-04-26T00:20:26.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f68227a9640 10 monclient: build_initial_monmap 2024-04-26T00:20:26.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f68227a9640 10 monclient: monmap: 2024-04-26T00:20:26.341 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:26.341 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:26.341 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:26.341 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:26.341 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:26.341 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:26.342 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:26.342 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:26.342 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:26.342 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:26.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f68227a9640 1 Processor -- start 2024-04-26T00:20:26.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f68227a9640 1 -- start start 2024-04-26T00:20:26.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f68227a9640 10 monclient: init 2024-04-26T00:20:26.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f68227a9640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:26.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f68227a9640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:20:26.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f68227a9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f681c071670 0x7f681c1cef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:26.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f68227a9640 10 monclient(hunting): picked mon.noname-b con 0x7f681c071670 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:26.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f68227a9640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f681c071a50 0x7f681c133f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:26.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f68227a9640 10 monclient(hunting): picked mon.noname-c con 0x7f681c071a50 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:26.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f68227a9640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f681c072180 0x7f681c134480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:26.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f68227a9640 10 monclient(hunting): picked mon.noname-a con 0x7f681c072180 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:26.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f68227a9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:26.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f68227a9640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f681c13c6f0 con 0x7f681c072180 2024-04-26T00:20:26.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f68227a9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:26.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f68227a9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f681c0a2260 con 0x7f681c071670 2024-04-26T00:20:26.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f68227a9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:26.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f68227a9640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f681c13c870 con 0x7f681c071a50 2024-04-26T00:20:26.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f68227a9640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:26.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f68227a9640 10 monclient(hunting): authenticate will time out at 1007.657345s 2024-04-26T00:20:26.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f681b7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f681c071a50 0x7f681c133f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:26.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f6820d46640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f681c072180 0x7f681c134480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:26.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f681bfff640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f681c071670 msgr2=0x7f681c1cef30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:20:26.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f681b7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f681c071a50 0x7f681c133f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:55406/0 (socket says 172.21.15.37:55406) 2024-04-26T00:20:26.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f681b7fe640 1 -- 172.21.15.37:0/165049472 learned_addr learned my addr 172.21.15.37:0/165049472 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:26.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f6820d46640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f681c072180 0x7f681c134480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:43754/0 (socket says 172.21.15.37:43754) 2024-04-26T00:20:26.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f6820d46640 10 monclient(hunting): get_auth_request con 0x7f681c072180 auth_method 0 2024-04-26T00:20:26.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f6820d46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:26.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f6820d46640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:26.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f6820d46640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:26.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f681bfff640 1 --2- 172.21.15.37:0/165049472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f681c071670 0x7f681c1cef30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:20:26.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f681b7fe640 10 monclient(hunting): get_auth_request con 0x7f681c071a50 auth_method 0 2024-04-26T00:20:26.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f681b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:26.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f681b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:26.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f681b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:26.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f6820d46640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:26.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f6820d46640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:26.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f6820d46640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:26.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f681b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:26.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f681b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:26.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f681b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:26.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f6820d46640 10 monclient(hunting): handle_auth_done global_id 4401 payload 824 2024-04-26T00:20:26.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f6820d46640 1 -- 172.21.15.37:0/165049472 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f681c071a50 msgr2=0x7f681c133f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:26.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f6820d46640 1 --2- 172.21.15.37:0/165049472 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f681c071a50 0x7f681c133f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:26.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f6820d46640 1 -- 172.21.15.37:0/165049472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f681c071670 msgr2=0x7f681c1cef30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:20:26.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f6820d46640 1 --2- 172.21.15.37:0/165049472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f681c071670 0x7f681c1cef30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:26.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f6820d46640 10 monclient: _finish_hunting 0 2024-04-26T00:20:26.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f6820d46640 1 monclient: found mon.noname-a 2024-04-26T00:20:26.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f6820d46640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:26.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f6820d46640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:26.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f6820d46640 1 -- 172.21.15.37:0/165049472 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f681c11f5a0 con 0x7f681c072180 2024-04-26T00:20:26.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.351+0000 7f681b7fe640 1 --2- 172.21.15.37:0/165049472 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f681c071a50 0x7f681c133f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:26.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f6820d46640 1 -- 172.21.15.37:0/165049472 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f681c072180 msgr2=0x7f681c134480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:26.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f6820d46640 1 --2- 172.21.15.37:0/165049472 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f681c072180 0x7f681c134480 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f68140ad000 tx=0x7f681401b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:26.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f6818ff9640 1 -- 172.21.15.37:0/165049472 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6814009c20 con 0x7f681c072180 2024-04-26T00:20:26.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f6818ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:26.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f6818ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:26.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f6818ff9640 10 monclient: dump: 2024-04-26T00:20:26.346 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:26.346 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:26.346 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:26.346 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:26.346 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:26.346 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:26.346 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:26.346 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:26.346 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:26.347 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:26.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f6818ff9640 10 monclient: _finish_auth 0 2024-04-26T00:20:26.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f6818ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:20:26.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f6818ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:26.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f6818ff9640 1 -- 172.21.15.37:0/165049472 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6814009de0 con 0x7f681c072180 2024-04-26T00:20:26.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f68227a9640 5 monclient: authenticate success, global_id 4401 2024-04-26T00:20:26.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f6818ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:26.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f6818ff9640 1 -- 172.21.15.37:0/165049472 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68140b9670 con 0x7f681c072180 2024-04-26T00:20:26.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f68227a9640 10 monclient: _renew_subs 2024-04-26T00:20:26.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f68227a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:26.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f68227a9640 1 -- 172.21.15.37:0/165049472 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f681c121060 con 0x7f681c072180 2024-04-26T00:20:26.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f6818ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:26.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f6818ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:26.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f6818ff9640 10 monclient: dump: 2024-04-26T00:20:26.348 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:26.348 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:26.348 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:26.348 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:26.348 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:26.348 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:26.348 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:26.348 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:26.348 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:26.348 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:26.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f681a7fc640 4 set_mon_vals no callback set 2024-04-26T00:20:26.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f68227a9640 10 monclient: _renew_subs 2024-04-26T00:20:26.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f68227a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:26.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f68227a9640 1 -- 172.21.15.37:0/165049472 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f681c1349c0 con 0x7f681c072180 2024-04-26T00:20:26.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f68227a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:26.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f68227a9640 1 --2- 172.21.15.37:0/165049472 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f681c11da30 0x7f681c11de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:26.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f68227a9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:26.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f68227a9640 1 -- 172.21.15.37:0/165049472 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f681c072d80 con 0x7f681c11da30 2024-04-26T00:20:26.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f6818ff9640 1 -- 172.21.15.37:0/165049472 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f68140b6070 con 0x7f681c072180 2024-04-26T00:20:26.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f681bfff640 1 --2- 172.21.15.37:0/165049472 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f681c11da30 0x7f681c11de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:26.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f681bfff640 10 monclient: get_auth_request con 0x7f681c11da30 auth_method 0 2024-04-26T00:20:26.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f681bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:26.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f681bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:26.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f681bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:26.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f681bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:26.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f681bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:26.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f681bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:26.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f6818ff9640 1 --2- 172.21.15.37:0/165049472 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6808035e80 0x7f6808038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:26.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f6818ff9640 1 -- 172.21.15.37:0/165049472 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 3871+0+0 (secure 0 0 0) 0x7f68140e3080 con 0x7f681c072180 2024-04-26T00:20:26.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f681bfff640 10 monclient(hunting): handle_auth_done global_id 4394 payload 824 2024-04-26T00:20:26.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f681bfff640 1 -- 172.21.15.37:0/165049472 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f681c134bf0 con 0x7f681c11da30 2024-04-26T00:20:26.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f681b7fe640 1 --2- 172.21.15.37:0/165049472 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6808035e80 0x7f6808038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:26.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f681b7fe640 10 monclient: get_auth_request con 0x7f6808035e80 auth_method 0 2024-04-26T00:20:26.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f681bfff640 1 -- 172.21.15.37:0/165049472 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f681c11da30 msgr2=0x7f681c11de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:26.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.355+0000 7f681bfff640 1 --2- 172.21.15.37:0/165049472 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f681c11da30 0x7f681c11de10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f680c0094c0 tx=0x7f680c00ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:26.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.359+0000 7f6818ff9640 1 -- 172.21.15.37:0/165049472 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f680c017070 con 0x7f681c11da30 2024-04-26T00:20:26.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.359+0000 7f6818ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:26.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.359+0000 7f6818ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:26.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.359+0000 7f6818ff9640 10 monclient: dump: 2024-04-26T00:20:26.350 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:26.350 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:26.350 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:26.350 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:26.350 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:26.351 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:26.351 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:26.351 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:26.351 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:26.351 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:26.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.359+0000 7f6818ff9640 1 -- 172.21.15.37:0/165049472 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f680c00cbc0 con 0x7f681c11da30 2024-04-26T00:20:26.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.359+0000 7f6818ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:26.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.359+0000 7f6818ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:26.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.359+0000 7f6818ff9640 1 -- 172.21.15.37:0/165049472 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f681c11da30 msgr2=0x7f681c11de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:26.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.359+0000 7f6818ff9640 1 --2- 172.21.15.37:0/165049472 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f681c11da30 0x7f681c11de10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f680c0094c0 tx=0x7f680c00ec90 comp rx=0 tx=0).stop 2024-04-26T00:20:26.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.359+0000 7f6818ff9640 1 -- 172.21.15.37:0/165049472 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f681c11da30 msgr2=0x7f681c11de10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:20:26.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.359+0000 7f6818ff9640 1 --2- 172.21.15.37:0/165049472 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f681c11da30 0x7f681c11de10 secure :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f680c0094c0 tx=0x7f680c00ec90 comp rx=0 tx=0).stop 2024-04-26T00:20:26.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.359+0000 7f681b7fe640 1 --2- 172.21.15.37:0/165049472 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6808035e80 0x7f6808038340 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f681c06cca0 tx=0x7f6810003ba0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:26.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.363+0000 7f68227a9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:20:26.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.363+0000 7f68227a9640 1 --2- 172.21.15.37:0/165049472 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f681c11e350 0x7f681c1c1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:26.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.363+0000 7f68227a9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:26.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.363+0000 7f68227a9640 1 -- 172.21.15.37:0/165049472 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f681c121a10 con 0x7f681c11e350 2024-04-26T00:20:26.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.363+0000 7f681bfff640 1 --2- 172.21.15.37:0/165049472 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f681c11e350 0x7f681c1c1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:26.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.363+0000 7f681bfff640 10 monclient: get_auth_request con 0x7f681c11e350 auth_method 0 2024-04-26T00:20:26.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.363+0000 7f681bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:26.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.363+0000 7f681bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:26.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.363+0000 7f681bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:26.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.363+0000 7f681bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:26.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.363+0000 7f681bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:26.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.363+0000 7f681bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:26.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.363+0000 7f681bfff640 10 monclient(hunting): handle_auth_done global_id 4397 payload 824 2024-04-26T00:20:26.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.363+0000 7f681bfff640 1 -- 172.21.15.37:0/165049472 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f681c04f510 con 0x7f681c11e350 2024-04-26T00:20:26.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.367+0000 7f681bfff640 1 -- 172.21.15.37:0/165049472 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f681c11e350 msgr2=0x7f681c1c1700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:26.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.367+0000 7f681bfff640 1 --2- 172.21.15.37:0/165049472 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f681c11e350 0x7f681c1c1700 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f680c0094c0 tx=0x7f680c004740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:26.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.367+0000 7f6818ff9640 1 -- 172.21.15.37:0/165049472 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f680c017070 con 0x7f681c11e350 2024-04-26T00:20:26.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.367+0000 7f6818ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:26.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.367+0000 7f6818ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:26.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.367+0000 7f6818ff9640 10 monclient: dump: 2024-04-26T00:20:26.354 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:26.355 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:26.355 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:26.355 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:26.355 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:26.355 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:26.355 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:26.355 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:26.355 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:26.355 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:26.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.367+0000 7f6818ff9640 1 -- 172.21.15.37:0/165049472 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4705 (secure 0 0 0) 0x7f680c0067c0 con 0x7f681c11e350 2024-04-26T00:20:26.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.367+0000 7f6818ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:20:26.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.367+0000 7f6818ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:26.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.367+0000 7f6818ff9640 1 -- 172.21.15.37:0/165049472 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f681c11e350 msgr2=0x7f681c1c1700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:26.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.367+0000 7f6818ff9640 1 --2- 172.21.15.37:0/165049472 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f681c11e350 0x7f681c1c1700 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f680c0094c0 tx=0x7f680c004740 comp rx=0 tx=0).stop 2024-04-26T00:20:26.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.367+0000 7f6818ff9640 1 -- 172.21.15.37:0/165049472 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f681c11e350 msgr2=0x7f681c1c1700 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:20:26.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.367+0000 7f6818ff9640 1 --2- 172.21.15.37:0/165049472 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f681c11e350 0x7f681c1c1700 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f680c0094c0 tx=0x7f680c004740 comp rx=0 tx=0).stop 2024-04-26T00:20:26.356 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:20:26.356 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:20:26.356 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:20:26.356 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:20:26.356 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 10, 2024-04-26T00:20:26.356 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 43475, 2024-04-26T00:20:26.356 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:20:26.356 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:20:26.357 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:20:26.357 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:26.357 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 2, 2024-04-26T00:20:26.357 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:20:26.357 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:20:26.357 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:20:26.357 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:20:26.357 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:20:26.357 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:20:26.357 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:20:26.357 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:20:26.357 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:20:26.357 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:20:26.357 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:20:26.357 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:20:26.358 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:20:26.358 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:20:26.358 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:26.358 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:20:26.358 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:20:26.358 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:20:26.358 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:20:26.358 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:20:26.358 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:20:26.358 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:20:26.358 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:20:26.358 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:20:26.358 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:20:26.358 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:20:26.359 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:20:26.359 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:20:26.359 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:26.359 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:26.359 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:20:26.359 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:20:26.359 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:20:26.359 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:20:26.359 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:20:26.359 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:20:26.359 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:20:26.359 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:20:26.359 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:20:26.360 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:20:26.360 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:20:26.360 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:20:26.360 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:20:26.360 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:20:26.360 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:20:26.360 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:20:26.360 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:20:26.360 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:20:26.360 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:20:26.360 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:20:26.360 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:20:26.360 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:20:26.360 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:20:26.360 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:20:26.361 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:20:26.361 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:20:26.361 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:20:26.361 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:20:26.361 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:26.361 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:20:26.361 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:26.361 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:20:26.361 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:26.361 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:20:26.361 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:20:26.361 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:20:26.361 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:20:26.362 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:26.362 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:20:26.362 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:20:26.362 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:26.362 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:26.362 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:26.362 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:20:26.362 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:20:26.362 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:26.362 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:26.362 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:26.362 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:26.362 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:20:26.363 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:20:26.363 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:20:26.363 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:20:26.363 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:20:26.363 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:26.363 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:26.363 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:20:26.363 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:20:26.363 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:20:26.363 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:20:26.363 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:26.363 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:20:26.363 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:20:26.363 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:26.364 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:26.364 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:26.364 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:20:26.364 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:20:26.364 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:26.364 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:26.364 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:26.364 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:26.364 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:20:26.364 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:20:26.364 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:20:26.364 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:20:26.364 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:20:26.364 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:26.364 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:26.365 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:20:26.365 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:20:26.365 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:20:26.365 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:20:26.365 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:26.365 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:20:26.365 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:20:26.365 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:26.365 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:26.365 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:26.365 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:20:26.365 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:20:26.365 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:26.365 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:26.365 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:26.366 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:26.366 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:20:26.366 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:20:26.366 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:20:26.366 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:20:26.366 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:20:26.366 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:26.366 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:26.366 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:26.366 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:20:26.366 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:20:26.366 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:26.366 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:20:26.366 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:20:26.366 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:20:26.367 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:26.367 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:26.367 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:20:26.367 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:26.367 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:20:26.367 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:20:26.367 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:20:26.367 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:26.367 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:26.367 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:20:26.367 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:26.367 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:20:26.367 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:20:26.367 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:20:26.367 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:26.368 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:26.368 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:26.368 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:20:26.368 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:20:26.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.367+0000 7f68227a9640 1 -- 172.21.15.37:0/165049472 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6808035e80 msgr2=0x7f6808038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:26.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.367+0000 7f68227a9640 1 --2- 172.21.15.37:0/165049472 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6808035e80 0x7f6808038340 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f681c06cca0 tx=0x7f6810003ba0 comp rx=0 tx=0).stop 2024-04-26T00:20:26.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.367+0000 7f68227a9640 10 monclient: shutdown 2024-04-26T00:20:26.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.367+0000 7f68227a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:26.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.367+0000 7f68227a9640 1 -- 172.21.15.37:0/165049472 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f681c072180 msgr2=0x7f681c134480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:26.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.367+0000 7f68227a9640 1 --2- 172.21.15.37:0/165049472 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f681c072180 0x7f681c134480 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f68140ad000 tx=0x7f681401b990 comp rx=0 tx=0).stop 2024-04-26T00:20:26.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.367+0000 7f6820d46640 1 -- 172.21.15.37:0/165049472 reap_dead start 2024-04-26T00:20:26.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.367+0000 7f6820d46640 1 -- 172.21.15.37:0/165049472 reap_dead start 2024-04-26T00:20:26.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.367+0000 7f68227a9640 1 -- 172.21.15.37:0/165049472 shutdown_connections 2024-04-26T00:20:26.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.367+0000 7f68227a9640 1 -- 172.21.15.37:0/165049472 >> 172.21.15.37:0/165049472 conn(0x7f681c0a24c0 msgr2=0x7f681c0a4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:26.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.367+0000 7f68227a9640 1 -- 172.21.15.37:0/165049472 shutdown_connections 2024-04-26T00:20:26.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:26.367+0000 7f68227a9640 1 -- 172.21.15.37:0/165049472 wait complete. 2024-04-26T00:20:26.370 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:20:26.371 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-04-26T00:20:26.833 DEBUG:teuthology.orchestra.run.smithi045:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0 2024-04-26T00:20:26.856 INFO:tasks.rados.rados.0.smithi045.stdout:adding op weight read -> 100 2024-04-26T00:20:26.856 INFO:tasks.rados.rados.0.smithi045.stdout:adding op weight write -> 50 2024-04-26T00:20:26.856 INFO:tasks.rados.rados.0.smithi045.stdout:adding op weight delete -> 50 2024-04-26T00:20:26.856 INFO:tasks.rados.rados.0.smithi045.stdout:adding op weight snap_create -> 50 2024-04-26T00:20:26.856 INFO:tasks.rados.rados.0.smithi045.stdout:adding op weight snap_remove -> 50 2024-04-26T00:20:26.856 INFO:tasks.rados.rados.0.smithi045.stdout:adding op weight rollback -> 50 2024-04-26T00:20:26.856 INFO:tasks.rados.rados.0.smithi045.stdout:adding op weight copy_from -> 50 2024-04-26T00:20:26.856 INFO:tasks.rados.rados.0.smithi045.stdout:adding op weight write_excl -> 50 2024-04-26T00:20:26.856 INFO:tasks.rados.rados.0.smithi045.stdout:ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev) 2024-04-26T00:20:26.856 INFO:tasks.rados.rados.0.smithi045.stdout:Configuration: 2024-04-26T00:20:26.856 INFO:tasks.rados.rados.0.smithi045.stdout: Number of operations: 4000 2024-04-26T00:20:26.856 INFO:tasks.rados.rados.0.smithi045.stdout: Number of objects: 50 2024-04-26T00:20:26.856 INFO:tasks.rados.rados.0.smithi045.stdout: Max in flight operations: 16 2024-04-26T00:20:26.857 INFO:tasks.rados.rados.0.smithi045.stdout: Object size (in bytes): 4000000 2024-04-26T00:20:26.857 INFO:tasks.rados.rados.0.smithi045.stdout: Write stride min: 400000 2024-04-26T00:20:26.857 INFO:tasks.rados.rados.0.smithi045.stdout: Write stride max: 800000 2024-04-26T00:20:27.809 INFO:tasks.rados.rados.0.smithi045.stdout:1: write initial oid 1 2024-04-26T00:20:27.809 INFO:tasks.rados.rados.0.smithi045.stdout:1: seq_num 0 ranges {416807=475249,1542129=474678} 2024-04-26T00:20:27.818 INFO:tasks.rados.rados.0.smithi045.stdout:1: writing smithi04526579-1 from 416807 to 892056 tid 1 2024-04-26T00:20:27.830 INFO:tasks.rados.rados.0.smithi045.stdout:1: writing smithi04526579-1 from 1542129 to 2016807 tid 2 2024-04-26T00:20:27.830 INFO:tasks.rados.rados.0.smithi045.stdout:2: write initial oid 2 2024-04-26T00:20:27.830 INFO:tasks.rados.rados.0.smithi045.stdout:2: seq_num 1 ranges {416807=475249,1542129=474678} 2024-04-26T00:20:27.838 INFO:tasks.rados.rados.0.smithi045.stdout:2: writing smithi04526579-2 from 416807 to 892056 tid 1 2024-04-26T00:20:27.850 INFO:tasks.rados.rados.0.smithi045.stdout:2: writing smithi04526579-2 from 1542129 to 2016807 tid 2 2024-04-26T00:20:27.850 INFO:tasks.rados.rados.0.smithi045.stdout:3: write initial oid 3 2024-04-26T00:20:27.850 INFO:tasks.rados.rados.0.smithi045.stdout:3: seq_num 2 ranges {433614=550498,1600611=433003} 2024-04-26T00:20:27.859 INFO:tasks.rados.rados.0.smithi045.stdout:3: writing smithi04526579-3 from 433614 to 984112 tid 1 2024-04-26T00:20:27.869 INFO:tasks.rados.rados.0.smithi045.stdout:3: writing smithi04526579-3 from 1600611 to 2033614 tid 2 2024-04-26T00:20:27.869 INFO:tasks.rados.rados.0.smithi045.stdout:4: write initial oid 4 2024-04-26T00:20:27.869 INFO:tasks.rados.rados.0.smithi045.stdout:4: seq_num 3 ranges {450421=625747,1659093=391328} 2024-04-26T00:20:27.878 INFO:tasks.rados.rados.0.smithi045.stdout:4: writing smithi04526579-4 from 450421 to 1076168 tid 1 2024-04-26T00:20:27.886 INFO:tasks.rados.rados.0.smithi045.stdout:4: writing smithi04526579-4 from 1659093 to 2050421 tid 2 2024-04-26T00:20:27.886 INFO:tasks.rados.rados.0.smithi045.stdout:5: write initial oid 5 2024-04-26T00:20:27.886 INFO:tasks.rados.rados.0.smithi045.stdout:5: seq_num 4 ranges {467228=700996,1717575=349653} 2024-04-26T00:20:27.893 INFO:tasks.rados.rados.0.smithi045.stdout:5: writing smithi04526579-5 from 467228 to 1168224 tid 1 2024-04-26T00:20:27.899 INFO:tasks.rados.rados.0.smithi045.stdout:5: writing smithi04526579-5 from 1717575 to 2067228 tid 2 2024-04-26T00:20:27.899 INFO:tasks.rados.rados.0.smithi045.stdout:6: write initial oid 6 2024-04-26T00:20:27.899 INFO:tasks.rados.rados.0.smithi045.stdout:6: seq_num 5 ranges {484035=776245,2059704=24331} 2024-04-26T00:20:27.907 INFO:tasks.rados.rados.0.smithi045.stdout:6: writing smithi04526579-6 from 484035 to 1260280 tid 1 2024-04-26T00:20:27.912 INFO:tasks.rados.rados.0.smithi045.stdout:6: writing smithi04526579-6 from 2059704 to 2084035 tid 2 2024-04-26T00:20:27.912 INFO:tasks.rados.rados.0.smithi045.stdout:7: write initial oid 7 2024-04-26T00:20:27.912 INFO:tasks.rados.rados.0.smithi045.stdout:7: seq_num 6 ranges {500842=451494,1718186=382656} 2024-04-26T00:20:27.918 INFO:tasks.rados.rados.0.smithi045.stdout:7: writing smithi04526579-7 from 500842 to 952336 tid 1 2024-04-26T00:20:27.924 INFO:tasks.rados.rados.0.smithi045.stdout:7: writing smithi04526579-7 from 1718186 to 2100842 tid 2 2024-04-26T00:20:27.925 INFO:tasks.rados.rados.0.smithi045.stdout:8: write initial oid 8 2024-04-26T00:20:27.925 INFO:tasks.rados.rados.0.smithi045.stdout:8: seq_num 7 ranges {517649=526743,1776668=340981} 2024-04-26T00:20:27.930 INFO:tasks.rados.rados.0.smithi045.stdout:8: writing smithi04526579-8 from 517649 to 1044392 tid 1 2024-04-26T00:20:27.937 INFO:tasks.rados.rados.0.smithi045.stdout:8: writing smithi04526579-8 from 1776668 to 2117649 tid 2 2024-04-26T00:20:27.937 INFO:tasks.rados.rados.0.smithi045.stdout:9: write initial oid 9 2024-04-26T00:20:27.937 INFO:tasks.rados.rados.0.smithi045.stdout:9: seq_num 8 ranges {534456=718345,1951503=182953} 2024-04-26T00:20:27.944 INFO:tasks.rados.rados.0.smithi045.stdout:9: writing smithi04526579-9 from 534456 to 1252801 tid 1 2024-04-26T00:20:27.949 INFO:tasks.rados.rados.0.smithi045.stdout:9: writing smithi04526579-9 from 1951503 to 2134456 tid 2 2024-04-26T00:20:27.949 INFO:tasks.rados.rados.0.smithi045.stdout:10: write initial oid 10 2024-04-26T00:20:27.949 INFO:tasks.rados.rados.0.smithi045.stdout:10: seq_num 9 ranges {551263=793594,1893632=257631} 2024-04-26T00:20:27.957 INFO:tasks.rados.rados.0.smithi045.stdout:10: writing smithi04526579-10 from 551263 to 1344857 tid 1 2024-04-26T00:20:27.961 INFO:tasks.rados.rados.0.smithi045.stdout:10: writing smithi04526579-10 from 1893632 to 2151263 tid 2 2024-04-26T00:20:27.961 INFO:tasks.rados.rados.0.smithi045.stdout:11: write initial oid 11 2024-04-26T00:20:27.961 INFO:tasks.rados.rados.0.smithi045.stdout:11: seq_num 10 ranges {568070=468843,1552114=615956} 2024-04-26T00:20:27.967 INFO:tasks.rados.rados.0.smithi045.stdout:11: writing smithi04526579-11 from 568070 to 1036913 tid 1 2024-04-26T00:20:27.974 INFO:tasks.rados.rados.0.smithi045.stdout:11: writing smithi04526579-11 from 1552114 to 2168070 tid 2 2024-04-26T00:20:27.974 INFO:tasks.rados.rados.0.smithi045.stdout:12: write initial oid 12 2024-04-26T00:20:27.974 INFO:tasks.rados.rados.0.smithi045.stdout:12: seq_num 11 ranges {584877=544092,1610596=562003} 2024-04-26T00:20:27.981 INFO:tasks.rados.rados.0.smithi045.stdout:12: writing smithi04526579-12 from 584877 to 1128969 tid 1 2024-04-26T00:20:27.986 INFO:tasks.rados.rados.0.smithi045.stdout:12: writing smithi04526579-12 from 1610596 to 2172599 tid 2 2024-04-26T00:20:27.986 INFO:tasks.rados.rados.0.smithi045.stdout:13: write initial oid 13 2024-04-26T00:20:27.986 INFO:tasks.rados.rados.0.smithi045.stdout:1: finishing write tid 1 to smithi04526579-1 2024-04-26T00:20:27.986 INFO:tasks.rados.rados.0.smithi045.stdout:13: seq_num 12 ranges {601684=619341,1669078=532606} 2024-04-26T00:20:27.993 INFO:tasks.rados.rados.0.smithi045.stdout:13: writing smithi04526579-13 from 601684 to 1221025 tid 1 2024-04-26T00:20:27.999 INFO:tasks.rados.rados.0.smithi045.stdout:13: writing smithi04526579-13 from 1669078 to 2201684 tid 2 2024-04-26T00:20:27.999 INFO:tasks.rados.rados.0.smithi045.stdout:14: write initial oid 14 2024-04-26T00:20:27.999 INFO:tasks.rados.rados.0.smithi045.stdout:14: seq_num 13 ranges {618491=694590,2011207=207284} 2024-04-26T00:20:28.006 INFO:tasks.rados.rados.0.smithi045.stdout:14: writing smithi04526579-14 from 618491 to 1313081 tid 1 2024-04-26T00:20:28.011 INFO:tasks.rados.rados.0.smithi045.stdout:14: writing smithi04526579-14 from 2011207 to 2218491 tid 2 2024-04-26T00:20:28.011 INFO:tasks.rados.rados.0.smithi045.stdout:15: write initial oid 15 2024-04-26T00:20:28.012 INFO:tasks.rados.rados.0.smithi045.stdout:15: seq_num 14 ranges {635298=769839,2069689=165609} 2024-04-26T00:20:28.019 INFO:tasks.rados.rados.0.smithi045.stdout:15: writing smithi04526579-15 from 635298 to 1405137 tid 1 2024-04-26T00:20:28.024 INFO:tasks.rados.rados.0.smithi045.stdout:15: writing smithi04526579-15 from 2069689 to 2235298 tid 2 2024-04-26T00:20:28.024 INFO:tasks.rados.rados.0.smithi045.stdout:16: write initial oid 16 2024-04-26T00:20:28.024 INFO:tasks.rados.rados.0.smithi045.stdout:16: seq_num 15 ranges {652105=445088,1728171=452988} 2024-04-26T00:20:28.031 INFO:tasks.rados.rados.0.smithi045.stdout:16: writing smithi04526579-16 from 652105 to 1097193 tid 1 2024-04-26T00:20:28.037 INFO:tasks.rados.rados.0.smithi045.stdout:16: writing smithi04526579-16 from 1728171 to 2181159 tid 2 2024-04-26T00:20:28.037 INFO:tasks.rados.rados.0.smithi045.stdout: waiting on 16 2024-04-26T00:20:28.038 INFO:tasks.rados.rados.0.smithi045.stdout:1: finishing write tid 2 to smithi04526579-1 2024-04-26T00:20:28.038 INFO:tasks.rados.rados.0.smithi045.stdout:1: finishing write tid 3 to smithi04526579-1 2024-04-26T00:20:28.038 INFO:tasks.rados.rados.0.smithi045.stdout:1: finishing write tid 4 to smithi04526579-1 2024-04-26T00:20:28.038 INFO:tasks.rados.rados.0.smithi045.stdout:1: oid 1 updating version 0 to 3 2024-04-26T00:20:28.038 INFO:tasks.rados.rados.0.smithi045.stdout:1: oid 1 version 3 is already newer than 2 2024-04-26T00:20:28.038 INFO:tasks.rados.rados.0.smithi045.stdout:1: oid 1 version 3 is already newer than 1 2024-04-26T00:20:28.038 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-26T00:20:28.038 INFO:tasks.rados.rados.0.smithi045.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2024-04-26T00:20:28.038 INFO:tasks.rados.rados.0.smithi045.stdout:2: finishing write tid 1 to smithi04526579-2 2024-04-26T00:20:28.038 INFO:tasks.rados.rados.0.smithi045.stdout:2: finishing write tid 2 to smithi04526579-2 2024-04-26T00:20:28.038 INFO:tasks.rados.rados.0.smithi045.stdout:2: finishing write tid 3 to smithi04526579-2 2024-04-26T00:20:28.038 INFO:tasks.rados.rados.0.smithi045.stdout:2: finishing write tid 4 to smithi04526579-2 2024-04-26T00:20:28.038 INFO:tasks.rados.rados.0.smithi045.stdout:2: oid 2 updating version 0 to 3 2024-04-26T00:20:28.038 INFO:tasks.rados.rados.0.smithi045.stdout:2: oid 2 version 3 is already newer than 2 2024-04-26T00:20:28.039 INFO:tasks.rados.rados.0.smithi045.stdout:2: oid 2 version 3 is already newer than 1 2024-04-26T00:20:28.039 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-26T00:20:28.039 INFO:tasks.rados.rados.0.smithi045.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2024-04-26T00:20:28.039 INFO:tasks.rados.rados.0.smithi045.stdout:3: finishing write tid 1 to smithi04526579-3 2024-04-26T00:20:28.039 INFO:tasks.rados.rados.0.smithi045.stdout:1: done (15 left) 2024-04-26T00:20:28.039 INFO:tasks.rados.rados.0.smithi045.stdout:2: done (14 left) 2024-04-26T00:20:28.039 INFO:tasks.rados.rados.0.smithi045.stdout:17: write initial oid 17 2024-04-26T00:20:28.039 INFO:tasks.rados.rados.0.smithi045.stdout:17: seq_num 16 ranges {668912=636690,1903006=365906} 2024-04-26T00:20:28.044 INFO:tasks.rados.rados.0.smithi045.stdout:17: writing smithi04526579-17 from 668912 to 1305602 tid 1 2024-04-26T00:20:28.050 INFO:tasks.rados.rados.0.smithi045.stdout:17: writing smithi04526579-17 from 1903006 to 2268912 tid 2 2024-04-26T00:20:28.050 INFO:tasks.rados.rados.0.smithi045.stdout:18: write initial oid 18 2024-04-26T00:20:28.050 INFO:tasks.rados.rados.0.smithi045.stdout:18: seq_num 17 ranges {685719=711939,1845135=440584} 2024-04-26T00:20:28.059 INFO:tasks.rados.rados.0.smithi045.stdout:18: writing smithi04526579-18 from 685719 to 1397658 tid 1 2024-04-26T00:20:28.064 INFO:tasks.rados.rados.0.smithi045.stdout:18: writing smithi04526579-18 from 1845135 to 2285719 tid 2 2024-04-26T00:20:28.064 INFO:tasks.rados.rados.0.smithi045.stdout: waiting on 16 2024-04-26T00:20:28.064 INFO:tasks.rados.rados.0.smithi045.stdout:3: finishing write tid 2 to smithi04526579-3 2024-04-26T00:20:28.064 INFO:tasks.rados.rados.0.smithi045.stdout:3: finishing write tid 3 to smithi04526579-3 2024-04-26T00:20:28.064 INFO:tasks.rados.rados.0.smithi045.stdout:3: finishing write tid 4 to smithi04526579-3 2024-04-26T00:20:28.064 INFO:tasks.rados.rados.0.smithi045.stdout:3: oid 3 updating version 0 to 1 2024-04-26T00:20:28.064 INFO:tasks.rados.rados.0.smithi045.stdout:3: oid 3 updating version 1 to 2 2024-04-26T00:20:28.064 INFO:tasks.rados.rados.0.smithi045.stdout:3: oid 3 updating version 2 to 3 2024-04-26T00:20:28.064 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-04-26T00:20:28.064 INFO:tasks.rados.rados.0.smithi045.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2024-04-26T00:20:28.064 INFO:tasks.rados.rados.0.smithi045.stdout:4: finishing write tid 1 to smithi04526579-4 2024-04-26T00:20:28.064 INFO:tasks.rados.rados.0.smithi045.stdout:4: finishing write tid 2 to smithi04526579-4 2024-04-26T00:20:28.065 INFO:tasks.rados.rados.0.smithi045.stdout:3: done (15 left) 2024-04-26T00:20:28.065 INFO:tasks.rados.rados.0.smithi045.stdout:19: write initial oid 19 2024-04-26T00:20:28.065 INFO:tasks.rados.rados.0.smithi045.stdout:19: seq_num 18 ranges {702526=787188,1903617=398909} 2024-04-26T00:20:28.072 INFO:tasks.rados.rados.0.smithi045.stdout:19: writing smithi04526579-19 from 702526 to 1489714 tid 1 2024-04-26T00:20:28.077 INFO:tasks.rados.rados.0.smithi045.stdout:19: writing smithi04526579-19 from 1903617 to 2302526 tid 2 2024-04-26T00:20:28.077 INFO:tasks.rados.rados.0.smithi045.stdout: waiting on 16 2024-04-26T00:20:28.077 INFO:tasks.rados.rados.0.smithi045.stdout:4: finishing write tid 3 to smithi04526579-4 2024-04-26T00:20:28.077 INFO:tasks.rados.rados.0.smithi045.stdout:4: finishing write tid 4 to smithi04526579-4 2024-04-26T00:20:28.077 INFO:tasks.rados.rados.0.smithi045.stdout:4: oid 4 updating version 0 to 6 2024-04-26T00:20:28.077 INFO:tasks.rados.rados.0.smithi045.stdout:4: oid 4 version 6 is already newer than 5 2024-04-26T00:20:28.077 INFO:tasks.rados.rados.0.smithi045.stdout:4: oid 4 version 6 is already newer than 4 2024-04-26T00:20:28.077 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 6 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-26T00:20:28.077 INFO:tasks.rados.rados.0.smithi045.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2024-04-26T00:20:28.077 INFO:tasks.rados.rados.0.smithi045.stdout:5: finishing write tid 1 to smithi04526579-5 2024-04-26T00:20:28.077 INFO:tasks.rados.rados.0.smithi045.stdout:5: finishing write tid 2 to smithi04526579-5 2024-04-26T00:20:28.077 INFO:tasks.rados.rados.0.smithi045.stdout:5: finishing write tid 3 to smithi04526579-5 2024-04-26T00:20:28.078 INFO:tasks.rados.rados.0.smithi045.stdout:4: done (15 left) 2024-04-26T00:20:28.078 INFO:tasks.rados.rados.0.smithi045.stdout:20: write initial oid 20 2024-04-26T00:20:28.078 INFO:tasks.rados.rados.0.smithi045.stdout:20: seq_num 19 ranges {719333=462437,1962099=357234} 2024-04-26T00:20:28.085 INFO:tasks.rados.rados.0.smithi045.stdout:20: writing smithi04526579-20 from 719333 to 1181770 tid 1 2024-04-26T00:20:28.090 INFO:tasks.rados.rados.0.smithi045.stdout:20: writing smithi04526579-20 from 1962099 to 2319333 tid 2 2024-04-26T00:20:28.090 INFO:tasks.rados.rados.0.smithi045.stdout: waiting on 16 2024-04-26T00:20:28.090 INFO:tasks.rados.rados.0.smithi045.stdout:5: finishing write tid 4 to smithi04526579-5 2024-04-26T00:20:28.090 INFO:tasks.rados.rados.0.smithi045.stdout:5: oid 5 updating version 0 to 1 2024-04-26T00:20:28.090 INFO:tasks.rados.rados.0.smithi045.stdout:5: oid 5 updating version 1 to 2 2024-04-26T00:20:28.090 INFO:tasks.rados.rados.0.smithi045.stdout:5: oid 5 updating version 2 to 3 2024-04-26T00:20:28.090 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-04-26T00:20:28.090 INFO:tasks.rados.rados.0.smithi045.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2024-04-26T00:20:28.090 INFO:tasks.rados.rados.0.smithi045.stdout:7: finishing write tid 1 to smithi04526579-7 2024-04-26T00:20:28.091 INFO:tasks.rados.rados.0.smithi045.stdout:5: done (15 left) 2024-04-26T00:20:28.091 INFO:tasks.rados.rados.0.smithi045.stdout:21: write initial oid 21 2024-04-26T00:20:28.091 INFO:tasks.rados.rados.0.smithi045.stdout:21: seq_num 20 ranges {736140=537686,2020581=315559} 2024-04-26T00:20:28.102 INFO:tasks.rados.rados.0.smithi045.stdout:21: writing smithi04526579-21 from 736140 to 1273826 tid 1 2024-04-26T00:20:28.103 INFO:tasks.rados.rados.0.smithi045.stdout:21: writing smithi04526579-21 from 2020581 to 2336140 tid 2 2024-04-26T00:20:28.103 INFO:tasks.rados.rados.0.smithi045.stdout: waiting on 16 2024-04-26T00:20:28.103 INFO:tasks.rados.rados.0.smithi045.stdout:6: finishing write tid 1 to smithi04526579-6 2024-04-26T00:20:28.103 INFO:tasks.rados.rados.0.smithi045.stdout:6: finishing write tid 2 to smithi04526579-6 2024-04-26T00:20:28.103 INFO:tasks.rados.rados.0.smithi045.stdout:6: finishing write tid 3 to smithi04526579-6 2024-04-26T00:20:28.103 INFO:tasks.rados.rados.0.smithi045.stdout:6: finishing write tid 4 to smithi04526579-6 2024-04-26T00:20:28.103 INFO:tasks.rados.rados.0.smithi045.stdout:6: oid 6 updating version 0 to 1 2024-04-26T00:20:28.103 INFO:tasks.rados.rados.0.smithi045.stdout:6: oid 6 updating version 1 to 2 2024-04-26T00:20:28.103 INFO:tasks.rados.rados.0.smithi045.stdout:6: oid 6 updating version 2 to 3 2024-04-26T00:20:28.103 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 3 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-04-26T00:20:28.104 INFO:tasks.rados.rados.0.smithi045.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2024-04-26T00:20:28.104 INFO:tasks.rados.rados.0.smithi045.stdout:7: finishing write tid 2 to smithi04526579-7 2024-04-26T00:20:28.104 INFO:tasks.rados.rados.0.smithi045.stdout:7: finishing write tid 3 to smithi04526579-7 2024-04-26T00:20:28.104 INFO:tasks.rados.rados.0.smithi045.stdout:6: done (15 left) 2024-04-26T00:20:28.104 INFO:tasks.rados.rados.0.smithi045.stdout:22: write initial oid 22 2024-04-26T00:20:28.104 INFO:tasks.rados.rados.0.smithi045.stdout:22: seq_num 21 ranges {752947=612935,1962710=390237} 2024-04-26T00:20:28.111 INFO:tasks.rados.rados.0.smithi045.stdout:22: writing smithi04526579-22 from 752947 to 1365882 tid 1 2024-04-26T00:20:28.116 INFO:tasks.rados.rados.0.smithi045.stdout:22: writing smithi04526579-22 from 1962710 to 2352947 tid 2 2024-04-26T00:20:28.117 INFO:tasks.rados.rados.0.smithi045.stdout: waiting on 16 2024-04-26T00:20:28.117 INFO:tasks.rados.rados.0.smithi045.stdout:7: finishing write tid 4 to smithi04526579-7 2024-04-26T00:20:28.117 INFO:tasks.rados.rados.0.smithi045.stdout:7: oid 7 updating version 0 to 6 2024-04-26T00:20:28.117 INFO:tasks.rados.rados.0.smithi045.stdout:7: oid 7 version 6 is already newer than 5 2024-04-26T00:20:28.117 INFO:tasks.rados.rados.0.smithi045.stdout:7: oid 7 version 6 is already newer than 4 2024-04-26T00:20:28.117 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 6 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-04-26T00:20:28.117 INFO:tasks.rados.rados.0.smithi045.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2024-04-26T00:20:28.117 INFO:tasks.rados.rados.0.smithi045.stdout:8: finishing write tid 1 to smithi04526579-8 2024-04-26T00:20:28.117 INFO:tasks.rados.rados.0.smithi045.stdout:8: finishing write tid 2 to smithi04526579-8 2024-04-26T00:20:28.117 INFO:tasks.rados.rados.0.smithi045.stdout:8: finishing write tid 3 to smithi04526579-8 2024-04-26T00:20:28.117 INFO:tasks.rados.rados.0.smithi045.stdout:7: done (15 left) 2024-04-26T00:20:28.117 INFO:tasks.rados.rados.0.smithi045.stdout:23: write initial oid 23 2024-04-26T00:20:28.118 INFO:tasks.rados.rados.0.smithi045.stdout:23: seq_num 22 ranges {769754=688184,2021192=348562} 2024-04-26T00:20:28.125 INFO:tasks.rados.rados.0.smithi045.stdout:23: writing smithi04526579-23 from 769754 to 1457938 tid 1 2024-04-26T00:20:28.130 INFO:tasks.rados.rados.0.smithi045.stdout:23: writing smithi04526579-23 from 2021192 to 2369754 tid 2 2024-04-26T00:20:28.130 INFO:tasks.rados.rados.0.smithi045.stdout: waiting on 16 2024-04-26T00:20:28.130 INFO:tasks.rados.rados.0.smithi045.stdout:8: finishing write tid 4 to smithi04526579-8 2024-04-26T00:20:28.130 INFO:tasks.rados.rados.0.smithi045.stdout:8: oid 8 updating version 0 to 1 2024-04-26T00:20:28.130 INFO:tasks.rados.rados.0.smithi045.stdout:8: oid 8 updating version 1 to 2 2024-04-26T00:20:28.130 INFO:tasks.rados.rados.0.smithi045.stdout:8: oid 8 updating version 2 to 3 2024-04-26T00:20:28.130 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 3 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-04-26T00:20:28.130 INFO:tasks.rados.rados.0.smithi045.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2024-04-26T00:20:28.130 INFO:tasks.rados.rados.0.smithi045.stdout:9: finishing write tid 1 to smithi04526579-9 2024-04-26T00:20:28.131 INFO:tasks.rados.rados.0.smithi045.stdout:8: done (15 left) 2024-04-26T00:20:28.131 INFO:tasks.rados.rados.0.smithi045.stdout:24: write initial oid 24 2024-04-26T00:20:28.131 INFO:tasks.rados.rados.0.smithi045.stdout:24: seq_num 23 ranges {786561=479786,1796027=467664} 2024-04-26T00:20:28.137 INFO:tasks.rados.rados.0.smithi045.stdout:24: writing smithi04526579-24 from 786561 to 1266347 tid 1 2024-04-26T00:20:28.143 INFO:tasks.rados.rados.0.smithi045.stdout:24: writing smithi04526579-24 from 1796027 to 2263691 tid 2 2024-04-26T00:20:28.143 INFO:tasks.rados.rados.0.smithi045.stdout: waiting on 16 2024-04-26T00:20:28.143 INFO:tasks.rados.rados.0.smithi045.stdout:9: finishing write tid 2 to smithi04526579-9 2024-04-26T00:20:28.143 INFO:tasks.rados.rados.0.smithi045.stdout:9: finishing write tid 3 to smithi04526579-9 2024-04-26T00:20:28.144 INFO:tasks.rados.rados.0.smithi045.stdout:9: finishing write tid 4 to smithi04526579-9 2024-04-26T00:20:28.144 INFO:tasks.rados.rados.0.smithi045.stdout:9: oid 9 updating version 0 to 9 2024-04-26T00:20:28.144 INFO:tasks.rados.rados.0.smithi045.stdout:9: oid 9 version 9 is already newer than 8 2024-04-26T00:20:28.144 INFO:tasks.rados.rados.0.smithi045.stdout:9: oid 9 version 9 is already newer than 7 2024-04-26T00:20:28.144 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 9 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-04-26T00:20:28.144 INFO:tasks.rados.rados.0.smithi045.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2024-04-26T00:20:28.144 INFO:tasks.rados.rados.0.smithi045.stdout:10: finishing write tid 1 to smithi04526579-10 2024-04-26T00:20:28.144 INFO:tasks.rados.rados.0.smithi045.stdout:10: finishing write tid 2 to smithi04526579-10 2024-04-26T00:20:28.144 INFO:tasks.rados.rados.0.smithi045.stdout:10: finishing write tid 3 to smithi04526579-10 2024-04-26T00:20:28.144 INFO:tasks.rados.rados.0.smithi045.stdout:10: finishing write tid 4 to smithi04526579-10 2024-04-26T00:20:28.144 INFO:tasks.rados.rados.0.smithi045.stdout:10: oid 10 updating version 0 to 1 2024-04-26T00:20:28.145 INFO:tasks.rados.rados.0.smithi045.stdout:10: oid 10 updating version 1 to 3 2024-04-26T00:20:28.145 INFO:tasks.rados.rados.0.smithi045.stdout:10: oid 10 version 3 is already newer than 2 2024-04-26T00:20:28.145 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 3 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-04-26T00:20:28.145 INFO:tasks.rados.rados.0.smithi045.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2024-04-26T00:20:28.145 INFO:tasks.rados.rados.0.smithi045.stdout:11: finishing write tid 1 to smithi04526579-11 2024-04-26T00:20:28.145 INFO:tasks.rados.rados.0.smithi045.stdout:11: finishing write tid 2 to smithi04526579-11 2024-04-26T00:20:28.145 INFO:tasks.rados.rados.0.smithi045.stdout:11: finishing write tid 3 to smithi04526579-11 2024-04-26T00:20:28.145 INFO:tasks.rados.rados.0.smithi045.stdout:11: finishing write tid 4 to smithi04526579-11 2024-04-26T00:20:28.145 INFO:tasks.rados.rados.0.smithi045.stdout:11: oid 11 updating version 0 to 1 2024-04-26T00:20:28.145 INFO:tasks.rados.rados.0.smithi045.stdout:11: oid 11 updating version 1 to 3 2024-04-26T00:20:28.145 INFO:tasks.rados.rados.0.smithi045.stdout:11: oid 11 version 3 is already newer than 2 2024-04-26T00:20:28.145 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 3 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-04-26T00:20:28.145 INFO:tasks.rados.rados.0.smithi045.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2024-04-26T00:20:28.145 INFO:tasks.rados.rados.0.smithi045.stdout:9: done (15 left) 2024-04-26T00:20:28.145 INFO:tasks.rados.rados.0.smithi045.stdout:10: done (14 left) 2024-04-26T00:20:28.146 INFO:tasks.rados.rados.0.smithi045.stdout:11: done (13 left) 2024-04-26T00:20:28.146 INFO:tasks.rados.rados.0.smithi045.stdout:25: write initial oid 25 2024-04-26T00:20:28.146 INFO:tasks.rados.rados.0.smithi045.stdout:25: seq_num 24 ranges {403368=555035,1454509=727675} 2024-04-26T00:20:28.149 INFO:tasks.rados.rados.0.smithi045.stdout:25: writing smithi04526579-25 from 403368 to 958403 tid 1 2024-04-26T00:20:28.155 INFO:tasks.rados.rados.0.smithi045.stdout:25: writing smithi04526579-25 from 1454509 to 2182184 tid 2 2024-04-26T00:20:28.155 INFO:tasks.rados.rados.0.smithi045.stdout:26: write initial oid 26 2024-04-26T00:20:28.155 INFO:tasks.rados.rados.0.smithi045.stdout:26: seq_num 25 ranges {420175=630284,1796638=471333} 2024-04-26T00:20:28.161 INFO:tasks.rados.rados.0.smithi045.stdout:26: writing smithi04526579-26 from 420175 to 1050459 tid 1 2024-04-26T00:20:28.168 INFO:tasks.rados.rados.0.smithi045.stdout:26: writing smithi04526579-26 from 1796638 to 2267971 tid 2 2024-04-26T00:20:28.168 INFO:tasks.rados.rados.0.smithi045.stdout:27: write initial oid 27 2024-04-26T00:20:28.168 INFO:tasks.rados.rados.0.smithi045.stdout:27: seq_num 26 ranges {436982=705533,1855120=581862} 2024-04-26T00:20:28.174 INFO:tasks.rados.rados.0.smithi045.stdout:27: writing smithi04526579-27 from 436982 to 1142515 tid 1 2024-04-26T00:20:28.183 INFO:tasks.rados.rados.0.smithi045.stdout:27: writing smithi04526579-27 from 1855120 to 2436982 tid 2 2024-04-26T00:20:28.183 INFO:tasks.rados.rados.0.smithi045.stdout: waiting on 16 2024-04-26T00:20:28.183 INFO:tasks.rados.rados.0.smithi045.stdout:12: finishing write tid 1 to smithi04526579-12 2024-04-26T00:20:28.183 INFO:tasks.rados.rados.0.smithi045.stdout:12: finishing write tid 2 to smithi04526579-12 2024-04-26T00:20:28.183 INFO:tasks.rados.rados.0.smithi045.stdout:12: finishing write tid 3 to smithi04526579-12 2024-04-26T00:20:28.183 INFO:tasks.rados.rados.0.smithi045.stdout:12: finishing write tid 4 to smithi04526579-12 2024-04-26T00:20:28.183 INFO:tasks.rados.rados.0.smithi045.stdout:12: oid 12 updating version 0 to 2 2024-04-26T00:20:28.183 INFO:tasks.rados.rados.0.smithi045.stdout:12: oid 12 version 2 is already newer than 1 2024-04-26T00:20:28.183 INFO:tasks.rados.rados.0.smithi045.stdout:12: oid 12 updating version 2 to 3 2024-04-26T00:20:28.183 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 3 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-04-26T00:20:28.184 INFO:tasks.rados.rados.0.smithi045.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2024-04-26T00:20:28.184 INFO:tasks.rados.rados.0.smithi045.stdout:13: finishing write tid 1 to smithi04526579-13 2024-04-26T00:20:28.184 INFO:tasks.rados.rados.0.smithi045.stdout:13: finishing write tid 2 to smithi04526579-13 2024-04-26T00:20:28.184 INFO:tasks.rados.rados.0.smithi045.stdout:13: finishing write tid 3 to smithi04526579-13 2024-04-26T00:20:28.184 INFO:tasks.rados.rados.0.smithi045.stdout:13: finishing write tid 4 to smithi04526579-13 2024-04-26T00:20:28.184 INFO:tasks.rados.rados.0.smithi045.stdout:13: oid 13 updating version 0 to 5 2024-04-26T00:20:28.184 INFO:tasks.rados.rados.0.smithi045.stdout:13: oid 13 updating version 5 to 6 2024-04-26T00:20:28.184 INFO:tasks.rados.rados.0.smithi045.stdout:13: oid 13 version 6 is already newer than 4 2024-04-26T00:20:28.184 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 6 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-04-26T00:20:28.184 INFO:tasks.rados.rados.0.smithi045.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2024-04-26T00:20:28.184 INFO:tasks.rados.rados.0.smithi045.stdout:12: done (15 left) 2024-04-26T00:20:28.184 INFO:tasks.rados.rados.0.smithi045.stdout:13: done (14 left) 2024-04-26T00:20:28.184 INFO:tasks.rados.rados.0.smithi045.stdout:28: write initial oid 28 2024-04-26T00:20:28.184 INFO:tasks.rados.rados.0.smithi045.stdout:28: seq_num 27 ranges {453789=780782,1913602=475002} 2024-04-26T00:20:28.189 INFO:tasks.rados.rados.0.smithi045.stdout:28: writing smithi04526579-28 from 453789 to 1234571 tid 1 2024-04-26T00:20:28.196 INFO:tasks.rados.rados.0.smithi045.stdout:28: writing smithi04526579-28 from 1913602 to 2388604 tid 2 2024-04-26T00:20:28.196 INFO:tasks.rados.rados.0.smithi045.stdout:29: write initial oid 29 2024-04-26T00:20:28.196 INFO:tasks.rados.rados.0.smithi045.stdout:29: seq_num 28 ranges {470596=456031,1572084=618660} 2024-04-26T00:20:28.201 INFO:tasks.rados.rados.0.smithi045.stdout:29: writing smithi04526579-29 from 470596 to 926627 tid 1 2024-04-26T00:20:28.210 INFO:tasks.rados.rados.0.smithi045.stdout:29: writing smithi04526579-29 from 1572084 to 2190744 tid 2 2024-04-26T00:20:28.210 INFO:tasks.rados.rados.0.smithi045.stdout: waiting on 16 2024-04-26T00:20:28.210 INFO:tasks.rados.rados.0.smithi045.stdout:14: finishing write tid 1 to smithi04526579-14 2024-04-26T00:20:28.210 INFO:tasks.rados.rados.0.smithi045.stdout:14: finishing write tid 2 to smithi04526579-14 2024-04-26T00:20:28.211 INFO:tasks.rados.rados.0.smithi045.stdout:14: finishing write tid 3 to smithi04526579-14 2024-04-26T00:20:28.211 INFO:tasks.rados.rados.0.smithi045.stdout:14: finishing write tid 4 to smithi04526579-14 2024-04-26T00:20:28.211 INFO:tasks.rados.rados.0.smithi045.stdout:14: oid 14 updating version 0 to 2 2024-04-26T00:20:28.211 INFO:tasks.rados.rados.0.smithi045.stdout:14: oid 14 updating version 2 to 3 2024-04-26T00:20:28.211 INFO:tasks.rados.rados.0.smithi045.stdout:14: oid 14 version 3 is already newer than 1 2024-04-26T00:20:28.211 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 3 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-04-26T00:20:28.211 INFO:tasks.rados.rados.0.smithi045.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2024-04-26T00:20:28.211 INFO:tasks.rados.rados.0.smithi045.stdout:15: finishing write tid 1 to smithi04526579-15 2024-04-26T00:20:28.211 INFO:tasks.rados.rados.0.smithi045.stdout:15: finishing write tid 2 to smithi04526579-15 2024-04-26T00:20:28.211 INFO:tasks.rados.rados.0.smithi045.stdout:15: finishing write tid 3 to smithi04526579-15 2024-04-26T00:20:28.211 INFO:tasks.rados.rados.0.smithi045.stdout:15: finishing write tid 4 to smithi04526579-15 2024-04-26T00:20:28.211 INFO:tasks.rados.rados.0.smithi045.stdout:15: oid 15 updating version 0 to 4 2024-04-26T00:20:28.211 INFO:tasks.rados.rados.0.smithi045.stdout:15: oid 15 updating version 4 to 5 2024-04-26T00:20:28.211 INFO:tasks.rados.rados.0.smithi045.stdout:15: oid 15 updating version 5 to 6 2024-04-26T00:20:28.211 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 6 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-04-26T00:20:28.212 INFO:tasks.rados.rados.0.smithi045.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2024-04-26T00:20:28.212 INFO:tasks.rados.rados.0.smithi045.stdout:16: finishing write tid 1 to smithi04526579-16 2024-04-26T00:20:28.212 INFO:tasks.rados.rados.0.smithi045.stdout:16: finishing write tid 2 to smithi04526579-16 2024-04-26T00:20:28.212 INFO:tasks.rados.rados.0.smithi045.stdout:16: finishing write tid 3 to smithi04526579-16 2024-04-26T00:20:28.212 INFO:tasks.rados.rados.0.smithi045.stdout:16: finishing write tid 4 to smithi04526579-16 2024-04-26T00:20:28.212 INFO:tasks.rados.rados.0.smithi045.stdout:16: oid 16 updating version 0 to 6 2024-04-26T00:20:28.212 INFO:tasks.rados.rados.0.smithi045.stdout:16: oid 16 version 6 is already newer than 5 2024-04-26T00:20:28.212 INFO:tasks.rados.rados.0.smithi045.stdout:16: oid 16 version 6 is already newer than 4 2024-04-26T00:20:28.212 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 6 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-04-26T00:20:28.212 INFO:tasks.rados.rados.0.smithi045.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2024-04-26T00:20:28.212 INFO:tasks.rados.rados.0.smithi045.stdout:17: finishing write tid 1 to smithi04526579-17 2024-04-26T00:20:28.212 INFO:tasks.rados.rados.0.smithi045.stdout:17: finishing write tid 2 to smithi04526579-17 2024-04-26T00:20:28.212 INFO:tasks.rados.rados.0.smithi045.stdout:17: finishing write tid 3 to smithi04526579-17 2024-04-26T00:20:28.212 INFO:tasks.rados.rados.0.smithi045.stdout:17: finishing write tid 4 to smithi04526579-17 2024-04-26T00:20:28.213 INFO:tasks.rados.rados.0.smithi045.stdout:17: oid 17 updating version 0 to 6 2024-04-26T00:20:28.213 INFO:tasks.rados.rados.0.smithi045.stdout:17: oid 17 version 6 is already newer than 5 2024-04-26T00:20:28.213 INFO:tasks.rados.rados.0.smithi045.stdout:17: oid 17 version 6 is already newer than 4 2024-04-26T00:20:28.213 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 6 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-04-26T00:20:28.213 INFO:tasks.rados.rados.0.smithi045.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2024-04-26T00:20:28.213 INFO:tasks.rados.rados.0.smithi045.stdout:18: finishing write tid 1 to smithi04526579-18 2024-04-26T00:20:28.213 INFO:tasks.rados.rados.0.smithi045.stdout:18: finishing write tid 2 to smithi04526579-18 2024-04-26T00:20:28.213 INFO:tasks.rados.rados.0.smithi045.stdout:18: finishing write tid 3 to smithi04526579-18 2024-04-26T00:20:28.213 INFO:tasks.rados.rados.0.smithi045.stdout:18: finishing write tid 4 to smithi04526579-18 2024-04-26T00:20:28.213 INFO:tasks.rados.rados.0.smithi045.stdout:18: oid 18 updating version 0 to 8 2024-04-26T00:20:28.214 INFO:tasks.rados.rados.0.smithi045.stdout:18: oid 18 version 8 is already newer than 7 2024-04-26T00:20:28.214 INFO:tasks.rados.rados.0.smithi045.stdout:18: oid 18 updating version 8 to 9 2024-04-26T00:20:28.214 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 9 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-04-26T00:20:28.214 INFO:tasks.rados.rados.0.smithi045.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2024-04-26T00:20:28.214 INFO:tasks.rados.rados.0.smithi045.stdout:19: finishing write tid 1 to smithi04526579-19 2024-04-26T00:20:28.214 INFO:tasks.rados.rados.0.smithi045.stdout:19: finishing write tid 2 to smithi04526579-19 2024-04-26T00:20:28.214 INFO:tasks.rados.rados.0.smithi045.stdout:19: finishing write tid 3 to smithi04526579-19 2024-04-26T00:20:28.214 INFO:tasks.rados.rados.0.smithi045.stdout:19: finishing write tid 4 to smithi04526579-19 2024-04-26T00:20:28.214 INFO:tasks.rados.rados.0.smithi045.stdout:19: oid 19 updating version 0 to 2 2024-04-26T00:20:28.214 INFO:tasks.rados.rados.0.smithi045.stdout:19: oid 19 version 2 is already newer than 1 2024-04-26T00:20:28.214 INFO:tasks.rados.rados.0.smithi045.stdout:19: oid 19 updating version 2 to 3 2024-04-26T00:20:28.215 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 3 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-04-26T00:20:28.215 INFO:tasks.rados.rados.0.smithi045.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2024-04-26T00:20:28.215 INFO:tasks.rados.rados.0.smithi045.stdout:20: finishing write tid 1 to smithi04526579-20 2024-04-26T00:20:28.215 INFO:tasks.rados.rados.0.smithi045.stdout:20: finishing write tid 2 to smithi04526579-20 2024-04-26T00:20:28.215 INFO:tasks.rados.rados.0.smithi045.stdout:21: finishing write tid 1 to smithi04526579-21 2024-04-26T00:20:28.215 INFO:tasks.rados.rados.0.smithi045.stdout:20: finishing write tid 3 to smithi04526579-20 2024-04-26T00:20:28.215 INFO:tasks.rados.rados.0.smithi045.stdout:21: finishing write tid 2 to smithi04526579-21 2024-04-26T00:20:28.215 INFO:tasks.rados.rados.0.smithi045.stdout:20: finishing write tid 4 to smithi04526579-20 2024-04-26T00:20:28.215 INFO:tasks.rados.rados.0.smithi045.stdout:20: oid 20 updating version 0 to 2 2024-04-26T00:20:28.215 INFO:tasks.rados.rados.0.smithi045.stdout:20: oid 20 version 2 is already newer than 1 2024-04-26T00:20:28.215 INFO:tasks.rados.rados.0.smithi045.stdout:20: oid 20 updating version 2 to 3 2024-04-26T00:20:28.215 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 3 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-04-26T00:20:28.215 INFO:tasks.rados.rados.0.smithi045.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2024-04-26T00:20:28.215 INFO:tasks.rados.rados.0.smithi045.stdout:21: finishing write tid 3 to smithi04526579-21 2024-04-26T00:20:28.215 INFO:tasks.rados.rados.0.smithi045.stdout:21: finishing write tid 4 to smithi04526579-21 2024-04-26T00:20:28.215 INFO:tasks.rados.rados.0.smithi045.stdout:21: oid 21 updating version 0 to 2 2024-04-26T00:20:28.216 INFO:tasks.rados.rados.0.smithi045.stdout:21: oid 21 updating version 2 to 3 2024-04-26T00:20:28.216 INFO:tasks.rados.rados.0.smithi045.stdout:21: oid 21 version 3 is already newer than 1 2024-04-26T00:20:28.216 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 3 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-04-26T00:20:28.216 INFO:tasks.rados.rados.0.smithi045.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2024-04-26T00:20:28.216 INFO:tasks.rados.rados.0.smithi045.stdout:22: finishing write tid 1 to smithi04526579-22 2024-04-26T00:20:28.216 INFO:tasks.rados.rados.0.smithi045.stdout:22: finishing write tid 2 to smithi04526579-22 2024-04-26T00:20:28.216 INFO:tasks.rados.rados.0.smithi045.stdout:22: finishing write tid 3 to smithi04526579-22 2024-04-26T00:20:28.216 INFO:tasks.rados.rados.0.smithi045.stdout:22: finishing write tid 4 to smithi04526579-22 2024-04-26T00:20:28.216 INFO:tasks.rados.rados.0.smithi045.stdout:22: oid 22 updating version 0 to 2 2024-04-26T00:20:28.216 INFO:tasks.rados.rados.0.smithi045.stdout:22: oid 22 version 2 is already newer than 1 2024-04-26T00:20:28.216 INFO:tasks.rados.rados.0.smithi045.stdout:22: oid 22 updating version 2 to 3 2024-04-26T00:20:28.216 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 3 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-04-26T00:20:28.216 INFO:tasks.rados.rados.0.smithi045.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2024-04-26T00:20:28.216 INFO:tasks.rados.rados.0.smithi045.stdout:23: finishing write tid 1 to smithi04526579-23 2024-04-26T00:20:28.216 INFO:tasks.rados.rados.0.smithi045.stdout:23: finishing write tid 2 to smithi04526579-23 2024-04-26T00:20:28.217 INFO:tasks.rados.rados.0.smithi045.stdout:23: finishing write tid 3 to smithi04526579-23 2024-04-26T00:20:28.217 INFO:tasks.rados.rados.0.smithi045.stdout:23: finishing write tid 4 to smithi04526579-23 2024-04-26T00:20:28.217 INFO:tasks.rados.rados.0.smithi045.stdout:23: oid 23 updating version 0 to 4 2024-04-26T00:20:28.217 INFO:tasks.rados.rados.0.smithi045.stdout:23: oid 23 updating version 4 to 5 2024-04-26T00:20:28.217 INFO:tasks.rados.rados.0.smithi045.stdout:23: oid 23 updating version 5 to 6 2024-04-26T00:20:28.217 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 6 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-04-26T00:20:28.217 INFO:tasks.rados.rados.0.smithi045.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2024-04-26T00:20:28.217 INFO:tasks.rados.rados.0.smithi045.stdout:24: finishing write tid 1 to smithi04526579-24 2024-04-26T00:20:28.217 INFO:tasks.rados.rados.0.smithi045.stdout:14: done (15 left) 2024-04-26T00:20:28.217 INFO:tasks.rados.rados.0.smithi045.stdout:15: done (14 left) 2024-04-26T00:20:28.217 INFO:tasks.rados.rados.0.smithi045.stdout:16: done (13 left) 2024-04-26T00:20:28.217 INFO:tasks.rados.rados.0.smithi045.stdout:17: done (12 left) 2024-04-26T00:20:28.217 INFO:tasks.rados.rados.0.smithi045.stdout:18: done (11 left) 2024-04-26T00:20:28.217 INFO:tasks.rados.rados.0.smithi045.stdout:19: done (10 left) 2024-04-26T00:20:28.218 INFO:tasks.rados.rados.0.smithi045.stdout:20: done (9 left) 2024-04-26T00:20:28.218 INFO:tasks.rados.rados.0.smithi045.stdout:21: done (8 left) 2024-04-26T00:20:28.218 INFO:tasks.rados.rados.0.smithi045.stdout:22: done (7 left) 2024-04-26T00:20:28.218 INFO:tasks.rados.rados.0.smithi045.stdout:23: done (6 left) 2024-04-26T00:20:28.218 INFO:tasks.rados.rados.0.smithi045.stdout:30: write initial oid 30 2024-04-26T00:20:28.218 INFO:tasks.rados.rados.0.smithi045.stdout:30: seq_num 29 ranges {487403=531280,1514213=478671} 2024-04-26T00:20:28.218 INFO:tasks.rados.rados.0.smithi045.stdout:30: writing smithi04526579-30 from 487403 to 1018683 tid 1 2024-04-26T00:20:28.220 INFO:tasks.rados.rados.0.smithi045.stdout:30: writing smithi04526579-30 from 1514213 to 1992884 tid 2 2024-04-26T00:20:28.220 INFO:tasks.rados.rados.0.smithi045.stdout:31: write initial oid 31 2024-04-26T00:20:28.220 INFO:tasks.rados.rados.0.smithi045.stdout:31: seq_num 30 ranges {504210=606529,1572695=622329} 2024-04-26T00:20:28.227 INFO:tasks.rados.rados.0.smithi045.stdout:31: writing smithi04526579-31 from 504210 to 1110739 tid 1 2024-04-26T00:20:28.233 INFO:tasks.rados.rados.0.smithi045.stdout:31: writing smithi04526579-31 from 1572695 to 2195024 tid 2 2024-04-26T00:20:28.233 INFO:tasks.rados.rados.0.smithi045.stdout:32: write initial oid 32 2024-04-26T00:20:28.233 INFO:tasks.rados.rados.0.smithi045.stdout:32: seq_num 31 ranges {521017=798131,1747530=482340} 2024-04-26T00:20:28.241 INFO:tasks.rados.rados.0.smithi045.stdout:32: writing smithi04526579-32 from 521017 to 1319148 tid 1 2024-04-26T00:20:28.246 INFO:tasks.rados.rados.0.smithi045.stdout:32: writing smithi04526579-32 from 1747530 to 2229870 tid 2 2024-04-26T00:20:28.246 INFO:tasks.rados.rados.0.smithi045.stdout:33: write initial oid 33 2024-04-26T00:20:28.246 INFO:tasks.rados.rados.0.smithi045.stdout:33: seq_num 32 ranges {537824=473380,1806012=625998} 2024-04-26T00:20:28.251 INFO:tasks.rados.rados.0.smithi045.stdout:33: writing smithi04526579-33 from 537824 to 1011204 tid 1 2024-04-26T00:20:28.259 INFO:tasks.rados.rados.0.smithi045.stdout:33: writing smithi04526579-33 from 1806012 to 2432010 tid 2 2024-04-26T00:20:28.259 INFO:tasks.rados.rados.0.smithi045.stdout:34: write initial oid 34 2024-04-26T00:20:28.260 INFO:tasks.rados.rados.0.smithi045.stdout:34: seq_num 33 ranges {554631=548629,1748141=486009} 2024-04-26T00:20:28.265 INFO:tasks.rados.rados.0.smithi045.stdout:34: writing smithi04526579-34 from 554631 to 1103260 tid 1 2024-04-26T00:20:28.272 INFO:tasks.rados.rados.0.smithi045.stdout:34: writing smithi04526579-34 from 1748141 to 2234150 tid 2 2024-04-26T00:20:28.272 INFO:tasks.rados.rados.0.smithi045.stdout:35: write initial oid 35 2024-04-26T00:20:28.272 INFO:tasks.rados.rados.0.smithi045.stdout:35: seq_num 34 ranges {571438=623878,1806623=629667} 2024-04-26T00:20:28.279 INFO:tasks.rados.rados.0.smithi045.stdout:35: writing smithi04526579-35 from 571438 to 1195316 tid 1 2024-04-26T00:20:28.286 INFO:tasks.rados.rados.0.smithi045.stdout:35: writing smithi04526579-35 from 1806623 to 2436290 tid 2 2024-04-26T00:20:28.286 INFO:tasks.rados.rados.0.smithi045.stdout:36: write initial oid 36 2024-04-26T00:20:28.286 INFO:tasks.rados.rados.0.smithi045.stdout:36: seq_num 35 ranges {588245=699127,1865105=489678} 2024-04-26T00:20:28.293 INFO:tasks.rados.rados.0.smithi045.stdout:36: writing smithi04526579-36 from 588245 to 1287372 tid 1 2024-04-26T00:20:28.300 INFO:tasks.rados.rados.0.smithi045.stdout:36: writing smithi04526579-36 from 1865105 to 2354783 tid 2 2024-04-26T00:20:28.300 INFO:tasks.rados.rados.0.smithi045.stdout:37: write initial oid 37 2024-04-26T00:20:28.300 INFO:tasks.rados.rados.0.smithi045.stdout:37: seq_num 36 ranges {605052=774376,1923587=633336} 2024-04-26T00:20:28.307 INFO:tasks.rados.rados.0.smithi045.stdout:37: writing smithi04526579-37 from 605052 to 1379428 tid 1 2024-04-26T00:20:28.314 INFO:tasks.rados.rados.0.smithi045.stdout:37: writing smithi04526579-37 from 1923587 to 2556923 tid 2 2024-04-26T00:20:28.314 INFO:tasks.rados.rados.0.smithi045.stdout:38: write initial oid 38 2024-04-26T00:20:28.314 INFO:tasks.rados.rados.0.smithi045.stdout:38: seq_num 37 ranges {621859=449625,1865716=756143} 2024-04-26T00:20:28.320 INFO:tasks.rados.rados.0.smithi045.stdout:38: writing smithi04526579-38 from 621859 to 1071484 tid 1 2024-04-26T00:20:28.329 INFO:tasks.rados.rados.0.smithi045.stdout:38: writing smithi04526579-38 from 1865716 to 2621859 tid 2 2024-04-26T00:20:28.329 INFO:tasks.rados.rados.0.smithi045.stdout:39: write initial oid 39 2024-04-26T00:20:28.330 INFO:tasks.rados.rados.0.smithi045.stdout:39: seq_num 38 ranges {638666=524874,1924198=637005} 2024-04-26T00:20:28.335 INFO:tasks.rados.rados.0.smithi045.stdout:39: writing smithi04526579-39 from 638666 to 1163540 tid 1 2024-04-26T00:20:28.344 INFO:tasks.rados.rados.0.smithi045.stdout:39: writing smithi04526579-39 from 1924198 to 2561203 tid 2 2024-04-26T00:20:28.344 INFO:tasks.rados.rados.0.smithi045.stdout: waiting on 16 2024-04-26T00:20:28.344 INFO:tasks.rados.rados.0.smithi045.stdout:24: finishing write tid 2 to smithi04526579-24 2024-04-26T00:20:28.344 INFO:tasks.rados.rados.0.smithi045.stdout:24: finishing write tid 3 to smithi04526579-24 2024-04-26T00:20:28.344 INFO:tasks.rados.rados.0.smithi045.stdout:24: finishing write tid 4 to smithi04526579-24 2024-04-26T00:20:28.344 INFO:tasks.rados.rados.0.smithi045.stdout:24: oid 24 updating version 0 to 11 2024-04-26T00:20:28.344 INFO:tasks.rados.rados.0.smithi045.stdout:24: oid 24 updating version 11 to 12 2024-04-26T00:20:28.345 INFO:tasks.rados.rados.0.smithi045.stdout:24: oid 24 version 12 is already newer than 10 2024-04-26T00:20:28.345 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 12 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-04-26T00:20:28.345 INFO:tasks.rados.rados.0.smithi045.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2024-04-26T00:20:28.345 INFO:tasks.rados.rados.0.smithi045.stdout:25: finishing write tid 1 to smithi04526579-25 2024-04-26T00:20:28.345 INFO:tasks.rados.rados.0.smithi045.stdout:25: finishing write tid 2 to smithi04526579-25 2024-04-26T00:20:28.345 INFO:tasks.rados.rados.0.smithi045.stdout:25: finishing write tid 3 to smithi04526579-25 2024-04-26T00:20:28.345 INFO:tasks.rados.rados.0.smithi045.stdout:25: finishing write tid 4 to smithi04526579-25 2024-04-26T00:20:28.345 INFO:tasks.rados.rados.0.smithi045.stdout:25: oid 25 updating version 0 to 14 2024-04-26T00:20:28.345 INFO:tasks.rados.rados.0.smithi045.stdout:25: oid 25 version 14 is already newer than 13 2024-04-26T00:20:28.345 INFO:tasks.rados.rados.0.smithi045.stdout:25: oid 25 updating version 14 to 15 2024-04-26T00:20:28.345 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 15 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-04-26T00:20:28.345 INFO:tasks.rados.rados.0.smithi045.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2024-04-26T00:20:28.345 INFO:tasks.rados.rados.0.smithi045.stdout:26: finishing write tid 1 to smithi04526579-26 2024-04-26T00:20:28.345 INFO:tasks.rados.rados.0.smithi045.stdout:26: finishing write tid 2 to smithi04526579-26 2024-04-26T00:20:28.345 INFO:tasks.rados.rados.0.smithi045.stdout:26: finishing write tid 3 to smithi04526579-26 2024-04-26T00:20:28.346 INFO:tasks.rados.rados.0.smithi045.stdout:26: finishing write tid 4 to smithi04526579-26 2024-04-26T00:20:28.346 INFO:tasks.rados.rados.0.smithi045.stdout:26: oid 26 updating version 0 to 7 2024-04-26T00:20:28.346 INFO:tasks.rados.rados.0.smithi045.stdout:26: oid 26 updating version 7 to 9 2024-04-26T00:20:28.346 INFO:tasks.rados.rados.0.smithi045.stdout:26: oid 26 version 9 is already newer than 8 2024-04-26T00:20:28.346 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 9 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-04-26T00:20:28.346 INFO:tasks.rados.rados.0.smithi045.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2024-04-26T00:20:28.346 INFO:tasks.rados.rados.0.smithi045.stdout:27: finishing write tid 1 to smithi04526579-27 2024-04-26T00:20:28.346 INFO:tasks.rados.rados.0.smithi045.stdout:27: finishing write tid 2 to smithi04526579-27 2024-04-26T00:20:28.346 INFO:tasks.rados.rados.0.smithi045.stdout:24: done (15 left) 2024-04-26T00:20:28.346 INFO:tasks.rados.rados.0.smithi045.stdout:25: done (14 left) 2024-04-26T00:20:28.346 INFO:tasks.rados.rados.0.smithi045.stdout:26: done (13 left) 2024-04-26T00:20:28.346 INFO:tasks.rados.rados.0.smithi045.stdout:40: write initial oid 40 2024-04-26T00:20:28.346 INFO:tasks.rados.rados.0.smithi045.stdout:40: seq_num 39 ranges {655473=716476,2099033=556440} 2024-04-26T00:20:28.351 INFO:tasks.rados.rados.0.smithi045.stdout:40: writing smithi04526579-40 from 655473 to 1371949 tid 1 2024-04-26T00:20:28.358 INFO:tasks.rados.rados.0.smithi045.stdout:40: writing smithi04526579-40 from 2099033 to 2655473 tid 2 2024-04-26T00:20:28.358 INFO:tasks.rados.rados.0.smithi045.stdout:41: write initial oid 41 2024-04-26T00:20:28.359 INFO:tasks.rados.rados.0.smithi045.stdout:41: seq_num 40 ranges {672280=791725,2157515=514765} 2024-04-26T00:20:28.367 INFO:tasks.rados.rados.0.smithi045.stdout:41: writing smithi04526579-41 from 672280 to 1464005 tid 1 2024-04-26T00:20:28.374 INFO:tasks.rados.rados.0.smithi045.stdout:41: writing smithi04526579-41 from 2157515 to 2672280 tid 2 2024-04-26T00:20:28.374 INFO:tasks.rados.rados.0.smithi045.stdout:42: write initial oid 42 2024-04-26T00:20:28.374 INFO:tasks.rados.rados.0.smithi045.stdout:42: seq_num 41 ranges {689087=466974,1699644=784332} 2024-04-26T00:20:28.381 INFO:tasks.rados.rados.0.smithi045.stdout:42: writing smithi04526579-42 from 689087 to 1156061 tid 1 2024-04-26T00:20:28.389 INFO:tasks.rados.rados.0.smithi045.stdout:42: writing smithi04526579-42 from 1699644 to 2483976 tid 2 2024-04-26T00:20:28.389 INFO:tasks.rados.rados.0.smithi045.stdout: waiting on 16 2024-04-26T00:20:28.389 INFO:tasks.rados.rados.0.smithi045.stdout:27: finishing write tid 3 to smithi04526579-27 2024-04-26T00:20:28.389 INFO:tasks.rados.rados.0.smithi045.stdout:27: finishing write tid 4 to smithi04526579-27 2024-04-26T00:20:28.389 INFO:tasks.rados.rados.0.smithi045.stdout:27: oid 27 updating version 0 to 5 2024-04-26T00:20:28.389 INFO:tasks.rados.rados.0.smithi045.stdout:27: oid 27 version 5 is already newer than 4 2024-04-26T00:20:28.389 INFO:tasks.rados.rados.0.smithi045.stdout:27: oid 27 updating version 5 to 6 2024-04-26T00:20:28.389 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 6 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-26T00:20:28.389 INFO:tasks.rados.rados.0.smithi045.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2024-04-26T00:20:28.389 INFO:tasks.rados.rados.0.smithi045.stdout:28: finishing write tid 1 to smithi04526579-28 2024-04-26T00:20:28.389 INFO:tasks.rados.rados.0.smithi045.stdout:28: finishing write tid 2 to smithi04526579-28 2024-04-26T00:20:28.389 INFO:tasks.rados.rados.0.smithi045.stdout:28: finishing write tid 3 to smithi04526579-28 2024-04-26T00:20:28.390 INFO:tasks.rados.rados.0.smithi045.stdout:28: finishing write tid 4 to smithi04526579-28 2024-04-26T00:20:28.390 INFO:tasks.rados.rados.0.smithi045.stdout:28: oid 28 updating version 0 to 11 2024-04-26T00:20:28.390 INFO:tasks.rados.rados.0.smithi045.stdout:28: oid 28 version 11 is already newer than 10 2024-04-26T00:20:28.390 INFO:tasks.rados.rados.0.smithi045.stdout:28: oid 28 updating version 11 to 12 2024-04-26T00:20:28.390 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 12 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-04-26T00:20:28.390 INFO:tasks.rados.rados.0.smithi045.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2024-04-26T00:20:28.390 INFO:tasks.rados.rados.0.smithi045.stdout:29: finishing write tid 1 to smithi04526579-29 2024-04-26T00:20:28.390 INFO:tasks.rados.rados.0.smithi045.stdout:27: done (15 left) 2024-04-26T00:20:28.390 INFO:tasks.rados.rados.0.smithi045.stdout:28: done (14 left) 2024-04-26T00:20:28.390 INFO:tasks.rados.rados.0.smithi045.stdout:43: write initial oid 43 2024-04-26T00:20:28.390 INFO:tasks.rados.rados.0.smithi045.stdout:43: seq_num 42 ranges {705894=542223,1758126=644343} 2024-04-26T00:20:28.395 INFO:tasks.rados.rados.0.smithi045.stdout:43: writing smithi04526579-43 from 705894 to 1248117 tid 1 2024-04-26T00:20:28.402 INFO:tasks.rados.rados.0.smithi045.stdout:43: writing smithi04526579-43 from 1758126 to 2402469 tid 2 2024-04-26T00:20:28.403 INFO:tasks.rados.rados.0.smithi045.stdout:44: write initial oid 44 2024-04-26T00:20:28.403 INFO:tasks.rados.rados.0.smithi045.stdout:44: seq_num 43 ranges {722701=617472,1816608=788001} 2024-04-26T00:20:28.410 INFO:tasks.rados.rados.0.smithi045.stdout:44: writing smithi04526579-44 from 722701 to 1340173 tid 1 2024-04-26T00:20:28.418 INFO:tasks.rados.rados.0.smithi045.stdout:44: writing smithi04526579-44 from 1816608 to 2604609 tid 2 2024-04-26T00:20:28.418 INFO:tasks.rados.rados.0.smithi045.stdout: waiting on 16 2024-04-26T00:20:28.418 INFO:tasks.rados.rados.0.smithi045.stdout:29: finishing write tid 2 to smithi04526579-29 2024-04-26T00:20:28.418 INFO:tasks.rados.rados.0.smithi045.stdout:29: finishing write tid 3 to smithi04526579-29 2024-04-26T00:20:28.418 INFO:tasks.rados.rados.0.smithi045.stdout:29: finishing write tid 4 to smithi04526579-29 2024-04-26T00:20:28.418 INFO:tasks.rados.rados.0.smithi045.stdout:29: oid 29 updating version 0 to 7 2024-04-26T00:20:28.418 INFO:tasks.rados.rados.0.smithi045.stdout:29: oid 29 updating version 7 to 8 2024-04-26T00:20:28.418 INFO:tasks.rados.rados.0.smithi045.stdout:29: oid 29 updating version 8 to 9 2024-04-26T00:20:28.418 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 9 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-26T00:20:28.418 INFO:tasks.rados.rados.0.smithi045.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2024-04-26T00:20:28.418 INFO:tasks.rados.rados.0.smithi045.stdout:30: finishing write tid 1 to smithi04526579-30 2024-04-26T00:20:28.418 INFO:tasks.rados.rados.0.smithi045.stdout:30: finishing write tid 2 to smithi04526579-30 2024-04-26T00:20:28.419 INFO:tasks.rados.rados.0.smithi045.stdout:29: done (15 left) 2024-04-26T00:20:28.419 INFO:tasks.rados.rados.0.smithi045.stdout:45: write initial oid 45 2024-04-26T00:20:28.419 INFO:tasks.rados.rados.0.smithi045.stdout:45: seq_num 44 ranges {739508=692721,1875090=648012} 2024-04-26T00:20:28.425 INFO:tasks.rados.rados.0.smithi045.stdout:45: writing smithi04526579-45 from 739508 to 1432229 tid 1 2024-04-26T00:20:28.432 INFO:tasks.rados.rados.0.smithi045.stdout:45: writing smithi04526579-45 from 1875090 to 2523102 tid 2 2024-04-26T00:20:28.432 INFO:tasks.rados.rados.0.smithi045.stdout: waiting on 16 2024-04-26T00:20:28.432 INFO:tasks.rados.rados.0.smithi045.stdout:30: finishing write tid 3 to smithi04526579-30 2024-04-26T00:20:28.432 INFO:tasks.rados.rados.0.smithi045.stdout:30: finishing write tid 4 to smithi04526579-30 2024-04-26T00:20:28.432 INFO:tasks.rados.rados.0.smithi045.stdout:30: oid 30 updating version 0 to 5 2024-04-26T00:20:28.432 INFO:tasks.rados.rados.0.smithi045.stdout:30: oid 30 updating version 5 to 6 2024-04-26T00:20:28.432 INFO:tasks.rados.rados.0.smithi045.stdout:30: oid 30 version 6 is already newer than 4 2024-04-26T00:20:28.432 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 6 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-04-26T00:20:28.432 INFO:tasks.rados.rados.0.smithi045.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2024-04-26T00:20:28.432 INFO:tasks.rados.rados.0.smithi045.stdout:31: finishing write tid 1 to smithi04526579-31 2024-04-26T00:20:28.433 INFO:tasks.rados.rados.0.smithi045.stdout:31: finishing write tid 2 to smithi04526579-31 2024-04-26T00:20:28.433 INFO:tasks.rados.rados.0.smithi045.stdout:30: done (15 left) 2024-04-26T00:20:28.433 INFO:tasks.rados.rados.0.smithi045.stdout:46: write initial oid 46 2024-04-26T00:20:28.433 INFO:tasks.rados.rados.0.smithi045.stdout:46: seq_num 45 ranges {756315=767970,1933572=791670} 2024-04-26T00:20:28.440 INFO:tasks.rados.rados.0.smithi045.stdout:46: writing smithi04526579-46 from 756315 to 1524285 tid 1 2024-04-26T00:20:28.448 INFO:tasks.rados.rados.0.smithi045.stdout:46: writing smithi04526579-46 from 1933572 to 2725242 tid 2 2024-04-26T00:20:28.448 INFO:tasks.rados.rados.0.smithi045.stdout: waiting on 16 2024-04-26T00:20:28.448 INFO:tasks.rados.rados.0.smithi045.stdout:31: finishing write tid 3 to smithi04526579-31 2024-04-26T00:20:28.448 INFO:tasks.rados.rados.0.smithi045.stdout:31: finishing write tid 4 to smithi04526579-31 2024-04-26T00:20:28.448 INFO:tasks.rados.rados.0.smithi045.stdout:31: oid 31 updating version 0 to 12 2024-04-26T00:20:28.448 INFO:tasks.rados.rados.0.smithi045.stdout:31: oid 31 version 12 is already newer than 10 2024-04-26T00:20:28.448 INFO:tasks.rados.rados.0.smithi045.stdout:31: oid 31 version 12 is already newer than 11 2024-04-26T00:20:28.449 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 12 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-04-26T00:20:28.449 INFO:tasks.rados.rados.0.smithi045.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2024-04-26T00:20:28.449 INFO:tasks.rados.rados.0.smithi045.stdout:32: finishing write tid 1 to smithi04526579-32 2024-04-26T00:20:28.449 INFO:tasks.rados.rados.0.smithi045.stdout:32: finishing write tid 2 to smithi04526579-32 2024-04-26T00:20:28.449 INFO:tasks.rados.rados.0.smithi045.stdout:31: done (15 left) 2024-04-26T00:20:28.449 INFO:tasks.rados.rados.0.smithi045.stdout:47: write initial oid 47 2024-04-26T00:20:28.449 INFO:tasks.rados.rados.0.smithi045.stdout:47: seq_num 46 ranges {773122=559572,1992054=651681} 2024-04-26T00:20:28.456 INFO:tasks.rados.rados.0.smithi045.stdout:47: writing smithi04526579-47 from 773122 to 1332694 tid 1 2024-04-26T00:20:28.463 INFO:tasks.rados.rados.0.smithi045.stdout:47: writing smithi04526579-47 from 1992054 to 2643735 tid 2 2024-04-26T00:20:28.463 INFO:tasks.rados.rados.0.smithi045.stdout: waiting on 16 2024-04-26T00:20:28.463 INFO:tasks.rados.rados.0.smithi045.stdout:32: finishing write tid 3 to smithi04526579-32 2024-04-26T00:20:28.464 INFO:tasks.rados.rados.0.smithi045.stdout:32: finishing write tid 4 to smithi04526579-32 2024-04-26T00:20:28.464 INFO:tasks.rados.rados.0.smithi045.stdout:32: oid 32 updating version 0 to 8 2024-04-26T00:20:28.464 INFO:tasks.rados.rados.0.smithi045.stdout:32: oid 32 version 8 is already newer than 7 2024-04-26T00:20:28.464 INFO:tasks.rados.rados.0.smithi045.stdout:32: oid 32 updating version 8 to 9 2024-04-26T00:20:28.464 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 9 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-04-26T00:20:28.464 INFO:tasks.rados.rados.0.smithi045.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2024-04-26T00:20:28.464 INFO:tasks.rados.rados.0.smithi045.stdout:33: finishing write tid 1 to smithi04526579-33 2024-04-26T00:20:28.464 INFO:tasks.rados.rados.0.smithi045.stdout:33: finishing write tid 2 to smithi04526579-33 2024-04-26T00:20:28.464 INFO:tasks.rados.rados.0.smithi045.stdout:32: done (15 left) 2024-04-26T00:20:28.464 INFO:tasks.rados.rados.0.smithi045.stdout:48: write initial oid 48 2024-04-26T00:20:28.464 INFO:tasks.rados.rados.0.smithi045.stdout:48: seq_num 47 ranges {789929=634821,2050536=739393} 2024-04-26T00:20:28.471 INFO:tasks.rados.rados.0.smithi045.stdout:48: writing smithi04526579-48 from 789929 to 1424750 tid 1 2024-04-26T00:20:28.479 INFO:tasks.rados.rados.0.smithi045.stdout:48: writing smithi04526579-48 from 2050536 to 2789929 tid 2 2024-04-26T00:20:28.479 INFO:tasks.rados.rados.0.smithi045.stdout: waiting on 16 2024-04-26T00:20:28.479 INFO:tasks.rados.rados.0.smithi045.stdout:33: finishing write tid 3 to smithi04526579-33 2024-04-26T00:20:28.480 INFO:tasks.rados.rados.0.smithi045.stdout:33: finishing write tid 4 to smithi04526579-33 2024-04-26T00:20:28.480 INFO:tasks.rados.rados.0.smithi045.stdout:33: oid 33 updating version 0 to 18 2024-04-26T00:20:28.480 INFO:tasks.rados.rados.0.smithi045.stdout:33: oid 33 version 18 is already newer than 16 2024-04-26T00:20:28.480 INFO:tasks.rados.rados.0.smithi045.stdout:33: oid 33 version 18 is already newer than 17 2024-04-26T00:20:28.480 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 18 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-04-26T00:20:28.480 INFO:tasks.rados.rados.0.smithi045.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2024-04-26T00:20:28.480 INFO:tasks.rados.rados.0.smithi045.stdout:34: finishing write tid 1 to smithi04526579-34 2024-04-26T00:20:28.480 INFO:tasks.rados.rados.0.smithi045.stdout:34: finishing write tid 2 to smithi04526579-34 2024-04-26T00:20:28.480 INFO:tasks.rados.rados.0.smithi045.stdout:33: done (15 left) 2024-04-26T00:20:28.480 INFO:tasks.rados.rados.0.smithi045.stdout:49: write initial oid 49 2024-04-26T00:20:28.480 INFO:tasks.rados.rados.0.smithi045.stdout:49: seq_num 48 ranges {406736=710070,1709018=655350} 2024-04-26T00:20:28.486 INFO:tasks.rados.rados.0.smithi045.stdout:49: writing smithi04526579-49 from 406736 to 1116806 tid 1 2024-04-26T00:20:28.493 INFO:tasks.rados.rados.0.smithi045.stdout:49: writing smithi04526579-49 from 1709018 to 2364368 tid 2 2024-04-26T00:20:28.493 INFO:tasks.rados.rados.0.smithi045.stdout: waiting on 16 2024-04-26T00:20:28.493 INFO:tasks.rados.rados.0.smithi045.stdout:34: finishing write tid 3 to smithi04526579-34 2024-04-26T00:20:28.493 INFO:tasks.rados.rados.0.smithi045.stdout:34: finishing write tid 4 to smithi04526579-34 2024-04-26T00:20:28.493 INFO:tasks.rados.rados.0.smithi045.stdout:34: oid 34 updating version 0 to 5 2024-04-26T00:20:28.493 INFO:tasks.rados.rados.0.smithi045.stdout:34: oid 34 version 5 is already newer than 4 2024-04-26T00:20:28.493 INFO:tasks.rados.rados.0.smithi045.stdout:34: oid 34 updating version 5 to 6 2024-04-26T00:20:28.494 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 6 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-04-26T00:20:28.494 INFO:tasks.rados.rados.0.smithi045.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2024-04-26T00:20:28.494 INFO:tasks.rados.rados.0.smithi045.stdout:35: finishing write tid 1 to smithi04526579-35 2024-04-26T00:20:28.494 INFO:tasks.rados.rados.0.smithi045.stdout:35: finishing write tid 2 to smithi04526579-35 2024-04-26T00:20:28.494 INFO:tasks.rados.rados.0.smithi045.stdout:34: done (15 left) 2024-04-26T00:20:28.494 INFO:tasks.rados.rados.0.smithi045.stdout:50: write initial oid 50 2024-04-26T00:20:28.494 INFO:tasks.rados.rados.0.smithi045.stdout:50: seq_num 49 ranges {423543=785319,1767500=799008} 2024-04-26T00:20:28.500 INFO:tasks.rados.rados.0.smithi045.stdout:50: writing smithi04526579-50 from 423543 to 1208862 tid 1 2024-04-26T00:20:28.509 INFO:tasks.rados.rados.0.smithi045.stdout:50: writing smithi04526579-50 from 1767500 to 2566508 tid 2 2024-04-26T00:20:28.509 INFO:tasks.rados.rados.0.smithi045.stdout: waiting on 16 2024-04-26T00:20:28.509 INFO:tasks.rados.rados.0.smithi045.stdout:35: finishing write tid 3 to smithi04526579-35 2024-04-26T00:20:28.510 INFO:tasks.rados.rados.0.smithi045.stdout:35: finishing write tid 4 to smithi04526579-35 2024-04-26T00:20:28.510 INFO:tasks.rados.rados.0.smithi045.stdout:35: oid 35 updating version 0 to 10 2024-04-26T00:20:28.510 INFO:tasks.rados.rados.0.smithi045.stdout:35: oid 35 updating version 10 to 11 2024-04-26T00:20:28.510 INFO:tasks.rados.rados.0.smithi045.stdout:35: oid 35 updating version 11 to 12 2024-04-26T00:20:28.510 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 12 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-04-26T00:20:28.510 INFO:tasks.rados.rados.0.smithi045.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2024-04-26T00:20:28.511 INFO:tasks.rados.rados.0.smithi045.stdout:36: finishing write tid 1 to smithi04526579-36 2024-04-26T00:20:28.511 INFO:tasks.rados.rados.0.smithi045.stdout:36: finishing write tid 2 to smithi04526579-36 2024-04-26T00:20:28.511 INFO:tasks.rados.rados.0.smithi045.stdout:36: finishing write tid 3 to smithi04526579-36 2024-04-26T00:20:28.511 INFO:tasks.rados.rados.0.smithi045.stdout:36: finishing write tid 4 to smithi04526579-36 2024-04-26T00:20:28.511 INFO:tasks.rados.rados.0.smithi045.stdout:36: oid 36 updating version 0 to 6 2024-04-26T00:20:28.511 INFO:tasks.rados.rados.0.smithi045.stdout:36: oid 36 version 6 is already newer than 4 2024-04-26T00:20:28.511 INFO:tasks.rados.rados.0.smithi045.stdout:36: oid 36 version 6 is already newer than 5 2024-04-26T00:20:28.511 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 6 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-04-26T00:20:28.511 INFO:tasks.rados.rados.0.smithi045.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2024-04-26T00:20:28.511 INFO:tasks.rados.rados.0.smithi045.stdout:37: finishing write tid 1 to smithi04526579-37 2024-04-26T00:20:28.511 INFO:tasks.rados.rados.0.smithi045.stdout:37: finishing write tid 2 to smithi04526579-37 2024-04-26T00:20:28.512 INFO:tasks.rados.rados.0.smithi045.stdout:37: finishing write tid 3 to smithi04526579-37 2024-04-26T00:20:28.512 INFO:tasks.rados.rados.0.smithi045.stdout:37: finishing write tid 4 to smithi04526579-37 2024-04-26T00:20:28.512 INFO:tasks.rados.rados.0.smithi045.stdout:37: oid 37 updating version 0 to 19 2024-04-26T00:20:28.512 INFO:tasks.rados.rados.0.smithi045.stdout:37: oid 37 updating version 19 to 21 2024-04-26T00:20:28.512 INFO:tasks.rados.rados.0.smithi045.stdout:37: oid 37 version 21 is already newer than 20 2024-04-26T00:20:28.512 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 21 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-04-26T00:20:28.512 INFO:tasks.rados.rados.0.smithi045.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2024-04-26T00:20:28.512 INFO:tasks.rados.rados.0.smithi045.stdout:38: finishing write tid 1 to smithi04526579-38 2024-04-26T00:20:28.512 INFO:tasks.rados.rados.0.smithi045.stdout:38: finishing write tid 2 to smithi04526579-38 2024-04-26T00:20:28.512 INFO:tasks.rados.rados.0.smithi045.stdout:35: done (15 left) 2024-04-26T00:20:28.512 INFO:tasks.rados.rados.0.smithi045.stdout:36: done (14 left) 2024-04-26T00:20:28.512 INFO:tasks.rados.rados.0.smithi045.stdout:37: done (13 left) 2024-04-26T00:20:28.513 INFO:tasks.rados.rados.0.smithi045.stdout:51: delete oid 21 current snap is 0 2024-04-26T00:20:28.513 INFO:tasks.rados.rados.0.smithi045.stdout:38: finishing write tid 3 to smithi04526579-38 2024-04-26T00:20:28.513 INFO:tasks.rados.rados.0.smithi045.stdout:38: finishing write tid 4 to smithi04526579-38 2024-04-26T00:20:28.513 INFO:tasks.rados.rados.0.smithi045.stdout:38: oid 38 updating version 0 to 4 2024-04-26T00:20:28.513 INFO:tasks.rados.rados.0.smithi045.stdout:38: oid 38 updating version 4 to 6 2024-04-26T00:20:28.513 INFO:tasks.rados.rados.0.smithi045.stdout:38: oid 38 version 6 is already newer than 5 2024-04-26T00:20:28.513 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 6 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-04-26T00:20:28.513 INFO:tasks.rados.rados.0.smithi045.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2024-04-26T00:20:28.513 INFO:tasks.rados.rados.0.smithi045.stdout:39: finishing write tid 1 to smithi04526579-39 2024-04-26T00:20:28.513 INFO:tasks.rados.rados.0.smithi045.stdout:39: finishing write tid 2 to smithi04526579-39 2024-04-26T00:20:28.513 INFO:tasks.rados.rados.0.smithi045.stdout:39: finishing write tid 3 to smithi04526579-39 2024-04-26T00:20:28.513 INFO:tasks.rados.rados.0.smithi045.stdout:39: finishing write tid 4 to smithi04526579-39 2024-04-26T00:20:28.513 INFO:tasks.rados.rados.0.smithi045.stdout:39: oid 39 updating version 0 to 9 2024-04-26T00:20:28.514 INFO:tasks.rados.rados.0.smithi045.stdout:39: oid 39 version 9 is already newer than 8 2024-04-26T00:20:28.514 INFO:tasks.rados.rados.0.smithi045.stdout:39: oid 39 version 9 is already newer than 7 2024-04-26T00:20:28.514 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 9 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-04-26T00:20:28.514 INFO:tasks.rados.rados.0.smithi045.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2024-04-26T00:20:28.514 INFO:tasks.rados.rados.0.smithi045.stdout:40: finishing write tid 1 to smithi04526579-40 2024-04-26T00:20:28.514 INFO:tasks.rados.rados.0.smithi045.stdout:40: finishing write tid 2 to smithi04526579-40 2024-04-26T00:20:28.514 INFO:tasks.rados.rados.0.smithi045.stdout:40: finishing write tid 3 to smithi04526579-40 2024-04-26T00:20:28.514 INFO:tasks.rados.rados.0.smithi045.stdout:40: finishing write tid 4 to smithi04526579-40 2024-04-26T00:20:28.515 INFO:tasks.rados.rados.0.smithi045.stdout:40: oid 40 updating version 0 to 8 2024-04-26T00:20:28.515 INFO:tasks.rados.rados.0.smithi045.stdout:40: oid 40 version 8 is already newer than 7 2024-04-26T00:20:28.515 INFO:tasks.rados.rados.0.smithi045.stdout:40: oid 40 updating version 8 to 9 2024-04-26T00:20:28.515 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 9 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-26T00:20:28.515 INFO:tasks.rados.rados.0.smithi045.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2024-04-26T00:20:28.515 INFO:tasks.rados.rados.0.smithi045.stdout:41: finishing write tid 1 to smithi04526579-41 2024-04-26T00:20:28.515 INFO:tasks.rados.rados.0.smithi045.stdout:41: finishing write tid 2 to smithi04526579-41 2024-04-26T00:20:28.515 INFO:tasks.rados.rados.0.smithi045.stdout:41: finishing write tid 3 to smithi04526579-41 2024-04-26T00:20:28.516 INFO:tasks.rados.rados.0.smithi045.stdout:41: finishing write tid 4 to smithi04526579-41 2024-04-26T00:20:28.516 INFO:tasks.rados.rados.0.smithi045.stdout:41: oid 41 updating version 0 to 15 2024-04-26T00:20:28.516 INFO:tasks.rados.rados.0.smithi045.stdout:41: oid 41 version 15 is already newer than 14 2024-04-26T00:20:28.516 INFO:tasks.rados.rados.0.smithi045.stdout:41: oid 41 version 15 is already newer than 13 2024-04-26T00:20:28.516 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 15 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-04-26T00:20:28.516 INFO:tasks.rados.rados.0.smithi045.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2024-04-26T00:20:28.516 INFO:tasks.rados.rados.0.smithi045.stdout:42: finishing write tid 1 to smithi04526579-42 2024-04-26T00:20:28.516 INFO:tasks.rados.rados.0.smithi045.stdout:42: finishing write tid 2 to smithi04526579-42 2024-04-26T00:20:28.516 INFO:tasks.rados.rados.0.smithi045.stdout:42: finishing write tid 3 to smithi04526579-42 2024-04-26T00:20:28.516 INFO:tasks.rados.rados.0.smithi045.stdout:42: finishing write tid 4 to smithi04526579-42 2024-04-26T00:20:28.516 INFO:tasks.rados.rados.0.smithi045.stdout:42: oid 42 updating version 0 to 13 2024-04-26T00:20:28.516 INFO:tasks.rados.rados.0.smithi045.stdout:42: oid 42 updating version 13 to 14 2024-04-26T00:20:28.516 INFO:tasks.rados.rados.0.smithi045.stdout:42: oid 42 updating version 14 to 15 2024-04-26T00:20:28.516 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 15 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-26T00:20:28.516 INFO:tasks.rados.rados.0.smithi045.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2024-04-26T00:20:28.517 INFO:tasks.rados.rados.0.smithi045.stdout:43: finishing write tid 1 to smithi04526579-43 2024-04-26T00:20:28.517 INFO:tasks.rados.rados.0.smithi045.stdout:43: finishing write tid 2 to smithi04526579-43 2024-04-26T00:20:28.517 INFO:tasks.rados.rados.0.smithi045.stdout:43: finishing write tid 3 to smithi04526579-43 2024-04-26T00:20:28.517 INFO:tasks.rados.rados.0.smithi045.stdout:43: finishing write tid 4 to smithi04526579-43 2024-04-26T00:20:28.517 INFO:tasks.rados.rados.0.smithi045.stdout:43: oid 43 updating version 0 to 8 2024-04-26T00:20:28.517 INFO:tasks.rados.rados.0.smithi045.stdout:43: oid 43 version 8 is already newer than 7 2024-04-26T00:20:28.517 INFO:tasks.rados.rados.0.smithi045.stdout:43: oid 43 updating version 8 to 9 2024-04-26T00:20:28.517 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 9 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-04-26T00:20:28.517 INFO:tasks.rados.rados.0.smithi045.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2024-04-26T00:20:28.517 INFO:tasks.rados.rados.0.smithi045.stdout:44: finishing write tid 1 to smithi04526579-44 2024-04-26T00:20:28.517 INFO:tasks.rados.rados.0.smithi045.stdout:44: finishing write tid 2 to smithi04526579-44 2024-04-26T00:20:28.517 INFO:tasks.rados.rados.0.smithi045.stdout:44: finishing write tid 3 to smithi04526579-44 2024-04-26T00:20:28.517 INFO:tasks.rados.rados.0.smithi045.stdout:44: finishing write tid 4 to smithi04526579-44 2024-04-26T00:20:28.517 INFO:tasks.rados.rados.0.smithi045.stdout:44: oid 44 updating version 0 to 11 2024-04-26T00:20:28.518 INFO:tasks.rados.rados.0.smithi045.stdout:44: oid 44 updating version 11 to 12 2024-04-26T00:20:28.518 INFO:tasks.rados.rados.0.smithi045.stdout:44: oid 44 version 12 is already newer than 10 2024-04-26T00:20:28.518 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 12 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-04-26T00:20:28.518 INFO:tasks.rados.rados.0.smithi045.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2024-04-26T00:20:28.518 INFO:tasks.rados.rados.0.smithi045.stdout:45: finishing write tid 1 to smithi04526579-45 2024-04-26T00:20:28.518 INFO:tasks.rados.rados.0.smithi045.stdout:45: finishing write tid 2 to smithi04526579-45 2024-04-26T00:20:28.518 INFO:tasks.rados.rados.0.smithi045.stdout:45: finishing write tid 3 to smithi04526579-45 2024-04-26T00:20:28.518 INFO:tasks.rados.rados.0.smithi045.stdout:45: finishing write tid 4 to smithi04526579-45 2024-04-26T00:20:28.518 INFO:tasks.rados.rados.0.smithi045.stdout:45: oid 45 updating version 0 to 4 2024-04-26T00:20:28.518 INFO:tasks.rados.rados.0.smithi045.stdout:45: oid 45 updating version 4 to 5 2024-04-26T00:20:28.518 INFO:tasks.rados.rados.0.smithi045.stdout:45: oid 45 updating version 5 to 6 2024-04-26T00:20:28.518 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 6 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-04-26T00:20:28.518 INFO:tasks.rados.rados.0.smithi045.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2024-04-26T00:20:28.518 INFO:tasks.rados.rados.0.smithi045.stdout:46: finishing write tid 1 to smithi04526579-46 2024-04-26T00:20:28.519 INFO:tasks.rados.rados.0.smithi045.stdout:46: finishing write tid 2 to smithi04526579-46 2024-04-26T00:20:28.519 INFO:tasks.rados.rados.0.smithi045.stdout:46: finishing write tid 3 to smithi04526579-46 2024-04-26T00:20:28.519 INFO:tasks.rados.rados.0.smithi045.stdout:46: finishing write tid 4 to smithi04526579-46 2024-04-26T00:20:28.519 INFO:tasks.rados.rados.0.smithi045.stdout:46: oid 46 updating version 0 to 9 2024-04-26T00:20:28.519 INFO:tasks.rados.rados.0.smithi045.stdout:46: oid 46 version 9 is already newer than 8 2024-04-26T00:20:28.519 INFO:tasks.rados.rados.0.smithi045.stdout:46: oid 46 version 9 is already newer than 7 2024-04-26T00:20:28.519 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 9 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-26T00:20:28.519 INFO:tasks.rados.rados.0.smithi045.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2024-04-26T00:20:28.519 INFO:tasks.rados.rados.0.smithi045.stdout:47: finishing write tid 1 to smithi04526579-47 2024-04-26T00:20:28.519 INFO:tasks.rados.rados.0.smithi045.stdout:47: finishing write tid 2 to smithi04526579-47 2024-04-26T00:20:28.519 INFO:tasks.rados.rados.0.smithi045.stdout:47: finishing write tid 3 to smithi04526579-47 2024-04-26T00:20:28.519 INFO:tasks.rados.rados.0.smithi045.stdout:47: finishing write tid 4 to smithi04526579-47 2024-04-26T00:20:28.519 INFO:tasks.rados.rados.0.smithi045.stdout:47: oid 47 updating version 0 to 7 2024-04-26T00:20:28.519 INFO:tasks.rados.rados.0.smithi045.stdout:47: oid 47 updating version 7 to 8 2024-04-26T00:20:28.520 INFO:tasks.rados.rados.0.smithi045.stdout:47: oid 47 updating version 8 to 9 2024-04-26T00:20:28.520 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 9 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-04-26T00:20:28.520 INFO:tasks.rados.rados.0.smithi045.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2024-04-26T00:20:28.520 INFO:tasks.rados.rados.0.smithi045.stdout:48: finishing write tid 1 to smithi04526579-48 2024-04-26T00:20:28.520 INFO:tasks.rados.rados.0.smithi045.stdout:48: finishing write tid 2 to smithi04526579-48 2024-04-26T00:20:28.521 INFO:tasks.rados.rados.0.smithi045.stdout:48: finishing write tid 3 to smithi04526579-48 2024-04-26T00:20:28.521 INFO:tasks.rados.rados.0.smithi045.stdout:48: finishing write tid 4 to smithi04526579-48 2024-04-26T00:20:28.521 INFO:tasks.rados.rados.0.smithi045.stdout:48: oid 48 updating version 0 to 12 2024-04-26T00:20:28.521 INFO:tasks.rados.rados.0.smithi045.stdout:48: oid 48 version 12 is already newer than 11 2024-04-26T00:20:28.521 INFO:tasks.rados.rados.0.smithi045.stdout:48: oid 48 version 12 is already newer than 10 2024-04-26T00:20:28.521 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 12 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-04-26T00:20:28.521 INFO:tasks.rados.rados.0.smithi045.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2024-04-26T00:20:28.521 INFO:tasks.rados.rados.0.smithi045.stdout:49: finishing write tid 1 to smithi04526579-49 2024-04-26T00:20:28.521 INFO:tasks.rados.rados.0.smithi045.stdout:49: finishing write tid 2 to smithi04526579-49 2024-04-26T00:20:28.521 INFO:tasks.rados.rados.0.smithi045.stdout:49: finishing write tid 3 to smithi04526579-49 2024-04-26T00:20:28.521 INFO:tasks.rados.rados.0.smithi045.stdout:49: finishing write tid 4 to smithi04526579-49 2024-04-26T00:20:28.522 INFO:tasks.rados.rados.0.smithi045.stdout:49: oid 49 updating version 0 to 15 2024-04-26T00:20:28.522 INFO:tasks.rados.rados.0.smithi045.stdout:49: oid 49 version 15 is already newer than 14 2024-04-26T00:20:28.522 INFO:tasks.rados.rados.0.smithi045.stdout:49: oid 49 version 15 is already newer than 13 2024-04-26T00:20:28.522 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 15 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-26T00:20:28.522 INFO:tasks.rados.rados.0.smithi045.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2024-04-26T00:20:28.522 INFO:tasks.rados.rados.0.smithi045.stdout:50: finishing write tid 1 to smithi04526579-50 2024-04-26T00:20:28.523 INFO:tasks.rados.rados.0.smithi045.stdout:38: done (13 left) 2024-04-26T00:20:28.523 INFO:tasks.rados.rados.0.smithi045.stdout:39: done (12 left) 2024-04-26T00:20:28.523 INFO:tasks.rados.rados.0.smithi045.stdout:40: done (11 left) 2024-04-26T00:20:28.523 INFO:tasks.rados.rados.0.smithi045.stdout:41: done (10 left) 2024-04-26T00:20:28.523 INFO:tasks.rados.rados.0.smithi045.stdout:42: done (9 left) 2024-04-26T00:20:28.523 INFO:tasks.rados.rados.0.smithi045.stdout:43: done (8 left) 2024-04-26T00:20:28.523 INFO:tasks.rados.rados.0.smithi045.stdout:44: done (7 left) 2024-04-26T00:20:28.523 INFO:tasks.rados.rados.0.smithi045.stdout:45: done (6 left) 2024-04-26T00:20:28.523 INFO:tasks.rados.rados.0.smithi045.stdout:46: done (5 left) 2024-04-26T00:20:28.524 INFO:tasks.rados.rados.0.smithi045.stdout:47: done (4 left) 2024-04-26T00:20:28.524 INFO:tasks.rados.rados.0.smithi045.stdout:48: done (3 left) 2024-04-26T00:20:28.524 INFO:tasks.rados.rados.0.smithi045.stdout:49: done (2 left) 2024-04-26T00:20:28.524 INFO:tasks.rados.rados.0.smithi045.stdout:51: done (1 left) 2024-04-26T00:20:28.524 INFO:tasks.rados.rados.0.smithi045.stdout:52: read oid 6 snap -1 2024-04-26T00:20:28.524 INFO:tasks.rados.rados.0.smithi045.stdout:52: expect (ObjNum 5 snap 0 seq_num 5) 2024-04-26T00:20:28.524 INFO:tasks.rados.rados.0.smithi045.stdout:53: rollback oid 27 current snap is 0 2024-04-26T00:20:28.524 INFO:tasks.rados.rados.0.smithi045.stdout:53: done (2 left) 2024-04-26T00:20:28.524 INFO:tasks.rados.rados.0.smithi045.stdout:54: write oid 23 current snap is 0 2024-04-26T00:20:28.524 INFO:tasks.rados.rados.0.smithi045.stdout:54: seq_num 51 ranges {457157=535817,1768111=402677} 2024-04-26T00:20:28.524 INFO:tasks.rados.rados.0.smithi045.stdout:54: writing smithi04526579-23 from 457157 to 992974 tid 1 2024-04-26T00:20:28.526 INFO:tasks.rados.rados.0.smithi045.stdout:54: writing smithi04526579-23 from 1768111 to 2170788 tid 2 2024-04-26T00:20:28.526 INFO:tasks.rados.rados.0.smithi045.stdout:55: copy_from oid 31 from oid 49 current snap is 0 2024-04-26T00:20:28.526 INFO:tasks.rados.rados.0.smithi045.stdout:50: finishing write tid 2 to smithi04526579-50 2024-04-26T00:20:28.526 INFO:tasks.rados.rados.0.smithi045.stdout:56: rollback oid 12 current snap is 0 2024-04-26T00:20:28.526 INFO:tasks.rados.rados.0.smithi045.stdout:50: finishing write tid 3 to smithi04526579-50 2024-04-26T00:20:28.527 INFO:tasks.rados.rados.0.smithi045.stdout:50: finishing write tid 4 to smithi04526579-50 2024-04-26T00:20:28.527 INFO:tasks.rados.rados.0.smithi045.stdout:50: oid 50 updating version 0 to 11 2024-04-26T00:20:28.527 INFO:tasks.rados.rados.0.smithi045.stdout:50: oid 50 updating version 11 to 12 2024-04-26T00:20:28.527 INFO:tasks.rados.rados.0.smithi045.stdout:50: oid 50 version 12 is already newer than 10 2024-04-26T00:20:28.527 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 12 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-04-26T00:20:28.527 INFO:tasks.rados.rados.0.smithi045.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2024-04-26T00:20:28.527 INFO:tasks.rados.rados.0.smithi045.stdout:52: expect (ObjNum 5 snap 0 seq_num 5) 2024-04-26T00:20:28.651 INFO:tasks.rados.rados.0.smithi045.stdout:54: finishing write tid 1 to smithi04526579-23 2024-04-26T00:20:28.651 INFO:tasks.rados.rados.0.smithi045.stdout:54: finishing write tid 2 to smithi04526579-23 2024-04-26T00:20:28.651 INFO:tasks.rados.rados.0.smithi045.stdout:54: finishing write tid 3 to smithi04526579-23 2024-04-26T00:20:28.651 INFO:tasks.rados.rados.0.smithi045.stdout:54: finishing write tid 4 to smithi04526579-23 2024-04-26T00:20:28.651 INFO:tasks.rados.rados.0.smithi045.stdout:54: oid 23 updating version 0 to 11 2024-04-26T00:20:28.651 INFO:tasks.rados.rados.0.smithi045.stdout:54: oid 23 version 11 is already newer than 10 2024-04-26T00:20:28.651 INFO:tasks.rados.rados.0.smithi045.stdout:54: oid 23 updating version 11 to 12 2024-04-26T00:20:28.651 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 12 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-04-26T00:20:28.651 INFO:tasks.rados.rados.0.smithi045.stdout:54: left oid 23 (ObjNum 51 snap 0 seq_num 51) 2024-04-26T00:20:28.652 INFO:tasks.rados.rados.0.smithi045.stdout:55: finishing copy_from racing read to smithi04526579-31 2024-04-26T00:20:28.652 INFO:tasks.rados.rados.0.smithi045.stdout:55: finishing copy_from to smithi04526579-31 2024-04-26T00:20:28.652 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 15 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-26T00:20:28.652 INFO:tasks.rados.rados.0.smithi045.stdout:50: done (4 left) 2024-04-26T00:20:28.652 INFO:tasks.rados.rados.0.smithi045.stdout:52: done (3 left) 2024-04-26T00:20:28.652 INFO:tasks.rados.rados.0.smithi045.stdout:54: done (2 left) 2024-04-26T00:20:28.652 INFO:tasks.rados.rados.0.smithi045.stdout:55: done (1 left) 2024-04-26T00:20:28.652 INFO:tasks.rados.rados.0.smithi045.stdout:56: done (0 left) 2024-04-26T00:20:28.652 INFO:tasks.rados.rados.0.smithi045.stdout:57: snap_create 2024-04-26T00:20:29.341 INFO:tasks.rados.rados.0.smithi045.stdout:57: done (0 left) 2024-04-26T00:20:29.341 INFO:tasks.rados.rados.0.smithi045.stdout:58: read oid 21 snap -1 2024-04-26T00:20:29.341 INFO:tasks.rados.rados.0.smithi045.stdout:58: expect deleted 2024-04-26T00:20:29.341 INFO:tasks.rados.rados.0.smithi045.stdout:59: copy_from oid 9 from oid 44 current snap is 1 2024-04-26T00:20:29.342 INFO:tasks.rados.rados.0.smithi045.stdout:60: rollback oid 20 current snap is 1 2024-04-26T00:20:29.342 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 20 to 0 2024-04-26T00:20:29.342 INFO:tasks.rados.rados.0.smithi045.stdout:61: read oid 39 snap 0 2024-04-26T00:20:29.342 INFO:tasks.rados.rados.0.smithi045.stdout:61: expect (ObjNum 38 snap 0 seq_num 38) 2024-04-26T00:20:29.342 INFO:tasks.rados.rados.0.smithi045.stdout:62: write (excl) oid 19 current snap is 1 2024-04-26T00:20:29.342 INFO:tasks.rados.rados.0.smithi045.stdout:62: seq_num 52 ranges {473964=611066,1826593=546335,2778819=95145} 2024-04-26T00:20:29.348 INFO:tasks.rados.rados.0.smithi045.stdout:62: writing smithi04526579-19 from 473964 to 1085030 tid 1 2024-04-26T00:20:29.356 INFO:tasks.rados.rados.0.smithi045.stdout:62: writing smithi04526579-19 from 1826593 to 2372928 tid 2 2024-04-26T00:20:29.359 INFO:tasks.rados.rados.0.smithi045.stdout:62: writing smithi04526579-19 from 2778819 to 2873964 tid 3 2024-04-26T00:20:29.359 INFO:tasks.rados.rados.0.smithi045.stdout:63: read oid 16 snap -1 2024-04-26T00:20:29.359 INFO:tasks.rados.rados.0.smithi045.stdout:63: expect (ObjNum 15 snap 0 seq_num 15) 2024-04-26T00:20:29.359 INFO:tasks.rados.rados.0.smithi045.stdout:64: snap_create 2024-04-26T00:20:29.367 INFO:tasks.rados.rados.0.smithi045.stdout:62: finishing write tid 1 to smithi04526579-19 2024-04-26T00:20:29.369 INFO:tasks.rados.rados.0.smithi045.stdout:60: finishing rollback tid 0 to smithi04526579-20 2024-04-26T00:20:29.371 INFO:tasks.rados.rados.0.smithi045.stdout:62: finishing write tid 2 to smithi04526579-19 2024-04-26T00:20:29.371 INFO:tasks.rados.rados.0.smithi045.stdout:62: finishing write tid 3 to smithi04526579-19 2024-04-26T00:20:29.371 INFO:tasks.rados.rados.0.smithi045.stdout:62: finishing write tid 4 to smithi04526579-19 2024-04-26T00:20:29.377 INFO:tasks.rados.rados.0.smithi045.stdout:63: expect (ObjNum 15 snap 0 seq_num 15) 2024-04-26T00:20:29.510 INFO:tasks.rados.rados.0.smithi045.stdout:62: finishing write tid 5 to smithi04526579-19 2024-04-26T00:20:29.510 INFO:tasks.rados.rados.0.smithi045.stdout:62: oid 19 updating version 0 to 18 2024-04-26T00:20:29.510 INFO:tasks.rados.rados.0.smithi045.stdout:62: oid 19 updating version 18 to 19 2024-04-26T00:20:29.510 INFO:tasks.rados.rados.0.smithi045.stdout:62: oid 19 updating version 19 to 20 2024-04-26T00:20:29.511 INFO:tasks.rados.rados.0.smithi045.stdout:62: oid 19 version 20 is already newer than 17 2024-04-26T00:20:29.511 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 20 (ObjNum 52 snap 1 seq_num 52) dirty exists 2024-04-26T00:20:29.511 INFO:tasks.rados.rados.0.smithi045.stdout:62: left oid 19 (ObjNum 52 snap 1 seq_num 52) 2024-04-26T00:20:29.511 INFO:tasks.rados.rados.0.smithi045.stdout:59: finishing copy_from racing read to smithi04526579-9 2024-04-26T00:20:29.511 INFO:tasks.rados.rados.0.smithi045.stdout:61: expect (ObjNum 38 snap 0 seq_num 38) 2024-04-26T00:20:29.596 INFO:tasks.rados.rados.0.smithi045.stdout:60: finishing rollback tid 1 to smithi04526579-20 2024-04-26T00:20:29.596 INFO:tasks.rados.rados.0.smithi045.stdout:60: finishing rollback tid 2 to smithi04526579-20 2024-04-26T00:20:29.596 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 16 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-04-26T00:20:29.596 INFO:tasks.rados.rados.0.smithi045.stdout:59: finishing copy_from to smithi04526579-9 2024-04-26T00:20:29.596 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 12 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-04-26T00:20:30.349 INFO:tasks.rados.rados.0.smithi045.stdout:58: done (6 left) 2024-04-26T00:20:30.349 INFO:tasks.rados.rados.0.smithi045.stdout:59: done (5 left) 2024-04-26T00:20:30.350 INFO:tasks.rados.rados.0.smithi045.stdout:60: done (4 left) 2024-04-26T00:20:30.350 INFO:tasks.rados.rados.0.smithi045.stdout:61: done (3 left) 2024-04-26T00:20:30.350 INFO:tasks.rados.rados.0.smithi045.stdout:62: done (2 left) 2024-04-26T00:20:30.350 INFO:tasks.rados.rados.0.smithi045.stdout:63: done (1 left) 2024-04-26T00:20:30.350 INFO:tasks.rados.rados.0.smithi045.stdout:64: done (0 left) 2024-04-26T00:20:30.350 INFO:tasks.rados.rados.0.smithi045.stdout:65: read oid 6 snap -1 2024-04-26T00:20:30.350 INFO:tasks.rados.rados.0.smithi045.stdout:65: expect (ObjNum 5 snap 0 seq_num 5) 2024-04-26T00:20:30.350 INFO:tasks.rados.rados.0.smithi045.stdout:66: snap_create 2024-04-26T00:20:30.363 INFO:tasks.rados.rados.0.smithi045.stdout:65: expect (ObjNum 5 snap 0 seq_num 5) 2024-04-26T00:20:31.357 INFO:tasks.rados.rados.0.smithi045.stdout:65: done (1 left) 2024-04-26T00:20:31.357 INFO:tasks.rados.rados.0.smithi045.stdout:66: done (0 left) 2024-04-26T00:20:31.358 INFO:tasks.rados.rados.0.smithi045.stdout:67: read oid 39 snap -1 2024-04-26T00:20:31.358 INFO:tasks.rados.rados.0.smithi045.stdout:67: expect (ObjNum 38 snap 0 seq_num 38) 2024-04-26T00:20:31.358 INFO:tasks.rados.rados.0.smithi045.stdout:68: read oid 6 snap -1 2024-04-26T00:20:31.358 INFO:tasks.rados.rados.0.smithi045.stdout:68: expect (ObjNum 5 snap 0 seq_num 5) 2024-04-26T00:20:31.358 INFO:tasks.rados.rados.0.smithi045.stdout:69: write (excl) oid 22 current snap is 3 2024-04-26T00:20:31.358 INFO:tasks.rados.rados.0.smithi045.stdout:69: seq_num 53 ranges {490771=686315,1885075=406346} 2024-04-26T00:20:31.373 INFO:tasks.rados.rados.0.smithi045.stdout:69: writing smithi04526579-22 from 490771 to 1177086 tid 1 2024-04-26T00:20:31.387 INFO:tasks.rados.rados.0.smithi045.stdout:69: writing smithi04526579-22 from 1885075 to 2291421 tid 2 2024-04-26T00:20:31.387 INFO:tasks.rados.rados.0.smithi045.stdout:70: snap_create 2024-04-26T00:20:31.387 INFO:tasks.rados.rados.0.smithi045.stdout:68: expect (ObjNum 5 snap 0 seq_num 5) 2024-04-26T00:20:31.460 INFO:tasks.rados.rados.0.smithi045.stdout:67: expect (ObjNum 38 snap 0 seq_num 38) 2024-04-26T00:20:31.547 INFO:tasks.rados.rados.0.smithi045.stdout:69: finishing write tid 1 to smithi04526579-22 2024-04-26T00:20:31.547 INFO:tasks.rados.rados.0.smithi045.stdout:69: finishing write tid 2 to smithi04526579-22 2024-04-26T00:20:31.547 INFO:tasks.rados.rados.0.smithi045.stdout:69: finishing write tid 3 to smithi04526579-22 2024-04-26T00:20:31.547 INFO:tasks.rados.rados.0.smithi045.stdout:69: finishing write tid 4 to smithi04526579-22 2024-04-26T00:20:31.547 INFO:tasks.rados.rados.0.smithi045.stdout:69: oid 22 updating version 0 to 13 2024-04-26T00:20:31.547 INFO:tasks.rados.rados.0.smithi045.stdout:69: oid 22 version 13 is already newer than 12 2024-04-26T00:20:31.547 INFO:tasks.rados.rados.0.smithi045.stdout:69: oid 22 version 13 is already newer than 11 2024-04-26T00:20:31.548 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 13 (ObjNum 53 snap 3 seq_num 53) dirty exists 2024-04-26T00:20:31.548 INFO:tasks.rados.rados.0.smithi045.stdout:69: left oid 22 (ObjNum 53 snap 3 seq_num 53) 2024-04-26T00:20:32.360 INFO:tasks.rados.rados.0.smithi045.stdout:67: done (3 left) 2024-04-26T00:20:32.360 INFO:tasks.rados.rados.0.smithi045.stdout:68: done (2 left) 2024-04-26T00:20:32.360 INFO:tasks.rados.rados.0.smithi045.stdout:69: done (1 left) 2024-04-26T00:20:32.360 INFO:tasks.rados.rados.0.smithi045.stdout:70: done (0 left) 2024-04-26T00:20:32.360 INFO:tasks.rados.rados.0.smithi045.stdout:71: write oid 25 current snap is 4 2024-04-26T00:20:32.360 INFO:tasks.rados.rados.0.smithi045.stdout:71: seq_num 54 ranges {507578=477917,1543557=550004,2833665=73913} 2024-04-26T00:20:32.372 INFO:tasks.rados.rados.0.smithi045.stdout:71: writing smithi04526579-25 from 507578 to 985495 tid 1 2024-04-26T00:20:32.387 INFO:tasks.rados.rados.0.smithi045.stdout:71: writing smithi04526579-25 from 1543557 to 2093561 tid 2 2024-04-26T00:20:32.398 INFO:tasks.rados.rados.0.smithi045.stdout:71: writing smithi04526579-25 from 2833665 to 2907578 tid 3 2024-04-26T00:20:32.398 INFO:tasks.rados.rados.0.smithi045.stdout:72: snap_remove snap 3 2024-04-26T00:20:33.382 INFO:tasks.rados.rados.0.smithi045.stdout:72: done (1 left) 2024-04-26T00:20:33.382 INFO:tasks.rados.rados.0.smithi045.stdout:73: write (excl) oid 50 current snap is 4 2024-04-26T00:20:33.382 INFO:tasks.rados.rados.0.smithi045.stdout:73: seq_num 55 ranges {524385=553166,1602039=410015,2577441=346944} 2024-04-26T00:20:33.396 INFO:tasks.rados.rados.0.smithi045.stdout:73: writing smithi04526579-50 from 524385 to 1077551 tid 1 2024-04-26T00:20:33.409 INFO:tasks.rados.rados.0.smithi045.stdout:73: writing smithi04526579-50 from 1602039 to 2012054 tid 2 2024-04-26T00:20:33.422 INFO:tasks.rados.rados.0.smithi045.stdout:73: writing smithi04526579-50 from 2577441 to 2924385 tid 3 2024-04-26T00:20:33.422 INFO:tasks.rados.rados.0.smithi045.stdout:74: rollback oid 20 current snap is 4 2024-04-26T00:20:33.422 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 20 to 0 2024-04-26T00:20:33.422 INFO:tasks.rados.rados.0.smithi045.stdout:71: finishing write tid 1 to smithi04526579-25 2024-04-26T00:20:33.422 INFO:tasks.rados.rados.0.smithi045.stdout:71: finishing write tid 2 to smithi04526579-25 2024-04-26T00:20:33.422 INFO:tasks.rados.rados.0.smithi045.stdout:71: finishing write tid 3 to smithi04526579-25 2024-04-26T00:20:33.422 INFO:tasks.rados.rados.0.smithi045.stdout:71: finishing write tid 4 to smithi04526579-25 2024-04-26T00:20:33.423 INFO:tasks.rados.rados.0.smithi045.stdout:71: finishing write tid 5 to smithi04526579-25 2024-04-26T00:20:33.423 INFO:tasks.rados.rados.0.smithi045.stdout:71: oid 25 updating version 0 to 23 2024-04-26T00:20:33.423 INFO:tasks.rados.rados.0.smithi045.stdout:71: oid 25 updating version 23 to 26 2024-04-26T00:20:33.423 INFO:tasks.rados.rados.0.smithi045.stdout:71: oid 25 version 26 is already newer than 25 2024-04-26T00:20:33.423 INFO:tasks.rados.rados.0.smithi045.stdout:71: oid 25 version 26 is already newer than 24 2024-04-26T00:20:33.423 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 26 (ObjNum 54 snap 4 seq_num 54) dirty exists 2024-04-26T00:20:33.423 INFO:tasks.rados.rados.0.smithi045.stdout:71: left oid 25 (ObjNum 54 snap 4 seq_num 54) 2024-04-26T00:20:33.423 INFO:tasks.rados.rados.0.smithi045.stdout:73: finishing write tid 1 to smithi04526579-50 2024-04-26T00:20:33.423 INFO:tasks.rados.rados.0.smithi045.stdout:73: finishing write tid 2 to smithi04526579-50 2024-04-26T00:20:33.423 INFO:tasks.rados.rados.0.smithi045.stdout:71: done (2 left) 2024-04-26T00:20:33.423 INFO:tasks.rados.rados.0.smithi045.stdout:75: read oid 43 snap -1 2024-04-26T00:20:33.423 INFO:tasks.rados.rados.0.smithi045.stdout:75: expect (ObjNum 42 snap 0 seq_num 42) 2024-04-26T00:20:33.423 INFO:tasks.rados.rados.0.smithi045.stdout:76: read oid 4 snap -1 2024-04-26T00:20:33.423 INFO:tasks.rados.rados.0.smithi045.stdout:76: expect (ObjNum 3 snap 0 seq_num 3) 2024-04-26T00:20:33.424 INFO:tasks.rados.rados.0.smithi045.stdout:77: rollback oid 10 current snap is 4 2024-04-26T00:20:33.424 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 10 to 2 2024-04-26T00:20:33.424 INFO:tasks.rados.rados.0.smithi045.stdout:78: write oid 18 current snap is 4 2024-04-26T00:20:33.424 INFO:tasks.rados.rados.0.smithi045.stdout:78: seq_num 56 ranges {541192=628415,1660521=553673,2888511=52681} 2024-04-26T00:20:33.439 INFO:tasks.rados.rados.0.smithi045.stdout:78: writing smithi04526579-18 from 541192 to 1169607 tid 1 2024-04-26T00:20:33.450 INFO:tasks.rados.rados.0.smithi045.stdout:78: writing smithi04526579-18 from 1660521 to 2214194 tid 2 2024-04-26T00:20:33.457 INFO:tasks.rados.rados.0.smithi045.stdout:78: writing smithi04526579-18 from 2888511 to 2941192 tid 3 2024-04-26T00:20:33.457 INFO:tasks.rados.rados.0.smithi045.stdout:79: read oid 38 snap -1 2024-04-26T00:20:33.457 INFO:tasks.rados.rados.0.smithi045.stdout:79: expect (ObjNum 37 snap 0 seq_num 37) 2024-04-26T00:20:33.457 INFO:tasks.rados.rados.0.smithi045.stdout:80: snap_remove snap 1 2024-04-26T00:20:34.387 INFO:tasks.rados.rados.0.smithi045.stdout:80: done (7 left) 2024-04-26T00:20:34.387 INFO:tasks.rados.rados.0.smithi045.stdout:81: snap_create 2024-04-26T00:20:34.387 INFO:tasks.rados.rados.0.smithi045.stdout:75: expect (ObjNum 42 snap 0 seq_num 42) 2024-04-26T00:20:34.533 INFO:tasks.rados.rados.0.smithi045.stdout:73: finishing write tid 3 to smithi04526579-50 2024-04-26T00:20:34.533 INFO:tasks.rados.rados.0.smithi045.stdout:73: finishing write tid 4 to smithi04526579-50 2024-04-26T00:20:34.533 INFO:tasks.rados.rados.0.smithi045.stdout:73: finishing write tid 5 to smithi04526579-50 2024-04-26T00:20:34.533 INFO:tasks.rados.rados.0.smithi045.stdout:73: oid 50 updating version 0 to 15 2024-04-26T00:20:34.533 INFO:tasks.rados.rados.0.smithi045.stdout:73: oid 50 version 15 is already newer than 14 2024-04-26T00:20:34.533 INFO:tasks.rados.rados.0.smithi045.stdout:73: oid 50 updating version 15 to 17 2024-04-26T00:20:34.533 INFO:tasks.rados.rados.0.smithi045.stdout:73: oid 50 version 17 is already newer than 16 2024-04-26T00:20:34.533 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 17 (ObjNum 55 snap 4 seq_num 55) dirty exists 2024-04-26T00:20:34.533 INFO:tasks.rados.rados.0.smithi045.stdout:73: left oid 50 (ObjNum 55 snap 4 seq_num 55) 2024-04-26T00:20:34.533 INFO:tasks.rados.rados.0.smithi045.stdout:74: finishing rollback tid 0 to smithi04526579-20 2024-04-26T00:20:34.533 INFO:tasks.rados.rados.0.smithi045.stdout:74: finishing rollback tid 1 to smithi04526579-20 2024-04-26T00:20:34.533 INFO:tasks.rados.rados.0.smithi045.stdout:74: finishing rollback tid 2 to smithi04526579-20 2024-04-26T00:20:34.534 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 20 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-04-26T00:20:34.534 INFO:tasks.rados.rados.0.smithi045.stdout:77: finishing rollback tid 0 to smithi04526579-10 2024-04-26T00:20:34.534 INFO:tasks.rados.rados.0.smithi045.stdout:77: finishing rollback tid 1 to smithi04526579-10 2024-04-26T00:20:34.534 INFO:tasks.rados.rados.0.smithi045.stdout:77: finishing rollback tid 2 to smithi04526579-10 2024-04-26T00:20:34.534 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 21 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-04-26T00:20:34.534 INFO:tasks.rados.rados.0.smithi045.stdout:76: expect (ObjNum 3 snap 0 seq_num 3) 2024-04-26T00:20:34.598 INFO:tasks.rados.rados.0.smithi045.stdout:78: finishing write tid 1 to smithi04526579-18 2024-04-26T00:20:34.598 INFO:tasks.rados.rados.0.smithi045.stdout:78: finishing write tid 2 to smithi04526579-18 2024-04-26T00:20:34.599 INFO:tasks.rados.rados.0.smithi045.stdout:78: finishing write tid 3 to smithi04526579-18 2024-04-26T00:20:34.599 INFO:tasks.rados.rados.0.smithi045.stdout:78: finishing write tid 4 to smithi04526579-18 2024-04-26T00:20:34.599 INFO:tasks.rados.rados.0.smithi045.stdout:78: finishing write tid 5 to smithi04526579-18 2024-04-26T00:20:34.599 INFO:tasks.rados.rados.0.smithi045.stdout:78: oid 18 updating version 0 to 30 2024-04-26T00:20:34.599 INFO:tasks.rados.rados.0.smithi045.stdout:78: oid 18 updating version 30 to 31 2024-04-26T00:20:34.599 INFO:tasks.rados.rados.0.smithi045.stdout:78: oid 18 updating version 31 to 32 2024-04-26T00:20:34.599 INFO:tasks.rados.rados.0.smithi045.stdout:78: oid 18 updating version 32 to 33 2024-04-26T00:20:34.599 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 33 (ObjNum 56 snap 4 seq_num 56) dirty exists 2024-04-26T00:20:34.599 INFO:tasks.rados.rados.0.smithi045.stdout:78: left oid 18 (ObjNum 56 snap 4 seq_num 56) 2024-04-26T00:20:34.599 INFO:tasks.rados.rados.0.smithi045.stdout:79: expect (ObjNum 37 snap 0 seq_num 37) 2024-04-26T00:20:35.390 INFO:tasks.rados.rados.0.smithi045.stdout:73: done (7 left) 2024-04-26T00:20:35.390 INFO:tasks.rados.rados.0.smithi045.stdout:74: done (6 left) 2024-04-26T00:20:35.390 INFO:tasks.rados.rados.0.smithi045.stdout:75: done (5 left) 2024-04-26T00:20:35.390 INFO:tasks.rados.rados.0.smithi045.stdout:76: done (4 left) 2024-04-26T00:20:35.390 INFO:tasks.rados.rados.0.smithi045.stdout:77: done (3 left) 2024-04-26T00:20:35.390 INFO:tasks.rados.rados.0.smithi045.stdout:78: done (2 left) 2024-04-26T00:20:35.390 INFO:tasks.rados.rados.0.smithi045.stdout:79: done (1 left) 2024-04-26T00:20:35.390 INFO:tasks.rados.rados.0.smithi045.stdout:81: done (0 left) 2024-04-26T00:20:35.390 INFO:tasks.rados.rados.0.smithi045.stdout:82: copy_from oid 5 from oid 19 current snap is 5 2024-04-26T00:20:35.390 INFO:tasks.rados.rados.0.smithi045.stdout:83: snap_create 2024-04-26T00:20:35.420 INFO:tasks.rados.rados.0.smithi045.stdout:82: finishing copy_from racing read to smithi04526579-5 2024-04-26T00:20:35.421 INFO:tasks.rados.rados.0.smithi045.stdout:82: finishing copy_from to smithi04526579-5 2024-04-26T00:20:35.421 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 20 (ObjNum 52 snap 1 seq_num 52) dirty exists 2024-04-26T00:20:36.392 INFO:tasks.rados.rados.0.smithi045.stdout:82: done (1 left) 2024-04-26T00:20:36.393 INFO:tasks.rados.rados.0.smithi045.stdout:83: done (0 left) 2024-04-26T00:20:36.393 INFO:tasks.rados.rados.0.smithi045.stdout:84: delete oid 21 current snap is 6 2024-04-26T00:20:36.400 INFO:tasks.rados.rados.0.smithi045.stdout:84: done (0 left) 2024-04-26T00:20:36.400 INFO:tasks.rados.rados.0.smithi045.stdout:85: write oid 50 current snap is 6 2024-04-26T00:20:36.400 INFO:tasks.rados.rados.0.smithi045.stdout:85: seq_num 58 ranges {574806=778913,2061132=557342} 2024-04-26T00:20:36.422 INFO:tasks.rados.rados.0.smithi045.stdout:85: writing smithi04526579-50 from 574806 to 1353719 tid 1 2024-04-26T00:20:36.443 INFO:tasks.rados.rados.0.smithi045.stdout:85: writing smithi04526579-50 from 2061132 to 2618474 tid 2 2024-04-26T00:20:36.443 INFO:tasks.rados.rados.0.smithi045.stdout:86: snap_create 2024-04-26T00:20:36.443 INFO:tasks.rados.rados.0.smithi045.stdout:85: finishing write tid 1 to smithi04526579-50 2024-04-26T00:20:36.449 INFO:tasks.rados.rados.0.smithi045.stdout:85: finishing write tid 2 to smithi04526579-50 2024-04-26T00:20:36.450 INFO:tasks.rados.rados.0.smithi045.stdout:85: finishing write tid 3 to smithi04526579-50 2024-04-26T00:20:36.450 INFO:tasks.rados.rados.0.smithi045.stdout:85: finishing write tid 4 to smithi04526579-50 2024-04-26T00:20:36.451 INFO:tasks.rados.rados.0.smithi045.stdout:85: oid 50 updating version 0 to 29 2024-04-26T00:20:36.451 INFO:tasks.rados.rados.0.smithi045.stdout:85: oid 50 version 29 is already newer than 28 2024-04-26T00:20:36.451 INFO:tasks.rados.rados.0.smithi045.stdout:85: oid 50 version 29 is already newer than 27 2024-04-26T00:20:36.451 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 29 (ObjNum 58 snap 6 seq_num 58) dirty exists 2024-04-26T00:20:36.451 INFO:tasks.rados.rados.0.smithi045.stdout:85: left oid 50 (ObjNum 58 snap 6 seq_num 58) 2024-04-26T00:20:37.416 INFO:tasks.rados.rados.0.smithi045.stdout:85: done (1 left) 2024-04-26T00:20:37.416 INFO:tasks.rados.rados.0.smithi045.stdout:86: done (0 left) 2024-04-26T00:20:37.416 INFO:tasks.rados.rados.0.smithi045.stdout:87: read oid 23 snap -1 2024-04-26T00:20:37.416 INFO:tasks.rados.rados.0.smithi045.stdout:87: expect (ObjNum 51 snap 0 seq_num 51) 2024-04-26T00:20:37.416 INFO:tasks.rados.rados.0.smithi045.stdout:88: delete oid 36 current snap is 7 2024-04-26T00:20:37.423 INFO:tasks.rados.rados.0.smithi045.stdout:88: done (1 left) 2024-04-26T00:20:37.424 INFO:tasks.rados.rados.0.smithi045.stdout:89: read oid 11 snap 2 2024-04-26T00:20:37.424 INFO:tasks.rados.rados.0.smithi045.stdout:89: expect (ObjNum 10 snap 0 seq_num 10) 2024-04-26T00:20:37.424 INFO:tasks.rados.rados.0.smithi045.stdout:90: snap_create 2024-04-26T00:20:37.434 INFO:tasks.rados.rados.0.smithi045.stdout:87: expect (ObjNum 51 snap 0 seq_num 51) 2024-04-26T00:20:37.619 INFO:tasks.rados.rados.0.smithi045.stdout:89: expect (ObjNum 10 snap 0 seq_num 10) 2024-04-26T00:20:37.895 INFO:tasks.rados.rados.0.smithi045.stdout:87: done (2 left) 2024-04-26T00:20:37.895 INFO:tasks.rados.rados.0.smithi045.stdout:89: done (1 left) 2024-04-26T00:20:37.895 INFO:tasks.rados.rados.0.smithi045.stdout:90: done (0 left) 2024-04-26T00:20:37.895 INFO:tasks.rados.rados.0.smithi045.stdout:91: rollback oid 24 current snap is 8 2024-04-26T00:20:37.895 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 24 to 0 2024-04-26T00:20:37.896 INFO:tasks.rados.rados.0.smithi045.stdout:92: delete oid 14 current snap is 8 2024-04-26T00:20:37.912 INFO:tasks.rados.rados.0.smithi045.stdout:91: finishing rollback tid 0 to smithi04526579-24 2024-04-26T00:20:37.912 INFO:tasks.rados.rados.0.smithi045.stdout:91: finishing rollback tid 1 to smithi04526579-24 2024-04-26T00:20:37.912 INFO:tasks.rados.rados.0.smithi045.stdout:92: done (1 left) 2024-04-26T00:20:37.912 INFO:tasks.rados.rados.0.smithi045.stdout:93: write oid 4 current snap is 8 2024-04-26T00:20:37.912 INFO:tasks.rados.rados.0.smithi045.stdout:93: seq_num 61 ranges {625227=721013,1952931=704669} 2024-04-26T00:20:37.934 INFO:tasks.rados.rados.0.smithi045.stdout:93: writing smithi04526579-4 from 625227 to 1346240 tid 1 2024-04-26T00:20:37.956 INFO:tasks.rados.rados.0.smithi045.stdout:93: writing smithi04526579-4 from 1952931 to 2657600 tid 2 2024-04-26T00:20:37.956 INFO:tasks.rados.rados.0.smithi045.stdout:94: snap_create 2024-04-26T00:20:37.956 INFO:tasks.rados.rados.0.smithi045.stdout:91: finishing rollback tid 2 to smithi04526579-24 2024-04-26T00:20:37.956 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 43 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-04-26T00:20:37.956 INFO:tasks.rados.rados.0.smithi045.stdout:93: finishing write tid 1 to smithi04526579-4 2024-04-26T00:20:37.962 INFO:tasks.rados.rados.0.smithi045.stdout:93: finishing write tid 2 to smithi04526579-4 2024-04-26T00:20:37.963 INFO:tasks.rados.rados.0.smithi045.stdout:93: finishing write tid 3 to smithi04526579-4 2024-04-26T00:20:37.964 INFO:tasks.rados.rados.0.smithi045.stdout:93: finishing write tid 4 to smithi04526579-4 2024-04-26T00:20:37.964 INFO:tasks.rados.rados.0.smithi045.stdout:93: oid 4 updating version 0 to 17 2024-04-26T00:20:37.964 INFO:tasks.rados.rados.0.smithi045.stdout:93: oid 4 updating version 17 to 19 2024-04-26T00:20:37.964 INFO:tasks.rados.rados.0.smithi045.stdout:93: oid 4 version 19 is already newer than 18 2024-04-26T00:20:37.964 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 19 (ObjNum 61 snap 8 seq_num 61) dirty exists 2024-04-26T00:20:37.964 INFO:tasks.rados.rados.0.smithi045.stdout:93: left oid 4 (ObjNum 61 snap 8 seq_num 61) 2024-04-26T00:20:38.896 INFO:tasks.rados.rados.0.smithi045.stdout:91: done (2 left) 2024-04-26T00:20:38.896 INFO:tasks.rados.rados.0.smithi045.stdout:93: done (1 left) 2024-04-26T00:20:38.896 INFO:tasks.rados.rados.0.smithi045.stdout:94: done (0 left) 2024-04-26T00:20:38.896 INFO:tasks.rados.rados.0.smithi045.stdout:95: write oid 2 current snap is 9 2024-04-26T00:20:38.896 INFO:tasks.rados.rados.0.smithi045.stdout:95: seq_num 62 ranges {642034=796262,1895060=564680} 2024-04-26T00:20:38.919 INFO:tasks.rados.rados.0.smithi045.stdout:95: writing smithi04526579-2 from 642034 to 1438296 tid 1 2024-04-26T00:20:38.936 INFO:tasks.rados.rados.0.smithi045.stdout:95: writing smithi04526579-2 from 1895060 to 2459740 tid 2 2024-04-26T00:20:38.936 INFO:tasks.rados.rados.0.smithi045.stdout:96: read oid 1 snap -1 2024-04-26T00:20:38.936 INFO:tasks.rados.rados.0.smithi045.stdout:96: expect (ObjNum 0 snap 0 seq_num 0) 2024-04-26T00:20:38.936 INFO:tasks.rados.rados.0.smithi045.stdout:97: read oid 48 snap 6 2024-04-26T00:20:38.936 INFO:tasks.rados.rados.0.smithi045.stdout:97: expect (ObjNum 47 snap 0 seq_num 47) 2024-04-26T00:20:38.937 INFO:tasks.rados.rados.0.smithi045.stdout:98: snap_remove snap 4 2024-04-26T00:20:39.908 INFO:tasks.rados.rados.0.smithi045.stdout:98: done (3 left) 2024-04-26T00:20:39.908 INFO:tasks.rados.rados.0.smithi045.stdout:99: snap_remove snap 5 2024-04-26T00:20:40.912 INFO:tasks.rados.rados.0.smithi045.stdout:99: done (3 left) 2024-04-26T00:20:40.913 INFO:tasks.rados.rados.0.smithi045.stdout:100: rollback oid 6 current snap is 9 2024-04-26T00:20:40.913 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 6 to 7 2024-04-26T00:20:40.913 INFO:tasks.rados.rados.0.smithi045.stdout:95: finishing write tid 1 to smithi04526579-2 2024-04-26T00:20:40.913 INFO:tasks.rados.rados.0.smithi045.stdout:96: expect (ObjNum 0 snap 0 seq_num 0) 2024-04-26T00:20:41.039 INFO:tasks.rados.rados.0.smithi045.stdout:95: finishing write tid 2 to smithi04526579-2 2024-04-26T00:20:41.039 INFO:tasks.rados.rados.0.smithi045.stdout:95: finishing write tid 3 to smithi04526579-2 2024-04-26T00:20:41.039 INFO:tasks.rados.rados.0.smithi045.stdout:95: finishing write tid 4 to smithi04526579-2 2024-04-26T00:20:41.039 INFO:tasks.rados.rados.0.smithi045.stdout:95: oid 2 updating version 0 to 22 2024-04-26T00:20:41.039 INFO:tasks.rados.rados.0.smithi045.stdout:95: oid 2 version 22 is already newer than 21 2024-04-26T00:20:41.039 INFO:tasks.rados.rados.0.smithi045.stdout:95: oid 2 updating version 22 to 23 2024-04-26T00:20:41.039 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 23 (ObjNum 62 snap 9 seq_num 62) dirty exists 2024-04-26T00:20:41.040 INFO:tasks.rados.rados.0.smithi045.stdout:95: left oid 2 (ObjNum 62 snap 9 seq_num 62) 2024-04-26T00:20:41.040 INFO:tasks.rados.rados.0.smithi045.stdout:97: expect (ObjNum 47 snap 0 seq_num 47) 2024-04-26T00:20:41.129 INFO:tasks.rados.rados.0.smithi045.stdout:100: finishing rollback tid 0 to smithi04526579-6 2024-04-26T00:20:41.129 INFO:tasks.rados.rados.0.smithi045.stdout:100: finishing rollback tid 1 to smithi04526579-6 2024-04-26T00:20:41.129 INFO:tasks.rados.rados.0.smithi045.stdout:100: finishing rollback tid 2 to smithi04526579-6 2024-04-26T00:20:41.129 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 10 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-04-26T00:20:41.129 INFO:tasks.rados.rados.0.smithi045.stdout:95: done (3 left) 2024-04-26T00:20:41.129 INFO:tasks.rados.rados.0.smithi045.stdout:96: done (2 left) 2024-04-26T00:20:41.129 INFO:tasks.rados.rados.0.smithi045.stdout:97: done (1 left) 2024-04-26T00:20:41.129 INFO:tasks.rados.rados.0.smithi045.stdout:100: done (0 left) 2024-04-26T00:20:41.129 INFO:tasks.rados.rados.0.smithi045.stdout:101: snap_remove snap 7 2024-04-26T00:20:41.922 INFO:tasks.rados.rados.0.smithi045.stdout:101: done (0 left) 2024-04-26T00:20:41.922 INFO:tasks.rados.rados.0.smithi045.stdout:102: read oid 45 snap -1 2024-04-26T00:20:41.922 INFO:tasks.rados.rados.0.smithi045.stdout:102: expect (ObjNum 44 snap 0 seq_num 44) 2024-04-26T00:20:41.922 INFO:tasks.rados.rados.0.smithi045.stdout:103: rollback oid 17 current snap is 9 2024-04-26T00:20:41.922 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 17 to 2 2024-04-26T00:20:41.922 INFO:tasks.rados.rados.0.smithi045.stdout:104: read oid 48 snap -1 2024-04-26T00:20:41.922 INFO:tasks.rados.rados.0.smithi045.stdout:104: expect (ObjNum 47 snap 0 seq_num 47) 2024-04-26T00:20:41.922 INFO:tasks.rados.rados.0.smithi045.stdout:105: rollback oid 39 current snap is 9 2024-04-26T00:20:41.923 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 39 to 0 2024-04-26T00:20:41.923 INFO:tasks.rados.rados.0.smithi045.stdout:106: rollback oid 4 current snap is 9 2024-04-26T00:20:41.923 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 4 to 2 2024-04-26T00:20:41.923 INFO:tasks.rados.rados.0.smithi045.stdout:107: snap_remove snap 8 2024-04-26T00:20:42.896 INFO:tasks.rados.rados.0.smithi045.stdout:107: done (5 left) 2024-04-26T00:20:42.896 INFO:tasks.rados.rados.0.smithi045.stdout:108: write (excl) oid 11 current snap is 9 2024-04-26T00:20:42.896 INFO:tasks.rados.rados.0.smithi045.stdout:108: seq_num 63 ranges {658841=471511,1553542=708338,2964119=94722} 2024-04-26T00:20:42.914 INFO:tasks.rados.rados.0.smithi045.stdout:108: writing smithi04526579-11 from 658841 to 1130352 tid 1 2024-04-26T00:20:42.935 INFO:tasks.rados.rados.0.smithi045.stdout:108: writing smithi04526579-11 from 1553542 to 2261880 tid 2 2024-04-26T00:20:43.019 INFO:tasks.rados.rados.0.smithi045.stdout:108: writing smithi04526579-11 from 2964119 to 3058841 tid 3 2024-04-26T00:20:43.019 INFO:tasks.rados.rados.0.smithi045.stdout:109: write oid 25 current snap is 9 2024-04-26T00:20:43.020 INFO:tasks.rados.rados.0.smithi045.stdout:109: seq_num 64 ranges {675648=546760,2012024=568349} 2024-04-26T00:20:43.020 INFO:tasks.rados.rados.0.smithi045.stdout:109: writing smithi04526579-25 from 675648 to 1222408 tid 1 2024-04-26T00:20:43.020 INFO:tasks.rados.rados.0.smithi045.stdout:109: writing smithi04526579-25 from 2012024 to 2580373 tid 2 2024-04-26T00:20:43.020 INFO:tasks.rados.rados.0.smithi045.stdout:110: delete oid 43 current snap is 9 2024-04-26T00:20:43.020 INFO:tasks.rados.rados.0.smithi045.stdout:103: finishing rollback tid 0 to smithi04526579-17 2024-04-26T00:20:43.020 INFO:tasks.rados.rados.0.smithi045.stdout:104: expect (ObjNum 47 snap 0 seq_num 47) 2024-04-26T00:20:43.088 INFO:tasks.rados.rados.0.smithi045.stdout:103: finishing rollback tid 1 to smithi04526579-17 2024-04-26T00:20:43.088 INFO:tasks.rados.rados.0.smithi045.stdout:103: finishing rollback tid 2 to smithi04526579-17 2024-04-26T00:20:43.088 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 37 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-04-26T00:20:43.088 INFO:tasks.rados.rados.0.smithi045.stdout:102: expect (ObjNum 44 snap 0 seq_num 44) 2024-04-26T00:20:43.175 INFO:tasks.rados.rados.0.smithi045.stdout:105: finishing rollback tid 0 to smithi04526579-39 2024-04-26T00:20:43.175 INFO:tasks.rados.rados.0.smithi045.stdout:102: done (7 left) 2024-04-26T00:20:43.176 INFO:tasks.rados.rados.0.smithi045.stdout:103: done (6 left) 2024-04-26T00:20:43.176 INFO:tasks.rados.rados.0.smithi045.stdout:104: done (5 left) 2024-04-26T00:20:43.176 INFO:tasks.rados.rados.0.smithi045.stdout:110: done (4 left) 2024-04-26T00:20:43.176 INFO:tasks.rados.rados.0.smithi045.stdout:111: snap_remove snap 6 2024-04-26T00:20:43.929 INFO:tasks.rados.rados.0.smithi045.stdout:111: done (4 left) 2024-04-26T00:20:43.930 INFO:tasks.rados.rados.0.smithi045.stdout:112: read oid 22 snap -1 2024-04-26T00:20:43.930 INFO:tasks.rados.rados.0.smithi045.stdout:112: expect (ObjNum 53 snap 3 seq_num 53) 2024-04-26T00:20:43.930 INFO:tasks.rados.rados.0.smithi045.stdout:113: write (excl) oid 18 current snap is 9 2024-04-26T00:20:43.930 INFO:tasks.rados.rados.0.smithi045.stdout:105: finishing rollback tid 1 to smithi04526579-39 2024-04-26T00:20:43.930 INFO:tasks.rados.rados.0.smithi045.stdout:106: finishing rollback tid 0 to smithi04526579-4 2024-04-26T00:20:43.930 INFO:tasks.rados.rados.0.smithi045.stdout:105: finishing rollback tid 2 to smithi04526579-39 2024-04-26T00:20:43.931 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 26 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-04-26T00:20:43.931 INFO:tasks.rados.rados.0.smithi045.stdout:106: finishing rollback tid 1 to smithi04526579-4 2024-04-26T00:20:43.931 INFO:tasks.rados.rados.0.smithi045.stdout:106: finishing rollback tid 2 to smithi04526579-4 2024-04-26T00:20:43.931 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 39 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-26T00:20:43.931 INFO:tasks.rados.rados.0.smithi045.stdout:113: seq_num 66 ranges {709262=697258,2012635=572018,3046388=62874} 2024-04-26T00:20:43.952 INFO:tasks.rados.rados.0.smithi045.stdout:113: writing smithi04526579-18 from 709262 to 1406520 tid 1 2024-04-26T00:20:43.971 INFO:tasks.rados.rados.0.smithi045.stdout:113: writing smithi04526579-18 from 2012635 to 2584653 tid 2 2024-04-26T00:20:43.980 INFO:tasks.rados.rados.0.smithi045.stdout:113: writing smithi04526579-18 from 3046388 to 3109262 tid 3 2024-04-26T00:20:43.981 INFO:tasks.rados.rados.0.smithi045.stdout:105: done (5 left) 2024-04-26T00:20:43.981 INFO:tasks.rados.rados.0.smithi045.stdout:106: done (4 left) 2024-04-26T00:20:43.981 INFO:tasks.rados.rados.0.smithi045.stdout:114: read oid 34 snap -1 2024-04-26T00:20:43.981 INFO:tasks.rados.rados.0.smithi045.stdout:114: expect (ObjNum 33 snap 0 seq_num 33) 2024-04-26T00:20:43.981 INFO:tasks.rados.rados.0.smithi045.stdout:115: write (excl) oid 3 current snap is 9 2024-04-26T00:20:43.981 INFO:tasks.rados.rados.0.smithi045.stdout:115: seq_num 67 ranges {726069=772507,2071117=715676} 2024-04-26T00:20:43.992 INFO:tasks.rados.rados.0.smithi045.stdout:115: writing smithi04526579-3 from 726069 to 1498576 tid 1 2024-04-26T00:20:44.005 INFO:tasks.rados.rados.0.smithi045.stdout:115: writing smithi04526579-3 from 2071117 to 2786793 tid 2 2024-04-26T00:20:44.006 INFO:tasks.rados.rados.0.smithi045.stdout:116: delete oid 1 current snap is 9 2024-04-26T00:20:44.006 INFO:tasks.rados.rados.0.smithi045.stdout:108: finishing write tid 1 to smithi04526579-11 2024-04-26T00:20:44.006 INFO:tasks.rados.rados.0.smithi045.stdout:108: finishing write tid 2 to smithi04526579-11 2024-04-26T00:20:44.006 INFO:tasks.rados.rados.0.smithi045.stdout:108: finishing write tid 3 to smithi04526579-11 2024-04-26T00:20:44.006 INFO:tasks.rados.rados.0.smithi045.stdout:108: finishing write tid 4 to smithi04526579-11 2024-04-26T00:20:44.006 INFO:tasks.rados.rados.0.smithi045.stdout:108: finishing write tid 5 to smithi04526579-11 2024-04-26T00:20:44.006 INFO:tasks.rados.rados.0.smithi045.stdout:108: oid 11 updating version 0 to 11 2024-04-26T00:20:44.006 INFO:tasks.rados.rados.0.smithi045.stdout:108: oid 11 version 11 is already newer than 9 2024-04-26T00:20:44.007 INFO:tasks.rados.rados.0.smithi045.stdout:108: oid 11 version 11 is already newer than 10 2024-04-26T00:20:44.007 INFO:tasks.rados.rados.0.smithi045.stdout:108: oid 11 version 11 is already newer than 8 2024-04-26T00:20:44.007 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 11 (ObjNum 63 snap 9 seq_num 63) dirty exists 2024-04-26T00:20:44.007 INFO:tasks.rados.rados.0.smithi045.stdout:108: left oid 11 (ObjNum 63 snap 9 seq_num 63) 2024-04-26T00:20:44.007 INFO:tasks.rados.rados.0.smithi045.stdout:109: finishing write tid 1 to smithi04526579-25 2024-04-26T00:20:44.007 INFO:tasks.rados.rados.0.smithi045.stdout:109: finishing write tid 2 to smithi04526579-25 2024-04-26T00:20:44.007 INFO:tasks.rados.rados.0.smithi045.stdout:109: finishing write tid 3 to smithi04526579-25 2024-04-26T00:20:44.007 INFO:tasks.rados.rados.0.smithi045.stdout:109: finishing write tid 4 to smithi04526579-25 2024-04-26T00:20:44.008 INFO:tasks.rados.rados.0.smithi045.stdout:109: oid 25 updating version 0 to 55 2024-04-26T00:20:44.008 INFO:tasks.rados.rados.0.smithi045.stdout:109: oid 25 version 55 is already newer than 54 2024-04-26T00:20:44.008 INFO:tasks.rados.rados.0.smithi045.stdout:109: oid 25 version 55 is already newer than 53 2024-04-26T00:20:44.008 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 55 (ObjNum 64 snap 9 seq_num 64) dirty exists 2024-04-26T00:20:44.008 INFO:tasks.rados.rados.0.smithi045.stdout:109: left oid 25 (ObjNum 64 snap 9 seq_num 64) 2024-04-26T00:20:44.008 INFO:tasks.rados.rados.0.smithi045.stdout:112: expect (ObjNum 53 snap 3 seq_num 53) 2024-04-26T00:20:44.184 INFO:tasks.rados.rados.0.smithi045.stdout:113: finishing write tid 1 to smithi04526579-18 2024-04-26T00:20:44.185 INFO:tasks.rados.rados.0.smithi045.stdout:113: finishing write tid 2 to smithi04526579-18 2024-04-26T00:20:44.185 INFO:tasks.rados.rados.0.smithi045.stdout:108: done (6 left) 2024-04-26T00:20:44.185 INFO:tasks.rados.rados.0.smithi045.stdout:109: done (5 left) 2024-04-26T00:20:44.185 INFO:tasks.rados.rados.0.smithi045.stdout:112: done (4 left) 2024-04-26T00:20:44.185 INFO:tasks.rados.rados.0.smithi045.stdout:116: done (3 left) 2024-04-26T00:20:44.186 INFO:tasks.rados.rados.0.smithi045.stdout:117: snap_remove snap 2 2024-04-26T00:20:44.935 INFO:tasks.rados.rados.0.smithi045.stdout:117: done (3 left) 2024-04-26T00:20:44.936 INFO:tasks.rados.rados.0.smithi045.stdout:118: write (excl) oid 47 current snap is 9 2024-04-26T00:20:44.936 INFO:tasks.rados.rados.0.smithi045.stdout:118: seq_num 69 ranges {759683=639358,1904434=719345,3128657=31026} 2024-04-26T00:20:44.955 INFO:tasks.rados.rados.0.smithi045.stdout:118: writing smithi04526579-47 from 759683 to 1399041 tid 1 2024-04-26T00:20:44.973 INFO:tasks.rados.rados.0.smithi045.stdout:118: writing smithi04526579-47 from 1904434 to 2623779 tid 2 2024-04-26T00:20:44.982 INFO:tasks.rados.rados.0.smithi045.stdout:118: writing smithi04526579-47 from 3128657 to 3159683 tid 3 2024-04-26T00:20:44.982 INFO:tasks.rados.rados.0.smithi045.stdout:119: snap_create 2024-04-26T00:20:44.982 INFO:tasks.rados.rados.0.smithi045.stdout:113: finishing write tid 3 to smithi04526579-18 2024-04-26T00:20:44.982 INFO:tasks.rados.rados.0.smithi045.stdout:113: finishing write tid 4 to smithi04526579-18 2024-04-26T00:20:44.983 INFO:tasks.rados.rados.0.smithi045.stdout:113: finishing write tid 5 to smithi04526579-18 2024-04-26T00:20:44.983 INFO:tasks.rados.rados.0.smithi045.stdout:113: oid 18 updating version 0 to 62 2024-04-26T00:20:44.983 INFO:tasks.rados.rados.0.smithi045.stdout:113: oid 18 version 62 is already newer than 61 2024-04-26T00:20:44.983 INFO:tasks.rados.rados.0.smithi045.stdout:113: oid 18 version 62 is already newer than 60 2024-04-26T00:20:44.983 INFO:tasks.rados.rados.0.smithi045.stdout:113: oid 18 updating version 62 to 63 2024-04-26T00:20:44.983 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 63 (ObjNum 66 snap 9 seq_num 66) dirty exists 2024-04-26T00:20:44.983 INFO:tasks.rados.rados.0.smithi045.stdout:113: left oid 18 (ObjNum 66 snap 9 seq_num 66) 2024-04-26T00:20:44.983 INFO:tasks.rados.rados.0.smithi045.stdout:114: expect (ObjNum 33 snap 0 seq_num 33) 2024-04-26T00:20:45.085 INFO:tasks.rados.rados.0.smithi045.stdout:115: finishing write tid 1 to smithi04526579-3 2024-04-26T00:20:45.085 INFO:tasks.rados.rados.0.smithi045.stdout:115: finishing write tid 2 to smithi04526579-3 2024-04-26T00:20:45.085 INFO:tasks.rados.rados.0.smithi045.stdout:115: finishing write tid 3 to smithi04526579-3 2024-04-26T00:20:45.085 INFO:tasks.rados.rados.0.smithi045.stdout:115: finishing write tid 4 to smithi04526579-3 2024-04-26T00:20:45.085 INFO:tasks.rados.rados.0.smithi045.stdout:115: oid 3 updating version 0 to 19 2024-04-26T00:20:45.086 INFO:tasks.rados.rados.0.smithi045.stdout:115: oid 3 version 19 is already newer than 18 2024-04-26T00:20:45.086 INFO:tasks.rados.rados.0.smithi045.stdout:115: oid 3 version 19 is already newer than 17 2024-04-26T00:20:45.086 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 19 (ObjNum 67 snap 9 seq_num 67) dirty exists 2024-04-26T00:20:45.086 INFO:tasks.rados.rados.0.smithi045.stdout:115: left oid 3 (ObjNum 67 snap 9 seq_num 67) 2024-04-26T00:20:45.086 INFO:tasks.rados.rados.0.smithi045.stdout:118: finishing write tid 1 to smithi04526579-47 2024-04-26T00:20:45.086 INFO:tasks.rados.rados.0.smithi045.stdout:118: finishing write tid 2 to smithi04526579-47 2024-04-26T00:20:45.086 INFO:tasks.rados.rados.0.smithi045.stdout:118: finishing write tid 3 to smithi04526579-47 2024-04-26T00:20:45.086 INFO:tasks.rados.rados.0.smithi045.stdout:118: finishing write tid 4 to smithi04526579-47 2024-04-26T00:20:45.086 INFO:tasks.rados.rados.0.smithi045.stdout:118: finishing write tid 5 to smithi04526579-47 2024-04-26T00:20:45.086 INFO:tasks.rados.rados.0.smithi045.stdout:118: oid 47 updating version 0 to 23 2024-04-26T00:20:45.086 INFO:tasks.rados.rados.0.smithi045.stdout:118: oid 47 updating version 23 to 24 2024-04-26T00:20:45.087 INFO:tasks.rados.rados.0.smithi045.stdout:118: oid 47 updating version 24 to 26 2024-04-26T00:20:45.087 INFO:tasks.rados.rados.0.smithi045.stdout:118: oid 47 version 26 is already newer than 25 2024-04-26T00:20:45.087 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 26 (ObjNum 69 snap 9 seq_num 69) dirty exists 2024-04-26T00:20:45.087 INFO:tasks.rados.rados.0.smithi045.stdout:118: left oid 47 (ObjNum 69 snap 9 seq_num 69) 2024-04-26T00:20:45.945 INFO:tasks.rados.rados.0.smithi045.stdout:113: done (4 left) 2024-04-26T00:20:45.945 INFO:tasks.rados.rados.0.smithi045.stdout:114: done (3 left) 2024-04-26T00:20:45.945 INFO:tasks.rados.rados.0.smithi045.stdout:115: done (2 left) 2024-04-26T00:20:45.945 INFO:tasks.rados.rados.0.smithi045.stdout:118: done (1 left) 2024-04-26T00:20:45.945 INFO:tasks.rados.rados.0.smithi045.stdout:119: done (0 left) 2024-04-26T00:20:45.945 INFO:tasks.rados.rados.0.smithi045.stdout:120: write (excl) oid 7 current snap is 10 2024-04-26T00:20:45.945 INFO:tasks.rados.rados.0.smithi045.stdout:120: seq_num 70 ranges {776490=714607,2246563=579356} 2024-04-26T00:20:45.967 INFO:tasks.rados.rados.0.smithi045.stdout:120: writing smithi04526579-7 from 776490 to 1491097 tid 1 2024-04-26T00:20:45.988 INFO:tasks.rados.rados.0.smithi045.stdout:120: writing smithi04526579-7 from 2246563 to 2825919 tid 2 2024-04-26T00:20:45.988 INFO:tasks.rados.rados.0.smithi045.stdout:121: read oid 5 snap -1 2024-04-26T00:20:45.988 INFO:tasks.rados.rados.0.smithi045.stdout:121: expect (ObjNum 52 snap 1 seq_num 52) 2024-04-26T00:20:45.988 INFO:tasks.rados.rados.0.smithi045.stdout:122: snap_create 2024-04-26T00:20:45.988 INFO:tasks.rados.rados.0.smithi045.stdout:120: finishing write tid 1 to smithi04526579-7 2024-04-26T00:20:46.004 INFO:tasks.rados.rados.0.smithi045.stdout:120: finishing write tid 2 to smithi04526579-7 2024-04-26T00:20:46.004 INFO:tasks.rados.rados.0.smithi045.stdout:120: finishing write tid 3 to smithi04526579-7 2024-04-26T00:20:46.004 INFO:tasks.rados.rados.0.smithi045.stdout:120: finishing write tid 4 to smithi04526579-7 2024-04-26T00:20:46.004 INFO:tasks.rados.rados.0.smithi045.stdout:120: oid 7 updating version 0 to 74 2024-04-26T00:20:46.004 INFO:tasks.rados.rados.0.smithi045.stdout:120: oid 7 updating version 74 to 75 2024-04-26T00:20:46.004 INFO:tasks.rados.rados.0.smithi045.stdout:120: oid 7 version 75 is already newer than 73 2024-04-26T00:20:46.005 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 75 (ObjNum 70 snap 10 seq_num 70) dirty exists 2024-04-26T00:20:46.005 INFO:tasks.rados.rados.0.smithi045.stdout:120: left oid 7 (ObjNum 70 snap 10 seq_num 70) 2024-04-26T00:20:46.014 INFO:tasks.rados.rados.0.smithi045.stdout:121: expect (ObjNum 52 snap 1 seq_num 52) 2024-04-26T00:20:46.371 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-04-26T00:20:46.372 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-04-26T00:20:46.372 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-26T00:20:46.372 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-26T00:20:46.376 INFO:tasks.ceph.mon.b:Started 2024-04-26T00:20:46.376 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:20:46.376 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:20:46.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb7234c3640 10 monclient: get_monmap_and_config 2024-04-26T00:20:46.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb7234c3640 10 monclient: build_initial_monmap 2024-04-26T00:20:46.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb7234c3640 10 monclient: monmap: 2024-04-26T00:20:46.509 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:46.509 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:46.509 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:46.533173+0000 2024-04-26T00:20:46.509 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:46.533173+0000 2024-04-26T00:20:46.509 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:46.509 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:46.510 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:46.510 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:46.510 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:46.510 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:46.510 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:46.510 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:46.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb7234c3640 1 Processor -- start 2024-04-26T00:20:46.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb7234c3640 1 -- start start 2024-04-26T00:20:46.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb7234c3640 10 monclient: init 2024-04-26T00:20:46.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb7234c3640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:46.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb7234c3640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-04-26T00:20:46.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb7234c3640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb71c0a0260 0x7fb71c0a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:46.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb7234c3640 10 monclient(hunting): picked mon.noname-a con 0x7fb71c0a0260 addr v2:172.21.15.37:3300/0 2024-04-26T00:20:46.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb7234c3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb71c134b30 0x7fb71c134f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:46.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb7234c3640 10 monclient(hunting): picked mon.noname-b con 0x7fb71c134b30 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:46.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb7234c3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb71c135490 addr v1:172.21.15.45:6789/0 2024-04-26T00:20:46.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb7234c3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb71c0a1130 con 0x7fb71c135490 2024-04-26T00:20:46.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb7234c3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:46.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb7234c3640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fb71c09ec10 con 0x7fb71c0a0260 2024-04-26T00:20:46.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb7234c3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:46.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb7234c3640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb71c09f0f0 con 0x7fb71c134b30 2024-04-26T00:20:46.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb7234c3640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:46.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb7234c3640 10 monclient(hunting): authenticate will time out at 1027.838853s 2024-04-26T00:20:46.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb72125f640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb71c0a0260 0x7fb71c0a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:46.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb72125f640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb71c0a0260 0x7fb71c0a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:45352/0 (socket says 172.21.15.37:45352) 2024-04-26T00:20:46.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb72125f640 1 -- 172.21.15.37:0/3661315442 learned_addr learned my addr 172.21.15.37:0/3661315442 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:46.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb72125f640 10 monclient(hunting): get_auth_request con 0x7fb71c0a0260 auth_method 0 2024-04-26T00:20:46.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb72125f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:46.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb72125f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:46.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb72125f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:46.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb720a5e640 1 --2- 172.21.15.37:0/3661315442 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb71c134b30 0x7fb71c134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:46.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb720a5e640 10 monclient(hunting): get_auth_request con 0x7fb71c134b30 auth_method 0 2024-04-26T00:20:46.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb720a5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:46.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb720a5e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:46.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb720a5e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:46.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb72125f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:46.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb72125f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:46.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb72125f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:46.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb720a5e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:46.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb720a5e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:46.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb720a5e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:46.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb72125f640 10 monclient(hunting): handle_auth_done global_id 4410 payload 274 2024-04-26T00:20:46.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb72125f640 1 -- 172.21.15.37:0/3661315442 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb71c134b30 msgr2=0x7fb71c134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:46.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb72125f640 1 --2- 172.21.15.37:0/3661315442 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb71c134b30 0x7fb71c134f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:46.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb72125f640 1 -- 172.21.15.37:0/3661315442 >> v1:172.21.15.45:6789/0 conn(0x7fb71c135490 legacy=0x7fb71c13be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:46.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb72125f640 10 monclient: _finish_hunting 0 2024-04-26T00:20:46.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb72125f640 1 monclient: found mon.noname-a 2024-04-26T00:20:46.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb72125f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:46.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb72125f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:46.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.531+0000 7fb72125f640 1 -- 172.21.15.37:0/3661315442 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb71c06cff0 con 0x7fb71c0a0260 2024-04-26T00:20:46.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.535+0000 7fb72125f640 1 -- 172.21.15.37:0/3661315442 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb71c0a0260 msgr2=0x7fb71c0a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:46.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.535+0000 7fb72125f640 1 --2- 172.21.15.37:0/3661315442 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb71c0a0260 0x7fb71c0a0640 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb7100098a0 tx=0x7fb71001c1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9bf149cf8cef0e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:46.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.535+0000 7fb70ffff640 1 -- 172.21.15.37:0/3661315442 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7100b0070 con 0x7fb71c0a0260 2024-04-26T00:20:46.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb70ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:46.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb70ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:46.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb70ffff640 10 monclient: dump: 2024-04-26T00:20:46.517 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:46.517 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:46.517 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:46.517 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:46.517 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:46.517 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:46.517 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:46.517 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:46.517 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:46.517 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:46.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb70ffff640 10 monclient: _finish_auth 0 2024-04-26T00:20:46.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb70ffff640 10 monclient: _check_auth_tickets 2024-04-26T00:20:46.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb70ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:46.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb7234c3640 5 monclient: authenticate success, global_id 4410 2024-04-26T00:20:46.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb70ffff640 1 -- 172.21.15.37:0/3661315442 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7100ab600 con 0x7fb71c0a0260 2024-04-26T00:20:46.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb7234c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:20:46.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb70ffff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:46.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb7234c3640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:46.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb7234c3640 4 set_mon_vals no callback set 2024-04-26T00:20:46.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb70ffff640 1 -- 172.21.15.37:0/3661315442 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7100ab9d0 con 0x7fb71c0a0260 2024-04-26T00:20:46.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb70ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:46.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb70ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:46.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb70ffff640 10 monclient: dump: 2024-04-26T00:20:46.518 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:46.518 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:46.518 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:46.518 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:46.519 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:46.519 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:46.519 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:46.519 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:46.519 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:46.519 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:46.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb7234c3640 10 monclient: shutdown 2024-04-26T00:20:46.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb7234c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:46.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb7234c3640 1 -- 172.21.15.37:0/3661315442 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb71c0a0260 msgr2=0x7fb71c0a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:46.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb7234c3640 1 --2- 172.21.15.37:0/3661315442 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb71c0a0260 0x7fb71c0a0640 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb7100098a0 tx=0x7fb71001c1e0 comp rx=0 tx=0).stop 2024-04-26T00:20:46.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb7234c3640 1 -- 172.21.15.37:0/3661315442 shutdown_connections 2024-04-26T00:20:46.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb7234c3640 1 --2- 172.21.15.37:0/3661315442 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb71c134b30 0x7fb71c134f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:46.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb7234c3640 1 --2- 172.21.15.37:0/3661315442 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb71c0a0260 0x7fb71c0a0640 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:46.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb7234c3640 1 -- 172.21.15.37:0/3661315442 >> 172.21.15.37:0/3661315442 conn(0x7fb71c0a1ad0 msgr2=0x7fb71c0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:46.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb7234c3640 1 -- 172.21.15.37:0/3661315442 shutdown_connections 2024-04-26T00:20:46.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb7234c3640 1 -- 172.21.15.37:0/3661315442 wait complete. 2024-04-26T00:20:46.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb7234c3640 10 monclient: build_initial_monmap 2024-04-26T00:20:46.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb7234c3640 10 monclient: monmap: 2024-04-26T00:20:46.524 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:46.524 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:46.524 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:46.524 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:46.524 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:46.524 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:46.524 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:46.525 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:46.525 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:46.525 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:46.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb7234c3640 1 Processor -- start 2024-04-26T00:20:46.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb7234c3640 1 -- start start 2024-04-26T00:20:46.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb7234c3640 10 monclient: init 2024-04-26T00:20:46.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb7234c3640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:46.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb7234c3640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:20:46.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb7234c3640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb71c0a0260 0x7fb71c13c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:46.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb7234c3640 10 monclient(hunting): picked mon.noname-a con 0x7fb71c0a0260 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:46.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb7234c3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb71c134b30 0x7fb71c1cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:46.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb7234c3640 10 monclient(hunting): picked mon.noname-b con 0x7fb71c134b30 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:46.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb7234c3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb71c135490 0x7fb71c1398d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:46.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb7234c3640 10 monclient(hunting): picked mon.noname-c con 0x7fb71c135490 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:46.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb7234c3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:46.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb7234c3640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb71c09f430 con 0x7fb71c0a0260 2024-04-26T00:20:46.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb7234c3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:46.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb7234c3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb71c09f2b0 con 0x7fb71c134b30 2024-04-26T00:20:46.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb7234c3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:46.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb7234c3640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb71c09fe90 con 0x7fb71c135490 2024-04-26T00:20:46.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb7234c3640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:46.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb7234c3640 10 monclient(hunting): authenticate will time out at 1027.846435s 2024-04-26T00:20:46.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb721a60640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb71c135490 0x7fb71c1398d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:46.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb72125f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb71c0a0260 0x7fb71c13c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:46.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb721a60640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb71c135490 0x7fb71c1398d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:58822/0 (socket says 172.21.15.37:58822) 2024-04-26T00:20:46.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb721a60640 1 -- 172.21.15.37:0/2025602200 learned_addr learned my addr 172.21.15.37:0/2025602200 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:46.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb721a60640 10 monclient(hunting): get_auth_request con 0x7fb71c135490 auth_method 0 2024-04-26T00:20:46.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb721a60640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:46.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb721a60640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:46.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb721a60640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:46.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb72125f640 10 monclient(hunting): get_auth_request con 0x7fb71c0a0260 auth_method 0 2024-04-26T00:20:46.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb72125f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:46.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb72125f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:46.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb72125f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:46.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb721a60640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:46.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb721a60640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:46.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb720a5e640 1 --2- 172.21.15.37:0/2025602200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb71c134b30 0x7fb71c1cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:46.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb721a60640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:46.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb72125f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:46.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb72125f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:46.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb72125f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:46.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb720a5e640 10 monclient(hunting): get_auth_request con 0x7fb71c134b30 auth_method 0 2024-04-26T00:20:46.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb720a5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:46.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb720a5e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:46.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb720a5e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:46.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb72125f640 10 monclient(hunting): handle_auth_done global_id 4413 payload 824 2024-04-26T00:20:46.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb720a5e640 1 -- 172.21.15.37:0/2025602200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb71c134b30 msgr2=0x7fb71c1cd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:20:46.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb720a5e640 1 -- 172.21.15.37:0/2025602200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb71c134b30 msgr2=0x7fb71c1cd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:20:46.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb720a5e640 1 --2- 172.21.15.37:0/2025602200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb71c134b30 0x7fb71c1cd1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:20:46.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb72125f640 1 -- 172.21.15.37:0/2025602200 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb71c135490 msgr2=0x7fb71c1398d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:46.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb72125f640 1 --2- 172.21.15.37:0/2025602200 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb71c135490 0x7fb71c1398d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:46.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb720a5e640 1 --2- 172.21.15.37:0/2025602200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb71c134b30 0x7fb71c1cd1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:20:46.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb72125f640 1 -- 172.21.15.37:0/2025602200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb71c134b30 msgr2=0x7fb71c1cd1a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:20:46.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb72125f640 1 --2- 172.21.15.37:0/2025602200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb71c134b30 0x7fb71c1cd1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:46.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb72125f640 10 monclient: _finish_hunting 0 2024-04-26T00:20:46.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb72125f640 1 monclient: found mon.noname-a 2024-04-26T00:20:46.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb72125f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:46.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb72125f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:46.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb72125f640 1 -- 172.21.15.37:0/2025602200 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb71c0729d0 con 0x7fb71c0a0260 2024-04-26T00:20:46.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.539+0000 7fb721a60640 1 --2- 172.21.15.37:0/2025602200 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb71c135490 0x7fb71c1398d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:46.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.543+0000 7fb72125f640 1 -- 172.21.15.37:0/2025602200 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb71c0a0260 msgr2=0x7fb71c13c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:46.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.543+0000 7fb72125f640 1 --2- 172.21.15.37:0/2025602200 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb71c0a0260 0x7fb71c13c030 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb7100ab000 tx=0x7fb710018f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:46.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.543+0000 7fb70dffb640 1 -- 172.21.15.37:0/2025602200 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7100b8020 con 0x7fb71c0a0260 2024-04-26T00:20:46.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.543+0000 7fb70dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:46.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.543+0000 7fb70dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:46.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.543+0000 7fb70dffb640 10 monclient: dump: 2024-04-26T00:20:46.529 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:46.529 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:46.530 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:46.530 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:46.530 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:46.530 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:46.530 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:46.530 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:46.530 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:46.530 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:46.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.543+0000 7fb70dffb640 10 monclient: _finish_auth 0 2024-04-26T00:20:46.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.543+0000 7fb70dffb640 10 monclient: _check_auth_tickets 2024-04-26T00:20:46.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.543+0000 7fb70dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:46.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.543+0000 7fb70dffb640 1 -- 172.21.15.37:0/2025602200 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7100b35b0 con 0x7fb71c0a0260 2024-04-26T00:20:46.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.543+0000 7fb70dffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:46.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.543+0000 7fb70dffb640 1 -- 172.21.15.37:0/2025602200 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7100b0040 con 0x7fb71c0a0260 2024-04-26T00:20:46.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.543+0000 7fb70dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:46.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.543+0000 7fb70dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:46.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.543+0000 7fb70dffb640 10 monclient: dump: 2024-04-26T00:20:46.531 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:46.531 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:46.531 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:46.531 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:46.531 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:46.531 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:46.531 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:46.531 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:46.531 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:46.531 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:46.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.543+0000 7fb7234c3640 5 monclient: authenticate success, global_id 4413 2024-04-26T00:20:46.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.543+0000 7fb70effd640 4 set_mon_vals no callback set 2024-04-26T00:20:46.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.543+0000 7fb7234c3640 10 monclient: _renew_subs 2024-04-26T00:20:46.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.543+0000 7fb7234c3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:46.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.543+0000 7fb7234c3640 1 -- 172.21.15.37:0/2025602200 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb71c132c30 con 0x7fb71c0a0260 2024-04-26T00:20:46.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.543+0000 7fb7234c3640 10 monclient: _renew_subs 2024-04-26T00:20:46.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.543+0000 7fb7234c3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:46.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.543+0000 7fb7234c3640 1 -- 172.21.15.37:0/2025602200 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb71c11b620 con 0x7fb71c0a0260 2024-04-26T00:20:46.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.543+0000 7fb7234c3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:46.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.543+0000 7fb7234c3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:46.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.543+0000 7fb7234c3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:46.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.543+0000 7fb7234c3640 1 -- 172.21.15.37:0/2025602200 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb71c066f40 con 0x7fb71c0a0260 2024-04-26T00:20:46.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.543+0000 7fb70dffb640 1 -- 172.21.15.37:0/2025602200 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb7100b3a00 con 0x7fb71c0a0260 2024-04-26T00:20:46.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.543+0000 7fb70dffb640 1 --2- 172.21.15.37:0/2025602200 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb700035f30 0x7fb7000383f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:46.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.543+0000 7fb70dffb640 1 -- 172.21.15.37:0/2025602200 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 3915+0+0 (secure 0 0 0) 0x7fb7100e1b60 con 0x7fb71c0a0260 2024-04-26T00:20:46.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.543+0000 7fb720a5e640 1 --2- 172.21.15.37:0/2025602200 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb700035f30 0x7fb7000383f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:46.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.543+0000 7fb720a5e640 10 monclient: get_auth_request con 0x7fb700035f30 auth_method 0 2024-04-26T00:20:46.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.547+0000 7fb720a5e640 1 --2- 172.21.15.37:0/2025602200 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb700035f30 0x7fb7000383f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb71c13b0d0 tx=0x7fb718008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:46.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.547+0000 7fb70dffb640 1 -- 172.21.15.37:0/2025602200 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fb71c0a1130 con 0x7fb71c0a0260 2024-04-26T00:20:46.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.547+0000 7fb70dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:46.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.547+0000 7fb70dffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:46.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.771+0000 7fb7234c3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:20:46.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.771+0000 7fb7234c3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:20:46.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.771+0000 7fb7234c3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:46.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.771+0000 7fb7234c3640 1 -- 172.21.15.37:0/2025602200 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb71c131e10 con 0x7fb71c0a0260 2024-04-26T00:20:46.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.775+0000 7fb70dffb640 1 -- 172.21.15.37:0/2025602200 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7fb7100bcc50 con 0x7fb71c0a0260 2024-04-26T00:20:46.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.775+0000 7fb70dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:20:46.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.775+0000 7fb70dffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:46.752 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":10,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:20:46.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.775+0000 7fb7234c3640 1 -- 172.21.15.37:0/2025602200 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb700035f30 msgr2=0x7fb7000383f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:46.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.775+0000 7fb7234c3640 1 --2- 172.21.15.37:0/2025602200 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb700035f30 0x7fb7000383f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb71c13b0d0 tx=0x7fb718008040 comp rx=0 tx=0).stop 2024-04-26T00:20:46.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.775+0000 7fb7234c3640 10 monclient: shutdown 2024-04-26T00:20:46.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.775+0000 7fb7234c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:46.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.775+0000 7fb7234c3640 1 -- 172.21.15.37:0/2025602200 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb71c0a0260 msgr2=0x7fb71c13c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:46.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.775+0000 7fb7234c3640 1 --2- 172.21.15.37:0/2025602200 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb71c0a0260 0x7fb71c13c030 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb7100ab000 tx=0x7fb710018f80 comp rx=0 tx=0).stop 2024-04-26T00:20:46.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.775+0000 7fb7234c3640 1 -- 172.21.15.37:0/2025602200 shutdown_connections 2024-04-26T00:20:46.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.775+0000 7fb7234c3640 1 --2- 172.21.15.37:0/2025602200 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb700035f30 0x7fb7000383f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:46.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.775+0000 7fb7234c3640 1 --2- 172.21.15.37:0/2025602200 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb71c135490 0x7fb71c1398d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:46.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.775+0000 7fb7234c3640 1 --2- 172.21.15.37:0/2025602200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb71c134b30 0x7fb71c1cd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:46.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.775+0000 7fb7234c3640 1 --2- 172.21.15.37:0/2025602200 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb71c0a0260 0x7fb71c13c030 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:46.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.775+0000 7fb7234c3640 1 -- 172.21.15.37:0/2025602200 >> 172.21.15.37:0/2025602200 conn(0x7fb71c0a1ad0 msgr2=0x7fb71c0a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:46.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.775+0000 7fb7234c3640 1 -- 172.21.15.37:0/2025602200 shutdown_connections 2024-04-26T00:20:46.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:46.775+0000 7fb7234c3640 1 -- 172.21.15.37:0/2025602200 wait complete. 2024-04-26T00:20:47.718 INFO:tasks.rados.rados.0.smithi045.stdout:120: done (2 left) 2024-04-26T00:20:47.718 INFO:tasks.rados.rados.0.smithi045.stdout:121: done (1 left) 2024-04-26T00:20:47.718 INFO:tasks.rados.rados.0.smithi045.stdout:122: done (0 left) 2024-04-26T00:20:47.718 INFO:tasks.rados.rados.0.smithi045.stdout:123: snap_remove snap 10 2024-04-26T00:20:47.969 INFO:tasks.rados.rados.0.smithi045.stdout:123: done (0 left) 2024-04-26T00:20:47.970 INFO:tasks.rados.rados.0.smithi045.stdout:124: read oid 33 snap -1 2024-04-26T00:20:47.970 INFO:tasks.rados.rados.0.smithi045.stdout:124: expect (ObjNum 32 snap 0 seq_num 32) 2024-04-26T00:20:47.970 INFO:tasks.rados.rados.0.smithi045.stdout:125: read oid 3 snap -1 2024-04-26T00:20:47.970 INFO:tasks.rados.rados.0.smithi045.stdout:125: expect (ObjNum 67 snap 9 seq_num 67) 2024-04-26T00:20:47.970 INFO:tasks.rados.rados.0.smithi045.stdout:126: read oid 13 snap -1 2024-04-26T00:20:47.970 INFO:tasks.rados.rados.0.smithi045.stdout:126: expect (ObjNum 12 snap 0 seq_num 12) 2024-04-26T00:20:47.970 INFO:tasks.rados.rados.0.smithi045.stdout:127: snap_remove snap 0 2024-04-26T00:20:48.974 INFO:tasks.rados.rados.0.smithi045.stdout:127: done (3 left) 2024-04-26T00:20:48.975 INFO:tasks.rados.rados.0.smithi045.stdout:128: copy_from oid 9 from oid 43 current snap is 11 2024-04-26T00:20:48.975 INFO:tasks.rados.rados.0.smithi045.stdout:125: expect (ObjNum 67 snap 9 seq_num 67) 2024-04-26T00:20:49.173 INFO:tasks.rados.rados.0.smithi045.stdout:126: expect (ObjNum 12 snap 0 seq_num 12) 2024-04-26T00:20:49.248 INFO:tasks.rados.rados.0.smithi045.stdout:124: expect (ObjNum 32 snap 0 seq_num 32) 2024-04-26T00:20:49.332 INFO:tasks.rados.rados.0.smithi045.stdout:128: finishing copy_from to smithi04526579-9 2024-04-26T00:20:49.332 INFO:tasks.rados.rados.0.smithi045.stdout:128: got expected ENOENT (src dne) 2024-04-26T00:20:49.332 INFO:tasks.rados.rados.0.smithi045.stdout:128: finishing copy_from racing read to smithi04526579-9 2024-04-26T00:20:49.332 INFO:tasks.rados.rados.0.smithi045.stdout:124: done (3 left) 2024-04-26T00:20:49.332 INFO:tasks.rados.rados.0.smithi045.stdout:125: done (2 left) 2024-04-26T00:20:49.332 INFO:tasks.rados.rados.0.smithi045.stdout:126: done (1 left) 2024-04-26T00:20:49.332 INFO:tasks.rados.rados.0.smithi045.stdout:128: done (0 left) 2024-04-26T00:20:49.332 INFO:tasks.rados.rados.0.smithi045.stdout:129: delete oid 18 current snap is 11 2024-04-26T00:20:49.341 INFO:tasks.rados.rados.0.smithi045.stdout:129: done (0 left) 2024-04-26T00:20:49.341 INFO:tasks.rados.rados.0.smithi045.stdout:130: copy_from oid 49 from oid 43 current snap is 11 2024-04-26T00:20:49.341 INFO:tasks.rados.rados.0.smithi045.stdout:131: copy_from oid 48 from oid 14 current snap is 11 2024-04-26T00:20:49.341 INFO:tasks.rados.rados.0.smithi045.stdout:132: rollback oid 41 current snap is 11 2024-04-26T00:20:49.341 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 41 to 9 2024-04-26T00:20:49.341 INFO:tasks.rados.rados.0.smithi045.stdout:133: write (excl) oid 25 current snap is 11 2024-04-26T00:20:49.341 INFO:tasks.rados.rados.0.smithi045.stdout:133: seq_num 72 ranges {410104=465105,1563527=583025,2810926=399178} 2024-04-26T00:20:49.351 INFO:tasks.rados.rados.0.smithi045.stdout:133: writing smithi04526579-25 from 410104 to 875209 tid 1 2024-04-26T00:20:49.362 INFO:tasks.rados.rados.0.smithi045.stdout:133: writing smithi04526579-25 from 1563527 to 2146552 tid 2 2024-04-26T00:20:49.371 INFO:tasks.rados.rados.0.smithi045.stdout:133: writing smithi04526579-25 from 2810926 to 3210104 tid 3 2024-04-26T00:20:49.371 INFO:tasks.rados.rados.0.smithi045.stdout:134: write oid 47 current snap is 11 2024-04-26T00:20:49.371 INFO:tasks.rados.rados.0.smithi045.stdout:134: seq_num 73 ranges {426911=540354,1622009=726683,3121996=104915} 2024-04-26T00:20:49.379 INFO:tasks.rados.rados.0.smithi045.stdout:134: writing smithi04526579-47 from 426911 to 967265 tid 1 2024-04-26T00:20:49.393 INFO:tasks.rados.rados.0.smithi045.stdout:134: writing smithi04526579-47 from 1622009 to 2348692 tid 2 2024-04-26T00:20:49.400 INFO:tasks.rados.rados.0.smithi045.stdout:134: writing smithi04526579-47 from 3121996 to 3226911 tid 3 2024-04-26T00:20:49.400 INFO:tasks.rados.rados.0.smithi045.stdout:135: write (excl) oid 26 current snap is 11 2024-04-26T00:20:49.400 INFO:tasks.rados.rados.0.smithi045.stdout:132: finishing rollback tid 0 to smithi04526579-41 2024-04-26T00:20:49.400 INFO:tasks.rados.rados.0.smithi045.stdout:135: seq_num 74 ranges {443718=615603,1564138=470341,2633066=610652} 2024-04-26T00:20:49.409 INFO:tasks.rados.rados.0.smithi045.stdout:135: writing smithi04526579-26 from 443718 to 1059321 tid 1 2024-04-26T00:20:49.417 INFO:tasks.rados.rados.0.smithi045.stdout:135: writing smithi04526579-26 from 1564138 to 2034479 tid 2 2024-04-26T00:20:49.426 INFO:tasks.rados.rados.0.smithi045.stdout:135: writing smithi04526579-26 from 2633066 to 3243718 tid 3 2024-04-26T00:20:49.426 INFO:tasks.rados.rados.0.smithi045.stdout:136: copy_from oid 36 from oid 29 current snap is 11 2024-04-26T00:20:49.426 INFO:tasks.rados.rados.0.smithi045.stdout:131: finishing copy_from to smithi04526579-48 2024-04-26T00:20:49.426 INFO:tasks.rados.rados.0.smithi045.stdout:131: got expected ENOENT (src dne) 2024-04-26T00:20:49.426 INFO:tasks.rados.rados.0.smithi045.stdout:130: finishing copy_from to smithi04526579-49 2024-04-26T00:20:49.426 INFO:tasks.rados.rados.0.smithi045.stdout:130: got expected ENOENT (src dne) 2024-04-26T00:20:49.426 INFO:tasks.rados.rados.0.smithi045.stdout:132: finishing rollback tid 1 to smithi04526579-41 2024-04-26T00:20:49.426 INFO:tasks.rados.rados.0.smithi045.stdout:130: finishing copy_from racing read to smithi04526579-49 2024-04-26T00:20:49.426 INFO:tasks.rados.rados.0.smithi045.stdout:131: finishing copy_from racing read to smithi04526579-48 2024-04-26T00:20:49.426 INFO:tasks.rados.rados.0.smithi045.stdout:132: finishing rollback tid 2 to smithi04526579-41 2024-04-26T00:20:49.426 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 25 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-04-26T00:20:49.426 INFO:tasks.rados.rados.0.smithi045.stdout:133: finishing write tid 1 to smithi04526579-25 2024-04-26T00:20:49.427 INFO:tasks.rados.rados.0.smithi045.stdout:133: finishing write tid 2 to smithi04526579-25 2024-04-26T00:20:49.427 INFO:tasks.rados.rados.0.smithi045.stdout:133: finishing write tid 3 to smithi04526579-25 2024-04-26T00:20:49.427 INFO:tasks.rados.rados.0.smithi045.stdout:133: finishing write tid 4 to smithi04526579-25 2024-04-26T00:20:49.427 INFO:tasks.rados.rados.0.smithi045.stdout:133: finishing write tid 5 to smithi04526579-25 2024-04-26T00:20:49.427 INFO:tasks.rados.rados.0.smithi045.stdout:133: oid 25 updating version 0 to 92 2024-04-26T00:20:49.427 INFO:tasks.rados.rados.0.smithi045.stdout:133: oid 25 version 92 is already newer than 90 2024-04-26T00:20:49.427 INFO:tasks.rados.rados.0.smithi045.stdout:133: oid 25 version 92 is already newer than 89 2024-04-26T00:20:49.427 INFO:tasks.rados.rados.0.smithi045.stdout:133: oid 25 version 92 is already newer than 91 2024-04-26T00:20:49.427 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 92 (ObjNum 72 snap 11 seq_num 72) dirty exists 2024-04-26T00:20:49.427 INFO:tasks.rados.rados.0.smithi045.stdout:133: left oid 25 (ObjNum 72 snap 11 seq_num 72) 2024-04-26T00:20:49.427 INFO:tasks.rados.rados.0.smithi045.stdout:134: finishing write tid 1 to smithi04526579-47 2024-04-26T00:20:49.427 INFO:tasks.rados.rados.0.smithi045.stdout:134: finishing write tid 2 to smithi04526579-47 2024-04-26T00:20:49.427 INFO:tasks.rados.rados.0.smithi045.stdout:134: finishing write tid 3 to smithi04526579-47 2024-04-26T00:20:49.427 INFO:tasks.rados.rados.0.smithi045.stdout:134: finishing write tid 4 to smithi04526579-47 2024-04-26T00:20:49.428 INFO:tasks.rados.rados.0.smithi045.stdout:134: finishing write tid 5 to smithi04526579-47 2024-04-26T00:20:49.428 INFO:tasks.rados.rados.0.smithi045.stdout:134: oid 47 updating version 0 to 34 2024-04-26T00:20:49.428 INFO:tasks.rados.rados.0.smithi045.stdout:134: oid 47 updating version 34 to 35 2024-04-26T00:20:49.428 INFO:tasks.rados.rados.0.smithi045.stdout:134: oid 47 version 35 is already newer than 33 2024-04-26T00:20:49.428 INFO:tasks.rados.rados.0.smithi045.stdout:134: oid 47 version 35 is already newer than 32 2024-04-26T00:20:49.428 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 35 (ObjNum 73 snap 11 seq_num 73) dirty exists 2024-04-26T00:20:49.428 INFO:tasks.rados.rados.0.smithi045.stdout:134: left oid 47 (ObjNum 73 snap 11 seq_num 73) 2024-04-26T00:20:49.428 INFO:tasks.rados.rados.0.smithi045.stdout:135: finishing write tid 1 to smithi04526579-26 2024-04-26T00:20:49.428 INFO:tasks.rados.rados.0.smithi045.stdout:135: finishing write tid 2 to smithi04526579-26 2024-04-26T00:20:49.428 INFO:tasks.rados.rados.0.smithi045.stdout:130: done (6 left) 2024-04-26T00:20:49.428 INFO:tasks.rados.rados.0.smithi045.stdout:131: done (5 left) 2024-04-26T00:20:49.428 INFO:tasks.rados.rados.0.smithi045.stdout:132: done (4 left) 2024-04-26T00:20:49.428 INFO:tasks.rados.rados.0.smithi045.stdout:133: done (3 left) 2024-04-26T00:20:49.428 INFO:tasks.rados.rados.0.smithi045.stdout:134: done (2 left) 2024-04-26T00:20:49.428 INFO:tasks.rados.rados.0.smithi045.stdout:137: read oid 18 snap -1 2024-04-26T00:20:49.429 INFO:tasks.rados.rados.0.smithi045.stdout:137: expect deleted 2024-04-26T00:20:49.429 INFO:tasks.rados.rados.0.smithi045.stdout:138: rollback oid 8 current snap is 11 2024-04-26T00:20:49.429 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 8 to 9 2024-04-26T00:20:49.429 INFO:tasks.rados.rados.0.smithi045.stdout:139: snap_create 2024-04-26T00:20:49.431 INFO:tasks.rados.rados.0.smithi045.stdout:135: finishing write tid 3 to smithi04526579-26 2024-04-26T00:20:49.431 INFO:tasks.rados.rados.0.smithi045.stdout:135: finishing write tid 4 to smithi04526579-26 2024-04-26T00:20:49.432 INFO:tasks.rados.rados.0.smithi045.stdout:135: finishing write tid 5 to smithi04526579-26 2024-04-26T00:20:49.432 INFO:tasks.rados.rados.0.smithi045.stdout:135: oid 26 updating version 0 to 28 2024-04-26T00:20:49.432 INFO:tasks.rados.rados.0.smithi045.stdout:135: oid 26 updating version 28 to 29 2024-04-26T00:20:49.432 INFO:tasks.rados.rados.0.smithi045.stdout:135: oid 26 updating version 29 to 30 2024-04-26T00:20:49.432 INFO:tasks.rados.rados.0.smithi045.stdout:135: oid 26 version 30 is already newer than 27 2024-04-26T00:20:49.432 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 30 (ObjNum 74 snap 11 seq_num 74) dirty exists 2024-04-26T00:20:49.433 INFO:tasks.rados.rados.0.smithi045.stdout:135: left oid 26 (ObjNum 74 snap 11 seq_num 74) 2024-04-26T00:20:49.438 INFO:tasks.rados.rados.0.smithi045.stdout:138: finishing rollback tid 0 to smithi04526579-8 2024-04-26T00:20:49.438 INFO:tasks.rados.rados.0.smithi045.stdout:138: finishing rollback tid 1 to smithi04526579-8 2024-04-26T00:20:49.439 INFO:tasks.rados.rados.0.smithi045.stdout:138: finishing rollback tid 2 to smithi04526579-8 2024-04-26T00:20:49.439 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 13 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-04-26T00:20:49.451 INFO:tasks.rados.rados.0.smithi045.stdout:136: finishing copy_from racing read to smithi04526579-36 2024-04-26T00:20:49.453 INFO:tasks.rados.rados.0.smithi045.stdout:136: finishing copy_from to smithi04526579-36 2024-04-26T00:20:49.453 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 9 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-26T00:20:49.771 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:20:49.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc545351640 10 monclient: get_monmap_and_config 2024-04-26T00:20:49.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc545351640 10 monclient: build_initial_monmap 2024-04-26T00:20:49.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc545351640 10 monclient: monmap: 2024-04-26T00:20:49.902 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:49.902 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:49.902 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:49.925267+0000 2024-04-26T00:20:49.902 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:49.925267+0000 2024-04-26T00:20:49.902 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:49.902 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:49.902 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:49.902 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:49.903 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:49.903 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:49.903 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:49.903 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:49.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc545351640 1 Processor -- start 2024-04-26T00:20:49.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc545351640 1 -- start start 2024-04-26T00:20:49.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc545351640 10 monclient: init 2024-04-26T00:20:49.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc545351640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:49.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc545351640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-04-26T00:20:49.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc545351640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc5400a0260 0x7fc5400a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:49.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc545351640 10 monclient(hunting): picked mon.noname-a con 0x7fc5400a0260 addr v2:172.21.15.37:3300/0 2024-04-26T00:20:49.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc545351640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc540134b30 0x7fc540134f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:49.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc545351640 10 monclient(hunting): picked mon.noname-b con 0x7fc540134b30 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:49.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc545351640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc540135490 addr v1:172.21.15.37:6789/0 2024-04-26T00:20:49.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc545351640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5400a1130 con 0x7fc540135490 2024-04-26T00:20:49.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc545351640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:49.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc545351640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fc54009ec10 con 0x7fc5400a0260 2024-04-26T00:20:49.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc545351640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:49.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc545351640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc54009f0f0 con 0x7fc540134b30 2024-04-26T00:20:49.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc545351640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:49.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc545351640 10 monclient(hunting): authenticate will time out at 1031.231125s 2024-04-26T00:20:49.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc53effd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc5400a0260 0x7fc5400a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:49.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc53f7fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fc540135490 0x7fc54013be60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:54950/0 (socket says 172.21.15.37:54950) 2024-04-26T00:20:49.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc5367fc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc540134b30 0x7fc540134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:49.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc53f7fe640 1 -- 172.21.15.37:0/3510429552 learned_addr learned my addr 172.21.15.37:0/3510429552 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:20:49.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc53effd640 10 monclient(hunting): get_auth_request con 0x7fc5400a0260 auth_method 0 2024-04-26T00:20:49.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc53effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:49.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc53effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:49.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc53effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:49.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc5367fc640 10 monclient(hunting): get_auth_request con 0x7fc540134b30 auth_method 0 2024-04-26T00:20:49.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc5367fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:49.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc5367fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:49.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc5367fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:49.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc5367fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:49.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc5367fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:49.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc5367fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:49.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc53effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:49.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc53effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:49.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc53effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:49.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc5367fc640 10 monclient(hunting): handle_auth_done global_id 4409 payload 274 2024-04-26T00:20:49.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc5367fc640 1 -- 172.21.15.37:0/3510429552 >> v2:172.21.15.37:3300/0 conn(0x7fc5400a0260 msgr2=0x7fc5400a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:49.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc5367fc640 1 --2- 172.21.15.37:0/3510429552 >> v2:172.21.15.37:3300/0 conn(0x7fc5400a0260 0x7fc5400a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:49.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc5367fc640 1 -- 172.21.15.37:0/3510429552 >> v1:172.21.15.37:6789/0 conn(0x7fc540135490 legacy=0x7fc54013be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:49.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc5367fc640 10 monclient: _finish_hunting 0 2024-04-26T00:20:49.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc5367fc640 1 monclient: found mon.noname-b 2024-04-26T00:20:49.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc5367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:49.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc5367fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:20:49.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.923+0000 7fc5367fc640 1 -- 172.21.15.37:0/3510429552 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc54006cff0 con 0x7fc540134b30 2024-04-26T00:20:49.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc5367fc640 1 -- 172.21.15.37:0/3510429552 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc540134b30 msgr2=0x7fc540134f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:49.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc5367fc640 1 --2- 172.21.15.37:0/3510429552 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc540134b30 0x7fc540134f50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc5300097c0 tx=0x7fc53001c140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81712a9ed387bda2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:49.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc53e7fc640 1 -- 172.21.15.37:0/3510429552 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5300b1070 con 0x7fc540134b30 2024-04-26T00:20:49.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc53e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:49.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc53e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:20:49.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc53e7fc640 10 monclient: dump: 2024-04-26T00:20:49.922 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:49.922 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:49.922 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:49.922 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:49.922 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:49.923 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:49.923 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:49.923 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:49.923 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:49.923 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:49.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc53e7fc640 10 monclient: _finish_auth 0 2024-04-26T00:20:49.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc53e7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:20:49.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc53e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:49.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc53e7fc640 1 -- 172.21.15.37:0/3510429552 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5300ac440 con 0x7fc540134b30 2024-04-26T00:20:49.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc53e7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:49.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc53e7fc640 1 -- 172.21.15.37:0/3510429552 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5300ac810 con 0x7fc540134b30 2024-04-26T00:20:49.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc545351640 5 monclient: authenticate success, global_id 4409 2024-04-26T00:20:49.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc545351640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:49.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc545351640 4 set_mon_vals no callback set 2024-04-26T00:20:49.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc53e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:49.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc53e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:49.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc53e7fc640 10 monclient: dump: 2024-04-26T00:20:49.925 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:49.925 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:49.925 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:49.925 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:49.925 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:49.925 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:49.925 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:49.926 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:49.926 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:49.926 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:49.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc545351640 10 monclient: shutdown 2024-04-26T00:20:49.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc545351640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:49.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc545351640 1 -- 172.21.15.37:0/3510429552 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc540134b30 msgr2=0x7fc540134f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:49.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc545351640 1 --2- 172.21.15.37:0/3510429552 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc540134b30 0x7fc540134f50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc5300097c0 tx=0x7fc53001c140 comp rx=0 tx=0).stop 2024-04-26T00:20:49.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc545351640 1 -- 172.21.15.37:0/3510429552 shutdown_connections 2024-04-26T00:20:49.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc545351640 1 --2- 172.21.15.37:0/3510429552 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc540134b30 0x7fc540134f50 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:49.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc545351640 1 --2- 172.21.15.37:0/3510429552 >> v2:172.21.15.37:3300/0 conn(0x7fc5400a0260 0x7fc5400a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:49.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc545351640 1 -- 172.21.15.37:0/3510429552 >> 172.21.15.37:0/3510429552 conn(0x7fc5400a1ad0 msgr2=0x7fc5400a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:49.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc545351640 1 -- 172.21.15.37:0/3510429552 shutdown_connections 2024-04-26T00:20:49.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc545351640 1 -- 172.21.15.37:0/3510429552 wait complete. 2024-04-26T00:20:49.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc545351640 10 monclient: build_initial_monmap 2024-04-26T00:20:49.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc545351640 10 monclient: monmap: 2024-04-26T00:20:49.928 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:49.928 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:49.928 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:49.928 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:49.928 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:49.928 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:49.928 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:49.929 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:49.929 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:49.929 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:49.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc545351640 1 Processor -- start 2024-04-26T00:20:49.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc545351640 1 -- start start 2024-04-26T00:20:49.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc545351640 10 monclient: init 2024-04-26T00:20:49.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc545351640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:49.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc545351640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:20:49.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc545351640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5400a0260 0x7fc540070dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:49.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc545351640 10 monclient(hunting): picked mon.noname-b con 0x7fc5400a0260 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:49.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc545351640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc540134b30 0x7fc5401318b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:49.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc545351640 10 monclient(hunting): picked mon.noname-c con 0x7fc540134b30 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:49.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc545351640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc540135490 0x7fc540131df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:49.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc545351640 10 monclient(hunting): picked mon.noname-a con 0x7fc540135490 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:49.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc545351640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:49.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc545351640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fc54009f430 con 0x7fc540135490 2024-04-26T00:20:49.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc545351640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:49.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc545351640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc54009f2b0 con 0x7fc5400a0260 2024-04-26T00:20:49.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc545351640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:49.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc545351640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc54009fe90 con 0x7fc540134b30 2024-04-26T00:20:49.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc545351640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:49.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc545351640 10 monclient(hunting): authenticate will time out at 1031.239438s 2024-04-26T00:20:49.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc53f7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc540135490 0x7fc540131df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:49.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc5367fc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc540134b30 0x7fc5401318b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:49.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc53f7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc540135490 0x7fc540131df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:37448/0 (socket says 172.21.15.37:37448) 2024-04-26T00:20:49.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc53f7fe640 1 -- 172.21.15.37:0/3563759556 learned_addr learned my addr 172.21.15.37:0/3563759556 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:49.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc53f7fe640 10 monclient(hunting): get_auth_request con 0x7fc540135490 auth_method 0 2024-04-26T00:20:49.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc53f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:49.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc53f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:49.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc53f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:49.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc5367fc640 1 --2- 172.21.15.37:0/3563759556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc540134b30 0x7fc5401318b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:33520/0 (socket says 172.21.15.37:33520) 2024-04-26T00:20:49.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc5367fc640 10 monclient(hunting): get_auth_request con 0x7fc540134b30 auth_method 0 2024-04-26T00:20:49.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc5367fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:49.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc5367fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:49.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc5367fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:49.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc53f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:49.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc53f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:49.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc53f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:49.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc5367fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:49.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc5367fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:49.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc5367fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:49.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc53effd640 1 --2- 172.21.15.37:0/3563759556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5400a0260 0x7fc540070dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:49.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc53effd640 10 monclient(hunting): get_auth_request con 0x7fc5400a0260 auth_method 0 2024-04-26T00:20:49.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc53effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:49.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc53effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:49.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc53effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:49.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc5367fc640 10 monclient(hunting): handle_auth_done global_id 4412 payload 824 2024-04-26T00:20:49.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc5367fc640 1 -- 172.21.15.37:0/3563759556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5400a0260 msgr2=0x7fc540070dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:49.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc5367fc640 1 --2- 172.21.15.37:0/3563759556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5400a0260 0x7fc540070dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:49.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc5367fc640 1 -- 172.21.15.37:0/3563759556 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc540135490 msgr2=0x7fc540131df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:49.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc5367fc640 1 --2- 172.21.15.37:0/3563759556 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc540135490 0x7fc540131df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:49.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc5367fc640 10 monclient: _finish_hunting 0 2024-04-26T00:20:49.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc5367fc640 1 monclient: found mon.noname-c 2024-04-26T00:20:49.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc5367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:49.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc5367fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:20:49.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc5367fc640 1 -- 172.21.15.37:0/3563759556 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc54011dca0 con 0x7fc540134b30 2024-04-26T00:20:49.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.931+0000 7fc53f7fe640 1 --2- 172.21.15.37:0/3563759556 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc540135490 0x7fc540131df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:49.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.935+0000 7fc5367fc640 1 -- 172.21.15.37:0/3563759556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc540134b30 msgr2=0x7fc5401318b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:49.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.935+0000 7fc5367fc640 1 --2- 172.21.15.37:0/3563759556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc540134b30 0x7fc5401318b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc5300ab000 tx=0x7fc530018ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:49.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.935+0000 7fc537fff640 1 -- 172.21.15.37:0/3563759556 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5300b9020 con 0x7fc540134b30 2024-04-26T00:20:49.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.935+0000 7fc537fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:49.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.935+0000 7fc537fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:20:49.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.935+0000 7fc537fff640 10 monclient: dump: 2024-04-26T00:20:49.936 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:49.936 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:49.936 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:49.936 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:49.936 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:49.936 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:49.936 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:49.937 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:49.937 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:49.937 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:49.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.935+0000 7fc537fff640 10 monclient: _finish_auth 0 2024-04-26T00:20:49.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.935+0000 7fc537fff640 10 monclient: _check_auth_tickets 2024-04-26T00:20:49.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.935+0000 7fc537fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:49.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.935+0000 7fc537fff640 1 -- 172.21.15.37:0/3563759556 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5300b4880 con 0x7fc540134b30 2024-04-26T00:20:49.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.935+0000 7fc545351640 5 monclient: authenticate success, global_id 4412 2024-04-26T00:20:49.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.935+0000 7fc545351640 10 monclient: _renew_subs 2024-04-26T00:20:49.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.935+0000 7fc545351640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:49.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.935+0000 7fc545351640 1 -- 172.21.15.37:0/3563759556 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc540071a10 con 0x7fc540134b30 2024-04-26T00:20:49.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.935+0000 7fc537fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:49.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.935+0000 7fc537fff640 1 -- 172.21.15.37:0/3563759556 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5300b1040 con 0x7fc540134b30 2024-04-26T00:20:49.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.935+0000 7fc537fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:49.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.935+0000 7fc537fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:49.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.935+0000 7fc537fff640 10 monclient: dump: 2024-04-26T00:20:49.939 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:49.939 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:49.939 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:49.939 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:49.939 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:49.939 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:49.940 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:49.940 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:49.940 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:49.940 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:49.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.935+0000 7fc53d7fa640 4 set_mon_vals no callback set 2024-04-26T00:20:49.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.935+0000 7fc545351640 10 monclient: _renew_subs 2024-04-26T00:20:49.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.935+0000 7fc545351640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:49.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.935+0000 7fc545351640 1 -- 172.21.15.37:0/3563759556 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc54011d9c0 con 0x7fc540134b30 2024-04-26T00:20:49.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.935+0000 7fc545351640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:49.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.935+0000 7fc545351640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:49.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.935+0000 7fc545351640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:49.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.935+0000 7fc545351640 1 -- 172.21.15.37:0/3563759556 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc540066f40 con 0x7fc540134b30 2024-04-26T00:20:49.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.935+0000 7fc537fff640 1 -- 172.21.15.37:0/3563759556 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc5300b4c70 con 0x7fc540134b30 2024-04-26T00:20:49.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.935+0000 7fc537fff640 1 --2- 172.21.15.37:0/3563759556 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc520035f00 0x7fc5200383c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:49.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.935+0000 7fc537fff640 1 -- 172.21.15.37:0/3563759556 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(40..40 src has 1..40) ==== 3975+0+0 (secure 0 0 0) 0x7fc5300e2590 con 0x7fc540134b30 2024-04-26T00:20:49.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.939+0000 7fc53effd640 1 --2- 172.21.15.37:0/3563759556 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc520035f00 0x7fc5200383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:49.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.939+0000 7fc53effd640 10 monclient: get_auth_request con 0x7fc520035f00 auth_method 0 2024-04-26T00:20:49.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.939+0000 7fc53effd640 1 --2- 172.21.15.37:0/3563759556 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc520035f00 0x7fc5200383c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc528009af0 tx=0x7fc528008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:49.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.943+0000 7fc537fff640 1 -- 172.21.15.37:0/3563759556 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc5300c3020 con 0x7fc540134b30 2024-04-26T00:20:49.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.943+0000 7fc537fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:49.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:49.943+0000 7fc537fff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:50.004 INFO:tasks.rados.rados.0.smithi045.stdout:135: done (4 left) 2024-04-26T00:20:50.004 INFO:tasks.rados.rados.0.smithi045.stdout:136: done (3 left) 2024-04-26T00:20:50.004 INFO:tasks.rados.rados.0.smithi045.stdout:137: done (2 left) 2024-04-26T00:20:50.004 INFO:tasks.rados.rados.0.smithi045.stdout:138: done (1 left) 2024-04-26T00:20:50.004 INFO:tasks.rados.rados.0.smithi045.stdout:139: done (0 left) 2024-04-26T00:20:50.004 INFO:tasks.rados.rados.0.smithi045.stdout:140: snap_create 2024-04-26T00:20:50.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:50.167+0000 7fc545351640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:20:50.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:50.167+0000 7fc545351640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:20:50.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:50.167+0000 7fc545351640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:50.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:50.167+0000 7fc545351640 1 -- 172.21.15.37:0/3563759556 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc54011e080 con 0x7fc540134b30 2024-04-26T00:20:50.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:50.167+0000 7fc537fff640 1 -- 172.21.15.37:0/3563759556 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7fc54011dca0 con 0x7fc540134b30 2024-04-26T00:20:50.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:50.167+0000 7fc537fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:20:50.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:50.167+0000 7fc537fff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:50.146 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":10,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":25,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:20:50.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:50.167+0000 7fc545351640 1 -- 172.21.15.37:0/3563759556 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc520035f00 msgr2=0x7fc5200383c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:50.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:50.167+0000 7fc545351640 1 --2- 172.21.15.37:0/3563759556 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc520035f00 0x7fc5200383c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc528009af0 tx=0x7fc528008040 comp rx=0 tx=0).stop 2024-04-26T00:20:50.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:50.167+0000 7fc545351640 10 monclient: shutdown 2024-04-26T00:20:50.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:50.167+0000 7fc545351640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:50.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:50.167+0000 7fc545351640 1 -- 172.21.15.37:0/3563759556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc540134b30 msgr2=0x7fc5401318b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:50.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:50.167+0000 7fc545351640 1 --2- 172.21.15.37:0/3563759556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc540134b30 0x7fc5401318b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc5300ab000 tx=0x7fc530018ee0 comp rx=0 tx=0).stop 2024-04-26T00:20:50.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:50.171+0000 7fc545351640 1 -- 172.21.15.37:0/3563759556 shutdown_connections 2024-04-26T00:20:50.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:50.171+0000 7fc545351640 1 --2- 172.21.15.37:0/3563759556 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc520035f00 0x7fc5200383c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:50.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:50.171+0000 7fc545351640 1 --2- 172.21.15.37:0/3563759556 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc540135490 0x7fc540131df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:50.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:50.171+0000 7fc545351640 1 --2- 172.21.15.37:0/3563759556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc540134b30 0x7fc5401318b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:50.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:50.171+0000 7fc545351640 1 --2- 172.21.15.37:0/3563759556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5400a0260 0x7fc540070dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:50.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:50.171+0000 7fc545351640 1 -- 172.21.15.37:0/3563759556 >> 172.21.15.37:0/3563759556 conn(0x7fc5400a1ad0 msgr2=0x7fc5400a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:50.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:50.171+0000 7fc545351640 1 -- 172.21.15.37:0/3563759556 shutdown_connections 2024-04-26T00:20:50.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:50.171+0000 7fc545351640 1 -- 172.21.15.37:0/3563759556 wait complete. 2024-04-26T00:20:51.009 INFO:tasks.rados.rados.0.smithi045.stdout:140: done (0 left) 2024-04-26T00:20:51.009 INFO:tasks.rados.rados.0.smithi045.stdout:141: write (excl) oid 41 current snap is 13 2024-04-26T00:20:51.010 INFO:tasks.rados.rados.0.smithi045.stdout:141: seq_num 75 ranges {460525=690852,1622620=730352,3060489=200036} 2024-04-26T00:20:51.020 INFO:tasks.rados.rados.0.smithi045.stdout:141: writing smithi04526579-41 from 460525 to 1151377 tid 1 2024-04-26T00:20:51.033 INFO:tasks.rados.rados.0.smithi045.stdout:141: writing smithi04526579-41 from 1622620 to 2352972 tid 2 2024-04-26T00:20:51.042 INFO:tasks.rados.rados.0.smithi045.stdout:141: writing smithi04526579-41 from 3060489 to 3260525 tid 3 2024-04-26T00:20:51.042 INFO:tasks.rados.rados.0.smithi045.stdout:142: write oid 13 current snap is 13 2024-04-26T00:20:51.043 INFO:tasks.rados.rados.0.smithi045.stdout:141: finishing write tid 1 to smithi04526579-41 2024-04-26T00:20:51.043 INFO:tasks.rados.rados.0.smithi045.stdout:141: finishing write tid 2 to smithi04526579-41 2024-04-26T00:20:51.043 INFO:tasks.rados.rados.0.smithi045.stdout:142: seq_num 76 ranges {477332=766101,1681102=474010,2687912=589420} 2024-04-26T00:20:51.054 INFO:tasks.rados.rados.0.smithi045.stdout:142: writing smithi04526579-13 from 477332 to 1243433 tid 1 2024-04-26T00:20:51.062 INFO:tasks.rados.rados.0.smithi045.stdout:142: writing smithi04526579-13 from 1681102 to 2155112 tid 2 2024-04-26T00:20:51.074 INFO:tasks.rados.rados.0.smithi045.stdout:142: writing smithi04526579-13 from 2687912 to 3277332 tid 3 2024-04-26T00:20:51.074 INFO:tasks.rados.rados.0.smithi045.stdout:143: delete oid 40 current snap is 13 2024-04-26T00:20:51.074 INFO:tasks.rados.rados.0.smithi045.stdout:141: finishing write tid 3 to smithi04526579-41 2024-04-26T00:20:51.074 INFO:tasks.rados.rados.0.smithi045.stdout:141: finishing write tid 4 to smithi04526579-41 2024-04-26T00:20:51.074 INFO:tasks.rados.rados.0.smithi045.stdout:141: finishing write tid 5 to smithi04526579-41 2024-04-26T00:20:51.074 INFO:tasks.rados.rados.0.smithi045.stdout:141: oid 41 updating version 0 to 35 2024-04-26T00:20:51.074 INFO:tasks.rados.rados.0.smithi045.stdout:141: oid 41 version 35 is already newer than 32 2024-04-26T00:20:51.074 INFO:tasks.rados.rados.0.smithi045.stdout:141: oid 41 version 35 is already newer than 33 2024-04-26T00:20:51.075 INFO:tasks.rados.rados.0.smithi045.stdout:141: oid 41 version 35 is already newer than 34 2024-04-26T00:20:51.075 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 35 (ObjNum 75 snap 13 seq_num 75) dirty exists 2024-04-26T00:20:51.075 INFO:tasks.rados.rados.0.smithi045.stdout:141: left oid 41 (ObjNum 75 snap 13 seq_num 75) 2024-04-26T00:20:51.075 INFO:tasks.rados.rados.0.smithi045.stdout:142: finishing write tid 1 to smithi04526579-13 2024-04-26T00:20:51.075 INFO:tasks.rados.rados.0.smithi045.stdout:142: finishing write tid 2 to smithi04526579-13 2024-04-26T00:20:51.080 INFO:tasks.rados.rados.0.smithi045.stdout:141: done (2 left) 2024-04-26T00:20:51.080 INFO:tasks.rados.rados.0.smithi045.stdout:143: done (1 left) 2024-04-26T00:20:51.080 INFO:tasks.rados.rados.0.smithi045.stdout:144: snap_remove snap 11 2024-04-26T00:20:52.020 INFO:tasks.rados.rados.0.smithi045.stdout:144: done (1 left) 2024-04-26T00:20:52.020 INFO:tasks.rados.rados.0.smithi045.stdout:145: write (excl) oid 21 current snap is 13 2024-04-26T00:20:52.021 INFO:tasks.rados.rados.0.smithi045.stdout:145: seq_num 78 ranges {510946=632952,1798066=477679,2742758=457217} 2024-04-26T00:20:52.031 INFO:tasks.rados.rados.0.smithi045.stdout:145: writing smithi04526579-21 from 510946 to 1143898 tid 1 2024-04-26T00:20:52.042 INFO:tasks.rados.rados.0.smithi045.stdout:145: writing smithi04526579-21 from 1798066 to 2275745 tid 2 2024-04-26T00:20:52.051 INFO:tasks.rados.rados.0.smithi045.stdout:145: writing smithi04526579-21 from 2742758 to 3199975 tid 3 2024-04-26T00:20:52.051 INFO:tasks.rados.rados.0.smithi045.stdout:146: delete oid 38 current snap is 13 2024-04-26T00:20:52.051 INFO:tasks.rados.rados.0.smithi045.stdout:142: finishing write tid 3 to smithi04526579-13 2024-04-26T00:20:52.051 INFO:tasks.rados.rados.0.smithi045.stdout:142: finishing write tid 4 to smithi04526579-13 2024-04-26T00:20:52.051 INFO:tasks.rados.rados.0.smithi045.stdout:142: finishing write tid 5 to smithi04526579-13 2024-04-26T00:20:52.052 INFO:tasks.rados.rados.0.smithi045.stdout:142: oid 13 updating version 0 to 18 2024-04-26T00:20:52.052 INFO:tasks.rados.rados.0.smithi045.stdout:142: oid 13 version 18 is already newer than 17 2024-04-26T00:20:52.052 INFO:tasks.rados.rados.0.smithi045.stdout:142: oid 13 version 18 is already newer than 16 2024-04-26T00:20:52.052 INFO:tasks.rados.rados.0.smithi045.stdout:142: oid 13 version 18 is already newer than 15 2024-04-26T00:20:52.052 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 18 (ObjNum 76 snap 13 seq_num 76) dirty exists 2024-04-26T00:20:52.052 INFO:tasks.rados.rados.0.smithi045.stdout:142: left oid 13 (ObjNum 76 snap 13 seq_num 76) 2024-04-26T00:20:52.057 INFO:tasks.rados.rados.0.smithi045.stdout:145: finishing write tid 1 to smithi04526579-21 2024-04-26T00:20:52.061 INFO:tasks.rados.rados.0.smithi045.stdout:145: finishing write tid 2 to smithi04526579-21 2024-04-26T00:20:52.063 INFO:tasks.rados.rados.0.smithi045.stdout:145: finishing write tid 3 to smithi04526579-21 2024-04-26T00:20:52.064 INFO:tasks.rados.rados.0.smithi045.stdout:145: finishing write tid 4 to smithi04526579-21 2024-04-26T00:20:52.064 INFO:tasks.rados.rados.0.smithi045.stdout:142: done (2 left) 2024-04-26T00:20:52.064 INFO:tasks.rados.rados.0.smithi045.stdout:146: done (1 left) 2024-04-26T00:20:52.064 INFO:tasks.rados.rados.0.smithi045.stdout:147: write (excl) oid 6 current snap is 13 2024-04-26T00:20:52.065 INFO:tasks.rados.rados.0.smithi045.stdout:147: seq_num 80 ranges {544560=783450,1915030=481348,2797604=479761} 2024-04-26T00:20:52.078 INFO:tasks.rados.rados.0.smithi045.stdout:147: writing smithi04526579-6 from 544560 to 1328010 tid 1 2024-04-26T00:20:52.089 INFO:tasks.rados.rados.0.smithi045.stdout:147: writing smithi04526579-6 from 1915030 to 2396378 tid 2 2024-04-26T00:20:52.099 INFO:tasks.rados.rados.0.smithi045.stdout:147: writing smithi04526579-6 from 2797604 to 3277365 tid 3 2024-04-26T00:20:52.099 INFO:tasks.rados.rados.0.smithi045.stdout:148: snap_remove snap 9 2024-04-26T00:20:52.899 INFO:tasks.rados.rados.0.smithi045.stdout:148: done (2 left) 2024-04-26T00:20:52.900 INFO:tasks.rados.rados.0.smithi045.stdout:149: snap_create 2024-04-26T00:20:52.900 INFO:tasks.rados.rados.0.smithi045.stdout:145: finishing write tid 5 to smithi04526579-21 2024-04-26T00:20:52.900 INFO:tasks.rados.rados.0.smithi045.stdout:145: oid 21 updating version 0 to 8 2024-04-26T00:20:52.900 INFO:tasks.rados.rados.0.smithi045.stdout:145: oid 21 version 8 is already newer than 6 2024-04-26T00:20:52.900 INFO:tasks.rados.rados.0.smithi045.stdout:145: oid 21 version 8 is already newer than 7 2024-04-26T00:20:52.900 INFO:tasks.rados.rados.0.smithi045.stdout:145: oid 21 updating version 8 to 9 2024-04-26T00:20:52.900 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 9 (ObjNum 78 snap 13 seq_num 78) dirty exists 2024-04-26T00:20:52.900 INFO:tasks.rados.rados.0.smithi045.stdout:145: left oid 21 (ObjNum 78 snap 13 seq_num 78) 2024-04-26T00:20:52.900 INFO:tasks.rados.rados.0.smithi045.stdout:147: finishing write tid 1 to smithi04526579-6 2024-04-26T00:20:52.900 INFO:tasks.rados.rados.0.smithi045.stdout:147: finishing write tid 2 to smithi04526579-6 2024-04-26T00:20:52.900 INFO:tasks.rados.rados.0.smithi045.stdout:147: finishing write tid 3 to smithi04526579-6 2024-04-26T00:20:52.900 INFO:tasks.rados.rados.0.smithi045.stdout:147: finishing write tid 4 to smithi04526579-6 2024-04-26T00:20:52.901 INFO:tasks.rados.rados.0.smithi045.stdout:147: finishing write tid 5 to smithi04526579-6 2024-04-26T00:20:52.901 INFO:tasks.rados.rados.0.smithi045.stdout:147: oid 6 updating version 0 to 24 2024-04-26T00:20:52.901 INFO:tasks.rados.rados.0.smithi045.stdout:147: oid 6 version 24 is already newer than 23 2024-04-26T00:20:52.901 INFO:tasks.rados.rados.0.smithi045.stdout:147: oid 6 updating version 24 to 25 2024-04-26T00:20:52.901 INFO:tasks.rados.rados.0.smithi045.stdout:147: oid 6 version 25 is already newer than 22 2024-04-26T00:20:52.901 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 25 (ObjNum 80 snap 13 seq_num 80) dirty exists 2024-04-26T00:20:52.901 INFO:tasks.rados.rados.0.smithi045.stdout:147: left oid 6 (ObjNum 80 snap 13 seq_num 80) 2024-04-26T00:20:53.165 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:20:53.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f25589d0640 10 monclient: get_monmap_and_config 2024-04-26T00:20:53.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f25589d0640 10 monclient: build_initial_monmap 2024-04-26T00:20:53.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f25589d0640 10 monclient: monmap: 2024-04-26T00:20:53.298 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:53.298 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:53.299 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:53.321209+0000 2024-04-26T00:20:53.299 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:53.321209+0000 2024-04-26T00:20:53.299 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:53.299 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:53.299 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:53.299 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:53.299 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:53.299 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:53.300 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:53.300 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:53.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f25589d0640 1 Processor -- start 2024-04-26T00:20:53.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f25589d0640 1 -- start start 2024-04-26T00:20:53.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f25589d0640 10 monclient: init 2024-04-26T00:20:53.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f25589d0640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:53.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f25589d0640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-04-26T00:20:53.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f25589d0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f25540a0260 addr v1:172.21.15.37:6789/0 2024-04-26T00:20:53.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f25589d0640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f2554137e40 0x7f255413a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:53.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f25589d0640 10 monclient(hunting): picked mon.noname-a con 0x7f2554137e40 addr v2:172.21.15.37:3300/0 2024-04-26T00:20:53.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f25589d0640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f255413a770 0x7f255413cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:53.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f25589d0640 10 monclient(hunting): picked mon.noname-b con 0x7f255413a770 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:53.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f25589d0640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25540a1130 con 0x7f25540a0260 2024-04-26T00:20:53.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f25589d0640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:53.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f25589d0640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f25540a0d50 con 0x7f2554137e40 2024-04-26T00:20:53.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f25589d0640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:53.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f25589d0640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f255409ec10 con 0x7f255413a770 2024-04-26T00:20:53.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f25589d0640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:53.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f25589d0640 10 monclient(hunting): authenticate will time out at 1034.626899s 2024-04-26T00:20:53.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f2551d9b640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f2554137e40 0x7f255413a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:53.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f255259c640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f25540a0260 0x7f25540a0640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:54956/0 (socket says 172.21.15.37:54956) 2024-04-26T00:20:53.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f2552d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f255413a770 0x7f255413cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:53.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f255259c640 1 -- 172.21.15.37:0/169413089 learned_addr learned my addr 172.21.15.37:0/169413089 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:20:53.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f2551d9b640 10 monclient(hunting): get_auth_request con 0x7f2554137e40 auth_method 0 2024-04-26T00:20:53.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f2551d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:53.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f2551d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:53.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f2551d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:53.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f2552d9d640 10 monclient(hunting): get_auth_request con 0x7f255413a770 auth_method 0 2024-04-26T00:20:53.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f2552d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:53.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f2552d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:53.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f2552d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:53.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f2551d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:53.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f2551d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:53.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f2551d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:53.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f2552d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:53.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f2552d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:53.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f2552d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:53.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f2552d9d640 10 monclient(hunting): handle_auth_done global_id 4415 payload 274 2024-04-26T00:20:53.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f2552d9d640 1 -- 172.21.15.37:0/169413089 >> v2:172.21.15.37:3300/0 conn(0x7f2554137e40 msgr2=0x7f255413a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:53.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f2552d9d640 1 --2- 172.21.15.37:0/169413089 >> v2:172.21.15.37:3300/0 conn(0x7f2554137e40 0x7f255413a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:53.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f2552d9d640 1 -- 172.21.15.37:0/169413089 >> v1:172.21.15.37:6789/0 conn(0x7f25540a0260 legacy=0x7f25540a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:53.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f2552d9d640 10 monclient: _finish_hunting 0 2024-04-26T00:20:53.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f2552d9d640 1 monclient: found mon.noname-b 2024-04-26T00:20:53.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f2552d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:53.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f2552d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:20:53.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f2552d9d640 1 -- 172.21.15.37:0/169413089 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f255406cff0 con 0x7f255413a770 2024-04-26T00:20:53.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.319+0000 7f2551d9b640 1 --2- 172.21.15.37:0/169413089 >> v2:172.21.15.37:3300/0 conn(0x7f2554137e40 0x7f255413a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:53.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.323+0000 7f2552d9d640 1 -- 172.21.15.37:0/169413089 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f255413a770 msgr2=0x7f255413cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:53.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.323+0000 7f2552d9d640 1 --2- 172.21.15.37:0/169413089 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f255413a770 0x7f255413cb60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f254c009d90 tx=0x7f254c01c610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=382e99c19627335c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:53.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.323+0000 7f255159a640 1 -- 172.21.15.37:0/169413089 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f254c0a6040 con 0x7f255413a770 2024-04-26T00:20:53.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f255159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:53.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f255159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:20:53.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f255159a640 10 monclient: dump: 2024-04-26T00:20:53.308 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:53.308 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:53.308 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:53.309 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:53.309 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:53.309 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:53.309 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:53.309 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:53.309 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:53.309 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:53.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f255159a640 10 monclient: _finish_auth 0 2024-04-26T00:20:53.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f255159a640 10 monclient: _check_auth_tickets 2024-04-26T00:20:53.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f255159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:53.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f255159a640 1 -- 172.21.15.37:0/169413089 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f254c0af070 con 0x7f255413a770 2024-04-26T00:20:53.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f25589d0640 5 monclient: authenticate success, global_id 4415 2024-04-26T00:20:53.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f25589d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:20:53.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f255159a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:53.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f25589d0640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:53.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f25589d0640 4 set_mon_vals no callback set 2024-04-26T00:20:53.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f255159a640 1 -- 172.21.15.37:0/169413089 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f254c0aa860 con 0x7f255413a770 2024-04-26T00:20:53.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f255159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:53.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f255159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:53.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f255159a640 10 monclient: dump: 2024-04-26T00:20:53.310 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:53.310 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:53.310 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:53.310 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:53.310 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:53.311 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:53.311 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:53.311 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:53.311 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:53.311 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:53.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f25589d0640 10 monclient: shutdown 2024-04-26T00:20:53.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f25589d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:53.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f25589d0640 1 -- 172.21.15.37:0/169413089 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f255413a770 msgr2=0x7f255413cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:53.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f25589d0640 1 --2- 172.21.15.37:0/169413089 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f255413a770 0x7f255413cb60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f254c009d90 tx=0x7f254c01c610 comp rx=0 tx=0).stop 2024-04-26T00:20:53.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f25589d0640 1 -- 172.21.15.37:0/169413089 shutdown_connections 2024-04-26T00:20:53.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f25589d0640 1 --2- 172.21.15.37:0/169413089 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f255413a770 0x7f255413cb60 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:53.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f25589d0640 1 --2- 172.21.15.37:0/169413089 >> v2:172.21.15.37:3300/0 conn(0x7f2554137e40 0x7f255413a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:53.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f25589d0640 1 -- 172.21.15.37:0/169413089 >> 172.21.15.37:0/169413089 conn(0x7f25540a1ad0 msgr2=0x7f25540a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:53.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f25589d0640 1 -- 172.21.15.37:0/169413089 shutdown_connections 2024-04-26T00:20:53.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f25589d0640 1 -- 172.21.15.37:0/169413089 wait complete. 2024-04-26T00:20:53.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f25589d0640 10 monclient: build_initial_monmap 2024-04-26T00:20:53.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f25589d0640 10 monclient: monmap: 2024-04-26T00:20:53.312 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:53.312 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:53.312 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:53.312 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:53.312 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:53.312 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:53.312 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:53.312 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:53.312 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:53.313 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:53.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f25589d0640 1 Processor -- start 2024-04-26T00:20:53.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f25589d0640 1 -- start start 2024-04-26T00:20:53.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f25589d0640 10 monclient: init 2024-04-26T00:20:53.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f25589d0640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:53.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f25589d0640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:20:53.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f25589d0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25540a0260 0x7f25541cb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:53.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f25589d0640 10 monclient(hunting): picked mon.noname-b con 0x7f25540a0260 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:53.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f25589d0640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2554137e40 0x7f2554135040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:53.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f25589d0640 10 monclient(hunting): picked mon.noname-a con 0x7f2554137e40 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:53.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f25589d0640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f255413a770 0x7f25541ceed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:53.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f25589d0640 10 monclient(hunting): picked mon.noname-c con 0x7f255413a770 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:53.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f25589d0640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:53.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f25589d0640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f255409f0f0 con 0x7f2554137e40 2024-04-26T00:20:53.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f25589d0640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:53.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f25589d0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f25540a00e0 con 0x7f25540a0260 2024-04-26T00:20:53.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f25589d0640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:53.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f25589d0640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f255409f4c0 con 0x7f255413a770 2024-04-26T00:20:53.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f25589d0640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:53.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f25589d0640 10 monclient(hunting): authenticate will time out at 1034.634318s 2024-04-26T00:20:53.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f2551d9b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2554137e40 0x7f2554135040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:53.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f2552d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f255413a770 0x7f25541ceed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:53.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f2551d9b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2554137e40 0x7f2554135040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:37470/0 (socket says 172.21.15.37:37470) 2024-04-26T00:20:53.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f2552d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f255413a770 0x7f25541ceed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:33528/0 (socket says 172.21.15.37:33528) 2024-04-26T00:20:53.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f2551d9b640 1 -- 172.21.15.37:0/4122898518 learned_addr learned my addr 172.21.15.37:0/4122898518 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:53.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f2551d9b640 10 monclient(hunting): get_auth_request con 0x7f2554137e40 auth_method 0 2024-04-26T00:20:53.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f2551d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:53.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f2551d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:53.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f2551d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:53.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f2552d9d640 10 monclient(hunting): get_auth_request con 0x7f255413a770 auth_method 0 2024-04-26T00:20:53.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f2552d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:53.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f2552d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:53.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f2552d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:53.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f2551d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:53.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f2551d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:53.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f2551d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:53.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f2552d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:53.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f2552d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:53.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f2552d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:53.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f2551d9b640 10 monclient(hunting): handle_auth_done global_id 4425 payload 824 2024-04-26T00:20:53.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f2551d9b640 1 -- 172.21.15.37:0/4122898518 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f255413a770 msgr2=0x7f25541ceed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:53.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f2551d9b640 1 --2- 172.21.15.37:0/4122898518 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f255413a770 0x7f25541ceed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:53.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f255259c640 1 --2- 172.21.15.37:0/4122898518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25540a0260 0x7f25541cb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:53.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f2551d9b640 1 -- 172.21.15.37:0/4122898518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25540a0260 msgr2=0x7f25541cb9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:53.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f2551d9b640 1 --2- 172.21.15.37:0/4122898518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25540a0260 0x7f25541cb9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:53.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f2551d9b640 10 monclient: _finish_hunting 0 2024-04-26T00:20:53.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f2551d9b640 1 monclient: found mon.noname-a 2024-04-26T00:20:53.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f2551d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:53.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f2551d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:53.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f2551d9b640 1 -- 172.21.15.37:0/4122898518 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2554121710 con 0x7f2554137e40 2024-04-26T00:20:53.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f2552d9d640 1 --2- 172.21.15.37:0/4122898518 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f255413a770 0x7f25541ceed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:53.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f255259c640 10 monclient: get_auth_request con 0x7f25540a0260 auth_method 0 2024-04-26T00:20:53.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.327+0000 7f255259c640 1 --2- 172.21.15.37:0/4122898518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25540a0260 0x7f25541cb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:20:53.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.331+0000 7f2551d9b640 1 -- 172.21.15.37:0/4122898518 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2554137e40 msgr2=0x7f2554135040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:53.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.331+0000 7f2551d9b640 1 --2- 172.21.15.37:0/4122898518 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2554137e40 0x7f2554135040 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f254400ebe0 tx=0x7f254400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:53.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.331+0000 7f2542ffd640 1 -- 172.21.15.37:0/4122898518 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2544017070 con 0x7f2554137e40 2024-04-26T00:20:53.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.331+0000 7f2542ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:53.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.331+0000 7f2542ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:53.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.331+0000 7f2542ffd640 10 monclient: dump: 2024-04-26T00:20:53.326 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:53.326 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:53.326 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:53.326 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:53.326 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:53.326 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:53.326 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:53.327 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:53.327 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:53.327 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:53.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.331+0000 7f2542ffd640 10 monclient: _finish_auth 0 2024-04-26T00:20:53.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.331+0000 7f2542ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:20:53.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.331+0000 7f2542ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:53.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.331+0000 7f2542ffd640 1 -- 172.21.15.37:0/4122898518 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f254400ce40 con 0x7f2554137e40 2024-04-26T00:20:53.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.331+0000 7f25589d0640 5 monclient: authenticate success, global_id 4425 2024-04-26T00:20:53.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.331+0000 7f25589d0640 10 monclient: _renew_subs 2024-04-26T00:20:53.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.331+0000 7f25589d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:53.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.331+0000 7f25589d0640 1 -- 172.21.15.37:0/4122898518 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2554121150 con 0x7f2554137e40 2024-04-26T00:20:53.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.331+0000 7f2542ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:53.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.331+0000 7f2542ffd640 1 -- 172.21.15.37:0/4122898518 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2544013690 con 0x7f2554137e40 2024-04-26T00:20:53.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.331+0000 7f2542ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:53.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.331+0000 7f2542ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:53.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.331+0000 7f2542ffd640 10 monclient: dump: 2024-04-26T00:20:53.328 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:53.328 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:53.328 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:53.328 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:53.328 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:53.328 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:53.328 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:53.328 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:53.328 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:53.328 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:53.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.331+0000 7f2543fff640 4 set_mon_vals no callback set 2024-04-26T00:20:53.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.331+0000 7f25589d0640 10 monclient: _renew_subs 2024-04-26T00:20:53.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.331+0000 7f25589d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:53.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.331+0000 7f25589d0640 1 -- 172.21.15.37:0/4122898518 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2554120260 con 0x7f2554137e40 2024-04-26T00:20:53.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.331+0000 7f25589d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:53.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.331+0000 7f25589d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:53.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.331+0000 7f25589d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:53.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.331+0000 7f25589d0640 1 -- 172.21.15.37:0/4122898518 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2554066f40 con 0x7f2554137e40 2024-04-26T00:20:53.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.331+0000 7f2542ffd640 1 -- 172.21.15.37:0/4122898518 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f254401a070 con 0x7f2554137e40 2024-04-26T00:20:53.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.331+0000 7f2542ffd640 1 --2- 172.21.15.37:0/4122898518 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2534035f00 0x7f25340383c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:53.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.331+0000 7f2542ffd640 1 -- 172.21.15.37:0/4122898518 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 4019+0+0 (secure 0 0 0) 0x7f2544048ae0 con 0x7f2554137e40 2024-04-26T00:20:53.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.331+0000 7f255259c640 1 --2- 172.21.15.37:0/4122898518 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2534035f00 0x7f25340383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:53.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.331+0000 7f255259c640 10 monclient: get_auth_request con 0x7f2534035f00 auth_method 0 2024-04-26T00:20:53.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.335+0000 7f255259c640 1 --2- 172.21.15.37:0/4122898518 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2534035f00 0x7f25340383c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2554120550 tx=0x7f2548009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:53.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.335+0000 7f2542ffd640 1 -- 172.21.15.37:0/4122898518 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f2554121710 con 0x7f2554137e40 2024-04-26T00:20:53.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.335+0000 7f2542ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:53.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.335+0000 7f2542ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:53.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.563+0000 7f25589d0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:20:53.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.563+0000 7f25589d0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:20:53.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.563+0000 7f25589d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:53.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.563+0000 7f25589d0640 1 -- 172.21.15.37:0/4122898518 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2554072010 con 0x7f2554137e40 2024-04-26T00:20:53.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.563+0000 7f2542ffd640 1 -- 172.21.15.37:0/4122898518 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f2544013830 con 0x7f2554137e40 2024-04-26T00:20:53.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.563+0000 7f2542ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:20:53.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.563+0000 7f2542ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:53.541 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:20:53.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.563+0000 7f25589d0640 1 -- 172.21.15.37:0/4122898518 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2534035f00 msgr2=0x7f25340383c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:53.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.563+0000 7f25589d0640 1 --2- 172.21.15.37:0/4122898518 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2534035f00 0x7f25340383c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2554120550 tx=0x7f2548009040 comp rx=0 tx=0).stop 2024-04-26T00:20:53.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.563+0000 7f25589d0640 10 monclient: shutdown 2024-04-26T00:20:53.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.563+0000 7f25589d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:53.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.563+0000 7f25589d0640 1 -- 172.21.15.37:0/4122898518 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2554137e40 msgr2=0x7f2554135040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:53.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.563+0000 7f25589d0640 1 --2- 172.21.15.37:0/4122898518 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2554137e40 0x7f2554135040 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f254400ebe0 tx=0x7f254400c520 comp rx=0 tx=0).stop 2024-04-26T00:20:53.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.563+0000 7f25589d0640 1 -- 172.21.15.37:0/4122898518 shutdown_connections 2024-04-26T00:20:53.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.563+0000 7f25589d0640 1 --2- 172.21.15.37:0/4122898518 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2534035f00 0x7f25340383c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:53.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.563+0000 7f25589d0640 1 --2- 172.21.15.37:0/4122898518 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f255413a770 0x7f25541ceed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:53.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.563+0000 7f25589d0640 1 --2- 172.21.15.37:0/4122898518 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2554137e40 0x7f2554135040 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:53.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.563+0000 7f25589d0640 1 --2- 172.21.15.37:0/4122898518 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25540a0260 0x7f25541cb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:53.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.563+0000 7f25589d0640 1 -- 172.21.15.37:0/4122898518 >> 172.21.15.37:0/4122898518 conn(0x7f25540a1ad0 msgr2=0x7f2554136030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:53.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.563+0000 7f25589d0640 1 -- 172.21.15.37:0/4122898518 shutdown_connections 2024-04-26T00:20:53.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.563+0000 7f25589d0640 1 -- 172.21.15.37:0/4122898518 wait complete. 2024-04-26T00:20:53.560 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:20:53.560 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:20:53.560 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:20:53.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff05cb3a640 10 monclient: get_monmap_and_config 2024-04-26T00:20:53.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff05cb3a640 10 monclient: build_initial_monmap 2024-04-26T00:20:53.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff05cb3a640 10 monclient: monmap: 2024-04-26T00:20:53.676 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:53.676 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:53.676 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:53.700379+0000 2024-04-26T00:20:53.676 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:53.700379+0000 2024-04-26T00:20:53.677 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:53.677 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:53.677 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:53.677 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:53.677 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:53.677 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:53.677 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:53.678 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:53.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff05cb3a640 1 Processor -- start 2024-04-26T00:20:53.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff05cb3a640 1 -- start start 2024-04-26T00:20:53.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff05cb3a640 10 monclient: init 2024-04-26T00:20:53.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff05cb3a640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:53.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff05cb3a640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-04-26T00:20:53.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff05cb3a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff058136f20 0x7ff05809bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:53.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff05cb3a640 10 monclient(hunting): picked mon.noname-b con 0x7ff058136f20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:53.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff05cb3a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff058137300 addr v1:172.21.15.37:6789/0 2024-04-26T00:20:53.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff05cb3a640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff05813c450 0x7ff05813e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:53.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff05cb3a640 10 monclient(hunting): picked mon.noname-a con 0x7ff05813c450 addr v2:172.21.15.37:3300/0 2024-04-26T00:20:53.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff05cb3a640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0580a50f0 con 0x7ff058137300 2024-04-26T00:20:53.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff05cb3a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:53.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff05cb3a640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7ff05809b330 con 0x7ff05813c450 2024-04-26T00:20:53.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff05cb3a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:53.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff05cb3a640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff05809b610 con 0x7ff058136f20 2024-04-26T00:20:53.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff05cb3a640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:53.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff05cb3a640 10 monclient(hunting): authenticate will time out at 1035.005993s 2024-04-26T00:20:53.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff056d9d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff05813c450 0x7ff05813e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:53.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff05659c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff058136f20 0x7ff05809bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:53.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff055d9b640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7ff058137300 0x7ff058137780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:54960/0 (socket says 172.21.15.37:54960) 2024-04-26T00:20:53.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff055d9b640 1 -- 172.21.15.37:0/4217385876 learned_addr learned my addr 172.21.15.37:0/4217385876 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:20:53.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff056d9d640 10 monclient(hunting): get_auth_request con 0x7ff05813c450 auth_method 0 2024-04-26T00:20:53.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff056d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:53.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff056d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:53.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff056d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:53.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff05659c640 10 monclient(hunting): get_auth_request con 0x7ff058136f20 auth_method 0 2024-04-26T00:20:53.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff05659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:53.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff05659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:53.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff05659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:53.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff056d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:53.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff056d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:53.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff056d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:53.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff05659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:53.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff05659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:53.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff05659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:53.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff056d9d640 10 monclient(hunting): handle_auth_done global_id 4428 payload 274 2024-04-26T00:20:53.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff056d9d640 1 -- 172.21.15.37:0/4217385876 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff058136f20 msgr2=0x7ff05809bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:53.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff056d9d640 1 --2- 172.21.15.37:0/4217385876 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff058136f20 0x7ff05809bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:53.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff056d9d640 1 -- 172.21.15.37:0/4217385876 >> v1:172.21.15.37:6789/0 conn(0x7ff058137300 legacy=0x7ff058137780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:53.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff056d9d640 10 monclient: _finish_hunting 0 2024-04-26T00:20:53.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff056d9d640 1 monclient: found mon.noname-a 2024-04-26T00:20:53.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff056d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:53.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff056d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:53.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.699+0000 7ff056d9d640 1 -- 172.21.15.37:0/4217385876 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff05809b8f0 con 0x7ff05813c450 2024-04-26T00:20:53.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.703+0000 7ff056d9d640 1 -- 172.21.15.37:0/4217385876 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff05813c450 msgr2=0x7ff05813e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:53.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.703+0000 7ff056d9d640 1 --2- 172.21.15.37:0/4217385876 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff05813c450 0x7ff05813e840 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff058137e90 tx=0x7ff05001c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=117c1afb8faed2c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:53.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.703+0000 7ff05559a640 1 -- 172.21.15.37:0/4217385876 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0500a66d0 con 0x7ff05813c450 2024-04-26T00:20:53.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.703+0000 7ff05559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:53.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.703+0000 7ff05559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:53.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.703+0000 7ff05559a640 10 monclient: dump: 2024-04-26T00:20:53.694 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:53.694 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:53.694 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:53.694 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:53.695 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:53.695 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:53.695 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:53.695 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:53.695 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:53.695 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:53.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.703+0000 7ff05559a640 10 monclient: _finish_auth 0 2024-04-26T00:20:53.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.703+0000 7ff05559a640 10 monclient: _check_auth_tickets 2024-04-26T00:20:53.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.703+0000 7ff05559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:53.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.703+0000 7ff05cb3a640 5 monclient: authenticate success, global_id 4428 2024-04-26T00:20:53.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.703+0000 7ff05559a640 1 -- 172.21.15.37:0/4217385876 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0500a6870 con 0x7ff05813c450 2024-04-26T00:20:53.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.703+0000 7ff05cb3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:20:53.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.703+0000 7ff05559a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:53.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.703+0000 7ff05cb3a640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:53.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.703+0000 7ff05cb3a640 4 set_mon_vals no callback set 2024-04-26T00:20:53.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.703+0000 7ff05559a640 1 -- 172.21.15.37:0/4217385876 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0500a6c40 con 0x7ff05813c450 2024-04-26T00:20:53.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.703+0000 7ff05559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:53.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.703+0000 7ff05559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:53.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.703+0000 7ff05559a640 10 monclient: dump: 2024-04-26T00:20:53.696 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:53.696 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:53.696 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:53.696 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:53.697 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:53.697 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:53.697 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:53.697 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:53.697 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:53.697 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:53.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.703+0000 7ff05cb3a640 10 monclient: shutdown 2024-04-26T00:20:53.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.703+0000 7ff05cb3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:53.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.703+0000 7ff05cb3a640 1 -- 172.21.15.37:0/4217385876 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff05813c450 msgr2=0x7ff05813e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:53.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.703+0000 7ff05cb3a640 1 --2- 172.21.15.37:0/4217385876 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff05813c450 0x7ff05813e840 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff058137e90 tx=0x7ff05001c840 comp rx=0 tx=0).stop 2024-04-26T00:20:53.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.703+0000 7ff05cb3a640 1 -- 172.21.15.37:0/4217385876 shutdown_connections 2024-04-26T00:20:53.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.703+0000 7ff05cb3a640 1 --2- 172.21.15.37:0/4217385876 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff05813c450 0x7ff05813e840 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:53.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.703+0000 7ff05cb3a640 1 --2- 172.21.15.37:0/4217385876 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff058136f20 0x7ff05809bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:53.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.703+0000 7ff05cb3a640 1 -- 172.21.15.37:0/4217385876 >> 172.21.15.37:0/4217385876 conn(0x7ff0580a61b0 msgr2=0x7ff0580a65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:53.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.703+0000 7ff05cb3a640 1 -- 172.21.15.37:0/4217385876 shutdown_connections 2024-04-26T00:20:53.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.703+0000 7ff05cb3a640 1 -- 172.21.15.37:0/4217385876 wait complete. 2024-04-26T00:20:53.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff05cb3a640 10 monclient: build_initial_monmap 2024-04-26T00:20:53.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff05cb3a640 10 monclient: monmap: 2024-04-26T00:20:53.699 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:53.699 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:53.699 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:53.699 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:53.699 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:53.699 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:53.699 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:53.699 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:53.699 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:53.700 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:53.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff05cb3a640 1 Processor -- start 2024-04-26T00:20:53.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff05cb3a640 1 -- start start 2024-04-26T00:20:53.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff05cb3a640 10 monclient: init 2024-04-26T00:20:53.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff05cb3a640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:53.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff05cb3a640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:20:53.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff05cb3a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff058136f20 0x7ff058134d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:53.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff05cb3a640 10 monclient(hunting): picked mon.noname-a con 0x7ff058136f20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:53.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff05cb3a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff058137300 0x7ff058137ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:53.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff05cb3a640 10 monclient(hunting): picked mon.noname-c con 0x7ff058137300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:53.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff05cb3a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff05813c450 0x7ff058138430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:53.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff05cb3a640 10 monclient(hunting): picked mon.noname-b con 0x7ff05813c450 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:53.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff05cb3a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:53.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff05cb3a640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff058141930 con 0x7ff058136f20 2024-04-26T00:20:53.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff05cb3a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:53.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff05cb3a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff0581417b0 con 0x7ff05813c450 2024-04-26T00:20:53.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff05cb3a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:53.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff05cb3a640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff058141ab0 con 0x7ff058137300 2024-04-26T00:20:53.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff05cb3a640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:53.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff05cb3a640 10 monclient(hunting): authenticate will time out at 1035.013623s 2024-04-26T00:20:53.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff055d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff058137300 0x7ff058137ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:53.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff05659c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff058136f20 0x7ff058134d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:53.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff05659c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff058136f20 0x7ff058134d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:37484/0 (socket says 172.21.15.37:37484) 2024-04-26T00:20:53.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff055d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff058137300 0x7ff058137ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:33552/0 (socket says 172.21.15.37:33552) 2024-04-26T00:20:53.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff05659c640 1 -- 172.21.15.37:0/690808160 learned_addr learned my addr 172.21.15.37:0/690808160 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:53.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff05659c640 10 monclient(hunting): get_auth_request con 0x7ff058136f20 auth_method 0 2024-04-26T00:20:53.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff05659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:53.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff05659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:53.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff05659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:53.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff055d9b640 10 monclient(hunting): get_auth_request con 0x7ff058137300 auth_method 0 2024-04-26T00:20:53.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff055d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:53.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff055d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:53.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff055d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:53.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff055d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:53.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff055d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:53.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff055d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:53.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff05659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:53.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff05659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:53.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff05659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:53.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff056d9d640 1 --2- 172.21.15.37:0/690808160 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff05813c450 0x7ff058138430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:53.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff056d9d640 10 monclient(hunting): get_auth_request con 0x7ff05813c450 auth_method 0 2024-04-26T00:20:53.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff056d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:53.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff056d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:53.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff056d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:53.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff055d9b640 10 monclient(hunting): handle_auth_done global_id 4424 payload 824 2024-04-26T00:20:53.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff055d9b640 1 -- 172.21.15.37:0/690808160 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff05813c450 msgr2=0x7ff058138430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:53.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff055d9b640 1 --2- 172.21.15.37:0/690808160 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff05813c450 0x7ff058138430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:53.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff055d9b640 1 -- 172.21.15.37:0/690808160 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff058136f20 msgr2=0x7ff058134d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:53.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff055d9b640 1 --2- 172.21.15.37:0/690808160 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff058136f20 0x7ff058134d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:53.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff055d9b640 10 monclient: _finish_hunting 0 2024-04-26T00:20:53.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff055d9b640 1 monclient: found mon.noname-c 2024-04-26T00:20:53.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff055d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:53.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff055d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:20:53.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff055d9b640 1 -- 172.21.15.37:0/690808160 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff05813a490 con 0x7ff058137300 2024-04-26T00:20:53.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.707+0000 7ff05659c640 1 --2- 172.21.15.37:0/690808160 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff058136f20 0x7ff058134d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:53.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff055d9b640 1 -- 172.21.15.37:0/690808160 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff058137300 msgr2=0x7ff058137ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:53.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff055d9b640 1 --2- 172.21.15.37:0/690808160 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff058137300 0x7ff058137ef0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff04c00c410 tx=0x7ff04c00c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:53.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff046ffd640 1 -- 172.21.15.37:0/690808160 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff04c017070 con 0x7ff058137300 2024-04-26T00:20:53.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff046ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:53.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff046ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:20:53.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff046ffd640 10 monclient: dump: 2024-04-26T00:20:53.705 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:53.705 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:53.705 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:53.705 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:53.705 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:53.705 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:53.705 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:53.705 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:53.705 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:53.706 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:53.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff046ffd640 10 monclient: _finish_auth 0 2024-04-26T00:20:53.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff046ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:20:53.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff046ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:53.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff046ffd640 1 -- 172.21.15.37:0/690808160 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff04c0070d0 con 0x7ff058137300 2024-04-26T00:20:53.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff046ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:53.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff046ffd640 1 -- 172.21.15.37:0/690808160 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff04c0074c0 con 0x7ff058137300 2024-04-26T00:20:53.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff046ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:53.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff046ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:53.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff046ffd640 10 monclient: dump: 2024-04-26T00:20:53.706 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:53.706 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:53.706 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:53.707 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:53.707 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:53.707 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:53.707 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:53.707 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:53.707 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:53.707 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:53.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff05cb3a640 5 monclient: authenticate success, global_id 4424 2024-04-26T00:20:53.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff05cb3a640 10 monclient: _renew_subs 2024-04-26T00:20:53.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff05cb3a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:53.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff054d99640 4 set_mon_vals no callback set 2024-04-26T00:20:53.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff05cb3a640 1 -- 172.21.15.37:0/690808160 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff05813b940 con 0x7ff058137300 2024-04-26T00:20:53.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff05cb3a640 10 monclient: _renew_subs 2024-04-26T00:20:53.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff05cb3a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:53.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff05cb3a640 1 -- 172.21.15.37:0/690808160 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff05813b380 con 0x7ff058137300 2024-04-26T00:20:53.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff05cb3a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:53.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff05cb3a640 1 --2- 172.21.15.37:0/690808160 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0581c6b40 0x7ff058138970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:53.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff05cb3a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:53.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff05cb3a640 1 -- 172.21.15.37:0/690808160 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff05813bb10 con 0x7ff0581c6b40 2024-04-26T00:20:53.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff05659c640 1 --2- 172.21.15.37:0/690808160 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0581c6b40 0x7ff058138970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:53.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff05659c640 10 monclient: get_auth_request con 0x7ff0581c6b40 auth_method 0 2024-04-26T00:20:53.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff05659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:53.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff05659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:53.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff05659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:53.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff046ffd640 1 -- 172.21.15.37:0/690808160 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff04c01a070 con 0x7ff058137300 2024-04-26T00:20:53.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff05659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:53.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff05659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:53.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff05659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:53.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff05659c640 10 monclient(hunting): handle_auth_done global_id 4434 payload 824 2024-04-26T00:20:53.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff05659c640 1 -- 172.21.15.37:0/690808160 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff05813b580 con 0x7ff0581c6b40 2024-04-26T00:20:53.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff046ffd640 1 --2- 172.21.15.37:0/690808160 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff038035eb0 0x7ff038038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:53.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff046ffd640 1 -- 172.21.15.37:0/690808160 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(44..44 src has 1..44) ==== 4019+0+0 (secure 0 0 0) 0x7ff04c047f60 con 0x7ff058137300 2024-04-26T00:20:53.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff056d9d640 1 --2- 172.21.15.37:0/690808160 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff038035eb0 0x7ff038038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:53.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.711+0000 7ff056d9d640 10 monclient: get_auth_request con 0x7ff038035eb0 auth_method 0 2024-04-26T00:20:53.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.715+0000 7ff05659c640 1 -- 172.21.15.37:0/690808160 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0581c6b40 msgr2=0x7ff058138970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:53.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.715+0000 7ff05659c640 1 --2- 172.21.15.37:0/690808160 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0581c6b40 0x7ff058138970 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff048009ec0 tx=0x7ff04800f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:53.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.715+0000 7ff046ffd640 1 -- 172.21.15.37:0/690808160 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff048004020 con 0x7ff0581c6b40 2024-04-26T00:20:53.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.715+0000 7ff046ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:53.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.715+0000 7ff046ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:53.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.715+0000 7ff046ffd640 10 monclient: dump: 2024-04-26T00:20:53.710 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:53.710 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:53.710 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:53.710 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:53.710 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:53.710 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:53.710 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:53.710 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:53.710 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:53.711 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:53.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.715+0000 7ff046ffd640 1 -- 172.21.15.37:0/690808160 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff048004020 con 0x7ff0581c6b40 2024-04-26T00:20:53.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.715+0000 7ff046ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:53.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.715+0000 7ff046ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:53.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.715+0000 7ff046ffd640 1 -- 172.21.15.37:0/690808160 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0581c6b40 msgr2=0x7ff058138970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:53.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.715+0000 7ff046ffd640 1 --2- 172.21.15.37:0/690808160 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0581c6b40 0x7ff058138970 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff048009ec0 tx=0x7ff04800f610 comp rx=0 tx=0).stop 2024-04-26T00:20:53.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.715+0000 7ff046ffd640 1 -- 172.21.15.37:0/690808160 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0581c6b40 msgr2=0x7ff058138970 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:20:53.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.715+0000 7ff046ffd640 1 --2- 172.21.15.37:0/690808160 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0581c6b40 0x7ff058138970 secure :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff048009ec0 tx=0x7ff04800f610 comp rx=0 tx=0).stop 2024-04-26T00:20:53.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.715+0000 7ff056d9d640 1 --2- 172.21.15.37:0/690808160 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff038035eb0 0x7ff038038370 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff05806e480 tx=0x7ff0500a7000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:53.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.719+0000 7ff05cb3a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:20:53.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.719+0000 7ff05cb3a640 1 --2- 172.21.15.37:0/690808160 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff058138eb0 0x7ff05809e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:53.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.719+0000 7ff05cb3a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:53.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.719+0000 7ff05cb3a640 1 -- 172.21.15.37:0/690808160 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff0581392c0 con 0x7ff058138eb0 2024-04-26T00:20:53.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.719+0000 7ff05659c640 1 --2- 172.21.15.37:0/690808160 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff058138eb0 0x7ff05809e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:53.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.719+0000 7ff05659c640 10 monclient: get_auth_request con 0x7ff058138eb0 auth_method 0 2024-04-26T00:20:53.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.719+0000 7ff05659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:53.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.719+0000 7ff05659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:53.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.719+0000 7ff05659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:53.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.719+0000 7ff05659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:53.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.719+0000 7ff05659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:53.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.719+0000 7ff05659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:53.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.719+0000 7ff05659c640 10 monclient(hunting): handle_auth_done global_id 4437 payload 824 2024-04-26T00:20:53.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.719+0000 7ff05659c640 1 -- 172.21.15.37:0/690808160 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff05804f510 con 0x7ff058138eb0 2024-04-26T00:20:53.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.723+0000 7ff05659c640 1 -- 172.21.15.37:0/690808160 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff058138eb0 msgr2=0x7ff05809e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:53.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.723+0000 7ff05659c640 1 --2- 172.21.15.37:0/690808160 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff058138eb0 0x7ff05809e300 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff04801b000 tx=0x7ff0480046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:53.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.723+0000 7ff046ffd640 1 -- 172.21.15.37:0/690808160 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff048003a30 con 0x7ff058138eb0 2024-04-26T00:20:53.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.723+0000 7ff046ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:53.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.723+0000 7ff046ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:53.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.723+0000 7ff046ffd640 10 monclient: dump: 2024-04-26T00:20:53.714 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:53.714 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:53.714 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:53.715 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:53.715 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:53.715 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:53.715 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:53.715 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:53.715 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:53.715 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:53.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.723+0000 7ff046ffd640 1 -- 172.21.15.37:0/690808160 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4888 (secure 0 0 0) 0x7ff048003a30 con 0x7ff058138eb0 2024-04-26T00:20:53.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.723+0000 7ff046ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:20:53.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.723+0000 7ff046ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:53.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.723+0000 7ff046ffd640 1 -- 172.21.15.37:0/690808160 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff058138eb0 msgr2=0x7ff05809e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:53.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.723+0000 7ff046ffd640 1 --2- 172.21.15.37:0/690808160 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff058138eb0 0x7ff05809e300 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff04801b000 tx=0x7ff0480046d0 comp rx=0 tx=0).stop 2024-04-26T00:20:53.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.723+0000 7ff046ffd640 1 -- 172.21.15.37:0/690808160 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff058138eb0 msgr2=0x7ff05809e300 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:20:53.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.723+0000 7ff046ffd640 1 --2- 172.21.15.37:0/690808160 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff058138eb0 0x7ff05809e300 secure :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff04801b000 tx=0x7ff0480046d0 comp rx=0 tx=0).stop 2024-04-26T00:20:53.716 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:20:53.716 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:20:53.716 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:20:53.716 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:20:53.716 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 12, 2024-04-26T00:20:53.716 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 70827, 2024-04-26T00:20:53.716 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:20:53.716 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:20:53.717 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:20:53.717 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:20:53.717 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:53.717 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 3, 2024-04-26T00:20:53.717 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:20:53.717 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:20:53.717 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:20:53.717 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:20:53.717 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:20:53.717 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:20:53.717 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:20:53.717 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:20:53.717 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:20:53.717 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:20:53.718 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:20:53.718 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:20:53.718 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:20:53.718 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:20:53.718 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:53.718 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:20:53.718 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:20:53.718 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:20:53.718 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:20:53.719 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:20:53.719 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:20:53.719 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:20:53.719 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:20:53.719 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:20:53.719 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:20:53.719 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:20:53.719 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:20:53.719 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:20:53.719 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:53.719 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:53.720 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:20:53.720 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:20:53.720 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:20:53.720 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:20:53.720 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:20:53.720 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:20:53.720 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:20:53.720 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:20:53.720 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:20:53.720 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:20:53.720 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:20:53.721 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:20:53.721 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:20:53.721 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:20:53.721 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:20:53.721 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:20:53.721 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:20:53.721 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:20:53.721 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:20:53.721 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:20:53.721 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:20:53.721 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:20:53.721 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:20:53.721 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:20:53.721 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:20:53.721 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:20:53.722 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:20:53.722 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:20:53.722 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:53.722 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:20:53.722 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:53.722 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:20:53.722 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:53.722 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:20:53.723 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:20:53.723 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:20:53.723 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:20:53.723 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:53.723 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:20:53.723 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:20:53.723 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:53.723 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:53.723 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:53.723 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:20:53.723 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:20:53.723 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:53.723 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:53.723 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:53.724 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:53.724 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:20:53.724 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:20:53.724 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:20:53.724 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:20:53.724 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:20:53.724 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:53.724 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:53.724 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:20:53.724 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:20:53.724 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:20:53.724 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:20:53.724 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:53.725 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:20:53.725 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:20:53.725 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:53.725 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:53.725 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:53.725 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:20:53.725 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:20:53.725 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:53.725 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:53.725 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:53.725 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:53.725 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:20:53.725 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:20:53.725 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:20:53.726 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:20:53.726 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:20:53.726 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:53.726 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:53.726 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:20:53.726 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:20:53.726 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:20:53.726 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:20:53.726 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:53.726 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:20:53.726 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:20:53.726 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:53.726 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:53.726 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:53.727 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:20:53.727 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:20:53.727 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:53.727 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:53.727 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:53.727 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:53.727 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:20:53.727 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:20:53.728 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:20:53.728 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:20:53.728 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:20:53.728 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:53.728 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:53.728 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:53.728 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:20:53.728 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:20:53.728 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:53.728 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:20:53.728 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:20:53.729 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:20:53.729 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:53.729 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:53.729 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:20:53.729 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:53.729 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:20:53.729 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:20:53.729 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:20:53.729 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:53.729 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:53.730 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:20:53.730 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:53.730 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:20:53.730 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:20:53.730 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:20:53.730 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:53.730 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:53.730 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:20:53.730 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:53.731 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:20:53.731 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:20:53.731 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:20:53.731 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:53.731 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:53.731 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:53.731 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:20:53.731 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:20:53.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.723+0000 7ff05cb3a640 1 -- 172.21.15.37:0/690808160 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff038035eb0 msgr2=0x7ff038038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:53.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.723+0000 7ff05cb3a640 1 --2- 172.21.15.37:0/690808160 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff038035eb0 0x7ff038038370 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff05806e480 tx=0x7ff0500a7000 comp rx=0 tx=0).stop 2024-04-26T00:20:53.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.723+0000 7ff05cb3a640 10 monclient: shutdown 2024-04-26T00:20:53.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.723+0000 7ff05cb3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:53.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.723+0000 7ff05cb3a640 1 -- 172.21.15.37:0/690808160 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff058137300 msgr2=0x7ff058137ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:53.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.723+0000 7ff05cb3a640 1 --2- 172.21.15.37:0/690808160 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff058137300 0x7ff058137ef0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff04c00c410 tx=0x7ff04c00c8d0 comp rx=0 tx=0).stop 2024-04-26T00:20:53.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.723+0000 7ff056d9d640 1 -- 172.21.15.37:0/690808160 reap_dead start 2024-04-26T00:20:53.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.723+0000 7ff056d9d640 1 -- 172.21.15.37:0/690808160 reap_dead start 2024-04-26T00:20:53.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.723+0000 7ff05cb3a640 1 -- 172.21.15.37:0/690808160 shutdown_connections 2024-04-26T00:20:53.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.723+0000 7ff05cb3a640 1 -- 172.21.15.37:0/690808160 >> 172.21.15.37:0/690808160 conn(0x7ff0580a61b0 msgr2=0x7ff0580a7a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:53.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.723+0000 7ff05cb3a640 1 -- 172.21.15.37:0/690808160 shutdown_connections 2024-04-26T00:20:53.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.723+0000 7ff05cb3a640 1 -- 172.21.15.37:0/690808160 wait complete. 2024-04-26T00:20:53.734 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:20:53.734 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:20:53.734 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:20:53.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.875+0000 7f876cef6640 10 monclient: get_monmap_and_config 2024-04-26T00:20:53.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.875+0000 7f876cef6640 10 monclient: build_initial_monmap 2024-04-26T00:20:53.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.875+0000 7f876cef6640 10 monclient: monmap: 2024-04-26T00:20:53.853 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:53.854 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:53.854 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:53.876998+0000 2024-04-26T00:20:53.854 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:53.876998+0000 2024-04-26T00:20:53.854 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:53.854 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:53.854 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:53.855 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:53.855 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:53.855 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:53.855 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:53.855 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:53.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.875+0000 7f876cef6640 1 Processor -- start 2024-04-26T00:20:53.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.875+0000 7f876cef6640 1 -- start start 2024-04-26T00:20:53.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.875+0000 7f876cef6640 10 monclient: init 2024-04-26T00:20:53.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.875+0000 7f876cef6640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:53.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.875+0000 7f876cef6640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-04-26T00:20:53.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.875+0000 7f876cef6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f87680a0300 addr v1:172.21.15.45:6789/0 2024-04-26T00:20:53.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.875+0000 7f876cef6640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f876809fc50 0x7f8768139ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:53.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.875+0000 7f876cef6640 10 monclient(hunting): picked mon.noname-a con 0x7f876809fc50 addr v2:172.21.15.37:3300/0 2024-04-26T00:20:53.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.875+0000 7f876cef6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f876813a400 addr v1:172.21.15.37:6789/0 2024-04-26T00:20:53.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.875+0000 7f876cef6640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f876809c190 con 0x7f876813a400 2024-04-26T00:20:53.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.875+0000 7f876cef6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87680a1110 con 0x7f87680a0300 2024-04-26T00:20:53.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.875+0000 7f876cef6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:53.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.875+0000 7f876cef6640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f87680a16d0 con 0x7f876809fc50 2024-04-26T00:20:53.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.875+0000 7f876cef6640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:53.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.875+0000 7f876cef6640 10 monclient(hunting): authenticate will time out at 1035.182645s 2024-04-26T00:20:53.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.875+0000 7f8766d9d640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f876813a400 0x7f876813c7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:54972/0 (socket says 172.21.15.37:54972) 2024-04-26T00:20:53.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.875+0000 7f8766d9d640 1 -- 172.21.15.37:0/3541423952 learned_addr learned my addr 172.21.15.37:0/3541423952 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:20:53.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.875+0000 7f8765d9b640 1 --2- 172.21.15.37:0/3541423952 >> v2:172.21.15.37:3300/0 conn(0x7f876809fc50 0x7f8768139ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:53.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.875+0000 7f8765d9b640 10 monclient(hunting): get_auth_request con 0x7f876809fc50 auth_method 0 2024-04-26T00:20:53.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.875+0000 7f8765d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:53.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.875+0000 7f8765d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:53.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.875+0000 7f8765d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:53.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.875+0000 7f8765d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:53.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.875+0000 7f8765d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:53.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.875+0000 7f8765d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:53.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.875+0000 7f8765d9b640 10 monclient(hunting): handle_auth_done global_id 4440 payload 274 2024-04-26T00:20:53.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.875+0000 7f8765d9b640 1 -- 172.21.15.37:0/3541423952 >> v1:172.21.15.45:6789/0 conn(0x7f87680a0300 legacy=0x7f87680a06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:53.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.875+0000 7f8765d9b640 1 -- 172.21.15.37:0/3541423952 >> v1:172.21.15.37:6789/0 conn(0x7f876813a400 legacy=0x7f876813c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:53.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.875+0000 7f8765d9b640 10 monclient: _finish_hunting 0 2024-04-26T00:20:53.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.875+0000 7f8765d9b640 1 monclient: found mon.noname-a 2024-04-26T00:20:53.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.875+0000 7f8765d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:53.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.875+0000 7f8765d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:53.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.875+0000 7f8765d9b640 1 -- 172.21.15.37:0/3541423952 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f876809f2a0 con 0x7f876809fc50 2024-04-26T00:20:53.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.879+0000 7f8765d9b640 1 -- 172.21.15.37:0/3541423952 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f876809fc50 msgr2=0x7f8768139ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:53.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.879+0000 7f8765d9b640 1 --2- 172.21.15.37:0/3541423952 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f876809fc50 0x7f8768139ec0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f876809f4d0 tx=0x7f875801c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50637342a0bbb1c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:53.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.879+0000 7f876559a640 1 -- 172.21.15.37:0/3541423952 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87580b0070 con 0x7f876809fc50 2024-04-26T00:20:53.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.879+0000 7f876559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:53.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.879+0000 7f876559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:53.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.879+0000 7f876559a640 10 monclient: dump: 2024-04-26T00:20:53.882 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:53.882 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:53.883 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:53.883 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:53.883 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:53.883 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:53.883 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:53.883 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:53.884 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:53.884 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:53.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.879+0000 7f876559a640 10 monclient: _finish_auth 0 2024-04-26T00:20:53.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.879+0000 7f876559a640 10 monclient: _check_auth_tickets 2024-04-26T00:20:53.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.879+0000 7f876559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:53.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.879+0000 7f876559a640 1 -- 172.21.15.37:0/3541423952 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87580ab600 con 0x7f876809fc50 2024-04-26T00:20:53.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.879+0000 7f876cef6640 5 monclient: authenticate success, global_id 4440 2024-04-26T00:20:53.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.879+0000 7f876cef6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:20:53.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.879+0000 7f876559a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:53.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.879+0000 7f876cef6640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:53.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.879+0000 7f876559a640 1 -- 172.21.15.37:0/3541423952 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87580ab9d0 con 0x7f876809fc50 2024-04-26T00:20:53.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.879+0000 7f876cef6640 4 set_mon_vals no callback set 2024-04-26T00:20:53.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.879+0000 7f876559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:53.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.879+0000 7f876559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:53.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.879+0000 7f876559a640 10 monclient: dump: 2024-04-26T00:20:53.887 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:53.887 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:53.887 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:53.887 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:53.887 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:53.887 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:53.888 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:53.888 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:53.888 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:53.888 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:53.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876cef6640 10 monclient: shutdown 2024-04-26T00:20:53.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876cef6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:53.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876cef6640 1 -- 172.21.15.37:0/3541423952 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f876809fc50 msgr2=0x7f8768139ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:53.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876cef6640 1 --2- 172.21.15.37:0/3541423952 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f876809fc50 0x7f8768139ec0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f876809f4d0 tx=0x7f875801c370 comp rx=0 tx=0).stop 2024-04-26T00:20:53.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876cef6640 1 -- 172.21.15.37:0/3541423952 shutdown_connections 2024-04-26T00:20:53.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876cef6640 1 --2- 172.21.15.37:0/3541423952 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f876809fc50 0x7f8768139ec0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:53.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876cef6640 1 -- 172.21.15.37:0/3541423952 >> 172.21.15.37:0/3541423952 conn(0x7f87680a1d50 msgr2=0x7f87680a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:53.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876cef6640 1 -- 172.21.15.37:0/3541423952 shutdown_connections 2024-04-26T00:20:53.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876cef6640 1 -- 172.21.15.37:0/3541423952 wait complete. 2024-04-26T00:20:53.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876cef6640 10 monclient: build_initial_monmap 2024-04-26T00:20:53.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876cef6640 10 monclient: monmap: 2024-04-26T00:20:53.890 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:53.890 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:53.890 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:53.891 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:53.891 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:53.891 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:53.891 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:53.891 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:53.891 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:53.891 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:53.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876cef6640 1 Processor -- start 2024-04-26T00:20:53.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876cef6640 1 -- start start 2024-04-26T00:20:53.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876cef6640 10 monclient: init 2024-04-26T00:20:53.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876cef6640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:53.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876cef6640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:20:53.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876cef6640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f876809fc50 0x7f876811b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:53.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876cef6640 10 monclient(hunting): picked mon.noname-a con 0x7f876809fc50 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:53.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876cef6640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f87680a0300 0x7f8768138fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:53.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876cef6640 10 monclient(hunting): picked mon.noname-c con 0x7f87680a0300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:53.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876cef6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f876813a400 0x7f876811c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:53.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876cef6640 10 monclient(hunting): picked mon.noname-b con 0x7f876813a400 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:53.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876cef6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:53.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876cef6640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f8768140ac0 con 0x7f876809fc50 2024-04-26T00:20:53.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876cef6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:53.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876cef6640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8768140940 con 0x7f876813a400 2024-04-26T00:20:53.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876cef6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:53.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876cef6640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8768140c40 con 0x7f87680a0300 2024-04-26T00:20:53.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876cef6640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:53.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876cef6640 10 monclient(hunting): authenticate will time out at 1035.189916s 2024-04-26T00:20:53.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876659c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f876809fc50 0x7f876811b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:53.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f8765d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f87680a0300 0x7f8768138fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:53.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876659c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f876809fc50 0x7f876811b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:37536/0 (socket says 172.21.15.37:37536) 2024-04-26T00:20:53.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f8765d9b640 10 monclient(hunting): get_auth_request con 0x7f87680a0300 auth_method 0 2024-04-26T00:20:53.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876659c640 1 -- 172.21.15.37:0/2031708634 learned_addr learned my addr 172.21.15.37:0/2031708634 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:53.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f8765d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:53.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f8765d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:53.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f8765d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:53.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876659c640 10 monclient(hunting): get_auth_request con 0x7f876809fc50 auth_method 0 2024-04-26T00:20:53.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:53.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:53.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:53.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f8765d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:53.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f8765d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:53.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f8765d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:53.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:53.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:53.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:53.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f8766d9d640 1 --2- 172.21.15.37:0/2031708634 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f876813a400 0x7f876811c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:53.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f8766d9d640 10 monclient(hunting): get_auth_request con 0x7f876813a400 auth_method 0 2024-04-26T00:20:53.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f8766d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:53.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f8766d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:53.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f8766d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:53.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876659c640 10 monclient(hunting): handle_auth_done global_id 4443 payload 824 2024-04-26T00:20:53.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876659c640 1 -- 172.21.15.37:0/2031708634 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f87680a0300 msgr2=0x7f8768138fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:53.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876659c640 1 --2- 172.21.15.37:0/2031708634 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f87680a0300 0x7f8768138fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:53.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876659c640 1 -- 172.21.15.37:0/2031708634 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f876813a400 msgr2=0x7f876811c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:53.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876659c640 1 --2- 172.21.15.37:0/2031708634 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f876813a400 0x7f876811c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:53.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876659c640 10 monclient: _finish_hunting 0 2024-04-26T00:20:53.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876659c640 1 monclient: found mon.noname-a 2024-04-26T00:20:53.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:53.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:53.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f876659c640 1 -- 172.21.15.37:0/2031708634 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f876813a990 con 0x7f876809fc50 2024-04-26T00:20:53.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.883+0000 7f8765d9b640 1 --2- 172.21.15.37:0/2031708634 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f87680a0300 0x7f8768138fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:53.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f876659c640 1 -- 172.21.15.37:0/2031708634 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f876809fc50 msgr2=0x7f876811b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:53.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f876659c640 1 --2- 172.21.15.37:0/2031708634 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f876809fc50 0x7f876811b8a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f875c00c450 tx=0x7f875c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:53.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f8756ffd640 1 -- 172.21.15.37:0/2031708634 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f875c012a40 con 0x7f876809fc50 2024-04-26T00:20:53.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f8756ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:53.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f8756ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:53.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f8756ffd640 10 monclient: dump: 2024-04-26T00:20:53.902 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:53.903 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:53.903 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:53.903 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:53.903 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:53.903 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:53.903 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:53.903 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:53.903 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:53.903 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:53.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f8756ffd640 10 monclient: _finish_auth 0 2024-04-26T00:20:53.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f8756ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:20:53.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f8756ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:53.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f8756ffd640 1 -- 172.21.15.37:0/2031708634 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f875c017070 con 0x7f876809fc50 2024-04-26T00:20:53.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f876cef6640 5 monclient: authenticate success, global_id 4443 2024-04-26T00:20:53.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f876cef6640 10 monclient: _renew_subs 2024-04-26T00:20:53.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f876cef6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:53.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f876cef6640 1 -- 172.21.15.37:0/2031708634 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f876809f550 con 0x7f876809fc50 2024-04-26T00:20:53.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f8756ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:53.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f8756ffd640 1 -- 172.21.15.37:0/2031708634 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f875c012e40 con 0x7f876809fc50 2024-04-26T00:20:53.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f8756ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:53.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f8756ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:53.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f8756ffd640 10 monclient: dump: 2024-04-26T00:20:53.904 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:53.904 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:53.905 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:53.905 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:53.905 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:53.905 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:53.905 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:53.905 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:53.905 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:53.905 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:53.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f8764d99640 4 set_mon_vals no callback set 2024-04-26T00:20:53.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f876cef6640 10 monclient: _renew_subs 2024-04-26T00:20:53.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f876cef6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:53.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f876cef6640 1 -- 172.21.15.37:0/2031708634 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8768135580 con 0x7f876809fc50 2024-04-26T00:20:53.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f8756ffd640 1 -- 172.21.15.37:0/2031708634 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f875c01c460 con 0x7f876809fc50 2024-04-26T00:20:53.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f876cef6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:53.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f876cef6640 1 --2- 172.21.15.37:0/2031708634 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f87681cd9c0 0x7f876802c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:53.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f876cef6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:53.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f876cef6640 1 -- 172.21.15.37:0/2031708634 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f876811e560 con 0x7f87681cd9c0 2024-04-26T00:20:53.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f8756ffd640 1 --2- 172.21.15.37:0/2031708634 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8748035e80 0x7f8748038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:53.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f8765d9b640 1 --2- 172.21.15.37:0/2031708634 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f87681cd9c0 0x7f876802c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:53.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f8765d9b640 10 monclient: get_auth_request con 0x7f87681cd9c0 auth_method 0 2024-04-26T00:20:53.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f8765d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:53.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f8765d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:53.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f8765d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:53.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f8756ffd640 1 -- 172.21.15.37:0/2031708634 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 4019+0+0 (secure 0 0 0) 0x7f875c0488b0 con 0x7f876809fc50 2024-04-26T00:20:53.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f8765d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:53.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f8765d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:53.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f8765d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:53.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f8765d9b640 10 monclient(hunting): handle_auth_done global_id 4430 payload 824 2024-04-26T00:20:53.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f8765d9b640 1 -- 172.21.15.37:0/2031708634 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f876804f510 con 0x7f87681cd9c0 2024-04-26T00:20:53.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f8766d9d640 1 --2- 172.21.15.37:0/2031708634 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8748035e80 0x7f8748038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:53.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.887+0000 7f8766d9d640 10 monclient: get_auth_request con 0x7f8748035e80 auth_method 0 2024-04-26T00:20:53.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.891+0000 7f8765d9b640 1 -- 172.21.15.37:0/2031708634 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f87681cd9c0 msgr2=0x7f876802c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:53.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.891+0000 7f8765d9b640 1 --2- 172.21.15.37:0/2031708634 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f87681cd9c0 0x7f876802c000 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f87580ad000 tx=0x7f8758019110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:53.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.891+0000 7f8756ffd640 1 -- 172.21.15.37:0/2031708634 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87580b0040 con 0x7f87681cd9c0 2024-04-26T00:20:53.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.891+0000 7f8756ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:53.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.891+0000 7f8756ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:53.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.891+0000 7f8756ffd640 10 monclient: dump: 2024-04-26T00:20:53.907 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:53.907 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:53.907 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:53.907 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:53.907 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:53.907 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:53.907 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:53.908 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:53.908 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:53.908 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:53.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.891+0000 7f8756ffd640 1 -- 172.21.15.37:0/2031708634 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f87580b1040 con 0x7f87681cd9c0 2024-04-26T00:20:53.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.891+0000 7f8756ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:53.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.891+0000 7f8756ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:53.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.891+0000 7f8756ffd640 1 -- 172.21.15.37:0/2031708634 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f87681cd9c0 msgr2=0x7f876802c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:53.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.891+0000 7f8756ffd640 1 --2- 172.21.15.37:0/2031708634 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f87681cd9c0 0x7f876802c000 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f87580ad000 tx=0x7f8758019110 comp rx=0 tx=0).stop 2024-04-26T00:20:53.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.891+0000 7f8756ffd640 1 -- 172.21.15.37:0/2031708634 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f87681cd9c0 msgr2=0x7f876802c000 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:20:53.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.891+0000 7f8756ffd640 1 --2- 172.21.15.37:0/2031708634 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f87681cd9c0 0x7f876802c000 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f87580ad000 tx=0x7f8758019110 comp rx=0 tx=0).stop 2024-04-26T00:20:53.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.891+0000 7f8766d9d640 1 --2- 172.21.15.37:0/2031708634 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8748035e80 0x7f8748038340 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f876811df70 tx=0x7f8760009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:53.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.895+0000 7f876cef6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:20:53.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.895+0000 7f876cef6640 1 --2- 172.21.15.37:0/2031708634 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f87681cdda0 0x7f876802c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:53.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.895+0000 7f876cef6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:53.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.895+0000 7f876cef6640 1 -- 172.21.15.37:0/2031708634 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8768139ff0 con 0x7f87681cdda0 2024-04-26T00:20:53.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.895+0000 7f8765d9b640 1 --2- 172.21.15.37:0/2031708634 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f87681cdda0 0x7f876802c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:53.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.895+0000 7f8765d9b640 10 monclient: get_auth_request con 0x7f87681cdda0 auth_method 0 2024-04-26T00:20:53.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.895+0000 7f8765d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:53.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.895+0000 7f8765d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:53.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.895+0000 7f8765d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:53.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.895+0000 7f8765d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:53.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.895+0000 7f8765d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:53.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.895+0000 7f8765d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:53.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.895+0000 7f8765d9b640 10 monclient(hunting): handle_auth_done global_id 4433 payload 824 2024-04-26T00:20:53.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.895+0000 7f8765d9b640 1 -- 172.21.15.37:0/2031708634 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f876813d580 con 0x7f87681cdda0 2024-04-26T00:20:53.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.899+0000 7f8765d9b640 1 -- 172.21.15.37:0/2031708634 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f87681cdda0 msgr2=0x7f876802c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:53.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.899+0000 7f8765d9b640 1 --2- 172.21.15.37:0/2031708634 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f87681cdda0 0x7f876802c540 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f875801c3b0 tx=0x7f8758009190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:53.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.899+0000 7f8756ffd640 1 -- 172.21.15.37:0/2031708634 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87580b0050 con 0x7f87681cdda0 2024-04-26T00:20:53.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.899+0000 7f8756ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:53.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.899+0000 7f8756ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:53.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.899+0000 7f8756ffd640 10 monclient: dump: 2024-04-26T00:20:53.910 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:53.910 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:53.910 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:53.910 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:53.910 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:53.910 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:53.910 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:53.910 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:53.910 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:53.910 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:53.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.899+0000 7f8756ffd640 1 -- 172.21.15.37:0/2031708634 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4716 (secure 0 0 0) 0x7f87580c6070 con 0x7f87681cdda0 2024-04-26T00:20:53.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.899+0000 7f8756ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:20:53.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.899+0000 7f8756ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:53.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.899+0000 7f8756ffd640 1 -- 172.21.15.37:0/2031708634 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f87681cdda0 msgr2=0x7f876802c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:53.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.899+0000 7f8756ffd640 1 --2- 172.21.15.37:0/2031708634 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f87681cdda0 0x7f876802c540 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f875801c3b0 tx=0x7f8758009190 comp rx=0 tx=0).stop 2024-04-26T00:20:53.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.899+0000 7f8756ffd640 1 -- 172.21.15.37:0/2031708634 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f87681cdda0 msgr2=0x7f876802c540 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:20:53.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.899+0000 7f8756ffd640 1 --2- 172.21.15.37:0/2031708634 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f87681cdda0 0x7f876802c540 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f875801c3b0 tx=0x7f8758009190 comp rx=0 tx=0).stop 2024-04-26T00:20:53.911 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:20:53.911 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:20:53.911 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:20:53.911 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:20:53.911 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 12, 2024-04-26T00:20:53.912 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 71007, 2024-04-26T00:20:53.912 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:20:53.912 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:20:53.912 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:20:53.912 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:20:53.912 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:53.912 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 3, 2024-04-26T00:20:53.912 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:20:53.912 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:20:53.912 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:20:53.912 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:20:53.912 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:20:53.912 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:20:53.912 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:20:53.912 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:20:53.913 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:20:53.913 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:20:53.913 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:20:53.913 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:20:53.913 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:20:53.913 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:20:53.913 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:53.913 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:20:53.913 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:20:53.913 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:20:53.913 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:20:53.913 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:20:53.913 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:20:53.913 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:20:53.913 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:20:53.913 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:20:53.914 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:20:53.914 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:20:53.914 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:20:53.914 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:20:53.914 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:53.914 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:53.914 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:20:53.914 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:20:53.914 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:20:53.914 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:20:53.914 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:20:53.914 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:20:53.914 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:20:53.914 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:20:53.915 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:20:53.915 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:20:53.915 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:20:53.915 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:20:53.915 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:20:53.915 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:20:53.915 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:20:53.915 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:20:53.915 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:20:53.915 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:20:53.915 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:20:53.915 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:20:53.915 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:20:53.915 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:20:53.915 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:20:53.916 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:20:53.916 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:20:53.916 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:20:53.916 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:20:53.916 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:20:53.916 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:53.916 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:20:53.916 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:53.916 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:20:53.916 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:53.916 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:20:53.916 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:20:53.916 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:20:53.916 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:20:53.916 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:53.917 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:20:53.917 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:20:53.917 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:53.917 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:53.917 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:53.917 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:20:53.917 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:20:53.917 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:53.917 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:53.917 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:53.917 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:53.917 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:20:53.917 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:20:53.917 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:20:53.918 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:20:53.918 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:20:53.918 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:53.918 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:53.918 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:20:53.918 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:20:53.918 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:20:53.918 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:20:53.918 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:53.918 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:20:53.918 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:20:53.918 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:53.918 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:53.918 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:53.918 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:20:53.919 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:20:53.919 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:53.919 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:53.919 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:53.919 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:53.919 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:20:53.919 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:20:53.919 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:20:53.919 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:20:53.919 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:20:53.919 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:53.919 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:53.919 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:20:53.919 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:20:53.919 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:20:53.920 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:20:53.920 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:53.920 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:20:53.920 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:20:53.920 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:53.920 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:53.920 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:53.920 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:20:53.920 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:20:53.920 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:53.920 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:53.920 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:53.920 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:53.920 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:20:53.920 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:20:53.921 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:20:53.921 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:20:53.921 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:20:53.921 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:53.921 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:53.921 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:53.921 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:20:53.921 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:20:53.921 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:53.921 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:20:53.921 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:20:53.921 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:20:53.921 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:53.921 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:53.921 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:20:53.922 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:53.922 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:20:53.922 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:20:53.922 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:20:53.922 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:53.922 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:53.922 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:20:53.922 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:53.922 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:20:53.922 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:20:53.922 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:20:53.922 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:53.923 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:53.923 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:53.923 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:20:53.923 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:20:53.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.899+0000 7f876cef6640 1 -- 172.21.15.37:0/2031708634 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8748035e80 msgr2=0x7f8748038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:53.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.899+0000 7f876cef6640 1 --2- 172.21.15.37:0/2031708634 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8748035e80 0x7f8748038340 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f876811df70 tx=0x7f8760009040 comp rx=0 tx=0).stop 2024-04-26T00:20:53.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.899+0000 7f876cef6640 10 monclient: shutdown 2024-04-26T00:20:53.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.899+0000 7f876cef6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:53.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.899+0000 7f876cef6640 1 -- 172.21.15.37:0/2031708634 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f876809fc50 msgr2=0x7f876811b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:53.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.899+0000 7f876cef6640 1 --2- 172.21.15.37:0/2031708634 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f876809fc50 0x7f876811b8a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f875c00c450 tx=0x7f875c00c910 comp rx=0 tx=0).stop 2024-04-26T00:20:53.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.899+0000 7f8766d9d640 1 -- 172.21.15.37:0/2031708634 reap_dead start 2024-04-26T00:20:53.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.899+0000 7f8766d9d640 1 -- 172.21.15.37:0/2031708634 reap_dead start 2024-04-26T00:20:53.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.899+0000 7f876cef6640 1 -- 172.21.15.37:0/2031708634 shutdown_connections 2024-04-26T00:20:53.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.899+0000 7f876cef6640 1 -- 172.21.15.37:0/2031708634 >> 172.21.15.37:0/2031708634 conn(0x7f87680a1d50 msgr2=0x7f876813c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:53.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.899+0000 7f876cef6640 1 -- 172.21.15.37:0/2031708634 shutdown_connections 2024-04-26T00:20:53.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:53.899+0000 7f876cef6640 1 -- 172.21.15.37:0/2031708634 wait complete. 2024-04-26T00:20:53.925 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:20:53.925 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:20:53.925 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:20:53.926 INFO:tasks.rados.rados.0.smithi045.stdout:145: done (2 left) 2024-04-26T00:20:53.926 INFO:tasks.rados.rados.0.smithi045.stdout:147: done (1 left) 2024-04-26T00:20:53.926 INFO:tasks.rados.rados.0.smithi045.stdout:149: done (0 left) 2024-04-26T00:20:53.926 INFO:tasks.rados.rados.0.smithi045.stdout:150: rollback oid 20 current snap is 14 2024-04-26T00:20:53.926 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 20 to 13 2024-04-26T00:20:53.927 INFO:tasks.rados.rados.0.smithi045.stdout:151: read oid 38 snap 13 2024-04-26T00:20:53.927 INFO:tasks.rados.rados.0.smithi045.stdout:151: expect deleted 2024-04-26T00:20:53.927 INFO:tasks.rados.rados.0.smithi045.stdout:152: snap_remove snap 12 2024-04-26T00:20:54.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f489836b640 10 monclient: get_monmap_and_config 2024-04-26T00:20:54.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f489836b640 10 monclient: build_initial_monmap 2024-04-26T00:20:54.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f489836b640 10 monclient: monmap: 2024-04-26T00:20:54.049 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:54.049 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:54.049 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:54.073322+0000 2024-04-26T00:20:54.050 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:54.073322+0000 2024-04-26T00:20:54.050 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:54.050 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:54.050 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:54.050 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:54.050 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:54.050 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:54.050 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:54.051 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:54.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f489836b640 1 Processor -- start 2024-04-26T00:20:54.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f489836b640 1 -- start start 2024-04-26T00:20:54.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f489836b640 10 monclient: init 2024-04-26T00:20:54.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f489836b640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:54.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f489836b640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-04-26T00:20:54.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f489836b640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f48900a0300 0x7f489009fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:54.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f489836b640 10 monclient(hunting): picked mon.noname-a con 0x7f48900a0300 addr v2:172.21.15.37:3300/0 2024-04-26T00:20:54.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f489836b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f48900a06e0 addr v1:172.21.15.45:6789/0 2024-04-26T00:20:54.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f489836b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f48901352b0 0x7f489013cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:54.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f489836b640 10 monclient(hunting): picked mon.noname-c con 0x7f48901352b0 addr v2:172.21.15.45:3300/0 2024-04-26T00:20:54.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f489836b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f489009c190 con 0x7f48900a06e0 2024-04-26T00:20:54.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f489836b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:54.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f489836b640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f48900a13f0 con 0x7f48900a0300 2024-04-26T00:20:54.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f489836b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:54.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f489836b640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f48900a1110 con 0x7f48901352b0 2024-04-26T00:20:54.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f489836b640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:54.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f489836b640 10 monclient(hunting): authenticate will time out at 1035.379257s 2024-04-26T00:20:54.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f4896107640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f48900a0300 0x7f489009fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:54.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f4896107640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f48900a0300 0x7f489009fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:37550/0 (socket says 172.21.15.37:37550) 2024-04-26T00:20:54.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f4896107640 1 -- 172.21.15.37:0/2964251376 learned_addr learned my addr 172.21.15.37:0/2964251376 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:54.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f4896107640 10 monclient(hunting): get_auth_request con 0x7f48900a0300 auth_method 0 2024-04-26T00:20:54.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f4896107640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:54.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f4896107640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:54.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f4896107640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:54.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f4896107640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:54.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f4896107640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:54.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f4896107640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:54.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f4896908640 1 --2- 172.21.15.37:0/2964251376 >> v2:172.21.15.45:3300/0 conn(0x7f48901352b0 0x7f489013cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:54.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f4896908640 10 monclient(hunting): get_auth_request con 0x7f48901352b0 auth_method 0 2024-04-26T00:20:54.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f4896908640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:54.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f4896908640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:54.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f4896908640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:54.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f4896107640 10 monclient(hunting): handle_auth_done global_id 4446 payload 274 2024-04-26T00:20:54.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f4896107640 1 -- 172.21.15.37:0/2964251376 >> v2:172.21.15.45:3300/0 conn(0x7f48901352b0 msgr2=0x7f489013cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:54.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f4896107640 1 --2- 172.21.15.37:0/2964251376 >> v2:172.21.15.45:3300/0 conn(0x7f48901352b0 0x7f489013cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:54.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f4896107640 1 -- 172.21.15.37:0/2964251376 >> v1:172.21.15.45:6789/0 conn(0x7f48900a06e0 legacy=0x7f4890134ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:54.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f4896107640 10 monclient: _finish_hunting 0 2024-04-26T00:20:54.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f4896107640 1 monclient: found mon.noname-a 2024-04-26T00:20:54.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f4896107640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:54.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f4896107640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:54.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.071+0000 7f4896107640 1 -- 172.21.15.37:0/2964251376 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48900a16d0 con 0x7f48900a0300 2024-04-26T00:20:54.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4896107640 1 -- 172.21.15.37:0/2964251376 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f48900a0300 msgr2=0x7f489009fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:54.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4896107640 1 --2- 172.21.15.37:0/2964251376 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f48900a0300 0x7f489009fc50 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f489006de30 tx=0x7f488801c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bc2fa436c6d9233 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:54.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4895105640 1 -- 172.21.15.37:0/2964251376 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48880b2070 con 0x7f48900a0300 2024-04-26T00:20:54.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4895105640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:54.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4895105640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:54.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4895105640 10 monclient: dump: 2024-04-26T00:20:54.057 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:54.057 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:54.058 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:54.058 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:54.058 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:54.058 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:54.058 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:54.058 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:54.058 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:54.058 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:54.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4895105640 10 monclient: _finish_auth 0 2024-04-26T00:20:54.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4895105640 10 monclient: _check_auth_tickets 2024-04-26T00:20:54.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4895105640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:54.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4895105640 1 -- 172.21.15.37:0/2964251376 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48880ad680 con 0x7f48900a0300 2024-04-26T00:20:54.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4895105640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:54.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4895105640 1 -- 172.21.15.37:0/2964251376 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48880ada50 con 0x7f48900a0300 2024-04-26T00:20:54.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f489836b640 5 monclient: authenticate success, global_id 4446 2024-04-26T00:20:54.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f489836b640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:54.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f489836b640 4 set_mon_vals no callback set 2024-04-26T00:20:54.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4895105640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:54.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4895105640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:54.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4895105640 10 monclient: dump: 2024-04-26T00:20:54.059 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:54.059 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:54.059 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:54.059 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:54.059 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:54.059 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:54.059 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:54.059 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:54.060 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:54.060 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:54.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f489836b640 10 monclient: shutdown 2024-04-26T00:20:54.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f489836b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:54.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f489836b640 1 -- 172.21.15.37:0/2964251376 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f48900a0300 msgr2=0x7f489009fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:54.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f489836b640 1 --2- 172.21.15.37:0/2964251376 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f48900a0300 0x7f489009fc50 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f489006de30 tx=0x7f488801c3a0 comp rx=0 tx=0).stop 2024-04-26T00:20:54.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f489836b640 1 -- 172.21.15.37:0/2964251376 shutdown_connections 2024-04-26T00:20:54.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f489836b640 1 --2- 172.21.15.37:0/2964251376 >> v2:172.21.15.45:3300/0 conn(0x7f48901352b0 0x7f489013cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:54.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f489836b640 1 --2- 172.21.15.37:0/2964251376 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f48900a0300 0x7f489009fc50 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:54.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f489836b640 1 -- 172.21.15.37:0/2964251376 >> 172.21.15.37:0/2964251376 conn(0x7f48900a1d50 msgr2=0x7f48900a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:54.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f489836b640 1 -- 172.21.15.37:0/2964251376 shutdown_connections 2024-04-26T00:20:54.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f489836b640 1 -- 172.21.15.37:0/2964251376 wait complete. 2024-04-26T00:20:54.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f489836b640 10 monclient: build_initial_monmap 2024-04-26T00:20:54.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f489836b640 10 monclient: monmap: 2024-04-26T00:20:54.060 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:54.061 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:54.061 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:54.061 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:54.061 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:54.061 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:54.061 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:54.061 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:54.061 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:54.061 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:54.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f489836b640 1 Processor -- start 2024-04-26T00:20:54.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f489836b640 1 -- start start 2024-04-26T00:20:54.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f489836b640 10 monclient: init 2024-04-26T00:20:54.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f489836b640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:54.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f489836b640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:20:54.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f489836b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f48900a0300 0x7f48901cf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:54.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f489836b640 10 monclient(hunting): picked mon.noname-a con 0x7f48900a0300 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:54.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f489836b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48900a06e0 0x7f489012fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:54.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f489836b640 10 monclient(hunting): picked mon.noname-b con 0x7f48900a06e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:54.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f489836b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48901352b0 0x7f4890139ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:54.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f489836b640 10 monclient(hunting): picked mon.noname-c con 0x7f48901352b0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:54.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f489836b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:54.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f489836b640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f489013fdc0 con 0x7f48900a0300 2024-04-26T00:20:54.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f489836b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:54.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f489836b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f489013fc40 con 0x7f48900a06e0 2024-04-26T00:20:54.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f489836b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:54.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f489836b640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f489013ff40 con 0x7f48901352b0 2024-04-26T00:20:54.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f489836b640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:54.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f489836b640 10 monclient(hunting): authenticate will time out at 1035.386555s 2024-04-26T00:20:54.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4896107640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f48900a0300 0x7f48901cf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:54.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4896908640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48901352b0 0x7f4890139ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:54.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4896908640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48901352b0 0x7f4890139ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:33594/0 (socket says 172.21.15.37:33594) 2024-04-26T00:20:54.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4896107640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f48900a0300 0x7f48901cf1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:37560/0 (socket says 172.21.15.37:37560) 2024-04-26T00:20:54.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4896908640 1 -- 172.21.15.37:0/3260610194 learned_addr learned my addr 172.21.15.37:0/3260610194 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:54.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4896908640 10 monclient(hunting): get_auth_request con 0x7f48901352b0 auth_method 0 2024-04-26T00:20:54.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4896908640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:54.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4896908640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:54.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4896908640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:54.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4896107640 10 monclient(hunting): get_auth_request con 0x7f48900a0300 auth_method 0 2024-04-26T00:20:54.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4896107640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:54.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4896107640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:54.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4896107640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:54.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4896908640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:54.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4896908640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:54.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4896908640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:54.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4896107640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:54.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4896107640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:54.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4896107640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:54.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4895906640 1 --2- 172.21.15.37:0/3260610194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48900a06e0 0x7f489012fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:54.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4895906640 10 monclient(hunting): get_auth_request con 0x7f48900a06e0 auth_method 0 2024-04-26T00:20:54.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4895906640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:54.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4895906640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:54.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4895906640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:54.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4896908640 10 monclient(hunting): handle_auth_done global_id 4436 payload 824 2024-04-26T00:20:54.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4896908640 1 -- 172.21.15.37:0/3260610194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48900a06e0 msgr2=0x7f489012fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:54.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4896908640 1 --2- 172.21.15.37:0/3260610194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48900a06e0 0x7f489012fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:54.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4896908640 1 -- 172.21.15.37:0/3260610194 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f48900a0300 msgr2=0x7f48901cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:54.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4896908640 1 --2- 172.21.15.37:0/3260610194 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f48900a0300 0x7f48901cf1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:54.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4896908640 10 monclient: _finish_hunting 0 2024-04-26T00:20:54.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4896908640 1 monclient: found mon.noname-c 2024-04-26T00:20:54.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4896908640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:54.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4896908640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:20:54.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4896908640 1 -- 172.21.15.37:0/3260610194 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48901309c0 con 0x7f48901352b0 2024-04-26T00:20:54.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4896107640 1 --2- 172.21.15.37:0/3260610194 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f48900a0300 0x7f48901cf1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:54.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.079+0000 7f4895906640 1 --2- 172.21.15.37:0/3260610194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48900a06e0 0x7f489012fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:20:54.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f4896908640 1 -- 172.21.15.37:0/3260610194 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48901352b0 msgr2=0x7f4890139ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:54.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f4896908640 1 --2- 172.21.15.37:0/3260610194 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48901352b0 0x7f4890139ba0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f488000d800 tx=0x7f488000dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:54.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f488effd640 1 -- 172.21.15.37:0/3260610194 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f488000b8d0 con 0x7f48901352b0 2024-04-26T00:20:54.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f488effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:54.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f488effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:20:54.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f488effd640 10 monclient: dump: 2024-04-26T00:20:54.070 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:54.071 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:54.071 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:54.071 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:54.071 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:54.071 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:54.071 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:54.071 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:54.071 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:54.071 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:54.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f488effd640 10 monclient: _finish_auth 0 2024-04-26T00:20:54.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f488effd640 10 monclient: _check_auth_tickets 2024-04-26T00:20:54.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f488effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:54.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f488effd640 1 -- 172.21.15.37:0/3260610194 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4880019070 con 0x7f48901352b0 2024-04-26T00:20:54.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f488effd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:54.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f488effd640 1 -- 172.21.15.37:0/3260610194 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f488000bcb0 con 0x7f48901352b0 2024-04-26T00:20:54.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f488effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:54.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f488effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:54.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f488effd640 10 monclient: dump: 2024-04-26T00:20:54.072 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:54.072 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:54.072 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:54.072 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:54.072 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:54.072 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:54.072 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:54.072 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:54.072 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:54.072 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:54.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f489836b640 5 monclient: authenticate success, global_id 4436 2024-04-26T00:20:54.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f489836b640 10 monclient: _renew_subs 2024-04-26T00:20:54.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f489836b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:54.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f489836b640 1 -- 172.21.15.37:0/3260610194 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f489011bc10 con 0x7f48901352b0 2024-04-26T00:20:54.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f488ffff640 4 set_mon_vals no callback set 2024-04-26T00:20:54.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f489836b640 10 monclient: _renew_subs 2024-04-26T00:20:54.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f489836b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:54.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f489836b640 1 -- 172.21.15.37:0/3260610194 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f48901d0070 con 0x7f48901352b0 2024-04-26T00:20:54.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f489836b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:54.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f489836b640 1 --2- 172.21.15.37:0/3260610194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4890130040 0x7f489013a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:54.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f489836b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:54.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f489836b640 1 -- 172.21.15.37:0/3260610194 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f489011c3c0 con 0x7f4890130040 2024-04-26T00:20:54.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f488effd640 1 -- 172.21.15.37:0/3260610194 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f488001e460 con 0x7f48901352b0 2024-04-26T00:20:54.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f488effd640 1 --2- 172.21.15.37:0/3260610194 >> v2:172.21.15.45:6812/2379918623 conn(0x7f487c035ed0 0x7f487c038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:54.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f488effd640 1 -- 172.21.15.37:0/3260610194 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(45..45 src has 1..45) ==== 3915+0+0 (secure 0 0 0) 0x7f4880048e70 con 0x7f48901352b0 2024-04-26T00:20:54.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f4896107640 1 --2- 172.21.15.37:0/3260610194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4890130040 0x7f489013a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:54.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f4896107640 10 monclient: get_auth_request con 0x7f4890130040 auth_method 0 2024-04-26T00:20:54.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f4896107640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:54.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f4896107640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:54.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f4896107640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:54.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f4895906640 1 --2- 172.21.15.37:0/3260610194 >> v2:172.21.15.45:6812/2379918623 conn(0x7f487c035ed0 0x7f487c038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:54.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f4896107640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:54.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f4896107640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:54.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f4896107640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:54.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f4895906640 10 monclient: get_auth_request con 0x7f487c035ed0 auth_method 0 2024-04-26T00:20:54.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f4896107640 10 monclient(hunting): handle_auth_done global_id 14110 payload 824 2024-04-26T00:20:54.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.083+0000 7f4896107640 1 -- 172.21.15.37:0/3260610194 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f489004f510 con 0x7f4890130040 2024-04-26T00:20:54.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.087+0000 7f4895906640 1 --2- 172.21.15.37:0/3260610194 >> v2:172.21.15.45:6812/2379918623 conn(0x7f487c035ed0 0x7f487c038390 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4890130bd0 tx=0x7f4884009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:54.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.087+0000 7f4896107640 1 -- 172.21.15.37:0/3260610194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4890130040 msgr2=0x7f489013a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:54.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.087+0000 7f4896107640 1 --2- 172.21.15.37:0/3260610194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4890130040 0x7f489013a0e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f489006caa0 tx=0x7f488801abe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:54.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.087+0000 7f488effd640 1 -- 172.21.15.37:0/3260610194 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48880b2040 con 0x7f4890130040 2024-04-26T00:20:54.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.087+0000 7f488effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:54.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.087+0000 7f488effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:20:54.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.087+0000 7f488effd640 10 monclient: dump: 2024-04-26T00:20:54.075 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:54.075 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:54.075 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:54.075 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:54.075 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:54.075 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:54.075 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:54.075 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:54.075 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:54.075 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:54.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.087+0000 7f488effd640 1 -- 172.21.15.37:0/3260610194 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f48880b8670 con 0x7f4890130040 2024-04-26T00:20:54.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.087+0000 7f488effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:54.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.087+0000 7f488effd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:54.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.087+0000 7f488effd640 1 -- 172.21.15.37:0/3260610194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4890130040 msgr2=0x7f489013a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:54.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.087+0000 7f488effd640 1 --2- 172.21.15.37:0/3260610194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4890130040 0x7f489013a0e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f489006caa0 tx=0x7f488801abe0 comp rx=0 tx=0).stop 2024-04-26T00:20:54.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.087+0000 7f488effd640 1 -- 172.21.15.37:0/3260610194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4890130040 msgr2=0x7f489013a0e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:20:54.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.087+0000 7f488effd640 1 --2- 172.21.15.37:0/3260610194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4890130040 0x7f489013a0e0 secure :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f489006caa0 tx=0x7f488801abe0 comp rx=0 tx=0).stop 2024-04-26T00:20:54.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.095+0000 7f489836b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:20:54.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.095+0000 7f489836b640 1 --2- 172.21.15.37:0/3260610194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f489013a620 0x7f4890135c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:54.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.095+0000 7f489836b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:54.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.095+0000 7f489836b640 1 -- 172.21.15.37:0/3260610194 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f489013aa30 con 0x7f489013a620 2024-04-26T00:20:54.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.095+0000 7f4896107640 1 --2- 172.21.15.37:0/3260610194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f489013a620 0x7f4890135c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:54.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.095+0000 7f4896107640 10 monclient: get_auth_request con 0x7f489013a620 auth_method 0 2024-04-26T00:20:54.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.095+0000 7f4896107640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:54.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.095+0000 7f4896107640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:54.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.095+0000 7f4896107640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:54.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.095+0000 7f4896107640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:54.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.095+0000 7f4896107640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:54.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.095+0000 7f4896107640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:54.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.095+0000 7f4896107640 10 monclient(hunting): handle_auth_done global_id 14113 payload 824 2024-04-26T00:20:54.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.095+0000 7f4896107640 1 -- 172.21.15.37:0/3260610194 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f489013b840 con 0x7f489013a620 2024-04-26T00:20:54.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.095+0000 7f4896107640 1 -- 172.21.15.37:0/3260610194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f489013a620 msgr2=0x7f4890135c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:20:54.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.095+0000 7f4896107640 1 -- 172.21.15.37:0/3260610194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f489013a620 msgr2=0x7f4890135c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:20:54.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.095+0000 7f4896107640 1 --2- 172.21.15.37:0/3260610194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f489013a620 0x7f4890135c90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f488801c3e0 tx=0x7f4888019140 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:20:54.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.095+0000 7f4896107640 1 --2- 172.21.15.37:0/3260610194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f489013a620 0x7f4890135c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:20:54.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.299+0000 7f4896107640 1 --2- 172.21.15.37:0/3260610194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f489013a620 0x7f4890135c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:54.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.299+0000 7f4896107640 10 monclient: get_auth_request con 0x7f489013a620 auth_method 0 2024-04-26T00:20:54.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.299+0000 7f4896107640 10 monclient: get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:54.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.299+0000 7f4896107640 10 monclient: _init_auth method 2 2024-04-26T00:20:54.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.299+0000 7f4896107640 10 monclient: _init_auth already have auth, reseting 2024-04-26T00:20:54.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.299+0000 7f4896107640 10 monclient: handle_auth_reply_more payload 9 2024-04-26T00:20:54.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.299+0000 7f4896107640 10 monclient: handle_auth_reply_more payload_len 9 2024-04-26T00:20:54.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.299+0000 7f4896107640 10 monclient: handle_auth_reply_more responding with 132 bytes 2024-04-26T00:20:54.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.299+0000 7f4896107640 10 monclient: handle_auth_done global_id 14113 payload 293 2024-04-26T00:20:54.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.299+0000 7f4896107640 1 -- 172.21.15.37:0/3260610194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f489013a620 msgr2=0x7f4890135c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:54.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.299+0000 7f4896107640 1 --2- 172.21.15.37:0/3260610194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f489013a620 0x7f4890135c90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f488801c1d0 tx=0x7f48880b9000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:54.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.303+0000 7f488effd640 1 -- 172.21.15.37:0/3260610194 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48880b2070 con 0x7f489013a620 2024-04-26T00:20:54.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.303+0000 7f488effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:54.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.303+0000 7f488effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:20:54.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.303+0000 7f488effd640 10 monclient: dump: 2024-04-26T00:20:54.280 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:54.280 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:54.280 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:54.280 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:54.280 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:54.280 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:54.280 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:54.280 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:54.280 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:54.281 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:54.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.303+0000 7f488effd640 1 -- 172.21.15.37:0/3260610194 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7f4888003620 con 0x7f489013a620 2024-04-26T00:20:54.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.303+0000 7f488effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:20:54.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.303+0000 7f488effd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:54.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.303+0000 7f488effd640 1 -- 172.21.15.37:0/3260610194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f489013a620 msgr2=0x7f4890135c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:54.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.303+0000 7f488effd640 1 --2- 172.21.15.37:0/3260610194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f489013a620 0x7f4890135c90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f488801c1d0 tx=0x7f48880b9000 comp rx=0 tx=0).stop 2024-04-26T00:20:54.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.303+0000 7f488effd640 1 -- 172.21.15.37:0/3260610194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f489013a620 msgr2=0x7f4890135c90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:20:54.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.303+0000 7f488effd640 1 --2- 172.21.15.37:0/3260610194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f489013a620 0x7f4890135c90 secure :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f488801c1d0 tx=0x7f48880b9000 comp rx=0 tx=0).stop 2024-04-26T00:20:54.282 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:20:54.282 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:20:54.282 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:20:54.282 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:20:54.283 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 12, 2024-04-26T00:20:54.283 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 7815, 2024-04-26T00:20:54.283 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:20:54.283 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:20:54.283 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:20:54.283 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:20:54.283 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:54.283 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 3, 2024-04-26T00:20:54.283 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:20:54.283 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:20:54.283 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:20:54.283 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:20:54.283 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:20:54.283 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:20:54.284 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:20:54.284 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:20:54.284 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:20:54.284 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:20:54.284 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:20:54.284 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:20:54.284 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:20:54.284 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:20:54.284 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:54.284 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:20:54.284 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:20:54.284 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:20:54.284 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:20:54.284 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:20:54.284 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:20:54.285 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:20:54.285 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:20:54.285 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:20:54.285 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:20:54.285 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:20:54.285 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:20:54.285 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:20:54.285 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:54.285 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:54.285 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:20:54.285 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:20:54.285 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:20:54.285 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:20:54.285 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:20:54.286 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:20:54.286 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:20:54.286 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:20:54.286 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:20:54.286 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:20:54.286 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:20:54.286 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:20:54.286 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:20:54.286 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:20:54.286 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:20:54.286 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:20:54.286 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:20:54.286 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:20:54.286 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:20:54.287 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:20:54.287 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:20:54.287 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:20:54.287 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:20:54.287 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:20:54.287 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:20:54.287 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:20:54.287 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:20:54.287 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:20:54.287 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:54.287 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:20:54.287 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:54.287 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:20:54.287 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:54.288 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:20:54.288 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:20:54.288 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:20:54.288 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:20:54.288 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:54.288 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:20:54.288 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:20:54.288 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:54.288 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:54.288 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:54.288 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:20:54.288 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:20:54.288 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:54.288 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:54.289 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:54.289 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:54.289 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:20:54.289 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:20:54.289 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:20:54.289 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:20:54.289 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:20:54.289 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:54.289 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:54.289 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:20:54.289 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:20:54.289 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:20:54.289 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:20:54.290 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:54.290 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:20:54.290 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:20:54.290 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:54.290 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:54.290 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:54.290 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:20:54.290 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:20:54.290 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:54.290 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:54.290 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:54.290 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:54.290 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:20:54.291 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:20:54.291 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:20:54.291 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:20:54.291 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:20:54.291 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:54.291 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:54.291 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:20:54.291 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:20:54.291 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:20:54.291 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:20:54.291 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:54.291 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:20:54.291 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:20:54.291 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:54.291 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:54.292 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:54.292 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:20:54.292 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:20:54.292 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:54.292 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:54.292 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:54.292 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:54.292 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:20:54.292 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:20:54.292 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:20:54.292 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:20:54.292 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:20:54.292 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:54.292 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:54.292 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:54.293 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:20:54.293 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:20:54.293 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:54.293 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:20:54.293 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:20:54.293 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:20:54.293 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:54.293 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:54.293 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:20:54.293 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:54.293 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:20:54.293 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:20:54.293 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:20:54.293 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:54.294 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:54.294 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:54.294 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:20:54.294 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:20:54.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.303+0000 7f489836b640 1 -- 172.21.15.37:0/3260610194 >> v2:172.21.15.45:6812/2379918623 conn(0x7f487c035ed0 msgr2=0x7f487c038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:54.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.303+0000 7f489836b640 1 --2- 172.21.15.37:0/3260610194 >> v2:172.21.15.45:6812/2379918623 conn(0x7f487c035ed0 0x7f487c038390 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4890130bd0 tx=0x7f4884009040 comp rx=0 tx=0).stop 2024-04-26T00:20:54.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.303+0000 7f489836b640 10 monclient: shutdown 2024-04-26T00:20:54.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.303+0000 7f489836b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:54.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.303+0000 7f489836b640 1 -- 172.21.15.37:0/3260610194 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48901352b0 msgr2=0x7f4890139ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:54.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.303+0000 7f489836b640 1 --2- 172.21.15.37:0/3260610194 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48901352b0 0x7f4890139ba0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f488000d800 tx=0x7f488000dcc0 comp rx=0 tx=0).stop 2024-04-26T00:20:54.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.303+0000 7f4896908640 1 -- 172.21.15.37:0/3260610194 reap_dead start 2024-04-26T00:20:54.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.303+0000 7f4896908640 1 -- 172.21.15.37:0/3260610194 reap_dead start 2024-04-26T00:20:54.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.303+0000 7f489836b640 1 -- 172.21.15.37:0/3260610194 shutdown_connections 2024-04-26T00:20:54.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.303+0000 7f489836b640 1 -- 172.21.15.37:0/3260610194 >> 172.21.15.37:0/3260610194 conn(0x7f48900a1d50 msgr2=0x7f48900a3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:54.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.303+0000 7f489836b640 1 -- 172.21.15.37:0/3260610194 shutdown_connections 2024-04-26T00:20:54.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.303+0000 7f489836b640 1 -- 172.21.15.37:0/3260610194 wait complete. 2024-04-26T00:20:54.306 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:20:54.306 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-26T00:20:54.306 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-26T00:20:54.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.463+0000 7fd688987640 10 monclient: get_monmap_and_config 2024-04-26T00:20:54.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.463+0000 7fd688987640 10 monclient: build_initial_monmap 2024-04-26T00:20:54.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.463+0000 7fd688987640 10 monclient: monmap: 2024-04-26T00:20:54.442 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:54.442 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:54.442 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:54.466449+0000 2024-04-26T00:20:54.443 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:54.466449+0000 2024-04-26T00:20:54.443 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:54.443 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:54.443 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:54.443 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:54.443 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:54.443 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:54.443 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:54.443 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:54.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.463+0000 7fd688987640 1 Processor -- start 2024-04-26T00:20:54.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.463+0000 7fd688987640 1 -- start start 2024-04-26T00:20:54.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.463+0000 7fd688987640 10 monclient: init 2024-04-26T00:20:54.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.467+0000 7fd688987640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:54.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.467+0000 7fd688987640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-04-26T00:20:54.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.467+0000 7fd688987640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd6840a0260 addr v1:172.21.15.45:6789/0 2024-04-26T00:20:54.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.467+0000 7fd688987640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd684137e40 0x7fd68413a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:54.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.467+0000 7fd688987640 10 monclient(hunting): picked mon.noname-c con 0x7fd684137e40 addr v2:172.21.15.45:3300/0 2024-04-26T00:20:54.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.467+0000 7fd688987640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd68413a770 0x7fd68413cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:54.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.467+0000 7fd688987640 10 monclient(hunting): picked mon.noname-b con 0x7fd68413a770 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:54.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.467+0000 7fd688987640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6840a1130 con 0x7fd6840a0260 2024-04-26T00:20:54.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.467+0000 7fd688987640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:54.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.467+0000 7fd688987640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fd6840a0d50 con 0x7fd684137e40 2024-04-26T00:20:54.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.467+0000 7fd688987640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:54.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.467+0000 7fd688987640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd68409ec10 con 0x7fd68413a770 2024-04-26T00:20:54.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.467+0000 7fd688987640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:54.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.467+0000 7fd688987640 10 monclient(hunting): authenticate will time out at 1035.772141s 2024-04-26T00:20:54.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.467+0000 7fd682d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd68413a770 0x7fd68413cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:54.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.467+0000 7fd682d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd68413a770 0x7fd68413cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:33604/0 (socket says 172.21.15.37:33604) 2024-04-26T00:20:54.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.467+0000 7fd682d9d640 1 -- 172.21.15.37:0/1574215824 learned_addr learned my addr 172.21.15.37:0/1574215824 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:54.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.467+0000 7fd682d9d640 10 monclient(hunting): get_auth_request con 0x7fd68413a770 auth_method 0 2024-04-26T00:20:54.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.467+0000 7fd682d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:54.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.467+0000 7fd682d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:54.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.467+0000 7fd682d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:54.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.467+0000 7fd682d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:54.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.467+0000 7fd682d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:54.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.467+0000 7fd682d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:54.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.467+0000 7fd681d9b640 1 --2- 172.21.15.37:0/1574215824 >> v2:172.21.15.45:3300/0 conn(0x7fd684137e40 0x7fd68413a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:54.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.467+0000 7fd681d9b640 10 monclient(hunting): get_auth_request con 0x7fd684137e40 auth_method 0 2024-04-26T00:20:54.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.467+0000 7fd681d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:54.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.467+0000 7fd681d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:54.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.467+0000 7fd681d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:54.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.467+0000 7fd682d9d640 10 monclient(hunting): handle_auth_done global_id 4439 payload 274 2024-04-26T00:20:54.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.467+0000 7fd682d9d640 1 -- 172.21.15.37:0/1574215824 >> v2:172.21.15.45:3300/0 conn(0x7fd684137e40 msgr2=0x7fd68413a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:54.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.467+0000 7fd682d9d640 1 --2- 172.21.15.37:0/1574215824 >> v2:172.21.15.45:3300/0 conn(0x7fd684137e40 0x7fd68413a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:54.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.467+0000 7fd682d9d640 1 -- 172.21.15.37:0/1574215824 >> v1:172.21.15.45:6789/0 conn(0x7fd6840a0260 legacy=0x7fd6840a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:54.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.467+0000 7fd682d9d640 10 monclient: _finish_hunting 0 2024-04-26T00:20:54.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.467+0000 7fd682d9d640 1 monclient: found mon.noname-b 2024-04-26T00:20:54.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.467+0000 7fd682d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:54.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.467+0000 7fd682d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:20:54.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.467+0000 7fd682d9d640 1 -- 172.21.15.37:0/1574215824 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd68406cff0 con 0x7fd68413a770 2024-04-26T00:20:54.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd682d9d640 1 -- 172.21.15.37:0/1574215824 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd68413a770 msgr2=0x7fd68413cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:54.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd682d9d640 1 --2- 172.21.15.37:0/1574215824 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd68413a770 0x7fd68413cb60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd67c009870 tx=0x7fd67c01c200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fffaab23c40cbd1d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:54.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd68159a640 1 -- 172.21.15.37:0/1574215824 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67c0b0070 con 0x7fd68413a770 2024-04-26T00:20:54.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd68159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:54.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd68159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:20:54.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd68159a640 10 monclient: dump: 2024-04-26T00:20:54.450 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:54.450 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:54.450 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:54.450 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:54.450 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:54.450 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:54.450 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:54.450 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:54.450 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:54.450 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:54.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd68159a640 10 monclient: _finish_auth 0 2024-04-26T00:20:54.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd68159a640 10 monclient: _check_auth_tickets 2024-04-26T00:20:54.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd68159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:54.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd68159a640 1 -- 172.21.15.37:0/1574215824 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd67c0ab600 con 0x7fd68413a770 2024-04-26T00:20:54.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd68159a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:54.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd68159a640 1 -- 172.21.15.37:0/1574215824 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67c0ab9d0 con 0x7fd68413a770 2024-04-26T00:20:54.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd688987640 5 monclient: authenticate success, global_id 4439 2024-04-26T00:20:54.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd688987640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:54.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd688987640 4 set_mon_vals no callback set 2024-04-26T00:20:54.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd68159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:54.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd68159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:54.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd68159a640 10 monclient: dump: 2024-04-26T00:20:54.451 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:54.451 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:54.451 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:54.451 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:54.451 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:54.452 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:54.452 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:54.452 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:54.452 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:54.452 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:54.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd688987640 10 monclient: shutdown 2024-04-26T00:20:54.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd688987640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:54.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd688987640 1 -- 172.21.15.37:0/1574215824 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd68413a770 msgr2=0x7fd68413cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:54.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd688987640 1 --2- 172.21.15.37:0/1574215824 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd68413a770 0x7fd68413cb60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd67c009870 tx=0x7fd67c01c200 comp rx=0 tx=0).stop 2024-04-26T00:20:54.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd688987640 1 -- 172.21.15.37:0/1574215824 shutdown_connections 2024-04-26T00:20:54.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd688987640 1 --2- 172.21.15.37:0/1574215824 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd68413a770 0x7fd68413cb60 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:54.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd688987640 1 --2- 172.21.15.37:0/1574215824 >> v2:172.21.15.45:3300/0 conn(0x7fd684137e40 0x7fd68413a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:54.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd688987640 1 -- 172.21.15.37:0/1574215824 >> 172.21.15.37:0/1574215824 conn(0x7fd6840a1ad0 msgr2=0x7fd6840a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:54.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd688987640 1 -- 172.21.15.37:0/1574215824 shutdown_connections 2024-04-26T00:20:54.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd688987640 1 -- 172.21.15.37:0/1574215824 wait complete. 2024-04-26T00:20:54.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd688987640 10 monclient: build_initial_monmap 2024-04-26T00:20:54.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd688987640 10 monclient: monmap: 2024-04-26T00:20:54.453 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:54.453 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:54.453 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:54.453 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:54.453 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:54.453 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:54.453 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:54.453 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:54.453 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:54.453 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:54.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd688987640 1 Processor -- start 2024-04-26T00:20:54.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd688987640 1 -- start start 2024-04-26T00:20:54.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd688987640 10 monclient: init 2024-04-26T00:20:54.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd688987640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:54.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd688987640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:20:54.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd688987640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6840a0260 0x7fd68411dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:54.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd688987640 10 monclient(hunting): picked mon.noname-b con 0x7fd6840a0260 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:54.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd688987640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd684137e40 0x7fd68411c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:54.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd688987640 10 monclient(hunting): picked mon.noname-a con 0x7fd684137e40 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:54.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd688987640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd68413a770 0x7fd6841cc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:54.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd688987640 10 monclient(hunting): picked mon.noname-c con 0x7fd68413a770 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:54.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd688987640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:54.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd688987640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd68409f4c0 con 0x7fd684137e40 2024-04-26T00:20:54.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd688987640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:54.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd688987640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd68409f0f0 con 0x7fd6840a0260 2024-04-26T00:20:54.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd688987640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:54.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd688987640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd68413f8c0 con 0x7fd68413a770 2024-04-26T00:20:54.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd688987640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:54.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd688987640 10 monclient(hunting): authenticate will time out at 1035.779300s 2024-04-26T00:20:54.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd682d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd68413a770 0x7fd6841cc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:54.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd681d9b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd684137e40 0x7fd68411c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:54.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd682d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd68413a770 0x7fd6841cc9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:33606/0 (socket says 172.21.15.37:33606) 2024-04-26T00:20:54.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd681d9b640 10 monclient(hunting): get_auth_request con 0x7fd684137e40 auth_method 0 2024-04-26T00:20:54.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd682d9d640 1 -- 172.21.15.37:0/3106116434 learned_addr learned my addr 172.21.15.37:0/3106116434 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:54.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd681d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:54.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd681d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:54.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd681d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:54.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd682d9d640 10 monclient(hunting): get_auth_request con 0x7fd68413a770 auth_method 0 2024-04-26T00:20:54.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd682d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:54.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd682d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:54.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd682d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:54.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd682d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:54.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd682d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:54.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd682d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:54.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd681d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:54.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd681d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:54.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd681d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:54.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd68259c640 1 --2- 172.21.15.37:0/3106116434 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6840a0260 0x7fd68411dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:54.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd68259c640 10 monclient(hunting): get_auth_request con 0x7fd6840a0260 auth_method 0 2024-04-26T00:20:54.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd68259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:54.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd68259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:54.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd68259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:54.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd682d9d640 10 monclient(hunting): handle_auth_done global_id 4442 payload 824 2024-04-26T00:20:54.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd682d9d640 1 -- 172.21.15.37:0/3106116434 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6840a0260 msgr2=0x7fd68411dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:54.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd682d9d640 1 --2- 172.21.15.37:0/3106116434 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6840a0260 0x7fd68411dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:54.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd682d9d640 1 -- 172.21.15.37:0/3106116434 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd684137e40 msgr2=0x7fd68411c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:54.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd682d9d640 1 --2- 172.21.15.37:0/3106116434 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd684137e40 0x7fd68411c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:54.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd682d9d640 10 monclient: _finish_hunting 0 2024-04-26T00:20:54.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd682d9d640 1 monclient: found mon.noname-c 2024-04-26T00:20:54.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd682d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:54.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd682d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:20:54.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.471+0000 7fd682d9d640 1 -- 172.21.15.37:0/3106116434 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6840729d0 con 0x7fd68413a770 2024-04-26T00:20:54.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.475+0000 7fd682d9d640 1 -- 172.21.15.37:0/3106116434 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd68413a770 msgr2=0x7fd6841cc9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:54.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.475+0000 7fd682d9d640 1 --2- 172.21.15.37:0/3106116434 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd68413a770 0x7fd6841cc9c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd67c01a8b0 tx=0x7fd67c018fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:54.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.475+0000 7fd67affd640 1 -- 172.21.15.37:0/3106116434 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67c0b0040 con 0x7fd68413a770 2024-04-26T00:20:54.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.475+0000 7fd67affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:54.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.475+0000 7fd67affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:20:54.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.475+0000 7fd67affd640 10 monclient: dump: 2024-04-26T00:20:54.461 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:54.461 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:54.461 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:54.461 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:54.461 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:54.461 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:54.461 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:54.461 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:54.461 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:54.461 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:54.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.475+0000 7fd67affd640 10 monclient: _finish_auth 0 2024-04-26T00:20:54.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.475+0000 7fd67affd640 10 monclient: _check_auth_tickets 2024-04-26T00:20:54.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.475+0000 7fd67affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:54.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.475+0000 7fd67affd640 1 -- 172.21.15.37:0/3106116434 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd67c0b9070 con 0x7fd68413a770 2024-04-26T00:20:54.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.475+0000 7fd67affd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:54.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.475+0000 7fd67affd640 1 -- 172.21.15.37:0/3106116434 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67c0b4440 con 0x7fd68413a770 2024-04-26T00:20:54.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.475+0000 7fd688987640 5 monclient: authenticate success, global_id 4442 2024-04-26T00:20:54.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.475+0000 7fd688987640 10 monclient: _renew_subs 2024-04-26T00:20:54.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.475+0000 7fd688987640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:54.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.475+0000 7fd688987640 1 -- 172.21.15.37:0/3106116434 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd68411e830 con 0x7fd68413a770 2024-04-26T00:20:54.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.475+0000 7fd680d99640 4 set_mon_vals no callback set 2024-04-26T00:20:54.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.475+0000 7fd67affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:54.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.475+0000 7fd67affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:54.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.475+0000 7fd67affd640 10 monclient: dump: 2024-04-26T00:20:54.462 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:54.462 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:54.462 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:54.462 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:54.462 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:54.462 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:54.463 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:54.463 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:54.463 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:54.463 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:54.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.475+0000 7fd688987640 10 monclient: _renew_subs 2024-04-26T00:20:54.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.475+0000 7fd688987640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:54.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.475+0000 7fd688987640 1 -- 172.21.15.37:0/3106116434 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd68411f2b0 con 0x7fd68413a770 2024-04-26T00:20:54.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.475+0000 7fd688987640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:54.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.475+0000 7fd688987640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:54.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.475+0000 7fd688987640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:54.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.475+0000 7fd688987640 1 -- 172.21.15.37:0/3106116434 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd684066f40 con 0x7fd68413a770 2024-04-26T00:20:54.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.475+0000 7fd67affd640 1 -- 172.21.15.37:0/3106116434 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd67c0b45e0 con 0x7fd68413a770 2024-04-26T00:20:54.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.475+0000 7fd67affd640 1 --2- 172.21.15.37:0/3106116434 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd664035eb0 0x7fd664038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:54.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.475+0000 7fd67affd640 1 -- 172.21.15.37:0/3106116434 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(45..45 src has 1..45) ==== 3915+0+0 (secure 0 0 0) 0x7fd67c0e23a0 con 0x7fd68413a770 2024-04-26T00:20:54.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.475+0000 7fd68259c640 1 --2- 172.21.15.37:0/3106116434 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd664035eb0 0x7fd664038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:54.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.475+0000 7fd68259c640 10 monclient: get_auth_request con 0x7fd664035eb0 auth_method 0 2024-04-26T00:20:54.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.479+0000 7fd68259c640 1 --2- 172.21.15.37:0/3106116434 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd664035eb0 0x7fd664038370 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd66c004e30 tx=0x7fd66c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:54.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.483+0000 7fd67affd640 1 -- 172.21.15.37:0/3106116434 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fd6840a1130 con 0x7fd68413a770 2024-04-26T00:20:54.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.483+0000 7fd67affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:54.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.483+0000 7fd67affd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:54.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.703+0000 7fd688987640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-26T00:20:54.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.703+0000 7fd688987640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-26T00:20:54.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.703+0000 7fd688987640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:54.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.703+0000 7fd688987640 1 -- 172.21.15.37:0/3106116434 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fd68411db10 con 0x7fd68413a770 2024-04-26T00:20:54.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.707+0000 7fd67affd640 1 -- 172.21.15.37:0/3106116434 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fd67c0e2150 con 0x7fd68413a770 2024-04-26T00:20:54.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.707+0000 7fd67affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-26T00:20:54.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.707+0000 7fd67affd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:54.686 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:20:54.707+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:20:54.686 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:20:54.707+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=8,config=2,health=10,kv=10,logm=70} crc {auth=202248012,config=692619927,health=2820191030,kv=1034996044,logm=2873402557}) 2024-04-26T00:20:54.686 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:20:54.707+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=6,config=2,health=11,kv=10,logm=71} crc {auth=1416320320,config=692619927,health=555465209,kv=1034996044,logm=2209347936}) 2024-04-26T00:20:54.686 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:20:54.707+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:20:54.686 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:20:54.707+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=8,config=2,health=10,kv=10,logm=70} crc {auth=202248012,config=692619927,health=2820191030,kv=1034996044,logm=2873402557}) 2024-04-26T00:20:54.686 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:20:54.707+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=8,config=2,health=10,kv=10,logm=70} crc {auth=202248012,config=692619927,health=2069958880,kv=1034996044,logm=3906042456}) 2024-04-26T00:20:54.687 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:20:54.711+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:20:54.688 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:20:54.711+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3794133007}) 2024-04-26T00:20:54.688 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:20:54.711+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4193878102}) 2024-04-26T00:20:54.688 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:20:54.711+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:20:54.688 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:20:54.711+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3794133007}) 2024-04-26T00:20:54.688 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:20:54.711+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4125197668}) 2024-04-26T00:20:54.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.711+0000 7fd688987640 1 -- 172.21.15.37:0/3106116434 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd664035eb0 msgr2=0x7fd664038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:54.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.711+0000 7fd688987640 1 --2- 172.21.15.37:0/3106116434 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd664035eb0 0x7fd664038370 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd66c004e30 tx=0x7fd66c009040 comp rx=0 tx=0).stop 2024-04-26T00:20:54.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.711+0000 7fd688987640 10 monclient: shutdown 2024-04-26T00:20:54.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.711+0000 7fd688987640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:54.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.711+0000 7fd688987640 1 -- 172.21.15.37:0/3106116434 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd68413a770 msgr2=0x7fd6841cc9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:54.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.711+0000 7fd688987640 1 --2- 172.21.15.37:0/3106116434 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd68413a770 0x7fd6841cc9c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd67c01a8b0 tx=0x7fd67c018fa0 comp rx=0 tx=0).stop 2024-04-26T00:20:54.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.711+0000 7fd688987640 1 -- 172.21.15.37:0/3106116434 shutdown_connections 2024-04-26T00:20:54.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.711+0000 7fd688987640 1 --2- 172.21.15.37:0/3106116434 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd664035eb0 0x7fd664038370 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:54.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.711+0000 7fd688987640 1 --2- 172.21.15.37:0/3106116434 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd68413a770 0x7fd6841cc9c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:54.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.711+0000 7fd688987640 1 --2- 172.21.15.37:0/3106116434 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd684137e40 0x7fd68411c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:54.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.711+0000 7fd688987640 1 --2- 172.21.15.37:0/3106116434 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd6840a0260 0x7fd68411dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:54.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.711+0000 7fd688987640 1 -- 172.21.15.37:0/3106116434 >> 172.21.15.37:0/3106116434 conn(0x7fd6840a1ad0 msgr2=0x7fd6840a3d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:54.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.711+0000 7fd688987640 1 -- 172.21.15.37:0/3106116434 shutdown_connections 2024-04-26T00:20:54.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:54.711+0000 7fd688987640 1 -- 172.21.15.37:0/3106116434 wait complete. 2024-04-26T00:20:54.690 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:20:54.711+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:20:54.690 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:20:54.711+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3103640446}) 2024-04-26T00:20:54.691 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:20:54.711+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3099101530}) 2024-04-26T00:20:54.691 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:20:54.711+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:20:54.691 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:20:54.711+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3103640446}) 2024-04-26T00:20:54.691 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:20:54.711+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2712390184}) 2024-04-26T00:20:54.692 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:20:54.715+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:20:54.692 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:20:54.715+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2646534500}) 2024-04-26T00:20:54.692 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:20:54.715+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2862868253}) 2024-04-26T00:20:54.692 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:20:54.715+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:20:54.692 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:20:54.715+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2646534500}) 2024-04-26T00:20:54.692 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:20:54.715+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=553365454}) 2024-04-26T00:20:54.695 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:20:54.715+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:20:54.695 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:20:54.715+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=3,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=10,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=5,osdmap=51} crc {logm=284324338,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339439,mgr_metadata=3686404438,mgrstat=3208497726,mon_config_key=1720404874,monmap=4194790699,osd_metadata=3306121363,osd_pg_creating=97454253,osd_snap=4044690964,osdmap=4227760458}) 2024-04-26T00:20:54.695 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:20:54.715+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=3,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=11,monmap=3,osd_metadata=6,osd_snap=5,osdmap=52} crc {logm=284324338,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=4165182441,mon_config_key=4061063117,monmap=3876866194,osd_metadata=3306121363,osd_snap=4044690964,osdmap=1199854944}) 2024-04-26T00:20:54.695 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:20:54.715+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:20:54.695 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:20:54.715+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=3,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=10,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=5,osdmap=51} crc {logm=284324338,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339439,mgr_metadata=3686404438,mgrstat=3208497726,mon_config_key=1720404874,monmap=4194790699,osd_metadata=3306121363,osd_pg_creating=97454253,osd_snap=4044690964,osdmap=4227760458}) 2024-04-26T00:20:54.695 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:20:54.715+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=3,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=10,monmap=1,osd_metadata=6,osd_pg_creating=1,osd_snap=4,osdmap=53} crc {logm=284324338,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=3208497726,mon_config_key=752439442,monmap=1228700967,osd_metadata=3306121363,osd_pg_creating=97454253,osd_snap=2959546430,osdmap=1235001894}) 2024-04-26T00:20:54.697 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:20:54.719+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:20:54.697 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:20:54.719+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=31} crc {osdmap=1595375745}) 2024-04-26T00:20:54.697 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:20:54.719+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=31} crc {osdmap=1232401116}) 2024-04-26T00:20:54.697 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:20:54.719+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:20:54.697 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:20:54.719+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=31} crc {osdmap=1595375745}) 2024-04-26T00:20:54.698 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:20:54.719+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=30} crc {osdmap=1810212833}) 2024-04-26T00:20:54.707 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-26T00:20:54.904 INFO:tasks.rados.rados.0.smithi045.stdout:152: done (2 left) 2024-04-26T00:20:54.904 INFO:tasks.rados.rados.0.smithi045.stdout:153: read oid 1 snap -1 2024-04-26T00:20:54.904 INFO:tasks.rados.rados.0.smithi045.stdout:153: expect deleted 2024-04-26T00:20:54.904 INFO:tasks.rados.rados.0.smithi045.stdout:154: read oid 22 snap 13 2024-04-26T00:20:54.905 INFO:tasks.rados.rados.0.smithi045.stdout:154: expect (ObjNum 53 snap 3 seq_num 53) 2024-04-26T00:20:54.905 INFO:tasks.rados.rados.0.smithi045.stdout:155: read oid 15 snap 13 2024-04-26T00:20:54.905 INFO:tasks.rados.rados.0.smithi045.stdout:155: expect (ObjNum 14 snap 0 seq_num 14) 2024-04-26T00:20:54.905 INFO:tasks.rados.rados.0.smithi045.stdout:156: read oid 34 snap -1 2024-04-26T00:20:54.905 INFO:tasks.rados.rados.0.smithi045.stdout:156: expect (ObjNum 33 snap 0 seq_num 33) 2024-04-26T00:20:54.905 INFO:tasks.rados.rados.0.smithi045.stdout:157: delete oid 37 current snap is 14 2024-04-26T00:20:54.906 INFO:tasks.rados.rados.0.smithi045.stdout:150: finishing rollback tid 0 to smithi04526579-20 2024-04-26T00:20:54.906 INFO:tasks.rados.rados.0.smithi045.stdout:150: finishing rollback tid 1 to smithi04526579-20 2024-04-26T00:20:54.906 INFO:tasks.rados.rados.0.smithi045.stdout:150: finishing rollback tid 2 to smithi04526579-20 2024-04-26T00:20:54.906 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 46 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-04-26T00:20:54.916 INFO:tasks.rados.rados.0.smithi045.stdout:156: expect (ObjNum 33 snap 0 seq_num 33) 2024-04-26T00:20:55.058 INFO:tasks.rados.rados.0.smithi045.stdout:154: expect (ObjNum 53 snap 3 seq_num 53) 2024-04-26T00:20:55.194 INFO:tasks.rados.rados.0.smithi045.stdout:155: expect (ObjNum 14 snap 0 seq_num 14) 2024-04-26T00:20:55.269 INFO:tasks.rados.rados.0.smithi045.stdout:150: done (6 left) 2024-04-26T00:20:55.269 INFO:tasks.rados.rados.0.smithi045.stdout:151: done (5 left) 2024-04-26T00:20:55.269 INFO:tasks.rados.rados.0.smithi045.stdout:153: done (4 left) 2024-04-26T00:20:55.269 INFO:tasks.rados.rados.0.smithi045.stdout:154: done (3 left) 2024-04-26T00:20:55.270 INFO:tasks.rados.rados.0.smithi045.stdout:155: done (2 left) 2024-04-26T00:20:55.270 INFO:tasks.rados.rados.0.smithi045.stdout:156: done (1 left) 2024-04-26T00:20:55.270 INFO:tasks.rados.rados.0.smithi045.stdout:157: done (0 left) 2024-04-26T00:20:55.270 INFO:tasks.rados.rados.0.smithi045.stdout:158: snap_create 2024-04-26T00:20:55.707 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:20:55.708 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:20:55.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52bb041640 10 monclient: get_monmap_and_config 2024-04-26T00:20:55.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52bb041640 10 monclient: build_initial_monmap 2024-04-26T00:20:55.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52bb041640 10 monclient: monmap: 2024-04-26T00:20:55.841 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:55.841 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:55.841 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:55.865232+0000 2024-04-26T00:20:55.841 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:55.865232+0000 2024-04-26T00:20:55.841 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:55.841 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:55.842 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:55.842 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:55.842 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:55.842 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:55.842 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:55.842 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:55.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52bb041640 1 Processor -- start 2024-04-26T00:20:55.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52bb041640 1 -- start start 2024-04-26T00:20:55.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52bb041640 10 monclient: init 2024-04-26T00:20:55.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52bb041640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:55.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52bb041640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-04-26T00:20:55.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52bb041640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52b4136e10 0x7f52b409c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:55.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52bb041640 10 monclient(hunting): picked mon.noname-b con 0x7f52b4136e10 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:55.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52bb041640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f52b41371f0 0x7f52b41375d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:55.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52bb041640 10 monclient(hunting): picked mon.noname-a con 0x7f52b41371f0 addr v2:172.21.15.37:3300/0 2024-04-26T00:20:55.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52bb041640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f52b4135d90 addr v1:172.21.15.45:6789/0 2024-04-26T00:20:55.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52bb041640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52b409b680 con 0x7f52b4135d90 2024-04-26T00:20:55.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52bb041640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:55.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52bb041640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f52b409b8f0 con 0x7f52b41371f0 2024-04-26T00:20:55.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52bb041640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:55.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52bb041640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f52b409bbd0 con 0x7f52b4136e10 2024-04-26T00:20:55.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52bb041640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:55.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52bb041640 10 monclient(hunting): authenticate will time out at 1037.170962s 2024-04-26T00:20:55.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52b8ddd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52b4136e10 0x7f52b409c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:55.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52b3fff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f52b41371f0 0x7f52b41375d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:55.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52b8ddd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52b4136e10 0x7f52b409c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:33614/0 (socket says 172.21.15.37:33614) 2024-04-26T00:20:55.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52b8ddd640 1 -- 172.21.15.37:0/2108868626 learned_addr learned my addr 172.21.15.37:0/2108868626 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:55.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52b3fff640 10 monclient(hunting): get_auth_request con 0x7f52b41371f0 auth_method 0 2024-04-26T00:20:55.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:55.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52b3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:55.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:55.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52b8ddd640 10 monclient(hunting): get_auth_request con 0x7f52b4136e10 auth_method 0 2024-04-26T00:20:55.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52b8ddd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:55.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52b8ddd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:55.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52b8ddd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:55.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52b8ddd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:55.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52b8ddd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:55.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52b8ddd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:55.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:55.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:55.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:55.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52b8ddd640 10 monclient(hunting): handle_auth_done global_id 4445 payload 274 2024-04-26T00:20:55.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52b8ddd640 1 -- 172.21.15.37:0/2108868626 >> v2:172.21.15.37:3300/0 conn(0x7f52b41371f0 msgr2=0x7f52b41375d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:55.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52b8ddd640 1 --2- 172.21.15.37:0/2108868626 >> v2:172.21.15.37:3300/0 conn(0x7f52b41371f0 0x7f52b41375d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:55.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52b8ddd640 1 -- 172.21.15.37:0/2108868626 >> v1:172.21.15.45:6789/0 conn(0x7f52b4135d90 legacy=0x7f52b4136240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:55.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52b8ddd640 10 monclient: _finish_hunting 0 2024-04-26T00:20:55.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52b8ddd640 1 monclient: found mon.noname-b 2024-04-26T00:20:55.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52b8ddd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:55.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52b8ddd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:20:55.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52b8ddd640 1 -- 172.21.15.37:0/2108868626 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52b4137b10 con 0x7f52b4136e10 2024-04-26T00:20:55.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.863+0000 7f52b3fff640 1 --2- 172.21.15.37:0/2108868626 >> v2:172.21.15.37:3300/0 conn(0x7f52b41371f0 0x7f52b41375d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:55.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.867+0000 7f52b8ddd640 1 -- 172.21.15.37:0/2108868626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52b4136e10 msgr2=0x7f52b409c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:55.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.867+0000 7f52b8ddd640 1 --2- 172.21.15.37:0/2108868626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52b4136e10 0x7f52b409c5b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f52a4009870 tx=0x7f52a401c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=700bd486713194b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:55.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b37fe640 1 -- 172.21.15.37:0/2108868626 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52a40b1070 con 0x7f52b4136e10 2024-04-26T00:20:55.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:55.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:20:55.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b37fe640 10 monclient: dump: 2024-04-26T00:20:55.852 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:55.852 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:55.852 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:55.852 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:55.852 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:55.852 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:55.852 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:55.852 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:55.852 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:55.852 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:55.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b37fe640 10 monclient: _finish_auth 0 2024-04-26T00:20:55.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b37fe640 10 monclient: _check_auth_tickets 2024-04-26T00:20:55.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:55.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b37fe640 1 -- 172.21.15.37:0/2108868626 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52a40ac620 con 0x7f52b4136e10 2024-04-26T00:20:55.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52bb041640 5 monclient: authenticate success, global_id 4445 2024-04-26T00:20:55.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52bb041640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:20:55.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b37fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:55.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52bb041640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:55.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b37fe640 1 -- 172.21.15.37:0/2108868626 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52a40ac9f0 con 0x7f52b4136e10 2024-04-26T00:20:55.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52bb041640 4 set_mon_vals no callback set 2024-04-26T00:20:55.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:55.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:55.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b37fe640 10 monclient: dump: 2024-04-26T00:20:55.853 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:55.853 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:55.853 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:55.853 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:55.853 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:55.853 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:55.854 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:55.854 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:55.854 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:55.854 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:55.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52bb041640 10 monclient: shutdown 2024-04-26T00:20:55.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52bb041640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:55.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52bb041640 1 -- 172.21.15.37:0/2108868626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52b4136e10 msgr2=0x7f52b409c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:55.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52bb041640 1 --2- 172.21.15.37:0/2108868626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52b4136e10 0x7f52b409c5b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f52a4009870 tx=0x7f52a401c3e0 comp rx=0 tx=0).stop 2024-04-26T00:20:55.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52bb041640 1 -- 172.21.15.37:0/2108868626 shutdown_connections 2024-04-26T00:20:55.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52bb041640 1 --2- 172.21.15.37:0/2108868626 >> v2:172.21.15.37:3300/0 conn(0x7f52b41371f0 0x7f52b41375d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:55.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52bb041640 1 --2- 172.21.15.37:0/2108868626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52b4136e10 0x7f52b409c5b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:55.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52bb041640 1 -- 172.21.15.37:0/2108868626 >> 172.21.15.37:0/2108868626 conn(0x7f52b40a64e0 msgr2=0x7f52b40a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:55.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52bb041640 1 -- 172.21.15.37:0/2108868626 shutdown_connections 2024-04-26T00:20:55.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52bb041640 1 -- 172.21.15.37:0/2108868626 wait complete. 2024-04-26T00:20:55.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52bb041640 10 monclient: build_initial_monmap 2024-04-26T00:20:55.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52bb041640 10 monclient: monmap: 2024-04-26T00:20:55.855 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:55.855 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:55.855 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:55.855 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:55.855 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:55.855 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:55.855 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:55.855 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:55.855 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:55.855 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:55.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52bb041640 1 Processor -- start 2024-04-26T00:20:55.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52bb041640 1 -- start start 2024-04-26T00:20:55.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52bb041640 10 monclient: init 2024-04-26T00:20:55.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52bb041640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:55.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52bb041640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:20:55.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52bb041640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52b4135d90 0x7f52b41cf270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:55.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52bb041640 10 monclient(hunting): picked mon.noname-b con 0x7f52b4135d90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:55.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52bb041640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52b4136e10 0x7f52b41cf910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:55.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52bb041640 10 monclient(hunting): picked mon.noname-c con 0x7f52b4136e10 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:55.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52bb041640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f52b41371f0 0x7f52b413a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:55.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52bb041640 10 monclient(hunting): picked mon.noname-a con 0x7f52b41371f0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:55.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52bb041640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:55.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52bb041640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f52b4141f10 con 0x7f52b41371f0 2024-04-26T00:20:55.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52bb041640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:55.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52bb041640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f52b4141d90 con 0x7f52b4135d90 2024-04-26T00:20:55.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52bb041640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:55.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52bb041640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f52b4142090 con 0x7f52b4136e10 2024-04-26T00:20:55.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52bb041640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:55.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52bb041640 10 monclient(hunting): authenticate will time out at 1037.178272s 2024-04-26T00:20:55.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b3fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52b4136e10 0x7f52b41cf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:55.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b95de640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f52b41371f0 0x7f52b413a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:55.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b3fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52b4136e10 0x7f52b41cf910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:33620/0 (socket says 172.21.15.37:33620) 2024-04-26T00:20:55.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b3fff640 1 -- 172.21.15.37:0/1876219155 learned_addr learned my addr 172.21.15.37:0/1876219155 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:55.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b95de640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f52b41371f0 0x7f52b413a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:37590/0 (socket says 172.21.15.37:37590) 2024-04-26T00:20:55.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b3fff640 10 monclient(hunting): get_auth_request con 0x7f52b4136e10 auth_method 0 2024-04-26T00:20:55.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:55.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:55.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:55.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b95de640 10 monclient(hunting): get_auth_request con 0x7f52b41371f0 auth_method 0 2024-04-26T00:20:55.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b95de640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:55.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b95de640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:55.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b95de640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:55.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b95de640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:55.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b95de640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:55.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b95de640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:55.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:55.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:55.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:55.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b8ddd640 1 --2- 172.21.15.37:0/1876219155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52b4135d90 0x7f52b41cf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:55.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b8ddd640 10 monclient(hunting): get_auth_request con 0x7f52b4135d90 auth_method 0 2024-04-26T00:20:55.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b8ddd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:55.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b8ddd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:55.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b8ddd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:55.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b95de640 10 monclient(hunting): handle_auth_done global_id 4458 payload 824 2024-04-26T00:20:55.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b95de640 1 -- 172.21.15.37:0/1876219155 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52b4136e10 msgr2=0x7f52b41cf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:55.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b95de640 1 --2- 172.21.15.37:0/1876219155 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52b4136e10 0x7f52b41cf910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:55.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b95de640 1 -- 172.21.15.37:0/1876219155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52b4135d90 msgr2=0x7f52b41cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:55.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b95de640 1 --2- 172.21.15.37:0/1876219155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52b4135d90 0x7f52b41cf270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:55.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b95de640 10 monclient: _finish_hunting 0 2024-04-26T00:20:55.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b95de640 1 monclient: found mon.noname-a 2024-04-26T00:20:55.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b95de640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:55.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b95de640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:55.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b95de640 1 -- 172.21.15.37:0/1876219155 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52b413c540 con 0x7f52b41371f0 2024-04-26T00:20:55.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.871+0000 7f52b3fff640 1 --2- 172.21.15.37:0/1876219155 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52b4136e10 0x7f52b41cf910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:55.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.875+0000 7f52b95de640 1 -- 172.21.15.37:0/1876219155 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f52b41371f0 msgr2=0x7f52b413a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:55.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.875+0000 7f52b95de640 1 --2- 172.21.15.37:0/1876219155 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f52b41371f0 0x7f52b413a8a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f52ac00c780 tx=0x7f52ac00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:55.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.875+0000 7f52b17fa640 1 -- 172.21.15.37:0/1876219155 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52ac016020 con 0x7f52b41371f0 2024-04-26T00:20:55.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.875+0000 7f52b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:55.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.875+0000 7f52b17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:55.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.875+0000 7f52b17fa640 10 monclient: dump: 2024-04-26T00:20:55.863 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:55.863 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:55.863 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:55.863 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:55.863 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:55.863 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:55.864 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:55.864 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:55.864 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:55.864 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:55.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.875+0000 7f52b17fa640 10 monclient: _finish_auth 0 2024-04-26T00:20:55.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.875+0000 7f52b17fa640 10 monclient: _check_auth_tickets 2024-04-26T00:20:55.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.875+0000 7f52b17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:55.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.875+0000 7f52b17fa640 1 -- 172.21.15.37:0/1876219155 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52ac00ac60 con 0x7f52b41371f0 2024-04-26T00:20:55.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.875+0000 7f52bb041640 5 monclient: authenticate success, global_id 4458 2024-04-26T00:20:55.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.875+0000 7f52bb041640 10 monclient: _renew_subs 2024-04-26T00:20:55.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.875+0000 7f52bb041640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:55.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.875+0000 7f52bb041640 1 -- 172.21.15.37:0/1876219155 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52b409aca0 con 0x7f52b41371f0 2024-04-26T00:20:55.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.875+0000 7f52b17fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:55.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.875+0000 7f52b17fa640 1 -- 172.21.15.37:0/1876219155 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52ac012660 con 0x7f52b41371f0 2024-04-26T00:20:55.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.875+0000 7f52b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:55.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.875+0000 7f52b17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:55.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.875+0000 7f52b17fa640 10 monclient: dump: 2024-04-26T00:20:55.865 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:55.865 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:55.865 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:55.865 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:55.865 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:55.865 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:55.865 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:55.865 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:55.865 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:55.865 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:55.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.875+0000 7f52b2ffd640 4 set_mon_vals no callback set 2024-04-26T00:20:55.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.875+0000 7f52bb041640 10 monclient: _renew_subs 2024-04-26T00:20:55.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.875+0000 7f52bb041640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:55.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.875+0000 7f52bb041640 1 -- 172.21.15.37:0/1876219155 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52b413df90 con 0x7f52b41371f0 2024-04-26T00:20:55.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.875+0000 7f52b17fa640 1 -- 172.21.15.37:0/1876219155 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f52ac019020 con 0x7f52b41371f0 2024-04-26T00:20:55.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.875+0000 7f52bb041640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:55.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.875+0000 7f52bb041640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:55.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.875+0000 7f52bb041640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:55.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.875+0000 7f52b17fa640 1 --2- 172.21.15.37:0/1876219155 >> v2:172.21.15.45:6812/2379918623 conn(0x7f52a0035e80 0x7f52a0038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:55.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.875+0000 7f52bb041640 1 -- 172.21.15.37:0/1876219155 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52b4075910 con 0x7f52b41371f0 2024-04-26T00:20:55.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.875+0000 7f52b17fa640 1 -- 172.21.15.37:0/1876219155 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 3943+0+0 (secure 0 0 0) 0x7f52ac048970 con 0x7f52b41371f0 2024-04-26T00:20:55.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.875+0000 7f52b8ddd640 1 --2- 172.21.15.37:0/1876219155 >> v2:172.21.15.45:6812/2379918623 conn(0x7f52a0035e80 0x7f52a0038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:55.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.875+0000 7f52b8ddd640 10 monclient: get_auth_request con 0x7f52a0035e80 auth_method 0 2024-04-26T00:20:55.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.879+0000 7f52b8ddd640 1 --2- 172.21.15.37:0/1876219155 >> v2:172.21.15.45:6812/2379918623 conn(0x7f52a0035e80 0x7f52a0038340 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f52b409d490 tx=0x7f52a40a7000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:55.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.879+0000 7f52b17fa640 1 -- 172.21.15.37:0/1876219155 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f52ac020030 con 0x7f52b41371f0 2024-04-26T00:20:55.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.879+0000 7f52b17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:55.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:55.879+0000 7f52b17fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:56.036 INFO:tasks.rados.rados.0.smithi045.stdout:158: done (0 left) 2024-04-26T00:20:56.036 INFO:tasks.rados.rados.0.smithi045.stdout:159: delete oid 28 current snap is 15 2024-04-26T00:20:56.043 INFO:tasks.rados.rados.0.smithi045.stdout:159: done (0 left) 2024-04-26T00:20:56.043 INFO:tasks.rados.rados.0.smithi045.stdout:160: read oid 40 snap -1 2024-04-26T00:20:56.043 INFO:tasks.rados.rados.0.smithi045.stdout:160: expect deleted 2024-04-26T00:20:56.043 INFO:tasks.rados.rados.0.smithi045.stdout:161: read oid 7 snap -1 2024-04-26T00:20:56.043 INFO:tasks.rados.rados.0.smithi045.stdout:161: expect (ObjNum 70 snap 10 seq_num 70) 2024-04-26T00:20:56.043 INFO:tasks.rados.rados.0.smithi045.stdout:162: write oid 30 current snap is 15 2024-04-26T00:20:56.043 INFO:tasks.rados.rados.0.smithi045.stdout:162: seq_num 83 ranges {594981=609197,1974123=745028,3279873=115108} 2024-04-26T00:20:56.068 INFO:tasks.rados.rados.0.smithi045.stdout:162: writing smithi04526579-30 from 594981 to 1204178 tid 1 2024-04-26T00:20:56.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.107+0000 7f52bb041640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:20:56.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.107+0000 7f52bb041640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:20:56.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.107+0000 7f52bb041640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:56.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.107+0000 7f52bb041640 1 -- 172.21.15.37:0/1876219155 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f52b4075ee0 con 0x7f52b41371f0 2024-04-26T00:20:56.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.107+0000 7f52b17fa640 1 -- 172.21.15.37:0/1876219155 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f52ac00ae10 con 0x7f52b41371f0 2024-04-26T00:20:56.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.107+0000 7f52b17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:20:56.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.107+0000 7f52b17fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:56.085 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:20:56.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.107+0000 7f52bb041640 1 -- 172.21.15.37:0/1876219155 >> v2:172.21.15.45:6812/2379918623 conn(0x7f52a0035e80 msgr2=0x7f52a0038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:56.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.107+0000 7f52bb041640 1 --2- 172.21.15.37:0/1876219155 >> v2:172.21.15.45:6812/2379918623 conn(0x7f52a0035e80 0x7f52a0038340 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f52b409d490 tx=0x7f52a40a7000 comp rx=0 tx=0).stop 2024-04-26T00:20:56.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.107+0000 7f52bb041640 10 monclient: shutdown 2024-04-26T00:20:56.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.107+0000 7f52bb041640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:56.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.107+0000 7f52bb041640 1 -- 172.21.15.37:0/1876219155 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f52b41371f0 msgr2=0x7f52b413a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:56.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.107+0000 7f52bb041640 1 --2- 172.21.15.37:0/1876219155 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f52b41371f0 0x7f52b413a8a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f52ac00c780 tx=0x7f52ac00cc40 comp rx=0 tx=0).stop 2024-04-26T00:20:56.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.107+0000 7f52bb041640 1 -- 172.21.15.37:0/1876219155 shutdown_connections 2024-04-26T00:20:56.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.107+0000 7f52bb041640 1 --2- 172.21.15.37:0/1876219155 >> v2:172.21.15.45:6812/2379918623 conn(0x7f52a0035e80 0x7f52a0038340 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:56.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.107+0000 7f52bb041640 1 --2- 172.21.15.37:0/1876219155 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f52b41371f0 0x7f52b413a8a0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:56.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.107+0000 7f52bb041640 1 --2- 172.21.15.37:0/1876219155 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52b4136e10 0x7f52b41cf910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:56.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.107+0000 7f52bb041640 1 --2- 172.21.15.37:0/1876219155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52b4135d90 0x7f52b41cf270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:56.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.107+0000 7f52bb041640 1 -- 172.21.15.37:0/1876219155 >> 172.21.15.37:0/1876219155 conn(0x7f52b40a64e0 msgr2=0x7f52b40a4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:56.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.107+0000 7f52bb041640 1 -- 172.21.15.37:0/1876219155 shutdown_connections 2024-04-26T00:20:56.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.111+0000 7f52bb041640 1 -- 172.21.15.37:0/1876219155 wait complete. 2024-04-26T00:20:56.092 INFO:tasks.rados.rados.0.smithi045.stdout:162: writing smithi04526579-30 from 1974123 to 2719151 tid 2 2024-04-26T00:20:56.101 INFO:tasks.rados.rados.0.smithi045.stdout:162: writing smithi04526579-30 from 3279873 to 3394981 tid 3 2024-04-26T00:20:56.102 INFO:tasks.rados.rados.0.smithi045.stdout:163: rollback oid 1 current snap is 15 2024-04-26T00:20:56.102 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 1 to 14 2024-04-26T00:20:56.102 INFO:tasks.rados.rados.0.smithi045.stdout:161: expect (ObjNum 70 snap 10 seq_num 70) 2024-04-26T00:20:56.103 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:20:56.103 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-26T00:20:56.103 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:20:56.104 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:20:56.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.239+0000 7f3e75019640 10 monclient: get_monmap_and_config 2024-04-26T00:20:56.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.239+0000 7f3e75019640 10 monclient: build_initial_monmap 2024-04-26T00:20:56.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.239+0000 7f3e75019640 10 monclient: monmap: 2024-04-26T00:20:56.219 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:56.220 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:56.220 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:56.243561+0000 2024-04-26T00:20:56.220 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:56.243561+0000 2024-04-26T00:20:56.220 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:56.220 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:56.220 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:56.220 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:56.220 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:56.220 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:56.220 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:56.220 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:56.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e75019640 1 Processor -- start 2024-04-26T00:20:56.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e75019640 1 -- start start 2024-04-26T00:20:56.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e75019640 10 monclient: init 2024-04-26T00:20:56.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e75019640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:56.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e75019640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-04-26T00:20:56.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e75019640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f3e700a3d90 0x7f3e7009bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:56.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e75019640 10 monclient(hunting): picked mon.noname-a con 0x7f3e700a3d90 addr v2:172.21.15.37:3300/0 2024-04-26T00:20:56.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e75019640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3e700a4170 addr v1:172.21.15.45:6789/0 2024-04-26T00:20:56.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e75019640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3e7013c3e0 0x7f3e7013e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:56.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e75019640 10 monclient(hunting): picked mon.noname-c con 0x7f3e7013c3e0 addr v2:172.21.15.45:3300/0 2024-04-26T00:20:56.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e75019640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e700a5150 con 0x7f3e700a4170 2024-04-26T00:20:56.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e75019640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:56.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e75019640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f3e7009b280 con 0x7f3e700a3d90 2024-04-26T00:20:56.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e75019640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:56.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e75019640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f3e7009b560 con 0x7f3e7013c3e0 2024-04-26T00:20:56.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e75019640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:56.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e75019640 10 monclient(hunting): authenticate will time out at 1037.549204s 2024-04-26T00:20:56.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e6ed9d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f3e700a3d90 0x7f3e7009bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:56.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e6ed9d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f3e700a3d90 0x7f3e7009bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:37598/0 (socket says 172.21.15.37:37598) 2024-04-26T00:20:56.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e6ed9d640 1 -- 172.21.15.37:0/3247929103 learned_addr learned my addr 172.21.15.37:0/3247929103 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:56.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e6ed9d640 10 monclient(hunting): get_auth_request con 0x7f3e700a3d90 auth_method 0 2024-04-26T00:20:56.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e6ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:56.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e6ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:56.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e6ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:56.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e6f59e640 1 --2- 172.21.15.37:0/3247929103 >> v2:172.21.15.45:3300/0 conn(0x7f3e7013c3e0 0x7f3e7013e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:56.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e6ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:56.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e6ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:56.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e6ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:56.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e6f59e640 10 monclient(hunting): get_auth_request con 0x7f3e7013c3e0 auth_method 0 2024-04-26T00:20:56.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e6f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:56.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e6f59e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:56.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e6f59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:56.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e6ed9d640 10 monclient(hunting): handle_auth_done global_id 4461 payload 274 2024-04-26T00:20:56.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e6ed9d640 1 -- 172.21.15.37:0/3247929103 >> v2:172.21.15.45:3300/0 conn(0x7f3e7013c3e0 msgr2=0x7f3e7013e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:56.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e6ed9d640 1 --2- 172.21.15.37:0/3247929103 >> v2:172.21.15.45:3300/0 conn(0x7f3e7013c3e0 0x7f3e7013e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:56.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e6ed9d640 1 -- 172.21.15.37:0/3247929103 >> v1:172.21.15.45:6789/0 conn(0x7f3e700a4170 legacy=0x7f3e700a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:56.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e6ed9d640 10 monclient: _finish_hunting 0 2024-04-26T00:20:56.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e6ed9d640 1 monclient: found mon.noname-a 2024-04-26T00:20:56.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e6ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:56.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e6ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:56.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.243+0000 7f3e6ed9d640 1 -- 172.21.15.37:0/3247929103 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e7006dc60 con 0x7f3e700a3d90 2024-04-26T00:20:56.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.247+0000 7f3e6ed9d640 1 -- 172.21.15.37:0/3247929103 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e700a3d90 msgr2=0x7f3e7009bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:56.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.247+0000 7f3e6ed9d640 1 --2- 172.21.15.37:0/3247929103 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e700a3d90 0x7f3e7009bf40 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3e700a6e00 tx=0x7f3e6801c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aa452dfe9e943ca server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:56.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.247+0000 7f3e6dd9b640 1 -- 172.21.15.37:0/3247929103 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e680a5640 con 0x7f3e700a3d90 2024-04-26T00:20:56.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.247+0000 7f3e6dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:56.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.247+0000 7f3e6dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:56.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.247+0000 7f3e6dd9b640 10 monclient: dump: 2024-04-26T00:20:56.235 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:56.236 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:56.236 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.236 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.236 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:56.236 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:56.236 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:56.236 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:56.236 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:56.236 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:56.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.247+0000 7f3e6dd9b640 10 monclient: _finish_auth 0 2024-04-26T00:20:56.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.247+0000 7f3e6dd9b640 10 monclient: _check_auth_tickets 2024-04-26T00:20:56.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.247+0000 7f3e6dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:56.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.247+0000 7f3e75019640 5 monclient: authenticate success, global_id 4461 2024-04-26T00:20:56.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.247+0000 7f3e75019640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:20:56.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.247+0000 7f3e6dd9b640 1 -- 172.21.15.37:0/3247929103 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e680a57e0 con 0x7f3e700a3d90 2024-04-26T00:20:56.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.247+0000 7f3e6dd9b640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:56.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.247+0000 7f3e75019640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:56.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.247+0000 7f3e75019640 4 set_mon_vals no callback set 2024-04-26T00:20:56.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.247+0000 7f3e6dd9b640 1 -- 172.21.15.37:0/3247929103 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e680a5bb0 con 0x7f3e700a3d90 2024-04-26T00:20:56.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.247+0000 7f3e6dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:56.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.247+0000 7f3e6dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:56.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.247+0000 7f3e6dd9b640 10 monclient: dump: 2024-04-26T00:20:56.237 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:56.237 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:56.237 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.237 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.237 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:56.237 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:56.238 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:56.238 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:56.238 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:56.238 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:56.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.247+0000 7f3e75019640 10 monclient: shutdown 2024-04-26T00:20:56.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.247+0000 7f3e75019640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:56.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.247+0000 7f3e75019640 1 -- 172.21.15.37:0/3247929103 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e700a3d90 msgr2=0x7f3e7009bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:56.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.247+0000 7f3e75019640 1 --2- 172.21.15.37:0/3247929103 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e700a3d90 0x7f3e7009bf40 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3e700a6e00 tx=0x7f3e6801c870 comp rx=0 tx=0).stop 2024-04-26T00:20:56.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.247+0000 7f3e75019640 1 -- 172.21.15.37:0/3247929103 shutdown_connections 2024-04-26T00:20:56.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.247+0000 7f3e75019640 1 --2- 172.21.15.37:0/3247929103 >> v2:172.21.15.45:3300/0 conn(0x7f3e7013c3e0 0x7f3e7013e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:56.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.247+0000 7f3e75019640 1 --2- 172.21.15.37:0/3247929103 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e700a3d90 0x7f3e7009bf40 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:56.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.247+0000 7f3e75019640 1 -- 172.21.15.37:0/3247929103 >> 172.21.15.37:0/3247929103 conn(0x7f3e700aa0a0 msgr2=0x7f3e700a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:56.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.247+0000 7f3e75019640 1 -- 172.21.15.37:0/3247929103 shutdown_connections 2024-04-26T00:20:56.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.247+0000 7f3e75019640 1 -- 172.21.15.37:0/3247929103 wait complete. 2024-04-26T00:20:56.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.247+0000 7f3e75019640 10 monclient: build_initial_monmap 2024-04-26T00:20:56.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.247+0000 7f3e75019640 10 monclient: monmap: 2024-04-26T00:20:56.239 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:56.239 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:56.239 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:56.239 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:56.239 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:56.239 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:56.239 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:56.239 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:56.239 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:56.239 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:56.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.247+0000 7f3e75019640 1 Processor -- start 2024-04-26T00:20:56.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.247+0000 7f3e75019640 1 -- start start 2024-04-26T00:20:56.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.247+0000 7f3e75019640 10 monclient: init 2024-04-26T00:20:56.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e75019640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:56.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e75019640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:20:56.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e75019640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e700a3d90 0x7f3e7013ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:56.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e75019640 10 monclient(hunting): picked mon.noname-b con 0x7f3e700a3d90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:56.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e75019640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3e700a4170 0x7f3e70121840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:56.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e75019640 10 monclient(hunting): picked mon.noname-c con 0x7f3e700a4170 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:56.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e75019640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e7013c3e0 0x7f3e70121d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:56.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e75019640 10 monclient(hunting): picked mon.noname-a con 0x7f3e7013c3e0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:56.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e75019640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:56.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e75019640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f3e70141740 con 0x7f3e7013c3e0 2024-04-26T00:20:56.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e75019640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:56.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e75019640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7009c690 con 0x7f3e700a3d90 2024-04-26T00:20:56.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e75019640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:56.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e75019640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f3e701418c0 con 0x7f3e700a4170 2024-04-26T00:20:56.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e75019640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:56.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e75019640 10 monclient(hunting): authenticate will time out at 1037.556296s 2024-04-26T00:20:56.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e6f59e640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e7013c3e0 0x7f3e70121d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:56.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e6e59c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3e700a4170 0x7f3e70121840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:56.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e6f59e640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e7013c3e0 0x7f3e70121d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:37602/0 (socket says 172.21.15.37:37602) 2024-04-26T00:20:56.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e6f59e640 1 -- 172.21.15.37:0/106736405 learned_addr learned my addr 172.21.15.37:0/106736405 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:56.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e6f59e640 10 monclient(hunting): get_auth_request con 0x7f3e7013c3e0 auth_method 0 2024-04-26T00:20:56.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e6f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:56.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e6f59e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:56.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e6f59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:56.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e6e59c640 10 monclient(hunting): get_auth_request con 0x7f3e700a4170 auth_method 0 2024-04-26T00:20:56.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e6e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:56.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e6e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:56.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e6e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:56.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e6f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:56.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e6f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:56.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e6f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:56.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e6e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:56.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e6e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:56.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e6e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:56.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e6ed9d640 1 --2- 172.21.15.37:0/106736405 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e700a3d90 0x7f3e7013ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:56.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e6ed9d640 10 monclient(hunting): get_auth_request con 0x7f3e700a3d90 auth_method 0 2024-04-26T00:20:56.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e6ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:56.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e6ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:56.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e6ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:56.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e6f59e640 10 monclient(hunting): handle_auth_done global_id 4464 payload 824 2024-04-26T00:20:56.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e6f59e640 1 -- 172.21.15.37:0/106736405 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3e700a4170 msgr2=0x7f3e70121840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:56.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e6f59e640 1 --2- 172.21.15.37:0/106736405 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3e700a4170 0x7f3e70121840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:56.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e6f59e640 1 -- 172.21.15.37:0/106736405 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e700a3d90 msgr2=0x7f3e7013ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:56.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e6f59e640 1 --2- 172.21.15.37:0/106736405 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e700a3d90 0x7f3e7013ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:56.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e6f59e640 10 monclient: _finish_hunting 0 2024-04-26T00:20:56.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e6f59e640 1 monclient: found mon.noname-a 2024-04-26T00:20:56.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e6f59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:56.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e6f59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:56.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e6f59e640 1 -- 172.21.15.37:0/106736405 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e701254f0 con 0x7f3e7013c3e0 2024-04-26T00:20:56.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e6e59c640 1 --2- 172.21.15.37:0/106736405 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3e700a4170 0x7f3e70121840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:56.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e6f59e640 1 -- 172.21.15.37:0/106736405 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e7013c3e0 msgr2=0x7f3e70121d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:56.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e6f59e640 1 --2- 172.21.15.37:0/106736405 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e7013c3e0 0x7f3e70121d80 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3e6000d800 tx=0x7f3e6000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:56.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e677fe640 1 -- 172.21.15.37:0/106736405 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e6000b8d0 con 0x7f3e7013c3e0 2024-04-26T00:20:56.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:56.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e677fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:56.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e677fe640 10 monclient: dump: 2024-04-26T00:20:56.244 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:56.244 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:56.244 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.244 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.244 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:56.244 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:56.244 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:56.244 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:56.244 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:56.244 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:56.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e677fe640 10 monclient: _finish_auth 0 2024-04-26T00:20:56.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e677fe640 10 monclient: _check_auth_tickets 2024-04-26T00:20:56.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:56.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e677fe640 1 -- 172.21.15.37:0/106736405 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e60019070 con 0x7f3e7013c3e0 2024-04-26T00:20:56.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e677fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:56.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e677fe640 1 -- 172.21.15.37:0/106736405 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e6000bcb0 con 0x7f3e7013c3e0 2024-04-26T00:20:56.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e75019640 5 monclient: authenticate success, global_id 4464 2024-04-26T00:20:56.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e75019640 10 monclient: _renew_subs 2024-04-26T00:20:56.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e75019640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:56.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e75019640 1 -- 172.21.15.37:0/106736405 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e701257d0 con 0x7f3e7013c3e0 2024-04-26T00:20:56.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e6d59a640 4 set_mon_vals no callback set 2024-04-26T00:20:56.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:56.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e677fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:56.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e677fe640 10 monclient: dump: 2024-04-26T00:20:56.245 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:56.245 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:56.246 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.246 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.246 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:56.246 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:56.246 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:56.246 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:56.246 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:56.246 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:56.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e75019640 10 monclient: _renew_subs 2024-04-26T00:20:56.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e75019640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:56.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.251+0000 7f3e75019640 1 -- 172.21.15.37:0/106736405 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e701234e0 con 0x7f3e7013c3e0 2024-04-26T00:20:56.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.255+0000 7f3e75019640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:56.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.255+0000 7f3e75019640 1 --2- 172.21.15.37:0/106736405 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e70124f30 0x7f3e701222c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:56.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.255+0000 7f3e75019640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:56.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.255+0000 7f3e75019640 1 -- 172.21.15.37:0/106736405 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7004f510 con 0x7f3e70124f30 2024-04-26T00:20:56.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.255+0000 7f3e677fe640 1 -- 172.21.15.37:0/106736405 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f3e6001e460 con 0x7f3e7013c3e0 2024-04-26T00:20:56.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.255+0000 7f3e6ed9d640 1 --2- 172.21.15.37:0/106736405 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e70124f30 0x7f3e701222c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:56.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.255+0000 7f3e6ed9d640 10 monclient: get_auth_request con 0x7f3e70124f30 auth_method 0 2024-04-26T00:20:56.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.255+0000 7f3e6ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:56.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.255+0000 7f3e6ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:56.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.255+0000 7f3e6ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:56.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.255+0000 7f3e6ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:56.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.255+0000 7f3e6ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:56.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.255+0000 7f3e677fe640 1 --2- 172.21.15.37:0/106736405 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3e50035e80 0x7f3e50038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:56.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.255+0000 7f3e6ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:56.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.255+0000 7f3e677fe640 1 -- 172.21.15.37:0/106736405 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 3915+0+0 (secure 0 0 0) 0x7f3e60048ee0 con 0x7f3e7013c3e0 2024-04-26T00:20:56.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.255+0000 7f3e6ed9d640 10 monclient(hunting): handle_auth_done global_id 4467 payload 824 2024-04-26T00:20:56.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.255+0000 7f3e6ed9d640 1 -- 172.21.15.37:0/106736405 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3e700008e0 con 0x7f3e70124f30 2024-04-26T00:20:56.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.255+0000 7f3e6e59c640 1 --2- 172.21.15.37:0/106736405 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3e50035e80 0x7f3e50038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:56.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.255+0000 7f3e6e59c640 10 monclient: get_auth_request con 0x7f3e50035e80 auth_method 0 2024-04-26T00:20:56.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.255+0000 7f3e6ed9d640 1 -- 172.21.15.37:0/106736405 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e70124f30 msgr2=0x7f3e701222c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:56.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.255+0000 7f3e6ed9d640 1 --2- 172.21.15.37:0/106736405 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e70124f30 0x7f3e701222c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3e680ae000 tx=0x7f3e68019610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:56.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.255+0000 7f3e677fe640 1 -- 172.21.15.37:0/106736405 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e680bd070 con 0x7f3e70124f30 2024-04-26T00:20:56.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.255+0000 7f3e677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:56.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.255+0000 7f3e677fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:56.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.255+0000 7f3e677fe640 10 monclient: dump: 2024-04-26T00:20:56.248 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:56.248 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:56.248 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.248 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.248 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:56.249 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:56.249 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:56.249 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:56.249 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:56.249 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:56.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.255+0000 7f3e677fe640 1 -- 172.21.15.37:0/106736405 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3e680b6830 con 0x7f3e70124f30 2024-04-26T00:20:56.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.255+0000 7f3e677fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:56.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.255+0000 7f3e677fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:56.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.255+0000 7f3e677fe640 1 -- 172.21.15.37:0/106736405 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e70124f30 msgr2=0x7f3e701222c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:56.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.255+0000 7f3e677fe640 1 --2- 172.21.15.37:0/106736405 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e70124f30 0x7f3e701222c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3e680ae000 tx=0x7f3e68019610 comp rx=0 tx=0).stop 2024-04-26T00:20:56.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.255+0000 7f3e677fe640 1 -- 172.21.15.37:0/106736405 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e70124f30 msgr2=0x7f3e701222c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:20:56.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.255+0000 7f3e677fe640 1 --2- 172.21.15.37:0/106736405 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e70124f30 0x7f3e701222c0 secure :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3e680ae000 tx=0x7f3e68019610 comp rx=0 tx=0).stop 2024-04-26T00:20:56.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.255+0000 7f3e6e59c640 1 --2- 172.21.15.37:0/106736405 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3e50035e80 0x7f3e50038340 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3e701230e0 tx=0x7f3e58007e40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:56.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.263+0000 7f3e75019640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:20:56.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.263+0000 7f3e75019640 1 --2- 172.21.15.37:0/106736405 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e70124260 0x7f3e70122800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:56.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.263+0000 7f3e75019640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:56.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.263+0000 7f3e75019640 1 -- 172.21.15.37:0/106736405 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7013b4d0 con 0x7f3e70124260 2024-04-26T00:20:56.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.263+0000 7f3e6ed9d640 1 --2- 172.21.15.37:0/106736405 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e70124260 0x7f3e70122800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:56.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.263+0000 7f3e6ed9d640 10 monclient: get_auth_request con 0x7f3e70124260 auth_method 0 2024-04-26T00:20:56.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.263+0000 7f3e6ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:56.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.263+0000 7f3e6ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:56.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.263+0000 7f3e6ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:56.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.263+0000 7f3e6ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:56.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.263+0000 7f3e6ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:56.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.263+0000 7f3e6ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:56.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.263+0000 7f3e6ed9d640 10 monclient(hunting): handle_auth_done global_id 4470 payload 824 2024-04-26T00:20:56.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.263+0000 7f3e6ed9d640 1 -- 172.21.15.37:0/106736405 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3e701d9e70 con 0x7f3e70124260 2024-04-26T00:20:56.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.263+0000 7f3e6ed9d640 1 -- 172.21.15.37:0/106736405 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e70124260 msgr2=0x7f3e70122800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:56.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.263+0000 7f3e6ed9d640 1 --2- 172.21.15.37:0/106736405 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e70124260 0x7f3e70122800 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3e6801c8b0 tx=0x7f3e680a7000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:56.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.263+0000 7f3e677fe640 1 -- 172.21.15.37:0/106736405 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e68004040 con 0x7f3e70124260 2024-04-26T00:20:56.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.263+0000 7f3e677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:56.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.263+0000 7f3e677fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:56.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.263+0000 7f3e677fe640 10 monclient: dump: 2024-04-26T00:20:56.252 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:56.253 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:56.253 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.253 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.253 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:56.253 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:56.253 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:56.253 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:56.253 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:56.253 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:56.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.263+0000 7f3e677fe640 1 -- 172.21.15.37:0/106736405 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4888 (secure 0 0 0) 0x7f3e68003fc0 con 0x7f3e70124260 2024-04-26T00:20:56.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.263+0000 7f3e677fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:20:56.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.263+0000 7f3e677fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:56.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.263+0000 7f3e677fe640 1 -- 172.21.15.37:0/106736405 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e70124260 msgr2=0x7f3e70122800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:56.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.263+0000 7f3e677fe640 1 --2- 172.21.15.37:0/106736405 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e70124260 0x7f3e70122800 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3e6801c8b0 tx=0x7f3e680a7000 comp rx=0 tx=0).stop 2024-04-26T00:20:56.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.263+0000 7f3e677fe640 1 -- 172.21.15.37:0/106736405 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e70124260 msgr2=0x7f3e70122800 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:20:56.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.263+0000 7f3e677fe640 1 --2- 172.21.15.37:0/106736405 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e70124260 0x7f3e70122800 secure :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3e6801c8b0 tx=0x7f3e680a7000 comp rx=0 tx=0).stop 2024-04-26T00:20:56.254 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:20:56.254 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:20:56.254 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:20:56.254 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:20:56.254 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 12, 2024-04-26T00:20:56.255 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 73367, 2024-04-26T00:20:56.255 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:20:56.255 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:20:56.255 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:20:56.255 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:20:56.255 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:56.255 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 5, 2024-04-26T00:20:56.255 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:20:56.255 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:20:56.255 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:20:56.255 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:20:56.255 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:20:56.255 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:20:56.255 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:20:56.256 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:20:56.256 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:20:56.256 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:20:56.256 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:20:56.256 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:20:56.256 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:20:56.256 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:20:56.256 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:56.256 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:20:56.256 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:20:56.256 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:20:56.256 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:20:56.256 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:20:56.256 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:20:56.257 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:20:56.257 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:20:56.257 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:20:56.257 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:20:56.257 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:20:56.257 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:20:56.257 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:20:56.257 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:56.257 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:56.257 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:20:56.257 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:20:56.257 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:20:56.257 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:20:56.257 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:20:56.257 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:20:56.258 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:20:56.258 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:20:56.258 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:20:56.258 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:20:56.258 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:20:56.258 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:20:56.258 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:20:56.258 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:20:56.258 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:20:56.258 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:20:56.258 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:20:56.258 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:20:56.258 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:20:56.258 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:20:56.259 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:20:56.259 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:20:56.259 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:20:56.259 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:20:56.259 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:20:56.259 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:20:56.259 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:20:56.259 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:20:56.259 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:56.259 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:20:56.259 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:56.259 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:20:56.259 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:56.259 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:20:56.260 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:20:56.260 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:20:56.260 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:20:56.260 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:56.260 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:20:56.260 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:20:56.260 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:56.260 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:56.260 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:56.260 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:20:56.260 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:20:56.260 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:56.260 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:56.260 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:56.261 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:56.261 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:20:56.261 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:20:56.261 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:20:56.261 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:20:56.261 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:20:56.261 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:56.261 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:56.261 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:20:56.261 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:20:56.261 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:20:56.261 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:20:56.261 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:56.261 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:20:56.261 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:20:56.262 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:56.262 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:56.262 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:56.262 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:20:56.262 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:20:56.262 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:56.262 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:56.262 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:56.262 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:56.262 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:20:56.262 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:20:56.262 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:20:56.262 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:20:56.263 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:20:56.263 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:56.263 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:56.263 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:20:56.263 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:20:56.263 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:20:56.263 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:20:56.263 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:56.263 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:20:56.263 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:20:56.263 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:56.263 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:56.263 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:56.263 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:20:56.263 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:20:56.264 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:56.264 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:56.264 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:56.264 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:56.264 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:20:56.264 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:20:56.264 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:20:56.264 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:20:56.264 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:20:56.264 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:56.264 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:56.264 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:56.264 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:20:56.264 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:20:56.265 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:56.265 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:20:56.265 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:20:56.265 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:20:56.265 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:56.265 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:56.265 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:20:56.265 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:56.265 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:20:56.265 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:20:56.265 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:20:56.265 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:56.265 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:56.265 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:20:56.266 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:56.266 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:20:56.266 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:20:56.266 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:20:56.266 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:56.266 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:56.266 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:20:56.266 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:56.266 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:20:56.266 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:20:56.266 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:20:56.266 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:56.266 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:56.267 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:56.267 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:20:56.267 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:20:56.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.267+0000 7f3e75019640 1 -- 172.21.15.37:0/106736405 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3e50035e80 msgr2=0x7f3e50038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:56.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.267+0000 7f3e75019640 1 --2- 172.21.15.37:0/106736405 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3e50035e80 0x7f3e50038340 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3e701230e0 tx=0x7f3e58007e40 comp rx=0 tx=0).stop 2024-04-26T00:20:56.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.267+0000 7f3e75019640 10 monclient: shutdown 2024-04-26T00:20:56.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.267+0000 7f3e75019640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:56.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.267+0000 7f3e75019640 1 -- 172.21.15.37:0/106736405 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e7013c3e0 msgr2=0x7f3e70121d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:56.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.267+0000 7f3e75019640 1 --2- 172.21.15.37:0/106736405 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e7013c3e0 0x7f3e70121d80 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3e6000d800 tx=0x7f3e6000dcc0 comp rx=0 tx=0).stop 2024-04-26T00:20:56.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.267+0000 7f3e6f59e640 1 -- 172.21.15.37:0/106736405 reap_dead start 2024-04-26T00:20:56.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.267+0000 7f3e6f59e640 1 -- 172.21.15.37:0/106736405 reap_dead start 2024-04-26T00:20:56.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.267+0000 7f3e75019640 1 -- 172.21.15.37:0/106736405 shutdown_connections 2024-04-26T00:20:56.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.267+0000 7f3e75019640 1 -- 172.21.15.37:0/106736405 >> 172.21.15.37:0/106736405 conn(0x7f3e700aa0a0 msgr2=0x7f3e700a8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:56.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.267+0000 7f3e75019640 1 -- 172.21.15.37:0/106736405 shutdown_connections 2024-04-26T00:20:56.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.267+0000 7f3e75019640 1 -- 172.21.15.37:0/106736405 wait complete. 2024-04-26T00:20:56.269 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:20:56.269 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:20:56.269 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:20:56.271 INFO:tasks.rados.rados.0.smithi045.stdout:162: finishing write tid 1 to smithi04526579-30 2024-04-26T00:20:56.271 INFO:tasks.rados.rados.0.smithi045.stdout:162: finishing write tid 2 to smithi04526579-30 2024-04-26T00:20:56.271 INFO:tasks.rados.rados.0.smithi045.stdout:163: finishing rollback tid 1 to smithi04526579-1 2024-04-26T00:20:56.271 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 0 (ObjNum 129 snap 0 seq_num 1476942376) dirty dne 2024-04-26T00:20:56.271 INFO:tasks.rados.rados.0.smithi045.stdout:162: finishing write tid 3 to smithi04526579-30 2024-04-26T00:20:56.271 INFO:tasks.rados.rados.0.smithi045.stdout:162: finishing write tid 4 to smithi04526579-30 2024-04-26T00:20:56.271 INFO:tasks.rados.rados.0.smithi045.stdout:162: finishing write tid 5 to smithi04526579-30 2024-04-26T00:20:56.271 INFO:tasks.rados.rados.0.smithi045.stdout:162: oid 30 updating version 0 to 34 2024-04-26T00:20:56.271 INFO:tasks.rados.rados.0.smithi045.stdout:162: oid 30 version 34 is already newer than 31 2024-04-26T00:20:56.271 INFO:tasks.rados.rados.0.smithi045.stdout:162: oid 30 version 34 is already newer than 32 2024-04-26T00:20:56.271 INFO:tasks.rados.rados.0.smithi045.stdout:162: oid 30 version 34 is already newer than 33 2024-04-26T00:20:56.271 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 34 (ObjNum 83 snap 15 seq_num 83) dirty exists 2024-04-26T00:20:56.271 INFO:tasks.rados.rados.0.smithi045.stdout:162: left oid 30 (ObjNum 83 snap 15 seq_num 83) 2024-04-26T00:20:56.272 INFO:tasks.rados.rados.0.smithi045.stdout:160: done (3 left) 2024-04-26T00:20:56.272 INFO:tasks.rados.rados.0.smithi045.stdout:161: done (2 left) 2024-04-26T00:20:56.272 INFO:tasks.rados.rados.0.smithi045.stdout:162: done (1 left) 2024-04-26T00:20:56.272 INFO:tasks.rados.rados.0.smithi045.stdout:163: done (0 left) 2024-04-26T00:20:56.272 INFO:tasks.rados.rados.0.smithi045.stdout:164: rollback oid 42 current snap is 15 2024-04-26T00:20:56.272 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 42 to 14 2024-04-26T00:20:56.272 INFO:tasks.rados.rados.0.smithi045.stdout:165: snap_remove snap 13 2024-04-26T00:20:56.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.403+0000 7fdedce1a640 10 monclient: get_monmap_and_config 2024-04-26T00:20:56.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.403+0000 7fdedce1a640 10 monclient: build_initial_monmap 2024-04-26T00:20:56.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.403+0000 7fdedce1a640 10 monclient: monmap: 2024-04-26T00:20:56.383 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:56.384 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:56.384 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:56.407817+0000 2024-04-26T00:20:56.384 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:56.407817+0000 2024-04-26T00:20:56.384 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:56.384 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:56.384 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:56.384 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:56.384 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:56.384 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:56.384 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:56.384 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:56.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.407+0000 7fdedce1a640 1 Processor -- start 2024-04-26T00:20:56.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.407+0000 7fdedce1a640 1 -- start start 2024-04-26T00:20:56.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.407+0000 7fdedce1a640 10 monclient: init 2024-04-26T00:20:56.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.407+0000 7fdedce1a640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:56.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.407+0000 7fdedce1a640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-04-26T00:20:56.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.407+0000 7fdedce1a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fded8136e80 addr v1:172.21.15.37:6789/0 2024-04-26T00:20:56.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.407+0000 7fdedce1a640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fded8072e70 0x7fded8137260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:56.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.407+0000 7fdedce1a640 10 monclient(hunting): picked mon.noname-a con 0x7fded8072e70 addr v2:172.21.15.37:3300/0 2024-04-26T00:20:56.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.407+0000 7fdedce1a640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fded813c1a0 0x7fded813e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:56.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.407+0000 7fdedce1a640 10 monclient(hunting): picked mon.noname-c con 0x7fded813c1a0 addr v2:172.21.15.45:3300/0 2024-04-26T00:20:56.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.407+0000 7fdedce1a640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fded81378b0 con 0x7fded8136e80 2024-04-26T00:20:56.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.407+0000 7fdedce1a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:56.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.407+0000 7fdedce1a640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fded806e2f0 con 0x7fded8072e70 2024-04-26T00:20:56.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.407+0000 7fdedce1a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:56.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.407+0000 7fdedce1a640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fded806d710 con 0x7fded813c1a0 2024-04-26T00:20:56.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.407+0000 7fdedce1a640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:56.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.407+0000 7fdedce1a640 10 monclient(hunting): authenticate will time out at 1037.713388s 2024-04-26T00:20:56.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.407+0000 7fded5d9b640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fded8072e70 0x7fded8137260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:56.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.407+0000 7fded659c640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fded8136e80 0x7fded8072760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:54978/0 (socket says 172.21.15.37:54978) 2024-04-26T00:20:56.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.407+0000 7fded659c640 1 -- 172.21.15.37:0/3190726021 learned_addr learned my addr 172.21.15.37:0/3190726021 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:20:56.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.407+0000 7fded5d9b640 10 monclient(hunting): get_auth_request con 0x7fded8072e70 auth_method 0 2024-04-26T00:20:56.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.407+0000 7fded5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:56.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.407+0000 7fded5d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:56.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.407+0000 7fded5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:56.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.407+0000 7fded5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:56.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.407+0000 7fded5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:56.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.407+0000 7fded5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:56.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.407+0000 7fded5d9b640 10 monclient(hunting): handle_auth_done global_id 4473 payload 274 2024-04-26T00:20:56.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.407+0000 7fded6d9d640 1 --2- 172.21.15.37:0/3190726021 >> v2:172.21.15.45:3300/0 conn(0x7fded813c1a0 0x7fded813e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:56.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.407+0000 7fded5d9b640 1 -- 172.21.15.37:0/3190726021 >> v2:172.21.15.45:3300/0 conn(0x7fded813c1a0 msgr2=0x7fded813e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:56.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.407+0000 7fded5d9b640 1 --2- 172.21.15.37:0/3190726021 >> v2:172.21.15.45:3300/0 conn(0x7fded813c1a0 0x7fded813e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:56.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.407+0000 7fded5d9b640 1 -- 172.21.15.37:0/3190726021 >> v1:172.21.15.37:6789/0 conn(0x7fded8136e80 legacy=0x7fded8072760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:56.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.407+0000 7fded5d9b640 10 monclient: _finish_hunting 0 2024-04-26T00:20:56.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.407+0000 7fded5d9b640 1 monclient: found mon.noname-a 2024-04-26T00:20:56.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.407+0000 7fded5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:56.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.407+0000 7fded5d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:56.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.407+0000 7fded5d9b640 1 -- 172.21.15.37:0/3190726021 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fded80a3840 con 0x7fded8072e70 2024-04-26T00:20:56.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.411+0000 7fded5d9b640 1 -- 172.21.15.37:0/3190726021 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fded8072e70 msgr2=0x7fded8137260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:56.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.411+0000 7fded5d9b640 1 --2- 172.21.15.37:0/3190726021 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fded8072e70 0x7fded8137260 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fded8075c10 tx=0x7fdecc01c300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6a466d9cc29bf7e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:56.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.411+0000 7fded559a640 1 -- 172.21.15.37:0/3190726021 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdecc0b0070 con 0x7fded8072e70 2024-04-26T00:20:56.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.411+0000 7fded559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:56.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.411+0000 7fded559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:56.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.411+0000 7fded559a640 10 monclient: dump: 2024-04-26T00:20:56.391 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:56.392 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:56.392 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.392 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.392 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:56.392 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:56.392 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:56.392 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:56.392 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:56.392 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:56.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.411+0000 7fded559a640 10 monclient: _finish_auth 0 2024-04-26T00:20:56.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.411+0000 7fded559a640 10 monclient: _check_auth_tickets 2024-04-26T00:20:56.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.411+0000 7fded559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:56.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.411+0000 7fded559a640 1 -- 172.21.15.37:0/3190726021 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdecc0ab600 con 0x7fded8072e70 2024-04-26T00:20:56.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.411+0000 7fded559a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:56.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.411+0000 7fded559a640 1 -- 172.21.15.37:0/3190726021 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdecc0ab9d0 con 0x7fded8072e70 2024-04-26T00:20:56.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.411+0000 7fdedce1a640 5 monclient: authenticate success, global_id 4473 2024-04-26T00:20:56.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.411+0000 7fdedce1a640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:56.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.411+0000 7fdedce1a640 4 set_mon_vals no callback set 2024-04-26T00:20:56.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.411+0000 7fded559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:56.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.411+0000 7fded559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:56.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.411+0000 7fded559a640 10 monclient: dump: 2024-04-26T00:20:56.393 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:56.393 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:56.393 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.393 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.393 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:56.393 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:56.393 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:56.393 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:56.393 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:56.393 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:56.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.411+0000 7fdedce1a640 10 monclient: shutdown 2024-04-26T00:20:56.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.411+0000 7fdedce1a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:56.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.411+0000 7fdedce1a640 1 -- 172.21.15.37:0/3190726021 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fded8072e70 msgr2=0x7fded8137260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:56.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.411+0000 7fdedce1a640 1 --2- 172.21.15.37:0/3190726021 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fded8072e70 0x7fded8137260 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fded8075c10 tx=0x7fdecc01c300 comp rx=0 tx=0).stop 2024-04-26T00:20:56.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.411+0000 7fdedce1a640 1 -- 172.21.15.37:0/3190726021 shutdown_connections 2024-04-26T00:20:56.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.411+0000 7fdedce1a640 1 --2- 172.21.15.37:0/3190726021 >> v2:172.21.15.45:3300/0 conn(0x7fded813c1a0 0x7fded813e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:56.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.411+0000 7fdedce1a640 1 --2- 172.21.15.37:0/3190726021 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fded8072e70 0x7fded8137260 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:56.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.411+0000 7fdedce1a640 1 -- 172.21.15.37:0/3190726021 >> 172.21.15.37:0/3190726021 conn(0x7fded80a7160 msgr2=0x7fded80a7540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:56.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.411+0000 7fdedce1a640 1 -- 172.21.15.37:0/3190726021 shutdown_connections 2024-04-26T00:20:56.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.411+0000 7fdedce1a640 1 -- 172.21.15.37:0/3190726021 wait complete. 2024-04-26T00:20:56.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.411+0000 7fdedce1a640 10 monclient: build_initial_monmap 2024-04-26T00:20:56.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.411+0000 7fdedce1a640 10 monclient: monmap: 2024-04-26T00:20:56.394 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:56.394 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:56.395 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:56.395 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:56.395 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:56.395 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:56.395 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:56.395 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:56.395 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:56.395 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:56.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.411+0000 7fdedce1a640 1 Processor -- start 2024-04-26T00:20:56.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.411+0000 7fdedce1a640 1 -- start start 2024-04-26T00:20:56.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.411+0000 7fdedce1a640 10 monclient: init 2024-04-26T00:20:56.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdedce1a640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:56.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdedce1a640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:20:56.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdedce1a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fded8072e70 0x7fded81cefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:56.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdedce1a640 10 monclient(hunting): picked mon.noname-c con 0x7fded8072e70 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:56.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdedce1a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fded8136e80 0x7fded81cf510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:56.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdedce1a640 10 monclient(hunting): picked mon.noname-b con 0x7fded8136e80 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:56.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdedce1a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fded813c1a0 0x7fded81cfa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:56.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdedce1a640 10 monclient(hunting): picked mon.noname-a con 0x7fded813c1a0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:56.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdedce1a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:56.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdedce1a640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fded8141680 con 0x7fded813c1a0 2024-04-26T00:20:56.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdedce1a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:56.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdedce1a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fded8141500 con 0x7fded8136e80 2024-04-26T00:20:56.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdedce1a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:56.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdedce1a640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fded8141800 con 0x7fded8072e70 2024-04-26T00:20:56.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdedce1a640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:56.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdedce1a640 10 monclient(hunting): authenticate will time out at 1037.720269s 2024-04-26T00:20:56.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fded659c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fded8072e70 0x7fded81cefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:56.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fded6d9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fded813c1a0 0x7fded81cfa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:56.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fded659c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fded8072e70 0x7fded81cefd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:33632/0 (socket says 172.21.15.37:33632) 2024-04-26T00:20:56.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fded659c640 1 -- 172.21.15.37:0/3912978951 learned_addr learned my addr 172.21.15.37:0/3912978951 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:56.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fded6d9d640 10 monclient(hunting): get_auth_request con 0x7fded813c1a0 auth_method 0 2024-04-26T00:20:56.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fded6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:56.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fded6d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:56.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fded6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:56.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fded659c640 10 monclient(hunting): get_auth_request con 0x7fded8072e70 auth_method 0 2024-04-26T00:20:56.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fded659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:56.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fded659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:56.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fded659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:56.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fded6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:56.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fded6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:56.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fded6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:56.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fded659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:56.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fded659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:56.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fded659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:56.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fded5d9b640 1 --2- 172.21.15.37:0/3912978951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fded8136e80 0x7fded81cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:56.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fded6d9d640 10 monclient(hunting): handle_auth_done global_id 4476 payload 824 2024-04-26T00:20:56.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fded6d9d640 1 -- 172.21.15.37:0/3912978951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fded8072e70 msgr2=0x7fded81cefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:56.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fded6d9d640 1 --2- 172.21.15.37:0/3912978951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fded8072e70 0x7fded81cefd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:56.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fded6d9d640 1 -- 172.21.15.37:0/3912978951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fded8136e80 msgr2=0x7fded81cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:56.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fded6d9d640 1 --2- 172.21.15.37:0/3912978951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fded8136e80 0x7fded81cf510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:56.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fded6d9d640 10 monclient: _finish_hunting 0 2024-04-26T00:20:56.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fded6d9d640 1 monclient: found mon.noname-a 2024-04-26T00:20:56.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fded6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:56.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fded6d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:20:56.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fded6d9d640 1 -- 172.21.15.37:0/3912978951 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fded80a68d0 con 0x7fded813c1a0 2024-04-26T00:20:56.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fded5d9b640 10 monclient: get_auth_request con 0x7fded8136e80 auth_method 0 2024-04-26T00:20:56.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fded5d9b640 1 --2- 172.21.15.37:0/3912978951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fded8136e80 0x7fded81cf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:20:56.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fded659c640 1 --2- 172.21.15.37:0/3912978951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fded8072e70 0x7fded81cefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:56.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fded6d9d640 1 -- 172.21.15.37:0/3912978951 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fded813c1a0 msgr2=0x7fded81cfa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:56.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fded6d9d640 1 --2- 172.21.15.37:0/3912978951 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fded813c1a0 0x7fded81cfa50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fded000ccb0 tx=0x7fded000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:56.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdec6ffd640 1 -- 172.21.15.37:0/3912978951 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded0003ba0 con 0x7fded813c1a0 2024-04-26T00:20:56.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdec6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:56.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdec6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:20:56.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdec6ffd640 10 monclient: dump: 2024-04-26T00:20:56.404 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:56.404 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:56.404 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.404 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.404 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:56.404 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:56.404 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:56.404 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:56.404 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:56.404 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:56.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdec6ffd640 10 monclient: _finish_auth 0 2024-04-26T00:20:56.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdec6ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:20:56.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdec6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:56.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdec6ffd640 1 -- 172.21.15.37:0/3912978951 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fded0003d40 con 0x7fded813c1a0 2024-04-26T00:20:56.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdec6ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:56.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdedce1a640 5 monclient: authenticate success, global_id 4476 2024-04-26T00:20:56.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdec6ffd640 1 -- 172.21.15.37:0/3912978951 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded0004240 con 0x7fded813c1a0 2024-04-26T00:20:56.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdec6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:56.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdec6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:20:56.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdec6ffd640 10 monclient: dump: 2024-04-26T00:20:56.405 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:56.405 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:56.405 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.405 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.405 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:56.405 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:56.405 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:56.405 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:56.405 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:56.405 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:56.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdedce1a640 10 monclient: _renew_subs 2024-04-26T00:20:56.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdedce1a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:56.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdedce1a640 1 -- 172.21.15.37:0/3912978951 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fded813a8c0 con 0x7fded813c1a0 2024-04-26T00:20:56.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdec7fff640 4 set_mon_vals no callback set 2024-04-26T00:20:56.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdedce1a640 10 monclient: _renew_subs 2024-04-26T00:20:56.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdedce1a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:20:56.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdedce1a640 1 -- 172.21.15.37:0/3912978951 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fded813bad0 con 0x7fded813c1a0 2024-04-26T00:20:56.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdedce1a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:56.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdedce1a640 1 --2- 172.21.15.37:0/3912978951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fded8139b80 0x7fded81364c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:56.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdedce1a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:56.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.415+0000 7fdedce1a640 1 -- 172.21.15.37:0/3912978951 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fded813b1d0 con 0x7fded8139b80 2024-04-26T00:20:56.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.419+0000 7fded659c640 1 --2- 172.21.15.37:0/3912978951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fded8139b80 0x7fded81364c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:56.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.419+0000 7fded659c640 10 monclient: get_auth_request con 0x7fded8139b80 auth_method 0 2024-04-26T00:20:56.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.419+0000 7fded659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:56.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.419+0000 7fded659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:56.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.419+0000 7fded659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:56.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.419+0000 7fded659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:56.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.419+0000 7fded659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:56.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.419+0000 7fded659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:56.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.419+0000 7fdec6ffd640 1 -- 172.21.15.37:0/3912978951 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fded0004420 con 0x7fded813c1a0 2024-04-26T00:20:56.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.419+0000 7fded659c640 10 monclient(hunting): handle_auth_done global_id 4457 payload 824 2024-04-26T00:20:56.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.419+0000 7fded659c640 1 -- 172.21.15.37:0/3912978951 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fded8073300 con 0x7fded8139b80 2024-04-26T00:20:56.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.419+0000 7fdec6ffd640 1 --2- 172.21.15.37:0/3912978951 >> v2:172.21.15.45:6812/2379918623 conn(0x7fdeb8035ed0 0x7fdeb8038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:56.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.419+0000 7fdec6ffd640 1 -- 172.21.15.37:0/3912978951 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 3915+0+0 (secure 0 0 0) 0x7fded0049480 con 0x7fded813c1a0 2024-04-26T00:20:56.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.419+0000 7fded5d9b640 1 --2- 172.21.15.37:0/3912978951 >> v2:172.21.15.45:6812/2379918623 conn(0x7fdeb8035ed0 0x7fdeb8038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:56.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.419+0000 7fded5d9b640 10 monclient: get_auth_request con 0x7fdeb8035ed0 auth_method 0 2024-04-26T00:20:56.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.419+0000 7fded659c640 1 -- 172.21.15.37:0/3912978951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fded8139b80 msgr2=0x7fded81364c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:56.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.419+0000 7fded659c640 1 --2- 172.21.15.37:0/3912978951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fded8139b80 0x7fded81364c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdec80072a0 tx=0x7fdec8007760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:56.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.419+0000 7fdec6ffd640 1 -- 172.21.15.37:0/3912978951 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec800fde0 con 0x7fded8139b80 2024-04-26T00:20:56.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.419+0000 7fdec6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:56.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.419+0000 7fdec6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:56.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.419+0000 7fdec6ffd640 10 monclient: dump: 2024-04-26T00:20:56.408 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:56.408 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:56.408 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.408 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.408 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:56.408 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:56.408 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:56.408 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:56.408 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:56.408 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:56.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.419+0000 7fdec6ffd640 1 -- 172.21.15.37:0/3912978951 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdec800fde0 con 0x7fded8139b80 2024-04-26T00:20:56.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.419+0000 7fdec6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:56.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.419+0000 7fdec6ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:56.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.419+0000 7fdec6ffd640 1 -- 172.21.15.37:0/3912978951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fded8139b80 msgr2=0x7fded81364c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:56.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.419+0000 7fdec6ffd640 1 --2- 172.21.15.37:0/3912978951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fded8139b80 0x7fded81364c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdec80072a0 tx=0x7fdec8007760 comp rx=0 tx=0).stop 2024-04-26T00:20:56.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.419+0000 7fdec6ffd640 1 -- 172.21.15.37:0/3912978951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fded8139b80 msgr2=0x7fded81364c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:20:56.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.419+0000 7fdec6ffd640 1 --2- 172.21.15.37:0/3912978951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fded8139b80 0x7fded81364c0 secure :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdec80072a0 tx=0x7fdec8007760 comp rx=0 tx=0).stop 2024-04-26T00:20:56.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.419+0000 7fded5d9b640 1 --2- 172.21.15.37:0/3912978951 >> v2:172.21.15.45:6812/2379918623 conn(0x7fdeb8035ed0 0x7fdeb8038390 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fded80a3130 tx=0x7fdecc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:56.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.427+0000 7fdedce1a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:20:56.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.427+0000 7fdedce1a640 1 --2- 172.21.15.37:0/3912978951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fded802c000 0x7fded802c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:56.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.427+0000 7fdedce1a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:56.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.427+0000 7fdedce1a640 1 -- 172.21.15.37:0/3912978951 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fded804f510 con 0x7fded802c000 2024-04-26T00:20:56.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.427+0000 7fded659c640 1 --2- 172.21.15.37:0/3912978951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fded802c000 0x7fded802c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:56.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.427+0000 7fded659c640 10 monclient: get_auth_request con 0x7fded802c000 auth_method 0 2024-04-26T00:20:56.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.427+0000 7fded659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:56.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.427+0000 7fded659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:56.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.427+0000 7fded659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:56.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.427+0000 7fded659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:56.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.427+0000 7fded659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:56.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.427+0000 7fded659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:56.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.427+0000 7fded659c640 10 monclient(hunting): handle_auth_done global_id 4460 payload 824 2024-04-26T00:20:56.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.427+0000 7fded659c640 1 -- 172.21.15.37:0/3912978951 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fded813cb60 con 0x7fded802c000 2024-04-26T00:20:56.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.427+0000 7fded659c640 1 -- 172.21.15.37:0/3912978951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fded802c000 msgr2=0x7fded802c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:56.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.427+0000 7fded659c640 1 --2- 172.21.15.37:0/3912978951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fded802c000 0x7fded802c3e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fdec8017000 tx=0x7fdec8004950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:56.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.427+0000 7fdec6ffd640 1 -- 172.21.15.37:0/3912978951 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec801ee40 con 0x7fded802c000 2024-04-26T00:20:56.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.427+0000 7fdec6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:56.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.427+0000 7fdec6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:56.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.427+0000 7fdec6ffd640 10 monclient: dump: 2024-04-26T00:20:56.412 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:56.412 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:56.412 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.412 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.412 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:56.412 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:56.412 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:56.412 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:56.412 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:56.412 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:56.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.431+0000 7fdec6ffd640 1 -- 172.21.15.37:0/3912978951 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4716 (secure 0 0 0) 0x7fdec8006240 con 0x7fded802c000 2024-04-26T00:20:56.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.431+0000 7fdec6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:20:56.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.431+0000 7fdec6ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:56.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.431+0000 7fdec6ffd640 1 -- 172.21.15.37:0/3912978951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fded802c000 msgr2=0x7fded802c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:56.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.431+0000 7fdec6ffd640 1 --2- 172.21.15.37:0/3912978951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fded802c000 0x7fded802c3e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fdec8017000 tx=0x7fdec8004950 comp rx=0 tx=0).stop 2024-04-26T00:20:56.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.431+0000 7fdec6ffd640 1 -- 172.21.15.37:0/3912978951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fded802c000 msgr2=0x7fded802c3e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:20:56.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.431+0000 7fdec6ffd640 1 --2- 172.21.15.37:0/3912978951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fded802c000 0x7fded802c3e0 secure :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fdec8017000 tx=0x7fdec8004950 comp rx=0 tx=0).stop 2024-04-26T00:20:56.413 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:20:56.413 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:20:56.413 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:20:56.413 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:20:56.414 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 12, 2024-04-26T00:20:56.414 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 73535, 2024-04-26T00:20:56.414 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:20:56.414 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:20:56.414 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:20:56.414 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:20:56.414 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:56.414 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 5, 2024-04-26T00:20:56.414 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:20:56.414 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:20:56.414 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:20:56.414 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:20:56.414 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:20:56.414 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:20:56.414 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:20:56.415 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:20:56.415 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:20:56.415 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:20:56.415 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:20:56.415 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:20:56.415 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:20:56.415 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:20:56.415 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:56.415 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:20:56.415 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:20:56.415 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:20:56.415 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:20:56.415 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:20:56.415 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:20:56.415 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:20:56.416 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:20:56.416 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:20:56.416 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:20:56.416 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:20:56.416 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:20:56.416 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:20:56.416 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:56.416 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:56.416 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:20:56.416 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:20:56.416 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:20:56.416 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:20:56.416 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:20:56.416 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:20:56.416 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:20:56.416 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:20:56.417 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:20:56.417 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:20:56.417 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:20:56.417 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:20:56.417 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:20:56.417 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:20:56.417 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:20:56.417 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:20:56.417 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:20:56.417 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:20:56.417 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:20:56.417 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:20:56.417 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:20:56.417 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:20:56.417 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:20:56.418 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:20:56.418 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:20:56.418 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:20:56.418 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:20:56.418 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:20:56.418 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:56.418 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:20:56.418 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:56.418 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:20:56.418 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:56.418 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:20:56.418 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:20:56.419 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:20:56.419 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:20:56.419 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:56.419 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:20:56.419 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:20:56.419 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:56.419 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:56.419 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:56.419 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:20:56.419 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:20:56.419 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:56.419 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:56.419 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:56.419 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:56.419 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:20:56.419 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:20:56.420 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:20:56.420 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:20:56.420 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:20:56.420 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:56.420 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:56.420 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:20:56.420 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:20:56.420 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:20:56.420 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:20:56.420 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:56.420 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:20:56.420 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:20:56.420 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:56.420 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:56.420 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:56.421 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:20:56.421 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:20:56.421 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:56.421 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:56.421 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:56.421 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:56.421 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:20:56.421 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:20:56.421 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:20:56.421 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:20:56.421 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:20:56.421 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:56.421 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:56.421 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:20:56.421 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:20:56.421 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:20:56.422 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:20:56.422 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:56.422 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:20:56.422 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:20:56.422 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:56.422 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:56.422 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:56.422 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:20:56.422 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:20:56.422 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:56.422 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:56.422 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:56.422 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:56.422 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:20:56.422 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:20:56.423 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:20:56.423 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:20:56.423 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:20:56.423 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:56.423 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:56.423 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:56.423 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:20:56.423 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:20:56.423 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:56.423 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:20:56.423 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:20:56.423 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:20:56.423 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:56.423 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:56.423 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:20:56.424 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:56.424 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:20:56.424 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:20:56.424 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:20:56.424 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:56.424 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:56.424 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:20:56.424 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:56.424 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:20:56.424 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:20:56.424 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:20:56.424 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:56.424 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:56.424 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:56.424 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:20:56.424 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:20:56.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.431+0000 7fdedce1a640 1 -- 172.21.15.37:0/3912978951 >> v2:172.21.15.45:6812/2379918623 conn(0x7fdeb8035ed0 msgr2=0x7fdeb8038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:56.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.431+0000 7fdedce1a640 1 --2- 172.21.15.37:0/3912978951 >> v2:172.21.15.45:6812/2379918623 conn(0x7fdeb8035ed0 0x7fdeb8038390 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fded80a3130 tx=0x7fdecc008040 comp rx=0 tx=0).stop 2024-04-26T00:20:56.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.431+0000 7fdedce1a640 10 monclient: shutdown 2024-04-26T00:20:56.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.431+0000 7fdedce1a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:56.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.431+0000 7fdedce1a640 1 -- 172.21.15.37:0/3912978951 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fded813c1a0 msgr2=0x7fded81cfa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:56.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.431+0000 7fdedce1a640 1 --2- 172.21.15.37:0/3912978951 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fded813c1a0 0x7fded81cfa50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fded000ccb0 tx=0x7fded000a660 comp rx=0 tx=0).stop 2024-04-26T00:20:56.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.431+0000 7fded6d9d640 1 -- 172.21.15.37:0/3912978951 reap_dead start 2024-04-26T00:20:56.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.431+0000 7fded6d9d640 1 -- 172.21.15.37:0/3912978951 reap_dead start 2024-04-26T00:20:56.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.431+0000 7fdedce1a640 1 -- 172.21.15.37:0/3912978951 shutdown_connections 2024-04-26T00:20:56.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.431+0000 7fdedce1a640 1 -- 172.21.15.37:0/3912978951 >> 172.21.15.37:0/3912978951 conn(0x7fded80a7160 msgr2=0x7fded80a5960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:56.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.431+0000 7fdedce1a640 1 -- 172.21.15.37:0/3912978951 shutdown_connections 2024-04-26T00:20:56.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.431+0000 7fdedce1a640 1 -- 172.21.15.37:0/3912978951 wait complete. 2024-04-26T00:20:56.427 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:20:56.427 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:20:56.427 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:20:56.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.559+0000 7f4d5cb09640 10 monclient: get_monmap_and_config 2024-04-26T00:20:56.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.559+0000 7f4d5cb09640 10 monclient: build_initial_monmap 2024-04-26T00:20:56.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.559+0000 7f4d5cb09640 10 monclient: monmap: 2024-04-26T00:20:56.538 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:56.538 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:56.538 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:56.562457+0000 2024-04-26T00:20:56.539 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:56.562457+0000 2024-04-26T00:20:56.539 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:56.539 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:56.539 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:56.539 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:56.539 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:56.539 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:56.539 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:56.539 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:56.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.559+0000 7f4d5cb09640 1 Processor -- start 2024-04-26T00:20:56.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.559+0000 7f4d5cb09640 1 -- start start 2024-04-26T00:20:56.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.559+0000 7f4d5cb09640 10 monclient: init 2024-04-26T00:20:56.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.559+0000 7f4d5cb09640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:56.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.559+0000 7f4d5cb09640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-04-26T00:20:56.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.559+0000 7f4d5cb09640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4d580a0300 addr v1:172.21.15.45:6789/0 2024-04-26T00:20:56.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.559+0000 7f4d5cb09640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4d5809fc50 addr v1:172.21.15.37:6789/0 2024-04-26T00:20:56.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.559+0000 7f4d5cb09640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d5813b5e0 0x7f4d5813d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:56.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.559+0000 7f4d5cb09640 10 monclient(hunting): picked mon.noname-b con 0x7f4d5813b5e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:56.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.559+0000 7f4d5cb09640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d5809c190 con 0x7f4d5809fc50 2024-04-26T00:20:56.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.559+0000 7f4d5cb09640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d580a1110 con 0x7f4d580a0300 2024-04-26T00:20:56.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.559+0000 7f4d5cb09640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:56.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.559+0000 7f4d5cb09640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4d580a16d0 con 0x7f4d5813b5e0 2024-04-26T00:20:56.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.559+0000 7f4d5cb09640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:56.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.559+0000 7f4d5cb09640 10 monclient(hunting): authenticate will time out at 1037.867892s 2024-04-26T00:20:56.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.563+0000 7f4d56d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d5813b5e0 0x7f4d5813d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:56.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.563+0000 7f4d56d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d5813b5e0 0x7f4d5813d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:33666/0 (socket says 172.21.15.37:33666) 2024-04-26T00:20:56.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.563+0000 7f4d55d9b640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f4d5809fc50 0x7f4d58139ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:54994/0 (socket says 172.21.15.37:54994) 2024-04-26T00:20:56.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.563+0000 7f4d56d9d640 1 -- 172.21.15.37:0/1013690648 learned_addr learned my addr 172.21.15.37:0/1013690648 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:56.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.563+0000 7f4d56d9d640 10 monclient(hunting): get_auth_request con 0x7f4d5813b5e0 auth_method 0 2024-04-26T00:20:56.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.563+0000 7f4d56d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:56.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.563+0000 7f4d56d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:56.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.563+0000 7f4d56d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:56.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.563+0000 7f4d56d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:56.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.563+0000 7f4d56d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:56.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.563+0000 7f4d56d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:56.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.563+0000 7f4d56d9d640 10 monclient(hunting): handle_auth_done global_id 4463 payload 274 2024-04-26T00:20:56.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.563+0000 7f4d56d9d640 1 -- 172.21.15.37:0/1013690648 >> v1:172.21.15.45:6789/0 conn(0x7f4d580a0300 legacy=0x7f4d580a06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:56.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.563+0000 7f4d56d9d640 1 -- 172.21.15.37:0/1013690648 >> v1:172.21.15.37:6789/0 conn(0x7f4d5809fc50 legacy=0x7f4d58139ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:56.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.563+0000 7f4d56d9d640 10 monclient: _finish_hunting 0 2024-04-26T00:20:56.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.563+0000 7f4d56d9d640 1 monclient: found mon.noname-b 2024-04-26T00:20:56.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.563+0000 7f4d56d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:56.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.563+0000 7f4d56d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:20:56.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.563+0000 7f4d56d9d640 1 -- 172.21.15.37:0/1013690648 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d5809f2a0 con 0x7f4d5813b5e0 2024-04-26T00:20:56.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d56d9d640 1 -- 172.21.15.37:0/1013690648 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d5813b5e0 msgr2=0x7f4d5813d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:56.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d56d9d640 1 --2- 172.21.15.37:0/1013690648 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d5813b5e0 0x7f4d5813d9d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4d50009e80 tx=0x7f4d5001c7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17d4810521737630 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:56.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5559a640 1 -- 172.21.15.37:0/1013690648 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d500a5670 con 0x7f4d5813b5e0 2024-04-26T00:20:56.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:56.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:20:56.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5559a640 10 monclient: dump: 2024-04-26T00:20:56.546 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:56.546 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:56.546 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.546 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.546 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:56.546 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:56.546 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:56.546 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:56.546 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:56.546 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:56.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5559a640 10 monclient: _finish_auth 0 2024-04-26T00:20:56.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5559a640 10 monclient: _check_auth_tickets 2024-04-26T00:20:56.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:56.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5559a640 1 -- 172.21.15.37:0/1013690648 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d500a5810 con 0x7f4d5813b5e0 2024-04-26T00:20:56.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5cb09640 5 monclient: authenticate success, global_id 4463 2024-04-26T00:20:56.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5cb09640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:20:56.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5559a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:56.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5cb09640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:56.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5cb09640 4 set_mon_vals no callback set 2024-04-26T00:20:56.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5559a640 1 -- 172.21.15.37:0/1013690648 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d500a5be0 con 0x7f4d5813b5e0 2024-04-26T00:20:56.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:56.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:56.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5559a640 10 monclient: dump: 2024-04-26T00:20:56.547 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:56.547 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:56.547 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.547 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.547 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:56.548 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:56.548 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:56.548 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:56.548 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:56.548 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:56.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5cb09640 10 monclient: shutdown 2024-04-26T00:20:56.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5cb09640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:56.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5cb09640 1 -- 172.21.15.37:0/1013690648 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d5813b5e0 msgr2=0x7f4d5813d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:56.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5cb09640 1 --2- 172.21.15.37:0/1013690648 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d5813b5e0 0x7f4d5813d9d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4d50009e80 tx=0x7f4d5001c7d0 comp rx=0 tx=0).stop 2024-04-26T00:20:56.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5cb09640 1 -- 172.21.15.37:0/1013690648 shutdown_connections 2024-04-26T00:20:56.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5cb09640 1 --2- 172.21.15.37:0/1013690648 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d5813b5e0 0x7f4d5813d9d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:56.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5cb09640 1 -- 172.21.15.37:0/1013690648 >> 172.21.15.37:0/1013690648 conn(0x7f4d580a1d50 msgr2=0x7f4d580a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:56.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5cb09640 1 -- 172.21.15.37:0/1013690648 shutdown_connections 2024-04-26T00:20:56.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5cb09640 1 -- 172.21.15.37:0/1013690648 wait complete. 2024-04-26T00:20:56.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5cb09640 10 monclient: build_initial_monmap 2024-04-26T00:20:56.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5cb09640 10 monclient: monmap: 2024-04-26T00:20:56.549 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:56.549 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:56.549 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:56.549 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:56.549 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:56.549 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:56.549 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:56.549 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:56.549 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:56.549 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:56.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5cb09640 1 Processor -- start 2024-04-26T00:20:56.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5cb09640 1 -- start start 2024-04-26T00:20:56.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5cb09640 10 monclient: init 2024-04-26T00:20:56.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5cb09640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:56.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5cb09640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:20:56.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5cb09640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4d5809fc50 0x7f4d5811b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:56.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5cb09640 10 monclient(hunting): picked mon.noname-a con 0x7f4d5809fc50 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:56.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5cb09640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d580a0300 0x7f4d58139a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:56.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5cb09640 10 monclient(hunting): picked mon.noname-b con 0x7f4d580a0300 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:56.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5cb09640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d5813b5e0 0x7f4d5811c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:56.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5cb09640 10 monclient(hunting): picked mon.noname-c con 0x7f4d5813b5e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:56.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5cb09640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:56.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5cb09640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f4d58140b00 con 0x7f4d5809fc50 2024-04-26T00:20:56.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5cb09640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:56.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5cb09640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4d58140980 con 0x7f4d580a0300 2024-04-26T00:20:56.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5cb09640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:56.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5cb09640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4d58140c80 con 0x7f4d5813b5e0 2024-04-26T00:20:56.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5cb09640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:56.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5cb09640 10 monclient(hunting): authenticate will time out at 1037.875152s 2024-04-26T00:20:56.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5659c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4d5809fc50 0x7f4d5811b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:56.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5659c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4d5809fc50 0x7f4d5811b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:37634/0 (socket says 172.21.15.37:37634) 2024-04-26T00:20:56.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5659c640 1 -- 172.21.15.37:0/128800030 learned_addr learned my addr 172.21.15.37:0/128800030 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:56.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5659c640 10 monclient(hunting): get_auth_request con 0x7f4d5809fc50 auth_method 0 2024-04-26T00:20:56.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:56.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:56.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:56.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d56d9d640 1 --2- 172.21.15.37:0/128800030 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d5813b5e0 0x7f4d5811c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:56.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d56d9d640 10 monclient(hunting): get_auth_request con 0x7f4d5813b5e0 auth_method 0 2024-04-26T00:20:56.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d56d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:56.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d56d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:56.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d56d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:56.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:56.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:56.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:56.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d56d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:56.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d56d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:56.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d56d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:56.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d56d9d640 10 monclient(hunting): handle_auth_done global_id 4466 payload 824 2024-04-26T00:20:56.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d55d9b640 1 --2- 172.21.15.37:0/128800030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d580a0300 0x7f4d58139a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:56.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d56d9d640 1 -- 172.21.15.37:0/128800030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d580a0300 msgr2=0x7f4d58139a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:56.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d56d9d640 1 --2- 172.21.15.37:0/128800030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d580a0300 0x7f4d58139a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:56.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d56d9d640 1 -- 172.21.15.37:0/128800030 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4d5809fc50 msgr2=0x7f4d5811b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:56.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d56d9d640 1 --2- 172.21.15.37:0/128800030 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4d5809fc50 0x7f4d5811b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:56.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d56d9d640 10 monclient: _finish_hunting 0 2024-04-26T00:20:56.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d56d9d640 1 monclient: found mon.noname-c 2024-04-26T00:20:56.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d56d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:56.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d56d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:20:56.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d56d9d640 1 -- 172.21.15.37:0/128800030 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d5809f550 con 0x7f4d5813b5e0 2024-04-26T00:20:56.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.567+0000 7f4d5659c640 1 --2- 172.21.15.37:0/128800030 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4d5809fc50 0x7f4d5811b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:56.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.571+0000 7f4d56d9d640 1 -- 172.21.15.37:0/128800030 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d5813b5e0 msgr2=0x7f4d5811c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:56.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.571+0000 7f4d56d9d640 1 --2- 172.21.15.37:0/128800030 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d5813b5e0 0x7f4d5811c310 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4d500ab000 tx=0x7f4d50019570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:56.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.571+0000 7f4d4effd640 1 -- 172.21.15.37:0/128800030 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d500bc070 con 0x7f4d5813b5e0 2024-04-26T00:20:56.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.571+0000 7f4d4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:56.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.571+0000 7f4d4effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:20:56.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.571+0000 7f4d4effd640 10 monclient: dump: 2024-04-26T00:20:56.557 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:56.557 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:56.557 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.557 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.557 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:56.558 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:56.558 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:56.558 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:56.558 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:56.558 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:56.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.571+0000 7f4d4effd640 10 monclient: _finish_auth 0 2024-04-26T00:20:56.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.571+0000 7f4d4effd640 10 monclient: _check_auth_tickets 2024-04-26T00:20:56.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.571+0000 7f4d4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:56.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.571+0000 7f4d4effd640 1 -- 172.21.15.37:0/128800030 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d500b8870 con 0x7f4d5813b5e0 2024-04-26T00:20:56.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.571+0000 7f4d5cb09640 5 monclient: authenticate success, global_id 4466 2024-04-26T00:20:56.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.571+0000 7f4d5cb09640 10 monclient: _renew_subs 2024-04-26T00:20:56.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.571+0000 7f4d5cb09640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:56.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.571+0000 7f4d5cb09640 1 -- 172.21.15.37:0/128800030 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d5811d8d0 con 0x7f4d5813b5e0 2024-04-26T00:20:56.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.571+0000 7f4d4effd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:56.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.571+0000 7f4d4effd640 1 -- 172.21.15.37:0/128800030 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d500a5670 con 0x7f4d5813b5e0 2024-04-26T00:20:56.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.571+0000 7f4d4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:56.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.571+0000 7f4d4effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:56.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.571+0000 7f4d4effd640 10 monclient: dump: 2024-04-26T00:20:56.559 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:56.559 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:56.559 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.559 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.559 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:56.559 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:56.559 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:56.559 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:56.559 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:56.559 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:56.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.571+0000 7f4d54d99640 4 set_mon_vals no callback set 2024-04-26T00:20:56.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.571+0000 7f4d5cb09640 10 monclient: _renew_subs 2024-04-26T00:20:56.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.571+0000 7f4d5cb09640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:56.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.571+0000 7f4d5cb09640 1 -- 172.21.15.37:0/128800030 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d5813a710 con 0x7f4d5813b5e0 2024-04-26T00:20:56.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.571+0000 7f4d5cb09640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:56.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.571+0000 7f4d5cb09640 1 --2- 172.21.15.37:0/128800030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d581ccf60 0x7f4d581cd340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:56.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.571+0000 7f4d5cb09640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:56.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.571+0000 7f4d5cb09640 1 -- 172.21.15.37:0/128800030 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4d5804f510 con 0x7f4d581ccf60 2024-04-26T00:20:56.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.571+0000 7f4d4effd640 1 -- 172.21.15.37:0/128800030 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f4d500b8ce0 con 0x7f4d5813b5e0 2024-04-26T00:20:56.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.571+0000 7f4d4effd640 1 --2- 172.21.15.37:0/128800030 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4d38035eb0 0x7f4d38038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:56.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.571+0000 7f4d4effd640 1 -- 172.21.15.37:0/128800030 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(47..47 src has 1..47) ==== 3915+0+0 (secure 0 0 0) 0x7f4d500e32d0 con 0x7f4d5813b5e0 2024-04-26T00:20:56.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.571+0000 7f4d5659c640 1 --2- 172.21.15.37:0/128800030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d581ccf60 0x7f4d581cd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:56.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.571+0000 7f4d5659c640 10 monclient: get_auth_request con 0x7f4d581ccf60 auth_method 0 2024-04-26T00:20:56.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.571+0000 7f4d5659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:56.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.571+0000 7f4d5659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:56.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.571+0000 7f4d5659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:56.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.571+0000 7f4d55d9b640 1 --2- 172.21.15.37:0/128800030 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4d38035eb0 0x7f4d38038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:56.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.571+0000 7f4d5659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:56.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.571+0000 7f4d5659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:56.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.571+0000 7f4d5659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:56.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.571+0000 7f4d55d9b640 10 monclient: get_auth_request con 0x7f4d38035eb0 auth_method 0 2024-04-26T00:20:56.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.575+0000 7f4d5659c640 10 monclient(hunting): handle_auth_done global_id 14131 payload 824 2024-04-26T00:20:56.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.575+0000 7f4d5659c640 1 -- 172.21.15.37:0/128800030 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4d5809eb60 con 0x7f4d581ccf60 2024-04-26T00:20:56.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.575+0000 7f4d55d9b640 1 --2- 172.21.15.37:0/128800030 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4d38035eb0 0x7f4d38038370 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4d58135990 tx=0x7f4d48009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:56.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.575+0000 7f4d5659c640 1 -- 172.21.15.37:0/128800030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d581ccf60 msgr2=0x7f4d581cd340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:56.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.575+0000 7f4d5659c640 1 --2- 172.21.15.37:0/128800030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d581ccf60 0x7f4d581cd340 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4d4000fb10 tx=0x7f4d400045a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:56.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.575+0000 7f4d4effd640 1 -- 172.21.15.37:0/128800030 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d40018030 con 0x7f4d581ccf60 2024-04-26T00:20:56.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.575+0000 7f4d4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:56.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.575+0000 7f4d4effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:20:56.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.575+0000 7f4d4effd640 10 monclient: dump: 2024-04-26T00:20:56.567 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:56.567 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:56.567 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.567 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.567 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:56.567 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:56.568 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:56.568 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:56.568 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:56.568 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:56.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.575+0000 7f4d4effd640 1 -- 172.21.15.37:0/128800030 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4d4000de50 con 0x7f4d581ccf60 2024-04-26T00:20:56.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.575+0000 7f4d4effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:56.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.575+0000 7f4d4effd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:56.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.575+0000 7f4d4effd640 1 -- 172.21.15.37:0/128800030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d581ccf60 msgr2=0x7f4d581cd340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:56.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.575+0000 7f4d4effd640 1 --2- 172.21.15.37:0/128800030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d581ccf60 0x7f4d581cd340 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4d4000fb10 tx=0x7f4d400045a0 comp rx=0 tx=0).stop 2024-04-26T00:20:56.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.575+0000 7f4d4effd640 1 -- 172.21.15.37:0/128800030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d581ccf60 msgr2=0x7f4d581cd340 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:20:56.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.575+0000 7f4d4effd640 1 --2- 172.21.15.37:0/128800030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d581ccf60 0x7f4d581cd340 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4d4000fb10 tx=0x7f4d400045a0 comp rx=0 tx=0).stop 2024-04-26T00:20:56.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.583+0000 7f4d5cb09640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:20:56.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.583+0000 7f4d5cb09640 1 --2- 172.21.15.37:0/128800030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d581cd880 0x7f4d58135c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:56.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.583+0000 7f4d5cb09640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:56.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.583+0000 7f4d5cb09640 1 -- 172.21.15.37:0/128800030 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4d5806ba50 con 0x7f4d581cd880 2024-04-26T00:20:56.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.583+0000 7f4d5659c640 1 --2- 172.21.15.37:0/128800030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d581cd880 0x7f4d58135c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:56.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.583+0000 7f4d5659c640 10 monclient: get_auth_request con 0x7f4d581cd880 auth_method 0 2024-04-26T00:20:56.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.583+0000 7f4d5659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:56.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.583+0000 7f4d5659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:56.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.583+0000 7f4d5659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:56.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.583+0000 7f4d5659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:56.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.583+0000 7f4d5659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:56.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.583+0000 7f4d5659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:56.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.583+0000 7f4d5659c640 10 monclient(hunting): handle_auth_done global_id 14134 payload 824 2024-04-26T00:20:56.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.583+0000 7f4d5659c640 1 -- 172.21.15.37:0/128800030 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4d581352f0 con 0x7f4d581cd880 2024-04-26T00:20:56.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.587+0000 7f4d5659c640 1 -- 172.21.15.37:0/128800030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d581cd880 msgr2=0x7f4d58135c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:56.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.587+0000 7f4d5659c640 1 --2- 172.21.15.37:0/128800030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d581cd880 0x7f4d58135c70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4d4000fb10 tx=0x7f4d40007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:56.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.587+0000 7f4d4effd640 1 -- 172.21.15.37:0/128800030 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d40004b60 con 0x7f4d581cd880 2024-04-26T00:20:56.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.587+0000 7f4d4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:56.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.587+0000 7f4d4effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:20:56.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.587+0000 7f4d4effd640 10 monclient: dump: 2024-04-26T00:20:56.570 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:56.570 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:56.570 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.570 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.570 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:56.570 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:56.570 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:56.570 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:56.570 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:56.570 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:56.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.587+0000 7f4d4effd640 1 -- 172.21.15.37:0/128800030 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7f4d40004b60 con 0x7f4d581cd880 2024-04-26T00:20:56.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.587+0000 7f4d4effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:20:56.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.587+0000 7f4d4effd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:56.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.587+0000 7f4d4effd640 1 -- 172.21.15.37:0/128800030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d581cd880 msgr2=0x7f4d58135c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:56.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.587+0000 7f4d4effd640 1 --2- 172.21.15.37:0/128800030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d581cd880 0x7f4d58135c70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4d4000fb10 tx=0x7f4d40007f90 comp rx=0 tx=0).stop 2024-04-26T00:20:56.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.587+0000 7f4d4effd640 1 -- 172.21.15.37:0/128800030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d581cd880 msgr2=0x7f4d58135c70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:20:56.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.587+0000 7f4d4effd640 1 --2- 172.21.15.37:0/128800030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d581cd880 0x7f4d58135c70 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4d4000fb10 tx=0x7f4d40007f90 comp rx=0 tx=0).stop 2024-04-26T00:20:56.571 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:20:56.571 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:20:56.571 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:20:56.572 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:20:56.572 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 12, 2024-04-26T00:20:56.572 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 10103, 2024-04-26T00:20:56.572 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:20:56.572 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:20:56.572 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:20:56.572 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:20:56.572 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:56.572 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 6, 2024-04-26T00:20:56.572 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:20:56.572 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:20:56.572 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:20:56.572 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:20:56.573 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:20:56.573 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:20:56.573 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:20:56.573 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:20:56.573 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:20:56.573 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:20:56.573 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:20:56.573 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:20:56.573 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:20:56.573 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:20:56.573 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:56.573 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:20:56.573 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:20:56.573 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:20:56.574 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:20:56.574 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:20:56.574 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:20:56.574 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:20:56.574 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:20:56.574 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:20:56.574 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:20:56.574 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:20:56.574 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:20:56.574 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:20:56.574 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:56.574 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:56.574 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:20:56.574 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:20:56.575 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:20:56.575 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:20:56.575 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:20:56.575 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:20:56.575 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:20:56.575 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:20:56.575 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:20:56.575 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:20:56.575 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:20:56.575 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:20:56.575 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:20:56.575 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:20:56.575 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:20:56.575 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:20:56.576 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:20:56.576 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:20:56.576 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:20:56.576 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:20:56.576 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:20:56.576 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:20:56.576 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:20:56.576 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:20:56.576 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:20:56.576 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:20:56.576 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:20:56.576 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:20:56.576 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:56.576 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:20:56.576 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:56.577 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:20:56.577 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:56.577 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:20:56.577 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:20:56.577 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:20:56.577 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:20:56.577 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:56.577 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:20:56.577 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:20:56.577 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:56.577 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:56.577 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:56.577 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:20:56.578 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:20:56.578 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:56.578 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:56.578 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:56.578 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:56.578 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:20:56.578 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:20:56.578 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:20:56.578 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:20:56.578 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:20:56.578 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:56.578 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:56.578 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:20:56.578 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:20:56.579 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:20:56.579 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:20:56.579 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:56.579 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:20:56.579 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:20:56.579 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:56.579 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:56.579 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:56.579 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:20:56.579 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:20:56.579 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:56.579 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:56.579 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:56.579 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:56.579 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:20:56.580 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:20:56.580 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:20:56.580 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:20:56.580 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:20:56.580 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:56.580 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:56.580 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:20:56.580 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:20:56.580 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:20:56.580 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:20:56.580 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:56.580 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:20:56.580 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:20:56.580 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:56.581 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:56.581 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:56.581 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:20:56.581 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:20:56.581 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:20:56.581 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:56.581 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:56.581 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:56.581 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:20:56.581 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:20:56.581 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:20:56.581 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:20:56.581 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:20:56.581 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:56.582 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:56.582 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:56.582 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:20:56.582 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:20:56.582 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:56.582 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:20:56.582 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:20:56.582 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:20:56.582 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:56.582 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:20:56.582 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:20:56.582 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:20:56.582 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:20:56.582 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:20:56.583 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:20:56.583 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:20:56.583 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:20:56.583 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:20:56.583 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:20:56.583 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:20:56.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.587+0000 7f4d5cb09640 1 -- 172.21.15.37:0/128800030 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4d38035eb0 msgr2=0x7f4d38038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:56.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.587+0000 7f4d5cb09640 1 --2- 172.21.15.37:0/128800030 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4d38035eb0 0x7f4d38038370 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4d58135990 tx=0x7f4d48009040 comp rx=0 tx=0).stop 2024-04-26T00:20:56.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.587+0000 7f4d5cb09640 10 monclient: shutdown 2024-04-26T00:20:56.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.587+0000 7f4d5cb09640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:56.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.587+0000 7f4d5cb09640 1 -- 172.21.15.37:0/128800030 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d5813b5e0 msgr2=0x7f4d5811c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:56.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.587+0000 7f4d5cb09640 1 --2- 172.21.15.37:0/128800030 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d5813b5e0 0x7f4d5811c310 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4d500ab000 tx=0x7f4d50019570 comp rx=0 tx=0).stop 2024-04-26T00:20:56.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.587+0000 7f4d56d9d640 1 -- 172.21.15.37:0/128800030 reap_dead start 2024-04-26T00:20:56.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.587+0000 7f4d5cb09640 1 -- 172.21.15.37:0/128800030 shutdown_connections 2024-04-26T00:20:56.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.587+0000 7f4d5cb09640 1 --2- 172.21.15.37:0/128800030 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d5813b5e0 0x7f4d5811c310 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:56.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.587+0000 7f4d5cb09640 1 -- 172.21.15.37:0/128800030 >> 172.21.15.37:0/128800030 conn(0x7f4d580a1d50 msgr2=0x7f4d580a3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:56.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.587+0000 7f4d5cb09640 1 -- 172.21.15.37:0/128800030 shutdown_connections 2024-04-26T00:20:56.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.587+0000 7f4d5cb09640 1 -- 172.21.15.37:0/128800030 wait complete. 2024-04-26T00:20:56.585 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:20:56.586 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-04-26T00:20:56.586 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-26T00:20:56.586 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-04-26T00:20:56.586 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-04-26T00:20:56.586 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-26T00:20:56.586 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T00:20:56.603 INFO:tasks.ceph.mon.b:Stopped 2024-04-26T00:20:56.603 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-26T00:20:56.603 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:20:56.704 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-04-26T00:20:56.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676c26f640 10 monclient: get_monmap_and_config 2024-04-26T00:20:56.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676c26f640 10 monclient: build_initial_monmap 2024-04-26T00:20:56.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676c26f640 10 monclient: monmap: 2024-04-26T00:20:56.713 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:56.713 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:56.713 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:20:56.736963+0000 2024-04-26T00:20:56.713 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:20:56.736963+0000 2024-04-26T00:20:56.713 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:56.713 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:56.713 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:20:56.713 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:20:56.713 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:20:56.713 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:20:56.713 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:20:56.713 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:56.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676c26f640 1 Processor -- start 2024-04-26T00:20:56.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676c26f640 1 -- start start 2024-04-26T00:20:56.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676c26f640 10 monclient: init 2024-04-26T00:20:56.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676c26f640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:56.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676c26f640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-04-26T00:20:56.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676c26f640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f67640a0260 0x7f67640a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:56.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676c26f640 10 monclient(hunting): picked mon.noname-c con 0x7f67640a0260 addr v2:172.21.15.45:3300/0 2024-04-26T00:20:56.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676c26f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6764134b30 0x7f6764134f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:56.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676c26f640 10 monclient(hunting): picked mon.noname-b con 0x7f6764134b30 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:56.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676c26f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6764135490 addr v1:172.21.15.37:6789/0 2024-04-26T00:20:56.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676c26f640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67640a1130 con 0x7f6764135490 2024-04-26T00:20:56.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676c26f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:56.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676c26f640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f676409ec10 con 0x7f67640a0260 2024-04-26T00:20:56.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676c26f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:56.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676c26f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f676409f0f0 con 0x7f6764134b30 2024-04-26T00:20:56.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676c26f640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:56.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676c26f640 10 monclient(hunting): authenticate will time out at 1038.042486s 2024-04-26T00:20:56.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676980a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6764134b30 0x7f6764134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:56.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676980a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6764134b30 0x7f6764134f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:33684/0 (socket says 172.21.15.37:33684) 2024-04-26T00:20:56.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676980a640 1 -- 172.21.15.37:0/3781363143 learned_addr learned my addr 172.21.15.37:0/3781363143 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:56.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676980a640 10 monclient(hunting): get_auth_request con 0x7f6764134b30 auth_method 0 2024-04-26T00:20:56.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676980a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:56.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676980a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:56.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676980a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:56.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676a00b640 1 -- 172.21.15.37:0/3781363143 >> v2:172.21.15.45:3300/0 conn(0x7f67640a0260 msgr2=0x7f67640a0640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:20:56.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676980a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:56.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676980a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:56.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676980a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:56.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676a00b640 1 --2- 172.21.15.37:0/3781363143 >> v2:172.21.15.45:3300/0 conn(0x7f67640a0260 0x7f67640a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:20:56.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676980a640 10 monclient(hunting): handle_auth_done global_id 4469 payload 274 2024-04-26T00:20:56.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676980a640 1 -- 172.21.15.37:0/3781363143 >> v2:172.21.15.45:3300/0 conn(0x7f67640a0260 msgr2=0x7f67640a0640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:20:56.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676980a640 1 --2- 172.21.15.37:0/3781363143 >> v2:172.21.15.45:3300/0 conn(0x7f67640a0260 0x7f67640a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:56.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676980a640 1 -- 172.21.15.37:0/3781363143 >> v1:172.21.15.37:6789/0 conn(0x7f6764135490 legacy=0x7f676413be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:20:56.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676980a640 10 monclient: _finish_hunting 0 2024-04-26T00:20:56.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676980a640 1 monclient: found mon.noname-b 2024-04-26T00:20:56.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676980a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:56.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676980a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:20:56.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.735+0000 7f676980a640 1 -- 172.21.15.37:0/3781363143 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f676406cff0 con 0x7f6764134b30 2024-04-26T00:20:56.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.739+0000 7f676980a640 1 -- 172.21.15.37:0/3781363143 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6764134b30 msgr2=0x7f6764134f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:56.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.739+0000 7f676980a640 1 --2- 172.21.15.37:0/3781363143 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6764134b30 0x7f6764134f50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6754009870 tx=0x7f675401c220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbb0d7ab3f7eee62 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:56.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.739+0000 7f6769009640 1 -- 172.21.15.37:0/3781363143 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67540b0070 con 0x7f6764134b30 2024-04-26T00:20:56.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.739+0000 7f6769009640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:56.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.739+0000 7f6769009640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:20:56.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.739+0000 7f6769009640 10 monclient: dump: 2024-04-26T00:20:56.720 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:56.720 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:56.720 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.721 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.721 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:56.721 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:56.721 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:56.721 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:56.721 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:56.721 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:56.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.739+0000 7f6769009640 10 monclient: _finish_auth 0 2024-04-26T00:20:56.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.739+0000 7f6769009640 10 monclient: _check_auth_tickets 2024-04-26T00:20:56.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.739+0000 7f6769009640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:56.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.739+0000 7f6769009640 1 -- 172.21.15.37:0/3781363143 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67540ab600 con 0x7f6764134b30 2024-04-26T00:20:56.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.739+0000 7f6769009640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:56.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.739+0000 7f6769009640 1 -- 172.21.15.37:0/3781363143 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67540ab9d0 con 0x7f6764134b30 2024-04-26T00:20:56.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.739+0000 7f6769009640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:56.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.739+0000 7f6769009640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:56.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.739+0000 7f6769009640 10 monclient: dump: 2024-04-26T00:20:56.722 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:56.722 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:56.722 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.722 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.722 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:56.722 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:56.722 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:56.722 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:56.722 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:56.723 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:56.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.739+0000 7f676c26f640 5 monclient: authenticate success, global_id 4469 2024-04-26T00:20:56.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.739+0000 7f676c26f640 10 monclient: get_monmap_and_config success 2024-04-26T00:20:56.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.739+0000 7f676c26f640 4 set_mon_vals no callback set 2024-04-26T00:20:56.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676c26f640 10 monclient: shutdown 2024-04-26T00:20:56.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676c26f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:56.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676c26f640 1 -- 172.21.15.37:0/3781363143 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6764134b30 msgr2=0x7f6764134f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:56.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676c26f640 1 --2- 172.21.15.37:0/3781363143 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6764134b30 0x7f6764134f50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6754009870 tx=0x7f675401c220 comp rx=0 tx=0).stop 2024-04-26T00:20:56.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676c26f640 1 -- 172.21.15.37:0/3781363143 shutdown_connections 2024-04-26T00:20:56.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676c26f640 1 --2- 172.21.15.37:0/3781363143 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6764134b30 0x7f6764134f50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:56.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676c26f640 1 --2- 172.21.15.37:0/3781363143 >> v2:172.21.15.45:3300/0 conn(0x7f67640a0260 0x7f67640a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:56.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676c26f640 1 -- 172.21.15.37:0/3781363143 >> 172.21.15.37:0/3781363143 conn(0x7f67640a1ad0 msgr2=0x7f67640a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:56.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676c26f640 1 -- 172.21.15.37:0/3781363143 shutdown_connections 2024-04-26T00:20:56.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676c26f640 1 -- 172.21.15.37:0/3781363143 wait complete. 2024-04-26T00:20:56.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676c26f640 10 monclient: build_initial_monmap 2024-04-26T00:20:56.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676c26f640 10 monclient: monmap: 2024-04-26T00:20:56.724 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:20:56.724 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:20:56.724 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:20:56.724 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:20:56.724 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:20:56.724 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:20:56.724 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:20:56.724 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:20:56.724 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:20:56.724 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:56.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676c26f640 1 Processor -- start 2024-04-26T00:20:56.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676c26f640 1 -- start start 2024-04-26T00:20:56.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676c26f640 10 monclient: init 2024-04-26T00:20:56.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676c26f640 10 monclient: _reopen_session rank -1 2024-04-26T00:20:56.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676c26f640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:20:56.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676c26f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67640a0260 0x7f6764070fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:56.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676c26f640 10 monclient(hunting): picked mon.noname-b con 0x7f67640a0260 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:20:56.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676c26f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6764134b30 0x7f67641319e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:56.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676c26f640 10 monclient(hunting): picked mon.noname-a con 0x7f6764134b30 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:20:56.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676c26f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6764135490 0x7f6764131f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:56.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676c26f640 10 monclient(hunting): picked mon.noname-c con 0x7f6764135490 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:20:56.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676c26f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:56.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676c26f640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f676409f430 con 0x7f6764134b30 2024-04-26T00:20:56.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676c26f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:56.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676c26f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f676409f2b0 con 0x7f67640a0260 2024-04-26T00:20:56.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676c26f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:20:56.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676c26f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f676409fe90 con 0x7f6764135490 2024-04-26T00:20:56.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676c26f640 10 monclient(hunting): _renew_subs 2024-04-26T00:20:56.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676c26f640 10 monclient(hunting): authenticate will time out at 1038.049759s 2024-04-26T00:20:56.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676a80c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6764135490 0x7f6764131f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:56.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676980a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6764134b30 0x7f67641319e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:56.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676a00b640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67640a0260 msgr2=0x7f6764070fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:20:56.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676a80c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6764135490 0x7f6764131f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:33686/0 (socket says 172.21.15.37:33686) 2024-04-26T00:20:56.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676a80c640 1 -- 172.21.15.37:0/3107162483 learned_addr learned my addr 172.21.15.37:0/3107162483 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:20:56.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676980a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6764134b30 0x7f67641319e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:37644/0 (socket says 172.21.15.37:37644) 2024-04-26T00:20:56.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676a80c640 10 monclient(hunting): get_auth_request con 0x7f6764135490 auth_method 0 2024-04-26T00:20:56.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676a80c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:56.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676a00b640 1 --2- 172.21.15.37:0/3107162483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67640a0260 0x7f6764070fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:20:56.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676a80c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:56.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676a80c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:56.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676980a640 10 monclient(hunting): get_auth_request con 0x7f6764134b30 auth_method 0 2024-04-26T00:20:56.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676980a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:20:56.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676980a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:20:56.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676980a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:20:56.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676a80c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:56.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676a80c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:56.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676a80c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:56.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676980a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:20:56.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676980a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:20:56.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676980a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:20:56.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676a80c640 10 monclient(hunting): handle_auth_done global_id 4472 payload 824 2024-04-26T00:20:56.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676a80c640 1 -- 172.21.15.37:0/3107162483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67640a0260 msgr2=0x7f6764070fc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:20:56.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676a80c640 1 --2- 172.21.15.37:0/3107162483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67640a0260 0x7f6764070fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:56.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676a80c640 1 -- 172.21.15.37:0/3107162483 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6764134b30 msgr2=0x7f67641319e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:56.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676a80c640 1 --2- 172.21.15.37:0/3107162483 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6764134b30 0x7f67641319e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:56.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676a80c640 10 monclient: _finish_hunting 0 2024-04-26T00:20:56.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676a80c640 1 monclient: found mon.noname-c 2024-04-26T00:20:56.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676a80c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:20:56.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676a80c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:20:56.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676a80c640 1 -- 172.21.15.37:0/3107162483 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f676411e390 con 0x7f6764135490 2024-04-26T00:20:56.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.743+0000 7f676980a640 1 --2- 172.21.15.37:0/3107162483 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6764134b30 0x7f67641319e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:20:56.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.747+0000 7f676a80c640 1 -- 172.21.15.37:0/3107162483 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6764135490 msgr2=0x7f6764131f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:20:56.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.747+0000 7f676a80c640 1 --2- 172.21.15.37:0/3107162483 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6764135490 0x7f6764131f20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f676000c3c0 tx=0x7f676000c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:56.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.747+0000 7f675affd640 1 -- 172.21.15.37:0/3107162483 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67600119a0 con 0x7f6764135490 2024-04-26T00:20:56.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.747+0000 7f675affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:56.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.747+0000 7f675affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:20:56.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.747+0000 7f675affd640 10 monclient: dump: 2024-04-26T00:20:56.733 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:56.733 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:56.733 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.733 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.733 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:56.733 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:56.733 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:56.733 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:56.733 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:56.733 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:56.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.747+0000 7f675affd640 10 monclient: _finish_auth 0 2024-04-26T00:20:56.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.747+0000 7f675affd640 10 monclient: _check_auth_tickets 2024-04-26T00:20:56.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.747+0000 7f675affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:20:56.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.747+0000 7f675affd640 1 -- 172.21.15.37:0/3107162483 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6760017070 con 0x7f6764135490 2024-04-26T00:20:56.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.747+0000 7f676c26f640 5 monclient: authenticate success, global_id 4472 2024-04-26T00:20:56.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.747+0000 7f676c26f640 10 monclient: _renew_subs 2024-04-26T00:20:56.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.747+0000 7f676c26f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:56.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.747+0000 7f676c26f640 1 -- 172.21.15.37:0/3107162483 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f676411bcb0 con 0x7f6764135490 2024-04-26T00:20:56.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.747+0000 7f675affd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:20:56.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.747+0000 7f675affd640 1 -- 172.21.15.37:0/3107162483 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6760011d80 con 0x7f6764135490 2024-04-26T00:20:56.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.747+0000 7f675affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:20:56.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.747+0000 7f675affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:20:56.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.747+0000 7f675affd640 10 monclient: dump: 2024-04-26T00:20:56.734 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:20:56.734 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:20:56.734 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.735 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:20:56.735 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:20:56.735 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:20:56.735 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:20:56.735 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:20:56.735 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:20:56.735 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:20:56.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.747+0000 7f6768808640 4 set_mon_vals no callback set 2024-04-26T00:20:56.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.747+0000 7f676c26f640 10 monclient: _renew_subs 2024-04-26T00:20:56.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.747+0000 7f676c26f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:56.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.747+0000 7f676c26f640 1 -- 172.21.15.37:0/3107162483 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f676411daf0 con 0x7f6764135490 2024-04-26T00:20:56.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.747+0000 7f675affd640 1 -- 172.21.15.37:0/3107162483 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f676001c460 con 0x7f6764135490 2024-04-26T00:20:56.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.747+0000 7f676c26f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:56.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.747+0000 7f676c26f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:56.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.747+0000 7f676c26f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:56.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.747+0000 7f676c26f640 1 -- 172.21.15.37:0/3107162483 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6764066f40 con 0x7f6764135490 2024-04-26T00:20:56.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.747+0000 7f675affd640 1 --2- 172.21.15.37:0/3107162483 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6748035e80 0x7f6748038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:20:56.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.747+0000 7f675affd640 1 -- 172.21.15.37:0/3107162483 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(47..47 src has 1..47) ==== 3915+0+0 (secure 0 0 0) 0x7f6760048740 con 0x7f6764135490 2024-04-26T00:20:56.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.747+0000 7f676a00b640 1 --2- 172.21.15.37:0/3107162483 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6748035e80 0x7f6748038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:20:56.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.747+0000 7f676a00b640 10 monclient: get_auth_request con 0x7f6748035e80 auth_method 0 2024-04-26T00:20:56.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.751+0000 7f676a00b640 1 --2- 172.21.15.37:0/3107162483 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6748035e80 0x7f6748038340 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f675c003cc0 tx=0x7f675c009270 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:20:56.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.751+0000 7f675affd640 1 -- 172.21.15.37:0/3107162483 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f67600474a0 con 0x7f6764135490 2024-04-26T00:20:56.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.751+0000 7f675affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:20:56.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.751+0000 7f675affd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:20:56.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.971+0000 7f676c26f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:20:56.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.971+0000 7f676c26f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:20:56.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.971+0000 7f676c26f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:20:56.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.971+0000 7f676c26f640 1 -- 172.21.15.37:0/3107162483 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f676411cdb0 con 0x7f6764135490 2024-04-26T00:20:56.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.971+0000 7f675affd640 1 -- 172.21.15.37:0/3107162483 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f676001a760 con 0x7f6764135490 2024-04-26T00:20:56.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.971+0000 7f675affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:20:56.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.971+0000 7f675affd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:20:56.949 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:20:56.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.971+0000 7f676c26f640 1 -- 172.21.15.37:0/3107162483 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6748035e80 msgr2=0x7f6748038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:56.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.971+0000 7f676c26f640 1 --2- 172.21.15.37:0/3107162483 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6748035e80 0x7f6748038340 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f675c003cc0 tx=0x7f675c009270 comp rx=0 tx=0).stop 2024-04-26T00:20:56.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.971+0000 7f676c26f640 10 monclient: shutdown 2024-04-26T00:20:56.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.971+0000 7f676c26f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:20:56.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.971+0000 7f676c26f640 1 -- 172.21.15.37:0/3107162483 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6764135490 msgr2=0x7f6764131f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:20:56.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.971+0000 7f676c26f640 1 --2- 172.21.15.37:0/3107162483 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6764135490 0x7f6764131f20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f676000c3c0 tx=0x7f676000c880 comp rx=0 tx=0).stop 2024-04-26T00:20:56.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.971+0000 7f676c26f640 1 -- 172.21.15.37:0/3107162483 shutdown_connections 2024-04-26T00:20:56.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.971+0000 7f676c26f640 1 --2- 172.21.15.37:0/3107162483 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6748035e80 0x7f6748038340 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:56.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.971+0000 7f676c26f640 1 --2- 172.21.15.37:0/3107162483 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6764135490 0x7f6764131f20 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:56.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.971+0000 7f676c26f640 1 --2- 172.21.15.37:0/3107162483 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6764134b30 0x7f67641319e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:56.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.971+0000 7f676c26f640 1 --2- 172.21.15.37:0/3107162483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67640a0260 0x7f6764070fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:20:56.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.971+0000 7f676c26f640 1 -- 172.21.15.37:0/3107162483 >> 172.21.15.37:0/3107162483 conn(0x7f67640a1ad0 msgr2=0x7f67640a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:20:56.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.971+0000 7f676c26f640 1 -- 172.21.15.37:0/3107162483 shutdown_connections 2024-04-26T00:20:56.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:20:56.971+0000 7f676c26f640 1 -- 172.21.15.37:0/3107162483 wait complete. 2024-04-26T00:20:59.967 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:21:00.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.111+0000 7f6314e89640 10 monclient: get_monmap_and_config 2024-04-26T00:21:00.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.111+0000 7f6314e89640 10 monclient: build_initial_monmap 2024-04-26T00:21:00.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.111+0000 7f6314e89640 10 monclient: monmap: 2024-04-26T00:21:00.090 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:00.090 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:00.090 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:21:00.113950+0000 2024-04-26T00:21:00.090 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:21:00.113950+0000 2024-04-26T00:21:00.090 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:00.090 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:00.090 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:21:00.090 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:21:00.090 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:21:00.091 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:21:00.091 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:21:00.091 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:00.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.111+0000 7f6314e89640 1 Processor -- start 2024-04-26T00:21:00.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.111+0000 7f6314e89640 1 -- start start 2024-04-26T00:21:00.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.111+0000 7f6314e89640 10 monclient: init 2024-04-26T00:21:00.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.111+0000 7f6314e89640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:00.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.111+0000 7f6314e89640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-04-26T00:21:00.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.111+0000 7f6314e89640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6310133b20 0x7f631009e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:00.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.111+0000 7f6314e89640 10 monclient(hunting): picked mon.noname-b con 0x7f6310133b20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:21:00.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.111+0000 7f6314e89640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6310133f00 addr v1:172.21.15.45:6789/0 2024-04-26T00:21:00.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.111+0000 7f6314e89640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6310139050 0x7f631013b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:00.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.111+0000 7f6314e89640 10 monclient(hunting): picked mon.noname-c con 0x7f6310139050 addr v2:172.21.15.45:3300/0 2024-04-26T00:21:00.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.111+0000 7f6314e89640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f631009fe20 con 0x7f6310133f00 2024-04-26T00:21:00.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.111+0000 7f6314e89640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:00.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.111+0000 7f6314e89640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f631009e620 con 0x7f6310139050 2024-04-26T00:21:00.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.111+0000 7f6314e89640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:00.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.111+0000 7f6314e89640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f6310072d00 con 0x7f6310133b20 2024-04-26T00:21:00.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.111+0000 7f6314e89640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:00.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.111+0000 7f6314e89640 10 monclient(hunting): authenticate will time out at 1041.419752s 2024-04-26T00:21:00.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.111+0000 7f630ed9d640 1 -- >> v2:172.21.15.45:3300/0 conn(0x7f6310139050 msgr2=0x7f631013b440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:21:00.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.111+0000 7f630dd9b640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f6310133f00 legacy=0x7f6310134320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:21:00.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.111+0000 7f630ed9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6310139050 0x7f631013b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:21:00.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.111+0000 7f630e59c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6310133b20 0x7f631009e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:00.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.111+0000 7f630e59c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6310133b20 0x7f631009e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:44208/0 (socket says 172.21.15.37:44208) 2024-04-26T00:21:00.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.115+0000 7f630e59c640 1 -- 172.21.15.37:0/3856491058 learned_addr learned my addr 172.21.15.37:0/3856491058 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:21:00.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.115+0000 7f630e59c640 10 monclient(hunting): get_auth_request con 0x7f6310133b20 auth_method 0 2024-04-26T00:21:00.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.115+0000 7f630e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:00.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.115+0000 7f630e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:00.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.115+0000 7f630e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:00.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.115+0000 7f630e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:00.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.115+0000 7f630e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:00.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.115+0000 7f630e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:00.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.115+0000 7f630e59c640 10 monclient(hunting): handle_auth_done global_id 4475 payload 274 2024-04-26T00:21:00.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.115+0000 7f630e59c640 1 -- 172.21.15.37:0/3856491058 >> v2:172.21.15.45:3300/0 conn(0x7f6310139050 msgr2=0x7f631013b440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:21:00.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.115+0000 7f630e59c640 1 --2- 172.21.15.37:0/3856491058 >> v2:172.21.15.45:3300/0 conn(0x7f6310139050 0x7f631013b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:00.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.115+0000 7f630e59c640 1 -- 172.21.15.37:0/3856491058 >> v1:172.21.15.45:6789/0 conn(0x7f6310133f00 legacy=0x7f6310134320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:21:00.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.115+0000 7f630e59c640 10 monclient: _finish_hunting 0 2024-04-26T00:21:00.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.115+0000 7f630e59c640 1 monclient: found mon.noname-b 2024-04-26T00:21:00.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.115+0000 7f630e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:00.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.115+0000 7f630e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:21:00.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.115+0000 7f630e59c640 1 -- 172.21.15.37:0/3856491058 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f631009ba60 con 0x7f6310133b20 2024-04-26T00:21:00.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630e59c640 1 -- 172.21.15.37:0/3856491058 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6310133b20 msgr2=0x7f631009e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:00.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630e59c640 1 --2- 172.21.15.37:0/3856491058 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6310133b20 0x7f631009e9d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6300009870 tx=0x7f630001c220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82a2ff07aba0e036 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:00.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630d59a640 1 -- 172.21.15.37:0/3856491058 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63000b0070 con 0x7f6310133b20 2024-04-26T00:21:00.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:00.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:21:00.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630d59a640 10 monclient: dump: 2024-04-26T00:21:00.098 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:00.098 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:00.098 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:00.098 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:00.098 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:00.098 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:00.099 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:00.099 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:00.099 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:00.099 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:00.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630d59a640 10 monclient: _finish_auth 0 2024-04-26T00:21:00.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630d59a640 10 monclient: _check_auth_tickets 2024-04-26T00:21:00.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:00.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630d59a640 1 -- 172.21.15.37:0/3856491058 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63000ab600 con 0x7f6310133b20 2024-04-26T00:21:00.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f6314e89640 5 monclient: authenticate success, global_id 4475 2024-04-26T00:21:00.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f6314e89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:21:00.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630d59a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:00.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f6314e89640 10 monclient: get_monmap_and_config success 2024-04-26T00:21:00.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630d59a640 1 -- 172.21.15.37:0/3856491058 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63000ab9d0 con 0x7f6310133b20 2024-04-26T00:21:00.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f6314e89640 4 set_mon_vals no callback set 2024-04-26T00:21:00.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:00.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:21:00.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630d59a640 10 monclient: dump: 2024-04-26T00:21:00.100 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:00.100 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:00.100 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:00.100 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:00.100 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:00.100 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:00.100 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:00.100 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:00.100 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:00.100 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:00.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f6314e89640 10 monclient: shutdown 2024-04-26T00:21:00.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f6314e89640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:00.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f6314e89640 1 -- 172.21.15.37:0/3856491058 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6310133b20 msgr2=0x7f631009e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:00.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f6314e89640 1 --2- 172.21.15.37:0/3856491058 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6310133b20 0x7f631009e9d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6300009870 tx=0x7f630001c220 comp rx=0 tx=0).stop 2024-04-26T00:21:00.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f6314e89640 1 -- 172.21.15.37:0/3856491058 shutdown_connections 2024-04-26T00:21:00.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f6314e89640 1 --2- 172.21.15.37:0/3856491058 >> v2:172.21.15.45:3300/0 conn(0x7f6310139050 0x7f631013b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:00.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f6314e89640 1 --2- 172.21.15.37:0/3856491058 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6310133b20 0x7f631009e9d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:00.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f6314e89640 1 -- 172.21.15.37:0/3856491058 >> 172.21.15.37:0/3856491058 conn(0x7f63100a1ad0 msgr2=0x7f63100a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:00.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f6314e89640 1 -- 172.21.15.37:0/3856491058 shutdown_connections 2024-04-26T00:21:00.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f6314e89640 1 -- 172.21.15.37:0/3856491058 wait complete. 2024-04-26T00:21:00.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f6314e89640 10 monclient: build_initial_monmap 2024-04-26T00:21:00.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f6314e89640 10 monclient: monmap: 2024-04-26T00:21:00.101 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:00.101 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:21:00.101 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:21:00.101 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:21:00.101 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:00.102 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:21:00.102 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:21:00.102 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:21:00.102 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:21:00.102 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:00.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f6314e89640 1 Processor -- start 2024-04-26T00:21:00.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f6314e89640 1 -- start start 2024-04-26T00:21:00.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f6314e89640 10 monclient: init 2024-04-26T00:21:00.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f6314e89640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:00.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f6314e89640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:21:00.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f6314e89640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6310133f00 0x7f63101cf410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:00.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f6314e89640 10 monclient(hunting): picked mon.noname-c con 0x7f6310133f00 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:21:00.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f6314e89640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6310139050 0x7f63101cf950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:00.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f6314e89640 10 monclient(hunting): picked mon.noname-b con 0x7f6310139050 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:21:00.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f6314e89640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f63101cb990 0x7f63101cbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:00.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f6314e89640 10 monclient(hunting): picked mon.noname-a con 0x7f63101cb990 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:21:00.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f6314e89640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:00.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f6314e89640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f631013c350 con 0x7f63101cb990 2024-04-26T00:21:00.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f6314e89640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:00.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f6314e89640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f631013c1d0 con 0x7f6310139050 2024-04-26T00:21:00.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f6314e89640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:00.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f6314e89640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f631013c4d0 con 0x7f6310133f00 2024-04-26T00:21:00.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f6314e89640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:00.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f6314e89640 10 monclient(hunting): authenticate will time out at 1041.427093s 2024-04-26T00:21:00.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630dd9b640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6310139050 msgr2=0x7f63101cf950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:21:00.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630e59c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6310133f00 0x7f63101cf410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:00.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630dd9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6310139050 0x7f63101cf950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:21:00.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630ed9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f63101cb990 0x7f63101cbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:00.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630ed9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f63101cb990 0x7f63101cbdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44050/0 (socket says 172.21.15.37:44050) 2024-04-26T00:21:00.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630ed9d640 1 -- 172.21.15.37:0/2433350915 learned_addr learned my addr 172.21.15.37:0/2433350915 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:21:00.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630e59c640 10 monclient(hunting): get_auth_request con 0x7f6310133f00 auth_method 0 2024-04-26T00:21:00.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:00.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:00.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:00.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630ed9d640 10 monclient(hunting): get_auth_request con 0x7f63101cb990 auth_method 0 2024-04-26T00:21:00.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:00.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:00.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:00.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:00.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:00.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:00.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:00.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:00.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:00.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630ed9d640 10 monclient(hunting): handle_auth_done global_id 4485 payload 824 2024-04-26T00:21:00.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630ed9d640 1 -- 172.21.15.37:0/2433350915 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6310133f00 msgr2=0x7f63101cf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:00.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630ed9d640 1 --2- 172.21.15.37:0/2433350915 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6310133f00 0x7f63101cf410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:00.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630ed9d640 1 -- 172.21.15.37:0/2433350915 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6310139050 msgr2=0x7f63101cf950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:21:00.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630ed9d640 1 --2- 172.21.15.37:0/2433350915 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6310139050 0x7f63101cf950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:00.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630ed9d640 10 monclient: _finish_hunting 0 2024-04-26T00:21:00.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630ed9d640 1 monclient: found mon.noname-a 2024-04-26T00:21:00.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:00.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:21:00.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.119+0000 7f630ed9d640 1 -- 172.21.15.37:0/2433350915 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f631006f910 con 0x7f63101cb990 2024-04-26T00:21:00.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.123+0000 7f630ed9d640 1 -- 172.21.15.37:0/2433350915 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f63101cb990 msgr2=0x7f63101cbdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:00.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.123+0000 7f630ed9d640 1 --2- 172.21.15.37:0/2433350915 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f63101cb990 0x7f63101cbdd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f630800ba10 tx=0x7f630800bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:00.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.123+0000 7f62feffd640 1 -- 172.21.15.37:0/2433350915 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f630800cd50 con 0x7f63101cb990 2024-04-26T00:21:00.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.123+0000 7f62feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:00.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.123+0000 7f62feffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:21:00.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.123+0000 7f62feffd640 10 monclient: dump: 2024-04-26T00:21:00.110 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:00.111 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:00.111 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:00.111 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:00.111 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:00.111 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:00.111 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:00.111 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:00.111 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:00.111 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:00.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.123+0000 7f62feffd640 10 monclient: _finish_auth 0 2024-04-26T00:21:00.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.123+0000 7f62feffd640 10 monclient: _check_auth_tickets 2024-04-26T00:21:00.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.123+0000 7f62feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:00.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.123+0000 7f6314e89640 5 monclient: authenticate success, global_id 4485 2024-04-26T00:21:00.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.123+0000 7f62feffd640 1 -- 172.21.15.37:0/2433350915 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6308010070 con 0x7f63101cb990 2024-04-26T00:21:00.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.123+0000 7f6314e89640 10 monclient: _renew_subs 2024-04-26T00:21:00.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.123+0000 7f6314e89640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:00.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.123+0000 7f6314e89640 1 -- 172.21.15.37:0/2433350915 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63100a0000 con 0x7f63101cb990 2024-04-26T00:21:00.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.123+0000 7f62feffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:00.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.123+0000 7f62feffd640 1 -- 172.21.15.37:0/2433350915 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63080156b0 con 0x7f63101cb990 2024-04-26T00:21:00.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.123+0000 7f62feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:00.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.123+0000 7f62feffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:21:00.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.123+0000 7f62feffd640 10 monclient: dump: 2024-04-26T00:21:00.112 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:00.112 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:00.112 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:00.112 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:00.112 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:00.112 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:00.113 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:00.113 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:00.113 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:00.113 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:00.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.123+0000 7f6314e89640 10 monclient: _renew_subs 2024-04-26T00:21:00.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.123+0000 7f62fffff640 4 set_mon_vals no callback set 2024-04-26T00:21:00.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.123+0000 7f6314e89640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:00.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.123+0000 7f6314e89640 1 -- 172.21.15.37:0/2433350915 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6310071c00 con 0x7f63101cb990 2024-04-26T00:21:00.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.123+0000 7f6314e89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:00.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.123+0000 7f6314e89640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:00.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.123+0000 7f6314e89640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:00.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.123+0000 7f6314e89640 1 -- 172.21.15.37:0/2433350915 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6310066f40 con 0x7f63101cb990 2024-04-26T00:21:00.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.123+0000 7f62feffd640 1 -- 172.21.15.37:0/2433350915 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f630801e3f0 con 0x7f63101cb990 2024-04-26T00:21:00.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.123+0000 7f62feffd640 1 --2- 172.21.15.37:0/2433350915 >> v2:172.21.15.45:6812/2379918623 conn(0x7f62f0035e80 0x7f62f0038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:00.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.123+0000 7f62feffd640 1 -- 172.21.15.37:0/2433350915 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 3915+0+0 (secure 0 0 0) 0x7f6308048cc0 con 0x7f63101cb990 2024-04-26T00:21:00.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.127+0000 7f630e59c640 1 --2- 172.21.15.37:0/2433350915 >> v2:172.21.15.45:6812/2379918623 conn(0x7f62f0035e80 0x7f62f0038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:00.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.127+0000 7f630e59c640 10 monclient: get_auth_request con 0x7f62f0035e80 auth_method 0 2024-04-26T00:21:00.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.127+0000 7f630e59c640 1 --2- 172.21.15.37:0/2433350915 >> v2:172.21.15.45:6812/2379918623 conn(0x7f62f0035e80 0x7f62f0038340 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f631006b640 tx=0x7f63000a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:00.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.131+0000 7f62feffd640 1 -- 172.21.15.37:0/2433350915 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f630801d3a0 con 0x7f63101cb990 2024-04-26T00:21:00.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.131+0000 7f62feffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:00.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.131+0000 7f62feffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:21:00.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.351+0000 7f6314e89640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:21:00.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.351+0000 7f6314e89640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:21:00.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.351+0000 7f6314e89640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:00.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.351+0000 7f6314e89640 1 -- 172.21.15.37:0/2433350915 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6310070690 con 0x7f63101cb990 2024-04-26T00:21:00.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.351+0000 7f62feffd640 1 -- 172.21.15.37:0/2433350915 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f630801eb50 con 0x7f63101cb990 2024-04-26T00:21:00.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.351+0000 7f62feffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:21:00.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.351+0000 7f62feffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:21:00.329 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:21:00.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.351+0000 7f6314e89640 1 -- 172.21.15.37:0/2433350915 >> v2:172.21.15.45:6812/2379918623 conn(0x7f62f0035e80 msgr2=0x7f62f0038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:00.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.351+0000 7f6314e89640 1 --2- 172.21.15.37:0/2433350915 >> v2:172.21.15.45:6812/2379918623 conn(0x7f62f0035e80 0x7f62f0038340 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f631006b640 tx=0x7f63000a6000 comp rx=0 tx=0).stop 2024-04-26T00:21:00.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.351+0000 7f6314e89640 10 monclient: shutdown 2024-04-26T00:21:00.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.351+0000 7f6314e89640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:00.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.351+0000 7f6314e89640 1 -- 172.21.15.37:0/2433350915 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f63101cb990 msgr2=0x7f63101cbdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:00.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.351+0000 7f6314e89640 1 --2- 172.21.15.37:0/2433350915 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f63101cb990 0x7f63101cbdd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f630800ba10 tx=0x7f630800bed0 comp rx=0 tx=0).stop 2024-04-26T00:21:00.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.351+0000 7f6314e89640 1 -- 172.21.15.37:0/2433350915 shutdown_connections 2024-04-26T00:21:00.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.351+0000 7f6314e89640 1 --2- 172.21.15.37:0/2433350915 >> v2:172.21.15.45:6812/2379918623 conn(0x7f62f0035e80 0x7f62f0038340 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:00.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.351+0000 7f6314e89640 1 --2- 172.21.15.37:0/2433350915 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f63101cb990 0x7f63101cbdd0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:00.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.351+0000 7f6314e89640 1 --2- 172.21.15.37:0/2433350915 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6310139050 0x7f63101cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:00.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.351+0000 7f6314e89640 1 --2- 172.21.15.37:0/2433350915 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6310133f00 0x7f63101cf410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:00.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.351+0000 7f6314e89640 1 -- 172.21.15.37:0/2433350915 >> 172.21.15.37:0/2433350915 conn(0x7f63100a1ad0 msgr2=0x7f631009c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:00.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.351+0000 7f6314e89640 1 -- 172.21.15.37:0/2433350915 shutdown_connections 2024-04-26T00:21:00.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:00.351+0000 7f6314e89640 1 -- 172.21.15.37:0/2433350915 wait complete. 2024-04-26T00:21:02.611 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-04-26T00:21:03.347 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:21:03.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.503+0000 7f176231d640 10 monclient: get_monmap_and_config 2024-04-26T00:21:03.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.503+0000 7f176231d640 10 monclient: build_initial_monmap 2024-04-26T00:21:03.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.503+0000 7f176231d640 10 monclient: monmap: 2024-04-26T00:21:03.481 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:03.481 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:03.481 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:21:03.505580+0000 2024-04-26T00:21:03.482 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:21:03.505580+0000 2024-04-26T00:21:03.482 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:03.482 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:03.482 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:21:03.482 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:21:03.482 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:21:03.482 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:21:03.482 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:21:03.482 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:03.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.503+0000 7f176231d640 1 Processor -- start 2024-04-26T00:21:03.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.503+0000 7f176231d640 1 -- start start 2024-04-26T00:21:03.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.503+0000 7f176231d640 10 monclient: init 2024-04-26T00:21:03.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.503+0000 7f176231d640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:03.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.503+0000 7f176231d640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-04-26T00:21:03.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.503+0000 7f176231d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f175c0a0260 0x7f175c0a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:03.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.503+0000 7f176231d640 10 monclient(hunting): picked mon.noname-c con 0x7f175c0a0260 addr v2:172.21.15.45:3300/0 2024-04-26T00:21:03.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.503+0000 7f176231d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f175c134b30 0x7f175c134f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:03.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.503+0000 7f176231d640 10 monclient(hunting): picked mon.noname-a con 0x7f175c134b30 addr v2:172.21.15.37:3300/0 2024-04-26T00:21:03.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.503+0000 7f176231d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f175c135490 addr v1:172.21.15.45:6789/0 2024-04-26T00:21:03.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.503+0000 7f176231d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f175c0a1130 con 0x7f175c135490 2024-04-26T00:21:03.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.503+0000 7f176231d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:03.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.503+0000 7f176231d640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f175c09ec10 con 0x7f175c134b30 2024-04-26T00:21:03.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.503+0000 7f176231d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:03.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.503+0000 7f176231d640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f175c09f0f0 con 0x7f175c0a0260 2024-04-26T00:21:03.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.503+0000 7f176231d640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:03.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.503+0000 7f176231d640 10 monclient(hunting): authenticate will time out at 1044.811451s 2024-04-26T00:21:03.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.503+0000 7f175bfff640 1 -- >> v2:172.21.15.45:3300/0 conn(0x7f175c0a0260 msgr2=0x7f175c0a0640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:21:03.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.503+0000 7f175bfff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f175c0a0260 0x7f175c0a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:21:03.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.503+0000 7f17608ba640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f175c135490 legacy=0x7f175c13be60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:21:03.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.503+0000 7f175b7fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f175c134b30 0x7f175c134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:03.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.503+0000 7f175b7fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f175c134b30 0x7f175c134f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44064/0 (socket says 172.21.15.37:44064) 2024-04-26T00:21:03.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.503+0000 7f175b7fe640 1 -- 172.21.15.37:0/4003515635 learned_addr learned my addr 172.21.15.37:0/4003515635 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:21:03.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.503+0000 7f175b7fe640 10 monclient(hunting): get_auth_request con 0x7f175c134b30 auth_method 0 2024-04-26T00:21:03.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.503+0000 7f175b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:03.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.503+0000 7f175b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:03.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.503+0000 7f175b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:03.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.503+0000 7f175b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:03.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.503+0000 7f175b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:03.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.503+0000 7f175b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:03.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.507+0000 7f175b7fe640 10 monclient(hunting): handle_auth_done global_id 4488 payload 274 2024-04-26T00:21:03.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.507+0000 7f175b7fe640 1 -- 172.21.15.37:0/4003515635 >> v2:172.21.15.45:3300/0 conn(0x7f175c0a0260 msgr2=0x7f175c0a0640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:21:03.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.507+0000 7f175b7fe640 1 --2- 172.21.15.37:0/4003515635 >> v2:172.21.15.45:3300/0 conn(0x7f175c0a0260 0x7f175c0a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:03.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.507+0000 7f175b7fe640 1 -- 172.21.15.37:0/4003515635 >> v1:172.21.15.45:6789/0 conn(0x7f175c135490 legacy=0x7f175c13be60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:21:03.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.507+0000 7f175b7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:21:03.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.507+0000 7f175b7fe640 1 monclient: found mon.noname-a 2024-04-26T00:21:03.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.507+0000 7f175b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:03.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.507+0000 7f175b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:21:03.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.507+0000 7f175b7fe640 1 -- 172.21.15.37:0/4003515635 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f175c06cff0 con 0x7f175c134b30 2024-04-26T00:21:03.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f175b7fe640 1 -- 172.21.15.37:0/4003515635 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f175c134b30 msgr2=0x7f175c134f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:03.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f175b7fe640 1 --2- 172.21.15.37:0/4003515635 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f175c134b30 0x7f175c134f50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f175c0a2ad0 tx=0x7f174401c1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca150029a92aa845 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:03.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f175affd640 1 -- 172.21.15.37:0/4003515635 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17440b0070 con 0x7f175c134b30 2024-04-26T00:21:03.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f175affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:03.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f175affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:21:03.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f175affd640 10 monclient: dump: 2024-04-26T00:21:03.492 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:03.492 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:03.492 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:03.493 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:03.493 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:03.493 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:03.493 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:03.493 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:03.493 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:03.493 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:03.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f175affd640 10 monclient: _finish_auth 0 2024-04-26T00:21:03.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f175affd640 10 monclient: _check_auth_tickets 2024-04-26T00:21:03.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f175affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:03.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f175affd640 1 -- 172.21.15.37:0/4003515635 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17440ab600 con 0x7f175c134b30 2024-04-26T00:21:03.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f175affd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:03.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f175affd640 1 -- 172.21.15.37:0/4003515635 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17440ab9d0 con 0x7f175c134b30 2024-04-26T00:21:03.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f176231d640 5 monclient: authenticate success, global_id 4488 2024-04-26T00:21:03.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f176231d640 10 monclient: get_monmap_and_config success 2024-04-26T00:21:03.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f176231d640 4 set_mon_vals no callback set 2024-04-26T00:21:03.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f175affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:03.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f175affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:21:03.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f175affd640 10 monclient: dump: 2024-04-26T00:21:03.494 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:03.494 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:03.494 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:03.494 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:03.494 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:03.495 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:03.495 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:03.495 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:03.495 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:03.495 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:03.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f176231d640 10 monclient: shutdown 2024-04-26T00:21:03.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f176231d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:03.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f176231d640 1 -- 172.21.15.37:0/4003515635 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f175c134b30 msgr2=0x7f175c134f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:03.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f176231d640 1 --2- 172.21.15.37:0/4003515635 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f175c134b30 0x7f175c134f50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f175c0a2ad0 tx=0x7f174401c1e0 comp rx=0 tx=0).stop 2024-04-26T00:21:03.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f176231d640 1 -- 172.21.15.37:0/4003515635 shutdown_connections 2024-04-26T00:21:03.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f176231d640 1 --2- 172.21.15.37:0/4003515635 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f175c134b30 0x7f175c134f50 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:03.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f176231d640 1 --2- 172.21.15.37:0/4003515635 >> v2:172.21.15.45:3300/0 conn(0x7f175c0a0260 0x7f175c0a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:03.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f176231d640 1 -- 172.21.15.37:0/4003515635 >> 172.21.15.37:0/4003515635 conn(0x7f175c0a1ad0 msgr2=0x7f175c0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:03.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f176231d640 1 -- 172.21.15.37:0/4003515635 shutdown_connections 2024-04-26T00:21:03.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f176231d640 1 -- 172.21.15.37:0/4003515635 wait complete. 2024-04-26T00:21:03.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f176231d640 10 monclient: build_initial_monmap 2024-04-26T00:21:03.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f176231d640 10 monclient: monmap: 2024-04-26T00:21:03.498 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:03.498 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:21:03.498 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:21:03.498 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:21:03.499 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:03.499 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:21:03.499 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:21:03.499 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:21:03.499 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:21:03.499 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:03.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f176231d640 1 Processor -- start 2024-04-26T00:21:03.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f176231d640 1 -- start start 2024-04-26T00:21:03.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f176231d640 10 monclient: init 2024-04-26T00:21:03.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f176231d640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:03.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f176231d640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:21:03.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f176231d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f175c0a0260 0x7f175c130940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:03.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f176231d640 10 monclient(hunting): picked mon.noname-b con 0x7f175c0a0260 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:21:03.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f176231d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f175c134b30 0x7f175c130e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:03.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f176231d640 10 monclient(hunting): picked mon.noname-a con 0x7f175c134b30 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:21:03.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f176231d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f175c135490 0x7f175c1313c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:03.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f176231d640 10 monclient(hunting): picked mon.noname-c con 0x7f175c135490 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:21:03.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f176231d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:03.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f176231d640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f175c09f430 con 0x7f175c134b30 2024-04-26T00:21:03.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f176231d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:03.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f176231d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f175c09f2b0 con 0x7f175c0a0260 2024-04-26T00:21:03.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f176231d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:03.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f176231d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f175c09fe90 con 0x7f175c135490 2024-04-26T00:21:03.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f176231d640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:03.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f176231d640 10 monclient(hunting): authenticate will time out at 1044.818732s 2024-04-26T00:21:03.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f175b7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f175c134b30 0x7f175c130e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:03.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f17608ba640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f175c135490 0x7f175c1313c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:03.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f175b7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f175c134b30 0x7f175c130e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44070/0 (socket says 172.21.15.37:44070) 2024-04-26T00:21:03.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f175b7fe640 1 -- 172.21.15.37:0/4215399200 learned_addr learned my addr 172.21.15.37:0/4215399200 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:21:03.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f175b7fe640 10 monclient(hunting): get_auth_request con 0x7f175c134b30 auth_method 0 2024-04-26T00:21:03.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f175bfff640 1 -- 172.21.15.37:0/4215399200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f175c0a0260 msgr2=0x7f175c130940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:21:03.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f175b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:03.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f175b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:03.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f175b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:03.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f175bfff640 1 --2- 172.21.15.37:0/4215399200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f175c0a0260 0x7f175c130940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:21:03.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f17608ba640 10 monclient(hunting): get_auth_request con 0x7f175c135490 auth_method 0 2024-04-26T00:21:03.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f17608ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:03.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f17608ba640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:03.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f17608ba640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:03.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f175b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:03.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f175b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:03.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f175b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:03.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f17608ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:03.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f17608ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:03.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f17608ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:03.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f175b7fe640 10 monclient(hunting): handle_auth_done global_id 4491 payload 824 2024-04-26T00:21:03.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f175b7fe640 1 -- 172.21.15.37:0/4215399200 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f175c135490 msgr2=0x7f175c1313c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:03.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f175b7fe640 1 --2- 172.21.15.37:0/4215399200 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f175c135490 0x7f175c1313c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:03.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f175b7fe640 1 -- 172.21.15.37:0/4215399200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f175c0a0260 msgr2=0x7f175c130940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:21:03.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f175b7fe640 1 --2- 172.21.15.37:0/4215399200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f175c0a0260 0x7f175c130940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:03.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f175b7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:21:03.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f175b7fe640 1 monclient: found mon.noname-a 2024-04-26T00:21:03.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f175b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:03.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f175b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:21:03.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.511+0000 7f175b7fe640 1 -- 172.21.15.37:0/4215399200 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f175c0729d0 con 0x7f175c134b30 2024-04-26T00:21:03.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.515+0000 7f175b7fe640 1 -- 172.21.15.37:0/4215399200 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f175c134b30 msgr2=0x7f175c130e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:03.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.515+0000 7f175b7fe640 1 --2- 172.21.15.37:0/4215399200 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f175c134b30 0x7f175c130e80 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1744018df0 tx=0x7f1744018f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:03.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.515+0000 7f175b7fe640 1 --2- 172.21.15.37:0/4215399200 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f175c134b30 0x7f175c130e80 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1744018df0 tx=0x7f1744018f80 comp rx=0 tx=0).queue_received will delay after 1.71409e+09 on 0x7f17440b1040 config(0 keys) 2024-04-26T00:21:03.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.515+0000 7f1758ff9640 1 -- 172.21.15.37:0/4215399200 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17440b9020 con 0x7f175c134b30 2024-04-26T00:21:03.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.515+0000 7f1758ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:03.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.515+0000 7f1758ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:21:03.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.515+0000 7f1758ff9640 10 monclient: dump: 2024-04-26T00:21:03.512 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:03.512 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:03.512 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:03.513 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:03.513 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:03.513 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:03.513 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:03.513 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:03.513 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:03.513 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:03.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.515+0000 7f1758ff9640 10 monclient: _finish_auth 0 2024-04-26T00:21:03.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.515+0000 7f1758ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:21:03.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.515+0000 7f1758ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:03.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.515+0000 7f176231d640 5 monclient: authenticate success, global_id 4491 2024-04-26T00:21:03.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.515+0000 7f176231d640 10 monclient: _renew_subs 2024-04-26T00:21:03.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.515+0000 7f1758ff9640 1 -- 172.21.15.37:0/4215399200 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17440b1040 con 0x7f175c134b30 2024-04-26T00:21:03.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.515+0000 7f176231d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:03.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.515+0000 7f176231d640 1 -- 172.21.15.37:0/4215399200 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f175c132490 con 0x7f175c134b30 2024-04-26T00:21:03.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.515+0000 7f1758ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:03.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.515+0000 7f175a7fc640 4 set_mon_vals no callback set 2024-04-26T00:21:03.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.515+0000 7f176231d640 10 monclient: _renew_subs 2024-04-26T00:21:03.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.515+0000 7f176231d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:03.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.515+0000 7f176231d640 1 -- 172.21.15.37:0/4215399200 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f175c06f890 con 0x7f175c134b30 2024-04-26T00:21:03.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.515+0000 7f1758ff9640 1 -- 172.21.15.37:0/4215399200 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17440b0050 con 0x7f175c134b30 2024-04-26T00:21:03.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.515+0000 7f1758ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:03.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.515+0000 7f1758ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:21:03.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.515+0000 7f1758ff9640 10 monclient: dump: 2024-04-26T00:21:03.516 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:03.516 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:03.516 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:03.516 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:03.516 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:03.516 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:03.516 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:03.516 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:03.517 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:03.517 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:03.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.515+0000 7f1758ff9640 1 -- 172.21.15.37:0/4215399200 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f17440bc020 con 0x7f175c134b30 2024-04-26T00:21:03.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.515+0000 7f176231d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:03.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.515+0000 7f176231d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:03.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.515+0000 7f176231d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:03.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.515+0000 7f176231d640 1 -- 172.21.15.37:0/4215399200 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f175c066f40 con 0x7f175c134b30 2024-04-26T00:21:03.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.515+0000 7f1758ff9640 1 --2- 172.21.15.37:0/4215399200 >> v2:172.21.15.45:6812/2379918623 conn(0x7f173c035e60 0x7f173c038320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:03.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.515+0000 7f175bfff640 1 --2- 172.21.15.37:0/4215399200 >> v2:172.21.15.45:6812/2379918623 conn(0x7f173c035e60 0x7f173c038320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:03.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.515+0000 7f175bfff640 10 monclient: get_auth_request con 0x7f173c035e60 auth_method 0 2024-04-26T00:21:03.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.519+0000 7f175bfff640 1 --2- 172.21.15.37:0/4215399200 >> v2:172.21.15.45:6812/2379918623 conn(0x7f173c035e60 0x7f173c038320 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1754003cc0 tx=0x7f1754009270 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:03.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:03.519+0000 7f1758ff9640 1 -- 172.21.15.37:0/4215399200 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 3915+0+0 (secure 0 0 0) 0x7f17440e29f0 con 0x7f175c134b30 2024-04-26T00:21:04.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:04.139+0000 7f1758ff9640 1 -- 172.21.15.37:0/4215399200 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f17440c3030 con 0x7f175c134b30 2024-04-26T00:21:04.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:04.139+0000 7f1758ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:04.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:04.139+0000 7f1758ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:21:04.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:04.375+0000 7f176231d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:21:04.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:04.375+0000 7f176231d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:21:04.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:04.375+0000 7f176231d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:04.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:04.375+0000 7f176231d640 1 -- 172.21.15.37:0/4215399200 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f175c06fa90 con 0x7f175c134b30 2024-04-26T00:21:04.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:04.375+0000 7f1758ff9640 1 -- 172.21.15.37:0/4215399200 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f175c0729d0 con 0x7f175c134b30 2024-04-26T00:21:04.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:04.375+0000 7f1758ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:21:04.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:04.375+0000 7f1758ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:21:04.355 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:21:04.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:04.379+0000 7f176231d640 1 -- 172.21.15.37:0/4215399200 >> v2:172.21.15.45:6812/2379918623 conn(0x7f173c035e60 msgr2=0x7f173c038320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:04.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:04.379+0000 7f176231d640 1 --2- 172.21.15.37:0/4215399200 >> v2:172.21.15.45:6812/2379918623 conn(0x7f173c035e60 0x7f173c038320 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1754003cc0 tx=0x7f1754009270 comp rx=0 tx=0).stop 2024-04-26T00:21:04.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:04.379+0000 7f176231d640 10 monclient: shutdown 2024-04-26T00:21:04.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:04.379+0000 7f176231d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:04.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:04.379+0000 7f176231d640 1 -- 172.21.15.37:0/4215399200 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f175c134b30 msgr2=0x7f175c130e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:04.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:04.379+0000 7f176231d640 1 --2- 172.21.15.37:0/4215399200 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f175c134b30 0x7f175c130e80 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1744018df0 tx=0x7f1744018f80 comp rx=0 tx=0).stop 2024-04-26T00:21:04.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:04.379+0000 7f176231d640 1 -- 172.21.15.37:0/4215399200 shutdown_connections 2024-04-26T00:21:04.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:04.379+0000 7f176231d640 1 --2- 172.21.15.37:0/4215399200 >> v2:172.21.15.45:6812/2379918623 conn(0x7f173c035e60 0x7f173c038320 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:04.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:04.379+0000 7f176231d640 1 --2- 172.21.15.37:0/4215399200 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f175c135490 0x7f175c1313c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:04.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:04.379+0000 7f176231d640 1 --2- 172.21.15.37:0/4215399200 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f175c134b30 0x7f175c130e80 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:04.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:04.379+0000 7f176231d640 1 --2- 172.21.15.37:0/4215399200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f175c0a0260 0x7f175c130940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:04.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:04.379+0000 7f176231d640 1 -- 172.21.15.37:0/4215399200 >> 172.21.15.37:0/4215399200 conn(0x7f175c0a1ad0 msgr2=0x7f175c0a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:04.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:04.379+0000 7f176231d640 1 -- 172.21.15.37:0/4215399200 shutdown_connections 2024-04-26T00:21:04.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:04.379+0000 7f176231d640 1 -- 172.21.15.37:0/4215399200 wait complete. 2024-04-26T00:21:07.374 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:21:07.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.527+0000 7fa1bb5cf640 10 monclient: get_monmap_and_config 2024-04-26T00:21:07.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.527+0000 7fa1bb5cf640 10 monclient: build_initial_monmap 2024-04-26T00:21:07.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.527+0000 7fa1bb5cf640 10 monclient: monmap: 2024-04-26T00:21:07.505 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:07.505 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:07.506 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:21:07.529662+0000 2024-04-26T00:21:07.506 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:21:07.529662+0000 2024-04-26T00:21:07.506 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:07.506 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:07.506 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:21:07.506 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:21:07.506 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:21:07.506 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:21:07.506 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:21:07.506 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:07.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.527+0000 7fa1bb5cf640 1 Processor -- start 2024-04-26T00:21:07.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.527+0000 7fa1bb5cf640 1 -- start start 2024-04-26T00:21:07.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.527+0000 7fa1bb5cf640 10 monclient: init 2024-04-26T00:21:07.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.531+0000 7fa1bb5cf640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:07.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.531+0000 7fa1bb5cf640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-04-26T00:21:07.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.531+0000 7fa1bb5cf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa1b4136e10 addr v1:172.21.15.45:6789/0 2024-04-26T00:21:07.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.531+0000 7fa1bb5cf640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa1b4137900 0x7fa1b409c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:07.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.531+0000 7fa1bb5cf640 10 monclient(hunting): picked mon.noname-c con 0x7fa1b4137900 addr v2:172.21.15.45:3300/0 2024-04-26T00:21:07.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.531+0000 7fa1bb5cf640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa1b413c340 0x7fa1b413e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:07.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.531+0000 7fa1bb5cf640 10 monclient(hunting): picked mon.noname-a con 0x7fa1b413c340 addr v2:172.21.15.37:3300/0 2024-04-26T00:21:07.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.531+0000 7fa1bb5cf640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1b409b680 con 0x7fa1b4136e10 2024-04-26T00:21:07.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.531+0000 7fa1bb5cf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:07.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.531+0000 7fa1bb5cf640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fa1b409b8f0 con 0x7fa1b413c340 2024-04-26T00:21:07.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.531+0000 7fa1bb5cf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:07.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.531+0000 7fa1bb5cf640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fa1b409bbd0 con 0x7fa1b4137900 2024-04-26T00:21:07.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.531+0000 7fa1bb5cf640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:07.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.531+0000 7fa1bb5cf640 10 monclient(hunting): authenticate will time out at 1048.836195s 2024-04-26T00:21:07.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.531+0000 7fa1b936b640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7fa1b4136e10 legacy=0x7fa1b41371f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:21:07.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.531+0000 7fa1b8b6a640 1 -- >> v2:172.21.15.45:3300/0 conn(0x7fa1b4137900 msgr2=0x7fa1b409c5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:21:07.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.531+0000 7fa1b8b6a640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa1b4137900 0x7fa1b409c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:21:07.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.531+0000 7fa1b9b6c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa1b413c340 0x7fa1b413e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:07.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.531+0000 7fa1b9b6c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa1b413c340 0x7fa1b413e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44078/0 (socket says 172.21.15.37:44078) 2024-04-26T00:21:07.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.531+0000 7fa1b9b6c640 1 -- 172.21.15.37:0/2297777174 learned_addr learned my addr 172.21.15.37:0/2297777174 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:21:07.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.531+0000 7fa1b9b6c640 10 monclient(hunting): get_auth_request con 0x7fa1b413c340 auth_method 0 2024-04-26T00:21:07.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.531+0000 7fa1b9b6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:07.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.531+0000 7fa1b9b6c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:07.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.531+0000 7fa1b9b6c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:07.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.531+0000 7fa1b9b6c640 1 -- 172.21.15.37:0/2297777174 >> v2:172.21.15.37:3300/0 conn(0x7fa1b413c340 msgr2=0x7fa1b413e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:21:07.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.531+0000 7fa1b9b6c640 1 -- 172.21.15.37:0/2297777174 >> v2:172.21.15.37:3300/0 conn(0x7fa1b413c340 msgr2=0x7fa1b413e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:21:07.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.531+0000 7fa1b9b6c640 1 --2- 172.21.15.37:0/2297777174 >> v2:172.21.15.37:3300/0 conn(0x7fa1b413c340 0x7fa1b413e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:21:07.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.531+0000 7fa1b9b6c640 1 --2- 172.21.15.37:0/2297777174 >> v2:172.21.15.37:3300/0 conn(0x7fa1b413c340 0x7fa1b413e730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:21:07.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.735+0000 7fa1b8b6a640 1 -- 172.21.15.37:0/2297777174 >> v2:172.21.15.45:3300/0 conn(0x7fa1b4137900 msgr2=0x7fa1b409c5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:21:07.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.735+0000 7fa1b9b6c640 1 --2- 172.21.15.37:0/2297777174 >> v2:172.21.15.37:3300/0 conn(0x7fa1b413c340 0x7fa1b413e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:07.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.735+0000 7fa1b936b640 1 -- 172.21.15.37:0/2297777174 >> v1:172.21.15.45:6789/0 conn(0x7fa1b4136e10 legacy=0x7fa1b41371f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:21:07.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.735+0000 7fa1b8b6a640 1 --2- 172.21.15.37:0/2297777174 >> v2:172.21.15.45:3300/0 conn(0x7fa1b4137900 0x7fa1b409c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:21:07.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.735+0000 7fa1b9b6c640 10 monclient(hunting): get_auth_request con 0x7fa1b413c340 auth_method 0 2024-04-26T00:21:07.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.735+0000 7fa1b9b6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:07.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.735+0000 7fa1b9b6c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:07.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.735+0000 7fa1b9b6c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:21:07.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.735+0000 7fa1b9b6c640 1 -- 172.21.15.37:0/2297777174 >> v2:172.21.15.37:3300/0 conn(0x7fa1b413c340 msgr2=0x7fa1b413e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:21:07.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.735+0000 7fa1b9b6c640 1 -- 172.21.15.37:0/2297777174 >> v2:172.21.15.37:3300/0 conn(0x7fa1b413c340 msgr2=0x7fa1b413e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:21:07.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.735+0000 7fa1b9b6c640 1 --2- 172.21.15.37:0/2297777174 >> v2:172.21.15.37:3300/0 conn(0x7fa1b413c340 0x7fa1b413e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:21:07.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:07.735+0000 7fa1b9b6c640 1 --2- 172.21.15.37:0/2297777174 >> v2:172.21.15.37:3300/0 conn(0x7fa1b413c340 0x7fa1b413e730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:21:08.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:08.139+0000 7fa1b8b6a640 1 -- 172.21.15.37:0/2297777174 >> v2:172.21.15.45:3300/0 conn(0x7fa1b4137900 msgr2=0x7fa1b409c5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:21:08.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:08.139+0000 7fa1b936b640 1 -- 172.21.15.37:0/2297777174 >> v1:172.21.15.45:6789/0 conn(0x7fa1b4136e10 legacy=0x7fa1b41371f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:21:08.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:08.139+0000 7fa1b8b6a640 1 --2- 172.21.15.37:0/2297777174 >> v2:172.21.15.45:3300/0 conn(0x7fa1b4137900 0x7fa1b409c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:21:08.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:08.139+0000 7fa1b9b6c640 1 --2- 172.21.15.37:0/2297777174 >> v2:172.21.15.37:3300/0 conn(0x7fa1b413c340 0x7fa1b413e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:08.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:08.139+0000 7fa1b9b6c640 10 monclient(hunting): get_auth_request con 0x7fa1b413c340 auth_method 0 2024-04-26T00:21:08.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:08.139+0000 7fa1b9b6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:08.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:08.139+0000 7fa1b9b6c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:08.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:08.139+0000 7fa1b9b6c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:21:08.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:08.139+0000 7fa1b9b6c640 1 -- 172.21.15.37:0/2297777174 >> v2:172.21.15.37:3300/0 conn(0x7fa1b413c340 msgr2=0x7fa1b413e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:21:08.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:08.139+0000 7fa1b9b6c640 1 -- 172.21.15.37:0/2297777174 >> v2:172.21.15.37:3300/0 conn(0x7fa1b413c340 msgr2=0x7fa1b413e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:21:08.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:08.139+0000 7fa1b9b6c640 1 --2- 172.21.15.37:0/2297777174 >> v2:172.21.15.37:3300/0 conn(0x7fa1b413c340 0x7fa1b413e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:21:08.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:08.139+0000 7fa1b9b6c640 1 --2- 172.21.15.37:0/2297777174 >> v2:172.21.15.37:3300/0 conn(0x7fa1b413c340 0x7fa1b413e730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:21:08.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:08.943+0000 7fa1b8b6a640 1 -- 172.21.15.37:0/2297777174 >> v2:172.21.15.45:3300/0 conn(0x7fa1b4137900 msgr2=0x7fa1b409c5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:21:08.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:08.943+0000 7fa1b8b6a640 1 --2- 172.21.15.37:0/2297777174 >> v2:172.21.15.45:3300/0 conn(0x7fa1b4137900 0x7fa1b409c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:21:08.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:08.943+0000 7fa1b936b640 1 -- 172.21.15.37:0/2297777174 >> v1:172.21.15.45:6789/0 conn(0x7fa1b4136e10 legacy=0x7fa1b41371f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:21:08.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:08.943+0000 7fa1b9b6c640 1 --2- 172.21.15.37:0/2297777174 >> v2:172.21.15.37:3300/0 conn(0x7fa1b413c340 0x7fa1b413e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:08.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:08.943+0000 7fa1b9b6c640 10 monclient(hunting): get_auth_request con 0x7fa1b413c340 auth_method 0 2024-04-26T00:21:08.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:08.943+0000 7fa1b9b6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:08.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:08.943+0000 7fa1b9b6c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:08.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:08.943+0000 7fa1b9b6c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:21:08.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:08.943+0000 7fa1b9b6c640 1 -- 172.21.15.37:0/2297777174 >> v2:172.21.15.37:3300/0 conn(0x7fa1b413c340 msgr2=0x7fa1b413e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:21:08.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:08.943+0000 7fa1b9b6c640 1 -- 172.21.15.37:0/2297777174 >> v2:172.21.15.37:3300/0 conn(0x7fa1b413c340 msgr2=0x7fa1b413e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:21:08.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:08.943+0000 7fa1b9b6c640 1 --2- 172.21.15.37:0/2297777174 >> v2:172.21.15.37:3300/0 conn(0x7fa1b413c340 0x7fa1b413e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:21:08.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:08.943+0000 7fa1b9b6c640 1 --2- 172.21.15.37:0/2297777174 >> v2:172.21.15.37:3300/0 conn(0x7fa1b413c340 0x7fa1b413e730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:21:10.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:10.531+0000 7fa1aaffd640 10 monclient(hunting): tick 2024-04-26T00:21:10.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:10.531+0000 7fa1aaffd640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:21:10.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:10.531+0000 7fa1aaffd640 1 monclient(hunting): continuing hunt 2024-04-26T00:21:10.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:10.531+0000 7fa1aaffd640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:21:10.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:10.531+0000 7fa1aaffd640 1 -- 172.21.15.37:0/2297777174 >> v2:172.21.15.45:3300/0 conn(0x7fa1b4137900 msgr2=0x7fa1b409c5b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:21:10.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:10.531+0000 7fa1aaffd640 1 --2- 172.21.15.37:0/2297777174 >> v2:172.21.15.45:3300/0 conn(0x7fa1b4137900 0x7fa1b409c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:10.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:10.531+0000 7fa1aaffd640 1 -- 172.21.15.37:0/2297777174 >> v2:172.21.15.37:3300/0 conn(0x7fa1b413c340 msgr2=0x7fa1b413e730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:21:10.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:10.531+0000 7fa1aaffd640 1 --2- 172.21.15.37:0/2297777174 >> v2:172.21.15.37:3300/0 conn(0x7fa1b413c340 0x7fa1b413e730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:10.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:10.531+0000 7fa1aaffd640 1 -- 172.21.15.37:0/2297777174 >> v1:172.21.15.45:6789/0 conn(0x7fa1b4136e10 legacy=0x7fa1b41371f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:21:10.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:10.531+0000 7fa1aaffd640 10 monclient: _add_conns ranks=[2,0] 2024-04-26T00:21:10.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:10.531+0000 7fa1aaffd640 1 --2- 172.21.15.37:0/2297777174 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1980036a0 0x7fa198005af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:10.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:10.531+0000 7fa1aaffd640 10 monclient(hunting): picked mon.noname-b con 0x7fa1980036a0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:21:10.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:10.531+0000 7fa1aaffd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa1980061d0 addr v1:172.21.15.37:6789/0 2024-04-26T00:21:10.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:10.531+0000 7fa1aaffd640 1 -- 172.21.15.37:0/2297777174 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1b40a5d90 con 0x7fa1980061d0 2024-04-26T00:21:10.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:10.531+0000 7fa1aaffd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:10.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:10.531+0000 7fa1aaffd640 1 -- 172.21.15.37:0/2297777174 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa1b409b8f0 con 0x7fa1980036a0 2024-04-26T00:21:10.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:10.531+0000 7fa1aaffd640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:10.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:10.531+0000 7fa1b936b640 1 --2- 172.21.15.37:0/2297777174 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1980036a0 0x7fa198005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:10.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:10.531+0000 7fa1b936b640 10 monclient(hunting): get_auth_request con 0x7fa1980036a0 auth_method 0 2024-04-26T00:21:10.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:10.531+0000 7fa1b936b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:10.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:10.531+0000 7fa1b936b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:10.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:10.531+0000 7fa1b936b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:10.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:10.531+0000 7fa1b936b640 1 -- 172.21.15.37:0/2297777174 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1980036a0 msgr2=0x7fa198005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:21:10.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:10.531+0000 7fa1b936b640 1 -- 172.21.15.37:0/2297777174 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1980036a0 msgr2=0x7fa198005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:21:10.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:10.531+0000 7fa1b936b640 1 --2- 172.21.15.37:0/2297777174 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1980036a0 0x7fa198005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:21:10.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:10.531+0000 7fa1b936b640 1 --2- 172.21.15.37:0/2297777174 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1980036a0 0x7fa198005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:21:10.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:10.531+0000 7fa1b8b6a640 1 -- 172.21.15.37:0/2297777174 >> v1:172.21.15.37:6789/0 conn(0x7fa1980061d0 legacy=0x7fa198008670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:10.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:10.735+0000 7fa1b936b640 1 --2- 172.21.15.37:0/2297777174 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1980036a0 0x7fa198005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:10.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:10.735+0000 7fa1b936b640 10 monclient(hunting): get_auth_request con 0x7fa1980036a0 auth_method 0 2024-04-26T00:21:10.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:10.735+0000 7fa1b936b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:10.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:10.735+0000 7fa1b936b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:10.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:10.735+0000 7fa1b936b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:21:10.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:10.735+0000 7fa1b936b640 1 -- 172.21.15.37:0/2297777174 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1980036a0 msgr2=0x7fa198005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:21:10.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:10.735+0000 7fa1b936b640 1 -- 172.21.15.37:0/2297777174 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1980036a0 msgr2=0x7fa198005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:21:10.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:10.735+0000 7fa1b936b640 1 --2- 172.21.15.37:0/2297777174 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1980036a0 0x7fa198005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:21:10.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:10.735+0000 7fa1b936b640 1 --2- 172.21.15.37:0/2297777174 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1980036a0 0x7fa198005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:21:11.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.115+0000 7fa1abfff640 1 -- 172.21.15.37:0/2297777174 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3189344247 0 0) 0x7fa1b40a5d90 con 0x7fa1980061d0 2024-04-26T00:21:11.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.115+0000 7fa1abfff640 10 monclient(hunting): _negotiate 2024-04-26T00:21:11.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.115+0000 7fa1abfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:11.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.115+0000 7fa1abfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:11.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.115+0000 7fa1abfff640 10 monclient(hunting): my global_id is 4494 2024-04-26T00:21:11.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.115+0000 7fa1abfff640 1 -- 172.21.15.37:0/2297777174 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1a0003530 con 0x7fa1980061d0 2024-04-26T00:21:11.093 INFO:tasks.rados.rados.0.smithi045.stdout:165: done (1 left) 2024-04-26T00:21:11.094 INFO:tasks.rados.rados.0.smithi045.stdout:166: snap_create 2024-04-26T00:21:11.094 INFO:tasks.rados.rados.0.smithi045.stdout:164: finishing rollback tid 0 to smithi04526579-42 2024-04-26T00:21:11.094 INFO:tasks.rados.rados.0.smithi045.stdout:164: finishing rollback tid 1 to smithi04526579-42 2024-04-26T00:21:11.094 INFO:tasks.rados.rados.0.smithi045.stdout:164: finishing rollback tid 2 to smithi04526579-42 2024-04-26T00:21:11.094 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 26 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-26T00:21:11.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1abfff640 1 -- 172.21.15.37:0/2297777174 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1474070232 0 0) 0x7fa1a0003530 con 0x7fa1980061d0 2024-04-26T00:21:11.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1abfff640 1 -- 172.21.15.37:0/2297777174 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1980036a0 msgr2=0x7fa198005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:21:11.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1abfff640 1 --2- 172.21.15.37:0/2297777174 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1980036a0 0x7fa198005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:11.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1abfff640 10 monclient: _finish_hunting 0 2024-04-26T00:21:11.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1abfff640 1 monclient: found mon.noname-a-legacy 2024-04-26T00:21:11.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:11.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1abfff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.37:6789/0 2024-04-26T00:21:11.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1abfff640 1 -- 172.21.15.37:0/2297777174 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1b409b680 con 0x7fa1980061d0 2024-04-26T00:21:11.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1abfff640 10 monclient: _finish_auth 0 2024-04-26T00:21:11.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1abfff640 10 monclient: _check_auth_tickets 2024-04-26T00:21:11.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:11.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1abfff640 1 -- 172.21.15.37:0/2297777174 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 866326386 0 0) 0x7fa1b00044d0 con 0x7fa1980061d0 2024-04-26T00:21:11.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:11.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1abfff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-26T00:21:11.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1abfff640 10 monclient: dump: 2024-04-26T00:21:11.100 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:11.100 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:11.100 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:11.100 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:11.100 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:11.100 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:11.100 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:11.101 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:11.101 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:11.101 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:11.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1abfff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] but i'm connected to v1:172.21.15.37:6789/0, reconnecting 2024-04-26T00:21:11.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1abfff640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:11.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1abfff640 1 -- 172.21.15.37:0/2297777174 >> v1:172.21.15.37:6789/0 conn(0x7fa1980061d0 legacy=0x7fa198008670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:11.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1b9b6c640 1 -- 172.21.15.37:0/2297777174 reap_dead start 2024-04-26T00:21:11.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1abfff640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:21:11.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1abfff640 1 --2- 172.21.15.37:0/2297777174 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1a00039c0 0x7fa1a0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:11.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1abfff640 10 monclient(hunting): picked mon.c con 0x7fa1a00039c0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:21:11.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1abfff640 1 --2- 172.21.15.37:0/2297777174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1a0004810 0x7fa1a0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:11.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1abfff640 10 monclient(hunting): picked mon.b con 0x7fa1a0004810 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:21:11.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1abfff640 1 --2- 172.21.15.37:0/2297777174 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1a0005220 0x7fa1a000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:11.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1abfff640 10 monclient(hunting): picked mon.a con 0x7fa1a0005220 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:21:11.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1abfff640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:11.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1abfff640 1 -- 172.21.15.37:0/2297777174 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b409b8f0 con 0x7fa1a0005220 2024-04-26T00:21:11.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1abfff640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:11.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1abfff640 1 -- 172.21.15.37:0/2297777174 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa1a000c150 con 0x7fa1a0004810 2024-04-26T00:21:11.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1abfff640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:11.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1abfff640 1 -- 172.21.15.37:0/2297777174 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa1a000c350 con 0x7fa1a00039c0 2024-04-26T00:21:11.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1abfff640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:11.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1abfff640 10 monclient(hunting): _finish_auth 0 2024-04-26T00:21:11.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1b8b6a640 1 -- 172.21.15.37:0/2297777174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1a0004810 msgr2=0x7fa1a0004c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:21:11.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1b936b640 1 --2- 172.21.15.37:0/2297777174 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1a00039c0 0x7fa1a0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:11.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1b8b6a640 1 --2- 172.21.15.37:0/2297777174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1a0004810 0x7fa1a0004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:21:11.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1b9b6c640 1 --2- 172.21.15.37:0/2297777174 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1a0005220 0x7fa1a000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:11.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1b936b640 10 monclient(hunting): get_auth_request con 0x7fa1a00039c0 auth_method 0 2024-04-26T00:21:11.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1b936b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:11.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1b936b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:11.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1b936b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:21:11.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1b9b6c640 10 monclient(hunting): get_auth_request con 0x7fa1a0005220 auth_method 0 2024-04-26T00:21:11.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1b9b6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:11.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1b9b6c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:11.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1b9b6c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:21:11.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1b9b6c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:11.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1b9b6c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:11.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1b9b6c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:21:11.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1b936b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:11.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1b936b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:11.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1b936b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:21:11.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1b9b6c640 10 monclient(hunting): handle_auth_done global_id 4494 payload 293 2024-04-26T00:21:11.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1b9b6c640 1 -- 172.21.15.37:0/2297777174 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1a00039c0 msgr2=0x7fa1a0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:21:11.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1b9b6c640 1 --2- 172.21.15.37:0/2297777174 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1a00039c0 0x7fa1a0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:11.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1b9b6c640 1 -- 172.21.15.37:0/2297777174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1a0004810 msgr2=0x7fa1a0004c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:21:11.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1b9b6c640 1 --2- 172.21.15.37:0/2297777174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1a0004810 0x7fa1a0004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:11.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1b9b6c640 10 monclient: _finish_hunting 0 2024-04-26T00:21:11.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1b9b6c640 1 monclient: found mon.a 2024-04-26T00:21:11.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1b9b6c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:11.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1b9b6c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:11.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1b9b6c640 1 -- 172.21.15.37:0/2297777174 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1a0003530 con 0x7fa1a0005220 2024-04-26T00:21:11.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1b9b6c640 10 monclient: _finish_auth 0 2024-04-26T00:21:11.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1b9b6c640 10 monclient: _check_auth_tickets 2024-04-26T00:21:11.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1b9b6c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:11.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1bb5cf640 5 monclient: authenticate success, global_id 4494 2024-04-26T00:21:11.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1bb5cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:21:11.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.119+0000 7fa1b936b640 1 --2- 172.21.15.37:0/2297777174 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1a00039c0 0x7fa1a0003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:21:11.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.127+0000 7fa1b9b6c640 1 -- 172.21.15.37:0/2297777174 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1a0005220 msgr2=0x7fa1a000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:11.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.127+0000 7fa1b9b6c640 1 --2- 172.21.15.37:0/2297777174 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1a0005220 0x7fa1a000b9d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa1b406c680 tx=0x7fa1ac01d0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd60cd3b47039525 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:11.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.127+0000 7fa1abfff640 1 -- 172.21.15.37:0/2297777174 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ac0b3070 con 0x7fa1a0005220 2024-04-26T00:21:11.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.127+0000 7fa1abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:11.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.127+0000 7fa1abfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:21:11.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.127+0000 7fa1abfff640 10 monclient: dump: 2024-04-26T00:21:11.107 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:11.107 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:11.107 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:11.107 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:11.108 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:11.108 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:11.108 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:11.108 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:11.108 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:11.108 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:11.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.127+0000 7fa1bb5cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:21:11.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.127+0000 7fa1abfff640 1 -- 172.21.15.37:0/2297777174 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1ac0ae410 con 0x7fa1a0005220 2024-04-26T00:21:11.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.127+0000 7fa1abfff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:11.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.127+0000 7fa1bb5cf640 10 monclient: get_monmap_and_config success 2024-04-26T00:21:11.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.127+0000 7fa1bb5cf640 4 set_mon_vals no callback set 2024-04-26T00:21:11.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.127+0000 7fa1abfff640 1 -- 172.21.15.37:0/2297777174 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ac0096d0 con 0x7fa1a0005220 2024-04-26T00:21:11.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.127+0000 7fa1abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:11.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.127+0000 7fa1abfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:21:11.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.127+0000 7fa1abfff640 10 monclient: dump: 2024-04-26T00:21:11.109 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:11.109 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:11.109 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:11.109 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:11.109 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:11.109 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:11.109 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:11.109 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:11.109 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:11.109 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:11.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.127+0000 7fa1bb5cf640 10 monclient: shutdown 2024-04-26T00:21:11.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.127+0000 7fa1bb5cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:11.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.127+0000 7fa1bb5cf640 1 -- 172.21.15.37:0/2297777174 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1a0005220 msgr2=0x7fa1a000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:11.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.127+0000 7fa1bb5cf640 1 --2- 172.21.15.37:0/2297777174 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1a0005220 0x7fa1a000b9d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa1b406c680 tx=0x7fa1ac01d0a0 comp rx=0 tx=0).stop 2024-04-26T00:21:11.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.127+0000 7fa1bb5cf640 1 -- 172.21.15.37:0/2297777174 shutdown_connections 2024-04-26T00:21:11.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.127+0000 7fa1bb5cf640 1 --2- 172.21.15.37:0/2297777174 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1a0005220 0x7fa1a000b9d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:11.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.127+0000 7fa1bb5cf640 1 --2- 172.21.15.37:0/2297777174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1a0004810 0x7fa1a0004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:11.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.127+0000 7fa1bb5cf640 1 --2- 172.21.15.37:0/2297777174 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1a00039c0 0x7fa1a0003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:11.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.127+0000 7fa1bb5cf640 1 -- 172.21.15.37:0/2297777174 >> 172.21.15.37:0/2297777174 conn(0x7fa1b40a64e0 msgr2=0x7fa1b40a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:11.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.127+0000 7fa1bb5cf640 1 -- 172.21.15.37:0/2297777174 shutdown_connections 2024-04-26T00:21:11.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.127+0000 7fa1bb5cf640 1 -- 172.21.15.37:0/2297777174 wait complete. 2024-04-26T00:21:11.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.127+0000 7fa1bb5cf640 10 monclient: build_initial_monmap 2024-04-26T00:21:11.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.127+0000 7fa1bb5cf640 10 monclient: monmap: 2024-04-26T00:21:11.110 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:11.110 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:21:11.110 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:21:11.110 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:21:11.110 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:11.111 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:21:11.111 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:21:11.111 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:21:11.111 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:21:11.111 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:11.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1bb5cf640 1 Processor -- start 2024-04-26T00:21:11.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1bb5cf640 1 -- start start 2024-04-26T00:21:11.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1bb5cf640 10 monclient: init 2024-04-26T00:21:11.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1bb5cf640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:11.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1bb5cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:21:11.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1bb5cf640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1a00039c0 0x7fa1b41cb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:11.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1bb5cf640 10 monclient(hunting): picked mon.noname-c con 0x7fa1a00039c0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:21:11.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1bb5cf640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1a0004810 0x7fa1b4134f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:11.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1bb5cf640 10 monclient(hunting): picked mon.noname-a con 0x7fa1a0004810 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:21:11.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1bb5cf640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1a0005220 0x7fa1b41354d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:11.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1bb5cf640 10 monclient(hunting): picked mon.noname-b con 0x7fa1a0005220 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:21:11.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1bb5cf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:11.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1bb5cf640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b409c730 con 0x7fa1a0004810 2024-04-26T00:21:11.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1bb5cf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:11.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1bb5cf640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b409c5b0 con 0x7fa1a0005220 2024-04-26T00:21:11.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1bb5cf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:11.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1bb5cf640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa1b409c8b0 con 0x7fa1a00039c0 2024-04-26T00:21:11.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1bb5cf640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:11.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1bb5cf640 10 monclient(hunting): authenticate will time out at 1052.437468s 2024-04-26T00:21:11.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1b936b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1a00039c0 0x7fa1b41cb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:11.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1b9b6c640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1a0005220 msgr2=0x7fa1b41354d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:21:11.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1b8b6a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1a0004810 0x7fa1b4134f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:11.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1b9b6c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1a0005220 0x7fa1b41354d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:21:11.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1b936b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1a00039c0 0x7fa1b41cb8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:59328/0 (socket says 172.21.15.37:59328) 2024-04-26T00:21:11.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1b936b640 1 -- 172.21.15.37:0/674714493 learned_addr learned my addr 172.21.15.37:0/674714493 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:21:11.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1b936b640 10 monclient(hunting): get_auth_request con 0x7fa1a00039c0 auth_method 0 2024-04-26T00:21:11.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1b936b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:11.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1b936b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:11.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1b936b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:11.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1b8b6a640 10 monclient(hunting): get_auth_request con 0x7fa1a0004810 auth_method 0 2024-04-26T00:21:11.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1b8b6a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:11.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1b8b6a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:11.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1b8b6a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:11.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1b8b6a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:11.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1b8b6a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:11.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1b8b6a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:11.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1b936b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:11.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1b936b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:11.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1b936b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:11.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1b8b6a640 10 monclient(hunting): handle_auth_done global_id 4497 payload 824 2024-04-26T00:21:11.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1b8b6a640 1 -- 172.21.15.37:0/674714493 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1a00039c0 msgr2=0x7fa1b41cb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:11.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1b8b6a640 1 --2- 172.21.15.37:0/674714493 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1a00039c0 0x7fa1b41cb8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:11.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1b8b6a640 1 -- 172.21.15.37:0/674714493 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1a0005220 msgr2=0x7fa1b41354d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:21:11.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1b8b6a640 1 --2- 172.21.15.37:0/674714493 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1a0005220 0x7fa1b41354d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:11.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1b8b6a640 10 monclient: _finish_hunting 0 2024-04-26T00:21:11.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1b8b6a640 1 monclient: found mon.noname-a 2024-04-26T00:21:11.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1b8b6a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:11.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1b8b6a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:21:11.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.131+0000 7fa1b8b6a640 1 -- 172.21.15.37:0/674714493 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1b413b900 con 0x7fa1a0004810 2024-04-26T00:21:11.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.135+0000 7fa1b8b6a640 1 -- 172.21.15.37:0/674714493 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1a0004810 msgr2=0x7fa1b4134f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:11.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.135+0000 7fa1b8b6a640 1 --2- 172.21.15.37:0/674714493 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1a0004810 0x7fa1b4134f90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa1b0000c10 tx=0x7fa1b000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:11.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.135+0000 7fa1a9ffb640 1 -- 172.21.15.37:0/674714493 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b001a070 con 0x7fa1a0004810 2024-04-26T00:21:11.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.135+0000 7fa1a9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:11.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.135+0000 7fa1a9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:21:11.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.135+0000 7fa1a9ffb640 10 monclient: dump: 2024-04-26T00:21:11.118 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:11.118 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:11.118 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:11.118 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:11.118 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:11.118 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:11.118 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:11.118 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:11.118 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:11.118 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:11.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.135+0000 7fa1a9ffb640 10 monclient: _finish_auth 0 2024-04-26T00:21:11.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.135+0000 7fa1a9ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:21:11.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.135+0000 7fa1a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:11.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.135+0000 7fa1a9ffb640 1 -- 172.21.15.37:0/674714493 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1b000de50 con 0x7fa1a0004810 2024-04-26T00:21:11.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.135+0000 7fa1a9ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:11.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.135+0000 7fa1a9ffb640 1 -- 172.21.15.37:0/674714493 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b0012940 con 0x7fa1a0004810 2024-04-26T00:21:11.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.135+0000 7fa1bb5cf640 5 monclient: authenticate success, global_id 4497 2024-04-26T00:21:11.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.135+0000 7fa1bb5cf640 10 monclient: _renew_subs 2024-04-26T00:21:11.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.135+0000 7fa1bb5cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:11.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.135+0000 7fa1ab7fe640 4 set_mon_vals no callback set 2024-04-26T00:21:11.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.135+0000 7fa1bb5cf640 1 -- 172.21.15.37:0/674714493 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1b4121940 con 0x7fa1a0004810 2024-04-26T00:21:11.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.135+0000 7fa1a9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:11.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.135+0000 7fa1a9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:21:11.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.135+0000 7fa1a9ffb640 10 monclient: dump: 2024-04-26T00:21:11.120 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:11.120 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:11.120 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:11.120 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:11.120 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:11.120 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:11.120 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:11.120 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:11.120 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:11.120 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:11.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.135+0000 7fa1bb5cf640 10 monclient: _renew_subs 2024-04-26T00:21:11.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.135+0000 7fa1bb5cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:11.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.135+0000 7fa1bb5cf640 1 -- 172.21.15.37:0/674714493 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1b4121380 con 0x7fa1a0004810 2024-04-26T00:21:11.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.135+0000 7fa1bb5cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:11.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.135+0000 7fa1bb5cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:11.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.135+0000 7fa1bb5cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:11.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.135+0000 7fa1bb5cf640 1 -- 172.21.15.37:0/674714493 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1b4075910 con 0x7fa1a0004810 2024-04-26T00:21:11.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.135+0000 7fa1a9ffb640 1 -- 172.21.15.37:0/674714493 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa1b001d070 con 0x7fa1a0004810 2024-04-26T00:21:11.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.135+0000 7fa1a9ffb640 1 --2- 172.21.15.37:0/674714493 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa1980364a0 0x7fa198038960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:11.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.135+0000 7fa1a9ffb640 1 -- 172.21.15.37:0/674714493 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 3943+0+0 (secure 0 0 0) 0x7fa1b004b170 con 0x7fa1a0004810 2024-04-26T00:21:11.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.135+0000 7fa1b936b640 1 --2- 172.21.15.37:0/674714493 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa1980364a0 0x7fa198038960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:11.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.135+0000 7fa1b936b640 10 monclient: get_auth_request con 0x7fa1980364a0 auth_method 0 2024-04-26T00:21:11.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.139+0000 7fa1b936b640 1 --2- 172.21.15.37:0/674714493 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa1980364a0 0x7fa198038960 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa1a4003a40 tx=0x7fa1a4003cd0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:11.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.139+0000 7fa1a9ffb640 1 -- 172.21.15.37:0/674714493 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa1b0024020 con 0x7fa1a0004810 2024-04-26T00:21:11.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.139+0000 7fa1a9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:11.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.139+0000 7fa1a9ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:21:11.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.379+0000 7fa1bb5cf640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:21:11.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.379+0000 7fa1bb5cf640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:21:11.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.379+0000 7fa1bb5cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:11.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.379+0000 7fa1bb5cf640 1 -- 172.21.15.37:0/674714493 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa1b4075ee0 con 0x7fa1a0004810 2024-04-26T00:21:11.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.379+0000 7fa1a9ffb640 1 -- 172.21.15.37:0/674714493 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7fa1b0012d40 con 0x7fa1a0004810 2024-04-26T00:21:11.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.379+0000 7fa1a9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:21:11.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.379+0000 7fa1a9ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:21:11.358 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":16,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:21:11.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.383+0000 7fa1bb5cf640 1 -- 172.21.15.37:0/674714493 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa1980364a0 msgr2=0x7fa198038960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:11.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.383+0000 7fa1bb5cf640 1 --2- 172.21.15.37:0/674714493 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa1980364a0 0x7fa198038960 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa1a4003a40 tx=0x7fa1a4003cd0 comp rx=0 tx=0).stop 2024-04-26T00:21:11.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.383+0000 7fa1bb5cf640 10 monclient: shutdown 2024-04-26T00:21:11.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.383+0000 7fa1bb5cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:11.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.383+0000 7fa1bb5cf640 1 -- 172.21.15.37:0/674714493 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1a0004810 msgr2=0x7fa1b4134f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:11.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.383+0000 7fa1bb5cf640 1 --2- 172.21.15.37:0/674714493 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1a0004810 0x7fa1b4134f90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa1b0000c10 tx=0x7fa1b000d4a0 comp rx=0 tx=0).stop 2024-04-26T00:21:11.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.383+0000 7fa1bb5cf640 1 -- 172.21.15.37:0/674714493 shutdown_connections 2024-04-26T00:21:11.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.383+0000 7fa1bb5cf640 1 --2- 172.21.15.37:0/674714493 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa1980364a0 0x7fa198038960 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:11.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.383+0000 7fa1bb5cf640 1 --2- 172.21.15.37:0/674714493 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1a0005220 0x7fa1b41354d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:11.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.383+0000 7fa1bb5cf640 1 --2- 172.21.15.37:0/674714493 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1a0004810 0x7fa1b4134f90 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:11.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.383+0000 7fa1bb5cf640 1 --2- 172.21.15.37:0/674714493 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1a00039c0 0x7fa1b41cb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:11.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.383+0000 7fa1bb5cf640 1 -- 172.21.15.37:0/674714493 >> 172.21.15.37:0/674714493 conn(0x7fa1b40a64e0 msgr2=0x7fa1b4135fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:11.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.383+0000 7fa1bb5cf640 1 -- 172.21.15.37:0/674714493 shutdown_connections 2024-04-26T00:21:11.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.383+0000 7fa1bb5cf640 1 -- 172.21.15.37:0/674714493 wait complete. 2024-04-26T00:21:11.377 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-26T00:21:11.377 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-04-26T00:21:11.378 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:21:11.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.515+0000 7f8496322640 10 monclient: get_monmap_and_config 2024-04-26T00:21:11.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.515+0000 7f8496322640 10 monclient: build_initial_monmap 2024-04-26T00:21:11.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.515+0000 7f8496322640 10 monclient: monmap: 2024-04-26T00:21:11.495 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:11.495 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:11.495 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:21:11.519622+0000 2024-04-26T00:21:11.496 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:21:11.519622+0000 2024-04-26T00:21:11.496 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:11.496 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:11.496 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:21:11.496 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:21:11.496 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:21:11.496 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:21:11.496 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:21:11.496 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:11.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.519+0000 7f8496322640 1 Processor -- start 2024-04-26T00:21:11.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.519+0000 7f8496322640 1 -- start start 2024-04-26T00:21:11.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.519+0000 7f8496322640 10 monclient: init 2024-04-26T00:21:11.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.519+0000 7f8496322640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:11.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.519+0000 7f8496322640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-04-26T00:21:11.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.519+0000 7f8496322640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8490136f20 0x7f849009bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:11.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.519+0000 7f8496322640 10 monclient(hunting): picked mon.noname-c con 0x7f8490136f20 addr v2:172.21.15.45:3300/0 2024-04-26T00:21:11.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.519+0000 7f8496322640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8490137300 addr v1:172.21.15.37:6789/0 2024-04-26T00:21:11.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.519+0000 7f8496322640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f849013c450 0x7f849013e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:11.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.519+0000 7f8496322640 10 monclient(hunting): picked mon.noname-b con 0x7f849013c450 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:21:11.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.519+0000 7f8496322640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84900a50f0 con 0x7f8490137300 2024-04-26T00:21:11.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.519+0000 7f8496322640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:11.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.519+0000 7f8496322640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f849009b330 con 0x7f8490136f20 2024-04-26T00:21:11.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.519+0000 7f8496322640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:11.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.519+0000 7f8496322640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f849009b610 con 0x7f849013c450 2024-04-26T00:21:11.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.519+0000 7f8496322640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:11.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.519+0000 7f8496322640 10 monclient(hunting): authenticate will time out at 1052.825382s 2024-04-26T00:21:11.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.519+0000 7f848ffff640 1 -- >> v2:172.21.15.45:3300/0 conn(0x7f8490136f20 msgr2=0x7f849009bff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:21:11.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.519+0000 7f848ffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8490136f20 0x7f849009bff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:21:11.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.519+0000 7f84948bf640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f849013c450 0x7f849013e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:11.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.519+0000 7f848f7fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f8490137300 0x7f84901377b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:49262/0 (socket says 172.21.15.37:49262) 2024-04-26T00:21:11.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.519+0000 7f84948bf640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f849013c450 0x7f849013e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:59338/0 (socket says 172.21.15.37:59338) 2024-04-26T00:21:11.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.519+0000 7f848f7fe640 1 -- 172.21.15.37:0/2180149970 learned_addr learned my addr 172.21.15.37:0/2180149970 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:21:11.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.519+0000 7f84948bf640 10 monclient(hunting): get_auth_request con 0x7f849013c450 auth_method 0 2024-04-26T00:21:11.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.519+0000 7f84948bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:11.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.519+0000 7f84948bf640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:11.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.519+0000 7f84948bf640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:11.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.519+0000 7f84948bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:11.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.519+0000 7f84948bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:11.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.519+0000 7f84948bf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:11.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.519+0000 7f84948bf640 10 monclient(hunting): handle_auth_done global_id 4487 payload 274 2024-04-26T00:21:11.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.519+0000 7f84948bf640 1 -- 172.21.15.37:0/2180149970 >> v2:172.21.15.45:3300/0 conn(0x7f8490136f20 msgr2=0x7f849009bff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:21:11.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.519+0000 7f84948bf640 1 --2- 172.21.15.37:0/2180149970 >> v2:172.21.15.45:3300/0 conn(0x7f8490136f20 0x7f849009bff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:11.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.519+0000 7f84948bf640 1 -- 172.21.15.37:0/2180149970 >> v1:172.21.15.37:6789/0 conn(0x7f8490137300 legacy=0x7f84901377b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:21:11.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.519+0000 7f84948bf640 10 monclient: _finish_hunting 0 2024-04-26T00:21:11.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.519+0000 7f84948bf640 1 monclient: found mon.noname-b 2024-04-26T00:21:11.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.519+0000 7f84948bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:11.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.519+0000 7f84948bf640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:21:11.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.519+0000 7f84948bf640 1 -- 172.21.15.37:0/2180149970 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f849009b8f0 con 0x7f849013c450 2024-04-26T00:21:11.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.523+0000 7f84948bf640 1 -- 172.21.15.37:0/2180149970 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f849013c450 msgr2=0x7f849013e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:11.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.523+0000 7f84948bf640 1 --2- 172.21.15.37:0/2180149970 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f849013c450 0x7f849013e840 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8480009d90 tx=0x7f848001c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a52a2282cdd977d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:11.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.523+0000 7f848effd640 1 -- 172.21.15.37:0/2180149970 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84800b2070 con 0x7f849013c450 2024-04-26T00:21:11.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.523+0000 7f848effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:11.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.523+0000 7f848effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:21:11.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.523+0000 7f848effd640 10 monclient: dump: 2024-04-26T00:21:11.505 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:11.505 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:11.505 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:11.505 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:11.506 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:11.506 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:11.506 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:11.506 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:11.506 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:11.506 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:11.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.523+0000 7f848effd640 10 monclient: _finish_auth 0 2024-04-26T00:21:11.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.523+0000 7f848effd640 10 monclient: _check_auth_tickets 2024-04-26T00:21:11.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.523+0000 7f848effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:11.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.523+0000 7f848effd640 1 -- 172.21.15.37:0/2180149970 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84800a5800 con 0x7f849013c450 2024-04-26T00:21:11.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.523+0000 7f848effd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:11.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.523+0000 7f848effd640 1 -- 172.21.15.37:0/2180149970 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84800a5bd0 con 0x7f849013c450 2024-04-26T00:21:11.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.523+0000 7f8496322640 5 monclient: authenticate success, global_id 4487 2024-04-26T00:21:11.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.523+0000 7f8496322640 10 monclient: get_monmap_and_config success 2024-04-26T00:21:11.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.523+0000 7f8496322640 4 set_mon_vals no callback set 2024-04-26T00:21:11.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.523+0000 7f848effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:11.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.523+0000 7f848effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:21:11.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.523+0000 7f848effd640 10 monclient: dump: 2024-04-26T00:21:11.507 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:11.507 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:11.507 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:11.507 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:11.507 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:11.508 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:11.508 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:11.508 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:11.508 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:11.508 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:11.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.523+0000 7f8496322640 10 monclient: shutdown 2024-04-26T00:21:11.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.523+0000 7f8496322640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:11.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.523+0000 7f8496322640 1 -- 172.21.15.37:0/2180149970 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f849013c450 msgr2=0x7f849013e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:11.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.523+0000 7f8496322640 1 --2- 172.21.15.37:0/2180149970 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f849013c450 0x7f849013e840 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8480009d90 tx=0x7f848001c8a0 comp rx=0 tx=0).stop 2024-04-26T00:21:11.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.523+0000 7f8496322640 1 -- 172.21.15.37:0/2180149970 shutdown_connections 2024-04-26T00:21:11.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.523+0000 7f8496322640 1 --2- 172.21.15.37:0/2180149970 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f849013c450 0x7f849013e840 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:11.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.523+0000 7f8496322640 1 --2- 172.21.15.37:0/2180149970 >> v2:172.21.15.45:3300/0 conn(0x7f8490136f20 0x7f849009bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:11.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.523+0000 7f8496322640 1 -- 172.21.15.37:0/2180149970 >> 172.21.15.37:0/2180149970 conn(0x7f84900a61b0 msgr2=0x7f84900a65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:11.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.523+0000 7f8496322640 1 -- 172.21.15.37:0/2180149970 shutdown_connections 2024-04-26T00:21:11.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.523+0000 7f8496322640 1 -- 172.21.15.37:0/2180149970 wait complete. 2024-04-26T00:21:11.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.523+0000 7f8496322640 10 monclient: build_initial_monmap 2024-04-26T00:21:11.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.523+0000 7f8496322640 10 monclient: monmap: 2024-04-26T00:21:11.509 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:11.509 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:21:11.509 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:21:11.509 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:21:11.509 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:11.509 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:21:11.509 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:21:11.509 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:21:11.509 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:21:11.509 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:11.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f8496322640 1 Processor -- start 2024-04-26T00:21:11.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f8496322640 1 -- start start 2024-04-26T00:21:11.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f8496322640 10 monclient: init 2024-04-26T00:21:11.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f8496322640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:11.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f8496322640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:21:11.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f8496322640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8490136f20 0x7f84901cedf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:11.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f8496322640 10 monclient(hunting): picked mon.noname-b con 0x7f8490136f20 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:21:11.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f8496322640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8490137300 0x7f8490134ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:11.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f8496322640 10 monclient(hunting): picked mon.noname-c con 0x7f8490137300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:21:11.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f8496322640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f849013c450 0x7f8490137ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:11.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f8496322640 10 monclient(hunting): picked mon.noname-a con 0x7f849013c450 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:21:11.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f8496322640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:11.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f8496322640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f8490141930 con 0x7f849013c450 2024-04-26T00:21:11.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f8496322640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:11.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f8496322640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f84901417b0 con 0x7f8490136f20 2024-04-26T00:21:11.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f8496322640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:11.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f8496322640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8490141ab0 con 0x7f8490137300 2024-04-26T00:21:11.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f8496322640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:11.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f8496322640 10 monclient(hunting): authenticate will time out at 1052.832823s 2024-04-26T00:21:11.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f848f7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8490137300 0x7f8490134ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:11.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f84948bf640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f849013c450 0x7f8490137ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:11.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f84948bf640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f849013c450 0x7f8490137ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:34912/0 (socket says 172.21.15.37:34912) 2024-04-26T00:21:11.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f84948bf640 1 -- 172.21.15.37:0/533705659 learned_addr learned my addr 172.21.15.37:0/533705659 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:21:11.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f84948bf640 10 monclient(hunting): get_auth_request con 0x7f849013c450 auth_method 0 2024-04-26T00:21:11.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f848ffff640 1 -- 172.21.15.37:0/533705659 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8490136f20 msgr2=0x7f84901cedf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:21:11.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f84948bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:11.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f84948bf640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:11.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f84948bf640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:11.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f848f7fe640 10 monclient(hunting): get_auth_request con 0x7f8490137300 auth_method 0 2024-04-26T00:21:11.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f848f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:11.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f848f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:11.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f848f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:11.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f848ffff640 1 --2- 172.21.15.37:0/533705659 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8490136f20 0x7f84901cedf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:21:11.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f84948bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:11.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f84948bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:11.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f84948bf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:11.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f848f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:11.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f848f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:11.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f848f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:11.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f84948bf640 10 monclient(hunting): handle_auth_done global_id 4500 payload 824 2024-04-26T00:21:11.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f84948bf640 1 -- 172.21.15.37:0/533705659 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8490137300 msgr2=0x7f8490134ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:11.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f84948bf640 1 --2- 172.21.15.37:0/533705659 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8490137300 0x7f8490134ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:11.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f84948bf640 1 -- 172.21.15.37:0/533705659 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8490136f20 msgr2=0x7f84901cedf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:21:11.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f84948bf640 1 --2- 172.21.15.37:0/533705659 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8490136f20 0x7f84901cedf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:11.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f84948bf640 10 monclient: _finish_hunting 0 2024-04-26T00:21:11.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f84948bf640 1 monclient: found mon.noname-a 2024-04-26T00:21:11.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f84948bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:11.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f84948bf640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:21:11.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f84948bf640 1 -- 172.21.15.37:0/533705659 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f849009ae30 con 0x7f849013c450 2024-04-26T00:21:11.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f848f7fe640 1 --2- 172.21.15.37:0/533705659 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8490137300 0x7f8490134ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:21:11.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f84948bf640 1 -- 172.21.15.37:0/533705659 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f849013c450 msgr2=0x7f8490137ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:11.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f84948bf640 1 --2- 172.21.15.37:0/533705659 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f849013c450 0x7f8490137ef0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8480009d90 tx=0x7f8480019640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:11.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f848cff9640 1 -- 172.21.15.37:0/533705659 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84800bd020 con 0x7f849013c450 2024-04-26T00:21:11.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f848cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:11.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f848cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:21:11.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f848cff9640 10 monclient: dump: 2024-04-26T00:21:11.517 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:11.517 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:11.517 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:11.517 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:11.518 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:11.518 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:11.518 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:11.518 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:11.518 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:11.518 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:11.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f848cff9640 10 monclient: _finish_auth 0 2024-04-26T00:21:11.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f848cff9640 10 monclient: _check_auth_tickets 2024-04-26T00:21:11.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f848cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:11.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f848cff9640 1 -- 172.21.15.37:0/533705659 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84800b8880 con 0x7f849013c450 2024-04-26T00:21:11.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f8496322640 5 monclient: authenticate success, global_id 4500 2024-04-26T00:21:11.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f8496322640 10 monclient: _renew_subs 2024-04-26T00:21:11.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f8496322640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:11.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f8496322640 1 -- 172.21.15.37:0/533705659 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8490139930 con 0x7f849013c450 2024-04-26T00:21:11.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f848cff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:11.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f848cff9640 1 -- 172.21.15.37:0/533705659 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84800b2040 con 0x7f849013c450 2024-04-26T00:21:11.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f848cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:11.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f848cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:21:11.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f848cff9640 10 monclient: dump: 2024-04-26T00:21:11.519 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:11.519 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:11.519 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:11.519 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:11.519 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:11.519 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:11.519 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:11.519 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:11.520 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:11.520 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:11.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f848e7fc640 4 set_mon_vals no callback set 2024-04-26T00:21:11.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f8496322640 10 monclient: _renew_subs 2024-04-26T00:21:11.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f8496322640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:11.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.527+0000 7f8496322640 1 -- 172.21.15.37:0/533705659 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f849013a1d0 con 0x7f849013c450 2024-04-26T00:21:11.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.531+0000 7f8496322640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:11.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.531+0000 7f8496322640 1 --2- 172.21.15.37:0/533705659 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f849002c000 0x7f849002c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:11.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.531+0000 7f8496322640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:11.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.531+0000 7f8496322640 1 -- 172.21.15.37:0/533705659 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f849013ba60 con 0x7f849002c000 2024-04-26T00:21:11.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.531+0000 7f848ffff640 1 --2- 172.21.15.37:0/533705659 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f849002c000 0x7f849002c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:11.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.531+0000 7f848ffff640 10 monclient: get_auth_request con 0x7f849002c000 auth_method 0 2024-04-26T00:21:11.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.531+0000 7f848ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:11.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.531+0000 7f848ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:11.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.531+0000 7f848ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:11.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.531+0000 7f848ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:11.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.531+0000 7f848ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:11.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.531+0000 7f848ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:11.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.531+0000 7f848ffff640 10 monclient(hunting): handle_auth_done global_id 4503 payload 824 2024-04-26T00:21:11.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.531+0000 7f848cff9640 1 -- 172.21.15.37:0/533705659 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f84800c0020 con 0x7f849013c450 2024-04-26T00:21:11.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.531+0000 7f848ffff640 1 -- 172.21.15.37:0/533705659 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f849004f510 con 0x7f849002c000 2024-04-26T00:21:11.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.531+0000 7f848cff9640 1 --2- 172.21.15.37:0/533705659 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8470035e80 0x7f8470038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:11.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.531+0000 7f848cff9640 1 -- 172.21.15.37:0/533705659 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 3943+0+0 (secure 0 0 0) 0x7f84800e3990 con 0x7f849013c450 2024-04-26T00:21:11.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.531+0000 7f848f7fe640 1 --2- 172.21.15.37:0/533705659 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8470035e80 0x7f8470038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:11.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.531+0000 7f848f7fe640 10 monclient: get_auth_request con 0x7f8470035e80 auth_method 0 2024-04-26T00:21:11.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.531+0000 7f848ffff640 1 -- 172.21.15.37:0/533705659 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f849002c000 msgr2=0x7f849002c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:11.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.531+0000 7f848ffff640 1 --2- 172.21.15.37:0/533705659 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f849002c000 0x7f849002c3e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f848800b450 tx=0x7f848800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:11.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.531+0000 7f848cff9640 1 -- 172.21.15.37:0/533705659 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8488005270 con 0x7f849002c000 2024-04-26T00:21:11.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.531+0000 7f848cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:11.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.531+0000 7f848cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:21:11.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.531+0000 7f848cff9640 10 monclient: dump: 2024-04-26T00:21:11.522 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:11.522 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:11.523 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:11.523 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:11.523 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:11.523 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:11.523 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:11.523 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:11.523 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:11.523 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:11.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.531+0000 7f848cff9640 1 -- 172.21.15.37:0/533705659 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8488015070 con 0x7f849002c000 2024-04-26T00:21:11.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.531+0000 7f848cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:11.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.531+0000 7f848cff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:21:11.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.531+0000 7f848cff9640 1 -- 172.21.15.37:0/533705659 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f849002c000 msgr2=0x7f849002c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:11.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.531+0000 7f848cff9640 1 --2- 172.21.15.37:0/533705659 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f849002c000 0x7f849002c3e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f848800b450 tx=0x7f848800b910 comp rx=0 tx=0).stop 2024-04-26T00:21:11.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.531+0000 7f848cff9640 1 -- 172.21.15.37:0/533705659 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f849002c000 msgr2=0x7f849002c3e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:21:11.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.531+0000 7f848cff9640 1 --2- 172.21.15.37:0/533705659 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f849002c000 0x7f849002c3e0 secure :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f848800b450 tx=0x7f848800b910 comp rx=0 tx=0).stop 2024-04-26T00:21:11.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.535+0000 7f848f7fe640 1 --2- 172.21.15.37:0/533705659 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8470035e80 0x7f8470038340 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8478007ca0 tx=0x7f8478007bf0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:11.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.539+0000 7f8496322640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:21:11.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.539+0000 7f8496322640 1 --2- 172.21.15.37:0/533705659 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f849002c920 0x7f849002cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:11.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.539+0000 7f8496322640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:11.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.539+0000 7f8496322640 1 -- 172.21.15.37:0/533705659 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f849013b3d0 con 0x7f849002c920 2024-04-26T00:21:11.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.539+0000 7f848ffff640 1 --2- 172.21.15.37:0/533705659 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f849002c920 0x7f849002cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:11.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.539+0000 7f848ffff640 10 monclient: get_auth_request con 0x7f849002c920 auth_method 0 2024-04-26T00:21:11.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.539+0000 7f848ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:11.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.539+0000 7f848ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:11.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.539+0000 7f848ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:11.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.539+0000 7f848ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:11.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.539+0000 7f848ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:11.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.539+0000 7f848ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:11.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.539+0000 7f848ffff640 10 monclient(hunting): handle_auth_done global_id 4506 payload 824 2024-04-26T00:21:11.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.539+0000 7f848ffff640 1 -- 172.21.15.37:0/533705659 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f849013b820 con 0x7f849002c920 2024-04-26T00:21:11.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.543+0000 7f848ffff640 1 -- 172.21.15.37:0/533705659 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f849002c920 msgr2=0x7f849002cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:11.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.543+0000 7f848ffff640 1 --2- 172.21.15.37:0/533705659 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f849002c920 0x7f849002cd30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f848800b450 tx=0x7f8488005a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:11.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.543+0000 7f848cff9640 1 -- 172.21.15.37:0/533705659 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f848801ae60 con 0x7f849002c920 2024-04-26T00:21:11.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.543+0000 7f848cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:11.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.543+0000 7f848cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:21:11.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.543+0000 7f848cff9640 10 monclient: dump: 2024-04-26T00:21:11.527 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:11.527 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:11.527 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:11.528 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:11.528 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:11.528 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:11.528 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:11.528 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:11.528 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:11.528 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:11.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.543+0000 7f848cff9640 1 -- 172.21.15.37:0/533705659 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4877 (secure 0 0 0) 0x7f8488010480 con 0x7f849002c920 2024-04-26T00:21:11.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.543+0000 7f848cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:21:11.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.543+0000 7f848cff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:21:11.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.543+0000 7f848cff9640 1 -- 172.21.15.37:0/533705659 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f849002c920 msgr2=0x7f849002cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:11.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.543+0000 7f848cff9640 1 --2- 172.21.15.37:0/533705659 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f849002c920 0x7f849002cd30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f848800b450 tx=0x7f8488005a30 comp rx=0 tx=0).stop 2024-04-26T00:21:11.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.543+0000 7f848cff9640 1 -- 172.21.15.37:0/533705659 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f849002c920 msgr2=0x7f849002cd30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:21:11.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.543+0000 7f848cff9640 1 --2- 172.21.15.37:0/533705659 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f849002c920 0x7f849002cd30 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:11.529 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:21:11.529 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:21:11.529 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:21:11.529 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:21:11.529 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 16, 2024-04-26T00:21:11.529 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 88647, 2024-04-26T00:21:11.530 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:21:11.530 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:21:11.530 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:21:11.530 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:11.530 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:21:11.530 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:21:11.530 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:21:11.530 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:21:11.530 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:21:11.530 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:21:11.530 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:21:11.530 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:21:11.530 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:21:11.530 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:21:11.531 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:21:11.531 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:21:11.531 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:21:11.531 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:21:11.531 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:21:11.531 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:11.531 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:21:11.531 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:21:11.531 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:21:11.531 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:21:11.531 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:21:11.531 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:21:11.531 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:21:11.531 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:21:11.532 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:21:11.532 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:21:11.532 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:21:11.532 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:21:11.532 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:21:11.532 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:11.532 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:11.532 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:21:11.532 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:21:11.532 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:21:11.533 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:21:11.533 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:21:11.533 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:21:11.533 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:21:11.533 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:21:11.533 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:21:11.533 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:21:11.533 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:21:11.533 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:21:11.533 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:21:11.533 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:21:11.533 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:21:11.533 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:21:11.533 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:21:11.534 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:21:11.534 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:21:11.534 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:21:11.534 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:21:11.534 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:21:11.534 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:21:11.534 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:21:11.534 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:21:11.534 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:21:11.534 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:21:11.534 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:21:11.534 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:11.534 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:21:11.534 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:11.535 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:21:11.535 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:11.535 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:21:11.535 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:21:11.535 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:21:11.535 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:21:11.535 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:11.535 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:21:11.535 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:21:11.535 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:11.535 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:11.535 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:11.535 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:21:11.535 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:21:11.536 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:11.536 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:11.536 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:11.536 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:11.536 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:21:11.536 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:21:11.536 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:21:11.536 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:21:11.536 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:21:11.536 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:11.536 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:11.536 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:21:11.536 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:21:11.536 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:21:11.537 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:21:11.537 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:11.537 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:21:11.537 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:21:11.537 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:11.537 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:11.537 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:11.537 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:21:11.537 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:21:11.537 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:11.537 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:11.538 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:11.538 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:11.538 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:21:11.538 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:21:11.538 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:21:11.538 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:21:11.538 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:21:11.538 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:11.539 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:11.539 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:21:11.539 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:21:11.539 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:21:11.539 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:21:11.539 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:11.539 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:21:11.539 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:21:11.539 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:11.539 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:11.539 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:11.539 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:21:11.539 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:21:11.539 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:11.539 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:11.540 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:11.540 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:11.540 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:21:11.540 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:21:11.540 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:21:11.540 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:21:11.540 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:21:11.540 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:11.540 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:11.540 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:11.540 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:21:11.540 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:21:11.540 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:11.540 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:21:11.541 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:21:11.541 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:21:11.541 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:11.541 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:11.541 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:21:11.541 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:11.541 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:21:11.541 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:21:11.541 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:21:11.541 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:11.541 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:11.541 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:21:11.541 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:11.541 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:21:11.542 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:21:11.542 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:21:11.542 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:11.542 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:11.542 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:21:11.542 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:11.542 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:21:11.542 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:21:11.542 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:21:11.542 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:11.542 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:11.542 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:11.542 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:21:11.543 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:21:11.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.543+0000 7f8496322640 1 -- 172.21.15.37:0/533705659 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8470035e80 msgr2=0x7f8470038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:11.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.543+0000 7f8496322640 1 --2- 172.21.15.37:0/533705659 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8470035e80 0x7f8470038340 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8478007ca0 tx=0x7f8478007bf0 comp rx=0 tx=0).stop 2024-04-26T00:21:11.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.543+0000 7f8496322640 10 monclient: shutdown 2024-04-26T00:21:11.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.543+0000 7f8496322640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:11.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.543+0000 7f8496322640 1 -- 172.21.15.37:0/533705659 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f849013c450 msgr2=0x7f8490137ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:11.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.543+0000 7f84948bf640 1 -- 172.21.15.37:0/533705659 reap_dead start 2024-04-26T00:21:11.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.543+0000 7f8496322640 1 --2- 172.21.15.37:0/533705659 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f849013c450 0x7f8490137ef0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8480009d90 tx=0x7f8480019640 comp rx=0 tx=0).stop 2024-04-26T00:21:11.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.543+0000 7f8496322640 1 -- 172.21.15.37:0/533705659 shutdown_connections 2024-04-26T00:21:11.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.543+0000 7f8496322640 1 --2- 172.21.15.37:0/533705659 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f849013c450 0x7f8490137ef0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:11.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.543+0000 7f8496322640 1 -- 172.21.15.37:0/533705659 >> 172.21.15.37:0/533705659 conn(0x7f84900a61b0 msgr2=0x7f84900a7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:11.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.543+0000 7f8496322640 1 -- 172.21.15.37:0/533705659 shutdown_connections 2024-04-26T00:21:11.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.543+0000 7f8496322640 1 -- 172.21.15.37:0/533705659 wait complete. 2024-04-26T00:21:11.545 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:21:11.545 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-04-26T00:21:11.546 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:21:11.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.687+0000 7ff9f8784640 10 monclient: get_monmap_and_config 2024-04-26T00:21:11.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.687+0000 7ff9f8784640 10 monclient: build_initial_monmap 2024-04-26T00:21:11.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.687+0000 7ff9f8784640 10 monclient: monmap: 2024-04-26T00:21:11.666 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:11.666 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:11.666 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:21:11.690171+0000 2024-04-26T00:21:11.666 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:21:11.690171+0000 2024-04-26T00:21:11.666 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:11.666 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:11.666 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:21:11.667 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:21:11.667 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:21:11.667 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:21:11.667 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:21:11.667 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:11.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.687+0000 7ff9f8784640 1 Processor -- start 2024-04-26T00:21:11.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.687+0000 7ff9f8784640 1 -- start start 2024-04-26T00:21:11.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.687+0000 7ff9f8784640 10 monclient: init 2024-04-26T00:21:11.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.687+0000 7ff9f8784640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:11.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.687+0000 7ff9f8784640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-04-26T00:21:11.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.687+0000 7ff9f8784640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff9f0133f70 0x7ff9f009c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:11.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.687+0000 7ff9f8784640 10 monclient(hunting): picked mon.noname-a con 0x7ff9f0133f70 addr v2:172.21.15.37:3300/0 2024-04-26T00:21:11.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.687+0000 7ff9f8784640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff9f0134350 addr v1:172.21.15.37:6789/0 2024-04-26T00:21:11.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.687+0000 7ff9f8784640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9f0134790 0x7ff9f01374b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:11.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.687+0000 7ff9f8784640 10 monclient(hunting): picked mon.noname-b con 0x7ff9f0134790 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:21:11.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.687+0000 7ff9f8784640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9f00a4fa0 con 0x7ff9f0134350 2024-04-26T00:21:11.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.687+0000 7ff9f8784640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:11.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.687+0000 7ff9f8784640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7ff9f00a51e0 con 0x7ff9f0133f70 2024-04-26T00:21:11.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.687+0000 7ff9f8784640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:11.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.687+0000 7ff9f8784640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff9f009b280 con 0x7ff9f0134790 2024-04-26T00:21:11.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.687+0000 7ff9f8784640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:11.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.687+0000 7ff9f8784640 10 monclient(hunting): authenticate will time out at 1052.995830s 2024-04-26T00:21:11.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.687+0000 7ff9f6520640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff9f0133f70 0x7ff9f009c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:11.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.691+0000 7ff9f6520640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff9f0133f70 0x7ff9f009c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:34932/0 (socket says 172.21.15.37:34932) 2024-04-26T00:21:11.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.691+0000 7ff9f5d1f640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7ff9f0134350 0x7ff9f0136da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:49270/0 (socket says 172.21.15.37:49270) 2024-04-26T00:21:11.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.691+0000 7ff9f6520640 1 -- 172.21.15.37:0/311268178 learned_addr learned my addr 172.21.15.37:0/311268178 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:21:11.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.691+0000 7ff9f6520640 10 monclient(hunting): get_auth_request con 0x7ff9f0133f70 auth_method 0 2024-04-26T00:21:11.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.691+0000 7ff9f6520640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:11.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.691+0000 7ff9f6520640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:11.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.691+0000 7ff9f6520640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:11.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.691+0000 7ff9f6d21640 1 --2- 172.21.15.37:0/311268178 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9f0134790 0x7ff9f01374b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:11.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.691+0000 7ff9f6d21640 10 monclient(hunting): get_auth_request con 0x7ff9f0134790 auth_method 0 2024-04-26T00:21:11.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.691+0000 7ff9f6d21640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:11.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.691+0000 7ff9f6d21640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:11.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.691+0000 7ff9f6d21640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:11.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.691+0000 7ff9f6520640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:11.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.691+0000 7ff9f6520640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:11.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.691+0000 7ff9f6520640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:11.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.691+0000 7ff9f6d21640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:11.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.691+0000 7ff9f6d21640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:11.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.691+0000 7ff9f6d21640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:11.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.691+0000 7ff9f6520640 10 monclient(hunting): handle_auth_done global_id 4509 payload 274 2024-04-26T00:21:11.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.691+0000 7ff9f6520640 1 -- 172.21.15.37:0/311268178 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9f0134790 msgr2=0x7ff9f01374b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:21:11.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.691+0000 7ff9f6520640 1 --2- 172.21.15.37:0/311268178 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9f0134790 0x7ff9f01374b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:11.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.691+0000 7ff9f6520640 1 -- 172.21.15.37:0/311268178 >> v1:172.21.15.37:6789/0 conn(0x7ff9f0134350 legacy=0x7ff9f0136da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:21:11.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.691+0000 7ff9f6520640 10 monclient: _finish_hunting 0 2024-04-26T00:21:11.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.691+0000 7ff9f6520640 1 monclient: found mon.noname-a 2024-04-26T00:21:11.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.691+0000 7ff9f6520640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:11.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.691+0000 7ff9f6520640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:21:11.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.691+0000 7ff9f6520640 1 -- 172.21.15.37:0/311268178 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9f009c680 con 0x7ff9f0133f70 2024-04-26T00:21:11.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f6520640 1 -- 172.21.15.37:0/311268178 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9f0133f70 msgr2=0x7ff9f009c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:11.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f6520640 1 --2- 172.21.15.37:0/311268178 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9f0133f70 0x7ff9f009c140 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec009870 tx=0x7ff9ec01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d8dac48ca1dea8a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:11.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f551e640 1 -- 172.21.15.37:0/311268178 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec0b0070 con 0x7ff9f0133f70 2024-04-26T00:21:11.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f551e640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:11.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f551e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:21:11.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f551e640 10 monclient: dump: 2024-04-26T00:21:11.681 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:11.681 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:11.681 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:11.681 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:11.681 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:11.681 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:11.681 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:11.681 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:11.681 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:11.681 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:11.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f551e640 10 monclient: _finish_auth 0 2024-04-26T00:21:11.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f551e640 10 monclient: _check_auth_tickets 2024-04-26T00:21:11.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f551e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:11.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f8784640 5 monclient: authenticate success, global_id 4509 2024-04-26T00:21:11.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f551e640 1 -- 172.21.15.37:0/311268178 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9ec0ab600 con 0x7ff9f0133f70 2024-04-26T00:21:11.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f8784640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:21:11.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f551e640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:11.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f8784640 10 monclient: get_monmap_and_config success 2024-04-26T00:21:11.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f8784640 4 set_mon_vals no callback set 2024-04-26T00:21:11.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f551e640 1 -- 172.21.15.37:0/311268178 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec0ab9d0 con 0x7ff9f0133f70 2024-04-26T00:21:11.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f551e640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:11.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f551e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:21:11.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f551e640 10 monclient: dump: 2024-04-26T00:21:11.682 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:11.682 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:11.682 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:11.682 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:11.683 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:11.683 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:11.683 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:11.683 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:11.683 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:11.683 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:11.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f8784640 10 monclient: shutdown 2024-04-26T00:21:11.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f8784640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:11.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f8784640 1 -- 172.21.15.37:0/311268178 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9f0133f70 msgr2=0x7ff9f009c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:11.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f8784640 1 --2- 172.21.15.37:0/311268178 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9f0133f70 0x7ff9f009c140 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec009870 tx=0x7ff9ec01c3e0 comp rx=0 tx=0).stop 2024-04-26T00:21:11.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f8784640 1 -- 172.21.15.37:0/311268178 shutdown_connections 2024-04-26T00:21:11.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f8784640 1 --2- 172.21.15.37:0/311268178 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9f0134790 0x7ff9f01374b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:11.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f8784640 1 --2- 172.21.15.37:0/311268178 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9f0133f70 0x7ff9f009c140 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:11.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f8784640 1 -- 172.21.15.37:0/311268178 >> 172.21.15.37:0/311268178 conn(0x7ff9f00a5f00 msgr2=0x7ff9f00a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:11.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f8784640 1 -- 172.21.15.37:0/311268178 shutdown_connections 2024-04-26T00:21:11.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f8784640 1 -- 172.21.15.37:0/311268178 wait complete. 2024-04-26T00:21:11.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f8784640 10 monclient: build_initial_monmap 2024-04-26T00:21:11.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f8784640 10 monclient: monmap: 2024-04-26T00:21:11.685 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:11.685 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:21:11.686 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:21:11.686 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:21:11.686 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:11.686 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:21:11.686 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:21:11.686 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:21:11.686 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:21:11.686 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:11.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f8784640 1 Processor -- start 2024-04-26T00:21:11.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f8784640 1 -- start start 2024-04-26T00:21:11.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f8784640 10 monclient: init 2024-04-26T00:21:11.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f8784640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:11.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f8784640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:21:11.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f8784640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9f0133f70 0x7ff9f01cf050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:11.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f8784640 10 monclient(hunting): picked mon.noname-b con 0x7ff9f0133f70 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:21:11.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f8784640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9f0134350 0x7ff9f013df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:11.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f8784640 10 monclient(hunting): picked mon.noname-c con 0x7ff9f0134350 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:21:11.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f8784640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9f0134790 0x7ff9f013e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:11.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f8784640 10 monclient(hunting): picked mon.noname-a con 0x7ff9f0134790 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:21:11.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f8784640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:11.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f8784640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff9f0141b10 con 0x7ff9f0134790 2024-04-26T00:21:11.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f8784640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:11.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f8784640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff9f0137c00 con 0x7ff9f0133f70 2024-04-26T00:21:11.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f8784640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:11.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f8784640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff9f0141c90 con 0x7ff9f0134350 2024-04-26T00:21:11.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f8784640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:11.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f8784640 10 monclient(hunting): authenticate will time out at 1053.003113s 2024-04-26T00:21:11.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f6520640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9f0133f70 msgr2=0x7ff9f01cf050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:21:11.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f5d1f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9f0134350 0x7ff9f013df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:11.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f6520640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9f0133f70 0x7ff9f01cf050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:21:11.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f6d21640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9f0134790 0x7ff9f013e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:11.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f5d1f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9f0134350 0x7ff9f013df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:59358/0 (socket says 172.21.15.37:59358) 2024-04-26T00:21:11.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f5d1f640 1 -- 172.21.15.37:0/319383102 learned_addr learned my addr 172.21.15.37:0/319383102 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:21:11.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f5d1f640 10 monclient(hunting): get_auth_request con 0x7ff9f0134350 auth_method 0 2024-04-26T00:21:11.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f5d1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:11.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f5d1f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:11.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f5d1f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:11.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f6d21640 10 monclient(hunting): get_auth_request con 0x7ff9f0134790 auth_method 0 2024-04-26T00:21:11.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f6d21640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:11.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f6d21640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:11.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f6d21640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:11.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f5d1f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:11.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f5d1f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:11.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f5d1f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:11.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f6d21640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:11.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f6d21640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:11.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f6d21640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:11.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f5d1f640 10 monclient(hunting): handle_auth_done global_id 4496 payload 824 2024-04-26T00:21:11.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f5d1f640 1 -- 172.21.15.37:0/319383102 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9f0133f70 msgr2=0x7ff9f01cf050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:21:11.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f5d1f640 1 --2- 172.21.15.37:0/319383102 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9f0133f70 0x7ff9f01cf050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:11.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f5d1f640 1 -- 172.21.15.37:0/319383102 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9f0134790 msgr2=0x7ff9f013e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:11.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f5d1f640 1 --2- 172.21.15.37:0/319383102 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9f0134790 0x7ff9f013e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:11.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f5d1f640 10 monclient: _finish_hunting 0 2024-04-26T00:21:11.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f5d1f640 1 monclient: found mon.noname-c 2024-04-26T00:21:11.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f5d1f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:11.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f5d1f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:21:11.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.695+0000 7ff9f5d1f640 1 -- 172.21.15.37:0/319383102 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9f013a720 con 0x7ff9f0134350 2024-04-26T00:21:11.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.699+0000 7ff9f5d1f640 1 -- 172.21.15.37:0/319383102 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9f0134350 msgr2=0x7ff9f013df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:11.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.699+0000 7ff9f5d1f640 1 --2- 172.21.15.37:0/319383102 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9f0134350 0x7ff9f013df40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff9e000c460 tx=0x7ff9e000c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:11.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.699+0000 7ff9e6ffd640 1 -- 172.21.15.37:0/319383102 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9e0017070 con 0x7ff9f0134350 2024-04-26T00:21:11.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.699+0000 7ff9e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:11.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.699+0000 7ff9e6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:21:11.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.699+0000 7ff9e6ffd640 10 monclient: dump: 2024-04-26T00:21:11.691 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:11.691 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:11.691 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:11.691 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:11.691 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:11.691 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:11.691 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:11.691 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:11.691 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:11.691 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:11.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.699+0000 7ff9e6ffd640 10 monclient: _finish_auth 0 2024-04-26T00:21:11.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.699+0000 7ff9e6ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:21:11.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.699+0000 7ff9e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:11.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.699+0000 7ff9e6ffd640 1 -- 172.21.15.37:0/319383102 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9e0005820 con 0x7ff9f0134350 2024-04-26T00:21:11.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.699+0000 7ff9e6ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:11.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.699+0000 7ff9e6ffd640 1 -- 172.21.15.37:0/319383102 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9e0005c10 con 0x7ff9f0134350 2024-04-26T00:21:11.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.699+0000 7ff9e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:11.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.699+0000 7ff9e6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:21:11.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.699+0000 7ff9e6ffd640 10 monclient: dump: 2024-04-26T00:21:11.692 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:11.692 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:11.692 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:11.692 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:11.692 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:11.692 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:11.692 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:11.692 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:11.692 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:11.693 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:11.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.699+0000 7ff9f8784640 5 monclient: authenticate success, global_id 4496 2024-04-26T00:21:11.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.699+0000 7ff9f8784640 10 monclient: _renew_subs 2024-04-26T00:21:11.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.699+0000 7ff9f8784640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:21:11.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.699+0000 7ff9e7fff640 4 set_mon_vals no callback set 2024-04-26T00:21:11.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.699+0000 7ff9f8784640 1 -- 172.21.15.37:0/319383102 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9f013c170 con 0x7ff9f0134350 2024-04-26T00:21:11.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.699+0000 7ff9f8784640 10 monclient: _renew_subs 2024-04-26T00:21:11.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.699+0000 7ff9f8784640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:21:11.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.699+0000 7ff9f8784640 1 -- 172.21.15.37:0/319383102 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9f013b4a0 con 0x7ff9f0134350 2024-04-26T00:21:11.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.699+0000 7ff9f8784640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:11.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.699+0000 7ff9f8784640 1 --2- 172.21.15.37:0/319383102 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9f0138b00 0x7ff9f0138ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:11.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.699+0000 7ff9f8784640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:11.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.699+0000 7ff9f8784640 1 -- 172.21.15.37:0/319383102 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff9f009a9e0 con 0x7ff9f0138b00 2024-04-26T00:21:11.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.699+0000 7ff9f6520640 1 --2- 172.21.15.37:0/319383102 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9f0138b00 0x7ff9f0138ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:11.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.699+0000 7ff9e6ffd640 1 -- 172.21.15.37:0/319383102 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff9e001a070 con 0x7ff9f0134350 2024-04-26T00:21:11.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.699+0000 7ff9f6520640 10 monclient: get_auth_request con 0x7ff9f0138b00 auth_method 0 2024-04-26T00:21:11.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.699+0000 7ff9f6520640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:11.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.699+0000 7ff9f6520640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:11.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.699+0000 7ff9f6520640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:11.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.699+0000 7ff9f6520640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:11.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.699+0000 7ff9f6520640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:11.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.699+0000 7ff9f6520640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:11.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.699+0000 7ff9e6ffd640 1 --2- 172.21.15.37:0/319383102 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff9d4035e80 0x7ff9d4038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:11.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.699+0000 7ff9e6ffd640 1 -- 172.21.15.37:0/319383102 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(48..48 src has 1..48) ==== 3943+0+0 (secure 0 0 0) 0x7ff9e0047f20 con 0x7ff9f0134350 2024-04-26T00:21:11.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.699+0000 7ff9f6520640 10 monclient(hunting): handle_auth_done global_id 4499 payload 824 2024-04-26T00:21:11.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.699+0000 7ff9f6520640 1 -- 172.21.15.37:0/319383102 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff9f013cb30 con 0x7ff9f0138b00 2024-04-26T00:21:11.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.703+0000 7ff9f6d21640 1 --2- 172.21.15.37:0/319383102 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff9d4035e80 0x7ff9d4038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:11.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.703+0000 7ff9f6d21640 10 monclient: get_auth_request con 0x7ff9d4035e80 auth_method 0 2024-04-26T00:21:11.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.703+0000 7ff9f6520640 1 -- 172.21.15.37:0/319383102 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9f0138b00 msgr2=0x7ff9f0138ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:11.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.703+0000 7ff9f6520640 1 --2- 172.21.15.37:0/319383102 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9f0138b00 0x7ff9f0138ee0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec0ad000 tx=0x7ff9ec01ac20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:11.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.703+0000 7ff9e6ffd640 1 -- 172.21.15.37:0/319383102 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec0ba020 con 0x7ff9f0138b00 2024-04-26T00:21:11.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.703+0000 7ff9e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:11.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.703+0000 7ff9e6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:21:11.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.703+0000 7ff9e6ffd640 10 monclient: dump: 2024-04-26T00:21:11.698 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:11.698 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:11.698 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:11.698 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:11.698 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:11.698 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:11.698 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:11.698 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:11.698 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:11.698 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:11.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.703+0000 7ff9e6ffd640 1 -- 172.21.15.37:0/319383102 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff9ec009490 con 0x7ff9f0138b00 2024-04-26T00:21:11.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.703+0000 7ff9e6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:11.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.703+0000 7ff9e6ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:21:11.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.703+0000 7ff9e6ffd640 1 -- 172.21.15.37:0/319383102 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9f0138b00 msgr2=0x7ff9f0138ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:11.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.703+0000 7ff9e6ffd640 1 --2- 172.21.15.37:0/319383102 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9f0138b00 0x7ff9f0138ee0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec0ad000 tx=0x7ff9ec01ac20 comp rx=0 tx=0).stop 2024-04-26T00:21:11.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.703+0000 7ff9e6ffd640 1 -- 172.21.15.37:0/319383102 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9f0138b00 msgr2=0x7ff9f0138ee0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:21:11.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.703+0000 7ff9e6ffd640 1 --2- 172.21.15.37:0/319383102 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9f0138b00 0x7ff9f0138ee0 secure :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec0ad000 tx=0x7ff9ec01ac20 comp rx=0 tx=0).stop 2024-04-26T00:21:11.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.703+0000 7ff9f6d21640 1 --2- 172.21.15.37:0/319383102 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff9d4035e80 0x7ff9d4038340 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff9f00767e0 tx=0x7ff9e800b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:11.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.711+0000 7ff9f8784640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:21:11.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.711+0000 7ff9f8784640 1 --2- 172.21.15.37:0/319383102 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9f0139420 0x7ff9f00a0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:11.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.711+0000 7ff9f8784640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:11.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.711+0000 7ff9f8784640 1 -- 172.21.15.37:0/319383102 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff9f004f510 con 0x7ff9f0139420 2024-04-26T00:21:11.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.711+0000 7ff9f6520640 1 --2- 172.21.15.37:0/319383102 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9f0139420 0x7ff9f00a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:11.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.711+0000 7ff9f6520640 10 monclient: get_auth_request con 0x7ff9f0139420 auth_method 0 2024-04-26T00:21:11.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.711+0000 7ff9f6520640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:11.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.711+0000 7ff9f6520640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:11.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.711+0000 7ff9f6520640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:11.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.711+0000 7ff9f6520640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:11.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.711+0000 7ff9f6520640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:11.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.711+0000 7ff9f6520640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:11.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.711+0000 7ff9f6520640 10 monclient(hunting): handle_auth_done global_id 4502 payload 824 2024-04-26T00:21:11.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.711+0000 7ff9f6520640 1 -- 172.21.15.37:0/319383102 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff9f00a8750 con 0x7ff9f0139420 2024-04-26T00:21:11.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.711+0000 7ff9f6520640 1 -- 172.21.15.37:0/319383102 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9f0139420 msgr2=0x7ff9f00a0800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:11.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.711+0000 7ff9f6520640 1 --2- 172.21.15.37:0/319383102 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9f0139420 0x7ff9f00a0800 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec009870 tx=0x7ff9ec019180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:11.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.711+0000 7ff9e6ffd640 1 -- 172.21.15.37:0/319383102 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec0c6070 con 0x7ff9f0139420 2024-04-26T00:21:11.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.711+0000 7ff9e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:11.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.711+0000 7ff9e6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:21:11.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.711+0000 7ff9e6ffd640 10 monclient: dump: 2024-04-26T00:21:11.701 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:11.701 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:11.701 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:11.701 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:11.701 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:11.701 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:11.701 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:11.702 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:11.702 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:11.702 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:11.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.711+0000 7ff9e6ffd640 1 -- 172.21.15.37:0/319383102 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4705 (secure 0 0 0) 0x7ff9ec0c1c70 con 0x7ff9f0139420 2024-04-26T00:21:11.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.711+0000 7ff9e6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:21:11.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.711+0000 7ff9e6ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:21:11.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.711+0000 7ff9e6ffd640 1 -- 172.21.15.37:0/319383102 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9f0139420 msgr2=0x7ff9f00a0800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:11.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.711+0000 7ff9e6ffd640 1 --2- 172.21.15.37:0/319383102 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9f0139420 0x7ff9f00a0800 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec009870 tx=0x7ff9ec019180 comp rx=0 tx=0).stop 2024-04-26T00:21:11.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.711+0000 7ff9e6ffd640 1 -- 172.21.15.37:0/319383102 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9f0139420 msgr2=0x7ff9f00a0800 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:21:11.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.711+0000 7ff9e6ffd640 1 --2- 172.21.15.37:0/319383102 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9f0139420 0x7ff9f00a0800 secure :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec009870 tx=0x7ff9ec019180 comp rx=0 tx=0).stop 2024-04-26T00:21:11.702 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:21:11.703 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:21:11.703 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:21:11.703 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:21:11.703 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 16, 2024-04-26T00:21:11.703 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 88819, 2024-04-26T00:21:11.703 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:21:11.703 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:21:11.703 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:21:11.703 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:11.703 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:21:11.703 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:21:11.703 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:21:11.703 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:21:11.704 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:21:11.704 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:21:11.704 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:21:11.704 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:21:11.704 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:21:11.704 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:21:11.704 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:21:11.704 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:21:11.704 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:21:11.704 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:21:11.704 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:21:11.704 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:11.704 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:21:11.704 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:21:11.704 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:21:11.705 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:21:11.705 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:21:11.705 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:21:11.705 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:21:11.705 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:21:11.705 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:21:11.705 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:21:11.705 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:21:11.705 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:21:11.705 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:21:11.705 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:11.705 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:11.705 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:21:11.705 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:21:11.706 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:21:11.706 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:21:11.706 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:21:11.706 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:21:11.706 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:21:11.706 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:21:11.706 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:21:11.706 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:21:11.706 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:21:11.706 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:21:11.706 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:21:11.706 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:21:11.706 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:21:11.706 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:21:11.707 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:21:11.707 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:21:11.707 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:21:11.707 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:21:11.707 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:21:11.707 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:21:11.707 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:21:11.707 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:21:11.707 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:21:11.707 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:21:11.707 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:21:11.707 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:21:11.707 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:11.707 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:21:11.708 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:11.708 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:21:11.708 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:11.708 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:21:11.708 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:21:11.708 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:21:11.708 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:21:11.708 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:11.708 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:21:11.708 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:21:11.708 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:11.708 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:11.708 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:11.708 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:21:11.708 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:21:11.709 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:11.709 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:11.709 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:11.709 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:11.709 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:21:11.709 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:21:11.709 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:21:11.709 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:21:11.709 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:21:11.709 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:11.709 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:11.709 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:21:11.709 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:21:11.709 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:21:11.710 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:21:11.710 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:11.710 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:21:11.710 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:21:11.710 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:11.710 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:11.710 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:11.710 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:21:11.710 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:21:11.710 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:11.710 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:11.710 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:11.710 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:11.711 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:21:11.711 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:21:11.711 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:21:11.711 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:21:11.711 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:21:11.714 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:11.714 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:11.714 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:21:11.714 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:21:11.714 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:21:11.714 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:21:11.714 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:11.714 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:21:11.714 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:21:11.714 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:11.715 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:11.715 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:11.715 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:21:11.715 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:21:11.715 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:11.715 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:11.715 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:11.715 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:11.715 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:21:11.715 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:21:11.715 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:21:11.715 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:21:11.715 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:21:11.715 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:11.715 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:11.716 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:11.716 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:21:11.716 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:21:11.716 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:11.716 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:21:11.716 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:21:11.716 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:21:11.716 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:11.716 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:11.716 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:21:11.716 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:11.716 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:21:11.716 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:21:11.716 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:21:11.716 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:11.717 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:11.717 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:21:11.717 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:11.717 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:21:11.717 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:21:11.717 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:21:11.717 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:11.717 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:11.717 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:11.717 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:21:11.717 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:21:11.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.715+0000 7ff9f8784640 1 -- 172.21.15.37:0/319383102 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff9d4035e80 msgr2=0x7ff9d4038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:11.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.715+0000 7ff9f8784640 1 --2- 172.21.15.37:0/319383102 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff9d4035e80 0x7ff9d4038340 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff9f00767e0 tx=0x7ff9e800b040 comp rx=0 tx=0).stop 2024-04-26T00:21:11.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.715+0000 7ff9f8784640 10 monclient: shutdown 2024-04-26T00:21:11.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.715+0000 7ff9f8784640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:11.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.715+0000 7ff9f8784640 1 -- 172.21.15.37:0/319383102 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9f0134350 msgr2=0x7ff9f013df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:11.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.715+0000 7ff9f8784640 1 --2- 172.21.15.37:0/319383102 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9f0134350 0x7ff9f013df40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff9e000c460 tx=0x7ff9e000c920 comp rx=0 tx=0).stop 2024-04-26T00:21:11.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.715+0000 7ff9f6d21640 1 -- 172.21.15.37:0/319383102 reap_dead start 2024-04-26T00:21:11.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.715+0000 7ff9f6d21640 1 -- 172.21.15.37:0/319383102 reap_dead start 2024-04-26T00:21:11.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.715+0000 7ff9f8784640 1 -- 172.21.15.37:0/319383102 shutdown_connections 2024-04-26T00:21:11.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.715+0000 7ff9f8784640 1 -- 172.21.15.37:0/319383102 >> 172.21.15.37:0/319383102 conn(0x7ff9f00a5f00 msgr2=0x7ff9f0135ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:11.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.715+0000 7ff9f8784640 1 -- 172.21.15.37:0/319383102 shutdown_connections 2024-04-26T00:21:11.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:11.715+0000 7ff9f8784640 1 -- 172.21.15.37:0/319383102 wait complete. 2024-04-26T00:21:11.720 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:21:11.720 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-04-26T00:21:12.102 INFO:tasks.rados.rados.0.smithi045.stdout:164: done (1 left) 2024-04-26T00:21:12.102 INFO:tasks.rados.rados.0.smithi045.stdout:166: done (0 left) 2024-04-26T00:21:12.102 INFO:tasks.rados.rados.0.smithi045.stdout:167: snap_create 2024-04-26T00:21:12.901 INFO:tasks.rados.rados.0.smithi045.stdout:167: done (0 left) 2024-04-26T00:21:12.901 INFO:tasks.rados.rados.0.smithi045.stdout:168: snap_create 2024-04-26T00:21:13.904 INFO:tasks.rados.rados.0.smithi045.stdout:168: done (0 left) 2024-04-26T00:21:13.904 INFO:tasks.rados.rados.0.smithi045.stdout:169: rollback oid 35 current snap is 18 2024-04-26T00:21:13.904 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 35 to 15 2024-04-26T00:21:13.904 INFO:tasks.rados.rados.0.smithi045.stdout:170: snap_create 2024-04-26T00:21:13.912 INFO:tasks.rados.rados.0.smithi045.stdout:169: finishing rollback tid 0 to smithi04526579-35 2024-04-26T00:21:13.913 INFO:tasks.rados.rados.0.smithi045.stdout:169: finishing rollback tid 1 to smithi04526579-35 2024-04-26T00:21:13.913 INFO:tasks.rados.rados.0.smithi045.stdout:169: finishing rollback tid 2 to smithi04526579-35 2024-04-26T00:21:13.913 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 32 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-04-26T00:21:14.905 INFO:tasks.rados.rados.0.smithi045.stdout:169: done (1 left) 2024-04-26T00:21:14.905 INFO:tasks.rados.rados.0.smithi045.stdout:170: done (0 left) 2024-04-26T00:21:14.905 INFO:tasks.rados.rados.0.smithi045.stdout:171: write (excl) oid 47 current snap is 19 2024-04-26T00:21:14.906 INFO:tasks.rados.rados.0.smithi045.stdout:171: seq_num 84 ranges {611788=400799,1748958=488686,2907296=504492} 2024-04-26T00:21:14.922 INFO:tasks.rados.rados.0.smithi045.stdout:171: writing smithi04526579-47 from 611788 to 1012587 tid 1 2024-04-26T00:21:14.942 INFO:tasks.rados.rados.0.smithi045.stdout:171: writing smithi04526579-47 from 1748958 to 2237644 tid 2 2024-04-26T00:21:14.960 INFO:tasks.rados.rados.0.smithi045.stdout:171: writing smithi04526579-47 from 2907296 to 3411788 tid 3 2024-04-26T00:21:14.961 INFO:tasks.rados.rados.0.smithi045.stdout:172: write oid 20 current snap is 19 2024-04-26T00:21:14.961 INFO:tasks.rados.rados.0.smithi045.stdout:172: seq_num 85 ranges {628595=476048,1807440=632344,2934719=452474} 2024-04-26T00:21:14.974 INFO:tasks.rados.rados.0.smithi045.stdout:172: writing smithi04526579-20 from 628595 to 1104643 tid 1 2024-04-26T00:21:14.986 INFO:tasks.rados.rados.0.smithi045.stdout:172: writing smithi04526579-20 from 1807440 to 2439784 tid 2 2024-04-26T00:21:14.995 INFO:tasks.rados.rados.0.smithi045.stdout:172: writing smithi04526579-20 from 2934719 to 3387193 tid 3 2024-04-26T00:21:14.995 INFO:tasks.rados.rados.0.smithi045.stdout:173: delete oid 5 current snap is 19 2024-04-26T00:21:14.995 INFO:tasks.rados.rados.0.smithi045.stdout:171: finishing write tid 1 to smithi04526579-47 2024-04-26T00:21:14.995 INFO:tasks.rados.rados.0.smithi045.stdout:171: finishing write tid 2 to smithi04526579-47 2024-04-26T00:21:14.995 INFO:tasks.rados.rados.0.smithi045.stdout:171: finishing write tid 3 to smithi04526579-47 2024-04-26T00:21:14.995 INFO:tasks.rados.rados.0.smithi045.stdout:171: finishing write tid 4 to smithi04526579-47 2024-04-26T00:21:14.995 INFO:tasks.rados.rados.0.smithi045.stdout:171: finishing write tid 5 to smithi04526579-47 2024-04-26T00:21:14.995 INFO:tasks.rados.rados.0.smithi045.stdout:171: oid 47 updating version 0 to 40 2024-04-26T00:21:14.995 INFO:tasks.rados.rados.0.smithi045.stdout:171: oid 47 updating version 40 to 41 2024-04-26T00:21:14.995 INFO:tasks.rados.rados.0.smithi045.stdout:171: oid 47 updating version 41 to 42 2024-04-26T00:21:14.996 INFO:tasks.rados.rados.0.smithi045.stdout:171: oid 47 version 42 is already newer than 39 2024-04-26T00:21:14.996 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 42 (ObjNum 84 snap 19 seq_num 84) dirty exists 2024-04-26T00:21:14.996 INFO:tasks.rados.rados.0.smithi045.stdout:171: left oid 47 (ObjNum 84 snap 19 seq_num 84) 2024-04-26T00:21:14.996 INFO:tasks.rados.rados.0.smithi045.stdout:172: finishing write tid 1 to smithi04526579-20 2024-04-26T00:21:14.996 INFO:tasks.rados.rados.0.smithi045.stdout:172: finishing write tid 2 to smithi04526579-20 2024-04-26T00:21:15.001 INFO:tasks.rados.rados.0.smithi045.stdout:172: finishing write tid 3 to smithi04526579-20 2024-04-26T00:21:15.003 INFO:tasks.rados.rados.0.smithi045.stdout:171: done (2 left) 2024-04-26T00:21:15.003 INFO:tasks.rados.rados.0.smithi045.stdout:173: done (1 left) 2024-04-26T00:21:15.003 INFO:tasks.rados.rados.0.smithi045.stdout:174: copy_from oid 39 from oid 37 current snap is 19 2024-04-26T00:21:15.003 INFO:tasks.rados.rados.0.smithi045.stdout:175: read oid 45 snap -1 2024-04-26T00:21:15.003 INFO:tasks.rados.rados.0.smithi045.stdout:175: expect (ObjNum 44 snap 0 seq_num 44) 2024-04-26T00:21:15.003 INFO:tasks.rados.rados.0.smithi045.stdout:176: read oid 11 snap -1 2024-04-26T00:21:15.003 INFO:tasks.rados.rados.0.smithi045.stdout:176: expect (ObjNum 63 snap 9 seq_num 63) 2024-04-26T00:21:15.003 INFO:tasks.rados.rados.0.smithi045.stdout:177: read oid 27 snap -1 2024-04-26T00:21:15.003 INFO:tasks.rados.rados.0.smithi045.stdout:177: expect (ObjNum 26 snap 0 seq_num 26) 2024-04-26T00:21:15.003 INFO:tasks.rados.rados.0.smithi045.stdout:178: read oid 13 snap -1 2024-04-26T00:21:15.003 INFO:tasks.rados.rados.0.smithi045.stdout:178: expect (ObjNum 76 snap 13 seq_num 76) 2024-04-26T00:21:15.004 INFO:tasks.rados.rados.0.smithi045.stdout:179: write oid 10 current snap is 19 2024-04-26T00:21:15.004 INFO:tasks.rados.rados.0.smithi045.stdout:179: seq_num 87 ranges {662209=626546,1808051=636013,2873212=588997} 2024-04-26T00:21:15.014 INFO:tasks.rados.rados.0.smithi045.stdout:179: writing smithi04526579-10 from 662209 to 1288755 tid 1 2024-04-26T00:21:15.024 INFO:tasks.rados.rados.0.smithi045.stdout:179: writing smithi04526579-10 from 1808051 to 2444064 tid 2 2024-04-26T00:21:15.033 INFO:tasks.rados.rados.0.smithi045.stdout:179: writing smithi04526579-10 from 2873212 to 3462209 tid 3 2024-04-26T00:21:15.033 INFO:tasks.rados.rados.0.smithi045.stdout:180: read oid 8 snap -1 2024-04-26T00:21:15.033 INFO:tasks.rados.rados.0.smithi045.stdout:180: expect (ObjNum 7 snap 0 seq_num 7) 2024-04-26T00:21:15.033 INFO:tasks.rados.rados.0.smithi045.stdout:181: snap_create 2024-04-26T00:21:15.033 INFO:tasks.rados.rados.0.smithi045.stdout:172: finishing write tid 4 to smithi04526579-20 2024-04-26T00:21:15.033 INFO:tasks.rados.rados.0.smithi045.stdout:172: finishing write tid 5 to smithi04526579-20 2024-04-26T00:21:15.033 INFO:tasks.rados.rados.0.smithi045.stdout:172: oid 20 updating version 0 to 55 2024-04-26T00:21:15.033 INFO:tasks.rados.rados.0.smithi045.stdout:172: oid 20 updating version 55 to 56 2024-04-26T00:21:15.033 INFO:tasks.rados.rados.0.smithi045.stdout:172: oid 20 updating version 56 to 57 2024-04-26T00:21:15.033 INFO:tasks.rados.rados.0.smithi045.stdout:172: oid 20 version 57 is already newer than 54 2024-04-26T00:21:15.033 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 57 (ObjNum 85 snap 19 seq_num 85) dirty exists 2024-04-26T00:21:15.034 INFO:tasks.rados.rados.0.smithi045.stdout:172: left oid 20 (ObjNum 85 snap 19 seq_num 85) 2024-04-26T00:21:15.034 INFO:tasks.rados.rados.0.smithi045.stdout:175: expect (ObjNum 44 snap 0 seq_num 44) 2024-04-26T00:21:15.180 INFO:tasks.rados.rados.0.smithi045.stdout:174: finishing copy_from to smithi04526579-39 2024-04-26T00:21:15.180 INFO:tasks.rados.rados.0.smithi045.stdout:174: got expected ENOENT (src dne) 2024-04-26T00:21:15.180 INFO:tasks.rados.rados.0.smithi045.stdout:177: expect (ObjNum 26 snap 0 seq_num 26) 2024-04-26T00:21:15.257 INFO:tasks.rados.rados.0.smithi045.stdout:174: finishing copy_from racing read to smithi04526579-39 2024-04-26T00:21:15.257 INFO:tasks.rados.rados.0.smithi045.stdout:176: expect (ObjNum 63 snap 9 seq_num 63) 2024-04-26T00:21:15.395 INFO:tasks.rados.rados.0.smithi045.stdout:179: finishing write tid 1 to smithi04526579-10 2024-04-26T00:21:15.395 INFO:tasks.rados.rados.0.smithi045.stdout:179: finishing write tid 2 to smithi04526579-10 2024-04-26T00:21:15.395 INFO:tasks.rados.rados.0.smithi045.stdout:178: expect (ObjNum 76 snap 13 seq_num 76) 2024-04-26T00:21:15.533 INFO:tasks.rados.rados.0.smithi045.stdout:179: finishing write tid 3 to smithi04526579-10 2024-04-26T00:21:15.534 INFO:tasks.rados.rados.0.smithi045.stdout:179: finishing write tid 4 to smithi04526579-10 2024-04-26T00:21:15.534 INFO:tasks.rados.rados.0.smithi045.stdout:179: finishing write tid 5 to smithi04526579-10 2024-04-26T00:21:15.534 INFO:tasks.rados.rados.0.smithi045.stdout:179: oid 10 updating version 0 to 64 2024-04-26T00:21:15.534 INFO:tasks.rados.rados.0.smithi045.stdout:179: oid 10 updating version 64 to 65 2024-04-26T00:21:15.534 INFO:tasks.rados.rados.0.smithi045.stdout:179: oid 10 version 65 is already newer than 63 2024-04-26T00:21:15.534 INFO:tasks.rados.rados.0.smithi045.stdout:179: oid 10 updating version 65 to 66 2024-04-26T00:21:15.534 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 66 (ObjNum 87 snap 19 seq_num 87) dirty exists 2024-04-26T00:21:15.534 INFO:tasks.rados.rados.0.smithi045.stdout:179: left oid 10 (ObjNum 87 snap 19 seq_num 87) 2024-04-26T00:21:15.534 INFO:tasks.rados.rados.0.smithi045.stdout:180: expect (ObjNum 7 snap 0 seq_num 7) 2024-04-26T00:21:15.923 INFO:tasks.rados.rados.0.smithi045.stdout:172: done (8 left) 2024-04-26T00:21:15.923 INFO:tasks.rados.rados.0.smithi045.stdout:174: done (7 left) 2024-04-26T00:21:15.923 INFO:tasks.rados.rados.0.smithi045.stdout:175: done (6 left) 2024-04-26T00:21:15.923 INFO:tasks.rados.rados.0.smithi045.stdout:176: done (5 left) 2024-04-26T00:21:15.923 INFO:tasks.rados.rados.0.smithi045.stdout:177: done (4 left) 2024-04-26T00:21:15.924 INFO:tasks.rados.rados.0.smithi045.stdout:178: done (3 left) 2024-04-26T00:21:15.924 INFO:tasks.rados.rados.0.smithi045.stdout:179: done (2 left) 2024-04-26T00:21:15.924 INFO:tasks.rados.rados.0.smithi045.stdout:180: done (1 left) 2024-04-26T00:21:15.924 INFO:tasks.rados.rados.0.smithi045.stdout:181: done (0 left) 2024-04-26T00:21:15.924 INFO:tasks.rados.rados.0.smithi045.stdout:182: snap_remove snap 17 2024-04-26T00:21:16.987 INFO:tasks.rados.rados.0.smithi045.stdout:182: done (0 left) 2024-04-26T00:21:16.987 INFO:tasks.rados.rados.0.smithi045.stdout:183: snap_create 2024-04-26T00:21:17.904 INFO:tasks.rados.rados.0.smithi045.stdout:183: done (0 left) 2024-04-26T00:21:17.904 INFO:tasks.rados.rados.0.smithi045.stdout:184: copy_from oid 6 from oid 49 current snap is 21 2024-04-26T00:21:17.904 INFO:tasks.rados.rados.0.smithi045.stdout:185: copy_from oid 33 from oid 24 current snap is 21 2024-04-26T00:21:17.904 INFO:tasks.rados.rados.0.smithi045.stdout:186: write (excl) oid 5 current snap is 21 2024-04-26T00:21:17.904 INFO:tasks.rados.rados.0.smithi045.stdout:186: seq_num 88 ranges {679016=701795,1866533=496024,2900635=402643} 2024-04-26T00:21:17.926 INFO:tasks.rados.rados.0.smithi045.stdout:186: writing smithi04526579-5 from 679016 to 1380811 tid 1 2024-04-26T00:21:17.943 INFO:tasks.rados.rados.0.smithi045.stdout:186: writing smithi04526579-5 from 1866533 to 2362557 tid 2 2024-04-26T00:21:17.958 INFO:tasks.rados.rados.0.smithi045.stdout:186: writing smithi04526579-5 from 2900635 to 3303278 tid 3 2024-04-26T00:21:17.958 INFO:tasks.rados.rados.0.smithi045.stdout:187: rollback oid 50 current snap is 21 2024-04-26T00:21:17.958 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 50 to 16 2024-04-26T00:21:17.959 INFO:tasks.rados.rados.0.smithi045.stdout:185: finishing copy_from racing read to smithi04526579-33 2024-04-26T00:21:17.959 INFO:tasks.rados.rados.0.smithi045.stdout:185: finishing copy_from to smithi04526579-33 2024-04-26T00:21:17.959 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 43 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-04-26T00:21:17.959 INFO:tasks.rados.rados.0.smithi045.stdout:186: finishing write tid 1 to smithi04526579-5 2024-04-26T00:21:17.959 INFO:tasks.rados.rados.0.smithi045.stdout:184: finishing copy_from racing read to smithi04526579-6 2024-04-26T00:21:17.959 INFO:tasks.rados.rados.0.smithi045.stdout:184: finishing copy_from to smithi04526579-6 2024-04-26T00:21:17.959 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 15 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-26T00:21:17.959 INFO:tasks.rados.rados.0.smithi045.stdout:186: finishing write tid 2 to smithi04526579-5 2024-04-26T00:21:17.959 INFO:tasks.rados.rados.0.smithi045.stdout:184: done (3 left) 2024-04-26T00:21:17.959 INFO:tasks.rados.rados.0.smithi045.stdout:185: done (2 left) 2024-04-26T00:21:17.959 INFO:tasks.rados.rados.0.smithi045.stdout:188: snap_remove snap 19 2024-04-26T00:21:18.914 INFO:tasks.rados.rados.0.smithi045.stdout:188: done (2 left) 2024-04-26T00:21:18.914 INFO:tasks.rados.rados.0.smithi045.stdout:189: read oid 20 snap 16 2024-04-26T00:21:18.914 INFO:tasks.rados.rados.0.smithi045.stdout:189: expect (ObjNum 19 snap 0 seq_num 19) 2024-04-26T00:21:18.914 INFO:tasks.rados.rados.0.smithi045.stdout:190: rollback oid 25 current snap is 21 2024-04-26T00:21:18.914 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 25 to 15 2024-04-26T00:21:18.915 INFO:tasks.rados.rados.0.smithi045.stdout:186: finishing write tid 3 to smithi04526579-5 2024-04-26T00:21:18.915 INFO:tasks.rados.rados.0.smithi045.stdout:186: finishing write tid 4 to smithi04526579-5 2024-04-26T00:21:18.915 INFO:tasks.rados.rados.0.smithi045.stdout:186: finishing write tid 5 to smithi04526579-5 2024-04-26T00:21:18.915 INFO:tasks.rados.rados.0.smithi045.stdout:186: oid 5 updating version 0 to 111 2024-04-26T00:21:18.915 INFO:tasks.rados.rados.0.smithi045.stdout:186: oid 5 version 111 is already newer than 110 2024-04-26T00:21:18.915 INFO:tasks.rados.rados.0.smithi045.stdout:186: oid 5 updating version 111 to 112 2024-04-26T00:21:18.915 INFO:tasks.rados.rados.0.smithi045.stdout:186: oid 5 version 112 is already newer than 109 2024-04-26T00:21:18.915 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 112 (ObjNum 88 snap 21 seq_num 88) dirty exists 2024-04-26T00:21:18.915 INFO:tasks.rados.rados.0.smithi045.stdout:186: left oid 5 (ObjNum 88 snap 21 seq_num 88) 2024-04-26T00:21:18.915 INFO:tasks.rados.rados.0.smithi045.stdout:187: finishing rollback tid 0 to smithi04526579-50 2024-04-26T00:21:18.915 INFO:tasks.rados.rados.0.smithi045.stdout:187: finishing rollback tid 1 to smithi04526579-50 2024-04-26T00:21:18.915 INFO:tasks.rados.rados.0.smithi045.stdout:187: finishing rollback tid 2 to smithi04526579-50 2024-04-26T00:21:18.915 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 72 (ObjNum 58 snap 6 seq_num 58) dirty exists 2024-04-26T00:21:18.916 INFO:tasks.rados.rados.0.smithi045.stdout:186: done (3 left) 2024-04-26T00:21:18.916 INFO:tasks.rados.rados.0.smithi045.stdout:187: done (2 left) 2024-04-26T00:21:18.916 INFO:tasks.rados.rados.0.smithi045.stdout:191: write (excl) oid 4 current snap is 21 2024-04-26T00:21:18.916 INFO:tasks.rados.rados.0.smithi045.stdout:191: seq_num 89 ranges {695823=777044,1925015=639682,3328058=167765} 2024-04-26T00:21:18.935 INFO:tasks.rados.rados.0.smithi045.stdout:191: writing smithi04526579-4 from 695823 to 1472867 tid 1 2024-04-26T00:21:18.951 INFO:tasks.rados.rados.0.smithi045.stdout:191: writing smithi04526579-4 from 1925015 to 2564697 tid 2 2024-04-26T00:21:18.964 INFO:tasks.rados.rados.0.smithi045.stdout:191: writing smithi04526579-4 from 3328058 to 3495823 tid 3 2024-04-26T00:21:18.964 INFO:tasks.rados.rados.0.smithi045.stdout:192: delete oid 13 current snap is 21 2024-04-26T00:21:18.964 INFO:tasks.rados.rados.0.smithi045.stdout:189: expect (ObjNum 19 snap 0 seq_num 19) 2024-04-26T00:21:19.077 INFO:tasks.rados.rados.0.smithi045.stdout:190: finishing rollback tid 0 to smithi04526579-25 2024-04-26T00:21:19.077 INFO:tasks.rados.rados.0.smithi045.stdout:190: finishing rollback tid 1 to smithi04526579-25 2024-04-26T00:21:19.077 INFO:tasks.rados.rados.0.smithi045.stdout:190: finishing rollback tid 2 to smithi04526579-25 2024-04-26T00:21:19.077 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 118 (ObjNum 72 snap 11 seq_num 72) dirty exists 2024-04-26T00:21:19.077 INFO:tasks.rados.rados.0.smithi045.stdout:191: finishing write tid 1 to smithi04526579-4 2024-04-26T00:21:19.077 INFO:tasks.rados.rados.0.smithi045.stdout:191: finishing write tid 2 to smithi04526579-4 2024-04-26T00:21:19.077 INFO:tasks.rados.rados.0.smithi045.stdout:191: finishing write tid 3 to smithi04526579-4 2024-04-26T00:21:19.078 INFO:tasks.rados.rados.0.smithi045.stdout:191: finishing write tid 4 to smithi04526579-4 2024-04-26T00:21:19.078 INFO:tasks.rados.rados.0.smithi045.stdout:191: finishing write tid 5 to smithi04526579-4 2024-04-26T00:21:19.078 INFO:tasks.rados.rados.0.smithi045.stdout:191: oid 4 updating version 0 to 61 2024-04-26T00:21:19.078 INFO:tasks.rados.rados.0.smithi045.stdout:191: oid 4 updating version 61 to 62 2024-04-26T00:21:19.078 INFO:tasks.rados.rados.0.smithi045.stdout:191: oid 4 version 62 is already newer than 60 2024-04-26T00:21:19.078 INFO:tasks.rados.rados.0.smithi045.stdout:191: oid 4 version 62 is already newer than 59 2024-04-26T00:21:19.078 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 62 (ObjNum 89 snap 21 seq_num 89) dirty exists 2024-04-26T00:21:19.078 INFO:tasks.rados.rados.0.smithi045.stdout:191: left oid 4 (ObjNum 89 snap 21 seq_num 89) 2024-04-26T00:21:19.078 INFO:tasks.rados.rados.0.smithi045.stdout:189: done (3 left) 2024-04-26T00:21:19.078 INFO:tasks.rados.rados.0.smithi045.stdout:190: done (2 left) 2024-04-26T00:21:19.078 INFO:tasks.rados.rados.0.smithi045.stdout:191: done (1 left) 2024-04-26T00:21:19.078 INFO:tasks.rados.rados.0.smithi045.stdout:192: done (0 left) 2024-04-26T00:21:19.078 INFO:tasks.rados.rados.0.smithi045.stdout:193: snap_create 2024-04-26T00:21:20.985 INFO:tasks.rados.rados.0.smithi045.stdout:193: done (0 left) 2024-04-26T00:21:20.985 INFO:tasks.rados.rados.0.smithi045.stdout:194: write oid 31 current snap is 22 2024-04-26T00:21:20.985 INFO:tasks.rados.rados.0.smithi045.stdout:194: seq_num 91 ranges {729437=527542,1925626=643351,3266551=262886} 2024-04-26T00:21:21.005 INFO:tasks.rados.rados.0.smithi045.stdout:194: writing smithi04526579-31 from 729437 to 1256979 tid 1 2024-04-26T00:21:21.027 INFO:tasks.rados.rados.0.smithi045.stdout:194: writing smithi04526579-31 from 1925626 to 2568977 tid 2 2024-04-26T00:21:21.042 INFO:tasks.rados.rados.0.smithi045.stdout:194: writing smithi04526579-31 from 3266551 to 3529437 tid 3 2024-04-26T00:21:21.042 INFO:tasks.rados.rados.0.smithi045.stdout:195: read oid 37 snap -1 2024-04-26T00:21:21.042 INFO:tasks.rados.rados.0.smithi045.stdout:195: expect deleted 2024-04-26T00:21:21.042 INFO:tasks.rados.rados.0.smithi045.stdout:196: delete oid 33 current snap is 22 2024-04-26T00:21:21.042 INFO:tasks.rados.rados.0.smithi045.stdout:194: finishing write tid 1 to smithi04526579-31 2024-04-26T00:21:21.042 INFO:tasks.rados.rados.0.smithi045.stdout:194: finishing write tid 2 to smithi04526579-31 2024-04-26T00:21:21.048 INFO:tasks.rados.rados.0.smithi045.stdout:194: finishing write tid 3 to smithi04526579-31 2024-04-26T00:21:21.048 INFO:tasks.rados.rados.0.smithi045.stdout:194: finishing write tid 4 to smithi04526579-31 2024-04-26T00:21:21.049 INFO:tasks.rados.rados.0.smithi045.stdout:194: finishing write tid 5 to smithi04526579-31 2024-04-26T00:21:21.049 INFO:tasks.rados.rados.0.smithi045.stdout:194: oid 31 updating version 0 to 66 2024-04-26T00:21:21.049 INFO:tasks.rados.rados.0.smithi045.stdout:194: oid 31 version 66 is already newer than 65 2024-04-26T00:21:21.049 INFO:tasks.rados.rados.0.smithi045.stdout:194: oid 31 version 66 is already newer than 64 2024-04-26T00:21:21.049 INFO:tasks.rados.rados.0.smithi045.stdout:194: oid 31 updating version 66 to 67 2024-04-26T00:21:21.049 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 67 (ObjNum 91 snap 22 seq_num 91) dirty exists 2024-04-26T00:21:21.049 INFO:tasks.rados.rados.0.smithi045.stdout:194: left oid 31 (ObjNum 91 snap 22 seq_num 91) 2024-04-26T00:21:21.051 INFO:tasks.rados.rados.0.smithi045.stdout:194: done (2 left) 2024-04-26T00:21:21.051 INFO:tasks.rados.rados.0.smithi045.stdout:195: done (1 left) 2024-04-26T00:21:21.051 INFO:tasks.rados.rados.0.smithi045.stdout:196: done (0 left) 2024-04-26T00:21:21.051 INFO:tasks.rados.rados.0.smithi045.stdout:197: delete oid 3 current snap is 22 2024-04-26T00:21:21.063 INFO:tasks.rados.rados.0.smithi045.stdout:197: done (0 left) 2024-04-26T00:21:21.063 INFO:tasks.rados.rados.0.smithi045.stdout:198: write (excl) oid 4 current snap is 22 2024-04-26T00:21:21.063 INFO:tasks.rados.rados.0.smithi045.stdout:198: seq_num 94 ranges {779858=469642,1817425=507031,2781526=586628} 2024-04-26T00:21:21.081 INFO:tasks.rados.rados.0.smithi045.stdout:198: writing smithi04526579-4 from 779858 to 1249500 tid 1 2024-04-26T00:21:21.097 INFO:tasks.rados.rados.0.smithi045.stdout:198: writing smithi04526579-4 from 1817425 to 2324456 tid 2 2024-04-26T00:21:21.110 INFO:tasks.rados.rados.0.smithi045.stdout:198: writing smithi04526579-4 from 2781526 to 3368154 tid 3 2024-04-26T00:21:21.110 INFO:tasks.rados.rados.0.smithi045.stdout:199: snap_remove snap 14 2024-04-26T00:21:21.995 INFO:tasks.rados.rados.0.smithi045.stdout:198: finishing write tid 1 to smithi04526579-4 2024-04-26T00:21:21.995 INFO:tasks.rados.rados.0.smithi045.stdout:198: finishing write tid 2 to smithi04526579-4 2024-04-26T00:21:21.995 INFO:tasks.rados.rados.0.smithi045.stdout:198: finishing write tid 3 to smithi04526579-4 2024-04-26T00:21:21.995 INFO:tasks.rados.rados.0.smithi045.stdout:198: finishing write tid 4 to smithi04526579-4 2024-04-26T00:21:21.995 INFO:tasks.rados.rados.0.smithi045.stdout:198: finishing write tid 5 to smithi04526579-4 2024-04-26T00:21:21.995 INFO:tasks.rados.rados.0.smithi045.stdout:198: oid 4 updating version 0 to 71 2024-04-26T00:21:21.995 INFO:tasks.rados.rados.0.smithi045.stdout:198: oid 4 updating version 71 to 72 2024-04-26T00:21:21.995 INFO:tasks.rados.rados.0.smithi045.stdout:198: oid 4 version 72 is already newer than 70 2024-04-26T00:21:21.996 INFO:tasks.rados.rados.0.smithi045.stdout:198: oid 4 version 72 is already newer than 69 2024-04-26T00:21:21.996 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 72 (ObjNum 94 snap 22 seq_num 94) dirty exists 2024-04-26T00:21:21.996 INFO:tasks.rados.rados.0.smithi045.stdout:198: left oid 4 (ObjNum 94 snap 22 seq_num 94) 2024-04-26T00:21:21.996 INFO:tasks.rados.rados.0.smithi045.stdout:198: done (1 left) 2024-04-26T00:21:21.996 INFO:tasks.rados.rados.0.smithi045.stdout:199: done (0 left) 2024-04-26T00:21:21.996 INFO:tasks.rados.rados.0.smithi045.stdout:200: write oid 28 current snap is 22 2024-04-26T00:21:21.996 INFO:tasks.rados.rados.0.smithi045.stdout:200: seq_num 95 ranges {796665=544891,1759554=650689,2976243=620422} 2024-04-26T00:21:22.010 INFO:tasks.rados.rados.0.smithi045.stdout:200: writing smithi04526579-28 from 796665 to 1341556 tid 1 2024-04-26T00:21:22.022 INFO:tasks.rados.rados.0.smithi045.stdout:200: writing smithi04526579-28 from 1759554 to 2410243 tid 2 2024-04-26T00:21:22.036 INFO:tasks.rados.rados.0.smithi045.stdout:200: writing smithi04526579-28 from 2976243 to 3596665 tid 3 2024-04-26T00:21:22.036 INFO:tasks.rados.rados.0.smithi045.stdout:201: write (excl) oid 22 current snap is 22 2024-04-26T00:21:22.036 INFO:tasks.rados.rados.0.smithi045.stdout:201: seq_num 96 ranges {413472=620140,1818036=510700,3120019=493453} 2024-04-26T00:21:22.047 INFO:tasks.rados.rados.0.smithi045.stdout:201: writing smithi04526579-22 from 413472 to 1033612 tid 1 2024-04-26T00:21:22.060 INFO:tasks.rados.rados.0.smithi045.stdout:201: writing smithi04526579-22 from 1818036 to 2328736 tid 2 2024-04-26T00:21:22.071 INFO:tasks.rados.rados.0.smithi045.stdout:201: writing smithi04526579-22 from 3120019 to 3613472 tid 3 2024-04-26T00:21:22.071 INFO:tasks.rados.rados.0.smithi045.stdout:202: read oid 49 snap 15 2024-04-26T00:21:22.071 INFO:tasks.rados.rados.0.smithi045.stdout:202: expect (ObjNum 48 snap 0 seq_num 48) 2024-04-26T00:21:22.071 INFO:tasks.rados.rados.0.smithi045.stdout:203: write oid 47 current snap is 22 2024-04-26T00:21:22.071 INFO:tasks.rados.rados.0.smithi045.stdout:203: seq_num 97 ranges {430279=695389,1876518=654358,3031089=536797} 2024-04-26T00:21:22.079 INFO:tasks.rados.rados.0.smithi045.stdout:203: writing smithi04526579-47 from 430279 to 1125668 tid 1 2024-04-26T00:21:22.089 INFO:tasks.rados.rados.0.smithi045.stdout:203: writing smithi04526579-47 from 1876518 to 2530876 tid 2 2024-04-26T00:21:22.096 INFO:tasks.rados.rados.0.smithi045.stdout:203: writing smithi04526579-47 from 3031089 to 3567886 tid 3 2024-04-26T00:21:22.096 INFO:tasks.rados.rados.0.smithi045.stdout:204: rollback oid 19 current snap is 22 2024-04-26T00:21:22.096 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 19 to 20 2024-04-26T00:21:22.096 INFO:tasks.rados.rados.0.smithi045.stdout:200: finishing write tid 1 to smithi04526579-28 2024-04-26T00:21:22.097 INFO:tasks.rados.rados.0.smithi045.stdout:205: copy_from oid 8 from oid 29 current snap is 22 2024-04-26T00:21:22.097 INFO:tasks.rados.rados.0.smithi045.stdout:206: write oid 12 current snap is 22 2024-04-26T00:21:22.097 INFO:tasks.rados.rados.0.smithi045.stdout:206: seq_num 98 ranges {447086=770638,1935000=798016,3458512=188574} 2024-04-26T00:21:22.103 INFO:tasks.rados.rados.0.smithi045.stdout:206: writing smithi04526579-12 from 447086 to 1217724 tid 1 2024-04-26T00:21:22.112 INFO:tasks.rados.rados.0.smithi045.stdout:206: writing smithi04526579-12 from 1935000 to 2733016 tid 2 2024-04-26T00:21:22.118 INFO:tasks.rados.rados.0.smithi045.stdout:206: writing smithi04526579-12 from 3458512 to 3647086 tid 3 2024-04-26T00:21:22.118 INFO:tasks.rados.rados.0.smithi045.stdout:207: read oid 15 snap -1 2024-04-26T00:21:22.119 INFO:tasks.rados.rados.0.smithi045.stdout:207: expect (ObjNum 14 snap 0 seq_num 14) 2024-04-26T00:21:22.119 INFO:tasks.rados.rados.0.smithi045.stdout:208: snap_remove snap 21 2024-04-26T00:21:22.905 INFO:tasks.rados.rados.0.smithi045.stdout:208: done (8 left) 2024-04-26T00:21:22.905 INFO:tasks.rados.rados.0.smithi045.stdout:209: write (excl) oid 43 current snap is 22 2024-04-26T00:21:22.905 INFO:tasks.rados.rados.0.smithi045.stdout:209: seq_num 99 ranges {463893=562240,1593482=658027,2685935=559341} 2024-04-26T00:21:22.911 INFO:tasks.rados.rados.0.smithi045.stdout:209: writing smithi04526579-43 from 463893 to 1026133 tid 1 2024-04-26T00:21:22.919 INFO:tasks.rados.rados.0.smithi045.stdout:209: writing smithi04526579-43 from 1593482 to 2251509 tid 2 2024-04-26T00:21:22.925 INFO:tasks.rados.rados.0.smithi045.stdout:209: writing smithi04526579-43 from 2685935 to 3245276 tid 3 2024-04-26T00:21:22.926 INFO:tasks.rados.rados.0.smithi045.stdout:210: read oid 33 snap 20 2024-04-26T00:21:22.926 INFO:tasks.rados.rados.0.smithi045.stdout:210: expect (ObjNum 32 snap 0 seq_num 32) 2024-04-26T00:21:22.926 INFO:tasks.rados.rados.0.smithi045.stdout:211: snap_create 2024-04-26T00:21:22.926 INFO:tasks.rados.rados.0.smithi045.stdout:200: finishing write tid 2 to smithi04526579-28 2024-04-26T00:21:22.926 INFO:tasks.rados.rados.0.smithi045.stdout:200: finishing write tid 3 to smithi04526579-28 2024-04-26T00:21:22.926 INFO:tasks.rados.rados.0.smithi045.stdout:200: finishing write tid 4 to smithi04526579-28 2024-04-26T00:21:22.926 INFO:tasks.rados.rados.0.smithi045.stdout:200: finishing write tid 5 to smithi04526579-28 2024-04-26T00:21:22.926 INFO:tasks.rados.rados.0.smithi045.stdout:200: oid 28 updating version 0 to 52 2024-04-26T00:21:22.926 INFO:tasks.rados.rados.0.smithi045.stdout:200: oid 28 version 52 is already newer than 50 2024-04-26T00:21:22.926 INFO:tasks.rados.rados.0.smithi045.stdout:200: oid 28 updating version 52 to 53 2024-04-26T00:21:22.926 INFO:tasks.rados.rados.0.smithi045.stdout:200: oid 28 version 53 is already newer than 51 2024-04-26T00:21:22.926 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 53 (ObjNum 95 snap 22 seq_num 95) dirty exists 2024-04-26T00:21:22.926 INFO:tasks.rados.rados.0.smithi045.stdout:200: left oid 28 (ObjNum 95 snap 22 seq_num 95) 2024-04-26T00:21:22.927 INFO:tasks.rados.rados.0.smithi045.stdout:201: finishing write tid 1 to smithi04526579-22 2024-04-26T00:21:22.927 INFO:tasks.rados.rados.0.smithi045.stdout:201: finishing write tid 2 to smithi04526579-22 2024-04-26T00:21:22.927 INFO:tasks.rados.rados.0.smithi045.stdout:201: finishing write tid 3 to smithi04526579-22 2024-04-26T00:21:22.927 INFO:tasks.rados.rados.0.smithi045.stdout:201: finishing write tid 4 to smithi04526579-22 2024-04-26T00:21:22.927 INFO:tasks.rados.rados.0.smithi045.stdout:201: finishing write tid 5 to smithi04526579-22 2024-04-26T00:21:22.927 INFO:tasks.rados.rados.0.smithi045.stdout:201: oid 22 updating version 0 to 37 2024-04-26T00:21:22.927 INFO:tasks.rados.rados.0.smithi045.stdout:201: oid 22 version 37 is already newer than 36 2024-04-26T00:21:22.927 INFO:tasks.rados.rados.0.smithi045.stdout:201: oid 22 version 37 is already newer than 34 2024-04-26T00:21:22.927 INFO:tasks.rados.rados.0.smithi045.stdout:201: oid 22 version 37 is already newer than 35 2024-04-26T00:21:22.927 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 37 (ObjNum 96 snap 22 seq_num 96) dirty exists 2024-04-26T00:21:22.927 INFO:tasks.rados.rados.0.smithi045.stdout:201: left oid 22 (ObjNum 96 snap 22 seq_num 96) 2024-04-26T00:21:22.927 INFO:tasks.rados.rados.0.smithi045.stdout:202: expect (ObjNum 48 snap 0 seq_num 48) 2024-04-26T00:21:23.074 INFO:tasks.rados.rados.0.smithi045.stdout:203: finishing write tid 1 to smithi04526579-47 2024-04-26T00:21:23.074 INFO:tasks.rados.rados.0.smithi045.stdout:203: finishing write tid 2 to smithi04526579-47 2024-04-26T00:21:23.074 INFO:tasks.rados.rados.0.smithi045.stdout:203: finishing write tid 3 to smithi04526579-47 2024-04-26T00:21:23.074 INFO:tasks.rados.rados.0.smithi045.stdout:204: finishing rollback tid 0 to smithi04526579-19 2024-04-26T00:21:23.074 INFO:tasks.rados.rados.0.smithi045.stdout:203: finishing write tid 4 to smithi04526579-47 2024-04-26T00:21:23.074 INFO:tasks.rados.rados.0.smithi045.stdout:203: finishing write tid 5 to smithi04526579-47 2024-04-26T00:21:23.074 INFO:tasks.rados.rados.0.smithi045.stdout:203: oid 47 updating version 0 to 52 2024-04-26T00:21:23.074 INFO:tasks.rados.rados.0.smithi045.stdout:203: oid 47 updating version 52 to 54 2024-04-26T00:21:23.075 INFO:tasks.rados.rados.0.smithi045.stdout:203: oid 47 updating version 54 to 55 2024-04-26T00:21:23.075 INFO:tasks.rados.rados.0.smithi045.stdout:203: oid 47 version 55 is already newer than 53 2024-04-26T00:21:23.075 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 55 (ObjNum 97 snap 22 seq_num 97) dirty exists 2024-04-26T00:21:23.075 INFO:tasks.rados.rados.0.smithi045.stdout:203: left oid 47 (ObjNum 97 snap 22 seq_num 97) 2024-04-26T00:21:23.075 INFO:tasks.rados.rados.0.smithi045.stdout:204: finishing rollback tid 1 to smithi04526579-19 2024-04-26T00:21:23.075 INFO:tasks.rados.rados.0.smithi045.stdout:204: finishing rollback tid 2 to smithi04526579-19 2024-04-26T00:21:23.075 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 42 (ObjNum 52 snap 1 seq_num 52) dirty exists 2024-04-26T00:21:23.075 INFO:tasks.rados.rados.0.smithi045.stdout:205: finishing copy_from racing read to smithi04526579-8 2024-04-26T00:21:23.075 INFO:tasks.rados.rados.0.smithi045.stdout:205: finishing copy_from to smithi04526579-8 2024-04-26T00:21:23.075 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 9 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-26T00:21:23.075 INFO:tasks.rados.rados.0.smithi045.stdout:206: finishing write tid 1 to smithi04526579-12 2024-04-26T00:21:23.075 INFO:tasks.rados.rados.0.smithi045.stdout:206: finishing write tid 2 to smithi04526579-12 2024-04-26T00:21:23.075 INFO:tasks.rados.rados.0.smithi045.stdout:206: finishing write tid 3 to smithi04526579-12 2024-04-26T00:21:23.076 INFO:tasks.rados.rados.0.smithi045.stdout:206: finishing write tid 4 to smithi04526579-12 2024-04-26T00:21:23.076 INFO:tasks.rados.rados.0.smithi045.stdout:207: expect (ObjNum 14 snap 0 seq_num 14) 2024-04-26T00:21:23.147 INFO:tasks.rados.rados.0.smithi045.stdout:206: finishing write tid 5 to smithi04526579-12 2024-04-26T00:21:23.148 INFO:tasks.rados.rados.0.smithi045.stdout:206: oid 12 updating version 0 to 24 2024-04-26T00:21:23.148 INFO:tasks.rados.rados.0.smithi045.stdout:206: oid 12 updating version 24 to 25 2024-04-26T00:21:23.148 INFO:tasks.rados.rados.0.smithi045.stdout:206: oid 12 version 25 is already newer than 23 2024-04-26T00:21:23.148 INFO:tasks.rados.rados.0.smithi045.stdout:206: oid 12 version 25 is already newer than 22 2024-04-26T00:21:23.148 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 25 (ObjNum 98 snap 22 seq_num 98) dirty exists 2024-04-26T00:21:23.148 INFO:tasks.rados.rados.0.smithi045.stdout:206: left oid 12 (ObjNum 98 snap 22 seq_num 98) 2024-04-26T00:21:23.148 INFO:tasks.rados.rados.0.smithi045.stdout:209: finishing write tid 1 to smithi04526579-43 2024-04-26T00:21:23.148 INFO:tasks.rados.rados.0.smithi045.stdout:209: finishing write tid 2 to smithi04526579-43 2024-04-26T00:21:23.148 INFO:tasks.rados.rados.0.smithi045.stdout:209: finishing write tid 3 to smithi04526579-43 2024-04-26T00:21:23.148 INFO:tasks.rados.rados.0.smithi045.stdout:209: finishing write tid 4 to smithi04526579-43 2024-04-26T00:21:23.148 INFO:tasks.rados.rados.0.smithi045.stdout:210: expect (ObjNum 32 snap 0 seq_num 32) 2024-04-26T00:21:23.230 INFO:tasks.rados.rados.0.smithi045.stdout:209: finishing write tid 5 to smithi04526579-43 2024-04-26T00:21:23.230 INFO:tasks.rados.rados.0.smithi045.stdout:209: oid 43 updating version 0 to 82 2024-04-26T00:21:23.230 INFO:tasks.rados.rados.0.smithi045.stdout:209: oid 43 version 82 is already newer than 79 2024-04-26T00:21:23.230 INFO:tasks.rados.rados.0.smithi045.stdout:209: oid 43 version 82 is already newer than 81 2024-04-26T00:21:23.231 INFO:tasks.rados.rados.0.smithi045.stdout:209: oid 43 version 82 is already newer than 80 2024-04-26T00:21:23.231 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 82 (ObjNum 99 snap 22 seq_num 99) dirty exists 2024-04-26T00:21:23.231 INFO:tasks.rados.rados.0.smithi045.stdout:209: left oid 43 (ObjNum 99 snap 22 seq_num 99) 2024-04-26T00:21:23.917 INFO:tasks.rados.rados.0.smithi045.stdout:200: done (10 left) 2024-04-26T00:21:23.917 INFO:tasks.rados.rados.0.smithi045.stdout:201: done (9 left) 2024-04-26T00:21:23.917 INFO:tasks.rados.rados.0.smithi045.stdout:202: done (8 left) 2024-04-26T00:21:23.917 INFO:tasks.rados.rados.0.smithi045.stdout:203: done (7 left) 2024-04-26T00:21:23.917 INFO:tasks.rados.rados.0.smithi045.stdout:204: done (6 left) 2024-04-26T00:21:23.917 INFO:tasks.rados.rados.0.smithi045.stdout:205: done (5 left) 2024-04-26T00:21:23.917 INFO:tasks.rados.rados.0.smithi045.stdout:206: done (4 left) 2024-04-26T00:21:23.917 INFO:tasks.rados.rados.0.smithi045.stdout:207: done (3 left) 2024-04-26T00:21:23.917 INFO:tasks.rados.rados.0.smithi045.stdout:209: done (2 left) 2024-04-26T00:21:23.917 INFO:tasks.rados.rados.0.smithi045.stdout:210: done (1 left) 2024-04-26T00:21:23.917 INFO:tasks.rados.rados.0.smithi045.stdout:211: done (0 left) 2024-04-26T00:21:23.917 INFO:tasks.rados.rados.0.smithi045.stdout:212: rollback oid 7 current snap is 23 2024-04-26T00:21:23.917 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 7 to 15 2024-04-26T00:21:23.918 INFO:tasks.rados.rados.0.smithi045.stdout:213: read oid 15 snap -1 2024-04-26T00:21:23.918 INFO:tasks.rados.rados.0.smithi045.stdout:213: expect (ObjNum 14 snap 0 seq_num 14) 2024-04-26T00:21:23.918 INFO:tasks.rados.rados.0.smithi045.stdout:214: snap_create 2024-04-26T00:21:23.921 INFO:tasks.rados.rados.0.smithi045.stdout:213: expect (ObjNum 14 snap 0 seq_num 14) 2024-04-26T00:21:23.995 INFO:tasks.rados.rados.0.smithi045.stdout:212: finishing rollback tid 0 to smithi04526579-7 2024-04-26T00:21:23.995 INFO:tasks.rados.rados.0.smithi045.stdout:212: finishing rollback tid 1 to smithi04526579-7 2024-04-26T00:21:23.995 INFO:tasks.rados.rados.0.smithi045.stdout:212: finishing rollback tid 2 to smithi04526579-7 2024-04-26T00:21:23.995 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 132 (ObjNum 70 snap 10 seq_num 70) dirty exists 2024-04-26T00:21:24.919 INFO:tasks.rados.rados.0.smithi045.stdout:212: done (2 left) 2024-04-26T00:21:24.919 INFO:tasks.rados.rados.0.smithi045.stdout:213: done (1 left) 2024-04-26T00:21:24.919 INFO:tasks.rados.rados.0.smithi045.stdout:214: done (0 left) 2024-04-26T00:21:24.919 INFO:tasks.rados.rados.0.smithi045.stdout:215: snap_remove snap 15 2024-04-26T00:21:25.940 INFO:tasks.rados.rados.0.smithi045.stdout:215: done (0 left) 2024-04-26T00:21:25.940 INFO:tasks.rados.rados.0.smithi045.stdout:216: write oid 46 current snap is 24 2024-04-26T00:21:25.940 INFO:tasks.rados.rados.0.smithi045.stdout:216: seq_num 100 ranges {480700=637489,1651964=401685,2713358=770613} 2024-04-26T00:21:25.946 INFO:tasks.rados.rados.0.smithi045.stdout:216: writing smithi04526579-46 from 480700 to 1118189 tid 1 2024-04-26T00:21:25.952 INFO:tasks.rados.rados.0.smithi045.stdout:216: writing smithi04526579-46 from 1651964 to 2053649 tid 2 2024-04-26T00:21:25.961 INFO:tasks.rados.rados.0.smithi045.stdout:216: writing smithi04526579-46 from 2713358 to 3483971 tid 3 2024-04-26T00:21:25.961 INFO:tasks.rados.rados.0.smithi045.stdout:217: rollback oid 29 current snap is 24 2024-04-26T00:21:25.961 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 29 to 16 2024-04-26T00:21:25.961 INFO:tasks.rados.rados.0.smithi045.stdout:218: read oid 3 snap -1 2024-04-26T00:21:25.961 INFO:tasks.rados.rados.0.smithi045.stdout:218: expect deleted 2024-04-26T00:21:25.961 INFO:tasks.rados.rados.0.smithi045.stdout:219: snap_create 2024-04-26T00:21:25.968 INFO:tasks.rados.rados.0.smithi045.stdout:216: finishing write tid 1 to smithi04526579-46 2024-04-26T00:21:25.969 INFO:tasks.rados.rados.0.smithi045.stdout:216: finishing write tid 2 to smithi04526579-46 2024-04-26T00:21:25.969 INFO:tasks.rados.rados.0.smithi045.stdout:217: finishing rollback tid 0 to smithi04526579-29 2024-04-26T00:21:25.970 INFO:tasks.rados.rados.0.smithi045.stdout:216: finishing write tid 3 to smithi04526579-46 2024-04-26T00:21:25.970 INFO:tasks.rados.rados.0.smithi045.stdout:217: finishing rollback tid 1 to smithi04526579-29 2024-04-26T00:21:25.971 INFO:tasks.rados.rados.0.smithi045.stdout:216: finishing write tid 4 to smithi04526579-46 2024-04-26T00:21:25.972 INFO:tasks.rados.rados.0.smithi045.stdout:217: finishing rollback tid 2 to smithi04526579-29 2024-04-26T00:21:25.972 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 42 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-26T00:21:25.972 INFO:tasks.rados.rados.0.smithi045.stdout:216: finishing write tid 5 to smithi04526579-46 2024-04-26T00:21:25.972 INFO:tasks.rados.rados.0.smithi045.stdout:216: oid 46 updating version 0 to 43 2024-04-26T00:21:25.972 INFO:tasks.rados.rados.0.smithi045.stdout:216: oid 46 updating version 43 to 44 2024-04-26T00:21:25.972 INFO:tasks.rados.rados.0.smithi045.stdout:216: oid 46 updating version 44 to 45 2024-04-26T00:21:25.972 INFO:tasks.rados.rados.0.smithi045.stdout:216: oid 46 updating version 45 to 46 2024-04-26T00:21:25.972 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 46 (ObjNum 100 snap 24 seq_num 100) dirty exists 2024-04-26T00:21:25.972 INFO:tasks.rados.rados.0.smithi045.stdout:216: left oid 46 (ObjNum 100 snap 24 seq_num 100) 2024-04-26T00:21:26.946 INFO:tasks.rados.rados.0.smithi045.stdout:216: done (3 left) 2024-04-26T00:21:26.946 INFO:tasks.rados.rados.0.smithi045.stdout:217: done (2 left) 2024-04-26T00:21:26.946 INFO:tasks.rados.rados.0.smithi045.stdout:218: done (1 left) 2024-04-26T00:21:26.946 INFO:tasks.rados.rados.0.smithi045.stdout:219: done (0 left) 2024-04-26T00:21:26.946 INFO:tasks.rados.rados.0.smithi045.stdout:220: read oid 41 snap 24 2024-04-26T00:21:26.946 INFO:tasks.rados.rados.0.smithi045.stdout:220: expect (ObjNum 75 snap 13 seq_num 75) 2024-04-26T00:21:26.947 INFO:tasks.rados.rados.0.smithi045.stdout:221: read oid 30 snap -1 2024-04-26T00:21:26.947 INFO:tasks.rados.rados.0.smithi045.stdout:221: expect (ObjNum 83 snap 15 seq_num 83) 2024-04-26T00:21:26.947 INFO:tasks.rados.rados.0.smithi045.stdout:222: delete oid 42 current snap is 25 2024-04-26T00:21:26.960 INFO:tasks.rados.rados.0.smithi045.stdout:220: expect (ObjNum 75 snap 13 seq_num 75) 2024-04-26T00:21:27.161 INFO:tasks.rados.rados.0.smithi045.stdout:221: expect (ObjNum 83 snap 15 seq_num 83) 2024-04-26T00:21:27.314 INFO:tasks.rados.rados.0.smithi045.stdout:220: done (2 left) 2024-04-26T00:21:27.315 INFO:tasks.rados.rados.0.smithi045.stdout:221: done (1 left) 2024-04-26T00:21:27.315 INFO:tasks.rados.rados.0.smithi045.stdout:222: done (0 left) 2024-04-26T00:21:27.315 INFO:tasks.rados.rados.0.smithi045.stdout:223: rollback oid 22 current snap is 25 2024-04-26T00:21:27.315 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 22 to 16 2024-04-26T00:21:27.315 INFO:tasks.rados.rados.0.smithi045.stdout:224: read oid 47 snap -1 2024-04-26T00:21:27.315 INFO:tasks.rados.rados.0.smithi045.stdout:224: expect (ObjNum 97 snap 22 seq_num 97) 2024-04-26T00:21:27.315 INFO:tasks.rados.rados.0.smithi045.stdout:225: snap_remove snap 24 2024-04-26T00:21:27.949 INFO:tasks.rados.rados.0.smithi045.stdout:225: done (2 left) 2024-04-26T00:21:27.949 INFO:tasks.rados.rados.0.smithi045.stdout:226: rollback oid 36 current snap is 25 2024-04-26T00:21:27.949 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 36 to 22 2024-04-26T00:21:27.949 INFO:tasks.rados.rados.0.smithi045.stdout:223: finishing rollback tid 0 to smithi04526579-22 2024-04-26T00:21:27.949 INFO:tasks.rados.rados.0.smithi045.stdout:223: finishing rollback tid 1 to smithi04526579-22 2024-04-26T00:21:27.949 INFO:tasks.rados.rados.0.smithi045.stdout:223: finishing rollback tid 2 to smithi04526579-22 2024-04-26T00:21:27.949 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 50 (ObjNum 53 snap 3 seq_num 53) dirty exists 2024-04-26T00:21:27.949 INFO:tasks.rados.rados.0.smithi045.stdout:224: expect (ObjNum 97 snap 22 seq_num 97) 2024-04-26T00:21:28.232 INFO:tasks.rados.rados.0.smithi045.stdout:226: finishing rollback tid 0 to smithi04526579-36 2024-04-26T00:21:28.232 INFO:tasks.rados.rados.0.smithi045.stdout:226: finishing rollback tid 1 to smithi04526579-36 2024-04-26T00:21:28.232 INFO:tasks.rados.rados.0.smithi045.stdout:223: done (2 left) 2024-04-26T00:21:28.232 INFO:tasks.rados.rados.0.smithi045.stdout:224: done (1 left) 2024-04-26T00:21:28.232 INFO:tasks.rados.rados.0.smithi045.stdout:227: write (excl) oid 41 current snap is 25 2024-04-26T00:21:28.232 INFO:tasks.rados.rados.0.smithi045.stdout:226: finishing rollback tid 2 to smithi04526579-36 2024-04-26T00:21:28.232 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 54 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-26T00:21:28.232 INFO:tasks.rados.rados.0.smithi045.stdout:227: seq_num 102 ranges {514314=787987,1768928=405354,2768204=509510} 2024-04-26T00:21:28.239 INFO:tasks.rados.rados.0.smithi045.stdout:227: writing smithi04526579-41 from 514314 to 1302301 tid 1 2024-04-26T00:21:28.243 INFO:tasks.rados.rados.0.smithi045.stdout:227: writing smithi04526579-41 from 1768928 to 2174282 tid 2 2024-04-26T00:21:28.249 INFO:tasks.rados.rados.0.smithi045.stdout:227: writing smithi04526579-41 from 2768204 to 3277714 tid 3 2024-04-26T00:21:28.250 INFO:tasks.rados.rados.0.smithi045.stdout:226: done (1 left) 2024-04-26T00:21:28.250 INFO:tasks.rados.rados.0.smithi045.stdout:228: snap_create 2024-04-26T00:21:28.260 INFO:tasks.rados.rados.0.smithi045.stdout:227: finishing write tid 1 to smithi04526579-41 2024-04-26T00:21:28.260 INFO:tasks.rados.rados.0.smithi045.stdout:227: finishing write tid 2 to smithi04526579-41 2024-04-26T00:21:28.260 INFO:tasks.rados.rados.0.smithi045.stdout:227: finishing write tid 3 to smithi04526579-41 2024-04-26T00:21:28.260 INFO:tasks.rados.rados.0.smithi045.stdout:227: finishing write tid 4 to smithi04526579-41 2024-04-26T00:21:28.260 INFO:tasks.rados.rados.0.smithi045.stdout:227: finishing write tid 5 to smithi04526579-41 2024-04-26T00:21:28.261 INFO:tasks.rados.rados.0.smithi045.stdout:227: oid 41 updating version 0 to 58 2024-04-26T00:21:28.261 INFO:tasks.rados.rados.0.smithi045.stdout:227: oid 41 version 58 is already newer than 57 2024-04-26T00:21:28.261 INFO:tasks.rados.rados.0.smithi045.stdout:227: oid 41 version 58 is already newer than 56 2024-04-26T00:21:28.261 INFO:tasks.rados.rados.0.smithi045.stdout:227: oid 41 version 58 is already newer than 55 2024-04-26T00:21:28.261 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 58 (ObjNum 102 snap 25 seq_num 102) dirty exists 2024-04-26T00:21:28.261 INFO:tasks.rados.rados.0.smithi045.stdout:227: left oid 41 (ObjNum 102 snap 25 seq_num 102) 2024-04-26T00:21:28.951 INFO:tasks.rados.rados.0.smithi045.stdout:227: done (1 left) 2024-04-26T00:21:28.952 INFO:tasks.rados.rados.0.smithi045.stdout:228: done (0 left) 2024-04-26T00:21:28.952 INFO:tasks.rados.rados.0.smithi045.stdout:229: delete oid 36 current snap is 26 2024-04-26T00:21:28.957 INFO:tasks.rados.rados.0.smithi045.stdout:229: done (0 left) 2024-04-26T00:21:28.958 INFO:tasks.rados.rados.0.smithi045.stdout:230: delete oid 33 current snap is 26 2024-04-26T00:21:28.965 INFO:tasks.rados.rados.0.smithi045.stdout:230: done (0 left) 2024-04-26T00:21:28.965 INFO:tasks.rados.rados.0.smithi045.stdout:231: snap_create 2024-04-26T00:21:29.968 INFO:tasks.rados.rados.0.smithi045.stdout:231: done (0 left) 2024-04-26T00:21:29.968 INFO:tasks.rados.rados.0.smithi045.stdout:232: read oid 1 snap -1 2024-04-26T00:21:29.968 INFO:tasks.rados.rados.0.smithi045.stdout:232: expect deleted 2024-04-26T00:21:29.968 INFO:tasks.rados.rados.0.smithi045.stdout:233: write oid 45 current snap is 27 2024-04-26T00:21:29.968 INFO:tasks.rados.rados.0.smithi045.stdout:233: seq_num 105 ranges {564735=613734,1828021=669034,3134120=630615} 2024-04-26T00:21:29.985 INFO:tasks.rados.rados.0.smithi045.stdout:233: writing smithi04526579-45 from 564735 to 1178469 tid 1 2024-04-26T00:21:30.007 INFO:tasks.rados.rados.0.smithi045.stdout:233: writing smithi04526579-45 from 1828021 to 2497055 tid 2 2024-04-26T00:21:30.026 INFO:tasks.rados.rados.0.smithi045.stdout:233: writing smithi04526579-45 from 3134120 to 3764735 tid 3 2024-04-26T00:21:30.026 INFO:tasks.rados.rados.0.smithi045.stdout:234: rollback oid 2 current snap is 27 2024-04-26T00:21:30.026 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 2 to 22 2024-04-26T00:21:30.026 INFO:tasks.rados.rados.0.smithi045.stdout:233: finishing write tid 1 to smithi04526579-45 2024-04-26T00:21:30.027 INFO:tasks.rados.rados.0.smithi045.stdout:233: finishing write tid 2 to smithi04526579-45 2024-04-26T00:21:30.027 INFO:tasks.rados.rados.0.smithi045.stdout:232: done (2 left) 2024-04-26T00:21:30.027 INFO:tasks.rados.rados.0.smithi045.stdout:235: write (excl) oid 31 current snap is 27 2024-04-26T00:21:30.027 INFO:tasks.rados.rados.0.smithi045.stdout:235: seq_num 106 ranges {581542=688983,1886503=412692,2761543=670951} 2024-04-26T00:21:30.041 INFO:tasks.rados.rados.0.smithi045.stdout:235: writing smithi04526579-31 from 581542 to 1270525 tid 1 2024-04-26T00:21:30.050 INFO:tasks.rados.rados.0.smithi045.stdout:235: writing smithi04526579-31 from 1886503 to 2299195 tid 2 2024-04-26T00:21:30.060 INFO:tasks.rados.rados.0.smithi045.stdout:235: writing smithi04526579-31 from 2761543 to 3432494 tid 3 2024-04-26T00:21:30.060 INFO:tasks.rados.rados.0.smithi045.stdout:236: snap_create 2024-04-26T00:21:30.060 INFO:tasks.rados.rados.0.smithi045.stdout:233: finishing write tid 3 to smithi04526579-45 2024-04-26T00:21:30.060 INFO:tasks.rados.rados.0.smithi045.stdout:233: finishing write tid 4 to smithi04526579-45 2024-04-26T00:21:30.060 INFO:tasks.rados.rados.0.smithi045.stdout:233: finishing write tid 5 to smithi04526579-45 2024-04-26T00:21:30.060 INFO:tasks.rados.rados.0.smithi045.stdout:233: oid 45 updating version 0 to 21 2024-04-26T00:21:30.060 INFO:tasks.rados.rados.0.smithi045.stdout:233: oid 45 version 21 is already newer than 19 2024-04-26T00:21:30.061 INFO:tasks.rados.rados.0.smithi045.stdout:233: oid 45 updating version 21 to 22 2024-04-26T00:21:30.061 INFO:tasks.rados.rados.0.smithi045.stdout:233: oid 45 version 22 is already newer than 20 2024-04-26T00:21:30.061 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 22 (ObjNum 105 snap 27 seq_num 105) dirty exists 2024-04-26T00:21:30.061 INFO:tasks.rados.rados.0.smithi045.stdout:233: left oid 45 (ObjNum 105 snap 27 seq_num 105) 2024-04-26T00:21:30.061 INFO:tasks.rados.rados.0.smithi045.stdout:234: finishing rollback tid 0 to smithi04526579-2 2024-04-26T00:21:30.061 INFO:tasks.rados.rados.0.smithi045.stdout:234: finishing rollback tid 1 to smithi04526579-2 2024-04-26T00:21:30.061 INFO:tasks.rados.rados.0.smithi045.stdout:234: finishing rollback tid 2 to smithi04526579-2 2024-04-26T00:21:30.061 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 88 (ObjNum 62 snap 9 seq_num 62) dirty exists 2024-04-26T00:21:30.061 INFO:tasks.rados.rados.0.smithi045.stdout:235: finishing write tid 1 to smithi04526579-31 2024-04-26T00:21:30.061 INFO:tasks.rados.rados.0.smithi045.stdout:235: finishing write tid 2 to smithi04526579-31 2024-04-26T00:21:30.067 INFO:tasks.rados.rados.0.smithi045.stdout:235: finishing write tid 3 to smithi04526579-31 2024-04-26T00:21:30.068 INFO:tasks.rados.rados.0.smithi045.stdout:235: finishing write tid 4 to smithi04526579-31 2024-04-26T00:21:30.068 INFO:tasks.rados.rados.0.smithi045.stdout:235: finishing write tid 5 to smithi04526579-31 2024-04-26T00:21:30.068 INFO:tasks.rados.rados.0.smithi045.stdout:235: oid 31 updating version 0 to 90 2024-04-26T00:21:30.068 INFO:tasks.rados.rados.0.smithi045.stdout:235: oid 31 updating version 90 to 91 2024-04-26T00:21:30.068 INFO:tasks.rados.rados.0.smithi045.stdout:235: oid 31 updating version 91 to 93 2024-04-26T00:21:30.068 INFO:tasks.rados.rados.0.smithi045.stdout:235: oid 31 version 93 is already newer than 92 2024-04-26T00:21:30.068 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 93 (ObjNum 106 snap 27 seq_num 106) dirty exists 2024-04-26T00:21:30.068 INFO:tasks.rados.rados.0.smithi045.stdout:235: left oid 31 (ObjNum 106 snap 27 seq_num 106) 2024-04-26T00:21:30.971 INFO:tasks.rados.rados.0.smithi045.stdout:233: done (3 left) 2024-04-26T00:21:30.971 INFO:tasks.rados.rados.0.smithi045.stdout:234: done (2 left) 2024-04-26T00:21:30.971 INFO:tasks.rados.rados.0.smithi045.stdout:235: done (1 left) 2024-04-26T00:21:30.972 INFO:tasks.rados.rados.0.smithi045.stdout:236: done (0 left) 2024-04-26T00:21:30.972 INFO:tasks.rados.rados.0.smithi045.stdout:237: rollback oid 23 current snap is 28 2024-04-26T00:21:30.972 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 23 to 16 2024-04-26T00:21:30.972 INFO:tasks.rados.rados.0.smithi045.stdout:238: copy_from oid 41 from oid 20 current snap is 28 2024-04-26T00:21:30.972 INFO:tasks.rados.rados.0.smithi045.stdout:239: write oid 40 current snap is 28 2024-04-26T00:21:30.972 INFO:tasks.rados.rados.0.smithi045.stdout:239: seq_num 107 ranges {598349=480585,1544985=672703,2905319=482223} 2024-04-26T00:21:30.981 INFO:tasks.rados.rados.0.smithi045.stdout:239: writing smithi04526579-40 from 598349 to 1078934 tid 1 2024-04-26T00:21:30.990 INFO:tasks.rados.rados.0.smithi045.stdout:239: writing smithi04526579-40 from 1544985 to 2217688 tid 2 2024-04-26T00:21:30.999 INFO:tasks.rados.rados.0.smithi045.stdout:239: writing smithi04526579-40 from 2905319 to 3387542 tid 3 2024-04-26T00:21:31.000 INFO:tasks.rados.rados.0.smithi045.stdout:240: delete oid 1 current snap is 28 2024-04-26T00:21:31.000 INFO:tasks.rados.rados.0.smithi045.stdout:237: finishing rollback tid 0 to smithi04526579-23 2024-04-26T00:21:31.000 INFO:tasks.rados.rados.0.smithi045.stdout:237: finishing rollback tid 1 to smithi04526579-23 2024-04-26T00:21:31.000 INFO:tasks.rados.rados.0.smithi045.stdout:237: finishing rollback tid 2 to smithi04526579-23 2024-04-26T00:21:31.000 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 33 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-04-26T00:21:31.000 INFO:tasks.rados.rados.0.smithi045.stdout:239: finishing write tid 1 to smithi04526579-40 2024-04-26T00:21:31.000 INFO:tasks.rados.rados.0.smithi045.stdout:239: finishing write tid 2 to smithi04526579-40 2024-04-26T00:21:31.005 INFO:tasks.rados.rados.0.smithi045.stdout:239: finishing write tid 3 to smithi04526579-40 2024-04-26T00:21:31.005 INFO:tasks.rados.rados.0.smithi045.stdout:239: finishing write tid 4 to smithi04526579-40 2024-04-26T00:21:31.006 INFO:tasks.rados.rados.0.smithi045.stdout:239: finishing write tid 5 to smithi04526579-40 2024-04-26T00:21:31.006 INFO:tasks.rados.rados.0.smithi045.stdout:239: oid 40 updating version 0 to 34 2024-04-26T00:21:31.006 INFO:tasks.rados.rados.0.smithi045.stdout:239: oid 40 updating version 34 to 37 2024-04-26T00:21:31.006 INFO:tasks.rados.rados.0.smithi045.stdout:239: oid 40 version 37 is already newer than 35 2024-04-26T00:21:31.006 INFO:tasks.rados.rados.0.smithi045.stdout:239: oid 40 version 37 is already newer than 36 2024-04-26T00:21:31.006 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 37 (ObjNum 107 snap 28 seq_num 107) dirty exists 2024-04-26T00:21:31.006 INFO:tasks.rados.rados.0.smithi045.stdout:239: left oid 40 (ObjNum 107 snap 28 seq_num 107) 2024-04-26T00:21:31.012 INFO:tasks.rados.rados.0.smithi045.stdout:238: finishing copy_from racing read to smithi04526579-41 2024-04-26T00:21:31.014 INFO:tasks.rados.rados.0.smithi045.stdout:238: finishing copy_from to smithi04526579-41 2024-04-26T00:21:31.014 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 57 (ObjNum 85 snap 19 seq_num 85) dirty exists 2024-04-26T00:21:31.015 INFO:tasks.rados.rados.0.smithi045.stdout:237: done (3 left) 2024-04-26T00:21:31.015 INFO:tasks.rados.rados.0.smithi045.stdout:238: done (2 left) 2024-04-26T00:21:31.015 INFO:tasks.rados.rados.0.smithi045.stdout:239: done (1 left) 2024-04-26T00:21:31.015 INFO:tasks.rados.rados.0.smithi045.stdout:240: done (0 left) 2024-04-26T00:21:31.015 INFO:tasks.rados.rados.0.smithi045.stdout:241: copy_from oid 3 from oid 50 current snap is 28 2024-04-26T00:21:31.016 INFO:tasks.rados.rados.0.smithi045.stdout:242: delete oid 31 current snap is 28 2024-04-26T00:21:31.026 INFO:tasks.rados.rados.0.smithi045.stdout:242: done (1 left) 2024-04-26T00:21:31.026 INFO:tasks.rados.rados.0.smithi045.stdout:243: write (excl) oid 33 current snap is 28 2024-04-26T00:21:31.026 INFO:tasks.rados.rados.0.smithi045.stdout:243: seq_num 110 ranges {648770=706332,2120431=420030,3271235=432392} 2024-04-26T00:21:31.040 INFO:tasks.rados.rados.0.smithi045.stdout:243: writing smithi04526579-33 from 648770 to 1355102 tid 1 2024-04-26T00:21:31.052 INFO:tasks.rados.rados.0.smithi045.stdout:243: writing smithi04526579-33 from 2120431 to 2540461 tid 2 2024-04-26T00:21:31.063 INFO:tasks.rados.rados.0.smithi045.stdout:243: writing smithi04526579-33 from 3271235 to 3703627 tid 3 2024-04-26T00:21:31.064 INFO:tasks.rados.rados.0.smithi045.stdout:244: write oid 11 current snap is 28 2024-04-26T00:21:31.064 INFO:tasks.rados.rados.0.smithi045.stdout:244: seq_num 111 ranges {665577=781581,2062560=563688,3182305=643664} 2024-04-26T00:21:31.075 INFO:tasks.rados.rados.0.smithi045.stdout:244: writing smithi04526579-11 from 665577 to 1447158 tid 1 2024-04-26T00:21:31.084 INFO:tasks.rados.rados.0.smithi045.stdout:244: writing smithi04526579-11 from 2062560 to 2626248 tid 2 2024-04-26T00:21:31.093 INFO:tasks.rados.rados.0.smithi045.stdout:244: writing smithi04526579-11 from 3182305 to 3825969 tid 3 2024-04-26T00:21:31.093 INFO:tasks.rados.rados.0.smithi045.stdout:245: write (excl) oid 2 current snap is 28 2024-04-26T00:21:31.093 INFO:tasks.rados.rados.0.smithi045.stdout:245: seq_num 112 ranges {682384=456830,1721042=423699,2809728=454936,3731357=151027} 2024-04-26T00:21:31.100 INFO:tasks.rados.rados.0.smithi045.stdout:245: writing smithi04526579-2 from 682384 to 1139214 tid 1 2024-04-26T00:21:31.106 INFO:tasks.rados.rados.0.smithi045.stdout:245: writing smithi04526579-2 from 1721042 to 2144741 tid 2 2024-04-26T00:21:31.112 INFO:tasks.rados.rados.0.smithi045.stdout:245: writing smithi04526579-2 from 2809728 to 3264664 tid 3 2024-04-26T00:21:31.116 INFO:tasks.rados.rados.0.smithi045.stdout:245: writing smithi04526579-2 from 3731357 to 3882384 tid 4 2024-04-26T00:21:31.116 INFO:tasks.rados.rados.0.smithi045.stdout:246: write oid 28 current snap is 28 2024-04-26T00:21:31.116 INFO:tasks.rados.rados.0.smithi045.stdout:246: seq_num 113 ranges {699191=532079,1779524=567357,2837151=666208} 2024-04-26T00:21:31.122 INFO:tasks.rados.rados.0.smithi045.stdout:246: writing smithi04526579-28 from 699191 to 1231270 tid 1 2024-04-26T00:21:31.129 INFO:tasks.rados.rados.0.smithi045.stdout:246: writing smithi04526579-28 from 1779524 to 2346881 tid 2 2024-04-26T00:21:31.137 INFO:tasks.rados.rados.0.smithi045.stdout:246: writing smithi04526579-28 from 2837151 to 3503359 tid 3 2024-04-26T00:21:31.137 INFO:tasks.rados.rados.0.smithi045.stdout:247: rollback oid 4 current snap is 28 2024-04-26T00:21:31.137 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 4 to 20 2024-04-26T00:21:31.137 INFO:tasks.rados.rados.0.smithi045.stdout:241: finishing copy_from racing read to smithi04526579-3 2024-04-26T00:21:31.137 INFO:tasks.rados.rados.0.smithi045.stdout:241: finishing copy_from to smithi04526579-3 2024-04-26T00:21:31.137 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 72 (ObjNum 58 snap 6 seq_num 58) dirty exists 2024-04-26T00:21:31.137 INFO:tasks.rados.rados.0.smithi045.stdout:243: finishing write tid 1 to smithi04526579-33 2024-04-26T00:21:31.137 INFO:tasks.rados.rados.0.smithi045.stdout:243: finishing write tid 2 to smithi04526579-33 2024-04-26T00:21:31.137 INFO:tasks.rados.rados.0.smithi045.stdout:243: finishing write tid 3 to smithi04526579-33 2024-04-26T00:21:31.137 INFO:tasks.rados.rados.0.smithi045.stdout:243: finishing write tid 4 to smithi04526579-33 2024-04-26T00:21:31.137 INFO:tasks.rados.rados.0.smithi045.stdout:243: finishing write tid 5 to smithi04526579-33 2024-04-26T00:21:31.138 INFO:tasks.rados.rados.0.smithi045.stdout:243: oid 33 updating version 0 to 142 2024-04-26T00:21:31.138 INFO:tasks.rados.rados.0.smithi045.stdout:243: oid 33 updating version 142 to 145 2024-04-26T00:21:31.138 INFO:tasks.rados.rados.0.smithi045.stdout:243: oid 33 version 145 is already newer than 143 2024-04-26T00:21:31.138 INFO:tasks.rados.rados.0.smithi045.stdout:243: oid 33 version 145 is already newer than 144 2024-04-26T00:21:31.138 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 145 (ObjNum 110 snap 28 seq_num 110) dirty exists 2024-04-26T00:21:31.138 INFO:tasks.rados.rados.0.smithi045.stdout:243: left oid 33 (ObjNum 110 snap 28 seq_num 110) 2024-04-26T00:21:31.138 INFO:tasks.rados.rados.0.smithi045.stdout:244: finishing write tid 1 to smithi04526579-11 2024-04-26T00:21:31.138 INFO:tasks.rados.rados.0.smithi045.stdout:244: finishing write tid 2 to smithi04526579-11 2024-04-26T00:21:31.138 INFO:tasks.rados.rados.0.smithi045.stdout:244: finishing write tid 3 to smithi04526579-11 2024-04-26T00:21:31.142 INFO:tasks.rados.rados.0.smithi045.stdout:244: finishing write tid 4 to smithi04526579-11 2024-04-26T00:21:31.142 INFO:tasks.rados.rados.0.smithi045.stdout:244: finishing write tid 5 to smithi04526579-11 2024-04-26T00:21:31.142 INFO:tasks.rados.rados.0.smithi045.stdout:244: oid 11 updating version 0 to 24 2024-04-26T00:21:31.143 INFO:tasks.rados.rados.0.smithi045.stdout:244: oid 11 updating version 24 to 27 2024-04-26T00:21:31.143 INFO:tasks.rados.rados.0.smithi045.stdout:244: oid 11 version 27 is already newer than 25 2024-04-26T00:21:31.143 INFO:tasks.rados.rados.0.smithi045.stdout:244: oid 11 version 27 is already newer than 26 2024-04-26T00:21:31.143 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 27 (ObjNum 111 snap 28 seq_num 111) dirty exists 2024-04-26T00:21:31.143 INFO:tasks.rados.rados.0.smithi045.stdout:244: left oid 11 (ObjNum 111 snap 28 seq_num 111) 2024-04-26T00:21:31.143 INFO:tasks.rados.rados.0.smithi045.stdout:245: finishing write tid 1 to smithi04526579-2 2024-04-26T00:21:31.143 INFO:tasks.rados.rados.0.smithi045.stdout:245: finishing write tid 2 to smithi04526579-2 2024-04-26T00:21:31.143 INFO:tasks.rados.rados.0.smithi045.stdout:245: finishing write tid 3 to smithi04526579-2 2024-04-26T00:21:31.143 INFO:tasks.rados.rados.0.smithi045.stdout:245: finishing write tid 4 to smithi04526579-2 2024-04-26T00:21:31.143 INFO:tasks.rados.rados.0.smithi045.stdout:245: finishing write tid 5 to smithi04526579-2 2024-04-26T00:21:31.143 INFO:tasks.rados.rados.0.smithi045.stdout:245: finishing write tid 6 to smithi04526579-2 2024-04-26T00:21:31.143 INFO:tasks.rados.rados.0.smithi045.stdout:245: oid 2 updating version 0 to 97 2024-04-26T00:21:31.143 INFO:tasks.rados.rados.0.smithi045.stdout:245: oid 2 updating version 97 to 98 2024-04-26T00:21:31.143 INFO:tasks.rados.rados.0.smithi045.stdout:245: oid 2 updating version 98 to 101 2024-04-26T00:21:31.144 INFO:tasks.rados.rados.0.smithi045.stdout:245: oid 2 version 101 is already newer than 100 2024-04-26T00:21:31.144 INFO:tasks.rados.rados.0.smithi045.stdout:245: oid 2 version 101 is already newer than 99 2024-04-26T00:21:31.144 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 101 (ObjNum 112 snap 28 seq_num 112) dirty exists 2024-04-26T00:21:31.144 INFO:tasks.rados.rados.0.smithi045.stdout:245: left oid 2 (ObjNum 112 snap 28 seq_num 112) 2024-04-26T00:21:31.144 INFO:tasks.rados.rados.0.smithi045.stdout:241: done (5 left) 2024-04-26T00:21:31.144 INFO:tasks.rados.rados.0.smithi045.stdout:243: done (4 left) 2024-04-26T00:21:31.144 INFO:tasks.rados.rados.0.smithi045.stdout:244: done (3 left) 2024-04-26T00:21:31.144 INFO:tasks.rados.rados.0.smithi045.stdout:245: done (2 left) 2024-04-26T00:21:31.144 INFO:tasks.rados.rados.0.smithi045.stdout:248: read oid 13 snap -1 2024-04-26T00:21:31.144 INFO:tasks.rados.rados.0.smithi045.stdout:248: expect deleted 2024-04-26T00:21:31.144 INFO:tasks.rados.rados.0.smithi045.stdout:249: write (excl) oid 42 current snap is 28 2024-04-26T00:21:31.144 INFO:tasks.rados.rados.0.smithi045.stdout:249: seq_num 114 ranges {715998=607328,1838006=427368,2864574=477480,3863835=52163} 2024-04-26T00:21:31.145 INFO:tasks.rados.rados.0.smithi045.stdout:249: writing smithi04526579-42 from 715998 to 1323326 tid 1 2024-04-26T00:21:31.149 INFO:tasks.rados.rados.0.smithi045.stdout:249: writing smithi04526579-42 from 1838006 to 2265374 tid 2 2024-04-26T00:21:31.155 INFO:tasks.rados.rados.0.smithi045.stdout:249: writing smithi04526579-42 from 2864574 to 3342054 tid 3 2024-04-26T00:21:31.158 INFO:tasks.rados.rados.0.smithi045.stdout:249: writing smithi04526579-42 from 3863835 to 3915998 tid 4 2024-04-26T00:21:31.158 INFO:tasks.rados.rados.0.smithi045.stdout:250: snap_remove snap 27 2024-04-26T00:21:31.721 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-04-26T00:21:31.721 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-04-26T00:21:31.722 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-26T00:21:31.722 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-26T00:21:31.725 INFO:tasks.ceph.mon.b:Started 2024-04-26T00:21:31.725 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:21:31.725 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:21:31.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f36340d2640 10 monclient: get_monmap_and_config 2024-04-26T00:21:31.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f36340d2640 10 monclient: build_initial_monmap 2024-04-26T00:21:31.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f36340d2640 10 monclient: monmap: 2024-04-26T00:21:31.857 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:31.857 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:31.857 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:21:31.881456+0000 2024-04-26T00:21:31.857 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:21:31.881456+0000 2024-04-26T00:21:31.857 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:31.857 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:31.858 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:21:31.858 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:21:31.858 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:21:31.858 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:21:31.858 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:21:31.858 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:31.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f36340d2640 1 Processor -- start 2024-04-26T00:21:31.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f36340d2640 1 -- start start 2024-04-26T00:21:31.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f36340d2640 10 monclient: init 2024-04-26T00:21:31.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f36340d2640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:31.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f36340d2640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-04-26T00:21:31.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f36340d2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f362c0a0260 addr v1:172.21.15.45:6789/0 2024-04-26T00:21:31.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f36340d2640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f362c137e40 0x7f362c13a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:31.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f36340d2640 10 monclient(hunting): picked mon.noname-a con 0x7f362c137e40 addr v2:172.21.15.37:3300/0 2024-04-26T00:21:31.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f36340d2640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f362c13a770 0x7f362c13cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:31.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f36340d2640 10 monclient(hunting): picked mon.noname-c con 0x7f362c13a770 addr v2:172.21.15.45:3300/0 2024-04-26T00:21:31.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f36340d2640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f362c0a1130 con 0x7f362c0a0260 2024-04-26T00:21:31.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f36340d2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:31.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f36340d2640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f362c0a0d50 con 0x7f362c137e40 2024-04-26T00:21:31.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f36340d2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:31.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f36340d2640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f362c09ec10 con 0x7f362c13a770 2024-04-26T00:21:31.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f36340d2640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:31.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f36340d2640 10 monclient(hunting): authenticate will time out at 1073.187105s 2024-04-26T00:21:31.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f363166d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f362c137e40 0x7f362c13a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:31.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f363166d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f362c137e40 0x7f362c13a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33704/0 (socket says 172.21.15.37:33704) 2024-04-26T00:21:31.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f363166d640 1 -- 172.21.15.37:0/1094156999 learned_addr learned my addr 172.21.15.37:0/1094156999 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:21:31.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f363166d640 10 monclient(hunting): get_auth_request con 0x7f362c137e40 auth_method 0 2024-04-26T00:21:31.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f363166d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:31.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f363166d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:31.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f363166d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:31.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f363266f640 1 --2- 172.21.15.37:0/1094156999 >> v2:172.21.15.45:3300/0 conn(0x7f362c13a770 0x7f362c13cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:31.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f363266f640 10 monclient(hunting): get_auth_request con 0x7f362c13a770 auth_method 0 2024-04-26T00:21:31.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f363266f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:31.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f363266f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:31.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f363266f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:31.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f363166d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:31.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f363166d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:31.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f363166d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:31.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f363266f640 1 -- 172.21.15.37:0/1094156999 >> v2:172.21.15.45:3300/0 conn(0x7f362c13a770 msgr2=0x7f362c13cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:21:31.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f363266f640 1 -- 172.21.15.37:0/1094156999 >> v2:172.21.15.45:3300/0 conn(0x7f362c13a770 msgr2=0x7f362c13cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:21:31.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f363266f640 1 --2- 172.21.15.37:0/1094156999 >> v2:172.21.15.45:3300/0 conn(0x7f362c13a770 0x7f362c13cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:21:31.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f363266f640 1 --2- 172.21.15.37:0/1094156999 >> v2:172.21.15.45:3300/0 conn(0x7f362c13a770 0x7f362c13cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:21:31.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f363166d640 10 monclient(hunting): handle_auth_done global_id 4515 payload 274 2024-04-26T00:21:31.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f363166d640 1 -- 172.21.15.37:0/1094156999 >> v2:172.21.15.45:3300/0 conn(0x7f362c13a770 msgr2=0x7f362c13cb60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:21:31.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f363166d640 1 --2- 172.21.15.37:0/1094156999 >> v2:172.21.15.45:3300/0 conn(0x7f362c13a770 0x7f362c13cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:31.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f363166d640 1 -- 172.21.15.37:0/1094156999 >> v1:172.21.15.45:6789/0 conn(0x7f362c0a0260 legacy=0x7f362c0a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:21:31.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f363166d640 10 monclient: _finish_hunting 0 2024-04-26T00:21:31.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f363166d640 1 monclient: found mon.noname-a 2024-04-26T00:21:31.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f363166d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:31.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f363166d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:21:31.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.879+0000 7f363166d640 1 -- 172.21.15.37:0/1094156999 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f362c06cff0 con 0x7f362c137e40 2024-04-26T00:21:31.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.883+0000 7f363166d640 1 -- 172.21.15.37:0/1094156999 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f362c137e40 msgr2=0x7f362c13a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:31.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.883+0000 7f363166d640 1 --2- 172.21.15.37:0/1094156999 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f362c137e40 0x7f362c13a230 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f362c0a1a60 tx=0x7f362801c1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8c65c626a9fe6b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:31.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f3630e6c640 1 -- 172.21.15.37:0/1094156999 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36280b0070 con 0x7f362c137e40 2024-04-26T00:21:31.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f3630e6c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:31.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f3630e6c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:21:31.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f3630e6c640 10 monclient: dump: 2024-04-26T00:21:31.871 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:31.871 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:31.871 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:31.871 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:31.871 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:31.871 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:31.871 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:31.871 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:31.871 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:31.871 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:31.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f3630e6c640 10 monclient: _finish_auth 0 2024-04-26T00:21:31.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f3630e6c640 10 monclient: _check_auth_tickets 2024-04-26T00:21:31.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f3630e6c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:31.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f36340d2640 5 monclient: authenticate success, global_id 4515 2024-04-26T00:21:31.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f36340d2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:21:31.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f3630e6c640 1 -- 172.21.15.37:0/1094156999 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36280ab600 con 0x7f362c137e40 2024-04-26T00:21:31.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f3630e6c640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:31.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f3630e6c640 1 -- 172.21.15.37:0/1094156999 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36280ab9d0 con 0x7f362c137e40 2024-04-26T00:21:31.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f36340d2640 10 monclient: get_monmap_and_config success 2024-04-26T00:21:31.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f3630e6c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:31.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f36340d2640 4 set_mon_vals no callback set 2024-04-26T00:21:31.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f3630e6c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:21:31.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f3630e6c640 10 monclient: dump: 2024-04-26T00:21:31.872 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:31.872 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:31.872 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:31.872 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:31.872 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:31.872 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:31.872 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:31.873 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:31.873 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:31.873 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:31.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f36340d2640 10 monclient: shutdown 2024-04-26T00:21:31.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f36340d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:31.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f36340d2640 1 -- 172.21.15.37:0/1094156999 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f362c137e40 msgr2=0x7f362c13a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:31.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f36340d2640 1 --2- 172.21.15.37:0/1094156999 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f362c137e40 0x7f362c13a230 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f362c0a1a60 tx=0x7f362801c1c0 comp rx=0 tx=0).stop 2024-04-26T00:21:31.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f36340d2640 1 -- 172.21.15.37:0/1094156999 shutdown_connections 2024-04-26T00:21:31.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f36340d2640 1 --2- 172.21.15.37:0/1094156999 >> v2:172.21.15.45:3300/0 conn(0x7f362c13a770 0x7f362c13cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:31.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f36340d2640 1 --2- 172.21.15.37:0/1094156999 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f362c137e40 0x7f362c13a230 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:31.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f36340d2640 1 -- 172.21.15.37:0/1094156999 >> 172.21.15.37:0/1094156999 conn(0x7f362c0a1ad0 msgr2=0x7f362c0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:31.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f36340d2640 1 -- 172.21.15.37:0/1094156999 shutdown_connections 2024-04-26T00:21:31.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f36340d2640 1 -- 172.21.15.37:0/1094156999 wait complete. 2024-04-26T00:21:31.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f36340d2640 10 monclient: build_initial_monmap 2024-04-26T00:21:31.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f36340d2640 10 monclient: monmap: 2024-04-26T00:21:31.874 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:31.874 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:21:31.874 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:21:31.874 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:21:31.874 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:31.874 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:21:31.874 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:21:31.874 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:21:31.874 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:21:31.874 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:31.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f36340d2640 1 Processor -- start 2024-04-26T00:21:31.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f36340d2640 1 -- start start 2024-04-26T00:21:31.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f36340d2640 10 monclient: init 2024-04-26T00:21:31.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f36340d2640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:31.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f36340d2640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:21:31.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f36340d2640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f362c0a0260 0x7f362c136fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:31.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f36340d2640 10 monclient(hunting): picked mon.noname-c con 0x7f362c0a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:21:31.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f36340d2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f362c137e40 0x7f362c1374e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:31.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f36340d2640 10 monclient(hunting): picked mon.noname-b con 0x7f362c137e40 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:21:31.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f36340d2640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f362c13a770 0x7f362c1cecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:31.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f36340d2640 10 monclient(hunting): picked mon.noname-a con 0x7f362c13a770 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:21:31.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f36340d2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:31.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f36340d2640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f362c09f0f0 con 0x7f362c13a770 2024-04-26T00:21:31.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f36340d2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:31.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f36340d2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f362c0a00a0 con 0x7f362c137e40 2024-04-26T00:21:31.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f36340d2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:31.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f36340d2640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f362c09f4c0 con 0x7f362c0a0260 2024-04-26T00:21:31.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f36340d2640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:31.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f36340d2640 10 monclient(hunting): authenticate will time out at 1073.194363s 2024-04-26T00:21:31.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f363266f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f362c13a770 0x7f362c1cecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:31.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f3631e6e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f362c0a0260 0x7f362c136fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:31.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f3631e6e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f362c0a0260 0x7f362c136fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:43428/0 (socket says 172.21.15.37:43428) 2024-04-26T00:21:31.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f363266f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f362c13a770 0x7f362c1cecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33710/0 (socket says 172.21.15.37:33710) 2024-04-26T00:21:31.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f3631e6e640 1 -- 172.21.15.37:0/2684644493 learned_addr learned my addr 172.21.15.37:0/2684644493 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:21:31.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f3631e6e640 10 monclient(hunting): get_auth_request con 0x7f362c0a0260 auth_method 0 2024-04-26T00:21:31.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f3631e6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:31.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f3631e6e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:31.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f3631e6e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:31.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f363266f640 10 monclient(hunting): get_auth_request con 0x7f362c13a770 auth_method 0 2024-04-26T00:21:31.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f363266f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:31.877 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f363266f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:31.877 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f363266f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:31.877 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f3631e6e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:31.877 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f3631e6e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:31.877 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f3631e6e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:31.877 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f363266f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:31.877 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f363266f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:31.877 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f363266f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:31.877 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f363166d640 1 --2- 172.21.15.37:0/2684644493 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f362c137e40 0x7f362c1374e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:31.877 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f363166d640 10 monclient(hunting): get_auth_request con 0x7f362c137e40 auth_method 0 2024-04-26T00:21:31.877 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f363166d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:31.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f363166d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:31.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f363166d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:31.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f3631e6e640 10 monclient(hunting): handle_auth_done global_id 4505 payload 824 2024-04-26T00:21:31.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f3631e6e640 1 -- 172.21.15.37:0/2684644493 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f362c137e40 msgr2=0x7f362c1374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:31.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f3631e6e640 1 --2- 172.21.15.37:0/2684644493 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f362c137e40 0x7f362c1374e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:31.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f3631e6e640 1 -- 172.21.15.37:0/2684644493 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f362c13a770 msgr2=0x7f362c1cecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:31.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f3631e6e640 1 --2- 172.21.15.37:0/2684644493 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f362c13a770 0x7f362c1cecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:31.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f3631e6e640 10 monclient: _finish_hunting 0 2024-04-26T00:21:31.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f3631e6e640 1 monclient: found mon.noname-c 2024-04-26T00:21:31.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f3631e6e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:31.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f3631e6e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:21:31.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f3631e6e640 1 -- 172.21.15.37:0/2684644493 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f362c122500 con 0x7f362c0a0260 2024-04-26T00:21:31.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.887+0000 7f363266f640 1 --2- 172.21.15.37:0/2684644493 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f362c13a770 0x7f362c1cecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:21:31.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.891+0000 7f3631e6e640 1 -- 172.21.15.37:0/2684644493 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f362c0a0260 msgr2=0x7f362c136fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:31.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.891+0000 7f3631e6e640 1 --2- 172.21.15.37:0/2684644493 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f362c0a0260 0x7f362c136fa0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f361c00c450 tx=0x7f361c00c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:31.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.891+0000 7f36227fc640 1 -- 172.21.15.37:0/2684644493 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f361c012a40 con 0x7f362c0a0260 2024-04-26T00:21:31.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.891+0000 7f36227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:31.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.891+0000 7f36227fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:21:31.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.891+0000 7f36227fc640 10 monclient: dump: 2024-04-26T00:21:31.879 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:31.879 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:31.879 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:31.879 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:31.879 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:31.879 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:31.879 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:31.879 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:31.879 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:31.879 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:31.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.891+0000 7f36227fc640 10 monclient: _finish_auth 0 2024-04-26T00:21:31.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.891+0000 7f36227fc640 10 monclient: _check_auth_tickets 2024-04-26T00:21:31.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.891+0000 7f36227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:31.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.891+0000 7f36340d2640 5 monclient: authenticate success, global_id 4505 2024-04-26T00:21:31.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.891+0000 7f36340d2640 10 monclient: _renew_subs 2024-04-26T00:21:31.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.891+0000 7f36340d2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:21:31.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.891+0000 7f36340d2640 1 -- 172.21.15.37:0/2684644493 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f362c09e1a0 con 0x7f362c0a0260 2024-04-26T00:21:31.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.891+0000 7f36340d2640 10 monclient: _renew_subs 2024-04-26T00:21:31.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.891+0000 7f36340d2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:21:31.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.891+0000 7f36340d2640 1 -- 172.21.15.37:0/2684644493 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f362c1230b0 con 0x7f362c0a0260 2024-04-26T00:21:31.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.891+0000 7f36227fc640 1 -- 172.21.15.37:0/2684644493 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f361c017070 con 0x7f362c0a0260 2024-04-26T00:21:31.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.891+0000 7f36227fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:31.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.891+0000 7f36227fc640 1 -- 172.21.15.37:0/2684644493 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f361c012e40 con 0x7f362c0a0260 2024-04-26T00:21:31.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.891+0000 7f36227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:31.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.891+0000 7f36227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:21:31.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.891+0000 7f36237fe640 4 set_mon_vals no callback set 2024-04-26T00:21:31.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.891+0000 7f36227fc640 10 monclient: dump: 2024-04-26T00:21:31.880 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:31.881 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:31.881 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:31.881 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:31.881 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:31.881 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:31.881 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:31.881 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:31.881 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:31.881 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:31.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.891+0000 7f36340d2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:31.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.891+0000 7f36340d2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:31.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.891+0000 7f36340d2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:21:31.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.891+0000 7f36340d2640 1 -- 172.21.15.37:0/2684644493 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3614005090 con 0x7f362c0a0260 2024-04-26T00:21:31.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.891+0000 7f36227fc640 1 -- 172.21.15.37:0/2684644493 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f361c01c460 con 0x7f362c0a0260 2024-04-26T00:21:31.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.891+0000 7f36227fc640 1 --2- 172.21.15.37:0/2684644493 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3610035e80 0x7f3610038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:31.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.891+0000 7f36227fc640 1 -- 172.21.15.37:0/2684644493 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(67..67 src has 1..67) ==== 3931+0+0 (secure 0 0 0) 0x7f361c048810 con 0x7f362c0a0260 2024-04-26T00:21:31.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.891+0000 7f363166d640 1 --2- 172.21.15.37:0/2684644493 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3610035e80 0x7f3610038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:31.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.891+0000 7f363166d640 10 monclient: get_auth_request con 0x7f3610035e80 auth_method 0 2024-04-26T00:21:31.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.895+0000 7f363166d640 1 --2- 172.21.15.37:0/2684644493 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3610035e80 0x7f3610038340 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f362c0a2ad0 tx=0x7f3628008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:31.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.895+0000 7f36227fc640 1 -- 172.21.15.37:0/2684644493 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f361c047560 con 0x7f362c0a0260 2024-04-26T00:21:31.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.895+0000 7f36227fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:31.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:31.895+0000 7f36227fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:21:31.982 INFO:tasks.rados.rados.0.smithi045.stdout:250: done (4 left) 2024-04-26T00:21:31.982 INFO:tasks.rados.rados.0.smithi045.stdout:251: write (excl) oid 6 current snap is 28 2024-04-26T00:21:31.982 INFO:tasks.rados.rados.0.smithi045.stdout:251: seq_num 115 ranges {732805=798930,2296488=571026,3291997=405105} 2024-04-26T00:21:31.990 INFO:tasks.rados.rados.0.smithi045.stdout:251: writing smithi04526579-6 from 732805 to 1531735 tid 1 2024-04-26T00:21:31.999 INFO:tasks.rados.rados.0.smithi045.stdout:251: writing smithi04526579-6 from 2296488 to 2867514 tid 2 2024-04-26T00:21:32.004 INFO:tasks.rados.rados.0.smithi045.stdout:251: writing smithi04526579-6 from 3291997 to 3697102 tid 3 2024-04-26T00:21:32.004 INFO:tasks.rados.rados.0.smithi045.stdout:252: read oid 16 snap -1 2024-04-26T00:21:32.004 INFO:tasks.rados.rados.0.smithi045.stdout:252: expect (ObjNum 15 snap 0 seq_num 15) 2024-04-26T00:21:32.005 INFO:tasks.rados.rados.0.smithi045.stdout:253: snap_remove snap 22 2024-04-26T00:21:32.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:32.127+0000 7f36340d2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:21:32.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:32.127+0000 7f36340d2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:21:32.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:32.127+0000 7f36340d2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:21:32.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:32.127+0000 7f36340d2640 1 -- 172.21.15.37:0/2684644493 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3614005380 con 0x7f362c0a0260 2024-04-26T00:21:32.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:32.127+0000 7f36227fc640 1 -- 172.21.15.37:0/2684644493 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f361c01b3c0 con 0x7f362c0a0260 2024-04-26T00:21:32.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:32.127+0000 7f36227fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:21:32.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:32.127+0000 7f36227fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:21:32.107 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":16,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:21:32.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:32.131+0000 7f36340d2640 1 -- 172.21.15.37:0/2684644493 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3610035e80 msgr2=0x7f3610038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:32.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:32.131+0000 7f36340d2640 1 --2- 172.21.15.37:0/2684644493 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3610035e80 0x7f3610038340 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f362c0a2ad0 tx=0x7f3628008040 comp rx=0 tx=0).stop 2024-04-26T00:21:32.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:32.131+0000 7f36340d2640 10 monclient: shutdown 2024-04-26T00:21:32.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:32.131+0000 7f36340d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:32.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:32.131+0000 7f36340d2640 1 -- 172.21.15.37:0/2684644493 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f362c0a0260 msgr2=0x7f362c136fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:32.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:32.131+0000 7f36340d2640 1 --2- 172.21.15.37:0/2684644493 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f362c0a0260 0x7f362c136fa0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f361c00c450 tx=0x7f361c00c910 comp rx=0 tx=0).stop 2024-04-26T00:21:32.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:32.131+0000 7f36340d2640 1 -- 172.21.15.37:0/2684644493 shutdown_connections 2024-04-26T00:21:32.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:32.131+0000 7f36340d2640 1 --2- 172.21.15.37:0/2684644493 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3610035e80 0x7f3610038340 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:32.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:32.131+0000 7f36340d2640 1 --2- 172.21.15.37:0/2684644493 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f362c13a770 0x7f362c1cecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:32.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:32.131+0000 7f36340d2640 1 --2- 172.21.15.37:0/2684644493 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f362c137e40 0x7f362c1374e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:32.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:32.131+0000 7f36340d2640 1 --2- 172.21.15.37:0/2684644493 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f362c0a0260 0x7f362c136fa0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:32.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:32.131+0000 7f36340d2640 1 -- 172.21.15.37:0/2684644493 >> 172.21.15.37:0/2684644493 conn(0x7f362c0a1ad0 msgr2=0x7f362c0a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:32.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:32.131+0000 7f36340d2640 1 -- 172.21.15.37:0/2684644493 shutdown_connections 2024-04-26T00:21:32.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:32.131+0000 7f36340d2640 1 -- 172.21.15.37:0/2684644493 wait complete. 2024-04-26T00:21:32.909 INFO:tasks.rados.rados.0.smithi045.stdout:253: done (6 left) 2024-04-26T00:21:32.909 INFO:tasks.rados.rados.0.smithi045.stdout:254: copy_from oid 50 from oid 25 current snap is 28 2024-04-26T00:21:32.909 INFO:tasks.rados.rados.0.smithi045.stdout:246: finishing write tid 1 to smithi04526579-28 2024-04-26T00:21:32.909 INFO:tasks.rados.rados.0.smithi045.stdout:246: finishing write tid 2 to smithi04526579-28 2024-04-26T00:21:32.909 INFO:tasks.rados.rados.0.smithi045.stdout:246: finishing write tid 3 to smithi04526579-28 2024-04-26T00:21:32.909 INFO:tasks.rados.rados.0.smithi045.stdout:246: finishing write tid 4 to smithi04526579-28 2024-04-26T00:21:32.909 INFO:tasks.rados.rados.0.smithi045.stdout:246: finishing write tid 5 to smithi04526579-28 2024-04-26T00:21:32.909 INFO:tasks.rados.rados.0.smithi045.stdout:246: oid 28 updating version 0 to 65 2024-04-26T00:21:32.909 INFO:tasks.rados.rados.0.smithi045.stdout:246: oid 28 version 65 is already newer than 64 2024-04-26T00:21:32.910 INFO:tasks.rados.rados.0.smithi045.stdout:246: oid 28 updating version 65 to 66 2024-04-26T00:21:32.910 INFO:tasks.rados.rados.0.smithi045.stdout:246: oid 28 version 66 is already newer than 63 2024-04-26T00:21:32.910 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 66 (ObjNum 113 snap 28 seq_num 113) dirty exists 2024-04-26T00:21:32.910 INFO:tasks.rados.rados.0.smithi045.stdout:246: left oid 28 (ObjNum 113 snap 28 seq_num 113) 2024-04-26T00:21:32.910 INFO:tasks.rados.rados.0.smithi045.stdout:247: finishing rollback tid 0 to smithi04526579-4 2024-04-26T00:21:32.910 INFO:tasks.rados.rados.0.smithi045.stdout:247: finishing rollback tid 1 to smithi04526579-4 2024-04-26T00:21:32.910 INFO:tasks.rados.rados.0.smithi045.stdout:247: finishing rollback tid 2 to smithi04526579-4 2024-04-26T00:21:32.910 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 105 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-26T00:21:32.910 INFO:tasks.rados.rados.0.smithi045.stdout:249: finishing write tid 1 to smithi04526579-42 2024-04-26T00:21:32.910 INFO:tasks.rados.rados.0.smithi045.stdout:249: finishing write tid 2 to smithi04526579-42 2024-04-26T00:21:32.910 INFO:tasks.rados.rados.0.smithi045.stdout:249: finishing write tid 3 to smithi04526579-42 2024-04-26T00:21:32.910 INFO:tasks.rados.rados.0.smithi045.stdout:249: finishing write tid 4 to smithi04526579-42 2024-04-26T00:21:32.910 INFO:tasks.rados.rados.0.smithi045.stdout:249: finishing write tid 5 to smithi04526579-42 2024-04-26T00:21:32.911 INFO:tasks.rados.rados.0.smithi045.stdout:249: finishing write tid 6 to smithi04526579-42 2024-04-26T00:21:32.911 INFO:tasks.rados.rados.0.smithi045.stdout:249: oid 42 updating version 0 to 54 2024-04-26T00:21:32.911 INFO:tasks.rados.rados.0.smithi045.stdout:249: oid 42 updating version 54 to 55 2024-04-26T00:21:32.911 INFO:tasks.rados.rados.0.smithi045.stdout:249: oid 42 updating version 55 to 56 2024-04-26T00:21:32.911 INFO:tasks.rados.rados.0.smithi045.stdout:249: oid 42 updating version 56 to 57 2024-04-26T00:21:32.911 INFO:tasks.rados.rados.0.smithi045.stdout:249: oid 42 version 57 is already newer than 53 2024-04-26T00:21:32.911 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 57 (ObjNum 114 snap 28 seq_num 114) dirty exists 2024-04-26T00:21:32.911 INFO:tasks.rados.rados.0.smithi045.stdout:249: left oid 42 (ObjNum 114 snap 28 seq_num 114) 2024-04-26T00:21:32.911 INFO:tasks.rados.rados.0.smithi045.stdout:251: finishing write tid 1 to smithi04526579-6 2024-04-26T00:21:32.911 INFO:tasks.rados.rados.0.smithi045.stdout:251: finishing write tid 2 to smithi04526579-6 2024-04-26T00:21:32.911 INFO:tasks.rados.rados.0.smithi045.stdout:251: finishing write tid 3 to smithi04526579-6 2024-04-26T00:21:32.911 INFO:tasks.rados.rados.0.smithi045.stdout:251: finishing write tid 4 to smithi04526579-6 2024-04-26T00:21:32.911 INFO:tasks.rados.rados.0.smithi045.stdout:252: expect (ObjNum 15 snap 0 seq_num 15) 2024-04-26T00:21:32.989 INFO:tasks.rados.rados.0.smithi045.stdout:251: finishing write tid 5 to smithi04526579-6 2024-04-26T00:21:32.989 INFO:tasks.rados.rados.0.smithi045.stdout:251: oid 6 updating version 0 to 50 2024-04-26T00:21:32.989 INFO:tasks.rados.rados.0.smithi045.stdout:251: oid 6 version 50 is already newer than 49 2024-04-26T00:21:32.990 INFO:tasks.rados.rados.0.smithi045.stdout:251: oid 6 updating version 50 to 51 2024-04-26T00:21:32.990 INFO:tasks.rados.rados.0.smithi045.stdout:251: oid 6 version 51 is already newer than 48 2024-04-26T00:21:32.990 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 51 (ObjNum 115 snap 28 seq_num 115) dirty exists 2024-04-26T00:21:32.990 INFO:tasks.rados.rados.0.smithi045.stdout:251: left oid 6 (ObjNum 115 snap 28 seq_num 115) 2024-04-26T00:21:32.990 INFO:tasks.rados.rados.0.smithi045.stdout:254: finishing copy_from racing read to smithi04526579-50 2024-04-26T00:21:32.990 INFO:tasks.rados.rados.0.smithi045.stdout:254: finishing copy_from to smithi04526579-50 2024-04-26T00:21:32.991 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 118 (ObjNum 72 snap 11 seq_num 72) dirty exists 2024-04-26T00:21:32.991 INFO:tasks.rados.rados.0.smithi045.stdout:246: done (6 left) 2024-04-26T00:21:32.991 INFO:tasks.rados.rados.0.smithi045.stdout:247: done (5 left) 2024-04-26T00:21:32.991 INFO:tasks.rados.rados.0.smithi045.stdout:248: done (4 left) 2024-04-26T00:21:32.991 INFO:tasks.rados.rados.0.smithi045.stdout:249: done (3 left) 2024-04-26T00:21:32.992 INFO:tasks.rados.rados.0.smithi045.stdout:251: done (2 left) 2024-04-26T00:21:32.992 INFO:tasks.rados.rados.0.smithi045.stdout:252: done (1 left) 2024-04-26T00:21:32.992 INFO:tasks.rados.rados.0.smithi045.stdout:254: done (0 left) 2024-04-26T00:21:32.992 INFO:tasks.rados.rados.0.smithi045.stdout:255: read oid 5 snap 25 2024-04-26T00:21:32.992 INFO:tasks.rados.rados.0.smithi045.stdout:255: expect (ObjNum 88 snap 21 seq_num 88) 2024-04-26T00:21:32.992 INFO:tasks.rados.rados.0.smithi045.stdout:256: write (excl) oid 15 current snap is 28 2024-04-26T00:21:32.992 INFO:tasks.rados.rados.0.smithi045.stdout:256: seq_num 116 ranges {749612=474179,1954970=431037,2919420=616377} 2024-04-26T00:21:32.995 INFO:tasks.rados.rados.0.smithi045.stdout:256: writing smithi04526579-15 from 749612 to 1223791 tid 1 2024-04-26T00:21:33.002 INFO:tasks.rados.rados.0.smithi045.stdout:256: writing smithi04526579-15 from 1954970 to 2386007 tid 2 2024-04-26T00:21:33.009 INFO:tasks.rados.rados.0.smithi045.stdout:256: writing smithi04526579-15 from 2919420 to 3535797 tid 3 2024-04-26T00:21:33.009 INFO:tasks.rados.rados.0.smithi045.stdout:257: write oid 41 current snap is 28 2024-04-26T00:21:33.009 INFO:tasks.rados.rados.0.smithi045.stdout:257: seq_num 117 ranges {766419=549428,2013452=574695,3346843=427649} 2024-04-26T00:21:33.016 INFO:tasks.rados.rados.0.smithi045.stdout:257: writing smithi04526579-41 from 766419 to 1315847 tid 1 2024-04-26T00:21:33.024 INFO:tasks.rados.rados.0.smithi045.stdout:257: writing smithi04526579-41 from 2013452 to 2588147 tid 2 2024-04-26T00:21:33.031 INFO:tasks.rados.rados.0.smithi045.stdout:257: writing smithi04526579-41 from 3346843 to 3774492 tid 3 2024-04-26T00:21:33.031 INFO:tasks.rados.rados.0.smithi045.stdout:258: copy_from oid 47 from oid 1 current snap is 28 2024-04-26T00:21:33.031 INFO:tasks.rados.rados.0.smithi045.stdout:256: finishing write tid 1 to smithi04526579-15 2024-04-26T00:21:33.031 INFO:tasks.rados.rados.0.smithi045.stdout:255: expect (ObjNum 88 snap 21 seq_num 88) 2024-04-26T00:21:33.202 INFO:tasks.rados.rados.0.smithi045.stdout:256: finishing write tid 2 to smithi04526579-15 2024-04-26T00:21:33.202 INFO:tasks.rados.rados.0.smithi045.stdout:256: finishing write tid 3 to smithi04526579-15 2024-04-26T00:21:33.202 INFO:tasks.rados.rados.0.smithi045.stdout:256: finishing write tid 4 to smithi04526579-15 2024-04-26T00:21:33.202 INFO:tasks.rados.rados.0.smithi045.stdout:256: finishing write tid 5 to smithi04526579-15 2024-04-26T00:21:33.202 INFO:tasks.rados.rados.0.smithi045.stdout:256: oid 15 updating version 0 to 77 2024-04-26T00:21:33.202 INFO:tasks.rados.rados.0.smithi045.stdout:256: oid 15 updating version 77 to 78 2024-04-26T00:21:33.202 INFO:tasks.rados.rados.0.smithi045.stdout:256: oid 15 version 78 is already newer than 76 2024-04-26T00:21:33.202 INFO:tasks.rados.rados.0.smithi045.stdout:256: oid 15 updating version 78 to 79 2024-04-26T00:21:33.202 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 79 (ObjNum 116 snap 28 seq_num 116) dirty exists 2024-04-26T00:21:33.202 INFO:tasks.rados.rados.0.smithi045.stdout:256: left oid 15 (ObjNum 116 snap 28 seq_num 116) 2024-04-26T00:21:33.203 INFO:tasks.rados.rados.0.smithi045.stdout:257: finishing write tid 1 to smithi04526579-41 2024-04-26T00:21:33.203 INFO:tasks.rados.rados.0.smithi045.stdout:257: finishing write tid 2 to smithi04526579-41 2024-04-26T00:21:33.203 INFO:tasks.rados.rados.0.smithi045.stdout:257: finishing write tid 3 to smithi04526579-41 2024-04-26T00:21:33.203 INFO:tasks.rados.rados.0.smithi045.stdout:257: finishing write tid 4 to smithi04526579-41 2024-04-26T00:21:33.203 INFO:tasks.rados.rados.0.smithi045.stdout:257: finishing write tid 5 to smithi04526579-41 2024-04-26T00:21:33.203 INFO:tasks.rados.rados.0.smithi045.stdout:257: oid 41 updating version 0 to 81 2024-04-26T00:21:33.203 INFO:tasks.rados.rados.0.smithi045.stdout:257: oid 41 updating version 81 to 83 2024-04-26T00:21:33.203 INFO:tasks.rados.rados.0.smithi045.stdout:257: oid 41 version 83 is already newer than 82 2024-04-26T00:21:33.203 INFO:tasks.rados.rados.0.smithi045.stdout:257: oid 41 version 83 is already newer than 80 2024-04-26T00:21:33.203 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 83 (ObjNum 117 snap 28 seq_num 117) dirty exists 2024-04-26T00:21:33.203 INFO:tasks.rados.rados.0.smithi045.stdout:257: left oid 41 (ObjNum 117 snap 28 seq_num 117) 2024-04-26T00:21:33.203 INFO:tasks.rados.rados.0.smithi045.stdout:258: finishing copy_from to smithi04526579-47 2024-04-26T00:21:33.203 INFO:tasks.rados.rados.0.smithi045.stdout:258: got expected ENOENT (src dne) 2024-04-26T00:21:33.203 INFO:tasks.rados.rados.0.smithi045.stdout:258: finishing copy_from racing read to smithi04526579-47 2024-04-26T00:21:33.204 INFO:tasks.rados.rados.0.smithi045.stdout:255: done (3 left) 2024-04-26T00:21:33.204 INFO:tasks.rados.rados.0.smithi045.stdout:256: done (2 left) 2024-04-26T00:21:33.204 INFO:tasks.rados.rados.0.smithi045.stdout:257: done (1 left) 2024-04-26T00:21:33.204 INFO:tasks.rados.rados.0.smithi045.stdout:258: done (0 left) 2024-04-26T00:21:33.204 INFO:tasks.rados.rados.0.smithi045.stdout:259: read oid 25 snap -1 2024-04-26T00:21:33.204 INFO:tasks.rados.rados.0.smithi045.stdout:259: expect (ObjNum 72 snap 11 seq_num 72) 2024-04-26T00:21:33.204 INFO:tasks.rados.rados.0.smithi045.stdout:260: write oid 43 current snap is 28 2024-04-26T00:21:33.204 INFO:tasks.rados.rados.0.smithi045.stdout:260: seq_num 118 ranges {783226=624677,2071934=434706,2974266=638921} 2024-04-26T00:21:33.212 INFO:tasks.rados.rados.0.smithi045.stdout:260: writing smithi04526579-43 from 783226 to 1407903 tid 1 2024-04-26T00:21:33.220 INFO:tasks.rados.rados.0.smithi045.stdout:260: writing smithi04526579-43 from 2071934 to 2506640 tid 2 2024-04-26T00:21:33.229 INFO:tasks.rados.rados.0.smithi045.stdout:260: writing smithi04526579-43 from 2974266 to 3613187 tid 3 2024-04-26T00:21:33.229 INFO:tasks.rados.rados.0.smithi045.stdout:261: write (excl) oid 44 current snap is 28 2024-04-26T00:21:33.229 INFO:tasks.rados.rados.0.smithi045.stdout:261: seq_num 119 ranges {400033=699926,1614063=385970} 2024-04-26T00:21:33.238 INFO:tasks.rados.rados.0.smithi045.stdout:261: writing smithi04526579-44 from 400033 to 1099959 tid 1 2024-04-26T00:21:33.245 INFO:tasks.rados.rados.0.smithi045.stdout:261: writing smithi04526579-44 from 1614063 to 2000033 tid 2 2024-04-26T00:21:33.245 INFO:tasks.rados.rados.0.smithi045.stdout:262: snap_remove snap 16 2024-04-26T00:21:33.993 INFO:tasks.rados.rados.0.smithi045.stdout:262: done (3 left) 2024-04-26T00:21:33.993 INFO:tasks.rados.rados.0.smithi045.stdout:263: snap_remove snap 25 2024-04-26T00:21:35.007 INFO:tasks.rados.rados.0.smithi045.stdout:263: done (3 left) 2024-04-26T00:21:35.007 INFO:tasks.rados.rados.0.smithi045.stdout:264: rollback oid 8 current snap is 28 2024-04-26T00:21:35.007 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 8 to 18 2024-04-26T00:21:35.008 INFO:tasks.rados.rados.0.smithi045.stdout:259: expect (ObjNum 72 snap 11 seq_num 72) 2024-04-26T00:21:35.126 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:21:35.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65bbcae640 10 monclient: get_monmap_and_config 2024-04-26T00:21:35.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65bbcae640 10 monclient: build_initial_monmap 2024-04-26T00:21:35.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65bbcae640 10 monclient: monmap: 2024-04-26T00:21:35.260 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:35.260 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:35.260 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:21:35.284295+0000 2024-04-26T00:21:35.260 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:21:35.284295+0000 2024-04-26T00:21:35.260 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:35.260 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:35.260 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:21:35.260 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:21:35.260 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:21:35.261 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:21:35.261 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:21:35.261 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:35.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65bbcae640 1 Processor -- start 2024-04-26T00:21:35.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65bbcae640 1 -- start start 2024-04-26T00:21:35.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65bbcae640 10 monclient: init 2024-04-26T00:21:35.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65bbcae640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:35.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65bbcae640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-04-26T00:21:35.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65bbcae640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f65b40a0260 0x7f65b4072e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:35.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65bbcae640 10 monclient(hunting): picked mon.noname-b con 0x7f65b40a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:21:35.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65bbcae640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f65b40a0640 addr v1:172.21.15.37:6789/0 2024-04-26T00:21:35.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65bbcae640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f65b41389e0 0x7f65b413add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:35.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65bbcae640 10 monclient(hunting): picked mon.noname-c con 0x7f65b41389e0 addr v2:172.21.15.45:3300/0 2024-04-26T00:21:35.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65bbcae640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65b40a12e0 con 0x7f65b40a0640 2024-04-26T00:21:35.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65bbcae640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:35.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65bbcae640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f65b406d050 con 0x7f65b41389e0 2024-04-26T00:21:35.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65bbcae640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:35.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65bbcae640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f65b409dcb0 con 0x7f65b40a0260 2024-04-26T00:21:35.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65bbcae640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:35.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65bbcae640 10 monclient(hunting): authenticate will time out at 1076.589978s 2024-04-26T00:21:35.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65b9a4a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f65b40a0260 0x7f65b4072e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:35.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65b9a4a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f65b40a0260 0x7f65b4072e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:43434/0 (socket says 172.21.15.37:43434) 2024-04-26T00:21:35.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65b9a4a640 1 -- 172.21.15.37:0/1462436444 learned_addr learned my addr 172.21.15.37:0/1462436444 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:21:35.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65b9a4a640 10 monclient(hunting): get_auth_request con 0x7f65b40a0260 auth_method 0 2024-04-26T00:21:35.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65b9a4a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:35.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65b9a4a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:35.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65b9a4a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:35.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65b9a4a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:35.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65b9a4a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:35.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65b9a4a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:35.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65ba24b640 1 --2- 172.21.15.37:0/1462436444 >> v2:172.21.15.45:3300/0 conn(0x7f65b41389e0 0x7f65b413add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:35.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65ba24b640 10 monclient(hunting): get_auth_request con 0x7f65b41389e0 auth_method 0 2024-04-26T00:21:35.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65ba24b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:35.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65ba24b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:35.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65ba24b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:35.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65b9a4a640 10 monclient(hunting): handle_auth_done global_id 4508 payload 274 2024-04-26T00:21:35.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65b9a4a640 1 -- 172.21.15.37:0/1462436444 >> v2:172.21.15.45:3300/0 conn(0x7f65b41389e0 msgr2=0x7f65b413add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:21:35.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65b9a4a640 1 --2- 172.21.15.37:0/1462436444 >> v2:172.21.15.45:3300/0 conn(0x7f65b41389e0 0x7f65b413add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:35.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65b9a4a640 1 -- 172.21.15.37:0/1462436444 >> v1:172.21.15.37:6789/0 conn(0x7f65b40a0640 legacy=0x7f65b40a0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:21:35.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65b9a4a640 10 monclient: _finish_hunting 0 2024-04-26T00:21:35.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65b9a4a640 1 monclient: found mon.noname-b 2024-04-26T00:21:35.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65b9a4a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:35.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65b9a4a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:21:35.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.283+0000 7f65b9a4a640 1 -- 172.21.15.37:0/1462436444 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65b40a16f0 con 0x7f65b40a0260 2024-04-26T00:21:35.267 INFO:tasks.rados.rados.0.smithi045.stdout:260: finishing write tid 1 to smithi04526579-43 2024-04-26T00:21:35.267 INFO:tasks.rados.rados.0.smithi045.stdout:260: finishing write tid 2 to smithi04526579-43 2024-04-26T00:21:35.267 INFO:tasks.rados.rados.0.smithi045.stdout:259: done (3 left) 2024-04-26T00:21:35.267 INFO:tasks.rados.rados.0.smithi045.stdout:265: write (excl) oid 29 current snap is 28 2024-04-26T00:21:35.267 INFO:tasks.rados.rados.0.smithi045.stdout:265: seq_num 120 ranges {416840=775175,1672545=344295} 2024-04-26T00:21:35.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.287+0000 7f65b9a4a640 1 -- 172.21.15.37:0/1462436444 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f65b40a0260 msgr2=0x7f65b4072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:35.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.287+0000 7f65b9a4a640 1 --2- 172.21.15.37:0/1462436444 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f65b40a0260 0x7f65b4072e10 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f65a800bb10 tx=0x7f65a801eb40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4d705908e03f814 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:35.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.287+0000 7f65b8a48640 1 -- 172.21.15.37:0/1462436444 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a80b4070 con 0x7f65b40a0260 2024-04-26T00:21:35.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.287+0000 7f65b8a48640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:35.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.287+0000 7f65b8a48640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:21:35.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.287+0000 7f65b8a48640 10 monclient: dump: 2024-04-26T00:21:35.271 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:35.271 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:35.271 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:35.272 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:35.272 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:35.272 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:35.272 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:35.272 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:35.272 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:35.272 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:35.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.287+0000 7f65b8a48640 10 monclient: _finish_auth 0 2024-04-26T00:21:35.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.287+0000 7f65b8a48640 10 monclient: _check_auth_tickets 2024-04-26T00:21:35.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.287+0000 7f65b8a48640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:35.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.287+0000 7f65bbcae640 5 monclient: authenticate success, global_id 4508 2024-04-26T00:21:35.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.287+0000 7f65bbcae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:21:35.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.287+0000 7f65b8a48640 1 -- 172.21.15.37:0/1462436444 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65a80a7950 con 0x7f65b40a0260 2024-04-26T00:21:35.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.287+0000 7f65b8a48640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:35.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.287+0000 7f65b8a48640 1 -- 172.21.15.37:0/1462436444 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a80a7d20 con 0x7f65b40a0260 2024-04-26T00:21:35.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.287+0000 7f65b8a48640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:35.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.287+0000 7f65b8a48640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:21:35.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.287+0000 7f65b8a48640 10 monclient: dump: 2024-04-26T00:21:35.273 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:35.273 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:35.273 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:35.273 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:35.273 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:35.273 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:35.273 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:35.273 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:35.274 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:35.274 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:35.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.287+0000 7f65bbcae640 10 monclient: get_monmap_and_config success 2024-04-26T00:21:35.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.287+0000 7f65bbcae640 4 set_mon_vals no callback set 2024-04-26T00:21:35.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65bbcae640 10 monclient: shutdown 2024-04-26T00:21:35.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65bbcae640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:35.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65bbcae640 1 -- 172.21.15.37:0/1462436444 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f65b40a0260 msgr2=0x7f65b4072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:35.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65bbcae640 1 --2- 172.21.15.37:0/1462436444 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f65b40a0260 0x7f65b4072e10 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f65a800bb10 tx=0x7f65a801eb40 comp rx=0 tx=0).stop 2024-04-26T00:21:35.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65bbcae640 1 -- 172.21.15.37:0/1462436444 shutdown_connections 2024-04-26T00:21:35.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65bbcae640 1 --2- 172.21.15.37:0/1462436444 >> v2:172.21.15.45:3300/0 conn(0x7f65b41389e0 0x7f65b413add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:35.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65bbcae640 1 --2- 172.21.15.37:0/1462436444 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f65b40a0260 0x7f65b4072e10 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:35.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65bbcae640 1 -- 172.21.15.37:0/1462436444 >> 172.21.15.37:0/1462436444 conn(0x7f65b40a1ad0 msgr2=0x7f65b40a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:35.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65bbcae640 1 -- 172.21.15.37:0/1462436444 shutdown_connections 2024-04-26T00:21:35.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65bbcae640 1 -- 172.21.15.37:0/1462436444 wait complete. 2024-04-26T00:21:35.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65bbcae640 10 monclient: build_initial_monmap 2024-04-26T00:21:35.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65bbcae640 10 monclient: monmap: 2024-04-26T00:21:35.275 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:35.275 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:21:35.275 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:21:35.275 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:21:35.275 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:35.275 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:21:35.275 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:21:35.275 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:21:35.275 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:21:35.275 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:35.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65bbcae640 1 Processor -- start 2024-04-26T00:21:35.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65bbcae640 1 -- start start 2024-04-26T00:21:35.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65bbcae640 10 monclient: init 2024-04-26T00:21:35.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65bbcae640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:35.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65bbcae640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:21:35.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65bbcae640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65b40a0260 0x7f65b41ca810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:35.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65bbcae640 10 monclient(hunting): picked mon.noname-a con 0x7f65b40a0260 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:21:35.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65bbcae640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f65b40a0640 0x7f65b4134730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:35.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65bbcae640 10 monclient(hunting): picked mon.noname-c con 0x7f65b40a0640 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:21:35.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65bbcae640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65b41389e0 0x7f65b411b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:35.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65bbcae640 10 monclient(hunting): picked mon.noname-b con 0x7f65b41389e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:21:35.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65bbcae640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:35.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65bbcae640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f65b413d530 con 0x7f65b40a0260 2024-04-26T00:21:35.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65bbcae640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:35.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65bbcae640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f65b409be60 con 0x7f65b41389e0 2024-04-26T00:21:35.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65bbcae640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:35.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65bbcae640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f65b413d6b0 con 0x7f65b40a0640 2024-04-26T00:21:35.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65bbcae640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:35.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65bbcae640 10 monclient(hunting): authenticate will time out at 1076.597740s 2024-04-26T00:21:35.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65b9a4a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65b40a0260 0x7f65b41ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:35.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65b9249640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f65b40a0640 0x7f65b4134730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:35.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65b9a4a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65b40a0260 0x7f65b41ca810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33728/0 (socket says 172.21.15.37:33728) 2024-04-26T00:21:35.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65b9249640 10 monclient(hunting): get_auth_request con 0x7f65b40a0640 auth_method 0 2024-04-26T00:21:35.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65b9a4a640 1 -- 172.21.15.37:0/1509194272 learned_addr learned my addr 172.21.15.37:0/1509194272 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:21:35.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65b9249640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:35.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65b9249640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:35.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65b9249640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:35.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65b9a4a640 10 monclient(hunting): get_auth_request con 0x7f65b40a0260 auth_method 0 2024-04-26T00:21:35.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65b9a4a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:35.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65b9a4a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:35.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65b9a4a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:35.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65b9249640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:35.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65b9249640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:35.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65b9249640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:35.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65b9a4a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:35.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65b9a4a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:35.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65b9a4a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:35.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65ba24b640 1 --2- 172.21.15.37:0/1509194272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65b41389e0 0x7f65b411b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:35.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65ba24b640 10 monclient(hunting): get_auth_request con 0x7f65b41389e0 auth_method 0 2024-04-26T00:21:35.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65ba24b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:35.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65ba24b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:35.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65ba24b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:35.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65b9249640 10 monclient(hunting): handle_auth_done global_id 4511 payload 824 2024-04-26T00:21:35.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65b9249640 1 -- 172.21.15.37:0/1509194272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65b41389e0 msgr2=0x7f65b411b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:35.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65b9249640 1 --2- 172.21.15.37:0/1509194272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65b41389e0 0x7f65b411b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:35.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65b9249640 1 -- 172.21.15.37:0/1509194272 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65b40a0260 msgr2=0x7f65b41ca810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:35.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65b9249640 1 --2- 172.21.15.37:0/1509194272 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65b40a0260 0x7f65b41ca810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:35.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65b9249640 10 monclient: _finish_hunting 0 2024-04-26T00:21:35.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65b9249640 1 monclient: found mon.noname-c 2024-04-26T00:21:35.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65b9249640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:35.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65b9249640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:21:35.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65b9249640 1 -- 172.21.15.37:0/1509194272 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65b41cb720 con 0x7f65b40a0640 2024-04-26T00:21:35.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.291+0000 7f65b9a4a640 1 --2- 172.21.15.37:0/1509194272 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65b40a0260 0x7f65b41ca810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:21:35.280 INFO:tasks.rados.rados.0.smithi045.stdout:265: writing smithi04526579-29 from 416840 to 1192015 tid 1 2024-04-26T00:21:35.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.295+0000 7f65b9249640 1 -- 172.21.15.37:0/1509194272 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f65b40a0640 msgr2=0x7f65b4134730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:35.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.295+0000 7f65b9249640 1 --2- 172.21.15.37:0/1509194272 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f65b40a0640 0x7f65b4134730 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f65b000c3e0 tx=0x7f65b000c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:35.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.295+0000 7f65a67fc640 1 -- 172.21.15.37:0/1509194272 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b00119a0 con 0x7f65b40a0640 2024-04-26T00:21:35.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.295+0000 7f65a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:35.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.295+0000 7f65a67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:21:35.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.295+0000 7f65a67fc640 10 monclient: dump: 2024-04-26T00:21:35.282 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:35.282 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:35.282 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:35.282 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:35.282 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:35.282 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:35.282 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:35.283 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:35.283 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:35.283 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:35.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.295+0000 7f65a67fc640 10 monclient: _finish_auth 0 2024-04-26T00:21:35.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.295+0000 7f65a67fc640 10 monclient: _check_auth_tickets 2024-04-26T00:21:35.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.295+0000 7f65a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:35.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.295+0000 7f65a67fc640 1 -- 172.21.15.37:0/1509194272 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65b0017070 con 0x7f65b40a0640 2024-04-26T00:21:35.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.295+0000 7f65bbcae640 5 monclient: authenticate success, global_id 4511 2024-04-26T00:21:35.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.295+0000 7f65bbcae640 10 monclient: _renew_subs 2024-04-26T00:21:35.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.295+0000 7f65bbcae640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:21:35.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.295+0000 7f65bbcae640 1 -- 172.21.15.37:0/1509194272 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65b411f0c0 con 0x7f65b40a0640 2024-04-26T00:21:35.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.295+0000 7f65a67fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:35.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.295+0000 7f65a67fc640 1 -- 172.21.15.37:0/1509194272 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b0011d80 con 0x7f65b40a0640 2024-04-26T00:21:35.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.295+0000 7f65a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:35.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.295+0000 7f65a67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:21:35.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.295+0000 7f65a67fc640 10 monclient: dump: 2024-04-26T00:21:35.284 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:35.284 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:35.284 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:35.284 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:35.284 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:35.284 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:35.284 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:35.284 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:35.284 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:35.284 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:35.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.295+0000 7f65a7fff640 4 set_mon_vals no callback set 2024-04-26T00:21:35.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.295+0000 7f65bbcae640 10 monclient: _renew_subs 2024-04-26T00:21:35.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.295+0000 7f65bbcae640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:21:35.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.295+0000 7f65bbcae640 1 -- 172.21.15.37:0/1509194272 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f65b411cd60 con 0x7f65b40a0640 2024-04-26T00:21:35.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.295+0000 7f65bbcae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:35.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.295+0000 7f65bbcae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:35.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.295+0000 7f65bbcae640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:21:35.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.295+0000 7f65bbcae640 1 -- 172.21.15.37:0/1509194272 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65b4066f40 con 0x7f65b40a0640 2024-04-26T00:21:35.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.295+0000 7f65a67fc640 1 -- 172.21.15.37:0/1509194272 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f65b001c460 con 0x7f65b40a0640 2024-04-26T00:21:35.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.295+0000 7f65a67fc640 1 --2- 172.21.15.37:0/1509194272 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6598035ed0 0x7f6598038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:35.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.295+0000 7f65a67fc640 1 -- 172.21.15.37:0/1509194272 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(71..71 src has 1..71) ==== 3975+0+0 (secure 0 0 0) 0x7f65b00488c0 con 0x7f65b40a0640 2024-04-26T00:21:35.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.295+0000 7f65b9a4a640 1 --2- 172.21.15.37:0/1509194272 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6598035ed0 0x7f6598038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:35.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.295+0000 7f65b9a4a640 10 monclient: get_auth_request con 0x7f6598035ed0 auth_method 0 2024-04-26T00:21:35.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.299+0000 7f65b9a4a640 1 --2- 172.21.15.37:0/1509194272 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6598035ed0 0x7f6598038390 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f65a80b2040 tx=0x7f65a80a9070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:35.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.299+0000 7f65a67fc640 1 -- 172.21.15.37:0/1509194272 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f65b41cb720 con 0x7f65b40a0640 2024-04-26T00:21:35.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.299+0000 7f65a67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:35.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.299+0000 7f65a67fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:21:35.287 INFO:tasks.rados.rados.0.smithi045.stdout:265: writing smithi04526579-29 from 1672545 to 2016840 tid 2 2024-04-26T00:21:35.287 INFO:tasks.rados.rados.0.smithi045.stdout:266: rollback oid 15 current snap is 28 2024-04-26T00:21:35.287 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 15 to 23 2024-04-26T00:21:35.287 INFO:tasks.rados.rados.0.smithi045.stdout:260: finishing write tid 3 to smithi04526579-43 2024-04-26T00:21:35.288 INFO:tasks.rados.rados.0.smithi045.stdout:260: finishing write tid 4 to smithi04526579-43 2024-04-26T00:21:35.288 INFO:tasks.rados.rados.0.smithi045.stdout:260: finishing write tid 5 to smithi04526579-43 2024-04-26T00:21:35.288 INFO:tasks.rados.rados.0.smithi045.stdout:260: oid 43 updating version 0 to 122 2024-04-26T00:21:35.288 INFO:tasks.rados.rados.0.smithi045.stdout:260: oid 43 version 122 is already newer than 119 2024-04-26T00:21:35.288 INFO:tasks.rados.rados.0.smithi045.stdout:260: oid 43 version 122 is already newer than 120 2024-04-26T00:21:35.288 INFO:tasks.rados.rados.0.smithi045.stdout:260: oid 43 version 122 is already newer than 121 2024-04-26T00:21:35.288 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 122 (ObjNum 118 snap 28 seq_num 118) dirty exists 2024-04-26T00:21:35.288 INFO:tasks.rados.rados.0.smithi045.stdout:260: left oid 43 (ObjNum 118 snap 28 seq_num 118) 2024-04-26T00:21:35.288 INFO:tasks.rados.rados.0.smithi045.stdout:261: finishing write tid 1 to smithi04526579-44 2024-04-26T00:21:35.288 INFO:tasks.rados.rados.0.smithi045.stdout:261: finishing write tid 2 to smithi04526579-44 2024-04-26T00:21:35.288 INFO:tasks.rados.rados.0.smithi045.stdout:261: finishing write tid 3 to smithi04526579-44 2024-04-26T00:21:35.288 INFO:tasks.rados.rados.0.smithi045.stdout:261: finishing write tid 4 to smithi04526579-44 2024-04-26T00:21:35.288 INFO:tasks.rados.rados.0.smithi045.stdout:261: oid 44 updating version 0 to 66 2024-04-26T00:21:35.288 INFO:tasks.rados.rados.0.smithi045.stdout:261: oid 44 version 66 is already newer than 65 2024-04-26T00:21:35.289 INFO:tasks.rados.rados.0.smithi045.stdout:261: oid 44 updating version 66 to 67 2024-04-26T00:21:35.289 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 67 (ObjNum 119 snap 28 seq_num 119) dirty exists 2024-04-26T00:21:35.289 INFO:tasks.rados.rados.0.smithi045.stdout:261: left oid 44 (ObjNum 119 snap 28 seq_num 119) 2024-04-26T00:21:35.289 INFO:tasks.rados.rados.0.smithi045.stdout:264: finishing rollback tid 0 to smithi04526579-8 2024-04-26T00:21:35.289 INFO:tasks.rados.rados.0.smithi045.stdout:264: finishing rollback tid 1 to smithi04526579-8 2024-04-26T00:21:35.289 INFO:tasks.rados.rados.0.smithi045.stdout:264: finishing rollback tid 2 to smithi04526579-8 2024-04-26T00:21:35.289 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 54 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-04-26T00:21:35.289 INFO:tasks.rados.rados.0.smithi045.stdout:265: finishing write tid 1 to smithi04526579-29 2024-04-26T00:21:35.289 INFO:tasks.rados.rados.0.smithi045.stdout:260: done (4 left) 2024-04-26T00:21:35.289 INFO:tasks.rados.rados.0.smithi045.stdout:261: done (3 left) 2024-04-26T00:21:35.289 INFO:tasks.rados.rados.0.smithi045.stdout:264: done (2 left) 2024-04-26T00:21:35.289 INFO:tasks.rados.rados.0.smithi045.stdout:267: rollback oid 30 current snap is 28 2024-04-26T00:21:35.289 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 30 to 26 2024-04-26T00:21:35.289 INFO:tasks.rados.rados.0.smithi045.stdout:268: snap_remove snap 18 2024-04-26T00:21:35.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.523+0000 7f65bbcae640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:21:35.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.523+0000 7f65bbcae640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:21:35.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.523+0000 7f65bbcae640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:21:35.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.523+0000 7f65bbcae640 1 -- 172.21.15.37:0/1509194272 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f65b406b9a0 con 0x7f65b40a0640 2024-04-26T00:21:35.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.527+0000 7f65a67fc640 1 -- 172.21.15.37:0/1509194272 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f65b0012420 con 0x7f65b40a0640 2024-04-26T00:21:35.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.527+0000 7f65a67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:21:35.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.527+0000 7f65a67fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:21:35.504 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":16,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:21:35.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.527+0000 7f65bbcae640 1 -- 172.21.15.37:0/1509194272 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6598035ed0 msgr2=0x7f6598038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:35.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.527+0000 7f65bbcae640 1 --2- 172.21.15.37:0/1509194272 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6598035ed0 0x7f6598038390 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f65a80b2040 tx=0x7f65a80a9070 comp rx=0 tx=0).stop 2024-04-26T00:21:35.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.527+0000 7f65bbcae640 10 monclient: shutdown 2024-04-26T00:21:35.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.527+0000 7f65bbcae640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:35.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.527+0000 7f65bbcae640 1 -- 172.21.15.37:0/1509194272 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f65b40a0640 msgr2=0x7f65b4134730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:35.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.527+0000 7f65bbcae640 1 --2- 172.21.15.37:0/1509194272 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f65b40a0640 0x7f65b4134730 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f65b000c3e0 tx=0x7f65b000c8a0 comp rx=0 tx=0).stop 2024-04-26T00:21:35.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.527+0000 7f65bbcae640 1 -- 172.21.15.37:0/1509194272 shutdown_connections 2024-04-26T00:21:35.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.527+0000 7f65bbcae640 1 --2- 172.21.15.37:0/1509194272 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6598035ed0 0x7f6598038390 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:35.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.527+0000 7f65bbcae640 1 --2- 172.21.15.37:0/1509194272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65b41389e0 0x7f65b411b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:35.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.527+0000 7f65bbcae640 1 --2- 172.21.15.37:0/1509194272 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f65b40a0640 0x7f65b4134730 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:35.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.527+0000 7f65bbcae640 1 --2- 172.21.15.37:0/1509194272 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65b40a0260 0x7f65b41ca810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:35.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.527+0000 7f65bbcae640 1 -- 172.21.15.37:0/1509194272 >> 172.21.15.37:0/1509194272 conn(0x7f65b40a1ad0 msgr2=0x7f65b41379e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:35.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.527+0000 7f65bbcae640 1 -- 172.21.15.37:0/1509194272 shutdown_connections 2024-04-26T00:21:35.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:35.527+0000 7f65bbcae640 1 -- 172.21.15.37:0/1509194272 wait complete. 2024-04-26T00:21:36.015 INFO:tasks.rados.rados.0.smithi045.stdout:268: done (3 left) 2024-04-26T00:21:36.015 INFO:tasks.rados.rados.0.smithi045.stdout:269: write (excl) oid 27 current snap is 28 2024-04-26T00:21:36.015 INFO:tasks.rados.rados.0.smithi045.stdout:269: seq_num 121 ranges {433647=450424,1331027=582033} 2024-04-26T00:21:36.030 INFO:tasks.rados.rados.0.smithi045.stdout:269: writing smithi04526579-27 from 433647 to 884071 tid 1 2024-04-26T00:21:36.050 INFO:tasks.rados.rados.0.smithi045.stdout:269: writing smithi04526579-27 from 1331027 to 1913060 tid 2 2024-04-26T00:21:36.050 INFO:tasks.rados.rados.0.smithi045.stdout:270: snap_create 2024-04-26T00:21:36.050 INFO:tasks.rados.rados.0.smithi045.stdout:266: finishing rollback tid 0 to smithi04526579-15 2024-04-26T00:21:36.051 INFO:tasks.rados.rados.0.smithi045.stdout:265: finishing write tid 2 to smithi04526579-29 2024-04-26T00:21:36.051 INFO:tasks.rados.rados.0.smithi045.stdout:266: finishing rollback tid 1 to smithi04526579-15 2024-04-26T00:21:36.051 INFO:tasks.rados.rados.0.smithi045.stdout:266: finishing rollback tid 2 to smithi04526579-15 2024-04-26T00:21:36.051 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 96 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-04-26T00:21:36.051 INFO:tasks.rados.rados.0.smithi045.stdout:265: finishing write tid 3 to smithi04526579-29 2024-04-26T00:21:36.051 INFO:tasks.rados.rados.0.smithi045.stdout:265: finishing write tid 4 to smithi04526579-29 2024-04-26T00:21:36.051 INFO:tasks.rados.rados.0.smithi045.stdout:265: oid 29 updating version 0 to 58 2024-04-26T00:21:36.051 INFO:tasks.rados.rados.0.smithi045.stdout:265: oid 29 version 58 is already newer than 57 2024-04-26T00:21:36.051 INFO:tasks.rados.rados.0.smithi045.stdout:265: oid 29 version 58 is already newer than 56 2024-04-26T00:21:36.051 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 58 (ObjNum 120 snap 28 seq_num 120) dirty exists 2024-04-26T00:21:36.051 INFO:tasks.rados.rados.0.smithi045.stdout:265: left oid 29 (ObjNum 120 snap 28 seq_num 120) 2024-04-26T00:21:36.051 INFO:tasks.rados.rados.0.smithi045.stdout:267: finishing rollback tid 0 to smithi04526579-30 2024-04-26T00:21:36.051 INFO:tasks.rados.rados.0.smithi045.stdout:267: finishing rollback tid 1 to smithi04526579-30 2024-04-26T00:21:36.051 INFO:tasks.rados.rados.0.smithi045.stdout:267: finishing rollback tid 2 to smithi04526579-30 2024-04-26T00:21:36.052 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 61 (ObjNum 83 snap 15 seq_num 83) dirty exists 2024-04-26T00:21:36.052 INFO:tasks.rados.rados.0.smithi045.stdout:269: finishing write tid 1 to smithi04526579-27 2024-04-26T00:21:36.057 INFO:tasks.rados.rados.0.smithi045.stdout:269: finishing write tid 2 to smithi04526579-27 2024-04-26T00:21:36.058 INFO:tasks.rados.rados.0.smithi045.stdout:269: finishing write tid 3 to smithi04526579-27 2024-04-26T00:21:36.059 INFO:tasks.rados.rados.0.smithi045.stdout:269: finishing write tid 4 to smithi04526579-27 2024-04-26T00:21:36.059 INFO:tasks.rados.rados.0.smithi045.stdout:269: oid 27 updating version 0 to 71 2024-04-26T00:21:36.059 INFO:tasks.rados.rados.0.smithi045.stdout:269: oid 27 updating version 71 to 72 2024-04-26T00:21:36.059 INFO:tasks.rados.rados.0.smithi045.stdout:269: oid 27 updating version 72 to 73 2024-04-26T00:21:36.059 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 73 (ObjNum 121 snap 28 seq_num 121) dirty exists 2024-04-26T00:21:36.059 INFO:tasks.rados.rados.0.smithi045.stdout:269: left oid 27 (ObjNum 121 snap 28 seq_num 121) 2024-04-26T00:21:37.016 INFO:tasks.rados.rados.0.smithi045.stdout:265: done (4 left) 2024-04-26T00:21:37.016 INFO:tasks.rados.rados.0.smithi045.stdout:266: done (3 left) 2024-04-26T00:21:37.016 INFO:tasks.rados.rados.0.smithi045.stdout:267: done (2 left) 2024-04-26T00:21:37.017 INFO:tasks.rados.rados.0.smithi045.stdout:269: done (1 left) 2024-04-26T00:21:37.017 INFO:tasks.rados.rados.0.smithi045.stdout:270: done (0 left) 2024-04-26T00:21:37.017 INFO:tasks.rados.rados.0.smithi045.stdout:271: delete oid 6 current snap is 29 2024-04-26T00:21:37.028 INFO:tasks.rados.rados.0.smithi045.stdout:271: done (0 left) 2024-04-26T00:21:37.028 INFO:tasks.rados.rados.0.smithi045.stdout:272: read oid 1 snap -1 2024-04-26T00:21:37.028 INFO:tasks.rados.rados.0.smithi045.stdout:272: expect deleted 2024-04-26T00:21:37.029 INFO:tasks.rados.rados.0.smithi045.stdout:273: read oid 24 snap 20 2024-04-26T00:21:37.029 INFO:tasks.rados.rados.0.smithi045.stdout:273: expect (ObjNum 23 snap 0 seq_num 23) 2024-04-26T00:21:37.029 INFO:tasks.rados.rados.0.smithi045.stdout:274: snap_remove snap 28 2024-04-26T00:21:38.033 INFO:tasks.rados.rados.0.smithi045.stdout:274: done (2 left) 2024-04-26T00:21:38.033 INFO:tasks.rados.rados.0.smithi045.stdout:275: snap_create 2024-04-26T00:21:38.033 INFO:tasks.rados.rados.0.smithi045.stdout:273: expect (ObjNum 23 snap 0 seq_num 23) 2024-04-26T00:21:38.524 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:21:38.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2ed17f6640 10 monclient: get_monmap_and_config 2024-04-26T00:21:38.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2ed17f6640 10 monclient: build_initial_monmap 2024-04-26T00:21:38.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2ed17f6640 10 monclient: monmap: 2024-04-26T00:21:38.657 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:38.657 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:38.657 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:21:38.681096+0000 2024-04-26T00:21:38.657 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:21:38.681096+0000 2024-04-26T00:21:38.657 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:38.657 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:38.657 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:21:38.657 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:21:38.657 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:21:38.657 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:21:38.657 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:21:38.657 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:38.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2ed17f6640 1 Processor -- start 2024-04-26T00:21:38.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2ed17f6640 1 -- start start 2024-04-26T00:21:38.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2ed17f6640 10 monclient: init 2024-04-26T00:21:38.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2ed17f6640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:38.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2ed17f6640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-04-26T00:21:38.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2ed17f6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2ecc0a0260 addr v1:172.21.15.37:6789/0 2024-04-26T00:21:38.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2ed17f6640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f2ecc137e40 0x7f2ecc13a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:38.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2ed17f6640 10 monclient(hunting): picked mon.noname-a con 0x7f2ecc137e40 addr v2:172.21.15.37:3300/0 2024-04-26T00:21:38.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2ed17f6640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ecc13a770 0x7f2ecc13cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:38.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2ed17f6640 10 monclient(hunting): picked mon.noname-b con 0x7f2ecc13a770 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:21:38.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2ed17f6640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ecc0a1130 con 0x7f2ecc0a0260 2024-04-26T00:21:38.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2ed17f6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:38.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2ed17f6640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f2ecc0a0d50 con 0x7f2ecc137e40 2024-04-26T00:21:38.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2ed17f6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:38.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2ed17f6640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2ecc09ec10 con 0x7f2ecc13a770 2024-04-26T00:21:38.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2ed17f6640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:38.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2ed17f6640 10 monclient(hunting): authenticate will time out at 1079.986828s 2024-04-26T00:21:38.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2ecaffd640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f2ecc0a0260 0x7f2ecc0a0640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:39960/0 (socket says 172.21.15.37:39960) 2024-04-26T00:21:38.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2ecaffd640 1 -- 172.21.15.37:0/336706728 learned_addr learned my addr 172.21.15.37:0/336706728 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:21:38.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2eca7fc640 1 --2- 172.21.15.37:0/336706728 >> v2:172.21.15.37:3300/0 conn(0x7f2ecc137e40 0x7f2ecc13a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:38.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2ecb7fe640 1 --2- 172.21.15.37:0/336706728 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ecc13a770 0x7f2ecc13cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:38.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2ecb7fe640 10 monclient(hunting): get_auth_request con 0x7f2ecc13a770 auth_method 0 2024-04-26T00:21:38.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2ecb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:38.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2ecb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:38.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2ecb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:38.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2eca7fc640 10 monclient(hunting): get_auth_request con 0x7f2ecc137e40 auth_method 0 2024-04-26T00:21:38.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2eca7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:38.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2eca7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:38.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2eca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:38.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2ecb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:38.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2ecb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:38.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2ecb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:38.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2eca7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:38.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2eca7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:38.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2eca7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:38.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2ecb7fe640 10 monclient(hunting): handle_auth_done global_id 4514 payload 274 2024-04-26T00:21:38.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2ecb7fe640 1 -- 172.21.15.37:0/336706728 >> v2:172.21.15.37:3300/0 conn(0x7f2ecc137e40 msgr2=0x7f2ecc13a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:21:38.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2ecb7fe640 1 --2- 172.21.15.37:0/336706728 >> v2:172.21.15.37:3300/0 conn(0x7f2ecc137e40 0x7f2ecc13a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:38.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2ecb7fe640 1 -- 172.21.15.37:0/336706728 >> v1:172.21.15.37:6789/0 conn(0x7f2ecc0a0260 legacy=0x7f2ecc0a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:21:38.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2ecb7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:21:38.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2ecb7fe640 1 monclient: found mon.noname-b 2024-04-26T00:21:38.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2ecb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:38.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2ecb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:21:38.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2ecb7fe640 1 -- 172.21.15.37:0/336706728 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ecc06cff0 con 0x7f2ecc13a770 2024-04-26T00:21:38.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.679+0000 7f2eca7fc640 1 --2- 172.21.15.37:0/336706728 >> v2:172.21.15.37:3300/0 conn(0x7f2ecc137e40 0x7f2ecc13a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:21:38.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.683+0000 7f2ecb7fe640 1 -- 172.21.15.37:0/336706728 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ecc13a770 msgr2=0x7f2ecc13cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:38.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.683+0000 7f2ecb7fe640 1 --2- 172.21.15.37:0/336706728 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ecc13a770 0x7f2ecc13cb60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc0097c0 tx=0x7f2ebc01c140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e96aa15209635944 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:38.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.683+0000 7f2ec9ffb640 1 -- 172.21.15.37:0/336706728 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ebc0b1070 con 0x7f2ecc13a770 2024-04-26T00:21:38.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.683+0000 7f2ec9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:38.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ec9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:21:38.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ec9ffb640 10 monclient: dump: 2024-04-26T00:21:38.667 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:38.667 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:38.667 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:38.667 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:38.667 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:38.667 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:38.667 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:38.667 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:38.668 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:38.668 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:38.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ec9ffb640 10 monclient: _finish_auth 0 2024-04-26T00:21:38.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ec9ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:21:38.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ec9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:38.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ec9ffb640 1 -- 172.21.15.37:0/336706728 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ebc0ac440 con 0x7f2ecc13a770 2024-04-26T00:21:38.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ec9ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:38.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ec9ffb640 1 -- 172.21.15.37:0/336706728 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ebc0ac810 con 0x7f2ecc13a770 2024-04-26T00:21:38.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ed17f6640 5 monclient: authenticate success, global_id 4514 2024-04-26T00:21:38.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ed17f6640 10 monclient: get_monmap_and_config success 2024-04-26T00:21:38.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ed17f6640 4 set_mon_vals no callback set 2024-04-26T00:21:38.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ec9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:38.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ec9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:21:38.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ec9ffb640 10 monclient: dump: 2024-04-26T00:21:38.668 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:38.669 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:38.669 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:38.669 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:38.669 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:38.669 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:38.669 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:38.669 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:38.669 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:38.669 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:38.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ed17f6640 10 monclient: shutdown 2024-04-26T00:21:38.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ed17f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:38.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ed17f6640 1 -- 172.21.15.37:0/336706728 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ecc13a770 msgr2=0x7f2ecc13cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:38.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ed17f6640 1 --2- 172.21.15.37:0/336706728 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ecc13a770 0x7f2ecc13cb60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc0097c0 tx=0x7f2ebc01c140 comp rx=0 tx=0).stop 2024-04-26T00:21:38.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ed17f6640 1 -- 172.21.15.37:0/336706728 shutdown_connections 2024-04-26T00:21:38.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ed17f6640 1 --2- 172.21.15.37:0/336706728 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ecc13a770 0x7f2ecc13cb60 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:38.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ed17f6640 1 --2- 172.21.15.37:0/336706728 >> v2:172.21.15.37:3300/0 conn(0x7f2ecc137e40 0x7f2ecc13a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:38.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ed17f6640 1 -- 172.21.15.37:0/336706728 >> 172.21.15.37:0/336706728 conn(0x7f2ecc0a1ad0 msgr2=0x7f2ecc0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:38.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ed17f6640 1 -- 172.21.15.37:0/336706728 shutdown_connections 2024-04-26T00:21:38.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ed17f6640 1 -- 172.21.15.37:0/336706728 wait complete. 2024-04-26T00:21:38.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ed17f6640 10 monclient: build_initial_monmap 2024-04-26T00:21:38.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ed17f6640 10 monclient: monmap: 2024-04-26T00:21:38.670 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:38.670 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:21:38.670 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:21:38.670 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:21:38.670 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:38.670 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:21:38.670 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:21:38.670 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:21:38.671 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:21:38.671 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:38.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ed17f6640 1 Processor -- start 2024-04-26T00:21:38.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ed17f6640 1 -- start start 2024-04-26T00:21:38.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ed17f6640 10 monclient: init 2024-04-26T00:21:38.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ed17f6640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:38.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ed17f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:21:38.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ed17f6640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ecc0a0260 0x7f2ecc11dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:38.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ed17f6640 10 monclient(hunting): picked mon.noname-a con 0x7f2ecc0a0260 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:21:38.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ed17f6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ecc137e40 0x7f2ecc11c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:38.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ed17f6640 10 monclient(hunting): picked mon.noname-b con 0x7f2ecc137e40 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:21:38.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ed17f6640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ecc13a770 0x7f2ecc1cc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:38.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ed17f6640 10 monclient(hunting): picked mon.noname-c con 0x7f2ecc13a770 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:21:38.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ed17f6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:38.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ed17f6640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f2ecc09f4c0 con 0x7f2ecc0a0260 2024-04-26T00:21:38.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ed17f6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:38.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ed17f6640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2ecc09f0f0 con 0x7f2ecc137e40 2024-04-26T00:21:38.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ed17f6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:38.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ed17f6640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2ecc13f8c0 con 0x7f2ecc13a770 2024-04-26T00:21:38.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ed17f6640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:38.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ed17f6640 10 monclient(hunting): authenticate will time out at 1079.994130s 2024-04-26T00:21:38.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ecaffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ecc0a0260 0x7f2ecc11dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:38.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ecb7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ecc13a770 0x7f2ecc1cc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:38.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ecaffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ecc0a0260 0x7f2ecc11dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33764/0 (socket says 172.21.15.37:33764) 2024-04-26T00:21:38.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ecaffd640 1 -- 172.21.15.37:0/4021820759 learned_addr learned my addr 172.21.15.37:0/4021820759 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:21:38.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ecaffd640 10 monclient(hunting): get_auth_request con 0x7f2ecc0a0260 auth_method 0 2024-04-26T00:21:38.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ecaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:38.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ecaffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:38.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ecaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:38.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ecb7fe640 10 monclient(hunting): get_auth_request con 0x7f2ecc13a770 auth_method 0 2024-04-26T00:21:38.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ecb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:38.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ecb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:38.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ecb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:38.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ecb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:38.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ecb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:38.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ecb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:38.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ecaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:38.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ecaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:38.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ecaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:38.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ecb7fe640 10 monclient(hunting): handle_auth_done global_id 4517 payload 824 2024-04-26T00:21:38.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2eca7fc640 1 --2- 172.21.15.37:0/4021820759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ecc137e40 0x7f2ecc11c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:38.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ecb7fe640 1 -- 172.21.15.37:0/4021820759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ecc137e40 msgr2=0x7f2ecc11c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:38.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ecb7fe640 1 --2- 172.21.15.37:0/4021820759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ecc137e40 0x7f2ecc11c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:38.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ecb7fe640 1 -- 172.21.15.37:0/4021820759 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ecc0a0260 msgr2=0x7f2ecc11dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:38.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ecb7fe640 1 --2- 172.21.15.37:0/4021820759 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ecc0a0260 0x7f2ecc11dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:38.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ecb7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:21:38.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ecb7fe640 1 monclient: found mon.noname-c 2024-04-26T00:21:38.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ecb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:38.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ecb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:21:38.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ecb7fe640 1 -- 172.21.15.37:0/4021820759 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ecc0729d0 con 0x7f2ecc13a770 2024-04-26T00:21:38.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.687+0000 7f2ecaffd640 1 --2- 172.21.15.37:0/4021820759 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ecc0a0260 0x7f2ecc11dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:21:38.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.691+0000 7f2ecb7fe640 1 -- 172.21.15.37:0/4021820759 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ecc13a770 msgr2=0x7f2ecc1cc9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:38.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.691+0000 7f2ecb7fe640 1 --2- 172.21.15.37:0/4021820759 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ecc13a770 0x7f2ecc1cc9c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc0ab000 tx=0x7f2ebc018ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:38.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.691+0000 7f2eb37fe640 1 -- 172.21.15.37:0/4021820759 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ebc0b9030 con 0x7f2ecc13a770 2024-04-26T00:21:38.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.691+0000 7f2eb37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:38.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.691+0000 7f2eb37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:21:38.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.691+0000 7f2eb37fe640 10 monclient: dump: 2024-04-26T00:21:38.676 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:38.677 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:38.677 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:38.677 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:38.677 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:38.677 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:38.677 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:38.677 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:38.677 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:38.677 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:38.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.691+0000 7f2eb37fe640 10 monclient: _finish_auth 0 2024-04-26T00:21:38.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.691+0000 7f2eb37fe640 10 monclient: _check_auth_tickets 2024-04-26T00:21:38.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.691+0000 7f2eb37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:38.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.691+0000 7f2eb37fe640 1 -- 172.21.15.37:0/4021820759 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ebc0b5410 con 0x7f2ecc13a770 2024-04-26T00:21:38.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.691+0000 7f2eb37fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:38.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.691+0000 7f2eb37fe640 1 -- 172.21.15.37:0/4021820759 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ebc0b1040 con 0x7f2ecc13a770 2024-04-26T00:21:38.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.691+0000 7f2eb37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:38.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.691+0000 7f2eb37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:21:38.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.691+0000 7f2eb37fe640 10 monclient: dump: 2024-04-26T00:21:38.678 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:38.678 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:38.678 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:38.678 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:38.678 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:38.678 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:38.678 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:38.678 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:38.678 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:38.678 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:38.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.691+0000 7f2ec8ff9640 4 set_mon_vals no callback set 2024-04-26T00:21:38.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.691+0000 7f2ed17f6640 5 monclient: authenticate success, global_id 4517 2024-04-26T00:21:38.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.691+0000 7f2ed17f6640 10 monclient: _renew_subs 2024-04-26T00:21:38.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.691+0000 7f2ed17f6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:21:38.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.691+0000 7f2ed17f6640 1 -- 172.21.15.37:0/4021820759 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ecc11e830 con 0x7f2ecc13a770 2024-04-26T00:21:38.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.691+0000 7f2ed17f6640 10 monclient: _renew_subs 2024-04-26T00:21:38.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.691+0000 7f2ed17f6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:21:38.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.691+0000 7f2ed17f6640 1 -- 172.21.15.37:0/4021820759 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ecc11f2b0 con 0x7f2ecc13a770 2024-04-26T00:21:38.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.691+0000 7f2ed17f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:38.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.691+0000 7f2ed17f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:38.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.691+0000 7f2ed17f6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:21:38.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.691+0000 7f2ed17f6640 1 -- 172.21.15.37:0/4021820759 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ecc066f40 con 0x7f2ecc13a770 2024-04-26T00:21:38.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.691+0000 7f2eb37fe640 1 -- 172.21.15.37:0/4021820759 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2ebc0b5840 con 0x7f2ecc13a770 2024-04-26T00:21:38.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.691+0000 7f2eb37fe640 1 --2- 172.21.15.37:0/4021820759 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2eb8035f00 0x7f2eb80383c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:38.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.691+0000 7f2eb37fe640 1 -- 172.21.15.37:0/4021820759 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(74..74 src has 1..74) ==== 3991+0+0 (secure 0 0 0) 0x7f2ebc0e20c0 con 0x7f2ecc13a770 2024-04-26T00:21:38.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.691+0000 7f2ecaffd640 1 --2- 172.21.15.37:0/4021820759 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2eb8035f00 0x7f2eb80383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:38.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.691+0000 7f2ecaffd640 10 monclient: get_auth_request con 0x7f2eb8035f00 auth_method 0 2024-04-26T00:21:38.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.695+0000 7f2ecaffd640 1 --2- 172.21.15.37:0/4021820759 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2eb8035f00 0x7f2eb80383c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2ec400d660 tx=0x7f2ec4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:38.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.695+0000 7f2eb37fe640 1 -- 172.21.15.37:0/4021820759 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f2ebc0c3020 con 0x7f2ecc13a770 2024-04-26T00:21:38.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.695+0000 7f2eb37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:38.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.695+0000 7f2eb37fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:21:38.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.915+0000 7f2ed17f6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:21:38.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.915+0000 7f2ed17f6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:21:38.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.915+0000 7f2ed17f6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:21:38.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.915+0000 7f2ed17f6640 1 -- 172.21.15.37:0/4021820759 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2ecc06b9a0 con 0x7f2ecc13a770 2024-04-26T00:21:38.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.915+0000 7f2eb37fe640 1 -- 172.21.15.37:0/4021820759 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f2ecc0a1130 con 0x7f2ecc13a770 2024-04-26T00:21:38.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.915+0000 7f2eb37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:21:38.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.915+0000 7f2eb37fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:21:38.894 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":18,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:21:38.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.919+0000 7f2ed17f6640 1 -- 172.21.15.37:0/4021820759 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2eb8035f00 msgr2=0x7f2eb80383c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:38.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.919+0000 7f2ed17f6640 1 --2- 172.21.15.37:0/4021820759 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2eb8035f00 0x7f2eb80383c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2ec400d660 tx=0x7f2ec4009040 comp rx=0 tx=0).stop 2024-04-26T00:21:38.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.919+0000 7f2ed17f6640 10 monclient: shutdown 2024-04-26T00:21:38.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.919+0000 7f2ed17f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:38.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.919+0000 7f2ed17f6640 1 -- 172.21.15.37:0/4021820759 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ecc13a770 msgr2=0x7f2ecc1cc9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:38.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.919+0000 7f2ed17f6640 1 --2- 172.21.15.37:0/4021820759 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ecc13a770 0x7f2ecc1cc9c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc0ab000 tx=0x7f2ebc018ee0 comp rx=0 tx=0).stop 2024-04-26T00:21:38.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.919+0000 7f2ed17f6640 1 -- 172.21.15.37:0/4021820759 shutdown_connections 2024-04-26T00:21:38.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.919+0000 7f2ed17f6640 1 --2- 172.21.15.37:0/4021820759 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2eb8035f00 0x7f2eb80383c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:38.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.919+0000 7f2ed17f6640 1 --2- 172.21.15.37:0/4021820759 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ecc13a770 0x7f2ecc1cc9c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:38.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.919+0000 7f2ed17f6640 1 --2- 172.21.15.37:0/4021820759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ecc137e40 0x7f2ecc11c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:38.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.919+0000 7f2ed17f6640 1 --2- 172.21.15.37:0/4021820759 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ecc0a0260 0x7f2ecc11dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:38.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.919+0000 7f2ed17f6640 1 -- 172.21.15.37:0/4021820759 >> 172.21.15.37:0/4021820759 conn(0x7f2ecc0a1ad0 msgr2=0x7f2ecc0a3d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:38.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.919+0000 7f2ed17f6640 1 -- 172.21.15.37:0/4021820759 shutdown_connections 2024-04-26T00:21:38.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:38.919+0000 7f2ed17f6640 1 -- 172.21.15.37:0/4021820759 wait complete. 2024-04-26T00:21:38.912 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:21:38.912 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:21:38.912 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:21:39.036 INFO:tasks.rados.rados.0.smithi045.stdout:272: done (2 left) 2024-04-26T00:21:39.036 INFO:tasks.rados.rados.0.smithi045.stdout:273: done (1 left) 2024-04-26T00:21:39.036 INFO:tasks.rados.rados.0.smithi045.stdout:275: done (0 left) 2024-04-26T00:21:39.036 INFO:tasks.rados.rados.0.smithi045.stdout:276: snap_remove snap 26 2024-04-26T00:21:39.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.059+0000 7fe803fef640 10 monclient: get_monmap_and_config 2024-04-26T00:21:39.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.059+0000 7fe803fef640 10 monclient: build_initial_monmap 2024-04-26T00:21:39.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.059+0000 7fe803fef640 10 monclient: monmap: 2024-04-26T00:21:39.037 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:39.037 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:39.037 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:21:39.061149+0000 2024-04-26T00:21:39.037 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:21:39.061149+0000 2024-04-26T00:21:39.037 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:39.037 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:39.037 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:21:39.037 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:21:39.037 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:21:39.037 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:21:39.037 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:21:39.037 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:39.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.059+0000 7fe803fef640 1 Processor -- start 2024-04-26T00:21:39.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.059+0000 7fe803fef640 1 -- start start 2024-04-26T00:21:39.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.059+0000 7fe803fef640 10 monclient: init 2024-04-26T00:21:39.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.059+0000 7fe803fef640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:39.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.059+0000 7fe803fef640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-04-26T00:21:39.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.059+0000 7fe803fef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe7fc09c7f0 addr v1:172.21.15.45:6789/0 2024-04-26T00:21:39.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.059+0000 7fe803fef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe7fc1350b0 addr v1:172.21.15.37:6789/0 2024-04-26T00:21:39.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.059+0000 7fe803fef640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fe7fc13b690 0x7fe7fc13da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:39.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.059+0000 7fe803fef640 10 monclient(hunting): picked mon.noname-a con 0x7fe7fc13b690 addr v2:172.21.15.37:3300/0 2024-04-26T00:21:39.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.059+0000 7fe803fef640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7fc135500 con 0x7fe7fc1350b0 2024-04-26T00:21:39.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.059+0000 7fe803fef640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7fc06bc70 con 0x7fe7fc09c7f0 2024-04-26T00:21:39.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.059+0000 7fe803fef640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:39.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.059+0000 7fe803fef640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fe7fc06df10 con 0x7fe7fc13b690 2024-04-26T00:21:39.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.059+0000 7fe803fef640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:39.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.059+0000 7fe803fef640 10 monclient(hunting): authenticate will time out at 1080.367814s 2024-04-26T00:21:39.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.059+0000 7fe80258c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fe7fc13b690 0x7fe7fc13da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:39.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.059+0000 7fe80158a640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fe7fc1350b0 0x7fe7fc139f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:39974/0 (socket says 172.21.15.37:39974) 2024-04-26T00:21:39.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.059+0000 7fe80158a640 1 -- 172.21.15.37:0/516351821 learned_addr learned my addr 172.21.15.37:0/516351821 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:21:39.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.063+0000 7fe80258c640 10 monclient(hunting): get_auth_request con 0x7fe7fc13b690 auth_method 0 2024-04-26T00:21:39.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.063+0000 7fe80258c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:39.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.063+0000 7fe80258c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:39.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.063+0000 7fe80258c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:39.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.063+0000 7fe80258c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:39.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.063+0000 7fe80258c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:39.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.063+0000 7fe80258c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:39.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.063+0000 7fe80258c640 10 monclient(hunting): handle_auth_done global_id 4530 payload 274 2024-04-26T00:21:39.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.063+0000 7fe80258c640 1 -- 172.21.15.37:0/516351821 >> v1:172.21.15.45:6789/0 conn(0x7fe7fc09c7f0 legacy=0x7fe7fc09cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:21:39.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.063+0000 7fe80258c640 1 -- 172.21.15.37:0/516351821 >> v1:172.21.15.37:6789/0 conn(0x7fe7fc1350b0 legacy=0x7fe7fc139f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:21:39.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.063+0000 7fe80258c640 10 monclient: _finish_hunting 0 2024-04-26T00:21:39.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.063+0000 7fe80258c640 1 monclient: found mon.noname-a 2024-04-26T00:21:39.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.063+0000 7fe80258c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:39.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.063+0000 7fe80258c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:21:39.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.063+0000 7fe80258c640 1 -- 172.21.15.37:0/516351821 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7fc06b460 con 0x7fe7fc13b690 2024-04-26T00:21:39.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.067+0000 7fe80258c640 1 -- 172.21.15.37:0/516351821 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7fc13b690 msgr2=0x7fe7fc13da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:39.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.067+0000 7fe80258c640 1 --2- 172.21.15.37:0/516351821 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7fc13b690 0x7fe7fc13da80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe7f4009d90 tx=0x7fe7f401c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58f36a88c5e24871 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:39.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.067+0000 7fe800d89640 1 -- 172.21.15.37:0/516351821 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f40ac070 con 0x7fe7fc13b690 2024-04-26T00:21:39.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.067+0000 7fe800d89640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:39.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.067+0000 7fe800d89640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:21:39.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.067+0000 7fe800d89640 10 monclient: dump: 2024-04-26T00:21:39.046 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:39.046 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:39.046 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:39.046 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:39.046 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:39.046 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:39.046 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:39.046 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:39.046 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:39.046 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:39.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.067+0000 7fe800d89640 10 monclient: _finish_auth 0 2024-04-26T00:21:39.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.067+0000 7fe800d89640 10 monclient: _check_auth_tickets 2024-04-26T00:21:39.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.067+0000 7fe800d89640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:39.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.067+0000 7fe800d89640 1 -- 172.21.15.37:0/516351821 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7f40a6760 con 0x7fe7fc13b690 2024-04-26T00:21:39.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.067+0000 7fe800d89640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:39.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.067+0000 7fe800d89640 1 -- 172.21.15.37:0/516351821 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f40a6b30 con 0x7fe7fc13b690 2024-04-26T00:21:39.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.067+0000 7fe800d89640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:39.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.067+0000 7fe800d89640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:21:39.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.067+0000 7fe800d89640 10 monclient: dump: 2024-04-26T00:21:39.047 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:39.047 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:39.047 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:39.047 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:39.047 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:39.047 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:39.047 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:39.047 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:39.048 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:39.048 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:39.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.067+0000 7fe803fef640 5 monclient: authenticate success, global_id 4530 2024-04-26T00:21:39.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.067+0000 7fe803fef640 10 monclient: get_monmap_and_config success 2024-04-26T00:21:39.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.067+0000 7fe803fef640 4 set_mon_vals no callback set 2024-04-26T00:21:39.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.067+0000 7fe803fef640 10 monclient: shutdown 2024-04-26T00:21:39.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.067+0000 7fe803fef640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:39.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.067+0000 7fe803fef640 1 -- 172.21.15.37:0/516351821 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7fc13b690 msgr2=0x7fe7fc13da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:39.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.067+0000 7fe803fef640 1 --2- 172.21.15.37:0/516351821 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7fc13b690 0x7fe7fc13da80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe7f4009d90 tx=0x7fe7f401c7b0 comp rx=0 tx=0).stop 2024-04-26T00:21:39.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.067+0000 7fe803fef640 1 -- 172.21.15.37:0/516351821 shutdown_connections 2024-04-26T00:21:39.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.067+0000 7fe803fef640 1 --2- 172.21.15.37:0/516351821 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7fc13b690 0x7fe7fc13da80 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:39.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.067+0000 7fe803fef640 1 -- 172.21.15.37:0/516351821 >> 172.21.15.37:0/516351821 conn(0x7fe7fc0a2c50 msgr2=0x7fe7fc0a3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:39.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.067+0000 7fe803fef640 1 -- 172.21.15.37:0/516351821 shutdown_connections 2024-04-26T00:21:39.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.067+0000 7fe803fef640 1 -- 172.21.15.37:0/516351821 wait complete. 2024-04-26T00:21:39.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe803fef640 10 monclient: build_initial_monmap 2024-04-26T00:21:39.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe803fef640 10 monclient: monmap: 2024-04-26T00:21:39.054 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:39.054 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:21:39.054 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:21:39.054 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:21:39.054 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:39.054 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:21:39.054 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:21:39.055 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:21:39.055 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:21:39.055 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:39.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe803fef640 1 Processor -- start 2024-04-26T00:21:39.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe803fef640 1 -- start start 2024-04-26T00:21:39.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe803fef640 10 monclient: init 2024-04-26T00:21:39.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe803fef640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:39.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe803fef640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:21:39.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe803fef640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7fc09c7f0 0x7fe7fc1ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:39.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe803fef640 10 monclient(hunting): picked mon.noname-b con 0x7fe7fc09c7f0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:21:39.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe803fef640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe7fc1350b0 0x7fe7fc1cd420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:39.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe803fef640 10 monclient(hunting): picked mon.noname-c con 0x7fe7fc1350b0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:21:39.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe803fef640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7fc13b690 0x7fe7fc06f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:39.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe803fef640 10 monclient(hunting): picked mon.noname-a con 0x7fe7fc13b690 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:21:39.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe803fef640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:39.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe803fef640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe7fc1409f0 con 0x7fe7fc13b690 2024-04-26T00:21:39.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe803fef640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:39.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe803fef640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe7fc06b990 con 0x7fe7fc09c7f0 2024-04-26T00:21:39.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe803fef640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:39.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe803fef640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe7fc140b70 con 0x7fe7fc1350b0 2024-04-26T00:21:39.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe803fef640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:39.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe803fef640 10 monclient(hunting): authenticate will time out at 1080.377343s 2024-04-26T00:21:39.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe80258c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7fc13b690 0x7fe7fc06f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:39.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe80158a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe7fc1350b0 0x7fe7fc1cd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:39.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe80258c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7fc13b690 0x7fe7fc06f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33792/0 (socket says 172.21.15.37:33792) 2024-04-26T00:21:39.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe80158a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe7fc1350b0 0x7fe7fc1cd420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:43482/0 (socket says 172.21.15.37:43482) 2024-04-26T00:21:39.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe80258c640 1 -- 172.21.15.37:0/2683985067 learned_addr learned my addr 172.21.15.37:0/2683985067 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:21:39.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe80258c640 10 monclient(hunting): get_auth_request con 0x7fe7fc13b690 auth_method 0 2024-04-26T00:21:39.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe80258c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:39.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe80258c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:39.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe80258c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:39.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe80158a640 10 monclient(hunting): get_auth_request con 0x7fe7fc1350b0 auth_method 0 2024-04-26T00:21:39.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe80158a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:39.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe80158a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:39.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe80158a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:39.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe80258c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:39.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe80258c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:39.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe801d8b640 1 --2- 172.21.15.37:0/2683985067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7fc09c7f0 0x7fe7fc1ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:39.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe80258c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:39.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe80158a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:39.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe80158a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:39.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe80158a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:39.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe801d8b640 10 monclient(hunting): get_auth_request con 0x7fe7fc09c7f0 auth_method 0 2024-04-26T00:21:39.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe801d8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:39.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe801d8b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:39.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe801d8b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:39.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe801d8b640 1 -- 172.21.15.37:0/2683985067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7fc09c7f0 msgr2=0x7fe7fc1ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:21:39.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe801d8b640 1 -- 172.21.15.37:0/2683985067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7fc09c7f0 msgr2=0x7fe7fc1ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:21:39.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe80258c640 10 monclient(hunting): handle_auth_done global_id 4533 payload 824 2024-04-26T00:21:39.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe801d8b640 1 --2- 172.21.15.37:0/2683985067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7fc09c7f0 0x7fe7fc1ccee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:21:39.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe801d8b640 1 --2- 172.21.15.37:0/2683985067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7fc09c7f0 0x7fe7fc1ccee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:21:39.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe80258c640 1 -- 172.21.15.37:0/2683985067 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe7fc1350b0 msgr2=0x7fe7fc1cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:39.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe80258c640 1 --2- 172.21.15.37:0/2683985067 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe7fc1350b0 0x7fe7fc1cd420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:39.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe80258c640 1 -- 172.21.15.37:0/2683985067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7fc09c7f0 msgr2=0x7fe7fc1ccee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:21:39.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe80258c640 1 --2- 172.21.15.37:0/2683985067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7fc09c7f0 0x7fe7fc1ccee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:39.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe80258c640 10 monclient: _finish_hunting 0 2024-04-26T00:21:39.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe80258c640 1 monclient: found mon.noname-a 2024-04-26T00:21:39.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe80258c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:39.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe80258c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:21:39.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe80258c640 1 -- 172.21.15.37:0/2683985067 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7fc13b210 con 0x7fe7fc13b690 2024-04-26T00:21:39.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.071+0000 7fe80158a640 1 --2- 172.21.15.37:0/2683985067 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe7fc1350b0 0x7fe7fc1cd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:21:39.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe80258c640 1 -- 172.21.15.37:0/2683985067 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7fc13b690 msgr2=0x7fe7fc06f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:39.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe80258c640 1 --2- 172.21.15.37:0/2683985067 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7fc13b690 0x7fe7fc06f420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe7f4009d90 tx=0x7fe7f4019550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:39.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe7f27fc640 1 -- 172.21.15.37:0/2683985067 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f40c0070 con 0x7fe7fc13b690 2024-04-26T00:21:39.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe7f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:39.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe7f27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:21:39.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe7f27fc640 10 monclient: dump: 2024-04-26T00:21:39.059 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:39.059 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:39.059 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:39.059 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:39.059 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:39.060 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:39.060 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:39.060 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:39.060 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:39.060 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:39.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe7f27fc640 10 monclient: _finish_auth 0 2024-04-26T00:21:39.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe7f27fc640 10 monclient: _check_auth_tickets 2024-04-26T00:21:39.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe7f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:39.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe7f27fc640 1 -- 172.21.15.37:0/2683985067 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7f40a6560 con 0x7fe7fc13b690 2024-04-26T00:21:39.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe7f27fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:39.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe7f27fc640 1 -- 172.21.15.37:0/2683985067 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f40ac050 con 0x7fe7fc13b690 2024-04-26T00:21:39.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe7f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:39.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe7f27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:21:39.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe7f27fc640 10 monclient: dump: 2024-04-26T00:21:39.060 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:39.061 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:39.061 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:39.061 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:39.061 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:39.061 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:39.061 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:39.061 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:39.061 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:39.061 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:39.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe7f3fff640 4 set_mon_vals no callback set 2024-04-26T00:21:39.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe803fef640 5 monclient: authenticate success, global_id 4533 2024-04-26T00:21:39.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe803fef640 10 monclient: _renew_subs 2024-04-26T00:21:39.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe803fef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:39.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe803fef640 1 -- 172.21.15.37:0/2683985067 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7fc072600 con 0x7fe7fc13b690 2024-04-26T00:21:39.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe803fef640 10 monclient: _renew_subs 2024-04-26T00:21:39.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe803fef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:39.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe803fef640 1 -- 172.21.15.37:0/2683985067 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7fc12b6d0 con 0x7fe7fc13b690 2024-04-26T00:21:39.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe7f27fc640 1 -- 172.21.15.37:0/2683985067 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe7f40af070 con 0x7fe7fc13b690 2024-04-26T00:21:39.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe803fef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:39.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe803fef640 1 --2- 172.21.15.37:0/2683985067 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7fc06f960 0x7fe7fc06fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:39.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe803fef640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:39.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe803fef640 1 -- 172.21.15.37:0/2683985067 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe7fc06c440 con 0x7fe7fc06f960 2024-04-26T00:21:39.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe7f27fc640 1 --2- 172.21.15.37:0/2683985067 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe7e0035e80 0x7fe7e0038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:39.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe7f27fc640 1 -- 172.21.15.37:0/2683985067 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 3963+0+0 (secure 0 0 0) 0x7fe7f40e3350 con 0x7fe7fc13b690 2024-04-26T00:21:39.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe801d8b640 1 --2- 172.21.15.37:0/2683985067 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7fc06f960 0x7fe7fc06fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:39.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe801d8b640 10 monclient: get_auth_request con 0x7fe7fc06f960 auth_method 0 2024-04-26T00:21:39.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe801d8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:39.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe801d8b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:39.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe801d8b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:39.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe801d8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:39.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe801d8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:39.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe801d8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:39.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe801d8b640 10 monclient(hunting): handle_auth_done global_id 4536 payload 824 2024-04-26T00:21:39.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe801d8b640 1 -- 172.21.15.37:0/2683985067 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe7fc04f510 con 0x7fe7fc06f960 2024-04-26T00:21:39.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe80158a640 1 --2- 172.21.15.37:0/2683985067 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe7e0035e80 0x7fe7e0038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:39.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.075+0000 7fe80158a640 10 monclient: get_auth_request con 0x7fe7e0035e80 auth_method 0 2024-04-26T00:21:39.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.079+0000 7fe801d8b640 1 -- 172.21.15.37:0/2683985067 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7fc06f960 msgr2=0x7fe7fc06fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:39.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.079+0000 7fe801d8b640 1 --2- 172.21.15.37:0/2683985067 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7fc06f960 0x7fe7fc06fd40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe7f800e4d0 tx=0x7fe7f800e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:39.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.079+0000 7fe7f27fc640 1 -- 172.21.15.37:0/2683985067 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f801a070 con 0x7fe7fc06f960 2024-04-26T00:21:39.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.079+0000 7fe7f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:39.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.079+0000 7fe7f27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:21:39.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.079+0000 7fe7f27fc640 10 monclient: dump: 2024-04-26T00:21:39.066 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:39.066 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:39.066 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:39.066 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:39.066 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:39.066 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:39.066 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:39.066 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:39.066 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:39.066 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:39.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.079+0000 7fe7f27fc640 1 -- 172.21.15.37:0/2683985067 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe7f8007450 con 0x7fe7fc06f960 2024-04-26T00:21:39.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.079+0000 7fe7f27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:39.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.079+0000 7fe7f27fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:21:39.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.079+0000 7fe7f27fc640 1 -- 172.21.15.37:0/2683985067 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7fc06f960 msgr2=0x7fe7fc06fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:39.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.079+0000 7fe7f27fc640 1 --2- 172.21.15.37:0/2683985067 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7fc06f960 0x7fe7fc06fd40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe7f800e4d0 tx=0x7fe7f800e990 comp rx=0 tx=0).stop 2024-04-26T00:21:39.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.079+0000 7fe7f27fc640 1 -- 172.21.15.37:0/2683985067 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7fc06f960 msgr2=0x7fe7fc06fd40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:21:39.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.079+0000 7fe7f27fc640 1 --2- 172.21.15.37:0/2683985067 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7fc06f960 0x7fe7fc06fd40 secure :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe7f800e4d0 tx=0x7fe7f800e990 comp rx=0 tx=0).stop 2024-04-26T00:21:39.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.079+0000 7fe80158a640 1 --2- 172.21.15.37:0/2683985067 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe7e0035e80 0x7fe7e0038340 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec007ca0 tx=0x7fe7ec007bf0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:39.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.083+0000 7fe803fef640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:21:39.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.083+0000 7fe803fef640 1 --2- 172.21.15.37:0/2683985067 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7fc070280 0x7fe7fc1337d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:39.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.083+0000 7fe803fef640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:39.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.083+0000 7fe803fef640 1 -- 172.21.15.37:0/2683985067 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe7fc06ca10 con 0x7fe7fc070280 2024-04-26T00:21:39.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.083+0000 7fe801d8b640 1 --2- 172.21.15.37:0/2683985067 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7fc070280 0x7fe7fc1337d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:39.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.083+0000 7fe801d8b640 10 monclient: get_auth_request con 0x7fe7fc070280 auth_method 0 2024-04-26T00:21:39.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.083+0000 7fe801d8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:39.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.083+0000 7fe801d8b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:39.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.083+0000 7fe801d8b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:39.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.083+0000 7fe801d8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:39.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.083+0000 7fe801d8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:39.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.083+0000 7fe801d8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:39.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.083+0000 7fe801d8b640 10 monclient(hunting): handle_auth_done global_id 4539 payload 824 2024-04-26T00:21:39.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.083+0000 7fe801d8b640 1 -- 172.21.15.37:0/2683985067 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe7fc0727d0 con 0x7fe7fc070280 2024-04-26T00:21:39.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.087+0000 7fe801d8b640 1 -- 172.21.15.37:0/2683985067 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7fc070280 msgr2=0x7fe7fc1337d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:39.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.087+0000 7fe801d8b640 1 --2- 172.21.15.37:0/2683985067 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7fc070280 0x7fe7fc1337d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe7f800e4d0 tx=0x7fe7f8003770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:39.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.087+0000 7fe7f27fc640 1 -- 172.21.15.37:0/2683985067 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f801a030 con 0x7fe7fc070280 2024-04-26T00:21:39.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.087+0000 7fe7f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:39.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.087+0000 7fe7f27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:21:39.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.087+0000 7fe7f27fc640 10 monclient: dump: 2024-04-26T00:21:39.069 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:39.069 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:39.069 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:39.069 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:39.069 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:39.069 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:39.069 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:39.069 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:39.069 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:39.069 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:39.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.087+0000 7fe7f27fc640 1 -- 172.21.15.37:0/2683985067 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4889 (secure 0 0 0) 0x7fe7f801b040 con 0x7fe7fc070280 2024-04-26T00:21:39.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.087+0000 7fe7f27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:21:39.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.087+0000 7fe7f27fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:21:39.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.087+0000 7fe7f27fc640 1 -- 172.21.15.37:0/2683985067 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7fc070280 msgr2=0x7fe7fc1337d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:39.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.087+0000 7fe7f27fc640 1 --2- 172.21.15.37:0/2683985067 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7fc070280 0x7fe7fc1337d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe7f800e4d0 tx=0x7fe7f8003770 comp rx=0 tx=0).stop 2024-04-26T00:21:39.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.087+0000 7fe7f27fc640 1 -- 172.21.15.37:0/2683985067 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7fc070280 msgr2=0x7fe7fc1337d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:21:39.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.087+0000 7fe7f27fc640 1 --2- 172.21.15.37:0/2683985067 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7fc070280 0x7fe7fc1337d0 secure :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe7f800e4d0 tx=0x7fe7f8003770 comp rx=0 tx=0).stop 2024-04-26T00:21:39.070 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:21:39.070 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:21:39.070 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:21:39.070 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:21:39.070 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 18, 2024-04-26T00:21:39.071 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 116190, 2024-04-26T00:21:39.071 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:21:39.071 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:21:39.071 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:21:39.071 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:21:39.071 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:39.071 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 1, 2024-04-26T00:21:39.071 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:21:39.071 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:21:39.071 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:21:39.071 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:21:39.071 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:21:39.071 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:21:39.071 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:21:39.071 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:21:39.072 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:21:39.072 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:21:39.072 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:21:39.072 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:21:39.072 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:21:39.072 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:21:39.072 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:39.072 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:21:39.072 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:21:39.072 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:21:39.072 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:21:39.072 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:21:39.072 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:21:39.072 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:21:39.073 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:21:39.073 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:21:39.073 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:21:39.073 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:21:39.073 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:21:39.073 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:21:39.073 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:39.073 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:39.073 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:21:39.073 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:21:39.073 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:21:39.073 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:21:39.073 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:21:39.073 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:21:39.073 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:21:39.074 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:21:39.074 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:21:39.074 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:21:39.074 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:21:39.074 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:21:39.074 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:21:39.074 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:21:39.074 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:21:39.074 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:21:39.074 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:21:39.074 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:21:39.074 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:21:39.074 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:21:39.074 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:21:39.075 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:21:39.075 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:21:39.075 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:21:39.075 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:21:39.075 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:21:39.075 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:21:39.075 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:21:39.075 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:39.075 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:21:39.075 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:39.075 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:21:39.075 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:39.075 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:21:39.075 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:21:39.075 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:21:39.076 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:21:39.076 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:39.076 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:21:39.076 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:21:39.076 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:39.076 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:39.076 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:39.076 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:21:39.076 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:21:39.076 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:39.076 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:39.076 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:39.076 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:39.076 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:21:39.076 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:21:39.077 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:21:39.077 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:21:39.077 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:21:39.077 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:39.077 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:39.077 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:21:39.077 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:21:39.077 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:21:39.077 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:21:39.077 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:39.077 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:21:39.077 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:21:39.077 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:39.077 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:39.077 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:39.078 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:21:39.078 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:21:39.078 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:39.078 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:39.078 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:39.078 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:39.078 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:21:39.078 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:21:39.078 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:21:39.078 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:21:39.078 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:21:39.078 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:39.078 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:39.079 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:21:39.079 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:21:39.079 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:21:39.079 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:21:39.079 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:39.079 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:21:39.079 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:21:39.079 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:39.079 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:39.079 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:39.079 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:21:39.079 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:21:39.079 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:39.079 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:39.080 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:39.080 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:39.080 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:21:39.080 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:21:39.080 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:21:39.080 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:21:39.080 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:21:39.080 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:39.080 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:39.080 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:39.080 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:21:39.080 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:21:39.080 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:39.080 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:21:39.081 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:21:39.081 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:21:39.081 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:39.081 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:39.081 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:21:39.081 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:39.081 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:21:39.081 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:21:39.081 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:21:39.081 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:39.081 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:39.081 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:21:39.081 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:39.081 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:21:39.081 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:21:39.082 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:21:39.082 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:39.082 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:39.082 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:21:39.082 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:39.082 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:21:39.082 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:21:39.082 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:21:39.082 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:39.082 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:39.082 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:39.082 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:21:39.082 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:21:39.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.087+0000 7fe803fef640 1 -- 172.21.15.37:0/2683985067 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe7e0035e80 msgr2=0x7fe7e0038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:39.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.087+0000 7fe803fef640 1 --2- 172.21.15.37:0/2683985067 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe7e0035e80 0x7fe7e0038340 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec007ca0 tx=0x7fe7ec007bf0 comp rx=0 tx=0).stop 2024-04-26T00:21:39.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.087+0000 7fe803fef640 10 monclient: shutdown 2024-04-26T00:21:39.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.087+0000 7fe803fef640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:39.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.087+0000 7fe803fef640 1 -- 172.21.15.37:0/2683985067 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7fc13b690 msgr2=0x7fe7fc06f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:39.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.087+0000 7fe803fef640 1 --2- 172.21.15.37:0/2683985067 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7fc13b690 0x7fe7fc06f420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe7f4009d90 tx=0x7fe7f4019550 comp rx=0 tx=0).stop 2024-04-26T00:21:39.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.087+0000 7fe80258c640 1 -- 172.21.15.37:0/2683985067 reap_dead start 2024-04-26T00:21:39.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.087+0000 7fe80258c640 1 -- 172.21.15.37:0/2683985067 reap_dead start 2024-04-26T00:21:39.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.087+0000 7fe803fef640 1 -- 172.21.15.37:0/2683985067 shutdown_connections 2024-04-26T00:21:39.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.087+0000 7fe803fef640 1 -- 172.21.15.37:0/2683985067 >> 172.21.15.37:0/2683985067 conn(0x7fe7fc0a2c50 msgr2=0x7fe7fc13ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:39.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.087+0000 7fe803fef640 1 -- 172.21.15.37:0/2683985067 shutdown_connections 2024-04-26T00:21:39.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.087+0000 7fe803fef640 1 -- 172.21.15.37:0/2683985067 wait complete. 2024-04-26T00:21:39.085 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:21:39.085 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:21:39.085 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:21:39.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.223+0000 7f5e97a22640 10 monclient: get_monmap_and_config 2024-04-26T00:21:39.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.223+0000 7f5e97a22640 10 monclient: build_initial_monmap 2024-04-26T00:21:39.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.223+0000 7f5e97a22640 10 monclient: monmap: 2024-04-26T00:21:39.203 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:39.203 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:39.203 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:21:39.227786+0000 2024-04-26T00:21:39.203 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:21:39.227786+0000 2024-04-26T00:21:39.203 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:39.204 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:39.204 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:21:39.204 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:21:39.204 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:21:39.204 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:21:39.204 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:21:39.204 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:39.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e97a22640 1 Processor -- start 2024-04-26T00:21:39.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e97a22640 1 -- start start 2024-04-26T00:21:39.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e97a22640 10 monclient: init 2024-04-26T00:21:39.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e97a22640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:39.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e97a22640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-04-26T00:21:39.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e97a22640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f5e900a2fe0 0x7f5e901351b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:39.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e97a22640 10 monclient(hunting): picked mon.noname-a con 0x7f5e900a2fe0 addr v2:172.21.15.37:3300/0 2024-04-26T00:21:39.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e97a22640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e901356f0 0x7f5e900a13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:39.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e97a22640 10 monclient(hunting): picked mon.noname-b con 0x7f5e901356f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:21:39.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e97a22640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5e900a1910 addr v1:172.21.15.37:6789/0 2024-04-26T00:21:39.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e97a22640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e90072ec0 con 0x7f5e900a1910 2024-04-26T00:21:39.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e97a22640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:39.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e97a22640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f5e900a33c0 con 0x7f5e900a2fe0 2024-04-26T00:21:39.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e97a22640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:39.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e97a22640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5e90071f00 con 0x7f5e901356f0 2024-04-26T00:21:39.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e97a22640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:39.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e97a22640 10 monclient(hunting): authenticate will time out at 1080.533677s 2024-04-26T00:21:39.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e957be640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f5e900a2fe0 0x7f5e901351b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:39.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e95fbf640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f5e900a1910 0x7f5e9013bdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:39988/0 (socket says 172.21.15.37:39988) 2024-04-26T00:21:39.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e94fbd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e901356f0 0x7f5e900a13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:39.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e95fbf640 1 -- 172.21.15.37:0/3704038815 learned_addr learned my addr 172.21.15.37:0/3704038815 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:21:39.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e957be640 10 monclient(hunting): get_auth_request con 0x7f5e900a2fe0 auth_method 0 2024-04-26T00:21:39.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e957be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:39.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e957be640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:39.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e957be640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:39.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e94fbd640 10 monclient(hunting): get_auth_request con 0x7f5e901356f0 auth_method 0 2024-04-26T00:21:39.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e94fbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:39.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e94fbd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:39.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e94fbd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:39.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e957be640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:39.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e957be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:39.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e957be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:39.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e94fbd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:39.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e94fbd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:39.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e94fbd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:39.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e957be640 10 monclient(hunting): handle_auth_done global_id 4542 payload 274 2024-04-26T00:21:39.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e957be640 1 -- 172.21.15.37:0/3704038815 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e901356f0 msgr2=0x7f5e900a13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:21:39.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e957be640 1 --2- 172.21.15.37:0/3704038815 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e901356f0 0x7f5e900a13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:39.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e957be640 1 -- 172.21.15.37:0/3704038815 >> v1:172.21.15.37:6789/0 conn(0x7f5e900a1910 legacy=0x7f5e9013bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:21:39.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e957be640 10 monclient: _finish_hunting 0 2024-04-26T00:21:39.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e957be640 1 monclient: found mon.noname-a 2024-04-26T00:21:39.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e957be640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:39.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e957be640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:21:39.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.227+0000 7f5e957be640 1 -- 172.21.15.37:0/3704038815 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e90134ac0 con 0x7f5e900a2fe0 2024-04-26T00:21:39.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.231+0000 7f5e957be640 1 -- 172.21.15.37:0/3704038815 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e900a2fe0 msgr2=0x7f5e901351b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:39.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.231+0000 7f5e957be640 1 --2- 172.21.15.37:0/3704038815 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e900a2fe0 0x7f5e901351b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c0097c0 tx=0x7f5e8c01c1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53dcb70c2f796a16 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:39.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.231+0000 7f5e87fff640 1 -- 172.21.15.37:0/3704038815 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e8c0b0070 con 0x7f5e900a2fe0 2024-04-26T00:21:39.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.231+0000 7f5e87fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:39.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.231+0000 7f5e87fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:21:39.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.231+0000 7f5e87fff640 10 monclient: dump: 2024-04-26T00:21:39.211 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:39.211 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:39.211 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:39.211 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:39.211 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:39.211 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:39.211 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:39.211 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:39.211 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:39.211 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:39.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.231+0000 7f5e87fff640 10 monclient: _finish_auth 0 2024-04-26T00:21:39.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.231+0000 7f5e87fff640 10 monclient: _check_auth_tickets 2024-04-26T00:21:39.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.231+0000 7f5e87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:39.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.231+0000 7f5e97a22640 5 monclient: authenticate success, global_id 4542 2024-04-26T00:21:39.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.231+0000 7f5e97a22640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:21:39.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.231+0000 7f5e87fff640 1 -- 172.21.15.37:0/3704038815 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e8c0ab600 con 0x7f5e900a2fe0 2024-04-26T00:21:39.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.231+0000 7f5e87fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:39.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.231+0000 7f5e87fff640 1 -- 172.21.15.37:0/3704038815 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e8c0ab9d0 con 0x7f5e900a2fe0 2024-04-26T00:21:39.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.231+0000 7f5e87fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:39.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.231+0000 7f5e87fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:21:39.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.231+0000 7f5e87fff640 10 monclient: dump: 2024-04-26T00:21:39.212 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:39.212 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:39.212 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:39.212 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:39.212 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:39.213 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:39.213 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:39.213 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:39.213 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:39.213 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:39.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.231+0000 7f5e97a22640 10 monclient: get_monmap_and_config success 2024-04-26T00:21:39.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.231+0000 7f5e97a22640 4 set_mon_vals no callback set 2024-04-26T00:21:39.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.231+0000 7f5e97a22640 10 monclient: shutdown 2024-04-26T00:21:39.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.231+0000 7f5e97a22640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:39.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.231+0000 7f5e97a22640 1 -- 172.21.15.37:0/3704038815 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e900a2fe0 msgr2=0x7f5e901351b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:39.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.231+0000 7f5e97a22640 1 --2- 172.21.15.37:0/3704038815 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e900a2fe0 0x7f5e901351b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c0097c0 tx=0x7f5e8c01c1d0 comp rx=0 tx=0).stop 2024-04-26T00:21:39.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.231+0000 7f5e97a22640 1 -- 172.21.15.37:0/3704038815 shutdown_connections 2024-04-26T00:21:39.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.231+0000 7f5e97a22640 1 --2- 172.21.15.37:0/3704038815 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e901356f0 0x7f5e900a13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:39.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.231+0000 7f5e97a22640 1 --2- 172.21.15.37:0/3704038815 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e900a2fe0 0x7f5e901351b0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:39.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.231+0000 7f5e97a22640 1 -- 172.21.15.37:0/3704038815 >> 172.21.15.37:0/3704038815 conn(0x7f5e900a24c0 msgr2=0x7f5e900a28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:39.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.231+0000 7f5e97a22640 1 -- 172.21.15.37:0/3704038815 shutdown_connections 2024-04-26T00:21:39.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.231+0000 7f5e97a22640 1 -- 172.21.15.37:0/3704038815 wait complete. 2024-04-26T00:21:39.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e97a22640 10 monclient: build_initial_monmap 2024-04-26T00:21:39.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e97a22640 10 monclient: monmap: 2024-04-26T00:21:39.218 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:39.218 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:21:39.218 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:21:39.218 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:21:39.218 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:39.218 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:21:39.218 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:21:39.218 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:21:39.218 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:21:39.219 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:39.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e97a22640 1 Processor -- start 2024-04-26T00:21:39.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e97a22640 1 -- start start 2024-04-26T00:21:39.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e97a22640 10 monclient: init 2024-04-26T00:21:39.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e97a22640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:39.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e97a22640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:21:39.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e97a22640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e900a1910 0x7f5e9013c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:39.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e97a22640 10 monclient(hunting): picked mon.noname-b con 0x7f5e900a1910 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:21:39.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e97a22640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e900a2fe0 0x7f5e9012e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:39.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e97a22640 10 monclient(hunting): picked mon.noname-c con 0x7f5e900a2fe0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:21:39.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e97a22640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e901356f0 0x7f5e9012ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:39.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e97a22640 10 monclient(hunting): picked mon.noname-a con 0x7f5e901356f0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:21:39.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e97a22640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:39.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e97a22640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f5e9006ed70 con 0x7f5e901356f0 2024-04-26T00:21:39.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e97a22640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:39.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e97a22640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5e9006ebf0 con 0x7f5e900a1910 2024-04-26T00:21:39.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e97a22640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:39.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e97a22640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5e9013fd30 con 0x7f5e900a2fe0 2024-04-26T00:21:39.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e97a22640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:39.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e97a22640 10 monclient(hunting): authenticate will time out at 1080.541162s 2024-04-26T00:21:39.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e94fbd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e900a2fe0 0x7f5e9012e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:39.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e94fbd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e900a2fe0 0x7f5e9012e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:43504/0 (socket says 172.21.15.37:43504) 2024-04-26T00:21:39.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e94fbd640 1 -- 172.21.15.37:0/351586473 learned_addr learned my addr 172.21.15.37:0/351586473 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:21:39.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e94fbd640 10 monclient(hunting): get_auth_request con 0x7f5e900a2fe0 auth_method 0 2024-04-26T00:21:39.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e94fbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:39.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e94fbd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:39.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e94fbd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:39.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e95fbf640 1 --2- 172.21.15.37:0/351586473 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e901356f0 0x7f5e9012ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:39.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e94fbd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:39.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e94fbd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:39.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e94fbd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:39.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e95fbf640 10 monclient(hunting): get_auth_request con 0x7f5e901356f0 auth_method 0 2024-04-26T00:21:39.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e95fbf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:39.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e95fbf640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:39.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e95fbf640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:39.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e95fbf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:39.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e95fbf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:39.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e95fbf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:39.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e94fbd640 10 monclient(hunting): handle_auth_done global_id 4526 payload 824 2024-04-26T00:21:39.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e957be640 1 --2- 172.21.15.37:0/351586473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e900a1910 0x7f5e9013c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:39.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e94fbd640 1 -- 172.21.15.37:0/351586473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e900a1910 msgr2=0x7f5e9013c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:39.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e94fbd640 1 --2- 172.21.15.37:0/351586473 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e900a1910 0x7f5e9013c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:39.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e94fbd640 1 -- 172.21.15.37:0/351586473 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e901356f0 msgr2=0x7f5e9012ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:39.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e94fbd640 1 --2- 172.21.15.37:0/351586473 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e901356f0 0x7f5e9012ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:39.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e94fbd640 10 monclient: _finish_hunting 0 2024-04-26T00:21:39.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e94fbd640 1 monclient: found mon.noname-c 2024-04-26T00:21:39.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e94fbd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:39.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e94fbd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:21:39.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e94fbd640 1 -- 172.21.15.37:0/351586473 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e90139fc0 con 0x7f5e900a2fe0 2024-04-26T00:21:39.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e94fbd640 1 -- 172.21.15.37:0/351586473 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e900a2fe0 msgr2=0x7f5e9012e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:39.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.235+0000 7f5e94fbd640 1 --2- 172.21.15.37:0/351586473 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e900a2fe0 0x7f5e9012e4f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5e8000e9b0 tx=0x7f5e8000ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:39.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e85ffb640 1 -- 172.21.15.37:0/351586473 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e80017070 con 0x7f5e900a2fe0 2024-04-26T00:21:39.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e85ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:39.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e85ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:21:39.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e85ffb640 10 monclient: dump: 2024-04-26T00:21:39.222 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:39.222 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:39.222 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:39.222 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:39.223 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:39.223 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:39.223 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:39.223 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:39.223 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:39.223 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:39.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e85ffb640 10 monclient: _finish_auth 0 2024-04-26T00:21:39.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e85ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:21:39.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:39.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e97a22640 5 monclient: authenticate success, global_id 4526 2024-04-26T00:21:39.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e97a22640 10 monclient: _renew_subs 2024-04-26T00:21:39.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e85ffb640 1 -- 172.21.15.37:0/351586473 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e8000cbb0 con 0x7f5e900a2fe0 2024-04-26T00:21:39.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e97a22640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:21:39.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e97a22640 1 -- 172.21.15.37:0/351586473 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e9013b330 con 0x7f5e900a2fe0 2024-04-26T00:21:39.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e85ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:39.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e85ffb640 1 -- 172.21.15.37:0/351586473 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e80013470 con 0x7f5e900a2fe0 2024-04-26T00:21:39.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e85ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:39.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e85ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:21:39.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e85ffb640 10 monclient: dump: 2024-04-26T00:21:39.224 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:39.224 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:39.224 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:39.224 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:39.224 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:39.224 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:39.224 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:39.224 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:39.224 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:39.224 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:39.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e97a22640 10 monclient: _renew_subs 2024-04-26T00:21:39.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e97a22640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:21:39.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e877fe640 4 set_mon_vals no callback set 2024-04-26T00:21:39.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e97a22640 1 -- 172.21.15.37:0/351586473 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e9013ca60 con 0x7f5e900a2fe0 2024-04-26T00:21:39.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e97a22640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:39.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e97a22640 1 --2- 172.21.15.37:0/351586473 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e9013b830 0x7f5e901cf230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:39.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e97a22640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:39.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e97a22640 1 -- 172.21.15.37:0/351586473 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5e9013c890 con 0x7f5e9013b830 2024-04-26T00:21:39.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e957be640 1 --2- 172.21.15.37:0/351586473 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e9013b830 0x7f5e901cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:39.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e957be640 10 monclient: get_auth_request con 0x7f5e9013b830 auth_method 0 2024-04-26T00:21:39.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e957be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:39.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e957be640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:39.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e957be640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:39.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e85ffb640 1 -- 172.21.15.37:0/351586473 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f5e8001a070 con 0x7f5e900a2fe0 2024-04-26T00:21:39.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e957be640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:39.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e957be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:39.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e957be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:39.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e957be640 10 monclient(hunting): handle_auth_done global_id 4529 payload 824 2024-04-26T00:21:39.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e957be640 1 -- 172.21.15.37:0/351586473 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5e9004f510 con 0x7f5e9013b830 2024-04-26T00:21:39.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e85ffb640 1 --2- 172.21.15.37:0/351586473 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5e74035e80 0x7f5e74038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:39.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e85ffb640 1 -- 172.21.15.37:0/351586473 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(75..75 src has 1..75) ==== 3963+0+0 (secure 0 0 0) 0x7f5e80048c10 con 0x7f5e900a2fe0 2024-04-26T00:21:39.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e95fbf640 1 --2- 172.21.15.37:0/351586473 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5e74035e80 0x7f5e74038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:39.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e95fbf640 10 monclient: get_auth_request con 0x7f5e74035e80 auth_method 0 2024-04-26T00:21:39.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e957be640 1 -- 172.21.15.37:0/351586473 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e9013b830 msgr2=0x7f5e901cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:39.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e957be640 1 --2- 172.21.15.37:0/351586473 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e9013b830 0x7f5e901cf230 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c0ad000 tx=0x7f5e8c01aa10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:39.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e85ffb640 1 -- 172.21.15.37:0/351586473 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e8c0bb020 con 0x7f5e9013b830 2024-04-26T00:21:39.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e85ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:39.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e85ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:21:39.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.239+0000 7f5e85ffb640 10 monclient: dump: 2024-04-26T00:21:39.228 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:39.229 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:39.229 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:39.229 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:39.229 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:39.229 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:39.229 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:39.229 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:39.229 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:39.229 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:39.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.243+0000 7f5e85ffb640 1 -- 172.21.15.37:0/351586473 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5e8c0abe30 con 0x7f5e9013b830 2024-04-26T00:21:39.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.243+0000 7f5e85ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:39.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.243+0000 7f5e85ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:21:39.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.243+0000 7f5e85ffb640 1 -- 172.21.15.37:0/351586473 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e9013b830 msgr2=0x7f5e901cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:39.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.243+0000 7f5e85ffb640 1 --2- 172.21.15.37:0/351586473 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e9013b830 0x7f5e901cf230 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c0ad000 tx=0x7f5e8c01aa10 comp rx=0 tx=0).stop 2024-04-26T00:21:39.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.243+0000 7f5e85ffb640 1 -- 172.21.15.37:0/351586473 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e9013b830 msgr2=0x7f5e901cf230 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:21:39.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.243+0000 7f5e85ffb640 1 --2- 172.21.15.37:0/351586473 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e9013b830 0x7f5e901cf230 secure :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c0ad000 tx=0x7f5e8c01aa10 comp rx=0 tx=0).stop 2024-04-26T00:21:39.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.243+0000 7f5e95fbf640 1 --2- 172.21.15.37:0/351586473 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5e74035e80 0x7f5e74038340 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5e8800c490 tx=0x7f5e88009b10 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:39.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.247+0000 7f5e97a22640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:21:39.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.247+0000 7f5e97a22640 1 --2- 172.21.15.37:0/351586473 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e901cf770 0x7f5e901cfb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:39.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.247+0000 7f5e97a22640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:39.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.247+0000 7f5e97a22640 1 -- 172.21.15.37:0/351586473 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5e901d00c0 con 0x7f5e901cf770 2024-04-26T00:21:39.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.247+0000 7f5e957be640 1 --2- 172.21.15.37:0/351586473 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e901cf770 0x7f5e901cfb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:39.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.247+0000 7f5e957be640 10 monclient: get_auth_request con 0x7f5e901cf770 auth_method 0 2024-04-26T00:21:39.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.247+0000 7f5e957be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:39.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.247+0000 7f5e957be640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:39.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.247+0000 7f5e957be640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:39.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.247+0000 7f5e957be640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:39.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.247+0000 7f5e957be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:39.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.247+0000 7f5e957be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:39.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.247+0000 7f5e957be640 10 monclient(hunting): handle_auth_done global_id 4532 payload 824 2024-04-26T00:21:39.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.247+0000 7f5e957be640 1 -- 172.21.15.37:0/351586473 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5e9013b500 con 0x7f5e901cf770 2024-04-26T00:21:39.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.251+0000 7f5e957be640 1 -- 172.21.15.37:0/351586473 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e901cf770 msgr2=0x7f5e901cfb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:39.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.251+0000 7f5e957be640 1 --2- 172.21.15.37:0/351586473 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e901cf770 0x7f5e901cfb80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c0097c0 tx=0x7f5e8c018f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:39.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.251+0000 7f5e85ffb640 1 -- 172.21.15.37:0/351586473 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e8c0bb030 con 0x7f5e901cf770 2024-04-26T00:21:39.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.251+0000 7f5e85ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:39.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.251+0000 7f5e85ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:21:39.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.251+0000 7f5e85ffb640 10 monclient: dump: 2024-04-26T00:21:39.231 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:39.231 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:39.231 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:39.231 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:39.231 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:39.231 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:39.232 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:39.232 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:39.232 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:39.232 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:39.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.251+0000 7f5e85ffb640 1 -- 172.21.15.37:0/351586473 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4717 (secure 0 0 0) 0x7f5e8c0c2860 con 0x7f5e901cf770 2024-04-26T00:21:39.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.251+0000 7f5e85ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:21:39.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.251+0000 7f5e85ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:21:39.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.251+0000 7f5e85ffb640 1 -- 172.21.15.37:0/351586473 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e901cf770 msgr2=0x7f5e901cfb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:39.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.251+0000 7f5e85ffb640 1 --2- 172.21.15.37:0/351586473 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e901cf770 0x7f5e901cfb80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c0097c0 tx=0x7f5e8c018f70 comp rx=0 tx=0).stop 2024-04-26T00:21:39.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.251+0000 7f5e85ffb640 1 -- 172.21.15.37:0/351586473 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e901cf770 msgr2=0x7f5e901cfb80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:21:39.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.251+0000 7f5e85ffb640 1 --2- 172.21.15.37:0/351586473 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e901cf770 0x7f5e901cfb80 secure :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c0097c0 tx=0x7f5e8c018f70 comp rx=0 tx=0).stop 2024-04-26T00:21:39.232 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:21:39.233 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:21:39.233 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:21:39.233 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:21:39.233 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 18, 2024-04-26T00:21:39.233 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 116358, 2024-04-26T00:21:39.233 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:21:39.233 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:21:39.233 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:21:39.233 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:21:39.233 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:39.233 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 1, 2024-04-26T00:21:39.233 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:21:39.233 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:21:39.233 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:21:39.234 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:21:39.234 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:21:39.234 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:21:39.234 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:21:39.234 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:21:39.234 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:21:39.234 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:21:39.234 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:21:39.234 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:21:39.234 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:21:39.234 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:21:39.234 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:39.234 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:21:39.235 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:21:39.235 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:21:39.235 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:21:39.235 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:21:39.235 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:21:39.235 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:21:39.235 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:21:39.235 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:21:39.235 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:21:39.235 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:21:39.235 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:21:39.235 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:21:39.235 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:39.235 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:39.235 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:21:39.236 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:21:39.236 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:21:39.236 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:21:39.236 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:21:39.236 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:21:39.236 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:21:39.236 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:21:39.236 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:21:39.236 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:21:39.236 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:21:39.236 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:21:39.236 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:21:39.236 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:21:39.236 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:21:39.236 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:21:39.236 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:21:39.237 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:21:39.237 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:21:39.237 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:21:39.237 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:21:39.237 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:21:39.237 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:21:39.237 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:21:39.237 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:21:39.237 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:21:39.237 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:21:39.237 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:21:39.237 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:39.237 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:21:39.237 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:39.238 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:21:39.238 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:39.238 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:21:39.238 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:21:39.238 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:21:39.238 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:21:39.238 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:39.238 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:21:39.238 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:21:39.238 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:39.238 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:39.238 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:39.238 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:21:39.238 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:21:39.239 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:39.239 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:39.239 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:39.239 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:39.239 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:21:39.239 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:21:39.239 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:21:39.239 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:21:39.239 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:21:39.239 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:39.239 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:39.239 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:21:39.239 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:21:39.239 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:21:39.239 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:21:39.240 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:39.240 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:21:39.240 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:21:39.240 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:39.240 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:39.240 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:39.240 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:21:39.240 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:21:39.240 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:39.240 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:39.240 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:39.241 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:39.241 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:21:39.241 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:21:39.241 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:21:39.241 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:21:39.241 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:21:39.241 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:39.241 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:39.241 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:21:39.241 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:21:39.241 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:21:39.241 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:21:39.241 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:39.241 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:21:39.241 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:21:39.241 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:39.242 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:39.242 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:39.242 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:21:39.242 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:21:39.242 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:39.242 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:39.242 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:39.242 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:39.242 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:21:39.242 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:21:39.242 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:21:39.242 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:21:39.242 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:21:39.242 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:39.242 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:39.243 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:39.243 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:21:39.243 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:21:39.243 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:39.243 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:21:39.243 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:21:39.243 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:21:39.243 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:39.243 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:39.243 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:21:39.243 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:39.243 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:21:39.243 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:21:39.243 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:21:39.243 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:39.244 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:39.244 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:21:39.244 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:39.244 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:21:39.244 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:21:39.244 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:21:39.244 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:39.244 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:39.244 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:39.244 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:21:39.244 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:21:39.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.251+0000 7f5e97a22640 1 -- 172.21.15.37:0/351586473 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5e74035e80 msgr2=0x7f5e74038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:39.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.251+0000 7f5e97a22640 1 --2- 172.21.15.37:0/351586473 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5e74035e80 0x7f5e74038340 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5e8800c490 tx=0x7f5e88009b10 comp rx=0 tx=0).stop 2024-04-26T00:21:39.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.251+0000 7f5e97a22640 10 monclient: shutdown 2024-04-26T00:21:39.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.251+0000 7f5e97a22640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:39.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.251+0000 7f5e97a22640 1 -- 172.21.15.37:0/351586473 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e900a2fe0 msgr2=0x7f5e9012e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:39.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.251+0000 7f5e97a22640 1 --2- 172.21.15.37:0/351586473 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e900a2fe0 0x7f5e9012e4f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5e8000e9b0 tx=0x7f5e8000ee70 comp rx=0 tx=0).stop 2024-04-26T00:21:39.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.251+0000 7f5e95fbf640 1 -- 172.21.15.37:0/351586473 reap_dead start 2024-04-26T00:21:39.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.251+0000 7f5e97a22640 1 -- 172.21.15.37:0/351586473 shutdown_connections 2024-04-26T00:21:39.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.251+0000 7f5e97a22640 1 --2- 172.21.15.37:0/351586473 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e900a2fe0 0x7f5e9012e4f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:39.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.251+0000 7f5e97a22640 1 -- 172.21.15.37:0/351586473 >> 172.21.15.37:0/351586473 conn(0x7f5e900a24c0 msgr2=0x7f5e900a4720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:39.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.251+0000 7f5e97a22640 1 -- 172.21.15.37:0/351586473 shutdown_connections 2024-04-26T00:21:39.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.251+0000 7f5e97a22640 1 -- 172.21.15.37:0/351586473 wait complete. 2024-04-26T00:21:39.247 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:21:39.247 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:21:39.247 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:21:39.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31faa4b640 10 monclient: get_monmap_and_config 2024-04-26T00:21:39.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31faa4b640 10 monclient: build_initial_monmap 2024-04-26T00:21:39.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31faa4b640 10 monclient: monmap: 2024-04-26T00:21:39.356 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:39.356 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:39.356 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:21:39.380461+0000 2024-04-26T00:21:39.356 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:21:39.380461+0000 2024-04-26T00:21:39.356 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:39.356 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:39.357 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:21:39.357 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:21:39.357 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:21:39.357 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:21:39.357 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:21:39.357 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:39.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31faa4b640 1 Processor -- start 2024-04-26T00:21:39.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31faa4b640 1 -- start start 2024-04-26T00:21:39.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31faa4b640 10 monclient: init 2024-04-26T00:21:39.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31faa4b640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:39.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31faa4b640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-04-26T00:21:39.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31faa4b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f31f409c780 0x7f31f409cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:39.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31faa4b640 10 monclient(hunting): picked mon.noname-c con 0x7f31f409c780 addr v2:172.21.15.45:3300/0 2024-04-26T00:21:39.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31faa4b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f31f4131960 addr v1:172.21.15.37:6789/0 2024-04-26T00:21:39.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31faa4b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f31f4131da0 0x7f31f40a2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:39.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31faa4b640 10 monclient(hunting): picked mon.noname-b con 0x7f31f4131da0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:21:39.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31faa4b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31f406ba00 con 0x7f31f4131960 2024-04-26T00:21:39.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31faa4b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:39.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31faa4b640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f31f406bc70 con 0x7f31f409c780 2024-04-26T00:21:39.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31faa4b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:39.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31faa4b640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f31f406bf50 con 0x7f31f4131da0 2024-04-26T00:21:39.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31faa4b640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:39.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31faa4b640 10 monclient(hunting): authenticate will time out at 1080.685970s 2024-04-26T00:21:39.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31f37fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f31f4131960 0x7f31f40a1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:39990/0 (socket says 172.21.15.37:39990) 2024-04-26T00:21:39.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31f37fe640 1 -- 172.21.15.37:0/2126966661 learned_addr learned my addr 172.21.15.37:0/2126966661 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:21:39.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31f8fe8640 1 --2- 172.21.15.37:0/2126966661 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f31f4131da0 0x7f31f40a2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:39.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31f8fe8640 10 monclient(hunting): get_auth_request con 0x7f31f4131da0 auth_method 0 2024-04-26T00:21:39.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31f8fe8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:39.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31f8fe8640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:39.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31f8fe8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:39.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31f8fe8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:39.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31f8fe8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:39.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31f8fe8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:39.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31f3fff640 1 --2- 172.21.15.37:0/2126966661 >> v2:172.21.15.45:3300/0 conn(0x7f31f409c780 0x7f31f409cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:39.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31f3fff640 10 monclient(hunting): get_auth_request con 0x7f31f409c780 auth_method 0 2024-04-26T00:21:39.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:39.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31f3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:39.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:39.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31f8fe8640 10 monclient(hunting): handle_auth_done global_id 4535 payload 274 2024-04-26T00:21:39.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31f8fe8640 1 -- 172.21.15.37:0/2126966661 >> v2:172.21.15.45:3300/0 conn(0x7f31f409c780 msgr2=0x7f31f409cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:21:39.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31f8fe8640 1 --2- 172.21.15.37:0/2126966661 >> v2:172.21.15.45:3300/0 conn(0x7f31f409c780 0x7f31f409cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:39.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31f8fe8640 1 -- 172.21.15.37:0/2126966661 >> v1:172.21.15.37:6789/0 conn(0x7f31f4131960 legacy=0x7f31f40a1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:21:39.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31f8fe8640 10 monclient: _finish_hunting 0 2024-04-26T00:21:39.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31f8fe8640 1 monclient: found mon.noname-b 2024-04-26T00:21:39.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31f8fe8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:39.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31f8fe8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:21:39.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.379+0000 7f31f8fe8640 1 -- 172.21.15.37:0/2126966661 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31f40a0010 con 0x7f31f4131da0 2024-04-26T00:21:39.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.383+0000 7f31f8fe8640 1 -- 172.21.15.37:0/2126966661 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f31f4131da0 msgr2=0x7f31f40a2270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:39.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.383+0000 7f31f8fe8640 1 --2- 172.21.15.37:0/2126966661 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f31f4131da0 0x7f31f40a2270 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f31ec009d90 tx=0x7f31ec01c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cab5511060e840b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:39.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.383+0000 7f31f2ffd640 1 -- 172.21.15.37:0/2126966661 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31ec0b1070 con 0x7f31f4131da0 2024-04-26T00:21:39.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.383+0000 7f31f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:39.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.383+0000 7f31f2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:21:39.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.383+0000 7f31f2ffd640 10 monclient: dump: 2024-04-26T00:21:39.371 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:39.372 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:39.372 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:39.372 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:39.372 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:39.372 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:39.372 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:39.372 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:39.372 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:39.372 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:39.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.383+0000 7f31f2ffd640 10 monclient: _finish_auth 0 2024-04-26T00:21:39.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.383+0000 7f31f2ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:21:39.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.383+0000 7f31f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:39.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.383+0000 7f31faa4b640 5 monclient: authenticate success, global_id 4535 2024-04-26T00:21:39.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.383+0000 7f31faa4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:21:39.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.383+0000 7f31f2ffd640 1 -- 172.21.15.37:0/2126966661 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31f40a29c0 con 0x7f31f4131da0 2024-04-26T00:21:39.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.383+0000 7f31f2ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:39.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.383+0000 7f31f2ffd640 1 -- 172.21.15.37:0/2126966661 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31ec0ac440 con 0x7f31f4131da0 2024-04-26T00:21:39.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.383+0000 7f31f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:39.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.383+0000 7f31f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:21:39.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.383+0000 7f31f2ffd640 10 monclient: dump: 2024-04-26T00:21:39.374 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:39.374 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:39.374 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:39.374 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:39.374 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:39.374 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:39.374 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:39.374 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:39.374 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:39.375 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:39.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.383+0000 7f31faa4b640 10 monclient: get_monmap_and_config success 2024-04-26T00:21:39.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.383+0000 7f31faa4b640 4 set_mon_vals no callback set 2024-04-26T00:21:39.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.383+0000 7f31faa4b640 10 monclient: shutdown 2024-04-26T00:21:39.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.383+0000 7f31faa4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:39.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.383+0000 7f31faa4b640 1 -- 172.21.15.37:0/2126966661 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f31f4131da0 msgr2=0x7f31f40a2270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:39.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.383+0000 7f31faa4b640 1 --2- 172.21.15.37:0/2126966661 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f31f4131da0 0x7f31f40a2270 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f31ec009d90 tx=0x7f31ec01c4d0 comp rx=0 tx=0).stop 2024-04-26T00:21:39.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.383+0000 7f31faa4b640 1 -- 172.21.15.37:0/2126966661 shutdown_connections 2024-04-26T00:21:39.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.383+0000 7f31faa4b640 1 --2- 172.21.15.37:0/2126966661 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f31f4131da0 0x7f31f40a2270 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:39.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.383+0000 7f31faa4b640 1 --2- 172.21.15.37:0/2126966661 >> v2:172.21.15.45:3300/0 conn(0x7f31f409c780 0x7f31f409cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:39.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.383+0000 7f31faa4b640 1 -- 172.21.15.37:0/2126966661 >> 172.21.15.37:0/2126966661 conn(0x7f31f40a2c50 msgr2=0x7f31f40a3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:39.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.383+0000 7f31faa4b640 1 -- 172.21.15.37:0/2126966661 shutdown_connections 2024-04-26T00:21:39.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.383+0000 7f31faa4b640 1 -- 172.21.15.37:0/2126966661 wait complete. 2024-04-26T00:21:39.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31faa4b640 10 monclient: build_initial_monmap 2024-04-26T00:21:39.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31faa4b640 10 monclient: monmap: 2024-04-26T00:21:39.376 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:39.376 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:21:39.376 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:21:39.376 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:21:39.376 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:39.376 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:21:39.377 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:21:39.377 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:21:39.377 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:21:39.377 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:39.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31faa4b640 1 Processor -- start 2024-04-26T00:21:39.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31faa4b640 1 -- start start 2024-04-26T00:21:39.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31faa4b640 10 monclient: init 2024-04-26T00:21:39.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31faa4b640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:39.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31faa4b640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:21:39.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31faa4b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f31f409c780 0x7f31f412eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:39.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31faa4b640 10 monclient(hunting): picked mon.noname-c con 0x7f31f409c780 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:21:39.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31faa4b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31f4131960 0x7f31f412b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:39.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31faa4b640 10 monclient(hunting): picked mon.noname-b con 0x7f31f4131960 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:21:39.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31faa4b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31f4131da0 0x7f31f412b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:39.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31faa4b640 10 monclient(hunting): picked mon.noname-a con 0x7f31f4131da0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:21:39.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31faa4b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:39.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31faa4b640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f31f4140310 con 0x7f31f4131da0 2024-04-26T00:21:39.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31faa4b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:39.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31faa4b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f31f4140190 con 0x7f31f4131960 2024-04-26T00:21:39.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31faa4b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:39.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31faa4b640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f31f4140490 con 0x7f31f409c780 2024-04-26T00:21:39.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31faa4b640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:39.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31faa4b640 10 monclient(hunting): authenticate will time out at 1080.693426s 2024-04-26T00:21:39.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31f3fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f31f409c780 0x7f31f412eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:39.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31f3fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f31f409c780 0x7f31f412eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:43530/0 (socket says 172.21.15.37:43530) 2024-04-26T00:21:39.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31f3fff640 1 -- 172.21.15.37:0/1919338775 learned_addr learned my addr 172.21.15.37:0/1919338775 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:21:39.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31f3fff640 10 monclient(hunting): get_auth_request con 0x7f31f409c780 auth_method 0 2024-04-26T00:21:39.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:39.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31f3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:39.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:39.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31f8fe8640 1 --2- 172.21.15.37:0/1919338775 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31f4131da0 0x7f31f412b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:39.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31f8fe8640 10 monclient(hunting): get_auth_request con 0x7f31f4131da0 auth_method 0 2024-04-26T00:21:39.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31f8fe8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:39.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31f8fe8640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:39.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31f8fe8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:39.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:39.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:39.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:39.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31f8fe8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:39.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31f8fe8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:39.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31f8fe8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:39.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31f37fe640 1 --2- 172.21.15.37:0/1919338775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31f4131960 0x7f31f412b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:39.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31f3fff640 10 monclient(hunting): handle_auth_done global_id 4538 payload 824 2024-04-26T00:21:39.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31f3fff640 1 -- 172.21.15.37:0/1919338775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31f4131960 msgr2=0x7f31f412b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:39.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31f3fff640 1 --2- 172.21.15.37:0/1919338775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31f4131960 0x7f31f412b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:39.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31f3fff640 1 -- 172.21.15.37:0/1919338775 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31f4131da0 msgr2=0x7f31f412b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:39.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31f3fff640 1 --2- 172.21.15.37:0/1919338775 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31f4131da0 0x7f31f412b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:39.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31f3fff640 10 monclient: _finish_hunting 0 2024-04-26T00:21:39.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31f3fff640 1 monclient: found mon.noname-c 2024-04-26T00:21:39.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:39.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31f3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:21:39.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31f3fff640 1 -- 172.21.15.37:0/1919338775 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31f40a0290 con 0x7f31f409c780 2024-04-26T00:21:39.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.387+0000 7f31f8fe8640 1 --2- 172.21.15.37:0/1919338775 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31f4131da0 0x7f31f412b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:21:39.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31f3fff640 1 -- 172.21.15.37:0/1919338775 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f31f409c780 msgr2=0x7f31f412eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:39.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31f3fff640 1 --2- 172.21.15.37:0/1919338775 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f31f409c780 0x7f31f412eb00 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f31e400d800 tx=0x7f31e400dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:39.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31f0ff9640 1 -- 172.21.15.37:0/1919338775 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e400b870 con 0x7f31f409c780 2024-04-26T00:21:39.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31f0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:39.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31f0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:21:39.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31f0ff9640 10 monclient: dump: 2024-04-26T00:21:39.383 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:39.383 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:39.383 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:39.383 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:39.383 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:39.384 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:39.384 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:39.384 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:39.384 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:39.384 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:39.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31f0ff9640 10 monclient: _finish_auth 0 2024-04-26T00:21:39.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31f0ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:21:39.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31f0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:39.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31f0ff9640 1 -- 172.21.15.37:0/1919338775 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31e4019070 con 0x7f31f409c780 2024-04-26T00:21:39.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31faa4b640 5 monclient: authenticate success, global_id 4538 2024-04-26T00:21:39.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31faa4b640 10 monclient: _renew_subs 2024-04-26T00:21:39.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31faa4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:21:39.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31faa4b640 1 -- 172.21.15.37:0/1919338775 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31f41307a0 con 0x7f31f409c780 2024-04-26T00:21:39.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31f0ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:39.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31f0ff9640 1 -- 172.21.15.37:0/1919338775 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e400bc10 con 0x7f31f409c780 2024-04-26T00:21:39.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31f0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:39.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31f0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:21:39.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31f0ff9640 10 monclient: dump: 2024-04-26T00:21:39.385 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:39.385 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:39.385 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:39.385 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:39.386 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:39.386 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:39.386 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:39.386 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:39.386 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:39.386 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:39.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31f27fc640 4 set_mon_vals no callback set 2024-04-26T00:21:39.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31faa4b640 10 monclient: _renew_subs 2024-04-26T00:21:39.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31faa4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:21:39.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31faa4b640 1 -- 172.21.15.37:0/1919338775 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f31f4072910 con 0x7f31f409c780 2024-04-26T00:21:39.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31faa4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:39.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31faa4b640 1 --2- 172.21.15.37:0/1919338775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31f412bc90 0x7f31f4099800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:39.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31faa4b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:39.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31faa4b640 1 -- 172.21.15.37:0/1919338775 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f31f40a29c0 con 0x7f31f412bc90 2024-04-26T00:21:39.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31f0ff9640 1 -- 172.21.15.37:0/1919338775 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f31e401e460 con 0x7f31f409c780 2024-04-26T00:21:39.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31f37fe640 1 --2- 172.21.15.37:0/1919338775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31f412bc90 0x7f31f4099800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:39.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31f0ff9640 1 --2- 172.21.15.37:0/1919338775 >> v2:172.21.15.45:6812/2379918623 conn(0x7f31e0035ed0 0x7f31e0038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:39.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31f37fe640 10 monclient: get_auth_request con 0x7f31f412bc90 auth_method 0 2024-04-26T00:21:39.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:39.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31f37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:39.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:39.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31f0ff9640 1 -- 172.21.15.37:0/1919338775 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(75..75 src has 1..75) ==== 3963+0+0 (secure 0 0 0) 0x7f31e4048f70 con 0x7f31f409c780 2024-04-26T00:21:39.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31f37fe640 1 -- 172.21.15.37:0/1919338775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31f412bc90 msgr2=0x7f31f4099800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:21:39.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31f37fe640 1 -- 172.21.15.37:0/1919338775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31f412bc90 msgr2=0x7f31f4099800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:21:39.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31f37fe640 1 --2- 172.21.15.37:0/1919338775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31f412bc90 0x7f31f4099800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:21:39.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31f37fe640 1 --2- 172.21.15.37:0/1919338775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31f412bc90 0x7f31f4099800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:21:39.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31f8fe8640 1 --2- 172.21.15.37:0/1919338775 >> v2:172.21.15.45:6812/2379918623 conn(0x7f31e0035ed0 0x7f31e0038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:39.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.391+0000 7f31f8fe8640 10 monclient: get_auth_request con 0x7f31e0035ed0 auth_method 0 2024-04-26T00:21:39.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.395+0000 7f31f8fe8640 1 --2- 172.21.15.37:0/1919338775 >> v2:172.21.15.45:6812/2379918623 conn(0x7f31e0035ed0 0x7f31e0038390 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f31f406b6b0 tx=0x7f31ec0a6070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:39.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.595+0000 7f31f37fe640 1 --2- 172.21.15.37:0/1919338775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31f412bc90 0x7f31f4099800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:39.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.595+0000 7f31f37fe640 10 monclient: get_auth_request con 0x7f31f412bc90 auth_method 0 2024-04-26T00:21:39.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.595+0000 7f31f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:39.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.595+0000 7f31f37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:39.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.595+0000 7f31f37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:21:39.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.595+0000 7f31f37fe640 1 -- 172.21.15.37:0/1919338775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31f412bc90 msgr2=0x7f31f4099800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:21:39.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.595+0000 7f31f37fe640 1 -- 172.21.15.37:0/1919338775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31f412bc90 msgr2=0x7f31f4099800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:21:39.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.595+0000 7f31f37fe640 1 --2- 172.21.15.37:0/1919338775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31f412bc90 0x7f31f4099800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:21:39.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.595+0000 7f31f37fe640 1 --2- 172.21.15.37:0/1919338775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31f412bc90 0x7f31f4099800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:21:39.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.999+0000 7f31f37fe640 1 --2- 172.21.15.37:0/1919338775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31f412bc90 0x7f31f4099800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:39.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.999+0000 7f31f37fe640 10 monclient: get_auth_request con 0x7f31f412bc90 auth_method 0 2024-04-26T00:21:39.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.999+0000 7f31f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:39.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.999+0000 7f31f37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:39.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.999+0000 7f31f37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:21:39.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.999+0000 7f31f37fe640 1 -- 172.21.15.37:0/1919338775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31f412bc90 msgr2=0x7f31f4099800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:21:39.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.999+0000 7f31f37fe640 1 -- 172.21.15.37:0/1919338775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31f412bc90 msgr2=0x7f31f4099800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:21:39.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.999+0000 7f31f37fe640 1 --2- 172.21.15.37:0/1919338775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31f412bc90 0x7f31f4099800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:21:39.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:39.999+0000 7f31f37fe640 1 --2- 172.21.15.37:0/1919338775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31f412bc90 0x7f31f4099800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:21:40.038 INFO:tasks.rados.rados.0.smithi045.stdout:276: done (0 left) 2024-04-26T00:21:40.038 INFO:tasks.rados.rados.0.smithi045.stdout:277: delete oid 7 current snap is 30 2024-04-26T00:21:40.059 INFO:tasks.rados.rados.0.smithi045.stdout:277: done (0 left) 2024-04-26T00:21:40.059 INFO:tasks.rados.rados.0.smithi045.stdout:278: delete oid 44 current snap is 30 2024-04-26T00:21:40.070 INFO:tasks.rados.rados.0.smithi045.stdout:278: done (0 left) 2024-04-26T00:21:40.070 INFO:tasks.rados.rados.0.smithi045.stdout:279: write (excl) oid 38 current snap is 30 2024-04-26T00:21:40.070 INFO:tasks.rados.rados.0.smithi045.stdout:279: seq_num 125 ranges {500875=467773,1564955=535920} 2024-04-26T00:21:40.085 INFO:tasks.rados.rados.0.smithi045.stdout:279: writing smithi04526579-38 from 500875 to 968648 tid 1 2024-04-26T00:21:40.104 INFO:tasks.rados.rados.0.smithi045.stdout:279: writing smithi04526579-38 from 1564955 to 2100875 tid 2 2024-04-26T00:21:40.104 INFO:tasks.rados.rados.0.smithi045.stdout:280: read oid 9 snap -1 2024-04-26T00:21:40.104 INFO:tasks.rados.rados.0.smithi045.stdout:280: expect (ObjNum 43 snap 0 seq_num 43) 2024-04-26T00:21:40.104 INFO:tasks.rados.rados.0.smithi045.stdout:281: write (excl) oid 30 current snap is 30 2024-04-26T00:21:40.105 INFO:tasks.rados.rados.0.smithi045.stdout:279: finishing write tid 1 to smithi04526579-38 2024-04-26T00:21:40.105 INFO:tasks.rados.rados.0.smithi045.stdout:281: seq_num 126 ranges {517682=543022,1623437=494245} 2024-04-26T00:21:40.121 INFO:tasks.rados.rados.0.smithi045.stdout:281: writing smithi04526579-30 from 517682 to 1060704 tid 1 2024-04-26T00:21:40.136 INFO:tasks.rados.rados.0.smithi045.stdout:281: writing smithi04526579-30 from 1623437 to 2117682 tid 2 2024-04-26T00:21:40.136 INFO:tasks.rados.rados.0.smithi045.stdout:282: read oid 37 snap -1 2024-04-26T00:21:40.136 INFO:tasks.rados.rados.0.smithi045.stdout:282: expect deleted 2024-04-26T00:21:40.136 INFO:tasks.rados.rados.0.smithi045.stdout:283: delete oid 46 current snap is 30 2024-04-26T00:21:40.140 INFO:tasks.rados.rados.0.smithi045.stdout:279: finishing write tid 2 to smithi04526579-38 2024-04-26T00:21:40.140 INFO:tasks.rados.rados.0.smithi045.stdout:279: finishing write tid 3 to smithi04526579-38 2024-04-26T00:21:40.140 INFO:tasks.rados.rados.0.smithi045.stdout:279: finishing write tid 4 to smithi04526579-38 2024-04-26T00:21:40.140 INFO:tasks.rados.rados.0.smithi045.stdout:279: oid 38 updating version 0 to 84 2024-04-26T00:21:40.140 INFO:tasks.rados.rados.0.smithi045.stdout:279: oid 38 version 84 is already newer than 83 2024-04-26T00:21:40.140 INFO:tasks.rados.rados.0.smithi045.stdout:279: oid 38 version 84 is already newer than 82 2024-04-26T00:21:40.141 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 84 (ObjNum 125 snap 30 seq_num 125) dirty exists 2024-04-26T00:21:40.141 INFO:tasks.rados.rados.0.smithi045.stdout:279: left oid 38 (ObjNum 125 snap 30 seq_num 125) 2024-04-26T00:21:40.141 INFO:tasks.rados.rados.0.smithi045.stdout:280: expect (ObjNum 43 snap 0 seq_num 43) 2024-04-26T00:21:40.258 INFO:tasks.rados.rados.0.smithi045.stdout:281: finishing write tid 1 to smithi04526579-30 2024-04-26T00:21:40.258 INFO:tasks.rados.rados.0.smithi045.stdout:281: finishing write tid 2 to smithi04526579-30 2024-04-26T00:21:40.258 INFO:tasks.rados.rados.0.smithi045.stdout:281: finishing write tid 3 to smithi04526579-30 2024-04-26T00:21:40.259 INFO:tasks.rados.rados.0.smithi045.stdout:281: finishing write tid 4 to smithi04526579-30 2024-04-26T00:21:40.259 INFO:tasks.rados.rados.0.smithi045.stdout:281: oid 30 updating version 0 to 77 2024-04-26T00:21:40.259 INFO:tasks.rados.rados.0.smithi045.stdout:281: oid 30 version 77 is already newer than 71 2024-04-26T00:21:40.259 INFO:tasks.rados.rados.0.smithi045.stdout:281: oid 30 version 77 is already newer than 76 2024-04-26T00:21:40.259 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 77 (ObjNum 126 snap 30 seq_num 126) dirty exists 2024-04-26T00:21:40.259 INFO:tasks.rados.rados.0.smithi045.stdout:281: left oid 30 (ObjNum 126 snap 30 seq_num 126) 2024-04-26T00:21:40.259 INFO:tasks.rados.rados.0.smithi045.stdout:279: done (4 left) 2024-04-26T00:21:40.259 INFO:tasks.rados.rados.0.smithi045.stdout:280: done (3 left) 2024-04-26T00:21:40.259 INFO:tasks.rados.rados.0.smithi045.stdout:281: done (2 left) 2024-04-26T00:21:40.259 INFO:tasks.rados.rados.0.smithi045.stdout:282: done (1 left) 2024-04-26T00:21:40.259 INFO:tasks.rados.rados.0.smithi045.stdout:283: done (0 left) 2024-04-26T00:21:40.259 INFO:tasks.rados.rados.0.smithi045.stdout:284: write (excl) oid 45 current snap is 30 2024-04-26T00:21:40.259 INFO:tasks.rados.rados.0.smithi045.stdout:284: seq_num 128 ranges {551296=693520,2024048=127248} 2024-04-26T00:21:40.275 INFO:tasks.rados.rados.0.smithi045.stdout:284: writing smithi04526579-45 from 551296 to 1244816 tid 1 2024-04-26T00:21:40.288 INFO:tasks.rados.rados.0.smithi045.stdout:284: writing smithi04526579-45 from 2024048 to 2151296 tid 2 2024-04-26T00:21:40.288 INFO:tasks.rados.rados.0.smithi045.stdout:285: write (excl) oid 6 current snap is 30 2024-04-26T00:21:40.289 INFO:tasks.rados.rados.0.smithi045.stdout:285: seq_num 129 ranges {568103=768769,2082530=85573} 2024-04-26T00:21:40.306 INFO:tasks.rados.rados.0.smithi045.stdout:285: writing smithi04526579-6 from 568103 to 1336872 tid 1 2024-04-26T00:21:40.317 INFO:tasks.rados.rados.0.smithi045.stdout:285: writing smithi04526579-6 from 2082530 to 2168103 tid 2 2024-04-26T00:21:40.317 INFO:tasks.rados.rados.0.smithi045.stdout:286: copy_from oid 17 from oid 49 current snap is 30 2024-04-26T00:21:40.317 INFO:tasks.rados.rados.0.smithi045.stdout:284: finishing write tid 1 to smithi04526579-45 2024-04-26T00:21:40.317 INFO:tasks.rados.rados.0.smithi045.stdout:284: finishing write tid 2 to smithi04526579-45 2024-04-26T00:21:40.317 INFO:tasks.rados.rados.0.smithi045.stdout:284: finishing write tid 3 to smithi04526579-45 2024-04-26T00:21:40.318 INFO:tasks.rados.rados.0.smithi045.stdout:284: finishing write tid 4 to smithi04526579-45 2024-04-26T00:21:40.318 INFO:tasks.rados.rados.0.smithi045.stdout:284: oid 45 updating version 0 to 51 2024-04-26T00:21:40.318 INFO:tasks.rados.rados.0.smithi045.stdout:284: oid 45 updating version 51 to 53 2024-04-26T00:21:40.318 INFO:tasks.rados.rados.0.smithi045.stdout:284: oid 45 version 53 is already newer than 52 2024-04-26T00:21:40.318 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 53 (ObjNum 128 snap 30 seq_num 128) dirty exists 2024-04-26T00:21:40.318 INFO:tasks.rados.rados.0.smithi045.stdout:284: left oid 45 (ObjNum 128 snap 30 seq_num 128) 2024-04-26T00:21:40.318 INFO:tasks.rados.rados.0.smithi045.stdout:284: done (2 left) 2024-04-26T00:21:40.318 INFO:tasks.rados.rados.0.smithi045.stdout:287: write oid 26 current snap is 30 2024-04-26T00:21:40.318 INFO:tasks.rados.rados.0.smithi045.stdout:287: seq_num 130 ranges {584910=560371,1857365=327545} 2024-04-26T00:21:40.330 INFO:tasks.rados.rados.0.smithi045.stdout:287: writing smithi04526579-26 from 584910 to 1145281 tid 1 2024-04-26T00:21:40.341 INFO:tasks.rados.rados.0.smithi045.stdout:287: writing smithi04526579-26 from 1857365 to 2184910 tid 2 2024-04-26T00:21:40.341 INFO:tasks.rados.rados.0.smithi045.stdout:288: write oid 40 current snap is 30 2024-04-26T00:21:40.342 INFO:tasks.rados.rados.0.smithi045.stdout:288: seq_num 131 ranges {601717=635620,1799494=402223} 2024-04-26T00:21:40.354 INFO:tasks.rados.rados.0.smithi045.stdout:288: writing smithi04526579-40 from 601717 to 1237337 tid 1 2024-04-26T00:21:40.363 INFO:tasks.rados.rados.0.smithi045.stdout:288: writing smithi04526579-40 from 1799494 to 2201717 tid 2 2024-04-26T00:21:40.364 INFO:tasks.rados.rados.0.smithi045.stdout:289: read oid 42 snap 29 2024-04-26T00:21:40.364 INFO:tasks.rados.rados.0.smithi045.stdout:289: expect (ObjNum 114 snap 28 seq_num 114) 2024-04-26T00:21:40.364 INFO:tasks.rados.rados.0.smithi045.stdout:290: copy_from oid 1 from oid 33 current snap is 30 2024-04-26T00:21:40.364 INFO:tasks.rados.rados.0.smithi045.stdout:285: finishing write tid 1 to smithi04526579-6 2024-04-26T00:21:40.364 INFO:tasks.rados.rados.0.smithi045.stdout:285: finishing write tid 2 to smithi04526579-6 2024-04-26T00:21:40.364 INFO:tasks.rados.rados.0.smithi045.stdout:285: finishing write tid 3 to smithi04526579-6 2024-04-26T00:21:40.364 INFO:tasks.rados.rados.0.smithi045.stdout:285: finishing write tid 4 to smithi04526579-6 2024-04-26T00:21:40.364 INFO:tasks.rados.rados.0.smithi045.stdout:285: oid 6 updating version 0 to 79 2024-04-26T00:21:40.364 INFO:tasks.rados.rados.0.smithi045.stdout:285: oid 6 version 79 is already newer than 78 2024-04-26T00:21:40.364 INFO:tasks.rados.rados.0.smithi045.stdout:285: oid 6 updating version 79 to 80 2024-04-26T00:21:40.364 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 80 (ObjNum 129 snap 30 seq_num 129) dirty exists 2024-04-26T00:21:40.364 INFO:tasks.rados.rados.0.smithi045.stdout:285: left oid 6 (ObjNum 129 snap 30 seq_num 129) 2024-04-26T00:21:40.364 INFO:tasks.rados.rados.0.smithi045.stdout:287: finishing write tid 1 to smithi04526579-26 2024-04-26T00:21:40.365 INFO:tasks.rados.rados.0.smithi045.stdout:286: finishing copy_from racing read to smithi04526579-17 2024-04-26T00:21:40.365 INFO:tasks.rados.rados.0.smithi045.stdout:286: finishing copy_from to smithi04526579-17 2024-04-26T00:21:40.365 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 15 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-26T00:21:40.365 INFO:tasks.rados.rados.0.smithi045.stdout:287: finishing write tid 2 to smithi04526579-26 2024-04-26T00:21:40.365 INFO:tasks.rados.rados.0.smithi045.stdout:287: finishing write tid 3 to smithi04526579-26 2024-04-26T00:21:40.365 INFO:tasks.rados.rados.0.smithi045.stdout:287: finishing write tid 4 to smithi04526579-26 2024-04-26T00:21:40.365 INFO:tasks.rados.rados.0.smithi045.stdout:287: oid 26 updating version 0 to 109 2024-04-26T00:21:40.365 INFO:tasks.rados.rados.0.smithi045.stdout:287: oid 26 version 109 is already newer than 108 2024-04-26T00:21:40.365 INFO:tasks.rados.rados.0.smithi045.stdout:287: oid 26 updating version 109 to 110 2024-04-26T00:21:40.365 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 110 (ObjNum 130 snap 30 seq_num 130) dirty exists 2024-04-26T00:21:40.365 INFO:tasks.rados.rados.0.smithi045.stdout:287: left oid 26 (ObjNum 130 snap 30 seq_num 130) 2024-04-26T00:21:40.365 INFO:tasks.rados.rados.0.smithi045.stdout:288: finishing write tid 1 to smithi04526579-40 2024-04-26T00:21:40.365 INFO:tasks.rados.rados.0.smithi045.stdout:285: done (5 left) 2024-04-26T00:21:40.365 INFO:tasks.rados.rados.0.smithi045.stdout:286: done (4 left) 2024-04-26T00:21:40.366 INFO:tasks.rados.rados.0.smithi045.stdout:287: done (3 left) 2024-04-26T00:21:40.366 INFO:tasks.rados.rados.0.smithi045.stdout:291: rollback oid 11 current snap is 30 2024-04-26T00:21:40.366 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 11 to 23 2024-04-26T00:21:40.366 INFO:tasks.rados.rados.0.smithi045.stdout:292: rollback oid 44 current snap is 30 2024-04-26T00:21:40.366 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 44 to 29 2024-04-26T00:21:40.366 INFO:tasks.rados.rados.0.smithi045.stdout:293: read oid 4 snap 20 2024-04-26T00:21:40.366 INFO:tasks.rados.rados.0.smithi045.stdout:293: expect (ObjNum 3 snap 0 seq_num 3) 2024-04-26T00:21:40.366 INFO:tasks.rados.rados.0.smithi045.stdout:294: rollback oid 28 current snap is 30 2024-04-26T00:21:40.366 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 28 to 29 2024-04-26T00:21:40.366 INFO:tasks.rados.rados.0.smithi045.stdout:295: delete oid 43 current snap is 30 2024-04-26T00:21:40.373 INFO:tasks.rados.rados.0.smithi045.stdout:288: finishing write tid 2 to smithi04526579-40 2024-04-26T00:21:40.374 INFO:tasks.rados.rados.0.smithi045.stdout:288: finishing write tid 3 to smithi04526579-40 2024-04-26T00:21:40.375 INFO:tasks.rados.rados.0.smithi045.stdout:288: finishing write tid 4 to smithi04526579-40 2024-04-26T00:21:40.375 INFO:tasks.rados.rados.0.smithi045.stdout:288: oid 40 updating version 0 to 56 2024-04-26T00:21:40.375 INFO:tasks.rados.rados.0.smithi045.stdout:288: oid 40 updating version 56 to 57 2024-04-26T00:21:40.375 INFO:tasks.rados.rados.0.smithi045.stdout:288: oid 40 version 57 is already newer than 55 2024-04-26T00:21:40.375 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 57 (ObjNum 131 snap 30 seq_num 131) dirty exists 2024-04-26T00:21:40.375 INFO:tasks.rados.rados.0.smithi045.stdout:288: left oid 40 (ObjNum 131 snap 30 seq_num 131) 2024-04-26T00:21:40.376 INFO:tasks.rados.rados.0.smithi045.stdout:294: finishing rollback tid 0 to smithi04526579-28 2024-04-26T00:21:40.377 INFO:tasks.rados.rados.0.smithi045.stdout:293: expect (ObjNum 3 snap 0 seq_num 3) 2024-04-26T00:21:40.466 INFO:tasks.rados.rados.0.smithi045.stdout:294: finishing rollback tid 1 to smithi04526579-28 2024-04-26T00:21:40.466 INFO:tasks.rados.rados.0.smithi045.stdout:294: finishing rollback tid 2 to smithi04526579-28 2024-04-26T00:21:40.466 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 114 (ObjNum 113 snap 28 seq_num 113) dirty exists 2024-04-26T00:21:40.466 INFO:tasks.rados.rados.0.smithi045.stdout:292: finishing rollback tid 1 to smithi04526579-44 2024-04-26T00:21:40.466 INFO:tasks.rados.rados.0.smithi045.stdout:292: finishing rollback tid 2 to smithi04526579-44 2024-04-26T00:21:40.466 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 86 (ObjNum 119 snap 28 seq_num 119) dirty exists 2024-04-26T00:21:40.466 INFO:tasks.rados.rados.0.smithi045.stdout:291: finishing rollback tid 0 to smithi04526579-11 2024-04-26T00:21:40.466 INFO:tasks.rados.rados.0.smithi045.stdout:291: finishing rollback tid 1 to smithi04526579-11 2024-04-26T00:21:40.467 INFO:tasks.rados.rados.0.smithi045.stdout:290: finishing copy_from racing read to smithi04526579-1 2024-04-26T00:21:40.467 INFO:tasks.rados.rados.0.smithi045.stdout:291: finishing rollback tid 2 to smithi04526579-11 2024-04-26T00:21:40.467 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 57 (ObjNum 63 snap 9 seq_num 63) dirty exists 2024-04-26T00:21:40.467 INFO:tasks.rados.rados.0.smithi045.stdout:290: finishing copy_from to smithi04526579-1 2024-04-26T00:21:40.467 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 145 (ObjNum 110 snap 28 seq_num 110) dirty exists 2024-04-26T00:21:40.467 INFO:tasks.rados.rados.0.smithi045.stdout:289: expect (ObjNum 114 snap 28 seq_num 114) 2024-04-26T00:21:40.596 INFO:tasks.rados.rados.0.smithi045.stdout:288: done (7 left) 2024-04-26T00:21:40.596 INFO:tasks.rados.rados.0.smithi045.stdout:289: done (6 left) 2024-04-26T00:21:40.596 INFO:tasks.rados.rados.0.smithi045.stdout:290: done (5 left) 2024-04-26T00:21:40.596 INFO:tasks.rados.rados.0.smithi045.stdout:291: done (4 left) 2024-04-26T00:21:40.596 INFO:tasks.rados.rados.0.smithi045.stdout:292: done (3 left) 2024-04-26T00:21:40.596 INFO:tasks.rados.rados.0.smithi045.stdout:293: done (2 left) 2024-04-26T00:21:40.596 INFO:tasks.rados.rados.0.smithi045.stdout:294: done (1 left) 2024-04-26T00:21:40.596 INFO:tasks.rados.rados.0.smithi045.stdout:295: done (0 left) 2024-04-26T00:21:40.596 INFO:tasks.rados.rados.0.smithi045.stdout:296: write oid 20 current snap is 30 2024-04-26T00:21:40.596 INFO:tasks.rados.rados.0.smithi045.stdout:296: seq_num 133 ranges {635331=786118,1916458=318873} 2024-04-26T00:21:40.612 INFO:tasks.rados.rados.0.smithi045.stdout:296: writing smithi04526579-20 from 635331 to 1421449 tid 1 2024-04-26T00:21:40.622 INFO:tasks.rados.rados.0.smithi045.stdout:296: writing smithi04526579-20 from 1916458 to 2235331 tid 2 2024-04-26T00:21:40.622 INFO:tasks.rados.rados.0.smithi045.stdout:297: read oid 28 snap -1 2024-04-26T00:21:40.622 INFO:tasks.rados.rados.0.smithi045.stdout:297: expect (ObjNum 113 snap 28 seq_num 113) 2024-04-26T00:21:40.622 INFO:tasks.rados.rados.0.smithi045.stdout:298: rollback oid 46 current snap is 30 2024-04-26T00:21:40.622 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 46 to 29 2024-04-26T00:21:40.622 INFO:tasks.rados.rados.0.smithi045.stdout:299: write (excl) oid 22 current snap is 30 2024-04-26T00:21:40.622 INFO:tasks.rados.rados.0.smithi045.stdout:299: seq_num 134 ranges {652138=461367,1574940=677198} 2024-04-26T00:21:40.635 INFO:tasks.rados.rados.0.smithi045.stdout:299: writing smithi04526579-22 from 652138 to 1113505 tid 1 2024-04-26T00:21:40.648 INFO:tasks.rados.rados.0.smithi045.stdout:299: writing smithi04526579-22 from 1574940 to 2252138 tid 2 2024-04-26T00:21:40.648 INFO:tasks.rados.rados.0.smithi045.stdout:300: read oid 43 snap -1 2024-04-26T00:21:40.648 INFO:tasks.rados.rados.0.smithi045.stdout:300: expect deleted 2024-04-26T00:21:40.648 INFO:tasks.rados.rados.0.smithi045.stdout:301: write oid 30 current snap is 30 2024-04-26T00:21:40.648 INFO:tasks.rados.rados.0.smithi045.stdout:301: seq_num 135 ranges {668945=536616,1633422=491363} 2024-04-26T00:21:40.661 INFO:tasks.rados.rados.0.smithi045.stdout:301: writing smithi04526579-30 from 668945 to 1205561 tid 1 2024-04-26T00:21:40.671 INFO:tasks.rados.rados.0.smithi045.stdout:301: writing smithi04526579-30 from 1633422 to 2124785 tid 2 2024-04-26T00:21:40.671 INFO:tasks.rados.rados.0.smithi045.stdout:302: write oid 26 current snap is 30 2024-04-26T00:21:40.671 INFO:tasks.rados.rados.0.smithi045.stdout:302: seq_num 136 ranges {685752=611865,1975551=310201} 2024-04-26T00:21:40.683 INFO:tasks.rados.rados.0.smithi045.stdout:302: writing smithi04526579-26 from 685752 to 1297617 tid 1 2024-04-26T00:21:40.693 INFO:tasks.rados.rados.0.smithi045.stdout:302: writing smithi04526579-26 from 1975551 to 2285752 tid 2 2024-04-26T00:21:40.693 INFO:tasks.rados.rados.0.smithi045.stdout:303: snap_create 2024-04-26T00:21:40.693 INFO:tasks.rados.rados.0.smithi045.stdout:298: finishing rollback tid 1 to smithi04526579-46 2024-04-26T00:21:40.693 INFO:tasks.rados.rados.0.smithi045.stdout:298: finishing rollback tid 2 to smithi04526579-46 2024-04-26T00:21:40.694 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 82 (ObjNum 100 snap 24 seq_num 100) dirty exists 2024-04-26T00:21:40.694 INFO:tasks.rados.rados.0.smithi045.stdout:297: expect (ObjNum 113 snap 28 seq_num 113) 2024-04-26T00:21:40.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:40.803+0000 7f31f37fe640 1 --2- 172.21.15.37:0/1919338775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31f412bc90 0x7f31f4099800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:40.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:40.803+0000 7f31f37fe640 10 monclient: get_auth_request con 0x7f31f412bc90 auth_method 0 2024-04-26T00:21:40.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:40.803+0000 7f31f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:40.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:40.803+0000 7f31f37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:40.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:40.803+0000 7f31f37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:21:40.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:40.803+0000 7f31f37fe640 1 -- 172.21.15.37:0/1919338775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31f412bc90 msgr2=0x7f31f4099800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:21:40.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:40.803+0000 7f31f37fe640 1 -- 172.21.15.37:0/1919338775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31f412bc90 msgr2=0x7f31f4099800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:21:40.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:40.803+0000 7f31f37fe640 1 --2- 172.21.15.37:0/1919338775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31f412bc90 0x7f31f4099800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:21:40.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:40.803+0000 7f31f37fe640 1 --2- 172.21.15.37:0/1919338775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31f412bc90 0x7f31f4099800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:21:40.906 INFO:tasks.rados.rados.0.smithi045.stdout:296: finishing write tid 1 to smithi04526579-20 2024-04-26T00:21:40.906 INFO:tasks.rados.rados.0.smithi045.stdout:296: finishing write tid 2 to smithi04526579-20 2024-04-26T00:21:40.906 INFO:tasks.rados.rados.0.smithi045.stdout:296: finishing write tid 3 to smithi04526579-20 2024-04-26T00:21:40.906 INFO:tasks.rados.rados.0.smithi045.stdout:296: finishing write tid 4 to smithi04526579-20 2024-04-26T00:21:40.906 INFO:tasks.rados.rados.0.smithi045.stdout:296: oid 20 updating version 0 to 89 2024-04-26T00:21:40.907 INFO:tasks.rados.rados.0.smithi045.stdout:296: oid 20 updating version 89 to 90 2024-04-26T00:21:40.907 INFO:tasks.rados.rados.0.smithi045.stdout:296: oid 20 version 90 is already newer than 88 2024-04-26T00:21:40.907 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 90 (ObjNum 133 snap 30 seq_num 133) dirty exists 2024-04-26T00:21:40.907 INFO:tasks.rados.rados.0.smithi045.stdout:296: left oid 20 (ObjNum 133 snap 30 seq_num 133) 2024-04-26T00:21:40.907 INFO:tasks.rados.rados.0.smithi045.stdout:299: finishing write tid 1 to smithi04526579-22 2024-04-26T00:21:40.907 INFO:tasks.rados.rados.0.smithi045.stdout:299: finishing write tid 2 to smithi04526579-22 2024-04-26T00:21:40.907 INFO:tasks.rados.rados.0.smithi045.stdout:299: finishing write tid 3 to smithi04526579-22 2024-04-26T00:21:40.907 INFO:tasks.rados.rados.0.smithi045.stdout:299: finishing write tid 4 to smithi04526579-22 2024-04-26T00:21:40.907 INFO:tasks.rados.rados.0.smithi045.stdout:299: oid 22 updating version 0 to 85 2024-04-26T00:21:40.907 INFO:tasks.rados.rados.0.smithi045.stdout:299: oid 22 version 85 is already newer than 84 2024-04-26T00:21:40.907 INFO:tasks.rados.rados.0.smithi045.stdout:299: oid 22 updating version 85 to 86 2024-04-26T00:21:40.907 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 86 (ObjNum 134 snap 30 seq_num 134) dirty exists 2024-04-26T00:21:40.907 INFO:tasks.rados.rados.0.smithi045.stdout:299: left oid 22 (ObjNum 134 snap 30 seq_num 134) 2024-04-26T00:21:40.907 INFO:tasks.rados.rados.0.smithi045.stdout:301: finishing write tid 1 to smithi04526579-30 2024-04-26T00:21:40.907 INFO:tasks.rados.rados.0.smithi045.stdout:301: finishing write tid 2 to smithi04526579-30 2024-04-26T00:21:40.908 INFO:tasks.rados.rados.0.smithi045.stdout:301: finishing write tid 3 to smithi04526579-30 2024-04-26T00:21:40.908 INFO:tasks.rados.rados.0.smithi045.stdout:301: finishing write tid 4 to smithi04526579-30 2024-04-26T00:21:40.908 INFO:tasks.rados.rados.0.smithi045.stdout:301: oid 30 updating version 0 to 81 2024-04-26T00:21:40.908 INFO:tasks.rados.rados.0.smithi045.stdout:301: oid 30 updating version 81 to 82 2024-04-26T00:21:40.908 INFO:tasks.rados.rados.0.smithi045.stdout:301: oid 30 updating version 82 to 83 2024-04-26T00:21:40.908 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 83 (ObjNum 135 snap 30 seq_num 135) dirty exists 2024-04-26T00:21:40.908 INFO:tasks.rados.rados.0.smithi045.stdout:301: left oid 30 (ObjNum 135 snap 30 seq_num 135) 2024-04-26T00:21:40.908 INFO:tasks.rados.rados.0.smithi045.stdout:302: finishing write tid 1 to smithi04526579-26 2024-04-26T00:21:40.908 INFO:tasks.rados.rados.0.smithi045.stdout:302: finishing write tid 2 to smithi04526579-26 2024-04-26T00:21:40.908 INFO:tasks.rados.rados.0.smithi045.stdout:302: finishing write tid 3 to smithi04526579-26 2024-04-26T00:21:40.908 INFO:tasks.rados.rados.0.smithi045.stdout:302: finishing write tid 4 to smithi04526579-26 2024-04-26T00:21:40.908 INFO:tasks.rados.rados.0.smithi045.stdout:302: oid 26 updating version 0 to 148 2024-04-26T00:21:40.908 INFO:tasks.rados.rados.0.smithi045.stdout:302: oid 26 version 148 is already newer than 146 2024-04-26T00:21:40.908 INFO:tasks.rados.rados.0.smithi045.stdout:302: oid 26 version 148 is already newer than 147 2024-04-26T00:21:40.909 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 148 (ObjNum 136 snap 30 seq_num 136) dirty exists 2024-04-26T00:21:40.909 INFO:tasks.rados.rados.0.smithi045.stdout:302: left oid 26 (ObjNum 136 snap 30 seq_num 136) 2024-04-26T00:21:41.577 INFO:tasks.rados.rados.0.smithi045.stdout:296: done (7 left) 2024-04-26T00:21:41.577 INFO:tasks.rados.rados.0.smithi045.stdout:297: done (6 left) 2024-04-26T00:21:41.578 INFO:tasks.rados.rados.0.smithi045.stdout:298: done (5 left) 2024-04-26T00:21:41.578 INFO:tasks.rados.rados.0.smithi045.stdout:299: done (4 left) 2024-04-26T00:21:41.578 INFO:tasks.rados.rados.0.smithi045.stdout:300: done (3 left) 2024-04-26T00:21:41.578 INFO:tasks.rados.rados.0.smithi045.stdout:301: done (2 left) 2024-04-26T00:21:41.578 INFO:tasks.rados.rados.0.smithi045.stdout:302: done (1 left) 2024-04-26T00:21:41.578 INFO:tasks.rados.rados.0.smithi045.stdout:303: done (0 left) 2024-04-26T00:21:41.578 INFO:tasks.rados.rados.0.smithi045.stdout:304: copy_from oid 43 from oid 16 current snap is 31 2024-04-26T00:21:41.578 INFO:tasks.rados.rados.0.smithi045.stdout:305: write (excl) oid 45 current snap is 31 2024-04-26T00:21:41.578 INFO:tasks.rados.rados.0.smithi045.stdout:305: seq_num 137 ranges {702559=403467,1750386=495032} 2024-04-26T00:21:41.586 INFO:tasks.rados.rados.0.smithi045.stdout:305: writing smithi04526579-45 from 702559 to 1106026 tid 1 2024-04-26T00:21:41.595 INFO:tasks.rados.rados.0.smithi045.stdout:305: writing smithi04526579-45 from 1750386 to 2245418 tid 2 2024-04-26T00:21:41.595 INFO:tasks.rados.rados.0.smithi045.stdout:306: write (excl) oid 11 current snap is 31 2024-04-26T00:21:41.595 INFO:tasks.rados.rados.0.smithi045.stdout:306: seq_num 138 ranges {719366=478716,1808868=510498} 2024-04-26T00:21:41.602 INFO:tasks.rados.rados.0.smithi045.stdout:306: writing smithi04526579-11 from 719366 to 1198082 tid 1 2024-04-26T00:21:41.609 INFO:tasks.rados.rados.0.smithi045.stdout:306: writing smithi04526579-11 from 1808868 to 2319366 tid 2 2024-04-26T00:21:41.609 INFO:tasks.rados.rados.0.smithi045.stdout:307: read oid 49 snap -1 2024-04-26T00:21:41.609 INFO:tasks.rados.rados.0.smithi045.stdout:307: expect (ObjNum 48 snap 0 seq_num 48) 2024-04-26T00:21:41.609 INFO:tasks.rados.rados.0.smithi045.stdout:308: write oid 6 current snap is 31 2024-04-26T00:21:41.610 INFO:tasks.rados.rados.0.smithi045.stdout:304: finishing copy_from racing read to smithi04526579-43 2024-04-26T00:21:41.610 INFO:tasks.rados.rados.0.smithi045.stdout:304: finishing copy_from to smithi04526579-43 2024-04-26T00:21:41.610 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 6 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-04-26T00:21:41.610 INFO:tasks.rados.rados.0.smithi045.stdout:305: finishing write tid 1 to smithi04526579-45 2024-04-26T00:21:41.610 INFO:tasks.rados.rados.0.smithi045.stdout:305: finishing write tid 2 to smithi04526579-45 2024-04-26T00:21:41.610 INFO:tasks.rados.rados.0.smithi045.stdout:305: finishing write tid 3 to smithi04526579-45 2024-04-26T00:21:41.610 INFO:tasks.rados.rados.0.smithi045.stdout:305: finishing write tid 4 to smithi04526579-45 2024-04-26T00:21:41.610 INFO:tasks.rados.rados.0.smithi045.stdout:305: oid 45 updating version 0 to 59 2024-04-26T00:21:41.610 INFO:tasks.rados.rados.0.smithi045.stdout:305: oid 45 updating version 59 to 61 2024-04-26T00:21:41.610 INFO:tasks.rados.rados.0.smithi045.stdout:305: oid 45 version 61 is already newer than 60 2024-04-26T00:21:41.610 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 61 (ObjNum 137 snap 31 seq_num 137) dirty exists 2024-04-26T00:21:41.610 INFO:tasks.rados.rados.0.smithi045.stdout:305: left oid 45 (ObjNum 137 snap 31 seq_num 137) 2024-04-26T00:21:41.611 INFO:tasks.rados.rados.0.smithi045.stdout:308: seq_num 139 ranges {736173=553965,1867350=468823} 2024-04-26T00:21:41.616 INFO:tasks.rados.rados.0.smithi045.stdout:308: writing smithi04526579-6 from 736173 to 1290138 tid 1 2024-04-26T00:21:41.623 INFO:tasks.rados.rados.0.smithi045.stdout:308: writing smithi04526579-6 from 1867350 to 2336173 tid 2 2024-04-26T00:21:41.623 INFO:tasks.rados.rados.0.smithi045.stdout:304: done (4 left) 2024-04-26T00:21:41.623 INFO:tasks.rados.rados.0.smithi045.stdout:305: done (3 left) 2024-04-26T00:21:41.623 INFO:tasks.rados.rados.0.smithi045.stdout:309: write (excl) oid 2 current snap is 31 2024-04-26T00:21:41.623 INFO:tasks.rados.rados.0.smithi045.stdout:309: seq_num 140 ranges {752980=629214,1809479=543501} 2024-04-26T00:21:41.631 INFO:tasks.rados.rados.0.smithi045.stdout:309: writing smithi04526579-2 from 752980 to 1382194 tid 1 2024-04-26T00:21:41.637 INFO:tasks.rados.rados.0.smithi045.stdout:309: writing smithi04526579-2 from 1809479 to 2352980 tid 2 2024-04-26T00:21:41.637 INFO:tasks.rados.rados.0.smithi045.stdout:310: read oid 18 snap -1 2024-04-26T00:21:41.637 INFO:tasks.rados.rados.0.smithi045.stdout:310: expect deleted 2024-04-26T00:21:41.637 INFO:tasks.rados.rados.0.smithi045.stdout:311: write oid 24 current snap is 31 2024-04-26T00:21:41.637 INFO:tasks.rados.rados.0.smithi045.stdout:306: finishing write tid 1 to smithi04526579-11 2024-04-26T00:21:41.637 INFO:tasks.rados.rados.0.smithi045.stdout:306: finishing write tid 2 to smithi04526579-11 2024-04-26T00:21:41.637 INFO:tasks.rados.rados.0.smithi045.stdout:311: seq_num 141 ranges {769787=704463,2267961=101826} 2024-04-26T00:21:41.645 INFO:tasks.rados.rados.0.smithi045.stdout:311: writing smithi04526579-24 from 769787 to 1474250 tid 1 2024-04-26T00:21:41.650 INFO:tasks.rados.rados.0.smithi045.stdout:311: writing smithi04526579-24 from 2267961 to 2369787 tid 2 2024-04-26T00:21:41.650 INFO:tasks.rados.rados.0.smithi045.stdout:307: expect (ObjNum 48 snap 0 seq_num 48) 2024-04-26T00:21:41.741 INFO:tasks.rados.rados.0.smithi045.stdout:306: finishing write tid 3 to smithi04526579-11 2024-04-26T00:21:41.741 INFO:tasks.rados.rados.0.smithi045.stdout:306: finishing write tid 4 to smithi04526579-11 2024-04-26T00:21:41.741 INFO:tasks.rados.rados.0.smithi045.stdout:306: oid 11 updating version 0 to 63 2024-04-26T00:21:41.741 INFO:tasks.rados.rados.0.smithi045.stdout:306: oid 11 updating version 63 to 65 2024-04-26T00:21:41.741 INFO:tasks.rados.rados.0.smithi045.stdout:306: oid 11 version 65 is already newer than 64 2024-04-26T00:21:41.741 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 65 (ObjNum 138 snap 31 seq_num 138) dirty exists 2024-04-26T00:21:41.741 INFO:tasks.rados.rados.0.smithi045.stdout:306: left oid 11 (ObjNum 138 snap 31 seq_num 138) 2024-04-26T00:21:41.741 INFO:tasks.rados.rados.0.smithi045.stdout:312: read oid 5 snap -1 2024-04-26T00:21:41.741 INFO:tasks.rados.rados.0.smithi045.stdout:312: expect (ObjNum 88 snap 21 seq_num 88) 2024-04-26T00:21:41.741 INFO:tasks.rados.rados.0.smithi045.stdout:306: done (6 left) 2024-04-26T00:21:41.741 INFO:tasks.rados.rados.0.smithi045.stdout:307: done (5 left) 2024-04-26T00:21:41.741 INFO:tasks.rados.rados.0.smithi045.stdout:313: write (excl) oid 17 current snap is 31 2024-04-26T00:21:41.741 INFO:tasks.rados.rados.0.smithi045.stdout:308: finishing write tid 1 to smithi04526579-6 2024-04-26T00:21:41.742 INFO:tasks.rados.rados.0.smithi045.stdout:308: finishing write tid 2 to smithi04526579-6 2024-04-26T00:21:41.742 INFO:tasks.rados.rados.0.smithi045.stdout:308: finishing write tid 3 to smithi04526579-6 2024-04-26T00:21:41.742 INFO:tasks.rados.rados.0.smithi045.stdout:313: seq_num 142 ranges {786594=779712,2326443=60151} 2024-04-26T00:21:41.749 INFO:tasks.rados.rados.0.smithi045.stdout:313: writing smithi04526579-17 from 786594 to 1566306 tid 1 2024-04-26T00:21:41.754 INFO:tasks.rados.rados.0.smithi045.stdout:313: writing smithi04526579-17 from 2326443 to 2386594 tid 2 2024-04-26T00:21:41.755 INFO:tasks.rados.rados.0.smithi045.stdout:314: delete oid 27 current snap is 31 2024-04-26T00:21:41.755 INFO:tasks.rados.rados.0.smithi045.stdout:308: finishing write tid 4 to smithi04526579-6 2024-04-26T00:21:41.755 INFO:tasks.rados.rados.0.smithi045.stdout:308: oid 6 updating version 0 to 85 2024-04-26T00:21:41.755 INFO:tasks.rados.rados.0.smithi045.stdout:308: oid 6 updating version 85 to 86 2024-04-26T00:21:41.755 INFO:tasks.rados.rados.0.smithi045.stdout:308: oid 6 updating version 86 to 87 2024-04-26T00:21:41.755 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 87 (ObjNum 139 snap 31 seq_num 139) dirty exists 2024-04-26T00:21:41.755 INFO:tasks.rados.rados.0.smithi045.stdout:308: left oid 6 (ObjNum 139 snap 31 seq_num 139) 2024-04-26T00:21:41.755 INFO:tasks.rados.rados.0.smithi045.stdout:309: finishing write tid 1 to smithi04526579-2 2024-04-26T00:21:41.755 INFO:tasks.rados.rados.0.smithi045.stdout:309: finishing write tid 2 to smithi04526579-2 2024-04-26T00:21:41.755 INFO:tasks.rados.rados.0.smithi045.stdout:309: finishing write tid 3 to smithi04526579-2 2024-04-26T00:21:41.755 INFO:tasks.rados.rados.0.smithi045.stdout:309: finishing write tid 4 to smithi04526579-2 2024-04-26T00:21:41.755 INFO:tasks.rados.rados.0.smithi045.stdout:309: oid 2 updating version 0 to 144 2024-04-26T00:21:41.755 INFO:tasks.rados.rados.0.smithi045.stdout:309: oid 2 updating version 144 to 145 2024-04-26T00:21:41.756 INFO:tasks.rados.rados.0.smithi045.stdout:309: oid 2 version 145 is already newer than 143 2024-04-26T00:21:41.756 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 145 (ObjNum 140 snap 31 seq_num 140) dirty exists 2024-04-26T00:21:41.756 INFO:tasks.rados.rados.0.smithi045.stdout:309: left oid 2 (ObjNum 140 snap 31 seq_num 140) 2024-04-26T00:21:41.756 INFO:tasks.rados.rados.0.smithi045.stdout:311: finishing write tid 1 to smithi04526579-24 2024-04-26T00:21:41.756 INFO:tasks.rados.rados.0.smithi045.stdout:311: finishing write tid 2 to smithi04526579-24 2024-04-26T00:21:41.756 INFO:tasks.rados.rados.0.smithi045.stdout:311: finishing write tid 3 to smithi04526579-24 2024-04-26T00:21:41.756 INFO:tasks.rados.rados.0.smithi045.stdout:311: finishing write tid 4 to smithi04526579-24 2024-04-26T00:21:41.756 INFO:tasks.rados.rados.0.smithi045.stdout:311: oid 24 updating version 0 to 169 2024-04-26T00:21:41.756 INFO:tasks.rados.rados.0.smithi045.stdout:311: oid 24 version 169 is already newer than 168 2024-04-26T00:21:41.756 INFO:tasks.rados.rados.0.smithi045.stdout:311: oid 24 version 169 is already newer than 167 2024-04-26T00:21:41.756 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 169 (ObjNum 141 snap 31 seq_num 141) dirty exists 2024-04-26T00:21:41.756 INFO:tasks.rados.rados.0.smithi045.stdout:311: left oid 24 (ObjNum 141 snap 31 seq_num 141) 2024-04-26T00:21:41.757 INFO:tasks.rados.rados.0.smithi045.stdout:312: expect (ObjNum 88 snap 21 seq_num 88) 2024-04-26T00:21:41.893 INFO:tasks.rados.rados.0.smithi045.stdout:313: finishing write tid 1 to smithi04526579-17 2024-04-26T00:21:41.893 INFO:tasks.rados.rados.0.smithi045.stdout:313: finishing write tid 2 to smithi04526579-17 2024-04-26T00:21:41.893 INFO:tasks.rados.rados.0.smithi045.stdout:313: finishing write tid 3 to smithi04526579-17 2024-04-26T00:21:41.893 INFO:tasks.rados.rados.0.smithi045.stdout:313: finishing write tid 4 to smithi04526579-17 2024-04-26T00:21:41.893 INFO:tasks.rados.rados.0.smithi045.stdout:313: oid 17 updating version 0 to 125 2024-04-26T00:21:41.893 INFO:tasks.rados.rados.0.smithi045.stdout:313: oid 17 updating version 125 to 126 2024-04-26T00:21:41.893 INFO:tasks.rados.rados.0.smithi045.stdout:313: oid 17 version 126 is already newer than 124 2024-04-26T00:21:41.893 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 126 (ObjNum 142 snap 31 seq_num 142) dirty exists 2024-04-26T00:21:41.893 INFO:tasks.rados.rados.0.smithi045.stdout:313: left oid 17 (ObjNum 142 snap 31 seq_num 142) 2024-04-26T00:21:41.893 INFO:tasks.rados.rados.0.smithi045.stdout:308: done (6 left) 2024-04-26T00:21:41.893 INFO:tasks.rados.rados.0.smithi045.stdout:309: done (5 left) 2024-04-26T00:21:41.893 INFO:tasks.rados.rados.0.smithi045.stdout:310: done (4 left) 2024-04-26T00:21:41.894 INFO:tasks.rados.rados.0.smithi045.stdout:311: done (3 left) 2024-04-26T00:21:41.894 INFO:tasks.rados.rados.0.smithi045.stdout:312: done (2 left) 2024-04-26T00:21:41.894 INFO:tasks.rados.rados.0.smithi045.stdout:313: done (1 left) 2024-04-26T00:21:41.894 INFO:tasks.rados.rados.0.smithi045.stdout:314: done (0 left) 2024-04-26T00:21:41.894 INFO:tasks.rados.rados.0.smithi045.stdout:315: snap_remove snap 20 2024-04-26T00:21:42.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.387+0000 7f31db7fe640 10 monclient: tick 2024-04-26T00:21:42.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.387+0000 7f31db7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:21:42.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.387+0000 7f31db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:42.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.387+0000 7f31db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:42.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.407+0000 7f31f37fe640 1 --2- 172.21.15.37:0/1919338775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31f412bc90 0x7f31f4099800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:42.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.407+0000 7f31f37fe640 10 monclient: get_auth_request con 0x7f31f412bc90 auth_method 0 2024-04-26T00:21:42.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.407+0000 7f31f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:42.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.407+0000 7f31f37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:42.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.407+0000 7f31f37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:21:42.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.407+0000 7f31f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:42.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.407+0000 7f31f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:42.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.407+0000 7f31f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:42.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.407+0000 7f31f37fe640 10 monclient(hunting): handle_auth_done global_id 24100 payload 824 2024-04-26T00:21:42.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.407+0000 7f31f37fe640 1 -- 172.21.15.37:0/1919338775 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f31f412c070 con 0x7f31f412bc90 2024-04-26T00:21:42.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.411+0000 7f31f37fe640 1 -- 172.21.15.37:0/1919338775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31f412bc90 msgr2=0x7f31f4099800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:42.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.411+0000 7f31f37fe640 1 --2- 172.21.15.37:0/1919338775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31f412bc90 0x7f31f4099800 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f31e8004fd0 tx=0x7f31e8012ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:42.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.411+0000 7f31f0ff9640 1 -- 172.21.15.37:0/1919338775 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e8010c30 con 0x7f31f412bc90 2024-04-26T00:21:42.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.411+0000 7f31f0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:42.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.411+0000 7f31f0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:21:42.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.411+0000 7f31f0ff9640 10 monclient: dump: 2024-04-26T00:21:42.388 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:42.388 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:42.388 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:42.388 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:42.388 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:42.388 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:42.388 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:42.388 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:42.388 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:42.388 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:42.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.411+0000 7f31f0ff9640 1 -- 172.21.15.37:0/1919338775 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f31e8010c30 con 0x7f31f412bc90 2024-04-26T00:21:42.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.411+0000 7f31f0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:42.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.411+0000 7f31f0ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:21:42.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.411+0000 7f31f0ff9640 1 -- 172.21.15.37:0/1919338775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31f412bc90 msgr2=0x7f31f4099800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:42.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.411+0000 7f31f0ff9640 1 --2- 172.21.15.37:0/1919338775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31f412bc90 0x7f31f4099800 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f31e8004fd0 tx=0x7f31e8012ab0 comp rx=0 tx=0).stop 2024-04-26T00:21:42.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.411+0000 7f31f0ff9640 1 -- 172.21.15.37:0/1919338775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31f412bc90 msgr2=0x7f31f4099800 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:21:42.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.411+0000 7f31f0ff9640 1 --2- 172.21.15.37:0/1919338775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31f412bc90 0x7f31f4099800 secure :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f31e8004fd0 tx=0x7f31e8012ab0 comp rx=0 tx=0).stop 2024-04-26T00:21:42.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.419+0000 7f31faa4b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:21:42.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.419+0000 7f31faa4b640 1 --2- 172.21.15.37:0/1919338775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31f402c000 0x7f31f402c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:42.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.419+0000 7f31faa4b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:42.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.419+0000 7f31faa4b640 1 -- 172.21.15.37:0/1919338775 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f31f412dd30 con 0x7f31f402c000 2024-04-26T00:21:42.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.419+0000 7f31f37fe640 1 --2- 172.21.15.37:0/1919338775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31f402c000 0x7f31f402c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:42.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.419+0000 7f31f37fe640 10 monclient: get_auth_request con 0x7f31f402c000 auth_method 0 2024-04-26T00:21:42.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.419+0000 7f31f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:42.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.419+0000 7f31f37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:42.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.419+0000 7f31f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:42.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.419+0000 7f31f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:42.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.419+0000 7f31f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:42.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.419+0000 7f31f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:42.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.419+0000 7f31f37fe640 10 monclient(hunting): handle_auth_done global_id 24103 payload 824 2024-04-26T00:21:42.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.419+0000 7f31f37fe640 1 -- 172.21.15.37:0/1919338775 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f31f412e230 con 0x7f31f402c000 2024-04-26T00:21:42.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.423+0000 7f31f37fe640 1 -- 172.21.15.37:0/1919338775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31f402c000 msgr2=0x7f31f402c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:42.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.423+0000 7f31f37fe640 1 --2- 172.21.15.37:0/1919338775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31f402c000 0x7f31f402c410 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f31e8004fd0 tx=0x7f31e8004950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:42.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.423+0000 7f31f0ff9640 1 -- 172.21.15.37:0/1919338775 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e801c400 con 0x7f31f402c000 2024-04-26T00:21:42.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.423+0000 7f31f0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:42.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.423+0000 7f31f0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:21:42.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.423+0000 7f31f0ff9640 10 monclient: dump: 2024-04-26T00:21:42.400 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:42.400 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:42.401 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:42.401 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:42.401 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:42.401 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:42.401 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:42.401 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:42.401 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:42.401 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:42.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.423+0000 7f31f0ff9640 1 -- 172.21.15.37:0/1919338775 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7f31e801c400 con 0x7f31f402c000 2024-04-26T00:21:42.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.423+0000 7f31f0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:21:42.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.423+0000 7f31f0ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:21:42.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.423+0000 7f31f0ff9640 1 -- 172.21.15.37:0/1919338775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31f402c000 msgr2=0x7f31f402c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:42.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.423+0000 7f31f0ff9640 1 --2- 172.21.15.37:0/1919338775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31f402c000 0x7f31f402c410 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f31e8004fd0 tx=0x7f31e8004950 comp rx=0 tx=0).stop 2024-04-26T00:21:42.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.423+0000 7f31f0ff9640 1 -- 172.21.15.37:0/1919338775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31f402c000 msgr2=0x7f31f402c410 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:21:42.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.423+0000 7f31f0ff9640 1 --2- 172.21.15.37:0/1919338775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31f402c000 0x7f31f402c410 secure :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f31e8004fd0 tx=0x7f31e8004950 comp rx=0 tx=0).stop 2024-04-26T00:21:42.402 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:21:42.403 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:21:42.403 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:21:42.403 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:21:42.403 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 18, 2024-04-26T00:21:42.403 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 10587, 2024-04-26T00:21:42.403 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:21:42.403 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:21:42.403 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:21:42.403 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:21:42.403 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:42.403 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 4, 2024-04-26T00:21:42.403 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:21:42.403 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:21:42.404 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:21:42.404 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:21:42.404 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:21:42.404 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:21:42.404 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:21:42.404 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:21:42.404 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:21:42.404 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:21:42.404 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:21:42.404 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:21:42.404 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:21:42.404 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:21:42.404 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:42.404 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:21:42.405 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:21:42.405 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:21:42.405 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:21:42.405 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:21:42.405 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:21:42.405 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:21:42.405 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:21:42.405 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:21:42.405 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:21:42.405 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:21:42.405 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:21:42.405 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:21:42.405 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:42.405 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:42.406 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:21:42.406 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:21:42.406 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:21:42.406 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:21:42.406 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:21:42.406 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:21:42.406 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:21:42.406 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:21:42.406 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:21:42.406 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:21:42.406 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:21:42.406 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:21:42.406 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:21:42.406 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:21:42.407 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:21:42.407 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:21:42.407 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:21:42.407 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:21:42.407 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:21:42.407 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:21:42.407 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:21:42.407 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:21:42.407 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:21:42.407 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:21:42.407 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:21:42.407 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:21:42.407 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:21:42.407 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:21:42.407 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:42.408 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:21:42.408 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:42.408 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:21:42.408 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:42.408 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:21:42.408 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:21:42.408 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:21:42.408 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:21:42.408 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:42.408 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:21:42.408 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:21:42.408 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:42.408 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:42.408 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:42.409 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:21:42.409 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:21:42.409 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:42.409 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:42.409 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:42.409 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:42.409 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:21:42.409 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:21:42.409 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:21:42.409 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:21:42.409 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:21:42.409 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:42.409 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:42.409 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:21:42.409 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:21:42.410 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:21:42.410 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:21:42.410 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:42.410 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:21:42.410 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:21:42.410 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:42.410 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:42.410 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:42.410 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:21:42.410 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:21:42.410 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:42.410 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:42.410 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:42.411 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:42.411 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:21:42.411 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:21:42.411 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:21:42.411 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:21:42.411 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:21:42.411 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:42.411 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:42.411 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:21:42.411 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:21:42.411 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:21:42.411 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:21:42.411 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:42.411 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:21:42.411 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:21:42.411 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:42.412 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:42.412 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:42.412 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:21:42.412 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:21:42.412 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:42.412 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:42.412 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:42.412 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:42.412 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:21:42.412 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:21:42.412 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:21:42.412 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:21:42.412 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:21:42.412 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:42.412 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:42.413 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:42.413 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:21:42.413 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:21:42.413 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:42.413 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:21:42.413 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:21:42.413 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:21:42.413 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:42.413 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:42.413 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:21:42.413 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:42.413 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:21:42.413 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:21:42.413 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:21:42.413 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:42.414 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:42.414 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:42.414 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:21:42.414 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:21:42.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.423+0000 7f31faa4b640 1 -- 172.21.15.37:0/1919338775 >> v2:172.21.15.45:6812/2379918623 conn(0x7f31e0035ed0 msgr2=0x7f31e0038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:42.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.423+0000 7f31faa4b640 1 --2- 172.21.15.37:0/1919338775 >> v2:172.21.15.45:6812/2379918623 conn(0x7f31e0035ed0 0x7f31e0038390 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f31f406b6b0 tx=0x7f31ec0a6070 comp rx=0 tx=0).stop 2024-04-26T00:21:42.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.423+0000 7f31faa4b640 10 monclient: shutdown 2024-04-26T00:21:42.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.423+0000 7f31faa4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:42.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.423+0000 7f31faa4b640 1 -- 172.21.15.37:0/1919338775 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f31f409c780 msgr2=0x7f31f412eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:42.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.423+0000 7f31faa4b640 1 --2- 172.21.15.37:0/1919338775 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f31f409c780 0x7f31f412eb00 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f31e400d800 tx=0x7f31e400dcc0 comp rx=0 tx=0).stop 2024-04-26T00:21:42.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.423+0000 7f31f8fe8640 1 -- 172.21.15.37:0/1919338775 reap_dead start 2024-04-26T00:21:42.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.423+0000 7f31f8fe8640 1 -- 172.21.15.37:0/1919338775 reap_dead start 2024-04-26T00:21:42.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.423+0000 7f31faa4b640 1 -- 172.21.15.37:0/1919338775 shutdown_connections 2024-04-26T00:21:42.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.423+0000 7f31faa4b640 1 -- 172.21.15.37:0/1919338775 >> 172.21.15.37:0/1919338775 conn(0x7f31f40a2c50 msgr2=0x7f31f4132770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:42.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.423+0000 7f31faa4b640 1 -- 172.21.15.37:0/1919338775 shutdown_connections 2024-04-26T00:21:42.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.423+0000 7f31faa4b640 1 -- 172.21.15.37:0/1919338775 wait complete. 2024-04-26T00:21:42.426 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:21:42.426 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-26T00:21:42.427 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-26T00:21:42.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.579+0000 7f5f431eb640 10 monclient: get_monmap_and_config 2024-04-26T00:21:42.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.579+0000 7f5f431eb640 10 monclient: build_initial_monmap 2024-04-26T00:21:42.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.579+0000 7f5f431eb640 10 monclient: monmap: 2024-04-26T00:21:42.557 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:42.557 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:42.557 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:21:42.581961+0000 2024-04-26T00:21:42.558 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:21:42.581961+0000 2024-04-26T00:21:42.558 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:42.558 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:42.558 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:21:42.558 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:21:42.558 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:21:42.558 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:21:42.558 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:21:42.558 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:42.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.579+0000 7f5f431eb640 1 Processor -- start 2024-04-26T00:21:42.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.579+0000 7f5f431eb640 1 -- start start 2024-04-26T00:21:42.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.579+0000 7f5f431eb640 10 monclient: init 2024-04-26T00:21:42.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.579+0000 7f5f431eb640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:42.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.579+0000 7f5f431eb640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-04-26T00:21:42.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.579+0000 7f5f431eb640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5f3c0a0260 0x7f5f3c0a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:42.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.579+0000 7f5f431eb640 10 monclient(hunting): picked mon.noname-c con 0x7f5f3c0a0260 addr v2:172.21.15.45:3300/0 2024-04-26T00:21:42.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.579+0000 7f5f431eb640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f5f3c134b30 0x7f5f3c134f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:42.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.579+0000 7f5f431eb640 10 monclient(hunting): picked mon.noname-a con 0x7f5f3c134b30 addr v2:172.21.15.37:3300/0 2024-04-26T00:21:42.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.579+0000 7f5f431eb640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f3c135490 0x7f5f3c13be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:42.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.579+0000 7f5f431eb640 10 monclient(hunting): picked mon.noname-b con 0x7f5f3c135490 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:21:42.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.579+0000 7f5f431eb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:42.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.579+0000 7f5f431eb640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f5f3c09ec10 con 0x7f5f3c134b30 2024-04-26T00:21:42.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.579+0000 7f5f431eb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:42.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.579+0000 7f5f431eb640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f5f3c09f0f0 con 0x7f5f3c0a0260 2024-04-26T00:21:42.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.579+0000 7f5f431eb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:42.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.579+0000 7f5f431eb640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5f3c0a1130 con 0x7f5f3c135490 2024-04-26T00:21:42.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.579+0000 7f5f431eb640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:42.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.579+0000 7f5f431eb640 10 monclient(hunting): authenticate will time out at 1083.887631s 2024-04-26T00:21:42.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.579+0000 7f5f41788640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f3c135490 0x7f5f3c13be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:42.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.579+0000 7f5f3bfff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f5f3c134b30 0x7f5f3c134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:42.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.579+0000 7f5f41788640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f3c135490 0x7f5f3c13be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:56300/0 (socket says 172.21.15.37:56300) 2024-04-26T00:21:42.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.579+0000 7f5f41788640 1 -- 172.21.15.37:0/2496716033 learned_addr learned my addr 172.21.15.37:0/2496716033 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:21:42.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.579+0000 7f5f3bfff640 10 monclient(hunting): get_auth_request con 0x7f5f3c134b30 auth_method 0 2024-04-26T00:21:42.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.579+0000 7f5f3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:42.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.579+0000 7f5f3bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:42.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.579+0000 7f5f3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:42.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.579+0000 7f5f41788640 10 monclient(hunting): get_auth_request con 0x7f5f3c135490 auth_method 0 2024-04-26T00:21:42.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.579+0000 7f5f41788640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:42.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.579+0000 7f5f41788640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:42.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.579+0000 7f5f41788640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:42.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.583+0000 7f5f41788640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:42.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.583+0000 7f5f41788640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:42.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.583+0000 7f5f41788640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:42.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.583+0000 7f5f40f87640 1 --2- 172.21.15.37:0/2496716033 >> v2:172.21.15.45:3300/0 conn(0x7f5f3c0a0260 0x7f5f3c0a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:42.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.583+0000 7f5f3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:42.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.583+0000 7f5f3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:42.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.583+0000 7f5f3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:42.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.583+0000 7f5f40f87640 10 monclient(hunting): get_auth_request con 0x7f5f3c0a0260 auth_method 0 2024-04-26T00:21:42.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.583+0000 7f5f40f87640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:42.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.583+0000 7f5f40f87640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:42.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.583+0000 7f5f40f87640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:42.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.583+0000 7f5f40f87640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:42.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.583+0000 7f5f40f87640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:42.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.583+0000 7f5f40f87640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:42.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.583+0000 7f5f41788640 10 monclient(hunting): handle_auth_done global_id 4541 payload 274 2024-04-26T00:21:42.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.583+0000 7f5f41788640 1 -- 172.21.15.37:0/2496716033 >> v2:172.21.15.45:3300/0 conn(0x7f5f3c0a0260 msgr2=0x7f5f3c0a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:21:42.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.583+0000 7f5f41788640 1 --2- 172.21.15.37:0/2496716033 >> v2:172.21.15.45:3300/0 conn(0x7f5f3c0a0260 0x7f5f3c0a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:42.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.583+0000 7f5f41788640 1 -- 172.21.15.37:0/2496716033 >> v2:172.21.15.37:3300/0 conn(0x7f5f3c134b30 msgr2=0x7f5f3c134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:21:42.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.583+0000 7f5f41788640 1 --2- 172.21.15.37:0/2496716033 >> v2:172.21.15.37:3300/0 conn(0x7f5f3c134b30 0x7f5f3c134f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:42.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.583+0000 7f5f41788640 10 monclient: _finish_hunting 0 2024-04-26T00:21:42.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.583+0000 7f5f41788640 1 monclient: found mon.noname-b 2024-04-26T00:21:42.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.583+0000 7f5f41788640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:42.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.583+0000 7f5f41788640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:21:42.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.583+0000 7f5f41788640 1 -- 172.21.15.37:0/2496716033 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f3c06cff0 con 0x7f5f3c135490 2024-04-26T00:21:42.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.583+0000 7f5f3bfff640 1 --2- 172.21.15.37:0/2496716033 >> v2:172.21.15.37:3300/0 conn(0x7f5f3c134b30 0x7f5f3c134f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:21:42.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f41788640 1 -- 172.21.15.37:0/2496716033 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f3c135490 msgr2=0x7f5f3c13be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:42.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f41788640 1 --2- 172.21.15.37:0/2496716033 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f3c135490 0x7f5f3c13be60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c0a2ad0 tx=0x7f5f2c01c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88e0d6f01efd4952 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:42.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f3b7fe640 1 -- 172.21.15.37:0/2496716033 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c0b0070 con 0x7f5f3c135490 2024-04-26T00:21:42.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f3b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:42.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f3b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:21:42.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f3b7fe640 10 monclient: dump: 2024-04-26T00:21:42.569 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:42.570 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:42.570 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:42.570 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:42.570 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:42.570 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:42.570 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:42.570 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:42.570 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:42.570 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:42.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f3b7fe640 10 monclient: _finish_auth 0 2024-04-26T00:21:42.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f3b7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:21:42.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:42.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f431eb640 5 monclient: authenticate success, global_id 4541 2024-04-26T00:21:42.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f3b7fe640 1 -- 172.21.15.37:0/2496716033 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f2c0ab400 con 0x7f5f3c135490 2024-04-26T00:21:42.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f3b7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:42.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f3b7fe640 1 -- 172.21.15.37:0/2496716033 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c0ab7d0 con 0x7f5f3c135490 2024-04-26T00:21:42.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f3b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:42.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f3b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:21:42.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f3b7fe640 10 monclient: dump: 2024-04-26T00:21:42.571 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:42.571 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:42.571 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:42.571 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:42.571 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:42.571 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:42.571 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:42.571 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:42.571 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:42.571 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:42.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f431eb640 10 monclient: get_monmap_and_config success 2024-04-26T00:21:42.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f431eb640 4 set_mon_vals no callback set 2024-04-26T00:21:42.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f431eb640 10 monclient: shutdown 2024-04-26T00:21:42.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f431eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:42.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f431eb640 1 -- 172.21.15.37:0/2496716033 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f3c135490 msgr2=0x7f5f3c13be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:42.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f431eb640 1 --2- 172.21.15.37:0/2496716033 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f3c135490 0x7f5f3c13be60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c0a2ad0 tx=0x7f5f2c01c3e0 comp rx=0 tx=0).stop 2024-04-26T00:21:42.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f431eb640 1 -- 172.21.15.37:0/2496716033 shutdown_connections 2024-04-26T00:21:42.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f431eb640 1 --2- 172.21.15.37:0/2496716033 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f3c135490 0x7f5f3c13be60 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:42.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f431eb640 1 --2- 172.21.15.37:0/2496716033 >> v2:172.21.15.37:3300/0 conn(0x7f5f3c134b30 0x7f5f3c134f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:42.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f431eb640 1 --2- 172.21.15.37:0/2496716033 >> v2:172.21.15.45:3300/0 conn(0x7f5f3c0a0260 0x7f5f3c0a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:42.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f431eb640 1 -- 172.21.15.37:0/2496716033 >> 172.21.15.37:0/2496716033 conn(0x7f5f3c0a1ad0 msgr2=0x7f5f3c0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:42.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f431eb640 1 -- 172.21.15.37:0/2496716033 shutdown_connections 2024-04-26T00:21:42.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f431eb640 1 -- 172.21.15.37:0/2496716033 wait complete. 2024-04-26T00:21:42.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f431eb640 10 monclient: build_initial_monmap 2024-04-26T00:21:42.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f431eb640 10 monclient: monmap: 2024-04-26T00:21:42.573 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:42.573 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:21:42.573 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:21:42.573 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:21:42.573 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:42.573 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:21:42.573 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:21:42.573 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:21:42.573 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:21:42.573 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:42.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f431eb640 1 Processor -- start 2024-04-26T00:21:42.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f431eb640 1 -- start start 2024-04-26T00:21:42.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f431eb640 10 monclient: init 2024-04-26T00:21:42.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f431eb640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:42.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f431eb640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:21:42.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f431eb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5f3c0a0260 0x7f5f3c131690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:42.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f431eb640 10 monclient(hunting): picked mon.noname-b con 0x7f5f3c0a0260 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:21:42.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f431eb640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f3c134b30 0x7f5f3c139650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:42.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f431eb640 10 monclient(hunting): picked mon.noname-c con 0x7f5f3c134b30 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:21:42.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f431eb640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5f3c135490 0x7f5f3c139b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:42.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f431eb640 10 monclient(hunting): picked mon.noname-a con 0x7f5f3c135490 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:21:42.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f431eb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:42.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f431eb640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f5f3c09f430 con 0x7f5f3c135490 2024-04-26T00:21:42.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f431eb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:42.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f431eb640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5f3c09f2b0 con 0x7f5f3c0a0260 2024-04-26T00:21:42.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f431eb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:42.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f431eb640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5f3c09fe90 con 0x7f5f3c134b30 2024-04-26T00:21:42.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f431eb640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:42.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f431eb640 10 monclient(hunting): authenticate will time out at 1083.894979s 2024-04-26T00:21:42.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f3bfff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f3c134b30 0x7f5f3c139650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:42.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f41788640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5f3c135490 0x7f5f3c139b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:42.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f3bfff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f3c134b30 0x7f5f3c139650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:56306/0 (socket says 172.21.15.37:56306) 2024-04-26T00:21:42.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f41788640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5f3c135490 0x7f5f3c139b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58524/0 (socket says 172.21.15.37:58524) 2024-04-26T00:21:42.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f3bfff640 1 -- 172.21.15.37:0/3529691555 learned_addr learned my addr 172.21.15.37:0/3529691555 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:21:42.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f3bfff640 10 monclient(hunting): get_auth_request con 0x7f5f3c134b30 auth_method 0 2024-04-26T00:21:42.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:42.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f3bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:42.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:42.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f41788640 10 monclient(hunting): get_auth_request con 0x7f5f3c135490 auth_method 0 2024-04-26T00:21:42.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f41788640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:42.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f41788640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:42.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f41788640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:42.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:42.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:42.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:42.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f41788640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:42.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f41788640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:42.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f41788640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:42.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f40f87640 1 --2- 172.21.15.37:0/3529691555 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5f3c0a0260 0x7f5f3c131690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:42.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f40f87640 10 monclient(hunting): get_auth_request con 0x7f5f3c0a0260 auth_method 0 2024-04-26T00:21:42.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f40f87640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:42.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f40f87640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:42.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f40f87640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:42.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f3bfff640 10 monclient(hunting): handle_auth_done global_id 4544 payload 824 2024-04-26T00:21:42.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f3bfff640 1 -- 172.21.15.37:0/3529691555 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5f3c0a0260 msgr2=0x7f5f3c131690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:42.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f3bfff640 1 --2- 172.21.15.37:0/3529691555 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5f3c0a0260 0x7f5f3c131690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:42.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f3bfff640 1 -- 172.21.15.37:0/3529691555 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5f3c135490 msgr2=0x7f5f3c139b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:42.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f3bfff640 1 --2- 172.21.15.37:0/3529691555 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5f3c135490 0x7f5f3c139b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:42.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f3bfff640 10 monclient: _finish_hunting 0 2024-04-26T00:21:42.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f3bfff640 1 monclient: found mon.noname-c 2024-04-26T00:21:42.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:42.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f3bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:21:42.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.587+0000 7f5f3bfff640 1 -- 172.21.15.37:0/3529691555 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f3c11d030 con 0x7f5f3c134b30 2024-04-26T00:21:42.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.591+0000 7f5f3bfff640 1 -- 172.21.15.37:0/3529691555 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f3c134b30 msgr2=0x7f5f3c139650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:42.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.591+0000 7f5f3bfff640 1 --2- 172.21.15.37:0/3529691555 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f3c134b30 0x7f5f3c139650 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5f2400ebe0 tx=0x7f5f2400c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:42.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.591+0000 7f5f397fa640 1 -- 172.21.15.37:0/3529691555 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f24017070 con 0x7f5f3c134b30 2024-04-26T00:21:42.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.591+0000 7f5f397fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:42.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.591+0000 7f5f397fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:21:42.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.591+0000 7f5f397fa640 10 monclient: dump: 2024-04-26T00:21:42.577 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:42.577 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:42.577 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:42.577 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:42.577 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:42.577 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:42.577 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:42.578 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:42.578 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:42.578 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:42.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.591+0000 7f5f397fa640 10 monclient: _finish_auth 0 2024-04-26T00:21:42.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.591+0000 7f5f397fa640 10 monclient: _check_auth_tickets 2024-04-26T00:21:42.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.591+0000 7f5f397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:42.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.591+0000 7f5f431eb640 5 monclient: authenticate success, global_id 4544 2024-04-26T00:21:42.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.591+0000 7f5f397fa640 1 -- 172.21.15.37:0/3529691555 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f2400cdc0 con 0x7f5f3c134b30 2024-04-26T00:21:42.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.591+0000 7f5f397fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:42.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.591+0000 7f5f397fa640 1 -- 172.21.15.37:0/3529691555 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f24013690 con 0x7f5f3c134b30 2024-04-26T00:21:42.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.591+0000 7f5f397fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:42.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.591+0000 7f5f397fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:21:42.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.591+0000 7f5f397fa640 10 monclient: dump: 2024-04-26T00:21:42.578 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:42.579 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:42.579 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:42.579 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:42.579 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:42.579 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:42.579 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:42.579 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:42.579 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:42.579 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:42.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.591+0000 7f5f3a7fc640 4 set_mon_vals no callback set 2024-04-26T00:21:42.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.591+0000 7f5f431eb640 10 monclient: _renew_subs 2024-04-26T00:21:42.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.591+0000 7f5f431eb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:21:42.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.591+0000 7f5f431eb640 1 -- 172.21.15.37:0/3529691555 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f3c11b620 con 0x7f5f3c134b30 2024-04-26T00:21:42.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.591+0000 7f5f431eb640 10 monclient: _renew_subs 2024-04-26T00:21:42.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.591+0000 7f5f431eb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:21:42.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.591+0000 7f5f431eb640 1 -- 172.21.15.37:0/3529691555 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f3c13b250 con 0x7f5f3c134b30 2024-04-26T00:21:42.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.591+0000 7f5f397fa640 1 -- 172.21.15.37:0/3529691555 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f5f2401a070 con 0x7f5f3c134b30 2024-04-26T00:21:42.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.591+0000 7f5f431eb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:42.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.591+0000 7f5f431eb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:42.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.591+0000 7f5f431eb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:21:42.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.591+0000 7f5f431eb640 1 -- 172.21.15.37:0/3529691555 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f3c11c900 con 0x7f5f3c134b30 2024-04-26T00:21:42.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.591+0000 7f5f397fa640 1 --2- 172.21.15.37:0/3529691555 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5f1c035f00 0x7f5f1c0383c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:42.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.591+0000 7f5f397fa640 1 -- 172.21.15.37:0/3529691555 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(77..77 src has 1..77) ==== 3947+0+0 (secure 0 0 0) 0x7f5f24048930 con 0x7f5f3c134b30 2024-04-26T00:21:42.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.591+0000 7f5f40f87640 1 --2- 172.21.15.37:0/3529691555 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5f1c035f00 0x7f5f1c0383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:42.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.595+0000 7f5f40f87640 10 monclient: get_auth_request con 0x7f5f1c035f00 auth_method 0 2024-04-26T00:21:42.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.595+0000 7f5f40f87640 1 --2- 172.21.15.37:0/3529691555 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5f1c035f00 0x7f5f1c0383c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5f340074a0 tx=0x7f5f3400b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:42.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.599+0000 7f5f397fa640 1 -- 172.21.15.37:0/3529691555 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5f3c11c900 con 0x7f5f3c134b30 2024-04-26T00:21:42.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.599+0000 7f5f397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:42.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.599+0000 7f5f397fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:21:42.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.823+0000 7f5f431eb640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-26T00:21:42.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.823+0000 7f5f431eb640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-26T00:21:42.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.823+0000 7f5f431eb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:21:42.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.823+0000 7f5f431eb640 1 -- 172.21.15.37:0/3529691555 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f5f3c066f40 con 0x7f5f3c134b30 2024-04-26T00:21:42.802 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.823+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:21:42.806 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.823+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=9,config=2,health=10,kv=10,logm=69} crc {auth=3993707498,config=692619927,health=2370057459,kv=1034996044,logm=2538665844}) 2024-04-26T00:21:42.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.823+0000 7f5f397fa640 1 -- 172.21.15.37:0/3529691555 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f5f24047670 con 0x7f5f3c134b30 2024-04-26T00:21:42.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.823+0000 7f5f397fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-26T00:21:42.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.823+0000 7f5f397fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:21:42.808 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.823+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=9,config=2,health=12,kv=9,logm=68} crc {auth=3993707498,config=692619927,health=3897446751,kv=3913313004,logm=2954801876}) 2024-04-26T00:21:42.808 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.823+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:21:42.808 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.823+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=9,config=2,health=10,kv=10,logm=69} crc {auth=3993707498,config=692619927,health=2370057459,kv=1034996044,logm=2538665844}) 2024-04-26T00:21:42.808 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.823+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=9,config=2,health=11,kv=10,logm=68} crc {auth=3993707498,config=692619927,health=661235312,kv=1034996044,logm=3576182413}) 2024-04-26T00:21:42.808 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.827+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:21:42.808 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.827+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2239003766}) 2024-04-26T00:21:42.808 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.827+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3051510195}) 2024-04-26T00:21:42.808 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.827+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:21:42.808 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.827+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2239003766}) 2024-04-26T00:21:42.808 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.827+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2705140827}) 2024-04-26T00:21:42.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.827+0000 7f5f431eb640 1 -- 172.21.15.37:0/3529691555 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5f1c035f00 msgr2=0x7f5f1c0383c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:42.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.827+0000 7f5f431eb640 1 --2- 172.21.15.37:0/3529691555 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5f1c035f00 0x7f5f1c0383c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5f340074a0 tx=0x7f5f3400b040 comp rx=0 tx=0).stop 2024-04-26T00:21:42.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.827+0000 7f5f431eb640 10 monclient: shutdown 2024-04-26T00:21:42.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.827+0000 7f5f431eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:42.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.827+0000 7f5f431eb640 1 -- 172.21.15.37:0/3529691555 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f3c134b30 msgr2=0x7f5f3c139650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:42.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.827+0000 7f5f431eb640 1 --2- 172.21.15.37:0/3529691555 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f3c134b30 0x7f5f3c139650 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5f2400ebe0 tx=0x7f5f2400c520 comp rx=0 tx=0).stop 2024-04-26T00:21:42.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.827+0000 7f5f431eb640 1 -- 172.21.15.37:0/3529691555 shutdown_connections 2024-04-26T00:21:42.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.827+0000 7f5f431eb640 1 --2- 172.21.15.37:0/3529691555 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5f1c035f00 0x7f5f1c0383c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:42.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.827+0000 7f5f431eb640 1 --2- 172.21.15.37:0/3529691555 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5f3c135490 0x7f5f3c139b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:42.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.827+0000 7f5f431eb640 1 --2- 172.21.15.37:0/3529691555 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f3c134b30 0x7f5f3c139650 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:42.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.827+0000 7f5f431eb640 1 --2- 172.21.15.37:0/3529691555 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5f3c0a0260 0x7f5f3c131690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:42.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.827+0000 7f5f431eb640 1 -- 172.21.15.37:0/3529691555 >> 172.21.15.37:0/3529691555 conn(0x7f5f3c0a1ad0 msgr2=0x7f5f3c071b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:42.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.827+0000 7f5f431eb640 1 -- 172.21.15.37:0/3529691555 shutdown_connections 2024-04-26T00:21:42.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:42.827+0000 7f5f431eb640 1 -- 172.21.15.37:0/3529691555 wait complete. 2024-04-26T00:21:42.811 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.827+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:21:42.812 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.831+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1397456320}) 2024-04-26T00:21:42.812 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.831+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2340777003}) 2024-04-26T00:21:42.812 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.831+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:21:42.812 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.831+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1397456320}) 2024-04-26T00:21:42.812 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.831+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=434743428}) 2024-04-26T00:21:42.816 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.839+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:21:42.817 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.839+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=588127416}) 2024-04-26T00:21:42.817 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.839+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4066206611}) 2024-04-26T00:21:42.817 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.839+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:21:42.817 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.839+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=588127416}) 2024-04-26T00:21:42.817 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.839+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2757872306}) 2024-04-26T00:21:42.819 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.839+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:21:42.819 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.839+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2926355052}) 2024-04-26T00:21:42.819 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.839+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2350253690}) 2024-04-26T00:21:42.819 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.839+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:21:42.819 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.839+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2926355052}) 2024-04-26T00:21:42.819 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.839+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=845069695}) 2024-04-26T00:21:42.822 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.843+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:21:42.822 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.843+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=74,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=3} crc {logm=3584320311,mdsmap=2840952586,mgr=2644196481,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=305983333,mon_config_key=2568173539}) 2024-04-26T00:21:42.823 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.843+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=74,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=4} crc {logm=1644720769,mdsmap=2840952586,mgr=2196320830,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=305983333,mon_config_key=1924277172}) 2024-04-26T00:21:42.823 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.843+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:21:42.823 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.843+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=74,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=3} crc {logm=3584320311,mdsmap=2840952586,mgr=2644196481,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=305983333,mon_config_key=2568173539}) 2024-04-26T00:21:42.823 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.843+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=71,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=8} crc {logm=1495609874,mdsmap=3140514009,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=2766381992,mon_config_key=1303549602}) 2024-04-26T00:21:42.824 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-26T00:21:42.834 INFO:tasks.rados.rados.0.smithi045.stdout:315: done (0 left) 2024-04-26T00:21:42.834 INFO:tasks.rados.rados.0.smithi045.stdout:316: write oid 27 current snap is 31 2024-04-26T00:21:42.834 INFO:tasks.rados.rados.0.smithi045.stdout:316: seq_num 144 ranges {420208=530210,1527054=766050} 2024-04-26T00:21:42.841 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.863+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:21:42.841 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.863+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=7,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=10,osdmap=74} crc {mon_config_key=3842760138,monmap=3876866194,osd_metadata=1540339580,osd_pg_creating=1217680737,osd_snap=3949407301,osdmap=1351134925}) 2024-04-26T00:21:42.841 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.863+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=6,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=11,osdmap=73} crc {mon_config_key=1804250073,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=1217680737,osd_snap=1466102038,osdmap=2500378804}) 2024-04-26T00:21:42.841 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.863+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:21:42.842 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.863+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=7,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=10,osdmap=74} crc {mon_config_key=3842760138,monmap=3876866194,osd_metadata=1540339580,osd_pg_creating=1217680737,osd_snap=3949407301,osdmap=1351134925}) 2024-04-26T00:21:42.842 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.863+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=7,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=10,osdmap=73} crc {mon_config_key=3842760138,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=1217680737,osd_snap=1850159317,osdmap=1426400410}) 2024-04-26T00:21:42.846 INFO:tasks.rados.rados.0.smithi045.stdout:316: writing smithi04526579-27 from 420208 to 950418 tid 1 2024-04-26T00:21:42.854 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.875+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:21:42.854 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.875+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=52} crc {osdmap=869407329}) 2024-04-26T00:21:42.854 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.875+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=51} crc {osdmap=3292384229}) 2024-04-26T00:21:42.854 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.875+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:21:42.854 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.875+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=52} crc {osdmap=869407329}) 2024-04-26T00:21:42.854 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:21:42.875+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=51} crc {osdmap=4146344741}) 2024-04-26T00:21:42.861 INFO:tasks.rados.rados.0.smithi045.stdout:316: writing smithi04526579-27 from 1527054 to 2293104 tid 2 2024-04-26T00:21:42.862 INFO:tasks.rados.rados.0.smithi045.stdout:317: snap_remove snap 29 2024-04-26T00:21:42.913 INFO:tasks.rados.rados.0.smithi045.stdout:317: done (1 left) 2024-04-26T00:21:42.913 INFO:tasks.rados.rados.0.smithi045.stdout:318: snap_remove snap 30 2024-04-26T00:21:43.825 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:21:43.825 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:21:43.923 INFO:tasks.rados.rados.0.smithi045.stdout:318: done (1 left) 2024-04-26T00:21:43.923 INFO:tasks.rados.rados.0.smithi045.stdout:319: copy_from oid 5 from oid 19 current snap is 31 2024-04-26T00:21:43.923 INFO:tasks.rados.rados.0.smithi045.stdout:316: finishing write tid 1 to smithi04526579-27 2024-04-26T00:21:43.923 INFO:tasks.rados.rados.0.smithi045.stdout:316: finishing write tid 2 to smithi04526579-27 2024-04-26T00:21:43.923 INFO:tasks.rados.rados.0.smithi045.stdout:316: finishing write tid 3 to smithi04526579-27 2024-04-26T00:21:43.923 INFO:tasks.rados.rados.0.smithi045.stdout:316: finishing write tid 4 to smithi04526579-27 2024-04-26T00:21:43.923 INFO:tasks.rados.rados.0.smithi045.stdout:316: oid 27 updating version 0 to 85 2024-04-26T00:21:43.923 INFO:tasks.rados.rados.0.smithi045.stdout:316: oid 27 version 85 is already newer than 84 2024-04-26T00:21:43.923 INFO:tasks.rados.rados.0.smithi045.stdout:316: oid 27 updating version 85 to 86 2024-04-26T00:21:43.923 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 86 (ObjNum 144 snap 31 seq_num 144) dirty exists 2024-04-26T00:21:43.923 INFO:tasks.rados.rados.0.smithi045.stdout:316: left oid 27 (ObjNum 144 snap 31 seq_num 144) 2024-04-26T00:21:43.923 INFO:tasks.rados.rados.0.smithi045.stdout:316: done (1 left) 2024-04-26T00:21:43.924 INFO:tasks.rados.rados.0.smithi045.stdout:320: snap_remove snap 23 2024-04-26T00:21:43.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb614a6640 10 monclient: get_monmap_and_config 2024-04-26T00:21:43.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb614a6640 10 monclient: build_initial_monmap 2024-04-26T00:21:43.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb614a6640 10 monclient: monmap: 2024-04-26T00:21:43.964 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:43.964 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:43.965 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:21:43.988996+0000 2024-04-26T00:21:43.965 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:21:43.988996+0000 2024-04-26T00:21:43.965 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:43.965 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:43.965 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:21:43.965 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:21:43.965 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:21:43.965 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:21:43.965 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:21:43.965 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:43.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb614a6640 1 Processor -- start 2024-04-26T00:21:43.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb614a6640 1 -- start start 2024-04-26T00:21:43.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb614a6640 10 monclient: init 2024-04-26T00:21:43.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb614a6640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:43.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb614a6640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-04-26T00:21:43.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb614a6640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7feb5c0a0260 0x7feb5c0a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:43.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb614a6640 10 monclient(hunting): picked mon.noname-c con 0x7feb5c0a0260 addr v2:172.21.15.45:3300/0 2024-04-26T00:21:43.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb614a6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feb5c134b30 addr v1:172.21.15.45:6789/0 2024-04-26T00:21:43.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb614a6640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb5c135660 0x7feb5c13ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:43.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb614a6640 10 monclient(hunting): picked mon.noname-b con 0x7feb5c135660 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:21:43.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb614a6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb5c0a1130 con 0x7feb5c134b30 2024-04-26T00:21:43.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb614a6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:43.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb614a6640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7feb5c09ec10 con 0x7feb5c0a0260 2024-04-26T00:21:43.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb614a6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:43.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb614a6640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7feb5c09f0f0 con 0x7feb5c135660 2024-04-26T00:21:43.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb614a6640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:43.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb614a6640 10 monclient(hunting): authenticate will time out at 1085.294653s 2024-04-26T00:21:43.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb5b7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb5c135660 0x7feb5c13ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:43.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb5b7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb5c135660 0x7feb5c13ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:56314/0 (socket says 172.21.15.37:56314) 2024-04-26T00:21:43.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb5b7fe640 1 -- 172.21.15.37:0/4012416348 learned_addr learned my addr 172.21.15.37:0/4012416348 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:21:43.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb5b7fe640 10 monclient(hunting): get_auth_request con 0x7feb5c135660 auth_method 0 2024-04-26T00:21:43.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:43.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb5b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:43.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:43.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb5b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:43.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb5b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:43.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb5b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:43.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb5affd640 1 --2- 172.21.15.37:0/4012416348 >> v2:172.21.15.45:3300/0 conn(0x7feb5c0a0260 0x7feb5c0a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:43.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb5affd640 10 monclient(hunting): get_auth_request con 0x7feb5c0a0260 auth_method 0 2024-04-26T00:21:43.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:43.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb5affd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:43.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb5affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:43.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb5b7fe640 10 monclient(hunting): handle_auth_done global_id 4547 payload 274 2024-04-26T00:21:43.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb5b7fe640 1 -- 172.21.15.37:0/4012416348 >> v2:172.21.15.45:3300/0 conn(0x7feb5c0a0260 msgr2=0x7feb5c0a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:21:43.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb5b7fe640 1 --2- 172.21.15.37:0/4012416348 >> v2:172.21.15.45:3300/0 conn(0x7feb5c0a0260 0x7feb5c0a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:43.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb5b7fe640 1 -- 172.21.15.37:0/4012416348 >> v1:172.21.15.45:6789/0 conn(0x7feb5c134b30 legacy=0x7feb5c134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:21:43.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb5b7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:21:43.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb5b7fe640 1 monclient: found mon.noname-b 2024-04-26T00:21:43.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb5b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:43.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb5b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:21:43.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.987+0000 7feb5b7fe640 1 -- 172.21.15.37:0/4012416348 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb5c06cff0 con 0x7feb5c135660 2024-04-26T00:21:43.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.991+0000 7feb5b7fe640 1 -- 172.21.15.37:0/4012416348 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb5c135660 msgr2=0x7feb5c13ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:43.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.991+0000 7feb5b7fe640 1 --2- 172.21.15.37:0/4012416348 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb5c135660 0x7feb5c13ce70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7feb54009e50 tx=0x7feb5401c6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74dbbf203a54a102 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:43.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.991+0000 7feb59ffb640 1 -- 172.21.15.37:0/4012416348 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb540a9460 con 0x7feb5c135660 2024-04-26T00:21:43.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.991+0000 7feb59ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:43.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.991+0000 7feb59ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:21:43.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.991+0000 7feb59ffb640 10 monclient: dump: 2024-04-26T00:21:43.975 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:43.975 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:43.975 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:43.975 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:43.975 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:43.975 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:43.975 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:43.975 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:43.975 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:43.976 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:43.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.991+0000 7feb59ffb640 10 monclient: _finish_auth 0 2024-04-26T00:21:43.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.991+0000 7feb59ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:21:43.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.991+0000 7feb59ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:43.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.991+0000 7feb614a6640 5 monclient: authenticate success, global_id 4547 2024-04-26T00:21:43.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.991+0000 7feb614a6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:21:43.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.991+0000 7feb59ffb640 1 -- 172.21.15.37:0/4012416348 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb540ae070 con 0x7feb5c135660 2024-04-26T00:21:43.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.991+0000 7feb59ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:43.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.991+0000 7feb614a6640 10 monclient: get_monmap_and_config success 2024-04-26T00:21:43.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.991+0000 7feb614a6640 4 set_mon_vals no callback set 2024-04-26T00:21:43.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.991+0000 7feb59ffb640 1 -- 172.21.15.37:0/4012416348 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb540a9820 con 0x7feb5c135660 2024-04-26T00:21:43.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.991+0000 7feb59ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:43.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.991+0000 7feb59ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:21:43.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.991+0000 7feb59ffb640 10 monclient: dump: 2024-04-26T00:21:43.976 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:43.977 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:43.977 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:43.977 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:43.977 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:43.977 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:43.977 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:43.977 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:43.977 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:43.977 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:43.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb614a6640 10 monclient: shutdown 2024-04-26T00:21:43.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb614a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:43.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb614a6640 1 -- 172.21.15.37:0/4012416348 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb5c135660 msgr2=0x7feb5c13ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:43.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb614a6640 1 --2- 172.21.15.37:0/4012416348 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb5c135660 0x7feb5c13ce70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7feb54009e50 tx=0x7feb5401c6d0 comp rx=0 tx=0).stop 2024-04-26T00:21:43.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb614a6640 1 -- 172.21.15.37:0/4012416348 shutdown_connections 2024-04-26T00:21:43.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb614a6640 1 --2- 172.21.15.37:0/4012416348 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb5c135660 0x7feb5c13ce70 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:43.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb614a6640 1 --2- 172.21.15.37:0/4012416348 >> v2:172.21.15.45:3300/0 conn(0x7feb5c0a0260 0x7feb5c0a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:43.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb614a6640 1 -- 172.21.15.37:0/4012416348 >> 172.21.15.37:0/4012416348 conn(0x7feb5c0a1ad0 msgr2=0x7feb5c0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:43.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb614a6640 1 -- 172.21.15.37:0/4012416348 shutdown_connections 2024-04-26T00:21:43.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb614a6640 1 -- 172.21.15.37:0/4012416348 wait complete. 2024-04-26T00:21:43.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb614a6640 10 monclient: build_initial_monmap 2024-04-26T00:21:43.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb614a6640 10 monclient: monmap: 2024-04-26T00:21:43.978 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:43.978 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:21:43.978 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:21:43.978 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:21:43.978 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:43.978 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:21:43.978 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:21:43.978 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:21:43.979 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:21:43.979 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:43.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb614a6640 1 Processor -- start 2024-04-26T00:21:43.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb614a6640 1 -- start start 2024-04-26T00:21:43.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb614a6640 10 monclient: init 2024-04-26T00:21:43.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb614a6640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:43.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb614a6640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:21:43.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb614a6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb5c0a0260 0x7feb5c1cce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:43.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb614a6640 10 monclient(hunting): picked mon.noname-b con 0x7feb5c0a0260 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:21:43.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb614a6640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb5c134b30 0x7feb5c131b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:43.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb614a6640 10 monclient(hunting): picked mon.noname-a con 0x7feb5c134b30 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:21:43.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb614a6640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb5c135660 0x7feb5c139d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:43.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb614a6640 10 monclient(hunting): picked mon.noname-c con 0x7feb5c135660 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:21:43.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb614a6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:43.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb614a6640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7feb5c0a0010 con 0x7feb5c134b30 2024-04-26T00:21:43.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb614a6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:43.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb614a6640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7feb5c09fe90 con 0x7feb5c0a0260 2024-04-26T00:21:43.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb614a6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:43.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb614a6640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7feb5c13fbd0 con 0x7feb5c135660 2024-04-26T00:21:43.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb614a6640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:43.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb614a6640 10 monclient(hunting): authenticate will time out at 1085.301953s 2024-04-26T00:21:43.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb5b7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb5c135660 0x7feb5c139d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:43.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb5a7fc640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb5c134b30 0x7feb5c131b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:43.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb5b7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb5c135660 0x7feb5c139d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:56326/0 (socket says 172.21.15.37:56326) 2024-04-26T00:21:43.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb5b7fe640 1 -- 172.21.15.37:0/786657598 learned_addr learned my addr 172.21.15.37:0/786657598 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:21:43.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb5b7fe640 10 monclient(hunting): get_auth_request con 0x7feb5c135660 auth_method 0 2024-04-26T00:21:43.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:43.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb5b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:43.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:43.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb5a7fc640 10 monclient(hunting): get_auth_request con 0x7feb5c134b30 auth_method 0 2024-04-26T00:21:43.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb5a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:43.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb5a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:43.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:43.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb5b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:43.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb5b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:43.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb5b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:43.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb5a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:43.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb5a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:43.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb5a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:43.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb5affd640 1 --2- 172.21.15.37:0/786657598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb5c0a0260 0x7feb5c1cce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:43.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb5affd640 10 monclient(hunting): get_auth_request con 0x7feb5c0a0260 auth_method 0 2024-04-26T00:21:43.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:43.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb5affd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:43.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb5affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:43.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb5a7fc640 10 monclient(hunting): handle_auth_done global_id 4557 payload 824 2024-04-26T00:21:43.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb5a7fc640 1 -- 172.21.15.37:0/786657598 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb5c135660 msgr2=0x7feb5c139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:43.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb5a7fc640 1 --2- 172.21.15.37:0/786657598 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb5c135660 0x7feb5c139d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:43.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb5a7fc640 1 -- 172.21.15.37:0/786657598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb5c0a0260 msgr2=0x7feb5c1cce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:43.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb5a7fc640 1 --2- 172.21.15.37:0/786657598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb5c0a0260 0x7feb5c1cce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:43.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb5a7fc640 10 monclient: _finish_hunting 0 2024-04-26T00:21:43.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb5a7fc640 1 monclient: found mon.noname-a 2024-04-26T00:21:43.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb5a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:43.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb5a7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:21:43.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb5a7fc640 1 -- 172.21.15.37:0/786657598 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb5c13b930 con 0x7feb5c134b30 2024-04-26T00:21:43.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.995+0000 7feb5b7fe640 1 --2- 172.21.15.37:0/786657598 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb5c135660 0x7feb5c139d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:21:43.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.999+0000 7feb5a7fc640 1 -- 172.21.15.37:0/786657598 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb5c134b30 msgr2=0x7feb5c131b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:43.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.999+0000 7feb5a7fc640 1 --2- 172.21.15.37:0/786657598 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb5c134b30 0x7feb5c131b70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7feb5000c3e0 tx=0x7feb5000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:43.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.999+0000 7feb437fe640 1 -- 172.21.15.37:0/786657598 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb500119a0 con 0x7feb5c134b30 2024-04-26T00:21:43.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.999+0000 7feb437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:43.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.999+0000 7feb437fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:21:43.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.999+0000 7feb437fe640 10 monclient: dump: 2024-04-26T00:21:43.984 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:43.985 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:43.985 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:43.985 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:43.985 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:43.985 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:43.985 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:43.985 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:43.985 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:43.985 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:43.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.999+0000 7feb437fe640 10 monclient: _finish_auth 0 2024-04-26T00:21:43.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.999+0000 7feb437fe640 10 monclient: _check_auth_tickets 2024-04-26T00:21:43.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.999+0000 7feb437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:43.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.999+0000 7feb437fe640 1 -- 172.21.15.37:0/786657598 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb50017070 con 0x7feb5c134b30 2024-04-26T00:21:43.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.999+0000 7feb614a6640 5 monclient: authenticate success, global_id 4557 2024-04-26T00:21:43.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.999+0000 7feb614a6640 10 monclient: _renew_subs 2024-04-26T00:21:43.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.999+0000 7feb614a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:43.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.999+0000 7feb614a6640 1 -- 172.21.15.37:0/786657598 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb5c0729d0 con 0x7feb5c134b30 2024-04-26T00:21:43.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.999+0000 7feb437fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:43.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.999+0000 7feb437fe640 1 -- 172.21.15.37:0/786657598 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb50011d80 con 0x7feb5c134b30 2024-04-26T00:21:43.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.999+0000 7feb437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:43.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.999+0000 7feb437fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:21:43.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.999+0000 7feb437fe640 10 monclient: dump: 2024-04-26T00:21:43.986 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:43.986 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:43.986 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:43.986 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:43.986 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:43.986 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:43.987 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:43.987 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:43.987 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:43.987 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:43.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.999+0000 7feb614a6640 10 monclient: _renew_subs 2024-04-26T00:21:43.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.999+0000 7feb614a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:43.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.999+0000 7feb614a6640 1 -- 172.21.15.37:0/786657598 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb5c11c500 con 0x7feb5c134b30 2024-04-26T00:21:43.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.999+0000 7feb58ff9640 4 set_mon_vals no callback set 2024-04-26T00:21:43.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.999+0000 7feb614a6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:43.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.999+0000 7feb614a6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:43.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.999+0000 7feb614a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:43.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.999+0000 7feb614a6640 1 -- 172.21.15.37:0/786657598 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb5c066f40 con 0x7feb5c134b30 2024-04-26T00:21:43.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.999+0000 7feb437fe640 1 -- 172.21.15.37:0/786657598 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7feb5001c460 con 0x7feb5c134b30 2024-04-26T00:21:43.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.999+0000 7feb437fe640 1 --2- 172.21.15.37:0/786657598 >> v2:172.21.15.45:6812/2379918623 conn(0x7feb48035e80 0x7feb48038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:43.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.999+0000 7feb437fe640 1 -- 172.21.15.37:0/786657598 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 3943+0+0 (secure 0 0 0) 0x7feb500488d0 con 0x7feb5c134b30 2024-04-26T00:21:43.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.999+0000 7feb5affd640 1 --2- 172.21.15.37:0/786657598 >> v2:172.21.15.45:6812/2379918623 conn(0x7feb48035e80 0x7feb48038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:43.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:43.999+0000 7feb5affd640 10 monclient: get_auth_request con 0x7feb48035e80 auth_method 0 2024-04-26T00:21:43.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.003+0000 7feb5affd640 1 --2- 172.21.15.37:0/786657598 >> v2:172.21.15.45:6812/2379918623 conn(0x7feb48035e80 0x7feb48038340 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7feb5c13cd40 tx=0x7feb4c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:43.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.003+0000 7feb437fe640 1 -- 172.21.15.37:0/786657598 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7feb5001a760 con 0x7feb5c134b30 2024-04-26T00:21:43.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.003+0000 7feb437fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:43.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.003+0000 7feb437fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:21:44.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.231+0000 7feb614a6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:21:44.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.231+0000 7feb614a6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:21:44.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.231+0000 7feb614a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:44.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.231+0000 7feb614a6640 1 -- 172.21.15.37:0/786657598 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7feb5c133580 con 0x7feb5c134b30 2024-04-26T00:21:44.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.231+0000 7feb437fe640 1 -- 172.21.15.37:0/786657598 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7feb50012420 con 0x7feb5c134b30 2024-04-26T00:21:44.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.231+0000 7feb437fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:21:44.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.231+0000 7feb437fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:21:44.211 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":18,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:21:44.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.235+0000 7feb614a6640 1 -- 172.21.15.37:0/786657598 >> v2:172.21.15.45:6812/2379918623 conn(0x7feb48035e80 msgr2=0x7feb48038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:44.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.235+0000 7feb614a6640 1 --2- 172.21.15.37:0/786657598 >> v2:172.21.15.45:6812/2379918623 conn(0x7feb48035e80 0x7feb48038340 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7feb5c13cd40 tx=0x7feb4c008040 comp rx=0 tx=0).stop 2024-04-26T00:21:44.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.235+0000 7feb614a6640 10 monclient: shutdown 2024-04-26T00:21:44.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.235+0000 7feb614a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:44.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.235+0000 7feb614a6640 1 -- 172.21.15.37:0/786657598 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb5c134b30 msgr2=0x7feb5c131b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:44.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.235+0000 7feb614a6640 1 --2- 172.21.15.37:0/786657598 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb5c134b30 0x7feb5c131b70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7feb5000c3e0 tx=0x7feb5000c8a0 comp rx=0 tx=0).stop 2024-04-26T00:21:44.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.235+0000 7feb614a6640 1 -- 172.21.15.37:0/786657598 shutdown_connections 2024-04-26T00:21:44.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.235+0000 7feb614a6640 1 --2- 172.21.15.37:0/786657598 >> v2:172.21.15.45:6812/2379918623 conn(0x7feb48035e80 0x7feb48038340 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:44.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.235+0000 7feb614a6640 1 --2- 172.21.15.37:0/786657598 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb5c135660 0x7feb5c139d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:44.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.235+0000 7feb614a6640 1 --2- 172.21.15.37:0/786657598 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb5c134b30 0x7feb5c131b70 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:44.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.235+0000 7feb614a6640 1 --2- 172.21.15.37:0/786657598 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feb5c0a0260 0x7feb5c1cce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:44.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.235+0000 7feb614a6640 1 -- 172.21.15.37:0/786657598 >> 172.21.15.37:0/786657598 conn(0x7feb5c0a1ad0 msgr2=0x7feb5c0a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:44.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.235+0000 7feb614a6640 1 -- 172.21.15.37:0/786657598 shutdown_connections 2024-04-26T00:21:44.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.235+0000 7feb614a6640 1 -- 172.21.15.37:0/786657598 wait complete. 2024-04-26T00:21:44.230 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:21:44.230 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-26T00:21:44.230 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:21:44.231 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:21:44.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa858c44640 10 monclient: get_monmap_and_config 2024-04-26T00:21:44.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa858c44640 10 monclient: build_initial_monmap 2024-04-26T00:21:44.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa858c44640 10 monclient: monmap: 2024-04-26T00:21:44.348 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:44.348 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:44.348 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:21:44.372461+0000 2024-04-26T00:21:44.348 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:21:44.372461+0000 2024-04-26T00:21:44.348 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:44.348 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:44.348 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:21:44.348 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:21:44.348 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:21:44.348 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:21:44.349 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:21:44.349 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:44.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa858c44640 1 Processor -- start 2024-04-26T00:21:44.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa858c44640 1 -- start start 2024-04-26T00:21:44.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa858c44640 10 monclient: init 2024-04-26T00:21:44.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa858c44640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:44.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa858c44640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-04-26T00:21:44.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa858c44640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa8540a3d90 addr v1:172.21.15.45:6789/0 2024-04-26T00:21:44.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa858c44640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa8540a4170 0x7fa8540a45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:44.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa858c44640 10 monclient(hunting): picked mon.noname-a con 0x7fa8540a4170 addr v2:172.21.15.37:3300/0 2024-04-26T00:21:44.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa858c44640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa85413c3e0 0x7fa85413e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:44.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa858c44640 10 monclient(hunting): picked mon.noname-b con 0x7fa85413c3e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:21:44.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa858c44640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8540a5150 con 0x7fa8540a3d90 2024-04-26T00:21:44.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa858c44640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:44.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa858c44640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fa85409c650 con 0x7fa8540a4170 2024-04-26T00:21:44.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa858c44640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:44.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa858c44640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa85409b280 con 0x7fa85413c3e0 2024-04-26T00:21:44.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa858c44640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:44.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa858c44640 10 monclient(hunting): authenticate will time out at 1085.678643s 2024-04-26T00:21:44.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa852d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa85413c3e0 0x7fa85413e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:44.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa84bfff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa8540a4170 0x7fa8540a45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:44.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa852d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa85413c3e0 0x7fa85413e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:56342/0 (socket says 172.21.15.37:56342) 2024-04-26T00:21:44.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa852d9d640 1 -- 172.21.15.37:0/2154613937 learned_addr learned my addr 172.21.15.37:0/2154613937 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:21:44.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa84bfff640 10 monclient(hunting): get_auth_request con 0x7fa8540a4170 auth_method 0 2024-04-26T00:21:44.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa84bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:44.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa84bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:44.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa84bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:44.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa852d9d640 10 monclient(hunting): get_auth_request con 0x7fa85413c3e0 auth_method 0 2024-04-26T00:21:44.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa852d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:44.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa852d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:44.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa852d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:44.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa84bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:44.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa84bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:44.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa84bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:44.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa852d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:44.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa852d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:44.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa852d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:44.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa84bfff640 10 monclient(hunting): handle_auth_done global_id 4560 payload 274 2024-04-26T00:21:44.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa84bfff640 1 -- 172.21.15.37:0/2154613937 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa85413c3e0 msgr2=0x7fa85413e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:21:44.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa84bfff640 1 --2- 172.21.15.37:0/2154613937 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa85413c3e0 0x7fa85413e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:44.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa84bfff640 1 -- 172.21.15.37:0/2154613937 >> v1:172.21.15.45:6789/0 conn(0x7fa8540a3d90 legacy=0x7fa85409bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:21:44.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa84bfff640 10 monclient: _finish_hunting 0 2024-04-26T00:21:44.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa84bfff640 1 monclient: found mon.noname-a 2024-04-26T00:21:44.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa84bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:44.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa84bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:21:44.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.371+0000 7fa84bfff640 1 -- 172.21.15.37:0/2154613937 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa85409b560 con 0x7fa8540a4170 2024-04-26T00:21:44.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.375+0000 7fa84bfff640 1 -- 172.21.15.37:0/2154613937 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8540a4170 msgr2=0x7fa8540a45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:44.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.375+0000 7fa84bfff640 1 --2- 172.21.15.37:0/2154613937 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8540a4170 0x7fa8540a45e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa83c0097c0 tx=0x7fa83c01c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7a8979bdc5370ac server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:44.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.375+0000 7fa851d9b640 1 -- 172.21.15.37:0/2154613937 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa83c0b0070 con 0x7fa8540a4170 2024-04-26T00:21:44.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.375+0000 7fa851d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:44.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.375+0000 7fa851d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:21:44.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.375+0000 7fa851d9b640 10 monclient: dump: 2024-04-26T00:21:44.356 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:44.356 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:44.356 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:44.356 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:44.356 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:44.357 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:44.357 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:44.357 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:44.357 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:44.357 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:44.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.375+0000 7fa851d9b640 10 monclient: _finish_auth 0 2024-04-26T00:21:44.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.375+0000 7fa851d9b640 10 monclient: _check_auth_tickets 2024-04-26T00:21:44.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.375+0000 7fa851d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:44.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.375+0000 7fa851d9b640 1 -- 172.21.15.37:0/2154613937 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa83c0ab600 con 0x7fa8540a4170 2024-04-26T00:21:44.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.375+0000 7fa858c44640 5 monclient: authenticate success, global_id 4560 2024-04-26T00:21:44.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.375+0000 7fa858c44640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:21:44.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.375+0000 7fa851d9b640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:44.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.375+0000 7fa858c44640 10 monclient: get_monmap_and_config success 2024-04-26T00:21:44.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.375+0000 7fa851d9b640 1 -- 172.21.15.37:0/2154613937 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa83c0ab9d0 con 0x7fa8540a4170 2024-04-26T00:21:44.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.375+0000 7fa858c44640 4 set_mon_vals no callback set 2024-04-26T00:21:44.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.375+0000 7fa851d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:44.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.375+0000 7fa851d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:21:44.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.375+0000 7fa851d9b640 10 monclient: dump: 2024-04-26T00:21:44.365 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:44.365 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:44.365 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:44.365 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:44.365 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:44.365 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:44.365 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:44.366 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:44.366 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:44.366 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:44.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa858c44640 10 monclient: shutdown 2024-04-26T00:21:44.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa858c44640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:44.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa858c44640 1 -- 172.21.15.37:0/2154613937 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8540a4170 msgr2=0x7fa8540a45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:44.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa858c44640 1 --2- 172.21.15.37:0/2154613937 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8540a4170 0x7fa8540a45e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa83c0097c0 tx=0x7fa83c01c320 comp rx=0 tx=0).stop 2024-04-26T00:21:44.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa858c44640 1 -- 172.21.15.37:0/2154613937 shutdown_connections 2024-04-26T00:21:44.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa858c44640 1 --2- 172.21.15.37:0/2154613937 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa85413c3e0 0x7fa85413e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:44.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa858c44640 1 --2- 172.21.15.37:0/2154613937 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8540a4170 0x7fa8540a45e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:44.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa858c44640 1 -- 172.21.15.37:0/2154613937 >> 172.21.15.37:0/2154613937 conn(0x7fa8540aa0a0 msgr2=0x7fa8540a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:44.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa858c44640 1 -- 172.21.15.37:0/2154613937 shutdown_connections 2024-04-26T00:21:44.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa858c44640 1 -- 172.21.15.37:0/2154613937 wait complete. 2024-04-26T00:21:44.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa858c44640 10 monclient: build_initial_monmap 2024-04-26T00:21:44.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa858c44640 10 monclient: monmap: 2024-04-26T00:21:44.367 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:44.367 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:21:44.367 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:21:44.367 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:21:44.367 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:44.367 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:21:44.367 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:21:44.367 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:21:44.367 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:21:44.367 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:44.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa858c44640 1 Processor -- start 2024-04-26T00:21:44.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa858c44640 1 -- start start 2024-04-26T00:21:44.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa858c44640 10 monclient: init 2024-04-26T00:21:44.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa858c44640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:44.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa858c44640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:21:44.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa858c44640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa8540a3d90 0x7fa8541ced20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:44.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa858c44640 10 monclient(hunting): picked mon.noname-c con 0x7fa8540a3d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:21:44.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa858c44640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8540a4170 0x7fa854136ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:44.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa858c44640 10 monclient(hunting): picked mon.noname-a con 0x7fa8540a4170 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:21:44.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa858c44640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa85413c3e0 0x7fa854139ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:44.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa858c44640 10 monclient(hunting): picked mon.noname-b con 0x7fa85413c3e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:21:44.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa858c44640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:44.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa858c44640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa8541418c0 con 0x7fa8540a4170 2024-04-26T00:21:44.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa858c44640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:44.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa858c44640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa854141740 con 0x7fa85413c3e0 2024-04-26T00:21:44.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa858c44640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:44.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa858c44640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa854141a40 con 0x7fa8540a3d90 2024-04-26T00:21:44.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa858c44640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:44.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa858c44640 10 monclient(hunting): authenticate will time out at 1085.685958s 2024-04-26T00:21:44.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa84bfff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8540a4170 0x7fa854136ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:44.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa85259c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa8540a3d90 0x7fa8541ced20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:44.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa84bfff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8540a4170 0x7fa854136ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58548/0 (socket says 172.21.15.37:58548) 2024-04-26T00:21:44.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa84bfff640 1 -- 172.21.15.37:0/3209026724 learned_addr learned my addr 172.21.15.37:0/3209026724 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:21:44.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa84bfff640 10 monclient(hunting): get_auth_request con 0x7fa8540a4170 auth_method 0 2024-04-26T00:21:44.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa84bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:44.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa84bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:44.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa84bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:44.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa85259c640 10 monclient(hunting): get_auth_request con 0x7fa8540a3d90 auth_method 0 2024-04-26T00:21:44.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa85259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:44.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa85259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:44.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa85259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:44.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa84bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:44.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa84bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:44.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa84bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:44.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa85259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:44.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa85259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:44.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa85259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:44.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa852d9d640 1 --2- 172.21.15.37:0/3209026724 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa85413c3e0 0x7fa854139ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:44.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa852d9d640 10 monclient(hunting): get_auth_request con 0x7fa85413c3e0 auth_method 0 2024-04-26T00:21:44.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa852d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:44.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa852d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:44.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa852d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:44.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa84bfff640 10 monclient(hunting): handle_auth_done global_id 4563 payload 824 2024-04-26T00:21:44.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa84bfff640 1 -- 172.21.15.37:0/3209026724 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa8540a3d90 msgr2=0x7fa8541ced20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:44.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa84bfff640 1 --2- 172.21.15.37:0/3209026724 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa8540a3d90 0x7fa8541ced20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:44.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa84bfff640 1 -- 172.21.15.37:0/3209026724 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa85413c3e0 msgr2=0x7fa854139ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:44.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa84bfff640 1 --2- 172.21.15.37:0/3209026724 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa85413c3e0 0x7fa854139ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:44.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa84bfff640 10 monclient: _finish_hunting 0 2024-04-26T00:21:44.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa84bfff640 1 monclient: found mon.noname-a 2024-04-26T00:21:44.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa84bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:44.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa84bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:21:44.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.379+0000 7fa84bfff640 1 -- 172.21.15.37:0/3209026724 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa85413c930 con 0x7fa8540a4170 2024-04-26T00:21:44.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa84bfff640 1 -- 172.21.15.37:0/3209026724 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8540a4170 msgr2=0x7fa854136ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:44.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa84bfff640 1 --2- 172.21.15.37:0/3209026724 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8540a4170 0x7fa854136ec0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa83c01a9d0 tx=0x7fa83c0190c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:44.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa84affd640 1 -- 172.21.15.37:0/3209026724 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa83c0b8020 con 0x7fa8540a4170 2024-04-26T00:21:44.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa84affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:44.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa84affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:21:44.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa84affd640 10 monclient: dump: 2024-04-26T00:21:44.371 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:44.372 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:44.372 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:44.372 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:44.372 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:44.372 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:44.372 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:44.372 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:44.372 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:44.372 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:44.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa84affd640 10 monclient: _finish_auth 0 2024-04-26T00:21:44.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa84affd640 10 monclient: _check_auth_tickets 2024-04-26T00:21:44.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa84affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:44.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa84affd640 1 -- 172.21.15.37:0/3209026724 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa83c0b4620 con 0x7fa8540a4170 2024-04-26T00:21:44.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa84affd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:44.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa84affd640 1 -- 172.21.15.37:0/3209026724 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa83c0b0040 con 0x7fa8540a4170 2024-04-26T00:21:44.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa84affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:44.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa84affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:21:44.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa84affd640 10 monclient: dump: 2024-04-26T00:21:44.373 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:44.373 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:44.373 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:44.373 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:44.373 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:44.373 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:44.373 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:44.373 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:44.373 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:44.373 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:44.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa858c44640 5 monclient: authenticate success, global_id 4563 2024-04-26T00:21:44.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa858c44640 10 monclient: _renew_subs 2024-04-26T00:21:44.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa850d99640 4 set_mon_vals no callback set 2024-04-26T00:21:44.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa858c44640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:44.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa858c44640 1 -- 172.21.15.37:0/3209026724 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa85413baf0 con 0x7fa8540a4170 2024-04-26T00:21:44.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa858c44640 10 monclient: _renew_subs 2024-04-26T00:21:44.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa858c44640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:44.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa858c44640 1 -- 172.21.15.37:0/3209026724 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa854120fb0 con 0x7fa8540a4170 2024-04-26T00:21:44.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa858c44640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:44.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa858c44640 1 --2- 172.21.15.37:0/3209026724 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa85413a3e0 0x7fa85413a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:44.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa858c44640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:44.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa858c44640 1 -- 172.21.15.37:0/3209026724 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa8541cfca0 con 0x7fa85413a3e0 2024-04-26T00:21:44.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa85259c640 1 --2- 172.21.15.37:0/3209026724 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa85413a3e0 0x7fa85413a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:44.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa85259c640 10 monclient: get_auth_request con 0x7fa85413a3e0 auth_method 0 2024-04-26T00:21:44.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa85259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:44.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa85259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:44.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa85259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:44.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa85259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:44.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa85259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:44.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa85259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:44.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa84affd640 1 -- 172.21.15.37:0/3209026724 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa83c0bb020 con 0x7fa8540a4170 2024-04-26T00:21:44.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa85259c640 10 monclient(hunting): handle_auth_done global_id 4566 payload 824 2024-04-26T00:21:44.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa85259c640 1 -- 172.21.15.37:0/3209026724 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa85404f510 con 0x7fa85413a3e0 2024-04-26T00:21:44.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa84affd640 1 --2- 172.21.15.37:0/3209026724 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa834035e80 0x7fa834038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:44.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa84affd640 1 -- 172.21.15.37:0/3209026724 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 3943+0+0 (secure 0 0 0) 0x7fa83c0e24e0 con 0x7fa8540a4170 2024-04-26T00:21:44.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa852d9d640 1 --2- 172.21.15.37:0/3209026724 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa834035e80 0x7fa834038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:44.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.383+0000 7fa852d9d640 10 monclient: get_auth_request con 0x7fa834035e80 auth_method 0 2024-04-26T00:21:44.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.387+0000 7fa85259c640 1 -- 172.21.15.37:0/3209026724 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa85413a3e0 msgr2=0x7fa85413a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:44.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.387+0000 7fa85259c640 1 --2- 172.21.15.37:0/3209026724 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa85413a3e0 0x7fa85413a7c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa84400f770 tx=0x7fa84400fc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:44.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.387+0000 7fa84affd640 1 -- 172.21.15.37:0/3209026724 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa84400fe40 con 0x7fa85413a3e0 2024-04-26T00:21:44.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.387+0000 7fa84affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:44.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.387+0000 7fa84affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:21:44.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.387+0000 7fa84affd640 10 monclient: dump: 2024-04-26T00:21:44.376 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:44.376 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:44.376 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:44.376 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:44.376 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:44.376 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:44.376 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:44.376 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:44.376 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:44.377 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:44.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.387+0000 7fa84affd640 1 -- 172.21.15.37:0/3209026724 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa844019070 con 0x7fa85413a3e0 2024-04-26T00:21:44.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.387+0000 7fa84affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:44.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.387+0000 7fa84affd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:21:44.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.387+0000 7fa84affd640 1 -- 172.21.15.37:0/3209026724 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa85413a3e0 msgr2=0x7fa85413a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:44.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.387+0000 7fa84affd640 1 --2- 172.21.15.37:0/3209026724 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa85413a3e0 0x7fa85413a7c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa84400f770 tx=0x7fa84400fc30 comp rx=0 tx=0).stop 2024-04-26T00:21:44.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.387+0000 7fa84affd640 1 -- 172.21.15.37:0/3209026724 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa85413a3e0 msgr2=0x7fa85413a7c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:21:44.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.387+0000 7fa84affd640 1 --2- 172.21.15.37:0/3209026724 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa85413a3e0 0x7fa85413a7c0 secure :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa84400f770 tx=0x7fa84400fc30 comp rx=0 tx=0).stop 2024-04-26T00:21:44.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.387+0000 7fa852d9d640 1 --2- 172.21.15.37:0/3209026724 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa834035e80 0x7fa834038340 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa854120cf0 tx=0x7fa84c00b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:44.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.391+0000 7fa858c44640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:21:44.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.391+0000 7fa858c44640 1 --2- 172.21.15.37:0/3209026724 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa85413ad00 0x7fa85409e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:44.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.391+0000 7fa858c44640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:44.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.391+0000 7fa858c44640 1 -- 172.21.15.37:0/3209026724 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa854121320 con 0x7fa85413ad00 2024-04-26T00:21:44.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.391+0000 7fa85259c640 1 --2- 172.21.15.37:0/3209026724 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa85413ad00 0x7fa85409e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:44.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.391+0000 7fa85259c640 10 monclient: get_auth_request con 0x7fa85413ad00 auth_method 0 2024-04-26T00:21:44.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.391+0000 7fa85259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:44.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.391+0000 7fa85259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:44.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.391+0000 7fa85259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:44.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.391+0000 7fa85259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:44.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.391+0000 7fa85259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:44.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.391+0000 7fa85259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:44.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.391+0000 7fa85259c640 10 monclient(hunting): handle_auth_done global_id 4569 payload 824 2024-04-26T00:21:44.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.391+0000 7fa85259c640 1 -- 172.21.15.37:0/3209026724 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa85413b110 con 0x7fa85413ad00 2024-04-26T00:21:44.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.395+0000 7fa85259c640 1 -- 172.21.15.37:0/3209026724 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa85413ad00 msgr2=0x7fa85409e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:44.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.395+0000 7fa85259c640 1 --2- 172.21.15.37:0/3209026724 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa85413ad00 0x7fa85409e080 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa84400f770 tx=0x7fa844003770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:44.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.395+0000 7fa84affd640 1 -- 172.21.15.37:0/3209026724 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa844006e00 con 0x7fa85413ad00 2024-04-26T00:21:44.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.395+0000 7fa84affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:44.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.395+0000 7fa84affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:21:44.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.395+0000 7fa84affd640 10 monclient: dump: 2024-04-26T00:21:44.380 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:44.380 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:44.380 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:44.380 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:44.380 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:44.381 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:44.381 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:44.381 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:44.381 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:44.381 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:44.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.395+0000 7fa84affd640 1 -- 172.21.15.37:0/3209026724 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4889 (secure 0 0 0) 0x7fa844006e00 con 0x7fa85413ad00 2024-04-26T00:21:44.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.395+0000 7fa84affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:21:44.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.395+0000 7fa84affd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:21:44.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.395+0000 7fa84affd640 1 -- 172.21.15.37:0/3209026724 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa85413ad00 msgr2=0x7fa85409e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:44.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.395+0000 7fa84affd640 1 --2- 172.21.15.37:0/3209026724 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa85413ad00 0x7fa85409e080 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa84400f770 tx=0x7fa844003770 comp rx=0 tx=0).stop 2024-04-26T00:21:44.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.395+0000 7fa84affd640 1 -- 172.21.15.37:0/3209026724 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa85413ad00 msgr2=0x7fa85409e080 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:21:44.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.395+0000 7fa84affd640 1 --2- 172.21.15.37:0/3209026724 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa85413ad00 0x7fa85409e080 secure :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa84400f770 tx=0x7fa844003770 comp rx=0 tx=0).stop 2024-04-26T00:21:44.382 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:21:44.382 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:21:44.382 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:21:44.382 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:21:44.382 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 18, 2024-04-26T00:21:44.382 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 121498, 2024-04-26T00:21:44.382 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:21:44.382 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:21:44.382 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:21:44.382 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:21:44.382 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:44.382 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 6, 2024-04-26T00:21:44.382 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:21:44.383 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:21:44.383 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:21:44.383 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:21:44.383 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:21:44.383 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:21:44.383 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:21:44.383 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:21:44.383 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:21:44.383 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:21:44.383 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:21:44.383 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:21:44.383 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:21:44.383 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:21:44.383 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:44.383 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:21:44.384 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:21:44.384 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:21:44.384 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:21:44.384 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:21:44.384 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:21:44.384 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:21:44.384 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:21:44.384 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:21:44.384 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:21:44.384 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:21:44.384 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:21:44.384 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:21:44.384 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:44.384 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:44.385 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:21:44.385 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:21:44.385 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:21:44.385 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:21:44.385 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:21:44.385 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:21:44.385 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:21:44.385 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:21:44.385 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:21:44.385 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:21:44.385 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:21:44.385 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:21:44.385 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:21:44.385 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:21:44.385 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:21:44.386 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:21:44.386 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:21:44.386 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:21:44.386 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:21:44.386 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:21:44.386 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:21:44.386 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:21:44.386 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:21:44.386 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:21:44.386 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:21:44.386 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:21:44.386 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:21:44.386 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:21:44.386 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:44.387 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:21:44.387 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:44.387 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:21:44.387 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:44.387 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:21:44.387 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:21:44.387 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:21:44.387 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:21:44.387 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:44.387 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:21:44.391 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:21:44.392 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:44.392 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:44.392 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:44.392 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:21:44.392 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:21:44.392 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:44.393 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:44.393 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:44.393 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:44.393 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:21:44.393 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:21:44.393 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:21:44.393 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:21:44.393 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:21:44.394 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:44.394 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:44.394 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:21:44.394 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:21:44.394 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:21:44.394 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:21:44.394 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:44.395 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:21:44.395 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:21:44.395 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:44.395 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:44.395 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:44.395 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:21:44.395 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:21:44.395 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:44.396 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:44.396 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:44.396 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:44.396 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:21:44.396 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:21:44.396 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:21:44.396 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:21:44.396 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:21:44.396 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:44.397 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:44.397 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:21:44.397 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:21:44.397 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:21:44.397 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:21:44.397 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:44.397 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:21:44.397 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:21:44.398 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:44.398 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:44.398 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:44.398 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:21:44.398 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:21:44.398 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:44.398 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:44.399 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:44.399 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:44.399 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:21:44.399 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:21:44.399 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:21:44.399 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:21:44.399 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:21:44.472 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:44.473 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:44.473 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:44.473 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:21:44.473 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:21:44.473 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:44.473 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:21:44.473 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:21:44.473 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:21:44.473 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:44.473 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:44.473 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:21:44.473 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:44.474 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:21:44.474 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:21:44.474 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:21:44.474 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:44.474 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:44.474 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:21:44.474 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:44.474 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:21:44.474 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:21:44.474 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:21:44.474 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:44.474 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:44.474 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:21:44.474 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:44.475 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:21:44.475 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:21:44.475 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:21:44.475 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:44.475 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:44.475 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:44.475 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:21:44.475 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:21:44.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.395+0000 7fa858c44640 1 -- 172.21.15.37:0/3209026724 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa834035e80 msgr2=0x7fa834038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:44.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.395+0000 7fa858c44640 1 --2- 172.21.15.37:0/3209026724 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa834035e80 0x7fa834038340 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa854120cf0 tx=0x7fa84c00b040 comp rx=0 tx=0).stop 2024-04-26T00:21:44.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.395+0000 7fa858c44640 10 monclient: shutdown 2024-04-26T00:21:44.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.395+0000 7fa858c44640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:44.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.395+0000 7fa858c44640 1 -- 172.21.15.37:0/3209026724 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8540a4170 msgr2=0x7fa854136ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:44.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.395+0000 7fa858c44640 1 --2- 172.21.15.37:0/3209026724 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8540a4170 0x7fa854136ec0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa83c01a9d0 tx=0x7fa83c0190c0 comp rx=0 tx=0).stop 2024-04-26T00:21:44.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.395+0000 7fa852d9d640 1 -- 172.21.15.37:0/3209026724 reap_dead start 2024-04-26T00:21:44.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.395+0000 7fa858c44640 1 -- 172.21.15.37:0/3209026724 shutdown_connections 2024-04-26T00:21:44.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.395+0000 7fa858c44640 1 --2- 172.21.15.37:0/3209026724 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8540a4170 0x7fa854136ec0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:44.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.395+0000 7fa858c44640 1 -- 172.21.15.37:0/3209026724 >> 172.21.15.37:0/3209026724 conn(0x7fa8540aa0a0 msgr2=0x7fa8540a8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:44.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.395+0000 7fa858c44640 1 -- 172.21.15.37:0/3209026724 shutdown_connections 2024-04-26T00:21:44.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.395+0000 7fa858c44640 1 -- 172.21.15.37:0/3209026724 wait complete. 2024-04-26T00:21:44.478 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:21:44.478 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:21:44.478 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:21:44.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f4825eb0640 10 monclient: get_monmap_and_config 2024-04-26T00:21:44.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f4825eb0640 10 monclient: build_initial_monmap 2024-04-26T00:21:44.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f4825eb0640 10 monclient: monmap: 2024-04-26T00:21:44.612 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:44.612 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:44.612 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:21:44.636795+0000 2024-04-26T00:21:44.612 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:21:44.636795+0000 2024-04-26T00:21:44.612 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:44.613 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:44.613 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:21:44.613 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:21:44.613 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:21:44.613 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:21:44.613 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:21:44.613 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:44.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f4825eb0640 1 Processor -- start 2024-04-26T00:21:44.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f4825eb0640 1 -- start start 2024-04-26T00:21:44.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f4825eb0640 10 monclient: init 2024-04-26T00:21:44.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f4825eb0640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:44.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f4825eb0640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-04-26T00:21:44.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f4825eb0640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f482009e220 0x7f482009fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:44.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f4825eb0640 10 monclient(hunting): picked mon.noname-c con 0x7f482009e220 addr v2:172.21.15.45:3300/0 2024-04-26T00:21:44.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f4825eb0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f48200a14f0 addr v1:172.21.15.37:6789/0 2024-04-26T00:21:44.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f4825eb0640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f482009e600 0x7f482013aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:44.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f4825eb0640 10 monclient(hunting): picked mon.noname-b con 0x7f482009e600 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:21:44.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f4825eb0640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48200a12b0 con 0x7f48200a14f0 2024-04-26T00:21:44.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f4825eb0640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:44.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f4825eb0640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f482009cd10 con 0x7f482009e220 2024-04-26T00:21:44.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f4825eb0640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:44.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f4825eb0640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f482009da20 con 0x7f482009e600 2024-04-26T00:21:44.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f4825eb0640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:44.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f4825eb0640 10 monclient(hunting): authenticate will time out at 1085.943237s 2024-04-26T00:21:44.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f481ffff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f482009e600 0x7f482013aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:44.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f481ffff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f482009e600 0x7f482013aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:56360/0 (socket says 172.21.15.37:56360) 2024-04-26T00:21:44.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f481effd640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f48200a14f0 0x7f48200a0190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:54106/0 (socket says 172.21.15.37:54106) 2024-04-26T00:21:44.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f481ffff640 1 -- 172.21.15.37:0/3056757562 learned_addr learned my addr 172.21.15.37:0/3056757562 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:21:44.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f481ffff640 10 monclient(hunting): get_auth_request con 0x7f482009e600 auth_method 0 2024-04-26T00:21:44.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f481ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:44.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f481ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:44.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f481ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:44.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f481ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:44.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f481ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:44.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f481ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:44.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f481f7fe640 1 --2- 172.21.15.37:0/3056757562 >> v2:172.21.15.45:3300/0 conn(0x7f482009e220 0x7f482009fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:44.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f481f7fe640 10 monclient(hunting): get_auth_request con 0x7f482009e220 auth_method 0 2024-04-26T00:21:44.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f481f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:44.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f481f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:44.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f481f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:44.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f481ffff640 10 monclient(hunting): handle_auth_done global_id 4559 payload 274 2024-04-26T00:21:44.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f481ffff640 1 -- 172.21.15.37:0/3056757562 >> v2:172.21.15.45:3300/0 conn(0x7f482009e220 msgr2=0x7f482009fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:21:44.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f481ffff640 1 --2- 172.21.15.37:0/3056757562 >> v2:172.21.15.45:3300/0 conn(0x7f482009e220 0x7f482009fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:44.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f481ffff640 1 -- 172.21.15.37:0/3056757562 >> v1:172.21.15.37:6789/0 conn(0x7f48200a14f0 legacy=0x7f48200a0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:21:44.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f481ffff640 10 monclient: _finish_hunting 0 2024-04-26T00:21:44.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f481ffff640 1 monclient: found mon.noname-b 2024-04-26T00:21:44.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f481ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:44.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f481ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:21:44.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.635+0000 7f481ffff640 1 -- 172.21.15.37:0/3056757562 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f482009dd00 con 0x7f482009e600 2024-04-26T00:21:44.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f481ffff640 1 -- 172.21.15.37:0/3056757562 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f482009e600 msgr2=0x7f482013aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:44.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f481ffff640 1 --2- 172.21.15.37:0/3056757562 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f482009e600 0x7f482013aa60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f481800af60 tx=0x7f481801d9b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=865be373dbf62f3f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:44.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f481e7fc640 1 -- 172.21.15.37:0/3056757562 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f481800a2d0 con 0x7f482009e600 2024-04-26T00:21:44.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f481e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:44.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f481e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:21:44.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f481e7fc640 10 monclient: dump: 2024-04-26T00:21:44.620 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:44.620 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:44.620 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:44.620 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:44.620 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:44.620 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:44.621 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:44.621 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:44.621 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:44.621 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:44.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f481e7fc640 10 monclient: _finish_auth 0 2024-04-26T00:21:44.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f481e7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:21:44.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f481e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:44.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f481e7fc640 1 -- 172.21.15.37:0/3056757562 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48200a0ab0 con 0x7f482009e600 2024-04-26T00:21:44.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f481e7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:44.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f481e7fc640 1 -- 172.21.15.37:0/3056757562 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48180a6900 con 0x7f482009e600 2024-04-26T00:21:44.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f4825eb0640 5 monclient: authenticate success, global_id 4559 2024-04-26T00:21:44.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f4825eb0640 10 monclient: get_monmap_and_config success 2024-04-26T00:21:44.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f4825eb0640 4 set_mon_vals no callback set 2024-04-26T00:21:44.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f481e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:44.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f481e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:21:44.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f481e7fc640 10 monclient: dump: 2024-04-26T00:21:44.622 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:44.622 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:44.622 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:44.622 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:44.622 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:44.622 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:44.622 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:44.622 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:44.622 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:44.622 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:44.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f4825eb0640 10 monclient: shutdown 2024-04-26T00:21:44.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f4825eb0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:44.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f4825eb0640 1 -- 172.21.15.37:0/3056757562 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f482009e600 msgr2=0x7f482013aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:44.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f4825eb0640 1 --2- 172.21.15.37:0/3056757562 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f482009e600 0x7f482013aa60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f481800af60 tx=0x7f481801d9b0 comp rx=0 tx=0).stop 2024-04-26T00:21:44.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f4825eb0640 1 -- 172.21.15.37:0/3056757562 shutdown_connections 2024-04-26T00:21:44.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f4825eb0640 1 --2- 172.21.15.37:0/3056757562 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f482009e600 0x7f482013aa60 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:44.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f4825eb0640 1 --2- 172.21.15.37:0/3056757562 >> v2:172.21.15.45:3300/0 conn(0x7f482009e220 0x7f482009fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:44.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f4825eb0640 1 -- 172.21.15.37:0/3056757562 >> 172.21.15.37:0/3056757562 conn(0x7f48200a1d50 msgr2=0x7f48200a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:44.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f4825eb0640 1 -- 172.21.15.37:0/3056757562 shutdown_connections 2024-04-26T00:21:44.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f4825eb0640 1 -- 172.21.15.37:0/3056757562 wait complete. 2024-04-26T00:21:44.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f4825eb0640 10 monclient: build_initial_monmap 2024-04-26T00:21:44.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f4825eb0640 10 monclient: monmap: 2024-04-26T00:21:44.629 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:44.629 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:21:44.629 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:21:44.629 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:21:44.629 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:44.629 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:21:44.629 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:21:44.629 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:21:44.629 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:21:44.629 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:44.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f4825eb0640 1 Processor -- start 2024-04-26T00:21:44.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f4825eb0640 1 -- start start 2024-04-26T00:21:44.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f4825eb0640 10 monclient: init 2024-04-26T00:21:44.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f4825eb0640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:44.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f4825eb0640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:21:44.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f4825eb0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f482009e220 0x7f48201cefc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:44.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f4825eb0640 10 monclient(hunting): picked mon.noname-b con 0x7f482009e220 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:21:44.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f4825eb0640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f482009e600 0x7f4820136f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:44.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f4825eb0640 10 monclient(hunting): picked mon.noname-c con 0x7f482009e600 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:21:44.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f4825eb0640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f48200a14f0 0x7f4820137480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:44.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f4825eb0640 10 monclient(hunting): picked mon.noname-a con 0x7f48200a14f0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:21:44.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f4825eb0640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:44.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f4825eb0640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f482013d430 con 0x7f48200a14f0 2024-04-26T00:21:44.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f4825eb0640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:44.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f4825eb0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f482013d2b0 con 0x7f482009e220 2024-04-26T00:21:44.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f4825eb0640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:44.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f4825eb0640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f482013d5b0 con 0x7f482009e600 2024-04-26T00:21:44.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f4825eb0640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:44.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f4825eb0640 10 monclient(hunting): authenticate will time out at 1085.951664s 2024-04-26T00:21:44.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f481ffff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f48200a14f0 0x7f4820137480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:44.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f481ffff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f48200a14f0 0x7f4820137480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58572/0 (socket says 172.21.15.37:58572) 2024-04-26T00:21:44.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f481ffff640 1 -- 172.21.15.37:0/3140204533 learned_addr learned my addr 172.21.15.37:0/3140204533 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:21:44.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f481ffff640 10 monclient(hunting): get_auth_request con 0x7f48200a14f0 auth_method 0 2024-04-26T00:21:44.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f481ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:44.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f481ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:44.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f481ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:44.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f481effd640 1 --2- 172.21.15.37:0/3140204533 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f482009e600 0x7f4820136f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:44.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f481ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:44.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f481ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:44.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f481ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:44.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f481effd640 10 monclient(hunting): get_auth_request con 0x7f482009e600 auth_method 0 2024-04-26T00:21:44.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f481effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:44.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f481effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:44.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.643+0000 7f481effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:44.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f481f7fe640 1 --2- 172.21.15.37:0/3140204533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f482009e220 0x7f48201cefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:44.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f481f7fe640 10 monclient(hunting): get_auth_request con 0x7f482009e220 auth_method 0 2024-04-26T00:21:44.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f481f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:44.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f481f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:44.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f481f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:44.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f481effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:44.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f481effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:44.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f481effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:44.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f481ffff640 10 monclient(hunting): handle_auth_done global_id 4572 payload 824 2024-04-26T00:21:44.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f481ffff640 1 -- 172.21.15.37:0/3140204533 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f482009e600 msgr2=0x7f4820136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:44.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f481ffff640 1 --2- 172.21.15.37:0/3140204533 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f482009e600 0x7f4820136f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:44.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f481ffff640 1 -- 172.21.15.37:0/3140204533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f482009e220 msgr2=0x7f48201cefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:44.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f481ffff640 1 --2- 172.21.15.37:0/3140204533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f482009e220 0x7f48201cefc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:44.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f481ffff640 10 monclient: _finish_hunting 0 2024-04-26T00:21:44.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f481ffff640 1 monclient: found mon.noname-a 2024-04-26T00:21:44.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f481ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:44.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f481ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:21:44.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f481ffff640 1 -- 172.21.15.37:0/3140204533 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4820138a70 con 0x7f48200a14f0 2024-04-26T00:21:44.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f481f7fe640 1 --2- 172.21.15.37:0/3140204533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f482009e220 0x7f48201cefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:21:44.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f481ffff640 1 -- 172.21.15.37:0/3140204533 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f48200a14f0 msgr2=0x7f4820137480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:44.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f481ffff640 1 --2- 172.21.15.37:0/3140204533 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f48200a14f0 0x7f4820137480 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f481800af60 tx=0x7f481801a750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:44.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f4807fff640 1 -- 172.21.15.37:0/3140204533 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4818008500 con 0x7f48200a14f0 2024-04-26T00:21:44.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f4807fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:44.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f4807fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:21:44.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f4807fff640 10 monclient: dump: 2024-04-26T00:21:44.634 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:44.634 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:44.634 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:44.634 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:44.634 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:44.634 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:44.634 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:44.634 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:44.634 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:44.634 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:44.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f4807fff640 10 monclient: _finish_auth 0 2024-04-26T00:21:44.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f4807fff640 10 monclient: _check_auth_tickets 2024-04-26T00:21:44.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f4807fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:44.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f4807fff640 1 -- 172.21.15.37:0/3140204533 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48180bd070 con 0x7f48200a14f0 2024-04-26T00:21:44.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f4825eb0640 5 monclient: authenticate success, global_id 4572 2024-04-26T00:21:44.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f4825eb0640 10 monclient: _renew_subs 2024-04-26T00:21:44.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f4825eb0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:44.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f4825eb0640 1 -- 172.21.15.37:0/3140204533 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f482009c970 con 0x7f48200a14f0 2024-04-26T00:21:44.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f4807fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:44.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f4807fff640 1 -- 172.21.15.37:0/3140204533 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4818008900 con 0x7f48200a14f0 2024-04-26T00:21:44.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f4807fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:44.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f4807fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:21:44.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f4807fff640 10 monclient: dump: 2024-04-26T00:21:44.635 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:44.636 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:44.636 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:44.636 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:44.636 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:44.636 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:44.636 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:44.636 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:44.636 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:44.636 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:44.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f4825eb0640 10 monclient: _renew_subs 2024-04-26T00:21:44.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f4825eb0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:44.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f4825eb0640 1 -- 172.21.15.37:0/3140204533 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4820123630 con 0x7f48200a14f0 2024-04-26T00:21:44.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f481dffb640 4 set_mon_vals no callback set 2024-04-26T00:21:44.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f4825eb0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:44.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f4825eb0640 1 --2- 172.21.15.37:0/3140204533 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48201cc280 0x7f482002c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:44.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f4825eb0640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:44.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f4825eb0640 1 -- 172.21.15.37:0/3140204533 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4820121fb0 con 0x7f48201cc280 2024-04-26T00:21:44.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f481f7fe640 1 --2- 172.21.15.37:0/3140204533 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48201cc280 0x7f482002c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:44.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f481f7fe640 10 monclient: get_auth_request con 0x7f48201cc280 auth_method 0 2024-04-26T00:21:44.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f481f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:44.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f481f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:44.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f481f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:44.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f481f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:44.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f481f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:44.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f481f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:44.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f4807fff640 1 -- 172.21.15.37:0/3140204533 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f4818008aa0 con 0x7f48200a14f0 2024-04-26T00:21:44.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f481f7fe640 10 monclient(hunting): handle_auth_done global_id 4565 payload 824 2024-04-26T00:21:44.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.647+0000 7f481f7fe640 1 -- 172.21.15.37:0/3140204533 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f48200a0ab0 con 0x7f48201cc280 2024-04-26T00:21:44.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.651+0000 7f4807fff640 1 --2- 172.21.15.37:0/3140204533 >> v2:172.21.15.45:6812/2379918623 conn(0x7f480c035eb0 0x7f480c038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:44.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.651+0000 7f4807fff640 1 -- 172.21.15.37:0/3140204533 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 3943+0+0 (secure 0 0 0) 0x7f48180e3b40 con 0x7f48200a14f0 2024-04-26T00:21:44.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.651+0000 7f481effd640 1 --2- 172.21.15.37:0/3140204533 >> v2:172.21.15.45:6812/2379918623 conn(0x7f480c035eb0 0x7f480c038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:44.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.651+0000 7f481effd640 10 monclient: get_auth_request con 0x7f480c035eb0 auth_method 0 2024-04-26T00:21:44.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.651+0000 7f481f7fe640 1 -- 172.21.15.37:0/3140204533 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48201cc280 msgr2=0x7f482002c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:44.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.651+0000 7f481f7fe640 1 --2- 172.21.15.37:0/3140204533 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48201cc280 0x7f482002c000 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f481000f420 tx=0x7f481000f8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:44.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.651+0000 7f4807fff640 1 -- 172.21.15.37:0/3140204533 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4810018020 con 0x7f48201cc280 2024-04-26T00:21:44.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.651+0000 7f4807fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:44.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.651+0000 7f4807fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:21:44.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.651+0000 7f4807fff640 10 monclient: dump: 2024-04-26T00:21:44.640 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:44.640 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:44.640 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:44.640 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:44.640 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:44.641 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:44.641 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:44.641 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:44.641 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:44.641 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:44.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.651+0000 7f4807fff640 1 -- 172.21.15.37:0/3140204533 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4810014a80 con 0x7f48201cc280 2024-04-26T00:21:44.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.651+0000 7f4807fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:44.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.651+0000 7f4807fff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:21:44.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.651+0000 7f4807fff640 1 -- 172.21.15.37:0/3140204533 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48201cc280 msgr2=0x7f482002c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:44.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.651+0000 7f4807fff640 1 --2- 172.21.15.37:0/3140204533 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48201cc280 0x7f482002c000 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f481000f420 tx=0x7f481000f8e0 comp rx=0 tx=0).stop 2024-04-26T00:21:44.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.651+0000 7f4807fff640 1 -- 172.21.15.37:0/3140204533 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48201cc280 msgr2=0x7f482002c000 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:21:44.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.651+0000 7f4807fff640 1 --2- 172.21.15.37:0/3140204533 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48201cc280 0x7f482002c000 secure :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f481000f420 tx=0x7f481000f8e0 comp rx=0 tx=0).stop 2024-04-26T00:21:44.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.651+0000 7f481effd640 1 --2- 172.21.15.37:0/3140204533 >> v2:172.21.15.45:6812/2379918623 conn(0x7f480c035eb0 0x7f480c038370 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f48201cef20 tx=0x7f4814007e70 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:44.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.659+0000 7f4825eb0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:21:44.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.659+0000 7f4825eb0640 1 --2- 172.21.15.37:0/3140204533 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48201cc660 0x7f482002c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:44.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.659+0000 7f4825eb0640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:44.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.659+0000 7f4825eb0640 1 -- 172.21.15.37:0/3140204533 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f482004f510 con 0x7f48201cc660 2024-04-26T00:21:44.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.659+0000 7f481f7fe640 1 --2- 172.21.15.37:0/3140204533 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48201cc660 0x7f482002c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:44.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.659+0000 7f481f7fe640 10 monclient: get_auth_request con 0x7f48201cc660 auth_method 0 2024-04-26T00:21:44.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.659+0000 7f481f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:44.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.659+0000 7f481f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:44.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.659+0000 7f481f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:44.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.659+0000 7f481f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:44.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.659+0000 7f481f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:44.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.659+0000 7f481f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:44.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.659+0000 7f481f7fe640 10 monclient(hunting): handle_auth_done global_id 4568 payload 824 2024-04-26T00:21:44.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.659+0000 7f481f7fe640 1 -- 172.21.15.37:0/3140204533 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f482011cfb0 con 0x7f48201cc660 2024-04-26T00:21:44.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.659+0000 7f481f7fe640 1 -- 172.21.15.37:0/3140204533 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48201cc660 msgr2=0x7f482002c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:44.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.659+0000 7f481f7fe640 1 --2- 172.21.15.37:0/3140204533 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48201cc660 0x7f482002c540 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4810019000 tx=0x7f48100046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:44.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.659+0000 7f4807fff640 1 -- 172.21.15.37:0/3140204533 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4810018070 con 0x7f48201cc660 2024-04-26T00:21:44.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.659+0000 7f4807fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:44.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.659+0000 7f4807fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:21:44.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.659+0000 7f4807fff640 10 monclient: dump: 2024-04-26T00:21:44.643 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:44.643 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:44.643 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:44.643 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:44.643 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:44.643 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:44.643 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:44.643 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:44.643 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:44.644 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:44.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.663+0000 7f4807fff640 1 -- 172.21.15.37:0/3140204533 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4717 (secure 0 0 0) 0x7f48100133c0 con 0x7f48201cc660 2024-04-26T00:21:44.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.663+0000 7f4807fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:21:44.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.663+0000 7f4807fff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:21:44.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.663+0000 7f4807fff640 1 -- 172.21.15.37:0/3140204533 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48201cc660 msgr2=0x7f482002c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:44.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.663+0000 7f4807fff640 1 --2- 172.21.15.37:0/3140204533 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48201cc660 0x7f482002c540 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4810019000 tx=0x7f48100046d0 comp rx=0 tx=0).stop 2024-04-26T00:21:44.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.663+0000 7f4807fff640 1 -- 172.21.15.37:0/3140204533 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48201cc660 msgr2=0x7f482002c540 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:21:44.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.663+0000 7f4807fff640 1 --2- 172.21.15.37:0/3140204533 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48201cc660 0x7f482002c540 secure :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4810019000 tx=0x7f48100046d0 comp rx=0 tx=0).stop 2024-04-26T00:21:44.644 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:21:44.644 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:21:44.645 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:21:44.645 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:21:44.645 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 18, 2024-04-26T00:21:44.645 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 121766, 2024-04-26T00:21:44.645 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:21:44.645 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:21:44.645 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:21:44.645 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:21:44.645 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:44.645 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 7, 2024-04-26T00:21:44.645 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:21:44.645 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:21:44.645 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:21:44.645 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:21:44.646 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:21:44.646 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:21:44.646 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:21:44.646 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:21:44.646 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:21:44.646 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:21:44.646 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:21:44.646 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:21:44.649 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:21:44.649 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:21:44.650 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:44.650 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:21:44.650 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:21:44.650 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:21:44.650 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:21:44.650 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:21:44.650 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:21:44.650 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:21:44.650 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:21:44.650 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:21:44.650 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:21:44.650 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:21:44.650 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:21:44.651 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:21:44.651 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:44.651 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:44.651 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:21:44.651 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:21:44.651 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:21:44.651 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:21:44.651 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:21:44.651 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:21:44.651 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:21:44.651 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:21:44.651 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:21:44.651 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:21:44.651 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:21:44.652 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:21:44.652 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:21:44.652 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:21:44.652 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:21:44.652 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:21:44.652 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:21:44.652 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:21:44.652 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:21:44.652 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:21:44.652 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:21:44.652 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:21:44.652 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:21:44.652 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:21:44.652 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:21:44.653 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:21:44.653 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:21:44.653 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:21:44.653 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:44.653 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:21:44.653 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:44.653 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:21:44.653 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:44.653 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:21:44.653 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:21:44.653 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:21:44.653 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:21:44.653 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:44.653 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:21:44.653 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:21:44.654 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:44.654 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:44.654 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:44.654 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:21:44.654 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:21:44.654 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:44.654 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:44.654 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:44.654 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:44.654 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:21:44.654 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:21:44.654 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:21:44.654 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:21:44.655 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:21:44.655 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:44.655 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:44.655 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:21:44.655 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:21:44.655 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:21:44.655 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:21:44.655 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:44.655 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:21:44.655 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:21:44.655 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:44.655 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:44.655 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:44.655 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:21:44.655 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:21:44.657 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:44.657 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:44.657 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:44.657 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:44.657 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:21:44.657 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:21:44.657 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:21:44.657 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:21:44.657 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:21:44.657 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:44.657 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:44.657 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:21:44.657 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:21:44.658 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:21:44.658 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:21:44.658 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:44.658 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:21:44.658 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:21:44.658 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:44.658 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:44.658 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:44.658 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:21:44.658 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:21:44.658 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:44.658 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:44.658 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:44.659 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:44.659 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:21:44.659 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:21:44.659 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:21:44.659 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:21:44.659 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:21:44.659 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:44.659 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:44.659 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:44.659 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:21:44.659 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:21:44.659 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:44.659 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:21:44.659 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:21:44.660 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:21:44.660 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:44.660 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:44.660 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:21:44.660 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:44.660 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:21:44.660 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:21:44.660 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:21:44.660 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:44.660 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:44.660 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:21:44.660 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:44.660 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:21:44.660 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:21:44.660 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:21:44.661 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:44.661 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:44.661 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:44.661 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:21:44.661 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:21:44.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.663+0000 7f4825eb0640 1 -- 172.21.15.37:0/3140204533 >> v2:172.21.15.45:6812/2379918623 conn(0x7f480c035eb0 msgr2=0x7f480c038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:44.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.663+0000 7f4825eb0640 1 --2- 172.21.15.37:0/3140204533 >> v2:172.21.15.45:6812/2379918623 conn(0x7f480c035eb0 0x7f480c038370 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f48201cef20 tx=0x7f4814007e70 comp rx=0 tx=0).stop 2024-04-26T00:21:44.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.663+0000 7f4825eb0640 10 monclient: shutdown 2024-04-26T00:21:44.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.663+0000 7f4825eb0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:44.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.663+0000 7f4825eb0640 1 -- 172.21.15.37:0/3140204533 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f48200a14f0 msgr2=0x7f4820137480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:44.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.663+0000 7f4825eb0640 1 --2- 172.21.15.37:0/3140204533 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f48200a14f0 0x7f4820137480 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f481800af60 tx=0x7f481801a750 comp rx=0 tx=0).stop 2024-04-26T00:21:44.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.663+0000 7f481ffff640 1 -- 172.21.15.37:0/3140204533 reap_dead start 2024-04-26T00:21:44.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.663+0000 7f481ffff640 1 -- 172.21.15.37:0/3140204533 reap_dead start 2024-04-26T00:21:44.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.663+0000 7f4825eb0640 1 -- 172.21.15.37:0/3140204533 shutdown_connections 2024-04-26T00:21:44.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.663+0000 7f4825eb0640 1 -- 172.21.15.37:0/3140204533 >> 172.21.15.37:0/3140204533 conn(0x7f48200a1d50 msgr2=0x7f4820137c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:44.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.663+0000 7f4825eb0640 1 -- 172.21.15.37:0/3140204533 shutdown_connections 2024-04-26T00:21:44.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.663+0000 7f4825eb0640 1 -- 172.21.15.37:0/3140204533 wait complete. 2024-04-26T00:21:44.663 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:21:44.663 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:21:44.664 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:21:44.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.803+0000 7fd75c39c640 10 monclient: get_monmap_and_config 2024-04-26T00:21:44.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.803+0000 7fd75c39c640 10 monclient: build_initial_monmap 2024-04-26T00:21:44.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.803+0000 7fd75c39c640 10 monclient: monmap: 2024-04-26T00:21:44.780 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:44.780 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:44.781 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:21:44.804499+0000 2024-04-26T00:21:44.781 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:21:44.804499+0000 2024-04-26T00:21:44.781 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:44.781 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:44.782 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:21:44.782 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:21:44.782 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:21:44.782 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:21:44.782 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:21:44.782 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:44.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.803+0000 7fd75c39c640 1 Processor -- start 2024-04-26T00:21:44.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.803+0000 7fd75c39c640 1 -- start start 2024-04-26T00:21:44.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.803+0000 7fd75c39c640 10 monclient: init 2024-04-26T00:21:44.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.803+0000 7fd75c39c640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:44.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.803+0000 7fd75c39c640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-04-26T00:21:44.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.803+0000 7fd75c39c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd754130eb0 addr v1:172.21.15.37:6789/0 2024-04-26T00:21:44.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.803+0000 7fd75c39c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd75409ef20 addr v1:172.21.15.45:6789/0 2024-04-26T00:21:44.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.803+0000 7fd75c39c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd75413a5d0 0x7fd75413c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:44.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.803+0000 7fd75c39c640 10 monclient(hunting): picked mon.noname-b con 0x7fd75413a5d0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:21:44.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.803+0000 7fd75c39c640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd75409c190 con 0x7fd754130eb0 2024-04-26T00:21:44.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.803+0000 7fd75c39c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7540a11c0 con 0x7fd75409ef20 2024-04-26T00:21:44.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.803+0000 7fd75c39c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:44.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.803+0000 7fd75c39c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd75409df60 con 0x7fd75413a5d0 2024-04-26T00:21:44.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.803+0000 7fd75c39c640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:44.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.803+0000 7fd75c39c640 10 monclient(hunting): authenticate will time out at 1086.110274s 2024-04-26T00:21:44.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.803+0000 7fd75a939640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd75413a5d0 0x7fd75413c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:44.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.803+0000 7fd75a138640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fd754130eb0 0x7fd754131290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:54114/0 (socket says 172.21.15.37:54114) 2024-04-26T00:21:44.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.803+0000 7fd75a138640 1 -- 172.21.15.37:0/2984281592 learned_addr learned my addr 172.21.15.37:0/2984281592 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:21:44.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.803+0000 7fd75a939640 10 monclient(hunting): get_auth_request con 0x7fd75413a5d0 auth_method 0 2024-04-26T00:21:44.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.803+0000 7fd75a939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:44.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.803+0000 7fd75a939640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:44.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.803+0000 7fd75a939640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:44.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.803+0000 7fd75a939640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:44.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.803+0000 7fd75a939640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:44.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.803+0000 7fd75a939640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:44.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.803+0000 7fd75a939640 10 monclient(hunting): handle_auth_done global_id 4571 payload 274 2024-04-26T00:21:44.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.803+0000 7fd75a939640 1 -- 172.21.15.37:0/2984281592 >> v1:172.21.15.45:6789/0 conn(0x7fd75409ef20 legacy=0x7fd754138eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:21:44.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.803+0000 7fd75a939640 1 -- 172.21.15.37:0/2984281592 >> v1:172.21.15.37:6789/0 conn(0x7fd754130eb0 legacy=0x7fd754131290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:21:44.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.803+0000 7fd75a939640 10 monclient: _finish_hunting 0 2024-04-26T00:21:44.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.803+0000 7fd75a939640 1 monclient: found mon.noname-b 2024-04-26T00:21:44.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.803+0000 7fd75a939640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:44.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.803+0000 7fd75a939640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:21:44.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.803+0000 7fd75a939640 1 -- 172.21.15.37:0/2984281592 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd75409dc80 con 0x7fd75413a5d0 2024-04-26T00:21:44.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.807+0000 7fd75a939640 1 -- 172.21.15.37:0/2984281592 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd75413a5d0 msgr2=0x7fd75413c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:44.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.807+0000 7fd75a939640 1 --2- 172.21.15.37:0/2984281592 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd75413a5d0 0x7fd75413c9c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd74c009dc0 tx=0x7fd74c01c7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef2f0798b47f030f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:44.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.807+0000 7fd759136640 1 -- 172.21.15.37:0/2984281592 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd74c0b2070 con 0x7fd75413a5d0 2024-04-26T00:21:44.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.807+0000 7fd759136640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:44.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.807+0000 7fd759136640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:21:44.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.807+0000 7fd759136640 10 monclient: dump: 2024-04-26T00:21:44.790 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:44.790 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:44.790 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:44.790 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:44.790 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:44.790 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:44.790 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:44.790 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:44.790 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:44.791 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:44.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.807+0000 7fd759136640 10 monclient: _finish_auth 0 2024-04-26T00:21:44.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.807+0000 7fd759136640 10 monclient: _check_auth_tickets 2024-04-26T00:21:44.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.807+0000 7fd759136640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:44.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.807+0000 7fd759136640 1 -- 172.21.15.37:0/2984281592 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd74c0a5920 con 0x7fd75413a5d0 2024-04-26T00:21:44.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.807+0000 7fd759136640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:44.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.807+0000 7fd759136640 1 -- 172.21.15.37:0/2984281592 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd74c0a5cf0 con 0x7fd75413a5d0 2024-04-26T00:21:44.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.807+0000 7fd75c39c640 5 monclient: authenticate success, global_id 4571 2024-04-26T00:21:44.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.807+0000 7fd75c39c640 10 monclient: get_monmap_and_config success 2024-04-26T00:21:44.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.807+0000 7fd75c39c640 4 set_mon_vals no callback set 2024-04-26T00:21:44.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.807+0000 7fd759136640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:44.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.807+0000 7fd759136640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:21:44.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.807+0000 7fd759136640 10 monclient: dump: 2024-04-26T00:21:44.791 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:44.792 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:44.792 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:44.792 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:44.792 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:44.792 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:44.792 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:44.792 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:44.792 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:44.792 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:44.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.807+0000 7fd75c39c640 10 monclient: shutdown 2024-04-26T00:21:44.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.807+0000 7fd75c39c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:44.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.807+0000 7fd75c39c640 1 -- 172.21.15.37:0/2984281592 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd75413a5d0 msgr2=0x7fd75413c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:44.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.807+0000 7fd75c39c640 1 --2- 172.21.15.37:0/2984281592 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd75413a5d0 0x7fd75413c9c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd74c009dc0 tx=0x7fd74c01c7a0 comp rx=0 tx=0).stop 2024-04-26T00:21:44.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.807+0000 7fd75c39c640 1 -- 172.21.15.37:0/2984281592 shutdown_connections 2024-04-26T00:21:44.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.807+0000 7fd75c39c640 1 --2- 172.21.15.37:0/2984281592 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd75413a5d0 0x7fd75413c9c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:44.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.807+0000 7fd75c39c640 1 -- 172.21.15.37:0/2984281592 >> 172.21.15.37:0/2984281592 conn(0x7fd7540a1d50 msgr2=0x7fd7540a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:44.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.807+0000 7fd75c39c640 1 -- 172.21.15.37:0/2984281592 shutdown_connections 2024-04-26T00:21:44.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.807+0000 7fd75c39c640 1 -- 172.21.15.37:0/2984281592 wait complete. 2024-04-26T00:21:44.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd75c39c640 10 monclient: build_initial_monmap 2024-04-26T00:21:44.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd75c39c640 10 monclient: monmap: 2024-04-26T00:21:44.793 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:44.793 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:21:44.793 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:21:44.793 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:21:44.793 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:44.793 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:21:44.793 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:21:44.794 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:21:44.794 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:21:44.794 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:44.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd75c39c640 1 Processor -- start 2024-04-26T00:21:44.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd75c39c640 1 -- start start 2024-04-26T00:21:44.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd75c39c640 10 monclient: init 2024-04-26T00:21:44.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd75c39c640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:44.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd75c39c640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:21:44.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd75c39c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd75409ef20 0x7fd7541c4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:44.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd75c39c640 10 monclient(hunting): picked mon.noname-a con 0x7fd75409ef20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:21:44.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd75c39c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd754130eb0 0x7fd7541393b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:44.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd75c39c640 10 monclient(hunting): picked mon.noname-c con 0x7fd754130eb0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:21:44.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd75c39c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd75413a5d0 0x7fd7541c76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:44.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd75c39c640 10 monclient(hunting): picked mon.noname-b con 0x7fd75413a5d0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:21:44.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd75c39c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:44.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd75c39c640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd75409fe10 con 0x7fd75409ef20 2024-04-26T00:21:44.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd75c39c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:44.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd75c39c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd75409fc90 con 0x7fd75413a5d0 2024-04-26T00:21:44.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd75c39c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:44.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd75c39c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd75409ff90 con 0x7fd754130eb0 2024-04-26T00:21:44.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd75c39c640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:44.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd75c39c640 10 monclient(hunting): authenticate will time out at 1086.117476s 2024-04-26T00:21:44.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd759937640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd754130eb0 0x7fd7541393b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:44.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd75a138640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd75409ef20 0x7fd7541c4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:44.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd759937640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd754130eb0 0x7fd7541393b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:56406/0 (socket says 172.21.15.37:56406) 2024-04-26T00:21:44.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd759937640 1 -- 172.21.15.37:0/1504225679 learned_addr learned my addr 172.21.15.37:0/1504225679 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:21:44.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd759937640 10 monclient(hunting): get_auth_request con 0x7fd754130eb0 auth_method 0 2024-04-26T00:21:44.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd759937640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:44.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd759937640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:44.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd759937640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:44.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd75a138640 10 monclient(hunting): get_auth_request con 0x7fd75409ef20 auth_method 0 2024-04-26T00:21:44.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd75a138640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:44.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd75a138640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:44.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd75a138640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:44.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd759937640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:44.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd759937640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:44.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd759937640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:44.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd75a939640 1 --2- 172.21.15.37:0/1504225679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd75413a5d0 0x7fd7541c76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:44.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd75a138640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:44.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd75a138640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:44.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd75a138640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:44.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd75a939640 10 monclient(hunting): get_auth_request con 0x7fd75413a5d0 auth_method 0 2024-04-26T00:21:44.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd75a939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:44.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd75a939640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:44.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd75a939640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:44.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd759937640 10 monclient(hunting): handle_auth_done global_id 4574 payload 824 2024-04-26T00:21:44.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd759937640 1 -- 172.21.15.37:0/1504225679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd75413a5d0 msgr2=0x7fd7541c76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:44.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd759937640 1 --2- 172.21.15.37:0/1504225679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd75413a5d0 0x7fd7541c76e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:44.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd759937640 1 -- 172.21.15.37:0/1504225679 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd75409ef20 msgr2=0x7fd7541c4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:44.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd759937640 1 --2- 172.21.15.37:0/1504225679 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd75409ef20 0x7fd7541c4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:44.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd759937640 10 monclient: _finish_hunting 0 2024-04-26T00:21:44.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd759937640 1 monclient: found mon.noname-c 2024-04-26T00:21:44.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd759937640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:44.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd759937640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:21:44.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.811+0000 7fd759937640 1 -- 172.21.15.37:0/1504225679 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd75412caa0 con 0x7fd754130eb0 2024-04-26T00:21:44.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd759937640 1 -- 172.21.15.37:0/1504225679 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd754130eb0 msgr2=0x7fd7541393b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:44.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd759937640 1 --2- 172.21.15.37:0/1504225679 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd754130eb0 0x7fd7541393b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd74400c450 tx=0x7fd74400c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:44.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd74affd640 1 -- 172.21.15.37:0/1504225679 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd744012a40 con 0x7fd754130eb0 2024-04-26T00:21:44.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd74affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:44.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd74affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:21:44.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd74affd640 10 monclient: dump: 2024-04-26T00:21:44.804 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:44.804 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:44.804 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:44.805 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:44.805 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:44.805 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:44.805 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:44.805 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:44.805 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:44.805 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:44.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd74affd640 10 monclient: _finish_auth 0 2024-04-26T00:21:44.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd74affd640 10 monclient: _check_auth_tickets 2024-04-26T00:21:44.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd74affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:44.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd75c39c640 5 monclient: authenticate success, global_id 4574 2024-04-26T00:21:44.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd74affd640 1 -- 172.21.15.37:0/1504225679 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd744017070 con 0x7fd754130eb0 2024-04-26T00:21:44.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd74affd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:44.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd75c39c640 10 monclient: _renew_subs 2024-04-26T00:21:44.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd74affd640 1 -- 172.21.15.37:0/1504225679 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd744012e40 con 0x7fd754130eb0 2024-04-26T00:21:44.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd75c39c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:21:44.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd75c39c640 1 -- 172.21.15.37:0/1504225679 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7540a1470 con 0x7fd754130eb0 2024-04-26T00:21:44.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd74affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:44.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd74affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:21:44.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd74affd640 10 monclient: dump: 2024-04-26T00:21:44.806 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:44.806 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:44.806 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:44.806 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:44.806 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:44.806 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:44.807 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:44.807 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:44.807 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:44.807 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:44.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd758935640 4 set_mon_vals no callback set 2024-04-26T00:21:44.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd75c39c640 10 monclient: _renew_subs 2024-04-26T00:21:44.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd75c39c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:21:44.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd75c39c640 1 -- 172.21.15.37:0/1504225679 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd754134e70 con 0x7fd754130eb0 2024-04-26T00:21:44.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd75c39c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:44.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd75c39c640 1 --2- 172.21.15.37:0/1504225679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd75412d770 0x7fd7541c7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:44.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd75c39c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:44.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd75c39c640 1 -- 172.21.15.37:0/1504225679 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd75412e530 con 0x7fd75412d770 2024-04-26T00:21:44.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd74affd640 1 -- 172.21.15.37:0/1504225679 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd74401c460 con 0x7fd754130eb0 2024-04-26T00:21:44.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd75a138640 1 --2- 172.21.15.37:0/1504225679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd75412d770 0x7fd7541c7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:44.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd74affd640 1 --2- 172.21.15.37:0/1504225679 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd738035ed0 0x7fd738038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:44.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd75a138640 10 monclient: get_auth_request con 0x7fd75412d770 auth_method 0 2024-04-26T00:21:44.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd75a138640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:44.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd75a138640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:44.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd75a138640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:44.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd74affd640 1 -- 172.21.15.37:0/1504225679 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(80..80 src has 1..80) ==== 3943+0+0 (secure 0 0 0) 0x7fd744048860 con 0x7fd754130eb0 2024-04-26T00:21:44.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd75a138640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:44.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd75a138640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:44.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd75a138640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:44.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd75a939640 1 --2- 172.21.15.37:0/1504225679 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd738035ed0 0x7fd738038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:44.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd75a939640 10 monclient: get_auth_request con 0x7fd738035ed0 auth_method 0 2024-04-26T00:21:44.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd75a138640 10 monclient(hunting): handle_auth_done global_id 24130 payload 824 2024-04-26T00:21:44.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.815+0000 7fd75a138640 1 -- 172.21.15.37:0/1504225679 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd75404f510 con 0x7fd75412d770 2024-04-26T00:21:44.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.819+0000 7fd75a138640 1 -- 172.21.15.37:0/1504225679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd75412d770 msgr2=0x7fd7541c7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:44.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.819+0000 7fd75a138640 1 --2- 172.21.15.37:0/1504225679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd75412d770 0x7fd7541c7c20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd75000eaa0 tx=0x7fd75000eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:44.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.819+0000 7fd74affd640 1 -- 172.21.15.37:0/1504225679 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd75000ee60 con 0x7fd75412d770 2024-04-26T00:21:44.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.819+0000 7fd74affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:44.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.819+0000 7fd74affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:21:44.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.819+0000 7fd74affd640 10 monclient: dump: 2024-04-26T00:21:44.809 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:44.809 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:44.809 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:44.809 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:44.809 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:44.809 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:44.810 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:44.810 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:44.810 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:44.810 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:44.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.819+0000 7fd75a939640 1 --2- 172.21.15.37:0/1504225679 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd738035ed0 0x7fd738038390 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd74c0ac000 tx=0x7fd74c0a7070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:44.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.819+0000 7fd74affd640 1 -- 172.21.15.37:0/1504225679 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd750019070 con 0x7fd75412d770 2024-04-26T00:21:44.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.819+0000 7fd74affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:44.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.819+0000 7fd74affd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:21:44.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.819+0000 7fd74affd640 1 -- 172.21.15.37:0/1504225679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd75412d770 msgr2=0x7fd7541c7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:44.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.819+0000 7fd74affd640 1 --2- 172.21.15.37:0/1504225679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd75412d770 0x7fd7541c7c20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd75000eaa0 tx=0x7fd75000eb90 comp rx=0 tx=0).stop 2024-04-26T00:21:44.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.819+0000 7fd74affd640 1 -- 172.21.15.37:0/1504225679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd75412d770 msgr2=0x7fd7541c7c20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:21:44.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.819+0000 7fd74affd640 1 --2- 172.21.15.37:0/1504225679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd75412d770 0x7fd7541c7c20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:44.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.823+0000 7fd75c39c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:21:44.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.823+0000 7fd75c39c640 1 --2- 172.21.15.37:0/1504225679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7541c8160 0x7fd754131a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:44.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.827+0000 7fd75c39c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:44.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.827+0000 7fd75c39c640 1 -- 172.21.15.37:0/1504225679 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd7541c8540 con 0x7fd7541c8160 2024-04-26T00:21:44.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.827+0000 7fd75a138640 1 --2- 172.21.15.37:0/1504225679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7541c8160 0x7fd754131a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:44.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.827+0000 7fd75a138640 10 monclient: get_auth_request con 0x7fd7541c8160 auth_method 0 2024-04-26T00:21:44.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.827+0000 7fd75a138640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:44.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.827+0000 7fd75a138640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:44.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.827+0000 7fd75a138640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:44.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.827+0000 7fd75a138640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:44.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.827+0000 7fd75a138640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:44.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.827+0000 7fd75a138640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:44.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.827+0000 7fd75a138640 10 monclient(hunting): handle_auth_done global_id 24133 payload 824 2024-04-26T00:21:44.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.827+0000 7fd75a138640 1 -- 172.21.15.37:0/1504225679 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd75412ec80 con 0x7fd7541c8160 2024-04-26T00:21:44.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.827+0000 7fd75a138640 1 -- 172.21.15.37:0/1504225679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7541c8160 msgr2=0x7fd754131a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:44.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.827+0000 7fd75a138640 1 --2- 172.21.15.37:0/1504225679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7541c8160 0x7fd754131a00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd7500070c0 tx=0x7fd750004950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:44.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.827+0000 7fd74affd640 1 -- 172.21.15.37:0/1504225679 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd75001ed10 con 0x7fd7541c8160 2024-04-26T00:21:44.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.827+0000 7fd74affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:44.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.827+0000 7fd74affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:21:44.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.827+0000 7fd74affd640 10 monclient: dump: 2024-04-26T00:21:44.814 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:44.814 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:44.814 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:44.814 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:44.814 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:44.814 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:44.814 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:44.814 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:44.814 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:44.815 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:44.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.831+0000 7fd74affd640 1 -- 172.21.15.37:0/1504225679 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7fd750014480 con 0x7fd7541c8160 2024-04-26T00:21:44.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.831+0000 7fd74affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:21:44.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.831+0000 7fd74affd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:21:44.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.831+0000 7fd74affd640 1 -- 172.21.15.37:0/1504225679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7541c8160 msgr2=0x7fd754131a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:44.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.831+0000 7fd74affd640 1 --2- 172.21.15.37:0/1504225679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7541c8160 0x7fd754131a00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd7500070c0 tx=0x7fd750004950 comp rx=0 tx=0).stop 2024-04-26T00:21:44.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.831+0000 7fd74affd640 1 -- 172.21.15.37:0/1504225679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7541c8160 msgr2=0x7fd754131a00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:21:44.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.831+0000 7fd74affd640 1 --2- 172.21.15.37:0/1504225679 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7541c8160 0x7fd754131a00 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd7500070c0 tx=0x7fd750004950 comp rx=0 tx=0).stop 2024-04-26T00:21:44.815 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:21:44.815 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:21:44.815 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:21:44.816 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:21:44.816 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 18, 2024-04-26T00:21:44.816 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 12995, 2024-04-26T00:21:44.816 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:21:44.816 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:21:44.816 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:21:44.816 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:21:44.816 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:44.816 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 7, 2024-04-26T00:21:44.816 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:21:44.816 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:21:44.816 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:21:44.816 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:21:44.816 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:21:44.816 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:21:44.817 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:21:44.817 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:21:44.817 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:21:44.817 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:21:44.817 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:21:44.817 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:21:44.817 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:21:44.817 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:21:44.817 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:44.817 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:21:44.817 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:21:44.817 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:21:44.817 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:21:44.817 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:21:44.817 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:21:44.818 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:21:44.818 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:21:44.818 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:21:44.818 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:21:44.818 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:21:44.818 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:21:44.818 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:21:44.818 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:44.818 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:44.818 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:21:44.818 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:21:44.818 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:21:44.818 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:21:44.818 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:21:44.819 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:21:44.819 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:21:44.819 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:21:44.819 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:21:44.819 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:21:44.819 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:21:44.819 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:21:44.819 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:21:44.819 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:21:44.819 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:21:44.819 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:21:44.819 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:21:44.819 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:21:44.819 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:21:44.819 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:21:44.819 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:21:44.820 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:21:44.820 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:21:44.820 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:21:44.820 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:21:44.820 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:21:44.820 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:21:44.820 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:21:44.820 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:44.820 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:21:44.820 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:44.820 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:21:44.820 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:44.820 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:21:44.820 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:21:44.820 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:21:44.821 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:21:44.821 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:44.821 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:21:44.821 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:21:44.821 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:44.821 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:44.821 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:44.821 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:21:44.821 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:21:44.821 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:44.821 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:44.821 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:44.821 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:44.821 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:21:44.822 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:21:44.822 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:21:44.822 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:21:44.822 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:21:44.822 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:44.822 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:44.822 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:21:44.822 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:21:44.822 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:21:44.822 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:21:44.822 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:44.822 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:21:44.822 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:21:44.822 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:44.822 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:44.823 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:44.823 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:21:44.823 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:21:44.823 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:44.823 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:44.823 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:44.823 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:44.823 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:21:44.823 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:21:44.823 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:21:44.823 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:21:44.823 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:21:44.823 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:44.823 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:44.823 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:21:44.823 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:21:44.824 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:21:44.824 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:21:44.824 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:44.824 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:21:44.824 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:21:44.824 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:44.824 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:44.824 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:44.824 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:21:44.824 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:21:44.824 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:44.824 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:44.824 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:44.824 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:44.824 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:21:44.824 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:21:44.824 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:21:44.825 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:21:44.825 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:21:44.825 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:44.825 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:44.825 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:44.825 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:21:44.825 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:21:44.825 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:44.825 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:21:44.825 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:21:44.825 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:21:44.825 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:44.825 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:44.825 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:21:44.825 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:44.825 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:21:44.826 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:21:44.826 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:21:44.826 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:44.826 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:44.826 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:44.826 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:21:44.826 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:21:44.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.831+0000 7fd75c39c640 1 -- 172.21.15.37:0/1504225679 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd738035ed0 msgr2=0x7fd738038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:44.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.831+0000 7fd75c39c640 1 --2- 172.21.15.37:0/1504225679 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd738035ed0 0x7fd738038390 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd74c0ac000 tx=0x7fd74c0a7070 comp rx=0 tx=0).stop 2024-04-26T00:21:44.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.831+0000 7fd75c39c640 10 monclient: shutdown 2024-04-26T00:21:44.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.831+0000 7fd75c39c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:44.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.831+0000 7fd75c39c640 1 -- 172.21.15.37:0/1504225679 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd754130eb0 msgr2=0x7fd7541393b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:44.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.831+0000 7fd75c39c640 1 --2- 172.21.15.37:0/1504225679 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd754130eb0 0x7fd7541393b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd74400c450 tx=0x7fd74400c910 comp rx=0 tx=0).stop 2024-04-26T00:21:44.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.831+0000 7fd75a939640 1 -- 172.21.15.37:0/1504225679 reap_dead start 2024-04-26T00:21:44.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.831+0000 7fd75a939640 1 -- 172.21.15.37:0/1504225679 reap_dead start 2024-04-26T00:21:44.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.831+0000 7fd75c39c640 1 -- 172.21.15.37:0/1504225679 shutdown_connections 2024-04-26T00:21:44.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.831+0000 7fd75c39c640 1 -- 172.21.15.37:0/1504225679 >> 172.21.15.37:0/1504225679 conn(0x7fd7540a1d50 msgr2=0x7fd75412f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:44.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.831+0000 7fd75c39c640 1 -- 172.21.15.37:0/1504225679 shutdown_connections 2024-04-26T00:21:44.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:44.831+0000 7fd75c39c640 1 -- 172.21.15.37:0/1504225679 wait complete. 2024-04-26T00:21:44.828 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:21:44.829 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-04-26T00:21:44.829 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-26T00:21:44.829 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-04-26T00:21:44.829 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-04-26T00:21:44.829 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-26T00:21:44.829 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T00:21:44.841 INFO:tasks.ceph.mon.c:Stopped 2024-04-26T00:21:44.841 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-26T00:21:44.841 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:21:44.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.019+0000 7f8c46e03640 10 monclient: get_monmap_and_config 2024-04-26T00:21:44.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.019+0000 7f8c46e03640 10 monclient: build_initial_monmap 2024-04-26T00:21:44.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.019+0000 7f8c46e03640 10 monclient: monmap: 2024-04-26T00:21:44.998 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:44.998 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:44.998 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:21:45.022532+0000 2024-04-26T00:21:44.998 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:21:45.022532+0000 2024-04-26T00:21:44.998 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:44.998 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:44.998 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:21:44.998 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:21:44.998 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:21:44.999 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:21:44.999 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:21:44.999 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:45.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.019+0000 7f8c46e03640 1 Processor -- start 2024-04-26T00:21:45.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.019+0000 7f8c46e03640 1 -- start start 2024-04-26T00:21:45.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.019+0000 7f8c46e03640 10 monclient: init 2024-04-26T00:21:45.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.023+0000 7f8c46e03640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:45.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.023+0000 7f8c46e03640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-04-26T00:21:45.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.023+0000 7f8c46e03640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8c40138330 addr v1:172.21.15.45:6789/0 2024-04-26T00:21:45.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.023+0000 7f8c46e03640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8c4009de00 0x7f8c401404d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:45.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.023+0000 7f8c46e03640 10 monclient(hunting): picked mon.noname-c con 0x7f8c4009de00 addr v2:172.21.15.45:3300/0 2024-04-26T00:21:45.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.023+0000 7f8c46e03640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8c40140a10 0x7f8c40142e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:45.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.023+0000 7f8c46e03640 10 monclient(hunting): picked mon.noname-b con 0x7f8c40140a10 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:21:45.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.023+0000 7f8c46e03640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c4009cb80 con 0x7f8c40138330 2024-04-26T00:21:45.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.023+0000 7f8c46e03640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:45.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.023+0000 7f8c46e03640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f8c4009d4b0 con 0x7f8c4009de00 2024-04-26T00:21:45.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.023+0000 7f8c46e03640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:45.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.023+0000 7f8c46e03640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8c4009ce60 con 0x7f8c40140a10 2024-04-26T00:21:45.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.023+0000 7f8c46e03640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:45.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.023+0000 7f8c46e03640 10 monclient(hunting): authenticate will time out at 1086.328305s 2024-04-26T00:21:45.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.023+0000 7f8c453a0640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8c40140a10 msgr2=0x7f8c40142e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:21:45.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.023+0000 7f8c453a0640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8c40140a10 0x7f8c40142e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:21:45.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.023+0000 7f8c3ffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8c4009de00 0x7f8c401404d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:45.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.023+0000 7f8c3ffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8c4009de00 0x7f8c401404d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.37:41466/0 (socket says 172.21.15.37:41466) 2024-04-26T00:21:45.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.023+0000 7f8c3ffff640 1 -- 172.21.15.37:0/1079692371 learned_addr learned my addr 172.21.15.37:0/1079692371 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:21:45.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.023+0000 7f8c44b9f640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f8c40138330 0x7f8c40138710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.37:51010/0 (socket says 172.21.15.37:51010) 2024-04-26T00:21:45.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.023+0000 7f8c3ffff640 10 monclient(hunting): get_auth_request con 0x7f8c4009de00 auth_method 0 2024-04-26T00:21:45.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.023+0000 7f8c3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:45.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.023+0000 7f8c3ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:45.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.023+0000 7f8c3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:45.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.023+0000 7f8c3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:45.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.023+0000 7f8c3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:45.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.023+0000 7f8c3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:45.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.023+0000 7f8c3ffff640 10 monclient(hunting): handle_auth_done global_id 24136 payload 274 2024-04-26T00:21:45.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.023+0000 7f8c3ffff640 1 -- 172.21.15.37:0/1079692371 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8c40140a10 msgr2=0x7f8c40142e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:21:45.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.023+0000 7f8c3ffff640 1 --2- 172.21.15.37:0/1079692371 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8c40140a10 0x7f8c40142e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:45.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.023+0000 7f8c3ffff640 1 -- 172.21.15.37:0/1079692371 >> v1:172.21.15.45:6789/0 conn(0x7f8c40138330 legacy=0x7f8c40138710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:21:45.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.023+0000 7f8c3ffff640 10 monclient: _finish_hunting 0 2024-04-26T00:21:45.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.023+0000 7f8c3ffff640 1 monclient: found mon.noname-c 2024-04-26T00:21:45.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.023+0000 7f8c3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:45.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.023+0000 7f8c3ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:21:45.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.023+0000 7f8c3ffff640 1 -- 172.21.15.37:0/1079692371 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c4009d140 con 0x7f8c4009de00 2024-04-26T00:21:45.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.027+0000 7f8c3ffff640 1 -- 172.21.15.37:0/1079692371 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c4009de00 msgr2=0x7f8c401404d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:45.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.027+0000 7f8c3ffff640 1 --2- 172.21.15.37:0/1079692371 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c4009de00 0x7f8c401404d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8c28009e50 tx=0x7f8c2801c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23834bc873bc292f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:45.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.027+0000 7f8c3f7fe640 1 -- 172.21.15.37:0/1079692371 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c280a55d0 con 0x7f8c4009de00 2024-04-26T00:21:45.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.027+0000 7f8c3f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:45.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.027+0000 7f8c3f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:21:45.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.027+0000 7f8c3f7fe640 10 monclient: dump: 2024-04-26T00:21:45.005 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:45.005 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:45.005 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:45.005 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:45.006 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:45.006 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:45.006 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:45.006 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:45.006 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:45.006 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:45.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.027+0000 7f8c3f7fe640 10 monclient: _finish_auth 0 2024-04-26T00:21:45.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.027+0000 7f8c3f7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:21:45.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.027+0000 7f8c3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:45.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.027+0000 7f8c3f7fe640 1 -- 172.21.15.37:0/1079692371 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c280a5770 con 0x7f8c4009de00 2024-04-26T00:21:45.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.027+0000 7f8c3f7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:45.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.027+0000 7f8c3f7fe640 1 -- 172.21.15.37:0/1079692371 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c280a5b40 con 0x7f8c4009de00 2024-04-26T00:21:45.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.027+0000 7f8c3f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:45.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.027+0000 7f8c3f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:21:45.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.027+0000 7f8c3f7fe640 10 monclient: dump: 2024-04-26T00:21:45.007 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:45.007 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:45.007 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:45.007 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:45.007 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:45.007 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:45.007 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:45.007 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:45.007 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:45.007 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:45.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.027+0000 7f8c46e03640 5 monclient: authenticate success, global_id 24136 2024-04-26T00:21:45.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.027+0000 7f8c46e03640 10 monclient: get_monmap_and_config success 2024-04-26T00:21:45.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.027+0000 7f8c46e03640 4 set_mon_vals no callback set 2024-04-26T00:21:45.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.027+0000 7f8c46e03640 10 monclient: shutdown 2024-04-26T00:21:45.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.027+0000 7f8c46e03640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:45.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.027+0000 7f8c46e03640 1 -- 172.21.15.37:0/1079692371 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c4009de00 msgr2=0x7f8c401404d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:45.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.027+0000 7f8c46e03640 1 --2- 172.21.15.37:0/1079692371 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c4009de00 0x7f8c401404d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8c28009e50 tx=0x7f8c2801c8d0 comp rx=0 tx=0).stop 2024-04-26T00:21:45.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.027+0000 7f8c46e03640 1 -- 172.21.15.37:0/1079692371 shutdown_connections 2024-04-26T00:21:45.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.027+0000 7f8c46e03640 1 --2- 172.21.15.37:0/1079692371 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8c40140a10 0x7f8c40142e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:45.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.027+0000 7f8c46e03640 1 --2- 172.21.15.37:0/1079692371 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c4009de00 0x7f8c401404d0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:45.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.027+0000 7f8c46e03640 1 -- 172.21.15.37:0/1079692371 >> 172.21.15.37:0/1079692371 conn(0x7f8c400a1ad0 msgr2=0x7f8c400a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:45.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.027+0000 7f8c46e03640 1 -- 172.21.15.37:0/1079692371 shutdown_connections 2024-04-26T00:21:45.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.027+0000 7f8c46e03640 1 -- 172.21.15.37:0/1079692371 wait complete. 2024-04-26T00:21:45.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.027+0000 7f8c46e03640 10 monclient: build_initial_monmap 2024-04-26T00:21:45.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.027+0000 7f8c46e03640 10 monclient: monmap: 2024-04-26T00:21:45.014 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:45.014 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:21:45.014 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:21:45.014 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:21:45.014 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:45.014 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:21:45.014 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:21:45.014 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:21:45.014 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:21:45.014 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:45.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c46e03640 1 Processor -- start 2024-04-26T00:21:45.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c46e03640 1 -- start start 2024-04-26T00:21:45.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c46e03640 10 monclient: init 2024-04-26T00:21:45.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c46e03640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:45.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c46e03640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:21:45.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c46e03640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8c4009de00 0x7f8c40071d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:45.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c46e03640 10 monclient(hunting): picked mon.noname-c con 0x7f8c4009de00 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:21:45.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c46e03640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8c40138330 0x7f8c40072260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:45.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c46e03640 10 monclient(hunting): picked mon.noname-a con 0x7f8c40138330 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:21:45.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c46e03640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c40140a10 0x7f8c401d3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:45.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c46e03640 10 monclient(hunting): picked mon.noname-b con 0x7f8c40140a10 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:21:45.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c46e03640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:45.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c46e03640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f8c40145ce0 con 0x7f8c40138330 2024-04-26T00:21:45.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c46e03640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:45.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c46e03640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8c40145b60 con 0x7f8c40140a10 2024-04-26T00:21:45.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c46e03640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:45.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c46e03640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8c40145e60 con 0x7f8c4009de00 2024-04-26T00:21:45.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c46e03640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:45.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c46e03640 10 monclient(hunting): authenticate will time out at 1086.337072s 2024-04-26T00:21:45.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c44b9f640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8c4009de00 msgr2=0x7f8c40071d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:21:45.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c44b9f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8c4009de00 0x7f8c40071d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:21:45.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c3ffff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8c40138330 0x7f8c40072260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:45.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c3ffff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8c40138330 0x7f8c40072260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58608/0 (socket says 172.21.15.37:58608) 2024-04-26T00:21:45.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c3ffff640 1 -- 172.21.15.37:0/3375279710 learned_addr learned my addr 172.21.15.37:0/3375279710 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:21:45.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c3ffff640 10 monclient(hunting): get_auth_request con 0x7f8c40138330 auth_method 0 2024-04-26T00:21:45.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:45.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c3ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:45.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:45.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:45.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:45.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:45.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c453a0640 1 --2- 172.21.15.37:0/3375279710 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c40140a10 0x7f8c401d3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:45.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c453a0640 10 monclient(hunting): get_auth_request con 0x7f8c40140a10 auth_method 0 2024-04-26T00:21:45.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c453a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:45.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c453a0640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:45.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c453a0640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:45.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c3ffff640 10 monclient(hunting): handle_auth_done global_id 4578 payload 824 2024-04-26T00:21:45.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c3ffff640 1 -- 172.21.15.37:0/3375279710 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8c4009de00 msgr2=0x7f8c40071d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:21:45.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c3ffff640 1 --2- 172.21.15.37:0/3375279710 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8c4009de00 0x7f8c40071d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:45.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c3ffff640 1 -- 172.21.15.37:0/3375279710 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c40140a10 msgr2=0x7f8c401d3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:45.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c3ffff640 1 --2- 172.21.15.37:0/3375279710 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c40140a10 0x7f8c401d3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:45.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c3ffff640 10 monclient: _finish_hunting 0 2024-04-26T00:21:45.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c3ffff640 1 monclient: found mon.noname-a 2024-04-26T00:21:45.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:45.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c3ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:21:45.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c3ffff640 1 -- 172.21.15.37:0/3375279710 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c400a2e60 con 0x7f8c40138330 2024-04-26T00:21:45.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c3ffff640 1 -- 172.21.15.37:0/3375279710 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8c40138330 msgr2=0x7f8c40072260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:45.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.031+0000 7f8c3ffff640 1 --2- 172.21.15.37:0/3375279710 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8c40138330 0x7f8c40072260 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8c2801af80 tx=0x7f8c28019670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:45.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.035+0000 7f8c3d7fa640 1 -- 172.21.15.37:0/3375279710 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c280bc020 con 0x7f8c40138330 2024-04-26T00:21:45.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.035+0000 7f8c3d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:45.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.035+0000 7f8c3d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:21:45.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.035+0000 7f8c3d7fa640 10 monclient: dump: 2024-04-26T00:21:45.018 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:45.018 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:45.018 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:45.019 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:45.019 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:45.019 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:45.019 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:45.019 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:45.019 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:45.019 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:45.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.035+0000 7f8c3d7fa640 10 monclient: _finish_auth 0 2024-04-26T00:21:45.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.035+0000 7f8c3d7fa640 10 monclient: _check_auth_tickets 2024-04-26T00:21:45.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.035+0000 7f8c3d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:45.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.035+0000 7f8c3d7fa640 1 -- 172.21.15.37:0/3375279710 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c280b8880 con 0x7f8c40138330 2024-04-26T00:21:45.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.035+0000 7f8c3d7fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:45.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.035+0000 7f8c3d7fa640 1 -- 172.21.15.37:0/3375279710 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c280a55d0 con 0x7f8c40138330 2024-04-26T00:21:45.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.035+0000 7f8c46e03640 5 monclient: authenticate success, global_id 4578 2024-04-26T00:21:45.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.035+0000 7f8c46e03640 10 monclient: _renew_subs 2024-04-26T00:21:45.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.035+0000 7f8c46e03640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:45.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.035+0000 7f8c46e03640 1 -- 172.21.15.37:0/3375279710 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c400727a0 con 0x7f8c40138330 2024-04-26T00:21:45.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.035+0000 7f8c3effd640 4 set_mon_vals no callback set 2024-04-26T00:21:45.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.035+0000 7f8c3d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:45.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.035+0000 7f8c3d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:21:45.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.035+0000 7f8c3d7fa640 10 monclient: dump: 2024-04-26T00:21:45.020 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:45.020 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:45.020 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:45.020 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:45.020 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:45.020 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:45.020 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:45.020 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:45.021 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:45.021 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:45.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.035+0000 7f8c46e03640 10 monclient: _renew_subs 2024-04-26T00:21:45.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.035+0000 7f8c46e03640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:45.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.035+0000 7f8c46e03640 1 -- 172.21.15.37:0/3375279710 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c40140040 con 0x7f8c40138330 2024-04-26T00:21:45.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.035+0000 7f8c46e03640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:45.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.035+0000 7f8c46e03640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:45.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.035+0000 7f8c46e03640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:45.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.035+0000 7f8c3d7fa640 1 -- 172.21.15.37:0/3375279710 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f8c280bf020 con 0x7f8c40138330 2024-04-26T00:21:45.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.035+0000 7f8c46e03640 1 -- 172.21.15.37:0/3375279710 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c40066f40 con 0x7f8c40138330 2024-04-26T00:21:45.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.035+0000 7f8c3d7fa640 1 --2- 172.21.15.37:0/3375279710 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8c20035e60 0x7f8c20038320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:45.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.035+0000 7f8c3d7fa640 1 -- 172.21.15.37:0/3375279710 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 3943+0+0 (secure 0 0 0) 0x7f8c280e3830 con 0x7f8c40138330 2024-04-26T00:21:45.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.035+0000 7f8c44b9f640 1 --2- 172.21.15.37:0/3375279710 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8c20035e60 0x7f8c20038320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:45.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.035+0000 7f8c44b9f640 10 monclient: get_auth_request con 0x7f8c20035e60 auth_method 0 2024-04-26T00:21:45.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.039+0000 7f8c44b9f640 1 --2- 172.21.15.37:0/3375279710 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8c20035e60 0x7f8c20038320 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8c30004920 tx=0x7f8c30007a70 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:45.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.039+0000 7f8c3d7fa640 1 -- 172.21.15.37:0/3375279710 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f8c280c6070 con 0x7f8c40138330 2024-04-26T00:21:45.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.039+0000 7f8c3d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:45.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.039+0000 7f8c3d7fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:21:45.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.267+0000 7f8c46e03640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:21:45.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.267+0000 7f8c46e03640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:21:45.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.267+0000 7f8c46e03640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:45.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.267+0000 7f8c46e03640 1 -- 172.21.15.37:0/3375279710 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8c4006b9a0 con 0x7f8c40138330 2024-04-26T00:21:45.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.267+0000 7f8c3d7fa640 1 -- 172.21.15.37:0/3375279710 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f8c280165e0 con 0x7f8c40138330 2024-04-26T00:21:45.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.267+0000 7f8c3d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:21:45.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.267+0000 7f8c3d7fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:21:45.245 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":18,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:21:45.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.267+0000 7f8c46e03640 1 -- 172.21.15.37:0/3375279710 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8c20035e60 msgr2=0x7f8c20038320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:45.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.267+0000 7f8c46e03640 1 --2- 172.21.15.37:0/3375279710 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8c20035e60 0x7f8c20038320 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8c30004920 tx=0x7f8c30007a70 comp rx=0 tx=0).stop 2024-04-26T00:21:45.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.267+0000 7f8c46e03640 10 monclient: shutdown 2024-04-26T00:21:45.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.267+0000 7f8c46e03640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:45.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.267+0000 7f8c46e03640 1 -- 172.21.15.37:0/3375279710 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8c40138330 msgr2=0x7f8c40072260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:45.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.267+0000 7f8c46e03640 1 --2- 172.21.15.37:0/3375279710 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8c40138330 0x7f8c40072260 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8c2801af80 tx=0x7f8c28019670 comp rx=0 tx=0).stop 2024-04-26T00:21:45.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.267+0000 7f8c46e03640 1 -- 172.21.15.37:0/3375279710 shutdown_connections 2024-04-26T00:21:45.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.267+0000 7f8c46e03640 1 --2- 172.21.15.37:0/3375279710 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8c20035e60 0x7f8c20038320 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:45.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.267+0000 7f8c46e03640 1 --2- 172.21.15.37:0/3375279710 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c40140a10 0x7f8c401d3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:45.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.267+0000 7f8c46e03640 1 --2- 172.21.15.37:0/3375279710 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8c40138330 0x7f8c40072260 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:45.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.267+0000 7f8c46e03640 1 --2- 172.21.15.37:0/3375279710 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8c4009de00 0x7f8c40071d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:45.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.267+0000 7f8c46e03640 1 -- 172.21.15.37:0/3375279710 >> 172.21.15.37:0/3375279710 conn(0x7f8c400a1ad0 msgr2=0x7f8c4009e260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:45.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.267+0000 7f8c46e03640 1 -- 172.21.15.37:0/3375279710 shutdown_connections 2024-04-26T00:21:45.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:45.267+0000 7f8c46e03640 1 -- 172.21.15.37:0/3375279710 wait complete. 2024-04-26T00:21:48.267 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:21:48.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc00407d640 10 monclient: get_monmap_and_config 2024-04-26T00:21:48.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc00407d640 10 monclient: build_initial_monmap 2024-04-26T00:21:48.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc00407d640 10 monclient: monmap: 2024-04-26T00:21:48.401 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:48.401 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:48.401 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:21:48.425220+0000 2024-04-26T00:21:48.401 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:21:48.425220+0000 2024-04-26T00:21:48.401 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:48.401 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:48.401 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:21:48.401 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:21:48.401 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:21:48.401 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:21:48.402 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:21:48.402 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:48.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc00407d640 1 Processor -- start 2024-04-26T00:21:48.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc00407d640 1 -- start start 2024-04-26T00:21:48.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc00407d640 10 monclient: init 2024-04-26T00:21:48.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc00407d640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:48.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc00407d640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-04-26T00:21:48.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc00407d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbffc0a0260 0x7fbffc0a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:48.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc00407d640 10 monclient(hunting): picked mon.noname-b con 0x7fbffc0a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:21:48.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc00407d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fbffc134b30 0x7fbffc134f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:48.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc00407d640 10 monclient(hunting): picked mon.noname-a con 0x7fbffc134b30 addr v2:172.21.15.37:3300/0 2024-04-26T00:21:48.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc00407d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbffc135490 addr v1:172.21.15.37:6789/0 2024-04-26T00:21:48.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc00407d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbffc0a1130 con 0x7fbffc135490 2024-04-26T00:21:48.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc00407d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:48.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc00407d640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fbffc09ec10 con 0x7fbffc134b30 2024-04-26T00:21:48.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc00407d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:48.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc00407d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fbffc09f0f0 con 0x7fbffc0a0260 2024-04-26T00:21:48.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc00407d640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:48.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc00407d640 10 monclient(hunting): authenticate will time out at 1089.730914s 2024-04-26T00:21:48.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc001e19640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbffc0a0260 msgr2=0x7fbffc0a0640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:21:48.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc001e19640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbffc0a0260 0x7fbffc0a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:21:48.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc001618640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fbffc134b30 0x7fbffc134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:48.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc00261a640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fbffc135490 0x7fbffc13be60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:54118/0 (socket says 172.21.15.37:54118) 2024-04-26T00:21:48.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc00261a640 1 -- 172.21.15.37:0/3438050007 learned_addr learned my addr 172.21.15.37:0/3438050007 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:21:48.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc001618640 10 monclient(hunting): get_auth_request con 0x7fbffc134b30 auth_method 0 2024-04-26T00:21:48.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc001618640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:48.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc001618640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:48.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc001618640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:48.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc001618640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:48.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc001618640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:48.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc001618640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:48.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc001618640 10 monclient(hunting): handle_auth_done global_id 4581 payload 274 2024-04-26T00:21:48.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc001618640 1 -- 172.21.15.37:0/3438050007 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbffc0a0260 msgr2=0x7fbffc0a0640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:21:48.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc001618640 1 --2- 172.21.15.37:0/3438050007 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbffc0a0260 0x7fbffc0a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:48.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc001618640 1 -- 172.21.15.37:0/3438050007 >> v1:172.21.15.37:6789/0 conn(0x7fbffc135490 legacy=0x7fbffc13be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:21:48.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc001618640 10 monclient: _finish_hunting 0 2024-04-26T00:21:48.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc001618640 1 monclient: found mon.noname-a 2024-04-26T00:21:48.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc001618640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:48.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc001618640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:21:48.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.423+0000 7fc001618640 1 -- 172.21.15.37:0/3438050007 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbffc06cff0 con 0x7fbffc134b30 2024-04-26T00:21:48.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc001618640 1 -- 172.21.15.37:0/3438050007 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbffc134b30 msgr2=0x7fbffc134f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:48.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc001618640 1 --2- 172.21.15.37:0/3438050007 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbffc134b30 0x7fbffc134f50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbfec0097f0 tx=0x7fbfec01c160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=914c4c0a8af9bb1e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:48.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc000e17640 1 -- 172.21.15.37:0/3438050007 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfec0b1070 con 0x7fbffc134b30 2024-04-26T00:21:48.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc000e17640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:48.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc000e17640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:21:48.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc000e17640 10 monclient: dump: 2024-04-26T00:21:48.408 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:48.408 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:48.408 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:48.408 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:48.409 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:48.409 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:48.409 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:48.409 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:48.409 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:48.409 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:48.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc000e17640 10 monclient: _finish_auth 0 2024-04-26T00:21:48.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc000e17640 10 monclient: _check_auth_tickets 2024-04-26T00:21:48.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc000e17640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:48.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc00407d640 5 monclient: authenticate success, global_id 4581 2024-04-26T00:21:48.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc00407d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:21:48.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc000e17640 1 -- 172.21.15.37:0/3438050007 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfec0ac440 con 0x7fbffc134b30 2024-04-26T00:21:48.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc000e17640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:48.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc00407d640 10 monclient: get_monmap_and_config success 2024-04-26T00:21:48.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc00407d640 4 set_mon_vals no callback set 2024-04-26T00:21:48.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc000e17640 1 -- 172.21.15.37:0/3438050007 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfec0ac810 con 0x7fbffc134b30 2024-04-26T00:21:48.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc000e17640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:48.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc000e17640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:21:48.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc000e17640 10 monclient: dump: 2024-04-26T00:21:48.410 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:48.410 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:48.410 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:48.410 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:48.410 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:48.410 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:48.410 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:48.410 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:48.410 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:48.410 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:48.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc00407d640 10 monclient: shutdown 2024-04-26T00:21:48.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc00407d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:48.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc00407d640 1 -- 172.21.15.37:0/3438050007 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbffc134b30 msgr2=0x7fbffc134f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:48.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc00407d640 1 --2- 172.21.15.37:0/3438050007 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbffc134b30 0x7fbffc134f50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbfec0097f0 tx=0x7fbfec01c160 comp rx=0 tx=0).stop 2024-04-26T00:21:48.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc00407d640 1 -- 172.21.15.37:0/3438050007 shutdown_connections 2024-04-26T00:21:48.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc00407d640 1 --2- 172.21.15.37:0/3438050007 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbffc134b30 0x7fbffc134f50 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:48.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc00407d640 1 --2- 172.21.15.37:0/3438050007 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbffc0a0260 0x7fbffc0a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:48.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc00407d640 1 -- 172.21.15.37:0/3438050007 >> 172.21.15.37:0/3438050007 conn(0x7fbffc0a1ad0 msgr2=0x7fbffc0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:48.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc00407d640 1 -- 172.21.15.37:0/3438050007 shutdown_connections 2024-04-26T00:21:48.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc00407d640 1 -- 172.21.15.37:0/3438050007 wait complete. 2024-04-26T00:21:48.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc00407d640 10 monclient: build_initial_monmap 2024-04-26T00:21:48.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc00407d640 10 monclient: monmap: 2024-04-26T00:21:48.416 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:48.416 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:21:48.416 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:21:48.416 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:21:48.416 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:48.416 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:21:48.416 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:21:48.417 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:21:48.417 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:21:48.417 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:48.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc00407d640 1 Processor -- start 2024-04-26T00:21:48.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc00407d640 1 -- start start 2024-04-26T00:21:48.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc00407d640 10 monclient: init 2024-04-26T00:21:48.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc00407d640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:48.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc00407d640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:21:48.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc00407d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbffc0a0260 0x7fbffc1cd010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:48.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc00407d640 10 monclient(hunting): picked mon.noname-a con 0x7fbffc0a0260 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:21:48.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc00407d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbffc134b30 0x7fbffc0715f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:48.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc00407d640 10 monclient(hunting): picked mon.noname-b con 0x7fbffc134b30 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:21:48.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc00407d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbffc135490 0x7fbffc071b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:48.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc00407d640 10 monclient(hunting): picked mon.noname-c con 0x7fbffc135490 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:21:48.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc00407d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:48.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc00407d640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fbffc09f430 con 0x7fbffc0a0260 2024-04-26T00:21:48.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc00407d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:48.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc00407d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbffc09f2b0 con 0x7fbffc134b30 2024-04-26T00:21:48.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc00407d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:48.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc00407d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fbffc09fe90 con 0x7fbffc135490 2024-04-26T00:21:48.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc00407d640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:48.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc00407d640 10 monclient(hunting): authenticate will time out at 1089.739528s 2024-04-26T00:21:48.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc00261a640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbffc135490 msgr2=0x7fbffc071b30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:21:48.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc00261a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbffc135490 0x7fbffc071b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:21:48.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc001e19640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbffc0a0260 0x7fbffc1cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:48.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc001e19640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbffc0a0260 0x7fbffc1cd010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58618/0 (socket says 172.21.15.37:58618) 2024-04-26T00:21:48.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc001e19640 1 -- 172.21.15.37:0/3615066959 learned_addr learned my addr 172.21.15.37:0/3615066959 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:21:48.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc001e19640 10 monclient(hunting): get_auth_request con 0x7fbffc0a0260 auth_method 0 2024-04-26T00:21:48.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc001e19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:48.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc001e19640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:48.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc001e19640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:48.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc001e19640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:48.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc001e19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:48.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc001e19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:48.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.431+0000 7fc001618640 1 --2- 172.21.15.37:0/3615066959 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbffc134b30 0x7fbffc0715f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:48.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fc001618640 10 monclient(hunting): get_auth_request con 0x7fbffc134b30 auth_method 0 2024-04-26T00:21:48.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fc001618640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:48.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fc001618640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:48.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fc001618640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:48.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fc001e19640 10 monclient(hunting): handle_auth_done global_id 4584 payload 824 2024-04-26T00:21:48.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fc001e19640 1 -- 172.21.15.37:0/3615066959 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbffc135490 msgr2=0x7fbffc071b30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:21:48.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fc001e19640 1 --2- 172.21.15.37:0/3615066959 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbffc135490 0x7fbffc071b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:48.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fc001e19640 1 -- 172.21.15.37:0/3615066959 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbffc134b30 msgr2=0x7fbffc0715f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:48.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fc001e19640 1 --2- 172.21.15.37:0/3615066959 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbffc134b30 0x7fbffc0715f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:48.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fc001e19640 10 monclient: _finish_hunting 0 2024-04-26T00:21:48.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fc001e19640 1 monclient: found mon.noname-a 2024-04-26T00:21:48.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fc001e19640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:48.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fc001e19640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:21:48.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fc001e19640 1 -- 172.21.15.37:0/3615066959 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbffc06e8c0 con 0x7fbffc0a0260 2024-04-26T00:21:48.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fc001e19640 1 -- 172.21.15.37:0/3615066959 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbffc0a0260 msgr2=0x7fbffc1cd010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:48.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fc001e19640 1 --2- 172.21.15.37:0/3615066959 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbffc0a0260 0x7fbffc1cd010 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbff800b550 tx=0x7fbff800ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:48.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fbff27fc640 1 -- 172.21.15.37:0/3615066959 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff80050f0 con 0x7fbffc0a0260 2024-04-26T00:21:48.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fbff27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:48.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fbff27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:21:48.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fbff27fc640 10 monclient: dump: 2024-04-26T00:21:48.420 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:48.421 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:48.421 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:48.421 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:48.421 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:48.421 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:48.421 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:48.421 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:48.421 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:48.421 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:48.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fbff27fc640 10 monclient: _finish_auth 0 2024-04-26T00:21:48.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fbff27fc640 10 monclient: _check_auth_tickets 2024-04-26T00:21:48.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fbff27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:48.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fc00407d640 5 monclient: authenticate success, global_id 4584 2024-04-26T00:21:48.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fbff27fc640 1 -- 172.21.15.37:0/3615066959 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbff8015070 con 0x7fbffc0a0260 2024-04-26T00:21:48.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fc00407d640 10 monclient: _renew_subs 2024-04-26T00:21:48.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fc00407d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:48.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fc00407d640 1 -- 172.21.15.37:0/3615066959 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbffc132470 con 0x7fbffc0a0260 2024-04-26T00:21:48.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fbff27fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:48.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fbff27fc640 1 -- 172.21.15.37:0/3615066959 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff80054d0 con 0x7fbffc0a0260 2024-04-26T00:21:48.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fbff27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:48.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fbff27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:21:48.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fbff27fc640 10 monclient: dump: 2024-04-26T00:21:48.422 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:48.422 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:48.422 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:48.422 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:48.422 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:48.422 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:48.422 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:48.423 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:48.423 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:48.423 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:48.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fc00407d640 10 monclient: _renew_subs 2024-04-26T00:21:48.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fbff37fe640 4 set_mon_vals no callback set 2024-04-26T00:21:48.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fc00407d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:48.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fc00407d640 1 -- 172.21.15.37:0/3615066959 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbffc072070 con 0x7fbffc0a0260 2024-04-26T00:21:48.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fc00407d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:48.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fc00407d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:48.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fc00407d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:48.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fc00407d640 1 -- 172.21.15.37:0/3615066959 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbffc066f40 con 0x7fbffc0a0260 2024-04-26T00:21:48.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fbff27fc640 1 -- 172.21.15.37:0/3615066959 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fbff8005670 con 0x7fbffc0a0260 2024-04-26T00:21:48.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fbff27fc640 1 --2- 172.21.15.37:0/3615066959 >> v2:172.21.15.45:6812/2379918623 conn(0x7fbfe0035eb0 0x7fbfe0038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:48.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.435+0000 7fbff27fc640 1 -- 172.21.15.37:0/3615066959 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 3943+0+0 (secure 0 0 0) 0x7fbff8048900 con 0x7fbffc0a0260 2024-04-26T00:21:48.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.439+0000 7fc001618640 1 --2- 172.21.15.37:0/3615066959 >> v2:172.21.15.45:6812/2379918623 conn(0x7fbfe0035eb0 0x7fbfe0038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:48.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.439+0000 7fc001618640 10 monclient: get_auth_request con 0x7fbfe0035eb0 auth_method 0 2024-04-26T00:21:48.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.439+0000 7fc001618640 1 --2- 172.21.15.37:0/3615066959 >> v2:172.21.15.45:6812/2379918623 conn(0x7fbfe0035eb0 0x7fbfe0038370 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbfec0ab000 tx=0x7fbfec0a6070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:48.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.443+0000 7fbff27fc640 1 -- 172.21.15.37:0/3615066959 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fbff800e340 con 0x7fbffc0a0260 2024-04-26T00:21:48.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.443+0000 7fbff27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:48.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.443+0000 7fbff27fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:21:48.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.667+0000 7fc00407d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:21:48.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.667+0000 7fc00407d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:21:48.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.667+0000 7fc00407d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:48.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.667+0000 7fc00407d640 1 -- 172.21.15.37:0/3615066959 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fbffc06b9a0 con 0x7fbffc0a0260 2024-04-26T00:21:48.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.671+0000 7fbff27fc640 1 -- 172.21.15.37:0/3615066959 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fbff8010660 con 0x7fbffc0a0260 2024-04-26T00:21:48.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.671+0000 7fbff27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:21:48.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.671+0000 7fbff27fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:21:48.648 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":18,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:21:48.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.671+0000 7fc00407d640 1 -- 172.21.15.37:0/3615066959 >> v2:172.21.15.45:6812/2379918623 conn(0x7fbfe0035eb0 msgr2=0x7fbfe0038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:48.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.671+0000 7fc00407d640 1 --2- 172.21.15.37:0/3615066959 >> v2:172.21.15.45:6812/2379918623 conn(0x7fbfe0035eb0 0x7fbfe0038370 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbfec0ab000 tx=0x7fbfec0a6070 comp rx=0 tx=0).stop 2024-04-26T00:21:48.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.671+0000 7fc00407d640 10 monclient: shutdown 2024-04-26T00:21:48.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.671+0000 7fc00407d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:48.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.671+0000 7fc00407d640 1 -- 172.21.15.37:0/3615066959 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbffc0a0260 msgr2=0x7fbffc1cd010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:48.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.671+0000 7fc00407d640 1 --2- 172.21.15.37:0/3615066959 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbffc0a0260 0x7fbffc1cd010 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbff800b550 tx=0x7fbff800ba10 comp rx=0 tx=0).stop 2024-04-26T00:21:48.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.671+0000 7fc00407d640 1 -- 172.21.15.37:0/3615066959 shutdown_connections 2024-04-26T00:21:48.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.671+0000 7fc00407d640 1 --2- 172.21.15.37:0/3615066959 >> v2:172.21.15.45:6812/2379918623 conn(0x7fbfe0035eb0 0x7fbfe0038370 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:48.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.671+0000 7fc00407d640 1 --2- 172.21.15.37:0/3615066959 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbffc135490 0x7fbffc071b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:48.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.671+0000 7fc00407d640 1 --2- 172.21.15.37:0/3615066959 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbffc134b30 0x7fbffc0715f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:48.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.671+0000 7fc00407d640 1 --2- 172.21.15.37:0/3615066959 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbffc0a0260 0x7fbffc1cd010 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:48.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.671+0000 7fc00407d640 1 -- 172.21.15.37:0/3615066959 >> 172.21.15.37:0/3615066959 conn(0x7fbffc0a1ad0 msgr2=0x7fbffc0a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:48.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.671+0000 7fc00407d640 1 -- 172.21.15.37:0/3615066959 shutdown_connections 2024-04-26T00:21:48.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:48.671+0000 7fc00407d640 1 -- 172.21.15.37:0/3615066959 wait complete. 2024-04-26T00:21:51.668 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:21:51.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.819+0000 7f175a860640 10 monclient: get_monmap_and_config 2024-04-26T00:21:51.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.819+0000 7f175a860640 10 monclient: build_initial_monmap 2024-04-26T00:21:51.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.819+0000 7f175a860640 10 monclient: monmap: 2024-04-26T00:21:51.797 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:51.797 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:51.798 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:21:51.821931+0000 2024-04-26T00:21:51.798 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:21:51.821931+0000 2024-04-26T00:21:51.798 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:51.798 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:51.798 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:21:51.798 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:21:51.798 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:21:51.798 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:21:51.798 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:21:51.798 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:51.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.819+0000 7f175a860640 1 Processor -- start 2024-04-26T00:21:51.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.819+0000 7f175a860640 1 -- start start 2024-04-26T00:21:51.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.819+0000 7f175a860640 10 monclient: init 2024-04-26T00:21:51.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.819+0000 7f175a860640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:51.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.819+0000 7f175a860640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-04-26T00:21:51.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.819+0000 7f175a860640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1754136e10 addr v1:172.21.15.37:6789/0 2024-04-26T00:21:51.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.819+0000 7f175a860640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1754137900 addr v1:172.21.15.45:6789/0 2024-04-26T00:21:51.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.819+0000 7f175a860640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f175409c5b0 0x7f175413fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:51.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.819+0000 7f175a860640 10 monclient(hunting): picked mon.noname-c con 0x7f175409c5b0 addr v2:172.21.15.45:3300/0 2024-04-26T00:21:51.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.819+0000 7f175a860640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f175409b680 con 0x7f1754136e10 2024-04-26T00:21:51.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.819+0000 7f175a860640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f175409bbd0 con 0x7f1754137900 2024-04-26T00:21:51.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.819+0000 7f175a860640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:51.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.819+0000 7f175a860640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f175409b8f0 con 0x7f175409c5b0 2024-04-26T00:21:51.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.819+0000 7f175a860640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:51.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.819+0000 7f175a860640 10 monclient(hunting): authenticate will time out at 1093.127881s 2024-04-26T00:21:51.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.823+0000 7f1753fff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f1754136e10 0x7f17541371f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:36480/0 (socket says 172.21.15.37:36480) 2024-04-26T00:21:51.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.823+0000 7f1753fff640 1 -- 172.21.15.37:0/2466418081 learned_addr learned my addr 172.21.15.37:0/2466418081 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:21:51.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.823+0000 7f1758dfd640 1 --2- 172.21.15.37:0/2466418081 >> v2:172.21.15.45:3300/0 conn(0x7f175409c5b0 0x7f175413fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:51.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.823+0000 7f1758dfd640 10 monclient(hunting): get_auth_request con 0x7f175409c5b0 auth_method 0 2024-04-26T00:21:51.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.823+0000 7f1758dfd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:51.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.823+0000 7f1758dfd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:51.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.823+0000 7f1758dfd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:51.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.823+0000 7f1758dfd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:51.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.823+0000 7f1758dfd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:51.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.823+0000 7f1758dfd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:51.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.823+0000 7f1758dfd640 10 monclient(hunting): handle_auth_done global_id 24145 payload 274 2024-04-26T00:21:51.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.823+0000 7f1758dfd640 1 -- 172.21.15.37:0/2466418081 >> v1:172.21.15.45:6789/0 conn(0x7f1754137900 legacy=0x7f175413c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:21:51.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.823+0000 7f1758dfd640 1 -- 172.21.15.37:0/2466418081 >> v1:172.21.15.37:6789/0 conn(0x7f1754136e10 legacy=0x7f17541371f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:21:51.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.823+0000 7f1758dfd640 10 monclient: _finish_hunting 0 2024-04-26T00:21:51.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.823+0000 7f1758dfd640 1 monclient: found mon.noname-c 2024-04-26T00:21:51.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.823+0000 7f1758dfd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:51.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.823+0000 7f1758dfd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:21:51.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.823+0000 7f1758dfd640 1 -- 172.21.15.37:0/2466418081 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17540a5d90 con 0x7f175409c5b0 2024-04-26T00:21:51.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.827+0000 7f1758dfd640 1 -- 172.21.15.37:0/2466418081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f175409c5b0 msgr2=0x7f175413fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:51.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.827+0000 7f1758dfd640 1 --2- 172.21.15.37:0/2466418081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f175409c5b0 0x7f175413fa70 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f174c009d90 tx=0x7f174c01c7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a2e4ef343cacffd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:51.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.827+0000 7f17537fe640 1 -- 172.21.15.37:0/2466418081 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f174c0b2070 con 0x7f175409c5b0 2024-04-26T00:21:51.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.827+0000 7f17537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:51.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.827+0000 7f17537fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:21:51.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.827+0000 7f17537fe640 10 monclient: dump: 2024-04-26T00:21:51.806 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:51.806 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:51.806 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:51.806 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:51.806 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:51.806 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:51.806 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:51.806 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:51.806 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:51.806 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:51.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.827+0000 7f17537fe640 10 monclient: _finish_auth 0 2024-04-26T00:21:51.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.827+0000 7f17537fe640 10 monclient: _check_auth_tickets 2024-04-26T00:21:51.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.827+0000 7f17537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:51.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.827+0000 7f17537fe640 1 -- 172.21.15.37:0/2466418081 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f174c0a5860 con 0x7f175409c5b0 2024-04-26T00:21:51.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.827+0000 7f17537fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:51.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.827+0000 7f17537fe640 1 -- 172.21.15.37:0/2466418081 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f174c0a5c30 con 0x7f175409c5b0 2024-04-26T00:21:51.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.827+0000 7f17537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:51.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.827+0000 7f17537fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:21:51.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.827+0000 7f17537fe640 10 monclient: dump: 2024-04-26T00:21:51.807 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:51.807 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:51.807 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:51.807 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:51.807 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:51.807 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:51.807 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:51.808 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:51.808 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:51.808 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:51.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.827+0000 7f175a860640 5 monclient: authenticate success, global_id 24145 2024-04-26T00:21:51.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.827+0000 7f175a860640 10 monclient: get_monmap_and_config success 2024-04-26T00:21:51.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.827+0000 7f175a860640 4 set_mon_vals no callback set 2024-04-26T00:21:51.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.827+0000 7f175a860640 10 monclient: shutdown 2024-04-26T00:21:51.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.827+0000 7f175a860640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:51.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.827+0000 7f175a860640 1 -- 172.21.15.37:0/2466418081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f175409c5b0 msgr2=0x7f175413fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:51.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.827+0000 7f175a860640 1 --2- 172.21.15.37:0/2466418081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f175409c5b0 0x7f175413fa70 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f174c009d90 tx=0x7f174c01c7b0 comp rx=0 tx=0).stop 2024-04-26T00:21:51.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.827+0000 7f175a860640 1 -- 172.21.15.37:0/2466418081 shutdown_connections 2024-04-26T00:21:51.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.827+0000 7f175a860640 1 --2- 172.21.15.37:0/2466418081 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f175409c5b0 0x7f175413fa70 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:51.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.827+0000 7f175a860640 1 -- 172.21.15.37:0/2466418081 >> 172.21.15.37:0/2466418081 conn(0x7f17540a64e0 msgr2=0x7f17540a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:51.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.827+0000 7f175a860640 1 -- 172.21.15.37:0/2466418081 shutdown_connections 2024-04-26T00:21:51.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.827+0000 7f175a860640 1 -- 172.21.15.37:0/2466418081 wait complete. 2024-04-26T00:21:51.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f175a860640 10 monclient: build_initial_monmap 2024-04-26T00:21:51.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f175a860640 10 monclient: monmap: 2024-04-26T00:21:51.813 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:51.813 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:21:51.813 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:21:51.813 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:21:51.813 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:51.813 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:21:51.813 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:21:51.813 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:21:51.814 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:21:51.814 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:51.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f175a860640 1 Processor -- start 2024-04-26T00:21:51.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f175a860640 1 -- start start 2024-04-26T00:21:51.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f175a860640 10 monclient: init 2024-04-26T00:21:51.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f175a860640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:51.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f175a860640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:21:51.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f175a860640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f175409c5b0 0x7f17541cf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:51.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f175a860640 10 monclient(hunting): picked mon.noname-c con 0x7f175409c5b0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:21:51.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f175a860640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1754136e10 0x7f175413b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:51.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f175a860640 10 monclient(hunting): picked mon.noname-a con 0x7f1754136e10 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:21:51.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f175a860640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1754137900 0x7f175413be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:51.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f175a860640 10 monclient(hunting): picked mon.noname-b con 0x7f1754137900 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:21:51.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f175a860640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:51.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f175a860640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f1754142b60 con 0x7f1754136e10 2024-04-26T00:21:51.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f175a860640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:51.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f175a860640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f17541429e0 con 0x7f1754137900 2024-04-26T00:21:51.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f175a860640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:51.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f175a860640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f1754142ce0 con 0x7f175409c5b0 2024-04-26T00:21:51.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f175a860640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:51.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f175a860640 10 monclient(hunting): authenticate will time out at 1093.137003s 2024-04-26T00:21:51.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f1753fff640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f175409c5b0 msgr2=0x7f17541cf3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:21:51.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f1753fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f175409c5b0 0x7f17541cf3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:21:51.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f174b7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1754136e10 0x7f175413b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:51.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f174b7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1754136e10 0x7f175413b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:41670/0 (socket says 172.21.15.37:41670) 2024-04-26T00:21:51.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f174b7fe640 1 -- 172.21.15.37:0/1014521330 learned_addr learned my addr 172.21.15.37:0/1014521330 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:21:51.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f174b7fe640 10 monclient(hunting): get_auth_request con 0x7f1754136e10 auth_method 0 2024-04-26T00:21:51.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f174b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:51.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f174b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:51.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f174b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:51.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f174b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:51.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f174b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:51.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f174b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:51.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f1758dfd640 1 --2- 172.21.15.37:0/1014521330 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1754137900 0x7f175413be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:51.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f1758dfd640 10 monclient(hunting): get_auth_request con 0x7f1754137900 auth_method 0 2024-04-26T00:21:51.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f1758dfd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:51.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f1758dfd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:51.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f1758dfd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:51.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f174b7fe640 10 monclient(hunting): handle_auth_done global_id 4587 payload 824 2024-04-26T00:21:51.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f174b7fe640 1 -- 172.21.15.37:0/1014521330 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f175409c5b0 msgr2=0x7f17541cf3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:21:51.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f174b7fe640 1 --2- 172.21.15.37:0/1014521330 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f175409c5b0 0x7f17541cf3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:51.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f174b7fe640 1 -- 172.21.15.37:0/1014521330 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1754137900 msgr2=0x7f175413be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:51.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f174b7fe640 1 --2- 172.21.15.37:0/1014521330 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1754137900 0x7f175413be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:51.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f174b7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:21:51.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f174b7fe640 1 monclient: found mon.noname-a 2024-04-26T00:21:51.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f174b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:51.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f174b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:21:51.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f174b7fe640 1 -- 172.21.15.37:0/1014521330 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f175413f5c0 con 0x7f1754136e10 2024-04-26T00:21:51.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f174b7fe640 1 -- 172.21.15.37:0/1014521330 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1754136e10 msgr2=0x7f175413b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:51.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.831+0000 7f174b7fe640 1 --2- 172.21.15.37:0/1014521330 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1754136e10 0x7f175413b910 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f174400c450 tx=0x7f174400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:51.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.835+0000 7f17517fa640 1 -- 172.21.15.37:0/1014521330 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1744012a40 con 0x7f1754136e10 2024-04-26T00:21:51.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.835+0000 7f17517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:51.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.835+0000 7f17517fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:21:51.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.835+0000 7f17517fa640 10 monclient: dump: 2024-04-26T00:21:51.818 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:51.818 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:51.818 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:51.818 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:51.818 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:51.818 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:51.818 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:51.818 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:51.818 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:51.818 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:51.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.835+0000 7f17517fa640 10 monclient: _finish_auth 0 2024-04-26T00:21:51.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.835+0000 7f17517fa640 10 monclient: _check_auth_tickets 2024-04-26T00:21:51.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.835+0000 7f17517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:51.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.835+0000 7f17517fa640 1 -- 172.21.15.37:0/1014521330 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1744017070 con 0x7f1754136e10 2024-04-26T00:21:51.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.835+0000 7f175a860640 5 monclient: authenticate success, global_id 4587 2024-04-26T00:21:51.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.835+0000 7f175a860640 10 monclient: _renew_subs 2024-04-26T00:21:51.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.835+0000 7f175a860640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:51.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.835+0000 7f175a860640 1 -- 172.21.15.37:0/1014521330 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f175413e110 con 0x7f1754136e10 2024-04-26T00:21:51.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.835+0000 7f17517fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:51.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.835+0000 7f17517fa640 1 -- 172.21.15.37:0/1014521330 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1744012e40 con 0x7f1754136e10 2024-04-26T00:21:51.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.835+0000 7f17517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:51.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.835+0000 7f17517fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:21:51.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.835+0000 7f17517fa640 10 monclient: dump: 2024-04-26T00:21:51.819 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:51.819 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:51.819 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:51.819 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:51.819 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:51.820 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:51.820 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:51.820 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:51.820 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:51.820 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:51.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.835+0000 7f175a860640 10 monclient: _renew_subs 2024-04-26T00:21:51.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.835+0000 7f175a860640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:51.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.835+0000 7f175a860640 1 -- 172.21.15.37:0/1014521330 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f175409aca0 con 0x7f1754136e10 2024-04-26T00:21:51.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.835+0000 7f1752ffd640 4 set_mon_vals no callback set 2024-04-26T00:21:51.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.835+0000 7f175a860640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:51.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.835+0000 7f175a860640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:51.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.835+0000 7f175a860640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:51.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.835+0000 7f175a860640 1 -- 172.21.15.37:0/1014521330 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1754075910 con 0x7f1754136e10 2024-04-26T00:21:51.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.835+0000 7f17517fa640 1 -- 172.21.15.37:0/1014521330 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f174401c460 con 0x7f1754136e10 2024-04-26T00:21:51.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.835+0000 7f17517fa640 1 --2- 172.21.15.37:0/1014521330 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1738035e30 0x7f17380382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:51.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.835+0000 7f17517fa640 1 -- 172.21.15.37:0/1014521330 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 3943+0+0 (secure 0 0 0) 0x7f1744049410 con 0x7f1754136e10 2024-04-26T00:21:51.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.835+0000 7f1753fff640 1 --2- 172.21.15.37:0/1014521330 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1738035e30 0x7f17380382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:51.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.835+0000 7f1753fff640 10 monclient: get_auth_request con 0x7f1738035e30 auth_method 0 2024-04-26T00:21:51.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.839+0000 7f1753fff640 1 --2- 172.21.15.37:0/1014521330 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1738035e30 0x7f17380382f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f173c004820 tx=0x7f173c007b30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:51.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.839+0000 7f17517fa640 1 -- 172.21.15.37:0/1014521330 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f1744024780 con 0x7f1754136e10 2024-04-26T00:21:51.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.839+0000 7f17517fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:51.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:51.839+0000 7f17517fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:21:52.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:52.059+0000 7f175a860640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:21:52.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:52.059+0000 7f175a860640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:21:52.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:52.059+0000 7f175a860640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:52.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:52.059+0000 7f175a860640 1 -- 172.21.15.37:0/1014521330 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1754066f20 con 0x7f1754136e10 2024-04-26T00:21:52.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:52.059+0000 7f17517fa640 1 -- 172.21.15.37:0/1014521330 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f17440471a0 con 0x7f1754136e10 2024-04-26T00:21:52.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:52.059+0000 7f17517fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:21:52.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:52.059+0000 7f17517fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:21:52.038 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":18,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:21:52.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:52.063+0000 7f175a860640 1 -- 172.21.15.37:0/1014521330 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1738035e30 msgr2=0x7f17380382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:52.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:52.063+0000 7f175a860640 1 --2- 172.21.15.37:0/1014521330 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1738035e30 0x7f17380382f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f173c004820 tx=0x7f173c007b30 comp rx=0 tx=0).stop 2024-04-26T00:21:52.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:52.063+0000 7f175a860640 10 monclient: shutdown 2024-04-26T00:21:52.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:52.063+0000 7f175a860640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:52.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:52.063+0000 7f175a860640 1 -- 172.21.15.37:0/1014521330 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1754136e10 msgr2=0x7f175413b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:52.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:52.063+0000 7f175a860640 1 --2- 172.21.15.37:0/1014521330 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1754136e10 0x7f175413b910 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f174400c450 tx=0x7f174400c910 comp rx=0 tx=0).stop 2024-04-26T00:21:52.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:52.063+0000 7f175a860640 1 -- 172.21.15.37:0/1014521330 shutdown_connections 2024-04-26T00:21:52.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:52.063+0000 7f175a860640 1 --2- 172.21.15.37:0/1014521330 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1738035e30 0x7f17380382f0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:52.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:52.063+0000 7f175a860640 1 --2- 172.21.15.37:0/1014521330 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1754137900 0x7f175413be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:52.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:52.063+0000 7f175a860640 1 --2- 172.21.15.37:0/1014521330 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1754136e10 0x7f175413b910 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:52.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:52.063+0000 7f175a860640 1 --2- 172.21.15.37:0/1014521330 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f175409c5b0 0x7f17541cf3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:52.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:52.063+0000 7f175a860640 1 -- 172.21.15.37:0/1014521330 >> 172.21.15.37:0/1014521330 conn(0x7f17540a64e0 msgr2=0x7f17540a26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:52.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:52.063+0000 7f175a860640 1 -- 172.21.15.37:0/1014521330 shutdown_connections 2024-04-26T00:21:52.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:52.063+0000 7f175a860640 1 -- 172.21.15.37:0/1014521330 wait complete. 2024-04-26T00:21:55.057 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:21:55.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.210+0000 7f9fffd62640 10 monclient: get_monmap_and_config 2024-04-26T00:21:55.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.210+0000 7f9fffd62640 10 monclient: build_initial_monmap 2024-04-26T00:21:55.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.210+0000 7f9fffd62640 10 monclient: monmap: 2024-04-26T00:21:55.189 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:55.190 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:55.190 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:21:55.214120+0000 2024-04-26T00:21:55.190 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:21:55.214120+0000 2024-04-26T00:21:55.190 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:55.190 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:55.190 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:21:55.190 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:21:55.190 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:21:55.190 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:21:55.190 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:21:55.190 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:55.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.210+0000 7f9fffd62640 1 Processor -- start 2024-04-26T00:21:55.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.210+0000 7f9fffd62640 1 -- start start 2024-04-26T00:21:55.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.210+0000 7f9fffd62640 10 monclient: init 2024-04-26T00:21:55.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.210+0000 7f9fffd62640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:55.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.210+0000 7f9fffd62640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-04-26T00:21:55.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.210+0000 7f9fffd62640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9ff8136e10 0x7f9ff809c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:55.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.210+0000 7f9fffd62640 10 monclient(hunting): picked mon.noname-c con 0x7f9ff8136e10 addr v2:172.21.15.45:3300/0 2024-04-26T00:21:55.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.210+0000 7f9fffd62640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9ff81371f0 addr v1:172.21.15.37:6789/0 2024-04-26T00:21:55.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.210+0000 7f9fffd62640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ff813c340 0x7f9ff813e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:55.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.210+0000 7f9fffd62640 10 monclient(hunting): picked mon.noname-b con 0x7f9ff813c340 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:21:55.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.214+0000 7f9fffd62640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ff809b680 con 0x7f9ff81371f0 2024-04-26T00:21:55.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.214+0000 7f9fffd62640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:55.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.214+0000 7f9fffd62640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f9ff809b8f0 con 0x7f9ff8136e10 2024-04-26T00:21:55.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.214+0000 7f9fffd62640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:55.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.214+0000 7f9fffd62640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9ff809bbd0 con 0x7f9ff813c340 2024-04-26T00:21:55.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.214+0000 7f9fffd62640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:55.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.214+0000 7f9fffd62640 10 monclient(hunting): authenticate will time out at 1096.520042s 2024-04-26T00:21:55.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.214+0000 7f9ffd2fd640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f9ff81371f0 0x7f9ff8137600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:36486/0 (socket says 172.21.15.37:36486) 2024-04-26T00:21:55.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.214+0000 7f9ffd2fd640 1 -- 172.21.15.37:0/581513554 learned_addr learned my addr 172.21.15.37:0/581513554 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:21:55.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.214+0000 7f9ffe2ff640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ff813c340 msgr2=0x7f9ff813e730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:21:55.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.214+0000 7f9ffe2ff640 1 --2- 172.21.15.37:0/581513554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ff813c340 0x7f9ff813e730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:21:55.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.214+0000 7f9ffdafe640 1 --2- 172.21.15.37:0/581513554 >> v2:172.21.15.45:3300/0 conn(0x7f9ff8136e10 0x7f9ff809c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:55.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.214+0000 7f9ffdafe640 10 monclient(hunting): get_auth_request con 0x7f9ff8136e10 auth_method 0 2024-04-26T00:21:55.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.214+0000 7f9ffdafe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:55.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.214+0000 7f9ffdafe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:55.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.214+0000 7f9ffdafe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:55.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.214+0000 7f9ffdafe640 1 -- 172.21.15.37:0/581513554 >> v2:172.21.15.45:3300/0 conn(0x7f9ff8136e10 msgr2=0x7f9ff809c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:21:55.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.214+0000 7f9ffdafe640 1 -- 172.21.15.37:0/581513554 >> v2:172.21.15.45:3300/0 conn(0x7f9ff8136e10 msgr2=0x7f9ff809c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:21:55.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.214+0000 7f9ffdafe640 1 --2- 172.21.15.37:0/581513554 >> v2:172.21.15.45:3300/0 conn(0x7f9ff8136e10 0x7f9ff809c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:21:55.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.214+0000 7f9ffdafe640 1 --2- 172.21.15.37:0/581513554 >> v2:172.21.15.45:3300/0 conn(0x7f9ff8136e10 0x7f9ff809c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:21:55.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.214+0000 7f9ffd2fd640 1 -- 172.21.15.37:0/581513554 >> v1:172.21.15.37:6789/0 conn(0x7f9ff81371f0 legacy=0x7f9ff8137600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:55.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.418+0000 7f9ffe2ff640 1 -- 172.21.15.37:0/581513554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ff813c340 msgr2=0x7f9ff813e730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:21:55.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.418+0000 7f9ffe2ff640 1 --2- 172.21.15.37:0/581513554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ff813c340 0x7f9ff813e730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:21:55.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.418+0000 7f9ffdafe640 1 --2- 172.21.15.37:0/581513554 >> v2:172.21.15.45:3300/0 conn(0x7f9ff8136e10 0x7f9ff809c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:55.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.418+0000 7f9ffdafe640 10 monclient(hunting): get_auth_request con 0x7f9ff8136e10 auth_method 0 2024-04-26T00:21:55.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.418+0000 7f9ffdafe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:55.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.418+0000 7f9ffdafe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:55.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.418+0000 7f9ffdafe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:21:55.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.418+0000 7f9ffdafe640 1 -- 172.21.15.37:0/581513554 >> v2:172.21.15.45:3300/0 conn(0x7f9ff8136e10 msgr2=0x7f9ff809c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:21:55.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.418+0000 7f9ffdafe640 1 -- 172.21.15.37:0/581513554 >> v2:172.21.15.45:3300/0 conn(0x7f9ff8136e10 msgr2=0x7f9ff809c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:21:55.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.418+0000 7f9ffdafe640 1 --2- 172.21.15.37:0/581513554 >> v2:172.21.15.45:3300/0 conn(0x7f9ff8136e10 0x7f9ff809c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:21:55.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.418+0000 7f9ffdafe640 1 --2- 172.21.15.37:0/581513554 >> v2:172.21.15.45:3300/0 conn(0x7f9ff8136e10 0x7f9ff809c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:21:55.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.822+0000 7f9ffe2ff640 1 -- 172.21.15.37:0/581513554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ff813c340 msgr2=0x7f9ff813e730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:21:55.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.822+0000 7f9ffe2ff640 1 --2- 172.21.15.37:0/581513554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ff813c340 0x7f9ff813e730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:21:55.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.822+0000 7f9ffdafe640 1 --2- 172.21.15.37:0/581513554 >> v2:172.21.15.45:3300/0 conn(0x7f9ff8136e10 0x7f9ff809c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:55.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.822+0000 7f9ffdafe640 10 monclient(hunting): get_auth_request con 0x7f9ff8136e10 auth_method 0 2024-04-26T00:21:55.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.822+0000 7f9ffdafe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:55.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.822+0000 7f9ffdafe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:55.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.822+0000 7f9ffdafe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:21:55.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.822+0000 7f9ffdafe640 1 -- 172.21.15.37:0/581513554 >> v2:172.21.15.45:3300/0 conn(0x7f9ff8136e10 msgr2=0x7f9ff809c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:21:55.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.822+0000 7f9ffdafe640 1 -- 172.21.15.37:0/581513554 >> v2:172.21.15.45:3300/0 conn(0x7f9ff8136e10 msgr2=0x7f9ff809c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:21:55.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.822+0000 7f9ffdafe640 1 --2- 172.21.15.37:0/581513554 >> v2:172.21.15.45:3300/0 conn(0x7f9ff8136e10 0x7f9ff809c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:21:55.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:55.822+0000 7f9ffdafe640 1 --2- 172.21.15.37:0/581513554 >> v2:172.21.15.45:3300/0 conn(0x7f9ff8136e10 0x7f9ff809c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:21:56.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:56.626+0000 7f9ffe2ff640 1 -- 172.21.15.37:0/581513554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ff813c340 msgr2=0x7f9ff813e730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:21:56.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:56.626+0000 7f9ffe2ff640 1 --2- 172.21.15.37:0/581513554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ff813c340 0x7f9ff813e730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:21:56.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:56.626+0000 7f9ffdafe640 1 --2- 172.21.15.37:0/581513554 >> v2:172.21.15.45:3300/0 conn(0x7f9ff8136e10 0x7f9ff809c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:56.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:56.626+0000 7f9ffdafe640 10 monclient(hunting): get_auth_request con 0x7f9ff8136e10 auth_method 0 2024-04-26T00:21:56.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:56.626+0000 7f9ffdafe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:56.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:56.626+0000 7f9ffdafe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:56.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:56.626+0000 7f9ffdafe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:21:56.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:56.626+0000 7f9ffdafe640 1 -- 172.21.15.37:0/581513554 >> v2:172.21.15.45:3300/0 conn(0x7f9ff8136e10 msgr2=0x7f9ff809c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:21:56.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:56.626+0000 7f9ffdafe640 1 -- 172.21.15.37:0/581513554 >> v2:172.21.15.45:3300/0 conn(0x7f9ff8136e10 msgr2=0x7f9ff809c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:21:56.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:56.626+0000 7f9ffdafe640 1 --2- 172.21.15.37:0/581513554 >> v2:172.21.15.45:3300/0 conn(0x7f9ff8136e10 0x7f9ff809c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:21:56.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:56.626+0000 7f9ffdafe640 1 --2- 172.21.15.37:0/581513554 >> v2:172.21.15.45:3300/0 conn(0x7f9ff8136e10 0x7f9ff809c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:21:58.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.214+0000 7f9fef7fe640 10 monclient(hunting): tick 2024-04-26T00:21:58.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.214+0000 7f9fef7fe640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:21:58.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.214+0000 7f9fef7fe640 1 monclient(hunting): continuing hunt 2024-04-26T00:21:58.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.214+0000 7f9fef7fe640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:21:58.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.214+0000 7f9fef7fe640 1 -- 172.21.15.37:0/581513554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ff813c340 msgr2=0x7f9ff813e730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:21:58.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.214+0000 7f9fef7fe640 1 --2- 172.21.15.37:0/581513554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ff813c340 0x7f9ff813e730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:58.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.214+0000 7f9fef7fe640 1 -- 172.21.15.37:0/581513554 >> v2:172.21.15.45:3300/0 conn(0x7f9ff8136e10 msgr2=0x7f9ff809c5b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:21:58.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.214+0000 7f9fef7fe640 1 --2- 172.21.15.37:0/581513554 >> v2:172.21.15.45:3300/0 conn(0x7f9ff8136e10 0x7f9ff809c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:58.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.214+0000 7f9fef7fe640 1 -- 172.21.15.37:0/581513554 >> v1:172.21.15.37:6789/0 conn(0x7f9ff81371f0 legacy=0x7f9ff8137600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:58.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.214+0000 7f9fef7fe640 10 monclient: _add_conns ranks=[1,3] 2024-04-26T00:21:58.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.214+0000 7f9fef7fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9fdc003770 addr v1:172.21.15.45:6789/0 2024-04-26T00:21:58.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.214+0000 7f9fef7fe640 1 --2- 172.21.15.37:0/581513554 >> v2:172.21.15.37:3300/0 conn(0x7f9fdc0074a0 0x7f9fdc009960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:58.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.214+0000 7f9fef7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f9fdc0074a0 addr v2:172.21.15.37:3300/0 2024-04-26T00:21:58.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.214+0000 7f9fef7fe640 1 -- 172.21.15.37:0/581513554 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ff80a5d90 con 0x7f9fdc003770 2024-04-26T00:21:58.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.214+0000 7f9fef7fe640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:58.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.214+0000 7f9fef7fe640 1 -- 172.21.15.37:0/581513554 --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f9ff809b8f0 con 0x7f9fdc0074a0 2024-04-26T00:21:58.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.214+0000 7f9fef7fe640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:58.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.214+0000 7f9ffd2fd640 1 --2- 172.21.15.37:0/581513554 >> v2:172.21.15.37:3300/0 conn(0x7f9fdc0074a0 0x7f9fdc009960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:58.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.214+0000 7f9ffd2fd640 10 monclient(hunting): get_auth_request con 0x7f9fdc0074a0 auth_method 0 2024-04-26T00:21:58.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.214+0000 7f9ffd2fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:58.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.214+0000 7f9ffd2fd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:58.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.214+0000 7f9ffd2fd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:58.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.214+0000 7f9ffd2fd640 1 -- 172.21.15.37:0/581513554 >> v2:172.21.15.37:3300/0 conn(0x7f9fdc0074a0 msgr2=0x7f9fdc009960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:21:58.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.214+0000 7f9ffd2fd640 1 -- 172.21.15.37:0/581513554 >> v2:172.21.15.37:3300/0 conn(0x7f9fdc0074a0 msgr2=0x7f9fdc009960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:21:58.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.214+0000 7f9ffd2fd640 1 --2- 172.21.15.37:0/581513554 >> v2:172.21.15.37:3300/0 conn(0x7f9fdc0074a0 0x7f9fdc009960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:21:58.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.214+0000 7f9ffd2fd640 1 --2- 172.21.15.37:0/581513554 >> v2:172.21.15.37:3300/0 conn(0x7f9fdc0074a0 0x7f9fdc009960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:21:58.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.214+0000 7f9ffdafe640 1 -- 172.21.15.37:0/581513554 >> v1:172.21.15.45:6789/0 conn(0x7f9fdc003770 legacy=0x7f9fdc005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:58.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.418+0000 7f9ffd2fd640 1 --2- 172.21.15.37:0/581513554 >> v2:172.21.15.37:3300/0 conn(0x7f9fdc0074a0 0x7f9fdc009960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:58.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.418+0000 7f9ffd2fd640 10 monclient(hunting): get_auth_request con 0x7f9fdc0074a0 auth_method 0 2024-04-26T00:21:58.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.418+0000 7f9ffd2fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:58.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.418+0000 7f9ffd2fd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:58.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.418+0000 7f9ffd2fd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:21:58.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.418+0000 7f9ffd2fd640 1 -- 172.21.15.37:0/581513554 >> v2:172.21.15.37:3300/0 conn(0x7f9fdc0074a0 msgr2=0x7f9fdc009960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:21:58.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.418+0000 7f9ffd2fd640 1 -- 172.21.15.37:0/581513554 >> v2:172.21.15.37:3300/0 conn(0x7f9fdc0074a0 msgr2=0x7f9fdc009960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:21:58.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.418+0000 7f9ffd2fd640 1 --2- 172.21.15.37:0/581513554 >> v2:172.21.15.37:3300/0 conn(0x7f9fdc0074a0 0x7f9fdc009960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:21:58.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.418+0000 7f9ffd2fd640 1 --2- 172.21.15.37:0/581513554 >> v2:172.21.15.37:3300/0 conn(0x7f9fdc0074a0 0x7f9fdc009960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:21:58.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.822+0000 7f9ffd2fd640 1 --2- 172.21.15.37:0/581513554 >> v2:172.21.15.37:3300/0 conn(0x7f9fdc0074a0 0x7f9fdc009960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:58.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.822+0000 7f9ffd2fd640 10 monclient(hunting): get_auth_request con 0x7f9fdc0074a0 auth_method 0 2024-04-26T00:21:58.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.822+0000 7f9ffd2fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:58.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.822+0000 7f9ffd2fd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:58.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.822+0000 7f9ffd2fd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:21:58.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.822+0000 7f9ffd2fd640 1 -- 172.21.15.37:0/581513554 >> v2:172.21.15.37:3300/0 conn(0x7f9fdc0074a0 msgr2=0x7f9fdc009960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:21:58.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.822+0000 7f9ffd2fd640 1 -- 172.21.15.37:0/581513554 >> v2:172.21.15.37:3300/0 conn(0x7f9fdc0074a0 msgr2=0x7f9fdc009960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:21:58.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.822+0000 7f9ffd2fd640 1 --2- 172.21.15.37:0/581513554 >> v2:172.21.15.37:3300/0 conn(0x7f9fdc0074a0 0x7f9fdc009960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:21:58.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.822+0000 7f9ffd2fd640 1 --2- 172.21.15.37:0/581513554 >> v2:172.21.15.37:3300/0 conn(0x7f9fdc0074a0 0x7f9fdc009960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:21:58.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.994+0000 7f9ffcafc640 1 -- 172.21.15.37:0/581513554 <== mon.1 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4013847407 0 0) 0x7f9ff80a5d90 con 0x7f9fdc003770 2024-04-26T00:21:58.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.994+0000 7f9ffcafc640 10 monclient(hunting): _negotiate 2024-04-26T00:21:58.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.994+0000 7f9ffcafc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:58.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.994+0000 7f9ffcafc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:58.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.994+0000 7f9ffcafc640 10 monclient(hunting): my global_id is 24151 2024-04-26T00:21:58.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.998+0000 7f9ffcafc640 1 -- 172.21.15.37:0/581513554 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9fe4003530 con 0x7f9fdc003770 2024-04-26T00:21:58.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.998+0000 7f9ffcafc640 1 -- 172.21.15.37:0/581513554 <== mon.1 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2140145146 0 0) 0x7f9fe4003530 con 0x7f9fdc003770 2024-04-26T00:21:58.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.998+0000 7f9ffcafc640 1 -- 172.21.15.37:0/581513554 >> v2:172.21.15.37:3300/0 conn(0x7f9fdc0074a0 msgr2=0x7f9fdc009960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:21:58.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.998+0000 7f9ffcafc640 1 --2- 172.21.15.37:0/581513554 >> v2:172.21.15.37:3300/0 conn(0x7f9fdc0074a0 0x7f9fdc009960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:58.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:58.998+0000 7f9ffcafc640 10 monclient: _finish_hunting 0 2024-04-26T00:21:58.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.002+0000 7f9ffcafc640 1 monclient: found mon.noname-c-legacy 2024-04-26T00:21:58.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.002+0000 7f9ffcafc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:58.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.002+0000 7f9ffcafc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.45:6789/0 2024-04-26T00:21:58.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.002+0000 7f9ffcafc640 1 -- 172.21.15.37:0/581513554 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fdc00b050 con 0x7f9fdc003770 2024-04-26T00:21:58.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.002+0000 7f9ffcafc640 10 monclient: _finish_auth 0 2024-04-26T00:21:58.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.002+0000 7f9ffcafc640 10 monclient: _check_auth_tickets 2024-04-26T00:21:58.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.002+0000 7f9ffcafc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:58.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.002+0000 7f9ffcafc640 1 -- 172.21.15.37:0/581513554 <== mon.1 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 866326386 0 0) 0x7f9ff4009d10 con 0x7f9fdc003770 2024-04-26T00:21:58.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.002+0000 7f9fffd62640 5 monclient: authenticate success, global_id 24151 2024-04-26T00:21:58.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.002+0000 7f9fffd62640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:21:58.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.002+0000 7f9ffcafc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:58.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.002+0000 7f9ffcafc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-04-26T00:21:58.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.002+0000 7f9ffcafc640 10 monclient: dump: 2024-04-26T00:21:58.984 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:58.984 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:58.984 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:58.984 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:58.984 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:58.984 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:58.984 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:58.984 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:58.984 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:58.984 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:58.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.002+0000 7f9ffcafc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] but i'm connected to v1:172.21.15.45:6789/0, reconnecting 2024-04-26T00:21:58.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.002+0000 7f9ffcafc640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:58.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.002+0000 7f9ffcafc640 1 -- 172.21.15.37:0/581513554 >> v1:172.21.15.45:6789/0 conn(0x7f9fdc003770 legacy=0x7f9fdc005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:58.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.002+0000 7f9ffcafc640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:21:58.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.002+0000 7f9ffcafc640 1 --2- 172.21.15.37:0/581513554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fe40039c0 0x7f9fe4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:58.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.002+0000 7f9ffcafc640 10 monclient(hunting): picked mon.b con 0x7f9fe40039c0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:21:58.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.002+0000 7f9ffcafc640 1 --2- 172.21.15.37:0/581513554 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9fe4004810 0x7f9fe4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:58.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.002+0000 7f9ffcafc640 10 monclient(hunting): picked mon.a con 0x7f9fe4004810 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:21:58.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.002+0000 7f9ffcafc640 1 --2- 172.21.15.37:0/581513554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ff813c340 0x7f9fe4005280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:58.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.002+0000 7f9ffcafc640 10 monclient(hunting): picked mon.c con 0x7f9ff813c340 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:21:58.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.002+0000 7f9ffcafc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:58.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.002+0000 7f9ffcafc640 1 -- 172.21.15.37:0/581513554 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff809b8f0 con 0x7f9fe4004810 2024-04-26T00:21:58.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.002+0000 7f9ffe2ff640 1 -- 172.21.15.37:0/581513554 reap_dead start 2024-04-26T00:21:58.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.002+0000 7f9ffe2ff640 1 -- 172.21.15.37:0/581513554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ff813c340 msgr2=0x7f9fe4005280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:21:58.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.002+0000 7f9ffcafc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:58.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.006+0000 7f9ffcafc640 1 -- 172.21.15.37:0/581513554 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe400bbc0 con 0x7f9fe40039c0 2024-04-26T00:21:58.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.002+0000 7f9ffdafe640 1 --2- 172.21.15.37:0/581513554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fe40039c0 0x7f9fe4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:58.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.006+0000 7f9ffcafc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:58.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.006+0000 7f9ffe2ff640 1 --2- 172.21.15.37:0/581513554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ff813c340 0x7f9fe4005280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:21:58.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.006+0000 7f9ffcafc640 1 -- 172.21.15.37:0/581513554 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9fe400bdc0 con 0x7f9ff813c340 2024-04-26T00:21:58.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.006+0000 7f9ffcafc640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:58.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.006+0000 7f9ffcafc640 10 monclient(hunting): _finish_auth 0 2024-04-26T00:21:58.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.006+0000 7f9fffd62640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-26T00:21:58.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.006+0000 7f9ffdafe640 10 monclient(hunting): get_auth_request con 0x7f9fe40039c0 auth_method 0 2024-04-26T00:21:58.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.006+0000 7f9ffdafe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:58.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.006+0000 7f9ffdafe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:58.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.006+0000 7f9ffdafe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:21:58.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.002+0000 7f9ffd2fd640 1 --2- 172.21.15.37:0/581513554 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9fe4004810 0x7f9fe4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:58.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.006+0000 7f9ffd2fd640 10 monclient(hunting): get_auth_request con 0x7f9fe4004810 auth_method 0 2024-04-26T00:21:58.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.006+0000 7f9ffd2fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:58.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.006+0000 7f9ffd2fd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:58.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.006+0000 7f9ffd2fd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:21:58.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.006+0000 7f9ffdafe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:58.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.006+0000 7f9ffdafe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:58.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.006+0000 7f9ffdafe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:21:58.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.006+0000 7f9ffd2fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:58.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.006+0000 7f9ffd2fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:58.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.006+0000 7f9ffd2fd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:21:58.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.006+0000 7f9ffdafe640 10 monclient(hunting): handle_auth_done global_id 24151 payload 293 2024-04-26T00:21:58.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.006+0000 7f9ffdafe640 1 -- 172.21.15.37:0/581513554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ff813c340 msgr2=0x7f9fe4005280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:21:58.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.006+0000 7f9ffdafe640 1 --2- 172.21.15.37:0/581513554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ff813c340 0x7f9fe4005280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:58.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.006+0000 7f9ffdafe640 1 -- 172.21.15.37:0/581513554 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9fe4004810 msgr2=0x7f9fe4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:21:58.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.006+0000 7f9ffdafe640 1 --2- 172.21.15.37:0/581513554 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9fe4004810 0x7f9fe4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:58.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.006+0000 7f9ffdafe640 10 monclient: _finish_hunting 0 2024-04-26T00:21:58.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.006+0000 7f9ffdafe640 1 monclient: found mon.b 2024-04-26T00:21:58.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.006+0000 7f9ffdafe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:58.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.006+0000 7f9ffdafe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.45:3300/0 2024-04-26T00:21:58.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.006+0000 7f9ffdafe640 1 -- 172.21.15.37:0/581513554 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fe4003530 con 0x7f9fe40039c0 2024-04-26T00:21:58.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.006+0000 7f9ffdafe640 10 monclient: _finish_auth 0 2024-04-26T00:21:58.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.006+0000 7f9ffdafe640 10 monclient: _check_auth_tickets 2024-04-26T00:21:58.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.006+0000 7f9ffdafe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:58.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.010+0000 7f9ffd2fd640 1 --2- 172.21.15.37:0/581513554 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9fe4004810 0x7f9fe4004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:21:58.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.018+0000 7f9ffdafe640 1 -- 172.21.15.37:0/581513554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fe40039c0 msgr2=0x7f9fe4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:58.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.018+0000 7f9ffdafe640 1 --2- 172.21.15.37:0/581513554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fe40039c0 0x7f9fe4003dc0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9ff4009610 tx=0x7f9ff401dbd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c044606691241d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:58.998 INFO:tasks.rados.rados.0.smithi045.stdout:320: done (1 left) 2024-04-26T00:21:58.998 INFO:tasks.rados.rados.0.smithi045.stdout:321: copy_from oid 24 from oid 25 current snap is 31 2024-04-26T00:21:58.998 INFO:tasks.rados.rados.0.smithi045.stdout:319: finishing copy_from racing read to smithi04526579-5 2024-04-26T00:21:58.998 INFO:tasks.rados.rados.0.smithi045.stdout:319: finishing copy_from to smithi04526579-5 2024-04-26T00:21:58.999 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 42 (ObjNum 52 snap 1 seq_num 52) dirty exists 2024-04-26T00:21:58.999 INFO:tasks.rados.rados.0.smithi045.stdout:319: done (1 left) 2024-04-26T00:21:58.999 INFO:tasks.rados.rados.0.smithi045.stdout:322: copy_from oid 35 from oid 11 current snap is 31 2024-04-26T00:21:58.999 INFO:tasks.rados.rados.0.smithi045.stdout:323: rollback oid 37 current snap is 31 2024-04-26T00:21:58.999 INFO:tasks.rados.rados.0.smithi045.stdout:323: done (2 left) 2024-04-26T00:21:58.999 INFO:tasks.rados.rados.0.smithi045.stdout:324: delete oid 39 current snap is 31 2024-04-26T00:21:59.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.022+0000 7f9ffcafc640 1 -- 172.21.15.37:0/581513554 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ff40ac070 con 0x7f9fe40039c0 2024-04-26T00:21:59.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.022+0000 7f9ffcafc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:59.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.022+0000 7f9ffcafc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:21:59.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.022+0000 7f9ffcafc640 10 monclient: dump: 2024-04-26T00:21:59.002 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:59.002 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:59.002 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:59.002 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:59.002 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:59.002 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:59.002 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:59.003 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:59.003 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:59.003 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:59.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.022+0000 7f9ffcafc640 1 -- 172.21.15.37:0/581513554 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ff40a6a20 con 0x7f9fe40039c0 2024-04-26T00:21:59.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.022+0000 7f9ffcafc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:59.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.022+0000 7f9ffcafc640 1 -- 172.21.15.37:0/581513554 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ff4004040 con 0x7f9fe40039c0 2024-04-26T00:21:59.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.022+0000 7f9ffcafc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:59.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.022+0000 7f9ffcafc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:21:59.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.022+0000 7f9ffcafc640 10 monclient: dump: 2024-04-26T00:21:59.004 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:59.004 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:59.004 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:59.004 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:59.004 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:59.004 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:59.004 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:59.004 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:59.004 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:59.004 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:59.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.022+0000 7f9fffd62640 10 monclient: get_monmap_and_config success 2024-04-26T00:21:59.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.022+0000 7f9fffd62640 4 set_mon_vals no callback set 2024-04-26T00:21:59.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.026+0000 7f9fffd62640 10 monclient: shutdown 2024-04-26T00:21:59.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.026+0000 7f9fffd62640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:59.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.026+0000 7f9fffd62640 1 -- 172.21.15.37:0/581513554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fe40039c0 msgr2=0x7f9fe4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:59.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.026+0000 7f9fffd62640 1 --2- 172.21.15.37:0/581513554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fe40039c0 0x7f9fe4003dc0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9ff4009610 tx=0x7f9ff401dbd0 comp rx=0 tx=0).stop 2024-04-26T00:21:59.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.026+0000 7f9fffd62640 1 -- 172.21.15.37:0/581513554 shutdown_connections 2024-04-26T00:21:59.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.026+0000 7f9fffd62640 1 --2- 172.21.15.37:0/581513554 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9fe4004810 0x7f9fe4004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:59.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.026+0000 7f9fffd62640 1 --2- 172.21.15.37:0/581513554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fe40039c0 0x7f9fe4003dc0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:59.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.026+0000 7f9fffd62640 1 --2- 172.21.15.37:0/581513554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ff813c340 0x7f9fe4005280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:59.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.026+0000 7f9fffd62640 1 -- 172.21.15.37:0/581513554 >> 172.21.15.37:0/581513554 conn(0x7f9ff80a64e0 msgr2=0x7f9ff80a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:59.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.026+0000 7f9fffd62640 1 -- 172.21.15.37:0/581513554 shutdown_connections 2024-04-26T00:21:59.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.026+0000 7f9fffd62640 1 -- 172.21.15.37:0/581513554 wait complete. 2024-04-26T00:21:59.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.026+0000 7f9fffd62640 10 monclient: build_initial_monmap 2024-04-26T00:21:59.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.026+0000 7f9fffd62640 10 monclient: monmap: 2024-04-26T00:21:59.009 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:59.009 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:21:59.009 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:21:59.009 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:21:59.009 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:59.009 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:21:59.009 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:21:59.009 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:21:59.009 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:21:59.009 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:59.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.026+0000 7f9fffd62640 1 Processor -- start 2024-04-26T00:21:59.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.026+0000 7f9fffd62640 1 -- start start 2024-04-26T00:21:59.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.026+0000 7f9fffd62640 10 monclient: init 2024-04-26T00:21:59.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.026+0000 7f9fffd62640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:59.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.026+0000 7f9fffd62640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:21:59.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.026+0000 7f9fffd62640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fe40039c0 0x7f9ff81cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:59.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.026+0000 7f9fffd62640 10 monclient(hunting): picked mon.noname-b con 0x7f9fe40039c0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:21:59.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.026+0000 7f9fffd62640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9fe4004810 0x7f9ff8134c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:59.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.026+0000 7f9fffd62640 10 monclient(hunting): picked mon.noname-a con 0x7f9fe4004810 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:21:59.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.026+0000 7f9fffd62640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ff813c340 0x7f9ff81351b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:59.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.026+0000 7f9fffd62640 10 monclient(hunting): picked mon.noname-c con 0x7f9ff813c340 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:21:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.026+0000 7f9fffd62640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.026+0000 7f9fffd62640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff8136930 con 0x7f9fe4004810 2024-04-26T00:21:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.026+0000 7f9fffd62640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.026+0000 7f9fffd62640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff81367b0 con 0x7f9fe40039c0 2024-04-26T00:21:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.026+0000 7f9fffd62640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.026+0000 7f9fffd62640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9ff8136ab0 con 0x7f9ff813c340 2024-04-26T00:21:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.026+0000 7f9fffd62640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.026+0000 7f9fffd62640 10 monclient(hunting): authenticate will time out at 1100.335572s 2024-04-26T00:21:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.026+0000 7f9ffe2ff640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ff813c340 msgr2=0x7f9ff81351b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:21:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.026+0000 7f9ffe2ff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ff813c340 0x7f9ff81351b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:21:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.030+0000 7f9ffd2fd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9fe4004810 0x7f9ff8134c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.030+0000 7f9ffd2fd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9fe4004810 0x7f9ff8134c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:41714/0 (socket says 172.21.15.37:41714) 2024-04-26T00:21:59.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.030+0000 7f9ffd2fd640 1 -- 172.21.15.37:0/1615926359 learned_addr learned my addr 172.21.15.37:0/1615926359 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:21:59.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.030+0000 7f9ffd2fd640 10 monclient(hunting): get_auth_request con 0x7f9fe4004810 auth_method 0 2024-04-26T00:21:59.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.030+0000 7f9ffdafe640 1 --2- 172.21.15.37:0/1615926359 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fe40039c0 0x7f9ff81cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:59.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.030+0000 7f9ffd2fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:59.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.030+0000 7f9ffd2fd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:59.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.030+0000 7f9ffd2fd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:59.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.030+0000 7f9ffdafe640 10 monclient(hunting): get_auth_request con 0x7f9fe40039c0 auth_method 0 2024-04-26T00:21:59.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.030+0000 7f9ffdafe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:59.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.030+0000 7f9ffdafe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:59.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.030+0000 7f9ffdafe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:59.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.030+0000 7f9ffd2fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:59.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.030+0000 7f9ffd2fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:59.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.030+0000 7f9ffd2fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:59.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.030+0000 7f9ffdafe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:59.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.030+0000 7f9ffdafe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:59.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.030+0000 7f9ffdafe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:59.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.030+0000 7f9ffd2fd640 10 monclient(hunting): handle_auth_done global_id 4590 payload 824 2024-04-26T00:21:59.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.030+0000 7f9ffd2fd640 1 -- 172.21.15.37:0/1615926359 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ff813c340 msgr2=0x7f9ff81351b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:21:59.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.030+0000 7f9ffd2fd640 1 --2- 172.21.15.37:0/1615926359 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ff813c340 0x7f9ff81351b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:59.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.030+0000 7f9ffd2fd640 1 -- 172.21.15.37:0/1615926359 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fe40039c0 msgr2=0x7f9ff81cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:59.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.030+0000 7f9ffd2fd640 1 --2- 172.21.15.37:0/1615926359 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fe40039c0 0x7f9ff81cf7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:59.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.030+0000 7f9ffd2fd640 10 monclient: _finish_hunting 0 2024-04-26T00:21:59.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.030+0000 7f9ffd2fd640 1 monclient: found mon.noname-a 2024-04-26T00:21:59.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.030+0000 7f9ffd2fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:59.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.030+0000 7f9ffd2fd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:21:59.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.030+0000 7f9ffd2fd640 1 -- 172.21.15.37:0/1615926359 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ff8125660 con 0x7f9fe4004810 2024-04-26T00:21:59.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.030+0000 7f9ffdafe640 1 --2- 172.21.15.37:0/1615926359 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fe40039c0 0x7f9ff81cf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:21:59.015 INFO:tasks.rados.rados.0.smithi045.stdout:324: done (2 left) 2024-04-26T00:21:59.015 INFO:tasks.rados.rados.0.smithi045.stdout:325: snap_create 2024-04-26T00:21:59.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.030+0000 7f9ffd2fd640 1 -- 172.21.15.37:0/1615926359 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9fe4004810 msgr2=0x7f9ff8134c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:59.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.030+0000 7f9ffd2fd640 1 --2- 172.21.15.37:0/1615926359 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9fe4004810 0x7f9ff8134c70 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9fe8015600 tx=0x7f9fe8015ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:59.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.034+0000 7f9fee7fc640 1 -- 172.21.15.37:0/1615926359 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe801b020 con 0x7f9fe4004810 2024-04-26T00:21:59.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.034+0000 7f9fee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:59.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.034+0000 7f9fee7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:21:59.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.034+0000 7f9fee7fc640 10 monclient: dump: 2024-04-26T00:21:59.017 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:59.017 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:59.017 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:59.017 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:59.017 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:59.017 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:59.018 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:59.018 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:59.018 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:59.018 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:59.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.034+0000 7f9fee7fc640 10 monclient: _finish_auth 0 2024-04-26T00:21:59.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.034+0000 7f9fee7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:21:59.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.034+0000 7f9fee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:59.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.034+0000 7f9fee7fc640 1 -- 172.21.15.37:0/1615926359 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fe8007490 con 0x7f9fe4004810 2024-04-26T00:21:59.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.034+0000 7f9fee7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:59.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.034+0000 7f9fee7fc640 1 -- 172.21.15.37:0/1615926359 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe8017520 con 0x7f9fe4004810 2024-04-26T00:21:59.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.034+0000 7f9fee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:59.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.034+0000 7f9fee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:21:59.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.034+0000 7f9fee7fc640 10 monclient: dump: 2024-04-26T00:21:59.019 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:59.019 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:59.019 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:59.019 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:59.019 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:59.019 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:59.019 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:59.019 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:59.019 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:59.019 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:59.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.034+0000 7f9fffd62640 5 monclient: authenticate success, global_id 4590 2024-04-26T00:21:59.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.034+0000 7f9fffd62640 10 monclient: _renew_subs 2024-04-26T00:21:59.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.034+0000 7f9fffd62640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.034+0000 7f9fffd62640 1 -- 172.21.15.37:0/1615926359 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ff81240f0 con 0x7f9fe4004810 2024-04-26T00:21:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.034+0000 7f9fffd62640 10 monclient: _renew_subs 2024-04-26T00:21:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.034+0000 7f9fffd62640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.034+0000 7f9fffd62640 1 -- 172.21.15.37:0/1615926359 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ff80a8890 con 0x7f9fe4004810 2024-04-26T00:21:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.034+0000 7f9fef7fe640 4 set_mon_vals no callback set 2024-04-26T00:21:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.034+0000 7f9fee7fc640 1 -- 172.21.15.37:0/1615926359 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9fe801e020 con 0x7f9fe4004810 2024-04-26T00:21:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.034+0000 7f9fffd62640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.034+0000 7f9fee7fc640 1 --2- 172.21.15.37:0/1615926359 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9fdc036810 0x7f9fdc038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.034+0000 7f9fee7fc640 1 -- 172.21.15.37:0/1615926359 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 3959+0+0 (secure 0 0 0) 0x7f9fe8049610 con 0x7f9fe4004810 2024-04-26T00:21:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.034+0000 7f9ffdafe640 1 --2- 172.21.15.37:0/1615926359 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9fdc036810 0x7f9fdc038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.034+0000 7f9fffd62640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.034+0000 7f9fffd62640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.034+0000 7f9fffd62640 1 -- 172.21.15.37:0/1615926359 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fe0005090 con 0x7f9fe4004810 2024-04-26T00:21:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.034+0000 7f9ffdafe640 10 monclient: get_auth_request con 0x7f9fdc036810 auth_method 0 2024-04-26T00:21:59.021 INFO:tasks.rados.rados.0.smithi045.stdout:322: finishing copy_from racing read to smithi04526579-35 2024-04-26T00:21:59.021 INFO:tasks.rados.rados.0.smithi045.stdout:321: finishing copy_from racing read to smithi04526579-24 2024-04-26T00:21:59.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.038+0000 7f9ffdafe640 1 --2- 172.21.15.37:0/1615926359 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9fdc036810 0x7f9fdc038cd0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9ff40ae000 tx=0x7f9ff4004ac0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:59.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.042+0000 7f9fee7fc640 1 -- 172.21.15.37:0/1615926359 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9fe8025020 con 0x7f9fe4004810 2024-04-26T00:21:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.042+0000 7f9fee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.042+0000 7f9fee7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:21:59.023 INFO:tasks.rados.rados.0.smithi045.stdout:321: finishing copy_from to smithi04526579-24 2024-04-26T00:21:59.023 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 118 (ObjNum 72 snap 11 seq_num 72) dirty exists 2024-04-26T00:21:59.023 INFO:tasks.rados.rados.0.smithi045.stdout:322: finishing copy_from to smithi04526579-35 2024-04-26T00:21:59.023 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 65 (ObjNum 138 snap 31 seq_num 138) dirty exists 2024-04-26T00:21:59.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.262+0000 7f9fffd62640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:21:59.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.262+0000 7f9fffd62640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:21:59.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.262+0000 7f9fffd62640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:59.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.262+0000 7f9fffd62640 1 -- 172.21.15.37:0/1615926359 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9fe0005380 con 0x7f9fe4004810 2024-04-26T00:21:59.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.266+0000 7f9fee7fc640 1 -- 172.21.15.37:0/1615926359 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7f9fe8013530 con 0x7f9fe4004810 2024-04-26T00:21:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.266+0000 7f9fee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:21:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.266+0000 7f9fee7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:21:59.244 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":22,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:21:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.266+0000 7f9fffd62640 1 -- 172.21.15.37:0/1615926359 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9fdc036810 msgr2=0x7f9fdc038cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.266+0000 7f9fffd62640 1 --2- 172.21.15.37:0/1615926359 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9fdc036810 0x7f9fdc038cd0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9ff40ae000 tx=0x7f9ff4004ac0 comp rx=0 tx=0).stop 2024-04-26T00:21:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.266+0000 7f9fffd62640 10 monclient: shutdown 2024-04-26T00:21:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.266+0000 7f9fffd62640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.266+0000 7f9fffd62640 1 -- 172.21.15.37:0/1615926359 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9fe4004810 msgr2=0x7f9ff8134c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.266+0000 7f9fffd62640 1 --2- 172.21.15.37:0/1615926359 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9fe4004810 0x7f9ff8134c70 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9fe8015600 tx=0x7f9fe8015ac0 comp rx=0 tx=0).stop 2024-04-26T00:21:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.266+0000 7f9fffd62640 1 -- 172.21.15.37:0/1615926359 shutdown_connections 2024-04-26T00:21:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.266+0000 7f9fffd62640 1 --2- 172.21.15.37:0/1615926359 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9fdc036810 0x7f9fdc038cd0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.266+0000 7f9fffd62640 1 --2- 172.21.15.37:0/1615926359 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ff813c340 0x7f9ff81351b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.266+0000 7f9fffd62640 1 --2- 172.21.15.37:0/1615926359 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9fe4004810 0x7f9ff8134c70 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.266+0000 7f9fffd62640 1 --2- 172.21.15.37:0/1615926359 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fe40039c0 0x7f9ff81cf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:59.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.266+0000 7f9fffd62640 1 -- 172.21.15.37:0/1615926359 >> 172.21.15.37:0/1615926359 conn(0x7f9ff80a64e0 msgr2=0x7f9ff80a8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:59.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.266+0000 7f9fffd62640 1 -- 172.21.15.37:0/1615926359 shutdown_connections 2024-04-26T00:21:59.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.266+0000 7f9fffd62640 1 -- 172.21.15.37:0/1615926359 wait complete. 2024-04-26T00:21:59.262 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-26T00:21:59.263 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-04-26T00:21:59.263 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:21:59.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.414+0000 7f07206d8640 10 monclient: get_monmap_and_config 2024-04-26T00:21:59.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.414+0000 7f07206d8640 10 monclient: build_initial_monmap 2024-04-26T00:21:59.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.414+0000 7f07206d8640 10 monclient: monmap: 2024-04-26T00:21:59.394 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:59.394 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:59.394 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:21:59.418757+0000 2024-04-26T00:21:59.394 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:21:59.418757+0000 2024-04-26T00:21:59.394 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:59.394 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:59.395 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:21:59.395 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:21:59.395 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:21:59.395 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:21:59.395 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:21:59.395 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:59.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.414+0000 7f07206d8640 1 Processor -- start 2024-04-26T00:21:59.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.414+0000 7f07206d8640 1 -- start start 2024-04-26T00:21:59.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.414+0000 7f07206d8640 10 monclient: init 2024-04-26T00:21:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.418+0000 7f07206d8640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.418+0000 7f07206d8640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-04-26T00:21:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.418+0000 7f07206d8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f07180a0300 addr v1:172.21.15.37:6789/0 2024-04-26T00:21:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.418+0000 7f07206d8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f071809fc50 addr v1:172.21.15.45:6789/0 2024-04-26T00:21:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.418+0000 7f07206d8640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f071813b5e0 0x7f071813d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.418+0000 7f07206d8640 10 monclient(hunting): picked mon.noname-a con 0x7f071813b5e0 addr v2:172.21.15.37:3300/0 2024-04-26T00:21:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.418+0000 7f07206d8640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f071809c190 con 0x7f07180a0300 2024-04-26T00:21:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.418+0000 7f07206d8640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07180a1110 con 0x7f071809fc50 2024-04-26T00:21:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.418+0000 7f07206d8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.418+0000 7f07206d8640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f07180a16d0 con 0x7f071813b5e0 2024-04-26T00:21:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.418+0000 7f07206d8640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.418+0000 7f07206d8640 10 monclient(hunting): authenticate will time out at 1100.724387s 2024-04-26T00:21:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.418+0000 7f071ec75640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f071813b5e0 0x7f071813d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:59.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.418+0000 7f071ec75640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f071813b5e0 0x7f071813d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:41726/0 (socket says 172.21.15.37:41726) 2024-04-26T00:21:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.418+0000 7f071ec75640 1 -- 172.21.15.37:0/1284513854 learned_addr learned my addr 172.21.15.37:0/1284513854 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:21:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.418+0000 7f071ec75640 10 monclient(hunting): get_auth_request con 0x7f071813b5e0 auth_method 0 2024-04-26T00:21:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.418+0000 7f071ec75640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.418+0000 7f071ec75640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.418+0000 7f071ec75640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.418+0000 7f071ec75640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.418+0000 7f071ec75640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.418+0000 7f071ec75640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.418+0000 7f071e474640 1 -- 172.21.15.37:0/1284513854 >> v1:172.21.15.37:6789/0 conn(0x7f07180a0300 legacy=0x7f07180a06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.418+0000 7f071ec75640 10 monclient(hunting): handle_auth_done global_id 4593 payload 274 2024-04-26T00:21:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.418+0000 7f071ec75640 1 -- 172.21.15.37:0/1284513854 >> v1:172.21.15.45:6789/0 conn(0x7f071809fc50 legacy=0x7f0718139ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:21:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.418+0000 7f071ec75640 1 -- 172.21.15.37:0/1284513854 >> v1:172.21.15.37:6789/0 conn(0x7f07180a0300 legacy=0x7f07180a06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.418+0000 7f071ec75640 10 monclient: _finish_hunting 0 2024-04-26T00:21:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.418+0000 7f071ec75640 1 monclient: found mon.noname-a 2024-04-26T00:21:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.418+0000 7f071ec75640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.418+0000 7f071ec75640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:21:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.418+0000 7f071ec75640 1 -- 172.21.15.37:0/1284513854 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f071809f2a0 con 0x7f071813b5e0 2024-04-26T00:21:59.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.422+0000 7f071ec75640 1 -- 172.21.15.37:0/1284513854 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f071813b5e0 msgr2=0x7f071813d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:59.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.422+0000 7f071ec75640 1 --2- 172.21.15.37:0/1284513854 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f071813b5e0 0x7f071813d9d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f071809f4d0 tx=0x7f071001c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90a282c3f7e6788f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:59.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.422+0000 7f071d472640 1 -- 172.21.15.37:0/1284513854 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07100a56e0 con 0x7f071813b5e0 2024-04-26T00:21:59.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.422+0000 7f071d472640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:59.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.422+0000 7f071d472640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:21:59.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.422+0000 7f071d472640 10 monclient: dump: 2024-04-26T00:21:59.404 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:59.404 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:59.404 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:59.404 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:59.404 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:59.404 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:59.404 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:59.404 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:59.404 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:59.404 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:59.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.422+0000 7f071d472640 10 monclient: _finish_auth 0 2024-04-26T00:21:59.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.422+0000 7f071d472640 10 monclient: _check_auth_tickets 2024-04-26T00:21:59.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.422+0000 7f071d472640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:59.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.422+0000 7f07206d8640 5 monclient: authenticate success, global_id 4593 2024-04-26T00:21:59.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.422+0000 7f07206d8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:21:59.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.422+0000 7f071d472640 1 -- 172.21.15.37:0/1284513854 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07100a5880 con 0x7f071813b5e0 2024-04-26T00:21:59.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.422+0000 7f071d472640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:59.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.422+0000 7f071d472640 1 -- 172.21.15.37:0/1284513854 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07100a5c50 con 0x7f071813b5e0 2024-04-26T00:21:59.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.422+0000 7f07206d8640 10 monclient: get_monmap_and_config success 2024-04-26T00:21:59.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.422+0000 7f07206d8640 4 set_mon_vals no callback set 2024-04-26T00:21:59.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.422+0000 7f071d472640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:59.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.422+0000 7f071d472640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:21:59.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.422+0000 7f071d472640 10 monclient: dump: 2024-04-26T00:21:59.405 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:59.405 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:59.405 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:59.406 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:59.406 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:59.406 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:59.406 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:59.406 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:59.406 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:59.406 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:59.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f07206d8640 10 monclient: shutdown 2024-04-26T00:21:59.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f07206d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:59.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f07206d8640 1 -- 172.21.15.37:0/1284513854 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f071813b5e0 msgr2=0x7f071813d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:59.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f07206d8640 1 --2- 172.21.15.37:0/1284513854 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f071813b5e0 0x7f071813d9d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f071809f4d0 tx=0x7f071001c910 comp rx=0 tx=0).stop 2024-04-26T00:21:59.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f07206d8640 1 -- 172.21.15.37:0/1284513854 shutdown_connections 2024-04-26T00:21:59.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f07206d8640 1 --2- 172.21.15.37:0/1284513854 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f071813b5e0 0x7f071813d9d0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:59.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f07206d8640 1 -- 172.21.15.37:0/1284513854 >> 172.21.15.37:0/1284513854 conn(0x7f07180a1d50 msgr2=0x7f07180a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:59.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f07206d8640 1 -- 172.21.15.37:0/1284513854 shutdown_connections 2024-04-26T00:21:59.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f07206d8640 1 -- 172.21.15.37:0/1284513854 wait complete. 2024-04-26T00:21:59.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f07206d8640 10 monclient: build_initial_monmap 2024-04-26T00:21:59.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f07206d8640 10 monclient: monmap: 2024-04-26T00:21:59.411 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:59.411 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:21:59.411 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:21:59.411 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:21:59.412 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:59.412 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:21:59.412 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:21:59.412 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:21:59.412 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:21:59.412 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:59.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f07206d8640 1 Processor -- start 2024-04-26T00:21:59.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f07206d8640 1 -- start start 2024-04-26T00:21:59.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f07206d8640 10 monclient: init 2024-04-26T00:21:59.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f07206d8640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:59.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f07206d8640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:21:59.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f07206d8640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f071809fc50 0x7f071811b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:59.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f07206d8640 10 monclient(hunting): picked mon.noname-a con 0x7f071809fc50 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:21:59.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f07206d8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07180a0300 0x7f071811bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:59.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f07206d8640 10 monclient(hunting): picked mon.noname-b con 0x7f07180a0300 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:21:59.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f07206d8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f071813b5e0 0x7f071811c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:59.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f07206d8640 10 monclient(hunting): picked mon.noname-c con 0x7f071813b5e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:21:59.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f07206d8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:59.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f07206d8640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f0718140ac0 con 0x7f071809fc50 2024-04-26T00:21:59.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f07206d8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:59.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f07206d8640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0718140940 con 0x7f07180a0300 2024-04-26T00:21:59.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f07206d8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:59.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f07206d8640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0718140c40 con 0x7f071813b5e0 2024-04-26T00:21:59.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f07206d8640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:59.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f07206d8640 10 monclient(hunting): authenticate will time out at 1100.734514s 2024-04-26T00:21:59.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f071ec75640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f071813b5e0 msgr2=0x7f071811c320 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:21:59.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f071ec75640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f071813b5e0 0x7f071811c320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:21:59.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f071e474640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f071809fc50 0x7f071811b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:59.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f071e474640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f071809fc50 0x7f071811b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:41736/0 (socket says 172.21.15.37:41736) 2024-04-26T00:21:59.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f071e474640 1 -- 172.21.15.37:0/4256359702 learned_addr learned my addr 172.21.15.37:0/4256359702 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:21:59.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f071e474640 10 monclient(hunting): get_auth_request con 0x7f071809fc50 auth_method 0 2024-04-26T00:21:59.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f071e474640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:59.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f071e474640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:59.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f071e474640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:59.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f071e474640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:59.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f071e474640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:59.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f071e474640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:59.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f071dc73640 1 --2- 172.21.15.37:0/4256359702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07180a0300 0x7f071811bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:59.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f071e474640 10 monclient(hunting): handle_auth_done global_id 4596 payload 824 2024-04-26T00:21:59.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f071e474640 1 -- 172.21.15.37:0/4256359702 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f071813b5e0 msgr2=0x7f071811c320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:21:59.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f071e474640 1 --2- 172.21.15.37:0/4256359702 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f071813b5e0 0x7f071811c320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:59.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f071e474640 1 -- 172.21.15.37:0/4256359702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07180a0300 msgr2=0x7f071811bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:59.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f071e474640 1 --2- 172.21.15.37:0/4256359702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07180a0300 0x7f071811bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:59.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f071e474640 10 monclient: _finish_hunting 0 2024-04-26T00:21:59.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f071e474640 1 monclient: found mon.noname-a 2024-04-26T00:21:59.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f071e474640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:59.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f071e474640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:21:59.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f071e474640 1 -- 172.21.15.37:0/4256359702 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f071811e500 con 0x7f071809fc50 2024-04-26T00:21:59.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f071dc73640 10 monclient: get_auth_request con 0x7f07180a0300 auth_method 0 2024-04-26T00:21:59.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.426+0000 7f071dc73640 1 --2- 172.21.15.37:0/4256359702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07180a0300 0x7f071811bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:21:59.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f071e474640 1 -- 172.21.15.37:0/4256359702 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f071809fc50 msgr2=0x7f071811b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:59.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f071e474640 1 --2- 172.21.15.37:0/4256359702 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f071809fc50 0x7f071811b8a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f071400c430 tx=0x7f071400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:59.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f070effd640 1 -- 172.21.15.37:0/4256359702 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0714017070 con 0x7f071809fc50 2024-04-26T00:21:59.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f070effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:59.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f070effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:21:59.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f070effd640 10 monclient: dump: 2024-04-26T00:21:59.415 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:59.415 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:59.416 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:59.416 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:59.416 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:59.416 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:59.416 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:59.416 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:59.416 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:59.416 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:59.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f070effd640 10 monclient: _finish_auth 0 2024-04-26T00:21:59.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f070effd640 10 monclient: _check_auth_tickets 2024-04-26T00:21:59.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f070effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:59.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f070effd640 1 -- 172.21.15.37:0/4256359702 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07140057d0 con 0x7f071809fc50 2024-04-26T00:21:59.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f070effd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:59.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f070effd640 1 -- 172.21.15.37:0/4256359702 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0714005ba0 con 0x7f071809fc50 2024-04-26T00:21:59.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f070effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:59.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f070effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:21:59.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f070effd640 10 monclient: dump: 2024-04-26T00:21:59.417 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:59.417 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:59.417 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:59.417 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:59.417 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:59.417 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:59.417 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:59.417 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:59.417 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:59.417 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:59.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f07206d8640 5 monclient: authenticate success, global_id 4596 2024-04-26T00:21:59.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f071cc71640 4 set_mon_vals no callback set 2024-04-26T00:21:59.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f07206d8640 10 monclient: _renew_subs 2024-04-26T00:21:59.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f07206d8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:59.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f07206d8640 1 -- 172.21.15.37:0/4256359702 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f071813a660 con 0x7f071809fc50 2024-04-26T00:21:59.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f07206d8640 10 monclient: _renew_subs 2024-04-26T00:21:59.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f07206d8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:59.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f07206d8640 1 -- 172.21.15.37:0/4256359702 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f071811dc60 con 0x7f071809fc50 2024-04-26T00:21:59.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f07206d8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:59.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f07206d8640 1 --2- 172.21.15.37:0/4256359702 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0718134350 0x7f0718134730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:59.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f07206d8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:59.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f07206d8640 1 -- 172.21.15.37:0/4256359702 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f071804f510 con 0x7f0718134350 2024-04-26T00:21:59.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f070effd640 1 -- 172.21.15.37:0/4256359702 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f071401a070 con 0x7f071809fc50 2024-04-26T00:21:59.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f071dc73640 1 --2- 172.21.15.37:0/4256359702 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0718134350 0x7f0718134730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:59.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f071dc73640 10 monclient: get_auth_request con 0x7f0718134350 auth_method 0 2024-04-26T00:21:59.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f071dc73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:59.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f071dc73640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:59.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f071dc73640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:59.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f071dc73640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:59.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f071dc73640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:59.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f071dc73640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:59.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f070effd640 1 --2- 172.21.15.37:0/4256359702 >> v2:172.21.15.45:6812/2379918623 conn(0x7f06fc035e30 0x7f06fc0382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:59.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f071dc73640 10 monclient(hunting): handle_auth_done global_id 4599 payload 824 2024-04-26T00:21:59.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f071dc73640 1 -- 172.21.15.37:0/4256359702 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f071809eb60 con 0x7f0718134350 2024-04-26T00:21:59.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f070effd640 1 -- 172.21.15.37:0/4256359702 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 3959+0+0 (secure 0 0 0) 0x7f0714047fc0 con 0x7f071809fc50 2024-04-26T00:21:59.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f071ec75640 1 --2- 172.21.15.37:0/4256359702 >> v2:172.21.15.45:6812/2379918623 conn(0x7f06fc035e30 0x7f06fc0382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:59.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.430+0000 7f071ec75640 10 monclient: get_auth_request con 0x7f06fc035e30 auth_method 0 2024-04-26T00:21:59.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.434+0000 7f071dc73640 1 -- 172.21.15.37:0/4256359702 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0718134350 msgr2=0x7f0718134730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:59.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.434+0000 7f071dc73640 1 --2- 172.21.15.37:0/4256359702 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0718134350 0x7f0718134730 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f071813a350 tx=0x7f070800d8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:59.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.434+0000 7f070effd640 1 -- 172.21.15.37:0/4256359702 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0708012a10 con 0x7f0718134350 2024-04-26T00:21:59.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.434+0000 7f070effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:59.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.434+0000 7f070effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:21:59.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.434+0000 7f070effd640 10 monclient: dump: 2024-04-26T00:21:59.422 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:59.422 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:59.422 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:59.422 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:59.422 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:59.422 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:59.422 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:59.422 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:59.422 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:59.422 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:59.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.434+0000 7f070effd640 1 -- 172.21.15.37:0/4256359702 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0708012a10 con 0x7f0718134350 2024-04-26T00:21:59.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.434+0000 7f070effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:59.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.434+0000 7f070effd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:21:59.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.434+0000 7f070effd640 1 -- 172.21.15.37:0/4256359702 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0718134350 msgr2=0x7f0718134730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:59.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.434+0000 7f070effd640 1 --2- 172.21.15.37:0/4256359702 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0718134350 0x7f0718134730 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f071813a350 tx=0x7f070800d8e0 comp rx=0 tx=0).stop 2024-04-26T00:21:59.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.434+0000 7f070effd640 1 -- 172.21.15.37:0/4256359702 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0718134350 msgr2=0x7f0718134730 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:21:59.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.434+0000 7f070effd640 1 --2- 172.21.15.37:0/4256359702 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0718134350 0x7f0718134730 secure :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f071813a350 tx=0x7f070800d8e0 comp rx=0 tx=0).stop 2024-04-26T00:21:59.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.434+0000 7f071ec75640 1 --2- 172.21.15.37:0/4256359702 >> v2:172.21.15.45:6812/2379918623 conn(0x7f06fc035e30 0x7f06fc0382f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f07180764d0 tx=0x7f07100a7070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:59.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.438+0000 7f07206d8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:21:59.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.438+0000 7f07206d8640 1 --2- 172.21.15.37:0/4256359702 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0718134c70 0x7f0718135c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:59.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.438+0000 7f07206d8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:59.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.438+0000 7f07206d8640 1 -- 172.21.15.37:0/4256359702 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f071813aaa0 con 0x7f0718134c70 2024-04-26T00:21:59.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.442+0000 7f071dc73640 1 --2- 172.21.15.37:0/4256359702 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0718134c70 0x7f0718135c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:59.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.442+0000 7f071dc73640 10 monclient: get_auth_request con 0x7f0718134c70 auth_method 0 2024-04-26T00:21:59.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.442+0000 7f071dc73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:59.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.442+0000 7f071dc73640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:59.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.442+0000 7f071dc73640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:59.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.442+0000 7f071dc73640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:59.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.442+0000 7f071dc73640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:59.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.442+0000 7f071dc73640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:59.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.442+0000 7f071dc73640 10 monclient(hunting): handle_auth_done global_id 4602 payload 824 2024-04-26T00:21:59.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.442+0000 7f071dc73640 1 -- 172.21.15.37:0/4256359702 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f071811de30 con 0x7f0718134c70 2024-04-26T00:21:59.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.442+0000 7f071dc73640 1 -- 172.21.15.37:0/4256359702 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0718134c70 msgr2=0x7f0718135c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:59.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.442+0000 7f071dc73640 1 --2- 172.21.15.37:0/4256359702 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0718134c70 0x7f0718135c70 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f07181cdfb0 tx=0x7f0708003770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:59.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.442+0000 7f070effd640 1 -- 172.21.15.37:0/4256359702 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0708028020 con 0x7f0718134c70 2024-04-26T00:21:59.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.442+0000 7f070effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:59.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.442+0000 7f070effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:21:59.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.442+0000 7f070effd640 10 monclient: dump: 2024-04-26T00:21:59.425 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:59.425 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:59.425 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:59.425 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:59.425 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:59.425 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:59.425 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:59.425 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:59.425 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:59.425 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:59.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.442+0000 7f070effd640 1 -- 172.21.15.37:0/4256359702 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4878 (secure 0 0 0) 0x7f0708023350 con 0x7f0718134c70 2024-04-26T00:21:59.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.442+0000 7f070effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:21:59.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.442+0000 7f070effd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:21:59.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.442+0000 7f070effd640 1 -- 172.21.15.37:0/4256359702 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0718134c70 msgr2=0x7f0718135c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:59.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.442+0000 7f070effd640 1 --2- 172.21.15.37:0/4256359702 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0718134c70 0x7f0718135c70 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f07181cdfb0 tx=0x7f0708003770 comp rx=0 tx=0).stop 2024-04-26T00:21:59.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.442+0000 7f070effd640 1 -- 172.21.15.37:0/4256359702 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0718134c70 msgr2=0x7f0718135c70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:21:59.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.442+0000 7f070effd640 1 --2- 172.21.15.37:0/4256359702 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0718134c70 0x7f0718135c70 secure :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f07181cdfb0 tx=0x7f0708003770 comp rx=0 tx=0).stop 2024-04-26T00:21:59.426 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:21:59.427 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:21:59.427 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:21:59.427 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:21:59.427 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 22, 2024-04-26T00:21:59.427 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 136546, 2024-04-26T00:21:59.427 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:21:59.427 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:21:59.427 INFO:teuthology.orchestra.run.smithi037.stdout: 1 2024-04-26T00:21:59.427 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:59.427 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:21:59.427 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:21:59.427 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:21:59.427 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:21:59.428 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:21:59.428 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:21:59.428 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:21:59.428 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:21:59.428 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:21:59.428 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:21:59.428 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:21:59.428 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:21:59.428 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:21:59.428 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:21:59.428 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:21:59.428 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:59.428 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:21:59.429 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:21:59.429 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:21:59.429 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:21:59.429 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:21:59.429 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:21:59.429 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:21:59.429 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:21:59.429 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:21:59.429 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:21:59.429 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:21:59.429 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:21:59.429 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:21:59.429 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:59.430 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:59.430 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:21:59.430 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:21:59.430 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:21:59.430 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:21:59.430 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:21:59.430 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:21:59.430 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:21:59.430 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:21:59.430 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:21:59.430 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:21:59.430 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:21:59.430 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:21:59.430 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:21:59.431 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:21:59.431 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:21:59.431 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:21:59.431 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:21:59.431 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:21:59.431 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:21:59.431 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:21:59.431 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:21:59.431 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:21:59.431 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:21:59.431 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:21:59.431 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:21:59.431 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:21:59.431 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:21:59.432 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:21:59.432 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:59.432 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:21:59.432 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:59.432 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:21:59.432 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:59.432 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:21:59.432 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:21:59.432 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:21:59.432 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:21:59.432 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:59.432 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:21:59.432 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:21:59.432 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:59.432 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:59.432 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:59.433 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:21:59.433 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:21:59.433 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:59.433 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:59.433 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:59.433 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:59.433 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:21:59.433 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:21:59.433 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:21:59.433 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:21:59.433 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:21:59.433 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:59.433 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:59.433 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:21:59.433 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:21:59.433 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:21:59.434 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:21:59.434 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:59.434 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:21:59.434 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:21:59.434 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:59.434 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:59.434 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:59.434 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:21:59.434 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:21:59.434 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:59.434 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:59.434 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:59.434 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:59.435 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:21:59.435 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:21:59.435 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:21:59.435 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:21:59.435 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:21:59.435 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:59.435 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:59.435 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:21:59.435 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:21:59.435 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:21:59.435 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:21:59.435 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:59.435 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:21:59.435 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:21:59.435 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:59.435 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:59.435 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:59.436 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:21:59.436 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:21:59.436 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:59.436 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:59.436 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:59.436 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:59.436 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:21:59.436 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:21:59.436 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:21:59.436 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:21:59.436 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:21:59.436 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:59.436 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:59.436 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:59.436 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:21:59.436 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:21:59.437 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:59.437 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:21:59.437 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:21:59.437 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:21:59.437 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:59.437 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:59.437 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:21:59.437 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:59.437 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:21:59.437 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:21:59.437 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:21:59.437 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:59.437 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:59.437 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:21:59.438 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:59.438 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:21:59.438 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:21:59.438 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:21:59.438 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:59.438 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:59.438 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:21:59.438 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:59.438 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:21:59.438 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:21:59.438 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:21:59.438 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:59.438 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:59.438 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:59.439 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:21:59.439 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:21:59.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.442+0000 7f07206d8640 1 -- 172.21.15.37:0/4256359702 >> v2:172.21.15.45:6812/2379918623 conn(0x7f06fc035e30 msgr2=0x7f06fc0382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:59.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.442+0000 7f07206d8640 1 --2- 172.21.15.37:0/4256359702 >> v2:172.21.15.45:6812/2379918623 conn(0x7f06fc035e30 0x7f06fc0382f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f07180764d0 tx=0x7f07100a7070 comp rx=0 tx=0).stop 2024-04-26T00:21:59.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.442+0000 7f07206d8640 10 monclient: shutdown 2024-04-26T00:21:59.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.442+0000 7f07206d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:59.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.442+0000 7f07206d8640 1 -- 172.21.15.37:0/4256359702 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f071809fc50 msgr2=0x7f071811b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:59.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.442+0000 7f07206d8640 1 --2- 172.21.15.37:0/4256359702 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f071809fc50 0x7f071811b8a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f071400c430 tx=0x7f071400c8f0 comp rx=0 tx=0).stop 2024-04-26T00:21:59.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.442+0000 7f071ec75640 1 -- 172.21.15.37:0/4256359702 reap_dead start 2024-04-26T00:21:59.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.442+0000 7f071ec75640 1 -- 172.21.15.37:0/4256359702 reap_dead start 2024-04-26T00:21:59.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.442+0000 7f07206d8640 1 -- 172.21.15.37:0/4256359702 shutdown_connections 2024-04-26T00:21:59.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.442+0000 7f07206d8640 1 -- 172.21.15.37:0/4256359702 >> 172.21.15.37:0/4256359702 conn(0x7f07180a1d50 msgr2=0x7f07180a3f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:59.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.442+0000 7f07206d8640 1 -- 172.21.15.37:0/4256359702 shutdown_connections 2024-04-26T00:21:59.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.442+0000 7f07206d8640 1 -- 172.21.15.37:0/4256359702 wait complete. 2024-04-26T00:21:59.441 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:21:59.441 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-04-26T00:21:59.441 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:21:59.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.582+0000 7f7e2ecbf640 10 monclient: get_monmap_and_config 2024-04-26T00:21:59.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.582+0000 7f7e2ecbf640 10 monclient: build_initial_monmap 2024-04-26T00:21:59.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.582+0000 7f7e2ecbf640 10 monclient: monmap: 2024-04-26T00:21:59.559 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:59.559 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:59.559 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:21:59.584114+0000 2024-04-26T00:21:59.560 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:21:59.584114+0000 2024-04-26T00:21:59.560 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:59.560 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:59.560 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:21:59.560 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:21:59.560 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:21:59.560 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:21:59.560 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:21:59.560 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.582+0000 7f7e2ecbf640 1 Processor -- start 2024-04-26T00:21:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.582+0000 7f7e2ecbf640 1 -- start start 2024-04-26T00:21:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.582+0000 7f7e2ecbf640 10 monclient: init 2024-04-26T00:21:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.582+0000 7f7e2ecbf640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.582+0000 7f7e2ecbf640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-04-26T00:21:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.582+0000 7f7e2ecbf640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7e280a3d90 0x7f7e2809bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.582+0000 7f7e2ecbf640 10 monclient(hunting): picked mon.noname-c con 0x7f7e280a3d90 addr v2:172.21.15.45:3300/0 2024-04-26T00:21:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.582+0000 7f7e2ecbf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7e280a4170 addr v1:172.21.15.45:6789/0 2024-04-26T00:21:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.582+0000 7f7e2ecbf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7e2813c3e0 addr v1:172.21.15.37:6789/0 2024-04-26T00:21:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.582+0000 7f7e2ecbf640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e280a5150 con 0x7f7e2813c3e0 2024-04-26T00:21:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.582+0000 7f7e2ecbf640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e2809b280 con 0x7f7e280a4170 2024-04-26T00:21:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.582+0000 7f7e2ecbf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.582+0000 7f7e2ecbf640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f7e2809b560 con 0x7f7e280a3d90 2024-04-26T00:21:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.582+0000 7f7e2ecbf640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.582+0000 7f7e2ecbf640 10 monclient(hunting): authenticate will time out at 1100.889825s 2024-04-26T00:21:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.582+0000 7f7e2d25c640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f7e2813c3e0 0x7f7e2813e7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:36500/0 (socket says 172.21.15.37:36500) 2024-04-26T00:21:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.582+0000 7f7e2d25c640 1 -- 172.21.15.37:0/1989976431 learned_addr learned my addr 172.21.15.37:0/1989976431 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:21:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.582+0000 7f7e2ca5b640 1 --2- 172.21.15.37:0/1989976431 >> v2:172.21.15.45:3300/0 conn(0x7f7e280a3d90 0x7f7e2809bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.582+0000 7f7e2ca5b640 10 monclient(hunting): get_auth_request con 0x7f7e280a3d90 auth_method 0 2024-04-26T00:21:59.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.582+0000 7f7e2ca5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:59.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.582+0000 7f7e2ca5b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:59.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.582+0000 7f7e2ca5b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:59.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.582+0000 7f7e2ca5b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:59.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.582+0000 7f7e2ca5b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:59.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.582+0000 7f7e2ca5b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:59.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.582+0000 7f7e2ca5b640 10 monclient(hunting): handle_auth_done global_id 24157 payload 274 2024-04-26T00:21:59.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.582+0000 7f7e2ca5b640 1 -- 172.21.15.37:0/1989976431 >> v1:172.21.15.45:6789/0 conn(0x7f7e280a4170 legacy=0x7f7e280a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:21:59.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.582+0000 7f7e2ca5b640 1 -- 172.21.15.37:0/1989976431 >> v1:172.21.15.37:6789/0 conn(0x7f7e2813c3e0 legacy=0x7f7e2813e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:21:59.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.582+0000 7f7e2ca5b640 10 monclient: _finish_hunting 0 2024-04-26T00:21:59.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.582+0000 7f7e2ca5b640 1 monclient: found mon.noname-c 2024-04-26T00:21:59.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.582+0000 7f7e2ca5b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:59.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.582+0000 7f7e2ca5b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:21:59.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.582+0000 7f7e2ca5b640 1 -- 172.21.15.37:0/1989976431 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e2806dc60 con 0x7f7e280a3d90 2024-04-26T00:21:59.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.586+0000 7f7e2ca5b640 1 -- 172.21.15.37:0/1989976431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e280a3d90 msgr2=0x7f7e2809bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:59.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.586+0000 7f7e2ca5b640 1 --2- 172.21.15.37:0/1989976431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e280a3d90 0x7f7e2809bf40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7e180097c0 tx=0x7f7e1801c2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=979461fbbf0aaddf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:59.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.586+0000 7f7e277fe640 1 -- 172.21.15.37:0/1989976431 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e180b0070 con 0x7f7e280a3d90 2024-04-26T00:21:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.586+0000 7f7e277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.586+0000 7f7e277fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:21:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.586+0000 7f7e277fe640 10 monclient: dump: 2024-04-26T00:21:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:59.567 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.586+0000 7f7e277fe640 10 monclient: _finish_auth 0 2024-04-26T00:21:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.586+0000 7f7e277fe640 10 monclient: _check_auth_tickets 2024-04-26T00:21:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.586+0000 7f7e277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.586+0000 7f7e277fe640 1 -- 172.21.15.37:0/1989976431 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e180ab600 con 0x7f7e280a3d90 2024-04-26T00:21:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.586+0000 7f7e277fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.586+0000 7f7e277fe640 1 -- 172.21.15.37:0/1989976431 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e180ab9d0 con 0x7f7e280a3d90 2024-04-26T00:21:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.586+0000 7f7e277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.586+0000 7f7e277fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:21:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.586+0000 7f7e277fe640 10 monclient: dump: 2024-04-26T00:21:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:59.569 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.586+0000 7f7e2ecbf640 5 monclient: authenticate success, global_id 24157 2024-04-26T00:21:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.586+0000 7f7e2ecbf640 10 monclient: get_monmap_and_config success 2024-04-26T00:21:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.586+0000 7f7e2ecbf640 4 set_mon_vals no callback set 2024-04-26T00:21:59.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ecbf640 10 monclient: shutdown 2024-04-26T00:21:59.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ecbf640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:59.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ecbf640 1 -- 172.21.15.37:0/1989976431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e280a3d90 msgr2=0x7f7e2809bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:59.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ecbf640 1 --2- 172.21.15.37:0/1989976431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e280a3d90 0x7f7e2809bf40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7e180097c0 tx=0x7f7e1801c2e0 comp rx=0 tx=0).stop 2024-04-26T00:21:59.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ecbf640 1 -- 172.21.15.37:0/1989976431 shutdown_connections 2024-04-26T00:21:59.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ecbf640 1 --2- 172.21.15.37:0/1989976431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e280a3d90 0x7f7e2809bf40 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:59.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ecbf640 1 -- 172.21.15.37:0/1989976431 >> 172.21.15.37:0/1989976431 conn(0x7f7e280aa0a0 msgr2=0x7f7e280a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:59.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ecbf640 1 -- 172.21.15.37:0/1989976431 shutdown_connections 2024-04-26T00:21:59.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ecbf640 1 -- 172.21.15.37:0/1989976431 wait complete. 2024-04-26T00:21:59.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ecbf640 10 monclient: build_initial_monmap 2024-04-26T00:21:59.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ecbf640 10 monclient: monmap: 2024-04-26T00:21:59.575 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:21:59.575 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:21:59.576 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:21:59.576 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:21:59.576 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:21:59.576 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:21:59.576 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:21:59.576 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:21:59.576 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:21:59.576 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:59.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ecbf640 1 Processor -- start 2024-04-26T00:21:59.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ecbf640 1 -- start start 2024-04-26T00:21:59.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ecbf640 10 monclient: init 2024-04-26T00:21:59.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ecbf640 10 monclient: _reopen_session rank -1 2024-04-26T00:21:59.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ecbf640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:21:59.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ecbf640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e280a3d90 0x7f7e281cebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:59.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ecbf640 10 monclient(hunting): picked mon.noname-a con 0x7f7e280a3d90 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:21:59.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ecbf640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7e280a4170 0x7f7e28136e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:59.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ecbf640 10 monclient(hunting): picked mon.noname-c con 0x7f7e280a4170 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:21:59.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ecbf640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e2813c3e0 0x7f7e28138f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:59.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ecbf640 10 monclient(hunting): picked mon.noname-b con 0x7f7e2813c3e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:21:59.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ecbf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:59.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ecbf640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f7e28142920 con 0x7f7e280a3d90 2024-04-26T00:21:59.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ecbf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:59.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ecbf640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7e2809c690 con 0x7f7e2813c3e0 2024-04-26T00:21:59.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ecbf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:59.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ecbf640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f7e28142aa0 con 0x7f7e280a4170 2024-04-26T00:21:59.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ecbf640 10 monclient(hunting): _renew_subs 2024-04-26T00:21:59.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ecbf640 10 monclient(hunting): authenticate will time out at 1100.898361s 2024-04-26T00:21:59.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e27fff640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7e280a4170 msgr2=0x7f7e28136e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:21:59.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e27fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7e280a4170 0x7f7e28136e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:21:59.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ca5b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e280a3d90 0x7f7e281cebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:59.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ca5b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e280a3d90 0x7f7e281cebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:41754/0 (socket says 172.21.15.37:41754) 2024-04-26T00:21:59.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ca5b640 1 -- 172.21.15.37:0/4235261836 learned_addr learned my addr 172.21.15.37:0/4235261836 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:21:59.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ca5b640 10 monclient(hunting): get_auth_request con 0x7f7e280a3d90 auth_method 0 2024-04-26T00:21:59.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ca5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:59.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ca5b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:59.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ca5b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:59.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ca5b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:59.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ca5b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:59.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ca5b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:59.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2d25c640 1 --2- 172.21.15.37:0/4235261836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e2813c3e0 0x7f7e28138f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:59.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2d25c640 10 monclient(hunting): get_auth_request con 0x7f7e2813c3e0 auth_method 0 2024-04-26T00:21:59.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2d25c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:59.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2d25c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:59.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2d25c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:59.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ca5b640 10 monclient(hunting): handle_auth_done global_id 4605 payload 824 2024-04-26T00:21:59.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ca5b640 1 -- 172.21.15.37:0/4235261836 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7e280a4170 msgr2=0x7f7e28136e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:21:59.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ca5b640 1 --2- 172.21.15.37:0/4235261836 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7e280a4170 0x7f7e28136e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:59.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ca5b640 1 -- 172.21.15.37:0/4235261836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e2813c3e0 msgr2=0x7f7e28138f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:59.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ca5b640 1 --2- 172.21.15.37:0/4235261836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e2813c3e0 0x7f7e28138f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:21:59.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ca5b640 10 monclient: _finish_hunting 0 2024-04-26T00:21:59.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ca5b640 1 monclient: found mon.noname-a 2024-04-26T00:21:59.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ca5b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:21:59.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ca5b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:21:59.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.590+0000 7f7e2ca5b640 1 -- 172.21.15.37:0/4235261836 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e281204d0 con 0x7f7e280a3d90 2024-04-26T00:21:59.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e2ca5b640 1 -- 172.21.15.37:0/4235261836 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e280a3d90 msgr2=0x7f7e281cebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:59.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e2ca5b640 1 --2- 172.21.15.37:0/4235261836 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e280a3d90 0x7f7e281cebc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7e18018ef0 tx=0x7f7e18019080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:59.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e257fa640 1 -- 172.21.15.37:0/4235261836 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e180b9070 con 0x7f7e280a3d90 2024-04-26T00:21:59.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e257fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:59.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e257fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:21:59.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e257fa640 10 monclient: dump: 2024-04-26T00:21:59.580 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:59.580 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:59.580 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:59.580 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:59.580 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:59.580 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:59.580 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:59.581 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:59.581 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:59.581 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:59.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e257fa640 10 monclient: _finish_auth 0 2024-04-26T00:21:59.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e257fa640 10 monclient: _check_auth_tickets 2024-04-26T00:21:59.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:21:59.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e257fa640 1 -- 172.21.15.37:0/4235261836 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e180b5aa0 con 0x7f7e280a3d90 2024-04-26T00:21:59.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e2ecbf640 5 monclient: authenticate success, global_id 4605 2024-04-26T00:21:59.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e2ecbf640 10 monclient: _renew_subs 2024-04-26T00:21:59.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e2ecbf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:59.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e2ecbf640 1 -- 172.21.15.37:0/4235261836 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e2813b630 con 0x7f7e280a3d90 2024-04-26T00:21:59.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e257fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:21:59.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e257fa640 1 -- 172.21.15.37:0/4235261836 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e180b0040 con 0x7f7e280a3d90 2024-04-26T00:21:59.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e257fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:59.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e257fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:21:59.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e257fa640 10 monclient: dump: 2024-04-26T00:21:59.582 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:59.582 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:59.582 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:59.582 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:59.582 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:59.582 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:59.582 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:59.582 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:59.582 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:59.582 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:59.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e267fc640 4 set_mon_vals no callback set 2024-04-26T00:21:59.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e2ecbf640 10 monclient: _renew_subs 2024-04-26T00:21:59.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e2ecbf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:21:59.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e2ecbf640 1 -- 172.21.15.37:0/4235261836 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e2809ada0 con 0x7f7e280a3d90 2024-04-26T00:21:59.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e2ecbf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:59.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e2ecbf640 1 --2- 172.21.15.37:0/4235261836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e281394b0 0x7f7e28139890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:59.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e2ecbf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:59.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e2ecbf640 1 -- 172.21.15.37:0/4235261836 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7e2811f9e0 con 0x7f7e281394b0 2024-04-26T00:21:59.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e257fa640 1 -- 172.21.15.37:0/4235261836 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f7e180bc070 con 0x7f7e280a3d90 2024-04-26T00:21:59.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e27fff640 1 --2- 172.21.15.37:0/4235261836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e281394b0 0x7f7e28139890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:59.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e257fa640 1 --2- 172.21.15.37:0/4235261836 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7e14035e30 0x7f7e140382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:59.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e27fff640 10 monclient: get_auth_request con 0x7f7e281394b0 auth_method 0 2024-04-26T00:21:59.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:59.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e27fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:59.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e27fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:59.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e257fa640 1 -- 172.21.15.37:0/4235261836 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 3959+0+0 (secure 0 0 0) 0x7f7e180e2be0 con 0x7f7e280a3d90 2024-04-26T00:21:59.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e27fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:59.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e27fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:59.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e27fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:59.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e2d25c640 1 --2- 172.21.15.37:0/4235261836 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7e14035e30 0x7f7e140382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:59.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e2d25c640 10 monclient: get_auth_request con 0x7f7e14035e30 auth_method 0 2024-04-26T00:21:59.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e27fff640 10 monclient(hunting): handle_auth_done global_id 24163 payload 824 2024-04-26T00:21:59.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.594+0000 7f7e27fff640 1 -- 172.21.15.37:0/4235261836 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7e28139dd0 con 0x7f7e281394b0 2024-04-26T00:21:59.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.598+0000 7f7e27fff640 1 -- 172.21.15.37:0/4235261836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e281394b0 msgr2=0x7f7e28139890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:59.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.598+0000 7f7e27fff640 1 --2- 172.21.15.37:0/4235261836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e281394b0 0x7f7e28139890 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c007fd0 tx=0x7f7e1c00c810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:59.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.598+0000 7f7e2d25c640 1 --2- 172.21.15.37:0/4235261836 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7e14035e30 0x7f7e140382f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7e20007af0 tx=0x7f7e20009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:59.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.598+0000 7f7e257fa640 1 -- 172.21.15.37:0/4235261836 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c017070 con 0x7f7e281394b0 2024-04-26T00:21:59.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.598+0000 7f7e257fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:59.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.598+0000 7f7e257fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:21:59.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.598+0000 7f7e257fa640 10 monclient: dump: 2024-04-26T00:21:59.586 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:59.586 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:59.586 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:59.586 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:59.586 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:59.586 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:59.587 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:59.587 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:59.587 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:59.587 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:59.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.598+0000 7f7e257fa640 1 -- 172.21.15.37:0/4235261836 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7e1c0057d0 con 0x7f7e281394b0 2024-04-26T00:21:59.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.598+0000 7f7e257fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:21:59.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.598+0000 7f7e257fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:21:59.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.598+0000 7f7e257fa640 1 -- 172.21.15.37:0/4235261836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e281394b0 msgr2=0x7f7e28139890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:59.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.598+0000 7f7e257fa640 1 --2- 172.21.15.37:0/4235261836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e281394b0 0x7f7e28139890 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c007fd0 tx=0x7f7e1c00c810 comp rx=0 tx=0).stop 2024-04-26T00:21:59.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.598+0000 7f7e257fa640 1 -- 172.21.15.37:0/4235261836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e281394b0 msgr2=0x7f7e28139890 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:21:59.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.598+0000 7f7e257fa640 1 --2- 172.21.15.37:0/4235261836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e281394b0 0x7f7e28139890 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c007fd0 tx=0x7f7e1c00c810 comp rx=0 tx=0).stop 2024-04-26T00:21:59.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.606+0000 7f7e2ecbf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:21:59.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.606+0000 7f7e2ecbf640 1 --2- 172.21.15.37:0/4235261836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e2809d070 0x7f7e2809d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:21:59.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.606+0000 7f7e2ecbf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:21:59.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.606+0000 7f7e2ecbf640 1 -- 172.21.15.37:0/4235261836 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7e2813b3b0 con 0x7f7e2809d070 2024-04-26T00:21:59.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.606+0000 7f7e27fff640 1 --2- 172.21.15.37:0/4235261836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e2809d070 0x7f7e2809d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:21:59.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.606+0000 7f7e27fff640 10 monclient: get_auth_request con 0x7f7e2809d070 auth_method 0 2024-04-26T00:21:59.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.606+0000 7f7e27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:21:59.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.606+0000 7f7e27fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:21:59.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.606+0000 7f7e27fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:21:59.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.606+0000 7f7e27fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:21:59.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.606+0000 7f7e27fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:21:59.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.606+0000 7f7e27fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:21:59.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.606+0000 7f7e27fff640 10 monclient(hunting): handle_auth_done global_id 24166 payload 824 2024-04-26T00:21:59.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.606+0000 7f7e27fff640 1 -- 172.21.15.37:0/4235261836 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7e281cfa40 con 0x7f7e2809d070 2024-04-26T00:21:59.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.610+0000 7f7e27fff640 1 -- 172.21.15.37:0/4235261836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e2809d070 msgr2=0x7f7e2809d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:21:59.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.610+0000 7f7e27fff640 1 --2- 172.21.15.37:0/4235261836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e2809d070 0x7f7e2809d480 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c011000 tx=0x7f7e1c0037f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:21:59.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.610+0000 7f7e257fa640 1 -- 172.21.15.37:0/4235261836 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c017070 con 0x7f7e2809d070 2024-04-26T00:21:59.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.610+0000 7f7e257fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:21:59.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.610+0000 7f7e257fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:21:59.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.610+0000 7f7e257fa640 10 monclient: dump: 2024-04-26T00:21:59.590 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:21:59.590 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:21:59.590 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:59.590 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:21:59.590 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:21:59.590 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:21:59.590 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:21:59.590 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:21:59.590 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:21:59.590 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:21:59.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.610+0000 7f7e257fa640 1 -- 172.21.15.37:0/4235261836 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4535 (secure 0 0 0) 0x7f7e1c018040 con 0x7f7e2809d070 2024-04-26T00:21:59.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.610+0000 7f7e257fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:21:59.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.610+0000 7f7e257fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:21:59.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.610+0000 7f7e257fa640 1 -- 172.21.15.37:0/4235261836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e2809d070 msgr2=0x7f7e2809d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:59.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.610+0000 7f7e257fa640 1 --2- 172.21.15.37:0/4235261836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e2809d070 0x7f7e2809d480 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c011000 tx=0x7f7e1c0037f0 comp rx=0 tx=0).stop 2024-04-26T00:21:59.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.610+0000 7f7e257fa640 1 -- 172.21.15.37:0/4235261836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e2809d070 msgr2=0x7f7e2809d480 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:21:59.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.610+0000 7f7e257fa640 1 --2- 172.21.15.37:0/4235261836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e2809d070 0x7f7e2809d480 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c011000 tx=0x7f7e1c0037f0 comp rx=0 tx=0).stop 2024-04-26T00:21:59.591 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:21:59.591 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:21:59.591 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:21:59.591 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:21:59.591 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 22, 2024-04-26T00:21:59.591 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 27774, 2024-04-26T00:21:59.592 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:21:59.592 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:21:59.592 INFO:teuthology.orchestra.run.smithi037.stdout: 1 2024-04-26T00:21:59.592 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:59.592 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:21:59.592 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:21:59.592 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:21:59.592 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:21:59.592 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:21:59.592 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:21:59.592 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:21:59.592 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:21:59.592 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:21:59.592 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:21:59.593 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:21:59.593 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:21:59.593 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:21:59.593 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:21:59.593 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:21:59.593 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:59.593 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:21:59.593 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:21:59.593 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:21:59.593 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:21:59.593 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:21:59.593 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:21:59.593 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:21:59.593 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:21:59.594 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:21:59.594 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:21:59.594 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:21:59.594 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:21:59.594 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:21:59.594 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:59.594 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:59.594 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:21:59.594 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:21:59.594 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:21:59.594 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:21:59.594 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:21:59.594 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:21:59.594 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:21:59.595 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:21:59.595 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:21:59.595 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:21:59.595 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:21:59.595 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:21:59.595 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:21:59.595 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:21:59.595 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:21:59.595 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:21:59.595 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:21:59.595 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:21:59.595 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:21:59.595 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:21:59.595 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:21:59.596 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:21:59.596 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:21:59.596 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:21:59.596 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:21:59.596 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:21:59.596 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:21:59.596 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:21:59.596 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:59.596 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:21:59.596 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:59.596 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:21:59.596 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:59.596 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:21:59.596 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:21:59.597 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:21:59.597 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:21:59.597 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:59.597 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:21:59.597 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:21:59.597 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:59.597 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:59.597 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:59.597 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:21:59.597 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:21:59.597 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:59.597 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:59.597 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:59.598 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:59.598 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:21:59.598 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:21:59.598 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:21:59.598 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:21:59.598 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:21:59.598 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:59.598 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:59.598 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:21:59.598 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:21:59.598 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:21:59.599 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:21:59.599 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:59.599 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:21:59.599 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:21:59.599 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:59.599 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:59.599 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:59.599 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:21:59.599 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:21:59.599 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:59.599 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:59.599 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:59.599 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:59.599 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:21:59.599 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:21:59.600 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:21:59.600 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:21:59.600 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:21:59.600 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:59.600 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:59.600 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:21:59.600 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:21:59.600 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:21:59.600 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:21:59.600 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:59.600 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:21:59.600 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:21:59.600 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:59.600 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:59.600 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:59.601 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:21:59.601 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:21:59.601 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:21:59.601 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:59.601 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:59.601 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:59.601 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:21:59.601 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:21:59.601 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:21:59.601 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:21:59.601 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:21:59.601 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:59.601 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:59.601 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:59.601 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:21:59.601 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:21:59.602 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:59.602 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:21:59.602 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:21:59.602 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:21:59.602 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:59.602 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:21:59.602 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:21:59.602 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:21:59.602 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:21:59.602 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:21:59.602 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:21:59.602 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:21:59.602 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:21:59.602 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:21:59.603 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:21:59.603 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:21:59.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.610+0000 7f7e2ecbf640 1 -- 172.21.15.37:0/4235261836 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7e14035e30 msgr2=0x7f7e140382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:59.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.610+0000 7f7e2ecbf640 1 --2- 172.21.15.37:0/4235261836 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7e14035e30 0x7f7e140382f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7e20007af0 tx=0x7f7e20009040 comp rx=0 tx=0).stop 2024-04-26T00:21:59.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.610+0000 7f7e2ecbf640 10 monclient: shutdown 2024-04-26T00:21:59.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.610+0000 7f7e2ecbf640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:21:59.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.610+0000 7f7e2ecbf640 1 -- 172.21.15.37:0/4235261836 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e280a3d90 msgr2=0x7f7e281cebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:21:59.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.610+0000 7f7e2ecbf640 1 --2- 172.21.15.37:0/4235261836 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e280a3d90 0x7f7e281cebc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7e18018ef0 tx=0x7f7e18019080 comp rx=0 tx=0).stop 2024-04-26T00:21:59.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.610+0000 7f7e2d25c640 1 -- 172.21.15.37:0/4235261836 reap_dead start 2024-04-26T00:21:59.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.610+0000 7f7e2d25c640 1 -- 172.21.15.37:0/4235261836 reap_dead start 2024-04-26T00:21:59.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.610+0000 7f7e2ecbf640 1 -- 172.21.15.37:0/4235261836 shutdown_connections 2024-04-26T00:21:59.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.610+0000 7f7e2ecbf640 1 -- 172.21.15.37:0/4235261836 >> 172.21.15.37:0/4235261836 conn(0x7f7e280aa0a0 msgr2=0x7f7e280a25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:21:59.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.610+0000 7f7e2ecbf640 1 -- 172.21.15.37:0/4235261836 shutdown_connections 2024-04-26T00:21:59.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:21:59.610+0000 7f7e2ecbf640 1 -- 172.21.15.37:0/4235261836 wait complete. 2024-04-26T00:21:59.606 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:21:59.606 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-04-26T00:21:59.997 INFO:tasks.rados.rados.0.smithi045.stdout:321: done (2 left) 2024-04-26T00:21:59.998 INFO:tasks.rados.rados.0.smithi045.stdout:322: done (1 left) 2024-04-26T00:21:59.998 INFO:tasks.rados.rados.0.smithi045.stdout:325: done (0 left) 2024-04-26T00:21:59.998 INFO:tasks.rados.rados.0.smithi045.stdout:326: write (excl) oid 7 current snap is 32 2024-04-26T00:21:59.998 INFO:tasks.rados.rados.0.smithi045.stdout:326: seq_num 146 ranges {453822=797061,1760371=653366} 2024-04-26T00:22:00.017 INFO:tasks.rados.rados.0.smithi045.stdout:326: writing smithi04526579-7 from 453822 to 1250883 tid 1 2024-04-26T00:22:00.038 INFO:tasks.rados.rados.0.smithi045.stdout:326: writing smithi04526579-7 from 1760371 to 2413737 tid 2 2024-04-26T00:22:00.038 INFO:tasks.rados.rados.0.smithi045.stdout:327: rollback oid 40 current snap is 32 2024-04-26T00:22:00.038 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 40 to 31 2024-04-26T00:22:00.038 INFO:tasks.rados.rados.0.smithi045.stdout:326: finishing write tid 1 to smithi04526579-7 2024-04-26T00:22:00.039 INFO:tasks.rados.rados.0.smithi045.stdout:328: rollback oid 31 current snap is 32 2024-04-26T00:22:00.039 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 31 to 31 2024-04-26T00:22:00.039 INFO:tasks.rados.rados.0.smithi045.stdout:329: delete oid 11 current snap is 32 2024-04-26T00:22:00.039 INFO:tasks.rados.rados.0.smithi045.stdout:328: finishing rollback tid 1 to smithi04526579-31 2024-04-26T00:22:00.039 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 0 (ObjNum 177 snap 0 seq_num 2241727840) dirty dne 2024-04-26T00:22:00.046 INFO:tasks.rados.rados.0.smithi045.stdout:326: finishing write tid 2 to smithi04526579-7 2024-04-26T00:22:00.046 INFO:tasks.rados.rados.0.smithi045.stdout:327: finishing rollback tid 0 to smithi04526579-40 2024-04-26T00:22:00.047 INFO:tasks.rados.rados.0.smithi045.stdout:327: finishing rollback tid 1 to smithi04526579-40 2024-04-26T00:22:00.047 INFO:tasks.rados.rados.0.smithi045.stdout:328: done (3 left) 2024-04-26T00:22:00.047 INFO:tasks.rados.rados.0.smithi045.stdout:329: done (2 left) 2024-04-26T00:22:00.048 INFO:tasks.rados.rados.0.smithi045.stdout:330: write oid 30 current snap is 32 2024-04-26T00:22:00.048 INFO:tasks.rados.rados.0.smithi045.stdout:330: seq_num 148 ranges {487436=547559,1760982=657035} 2024-04-26T00:22:00.061 INFO:tasks.rados.rados.0.smithi045.stdout:330: writing smithi04526579-30 from 487436 to 1034995 tid 1 2024-04-26T00:22:00.079 INFO:tasks.rados.rados.0.smithi045.stdout:330: writing smithi04526579-30 from 1760982 to 2418017 tid 2 2024-04-26T00:22:00.079 INFO:tasks.rados.rados.0.smithi045.stdout:331: rollback oid 44 current snap is 32 2024-04-26T00:22:00.079 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 44 to 31 2024-04-26T00:22:00.079 INFO:tasks.rados.rados.0.smithi045.stdout:326: finishing write tid 3 to smithi04526579-7 2024-04-26T00:22:00.079 INFO:tasks.rados.rados.0.smithi045.stdout:327: finishing rollback tid 2 to smithi04526579-40 2024-04-26T00:22:00.080 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 69 (ObjNum 131 snap 30 seq_num 131) dirty exists 2024-04-26T00:22:00.080 INFO:tasks.rados.rados.0.smithi045.stdout:326: finishing write tid 4 to smithi04526579-7 2024-04-26T00:22:00.080 INFO:tasks.rados.rados.0.smithi045.stdout:326: oid 7 updating version 0 to 193 2024-04-26T00:22:00.080 INFO:tasks.rados.rados.0.smithi045.stdout:326: oid 7 updating version 193 to 195 2024-04-26T00:22:00.080 INFO:tasks.rados.rados.0.smithi045.stdout:326: oid 7 version 195 is already newer than 194 2024-04-26T00:22:00.080 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 195 (ObjNum 146 snap 32 seq_num 146) dirty exists 2024-04-26T00:22:00.080 INFO:tasks.rados.rados.0.smithi045.stdout:326: left oid 7 (ObjNum 146 snap 32 seq_num 146) 2024-04-26T00:22:00.080 INFO:tasks.rados.rados.0.smithi045.stdout:330: finishing write tid 1 to smithi04526579-30 2024-04-26T00:22:00.080 INFO:tasks.rados.rados.0.smithi045.stdout:326: done (3 left) 2024-04-26T00:22:00.080 INFO:tasks.rados.rados.0.smithi045.stdout:327: done (2 left) 2024-04-26T00:22:00.080 INFO:tasks.rados.rados.0.smithi045.stdout:332: read oid 25 snap -1 2024-04-26T00:22:00.080 INFO:tasks.rados.rados.0.smithi045.stdout:332: expect (ObjNum 72 snap 11 seq_num 72) 2024-04-26T00:22:00.080 INFO:tasks.rados.rados.0.smithi045.stdout:333: rollback oid 5 current snap is 32 2024-04-26T00:22:00.080 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 5 to 31 2024-04-26T00:22:00.081 INFO:tasks.rados.rados.0.smithi045.stdout:334: read oid 26 snap 31 2024-04-26T00:22:00.081 INFO:tasks.rados.rados.0.smithi045.stdout:334: expect (ObjNum 136 snap 30 seq_num 136) 2024-04-26T00:22:00.081 INFO:tasks.rados.rados.0.smithi045.stdout:335: write oid 31 current snap is 32 2024-04-26T00:22:00.081 INFO:tasks.rados.rados.0.smithi045.stdout:335: seq_num 149 ranges {504243=622808,1819464=517046} 2024-04-26T00:22:00.092 INFO:tasks.rados.rados.0.smithi045.stdout:335: writing smithi04526579-31 from 504243 to 1127051 tid 1 2024-04-26T00:22:00.104 INFO:tasks.rados.rados.0.smithi045.stdout:335: writing smithi04526579-31 from 1819464 to 2336510 tid 2 2024-04-26T00:22:00.104 INFO:tasks.rados.rados.0.smithi045.stdout:336: read oid 7 snap -1 2024-04-26T00:22:00.104 INFO:tasks.rados.rados.0.smithi045.stdout:336: expect (ObjNum 146 snap 32 seq_num 146) 2024-04-26T00:22:00.104 INFO:tasks.rados.rados.0.smithi045.stdout:337: rollback oid 23 current snap is 32 2024-04-26T00:22:00.104 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 23 to 31 2024-04-26T00:22:00.104 INFO:tasks.rados.rados.0.smithi045.stdout:331: finishing rollback tid 0 to smithi04526579-44 2024-04-26T00:22:00.104 INFO:tasks.rados.rados.0.smithi045.stdout:331: finishing rollback tid 1 to smithi04526579-44 2024-04-26T00:22:00.104 INFO:tasks.rados.rados.0.smithi045.stdout:331: finishing rollback tid 2 to smithi04526579-44 2024-04-26T00:22:00.104 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 107 (ObjNum 119 snap 28 seq_num 119) dirty exists 2024-04-26T00:22:00.104 INFO:tasks.rados.rados.0.smithi045.stdout:334: expect (ObjNum 136 snap 30 seq_num 136) 2024-04-26T00:22:00.299 INFO:tasks.rados.rados.0.smithi045.stdout:330: finishing write tid 2 to smithi04526579-30 2024-04-26T00:22:00.299 INFO:tasks.rados.rados.0.smithi045.stdout:330: finishing write tid 3 to smithi04526579-30 2024-04-26T00:22:00.299 INFO:tasks.rados.rados.0.smithi045.stdout:330: finishing write tid 4 to smithi04526579-30 2024-04-26T00:22:00.299 INFO:tasks.rados.rados.0.smithi045.stdout:330: oid 30 updating version 0 to 103 2024-04-26T00:22:00.299 INFO:tasks.rados.rados.0.smithi045.stdout:330: oid 30 version 103 is already newer than 102 2024-04-26T00:22:00.299 INFO:tasks.rados.rados.0.smithi045.stdout:330: oid 30 version 103 is already newer than 101 2024-04-26T00:22:00.299 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 103 (ObjNum 148 snap 32 seq_num 148) dirty exists 2024-04-26T00:22:00.299 INFO:tasks.rados.rados.0.smithi045.stdout:330: left oid 30 (ObjNum 148 snap 32 seq_num 148) 2024-04-26T00:22:00.299 INFO:tasks.rados.rados.0.smithi045.stdout:332: expect (ObjNum 72 snap 11 seq_num 72) 2024-04-26T00:22:00.505 INFO:tasks.rados.rados.0.smithi045.stdout:333: finishing rollback tid 0 to smithi04526579-5 2024-04-26T00:22:00.505 INFO:tasks.rados.rados.0.smithi045.stdout:333: finishing rollback tid 1 to smithi04526579-5 2024-04-26T00:22:00.506 INFO:tasks.rados.rados.0.smithi045.stdout:333: finishing rollback tid 2 to smithi04526579-5 2024-04-26T00:22:00.506 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 199 (ObjNum 52 snap 1 seq_num 52) dirty exists 2024-04-26T00:22:00.506 INFO:tasks.rados.rados.0.smithi045.stdout:335: finishing write tid 1 to smithi04526579-31 2024-04-26T00:22:00.506 INFO:tasks.rados.rados.0.smithi045.stdout:335: finishing write tid 2 to smithi04526579-31 2024-04-26T00:22:00.506 INFO:tasks.rados.rados.0.smithi045.stdout:337: finishing rollback tid 0 to smithi04526579-23 2024-04-26T00:22:00.506 INFO:tasks.rados.rados.0.smithi045.stdout:337: finishing rollback tid 1 to smithi04526579-23 2024-04-26T00:22:00.506 INFO:tasks.rados.rados.0.smithi045.stdout:337: finishing rollback tid 2 to smithi04526579-23 2024-04-26T00:22:00.506 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 73 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-04-26T00:22:00.506 INFO:tasks.rados.rados.0.smithi045.stdout:335: finishing write tid 3 to smithi04526579-31 2024-04-26T00:22:00.506 INFO:tasks.rados.rados.0.smithi045.stdout:335: finishing write tid 4 to smithi04526579-31 2024-04-26T00:22:00.506 INFO:tasks.rados.rados.0.smithi045.stdout:335: oid 31 updating version 0 to 164 2024-04-26T00:22:00.506 INFO:tasks.rados.rados.0.smithi045.stdout:335: oid 31 version 164 is already newer than 162 2024-04-26T00:22:00.507 INFO:tasks.rados.rados.0.smithi045.stdout:335: oid 31 version 164 is already newer than 163 2024-04-26T00:22:00.507 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 164 (ObjNum 149 snap 32 seq_num 149) dirty exists 2024-04-26T00:22:00.507 INFO:tasks.rados.rados.0.smithi045.stdout:335: left oid 31 (ObjNum 149 snap 32 seq_num 149) 2024-04-26T00:22:00.507 INFO:tasks.rados.rados.0.smithi045.stdout:336: expect (ObjNum 146 snap 32 seq_num 146) 2024-04-26T00:22:00.582 INFO:tasks.rados.rados.0.smithi045.stdout:330: done (7 left) 2024-04-26T00:22:00.582 INFO:tasks.rados.rados.0.smithi045.stdout:331: done (6 left) 2024-04-26T00:22:00.582 INFO:tasks.rados.rados.0.smithi045.stdout:332: done (5 left) 2024-04-26T00:22:00.582 INFO:tasks.rados.rados.0.smithi045.stdout:333: done (4 left) 2024-04-26T00:22:00.582 INFO:tasks.rados.rados.0.smithi045.stdout:334: done (3 left) 2024-04-26T00:22:00.582 INFO:tasks.rados.rados.0.smithi045.stdout:335: done (2 left) 2024-04-26T00:22:00.582 INFO:tasks.rados.rados.0.smithi045.stdout:336: done (1 left) 2024-04-26T00:22:00.582 INFO:tasks.rados.rados.0.smithi045.stdout:337: done (0 left) 2024-04-26T00:22:00.582 INFO:tasks.rados.rados.0.smithi045.stdout:338: write (excl) oid 33 current snap is 32 2024-04-26T00:22:00.583 INFO:tasks.rados.rados.0.smithi045.stdout:338: seq_num 150 ranges {521050=698057,1877946=643104} 2024-04-26T00:22:00.601 INFO:tasks.rados.rados.0.smithi045.stdout:338: writing smithi04526579-33 from 521050 to 1219107 tid 1 2024-04-26T00:22:00.623 INFO:tasks.rados.rados.0.smithi045.stdout:338: writing smithi04526579-33 from 1877946 to 2521050 tid 2 2024-04-26T00:22:00.623 INFO:tasks.rados.rados.0.smithi045.stdout:339: read oid 42 snap 31 2024-04-26T00:22:00.623 INFO:tasks.rados.rados.0.smithi045.stdout:339: expect (ObjNum 114 snap 28 seq_num 114) 2024-04-26T00:22:00.623 INFO:tasks.rados.rados.0.smithi045.stdout:340: read oid 50 snap -1 2024-04-26T00:22:00.623 INFO:tasks.rados.rados.0.smithi045.stdout:340: expect (ObjNum 72 snap 11 seq_num 72) 2024-04-26T00:22:00.624 INFO:tasks.rados.rados.0.smithi045.stdout:341: read oid 47 snap -1 2024-04-26T00:22:00.624 INFO:tasks.rados.rados.0.smithi045.stdout:341: expect (ObjNum 97 snap 22 seq_num 97) 2024-04-26T00:22:00.624 INFO:tasks.rados.rados.0.smithi045.stdout:342: read oid 25 snap -1 2024-04-26T00:22:00.624 INFO:tasks.rados.rados.0.smithi045.stdout:342: expect (ObjNum 72 snap 11 seq_num 72) 2024-04-26T00:22:00.624 INFO:tasks.rados.rados.0.smithi045.stdout:343: write (excl) oid 14 current snap is 32 2024-04-26T00:22:00.624 INFO:tasks.rados.rados.0.smithi045.stdout:343: seq_num 151 ranges {537857=773306,1936428=520715} 2024-04-26T00:22:00.647 INFO:tasks.rados.rados.0.smithi045.stdout:343: writing smithi04526579-14 from 537857 to 1311163 tid 1 2024-04-26T00:22:00.659 INFO:tasks.rados.rados.0.smithi045.stdout:343: writing smithi04526579-14 from 1936428 to 2457143 tid 2 2024-04-26T00:22:00.659 INFO:tasks.rados.rados.0.smithi045.stdout:344: write oid 45 current snap is 32 2024-04-26T00:22:00.659 INFO:tasks.rados.rados.0.smithi045.stdout:344: seq_num 152 ranges {554664=448555,1478557=664373} 2024-04-26T00:22:00.668 INFO:tasks.rados.rados.0.smithi045.stdout:344: writing smithi04526579-45 from 554664 to 1003219 tid 1 2024-04-26T00:22:00.678 INFO:tasks.rados.rados.0.smithi045.stdout:344: writing smithi04526579-45 from 1478557 to 2142930 tid 2 2024-04-26T00:22:00.678 INFO:tasks.rados.rados.0.smithi045.stdout:345: read oid 20 snap -1 2024-04-26T00:22:00.678 INFO:tasks.rados.rados.0.smithi045.stdout:345: expect (ObjNum 133 snap 30 seq_num 133) 2024-04-26T00:22:00.678 INFO:tasks.rados.rados.0.smithi045.stdout:346: write (excl) oid 18 current snap is 32 2024-04-26T00:22:00.678 INFO:tasks.rados.rados.0.smithi045.stdout:346: seq_num 153 ranges {571471=640157,1653392=524384} 2024-04-26T00:22:00.687 INFO:tasks.rados.rados.0.smithi045.stdout:346: writing smithi04526579-18 from 571471 to 1211628 tid 1 2024-04-26T00:22:00.694 INFO:tasks.rados.rados.0.smithi045.stdout:346: writing smithi04526579-18 from 1653392 to 2177776 tid 2 2024-04-26T00:22:00.694 INFO:tasks.rados.rados.0.smithi045.stdout:347: copy_from oid 4 from oid 9 current snap is 32 2024-04-26T00:22:00.695 INFO:tasks.rados.rados.0.smithi045.stdout:348: write oid 28 current snap is 32 2024-04-26T00:22:00.695 INFO:tasks.rados.rados.0.smithi045.stdout:348: seq_num 154 ranges {588278=715406,1711874=668042} 2024-04-26T00:22:00.702 INFO:tasks.rados.rados.0.smithi045.stdout:348: writing smithi04526579-28 from 588278 to 1303684 tid 1 2024-04-26T00:22:00.710 INFO:tasks.rados.rados.0.smithi045.stdout:348: writing smithi04526579-28 from 1711874 to 2379916 tid 2 2024-04-26T00:22:00.711 INFO:tasks.rados.rados.0.smithi045.stdout:349: read oid 44 snap -1 2024-04-26T00:22:00.711 INFO:tasks.rados.rados.0.smithi045.stdout:349: expect (ObjNum 119 snap 28 seq_num 119) 2024-04-26T00:22:00.711 INFO:tasks.rados.rados.0.smithi045.stdout:350: copy_from oid 30 from oid 31 current snap is 32 2024-04-26T00:22:00.711 INFO:tasks.rados.rados.0.smithi045.stdout:338: finishing write tid 1 to smithi04526579-33 2024-04-26T00:22:00.711 INFO:tasks.rados.rados.0.smithi045.stdout:340: expect (ObjNum 72 snap 11 seq_num 72) 2024-04-26T00:22:00.975 INFO:tasks.rados.rados.0.smithi045.stdout:339: expect (ObjNum 114 snap 28 seq_num 114) 2024-04-26T00:22:01.106 INFO:tasks.rados.rados.0.smithi045.stdout:341: expect (ObjNum 97 snap 22 seq_num 97) 2024-04-26T00:22:01.392 INFO:tasks.rados.rados.0.smithi045.stdout:342: expect (ObjNum 72 snap 11 seq_num 72) 2024-04-26T00:22:01.595 INFO:tasks.rados.rados.0.smithi045.stdout:338: finishing write tid 2 to smithi04526579-33 2024-04-26T00:22:01.595 INFO:tasks.rados.rados.0.smithi045.stdout:338: finishing write tid 3 to smithi04526579-33 2024-04-26T00:22:01.595 INFO:tasks.rados.rados.0.smithi045.stdout:338: finishing write tid 4 to smithi04526579-33 2024-04-26T00:22:01.595 INFO:tasks.rados.rados.0.smithi045.stdout:338: oid 33 updating version 0 to 203 2024-04-26T00:22:01.595 INFO:tasks.rados.rados.0.smithi045.stdout:338: oid 33 version 203 is already newer than 202 2024-04-26T00:22:01.595 INFO:tasks.rados.rados.0.smithi045.stdout:338: oid 33 version 203 is already newer than 201 2024-04-26T00:22:01.595 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 203 (ObjNum 150 snap 32 seq_num 150) dirty exists 2024-04-26T00:22:01.596 INFO:tasks.rados.rados.0.smithi045.stdout:338: left oid 33 (ObjNum 150 snap 32 seq_num 150) 2024-04-26T00:22:01.596 INFO:tasks.rados.rados.0.smithi045.stdout:343: finishing write tid 1 to smithi04526579-14 2024-04-26T00:22:01.596 INFO:tasks.rados.rados.0.smithi045.stdout:343: finishing write tid 2 to smithi04526579-14 2024-04-26T00:22:01.596 INFO:tasks.rados.rados.0.smithi045.stdout:343: finishing write tid 3 to smithi04526579-14 2024-04-26T00:22:01.596 INFO:tasks.rados.rados.0.smithi045.stdout:343: finishing write tid 4 to smithi04526579-14 2024-04-26T00:22:01.596 INFO:tasks.rados.rados.0.smithi045.stdout:343: oid 14 updating version 0 to 21 2024-04-26T00:22:01.596 INFO:tasks.rados.rados.0.smithi045.stdout:343: oid 14 updating version 21 to 22 2024-04-26T00:22:01.596 INFO:tasks.rados.rados.0.smithi045.stdout:343: oid 14 updating version 22 to 23 2024-04-26T00:22:01.596 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 23 (ObjNum 151 snap 32 seq_num 151) dirty exists 2024-04-26T00:22:01.596 INFO:tasks.rados.rados.0.smithi045.stdout:343: left oid 14 (ObjNum 151 snap 32 seq_num 151) 2024-04-26T00:22:01.596 INFO:tasks.rados.rados.0.smithi045.stdout:344: finishing write tid 1 to smithi04526579-45 2024-04-26T00:22:01.596 INFO:tasks.rados.rados.0.smithi045.stdout:344: finishing write tid 2 to smithi04526579-45 2024-04-26T00:22:01.596 INFO:tasks.rados.rados.0.smithi045.stdout:345: expect (ObjNum 133 snap 30 seq_num 133) 2024-04-26T00:22:01.694 INFO:tasks.rados.rados.0.smithi045.stdout:344: finishing write tid 3 to smithi04526579-45 2024-04-26T00:22:01.694 INFO:tasks.rados.rados.0.smithi045.stdout:344: finishing write tid 4 to smithi04526579-45 2024-04-26T00:22:01.694 INFO:tasks.rados.rados.0.smithi045.stdout:344: oid 45 updating version 0 to 87 2024-04-26T00:22:01.694 INFO:tasks.rados.rados.0.smithi045.stdout:344: oid 45 version 87 is already newer than 85 2024-04-26T00:22:01.695 INFO:tasks.rados.rados.0.smithi045.stdout:344: oid 45 version 87 is already newer than 86 2024-04-26T00:22:01.695 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 87 (ObjNum 152 snap 32 seq_num 152) dirty exists 2024-04-26T00:22:01.695 INFO:tasks.rados.rados.0.smithi045.stdout:344: left oid 45 (ObjNum 152 snap 32 seq_num 152) 2024-04-26T00:22:01.695 INFO:tasks.rados.rados.0.smithi045.stdout:346: finishing write tid 1 to smithi04526579-18 2024-04-26T00:22:01.695 INFO:tasks.rados.rados.0.smithi045.stdout:346: finishing write tid 2 to smithi04526579-18 2024-04-26T00:22:01.695 INFO:tasks.rados.rados.0.smithi045.stdout:346: finishing write tid 3 to smithi04526579-18 2024-04-26T00:22:01.695 INFO:tasks.rados.rados.0.smithi045.stdout:346: finishing write tid 4 to smithi04526579-18 2024-04-26T00:22:01.695 INFO:tasks.rados.rados.0.smithi045.stdout:346: oid 18 updating version 0 to 204 2024-04-26T00:22:01.695 INFO:tasks.rados.rados.0.smithi045.stdout:346: oid 18 updating version 204 to 206 2024-04-26T00:22:01.695 INFO:tasks.rados.rados.0.smithi045.stdout:346: oid 18 version 206 is already newer than 205 2024-04-26T00:22:01.696 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 206 (ObjNum 153 snap 32 seq_num 153) dirty exists 2024-04-26T00:22:01.696 INFO:tasks.rados.rados.0.smithi045.stdout:346: left oid 18 (ObjNum 153 snap 32 seq_num 153) 2024-04-26T00:22:01.696 INFO:tasks.rados.rados.0.smithi045.stdout:348: finishing write tid 1 to smithi04526579-28 2024-04-26T00:22:01.696 INFO:tasks.rados.rados.0.smithi045.stdout:348: finishing write tid 2 to smithi04526579-28 2024-04-26T00:22:01.696 INFO:tasks.rados.rados.0.smithi045.stdout:348: finishing write tid 3 to smithi04526579-28 2024-04-26T00:22:01.696 INFO:tasks.rados.rados.0.smithi045.stdout:348: finishing write tid 4 to smithi04526579-28 2024-04-26T00:22:01.696 INFO:tasks.rados.rados.0.smithi045.stdout:348: oid 28 updating version 0 to 149 2024-04-26T00:22:01.696 INFO:tasks.rados.rados.0.smithi045.stdout:348: oid 28 updating version 149 to 150 2024-04-26T00:22:01.696 INFO:tasks.rados.rados.0.smithi045.stdout:348: oid 28 updating version 150 to 151 2024-04-26T00:22:01.696 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 151 (ObjNum 154 snap 32 seq_num 154) dirty exists 2024-04-26T00:22:01.696 INFO:tasks.rados.rados.0.smithi045.stdout:348: left oid 28 (ObjNum 154 snap 32 seq_num 154) 2024-04-26T00:22:01.697 INFO:tasks.rados.rados.0.smithi045.stdout:347: finishing copy_from racing read to smithi04526579-4 2024-04-26T00:22:01.697 INFO:tasks.rados.rados.0.smithi045.stdout:347: finishing copy_from to smithi04526579-4 2024-04-26T00:22:01.697 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 12 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-04-26T00:22:01.697 INFO:tasks.rados.rados.0.smithi045.stdout:349: expect (ObjNum 119 snap 28 seq_num 119) 2024-04-26T00:22:01.776 INFO:tasks.rados.rados.0.smithi045.stdout:350: finishing copy_from racing read to smithi04526579-30 2024-04-26T00:22:01.776 INFO:tasks.rados.rados.0.smithi045.stdout:350: finishing copy_from to smithi04526579-30 2024-04-26T00:22:01.776 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 164 (ObjNum 149 snap 32 seq_num 149) dirty exists 2024-04-26T00:22:01.776 INFO:tasks.rados.rados.0.smithi045.stdout:338: done (12 left) 2024-04-26T00:22:01.776 INFO:tasks.rados.rados.0.smithi045.stdout:339: done (11 left) 2024-04-26T00:22:01.776 INFO:tasks.rados.rados.0.smithi045.stdout:340: done (10 left) 2024-04-26T00:22:01.776 INFO:tasks.rados.rados.0.smithi045.stdout:341: done (9 left) 2024-04-26T00:22:01.776 INFO:tasks.rados.rados.0.smithi045.stdout:342: done (8 left) 2024-04-26T00:22:01.776 INFO:tasks.rados.rados.0.smithi045.stdout:343: done (7 left) 2024-04-26T00:22:01.776 INFO:tasks.rados.rados.0.smithi045.stdout:344: done (6 left) 2024-04-26T00:22:01.776 INFO:tasks.rados.rados.0.smithi045.stdout:345: done (5 left) 2024-04-26T00:22:01.776 INFO:tasks.rados.rados.0.smithi045.stdout:346: done (4 left) 2024-04-26T00:22:01.777 INFO:tasks.rados.rados.0.smithi045.stdout:347: done (3 left) 2024-04-26T00:22:01.777 INFO:tasks.rados.rados.0.smithi045.stdout:348: done (2 left) 2024-04-26T00:22:01.777 INFO:tasks.rados.rados.0.smithi045.stdout:349: done (1 left) 2024-04-26T00:22:01.777 INFO:tasks.rados.rados.0.smithi045.stdout:350: done (0 left) 2024-04-26T00:22:01.777 INFO:tasks.rados.rados.0.smithi045.stdout:351: snap_create 2024-04-26T00:22:02.008 INFO:tasks.rados.rados.0.smithi045.stdout:351: done (0 left) 2024-04-26T00:22:02.008 INFO:tasks.rados.rados.0.smithi045.stdout:352: copy_from oid 19 from oid 29 current snap is 33 2024-04-26T00:22:02.009 INFO:tasks.rados.rados.0.smithi045.stdout:353: read oid 49 snap -1 2024-04-26T00:22:02.009 INFO:tasks.rados.rados.0.smithi045.stdout:353: expect (ObjNum 48 snap 0 seq_num 48) 2024-04-26T00:22:02.009 INFO:tasks.rados.rados.0.smithi045.stdout:354: write (excl) oid 11 current snap is 33 2024-04-26T00:22:02.009 INFO:tasks.rados.rados.0.smithi045.stdout:354: seq_num 155 ranges {605085=790655,2170356=434729} 2024-04-26T00:22:02.031 INFO:tasks.rados.rados.0.smithi045.stdout:354: writing smithi04526579-11 from 605085 to 1395740 tid 1 2024-04-26T00:22:02.051 INFO:tasks.rados.rados.0.smithi045.stdout:354: writing smithi04526579-11 from 2170356 to 2605085 tid 2 2024-04-26T00:22:02.051 INFO:tasks.rados.rados.0.smithi045.stdout:355: write (excl) oid 23 current snap is 33 2024-04-26T00:22:02.051 INFO:tasks.rados.rados.0.smithi045.stdout:355: seq_num 156 ranges {621892=465904,1712485=671711} 2024-04-26T00:22:02.066 INFO:tasks.rados.rados.0.smithi045.stdout:355: writing smithi04526579-23 from 621892 to 1087796 tid 1 2024-04-26T00:22:02.081 INFO:tasks.rados.rados.0.smithi045.stdout:355: writing smithi04526579-23 from 1712485 to 2384196 tid 2 2024-04-26T00:22:02.082 INFO:tasks.rados.rados.0.smithi045.stdout:356: delete oid 38 current snap is 33 2024-04-26T00:22:02.082 INFO:tasks.rados.rados.0.smithi045.stdout:353: expect (ObjNum 48 snap 0 seq_num 48) 2024-04-26T00:22:02.193 INFO:tasks.rados.rados.0.smithi045.stdout:352: finishing copy_from racing read to smithi04526579-19 2024-04-26T00:22:02.193 INFO:tasks.rados.rados.0.smithi045.stdout:354: finishing write tid 1 to smithi04526579-11 2024-04-26T00:22:02.193 INFO:tasks.rados.rados.0.smithi045.stdout:352: finishing copy_from to smithi04526579-19 2024-04-26T00:22:02.193 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 58 (ObjNum 120 snap 28 seq_num 120) dirty exists 2024-04-26T00:22:02.193 INFO:tasks.rados.rados.0.smithi045.stdout:354: finishing write tid 2 to smithi04526579-11 2024-04-26T00:22:02.193 INFO:tasks.rados.rados.0.smithi045.stdout:354: finishing write tid 3 to smithi04526579-11 2024-04-26T00:22:02.193 INFO:tasks.rados.rados.0.smithi045.stdout:354: finishing write tid 4 to smithi04526579-11 2024-04-26T00:22:02.193 INFO:tasks.rados.rados.0.smithi045.stdout:354: oid 11 updating version 0 to 88 2024-04-26T00:22:02.193 INFO:tasks.rados.rados.0.smithi045.stdout:354: oid 11 updating version 88 to 89 2024-04-26T00:22:02.193 INFO:tasks.rados.rados.0.smithi045.stdout:354: oid 11 updating version 89 to 90 2024-04-26T00:22:02.193 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 90 (ObjNum 155 snap 33 seq_num 155) dirty exists 2024-04-26T00:22:02.193 INFO:tasks.rados.rados.0.smithi045.stdout:354: left oid 11 (ObjNum 155 snap 33 seq_num 155) 2024-04-26T00:22:02.193 INFO:tasks.rados.rados.0.smithi045.stdout:355: finishing write tid 1 to smithi04526579-23 2024-04-26T00:22:02.194 INFO:tasks.rados.rados.0.smithi045.stdout:355: finishing write tid 2 to smithi04526579-23 2024-04-26T00:22:02.194 INFO:tasks.rados.rados.0.smithi045.stdout:355: finishing write tid 3 to smithi04526579-23 2024-04-26T00:22:02.194 INFO:tasks.rados.rados.0.smithi045.stdout:355: finishing write tid 4 to smithi04526579-23 2024-04-26T00:22:02.194 INFO:tasks.rados.rados.0.smithi045.stdout:355: oid 23 updating version 0 to 77 2024-04-26T00:22:02.194 INFO:tasks.rados.rados.0.smithi045.stdout:355: oid 23 version 77 is already newer than 76 2024-04-26T00:22:02.194 INFO:tasks.rados.rados.0.smithi045.stdout:355: oid 23 version 77 is already newer than 75 2024-04-26T00:22:02.194 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 77 (ObjNum 156 snap 33 seq_num 156) dirty exists 2024-04-26T00:22:02.194 INFO:tasks.rados.rados.0.smithi045.stdout:355: left oid 23 (ObjNum 156 snap 33 seq_num 156) 2024-04-26T00:22:02.194 INFO:tasks.rados.rados.0.smithi045.stdout:352: done (4 left) 2024-04-26T00:22:02.194 INFO:tasks.rados.rados.0.smithi045.stdout:353: done (3 left) 2024-04-26T00:22:02.194 INFO:tasks.rados.rados.0.smithi045.stdout:354: done (2 left) 2024-04-26T00:22:02.194 INFO:tasks.rados.rados.0.smithi045.stdout:355: done (1 left) 2024-04-26T00:22:02.194 INFO:tasks.rados.rados.0.smithi045.stdout:356: done (0 left) 2024-04-26T00:22:02.194 INFO:tasks.rados.rados.0.smithi045.stdout:357: rollback oid 32 current snap is 33 2024-04-26T00:22:02.194 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 32 to 31 2024-04-26T00:22:02.195 INFO:tasks.rados.rados.0.smithi045.stdout:358: snap_create 2024-04-26T00:22:02.201 INFO:tasks.rados.rados.0.smithi045.stdout:357: finishing rollback tid 0 to smithi04526579-32 2024-04-26T00:22:02.203 INFO:tasks.rados.rados.0.smithi045.stdout:357: finishing rollback tid 1 to smithi04526579-32 2024-04-26T00:22:02.203 INFO:tasks.rados.rados.0.smithi045.stdout:357: finishing rollback tid 2 to smithi04526579-32 2024-04-26T00:22:02.203 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 101 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-04-26T00:22:02.913 INFO:tasks.rados.rados.0.smithi045.stdout:357: done (1 left) 2024-04-26T00:22:02.913 INFO:tasks.rados.rados.0.smithi045.stdout:358: done (0 left) 2024-04-26T00:22:02.913 INFO:tasks.rados.rados.0.smithi045.stdout:359: delete oid 45 current snap is 34 2024-04-26T00:22:02.928 INFO:tasks.rados.rados.0.smithi045.stdout:359: done (0 left) 2024-04-26T00:22:02.928 INFO:tasks.rados.rados.0.smithi045.stdout:360: delete oid 39 current snap is 34 2024-04-26T00:22:02.932 INFO:tasks.rados.rados.0.smithi045.stdout:360: done (0 left) 2024-04-26T00:22:02.932 INFO:tasks.rados.rados.0.smithi045.stdout:361: write (excl) oid 39 current snap is 34 2024-04-26T00:22:02.932 INFO:tasks.rados.rados.0.smithi045.stdout:361: seq_num 160 ranges {689120=483253,1946413=679049} 2024-04-26T00:22:02.951 INFO:tasks.rados.rados.0.smithi045.stdout:361: writing smithi04526579-39 from 689120 to 1172373 tid 1 2024-04-26T00:22:02.975 INFO:tasks.rados.rados.0.smithi045.stdout:361: writing smithi04526579-39 from 1946413 to 2625462 tid 2 2024-04-26T00:22:02.975 INFO:tasks.rados.rados.0.smithi045.stdout:362: snap_create 2024-04-26T00:22:02.975 INFO:tasks.rados.rados.0.smithi045.stdout:361: finishing write tid 1 to smithi04526579-39 2024-04-26T00:22:02.982 INFO:tasks.rados.rados.0.smithi045.stdout:361: finishing write tid 2 to smithi04526579-39 2024-04-26T00:22:02.982 INFO:tasks.rados.rados.0.smithi045.stdout:361: finishing write tid 3 to smithi04526579-39 2024-04-26T00:22:02.983 INFO:tasks.rados.rados.0.smithi045.stdout:361: finishing write tid 4 to smithi04526579-39 2024-04-26T00:22:02.983 INFO:tasks.rados.rados.0.smithi045.stdout:361: oid 39 updating version 0 to 113 2024-04-26T00:22:02.983 INFO:tasks.rados.rados.0.smithi045.stdout:361: oid 39 version 113 is already newer than 112 2024-04-26T00:22:02.983 INFO:tasks.rados.rados.0.smithi045.stdout:361: oid 39 version 113 is already newer than 111 2024-04-26T00:22:02.983 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 113 (ObjNum 160 snap 34 seq_num 160) dirty exists 2024-04-26T00:22:02.983 INFO:tasks.rados.rados.0.smithi045.stdout:361: left oid 39 (ObjNum 160 snap 34 seq_num 160) 2024-04-26T00:22:03.925 INFO:tasks.rados.rados.0.smithi045.stdout:361: done (1 left) 2024-04-26T00:22:03.925 INFO:tasks.rados.rados.0.smithi045.stdout:362: done (0 left) 2024-04-26T00:22:03.926 INFO:tasks.rados.rados.0.smithi045.stdout:363: snap_create 2024-04-26T00:22:04.933 INFO:tasks.rados.rados.0.smithi045.stdout:363: done (0 left) 2024-04-26T00:22:04.933 INFO:tasks.rados.rados.0.smithi045.stdout:364: snap_remove snap 31 2024-04-26T00:22:05.941 INFO:tasks.rados.rados.0.smithi045.stdout:364: done (0 left) 2024-04-26T00:22:05.941 INFO:tasks.rados.rados.0.smithi045.stdout:365: write (excl) oid 24 current snap is 36 2024-04-26T00:22:05.942 INFO:tasks.rados.rados.0.smithi045.stdout:365: seq_num 161 ranges {705927=558502,2004895=422707} 2024-04-26T00:22:05.962 INFO:tasks.rados.rados.0.smithi045.stdout:365: writing smithi04526579-24 from 705927 to 1264429 tid 1 2024-04-26T00:22:05.981 INFO:tasks.rados.rados.0.smithi045.stdout:365: writing smithi04526579-24 from 2004895 to 2427602 tid 2 2024-04-26T00:22:05.981 INFO:tasks.rados.rados.0.smithi045.stdout:366: rollback oid 32 current snap is 36 2024-04-26T00:22:05.982 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 32 to 32 2024-04-26T00:22:05.982 INFO:tasks.rados.rados.0.smithi045.stdout:365: finishing write tid 1 to smithi04526579-24 2024-04-26T00:22:05.982 INFO:tasks.rados.rados.0.smithi045.stdout:367: rollback oid 22 current snap is 36 2024-04-26T00:22:05.982 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 22 to 34 2024-04-26T00:22:05.982 INFO:tasks.rados.rados.0.smithi045.stdout:368: snap_create 2024-04-26T00:22:05.994 INFO:tasks.rados.rados.0.smithi045.stdout:365: finishing write tid 2 to smithi04526579-24 2024-04-26T00:22:05.994 INFO:tasks.rados.rados.0.smithi045.stdout:365: finishing write tid 3 to smithi04526579-24 2024-04-26T00:22:05.995 INFO:tasks.rados.rados.0.smithi045.stdout:365: finishing write tid 4 to smithi04526579-24 2024-04-26T00:22:05.995 INFO:tasks.rados.rados.0.smithi045.stdout:365: oid 24 updating version 0 to 213 2024-04-26T00:22:05.995 INFO:tasks.rados.rados.0.smithi045.stdout:365: oid 24 version 213 is already newer than 212 2024-04-26T00:22:05.995 INFO:tasks.rados.rados.0.smithi045.stdout:365: oid 24 updating version 213 to 214 2024-04-26T00:22:05.995 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 214 (ObjNum 161 snap 36 seq_num 161) dirty exists 2024-04-26T00:22:05.995 INFO:tasks.rados.rados.0.smithi045.stdout:365: left oid 24 (ObjNum 161 snap 36 seq_num 161) 2024-04-26T00:22:05.996 INFO:tasks.rados.rados.0.smithi045.stdout:366: finishing rollback tid 0 to smithi04526579-32 2024-04-26T00:22:05.997 INFO:tasks.rados.rados.0.smithi045.stdout:366: finishing rollback tid 1 to smithi04526579-32 2024-04-26T00:22:05.997 INFO:tasks.rados.rados.0.smithi045.stdout:366: finishing rollback tid 2 to smithi04526579-32 2024-04-26T00:22:05.997 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 109 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-04-26T00:22:05.998 INFO:tasks.rados.rados.0.smithi045.stdout:367: finishing rollback tid 0 to smithi04526579-22 2024-04-26T00:22:06.000 INFO:tasks.rados.rados.0.smithi045.stdout:367: finishing rollback tid 1 to smithi04526579-22 2024-04-26T00:22:06.000 INFO:tasks.rados.rados.0.smithi045.stdout:367: finishing rollback tid 2 to smithi04526579-22 2024-04-26T00:22:06.000 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 106 (ObjNum 134 snap 30 seq_num 134) dirty exists 2024-04-26T00:22:06.953 INFO:tasks.rados.rados.0.smithi045.stdout:365: done (3 left) 2024-04-26T00:22:06.954 INFO:tasks.rados.rados.0.smithi045.stdout:366: done (2 left) 2024-04-26T00:22:06.954 INFO:tasks.rados.rados.0.smithi045.stdout:367: done (1 left) 2024-04-26T00:22:06.954 INFO:tasks.rados.rados.0.smithi045.stdout:368: done (0 left) 2024-04-26T00:22:06.954 INFO:tasks.rados.rados.0.smithi045.stdout:369: delete oid 21 current snap is 37 2024-04-26T00:22:06.961 INFO:tasks.rados.rados.0.smithi045.stdout:369: done (0 left) 2024-04-26T00:22:06.961 INFO:tasks.rados.rados.0.smithi045.stdout:370: snap_remove snap 34 2024-04-26T00:22:07.916 INFO:tasks.rados.rados.0.smithi045.stdout:370: done (0 left) 2024-04-26T00:22:07.916 INFO:tasks.rados.rados.0.smithi045.stdout:371: write (excl) oid 15 current snap is 37 2024-04-26T00:22:07.916 INFO:tasks.rados.rados.0.smithi045.stdout:371: seq_num 163 ranges {739541=709000,2121859=426376} 2024-04-26T00:22:07.939 INFO:tasks.rados.rados.0.smithi045.stdout:371: writing smithi04526579-15 from 739541 to 1448541 tid 1 2024-04-26T00:22:07.960 INFO:tasks.rados.rados.0.smithi045.stdout:371: writing smithi04526579-15 from 2121859 to 2548235 tid 2 2024-04-26T00:22:07.960 INFO:tasks.rados.rados.0.smithi045.stdout:372: snap_create 2024-04-26T00:22:07.960 INFO:tasks.rados.rados.0.smithi045.stdout:371: finishing write tid 1 to smithi04526579-15 2024-04-26T00:22:07.966 INFO:tasks.rados.rados.0.smithi045.stdout:371: finishing write tid 2 to smithi04526579-15 2024-04-26T00:22:07.967 INFO:tasks.rados.rados.0.smithi045.stdout:371: finishing write tid 3 to smithi04526579-15 2024-04-26T00:22:07.967 INFO:tasks.rados.rados.0.smithi045.stdout:371: finishing write tid 4 to smithi04526579-15 2024-04-26T00:22:07.968 INFO:tasks.rados.rados.0.smithi045.stdout:371: oid 15 updating version 0 to 154 2024-04-26T00:22:07.968 INFO:tasks.rados.rados.0.smithi045.stdout:371: oid 15 version 154 is already newer than 153 2024-04-26T00:22:07.968 INFO:tasks.rados.rados.0.smithi045.stdout:371: oid 15 version 154 is already newer than 152 2024-04-26T00:22:07.968 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 154 (ObjNum 163 snap 37 seq_num 163) dirty exists 2024-04-26T00:22:07.968 INFO:tasks.rados.rados.0.smithi045.stdout:371: left oid 15 (ObjNum 163 snap 37 seq_num 163) 2024-04-26T00:22:08.980 INFO:tasks.rados.rados.0.smithi045.stdout:371: done (1 left) 2024-04-26T00:22:08.980 INFO:tasks.rados.rados.0.smithi045.stdout:372: done (0 left) 2024-04-26T00:22:08.980 INFO:tasks.rados.rados.0.smithi045.stdout:373: write (excl) oid 7 current snap is 38 2024-04-26T00:22:08.980 INFO:tasks.rados.rados.0.smithi045.stdout:373: seq_num 164 ranges {756348=784249,2063988=686387} 2024-04-26T00:22:09.005 INFO:tasks.rados.rados.0.smithi045.stdout:373: writing smithi04526579-7 from 756348 to 1540597 tid 1 2024-04-26T00:22:09.025 INFO:tasks.rados.rados.0.smithi045.stdout:373: writing smithi04526579-7 from 2063988 to 2750375 tid 2 2024-04-26T00:22:09.025 INFO:tasks.rados.rados.0.smithi045.stdout:374: write (excl) oid 18 current snap is 38 2024-04-26T00:22:09.025 INFO:tasks.rados.rados.0.smithi045.stdout:373: finishing write tid 1 to smithi04526579-7 2024-04-26T00:22:09.025 INFO:tasks.rados.rados.0.smithi045.stdout:374: seq_num 165 ranges {773155=459498,1722470=430045,2648676=124479} 2024-04-26T00:22:09.045 INFO:tasks.rados.rados.0.smithi045.stdout:374: writing smithi04526579-18 from 773155 to 1232653 tid 1 2024-04-26T00:22:09.058 INFO:tasks.rados.rados.0.smithi045.stdout:374: writing smithi04526579-18 from 1722470 to 2152515 tid 2 2024-04-26T00:22:09.067 INFO:tasks.rados.rados.0.smithi045.stdout:374: writing smithi04526579-18 from 2648676 to 2773155 tid 3 2024-04-26T00:22:09.067 INFO:tasks.rados.rados.0.smithi045.stdout:375: snap_create 2024-04-26T00:22:09.067 INFO:tasks.rados.rados.0.smithi045.stdout:373: finishing write tid 2 to smithi04526579-7 2024-04-26T00:22:09.067 INFO:tasks.rados.rados.0.smithi045.stdout:373: finishing write tid 3 to smithi04526579-7 2024-04-26T00:22:09.067 INFO:tasks.rados.rados.0.smithi045.stdout:373: finishing write tid 4 to smithi04526579-7 2024-04-26T00:22:09.067 INFO:tasks.rados.rados.0.smithi045.stdout:373: oid 7 updating version 0 to 218 2024-04-26T00:22:09.067 INFO:tasks.rados.rados.0.smithi045.stdout:373: oid 7 updating version 218 to 220 2024-04-26T00:22:09.067 INFO:tasks.rados.rados.0.smithi045.stdout:373: oid 7 version 220 is already newer than 219 2024-04-26T00:22:09.067 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 220 (ObjNum 164 snap 38 seq_num 164) dirty exists 2024-04-26T00:22:09.068 INFO:tasks.rados.rados.0.smithi045.stdout:373: left oid 7 (ObjNum 164 snap 38 seq_num 164) 2024-04-26T00:22:09.068 INFO:tasks.rados.rados.0.smithi045.stdout:374: finishing write tid 1 to smithi04526579-18 2024-04-26T00:22:09.068 INFO:tasks.rados.rados.0.smithi045.stdout:374: finishing write tid 2 to smithi04526579-18 2024-04-26T00:22:09.074 INFO:tasks.rados.rados.0.smithi045.stdout:374: finishing write tid 3 to smithi04526579-18 2024-04-26T00:22:09.077 INFO:tasks.rados.rados.0.smithi045.stdout:374: finishing write tid 4 to smithi04526579-18 2024-04-26T00:22:09.078 INFO:tasks.rados.rados.0.smithi045.stdout:374: finishing write tid 5 to smithi04526579-18 2024-04-26T00:22:09.078 INFO:tasks.rados.rados.0.smithi045.stdout:374: oid 18 updating version 0 to 222 2024-04-26T00:22:09.078 INFO:tasks.rados.rados.0.smithi045.stdout:374: oid 18 updating version 222 to 224 2024-04-26T00:22:09.078 INFO:tasks.rados.rados.0.smithi045.stdout:374: oid 18 version 224 is already newer than 223 2024-04-26T00:22:09.078 INFO:tasks.rados.rados.0.smithi045.stdout:374: oid 18 updating version 224 to 225 2024-04-26T00:22:09.078 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 225 (ObjNum 165 snap 38 seq_num 165) dirty exists 2024-04-26T00:22:09.078 INFO:tasks.rados.rados.0.smithi045.stdout:374: left oid 18 (ObjNum 165 snap 38 seq_num 165) 2024-04-26T00:22:09.988 INFO:tasks.rados.rados.0.smithi045.stdout:373: done (2 left) 2024-04-26T00:22:09.988 INFO:tasks.rados.rados.0.smithi045.stdout:374: done (1 left) 2024-04-26T00:22:09.988 INFO:tasks.rados.rados.0.smithi045.stdout:375: done (0 left) 2024-04-26T00:22:09.988 INFO:tasks.rados.rados.0.smithi045.stdout:376: rollback oid 17 current snap is 39 2024-04-26T00:22:09.988 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 17 to 33 2024-04-26T00:22:09.988 INFO:tasks.rados.rados.0.smithi045.stdout:377: write oid 40 current snap is 39 2024-04-26T00:22:09.988 INFO:tasks.rados.rados.0.smithi045.stdout:377: seq_num 166 ranges {789962=534747,1780952=690056} 2024-04-26T00:22:10.003 INFO:tasks.rados.rados.0.smithi045.stdout:377: writing smithi04526579-40 from 789962 to 1324709 tid 1 2024-04-26T00:22:10.016 INFO:tasks.rados.rados.0.smithi045.stdout:377: writing smithi04526579-40 from 1780952 to 2471008 tid 2 2024-04-26T00:22:10.016 INFO:tasks.rados.rados.0.smithi045.stdout:378: read oid 7 snap -1 2024-04-26T00:22:10.017 INFO:tasks.rados.rados.0.smithi045.stdout:378: expect (ObjNum 164 snap 38 seq_num 164) 2024-04-26T00:22:10.017 INFO:tasks.rados.rados.0.smithi045.stdout:379: snap_remove snap 38 2024-04-26T00:22:11.498 INFO:tasks.rados.rados.0.smithi045.stdout:379: done (3 left) 2024-04-26T00:22:11.499 INFO:tasks.rados.rados.0.smithi045.stdout:380: rollback oid 43 current snap is 39 2024-04-26T00:22:11.499 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 43 to 33 2024-04-26T00:22:11.499 INFO:tasks.rados.rados.0.smithi045.stdout:376: finishing rollback tid 0 to smithi04526579-17 2024-04-26T00:22:11.499 INFO:tasks.rados.rados.0.smithi045.stdout:376: finishing rollback tid 1 to smithi04526579-17 2024-04-26T00:22:11.499 INFO:tasks.rados.rados.0.smithi045.stdout:376: finishing rollback tid 2 to smithi04526579-17 2024-04-26T00:22:11.499 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 138 (ObjNum 142 snap 31 seq_num 142) dirty exists 2024-04-26T00:22:11.499 INFO:tasks.rados.rados.0.smithi045.stdout:377: finishing write tid 1 to smithi04526579-40 2024-04-26T00:22:11.499 INFO:tasks.rados.rados.0.smithi045.stdout:377: finishing write tid 2 to smithi04526579-40 2024-04-26T00:22:11.499 INFO:tasks.rados.rados.0.smithi045.stdout:377: finishing write tid 3 to smithi04526579-40 2024-04-26T00:22:11.499 INFO:tasks.rados.rados.0.smithi045.stdout:377: finishing write tid 4 to smithi04526579-40 2024-04-26T00:22:11.499 INFO:tasks.rados.rados.0.smithi045.stdout:377: oid 40 updating version 0 to 85 2024-04-26T00:22:11.499 INFO:tasks.rados.rados.0.smithi045.stdout:377: oid 40 version 85 is already newer than 84 2024-04-26T00:22:11.500 INFO:tasks.rados.rados.0.smithi045.stdout:377: oid 40 version 85 is already newer than 83 2024-04-26T00:22:11.500 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 85 (ObjNum 166 snap 39 seq_num 166) dirty exists 2024-04-26T00:22:11.500 INFO:tasks.rados.rados.0.smithi045.stdout:377: left oid 40 (ObjNum 166 snap 39 seq_num 166) 2024-04-26T00:22:11.500 INFO:tasks.rados.rados.0.smithi045.stdout:378: expect (ObjNum 164 snap 38 seq_num 164) 2024-04-26T00:22:11.685 INFO:tasks.rados.rados.0.smithi045.stdout:380: finishing rollback tid 0 to smithi04526579-43 2024-04-26T00:22:11.685 INFO:tasks.rados.rados.0.smithi045.stdout:380: finishing rollback tid 1 to smithi04526579-43 2024-04-26T00:22:11.685 INFO:tasks.rados.rados.0.smithi045.stdout:380: finishing rollback tid 2 to smithi04526579-43 2024-04-26T00:22:11.685 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 142 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-04-26T00:22:11.685 INFO:tasks.rados.rados.0.smithi045.stdout:376: done (3 left) 2024-04-26T00:22:11.685 INFO:tasks.rados.rados.0.smithi045.stdout:377: done (2 left) 2024-04-26T00:22:11.685 INFO:tasks.rados.rados.0.smithi045.stdout:378: done (1 left) 2024-04-26T00:22:11.686 INFO:tasks.rados.rados.0.smithi045.stdout:380: done (0 left) 2024-04-26T00:22:11.686 INFO:tasks.rados.rados.0.smithi045.stdout:381: snap_create 2024-04-26T00:22:12.499 INFO:tasks.rados.rados.0.smithi045.stdout:381: done (0 left) 2024-04-26T00:22:12.499 INFO:tasks.rados.rados.0.smithi045.stdout:382: snap_remove snap 36 2024-04-26T00:22:12.915 INFO:tasks.rados.rados.0.smithi045.stdout:382: done (0 left) 2024-04-26T00:22:12.916 INFO:tasks.rados.rados.0.smithi045.stdout:383: rollback oid 13 current snap is 40 2024-04-26T00:22:12.916 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 13 to 32 2024-04-26T00:22:12.916 INFO:tasks.rados.rados.0.smithi045.stdout:384: write oid 33 current snap is 40 2024-04-26T00:22:12.916 INFO:tasks.rados.rados.0.smithi045.stdout:384: seq_num 167 ranges {406769=609996,1439434=433714,2303522=503247} 2024-04-26T00:22:12.933 INFO:tasks.rados.rados.0.smithi045.stdout:384: writing smithi04526579-33 from 406769 to 1016765 tid 1 2024-04-26T00:22:12.948 INFO:tasks.rados.rados.0.smithi045.stdout:384: writing smithi04526579-33 from 1439434 to 1873148 tid 2 2024-04-26T00:22:12.963 INFO:tasks.rados.rados.0.smithi045.stdout:384: writing smithi04526579-33 from 2303522 to 2806769 tid 3 2024-04-26T00:22:12.964 INFO:tasks.rados.rados.0.smithi045.stdout:385: read oid 8 snap -1 2024-04-26T00:22:12.964 INFO:tasks.rados.rados.0.smithi045.stdout:385: expect (ObjNum 7 snap 0 seq_num 7) 2024-04-26T00:22:12.964 INFO:tasks.rados.rados.0.smithi045.stdout:386: write (excl) oid 37 current snap is 40 2024-04-26T00:22:12.964 INFO:tasks.rados.rados.0.smithi045.stdout:383: finishing rollback tid 1 to smithi04526579-13 2024-04-26T00:22:12.964 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 0 (ObjNum 177 snap 0 seq_num 2241675456) dirty dne 2024-04-26T00:22:12.964 INFO:tasks.rados.rados.0.smithi045.stdout:384: finishing write tid 1 to smithi04526579-33 2024-04-26T00:22:12.964 INFO:tasks.rados.rados.0.smithi045.stdout:384: finishing write tid 2 to smithi04526579-33 2024-04-26T00:22:12.964 INFO:tasks.rados.rados.0.smithi045.stdout:386: seq_num 168 ranges {423576=401598,1497916=693725} 2024-04-26T00:22:12.977 INFO:tasks.rados.rados.0.smithi045.stdout:386: writing smithi04526579-37 from 423576 to 825174 tid 1 2024-04-26T00:22:12.997 INFO:tasks.rados.rados.0.smithi045.stdout:386: writing smithi04526579-37 from 1497916 to 2191641 tid 2 2024-04-26T00:22:12.997 INFO:tasks.rados.rados.0.smithi045.stdout:383: done (3 left) 2024-04-26T00:22:12.997 INFO:tasks.rados.rados.0.smithi045.stdout:387: snap_create 2024-04-26T00:22:12.997 INFO:tasks.rados.rados.0.smithi045.stdout:384: finishing write tid 3 to smithi04526579-33 2024-04-26T00:22:12.997 INFO:tasks.rados.rados.0.smithi045.stdout:384: finishing write tid 4 to smithi04526579-33 2024-04-26T00:22:12.997 INFO:tasks.rados.rados.0.smithi045.stdout:384: finishing write tid 5 to smithi04526579-33 2024-04-26T00:22:12.998 INFO:tasks.rados.rados.0.smithi045.stdout:384: oid 33 updating version 0 to 233 2024-04-26T00:22:12.998 INFO:tasks.rados.rados.0.smithi045.stdout:384: oid 33 version 233 is already newer than 231 2024-04-26T00:22:12.998 INFO:tasks.rados.rados.0.smithi045.stdout:384: oid 33 updating version 233 to 234 2024-04-26T00:22:12.998 INFO:tasks.rados.rados.0.smithi045.stdout:384: oid 33 version 234 is already newer than 232 2024-04-26T00:22:12.998 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 234 (ObjNum 167 snap 40 seq_num 167) dirty exists 2024-04-26T00:22:12.998 INFO:tasks.rados.rados.0.smithi045.stdout:384: left oid 33 (ObjNum 167 snap 40 seq_num 167) 2024-04-26T00:22:12.998 INFO:tasks.rados.rados.0.smithi045.stdout:385: expect (ObjNum 7 snap 0 seq_num 7) 2024-04-26T00:22:13.092 INFO:tasks.rados.rados.0.smithi045.stdout:386: finishing write tid 1 to smithi04526579-37 2024-04-26T00:22:13.092 INFO:tasks.rados.rados.0.smithi045.stdout:386: finishing write tid 2 to smithi04526579-37 2024-04-26T00:22:13.092 INFO:tasks.rados.rados.0.smithi045.stdout:386: finishing write tid 3 to smithi04526579-37 2024-04-26T00:22:13.092 INFO:tasks.rados.rados.0.smithi045.stdout:386: finishing write tid 4 to smithi04526579-37 2024-04-26T00:22:13.092 INFO:tasks.rados.rados.0.smithi045.stdout:386: oid 37 updating version 0 to 235 2024-04-26T00:22:13.092 INFO:tasks.rados.rados.0.smithi045.stdout:386: oid 37 updating version 235 to 236 2024-04-26T00:22:13.093 INFO:tasks.rados.rados.0.smithi045.stdout:386: oid 37 updating version 236 to 237 2024-04-26T00:22:13.093 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 237 (ObjNum 168 snap 40 seq_num 168) dirty exists 2024-04-26T00:22:13.093 INFO:tasks.rados.rados.0.smithi045.stdout:386: left oid 37 (ObjNum 168 snap 40 seq_num 168) 2024-04-26T00:22:13.930 INFO:tasks.rados.rados.0.smithi045.stdout:384: done (3 left) 2024-04-26T00:22:13.931 INFO:tasks.rados.rados.0.smithi045.stdout:385: done (2 left) 2024-04-26T00:22:13.931 INFO:tasks.rados.rados.0.smithi045.stdout:386: done (1 left) 2024-04-26T00:22:13.931 INFO:tasks.rados.rados.0.smithi045.stdout:387: done (0 left) 2024-04-26T00:22:13.931 INFO:tasks.rados.rados.0.smithi045.stdout:388: snap_create 2024-04-26T00:22:14.932 INFO:tasks.rados.rados.0.smithi045.stdout:388: done (0 left) 2024-04-26T00:22:14.932 INFO:tasks.rados.rados.0.smithi045.stdout:389: snap_create 2024-04-26T00:22:15.934 INFO:tasks.rados.rados.0.smithi045.stdout:389: done (0 left) 2024-04-26T00:22:15.935 INFO:tasks.rados.rados.0.smithi045.stdout:390: read oid 45 snap -1 2024-04-26T00:22:15.935 INFO:tasks.rados.rados.0.smithi045.stdout:390: expect deleted 2024-04-26T00:22:15.935 INFO:tasks.rados.rados.0.smithi045.stdout:391: rollback oid 34 current snap is 43 2024-04-26T00:22:15.935 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 34 to 37 2024-04-26T00:22:15.935 INFO:tasks.rados.rados.0.smithi045.stdout:392: snap_create 2024-04-26T00:22:15.945 INFO:tasks.rados.rados.0.smithi045.stdout:391: finishing rollback tid 0 to smithi04526579-34 2024-04-26T00:22:15.946 INFO:tasks.rados.rados.0.smithi045.stdout:391: finishing rollback tid 1 to smithi04526579-34 2024-04-26T00:22:15.946 INFO:tasks.rados.rados.0.smithi045.stdout:391: finishing rollback tid 2 to smithi04526579-34 2024-04-26T00:22:15.946 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 80 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-04-26T00:22:16.937 INFO:tasks.rados.rados.0.smithi045.stdout:390: done (2 left) 2024-04-26T00:22:16.937 INFO:tasks.rados.rados.0.smithi045.stdout:391: done (1 left) 2024-04-26T00:22:16.937 INFO:tasks.rados.rados.0.smithi045.stdout:392: done (0 left) 2024-04-26T00:22:16.937 INFO:tasks.rados.rados.0.smithi045.stdout:393: snap_remove snap 40 2024-04-26T00:22:17.920 INFO:tasks.rados.rados.0.smithi045.stdout:393: done (0 left) 2024-04-26T00:22:17.920 INFO:tasks.rados.rados.0.smithi045.stdout:394: snap_create 2024-04-26T00:22:18.920 INFO:tasks.rados.rados.0.smithi045.stdout:394: done (0 left) 2024-04-26T00:22:18.920 INFO:tasks.rados.rados.0.smithi045.stdout:395: write (excl) oid 31 current snap is 45 2024-04-26T00:22:18.920 INFO:tasks.rados.rados.0.smithi045.stdout:395: seq_num 169 ranges {440383=476847,1556398=437383,2474721=365662} 2024-04-26T00:22:18.935 INFO:tasks.rados.rados.0.smithi045.stdout:395: writing smithi04526579-31 from 440383 to 917230 tid 1 2024-04-26T00:22:18.952 INFO:tasks.rados.rados.0.smithi045.stdout:395: writing smithi04526579-31 from 1556398 to 1993781 tid 2 2024-04-26T00:22:18.967 INFO:tasks.rados.rados.0.smithi045.stdout:395: writing smithi04526579-31 from 2474721 to 2840383 tid 3 2024-04-26T00:22:18.967 INFO:tasks.rados.rados.0.smithi045.stdout:396: write oid 9 current snap is 45 2024-04-26T00:22:18.967 INFO:tasks.rados.rados.0.smithi045.stdout:396: seq_num 170 ranges {457190=552096,1614880=581041,2785791=71399} 2024-04-26T00:22:18.980 INFO:tasks.rados.rados.0.smithi045.stdout:396: writing smithi04526579-9 from 457190 to 1009286 tid 1 2024-04-26T00:22:18.993 INFO:tasks.rados.rados.0.smithi045.stdout:396: writing smithi04526579-9 from 1614880 to 2195921 tid 2 2024-04-26T00:22:19.000 INFO:tasks.rados.rados.0.smithi045.stdout:396: writing smithi04526579-9 from 2785791 to 2857190 tid 3 2024-04-26T00:22:19.000 INFO:tasks.rados.rados.0.smithi045.stdout:397: copy_from oid 38 from oid 32 current snap is 45 2024-04-26T00:22:19.000 INFO:tasks.rados.rados.0.smithi045.stdout:395: finishing write tid 1 to smithi04526579-31 2024-04-26T00:22:19.000 INFO:tasks.rados.rados.0.smithi045.stdout:395: finishing write tid 2 to smithi04526579-31 2024-04-26T00:22:19.000 INFO:tasks.rados.rados.0.smithi045.stdout:395: finishing write tid 3 to smithi04526579-31 2024-04-26T00:22:19.000 INFO:tasks.rados.rados.0.smithi045.stdout:395: finishing write tid 4 to smithi04526579-31 2024-04-26T00:22:19.000 INFO:tasks.rados.rados.0.smithi045.stdout:398: write (excl) oid 48 current snap is 45 2024-04-26T00:22:19.000 INFO:tasks.rados.rados.0.smithi045.stdout:398: seq_num 171 ranges {473997=627345,1673362=441052,2529567=344430} 2024-04-26T00:22:19.009 INFO:tasks.rados.rados.0.smithi045.stdout:398: writing smithi04526579-48 from 473997 to 1101342 tid 1 2024-04-26T00:22:19.017 INFO:tasks.rados.rados.0.smithi045.stdout:398: writing smithi04526579-48 from 1673362 to 2114414 tid 2 2024-04-26T00:22:19.023 INFO:tasks.rados.rados.0.smithi045.stdout:398: writing smithi04526579-48 from 2529567 to 2873997 tid 3 2024-04-26T00:22:19.023 INFO:tasks.rados.rados.0.smithi045.stdout:399: copy_from oid 42 from oid 1 current snap is 45 2024-04-26T00:22:19.023 INFO:tasks.rados.rados.0.smithi045.stdout:395: finishing write tid 5 to smithi04526579-31 2024-04-26T00:22:19.023 INFO:tasks.rados.rados.0.smithi045.stdout:395: oid 31 updating version 0 to 171 2024-04-26T00:22:19.024 INFO:tasks.rados.rados.0.smithi045.stdout:395: oid 31 version 171 is already newer than 170 2024-04-26T00:22:19.024 INFO:tasks.rados.rados.0.smithi045.stdout:395: oid 31 updating version 171 to 172 2024-04-26T00:22:19.024 INFO:tasks.rados.rados.0.smithi045.stdout:395: oid 31 updating version 172 to 173 2024-04-26T00:22:19.024 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 173 (ObjNum 169 snap 45 seq_num 169) dirty exists 2024-04-26T00:22:19.024 INFO:tasks.rados.rados.0.smithi045.stdout:395: left oid 31 (ObjNum 169 snap 45 seq_num 169) 2024-04-26T00:22:19.024 INFO:tasks.rados.rados.0.smithi045.stdout:396: finishing write tid 1 to smithi04526579-9 2024-04-26T00:22:19.024 INFO:tasks.rados.rados.0.smithi045.stdout:395: done (4 left) 2024-04-26T00:22:19.024 INFO:tasks.rados.rados.0.smithi045.stdout:400: delete oid 17 current snap is 45 2024-04-26T00:22:19.024 INFO:tasks.rados.rados.0.smithi045.stdout:396: finishing write tid 2 to smithi04526579-9 2024-04-26T00:22:19.024 INFO:tasks.rados.rados.0.smithi045.stdout:396: finishing write tid 3 to smithi04526579-9 2024-04-26T00:22:19.024 INFO:tasks.rados.rados.0.smithi045.stdout:396: finishing write tid 4 to smithi04526579-9 2024-04-26T00:22:19.024 INFO:tasks.rados.rados.0.smithi045.stdout:396: finishing write tid 5 to smithi04526579-9 2024-04-26T00:22:19.024 INFO:tasks.rados.rados.0.smithi045.stdout:396: oid 9 updating version 0 to 176 2024-04-26T00:22:19.024 INFO:tasks.rados.rados.0.smithi045.stdout:396: oid 9 updating version 176 to 177 2024-04-26T00:22:19.024 INFO:tasks.rados.rados.0.smithi045.stdout:396: oid 9 version 177 is already newer than 175 2024-04-26T00:22:19.025 INFO:tasks.rados.rados.0.smithi045.stdout:396: oid 9 updating version 177 to 178 2024-04-26T00:22:19.025 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 178 (ObjNum 170 snap 45 seq_num 170) dirty exists 2024-04-26T00:22:19.025 INFO:tasks.rados.rados.0.smithi045.stdout:396: left oid 9 (ObjNum 170 snap 45 seq_num 170) 2024-04-26T00:22:19.032 INFO:tasks.rados.rados.0.smithi045.stdout:396: done (4 left) 2024-04-26T00:22:19.032 INFO:tasks.rados.rados.0.smithi045.stdout:400: done (3 left) 2024-04-26T00:22:19.033 INFO:tasks.rados.rados.0.smithi045.stdout:401: read oid 8 snap -1 2024-04-26T00:22:19.033 INFO:tasks.rados.rados.0.smithi045.stdout:401: expect (ObjNum 7 snap 0 seq_num 7) 2024-04-26T00:22:19.033 INFO:tasks.rados.rados.0.smithi045.stdout:402: read oid 35 snap -1 2024-04-26T00:22:19.033 INFO:tasks.rados.rados.0.smithi045.stdout:402: expect (ObjNum 138 snap 31 seq_num 138) 2024-04-26T00:22:19.033 INFO:tasks.rados.rados.0.smithi045.stdout:403: copy_from oid 33 from oid 16 current snap is 45 2024-04-26T00:22:19.033 INFO:tasks.rados.rados.0.smithi045.stdout:404: copy_from oid 50 from oid 26 current snap is 45 2024-04-26T00:22:19.033 INFO:tasks.rados.rados.0.smithi045.stdout:405: write (excl) oid 34 current snap is 45 2024-04-26T00:22:19.033 INFO:tasks.rados.rados.0.smithi045.stdout:405: seq_num 173 ranges {507611=777843,2073973=444721} 2024-04-26T00:22:19.047 INFO:tasks.rados.rados.0.smithi045.stdout:405: writing smithi04526579-34 from 507611 to 1285454 tid 1 2024-04-26T00:22:19.058 INFO:tasks.rados.rados.0.smithi045.stdout:405: writing smithi04526579-34 from 2073973 to 2518694 tid 2 2024-04-26T00:22:19.058 INFO:tasks.rados.rados.0.smithi045.stdout:406: snap_create 2024-04-26T00:22:19.058 INFO:tasks.rados.rados.0.smithi045.stdout:401: expect (ObjNum 7 snap 0 seq_num 7) 2024-04-26T00:22:19.187 INFO:tasks.rados.rados.0.smithi045.stdout:402: expect (ObjNum 138 snap 31 seq_num 138) 2024-04-26T00:22:19.320 INFO:tasks.rados.rados.0.smithi045.stdout:404: finishing copy_from racing read to smithi04526579-50 2024-04-26T00:22:19.320 INFO:tasks.rados.rados.0.smithi045.stdout:403: finishing copy_from racing read to smithi04526579-33 2024-04-26T00:22:19.321 INFO:tasks.rados.rados.0.smithi045.stdout:403: finishing copy_from to smithi04526579-33 2024-04-26T00:22:19.321 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 6 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-04-26T00:22:19.321 INFO:tasks.rados.rados.0.smithi045.stdout:399: finishing copy_from racing read to smithi04526579-42 2024-04-26T00:22:19.321 INFO:tasks.rados.rados.0.smithi045.stdout:399: finishing copy_from to smithi04526579-42 2024-04-26T00:22:19.321 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 145 (ObjNum 110 snap 28 seq_num 110) dirty exists 2024-04-26T00:22:19.321 INFO:tasks.rados.rados.0.smithi045.stdout:404: finishing copy_from to smithi04526579-50 2024-04-26T00:22:19.321 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 148 (ObjNum 136 snap 30 seq_num 136) dirty exists 2024-04-26T00:22:19.321 INFO:tasks.rados.rados.0.smithi045.stdout:397: finishing copy_from racing read to smithi04526579-38 2024-04-26T00:22:19.321 INFO:tasks.rados.rados.0.smithi045.stdout:397: finishing copy_from to smithi04526579-38 2024-04-26T00:22:19.321 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 109 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-04-26T00:22:19.321 INFO:tasks.rados.rados.0.smithi045.stdout:398: finishing write tid 1 to smithi04526579-48 2024-04-26T00:22:19.321 INFO:tasks.rados.rados.0.smithi045.stdout:398: finishing write tid 2 to smithi04526579-48 2024-04-26T00:22:19.321 INFO:tasks.rados.rados.0.smithi045.stdout:398: finishing write tid 3 to smithi04526579-48 2024-04-26T00:22:19.321 INFO:tasks.rados.rados.0.smithi045.stdout:398: finishing write tid 4 to smithi04526579-48 2024-04-26T00:22:19.321 INFO:tasks.rados.rados.0.smithi045.stdout:398: finishing write tid 5 to smithi04526579-48 2024-04-26T00:22:19.322 INFO:tasks.rados.rados.0.smithi045.stdout:398: oid 48 updating version 0 to 86 2024-04-26T00:22:19.322 INFO:tasks.rados.rados.0.smithi045.stdout:398: oid 48 version 86 is already newer than 84 2024-04-26T00:22:19.322 INFO:tasks.rados.rados.0.smithi045.stdout:398: oid 48 updating version 86 to 87 2024-04-26T00:22:19.322 INFO:tasks.rados.rados.0.smithi045.stdout:398: oid 48 version 87 is already newer than 85 2024-04-26T00:22:19.322 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 87 (ObjNum 171 snap 45 seq_num 171) dirty exists 2024-04-26T00:22:19.322 INFO:tasks.rados.rados.0.smithi045.stdout:398: left oid 48 (ObjNum 171 snap 45 seq_num 171) 2024-04-26T00:22:19.322 INFO:tasks.rados.rados.0.smithi045.stdout:405: finishing write tid 1 to smithi04526579-34 2024-04-26T00:22:19.322 INFO:tasks.rados.rados.0.smithi045.stdout:405: finishing write tid 2 to smithi04526579-34 2024-04-26T00:22:19.322 INFO:tasks.rados.rados.0.smithi045.stdout:405: finishing write tid 3 to smithi04526579-34 2024-04-26T00:22:19.322 INFO:tasks.rados.rados.0.smithi045.stdout:405: finishing write tid 4 to smithi04526579-34 2024-04-26T00:22:19.322 INFO:tasks.rados.rados.0.smithi045.stdout:405: oid 34 updating version 0 to 91 2024-04-26T00:22:19.322 INFO:tasks.rados.rados.0.smithi045.stdout:405: oid 34 version 91 is already newer than 89 2024-04-26T00:22:19.322 INFO:tasks.rados.rados.0.smithi045.stdout:405: oid 34 version 91 is already newer than 90 2024-04-26T00:22:19.322 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 91 (ObjNum 173 snap 45 seq_num 173) dirty exists 2024-04-26T00:22:19.322 INFO:tasks.rados.rados.0.smithi045.stdout:405: left oid 34 (ObjNum 173 snap 45 seq_num 173) 2024-04-26T00:22:19.607 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-04-26T00:22:19.607 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-04-26T00:22:19.607 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-26T00:22:19.607 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-26T00:22:19.610 INFO:tasks.ceph.mon.c:Started 2024-04-26T00:22:19.610 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:22:19.610 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:22:19.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.754+0000 7f9892db5640 10 monclient: get_monmap_and_config 2024-04-26T00:22:19.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.754+0000 7f9892db5640 10 monclient: build_initial_monmap 2024-04-26T00:22:19.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.754+0000 7f9892db5640 10 monclient: monmap: 2024-04-26T00:22:19.733 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:22:19.733 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:19.733 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:22:19.757561+0000 2024-04-26T00:22:19.733 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:22:19.757561+0000 2024-04-26T00:22:19.733 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:22:19.733 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:19.733 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:22:19.733 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:22:19.733 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:22:19.733 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:22:19.733 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:22:19.733 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:19.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.754+0000 7f9892db5640 1 Processor -- start 2024-04-26T00:22:19.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.754+0000 7f9892db5640 1 -- start start 2024-04-26T00:22:19.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.754+0000 7f9892db5640 10 monclient: init 2024-04-26T00:22:19.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.754+0000 7f9892db5640 10 monclient: _reopen_session rank -1 2024-04-26T00:22:19.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.754+0000 7f9892db5640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-04-26T00:22:19.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.754+0000 7f9892db5640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f988c136e10 0x7f988c09c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:19.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.754+0000 7f9892db5640 10 monclient(hunting): picked mon.noname-b con 0x7f988c136e10 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:22:19.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.754+0000 7f9892db5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f988c1371f0 addr v1:172.21.15.37:6789/0 2024-04-26T00:22:19.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.754+0000 7f9892db5640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f988c13c340 0x7f988c13e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:19.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.754+0000 7f9892db5640 10 monclient(hunting): picked mon.noname-a con 0x7f988c13c340 addr v2:172.21.15.37:3300/0 2024-04-26T00:22:19.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.754+0000 7f9892db5640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f988c09b680 con 0x7f988c1371f0 2024-04-26T00:22:19.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.754+0000 7f9892db5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:19.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.754+0000 7f9892db5640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f988c09b8f0 con 0x7f988c13c340 2024-04-26T00:22:19.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.754+0000 7f9892db5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:19.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.754+0000 7f9892db5640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f988c09bbd0 con 0x7f988c136e10 2024-04-26T00:22:19.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.754+0000 7f9892db5640 10 monclient(hunting): _renew_subs 2024-04-26T00:22:19.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.754+0000 7f9892db5640 10 monclient(hunting): authenticate will time out at 1121.062923s 2024-04-26T00:22:19.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.754+0000 7f9890b51640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f988c136e10 0x7f988c09c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:19.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.754+0000 7f9891352640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f988c13c340 0x7f988c13e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:19.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.754+0000 7f9890b51640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f988c136e10 0x7f988c09c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:55584/0 (socket says 172.21.15.37:55584) 2024-04-26T00:22:19.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.754+0000 7f9890b51640 1 -- 172.21.15.37:0/3554638272 learned_addr learned my addr 172.21.15.37:0/3554638272 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:22:19.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.754+0000 7f9891352640 1 -- 172.21.15.37:0/3554638272 >> v2:172.21.15.37:3300/0 conn(0x7f988c13c340 msgr2=0x7f988c13e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:22:19.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.754+0000 7f9890b51640 10 monclient(hunting): get_auth_request con 0x7f988c136e10 auth_method 0 2024-04-26T00:22:19.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.754+0000 7f9891352640 1 -- 172.21.15.37:0/3554638272 >> v2:172.21.15.37:3300/0 conn(0x7f988c13c340 msgr2=0x7f988c13e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:22:19.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.754+0000 7f9890b51640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:19.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.754+0000 7f9891352640 1 --2- 172.21.15.37:0/3554638272 >> v2:172.21.15.37:3300/0 conn(0x7f988c13c340 0x7f988c13e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:22:19.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.754+0000 7f9890b51640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:19.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.754+0000 7f9890b51640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:19.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.754+0000 7f988bfff640 1 --1- 172.21.15.37:0/3554638272 >> v1:172.21.15.37:6789/0 conn(0x7f988c1371f0 0x7f988c1375d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:35472/0 (socket says 172.21.15.37:35472) 2024-04-26T00:22:19.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.754+0000 7f9891352640 1 --2- 172.21.15.37:0/3554638272 >> v2:172.21.15.37:3300/0 conn(0x7f988c13c340 0x7f988c13e730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:22:19.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.754+0000 7f9890b51640 1 -- 172.21.15.37:0/3554638272 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f988c136e10 msgr2=0x7f988c09c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:22:19.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.754+0000 7f9890b51640 1 -- 172.21.15.37:0/3554638272 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f988c136e10 msgr2=0x7f988c09c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:22:19.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.754+0000 7f9890b51640 1 --2- 172.21.15.37:0/3554638272 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f988c136e10 0x7f988c09c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:22:19.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.754+0000 7f9890b51640 1 --2- 172.21.15.37:0/3554638272 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f988c136e10 0x7f988c09c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:22:19.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988bfff640 1 -- 172.21.15.37:0/3554638272 >> v1:172.21.15.37:6789/0 conn(0x7f988c1371f0 legacy=0x7f988c1375d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:19.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 1 -- 172.21.15.37:0/3554638272 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1867623989 0 0) 0x7f988c09b680 con 0x7f988c1371f0 2024-04-26T00:22:19.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 10 monclient(hunting): _negotiate 2024-04-26T00:22:19.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:19.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:19.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 10 monclient(hunting): my global_id is 4608 2024-04-26T00:22:19.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 1 -- 172.21.15.37:0/3554638272 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9874003530 con 0x7f988c1371f0 2024-04-26T00:22:19.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 1 -- 172.21.15.37:0/3554638272 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2817114328 0 0) 0x7f9874003530 con 0x7f988c1371f0 2024-04-26T00:22:19.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 1 -- 172.21.15.37:0/3554638272 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f988c136e10 msgr2=0x7f988c09c5b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:22:19.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 1 --2- 172.21.15.37:0/3554638272 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f988c136e10 0x7f988c09c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:19.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 1 -- 172.21.15.37:0/3554638272 >> v2:172.21.15.37:3300/0 conn(0x7f988c13c340 msgr2=0x7f988c13e730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:22:19.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 1 --2- 172.21.15.37:0/3554638272 >> v2:172.21.15.37:3300/0 conn(0x7f988c13c340 0x7f988c13e730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:19.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:22:19.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 1 monclient: found mon.noname-a-legacy 2024-04-26T00:22:19.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:22:19.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.37:6789/0 2024-04-26T00:22:19.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 1 -- 172.21.15.37:0/3554638272 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f988c0a5d90 con 0x7f988c1371f0 2024-04-26T00:22:19.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 10 monclient: _finish_auth 0 2024-04-26T00:22:19.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:22:19.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:22:19.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 1 -- 172.21.15.37:0/3554638272 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 866326386 0 0) 0x7f98800019f0 con 0x7f988c1371f0 2024-04-26T00:22:19.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:19.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-26T00:22:19.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 10 monclient: dump: 2024-04-26T00:22:19.742 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:19.742 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:19.742 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:19.742 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:19.742 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:19.742 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:19.742 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:19.742 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:19.743 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:19.743 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:19.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] but i'm connected to v1:172.21.15.37:6789/0, reconnecting 2024-04-26T00:22:19.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 10 monclient: _reopen_session rank -1 2024-04-26T00:22:19.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 1 -- 172.21.15.37:0/3554638272 >> v1:172.21.15.37:6789/0 conn(0x7f988c1371f0 legacy=0x7f988c1375d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:19.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:22:19.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 1 --2- 172.21.15.37:0/3554638272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98740039c0 0x7f9874003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:19.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 10 monclient(hunting): picked mon.b con 0x7f98740039c0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:22:19.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 1 --2- 172.21.15.37:0/3554638272 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9874004810 0x7f9874004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:19.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 10 monclient(hunting): picked mon.a con 0x7f9874004810 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:22:19.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 1 --2- 172.21.15.37:0/3554638272 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f988c136e10 0x7f987400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:19.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 10 monclient(hunting): picked mon.c con 0x7f988c136e10 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:22:19.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:19.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 1 -- 172.21.15.37:0/3554638272 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f988c09b8f0 con 0x7f9874004810 2024-04-26T00:22:19.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:19.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 1 -- 172.21.15.37:0/3554638272 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f988c09bbd0 con 0x7f98740039c0 2024-04-26T00:22:19.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:19.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 1 -- 172.21.15.37:0/3554638272 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f98740055b0 con 0x7f988c136e10 2024-04-26T00:22:19.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 10 monclient(hunting): _renew_subs 2024-04-26T00:22:19.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988b7fe640 10 monclient(hunting): _finish_auth 0 2024-04-26T00:22:19.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988bfff640 1 --2- 172.21.15.37:0/3554638272 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9874004810 0x7f9874004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:19.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988bfff640 10 monclient(hunting): get_auth_request con 0x7f9874004810 auth_method 0 2024-04-26T00:22:19.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:19.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:19.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:22:19.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:19.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:19.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f9891352640 1 --2- 172.21.15.37:0/3554638272 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f988c136e10 0x7f987400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:19.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:22:19.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f9891352640 10 monclient(hunting): get_auth_request con 0x7f988c136e10 auth_method 0 2024-04-26T00:22:19.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f9891352640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:19.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f9891352640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:19.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f9891352640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:22:19.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988bfff640 10 monclient(hunting): handle_auth_done global_id 4608 payload 293 2024-04-26T00:22:19.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f9890b51640 1 --2- 172.21.15.37:0/3554638272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98740039c0 0x7f9874003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:19.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988bfff640 1 -- 172.21.15.37:0/3554638272 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f988c136e10 msgr2=0x7f987400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:22:19.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988bfff640 1 --2- 172.21.15.37:0/3554638272 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f988c136e10 0x7f987400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:19.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988bfff640 1 -- 172.21.15.37:0/3554638272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98740039c0 msgr2=0x7f9874003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:22:19.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988bfff640 1 --2- 172.21.15.37:0/3554638272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98740039c0 0x7f9874003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:19.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988bfff640 10 monclient: _finish_hunting 0 2024-04-26T00:22:19.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988bfff640 1 monclient: found mon.a 2024-04-26T00:22:19.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:22:19.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988bfff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:22:19.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988bfff640 1 -- 172.21.15.37:0/3554638272 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9874003530 con 0x7f9874004810 2024-04-26T00:22:19.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988bfff640 10 monclient: _finish_auth 0 2024-04-26T00:22:19.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988bfff640 10 monclient: _check_auth_tickets 2024-04-26T00:22:19.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f988bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:22:19.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f9892db5640 5 monclient: authenticate success, global_id 4608 2024-04-26T00:22:19.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f9892db5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:22:19.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.758+0000 7f9891352640 1 --2- 172.21.15.37:0/3554638272 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f988c136e10 0x7f987400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:22:19.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.762+0000 7f988bfff640 1 -- 172.21.15.37:0/3554638272 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9874004810 msgr2=0x7f9874004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:19.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.762+0000 7f988bfff640 1 --2- 172.21.15.37:0/3554638272 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9874004810 0x7f9874004c90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f988000c3b0 tx=0x7f988001cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d630550ea5826de2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:19.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.762+0000 7f988b7fe640 1 -- 172.21.15.37:0/3554638272 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f988000c0e0 con 0x7f9874004810 2024-04-26T00:22:19.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.762+0000 7f988b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:19.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.762+0000 7f988b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:22:19.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.762+0000 7f988b7fe640 10 monclient: dump: 2024-04-26T00:22:19.752 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:19.752 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:19.752 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:19.752 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:19.752 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:19.752 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:19.752 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:19.753 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:19.753 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:19.753 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:19.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.762+0000 7f988b7fe640 1 -- 172.21.15.37:0/3554638272 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98800a9070 con 0x7f9874004810 2024-04-26T00:22:19.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.762+0000 7f9892db5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:22:19.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.762+0000 7f988b7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:22:19.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.762+0000 7f988b7fe640 1 -- 172.21.15.37:0/3554638272 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98800a5d10 con 0x7f9874004810 2024-04-26T00:22:19.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.762+0000 7f988b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:19.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.762+0000 7f988b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:22:19.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.762+0000 7f988b7fe640 10 monclient: dump: 2024-04-26T00:22:19.753 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:19.753 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:19.753 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:19.753 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:19.754 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:19.754 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:19.754 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:19.754 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:19.754 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:19.754 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:19.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.762+0000 7f9892db5640 10 monclient: get_monmap_and_config success 2024-04-26T00:22:19.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.762+0000 7f9892db5640 4 set_mon_vals no callback set 2024-04-26T00:22:19.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.762+0000 7f9892db5640 10 monclient: shutdown 2024-04-26T00:22:19.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.762+0000 7f9892db5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:22:19.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.762+0000 7f9892db5640 1 -- 172.21.15.37:0/3554638272 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9874004810 msgr2=0x7f9874004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:19.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.762+0000 7f9892db5640 1 --2- 172.21.15.37:0/3554638272 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9874004810 0x7f9874004c90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f988000c3b0 tx=0x7f988001cd70 comp rx=0 tx=0).stop 2024-04-26T00:22:19.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.762+0000 7f9891352640 1 -- 172.21.15.37:0/3554638272 reap_dead start 2024-04-26T00:22:19.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.762+0000 7f9892db5640 1 -- 172.21.15.37:0/3554638272 shutdown_connections 2024-04-26T00:22:19.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.762+0000 7f9892db5640 1 -- 172.21.15.37:0/3554638272 >> 172.21.15.37:0/3554638272 conn(0x7f988c0a64e0 msgr2=0x7f988c0a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:22:19.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.762+0000 7f9892db5640 1 -- 172.21.15.37:0/3554638272 shutdown_connections 2024-04-26T00:22:19.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.762+0000 7f9892db5640 1 -- 172.21.15.37:0/3554638272 wait complete. 2024-04-26T00:22:19.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9892db5640 10 monclient: build_initial_monmap 2024-04-26T00:22:19.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9892db5640 10 monclient: monmap: 2024-04-26T00:22:19.755 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:22:19.755 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:22:19.755 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:22:19.755 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:22:19.755 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:22:19.755 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:22:19.755 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:22:19.755 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:22:19.755 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:22:19.756 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:19.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9892db5640 1 Processor -- start 2024-04-26T00:22:19.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9892db5640 1 -- start start 2024-04-26T00:22:19.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9892db5640 10 monclient: init 2024-04-26T00:22:19.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9892db5640 10 monclient: _reopen_session rank -1 2024-04-26T00:22:19.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9892db5640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:22:19.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9892db5640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9874004810 0x7f988c1cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:19.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9892db5640 10 monclient(hunting): picked mon.noname-c con 0x7f9874004810 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:22:19.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9892db5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f988c1cc400 0x7f988c1351f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:19.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9892db5640 10 monclient(hunting): picked mon.noname-b con 0x7f988c1cc400 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:22:19.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9892db5640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f988c135730 0x7f988c1ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:19.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9892db5640 10 monclient(hunting): picked mon.noname-a con 0x7f988c135730 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:22:19.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9892db5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:19.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9892db5640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f988c13f690 con 0x7f988c135730 2024-04-26T00:22:19.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9892db5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:19.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9892db5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f988c09c9a0 con 0x7f988c1cc400 2024-04-26T00:22:19.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9892db5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:19.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9892db5640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f988c13f810 con 0x7f9874004810 2024-04-26T00:22:19.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9892db5640 10 monclient(hunting): _renew_subs 2024-04-26T00:22:19.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9892db5640 10 monclient(hunting): authenticate will time out at 1121.073027s 2024-04-26T00:22:19.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9890b51640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9874004810 0x7f988c1cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:19.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9891352640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f988c135730 0x7f988c1ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:19.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9890b51640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9874004810 0x7f988c1cb9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:55608/0 (socket says 172.21.15.37:55608) 2024-04-26T00:22:19.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9890b51640 1 -- 172.21.15.37:0/3199101267 learned_addr learned my addr 172.21.15.37:0/3199101267 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:22:19.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9890b51640 10 monclient(hunting): get_auth_request con 0x7f9874004810 auth_method 0 2024-04-26T00:22:19.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9890b51640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:19.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9890b51640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:19.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9890b51640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:19.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9891352640 10 monclient(hunting): get_auth_request con 0x7f988c135730 auth_method 0 2024-04-26T00:22:19.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9891352640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:19.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9891352640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:19.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9891352640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:19.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9890b51640 1 -- 172.21.15.37:0/3199101267 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9874004810 msgr2=0x7f988c1cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:22:19.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9890b51640 1 -- 172.21.15.37:0/3199101267 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9874004810 msgr2=0x7f988c1cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:22:19.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9890b51640 1 --2- 172.21.15.37:0/3199101267 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9874004810 0x7f988c1cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:22:19.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9891352640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:19.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9891352640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:19.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9891352640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:19.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9890b51640 1 --2- 172.21.15.37:0/3199101267 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9874004810 0x7f988c1cb9a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:22:19.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f988bfff640 1 --2- 172.21.15.37:0/3199101267 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f988c1cc400 0x7f988c1351f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:19.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9891352640 10 monclient(hunting): handle_auth_done global_id 4611 payload 824 2024-04-26T00:22:19.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9891352640 1 -- 172.21.15.37:0/3199101267 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9874004810 msgr2=0x7f988c1cb9a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:22:19.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9891352640 1 --2- 172.21.15.37:0/3199101267 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9874004810 0x7f988c1cb9a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:19.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9891352640 1 -- 172.21.15.37:0/3199101267 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f988c1cc400 msgr2=0x7f988c1351f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:19.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9891352640 1 --2- 172.21.15.37:0/3199101267 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f988c1cc400 0x7f988c1351f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:19.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9891352640 10 monclient: _finish_hunting 0 2024-04-26T00:22:19.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9891352640 1 monclient: found mon.noname-a 2024-04-26T00:22:19.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9891352640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:22:19.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9891352640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:22:19.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9891352640 1 -- 172.21.15.37:0/3199101267 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f988c0a0410 con 0x7f988c135730 2024-04-26T00:22:19.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f988bfff640 10 monclient: get_auth_request con 0x7f988c1cc400 auth_method 0 2024-04-26T00:22:19.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f988bfff640 1 --2- 172.21.15.37:0/3199101267 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f988c1cc400 0x7f988c1351f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:22:19.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9891352640 1 -- 172.21.15.37:0/3199101267 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f988c135730 msgr2=0x7f988c1ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:19.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.766+0000 7f9891352640 1 --2- 172.21.15.37:0/3199101267 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f988c135730 0x7f988c1ceea0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f988c06e430 tx=0x7f988400c6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:19.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.770+0000 7f98897fa640 1 -- 172.21.15.37:0/3199101267 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9884003ba0 con 0x7f988c135730 2024-04-26T00:22:19.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.770+0000 7f98897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:19.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.770+0000 7f98897fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:22:19.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.770+0000 7f98897fa640 10 monclient: dump: 2024-04-26T00:22:19.760 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:19.760 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:19.760 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:19.760 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:19.760 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:19.760 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:19.760 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:19.760 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:19.760 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:19.760 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:19.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.770+0000 7f98897fa640 10 monclient: _finish_auth 0 2024-04-26T00:22:19.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.770+0000 7f98897fa640 10 monclient: _check_auth_tickets 2024-04-26T00:22:19.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.770+0000 7f98897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:22:19.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.770+0000 7f9892db5640 5 monclient: authenticate success, global_id 4611 2024-04-26T00:22:19.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.770+0000 7f9892db5640 10 monclient: _renew_subs 2024-04-26T00:22:19.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.770+0000 7f98897fa640 1 -- 172.21.15.37:0/3199101267 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9884013070 con 0x7f988c135730 2024-04-26T00:22:19.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.770+0000 7f9892db5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:22:19.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.770+0000 7f9892db5640 1 -- 172.21.15.37:0/3199101267 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f988c121300 con 0x7f988c135730 2024-04-26T00:22:19.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.770+0000 7f98897fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:22:19.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.770+0000 7f98897fa640 1 -- 172.21.15.37:0/3199101267 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9884004030 con 0x7f988c135730 2024-04-26T00:22:19.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.770+0000 7f98897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:19.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.770+0000 7f98897fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:22:19.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.770+0000 7f98897fa640 10 monclient: dump: 2024-04-26T00:22:19.761 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:19.761 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:19.761 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:19.761 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:19.761 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:19.761 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:19.762 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:19.762 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:19.762 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:19.762 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:19.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.770+0000 7f9892db5640 10 monclient: _renew_subs 2024-04-26T00:22:19.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.770+0000 7f988affd640 4 set_mon_vals no callback set 2024-04-26T00:22:19.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.770+0000 7f9892db5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:22:19.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.770+0000 7f9892db5640 1 -- 172.21.15.37:0/3199101267 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f988c120630 con 0x7f988c135730 2024-04-26T00:22:19.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.770+0000 7f9892db5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:19.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.770+0000 7f9892db5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:19.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.770+0000 7f9892db5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:22:19.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.770+0000 7f98897fa640 1 -- 172.21.15.37:0/3199101267 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f98840041d0 con 0x7f988c135730 2024-04-26T00:22:19.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.770+0000 7f9892db5640 1 -- 172.21.15.37:0/3199101267 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f988c075910 con 0x7f988c135730 2024-04-26T00:22:19.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.770+0000 7f98897fa640 1 --2- 172.21.15.37:0/3199101267 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9878035ed0 0x7f9878038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:19.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.770+0000 7f98897fa640 1 -- 172.21.15.37:0/3199101267 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 3931+0+0 (secure 0 0 0) 0x7f98840491c0 con 0x7f988c135730 2024-04-26T00:22:19.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.770+0000 7f9890b51640 1 --2- 172.21.15.37:0/3199101267 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9878035ed0 0x7f9878038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:19.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.770+0000 7f9890b51640 10 monclient: get_auth_request con 0x7f9878035ed0 auth_method 0 2024-04-26T00:22:19.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.774+0000 7f9890b51640 1 --2- 172.21.15.37:0/3199101267 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9878035ed0 0x7f9878038390 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f987c00d180 tx=0x7f987c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:19.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.774+0000 7f98897fa640 1 -- 172.21.15.37:0/3199101267 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9884047f10 con 0x7f988c135730 2024-04-26T00:22:19.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.774+0000 7f98897fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:19.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.774+0000 7f98897fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:22:19.926 INFO:tasks.rados.rados.0.smithi045.stdout:397: done (8 left) 2024-04-26T00:22:19.926 INFO:tasks.rados.rados.0.smithi045.stdout:398: done (7 left) 2024-04-26T00:22:19.926 INFO:tasks.rados.rados.0.smithi045.stdout:399: done (6 left) 2024-04-26T00:22:19.926 INFO:tasks.rados.rados.0.smithi045.stdout:401: done (5 left) 2024-04-26T00:22:19.926 INFO:tasks.rados.rados.0.smithi045.stdout:402: done (4 left) 2024-04-26T00:22:19.927 INFO:tasks.rados.rados.0.smithi045.stdout:403: done (3 left) 2024-04-26T00:22:19.927 INFO:tasks.rados.rados.0.smithi045.stdout:404: done (2 left) 2024-04-26T00:22:19.927 INFO:tasks.rados.rados.0.smithi045.stdout:405: done (1 left) 2024-04-26T00:22:19.927 INFO:tasks.rados.rados.0.smithi045.stdout:406: done (0 left) 2024-04-26T00:22:19.927 INFO:tasks.rados.rados.0.smithi045.stdout:407: rollback oid 8 current snap is 46 2024-04-26T00:22:19.927 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 8 to 45 2024-04-26T00:22:19.927 INFO:tasks.rados.rados.0.smithi045.stdout:408: delete oid 28 current snap is 46 2024-04-26T00:22:19.933 INFO:tasks.rados.rados.0.smithi045.stdout:408: done (1 left) 2024-04-26T00:22:19.933 INFO:tasks.rados.rados.0.smithi045.stdout:409: snap_remove snap 44 2024-04-26T00:22:19.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.998+0000 7f9892db5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:22:19.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.998+0000 7f9892db5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:22:19.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.998+0000 7f9892db5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:22:19.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.998+0000 7f9892db5640 1 -- 172.21.15.37:0/3199101267 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f988c066f40 con 0x7f988c135730 2024-04-26T00:22:19.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.998+0000 7f98897fa640 1 -- 172.21.15.37:0/3199101267 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f988400a520 con 0x7f988c135730 2024-04-26T00:22:19.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.998+0000 7f98897fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:22:19.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:19.998+0000 7f98897fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:22:19.976 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":22,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:22:19.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:20.002+0000 7f9892db5640 1 -- 172.21.15.37:0/3199101267 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9878035ed0 msgr2=0x7f9878038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:19.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:20.002+0000 7f9892db5640 1 --2- 172.21.15.37:0/3199101267 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9878035ed0 0x7f9878038390 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f987c00d180 tx=0x7f987c008040 comp rx=0 tx=0).stop 2024-04-26T00:22:19.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:20.002+0000 7f9892db5640 10 monclient: shutdown 2024-04-26T00:22:19.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:20.002+0000 7f9892db5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:22:19.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:20.002+0000 7f9892db5640 1 -- 172.21.15.37:0/3199101267 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f988c135730 msgr2=0x7f988c1ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:19.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:20.002+0000 7f9892db5640 1 --2- 172.21.15.37:0/3199101267 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f988c135730 0x7f988c1ceea0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f988c06e430 tx=0x7f988400c6e0 comp rx=0 tx=0).stop 2024-04-26T00:22:19.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:20.002+0000 7f9892db5640 1 -- 172.21.15.37:0/3199101267 shutdown_connections 2024-04-26T00:22:19.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:20.002+0000 7f9892db5640 1 --2- 172.21.15.37:0/3199101267 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9878035ed0 0x7f9878038390 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:19.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:20.002+0000 7f9892db5640 1 --2- 172.21.15.37:0/3199101267 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f988c135730 0x7f988c1ceea0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:19.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:20.002+0000 7f9892db5640 1 --2- 172.21.15.37:0/3199101267 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f988c1cc400 0x7f988c1351f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:19.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:20.002+0000 7f9892db5640 1 --2- 172.21.15.37:0/3199101267 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9874004810 0x7f988c1cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:19.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:20.002+0000 7f9892db5640 1 -- 172.21.15.37:0/3199101267 >> 172.21.15.37:0/3199101267 conn(0x7f988c0a64e0 msgr2=0x7f988c0a82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:22:19.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:20.002+0000 7f9892db5640 1 -- 172.21.15.37:0/3199101267 shutdown_connections 2024-04-26T00:22:19.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:20.002+0000 7f9892db5640 1 -- 172.21.15.37:0/3199101267 wait complete. 2024-04-26T00:22:20.935 INFO:tasks.rados.rados.0.smithi045.stdout:407: finishing rollback tid 0 to smithi04526579-8 2024-04-26T00:22:20.935 INFO:tasks.rados.rados.0.smithi045.stdout:407: finishing rollback tid 1 to smithi04526579-8 2024-04-26T00:22:20.935 INFO:tasks.rados.rados.0.smithi045.stdout:407: finishing rollback tid 2 to smithi04526579-8 2024-04-26T00:22:20.935 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 82 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-04-26T00:22:20.936 INFO:tasks.rados.rados.0.smithi045.stdout:407: done (1 left) 2024-04-26T00:22:20.936 INFO:tasks.rados.rados.0.smithi045.stdout:409: done (0 left) 2024-04-26T00:22:20.936 INFO:tasks.rados.rados.0.smithi045.stdout:410: read oid 30 snap -1 2024-04-26T00:22:20.936 INFO:tasks.rados.rados.0.smithi045.stdout:410: expect (ObjNum 149 snap 32 seq_num 149) 2024-04-26T00:22:20.936 INFO:tasks.rados.rados.0.smithi045.stdout:411: write (excl) oid 28 current snap is 46 2024-04-26T00:22:20.936 INFO:tasks.rados.rados.0.smithi045.stdout:411: seq_num 175 ranges {541225=644694,1907290=448390} 2024-04-26T00:22:20.948 INFO:tasks.rados.rados.0.smithi045.stdout:411: writing smithi04526579-28 from 541225 to 1185919 tid 1 2024-04-26T00:22:20.963 INFO:tasks.rados.rados.0.smithi045.stdout:411: writing smithi04526579-28 from 1907290 to 2355680 tid 2 2024-04-26T00:22:20.964 INFO:tasks.rados.rados.0.smithi045.stdout:412: copy_from oid 14 from oid 46 current snap is 46 2024-04-26T00:22:20.964 INFO:tasks.rados.rados.0.smithi045.stdout:410: expect (ObjNum 149 snap 32 seq_num 149) 2024-04-26T00:22:21.104 INFO:tasks.rados.rados.0.smithi045.stdout:411: finishing write tid 1 to smithi04526579-28 2024-04-26T00:22:21.104 INFO:tasks.rados.rados.0.smithi045.stdout:411: finishing write tid 2 to smithi04526579-28 2024-04-26T00:22:21.104 INFO:tasks.rados.rados.0.smithi045.stdout:411: finishing write tid 3 to smithi04526579-28 2024-04-26T00:22:21.104 INFO:tasks.rados.rados.0.smithi045.stdout:411: finishing write tid 4 to smithi04526579-28 2024-04-26T00:22:21.105 INFO:tasks.rados.rados.0.smithi045.stdout:411: oid 28 updating version 0 to 157 2024-04-26T00:22:21.105 INFO:tasks.rados.rados.0.smithi045.stdout:411: oid 28 updating version 157 to 158 2024-04-26T00:22:21.105 INFO:tasks.rados.rados.0.smithi045.stdout:411: oid 28 version 158 is already newer than 156 2024-04-26T00:22:21.105 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 158 (ObjNum 175 snap 46 seq_num 175) dirty exists 2024-04-26T00:22:21.105 INFO:tasks.rados.rados.0.smithi045.stdout:411: left oid 28 (ObjNum 175 snap 46 seq_num 175) 2024-04-26T00:22:21.105 INFO:tasks.rados.rados.0.smithi045.stdout:412: finishing copy_from racing read to smithi04526579-14 2024-04-26T00:22:21.105 INFO:tasks.rados.rados.0.smithi045.stdout:412: finishing copy_from to smithi04526579-14 2024-04-26T00:22:21.105 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 82 (ObjNum 100 snap 24 seq_num 100) dirty exists 2024-04-26T00:22:21.105 INFO:tasks.rados.rados.0.smithi045.stdout:410: done (2 left) 2024-04-26T00:22:21.105 INFO:tasks.rados.rados.0.smithi045.stdout:411: done (1 left) 2024-04-26T00:22:21.105 INFO:tasks.rados.rados.0.smithi045.stdout:412: done (0 left) 2024-04-26T00:22:21.105 INFO:tasks.rados.rados.0.smithi045.stdout:413: snap_remove snap 39 2024-04-26T00:22:21.943 INFO:tasks.rados.rados.0.smithi045.stdout:413: done (0 left) 2024-04-26T00:22:21.943 INFO:tasks.rados.rados.0.smithi045.stdout:414: read oid 25 snap -1 2024-04-26T00:22:21.943 INFO:tasks.rados.rados.0.smithi045.stdout:414: expect (ObjNum 72 snap 11 seq_num 72) 2024-04-26T00:22:21.943 INFO:tasks.rados.rados.0.smithi045.stdout:415: delete oid 32 current snap is 46 2024-04-26T00:22:21.955 INFO:tasks.rados.rados.0.smithi045.stdout:414: expect (ObjNum 72 snap 11 seq_num 72) 2024-04-26T00:22:22.178 INFO:tasks.rados.rados.0.smithi045.stdout:414: done (1 left) 2024-04-26T00:22:22.178 INFO:tasks.rados.rados.0.smithi045.stdout:415: done (0 left) 2024-04-26T00:22:22.178 INFO:tasks.rados.rados.0.smithi045.stdout:416: read oid 25 snap -1 2024-04-26T00:22:22.178 INFO:tasks.rados.rados.0.smithi045.stdout:416: expect (ObjNum 72 snap 11 seq_num 72) 2024-04-26T00:22:22.178 INFO:tasks.rados.rados.0.smithi045.stdout:417: write (excl) oid 49 current snap is 46 2024-04-26T00:22:22.178 INFO:tasks.rados.rados.0.smithi045.stdout:417: seq_num 177 ranges {574839=795192,1907901=452059} 2024-04-26T00:22:22.204 INFO:tasks.rados.rados.0.smithi045.stdout:417: writing smithi04526579-49 from 574839 to 1370031 tid 1 2024-04-26T00:22:22.221 INFO:tasks.rados.rados.0.smithi045.stdout:417: writing smithi04526579-49 from 1907901 to 2359960 tid 2 2024-04-26T00:22:22.221 INFO:tasks.rados.rados.0.smithi045.stdout:418: delete oid 16 current snap is 46 2024-04-26T00:22:22.221 INFO:tasks.rados.rados.0.smithi045.stdout:416: expect (ObjNum 72 snap 11 seq_num 72) 2024-04-26T00:22:22.452 INFO:tasks.rados.rados.0.smithi045.stdout:417: finishing write tid 1 to smithi04526579-49 2024-04-26T00:22:22.452 INFO:tasks.rados.rados.0.smithi045.stdout:417: finishing write tid 2 to smithi04526579-49 2024-04-26T00:22:22.453 INFO:tasks.rados.rados.0.smithi045.stdout:417: finishing write tid 3 to smithi04526579-49 2024-04-26T00:22:22.453 INFO:tasks.rados.rados.0.smithi045.stdout:417: finishing write tid 4 to smithi04526579-49 2024-04-26T00:22:22.453 INFO:tasks.rados.rados.0.smithi045.stdout:417: oid 49 updating version 0 to 103 2024-04-26T00:22:22.453 INFO:tasks.rados.rados.0.smithi045.stdout:417: oid 49 version 103 is already newer than 97 2024-04-26T00:22:22.453 INFO:tasks.rados.rados.0.smithi045.stdout:417: oid 49 version 103 is already newer than 102 2024-04-26T00:22:22.453 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 103 (ObjNum 177 snap 46 seq_num 177) dirty exists 2024-04-26T00:22:22.453 INFO:tasks.rados.rados.0.smithi045.stdout:417: left oid 49 (ObjNum 177 snap 46 seq_num 177) 2024-04-26T00:22:22.453 INFO:tasks.rados.rados.0.smithi045.stdout:416: done (2 left) 2024-04-26T00:22:22.453 INFO:tasks.rados.rados.0.smithi045.stdout:417: done (1 left) 2024-04-26T00:22:22.453 INFO:tasks.rados.rados.0.smithi045.stdout:418: done (0 left) 2024-04-26T00:22:22.453 INFO:tasks.rados.rados.0.smithi045.stdout:419: delete oid 23 current snap is 46 2024-04-26T00:22:22.460 INFO:tasks.rados.rados.0.smithi045.stdout:419: done (0 left) 2024-04-26T00:22:22.460 INFO:tasks.rados.rados.0.smithi045.stdout:420: rollback oid 40 current snap is 46 2024-04-26T00:22:22.461 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 40 to 32 2024-04-26T00:22:22.461 INFO:tasks.rados.rados.0.smithi045.stdout:421: snap_create 2024-04-26T00:22:22.469 INFO:tasks.rados.rados.0.smithi045.stdout:420: finishing rollback tid 0 to smithi04526579-40 2024-04-26T00:22:22.470 INFO:tasks.rados.rados.0.smithi045.stdout:420: finishing rollback tid 1 to smithi04526579-40 2024-04-26T00:22:22.471 INFO:tasks.rados.rados.0.smithi045.stdout:420: finishing rollback tid 2 to smithi04526579-40 2024-04-26T00:22:22.471 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 95 (ObjNum 131 snap 30 seq_num 131) dirty exists 2024-04-26T00:22:22.921 INFO:tasks.rados.rados.0.smithi045.stdout:420: done (1 left) 2024-04-26T00:22:22.921 INFO:tasks.rados.rados.0.smithi045.stdout:421: done (0 left) 2024-04-26T00:22:22.921 INFO:tasks.rados.rados.0.smithi045.stdout:422: write (excl) oid 28 current snap is 47 2024-04-26T00:22:22.921 INFO:tasks.rados.rados.0.smithi045.stdout:422: seq_num 180 ranges {625260=620939,1683347=599386,2943668=81592} 2024-04-26T00:22:22.941 INFO:tasks.rados.rados.0.smithi045.stdout:422: writing smithi04526579-28 from 625260 to 1246199 tid 1 2024-04-26T00:22:22.959 INFO:tasks.rados.rados.0.smithi045.stdout:422: writing smithi04526579-28 from 1683347 to 2282733 tid 2 2024-04-26T00:22:22.972 INFO:tasks.rados.rados.0.smithi045.stdout:422: writing smithi04526579-28 from 2943668 to 3025260 tid 3 2024-04-26T00:22:22.972 INFO:tasks.rados.rados.0.smithi045.stdout:423: write oid 47 current snap is 47 2024-04-26T00:22:22.972 INFO:tasks.rados.rados.0.smithi045.stdout:422: finishing write tid 1 to smithi04526579-28 2024-04-26T00:22:22.972 INFO:tasks.rados.rados.0.smithi045.stdout:422: finishing write tid 2 to smithi04526579-28 2024-04-26T00:22:22.972 INFO:tasks.rados.rados.0.smithi045.stdout:423: seq_num 181 ranges {642067=696188,2025476=459397,2971091=70976} 2024-04-26T00:22:22.990 INFO:tasks.rados.rados.0.smithi045.stdout:423: writing smithi04526579-47 from 642067 to 1338255 tid 1 2024-04-26T00:22:22.998 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:22:23.007 INFO:tasks.rados.rados.0.smithi045.stdout:423: writing smithi04526579-47 from 2025476 to 2484873 tid 2 2024-04-26T00:22:23.014 INFO:tasks.rados.rados.0.smithi045.stdout:423: writing smithi04526579-47 from 2971091 to 3042067 tid 3 2024-04-26T00:22:23.014 INFO:tasks.rados.rados.0.smithi045.stdout:424: snap_create 2024-04-26T00:22:23.014 INFO:tasks.rados.rados.0.smithi045.stdout:422: finishing write tid 3 to smithi04526579-28 2024-04-26T00:22:23.014 INFO:tasks.rados.rados.0.smithi045.stdout:422: finishing write tid 4 to smithi04526579-28 2024-04-26T00:22:23.015 INFO:tasks.rados.rados.0.smithi045.stdout:422: finishing write tid 5 to smithi04526579-28 2024-04-26T00:22:23.015 INFO:tasks.rados.rados.0.smithi045.stdout:422: oid 28 updating version 0 to 159 2024-04-26T00:22:23.015 INFO:tasks.rados.rados.0.smithi045.stdout:422: oid 28 updating version 159 to 162 2024-04-26T00:22:23.015 INFO:tasks.rados.rados.0.smithi045.stdout:422: oid 28 version 162 is already newer than 161 2024-04-26T00:22:23.015 INFO:tasks.rados.rados.0.smithi045.stdout:422: oid 28 version 162 is already newer than 160 2024-04-26T00:22:23.015 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 162 (ObjNum 180 snap 47 seq_num 180) dirty exists 2024-04-26T00:22:23.015 INFO:tasks.rados.rados.0.smithi045.stdout:422: left oid 28 (ObjNum 180 snap 47 seq_num 180) 2024-04-26T00:22:23.015 INFO:tasks.rados.rados.0.smithi045.stdout:423: finishing write tid 1 to smithi04526579-47 2024-04-26T00:22:23.015 INFO:tasks.rados.rados.0.smithi045.stdout:423: finishing write tid 2 to smithi04526579-47 2024-04-26T00:22:23.018 INFO:tasks.rados.rados.0.smithi045.stdout:423: finishing write tid 3 to smithi04526579-47 2024-04-26T00:22:23.019 INFO:tasks.rados.rados.0.smithi045.stdout:423: finishing write tid 4 to smithi04526579-47 2024-04-26T00:22:23.020 INFO:tasks.rados.rados.0.smithi045.stdout:423: finishing write tid 5 to smithi04526579-47 2024-04-26T00:22:23.020 INFO:tasks.rados.rados.0.smithi045.stdout:423: oid 47 updating version 0 to 107 2024-04-26T00:22:23.020 INFO:tasks.rados.rados.0.smithi045.stdout:423: oid 47 version 107 is already newer than 106 2024-04-26T00:22:23.020 INFO:tasks.rados.rados.0.smithi045.stdout:423: oid 47 updating version 107 to 108 2024-04-26T00:22:23.020 INFO:tasks.rados.rados.0.smithi045.stdout:423: oid 47 version 108 is already newer than 105 2024-04-26T00:22:23.020 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 108 (ObjNum 181 snap 47 seq_num 181) dirty exists 2024-04-26T00:22:23.020 INFO:tasks.rados.rados.0.smithi045.stdout:423: left oid 47 (ObjNum 181 snap 47 seq_num 181) 2024-04-26T00:22:23.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.150+0000 7f04de0c4640 10 monclient: get_monmap_and_config 2024-04-26T00:22:23.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.150+0000 7f04de0c4640 10 monclient: build_initial_monmap 2024-04-26T00:22:23.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.150+0000 7f04de0c4640 10 monclient: monmap: 2024-04-26T00:22:23.129 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:22:23.130 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:23.130 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:22:23.154071+0000 2024-04-26T00:22:23.130 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:22:23.154071+0000 2024-04-26T00:22:23.130 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:22:23.130 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:23.130 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:22:23.130 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:22:23.130 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:22:23.130 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:22:23.130 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:22:23.130 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:23.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.150+0000 7f04de0c4640 1 Processor -- start 2024-04-26T00:22:23.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.150+0000 7f04de0c4640 1 -- start start 2024-04-26T00:22:23.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.150+0000 7f04de0c4640 10 monclient: init 2024-04-26T00:22:23.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.150+0000 7f04de0c4640 10 monclient: _reopen_session rank -1 2024-04-26T00:22:23.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.150+0000 7f04de0c4640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-04-26T00:22:23.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.150+0000 7f04de0c4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f04d8136da0 addr v1:172.21.15.37:6789/0 2024-04-26T00:22:23.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.150+0000 7f04de0c4640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f04d8137180 0x7f04d81375c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:23.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.150+0000 7f04de0c4640 10 monclient(hunting): picked mon.noname-a con 0x7f04d8137180 addr v2:172.21.15.37:3300/0 2024-04-26T00:22:23.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.150+0000 7f04de0c4640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f04d813c2d0 0x7f04d813e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:23.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.150+0000 7f04de0c4640 10 monclient(hunting): picked mon.noname-b con 0x7f04d813c2d0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:22:23.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.150+0000 7f04de0c4640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04d809b4f0 con 0x7f04d8136da0 2024-04-26T00:22:23.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.150+0000 7f04de0c4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:23.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.150+0000 7f04de0c4640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f04d809b210 con 0x7f04d8137180 2024-04-26T00:22:23.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.150+0000 7f04de0c4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:23.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.150+0000 7f04de0c4640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f04d809b7d0 con 0x7f04d813c2d0 2024-04-26T00:22:23.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.150+0000 7f04de0c4640 10 monclient(hunting): _renew_subs 2024-04-26T00:22:23.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.154+0000 7f04de0c4640 10 monclient(hunting): authenticate will time out at 1124.460012s 2024-04-26T00:22:23.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.154+0000 7f04d77fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f04d8136da0 0x7f04d809bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:35480/0 (socket says 172.21.15.37:35480) 2024-04-26T00:22:23.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.154+0000 7f04d77fe640 1 -- 172.21.15.37:0/4156315554 learned_addr learned my addr 172.21.15.37:0/4156315554 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:22:23.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.154+0000 7f04d6ffd640 1 --2- 172.21.15.37:0/4156315554 >> v2:172.21.15.37:3300/0 conn(0x7f04d8137180 0x7f04d81375c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:23.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.154+0000 7f04d7fff640 1 --2- 172.21.15.37:0/4156315554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f04d813c2d0 0x7f04d813e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:23.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.154+0000 7f04d6ffd640 10 monclient(hunting): get_auth_request con 0x7f04d8137180 auth_method 0 2024-04-26T00:22:23.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.154+0000 7f04d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:23.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.154+0000 7f04d6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:23.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.154+0000 7f04d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:23.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.154+0000 7f04d7fff640 10 monclient(hunting): get_auth_request con 0x7f04d813c2d0 auth_method 0 2024-04-26T00:22:23.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.154+0000 7f04d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:23.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.154+0000 7f04d7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:23.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.154+0000 7f04d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:23.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.154+0000 7f04d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:23.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.154+0000 7f04d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:23.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.154+0000 7f04d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:23.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.154+0000 7f04d7fff640 1 -- 172.21.15.37:0/4156315554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f04d813c2d0 msgr2=0x7f04d813e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:22:23.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.154+0000 7f04d7fff640 1 -- 172.21.15.37:0/4156315554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f04d813c2d0 msgr2=0x7f04d813e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:22:23.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.154+0000 7f04d7fff640 1 --2- 172.21.15.37:0/4156315554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f04d813c2d0 0x7f04d813e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:22:23.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.154+0000 7f04d7fff640 1 --2- 172.21.15.37:0/4156315554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f04d813c2d0 0x7f04d813e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:22:23.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.154+0000 7f04d6ffd640 10 monclient(hunting): handle_auth_done global_id 4614 payload 274 2024-04-26T00:22:23.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.154+0000 7f04d6ffd640 1 -- 172.21.15.37:0/4156315554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f04d813c2d0 msgr2=0x7f04d813e6c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:22:23.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.154+0000 7f04d6ffd640 1 --2- 172.21.15.37:0/4156315554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f04d813c2d0 0x7f04d813e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:23.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.154+0000 7f04d6ffd640 1 -- 172.21.15.37:0/4156315554 >> v1:172.21.15.37:6789/0 conn(0x7f04d8136da0 legacy=0x7f04d809bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:22:23.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.154+0000 7f04d6ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:22:23.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.154+0000 7f04d6ffd640 1 monclient: found mon.noname-a 2024-04-26T00:22:23.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.154+0000 7f04d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:22:23.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.154+0000 7f04d6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:22:23.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.154+0000 7f04d6ffd640 1 -- 172.21.15.37:0/4156315554 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04d8137b00 con 0x7f04d8137180 2024-04-26T00:22:23.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d6ffd640 1 -- 172.21.15.37:0/4156315554 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f04d8137180 msgr2=0x7f04d81375c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:23.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d6ffd640 1 --2- 172.21.15.37:0/4156315554 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f04d8137180 0x7f04d81375c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f04cc0097c0 tx=0x7f04cc01c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dec4deb4a2e27451 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:23.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d67fc640 1 -- 172.21.15.37:0/4156315554 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04cc0b0070 con 0x7f04d8137180 2024-04-26T00:22:23.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:23.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:22:23.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d67fc640 10 monclient: dump: 2024-04-26T00:22:23.143 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:23.143 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:23.143 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:23.143 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:23.143 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:23.143 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:23.143 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:23.143 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:23.143 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:23.143 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:23.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d67fc640 10 monclient: _finish_auth 0 2024-04-26T00:22:23.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d67fc640 10 monclient: _check_auth_tickets 2024-04-26T00:22:23.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:22:23.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d67fc640 1 -- 172.21.15.37:0/4156315554 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04cc0ab600 con 0x7f04d8137180 2024-04-26T00:22:23.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d67fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:22:23.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d67fc640 1 -- 172.21.15.37:0/4156315554 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04cc0ab9d0 con 0x7f04d8137180 2024-04-26T00:22:23.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04de0c4640 5 monclient: authenticate success, global_id 4614 2024-04-26T00:22:23.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04de0c4640 10 monclient: get_monmap_and_config success 2024-04-26T00:22:23.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04de0c4640 4 set_mon_vals no callback set 2024-04-26T00:22:23.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:23.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:22:23.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d67fc640 10 monclient: dump: 2024-04-26T00:22:23.144 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:23.144 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:23.144 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:23.145 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:23.145 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:23.145 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:23.145 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:23.145 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:23.145 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:23.145 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:23.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04de0c4640 10 monclient: shutdown 2024-04-26T00:22:23.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04de0c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:22:23.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04de0c4640 1 -- 172.21.15.37:0/4156315554 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f04d8137180 msgr2=0x7f04d81375c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:23.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04de0c4640 1 --2- 172.21.15.37:0/4156315554 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f04d8137180 0x7f04d81375c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f04cc0097c0 tx=0x7f04cc01c320 comp rx=0 tx=0).stop 2024-04-26T00:22:23.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04de0c4640 1 -- 172.21.15.37:0/4156315554 shutdown_connections 2024-04-26T00:22:23.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04de0c4640 1 --2- 172.21.15.37:0/4156315554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f04d813c2d0 0x7f04d813e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:23.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04de0c4640 1 --2- 172.21.15.37:0/4156315554 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f04d8137180 0x7f04d81375c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:23.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04de0c4640 1 -- 172.21.15.37:0/4156315554 >> 172.21.15.37:0/4156315554 conn(0x7f04d80a5f00 msgr2=0x7f04d80a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:22:23.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04de0c4640 1 -- 172.21.15.37:0/4156315554 shutdown_connections 2024-04-26T00:22:23.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04de0c4640 1 -- 172.21.15.37:0/4156315554 wait complete. 2024-04-26T00:22:23.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04de0c4640 10 monclient: build_initial_monmap 2024-04-26T00:22:23.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04de0c4640 10 monclient: monmap: 2024-04-26T00:22:23.146 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:22:23.146 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:22:23.146 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:22:23.146 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:22:23.146 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:22:23.146 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:22:23.146 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:22:23.146 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:22:23.146 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:22:23.146 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:23.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04de0c4640 1 Processor -- start 2024-04-26T00:22:23.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04de0c4640 1 -- start start 2024-04-26T00:22:23.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04de0c4640 10 monclient: init 2024-04-26T00:22:23.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04de0c4640 10 monclient: _reopen_session rank -1 2024-04-26T00:22:23.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04de0c4640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:22:23.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04de0c4640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f04d8136da0 0x7f04d81cf140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:23.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04de0c4640 10 monclient(hunting): picked mon.noname-a con 0x7f04d8136da0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:22:23.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04de0c4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04d8137180 0x7f04d813b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:23.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04de0c4640 10 monclient(hunting): picked mon.noname-b con 0x7f04d8137180 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:22:23.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04de0c4640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f04d813c2d0 0x7f04d809e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:23.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04de0c4640 10 monclient(hunting): picked mon.noname-c con 0x7f04d813c2d0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:22:23.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04de0c4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:23.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04de0c4640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f04d81417b0 con 0x7f04d8136da0 2024-04-26T00:22:23.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04de0c4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:23.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04de0c4640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f04d8141630 con 0x7f04d8137180 2024-04-26T00:22:23.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04de0c4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:23.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04de0c4640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f04d8141930 con 0x7f04d813c2d0 2024-04-26T00:22:23.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04de0c4640 10 monclient(hunting): _renew_subs 2024-04-26T00:22:23.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04de0c4640 10 monclient(hunting): authenticate will time out at 1124.467220s 2024-04-26T00:22:23.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d77fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f04d8136da0 0x7f04d81cf140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:23.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d7fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f04d813c2d0 0x7f04d809e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:23.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d77fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f04d8136da0 0x7f04d81cf140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:55686/0 (socket says 172.21.15.37:55686) 2024-04-26T00:22:23.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d77fe640 1 -- 172.21.15.37:0/600340762 learned_addr learned my addr 172.21.15.37:0/600340762 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:22:23.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d7fff640 10 monclient(hunting): get_auth_request con 0x7f04d813c2d0 auth_method 0 2024-04-26T00:22:23.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:23.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:23.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:23.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d77fe640 10 monclient(hunting): get_auth_request con 0x7f04d8136da0 auth_method 0 2024-04-26T00:22:23.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:23.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d77fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:23.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:23.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d7fff640 1 -- 172.21.15.37:0/600340762 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f04d813c2d0 msgr2=0x7f04d809e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:22:23.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d7fff640 1 -- 172.21.15.37:0/600340762 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f04d813c2d0 msgr2=0x7f04d809e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:22:23.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d7fff640 1 --2- 172.21.15.37:0/600340762 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f04d813c2d0 0x7f04d809e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:22:23.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:23.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:23.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:23.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d7fff640 1 --2- 172.21.15.37:0/600340762 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f04d813c2d0 0x7f04d809e320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:22:23.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d6ffd640 1 --2- 172.21.15.37:0/600340762 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04d8137180 0x7f04d813b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:23.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d6ffd640 10 monclient(hunting): get_auth_request con 0x7f04d8137180 auth_method 0 2024-04-26T00:22:23.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:23.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:23.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:23.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d77fe640 10 monclient(hunting): handle_auth_done global_id 4617 payload 824 2024-04-26T00:22:23.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d77fe640 1 -- 172.21.15.37:0/600340762 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f04d813c2d0 msgr2=0x7f04d809e320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:22:23.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d77fe640 1 --2- 172.21.15.37:0/600340762 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f04d813c2d0 0x7f04d809e320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:23.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d77fe640 1 -- 172.21.15.37:0/600340762 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04d8137180 msgr2=0x7f04d813b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:23.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d77fe640 1 --2- 172.21.15.37:0/600340762 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04d8137180 0x7f04d813b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:23.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d77fe640 10 monclient: _finish_hunting 0 2024-04-26T00:22:23.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d77fe640 1 monclient: found mon.noname-a 2024-04-26T00:22:23.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:22:23.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:22:23.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.158+0000 7f04d77fe640 1 -- 172.21.15.37:0/600340762 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04d809aca0 con 0x7f04d8136da0 2024-04-26T00:22:23.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.162+0000 7f04d77fe640 1 -- 172.21.15.37:0/600340762 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f04d8136da0 msgr2=0x7f04d81cf140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:23.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.162+0000 7f04d77fe640 1 --2- 172.21.15.37:0/600340762 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f04d8136da0 0x7f04d81cf140 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f04c800c4a0 tx=0x7f04c800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:23.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.162+0000 7f04bffff640 1 -- 172.21.15.37:0/600340762 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c8017070 con 0x7f04d8136da0 2024-04-26T00:22:23.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.162+0000 7f04bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:23.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.162+0000 7f04bffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:22:23.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.162+0000 7f04bffff640 10 monclient: dump: 2024-04-26T00:22:23.152 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:23.152 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:23.152 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:23.152 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:23.152 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:23.152 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:23.152 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:23.152 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:23.152 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:23.152 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:23.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.162+0000 7f04bffff640 10 monclient: _finish_auth 0 2024-04-26T00:22:23.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.162+0000 7f04bffff640 10 monclient: _check_auth_tickets 2024-04-26T00:22:23.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.162+0000 7f04bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:22:23.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.162+0000 7f04bffff640 1 -- 172.21.15.37:0/600340762 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04c8007070 con 0x7f04d8136da0 2024-04-26T00:22:23.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.162+0000 7f04bffff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:22:23.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.162+0000 7f04bffff640 1 -- 172.21.15.37:0/600340762 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c8007460 con 0x7f04d8136da0 2024-04-26T00:22:23.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.162+0000 7f04bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:23.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.162+0000 7f04bffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:22:23.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.162+0000 7f04bffff640 10 monclient: dump: 2024-04-26T00:22:23.153 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:23.153 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:23.153 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:23.153 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:23.153 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:23.153 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:23.153 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:23.153 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:23.154 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:23.154 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:23.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.162+0000 7f04de0c4640 5 monclient: authenticate success, global_id 4617 2024-04-26T00:22:23.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.162+0000 7f04de0c4640 10 monclient: _renew_subs 2024-04-26T00:22:23.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.162+0000 7f04de0c4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:22:23.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.162+0000 7f04de0c4640 1 -- 172.21.15.37:0/600340762 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04d80a86f0 con 0x7f04d8136da0 2024-04-26T00:22:23.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.162+0000 7f04d5ffb640 4 set_mon_vals no callback set 2024-04-26T00:22:23.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.162+0000 7f04de0c4640 10 monclient: _renew_subs 2024-04-26T00:22:23.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.162+0000 7f04de0c4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:22:23.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.162+0000 7f04de0c4640 1 -- 172.21.15.37:0/600340762 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04d81399d0 con 0x7f04d8136da0 2024-04-26T00:22:23.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.162+0000 7f04de0c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:23.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.162+0000 7f04de0c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:23.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.162+0000 7f04de0c4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:22:23.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.162+0000 7f04de0c4640 1 -- 172.21.15.37:0/600340762 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04d8066f40 con 0x7f04d8136da0 2024-04-26T00:22:23.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.162+0000 7f04bffff640 1 -- 172.21.15.37:0/600340762 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f04c801a070 con 0x7f04d8136da0 2024-04-26T00:22:23.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.162+0000 7f04bffff640 1 --2- 172.21.15.37:0/600340762 >> v2:172.21.15.45:6812/2379918623 conn(0x7f04c4035f00 0x7f04c40383c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:23.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.162+0000 7f04bffff640 1 -- 172.21.15.37:0/600340762 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 3975+0+0 (secure 0 0 0) 0x7f04c8047fb0 con 0x7f04d8136da0 2024-04-26T00:22:23.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.166+0000 7f04d6ffd640 1 --2- 172.21.15.37:0/600340762 >> v2:172.21.15.45:6812/2379918623 conn(0x7f04c4035f00 0x7f04c40383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:23.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.166+0000 7f04d6ffd640 10 monclient: get_auth_request con 0x7f04c4035f00 auth_method 0 2024-04-26T00:22:23.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.166+0000 7f04d6ffd640 1 --2- 172.21.15.37:0/600340762 >> v2:172.21.15.45:6812/2379918623 conn(0x7f04c4035f00 0x7f04c40383c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f04cc0ab000 tx=0x7f04cc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:23.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.170+0000 7f04bffff640 1 -- 172.21.15.37:0/600340762 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f04c8010300 con 0x7f04d8136da0 2024-04-26T00:22:23.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.170+0000 7f04bffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:23.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.170+0000 7f04bffff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:22:23.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.394+0000 7f04de0c4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:22:23.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.394+0000 7f04de0c4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:22:23.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.394+0000 7f04de0c4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:22:23.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.394+0000 7f04de0c4640 1 -- 172.21.15.37:0/600340762 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f04d81382a0 con 0x7f04d8136da0 2024-04-26T00:22:23.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.394+0000 7f04bffff640 1 -- 172.21.15.37:0/600340762 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f04c801b460 con 0x7f04d8136da0 2024-04-26T00:22:23.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.394+0000 7f04bffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:22:23.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.394+0000 7f04bffff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:22:23.372 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":22,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:22:23.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.394+0000 7f04de0c4640 1 -- 172.21.15.37:0/600340762 >> v2:172.21.15.45:6812/2379918623 conn(0x7f04c4035f00 msgr2=0x7f04c40383c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:23.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.394+0000 7f04de0c4640 1 --2- 172.21.15.37:0/600340762 >> v2:172.21.15.45:6812/2379918623 conn(0x7f04c4035f00 0x7f04c40383c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f04cc0ab000 tx=0x7f04cc008040 comp rx=0 tx=0).stop 2024-04-26T00:22:23.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.394+0000 7f04de0c4640 10 monclient: shutdown 2024-04-26T00:22:23.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.394+0000 7f04de0c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:22:23.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.394+0000 7f04de0c4640 1 -- 172.21.15.37:0/600340762 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f04d8136da0 msgr2=0x7f04d81cf140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:23.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.394+0000 7f04de0c4640 1 --2- 172.21.15.37:0/600340762 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f04d8136da0 0x7f04d81cf140 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f04c800c4a0 tx=0x7f04c800c960 comp rx=0 tx=0).stop 2024-04-26T00:22:23.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.394+0000 7f04de0c4640 1 -- 172.21.15.37:0/600340762 shutdown_connections 2024-04-26T00:22:23.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.394+0000 7f04de0c4640 1 --2- 172.21.15.37:0/600340762 >> v2:172.21.15.45:6812/2379918623 conn(0x7f04c4035f00 0x7f04c40383c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:23.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.394+0000 7f04de0c4640 1 --2- 172.21.15.37:0/600340762 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f04d813c2d0 0x7f04d809e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:23.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.394+0000 7f04de0c4640 1 --2- 172.21.15.37:0/600340762 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04d8137180 0x7f04d813b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:23.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.394+0000 7f04de0c4640 1 --2- 172.21.15.37:0/600340762 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f04d8136da0 0x7f04d81cf140 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:23.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.394+0000 7f04de0c4640 1 -- 172.21.15.37:0/600340762 >> 172.21.15.37:0/600340762 conn(0x7f04d80a5f00 msgr2=0x7f04d80a3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:22:23.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.394+0000 7f04de0c4640 1 -- 172.21.15.37:0/600340762 shutdown_connections 2024-04-26T00:22:23.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:23.394+0000 7f04de0c4640 1 -- 172.21.15.37:0/600340762 wait complete. 2024-04-26T00:22:23.952 INFO:tasks.rados.rados.0.smithi045.stdout:422: done (2 left) 2024-04-26T00:22:23.952 INFO:tasks.rados.rados.0.smithi045.stdout:423: done (1 left) 2024-04-26T00:22:23.952 INFO:tasks.rados.rados.0.smithi045.stdout:424: done (0 left) 2024-04-26T00:22:23.952 INFO:tasks.rados.rados.0.smithi045.stdout:425: snap_create 2024-04-26T00:22:24.954 INFO:tasks.rados.rados.0.smithi045.stdout:425: done (0 left) 2024-04-26T00:22:24.954 INFO:tasks.rados.rados.0.smithi045.stdout:426: write oid 3 current snap is 49 2024-04-26T00:22:24.954 INFO:tasks.rados.rados.0.smithi045.stdout:426: seq_num 182 ranges {658874=771437,2083958=603055} 2024-04-26T00:22:24.974 INFO:tasks.rados.rados.0.smithi045.stdout:426: writing smithi04526579-3 from 658874 to 1430311 tid 1 2024-04-26T00:22:24.993 INFO:tasks.rados.rados.0.smithi045.stdout:426: writing smithi04526579-3 from 2083958 to 2687013 tid 2 2024-04-26T00:22:24.993 INFO:tasks.rados.rados.0.smithi045.stdout:427: read oid 36 snap -1 2024-04-26T00:22:24.993 INFO:tasks.rados.rados.0.smithi045.stdout:427: expect deleted 2024-04-26T00:22:24.993 INFO:tasks.rados.rados.0.smithi045.stdout:428: read oid 43 snap -1 2024-04-26T00:22:24.993 INFO:tasks.rados.rados.0.smithi045.stdout:428: expect (ObjNum 15 snap 0 seq_num 15) 2024-04-26T00:22:24.993 INFO:tasks.rados.rados.0.smithi045.stdout:429: read oid 34 snap -1 2024-04-26T00:22:24.993 INFO:tasks.rados.rados.0.smithi045.stdout:429: expect (ObjNum 173 snap 45 seq_num 173) 2024-04-26T00:22:24.993 INFO:tasks.rados.rados.0.smithi045.stdout:430: read oid 46 snap -1 2024-04-26T00:22:24.994 INFO:tasks.rados.rados.0.smithi045.stdout:430: expect (ObjNum 100 snap 24 seq_num 100) 2024-04-26T00:22:24.994 INFO:tasks.rados.rados.0.smithi045.stdout:431: snap_remove snap 33 2024-04-26T00:22:25.973 INFO:tasks.rados.rados.0.smithi045.stdout:431: done (5 left) 2024-04-26T00:22:25.973 INFO:tasks.rados.rados.0.smithi045.stdout:432: snap_remove snap 47 2024-04-26T00:22:26.392 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:22:26.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16cc230640 10 monclient: get_monmap_and_config 2024-04-26T00:22:26.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16cc230640 10 monclient: build_initial_monmap 2024-04-26T00:22:26.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16cc230640 10 monclient: monmap: 2024-04-26T00:22:26.524 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:22:26.525 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:26.525 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:22:26.549392+0000 2024-04-26T00:22:26.525 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:22:26.549392+0000 2024-04-26T00:22:26.525 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:22:26.525 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:26.525 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:22:26.525 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:22:26.525 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:22:26.525 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:22:26.525 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:22:26.525 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:26.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16cc230640 1 Processor -- start 2024-04-26T00:22:26.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16cc230640 1 -- start start 2024-04-26T00:22:26.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16cc230640 10 monclient: init 2024-04-26T00:22:26.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16cc230640 10 monclient: _reopen_session rank -1 2024-04-26T00:22:26.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16cc230640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-04-26T00:22:26.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16cc230640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f16c4136e10 addr v1:172.21.15.37:6789/0 2024-04-26T00:22:26.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16cc230640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16c4137900 0x7f16c409c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:26.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16cc230640 10 monclient(hunting): picked mon.noname-b con 0x7f16c4137900 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:22:26.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16cc230640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f16c413c340 0x7f16c413e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:26.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16cc230640 10 monclient(hunting): picked mon.noname-a con 0x7f16c413c340 addr v2:172.21.15.37:3300/0 2024-04-26T00:22:26.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16cc230640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16c409b680 con 0x7f16c4136e10 2024-04-26T00:22:26.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16cc230640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:26.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16cc230640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f16c409b8f0 con 0x7f16c413c340 2024-04-26T00:22:26.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16cc230640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:26.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16cc230640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f16c409bbd0 con 0x7f16c4137900 2024-04-26T00:22:26.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16cc230640 10 monclient(hunting): _renew_subs 2024-04-26T00:22:26.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16cc230640 10 monclient(hunting): authenticate will time out at 1127.855307s 2024-04-26T00:22:26.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16ca7cd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f16c413c340 0x7f16c413e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:26.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16c9fcc640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f16c4136e10 0x7f16c41371f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:35494/0 (socket says 172.21.15.37:35494) 2024-04-26T00:22:26.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16c97cb640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16c4137900 0x7f16c409c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:26.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16c9fcc640 1 -- 172.21.15.37:0/3189888758 learned_addr learned my addr 172.21.15.37:0/3189888758 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:22:26.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16ca7cd640 10 monclient(hunting): get_auth_request con 0x7f16c413c340 auth_method 0 2024-04-26T00:22:26.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16ca7cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:26.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16ca7cd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:26.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16ca7cd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:26.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16c97cb640 10 monclient(hunting): get_auth_request con 0x7f16c4137900 auth_method 0 2024-04-26T00:22:26.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16c97cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:26.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16c97cb640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:26.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16c97cb640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:26.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16ca7cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:26.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16ca7cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:26.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16ca7cd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:26.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16c97cb640 1 -- 172.21.15.37:0/3189888758 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16c4137900 msgr2=0x7f16c409c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:22:26.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16c97cb640 1 -- 172.21.15.37:0/3189888758 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16c4137900 msgr2=0x7f16c409c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:22:26.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16c97cb640 1 --2- 172.21.15.37:0/3189888758 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16c4137900 0x7f16c409c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:22:26.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16c97cb640 1 --2- 172.21.15.37:0/3189888758 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16c4137900 0x7f16c409c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:22:26.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16ca7cd640 10 monclient(hunting): handle_auth_done global_id 4620 payload 274 2024-04-26T00:22:26.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16ca7cd640 1 -- 172.21.15.37:0/3189888758 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16c4137900 msgr2=0x7f16c409c5b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:22:26.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16ca7cd640 1 --2- 172.21.15.37:0/3189888758 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16c4137900 0x7f16c409c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:26.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16ca7cd640 1 -- 172.21.15.37:0/3189888758 >> v1:172.21.15.37:6789/0 conn(0x7f16c4136e10 legacy=0x7f16c41371f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:22:26.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16ca7cd640 10 monclient: _finish_hunting 0 2024-04-26T00:22:26.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16ca7cd640 1 monclient: found mon.noname-a 2024-04-26T00:22:26.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16ca7cd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:22:26.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16ca7cd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:22:26.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.546+0000 7f16ca7cd640 1 -- 172.21.15.37:0/3189888758 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16c40a5d90 con 0x7f16c413c340 2024-04-26T00:22:26.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16ca7cd640 1 -- 172.21.15.37:0/3189888758 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16c413c340 msgr2=0x7f16c413e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:26.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16ca7cd640 1 --2- 172.21.15.37:0/3189888758 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16c413c340 0x7f16c413e730 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f16bc009dc0 tx=0x7f16bc01c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8108c5f8ca13248 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:26.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16c8fca640 1 -- 172.21.15.37:0/3189888758 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16bc0ac070 con 0x7f16c413c340 2024-04-26T00:22:26.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16c8fca640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:26.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16c8fca640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:22:26.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16c8fca640 10 monclient: dump: 2024-04-26T00:22:26.538 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:26.538 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:26.538 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:26.538 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:26.539 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:26.539 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:26.539 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:26.539 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:26.539 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:26.539 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:26.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16c8fca640 10 monclient: _finish_auth 0 2024-04-26T00:22:26.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16c8fca640 10 monclient: _check_auth_tickets 2024-04-26T00:22:26.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16c8fca640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:22:26.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16c8fca640 1 -- 172.21.15.37:0/3189888758 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16bc0a6660 con 0x7f16c413c340 2024-04-26T00:22:26.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16cc230640 5 monclient: authenticate success, global_id 4620 2024-04-26T00:22:26.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16cc230640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:22:26.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16c8fca640 10 monclient: handle_config config(0 keys) 2024-04-26T00:22:26.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16cc230640 10 monclient: get_monmap_and_config success 2024-04-26T00:22:26.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16c8fca640 1 -- 172.21.15.37:0/3189888758 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16bc0a6a30 con 0x7f16c413c340 2024-04-26T00:22:26.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16cc230640 4 set_mon_vals no callback set 2024-04-26T00:22:26.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16c8fca640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:26.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16c8fca640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:22:26.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16c8fca640 10 monclient: dump: 2024-04-26T00:22:26.540 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:26.540 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:26.540 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:26.540 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:26.540 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:26.540 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:26.540 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:26.541 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:26.541 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:26.541 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:26.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16cc230640 10 monclient: shutdown 2024-04-26T00:22:26.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16cc230640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:22:26.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16cc230640 1 -- 172.21.15.37:0/3189888758 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16c413c340 msgr2=0x7f16c413e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:26.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16cc230640 1 --2- 172.21.15.37:0/3189888758 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16c413c340 0x7f16c413e730 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f16bc009dc0 tx=0x7f16bc01c7d0 comp rx=0 tx=0).stop 2024-04-26T00:22:26.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16cc230640 1 -- 172.21.15.37:0/3189888758 shutdown_connections 2024-04-26T00:22:26.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16cc230640 1 --2- 172.21.15.37:0/3189888758 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16c413c340 0x7f16c413e730 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:26.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16cc230640 1 --2- 172.21.15.37:0/3189888758 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16c4137900 0x7f16c409c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:26.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16cc230640 1 -- 172.21.15.37:0/3189888758 >> 172.21.15.37:0/3189888758 conn(0x7f16c40a64e0 msgr2=0x7f16c40a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:22:26.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16cc230640 1 -- 172.21.15.37:0/3189888758 shutdown_connections 2024-04-26T00:22:26.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16cc230640 1 -- 172.21.15.37:0/3189888758 wait complete. 2024-04-26T00:22:26.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16cc230640 10 monclient: build_initial_monmap 2024-04-26T00:22:26.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16cc230640 10 monclient: monmap: 2024-04-26T00:22:26.542 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:22:26.542 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:22:26.542 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:22:26.542 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:22:26.542 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:22:26.542 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:22:26.542 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:22:26.542 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:22:26.542 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:22:26.542 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:26.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16cc230640 1 Processor -- start 2024-04-26T00:22:26.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16cc230640 1 -- start start 2024-04-26T00:22:26.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16cc230640 10 monclient: init 2024-04-26T00:22:26.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16cc230640 10 monclient: _reopen_session rank -1 2024-04-26T00:22:26.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16cc230640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:22:26.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16cc230640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16c4136e10 0x7f16c411fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:26.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16cc230640 10 monclient(hunting): picked mon.noname-a con 0x7f16c4136e10 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:22:26.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16cc230640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f16c4137900 0x7f16c4139460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:26.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16cc230640 10 monclient(hunting): picked mon.noname-b con 0x7f16c4137900 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:22:26.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16cc230640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16c413c340 0x7f16c41361a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:26.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16cc230640 10 monclient(hunting): picked mon.noname-c con 0x7f16c413c340 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:22:26.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16cc230640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:26.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16cc230640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f16c4141820 con 0x7f16c4136e10 2024-04-26T00:22:26.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16cc230640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:26.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16cc230640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f16c41416a0 con 0x7f16c4137900 2024-04-26T00:22:26.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16cc230640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:26.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16cc230640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f16c41419a0 con 0x7f16c413c340 2024-04-26T00:22:26.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16cc230640 10 monclient(hunting): _renew_subs 2024-04-26T00:22:26.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16cc230640 10 monclient(hunting): authenticate will time out at 1127.862357s 2024-04-26T00:22:26.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16c9fcc640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16c4136e10 0x7f16c411fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:26.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16ca7cd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16c413c340 0x7f16c41361a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:26.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16c9fcc640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16c4136e10 0x7f16c411fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:55704/0 (socket says 172.21.15.37:55704) 2024-04-26T00:22:26.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16c9fcc640 1 -- 172.21.15.37:0/951863180 learned_addr learned my addr 172.21.15.37:0/951863180 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:22:26.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16c9fcc640 10 monclient(hunting): get_auth_request con 0x7f16c4136e10 auth_method 0 2024-04-26T00:22:26.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16c9fcc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:26.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16c9fcc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:26.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16c9fcc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:26.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16ca7cd640 10 monclient(hunting): get_auth_request con 0x7f16c413c340 auth_method 0 2024-04-26T00:22:26.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16ca7cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:26.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16ca7cd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:26.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16ca7cd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:26.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16c9fcc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:26.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16c9fcc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:26.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16c9fcc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:26.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16ca7cd640 1 -- 172.21.15.37:0/951863180 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16c413c340 msgr2=0x7f16c41361a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:22:26.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16ca7cd640 1 -- 172.21.15.37:0/951863180 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16c413c340 msgr2=0x7f16c41361a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:22:26.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16ca7cd640 1 --2- 172.21.15.37:0/951863180 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16c413c340 0x7f16c41361a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:22:26.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16ca7cd640 1 --2- 172.21.15.37:0/951863180 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16c413c340 0x7f16c41361a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:22:26.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16c9fcc640 10 monclient(hunting): handle_auth_done global_id 4623 payload 824 2024-04-26T00:22:26.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16c9fcc640 1 -- 172.21.15.37:0/951863180 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16c413c340 msgr2=0x7f16c41361a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:22:26.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16c9fcc640 1 --2- 172.21.15.37:0/951863180 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16c413c340 0x7f16c41361a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:26.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16c9fcc640 1 -- 172.21.15.37:0/951863180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f16c4137900 msgr2=0x7f16c4139460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:26.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16c9fcc640 1 --2- 172.21.15.37:0/951863180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f16c4137900 0x7f16c4139460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:26.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16c9fcc640 10 monclient: _finish_hunting 0 2024-04-26T00:22:26.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16c9fcc640 1 monclient: found mon.noname-a 2024-04-26T00:22:26.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16c9fcc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:22:26.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16c9fcc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:22:26.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.554+0000 7f16c9fcc640 1 -- 172.21.15.37:0/951863180 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16c40a84a0 con 0x7f16c4136e10 2024-04-26T00:22:26.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.558+0000 7f16c9fcc640 1 -- 172.21.15.37:0/951863180 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16c4136e10 msgr2=0x7f16c411fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:26.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.558+0000 7f16c9fcc640 1 --2- 172.21.15.37:0/951863180 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16c4136e10 0x7f16c411fc30 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f16c000c470 tx=0x7f16c000c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:26.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.558+0000 7f16ba7fc640 1 -- 172.21.15.37:0/951863180 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c0017070 con 0x7f16c4136e10 2024-04-26T00:22:26.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.558+0000 7f16ba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:26.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.558+0000 7f16ba7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:22:26.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.558+0000 7f16ba7fc640 10 monclient: dump: 2024-04-26T00:22:26.546 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:26.547 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:26.547 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:26.547 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:26.547 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:26.547 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:26.547 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:26.547 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:26.547 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:26.547 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:26.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.558+0000 7f16ba7fc640 10 monclient: _finish_auth 0 2024-04-26T00:22:26.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.558+0000 7f16ba7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:22:26.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.558+0000 7f16ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:22:26.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.558+0000 7f16ba7fc640 1 -- 172.21.15.37:0/951863180 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16c00070d0 con 0x7f16c4136e10 2024-04-26T00:22:26.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.558+0000 7f16cc230640 5 monclient: authenticate success, global_id 4623 2024-04-26T00:22:26.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.558+0000 7f16cc230640 10 monclient: _renew_subs 2024-04-26T00:22:26.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.558+0000 7f16cc230640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:22:26.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.558+0000 7f16cc230640 1 -- 172.21.15.37:0/951863180 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16c413a420 con 0x7f16c4136e10 2024-04-26T00:22:26.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.558+0000 7f16ba7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:22:26.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.558+0000 7f16ba7fc640 1 -- 172.21.15.37:0/951863180 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c00074c0 con 0x7f16c4136e10 2024-04-26T00:22:26.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.558+0000 7f16ba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:26.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.558+0000 7f16ba7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:22:26.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.558+0000 7f16ba7fc640 10 monclient: dump: 2024-04-26T00:22:26.548 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:26.548 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:26.548 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:26.548 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:26.548 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:26.548 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:26.548 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:26.549 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:26.549 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:26.549 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:26.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.558+0000 7f16bbfff640 4 set_mon_vals no callback set 2024-04-26T00:22:26.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.558+0000 7f16cc230640 10 monclient: _renew_subs 2024-04-26T00:22:26.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.558+0000 7f16cc230640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:22:26.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.558+0000 7f16cc230640 1 -- 172.21.15.37:0/951863180 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16c4120b50 con 0x7f16c4136e10 2024-04-26T00:22:26.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.558+0000 7f16cc230640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:26.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.558+0000 7f16cc230640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:26.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.558+0000 7f16cc230640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:22:26.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.558+0000 7f16cc230640 1 -- 172.21.15.37:0/951863180 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16c4075910 con 0x7f16c4136e10 2024-04-26T00:22:26.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.558+0000 7f16ba7fc640 1 -- 172.21.15.37:0/951863180 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f16c001a070 con 0x7f16c4136e10 2024-04-26T00:22:26.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.558+0000 7f16ba7fc640 1 --2- 172.21.15.37:0/951863180 >> v2:172.21.15.45:6812/2379918623 conn(0x7f16a8035ed0 0x7f16a8038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:26.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.558+0000 7f16ba7fc640 1 -- 172.21.15.37:0/951863180 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 3959+0+0 (secure 0 0 0) 0x7f16c00480a0 con 0x7f16c4136e10 2024-04-26T00:22:26.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.558+0000 7f16c97cb640 1 --2- 172.21.15.37:0/951863180 >> v2:172.21.15.45:6812/2379918623 conn(0x7f16a8035ed0 0x7f16a8038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:26.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.558+0000 7f16c97cb640 10 monclient: get_auth_request con 0x7f16a8035ed0 auth_method 0 2024-04-26T00:22:26.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.562+0000 7f16c97cb640 1 --2- 172.21.15.37:0/951863180 >> v2:172.21.15.45:6812/2379918623 conn(0x7f16a8035ed0 0x7f16a8038390 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f16b4006f10 tx=0x7f16b4006e20 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:26.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.562+0000 7f16ba7fc640 1 -- 172.21.15.37:0/951863180 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f16c0010300 con 0x7f16c4136e10 2024-04-26T00:22:26.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.562+0000 7f16ba7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:26.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.562+0000 7f16ba7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:22:26.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.786+0000 7f16cc230640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:22:26.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.786+0000 7f16cc230640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:22:26.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.786+0000 7f16cc230640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:22:26.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.786+0000 7f16cc230640 1 -- 172.21.15.37:0/951863180 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f16c4066f40 con 0x7f16c4136e10 2024-04-26T00:22:26.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.790+0000 7f16ba7fc640 1 -- 172.21.15.37:0/951863180 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f16c001b460 con 0x7f16c4136e10 2024-04-26T00:22:26.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.790+0000 7f16ba7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:22:26.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.790+0000 7f16ba7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:22:26.767 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":24,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:22:26.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.790+0000 7f16cc230640 1 -- 172.21.15.37:0/951863180 >> v2:172.21.15.45:6812/2379918623 conn(0x7f16a8035ed0 msgr2=0x7f16a8038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:26.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.790+0000 7f16cc230640 1 --2- 172.21.15.37:0/951863180 >> v2:172.21.15.45:6812/2379918623 conn(0x7f16a8035ed0 0x7f16a8038390 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f16b4006f10 tx=0x7f16b4006e20 comp rx=0 tx=0).stop 2024-04-26T00:22:26.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.790+0000 7f16cc230640 10 monclient: shutdown 2024-04-26T00:22:26.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.790+0000 7f16cc230640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:22:26.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.790+0000 7f16cc230640 1 -- 172.21.15.37:0/951863180 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16c4136e10 msgr2=0x7f16c411fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:26.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.790+0000 7f16cc230640 1 --2- 172.21.15.37:0/951863180 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16c4136e10 0x7f16c411fc30 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f16c000c470 tx=0x7f16c000c930 comp rx=0 tx=0).stop 2024-04-26T00:22:26.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.790+0000 7f16cc230640 1 -- 172.21.15.37:0/951863180 shutdown_connections 2024-04-26T00:22:26.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.790+0000 7f16cc230640 1 --2- 172.21.15.37:0/951863180 >> v2:172.21.15.45:6812/2379918623 conn(0x7f16a8035ed0 0x7f16a8038390 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:26.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.790+0000 7f16cc230640 1 --2- 172.21.15.37:0/951863180 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16c413c340 0x7f16c41361a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:26.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.790+0000 7f16cc230640 1 --2- 172.21.15.37:0/951863180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f16c4137900 0x7f16c4139460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:26.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.790+0000 7f16cc230640 1 --2- 172.21.15.37:0/951863180 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16c4136e10 0x7f16c411fc30 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:26.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.790+0000 7f16cc230640 1 -- 172.21.15.37:0/951863180 >> 172.21.15.37:0/951863180 conn(0x7f16c40a64e0 msgr2=0x7f16c413d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:22:26.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.790+0000 7f16cc230640 1 -- 172.21.15.37:0/951863180 shutdown_connections 2024-04-26T00:22:26.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.790+0000 7f16cc230640 1 -- 172.21.15.37:0/951863180 wait complete. 2024-04-26T00:22:26.786 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:22:26.786 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:22:26.786 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:22:26.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a86ef640 10 monclient: get_monmap_and_config 2024-04-26T00:22:26.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a86ef640 10 monclient: build_initial_monmap 2024-04-26T00:22:26.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a86ef640 10 monclient: monmap: 2024-04-26T00:22:26.905 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:22:26.905 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:26.905 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:22:26.929497+0000 2024-04-26T00:22:26.905 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:22:26.929497+0000 2024-04-26T00:22:26.906 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:22:26.906 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:26.906 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:22:26.906 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:22:26.906 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:22:26.907 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:22:26.907 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:22:26.907 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:26.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a86ef640 1 Processor -- start 2024-04-26T00:22:26.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a86ef640 1 -- start start 2024-04-26T00:22:26.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a86ef640 10 monclient: init 2024-04-26T00:22:26.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a86ef640 10 monclient: _reopen_session rank -1 2024-04-26T00:22:26.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a86ef640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-04-26T00:22:26.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a86ef640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86a00a0300 0x7f86a009fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:26.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a86ef640 10 monclient(hunting): picked mon.noname-b con 0x7f86a00a0300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:22:26.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a86ef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f86a00a06e0 addr v1:172.21.15.45:6789/0 2024-04-26T00:22:26.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a86ef640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f86a01352b0 0x7f86a013cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:26.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a86ef640 10 monclient(hunting): picked mon.noname-a con 0x7f86a01352b0 addr v2:172.21.15.37:3300/0 2024-04-26T00:22:26.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a86ef640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86a009c190 con 0x7f86a00a06e0 2024-04-26T00:22:26.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a86ef640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:26.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a86ef640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f86a00a13f0 con 0x7f86a01352b0 2024-04-26T00:22:26.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a86ef640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:26.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a86ef640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f86a00a1110 con 0x7f86a00a0300 2024-04-26T00:22:26.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a86ef640 10 monclient(hunting): _renew_subs 2024-04-26T00:22:26.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a86ef640 10 monclient(hunting): authenticate will time out at 1128.235397s 2024-04-26T00:22:26.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a648b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86a00a0300 0x7f86a009fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:26.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a648b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86a00a0300 0x7f86a009fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:55664/0 (socket says 172.21.15.37:55664) 2024-04-26T00:22:26.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a648b640 1 -- 172.21.15.37:0/2671923444 learned_addr learned my addr 172.21.15.37:0/2671923444 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:22:26.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a648b640 10 monclient(hunting): get_auth_request con 0x7f86a00a0300 auth_method 0 2024-04-26T00:22:26.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a648b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:26.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a648b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:26.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a648b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:26.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a6c8c640 1 --2- 172.21.15.37:0/2671923444 >> v2:172.21.15.37:3300/0 conn(0x7f86a01352b0 0x7f86a013cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:26.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a6c8c640 10 monclient(hunting): get_auth_request con 0x7f86a01352b0 auth_method 0 2024-04-26T00:22:26.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a6c8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:26.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a6c8c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:26.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a6c8c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:26.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a648b640 1 -- 172.21.15.37:0/2671923444 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86a00a0300 msgr2=0x7f86a009fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:22:26.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a648b640 1 -- 172.21.15.37:0/2671923444 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86a00a0300 msgr2=0x7f86a009fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:22:26.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a648b640 1 --2- 172.21.15.37:0/2671923444 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86a00a0300 0x7f86a009fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:22:26.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a6c8c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:26.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a6c8c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:26.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a6c8c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:26.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a648b640 1 --2- 172.21.15.37:0/2671923444 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86a00a0300 0x7f86a009fc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:22:26.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a6c8c640 10 monclient(hunting): handle_auth_done global_id 4626 payload 274 2024-04-26T00:22:26.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a6c8c640 1 -- 172.21.15.37:0/2671923444 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86a00a0300 msgr2=0x7f86a009fc50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:22:26.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a6c8c640 1 --2- 172.21.15.37:0/2671923444 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86a00a0300 0x7f86a009fc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:26.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a6c8c640 1 -- 172.21.15.37:0/2671923444 >> v1:172.21.15.45:6789/0 conn(0x7f86a00a06e0 legacy=0x7f86a0134ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:22:26.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a6c8c640 10 monclient: _finish_hunting 0 2024-04-26T00:22:26.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a6c8c640 1 monclient: found mon.noname-a 2024-04-26T00:22:26.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a6c8c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:22:26.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a6c8c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:22:26.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.926+0000 7f86a6c8c640 1 -- 172.21.15.37:0/2671923444 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86a00a16d0 con 0x7f86a01352b0 2024-04-26T00:22:26.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a6c8c640 1 -- 172.21.15.37:0/2671923444 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86a01352b0 msgr2=0x7f86a013cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:26.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a6c8c640 1 --2- 172.21.15.37:0/2671923444 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86a01352b0 0x7f86a013cee0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8698009d90 tx=0x7f869801c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e71b630ed19ecce2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:26.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a5489640 1 -- 172.21.15.37:0/2671923444 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86980b2070 con 0x7f86a01352b0 2024-04-26T00:22:26.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a5489640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:26.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a5489640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:22:26.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a5489640 10 monclient: dump: 2024-04-26T00:22:26.919 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:26.919 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:26.919 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:26.919 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:26.919 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:26.919 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:26.919 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:26.919 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:26.919 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:26.919 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:26.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a5489640 10 monclient: _finish_auth 0 2024-04-26T00:22:26.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a5489640 10 monclient: _check_auth_tickets 2024-04-26T00:22:26.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a5489640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:22:26.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a5489640 1 -- 172.21.15.37:0/2671923444 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86980a5850 con 0x7f86a01352b0 2024-04-26T00:22:26.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a86ef640 5 monclient: authenticate success, global_id 4626 2024-04-26T00:22:26.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a86ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:22:26.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a5489640 10 monclient: handle_config config(0 keys) 2024-04-26T00:22:26.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a86ef640 10 monclient: get_monmap_and_config success 2024-04-26T00:22:26.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a86ef640 4 set_mon_vals no callback set 2024-04-26T00:22:26.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a5489640 1 -- 172.21.15.37:0/2671923444 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86980a5c20 con 0x7f86a01352b0 2024-04-26T00:22:26.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a5489640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:26.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a5489640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:22:26.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a5489640 10 monclient: dump: 2024-04-26T00:22:26.920 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:26.920 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:26.921 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:26.921 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:26.921 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:26.921 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:26.921 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:26.921 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:26.921 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:26.921 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:26.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a86ef640 10 monclient: shutdown 2024-04-26T00:22:26.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a86ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:22:26.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a86ef640 1 -- 172.21.15.37:0/2671923444 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86a01352b0 msgr2=0x7f86a013cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:26.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a86ef640 1 --2- 172.21.15.37:0/2671923444 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86a01352b0 0x7f86a013cee0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8698009d90 tx=0x7f869801c7f0 comp rx=0 tx=0).stop 2024-04-26T00:22:26.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a86ef640 1 -- 172.21.15.37:0/2671923444 shutdown_connections 2024-04-26T00:22:26.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a86ef640 1 --2- 172.21.15.37:0/2671923444 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86a01352b0 0x7f86a013cee0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:26.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a86ef640 1 --2- 172.21.15.37:0/2671923444 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86a00a0300 0x7f86a009fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:26.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a86ef640 1 -- 172.21.15.37:0/2671923444 >> 172.21.15.37:0/2671923444 conn(0x7f86a00a1d50 msgr2=0x7f86a00a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:22:26.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a86ef640 1 -- 172.21.15.37:0/2671923444 shutdown_connections 2024-04-26T00:22:26.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a86ef640 1 -- 172.21.15.37:0/2671923444 wait complete. 2024-04-26T00:22:26.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a86ef640 10 monclient: build_initial_monmap 2024-04-26T00:22:26.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a86ef640 10 monclient: monmap: 2024-04-26T00:22:26.922 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:22:26.922 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:22:26.922 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:22:26.922 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:22:26.922 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:22:26.922 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:22:26.922 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:22:26.922 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:22:26.922 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:22:26.923 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:26.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a86ef640 1 Processor -- start 2024-04-26T00:22:26.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a86ef640 1 -- start start 2024-04-26T00:22:26.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a86ef640 10 monclient: init 2024-04-26T00:22:26.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a86ef640 10 monclient: _reopen_session rank -1 2024-04-26T00:22:26.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a86ef640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:22:26.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a86ef640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86a00a0300 0x7f86a012ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:26.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a86ef640 10 monclient(hunting): picked mon.noname-a con 0x7f86a00a0300 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:22:26.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a86ef640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86a00a06e0 0x7f86a0130470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:26.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a86ef640 10 monclient(hunting): picked mon.noname-c con 0x7f86a00a06e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:22:26.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a86ef640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86a01352b0 0x7f86a01cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:26.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a86ef640 10 monclient(hunting): picked mon.noname-b con 0x7f86a01352b0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:22:26.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a86ef640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:26.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a86ef640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f86a013fc40 con 0x7f86a00a0300 2024-04-26T00:22:26.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a86ef640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:26.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a86ef640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f86a013fdc0 con 0x7f86a01352b0 2024-04-26T00:22:26.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a86ef640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:26.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a86ef640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f86a013ff40 con 0x7f86a00a06e0 2024-04-26T00:22:26.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a86ef640 10 monclient(hunting): _renew_subs 2024-04-26T00:22:26.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a86ef640 10 monclient(hunting): authenticate will time out at 1128.242434s 2024-04-26T00:22:26.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a5c8a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86a00a06e0 0x7f86a0130470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:26.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a648b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86a00a0300 0x7f86a012ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:26.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a5c8a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86a00a06e0 0x7f86a0130470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:55680/0 (socket says 172.21.15.37:55680) 2024-04-26T00:22:26.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a5c8a640 1 -- 172.21.15.37:0/1539356996 learned_addr learned my addr 172.21.15.37:0/1539356996 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:22:26.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a5c8a640 10 monclient(hunting): get_auth_request con 0x7f86a00a06e0 auth_method 0 2024-04-26T00:22:26.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a5c8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:26.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a5c8a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:26.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a5c8a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:26.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a648b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86a00a0300 0x7f86a012ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:55726/0 (socket says 172.21.15.37:55726) 2024-04-26T00:22:26.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a648b640 10 monclient(hunting): get_auth_request con 0x7f86a00a0300 auth_method 0 2024-04-26T00:22:26.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a648b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:26.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a648b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:26.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a648b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:26.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a5c8a640 1 -- 172.21.15.37:0/1539356996 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86a00a06e0 msgr2=0x7f86a0130470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:22:26.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a5c8a640 1 -- 172.21.15.37:0/1539356996 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86a00a06e0 msgr2=0x7f86a0130470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:22:26.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a5c8a640 1 --2- 172.21.15.37:0/1539356996 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86a00a06e0 0x7f86a0130470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:22:26.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a5c8a640 1 --2- 172.21.15.37:0/1539356996 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86a00a06e0 0x7f86a0130470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:22:26.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a648b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:26.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a648b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:26.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a648b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:26.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a6c8c640 1 --2- 172.21.15.37:0/1539356996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86a01352b0 0x7f86a01cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:26.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a6c8c640 10 monclient(hunting): get_auth_request con 0x7f86a01352b0 auth_method 0 2024-04-26T00:22:26.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a6c8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:26.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a6c8c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:26.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a6c8c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:26.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a648b640 10 monclient(hunting): handle_auth_done global_id 4629 payload 824 2024-04-26T00:22:26.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a648b640 1 -- 172.21.15.37:0/1539356996 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86a00a06e0 msgr2=0x7f86a0130470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:22:26.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a648b640 1 --2- 172.21.15.37:0/1539356996 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86a00a06e0 0x7f86a0130470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:26.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a648b640 1 -- 172.21.15.37:0/1539356996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86a01352b0 msgr2=0x7f86a01cd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:26.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a648b640 1 --2- 172.21.15.37:0/1539356996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86a01352b0 0x7f86a01cd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:26.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a648b640 10 monclient: _finish_hunting 0 2024-04-26T00:22:26.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a648b640 1 monclient: found mon.noname-a 2024-04-26T00:22:26.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a648b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:22:26.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a648b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:22:26.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.934+0000 7f86a648b640 1 -- 172.21.15.37:0/1539356996 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86a013a1d0 con 0x7f86a00a0300 2024-04-26T00:22:26.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f86a648b640 1 -- 172.21.15.37:0/1539356996 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86a00a0300 msgr2=0x7f86a012ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:26.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f86a648b640 1 --2- 172.21.15.37:0/1539356996 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86a00a0300 0x7f86a012ff30 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f869c009a90 tx=0x7f869c009f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:26.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f8696ffd640 1 -- 172.21.15.37:0/1539356996 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f869c01d020 con 0x7f86a00a0300 2024-04-26T00:22:26.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f8696ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:26.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f8696ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:22:26.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f8696ffd640 10 monclient: dump: 2024-04-26T00:22:26.931 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:26.931 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:26.931 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:26.931 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:26.931 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:26.931 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:26.931 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:26.931 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:26.932 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:26.932 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:26.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f8696ffd640 10 monclient: _finish_auth 0 2024-04-26T00:22:26.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f8696ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:22:26.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f8696ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:22:26.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f8696ffd640 1 -- 172.21.15.37:0/1539356996 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f869c004020 con 0x7f86a00a0300 2024-04-26T00:22:26.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f8696ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:22:26.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f8696ffd640 1 -- 172.21.15.37:0/1539356996 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f869c004410 con 0x7f86a00a0300 2024-04-26T00:22:26.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f8696ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:26.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f8696ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:22:26.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f8696ffd640 10 monclient: dump: 2024-04-26T00:22:26.933 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:26.933 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:26.933 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:26.933 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:26.933 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:26.933 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:26.933 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:26.933 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:26.933 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:26.933 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:26.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f86a86ef640 5 monclient: authenticate success, global_id 4629 2024-04-26T00:22:26.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f86a86ef640 10 monclient: _renew_subs 2024-04-26T00:22:26.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f86a4c88640 4 set_mon_vals no callback set 2024-04-26T00:22:26.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f86a86ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:22:26.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f86a86ef640 1 -- 172.21.15.37:0/1539356996 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86a009f550 con 0x7f86a00a0300 2024-04-26T00:22:26.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f86a86ef640 10 monclient: _renew_subs 2024-04-26T00:22:26.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f86a86ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:22:26.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f86a86ef640 1 -- 172.21.15.37:0/1539356996 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86a013b680 con 0x7f86a00a0300 2024-04-26T00:22:26.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f86a86ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:26.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f86a86ef640 1 --2- 172.21.15.37:0/1539356996 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86a002c000 0x7f86a002c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:26.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f86a86ef640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:26.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f86a86ef640 1 -- 172.21.15.37:0/1539356996 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f86a00a0cd0 con 0x7f86a002c000 2024-04-26T00:22:26.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f86a5c8a640 1 --2- 172.21.15.37:0/1539356996 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86a002c000 0x7f86a002c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:26.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f86a5c8a640 10 monclient: get_auth_request con 0x7f86a002c000 auth_method 0 2024-04-26T00:22:26.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f86a5c8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:26.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f86a5c8a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:26.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f86a5c8a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:26.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f8696ffd640 1 -- 172.21.15.37:0/1539356996 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f869c012900 con 0x7f86a00a0300 2024-04-26T00:22:26.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f86a5c8a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:26.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f86a5c8a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:26.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f86a5c8a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:26.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f8696ffd640 1 --2- 172.21.15.37:0/1539356996 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8684035ed0 0x7f8684038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:26.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f86a5c8a640 10 monclient(hunting): handle_auth_done global_id 4632 payload 824 2024-04-26T00:22:26.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f86a5c8a640 1 -- 172.21.15.37:0/1539356996 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f86a009e760 con 0x7f86a002c000 2024-04-26T00:22:26.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f8696ffd640 1 -- 172.21.15.37:0/1539356996 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 3959+0+0 (secure 0 0 0) 0x7f869c04caa0 con 0x7f86a00a0300 2024-04-26T00:22:26.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f86a6c8c640 1 --2- 172.21.15.37:0/1539356996 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8684035ed0 0x7f8684038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:26.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.938+0000 7f86a6c8c640 10 monclient: get_auth_request con 0x7f8684035ed0 auth_method 0 2024-04-26T00:22:26.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.942+0000 7f86a5c8a640 1 -- 172.21.15.37:0/1539356996 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86a002c000 msgr2=0x7f86a002c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:26.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.942+0000 7f86a5c8a640 1 --2- 172.21.15.37:0/1539356996 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86a002c000 0x7f86a002c3e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f86a0139270 tx=0x7f869000e980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:26.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.942+0000 7f8696ffd640 1 -- 172.21.15.37:0/1539356996 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8690007450 con 0x7f86a002c000 2024-04-26T00:22:26.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.942+0000 7f8696ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:26.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.942+0000 7f8696ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:22:26.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.942+0000 7f8696ffd640 10 monclient: dump: 2024-04-26T00:22:26.936 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:26.936 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:26.936 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:26.936 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:26.936 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:26.936 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:26.936 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:26.936 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:26.936 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:26.937 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:26.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.942+0000 7f8696ffd640 1 -- 172.21.15.37:0/1539356996 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8690007450 con 0x7f86a002c000 2024-04-26T00:22:26.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.942+0000 7f8696ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:26.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.942+0000 7f8696ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:22:26.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.942+0000 7f8696ffd640 1 -- 172.21.15.37:0/1539356996 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86a002c000 msgr2=0x7f86a002c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:26.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.942+0000 7f8696ffd640 1 --2- 172.21.15.37:0/1539356996 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86a002c000 0x7f86a002c3e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f86a0139270 tx=0x7f869000e980 comp rx=0 tx=0).stop 2024-04-26T00:22:26.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.942+0000 7f8696ffd640 1 -- 172.21.15.37:0/1539356996 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86a002c000 msgr2=0x7f86a002c3e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:22:26.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.942+0000 7f8696ffd640 1 --2- 172.21.15.37:0/1539356996 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86a002c000 0x7f86a002c3e0 secure :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f86a0139270 tx=0x7f869000e980 comp rx=0 tx=0).stop 2024-04-26T00:22:26.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.942+0000 7f86a6c8c640 1 --2- 172.21.15.37:0/1539356996 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8684035ed0 0x7f8684038390 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f86a006d330 tx=0x7f86980a7070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:26.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.946+0000 7f86a86ef640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:22:26.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.946+0000 7f86a86ef640 1 --2- 172.21.15.37:0/1539356996 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86a002c920 0x7f86a002cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:26.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.946+0000 7f86a86ef640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:26.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.946+0000 7f86a86ef640 1 -- 172.21.15.37:0/1539356996 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f86a009eb60 con 0x7f86a002c920 2024-04-26T00:22:26.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.946+0000 7f86a5c8a640 1 --2- 172.21.15.37:0/1539356996 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86a002c920 0x7f86a002cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:26.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.946+0000 7f86a5c8a640 10 monclient: get_auth_request con 0x7f86a002c920 auth_method 0 2024-04-26T00:22:26.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.946+0000 7f86a5c8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:26.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.946+0000 7f86a5c8a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:26.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.946+0000 7f86a5c8a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:26.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.946+0000 7f86a5c8a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:26.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.946+0000 7f86a5c8a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:26.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.946+0000 7f86a5c8a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:26.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.946+0000 7f86a5c8a640 10 monclient(hunting): handle_auth_done global_id 4635 payload 824 2024-04-26T00:22:26.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.946+0000 7f86a5c8a640 1 -- 172.21.15.37:0/1539356996 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f86a0133a30 con 0x7f86a002c920 2024-04-26T00:22:26.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.950+0000 7f86a5c8a640 1 -- 172.21.15.37:0/1539356996 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86a002c920 msgr2=0x7f86a002cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:26.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.950+0000 7f86a5c8a640 1 --2- 172.21.15.37:0/1539356996 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86a002c920 0x7f86a002cd30 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8690018000 tx=0x7f8690003770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:26.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.950+0000 7f8696ffd640 1 -- 172.21.15.37:0/1539356996 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f869001eb70 con 0x7f86a002c920 2024-04-26T00:22:26.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.950+0000 7f8696ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:26.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.950+0000 7f8696ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:22:26.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.950+0000 7f8696ffd640 10 monclient: dump: 2024-04-26T00:22:26.939 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:26.939 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:26.939 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:26.939 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:26.939 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:26.939 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:26.939 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:26.939 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:26.939 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:26.939 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:26.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.950+0000 7f8696ffd640 1 -- 172.21.15.37:0/1539356996 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4719 (secure 0 0 0) 0x7f869001eb70 con 0x7f86a002c920 2024-04-26T00:22:26.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.950+0000 7f8696ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:22:26.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.950+0000 7f8696ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:22:26.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.950+0000 7f8696ffd640 1 -- 172.21.15.37:0/1539356996 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86a002c920 msgr2=0x7f86a002cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:26.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.950+0000 7f8696ffd640 1 --2- 172.21.15.37:0/1539356996 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86a002c920 0x7f86a002cd30 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8690018000 tx=0x7f8690003770 comp rx=0 tx=0).stop 2024-04-26T00:22:26.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.950+0000 7f8696ffd640 1 -- 172.21.15.37:0/1539356996 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86a002c920 msgr2=0x7f86a002cd30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:22:26.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.950+0000 7f8696ffd640 1 --2- 172.21.15.37:0/1539356996 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86a002c920 0x7f86a002cd30 secure :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8690018000 tx=0x7f8690003770 comp rx=0 tx=0).stop 2024-04-26T00:22:26.940 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:22:26.940 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:22:26.940 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:22:26.940 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:22:26.940 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 24, 2024-04-26T00:22:26.940 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 164054, 2024-04-26T00:22:26.941 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:22:26.941 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:22:26.941 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:22:26.941 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:22:26.941 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:22:26.941 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 1, 2024-04-26T00:22:26.941 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:22:26.941 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:22:26.941 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:22:26.941 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:22:26.941 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:22:26.941 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:22:26.941 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:22:26.941 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:22:26.941 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:22:26.942 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:22:26.942 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:22:26.942 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:22:26.942 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:22:26.942 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:22:26.942 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:22:26.942 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:22:26.942 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:22:26.942 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:22:26.942 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:22:26.942 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:22:26.942 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:22:26.942 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:22:26.942 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:22:26.943 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:22:26.943 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:22:26.943 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:22:26.943 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:22:26.943 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:22:26.943 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:26.943 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:26.943 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:22:26.943 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:22:26.943 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:22:26.943 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:22:26.943 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:22:26.943 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:22:26.943 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:22:26.944 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:22:26.944 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:22:26.944 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:22:26.944 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:22:26.944 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:22:26.944 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:22:26.944 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:22:26.944 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:22:26.944 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:22:26.944 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:22:26.944 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:22:26.944 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:22:26.944 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:22:26.944 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:22:26.945 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:22:26.945 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:22:26.945 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:22:26.945 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:22:26.945 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:22:26.945 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:22:26.945 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:22:26.945 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:22:26.945 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:22:26.945 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:26.945 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:22:26.945 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:26.945 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:22:26.945 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:22:26.946 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:22:26.946 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:22:26.946 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:26.946 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:22:26.946 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:22:26.946 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:26.946 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:26.946 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:26.946 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:22:26.946 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:22:26.946 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:26.946 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:26.946 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:26.946 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:26.947 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:22:26.947 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:22:26.947 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:22:26.947 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:22:26.947 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:22:26.947 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:26.947 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:26.947 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:22:26.947 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:22:26.947 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:22:26.947 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:22:26.947 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:26.947 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:22:26.947 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:22:26.947 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:26.948 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:26.948 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:26.948 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:22:26.948 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:22:26.948 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:26.948 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:26.948 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:26.948 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:26.948 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:22:26.948 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:22:26.948 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:22:26.948 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:22:26.948 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:22:26.948 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:26.949 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:26.949 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:22:26.949 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:22:26.949 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:22:26.949 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:22:26.949 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:26.949 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:22:26.949 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:22:26.949 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:26.949 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:26.949 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:26.949 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:22:26.949 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:22:26.949 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:26.950 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:26.950 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:26.950 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:26.950 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:22:26.950 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:22:26.950 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:22:26.950 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:22:26.950 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:22:26.950 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:26.950 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:26.950 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:26.950 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:22:26.950 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:22:26.950 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:26.951 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:22:26.951 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:22:26.951 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:22:26.951 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:26.951 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:22:26.951 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:22:26.951 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:26.951 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:22:26.951 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:22:26.951 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:22:26.951 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:26.951 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:22:26.951 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:22:26.951 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:26.952 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:22:26.952 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:22:26.952 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:22:26.952 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:26.952 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:26.952 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:26.952 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:22:26.952 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:22:26.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.950+0000 7f86a86ef640 1 -- 172.21.15.37:0/1539356996 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8684035ed0 msgr2=0x7f8684038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:26.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.950+0000 7f86a86ef640 1 --2- 172.21.15.37:0/1539356996 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8684035ed0 0x7f8684038390 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f86a006d330 tx=0x7f86980a7070 comp rx=0 tx=0).stop 2024-04-26T00:22:26.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.950+0000 7f86a86ef640 10 monclient: shutdown 2024-04-26T00:22:26.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.950+0000 7f86a86ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:22:26.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.950+0000 7f86a86ef640 1 -- 172.21.15.37:0/1539356996 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86a00a0300 msgr2=0x7f86a012ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:26.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.950+0000 7f86a86ef640 1 --2- 172.21.15.37:0/1539356996 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86a00a0300 0x7f86a012ff30 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f869c009a90 tx=0x7f869c009f50 comp rx=0 tx=0).stop 2024-04-26T00:22:26.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.950+0000 7f86a6c8c640 1 -- 172.21.15.37:0/1539356996 reap_dead start 2024-04-26T00:22:26.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.950+0000 7f86a6c8c640 1 -- 172.21.15.37:0/1539356996 reap_dead start 2024-04-26T00:22:26.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.950+0000 7f86a86ef640 1 -- 172.21.15.37:0/1539356996 shutdown_connections 2024-04-26T00:22:26.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.950+0000 7f86a86ef640 1 -- 172.21.15.37:0/1539356996 >> 172.21.15.37:0/1539356996 conn(0x7f86a00a1d50 msgr2=0x7f86a0130c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:22:26.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.950+0000 7f86a86ef640 1 -- 172.21.15.37:0/1539356996 shutdown_connections 2024-04-26T00:22:26.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:26.950+0000 7f86a86ef640 1 -- 172.21.15.37:0/1539356996 wait complete. 2024-04-26T00:22:26.954 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:22:26.955 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:22:26.955 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:22:26.971 INFO:tasks.rados.rados.0.smithi045.stdout:432: done (5 left) 2024-04-26T00:22:26.971 INFO:tasks.rados.rados.0.smithi045.stdout:433: snap_remove snap 48 2024-04-26T00:22:27.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62f2300640 10 monclient: get_monmap_and_config 2024-04-26T00:22:27.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62f2300640 10 monclient: build_initial_monmap 2024-04-26T00:22:27.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62f2300640 10 monclient: monmap: 2024-04-26T00:22:27.071 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:22:27.071 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:27.071 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:22:27.095982+0000 2024-04-26T00:22:27.071 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:22:27.095982+0000 2024-04-26T00:22:27.071 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:22:27.071 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:27.071 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:22:27.072 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:22:27.072 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:22:27.072 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:22:27.072 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:22:27.072 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:27.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62f2300640 1 Processor -- start 2024-04-26T00:22:27.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62f2300640 1 -- start start 2024-04-26T00:22:27.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62f2300640 10 monclient: init 2024-04-26T00:22:27.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62f2300640 10 monclient: _reopen_session rank -1 2024-04-26T00:22:27.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62f2300640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-04-26T00:22:27.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62f2300640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f62ec0a3d90 0x7f62ec09bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:27.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62f2300640 10 monclient(hunting): picked mon.noname-a con 0x7f62ec0a3d90 addr v2:172.21.15.37:3300/0 2024-04-26T00:22:27.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62f2300640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f62ec0a4170 0x7f62ec0a45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:27.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62f2300640 10 monclient(hunting): picked mon.noname-c con 0x7f62ec0a4170 addr v2:172.21.15.45:3300/0 2024-04-26T00:22:27.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62f2300640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec133b20 0x7f62ec133f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:27.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62f2300640 10 monclient(hunting): picked mon.noname-b con 0x7f62ec133b20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:22:27.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62f2300640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:27.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62f2300640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f62ec0a5150 con 0x7f62ec0a3d90 2024-04-26T00:22:27.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62f2300640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:27.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62f2300640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f62ec09b280 con 0x7f62ec0a4170 2024-04-26T00:22:27.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62f2300640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:27.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62f2300640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f62ec09b560 con 0x7f62ec133b20 2024-04-26T00:22:27.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62f2300640 10 monclient(hunting): _renew_subs 2024-04-26T00:22:27.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62f2300640 10 monclient(hunting): authenticate will time out at 1128.401595s 2024-04-26T00:22:27.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62ebfff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f62ec0a3d90 0x7f62ec09bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:27.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62f089d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec133b20 0x7f62ec133f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:27.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62ebfff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f62ec0a3d90 0x7f62ec09bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:55756/0 (socket says 172.21.15.37:55756) 2024-04-26T00:22:27.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62ebfff640 1 -- 172.21.15.37:0/3092834794 learned_addr learned my addr 172.21.15.37:0/3092834794 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:22:27.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62ebfff640 10 monclient(hunting): get_auth_request con 0x7f62ec0a3d90 auth_method 0 2024-04-26T00:22:27.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:27.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62ebfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:27.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:27.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62f089d640 10 monclient(hunting): get_auth_request con 0x7f62ec133b20 auth_method 0 2024-04-26T00:22:27.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62f089d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:27.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62f089d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:27.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62f089d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:27.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:27.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:27.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:27.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62f089d640 1 -- 172.21.15.37:0/3092834794 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec133b20 msgr2=0x7f62ec133f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:22:27.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62f089d640 1 -- 172.21.15.37:0/3092834794 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec133b20 msgr2=0x7f62ec133f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:22:27.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62f089d640 1 --2- 172.21.15.37:0/3092834794 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec133b20 0x7f62ec133f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:22:27.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62f089d640 1 --2- 172.21.15.37:0/3092834794 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec133b20 0x7f62ec133f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:22:27.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62eb7fe640 1 --2- 172.21.15.37:0/3092834794 >> v2:172.21.15.45:3300/0 conn(0x7f62ec0a4170 0x7f62ec0a45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:27.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62eb7fe640 10 monclient(hunting): get_auth_request con 0x7f62ec0a4170 auth_method 0 2024-04-26T00:22:27.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:27.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62eb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:27.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:27.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62ebfff640 10 monclient(hunting): handle_auth_done global_id 4638 payload 274 2024-04-26T00:22:27.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62ebfff640 1 -- 172.21.15.37:0/3092834794 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec133b20 msgr2=0x7f62ec133f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:22:27.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62ebfff640 1 --2- 172.21.15.37:0/3092834794 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec133b20 0x7f62ec133f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:27.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62ebfff640 1 -- 172.21.15.37:0/3092834794 >> v2:172.21.15.45:3300/0 conn(0x7f62ec0a4170 msgr2=0x7f62ec0a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:22:27.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62ebfff640 1 --2- 172.21.15.37:0/3092834794 >> v2:172.21.15.45:3300/0 conn(0x7f62ec0a4170 0x7f62ec0a45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:27.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62ebfff640 10 monclient: _finish_hunting 0 2024-04-26T00:22:27.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62ebfff640 1 monclient: found mon.noname-a 2024-04-26T00:22:27.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:22:27.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62ebfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:22:27.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.094+0000 7f62ebfff640 1 -- 172.21.15.37:0/3092834794 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62ec0a4b20 con 0x7f62ec0a3d90 2024-04-26T00:22:27.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.098+0000 7f62ebfff640 1 -- 172.21.15.37:0/3092834794 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f62ec0a3d90 msgr2=0x7f62ec09bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:27.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.098+0000 7f62ebfff640 1 --2- 172.21.15.37:0/3092834794 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f62ec0a3d90 0x7f62ec09bf40 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f62ec09d140 tx=0x7f62d401c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47acf24337f7a07c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:27.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.098+0000 7f62eaffd640 1 -- 172.21.15.37:0/3092834794 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d40a5640 con 0x7f62ec0a3d90 2024-04-26T00:22:27.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.098+0000 7f62eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:27.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.098+0000 7f62eaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:22:27.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.098+0000 7f62eaffd640 10 monclient: dump: 2024-04-26T00:22:27.086 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:27.086 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:27.086 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:27.086 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:27.086 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:27.086 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:27.086 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:27.086 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:27.086 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:27.086 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:27.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.098+0000 7f62eaffd640 10 monclient: _finish_auth 0 2024-04-26T00:22:27.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.098+0000 7f62eaffd640 10 monclient: _check_auth_tickets 2024-04-26T00:22:27.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.098+0000 7f62eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:22:27.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.098+0000 7f62f2300640 5 monclient: authenticate success, global_id 4638 2024-04-26T00:22:27.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.098+0000 7f62f2300640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:22:27.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.098+0000 7f62eaffd640 1 -- 172.21.15.37:0/3092834794 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62d40a57e0 con 0x7f62ec0a3d90 2024-04-26T00:22:27.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.098+0000 7f62eaffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:22:27.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.098+0000 7f62f2300640 10 monclient: get_monmap_and_config success 2024-04-26T00:22:27.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.098+0000 7f62f2300640 4 set_mon_vals no callback set 2024-04-26T00:22:27.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.098+0000 7f62eaffd640 1 -- 172.21.15.37:0/3092834794 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d40a5bb0 con 0x7f62ec0a3d90 2024-04-26T00:22:27.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.098+0000 7f62eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:27.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.098+0000 7f62eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:22:27.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.098+0000 7f62eaffd640 10 monclient: dump: 2024-04-26T00:22:27.087 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:27.088 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:27.088 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:27.088 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:27.088 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:27.088 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:27.088 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:27.088 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:27.088 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:27.088 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:27.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.098+0000 7f62f2300640 10 monclient: shutdown 2024-04-26T00:22:27.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.098+0000 7f62f2300640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:22:27.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.098+0000 7f62f2300640 1 -- 172.21.15.37:0/3092834794 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f62ec0a3d90 msgr2=0x7f62ec09bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:27.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.098+0000 7f62f2300640 1 --2- 172.21.15.37:0/3092834794 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f62ec0a3d90 0x7f62ec09bf40 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f62ec09d140 tx=0x7f62d401c8b0 comp rx=0 tx=0).stop 2024-04-26T00:22:27.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.098+0000 7f62f2300640 1 -- 172.21.15.37:0/3092834794 shutdown_connections 2024-04-26T00:22:27.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.098+0000 7f62f2300640 1 --2- 172.21.15.37:0/3092834794 >> v2:172.21.15.45:3300/0 conn(0x7f62ec0a4170 0x7f62ec0a45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:27.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.098+0000 7f62f2300640 1 --2- 172.21.15.37:0/3092834794 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec133b20 0x7f62ec133f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:27.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.098+0000 7f62f2300640 1 --2- 172.21.15.37:0/3092834794 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f62ec0a3d90 0x7f62ec09bf40 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:27.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.098+0000 7f62f2300640 1 -- 172.21.15.37:0/3092834794 >> 172.21.15.37:0/3092834794 conn(0x7f62ec0aa0a0 msgr2=0x7f62ec0a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:22:27.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.098+0000 7f62f2300640 1 -- 172.21.15.37:0/3092834794 shutdown_connections 2024-04-26T00:22:27.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.098+0000 7f62f2300640 1 -- 172.21.15.37:0/3092834794 wait complete. 2024-04-26T00:22:27.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62f2300640 10 monclient: build_initial_monmap 2024-04-26T00:22:27.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62f2300640 10 monclient: monmap: 2024-04-26T00:22:27.089 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:22:27.089 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:22:27.089 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:22:27.089 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:22:27.089 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:22:27.089 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:22:27.089 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:22:27.090 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:22:27.090 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:22:27.090 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:27.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62f2300640 1 Processor -- start 2024-04-26T00:22:27.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62f2300640 1 -- start start 2024-04-26T00:22:27.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62f2300640 10 monclient: init 2024-04-26T00:22:27.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62f2300640 10 monclient: _reopen_session rank -1 2024-04-26T00:22:27.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62f2300640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:22:27.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62f2300640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62ec0a3d90 0x7f62ec132e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:27.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62f2300640 10 monclient(hunting): picked mon.noname-b con 0x7f62ec0a3d90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:22:27.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62f2300640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f62ec0a4170 0x7f62ec1333a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:27.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62f2300640 10 monclient(hunting): picked mon.noname-a con 0x7f62ec0a4170 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:22:27.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62f2300640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec133b20 0x7f62ec1cf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:27.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62f2300640 10 monclient(hunting): picked mon.noname-c con 0x7f62ec133b20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:22:27.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62f2300640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:27.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62f2300640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec140c10 con 0x7f62ec0a4170 2024-04-26T00:22:27.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62f2300640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:27.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62f2300640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec09c690 con 0x7f62ec0a3d90 2024-04-26T00:22:27.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62f2300640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:27.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62f2300640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f62ec140d90 con 0x7f62ec133b20 2024-04-26T00:22:27.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62f2300640 10 monclient(hunting): _renew_subs 2024-04-26T00:22:27.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62f2300640 10 monclient(hunting): authenticate will time out at 1128.409240s 2024-04-26T00:22:27.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62eb7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f62ec0a4170 0x7f62ec1333a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:27.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62eb7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f62ec0a4170 0x7f62ec1333a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:55766/0 (socket says 172.21.15.37:55766) 2024-04-26T00:22:27.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62eb7fe640 1 -- 172.21.15.37:0/4185693357 learned_addr learned my addr 172.21.15.37:0/4185693357 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:22:27.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62eb7fe640 10 monclient(hunting): get_auth_request con 0x7f62ec0a4170 auth_method 0 2024-04-26T00:22:27.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:27.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62eb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:27.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:27.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62f089d640 1 --2- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec133b20 0x7f62ec1cf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:27.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62f089d640 10 monclient(hunting): get_auth_request con 0x7f62ec133b20 auth_method 0 2024-04-26T00:22:27.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62f089d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:27.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62f089d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:27.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62f089d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:27.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:27.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:27.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:27.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62f089d640 1 -- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec133b20 msgr2=0x7f62ec1cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:22:27.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62ebfff640 1 --2- 172.21.15.37:0/4185693357 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62ec0a3d90 0x7f62ec132e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:27.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62f089d640 1 -- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec133b20 msgr2=0x7f62ec1cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:22:27.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62f089d640 1 --2- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec133b20 0x7f62ec1cf1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:22:27.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62f089d640 1 --2- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec133b20 0x7f62ec1cf1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:22:27.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62ebfff640 10 monclient(hunting): get_auth_request con 0x7f62ec0a3d90 auth_method 0 2024-04-26T00:22:27.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:27.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62ebfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:27.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:27.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62eb7fe640 10 monclient(hunting): handle_auth_done global_id 4641 payload 824 2024-04-26T00:22:27.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62eb7fe640 1 -- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec133b20 msgr2=0x7f62ec1cf1a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:22:27.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62eb7fe640 1 --2- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec133b20 0x7f62ec1cf1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:27.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62eb7fe640 1 -- 172.21.15.37:0/4185693357 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62ec0a3d90 msgr2=0x7f62ec132e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:27.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62eb7fe640 1 --2- 172.21.15.37:0/4185693357 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62ec0a3d90 0x7f62ec132e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:27.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62eb7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:22:27.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62eb7fe640 1 monclient: found mon.noname-a 2024-04-26T00:22:27.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:22:27.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62eb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:22:27.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.102+0000 7f62eb7fe640 1 -- 172.21.15.37:0/4185693357 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62ec13cb70 con 0x7f62ec0a4170 2024-04-26T00:22:27.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62eb7fe640 1 -- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f62ec0a4170 msgr2=0x7f62ec1333a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:27.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62eb7fe640 1 --2- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f62ec0a4170 0x7f62ec1333a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f62e400d800 tx=0x7f62e400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:27.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62e8ff9640 1 -- 172.21.15.37:0/4185693357 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e400b8d0 con 0x7f62ec0a4170 2024-04-26T00:22:27.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62e8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:27.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62e8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:22:27.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62e8ff9640 10 monclient: dump: 2024-04-26T00:22:27.094 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:27.094 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:27.094 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:27.094 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:27.094 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:27.094 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:27.094 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:27.095 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:27.095 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:27.095 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:27.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62e8ff9640 10 monclient: _finish_auth 0 2024-04-26T00:22:27.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62e8ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:22:27.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62e8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:22:27.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62e8ff9640 1 -- 172.21.15.37:0/4185693357 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62e4019070 con 0x7f62ec0a4170 2024-04-26T00:22:27.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62e8ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:22:27.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62e8ff9640 1 -- 172.21.15.37:0/4185693357 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e400bc70 con 0x7f62ec0a4170 2024-04-26T00:22:27.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62e8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:27.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62e8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:22:27.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62e8ff9640 10 monclient: dump: 2024-04-26T00:22:27.095 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:27.095 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:27.095 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:27.096 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:27.096 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:27.096 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:27.096 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:27.096 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:27.096 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:27.096 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:27.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62ea7fc640 4 set_mon_vals no callback set 2024-04-26T00:22:27.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62f2300640 5 monclient: authenticate success, global_id 4641 2024-04-26T00:22:27.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62f2300640 10 monclient: _renew_subs 2024-04-26T00:22:27.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62f2300640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:22:27.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62f2300640 1 -- 172.21.15.37:0/4185693357 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62ec09ada0 con 0x7f62ec0a4170 2024-04-26T00:22:27.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62f2300640 10 monclient: _renew_subs 2024-04-26T00:22:27.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62f2300640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:22:27.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62f2300640 1 -- 172.21.15.37:0/4185693357 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62ec13a880 con 0x7f62ec0a4170 2024-04-26T00:22:27.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62f2300640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:27.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62f2300640 1 --2- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec13c790 0x7f62ec1cf6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:27.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62f2300640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:27.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62f2300640 1 -- 172.21.15.37:0/4185693357 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f62ec13b210 con 0x7f62ec13c790 2024-04-26T00:22:27.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62ebfff640 1 --2- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec13c790 0x7f62ec1cf6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:27.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62ebfff640 10 monclient: get_auth_request con 0x7f62ec13c790 auth_method 0 2024-04-26T00:22:27.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:27.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62ebfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:27.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:27.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62ebfff640 1 -- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec13c790 msgr2=0x7f62ec1cf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:22:27.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62ebfff640 1 -- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec13c790 msgr2=0x7f62ec1cf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:22:27.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62ebfff640 1 --2- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec13c790 0x7f62ec1cf6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:22:27.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62ebfff640 1 --2- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec13c790 0x7f62ec1cf6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:22:27.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62e8ff9640 1 -- 172.21.15.37:0/4185693357 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f62e401e460 con 0x7f62ec0a4170 2024-04-26T00:22:27.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62e8ff9640 1 --2- 172.21.15.37:0/4185693357 >> v2:172.21.15.45:6812/2379918623 conn(0x7f62d0035eb0 0x7f62d0038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:27.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62e8ff9640 1 -- 172.21.15.37:0/4185693357 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 3975+0+0 (secure 0 0 0) 0x7f62e4048ff0 con 0x7f62ec0a4170 2024-04-26T00:22:27.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62f089d640 1 --2- 172.21.15.37:0/4185693357 >> v2:172.21.15.45:6812/2379918623 conn(0x7f62d0035eb0 0x7f62d0038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:27.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.106+0000 7f62f089d640 10 monclient: get_auth_request con 0x7f62d0035eb0 auth_method 0 2024-04-26T00:22:27.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.110+0000 7f62f089d640 1 --2- 172.21.15.37:0/4185693357 >> v2:172.21.15.45:6812/2379918623 conn(0x7f62d0035eb0 0x7f62d0038370 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f62ec13b5b0 tx=0x7f62dc0092e0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:27.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.310+0000 7f62ebfff640 1 --2- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec13c790 0x7f62ec1cf6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:27.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.310+0000 7f62ebfff640 10 monclient: get_auth_request con 0x7f62ec13c790 auth_method 0 2024-04-26T00:22:27.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.310+0000 7f62ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:27.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.310+0000 7f62ebfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:27.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.310+0000 7f62ebfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:22:27.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.310+0000 7f62ebfff640 1 -- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec13c790 msgr2=0x7f62ec1cf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:22:27.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.310+0000 7f62ebfff640 1 -- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec13c790 msgr2=0x7f62ec1cf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:22:27.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.310+0000 7f62ebfff640 1 --2- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec13c790 0x7f62ec1cf6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:22:27.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.310+0000 7f62ebfff640 1 --2- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec13c790 0x7f62ec1cf6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:22:27.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.714+0000 7f62ebfff640 1 --2- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec13c790 0x7f62ec1cf6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:27.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.714+0000 7f62ebfff640 10 monclient: get_auth_request con 0x7f62ec13c790 auth_method 0 2024-04-26T00:22:27.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.714+0000 7f62ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:27.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.714+0000 7f62ebfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:27.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.714+0000 7f62ebfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:22:27.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.714+0000 7f62ebfff640 1 -- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec13c790 msgr2=0x7f62ec1cf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:22:27.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.714+0000 7f62ebfff640 1 -- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec13c790 msgr2=0x7f62ec1cf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:22:27.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.714+0000 7f62ebfff640 1 --2- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec13c790 0x7f62ec1cf6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:22:27.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:27.714+0000 7f62ebfff640 1 --2- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec13c790 0x7f62ec1cf6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:22:27.924 INFO:tasks.rados.rados.0.smithi045.stdout:433: done (5 left) 2024-04-26T00:22:27.924 INFO:tasks.rados.rados.0.smithi045.stdout:434: read oid 20 snap 45 2024-04-26T00:22:27.924 INFO:tasks.rados.rados.0.smithi045.stdout:434: expect (ObjNum 133 snap 30 seq_num 133) 2024-04-26T00:22:27.924 INFO:tasks.rados.rados.0.smithi045.stdout:435: read oid 11 snap -1 2024-04-26T00:22:27.924 INFO:tasks.rados.rados.0.smithi045.stdout:435: expect (ObjNum 155 snap 33 seq_num 155) 2024-04-26T00:22:27.924 INFO:tasks.rados.rados.0.smithi045.stdout:436: write oid 10 current snap is 49 2024-04-26T00:22:27.924 INFO:tasks.rados.rados.0.smithi045.stdout:436: seq_num 183 ranges {675681=563039,1858793=746713,3025937=49744} 2024-04-26T00:22:27.949 INFO:tasks.rados.rados.0.smithi045.stdout:436: writing smithi04526579-10 from 675681 to 1238720 tid 1 2024-04-26T00:22:27.972 INFO:tasks.rados.rados.0.smithi045.stdout:436: writing smithi04526579-10 from 1858793 to 2605506 tid 2 2024-04-26T00:22:27.979 INFO:tasks.rados.rados.0.smithi045.stdout:436: writing smithi04526579-10 from 3025937 to 3075681 tid 3 2024-04-26T00:22:27.979 INFO:tasks.rados.rados.0.smithi045.stdout:437: write oid 47 current snap is 49 2024-04-26T00:22:27.979 INFO:tasks.rados.rados.0.smithi045.stdout:426: finishing write tid 1 to smithi04526579-3 2024-04-26T00:22:27.979 INFO:tasks.rados.rados.0.smithi045.stdout:430: expect (ObjNum 100 snap 24 seq_num 100) 2024-04-26T00:22:28.144 INFO:tasks.rados.rados.0.smithi045.stdout:426: finishing write tid 2 to smithi04526579-3 2024-04-26T00:22:28.144 INFO:tasks.rados.rados.0.smithi045.stdout:429: expect (ObjNum 173 snap 45 seq_num 173) 2024-04-26T00:22:28.277 INFO:tasks.rados.rados.0.smithi045.stdout:426: finishing write tid 3 to smithi04526579-3 2024-04-26T00:22:28.277 INFO:tasks.rados.rados.0.smithi045.stdout:426: finishing write tid 4 to smithi04526579-3 2024-04-26T00:22:28.277 INFO:tasks.rados.rados.0.smithi045.stdout:426: oid 3 updating version 0 to 112 2024-04-26T00:22:28.277 INFO:tasks.rados.rados.0.smithi045.stdout:426: oid 3 version 112 is already newer than 111 2024-04-26T00:22:28.278 INFO:tasks.rados.rados.0.smithi045.stdout:426: oid 3 version 112 is already newer than 110 2024-04-26T00:22:28.278 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 112 (ObjNum 182 snap 49 seq_num 182) dirty exists 2024-04-26T00:22:28.278 INFO:tasks.rados.rados.0.smithi045.stdout:426: left oid 3 (ObjNum 182 snap 49 seq_num 182) 2024-04-26T00:22:28.278 INFO:tasks.rados.rados.0.smithi045.stdout:428: expect (ObjNum 15 snap 0 seq_num 15) 2024-04-26T00:22:28.350 INFO:tasks.rados.rados.0.smithi045.stdout:434: expect (ObjNum 133 snap 30 seq_num 133) 2024-04-26T00:22:28.449 INFO:tasks.rados.rados.0.smithi045.stdout:435: expect (ObjNum 155 snap 33 seq_num 155) 2024-04-26T00:22:28.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:28.518+0000 7f62ebfff640 1 --2- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec13c790 0x7f62ec1cf6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:28.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:28.518+0000 7f62ebfff640 10 monclient: get_auth_request con 0x7f62ec13c790 auth_method 0 2024-04-26T00:22:28.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:28.518+0000 7f62ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:28.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:28.518+0000 7f62ebfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:28.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:28.518+0000 7f62ebfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:22:28.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:28.518+0000 7f62ebfff640 1 -- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec13c790 msgr2=0x7f62ec1cf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:22:28.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:28.518+0000 7f62ebfff640 1 -- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec13c790 msgr2=0x7f62ec1cf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:22:28.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:28.518+0000 7f62ebfff640 1 --2- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec13c790 0x7f62ec1cf6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:22:28.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:28.518+0000 7f62ebfff640 1 --2- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec13c790 0x7f62ec1cf6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:22:28.533 INFO:tasks.rados.rados.0.smithi045.stdout:436: finishing write tid 1 to smithi04526579-10 2024-04-26T00:22:28.533 INFO:tasks.rados.rados.0.smithi045.stdout:436: finishing write tid 2 to smithi04526579-10 2024-04-26T00:22:28.533 INFO:tasks.rados.rados.0.smithi045.stdout:436: finishing write tid 3 to smithi04526579-10 2024-04-26T00:22:28.533 INFO:tasks.rados.rados.0.smithi045.stdout:436: finishing write tid 4 to smithi04526579-10 2024-04-26T00:22:28.533 INFO:tasks.rados.rados.0.smithi045.stdout:436: finishing write tid 5 to smithi04526579-10 2024-04-26T00:22:28.533 INFO:tasks.rados.rados.0.smithi045.stdout:436: oid 10 updating version 0 to 169 2024-04-26T00:22:28.533 INFO:tasks.rados.rados.0.smithi045.stdout:436: oid 10 updating version 169 to 170 2024-04-26T00:22:28.533 INFO:tasks.rados.rados.0.smithi045.stdout:436: oid 10 updating version 170 to 171 2024-04-26T00:22:28.533 INFO:tasks.rados.rados.0.smithi045.stdout:436: oid 10 version 171 is already newer than 168 2024-04-26T00:22:28.533 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 171 (ObjNum 183 snap 49 seq_num 183) dirty exists 2024-04-26T00:22:28.533 INFO:tasks.rados.rados.0.smithi045.stdout:436: left oid 10 (ObjNum 183 snap 49 seq_num 183) 2024-04-26T00:22:28.533 INFO:tasks.rados.rados.0.smithi045.stdout:437: seq_num 184 ranges {692488=638288,1917275=606724} 2024-04-26T00:22:28.554 INFO:tasks.rados.rados.0.smithi045.stdout:437: writing smithi04526579-47 from 692488 to 1330776 tid 1 2024-04-26T00:22:28.574 INFO:tasks.rados.rados.0.smithi045.stdout:437: writing smithi04526579-47 from 1917275 to 2523999 tid 2 2024-04-26T00:22:28.575 INFO:tasks.rados.rados.0.smithi045.stdout:426: done (8 left) 2024-04-26T00:22:28.575 INFO:tasks.rados.rados.0.smithi045.stdout:427: done (7 left) 2024-04-26T00:22:28.575 INFO:tasks.rados.rados.0.smithi045.stdout:428: done (6 left) 2024-04-26T00:22:28.575 INFO:tasks.rados.rados.0.smithi045.stdout:429: done (5 left) 2024-04-26T00:22:28.575 INFO:tasks.rados.rados.0.smithi045.stdout:430: done (4 left) 2024-04-26T00:22:28.575 INFO:tasks.rados.rados.0.smithi045.stdout:434: done (3 left) 2024-04-26T00:22:28.575 INFO:tasks.rados.rados.0.smithi045.stdout:435: done (2 left) 2024-04-26T00:22:28.575 INFO:tasks.rados.rados.0.smithi045.stdout:436: done (1 left) 2024-04-26T00:22:28.575 INFO:tasks.rados.rados.0.smithi045.stdout:438: write (excl) oid 39 current snap is 49 2024-04-26T00:22:28.575 INFO:tasks.rados.rados.0.smithi045.stdout:438: seq_num 185 ranges {709295=713537,1859404=750382} 2024-04-26T00:22:28.595 INFO:tasks.rados.rados.0.smithi045.stdout:438: writing smithi04526579-39 from 709295 to 1422832 tid 1 2024-04-26T00:22:28.609 INFO:tasks.rados.rados.0.smithi045.stdout:438: writing smithi04526579-39 from 1859404 to 2609786 tid 2 2024-04-26T00:22:28.609 INFO:tasks.rados.rados.0.smithi045.stdout:439: read oid 7 snap -1 2024-04-26T00:22:28.609 INFO:tasks.rados.rados.0.smithi045.stdout:439: expect (ObjNum 164 snap 38 seq_num 164) 2024-04-26T00:22:28.609 INFO:tasks.rados.rados.0.smithi045.stdout:440: write (excl) oid 32 current snap is 49 2024-04-26T00:22:28.609 INFO:tasks.rados.rados.0.smithi045.stdout:437: finishing write tid 1 to smithi04526579-47 2024-04-26T00:22:28.609 INFO:tasks.rados.rados.0.smithi045.stdout:437: finishing write tid 2 to smithi04526579-47 2024-04-26T00:22:28.609 INFO:tasks.rados.rados.0.smithi045.stdout:437: finishing write tid 3 to smithi04526579-47 2024-04-26T00:22:28.609 INFO:tasks.rados.rados.0.smithi045.stdout:437: finishing write tid 4 to smithi04526579-47 2024-04-26T00:22:28.610 INFO:tasks.rados.rados.0.smithi045.stdout:437: oid 47 updating version 0 to 128 2024-04-26T00:22:28.610 INFO:tasks.rados.rados.0.smithi045.stdout:437: oid 47 version 128 is already newer than 127 2024-04-26T00:22:28.610 INFO:tasks.rados.rados.0.smithi045.stdout:437: oid 47 updating version 128 to 129 2024-04-26T00:22:28.610 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 129 (ObjNum 184 snap 49 seq_num 184) dirty exists 2024-04-26T00:22:28.610 INFO:tasks.rados.rados.0.smithi045.stdout:437: left oid 47 (ObjNum 184 snap 49 seq_num 184) 2024-04-26T00:22:28.610 INFO:tasks.rados.rados.0.smithi045.stdout:438: finishing write tid 1 to smithi04526579-39 2024-04-26T00:22:28.610 INFO:tasks.rados.rados.0.smithi045.stdout:440: seq_num 186 ranges {726102=788786,1917886=610393,3108206=17896} 2024-04-26T00:22:28.626 INFO:tasks.rados.rados.0.smithi045.stdout:440: writing smithi04526579-32 from 726102 to 1514888 tid 1 2024-04-26T00:22:28.638 INFO:tasks.rados.rados.0.smithi045.stdout:440: writing smithi04526579-32 from 1917886 to 2528279 tid 2 2024-04-26T00:22:28.645 INFO:tasks.rados.rados.0.smithi045.stdout:440: writing smithi04526579-32 from 3108206 to 3126102 tid 3 2024-04-26T00:22:28.645 INFO:tasks.rados.rados.0.smithi045.stdout:437: done (3 left) 2024-04-26T00:22:28.645 INFO:tasks.rados.rados.0.smithi045.stdout:441: read oid 3 snap -1 2024-04-26T00:22:28.646 INFO:tasks.rados.rados.0.smithi045.stdout:441: expect (ObjNum 182 snap 49 seq_num 182) 2024-04-26T00:22:28.646 INFO:tasks.rados.rados.0.smithi045.stdout:442: delete oid 6 current snap is 49 2024-04-26T00:22:28.646 INFO:tasks.rados.rados.0.smithi045.stdout:439: expect (ObjNum 164 snap 38 seq_num 164) 2024-04-26T00:22:28.770 INFO:tasks.rados.rados.0.smithi045.stdout:438: finishing write tid 2 to smithi04526579-39 2024-04-26T00:22:28.770 INFO:tasks.rados.rados.0.smithi045.stdout:438: finishing write tid 3 to smithi04526579-39 2024-04-26T00:22:28.770 INFO:tasks.rados.rados.0.smithi045.stdout:438: finishing write tid 4 to smithi04526579-39 2024-04-26T00:22:28.770 INFO:tasks.rados.rados.0.smithi045.stdout:438: oid 39 updating version 0 to 122 2024-04-26T00:22:28.770 INFO:tasks.rados.rados.0.smithi045.stdout:438: oid 39 version 122 is already newer than 120 2024-04-26T00:22:28.771 INFO:tasks.rados.rados.0.smithi045.stdout:438: oid 39 version 122 is already newer than 121 2024-04-26T00:22:28.771 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 122 (ObjNum 185 snap 49 seq_num 185) dirty exists 2024-04-26T00:22:28.771 INFO:tasks.rados.rados.0.smithi045.stdout:438: left oid 39 (ObjNum 185 snap 49 seq_num 185) 2024-04-26T00:22:28.771 INFO:tasks.rados.rados.0.smithi045.stdout:440: finishing write tid 1 to smithi04526579-32 2024-04-26T00:22:28.771 INFO:tasks.rados.rados.0.smithi045.stdout:440: finishing write tid 2 to smithi04526579-32 2024-04-26T00:22:28.771 INFO:tasks.rados.rados.0.smithi045.stdout:440: finishing write tid 3 to smithi04526579-32 2024-04-26T00:22:28.771 INFO:tasks.rados.rados.0.smithi045.stdout:440: finishing write tid 4 to smithi04526579-32 2024-04-26T00:22:28.771 INFO:tasks.rados.rados.0.smithi045.stdout:440: finishing write tid 5 to smithi04526579-32 2024-04-26T00:22:28.771 INFO:tasks.rados.rados.0.smithi045.stdout:440: oid 32 updating version 0 to 148 2024-04-26T00:22:28.771 INFO:tasks.rados.rados.0.smithi045.stdout:440: oid 32 version 148 is already newer than 147 2024-04-26T00:22:28.771 INFO:tasks.rados.rados.0.smithi045.stdout:440: oid 32 updating version 148 to 149 2024-04-26T00:22:28.771 INFO:tasks.rados.rados.0.smithi045.stdout:440: oid 32 updating version 149 to 150 2024-04-26T00:22:28.771 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 150 (ObjNum 186 snap 49 seq_num 186) dirty exists 2024-04-26T00:22:28.771 INFO:tasks.rados.rados.0.smithi045.stdout:440: left oid 32 (ObjNum 186 snap 49 seq_num 186) 2024-04-26T00:22:28.772 INFO:tasks.rados.rados.0.smithi045.stdout:441: expect (ObjNum 182 snap 49 seq_num 182) 2024-04-26T00:22:28.986 INFO:tasks.rados.rados.0.smithi045.stdout:438: done (4 left) 2024-04-26T00:22:28.986 INFO:tasks.rados.rados.0.smithi045.stdout:439: done (3 left) 2024-04-26T00:22:28.987 INFO:tasks.rados.rados.0.smithi045.stdout:440: done (2 left) 2024-04-26T00:22:28.987 INFO:tasks.rados.rados.0.smithi045.stdout:441: done (1 left) 2024-04-26T00:22:28.987 INFO:tasks.rados.rados.0.smithi045.stdout:442: done (0 left) 2024-04-26T00:22:28.987 INFO:tasks.rados.rados.0.smithi045.stdout:443: write oid 1 current snap is 49 2024-04-26T00:22:28.987 INFO:tasks.rados.rados.0.smithi045.stdout:443: seq_num 188 ranges {759716=539284,2034850=614062} 2024-04-26T00:22:29.005 INFO:tasks.rados.rados.0.smithi045.stdout:443: writing smithi04526579-1 from 759716 to 1299000 tid 1 2024-04-26T00:22:29.026 INFO:tasks.rados.rados.0.smithi045.stdout:443: writing smithi04526579-1 from 2034850 to 2648912 tid 2 2024-04-26T00:22:29.026 INFO:tasks.rados.rados.0.smithi045.stdout:444: write (excl) oid 2 current snap is 49 2024-04-26T00:22:29.026 INFO:tasks.rados.rados.0.smithi045.stdout:444: seq_num 189 ranges {776523=614533,1976979=757720} 2024-04-26T00:22:29.045 INFO:tasks.rados.rados.0.smithi045.stdout:444: writing smithi04526579-2 from 776523 to 1391056 tid 1 2024-04-26T00:22:29.060 INFO:tasks.rados.rados.0.smithi045.stdout:444: writing smithi04526579-2 from 1976979 to 2734699 tid 2 2024-04-26T00:22:29.060 INFO:tasks.rados.rados.0.smithi045.stdout:445: read oid 41 snap -1 2024-04-26T00:22:29.060 INFO:tasks.rados.rados.0.smithi045.stdout:445: expect (ObjNum 117 snap 28 seq_num 117) 2024-04-26T00:22:29.060 INFO:tasks.rados.rados.0.smithi045.stdout:446: write (excl) oid 26 current snap is 49 2024-04-26T00:22:29.060 INFO:tasks.rados.rados.0.smithi045.stdout:446: seq_num 190 ranges {793330=689782,2035461=617731,3101545=91785} 2024-04-26T00:22:29.073 INFO:tasks.rados.rados.0.smithi045.stdout:446: writing smithi04526579-26 from 793330 to 1483112 tid 1 2024-04-26T00:22:29.083 INFO:tasks.rados.rados.0.smithi045.stdout:446: writing smithi04526579-26 from 2035461 to 2653192 tid 2 2024-04-26T00:22:29.087 INFO:tasks.rados.rados.0.smithi045.stdout:446: writing smithi04526579-26 from 3101545 to 3193330 tid 3 2024-04-26T00:22:29.087 INFO:tasks.rados.rados.0.smithi045.stdout:447: write (excl) oid 49 current snap is 49 2024-04-26T00:22:29.087 INFO:tasks.rados.rados.0.smithi045.stdout:447: seq_num 191 ranges {410137=481384,1410296=761389,2728968=481169} 2024-04-26T00:22:29.093 INFO:tasks.rados.rados.0.smithi045.stdout:447: writing smithi04526579-49 from 410137 to 891521 tid 1 2024-04-26T00:22:29.102 INFO:tasks.rados.rados.0.smithi045.stdout:447: writing smithi04526579-49 from 1410296 to 2171685 tid 2 2024-04-26T00:22:29.108 INFO:tasks.rados.rados.0.smithi045.stdout:447: writing smithi04526579-49 from 2728968 to 3210137 tid 3 2024-04-26T00:22:29.108 INFO:tasks.rados.rados.0.smithi045.stdout:448: delete oid 44 current snap is 49 2024-04-26T00:22:29.109 INFO:tasks.rados.rados.0.smithi045.stdout:443: finishing write tid 1 to smithi04526579-1 2024-04-26T00:22:29.109 INFO:tasks.rados.rados.0.smithi045.stdout:443: finishing write tid 2 to smithi04526579-1 2024-04-26T00:22:29.109 INFO:tasks.rados.rados.0.smithi045.stdout:443: finishing write tid 3 to smithi04526579-1 2024-04-26T00:22:29.109 INFO:tasks.rados.rados.0.smithi045.stdout:443: finishing write tid 4 to smithi04526579-1 2024-04-26T00:22:29.109 INFO:tasks.rados.rados.0.smithi045.stdout:443: oid 1 updating version 0 to 170 2024-04-26T00:22:29.109 INFO:tasks.rados.rados.0.smithi045.stdout:443: oid 1 version 170 is already newer than 168 2024-04-26T00:22:29.109 INFO:tasks.rados.rados.0.smithi045.stdout:443: oid 1 version 170 is already newer than 169 2024-04-26T00:22:29.109 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 170 (ObjNum 188 snap 49 seq_num 188) dirty exists 2024-04-26T00:22:29.109 INFO:tasks.rados.rados.0.smithi045.stdout:443: left oid 1 (ObjNum 188 snap 49 seq_num 188) 2024-04-26T00:22:29.109 INFO:tasks.rados.rados.0.smithi045.stdout:444: finishing write tid 1 to smithi04526579-2 2024-04-26T00:22:29.110 INFO:tasks.rados.rados.0.smithi045.stdout:444: finishing write tid 2 to smithi04526579-2 2024-04-26T00:22:29.110 INFO:tasks.rados.rados.0.smithi045.stdout:444: finishing write tid 3 to smithi04526579-2 2024-04-26T00:22:29.110 INFO:tasks.rados.rados.0.smithi045.stdout:444: finishing write tid 4 to smithi04526579-2 2024-04-26T00:22:29.110 INFO:tasks.rados.rados.0.smithi045.stdout:444: oid 2 updating version 0 to 193 2024-04-26T00:22:29.110 INFO:tasks.rados.rados.0.smithi045.stdout:444: oid 2 updating version 193 to 194 2024-04-26T00:22:29.110 INFO:tasks.rados.rados.0.smithi045.stdout:444: oid 2 version 194 is already newer than 192 2024-04-26T00:22:29.110 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 194 (ObjNum 189 snap 49 seq_num 189) dirty exists 2024-04-26T00:22:29.110 INFO:tasks.rados.rados.0.smithi045.stdout:444: left oid 2 (ObjNum 189 snap 49 seq_num 189) 2024-04-26T00:22:29.110 INFO:tasks.rados.rados.0.smithi045.stdout:445: expect (ObjNum 117 snap 28 seq_num 117) 2024-04-26T00:22:29.397 INFO:tasks.rados.rados.0.smithi045.stdout:446: finishing write tid 1 to smithi04526579-26 2024-04-26T00:22:29.398 INFO:tasks.rados.rados.0.smithi045.stdout:446: finishing write tid 2 to smithi04526579-26 2024-04-26T00:22:29.398 INFO:tasks.rados.rados.0.smithi045.stdout:446: finishing write tid 3 to smithi04526579-26 2024-04-26T00:22:29.398 INFO:tasks.rados.rados.0.smithi045.stdout:446: finishing write tid 4 to smithi04526579-26 2024-04-26T00:22:29.398 INFO:tasks.rados.rados.0.smithi045.stdout:446: finishing write tid 5 to smithi04526579-26 2024-04-26T00:22:29.398 INFO:tasks.rados.rados.0.smithi045.stdout:446: oid 26 updating version 0 to 173 2024-04-26T00:22:29.398 INFO:tasks.rados.rados.0.smithi045.stdout:446: oid 26 updating version 173 to 175 2024-04-26T00:22:29.398 INFO:tasks.rados.rados.0.smithi045.stdout:446: oid 26 version 175 is already newer than 174 2024-04-26T00:22:29.398 INFO:tasks.rados.rados.0.smithi045.stdout:446: oid 26 version 175 is already newer than 172 2024-04-26T00:22:29.398 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 175 (ObjNum 190 snap 49 seq_num 190) dirty exists 2024-04-26T00:22:29.398 INFO:tasks.rados.rados.0.smithi045.stdout:446: left oid 26 (ObjNum 190 snap 49 seq_num 190) 2024-04-26T00:22:29.398 INFO:tasks.rados.rados.0.smithi045.stdout:447: finishing write tid 1 to smithi04526579-49 2024-04-26T00:22:29.398 INFO:tasks.rados.rados.0.smithi045.stdout:447: finishing write tid 2 to smithi04526579-49 2024-04-26T00:22:29.398 INFO:tasks.rados.rados.0.smithi045.stdout:447: finishing write tid 3 to smithi04526579-49 2024-04-26T00:22:29.399 INFO:tasks.rados.rados.0.smithi045.stdout:447: finishing write tid 4 to smithi04526579-49 2024-04-26T00:22:29.399 INFO:tasks.rados.rados.0.smithi045.stdout:447: finishing write tid 5 to smithi04526579-49 2024-04-26T00:22:29.399 INFO:tasks.rados.rados.0.smithi045.stdout:447: oid 49 updating version 0 to 134 2024-04-26T00:22:29.399 INFO:tasks.rados.rados.0.smithi045.stdout:447: oid 49 version 134 is already newer than 131 2024-04-26T00:22:29.399 INFO:tasks.rados.rados.0.smithi045.stdout:447: oid 49 version 134 is already newer than 132 2024-04-26T00:22:29.399 INFO:tasks.rados.rados.0.smithi045.stdout:447: oid 49 version 134 is already newer than 133 2024-04-26T00:22:29.399 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 134 (ObjNum 191 snap 49 seq_num 191) dirty exists 2024-04-26T00:22:29.399 INFO:tasks.rados.rados.0.smithi045.stdout:447: left oid 49 (ObjNum 191 snap 49 seq_num 191) 2024-04-26T00:22:29.399 INFO:tasks.rados.rados.0.smithi045.stdout:443: done (5 left) 2024-04-26T00:22:29.399 INFO:tasks.rados.rados.0.smithi045.stdout:444: done (4 left) 2024-04-26T00:22:29.399 INFO:tasks.rados.rados.0.smithi045.stdout:445: done (3 left) 2024-04-26T00:22:29.399 INFO:tasks.rados.rados.0.smithi045.stdout:446: done (2 left) 2024-04-26T00:22:29.399 INFO:tasks.rados.rados.0.smithi045.stdout:447: done (1 left) 2024-04-26T00:22:29.399 INFO:tasks.rados.rados.0.smithi045.stdout:448: done (0 left) 2024-04-26T00:22:29.399 INFO:tasks.rados.rados.0.smithi045.stdout:449: snap_remove snap 42 2024-04-26T00:22:30.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.102+0000 7f62e37fe640 10 monclient: tick 2024-04-26T00:22:30.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.102+0000 7f62e37fe640 10 monclient: _check_auth_tickets 2024-04-26T00:22:30.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.102+0000 7f62e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:22:30.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.102+0000 7f62e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:22:30.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.122+0000 7f62ebfff640 1 --2- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec13c790 0x7f62ec1cf6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:30.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.122+0000 7f62ebfff640 10 monclient: get_auth_request con 0x7f62ec13c790 auth_method 0 2024-04-26T00:22:30.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.122+0000 7f62ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:30.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.122+0000 7f62ebfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:30.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.122+0000 7f62ebfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:22:30.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.122+0000 7f62ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:30.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.122+0000 7f62ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:30.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.122+0000 7f62ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:30.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.122+0000 7f62ebfff640 10 monclient(hunting): handle_auth_done global_id 34100 payload 824 2024-04-26T00:22:30.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.122+0000 7f62ebfff640 1 -- 172.21.15.37:0/4185693357 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f62ec04f510 con 0x7f62ec13c790 2024-04-26T00:22:30.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.122+0000 7f62ebfff640 1 -- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec13c790 msgr2=0x7f62ec1cf6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:30.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.122+0000 7f62ebfff640 1 --2- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec13c790 0x7f62ec1cf6e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f62d40ae000 tx=0x7f62d4019650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:30.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.126+0000 7f62e8ff9640 1 -- 172.21.15.37:0/4185693357 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d40a5a50 con 0x7f62ec13c790 2024-04-26T00:22:30.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.126+0000 7f62e8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:30.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.126+0000 7f62e8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:22:30.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.126+0000 7f62e8ff9640 10 monclient: dump: 2024-04-26T00:22:30.103 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:30.103 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:30.103 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:30.103 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:30.103 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:30.103 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:30.104 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:30.104 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:30.104 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:30.104 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:30.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.126+0000 7f62e8ff9640 1 -- 172.21.15.37:0/4185693357 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f62d40c1470 con 0x7f62ec13c790 2024-04-26T00:22:30.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.126+0000 7f62e8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:30.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.126+0000 7f62e8ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:22:30.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.126+0000 7f62e8ff9640 1 -- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec13c790 msgr2=0x7f62ec1cf6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:30.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.126+0000 7f62e8ff9640 1 --2- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec13c790 0x7f62ec1cf6e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f62d40ae000 tx=0x7f62d4019650 comp rx=0 tx=0).stop 2024-04-26T00:22:30.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.126+0000 7f62e8ff9640 1 -- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec13c790 msgr2=0x7f62ec1cf6e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:22:30.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.126+0000 7f62e8ff9640 1 --2- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec13c790 0x7f62ec1cf6e0 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f62d40ae000 tx=0x7f62d4019650 comp rx=0 tx=0).stop 2024-04-26T00:22:30.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.134+0000 7f62f2300640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:22:30.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.134+0000 7f62f2300640 1 --2- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec1cfc20 0x7f62ec134b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:30.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.134+0000 7f62f2300640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:30.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.134+0000 7f62f2300640 1 -- 172.21.15.37:0/4185693357 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f62ec1d0030 con 0x7f62ec1cfc20 2024-04-26T00:22:30.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.134+0000 7f62ebfff640 1 --2- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec1cfc20 0x7f62ec134b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:30.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.134+0000 7f62ebfff640 10 monclient: get_auth_request con 0x7f62ec1cfc20 auth_method 0 2024-04-26T00:22:30.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.134+0000 7f62ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:30.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.134+0000 7f62ebfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:30.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.134+0000 7f62ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:30.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.134+0000 7f62ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:30.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.134+0000 7f62ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:30.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.134+0000 7f62ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:30.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.134+0000 7f62ebfff640 10 monclient(hunting): handle_auth_done global_id 34103 payload 824 2024-04-26T00:22:30.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.134+0000 7f62ebfff640 1 -- 172.21.15.37:0/4185693357 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f62ec1338e0 con 0x7f62ec1cfc20 2024-04-26T00:22:30.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.138+0000 7f62ebfff640 1 -- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec1cfc20 msgr2=0x7f62ec134b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:30.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.138+0000 7f62ebfff640 1 --2- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec1cfc20 0x7f62ec134b30 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f62d401af60 tx=0x7f62d4009810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:30.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.138+0000 7f62e8ff9640 1 -- 172.21.15.37:0/4185693357 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d40cc070 con 0x7f62ec1cfc20 2024-04-26T00:22:30.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.138+0000 7f62e8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:30.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.138+0000 7f62e8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:22:30.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.138+0000 7f62e8ff9640 10 monclient: dump: 2024-04-26T00:22:30.116 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:30.117 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:30.117 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:30.117 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:30.117 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:30.117 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:30.117 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:30.117 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:30.117 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:30.117 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:30.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.138+0000 7f62e8ff9640 1 -- 172.21.15.37:0/4185693357 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7f62d40ad3f0 con 0x7f62ec1cfc20 2024-04-26T00:22:30.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.138+0000 7f62e8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:22:30.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.138+0000 7f62e8ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:22:30.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.138+0000 7f62e8ff9640 1 -- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec1cfc20 msgr2=0x7f62ec134b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:30.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.138+0000 7f62e8ff9640 1 --2- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec1cfc20 0x7f62ec134b30 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f62d401af60 tx=0x7f62d4009810 comp rx=0 tx=0).stop 2024-04-26T00:22:30.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.138+0000 7f62e8ff9640 1 -- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec1cfc20 msgr2=0x7f62ec134b30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:22:30.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.138+0000 7f62e8ff9640 1 --2- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec1cfc20 0x7f62ec134b30 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f62d401af60 tx=0x7f62d4009810 comp rx=0 tx=0).stop 2024-04-26T00:22:30.118 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:22:30.119 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:22:30.119 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:22:30.119 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:22:30.119 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 24, 2024-04-26T00:22:30.119 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 10415, 2024-04-26T00:22:30.119 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:22:30.119 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:22:30.119 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:22:30.119 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:22:30.119 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:22:30.119 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 4, 2024-04-26T00:22:30.119 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:22:30.120 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:22:30.120 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:22:30.120 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:22:30.120 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:22:30.120 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:22:30.120 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:22:30.120 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:22:30.120 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:22:30.120 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:22:30.120 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:22:30.120 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:22:30.120 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:22:30.120 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:22:30.120 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:22:30.120 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:22:30.120 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:22:30.121 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:22:30.121 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:22:30.121 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:22:30.121 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:22:30.121 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:22:30.121 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:22:30.121 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:22:30.121 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:22:30.121 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:22:30.121 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:22:30.121 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:22:30.121 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:30.121 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:30.121 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:22:30.121 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:22:30.122 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:22:30.122 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:22:30.122 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:22:30.122 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:22:30.122 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:22:30.122 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:22:30.122 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:22:30.122 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:22:30.122 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:22:30.122 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:22:30.122 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:22:30.122 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:22:30.122 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:22:30.122 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:22:30.123 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:22:30.123 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:22:30.123 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:22:30.123 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:22:30.123 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:22:30.123 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:22:30.123 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:22:30.123 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:22:30.123 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:22:30.123 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:22:30.123 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:22:30.123 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:22:30.123 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:22:30.123 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:22:30.124 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:30.124 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:22:30.124 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:30.124 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:22:30.124 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:22:30.124 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:22:30.124 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:22:30.124 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:30.124 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:22:30.124 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:22:30.124 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:30.124 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:30.124 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:30.125 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:22:30.125 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:22:30.125 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:30.125 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:30.125 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:30.125 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:30.125 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:22:30.125 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:22:30.125 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:22:30.125 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:22:30.125 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:22:30.125 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:30.126 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:30.126 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:22:30.126 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:22:30.126 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:22:30.126 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:22:30.126 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:30.126 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:22:30.126 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:22:30.126 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:30.126 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:30.127 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:30.127 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:22:30.127 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:22:30.127 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:30.127 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:30.127 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:30.127 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:30.127 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:22:30.127 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:22:30.127 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:22:30.127 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:22:30.127 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:22:30.128 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:30.128 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:30.128 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:22:30.128 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:22:30.128 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:22:30.128 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:22:30.128 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:30.128 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:22:30.128 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:22:30.129 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:30.129 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:30.129 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:30.129 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:22:30.129 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:22:30.129 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:30.129 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:30.129 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:30.129 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:30.130 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:22:30.130 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:22:30.130 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:22:30.130 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:22:30.130 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:22:30.130 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:30.130 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:30.130 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:30.130 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:22:30.130 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:22:30.131 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:30.131 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:22:30.131 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:22:30.131 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:22:30.131 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:30.131 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:22:30.131 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:22:30.131 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:30.131 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:22:30.132 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:22:30.132 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:22:30.132 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:30.132 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:30.132 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:30.132 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:22:30.132 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:22:30.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.138+0000 7f62f2300640 1 -- 172.21.15.37:0/4185693357 >> v2:172.21.15.45:6812/2379918623 conn(0x7f62d0035eb0 msgr2=0x7f62d0038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:30.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.138+0000 7f62f2300640 1 --2- 172.21.15.37:0/4185693357 >> v2:172.21.15.45:6812/2379918623 conn(0x7f62d0035eb0 0x7f62d0038370 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f62ec13b5b0 tx=0x7f62dc0092e0 comp rx=0 tx=0).stop 2024-04-26T00:22:30.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.138+0000 7f62f2300640 10 monclient: shutdown 2024-04-26T00:22:30.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.138+0000 7f62f2300640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:22:30.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.138+0000 7f62f2300640 1 -- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f62ec0a4170 msgr2=0x7f62ec1333a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:30.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.138+0000 7f62f2300640 1 --2- 172.21.15.37:0/4185693357 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f62ec0a4170 0x7f62ec1333a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f62e400d800 tx=0x7f62e400dcc0 comp rx=0 tx=0).stop 2024-04-26T00:22:30.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.138+0000 7f62f089d640 1 -- 172.21.15.37:0/4185693357 reap_dead start 2024-04-26T00:22:30.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.138+0000 7f62f089d640 1 -- 172.21.15.37:0/4185693357 reap_dead start 2024-04-26T00:22:30.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.138+0000 7f62f2300640 1 -- 172.21.15.37:0/4185693357 shutdown_connections 2024-04-26T00:22:30.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.138+0000 7f62f2300640 1 -- 172.21.15.37:0/4185693357 >> 172.21.15.37:0/4185693357 conn(0x7f62ec0aa0a0 msgr2=0x7f62ec0a7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:22:30.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.138+0000 7f62f2300640 1 -- 172.21.15.37:0/4185693357 shutdown_connections 2024-04-26T00:22:30.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.138+0000 7f62f2300640 1 -- 172.21.15.37:0/4185693357 wait complete. 2024-04-26T00:22:30.142 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:22:30.142 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:22:30.142 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:22:30.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd52983c640 10 monclient: get_monmap_and_config 2024-04-26T00:22:30.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd52983c640 10 monclient: build_initial_monmap 2024-04-26T00:22:30.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd52983c640 10 monclient: monmap: 2024-04-26T00:22:30.272 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:22:30.272 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:30.272 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:22:30.296629+0000 2024-04-26T00:22:30.272 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:22:30.296629+0000 2024-04-26T00:22:30.272 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:22:30.272 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:30.272 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:22:30.272 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:22:30.272 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:22:30.273 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:22:30.273 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:22:30.273 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:30.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd52983c640 1 Processor -- start 2024-04-26T00:22:30.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd52983c640 1 -- start start 2024-04-26T00:22:30.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd52983c640 10 monclient: init 2024-04-26T00:22:30.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd52983c640 10 monclient: _reopen_session rank -1 2024-04-26T00:22:30.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd52983c640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-04-26T00:22:30.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd52983c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd5240a3d90 0x7fd52409bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:30.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd52983c640 10 monclient(hunting): picked mon.noname-a con 0x7fd5240a3d90 addr v2:172.21.15.37:3300/0 2024-04-26T00:22:30.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd52983c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd5240a4170 addr v1:172.21.15.37:6789/0 2024-04-26T00:22:30.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd52983c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd52413c3e0 0x7fd52413e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:30.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd52983c640 10 monclient(hunting): picked mon.noname-b con 0x7fd52413c3e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:22:30.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd52983c640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5240a5150 con 0x7fd5240a4170 2024-04-26T00:22:30.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd52983c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:30.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd52983c640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fd52409b280 con 0x7fd5240a3d90 2024-04-26T00:22:30.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd52983c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:30.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd52983c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd52409b560 con 0x7fd52413c3e0 2024-04-26T00:22:30.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd52983c640 10 monclient(hunting): _renew_subs 2024-04-26T00:22:30.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd52983c640 10 monclient(hunting): authenticate will time out at 1131.602920s 2024-04-26T00:22:30.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd5237fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd52413c3e0 0x7fd52413e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:30.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd5237fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd52413c3e0 0x7fd52413e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:53542/0 (socket says 172.21.15.37:53542) 2024-04-26T00:22:30.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd5237fe640 1 -- 172.21.15.37:0/2382294501 learned_addr learned my addr 172.21.15.37:0/2382294501 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:22:30.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd5237fe640 10 monclient(hunting): get_auth_request con 0x7fd52413c3e0 auth_method 0 2024-04-26T00:22:30.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd5237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:30.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd5237fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:30.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd5237fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:30.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd522ffd640 1 --2- 172.21.15.37:0/2382294501 >> v2:172.21.15.37:3300/0 conn(0x7fd5240a3d90 0x7fd52409bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:30.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd522ffd640 10 monclient(hunting): get_auth_request con 0x7fd5240a3d90 auth_method 0 2024-04-26T00:22:30.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd522ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:30.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd522ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:30.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd522ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:30.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd5237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:30.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd5237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:30.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd5237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:30.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd522ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:30.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd522ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:30.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd522ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:30.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd5237fe640 10 monclient(hunting): handle_auth_done global_id 34106 payload 274 2024-04-26T00:22:30.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd5237fe640 1 -- 172.21.15.37:0/2382294501 >> v2:172.21.15.37:3300/0 conn(0x7fd5240a3d90 msgr2=0x7fd52409bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:22:30.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd5237fe640 1 --2- 172.21.15.37:0/2382294501 >> v2:172.21.15.37:3300/0 conn(0x7fd5240a3d90 0x7fd52409bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:30.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd5237fe640 1 -- 172.21.15.37:0/2382294501 >> v1:172.21.15.37:6789/0 conn(0x7fd5240a4170 legacy=0x7fd5240a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:22:30.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd5237fe640 10 monclient: _finish_hunting 0 2024-04-26T00:22:30.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd5237fe640 1 monclient: found mon.noname-b 2024-04-26T00:22:30.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd5237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:22:30.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd5237fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:22:30.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.294+0000 7fd5237fe640 1 -- 172.21.15.37:0/2382294501 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd52406dc60 con 0x7fd52413c3e0 2024-04-26T00:22:30.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.298+0000 7fd5237fe640 1 -- 172.21.15.37:0/2382294501 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd52413c3e0 msgr2=0x7fd52413e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:30.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.298+0000 7fd5237fe640 1 --2- 172.21.15.37:0/2382294501 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd52413c3e0 0x7fd52413e7d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd51c009e50 tx=0x7fd51c01c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e012e1756ed7c2cc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:30.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd521ffb640 1 -- 172.21.15.37:0/2382294501 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd51c0a5530 con 0x7fd52413c3e0 2024-04-26T00:22:30.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd521ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:30.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd521ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:22:30.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd521ffb640 10 monclient: dump: 2024-04-26T00:22:30.283 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:30.283 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:30.283 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:30.283 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:30.283 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:30.283 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:30.283 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:30.284 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:30.284 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:30.284 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:30.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd521ffb640 10 monclient: _finish_auth 0 2024-04-26T00:22:30.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd521ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:22:30.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd521ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:22:30.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd521ffb640 1 -- 172.21.15.37:0/2382294501 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd51c0a56d0 con 0x7fd52413c3e0 2024-04-26T00:22:30.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd521ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:22:30.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd521ffb640 1 -- 172.21.15.37:0/2382294501 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd51c0a5aa0 con 0x7fd52413c3e0 2024-04-26T00:22:30.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd52983c640 5 monclient: authenticate success, global_id 34106 2024-04-26T00:22:30.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd52983c640 10 monclient: get_monmap_and_config success 2024-04-26T00:22:30.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd52983c640 4 set_mon_vals no callback set 2024-04-26T00:22:30.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd521ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:30.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd521ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:22:30.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd521ffb640 10 monclient: dump: 2024-04-26T00:22:30.285 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:30.285 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:30.285 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:30.285 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:30.285 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:30.285 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:30.285 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:30.285 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:30.285 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:30.285 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:30.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd52983c640 10 monclient: shutdown 2024-04-26T00:22:30.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd52983c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:22:30.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd52983c640 1 -- 172.21.15.37:0/2382294501 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd52413c3e0 msgr2=0x7fd52413e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:30.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd52983c640 1 --2- 172.21.15.37:0/2382294501 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd52413c3e0 0x7fd52413e7d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd51c009e50 tx=0x7fd51c01c8b0 comp rx=0 tx=0).stop 2024-04-26T00:22:30.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd52983c640 1 -- 172.21.15.37:0/2382294501 shutdown_connections 2024-04-26T00:22:30.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd52983c640 1 --2- 172.21.15.37:0/2382294501 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd52413c3e0 0x7fd52413e7d0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:30.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd52983c640 1 --2- 172.21.15.37:0/2382294501 >> v2:172.21.15.37:3300/0 conn(0x7fd5240a3d90 0x7fd52409bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:30.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd52983c640 1 -- 172.21.15.37:0/2382294501 >> 172.21.15.37:0/2382294501 conn(0x7fd5240aa0a0 msgr2=0x7fd5240a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:22:30.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd52983c640 1 -- 172.21.15.37:0/2382294501 shutdown_connections 2024-04-26T00:22:30.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd52983c640 1 -- 172.21.15.37:0/2382294501 wait complete. 2024-04-26T00:22:30.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd52983c640 10 monclient: build_initial_monmap 2024-04-26T00:22:30.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd52983c640 10 monclient: monmap: 2024-04-26T00:22:30.286 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:22:30.286 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:22:30.286 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:22:30.286 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:22:30.286 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:22:30.286 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:22:30.287 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:22:30.287 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:22:30.287 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:22:30.287 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:30.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd52983c640 1 Processor -- start 2024-04-26T00:22:30.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd52983c640 1 -- start start 2024-04-26T00:22:30.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd52983c640 10 monclient: init 2024-04-26T00:22:30.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd52983c640 10 monclient: _reopen_session rank -1 2024-04-26T00:22:30.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd52983c640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:22:30.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd52983c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd5240a3d90 0x7fd5241cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:30.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd52983c640 10 monclient(hunting): picked mon.noname-c con 0x7fd5240a3d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:22:30.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd52983c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd5240a4170 0x7fd524136e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:30.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd52983c640 10 monclient(hunting): picked mon.noname-a con 0x7fd5240a4170 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:22:30.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd52983c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd52413c3e0 0x7fd524139da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:30.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd52983c640 10 monclient(hunting): picked mon.noname-b con 0x7fd52413c3e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:22:30.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd52983c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:30.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd52983c640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd5241418c0 con 0x7fd5240a4170 2024-04-26T00:22:30.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd52983c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:30.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd52983c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd524141740 con 0x7fd52413c3e0 2024-04-26T00:22:30.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd52983c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:30.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd52983c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd524141a40 con 0x7fd5240a3d90 2024-04-26T00:22:30.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd52983c640 10 monclient(hunting): _renew_subs 2024-04-26T00:22:30.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd52983c640 10 monclient(hunting): authenticate will time out at 1131.610258s 2024-04-26T00:22:30.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd5227fc640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd5240a4170 0x7fd524136e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:30.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd5227fc640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd5240a4170 0x7fd524136e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44960/0 (socket says 172.21.15.37:44960) 2024-04-26T00:22:30.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd5227fc640 1 -- 172.21.15.37:0/3872704976 learned_addr learned my addr 172.21.15.37:0/3872704976 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:22:30.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd5227fc640 10 monclient(hunting): get_auth_request con 0x7fd5240a4170 auth_method 0 2024-04-26T00:22:30.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd5227fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:30.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd5227fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:30.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd5227fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:30.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd522ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd5240a3d90 0x7fd5241cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:30.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd522ffd640 10 monclient(hunting): get_auth_request con 0x7fd5240a3d90 auth_method 0 2024-04-26T00:22:30.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd522ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:30.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd522ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:30.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd522ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:30.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd5227fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:30.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd5227fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:30.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd5227fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:30.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd522ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:30.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd522ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:30.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd522ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:30.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd5227fc640 10 monclient(hunting): handle_auth_done global_id 4647 payload 824 2024-04-26T00:22:30.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd5227fc640 1 -- 172.21.15.37:0/3872704976 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd5240a3d90 msgr2=0x7fd5241cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:30.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd5237fe640 1 --2- 172.21.15.37:0/3872704976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd52413c3e0 0x7fd524139da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:30.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd5227fc640 1 --2- 172.21.15.37:0/3872704976 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd5240a3d90 0x7fd5241cecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:30.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd5227fc640 1 -- 172.21.15.37:0/3872704976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd52413c3e0 msgr2=0x7fd524139da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:30.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd5227fc640 1 --2- 172.21.15.37:0/3872704976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd52413c3e0 0x7fd524139da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:30.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd5227fc640 10 monclient: _finish_hunting 0 2024-04-26T00:22:30.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd5227fc640 1 monclient: found mon.noname-a 2024-04-26T00:22:30.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd5227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:22:30.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd5227fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:22:30.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd5227fc640 1 -- 172.21.15.37:0/3872704976 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd52409ada0 con 0x7fd5240a4170 2024-04-26T00:22:30.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd5237fe640 10 monclient: get_auth_request con 0x7fd52413c3e0 auth_method 0 2024-04-26T00:22:30.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.302+0000 7fd5237fe640 1 --2- 172.21.15.37:0/3872704976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd52413c3e0 0x7fd524139da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:22:30.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd5227fc640 1 -- 172.21.15.37:0/3872704976 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd5240a4170 msgr2=0x7fd524136e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:30.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd5227fc640 1 --2- 172.21.15.37:0/3872704976 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd5240a4170 0x7fd524136e60 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd51800c450 tx=0x7fd51800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:30.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd50bfff640 1 -- 172.21.15.37:0/3872704976 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd518012a40 con 0x7fd5240a4170 2024-04-26T00:22:30.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd50bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:30.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd50bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:22:30.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd50bfff640 10 monclient: dump: 2024-04-26T00:22:30.294 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:30.294 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:30.294 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:30.294 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:30.294 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:30.294 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:30.294 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:30.294 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:30.294 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:30.295 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:30.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd50bfff640 10 monclient: _finish_auth 0 2024-04-26T00:22:30.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd50bfff640 10 monclient: _check_auth_tickets 2024-04-26T00:22:30.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd50bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:22:30.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd50bfff640 1 -- 172.21.15.37:0/3872704976 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd518017070 con 0x7fd5240a4170 2024-04-26T00:22:30.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd52983c640 5 monclient: authenticate success, global_id 4647 2024-04-26T00:22:30.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd52983c640 10 monclient: _renew_subs 2024-04-26T00:22:30.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd52983c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:22:30.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd52983c640 1 -- 172.21.15.37:0/3872704976 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd52413cae0 con 0x7fd5240a4170 2024-04-26T00:22:30.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd50bfff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:22:30.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd50bfff640 1 -- 172.21.15.37:0/3872704976 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd518012e40 con 0x7fd5240a4170 2024-04-26T00:22:30.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd50bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:30.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd50bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:22:30.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd50bfff640 10 monclient: dump: 2024-04-26T00:22:30.296 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:30.296 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:30.296 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:30.296 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:30.296 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:30.296 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:30.296 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:30.296 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:30.296 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:30.296 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:30.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd5217fa640 4 set_mon_vals no callback set 2024-04-26T00:22:30.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd52983c640 10 monclient: _renew_subs 2024-04-26T00:22:30.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd52983c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:22:30.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd52983c640 1 -- 172.21.15.37:0/3872704976 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5241217f0 con 0x7fd5240a4170 2024-04-26T00:22:30.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd52983c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:30.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd52983c640 1 --2- 172.21.15.37:0/3872704976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd52402c000 0x7fd52402c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:30.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd50bfff640 1 -- 172.21.15.37:0/3872704976 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd51801c460 con 0x7fd5240a4170 2024-04-26T00:22:30.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd52983c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:30.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd52983c640 1 -- 172.21.15.37:0/3872704976 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd5241cfc00 con 0x7fd52402c000 2024-04-26T00:22:30.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd50bfff640 1 --2- 172.21.15.37:0/3872704976 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd510035e80 0x7fd510038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:30.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd50bfff640 1 -- 172.21.15.37:0/3872704976 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 3975+0+0 (secure 0 0 0) 0x7fd518048880 con 0x7fd5240a4170 2024-04-26T00:22:30.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd5237fe640 1 --2- 172.21.15.37:0/3872704976 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd510035e80 0x7fd510038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:30.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd5237fe640 10 monclient: get_auth_request con 0x7fd510035e80 auth_method 0 2024-04-26T00:22:30.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd522ffd640 1 --2- 172.21.15.37:0/3872704976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd52402c000 0x7fd52402c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:30.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd522ffd640 10 monclient: get_auth_request con 0x7fd52402c000 auth_method 0 2024-04-26T00:22:30.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd522ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:30.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd522ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:30.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd522ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:30.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd522ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:30.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd522ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:30.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd522ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:30.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd522ffd640 10 monclient(hunting): handle_auth_done global_id 24181 payload 824 2024-04-26T00:22:30.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.306+0000 7fd522ffd640 1 -- 172.21.15.37:0/3872704976 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd52404f510 con 0x7fd52402c000 2024-04-26T00:22:30.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.310+0000 7fd5237fe640 1 --2- 172.21.15.37:0/3872704976 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd510035e80 0x7fd510038340 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd51c0ab000 tx=0x7fd51c0a7070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:30.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.310+0000 7fd522ffd640 1 -- 172.21.15.37:0/3872704976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd52402c000 msgr2=0x7fd52402c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:30.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.310+0000 7fd522ffd640 1 --2- 172.21.15.37:0/3872704976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd52402c000 0x7fd52402c3e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd514011900 tx=0x7fd514011dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:30.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.310+0000 7fd50bfff640 1 -- 172.21.15.37:0/3872704976 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd514019070 con 0x7fd52402c000 2024-04-26T00:22:30.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.310+0000 7fd50bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:30.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.310+0000 7fd50bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:22:30.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.310+0000 7fd50bfff640 10 monclient: dump: 2024-04-26T00:22:30.299 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:30.299 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:30.299 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:30.299 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:30.299 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:30.299 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:30.299 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:30.299 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:30.299 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:30.299 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:30.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.310+0000 7fd50bfff640 1 -- 172.21.15.37:0/3872704976 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd51400f890 con 0x7fd52402c000 2024-04-26T00:22:30.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.310+0000 7fd50bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:30.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.310+0000 7fd50bfff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:22:30.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.310+0000 7fd50bfff640 1 -- 172.21.15.37:0/3872704976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd52402c000 msgr2=0x7fd52402c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:30.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.310+0000 7fd50bfff640 1 --2- 172.21.15.37:0/3872704976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd52402c000 0x7fd52402c3e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd514011900 tx=0x7fd514011dc0 comp rx=0 tx=0).stop 2024-04-26T00:22:30.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.310+0000 7fd50bfff640 1 -- 172.21.15.37:0/3872704976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd52402c000 msgr2=0x7fd52402c3e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:22:30.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.310+0000 7fd50bfff640 1 --2- 172.21.15.37:0/3872704976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd52402c000 0x7fd52402c3e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:30.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.318+0000 7fd52983c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:22:30.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.318+0000 7fd52983c640 1 --2- 172.21.15.37:0/3872704976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd52402c920 0x7fd52402cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:30.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.318+0000 7fd52983c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:30.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.318+0000 7fd52983c640 1 -- 172.21.15.37:0/3872704976 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd52409e880 con 0x7fd52402c920 2024-04-26T00:22:30.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.318+0000 7fd522ffd640 1 --2- 172.21.15.37:0/3872704976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd52402c920 0x7fd52402cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:30.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.318+0000 7fd522ffd640 10 monclient: get_auth_request con 0x7fd52402c920 auth_method 0 2024-04-26T00:22:30.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.318+0000 7fd522ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:30.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.318+0000 7fd522ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:30.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.318+0000 7fd522ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:30.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.318+0000 7fd522ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:30.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.318+0000 7fd522ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:30.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.318+0000 7fd522ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:30.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.318+0000 7fd522ffd640 10 monclient(hunting): handle_auth_done global_id 24184 payload 824 2024-04-26T00:22:30.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.318+0000 7fd522ffd640 1 -- 172.21.15.37:0/3872704976 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd524075dc0 con 0x7fd52402c920 2024-04-26T00:22:30.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.322+0000 7fd522ffd640 1 -- 172.21.15.37:0/3872704976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd52402c920 msgr2=0x7fd52402cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:30.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.322+0000 7fd522ffd640 1 --2- 172.21.15.37:0/3872704976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd52402c920 0x7fd52402cd30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd514011900 tx=0x7fd5140046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:30.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.322+0000 7fd50bfff640 1 -- 172.21.15.37:0/3872704976 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd514019040 con 0x7fd52402c920 2024-04-26T00:22:30.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.322+0000 7fd50bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:30.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.322+0000 7fd50bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:22:30.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.322+0000 7fd50bfff640 10 monclient: dump: 2024-04-26T00:22:30.303 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:30.303 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:30.303 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:30.303 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:30.303 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:30.303 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:30.303 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:30.303 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:30.303 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:30.303 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:30.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.322+0000 7fd50bfff640 1 -- 172.21.15.37:0/3872704976 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4716 (secure 0 0 0) 0x7fd51401e660 con 0x7fd52402c920 2024-04-26T00:22:30.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.322+0000 7fd50bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:22:30.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.322+0000 7fd50bfff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:22:30.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.322+0000 7fd50bfff640 1 -- 172.21.15.37:0/3872704976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd52402c920 msgr2=0x7fd52402cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:30.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.322+0000 7fd50bfff640 1 --2- 172.21.15.37:0/3872704976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd52402c920 0x7fd52402cd30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd514011900 tx=0x7fd5140046d0 comp rx=0 tx=0).stop 2024-04-26T00:22:30.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.322+0000 7fd50bfff640 1 -- 172.21.15.37:0/3872704976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd52402c920 msgr2=0x7fd52402cd30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:22:30.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.322+0000 7fd50bfff640 1 --2- 172.21.15.37:0/3872704976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd52402c920 0x7fd52402cd30 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd514011900 tx=0x7fd5140046d0 comp rx=0 tx=0).stop 2024-04-26T00:22:30.304 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:22:30.304 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:22:30.304 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:22:30.304 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:22:30.304 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 24, 2024-04-26T00:22:30.304 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 58485, 2024-04-26T00:22:30.304 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:22:30.304 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:22:30.305 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:22:30.305 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:22:30.305 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:22:30.305 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 4, 2024-04-26T00:22:30.305 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:22:30.305 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:22:30.305 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:22:30.305 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:22:30.305 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:22:30.305 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:22:30.305 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:22:30.305 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:22:30.305 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:22:30.305 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:22:30.305 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:22:30.305 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:22:30.306 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:22:30.306 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:22:30.306 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:22:30.306 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:22:30.306 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:22:30.306 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:22:30.306 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:22:30.306 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:22:30.306 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:22:30.306 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:22:30.306 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:22:30.306 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:22:30.306 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:22:30.306 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:22:30.307 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:22:30.307 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:22:30.307 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:30.307 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:30.307 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:22:30.307 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:22:30.307 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:22:30.307 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:22:30.307 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:22:30.307 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:22:30.307 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:22:30.307 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:22:30.307 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:22:30.307 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:22:30.307 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:22:30.307 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:22:30.308 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:22:30.308 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:22:30.308 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:22:30.308 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:22:30.308 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:22:30.308 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:22:30.308 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:22:30.308 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:22:30.308 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:22:30.308 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:22:30.308 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:22:30.308 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:22:30.309 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:22:30.309 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:22:30.309 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:22:30.309 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:22:30.309 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:22:30.309 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:22:30.309 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:30.309 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:22:30.309 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:30.309 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:22:30.309 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:22:30.309 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:22:30.309 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:22:30.310 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:30.310 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:22:30.310 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:22:30.310 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:30.310 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:30.310 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:30.310 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:22:30.310 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:22:30.310 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:30.310 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:30.310 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:30.310 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:30.310 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:22:30.311 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:22:30.311 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:22:30.311 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:22:30.311 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:22:30.311 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:30.311 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:30.311 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:22:30.311 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:22:30.311 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:22:30.311 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:22:30.311 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:30.311 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:22:30.312 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:22:30.312 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:30.312 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:30.312 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:30.312 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:22:30.312 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:22:30.312 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:30.312 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:30.312 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:30.312 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:30.312 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:22:30.313 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:22:30.313 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:22:30.313 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:22:30.313 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:22:30.313 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:30.313 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:30.313 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:22:30.313 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:22:30.313 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:22:30.313 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:22:30.313 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:30.314 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:22:30.314 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:22:30.314 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:30.314 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:30.314 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:30.314 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:22:30.314 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:22:30.314 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:30.314 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:30.314 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:30.314 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:30.314 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:22:30.314 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:22:30.314 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:22:30.315 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:22:30.315 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:22:30.315 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:30.315 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:30.315 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:30.315 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:22:30.315 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:22:30.315 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:30.315 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:22:30.315 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:22:30.315 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:22:30.315 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:30.315 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:22:30.315 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:22:30.315 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:30.316 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:22:30.316 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:22:30.316 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:22:30.316 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:30.316 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:22:30.316 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:22:30.316 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:30.316 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:22:30.316 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:22:30.316 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:22:30.316 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:30.316 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:30.317 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:30.317 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:22:30.317 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:22:30.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.322+0000 7fd52983c640 1 -- 172.21.15.37:0/3872704976 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd510035e80 msgr2=0x7fd510038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:30.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.322+0000 7fd52983c640 1 --2- 172.21.15.37:0/3872704976 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd510035e80 0x7fd510038340 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd51c0ab000 tx=0x7fd51c0a7070 comp rx=0 tx=0).stop 2024-04-26T00:22:30.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.322+0000 7fd52983c640 10 monclient: shutdown 2024-04-26T00:22:30.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.322+0000 7fd52983c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:22:30.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.322+0000 7fd52983c640 1 -- 172.21.15.37:0/3872704976 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd5240a4170 msgr2=0x7fd524136e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:30.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.322+0000 7fd52983c640 1 --2- 172.21.15.37:0/3872704976 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd5240a4170 0x7fd524136e60 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd51800c450 tx=0x7fd51800c910 comp rx=0 tx=0).stop 2024-04-26T00:22:30.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.322+0000 7fd5237fe640 1 -- 172.21.15.37:0/3872704976 reap_dead start 2024-04-26T00:22:30.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.322+0000 7fd52983c640 1 -- 172.21.15.37:0/3872704976 shutdown_connections 2024-04-26T00:22:30.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.322+0000 7fd52983c640 1 --2- 172.21.15.37:0/3872704976 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd5240a4170 0x7fd524136e60 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:30.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.322+0000 7fd52983c640 1 -- 172.21.15.37:0/3872704976 >> 172.21.15.37:0/3872704976 conn(0x7fd5240aa0a0 msgr2=0x7fd52413ccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:22:30.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.322+0000 7fd52983c640 1 -- 172.21.15.37:0/3872704976 shutdown_connections 2024-04-26T00:22:30.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.322+0000 7fd52983c640 1 -- 172.21.15.37:0/3872704976 wait complete. 2024-04-26T00:22:30.319 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:22:30.319 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-26T00:22:30.319 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-26T00:22:30.359 INFO:tasks.rados.rados.0.smithi045.stdout:449: done (0 left) 2024-04-26T00:22:30.360 INFO:tasks.rados.rados.0.smithi045.stdout:450: snap_create 2024-04-26T00:22:30.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.462+0000 7efc651dd640 10 monclient: get_monmap_and_config 2024-04-26T00:22:30.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.462+0000 7efc651dd640 10 monclient: build_initial_monmap 2024-04-26T00:22:30.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.462+0000 7efc651dd640 10 monclient: monmap: 2024-04-26T00:22:30.441 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:22:30.441 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:30.442 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:22:30.466526+0000 2024-04-26T00:22:30.442 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:22:30.466526+0000 2024-04-26T00:22:30.442 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:22:30.442 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:30.442 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:22:30.442 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:22:30.442 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:22:30.442 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:22:30.442 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:22:30.442 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:30.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.462+0000 7efc651dd640 1 Processor -- start 2024-04-26T00:22:30.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.462+0000 7efc651dd640 1 -- start start 2024-04-26T00:22:30.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.462+0000 7efc651dd640 10 monclient: init 2024-04-26T00:22:30.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.462+0000 7efc651dd640 10 monclient: _reopen_session rank -1 2024-04-26T00:22:30.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc651dd640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-04-26T00:22:30.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc651dd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7efc60133b20 0x7efc6009e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:30.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc651dd640 10 monclient(hunting): picked mon.noname-a con 0x7efc60133b20 addr v2:172.21.15.37:3300/0 2024-04-26T00:22:30.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc651dd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efc60133f00 0x7efc60134350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:30.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc651dd640 10 monclient(hunting): picked mon.noname-b con 0x7efc60133f00 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:22:30.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc651dd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efc60130770 addr v1:172.21.15.45:6789/0 2024-04-26T00:22:30.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc651dd640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc6009fe20 con 0x7efc60130770 2024-04-26T00:22:30.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc651dd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:30.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc651dd640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7efc60072d00 con 0x7efc60133b20 2024-04-26T00:22:30.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc651dd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:30.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc651dd640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7efc6009ba60 con 0x7efc60133f00 2024-04-26T00:22:30.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc651dd640 10 monclient(hunting): _renew_subs 2024-04-26T00:22:30.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc651dd640 10 monclient(hunting): authenticate will time out at 1131.772106s 2024-04-26T00:22:30.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc5ed9d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7efc60133b20 0x7efc6009e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:30.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc57fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efc60133f00 0x7efc60134350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:30.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc5ed9d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7efc60133b20 0x7efc6009e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44972/0 (socket says 172.21.15.37:44972) 2024-04-26T00:22:30.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc5ed9d640 1 -- 172.21.15.37:0/824987969 learned_addr learned my addr 172.21.15.37:0/824987969 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:22:30.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc5ed9d640 10 monclient(hunting): get_auth_request con 0x7efc60133b20 auth_method 0 2024-04-26T00:22:30.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc5ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:30.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc5ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:30.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc5ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:30.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc57fff640 10 monclient(hunting): get_auth_request con 0x7efc60133f00 auth_method 0 2024-04-26T00:22:30.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc57fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:30.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc57fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:30.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc57fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:30.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc57fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:30.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc57fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:30.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc57fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:30.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc5ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:30.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc5ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:30.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc5ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:30.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc57fff640 10 monclient(hunting): handle_auth_done global_id 34112 payload 274 2024-04-26T00:22:30.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc57fff640 1 -- 172.21.15.37:0/824987969 >> v2:172.21.15.37:3300/0 conn(0x7efc60133b20 msgr2=0x7efc6009e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:22:30.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc57fff640 1 --2- 172.21.15.37:0/824987969 >> v2:172.21.15.37:3300/0 conn(0x7efc60133b20 0x7efc6009e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:30.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc57fff640 1 -- 172.21.15.37:0/824987969 >> v1:172.21.15.45:6789/0 conn(0x7efc60130770 legacy=0x7efc60130be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:22:30.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc57fff640 10 monclient: _finish_hunting 0 2024-04-26T00:22:30.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc57fff640 1 monclient: found mon.noname-b 2024-04-26T00:22:30.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:22:30.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc57fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:22:30.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc57fff640 1 -- 172.21.15.37:0/824987969 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc6009dd10 con 0x7efc60133f00 2024-04-26T00:22:30.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.466+0000 7efc5ed9d640 1 --2- 172.21.15.37:0/824987969 >> v2:172.21.15.37:3300/0 conn(0x7efc60133b20 0x7efc6009e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:22:30.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.470+0000 7efc57fff640 1 -- 172.21.15.37:0/824987969 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efc60133f00 msgr2=0x7efc60134350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:30.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.470+0000 7efc57fff640 1 --2- 172.21.15.37:0/824987969 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efc60133f00 0x7efc60134350 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7efc480097c0 tx=0x7efc4801c110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3732c926f19dd631 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:30.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.470+0000 7efc5e59c640 1 -- 172.21.15.37:0/824987969 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc480b0070 con 0x7efc60133f00 2024-04-26T00:22:30.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.470+0000 7efc5e59c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:30.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.470+0000 7efc5e59c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:22:30.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.470+0000 7efc5e59c640 10 monclient: dump: 2024-04-26T00:22:30.449 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:30.449 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:30.449 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:30.449 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:30.449 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:30.449 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:30.450 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:30.450 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:30.450 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:30.450 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:30.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.470+0000 7efc5e59c640 10 monclient: _finish_auth 0 2024-04-26T00:22:30.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.470+0000 7efc5e59c640 10 monclient: _check_auth_tickets 2024-04-26T00:22:30.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.470+0000 7efc5e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:22:30.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.470+0000 7efc5e59c640 1 -- 172.21.15.37:0/824987969 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc480ab600 con 0x7efc60133f00 2024-04-26T00:22:30.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.470+0000 7efc5e59c640 10 monclient: handle_config config(0 keys) 2024-04-26T00:22:30.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.470+0000 7efc5e59c640 1 -- 172.21.15.37:0/824987969 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc480ab9d0 con 0x7efc60133f00 2024-04-26T00:22:30.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.470+0000 7efc651dd640 5 monclient: authenticate success, global_id 34112 2024-04-26T00:22:30.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.470+0000 7efc651dd640 10 monclient: get_monmap_and_config success 2024-04-26T00:22:30.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.470+0000 7efc651dd640 4 set_mon_vals no callback set 2024-04-26T00:22:30.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.470+0000 7efc5e59c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:30.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.470+0000 7efc5e59c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:22:30.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.470+0000 7efc5e59c640 10 monclient: dump: 2024-04-26T00:22:30.451 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:30.451 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:30.451 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:30.451 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:30.451 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:30.451 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:30.451 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:30.451 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:30.451 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:30.451 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:30.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.470+0000 7efc651dd640 10 monclient: shutdown 2024-04-26T00:22:30.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.470+0000 7efc651dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:22:30.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.470+0000 7efc651dd640 1 -- 172.21.15.37:0/824987969 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efc60133f00 msgr2=0x7efc60134350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:30.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.470+0000 7efc651dd640 1 --2- 172.21.15.37:0/824987969 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efc60133f00 0x7efc60134350 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7efc480097c0 tx=0x7efc4801c110 comp rx=0 tx=0).stop 2024-04-26T00:22:30.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.470+0000 7efc651dd640 1 -- 172.21.15.37:0/824987969 shutdown_connections 2024-04-26T00:22:30.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.470+0000 7efc651dd640 1 --2- 172.21.15.37:0/824987969 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efc60133f00 0x7efc60134350 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:30.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.470+0000 7efc651dd640 1 --2- 172.21.15.37:0/824987969 >> v2:172.21.15.37:3300/0 conn(0x7efc60133b20 0x7efc6009e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:30.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.470+0000 7efc651dd640 1 -- 172.21.15.37:0/824987969 >> 172.21.15.37:0/824987969 conn(0x7efc600a1ad0 msgr2=0x7efc600a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:22:30.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.470+0000 7efc651dd640 1 -- 172.21.15.37:0/824987969 shutdown_connections 2024-04-26T00:22:30.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.470+0000 7efc651dd640 1 -- 172.21.15.37:0/824987969 wait complete. 2024-04-26T00:22:30.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.470+0000 7efc651dd640 10 monclient: build_initial_monmap 2024-04-26T00:22:30.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.470+0000 7efc651dd640 10 monclient: monmap: 2024-04-26T00:22:30.455 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:22:30.455 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:22:30.455 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:22:30.456 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:22:30.456 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:22:30.456 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:22:30.456 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:22:30.456 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:22:30.456 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:22:30.456 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:30.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.470+0000 7efc651dd640 1 Processor -- start 2024-04-26T00:22:30.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.470+0000 7efc651dd640 1 -- start start 2024-04-26T00:22:30.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.470+0000 7efc651dd640 10 monclient: init 2024-04-26T00:22:30.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.470+0000 7efc651dd640 10 monclient: _reopen_session rank -1 2024-04-26T00:22:30.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.470+0000 7efc651dd640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:22:30.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.470+0000 7efc651dd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc60130770 0x7efc6013a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:30.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.470+0000 7efc651dd640 10 monclient(hunting): picked mon.noname-b con 0x7efc60130770 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:22:30.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.470+0000 7efc651dd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efc60133b20 0x7efc6012f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:30.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.470+0000 7efc651dd640 10 monclient(hunting): picked mon.noname-a con 0x7efc60133b20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:22:30.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc651dd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efc60133f00 0x7efc601cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:30.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc651dd640 10 monclient(hunting): picked mon.noname-c con 0x7efc60133f00 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:22:30.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc651dd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:30.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc651dd640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7efc600a1750 con 0x7efc60133b20 2024-04-26T00:22:30.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc651dd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:30.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc651dd640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7efc600a15d0 con 0x7efc60130770 2024-04-26T00:22:30.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc651dd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:30.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc651dd640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7efc600a0a70 con 0x7efc60133f00 2024-04-26T00:22:30.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc651dd640 10 monclient(hunting): _renew_subs 2024-04-26T00:22:30.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc651dd640 10 monclient(hunting): authenticate will time out at 1131.780044s 2024-04-26T00:22:30.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc57fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efc60133b20 0x7efc6012f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:30.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc5f59e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efc60133f00 0x7efc601cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:30.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc5f59e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efc60133f00 0x7efc601cf0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:53566/0 (socket says 172.21.15.37:53566) 2024-04-26T00:22:30.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc5f59e640 1 -- 172.21.15.37:0/3413724088 learned_addr learned my addr 172.21.15.37:0/3413724088 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:22:30.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc5f59e640 10 monclient(hunting): get_auth_request con 0x7efc60133f00 auth_method 0 2024-04-26T00:22:30.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc5f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:30.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc5f59e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:30.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc5f59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:30.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc57fff640 10 monclient(hunting): get_auth_request con 0x7efc60133b20 auth_method 0 2024-04-26T00:22:30.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc57fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:30.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc57fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:30.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc57fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:30.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc5f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:30.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc5f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:30.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc5f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:30.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc57fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:30.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc57fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:30.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc57fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:30.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc5ed9d640 1 --2- 172.21.15.37:0/3413724088 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc60130770 0x7efc6013a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:30.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc5ed9d640 10 monclient(hunting): get_auth_request con 0x7efc60130770 auth_method 0 2024-04-26T00:22:30.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc5ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:30.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc5ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:30.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc5ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:30.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc5f59e640 10 monclient(hunting): handle_auth_done global_id 34115 payload 824 2024-04-26T00:22:30.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc5f59e640 1 -- 172.21.15.37:0/3413724088 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc60130770 msgr2=0x7efc6013a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:30.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc5f59e640 1 --2- 172.21.15.37:0/3413724088 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc60130770 0x7efc6013a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:30.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc5f59e640 1 -- 172.21.15.37:0/3413724088 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efc60133b20 msgr2=0x7efc6012f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:30.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc5f59e640 1 --2- 172.21.15.37:0/3413724088 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efc60133b20 0x7efc6012f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:30.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc5f59e640 10 monclient: _finish_hunting 0 2024-04-26T00:22:30.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc5f59e640 1 monclient: found mon.noname-c 2024-04-26T00:22:30.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc5f59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:22:30.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc5f59e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:22:30.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc5f59e640 1 -- 172.21.15.37:0/3413724088 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc6011bae0 con 0x7efc60133f00 2024-04-26T00:22:30.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc5f59e640 1 -- 172.21.15.37:0/3413724088 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efc60133f00 msgr2=0x7efc601cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:30.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc5f59e640 1 --2- 172.21.15.37:0/3413724088 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efc60133f00 0x7efc601cf0d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7efc5800c780 tx=0x7efc5800cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:30.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc577fe640 1 -- 172.21.15.37:0/3413724088 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc58017070 con 0x7efc60133f00 2024-04-26T00:22:30.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:30.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc577fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:22:30.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc577fe640 10 monclient: dump: 2024-04-26T00:22:30.460 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:30.460 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:30.460 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:30.460 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:30.460 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:30.460 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:30.460 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:30.460 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:30.461 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:30.461 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:30.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc577fe640 10 monclient: _finish_auth 0 2024-04-26T00:22:30.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc577fe640 10 monclient: _check_auth_tickets 2024-04-26T00:22:30.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:22:30.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc577fe640 1 -- 172.21.15.37:0/3413724088 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc58005070 con 0x7efc60133f00 2024-04-26T00:22:30.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc651dd640 5 monclient: authenticate success, global_id 34115 2024-04-26T00:22:30.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc651dd640 10 monclient: _renew_subs 2024-04-26T00:22:30.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc651dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:22:30.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc651dd640 1 -- 172.21.15.37:0/3413724088 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc6011dc00 con 0x7efc60133f00 2024-04-26T00:22:30.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc577fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:22:30.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc577fe640 1 -- 172.21.15.37:0/3413724088 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc58005460 con 0x7efc60133f00 2024-04-26T00:22:30.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:30.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:22:30.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc577fe640 10 monclient: dump: 2024-04-26T00:22:30.461 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:30.462 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:30.462 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:30.462 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:30.462 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:30.462 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:30.462 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:30.462 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:30.462 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:30.462 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:30.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc5dd9b640 4 set_mon_vals no callback set 2024-04-26T00:22:30.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc651dd640 10 monclient: _renew_subs 2024-04-26T00:22:30.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc651dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:22:30.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc651dd640 1 -- 172.21.15.37:0/3413724088 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efc6011c730 con 0x7efc60133f00 2024-04-26T00:22:30.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc651dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:30.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc651dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:30.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc651dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:22:30.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.478+0000 7efc577fe640 1 -- 172.21.15.37:0/3413724088 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7efc5801a070 con 0x7efc60133f00 2024-04-26T00:22:30.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.474+0000 7efc651dd640 1 -- 172.21.15.37:0/3413724088 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc60066f40 con 0x7efc60133f00 2024-04-26T00:22:30.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.478+0000 7efc577fe640 1 --2- 172.21.15.37:0/3413724088 >> v2:172.21.15.45:6812/2379918623 conn(0x7efc40035e80 0x7efc40038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:30.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.478+0000 7efc577fe640 1 -- 172.21.15.37:0/3413724088 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(110..110 src has 1..110) ==== 3991+0+0 (secure 0 0 0) 0x7efc580481e0 con 0x7efc60133f00 2024-04-26T00:22:30.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.478+0000 7efc5ed9d640 1 --2- 172.21.15.37:0/3413724088 >> v2:172.21.15.45:6812/2379918623 conn(0x7efc40035e80 0x7efc40038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:30.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.478+0000 7efc5ed9d640 10 monclient: get_auth_request con 0x7efc40035e80 auth_method 0 2024-04-26T00:22:30.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.478+0000 7efc5ed9d640 1 --2- 172.21.15.37:0/3413724088 >> v2:172.21.15.45:6812/2379918623 conn(0x7efc40035e80 0x7efc40038340 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7efc5000b8d0 tx=0x7efc5000b450 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:30.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.482+0000 7efc577fe640 1 -- 172.21.15.37:0/3413724088 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7efc6011bae0 con 0x7efc60133f00 2024-04-26T00:22:30.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.482+0000 7efc577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:30.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.482+0000 7efc577fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:22:30.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.702+0000 7efc651dd640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-26T00:22:30.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.702+0000 7efc651dd640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-26T00:22:30.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.702+0000 7efc651dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:22:30.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.702+0000 7efc651dd640 1 -- 172.21.15.37:0/3413724088 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7efc6006b9a0 con 0x7efc60133f00 2024-04-26T00:22:30.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.706+0000 7efc577fe640 1 -- 172.21.15.37:0/3413724088 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7efc58046ef0 con 0x7efc60133f00 2024-04-26T00:22:30.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.706+0000 7efc577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-26T00:22:30.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.706+0000 7efc577fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:22:30.686 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.706+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:22:30.686 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.706+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=9,config=2,health=12,kv=9,logm=68} crc {auth=484207421,config=692619927,health=2157093694,kv=966672740,logm=2644585761}) 2024-04-26T00:22:30.686 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.706+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=9,config=2,health=12,kv=9,logm=68} crc {auth=254430421,config=692619927,health=2157093694,kv=2491615610,logm=2701017657}) 2024-04-26T00:22:30.686 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.706+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:22:30.686 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.706+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=9,config=2,health=12,kv=9,logm=68} crc {auth=484207421,config=692619927,health=2157093694,kv=966672740,logm=2644585761}) 2024-04-26T00:22:30.687 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.706+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=9,config=2,health=12,kv=10,logm=67} crc {auth=2936125017,config=692619927,health=2157093694,kv=3485254834,logm=1986274163}) 2024-04-26T00:22:30.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.710+0000 7efc651dd640 1 -- 172.21.15.37:0/3413724088 >> v2:172.21.15.45:6812/2379918623 conn(0x7efc40035e80 msgr2=0x7efc40038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:30.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.710+0000 7efc651dd640 1 --2- 172.21.15.37:0/3413724088 >> v2:172.21.15.45:6812/2379918623 conn(0x7efc40035e80 0x7efc40038340 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7efc5000b8d0 tx=0x7efc5000b450 comp rx=0 tx=0).stop 2024-04-26T00:22:30.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.710+0000 7efc651dd640 10 monclient: shutdown 2024-04-26T00:22:30.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.710+0000 7efc651dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:22:30.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.710+0000 7efc651dd640 1 -- 172.21.15.37:0/3413724088 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efc60133f00 msgr2=0x7efc601cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:30.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.710+0000 7efc651dd640 1 --2- 172.21.15.37:0/3413724088 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efc60133f00 0x7efc601cf0d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7efc5800c780 tx=0x7efc5800cc40 comp rx=0 tx=0).stop 2024-04-26T00:22:30.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.710+0000 7efc651dd640 1 -- 172.21.15.37:0/3413724088 shutdown_connections 2024-04-26T00:22:30.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.710+0000 7efc651dd640 1 --2- 172.21.15.37:0/3413724088 >> v2:172.21.15.45:6812/2379918623 conn(0x7efc40035e80 0x7efc40038340 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:30.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.710+0000 7efc651dd640 1 --2- 172.21.15.37:0/3413724088 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efc60133f00 0x7efc601cf0d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:30.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.710+0000 7efc651dd640 1 --2- 172.21.15.37:0/3413724088 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efc60133b20 0x7efc6012f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:30.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.710+0000 7efc651dd640 1 --2- 172.21.15.37:0/3413724088 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc60130770 0x7efc6013a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:30.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.710+0000 7efc651dd640 1 -- 172.21.15.37:0/3413724088 >> 172.21.15.37:0/3413724088 conn(0x7efc600a1ad0 msgr2=0x7efc600a4830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:22:30.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.710+0000 7efc651dd640 1 -- 172.21.15.37:0/3413724088 shutdown_connections 2024-04-26T00:22:30.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:30.710+0000 7efc651dd640 1 -- 172.21.15.37:0/3413724088 wait complete. 2024-04-26T00:22:30.690 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.710+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:22:30.690 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.710+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=966245268}) 2024-04-26T00:22:30.690 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.710+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3848442039}) 2024-04-26T00:22:30.690 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.710+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:22:30.690 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.710+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=966245268}) 2024-04-26T00:22:30.690 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.710+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2657207037}) 2024-04-26T00:22:30.697 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.718+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:22:30.697 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.718+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=695409333}) 2024-04-26T00:22:30.697 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.718+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3237490383}) 2024-04-26T00:22:30.697 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.718+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:22:30.697 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.718+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=695409333}) 2024-04-26T00:22:30.697 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.718+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2168798247}) 2024-04-26T00:22:30.699 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.722+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:22:30.699 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.722+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3893460179}) 2024-04-26T00:22:30.699 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.722+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3494818968}) 2024-04-26T00:22:30.699 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.722+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:22:30.699 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.722+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3893460179}) 2024-04-26T00:22:30.699 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.722+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1061716356}) 2024-04-26T00:22:30.702 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.722+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:22:30.702 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.722+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1572625265}) 2024-04-26T00:22:30.702 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.722+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4286667434}) 2024-04-26T00:22:30.702 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.722+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:22:30.703 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.722+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1572625265}) 2024-04-26T00:22:30.703 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.722+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2782670302}) 2024-04-26T00:22:30.704 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-26T00:22:30.705 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.726+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:22:30.705 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.726+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3487248681}) 2024-04-26T00:22:30.705 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.726+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1129358360}) 2024-04-26T00:22:30.705 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.726+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:22:30.705 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.726+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3487248681}) 2024-04-26T00:22:30.706 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.726+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2420122332}) 2024-04-26T00:22:30.708 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.730+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:22:30.708 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.730+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1916457246}) 2024-04-26T00:22:30.708 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.730+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=308292194}) 2024-04-26T00:22:30.708 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.730+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:22:30.708 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.730+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1916457246}) 2024-04-26T00:22:30.708 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.730+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2723369837}) 2024-04-26T00:22:30.712 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.734+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:22:30.712 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.734+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=39,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=8,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=14,osdmap=6} crc {logm=2605907001,mdsmap=2840952586,mgr=2644196481,mgr_metadata=3686404438,mgrstat=1412815597,mon_config_key=1394720687,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=1388750497,osd_snap=775020172,osdmap=2629668331}) 2024-04-26T00:22:30.712 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.734+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=38,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=15,osdmap=4} crc {logm=1547433599,mdsmap=1401145748,mgr=2644196481,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=896721941,mon_config_key=4061063117,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=1388750498,osd_snap=436527572,osdmap=3748732073}) 2024-04-26T00:22:30.712 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.734+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:22:30.712 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.734+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=39,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=8,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=14,osdmap=6} crc {logm=2605907001,mdsmap=2840952586,mgr=2644196481,mgr_metadata=3686404438,mgrstat=1412815597,mon_config_key=1394720687,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=1388750497,osd_snap=775020172,osdmap=2629668331}) 2024-04-26T00:22:30.712 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.734+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=39,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=15,osdmap=4} crc {logm=1859091636,mdsmap=2840952586,mgr=3332899993,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=298308717,mon_config_key=4061063117,monmap=3876866194,osd_metadata=839604482,osd_pg_creating=1388750497,osd_snap=221961225,osdmap=3748732073}) 2024-04-26T00:22:30.715 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.738+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:22:30.715 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.738+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3383629443}) 2024-04-26T00:22:30.715 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.738+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4101383072}) 2024-04-26T00:22:30.715 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.738+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:22:30.715 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.738+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3383629443}) 2024-04-26T00:22:30.715 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.738+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2325352102}) 2024-04-26T00:22:30.723 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.746+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:22:30.723 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.746+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=72} crc {osdmap=317276916}) 2024-04-26T00:22:30.724 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.746+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=71} crc {osdmap=3917117145}) 2024-04-26T00:22:30.724 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.746+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:22:30.724 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.746+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=72} crc {osdmap=317276916}) 2024-04-26T00:22:30.724 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:22:30.746+0000 7f4a941da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=70} crc {osdmap=1433883643}) 2024-04-26T00:22:31.370 INFO:tasks.rados.rados.0.smithi045.stdout:450: done (0 left) 2024-04-26T00:22:31.371 INFO:tasks.rados.rados.0.smithi045.stdout:451: read oid 17 snap 43 2024-04-26T00:22:31.371 INFO:tasks.rados.rados.0.smithi045.stdout:451: expect (ObjNum 142 snap 31 seq_num 142) 2024-04-26T00:22:31.371 INFO:tasks.rados.rados.0.smithi045.stdout:452: rollback oid 37 current snap is 50 2024-04-26T00:22:31.371 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 37 to 46 2024-04-26T00:22:31.371 INFO:tasks.rados.rados.0.smithi045.stdout:453: read oid 36 snap -1 2024-04-26T00:22:31.371 INFO:tasks.rados.rados.0.smithi045.stdout:453: expect deleted 2024-04-26T00:22:31.371 INFO:tasks.rados.rados.0.smithi045.stdout:454: copy_from oid 41 from oid 43 current snap is 50 2024-04-26T00:22:31.371 INFO:tasks.rados.rados.0.smithi045.stdout:455: read oid 28 snap -1 2024-04-26T00:22:31.371 INFO:tasks.rados.rados.0.smithi045.stdout:455: expect (ObjNum 180 snap 47 seq_num 180) 2024-04-26T00:22:31.372 INFO:tasks.rados.rados.0.smithi045.stdout:456: delete oid 31 current snap is 50 2024-04-26T00:22:31.381 INFO:tasks.rados.rados.0.smithi045.stdout:451: expect (ObjNum 142 snap 31 seq_num 142) 2024-04-26T00:22:31.526 INFO:tasks.rados.rados.0.smithi045.stdout:452: finishing rollback tid 0 to smithi04526579-37 2024-04-26T00:22:31.527 INFO:tasks.rados.rados.0.smithi045.stdout:455: expect (ObjNum 180 snap 47 seq_num 180) 2024-04-26T00:22:31.673 INFO:tasks.rados.rados.0.smithi045.stdout:452: finishing rollback tid 1 to smithi04526579-37 2024-04-26T00:22:31.674 INFO:tasks.rados.rados.0.smithi045.stdout:452: finishing rollback tid 2 to smithi04526579-37 2024-04-26T00:22:31.674 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 257 (ObjNum 168 snap 40 seq_num 168) dirty exists 2024-04-26T00:22:31.674 INFO:tasks.rados.rados.0.smithi045.stdout:454: finishing copy_from racing read to smithi04526579-41 2024-04-26T00:22:31.674 INFO:tasks.rados.rados.0.smithi045.stdout:454: finishing copy_from to smithi04526579-41 2024-04-26T00:22:31.674 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 142 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-04-26T00:22:31.674 INFO:tasks.rados.rados.0.smithi045.stdout:451: done (5 left) 2024-04-26T00:22:31.674 INFO:tasks.rados.rados.0.smithi045.stdout:452: done (4 left) 2024-04-26T00:22:31.674 INFO:tasks.rados.rados.0.smithi045.stdout:453: done (3 left) 2024-04-26T00:22:31.674 INFO:tasks.rados.rados.0.smithi045.stdout:454: done (2 left) 2024-04-26T00:22:31.674 INFO:tasks.rados.rados.0.smithi045.stdout:455: done (1 left) 2024-04-26T00:22:31.674 INFO:tasks.rados.rados.0.smithi045.stdout:456: done (0 left) 2024-04-26T00:22:31.674 INFO:tasks.rados.rados.0.smithi045.stdout:457: read oid 2 snap -1 2024-04-26T00:22:31.675 INFO:tasks.rados.rados.0.smithi045.stdout:457: expect (ObjNum 189 snap 49 seq_num 189) 2024-04-26T00:22:31.675 INFO:tasks.rados.rados.0.smithi045.stdout:458: delete oid 41 current snap is 50 2024-04-26T00:22:31.677 INFO:tasks.rados.rados.0.smithi045.stdout:458: done (1 left) 2024-04-26T00:22:31.677 INFO:tasks.rados.rados.0.smithi045.stdout:459: write (excl) oid 4 current snap is 50 2024-04-26T00:22:31.677 INFO:tasks.rados.rados.0.smithi045.stdout:459: seq_num 195 ranges {477365=782380,1927871=768727,3122307=155058} 2024-04-26T00:22:31.697 INFO:tasks.rados.rados.0.smithi045.stdout:459: writing smithi04526579-4 from 477365 to 1259745 tid 1 2024-04-26T00:22:31.705 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:22:31.705 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:22:31.721 INFO:tasks.rados.rados.0.smithi045.stdout:459: writing smithi04526579-4 from 1927871 to 2696598 tid 2 2024-04-26T00:22:31.729 INFO:tasks.rados.rados.0.smithi045.stdout:459: writing smithi04526579-4 from 3122307 to 3277365 tid 3 2024-04-26T00:22:31.729 INFO:tasks.rados.rados.0.smithi045.stdout:460: copy_from oid 16 from oid 29 current snap is 50 2024-04-26T00:22:31.729 INFO:tasks.rados.rados.0.smithi045.stdout:457: expect (ObjNum 189 snap 49 seq_num 189) 2024-04-26T00:22:31.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.862+0000 7f4d11463640 10 monclient: get_monmap_and_config 2024-04-26T00:22:31.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.862+0000 7f4d11463640 10 monclient: build_initial_monmap 2024-04-26T00:22:31.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.862+0000 7f4d11463640 10 monclient: monmap: 2024-04-26T00:22:31.840 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:22:31.840 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:31.841 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:22:31.865364+0000 2024-04-26T00:22:31.841 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:22:31.865364+0000 2024-04-26T00:22:31.841 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:22:31.841 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:31.841 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:22:31.841 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:22:31.841 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:22:31.841 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:22:31.841 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:22:31.841 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:31.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.862+0000 7f4d11463640 1 Processor -- start 2024-04-26T00:22:31.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.862+0000 7f4d11463640 1 -- start start 2024-04-26T00:22:31.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.862+0000 7f4d11463640 10 monclient: init 2024-04-26T00:22:31.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.862+0000 7f4d11463640 10 monclient: _reopen_session rank -1 2024-04-26T00:22:31.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.862+0000 7f4d11463640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-04-26T00:22:31.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.862+0000 7f4d11463640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4d0c071fc0 addr v1:172.21.15.37:6789/0 2024-04-26T00:22:31.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.862+0000 7f4d11463640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f4d0c0723a0 0x7f4d0c072810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:31.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.862+0000 7f4d11463640 10 monclient(hunting): picked mon.noname-a con 0x7f4d0c0723a0 addr v2:172.21.15.37:3300/0 2024-04-26T00:22:31.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.862+0000 7f4d11463640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4d0c13c340 addr v1:172.21.15.45:6789/0 2024-04-26T00:22:31.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.862+0000 7f4d11463640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d0c073200 con 0x7f4d0c071fc0 2024-04-26T00:22:31.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.862+0000 7f4d11463640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d0c0a0c20 con 0x7f4d0c13c340 2024-04-26T00:22:31.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.862+0000 7f4d11463640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:31.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.862+0000 7f4d11463640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f4d0c0a0f00 con 0x7f4d0c0723a0 2024-04-26T00:22:31.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.862+0000 7f4d11463640 10 monclient(hunting): _renew_subs 2024-04-26T00:22:31.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.862+0000 7f4d11463640 10 monclient(hunting): authenticate will time out at 1133.171067s 2024-04-26T00:22:31.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.862+0000 7f4d0a7fc640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f4d0c0723a0 0x7f4d0c072810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:31.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.862+0000 7f4d0affd640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f4d0c071fc0 0x7f4d0c137240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:51972/0 (socket says 172.21.15.37:51972) 2024-04-26T00:22:31.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.862+0000 7f4d0affd640 1 -- 172.21.15.37:0/3357040800 learned_addr learned my addr 172.21.15.37:0/3357040800 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:22:31.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.862+0000 7f4d0a7fc640 10 monclient(hunting): get_auth_request con 0x7f4d0c0723a0 auth_method 0 2024-04-26T00:22:31.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.862+0000 7f4d0a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:31.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.862+0000 7f4d0a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:31.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.862+0000 7f4d0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:31.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.862+0000 7f4d0a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:31.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.862+0000 7f4d0a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:31.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.862+0000 7f4d0a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:31.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.862+0000 7f4d0a7fc640 10 monclient(hunting): handle_auth_done global_id 4656 payload 274 2024-04-26T00:22:31.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.862+0000 7f4d0a7fc640 1 -- 172.21.15.37:0/3357040800 >> v1:172.21.15.45:6789/0 conn(0x7f4d0c13c340 legacy=0x7f4d0c13e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:22:31.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.862+0000 7f4d0a7fc640 1 -- 172.21.15.37:0/3357040800 >> v1:172.21.15.37:6789/0 conn(0x7f4d0c071fc0 legacy=0x7f4d0c137240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:22:31.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.862+0000 7f4d0a7fc640 10 monclient: _finish_hunting 0 2024-04-26T00:22:31.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.862+0000 7f4d0a7fc640 1 monclient: found mon.noname-a 2024-04-26T00:22:31.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.862+0000 7f4d0a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:22:31.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.862+0000 7f4d0a7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:22:31.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.862+0000 7f4d0a7fc640 1 -- 172.21.15.37:0/3357040800 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d0c137950 con 0x7f4d0c0723a0 2024-04-26T00:22:31.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.866+0000 7f4d0a7fc640 1 -- 172.21.15.37:0/3357040800 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4d0c0723a0 msgr2=0x7f4d0c072810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:31.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.866+0000 7f4d0a7fc640 1 --2- 172.21.15.37:0/3357040800 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4d0c0723a0 0x7f4d0c072810 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4d000097c0 tx=0x7f4d0001c2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eafdb0793ea6b127 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:31.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.866+0000 7f4d09ffb640 1 -- 172.21.15.37:0/3357040800 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d000b0070 con 0x7f4d0c0723a0 2024-04-26T00:22:31.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d09ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:31.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d09ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:22:31.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d09ffb640 10 monclient: dump: 2024-04-26T00:22:31.848 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:31.848 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:31.848 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:31.848 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:31.848 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:31.848 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:31.848 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:31.848 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:31.848 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:31.848 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:31.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d09ffb640 10 monclient: _finish_auth 0 2024-04-26T00:22:31.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d09ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:22:31.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:22:31.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d09ffb640 1 -- 172.21.15.37:0/3357040800 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d000ab600 con 0x7f4d0c0723a0 2024-04-26T00:22:31.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d11463640 5 monclient: authenticate success, global_id 4656 2024-04-26T00:22:31.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d11463640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:22:31.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d09ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:22:31.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d11463640 10 monclient: get_monmap_and_config success 2024-04-26T00:22:31.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d11463640 4 set_mon_vals no callback set 2024-04-26T00:22:31.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d09ffb640 1 -- 172.21.15.37:0/3357040800 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d000ab9d0 con 0x7f4d0c0723a0 2024-04-26T00:22:31.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d09ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:31.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d09ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:22:31.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d09ffb640 10 monclient: dump: 2024-04-26T00:22:31.849 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:31.849 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:31.849 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:31.849 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:31.849 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:31.849 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:31.850 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:31.850 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:31.850 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:31.850 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:31.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d11463640 10 monclient: shutdown 2024-04-26T00:22:31.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d11463640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:22:31.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d11463640 1 -- 172.21.15.37:0/3357040800 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4d0c0723a0 msgr2=0x7f4d0c072810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:31.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d11463640 1 --2- 172.21.15.37:0/3357040800 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4d0c0723a0 0x7f4d0c072810 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4d000097c0 tx=0x7f4d0001c2e0 comp rx=0 tx=0).stop 2024-04-26T00:22:31.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d11463640 1 -- 172.21.15.37:0/3357040800 shutdown_connections 2024-04-26T00:22:31.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d11463640 1 --2- 172.21.15.37:0/3357040800 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4d0c0723a0 0x7f4d0c072810 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:31.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d11463640 1 -- 172.21.15.37:0/3357040800 >> 172.21.15.37:0/3357040800 conn(0x7f4d0c0a1ad0 msgr2=0x7f4d0c0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:22:31.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d11463640 1 -- 172.21.15.37:0/3357040800 shutdown_connections 2024-04-26T00:22:31.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d11463640 1 -- 172.21.15.37:0/3357040800 wait complete. 2024-04-26T00:22:31.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d11463640 10 monclient: build_initial_monmap 2024-04-26T00:22:31.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d11463640 10 monclient: monmap: 2024-04-26T00:22:31.855 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:22:31.855 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:22:31.855 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:22:31.855 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:22:31.855 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:22:31.855 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:22:31.855 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:22:31.855 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:22:31.855 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:22:31.855 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:31.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d11463640 1 Processor -- start 2024-04-26T00:22:31.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d11463640 1 -- start start 2024-04-26T00:22:31.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d11463640 10 monclient: init 2024-04-26T00:22:31.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d11463640 10 monclient: _reopen_session rank -1 2024-04-26T00:22:31.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d11463640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:22:31.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d11463640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d0c071fc0 0x7f4d0c1336a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:31.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d11463640 10 monclient(hunting): picked mon.noname-b con 0x7f4d0c071fc0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:22:31.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d11463640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d0c0723a0 0x7f4d0c1cee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:31.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d11463640 10 monclient(hunting): picked mon.noname-c con 0x7f4d0c0723a0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:22:31.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d11463640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4d0c13c340 0x7f4d0c1cf360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:31.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d11463640 10 monclient(hunting): picked mon.noname-a con 0x7f4d0c13c340 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:22:31.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d11463640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:31.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d11463640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f4d0c142a40 con 0x7f4d0c13c340 2024-04-26T00:22:31.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d11463640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:31.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d11463640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4d0c1428c0 con 0x7f4d0c071fc0 2024-04-26T00:22:31.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d11463640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:31.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d11463640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4d0c142bc0 con 0x7f4d0c0723a0 2024-04-26T00:22:31.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d11463640 10 monclient(hunting): _renew_subs 2024-04-26T00:22:31.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d11463640 10 monclient(hunting): authenticate will time out at 1133.178238s 2024-04-26T00:22:31.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d0a7fc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d0c0723a0 0x7f4d0c1cee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:31.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d0b7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4d0c13c340 0x7f4d0c1cf360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:31.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d0a7fc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d0c0723a0 0x7f4d0c1cee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:53578/0 (socket says 172.21.15.37:53578) 2024-04-26T00:22:31.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d0a7fc640 1 -- 172.21.15.37:0/2183206712 learned_addr learned my addr 172.21.15.37:0/2183206712 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:22:31.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d0a7fc640 10 monclient(hunting): get_auth_request con 0x7f4d0c0723a0 auth_method 0 2024-04-26T00:22:31.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d0a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:31.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d0a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:31.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:31.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d0b7fe640 10 monclient(hunting): get_auth_request con 0x7f4d0c13c340 auth_method 0 2024-04-26T00:22:31.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d0b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:31.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d0b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:31.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:31.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d0a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:31.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d0a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:31.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d0a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:31.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d0b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:31.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d0b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:31.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d0b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:31.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d0affd640 1 --2- 172.21.15.37:0/2183206712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d0c071fc0 0x7f4d0c1336a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:31.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d0affd640 10 monclient(hunting): get_auth_request con 0x7f4d0c071fc0 auth_method 0 2024-04-26T00:22:31.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d0affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:31.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d0affd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:31.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d0affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:31.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d0affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:31.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d0affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:31.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d0affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:31.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d0a7fc640 10 monclient(hunting): handle_auth_done global_id 34118 payload 824 2024-04-26T00:22:31.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d0a7fc640 1 -- 172.21.15.37:0/2183206712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d0c071fc0 msgr2=0x7f4d0c1336a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:31.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d0a7fc640 1 --2- 172.21.15.37:0/2183206712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d0c071fc0 0x7f4d0c1336a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:31.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d0a7fc640 1 -- 172.21.15.37:0/2183206712 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4d0c13c340 msgr2=0x7f4d0c1cf360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:31.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d0a7fc640 1 --2- 172.21.15.37:0/2183206712 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4d0c13c340 0x7f4d0c1cf360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:31.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d0a7fc640 10 monclient: _finish_hunting 0 2024-04-26T00:22:31.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d0a7fc640 1 monclient: found mon.noname-c 2024-04-26T00:22:31.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d0a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:22:31.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d0a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:22:31.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d0a7fc640 1 -- 172.21.15.37:0/2183206712 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d0c13a300 con 0x7f4d0c0723a0 2024-04-26T00:22:31.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.870+0000 7f4d0b7fe640 1 --2- 172.21.15.37:0/2183206712 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4d0c13c340 0x7f4d0c1cf360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:22:31.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.874+0000 7f4d0a7fc640 1 -- 172.21.15.37:0/2183206712 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d0c0723a0 msgr2=0x7f4d0c1cee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:31.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.874+0000 7f4d0a7fc640 1 --2- 172.21.15.37:0/2183206712 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d0c0723a0 0x7f4d0c1cee20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4d000ae040 tx=0x7f4d0001ab20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:31.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.874+0000 7f4cf37fe640 1 -- 172.21.15.37:0/2183206712 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d000b0050 con 0x7f4d0c0723a0 2024-04-26T00:22:31.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.874+0000 7f4cf37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:31.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.874+0000 7f4cf37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:22:31.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.874+0000 7f4cf37fe640 10 monclient: dump: 2024-04-26T00:22:31.859 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:31.860 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:31.860 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:31.860 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:31.860 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:31.860 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:31.860 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:31.860 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:31.860 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:31.860 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:31.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.874+0000 7f4cf37fe640 10 monclient: _finish_auth 0 2024-04-26T00:22:31.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.874+0000 7f4cf37fe640 10 monclient: _check_auth_tickets 2024-04-26T00:22:31.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.874+0000 7f4cf37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:22:31.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.874+0000 7f4d11463640 5 monclient: authenticate success, global_id 34118 2024-04-26T00:22:31.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.874+0000 7f4cf37fe640 1 -- 172.21.15.37:0/2183206712 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d000bc070 con 0x7f4d0c0723a0 2024-04-26T00:22:31.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.874+0000 7f4cf37fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:22:31.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.874+0000 7f4cf37fe640 1 -- 172.21.15.37:0/2183206712 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d000b5c80 con 0x7f4d0c0723a0 2024-04-26T00:22:31.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.874+0000 7f4d11463640 10 monclient: _renew_subs 2024-04-26T00:22:31.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.874+0000 7f4d11463640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:22:31.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.874+0000 7f4d11463640 1 -- 172.21.15.37:0/2183206712 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d0c13a020 con 0x7f4d0c0723a0 2024-04-26T00:22:31.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.874+0000 7f4cf37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:31.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.874+0000 7f4cf37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:22:31.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.874+0000 7f4d097fa640 4 set_mon_vals no callback set 2024-04-26T00:22:31.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.874+0000 7f4cf37fe640 10 monclient: dump: 2024-04-26T00:22:31.861 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:31.861 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:31.861 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:31.861 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:31.861 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:31.861 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:31.861 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:31.862 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:31.862 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:31.862 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:31.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.874+0000 7f4d11463640 10 monclient: _renew_subs 2024-04-26T00:22:31.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.874+0000 7f4d11463640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:22:31.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.874+0000 7f4d11463640 1 -- 172.21.15.37:0/2183206712 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d0c13b370 con 0x7f4d0c0723a0 2024-04-26T00:22:31.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.874+0000 7f4d11463640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:31.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.874+0000 7f4d11463640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:31.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.874+0000 7f4d11463640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:22:31.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.874+0000 7f4d11463640 1 -- 172.21.15.37:0/2183206712 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d0c066f40 con 0x7f4d0c0723a0 2024-04-26T00:22:31.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.874+0000 7f4cf37fe640 1 -- 172.21.15.37:0/2183206712 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f4d000ab4c0 con 0x7f4d0c0723a0 2024-04-26T00:22:31.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.874+0000 7f4cf37fe640 1 --2- 172.21.15.37:0/2183206712 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4cf8035eb0 0x7f4cf8038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:31.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.874+0000 7f4cf37fe640 1 -- 172.21.15.37:0/2183206712 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(111..111 src has 1..111) ==== 3963+0+0 (secure 0 0 0) 0x7f4d000e34f0 con 0x7f4d0c0723a0 2024-04-26T00:22:31.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.874+0000 7f4d0affd640 1 --2- 172.21.15.37:0/2183206712 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4cf8035eb0 0x7f4cf8038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:31.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.874+0000 7f4d0affd640 10 monclient: get_auth_request con 0x7f4cf8035eb0 auth_method 0 2024-04-26T00:22:31.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.878+0000 7f4d0affd640 1 --2- 172.21.15.37:0/2183206712 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4cf8035eb0 0x7f4cf8038370 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc005630 tx=0x7f4cfc0055c0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:31.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.878+0000 7f4cf37fe640 1 -- 172.21.15.37:0/2183206712 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f4d0c13a300 con 0x7f4d0c0723a0 2024-04-26T00:22:31.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.878+0000 7f4cf37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:31.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:31.878+0000 7f4cf37fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:22:31.975 INFO:tasks.rados.rados.0.smithi045.stdout:459: finishing write tid 1 to smithi04526579-4 2024-04-26T00:22:31.975 INFO:tasks.rados.rados.0.smithi045.stdout:459: finishing write tid 2 to smithi04526579-4 2024-04-26T00:22:31.975 INFO:tasks.rados.rados.0.smithi045.stdout:457: done (2 left) 2024-04-26T00:22:31.975 INFO:tasks.rados.rados.0.smithi045.stdout:461: rollback oid 11 current snap is 50 2024-04-26T00:22:31.975 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 11 to 41 2024-04-26T00:22:31.975 INFO:tasks.rados.rados.0.smithi045.stdout:459: finishing write tid 3 to smithi04526579-4 2024-04-26T00:22:31.975 INFO:tasks.rados.rados.0.smithi045.stdout:459: finishing write tid 4 to smithi04526579-4 2024-04-26T00:22:31.975 INFO:tasks.rados.rados.0.smithi045.stdout:459: finishing write tid 5 to smithi04526579-4 2024-04-26T00:22:31.975 INFO:tasks.rados.rados.0.smithi045.stdout:459: oid 4 updating version 0 to 206 2024-04-26T00:22:31.975 INFO:tasks.rados.rados.0.smithi045.stdout:459: oid 4 version 206 is already newer than 205 2024-04-26T00:22:31.975 INFO:tasks.rados.rados.0.smithi045.stdout:459: oid 4 version 206 is already newer than 204 2024-04-26T00:22:31.975 INFO:tasks.rados.rados.0.smithi045.stdout:459: oid 4 updating version 206 to 207 2024-04-26T00:22:31.975 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 207 (ObjNum 195 snap 50 seq_num 195) dirty exists 2024-04-26T00:22:31.976 INFO:tasks.rados.rados.0.smithi045.stdout:459: left oid 4 (ObjNum 195 snap 50 seq_num 195) 2024-04-26T00:22:31.976 INFO:tasks.rados.rados.0.smithi045.stdout:460: finishing copy_from racing read to smithi04526579-16 2024-04-26T00:22:31.976 INFO:tasks.rados.rados.0.smithi045.stdout:460: finishing copy_from to smithi04526579-16 2024-04-26T00:22:31.976 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 58 (ObjNum 120 snap 28 seq_num 120) dirty exists 2024-04-26T00:22:31.976 INFO:tasks.rados.rados.0.smithi045.stdout:459: done (2 left) 2024-04-26T00:22:31.976 INFO:tasks.rados.rados.0.smithi045.stdout:460: done (1 left) 2024-04-26T00:22:31.976 INFO:tasks.rados.rados.0.smithi045.stdout:462: read oid 48 snap -1 2024-04-26T00:22:31.976 INFO:tasks.rados.rados.0.smithi045.stdout:462: expect (ObjNum 171 snap 45 seq_num 171) 2024-04-26T00:22:31.976 INFO:tasks.rados.rados.0.smithi045.stdout:463: write oid 32 current snap is 50 2024-04-26T00:22:31.976 INFO:tasks.rados.rados.0.smithi045.stdout:463: seq_num 196 ranges {494172=457629,1586353=512385,2749730=544442} 2024-04-26T00:22:31.991 INFO:tasks.rados.rados.0.smithi045.stdout:463: writing smithi04526579-32 from 494172 to 951801 tid 1 2024-04-26T00:22:32.009 INFO:tasks.rados.rados.0.smithi045.stdout:463: writing smithi04526579-32 from 1586353 to 2098738 tid 2 2024-04-26T00:22:32.028 INFO:tasks.rados.rados.0.smithi045.stdout:463: writing smithi04526579-32 from 2749730 to 3294172 tid 3 2024-04-26T00:22:32.029 INFO:tasks.rados.rados.0.smithi045.stdout:464: copy_from oid 46 from oid 6 current snap is 50 2024-04-26T00:22:32.029 INFO:tasks.rados.rados.0.smithi045.stdout:462: expect (ObjNum 171 snap 45 seq_num 171) 2024-04-26T00:22:32.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.110+0000 7f4d11463640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:22:32.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.110+0000 7f4d11463640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:22:32.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.110+0000 7f4d11463640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:22:32.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.110+0000 7f4d11463640 1 -- 172.21.15.37:0/2183206712 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4d0c06b9a0 con 0x7f4d0c0723a0 2024-04-26T00:22:32.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.114+0000 7f4cf37fe640 1 -- 172.21.15.37:0/2183206712 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f4d00015ff0 con 0x7f4d0c0723a0 2024-04-26T00:22:32.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.114+0000 7f4cf37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:22:32.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.114+0000 7f4cf37fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:22:32.091 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":24,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:22:32.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.118+0000 7f4d11463640 1 -- 172.21.15.37:0/2183206712 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4cf8035eb0 msgr2=0x7f4cf8038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:32.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.118+0000 7f4d11463640 1 --2- 172.21.15.37:0/2183206712 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4cf8035eb0 0x7f4cf8038370 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc005630 tx=0x7f4cfc0055c0 comp rx=0 tx=0).stop 2024-04-26T00:22:32.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.118+0000 7f4d11463640 10 monclient: shutdown 2024-04-26T00:22:32.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.118+0000 7f4d11463640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:22:32.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.118+0000 7f4d11463640 1 -- 172.21.15.37:0/2183206712 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d0c0723a0 msgr2=0x7f4d0c1cee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:32.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.118+0000 7f4d11463640 1 --2- 172.21.15.37:0/2183206712 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d0c0723a0 0x7f4d0c1cee20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4d000ae040 tx=0x7f4d0001ab20 comp rx=0 tx=0).stop 2024-04-26T00:22:32.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.118+0000 7f4d11463640 1 -- 172.21.15.37:0/2183206712 shutdown_connections 2024-04-26T00:22:32.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.118+0000 7f4d11463640 1 --2- 172.21.15.37:0/2183206712 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4cf8035eb0 0x7f4cf8038370 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:32.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.118+0000 7f4d11463640 1 --2- 172.21.15.37:0/2183206712 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4d0c13c340 0x7f4d0c1cf360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:32.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.118+0000 7f4d11463640 1 --2- 172.21.15.37:0/2183206712 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d0c0723a0 0x7f4d0c1cee20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:32.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.118+0000 7f4d11463640 1 --2- 172.21.15.37:0/2183206712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d0c071fc0 0x7f4d0c1336a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:32.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.118+0000 7f4d11463640 1 -- 172.21.15.37:0/2183206712 >> 172.21.15.37:0/2183206712 conn(0x7f4d0c0a1ad0 msgr2=0x7f4d0c0a3d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:22:32.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.118+0000 7f4d11463640 1 -- 172.21.15.37:0/2183206712 shutdown_connections 2024-04-26T00:22:32.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.118+0000 7f4d11463640 1 -- 172.21.15.37:0/2183206712 wait complete. 2024-04-26T00:22:32.112 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:22:32.112 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-26T00:22:32.112 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:22:32.113 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:22:32.157 INFO:tasks.rados.rados.0.smithi045.stdout:461: finishing rollback tid 0 to smithi04526579-11 2024-04-26T00:22:32.157 INFO:tasks.rados.rados.0.smithi045.stdout:461: finishing rollback tid 1 to smithi04526579-11 2024-04-26T00:22:32.157 INFO:tasks.rados.rados.0.smithi045.stdout:461: finishing rollback tid 2 to smithi04526579-11 2024-04-26T00:22:32.157 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 102 (ObjNum 155 snap 33 seq_num 155) dirty exists 2024-04-26T00:22:32.157 INFO:tasks.rados.rados.0.smithi045.stdout:463: finishing write tid 1 to smithi04526579-32 2024-04-26T00:22:32.157 INFO:tasks.rados.rados.0.smithi045.stdout:463: finishing write tid 2 to smithi04526579-32 2024-04-26T00:22:32.157 INFO:tasks.rados.rados.0.smithi045.stdout:463: finishing write tid 3 to smithi04526579-32 2024-04-26T00:22:32.158 INFO:tasks.rados.rados.0.smithi045.stdout:463: finishing write tid 4 to smithi04526579-32 2024-04-26T00:22:32.158 INFO:tasks.rados.rados.0.smithi045.stdout:463: finishing write tid 5 to smithi04526579-32 2024-04-26T00:22:32.158 INFO:tasks.rados.rados.0.smithi045.stdout:463: oid 32 updating version 0 to 151 2024-04-26T00:22:32.158 INFO:tasks.rados.rados.0.smithi045.stdout:463: oid 32 updating version 151 to 153 2024-04-26T00:22:32.158 INFO:tasks.rados.rados.0.smithi045.stdout:463: oid 32 updating version 153 to 154 2024-04-26T00:22:32.158 INFO:tasks.rados.rados.0.smithi045.stdout:463: oid 32 version 154 is already newer than 152 2024-04-26T00:22:32.158 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 154 (ObjNum 196 snap 50 seq_num 196) dirty exists 2024-04-26T00:22:32.158 INFO:tasks.rados.rados.0.smithi045.stdout:463: left oid 32 (ObjNum 196 snap 50 seq_num 196) 2024-04-26T00:22:32.158 INFO:tasks.rados.rados.0.smithi045.stdout:461: done (3 left) 2024-04-26T00:22:32.158 INFO:tasks.rados.rados.0.smithi045.stdout:462: done (2 left) 2024-04-26T00:22:32.158 INFO:tasks.rados.rados.0.smithi045.stdout:463: done (1 left) 2024-04-26T00:22:32.158 INFO:tasks.rados.rados.0.smithi045.stdout:465: copy_from oid 18 from oid 31 current snap is 50 2024-04-26T00:22:32.158 INFO:tasks.rados.rados.0.smithi045.stdout:466: snap_create 2024-04-26T00:22:32.159 INFO:tasks.rados.rados.0.smithi045.stdout:465: finishing copy_from to smithi04526579-18 2024-04-26T00:22:32.159 INFO:tasks.rados.rados.0.smithi045.stdout:465: got expected ENOENT (src dne) 2024-04-26T00:22:32.159 INFO:tasks.rados.rados.0.smithi045.stdout:465: finishing copy_from racing read to smithi04526579-18 2024-04-26T00:22:32.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.250+0000 7f0550fc7640 10 monclient: get_monmap_and_config 2024-04-26T00:22:32.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.250+0000 7f0550fc7640 10 monclient: build_initial_monmap 2024-04-26T00:22:32.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.250+0000 7f0550fc7640 10 monclient: monmap: 2024-04-26T00:22:32.230 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:22:32.231 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:32.231 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:22:32.255528+0000 2024-04-26T00:22:32.231 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:22:32.255528+0000 2024-04-26T00:22:32.231 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:22:32.231 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:32.231 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:22:32.231 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:22:32.231 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:22:32.231 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:22:32.231 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:22:32.231 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:32.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f0550fc7640 1 Processor -- start 2024-04-26T00:22:32.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f0550fc7640 1 -- start start 2024-04-26T00:22:32.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f0550fc7640 10 monclient: init 2024-04-26T00:22:32.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f0550fc7640 10 monclient: _reopen_session rank -1 2024-04-26T00:22:32.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f0550fc7640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-04-26T00:22:32.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f0550fc7640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f054c0a0300 0x7f054c09fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:32.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f0550fc7640 10 monclient(hunting): picked mon.noname-c con 0x7f054c0a0300 addr v2:172.21.15.45:3300/0 2024-04-26T00:22:32.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f0550fc7640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f054c0a06e0 0x7f054c134ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:32.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f0550fc7640 10 monclient(hunting): picked mon.noname-b con 0x7f054c0a06e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:22:32.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f0550fc7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f054c1350e0 addr v1:172.21.15.37:6789/0 2024-04-26T00:22:32.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f0550fc7640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f054c09c190 con 0x7f054c1350e0 2024-04-26T00:22:32.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f0550fc7640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:32.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f0550fc7640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f054c0a13f0 con 0x7f054c0a0300 2024-04-26T00:22:32.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f0550fc7640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:32.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f0550fc7640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f054c0a1110 con 0x7f054c0a06e0 2024-04-26T00:22:32.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f0550fc7640 10 monclient(hunting): _renew_subs 2024-04-26T00:22:32.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f0550fc7640 10 monclient(hunting): authenticate will time out at 1133.561457s 2024-04-26T00:22:32.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f054ad9d640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f054c1350e0 0x7f054c13bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:51986/0 (socket says 172.21.15.37:51986) 2024-04-26T00:22:32.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f054ad9d640 1 -- 172.21.15.37:0/3741628770 learned_addr learned my addr 172.21.15.37:0/3741628770 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:22:32.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f0549d9b640 1 --2- 172.21.15.37:0/3741628770 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f054c0a06e0 0x7f054c134ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:32.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f0549d9b640 10 monclient(hunting): get_auth_request con 0x7f054c0a06e0 auth_method 0 2024-04-26T00:22:32.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f0549d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:32.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f0549d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:32.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f0549d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:32.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f0549d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:32.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f0549d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:32.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f054a59c640 1 --2- 172.21.15.37:0/3741628770 >> v2:172.21.15.45:3300/0 conn(0x7f054c0a0300 0x7f054c09fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:32.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f0549d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:32.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f054a59c640 10 monclient(hunting): get_auth_request con 0x7f054c0a0300 auth_method 0 2024-04-26T00:22:32.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f054a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:32.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f054a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:32.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f054a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:32.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f054a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:32.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f054a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:32.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f054a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:32.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f0549d9b640 10 monclient(hunting): handle_auth_done global_id 34121 payload 274 2024-04-26T00:22:32.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f0549d9b640 1 -- 172.21.15.37:0/3741628770 >> v2:172.21.15.45:3300/0 conn(0x7f054c0a0300 msgr2=0x7f054c09fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:22:32.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f0549d9b640 1 --2- 172.21.15.37:0/3741628770 >> v2:172.21.15.45:3300/0 conn(0x7f054c0a0300 0x7f054c09fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:32.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f0549d9b640 1 -- 172.21.15.37:0/3741628770 >> v1:172.21.15.37:6789/0 conn(0x7f054c1350e0 legacy=0x7f054c13bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:22:32.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f0549d9b640 10 monclient: _finish_hunting 0 2024-04-26T00:22:32.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f0549d9b640 1 monclient: found mon.noname-b 2024-04-26T00:22:32.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f0549d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:22:32.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f0549d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:22:32.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.254+0000 7f0549d9b640 1 -- 172.21.15.37:0/3741628770 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f054c0a16d0 con 0x7f054c0a06e0 2024-04-26T00:22:32.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.258+0000 7f0549d9b640 1 -- 172.21.15.37:0/3741628770 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f054c0a06e0 msgr2=0x7f054c134ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:32.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.258+0000 7f0549d9b640 1 --2- 172.21.15.37:0/3741628770 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f054c0a06e0 0x7f054c134ba0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f05400097c0 tx=0x7f054001c350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1baae50cf29c3aca server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:32.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.258+0000 7f054959a640 1 -- 172.21.15.37:0/3741628770 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05400b0070 con 0x7f054c0a06e0 2024-04-26T00:22:32.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.258+0000 7f054959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:32.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.258+0000 7f054959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:22:32.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.258+0000 7f054959a640 10 monclient: dump: 2024-04-26T00:22:32.239 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:32.239 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:32.239 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.239 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.239 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:32.239 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:32.239 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:32.239 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:32.239 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:32.239 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:32.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.258+0000 7f054959a640 10 monclient: _finish_auth 0 2024-04-26T00:22:32.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.258+0000 7f054959a640 10 monclient: _check_auth_tickets 2024-04-26T00:22:32.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.258+0000 7f054959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:22:32.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.258+0000 7f054959a640 1 -- 172.21.15.37:0/3741628770 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05400ab600 con 0x7f054c0a06e0 2024-04-26T00:22:32.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.258+0000 7f0550fc7640 5 monclient: authenticate success, global_id 34121 2024-04-26T00:22:32.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.258+0000 7f0550fc7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:22:32.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.258+0000 7f054959a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:22:32.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.258+0000 7f0550fc7640 10 monclient: get_monmap_and_config success 2024-04-26T00:22:32.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.258+0000 7f054959a640 1 -- 172.21.15.37:0/3741628770 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05400ab9d0 con 0x7f054c0a06e0 2024-04-26T00:22:32.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.258+0000 7f0550fc7640 4 set_mon_vals no callback set 2024-04-26T00:22:32.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.258+0000 7f054959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:32.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.258+0000 7f054959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:22:32.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.258+0000 7f054959a640 10 monclient: dump: 2024-04-26T00:22:32.240 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:32.240 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:32.240 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.240 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.240 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:32.240 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:32.240 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:32.241 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:32.241 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:32.241 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:32.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.258+0000 7f0550fc7640 10 monclient: shutdown 2024-04-26T00:22:32.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.258+0000 7f0550fc7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:22:32.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.258+0000 7f0550fc7640 1 -- 172.21.15.37:0/3741628770 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f054c0a06e0 msgr2=0x7f054c134ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:32.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.258+0000 7f0550fc7640 1 --2- 172.21.15.37:0/3741628770 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f054c0a06e0 0x7f054c134ba0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f05400097c0 tx=0x7f054001c350 comp rx=0 tx=0).stop 2024-04-26T00:22:32.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.258+0000 7f0550fc7640 1 -- 172.21.15.37:0/3741628770 shutdown_connections 2024-04-26T00:22:32.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.258+0000 7f0550fc7640 1 --2- 172.21.15.37:0/3741628770 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f054c0a06e0 0x7f054c134ba0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:32.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.258+0000 7f0550fc7640 1 --2- 172.21.15.37:0/3741628770 >> v2:172.21.15.45:3300/0 conn(0x7f054c0a0300 0x7f054c09fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:32.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.258+0000 7f0550fc7640 1 -- 172.21.15.37:0/3741628770 >> 172.21.15.37:0/3741628770 conn(0x7f054c0a1d50 msgr2=0x7f054c0a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:22:32.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.258+0000 7f0550fc7640 1 -- 172.21.15.37:0/3741628770 shutdown_connections 2024-04-26T00:22:32.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.258+0000 7f0550fc7640 1 -- 172.21.15.37:0/3741628770 wait complete. 2024-04-26T00:22:32.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.258+0000 7f0550fc7640 10 monclient: build_initial_monmap 2024-04-26T00:22:32.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.258+0000 7f0550fc7640 10 monclient: monmap: 2024-04-26T00:22:32.242 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:22:32.242 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:22:32.242 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:22:32.242 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:22:32.242 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:22:32.242 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:22:32.242 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:22:32.242 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:22:32.242 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:22:32.242 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:32.243 INFO:tasks.rados.rados.0.smithi045.stdout:464: finishing copy_from to smithi04526579-46 2024-04-26T00:22:32.243 INFO:tasks.rados.rados.0.smithi045.stdout:464: got expected ENOENT (src dne) 2024-04-26T00:22:32.243 INFO:tasks.rados.rados.0.smithi045.stdout:464: finishing copy_from racing read to smithi04526579-46 2024-04-26T00:22:32.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f0550fc7640 1 Processor -- start 2024-04-26T00:22:32.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f0550fc7640 1 -- start start 2024-04-26T00:22:32.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f0550fc7640 10 monclient: init 2024-04-26T00:22:32.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f0550fc7640 10 monclient: _reopen_session rank -1 2024-04-26T00:22:32.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f0550fc7640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:22:32.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f0550fc7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f054c0a0300 0x7f054c1ca950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:32.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f0550fc7640 10 monclient(hunting): picked mon.noname-b con 0x7f054c0a0300 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:22:32.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f0550fc7640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f054c0a06e0 0x7f054c13c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:32.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f0550fc7640 10 monclient(hunting): picked mon.noname-a con 0x7f054c0a06e0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:22:32.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f0550fc7640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f054c1350e0 0x7f054c13c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:32.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f0550fc7640 10 monclient(hunting): picked mon.noname-c con 0x7f054c1350e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:22:32.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f0550fc7640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:32.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f0550fc7640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f054c0a0c40 con 0x7f054c0a06e0 2024-04-26T00:22:32.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f0550fc7640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:32.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f0550fc7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f054c0a0ac0 con 0x7f054c0a0300 2024-04-26T00:22:32.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f0550fc7640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:32.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f0550fc7640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f054c13fe10 con 0x7f054c1350e0 2024-04-26T00:22:32.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f0550fc7640 10 monclient(hunting): _renew_subs 2024-04-26T00:22:32.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f0550fc7640 10 monclient(hunting): authenticate will time out at 1133.568842s 2024-04-26T00:22:32.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f054ad9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f054c1350e0 0x7f054c13c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:32.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f0549d9b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f054c0a06e0 0x7f054c13c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:32.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f054ad9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f054c1350e0 0x7f054c13c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:53586/0 (socket says 172.21.15.37:53586) 2024-04-26T00:22:32.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f054ad9d640 1 -- 172.21.15.37:0/574740905 learned_addr learned my addr 172.21.15.37:0/574740905 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:22:32.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f0549d9b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f054c0a06e0 0x7f054c13c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:45032/0 (socket says 172.21.15.37:45032) 2024-04-26T00:22:32.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f054ad9d640 10 monclient(hunting): get_auth_request con 0x7f054c1350e0 auth_method 0 2024-04-26T00:22:32.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f054ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:32.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f054ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:32.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f054ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:32.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f0549d9b640 10 monclient(hunting): get_auth_request con 0x7f054c0a06e0 auth_method 0 2024-04-26T00:22:32.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f0549d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:32.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f0549d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:32.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f0549d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:32.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f054ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:32.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f054ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:32.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f054ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:32.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f0549d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:32.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f0549d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:32.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f0549d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:32.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f054a59c640 1 --2- 172.21.15.37:0/574740905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f054c0a0300 0x7f054c1ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:32.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f054a59c640 10 monclient(hunting): get_auth_request con 0x7f054c0a0300 auth_method 0 2024-04-26T00:22:32.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f054a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:32.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f054a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:32.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f054a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:32.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f054ad9d640 10 monclient(hunting): handle_auth_done global_id 34124 payload 824 2024-04-26T00:22:32.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f054ad9d640 1 -- 172.21.15.37:0/574740905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f054c0a0300 msgr2=0x7f054c1ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:32.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f054ad9d640 1 --2- 172.21.15.37:0/574740905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f054c0a0300 0x7f054c1ca950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:32.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f054ad9d640 1 -- 172.21.15.37:0/574740905 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f054c0a06e0 msgr2=0x7f054c13c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:32.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f054ad9d640 1 --2- 172.21.15.37:0/574740905 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f054c0a06e0 0x7f054c13c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:32.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f054ad9d640 10 monclient: _finish_hunting 0 2024-04-26T00:22:32.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f054ad9d640 1 monclient: found mon.noname-c 2024-04-26T00:22:32.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f054ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:22:32.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f054ad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:22:32.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f054ad9d640 1 -- 172.21.15.37:0/574740905 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f054c13ce40 con 0x7f054c1350e0 2024-04-26T00:22:32.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f054ad9d640 1 -- 172.21.15.37:0/574740905 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f054c1350e0 msgr2=0x7f054c13c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:32.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f054ad9d640 1 --2- 172.21.15.37:0/574740905 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f054c1350e0 0x7f054c13c900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f054400c420 tx=0x7f054400c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:32.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f053affd640 1 -- 172.21.15.37:0/574740905 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0544017070 con 0x7f054c1350e0 2024-04-26T00:22:32.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f053affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:32.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f053affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:22:32.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f053affd640 10 monclient: dump: 2024-04-26T00:22:32.252 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:32.252 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:32.252 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.252 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.252 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:32.253 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:32.253 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:32.253 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:32.253 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:32.253 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:32.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f053affd640 10 monclient: _finish_auth 0 2024-04-26T00:22:32.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f053affd640 10 monclient: _check_auth_tickets 2024-04-26T00:22:32.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f053affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:22:32.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f053affd640 1 -- 172.21.15.37:0/574740905 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05440070d0 con 0x7f054c1350e0 2024-04-26T00:22:32.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f053affd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:22:32.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f053affd640 1 -- 172.21.15.37:0/574740905 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05440074c0 con 0x7f054c1350e0 2024-04-26T00:22:32.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f0550fc7640 5 monclient: authenticate success, global_id 34124 2024-04-26T00:22:32.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f0550fc7640 10 monclient: _renew_subs 2024-04-26T00:22:32.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f0550fc7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:22:32.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f0550fc7640 1 -- 172.21.15.37:0/574740905 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f054c06e580 con 0x7f054c1350e0 2024-04-26T00:22:32.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f0548d99640 4 set_mon_vals no callback set 2024-04-26T00:22:32.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f053affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:32.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f053affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:22:32.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f053affd640 10 monclient: dump: 2024-04-26T00:22:32.254 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:32.254 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:32.254 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.254 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.254 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:32.254 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:32.254 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:32.254 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:32.254 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:32.254 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:32.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f0550fc7640 10 monclient: _renew_subs 2024-04-26T00:22:32.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f0550fc7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:22:32.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.262+0000 7f0550fc7640 1 -- 172.21.15.37:0/574740905 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f054c1395e0 con 0x7f054c1350e0 2024-04-26T00:22:32.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.266+0000 7f0550fc7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:32.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.266+0000 7f0550fc7640 1 --2- 172.21.15.37:0/574740905 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f054c1cd430 0x7f054c1cd810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:32.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.266+0000 7f0550fc7640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:32.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.266+0000 7f0550fc7640 1 -- 172.21.15.37:0/574740905 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f054c139de0 con 0x7f054c1cd430 2024-04-26T00:22:32.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.266+0000 7f054a59c640 1 --2- 172.21.15.37:0/574740905 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f054c1cd430 0x7f054c1cd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:32.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.266+0000 7f054a59c640 10 monclient: get_auth_request con 0x7f054c1cd430 auth_method 0 2024-04-26T00:22:32.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.266+0000 7f054a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:32.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.266+0000 7f054a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:32.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.266+0000 7f054a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:32.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.266+0000 7f054a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:32.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.266+0000 7f054a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:32.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.266+0000 7f054a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:32.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.266+0000 7f053affd640 1 -- 172.21.15.37:0/574740905 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f054401a070 con 0x7f054c1350e0 2024-04-26T00:22:32.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.266+0000 7f054a59c640 10 monclient(hunting): handle_auth_done global_id 4665 payload 824 2024-04-26T00:22:32.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.266+0000 7f054a59c640 1 -- 172.21.15.37:0/574740905 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f054c04f510 con 0x7f054c1cd430 2024-04-26T00:22:32.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.266+0000 7f053affd640 1 --2- 172.21.15.37:0/574740905 >> v2:172.21.15.45:6812/2379918623 conn(0x7f052c035ed0 0x7f052c038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:32.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.266+0000 7f053affd640 1 -- 172.21.15.37:0/574740905 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(111..111 src has 1..111) ==== 3963+0+0 (secure 0 0 0) 0x7f0544047ef0 con 0x7f054c1350e0 2024-04-26T00:22:32.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.266+0000 7f0549d9b640 1 --2- 172.21.15.37:0/574740905 >> v2:172.21.15.45:6812/2379918623 conn(0x7f052c035ed0 0x7f052c038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:32.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.266+0000 7f0549d9b640 10 monclient: get_auth_request con 0x7f052c035ed0 auth_method 0 2024-04-26T00:22:32.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.266+0000 7f054a59c640 1 -- 172.21.15.37:0/574740905 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f054c1cd430 msgr2=0x7f054c1cd810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:32.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.266+0000 7f054a59c640 1 --2- 172.21.15.37:0/574740905 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f054c1cd430 0x7f054c1cd810 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f053c007f00 tx=0x7f053c0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:32.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.266+0000 7f053affd640 1 -- 172.21.15.37:0/574740905 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f053c011ea0 con 0x7f054c1cd430 2024-04-26T00:22:32.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.266+0000 7f053affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:32.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.266+0000 7f053affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:22:32.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.266+0000 7f053affd640 10 monclient: dump: 2024-04-26T00:22:32.256 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:32.257 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:32.257 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.257 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.257 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:32.257 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:32.257 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:32.257 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:32.257 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:32.257 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:32.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.266+0000 7f053affd640 1 -- 172.21.15.37:0/574740905 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f053c0039a0 con 0x7f054c1cd430 2024-04-26T00:22:32.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.266+0000 7f053affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:32.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.266+0000 7f053affd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:22:32.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.266+0000 7f053affd640 1 -- 172.21.15.37:0/574740905 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f054c1cd430 msgr2=0x7f054c1cd810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:32.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.266+0000 7f053affd640 1 --2- 172.21.15.37:0/574740905 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f054c1cd430 0x7f054c1cd810 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f053c007f00 tx=0x7f053c0115a0 comp rx=0 tx=0).stop 2024-04-26T00:22:32.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.266+0000 7f053affd640 1 -- 172.21.15.37:0/574740905 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f054c1cd430 msgr2=0x7f054c1cd810 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:22:32.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.266+0000 7f053affd640 1 --2- 172.21.15.37:0/574740905 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f054c1cd430 0x7f054c1cd810 secure :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f053c007f00 tx=0x7f053c0115a0 comp rx=0 tx=0).stop 2024-04-26T00:22:32.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.270+0000 7f0549d9b640 1 --2- 172.21.15.37:0/574740905 >> v2:172.21.15.45:6812/2379918623 conn(0x7f052c035ed0 0x7f052c038390 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f05400ab000 tx=0x7f05400a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:32.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.274+0000 7f0550fc7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:22:32.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.274+0000 7f0550fc7640 1 --2- 172.21.15.37:0/574740905 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f054c1cdd50 0x7f054c070010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:32.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.274+0000 7f0550fc7640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:32.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.274+0000 7f0550fc7640 1 -- 172.21.15.37:0/574740905 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f054c06f0d0 con 0x7f054c1cdd50 2024-04-26T00:22:32.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.274+0000 7f054a59c640 1 --2- 172.21.15.37:0/574740905 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f054c1cdd50 0x7f054c070010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:32.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.274+0000 7f054a59c640 10 monclient: get_auth_request con 0x7f054c1cdd50 auth_method 0 2024-04-26T00:22:32.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.274+0000 7f054a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:32.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.274+0000 7f054a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:32.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.274+0000 7f054a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:32.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.274+0000 7f054a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:32.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.274+0000 7f054a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:32.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.274+0000 7f054a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:32.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.274+0000 7f054a59c640 10 monclient(hunting): handle_auth_done global_id 4668 payload 824 2024-04-26T00:22:32.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.274+0000 7f054a59c640 1 -- 172.21.15.37:0/574740905 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f054c09eb40 con 0x7f054c1cdd50 2024-04-26T00:22:32.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.278+0000 7f054a59c640 1 -- 172.21.15.37:0/574740905 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f054c1cdd50 msgr2=0x7f054c070010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:32.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.278+0000 7f054a59c640 1 --2- 172.21.15.37:0/574740905 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f054c1cdd50 0x7f054c070010 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f053c007f00 tx=0x7f053c004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:32.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.278+0000 7f053affd640 1 -- 172.21.15.37:0/574740905 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f053c004250 con 0x7f054c1cdd50 2024-04-26T00:22:32.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.278+0000 7f053affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:32.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.278+0000 7f053affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:22:32.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.278+0000 7f053affd640 10 monclient: dump: 2024-04-26T00:22:32.261 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:32.261 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:32.261 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.261 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.261 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:32.261 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:32.261 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:32.261 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:32.261 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:32.261 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:32.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.278+0000 7f053affd640 1 -- 172.21.15.37:0/574740905 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4719 (secure 0 0 0) 0x7f053c004230 con 0x7f054c1cdd50 2024-04-26T00:22:32.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.278+0000 7f053affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:22:32.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.278+0000 7f053affd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:22:32.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.278+0000 7f053affd640 1 -- 172.21.15.37:0/574740905 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f054c1cdd50 msgr2=0x7f054c070010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:32.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.278+0000 7f053affd640 1 --2- 172.21.15.37:0/574740905 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f054c1cdd50 0x7f054c070010 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f053c007f00 tx=0x7f053c004a50 comp rx=0 tx=0).stop 2024-04-26T00:22:32.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.278+0000 7f053affd640 1 -- 172.21.15.37:0/574740905 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f054c1cdd50 msgr2=0x7f054c070010 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:22:32.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.278+0000 7f053affd640 1 --2- 172.21.15.37:0/574740905 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f054c1cdd50 0x7f054c070010 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:32.262 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:22:32.262 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:22:32.262 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:22:32.262 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:22:32.263 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 24, 2024-04-26T00:22:32.263 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 169382, 2024-04-26T00:22:32.263 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:22:32.263 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:22:32.263 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:22:32.263 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:22:32.263 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:22:32.263 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 6, 2024-04-26T00:22:32.263 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:22:32.263 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:22:32.263 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:22:32.263 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:22:32.263 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:22:32.263 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:22:32.263 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:22:32.264 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:22:32.264 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:22:32.264 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:22:32.264 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:22:32.264 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:22:32.264 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:22:32.264 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:22:32.264 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:22:32.264 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:22:32.264 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:22:32.264 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:22:32.264 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:22:32.264 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:22:32.264 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:22:32.265 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:22:32.265 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:22:32.265 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:22:32.265 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:22:32.265 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:22:32.265 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:22:32.265 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:22:32.265 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:32.265 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:32.265 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:22:32.265 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:22:32.265 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:22:32.265 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:22:32.265 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:22:32.265 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:22:32.266 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:22:32.266 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:22:32.266 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:22:32.266 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:22:32.266 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:22:32.266 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:22:32.266 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:22:32.266 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:22:32.266 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:22:32.266 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:22:32.266 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:22:32.266 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:22:32.266 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:22:32.266 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:22:32.267 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:22:32.267 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:22:32.267 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:22:32.267 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:22:32.267 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:22:32.267 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:22:32.267 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:22:32.267 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:22:32.267 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:22:32.267 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:22:32.267 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:32.267 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:22:32.267 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:32.268 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:22:32.268 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:22:32.268 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:22:32.268 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:22:32.268 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:32.268 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:22:32.268 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:22:32.268 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:32.268 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:32.268 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:32.268 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:22:32.268 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:22:32.268 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:32.268 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:32.269 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:32.269 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:32.269 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:22:32.269 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:22:32.269 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:22:32.269 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:22:32.269 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:22:32.269 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:32.269 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:32.269 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:22:32.269 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:22:32.270 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:22:32.270 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:22:32.270 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:32.270 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:22:32.270 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:22:32.270 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:32.270 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:32.270 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:32.270 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:22:32.270 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:22:32.270 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:32.270 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:32.270 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:32.270 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:32.271 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:22:32.271 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:22:32.271 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:22:32.271 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:22:32.271 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:22:32.271 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:32.271 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:32.271 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:22:32.271 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:22:32.271 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:22:32.271 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:22:32.271 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:32.271 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:22:32.271 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:22:32.271 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:32.272 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:32.272 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:32.272 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:22:32.272 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:22:32.272 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:32.272 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:32.272 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:32.272 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:32.272 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:22:32.272 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:22:32.272 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:22:32.272 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:22:32.272 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:22:32.272 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:32.272 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:32.273 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:32.273 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:22:32.273 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:22:32.273 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:32.273 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:22:32.273 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:22:32.273 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:22:32.273 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:32.273 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:22:32.273 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:22:32.273 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:32.273 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:22:32.273 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:22:32.273 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:22:32.273 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:32.274 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:22:32.274 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:22:32.274 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:32.274 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:22:32.274 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:22:32.274 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:22:32.274 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:32.274 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:32.274 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:32.274 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:22:32.274 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:22:32.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.278+0000 7f0550fc7640 1 -- 172.21.15.37:0/574740905 >> v2:172.21.15.45:6812/2379918623 conn(0x7f052c035ed0 msgr2=0x7f052c038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:32.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.278+0000 7f0550fc7640 1 --2- 172.21.15.37:0/574740905 >> v2:172.21.15.45:6812/2379918623 conn(0x7f052c035ed0 0x7f052c038390 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f05400ab000 tx=0x7f05400a6000 comp rx=0 tx=0).stop 2024-04-26T00:22:32.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.278+0000 7f0550fc7640 10 monclient: shutdown 2024-04-26T00:22:32.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.278+0000 7f0550fc7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:22:32.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.278+0000 7f0550fc7640 1 -- 172.21.15.37:0/574740905 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f054c1350e0 msgr2=0x7f054c13c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:32.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.278+0000 7f0550fc7640 1 --2- 172.21.15.37:0/574740905 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f054c1350e0 0x7f054c13c900 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f054400c420 tx=0x7f054400c8e0 comp rx=0 tx=0).stop 2024-04-26T00:22:32.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.278+0000 7f054ad9d640 1 -- 172.21.15.37:0/574740905 reap_dead start 2024-04-26T00:22:32.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.278+0000 7f054ad9d640 1 -- 172.21.15.37:0/574740905 reap_dead start 2024-04-26T00:22:32.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.278+0000 7f0550fc7640 1 -- 172.21.15.37:0/574740905 shutdown_connections 2024-04-26T00:22:32.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.278+0000 7f0550fc7640 1 -- 172.21.15.37:0/574740905 >> 172.21.15.37:0/574740905 conn(0x7f054c0a1d50 msgr2=0x7f054c0a3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:22:32.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.278+0000 7f0550fc7640 1 -- 172.21.15.37:0/574740905 shutdown_connections 2024-04-26T00:22:32.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.278+0000 7f0550fc7640 1 -- 172.21.15.37:0/574740905 wait complete. 2024-04-26T00:22:32.277 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:22:32.277 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:22:32.277 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:22:32.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.414+0000 7f8e5854b640 10 monclient: get_monmap_and_config 2024-04-26T00:22:32.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.414+0000 7f8e5854b640 10 monclient: build_initial_monmap 2024-04-26T00:22:32.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.414+0000 7f8e5854b640 10 monclient: monmap: 2024-04-26T00:22:32.391 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:22:32.391 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:32.391 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:22:32.416367+0000 2024-04-26T00:22:32.392 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:22:32.416367+0000 2024-04-26T00:22:32.392 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:22:32.392 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:32.392 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:22:32.392 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:22:32.392 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:22:32.392 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:22:32.392 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:22:32.392 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:32.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.414+0000 7f8e5854b640 1 Processor -- start 2024-04-26T00:22:32.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.414+0000 7f8e5854b640 1 -- start start 2024-04-26T00:22:32.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.414+0000 7f8e5854b640 10 monclient: init 2024-04-26T00:22:32.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.414+0000 7f8e5854b640 10 monclient: _reopen_session rank -1 2024-04-26T00:22:32.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.414+0000 7f8e5854b640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-04-26T00:22:32.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.414+0000 7f8e5854b640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f8e500a3d90 0x7f8e5009bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:32.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.414+0000 7f8e5854b640 10 monclient(hunting): picked mon.noname-a con 0x7f8e500a3d90 addr v2:172.21.15.37:3300/0 2024-04-26T00:22:32.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.414+0000 7f8e5854b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8e500a4170 0x7f8e500a45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:32.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.414+0000 7f8e5854b640 10 monclient(hunting): picked mon.noname-c con 0x7f8e500a4170 addr v2:172.21.15.45:3300/0 2024-04-26T00:22:32.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.414+0000 7f8e5854b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8e50133b20 addr v1:172.21.15.45:6789/0 2024-04-26T00:22:32.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.414+0000 7f8e5854b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e500a5150 con 0x7f8e50133b20 2024-04-26T00:22:32.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.414+0000 7f8e5854b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:32.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.414+0000 7f8e5854b640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f8e5009b280 con 0x7f8e500a3d90 2024-04-26T00:22:32.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.414+0000 7f8e5854b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:32.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.414+0000 7f8e5854b640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f8e5009b560 con 0x7f8e500a4170 2024-04-26T00:22:32.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.414+0000 7f8e5854b640 10 monclient(hunting): _renew_subs 2024-04-26T00:22:32.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.414+0000 7f8e5854b640 10 monclient(hunting): authenticate will time out at 1133.722634s 2024-04-26T00:22:32.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.414+0000 7f8e562e7640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f8e500a3d90 0x7f8e5009bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:32.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.414+0000 7f8e562e7640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f8e500a3d90 0x7f8e5009bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:45064/0 (socket says 172.21.15.37:45064) 2024-04-26T00:22:32.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.414+0000 7f8e562e7640 1 -- 172.21.15.37:0/4261354123 learned_addr learned my addr 172.21.15.37:0/4261354123 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:22:32.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.414+0000 7f8e562e7640 10 monclient(hunting): get_auth_request con 0x7f8e500a3d90 auth_method 0 2024-04-26T00:22:32.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.414+0000 7f8e562e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:32.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.414+0000 7f8e562e7640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:32.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.414+0000 7f8e562e7640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:32.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.414+0000 7f8e562e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:32.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.414+0000 7f8e562e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:32.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.414+0000 7f8e562e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:32.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.414+0000 7f8e562e7640 10 monclient(hunting): handle_auth_done global_id 4671 payload 274 2024-04-26T00:22:32.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.414+0000 7f8e562e7640 1 -- 172.21.15.37:0/4261354123 >> v2:172.21.15.45:3300/0 conn(0x7f8e500a4170 msgr2=0x7f8e500a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:22:32.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.414+0000 7f8e562e7640 1 --2- 172.21.15.37:0/4261354123 >> v2:172.21.15.45:3300/0 conn(0x7f8e500a4170 0x7f8e500a45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:32.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.414+0000 7f8e562e7640 1 -- 172.21.15.37:0/4261354123 >> v1:172.21.15.45:6789/0 conn(0x7f8e50133b20 legacy=0x7f8e50133f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:22:32.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.414+0000 7f8e562e7640 10 monclient: _finish_hunting 0 2024-04-26T00:22:32.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.414+0000 7f8e562e7640 1 monclient: found mon.noname-a 2024-04-26T00:22:32.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.414+0000 7f8e562e7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:22:32.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.414+0000 7f8e562e7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:22:32.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.414+0000 7f8e562e7640 1 -- 172.21.15.37:0/4261354123 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e500a4b20 con 0x7f8e500a3d90 2024-04-26T00:22:32.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.418+0000 7f8e562e7640 1 -- 172.21.15.37:0/4261354123 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8e500a3d90 msgr2=0x7f8e5009bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:32.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.418+0000 7f8e562e7640 1 --2- 172.21.15.37:0/4261354123 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8e500a3d90 0x7f8e5009bf40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8e5009d140 tx=0x7f8e4001c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bf2df453c77340a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:32.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.418+0000 7f8e552e5640 1 -- 172.21.15.37:0/4261354123 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e400b0070 con 0x7f8e500a3d90 2024-04-26T00:22:32.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.418+0000 7f8e552e5640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:32.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.418+0000 7f8e552e5640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:22:32.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.418+0000 7f8e552e5640 10 monclient: dump: 2024-04-26T00:22:32.400 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:32.400 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:32.400 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.400 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.400 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:32.400 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:32.400 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:32.400 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:32.400 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:32.400 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:32.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.418+0000 7f8e552e5640 10 monclient: _finish_auth 0 2024-04-26T00:22:32.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.418+0000 7f8e552e5640 10 monclient: _check_auth_tickets 2024-04-26T00:22:32.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.418+0000 7f8e552e5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:22:32.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.418+0000 7f8e552e5640 1 -- 172.21.15.37:0/4261354123 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e400ab600 con 0x7f8e500a3d90 2024-04-26T00:22:32.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.418+0000 7f8e552e5640 10 monclient: handle_config config(0 keys) 2024-04-26T00:22:32.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.418+0000 7f8e552e5640 1 -- 172.21.15.37:0/4261354123 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e400ab9d0 con 0x7f8e500a3d90 2024-04-26T00:22:32.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.418+0000 7f8e5854b640 5 monclient: authenticate success, global_id 4671 2024-04-26T00:22:32.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.418+0000 7f8e5854b640 10 monclient: get_monmap_and_config success 2024-04-26T00:22:32.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.418+0000 7f8e5854b640 4 set_mon_vals no callback set 2024-04-26T00:22:32.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.418+0000 7f8e552e5640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:32.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.418+0000 7f8e552e5640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:22:32.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.418+0000 7f8e552e5640 10 monclient: dump: 2024-04-26T00:22:32.401 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:32.402 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:32.402 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.402 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.402 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:32.402 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:32.402 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:32.402 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:32.402 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:32.402 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:32.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e5854b640 10 monclient: shutdown 2024-04-26T00:22:32.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e5854b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:22:32.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e5854b640 1 -- 172.21.15.37:0/4261354123 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8e500a3d90 msgr2=0x7f8e5009bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:32.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e5854b640 1 --2- 172.21.15.37:0/4261354123 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8e500a3d90 0x7f8e5009bf40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8e5009d140 tx=0x7f8e4001c3a0 comp rx=0 tx=0).stop 2024-04-26T00:22:32.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e5854b640 1 -- 172.21.15.37:0/4261354123 shutdown_connections 2024-04-26T00:22:32.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e5854b640 1 --2- 172.21.15.37:0/4261354123 >> v2:172.21.15.45:3300/0 conn(0x7f8e500a4170 0x7f8e500a45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:32.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e5854b640 1 --2- 172.21.15.37:0/4261354123 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8e500a3d90 0x7f8e5009bf40 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:32.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e5854b640 1 -- 172.21.15.37:0/4261354123 >> 172.21.15.37:0/4261354123 conn(0x7f8e500aa0a0 msgr2=0x7f8e500a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:22:32.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e5854b640 1 -- 172.21.15.37:0/4261354123 shutdown_connections 2024-04-26T00:22:32.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e5854b640 1 -- 172.21.15.37:0/4261354123 wait complete. 2024-04-26T00:22:32.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e5854b640 10 monclient: build_initial_monmap 2024-04-26T00:22:32.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e5854b640 10 monclient: monmap: 2024-04-26T00:22:32.403 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:22:32.403 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:22:32.404 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:22:32.404 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:22:32.404 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:22:32.404 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:22:32.404 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:22:32.404 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:22:32.404 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:22:32.404 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:32.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e5854b640 1 Processor -- start 2024-04-26T00:22:32.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e5854b640 1 -- start start 2024-04-26T00:22:32.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e5854b640 10 monclient: init 2024-04-26T00:22:32.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e5854b640 10 monclient: _reopen_session rank -1 2024-04-26T00:22:32.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e5854b640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:22:32.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e5854b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e500a3d90 0x7f8e5013e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:32.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e5854b640 10 monclient(hunting): picked mon.noname-b con 0x7f8e500a3d90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:22:32.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e5854b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e500a4170 0x7f8e5013e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:32.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e5854b640 10 monclient(hunting): picked mon.noname-c con 0x7f8e500a4170 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:22:32.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e5854b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8e50133b20 0x7f8e501cf560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:32.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e5854b640 10 monclient(hunting): picked mon.noname-a con 0x7f8e50133b20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:22:32.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e5854b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:32.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e5854b640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f8e50141e30 con 0x7f8e50133b20 2024-04-26T00:22:32.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e5854b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:32.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e5854b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8e5009c690 con 0x7f8e500a3d90 2024-04-26T00:22:32.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e5854b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:32.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e5854b640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8e50141fb0 con 0x7f8e500a4170 2024-04-26T00:22:32.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e5854b640 10 monclient(hunting): _renew_subs 2024-04-26T00:22:32.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e5854b640 10 monclient(hunting): authenticate will time out at 1133.729640s 2024-04-26T00:22:32.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e55ae6640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e500a4170 0x7f8e5013e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:32.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e55ae6640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e500a4170 0x7f8e5013e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:53600/0 (socket says 172.21.15.37:53600) 2024-04-26T00:22:32.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e55ae6640 1 -- 172.21.15.37:0/2306670105 learned_addr learned my addr 172.21.15.37:0/2306670105 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:22:32.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e55ae6640 10 monclient(hunting): get_auth_request con 0x7f8e500a4170 auth_method 0 2024-04-26T00:22:32.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e55ae6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:32.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e55ae6640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:32.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e55ae6640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:32.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e56ae8640 1 --2- 172.21.15.37:0/2306670105 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8e50133b20 0x7f8e501cf560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:32.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e56ae8640 10 monclient(hunting): get_auth_request con 0x7f8e50133b20 auth_method 0 2024-04-26T00:22:32.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e56ae8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:32.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e56ae8640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:32.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e56ae8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:32.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e55ae6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:32.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e55ae6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:32.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e55ae6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:32.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e56ae8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:32.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e56ae8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:32.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e56ae8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:32.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e562e7640 1 --2- 172.21.15.37:0/2306670105 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e500a3d90 0x7f8e5013e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:32.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e562e7640 10 monclient(hunting): get_auth_request con 0x7f8e500a3d90 auth_method 0 2024-04-26T00:22:32.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e562e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:32.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e562e7640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:32.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e562e7640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:32.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e55ae6640 10 monclient(hunting): handle_auth_done global_id 34127 payload 824 2024-04-26T00:22:32.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e55ae6640 1 -- 172.21.15.37:0/2306670105 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e500a3d90 msgr2=0x7f8e5013e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:32.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e55ae6640 1 --2- 172.21.15.37:0/2306670105 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8e500a3d90 0x7f8e5013e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:32.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e55ae6640 1 -- 172.21.15.37:0/2306670105 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8e50133b20 msgr2=0x7f8e501cf560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:32.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e55ae6640 1 --2- 172.21.15.37:0/2306670105 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8e50133b20 0x7f8e501cf560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:32.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e55ae6640 10 monclient: _finish_hunting 0 2024-04-26T00:22:32.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e55ae6640 1 monclient: found mon.noname-c 2024-04-26T00:22:32.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e55ae6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:22:32.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e55ae6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:22:32.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.422+0000 7f8e55ae6640 1 -- 172.21.15.37:0/2306670105 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e5013abb0 con 0x7f8e500a4170 2024-04-26T00:22:32.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e55ae6640 1 -- 172.21.15.37:0/2306670105 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e500a4170 msgr2=0x7f8e5013e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:32.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e55ae6640 1 --2- 172.21.15.37:0/2306670105 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e500a4170 0x7f8e5013e910 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c00b580 tx=0x7f8e4c00ba40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:32.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e46ffd640 1 -- 172.21.15.37:0/2306670105 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c014070 con 0x7f8e500a4170 2024-04-26T00:22:32.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e46ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:32.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e46ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:22:32.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e46ffd640 10 monclient: dump: 2024-04-26T00:22:32.414 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:32.415 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:32.415 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.415 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.415 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:32.415 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:32.415 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:32.415 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:32.415 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:32.415 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:32.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e46ffd640 10 monclient: _finish_auth 0 2024-04-26T00:22:32.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e46ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:22:32.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:22:32.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e46ffd640 1 -- 172.21.15.37:0/2306670105 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e4c010aa0 con 0x7f8e500a4170 2024-04-26T00:22:32.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e5854b640 5 monclient: authenticate success, global_id 34127 2024-04-26T00:22:32.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e46ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:22:32.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e46ffd640 1 -- 172.21.15.37:0/2306670105 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c00f3c0 con 0x7f8e500a4170 2024-04-26T00:22:32.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e46ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:32.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e46ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:22:32.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e46ffd640 10 monclient: dump: 2024-04-26T00:22:32.416 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:32.416 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:32.416 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.416 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.417 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:32.417 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:32.417 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:32.417 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:32.417 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:32.417 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:32.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e54ae4640 4 set_mon_vals no callback set 2024-04-26T00:22:32.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e5854b640 10 monclient: _renew_subs 2024-04-26T00:22:32.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e5854b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:22:32.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e5854b640 1 -- 172.21.15.37:0/2306670105 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e5009ada0 con 0x7f8e500a4170 2024-04-26T00:22:32.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e5854b640 10 monclient: _renew_subs 2024-04-26T00:22:32.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e5854b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:22:32.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e5854b640 1 -- 172.21.15.37:0/2306670105 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e5013c600 con 0x7f8e500a4170 2024-04-26T00:22:32.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e5854b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:32.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e5854b640 1 --2- 172.21.15.37:0/2306670105 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e501cfaa0 0x7f8e501cfe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:32.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e5854b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:32.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e5854b640 1 -- 172.21.15.37:0/2306670105 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8e500a0890 con 0x7f8e501cfaa0 2024-04-26T00:22:32.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e562e7640 1 --2- 172.21.15.37:0/2306670105 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e501cfaa0 0x7f8e501cfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:32.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e562e7640 10 monclient: get_auth_request con 0x7f8e501cfaa0 auth_method 0 2024-04-26T00:22:32.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e562e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:32.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e562e7640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:32.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e562e7640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:32.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e562e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:32.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e562e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:32.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e46ffd640 1 -- 172.21.15.37:0/2306670105 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f8e4c017070 con 0x7f8e500a4170 2024-04-26T00:22:32.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e562e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:32.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e562e7640 10 monclient(hunting): handle_auth_done global_id 34130 payload 824 2024-04-26T00:22:32.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e562e7640 1 -- 172.21.15.37:0/2306670105 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8e501d03c0 con 0x7f8e501cfaa0 2024-04-26T00:22:32.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e46ffd640 1 --2- 172.21.15.37:0/2306670105 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8e34035ed0 0x7f8e34038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:32.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e46ffd640 1 -- 172.21.15.37:0/2306670105 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(111..111 src has 1..111) ==== 3963+0+0 (secure 0 0 0) 0x7f8e4c0488e0 con 0x7f8e500a4170 2024-04-26T00:22:32.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e56ae8640 1 --2- 172.21.15.37:0/2306670105 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8e34035ed0 0x7f8e34038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:32.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.426+0000 7f8e56ae8640 10 monclient: get_auth_request con 0x7f8e34035ed0 auth_method 0 2024-04-26T00:22:32.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.430+0000 7f8e562e7640 1 -- 172.21.15.37:0/2306670105 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e501cfaa0 msgr2=0x7f8e501cfe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:32.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.430+0000 7f8e562e7640 1 --2- 172.21.15.37:0/2306670105 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e501cfaa0 0x7f8e501cfe80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8e5009a9e0 tx=0x7f8e40019140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:32.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.430+0000 7f8e46ffd640 1 -- 172.21.15.37:0/2306670105 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e400ba020 con 0x7f8e501cfaa0 2024-04-26T00:22:32.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.430+0000 7f8e46ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:32.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.430+0000 7f8e46ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:22:32.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.430+0000 7f8e46ffd640 10 monclient: dump: 2024-04-26T00:22:32.420 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:32.420 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:32.420 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.420 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.420 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:32.420 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:32.420 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:32.421 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:32.421 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:32.421 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:32.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.430+0000 7f8e46ffd640 1 -- 172.21.15.37:0/2306670105 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8e400b6470 con 0x7f8e501cfaa0 2024-04-26T00:22:32.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.430+0000 7f8e46ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:32.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.430+0000 7f8e46ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:22:32.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.430+0000 7f8e46ffd640 1 -- 172.21.15.37:0/2306670105 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e501cfaa0 msgr2=0x7f8e501cfe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:32.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.430+0000 7f8e46ffd640 1 --2- 172.21.15.37:0/2306670105 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e501cfaa0 0x7f8e501cfe80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8e5009a9e0 tx=0x7f8e40019140 comp rx=0 tx=0).stop 2024-04-26T00:22:32.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.430+0000 7f8e46ffd640 1 -- 172.21.15.37:0/2306670105 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e501cfaa0 msgr2=0x7f8e501cfe80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:22:32.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.430+0000 7f8e46ffd640 1 --2- 172.21.15.37:0/2306670105 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e501cfaa0 0x7f8e501cfe80 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8e5009a9e0 tx=0x7f8e40019140 comp rx=0 tx=0).stop 2024-04-26T00:22:32.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.430+0000 7f8e56ae8640 1 --2- 172.21.15.37:0/2306670105 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8e34035ed0 0x7f8e34038390 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8e48001ea0 tx=0x7f8e48009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:32.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.434+0000 7f8e5854b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:22:32.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.434+0000 7f8e5854b640 1 --2- 172.21.15.37:0/2306670105 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e50134b30 0x7f8e50134f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:32.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.434+0000 7f8e5854b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:32.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.434+0000 7f8e5854b640 1 -- 172.21.15.37:0/2306670105 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8e5004f510 con 0x7f8e50134b30 2024-04-26T00:22:32.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.434+0000 7f8e562e7640 1 --2- 172.21.15.37:0/2306670105 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e50134b30 0x7f8e50134f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:32.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.434+0000 7f8e562e7640 10 monclient: get_auth_request con 0x7f8e50134b30 auth_method 0 2024-04-26T00:22:32.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.434+0000 7f8e562e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:32.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.434+0000 7f8e562e7640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:32.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.434+0000 7f8e562e7640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:32.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.434+0000 7f8e562e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:32.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.434+0000 7f8e562e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:32.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.434+0000 7f8e562e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:32.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.434+0000 7f8e562e7640 10 monclient(hunting): handle_auth_done global_id 34133 payload 824 2024-04-26T00:22:32.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.434+0000 7f8e562e7640 1 -- 172.21.15.37:0/2306670105 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8e5013b930 con 0x7f8e50134b30 2024-04-26T00:22:32.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.438+0000 7f8e562e7640 1 -- 172.21.15.37:0/2306670105 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e50134b30 msgr2=0x7f8e50134f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:32.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.438+0000 7f8e562e7640 1 --2- 172.21.15.37:0/2306670105 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e50134b30 0x7f8e50134f40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8e40000c10 tx=0x7f8e400a8000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:32.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.438+0000 7f8e46ffd640 1 -- 172.21.15.37:0/2306670105 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e400ba070 con 0x7f8e50134b30 2024-04-26T00:22:32.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.438+0000 7f8e46ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:32.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.438+0000 7f8e46ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:22:32.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.438+0000 7f8e46ffd640 10 monclient: dump: 2024-04-26T00:22:32.425 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:32.425 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:32.425 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.425 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.425 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:32.425 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:32.426 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:32.426 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:32.426 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:32.426 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:32.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.438+0000 7f8e46ffd640 1 -- 172.21.15.37:0/2306670105 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7f8e400c1220 con 0x7f8e50134b30 2024-04-26T00:22:32.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.438+0000 7f8e46ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:22:32.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.438+0000 7f8e46ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:22:32.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.438+0000 7f8e46ffd640 1 -- 172.21.15.37:0/2306670105 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e50134b30 msgr2=0x7f8e50134f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:32.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.438+0000 7f8e46ffd640 1 --2- 172.21.15.37:0/2306670105 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e50134b30 0x7f8e50134f40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8e40000c10 tx=0x7f8e400a8000 comp rx=0 tx=0).stop 2024-04-26T00:22:32.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.438+0000 7f8e46ffd640 1 -- 172.21.15.37:0/2306670105 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e50134b30 msgr2=0x7f8e50134f40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:22:32.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.438+0000 7f8e46ffd640 1 --2- 172.21.15.37:0/2306670105 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e50134b30 0x7f8e50134f40 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8e40000c10 tx=0x7f8e400a8000 comp rx=0 tx=0).stop 2024-04-26T00:22:32.427 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:22:32.427 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:22:32.427 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:22:32.427 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:22:32.427 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 24, 2024-04-26T00:22:32.427 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 12715, 2024-04-26T00:22:32.427 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:22:32.427 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:22:32.427 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:22:32.427 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:22:32.428 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:22:32.428 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 6, 2024-04-26T00:22:32.428 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:22:32.428 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:22:32.428 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:22:32.428 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:22:32.428 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:22:32.428 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:22:32.428 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:22:32.428 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:22:32.428 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:22:32.429 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:22:32.429 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:22:32.429 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:22:32.429 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:22:32.429 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:22:32.429 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:22:32.429 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:22:32.429 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:22:32.429 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:22:32.429 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:22:32.429 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:22:32.430 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:22:32.430 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:22:32.430 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:22:32.430 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:22:32.430 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:22:32.430 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:22:32.430 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:22:32.430 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:22:32.430 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:32.430 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:32.430 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:22:32.430 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:22:32.431 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:22:32.431 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:22:32.431 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:22:32.431 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:22:32.431 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:22:32.431 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:22:32.431 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:22:32.431 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:22:32.431 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:22:32.431 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:22:32.431 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:22:32.432 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:22:32.432 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:22:32.432 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:22:32.432 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:22:32.432 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:22:32.432 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:22:32.432 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:22:32.432 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:22:32.432 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:22:32.432 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:22:32.432 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:22:32.432 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:22:32.433 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:22:32.433 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:22:32.433 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:22:32.433 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:22:32.433 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:22:32.433 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:32.433 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:22:32.433 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:32.433 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:22:32.433 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:22:32.433 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:22:32.434 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:22:32.434 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:32.434 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:22:32.434 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:22:32.434 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:32.434 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:32.434 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:32.434 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:22:32.434 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:22:32.434 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:32.434 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:32.434 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:32.434 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:32.435 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:22:32.435 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:22:32.435 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:22:32.435 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:22:32.435 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:22:32.435 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:32.435 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:32.435 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:22:32.435 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:22:32.435 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:22:32.435 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:22:32.435 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:32.436 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:22:32.436 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:22:32.436 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:32.436 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:32.436 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:32.436 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:22:32.436 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:22:32.436 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:32.436 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:32.436 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:32.436 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:32.437 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:22:32.437 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:22:32.437 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:22:32.437 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:22:32.437 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:22:32.437 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:32.437 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:32.437 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:22:32.437 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:22:32.437 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:22:32.437 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:22:32.437 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:32.437 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:22:32.438 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:22:32.438 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:32.438 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:32.438 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:32.438 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:22:32.438 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:22:32.438 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:32.438 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:32.438 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:32.438 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:32.438 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:22:32.438 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:22:32.439 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:22:32.439 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:22:32.439 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:22:32.439 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:32.439 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:32.439 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:32.439 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:22:32.439 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:22:32.439 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:32.439 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:22:32.439 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:22:32.439 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:22:32.440 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:32.440 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:22:32.440 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:22:32.440 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:32.440 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:22:32.440 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:22:32.440 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:22:32.440 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:32.440 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:32.440 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:32.440 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:22:32.440 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:22:32.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.438+0000 7f8e5854b640 1 -- 172.21.15.37:0/2306670105 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8e34035ed0 msgr2=0x7f8e34038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:32.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.438+0000 7f8e5854b640 1 --2- 172.21.15.37:0/2306670105 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8e34035ed0 0x7f8e34038390 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8e48001ea0 tx=0x7f8e48009040 comp rx=0 tx=0).stop 2024-04-26T00:22:32.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.438+0000 7f8e5854b640 10 monclient: shutdown 2024-04-26T00:22:32.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.438+0000 7f8e5854b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:22:32.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.438+0000 7f8e5854b640 1 -- 172.21.15.37:0/2306670105 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e500a4170 msgr2=0x7f8e5013e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:32.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.438+0000 7f8e56ae8640 1 -- 172.21.15.37:0/2306670105 reap_dead start 2024-04-26T00:22:32.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.438+0000 7f8e5854b640 1 --2- 172.21.15.37:0/2306670105 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e500a4170 0x7f8e5013e910 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c00b580 tx=0x7f8e4c00ba40 comp rx=0 tx=0).stop 2024-04-26T00:22:32.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.438+0000 7f8e5854b640 1 -- 172.21.15.37:0/2306670105 shutdown_connections 2024-04-26T00:22:32.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.438+0000 7f8e5854b640 1 --2- 172.21.15.37:0/2306670105 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e500a4170 0x7f8e5013e910 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:32.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.438+0000 7f8e5854b640 1 -- 172.21.15.37:0/2306670105 >> 172.21.15.37:0/2306670105 conn(0x7f8e500aa0a0 msgr2=0x7f8e500a2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:22:32.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.438+0000 7f8e5854b640 1 -- 172.21.15.37:0/2306670105 shutdown_connections 2024-04-26T00:22:32.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.438+0000 7f8e5854b640 1 -- 172.21.15.37:0/2306670105 wait complete. 2024-04-26T00:22:32.443 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:22:32.443 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:22:32.444 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:22:32.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.582+0000 7ff358bfb640 10 monclient: get_monmap_and_config 2024-04-26T00:22:32.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.582+0000 7ff358bfb640 10 monclient: build_initial_monmap 2024-04-26T00:22:32.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.582+0000 7ff358bfb640 10 monclient: monmap: 2024-04-26T00:22:32.560 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:22:32.560 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:32.560 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:22:32.584555+0000 2024-04-26T00:22:32.560 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:22:32.584555+0000 2024-04-26T00:22:32.560 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:22:32.560 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:32.560 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:22:32.560 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:22:32.561 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:22:32.561 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:22:32.561 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:22:32.561 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:32.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.582+0000 7ff358bfb640 1 Processor -- start 2024-04-26T00:22:32.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.582+0000 7ff358bfb640 1 -- start start 2024-04-26T00:22:32.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.582+0000 7ff358bfb640 10 monclient: init 2024-04-26T00:22:32.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.582+0000 7ff358bfb640 10 monclient: _reopen_session rank -1 2024-04-26T00:22:32.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.582+0000 7ff358bfb640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-04-26T00:22:32.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.582+0000 7ff358bfb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff354130eb0 addr v1:172.21.15.37:6789/0 2024-04-26T00:22:32.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.582+0000 7ff358bfb640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff35409ef20 0x7ff354138eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:32.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.582+0000 7ff358bfb640 10 monclient(hunting): picked mon.noname-b con 0x7ff35409ef20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:22:32.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.582+0000 7ff358bfb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff3541393f0 addr v1:172.21.15.45:6789/0 2024-04-26T00:22:32.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.582+0000 7ff358bfb640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff35409c190 con 0x7ff354130eb0 2024-04-26T00:22:32.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.582+0000 7ff358bfb640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3540a11c0 con 0x7ff3541393f0 2024-04-26T00:22:32.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.582+0000 7ff358bfb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:32.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.582+0000 7ff358bfb640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff35409df60 con 0x7ff35409ef20 2024-04-26T00:22:32.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.582+0000 7ff358bfb640 10 monclient(hunting): _renew_subs 2024-04-26T00:22:32.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.582+0000 7ff358bfb640 10 monclient(hunting): authenticate will time out at 1133.890257s 2024-04-26T00:22:32.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.582+0000 7ff35259c640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7ff354130eb0 0x7ff354131290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:51988/0 (socket says 172.21.15.37:51988) 2024-04-26T00:22:32.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.582+0000 7ff35259c640 1 -- 172.21.15.37:0/1447051157 learned_addr learned my addr 172.21.15.37:0/1447051157 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:22:32.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.582+0000 7ff351d9b640 1 --2- 172.21.15.37:0/1447051157 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff35409ef20 0x7ff354138eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:32.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.582+0000 7ff351d9b640 10 monclient(hunting): get_auth_request con 0x7ff35409ef20 auth_method 0 2024-04-26T00:22:32.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.582+0000 7ff351d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:32.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.582+0000 7ff351d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:32.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.582+0000 7ff351d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:32.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.582+0000 7ff351d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:32.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.582+0000 7ff351d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:32.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.582+0000 7ff351d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:32.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.582+0000 7ff351d9b640 10 monclient(hunting): handle_auth_done global_id 34136 payload 274 2024-04-26T00:22:32.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.582+0000 7ff351d9b640 1 -- 172.21.15.37:0/1447051157 >> v1:172.21.15.45:6789/0 conn(0x7ff3541393f0 legacy=0x7ff35413b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:22:32.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.582+0000 7ff351d9b640 1 -- 172.21.15.37:0/1447051157 >> v1:172.21.15.37:6789/0 conn(0x7ff354130eb0 legacy=0x7ff354131290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:22:32.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.582+0000 7ff351d9b640 10 monclient: _finish_hunting 0 2024-04-26T00:22:32.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.582+0000 7ff351d9b640 1 monclient: found mon.noname-b 2024-04-26T00:22:32.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.582+0000 7ff351d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:22:32.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.582+0000 7ff351d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:22:32.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.582+0000 7ff351d9b640 1 -- 172.21.15.37:0/1447051157 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff35409dc80 con 0x7ff35409ef20 2024-04-26T00:22:32.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.586+0000 7ff351d9b640 1 -- 172.21.15.37:0/1447051157 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff35409ef20 msgr2=0x7ff354138eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:32.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.586+0000 7ff351d9b640 1 --2- 172.21.15.37:0/1447051157 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff35409ef20 0x7ff354138eb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff344009dc0 tx=0x7ff34401c7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcc4ae63b61b6a52 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:32.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.586+0000 7ff35159a640 1 -- 172.21.15.37:0/1447051157 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3440b2070 con 0x7ff35409ef20 2024-04-26T00:22:32.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.586+0000 7ff35159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:32.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.586+0000 7ff35159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:22:32.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.586+0000 7ff35159a640 10 monclient: dump: 2024-04-26T00:22:32.575 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:32.575 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:32.575 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.575 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.575 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:32.575 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:32.575 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:32.575 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:32.575 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:32.575 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:32.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.586+0000 7ff35159a640 10 monclient: _finish_auth 0 2024-04-26T00:22:32.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.586+0000 7ff35159a640 10 monclient: _check_auth_tickets 2024-04-26T00:22:32.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.586+0000 7ff35159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:22:32.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.586+0000 7ff35159a640 1 -- 172.21.15.37:0/1447051157 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3440a58a0 con 0x7ff35409ef20 2024-04-26T00:22:32.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.586+0000 7ff358bfb640 5 monclient: authenticate success, global_id 34136 2024-04-26T00:22:32.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.586+0000 7ff358bfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:22:32.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.586+0000 7ff35159a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:22:32.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.586+0000 7ff358bfb640 10 monclient: get_monmap_and_config success 2024-04-26T00:22:32.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.586+0000 7ff358bfb640 4 set_mon_vals no callback set 2024-04-26T00:22:32.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.586+0000 7ff35159a640 1 -- 172.21.15.37:0/1447051157 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3440a5c70 con 0x7ff35409ef20 2024-04-26T00:22:32.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.586+0000 7ff35159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:32.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.586+0000 7ff35159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:22:32.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.586+0000 7ff35159a640 10 monclient: dump: 2024-04-26T00:22:32.576 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:32.577 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:32.577 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.577 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.577 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:32.577 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:32.577 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:32.577 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:32.577 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:32.577 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:32.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.586+0000 7ff358bfb640 10 monclient: shutdown 2024-04-26T00:22:32.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.586+0000 7ff358bfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:22:32.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.586+0000 7ff358bfb640 1 -- 172.21.15.37:0/1447051157 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff35409ef20 msgr2=0x7ff354138eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:32.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.586+0000 7ff358bfb640 1 --2- 172.21.15.37:0/1447051157 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff35409ef20 0x7ff354138eb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff344009dc0 tx=0x7ff34401c7a0 comp rx=0 tx=0).stop 2024-04-26T00:22:32.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff358bfb640 1 -- 172.21.15.37:0/1447051157 shutdown_connections 2024-04-26T00:22:32.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff358bfb640 1 --2- 172.21.15.37:0/1447051157 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff35409ef20 0x7ff354138eb0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:32.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff358bfb640 1 -- 172.21.15.37:0/1447051157 >> 172.21.15.37:0/1447051157 conn(0x7ff3540a1d50 msgr2=0x7ff3540a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:22:32.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff358bfb640 1 -- 172.21.15.37:0/1447051157 shutdown_connections 2024-04-26T00:22:32.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff358bfb640 1 -- 172.21.15.37:0/1447051157 wait complete. 2024-04-26T00:22:32.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff358bfb640 10 monclient: build_initial_monmap 2024-04-26T00:22:32.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff358bfb640 10 monclient: monmap: 2024-04-26T00:22:32.578 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:22:32.578 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:22:32.578 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:22:32.578 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:22:32.578 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:22:32.578 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:22:32.578 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:22:32.578 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:22:32.579 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:22:32.579 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:32.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff358bfb640 1 Processor -- start 2024-04-26T00:22:32.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff358bfb640 1 -- start start 2024-04-26T00:22:32.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff358bfb640 10 monclient: init 2024-04-26T00:22:32.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff358bfb640 10 monclient: _reopen_session rank -1 2024-04-26T00:22:32.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff358bfb640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:22:32.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff358bfb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff35409ef20 0x7ff354072050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:32.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff358bfb640 10 monclient(hunting): picked mon.noname-b con 0x7ff35409ef20 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:22:32.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff358bfb640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff354130eb0 0x7ff3541cf700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:32.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff358bfb640 10 monclient(hunting): picked mon.noname-a con 0x7ff354130eb0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:22:32.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff358bfb640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3541393f0 0x7ff3541cfc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:32.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff358bfb640 10 monclient(hunting): picked mon.noname-c con 0x7ff3541393f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:22:32.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff358bfb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:32.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff358bfb640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff35413f930 con 0x7ff354130eb0 2024-04-26T00:22:32.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff358bfb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:32.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff358bfb640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff35409bda0 con 0x7ff35409ef20 2024-04-26T00:22:32.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff358bfb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:32.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff358bfb640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff35413fab0 con 0x7ff3541393f0 2024-04-26T00:22:32.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff358bfb640 10 monclient(hunting): _renew_subs 2024-04-26T00:22:32.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff358bfb640 10 monclient(hunting): authenticate will time out at 1133.898116s 2024-04-26T00:22:32.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff351d9b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff354130eb0 0x7ff3541cf700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:32.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff352d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3541393f0 0x7ff3541cfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:32.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff351d9b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff354130eb0 0x7ff3541cf700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:45086/0 (socket says 172.21.15.37:45086) 2024-04-26T00:22:32.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff351d9b640 1 -- 172.21.15.37:0/3673638956 learned_addr learned my addr 172.21.15.37:0/3673638956 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:22:32.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff351d9b640 10 monclient(hunting): get_auth_request con 0x7ff354130eb0 auth_method 0 2024-04-26T00:22:32.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff351d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:32.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff351d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:32.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff351d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:32.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff352d9d640 10 monclient(hunting): get_auth_request con 0x7ff3541393f0 auth_method 0 2024-04-26T00:22:32.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff352d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:32.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff352d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:32.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff352d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:32.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff351d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:32.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff351d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:32.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff351d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:32.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff352d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:32.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff352d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:32.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff352d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:32.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff35259c640 1 --2- 172.21.15.37:0/3673638956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff35409ef20 0x7ff354072050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:32.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff35259c640 10 monclient(hunting): get_auth_request con 0x7ff35409ef20 auth_method 0 2024-04-26T00:22:32.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff35259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:32.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff35259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:32.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff35259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:32.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff352d9d640 10 monclient(hunting): handle_auth_done global_id 34139 payload 824 2024-04-26T00:22:32.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff352d9d640 1 -- 172.21.15.37:0/3673638956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff35409ef20 msgr2=0x7ff354072050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:32.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff352d9d640 1 --2- 172.21.15.37:0/3673638956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff35409ef20 0x7ff354072050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:32.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff352d9d640 1 -- 172.21.15.37:0/3673638956 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff354130eb0 msgr2=0x7ff3541cf700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:32.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff352d9d640 1 --2- 172.21.15.37:0/3673638956 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff354130eb0 0x7ff3541cf700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:32.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff352d9d640 10 monclient: _finish_hunting 0 2024-04-26T00:22:32.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff352d9d640 1 monclient: found mon.noname-c 2024-04-26T00:22:32.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff352d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:22:32.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff352d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:22:32.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff352d9d640 1 -- 172.21.15.37:0/3673638956 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3541d0180 con 0x7ff3541393f0 2024-04-26T00:22:32.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.590+0000 7ff351d9b640 1 --2- 172.21.15.37:0/3673638956 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff354130eb0 0x7ff3541cf700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:22:32.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff352d9d640 1 -- 172.21.15.37:0/3673638956 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3541393f0 msgr2=0x7ff3541cfc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:32.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff352d9d640 1 --2- 172.21.15.37:0/3673638956 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3541393f0 0x7ff3541cfc40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff34c00c3e0 tx=0x7ff34c00c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:32.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff34affd640 1 -- 172.21.15.37:0/3673638956 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff34c0119a0 con 0x7ff3541393f0 2024-04-26T00:22:32.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff34affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:32.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff34affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:22:32.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff34affd640 10 monclient: dump: 2024-04-26T00:22:32.583 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:32.583 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:32.583 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.583 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.583 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:32.583 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:32.584 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:32.584 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:32.584 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:32.584 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:32.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff34affd640 10 monclient: _finish_auth 0 2024-04-26T00:22:32.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff34affd640 10 monclient: _check_auth_tickets 2024-04-26T00:22:32.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff34affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:22:32.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff358bfb640 5 monclient: authenticate success, global_id 34139 2024-04-26T00:22:32.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff34affd640 1 -- 172.21.15.37:0/3673638956 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff34c017070 con 0x7ff3541393f0 2024-04-26T00:22:32.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff34affd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:22:32.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff34affd640 1 -- 172.21.15.37:0/3673638956 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff34c011d80 con 0x7ff3541393f0 2024-04-26T00:22:32.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff358bfb640 10 monclient: _renew_subs 2024-04-26T00:22:32.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff358bfb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:22:32.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff358bfb640 1 -- 172.21.15.37:0/3673638956 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff35412fb90 con 0x7ff3541393f0 2024-04-26T00:22:32.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff34bfff640 4 set_mon_vals no callback set 2024-04-26T00:22:32.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff34affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:32.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff34affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:22:32.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff34affd640 10 monclient: dump: 2024-04-26T00:22:32.585 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:32.585 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:32.585 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.585 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.585 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:32.585 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:32.585 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:32.586 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:32.586 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:32.586 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:32.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff358bfb640 10 monclient: _renew_subs 2024-04-26T00:22:32.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff358bfb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:22:32.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff358bfb640 1 -- 172.21.15.37:0/3673638956 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff354138a30 con 0x7ff3541393f0 2024-04-26T00:22:32.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff358bfb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:32.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff358bfb640 1 --2- 172.21.15.37:0/3673638956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff354133410 0x7ff354131a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:32.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff358bfb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:32.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff358bfb640 1 -- 172.21.15.37:0/3673638956 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff35404f510 con 0x7ff354133410 2024-04-26T00:22:32.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff34affd640 1 -- 172.21.15.37:0/3673638956 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff34c01c460 con 0x7ff3541393f0 2024-04-26T00:22:32.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff34affd640 1 --2- 172.21.15.37:0/3673638956 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff334035e80 0x7ff334038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:32.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff34affd640 1 -- 172.21.15.37:0/3673638956 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(111..111 src has 1..111) ==== 3963+0+0 (secure 0 0 0) 0x7ff34c0485d0 con 0x7ff3541393f0 2024-04-26T00:22:32.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff35259c640 1 --2- 172.21.15.37:0/3673638956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff354133410 0x7ff354131a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:32.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff35259c640 10 monclient: get_auth_request con 0x7ff354133410 auth_method 0 2024-04-26T00:22:32.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff35259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:32.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff35259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:32.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff35259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:32.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff35259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:32.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff35259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:32.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff35259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:32.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff351d9b640 1 --2- 172.21.15.37:0/3673638956 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff334035e80 0x7ff334038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:32.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff351d9b640 10 monclient: get_auth_request con 0x7ff334035e80 auth_method 0 2024-04-26T00:22:32.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff35259c640 10 monclient(hunting): handle_auth_done global_id 24205 payload 824 2024-04-26T00:22:32.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.594+0000 7ff35259c640 1 -- 172.21.15.37:0/3673638956 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff3540a0f60 con 0x7ff354133410 2024-04-26T00:22:32.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.598+0000 7ff35259c640 1 -- 172.21.15.37:0/3673638956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff354133410 msgr2=0x7ff354131a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:32.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.598+0000 7ff35259c640 1 --2- 172.21.15.37:0/3673638956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff354133410 0x7ff354131a00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff33c00e500 tx=0x7ff33c00e9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:32.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.598+0000 7ff351d9b640 1 --2- 172.21.15.37:0/3673638956 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff334035e80 0x7ff334038340 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff344000f90 tx=0x7ff3440a7070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:32.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.598+0000 7ff34affd640 1 -- 172.21.15.37:0/3673638956 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff33c004e40 con 0x7ff354133410 2024-04-26T00:22:32.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.598+0000 7ff34affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:32.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.598+0000 7ff34affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:22:32.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.598+0000 7ff34affd640 10 monclient: dump: 2024-04-26T00:22:32.590 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:32.590 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:32.590 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.591 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.591 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:32.591 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:32.591 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:32.591 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:32.591 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:32.591 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:32.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.598+0000 7ff34affd640 1 -- 172.21.15.37:0/3673638956 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff33c018070 con 0x7ff354133410 2024-04-26T00:22:32.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.598+0000 7ff34affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:32.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.598+0000 7ff34affd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:22:32.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.598+0000 7ff34affd640 1 -- 172.21.15.37:0/3673638956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff354133410 msgr2=0x7ff354131a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:32.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.598+0000 7ff34affd640 1 --2- 172.21.15.37:0/3673638956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff354133410 0x7ff354131a00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff33c00e500 tx=0x7ff33c00e9c0 comp rx=0 tx=0).stop 2024-04-26T00:22:32.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.598+0000 7ff34affd640 1 -- 172.21.15.37:0/3673638956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff354133410 msgr2=0x7ff354131a00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:22:32.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.598+0000 7ff34affd640 1 --2- 172.21.15.37:0/3673638956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff354133410 0x7ff354131a00 secure :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff33c00e500 tx=0x7ff33c00e9c0 comp rx=0 tx=0).stop 2024-04-26T00:22:32.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.606+0000 7ff358bfb640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:22:32.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.606+0000 7ff358bfb640 1 --2- 172.21.15.37:0/3673638956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff354131f40 0x7ff354132350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:32.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.606+0000 7ff358bfb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:32.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.606+0000 7ff358bfb640 1 -- 172.21.15.37:0/3673638956 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff354132890 con 0x7ff354131f40 2024-04-26T00:22:32.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.606+0000 7ff35259c640 1 --2- 172.21.15.37:0/3673638956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff354131f40 0x7ff354132350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:32.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.606+0000 7ff35259c640 10 monclient: get_auth_request con 0x7ff354131f40 auth_method 0 2024-04-26T00:22:32.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.606+0000 7ff35259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:32.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.606+0000 7ff35259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:32.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.606+0000 7ff35259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:32.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.606+0000 7ff35259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:32.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.606+0000 7ff35259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:32.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.606+0000 7ff35259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:32.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.606+0000 7ff35259c640 10 monclient(hunting): handle_auth_done global_id 24208 payload 824 2024-04-26T00:22:32.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.606+0000 7ff35259c640 1 -- 172.21.15.37:0/3673638956 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff354135420 con 0x7ff354131f40 2024-04-26T00:22:32.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.610+0000 7ff35259c640 1 -- 172.21.15.37:0/3673638956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff354131f40 msgr2=0x7ff354132350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:32.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.610+0000 7ff35259c640 1 --2- 172.21.15.37:0/3673638956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff354131f40 0x7ff354132350 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff33c00e4d0 tx=0x7ff33c004950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:32.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.610+0000 7ff34affd640 1 -- 172.21.15.37:0/3673638956 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff33c0064e0 con 0x7ff354131f40 2024-04-26T00:22:32.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.610+0000 7ff34affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:32.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.610+0000 7ff34affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:22:32.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.610+0000 7ff34affd640 10 monclient: dump: 2024-04-26T00:22:32.593 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:32.593 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:32.593 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.593 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.593 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:32.593 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:32.593 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:32.593 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:32.593 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:32.594 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:32.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.610+0000 7ff34affd640 1 -- 172.21.15.37:0/3673638956 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4716 (secure 0 0 0) 0x7ff33c0064e0 con 0x7ff354131f40 2024-04-26T00:22:32.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.610+0000 7ff34affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:22:32.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.610+0000 7ff34affd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:22:32.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.610+0000 7ff34affd640 1 -- 172.21.15.37:0/3673638956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff354131f40 msgr2=0x7ff354132350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:32.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.610+0000 7ff34affd640 1 --2- 172.21.15.37:0/3673638956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff354131f40 0x7ff354132350 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff33c00e4d0 tx=0x7ff33c004950 comp rx=0 tx=0).stop 2024-04-26T00:22:32.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.610+0000 7ff34affd640 1 -- 172.21.15.37:0/3673638956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff354131f40 msgr2=0x7ff354132350 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:22:32.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.610+0000 7ff34affd640 1 --2- 172.21.15.37:0/3673638956 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff354131f40 0x7ff354132350 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff33c00e4d0 tx=0x7ff33c004950 comp rx=0 tx=0).stop 2024-04-26T00:22:32.594 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:22:32.594 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:22:32.595 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:22:32.595 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:22:32.595 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 24, 2024-04-26T00:22:32.595 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 60773, 2024-04-26T00:22:32.595 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:22:32.595 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:22:32.595 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:22:32.595 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:22:32.595 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:22:32.595 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 7, 2024-04-26T00:22:32.595 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:22:32.595 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:22:32.595 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:22:32.595 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:22:32.596 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:22:32.596 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:22:32.596 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:22:32.596 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:22:32.596 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:22:32.596 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:22:32.596 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:22:32.596 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:22:32.596 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:22:32.596 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:22:32.596 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:22:32.596 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:22:32.596 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:22:32.596 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:22:32.597 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:22:32.597 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:22:32.597 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:22:32.597 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:22:32.597 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:22:32.597 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:22:32.597 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:22:32.597 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:22:32.597 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:22:32.597 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:22:32.597 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:32.597 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:32.597 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:22:32.598 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:22:32.598 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:22:32.598 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:22:32.598 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:22:32.598 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:22:32.598 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:22:32.598 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:22:32.598 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:22:32.598 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:22:32.598 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:22:32.598 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:22:32.598 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:22:32.598 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:22:32.599 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:22:32.599 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:22:32.599 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:22:32.599 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:22:32.599 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:22:32.599 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:22:32.599 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:22:32.599 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:22:32.599 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:22:32.599 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:22:32.599 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:22:32.599 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:22:32.600 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:22:32.600 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:22:32.600 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:22:32.600 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:22:32.600 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:32.600 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:22:32.600 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:32.600 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:22:32.600 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:22:32.600 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:22:32.600 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:22:32.600 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:32.600 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:22:32.601 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:22:32.601 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:32.601 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:32.601 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:32.601 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:22:32.601 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:22:32.601 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:32.601 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:32.601 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:32.601 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:32.601 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:22:32.601 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:22:32.601 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:22:32.602 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:22:32.602 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:22:32.602 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:32.602 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:32.602 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:22:32.602 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:22:32.602 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:22:32.602 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:22:32.603 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:32.603 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:22:32.603 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:22:32.603 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:32.603 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:32.603 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:32.603 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:22:32.603 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:22:32.603 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:32.603 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:32.603 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:32.603 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:32.603 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:22:32.603 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:22:32.604 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:22:32.604 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:22:32.604 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:22:32.604 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:32.604 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:32.604 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:22:32.604 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:22:32.604 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:22:32.604 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:22:32.604 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:32.604 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:22:32.604 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:22:32.604 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:32.605 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:32.605 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:32.605 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:22:32.605 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:22:32.605 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:32.605 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:32.605 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:32.605 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:32.605 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:22:32.605 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:22:32.605 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:22:32.605 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:22:32.605 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:22:32.606 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:32.606 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:32.606 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:32.606 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:22:32.606 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:22:32.606 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:32.606 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:22:32.606 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:22:32.606 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:22:32.606 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:32.606 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:22:32.606 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:22:32.606 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:32.607 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:22:32.607 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:22:32.607 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:22:32.607 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:32.607 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:22:32.607 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:22:32.607 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:32.607 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:22:32.607 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:22:32.607 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:22:32.607 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:32.607 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:32.607 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:32.607 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:22:32.608 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:22:32.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.610+0000 7ff358bfb640 1 -- 172.21.15.37:0/3673638956 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff334035e80 msgr2=0x7ff334038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:32.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.610+0000 7ff358bfb640 1 --2- 172.21.15.37:0/3673638956 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff334035e80 0x7ff334038340 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff344000f90 tx=0x7ff3440a7070 comp rx=0 tx=0).stop 2024-04-26T00:22:32.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.610+0000 7ff358bfb640 10 monclient: shutdown 2024-04-26T00:22:32.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.610+0000 7ff358bfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:22:32.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.610+0000 7ff358bfb640 1 -- 172.21.15.37:0/3673638956 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3541393f0 msgr2=0x7ff3541cfc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:32.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.610+0000 7ff358bfb640 1 --2- 172.21.15.37:0/3673638956 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3541393f0 0x7ff3541cfc40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff34c00c3e0 tx=0x7ff34c00c8a0 comp rx=0 tx=0).stop 2024-04-26T00:22:32.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.610+0000 7ff352d9d640 1 -- 172.21.15.37:0/3673638956 reap_dead start 2024-04-26T00:22:32.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.610+0000 7ff352d9d640 1 -- 172.21.15.37:0/3673638956 reap_dead start 2024-04-26T00:22:32.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.610+0000 7ff358bfb640 1 -- 172.21.15.37:0/3673638956 shutdown_connections 2024-04-26T00:22:32.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.610+0000 7ff358bfb640 1 -- 172.21.15.37:0/3673638956 >> 172.21.15.37:0/3673638956 conn(0x7ff3540a1d50 msgr2=0x7ff354131290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:22:32.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.610+0000 7ff358bfb640 1 -- 172.21.15.37:0/3673638956 shutdown_connections 2024-04-26T00:22:32.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.610+0000 7ff358bfb640 1 -- 172.21.15.37:0/3673638956 wait complete. 2024-04-26T00:22:32.610 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:22:32.610 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-04-26T00:22:32.610 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-26T00:22:32.611 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-04-26T00:22:32.611 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-04-26T00:22:32.611 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-26T00:22:32.611 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T00:22:32.624 INFO:tasks.ceph.mon.a:Stopped 2024-04-26T00:22:32.624 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-26T00:22:32.624 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:22:32.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f562c81c640 10 monclient: get_monmap_and_config 2024-04-26T00:22:32.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f562c81c640 10 monclient: build_initial_monmap 2024-04-26T00:22:32.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f562c81c640 10 monclient: monmap: 2024-04-26T00:22:32.788 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:22:32.788 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:32.788 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:22:32.813270+0000 2024-04-26T00:22:32.789 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:22:32.813270+0000 2024-04-26T00:22:32.789 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:22:32.789 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:32.789 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:22:32.789 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:22:32.789 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:22:32.789 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:22:32.789 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:22:32.789 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:32.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f562c81c640 1 Processor -- start 2024-04-26T00:22:32.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f562c81c640 1 -- start start 2024-04-26T00:22:32.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f562c81c640 10 monclient: init 2024-04-26T00:22:32.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f562c81c640 10 monclient: _reopen_session rank -1 2024-04-26T00:22:32.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f562c81c640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-04-26T00:22:32.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f562c81c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5628136da0 0x7f562809bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:32.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f562c81c640 10 monclient(hunting): picked mon.noname-c con 0x7f5628136da0 addr v2:172.21.15.45:3300/0 2024-04-26T00:22:32.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f562c81c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f562809c410 0x7f5628137180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:32.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f562c81c640 10 monclient(hunting): picked mon.noname-a con 0x7f562809c410 addr v2:172.21.15.37:3300/0 2024-04-26T00:22:32.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f562c81c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56281376c0 0x7f5628133b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:32.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f562c81c640 10 monclient(hunting): picked mon.noname-b con 0x7f56281376c0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:22:32.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f562c81c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:32.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f562c81c640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f562809b210 con 0x7f562809c410 2024-04-26T00:22:32.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f562c81c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:32.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f562c81c640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f562809b4f0 con 0x7f5628136da0 2024-04-26T00:22:32.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f562c81c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:32.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f562c81c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f562809b7d0 con 0x7f56281376c0 2024-04-26T00:22:32.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f562c81c640 10 monclient(hunting): _renew_subs 2024-04-26T00:22:32.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f562c81c640 10 monclient(hunting): authenticate will time out at 1134.118925s 2024-04-26T00:22:32.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f5625d9b640 1 -- >> v2:172.21.15.37:3300/0 conn(0x7f562809c410 msgr2=0x7f5628137180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:22:32.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f5625d9b640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f562809c410 0x7f5628137180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:22:32.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f5626d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56281376c0 0x7f5628133b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:32.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f5626d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56281376c0 0x7f5628133b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:53670/0 (socket says 172.21.15.37:53670) 2024-04-26T00:22:32.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f5626d9d640 1 -- 172.21.15.37:0/46253556 learned_addr learned my addr 172.21.15.37:0/46253556 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:22:32.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f5626d9d640 10 monclient(hunting): get_auth_request con 0x7f56281376c0 auth_method 0 2024-04-26T00:22:32.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f5626d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:32.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f5626d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:32.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f5626d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:32.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f5626d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:32.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f5626d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:32.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f5626d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:32.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f562659c640 1 --2- 172.21.15.37:0/46253556 >> v2:172.21.15.45:3300/0 conn(0x7f5628136da0 0x7f562809bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:32.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f562659c640 10 monclient(hunting): get_auth_request con 0x7f5628136da0 auth_method 0 2024-04-26T00:22:32.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f562659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:32.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f562659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:32.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f562659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:32.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f5626d9d640 10 monclient(hunting): handle_auth_done global_id 34142 payload 274 2024-04-26T00:22:32.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f5626d9d640 1 -- 172.21.15.37:0/46253556 >> v2:172.21.15.45:3300/0 conn(0x7f5628136da0 msgr2=0x7f562809bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:22:32.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f5626d9d640 1 --2- 172.21.15.37:0/46253556 >> v2:172.21.15.45:3300/0 conn(0x7f5628136da0 0x7f562809bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:32.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f5626d9d640 1 -- 172.21.15.37:0/46253556 >> v2:172.21.15.37:3300/0 conn(0x7f562809c410 msgr2=0x7f5628137180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:22:32.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f5626d9d640 1 --2- 172.21.15.37:0/46253556 >> v2:172.21.15.37:3300/0 conn(0x7f562809c410 0x7f5628137180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:32.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f5626d9d640 10 monclient: _finish_hunting 0 2024-04-26T00:22:32.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f5626d9d640 1 monclient: found mon.noname-b 2024-04-26T00:22:32.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f5626d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:22:32.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f5626d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:22:32.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.810+0000 7f5626d9d640 1 -- 172.21.15.37:0/46253556 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5628137b00 con 0x7f56281376c0 2024-04-26T00:22:32.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.814+0000 7f5626d9d640 1 -- 172.21.15.37:0/46253556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56281376c0 msgr2=0x7f5628133b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:32.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.814+0000 7f5626d9d640 1 --2- 172.21.15.37:0/46253556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56281376c0 0x7f5628133b90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f56280a58a0 tx=0x7f562001c610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=881afc57a6f00ef2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:32.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.814+0000 7f562559a640 1 -- 172.21.15.37:0/46253556 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56200a64c0 con 0x7f56281376c0 2024-04-26T00:22:32.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.814+0000 7f562559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:32.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.814+0000 7f562559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:22:32.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.814+0000 7f562559a640 10 monclient: dump: 2024-04-26T00:22:32.802 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:32.802 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:32.802 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.802 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.802 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:32.803 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:32.803 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:32.803 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:32.803 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:32.803 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:32.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562559a640 10 monclient: _finish_auth 0 2024-04-26T00:22:32.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562559a640 10 monclient: _check_auth_tickets 2024-04-26T00:22:32.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:22:32.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562c81c640 5 monclient: authenticate success, global_id 34142 2024-04-26T00:22:32.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562559a640 1 -- 172.21.15.37:0/46253556 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56200a6660 con 0x7f56281376c0 2024-04-26T00:22:32.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562c81c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:22:32.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562559a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:22:32.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562c81c640 10 monclient: get_monmap_and_config success 2024-04-26T00:22:32.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562c81c640 4 set_mon_vals no callback set 2024-04-26T00:22:32.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562559a640 1 -- 172.21.15.37:0/46253556 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56200a6a30 con 0x7f56281376c0 2024-04-26T00:22:32.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:32.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:22:32.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562559a640 10 monclient: dump: 2024-04-26T00:22:32.804 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:32.804 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:32.804 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.804 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.804 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:32.804 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:32.805 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:32.805 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:32.805 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:32.805 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:32.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562c81c640 10 monclient: shutdown 2024-04-26T00:22:32.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562c81c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:22:32.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562c81c640 1 -- 172.21.15.37:0/46253556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56281376c0 msgr2=0x7f5628133b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:32.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562c81c640 1 --2- 172.21.15.37:0/46253556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56281376c0 0x7f5628133b90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f56280a58a0 tx=0x7f562001c610 comp rx=0 tx=0).stop 2024-04-26T00:22:32.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562c81c640 1 -- 172.21.15.37:0/46253556 shutdown_connections 2024-04-26T00:22:32.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562c81c640 1 --2- 172.21.15.37:0/46253556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56281376c0 0x7f5628133b90 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:32.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562c81c640 1 --2- 172.21.15.37:0/46253556 >> v2:172.21.15.37:3300/0 conn(0x7f562809c410 0x7f5628137180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:32.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562c81c640 1 --2- 172.21.15.37:0/46253556 >> v2:172.21.15.45:3300/0 conn(0x7f5628136da0 0x7f562809bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:32.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562c81c640 1 -- 172.21.15.37:0/46253556 >> 172.21.15.37:0/46253556 conn(0x7f56280a5f00 msgr2=0x7f56280a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:22:32.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562c81c640 1 -- 172.21.15.37:0/46253556 shutdown_connections 2024-04-26T00:22:32.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562c81c640 1 -- 172.21.15.37:0/46253556 wait complete. 2024-04-26T00:22:32.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562c81c640 10 monclient: build_initial_monmap 2024-04-26T00:22:32.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562c81c640 10 monclient: monmap: 2024-04-26T00:22:32.806 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:22:32.806 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:22:32.806 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:22:32.806 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:22:32.806 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:22:32.806 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:22:32.807 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:22:32.807 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:22:32.807 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:22:32.807 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:32.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562c81c640 1 Processor -- start 2024-04-26T00:22:32.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562c81c640 1 -- start start 2024-04-26T00:22:32.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562c81c640 10 monclient: init 2024-04-26T00:22:32.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562c81c640 10 monclient: _reopen_session rank -1 2024-04-26T00:22:32.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562c81c640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:22:32.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562c81c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f562809c410 0x7f56281cc130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:32.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562c81c640 10 monclient(hunting): picked mon.noname-c con 0x7f562809c410 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:22:32.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562c81c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5628136da0 0x7f562813d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:32.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562c81c640 10 monclient(hunting): picked mon.noname-a con 0x7f5628136da0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:22:32.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562c81c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56281376c0 0x7f56281cf670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:32.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562c81c640 10 monclient(hunting): picked mon.noname-b con 0x7f56281376c0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:22:32.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562c81c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:32.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562c81c640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f5628140e90 con 0x7f5628136da0 2024-04-26T00:22:32.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562c81c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:32.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562c81c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5628140d10 con 0x7f56281376c0 2024-04-26T00:22:32.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562c81c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:32.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562c81c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5628141010 con 0x7f562809c410 2024-04-26T00:22:32.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562c81c640 10 monclient(hunting): _renew_subs 2024-04-26T00:22:32.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562c81c640 10 monclient(hunting): authenticate will time out at 1134.126426s 2024-04-26T00:22:32.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f5625d9b640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5628136da0 msgr2=0x7f562813d3d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:22:32.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f5625d9b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5628136da0 0x7f562813d3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:22:32.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562659c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f562809c410 0x7f56281cc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:32.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562659c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f562809c410 0x7f56281cc130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:53678/0 (socket says 172.21.15.37:53678) 2024-04-26T00:22:32.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562659c640 1 -- 172.21.15.37:0/705321025 learned_addr learned my addr 172.21.15.37:0/705321025 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:22:32.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562659c640 10 monclient(hunting): get_auth_request con 0x7f562809c410 auth_method 0 2024-04-26T00:22:32.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:32.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:32.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:32.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:32.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:32.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f5626d9d640 1 --2- 172.21.15.37:0/705321025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56281376c0 0x7f56281cf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:32.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:32.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f5626d9d640 10 monclient(hunting): get_auth_request con 0x7f56281376c0 auth_method 0 2024-04-26T00:22:32.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f5626d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:32.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f5626d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:32.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f5626d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:32.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562659c640 10 monclient(hunting): handle_auth_done global_id 34145 payload 824 2024-04-26T00:22:32.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562659c640 1 -- 172.21.15.37:0/705321025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56281376c0 msgr2=0x7f56281cf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:32.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562659c640 1 --2- 172.21.15.37:0/705321025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56281376c0 0x7f56281cf670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:32.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562659c640 1 -- 172.21.15.37:0/705321025 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5628136da0 msgr2=0x7f562813d3d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:22:32.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562659c640 1 --2- 172.21.15.37:0/705321025 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5628136da0 0x7f562813d3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:32.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562659c640 10 monclient: _finish_hunting 0 2024-04-26T00:22:32.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562659c640 1 monclient: found mon.noname-c 2024-04-26T00:22:32.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:22:32.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:22:32.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f562659c640 1 -- 172.21.15.37:0/705321025 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5628138870 con 0x7f562809c410 2024-04-26T00:22:32.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.818+0000 7f5626d9d640 1 --2- 172.21.15.37:0/705321025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56281376c0 0x7f56281cf670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:22:32.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.822+0000 7f562659c640 1 -- 172.21.15.37:0/705321025 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f562809c410 msgr2=0x7f56281cc130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:32.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.822+0000 7f562659c640 1 --2- 172.21.15.37:0/705321025 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f562809c410 0x7f56281cc130 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f561800d800 tx=0x7f561800dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:32.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.822+0000 7f5616ffd640 1 -- 172.21.15.37:0/705321025 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f561800b8d0 con 0x7f562809c410 2024-04-26T00:22:32.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.822+0000 7f5616ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:32.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.822+0000 7f5616ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:22:32.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.822+0000 7f5616ffd640 10 monclient: dump: 2024-04-26T00:22:32.811 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:32.811 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:32.811 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.811 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.812 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:32.812 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:32.812 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:32.812 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:32.812 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:32.812 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:32.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.822+0000 7f5616ffd640 10 monclient: _finish_auth 0 2024-04-26T00:22:32.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.822+0000 7f5616ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:22:32.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.822+0000 7f5616ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:22:32.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.822+0000 7f5616ffd640 1 -- 172.21.15.37:0/705321025 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5618019070 con 0x7f562809c410 2024-04-26T00:22:32.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.822+0000 7f5616ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:22:32.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.822+0000 7f5616ffd640 1 -- 172.21.15.37:0/705321025 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f561800bc70 con 0x7f562809c410 2024-04-26T00:22:32.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.822+0000 7f562c81c640 5 monclient: authenticate success, global_id 34145 2024-04-26T00:22:32.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.822+0000 7f562c81c640 10 monclient: _renew_subs 2024-04-26T00:22:32.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.822+0000 7f562c81c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:22:32.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.822+0000 7f562c81c640 1 -- 172.21.15.37:0/705321025 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56281395e0 con 0x7f562809c410 2024-04-26T00:22:32.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.822+0000 7f5616ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:32.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.822+0000 7f5617fff640 4 set_mon_vals no callback set 2024-04-26T00:22:32.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.822+0000 7f5616ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:22:32.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.822+0000 7f5616ffd640 10 monclient: dump: 2024-04-26T00:22:32.813 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:32.813 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:32.813 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.813 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:32.814 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:32.814 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:32.814 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:32.814 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:32.814 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:32.814 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:32.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.822+0000 7f562c81c640 10 monclient: _renew_subs 2024-04-26T00:22:32.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.822+0000 7f562c81c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:22:32.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.822+0000 7f562c81c640 1 -- 172.21.15.37:0/705321025 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5628139ba0 con 0x7f562809c410 2024-04-26T00:22:32.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.822+0000 7f5616ffd640 1 -- 172.21.15.37:0/705321025 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f561801e460 con 0x7f562809c410 2024-04-26T00:22:32.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.822+0000 7f562c81c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:32.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.822+0000 7f562c81c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:32.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.822+0000 7f562c81c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:22:32.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.822+0000 7f562c81c640 1 -- 172.21.15.37:0/705321025 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5628066f40 con 0x7f562809c410 2024-04-26T00:22:32.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.822+0000 7f5616ffd640 1 --2- 172.21.15.37:0/705321025 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5608035e80 0x7f5608038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:32.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.822+0000 7f5616ffd640 1 -- 172.21.15.37:0/705321025 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(111..111 src has 1..111) ==== 3963+0+0 (secure 0 0 0) 0x7f5618049030 con 0x7f562809c410 2024-04-26T00:22:32.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.822+0000 7f5625d9b640 1 --2- 172.21.15.37:0/705321025 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5608035e80 0x7f5608038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:32.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.822+0000 7f5625d9b640 10 monclient: get_auth_request con 0x7f5608035e80 auth_method 0 2024-04-26T00:22:32.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.826+0000 7f5625d9b640 1 --2- 172.21.15.37:0/705321025 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5608035e80 0x7f5608038340 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f561c006fd0 tx=0x7f561c006d40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:32.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.826+0000 7f5616ffd640 1 -- 172.21.15.37:0/705321025 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f561800be10 con 0x7f562809c410 2024-04-26T00:22:32.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.826+0000 7f5616ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:32.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:32.826+0000 7f5616ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:22:33.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:33.050+0000 7f562c81c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:22:33.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:33.050+0000 7f562c81c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:22:33.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:33.050+0000 7f562c81c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:22:33.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:33.050+0000 7f562c81c640 1 -- 172.21.15.37:0/705321025 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5628131aa0 con 0x7f562809c410 2024-04-26T00:22:33.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:33.050+0000 7f5616ffd640 1 -- 172.21.15.37:0/705321025 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f5618047d60 con 0x7f562809c410 2024-04-26T00:22:33.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:33.050+0000 7f5616ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:22:33.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:33.050+0000 7f5616ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:22:33.030 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":24,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:22:33.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:33.054+0000 7f562c81c640 1 -- 172.21.15.37:0/705321025 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5608035e80 msgr2=0x7f5608038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:33.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:33.054+0000 7f562c81c640 1 --2- 172.21.15.37:0/705321025 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5608035e80 0x7f5608038340 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f561c006fd0 tx=0x7f561c006d40 comp rx=0 tx=0).stop 2024-04-26T00:22:33.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:33.054+0000 7f562c81c640 10 monclient: shutdown 2024-04-26T00:22:33.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:33.054+0000 7f562c81c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:22:33.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:33.054+0000 7f562c81c640 1 -- 172.21.15.37:0/705321025 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f562809c410 msgr2=0x7f56281cc130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:33.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:33.054+0000 7f562c81c640 1 --2- 172.21.15.37:0/705321025 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f562809c410 0x7f56281cc130 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f561800d800 tx=0x7f561800dcc0 comp rx=0 tx=0).stop 2024-04-26T00:22:33.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:33.054+0000 7f562c81c640 1 -- 172.21.15.37:0/705321025 shutdown_connections 2024-04-26T00:22:33.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:33.054+0000 7f562c81c640 1 --2- 172.21.15.37:0/705321025 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5608035e80 0x7f5608038340 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:33.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:33.054+0000 7f562c81c640 1 --2- 172.21.15.37:0/705321025 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56281376c0 0x7f56281cf670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:33.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:33.054+0000 7f562c81c640 1 --2- 172.21.15.37:0/705321025 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5628136da0 0x7f562813d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:33.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:33.054+0000 7f562c81c640 1 --2- 172.21.15.37:0/705321025 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f562809c410 0x7f56281cc130 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:33.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:33.054+0000 7f562c81c640 1 -- 172.21.15.37:0/705321025 >> 172.21.15.37:0/705321025 conn(0x7f56280a5f00 msgr2=0x7f5628130bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:22:33.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:33.054+0000 7f562c81c640 1 -- 172.21.15.37:0/705321025 shutdown_connections 2024-04-26T00:22:33.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:33.054+0000 7f562c81c640 1 -- 172.21.15.37:0/705321025 wait complete. 2024-04-26T00:22:36.048 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:22:36.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2473893640 10 monclient: get_monmap_and_config 2024-04-26T00:22:36.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2473893640 10 monclient: build_initial_monmap 2024-04-26T00:22:36.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2473893640 10 monclient: monmap: 2024-04-26T00:22:36.180 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:22:36.181 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:36.181 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:22:36.205618+0000 2024-04-26T00:22:36.181 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:22:36.205618+0000 2024-04-26T00:22:36.181 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:22:36.181 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:36.181 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:22:36.181 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:22:36.181 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:22:36.181 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:22:36.181 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:22:36.181 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:36.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2473893640 1 Processor -- start 2024-04-26T00:22:36.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2473893640 1 -- start start 2024-04-26T00:22:36.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2473893640 10 monclient: init 2024-04-26T00:22:36.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2473893640 10 monclient: _reopen_session rank -1 2024-04-26T00:22:36.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2473893640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-04-26T00:22:36.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2473893640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f246c136e10 addr v1:172.21.15.37:6789/0 2024-04-26T00:22:36.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2473893640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f246c137900 0x7f246c09c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:36.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2473893640 10 monclient(hunting): picked mon.noname-a con 0x7f246c137900 addr v2:172.21.15.37:3300/0 2024-04-26T00:22:36.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2473893640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f246c13c340 0x7f246c13e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:36.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2473893640 10 monclient(hunting): picked mon.noname-b con 0x7f246c13c340 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:22:36.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2473893640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f246c09b680 con 0x7f246c136e10 2024-04-26T00:22:36.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2473893640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:36.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2473893640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f246c09b8f0 con 0x7f246c137900 2024-04-26T00:22:36.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2473893640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:36.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2473893640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f246c09bbd0 con 0x7f246c13c340 2024-04-26T00:22:36.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2473893640 10 monclient(hunting): _renew_subs 2024-04-26T00:22:36.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f247162f640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f246c136e10 legacy=0x7f246c1371f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:22:36.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2473893640 10 monclient(hunting): authenticate will time out at 1137.511381s 2024-04-26T00:22:36.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2470e2e640 1 -- >> v2:172.21.15.37:3300/0 conn(0x7f246c137900 msgr2=0x7f246c09c5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:22:36.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2470e2e640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f246c137900 0x7f246c09c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:22:36.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2471e30640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f246c13c340 0x7f246c13e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:36.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2471e30640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f246c13c340 0x7f246c13e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:53698/0 (socket says 172.21.15.37:53698) 2024-04-26T00:22:36.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2471e30640 1 -- 172.21.15.37:0/1253155928 learned_addr learned my addr 172.21.15.37:0/1253155928 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:22:36.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2471e30640 10 monclient(hunting): get_auth_request con 0x7f246c13c340 auth_method 0 2024-04-26T00:22:36.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2471e30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:36.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2471e30640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:36.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2471e30640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:36.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2471e30640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:36.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2471e30640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:36.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2471e30640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:36.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2471e30640 10 monclient(hunting): handle_auth_done global_id 34148 payload 274 2024-04-26T00:22:36.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2471e30640 1 -- 172.21.15.37:0/1253155928 >> v2:172.21.15.37:3300/0 conn(0x7f246c137900 msgr2=0x7f246c09c5b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:22:36.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2471e30640 1 --2- 172.21.15.37:0/1253155928 >> v2:172.21.15.37:3300/0 conn(0x7f246c137900 0x7f246c09c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:36.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2471e30640 1 -- 172.21.15.37:0/1253155928 >> v1:172.21.15.37:6789/0 conn(0x7f246c136e10 legacy=0x7f246c1371f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:22:36.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2471e30640 10 monclient: _finish_hunting 0 2024-04-26T00:22:36.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2471e30640 1 monclient: found mon.noname-b 2024-04-26T00:22:36.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2471e30640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:22:36.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2471e30640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:22:36.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.202+0000 7f2471e30640 1 -- 172.21.15.37:0/1253155928 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f246c0a5d90 con 0x7f246c13c340 2024-04-26T00:22:36.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2471e30640 1 -- 172.21.15.37:0/1253155928 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f246c13c340 msgr2=0x7f246c13e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:36.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2471e30640 1 --2- 172.21.15.37:0/1253155928 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f246c13c340 0x7f246c13e730 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2464009e50 tx=0x7f246401c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67d2115c019ccb55 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:36.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f245ffff640 1 -- 172.21.15.37:0/1253155928 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24640a5550 con 0x7f246c13c340 2024-04-26T00:22:36.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f245ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:36.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f245ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:22:36.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f245ffff640 10 monclient: dump: 2024-04-26T00:22:36.188 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:36.188 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:36.188 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:36.189 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:36.189 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:36.189 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:36.189 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:36.189 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:36.189 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:36.189 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:36.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f245ffff640 10 monclient: _finish_auth 0 2024-04-26T00:22:36.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f245ffff640 10 monclient: _check_auth_tickets 2024-04-26T00:22:36.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f245ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:22:36.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2473893640 5 monclient: authenticate success, global_id 34148 2024-04-26T00:22:36.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f245ffff640 1 -- 172.21.15.37:0/1253155928 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24640a56f0 con 0x7f246c13c340 2024-04-26T00:22:36.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f245ffff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:22:36.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2473893640 10 monclient: get_monmap_and_config success 2024-04-26T00:22:36.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f245ffff640 1 -- 172.21.15.37:0/1253155928 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24640a5ac0 con 0x7f246c13c340 2024-04-26T00:22:36.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f245ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:36.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2473893640 4 set_mon_vals no callback set 2024-04-26T00:22:36.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f245ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:22:36.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f245ffff640 10 monclient: dump: 2024-04-26T00:22:36.190 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:36.191 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:36.191 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:36.191 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:36.191 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:36.191 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:36.191 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:36.191 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:36.191 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:36.191 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:36.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2473893640 10 monclient: shutdown 2024-04-26T00:22:36.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2473893640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:22:36.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2473893640 1 -- 172.21.15.37:0/1253155928 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f246c13c340 msgr2=0x7f246c13e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:36.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2473893640 1 --2- 172.21.15.37:0/1253155928 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f246c13c340 0x7f246c13e730 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2464009e50 tx=0x7f246401c8d0 comp rx=0 tx=0).stop 2024-04-26T00:22:36.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2473893640 1 -- 172.21.15.37:0/1253155928 shutdown_connections 2024-04-26T00:22:36.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2473893640 1 --2- 172.21.15.37:0/1253155928 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f246c13c340 0x7f246c13e730 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:36.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2473893640 1 --2- 172.21.15.37:0/1253155928 >> v2:172.21.15.37:3300/0 conn(0x7f246c137900 0x7f246c09c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:36.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2473893640 1 -- 172.21.15.37:0/1253155928 >> 172.21.15.37:0/1253155928 conn(0x7f246c0a64e0 msgr2=0x7f246c0a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:22:36.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2473893640 1 -- 172.21.15.37:0/1253155928 shutdown_connections 2024-04-26T00:22:36.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2473893640 1 -- 172.21.15.37:0/1253155928 wait complete. 2024-04-26T00:22:36.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2473893640 10 monclient: build_initial_monmap 2024-04-26T00:22:36.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2473893640 10 monclient: monmap: 2024-04-26T00:22:36.192 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:22:36.192 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:22:36.192 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:22:36.192 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:22:36.192 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:22:36.192 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:22:36.192 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:22:36.192 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:22:36.192 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:22:36.192 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:36.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2473893640 1 Processor -- start 2024-04-26T00:22:36.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2473893640 1 -- start start 2024-04-26T00:22:36.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2473893640 10 monclient: init 2024-04-26T00:22:36.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2473893640 10 monclient: _reopen_session rank -1 2024-04-26T00:22:36.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2473893640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:22:36.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2473893640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f246c136e10 0x7f246c13b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:36.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2473893640 10 monclient(hunting): picked mon.noname-b con 0x7f246c136e10 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:22:36.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2473893640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f246c137900 0x7f246c135d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:36.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2473893640 10 monclient(hunting): picked mon.noname-c con 0x7f246c137900 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:22:36.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2473893640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f246c13c340 0x7f246c1ceb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:36.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2473893640 10 monclient(hunting): picked mon.noname-a con 0x7f246c13c340 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:22:36.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2473893640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:36.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2473893640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f246c141820 con 0x7f246c13c340 2024-04-26T00:22:36.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2473893640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:36.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2473893640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f246c1416a0 con 0x7f246c136e10 2024-04-26T00:22:36.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2473893640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:36.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2473893640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f246c1419a0 con 0x7f246c137900 2024-04-26T00:22:36.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2473893640 10 monclient(hunting): _renew_subs 2024-04-26T00:22:36.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2473893640 10 monclient(hunting): authenticate will time out at 1137.518783s 2024-04-26T00:22:36.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2471e30640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f246c13c340 msgr2=0x7f246c1ceb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:22:36.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2471e30640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f246c13c340 0x7f246c1ceb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:22:36.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2470e2e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f246c137900 0x7f246c135d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:36.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2470e2e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f246c137900 0x7f246c135d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:53702/0 (socket says 172.21.15.37:53702) 2024-04-26T00:22:36.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2470e2e640 1 -- 172.21.15.37:0/340550867 learned_addr learned my addr 172.21.15.37:0/340550867 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:22:36.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2470e2e640 10 monclient(hunting): get_auth_request con 0x7f246c137900 auth_method 0 2024-04-26T00:22:36.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2470e2e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:36.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2470e2e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:36.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2470e2e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:36.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2470e2e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:36.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2470e2e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:36.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2470e2e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:36.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f247162f640 1 --2- 172.21.15.37:0/340550867 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f246c136e10 0x7f246c13b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:36.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2470e2e640 10 monclient(hunting): handle_auth_done global_id 34151 payload 824 2024-04-26T00:22:36.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2470e2e640 1 -- 172.21.15.37:0/340550867 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f246c136e10 msgr2=0x7f246c13b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:36.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2470e2e640 1 --2- 172.21.15.37:0/340550867 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f246c136e10 0x7f246c13b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:36.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2470e2e640 1 -- 172.21.15.37:0/340550867 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f246c13c340 msgr2=0x7f246c1ceb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:22:36.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2470e2e640 1 --2- 172.21.15.37:0/340550867 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f246c13c340 0x7f246c1ceb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:36.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2470e2e640 10 monclient: _finish_hunting 0 2024-04-26T00:22:36.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2470e2e640 1 monclient: found mon.noname-c 2024-04-26T00:22:36.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2470e2e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:22:36.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2470e2e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:22:36.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f2470e2e640 1 -- 172.21.15.37:0/340550867 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f246c09aca0 con 0x7f246c137900 2024-04-26T00:22:36.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f247162f640 10 monclient: get_auth_request con 0x7f246c136e10 auth_method 0 2024-04-26T00:22:36.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.210+0000 7f247162f640 1 --2- 172.21.15.37:0/340550867 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f246c136e10 0x7f246c13b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:22:36.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.214+0000 7f2470e2e640 1 -- 172.21.15.37:0/340550867 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f246c137900 msgr2=0x7f246c135d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:36.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.214+0000 7f2470e2e640 1 --2- 172.21.15.37:0/340550867 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f246c137900 0x7f246c135d00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f246800b580 tx=0x7f246800ba40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:36.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.214+0000 7f245dffb640 1 -- 172.21.15.37:0/340550867 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2468017070 con 0x7f246c137900 2024-04-26T00:22:36.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.214+0000 7f245dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:36.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.214+0000 7f245dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:22:36.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.214+0000 7f245dffb640 10 monclient: dump: 2024-04-26T00:22:36.200 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:36.201 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:36.201 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:36.201 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:36.201 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:36.201 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:36.201 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:36.201 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:36.201 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:36.201 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:36.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.214+0000 7f245dffb640 10 monclient: _finish_auth 0 2024-04-26T00:22:36.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.214+0000 7f245dffb640 10 monclient: _check_auth_tickets 2024-04-26T00:22:36.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.214+0000 7f245dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:22:36.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.214+0000 7f2473893640 5 monclient: authenticate success, global_id 34151 2024-04-26T00:22:36.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.214+0000 7f2473893640 10 monclient: _renew_subs 2024-04-26T00:22:36.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.214+0000 7f245dffb640 1 -- 172.21.15.37:0/340550867 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2468003de0 con 0x7f246c137900 2024-04-26T00:22:36.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.214+0000 7f2473893640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:22:36.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.214+0000 7f2473893640 1 -- 172.21.15.37:0/340550867 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f246c13a5d0 con 0x7f246c137900 2024-04-26T00:22:36.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.214+0000 7f245dffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:22:36.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.214+0000 7f245dffb640 1 -- 172.21.15.37:0/340550867 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24680052f0 con 0x7f246c137900 2024-04-26T00:22:36.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.214+0000 7f245dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:36.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.214+0000 7f245dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:22:36.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.214+0000 7f245dffb640 10 monclient: dump: 2024-04-26T00:22:36.202 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:36.202 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:36.202 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:36.202 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:36.202 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:36.202 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:36.203 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:36.203 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:36.203 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:36.203 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:36.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.214+0000 7f245f7fe640 4 set_mon_vals no callback set 2024-04-26T00:22:36.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.214+0000 7f2473893640 10 monclient: _renew_subs 2024-04-26T00:22:36.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.214+0000 7f2473893640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:22:36.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.214+0000 7f2473893640 1 -- 172.21.15.37:0/340550867 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f246c13bb80 con 0x7f246c137900 2024-04-26T00:22:36.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.214+0000 7f2473893640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:36.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.214+0000 7f2473893640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:36.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.214+0000 7f2473893640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:22:36.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.214+0000 7f2473893640 1 -- 172.21.15.37:0/340550867 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f246c066f40 con 0x7f246c137900 2024-04-26T00:22:36.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.214+0000 7f245dffb640 1 -- 172.21.15.37:0/340550867 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f246801a070 con 0x7f246c137900 2024-04-26T00:22:36.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.214+0000 7f245dffb640 1 --2- 172.21.15.37:0/340550867 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2450035e80 0x7f2450038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:36.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.214+0000 7f245dffb640 1 -- 172.21.15.37:0/340550867 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(111..111 src has 1..111) ==== 3963+0+0 (secure 0 0 0) 0x7f2468048000 con 0x7f246c137900 2024-04-26T00:22:36.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.214+0000 7f247162f640 1 --2- 172.21.15.37:0/340550867 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2450035e80 0x7f2450038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:36.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.214+0000 7f247162f640 10 monclient: get_auth_request con 0x7f2450035e80 auth_method 0 2024-04-26T00:22:36.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.218+0000 7f247162f640 1 --2- 172.21.15.37:0/340550867 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2450035e80 0x7f2450038340 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2460006fd0 tx=0x7f2460008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:36.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.218+0000 7f245dffb640 1 -- 172.21.15.37:0/340550867 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f2468010a00 con 0x7f246c137900 2024-04-26T00:22:36.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.218+0000 7f245dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:36.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.218+0000 7f245dffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:22:36.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.450+0000 7f2473893640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:22:36.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.450+0000 7f2473893640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:22:36.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.450+0000 7f2473893640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:22:36.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.450+0000 7f2473893640 1 -- 172.21.15.37:0/340550867 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f246c120030 con 0x7f246c137900 2024-04-26T00:22:36.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.450+0000 7f245dffb640 1 -- 172.21.15.37:0/340550867 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f2468010c60 con 0x7f246c137900 2024-04-26T00:22:36.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.450+0000 7f245dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:22:36.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.450+0000 7f245dffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:22:36.428 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":24,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:22:36.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.450+0000 7f2473893640 1 -- 172.21.15.37:0/340550867 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2450035e80 msgr2=0x7f2450038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:36.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.450+0000 7f2473893640 1 --2- 172.21.15.37:0/340550867 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2450035e80 0x7f2450038340 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2460006fd0 tx=0x7f2460008040 comp rx=0 tx=0).stop 2024-04-26T00:22:36.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.450+0000 7f2473893640 10 monclient: shutdown 2024-04-26T00:22:36.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.450+0000 7f2473893640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:22:36.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.450+0000 7f2473893640 1 -- 172.21.15.37:0/340550867 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f246c137900 msgr2=0x7f246c135d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:36.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.450+0000 7f2473893640 1 --2- 172.21.15.37:0/340550867 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f246c137900 0x7f246c135d00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f246800b580 tx=0x7f246800ba40 comp rx=0 tx=0).stop 2024-04-26T00:22:36.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.450+0000 7f2473893640 1 -- 172.21.15.37:0/340550867 shutdown_connections 2024-04-26T00:22:36.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.450+0000 7f2473893640 1 --2- 172.21.15.37:0/340550867 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2450035e80 0x7f2450038340 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:36.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.450+0000 7f2473893640 1 --2- 172.21.15.37:0/340550867 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f246c13c340 0x7f246c1ceb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:36.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.450+0000 7f2473893640 1 --2- 172.21.15.37:0/340550867 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f246c137900 0x7f246c135d00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:36.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.450+0000 7f2473893640 1 --2- 172.21.15.37:0/340550867 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f246c136e10 0x7f246c13b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:36.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.450+0000 7f2473893640 1 -- 172.21.15.37:0/340550867 >> 172.21.15.37:0/340550867 conn(0x7f246c0a64e0 msgr2=0x7f246c0a8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:22:36.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.450+0000 7f2473893640 1 -- 172.21.15.37:0/340550867 shutdown_connections 2024-04-26T00:22:36.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:36.450+0000 7f2473893640 1 -- 172.21.15.37:0/340550867 wait complete. 2024-04-26T00:22:39.448 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:22:39.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.602+0000 7f909df6f640 10 monclient: get_monmap_and_config 2024-04-26T00:22:39.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.602+0000 7f909df6f640 10 monclient: build_initial_monmap 2024-04-26T00:22:39.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.602+0000 7f909df6f640 10 monclient: monmap: 2024-04-26T00:22:39.580 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:22:39.581 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:39.581 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:22:39.605413+0000 2024-04-26T00:22:39.581 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:22:39.605413+0000 2024-04-26T00:22:39.581 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:22:39.581 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:39.581 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:22:39.581 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:22:39.581 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:22:39.581 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:22:39.581 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:22:39.581 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:39.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.602+0000 7f909df6f640 1 Processor -- start 2024-04-26T00:22:39.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.602+0000 7f909df6f640 1 -- start start 2024-04-26T00:22:39.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.602+0000 7f909df6f640 10 monclient: init 2024-04-26T00:22:39.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.602+0000 7f909df6f640 10 monclient: _reopen_session rank -1 2024-04-26T00:22:39.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.602+0000 7f909df6f640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-04-26T00:22:39.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.602+0000 7f909df6f640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f90980a0260 0x7f90980a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:39.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.602+0000 7f909df6f640 10 monclient(hunting): picked mon.noname-c con 0x7f90980a0260 addr v2:172.21.15.45:3300/0 2024-04-26T00:22:39.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.602+0000 7f909df6f640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9098134b30 0x7f9098134f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:39.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.602+0000 7f909df6f640 10 monclient(hunting): picked mon.noname-a con 0x7f9098134b30 addr v2:172.21.15.37:3300/0 2024-04-26T00:22:39.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.602+0000 7f909df6f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9098135490 addr v1:172.21.15.45:6789/0 2024-04-26T00:22:39.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.602+0000 7f909df6f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90980a1130 con 0x7f9098135490 2024-04-26T00:22:39.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.602+0000 7f909df6f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:39.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.602+0000 7f909df6f640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f909809ec10 con 0x7f9098134b30 2024-04-26T00:22:39.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.602+0000 7f909df6f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:39.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.602+0000 7f909df6f640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f909809f0f0 con 0x7f90980a0260 2024-04-26T00:22:39.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.602+0000 7f909df6f640 10 monclient(hunting): _renew_subs 2024-04-26T00:22:39.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.602+0000 7f909df6f640 10 monclient(hunting): authenticate will time out at 1140.911115s 2024-04-26T00:22:39.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.602+0000 7f9096ffd640 1 -- >> v2:172.21.15.37:3300/0 conn(0x7f9098134b30 msgr2=0x7f9098134f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:22:39.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.602+0000 7f9096ffd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9098134b30 0x7f9098134f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:22:39.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.602+0000 7f90977fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f90980a0260 0x7f90980a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:39.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.602+0000 7f9097fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f9098135490 0x7f909813be60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.37:46074/0 (socket says 172.21.15.37:46074) 2024-04-26T00:22:39.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.602+0000 7f90977fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f90980a0260 0x7f90980a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.37:33026/0 (socket says 172.21.15.37:33026) 2024-04-26T00:22:39.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.602+0000 7f9097fff640 1 -- 172.21.15.37:0/1875420309 learned_addr learned my addr 172.21.15.37:0/1875420309 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:22:39.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.602+0000 7f90977fe640 10 monclient(hunting): get_auth_request con 0x7f90980a0260 auth_method 0 2024-04-26T00:22:39.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.602+0000 7f90977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:39.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.602+0000 7f90977fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:39.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.602+0000 7f90977fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:39.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.602+0000 7f90977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:39.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.602+0000 7f90977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:39.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.602+0000 7f90977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:39.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.606+0000 7f90977fe640 10 monclient(hunting): handle_auth_done global_id 24217 payload 274 2024-04-26T00:22:39.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.606+0000 7f90977fe640 1 -- 172.21.15.37:0/1875420309 >> v2:172.21.15.37:3300/0 conn(0x7f9098134b30 msgr2=0x7f9098134f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:22:39.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.606+0000 7f90977fe640 1 --2- 172.21.15.37:0/1875420309 >> v2:172.21.15.37:3300/0 conn(0x7f9098134b30 0x7f9098134f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:39.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.606+0000 7f90977fe640 1 -- 172.21.15.37:0/1875420309 >> v1:172.21.15.45:6789/0 conn(0x7f9098135490 legacy=0x7f909813be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:22:39.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.606+0000 7f90977fe640 10 monclient: _finish_hunting 0 2024-04-26T00:22:39.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.606+0000 7f90977fe640 1 monclient: found mon.noname-c 2024-04-26T00:22:39.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.606+0000 7f90977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:22:39.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.606+0000 7f90977fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:22:39.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.606+0000 7f90977fe640 1 -- 172.21.15.37:0/1875420309 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f909806cff0 con 0x7f90980a0260 2024-04-26T00:22:39.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f90977fe640 1 -- 172.21.15.37:0/1875420309 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90980a0260 msgr2=0x7f90980a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:39.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f90977fe640 1 --2- 172.21.15.37:0/1875420309 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90980a0260 0x7f90980a0640 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f90980a2ad0 tx=0x7f908001c140 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e7d4ab8cd782c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:39.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f90967fc640 1 -- 172.21.15.37:0/1875420309 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90800b1070 con 0x7f90980a0260 2024-04-26T00:22:39.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f90967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:39.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f90967fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:22:39.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f90967fc640 10 monclient: dump: 2024-04-26T00:22:39.588 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:39.588 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:39.588 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:39.588 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:39.588 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:39.588 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:39.588 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:39.588 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:39.588 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:39.589 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:39.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f90967fc640 10 monclient: _finish_auth 0 2024-04-26T00:22:39.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f90967fc640 10 monclient: _check_auth_tickets 2024-04-26T00:22:39.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f90967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:22:39.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f90967fc640 1 -- 172.21.15.37:0/1875420309 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90800ac440 con 0x7f90980a0260 2024-04-26T00:22:39.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f90967fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:22:39.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f90967fc640 1 -- 172.21.15.37:0/1875420309 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90800ac810 con 0x7f90980a0260 2024-04-26T00:22:39.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f909df6f640 5 monclient: authenticate success, global_id 24217 2024-04-26T00:22:39.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f909df6f640 10 monclient: get_monmap_and_config success 2024-04-26T00:22:39.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f909df6f640 4 set_mon_vals no callback set 2024-04-26T00:22:39.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f90967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:39.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f90967fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:22:39.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f90967fc640 10 monclient: dump: 2024-04-26T00:22:39.589 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:39.590 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:39.590 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:39.590 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:39.590 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:39.590 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:39.590 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:39.590 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:39.590 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:39.590 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:39.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f909df6f640 10 monclient: shutdown 2024-04-26T00:22:39.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f909df6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:22:39.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f909df6f640 1 -- 172.21.15.37:0/1875420309 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90980a0260 msgr2=0x7f90980a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:39.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f909df6f640 1 --2- 172.21.15.37:0/1875420309 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90980a0260 0x7f90980a0640 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f90980a2ad0 tx=0x7f908001c140 comp rx=0 tx=0).stop 2024-04-26T00:22:39.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f909df6f640 1 -- 172.21.15.37:0/1875420309 shutdown_connections 2024-04-26T00:22:39.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f909df6f640 1 --2- 172.21.15.37:0/1875420309 >> v2:172.21.15.37:3300/0 conn(0x7f9098134b30 0x7f9098134f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:39.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f909df6f640 1 --2- 172.21.15.37:0/1875420309 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90980a0260 0x7f90980a0640 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:39.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f909df6f640 1 -- 172.21.15.37:0/1875420309 >> 172.21.15.37:0/1875420309 conn(0x7f90980a1ad0 msgr2=0x7f90980a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:22:39.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f909df6f640 1 -- 172.21.15.37:0/1875420309 shutdown_connections 2024-04-26T00:22:39.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f909df6f640 1 -- 172.21.15.37:0/1875420309 wait complete. 2024-04-26T00:22:39.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f909df6f640 10 monclient: build_initial_monmap 2024-04-26T00:22:39.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f909df6f640 10 monclient: monmap: 2024-04-26T00:22:39.596 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:22:39.596 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:22:39.596 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:22:39.596 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:22:39.596 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:22:39.596 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:22:39.596 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:22:39.596 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:22:39.596 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:22:39.596 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:39.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f909df6f640 1 Processor -- start 2024-04-26T00:22:39.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f909df6f640 1 -- start start 2024-04-26T00:22:39.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f909df6f640 10 monclient: init 2024-04-26T00:22:39.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f909df6f640 10 monclient: _reopen_session rank -1 2024-04-26T00:22:39.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f909df6f640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:22:39.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f909df6f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f90980a0260 0x7f90981322b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:39.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f909df6f640 10 monclient(hunting): picked mon.noname-c con 0x7f90980a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:22:39.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f909df6f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9098134b30 0x7f909809c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:39.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f909df6f640 10 monclient(hunting): picked mon.noname-a con 0x7f9098134b30 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:22:39.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f909df6f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9098135490 0x7f909809ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:39.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f909df6f640 10 monclient(hunting): picked mon.noname-b con 0x7f9098135490 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:22:39.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f909df6f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:39.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f909df6f640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f909809f430 con 0x7f9098134b30 2024-04-26T00:22:39.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f909df6f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:39.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f909df6f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f909809f2b0 con 0x7f9098135490 2024-04-26T00:22:39.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f909df6f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:39.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f909df6f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f909809fe90 con 0x7f90980a0260 2024-04-26T00:22:39.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f909df6f640 10 monclient(hunting): _renew_subs 2024-04-26T00:22:39.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f909df6f640 10 monclient(hunting): authenticate will time out at 1140.919543s 2024-04-26T00:22:39.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f9096ffd640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9098134b30 msgr2=0x7f909809c4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:22:39.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f9096ffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9098134b30 0x7f909809c4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:22:39.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f90977fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f90980a0260 0x7f90981322b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:39.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f90977fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f90980a0260 0x7f90981322b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:53718/0 (socket says 172.21.15.37:53718) 2024-04-26T00:22:39.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f90977fe640 1 -- 172.21.15.37:0/546704805 learned_addr learned my addr 172.21.15.37:0/546704805 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:22:39.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f90977fe640 10 monclient(hunting): get_auth_request con 0x7f90980a0260 auth_method 0 2024-04-26T00:22:39.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f90977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:39.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f90977fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:39.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f90977fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:39.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f90977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:39.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f90977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:39.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.610+0000 7f90977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:39.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f9097fff640 1 --2- 172.21.15.37:0/546704805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9098135490 0x7f909809ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:39.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f9097fff640 10 monclient(hunting): get_auth_request con 0x7f9098135490 auth_method 0 2024-04-26T00:22:39.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f9097fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:39.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f9097fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:39.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f9097fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:39.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f90977fe640 10 monclient(hunting): handle_auth_done global_id 34154 payload 824 2024-04-26T00:22:39.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f90977fe640 1 -- 172.21.15.37:0/546704805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9098135490 msgr2=0x7f909809ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:39.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f90977fe640 1 --2- 172.21.15.37:0/546704805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9098135490 0x7f909809ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:39.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f90977fe640 1 -- 172.21.15.37:0/546704805 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9098134b30 msgr2=0x7f909809c4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:22:39.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f90977fe640 1 --2- 172.21.15.37:0/546704805 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9098134b30 0x7f909809c4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:39.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f90977fe640 10 monclient: _finish_hunting 0 2024-04-26T00:22:39.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f90977fe640 1 monclient: found mon.noname-c 2024-04-26T00:22:39.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f90977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:22:39.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f90977fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:22:39.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f90977fe640 1 -- 172.21.15.37:0/546704805 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f909813b670 con 0x7f90980a0260 2024-04-26T00:22:39.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f90977fe640 1 -- 172.21.15.37:0/546704805 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f90980a0260 msgr2=0x7f90981322b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:39.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f90977fe640 1 --2- 172.21.15.37:0/546704805 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f90980a0260 0x7f90981322b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9080018d50 tx=0x7f9080018ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:39.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f908ffff640 1 -- 172.21.15.37:0/546704805 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90800ba020 con 0x7f90980a0260 2024-04-26T00:22:39.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f908ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:39.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f908ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:22:39.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f908ffff640 10 monclient: dump: 2024-04-26T00:22:39.600 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:39.600 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:39.600 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:39.600 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:39.600 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:39.601 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:39.601 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:39.601 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:39.601 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:39.601 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:39.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f908ffff640 10 monclient: _finish_auth 0 2024-04-26T00:22:39.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f908ffff640 10 monclient: _check_auth_tickets 2024-04-26T00:22:39.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f908ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:22:39.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f909df6f640 5 monclient: authenticate success, global_id 34154 2024-04-26T00:22:39.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f908ffff640 1 -- 172.21.15.37:0/546704805 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90800b66a0 con 0x7f90980a0260 2024-04-26T00:22:39.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f909df6f640 10 monclient: _renew_subs 2024-04-26T00:22:39.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f909df6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:22:39.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f909df6f640 1 -- 172.21.15.37:0/546704805 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f909809e150 con 0x7f90980a0260 2024-04-26T00:22:39.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f908ffff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:22:39.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f908ffff640 1 -- 172.21.15.37:0/546704805 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90800b1040 con 0x7f90980a0260 2024-04-26T00:22:39.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f908ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:39.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f908ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:22:39.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f908ffff640 10 monclient: dump: 2024-04-26T00:22:39.602 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:39.602 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:39.602 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:39.602 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:39.602 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:39.602 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:39.602 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:39.602 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:39.602 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:39.602 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:39.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f909df6f640 10 monclient: _renew_subs 2024-04-26T00:22:39.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f9095ffb640 4 set_mon_vals no callback set 2024-04-26T00:22:39.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f909df6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:22:39.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f909df6f640 1 -- 172.21.15.37:0/546704805 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f909813bd00 con 0x7f90980a0260 2024-04-26T00:22:39.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f909df6f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:39.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f909df6f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:39.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f909df6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:22:39.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f908ffff640 1 -- 172.21.15.37:0/546704805 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f90800bd020 con 0x7f90980a0260 2024-04-26T00:22:39.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f909df6f640 1 -- 172.21.15.37:0/546704805 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9098066f40 con 0x7f90980a0260 2024-04-26T00:22:39.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f908ffff640 1 --2- 172.21.15.37:0/546704805 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9078035e80 0x7f9078038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:39.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.614+0000 7f908ffff640 1 -- 172.21.15.37:0/546704805 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(111..111 src has 1..111) ==== 3963+0+0 (secure 0 0 0) 0x7f90800e2b40 con 0x7f90980a0260 2024-04-26T00:22:39.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.618+0000 7f9096ffd640 1 --2- 172.21.15.37:0/546704805 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9078035e80 0x7f9078038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:39.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.618+0000 7f9096ffd640 10 monclient: get_auth_request con 0x7f9078035e80 auth_method 0 2024-04-26T00:22:39.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.618+0000 7f9096ffd640 1 --2- 172.21.15.37:0/546704805 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9078035e80 0x7f9078038340 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9088006fd0 tx=0x7f9088006d40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:39.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.622+0000 7f908ffff640 1 -- 172.21.15.37:0/546704805 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f90800c4030 con 0x7f90980a0260 2024-04-26T00:22:39.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.622+0000 7f908ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:39.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.622+0000 7f908ffff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:22:39.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.842+0000 7f909df6f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:22:39.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.842+0000 7f909df6f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:22:39.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.842+0000 7f909df6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:22:39.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.842+0000 7f909df6f640 1 -- 172.21.15.37:0/546704805 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f909813b280 con 0x7f90980a0260 2024-04-26T00:22:39.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.842+0000 7f908ffff640 1 -- 172.21.15.37:0/546704805 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f90980a1130 con 0x7f90980a0260 2024-04-26T00:22:39.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.842+0000 7f908ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:22:39.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.842+0000 7f908ffff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:22:39.822 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":24,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:22:39.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.846+0000 7f909df6f640 1 -- 172.21.15.37:0/546704805 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9078035e80 msgr2=0x7f9078038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:39.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.846+0000 7f909df6f640 1 --2- 172.21.15.37:0/546704805 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9078035e80 0x7f9078038340 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9088006fd0 tx=0x7f9088006d40 comp rx=0 tx=0).stop 2024-04-26T00:22:39.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.846+0000 7f909df6f640 10 monclient: shutdown 2024-04-26T00:22:39.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.846+0000 7f909df6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:22:39.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.846+0000 7f909df6f640 1 -- 172.21.15.37:0/546704805 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f90980a0260 msgr2=0x7f90981322b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:39.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.846+0000 7f909df6f640 1 --2- 172.21.15.37:0/546704805 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f90980a0260 0x7f90981322b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9080018d50 tx=0x7f9080018ee0 comp rx=0 tx=0).stop 2024-04-26T00:22:39.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.846+0000 7f909df6f640 1 -- 172.21.15.37:0/546704805 shutdown_connections 2024-04-26T00:22:39.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.846+0000 7f909df6f640 1 --2- 172.21.15.37:0/546704805 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9078035e80 0x7f9078038340 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:39.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.846+0000 7f909df6f640 1 --2- 172.21.15.37:0/546704805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9098135490 0x7f909809ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:39.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.846+0000 7f909df6f640 1 --2- 172.21.15.37:0/546704805 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9098134b30 0x7f909809c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:39.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.846+0000 7f909df6f640 1 --2- 172.21.15.37:0/546704805 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f90980a0260 0x7f90981322b0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:39.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.846+0000 7f909df6f640 1 -- 172.21.15.37:0/546704805 >> 172.21.15.37:0/546704805 conn(0x7f90980a1ad0 msgr2=0x7f90980a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:22:39.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.846+0000 7f909df6f640 1 -- 172.21.15.37:0/546704805 shutdown_connections 2024-04-26T00:22:39.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:39.846+0000 7f909df6f640 1 -- 172.21.15.37:0/546704805 wait complete. 2024-04-26T00:22:42.840 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:22:42.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:42.994+0000 7f78fa8e3640 10 monclient: get_monmap_and_config 2024-04-26T00:22:42.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:42.994+0000 7f78fa8e3640 10 monclient: build_initial_monmap 2024-04-26T00:22:42.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:42.994+0000 7f78fa8e3640 10 monclient: monmap: 2024-04-26T00:22:42.974 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:22:42.974 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:42.974 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:22:42.998022+0000 2024-04-26T00:22:42.974 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:22:42.998022+0000 2024-04-26T00:22:42.974 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:22:42.974 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:42.974 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:22:42.974 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:22:42.974 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:22:42.974 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:22:42.974 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:22:42.974 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:42.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:42.994+0000 7f78fa8e3640 1 Processor -- start 2024-04-26T00:22:42.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:42.994+0000 7f78fa8e3640 1 -- start start 2024-04-26T00:22:42.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:42.994+0000 7f78fa8e3640 10 monclient: init 2024-04-26T00:22:42.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:42.994+0000 7f78fa8e3640 10 monclient: _reopen_session rank -1 2024-04-26T00:22:42.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:42.994+0000 7f78fa8e3640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-04-26T00:22:42.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:42.994+0000 7f78fa8e3640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f78f4133b20 0x7f78f409e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:42.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:42.994+0000 7f78fa8e3640 10 monclient(hunting): picked mon.noname-a con 0x7f78f4133b20 addr v2:172.21.15.37:3300/0 2024-04-26T00:22:42.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:42.994+0000 7f78fa8e3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f78f4133f00 addr v1:172.21.15.37:6789/0 2024-04-26T00:22:42.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:42.994+0000 7f78fa8e3640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f78f4139050 0x7f78f413b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:42.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:42.994+0000 7f78fa8e3640 10 monclient(hunting): picked mon.noname-c con 0x7f78f4139050 addr v2:172.21.15.45:3300/0 2024-04-26T00:22:42.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:42.994+0000 7f78fa8e3640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78f409fe20 con 0x7f78f4133f00 2024-04-26T00:22:42.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:42.994+0000 7f78fa8e3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:42.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:42.994+0000 7f78fa8e3640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f78f4072d00 con 0x7f78f4133b20 2024-04-26T00:22:42.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:42.994+0000 7f78fa8e3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:42.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:42.994+0000 7f78fa8e3640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f78f409ba60 con 0x7f78f4139050 2024-04-26T00:22:42.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:42.994+0000 7f78fa8e3640 10 monclient(hunting): _renew_subs 2024-04-26T00:22:42.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:42.994+0000 7f78fa8e3640 10 monclient(hunting): authenticate will time out at 1144.303780s 2024-04-26T00:22:42.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:42.994+0000 7f78f3fff640 1 -- >> v2:172.21.15.37:3300/0 conn(0x7f78f4133b20 msgr2=0x7f78f409e9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:22:42.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:42.994+0000 7f78f3fff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f78f4133b20 0x7f78f409e9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:22:42.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:42.994+0000 7f78f37fe640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f78f4133f00 legacy=0x7f78f4134350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:22:42.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:42.998+0000 7f78f8e80640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f78f4139050 0x7f78f413b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:42.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:42.998+0000 7f78f8e80640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f78f4139050 0x7f78f413b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.37:51104/0 (socket says 172.21.15.37:51104) 2024-04-26T00:22:42.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:42.998+0000 7f78f8e80640 1 -- 172.21.15.37:0/924154804 learned_addr learned my addr 172.21.15.37:0/924154804 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:22:42.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:42.998+0000 7f78f8e80640 10 monclient(hunting): get_auth_request con 0x7f78f4139050 auth_method 0 2024-04-26T00:22:42.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:42.998+0000 7f78f8e80640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:42.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:42.998+0000 7f78f8e80640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:42.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:42.998+0000 7f78f8e80640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:42.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:42.998+0000 7f78f8e80640 1 -- 172.21.15.37:0/924154804 >> v2:172.21.15.45:3300/0 conn(0x7f78f4139050 msgr2=0x7f78f413b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:22:42.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:42.998+0000 7f78f8e80640 1 -- 172.21.15.37:0/924154804 >> v2:172.21.15.45:3300/0 conn(0x7f78f4139050 msgr2=0x7f78f413b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:22:42.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:42.998+0000 7f78f8e80640 1 --2- 172.21.15.37:0/924154804 >> v2:172.21.15.45:3300/0 conn(0x7f78f4139050 0x7f78f413b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:22:42.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:42.998+0000 7f78f8e80640 1 --2- 172.21.15.37:0/924154804 >> v2:172.21.15.45:3300/0 conn(0x7f78f4139050 0x7f78f413b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:22:43.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:43.198+0000 7f78f3fff640 1 -- 172.21.15.37:0/924154804 >> v2:172.21.15.37:3300/0 conn(0x7f78f4133b20 msgr2=0x7f78f409e9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:22:43.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:43.198+0000 7f78f37fe640 1 -- 172.21.15.37:0/924154804 >> v1:172.21.15.37:6789/0 conn(0x7f78f4133f00 legacy=0x7f78f4134350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:22:43.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:43.198+0000 7f78f3fff640 1 --2- 172.21.15.37:0/924154804 >> v2:172.21.15.37:3300/0 conn(0x7f78f4133b20 0x7f78f409e9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:22:43.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:43.202+0000 7f78f8e80640 1 --2- 172.21.15.37:0/924154804 >> v2:172.21.15.45:3300/0 conn(0x7f78f4139050 0x7f78f413b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:43.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:43.202+0000 7f78f8e80640 10 monclient(hunting): get_auth_request con 0x7f78f4139050 auth_method 0 2024-04-26T00:22:43.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:43.202+0000 7f78f8e80640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:43.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:43.202+0000 7f78f8e80640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:43.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:43.202+0000 7f78f8e80640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:22:43.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:43.202+0000 7f78f8e80640 1 -- 172.21.15.37:0/924154804 >> v2:172.21.15.45:3300/0 conn(0x7f78f4139050 msgr2=0x7f78f413b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:22:43.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:43.202+0000 7f78f8e80640 1 -- 172.21.15.37:0/924154804 >> v2:172.21.15.45:3300/0 conn(0x7f78f4139050 msgr2=0x7f78f413b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:22:43.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:43.202+0000 7f78f8e80640 1 --2- 172.21.15.37:0/924154804 >> v2:172.21.15.45:3300/0 conn(0x7f78f4139050 0x7f78f413b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:22:43.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:43.202+0000 7f78f8e80640 1 --2- 172.21.15.37:0/924154804 >> v2:172.21.15.45:3300/0 conn(0x7f78f4139050 0x7f78f413b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:22:43.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:43.602+0000 7f78f37fe640 1 -- 172.21.15.37:0/924154804 >> v1:172.21.15.37:6789/0 conn(0x7f78f4133f00 legacy=0x7f78f4134350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:22:43.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:43.602+0000 7f78f3fff640 1 -- 172.21.15.37:0/924154804 >> v2:172.21.15.37:3300/0 conn(0x7f78f4133b20 msgr2=0x7f78f409e9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:22:43.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:43.602+0000 7f78f3fff640 1 --2- 172.21.15.37:0/924154804 >> v2:172.21.15.37:3300/0 conn(0x7f78f4133b20 0x7f78f409e9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:22:43.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:43.606+0000 7f78f8e80640 1 --2- 172.21.15.37:0/924154804 >> v2:172.21.15.45:3300/0 conn(0x7f78f4139050 0x7f78f413b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:43.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:43.606+0000 7f78f8e80640 10 monclient(hunting): get_auth_request con 0x7f78f4139050 auth_method 0 2024-04-26T00:22:43.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:43.606+0000 7f78f8e80640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:43.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:43.606+0000 7f78f8e80640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:43.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:43.606+0000 7f78f8e80640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:22:43.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:43.606+0000 7f78f8e80640 1 -- 172.21.15.37:0/924154804 >> v2:172.21.15.45:3300/0 conn(0x7f78f4139050 msgr2=0x7f78f413b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:22:43.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:43.606+0000 7f78f8e80640 1 -- 172.21.15.37:0/924154804 >> v2:172.21.15.45:3300/0 conn(0x7f78f4139050 msgr2=0x7f78f413b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:22:43.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:43.606+0000 7f78f8e80640 1 --2- 172.21.15.37:0/924154804 >> v2:172.21.15.45:3300/0 conn(0x7f78f4139050 0x7f78f413b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:22:43.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:43.606+0000 7f78f8e80640 1 --2- 172.21.15.37:0/924154804 >> v2:172.21.15.45:3300/0 conn(0x7f78f4139050 0x7f78f413b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:22:44.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:44.406+0000 7f78f37fe640 1 -- 172.21.15.37:0/924154804 >> v1:172.21.15.37:6789/0 conn(0x7f78f4133f00 legacy=0x7f78f4134350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:22:44.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:44.406+0000 7f78f3fff640 1 -- 172.21.15.37:0/924154804 >> v2:172.21.15.37:3300/0 conn(0x7f78f4133b20 msgr2=0x7f78f409e9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:22:44.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:44.406+0000 7f78f3fff640 1 --2- 172.21.15.37:0/924154804 >> v2:172.21.15.37:3300/0 conn(0x7f78f4133b20 0x7f78f409e9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:22:44.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:44.410+0000 7f78f8e80640 1 --2- 172.21.15.37:0/924154804 >> v2:172.21.15.45:3300/0 conn(0x7f78f4139050 0x7f78f413b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:44.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:44.410+0000 7f78f8e80640 10 monclient(hunting): get_auth_request con 0x7f78f4139050 auth_method 0 2024-04-26T00:22:44.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:44.410+0000 7f78f8e80640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:44.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:44.410+0000 7f78f8e80640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:44.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:44.410+0000 7f78f8e80640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:22:44.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:44.410+0000 7f78f8e80640 1 -- 172.21.15.37:0/924154804 >> v2:172.21.15.45:3300/0 conn(0x7f78f4139050 msgr2=0x7f78f413b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:22:44.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:44.410+0000 7f78f8e80640 1 -- 172.21.15.37:0/924154804 >> v2:172.21.15.45:3300/0 conn(0x7f78f4139050 msgr2=0x7f78f413b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:22:44.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:44.410+0000 7f78f8e80640 1 --2- 172.21.15.37:0/924154804 >> v2:172.21.15.45:3300/0 conn(0x7f78f4139050 0x7f78f413b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:22:44.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:44.410+0000 7f78f8e80640 1 --2- 172.21.15.37:0/924154804 >> v2:172.21.15.45:3300/0 conn(0x7f78f4139050 0x7f78f413b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:22:45.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:45.994+0000 7f78f1ffb640 10 monclient(hunting): tick 2024-04-26T00:22:45.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:45.994+0000 7f78f1ffb640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:22:45.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:45.994+0000 7f78f1ffb640 1 monclient(hunting): continuing hunt 2024-04-26T00:22:45.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:45.994+0000 7f78f1ffb640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:22:45.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:45.994+0000 7f78f1ffb640 1 -- 172.21.15.37:0/924154804 >> v2:172.21.15.45:3300/0 conn(0x7f78f4139050 msgr2=0x7f78f413b440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:22:45.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:45.994+0000 7f78f1ffb640 1 --2- 172.21.15.37:0/924154804 >> v2:172.21.15.45:3300/0 conn(0x7f78f4139050 0x7f78f413b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:45.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:45.994+0000 7f78f1ffb640 1 -- 172.21.15.37:0/924154804 >> v2:172.21.15.37:3300/0 conn(0x7f78f4133b20 msgr2=0x7f78f409e9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:22:45.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:45.994+0000 7f78f1ffb640 1 --2- 172.21.15.37:0/924154804 >> v2:172.21.15.37:3300/0 conn(0x7f78f4133b20 0x7f78f409e9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:45.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:45.994+0000 7f78f1ffb640 1 -- 172.21.15.37:0/924154804 >> v1:172.21.15.37:6789/0 conn(0x7f78f4133f00 legacy=0x7f78f4134350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:22:45.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:45.994+0000 7f78f1ffb640 10 monclient: _add_conns ranks=[1,2] 2024-04-26T00:22:45.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:45.994+0000 7f78f1ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f78e00036a0 addr v1:172.21.15.45:6789/0 2024-04-26T00:22:45.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:45.994+0000 7f78f1ffb640 1 --2- 172.21.15.37:0/924154804 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78e0007370 0x7f78e0009830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:45.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:45.998+0000 7f78f1ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f78e0007370 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:22:45.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:45.998+0000 7f78f1ffb640 1 -- 172.21.15.37:0/924154804 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78f409dd10 con 0x7f78e00036a0 2024-04-26T00:22:45.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:45.998+0000 7f78f1ffb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:45.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:45.998+0000 7f78f1ffb640 1 -- 172.21.15.37:0/924154804 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f78f4072d00 con 0x7f78e0007370 2024-04-26T00:22:45.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:45.998+0000 7f78f1ffb640 10 monclient(hunting): _renew_subs 2024-04-26T00:22:45.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:45.998+0000 7f78f37fe640 1 --2- 172.21.15.37:0/924154804 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78e0007370 0x7f78e0009830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:45.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:45.998+0000 7f78f37fe640 10 monclient(hunting): get_auth_request con 0x7f78e0007370 auth_method 0 2024-04-26T00:22:45.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:45.998+0000 7f78f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:45.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:45.998+0000 7f78f37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:45.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:45.998+0000 7f78f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:45.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:45.998+0000 7f78f37fe640 1 -- 172.21.15.37:0/924154804 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78e0007370 msgr2=0x7f78e0009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:22:45.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:45.998+0000 7f78f37fe640 1 -- 172.21.15.37:0/924154804 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78e0007370 msgr2=0x7f78e0009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:22:45.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:45.998+0000 7f78f37fe640 1 --2- 172.21.15.37:0/924154804 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78e0007370 0x7f78e0009830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:22:45.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:45.998+0000 7f78f37fe640 1 --2- 172.21.15.37:0/924154804 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78e0007370 0x7f78e0009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:22:45.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:45.998+0000 7f78f3fff640 1 -- 172.21.15.37:0/924154804 >> v1:172.21.15.45:6789/0 conn(0x7f78e00036a0 legacy=0x7f78e0005b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:46.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:46.202+0000 7f78f37fe640 1 --2- 172.21.15.37:0/924154804 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78e0007370 0x7f78e0009830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:46.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:46.202+0000 7f78f37fe640 10 monclient(hunting): get_auth_request con 0x7f78e0007370 auth_method 0 2024-04-26T00:22:46.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:46.202+0000 7f78f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:46.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:46.202+0000 7f78f37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:46.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:46.202+0000 7f78f37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:22:46.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:46.202+0000 7f78f37fe640 1 -- 172.21.15.37:0/924154804 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78e0007370 msgr2=0x7f78e0009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:22:46.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:46.202+0000 7f78f37fe640 1 -- 172.21.15.37:0/924154804 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78e0007370 msgr2=0x7f78e0009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:22:46.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:46.202+0000 7f78f37fe640 1 --2- 172.21.15.37:0/924154804 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78e0007370 0x7f78e0009830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:22:46.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:46.202+0000 7f78f37fe640 1 --2- 172.21.15.37:0/924154804 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78e0007370 0x7f78e0009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:22:46.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:46.606+0000 7f78f37fe640 1 --2- 172.21.15.37:0/924154804 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78e0007370 0x7f78e0009830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:46.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:46.606+0000 7f78f37fe640 10 monclient(hunting): get_auth_request con 0x7f78e0007370 auth_method 0 2024-04-26T00:22:46.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:46.606+0000 7f78f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:46.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:46.606+0000 7f78f37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:46.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:46.606+0000 7f78f37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:22:46.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:46.606+0000 7f78f37fe640 1 -- 172.21.15.37:0/924154804 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78e0007370 msgr2=0x7f78e0009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:22:46.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:46.606+0000 7f78f37fe640 1 -- 172.21.15.37:0/924154804 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78e0007370 msgr2=0x7f78e0009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:22:46.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:46.606+0000 7f78f37fe640 1 --2- 172.21.15.37:0/924154804 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78e0007370 0x7f78e0009830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:22:46.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:46.606+0000 7f78f37fe640 1 --2- 172.21.15.37:0/924154804 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78e0007370 0x7f78e0009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:22:47.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.410+0000 7f78f37fe640 1 --2- 172.21.15.37:0/924154804 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78e0007370 0x7f78e0009830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:47.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.410+0000 7f78f37fe640 10 monclient(hunting): get_auth_request con 0x7f78e0007370 auth_method 0 2024-04-26T00:22:47.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.410+0000 7f78f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:47.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.410+0000 7f78f37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:47.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.410+0000 7f78f37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:22:47.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.410+0000 7f78f37fe640 1 -- 172.21.15.37:0/924154804 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78e0007370 msgr2=0x7f78e0009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:22:47.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.410+0000 7f78f37fe640 1 -- 172.21.15.37:0/924154804 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78e0007370 msgr2=0x7f78e0009830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:22:47.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.410+0000 7f78f37fe640 1 --2- 172.21.15.37:0/924154804 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78e0007370 0x7f78e0009830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:22:47.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.410+0000 7f78f37fe640 1 --2- 172.21.15.37:0/924154804 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78e0007370 0x7f78e0009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:22:47.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f2ffd640 1 -- 172.21.15.37:0/924154804 <== mon.1 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 375663715 0 0) 0x7f78f409dd10 con 0x7f78e00036a0 2024-04-26T00:22:47.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f2ffd640 10 monclient(hunting): _negotiate 2024-04-26T00:22:47.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:47.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:47.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f2ffd640 10 monclient(hunting): my global_id is 24223 2024-04-26T00:22:47.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f2ffd640 1 -- 172.21.15.37:0/924154804 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78dc003530 con 0x7f78e00036a0 2024-04-26T00:22:47.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f2ffd640 1 -- 172.21.15.37:0/924154804 <== mon.1 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2101702838 0 0) 0x7f78dc003530 con 0x7f78e00036a0 2024-04-26T00:22:47.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f2ffd640 1 -- 172.21.15.37:0/924154804 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78e0007370 msgr2=0x7f78e0009830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:22:47.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f2ffd640 1 --2- 172.21.15.37:0/924154804 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78e0007370 0x7f78e0009830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:47.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f2ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:22:47.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f2ffd640 1 monclient: found mon.noname-c-legacy 2024-04-26T00:22:47.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:22:47.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f2ffd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.45:6789/0 2024-04-26T00:22:47.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f2ffd640 1 -- 172.21.15.37:0/924154804 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78e000b030 con 0x7f78e00036a0 2024-04-26T00:22:47.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f2ffd640 10 monclient: _finish_auth 0 2024-04-26T00:22:47.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f2ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:22:47.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:22:47.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f2ffd640 1 -- 172.21.15.37:0/924154804 <== mon.1 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 866326386 0 0) 0x7f78e4004370 con 0x7f78e00036a0 2024-04-26T00:22:47.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:47.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f2ffd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-04-26T00:22:47.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f2ffd640 10 monclient: dump: 2024-04-26T00:22:47.428 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:47.428 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:47.428 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:47.428 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:47.428 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:47.428 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:47.428 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:47.428 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:47.428 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:47.428 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:47.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f2ffd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] but i'm connected to v1:172.21.15.45:6789/0, reconnecting 2024-04-26T00:22:47.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f2ffd640 10 monclient: _reopen_session rank -1 2024-04-26T00:22:47.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f2ffd640 1 -- 172.21.15.37:0/924154804 >> v1:172.21.15.45:6789/0 conn(0x7f78e00036a0 legacy=0x7f78e0005b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:47.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f8e80640 1 -- 172.21.15.37:0/924154804 reap_dead start 2024-04-26T00:22:47.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f2ffd640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:22:47.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f2ffd640 1 --2- 172.21.15.37:0/924154804 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78dc0039c0 0x7f78dc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:47.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f2ffd640 10 monclient(hunting): picked mon.c con 0x7f78dc0039c0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:22:47.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f2ffd640 1 --2- 172.21.15.37:0/924154804 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78dc004810 0x7f78dc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:47.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f2ffd640 10 monclient(hunting): picked mon.a con 0x7f78dc004810 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:22:47.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f2ffd640 1 --2- 172.21.15.37:0/924154804 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78dc005220 0x7f78dc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:47.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f2ffd640 10 monclient(hunting): picked mon.b con 0x7f78dc005220 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:22:47.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f2ffd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:47.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f2ffd640 1 -- 172.21.15.37:0/924154804 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f78f4072d00 con 0x7f78dc004810 2024-04-26T00:22:47.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f2ffd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:47.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f2ffd640 1 -- 172.21.15.37:0/924154804 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f78dc00c150 con 0x7f78dc005220 2024-04-26T00:22:47.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f2ffd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:47.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f2ffd640 1 -- 172.21.15.37:0/924154804 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f78dc00c350 con 0x7f78dc0039c0 2024-04-26T00:22:47.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f2ffd640 10 monclient(hunting): _renew_subs 2024-04-26T00:22:47.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f37fe640 1 -- 172.21.15.37:0/924154804 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78dc004810 msgr2=0x7f78dc004c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:22:47.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f2ffd640 10 monclient(hunting): _finish_auth 0 2024-04-26T00:22:47.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f37fe640 1 --2- 172.21.15.37:0/924154804 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78dc004810 0x7f78dc004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:22:47.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f3fff640 1 --2- 172.21.15.37:0/924154804 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78dc0039c0 0x7f78dc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:47.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f3fff640 10 monclient(hunting): get_auth_request con 0x7f78dc0039c0 auth_method 0 2024-04-26T00:22:47.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:47.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:47.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:22:47.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f8e80640 1 --2- 172.21.15.37:0/924154804 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78dc005220 0x7f78dc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:47.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f8e80640 10 monclient(hunting): get_auth_request con 0x7f78dc005220 auth_method 0 2024-04-26T00:22:47.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f8e80640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:47.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f8e80640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:47.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f8e80640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:22:47.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:47.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:47.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:22:47.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f8e80640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:47.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f8e80640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:47.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f8e80640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:22:47.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f3fff640 10 monclient(hunting): handle_auth_done global_id 24223 payload 293 2024-04-26T00:22:47.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f3fff640 1 -- 172.21.15.37:0/924154804 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78dc005220 msgr2=0x7f78dc00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:22:47.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f3fff640 1 --2- 172.21.15.37:0/924154804 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78dc005220 0x7f78dc00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:47.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f3fff640 1 -- 172.21.15.37:0/924154804 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78dc004810 msgr2=0x7f78dc004c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:22:47.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f3fff640 1 --2- 172.21.15.37:0/924154804 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78dc004810 0x7f78dc004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:47.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f3fff640 10 monclient: _finish_hunting 0 2024-04-26T00:22:47.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f3fff640 1 monclient: found mon.c 2024-04-26T00:22:47.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:22:47.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f3fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:22:47.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f3fff640 1 -- 172.21.15.37:0/924154804 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78dc003530 con 0x7f78dc0039c0 2024-04-26T00:22:47.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f3fff640 10 monclient: _finish_auth 0 2024-04-26T00:22:47.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f3fff640 10 monclient: _check_auth_tickets 2024-04-26T00:22:47.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:22:47.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78fa8e3640 5 monclient: authenticate success, global_id 24223 2024-04-26T00:22:47.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.446+0000 7f78fa8e3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:22:47.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.454+0000 7f78f3fff640 1 -- 172.21.15.37:0/924154804 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78dc0039c0 msgr2=0x7f78dc003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:47.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.454+0000 7f78f3fff640 1 --2- 172.21.15.37:0/924154804 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78dc0039c0 0x7f78dc003dc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f78e4004590 tx=0x7f78e401cca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ba9f62b381b322b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:47.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.458+0000 7f78f2ffd640 1 -- 172.21.15.37:0/924154804 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e400c0e0 con 0x7f78dc0039c0 2024-04-26T00:22:47.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.458+0000 7f78f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:47.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.458+0000 7f78f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:22:47.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.458+0000 7f78f2ffd640 10 monclient: dump: 2024-04-26T00:22:47.436 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:47.436 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:47.436 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:47.436 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:47.436 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:47.436 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:47.436 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:47.436 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:47.436 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:47.436 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:47.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.458+0000 7f78fa8e3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:22:47.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.458+0000 7f78f2ffd640 1 -- 172.21.15.37:0/924154804 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78e40a9070 con 0x7f78dc0039c0 2024-04-26T00:22:47.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.458+0000 7f78f2ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:22:47.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.458+0000 7f78f2ffd640 1 -- 172.21.15.37:0/924154804 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e4006070 con 0x7f78dc0039c0 2024-04-26T00:22:47.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.458+0000 7f78fa8e3640 10 monclient: get_monmap_and_config success 2024-04-26T00:22:47.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.458+0000 7f78fa8e3640 4 set_mon_vals no callback set 2024-04-26T00:22:47.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.458+0000 7f78f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:47.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.458+0000 7f78f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:22:47.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.458+0000 7f78f2ffd640 10 monclient: dump: 2024-04-26T00:22:47.438 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:47.438 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:47.438 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:47.438 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:47.438 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:47.439 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:47.439 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:47.439 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:47.439 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:47.439 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:47.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.458+0000 7f78fa8e3640 10 monclient: shutdown 2024-04-26T00:22:47.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.458+0000 7f78fa8e3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:22:47.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.458+0000 7f78fa8e3640 1 -- 172.21.15.37:0/924154804 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78dc0039c0 msgr2=0x7f78dc003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:47.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.458+0000 7f78fa8e3640 1 --2- 172.21.15.37:0/924154804 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78dc0039c0 0x7f78dc003dc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f78e4004590 tx=0x7f78e401cca0 comp rx=0 tx=0).stop 2024-04-26T00:22:47.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.458+0000 7f78fa8e3640 1 -- 172.21.15.37:0/924154804 shutdown_connections 2024-04-26T00:22:47.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.458+0000 7f78fa8e3640 1 --2- 172.21.15.37:0/924154804 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78dc005220 0x7f78dc00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:47.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.458+0000 7f78fa8e3640 1 --2- 172.21.15.37:0/924154804 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78dc004810 0x7f78dc004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:47.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.458+0000 7f78fa8e3640 1 --2- 172.21.15.37:0/924154804 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78dc0039c0 0x7f78dc003dc0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:47.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.458+0000 7f78fa8e3640 1 -- 172.21.15.37:0/924154804 >> 172.21.15.37:0/924154804 conn(0x7f78f40a1ad0 msgr2=0x7f78f40a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:22:47.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.458+0000 7f78fa8e3640 1 -- 172.21.15.37:0/924154804 shutdown_connections 2024-04-26T00:22:47.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.458+0000 7f78fa8e3640 1 -- 172.21.15.37:0/924154804 wait complete. 2024-04-26T00:22:47.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78fa8e3640 10 monclient: build_initial_monmap 2024-04-26T00:22:47.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78fa8e3640 10 monclient: monmap: 2024-04-26T00:22:47.443 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:22:47.443 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:22:47.443 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:22:47.443 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:22:47.443 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:22:47.443 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:22:47.443 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:22:47.443 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:22:47.443 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:22:47.443 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:47.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78fa8e3640 1 Processor -- start 2024-04-26T00:22:47.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78fa8e3640 1 -- start start 2024-04-26T00:22:47.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78fa8e3640 10 monclient: init 2024-04-26T00:22:47.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78fa8e3640 10 monclient: _reopen_session rank -1 2024-04-26T00:22:47.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78fa8e3640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:22:47.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78fa8e3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78dc0039c0 0x7f78f412d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:47.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78fa8e3640 10 monclient(hunting): picked mon.noname-b con 0x7f78dc0039c0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:22:47.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78fa8e3640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78dc004810 0x7f78f412dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:47.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78fa8e3640 10 monclient(hunting): picked mon.noname-a con 0x7f78dc004810 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:22:47.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78fa8e3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78dc005220 0x7f78f406f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:47.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78fa8e3640 10 monclient(hunting): picked mon.noname-c con 0x7f78dc005220 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:22:47.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78fa8e3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:47.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78fa8e3640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f78f4134870 con 0x7f78dc004810 2024-04-26T00:22:47.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78fa8e3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:47.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78fa8e3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f78f41346f0 con 0x7f78dc0039c0 2024-04-26T00:22:47.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78fa8e3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:47.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78fa8e3640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f78f4130770 con 0x7f78dc005220 2024-04-26T00:22:47.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78fa8e3640 10 monclient(hunting): _renew_subs 2024-04-26T00:22:47.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78fa8e3640 10 monclient(hunting): authenticate will time out at 1148.769919s 2024-04-26T00:22:47.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78f37fe640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78dc004810 msgr2=0x7f78f412dd30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:22:47.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78f37fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78dc004810 0x7f78f412dd30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:22:47.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78f8e80640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78dc005220 0x7f78f406f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:47.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78f8e80640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78dc005220 0x7f78f406f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:50506/0 (socket says 172.21.15.37:50506) 2024-04-26T00:22:47.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78f8e80640 1 -- 172.21.15.37:0/2023553726 learned_addr learned my addr 172.21.15.37:0/2023553726 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:22:47.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78f8e80640 10 monclient(hunting): get_auth_request con 0x7f78dc005220 auth_method 0 2024-04-26T00:22:47.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78f8e80640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:47.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78f8e80640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:47.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78f8e80640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:47.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78f3fff640 1 --2- 172.21.15.37:0/2023553726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78dc0039c0 0x7f78f412d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:47.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78f8e80640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:47.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78f8e80640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:47.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78f8e80640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:47.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78f3fff640 10 monclient(hunting): get_auth_request con 0x7f78dc0039c0 auth_method 0 2024-04-26T00:22:47.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:47.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78f3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:47.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:47.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:47.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:47.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:47.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78f8e80640 10 monclient(hunting): handle_auth_done global_id 34157 payload 824 2024-04-26T00:22:47.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78f8e80640 1 -- 172.21.15.37:0/2023553726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78dc0039c0 msgr2=0x7f78f412d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:47.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78f8e80640 1 --2- 172.21.15.37:0/2023553726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78dc0039c0 0x7f78f412d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:47.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78f8e80640 1 -- 172.21.15.37:0/2023553726 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78dc004810 msgr2=0x7f78f412dd30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:22:47.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78f8e80640 1 --2- 172.21.15.37:0/2023553726 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78dc004810 0x7f78f412dd30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:47.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78f8e80640 10 monclient: _finish_hunting 0 2024-04-26T00:22:47.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78f8e80640 1 monclient: found mon.noname-c 2024-04-26T00:22:47.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78f8e80640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:22:47.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78f8e80640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:22:47.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.462+0000 7f78f8e80640 1 -- 172.21.15.37:0/2023553726 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78f41c0810 con 0x7f78dc005220 2024-04-26T00:22:47.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.466+0000 7f78f8e80640 1 -- 172.21.15.37:0/2023553726 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78dc005220 msgr2=0x7f78f406f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:47.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.466+0000 7f78f8e80640 1 --2- 172.21.15.37:0/2023553726 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78dc005220 0x7f78f406f520 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f78e0007340 tx=0x7f78ec015d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:47.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.466+0000 7f78f0ff9640 1 -- 172.21.15.37:0/2023553726 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78ec01c030 con 0x7f78dc005220 2024-04-26T00:22:47.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.466+0000 7f78f0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:47.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.466+0000 7f78f0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:22:47.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.466+0000 7f78f0ff9640 10 monclient: dump: 2024-04-26T00:22:47.450 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:47.450 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:47.450 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:47.450 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:47.451 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:47.451 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:47.451 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:47.451 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:47.451 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:47.451 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:47.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.466+0000 7f78f0ff9640 10 monclient: _finish_auth 0 2024-04-26T00:22:47.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.466+0000 7f78f0ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:22:47.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.466+0000 7f78f0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:22:47.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.466+0000 7f78f0ff9640 1 -- 172.21.15.37:0/2023553726 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78ec013c20 con 0x7f78dc005220 2024-04-26T00:22:47.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.466+0000 7f78f0ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:22:47.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.466+0000 7f78f0ff9640 1 -- 172.21.15.37:0/2023553726 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78ec013e10 con 0x7f78dc005220 2024-04-26T00:22:47.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.466+0000 7f78f0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:47.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.466+0000 7f78f27fc640 4 set_mon_vals no callback set 2024-04-26T00:22:47.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.466+0000 7f78f0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:22:47.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.466+0000 7f78f0ff9640 10 monclient: dump: 2024-04-26T00:22:47.452 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:47.452 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:47.452 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:47.452 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:47.452 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:47.452 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:47.452 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:47.452 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:47.452 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:47.452 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:47.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.466+0000 7f78fa8e3640 5 monclient: authenticate success, global_id 34157 2024-04-26T00:22:47.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.466+0000 7f78fa8e3640 10 monclient: _renew_subs 2024-04-26T00:22:47.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.466+0000 7f78fa8e3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:22:47.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.466+0000 7f78fa8e3640 1 -- 172.21.15.37:0/2023553726 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78f411bf60 con 0x7f78dc005220 2024-04-26T00:22:47.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.466+0000 7f78fa8e3640 10 monclient: _renew_subs 2024-04-26T00:22:47.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.466+0000 7f78fa8e3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:22:47.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.466+0000 7f78fa8e3640 1 -- 172.21.15.37:0/2023553726 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f78f4071090 con 0x7f78dc005220 2024-04-26T00:22:47.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.466+0000 7f78f0ff9640 1 -- 172.21.15.37:0/2023553726 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f78ec01f030 con 0x7f78dc005220 2024-04-26T00:22:47.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.466+0000 7f78fa8e3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:47.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.466+0000 7f78fa8e3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:47.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.466+0000 7f78fa8e3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:22:47.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.466+0000 7f78fa8e3640 1 -- 172.21.15.37:0/2023553726 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78f4066f40 con 0x7f78dc005220 2024-04-26T00:22:47.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.466+0000 7f78f0ff9640 1 --2- 172.21.15.37:0/2023553726 >> v2:172.21.15.45:6812/2379918623 conn(0x7f78e0036940 0x7f78e0038e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:47.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.466+0000 7f78f0ff9640 1 -- 172.21.15.37:0/2023553726 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(111..111 src has 1..111) ==== 3963+0+0 (secure 0 0 0) 0x7f78ec0497e0 con 0x7f78dc005220 2024-04-26T00:22:47.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.466+0000 7f78f3fff640 1 --2- 172.21.15.37:0/2023553726 >> v2:172.21.15.45:6812/2379918623 conn(0x7f78e0036940 0x7f78e0038e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:47.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.466+0000 7f78f3fff640 10 monclient: get_auth_request con 0x7f78e0036940 auth_method 0 2024-04-26T00:22:47.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.470+0000 7f78f3fff640 1 --2- 172.21.15.37:0/2023553726 >> v2:172.21.15.45:6812/2379918623 conn(0x7f78e0036940 0x7f78e0038e00 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f78dc0057f0 tx=0x7f78e40a7070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:47.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.474+0000 7f78f0ff9640 1 -- 172.21.15.37:0/2023553726 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f78f41c0810 con 0x7f78dc005220 2024-04-26T00:22:47.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.474+0000 7f78f0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:47.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.474+0000 7f78f0ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:22:47.488 INFO:tasks.rados.rados.0.smithi045.stdout:464: done (2 left) 2024-04-26T00:22:47.488 INFO:tasks.rados.rados.0.smithi045.stdout:465: done (1 left) 2024-04-26T00:22:47.488 INFO:tasks.rados.rados.0.smithi045.stdout:466: done (0 left) 2024-04-26T00:22:47.488 INFO:tasks.rados.rados.0.smithi045.stdout:467: write (excl) oid 16 current snap is 51 2024-04-26T00:22:47.489 INFO:tasks.rados.rados.0.smithi045.stdout:467: seq_num 197 ranges {510979=532878,1528482=772396,3060800=250179} 2024-04-26T00:22:47.506 INFO:tasks.rados.rados.0.smithi045.stdout:467: writing smithi04526579-16 from 510979 to 1043857 tid 1 2024-04-26T00:22:47.526 INFO:tasks.rados.rados.0.smithi045.stdout:467: writing smithi04526579-16 from 1528482 to 2300878 tid 2 2024-04-26T00:22:47.542 INFO:tasks.rados.rados.0.smithi045.stdout:467: writing smithi04526579-16 from 3060800 to 3310979 tid 3 2024-04-26T00:22:47.542 INFO:tasks.rados.rados.0.smithi045.stdout:468: delete oid 48 current snap is 51 2024-04-26T00:22:47.542 INFO:tasks.rados.rados.0.smithi045.stdout:467: finishing write tid 1 to smithi04526579-16 2024-04-26T00:22:47.542 INFO:tasks.rados.rados.0.smithi045.stdout:467: finishing write tid 2 to smithi04526579-16 2024-04-26T00:22:47.547 INFO:tasks.rados.rados.0.smithi045.stdout:467: finishing write tid 3 to smithi04526579-16 2024-04-26T00:22:47.547 INFO:tasks.rados.rados.0.smithi045.stdout:467: finishing write tid 4 to smithi04526579-16 2024-04-26T00:22:47.548 INFO:tasks.rados.rados.0.smithi045.stdout:467: finishing write tid 5 to smithi04526579-16 2024-04-26T00:22:47.548 INFO:tasks.rados.rados.0.smithi045.stdout:467: oid 16 updating version 0 to 87 2024-04-26T00:22:47.548 INFO:tasks.rados.rados.0.smithi045.stdout:467: oid 16 version 87 is already newer than 84 2024-04-26T00:22:47.549 INFO:tasks.rados.rados.0.smithi045.stdout:467: oid 16 version 87 is already newer than 86 2024-04-26T00:22:47.549 INFO:tasks.rados.rados.0.smithi045.stdout:467: oid 16 version 87 is already newer than 85 2024-04-26T00:22:47.549 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 87 (ObjNum 197 snap 51 seq_num 197) dirty exists 2024-04-26T00:22:47.549 INFO:tasks.rados.rados.0.smithi045.stdout:467: left oid 16 (ObjNum 197 snap 51 seq_num 197) 2024-04-26T00:22:47.550 INFO:tasks.rados.rados.0.smithi045.stdout:467: done (1 left) 2024-04-26T00:22:47.550 INFO:tasks.rados.rados.0.smithi045.stdout:468: done (0 left) 2024-04-26T00:22:47.550 INFO:tasks.rados.rados.0.smithi045.stdout:469: read oid 19 snap -1 2024-04-26T00:22:47.550 INFO:tasks.rados.rados.0.smithi045.stdout:469: expect (ObjNum 120 snap 28 seq_num 120) 2024-04-26T00:22:47.550 INFO:tasks.rados.rados.0.smithi045.stdout:470: read oid 44 snap -1 2024-04-26T00:22:47.550 INFO:tasks.rados.rados.0.smithi045.stdout:470: expect deleted 2024-04-26T00:22:47.550 INFO:tasks.rados.rados.0.smithi045.stdout:471: snap_create 2024-04-26T00:22:47.562 INFO:tasks.rados.rados.0.smithi045.stdout:469: expect (ObjNum 120 snap 28 seq_num 120) 2024-04-26T00:22:47.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.726+0000 7f78fa8e3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:22:47.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.726+0000 7f78fa8e3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:22:47.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.726+0000 7f78fa8e3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:22:47.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.726+0000 7f78fa8e3640 1 -- 172.21.15.37:0/2023553726 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f78f409ff50 con 0x7f78dc005220 2024-04-26T00:22:47.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.726+0000 7f78f0ff9640 1 -- 172.21.15.37:0/2023553726 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7f78ec048510 con 0x7f78dc005220 2024-04-26T00:22:47.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.726+0000 7f78f0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:22:47.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.726+0000 7f78f0ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:22:47.707 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":26,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:22:47.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.730+0000 7f78fa8e3640 1 -- 172.21.15.37:0/2023553726 >> v2:172.21.15.45:6812/2379918623 conn(0x7f78e0036940 msgr2=0x7f78e0038e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:47.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.730+0000 7f78fa8e3640 1 --2- 172.21.15.37:0/2023553726 >> v2:172.21.15.45:6812/2379918623 conn(0x7f78e0036940 0x7f78e0038e00 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f78dc0057f0 tx=0x7f78e40a7070 comp rx=0 tx=0).stop 2024-04-26T00:22:47.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.730+0000 7f78fa8e3640 10 monclient: shutdown 2024-04-26T00:22:47.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.730+0000 7f78fa8e3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:22:47.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.730+0000 7f78fa8e3640 1 -- 172.21.15.37:0/2023553726 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78dc005220 msgr2=0x7f78f406f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:47.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.730+0000 7f78fa8e3640 1 --2- 172.21.15.37:0/2023553726 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78dc005220 0x7f78f406f520 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f78e0007340 tx=0x7f78ec015d70 comp rx=0 tx=0).stop 2024-04-26T00:22:47.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.730+0000 7f78fa8e3640 1 -- 172.21.15.37:0/2023553726 shutdown_connections 2024-04-26T00:22:47.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.730+0000 7f78fa8e3640 1 --2- 172.21.15.37:0/2023553726 >> v2:172.21.15.45:6812/2379918623 conn(0x7f78e0036940 0x7f78e0038e00 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:47.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.730+0000 7f78fa8e3640 1 --2- 172.21.15.37:0/2023553726 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78dc005220 0x7f78f406f520 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:47.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.730+0000 7f78fa8e3640 1 --2- 172.21.15.37:0/2023553726 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78dc004810 0x7f78f412dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:47.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.730+0000 7f78fa8e3640 1 --2- 172.21.15.37:0/2023553726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78dc0039c0 0x7f78f412d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:47.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.730+0000 7f78fa8e3640 1 -- 172.21.15.37:0/2023553726 >> 172.21.15.37:0/2023553726 conn(0x7f78f40a1ad0 msgr2=0x7f78f409cfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:22:47.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.730+0000 7f78fa8e3640 1 -- 172.21.15.37:0/2023553726 shutdown_connections 2024-04-26T00:22:47.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.730+0000 7f78fa8e3640 1 -- 172.21.15.37:0/2023553726 wait complete. 2024-04-26T00:22:47.726 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-26T00:22:47.726 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-04-26T00:22:47.726 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:22:47.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f78a2615640 10 monclient: get_monmap_and_config 2024-04-26T00:22:47.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f78a2615640 10 monclient: build_initial_monmap 2024-04-26T00:22:47.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f78a2615640 10 monclient: monmap: 2024-04-26T00:22:47.844 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:22:47.844 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:47.844 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:22:47.868774+0000 2024-04-26T00:22:47.844 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:22:47.868774+0000 2024-04-26T00:22:47.844 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:22:47.844 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:47.844 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:22:47.844 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:22:47.844 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:22:47.844 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:22:47.844 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:22:47.844 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:47.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f78a2615640 1 Processor -- start 2024-04-26T00:22:47.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f78a2615640 1 -- start start 2024-04-26T00:22:47.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f78a2615640 10 monclient: init 2024-04-26T00:22:47.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f78a2615640 10 monclient: _reopen_session rank -1 2024-04-26T00:22:47.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f78a2615640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-04-26T00:22:47.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f78a2615640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f789c130eb0 0x7f789c09ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:47.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f78a2615640 10 monclient(hunting): picked mon.noname-b con 0x7f789c130eb0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:22:47.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f78a2615640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f789c131290 addr v1:172.21.15.37:6789/0 2024-04-26T00:22:47.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f78a2615640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f789c1342a0 0x7f789c13bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:47.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f78a2615640 10 monclient(hunting): picked mon.noname-a con 0x7f789c1342a0 addr v2:172.21.15.37:3300/0 2024-04-26T00:22:47.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f78a2615640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f789c09c190 con 0x7f789c131290 2024-04-26T00:22:47.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f78a2615640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:47.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f78a2615640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f789c09df60 con 0x7f789c1342a0 2024-04-26T00:22:47.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f78a2615640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:47.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f78a2615640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f789c0a11c0 con 0x7f789c130eb0 2024-04-26T00:22:47.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f78a2615640 10 monclient(hunting): _renew_subs 2024-04-26T00:22:47.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f78a2615640 10 monclient(hunting): authenticate will time out at 1149.174466s 2024-04-26T00:22:47.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f78a0bb2640 1 -- >> v2:172.21.15.37:3300/0 conn(0x7f789c1342a0 msgr2=0x7f789c13bed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:22:47.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f78a0bb2640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f789c1342a0 0x7f789c13bed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:22:47.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f789bfff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f789c130eb0 0x7f789c09ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:47.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f789b7fe640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f789c131290 legacy=0x7f789c133b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:22:47.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f789bfff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f789c130eb0 0x7f789c09ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:50508/0 (socket says 172.21.15.37:50508) 2024-04-26T00:22:47.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f789bfff640 1 -- 172.21.15.37:0/1333482736 learned_addr learned my addr 172.21.15.37:0/1333482736 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:22:47.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f789bfff640 10 monclient(hunting): get_auth_request con 0x7f789c130eb0 auth_method 0 2024-04-26T00:22:47.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f789bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:47.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f789bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:47.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f789bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:47.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f789bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:47.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f789bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:47.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f789bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:47.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f789bfff640 10 monclient(hunting): handle_auth_done global_id 34160 payload 274 2024-04-26T00:22:47.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f789bfff640 1 -- 172.21.15.37:0/1333482736 >> v2:172.21.15.37:3300/0 conn(0x7f789c1342a0 msgr2=0x7f789c13bed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:22:47.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f789bfff640 1 --2- 172.21.15.37:0/1333482736 >> v2:172.21.15.37:3300/0 conn(0x7f789c1342a0 0x7f789c13bed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:47.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f789bfff640 1 -- 172.21.15.37:0/1333482736 >> v1:172.21.15.37:6789/0 conn(0x7f789c131290 legacy=0x7f789c133b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:22:47.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f789bfff640 10 monclient: _finish_hunting 0 2024-04-26T00:22:47.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f789bfff640 1 monclient: found mon.noname-b 2024-04-26T00:22:47.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f789bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:22:47.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f789bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:22:47.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.866+0000 7f789bfff640 1 -- 172.21.15.37:0/1333482736 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f789c09dc80 con 0x7f789c130eb0 2024-04-26T00:22:47.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.870+0000 7f789bfff640 1 -- 172.21.15.37:0/1333482736 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f789c130eb0 msgr2=0x7f789c09ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:47.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.870+0000 7f789bfff640 1 --2- 172.21.15.37:0/1333482736 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f789c130eb0 0x7f789c09ef20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f788c00bb10 tx=0x7f788c01ead0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1402434171124b9c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:47.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.870+0000 7f789affd640 1 -- 172.21.15.37:0/1333482736 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f788c0b4070 con 0x7f789c130eb0 2024-04-26T00:22:47.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.870+0000 7f789affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:47.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.870+0000 7f789affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:22:47.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.870+0000 7f789affd640 10 monclient: dump: 2024-04-26T00:22:47.852 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:47.852 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:47.852 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:47.852 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:47.852 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:47.852 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:47.852 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:47.852 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:47.852 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:47.852 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:47.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.870+0000 7f789affd640 10 monclient: _finish_auth 0 2024-04-26T00:22:47.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.870+0000 7f789affd640 10 monclient: _check_auth_tickets 2024-04-26T00:22:47.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.870+0000 7f789affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:22:47.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.870+0000 7f789affd640 1 -- 172.21.15.37:0/1333482736 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f788c0a7a50 con 0x7f789c130eb0 2024-04-26T00:22:47.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.870+0000 7f78a2615640 5 monclient: authenticate success, global_id 34160 2024-04-26T00:22:47.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.870+0000 7f78a2615640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:22:47.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.870+0000 7f789affd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:22:47.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.870+0000 7f78a2615640 10 monclient: get_monmap_and_config success 2024-04-26T00:22:47.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.870+0000 7f78a2615640 4 set_mon_vals no callback set 2024-04-26T00:22:47.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.870+0000 7f789affd640 1 -- 172.21.15.37:0/1333482736 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f788c0a7e20 con 0x7f789c130eb0 2024-04-26T00:22:47.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.870+0000 7f789affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:47.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.870+0000 7f789affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:22:47.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.870+0000 7f789affd640 10 monclient: dump: 2024-04-26T00:22:47.853 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:47.854 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:47.854 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:47.854 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:47.854 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:47.854 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:47.854 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:47.854 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:47.854 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:47.854 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:47.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f78a2615640 10 monclient: shutdown 2024-04-26T00:22:47.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f78a2615640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:22:47.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f78a2615640 1 -- 172.21.15.37:0/1333482736 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f789c130eb0 msgr2=0x7f789c09ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:47.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f78a2615640 1 --2- 172.21.15.37:0/1333482736 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f789c130eb0 0x7f789c09ef20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f788c00bb10 tx=0x7f788c01ead0 comp rx=0 tx=0).stop 2024-04-26T00:22:47.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f78a2615640 1 -- 172.21.15.37:0/1333482736 shutdown_connections 2024-04-26T00:22:47.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f78a2615640 1 --2- 172.21.15.37:0/1333482736 >> v2:172.21.15.37:3300/0 conn(0x7f789c1342a0 0x7f789c13bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:47.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f78a2615640 1 --2- 172.21.15.37:0/1333482736 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f789c130eb0 0x7f789c09ef20 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:47.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f78a2615640 1 -- 172.21.15.37:0/1333482736 >> 172.21.15.37:0/1333482736 conn(0x7f789c0a1d50 msgr2=0x7f789c0a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:22:47.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f78a2615640 1 -- 172.21.15.37:0/1333482736 shutdown_connections 2024-04-26T00:22:47.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f78a2615640 1 -- 172.21.15.37:0/1333482736 wait complete. 2024-04-26T00:22:47.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f78a2615640 10 monclient: build_initial_monmap 2024-04-26T00:22:47.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f78a2615640 10 monclient: monmap: 2024-04-26T00:22:47.855 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:22:47.855 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:22:47.855 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:22:47.855 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:22:47.855 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:22:47.855 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:22:47.855 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:22:47.855 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:22:47.856 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:22:47.856 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:47.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f78a2615640 1 Processor -- start 2024-04-26T00:22:47.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f78a2615640 1 -- start start 2024-04-26T00:22:47.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f78a2615640 10 monclient: init 2024-04-26T00:22:47.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f78a2615640 10 monclient: _reopen_session rank -1 2024-04-26T00:22:47.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f78a2615640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:22:47.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f78a2615640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f789c130eb0 0x7f789c1cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:47.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f78a2615640 10 monclient(hunting): picked mon.noname-a con 0x7f789c130eb0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:22:47.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f78a2615640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f789c131290 0x7f789c12f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:47.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f78a2615640 10 monclient(hunting): picked mon.noname-c con 0x7f789c131290 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:22:47.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f78a2615640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f789c1342a0 0x7f789c12f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:47.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f78a2615640 10 monclient(hunting): picked mon.noname-b con 0x7f789c1342a0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:22:47.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f78a2615640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:47.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f78a2615640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f789c0a05e0 con 0x7f789c130eb0 2024-04-26T00:22:47.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f78a2615640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:47.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f78a2615640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f789c0a0460 con 0x7f789c1342a0 2024-04-26T00:22:47.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f78a2615640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:47.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f789bfff640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f789c130eb0 msgr2=0x7f789c1cf7d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:22:47.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f78a2615640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f789c0a0760 con 0x7f789c131290 2024-04-26T00:22:47.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f78a2615640 10 monclient(hunting): _renew_subs 2024-04-26T00:22:47.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f78a2615640 10 monclient(hunting): authenticate will time out at 1149.181875s 2024-04-26T00:22:47.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f789bfff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f789c130eb0 0x7f789c1cf7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:22:47.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f789b7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f789c131290 0x7f789c12f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:47.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f789b7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f789c131290 0x7f789c12f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:50520/0 (socket says 172.21.15.37:50520) 2024-04-26T00:22:47.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f789b7fe640 1 -- 172.21.15.37:0/36524231 learned_addr learned my addr 172.21.15.37:0/36524231 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:22:47.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f789b7fe640 10 monclient(hunting): get_auth_request con 0x7f789c131290 auth_method 0 2024-04-26T00:22:47.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f789b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:47.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f789b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:47.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f789b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:47.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f789b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:47.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f789b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:47.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f789b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:47.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f789b7fe640 10 monclient(hunting): handle_auth_done global_id 34163 payload 824 2024-04-26T00:22:47.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f789b7fe640 1 -- 172.21.15.37:0/36524231 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f789c1342a0 msgr2=0x7f789c12f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:47.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f78a0bb2640 1 --2- 172.21.15.37:0/36524231 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f789c1342a0 0x7f789c12f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:47.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f789b7fe640 1 --2- 172.21.15.37:0/36524231 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f789c1342a0 0x7f789c12f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:47.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f789b7fe640 1 -- 172.21.15.37:0/36524231 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f789c130eb0 msgr2=0x7f789c1cf7d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:22:47.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f789b7fe640 1 --2- 172.21.15.37:0/36524231 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f789c130eb0 0x7f789c1cf7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:47.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f789b7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:22:47.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f789b7fe640 1 monclient: found mon.noname-c 2024-04-26T00:22:47.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f789b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:22:47.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f789b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:22:47.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.874+0000 7f789b7fe640 1 -- 172.21.15.37:0/36524231 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f789c13b9c0 con 0x7f789c131290 2024-04-26T00:22:47.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f789b7fe640 1 -- 172.21.15.37:0/36524231 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f789c131290 msgr2=0x7f789c12f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:47.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f789b7fe640 1 --2- 172.21.15.37:0/36524231 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f789c131290 0x7f789c12f070 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f789000b550 tx=0x7f789000ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:47.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f7898ff9640 1 -- 172.21.15.37:0/36524231 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78900050f0 con 0x7f789c131290 2024-04-26T00:22:47.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f7898ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:47.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f7898ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:22:47.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f7898ff9640 10 monclient: dump: 2024-04-26T00:22:47.863 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:47.864 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:47.864 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:47.864 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:47.864 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:47.864 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:47.864 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:47.864 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:47.864 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:47.864 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:47.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f7898ff9640 10 monclient: _finish_auth 0 2024-04-26T00:22:47.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f7898ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:22:47.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f7898ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:22:47.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f7898ff9640 1 -- 172.21.15.37:0/36524231 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7890015070 con 0x7f789c131290 2024-04-26T00:22:47.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f78a2615640 5 monclient: authenticate success, global_id 34163 2024-04-26T00:22:47.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f78a2615640 10 monclient: _renew_subs 2024-04-26T00:22:47.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f78a2615640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:22:47.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f78a2615640 1 -- 172.21.15.37:0/36524231 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f789c11d530 con 0x7f789c131290 2024-04-26T00:22:47.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f7898ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:22:47.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f7898ff9640 1 -- 172.21.15.37:0/36524231 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78900054d0 con 0x7f789c131290 2024-04-26T00:22:47.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f7898ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:47.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f7898ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:22:47.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f7898ff9640 10 monclient: dump: 2024-04-26T00:22:47.865 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:47.865 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:47.865 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:47.865 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:47.865 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:47.865 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:47.865 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:47.866 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:47.866 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:47.866 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:47.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f78a2615640 10 monclient: _renew_subs 2024-04-26T00:22:47.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f78a2615640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:22:47.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f78a2615640 1 -- 172.21.15.37:0/36524231 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f789c1cfd10 con 0x7f789c131290 2024-04-26T00:22:47.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f789a7fc640 4 set_mon_vals no callback set 2024-04-26T00:22:47.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f78a2615640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:47.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f78a2615640 1 --2- 172.21.15.37:0/36524231 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f789c139e30 0x7f789c13a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:47.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f78a2615640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:47.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f78a2615640 1 -- 172.21.15.37:0/36524231 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f789c04f510 con 0x7f789c139e30 2024-04-26T00:22:47.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f7898ff9640 1 -- 172.21.15.37:0/36524231 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f7890005670 con 0x7f789c131290 2024-04-26T00:22:47.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f789bfff640 1 --2- 172.21.15.37:0/36524231 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f789c139e30 0x7f789c13a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:47.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f789bfff640 10 monclient: get_auth_request con 0x7f789c139e30 auth_method 0 2024-04-26T00:22:47.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f789bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:47.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f789bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:47.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f789bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:47.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f789bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:47.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f789bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:47.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f789bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:47.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f7898ff9640 1 --2- 172.21.15.37:0/36524231 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7888035e80 0x7f7888038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:47.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f789bfff640 10 monclient(hunting): handle_auth_done global_id 34166 payload 824 2024-04-26T00:22:47.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f789bfff640 1 -- 172.21.15.37:0/36524231 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f789c0a0f60 con 0x7f789c139e30 2024-04-26T00:22:47.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f7898ff9640 1 -- 172.21.15.37:0/36524231 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(112..112 src has 1..112) ==== 3963+0+0 (secure 0 0 0) 0x7f78900489e0 con 0x7f789c131290 2024-04-26T00:22:47.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f78a0bb2640 1 --2- 172.21.15.37:0/36524231 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7888035e80 0x7f7888038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:47.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.878+0000 7f78a0bb2640 10 monclient: get_auth_request con 0x7f7888035e80 auth_method 0 2024-04-26T00:22:47.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.882+0000 7f789bfff640 1 -- 172.21.15.37:0/36524231 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f789c139e30 msgr2=0x7f789c13a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:47.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.882+0000 7f789bfff640 1 --2- 172.21.15.37:0/36524231 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f789c139e30 0x7f789c13a210 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f788c0b1000 tx=0x7f788c01d310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:47.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.882+0000 7f7898ff9640 1 -- 172.21.15.37:0/36524231 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f788c0b4040 con 0x7f789c139e30 2024-04-26T00:22:47.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.882+0000 7f7898ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:47.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.882+0000 7f7898ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:22:47.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.882+0000 7f7898ff9640 10 monclient: dump: 2024-04-26T00:22:47.868 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:47.868 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:47.868 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:47.868 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:47.868 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:47.868 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:47.868 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:47.868 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:47.868 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:47.868 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:47.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.882+0000 7f7898ff9640 1 -- 172.21.15.37:0/36524231 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f788c00b730 con 0x7f789c139e30 2024-04-26T00:22:47.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.882+0000 7f7898ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:47.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.882+0000 7f7898ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:22:47.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.882+0000 7f7898ff9640 1 -- 172.21.15.37:0/36524231 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f789c139e30 msgr2=0x7f789c13a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:47.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.882+0000 7f7898ff9640 1 --2- 172.21.15.37:0/36524231 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f789c139e30 0x7f789c13a210 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f788c0b1000 tx=0x7f788c01d310 comp rx=0 tx=0).stop 2024-04-26T00:22:47.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.882+0000 7f7898ff9640 1 -- 172.21.15.37:0/36524231 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f789c139e30 msgr2=0x7f789c13a210 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:22:47.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.882+0000 7f7898ff9640 1 --2- 172.21.15.37:0/36524231 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f789c139e30 0x7f789c13a210 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f788c0b1000 tx=0x7f788c01d310 comp rx=0 tx=0).stop 2024-04-26T00:22:47.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.882+0000 7f78a0bb2640 1 --2- 172.21.15.37:0/36524231 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7888035e80 0x7f7888038340 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f789c11d020 tx=0x7f7894008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:47.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.886+0000 7f78a2615640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:22:47.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.886+0000 7f78a2615640 1 --2- 172.21.15.37:0/36524231 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f789c13a750 0x7f789c06fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:47.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.886+0000 7f78a2615640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:47.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.886+0000 7f78a2615640 1 -- 172.21.15.37:0/36524231 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f789c11d2a0 con 0x7f789c13a750 2024-04-26T00:22:47.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.886+0000 7f789bfff640 1 --2- 172.21.15.37:0/36524231 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f789c13a750 0x7f789c06fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:47.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.886+0000 7f789bfff640 10 monclient: get_auth_request con 0x7f789c13a750 auth_method 0 2024-04-26T00:22:47.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.886+0000 7f789bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:47.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.886+0000 7f789bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:47.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.886+0000 7f789bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:47.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.886+0000 7f789bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:47.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.886+0000 7f789bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:47.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.886+0000 7f789bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:47.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.886+0000 7f789bfff640 10 monclient(hunting): handle_auth_done global_id 34169 payload 824 2024-04-26T00:22:47.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.886+0000 7f789bfff640 1 -- 172.21.15.37:0/36524231 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f789c06ba50 con 0x7f789c13a750 2024-04-26T00:22:47.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.890+0000 7f789bfff640 1 -- 172.21.15.37:0/36524231 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f789c13a750 msgr2=0x7f789c06fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:47.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.890+0000 7f789bfff640 1 --2- 172.21.15.37:0/36524231 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f789c13a750 0x7f789c06fb70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f788c01f370 tx=0x7f788c01b870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:47.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.890+0000 7f7898ff9640 1 -- 172.21.15.37:0/36524231 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f788c0b4040 con 0x7f789c13a750 2024-04-26T00:22:47.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.890+0000 7f7898ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:47.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.890+0000 7f7898ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:22:47.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.890+0000 7f7898ff9640 10 monclient: dump: 2024-04-26T00:22:47.872 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:47.872 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:47.873 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:47.873 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:47.873 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:47.873 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:47.873 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:47.873 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:47.873 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:47.873 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:47.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.890+0000 7f7898ff9640 1 -- 172.21.15.37:0/36524231 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4707 (secure 0 0 0) 0x7f788c0af6d0 con 0x7f789c13a750 2024-04-26T00:22:47.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.890+0000 7f7898ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:22:47.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.890+0000 7f7898ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:22:47.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.890+0000 7f7898ff9640 1 -- 172.21.15.37:0/36524231 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f789c13a750 msgr2=0x7f789c06fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:47.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.890+0000 7f7898ff9640 1 --2- 172.21.15.37:0/36524231 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f789c13a750 0x7f789c06fb70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f788c01f370 tx=0x7f788c01b870 comp rx=0 tx=0).stop 2024-04-26T00:22:47.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.890+0000 7f7898ff9640 1 -- 172.21.15.37:0/36524231 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f789c13a750 msgr2=0x7f789c06fb70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:22:47.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.890+0000 7f7898ff9640 1 --2- 172.21.15.37:0/36524231 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f789c13a750 0x7f789c06fb70 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f788c01f370 tx=0x7f788c01b870 comp rx=0 tx=0).stop 2024-04-26T00:22:47.874 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:22:47.874 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:22:47.874 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:22:47.874 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:22:47.874 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 26, 2024-04-26T00:22:47.874 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 28167, 2024-04-26T00:22:47.874 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:22:47.874 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:22:47.874 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:22:47.874 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:22:47.874 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:22:47.875 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:22:47.875 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:22:47.875 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:22:47.875 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:22:47.875 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:22:47.875 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:22:47.875 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:22:47.875 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:22:47.875 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:22:47.875 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:22:47.875 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:22:47.875 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:22:47.875 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:22:47.875 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:22:47.875 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:22:47.876 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:22:47.876 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:22:47.876 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:22:47.876 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:22:47.876 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:22:47.876 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:22:47.876 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:22:47.876 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:22:47.876 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:22:47.876 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:22:47.876 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:22:47.876 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:22:47.876 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:22:47.876 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:47.877 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:47.877 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:22:47.877 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:22:47.877 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:22:47.877 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:22:47.877 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:22:47.877 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:22:47.877 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:22:47.877 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:22:47.877 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:22:47.877 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:22:47.877 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:22:47.877 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:22:47.877 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:22:47.877 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:22:47.878 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:22:47.878 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:22:47.878 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:22:47.878 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:22:47.878 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:22:47.878 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:22:47.878 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:22:47.878 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:22:47.878 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:22:47.878 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:22:47.878 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:22:47.878 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:22:47.878 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:22:47.878 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:22:47.879 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:22:47.879 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:22:47.879 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:47.879 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:22:47.879 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:47.879 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:22:47.879 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:22:47.879 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:22:47.879 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:22:47.879 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:47.879 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:22:47.879 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:22:47.879 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:47.879 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:47.879 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:47.880 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:22:47.880 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:22:47.880 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:47.880 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:47.880 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:47.880 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:47.880 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:22:47.880 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:22:47.880 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:22:47.880 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:22:47.880 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:22:47.880 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:47.880 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:47.880 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:22:47.881 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:22:47.881 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:22:47.881 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:22:47.881 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:47.881 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:22:47.881 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:22:47.881 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:47.881 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:47.881 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:47.881 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:22:47.881 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:22:47.881 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:47.881 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:47.881 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:47.882 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:47.882 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:22:47.882 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:22:47.882 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:22:47.882 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:22:47.882 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:22:47.882 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:47.882 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:47.882 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:22:47.882 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:22:47.882 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:22:47.882 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:22:47.882 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:47.882 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:22:47.882 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:22:47.883 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:47.883 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:47.883 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:47.883 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:22:47.883 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:22:47.883 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:47.883 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:47.883 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:47.883 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:47.883 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:22:47.883 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:22:47.883 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:22:47.883 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:22:47.884 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:22:47.884 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:47.884 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:47.884 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:47.884 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:22:47.884 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:22:47.884 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:47.884 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:22:47.884 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:22:47.884 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:22:47.884 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:47.884 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:22:47.884 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:22:47.884 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:47.884 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:22:47.885 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:22:47.885 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:22:47.885 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:47.885 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:22:47.885 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:22:47.885 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:47.885 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:22:47.885 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:22:47.885 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:22:47.885 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:47.885 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:47.885 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:47.885 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:22:47.885 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:22:47.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.890+0000 7f78a2615640 1 -- 172.21.15.37:0/36524231 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7888035e80 msgr2=0x7f7888038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:47.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.890+0000 7f78a2615640 1 --2- 172.21.15.37:0/36524231 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7888035e80 0x7f7888038340 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f789c11d020 tx=0x7f7894008040 comp rx=0 tx=0).stop 2024-04-26T00:22:47.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.890+0000 7f78a2615640 10 monclient: shutdown 2024-04-26T00:22:47.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.890+0000 7f78a2615640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:22:47.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.890+0000 7f78a2615640 1 -- 172.21.15.37:0/36524231 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f789c131290 msgr2=0x7f789c12f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:47.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.890+0000 7f78a2615640 1 --2- 172.21.15.37:0/36524231 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f789c131290 0x7f789c12f070 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f789000b550 tx=0x7f789000ba10 comp rx=0 tx=0).stop 2024-04-26T00:22:47.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.890+0000 7f78a0bb2640 1 -- 172.21.15.37:0/36524231 reap_dead start 2024-04-26T00:22:47.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.890+0000 7f78a0bb2640 1 -- 172.21.15.37:0/36524231 reap_dead start 2024-04-26T00:22:47.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.890+0000 7f78a2615640 1 -- 172.21.15.37:0/36524231 shutdown_connections 2024-04-26T00:22:47.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.890+0000 7f78a2615640 1 -- 172.21.15.37:0/36524231 >> 172.21.15.37:0/36524231 conn(0x7f789c0a1d50 msgr2=0x7f789c09b130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:22:47.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.890+0000 7f78a2615640 1 -- 172.21.15.37:0/36524231 shutdown_connections 2024-04-26T00:22:47.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:47.890+0000 7f78a2615640 1 -- 172.21.15.37:0/36524231 wait complete. 2024-04-26T00:22:47.888 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:22:47.888 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-04-26T00:22:47.888 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:22:48.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f26996c1640 10 monclient: get_monmap_and_config 2024-04-26T00:22:48.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f26996c1640 10 monclient: build_initial_monmap 2024-04-26T00:22:48.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f26996c1640 10 monclient: monmap: 2024-04-26T00:22:48.008 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:22:48.009 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:48.009 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:22:48.033592+0000 2024-04-26T00:22:48.009 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:22:48.033592+0000 2024-04-26T00:22:48.009 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:22:48.009 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:48.009 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:22:48.009 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:22:48.009 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:22:48.009 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:22:48.009 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:22:48.009 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:48.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f26996c1640 1 Processor -- start 2024-04-26T00:22:48.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f26996c1640 1 -- start start 2024-04-26T00:22:48.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f26996c1640 10 monclient: init 2024-04-26T00:22:48.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f26996c1640 10 monclient: _reopen_session rank -1 2024-04-26T00:22:48.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f26996c1640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-04-26T00:22:48.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f26996c1640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2694130eb0 0x7f269409ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:48.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f26996c1640 10 monclient(hunting): picked mon.noname-c con 0x7f2694130eb0 addr v2:172.21.15.45:3300/0 2024-04-26T00:22:48.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f26996c1640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2694131290 0x7f2694133b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:48.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f26996c1640 10 monclient(hunting): picked mon.noname-b con 0x7f2694131290 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:22:48.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f26996c1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f26941340d0 addr v1:172.21.15.37:6789/0 2024-04-26T00:22:48.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f26996c1640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f269409c190 con 0x7f26941340d0 2024-04-26T00:22:48.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f26996c1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:48.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f26996c1640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f269409df60 con 0x7f2694130eb0 2024-04-26T00:22:48.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f26996c1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:48.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f26996c1640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f26940a11c0 con 0x7f2694131290 2024-04-26T00:22:48.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f26996c1640 10 monclient(hunting): _renew_subs 2024-04-26T00:22:48.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f26996c1640 10 monclient(hunting): authenticate will time out at 1149.339491s 2024-04-26T00:22:48.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f26937fe640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f26941340d0 legacy=0x7f269413aec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:22:48.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f26927fc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2694131290 0x7f2694133b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:48.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f26927fc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2694131290 0x7f2694133b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:50550/0 (socket says 172.21.15.37:50550) 2024-04-26T00:22:48.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f26927fc640 1 -- 172.21.15.37:0/990050875 learned_addr learned my addr 172.21.15.37:0/990050875 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:22:48.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f26927fc640 10 monclient(hunting): get_auth_request con 0x7f2694131290 auth_method 0 2024-04-26T00:22:48.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f26927fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:48.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f26927fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:48.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f26927fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:48.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f26927fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:48.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f26927fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:48.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f26927fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:48.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f26927fc640 10 monclient(hunting): handle_auth_done global_id 34172 payload 274 2024-04-26T00:22:48.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f2692ffd640 1 --2- 172.21.15.37:0/990050875 >> v2:172.21.15.45:3300/0 conn(0x7f2694130eb0 0x7f269409ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:48.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f26927fc640 1 -- 172.21.15.37:0/990050875 >> v2:172.21.15.45:3300/0 conn(0x7f2694130eb0 msgr2=0x7f269409ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:22:48.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f26927fc640 1 --2- 172.21.15.37:0/990050875 >> v2:172.21.15.45:3300/0 conn(0x7f2694130eb0 0x7f269409ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:48.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f26927fc640 1 -- 172.21.15.37:0/990050875 >> v1:172.21.15.37:6789/0 conn(0x7f26941340d0 legacy=0x7f269413aec0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:22:48.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f26927fc640 10 monclient: _finish_hunting 0 2024-04-26T00:22:48.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f26927fc640 1 monclient: found mon.noname-b 2024-04-26T00:22:48.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f26927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:22:48.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f26927fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:22:48.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f26927fc640 1 -- 172.21.15.37:0/990050875 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f269409dc80 con 0x7f2694131290 2024-04-26T00:22:48.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f2692ffd640 10 monclient: get_auth_request con 0x7f2694130eb0 auth_method 0 2024-04-26T00:22:48.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.030+0000 7f2692ffd640 1 --2- 172.21.15.37:0/990050875 >> v2:172.21.15.45:3300/0 conn(0x7f2694130eb0 0x7f269409ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:22:48.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26927fc640 1 -- 172.21.15.37:0/990050875 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2694131290 msgr2=0x7f2694133b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:48.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26927fc640 1 --2- 172.21.15.37:0/990050875 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2694131290 0x7f2694133b90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2684009870 tx=0x7f268401c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de28e436a49d9145 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:48.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f2691ffb640 1 -- 172.21.15.37:0/990050875 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26840b0070 con 0x7f2694131290 2024-04-26T00:22:48.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f2691ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:48.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f2691ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:22:48.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f2691ffb640 10 monclient: dump: 2024-04-26T00:22:48.017 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:48.017 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:48.017 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:48.017 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:48.017 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:48.017 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:48.017 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:48.017 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:48.017 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:48.017 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:48.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f2691ffb640 10 monclient: _finish_auth 0 2024-04-26T00:22:48.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f2691ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:22:48.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f2691ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:22:48.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f2691ffb640 1 -- 172.21.15.37:0/990050875 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26840ab600 con 0x7f2694131290 2024-04-26T00:22:48.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f2691ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:22:48.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f2691ffb640 1 -- 172.21.15.37:0/990050875 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26840ab9d0 con 0x7f2694131290 2024-04-26T00:22:48.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f2691ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:48.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f2691ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:22:48.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f2691ffb640 10 monclient: dump: 2024-04-26T00:22:48.018 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:48.018 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:48.018 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:48.018 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:48.018 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:48.018 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:48.018 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:48.019 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:48.019 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:48.019 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:48.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26996c1640 5 monclient: authenticate success, global_id 34172 2024-04-26T00:22:48.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26996c1640 10 monclient: get_monmap_and_config success 2024-04-26T00:22:48.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26996c1640 4 set_mon_vals no callback set 2024-04-26T00:22:48.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26996c1640 10 monclient: shutdown 2024-04-26T00:22:48.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26996c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:22:48.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26996c1640 1 -- 172.21.15.37:0/990050875 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2694131290 msgr2=0x7f2694133b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:48.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26996c1640 1 --2- 172.21.15.37:0/990050875 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2694131290 0x7f2694133b90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2684009870 tx=0x7f268401c3e0 comp rx=0 tx=0).stop 2024-04-26T00:22:48.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26996c1640 1 -- 172.21.15.37:0/990050875 shutdown_connections 2024-04-26T00:22:48.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26996c1640 1 --2- 172.21.15.37:0/990050875 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2694131290 0x7f2694133b90 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:48.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26996c1640 1 --2- 172.21.15.37:0/990050875 >> v2:172.21.15.45:3300/0 conn(0x7f2694130eb0 0x7f269409ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:48.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26996c1640 1 -- 172.21.15.37:0/990050875 >> 172.21.15.37:0/990050875 conn(0x7f26940a1d50 msgr2=0x7f26940a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:22:48.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26996c1640 1 -- 172.21.15.37:0/990050875 shutdown_connections 2024-04-26T00:22:48.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26996c1640 1 -- 172.21.15.37:0/990050875 wait complete. 2024-04-26T00:22:48.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26996c1640 10 monclient: build_initial_monmap 2024-04-26T00:22:48.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26996c1640 10 monclient: monmap: 2024-04-26T00:22:48.020 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:22:48.020 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:22:48.020 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:22:48.020 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:22:48.020 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:22:48.020 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:22:48.020 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:22:48.020 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:22:48.020 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:22:48.020 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:48.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26996c1640 1 Processor -- start 2024-04-26T00:22:48.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26996c1640 1 -- start start 2024-04-26T00:22:48.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26996c1640 10 monclient: init 2024-04-26T00:22:48.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26996c1640 10 monclient: _reopen_session rank -1 2024-04-26T00:22:48.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26996c1640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:22:48.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26996c1640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2694130eb0 0x7f269412dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:48.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26996c1640 10 monclient(hunting): picked mon.noname-c con 0x7f2694130eb0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:22:48.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26996c1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2694131290 0x7f269412e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:48.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26996c1640 10 monclient(hunting): picked mon.noname-b con 0x7f2694131290 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:22:48.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26996c1640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26941340d0 0x7f26941355d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:48.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26996c1640 10 monclient(hunting): picked mon.noname-a con 0x7f26941340d0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:22:48.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26996c1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:48.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26996c1640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f26941317f0 con 0x7f26941340d0 2024-04-26T00:22:48.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26996c1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:48.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26996c1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2694131670 con 0x7f2694131290 2024-04-26T00:22:48.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26996c1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:48.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26996c1640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f269413cdf0 con 0x7f2694130eb0 2024-04-26T00:22:48.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26996c1640 10 monclient(hunting): _renew_subs 2024-04-26T00:22:48.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26996c1640 10 monclient(hunting): authenticate will time out at 1149.346861s 2024-04-26T00:22:48.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26937fe640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26941340d0 msgr2=0x7f26941355d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:22:48.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26937fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26941340d0 0x7f26941355d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:22:48.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f2692ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2694130eb0 0x7f269412dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:48.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f2692ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2694130eb0 0x7f269412dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:50566/0 (socket says 172.21.15.37:50566) 2024-04-26T00:22:48.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f2692ffd640 1 -- 172.21.15.37:0/3994786960 learned_addr learned my addr 172.21.15.37:0/3994786960 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:22:48.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f2692ffd640 10 monclient(hunting): get_auth_request con 0x7f2694130eb0 auth_method 0 2024-04-26T00:22:48.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f2692ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:48.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f2692ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:48.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f2692ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:48.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f2692ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:48.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f2692ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:48.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f2692ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:48.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26927fc640 1 --2- 172.21.15.37:0/3994786960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2694131290 0x7f269412e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:48.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26927fc640 10 monclient(hunting): get_auth_request con 0x7f2694131290 auth_method 0 2024-04-26T00:22:48.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26927fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:48.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26927fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:48.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f26927fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:48.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f2692ffd640 10 monclient(hunting): handle_auth_done global_id 34175 payload 824 2024-04-26T00:22:48.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f2692ffd640 1 -- 172.21.15.37:0/3994786960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2694131290 msgr2=0x7f269412e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:48.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f2692ffd640 1 --2- 172.21.15.37:0/3994786960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2694131290 0x7f269412e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:48.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f2692ffd640 1 -- 172.21.15.37:0/3994786960 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26941340d0 msgr2=0x7f26941355d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:22:48.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f2692ffd640 1 --2- 172.21.15.37:0/3994786960 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26941340d0 0x7f26941355d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:48.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f2692ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:22:48.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f2692ffd640 1 monclient: found mon.noname-c 2024-04-26T00:22:48.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f2692ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:22:48.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f2692ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:22:48.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.038+0000 7f2692ffd640 1 -- 172.21.15.37:0/3994786960 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26940a1470 con 0x7f2694130eb0 2024-04-26T00:22:48.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f2692ffd640 1 -- 172.21.15.37:0/3994786960 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2694130eb0 msgr2=0x7f269412dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:48.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f2692ffd640 1 --2- 172.21.15.37:0/3994786960 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2694130eb0 0x7f269412dcb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f267c00c7c0 tx=0x7f267c00cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:48.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f268b7fe640 1 -- 172.21.15.37:0/3994786960 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267c007c40 con 0x7f2694130eb0 2024-04-26T00:22:48.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f268b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:48.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f268b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:22:48.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f268b7fe640 10 monclient: dump: 2024-04-26T00:22:48.028 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:48.028 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:48.028 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:48.028 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:48.028 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:48.028 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:48.028 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:48.029 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:48.029 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:48.029 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:48.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f268b7fe640 10 monclient: _finish_auth 0 2024-04-26T00:22:48.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f268b7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:22:48.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f268b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:22:48.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f268b7fe640 1 -- 172.21.15.37:0/3994786960 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f267c007de0 con 0x7f2694130eb0 2024-04-26T00:22:48.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f268b7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:22:48.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f268b7fe640 1 -- 172.21.15.37:0/3994786960 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267c011630 con 0x7f2694130eb0 2024-04-26T00:22:48.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f268b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:48.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f268b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:22:48.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f268b7fe640 10 monclient: dump: 2024-04-26T00:22:48.029 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:48.029 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:48.030 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:48.030 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:48.030 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:48.030 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:48.030 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:48.030 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:48.030 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:48.030 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:48.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f26996c1640 5 monclient: authenticate success, global_id 34175 2024-04-26T00:22:48.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f26996c1640 10 monclient: _renew_subs 2024-04-26T00:22:48.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f2690ff9640 4 set_mon_vals no callback set 2024-04-26T00:22:48.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f26996c1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:22:48.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f26996c1640 1 -- 172.21.15.37:0/3994786960 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f269413c160 con 0x7f2694130eb0 2024-04-26T00:22:48.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f26996c1640 10 monclient: _renew_subs 2024-04-26T00:22:48.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f26996c1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:22:48.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f26996c1640 1 -- 172.21.15.37:0/3994786960 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2694139560 con 0x7f2694130eb0 2024-04-26T00:22:48.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f26996c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:48.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f26996c1640 1 --2- 172.21.15.37:0/3994786960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2694135b10 0x7f2694135f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:48.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f26996c1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:48.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f26996c1640 1 -- 172.21.15.37:0/3994786960 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f269404f510 con 0x7f2694135b10 2024-04-26T00:22:48.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f268b7fe640 1 -- 172.21.15.37:0/3994786960 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f267c0117d0 con 0x7f2694130eb0 2024-04-26T00:22:48.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f268b7fe640 1 --2- 172.21.15.37:0/3994786960 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2678035e80 0x7f2678038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:48.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f268b7fe640 1 -- 172.21.15.37:0/3994786960 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(112..112 src has 1..112) ==== 3963+0+0 (secure 0 0 0) 0x7f267c048ce0 con 0x7f2694130eb0 2024-04-26T00:22:48.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f26927fc640 1 --2- 172.21.15.37:0/3994786960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2694135b10 0x7f2694135f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:48.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f26927fc640 10 monclient: get_auth_request con 0x7f2694135b10 auth_method 0 2024-04-26T00:22:48.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f26927fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:48.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f26927fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:48.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f26927fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:48.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f26927fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:48.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f26927fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:48.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f26927fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:48.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f26937fe640 1 --2- 172.21.15.37:0/3994786960 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2678035e80 0x7f2678038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:48.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f26937fe640 10 monclient: get_auth_request con 0x7f2678035e80 auth_method 0 2024-04-26T00:22:48.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f26927fc640 10 monclient(hunting): handle_auth_done global_id 24232 payload 824 2024-04-26T00:22:48.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.042+0000 7f26927fc640 1 -- 172.21.15.37:0/3994786960 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f26940a0f60 con 0x7f2694135b10 2024-04-26T00:22:48.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.046+0000 7f26927fc640 1 -- 172.21.15.37:0/3994786960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2694135b10 msgr2=0x7f2694135f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:48.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.046+0000 7f26927fc640 1 --2- 172.21.15.37:0/3994786960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2694135b10 0x7f2694135f10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f26840ab000 tx=0x7f2684019180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:48.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.046+0000 7f26937fe640 1 --2- 172.21.15.37:0/3994786960 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2678035e80 0x7f2678038340 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f268c003cc0 tx=0x7f268c0091f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:48.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.046+0000 7f268b7fe640 1 -- 172.21.15.37:0/3994786960 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26840ba020 con 0x7f2694135b10 2024-04-26T00:22:48.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.046+0000 7f268b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:48.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.046+0000 7f268b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:22:48.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.046+0000 7f268b7fe640 10 monclient: dump: 2024-04-26T00:22:48.033 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:48.033 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:48.034 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:48.034 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:48.034 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:48.034 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:48.034 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:48.034 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:48.034 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:48.034 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:48.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.046+0000 7f268b7fe640 1 -- 172.21.15.37:0/3994786960 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f26840b6470 con 0x7f2694135b10 2024-04-26T00:22:48.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.046+0000 7f268b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:22:48.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.046+0000 7f268b7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:22:48.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.046+0000 7f268b7fe640 1 -- 172.21.15.37:0/3994786960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2694135b10 msgr2=0x7f2694135f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:48.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.046+0000 7f268b7fe640 1 --2- 172.21.15.37:0/3994786960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2694135b10 0x7f2694135f10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f26840ab000 tx=0x7f2684019180 comp rx=0 tx=0).stop 2024-04-26T00:22:48.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.046+0000 7f268b7fe640 1 -- 172.21.15.37:0/3994786960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2694135b10 msgr2=0x7f2694135f10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:22:48.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.046+0000 7f268b7fe640 1 --2- 172.21.15.37:0/3994786960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2694135b10 0x7f2694135f10 secure :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f26840ab000 tx=0x7f2684019180 comp rx=0 tx=0).stop 2024-04-26T00:22:48.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.054+0000 7f26996c1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:22:48.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.054+0000 7f26996c1640 1 --2- 172.21.15.37:0/3994786960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2694136450 0x7f2694131a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:22:48.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.054+0000 7f26996c1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:22:48.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.054+0000 7f26996c1640 1 -- 172.21.15.37:0/3994786960 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2694137270 con 0x7f2694136450 2024-04-26T00:22:48.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.054+0000 7f26927fc640 1 --2- 172.21.15.37:0/3994786960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2694136450 0x7f2694131a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:22:48.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.054+0000 7f26927fc640 10 monclient: get_auth_request con 0x7f2694136450 auth_method 0 2024-04-26T00:22:48.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.054+0000 7f26927fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:22:48.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.054+0000 7f26927fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:22:48.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.054+0000 7f26927fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:22:48.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.054+0000 7f26927fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:22:48.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.054+0000 7f26927fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:22:48.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.054+0000 7f26927fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:22:48.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.054+0000 7f26927fc640 10 monclient(hunting): handle_auth_done global_id 24235 payload 824 2024-04-26T00:22:48.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.054+0000 7f26927fc640 1 -- 172.21.15.37:0/3994786960 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2694137b50 con 0x7f2694136450 2024-04-26T00:22:48.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.058+0000 7f26927fc640 1 -- 172.21.15.37:0/3994786960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2694136450 msgr2=0x7f2694131a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:22:48.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.058+0000 7f26927fc640 1 --2- 172.21.15.37:0/3994786960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2694136450 0x7f2694131a00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2684000c10 tx=0x7f26840a8000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:22:48.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.058+0000 7f268b7fe640 1 -- 172.21.15.37:0/3994786960 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26840ba030 con 0x7f2694136450 2024-04-26T00:22:48.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.058+0000 7f268b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:22:48.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.058+0000 7f268b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:22:48.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.058+0000 7f268b7fe640 10 monclient: dump: 2024-04-26T00:22:48.036 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:22:48.037 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:22:48.037 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:48.037 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:22:48.037 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:22:48.037 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:22:48.037 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:22:48.037 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:22:48.037 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:22:48.037 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:22:48.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.058+0000 7f268b7fe640 1 -- 172.21.15.37:0/3994786960 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4875 (secure 0 0 0) 0x7f26840c1220 con 0x7f2694136450 2024-04-26T00:22:48.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.058+0000 7f268b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:22:48.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.058+0000 7f268b7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:22:48.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.058+0000 7f268b7fe640 1 -- 172.21.15.37:0/3994786960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2694136450 msgr2=0x7f2694131a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:48.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.058+0000 7f268b7fe640 1 --2- 172.21.15.37:0/3994786960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2694136450 0x7f2694131a00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2684000c10 tx=0x7f26840a8000 comp rx=0 tx=0).stop 2024-04-26T00:22:48.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.058+0000 7f268b7fe640 1 -- 172.21.15.37:0/3994786960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2694136450 msgr2=0x7f2694131a00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:22:48.038 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:22:48.038 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:22:48.038 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:22:48.039 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:22:48.039 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 26, 2024-04-26T00:22:48.039 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 76225, 2024-04-26T00:22:48.039 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:22:48.039 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:22:48.039 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:22:48.039 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:22:48.039 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:22:48.039 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:22:48.039 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:22:48.039 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:22:48.039 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:22:48.039 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:22:48.040 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:22:48.040 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:22:48.040 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:22:48.040 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:22:48.040 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:22:48.040 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:22:48.040 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:22:48.040 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:22:48.040 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:22:48.040 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:22:48.040 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:22:48.040 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:22:48.040 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:22:48.040 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:22:48.041 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:22:48.041 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:22:48.041 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:22:48.041 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:22:48.041 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:22:48.041 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:22:48.041 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:22:48.041 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:22:48.041 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:22:48.041 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:48.041 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:48.041 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:22:48.041 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:22:48.041 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:22:48.041 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:22:48.042 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:22:48.042 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:22:48.042 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:22:48.042 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:22:48.042 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:22:48.042 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:22:48.042 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:22:48.042 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:22:48.042 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:22:48.042 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:22:48.042 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:22:48.042 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:22:48.042 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:22:48.042 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:22:48.043 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:22:48.043 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:22:48.043 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:22:48.043 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:22:48.043 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:22:48.043 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:22:48.043 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:22:48.043 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:22:48.043 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:22:48.043 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:22:48.043 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:22:48.043 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:22:48.043 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:48.043 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:22:48.043 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:48.044 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:22:48.044 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:22:48.044 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:22:48.044 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:22:48.044 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:48.044 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:22:48.044 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:22:48.044 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:48.044 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:48.044 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:48.044 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:22:48.044 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:22:48.044 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:48.044 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:48.045 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:48.045 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:48.045 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:22:48.045 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:22:48.045 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:22:48.045 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:22:48.045 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:22:48.045 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:48.045 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:48.045 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:22:48.045 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:22:48.045 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:22:48.045 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:22:48.045 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:48.045 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:22:48.046 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:22:48.046 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:48.046 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:48.046 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:48.046 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:22:48.046 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:22:48.046 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:48.046 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:48.046 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:48.046 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:48.046 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:22:48.046 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:22:48.046 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:22:48.046 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:22:48.047 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:22:48.047 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:48.047 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:48.047 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:22:48.047 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:22:48.047 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:22:48.047 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:22:48.047 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:48.047 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:22:48.047 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:22:48.047 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:48.047 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:48.047 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:48.047 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:22:48.047 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:22:48.048 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:22:48.048 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:48.048 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:48.048 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:48.048 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:22:48.048 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:22:48.048 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:22:48.048 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:22:48.048 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:22:48.048 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:48.048 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:48.048 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:48.048 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:22:48.048 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:22:48.048 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:48.049 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:22:48.049 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:22:48.049 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:22:48.049 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:48.049 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:22:48.049 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:22:48.049 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:48.049 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:22:48.049 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:22:48.049 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:22:48.049 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:48.049 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:22:48.049 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:22:48.049 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:48.049 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:22:48.049 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:22:48.050 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:22:48.050 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:48.050 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:22:48.050 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:22:48.050 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:22:48.050 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:22:48.050 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:22:48.050 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:22:48.050 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:22:48.050 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:22:48.050 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:22:48.050 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:22:48.050 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:22:48.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.058+0000 7f268b7fe640 1 --2- 172.21.15.37:0/3994786960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2694136450 0x7f2694131a00 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:22:48.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.058+0000 7f26996c1640 1 -- 172.21.15.37:0/3994786960 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2678035e80 msgr2=0x7f2678038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:48.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.058+0000 7f26996c1640 1 --2- 172.21.15.37:0/3994786960 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2678035e80 0x7f2678038340 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f268c003cc0 tx=0x7f268c0091f0 comp rx=0 tx=0).stop 2024-04-26T00:22:48.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.058+0000 7f26996c1640 10 monclient: shutdown 2024-04-26T00:22:48.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.058+0000 7f26996c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:22:48.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.058+0000 7f26996c1640 1 -- 172.21.15.37:0/3994786960 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2694130eb0 msgr2=0x7f269412dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:22:48.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.058+0000 7f26996c1640 1 --2- 172.21.15.37:0/3994786960 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2694130eb0 0x7f269412dcb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f267c00c7c0 tx=0x7f267c00cc80 comp rx=0 tx=0).stop 2024-04-26T00:22:48.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.058+0000 7f26937fe640 1 -- 172.21.15.37:0/3994786960 reap_dead start 2024-04-26T00:22:48.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.058+0000 7f26937fe640 1 -- 172.21.15.37:0/3994786960 reap_dead start 2024-04-26T00:22:48.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.058+0000 7f26996c1640 1 -- 172.21.15.37:0/3994786960 shutdown_connections 2024-04-26T00:22:48.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.058+0000 7f26996c1640 1 -- 172.21.15.37:0/3994786960 >> 172.21.15.37:0/3994786960 conn(0x7f26940a1d50 msgr2=0x7f26940a04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:22:48.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.058+0000 7f26996c1640 1 -- 172.21.15.37:0/3994786960 shutdown_connections 2024-04-26T00:22:48.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:22:48.058+0000 7f26996c1640 1 -- 172.21.15.37:0/3994786960 wait complete. 2024-04-26T00:22:48.054 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:22:48.054 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-04-26T00:22:48.499 INFO:tasks.rados.rados.0.smithi045.stdout:469: done (2 left) 2024-04-26T00:22:48.499 INFO:tasks.rados.rados.0.smithi045.stdout:470: done (1 left) 2024-04-26T00:22:48.499 INFO:tasks.rados.rados.0.smithi045.stdout:471: done (0 left) 2024-04-26T00:22:48.499 INFO:tasks.rados.rados.0.smithi045.stdout:472: copy_from oid 23 from oid 44 current snap is 52 2024-04-26T00:22:48.499 INFO:tasks.rados.rados.0.smithi045.stdout:473: delete oid 3 current snap is 52 2024-04-26T00:22:48.509 INFO:tasks.rados.rados.0.smithi045.stdout:472: finishing copy_from to smithi04526579-23 2024-04-26T00:22:48.509 INFO:tasks.rados.rados.0.smithi045.stdout:472: got expected ENOENT (src dne) 2024-04-26T00:22:48.509 INFO:tasks.rados.rados.0.smithi045.stdout:472: finishing copy_from racing read to smithi04526579-23 2024-04-26T00:22:48.512 INFO:tasks.rados.rados.0.smithi045.stdout:472: done (1 left) 2024-04-26T00:22:48.512 INFO:tasks.rados.rados.0.smithi045.stdout:473: done (0 left) 2024-04-26T00:22:48.513 INFO:tasks.rados.rados.0.smithi045.stdout:474: read oid 41 snap 43 2024-04-26T00:22:48.513 INFO:tasks.rados.rados.0.smithi045.stdout:474: expect (ObjNum 117 snap 28 seq_num 117) 2024-04-26T00:22:48.513 INFO:tasks.rados.rados.0.smithi045.stdout:475: snap_create 2024-04-26T00:22:48.522 INFO:tasks.rados.rados.0.smithi045.stdout:474: expect (ObjNum 117 snap 28 seq_num 117) 2024-04-26T00:22:49.507 INFO:tasks.rados.rados.0.smithi045.stdout:474: done (1 left) 2024-04-26T00:22:49.507 INFO:tasks.rados.rados.0.smithi045.stdout:475: done (0 left) 2024-04-26T00:22:49.507 INFO:tasks.rados.rados.0.smithi045.stdout:476: write (excl) oid 29 current snap is 53 2024-04-26T00:22:49.507 INFO:tasks.rados.rados.0.smithi045.stdout:476: seq_num 200 ranges {561400=474978,1820281=519723,2859422=457579} 2024-04-26T00:22:49.523 INFO:tasks.rados.rados.0.smithi045.stdout:476: writing smithi04526579-29 from 561400 to 1036378 tid 1 2024-04-26T00:22:49.545 INFO:tasks.rados.rados.0.smithi045.stdout:476: writing smithi04526579-29 from 1820281 to 2340004 tid 2 2024-04-26T00:22:49.562 INFO:tasks.rados.rados.0.smithi045.stdout:476: writing smithi04526579-29 from 2859422 to 3317001 tid 3 2024-04-26T00:22:49.562 INFO:tasks.rados.rados.0.smithi045.stdout:477: copy_from oid 10 from oid 18 current snap is 53 2024-04-26T00:22:49.562 INFO:tasks.rados.rados.0.smithi045.stdout:476: finishing write tid 1 to smithi04526579-29 2024-04-26T00:22:49.562 INFO:tasks.rados.rados.0.smithi045.stdout:476: finishing write tid 2 to smithi04526579-29 2024-04-26T00:22:49.562 INFO:tasks.rados.rados.0.smithi045.stdout:478: copy_from oid 39 from oid 11 current snap is 53 2024-04-26T00:22:49.563 INFO:tasks.rados.rados.0.smithi045.stdout:479: write oid 3 current snap is 53 2024-04-26T00:22:49.563 INFO:tasks.rados.rados.0.smithi045.stdout:479: seq_num 201 ranges {578207=550227,1762410=779734,3286845=91362} 2024-04-26T00:22:49.576 INFO:tasks.rados.rados.0.smithi045.stdout:479: writing smithi04526579-3 from 578207 to 1128434 tid 1 2024-04-26T00:22:49.591 INFO:tasks.rados.rados.0.smithi045.stdout:479: writing smithi04526579-3 from 1762410 to 2542144 tid 2 2024-04-26T00:22:49.599 INFO:tasks.rados.rados.0.smithi045.stdout:479: writing smithi04526579-3 from 3286845 to 3378207 tid 3 2024-04-26T00:22:49.599 INFO:tasks.rados.rados.0.smithi045.stdout:480: copy_from oid 9 from oid 26 current snap is 53 2024-04-26T00:22:49.599 INFO:tasks.rados.rados.0.smithi045.stdout:476: finishing write tid 3 to smithi04526579-29 2024-04-26T00:22:49.599 INFO:tasks.rados.rados.0.smithi045.stdout:476: finishing write tid 4 to smithi04526579-29 2024-04-26T00:22:49.599 INFO:tasks.rados.rados.0.smithi045.stdout:476: finishing write tid 5 to smithi04526579-29 2024-04-26T00:22:49.599 INFO:tasks.rados.rados.0.smithi045.stdout:476: oid 29 updating version 0 to 95 2024-04-26T00:22:49.599 INFO:tasks.rados.rados.0.smithi045.stdout:476: oid 29 version 95 is already newer than 92 2024-04-26T00:22:49.599 INFO:tasks.rados.rados.0.smithi045.stdout:476: oid 29 version 95 is already newer than 94 2024-04-26T00:22:49.599 INFO:tasks.rados.rados.0.smithi045.stdout:476: oid 29 version 95 is already newer than 93 2024-04-26T00:22:49.599 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 95 (ObjNum 200 snap 53 seq_num 200) dirty exists 2024-04-26T00:22:49.599 INFO:tasks.rados.rados.0.smithi045.stdout:476: left oid 29 (ObjNum 200 snap 53 seq_num 200) 2024-04-26T00:22:49.600 INFO:tasks.rados.rados.0.smithi045.stdout:477: finishing copy_from racing read to smithi04526579-10 2024-04-26T00:22:49.600 INFO:tasks.rados.rados.0.smithi045.stdout:479: finishing write tid 1 to smithi04526579-3 2024-04-26T00:22:49.600 INFO:tasks.rados.rados.0.smithi045.stdout:478: finishing copy_from racing read to smithi04526579-39 2024-04-26T00:22:49.600 INFO:tasks.rados.rados.0.smithi045.stdout:478: finishing copy_from to smithi04526579-39 2024-04-26T00:22:49.600 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 102 (ObjNum 155 snap 33 seq_num 155) dirty exists 2024-04-26T00:22:49.600 INFO:tasks.rados.rados.0.smithi045.stdout:476: done (4 left) 2024-04-26T00:22:49.600 INFO:tasks.rados.rados.0.smithi045.stdout:478: done (3 left) 2024-04-26T00:22:49.600 INFO:tasks.rados.rados.0.smithi045.stdout:481: write (excl) oid 4 current snap is 53 2024-04-26T00:22:49.600 INFO:tasks.rados.rados.0.smithi045.stdout:481: seq_num 202 ranges {595014=625476,1820892=523392,2797915=596476} 2024-04-26T00:22:49.608 INFO:tasks.rados.rados.0.smithi045.stdout:481: writing smithi04526579-4 from 595014 to 1220490 tid 1 2024-04-26T00:22:49.617 INFO:tasks.rados.rados.0.smithi045.stdout:481: writing smithi04526579-4 from 1820892 to 2344284 tid 2 2024-04-26T00:22:49.626 INFO:tasks.rados.rados.0.smithi045.stdout:481: writing smithi04526579-4 from 2797915 to 3394391 tid 3 2024-04-26T00:22:49.626 INFO:tasks.rados.rados.0.smithi045.stdout:482: snap_remove snap 49 2024-04-26T00:22:49.717 INFO:tasks.rados.rados.0.smithi045.stdout:482: done (4 left) 2024-04-26T00:22:49.718 INFO:tasks.rados.rados.0.smithi045.stdout:483: delete oid 38 current snap is 53 2024-04-26T00:22:49.718 INFO:tasks.rados.rados.0.smithi045.stdout:477: finishing copy_from to smithi04526579-10 2024-04-26T00:22:49.718 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 225 (ObjNum 165 snap 38 seq_num 165) dirty exists 2024-04-26T00:22:49.718 INFO:tasks.rados.rados.0.smithi045.stdout:479: finishing write tid 2 to smithi04526579-3 2024-04-26T00:22:49.718 INFO:tasks.rados.rados.0.smithi045.stdout:479: finishing write tid 3 to smithi04526579-3 2024-04-26T00:22:49.718 INFO:tasks.rados.rados.0.smithi045.stdout:479: finishing write tid 4 to smithi04526579-3 2024-04-26T00:22:49.718 INFO:tasks.rados.rados.0.smithi045.stdout:479: finishing write tid 5 to smithi04526579-3 2024-04-26T00:22:49.718 INFO:tasks.rados.rados.0.smithi045.stdout:479: oid 3 updating version 0 to 150 2024-04-26T00:22:49.718 INFO:tasks.rados.rados.0.smithi045.stdout:479: oid 3 updating version 150 to 151 2024-04-26T00:22:49.718 INFO:tasks.rados.rados.0.smithi045.stdout:479: oid 3 version 151 is already newer than 149 2024-04-26T00:22:49.718 INFO:tasks.rados.rados.0.smithi045.stdout:479: oid 3 updating version 151 to 152 2024-04-26T00:22:49.718 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 152 (ObjNum 201 snap 53 seq_num 201) dirty exists 2024-04-26T00:22:49.719 INFO:tasks.rados.rados.0.smithi045.stdout:479: left oid 3 (ObjNum 201 snap 53 seq_num 201) 2024-04-26T00:22:49.719 INFO:tasks.rados.rados.0.smithi045.stdout:480: finishing copy_from racing read to smithi04526579-9 2024-04-26T00:22:49.719 INFO:tasks.rados.rados.0.smithi045.stdout:480: finishing copy_from to smithi04526579-9 2024-04-26T00:22:49.719 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 175 (ObjNum 190 snap 49 seq_num 190) dirty exists 2024-04-26T00:22:49.719 INFO:tasks.rados.rados.0.smithi045.stdout:481: finishing write tid 1 to smithi04526579-4 2024-04-26T00:22:49.719 INFO:tasks.rados.rados.0.smithi045.stdout:481: finishing write tid 2 to smithi04526579-4 2024-04-26T00:22:49.719 INFO:tasks.rados.rados.0.smithi045.stdout:481: finishing write tid 3 to smithi04526579-4 2024-04-26T00:22:49.719 INFO:tasks.rados.rados.0.smithi045.stdout:481: finishing write tid 4 to smithi04526579-4 2024-04-26T00:22:49.719 INFO:tasks.rados.rados.0.smithi045.stdout:481: finishing write tid 5 to smithi04526579-4 2024-04-26T00:22:49.719 INFO:tasks.rados.rados.0.smithi045.stdout:481: oid 4 updating version 0 to 212 2024-04-26T00:22:49.719 INFO:tasks.rados.rados.0.smithi045.stdout:481: oid 4 updating version 212 to 214 2024-04-26T00:22:49.719 INFO:tasks.rados.rados.0.smithi045.stdout:481: oid 4 version 214 is already newer than 213 2024-04-26T00:22:49.719 INFO:tasks.rados.rados.0.smithi045.stdout:481: oid 4 version 214 is already newer than 211 2024-04-26T00:22:49.719 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 214 (ObjNum 202 snap 53 seq_num 202) dirty exists 2024-04-26T00:22:49.720 INFO:tasks.rados.rados.0.smithi045.stdout:481: left oid 4 (ObjNum 202 snap 53 seq_num 202) 2024-04-26T00:22:49.733 INFO:tasks.rados.rados.0.smithi045.stdout:477: done (4 left) 2024-04-26T00:22:49.733 INFO:tasks.rados.rados.0.smithi045.stdout:479: done (3 left) 2024-04-26T00:22:49.733 INFO:tasks.rados.rados.0.smithi045.stdout:480: done (2 left) 2024-04-26T00:22:49.733 INFO:tasks.rados.rados.0.smithi045.stdout:481: done (1 left) 2024-04-26T00:22:49.734 INFO:tasks.rados.rados.0.smithi045.stdout:483: done (0 left) 2024-04-26T00:22:49.734 INFO:tasks.rados.rados.0.smithi045.stdout:484: write (excl) oid 13 current snap is 53 2024-04-26T00:22:49.734 INFO:tasks.rados.rados.0.smithi045.stdout:484: seq_num 204 ranges {628628=775974,1937856=527061,3252761=175867} 2024-04-26T00:22:49.754 INFO:tasks.rados.rados.0.smithi045.stdout:484: writing smithi04526579-13 from 628628 to 1404602 tid 1 2024-04-26T00:22:49.765 INFO:tasks.rados.rados.0.smithi045.stdout:484: writing smithi04526579-13 from 1937856 to 2464917 tid 2 2024-04-26T00:22:49.777 INFO:tasks.rados.rados.0.smithi045.stdout:484: writing smithi04526579-13 from 3252761 to 3428628 tid 3 2024-04-26T00:22:49.777 INFO:tasks.rados.rados.0.smithi045.stdout:485: rollback oid 25 current snap is 53 2024-04-26T00:22:49.777 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 25 to 46 2024-04-26T00:22:49.777 INFO:tasks.rados.rados.0.smithi045.stdout:484: finishing write tid 1 to smithi04526579-13 2024-04-26T00:22:49.777 INFO:tasks.rados.rados.0.smithi045.stdout:484: finishing write tid 2 to smithi04526579-13 2024-04-26T00:22:49.777 INFO:tasks.rados.rados.0.smithi045.stdout:486: write (excl) oid 47 current snap is 53 2024-04-26T00:22:49.777 INFO:tasks.rados.rados.0.smithi045.stdout:486: seq_num 205 ranges {645435=451223,1879985=787072,3280184=165251} 2024-04-26T00:22:49.789 INFO:tasks.rados.rados.0.smithi045.stdout:486: writing smithi04526579-47 from 645435 to 1096658 tid 1 2024-04-26T00:22:49.803 INFO:tasks.rados.rados.0.smithi045.stdout:486: writing smithi04526579-47 from 1879985 to 2667057 tid 2 2024-04-26T00:22:49.810 INFO:tasks.rados.rados.0.smithi045.stdout:486: writing smithi04526579-47 from 3280184 to 3445435 tid 3 2024-04-26T00:22:49.811 INFO:tasks.rados.rados.0.smithi045.stdout:487: copy_from oid 20 from oid 4 current snap is 53 2024-04-26T00:22:49.811 INFO:tasks.rados.rados.0.smithi045.stdout:484: finishing write tid 3 to smithi04526579-13 2024-04-26T00:22:49.811 INFO:tasks.rados.rados.0.smithi045.stdout:488: write oid 48 current snap is 53 2024-04-26T00:22:49.811 INFO:tasks.rados.rados.0.smithi045.stdout:484: finishing write tid 4 to smithi04526579-13 2024-04-26T00:22:49.811 INFO:tasks.rados.rados.0.smithi045.stdout:484: finishing write tid 5 to smithi04526579-13 2024-04-26T00:22:49.811 INFO:tasks.rados.rados.0.smithi045.stdout:484: oid 13 updating version 0 to 101 2024-04-26T00:22:49.811 INFO:tasks.rados.rados.0.smithi045.stdout:484: oid 13 version 101 is already newer than 99 2024-04-26T00:22:49.811 INFO:tasks.rados.rados.0.smithi045.stdout:484: oid 13 version 101 is already newer than 98 2024-04-26T00:22:49.812 INFO:tasks.rados.rados.0.smithi045.stdout:484: oid 13 version 101 is already newer than 100 2024-04-26T00:22:49.812 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 101 (ObjNum 204 snap 53 seq_num 204) dirty exists 2024-04-26T00:22:49.812 INFO:tasks.rados.rados.0.smithi045.stdout:484: left oid 13 (ObjNum 204 snap 53 seq_num 204) 2024-04-26T00:22:49.812 INFO:tasks.rados.rados.0.smithi045.stdout:485: finishing rollback tid 0 to smithi04526579-25 2024-04-26T00:22:49.812 INFO:tasks.rados.rados.0.smithi045.stdout:485: finishing rollback tid 1 to smithi04526579-25 2024-04-26T00:22:49.812 INFO:tasks.rados.rados.0.smithi045.stdout:485: finishing rollback tid 2 to smithi04526579-25 2024-04-26T00:22:49.812 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 263 (ObjNum 72 snap 11 seq_num 72) dirty exists 2024-04-26T00:22:49.813 INFO:tasks.rados.rados.0.smithi045.stdout:488: seq_num 206 ranges {662242=642825,2054820=530730,3307607=154635} 2024-04-26T00:22:49.820 INFO:tasks.rados.rados.0.smithi045.stdout:488: writing smithi04526579-48 from 662242 to 1305067 tid 1 2024-04-26T00:22:49.830 INFO:tasks.rados.rados.0.smithi045.stdout:488: writing smithi04526579-48 from 2054820 to 2585550 tid 2 2024-04-26T00:22:49.838 INFO:tasks.rados.rados.0.smithi045.stdout:488: writing smithi04526579-48 from 3307607 to 3462242 tid 3 2024-04-26T00:22:49.838 INFO:tasks.rados.rados.0.smithi045.stdout:484: done (4 left) 2024-04-26T00:22:49.838 INFO:tasks.rados.rados.0.smithi045.stdout:485: done (3 left) 2024-04-26T00:22:49.838 INFO:tasks.rados.rados.0.smithi045.stdout:489: delete oid 37 current snap is 53 2024-04-26T00:22:49.838 INFO:tasks.rados.rados.0.smithi045.stdout:486: finishing write tid 1 to smithi04526579-47 2024-04-26T00:22:49.839 INFO:tasks.rados.rados.0.smithi045.stdout:486: finishing write tid 2 to smithi04526579-47 2024-04-26T00:22:49.839 INFO:tasks.rados.rados.0.smithi045.stdout:486: finishing write tid 3 to smithi04526579-47 2024-04-26T00:22:49.839 INFO:tasks.rados.rados.0.smithi045.stdout:486: finishing write tid 4 to smithi04526579-47 2024-04-26T00:22:49.839 INFO:tasks.rados.rados.0.smithi045.stdout:486: finishing write tid 5 to smithi04526579-47 2024-04-26T00:22:49.839 INFO:tasks.rados.rados.0.smithi045.stdout:486: oid 47 updating version 0 to 154 2024-04-26T00:22:49.839 INFO:tasks.rados.rados.0.smithi045.stdout:486: oid 47 updating version 154 to 160 2024-04-26T00:22:49.839 INFO:tasks.rados.rados.0.smithi045.stdout:486: oid 47 version 160 is already newer than 159 2024-04-26T00:22:49.839 INFO:tasks.rados.rados.0.smithi045.stdout:486: oid 47 updating version 160 to 161 2024-04-26T00:22:49.839 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 161 (ObjNum 205 snap 53 seq_num 205) dirty exists 2024-04-26T00:22:49.839 INFO:tasks.rados.rados.0.smithi045.stdout:486: left oid 47 (ObjNum 205 snap 53 seq_num 205) 2024-04-26T00:22:49.839 INFO:tasks.rados.rados.0.smithi045.stdout:488: finishing write tid 1 to smithi04526579-48 2024-04-26T00:22:49.839 INFO:tasks.rados.rados.0.smithi045.stdout:487: finishing copy_from racing read to smithi04526579-20 2024-04-26T00:22:49.839 INFO:tasks.rados.rados.0.smithi045.stdout:488: finishing write tid 2 to smithi04526579-48 2024-04-26T00:22:49.841 INFO:tasks.rados.rados.0.smithi045.stdout:487: finishing copy_from to smithi04526579-20 2024-04-26T00:22:49.841 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 214 (ObjNum 202 snap 53 seq_num 202) dirty exists 2024-04-26T00:22:49.843 INFO:tasks.rados.rados.0.smithi045.stdout:488: finishing write tid 3 to smithi04526579-48 2024-04-26T00:22:49.844 INFO:tasks.rados.rados.0.smithi045.stdout:486: done (3 left) 2024-04-26T00:22:49.844 INFO:tasks.rados.rados.0.smithi045.stdout:487: done (2 left) 2024-04-26T00:22:49.844 INFO:tasks.rados.rados.0.smithi045.stdout:489: done (1 left) 2024-04-26T00:22:49.844 INFO:tasks.rados.rados.0.smithi045.stdout:490: read oid 42 snap -1 2024-04-26T00:22:49.844 INFO:tasks.rados.rados.0.smithi045.stdout:490: expect (ObjNum 110 snap 28 seq_num 110) 2024-04-26T00:22:49.845 INFO:tasks.rados.rados.0.smithi045.stdout:491: copy_from oid 10 from oid 44 current snap is 53 2024-04-26T00:22:49.845 INFO:tasks.rados.rados.0.smithi045.stdout:488: finishing write tid 4 to smithi04526579-48 2024-04-26T00:22:49.845 INFO:tasks.rados.rados.0.smithi045.stdout:492: read oid 3 snap -1 2024-04-26T00:22:49.845 INFO:tasks.rados.rados.0.smithi045.stdout:492: expect (ObjNum 201 snap 53 seq_num 201) 2024-04-26T00:22:49.845 INFO:tasks.rados.rados.0.smithi045.stdout:493: delete oid 21 current snap is 53 2024-04-26T00:22:49.846 INFO:tasks.rados.rados.0.smithi045.stdout:488: finishing write tid 5 to smithi04526579-48 2024-04-26T00:22:49.846 INFO:tasks.rados.rados.0.smithi045.stdout:488: oid 48 updating version 0 to 163 2024-04-26T00:22:49.846 INFO:tasks.rados.rados.0.smithi045.stdout:488: oid 48 updating version 163 to 165 2024-04-26T00:22:49.846 INFO:tasks.rados.rados.0.smithi045.stdout:488: oid 48 version 165 is already newer than 164 2024-04-26T00:22:49.846 INFO:tasks.rados.rados.0.smithi045.stdout:488: oid 48 version 165 is already newer than 162 2024-04-26T00:22:49.846 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 165 (ObjNum 206 snap 53 seq_num 206) dirty exists 2024-04-26T00:22:49.846 INFO:tasks.rados.rados.0.smithi045.stdout:488: left oid 48 (ObjNum 206 snap 53 seq_num 206) 2024-04-26T00:22:49.846 INFO:tasks.rados.rados.0.smithi045.stdout:491: finishing copy_from to smithi04526579-10 2024-04-26T00:22:49.846 INFO:tasks.rados.rados.0.smithi045.stdout:491: got expected ENOENT (src dne) 2024-04-26T00:22:49.846 INFO:tasks.rados.rados.0.smithi045.stdout:491: finishing copy_from racing read to smithi04526579-10 2024-04-26T00:22:49.848 INFO:tasks.rados.rados.0.smithi045.stdout:488: done (4 left) 2024-04-26T00:22:49.848 INFO:tasks.rados.rados.0.smithi045.stdout:491: done (3 left) 2024-04-26T00:22:49.848 INFO:tasks.rados.rados.0.smithi045.stdout:493: done (2 left) 2024-04-26T00:22:49.848 INFO:tasks.rados.rados.0.smithi045.stdout:494: read oid 13 snap -1 2024-04-26T00:22:49.848 INFO:tasks.rados.rados.0.smithi045.stdout:494: expect (ObjNum 204 snap 53 seq_num 204) 2024-04-26T00:22:49.848 INFO:tasks.rados.rados.0.smithi045.stdout:495: write (excl) oid 26 current snap is 53 2024-04-26T00:22:49.848 INFO:tasks.rados.rados.0.smithi045.stdout:495: seq_num 209 ranges {712663=468572,1713913=678057,2873523=591733} 2024-04-26T00:22:49.860 INFO:tasks.rados.rados.0.smithi045.stdout:495: writing smithi04526579-26 from 712663 to 1181235 tid 1 2024-04-26T00:22:49.870 INFO:tasks.rados.rados.0.smithi045.stdout:495: writing smithi04526579-26 from 1713913 to 2391970 tid 2 2024-04-26T00:22:49.879 INFO:tasks.rados.rados.0.smithi045.stdout:495: writing smithi04526579-26 from 2873523 to 3465256 tid 3 2024-04-26T00:22:49.879 INFO:tasks.rados.rados.0.smithi045.stdout:496: read oid 32 snap -1 2024-04-26T00:22:49.879 INFO:tasks.rados.rados.0.smithi045.stdout:496: expect (ObjNum 196 snap 50 seq_num 196) 2024-04-26T00:22:49.879 INFO:tasks.rados.rados.0.smithi045.stdout:497: read oid 37 snap -1 2024-04-26T00:22:49.879 INFO:tasks.rados.rados.0.smithi045.stdout:497: expect deleted 2024-04-26T00:22:49.879 INFO:tasks.rados.rados.0.smithi045.stdout:498: write oid 27 current snap is 53 2024-04-26T00:22:49.879 INFO:tasks.rados.rados.0.smithi045.stdout:498: seq_num 210 ranges {729470=543821,1772395=538068,2900946=403005} 2024-04-26T00:22:49.889 INFO:tasks.rados.rados.0.smithi045.stdout:498: writing smithi04526579-27 from 729470 to 1273291 tid 1 2024-04-26T00:22:49.896 INFO:tasks.rados.rados.0.smithi045.stdout:498: writing smithi04526579-27 from 1772395 to 2310463 tid 2 2024-04-26T00:22:49.903 INFO:tasks.rados.rados.0.smithi045.stdout:498: writing smithi04526579-27 from 2900946 to 3303951 tid 3 2024-04-26T00:22:49.903 INFO:tasks.rados.rados.0.smithi045.stdout:499: write oid 15 current snap is 53 2024-04-26T00:22:49.903 INFO:tasks.rados.rados.0.smithi045.stdout:499: seq_num 211 ranges {746277=619070,1830877=681726,2928369=617908} 2024-04-26T00:22:49.912 INFO:tasks.rados.rados.0.smithi045.stdout:499: writing smithi04526579-15 from 746277 to 1365347 tid 1 2024-04-26T00:22:49.919 INFO:tasks.rados.rados.0.smithi045.stdout:499: writing smithi04526579-15 from 1830877 to 2512603 tid 2 2024-04-26T00:22:49.925 INFO:tasks.rados.rados.0.smithi045.stdout:499: writing smithi04526579-15 from 2928369 to 3546277 tid 3 2024-04-26T00:22:49.925 INFO:tasks.rados.rados.0.smithi045.stdout:500: snap_remove snap 43 2024-04-26T00:22:50.727 INFO:tasks.rados.rados.0.smithi045.stdout:500: done (8 left) 2024-04-26T00:22:50.727 INFO:tasks.rados.rados.0.smithi045.stdout:501: write oid 46 current snap is 53 2024-04-26T00:22:50.727 INFO:tasks.rados.rados.0.smithi045.stdout:501: seq_num 212 ranges {763084=694319,1889359=541737,2955792=541902} 2024-04-26T00:22:50.735 INFO:tasks.rados.rados.0.smithi045.stdout:501: writing smithi04526579-46 from 763084 to 1457403 tid 1 2024-04-26T00:22:50.740 INFO:tasks.rados.rados.0.smithi045.stdout:501: writing smithi04526579-46 from 1889359 to 2431096 tid 2 2024-04-26T00:22:50.747 INFO:tasks.rados.rados.0.smithi045.stdout:501: writing smithi04526579-46 from 2955792 to 3497694 tid 3 2024-04-26T00:22:50.747 INFO:tasks.rados.rados.0.smithi045.stdout:502: write (excl) oid 43 current snap is 53 2024-04-26T00:22:50.747 INFO:tasks.rados.rados.0.smithi045.stdout:502: seq_num 213 ranges {779891=485921,1947841=685395,3383215=196676} 2024-04-26T00:22:50.755 INFO:tasks.rados.rados.0.smithi045.stdout:502: writing smithi04526579-43 from 779891 to 1265812 tid 1 2024-04-26T00:22:50.763 INFO:tasks.rados.rados.0.smithi045.stdout:502: writing smithi04526579-43 from 1947841 to 2633236 tid 2 2024-04-26T00:22:50.769 INFO:tasks.rados.rados.0.smithi045.stdout:502: writing smithi04526579-43 from 3383215 to 3579891 tid 3 2024-04-26T00:22:50.769 INFO:tasks.rados.rados.0.smithi045.stdout:503: snap_remove snap 41 2024-04-26T00:22:51.735 INFO:tasks.rados.rados.0.smithi045.stdout:503: done (10 left) 2024-04-26T00:22:51.736 INFO:tasks.rados.rados.0.smithi045.stdout:504: read oid 6 snap 52 2024-04-26T00:22:51.736 INFO:tasks.rados.rados.0.smithi045.stdout:504: expect deleted 2024-04-26T00:22:51.736 INFO:tasks.rados.rados.0.smithi045.stdout:505: snap_create 2024-04-26T00:22:51.736 INFO:tasks.rados.rados.0.smithi045.stdout:490: expect (ObjNum 110 snap 28 seq_num 110) 2024-04-26T00:22:51.896 INFO:tasks.rados.rados.0.smithi045.stdout:494: expect (ObjNum 204 snap 53 seq_num 204) 2024-04-26T00:22:52.008 INFO:tasks.rados.rados.0.smithi045.stdout:492: expect (ObjNum 201 snap 53 seq_num 201) 2024-04-26T00:22:52.120 INFO:tasks.rados.rados.0.smithi045.stdout:495: finishing write tid 1 to smithi04526579-26 2024-04-26T00:22:52.120 INFO:tasks.rados.rados.0.smithi045.stdout:495: finishing write tid 2 to smithi04526579-26 2024-04-26T00:22:52.120 INFO:tasks.rados.rados.0.smithi045.stdout:495: finishing write tid 3 to smithi04526579-26 2024-04-26T00:22:52.120 INFO:tasks.rados.rados.0.smithi045.stdout:495: finishing write tid 4 to smithi04526579-26 2024-04-26T00:22:52.120 INFO:tasks.rados.rados.0.smithi045.stdout:495: finishing write tid 5 to smithi04526579-26 2024-04-26T00:22:52.120 INFO:tasks.rados.rados.0.smithi045.stdout:495: oid 26 updating version 0 to 188 2024-04-26T00:22:52.120 INFO:tasks.rados.rados.0.smithi045.stdout:495: oid 26 updating version 188 to 191 2024-04-26T00:22:52.120 INFO:tasks.rados.rados.0.smithi045.stdout:495: oid 26 version 191 is already newer than 189 2024-04-26T00:22:52.120 INFO:tasks.rados.rados.0.smithi045.stdout:495: oid 26 version 191 is already newer than 190 2024-04-26T00:22:52.120 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 191 (ObjNum 209 snap 53 seq_num 209) dirty exists 2024-04-26T00:22:52.120 INFO:tasks.rados.rados.0.smithi045.stdout:495: left oid 26 (ObjNum 209 snap 53 seq_num 209) 2024-04-26T00:22:52.120 INFO:tasks.rados.rados.0.smithi045.stdout:496: expect (ObjNum 196 snap 50 seq_num 196) 2024-04-26T00:22:52.257 INFO:tasks.rados.rados.0.smithi045.stdout:498: finishing write tid 1 to smithi04526579-27 2024-04-26T00:22:52.257 INFO:tasks.rados.rados.0.smithi045.stdout:498: finishing write tid 2 to smithi04526579-27 2024-04-26T00:22:52.257 INFO:tasks.rados.rados.0.smithi045.stdout:498: finishing write tid 3 to smithi04526579-27 2024-04-26T00:22:52.257 INFO:tasks.rados.rados.0.smithi045.stdout:498: finishing write tid 4 to smithi04526579-27 2024-04-26T00:22:52.257 INFO:tasks.rados.rados.0.smithi045.stdout:498: finishing write tid 5 to smithi04526579-27 2024-04-26T00:22:52.257 INFO:tasks.rados.rados.0.smithi045.stdout:498: oid 27 updating version 0 to 158 2024-04-26T00:22:52.257 INFO:tasks.rados.rados.0.smithi045.stdout:498: oid 27 version 158 is already newer than 156 2024-04-26T00:22:52.257 INFO:tasks.rados.rados.0.smithi045.stdout:498: oid 27 version 158 is already newer than 157 2024-04-26T00:22:52.257 INFO:tasks.rados.rados.0.smithi045.stdout:498: oid 27 version 158 is already newer than 155 2024-04-26T00:22:52.257 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 158 (ObjNum 210 snap 53 seq_num 210) dirty exists 2024-04-26T00:22:52.258 INFO:tasks.rados.rados.0.smithi045.stdout:498: left oid 27 (ObjNum 210 snap 53 seq_num 210) 2024-04-26T00:22:52.258 INFO:tasks.rados.rados.0.smithi045.stdout:499: finishing write tid 1 to smithi04526579-15 2024-04-26T00:22:52.258 INFO:tasks.rados.rados.0.smithi045.stdout:499: finishing write tid 2 to smithi04526579-15 2024-04-26T00:22:52.258 INFO:tasks.rados.rados.0.smithi045.stdout:499: finishing write tid 3 to smithi04526579-15 2024-04-26T00:22:52.258 INFO:tasks.rados.rados.0.smithi045.stdout:499: finishing write tid 4 to smithi04526579-15 2024-04-26T00:22:52.258 INFO:tasks.rados.rados.0.smithi045.stdout:499: finishing write tid 5 to smithi04526579-15 2024-04-26T00:22:52.258 INFO:tasks.rados.rados.0.smithi045.stdout:499: oid 15 updating version 0 to 193 2024-04-26T00:22:52.258 INFO:tasks.rados.rados.0.smithi045.stdout:499: oid 15 updating version 193 to 196 2024-04-26T00:22:52.258 INFO:tasks.rados.rados.0.smithi045.stdout:499: oid 15 version 196 is already newer than 194 2024-04-26T00:22:52.258 INFO:tasks.rados.rados.0.smithi045.stdout:499: oid 15 version 196 is already newer than 195 2024-04-26T00:22:52.258 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 196 (ObjNum 211 snap 53 seq_num 211) dirty exists 2024-04-26T00:22:52.258 INFO:tasks.rados.rados.0.smithi045.stdout:499: left oid 15 (ObjNum 211 snap 53 seq_num 211) 2024-04-26T00:22:52.258 INFO:tasks.rados.rados.0.smithi045.stdout:502: finishing write tid 1 to smithi04526579-43 2024-04-26T00:22:52.259 INFO:tasks.rados.rados.0.smithi045.stdout:501: finishing write tid 1 to smithi04526579-46 2024-04-26T00:22:52.259 INFO:tasks.rados.rados.0.smithi045.stdout:501: finishing write tid 2 to smithi04526579-46 2024-04-26T00:22:52.259 INFO:tasks.rados.rados.0.smithi045.stdout:501: finishing write tid 3 to smithi04526579-46 2024-04-26T00:22:52.259 INFO:tasks.rados.rados.0.smithi045.stdout:501: finishing write tid 4 to smithi04526579-46 2024-04-26T00:22:52.259 INFO:tasks.rados.rados.0.smithi045.stdout:501: finishing write tid 5 to smithi04526579-46 2024-04-26T00:22:52.259 INFO:tasks.rados.rados.0.smithi045.stdout:501: oid 46 updating version 0 to 112 2024-04-26T00:22:52.259 INFO:tasks.rados.rados.0.smithi045.stdout:501: oid 46 updating version 112 to 114 2024-04-26T00:22:52.259 INFO:tasks.rados.rados.0.smithi045.stdout:501: oid 46 updating version 114 to 115 2024-04-26T00:22:52.259 INFO:tasks.rados.rados.0.smithi045.stdout:501: oid 46 version 115 is already newer than 113 2024-04-26T00:22:52.259 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 115 (ObjNum 212 snap 53 seq_num 212) dirty exists 2024-04-26T00:22:52.259 INFO:tasks.rados.rados.0.smithi045.stdout:501: left oid 46 (ObjNum 212 snap 53 seq_num 212) 2024-04-26T00:22:52.259 INFO:tasks.rados.rados.0.smithi045.stdout:502: finishing write tid 2 to smithi04526579-43 2024-04-26T00:22:52.259 INFO:tasks.rados.rados.0.smithi045.stdout:502: finishing write tid 3 to smithi04526579-43 2024-04-26T00:22:52.259 INFO:tasks.rados.rados.0.smithi045.stdout:502: finishing write tid 4 to smithi04526579-43 2024-04-26T00:22:52.260 INFO:tasks.rados.rados.0.smithi045.stdout:502: finishing write tid 5 to smithi04526579-43 2024-04-26T00:22:52.260 INFO:tasks.rados.rados.0.smithi045.stdout:502: oid 43 updating version 0 to 227 2024-04-26T00:22:52.260 INFO:tasks.rados.rados.0.smithi045.stdout:502: oid 43 updating version 227 to 228 2024-04-26T00:22:52.260 INFO:tasks.rados.rados.0.smithi045.stdout:502: oid 43 version 228 is already newer than 226 2024-04-26T00:22:52.260 INFO:tasks.rados.rados.0.smithi045.stdout:502: oid 43 updating version 228 to 229 2024-04-26T00:22:52.260 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 229 (ObjNum 213 snap 53 seq_num 213) dirty exists 2024-04-26T00:22:52.260 INFO:tasks.rados.rados.0.smithi045.stdout:502: left oid 43 (ObjNum 213 snap 53 seq_num 213) 2024-04-26T00:22:52.741 INFO:tasks.rados.rados.0.smithi045.stdout:490: done (11 left) 2024-04-26T00:22:52.741 INFO:tasks.rados.rados.0.smithi045.stdout:492: done (10 left) 2024-04-26T00:22:52.741 INFO:tasks.rados.rados.0.smithi045.stdout:494: done (9 left) 2024-04-26T00:22:52.741 INFO:tasks.rados.rados.0.smithi045.stdout:495: done (8 left) 2024-04-26T00:22:52.741 INFO:tasks.rados.rados.0.smithi045.stdout:496: done (7 left) 2024-04-26T00:22:52.742 INFO:tasks.rados.rados.0.smithi045.stdout:497: done (6 left) 2024-04-26T00:22:52.742 INFO:tasks.rados.rados.0.smithi045.stdout:498: done (5 left) 2024-04-26T00:22:52.742 INFO:tasks.rados.rados.0.smithi045.stdout:499: done (4 left) 2024-04-26T00:22:52.742 INFO:tasks.rados.rados.0.smithi045.stdout:501: done (3 left) 2024-04-26T00:22:52.742 INFO:tasks.rados.rados.0.smithi045.stdout:502: done (2 left) 2024-04-26T00:22:52.742 INFO:tasks.rados.rados.0.smithi045.stdout:504: done (1 left) 2024-04-26T00:22:52.742 INFO:tasks.rados.rados.0.smithi045.stdout:505: done (0 left) 2024-04-26T00:22:52.742 INFO:tasks.rados.rados.0.smithi045.stdout:506: delete oid 15 current snap is 54 2024-04-26T00:22:52.760 INFO:tasks.rados.rados.0.smithi045.stdout:506: done (0 left) 2024-04-26T00:22:52.760 INFO:tasks.rados.rados.0.smithi045.stdout:507: snap_create 2024-04-26T00:22:54.369 INFO:tasks.rados.rados.0.smithi045.stdout:507: done (0 left) 2024-04-26T00:22:54.369 INFO:tasks.rados.rados.0.smithi045.stdout:508: read oid 38 snap -1 2024-04-26T00:22:54.370 INFO:tasks.rados.rados.0.smithi045.stdout:508: expect deleted 2024-04-26T00:22:54.370 INFO:tasks.rados.rados.0.smithi045.stdout:509: write oid 37 current snap is 55 2024-04-26T00:22:54.370 INFO:tasks.rados.rados.0.smithi045.stdout:509: seq_num 215 ranges {413505=636419,1664805=689064,3038061=492071} 2024-04-26T00:22:54.375 INFO:tasks.rados.rados.0.smithi045.stdout:509: writing smithi04526579-37 from 413505 to 1049924 tid 1 2024-04-26T00:22:54.382 INFO:tasks.rados.rados.0.smithi045.stdout:509: writing smithi04526579-37 from 1664805 to 2353869 tid 2 2024-04-26T00:22:54.390 INFO:tasks.rados.rados.0.smithi045.stdout:509: writing smithi04526579-37 from 3038061 to 3530132 tid 3 2024-04-26T00:22:54.391 INFO:tasks.rados.rados.0.smithi045.stdout:510: delete oid 6 current snap is 55 2024-04-26T00:22:54.392 INFO:tasks.rados.rados.0.smithi045.stdout:508: done (2 left) 2024-04-26T00:22:54.392 INFO:tasks.rados.rados.0.smithi045.stdout:510: done (1 left) 2024-04-26T00:22:54.392 INFO:tasks.rados.rados.0.smithi045.stdout:511: write oid 10 current snap is 55 2024-04-26T00:22:54.393 INFO:tasks.rados.rados.0.smithi045.stdout:511: seq_num 217 ranges {447119=786917,1665416=692733,2976554=514615} 2024-04-26T00:22:54.399 INFO:tasks.rados.rados.0.smithi045.stdout:511: writing smithi04526579-10 from 447119 to 1234036 tid 1 2024-04-26T00:22:54.406 INFO:tasks.rados.rados.0.smithi045.stdout:511: writing smithi04526579-10 from 1665416 to 2358149 tid 2 2024-04-26T00:22:54.413 INFO:tasks.rados.rados.0.smithi045.stdout:511: writing smithi04526579-10 from 2976554 to 3491169 tid 3 2024-04-26T00:22:54.413 INFO:tasks.rados.rados.0.smithi045.stdout:512: rollback oid 42 current snap is 55 2024-04-26T00:22:54.413 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 42 to 32 2024-04-26T00:22:54.414 INFO:tasks.rados.rados.0.smithi045.stdout:509: finishing write tid 1 to smithi04526579-37 2024-04-26T00:22:54.414 INFO:tasks.rados.rados.0.smithi045.stdout:509: finishing write tid 2 to smithi04526579-37 2024-04-26T00:22:54.414 INFO:tasks.rados.rados.0.smithi045.stdout:509: finishing write tid 3 to smithi04526579-37 2024-04-26T00:22:54.414 INFO:tasks.rados.rados.0.smithi045.stdout:509: finishing write tid 4 to smithi04526579-37 2024-04-26T00:22:54.414 INFO:tasks.rados.rados.0.smithi045.stdout:509: finishing write tid 5 to smithi04526579-37 2024-04-26T00:22:54.414 INFO:tasks.rados.rados.0.smithi045.stdout:509: oid 37 updating version 0 to 280 2024-04-26T00:22:54.414 INFO:tasks.rados.rados.0.smithi045.stdout:509: oid 37 updating version 280 to 281 2024-04-26T00:22:54.414 INFO:tasks.rados.rados.0.smithi045.stdout:509: oid 37 version 281 is already newer than 279 2024-04-26T00:22:54.414 INFO:tasks.rados.rados.0.smithi045.stdout:509: oid 37 version 281 is already newer than 278 2024-04-26T00:22:54.414 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 281 (ObjNum 215 snap 55 seq_num 215) dirty exists 2024-04-26T00:22:54.415 INFO:tasks.rados.rados.0.smithi045.stdout:509: left oid 37 (ObjNum 215 snap 55 seq_num 215) 2024-04-26T00:22:54.415 INFO:tasks.rados.rados.0.smithi045.stdout:509: done (2 left) 2024-04-26T00:22:54.415 INFO:tasks.rados.rados.0.smithi045.stdout:513: write oid 5 current snap is 55 2024-04-26T00:22:54.415 INFO:tasks.rados.rados.0.smithi045.stdout:513: seq_num 218 ranges {463926=462166,1723898=436391,2603977=725887} 2024-04-26T00:22:54.418 INFO:tasks.rados.rados.0.smithi045.stdout:513: writing smithi04526579-5 from 463926 to 926092 tid 1 2024-04-26T00:22:54.425 INFO:tasks.rados.rados.0.smithi045.stdout:513: writing smithi04526579-5 from 1723898 to 2160289 tid 2 2024-04-26T00:22:54.433 INFO:tasks.rados.rados.0.smithi045.stdout:513: writing smithi04526579-5 from 2603977 to 3329864 tid 3 2024-04-26T00:22:54.433 INFO:tasks.rados.rados.0.smithi045.stdout:514: rollback oid 41 current snap is 55 2024-04-26T00:22:54.433 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 41 to 51 2024-04-26T00:22:54.433 INFO:tasks.rados.rados.0.smithi045.stdout:511: finishing write tid 1 to smithi04526579-10 2024-04-26T00:22:54.433 INFO:tasks.rados.rados.0.smithi045.stdout:511: finishing write tid 2 to smithi04526579-10 2024-04-26T00:22:54.434 INFO:tasks.rados.rados.0.smithi045.stdout:511: finishing write tid 3 to smithi04526579-10 2024-04-26T00:22:54.434 INFO:tasks.rados.rados.0.smithi045.stdout:511: finishing write tid 4 to smithi04526579-10 2024-04-26T00:22:54.434 INFO:tasks.rados.rados.0.smithi045.stdout:511: finishing write tid 5 to smithi04526579-10 2024-04-26T00:22:54.434 INFO:tasks.rados.rados.0.smithi045.stdout:511: oid 10 updating version 0 to 230 2024-04-26T00:22:54.434 INFO:tasks.rados.rados.0.smithi045.stdout:511: oid 10 updating version 230 to 231 2024-04-26T00:22:54.434 INFO:tasks.rados.rados.0.smithi045.stdout:511: oid 10 updating version 231 to 233 2024-04-26T00:22:54.434 INFO:tasks.rados.rados.0.smithi045.stdout:511: oid 10 version 233 is already newer than 232 2024-04-26T00:22:54.434 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 233 (ObjNum 217 snap 55 seq_num 217) dirty exists 2024-04-26T00:22:54.434 INFO:tasks.rados.rados.0.smithi045.stdout:511: left oid 10 (ObjNum 217 snap 55 seq_num 217) 2024-04-26T00:22:54.434 INFO:tasks.rados.rados.0.smithi045.stdout:512: finishing rollback tid 0 to smithi04526579-42 2024-04-26T00:22:54.434 INFO:tasks.rados.rados.0.smithi045.stdout:512: finishing rollback tid 1 to smithi04526579-42 2024-04-26T00:22:54.434 INFO:tasks.rados.rados.0.smithi045.stdout:512: finishing rollback tid 2 to smithi04526579-42 2024-04-26T00:22:54.434 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 161 (ObjNum 114 snap 28 seq_num 114) dirty exists 2024-04-26T00:22:54.434 INFO:tasks.rados.rados.0.smithi045.stdout:513: finishing write tid 1 to smithi04526579-5 2024-04-26T00:22:54.435 INFO:tasks.rados.rados.0.smithi045.stdout:513: finishing write tid 2 to smithi04526579-5 2024-04-26T00:22:54.435 INFO:tasks.rados.rados.0.smithi045.stdout:511: done (3 left) 2024-04-26T00:22:54.435 INFO:tasks.rados.rados.0.smithi045.stdout:512: done (2 left) 2024-04-26T00:22:54.435 INFO:tasks.rados.rados.0.smithi045.stdout:515: copy_from oid 32 from oid 50 current snap is 55 2024-04-26T00:22:54.435 INFO:tasks.rados.rados.0.smithi045.stdout:516: copy_from oid 16 from oid 46 current snap is 55 2024-04-26T00:22:54.435 INFO:tasks.rados.rados.0.smithi045.stdout:517: delete oid 23 current snap is 55 2024-04-26T00:22:54.435 INFO:tasks.rados.rados.0.smithi045.stdout:517: done (4 left) 2024-04-26T00:22:54.435 INFO:tasks.rados.rados.0.smithi045.stdout:518: write oid 19 current snap is 55 2024-04-26T00:22:54.435 INFO:tasks.rados.rados.0.smithi045.stdout:518: seq_num 220 ranges {497540=612664,1840862=440060,3058823=464784} 2024-04-26T00:22:54.443 INFO:tasks.rados.rados.0.smithi045.stdout:518: writing smithi04526579-19 from 497540 to 1110204 tid 1 2024-04-26T00:22:54.450 INFO:tasks.rados.rados.0.smithi045.stdout:518: writing smithi04526579-19 from 1840862 to 2280922 tid 2 2024-04-26T00:22:54.458 INFO:tasks.rados.rados.0.smithi045.stdout:518: writing smithi04526579-19 from 3058823 to 3523607 tid 3 2024-04-26T00:22:54.458 INFO:tasks.rados.rados.0.smithi045.stdout:519: write oid 14 current snap is 55 2024-04-26T00:22:54.458 INFO:tasks.rados.rados.0.smithi045.stdout:519: seq_num 221 ranges {514347=404266,1499344=700071,2686246=676056} 2024-04-26T00:22:54.463 INFO:tasks.rados.rados.0.smithi045.stdout:519: writing smithi04526579-14 from 514347 to 918613 tid 1 2024-04-26T00:22:54.470 INFO:tasks.rados.rados.0.smithi045.stdout:519: writing smithi04526579-14 from 1499344 to 2199415 tid 2 2024-04-26T00:22:54.477 INFO:tasks.rados.rados.0.smithi045.stdout:519: writing smithi04526579-14 from 2686246 to 3362302 tid 3 2024-04-26T00:22:54.478 INFO:tasks.rados.rados.0.smithi045.stdout:520: snap_create 2024-04-26T00:22:54.478 INFO:tasks.rados.rados.0.smithi045.stdout:514: finishing rollback tid 1 to smithi04526579-41 2024-04-26T00:22:54.478 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 184 (ObjNum 209 snap 0 seq_num 3717656421) dirty dne 2024-04-26T00:22:54.478 INFO:tasks.rados.rados.0.smithi045.stdout:513: finishing write tid 3 to smithi04526579-5 2024-04-26T00:22:54.478 INFO:tasks.rados.rados.0.smithi045.stdout:513: finishing write tid 4 to smithi04526579-5 2024-04-26T00:22:54.478 INFO:tasks.rados.rados.0.smithi045.stdout:513: finishing write tid 5 to smithi04526579-5 2024-04-26T00:22:54.478 INFO:tasks.rados.rados.0.smithi045.stdout:513: oid 5 updating version 0 to 284 2024-04-26T00:22:54.478 INFO:tasks.rados.rados.0.smithi045.stdout:513: oid 5 updating version 284 to 285 2024-04-26T00:22:54.478 INFO:tasks.rados.rados.0.smithi045.stdout:513: oid 5 version 285 is already newer than 283 2024-04-26T00:22:54.478 INFO:tasks.rados.rados.0.smithi045.stdout:513: oid 5 updating version 285 to 286 2024-04-26T00:22:54.478 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 286 (ObjNum 218 snap 55 seq_num 218) dirty exists 2024-04-26T00:22:54.478 INFO:tasks.rados.rados.0.smithi045.stdout:513: left oid 5 (ObjNum 218 snap 55 seq_num 218) 2024-04-26T00:22:54.478 INFO:tasks.rados.rados.0.smithi045.stdout:516: finishing copy_from racing read to smithi04526579-16 2024-04-26T00:22:54.478 INFO:tasks.rados.rados.0.smithi045.stdout:515: finishing copy_from racing read to smithi04526579-32 2024-04-26T00:22:54.479 INFO:tasks.rados.rados.0.smithi045.stdout:516: finishing copy_from to smithi04526579-16 2024-04-26T00:22:54.479 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 115 (ObjNum 212 snap 53 seq_num 212) dirty exists 2024-04-26T00:22:54.479 INFO:tasks.rados.rados.0.smithi045.stdout:519: finishing write tid 1 to smithi04526579-14 2024-04-26T00:22:54.479 INFO:tasks.rados.rados.0.smithi045.stdout:515: finishing copy_from to smithi04526579-32 2024-04-26T00:22:54.479 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 148 (ObjNum 136 snap 30 seq_num 136) dirty exists 2024-04-26T00:22:54.479 INFO:tasks.rados.rados.0.smithi045.stdout:518: finishing write tid 1 to smithi04526579-19 2024-04-26T00:22:54.479 INFO:tasks.rados.rados.0.smithi045.stdout:518: finishing write tid 2 to smithi04526579-19 2024-04-26T00:22:54.479 INFO:tasks.rados.rados.0.smithi045.stdout:518: finishing write tid 3 to smithi04526579-19 2024-04-26T00:22:54.479 INFO:tasks.rados.rados.0.smithi045.stdout:518: finishing write tid 4 to smithi04526579-19 2024-04-26T00:22:54.479 INFO:tasks.rados.rados.0.smithi045.stdout:519: finishing write tid 2 to smithi04526579-14 2024-04-26T00:22:54.479 INFO:tasks.rados.rados.0.smithi045.stdout:518: finishing write tid 5 to smithi04526579-19 2024-04-26T00:22:54.479 INFO:tasks.rados.rados.0.smithi045.stdout:518: oid 19 updating version 0 to 163 2024-04-26T00:22:54.479 INFO:tasks.rados.rados.0.smithi045.stdout:518: oid 19 updating version 163 to 165 2024-04-26T00:22:54.479 INFO:tasks.rados.rados.0.smithi045.stdout:518: oid 19 version 165 is already newer than 164 2024-04-26T00:22:54.480 INFO:tasks.rados.rados.0.smithi045.stdout:518: oid 19 updating version 165 to 166 2024-04-26T00:22:54.480 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 166 (ObjNum 220 snap 55 seq_num 220) dirty exists 2024-04-26T00:22:54.480 INFO:tasks.rados.rados.0.smithi045.stdout:518: left oid 19 (ObjNum 220 snap 55 seq_num 220) 2024-04-26T00:22:54.483 INFO:tasks.rados.rados.0.smithi045.stdout:519: finishing write tid 3 to smithi04526579-14 2024-04-26T00:22:54.483 INFO:tasks.rados.rados.0.smithi045.stdout:519: finishing write tid 4 to smithi04526579-14 2024-04-26T00:22:54.484 INFO:tasks.rados.rados.0.smithi045.stdout:519: finishing write tid 5 to smithi04526579-14 2024-04-26T00:22:54.484 INFO:tasks.rados.rados.0.smithi045.stdout:519: oid 14 updating version 0 to 118 2024-04-26T00:22:54.484 INFO:tasks.rados.rados.0.smithi045.stdout:519: oid 14 version 118 is already newer than 117 2024-04-26T00:22:54.484 INFO:tasks.rados.rados.0.smithi045.stdout:519: oid 14 updating version 118 to 119 2024-04-26T00:22:54.484 INFO:tasks.rados.rados.0.smithi045.stdout:519: oid 14 version 119 is already newer than 116 2024-04-26T00:22:54.484 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 119 (ObjNum 221 snap 55 seq_num 221) dirty exists 2024-04-26T00:22:54.484 INFO:tasks.rados.rados.0.smithi045.stdout:519: left oid 14 (ObjNum 221 snap 55 seq_num 221) 2024-04-26T00:22:54.719 INFO:tasks.rados.rados.0.smithi045.stdout:513: done (6 left) 2024-04-26T00:22:54.719 INFO:tasks.rados.rados.0.smithi045.stdout:514: done (5 left) 2024-04-26T00:22:54.719 INFO:tasks.rados.rados.0.smithi045.stdout:515: done (4 left) 2024-04-26T00:22:54.719 INFO:tasks.rados.rados.0.smithi045.stdout:516: done (3 left) 2024-04-26T00:22:54.720 INFO:tasks.rados.rados.0.smithi045.stdout:518: done (2 left) 2024-04-26T00:22:54.720 INFO:tasks.rados.rados.0.smithi045.stdout:519: done (1 left) 2024-04-26T00:22:54.720 INFO:tasks.rados.rados.0.smithi045.stdout:520: done (0 left) 2024-04-26T00:22:54.720 INFO:tasks.rados.rados.0.smithi045.stdout:521: copy_from oid 29 from oid 40 current snap is 56 2024-04-26T00:22:54.720 INFO:tasks.rados.rados.0.smithi045.stdout:522: delete oid 8 current snap is 56 2024-04-26T00:22:54.730 INFO:tasks.rados.rados.0.smithi045.stdout:522: done (1 left) 2024-04-26T00:22:54.730 INFO:tasks.rados.rados.0.smithi045.stdout:523: snap_create 2024-04-26T00:22:54.745 INFO:tasks.rados.rados.0.smithi045.stdout:521: finishing copy_from racing read to smithi04526579-29 2024-04-26T00:22:54.750 INFO:tasks.rados.rados.0.smithi045.stdout:521: finishing copy_from to smithi04526579-29 2024-04-26T00:22:54.750 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 95 (ObjNum 131 snap 30 seq_num 131) dirty exists 2024-04-26T00:22:55.726 INFO:tasks.rados.rados.0.smithi045.stdout:521: done (1 left) 2024-04-26T00:22:55.726 INFO:tasks.rados.rados.0.smithi045.stdout:523: done (0 left) 2024-04-26T00:22:55.727 INFO:tasks.rados.rados.0.smithi045.stdout:524: snap_create 2024-04-26T00:22:56.728 INFO:tasks.rados.rados.0.smithi045.stdout:524: done (0 left) 2024-04-26T00:22:56.728 INFO:tasks.rados.rados.0.smithi045.stdout:525: read oid 26 snap -1 2024-04-26T00:22:56.728 INFO:tasks.rados.rados.0.smithi045.stdout:525: expect (ObjNum 209 snap 53 seq_num 209) 2024-04-26T00:22:56.729 INFO:tasks.rados.rados.0.smithi045.stdout:526: snap_remove snap 53 2024-04-26T00:22:57.743 INFO:tasks.rados.rados.0.smithi045.stdout:526: done (1 left) 2024-04-26T00:22:57.743 INFO:tasks.rados.rados.0.smithi045.stdout:527: snap_create 2024-04-26T00:22:57.743 INFO:tasks.rados.rados.0.smithi045.stdout:525: expect (ObjNum 209 snap 53 seq_num 209) 2024-04-26T00:22:58.745 INFO:tasks.rados.rados.0.smithi045.stdout:525: done (1 left) 2024-04-26T00:22:58.745 INFO:tasks.rados.rados.0.smithi045.stdout:527: done (0 left) 2024-04-26T00:22:58.745 INFO:tasks.rados.rados.0.smithi045.stdout:528: copy_from oid 6 from oid 39 current snap is 59 2024-04-26T00:22:58.745 INFO:tasks.rados.rados.0.smithi045.stdout:529: snap_remove snap 54 2024-04-26T00:22:59.722 INFO:tasks.rados.rados.0.smithi045.stdout:529: done (1 left) 2024-04-26T00:22:59.722 INFO:tasks.rados.rados.0.smithi045.stdout:530: delete oid 25 current snap is 59 2024-04-26T00:22:59.722 INFO:tasks.rados.rados.0.smithi045.stdout:528: finishing copy_from racing read to smithi04526579-6 2024-04-26T00:22:59.722 INFO:tasks.rados.rados.0.smithi045.stdout:528: finishing copy_from to smithi04526579-6 2024-04-26T00:22:59.722 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 102 (ObjNum 155 snap 33 seq_num 155) dirty exists 2024-04-26T00:22:59.739 INFO:tasks.rados.rados.0.smithi045.stdout:528: done (1 left) 2024-04-26T00:22:59.739 INFO:tasks.rados.rados.0.smithi045.stdout:530: done (0 left) 2024-04-26T00:22:59.739 INFO:tasks.rados.rados.0.smithi045.stdout:531: read oid 21 snap -1 2024-04-26T00:22:59.739 INFO:tasks.rados.rados.0.smithi045.stdout:531: expect deleted 2024-04-26T00:22:59.740 INFO:tasks.rados.rados.0.smithi045.stdout:532: write oid 41 current snap is 59 2024-04-26T00:22:59.740 INFO:tasks.rados.rados.0.smithi045.stdout:532: seq_num 224 ranges {564768=630013,1674790=447398,2768515=626225} 2024-04-26T00:22:59.759 INFO:tasks.rados.rados.0.smithi045.stdout:532: writing smithi04526579-41 from 564768 to 1194781 tid 1 2024-04-26T00:22:59.774 INFO:tasks.rados.rados.0.smithi045.stdout:532: writing smithi04526579-41 from 1674790 to 2122188 tid 2 2024-04-26T00:22:59.794 INFO:tasks.rados.rados.0.smithi045.stdout:532: writing smithi04526579-41 from 2768515 to 3394740 tid 3 2024-04-26T00:22:59.794 INFO:tasks.rados.rados.0.smithi045.stdout:533: delete oid 7 current snap is 59 2024-04-26T00:22:59.794 INFO:tasks.rados.rados.0.smithi045.stdout:532: finishing write tid 1 to smithi04526579-41 2024-04-26T00:22:59.794 INFO:tasks.rados.rados.0.smithi045.stdout:532: finishing write tid 2 to smithi04526579-41 2024-04-26T00:22:59.801 INFO:tasks.rados.rados.0.smithi045.stdout:532: finishing write tid 3 to smithi04526579-41 2024-04-26T00:22:59.802 INFO:tasks.rados.rados.0.smithi045.stdout:532: finishing write tid 4 to smithi04526579-41 2024-04-26T00:22:59.802 INFO:tasks.rados.rados.0.smithi045.stdout:532: finishing write tid 5 to smithi04526579-41 2024-04-26T00:22:59.802 INFO:tasks.rados.rados.0.smithi045.stdout:532: oid 41 updating version 0 to 224 2024-04-26T00:22:59.802 INFO:tasks.rados.rados.0.smithi045.stdout:532: oid 41 version 224 is already newer than 223 2024-04-26T00:22:59.802 INFO:tasks.rados.rados.0.smithi045.stdout:532: oid 41 updating version 224 to 225 2024-04-26T00:22:59.803 INFO:tasks.rados.rados.0.smithi045.stdout:532: oid 41 version 225 is already newer than 222 2024-04-26T00:22:59.803 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 225 (ObjNum 224 snap 59 seq_num 224) dirty exists 2024-04-26T00:22:59.803 INFO:tasks.rados.rados.0.smithi045.stdout:532: left oid 41 (ObjNum 224 snap 59 seq_num 224) 2024-04-26T00:22:59.803 INFO:tasks.rados.rados.0.smithi045.stdout:531: done (2 left) 2024-04-26T00:22:59.803 INFO:tasks.rados.rados.0.smithi045.stdout:532: done (1 left) 2024-04-26T00:22:59.803 INFO:tasks.rados.rados.0.smithi045.stdout:533: done (0 left) 2024-04-26T00:22:59.803 INFO:tasks.rados.rados.0.smithi045.stdout:534: write (excl) oid 31 current snap is 59 2024-04-26T00:22:59.804 INFO:tasks.rados.rados.0.smithi045.stdout:534: seq_num 226 ranges {598382=780511,2075401=451067,3107008=648769} 2024-04-26T00:22:59.821 INFO:tasks.rados.rados.0.smithi045.stdout:534: writing smithi04526579-31 from 598382 to 1378893 tid 1 2024-04-26T00:22:59.835 INFO:tasks.rados.rados.0.smithi045.stdout:534: writing smithi04526579-31 from 2075401 to 2526468 tid 2 2024-04-26T00:22:59.849 INFO:tasks.rados.rados.0.smithi045.stdout:534: writing smithi04526579-31 from 3107008 to 3755777 tid 3 2024-04-26T00:22:59.849 INFO:tasks.rados.rados.0.smithi045.stdout:535: write (excl) oid 8 current snap is 59 2024-04-26T00:22:59.849 INFO:tasks.rados.rados.0.smithi045.stdout:535: seq_num 227 ranges {615189=455760,1733883=711078,3134431=460041} 2024-04-26T00:22:59.858 INFO:tasks.rados.rados.0.smithi045.stdout:535: writing smithi04526579-8 from 615189 to 1070949 tid 1 2024-04-26T00:22:59.869 INFO:tasks.rados.rados.0.smithi045.stdout:535: writing smithi04526579-8 from 1733883 to 2444961 tid 2 2024-04-26T00:22:59.878 INFO:tasks.rados.rados.0.smithi045.stdout:535: writing smithi04526579-8 from 3134431 to 3594472 tid 3 2024-04-26T00:22:59.878 INFO:tasks.rados.rados.0.smithi045.stdout:536: delete oid 22 current snap is 59 2024-04-26T00:22:59.878 INFO:tasks.rados.rados.0.smithi045.stdout:534: finishing write tid 1 to smithi04526579-31 2024-04-26T00:22:59.878 INFO:tasks.rados.rados.0.smithi045.stdout:534: finishing write tid 2 to smithi04526579-31 2024-04-26T00:22:59.878 INFO:tasks.rados.rados.0.smithi045.stdout:534: finishing write tid 3 to smithi04526579-31 2024-04-26T00:22:59.878 INFO:tasks.rados.rados.0.smithi045.stdout:534: finishing write tid 4 to smithi04526579-31 2024-04-26T00:22:59.878 INFO:tasks.rados.rados.0.smithi045.stdout:534: finishing write tid 5 to smithi04526579-31 2024-04-26T00:22:59.879 INFO:tasks.rados.rados.0.smithi045.stdout:534: oid 31 updating version 0 to 238 2024-04-26T00:22:59.879 INFO:tasks.rados.rados.0.smithi045.stdout:534: oid 31 version 238 is already newer than 237 2024-04-26T00:22:59.879 INFO:tasks.rados.rados.0.smithi045.stdout:534: oid 31 updating version 238 to 239 2024-04-26T00:22:59.879 INFO:tasks.rados.rados.0.smithi045.stdout:534: oid 31 updating version 239 to 240 2024-04-26T00:22:59.879 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 240 (ObjNum 226 snap 59 seq_num 226) dirty exists 2024-04-26T00:22:59.879 INFO:tasks.rados.rados.0.smithi045.stdout:534: left oid 31 (ObjNum 226 snap 59 seq_num 226) 2024-04-26T00:22:59.879 INFO:tasks.rados.rados.0.smithi045.stdout:535: finishing write tid 1 to smithi04526579-8 2024-04-26T00:22:59.879 INFO:tasks.rados.rados.0.smithi045.stdout:535: finishing write tid 2 to smithi04526579-8 2024-04-26T00:22:59.885 INFO:tasks.rados.rados.0.smithi045.stdout:535: finishing write tid 3 to smithi04526579-8 2024-04-26T00:22:59.886 INFO:tasks.rados.rados.0.smithi045.stdout:534: done (2 left) 2024-04-26T00:22:59.886 INFO:tasks.rados.rados.0.smithi045.stdout:536: done (1 left) 2024-04-26T00:22:59.886 INFO:tasks.rados.rados.0.smithi045.stdout:537: write (excl) oid 37 current snap is 59 2024-04-26T00:22:59.886 INFO:tasks.rados.rados.0.smithi045.stdout:537: seq_num 229 ranges {648803=722611,1967200=714747,3421983=426820} 2024-04-26T00:22:59.896 INFO:tasks.rados.rados.0.smithi045.stdout:537: writing smithi04526579-37 from 648803 to 1371414 tid 1 2024-04-26T00:22:59.905 INFO:tasks.rados.rados.0.smithi045.stdout:537: writing smithi04526579-37 from 1967200 to 2681947 tid 2 2024-04-26T00:22:59.913 INFO:tasks.rados.rados.0.smithi045.stdout:537: writing smithi04526579-37 from 3421983 to 3848803 tid 3 2024-04-26T00:22:59.913 INFO:tasks.rados.rados.0.smithi045.stdout:538: write oid 24 current snap is 59 2024-04-26T00:22:59.913 INFO:tasks.rados.rados.0.smithi045.stdout:538: seq_num 230 ranges {665610=797860,1909329=458405,2816700=410210} 2024-04-26T00:22:59.923 INFO:tasks.rados.rados.0.smithi045.stdout:538: writing smithi04526579-24 from 665610 to 1463470 tid 1 2024-04-26T00:22:59.929 INFO:tasks.rados.rados.0.smithi045.stdout:538: writing smithi04526579-24 from 1909329 to 2367734 tid 2 2024-04-26T00:22:59.935 INFO:tasks.rados.rados.0.smithi045.stdout:538: writing smithi04526579-24 from 2816700 to 3226910 tid 3 2024-04-26T00:22:59.935 INFO:tasks.rados.rados.0.smithi045.stdout:539: write oid 15 current snap is 59 2024-04-26T00:22:59.935 INFO:tasks.rados.rados.0.smithi045.stdout:539: seq_num 231 ranges {682417=473109,1567811=602063,2844123=621482} 2024-04-26T00:22:59.942 INFO:tasks.rados.rados.0.smithi045.stdout:539: writing smithi04526579-15 from 682417 to 1155526 tid 1 2024-04-26T00:22:59.948 INFO:tasks.rados.rados.0.smithi045.stdout:539: writing smithi04526579-15 from 1567811 to 2169874 tid 2 2024-04-26T00:22:59.956 INFO:tasks.rados.rados.0.smithi045.stdout:539: writing smithi04526579-15 from 2844123 to 3465605 tid 3 2024-04-26T00:22:59.956 INFO:tasks.rados.rados.0.smithi045.stdout:540: snap_remove snap 46 2024-04-26T00:23:01.369 INFO:tasks.rados.rados.0.smithi045.stdout:540: done (4 left) 2024-04-26T00:23:01.370 INFO:tasks.rados.rados.0.smithi045.stdout:541: write oid 45 current snap is 59 2024-04-26T00:23:01.370 INFO:tasks.rados.rados.0.smithi045.stdout:541: seq_num 232 ranges {699224=548358,2026293=462074,3271546=432754} 2024-04-26T00:23:01.376 INFO:tasks.rados.rados.0.smithi045.stdout:541: writing smithi04526579-45 from 699224 to 1247582 tid 1 2024-04-26T00:23:01.384 INFO:tasks.rados.rados.0.smithi045.stdout:541: writing smithi04526579-45 from 2026293 to 2488367 tid 2 2024-04-26T00:23:01.391 INFO:tasks.rados.rados.0.smithi045.stdout:541: writing smithi04526579-45 from 3271546 to 3704300 tid 3 2024-04-26T00:23:01.391 INFO:tasks.rados.rados.0.smithi045.stdout:542: delete oid 16 current snap is 59 2024-04-26T00:23:01.391 INFO:tasks.rados.rados.0.smithi045.stdout:535: finishing write tid 4 to smithi04526579-8 2024-04-26T00:23:01.391 INFO:tasks.rados.rados.0.smithi045.stdout:535: finishing write tid 5 to smithi04526579-8 2024-04-26T00:23:01.391 INFO:tasks.rados.rados.0.smithi045.stdout:535: oid 8 updating version 0 to 124 2024-04-26T00:23:01.391 INFO:tasks.rados.rados.0.smithi045.stdout:535: oid 8 version 124 is already newer than 123 2024-04-26T00:23:01.391 INFO:tasks.rados.rados.0.smithi045.stdout:535: oid 8 version 124 is already newer than 122 2024-04-26T00:23:01.391 INFO:tasks.rados.rados.0.smithi045.stdout:535: oid 8 updating version 124 to 125 2024-04-26T00:23:01.391 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 125 (ObjNum 227 snap 59 seq_num 227) dirty exists 2024-04-26T00:23:01.391 INFO:tasks.rados.rados.0.smithi045.stdout:535: left oid 8 (ObjNum 227 snap 59 seq_num 227) 2024-04-26T00:23:01.391 INFO:tasks.rados.rados.0.smithi045.stdout:537: finishing write tid 1 to smithi04526579-37 2024-04-26T00:23:01.392 INFO:tasks.rados.rados.0.smithi045.stdout:537: finishing write tid 2 to smithi04526579-37 2024-04-26T00:23:01.392 INFO:tasks.rados.rados.0.smithi045.stdout:537: finishing write tid 3 to smithi04526579-37 2024-04-26T00:23:01.392 INFO:tasks.rados.rados.0.smithi045.stdout:537: finishing write tid 4 to smithi04526579-37 2024-04-26T00:23:01.392 INFO:tasks.rados.rados.0.smithi045.stdout:537: finishing write tid 5 to smithi04526579-37 2024-04-26T00:23:01.392 INFO:tasks.rados.rados.0.smithi045.stdout:537: oid 37 updating version 0 to 296 2024-04-26T00:23:01.392 INFO:tasks.rados.rados.0.smithi045.stdout:537: oid 37 updating version 296 to 298 2024-04-26T00:23:01.392 INFO:tasks.rados.rados.0.smithi045.stdout:537: oid 37 updating version 298 to 299 2024-04-26T00:23:01.392 INFO:tasks.rados.rados.0.smithi045.stdout:537: oid 37 version 299 is already newer than 297 2024-04-26T00:23:01.392 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 299 (ObjNum 229 snap 59 seq_num 229) dirty exists 2024-04-26T00:23:01.392 INFO:tasks.rados.rados.0.smithi045.stdout:537: left oid 37 (ObjNum 229 snap 59 seq_num 229) 2024-04-26T00:23:01.392 INFO:tasks.rados.rados.0.smithi045.stdout:538: finishing write tid 1 to smithi04526579-24 2024-04-26T00:23:01.392 INFO:tasks.rados.rados.0.smithi045.stdout:538: finishing write tid 2 to smithi04526579-24 2024-04-26T00:23:01.392 INFO:tasks.rados.rados.0.smithi045.stdout:538: finishing write tid 3 to smithi04526579-24 2024-04-26T00:23:01.392 INFO:tasks.rados.rados.0.smithi045.stdout:538: finishing write tid 4 to smithi04526579-24 2024-04-26T00:23:01.393 INFO:tasks.rados.rados.0.smithi045.stdout:538: finishing write tid 5 to smithi04526579-24 2024-04-26T00:23:01.393 INFO:tasks.rados.rados.0.smithi045.stdout:538: oid 24 updating version 0 to 301 2024-04-26T00:23:01.393 INFO:tasks.rados.rados.0.smithi045.stdout:538: oid 24 updating version 301 to 303 2024-04-26T00:23:01.393 INFO:tasks.rados.rados.0.smithi045.stdout:538: oid 24 updating version 303 to 304 2024-04-26T00:23:01.393 INFO:tasks.rados.rados.0.smithi045.stdout:538: oid 24 version 304 is already newer than 302 2024-04-26T00:23:01.393 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 304 (ObjNum 230 snap 59 seq_num 230) dirty exists 2024-04-26T00:23:01.393 INFO:tasks.rados.rados.0.smithi045.stdout:538: left oid 24 (ObjNum 230 snap 59 seq_num 230) 2024-04-26T00:23:01.393 INFO:tasks.rados.rados.0.smithi045.stdout:539: finishing write tid 1 to smithi04526579-15 2024-04-26T00:23:01.393 INFO:tasks.rados.rados.0.smithi045.stdout:539: finishing write tid 2 to smithi04526579-15 2024-04-26T00:23:01.393 INFO:tasks.rados.rados.0.smithi045.stdout:539: finishing write tid 3 to smithi04526579-15 2024-04-26T00:23:01.393 INFO:tasks.rados.rados.0.smithi045.stdout:539: finishing write tid 4 to smithi04526579-15 2024-04-26T00:23:01.393 INFO:tasks.rados.rados.0.smithi045.stdout:539: finishing write tid 5 to smithi04526579-15 2024-04-26T00:23:01.393 INFO:tasks.rados.rados.0.smithi045.stdout:539: oid 15 updating version 0 to 226 2024-04-26T00:23:01.394 INFO:tasks.rados.rados.0.smithi045.stdout:539: oid 15 updating version 226 to 227 2024-04-26T00:23:01.394 INFO:tasks.rados.rados.0.smithi045.stdout:539: oid 15 updating version 227 to 229 2024-04-26T00:23:01.394 INFO:tasks.rados.rados.0.smithi045.stdout:539: oid 15 version 229 is already newer than 228 2024-04-26T00:23:01.394 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 229 (ObjNum 231 snap 59 seq_num 231) dirty exists 2024-04-26T00:23:01.394 INFO:tasks.rados.rados.0.smithi045.stdout:539: left oid 15 (ObjNum 231 snap 59 seq_num 231) 2024-04-26T00:23:01.394 INFO:tasks.rados.rados.0.smithi045.stdout:541: finishing write tid 1 to smithi04526579-45 2024-04-26T00:23:01.394 INFO:tasks.rados.rados.0.smithi045.stdout:541: finishing write tid 2 to smithi04526579-45 2024-04-26T00:23:01.397 INFO:tasks.rados.rados.0.smithi045.stdout:541: finishing write tid 3 to smithi04526579-45 2024-04-26T00:23:01.399 INFO:tasks.rados.rados.0.smithi045.stdout:541: finishing write tid 4 to smithi04526579-45 2024-04-26T00:23:01.399 INFO:tasks.rados.rados.0.smithi045.stdout:541: finishing write tid 5 to smithi04526579-45 2024-04-26T00:23:01.399 INFO:tasks.rados.rados.0.smithi045.stdout:541: oid 45 updating version 0 to 111 2024-04-26T00:23:01.399 INFO:tasks.rados.rados.0.smithi045.stdout:541: oid 45 updating version 111 to 113 2024-04-26T00:23:01.400 INFO:tasks.rados.rados.0.smithi045.stdout:541: oid 45 updating version 113 to 114 2024-04-26T00:23:01.400 INFO:tasks.rados.rados.0.smithi045.stdout:541: oid 45 version 114 is already newer than 112 2024-04-26T00:23:01.400 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 114 (ObjNum 232 snap 59 seq_num 232) dirty exists 2024-04-26T00:23:01.400 INFO:tasks.rados.rados.0.smithi045.stdout:541: left oid 45 (ObjNum 232 snap 59 seq_num 232) 2024-04-26T00:23:01.401 INFO:tasks.rados.rados.0.smithi045.stdout:535: done (5 left) 2024-04-26T00:23:01.401 INFO:tasks.rados.rados.0.smithi045.stdout:537: done (4 left) 2024-04-26T00:23:01.401 INFO:tasks.rados.rados.0.smithi045.stdout:538: done (3 left) 2024-04-26T00:23:01.401 INFO:tasks.rados.rados.0.smithi045.stdout:539: done (2 left) 2024-04-26T00:23:01.401 INFO:tasks.rados.rados.0.smithi045.stdout:541: done (1 left) 2024-04-26T00:23:01.401 INFO:tasks.rados.rados.0.smithi045.stdout:542: done (0 left) 2024-04-26T00:23:01.402 INFO:tasks.rados.rados.0.smithi045.stdout:543: read oid 16 snap -1 2024-04-26T00:23:01.402 INFO:tasks.rados.rados.0.smithi045.stdout:543: expect deleted 2024-04-26T00:23:01.402 INFO:tasks.rados.rados.0.smithi045.stdout:544: copy_from oid 33 from oid 23 current snap is 59 2024-04-26T00:23:01.402 INFO:tasks.rados.rados.0.smithi045.stdout:545: read oid 41 snap -1 2024-04-26T00:23:01.402 INFO:tasks.rados.rados.0.smithi045.stdout:545: expect (ObjNum 224 snap 59 seq_num 224) 2024-04-26T00:23:01.402 INFO:tasks.rados.rados.0.smithi045.stdout:546: rollback oid 42 current snap is 59 2024-04-26T00:23:01.402 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 42 to 32 2024-04-26T00:23:01.402 INFO:tasks.rados.rados.0.smithi045.stdout:547: delete oid 38 current snap is 59 2024-04-26T00:23:01.403 INFO:tasks.rados.rados.0.smithi045.stdout:544: finishing copy_from to smithi04526579-33 2024-04-26T00:23:01.403 INFO:tasks.rados.rados.0.smithi045.stdout:544: got expected ENOENT (src dne) 2024-04-26T00:23:01.403 INFO:tasks.rados.rados.0.smithi045.stdout:544: finishing copy_from racing read to smithi04526579-33 2024-04-26T00:23:01.404 INFO:tasks.rados.rados.0.smithi045.stdout:543: done (4 left) 2024-04-26T00:23:01.404 INFO:tasks.rados.rados.0.smithi045.stdout:544: done (3 left) 2024-04-26T00:23:01.404 INFO:tasks.rados.rados.0.smithi045.stdout:547: done (2 left) 2024-04-26T00:23:01.404 INFO:tasks.rados.rados.0.smithi045.stdout:548: copy_from oid 22 from oid 38 current snap is 59 2024-04-26T00:23:01.405 INFO:tasks.rados.rados.0.smithi045.stdout:549: copy_from oid 24 from oid 3 current snap is 59 2024-04-26T00:23:01.405 INFO:tasks.rados.rados.0.smithi045.stdout:550: snap_create 2024-04-26T00:23:01.411 INFO:tasks.rados.rados.0.smithi045.stdout:546: finishing rollback tid 0 to smithi04526579-42 2024-04-26T00:23:01.412 INFO:tasks.rados.rados.0.smithi045.stdout:548: finishing copy_from to smithi04526579-22 2024-04-26T00:23:01.413 INFO:tasks.rados.rados.0.smithi045.stdout:548: got expected ENOENT (src dne) 2024-04-26T00:23:01.413 INFO:tasks.rados.rados.0.smithi045.stdout:546: finishing rollback tid 1 to smithi04526579-42 2024-04-26T00:23:01.413 INFO:tasks.rados.rados.0.smithi045.stdout:546: finishing rollback tid 2 to smithi04526579-42 2024-04-26T00:23:01.413 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 188 (ObjNum 114 snap 28 seq_num 114) dirty exists 2024-04-26T00:23:01.413 INFO:tasks.rados.rados.0.smithi045.stdout:548: finishing copy_from racing read to smithi04526579-22 2024-04-26T00:23:01.418 INFO:tasks.rados.rados.0.smithi045.stdout:545: expect (ObjNum 224 snap 59 seq_num 224) 2024-04-26T00:23:01.583 INFO:tasks.rados.rados.0.smithi045.stdout:549: finishing copy_from racing read to smithi04526579-24 2024-04-26T00:23:01.583 INFO:tasks.rados.rados.0.smithi045.stdout:549: finishing copy_from to smithi04526579-24 2024-04-26T00:23:01.583 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 152 (ObjNum 201 snap 53 seq_num 201) dirty exists 2024-04-26T00:23:01.780 INFO:tasks.rados.rados.0.smithi045.stdout:545: done (4 left) 2024-04-26T00:23:01.781 INFO:tasks.rados.rados.0.smithi045.stdout:546: done (3 left) 2024-04-26T00:23:01.781 INFO:tasks.rados.rados.0.smithi045.stdout:548: done (2 left) 2024-04-26T00:23:01.781 INFO:tasks.rados.rados.0.smithi045.stdout:549: done (1 left) 2024-04-26T00:23:01.781 INFO:tasks.rados.rados.0.smithi045.stdout:550: done (0 left) 2024-04-26T00:23:01.781 INFO:tasks.rados.rados.0.smithi045.stdout:551: write (excl) oid 37 current snap is 60 2024-04-26T00:23:01.781 INFO:tasks.rados.rados.0.smithi045.stdout:551: seq_num 235 ranges {749645=774105,2085386=609401,3237462=712183} 2024-04-26T00:23:01.790 INFO:tasks.rados.rados.0.smithi045.stdout:551: writing smithi04526579-37 from 749645 to 1523750 tid 1 2024-04-26T00:23:01.798 INFO:tasks.rados.rados.0.smithi045.stdout:551: writing smithi04526579-37 from 2085386 to 2694787 tid 2 2024-04-26T00:23:01.807 INFO:tasks.rados.rados.0.smithi045.stdout:551: writing smithi04526579-37 from 3237462 to 3949645 tid 3 2024-04-26T00:23:01.807 INFO:tasks.rados.rados.0.smithi045.stdout:552: write oid 2 current snap is 60 2024-04-26T00:23:01.807 INFO:tasks.rados.rados.0.smithi045.stdout:552: seq_num 236 ranges {766452=565707,1860221=469412,2981238=594195} 2024-04-26T00:23:01.815 INFO:tasks.rados.rados.0.smithi045.stdout:552: writing smithi04526579-2 from 766452 to 1332159 tid 1 2024-04-26T00:23:01.821 INFO:tasks.rados.rados.0.smithi045.stdout:552: writing smithi04526579-2 from 1860221 to 2329633 tid 2 2024-04-26T00:23:01.830 INFO:tasks.rados.rados.0.smithi045.stdout:552: writing smithi04526579-2 from 2981238 to 3575433 tid 3 2024-04-26T00:23:01.830 INFO:tasks.rados.rados.0.smithi045.stdout:553: write (excl) oid 14 current snap is 60 2024-04-26T00:23:01.830 INFO:tasks.rados.rados.0.smithi045.stdout:553: seq_num 237 ranges {783259=640956,1918703=613070,3008661=405467,3821939=161320} 2024-04-26T00:23:01.841 INFO:tasks.rados.rados.0.smithi045.stdout:553: writing smithi04526579-14 from 783259 to 1424215 tid 1 2024-04-26T00:23:01.847 INFO:tasks.rados.rados.0.smithi045.stdout:553: writing smithi04526579-14 from 1918703 to 2531773 tid 2 2024-04-26T00:23:01.852 INFO:tasks.rados.rados.0.smithi045.stdout:553: writing smithi04526579-14 from 3008661 to 3414128 tid 3 2024-04-26T00:23:01.856 INFO:tasks.rados.rados.0.smithi045.stdout:553: writing smithi04526579-14 from 3821939 to 3983259 tid 4 2024-04-26T00:23:01.856 INFO:tasks.rados.rados.0.smithi045.stdout:554: read oid 17 snap -1 2024-04-26T00:23:01.856 INFO:tasks.rados.rados.0.smithi045.stdout:554: expect deleted 2024-04-26T00:23:01.856 INFO:tasks.rados.rados.0.smithi045.stdout:555: write oid 1 current snap is 60 2024-04-26T00:23:01.856 INFO:tasks.rados.rados.0.smithi045.stdout:555: seq_num 238 ranges {400066=716205,1860832=139234} 2024-04-26T00:23:01.863 INFO:tasks.rados.rados.0.smithi045.stdout:555: writing smithi04526579-1 from 400066 to 1116271 tid 1 2024-04-26T00:23:01.869 INFO:tasks.rados.rados.0.smithi045.stdout:555: writing smithi04526579-1 from 1860832 to 2000066 tid 2 2024-04-26T00:23:01.869 INFO:tasks.rados.rados.0.smithi045.stdout:556: read oid 19 snap 51 2024-04-26T00:23:01.869 INFO:tasks.rados.rados.0.smithi045.stdout:556: expect (ObjNum 120 snap 28 seq_num 120) 2024-04-26T00:23:01.869 INFO:tasks.rados.rados.0.smithi045.stdout:557: read oid 7 snap 45 2024-04-26T00:23:01.869 INFO:tasks.rados.rados.0.smithi045.stdout:557: expect (ObjNum 164 snap 38 seq_num 164) 2024-04-26T00:23:01.869 INFO:tasks.rados.rados.0.smithi045.stdout:558: rollback oid 44 current snap is 60 2024-04-26T00:23:01.869 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 44 to 32 2024-04-26T00:23:01.869 INFO:tasks.rados.rados.0.smithi045.stdout:559: snap_remove snap 52 2024-04-26T00:23:02.784 INFO:tasks.rados.rados.0.smithi045.stdout:559: done (8 left) 2024-04-26T00:23:02.784 INFO:tasks.rados.rados.0.smithi045.stdout:560: delete oid 41 current snap is 60 2024-04-26T00:23:02.784 INFO:tasks.rados.rados.0.smithi045.stdout:551: finishing write tid 1 to smithi04526579-37 2024-04-26T00:23:02.784 INFO:tasks.rados.rados.0.smithi045.stdout:551: finishing write tid 2 to smithi04526579-37 2024-04-26T00:23:02.784 INFO:tasks.rados.rados.0.smithi045.stdout:551: finishing write tid 3 to smithi04526579-37 2024-04-26T00:23:02.784 INFO:tasks.rados.rados.0.smithi045.stdout:551: finishing write tid 4 to smithi04526579-37 2024-04-26T00:23:02.784 INFO:tasks.rados.rados.0.smithi045.stdout:551: finishing write tid 5 to smithi04526579-37 2024-04-26T00:23:02.784 INFO:tasks.rados.rados.0.smithi045.stdout:551: oid 37 updating version 0 to 317 2024-04-26T00:23:02.785 INFO:tasks.rados.rados.0.smithi045.stdout:551: oid 37 updating version 317 to 320 2024-04-26T00:23:02.785 INFO:tasks.rados.rados.0.smithi045.stdout:551: oid 37 version 320 is already newer than 319 2024-04-26T00:23:02.785 INFO:tasks.rados.rados.0.smithi045.stdout:551: oid 37 version 320 is already newer than 318 2024-04-26T00:23:02.785 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 320 (ObjNum 235 snap 60 seq_num 235) dirty exists 2024-04-26T00:23:02.785 INFO:tasks.rados.rados.0.smithi045.stdout:551: left oid 37 (ObjNum 235 snap 60 seq_num 235) 2024-04-26T00:23:02.785 INFO:tasks.rados.rados.0.smithi045.stdout:552: finishing write tid 1 to smithi04526579-2 2024-04-26T00:23:02.785 INFO:tasks.rados.rados.0.smithi045.stdout:552: finishing write tid 2 to smithi04526579-2 2024-04-26T00:23:02.785 INFO:tasks.rados.rados.0.smithi045.stdout:552: finishing write tid 3 to smithi04526579-2 2024-04-26T00:23:02.785 INFO:tasks.rados.rados.0.smithi045.stdout:552: finishing write tid 4 to smithi04526579-2 2024-04-26T00:23:02.785 INFO:tasks.rados.rados.0.smithi045.stdout:552: finishing write tid 5 to smithi04526579-2 2024-04-26T00:23:02.785 INFO:tasks.rados.rados.0.smithi045.stdout:552: oid 2 updating version 0 to 253 2024-04-26T00:23:02.785 INFO:tasks.rados.rados.0.smithi045.stdout:552: oid 2 version 253 is already newer than 252 2024-04-26T00:23:02.785 INFO:tasks.rados.rados.0.smithi045.stdout:552: oid 2 version 253 is already newer than 250 2024-04-26T00:23:02.785 INFO:tasks.rados.rados.0.smithi045.stdout:552: oid 2 version 253 is already newer than 251 2024-04-26T00:23:02.786 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 253 (ObjNum 236 snap 60 seq_num 236) dirty exists 2024-04-26T00:23:02.786 INFO:tasks.rados.rados.0.smithi045.stdout:552: left oid 2 (ObjNum 236 snap 60 seq_num 236) 2024-04-26T00:23:02.786 INFO:tasks.rados.rados.0.smithi045.stdout:553: finishing write tid 1 to smithi04526579-14 2024-04-26T00:23:02.786 INFO:tasks.rados.rados.0.smithi045.stdout:553: finishing write tid 2 to smithi04526579-14 2024-04-26T00:23:02.786 INFO:tasks.rados.rados.0.smithi045.stdout:553: finishing write tid 3 to smithi04526579-14 2024-04-26T00:23:02.786 INFO:tasks.rados.rados.0.smithi045.stdout:553: finishing write tid 4 to smithi04526579-14 2024-04-26T00:23:02.786 INFO:tasks.rados.rados.0.smithi045.stdout:555: finishing write tid 1 to smithi04526579-1 2024-04-26T00:23:02.786 INFO:tasks.rados.rados.0.smithi045.stdout:555: finishing write tid 2 to smithi04526579-1 2024-04-26T00:23:02.786 INFO:tasks.rados.rados.0.smithi045.stdout:555: finishing write tid 3 to smithi04526579-1 2024-04-26T00:23:02.786 INFO:tasks.rados.rados.0.smithi045.stdout:555: finishing write tid 4 to smithi04526579-1 2024-04-26T00:23:02.786 INFO:tasks.rados.rados.0.smithi045.stdout:555: oid 1 updating version 0 to 243 2024-04-26T00:23:02.786 INFO:tasks.rados.rados.0.smithi045.stdout:555: oid 1 version 243 is already newer than 241 2024-04-26T00:23:02.786 INFO:tasks.rados.rados.0.smithi045.stdout:555: oid 1 version 243 is already newer than 242 2024-04-26T00:23:02.787 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 243 (ObjNum 238 snap 60 seq_num 238) dirty exists 2024-04-26T00:23:02.787 INFO:tasks.rados.rados.0.smithi045.stdout:555: left oid 1 (ObjNum 238 snap 60 seq_num 238) 2024-04-26T00:23:02.787 INFO:tasks.rados.rados.0.smithi045.stdout:558: finishing rollback tid 1 to smithi04526579-44 2024-04-26T00:23:02.787 INFO:tasks.rados.rados.0.smithi045.stdout:553: finishing write tid 5 to smithi04526579-14 2024-04-26T00:23:02.787 INFO:tasks.rados.rados.0.smithi045.stdout:558: finishing rollback tid 2 to smithi04526579-44 2024-04-26T00:23:02.787 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 216 (ObjNum 119 snap 28 seq_num 119) dirty exists 2024-04-26T00:23:02.787 INFO:tasks.rados.rados.0.smithi045.stdout:553: finishing write tid 6 to smithi04526579-14 2024-04-26T00:23:02.787 INFO:tasks.rados.rados.0.smithi045.stdout:553: oid 14 updating version 0 to 124 2024-04-26T00:23:02.787 INFO:tasks.rados.rados.0.smithi045.stdout:553: oid 14 updating version 124 to 125 2024-04-26T00:23:02.787 INFO:tasks.rados.rados.0.smithi045.stdout:553: oid 14 version 125 is already newer than 123 2024-04-26T00:23:02.787 INFO:tasks.rados.rados.0.smithi045.stdout:553: oid 14 version 125 is already newer than 121 2024-04-26T00:23:02.787 INFO:tasks.rados.rados.0.smithi045.stdout:553: oid 14 version 125 is already newer than 122 2024-04-26T00:23:02.787 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 125 (ObjNum 237 snap 60 seq_num 237) dirty exists 2024-04-26T00:23:02.787 INFO:tasks.rados.rados.0.smithi045.stdout:553: left oid 14 (ObjNum 237 snap 60 seq_num 237) 2024-04-26T00:23:02.788 INFO:tasks.rados.rados.0.smithi045.stdout:556: expect (ObjNum 120 snap 28 seq_num 120) 2024-04-26T00:23:02.883 INFO:tasks.rados.rados.0.smithi045.stdout:557: expect (ObjNum 164 snap 38 seq_num 164) 2024-04-26T00:23:03.011 INFO:tasks.rados.rados.0.smithi045.stdout:551: done (8 left) 2024-04-26T00:23:03.011 INFO:tasks.rados.rados.0.smithi045.stdout:552: done (7 left) 2024-04-26T00:23:03.011 INFO:tasks.rados.rados.0.smithi045.stdout:553: done (6 left) 2024-04-26T00:23:03.011 INFO:tasks.rados.rados.0.smithi045.stdout:554: done (5 left) 2024-04-26T00:23:03.011 INFO:tasks.rados.rados.0.smithi045.stdout:555: done (4 left) 2024-04-26T00:23:03.011 INFO:tasks.rados.rados.0.smithi045.stdout:556: done (3 left) 2024-04-26T00:23:03.012 INFO:tasks.rados.rados.0.smithi045.stdout:557: done (2 left) 2024-04-26T00:23:03.012 INFO:tasks.rados.rados.0.smithi045.stdout:558: done (1 left) 2024-04-26T00:23:03.012 INFO:tasks.rados.rados.0.smithi045.stdout:560: done (0 left) 2024-04-26T00:23:03.012 INFO:tasks.rados.rados.0.smithi045.stdout:561: rollback oid 37 current snap is 60 2024-04-26T00:23:03.012 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 37 to 58 2024-04-26T00:23:03.012 INFO:tasks.rados.rados.0.smithi045.stdout:562: delete oid 43 current snap is 60 2024-04-26T00:23:03.016 INFO:tasks.rados.rados.0.smithi045.stdout:562: done (1 left) 2024-04-26T00:23:03.016 INFO:tasks.rados.rados.0.smithi045.stdout:563: read oid 48 snap -1 2024-04-26T00:23:03.016 INFO:tasks.rados.rados.0.smithi045.stdout:563: expect (ObjNum 206 snap 53 seq_num 206) 2024-04-26T00:23:03.016 INFO:tasks.rados.rados.0.smithi045.stdout:564: rollback oid 13 current snap is 60 2024-04-26T00:23:03.016 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 13 to 55 2024-04-26T00:23:03.017 INFO:tasks.rados.rados.0.smithi045.stdout:565: read oid 18 snap -1 2024-04-26T00:23:03.017 INFO:tasks.rados.rados.0.smithi045.stdout:565: expect (ObjNum 165 snap 38 seq_num 165) 2024-04-26T00:23:03.017 INFO:tasks.rados.rados.0.smithi045.stdout:566: read oid 33 snap 57 2024-04-26T00:23:03.017 INFO:tasks.rados.rados.0.smithi045.stdout:566: expect (ObjNum 15 snap 0 seq_num 15) 2024-04-26T00:23:03.017 INFO:tasks.rados.rados.0.smithi045.stdout:567: snap_remove snap 35 2024-04-26T00:23:03.801 INFO:tasks.rados.rados.0.smithi045.stdout:567: done (5 left) 2024-04-26T00:23:03.801 INFO:tasks.rados.rados.0.smithi045.stdout:568: write (excl) oid 10 current snap is 60 2024-04-26T00:23:03.801 INFO:tasks.rados.rados.0.smithi045.stdout:561: finishing rollback tid 0 to smithi04526579-37 2024-04-26T00:23:03.801 INFO:tasks.rados.rados.0.smithi045.stdout:564: finishing rollback tid 0 to smithi04526579-13 2024-04-26T00:23:03.801 INFO:tasks.rados.rados.0.smithi045.stdout:564: finishing rollback tid 1 to smithi04526579-13 2024-04-26T00:23:03.801 INFO:tasks.rados.rados.0.smithi045.stdout:564: finishing rollback tid 2 to smithi04526579-13 2024-04-26T00:23:03.801 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 137 (ObjNum 204 snap 53 seq_num 204) dirty exists 2024-04-26T00:23:03.801 INFO:tasks.rados.rados.0.smithi045.stdout:561: finishing rollback tid 1 to smithi04526579-37 2024-04-26T00:23:03.801 INFO:tasks.rados.rados.0.smithi045.stdout:561: finishing rollback tid 2 to smithi04526579-37 2024-04-26T00:23:03.801 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 333 (ObjNum 215 snap 55 seq_num 215) dirty exists 2024-04-26T00:23:03.802 INFO:tasks.rados.rados.0.smithi045.stdout:565: expect (ObjNum 165 snap 38 seq_num 165) 2024-04-26T00:23:03.938 INFO:tasks.rados.rados.0.smithi045.stdout:563: expect (ObjNum 206 snap 53 seq_num 206) 2024-04-26T00:23:04.051 INFO:tasks.rados.rados.0.smithi045.stdout:566: expect (ObjNum 15 snap 0 seq_num 15) 2024-04-26T00:23:04.124 INFO:tasks.rados.rados.0.smithi045.stdout:568: seq_num 241 ranges {450487=541952,1636278=414209} 2024-04-26T00:23:04.140 INFO:tasks.rados.rados.0.smithi045.stdout:568: writing smithi04526579-10 from 450487 to 992439 tid 1 2024-04-26T00:23:04.158 INFO:tasks.rados.rados.0.smithi045.stdout:568: writing smithi04526579-10 from 1636278 to 2050487 tid 2 2024-04-26T00:23:04.158 INFO:tasks.rados.rados.0.smithi045.stdout:561: done (5 left) 2024-04-26T00:23:04.158 INFO:tasks.rados.rados.0.smithi045.stdout:563: done (4 left) 2024-04-26T00:23:04.158 INFO:tasks.rados.rados.0.smithi045.stdout:564: done (3 left) 2024-04-26T00:23:04.158 INFO:tasks.rados.rados.0.smithi045.stdout:565: done (2 left) 2024-04-26T00:23:04.158 INFO:tasks.rados.rados.0.smithi045.stdout:566: done (1 left) 2024-04-26T00:23:04.158 INFO:tasks.rados.rados.0.smithi045.stdout:569: snap_create 2024-04-26T00:23:04.159 INFO:tasks.rados.rados.0.smithi045.stdout:568: finishing write tid 1 to smithi04526579-10 2024-04-26T00:23:04.165 INFO:tasks.rados.rados.0.smithi045.stdout:568: finishing write tid 2 to smithi04526579-10 2024-04-26T00:23:04.166 INFO:tasks.rados.rados.0.smithi045.stdout:568: finishing write tid 3 to smithi04526579-10 2024-04-26T00:23:04.166 INFO:tasks.rados.rados.0.smithi045.stdout:568: finishing write tid 4 to smithi04526579-10 2024-04-26T00:23:04.167 INFO:tasks.rados.rados.0.smithi045.stdout:568: oid 10 updating version 0 to 235 2024-04-26T00:23:04.167 INFO:tasks.rados.rados.0.smithi045.stdout:568: oid 10 updating version 235 to 237 2024-04-26T00:23:04.167 INFO:tasks.rados.rados.0.smithi045.stdout:568: oid 10 version 237 is already newer than 236 2024-04-26T00:23:04.167 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 237 (ObjNum 241 snap 60 seq_num 241) dirty exists 2024-04-26T00:23:04.167 INFO:tasks.rados.rados.0.smithi045.stdout:568: left oid 10 (ObjNum 241 snap 60 seq_num 241) 2024-04-26T00:23:04.724 INFO:tasks.rados.rados.0.smithi045.stdout:568: done (1 left) 2024-04-26T00:23:04.725 INFO:tasks.rados.rados.0.smithi045.stdout:569: done (0 left) 2024-04-26T00:23:04.725 INFO:tasks.rados.rados.0.smithi045.stdout:570: rollback oid 15 current snap is 61 2024-04-26T00:23:04.725 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 15 to 58 2024-04-26T00:23:04.725 INFO:tasks.rados.rados.0.smithi045.stdout:571: write oid 22 current snap is 61 2024-04-26T00:23:04.725 INFO:tasks.rados.rados.0.smithi045.stdout:571: seq_num 242 ranges {467294=617201,1578407=488887} 2024-04-26T00:23:04.744 INFO:tasks.rados.rados.0.smithi045.stdout:571: writing smithi04526579-22 from 467294 to 1084495 tid 1 2024-04-26T00:23:04.761 INFO:tasks.rados.rados.0.smithi045.stdout:571: writing smithi04526579-22 from 1578407 to 2067294 tid 2 2024-04-26T00:23:04.761 INFO:tasks.rados.rados.0.smithi045.stdout:572: rollback oid 49 current snap is 61 2024-04-26T00:23:04.761 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 49 to 32 2024-04-26T00:23:04.761 INFO:tasks.rados.rados.0.smithi045.stdout:570: finishing rollback tid 0 to smithi04526579-15 2024-04-26T00:23:04.761 INFO:tasks.rados.rados.0.smithi045.stdout:570: finishing rollback tid 1 to smithi04526579-15 2024-04-26T00:23:04.761 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 264 (ObjNum 129 snap 0 seq_num 2242059664) dirty dne 2024-04-26T00:23:04.761 INFO:tasks.rados.rados.0.smithi045.stdout:571: finishing write tid 1 to smithi04526579-22 2024-04-26T00:23:04.761 INFO:tasks.rados.rados.0.smithi045.stdout:570: done (2 left) 2024-04-26T00:23:04.762 INFO:tasks.rados.rados.0.smithi045.stdout:573: write (excl) oid 6 current snap is 61 2024-04-26T00:23:04.762 INFO:tasks.rados.rados.0.smithi045.stdout:573: seq_num 243 ranges {484101=692450,1636889=447212} 2024-04-26T00:23:04.779 INFO:tasks.rados.rados.0.smithi045.stdout:573: writing smithi04526579-6 from 484101 to 1176551 tid 1 2024-04-26T00:23:04.791 INFO:tasks.rados.rados.0.smithi045.stdout:573: writing smithi04526579-6 from 1636889 to 2084101 tid 2 2024-04-26T00:23:04.791 INFO:tasks.rados.rados.0.smithi045.stdout:574: delete oid 18 current snap is 61 2024-04-26T00:23:04.791 INFO:tasks.rados.rados.0.smithi045.stdout:572: finishing rollback tid 0 to smithi04526579-49 2024-04-26T00:23:04.791 INFO:tasks.rados.rados.0.smithi045.stdout:571: finishing write tid 2 to smithi04526579-22 2024-04-26T00:23:04.791 INFO:tasks.rados.rados.0.smithi045.stdout:571: finishing write tid 3 to smithi04526579-22 2024-04-26T00:23:04.791 INFO:tasks.rados.rados.0.smithi045.stdout:571: finishing write tid 4 to smithi04526579-22 2024-04-26T00:23:04.791 INFO:tasks.rados.rados.0.smithi045.stdout:571: oid 22 updating version 0 to 134 2024-04-26T00:23:04.791 INFO:tasks.rados.rados.0.smithi045.stdout:571: oid 22 version 134 is already newer than 132 2024-04-26T00:23:04.791 INFO:tasks.rados.rados.0.smithi045.stdout:571: oid 22 version 134 is already newer than 133 2024-04-26T00:23:04.792 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 134 (ObjNum 242 snap 61 seq_num 242) dirty exists 2024-04-26T00:23:04.792 INFO:tasks.rados.rados.0.smithi045.stdout:571: left oid 22 (ObjNum 242 snap 61 seq_num 242) 2024-04-26T00:23:04.792 INFO:tasks.rados.rados.0.smithi045.stdout:572: finishing rollback tid 1 to smithi04526579-49 2024-04-26T00:23:04.792 INFO:tasks.rados.rados.0.smithi045.stdout:572: finishing rollback tid 2 to smithi04526579-49 2024-04-26T00:23:04.792 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 209 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-26T00:23:04.792 INFO:tasks.rados.rados.0.smithi045.stdout:573: finishing write tid 1 to smithi04526579-6 2024-04-26T00:23:04.796 INFO:tasks.rados.rados.0.smithi045.stdout:573: finishing write tid 2 to smithi04526579-6 2024-04-26T00:23:04.797 INFO:tasks.rados.rados.0.smithi045.stdout:573: finishing write tid 3 to smithi04526579-6 2024-04-26T00:23:04.798 INFO:tasks.rados.rados.0.smithi045.stdout:573: finishing write tid 4 to smithi04526579-6 2024-04-26T00:23:04.798 INFO:tasks.rados.rados.0.smithi045.stdout:573: oid 6 updating version 0 to 166 2024-04-26T00:23:04.798 INFO:tasks.rados.rados.0.smithi045.stdout:573: oid 6 updating version 166 to 167 2024-04-26T00:23:04.798 INFO:tasks.rados.rados.0.smithi045.stdout:573: oid 6 updating version 167 to 168 2024-04-26T00:23:04.798 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 168 (ObjNum 243 snap 61 seq_num 243) dirty exists 2024-04-26T00:23:04.798 INFO:tasks.rados.rados.0.smithi045.stdout:573: left oid 6 (ObjNum 243 snap 61 seq_num 243) 2024-04-26T00:23:04.804 INFO:tasks.rados.rados.0.smithi045.stdout:571: done (3 left) 2024-04-26T00:23:04.804 INFO:tasks.rados.rados.0.smithi045.stdout:572: done (2 left) 2024-04-26T00:23:04.804 INFO:tasks.rados.rados.0.smithi045.stdout:573: done (1 left) 2024-04-26T00:23:04.804 INFO:tasks.rados.rados.0.smithi045.stdout:574: done (0 left) 2024-04-26T00:23:04.805 INFO:tasks.rados.rados.0.smithi045.stdout:575: snap_create 2024-04-26T00:23:05.808 INFO:tasks.rados.rados.0.smithi045.stdout:575: done (0 left) 2024-04-26T00:23:05.809 INFO:tasks.rados.rados.0.smithi045.stdout:576: read oid 50 snap -1 2024-04-26T00:23:05.809 INFO:tasks.rados.rados.0.smithi045.stdout:576: expect (ObjNum 136 snap 30 seq_num 136) 2024-04-26T00:23:05.809 INFO:tasks.rados.rados.0.smithi045.stdout:577: read oid 3 snap -1 2024-04-26T00:23:05.809 INFO:tasks.rados.rados.0.smithi045.stdout:577: expect (ObjNum 201 snap 53 seq_num 201) 2024-04-26T00:23:05.809 INFO:tasks.rados.rados.0.smithi045.stdout:578: write (excl) oid 6 current snap is 62 2024-04-26T00:23:05.809 INFO:tasks.rados.rados.0.smithi045.stdout:578: seq_num 245 ranges {517715=559301,1870206=247509} 2024-04-26T00:23:05.821 INFO:tasks.rados.rados.0.smithi045.stdout:578: writing smithi04526579-6 from 517715 to 1077016 tid 1 2024-04-26T00:23:05.835 INFO:tasks.rados.rados.0.smithi045.stdout:578: writing smithi04526579-6 from 1870206 to 2117715 tid 2 2024-04-26T00:23:05.835 INFO:tasks.rados.rados.0.smithi045.stdout:579: snap_create 2024-04-26T00:23:05.835 INFO:tasks.rados.rados.0.smithi045.stdout:576: expect (ObjNum 136 snap 30 seq_num 136) 2024-04-26T00:23:06.003 INFO:tasks.rados.rados.0.smithi045.stdout:577: expect (ObjNum 201 snap 53 seq_num 201) 2024-04-26T00:23:06.114 INFO:tasks.rados.rados.0.smithi045.stdout:578: finishing write tid 1 to smithi04526579-6 2024-04-26T00:23:06.114 INFO:tasks.rados.rados.0.smithi045.stdout:578: finishing write tid 2 to smithi04526579-6 2024-04-26T00:23:06.114 INFO:tasks.rados.rados.0.smithi045.stdout:578: finishing write tid 3 to smithi04526579-6 2024-04-26T00:23:06.114 INFO:tasks.rados.rados.0.smithi045.stdout:578: finishing write tid 4 to smithi04526579-6 2024-04-26T00:23:06.115 INFO:tasks.rados.rados.0.smithi045.stdout:578: oid 6 updating version 0 to 171 2024-04-26T00:23:06.115 INFO:tasks.rados.rados.0.smithi045.stdout:578: oid 6 version 171 is already newer than 169 2024-04-26T00:23:06.115 INFO:tasks.rados.rados.0.smithi045.stdout:578: oid 6 version 171 is already newer than 170 2024-04-26T00:23:06.115 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 171 (ObjNum 245 snap 62 seq_num 245) dirty exists 2024-04-26T00:23:06.115 INFO:tasks.rados.rados.0.smithi045.stdout:578: left oid 6 (ObjNum 245 snap 62 seq_num 245) 2024-04-26T00:23:06.817 INFO:tasks.rados.rados.0.smithi045.stdout:576: done (3 left) 2024-04-26T00:23:06.817 INFO:tasks.rados.rados.0.smithi045.stdout:577: done (2 left) 2024-04-26T00:23:06.817 INFO:tasks.rados.rados.0.smithi045.stdout:578: done (1 left) 2024-04-26T00:23:06.817 INFO:tasks.rados.rados.0.smithi045.stdout:579: done (0 left) 2024-04-26T00:23:06.817 INFO:tasks.rados.rados.0.smithi045.stdout:580: snap_remove snap 62 2024-04-26T00:23:07.823 INFO:tasks.rados.rados.0.smithi045.stdout:580: done (0 left) 2024-04-26T00:23:07.824 INFO:tasks.rados.rados.0.smithi045.stdout:581: write (excl) oid 33 current snap is 63 2024-04-26T00:23:07.824 INFO:tasks.rados.rados.0.smithi045.stdout:581: seq_num 246 ranges {534522=634550,1812335=322187} 2024-04-26T00:23:07.830 INFO:tasks.rados.rados.0.smithi045.stdout:581: writing smithi04526579-33 from 534522 to 1169072 tid 1 2024-04-26T00:23:07.835 INFO:tasks.rados.rados.0.smithi045.stdout:581: writing smithi04526579-33 from 1812335 to 2134522 tid 2 2024-04-26T00:23:07.835 INFO:tasks.rados.rados.0.smithi045.stdout:582: write oid 28 current snap is 63 2024-04-26T00:23:07.836 INFO:tasks.rados.rados.0.smithi045.stdout:582: seq_num 247 ranges {551329=709799,1870817=280512} 2024-04-26T00:23:07.842 INFO:tasks.rados.rados.0.smithi045.stdout:582: writing smithi04526579-28 from 551329 to 1261128 tid 1 2024-04-26T00:23:07.850 INFO:tasks.rados.rados.0.smithi045.stdout:582: writing smithi04526579-28 from 1870817 to 2151329 tid 2 2024-04-26T00:23:07.851 INFO:tasks.rados.rados.0.smithi045.stdout:583: copy_from oid 18 from oid 9 current snap is 63 2024-04-26T00:23:07.851 INFO:tasks.rados.rados.0.smithi045.stdout:584: copy_from oid 3 from oid 44 current snap is 63 2024-04-26T00:23:07.851 INFO:tasks.rados.rados.0.smithi045.stdout:585: copy_from oid 40 from oid 26 current snap is 63 2024-04-26T00:23:07.851 INFO:tasks.rados.rados.0.smithi045.stdout:586: rollback oid 41 current snap is 63 2024-04-26T00:23:07.851 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 41 to 59 2024-04-26T00:23:07.851 INFO:tasks.rados.rados.0.smithi045.stdout:587: snap_create 2024-04-26T00:23:07.853 INFO:tasks.rados.rados.0.smithi045.stdout:581: finishing write tid 1 to smithi04526579-33 2024-04-26T00:23:07.855 INFO:tasks.rados.rados.0.smithi045.stdout:581: finishing write tid 2 to smithi04526579-33 2024-04-26T00:23:07.856 INFO:tasks.rados.rados.0.smithi045.stdout:581: finishing write tid 3 to smithi04526579-33 2024-04-26T00:23:07.857 INFO:tasks.rados.rados.0.smithi045.stdout:581: finishing write tid 4 to smithi04526579-33 2024-04-26T00:23:07.857 INFO:tasks.rados.rados.0.smithi045.stdout:581: oid 33 updating version 0 to 351 2024-04-26T00:23:07.857 INFO:tasks.rados.rados.0.smithi045.stdout:581: oid 33 version 351 is already newer than 350 2024-04-26T00:23:07.857 INFO:tasks.rados.rados.0.smithi045.stdout:581: oid 33 version 351 is already newer than 349 2024-04-26T00:23:07.857 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 351 (ObjNum 246 snap 63 seq_num 246) dirty exists 2024-04-26T00:23:07.857 INFO:tasks.rados.rados.0.smithi045.stdout:581: left oid 33 (ObjNum 246 snap 63 seq_num 246) 2024-04-26T00:23:07.870 INFO:tasks.rados.rados.0.smithi045.stdout:582: finishing write tid 1 to smithi04526579-28 2024-04-26T00:23:07.870 INFO:tasks.rados.rados.0.smithi045.stdout:582: finishing write tid 2 to smithi04526579-28 2024-04-26T00:23:07.870 INFO:tasks.rados.rados.0.smithi045.stdout:582: finishing write tid 3 to smithi04526579-28 2024-04-26T00:23:07.871 INFO:tasks.rados.rados.0.smithi045.stdout:582: finishing write tid 4 to smithi04526579-28 2024-04-26T00:23:07.871 INFO:tasks.rados.rados.0.smithi045.stdout:582: oid 28 updating version 0 to 266 2024-04-26T00:23:07.871 INFO:tasks.rados.rados.0.smithi045.stdout:582: oid 28 updating version 266 to 268 2024-04-26T00:23:07.871 INFO:tasks.rados.rados.0.smithi045.stdout:582: oid 28 version 268 is already newer than 267 2024-04-26T00:23:07.871 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 268 (ObjNum 247 snap 63 seq_num 247) dirty exists 2024-04-26T00:23:07.871 INFO:tasks.rados.rados.0.smithi045.stdout:582: left oid 28 (ObjNum 247 snap 63 seq_num 247) 2024-04-26T00:23:07.872 INFO:tasks.rados.rados.0.smithi045.stdout:586: finishing rollback tid 1 to smithi04526579-41 2024-04-26T00:23:07.872 INFO:tasks.rados.rados.0.smithi045.stdout:586: finishing rollback tid 2 to smithi04526579-41 2024-04-26T00:23:07.872 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 270 (ObjNum 224 snap 59 seq_num 224) dirty exists 2024-04-26T00:23:07.883 INFO:tasks.rados.rados.0.smithi045.stdout:583: finishing copy_from racing read to smithi04526579-18 2024-04-26T00:23:07.886 INFO:tasks.rados.rados.0.smithi045.stdout:583: finishing copy_from to smithi04526579-18 2024-04-26T00:23:07.886 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 175 (ObjNum 190 snap 49 seq_num 190) dirty exists 2024-04-26T00:23:07.887 INFO:tasks.rados.rados.0.smithi045.stdout:584: finishing copy_from racing read to smithi04526579-3 2024-04-26T00:23:07.888 INFO:tasks.rados.rados.0.smithi045.stdout:584: finishing copy_from to smithi04526579-3 2024-04-26T00:23:07.888 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 216 (ObjNum 119 snap 28 seq_num 119) dirty exists 2024-04-26T00:23:07.890 INFO:tasks.rados.rados.0.smithi045.stdout:585: finishing copy_from racing read to smithi04526579-40 2024-04-26T00:23:07.896 INFO:tasks.rados.rados.0.smithi045.stdout:585: finishing copy_from to smithi04526579-40 2024-04-26T00:23:07.896 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 191 (ObjNum 209 snap 53 seq_num 209) dirty exists 2024-04-26T00:23:08.055 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-04-26T00:23:08.055 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-04-26T00:23:08.055 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-26T00:23:08.055 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-26T00:23:08.058 INFO:tasks.ceph.mon.a:Started 2024-04-26T00:23:08.058 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:23:08.058 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:23:08.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.218+0000 7f0d9cbe1640 10 monclient: get_monmap_and_config 2024-04-26T00:23:08.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.218+0000 7f0d9cbe1640 10 monclient: build_initial_monmap 2024-04-26T00:23:08.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.218+0000 7f0d9cbe1640 10 monclient: monmap: 2024-04-26T00:23:08.195 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:23:08.195 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:08.195 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:23:08.220633+0000 2024-04-26T00:23:08.195 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:23:08.220633+0000 2024-04-26T00:23:08.195 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:23:08.196 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:08.196 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:23:08.196 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:23:08.196 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:23:08.196 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:23:08.196 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:23:08.196 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:08.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.218+0000 7f0d9cbe1640 1 Processor -- start 2024-04-26T00:23:08.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.218+0000 7f0d9cbe1640 1 -- start start 2024-04-26T00:23:08.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.218+0000 7f0d9cbe1640 10 monclient: init 2024-04-26T00:23:08.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.218+0000 7f0d9cbe1640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:08.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.218+0000 7f0d9cbe1640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-04-26T00:23:08.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.218+0000 7f0d9cbe1640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f0d98136e10 0x7f0d9809c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:08.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.218+0000 7f0d9cbe1640 10 monclient(hunting): picked mon.noname-a con 0x7f0d98136e10 addr v2:172.21.15.37:3300/0 2024-04-26T00:23:08.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.218+0000 7f0d9cbe1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0d981371f0 addr v1:172.21.15.45:6789/0 2024-04-26T00:23:08.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.218+0000 7f0d9cbe1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0d9813c340 addr v1:172.21.15.37:6789/0 2024-04-26T00:23:08.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.218+0000 7f0d9cbe1640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d9809b680 con 0x7f0d9813c340 2024-04-26T00:23:08.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.218+0000 7f0d9cbe1640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d9809bbd0 con 0x7f0d981371f0 2024-04-26T00:23:08.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.218+0000 7f0d9cbe1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:08.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.218+0000 7f0d9cbe1640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f0d9809b8f0 con 0x7f0d98136e10 2024-04-26T00:23:08.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.218+0000 7f0d9cbe1640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:08.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.218+0000 7f0d9cbe1640 10 monclient(hunting): authenticate will time out at 1169.526234s 2024-04-26T00:23:08.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.218+0000 7f0d9659c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f0d98136e10 0x7f0d9809c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:08.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.218+0000 7f0d96d9d640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f0d9813c340 0x7f0d9813e730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:48128/0 (socket says 172.21.15.37:48128) 2024-04-26T00:23:08.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.218+0000 7f0d9659c640 10 monclient(hunting): get_auth_request con 0x7f0d98136e10 auth_method 0 2024-04-26T00:23:08.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.218+0000 7f0d96d9d640 1 -- 172.21.15.37:0/3254894146 learned_addr learned my addr 172.21.15.37:0/3254894146 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:23:08.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.218+0000 7f0d9659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:08.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.218+0000 7f0d9659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:08.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.218+0000 7f0d9659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:08.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.218+0000 7f0d9659c640 1 -- 172.21.15.37:0/3254894146 >> v2:172.21.15.37:3300/0 conn(0x7f0d98136e10 msgr2=0x7f0d9809c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:23:08.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.218+0000 7f0d9659c640 1 -- 172.21.15.37:0/3254894146 >> v2:172.21.15.37:3300/0 conn(0x7f0d98136e10 msgr2=0x7f0d9809c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:23:08.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.218+0000 7f0d9659c640 1 --2- 172.21.15.37:0/3254894146 >> v2:172.21.15.37:3300/0 conn(0x7f0d98136e10 0x7f0d9809c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:23:08.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.218+0000 7f0d9659c640 1 --2- 172.21.15.37:0/3254894146 >> v2:172.21.15.37:3300/0 conn(0x7f0d98136e10 0x7f0d9809c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:23:08.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d96d9d640 1 -- 172.21.15.37:0/3254894146 >> v1:172.21.15.37:6789/0 conn(0x7f0d9813c340 legacy=0x7f0d9813e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:08.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d8dd9b640 1 -- 172.21.15.37:0/3254894146 >> v1:172.21.15.45:6789/0 conn(0x7f0d981371f0 legacy=0x7f0d98137600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:08.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d95d9b640 1 -- 172.21.15.37:0/3254894146 <== mon.1 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1200060269 0 0) 0x7f0d9809bbd0 con 0x7f0d981371f0 2024-04-26T00:23:08.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d95d9b640 10 monclient(hunting): _negotiate 2024-04-26T00:23:08.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d95d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:08.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d95d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:08.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d95d9b640 10 monclient(hunting): my global_id is 24238 2024-04-26T00:23:08.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d95d9b640 1 -- 172.21.15.37:0/3254894146 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d84003530 con 0x7f0d981371f0 2024-04-26T00:23:08.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d95d9b640 1 -- 172.21.15.37:0/3254894146 <== mon.1 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 738635573 0 0) 0x7f0d84003530 con 0x7f0d981371f0 2024-04-26T00:23:08.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d95d9b640 1 -- 172.21.15.37:0/3254894146 >> v2:172.21.15.37:3300/0 conn(0x7f0d98136e10 msgr2=0x7f0d9809c5b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:23:08.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d95d9b640 1 --2- 172.21.15.37:0/3254894146 >> v2:172.21.15.37:3300/0 conn(0x7f0d98136e10 0x7f0d9809c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:08.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d95d9b640 1 -- 172.21.15.37:0/3254894146 >> v1:172.21.15.37:6789/0 conn(0x7f0d9813c340 legacy=0x7f0d9813e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:08.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d95d9b640 10 monclient: _finish_hunting 0 2024-04-26T00:23:08.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d95d9b640 1 monclient: found mon.noname-c-legacy 2024-04-26T00:23:08.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d95d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:08.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d95d9b640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.45:6789/0 2024-04-26T00:23:08.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d95d9b640 1 -- 172.21.15.37:0/3254894146 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d980a5d90 con 0x7f0d981371f0 2024-04-26T00:23:08.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d95d9b640 10 monclient: _finish_auth 0 2024-04-26T00:23:08.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d95d9b640 10 monclient: _check_auth_tickets 2024-04-26T00:23:08.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d95d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:08.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d95d9b640 1 -- 172.21.15.37:0/3254894146 <== mon.1 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 866326386 0 0) 0x7f0d80001980 con 0x7f0d981371f0 2024-04-26T00:23:08.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d95d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:08.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d95d9b640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-04-26T00:23:08.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d95d9b640 10 monclient: dump: 2024-04-26T00:23:08.205 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:08.205 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:08.205 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:08.205 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:08.205 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:08.205 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:08.205 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:08.205 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:08.205 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:08.206 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:08.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d95d9b640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] but i'm connected to v1:172.21.15.45:6789/0, reconnecting 2024-04-26T00:23:08.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d95d9b640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:08.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d95d9b640 1 -- 172.21.15.37:0/3254894146 >> v1:172.21.15.45:6789/0 conn(0x7f0d981371f0 legacy=0x7f0d98137600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:08.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d95d9b640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:23:08.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d95d9b640 1 --2- 172.21.15.37:0/3254894146 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d84003a90 0x7f0d84003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:08.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d95d9b640 10 monclient(hunting): picked mon.a con 0x7f0d84003a90 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:23:08.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d95d9b640 1 --2- 172.21.15.37:0/3254894146 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d84004810 0x7f0d84004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:08.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d95d9b640 10 monclient(hunting): picked mon.c con 0x7f0d84004810 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:23:08.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d95d9b640 1 --2- 172.21.15.37:0/3254894146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d84005420 0x7f0d8400ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:08.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d95d9b640 10 monclient(hunting): picked mon.b con 0x7f0d84005420 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:23:08.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d95d9b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:08.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d95d9b640 1 -- 172.21.15.37:0/3254894146 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f0d9809b8f0 con 0x7f0d84003a90 2024-04-26T00:23:08.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d95d9b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:08.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d95d9b640 1 -- 172.21.15.37:0/3254894146 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0d8400c330 con 0x7f0d84005420 2024-04-26T00:23:08.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d95d9b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:08.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d95d9b640 1 -- 172.21.15.37:0/3254894146 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0d8400c530 con 0x7f0d84004810 2024-04-26T00:23:08.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d95d9b640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:08.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d95d9b640 10 monclient(hunting): _finish_auth 0 2024-04-26T00:23:08.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d9659c640 1 --2- 172.21.15.37:0/3254894146 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d84003a90 0x7f0d84003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:08.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d8dd9b640 1 --2- 172.21.15.37:0/3254894146 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d84004810 0x7f0d84004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:08.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d9659c640 10 monclient(hunting): get_auth_request con 0x7f0d84003a90 auth_method 0 2024-04-26T00:23:08.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d9659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:08.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d9659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:08.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d9659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:23:08.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d8dd9b640 10 monclient(hunting): get_auth_request con 0x7f0d84004810 auth_method 0 2024-04-26T00:23:08.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d8dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:08.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d8dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:08.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d8dd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:23:08.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d9659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:08.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d9659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:08.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d9659c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:23:08.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d8dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:08.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d8dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:08.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d8dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:23:08.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d96d9d640 1 --2- 172.21.15.37:0/3254894146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d84005420 0x7f0d8400ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:08.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d96d9d640 10 monclient(hunting): get_auth_request con 0x7f0d84005420 auth_method 0 2024-04-26T00:23:08.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d96d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:08.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d96d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:08.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d96d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:23:08.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d8dd9b640 10 monclient(hunting): handle_auth_done global_id 24238 payload 293 2024-04-26T00:23:08.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d8dd9b640 1 -- 172.21.15.37:0/3254894146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d84005420 msgr2=0x7f0d8400ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:23:08.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d8dd9b640 1 --2- 172.21.15.37:0/3254894146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d84005420 0x7f0d8400ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:08.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d8dd9b640 1 -- 172.21.15.37:0/3254894146 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d84003a90 msgr2=0x7f0d84003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:23:08.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d8dd9b640 1 --2- 172.21.15.37:0/3254894146 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d84003a90 0x7f0d84003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:08.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d8dd9b640 10 monclient: _finish_hunting 0 2024-04-26T00:23:08.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d8dd9b640 1 monclient: found mon.c 2024-04-26T00:23:08.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d8dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:08.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d8dd9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:08.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d8dd9b640 1 -- 172.21.15.37:0/3254894146 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d84003530 con 0x7f0d84004810 2024-04-26T00:23:08.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d8dd9b640 10 monclient: _finish_auth 0 2024-04-26T00:23:08.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d8dd9b640 10 monclient: _check_auth_tickets 2024-04-26T00:23:08.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d8dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:08.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d96d9d640 1 -- 172.21.15.37:0/3254894146 reap_dead start 2024-04-26T00:23:08.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d9659c640 1 --2- 172.21.15.37:0/3254894146 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d84003a90 0x7f0d84003eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:23:08.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d9cbe1640 5 monclient: authenticate success, global_id 24238 2024-04-26T00:23:08.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.222+0000 7f0d9cbe1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:23:08.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.226+0000 7f0d8dd9b640 1 -- 172.21.15.37:0/3254894146 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d84004810 msgr2=0x7f0d84004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:23:08.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.226+0000 7f0d8dd9b640 1 -- 172.21.15.37:0/3254894146 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d84004810 msgr2=0x7f0d84004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:23:08.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.226+0000 7f0d8dd9b640 1 --2- 172.21.15.37:0/3254894146 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d84004810 0x7f0d84004c90 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7f0d800014f0 tx=0x7f0d8001cc70 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:23:08.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.226+0000 7f0d8dd9b640 1 --2- 172.21.15.37:0/3254894146 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d84004810 0x7f0d84004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:23:08.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d8dd9b640 1 --2- 172.21.15.37:0/3254894146 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d84004810 0x7f0d84004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:08.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d8dd9b640 10 monclient: get_auth_request con 0x7f0d84004810 auth_method 0 2024-04-26T00:23:08.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d8dd9b640 0 --2- 172.21.15.37:0/3254894146 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d84004810 0x7f0d84004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-04-26T00:23:08.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d8dd9b640 1 --2- 172.21.15.37:0/3254894146 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d84004810 0x7f0d84004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:08.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d95d9b640 10 monclient: ms_handle_reset current mon [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:23:08.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d95d9b640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:08.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d95d9b640 1 -- 172.21.15.37:0/3254894146 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d84004810 msgr2=0x7f0d84004c90 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-04-26T00:23:08.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d95d9b640 1 --2- 172.21.15.37:0/3254894146 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d84004810 0x7f0d84004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:08.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d95d9b640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:23:08.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d95d9b640 1 --2- 172.21.15.37:0/3254894146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d84003eb0 0x7f0d840099c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:08.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d95d9b640 10 monclient(hunting): picked mon.b con 0x7f0d84003eb0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:23:08.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d95d9b640 1 --2- 172.21.15.37:0/3254894146 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d8400a090 0x7f0d8400a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:08.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d95d9b640 10 monclient(hunting): picked mon.c con 0x7f0d8400a090 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:23:08.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d95d9b640 1 --2- 172.21.15.37:0/3254894146 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d8400abd0 0x7f0d8400b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:08.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d95d9b640 10 monclient(hunting): picked mon.a con 0x7f0d8400abd0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:23:08.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d95d9b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:08.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d95d9b640 1 -- 172.21.15.37:0/3254894146 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f0d8400b960 con 0x7f0d8400abd0 2024-04-26T00:23:08.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d95d9b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:08.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d95d9b640 1 -- 172.21.15.37:0/3254894146 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0d8400bb10 con 0x7f0d84003eb0 2024-04-26T00:23:08.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d95d9b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:08.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d95d9b640 1 -- 172.21.15.37:0/3254894146 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0d8400bd10 con 0x7f0d8400a090 2024-04-26T00:23:08.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d95d9b640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:08.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d95d9b640 10 monclient(hunting): ms_handle_reset hunted mon [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:23:08.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d8dd9b640 1 --2- 172.21.15.37:0/3254894146 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d8400a090 0x7f0d8400a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:08.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d8dd9b640 10 monclient(hunting): get_auth_request con 0x7f0d8400a090 auth_method 0 2024-04-26T00:23:08.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d8dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:08.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d8dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:08.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d8dd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:23:08.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d96d9d640 1 --2- 172.21.15.37:0/3254894146 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d8400abd0 0x7f0d8400b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:08.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d9659c640 1 --2- 172.21.15.37:0/3254894146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d84003eb0 0x7f0d840099c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:08.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d96d9d640 10 monclient(hunting): get_auth_request con 0x7f0d8400abd0 auth_method 0 2024-04-26T00:23:08.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d96d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:08.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d96d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:08.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d96d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:23:08.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d9659c640 10 monclient(hunting): get_auth_request con 0x7f0d84003eb0 auth_method 0 2024-04-26T00:23:08.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d9659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:08.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d9659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:08.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d9659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:23:08.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d8dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:08.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d8dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:08.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d8dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:23:08.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d96d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:08.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d96d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:08.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d96d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:23:08.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d9659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:08.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d9659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:08.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d9659c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:23:08.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d96d9d640 10 monclient(hunting): handle_auth_done global_id 24238 payload 293 2024-04-26T00:23:08.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d96d9d640 1 -- 172.21.15.37:0/3254894146 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d8400a090 msgr2=0x7f0d8400a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:23:08.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d96d9d640 1 --2- 172.21.15.37:0/3254894146 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d8400a090 0x7f0d8400a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:08.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d96d9d640 1 -- 172.21.15.37:0/3254894146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d84003eb0 msgr2=0x7f0d840099c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:23:08.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d96d9d640 1 --2- 172.21.15.37:0/3254894146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d84003eb0 0x7f0d840099c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:08.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d96d9d640 10 monclient: _finish_hunting 0 2024-04-26T00:23:08.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d96d9d640 1 monclient: found mon.a 2024-04-26T00:23:08.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d96d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:08.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d96d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:23:08.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d96d9d640 1 -- 172.21.15.37:0/3254894146 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=2+}) -- 0x7f0d9809bbd0 con 0x7f0d8400abd0 2024-04-26T00:23:08.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d96d9d640 10 monclient: _finish_auth 0 2024-04-26T00:23:08.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d96d9d640 10 monclient: _check_auth_tickets 2024-04-26T00:23:08.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d96d9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:08.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.430+0000 7f0d8dd9b640 1 --2- 172.21.15.37:0/3254894146 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d8400a090 0x7f0d8400a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:23:08.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.434+0000 7f0d96d9d640 1 -- 172.21.15.37:0/3254894146 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d8400abd0 msgr2=0x7f0d8400b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:08.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.434+0000 7f0d96d9d640 1 --2- 172.21.15.37:0/3254894146 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d8400abd0 0x7f0d8400b050 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0d9806e430 tx=0x7f0d80007a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d2236a50ddbf2e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:08.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.434+0000 7f0d95d9b640 1 -- 172.21.15.37:0/3254894146 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d90019070 con 0x7f0d8400abd0 2024-04-26T00:23:08.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.434+0000 7f0d95d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:08.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.434+0000 7f0d95d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:23:08.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.434+0000 7f0d95d9b640 10 monclient: dump: 2024-04-26T00:23:08.417 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:08.417 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:08.417 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:08.417 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:08.417 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:08.417 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:08.417 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:08.417 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:08.418 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:08.418 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:08.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.434+0000 7f0d9cbe1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:23:08.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.434+0000 7f0d95d9b640 1 -- 172.21.15.37:0/3254894146 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d90014480 con 0x7f0d8400abd0 2024-04-26T00:23:08.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.434+0000 7f0d95d9b640 10 monclient: handle_config config(0 keys) 2024-04-26T00:23:08.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.434+0000 7f0d9cbe1640 10 monclient: get_monmap_and_config success 2024-04-26T00:23:08.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.434+0000 7f0d9cbe1640 4 set_mon_vals no callback set 2024-04-26T00:23:08.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.434+0000 7f0d9cbe1640 10 monclient: shutdown 2024-04-26T00:23:08.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.434+0000 7f0d9cbe1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:23:08.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.434+0000 7f0d9cbe1640 1 -- 172.21.15.37:0/3254894146 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d8400abd0 msgr2=0x7f0d8400b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:08.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.434+0000 7f0d9cbe1640 1 --2- 172.21.15.37:0/3254894146 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d8400abd0 0x7f0d8400b050 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0d9806e430 tx=0x7f0d80007a60 comp rx=0 tx=0).stop 2024-04-26T00:23:08.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.434+0000 7f0d9cbe1640 1 -- 172.21.15.37:0/3254894146 shutdown_connections 2024-04-26T00:23:08.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.434+0000 7f0d9cbe1640 1 --2- 172.21.15.37:0/3254894146 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d8400abd0 0x7f0d8400b050 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:08.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.434+0000 7f0d9cbe1640 1 --2- 172.21.15.37:0/3254894146 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d8400a090 0x7f0d8400a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:08.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.434+0000 7f0d9cbe1640 1 --2- 172.21.15.37:0/3254894146 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d84003eb0 0x7f0d840099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:08.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.434+0000 7f0d9cbe1640 1 -- 172.21.15.37:0/3254894146 >> 172.21.15.37:0/3254894146 conn(0x7f0d980a64e0 msgr2=0x7f0d980a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:23:08.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.434+0000 7f0d9cbe1640 1 -- 172.21.15.37:0/3254894146 shutdown_connections 2024-04-26T00:23:08.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.434+0000 7f0d9cbe1640 1 -- 172.21.15.37:0/3254894146 wait complete. 2024-04-26T00:23:08.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.434+0000 7f0d9cbe1640 10 monclient: build_initial_monmap 2024-04-26T00:23:08.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.434+0000 7f0d9cbe1640 10 monclient: monmap: 2024-04-26T00:23:08.419 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:23:08.419 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:23:08.419 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:23:08.419 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:23:08.419 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:23:08.419 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:23:08.419 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:23:08.419 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:23:08.420 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:23:08.420 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:08.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.434+0000 7f0d9cbe1640 1 Processor -- start 2024-04-26T00:23:08.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d9cbe1640 1 -- start start 2024-04-26T00:23:08.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d9cbe1640 10 monclient: init 2024-04-26T00:23:08.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d9cbe1640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:08.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d9cbe1640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:23:08.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d9cbe1640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d84003eb0 0x7f0d981cf190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:08.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d9cbe1640 10 monclient(hunting): picked mon.noname-a con 0x7f0d84003eb0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:23:08.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d9cbe1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d8400a090 0x7f0d98121420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:08.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d9cbe1640 10 monclient(hunting): picked mon.noname-b con 0x7f0d8400a090 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:23:08.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d9cbe1640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d8400abd0 0x7f0d981cf820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:08.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d9cbe1640 10 monclient(hunting): picked mon.noname-c con 0x7f0d8400abd0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:23:08.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d9cbe1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:08.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d9cbe1640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f0d98136120 con 0x7f0d84003eb0 2024-04-26T00:23:08.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d9cbe1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:08.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d9cbe1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0d98135fa0 con 0x7f0d8400a090 2024-04-26T00:23:08.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d9cbe1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:08.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d9cbe1640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0d981362a0 con 0x7f0d8400abd0 2024-04-26T00:23:08.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d9cbe1640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:08.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d9cbe1640 10 monclient(hunting): authenticate will time out at 1169.744873s 2024-04-26T00:23:08.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d96d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d8400abd0 0x7f0d981cf820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:08.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d96d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d8400abd0 0x7f0d981cf820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:34442/0 (socket says 172.21.15.37:34442) 2024-04-26T00:23:08.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d96d9d640 1 -- 172.21.15.37:0/1213280360 learned_addr learned my addr 172.21.15.37:0/1213280360 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:23:08.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d96d9d640 10 monclient(hunting): get_auth_request con 0x7f0d8400abd0 auth_method 0 2024-04-26T00:23:08.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d96d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:08.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d96d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:08.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d96d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:08.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d9659c640 1 --2- 172.21.15.37:0/1213280360 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d84003eb0 0x7f0d981cf190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:08.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d9659c640 10 monclient(hunting): get_auth_request con 0x7f0d84003eb0 auth_method 0 2024-04-26T00:23:08.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d9659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:08.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d9659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:08.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d9659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:08.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d96d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:08.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d96d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:08.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d96d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:08.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d8dd9b640 1 --2- 172.21.15.37:0/1213280360 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d8400a090 0x7f0d98121420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:08.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d9659c640 1 -- 172.21.15.37:0/1213280360 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d84003eb0 msgr2=0x7f0d981cf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:23:08.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d9659c640 1 -- 172.21.15.37:0/1213280360 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d84003eb0 msgr2=0x7f0d981cf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:23:08.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d9659c640 1 --2- 172.21.15.37:0/1213280360 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d84003eb0 0x7f0d981cf190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:23:08.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d8dd9b640 10 monclient(hunting): get_auth_request con 0x7f0d8400a090 auth_method 0 2024-04-26T00:23:08.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d8dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:08.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d8dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:08.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d8dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:08.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d9659c640 1 --2- 172.21.15.37:0/1213280360 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d84003eb0 0x7f0d981cf190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:23:08.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d96d9d640 10 monclient(hunting): handle_auth_done global_id 34178 payload 824 2024-04-26T00:23:08.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d96d9d640 1 -- 172.21.15.37:0/1213280360 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d8400a090 msgr2=0x7f0d98121420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:08.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d96d9d640 1 --2- 172.21.15.37:0/1213280360 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d8400a090 0x7f0d98121420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:08.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d96d9d640 1 -- 172.21.15.37:0/1213280360 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d84003eb0 msgr2=0x7f0d981cf190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:23:08.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d96d9d640 1 --2- 172.21.15.37:0/1213280360 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d84003eb0 0x7f0d981cf190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:08.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d96d9d640 10 monclient: _finish_hunting 0 2024-04-26T00:23:08.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d96d9d640 1 monclient: found mon.noname-c 2024-04-26T00:23:08.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d96d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:08.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d96d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:23:08.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d96d9d640 1 -- 172.21.15.37:0/1213280360 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d981c2520 con 0x7f0d8400abd0 2024-04-26T00:23:08.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d96d9d640 1 -- 172.21.15.37:0/1213280360 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d8400abd0 msgr2=0x7f0d981cf820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:08.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.438+0000 7f0d96d9d640 1 --2- 172.21.15.37:0/1213280360 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d8400abd0 0x7f0d981cf820 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0d9806cac0 tx=0x7f0d90001860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:08.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d8f7fe640 1 -- 172.21.15.37:0/1213280360 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d90019070 con 0x7f0d8400abd0 2024-04-26T00:23:08.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d8f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:08.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d8f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:23:08.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d8f7fe640 10 monclient: dump: 2024-04-26T00:23:08.429 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:08.430 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:08.430 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:08.430 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:08.430 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:08.430 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:08.430 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:08.430 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:08.430 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:08.430 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:08.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d8f7fe640 10 monclient: _finish_auth 0 2024-04-26T00:23:08.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d8f7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:23:08.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:08.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d8f7fe640 1 -- 172.21.15.37:0/1213280360 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d90024070 con 0x7f0d8400abd0 2024-04-26T00:23:08.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d9cbe1640 5 monclient: authenticate success, global_id 34178 2024-04-26T00:23:08.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d9cbe1640 10 monclient: _renew_subs 2024-04-26T00:23:08.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d9cbe1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:08.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d9cbe1640 1 -- 172.21.15.37:0/1213280360 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d980a3760 con 0x7f0d8400abd0 2024-04-26T00:23:08.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d8f7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:23:08.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d8f7fe640 1 -- 172.21.15.37:0/1213280360 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d900020d0 con 0x7f0d8400abd0 2024-04-26T00:23:08.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d8f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:08.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d8f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:23:08.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d8f7fe640 10 monclient: dump: 2024-04-26T00:23:08.431 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:08.431 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:08.431 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:08.431 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:08.432 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:08.432 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:08.432 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:08.432 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:08.432 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:08.432 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:08.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d9559a640 4 set_mon_vals no callback set 2024-04-26T00:23:08.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d9cbe1640 10 monclient: _renew_subs 2024-04-26T00:23:08.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d9cbe1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:08.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d9cbe1640 1 -- 172.21.15.37:0/1213280360 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d98122c20 con 0x7f0d8400abd0 2024-04-26T00:23:08.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d96d9d640 1 -- 172.21.15.37:0/1213280360 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d8400abd0 msgr2=0x7f0d981cf820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:23:08.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d96d9d640 1 -- 172.21.15.37:0/1213280360 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d8400abd0 msgr2=0x7f0d981cf820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:23:08.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d96d9d640 1 --2- 172.21.15.37:0/1213280360 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d8400abd0 0x7f0d981cf820 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0d9806cac0 tx=0x7f0d90001860 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:23:08.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d96d9d640 1 --2- 172.21.15.37:0/1213280360 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d8400abd0 0x7f0d981cf820 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0d9806cac0 tx=0x7f0d90001860 comp rx=0 tx=0).stop 2024-04-26T00:23:08.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d8f7fe640 10 monclient: ms_handle_reset current mon [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:23:08.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d8f7fe640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:08.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d8f7fe640 1 -- 172.21.15.37:0/1213280360 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d8400abd0 msgr2=0x7f0d981cf820 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:23:08.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d8f7fe640 1 --2- 172.21.15.37:0/1213280360 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d8400abd0 0x7f0d981cf820 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:08.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d8f7fe640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:23:08.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d8f7fe640 1 --2- 172.21.15.37:0/1213280360 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d78003870 0x7f0d78005c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:08.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d8f7fe640 10 monclient(hunting): picked mon.c con 0x7f0d78003870 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:23:08.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d8f7fe640 1 --2- 172.21.15.37:0/1213280360 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d8400a090 0x7f0d78008760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:08.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d8f7fe640 10 monclient(hunting): picked mon.b con 0x7f0d8400a090 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:23:08.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d8f7fe640 1 --2- 172.21.15.37:0/1213280360 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d84003eb0 0x7f0d7800b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:08.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d8f7fe640 10 monclient(hunting): picked mon.a con 0x7f0d84003eb0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:23:08.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d8f7fe640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:08.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d8f7fe640 1 -- 172.21.15.37:0/1213280360 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f0d7800bbf0 con 0x7f0d84003eb0 2024-04-26T00:23:08.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d8f7fe640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:08.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d8f7fe640 1 -- 172.21.15.37:0/1213280360 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0d7800bd70 con 0x7f0d8400a090 2024-04-26T00:23:08.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d8f7fe640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:08.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d8f7fe640 1 -- 172.21.15.37:0/1213280360 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0d7800bef0 con 0x7f0d78003870 2024-04-26T00:23:08.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d8f7fe640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:08.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d9659c640 1 --2- 172.21.15.37:0/1213280360 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d78003870 0x7f0d78005c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:08.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d9659c640 10 monclient(hunting): get_auth_request con 0x7f0d78003870 auth_method 0 2024-04-26T00:23:08.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d9659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:08.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d9659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:08.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d9659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:23:08.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d96d9d640 1 --2- 172.21.15.37:0/1213280360 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d84003eb0 0x7f0d7800b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:08.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d96d9d640 10 monclient(hunting): get_auth_request con 0x7f0d84003eb0 auth_method 0 2024-04-26T00:23:08.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d96d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:08.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d96d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:08.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d96d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:23:08.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d9659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:08.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d9cbe1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:08.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d9659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:08.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d9659c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:23:08.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d9cbe1640 10 monclient(hunting): _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:08.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d96d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:08.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d96d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:08.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d8dd9b640 1 --2- 172.21.15.37:0/1213280360 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d8400a090 0x7f0d78008760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:08.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d96d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:23:08.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d8dd9b640 10 monclient(hunting): get_auth_request con 0x7f0d8400a090 auth_method 0 2024-04-26T00:23:08.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d8dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:08.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d8dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:08.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d8dd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:23:08.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d9659c640 10 monclient(hunting): handle_auth_done global_id 34178 payload 293 2024-04-26T00:23:08.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d9659c640 1 -- 172.21.15.37:0/1213280360 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d8400a090 msgr2=0x7f0d78008760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:08.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d9659c640 1 --2- 172.21.15.37:0/1213280360 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d8400a090 0x7f0d78008760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:08.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d9659c640 1 -- 172.21.15.37:0/1213280360 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d84003eb0 msgr2=0x7f0d7800b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:08.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d9659c640 1 --2- 172.21.15.37:0/1213280360 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d84003eb0 0x7f0d7800b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:08.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d9659c640 10 monclient: _finish_hunting 0 2024-04-26T00:23:08.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d9659c640 1 monclient: found mon.c 2024-04-26T00:23:08.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d9659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:08.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d9659c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:08.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d9659c640 1 -- 172.21.15.37:0/1213280360 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7f0d7800c210 con 0x7f0d78003870 2024-04-26T00:23:08.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d9659c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:08.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d9659c640 1 -- 172.21.15.37:0/1213280360 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d98066f40 con 0x7f0d78003870 2024-04-26T00:23:08.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d9659c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:08.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d9659c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:08.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d9659c640 1 -- 172.21.15.37:0/1213280360 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d8800cc60 con 0x7f0d78003870 2024-04-26T00:23:08.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d9659c640 10 monclient: _finish_auth 0 2024-04-26T00:23:08.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d9659c640 10 monclient: _check_auth_tickets 2024-04-26T00:23:08.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.442+0000 7f0d9659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:08.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.446+0000 7f0d9659c640 1 -- 172.21.15.37:0/1213280360 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d78003870 msgr2=0x7f0d78005c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:08.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.446+0000 7f0d9659c640 1 --2- 172.21.15.37:0/1213280360 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d78003870 0x7f0d78005c60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0d8800ce90 tx=0x7f0d90009030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:08.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.446+0000 7f0d8f7fe640 1 -- 172.21.15.37:0/1213280360 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d8800f040 con 0x7f0d78003870 2024-04-26T00:23:08.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.446+0000 7f0d8f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:08.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.446+0000 7f0d8f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:23:08.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.446+0000 7f0d8f7fe640 10 monclient: dump: 2024-04-26T00:23:08.437 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:08.437 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:08.437 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:08.437 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:08.437 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:08.437 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:08.437 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:08.438 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:08.438 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:08.438 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:08.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.446+0000 7f0d8f7fe640 1 -- 172.21.15.37:0/1213280360 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d8801e430 con 0x7f0d78003870 2024-04-26T00:23:08.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.446+0000 7f0d8f7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:23:08.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.446+0000 7f0d94d99640 4 set_mon_vals no callback set 2024-04-26T00:23:08.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.446+0000 7f0d8f7fe640 1 -- 172.21.15.37:0/1213280360 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f0d88016430 con 0x7f0d78003870 2024-04-26T00:23:08.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.446+0000 7f0d8f7fe640 1 --2- 172.21.15.37:0/1213280360 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0d8400abd0 0x7f0d780407e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:08.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.446+0000 7f0d8f7fe640 1 -- 172.21.15.37:0/1213280360 <== mon.2 v2:172.21.15.37:3301/0 4 ==== osd_map(133..133 src has 1..133) ==== 3975+0+0 (secure 0 0 0) 0x7f0d88047ed0 con 0x7f0d78003870 2024-04-26T00:23:08.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.446+0000 7f0d8dd9b640 1 --2- 172.21.15.37:0/1213280360 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0d8400abd0 0x7f0d780407e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:08.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.446+0000 7f0d8dd9b640 10 monclient: get_auth_request con 0x7f0d8400abd0 auth_method 0 2024-04-26T00:23:08.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.450+0000 7f0d8dd9b640 1 --2- 172.21.15.37:0/1213280360 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0d8400abd0 0x7f0d780407e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f0d8001ccb0 tx=0x7f0d800a7070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:08.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.450+0000 7f0d8f7fe640 1 -- 172.21.15.37:0/1213280360 <== mon.2 v2:172.21.15.37:3301/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f0d88045c20 con 0x7f0d78003870 2024-04-26T00:23:08.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.450+0000 7f0d8f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:08.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.450+0000 7f0d8f7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:23:08.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.458+0000 7f0d8f7fe640 1 -- 172.21.15.37:0/1213280360 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f0d88045fa0 con 0x7f0d78003870 2024-04-26T00:23:08.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.458+0000 7f0d8f7fe640 10 monclient: handle_mon_command_ack 1 not found 2024-04-26T00:23:08.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.678+0000 7f0d9cbe1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:23:08.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.678+0000 7f0d9cbe1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:23:08.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.678+0000 7f0d9cbe1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:08.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.678+0000 7f0d9cbe1640 1 -- 172.21.15.37:0/1213280360 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0d981358a0 con 0x7f0d78003870 2024-04-26T00:23:08.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.678+0000 7f0d8f7fe640 1 -- 172.21.15.37:0/1213280360 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f0d88026420 con 0x7f0d78003870 2024-04-26T00:23:08.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.678+0000 7f0d8f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:23:08.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.678+0000 7f0d8f7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:23:08.657 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":26,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:23:08.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.682+0000 7f0d9cbe1640 1 -- 172.21.15.37:0/1213280360 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0d8400abd0 msgr2=0x7f0d780407e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:08.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.682+0000 7f0d9cbe1640 1 --2- 172.21.15.37:0/1213280360 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0d8400abd0 0x7f0d780407e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f0d8001ccb0 tx=0x7f0d800a7070 comp rx=0 tx=0).stop 2024-04-26T00:23:08.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.682+0000 7f0d9cbe1640 10 monclient: shutdown 2024-04-26T00:23:08.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.682+0000 7f0d9cbe1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:23:08.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.682+0000 7f0d9cbe1640 1 -- 172.21.15.37:0/1213280360 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d78003870 msgr2=0x7f0d78005c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:08.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.682+0000 7f0d9cbe1640 1 --2- 172.21.15.37:0/1213280360 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d78003870 0x7f0d78005c60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0d8800ce90 tx=0x7f0d90009030 comp rx=0 tx=0).stop 2024-04-26T00:23:08.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.682+0000 7f0d9cbe1640 1 -- 172.21.15.37:0/1213280360 shutdown_connections 2024-04-26T00:23:08.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.682+0000 7f0d9cbe1640 1 --2- 172.21.15.37:0/1213280360 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0d8400abd0 0x7f0d780407e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:08.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.682+0000 7f0d9cbe1640 1 --2- 172.21.15.37:0/1213280360 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d84003eb0 0x7f0d7800b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:08.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.682+0000 7f0d9cbe1640 1 --2- 172.21.15.37:0/1213280360 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d8400a090 0x7f0d78008760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:08.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.682+0000 7f0d9cbe1640 1 --2- 172.21.15.37:0/1213280360 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d78003870 0x7f0d78005c60 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:08.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.682+0000 7f0d9cbe1640 1 -- 172.21.15.37:0/1213280360 >> 172.21.15.37:0/1213280360 conn(0x7f0d980a64e0 msgr2=0x7f0d9813ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:23:08.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.682+0000 7f0d9cbe1640 1 -- 172.21.15.37:0/1213280360 shutdown_connections 2024-04-26T00:23:08.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:08.682+0000 7f0d9cbe1640 1 -- 172.21.15.37:0/1213280360 wait complete. 2024-04-26T00:23:08.837 INFO:tasks.rados.rados.0.smithi045.stdout:581: done (6 left) 2024-04-26T00:23:08.837 INFO:tasks.rados.rados.0.smithi045.stdout:582: done (5 left) 2024-04-26T00:23:08.837 INFO:tasks.rados.rados.0.smithi045.stdout:583: done (4 left) 2024-04-26T00:23:08.837 INFO:tasks.rados.rados.0.smithi045.stdout:584: done (3 left) 2024-04-26T00:23:08.837 INFO:tasks.rados.rados.0.smithi045.stdout:585: done (2 left) 2024-04-26T00:23:08.837 INFO:tasks.rados.rados.0.smithi045.stdout:586: done (1 left) 2024-04-26T00:23:08.837 INFO:tasks.rados.rados.0.smithi045.stdout:587: done (0 left) 2024-04-26T00:23:08.837 INFO:tasks.rados.rados.0.smithi045.stdout:588: rollback oid 34 current snap is 64 2024-04-26T00:23:08.837 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 34 to 57 2024-04-26T00:23:08.837 INFO:tasks.rados.rados.0.smithi045.stdout:589: write oid 43 current snap is 64 2024-04-26T00:23:08.837 INFO:tasks.rados.rados.0.smithi045.stdout:589: seq_num 248 ranges {568136=785048,1929299=238837} 2024-04-26T00:23:08.852 INFO:tasks.rados.rados.0.smithi045.stdout:589: writing smithi04526579-43 from 568136 to 1353184 tid 1 2024-04-26T00:23:08.862 INFO:tasks.rados.rados.0.smithi045.stdout:589: writing smithi04526579-43 from 1929299 to 2168136 tid 2 2024-04-26T00:23:08.862 INFO:tasks.rados.rados.0.smithi045.stdout:590: write oid 20 current snap is 64 2024-04-26T00:23:08.862 INFO:tasks.rados.rados.0.smithi045.stdout:588: finishing rollback tid 0 to smithi04526579-34 2024-04-26T00:23:08.862 INFO:tasks.rados.rados.0.smithi045.stdout:588: finishing rollback tid 1 to smithi04526579-34 2024-04-26T00:23:08.862 INFO:tasks.rados.rados.0.smithi045.stdout:590: seq_num 249 ranges {584943=460297,1587781=597162} 2024-04-26T00:23:08.873 INFO:tasks.rados.rados.0.smithi045.stdout:590: writing smithi04526579-20 from 584943 to 1045240 tid 1 2024-04-26T00:23:08.886 INFO:tasks.rados.rados.0.smithi045.stdout:590: writing smithi04526579-20 from 1587781 to 2184943 tid 2 2024-04-26T00:23:08.886 INFO:tasks.rados.rados.0.smithi045.stdout:591: copy_from oid 28 from oid 41 current snap is 64 2024-04-26T00:23:08.887 INFO:tasks.rados.rados.0.smithi045.stdout:588: finishing rollback tid 2 to smithi04526579-34 2024-04-26T00:23:08.887 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 219 (ObjNum 173 snap 45 seq_num 173) dirty exists 2024-04-26T00:23:08.887 INFO:tasks.rados.rados.0.smithi045.stdout:588: done (3 left) 2024-04-26T00:23:08.887 INFO:tasks.rados.rados.0.smithi045.stdout:592: copy_from oid 32 from oid 22 current snap is 64 2024-04-26T00:23:08.887 INFO:tasks.rados.rados.0.smithi045.stdout:589: finishing write tid 1 to smithi04526579-43 2024-04-26T00:23:08.887 INFO:tasks.rados.rados.0.smithi045.stdout:589: finishing write tid 2 to smithi04526579-43 2024-04-26T00:23:08.888 INFO:tasks.rados.rados.0.smithi045.stdout:589: finishing write tid 3 to smithi04526579-43 2024-04-26T00:23:08.888 INFO:tasks.rados.rados.0.smithi045.stdout:589: finishing write tid 4 to smithi04526579-43 2024-04-26T00:23:08.888 INFO:tasks.rados.rados.0.smithi045.stdout:589: oid 43 updating version 0 to 238 2024-04-26T00:23:08.888 INFO:tasks.rados.rados.0.smithi045.stdout:589: oid 43 updating version 238 to 240 2024-04-26T00:23:08.888 INFO:tasks.rados.rados.0.smithi045.stdout:589: oid 43 version 240 is already newer than 239 2024-04-26T00:23:08.888 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 240 (ObjNum 248 snap 64 seq_num 248) dirty exists 2024-04-26T00:23:08.888 INFO:tasks.rados.rados.0.smithi045.stdout:589: left oid 43 (ObjNum 248 snap 64 seq_num 248) 2024-04-26T00:23:08.888 INFO:tasks.rados.rados.0.smithi045.stdout:589: done (3 left) 2024-04-26T00:23:08.888 INFO:tasks.rados.rados.0.smithi045.stdout:593: read oid 45 snap 57 2024-04-26T00:23:08.888 INFO:tasks.rados.rados.0.smithi045.stdout:593: expect deleted 2024-04-26T00:23:08.888 INFO:tasks.rados.rados.0.smithi045.stdout:594: copy_from oid 23 from oid 18 current snap is 64 2024-04-26T00:23:08.888 INFO:tasks.rados.rados.0.smithi045.stdout:595: read oid 4 snap 51 2024-04-26T00:23:08.889 INFO:tasks.rados.rados.0.smithi045.stdout:595: expect (ObjNum 195 snap 50 seq_num 195) 2024-04-26T00:23:08.889 INFO:tasks.rados.rados.0.smithi045.stdout:596: copy_from oid 12 from oid 14 current snap is 64 2024-04-26T00:23:08.889 INFO:tasks.rados.rados.0.smithi045.stdout:597: delete oid 47 current snap is 64 2024-04-26T00:23:08.897 INFO:tasks.rados.rados.0.smithi045.stdout:593: done (7 left) 2024-04-26T00:23:08.898 INFO:tasks.rados.rados.0.smithi045.stdout:597: done (6 left) 2024-04-26T00:23:08.898 INFO:tasks.rados.rados.0.smithi045.stdout:598: write oid 19 current snap is 64 2024-04-26T00:23:08.898 INFO:tasks.rados.rados.0.smithi045.stdout:598: seq_num 251 ranges {618557=727148,2104745=113812} 2024-04-26T00:23:08.915 INFO:tasks.rados.rados.0.smithi045.stdout:598: writing smithi04526579-19 from 618557 to 1345705 tid 1 2024-04-26T00:23:08.927 INFO:tasks.rados.rados.0.smithi045.stdout:598: writing smithi04526579-19 from 2104745 to 2218557 tid 2 2024-04-26T00:23:08.927 INFO:tasks.rados.rados.0.smithi045.stdout:599: snap_remove snap 50 2024-04-26T00:23:09.728 INFO:tasks.rados.rados.0.smithi045.stdout:599: done (7 left) 2024-04-26T00:23:09.728 INFO:tasks.rados.rados.0.smithi045.stdout:600: delete oid 48 current snap is 64 2024-04-26T00:23:09.728 INFO:tasks.rados.rados.0.smithi045.stdout:590: finishing write tid 1 to smithi04526579-20 2024-04-26T00:23:09.728 INFO:tasks.rados.rados.0.smithi045.stdout:590: finishing write tid 2 to smithi04526579-20 2024-04-26T00:23:09.729 INFO:tasks.rados.rados.0.smithi045.stdout:590: finishing write tid 3 to smithi04526579-20 2024-04-26T00:23:09.729 INFO:tasks.rados.rados.0.smithi045.stdout:590: finishing write tid 4 to smithi04526579-20 2024-04-26T00:23:09.729 INFO:tasks.rados.rados.0.smithi045.stdout:590: oid 20 updating version 0 to 217 2024-04-26T00:23:09.729 INFO:tasks.rados.rados.0.smithi045.stdout:590: oid 20 updating version 217 to 218 2024-04-26T00:23:09.729 INFO:tasks.rados.rados.0.smithi045.stdout:590: oid 20 updating version 218 to 219 2024-04-26T00:23:09.729 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 219 (ObjNum 249 snap 64 seq_num 249) dirty exists 2024-04-26T00:23:09.729 INFO:tasks.rados.rados.0.smithi045.stdout:590: left oid 20 (ObjNum 249 snap 64 seq_num 249) 2024-04-26T00:23:09.729 INFO:tasks.rados.rados.0.smithi045.stdout:592: finishing copy_from racing read to smithi04526579-32 2024-04-26T00:23:09.729 INFO:tasks.rados.rados.0.smithi045.stdout:592: finishing copy_from to smithi04526579-32 2024-04-26T00:23:09.729 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 134 (ObjNum 242 snap 61 seq_num 242) dirty exists 2024-04-26T00:23:09.729 INFO:tasks.rados.rados.0.smithi045.stdout:591: finishing copy_from racing read to smithi04526579-28 2024-04-26T00:23:09.729 INFO:tasks.rados.rados.0.smithi045.stdout:595: expect (ObjNum 195 snap 50 seq_num 195) 2024-04-26T00:23:09.934 INFO:tasks.rados.rados.0.smithi045.stdout:591: finishing copy_from to smithi04526579-28 2024-04-26T00:23:09.934 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 270 (ObjNum 224 snap 59 seq_num 224) dirty exists 2024-04-26T00:23:09.934 INFO:tasks.rados.rados.0.smithi045.stdout:596: finishing copy_from racing read to smithi04526579-12 2024-04-26T00:23:09.934 INFO:tasks.rados.rados.0.smithi045.stdout:590: done (7 left) 2024-04-26T00:23:09.934 INFO:tasks.rados.rados.0.smithi045.stdout:591: done (6 left) 2024-04-26T00:23:09.934 INFO:tasks.rados.rados.0.smithi045.stdout:592: done (5 left) 2024-04-26T00:23:09.934 INFO:tasks.rados.rados.0.smithi045.stdout:595: done (4 left) 2024-04-26T00:23:09.934 INFO:tasks.rados.rados.0.smithi045.stdout:600: done (3 left) 2024-04-26T00:23:09.934 INFO:tasks.rados.rados.0.smithi045.stdout:601: delete oid 15 current snap is 64 2024-04-26T00:23:09.935 INFO:tasks.rados.rados.0.smithi045.stdout:594: finishing copy_from racing read to smithi04526579-23 2024-04-26T00:23:09.935 INFO:tasks.rados.rados.0.smithi045.stdout:598: finishing write tid 1 to smithi04526579-19 2024-04-26T00:23:09.935 INFO:tasks.rados.rados.0.smithi045.stdout:598: finishing write tid 2 to smithi04526579-19 2024-04-26T00:23:09.935 INFO:tasks.rados.rados.0.smithi045.stdout:598: finishing write tid 3 to smithi04526579-19 2024-04-26T00:23:09.935 INFO:tasks.rados.rados.0.smithi045.stdout:598: finishing write tid 4 to smithi04526579-19 2024-04-26T00:23:09.935 INFO:tasks.rados.rados.0.smithi045.stdout:598: oid 19 updating version 0 to 209 2024-04-26T00:23:09.935 INFO:tasks.rados.rados.0.smithi045.stdout:598: oid 19 updating version 209 to 210 2024-04-26T00:23:09.935 INFO:tasks.rados.rados.0.smithi045.stdout:598: oid 19 version 210 is already newer than 208 2024-04-26T00:23:09.935 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 210 (ObjNum 251 snap 64 seq_num 251) dirty exists 2024-04-26T00:23:09.935 INFO:tasks.rados.rados.0.smithi045.stdout:598: left oid 19 (ObjNum 251 snap 64 seq_num 251) 2024-04-26T00:23:09.935 INFO:tasks.rados.rados.0.smithi045.stdout:594: finishing copy_from to smithi04526579-23 2024-04-26T00:23:09.935 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 175 (ObjNum 190 snap 49 seq_num 190) dirty exists 2024-04-26T00:23:09.935 INFO:tasks.rados.rados.0.smithi045.stdout:596: finishing copy_from to smithi04526579-12 2024-04-26T00:23:09.935 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 125 (ObjNum 237 snap 60 seq_num 237) dirty exists 2024-04-26T00:23:09.936 INFO:tasks.rados.rados.0.smithi045.stdout:594: done (3 left) 2024-04-26T00:23:09.936 INFO:tasks.rados.rados.0.smithi045.stdout:596: done (2 left) 2024-04-26T00:23:09.937 INFO:tasks.rados.rados.0.smithi045.stdout:598: done (1 left) 2024-04-26T00:23:09.937 INFO:tasks.rados.rados.0.smithi045.stdout:601: done (0 left) 2024-04-26T00:23:09.937 INFO:tasks.rados.rados.0.smithi045.stdout:602: rollback oid 15 current snap is 64 2024-04-26T00:23:09.937 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 15 to 37 2024-04-26T00:23:09.937 INFO:tasks.rados.rados.0.smithi045.stdout:603: write (excl) oid 26 current snap is 64 2024-04-26T00:23:09.937 INFO:tasks.rados.rados.0.smithi045.stdout:603: seq_num 254 ranges {668978=552895,1763838=505140} 2024-04-26T00:23:09.949 INFO:tasks.rados.rados.0.smithi045.stdout:603: writing smithi04526579-26 from 668978 to 1221873 tid 1 2024-04-26T00:23:09.961 INFO:tasks.rados.rados.0.smithi045.stdout:603: writing smithi04526579-26 from 1763838 to 2268978 tid 2 2024-04-26T00:23:09.961 INFO:tasks.rados.rados.0.smithi045.stdout:604: rollback oid 28 current snap is 64 2024-04-26T00:23:09.961 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 28 to 59 2024-04-26T00:23:09.962 INFO:tasks.rados.rados.0.smithi045.stdout:602: finishing rollback tid 1 to smithi04526579-15 2024-04-26T00:23:09.962 INFO:tasks.rados.rados.0.smithi045.stdout:602: finishing rollback tid 2 to smithi04526579-15 2024-04-26T00:23:09.962 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 283 (ObjNum 163 snap 37 seq_num 163) dirty exists 2024-04-26T00:23:09.962 INFO:tasks.rados.rados.0.smithi045.stdout:603: finishing write tid 1 to smithi04526579-26 2024-04-26T00:23:09.962 INFO:tasks.rados.rados.0.smithi045.stdout:602: done (2 left) 2024-04-26T00:23:09.962 INFO:tasks.rados.rados.0.smithi045.stdout:605: snap_remove snap 58 2024-04-26T00:23:10.848 INFO:tasks.rados.rados.0.smithi045.stdout:605: done (2 left) 2024-04-26T00:23:10.848 INFO:tasks.rados.rados.0.smithi045.stdout:606: snap_remove snap 57 2024-04-26T00:23:11.676 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:23:11.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.830+0000 7fa25409a640 10 monclient: get_monmap_and_config 2024-04-26T00:23:11.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.830+0000 7fa25409a640 10 monclient: build_initial_monmap 2024-04-26T00:23:11.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.830+0000 7fa25409a640 10 monclient: monmap: 2024-04-26T00:23:11.810 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:23:11.810 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:11.810 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:23:11.835328+0000 2024-04-26T00:23:11.810 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:23:11.835328+0000 2024-04-26T00:23:11.810 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:23:11.810 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:11.811 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:23:11.811 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:23:11.811 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:23:11.811 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:23:11.811 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:23:11.811 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:11.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa25409a640 1 Processor -- start 2024-04-26T00:23:11.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa25409a640 1 -- start start 2024-04-26T00:23:11.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa25409a640 10 monclient: init 2024-04-26T00:23:11.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa25409a640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:11.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa25409a640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-04-26T00:23:11.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa25409a640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa24c136da0 0x7fa24c09bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:11.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa25409a640 10 monclient(hunting): picked mon.noname-c con 0x7fa24c136da0 addr v2:172.21.15.45:3300/0 2024-04-26T00:23:11.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa25409a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa24c09c410 addr v1:172.21.15.45:6789/0 2024-04-26T00:23:11.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa25409a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa24c137890 0x7fa24c13e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:11.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa25409a640 10 monclient(hunting): picked mon.noname-b con 0x7fa24c137890 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:23:11.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa25409a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa24c09b4f0 con 0x7fa24c09c410 2024-04-26T00:23:11.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa25409a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:11.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa25409a640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fa24c09bb40 con 0x7fa24c136da0 2024-04-26T00:23:11.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa25409a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:11.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa25409a640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa24c09b210 con 0x7fa24c137890 2024-04-26T00:23:11.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa25409a640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:11.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa25409a640 10 monclient(hunting): authenticate will time out at 1173.141469s 2024-04-26T00:23:11.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa252637640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa24c137890 0x7fa24c13e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:11.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa252637640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa24c137890 0x7fa24c13e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:38896/0 (socket says 172.21.15.37:38896) 2024-04-26T00:23:11.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa252637640 1 -- 172.21.15.37:0/4039864784 learned_addr learned my addr 172.21.15.37:0/4039864784 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:23:11.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa252637640 10 monclient(hunting): get_auth_request con 0x7fa24c137890 auth_method 0 2024-04-26T00:23:11.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa252637640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:11.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa252637640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:11.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa252637640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:11.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa252637640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:11.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa252637640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:11.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa252637640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:11.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa251e36640 1 --2- 172.21.15.37:0/4039864784 >> v2:172.21.15.45:3300/0 conn(0x7fa24c136da0 0x7fa24c09bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:11.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa251e36640 10 monclient(hunting): get_auth_request con 0x7fa24c136da0 auth_method 0 2024-04-26T00:23:11.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa251e36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:11.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa251e36640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:11.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa251e36640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:11.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa251e36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:11.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa251e36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:11.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa251e36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:11.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa252637640 10 monclient(hunting): handle_auth_done global_id 34181 payload 274 2024-04-26T00:23:11.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa252637640 1 -- 172.21.15.37:0/4039864784 >> v2:172.21.15.45:3300/0 conn(0x7fa24c136da0 msgr2=0x7fa24c09bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:23:11.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa252637640 1 --2- 172.21.15.37:0/4039864784 >> v2:172.21.15.45:3300/0 conn(0x7fa24c136da0 0x7fa24c09bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:11.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa252637640 1 -- 172.21.15.37:0/4039864784 >> v1:172.21.15.45:6789/0 conn(0x7fa24c09c410 legacy=0x7fa24c137180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:23:11.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa252637640 10 monclient: _finish_hunting 0 2024-04-26T00:23:11.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa252637640 1 monclient: found mon.noname-b 2024-04-26T00:23:11.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa252637640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:11.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa252637640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:23:11.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.834+0000 7fa252637640 1 -- 172.21.15.37:0/4039864784 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa24c09b7d0 con 0x7fa24c137890 2024-04-26T00:23:11.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.838+0000 7fa252637640 1 -- 172.21.15.37:0/4039864784 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa24c137890 msgr2=0x7fa24c13e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:11.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.838+0000 7fa252637640 1 --2- 172.21.15.37:0/4039864784 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa24c137890 0x7fa24c13e2e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa244009e50 tx=0x7fa24401c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d2630be37fd5b2f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:11.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.838+0000 7fa250e34640 1 -- 172.21.15.37:0/4039864784 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2440a5750 con 0x7fa24c137890 2024-04-26T00:23:11.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.838+0000 7fa250e34640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:11.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.838+0000 7fa250e34640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:23:11.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.838+0000 7fa250e34640 10 monclient: dump: 2024-04-26T00:23:11.818 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:11.818 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:11.818 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:11.818 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:11.818 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:11.818 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:11.818 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:11.818 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:11.818 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:11.818 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:11.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.838+0000 7fa250e34640 10 monclient: _finish_auth 0 2024-04-26T00:23:11.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.838+0000 7fa250e34640 10 monclient: _check_auth_tickets 2024-04-26T00:23:11.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.838+0000 7fa250e34640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:11.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.838+0000 7fa250e34640 1 -- 172.21.15.37:0/4039864784 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2440a58f0 con 0x7fa24c137890 2024-04-26T00:23:11.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.838+0000 7fa25409a640 5 monclient: authenticate success, global_id 34181 2024-04-26T00:23:11.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.838+0000 7fa25409a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:23:11.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.838+0000 7fa250e34640 10 monclient: handle_config config(0 keys) 2024-04-26T00:23:11.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.838+0000 7fa25409a640 10 monclient: get_monmap_and_config success 2024-04-26T00:23:11.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.838+0000 7fa25409a640 4 set_mon_vals no callback set 2024-04-26T00:23:11.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.838+0000 7fa250e34640 1 -- 172.21.15.37:0/4039864784 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2440a5cc0 con 0x7fa24c137890 2024-04-26T00:23:11.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.838+0000 7fa250e34640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:11.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.838+0000 7fa250e34640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:23:11.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.838+0000 7fa250e34640 10 monclient: dump: 2024-04-26T00:23:11.819 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:11.819 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:11.819 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:11.819 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:11.820 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:11.820 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:11.820 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:11.820 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:11.820 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:11.820 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:11.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.838+0000 7fa25409a640 10 monclient: shutdown 2024-04-26T00:23:11.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.838+0000 7fa25409a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:23:11.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.838+0000 7fa25409a640 1 -- 172.21.15.37:0/4039864784 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa24c137890 msgr2=0x7fa24c13e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:11.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.838+0000 7fa25409a640 1 --2- 172.21.15.37:0/4039864784 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa24c137890 0x7fa24c13e2e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa244009e50 tx=0x7fa24401c8b0 comp rx=0 tx=0).stop 2024-04-26T00:23:11.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.838+0000 7fa25409a640 1 -- 172.21.15.37:0/4039864784 shutdown_connections 2024-04-26T00:23:11.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.838+0000 7fa25409a640 1 --2- 172.21.15.37:0/4039864784 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa24c137890 0x7fa24c13e2e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:11.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.838+0000 7fa25409a640 1 --2- 172.21.15.37:0/4039864784 >> v2:172.21.15.45:3300/0 conn(0x7fa24c136da0 0x7fa24c09bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:11.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.838+0000 7fa25409a640 1 -- 172.21.15.37:0/4039864784 >> 172.21.15.37:0/4039864784 conn(0x7fa24c0a5f00 msgr2=0x7fa24c0a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:23:11.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.838+0000 7fa25409a640 1 -- 172.21.15.37:0/4039864784 shutdown_connections 2024-04-26T00:23:11.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.838+0000 7fa25409a640 1 -- 172.21.15.37:0/4039864784 wait complete. 2024-04-26T00:23:11.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.838+0000 7fa25409a640 10 monclient: build_initial_monmap 2024-04-26T00:23:11.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.838+0000 7fa25409a640 10 monclient: monmap: 2024-04-26T00:23:11.825 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:23:11.825 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:23:11.825 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:23:11.825 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:23:11.825 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:23:11.825 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:23:11.825 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:23:11.825 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:23:11.825 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:23:11.825 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:11.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa25409a640 1 Processor -- start 2024-04-26T00:23:11.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa25409a640 1 -- start start 2024-04-26T00:23:11.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa25409a640 10 monclient: init 2024-04-26T00:23:11.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa25409a640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:11.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa25409a640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:23:11.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa25409a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa24c09c410 0x7fa24c13d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:11.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa25409a640 10 monclient(hunting): picked mon.noname-a con 0x7fa24c09c410 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:23:11.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa25409a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa24c136da0 0x7fa24c135c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:11.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa25409a640 10 monclient(hunting): picked mon.noname-b con 0x7fa24c136da0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:23:11.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa25409a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa24c137890 0x7fa24c1ce8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:11.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa25409a640 10 monclient(hunting): picked mon.noname-c con 0x7fa24c137890 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:23:11.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa25409a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:11.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa25409a640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa24c1414b0 con 0x7fa24c09c410 2024-04-26T00:23:11.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa25409a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:11.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa25409a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa24c141330 con 0x7fa24c136da0 2024-04-26T00:23:11.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa25409a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:11.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa25409a640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa24c141630 con 0x7fa24c137890 2024-04-26T00:23:11.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa25409a640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:11.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa25409a640 10 monclient(hunting): authenticate will time out at 1173.148879s 2024-04-26T00:23:11.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa252637640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa24c137890 0x7fa24c1ce8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:11.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa251e36640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa24c09c410 0x7fa24c13d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:11.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa252637640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa24c137890 0x7fa24c1ce8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:38906/0 (socket says 172.21.15.37:38906) 2024-04-26T00:23:11.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa252637640 1 -- 172.21.15.37:0/4222197514 learned_addr learned my addr 172.21.15.37:0/4222197514 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:23:11.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa252637640 10 monclient(hunting): get_auth_request con 0x7fa24c137890 auth_method 0 2024-04-26T00:23:11.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa252637640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:11.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa252637640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:11.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa252637640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:11.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa251e36640 10 monclient(hunting): get_auth_request con 0x7fa24c09c410 auth_method 0 2024-04-26T00:23:11.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa251e36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:11.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa251e36640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:11.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa251e36640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:11.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa252637640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:11.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa252637640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:11.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa252637640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:11.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa251e36640 1 -- 172.21.15.37:0/4222197514 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa24c09c410 msgr2=0x7fa24c13d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:23:11.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa251e36640 1 -- 172.21.15.37:0/4222197514 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa24c09c410 msgr2=0x7fa24c13d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:23:11.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa251e36640 1 --2- 172.21.15.37:0/4222197514 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa24c09c410 0x7fa24c13d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:23:11.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa251e36640 1 --2- 172.21.15.37:0/4222197514 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa24c09c410 0x7fa24c13d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:23:11.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa251635640 1 --2- 172.21.15.37:0/4222197514 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa24c136da0 0x7fa24c135c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:11.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa251635640 10 monclient(hunting): get_auth_request con 0x7fa24c136da0 auth_method 0 2024-04-26T00:23:11.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa251635640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:11.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa251635640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:11.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa251635640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:11.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa252637640 10 monclient(hunting): handle_auth_done global_id 34184 payload 824 2024-04-26T00:23:11.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa252637640 1 -- 172.21.15.37:0/4222197514 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa24c136da0 msgr2=0x7fa24c135c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:11.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa252637640 1 --2- 172.21.15.37:0/4222197514 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa24c136da0 0x7fa24c135c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:11.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa252637640 1 -- 172.21.15.37:0/4222197514 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa24c09c410 msgr2=0x7fa24c13d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:23:11.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa252637640 1 --2- 172.21.15.37:0/4222197514 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa24c09c410 0x7fa24c13d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:11.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa252637640 10 monclient: _finish_hunting 0 2024-04-26T00:23:11.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa252637640 1 monclient: found mon.noname-c 2024-04-26T00:23:11.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa252637640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:11.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa252637640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:23:11.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa252637640 1 -- 172.21.15.37:0/4222197514 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa24c13df30 con 0x7fa24c137890 2024-04-26T00:23:11.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa252637640 1 -- 172.21.15.37:0/4222197514 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa24c137890 msgr2=0x7fa24c1ce8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:11.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa252637640 1 --2- 172.21.15.37:0/4222197514 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa24c137890 0x7fa24c1ce8b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa2440af040 tx=0x7fa244019650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:11.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa2427fc640 1 -- 172.21.15.37:0/4222197514 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2440bc020 con 0x7fa24c137890 2024-04-26T00:23:11.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa2427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:11.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa2427fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:23:11.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa2427fc640 10 monclient: dump: 2024-04-26T00:23:11.829 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:11.829 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:11.830 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:11.830 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:11.830 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:11.830 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:11.830 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:11.830 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:11.830 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:11.830 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:11.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa2427fc640 10 monclient: _finish_auth 0 2024-04-26T00:23:11.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa2427fc640 10 monclient: _check_auth_tickets 2024-04-26T00:23:11.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa2427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:11.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa2427fc640 1 -- 172.21.15.37:0/4222197514 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2440b8a90 con 0x7fa24c137890 2024-04-26T00:23:11.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa25409a640 5 monclient: authenticate success, global_id 34184 2024-04-26T00:23:11.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa25409a640 10 monclient: _renew_subs 2024-04-26T00:23:11.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa25409a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:11.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa25409a640 1 -- 172.21.15.37:0/4222197514 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa24c1202b0 con 0x7fa24c137890 2024-04-26T00:23:11.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa2427fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:23:11.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa2427fc640 1 -- 172.21.15.37:0/4222197514 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2440a5750 con 0x7fa24c137890 2024-04-26T00:23:11.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa2427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:11.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa2427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:23:11.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa2427fc640 10 monclient: dump: 2024-04-26T00:23:11.831 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:11.831 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:11.831 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:11.831 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:11.831 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:11.831 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:11.831 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:11.832 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:11.832 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:11.832 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:11.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa25409a640 10 monclient: _renew_subs 2024-04-26T00:23:11.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa243fff640 4 set_mon_vals no callback set 2024-04-26T00:23:11.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.842+0000 7fa25409a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:11.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.846+0000 7fa25409a640 1 -- 172.21.15.37:0/4222197514 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa24c13bf80 con 0x7fa24c137890 2024-04-26T00:23:11.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.846+0000 7fa25409a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:11.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.846+0000 7fa25409a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:11.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.846+0000 7fa25409a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:11.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.846+0000 7fa25409a640 1 -- 172.21.15.37:0/4222197514 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa24c11fa50 con 0x7fa24c137890 2024-04-26T00:23:11.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.846+0000 7fa2427fc640 1 -- 172.21.15.37:0/4222197514 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa2440bf020 con 0x7fa24c137890 2024-04-26T00:23:11.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.846+0000 7fa2427fc640 1 --2- 172.21.15.37:0/4222197514 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa230035e80 0x7fa230038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:11.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.846+0000 7fa2427fc640 1 -- 172.21.15.37:0/4222197514 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(136..136 src has 1..136) ==== 3959+0+0 (secure 0 0 0) 0x7fa2440e3e90 con 0x7fa24c137890 2024-04-26T00:23:11.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.846+0000 7fa251e36640 1 --2- 172.21.15.37:0/4222197514 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa230035e80 0x7fa230038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:11.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.846+0000 7fa251e36640 10 monclient: get_auth_request con 0x7fa230035e80 auth_method 0 2024-04-26T00:23:11.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.850+0000 7fa251e36640 1 --2- 172.21.15.37:0/4222197514 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa230035e80 0x7fa230038340 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa248009520 tx=0x7fa248008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:11.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.850+0000 7fa2427fc640 1 -- 172.21.15.37:0/4222197514 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa2440c6030 con 0x7fa24c137890 2024-04-26T00:23:11.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.850+0000 7fa2427fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:11.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:11.850+0000 7fa2427fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:23:11.895 INFO:tasks.rados.rados.0.smithi045.stdout:606: done (2 left) 2024-04-26T00:23:11.895 INFO:tasks.rados.rados.0.smithi045.stdout:607: snap_create 2024-04-26T00:23:11.895 INFO:tasks.rados.rados.0.smithi045.stdout:603: finishing write tid 2 to smithi04526579-26 2024-04-26T00:23:11.895 INFO:tasks.rados.rados.0.smithi045.stdout:603: finishing write tid 3 to smithi04526579-26 2024-04-26T00:23:11.895 INFO:tasks.rados.rados.0.smithi045.stdout:603: finishing write tid 4 to smithi04526579-26 2024-04-26T00:23:11.895 INFO:tasks.rados.rados.0.smithi045.stdout:603: oid 26 updating version 0 to 285 2024-04-26T00:23:11.895 INFO:tasks.rados.rados.0.smithi045.stdout:603: oid 26 updating version 285 to 286 2024-04-26T00:23:11.895 INFO:tasks.rados.rados.0.smithi045.stdout:603: oid 26 updating version 286 to 287 2024-04-26T00:23:11.895 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 287 (ObjNum 254 snap 64 seq_num 254) dirty exists 2024-04-26T00:23:11.895 INFO:tasks.rados.rados.0.smithi045.stdout:603: left oid 26 (ObjNum 254 snap 64 seq_num 254) 2024-04-26T00:23:11.896 INFO:tasks.rados.rados.0.smithi045.stdout:604: finishing rollback tid 0 to smithi04526579-28 2024-04-26T00:23:11.896 INFO:tasks.rados.rados.0.smithi045.stdout:604: finishing rollback tid 1 to smithi04526579-28 2024-04-26T00:23:11.896 INFO:tasks.rados.rados.0.smithi045.stdout:604: finishing rollback tid 2 to smithi04526579-28 2024-04-26T00:23:11.896 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 290 (ObjNum 180 snap 47 seq_num 180) dirty exists 2024-04-26T00:23:12.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:12.090+0000 7fa25409a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:23:12.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:12.090+0000 7fa25409a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:23:12.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:12.090+0000 7fa25409a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:12.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:12.090+0000 7fa25409a640 1 -- 172.21.15.37:0/4222197514 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa24c066f40 con 0x7fa24c137890 2024-04-26T00:23:12.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:12.094+0000 7fa2427fc640 1 -- 172.21.15.37:0/4222197514 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7fa24c13df30 con 0x7fa24c137890 2024-04-26T00:23:12.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:12.094+0000 7fa2427fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:23:12.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:12.094+0000 7fa2427fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:23:12.071 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":26,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:23:12.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:12.094+0000 7fa25409a640 1 -- 172.21.15.37:0/4222197514 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa230035e80 msgr2=0x7fa230038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:12.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:12.094+0000 7fa25409a640 1 --2- 172.21.15.37:0/4222197514 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa230035e80 0x7fa230038340 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa248009520 tx=0x7fa248008040 comp rx=0 tx=0).stop 2024-04-26T00:23:12.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:12.094+0000 7fa25409a640 10 monclient: shutdown 2024-04-26T00:23:12.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:12.094+0000 7fa25409a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:23:12.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:12.094+0000 7fa25409a640 1 -- 172.21.15.37:0/4222197514 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa24c137890 msgr2=0x7fa24c1ce8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:12.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:12.094+0000 7fa25409a640 1 --2- 172.21.15.37:0/4222197514 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa24c137890 0x7fa24c1ce8b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa2440af040 tx=0x7fa244019650 comp rx=0 tx=0).stop 2024-04-26T00:23:12.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:12.094+0000 7fa25409a640 1 -- 172.21.15.37:0/4222197514 shutdown_connections 2024-04-26T00:23:12.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:12.094+0000 7fa25409a640 1 --2- 172.21.15.37:0/4222197514 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa230035e80 0x7fa230038340 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:12.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:12.094+0000 7fa25409a640 1 --2- 172.21.15.37:0/4222197514 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa24c137890 0x7fa24c1ce8b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:12.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:12.094+0000 7fa25409a640 1 --2- 172.21.15.37:0/4222197514 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa24c136da0 0x7fa24c135c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:12.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:12.094+0000 7fa25409a640 1 --2- 172.21.15.37:0/4222197514 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa24c09c410 0x7fa24c13d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:12.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:12.094+0000 7fa25409a640 1 -- 172.21.15.37:0/4222197514 >> 172.21.15.37:0/4222197514 conn(0x7fa24c0a5f00 msgr2=0x7fa24c0a7b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:23:12.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:12.094+0000 7fa25409a640 1 -- 172.21.15.37:0/4222197514 shutdown_connections 2024-04-26T00:23:12.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:12.094+0000 7fa25409a640 1 -- 172.21.15.37:0/4222197514 wait complete. 2024-04-26T00:23:12.900 INFO:tasks.rados.rados.0.smithi045.stdout:603: done (2 left) 2024-04-26T00:23:12.900 INFO:tasks.rados.rados.0.smithi045.stdout:604: done (1 left) 2024-04-26T00:23:12.900 INFO:tasks.rados.rados.0.smithi045.stdout:607: done (0 left) 2024-04-26T00:23:12.900 INFO:tasks.rados.rados.0.smithi045.stdout:608: snap_remove snap 64 2024-04-26T00:23:13.910 INFO:tasks.rados.rados.0.smithi045.stdout:608: done (0 left) 2024-04-26T00:23:13.910 INFO:tasks.rados.rados.0.smithi045.stdout:609: delete oid 5 current snap is 65 2024-04-26T00:23:13.927 INFO:tasks.rados.rados.0.smithi045.stdout:609: done (0 left) 2024-04-26T00:23:13.927 INFO:tasks.rados.rados.0.smithi045.stdout:610: snap_create 2024-04-26T00:23:14.727 INFO:tasks.rados.rados.0.smithi045.stdout:610: done (0 left) 2024-04-26T00:23:14.727 INFO:tasks.rados.rados.0.smithi045.stdout:611: write oid 49 current snap is 66 2024-04-26T00:23:14.727 INFO:tasks.rados.rados.0.smithi045.stdout:611: seq_num 256 ranges {702592=703393,1880802=421790} 2024-04-26T00:23:14.749 INFO:tasks.rados.rados.0.smithi045.stdout:611: writing smithi04526579-49 from 702592 to 1405985 tid 1 2024-04-26T00:23:14.765 INFO:tasks.rados.rados.0.smithi045.stdout:611: writing smithi04526579-49 from 1880802 to 2302592 tid 2 2024-04-26T00:23:14.765 INFO:tasks.rados.rados.0.smithi045.stdout:612: snap_create 2024-04-26T00:23:14.765 INFO:tasks.rados.rados.0.smithi045.stdout:611: finishing write tid 1 to smithi04526579-49 2024-04-26T00:23:14.773 INFO:tasks.rados.rados.0.smithi045.stdout:611: finishing write tid 2 to smithi04526579-49 2024-04-26T00:23:14.773 INFO:tasks.rados.rados.0.smithi045.stdout:611: finishing write tid 3 to smithi04526579-49 2024-04-26T00:23:14.773 INFO:tasks.rados.rados.0.smithi045.stdout:611: finishing write tid 4 to smithi04526579-49 2024-04-26T00:23:14.773 INFO:tasks.rados.rados.0.smithi045.stdout:611: oid 49 updating version 0 to 253 2024-04-26T00:23:14.773 INFO:tasks.rados.rados.0.smithi045.stdout:611: oid 49 version 253 is already newer than 251 2024-04-26T00:23:14.774 INFO:tasks.rados.rados.0.smithi045.stdout:611: oid 49 version 253 is already newer than 252 2024-04-26T00:23:14.774 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 253 (ObjNum 256 snap 66 seq_num 256) dirty exists 2024-04-26T00:23:14.774 INFO:tasks.rados.rados.0.smithi045.stdout:611: left oid 49 (ObjNum 256 snap 66 seq_num 256) 2024-04-26T00:23:15.090 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:23:15.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.242+0000 7f13a88cf640 10 monclient: get_monmap_and_config 2024-04-26T00:23:15.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.242+0000 7f13a88cf640 10 monclient: build_initial_monmap 2024-04-26T00:23:15.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.242+0000 7f13a88cf640 10 monclient: monmap: 2024-04-26T00:23:15.220 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:23:15.220 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:15.220 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:23:15.245832+0000 2024-04-26T00:23:15.221 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:23:15.245832+0000 2024-04-26T00:23:15.221 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:23:15.221 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:15.221 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:23:15.221 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:23:15.221 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:23:15.221 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:23:15.221 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:23:15.221 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:15.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.242+0000 7f13a88cf640 1 Processor -- start 2024-04-26T00:23:15.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.242+0000 7f13a88cf640 1 -- start start 2024-04-26T00:23:15.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.242+0000 7f13a88cf640 10 monclient: init 2024-04-26T00:23:15.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.242+0000 7f13a88cf640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:15.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.242+0000 7f13a88cf640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-04-26T00:23:15.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.242+0000 7f13a88cf640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f13a4133b20 0x7f13a409e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:15.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.242+0000 7f13a88cf640 10 monclient(hunting): picked mon.noname-c con 0x7f13a4133b20 addr v2:172.21.15.45:3300/0 2024-04-26T00:23:15.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.242+0000 7f13a88cf640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13a4133f00 0x7f13a4134350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:15.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.242+0000 7f13a88cf640 10 monclient(hunting): picked mon.noname-b con 0x7f13a4133f00 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:23:15.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.242+0000 7f13a88cf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f13a4130770 addr v1:172.21.15.37:6789/0 2024-04-26T00:23:15.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.242+0000 7f13a88cf640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13a409fe20 con 0x7f13a4130770 2024-04-26T00:23:15.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.242+0000 7f13a88cf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:15.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.242+0000 7f13a88cf640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f13a4072d00 con 0x7f13a4133b20 2024-04-26T00:23:15.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.242+0000 7f13a88cf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:15.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.242+0000 7f13a88cf640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f13a409ba60 con 0x7f13a4133f00 2024-04-26T00:23:15.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.242+0000 7f13a88cf640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:15.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.242+0000 7f13a88cf640 10 monclient(hunting): authenticate will time out at 1176.551500s 2024-04-26T00:23:15.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.242+0000 7f13a1d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13a4133f00 0x7f13a4134350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:15.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.242+0000 7f13a2d9d640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f13a4130770 0x7f13a4130be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:42420/0 (socket says 172.21.15.37:42420) 2024-04-26T00:23:15.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.242+0000 7f13a2d9d640 1 -- 172.21.15.37:0/1988509149 learned_addr learned my addr 172.21.15.37:0/1988509149 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:23:15.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.242+0000 7f13a1d9b640 10 monclient(hunting): get_auth_request con 0x7f13a4133f00 auth_method 0 2024-04-26T00:23:15.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.242+0000 7f13a1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:15.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.242+0000 7f13a1d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:15.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.242+0000 7f13a1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:15.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.242+0000 7f13a259c640 1 --2- 172.21.15.37:0/1988509149 >> v2:172.21.15.45:3300/0 conn(0x7f13a4133b20 0x7f13a409e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:15.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.242+0000 7f13a1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:15.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.242+0000 7f13a1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:15.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.242+0000 7f13a1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:15.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.242+0000 7f13a259c640 10 monclient(hunting): get_auth_request con 0x7f13a4133b20 auth_method 0 2024-04-26T00:23:15.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.242+0000 7f13a259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:15.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.242+0000 7f13a259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:15.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.242+0000 7f13a259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:15.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.246+0000 7f13a259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:15.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.246+0000 7f13a259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:15.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.246+0000 7f13a259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:15.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.246+0000 7f13a1d9b640 10 monclient(hunting): handle_auth_done global_id 34187 payload 274 2024-04-26T00:23:15.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.246+0000 7f13a1d9b640 1 -- 172.21.15.37:0/1988509149 >> v2:172.21.15.45:3300/0 conn(0x7f13a4133b20 msgr2=0x7f13a409e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:23:15.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.246+0000 7f13a1d9b640 1 --2- 172.21.15.37:0/1988509149 >> v2:172.21.15.45:3300/0 conn(0x7f13a4133b20 0x7f13a409e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:15.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.246+0000 7f13a1d9b640 1 -- 172.21.15.37:0/1988509149 >> v1:172.21.15.37:6789/0 conn(0x7f13a4130770 legacy=0x7f13a4130be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:23:15.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.246+0000 7f13a1d9b640 10 monclient: _finish_hunting 0 2024-04-26T00:23:15.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.246+0000 7f13a1d9b640 1 monclient: found mon.noname-b 2024-04-26T00:23:15.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.246+0000 7f13a1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:15.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.246+0000 7f13a1d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:23:15.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.246+0000 7f13a1d9b640 1 -- 172.21.15.37:0/1988509149 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13a409dd10 con 0x7f13a4133f00 2024-04-26T00:23:15.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.250+0000 7f13a1d9b640 1 -- 172.21.15.37:0/1988509149 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13a4133f00 msgr2=0x7f13a4134350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:15.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.250+0000 7f13a1d9b640 1 --2- 172.21.15.37:0/1988509149 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13a4133f00 0x7f13a4134350 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f13980097c0 tx=0x7f139801c110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d95e263502c799c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:15.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.250+0000 7f13a159a640 1 -- 172.21.15.37:0/1988509149 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13980b0070 con 0x7f13a4133f00 2024-04-26T00:23:15.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.250+0000 7f13a159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:15.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.250+0000 7f13a159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:23:15.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.250+0000 7f13a159a640 10 monclient: dump: 2024-04-26T00:23:15.228 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:15.228 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:15.228 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:15.228 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:15.228 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:15.228 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:15.228 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:15.228 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:15.228 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:15.228 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:15.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.250+0000 7f13a159a640 10 monclient: _finish_auth 0 2024-04-26T00:23:15.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.250+0000 7f13a159a640 10 monclient: _check_auth_tickets 2024-04-26T00:23:15.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.250+0000 7f13a159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:15.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.250+0000 7f13a159a640 1 -- 172.21.15.37:0/1988509149 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13980ab600 con 0x7f13a4133f00 2024-04-26T00:23:15.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.250+0000 7f13a159a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:23:15.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.250+0000 7f13a159a640 1 -- 172.21.15.37:0/1988509149 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13980ab9d0 con 0x7f13a4133f00 2024-04-26T00:23:15.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.250+0000 7f13a159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:15.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.250+0000 7f13a159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:23:15.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.250+0000 7f13a159a640 10 monclient: dump: 2024-04-26T00:23:15.229 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:15.229 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:15.229 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:15.229 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:15.229 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:15.229 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:15.229 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:15.229 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:15.229 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:15.230 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:15.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.250+0000 7f13a88cf640 5 monclient: authenticate success, global_id 34187 2024-04-26T00:23:15.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.250+0000 7f13a88cf640 10 monclient: get_monmap_and_config success 2024-04-26T00:23:15.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.250+0000 7f13a88cf640 4 set_mon_vals no callback set 2024-04-26T00:23:15.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.250+0000 7f13a88cf640 10 monclient: shutdown 2024-04-26T00:23:15.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.250+0000 7f13a88cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:23:15.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.250+0000 7f13a88cf640 1 -- 172.21.15.37:0/1988509149 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13a4133f00 msgr2=0x7f13a4134350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:15.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.250+0000 7f13a88cf640 1 --2- 172.21.15.37:0/1988509149 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13a4133f00 0x7f13a4134350 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f13980097c0 tx=0x7f139801c110 comp rx=0 tx=0).stop 2024-04-26T00:23:15.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.250+0000 7f13a88cf640 1 -- 172.21.15.37:0/1988509149 shutdown_connections 2024-04-26T00:23:15.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.250+0000 7f13a88cf640 1 --2- 172.21.15.37:0/1988509149 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13a4133f00 0x7f13a4134350 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:15.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.250+0000 7f13a88cf640 1 --2- 172.21.15.37:0/1988509149 >> v2:172.21.15.45:3300/0 conn(0x7f13a4133b20 0x7f13a409e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:15.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.250+0000 7f13a88cf640 1 -- 172.21.15.37:0/1988509149 >> 172.21.15.37:0/1988509149 conn(0x7f13a40a1ad0 msgr2=0x7f13a40a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:23:15.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.250+0000 7f13a88cf640 1 -- 172.21.15.37:0/1988509149 shutdown_connections 2024-04-26T00:23:15.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.250+0000 7f13a88cf640 1 -- 172.21.15.37:0/1988509149 wait complete. 2024-04-26T00:23:15.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.250+0000 7f13a88cf640 10 monclient: build_initial_monmap 2024-04-26T00:23:15.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.250+0000 7f13a88cf640 10 monclient: monmap: 2024-04-26T00:23:15.235 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:23:15.235 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:23:15.235 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:23:15.235 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:23:15.235 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:23:15.235 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:23:15.235 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:23:15.235 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:23:15.235 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:23:15.235 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:15.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.250+0000 7f13a88cf640 1 Processor -- start 2024-04-26T00:23:15.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.250+0000 7f13a88cf640 1 -- start start 2024-04-26T00:23:15.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.250+0000 7f13a88cf640 10 monclient: init 2024-04-26T00:23:15.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.250+0000 7f13a88cf640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:15.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.250+0000 7f13a88cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:23:15.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a88cf640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13a4130770 0x7f13a41cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:15.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a88cf640 10 monclient(hunting): picked mon.noname-c con 0x7f13a4130770 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:23:15.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a88cf640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f13a4133b20 0x7f13a413ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:15.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a88cf640 10 monclient(hunting): picked mon.noname-a con 0x7f13a4133b20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:23:15.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a88cf640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13a4133f00 0x7f13a413b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:15.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a88cf640 10 monclient(hunting): picked mon.noname-b con 0x7f13a4133f00 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:23:15.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a88cf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:15.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a88cf640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f13a40a1750 con 0x7f13a4133b20 2024-04-26T00:23:15.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a88cf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:15.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a88cf640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f13a40a15d0 con 0x7f13a4133f00 2024-04-26T00:23:15.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a88cf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:15.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a88cf640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f13a413e890 con 0x7f13a4130770 2024-04-26T00:23:15.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a88cf640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:15.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a88cf640 10 monclient(hunting): authenticate will time out at 1176.560091s 2024-04-26T00:23:15.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a259c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13a4130770 0x7f13a41cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:15.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a1d9b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f13a4133b20 0x7f13a413ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:15.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a259c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13a4130770 0x7f13a41cccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:38934/0 (socket says 172.21.15.37:38934) 2024-04-26T00:23:15.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a259c640 1 -- 172.21.15.37:0/3055466061 learned_addr learned my addr 172.21.15.37:0/3055466061 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:23:15.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a259c640 10 monclient(hunting): get_auth_request con 0x7f13a4130770 auth_method 0 2024-04-26T00:23:15.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:15.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:15.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:15.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a1d9b640 10 monclient(hunting): get_auth_request con 0x7f13a4133b20 auth_method 0 2024-04-26T00:23:15.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:15.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a1d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:15.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:15.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:15.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:15.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:15.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a1d9b640 1 -- 172.21.15.37:0/3055466061 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f13a4133b20 msgr2=0x7f13a413ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:23:15.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a1d9b640 1 -- 172.21.15.37:0/3055466061 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f13a4133b20 msgr2=0x7f13a413ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:23:15.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a1d9b640 1 --2- 172.21.15.37:0/3055466061 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f13a4133b20 0x7f13a413ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:23:15.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a1d9b640 1 --2- 172.21.15.37:0/3055466061 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f13a4133b20 0x7f13a413ae00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:23:15.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a2d9d640 1 --2- 172.21.15.37:0/3055466061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13a4133f00 0x7f13a413b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:15.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a2d9d640 10 monclient(hunting): get_auth_request con 0x7f13a4133f00 auth_method 0 2024-04-26T00:23:15.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:15.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a2d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:15.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:15.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a259c640 10 monclient(hunting): handle_auth_done global_id 34190 payload 824 2024-04-26T00:23:15.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a259c640 1 -- 172.21.15.37:0/3055466061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13a4133f00 msgr2=0x7f13a413b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:15.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a259c640 1 --2- 172.21.15.37:0/3055466061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13a4133f00 0x7f13a413b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:15.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a259c640 1 -- 172.21.15.37:0/3055466061 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f13a4133b20 msgr2=0x7f13a413ae00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:23:15.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a259c640 1 --2- 172.21.15.37:0/3055466061 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f13a4133b20 0x7f13a413ae00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:15.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a259c640 10 monclient: _finish_hunting 0 2024-04-26T00:23:15.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a259c640 1 monclient: found mon.noname-c 2024-04-26T00:23:15.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:15.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:23:15.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a259c640 1 -- 172.21.15.37:0/3055466061 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13a41cd380 con 0x7f13a4130770 2024-04-26T00:23:15.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a259c640 1 -- 172.21.15.37:0/3055466061 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13a4130770 msgr2=0x7f13a41cccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:15.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a259c640 1 --2- 172.21.15.37:0/3055466061 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13a4130770 0x7f13a41cccd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f139400e930 tx=0x7f139400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:15.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f1392ffd640 1 -- 172.21.15.37:0/3055466061 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1394017070 con 0x7f13a4130770 2024-04-26T00:23:15.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f1392ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:15.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f1392ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:23:15.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f1392ffd640 10 monclient: dump: 2024-04-26T00:23:15.240 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:15.240 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:15.240 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:15.240 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:15.240 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:15.240 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:15.240 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:15.240 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:15.240 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:15.240 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:15.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f1392ffd640 10 monclient: _finish_auth 0 2024-04-26T00:23:15.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f1392ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:23:15.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f1392ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:15.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f1392ffd640 1 -- 172.21.15.37:0/3055466061 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f139400cc10 con 0x7f13a4130770 2024-04-26T00:23:15.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a88cf640 5 monclient: authenticate success, global_id 34190 2024-04-26T00:23:15.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a88cf640 10 monclient: _renew_subs 2024-04-26T00:23:15.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a88cf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:15.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a88cf640 1 -- 172.21.15.37:0/3055466061 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13a409cc30 con 0x7f13a4130770 2024-04-26T00:23:15.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f1392ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:23:15.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f1392ffd640 1 -- 172.21.15.37:0/3055466061 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1394013470 con 0x7f13a4130770 2024-04-26T00:23:15.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f1392ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:15.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f1392ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:23:15.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f1392ffd640 10 monclient: dump: 2024-04-26T00:23:15.241 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:15.241 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:15.241 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:15.241 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:15.242 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:15.242 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:15.242 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:15.242 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:15.242 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:15.242 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:15.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a0d99640 4 set_mon_vals no callback set 2024-04-26T00:23:15.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a88cf640 10 monclient: _renew_subs 2024-04-26T00:23:15.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a88cf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:15.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.254+0000 7f13a88cf640 1 -- 172.21.15.37:0/3055466061 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f13a4138e00 con 0x7f13a4130770 2024-04-26T00:23:15.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.258+0000 7f1392ffd640 1 -- 172.21.15.37:0/3055466061 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f139401a070 con 0x7f13a4130770 2024-04-26T00:23:15.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.258+0000 7f13a88cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:15.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.258+0000 7f13a88cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:15.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.258+0000 7f13a88cf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:15.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.258+0000 7f13a88cf640 1 -- 172.21.15.37:0/3055466061 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13a4066f40 con 0x7f13a4130770 2024-04-26T00:23:15.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.258+0000 7f1392ffd640 1 --2- 172.21.15.37:0/3055466061 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1384035eb0 0x7f1384038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:15.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.258+0000 7f1392ffd640 1 -- 172.21.15.37:0/3055466061 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(140..140 src has 1..140) ==== 3959+0+0 (secure 0 0 0) 0x7f1394048c50 con 0x7f13a4130770 2024-04-26T00:23:15.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.258+0000 7f13a1d9b640 1 --2- 172.21.15.37:0/3055466061 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1384035eb0 0x7f1384038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:15.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.258+0000 7f13a1d9b640 10 monclient: get_auth_request con 0x7f1384035eb0 auth_method 0 2024-04-26T00:23:15.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.258+0000 7f13a1d9b640 1 --2- 172.21.15.37:0/3055466061 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1384035eb0 0x7f1384038370 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f13980ab000 tx=0x7f13980a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:15.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.262+0000 7f1392ffd640 1 -- 172.21.15.37:0/3055466061 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f13a41cd380 con 0x7f13a4130770 2024-04-26T00:23:15.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.262+0000 7f1392ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:15.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.262+0000 7f1392ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:23:15.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.486+0000 7f13a88cf640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:23:15.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.486+0000 7f13a88cf640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:23:15.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.486+0000 7f13a88cf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:15.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.486+0000 7f13a88cf640 1 -- 172.21.15.37:0/3055466061 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f13a4138130 con 0x7f13a4130770 2024-04-26T00:23:15.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.486+0000 7f1392ffd640 1 -- 172.21.15.37:0/3055466061 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f1394013e10 con 0x7f13a4130770 2024-04-26T00:23:15.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.486+0000 7f1392ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:23:15.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.486+0000 7f1392ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:23:15.465 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:23:15.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.490+0000 7f13a88cf640 1 -- 172.21.15.37:0/3055466061 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1384035eb0 msgr2=0x7f1384038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:15.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.490+0000 7f13a88cf640 1 --2- 172.21.15.37:0/3055466061 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1384035eb0 0x7f1384038370 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f13980ab000 tx=0x7f13980a6000 comp rx=0 tx=0).stop 2024-04-26T00:23:15.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.490+0000 7f13a88cf640 10 monclient: shutdown 2024-04-26T00:23:15.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.490+0000 7f13a88cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:23:15.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.490+0000 7f13a88cf640 1 -- 172.21.15.37:0/3055466061 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13a4130770 msgr2=0x7f13a41cccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:15.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.490+0000 7f13a88cf640 1 --2- 172.21.15.37:0/3055466061 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13a4130770 0x7f13a41cccd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f139400e930 tx=0x7f139400edf0 comp rx=0 tx=0).stop 2024-04-26T00:23:15.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.490+0000 7f13a88cf640 1 -- 172.21.15.37:0/3055466061 shutdown_connections 2024-04-26T00:23:15.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.490+0000 7f13a88cf640 1 --2- 172.21.15.37:0/3055466061 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1384035eb0 0x7f1384038370 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:15.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.490+0000 7f13a88cf640 1 --2- 172.21.15.37:0/3055466061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f13a4133f00 0x7f13a413b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:15.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.490+0000 7f13a88cf640 1 --2- 172.21.15.37:0/3055466061 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f13a4133b20 0x7f13a413ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:15.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.490+0000 7f13a88cf640 1 --2- 172.21.15.37:0/3055466061 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13a4130770 0x7f13a41cccd0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:15.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.490+0000 7f13a88cf640 1 -- 172.21.15.37:0/3055466061 >> 172.21.15.37:0/3055466061 conn(0x7f13a40a1ad0 msgr2=0x7f13a40a02e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:23:15.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.490+0000 7f13a88cf640 1 -- 172.21.15.37:0/3055466061 shutdown_connections 2024-04-26T00:23:15.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.490+0000 7f13a88cf640 1 -- 172.21.15.37:0/3055466061 wait complete. 2024-04-26T00:23:15.483 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:23:15.484 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:23:15.484 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:23:15.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.630+0000 7f3182431640 10 monclient: get_monmap_and_config 2024-04-26T00:23:15.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.630+0000 7f3182431640 10 monclient: build_initial_monmap 2024-04-26T00:23:15.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.630+0000 7f3182431640 10 monclient: monmap: 2024-04-26T00:23:15.610 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:23:15.610 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:15.610 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:23:15.635677+0000 2024-04-26T00:23:15.610 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:23:15.635677+0000 2024-04-26T00:23:15.610 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:23:15.611 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:15.611 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:23:15.611 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:23:15.611 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:23:15.611 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:23:15.611 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:23:15.611 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:15.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f3182431640 1 Processor -- start 2024-04-26T00:23:15.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f3182431640 1 -- start start 2024-04-26T00:23:15.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f3182431640 10 monclient: init 2024-04-26T00:23:15.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f3182431640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:15.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f3182431640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-04-26T00:23:15.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f3182431640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f317c0a0300 0x7f317c09fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:15.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f3182431640 10 monclient(hunting): picked mon.noname-c con 0x7f317c0a0300 addr v2:172.21.15.45:3300/0 2024-04-26T00:23:15.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f3182431640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f317c0a06e0 0x7f317c134ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:15.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f3182431640 10 monclient(hunting): picked mon.noname-a con 0x7f317c0a06e0 addr v2:172.21.15.37:3300/0 2024-04-26T00:23:15.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f3182431640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f317c1350e0 0x7f317c135550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:15.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f3182431640 10 monclient(hunting): picked mon.noname-b con 0x7f317c1350e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:23:15.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f3182431640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:15.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f3182431640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f317c0a13f0 con 0x7f317c0a06e0 2024-04-26T00:23:15.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f3182431640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:15.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f3182431640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f317c09c190 con 0x7f317c0a0300 2024-04-26T00:23:15.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f3182431640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:15.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f3182431640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f317c0a1110 con 0x7f317c1350e0 2024-04-26T00:23:15.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f3182431640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:15.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f3182431640 10 monclient(hunting): authenticate will time out at 1176.941310s 2024-04-26T00:23:15.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f317b7fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f317c0a06e0 0x7f317c134ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:15.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f31809ce640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f317c1350e0 0x7f317c135550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:15.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f317b7fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f317c0a06e0 0x7f317c134ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:48290/0 (socket says 172.21.15.37:48290) 2024-04-26T00:23:15.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f317b7fe640 1 -- 172.21.15.37:0/875900498 learned_addr learned my addr 172.21.15.37:0/875900498 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:23:15.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f31809ce640 10 monclient(hunting): get_auth_request con 0x7f317c1350e0 auth_method 0 2024-04-26T00:23:15.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f31809ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:15.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f31809ce640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:15.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f31809ce640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:15.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f317b7fe640 10 monclient(hunting): get_auth_request con 0x7f317c0a06e0 auth_method 0 2024-04-26T00:23:15.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f317b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:15.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f317b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:15.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f317b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:15.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f31809ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:15.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f31809ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:15.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f317bfff640 1 --2- 172.21.15.37:0/875900498 >> v2:172.21.15.45:3300/0 conn(0x7f317c0a0300 0x7f317c09fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:15.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f31809ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:15.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f317b7fe640 1 -- 172.21.15.37:0/875900498 >> v2:172.21.15.37:3300/0 conn(0x7f317c0a06e0 msgr2=0x7f317c134ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:23:15.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f317b7fe640 1 -- 172.21.15.37:0/875900498 >> v2:172.21.15.37:3300/0 conn(0x7f317c0a06e0 msgr2=0x7f317c134ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:23:15.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f317b7fe640 1 --2- 172.21.15.37:0/875900498 >> v2:172.21.15.37:3300/0 conn(0x7f317c0a06e0 0x7f317c134ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:23:15.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f317bfff640 10 monclient(hunting): get_auth_request con 0x7f317c0a0300 auth_method 0 2024-04-26T00:23:15.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f317bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:15.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f317bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:15.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f317bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:15.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f317b7fe640 1 --2- 172.21.15.37:0/875900498 >> v2:172.21.15.37:3300/0 conn(0x7f317c0a06e0 0x7f317c134ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:23:15.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f317bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:15.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f317bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:15.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f317bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:15.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f31809ce640 10 monclient(hunting): handle_auth_done global_id 34193 payload 274 2024-04-26T00:23:15.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f31809ce640 1 -- 172.21.15.37:0/875900498 >> v2:172.21.15.45:3300/0 conn(0x7f317c0a0300 msgr2=0x7f317c09fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:23:15.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f31809ce640 1 --2- 172.21.15.37:0/875900498 >> v2:172.21.15.45:3300/0 conn(0x7f317c0a0300 0x7f317c09fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:15.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f31809ce640 1 -- 172.21.15.37:0/875900498 >> v2:172.21.15.37:3300/0 conn(0x7f317c0a06e0 msgr2=0x7f317c134ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:23:15.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f31809ce640 1 --2- 172.21.15.37:0/875900498 >> v2:172.21.15.37:3300/0 conn(0x7f317c0a06e0 0x7f317c134ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:15.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f31809ce640 10 monclient: _finish_hunting 0 2024-04-26T00:23:15.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f31809ce640 1 monclient: found mon.noname-b 2024-04-26T00:23:15.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f31809ce640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:15.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f31809ce640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:23:15.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.634+0000 7f31809ce640 1 -- 172.21.15.37:0/875900498 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f317c0a16d0 con 0x7f317c1350e0 2024-04-26T00:23:15.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.638+0000 7f31809ce640 1 -- 172.21.15.37:0/875900498 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f317c1350e0 msgr2=0x7f317c135550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:15.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.638+0000 7f31809ce640 1 --2- 172.21.15.37:0/875900498 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f317c1350e0 0x7f317c135550 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f317c0764d0 tx=0x7f317401c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=663270405a06e392 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:15.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.638+0000 7f317affd640 1 -- 172.21.15.37:0/875900498 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31740b2070 con 0x7f317c1350e0 2024-04-26T00:23:15.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.638+0000 7f317affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:15.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.638+0000 7f317affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:23:15.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.638+0000 7f317affd640 10 monclient: dump: 2024-04-26T00:23:15.624 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:15.624 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:15.624 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:15.624 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:15.624 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:15.624 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:15.625 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:15.625 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:15.625 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:15.625 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:15.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.638+0000 7f317affd640 10 monclient: _finish_auth 0 2024-04-26T00:23:15.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.638+0000 7f317affd640 10 monclient: _check_auth_tickets 2024-04-26T00:23:15.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.638+0000 7f317affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:15.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.638+0000 7f317affd640 1 -- 172.21.15.37:0/875900498 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31740a5780 con 0x7f317c1350e0 2024-04-26T00:23:15.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.638+0000 7f317affd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:23:15.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.638+0000 7f317affd640 1 -- 172.21.15.37:0/875900498 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31740a5b50 con 0x7f317c1350e0 2024-04-26T00:23:15.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.638+0000 7f3182431640 5 monclient: authenticate success, global_id 34193 2024-04-26T00:23:15.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.638+0000 7f3182431640 10 monclient: get_monmap_and_config success 2024-04-26T00:23:15.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.638+0000 7f3182431640 4 set_mon_vals no callback set 2024-04-26T00:23:15.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.638+0000 7f317affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:15.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.638+0000 7f317affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:23:15.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.638+0000 7f317affd640 10 monclient: dump: 2024-04-26T00:23:15.626 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:15.626 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:15.626 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:15.626 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:15.626 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:15.626 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:15.626 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:15.626 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:15.626 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:15.627 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:15.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.638+0000 7f3182431640 10 monclient: shutdown 2024-04-26T00:23:15.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.638+0000 7f3182431640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:23:15.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.638+0000 7f3182431640 1 -- 172.21.15.37:0/875900498 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f317c1350e0 msgr2=0x7f317c135550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:15.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.638+0000 7f3182431640 1 --2- 172.21.15.37:0/875900498 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f317c1350e0 0x7f317c135550 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f317c0764d0 tx=0x7f317401c7f0 comp rx=0 tx=0).stop 2024-04-26T00:23:15.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.638+0000 7f3182431640 1 -- 172.21.15.37:0/875900498 shutdown_connections 2024-04-26T00:23:15.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.638+0000 7f3182431640 1 --2- 172.21.15.37:0/875900498 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f317c1350e0 0x7f317c135550 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:15.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.638+0000 7f3182431640 1 --2- 172.21.15.37:0/875900498 >> v2:172.21.15.37:3300/0 conn(0x7f317c0a06e0 0x7f317c134ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:15.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.638+0000 7f3182431640 1 --2- 172.21.15.37:0/875900498 >> v2:172.21.15.45:3300/0 conn(0x7f317c0a0300 0x7f317c09fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:15.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.638+0000 7f3182431640 1 -- 172.21.15.37:0/875900498 >> 172.21.15.37:0/875900498 conn(0x7f317c0a1d50 msgr2=0x7f317c0a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:23:15.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.638+0000 7f3182431640 1 -- 172.21.15.37:0/875900498 shutdown_connections 2024-04-26T00:23:15.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.638+0000 7f3182431640 1 -- 172.21.15.37:0/875900498 wait complete. 2024-04-26T00:23:15.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.638+0000 7f3182431640 10 monclient: build_initial_monmap 2024-04-26T00:23:15.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.638+0000 7f3182431640 10 monclient: monmap: 2024-04-26T00:23:15.628 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:23:15.628 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:23:15.628 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:23:15.628 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:23:15.628 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:23:15.628 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:23:15.628 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:23:15.628 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:23:15.628 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:23:15.628 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:15.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f3182431640 1 Processor -- start 2024-04-26T00:23:15.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f3182431640 1 -- start start 2024-04-26T00:23:15.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f3182431640 10 monclient: init 2024-04-26T00:23:15.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f3182431640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:15.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f3182431640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:23:15.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f3182431640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c0a0300 0x7f317c1ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:15.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f3182431640 10 monclient(hunting): picked mon.noname-a con 0x7f317c0a0300 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:23:15.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f3182431640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f317c0a06e0 0x7f317c1cd530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:15.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f3182431640 10 monclient(hunting): picked mon.noname-c con 0x7f317c0a06e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:23:15.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f3182431640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f317c1350e0 0x7f317c1cda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:15.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f3182431640 10 monclient(hunting): picked mon.noname-b con 0x7f317c1350e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:23:15.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f3182431640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:15.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f3182431640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f317c13e6f0 con 0x7f317c0a0300 2024-04-26T00:23:15.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f3182431640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:15.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f3182431640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f317c13e870 con 0x7f317c1350e0 2024-04-26T00:23:15.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f3182431640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:15.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f3182431640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f317c13e9f0 con 0x7f317c0a06e0 2024-04-26T00:23:15.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f3182431640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:15.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f3182431640 10 monclient(hunting): authenticate will time out at 1176.948657s 2024-04-26T00:23:15.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f317bfff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c0a0300 0x7f317c1ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:15.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f317b7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f317c0a06e0 0x7f317c1cd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:15.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f317bfff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c0a0300 0x7f317c1ccff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:48298/0 (socket says 172.21.15.37:48298) 2024-04-26T00:23:15.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f317bfff640 1 -- 172.21.15.37:0/3201617685 learned_addr learned my addr 172.21.15.37:0/3201617685 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:23:15.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f317bfff640 10 monclient(hunting): get_auth_request con 0x7f317c0a0300 auth_method 0 2024-04-26T00:23:15.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f317bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:15.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f317bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:15.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f317bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:15.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f317b7fe640 10 monclient(hunting): get_auth_request con 0x7f317c0a06e0 auth_method 0 2024-04-26T00:23:15.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f317b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:15.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f317b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:15.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f317b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:15.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f317bfff640 1 -- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c0a0300 msgr2=0x7f317c1ccff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:23:15.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f317bfff640 1 -- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c0a0300 msgr2=0x7f317c1ccff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:23:15.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f317bfff640 1 --2- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c0a0300 0x7f317c1ccff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:23:15.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f317b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:15.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f317b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:15.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f317b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:15.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f317bfff640 1 --2- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c0a0300 0x7f317c1ccff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:23:15.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f31809ce640 1 --2- 172.21.15.37:0/3201617685 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f317c1350e0 0x7f317c1cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:15.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f31809ce640 10 monclient(hunting): get_auth_request con 0x7f317c1350e0 auth_method 0 2024-04-26T00:23:15.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f31809ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:15.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f31809ce640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:15.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f31809ce640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:15.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f317b7fe640 10 monclient(hunting): handle_auth_done global_id 34196 payload 824 2024-04-26T00:23:15.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f317b7fe640 1 -- 172.21.15.37:0/3201617685 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f317c1350e0 msgr2=0x7f317c1cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:15.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f317b7fe640 1 --2- 172.21.15.37:0/3201617685 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f317c1350e0 0x7f317c1cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:15.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f317b7fe640 1 -- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c0a0300 msgr2=0x7f317c1ccff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:23:15.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f317b7fe640 1 --2- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c0a0300 0x7f317c1ccff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:15.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f317b7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:23:15.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f317b7fe640 1 monclient: found mon.noname-c 2024-04-26T00:23:15.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f317b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:15.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f317b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:23:15.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f317b7fe640 1 -- 172.21.15.37:0/3201617685 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f317c13a710 con 0x7f317c0a06e0 2024-04-26T00:23:15.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f317b7fe640 1 -- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f317c0a06e0 msgr2=0x7f317c1cd530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:15.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f317b7fe640 1 --2- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f317c0a06e0 0x7f317c1cd530 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f317000e810 tx=0x7f317000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:15.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f3178ff9640 1 -- 172.21.15.37:0/3201617685 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3170017070 con 0x7f317c0a06e0 2024-04-26T00:23:15.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f3178ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:15.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f3178ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:23:15.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f3178ff9640 10 monclient: dump: 2024-04-26T00:23:15.632 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:15.632 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:15.632 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:15.633 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:15.633 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:15.633 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:15.633 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:15.633 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:15.633 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:15.633 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:15.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f3178ff9640 10 monclient: _finish_auth 0 2024-04-26T00:23:15.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f3178ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:23:15.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f3178ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:15.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f3178ff9640 1 -- 172.21.15.37:0/3201617685 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3170004260 con 0x7f317c0a06e0 2024-04-26T00:23:15.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f3178ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:23:15.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f3178ff9640 1 -- 172.21.15.37:0/3201617685 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f317000f040 con 0x7f317c0a06e0 2024-04-26T00:23:15.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f3182431640 5 monclient: authenticate success, global_id 34196 2024-04-26T00:23:15.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f3182431640 10 monclient: _renew_subs 2024-04-26T00:23:15.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f3182431640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:15.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f3182431640 1 -- 172.21.15.37:0/3201617685 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f317c13a9f0 con 0x7f317c0a06e0 2024-04-26T00:23:15.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f3178ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:15.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f3178ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:23:15.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f3178ff9640 10 monclient: dump: 2024-04-26T00:23:15.634 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:15.634 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:15.634 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:15.634 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:15.634 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:15.634 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:15.634 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:15.634 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:15.634 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:15.635 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:15.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f317a7fc640 4 set_mon_vals no callback set 2024-04-26T00:23:15.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f3182431640 10 monclient: _renew_subs 2024-04-26T00:23:15.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f3182431640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:15.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.642+0000 7f3182431640 1 -- 172.21.15.37:0/3201617685 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f317c09f550 con 0x7f317c0a06e0 2024-04-26T00:23:15.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.646+0000 7f3182431640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:15.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.646+0000 7f3182431640 1 --2- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c12ea60 0x7f317c12ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:15.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.646+0000 7f3182431640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:15.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.646+0000 7f3182431640 1 -- 172.21.15.37:0/3201617685 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f317c139370 con 0x7f317c12ea60 2024-04-26T00:23:15.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.646+0000 7f3178ff9640 1 -- 172.21.15.37:0/3201617685 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f317000c6d0 con 0x7f317c0a06e0 2024-04-26T00:23:15.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.646+0000 7f317bfff640 1 --2- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c12ea60 0x7f317c12ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:15.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.646+0000 7f317bfff640 10 monclient: get_auth_request con 0x7f317c12ea60 auth_method 0 2024-04-26T00:23:15.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.646+0000 7f317bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:15.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.646+0000 7f317bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:15.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.646+0000 7f317bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:15.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.646+0000 7f317bfff640 1 -- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c12ea60 msgr2=0x7f317c12ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:23:15.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.646+0000 7f317bfff640 1 -- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c12ea60 msgr2=0x7f317c12ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:23:15.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.646+0000 7f317bfff640 1 --2- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c12ea60 0x7f317c12ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:23:15.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.646+0000 7f3178ff9640 1 --2- 172.21.15.37:0/3201617685 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3168035ed0 0x7f3168038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:15.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.646+0000 7f317bfff640 1 --2- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c12ea60 0x7f317c12ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:23:15.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.646+0000 7f3178ff9640 1 -- 172.21.15.37:0/3201617685 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(140..140 src has 1..140) ==== 3959+0+0 (secure 0 0 0) 0x7f3170048510 con 0x7f317c0a06e0 2024-04-26T00:23:15.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.646+0000 7f31809ce640 1 --2- 172.21.15.37:0/3201617685 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3168035ed0 0x7f3168038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:15.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.646+0000 7f31809ce640 10 monclient: get_auth_request con 0x7f3168035ed0 auth_method 0 2024-04-26T00:23:15.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.650+0000 7f31809ce640 1 --2- 172.21.15.37:0/3201617685 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3168035ed0 0x7f3168038390 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f317c06caa0 tx=0x7f31740a7070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:15.739 INFO:tasks.rados.rados.0.smithi045.stdout:611: done (1 left) 2024-04-26T00:23:15.739 INFO:tasks.rados.rados.0.smithi045.stdout:612: done (0 left) 2024-04-26T00:23:15.739 INFO:tasks.rados.rados.0.smithi045.stdout:613: snap_remove snap 65 2024-04-26T00:23:15.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.850+0000 7f317bfff640 1 --2- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c12ea60 0x7f317c12ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:15.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.850+0000 7f317bfff640 10 monclient: get_auth_request con 0x7f317c12ea60 auth_method 0 2024-04-26T00:23:15.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.850+0000 7f317bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:15.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.850+0000 7f317bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:15.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.850+0000 7f317bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:23:15.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.850+0000 7f317bfff640 1 -- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c12ea60 msgr2=0x7f317c12ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:23:15.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.850+0000 7f317bfff640 1 -- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c12ea60 msgr2=0x7f317c12ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:23:15.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.850+0000 7f317bfff640 1 --2- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c12ea60 0x7f317c12ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:23:15.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:15.850+0000 7f317bfff640 1 --2- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c12ea60 0x7f317c12ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:23:16.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:16.254+0000 7f317bfff640 1 --2- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c12ea60 0x7f317c12ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:16.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:16.254+0000 7f317bfff640 10 monclient: get_auth_request con 0x7f317c12ea60 auth_method 0 2024-04-26T00:23:16.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:16.254+0000 7f317bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:16.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:16.254+0000 7f317bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:16.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:16.254+0000 7f317bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:23:16.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:16.254+0000 7f317bfff640 1 -- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c12ea60 msgr2=0x7f317c12ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:23:16.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:16.254+0000 7f317bfff640 1 -- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c12ea60 msgr2=0x7f317c12ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:23:16.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:16.254+0000 7f317bfff640 1 --2- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c12ea60 0x7f317c12ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:23:16.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:16.254+0000 7f317bfff640 1 --2- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c12ea60 0x7f317c12ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:23:16.741 INFO:tasks.rados.rados.0.smithi045.stdout:613: done (0 left) 2024-04-26T00:23:16.741 INFO:tasks.rados.rados.0.smithi045.stdout:614: snap_remove snap 60 2024-04-26T00:23:17.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:17.058+0000 7f317bfff640 1 --2- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c12ea60 0x7f317c12ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:17.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:17.058+0000 7f317bfff640 10 monclient: get_auth_request con 0x7f317c12ea60 auth_method 0 2024-04-26T00:23:17.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:17.058+0000 7f317bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:17.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:17.058+0000 7f317bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:17.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:17.058+0000 7f317bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:23:17.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:17.058+0000 7f317bfff640 1 -- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c12ea60 msgr2=0x7f317c12ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:23:17.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:17.058+0000 7f317bfff640 1 -- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c12ea60 msgr2=0x7f317c12ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:23:17.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:17.058+0000 7f317bfff640 1 --2- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c12ea60 0x7f317c12ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:23:17.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:17.058+0000 7f317bfff640 1 --2- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c12ea60 0x7f317c12ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:23:17.744 INFO:tasks.rados.rados.0.smithi045.stdout:614: done (0 left) 2024-04-26T00:23:17.744 INFO:tasks.rados.rados.0.smithi045.stdout:615: copy_from oid 45 from oid 14 current snap is 67 2024-04-26T00:23:17.744 INFO:tasks.rados.rados.0.smithi045.stdout:616: rollback oid 15 current snap is 67 2024-04-26T00:23:17.744 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 15 to 37 2024-04-26T00:23:17.745 INFO:tasks.rados.rados.0.smithi045.stdout:617: snap_create 2024-04-26T00:23:17.765 INFO:tasks.rados.rados.0.smithi045.stdout:616: finishing rollback tid 0 to smithi04526579-15 2024-04-26T00:23:17.766 INFO:tasks.rados.rados.0.smithi045.stdout:616: finishing rollback tid 1 to smithi04526579-15 2024-04-26T00:23:17.766 INFO:tasks.rados.rados.0.smithi045.stdout:616: finishing rollback tid 2 to smithi04526579-15 2024-04-26T00:23:17.766 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 306 (ObjNum 163 snap 37 seq_num 163) dirty exists 2024-04-26T00:23:17.783 INFO:tasks.rados.rados.0.smithi045.stdout:615: finishing copy_from racing read to smithi04526579-45 2024-04-26T00:23:17.793 INFO:tasks.rados.rados.0.smithi045.stdout:615: finishing copy_from to smithi04526579-45 2024-04-26T00:23:17.793 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 125 (ObjNum 237 snap 60 seq_num 237) dirty exists 2024-04-26T00:23:18.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.642+0000 7f31637fe640 10 monclient: tick 2024-04-26T00:23:18.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.642+0000 7f31637fe640 10 monclient: _check_auth_tickets 2024-04-26T00:23:18.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.642+0000 7f31637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:18.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.642+0000 7f31637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:18.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.662+0000 7f317bfff640 1 --2- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c12ea60 0x7f317c12ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:18.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.662+0000 7f317bfff640 10 monclient: get_auth_request con 0x7f317c12ea60 auth_method 0 2024-04-26T00:23:18.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.662+0000 7f317bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:18.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.662+0000 7f317bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:18.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.662+0000 7f317bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:23:18.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.662+0000 7f317bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:18.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.662+0000 7f317bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:18.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.662+0000 7f317bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:18.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.662+0000 7f317bfff640 10 monclient(hunting): handle_auth_done global_id 44097 payload 824 2024-04-26T00:23:18.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.662+0000 7f317bfff640 1 -- 172.21.15.37:0/3201617685 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f317c1389e0 con 0x7f317c12ea60 2024-04-26T00:23:18.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.662+0000 7f317bfff640 1 -- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c12ea60 msgr2=0x7f317c12ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:18.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.662+0000 7f317bfff640 1 --2- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c12ea60 0x7f317c12ee40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f316c019a00 tx=0x7f316c019ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:18.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.666+0000 7f3178ff9640 1 -- 172.21.15.37:0/3201617685 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f316c004230 con 0x7f317c12ea60 2024-04-26T00:23:18.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.666+0000 7f3178ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:18.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.666+0000 7f3178ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:23:18.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.666+0000 7f3178ff9640 10 monclient: dump: 2024-04-26T00:23:18.643 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:18.643 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:18.643 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:18.643 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:18.643 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:18.643 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:18.643 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:18.643 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:18.643 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:18.643 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:18.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.666+0000 7f3178ff9640 1 -- 172.21.15.37:0/3201617685 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f316c017e50 con 0x7f317c12ea60 2024-04-26T00:23:18.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.666+0000 7f3178ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:18.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.666+0000 7f3178ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:23:18.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.666+0000 7f3178ff9640 1 -- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c12ea60 msgr2=0x7f317c12ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:18.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.666+0000 7f3178ff9640 1 --2- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c12ea60 0x7f317c12ee40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f316c019a00 tx=0x7f316c019ec0 comp rx=0 tx=0).stop 2024-04-26T00:23:18.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.666+0000 7f3178ff9640 1 -- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c12ea60 msgr2=0x7f317c12ee40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:23:18.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.666+0000 7f3178ff9640 1 --2- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c12ea60 0x7f317c12ee40 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f316c019a00 tx=0x7f316c019ec0 comp rx=0 tx=0).stop 2024-04-26T00:23:18.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.674+0000 7f3182431640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:23:18.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.674+0000 7f3182431640 1 --2- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c12f380 0x7f317c06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:18.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.674+0000 7f3182431640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:18.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.674+0000 7f3182431640 1 -- 172.21.15.37:0/3201617685 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f317c13a430 con 0x7f317c12f380 2024-04-26T00:23:18.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.674+0000 7f317bfff640 1 --2- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c12f380 0x7f317c06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:18.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.674+0000 7f317bfff640 10 monclient: get_auth_request con 0x7f317c12f380 auth_method 0 2024-04-26T00:23:18.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.674+0000 7f317bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:18.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.674+0000 7f317bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:18.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.674+0000 7f317bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:18.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.674+0000 7f317bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:18.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.674+0000 7f317bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:18.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.674+0000 7f317bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:18.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.674+0000 7f317bfff640 10 monclient(hunting): handle_auth_done global_id 44100 payload 824 2024-04-26T00:23:18.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.674+0000 7f317bfff640 1 -- 172.21.15.37:0/3201617685 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f317c04f510 con 0x7f317c12f380 2024-04-26T00:23:18.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.678+0000 7f317bfff640 1 -- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c12f380 msgr2=0x7f317c06ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:18.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.678+0000 7f317bfff640 1 --2- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c12f380 0x7f317c06ea10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f316c009fd0 tx=0x7f316c0046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:18.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.678+0000 7f3178ff9640 1 -- 172.21.15.37:0/3201617685 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f316c028040 con 0x7f317c12f380 2024-04-26T00:23:18.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.678+0000 7f3178ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:18.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.678+0000 7f3178ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:23:18.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.678+0000 7f3178ff9640 10 monclient: dump: 2024-04-26T00:23:18.655 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:18.655 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:18.655 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:18.655 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:18.655 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:18.655 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:18.656 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:18.656 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:18.656 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:18.656 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:18.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.678+0000 7f3178ff9640 1 -- 172.21.15.37:0/3201617685 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7f316c003c40 con 0x7f317c12f380 2024-04-26T00:23:18.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.678+0000 7f3178ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:23:18.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.678+0000 7f3178ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:23:18.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.678+0000 7f3178ff9640 1 -- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c12f380 msgr2=0x7f317c06ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:18.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.678+0000 7f3178ff9640 1 --2- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c12f380 0x7f317c06ea10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f316c009fd0 tx=0x7f316c0046d0 comp rx=0 tx=0).stop 2024-04-26T00:23:18.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.678+0000 7f3178ff9640 1 -- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c12f380 msgr2=0x7f317c06ea10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:23:18.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.678+0000 7f3178ff9640 1 --2- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f317c12f380 0x7f317c06ea10 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f316c009fd0 tx=0x7f316c0046d0 comp rx=0 tx=0).stop 2024-04-26T00:23:18.656 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:23:18.657 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:23:18.657 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:23:18.657 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:23:18.657 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 28, 2024-04-26T00:23:18.657 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 10519, 2024-04-26T00:23:18.657 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:23:18.657 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:23:18.657 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:23:18.657 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:23:18.657 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:18.657 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 4, 2024-04-26T00:23:18.657 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:23:18.657 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:23:18.658 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:23:18.658 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:23:18.658 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:23:18.658 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:23:18.658 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:23:18.658 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:23:18.658 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:23:18.658 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:23:18.658 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:23:18.658 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:23:18.658 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:23:18.658 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:23:18.658 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:18.658 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:23:18.658 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:23:18.659 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:23:18.659 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:23:18.659 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:23:18.659 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:23:18.659 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:23:18.659 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:23:18.659 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:23:18.659 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:23:18.659 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:23:18.659 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:23:18.659 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:23:18.659 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:18.659 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:18.659 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:23:18.660 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:23:18.660 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:23:18.660 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:23:18.660 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:23:18.660 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:23:18.660 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:23:18.660 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:23:18.660 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:23:18.660 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:23:18.660 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:23:18.660 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:23:18.660 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:23:18.660 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:23:18.660 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:23:18.660 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:23:18.661 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:23:18.661 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:23:18.661 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:23:18.661 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:23:18.661 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:23:18.661 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:23:18.661 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:23:18.661 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:23:18.661 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:23:18.661 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:23:18.661 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:23:18.661 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:23:18.661 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:18.661 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:23:18.662 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:18.662 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:23:18.662 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:18.662 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:23:18.662 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:23:18.662 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:23:18.662 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:23:18.662 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:18.662 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:23:18.662 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:23:18.662 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:18.662 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:18.662 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:18.662 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:23:18.663 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:23:18.663 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:18.663 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:18.663 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:18.663 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:18.663 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:23:18.663 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:23:18.663 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:23:18.663 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:23:18.663 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:23:18.663 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:18.663 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:18.663 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:23:18.663 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:23:18.663 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:23:18.664 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:23:18.664 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:18.664 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:23:18.664 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:23:18.664 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:18.664 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:18.664 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:18.664 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:23:18.664 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:23:18.664 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:18.664 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:18.664 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:18.664 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:18.664 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:23:18.665 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:23:18.665 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:23:18.665 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:23:18.665 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:23:18.665 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:18.665 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:18.665 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:23:18.665 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:23:18.665 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:23:18.665 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:23:18.665 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:18.665 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:23:18.665 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:23:18.665 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:18.666 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:18.666 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:18.666 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:23:18.666 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:23:18.666 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:18.666 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:18.666 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:18.666 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:18.666 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:23:18.666 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:23:18.666 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:23:18.666 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:23:18.666 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:23:18.666 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:18.667 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:18.667 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:18.667 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:23:18.667 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:23:18.667 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:18.667 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:23:18.667 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:23:18.667 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:23:18.667 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:18.667 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:18.667 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:23:18.667 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:18.667 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:23:18.667 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:23:18.667 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:23:18.667 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:18.668 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:18.668 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:18.668 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:23:18.668 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:23:18.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.678+0000 7f3182431640 1 -- 172.21.15.37:0/3201617685 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3168035ed0 msgr2=0x7f3168038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:18.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.678+0000 7f3182431640 1 --2- 172.21.15.37:0/3201617685 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3168035ed0 0x7f3168038390 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f317c06caa0 tx=0x7f31740a7070 comp rx=0 tx=0).stop 2024-04-26T00:23:18.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.678+0000 7f3182431640 10 monclient: shutdown 2024-04-26T00:23:18.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.678+0000 7f3182431640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:23:18.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.678+0000 7f3182431640 1 -- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f317c0a06e0 msgr2=0x7f317c1cd530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:18.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.678+0000 7f3182431640 1 --2- 172.21.15.37:0/3201617685 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f317c0a06e0 0x7f317c1cd530 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f317000e810 tx=0x7f317000ecd0 comp rx=0 tx=0).stop 2024-04-26T00:23:18.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.678+0000 7f31809ce640 1 -- 172.21.15.37:0/3201617685 reap_dead start 2024-04-26T00:23:18.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.678+0000 7f31809ce640 1 -- 172.21.15.37:0/3201617685 reap_dead start 2024-04-26T00:23:18.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.678+0000 7f3182431640 1 -- 172.21.15.37:0/3201617685 shutdown_connections 2024-04-26T00:23:18.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.678+0000 7f3182431640 1 -- 172.21.15.37:0/3201617685 >> 172.21.15.37:0/3201617685 conn(0x7f317c0a1d50 msgr2=0x7f317c131c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:23:18.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.678+0000 7f3182431640 1 -- 172.21.15.37:0/3201617685 shutdown_connections 2024-04-26T00:23:18.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.678+0000 7f3182431640 1 -- 172.21.15.37:0/3201617685 wait complete. 2024-04-26T00:23:18.680 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:23:18.680 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:23:18.680 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:23:18.792 INFO:tasks.rados.rados.0.smithi045.stdout:615: done (2 left) 2024-04-26T00:23:18.792 INFO:tasks.rados.rados.0.smithi045.stdout:616: done (1 left) 2024-04-26T00:23:18.792 INFO:tasks.rados.rados.0.smithi045.stdout:617: done (0 left) 2024-04-26T00:23:18.792 INFO:tasks.rados.rados.0.smithi045.stdout:618: write oid 47 current snap is 68 2024-04-26T00:23:18.792 INFO:tasks.rados.rados.0.smithi045.stdout:618: seq_num 257 ranges {719399=778642,1939284=380115} 2024-04-26T00:23:18.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1ba0cf640 10 monclient: get_monmap_and_config 2024-04-26T00:23:18.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1ba0cf640 10 monclient: build_initial_monmap 2024-04-26T00:23:18.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1ba0cf640 10 monclient: monmap: 2024-04-26T00:23:18.811 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:23:18.811 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:18.811 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:23:18.836502+0000 2024-04-26T00:23:18.811 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:23:18.836502+0000 2024-04-26T00:23:18.811 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:23:18.811 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:18.811 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:23:18.811 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:23:18.811 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:23:18.811 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:23:18.812 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:23:18.812 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:18.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1ba0cf640 1 Processor -- start 2024-04-26T00:23:18.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1ba0cf640 1 -- start start 2024-04-26T00:23:18.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1ba0cf640 10 monclient: init 2024-04-26T00:23:18.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1ba0cf640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:18.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1ba0cf640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-04-26T00:23:18.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1ba0cf640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb1b409ec20 0x7fb1b4131b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:18.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1ba0cf640 10 monclient(hunting): picked mon.noname-b con 0x7fb1b409ec20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:23:18.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1ba0cf640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb1b409b720 0x7fb1b4134bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:18.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1ba0cf640 10 monclient(hunting): picked mon.noname-a con 0x7fb1b409b720 addr v2:172.21.15.37:3300/0 2024-04-26T00:23:18.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1ba0cf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb1b409bb80 addr v1:172.21.15.37:6789/0 2024-04-26T00:23:18.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1ba0cf640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1b40999c0 con 0x7fb1b409bb80 2024-04-26T00:23:18.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1ba0cf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:18.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1ba0cf640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fb1b409c550 con 0x7fb1b409b720 2024-04-26T00:23:18.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1ba0cf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:18.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1ba0cf640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb1b409f000 con 0x7fb1b409ec20 2024-04-26T00:23:18.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1ba0cf640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:18.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1ba0cf640 10 monclient(hunting): authenticate will time out at 1180.142934s 2024-04-26T00:23:18.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1b37fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb1b409ec20 0x7fb1b4131b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:18.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1abfff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb1b409b720 0x7fb1b4134bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:18.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1b37fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb1b409ec20 0x7fb1b4131b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:38954/0 (socket says 172.21.15.37:38954) 2024-04-26T00:23:18.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1b3fff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fb1b409bb80 0x7fb1b41350f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:42436/0 (socket says 172.21.15.37:42436) 2024-04-26T00:23:18.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1abfff640 10 monclient(hunting): get_auth_request con 0x7fb1b409b720 auth_method 0 2024-04-26T00:23:18.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1b37fe640 1 -- 172.21.15.37:0/3662398724 learned_addr learned my addr 172.21.15.37:0/3662398724 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:23:18.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:18.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1abfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:18.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1abfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:18.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1b37fe640 10 monclient(hunting): get_auth_request con 0x7fb1b409ec20 auth_method 0 2024-04-26T00:23:18.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:18.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1b37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:18.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:18.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:18.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:18.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:18.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:18.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:18.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:18.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1abfff640 10 monclient(hunting): handle_auth_done global_id 44103 payload 274 2024-04-26T00:23:18.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1abfff640 1 -- 172.21.15.37:0/3662398724 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb1b409ec20 msgr2=0x7fb1b4131b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:23:18.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1abfff640 1 --2- 172.21.15.37:0/3662398724 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb1b409ec20 0x7fb1b4131b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:18.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1abfff640 1 -- 172.21.15.37:0/3662398724 >> v1:172.21.15.37:6789/0 conn(0x7fb1b409bb80 legacy=0x7fb1b41350f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:23:18.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1abfff640 10 monclient: _finish_hunting 0 2024-04-26T00:23:18.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1abfff640 1 monclient: found mon.noname-a 2024-04-26T00:23:18.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:18.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1abfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:23:18.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.834+0000 7fb1abfff640 1 -- 172.21.15.37:0/3662398724 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1b4099c30 con 0x7fb1b409b720 2024-04-26T00:23:18.818 INFO:tasks.rados.rados.0.smithi045.stdout:618: writing smithi04526579-47 from 719399 to 1498041 tid 1 2024-04-26T00:23:18.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.838+0000 7fb1abfff640 1 -- 172.21.15.37:0/3662398724 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb1b409b720 msgr2=0x7fb1b4134bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:18.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.838+0000 7fb1abfff640 1 --2- 172.21.15.37:0/3662398724 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb1b409b720 0x7fb1b4134bb0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb19c0097c0 tx=0x7fb19c01c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3329dde436a6c955 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:18.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1b2ffd640 1 -- 172.21.15.37:0/3662398724 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb19c0b0070 con 0x7fb1b409b720 2024-04-26T00:23:18.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:18.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1b2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:23:18.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1b2ffd640 10 monclient: dump: 2024-04-26T00:23:18.819 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:18.819 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:18.820 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:18.820 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:18.820 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:18.820 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:18.820 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:18.820 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:18.820 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:18.820 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:18.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1b2ffd640 10 monclient: _finish_auth 0 2024-04-26T00:23:18.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1b2ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:23:18.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:18.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1b2ffd640 1 -- 172.21.15.37:0/3662398724 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb19c0ab600 con 0x7fb1b409b720 2024-04-26T00:23:18.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1b2ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:23:18.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1b2ffd640 1 -- 172.21.15.37:0/3662398724 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb19c0ab9d0 con 0x7fb1b409b720 2024-04-26T00:23:18.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1ba0cf640 5 monclient: authenticate success, global_id 44103 2024-04-26T00:23:18.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1ba0cf640 10 monclient: get_monmap_and_config success 2024-04-26T00:23:18.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1ba0cf640 4 set_mon_vals no callback set 2024-04-26T00:23:18.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:18.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1b2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:23:18.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1b2ffd640 10 monclient: dump: 2024-04-26T00:23:18.821 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:18.821 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:18.821 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:18.821 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:18.821 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:18.821 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:18.821 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:18.821 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:18.821 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:18.822 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:18.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1ba0cf640 10 monclient: shutdown 2024-04-26T00:23:18.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1ba0cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:23:18.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1ba0cf640 1 -- 172.21.15.37:0/3662398724 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb1b409b720 msgr2=0x7fb1b4134bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:18.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1ba0cf640 1 --2- 172.21.15.37:0/3662398724 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb1b409b720 0x7fb1b4134bb0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb19c0097c0 tx=0x7fb19c01c320 comp rx=0 tx=0).stop 2024-04-26T00:23:18.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1ba0cf640 1 -- 172.21.15.37:0/3662398724 shutdown_connections 2024-04-26T00:23:18.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1ba0cf640 1 --2- 172.21.15.37:0/3662398724 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb1b409b720 0x7fb1b4134bb0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:18.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1ba0cf640 1 --2- 172.21.15.37:0/3662398724 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb1b409ec20 0x7fb1b4131b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:18.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1ba0cf640 1 -- 172.21.15.37:0/3662398724 >> 172.21.15.37:0/3662398724 conn(0x7fb1b40a5710 msgr2=0x7fb1b40a5b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:23:18.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1ba0cf640 1 -- 172.21.15.37:0/3662398724 shutdown_connections 2024-04-26T00:23:18.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1ba0cf640 1 -- 172.21.15.37:0/3662398724 wait complete. 2024-04-26T00:23:18.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1ba0cf640 10 monclient: build_initial_monmap 2024-04-26T00:23:18.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1ba0cf640 10 monclient: monmap: 2024-04-26T00:23:18.827 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:23:18.827 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:23:18.827 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:23:18.827 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:23:18.827 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:23:18.827 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:23:18.827 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:23:18.827 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:23:18.827 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:23:18.827 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:18.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1ba0cf640 1 Processor -- start 2024-04-26T00:23:18.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1ba0cf640 1 -- start start 2024-04-26T00:23:18.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1ba0cf640 10 monclient: init 2024-04-26T00:23:18.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1ba0cf640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:18.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1ba0cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:23:18.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1ba0cf640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb1b409b720 0x7fb1b4071c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:18.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1ba0cf640 10 monclient(hunting): picked mon.noname-c con 0x7fb1b409b720 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:23:18.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1ba0cf640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb1b409bb80 0x7fb1b4072140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:18.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1ba0cf640 10 monclient(hunting): picked mon.noname-a con 0x7fb1b409bb80 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:23:18.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1ba0cf640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1b409ec20 0x7fb1b4072680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:18.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1ba0cf640 10 monclient(hunting): picked mon.noname-b con 0x7fb1b409ec20 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:23:18.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1ba0cf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:18.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1ba0cf640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b41359c0 con 0x7fb1b409bb80 2024-04-26T00:23:18.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1ba0cf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:18.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1ba0cf640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b4135840 con 0x7fb1b409ec20 2024-04-26T00:23:18.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1ba0cf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:18.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1ba0cf640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb1b413f710 con 0x7fb1b409b720 2024-04-26T00:23:18.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1ba0cf640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:18.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1ba0cf640 10 monclient(hunting): authenticate will time out at 1180.150750s 2024-04-26T00:23:18.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1abfff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb1b409bb80 0x7fb1b4072140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:18.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1b37fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb1b409b720 0x7fb1b4071c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:18.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1abfff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb1b409bb80 0x7fb1b4072140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:48374/0 (socket says 172.21.15.37:48374) 2024-04-26T00:23:18.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1abfff640 1 -- 172.21.15.37:0/689485355 learned_addr learned my addr 172.21.15.37:0/689485355 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:23:18.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1abfff640 10 monclient(hunting): get_auth_request con 0x7fb1b409bb80 auth_method 0 2024-04-26T00:23:18.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:18.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1abfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:18.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1abfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:18.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1b37fe640 10 monclient(hunting): get_auth_request con 0x7fb1b409b720 auth_method 0 2024-04-26T00:23:18.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:18.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1b37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:18.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:18.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:18.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:18.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:18.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:18.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:18.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:18.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1b3fff640 1 --2- 172.21.15.37:0/689485355 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1b409ec20 0x7fb1b4072680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:18.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1abfff640 10 monclient(hunting): handle_auth_done global_id 44106 payload 824 2024-04-26T00:23:18.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1abfff640 1 -- 172.21.15.37:0/689485355 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb1b409b720 msgr2=0x7fb1b4071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:18.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1abfff640 1 --2- 172.21.15.37:0/689485355 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb1b409b720 0x7fb1b4071c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:18.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1abfff640 1 -- 172.21.15.37:0/689485355 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1b409ec20 msgr2=0x7fb1b4072680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:18.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1abfff640 1 --2- 172.21.15.37:0/689485355 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1b409ec20 0x7fb1b4072680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:18.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1abfff640 10 monclient: _finish_hunting 0 2024-04-26T00:23:18.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1abfff640 1 monclient: found mon.noname-a 2024-04-26T00:23:18.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:18.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1abfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:23:18.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1abfff640 1 -- 172.21.15.37:0/689485355 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1b412ed30 con 0x7fb1b409bb80 2024-04-26T00:23:18.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1b3fff640 10 monclient: get_auth_request con 0x7fb1b409ec20 auth_method 0 2024-04-26T00:23:18.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1b37fe640 1 --2- 172.21.15.37:0/689485355 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb1b409b720 0x7fb1b4071c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:23:18.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.842+0000 7fb1b3fff640 1 --2- 172.21.15.37:0/689485355 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb1b409ec20 0x7fb1b4072680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:23:18.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1abfff640 1 -- 172.21.15.37:0/689485355 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb1b409bb80 msgr2=0x7fb1b4072140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:18.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1abfff640 1 --2- 172.21.15.37:0/689485355 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb1b409bb80 0x7fb1b4072140 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb19c01a9d0 tx=0x7fb19c0190c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:18.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1b0ff9640 1 -- 172.21.15.37:0/689485355 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb19c0b8020 con 0x7fb1b409bb80 2024-04-26T00:23:18.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1b0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:18.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1b0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:23:18.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1b0ff9640 10 monclient: dump: 2024-04-26T00:23:18.831 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:18.831 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:18.831 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:18.831 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:18.832 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:18.832 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:18.832 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:18.832 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:18.832 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:18.832 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:18.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1b0ff9640 10 monclient: _finish_auth 0 2024-04-26T00:23:18.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1b0ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:23:18.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1b0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:18.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1ba0cf640 5 monclient: authenticate success, global_id 44106 2024-04-26T00:23:18.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1ba0cf640 10 monclient: _renew_subs 2024-04-26T00:23:18.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1ba0cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:23:18.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1ba0cf640 1 -- 172.21.15.37:0/689485355 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1b40a17f0 con 0x7fb1b409bb80 2024-04-26T00:23:18.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1b0ff9640 1 -- 172.21.15.37:0/689485355 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb19c0b4810 con 0x7fb1b409bb80 2024-04-26T00:23:18.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1b0ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:23:18.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1b0ff9640 1 -- 172.21.15.37:0/689485355 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb19c0b0040 con 0x7fb1b409bb80 2024-04-26T00:23:18.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1b0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:18.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1b0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:23:18.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1b0ff9640 10 monclient: dump: 2024-04-26T00:23:18.833 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:18.833 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:18.833 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:18.833 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:18.833 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:18.833 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:18.833 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:18.833 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:18.833 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:18.833 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:18.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1b27fc640 4 set_mon_vals no callback set 2024-04-26T00:23:18.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1ba0cf640 10 monclient: _renew_subs 2024-04-26T00:23:18.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1ba0cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:23:18.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1ba0cf640 1 -- 172.21.15.37:0/689485355 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1b409e870 con 0x7fb1b409bb80 2024-04-26T00:23:18.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1ba0cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:18.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1ba0cf640 1 --2- 172.21.15.37:0/689485355 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb1b41312d0 0x7fb1b406e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:18.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1ba0cf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:18.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1ba0cf640 1 -- 172.21.15.37:0/689485355 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb1b40737f0 con 0x7fb1b41312d0 2024-04-26T00:23:18.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1b0ff9640 1 -- 172.21.15.37:0/689485355 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb19c0bb020 con 0x7fb1b409bb80 2024-04-26T00:23:18.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1b37fe640 1 --2- 172.21.15.37:0/689485355 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb1b41312d0 0x7fb1b406e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:18.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1b37fe640 10 monclient: get_auth_request con 0x7fb1b41312d0 auth_method 0 2024-04-26T00:23:18.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:18.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1b37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:18.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:18.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1b0ff9640 1 --2- 172.21.15.37:0/689485355 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb194035e80 0x7fb194038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:18.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:18.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:18.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:18.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1b0ff9640 1 -- 172.21.15.37:0/689485355 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 3931+0+0 (secure 0 0 0) 0x7fb19c0e2720 con 0x7fb1b409bb80 2024-04-26T00:23:18.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1b37fe640 10 monclient(hunting): handle_auth_done global_id 34205 payload 824 2024-04-26T00:23:18.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1b37fe640 1 -- 172.21.15.37:0/689485355 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb1b404f510 con 0x7fb1b41312d0 2024-04-26T00:23:18.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1b3fff640 1 --2- 172.21.15.37:0/689485355 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb194035e80 0x7fb194038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:18.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.846+0000 7fb1b3fff640 10 monclient: get_auth_request con 0x7fb194035e80 auth_method 0 2024-04-26T00:23:18.838 INFO:tasks.rados.rados.0.smithi045.stdout:618: writing smithi04526579-47 from 1939284 to 2319399 tid 2 2024-04-26T00:23:18.838 INFO:tasks.rados.rados.0.smithi045.stdout:619: write oid 1 current snap is 68 2024-04-26T00:23:18.838 INFO:tasks.rados.rados.0.smithi045.stdout:618: finishing write tid 1 to smithi04526579-47 2024-04-26T00:23:18.838 INFO:tasks.rados.rados.0.smithi045.stdout:619: seq_num 258 ranges {736206=453891,1881413=454793} 2024-04-26T00:23:18.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.850+0000 7fb1b37fe640 1 -- 172.21.15.37:0/689485355 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb1b41312d0 msgr2=0x7fb1b406e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:18.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.850+0000 7fb1b37fe640 1 --2- 172.21.15.37:0/689485355 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb1b41312d0 0x7fb1b406e870 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4011cf0 tx=0x7fb1a400f610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:18.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.850+0000 7fb1b0ff9640 1 -- 172.21.15.37:0/689485355 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a4004020 con 0x7fb1b41312d0 2024-04-26T00:23:18.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.850+0000 7fb1b0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:18.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.850+0000 7fb1b0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:23:18.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.850+0000 7fb1b0ff9640 10 monclient: dump: 2024-04-26T00:23:18.840 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:18.840 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:18.840 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:18.840 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:18.840 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:18.841 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:18.841 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:18.841 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:18.841 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:18.841 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:18.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.850+0000 7fb1b0ff9640 1 -- 172.21.15.37:0/689485355 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb1a4004020 con 0x7fb1b41312d0 2024-04-26T00:23:18.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.850+0000 7fb1b0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:18.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.850+0000 7fb1b0ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:23:18.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.850+0000 7fb1b0ff9640 1 -- 172.21.15.37:0/689485355 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb1b41312d0 msgr2=0x7fb1b406e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:18.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.850+0000 7fb1b0ff9640 1 --2- 172.21.15.37:0/689485355 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb1b41312d0 0x7fb1b406e870 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4011cf0 tx=0x7fb1a400f610 comp rx=0 tx=0).stop 2024-04-26T00:23:18.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.850+0000 7fb1b0ff9640 1 -- 172.21.15.37:0/689485355 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb1b41312d0 msgr2=0x7fb1b406e870 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:23:18.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.850+0000 7fb1b0ff9640 1 --2- 172.21.15.37:0/689485355 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb1b41312d0 0x7fb1b406e870 secure :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4011cf0 tx=0x7fb1a400f610 comp rx=0 tx=0).stop 2024-04-26T00:23:18.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.850+0000 7fb1b3fff640 1 --2- 172.21.15.37:0/689485355 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb194035e80 0x7fb194038340 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb1b409bfd0 tx=0x7fb1ac009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:18.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.858+0000 7fb1ba0cf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:23:18.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.858+0000 7fb1ba0cf640 1 --2- 172.21.15.37:0/689485355 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb1b406edb0 0x7fb1b406f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:18.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.858+0000 7fb1ba0cf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:18.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.858+0000 7fb1ba0cf640 1 -- 172.21.15.37:0/689485355 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb1b40a3770 con 0x7fb1b406edb0 2024-04-26T00:23:18.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.858+0000 7fb1b37fe640 1 --2- 172.21.15.37:0/689485355 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb1b406edb0 0x7fb1b406f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:18.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.858+0000 7fb1b37fe640 10 monclient: get_auth_request con 0x7fb1b406edb0 auth_method 0 2024-04-26T00:23:18.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.858+0000 7fb1b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:18.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.858+0000 7fb1b37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:18.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.858+0000 7fb1b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:18.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.858+0000 7fb1b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:18.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.858+0000 7fb1b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:18.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.858+0000 7fb1b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:18.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.858+0000 7fb1b37fe640 10 monclient(hunting): handle_auth_done global_id 34208 payload 824 2024-04-26T00:23:18.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.858+0000 7fb1b37fe640 1 -- 172.21.15.37:0/689485355 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb1b41318f0 con 0x7fb1b406edb0 2024-04-26T00:23:18.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.858+0000 7fb1b37fe640 1 -- 172.21.15.37:0/689485355 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb1b406edb0 msgr2=0x7fb1b406f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:18.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.858+0000 7fb1b37fe640 1 --2- 172.21.15.37:0/689485355 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb1b406edb0 0x7fb1b406f1c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4011cf0 tx=0x7fb1a40046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:18.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.858+0000 7fb1b0ff9640 1 -- 172.21.15.37:0/689485355 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a4003a30 con 0x7fb1b406edb0 2024-04-26T00:23:18.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.858+0000 7fb1b0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:18.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.858+0000 7fb1b0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:23:18.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.858+0000 7fb1b0ff9640 10 monclient: dump: 2024-04-26T00:23:18.843 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:18.843 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:18.843 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:18.843 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:18.843 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:18.843 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:18.843 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:18.843 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:18.843 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:18.843 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:18.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.862+0000 7fb1b0ff9640 1 -- 172.21.15.37:0/689485355 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4718 (secure 0 0 0) 0x7fb1a4003a30 con 0x7fb1b406edb0 2024-04-26T00:23:18.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.862+0000 7fb1b0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:23:18.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.862+0000 7fb1b0ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:23:18.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.862+0000 7fb1b0ff9640 1 -- 172.21.15.37:0/689485355 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb1b406edb0 msgr2=0x7fb1b406f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:18.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.862+0000 7fb1b0ff9640 1 --2- 172.21.15.37:0/689485355 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb1b406edb0 0x7fb1b406f1c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4011cf0 tx=0x7fb1a40046d0 comp rx=0 tx=0).stop 2024-04-26T00:23:18.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.862+0000 7fb1b0ff9640 1 -- 172.21.15.37:0/689485355 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb1b406edb0 msgr2=0x7fb1b406f1c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:23:18.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.862+0000 7fb1b0ff9640 1 --2- 172.21.15.37:0/689485355 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb1b406edb0 0x7fb1b406f1c0 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4011cf0 tx=0x7fb1a40046d0 comp rx=0 tx=0).stop 2024-04-26T00:23:18.844 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:23:18.844 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:23:18.844 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:23:18.845 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:23:18.845 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 28, 2024-04-26T00:23:18.845 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 59135, 2024-04-26T00:23:18.845 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:23:18.845 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:23:18.845 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:23:18.845 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:23:18.845 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:18.845 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 4, 2024-04-26T00:23:18.845 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:23:18.845 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:23:18.845 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:23:18.845 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:23:18.845 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:23:18.845 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:23:18.846 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:23:18.846 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:23:18.846 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:23:18.846 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:23:18.846 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:23:18.846 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:23:18.846 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:23:18.846 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:23:18.846 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:18.846 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:23:18.846 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:23:18.846 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:23:18.846 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:23:18.846 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:23:18.847 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:23:18.847 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:23:18.847 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:23:18.847 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:23:18.847 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:23:18.847 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:23:18.847 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:23:18.847 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:23:18.847 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:18.847 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:18.847 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:23:18.847 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:23:18.847 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:23:18.847 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:23:18.847 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:23:18.848 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:23:18.848 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:23:18.848 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:23:18.848 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:23:18.848 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:23:18.848 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:23:18.848 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:23:18.848 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:23:18.848 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:23:18.848 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:23:18.848 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:23:18.848 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:23:18.848 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:23:18.848 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:23:18.848 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:23:18.849 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:23:18.849 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:23:18.849 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:23:18.849 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:23:18.849 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:23:18.849 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:23:18.849 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:23:18.849 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:23:18.849 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:18.849 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:23:18.849 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:18.849 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:23:18.849 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:18.849 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:23:18.850 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:23:18.850 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:23:18.850 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:23:18.850 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:18.850 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:23:18.850 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:23:18.850 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:18.850 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:18.850 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:18.850 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:23:18.850 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:23:18.850 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:18.850 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:18.850 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:18.850 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:18.851 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:23:18.851 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:23:18.851 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:23:18.851 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:23:18.851 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:23:18.851 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:18.851 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:18.851 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:23:18.851 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:23:18.851 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:23:18.851 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:23:18.851 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:18.851 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:23:18.851 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:23:18.852 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:18.852 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:18.852 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:18.852 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:23:18.852 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:23:18.852 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:18.852 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:18.852 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:18.852 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:18.852 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:23:18.852 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:23:18.852 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:23:18.852 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:23:18.852 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:23:18.852 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:18.853 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:18.853 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:23:18.853 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:23:18.853 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:23:18.853 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:23:18.853 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:18.853 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:23:18.853 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:23:18.853 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:18.853 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:18.853 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:18.853 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:23:18.853 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:23:18.853 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:18.854 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:18.854 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:18.854 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:18.854 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:23:18.854 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:23:18.854 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:23:18.854 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:23:18.854 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:23:18.854 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:18.854 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:18.854 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:18.854 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:23:18.854 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:23:18.854 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:18.854 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:23:18.855 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:23:18.855 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:23:18.855 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:18.855 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:18.855 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:23:18.855 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:18.855 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:23:18.855 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:23:18.855 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:23:18.855 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:18.855 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:18.855 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:23:18.855 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:18.855 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:23:18.856 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:23:18.856 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:23:18.856 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:18.856 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:18.856 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:18.856 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:23:18.856 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:23:18.856 INFO:tasks.rados.rados.0.smithi045.stdout:619: writing smithi04526579-1 from 736206 to 1190097 tid 1 2024-04-26T00:23:18.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.862+0000 7fb1ba0cf640 1 -- 172.21.15.37:0/689485355 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb194035e80 msgr2=0x7fb194038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:18.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.862+0000 7fb1ba0cf640 1 --2- 172.21.15.37:0/689485355 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb194035e80 0x7fb194038340 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb1b409bfd0 tx=0x7fb1ac009040 comp rx=0 tx=0).stop 2024-04-26T00:23:18.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.862+0000 7fb1ba0cf640 10 monclient: shutdown 2024-04-26T00:23:18.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.862+0000 7fb1ba0cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:23:18.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.862+0000 7fb1ba0cf640 1 -- 172.21.15.37:0/689485355 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb1b409bb80 msgr2=0x7fb1b4072140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:18.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.862+0000 7fb1ba0cf640 1 --2- 172.21.15.37:0/689485355 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb1b409bb80 0x7fb1b4072140 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb19c01a9d0 tx=0x7fb19c0190c0 comp rx=0 tx=0).stop 2024-04-26T00:23:18.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.862+0000 7fb1b3fff640 1 -- 172.21.15.37:0/689485355 reap_dead start 2024-04-26T00:23:18.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.862+0000 7fb1b3fff640 1 -- 172.21.15.37:0/689485355 reap_dead start 2024-04-26T00:23:18.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.862+0000 7fb1ba0cf640 1 -- 172.21.15.37:0/689485355 shutdown_connections 2024-04-26T00:23:18.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.862+0000 7fb1ba0cf640 1 -- 172.21.15.37:0/689485355 >> 172.21.15.37:0/689485355 conn(0x7fb1b40a5710 msgr2=0x7fb1b4098230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:23:18.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.862+0000 7fb1ba0cf640 1 -- 172.21.15.37:0/689485355 shutdown_connections 2024-04-26T00:23:18.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:18.862+0000 7fb1ba0cf640 1 -- 172.21.15.37:0/689485355 wait complete. 2024-04-26T00:23:18.859 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:23:18.859 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:23:18.859 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:23:18.868 INFO:tasks.rados.rados.0.smithi045.stdout:619: writing smithi04526579-1 from 1881413 to 2336206 tid 2 2024-04-26T00:23:18.868 INFO:tasks.rados.rados.0.smithi045.stdout:620: read oid 7 snap -1 2024-04-26T00:23:18.869 INFO:tasks.rados.rados.0.smithi045.stdout:620: expect deleted 2024-04-26T00:23:18.869 INFO:tasks.rados.rados.0.smithi045.stdout:621: read oid 4 snap -1 2024-04-26T00:23:18.869 INFO:tasks.rados.rados.0.smithi045.stdout:621: expect (ObjNum 202 snap 53 seq_num 202) 2024-04-26T00:23:18.869 INFO:tasks.rados.rados.0.smithi045.stdout:622: delete oid 11 current snap is 68 2024-04-26T00:23:18.869 INFO:tasks.rados.rados.0.smithi045.stdout:618: finishing write tid 2 to smithi04526579-47 2024-04-26T00:23:18.869 INFO:tasks.rados.rados.0.smithi045.stdout:618: finishing write tid 3 to smithi04526579-47 2024-04-26T00:23:18.869 INFO:tasks.rados.rados.0.smithi045.stdout:618: finishing write tid 4 to smithi04526579-47 2024-04-26T00:23:18.869 INFO:tasks.rados.rados.0.smithi045.stdout:618: oid 47 updating version 0 to 266 2024-04-26T00:23:18.869 INFO:tasks.rados.rados.0.smithi045.stdout:618: oid 47 updating version 266 to 268 2024-04-26T00:23:18.869 INFO:tasks.rados.rados.0.smithi045.stdout:618: oid 47 version 268 is already newer than 267 2024-04-26T00:23:18.869 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 268 (ObjNum 257 snap 68 seq_num 257) dirty exists 2024-04-26T00:23:18.869 INFO:tasks.rados.rados.0.smithi045.stdout:618: left oid 47 (ObjNum 257 snap 68 seq_num 257) 2024-04-26T00:23:18.869 INFO:tasks.rados.rados.0.smithi045.stdout:619: finishing write tid 1 to smithi04526579-1 2024-04-26T00:23:18.874 INFO:tasks.rados.rados.0.smithi045.stdout:618: done (4 left) 2024-04-26T00:23:18.874 INFO:tasks.rados.rados.0.smithi045.stdout:620: done (3 left) 2024-04-26T00:23:18.874 INFO:tasks.rados.rados.0.smithi045.stdout:622: done (2 left) 2024-04-26T00:23:18.874 INFO:tasks.rados.rados.0.smithi045.stdout:623: write oid 35 current snap is 68 2024-04-26T00:23:18.874 INFO:tasks.rados.rados.0.smithi045.stdout:623: seq_num 260 ranges {769820=720742,2114730=255090} 2024-04-26T00:23:18.893 INFO:tasks.rados.rados.0.smithi045.stdout:623: writing smithi04526579-35 from 769820 to 1490562 tid 1 2024-04-26T00:23:18.904 INFO:tasks.rados.rados.0.smithi045.stdout:623: writing smithi04526579-35 from 2114730 to 2369820 tid 2 2024-04-26T00:23:18.904 INFO:tasks.rados.rados.0.smithi045.stdout:624: copy_from oid 27 from oid 33 current snap is 68 2024-04-26T00:23:18.904 INFO:tasks.rados.rados.0.smithi045.stdout:619: finishing write tid 2 to smithi04526579-1 2024-04-26T00:23:18.904 INFO:tasks.rados.rados.0.smithi045.stdout:619: finishing write tid 3 to smithi04526579-1 2024-04-26T00:23:18.905 INFO:tasks.rados.rados.0.smithi045.stdout:619: finishing write tid 4 to smithi04526579-1 2024-04-26T00:23:18.905 INFO:tasks.rados.rados.0.smithi045.stdout:619: oid 1 updating version 0 to 316 2024-04-26T00:23:18.905 INFO:tasks.rados.rados.0.smithi045.stdout:619: oid 1 version 316 is already newer than 314 2024-04-26T00:23:18.905 INFO:tasks.rados.rados.0.smithi045.stdout:619: oid 1 version 316 is already newer than 315 2024-04-26T00:23:18.905 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 316 (ObjNum 258 snap 68 seq_num 258) dirty exists 2024-04-26T00:23:18.905 INFO:tasks.rados.rados.0.smithi045.stdout:619: left oid 1 (ObjNum 258 snap 68 seq_num 258) 2024-04-26T00:23:18.905 INFO:tasks.rados.rados.0.smithi045.stdout:623: finishing write tid 1 to smithi04526579-35 2024-04-26T00:23:18.905 INFO:tasks.rados.rados.0.smithi045.stdout:619: done (3 left) 2024-04-26T00:23:18.905 INFO:tasks.rados.rados.0.smithi045.stdout:625: copy_from oid 36 from oid 11 current snap is 68 2024-04-26T00:23:18.905 INFO:tasks.rados.rados.0.smithi045.stdout:626: snap_remove snap 37 2024-04-26T00:23:18.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.002+0000 7f0f86b1f640 10 monclient: get_monmap_and_config 2024-04-26T00:23:18.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.002+0000 7f0f86b1f640 10 monclient: build_initial_monmap 2024-04-26T00:23:18.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.002+0000 7f0f86b1f640 10 monclient: monmap: 2024-04-26T00:23:18.979 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:23:18.979 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:18.979 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:23:19.004475+0000 2024-04-26T00:23:18.979 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:23:19.004475+0000 2024-04-26T00:23:18.979 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:23:18.979 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:18.979 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:23:18.979 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:23:18.980 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:23:18.980 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:23:18.980 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:23:18.980 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:18.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.002+0000 7f0f86b1f640 1 Processor -- start 2024-04-26T00:23:18.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.002+0000 7f0f86b1f640 1 -- start start 2024-04-26T00:23:18.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.002+0000 7f0f86b1f640 10 monclient: init 2024-04-26T00:23:18.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.002+0000 7f0f86b1f640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:18.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.002+0000 7f0f86b1f640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-04-26T00:23:18.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.002+0000 7f0f86b1f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0f80130eb0 addr v1:172.21.15.37:6789/0 2024-04-26T00:23:18.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.002+0000 7f0f86b1f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f8009ef20 0x7f0f80138eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:18.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.002+0000 7f0f86b1f640 10 monclient(hunting): picked mon.noname-b con 0x7f0f8009ef20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:23:18.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.002+0000 7f0f86b1f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0f801393f0 addr v1:172.21.15.45:6789/0 2024-04-26T00:23:18.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.002+0000 7f0f86b1f640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f8009c190 con 0x7f0f80130eb0 2024-04-26T00:23:18.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.002+0000 7f0f86b1f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f800a11c0 con 0x7f0f801393f0 2024-04-26T00:23:18.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.002+0000 7f0f86b1f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:18.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.002+0000 7f0f86b1f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0f8009df60 con 0x7f0f8009ef20 2024-04-26T00:23:18.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.002+0000 7f0f86b1f640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:18.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.002+0000 7f0f86b1f640 10 monclient(hunting): authenticate will time out at 1180.310705s 2024-04-26T00:23:18.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.002+0000 7f0f848bb640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f0f80130eb0 0x7f0f80131290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:42444/0 (socket says 172.21.15.37:42444) 2024-04-26T00:23:18.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.002+0000 7f0f77fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f8009ef20 0x7f0f80138eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:18.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.002+0000 7f0f848bb640 1 -- 172.21.15.37:0/2982658753 learned_addr learned my addr 172.21.15.37:0/2982658753 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:23:18.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.002+0000 7f0f77fff640 10 monclient(hunting): get_auth_request con 0x7f0f8009ef20 auth_method 0 2024-04-26T00:23:18.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.002+0000 7f0f77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:18.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.002+0000 7f0f77fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:18.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.002+0000 7f0f77fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:18.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.002+0000 7f0f77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:18.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.002+0000 7f0f77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:18.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.002+0000 7f0f77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:18.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.002+0000 7f0f77fff640 10 monclient(hunting): handle_auth_done global_id 34211 payload 274 2024-04-26T00:23:18.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.002+0000 7f0f77fff640 1 -- 172.21.15.37:0/2982658753 >> v1:172.21.15.45:6789/0 conn(0x7f0f801393f0 legacy=0x7f0f8013b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:23:18.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.002+0000 7f0f77fff640 1 -- 172.21.15.37:0/2982658753 >> v1:172.21.15.37:6789/0 conn(0x7f0f80130eb0 legacy=0x7f0f80131290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:23:18.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.002+0000 7f0f77fff640 10 monclient: _finish_hunting 0 2024-04-26T00:23:18.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.002+0000 7f0f77fff640 1 monclient: found mon.noname-b 2024-04-26T00:23:18.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.002+0000 7f0f77fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:18.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.002+0000 7f0f77fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:23:18.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.002+0000 7f0f77fff640 1 -- 172.21.15.37:0/2982658753 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f8009dc80 con 0x7f0f8009ef20 2024-04-26T00:23:18.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.006+0000 7f0f77fff640 1 -- 172.21.15.37:0/2982658753 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f8009ef20 msgr2=0x7f0f80138eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:18.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.006+0000 7f0f77fff640 1 --2- 172.21.15.37:0/2982658753 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f8009ef20 0x7f0f80138eb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0f700097f0 tx=0x7f0f7001c2f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfd0d32bee6ff460 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:18.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.006+0000 7f0f777fe640 1 -- 172.21.15.37:0/2982658753 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f700b0070 con 0x7f0f8009ef20 2024-04-26T00:23:18.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.006+0000 7f0f777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:18.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.006+0000 7f0f777fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:23:18.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.006+0000 7f0f777fe640 10 monclient: dump: 2024-04-26T00:23:18.987 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:18.987 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:18.987 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:18.987 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:18.987 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:18.987 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:18.987 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:18.987 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:18.988 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:18.988 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:18.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.006+0000 7f0f777fe640 10 monclient: _finish_auth 0 2024-04-26T00:23:18.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.006+0000 7f0f777fe640 10 monclient: _check_auth_tickets 2024-04-26T00:23:18.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.006+0000 7f0f777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:18.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.006+0000 7f0f777fe640 1 -- 172.21.15.37:0/2982658753 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f700ab600 con 0x7f0f8009ef20 2024-04-26T00:23:18.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.006+0000 7f0f777fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:23:18.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.006+0000 7f0f777fe640 1 -- 172.21.15.37:0/2982658753 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f700ab9d0 con 0x7f0f8009ef20 2024-04-26T00:23:18.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.006+0000 7f0f86b1f640 5 monclient: authenticate success, global_id 34211 2024-04-26T00:23:18.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.006+0000 7f0f86b1f640 10 monclient: get_monmap_and_config success 2024-04-26T00:23:18.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.006+0000 7f0f86b1f640 4 set_mon_vals no callback set 2024-04-26T00:23:18.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.006+0000 7f0f777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:18.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.006+0000 7f0f777fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:23:18.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.006+0000 7f0f777fe640 10 monclient: dump: 2024-04-26T00:23:18.988 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:18.989 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:18.989 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:18.989 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:18.989 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:18.989 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:18.989 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:18.989 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:18.989 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:18.989 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:18.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f86b1f640 10 monclient: shutdown 2024-04-26T00:23:18.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f86b1f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:23:18.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f86b1f640 1 -- 172.21.15.37:0/2982658753 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f8009ef20 msgr2=0x7f0f80138eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:18.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f86b1f640 1 --2- 172.21.15.37:0/2982658753 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f8009ef20 0x7f0f80138eb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0f700097f0 tx=0x7f0f7001c2f0 comp rx=0 tx=0).stop 2024-04-26T00:23:18.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f86b1f640 1 -- 172.21.15.37:0/2982658753 shutdown_connections 2024-04-26T00:23:18.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f86b1f640 1 --2- 172.21.15.37:0/2982658753 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f8009ef20 0x7f0f80138eb0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:18.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f86b1f640 1 -- 172.21.15.37:0/2982658753 >> 172.21.15.37:0/2982658753 conn(0x7f0f800a1d50 msgr2=0x7f0f800a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:23:18.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f86b1f640 1 -- 172.21.15.37:0/2982658753 shutdown_connections 2024-04-26T00:23:18.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f86b1f640 1 -- 172.21.15.37:0/2982658753 wait complete. 2024-04-26T00:23:18.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f86b1f640 10 monclient: build_initial_monmap 2024-04-26T00:23:18.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f86b1f640 10 monclient: monmap: 2024-04-26T00:23:18.995 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:23:18.995 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:23:18.995 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:23:18.995 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:23:18.995 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:23:18.995 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:23:18.995 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:23:18.995 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:23:18.995 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:23:18.995 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:18.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f86b1f640 1 Processor -- start 2024-04-26T00:23:18.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f86b1f640 1 -- start start 2024-04-26T00:23:18.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f86b1f640 10 monclient: init 2024-04-26T00:23:18.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f86b1f640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:18.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f86b1f640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:23:18.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f86b1f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0f8009ef20 0x7f0f80131e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:18.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f86b1f640 10 monclient(hunting): picked mon.noname-a con 0x7f0f8009ef20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:23:18.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f86b1f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f80130eb0 0x7f0f80132360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:18.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f86b1f640 10 monclient(hunting): picked mon.noname-b con 0x7f0f80130eb0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:23:18.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f86b1f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f801393f0 0x7f0f801328a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:18.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f86b1f640 10 monclient(hunting): picked mon.noname-c con 0x7f0f801393f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:23:18.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f86b1f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:18.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f86b1f640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f0f8009ffe0 con 0x7f0f8009ef20 2024-04-26T00:23:18.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f86b1f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:18.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f86b1f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0f8009fe60 con 0x7f0f80130eb0 2024-04-26T00:23:18.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f86b1f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:18.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f86b1f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0f800a0160 con 0x7f0f801393f0 2024-04-26T00:23:18.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f86b1f640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:18.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f86b1f640 10 monclient(hunting): authenticate will time out at 1180.318083s 2024-04-26T00:23:18.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f848bb640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0f8009ef20 0x7f0f80131e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:18.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f848bb640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0f8009ef20 0x7f0f80131e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:48382/0 (socket says 172.21.15.37:48382) 2024-04-26T00:23:18.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f850bc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f801393f0 0x7f0f801328a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:18.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f848bb640 1 -- 172.21.15.37:0/3847399626 learned_addr learned my addr 172.21.15.37:0/3847399626 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:23:18.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f848bb640 10 monclient(hunting): get_auth_request con 0x7f0f8009ef20 auth_method 0 2024-04-26T00:23:18.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f848bb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:18.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f848bb640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:18.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f848bb640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:18.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f850bc640 10 monclient(hunting): get_auth_request con 0x7f0f801393f0 auth_method 0 2024-04-26T00:23:18.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f850bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:18.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f850bc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:18.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f850bc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:18.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f850bc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:18.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f850bc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:18.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f850bc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:18.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f848bb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:18.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f848bb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:18.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f77fff640 1 --2- 172.21.15.37:0/3847399626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f80130eb0 0x7f0f80132360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:18.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f848bb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:18.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f77fff640 10 monclient(hunting): get_auth_request con 0x7f0f80130eb0 auth_method 0 2024-04-26T00:23:18.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:18.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f77fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:18.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f77fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:18.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f850bc640 10 monclient(hunting): handle_auth_done global_id 34214 payload 824 2024-04-26T00:23:18.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f850bc640 1 -- 172.21.15.37:0/3847399626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f80130eb0 msgr2=0x7f0f80132360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:18.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f850bc640 1 --2- 172.21.15.37:0/3847399626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f80130eb0 0x7f0f80132360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:18.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f850bc640 1 -- 172.21.15.37:0/3847399626 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0f8009ef20 msgr2=0x7f0f80131e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:18.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f850bc640 1 --2- 172.21.15.37:0/3847399626 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0f8009ef20 0x7f0f80131e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:18.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f850bc640 10 monclient: _finish_hunting 0 2024-04-26T00:23:18.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f850bc640 1 monclient: found mon.noname-c 2024-04-26T00:23:18.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f850bc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:18.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f850bc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:23:18.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f850bc640 1 -- 172.21.15.37:0/3847399626 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f801343c0 con 0x7f0f801393f0 2024-04-26T00:23:18.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.010+0000 7f0f848bb640 1 --2- 172.21.15.37:0/3847399626 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0f8009ef20 0x7f0f80131e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:23:18.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f850bc640 1 -- 172.21.15.37:0/3847399626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f801393f0 msgr2=0x7f0f801328a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:18.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f850bc640 1 --2- 172.21.15.37:0/3847399626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f801393f0 0x7f0f801328a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0f7800c750 tx=0x7f0f7800cc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:18.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f757fa640 1 -- 172.21.15.37:0/3847399626 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f78016020 con 0x7f0f801393f0 2024-04-26T00:23:18.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f757fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:18.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f757fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:23:18.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f757fa640 10 monclient: dump: 2024-04-26T00:23:19.000 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:19.000 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:19.000 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:19.000 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:19.000 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:19.000 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:19.000 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:19.000 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:19.000 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:19.000 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:19.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f757fa640 10 monclient: _finish_auth 0 2024-04-26T00:23:19.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f757fa640 10 monclient: _check_auth_tickets 2024-04-26T00:23:19.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:19.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f757fa640 1 -- 172.21.15.37:0/3847399626 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f7800aa60 con 0x7f0f801393f0 2024-04-26T00:23:19.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f757fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:23:19.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f757fa640 1 -- 172.21.15.37:0/3847399626 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f7800ae50 con 0x7f0f801393f0 2024-04-26T00:23:19.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f86b1f640 5 monclient: authenticate success, global_id 34214 2024-04-26T00:23:19.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f86b1f640 10 monclient: _renew_subs 2024-04-26T00:23:19.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f86b1f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:19.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f86b1f640 1 -- 172.21.15.37:0/3847399626 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f80133880 con 0x7f0f801393f0 2024-04-26T00:23:19.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f757fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:19.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f757fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:23:19.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f757fa640 10 monclient: dump: 2024-04-26T00:23:19.003 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:19.003 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:19.003 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:19.003 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:19.003 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:19.003 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:19.003 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:19.003 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:19.003 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:19.003 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:19.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f76ffd640 4 set_mon_vals no callback set 2024-04-26T00:23:19.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f86b1f640 10 monclient: _renew_subs 2024-04-26T00:23:19.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f86b1f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:19.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f86b1f640 1 -- 172.21.15.37:0/3847399626 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f800a49b0 con 0x7f0f801393f0 2024-04-26T00:23:19.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f86b1f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:19.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f86b1f640 1 --2- 172.21.15.37:0/3847399626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f80135a30 0x7f0f80132de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:19.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f86b1f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:19.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f86b1f640 1 -- 172.21.15.37:0/3847399626 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0f8004f510 con 0x7f0f80135a30 2024-04-26T00:23:19.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f757fa640 1 -- 172.21.15.37:0/3847399626 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f0f78019020 con 0x7f0f801393f0 2024-04-26T00:23:19.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f848bb640 1 --2- 172.21.15.37:0/3847399626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f80135a30 0x7f0f80132de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:19.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f848bb640 10 monclient: get_auth_request con 0x7f0f80135a30 auth_method 0 2024-04-26T00:23:19.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f848bb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:19.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f848bb640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:19.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f848bb640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:19.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f757fa640 1 --2- 172.21.15.37:0/3847399626 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0f6c036070 0x7f0f6c038530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:19.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f757fa640 1 -- 172.21.15.37:0/3847399626 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(144..144 src has 1..144) ==== 3931+0+0 (secure 0 0 0) 0x7f0f78048bf0 con 0x7f0f801393f0 2024-04-26T00:23:19.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f848bb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:19.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f848bb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:19.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f848bb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:19.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f77fff640 1 --2- 172.21.15.37:0/3847399626 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0f6c036070 0x7f0f6c038530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:19.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f77fff640 10 monclient: get_auth_request con 0x7f0f6c036070 auth_method 0 2024-04-26T00:23:19.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f848bb640 10 monclient(hunting): handle_auth_done global_id 24265 payload 824 2024-04-26T00:23:19.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.014+0000 7f0f848bb640 1 -- 172.21.15.37:0/3847399626 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0f800a0f60 con 0x7f0f80135a30 2024-04-26T00:23:19.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.018+0000 7f0f848bb640 1 -- 172.21.15.37:0/3847399626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f80135a30 msgr2=0x7f0f80132de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:19.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.018+0000 7f0f848bb640 1 --2- 172.21.15.37:0/3847399626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f80135a30 0x7f0f80132de0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c0072a0 tx=0x7f0f7c007760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:19.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.018+0000 7f0f757fa640 1 -- 172.21.15.37:0/3847399626 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c00fde0 con 0x7f0f80135a30 2024-04-26T00:23:19.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.018+0000 7f0f757fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:19.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.018+0000 7f0f757fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:23:19.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.018+0000 7f0f757fa640 10 monclient: dump: 2024-04-26T00:23:19.007 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:19.007 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:19.007 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:19.007 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:19.007 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:19.008 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:19.008 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:19.008 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:19.008 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:19.008 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:19.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.018+0000 7f0f77fff640 1 --2- 172.21.15.37:0/3847399626 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0f6c036070 0x7f0f6c038530 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0f700ab000 tx=0x7f0f70008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:19.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.018+0000 7f0f757fa640 1 -- 172.21.15.37:0/3847399626 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0f7c00fde0 con 0x7f0f80135a30 2024-04-26T00:23:19.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.018+0000 7f0f757fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:19.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.018+0000 7f0f757fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:23:19.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.018+0000 7f0f757fa640 1 -- 172.21.15.37:0/3847399626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f80135a30 msgr2=0x7f0f80132de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:19.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.018+0000 7f0f757fa640 1 --2- 172.21.15.37:0/3847399626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f80135a30 0x7f0f80132de0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c0072a0 tx=0x7f0f7c007760 comp rx=0 tx=0).stop 2024-04-26T00:23:19.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.018+0000 7f0f757fa640 1 -- 172.21.15.37:0/3847399626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f80135a30 msgr2=0x7f0f80132de0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:23:19.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.018+0000 7f0f757fa640 1 --2- 172.21.15.37:0/3847399626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f80135a30 0x7f0f80132de0 secure :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c0072a0 tx=0x7f0f7c007760 comp rx=0 tx=0).stop 2024-04-26T00:23:19.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.026+0000 7f0f86b1f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:23:19.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.026+0000 7f0f86b1f640 1 --2- 172.21.15.37:0/3847399626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f800a36b0 0x7f0f800a3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:19.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.026+0000 7f0f86b1f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:19.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.026+0000 7f0f86b1f640 1 -- 172.21.15.37:0/3847399626 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0f80135390 con 0x7f0f800a36b0 2024-04-26T00:23:19.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.026+0000 7f0f848bb640 1 --2- 172.21.15.37:0/3847399626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f800a36b0 0x7f0f800a3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:19.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.026+0000 7f0f848bb640 10 monclient: get_auth_request con 0x7f0f800a36b0 auth_method 0 2024-04-26T00:23:19.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.026+0000 7f0f848bb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:19.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.026+0000 7f0f848bb640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:19.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.026+0000 7f0f848bb640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:19.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.026+0000 7f0f848bb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:19.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.026+0000 7f0f848bb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:19.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.026+0000 7f0f848bb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:19.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.026+0000 7f0f848bb640 10 monclient(hunting): handle_auth_done global_id 24268 payload 824 2024-04-26T00:23:19.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.026+0000 7f0f848bb640 1 -- 172.21.15.37:0/3847399626 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0f8006ba50 con 0x7f0f800a36b0 2024-04-26T00:23:19.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.030+0000 7f0f848bb640 1 -- 172.21.15.37:0/3847399626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f800a36b0 msgr2=0x7f0f800a3a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:19.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.030+0000 7f0f848bb640 1 --2- 172.21.15.37:0/3847399626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f800a36b0 0x7f0f800a3a90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c0072a0 tx=0x7f0f7c004950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:19.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.030+0000 7f0f757fa640 1 -- 172.21.15.37:0/3847399626 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c006240 con 0x7f0f800a36b0 2024-04-26T00:23:19.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.030+0000 7f0f757fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:19.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.030+0000 7f0f757fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:23:19.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.030+0000 7f0f757fa640 10 monclient: dump: 2024-04-26T00:23:19.010 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:19.010 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:19.010 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:19.010 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:19.010 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:19.010 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:19.010 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:19.010 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:19.011 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:19.011 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:19.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.030+0000 7f0f757fa640 1 -- 172.21.15.37:0/3847399626 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f0f7c006240 con 0x7f0f800a36b0 2024-04-26T00:23:19.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.030+0000 7f0f757fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:23:19.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.030+0000 7f0f757fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:23:19.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.030+0000 7f0f757fa640 1 -- 172.21.15.37:0/3847399626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f800a36b0 msgr2=0x7f0f800a3a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:19.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.030+0000 7f0f757fa640 1 --2- 172.21.15.37:0/3847399626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f800a36b0 0x7f0f800a3a90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c0072a0 tx=0x7f0f7c004950 comp rx=0 tx=0).stop 2024-04-26T00:23:19.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.030+0000 7f0f757fa640 1 -- 172.21.15.37:0/3847399626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f800a36b0 msgr2=0x7f0f800a3a90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:23:19.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.030+0000 7f0f757fa640 1 --2- 172.21.15.37:0/3847399626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f800a36b0 0x7f0f800a3a90 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c0072a0 tx=0x7f0f7c004950 comp rx=0 tx=0).stop 2024-04-26T00:23:19.012 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:23:19.012 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:23:19.012 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:23:19.012 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:23:19.012 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 28, 2024-04-26T00:23:19.012 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 107197, 2024-04-26T00:23:19.012 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:23:19.013 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:23:19.013 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:23:19.013 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:23:19.013 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:19.013 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 4, 2024-04-26T00:23:19.013 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:23:19.013 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:23:19.013 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:23:19.013 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:23:19.013 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:23:19.013 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:23:19.013 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:23:19.013 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:23:19.013 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:23:19.014 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:23:19.014 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:23:19.014 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:23:19.014 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:23:19.014 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:23:19.014 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:19.014 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:23:19.014 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:23:19.014 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:23:19.014 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:23:19.014 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:23:19.014 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:23:19.014 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:23:19.014 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:23:19.014 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:23:19.015 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:23:19.015 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:23:19.015 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:23:19.015 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:23:19.015 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:19.015 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:19.015 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:23:19.015 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:23:19.015 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:23:19.015 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:23:19.015 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:23:19.015 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:23:19.015 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:23:19.015 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:23:19.015 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:23:19.016 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:23:19.016 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:23:19.016 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:23:19.016 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:23:19.016 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:23:19.016 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:23:19.016 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:23:19.016 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:23:19.016 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:23:19.016 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:23:19.016 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:23:19.016 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:23:19.016 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:23:19.016 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:23:19.016 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:23:19.017 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:23:19.017 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:23:19.017 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:23:19.017 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:23:19.017 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:19.017 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:23:19.017 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:19.017 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:23:19.017 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:19.017 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:23:19.017 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:23:19.017 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:23:19.017 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:23:19.017 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:19.018 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:23:19.018 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:23:19.018 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:19.018 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:19.018 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:19.018 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:23:19.018 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:23:19.018 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:19.018 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:19.018 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:19.018 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:19.018 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:23:19.018 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:23:19.018 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:23:19.018 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:23:19.019 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:23:19.019 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:19.019 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:19.019 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:23:19.019 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:23:19.019 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:23:19.019 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:23:19.019 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:19.019 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:23:19.019 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:23:19.019 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:19.019 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:19.019 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:19.019 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:23:19.019 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:23:19.020 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:19.020 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:19.020 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:19.020 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:19.020 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:23:19.020 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:23:19.020 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:23:19.020 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:23:19.020 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:23:19.020 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:19.020 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:19.020 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:23:19.020 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:23:19.020 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:23:19.021 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:23:19.021 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:19.021 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:23:19.021 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:23:19.021 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:19.021 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:19.021 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:19.021 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:23:19.021 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:23:19.021 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:19.021 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:19.021 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:19.021 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:19.021 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:23:19.022 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:23:19.022 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:23:19.022 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:23:19.022 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:23:19.022 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:19.022 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:19.022 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:19.022 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:23:19.022 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:23:19.022 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:19.022 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:23:19.022 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:23:19.022 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:23:19.022 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:19.022 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:19.023 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:23:19.023 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:19.023 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:23:19.023 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:23:19.023 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:23:19.023 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:19.023 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:19.023 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:23:19.023 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:19.023 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:23:19.023 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:23:19.023 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:23:19.023 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:19.023 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:19.023 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:23:19.024 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:19.024 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:23:19.024 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:23:19.024 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:23:19.024 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:19.024 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:19.024 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:19.024 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:23:19.024 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:23:19.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.030+0000 7f0f86b1f640 1 -- 172.21.15.37:0/3847399626 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0f6c036070 msgr2=0x7f0f6c038530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:19.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.030+0000 7f0f86b1f640 1 --2- 172.21.15.37:0/3847399626 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0f6c036070 0x7f0f6c038530 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0f700ab000 tx=0x7f0f70008040 comp rx=0 tx=0).stop 2024-04-26T00:23:19.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.030+0000 7f0f86b1f640 10 monclient: shutdown 2024-04-26T00:23:19.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.030+0000 7f0f86b1f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:23:19.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.030+0000 7f0f86b1f640 1 -- 172.21.15.37:0/3847399626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f801393f0 msgr2=0x7f0f801328a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:19.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.030+0000 7f0f86b1f640 1 --2- 172.21.15.37:0/3847399626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f801393f0 0x7f0f801328a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0f7800c750 tx=0x7f0f7800cc10 comp rx=0 tx=0).stop 2024-04-26T00:23:19.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.030+0000 7f0f850bc640 1 -- 172.21.15.37:0/3847399626 reap_dead start 2024-04-26T00:23:19.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.030+0000 7f0f86b1f640 1 -- 172.21.15.37:0/3847399626 shutdown_connections 2024-04-26T00:23:19.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.030+0000 7f0f86b1f640 1 --2- 172.21.15.37:0/3847399626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f801393f0 0x7f0f801328a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:19.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.030+0000 7f0f86b1f640 1 -- 172.21.15.37:0/3847399626 >> 172.21.15.37:0/3847399626 conn(0x7f0f800a1d50 msgr2=0x7f0f8012ef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:23:19.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.030+0000 7f0f86b1f640 1 -- 172.21.15.37:0/3847399626 shutdown_connections 2024-04-26T00:23:19.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.030+0000 7f0f86b1f640 1 -- 172.21.15.37:0/3847399626 wait complete. 2024-04-26T00:23:19.026 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:23:19.026 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-26T00:23:19.027 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-26T00:23:19.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b475b640 10 monclient: get_monmap_and_config 2024-04-26T00:23:19.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b475b640 10 monclient: build_initial_monmap 2024-04-26T00:23:19.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b475b640 10 monclient: monmap: 2024-04-26T00:23:19.143 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:23:19.143 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:19.143 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:23:19.168374+0000 2024-04-26T00:23:19.143 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:23:19.168374+0000 2024-04-26T00:23:19.143 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:23:19.143 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:19.143 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:23:19.143 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:23:19.143 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:23:19.143 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:23:19.144 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:23:19.144 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:19.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b475b640 1 Processor -- start 2024-04-26T00:23:19.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b475b640 1 -- start start 2024-04-26T00:23:19.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b475b640 10 monclient: init 2024-04-26T00:23:19.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b475b640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:19.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b475b640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-04-26T00:23:19.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b475b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb3ac0a0260 addr v1:172.21.15.45:6789/0 2024-04-26T00:23:19.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b475b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb3ac137e40 0x7fb3ac13a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:19.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b475b640 10 monclient(hunting): picked mon.noname-c con 0x7fb3ac137e40 addr v2:172.21.15.45:3300/0 2024-04-26T00:23:19.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b475b640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb3ac13a770 0x7fb3ac13cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:19.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b475b640 10 monclient(hunting): picked mon.noname-a con 0x7fb3ac13a770 addr v2:172.21.15.37:3300/0 2024-04-26T00:23:19.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b475b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3ac0a1130 con 0x7fb3ac0a0260 2024-04-26T00:23:19.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b475b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:19.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b475b640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fb3ac0a0d50 con 0x7fb3ac13a770 2024-04-26T00:23:19.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b475b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:19.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b475b640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fb3ac09ec10 con 0x7fb3ac137e40 2024-04-26T00:23:19.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b475b640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:19.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b475b640 10 monclient(hunting): authenticate will time out at 1180.473991s 2024-04-26T00:23:19.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b2cf8640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb3ac13a770 0x7fb3ac13cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:19.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b2cf8640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb3ac13a770 0x7fb3ac13cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:48384/0 (socket says 172.21.15.37:48384) 2024-04-26T00:23:19.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b2cf8640 1 -- 172.21.15.37:0/3812209542 learned_addr learned my addr 172.21.15.37:0/3812209542 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:23:19.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b2cf8640 10 monclient(hunting): get_auth_request con 0x7fb3ac13a770 auth_method 0 2024-04-26T00:23:19.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b2cf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:19.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b2cf8640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:19.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b2cf8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:19.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b2cf8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:19.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b2cf8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:19.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b2cf8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:19.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b1cf6640 1 --2- 172.21.15.37:0/3812209542 >> v2:172.21.15.45:3300/0 conn(0x7fb3ac137e40 0x7fb3ac13a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:19.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b1cf6640 10 monclient(hunting): get_auth_request con 0x7fb3ac137e40 auth_method 0 2024-04-26T00:23:19.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b1cf6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:19.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b1cf6640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:19.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b1cf6640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:19.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b2cf8640 10 monclient(hunting): handle_auth_done global_id 44112 payload 274 2024-04-26T00:23:19.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b2cf8640 1 -- 172.21.15.37:0/3812209542 >> v2:172.21.15.45:3300/0 conn(0x7fb3ac137e40 msgr2=0x7fb3ac13a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:23:19.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b2cf8640 1 --2- 172.21.15.37:0/3812209542 >> v2:172.21.15.45:3300/0 conn(0x7fb3ac137e40 0x7fb3ac13a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:19.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b2cf8640 1 -- 172.21.15.37:0/3812209542 >> v1:172.21.15.45:6789/0 conn(0x7fb3ac0a0260 legacy=0x7fb3ac0a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:23:19.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b2cf8640 10 monclient: _finish_hunting 0 2024-04-26T00:23:19.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b2cf8640 1 monclient: found mon.noname-a 2024-04-26T00:23:19.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b2cf8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:19.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b2cf8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:23:19.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.166+0000 7fb3b2cf8640 1 -- 172.21.15.37:0/3812209542 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3ac06cff0 con 0x7fb3ac13a770 2024-04-26T00:23:19.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b2cf8640 1 -- 172.21.15.37:0/3812209542 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3ac13a770 msgr2=0x7fb3ac13cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:19.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b2cf8640 1 --2- 172.21.15.37:0/3812209542 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3ac13a770 0x7fb3ac13cb60 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac09f690 tx=0x7fb3a401c690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=236be2a4f22a58ae server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:19.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b14f5640 1 -- 172.21.15.37:0/3812209542 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a40a9460 con 0x7fb3ac13a770 2024-04-26T00:23:19.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b14f5640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:19.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b14f5640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:23:19.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b14f5640 10 monclient: dump: 2024-04-26T00:23:19.151 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:19.151 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:19.151 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:19.151 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:19.151 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:19.151 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:19.151 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:19.151 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:19.151 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:19.151 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:19.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b14f5640 10 monclient: _finish_auth 0 2024-04-26T00:23:19.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b14f5640 10 monclient: _check_auth_tickets 2024-04-26T00:23:19.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b14f5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:19.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b14f5640 1 -- 172.21.15.37:0/3812209542 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3a40ae070 con 0x7fb3ac13a770 2024-04-26T00:23:19.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b14f5640 10 monclient: handle_config config(0 keys) 2024-04-26T00:23:19.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b14f5640 1 -- 172.21.15.37:0/3812209542 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a40a9820 con 0x7fb3ac13a770 2024-04-26T00:23:19.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b14f5640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:19.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b14f5640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:23:19.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b14f5640 10 monclient: dump: 2024-04-26T00:23:19.152 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:19.152 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:19.152 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:19.152 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:19.152 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:19.153 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:19.153 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:19.153 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:19.153 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:19.153 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:19.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b475b640 5 monclient: authenticate success, global_id 44112 2024-04-26T00:23:19.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b475b640 10 monclient: get_monmap_and_config success 2024-04-26T00:23:19.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b475b640 4 set_mon_vals no callback set 2024-04-26T00:23:19.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b475b640 10 monclient: shutdown 2024-04-26T00:23:19.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b475b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:23:19.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b475b640 1 -- 172.21.15.37:0/3812209542 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3ac13a770 msgr2=0x7fb3ac13cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:19.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b475b640 1 --2- 172.21.15.37:0/3812209542 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3ac13a770 0x7fb3ac13cb60 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac09f690 tx=0x7fb3a401c690 comp rx=0 tx=0).stop 2024-04-26T00:23:19.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b475b640 1 -- 172.21.15.37:0/3812209542 shutdown_connections 2024-04-26T00:23:19.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b475b640 1 --2- 172.21.15.37:0/3812209542 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3ac13a770 0x7fb3ac13cb60 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:19.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b475b640 1 --2- 172.21.15.37:0/3812209542 >> v2:172.21.15.45:3300/0 conn(0x7fb3ac137e40 0x7fb3ac13a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:19.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b475b640 1 -- 172.21.15.37:0/3812209542 >> 172.21.15.37:0/3812209542 conn(0x7fb3ac0a1ad0 msgr2=0x7fb3ac0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:23:19.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b475b640 1 -- 172.21.15.37:0/3812209542 shutdown_connections 2024-04-26T00:23:19.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b475b640 1 -- 172.21.15.37:0/3812209542 wait complete. 2024-04-26T00:23:19.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b475b640 10 monclient: build_initial_monmap 2024-04-26T00:23:19.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b475b640 10 monclient: monmap: 2024-04-26T00:23:19.157 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:23:19.157 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:23:19.157 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:23:19.157 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:23:19.157 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:23:19.157 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:23:19.157 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:23:19.157 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:23:19.158 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:23:19.158 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:19.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b475b640 1 Processor -- start 2024-04-26T00:23:19.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b475b640 1 -- start start 2024-04-26T00:23:19.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b475b640 10 monclient: init 2024-04-26T00:23:19.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b475b640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:19.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b475b640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:23:19.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b475b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3ac0a0260 0x7fb3ac1cc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:19.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b475b640 10 monclient(hunting): picked mon.noname-b con 0x7fb3ac0a0260 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:23:19.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b475b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3ac137e40 0x7fb3ac1368e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:19.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b475b640 10 monclient(hunting): picked mon.noname-a con 0x7fb3ac137e40 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:23:19.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b475b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb3ac13a770 0x7fb3ac11d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:19.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b475b640 10 monclient(hunting): picked mon.noname-c con 0x7fb3ac13a770 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:23:19.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b475b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:19.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b475b640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ac09f0f0 con 0x7fb3ac137e40 2024-04-26T00:23:19.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b475b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:19.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b475b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ac0a00a0 con 0x7fb3ac0a0260 2024-04-26T00:23:19.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b475b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:19.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b475b640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb3ac09f4c0 con 0x7fb3ac13a770 2024-04-26T00:23:19.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b475b640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:19.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b475b640 10 monclient(hunting): authenticate will time out at 1180.483242s 2024-04-26T00:23:19.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b1cf6640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3ac137e40 0x7fb3ac1368e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:19.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b1cf6640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3ac137e40 0x7fb3ac1368e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:48386/0 (socket says 172.21.15.37:48386) 2024-04-26T00:23:19.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b1cf6640 1 -- 172.21.15.37:0/505665327 learned_addr learned my addr 172.21.15.37:0/505665327 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:23:19.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b1cf6640 10 monclient(hunting): get_auth_request con 0x7fb3ac137e40 auth_method 0 2024-04-26T00:23:19.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b1cf6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:19.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b1cf6640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:19.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b1cf6640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:19.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b2cf8640 1 --2- 172.21.15.37:0/505665327 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb3ac13a770 0x7fb3ac11d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:19.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b2cf8640 10 monclient(hunting): get_auth_request con 0x7fb3ac13a770 auth_method 0 2024-04-26T00:23:19.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b2cf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:19.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b2cf8640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:19.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b2cf8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:19.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b1cf6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:19.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b1cf6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:19.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b1cf6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:19.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b2cf8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:19.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b2cf8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:19.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b2cf8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:19.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b24f7640 1 --2- 172.21.15.37:0/505665327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3ac0a0260 0x7fb3ac1cc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:19.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b24f7640 10 monclient(hunting): get_auth_request con 0x7fb3ac0a0260 auth_method 0 2024-04-26T00:23:19.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b24f7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:19.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b24f7640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:19.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b24f7640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:19.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b1cf6640 10 monclient(hunting): handle_auth_done global_id 44115 payload 824 2024-04-26T00:23:19.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b1cf6640 1 -- 172.21.15.37:0/505665327 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb3ac13a770 msgr2=0x7fb3ac11d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:19.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b1cf6640 1 --2- 172.21.15.37:0/505665327 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb3ac13a770 0x7fb3ac11d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:19.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b1cf6640 1 -- 172.21.15.37:0/505665327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3ac0a0260 msgr2=0x7fb3ac1cc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:19.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b1cf6640 1 --2- 172.21.15.37:0/505665327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3ac0a0260 0x7fb3ac1cc7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:19.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b1cf6640 10 monclient: _finish_hunting 0 2024-04-26T00:23:19.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b1cf6640 1 monclient: found mon.noname-a 2024-04-26T00:23:19.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b1cf6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:19.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b1cf6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:23:19.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.174+0000 7fb3b1cf6640 1 -- 172.21.15.37:0/505665327 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3ac11ef80 con 0x7fb3ac137e40 2024-04-26T00:23:19.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.178+0000 7fb3b1cf6640 1 -- 172.21.15.37:0/505665327 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3ac137e40 msgr2=0x7fb3ac1368e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:19.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.178+0000 7fb3b1cf6640 1 --2- 172.21.15.37:0/505665327 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3ac137e40 0x7fb3ac1368e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb39c00d800 tx=0x7fb39c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:19.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.182+0000 7fb3a2ffd640 1 -- 172.21.15.37:0/505665327 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb39c00b8d0 con 0x7fb3ac137e40 2024-04-26T00:23:19.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.182+0000 7fb3a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:19.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.182+0000 7fb3a2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:23:19.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.182+0000 7fb3a2ffd640 10 monclient: dump: 2024-04-26T00:23:19.163 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:19.163 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:19.163 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:19.164 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:19.164 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:19.164 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:19.164 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:19.164 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:19.164 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:19.164 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:19.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.182+0000 7fb3a2ffd640 10 monclient: _finish_auth 0 2024-04-26T00:23:19.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.182+0000 7fb3a2ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:23:19.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.182+0000 7fb3a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:19.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.182+0000 7fb3a2ffd640 1 -- 172.21.15.37:0/505665327 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb39c019070 con 0x7fb3ac137e40 2024-04-26T00:23:19.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.182+0000 7fb3b475b640 5 monclient: authenticate success, global_id 44115 2024-04-26T00:23:19.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.182+0000 7fb3b475b640 10 monclient: _renew_subs 2024-04-26T00:23:19.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.182+0000 7fb3b475b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:23:19.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.182+0000 7fb3b475b640 1 -- 172.21.15.37:0/505665327 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3ac1cd6c0 con 0x7fb3ac137e40 2024-04-26T00:23:19.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.182+0000 7fb3a2ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:23:19.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.182+0000 7fb3a2ffd640 1 -- 172.21.15.37:0/505665327 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb39c00bc70 con 0x7fb3ac137e40 2024-04-26T00:23:19.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.182+0000 7fb3a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:19.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.182+0000 7fb3a2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:23:19.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.182+0000 7fb3a2ffd640 10 monclient: dump: 2024-04-26T00:23:19.165 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:19.165 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:19.165 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:19.165 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:19.165 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:19.165 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:19.165 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:19.165 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:19.165 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:19.166 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:19.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.182+0000 7fb3b0cf4640 4 set_mon_vals no callback set 2024-04-26T00:23:19.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.182+0000 7fb3b475b640 10 monclient: _renew_subs 2024-04-26T00:23:19.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.182+0000 7fb3b475b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:23:19.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.182+0000 7fb3b475b640 1 -- 172.21.15.37:0/505665327 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3ac121270 con 0x7fb3ac137e40 2024-04-26T00:23:19.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.182+0000 7fb3a2ffd640 1 -- 172.21.15.37:0/505665327 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb39c01e460 con 0x7fb3ac137e40 2024-04-26T00:23:19.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.182+0000 7fb3b475b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:19.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.182+0000 7fb3b475b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:19.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.182+0000 7fb3b475b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:23:19.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.182+0000 7fb3b475b640 1 -- 172.21.15.37:0/505665327 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3ac066f40 con 0x7fb3ac137e40 2024-04-26T00:23:19.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.182+0000 7fb3a2ffd640 1 --2- 172.21.15.37:0/505665327 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb390035e80 0x7fb390038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:19.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.182+0000 7fb3a2ffd640 1 -- 172.21.15.37:0/505665327 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 3931+0+0 (secure 0 0 0) 0x7fb39c048f70 con 0x7fb3ac137e40 2024-04-26T00:23:19.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.182+0000 7fb3b24f7640 1 --2- 172.21.15.37:0/505665327 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb390035e80 0x7fb390038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:19.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.182+0000 7fb3b24f7640 10 monclient: get_auth_request con 0x7fb390035e80 auth_method 0 2024-04-26T00:23:19.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.186+0000 7fb3b24f7640 1 --2- 172.21.15.37:0/505665327 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb390035e80 0x7fb390038340 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8004e30 tx=0x7fb3a8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:19.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.186+0000 7fb3a2ffd640 1 -- 172.21.15.37:0/505665327 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fb39c00be10 con 0x7fb3ac137e40 2024-04-26T00:23:19.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.186+0000 7fb3a2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:19.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.186+0000 7fb3a2ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:23:19.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.410+0000 7fb3b475b640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-26T00:23:19.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.410+0000 7fb3b475b640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-26T00:23:19.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.410+0000 7fb3b475b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:23:19.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.410+0000 7fb3b475b640 1 -- 172.21.15.37:0/505665327 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fb3ac06b9a0 con 0x7fb3ac137e40 2024-04-26T00:23:19.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.414+0000 7fb3a2ffd640 1 -- 172.21.15.37:0/505665327 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fb39c047cc0 con 0x7fb3ac137e40 2024-04-26T00:23:19.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.414+0000 7fb3a2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-26T00:23:19.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.414+0000 7fb3a2ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:23:19.393 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.414+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:23:19.393 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.414+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=10,config=2,health=12,kv=10,logm=66} crc {auth=1867396482,config=692619927,health=2157093694,kv=1034996044,logm=3545549666}) 2024-04-26T00:23:19.393 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.414+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=11,config=2,health=12,kv=9,logm=66} crc {auth=468025368,config=692619927,health=2157093694,kv=3680767304,logm=1418391198}) 2024-04-26T00:23:19.393 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.414+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:23:19.393 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.414+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=10,config=2,health=12,kv=10,logm=66} crc {auth=1867396482,config=692619927,health=2157093694,kv=1034996044,logm=3545549666}) 2024-04-26T00:23:19.393 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.414+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=11,config=1,health=10,kv=10,logm=68} crc {auth=468025368,config=1228700967,health=1218526840,kv=1034996044,logm=41882698}) 2024-04-26T00:23:19.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.418+0000 7fb3b475b640 1 -- 172.21.15.37:0/505665327 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb390035e80 msgr2=0x7fb390038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:19.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.418+0000 7fb3b475b640 1 --2- 172.21.15.37:0/505665327 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb390035e80 0x7fb390038340 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8004e30 tx=0x7fb3a8009040 comp rx=0 tx=0).stop 2024-04-26T00:23:19.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.418+0000 7fb3b475b640 10 monclient: shutdown 2024-04-26T00:23:19.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.418+0000 7fb3b475b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:23:19.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.418+0000 7fb3b475b640 1 -- 172.21.15.37:0/505665327 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3ac137e40 msgr2=0x7fb3ac1368e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:19.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.418+0000 7fb3b475b640 1 --2- 172.21.15.37:0/505665327 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3ac137e40 0x7fb3ac1368e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb39c00d800 tx=0x7fb39c00dcc0 comp rx=0 tx=0).stop 2024-04-26T00:23:19.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.418+0000 7fb3b475b640 1 -- 172.21.15.37:0/505665327 shutdown_connections 2024-04-26T00:23:19.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.418+0000 7fb3b475b640 1 --2- 172.21.15.37:0/505665327 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb390035e80 0x7fb390038340 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:19.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.418+0000 7fb3b475b640 1 --2- 172.21.15.37:0/505665327 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb3ac13a770 0x7fb3ac11d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:19.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.418+0000 7fb3b475b640 1 --2- 172.21.15.37:0/505665327 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3ac137e40 0x7fb3ac1368e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:19.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.418+0000 7fb3b475b640 1 --2- 172.21.15.37:0/505665327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3ac0a0260 0x7fb3ac1cc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:19.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.418+0000 7fb3b475b640 1 -- 172.21.15.37:0/505665327 >> 172.21.15.37:0/505665327 conn(0x7fb3ac0a1ad0 msgr2=0x7fb3ac138610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:23:19.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.418+0000 7fb3b475b640 1 -- 172.21.15.37:0/505665327 shutdown_connections 2024-04-26T00:23:19.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:19.418+0000 7fb3b475b640 1 -- 172.21.15.37:0/505665327 wait complete. 2024-04-26T00:23:19.413 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-26T00:23:19.497 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.518+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:23:19.498 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.518+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4064446543}) 2024-04-26T00:23:19.498 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.518+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3140019335}) 2024-04-26T00:23:19.498 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.518+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:23:19.498 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.518+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4064446543}) 2024-04-26T00:23:19.498 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.518+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1483753718}) 2024-04-26T00:23:19.500 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.522+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:23:19.500 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.522+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2754494710}) 2024-04-26T00:23:19.500 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.522+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2917344452}) 2024-04-26T00:23:19.501 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.522+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:23:19.501 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.522+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2754494710}) 2024-04-26T00:23:19.501 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.522+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1643361503}) 2024-04-26T00:23:19.723 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.746+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:23:19.723 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.746+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3215516529}) 2024-04-26T00:23:19.723 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.746+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1037275668}) 2024-04-26T00:23:19.724 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.746+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:23:19.724 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.746+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3215516529}) 2024-04-26T00:23:19.724 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.746+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=819775336}) 2024-04-26T00:23:19.733 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.754+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:23:19.734 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.754+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2799474237}) 2024-04-26T00:23:19.734 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.754+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3994982985}) 2024-04-26T00:23:19.734 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.754+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:23:19.734 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.754+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2799474237}) 2024-04-26T00:23:19.734 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.754+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=175509816}) 2024-04-26T00:23:19.741 INFO:tasks.rados.rados.0.smithi045.stdout:626: done (4 left) 2024-04-26T00:23:19.741 INFO:tasks.rados.rados.0.smithi045.stdout:627: snap_create 2024-04-26T00:23:19.741 INFO:tasks.rados.rados.0.smithi045.stdout:623: finishing write tid 2 to smithi04526579-35 2024-04-26T00:23:19.741 INFO:tasks.rados.rados.0.smithi045.stdout:623: finishing write tid 3 to smithi04526579-35 2024-04-26T00:23:19.741 INFO:tasks.rados.rados.0.smithi045.stdout:623: finishing write tid 4 to smithi04526579-35 2024-04-26T00:23:19.741 INFO:tasks.rados.rados.0.smithi045.stdout:623: oid 35 updating version 0 to 236 2024-04-26T00:23:19.741 INFO:tasks.rados.rados.0.smithi045.stdout:623: oid 35 updating version 236 to 237 2024-04-26T00:23:19.741 INFO:tasks.rados.rados.0.smithi045.stdout:623: oid 35 updating version 237 to 238 2024-04-26T00:23:19.741 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 238 (ObjNum 260 snap 68 seq_num 260) dirty exists 2024-04-26T00:23:19.742 INFO:tasks.rados.rados.0.smithi045.stdout:623: left oid 35 (ObjNum 260 snap 68 seq_num 260) 2024-04-26T00:23:19.742 INFO:tasks.rados.rados.0.smithi045.stdout:625: finishing copy_from to smithi04526579-36 2024-04-26T00:23:19.742 INFO:tasks.rados.rados.0.smithi045.stdout:625: got expected ENOENT (src dne) 2024-04-26T00:23:19.742 INFO:tasks.rados.rados.0.smithi045.stdout:625: finishing copy_from racing read to smithi04526579-36 2024-04-26T00:23:19.742 INFO:tasks.rados.rados.0.smithi045.stdout:624: finishing copy_from racing read to smithi04526579-27 2024-04-26T00:23:19.742 INFO:tasks.rados.rados.0.smithi045.stdout:624: finishing copy_from to smithi04526579-27 2024-04-26T00:23:19.742 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 351 (ObjNum 246 snap 63 seq_num 246) dirty exists 2024-04-26T00:23:19.742 INFO:tasks.rados.rados.0.smithi045.stdout:621: expect (ObjNum 202 snap 53 seq_num 202) 2024-04-26T00:23:19.797 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.818+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:23:19.797 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.818+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3922664434}) 2024-04-26T00:23:19.797 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.818+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3316703336}) 2024-04-26T00:23:19.798 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.818+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:23:19.798 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.818+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3922664434}) 2024-04-26T00:23:19.798 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.818+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3345683866}) 2024-04-26T00:23:19.803 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.826+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:23:19.804 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.826+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1430517973}) 2024-04-26T00:23:19.804 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.826+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=94973648}) 2024-04-26T00:23:19.804 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.826+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:23:19.804 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.826+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1430517973}) 2024-04-26T00:23:19.804 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.826+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4286445215}) 2024-04-26T00:23:19.814 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.838+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:23:19.815 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.838+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1760377396}) 2024-04-26T00:23:19.815 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.838+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=953193989}) 2024-04-26T00:23:19.815 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.838+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:23:19.815 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.838+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1760377396}) 2024-04-26T00:23:19.815 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:19.838+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1175221866}) 2024-04-26T00:23:20.188 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:20.210+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:23:20.188 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:20.210+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3211044774}) 2024-04-26T00:23:20.188 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:20.210+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2489172401}) 2024-04-26T00:23:20.188 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:20.210+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:23:20.188 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:20.210+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3211044774}) 2024-04-26T00:23:20.188 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:20.210+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2242997094}) 2024-04-26T00:23:20.190 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:20.214+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:23:20.190 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:20.214+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=64,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=2} crc {logm=779143191,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=720958394,mon_config_key=216630890,monmap=4194790699}) 2024-04-26T00:23:20.190 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:20.214+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=61,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=11,monmap=3,osd_metadata=3} crc {logm=3504320767,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=3414169443,mon_config_key=4061063117,monmap=3876866194,osd_metadata=1931190829}) 2024-04-26T00:23:20.191 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:20.214+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:23:20.191 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:20.214+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=64,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=2} crc {logm=779143191,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=720958394,mon_config_key=216630890,monmap=4194790699}) 2024-04-26T00:23:20.191 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:20.214+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=61,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=11,mon_config_key=11,monmap=3,osd_metadata=2} crc {logm=2776025135,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgrstat=1773766707,mon_config_key=318482329,monmap=3876866194,osd_metadata=2003155415}) 2024-04-26T00:23:20.193 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:20.214+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:23:20.193 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:20.214+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {monmap=1,osd_metadata=6,osd_pg_creating=1,osd_snap=21,osdmap=71} crc {monmap=1228700967,osd_metadata=3306121363,osd_pg_creating=3091265077,osd_snap=289037460,osdmap=3101264114}) 2024-04-26T00:23:20.193 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:20.214+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {monmap=1,osd_metadata=6,osd_pg_creating=1,osd_snap=22,osdmap=70} crc {monmap=1228700967,osd_metadata=3306121363,osd_pg_creating=3091265077,osd_snap=2637996987,osdmap=571460995}) 2024-04-26T00:23:20.193 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:20.214+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:23:20.193 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:20.214+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {monmap=1,osd_metadata=6,osd_pg_creating=1,osd_snap=21,osdmap=71} crc {monmap=1228700967,osd_metadata=3306121363,osd_pg_creating=3091265077,osd_snap=289037460,osdmap=3101264114}) 2024-04-26T00:23:20.193 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:20.214+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {monmap=1,osd_metadata=6,osd_pg_creating=1,osd_snap=23,osdmap=69} crc {monmap=1228700967,osd_metadata=3306121363,osd_pg_creating=3091265078,osd_snap=2018462217,osdmap=1620654574}) 2024-04-26T00:23:20.196 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:20.218+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:23:20.196 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:20.218+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3565026294}) 2024-04-26T00:23:20.197 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:20.218+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3117542799}) 2024-04-26T00:23:20.197 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:20.218+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:23:20.197 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:20.218+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3565026294}) 2024-04-26T00:23:20.197 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:20.218+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1657056015}) 2024-04-26T00:23:20.382 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:20.402+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:23:20.382 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:20.402+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=51} crc {osdmap=1526491027}) 2024-04-26T00:23:20.382 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:20.402+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=50} crc {osdmap=2126435966}) 2024-04-26T00:23:20.382 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:20.402+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:23:20.382 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:20.402+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=51} crc {osdmap=1526491027}) 2024-04-26T00:23:20.382 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:23:20.402+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=49} crc {osdmap=494195343}) 2024-04-26T00:23:20.413 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:23:20.414 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:23:20.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbeb0d02640 10 monclient: get_monmap_and_config 2024-04-26T00:23:20.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbeb0d02640 10 monclient: build_initial_monmap 2024-04-26T00:23:20.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbeb0d02640 10 monclient: monmap: 2024-04-26T00:23:20.548 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:23:20.548 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:20.548 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:23:20.573052+0000 2024-04-26T00:23:20.548 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:23:20.573052+0000 2024-04-26T00:23:20.548 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:23:20.548 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:20.548 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:23:20.548 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:23:20.548 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:23:20.548 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:23:20.548 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:23:20.548 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:20.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbeb0d02640 1 Processor -- start 2024-04-26T00:23:20.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbeb0d02640 1 -- start start 2024-04-26T00:23:20.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbeb0d02640 10 monclient: init 2024-04-26T00:23:20.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbeb0d02640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:20.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbeb0d02640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-04-26T00:23:20.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbeb0d02640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbeac136da0 0x7fbeac09bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:20.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbeb0d02640 10 monclient(hunting): picked mon.noname-c con 0x7fbeac136da0 addr v2:172.21.15.45:3300/0 2024-04-26T00:23:20.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbeb0d02640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fbeac09c410 0x7fbeac137180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:20.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbeb0d02640 10 monclient(hunting): picked mon.noname-a con 0x7fbeac09c410 addr v2:172.21.15.37:3300/0 2024-04-26T00:23:20.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbeb0d02640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbeac1376c0 0x7fbeac133b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:20.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbeb0d02640 10 monclient(hunting): picked mon.noname-b con 0x7fbeac1376c0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:23:20.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbeb0d02640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:20.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbeb0d02640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fbeac09b210 con 0x7fbeac09c410 2024-04-26T00:23:20.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbeb0d02640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:20.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbeb0d02640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fbeac09b4f0 con 0x7fbeac136da0 2024-04-26T00:23:20.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbeb0d02640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:20.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbeb0d02640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fbeac09b7d0 con 0x7fbeac1376c0 2024-04-26T00:23:20.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbeb0d02640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:20.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbeb0d02640 10 monclient(hunting): authenticate will time out at 1181.878813s 2024-04-26T00:23:20.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbea9d9b640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fbeac09c410 0x7fbeac137180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:20.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbea9d9b640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fbeac09c410 0x7fbeac137180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58058/0 (socket says 172.21.15.37:58058) 2024-04-26T00:23:20.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbea9d9b640 1 -- 172.21.15.37:0/2462042634 learned_addr learned my addr 172.21.15.37:0/2462042634 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:23:20.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbea9d9b640 10 monclient(hunting): get_auth_request con 0x7fbeac09c410 auth_method 0 2024-04-26T00:23:20.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbea9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:20.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbea9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:20.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbea9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:20.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbeaad9d640 1 --2- 172.21.15.37:0/2462042634 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbeac1376c0 0x7fbeac133b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:20.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbeaad9d640 10 monclient(hunting): get_auth_request con 0x7fbeac1376c0 auth_method 0 2024-04-26T00:23:20.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbeaad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:20.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbeaad9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:20.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbeaad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:20.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbea9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:20.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbea9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:20.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbea9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:20.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbeaa59c640 1 --2- 172.21.15.37:0/2462042634 >> v2:172.21.15.45:3300/0 conn(0x7fbeac136da0 0x7fbeac09bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:20.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbeaa59c640 10 monclient(hunting): get_auth_request con 0x7fbeac136da0 auth_method 0 2024-04-26T00:23:20.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbeaa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:20.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbeaa59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:20.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbeaa59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:20.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbeaad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:20.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbeaad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:20.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbeaad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:20.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbea9d9b640 10 monclient(hunting): handle_auth_done global_id 44118 payload 274 2024-04-26T00:23:20.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbea9d9b640 1 -- 172.21.15.37:0/2462042634 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbeac1376c0 msgr2=0x7fbeac133b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:23:20.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbea9d9b640 1 --2- 172.21.15.37:0/2462042634 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbeac1376c0 0x7fbeac133b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:20.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbea9d9b640 1 -- 172.21.15.37:0/2462042634 >> v2:172.21.15.45:3300/0 conn(0x7fbeac136da0 msgr2=0x7fbeac09bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:23:20.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbea9d9b640 1 --2- 172.21.15.37:0/2462042634 >> v2:172.21.15.45:3300/0 conn(0x7fbeac136da0 0x7fbeac09bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:20.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbea9d9b640 10 monclient: _finish_hunting 0 2024-04-26T00:23:20.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbea9d9b640 1 monclient: found mon.noname-a 2024-04-26T00:23:20.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbea9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:20.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbea9d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:23:20.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.570+0000 7fbea9d9b640 1 -- 172.21.15.37:0/2462042634 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeac137b00 con 0x7fbeac09c410 2024-04-26T00:23:20.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.574+0000 7fbea9d9b640 1 -- 172.21.15.37:0/2462042634 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbeac09c410 msgr2=0x7fbeac137180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:20.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.574+0000 7fbea9d9b640 1 --2- 172.21.15.37:0/2462042634 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbeac09c410 0x7fbeac137180 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fbeac0a58a0 tx=0x7fbea001c120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b1a41fe255d610e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:20.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbea959a640 1 -- 172.21.15.37:0/2462042634 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea00b0070 con 0x7fbeac09c410 2024-04-26T00:23:20.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbea959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:20.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbea959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:23:20.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbea959a640 10 monclient: dump: 2024-04-26T00:23:20.559 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:20.559 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:20.559 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:20.559 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:20.559 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:20.559 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:20.559 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:20.559 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:20.559 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:20.559 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:20.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbea959a640 10 monclient: _finish_auth 0 2024-04-26T00:23:20.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbea959a640 10 monclient: _check_auth_tickets 2024-04-26T00:23:20.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbea959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:20.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeb0d02640 5 monclient: authenticate success, global_id 44118 2024-04-26T00:23:20.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbea959a640 1 -- 172.21.15.37:0/2462042634 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbea00ab600 con 0x7fbeac09c410 2024-04-26T00:23:20.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeb0d02640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:23:20.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbea959a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:23:20.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeb0d02640 10 monclient: get_monmap_and_config success 2024-04-26T00:23:20.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeb0d02640 4 set_mon_vals no callback set 2024-04-26T00:23:20.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbea959a640 1 -- 172.21.15.37:0/2462042634 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea00ab9d0 con 0x7fbeac09c410 2024-04-26T00:23:20.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbea959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:20.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbea959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:23:20.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbea959a640 10 monclient: dump: 2024-04-26T00:23:20.561 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:20.561 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:20.561 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:20.561 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:20.561 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:20.561 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:20.561 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:20.561 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:20.561 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:20.561 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:20.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeb0d02640 10 monclient: shutdown 2024-04-26T00:23:20.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeb0d02640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:23:20.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeb0d02640 1 -- 172.21.15.37:0/2462042634 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbeac09c410 msgr2=0x7fbeac137180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:20.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeb0d02640 1 --2- 172.21.15.37:0/2462042634 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbeac09c410 0x7fbeac137180 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fbeac0a58a0 tx=0x7fbea001c120 comp rx=0 tx=0).stop 2024-04-26T00:23:20.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeb0d02640 1 -- 172.21.15.37:0/2462042634 shutdown_connections 2024-04-26T00:23:20.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeb0d02640 1 --2- 172.21.15.37:0/2462042634 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbeac1376c0 0x7fbeac133b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:20.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeb0d02640 1 --2- 172.21.15.37:0/2462042634 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbeac09c410 0x7fbeac137180 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:20.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeb0d02640 1 --2- 172.21.15.37:0/2462042634 >> v2:172.21.15.45:3300/0 conn(0x7fbeac136da0 0x7fbeac09bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:20.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeb0d02640 1 -- 172.21.15.37:0/2462042634 >> 172.21.15.37:0/2462042634 conn(0x7fbeac0a5f00 msgr2=0x7fbeac0a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:23:20.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeb0d02640 1 -- 172.21.15.37:0/2462042634 shutdown_connections 2024-04-26T00:23:20.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeb0d02640 1 -- 172.21.15.37:0/2462042634 wait complete. 2024-04-26T00:23:20.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeb0d02640 10 monclient: build_initial_monmap 2024-04-26T00:23:20.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeb0d02640 10 monclient: monmap: 2024-04-26T00:23:20.562 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:23:20.562 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:23:20.562 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:23:20.562 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:23:20.562 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:23:20.562 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:23:20.563 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:23:20.563 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:23:20.563 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:23:20.563 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:20.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeb0d02640 1 Processor -- start 2024-04-26T00:23:20.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeb0d02640 1 -- start start 2024-04-26T00:23:20.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeb0d02640 10 monclient: init 2024-04-26T00:23:20.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeb0d02640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:20.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeb0d02640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:23:20.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeb0d02640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbeac09c410 0x7fbeac13d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:20.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeb0d02640 10 monclient(hunting): picked mon.noname-c con 0x7fbeac09c410 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:23:20.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeb0d02640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbeac136da0 0x7fbeac137db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:20.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeb0d02640 10 monclient(hunting): picked mon.noname-b con 0x7fbeac136da0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:23:20.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeb0d02640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbeac1376c0 0x7fbeac1382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:20.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeb0d02640 10 monclient(hunting): picked mon.noname-a con 0x7fbeac1376c0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:23:20.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeb0d02640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:20.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeb0d02640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fbeac140e90 con 0x7fbeac1376c0 2024-04-26T00:23:20.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeb0d02640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:20.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeb0d02640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbeac140d10 con 0x7fbeac136da0 2024-04-26T00:23:20.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeb0d02640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:20.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeb0d02640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fbeac141010 con 0x7fbeac09c410 2024-04-26T00:23:20.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeb0d02640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:20.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeb0d02640 10 monclient(hunting): authenticate will time out at 1181.886267s 2024-04-26T00:23:20.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeaad9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbeac1376c0 0x7fbeac1382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:20.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeaa59c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbeac09c410 0x7fbeac13d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:20.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeaad9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbeac1376c0 0x7fbeac1382f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58068/0 (socket says 172.21.15.37:58068) 2024-04-26T00:23:20.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeaad9d640 1 -- 172.21.15.37:0/451546293 learned_addr learned my addr 172.21.15.37:0/451546293 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:23:20.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeaad9d640 10 monclient(hunting): get_auth_request con 0x7fbeac1376c0 auth_method 0 2024-04-26T00:23:20.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeaad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:20.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeaad9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:20.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeaad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:20.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeaa59c640 10 monclient(hunting): get_auth_request con 0x7fbeac09c410 auth_method 0 2024-04-26T00:23:20.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeaa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:20.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeaa59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:20.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeaa59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:20.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeaad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:20.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeaad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:20.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeaad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:20.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeaa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:20.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeaa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:20.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeaa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:20.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbea9d9b640 1 --2- 172.21.15.37:0/451546293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbeac136da0 0x7fbeac137db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:20.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeaad9d640 10 monclient(hunting): handle_auth_done global_id 44121 payload 824 2024-04-26T00:23:20.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeaad9d640 1 -- 172.21.15.37:0/451546293 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbeac09c410 msgr2=0x7fbeac13d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:20.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeaad9d640 1 --2- 172.21.15.37:0/451546293 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbeac09c410 0x7fbeac13d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:20.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeaad9d640 1 -- 172.21.15.37:0/451546293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbeac136da0 msgr2=0x7fbeac137db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:20.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeaad9d640 1 --2- 172.21.15.37:0/451546293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbeac136da0 0x7fbeac137db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:20.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeaad9d640 10 monclient: _finish_hunting 0 2024-04-26T00:23:20.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeaad9d640 1 monclient: found mon.noname-a 2024-04-26T00:23:20.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeaad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:20.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeaad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:23:20.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeaad9d640 1 -- 172.21.15.37:0/451546293 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeac09fe50 con 0x7fbeac1376c0 2024-04-26T00:23:20.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbea9d9b640 10 monclient: get_auth_request con 0x7fbeac136da0 auth_method 0 2024-04-26T00:23:20.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbeaa59c640 1 --2- 172.21.15.37:0/451546293 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbeac09c410 0x7fbeac13d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:23:20.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.578+0000 7fbea9d9b640 1 --2- 172.21.15.37:0/451546293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbeac136da0 0x7fbeac137db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:23:20.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.582+0000 7fbeaad9d640 1 -- 172.21.15.37:0/451546293 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbeac1376c0 msgr2=0x7fbeac1382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:20.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.582+0000 7fbeaad9d640 1 --2- 172.21.15.37:0/451546293 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbeac1376c0 0x7fbeac1382f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbea400efc0 tx=0x7fbea400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:20.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.582+0000 7fbe9affd640 1 -- 172.21.15.37:0/451546293 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea4018020 con 0x7fbeac1376c0 2024-04-26T00:23:20.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.582+0000 7fbe9affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:20.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.582+0000 7fbe9affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:23:20.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.582+0000 7fbe9affd640 10 monclient: dump: 2024-04-26T00:23:20.569 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:20.569 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:20.569 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:20.569 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:20.569 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:20.569 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:20.569 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:20.569 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:20.569 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:20.569 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:20.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.582+0000 7fbe9affd640 10 monclient: _finish_auth 0 2024-04-26T00:23:20.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.582+0000 7fbe9affd640 10 monclient: _check_auth_tickets 2024-04-26T00:23:20.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.582+0000 7fbe9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:20.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.582+0000 7fbe9affd640 1 -- 172.21.15.37:0/451546293 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbea4014680 con 0x7fbeac1376c0 2024-04-26T00:23:20.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.582+0000 7fbeb0d02640 5 monclient: authenticate success, global_id 44121 2024-04-26T00:23:20.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.582+0000 7fbeb0d02640 10 monclient: _renew_subs 2024-04-26T00:23:20.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.582+0000 7fbeb0d02640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:23:20.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.582+0000 7fbeb0d02640 1 -- 172.21.15.37:0/451546293 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbeac0a8b40 con 0x7fbeac1376c0 2024-04-26T00:23:20.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.582+0000 7fbe9affd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:23:20.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.582+0000 7fbe9affd640 1 -- 172.21.15.37:0/451546293 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea4014a70 con 0x7fbeac1376c0 2024-04-26T00:23:20.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.582+0000 7fbe9affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:20.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.582+0000 7fbe9affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:23:20.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.582+0000 7fbe9affd640 10 monclient: dump: 2024-04-26T00:23:20.570 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:20.570 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:20.570 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:20.570 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:20.570 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:20.571 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:20.571 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:20.571 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:20.571 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:20.571 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:20.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.582+0000 7fbeb0d02640 10 monclient: _renew_subs 2024-04-26T00:23:20.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.582+0000 7fbeb0d02640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:23:20.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.582+0000 7fbeb0d02640 1 -- 172.21.15.37:0/451546293 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbeac13aca0 con 0x7fbeac1376c0 2024-04-26T00:23:20.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.582+0000 7fbe9bfff640 4 set_mon_vals no callback set 2024-04-26T00:23:20.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.582+0000 7fbe9affd640 1 -- 172.21.15.37:0/451546293 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fbea401b020 con 0x7fbeac1376c0 2024-04-26T00:23:20.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.582+0000 7fbeb0d02640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:20.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.582+0000 7fbeb0d02640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:20.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.582+0000 7fbeb0d02640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:23:20.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.582+0000 7fbeb0d02640 1 -- 172.21.15.37:0/451546293 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbeac066f40 con 0x7fbeac1376c0 2024-04-26T00:23:20.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.582+0000 7fbe9affd640 1 --2- 172.21.15.37:0/451546293 >> v2:172.21.15.45:6812/2379918623 conn(0x7fbe8c035e80 0x7fbe8c038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:20.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.582+0000 7fbe9affd640 1 -- 172.21.15.37:0/451546293 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(145..145 src has 1..145) ==== 4003+0+0 (secure 0 0 0) 0x7fbea4049860 con 0x7fbeac1376c0 2024-04-26T00:23:20.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.582+0000 7fbeaa59c640 1 --2- 172.21.15.37:0/451546293 >> v2:172.21.15.45:6812/2379918623 conn(0x7fbe8c035e80 0x7fbe8c038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:20.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.582+0000 7fbeaa59c640 10 monclient: get_auth_request con 0x7fbe8c035e80 auth_method 0 2024-04-26T00:23:20.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.586+0000 7fbeaa59c640 1 --2- 172.21.15.37:0/451546293 >> v2:172.21.15.45:6812/2379918623 conn(0x7fbe8c035e80 0x7fbe8c038340 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c007e50 tx=0x7fbe9c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:20.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.586+0000 7fbe9affd640 1 -- 172.21.15.37:0/451546293 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fbea4022030 con 0x7fbeac1376c0 2024-04-26T00:23:20.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.586+0000 7fbe9affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:20.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.586+0000 7fbe9affd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:23:20.735 INFO:tasks.rados.rados.0.smithi045.stdout:621: done (4 left) 2024-04-26T00:23:20.735 INFO:tasks.rados.rados.0.smithi045.stdout:623: done (3 left) 2024-04-26T00:23:20.735 INFO:tasks.rados.rados.0.smithi045.stdout:624: done (2 left) 2024-04-26T00:23:20.735 INFO:tasks.rados.rados.0.smithi045.stdout:625: done (1 left) 2024-04-26T00:23:20.735 INFO:tasks.rados.rados.0.smithi045.stdout:627: done (0 left) 2024-04-26T00:23:20.736 INFO:tasks.rados.rados.0.smithi045.stdout:628: snap_create 2024-04-26T00:23:20.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.810+0000 7fbeb0d02640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:23:20.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.810+0000 7fbeb0d02640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:23:20.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.810+0000 7fbeb0d02640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:23:20.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.810+0000 7fbeb0d02640 1 -- 172.21.15.37:0/451546293 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fbeac134cc0 con 0x7fbeac1376c0 2024-04-26T00:23:20.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.810+0000 7fbe9affd640 1 -- 172.21.15.37:0/451546293 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fbea4010040 con 0x7fbeac1376c0 2024-04-26T00:23:20.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.810+0000 7fbe9affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:23:20.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.810+0000 7fbe9affd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:23:20.789 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:23:20.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.814+0000 7fbeb0d02640 1 -- 172.21.15.37:0/451546293 >> v2:172.21.15.45:6812/2379918623 conn(0x7fbe8c035e80 msgr2=0x7fbe8c038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:20.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.814+0000 7fbeb0d02640 1 --2- 172.21.15.37:0/451546293 >> v2:172.21.15.45:6812/2379918623 conn(0x7fbe8c035e80 0x7fbe8c038340 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c007e50 tx=0x7fbe9c008040 comp rx=0 tx=0).stop 2024-04-26T00:23:20.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.814+0000 7fbeb0d02640 10 monclient: shutdown 2024-04-26T00:23:20.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.814+0000 7fbeb0d02640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:23:20.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.814+0000 7fbeb0d02640 1 -- 172.21.15.37:0/451546293 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbeac1376c0 msgr2=0x7fbeac1382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:20.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.814+0000 7fbeb0d02640 1 --2- 172.21.15.37:0/451546293 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbeac1376c0 0x7fbeac1382f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbea400efc0 tx=0x7fbea400c490 comp rx=0 tx=0).stop 2024-04-26T00:23:20.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.814+0000 7fbeb0d02640 1 -- 172.21.15.37:0/451546293 shutdown_connections 2024-04-26T00:23:20.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.814+0000 7fbeb0d02640 1 --2- 172.21.15.37:0/451546293 >> v2:172.21.15.45:6812/2379918623 conn(0x7fbe8c035e80 0x7fbe8c038340 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:20.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.814+0000 7fbeb0d02640 1 --2- 172.21.15.37:0/451546293 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbeac1376c0 0x7fbeac1382f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:20.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.814+0000 7fbeb0d02640 1 --2- 172.21.15.37:0/451546293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbeac136da0 0x7fbeac137db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:20.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.814+0000 7fbeb0d02640 1 --2- 172.21.15.37:0/451546293 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbeac09c410 0x7fbeac13d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:20.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.814+0000 7fbeb0d02640 1 -- 172.21.15.37:0/451546293 >> 172.21.15.37:0/451546293 conn(0x7fbeac0a5f00 msgr2=0x7fbeac1354a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:23:20.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.814+0000 7fbeb0d02640 1 -- 172.21.15.37:0/451546293 shutdown_connections 2024-04-26T00:23:20.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.814+0000 7fbeb0d02640 1 -- 172.21.15.37:0/451546293 wait complete. 2024-04-26T00:23:20.808 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:23:20.808 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-26T00:23:20.809 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:23:20.809 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:23:20.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe91c4f2640 10 monclient: get_monmap_and_config 2024-04-26T00:23:20.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe91c4f2640 10 monclient: build_initial_monmap 2024-04-26T00:23:20.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe91c4f2640 10 monclient: monmap: 2024-04-26T00:23:20.939 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:23:20.939 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:20.939 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:23:20.964267+0000 2024-04-26T00:23:20.939 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:23:20.964267+0000 2024-04-26T00:23:20.939 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:23:20.939 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:20.939 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:23:20.939 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:23:20.939 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:23:20.940 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:23:20.940 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:23:20.940 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:20.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe91c4f2640 1 Processor -- start 2024-04-26T00:23:20.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe91c4f2640 1 -- start start 2024-04-26T00:23:20.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe91c4f2640 10 monclient: init 2024-04-26T00:23:20.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe91c4f2640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:20.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe91c4f2640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-04-26T00:23:20.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe91c4f2640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe914130eb0 0x7fe91409ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:20.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe91c4f2640 10 monclient(hunting): picked mon.noname-b con 0x7fe914130eb0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:23:20.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe91c4f2640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fe914131290 0x7fe914133b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:20.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe91c4f2640 10 monclient(hunting): picked mon.noname-a con 0x7fe914131290 addr v2:172.21.15.37:3300/0 2024-04-26T00:23:20.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe91c4f2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe9141340d0 addr v1:172.21.15.45:6789/0 2024-04-26T00:23:20.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe91c4f2640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe91409c190 con 0x7fe9141340d0 2024-04-26T00:23:20.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe91c4f2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:20.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe91c4f2640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fe91409df60 con 0x7fe914131290 2024-04-26T00:23:20.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe91c4f2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:20.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe91c4f2640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe9140a11c0 con 0x7fe914130eb0 2024-04-26T00:23:20.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe91c4f2640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:20.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe91c4f2640 10 monclient(hunting): authenticate will time out at 1182.269936s 2024-04-26T00:23:20.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe919a8d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fe914131290 0x7fe914133b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:20.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe91a28e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe914130eb0 0x7fe91409ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:20.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe919a8d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fe914131290 0x7fe914133b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58072/0 (socket says 172.21.15.37:58072) 2024-04-26T00:23:20.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe919a8d640 1 -- 172.21.15.37:0/2793485755 learned_addr learned my addr 172.21.15.37:0/2793485755 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:23:20.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe919a8d640 10 monclient(hunting): get_auth_request con 0x7fe914131290 auth_method 0 2024-04-26T00:23:20.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe919a8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:20.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe919a8d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:20.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe919a8d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:20.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe91a28e640 10 monclient(hunting): get_auth_request con 0x7fe914130eb0 auth_method 0 2024-04-26T00:23:20.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe91a28e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:20.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe91a28e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:20.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe91a28e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:20.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe919a8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:20.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe919a8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:20.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe919a8d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:20.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe91a28e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:20.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe91a28e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:20.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe91a28e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:20.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe91a28e640 10 monclient(hunting): handle_auth_done global_id 34226 payload 274 2024-04-26T00:23:20.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe91a28e640 1 -- 172.21.15.37:0/2793485755 >> v2:172.21.15.37:3300/0 conn(0x7fe914131290 msgr2=0x7fe914133b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:23:20.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe91a28e640 1 --2- 172.21.15.37:0/2793485755 >> v2:172.21.15.37:3300/0 conn(0x7fe914131290 0x7fe914133b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:20.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe91a28e640 1 -- 172.21.15.37:0/2793485755 >> v1:172.21.15.45:6789/0 conn(0x7fe9141340d0 legacy=0x7fe91413aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:23:20.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe91a28e640 10 monclient: _finish_hunting 0 2024-04-26T00:23:20.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe91a28e640 1 monclient: found mon.noname-b 2024-04-26T00:23:20.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe91a28e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:20.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe91a28e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:23:20.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe91a28e640 1 -- 172.21.15.37:0/2793485755 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe91409dc80 con 0x7fe914130eb0 2024-04-26T00:23:20.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.962+0000 7fe919a8d640 1 --2- 172.21.15.37:0/2793485755 >> v2:172.21.15.37:3300/0 conn(0x7fe914131290 0x7fe914133b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:23:20.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.966+0000 7fe91a28e640 1 -- 172.21.15.37:0/2793485755 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe914130eb0 msgr2=0x7fe91409ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:20.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.966+0000 7fe91a28e640 1 --2- 172.21.15.37:0/2793485755 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe914130eb0 0x7fe91409ef20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe90c009d90 tx=0x7fe90c01c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1397ff90f1b768aa server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:20.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.966+0000 7fe91928c640 1 -- 172.21.15.37:0/2793485755 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe90c0b2070 con 0x7fe914130eb0 2024-04-26T00:23:20.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.966+0000 7fe91928c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:20.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.966+0000 7fe91928c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:23:20.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.966+0000 7fe91928c640 10 monclient: dump: 2024-04-26T00:23:20.950 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:20.950 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:20.950 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:20.950 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:20.950 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:20.950 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:20.950 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:20.950 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:20.951 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:20.951 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:20.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.966+0000 7fe91928c640 10 monclient: _finish_auth 0 2024-04-26T00:23:20.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.966+0000 7fe91928c640 10 monclient: _check_auth_tickets 2024-04-26T00:23:20.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.966+0000 7fe91928c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:20.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.966+0000 7fe91928c640 1 -- 172.21.15.37:0/2793485755 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe90c0a5780 con 0x7fe914130eb0 2024-04-26T00:23:20.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.966+0000 7fe91c4f2640 5 monclient: authenticate success, global_id 34226 2024-04-26T00:23:20.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.966+0000 7fe91c4f2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:23:20.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.966+0000 7fe91928c640 10 monclient: handle_config config(0 keys) 2024-04-26T00:23:20.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.966+0000 7fe91c4f2640 10 monclient: get_monmap_and_config success 2024-04-26T00:23:20.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.966+0000 7fe91928c640 1 -- 172.21.15.37:0/2793485755 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe90c0a5b50 con 0x7fe914130eb0 2024-04-26T00:23:20.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.966+0000 7fe91928c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:20.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.966+0000 7fe91c4f2640 4 set_mon_vals no callback set 2024-04-26T00:23:20.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.966+0000 7fe91928c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:23:20.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.966+0000 7fe91928c640 10 monclient: dump: 2024-04-26T00:23:20.952 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:20.952 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:20.952 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:20.952 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:20.952 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:20.952 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:20.952 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:20.952 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:20.952 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:20.952 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:20.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.966+0000 7fe91c4f2640 10 monclient: shutdown 2024-04-26T00:23:20.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.966+0000 7fe91c4f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:23:20.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.966+0000 7fe91c4f2640 1 -- 172.21.15.37:0/2793485755 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe914130eb0 msgr2=0x7fe91409ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:20.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.966+0000 7fe91c4f2640 1 --2- 172.21.15.37:0/2793485755 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe914130eb0 0x7fe91409ef20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe90c009d90 tx=0x7fe90c01c7f0 comp rx=0 tx=0).stop 2024-04-26T00:23:20.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.966+0000 7fe91c4f2640 1 -- 172.21.15.37:0/2793485755 shutdown_connections 2024-04-26T00:23:20.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.966+0000 7fe91c4f2640 1 --2- 172.21.15.37:0/2793485755 >> v2:172.21.15.37:3300/0 conn(0x7fe914131290 0x7fe914133b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:20.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.966+0000 7fe91c4f2640 1 --2- 172.21.15.37:0/2793485755 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe914130eb0 0x7fe91409ef20 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:20.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.966+0000 7fe91c4f2640 1 -- 172.21.15.37:0/2793485755 >> 172.21.15.37:0/2793485755 conn(0x7fe9140a1d50 msgr2=0x7fe9140a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:23:20.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.966+0000 7fe91c4f2640 1 -- 172.21.15.37:0/2793485755 shutdown_connections 2024-04-26T00:23:20.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.966+0000 7fe91c4f2640 1 -- 172.21.15.37:0/2793485755 wait complete. 2024-04-26T00:23:20.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe91c4f2640 10 monclient: build_initial_monmap 2024-04-26T00:23:20.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe91c4f2640 10 monclient: monmap: 2024-04-26T00:23:20.953 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:23:20.953 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:23:20.953 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:23:20.953 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:23:20.954 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:23:20.954 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:23:20.954 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:23:20.954 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:23:20.954 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:23:20.954 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:20.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe91c4f2640 1 Processor -- start 2024-04-26T00:23:20.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe91c4f2640 1 -- start start 2024-04-26T00:23:20.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe91c4f2640 10 monclient: init 2024-04-26T00:23:20.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe91c4f2640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:20.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe91c4f2640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:23:20.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe91c4f2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe914130eb0 0x7fe91412ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:20.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe91c4f2640 10 monclient(hunting): picked mon.noname-b con 0x7fe914130eb0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:23:20.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe91c4f2640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe914131290 0x7fe91412f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:20.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe91c4f2640 10 monclient(hunting): picked mon.noname-c con 0x7fe914131290 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:23:20.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe91c4f2640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe9141340d0 0x7fe914137a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:20.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe91c4f2640 10 monclient(hunting): picked mon.noname-a con 0x7fe9141340d0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:23:20.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe91c4f2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:20.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe91c4f2640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe9141317f0 con 0x7fe9141340d0 2024-04-26T00:23:20.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe91c4f2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:20.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe91c4f2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe914131670 con 0x7fe914130eb0 2024-04-26T00:23:20.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe91c4f2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:20.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe91c4f2640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe91413ee00 con 0x7fe914131290 2024-04-26T00:23:20.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe91c4f2640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:20.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe91c4f2640 10 monclient(hunting): authenticate will time out at 1182.277284s 2024-04-26T00:23:20.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe919a8d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe914131290 0x7fe91412f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:20.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe91aa8f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe9141340d0 0x7fe914137a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:20.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe919a8d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe914131290 0x7fe91412f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:46744/0 (socket says 172.21.15.37:46744) 2024-04-26T00:23:20.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe919a8d640 1 -- 172.21.15.37:0/3526924309 learned_addr learned my addr 172.21.15.37:0/3526924309 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:23:20.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe919a8d640 10 monclient(hunting): get_auth_request con 0x7fe914131290 auth_method 0 2024-04-26T00:23:20.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe919a8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:20.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe919a8d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:20.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe919a8d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:20.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe91aa8f640 10 monclient(hunting): get_auth_request con 0x7fe9141340d0 auth_method 0 2024-04-26T00:23:20.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe91aa8f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:20.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe91aa8f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:20.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe91aa8f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:20.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe919a8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:20.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe919a8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:20.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe919a8d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:20.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe91aa8f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:20.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe91aa8f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:20.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe91a28e640 1 --2- 172.21.15.37:0/3526924309 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe914130eb0 0x7fe91412ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:20.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe91aa8f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:20.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe91a28e640 10 monclient(hunting): get_auth_request con 0x7fe914130eb0 auth_method 0 2024-04-26T00:23:20.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe91a28e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:20.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe91a28e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:20.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe91a28e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:20.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe919a8d640 10 monclient(hunting): handle_auth_done global_id 34229 payload 824 2024-04-26T00:23:20.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe919a8d640 1 -- 172.21.15.37:0/3526924309 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe914130eb0 msgr2=0x7fe91412ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:20.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe919a8d640 1 --2- 172.21.15.37:0/3526924309 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe914130eb0 0x7fe91412ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:20.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe919a8d640 1 -- 172.21.15.37:0/3526924309 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe9141340d0 msgr2=0x7fe914137a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:20.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe919a8d640 1 --2- 172.21.15.37:0/3526924309 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe9141340d0 0x7fe914137a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:20.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe919a8d640 10 monclient: _finish_hunting 0 2024-04-26T00:23:20.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe919a8d640 1 monclient: found mon.noname-c 2024-04-26T00:23:20.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe919a8d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:20.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe919a8d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:23:20.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe919a8d640 1 -- 172.21.15.37:0/3526924309 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe91413a940 con 0x7fe914131290 2024-04-26T00:23:20.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe91a28e640 1 --2- 172.21.15.37:0/3526924309 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe914130eb0 0x7fe91412ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:23:20.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.970+0000 7fe91aa8f640 1 --2- 172.21.15.37:0/3526924309 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe9141340d0 0x7fe914137a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:23:20.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe919a8d640 1 -- 172.21.15.37:0/3526924309 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe914131290 msgr2=0x7fe91412f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:20.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe919a8d640 1 --2- 172.21.15.37:0/3526924309 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe914131290 0x7fe91412f460 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe90800ea00 tx=0x7fe90800eec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:20.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe912ffd640 1 -- 172.21.15.37:0/3526924309 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe908017070 con 0x7fe914131290 2024-04-26T00:23:20.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe912ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:20.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe912ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:23:20.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe912ffd640 10 monclient: dump: 2024-04-26T00:23:20.963 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:20.963 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:20.963 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:20.963 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:20.963 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:20.963 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:20.963 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:20.963 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:20.963 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:20.963 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:20.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe912ffd640 10 monclient: _finish_auth 0 2024-04-26T00:23:20.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe912ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:23:20.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe912ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:20.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe91c4f2640 5 monclient: authenticate success, global_id 34229 2024-04-26T00:23:20.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe91c4f2640 10 monclient: _renew_subs 2024-04-26T00:23:20.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe91c4f2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:20.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe91c4f2640 1 -- 172.21.15.37:0/3526924309 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe91412fc20 con 0x7fe914131290 2024-04-26T00:23:20.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe912ffd640 1 -- 172.21.15.37:0/3526924309 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe90800cc70 con 0x7fe914131290 2024-04-26T00:23:20.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe912ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:23:20.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe912ffd640 1 -- 172.21.15.37:0/3526924309 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe908013470 con 0x7fe914131290 2024-04-26T00:23:20.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe912ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:20.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe912ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:23:20.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe912ffd640 10 monclient: dump: 2024-04-26T00:23:20.964 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:20.964 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:20.964 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:20.964 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:20.964 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:20.965 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:20.965 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:20.965 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:20.965 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:20.965 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:20.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe91c4f2640 10 monclient: _renew_subs 2024-04-26T00:23:20.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe91c4f2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:20.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe91c4f2640 1 -- 172.21.15.37:0/3526924309 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9140a0970 con 0x7fe914131290 2024-04-26T00:23:20.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe918a8b640 4 set_mon_vals no callback set 2024-04-26T00:23:20.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe91c4f2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:20.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe91c4f2640 1 --2- 172.21.15.37:0/3526924309 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe91413b320 0x7fe914137f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:20.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe91c4f2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:20.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe91c4f2640 1 -- 172.21.15.37:0/3526924309 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe91413a2e0 con 0x7fe91413b320 2024-04-26T00:23:20.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe91a28e640 1 --2- 172.21.15.37:0/3526924309 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe91413b320 0x7fe914137f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:20.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe91a28e640 10 monclient: get_auth_request con 0x7fe91413b320 auth_method 0 2024-04-26T00:23:20.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe91a28e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:20.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe91a28e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:20.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe91a28e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:20.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe91a28e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:20.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe91a28e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:20.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe91a28e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:20.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe912ffd640 1 -- 172.21.15.37:0/3526924309 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe90801a070 con 0x7fe914131290 2024-04-26T00:23:20.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe91a28e640 10 monclient(hunting): handle_auth_done global_id 44130 payload 824 2024-04-26T00:23:20.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe91a28e640 1 -- 172.21.15.37:0/3526924309 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe91413a780 con 0x7fe91413b320 2024-04-26T00:23:20.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe912ffd640 1 --2- 172.21.15.37:0/3526924309 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe900035e80 0x7fe900038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:20.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe912ffd640 1 -- 172.21.15.37:0/3526924309 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(146..146 src has 1..146) ==== 3931+0+0 (secure 0 0 0) 0x7fe908048aa0 con 0x7fe914131290 2024-04-26T00:23:20.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe91aa8f640 1 --2- 172.21.15.37:0/3526924309 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe900035e80 0x7fe900038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:20.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe91aa8f640 10 monclient: get_auth_request con 0x7fe900035e80 auth_method 0 2024-04-26T00:23:20.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe91a28e640 1 -- 172.21.15.37:0/3526924309 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe91413b320 msgr2=0x7fe914137f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:20.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.974+0000 7fe91a28e640 1 --2- 172.21.15.37:0/3526924309 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe91413b320 0x7fe914137f70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe90c0af000 tx=0x7fe90c019590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:20.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.978+0000 7fe912ffd640 1 -- 172.21.15.37:0/3526924309 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe90c0bd070 con 0x7fe91413b320 2024-04-26T00:23:20.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.978+0000 7fe912ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:20.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.978+0000 7fe912ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:23:20.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.978+0000 7fe912ffd640 10 monclient: dump: 2024-04-26T00:23:20.967 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:20.967 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:20.967 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:20.967 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:20.967 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:20.967 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:20.967 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:20.967 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:20.968 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:20.968 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:20.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.978+0000 7fe912ffd640 1 -- 172.21.15.37:0/3526924309 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe90c0b8640 con 0x7fe91413b320 2024-04-26T00:23:20.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.978+0000 7fe912ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:20.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.978+0000 7fe912ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:23:20.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.978+0000 7fe912ffd640 1 -- 172.21.15.37:0/3526924309 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe91413b320 msgr2=0x7fe914137f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:20.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.978+0000 7fe912ffd640 1 --2- 172.21.15.37:0/3526924309 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe91413b320 0x7fe914137f70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe90c0af000 tx=0x7fe90c019590 comp rx=0 tx=0).stop 2024-04-26T00:23:20.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.978+0000 7fe912ffd640 1 -- 172.21.15.37:0/3526924309 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe91413b320 msgr2=0x7fe914137f70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:23:20.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.978+0000 7fe912ffd640 1 --2- 172.21.15.37:0/3526924309 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe91413b320 0x7fe914137f70 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe90c0af000 tx=0x7fe90c019590 comp rx=0 tx=0).stop 2024-04-26T00:23:20.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.978+0000 7fe91aa8f640 1 --2- 172.21.15.37:0/3526924309 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe900035e80 0x7fe900038340 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe904007fd0 tx=0x7fe904007dc0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:20.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.982+0000 7fe91c4f2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:23:20.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.982+0000 7fe91c4f2640 1 --2- 172.21.15.37:0/3526924309 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe9141384b0 0x7fe91406fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:20.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.982+0000 7fe91c4f2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:20.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.982+0000 7fe91c4f2640 1 -- 172.21.15.37:0/3526924309 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe9141388c0 con 0x7fe9141384b0 2024-04-26T00:23:20.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.982+0000 7fe91a28e640 1 --2- 172.21.15.37:0/3526924309 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe9141384b0 0x7fe91406fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:20.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.982+0000 7fe91a28e640 10 monclient: get_auth_request con 0x7fe9141384b0 auth_method 0 2024-04-26T00:23:20.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.982+0000 7fe91a28e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:20.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.982+0000 7fe91a28e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:20.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.982+0000 7fe91a28e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:20.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.982+0000 7fe91a28e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:20.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.982+0000 7fe91a28e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:20.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.982+0000 7fe91a28e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:20.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.982+0000 7fe91a28e640 10 monclient(hunting): handle_auth_done global_id 44133 payload 824 2024-04-26T00:23:20.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.982+0000 7fe91a28e640 1 -- 172.21.15.37:0/3526924309 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe91404f510 con 0x7fe9141384b0 2024-04-26T00:23:20.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.986+0000 7fe91a28e640 1 -- 172.21.15.37:0/3526924309 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe9141384b0 msgr2=0x7fe91406fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:20.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.986+0000 7fe91a28e640 1 --2- 172.21.15.37:0/3526924309 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe9141384b0 0x7fe91406fb70 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe90c009d90 tx=0x7fe90c0a7000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:20.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.986+0000 7fe912ffd640 1 -- 172.21.15.37:0/3526924309 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe90c0b2040 con 0x7fe9141384b0 2024-04-26T00:23:20.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.986+0000 7fe912ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:20.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.986+0000 7fe912ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:23:20.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.986+0000 7fe912ffd640 10 monclient: dump: 2024-04-26T00:23:20.971 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:20.971 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:20.971 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:20.971 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:20.971 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:20.971 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:20.972 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:20.972 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:20.972 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:20.972 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:20.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.986+0000 7fe912ffd640 1 -- 172.21.15.37:0/3526924309 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7fe90c0c2d20 con 0x7fe9141384b0 2024-04-26T00:23:20.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.986+0000 7fe912ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:23:20.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.986+0000 7fe912ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:23:20.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.986+0000 7fe912ffd640 1 -- 172.21.15.37:0/3526924309 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe9141384b0 msgr2=0x7fe91406fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:20.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.986+0000 7fe912ffd640 1 --2- 172.21.15.37:0/3526924309 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe9141384b0 0x7fe91406fb70 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe90c009d90 tx=0x7fe90c0a7000 comp rx=0 tx=0).stop 2024-04-26T00:23:20.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.986+0000 7fe912ffd640 1 -- 172.21.15.37:0/3526924309 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe9141384b0 msgr2=0x7fe91406fb70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:23:20.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.986+0000 7fe912ffd640 1 --2- 172.21.15.37:0/3526924309 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe9141384b0 0x7fe91406fb70 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe90c009d90 tx=0x7fe90c0a7000 comp rx=0 tx=0).stop 2024-04-26T00:23:20.972 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:23:20.973 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:23:20.973 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:23:20.973 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:23:20.973 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 28, 2024-04-26T00:23:20.973 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 12827, 2024-04-26T00:23:20.973 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:23:20.973 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:23:20.973 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:23:20.973 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:23:20.973 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:20.973 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 6, 2024-04-26T00:23:20.973 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:23:20.973 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:23:20.974 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:23:20.974 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:23:20.974 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:23:20.974 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:23:20.974 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:23:20.974 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:23:20.974 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:23:20.974 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:23:20.974 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:23:20.974 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:23:20.974 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:23:20.974 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:23:20.974 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:20.974 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:23:20.975 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:23:20.975 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:23:20.975 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:23:20.975 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:23:20.975 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:23:20.975 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:23:20.975 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:23:20.975 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:23:20.975 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:23:20.975 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:23:20.975 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:23:20.975 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:23:20.975 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:20.975 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:20.976 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:23:20.976 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:23:20.976 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:23:20.976 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:23:20.976 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:23:20.976 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:23:20.976 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:23:20.976 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:23:20.976 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:23:20.976 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:23:20.976 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:23:20.976 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:23:20.976 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:23:20.976 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:23:20.977 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:23:20.977 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:23:20.977 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:23:20.977 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:23:20.977 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:23:20.977 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:23:20.977 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:23:20.977 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:23:20.977 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:23:20.977 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:23:20.977 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:23:20.978 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:23:20.978 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:23:20.978 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:23:20.978 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:20.978 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:23:20.978 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:20.978 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:23:20.978 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:20.978 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:23:20.978 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:23:20.978 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:23:20.978 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:23:20.979 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:20.979 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:23:20.979 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:23:20.979 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:20.979 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:20.979 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:20.979 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:23:20.979 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:23:20.979 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:20.979 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:20.979 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:20.979 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:20.979 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:23:20.979 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:23:20.980 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:23:20.980 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:23:20.980 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:23:20.980 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:20.980 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:20.980 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:23:20.980 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:23:20.980 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:23:20.980 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:23:20.980 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:20.980 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:23:20.980 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:23:20.980 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:20.980 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:20.981 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:20.981 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:23:20.981 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:23:20.981 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:20.981 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:20.981 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:20.981 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:20.981 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:23:20.981 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:23:20.981 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:23:20.981 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:23:20.981 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:23:20.981 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:20.981 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:20.982 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:23:20.982 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:23:20.982 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:23:20.982 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:23:20.982 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:20.982 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:23:20.982 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:23:20.982 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:20.982 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:20.982 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:20.982 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:23:20.982 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:23:20.982 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:20.983 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:20.983 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:20.983 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:20.983 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:23:20.983 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:23:20.983 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:23:20.983 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:23:20.983 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:23:20.983 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:20.983 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:20.983 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:20.983 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:23:20.983 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:23:20.983 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:20.984 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:23:20.984 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:23:20.984 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:23:20.984 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:20.984 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:20.984 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:23:20.984 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:20.984 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:23:20.984 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:23:20.984 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:23:20.984 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:20.984 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:20.985 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:20.985 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:23:20.985 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:23:20.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.986+0000 7fe91c4f2640 1 -- 172.21.15.37:0/3526924309 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe900035e80 msgr2=0x7fe900038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:20.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.986+0000 7fe91c4f2640 1 --2- 172.21.15.37:0/3526924309 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe900035e80 0x7fe900038340 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe904007fd0 tx=0x7fe904007dc0 comp rx=0 tx=0).stop 2024-04-26T00:23:20.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.986+0000 7fe91c4f2640 10 monclient: shutdown 2024-04-26T00:23:20.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.986+0000 7fe91c4f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:23:20.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.986+0000 7fe91c4f2640 1 -- 172.21.15.37:0/3526924309 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe914131290 msgr2=0x7fe91412f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:20.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.986+0000 7fe91c4f2640 1 --2- 172.21.15.37:0/3526924309 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe914131290 0x7fe91412f460 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe90800ea00 tx=0x7fe90800eec0 comp rx=0 tx=0).stop 2024-04-26T00:23:20.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.986+0000 7fe91aa8f640 1 -- 172.21.15.37:0/3526924309 reap_dead start 2024-04-26T00:23:20.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.986+0000 7fe91c4f2640 1 -- 172.21.15.37:0/3526924309 shutdown_connections 2024-04-26T00:23:20.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.986+0000 7fe91c4f2640 1 --2- 172.21.15.37:0/3526924309 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe914131290 0x7fe91412f460 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:20.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.986+0000 7fe91c4f2640 1 -- 172.21.15.37:0/3526924309 >> 172.21.15.37:0/3526924309 conn(0x7fe9140a1d50 msgr2=0x7fe9140a3930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:23:20.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.986+0000 7fe91c4f2640 1 -- 172.21.15.37:0/3526924309 shutdown_connections 2024-04-26T00:23:20.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:20.986+0000 7fe91c4f2640 1 -- 172.21.15.37:0/3526924309 wait complete. 2024-04-26T00:23:20.987 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:23:20.987 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:23:20.987 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:23:21.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.126+0000 7f190f802640 10 monclient: get_monmap_and_config 2024-04-26T00:23:21.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.126+0000 7f190f802640 10 monclient: build_initial_monmap 2024-04-26T00:23:21.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.126+0000 7f190f802640 10 monclient: monmap: 2024-04-26T00:23:21.106 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:23:21.106 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:21.106 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:23:21.131573+0000 2024-04-26T00:23:21.106 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:23:21.131573+0000 2024-04-26T00:23:21.106 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:23:21.106 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:21.107 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:23:21.107 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:23:21.107 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:23:21.107 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:23:21.107 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:23:21.107 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:21.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190f802640 1 Processor -- start 2024-04-26T00:23:21.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190f802640 1 -- start start 2024-04-26T00:23:21.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190f802640 10 monclient: init 2024-04-26T00:23:21.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190f802640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:21.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190f802640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-04-26T00:23:21.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190f802640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19080a3d90 0x7f190809bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:21.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190f802640 10 monclient(hunting): picked mon.noname-b con 0x7f19080a3d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:23:21.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190f802640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f19080a4170 0x7f19080a45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:21.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190f802640 10 monclient(hunting): picked mon.noname-a con 0x7f19080a4170 addr v2:172.21.15.37:3300/0 2024-04-26T00:23:21.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190f802640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1908133b20 addr v1:172.21.15.37:6789/0 2024-04-26T00:23:21.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190f802640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19080a5150 con 0x7f1908133b20 2024-04-26T00:23:21.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190f802640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:21.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190f802640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f190809b280 con 0x7f19080a4170 2024-04-26T00:23:21.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190f802640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:21.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190f802640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f190809b560 con 0x7f19080a3d90 2024-04-26T00:23:21.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190f802640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:21.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190f802640 10 monclient(hunting): authenticate will time out at 1182.437210s 2024-04-26T00:23:21.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190d59e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19080a3d90 0x7f190809bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:21.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190dd9f640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f1908133b20 0x7f1908133f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:50546/0 (socket says 172.21.15.37:50546) 2024-04-26T00:23:21.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190dd9f640 1 -- 172.21.15.37:0/3884139459 learned_addr learned my addr 172.21.15.37:0/3884139459 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:23:21.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190cd9d640 1 --2- 172.21.15.37:0/3884139459 >> v2:172.21.15.37:3300/0 conn(0x7f19080a4170 0x7f19080a45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:21.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190d59e640 10 monclient(hunting): get_auth_request con 0x7f19080a3d90 auth_method 0 2024-04-26T00:23:21.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190d59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:21.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190d59e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:21.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190d59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:21.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190cd9d640 10 monclient(hunting): get_auth_request con 0x7f19080a4170 auth_method 0 2024-04-26T00:23:21.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190cd9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:21.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190cd9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:21.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190cd9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:21.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190d59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:21.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190d59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:21.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190d59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:21.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190cd9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:21.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190cd9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:21.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190cd9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:21.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190cd9d640 10 monclient(hunting): handle_auth_done global_id 44136 payload 274 2024-04-26T00:23:21.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190cd9d640 1 -- 172.21.15.37:0/3884139459 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19080a3d90 msgr2=0x7f190809bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:23:21.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190cd9d640 1 --2- 172.21.15.37:0/3884139459 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19080a3d90 0x7f190809bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:21.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190cd9d640 1 -- 172.21.15.37:0/3884139459 >> v1:172.21.15.37:6789/0 conn(0x7f1908133b20 legacy=0x7f1908133f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:23:21.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190cd9d640 10 monclient: _finish_hunting 0 2024-04-26T00:23:21.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190cd9d640 1 monclient: found mon.noname-a 2024-04-26T00:23:21.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190cd9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:21.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190cd9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:23:21.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.130+0000 7f190cd9d640 1 -- 172.21.15.37:0/3884139459 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19080a4af0 con 0x7f19080a4170 2024-04-26T00:23:21.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.134+0000 7f190cd9d640 1 -- 172.21.15.37:0/3884139459 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19080a4170 msgr2=0x7f19080a45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:21.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.134+0000 7f190cd9d640 1 --2- 172.21.15.37:0/3884139459 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19080a4170 0x7f19080a45b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f18f80097c0 tx=0x7f18f801c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9a80729063fcea9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:21.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.134+0000 7f18fffff640 1 -- 172.21.15.37:0/3884139459 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f80b0070 con 0x7f19080a4170 2024-04-26T00:23:21.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.134+0000 7f18fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:21.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.134+0000 7f18fffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:23:21.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.134+0000 7f18fffff640 10 monclient: dump: 2024-04-26T00:23:21.117 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:21.117 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:21.117 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:21.117 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:21.117 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:21.117 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:21.117 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:21.117 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:21.118 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:21.118 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:21.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.134+0000 7f18fffff640 10 monclient: _finish_auth 0 2024-04-26T00:23:21.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.134+0000 7f18fffff640 10 monclient: _check_auth_tickets 2024-04-26T00:23:21.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.134+0000 7f18fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:21.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.134+0000 7f190f802640 5 monclient: authenticate success, global_id 44136 2024-04-26T00:23:21.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.134+0000 7f18fffff640 1 -- 172.21.15.37:0/3884139459 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18f80ab400 con 0x7f19080a4170 2024-04-26T00:23:21.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.134+0000 7f18fffff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:23:21.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.134+0000 7f190f802640 10 monclient: get_monmap_and_config success 2024-04-26T00:23:21.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.134+0000 7f190f802640 4 set_mon_vals no callback set 2024-04-26T00:23:21.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.134+0000 7f18fffff640 1 -- 172.21.15.37:0/3884139459 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f80ab7d0 con 0x7f19080a4170 2024-04-26T00:23:21.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.134+0000 7f18fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:21.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.134+0000 7f18fffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:23:21.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.134+0000 7f18fffff640 10 monclient: dump: 2024-04-26T00:23:21.119 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:21.119 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:21.119 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:21.119 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:21.119 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:21.119 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:21.119 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:21.119 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:21.119 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:21.119 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:21.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.134+0000 7f190f802640 10 monclient: shutdown 2024-04-26T00:23:21.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.134+0000 7f190f802640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:23:21.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.134+0000 7f190f802640 1 -- 172.21.15.37:0/3884139459 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19080a4170 msgr2=0x7f19080a45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:21.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.134+0000 7f190f802640 1 --2- 172.21.15.37:0/3884139459 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19080a4170 0x7f19080a45b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f18f80097c0 tx=0x7f18f801c320 comp rx=0 tx=0).stop 2024-04-26T00:23:21.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.134+0000 7f190f802640 1 -- 172.21.15.37:0/3884139459 shutdown_connections 2024-04-26T00:23:21.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.134+0000 7f190f802640 1 --2- 172.21.15.37:0/3884139459 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19080a4170 0x7f19080a45b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:21.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.134+0000 7f190f802640 1 --2- 172.21.15.37:0/3884139459 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19080a3d90 0x7f190809bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:21.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.134+0000 7f190f802640 1 -- 172.21.15.37:0/3884139459 >> 172.21.15.37:0/3884139459 conn(0x7f19080aa0a0 msgr2=0x7f19080a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:23:21.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.134+0000 7f190f802640 1 -- 172.21.15.37:0/3884139459 shutdown_connections 2024-04-26T00:23:21.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.134+0000 7f190f802640 1 -- 172.21.15.37:0/3884139459 wait complete. 2024-04-26T00:23:21.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.134+0000 7f190f802640 10 monclient: build_initial_monmap 2024-04-26T00:23:21.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.134+0000 7f190f802640 10 monclient: monmap: 2024-04-26T00:23:21.120 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:23:21.121 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:23:21.121 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:23:21.121 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:23:21.121 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:23:21.121 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:23:21.121 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:23:21.121 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:23:21.121 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:23:21.121 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:21.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.134+0000 7f190f802640 1 Processor -- start 2024-04-26T00:23:21.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190f802640 1 -- start start 2024-04-26T00:23:21.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190f802640 10 monclient: init 2024-04-26T00:23:21.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190f802640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:21.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190f802640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:23:21.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190f802640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19080a3d90 0x7f19081cf580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:21.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190f802640 10 monclient(hunting): picked mon.noname-a con 0x7f19080a3d90 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:23:21.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190f802640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19080a4170 0x7f190813e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:21.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190f802640 10 monclient(hunting): picked mon.noname-c con 0x7f19080a4170 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:23:21.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190f802640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1908133b20 0x7f190813e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:21.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190f802640 10 monclient(hunting): picked mon.noname-b con 0x7f1908133b20 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:23:21.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190f802640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:21.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190f802640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f1908141e30 con 0x7f19080a3d90 2024-04-26T00:23:21.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190f802640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:21.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190f802640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f190809c690 con 0x7f1908133b20 2024-04-26T00:23:21.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190f802640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:21.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190f802640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f1908141fb0 con 0x7f19080a4170 2024-04-26T00:23:21.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190f802640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:21.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190f802640 10 monclient(hunting): authenticate will time out at 1182.444564s 2024-04-26T00:23:21.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190d59e640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19080a3d90 0x7f19081cf580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:21.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190cd9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19080a4170 0x7f190813e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:21.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190d59e640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19080a3d90 0x7f19081cf580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58118/0 (socket says 172.21.15.37:58118) 2024-04-26T00:23:21.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190cd9d640 10 monclient(hunting): get_auth_request con 0x7f19080a4170 auth_method 0 2024-04-26T00:23:21.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190cd9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:21.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190cd9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:21.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190cd9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:21.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190d59e640 1 -- 172.21.15.37:0/217585375 learned_addr learned my addr 172.21.15.37:0/217585375 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:23:21.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190d59e640 10 monclient(hunting): get_auth_request con 0x7f19080a3d90 auth_method 0 2024-04-26T00:23:21.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190d59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:21.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190d59e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:21.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190d59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:21.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190cd9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:21.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190cd9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:21.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190cd9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:21.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190d59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:21.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190d59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:21.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190dd9f640 1 --2- 172.21.15.37:0/217585375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1908133b20 0x7f190813e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:21.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190d59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:21.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190dd9f640 10 monclient(hunting): get_auth_request con 0x7f1908133b20 auth_method 0 2024-04-26T00:23:21.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190dd9f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:21.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190dd9f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:21.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190dd9f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:21.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190cd9d640 10 monclient(hunting): handle_auth_done global_id 34235 payload 824 2024-04-26T00:23:21.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190cd9d640 1 -- 172.21.15.37:0/217585375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1908133b20 msgr2=0x7f190813e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:21.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190cd9d640 1 --2- 172.21.15.37:0/217585375 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1908133b20 0x7f190813e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:21.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190cd9d640 1 -- 172.21.15.37:0/217585375 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19080a3d90 msgr2=0x7f19081cf580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:21.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190cd9d640 1 --2- 172.21.15.37:0/217585375 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19080a3d90 0x7f19081cf580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:21.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190cd9d640 10 monclient: _finish_hunting 0 2024-04-26T00:23:21.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190cd9d640 1 monclient: found mon.noname-c 2024-04-26T00:23:21.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190cd9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:21.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190cd9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:23:21.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190cd9d640 1 -- 172.21.15.37:0/217585375 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f190813ce30 con 0x7f19080a4170 2024-04-26T00:23:21.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190d59e640 1 --2- 172.21.15.37:0/217585375 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19080a3d90 0x7f19081cf580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:23:21.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190cd9d640 1 -- 172.21.15.37:0/217585375 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19080a4170 msgr2=0x7f190813e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:21.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190cd9d640 1 --2- 172.21.15.37:0/217585375 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19080a4170 0x7f190813e360 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f18f80ab000 tx=0x7f18f80190c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:21.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f18fdffb640 1 -- 172.21.15.37:0/217585375 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f80b8020 con 0x7f19080a4170 2024-04-26T00:23:21.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f18fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:21.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f18fdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:23:21.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f18fdffb640 10 monclient: dump: 2024-04-26T00:23:21.129 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:21.129 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:21.129 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:21.129 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:21.129 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:21.129 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:21.130 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:21.130 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:21.130 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:21.130 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:21.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f18fdffb640 10 monclient: _finish_auth 0 2024-04-26T00:23:21.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f18fdffb640 10 monclient: _check_auth_tickets 2024-04-26T00:23:21.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f18fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:21.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f18fdffb640 1 -- 172.21.15.37:0/217585375 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18f80b4650 con 0x7f19080a4170 2024-04-26T00:23:21.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f18fdffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:23:21.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f18fdffb640 1 -- 172.21.15.37:0/217585375 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f80b0040 con 0x7f19080a4170 2024-04-26T00:23:21.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190f802640 5 monclient: authenticate success, global_id 34235 2024-04-26T00:23:21.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190f802640 10 monclient: _renew_subs 2024-04-26T00:23:21.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190f802640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:21.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190f802640 1 -- 172.21.15.37:0/217585375 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f190813b420 con 0x7f19080a4170 2024-04-26T00:23:21.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f18ff7fe640 4 set_mon_vals no callback set 2024-04-26T00:23:21.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f18fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:21.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f18fdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:23:21.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f18fdffb640 10 monclient: dump: 2024-04-26T00:23:21.131 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:21.131 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:21.131 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:21.131 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:21.131 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:21.131 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:21.131 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:21.131 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:21.131 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:21.131 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:21.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190f802640 10 monclient: _renew_subs 2024-04-26T00:23:21.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190f802640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:21.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.138+0000 7f190f802640 1 -- 172.21.15.37:0/217585375 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f19080a9140 con 0x7f19080a4170 2024-04-26T00:23:21.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.142+0000 7f190f802640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:21.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.142+0000 7f190f802640 1 --2- 172.21.15.37:0/217585375 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f190813ca50 0x7f1908138f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:21.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.142+0000 7f190f802640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:21.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.142+0000 7f190f802640 1 -- 172.21.15.37:0/217585375 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f19081cfac0 con 0x7f190813ca50 2024-04-26T00:23:21.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.142+0000 7f190d59e640 1 --2- 172.21.15.37:0/217585375 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f190813ca50 0x7f1908138f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:21.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.142+0000 7f190d59e640 10 monclient: get_auth_request con 0x7f190813ca50 auth_method 0 2024-04-26T00:23:21.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.142+0000 7f190d59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:21.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.142+0000 7f190d59e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:21.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.142+0000 7f190d59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:21.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.142+0000 7f18fdffb640 1 -- 172.21.15.37:0/217585375 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f18f80bb020 con 0x7f19080a4170 2024-04-26T00:23:21.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.142+0000 7f190d59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:21.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.142+0000 7f190d59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:21.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.142+0000 7f190d59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:21.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.142+0000 7f190d59e640 10 monclient(hunting): handle_auth_done global_id 34238 payload 824 2024-04-26T00:23:21.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.142+0000 7f190d59e640 1 -- 172.21.15.37:0/217585375 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f190804f510 con 0x7f190813ca50 2024-04-26T00:23:21.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.142+0000 7f18fdffb640 1 --2- 172.21.15.37:0/217585375 >> v2:172.21.15.45:6812/2379918623 conn(0x7f18ec035e80 0x7f18ec038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:21.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.142+0000 7f18fdffb640 1 -- 172.21.15.37:0/217585375 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(146..146 src has 1..146) ==== 3931+0+0 (secure 0 0 0) 0x7f18f80e2370 con 0x7f19080a4170 2024-04-26T00:23:21.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.142+0000 7f190dd9f640 1 --2- 172.21.15.37:0/217585375 >> v2:172.21.15.45:6812/2379918623 conn(0x7f18ec035e80 0x7f18ec038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:21.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.142+0000 7f190dd9f640 10 monclient: get_auth_request con 0x7f18ec035e80 auth_method 0 2024-04-26T00:23:21.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.142+0000 7f190d59e640 1 -- 172.21.15.37:0/217585375 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f190813ca50 msgr2=0x7f1908138f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:21.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.142+0000 7f190d59e640 1 --2- 172.21.15.37:0/217585375 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f190813ca50 0x7f1908138f40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f190813a740 tx=0x7f1904007a60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:21.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.142+0000 7f18fdffb640 1 -- 172.21.15.37:0/217585375 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f190401c030 con 0x7f190813ca50 2024-04-26T00:23:21.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.142+0000 7f18fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:21.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.142+0000 7f18fdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:23:21.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.142+0000 7f18fdffb640 10 monclient: dump: 2024-04-26T00:23:21.133 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:21.133 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:21.133 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:21.133 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:21.133 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:21.134 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:21.134 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:21.134 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:21.134 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:21.134 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:21.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.142+0000 7f18fdffb640 1 -- 172.21.15.37:0/217585375 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1904012aa0 con 0x7f190813ca50 2024-04-26T00:23:21.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.142+0000 7f18fdffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:21.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.142+0000 7f18fdffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:23:21.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.142+0000 7f18fdffb640 1 -- 172.21.15.37:0/217585375 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f190813ca50 msgr2=0x7f1908138f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:21.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.142+0000 7f18fdffb640 1 --2- 172.21.15.37:0/217585375 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f190813ca50 0x7f1908138f40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f190813a740 tx=0x7f1904007a60 comp rx=0 tx=0).stop 2024-04-26T00:23:21.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.142+0000 7f18fdffb640 1 -- 172.21.15.37:0/217585375 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f190813ca50 msgr2=0x7f1908138f40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:23:21.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.142+0000 7f18fdffb640 1 --2- 172.21.15.37:0/217585375 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f190813ca50 0x7f1908138f40 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f190813a740 tx=0x7f1904007a60 comp rx=0 tx=0).stop 2024-04-26T00:23:21.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.146+0000 7f190dd9f640 1 --2- 172.21.15.37:0/217585375 >> v2:172.21.15.45:6812/2379918623 conn(0x7f18ec035e80 0x7f18ec038340 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f190813bc50 tx=0x7f1900009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:21.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.150+0000 7f190f802640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:23:21.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.150+0000 7f190f802640 1 --2- 172.21.15.37:0/217585375 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1908139480 0x7f1908139890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:21.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.150+0000 7f190f802640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:21.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.150+0000 7f190f802640 1 -- 172.21.15.37:0/217585375 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f1908139dd0 con 0x7f1908139480 2024-04-26T00:23:21.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.150+0000 7f190d59e640 1 --2- 172.21.15.37:0/217585375 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1908139480 0x7f1908139890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:21.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.150+0000 7f190d59e640 10 monclient: get_auth_request con 0x7f1908139480 auth_method 0 2024-04-26T00:23:21.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.150+0000 7f190d59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:21.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.150+0000 7f190d59e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:21.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.150+0000 7f190d59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:21.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.150+0000 7f190d59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:21.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.150+0000 7f190d59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:21.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.150+0000 7f190d59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:21.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.150+0000 7f190d59e640 10 monclient(hunting): handle_auth_done global_id 34241 payload 824 2024-04-26T00:23:21.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.150+0000 7f190d59e640 1 -- 172.21.15.37:0/217585375 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1908075dc0 con 0x7f1908139480 2024-04-26T00:23:21.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.154+0000 7f190d59e640 1 -- 172.21.15.37:0/217585375 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1908139480 msgr2=0x7f1908139890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:21.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.154+0000 7f190d59e640 1 --2- 172.21.15.37:0/217585375 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1908139480 0x7f1908139890 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f190813a740 tx=0x7f1904004a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:21.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.154+0000 7f18fdffb640 1 -- 172.21.15.37:0/217585375 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1904022d30 con 0x7f1908139480 2024-04-26T00:23:21.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.154+0000 7f18fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:21.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.154+0000 7f18fdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:23:21.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.154+0000 7f18fdffb640 10 monclient: dump: 2024-04-26T00:23:21.137 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:21.137 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:21.137 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:21.137 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:21.137 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:21.137 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:21.137 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:21.137 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:21.137 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:21.137 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:21.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.154+0000 7f18fdffb640 1 -- 172.21.15.37:0/217585375 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4718 (secure 0 0 0) 0x7f1904004000 con 0x7f1908139480 2024-04-26T00:23:21.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.154+0000 7f18fdffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:23:21.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.154+0000 7f18fdffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:23:21.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.154+0000 7f18fdffb640 1 -- 172.21.15.37:0/217585375 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1908139480 msgr2=0x7f1908139890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:21.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.154+0000 7f18fdffb640 1 --2- 172.21.15.37:0/217585375 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1908139480 0x7f1908139890 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f190813a740 tx=0x7f1904004a50 comp rx=0 tx=0).stop 2024-04-26T00:23:21.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.154+0000 7f18fdffb640 1 -- 172.21.15.37:0/217585375 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1908139480 msgr2=0x7f1908139890 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:23:21.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.154+0000 7f18fdffb640 1 --2- 172.21.15.37:0/217585375 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1908139480 0x7f1908139890 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f190813a740 tx=0x7f1904004a50 comp rx=0 tx=0).stop 2024-04-26T00:23:21.138 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:23:21.138 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:23:21.138 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:23:21.138 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:23:21.139 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 28, 2024-04-26T00:23:21.139 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 61431, 2024-04-26T00:23:21.139 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:23:21.139 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:23:21.139 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:23:21.139 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:23:21.139 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:21.139 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 6, 2024-04-26T00:23:21.139 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:23:21.139 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:23:21.139 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:23:21.139 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:23:21.139 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:23:21.139 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:23:21.140 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:23:21.140 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:23:21.140 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:23:21.140 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:23:21.140 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:23:21.140 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:23:21.140 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:23:21.140 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:23:21.140 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:21.140 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:23:21.140 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:23:21.140 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:23:21.140 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:23:21.140 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:23:21.140 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:23:21.140 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:23:21.141 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:23:21.141 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:23:21.141 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:23:21.141 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:23:21.141 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:23:21.141 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:23:21.141 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:21.141 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:21.141 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:23:21.141 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:23:21.141 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:23:21.141 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:23:21.141 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:23:21.141 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:23:21.141 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:23:21.141 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:23:21.142 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:23:21.142 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:23:21.142 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:23:21.142 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:23:21.142 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:23:21.142 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:23:21.142 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:23:21.142 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:23:21.142 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:23:21.142 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:23:21.142 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:23:21.142 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:23:21.142 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:23:21.142 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:23:21.142 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:23:21.143 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:23:21.143 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:23:21.143 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:23:21.143 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:23:21.143 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:23:21.143 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:21.143 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:23:21.143 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:21.143 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:23:21.143 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:21.143 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:23:21.143 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:23:21.143 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:23:21.143 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:23:21.144 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:21.144 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:23:21.144 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:23:21.144 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:21.144 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:21.144 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:21.144 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:23:21.144 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:23:21.144 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:21.144 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:21.144 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:21.144 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:21.145 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:23:21.145 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:23:21.145 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:23:21.145 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:23:21.145 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:23:21.145 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:21.145 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:21.145 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:23:21.145 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:23:21.145 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:23:21.145 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:23:21.145 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:21.145 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:23:21.145 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:23:21.146 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:21.146 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:21.146 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:21.146 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:23:21.146 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:23:21.146 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:21.146 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:21.146 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:21.146 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:21.146 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:23:21.146 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:23:21.146 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:23:21.146 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:23:21.146 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:23:21.147 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:21.147 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:21.147 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:23:21.147 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:23:21.147 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:23:21.147 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:23:21.147 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:21.147 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:23:21.147 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:23:21.147 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:21.147 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:21.147 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:21.147 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:23:21.147 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:23:21.147 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:21.148 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:21.148 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:21.148 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:21.148 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:23:21.148 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:23:21.148 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:23:21.148 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:23:21.148 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:23:21.148 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:21.148 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:21.148 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:21.148 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:23:21.148 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:23:21.148 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:21.148 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:23:21.149 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:23:21.149 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:23:21.149 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:21.149 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:21.149 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:23:21.149 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:21.149 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:23:21.149 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:23:21.149 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:23:21.149 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:21.149 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:21.149 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:23:21.149 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:21.149 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:23:21.150 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:23:21.150 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:23:21.150 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:21.150 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:21.150 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:21.150 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:23:21.150 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:23:21.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.154+0000 7f190f802640 1 -- 172.21.15.37:0/217585375 >> v2:172.21.15.45:6812/2379918623 conn(0x7f18ec035e80 msgr2=0x7f18ec038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:21.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.154+0000 7f190f802640 1 --2- 172.21.15.37:0/217585375 >> v2:172.21.15.45:6812/2379918623 conn(0x7f18ec035e80 0x7f18ec038340 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f190813bc50 tx=0x7f1900009040 comp rx=0 tx=0).stop 2024-04-26T00:23:21.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.154+0000 7f190f802640 10 monclient: shutdown 2024-04-26T00:23:21.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.154+0000 7f190f802640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:23:21.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.154+0000 7f190f802640 1 -- 172.21.15.37:0/217585375 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19080a4170 msgr2=0x7f190813e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:21.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.154+0000 7f190f802640 1 --2- 172.21.15.37:0/217585375 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19080a4170 0x7f190813e360 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f18f80ab000 tx=0x7f18f80190c0 comp rx=0 tx=0).stop 2024-04-26T00:23:21.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.154+0000 7f190dd9f640 1 -- 172.21.15.37:0/217585375 reap_dead start 2024-04-26T00:23:21.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.154+0000 7f190dd9f640 1 -- 172.21.15.37:0/217585375 reap_dead start 2024-04-26T00:23:21.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.154+0000 7f190f802640 1 -- 172.21.15.37:0/217585375 shutdown_connections 2024-04-26T00:23:21.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.154+0000 7f190f802640 1 -- 172.21.15.37:0/217585375 >> 172.21.15.37:0/217585375 conn(0x7f19080aa0a0 msgr2=0x7f19080a2200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:23:21.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.154+0000 7f190f802640 1 -- 172.21.15.37:0/217585375 shutdown_connections 2024-04-26T00:23:21.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.154+0000 7f190f802640 1 -- 172.21.15.37:0/217585375 wait complete. 2024-04-26T00:23:21.152 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:23:21.152 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:23:21.153 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:23:21.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.290+0000 7f2fba56d640 10 monclient: get_monmap_and_config 2024-04-26T00:23:21.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.290+0000 7f2fba56d640 10 monclient: build_initial_monmap 2024-04-26T00:23:21.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.290+0000 7f2fba56d640 10 monclient: monmap: 2024-04-26T00:23:21.269 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:23:21.269 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:21.269 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:23:21.294677+0000 2024-04-26T00:23:21.269 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:23:21.294677+0000 2024-04-26T00:23:21.269 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:23:21.270 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:21.270 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:23:21.270 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:23:21.270 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:23:21.270 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:23:21.270 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:23:21.270 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:21.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.290+0000 7f2fba56d640 1 Processor -- start 2024-04-26T00:23:21.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.290+0000 7f2fba56d640 1 -- start start 2024-04-26T00:23:21.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.290+0000 7f2fba56d640 10 monclient: init 2024-04-26T00:23:21.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.294+0000 7f2fba56d640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:21.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.294+0000 7f2fba56d640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-04-26T00:23:21.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.294+0000 7f2fba56d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f2fb4131a80 0x7f2fb4131e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:21.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.294+0000 7f2fba56d640 10 monclient(hunting): picked mon.noname-a con 0x7f2fb4131a80 addr v2:172.21.15.37:3300/0 2024-04-26T00:23:21.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.294+0000 7f2fba56d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2fb41315c0 0x7f2fb40a13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:21.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.294+0000 7f2fba56d640 10 monclient(hunting): picked mon.noname-c con 0x7f2fb41315c0 addr v2:172.21.15.45:3300/0 2024-04-26T00:23:21.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.294+0000 7f2fba56d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2fb40a1910 addr v1:172.21.15.37:6789/0 2024-04-26T00:23:21.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.294+0000 7f2fba56d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fb4071c20 con 0x7f2fb40a1910 2024-04-26T00:23:21.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.294+0000 7f2fba56d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:21.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.294+0000 7f2fba56d640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f2fb4072f80 con 0x7f2fb4131a80 2024-04-26T00:23:21.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.294+0000 7f2fba56d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:21.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.294+0000 7f2fba56d640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f2fb4071f00 con 0x7f2fb41315c0 2024-04-26T00:23:21.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.294+0000 7f2fba56d640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:21.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.294+0000 7f2fba56d640 10 monclient(hunting): authenticate will time out at 1182.600469s 2024-04-26T00:23:21.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.294+0000 7f2fb3fff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f2fb4131a80 0x7f2fb4131e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:21.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.294+0000 7f2fb8b0a640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f2fb40a1910 0x7f2fb413be70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:50550/0 (socket says 172.21.15.37:50550) 2024-04-26T00:23:21.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.294+0000 7f2fb8b0a640 1 -- 172.21.15.37:0/3814630778 learned_addr learned my addr 172.21.15.37:0/3814630778 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:23:21.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.294+0000 7f2fb3fff640 10 monclient(hunting): get_auth_request con 0x7f2fb4131a80 auth_method 0 2024-04-26T00:23:21.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.294+0000 7f2fb3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:21.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.294+0000 7f2fb3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:21.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.294+0000 7f2fb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:21.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.294+0000 7f2fb3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:21.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.294+0000 7f2fb3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:21.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.294+0000 7f2fb3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:21.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.294+0000 7f2fb37fe640 1 --2- 172.21.15.37:0/3814630778 >> v2:172.21.15.45:3300/0 conn(0x7f2fb41315c0 0x7f2fb40a13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:21.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.294+0000 7f2fb37fe640 10 monclient(hunting): get_auth_request con 0x7f2fb41315c0 auth_method 0 2024-04-26T00:23:21.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.294+0000 7f2fb37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:21.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.294+0000 7f2fb37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:21.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.294+0000 7f2fb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:21.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.294+0000 7f2fb3fff640 10 monclient(hunting): handle_auth_done global_id 44142 payload 274 2024-04-26T00:23:21.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.294+0000 7f2fb3fff640 1 -- 172.21.15.37:0/3814630778 >> v2:172.21.15.45:3300/0 conn(0x7f2fb41315c0 msgr2=0x7f2fb40a13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:23:21.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.294+0000 7f2fb3fff640 1 --2- 172.21.15.37:0/3814630778 >> v2:172.21.15.45:3300/0 conn(0x7f2fb41315c0 0x7f2fb40a13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:21.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.294+0000 7f2fb3fff640 1 -- 172.21.15.37:0/3814630778 >> v1:172.21.15.37:6789/0 conn(0x7f2fb40a1910 legacy=0x7f2fb413be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:23:21.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.294+0000 7f2fb3fff640 10 monclient: _finish_hunting 0 2024-04-26T00:23:21.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.294+0000 7f2fb3fff640 1 monclient: found mon.noname-a 2024-04-26T00:23:21.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.294+0000 7f2fb3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:21.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.294+0000 7f2fb3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:23:21.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.294+0000 7f2fb3fff640 1 -- 172.21.15.37:0/3814630778 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fb40721e0 con 0x7f2fb4131a80 2024-04-26T00:23:21.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fb3fff640 1 -- 172.21.15.37:0/3814630778 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2fb4131a80 msgr2=0x7f2fb4131e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:21.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fb3fff640 1 --2- 172.21.15.37:0/3814630778 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2fb4131a80 0x7f2fb4131e60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2fb406c520 tx=0x7f2fa401c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=163d9244efa5ef69 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:21.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fb2ffd640 1 -- 172.21.15.37:0/3814630778 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa40b2070 con 0x7f2fb4131a80 2024-04-26T00:23:21.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fb2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:21.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fb2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:23:21.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fb2ffd640 10 monclient: dump: 2024-04-26T00:23:21.280 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:21.280 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:21.281 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:21.281 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:21.281 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:21.281 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:21.281 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:21.281 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:21.281 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:21.281 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:21.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fb2ffd640 10 monclient: _finish_auth 0 2024-04-26T00:23:21.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fb2ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:23:21.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fb2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:21.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fb2ffd640 1 -- 172.21.15.37:0/3814630778 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fa40a5780 con 0x7f2fb4131a80 2024-04-26T00:23:21.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fba56d640 5 monclient: authenticate success, global_id 44142 2024-04-26T00:23:21.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fba56d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:23:21.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fb2ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:23:21.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fba56d640 10 monclient: get_monmap_and_config success 2024-04-26T00:23:21.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fba56d640 4 set_mon_vals no callback set 2024-04-26T00:23:21.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fb2ffd640 1 -- 172.21.15.37:0/3814630778 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa40a5b50 con 0x7f2fb4131a80 2024-04-26T00:23:21.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fb2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:21.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fb2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:23:21.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fb2ffd640 10 monclient: dump: 2024-04-26T00:23:21.282 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:21.282 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:21.282 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:21.282 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:21.282 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:21.282 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:21.282 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:21.283 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:21.283 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:21.283 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:21.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fba56d640 10 monclient: shutdown 2024-04-26T00:23:21.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fba56d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:23:21.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fba56d640 1 -- 172.21.15.37:0/3814630778 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2fb4131a80 msgr2=0x7f2fb4131e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:21.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fba56d640 1 --2- 172.21.15.37:0/3814630778 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2fb4131a80 0x7f2fb4131e60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2fb406c520 tx=0x7f2fa401c7b0 comp rx=0 tx=0).stop 2024-04-26T00:23:21.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fba56d640 1 -- 172.21.15.37:0/3814630778 shutdown_connections 2024-04-26T00:23:21.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fba56d640 1 --2- 172.21.15.37:0/3814630778 >> v2:172.21.15.45:3300/0 conn(0x7f2fb41315c0 0x7f2fb40a13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:21.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fba56d640 1 --2- 172.21.15.37:0/3814630778 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2fb4131a80 0x7f2fb4131e60 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:21.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fba56d640 1 -- 172.21.15.37:0/3814630778 >> 172.21.15.37:0/3814630778 conn(0x7f2fb40a24c0 msgr2=0x7f2fb40a28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:23:21.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fba56d640 1 -- 172.21.15.37:0/3814630778 shutdown_connections 2024-04-26T00:23:21.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fba56d640 1 -- 172.21.15.37:0/3814630778 wait complete. 2024-04-26T00:23:21.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fba56d640 10 monclient: build_initial_monmap 2024-04-26T00:23:21.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fba56d640 10 monclient: monmap: 2024-04-26T00:23:21.284 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:23:21.284 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:23:21.284 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:23:21.284 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:23:21.284 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:23:21.284 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:23:21.284 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:23:21.284 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:23:21.284 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:23:21.284 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:21.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fba56d640 1 Processor -- start 2024-04-26T00:23:21.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fba56d640 1 -- start start 2024-04-26T00:23:21.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fba56d640 10 monclient: init 2024-04-26T00:23:21.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fba56d640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:21.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fba56d640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:23:21.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fba56d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fb40a1910 0x7f2fb413c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:21.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fba56d640 10 monclient(hunting): picked mon.noname-b con 0x7f2fb40a1910 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:23:21.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fba56d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2fb41315c0 0x7f2fb41cf2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:21.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fba56d640 10 monclient(hunting): picked mon.noname-c con 0x7f2fb41315c0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:23:21.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fba56d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2fb4131a80 0x7f2fb41cf810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:21.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fba56d640 10 monclient(hunting): picked mon.noname-a con 0x7f2fb4131a80 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:23:21.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fba56d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:21.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fba56d640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb40a33c0 con 0x7f2fb4131a80 2024-04-26T00:23:21.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fba56d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:21.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fba56d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb40a3240 con 0x7f2fb40a1910 2024-04-26T00:23:21.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fba56d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:21.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fba56d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2fb413fdb0 con 0x7f2fb41315c0 2024-04-26T00:23:21.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fba56d640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:21.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.298+0000 7f2fba56d640 10 monclient(hunting): authenticate will time out at 1182.607973s 2024-04-26T00:23:21.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb37fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2fb41315c0 0x7f2fb41cf2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:21.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb37fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2fb41315c0 0x7f2fb41cf2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:46788/0 (socket says 172.21.15.37:46788) 2024-04-26T00:23:21.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb8b0a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2fb4131a80 0x7f2fb41cf810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:21.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb37fe640 1 -- 172.21.15.37:0/1667613426 learned_addr learned my addr 172.21.15.37:0/1667613426 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:23:21.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb37fe640 10 monclient(hunting): get_auth_request con 0x7f2fb41315c0 auth_method 0 2024-04-26T00:23:21.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:21.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:21.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:21.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb8b0a640 10 monclient(hunting): get_auth_request con 0x7f2fb4131a80 auth_method 0 2024-04-26T00:23:21.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb8b0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:21.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb8b0a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:21.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb8b0a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:21.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:21.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:21.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:21.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb3fff640 1 --2- 172.21.15.37:0/1667613426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fb40a1910 0x7f2fb413c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:21.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb3fff640 10 monclient(hunting): get_auth_request con 0x7f2fb40a1910 auth_method 0 2024-04-26T00:23:21.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:21.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:21.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:21.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb8b0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:21.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb8b0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:21.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb8b0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:21.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:21.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:21.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:21.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb37fe640 10 monclient(hunting): handle_auth_done global_id 34244 payload 824 2024-04-26T00:23:21.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb37fe640 1 -- 172.21.15.37:0/1667613426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fb40a1910 msgr2=0x7f2fb413c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:21.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb37fe640 1 --2- 172.21.15.37:0/1667613426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fb40a1910 0x7f2fb413c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:21.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb37fe640 1 -- 172.21.15.37:0/1667613426 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2fb4131a80 msgr2=0x7f2fb41cf810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:21.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb37fe640 1 --2- 172.21.15.37:0/1667613426 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2fb4131a80 0x7f2fb41cf810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:21.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb37fe640 10 monclient: _finish_hunting 0 2024-04-26T00:23:21.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb37fe640 1 monclient: found mon.noname-c 2024-04-26T00:23:21.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:21.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:23:21.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb37fe640 1 -- 172.21.15.37:0/1667613426 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fb4139a20 con 0x7f2fb41315c0 2024-04-26T00:23:21.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb37fe640 1 -- 172.21.15.37:0/1667613426 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2fb41315c0 msgr2=0x7f2fb41cf2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:21.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb37fe640 1 --2- 172.21.15.37:0/1667613426 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2fb41315c0 0x7f2fb41cf2d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2fa800d830 tx=0x7f2fa800dcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:21.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb0ff9640 1 -- 172.21.15.37:0/1667613426 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa800b8d0 con 0x7f2fb41315c0 2024-04-26T00:23:21.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:21.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:23:21.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb0ff9640 10 monclient: dump: 2024-04-26T00:23:21.291 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:21.291 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:21.292 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:21.292 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:21.292 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:21.292 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:21.292 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:21.292 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:21.292 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:21.292 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:21.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb0ff9640 10 monclient: _finish_auth 0 2024-04-26T00:23:21.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb0ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:23:21.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:21.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb0ff9640 1 -- 172.21.15.37:0/1667613426 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fa8019070 con 0x7f2fb41315c0 2024-04-26T00:23:21.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fba56d640 5 monclient: authenticate success, global_id 34244 2024-04-26T00:23:21.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fba56d640 10 monclient: _renew_subs 2024-04-26T00:23:21.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fba56d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:21.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fba56d640 1 -- 172.21.15.37:0/1667613426 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fb413bd80 con 0x7f2fb41315c0 2024-04-26T00:23:21.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb0ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:23:21.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb0ff9640 1 -- 172.21.15.37:0/1667613426 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa800bc70 con 0x7f2fb41315c0 2024-04-26T00:23:21.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:21.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:23:21.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb0ff9640 10 monclient: dump: 2024-04-26T00:23:21.293 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:21.293 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:21.293 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:21.293 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:21.293 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:21.293 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:21.293 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:21.293 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:21.294 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:21.294 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:21.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fb27fc640 4 set_mon_vals no callback set 2024-04-26T00:23:21.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fba56d640 10 monclient: _renew_subs 2024-04-26T00:23:21.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fba56d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:21.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fba56d640 1 -- 172.21.15.37:0/1667613426 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fb413ca40 con 0x7f2fb41315c0 2024-04-26T00:23:21.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fba56d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:21.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fba56d640 1 --2- 172.21.15.37:0/1667613426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fb41cfd50 0x7f2fb412eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:21.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fba56d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:21.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.302+0000 7f2fba56d640 1 -- 172.21.15.37:0/1667613426 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb41d0130 con 0x7f2fb41cfd50 2024-04-26T00:23:21.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.306+0000 7f2fb0ff9640 1 -- 172.21.15.37:0/1667613426 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2fa801e460 con 0x7f2fb41315c0 2024-04-26T00:23:21.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.306+0000 7f2fb0ff9640 1 --2- 172.21.15.37:0/1667613426 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2fa0035e80 0x7f2fa0038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:21.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.306+0000 7f2fb3fff640 1 --2- 172.21.15.37:0/1667613426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fb41cfd50 0x7f2fb412eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:21.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.306+0000 7f2fb3fff640 10 monclient: get_auth_request con 0x7f2fb41cfd50 auth_method 0 2024-04-26T00:23:21.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.306+0000 7f2fb3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:21.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.306+0000 7f2fb3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:21.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.306+0000 7f2fb0ff9640 1 -- 172.21.15.37:0/1667613426 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(146..146 src has 1..146) ==== 3931+0+0 (secure 0 0 0) 0x7f2fa8048f70 con 0x7f2fb41315c0 2024-04-26T00:23:21.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.306+0000 7f2fb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:21.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.306+0000 7f2fb3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:21.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.306+0000 7f2fb3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:21.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.306+0000 7f2fb3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:21.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.306+0000 7f2fb8b0a640 1 --2- 172.21.15.37:0/1667613426 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2fa0035e80 0x7f2fa0038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:21.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.306+0000 7f2fb8b0a640 10 monclient: get_auth_request con 0x7f2fa0035e80 auth_method 0 2024-04-26T00:23:21.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.306+0000 7f2fb3fff640 10 monclient(hunting): handle_auth_done global_id 24292 payload 824 2024-04-26T00:23:21.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.306+0000 7f2fb3fff640 1 -- 172.21.15.37:0/1667613426 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2fb413b5f0 con 0x7f2fb41cfd50 2024-04-26T00:23:21.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.306+0000 7f2fb3fff640 1 -- 172.21.15.37:0/1667613426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fb41cfd50 msgr2=0x7f2fb412eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:21.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.306+0000 7f2fb3fff640 1 --2- 172.21.15.37:0/1667613426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fb41cfd50 0x7f2fb412eaf0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2fa40af000 tx=0x7f2fa4019550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:21.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.306+0000 7f2fb8b0a640 1 --2- 172.21.15.37:0/1667613426 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2fa0035e80 0x7f2fa0038340 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2fac005fd0 tx=0x7f2fac007bf0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:21.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.306+0000 7f2fb0ff9640 1 -- 172.21.15.37:0/1667613426 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa40b2040 con 0x7f2fb41cfd50 2024-04-26T00:23:21.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.306+0000 7f2fb0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:21.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.306+0000 7f2fb0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:23:21.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.306+0000 7f2fb0ff9640 10 monclient: dump: 2024-04-26T00:23:21.296 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:21.296 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:21.296 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:21.296 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:21.296 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:21.296 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:21.296 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:21.296 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:21.296 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:21.296 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:21.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.310+0000 7f2fb0ff9640 1 -- 172.21.15.37:0/1667613426 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2fa40b3040 con 0x7f2fb41cfd50 2024-04-26T00:23:21.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.310+0000 7f2fb0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:21.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.310+0000 7f2fb0ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:23:21.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.310+0000 7f2fb0ff9640 1 -- 172.21.15.37:0/1667613426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fb41cfd50 msgr2=0x7f2fb412eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:21.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.310+0000 7f2fb0ff9640 1 --2- 172.21.15.37:0/1667613426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fb41cfd50 0x7f2fb412eaf0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2fa40af000 tx=0x7f2fa4019550 comp rx=0 tx=0).stop 2024-04-26T00:23:21.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.310+0000 7f2fb0ff9640 1 -- 172.21.15.37:0/1667613426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fb41cfd50 msgr2=0x7f2fb412eaf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:23:21.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.310+0000 7f2fb0ff9640 1 --2- 172.21.15.37:0/1667613426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fb41cfd50 0x7f2fb412eaf0 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2fa40af000 tx=0x7f2fa4019550 comp rx=0 tx=0).stop 2024-04-26T00:23:21.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.314+0000 7f2fba56d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:23:21.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.314+0000 7f2fba56d640 1 --2- 172.21.15.37:0/1667613426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fb402c000 0x7f2fb402c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:21.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.314+0000 7f2fba56d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:21.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.314+0000 7f2fba56d640 1 -- 172.21.15.37:0/1667613426 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb404f510 con 0x7f2fb402c000 2024-04-26T00:23:21.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.314+0000 7f2fb3fff640 1 --2- 172.21.15.37:0/1667613426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fb402c000 0x7f2fb402c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:21.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.314+0000 7f2fb3fff640 10 monclient: get_auth_request con 0x7f2fb402c000 auth_method 0 2024-04-26T00:23:21.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.314+0000 7f2fb3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:21.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.314+0000 7f2fb3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:21.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.314+0000 7f2fb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:21.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.314+0000 7f2fb3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:21.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.314+0000 7f2fb3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:21.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.314+0000 7f2fb3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:21.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.314+0000 7f2fb3fff640 10 monclient(hunting): handle_auth_done global_id 24295 payload 824 2024-04-26T00:23:21.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.314+0000 7f2fb3fff640 1 -- 172.21.15.37:0/1667613426 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2fb4133dc0 con 0x7f2fb402c000 2024-04-26T00:23:21.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.318+0000 7f2fb3fff640 1 -- 172.21.15.37:0/1667613426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fb402c000 msgr2=0x7f2fb402c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:21.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.318+0000 7f2fb3fff640 1 --2- 172.21.15.37:0/1667613426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fb402c000 0x7f2fb402c410 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2fa401ae60 tx=0x7f2fa40a7000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:21.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.318+0000 7f2fb0ff9640 1 -- 172.21.15.37:0/1667613426 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa40c4850 con 0x7f2fb402c000 2024-04-26T00:23:21.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.318+0000 7f2fb0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:21.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.318+0000 7f2fb0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:23:21.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.318+0000 7f2fb0ff9640 10 monclient: dump: 2024-04-26T00:23:21.300 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:21.300 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:21.300 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:21.300 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:21.300 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:21.300 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:21.300 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:21.300 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:21.300 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:21.300 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:21.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.318+0000 7f2fb0ff9640 1 -- 172.21.15.37:0/1667613426 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f2fa40c4850 con 0x7f2fb402c000 2024-04-26T00:23:21.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.318+0000 7f2fb0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:23:21.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.318+0000 7f2fb0ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:23:21.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.318+0000 7f2fb0ff9640 1 -- 172.21.15.37:0/1667613426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fb402c000 msgr2=0x7f2fb402c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:21.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.318+0000 7f2fb0ff9640 1 --2- 172.21.15.37:0/1667613426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fb402c000 0x7f2fb402c410 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2fa401ae60 tx=0x7f2fa40a7000 comp rx=0 tx=0).stop 2024-04-26T00:23:21.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.318+0000 7f2fb0ff9640 1 -- 172.21.15.37:0/1667613426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fb402c000 msgr2=0x7f2fb402c410 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:23:21.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.318+0000 7f2fb0ff9640 1 --2- 172.21.15.37:0/1667613426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2fb402c000 0x7f2fb402c410 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2fa401ae60 tx=0x7f2fa40a7000 comp rx=0 tx=0).stop 2024-04-26T00:23:21.301 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:23:21.301 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:23:21.301 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:23:21.301 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:23:21.301 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 28, 2024-04-26T00:23:21.301 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 109489, 2024-04-26T00:23:21.302 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:23:21.302 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:23:21.302 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:23:21.302 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:23:21.302 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:21.302 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 6, 2024-04-26T00:23:21.302 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:23:21.302 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:23:21.302 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:23:21.302 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:23:21.302 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:23:21.302 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:23:21.303 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:23:21.303 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:23:21.303 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:23:21.303 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:23:21.303 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:23:21.303 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:23:21.303 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:23:21.303 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:23:21.303 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:21.303 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:23:21.303 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:23:21.303 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:23:21.303 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:23:21.303 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:23:21.303 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:23:21.304 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:23:21.304 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:23:21.304 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:23:21.304 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:23:21.304 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:23:21.304 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:23:21.304 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:23:21.304 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:21.304 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:21.304 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:23:21.304 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:23:21.304 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:23:21.304 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:23:21.304 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:23:21.304 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:23:21.305 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:23:21.305 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:23:21.305 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:23:21.305 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:23:21.305 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:23:21.305 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:23:21.305 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:23:21.305 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:23:21.305 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:23:21.305 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:23:21.305 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:23:21.305 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:23:21.305 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:23:21.305 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:23:21.305 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:23:21.306 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:23:21.306 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:23:21.306 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:23:21.306 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:23:21.306 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:23:21.306 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:23:21.306 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:23:21.306 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:21.306 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:23:21.306 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:21.306 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:23:21.307 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:21.307 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:23:21.307 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:23:21.307 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:23:21.307 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:23:21.307 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:21.307 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:23:21.307 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:23:21.307 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:21.307 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:21.307 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:21.307 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:23:21.307 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:23:21.307 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:21.308 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:21.308 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:21.308 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:21.308 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:23:21.308 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:23:21.308 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:23:21.308 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:23:21.308 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:23:21.308 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:21.308 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:21.308 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:23:21.308 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:23:21.308 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:23:21.308 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:23:21.308 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:21.309 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:23:21.309 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:23:21.309 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:21.309 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:21.309 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:21.309 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:23:21.309 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:23:21.309 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:21.309 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:21.309 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:21.309 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:21.309 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:23:21.309 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:23:21.309 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:23:21.309 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:23:21.310 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:23:21.310 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:21.310 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:21.310 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:23:21.310 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:23:21.310 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:23:21.310 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:23:21.310 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:21.310 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:23:21.310 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:23:21.310 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:21.310 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:21.310 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:21.310 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:23:21.311 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:23:21.311 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:21.311 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:21.311 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:21.311 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:21.311 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:23:21.311 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:23:21.311 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:23:21.311 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:23:21.311 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:23:21.311 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:21.311 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:21.311 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:21.311 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:23:21.312 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:23:21.312 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:21.312 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:23:21.312 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:23:21.312 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:23:21.312 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:21.312 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:21.312 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:23:21.312 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:21.312 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:23:21.312 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:23:21.312 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:23:21.312 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:21.312 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:21.313 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:23:21.313 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:21.313 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:23:21.313 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:23:21.313 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:23:21.313 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:21.313 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:21.313 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:23:21.313 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:21.313 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:23:21.313 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:23:21.313 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:23:21.313 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:21.313 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:21.313 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:21.314 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:23:21.314 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:23:21.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.318+0000 7f2fba56d640 1 -- 172.21.15.37:0/1667613426 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2fa0035e80 msgr2=0x7f2fa0038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:21.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.318+0000 7f2fba56d640 1 --2- 172.21.15.37:0/1667613426 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2fa0035e80 0x7f2fa0038340 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2fac005fd0 tx=0x7f2fac007bf0 comp rx=0 tx=0).stop 2024-04-26T00:23:21.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.318+0000 7f2fba56d640 10 monclient: shutdown 2024-04-26T00:23:21.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.318+0000 7f2fba56d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:23:21.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.318+0000 7f2fba56d640 1 -- 172.21.15.37:0/1667613426 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2fb41315c0 msgr2=0x7f2fb41cf2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:21.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.318+0000 7f2fba56d640 1 --2- 172.21.15.37:0/1667613426 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2fb41315c0 0x7f2fb41cf2d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2fa800d830 tx=0x7f2fa800dcf0 comp rx=0 tx=0).stop 2024-04-26T00:23:21.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.318+0000 7f2fb8b0a640 1 -- 172.21.15.37:0/1667613426 reap_dead start 2024-04-26T00:23:21.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.322+0000 7f2fb8b0a640 1 -- 172.21.15.37:0/1667613426 reap_dead start 2024-04-26T00:23:21.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.322+0000 7f2fba56d640 1 -- 172.21.15.37:0/1667613426 shutdown_connections 2024-04-26T00:23:21.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.322+0000 7f2fba56d640 1 -- 172.21.15.37:0/1667613426 >> 172.21.15.37:0/1667613426 conn(0x7f2fb40a24c0 msgr2=0x7f2fb412faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:23:21.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.322+0000 7f2fba56d640 1 -- 172.21.15.37:0/1667613426 shutdown_connections 2024-04-26T00:23:21.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.322+0000 7f2fba56d640 1 -- 172.21.15.37:0/1667613426 wait complete. 2024-04-26T00:23:21.316 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:23:21.316 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-04-26T00:23:21.317 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-26T00:23:21.317 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-04-26T00:23:21.317 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-04-26T00:23:21.317 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-26T00:23:21.317 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T00:23:21.329 INFO:tasks.ceph.mon.a:Stopped 2024-04-26T00:23:21.329 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-26T00:23:21.329 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:23:21.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.514+0000 7f6a8a5ad640 10 monclient: get_monmap_and_config 2024-04-26T00:23:21.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.514+0000 7f6a8a5ad640 10 monclient: build_initial_monmap 2024-04-26T00:23:21.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.514+0000 7f6a8a5ad640 10 monclient: monmap: 2024-04-26T00:23:21.492 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:23:21.492 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:21.492 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:23:21.517304+0000 2024-04-26T00:23:21.492 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:23:21.517304+0000 2024-04-26T00:23:21.492 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:23:21.492 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:21.492 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:23:21.492 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:23:21.492 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:23:21.492 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:23:21.492 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:23:21.493 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:21.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.514+0000 7f6a8a5ad640 1 Processor -- start 2024-04-26T00:23:21.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.514+0000 7f6a8a5ad640 1 -- start start 2024-04-26T00:23:21.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.514+0000 7f6a8a5ad640 10 monclient: init 2024-04-26T00:23:21.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.514+0000 7f6a8a5ad640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:21.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.514+0000 7f6a8a5ad640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-04-26T00:23:21.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.514+0000 7f6a8a5ad640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6a84136e10 addr v1:172.21.15.45:6789/0 2024-04-26T00:23:21.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.514+0000 7f6a8a5ad640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6a84137900 addr v1:172.21.15.37:6789/0 2024-04-26T00:23:21.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.514+0000 7f6a8a5ad640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6a8409c5b0 0x7f6a8413fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:21.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.514+0000 7f6a8a5ad640 10 monclient(hunting): picked mon.noname-a con 0x7f6a8409c5b0 addr v2:172.21.15.37:3300/0 2024-04-26T00:23:21.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.514+0000 7f6a8a5ad640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a8409b680 con 0x7f6a84137900 2024-04-26T00:23:21.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.514+0000 7f6a8a5ad640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a8409bbd0 con 0x7f6a84136e10 2024-04-26T00:23:21.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.514+0000 7f6a8a5ad640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:21.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.514+0000 7f6a8a5ad640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f6a8409b8f0 con 0x7f6a8409c5b0 2024-04-26T00:23:21.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.514+0000 7f6a8a5ad640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:21.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.514+0000 7f6a8a5ad640 10 monclient(hunting): authenticate will time out at 1182.823091s 2024-04-26T00:23:21.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.514+0000 7f6a837fe640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f6a84137900 legacy=0x7f6a8413c340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:23:21.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.514+0000 7f6a88b4a640 1 -- >> v2:172.21.15.37:3300/0 conn(0x7f6a8409c5b0 msgr2=0x7f6a8413fa70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:23:21.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.514+0000 7f6a88b4a640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6a8409c5b0 0x7f6a8413fa70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:23:21.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.514+0000 7f6a83fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f6a84136e10 0x7f6a841371f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.37:49218/0 (socket says 172.21.15.37:49218) 2024-04-26T00:23:21.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.514+0000 7f6a83fff640 1 -- 172.21.15.37:0/1736430943 learned_addr learned my addr 172.21.15.37:0/1736430943 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:23:21.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a83fff640 1 -- 172.21.15.37:0/1736430943 >> v1:172.21.15.45:6789/0 conn(0x7f6a84136e10 legacy=0x7f6a841371f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:21.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a82ffd640 1 -- 172.21.15.37:0/1736430943 <== mon.1 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 324243298 0 0) 0x7f6a8409bbd0 con 0x7f6a84136e10 2024-04-26T00:23:21.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a82ffd640 10 monclient(hunting): _negotiate 2024-04-26T00:23:21.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a82ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:21.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:21.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a82ffd640 10 monclient(hunting): my global_id is 24298 2024-04-26T00:23:21.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a82ffd640 1 -- 172.21.15.37:0/1736430943 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a6c003530 con 0x7f6a84136e10 2024-04-26T00:23:21.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a82ffd640 1 -- 172.21.15.37:0/1736430943 <== mon.1 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3413684758 0 0) 0x7f6a6c003530 con 0x7f6a84136e10 2024-04-26T00:23:21.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a82ffd640 1 -- 172.21.15.37:0/1736430943 >> v2:172.21.15.37:3300/0 conn(0x7f6a8409c5b0 msgr2=0x7f6a8413fa70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:23:21.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a82ffd640 1 --2- 172.21.15.37:0/1736430943 >> v2:172.21.15.37:3300/0 conn(0x7f6a8409c5b0 0x7f6a8413fa70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:21.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a82ffd640 1 -- 172.21.15.37:0/1736430943 >> v1:172.21.15.37:6789/0 conn(0x7f6a84137900 legacy=0x7f6a8413c340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:23:21.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a82ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:23:21.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a82ffd640 1 monclient: found mon.noname-c-legacy 2024-04-26T00:23:21.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a82ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:21.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a82ffd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.45:6789/0 2024-04-26T00:23:21.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a82ffd640 1 -- 172.21.15.37:0/1736430943 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a840a5d90 con 0x7f6a84136e10 2024-04-26T00:23:21.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a82ffd640 10 monclient: _finish_auth 0 2024-04-26T00:23:21.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a82ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:23:21.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a82ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:21.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a82ffd640 1 -- 172.21.15.37:0/1736430943 <== mon.1 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 866326386 0 0) 0x7f6a74001930 con 0x7f6a84136e10 2024-04-26T00:23:21.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a82ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:21.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a82ffd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-04-26T00:23:21.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a82ffd640 10 monclient: dump: 2024-04-26T00:23:21.500 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:21.500 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:21.500 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:21.500 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:21.500 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:21.500 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:21.500 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:21.500 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:21.500 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:21.500 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:21.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a82ffd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] but i'm connected to v1:172.21.15.45:6789/0, reconnecting 2024-04-26T00:23:21.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a82ffd640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:21.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a82ffd640 1 -- 172.21.15.37:0/1736430943 >> v1:172.21.15.45:6789/0 conn(0x7f6a84136e10 legacy=0x7f6a841371f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:21.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a82ffd640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:23:21.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a82ffd640 1 --2- 172.21.15.37:0/1736430943 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a6c0039c0 0x7f6a6c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:21.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a82ffd640 10 monclient(hunting): picked mon.a con 0x7f6a6c0039c0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:23:21.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a82ffd640 1 --2- 172.21.15.37:0/1736430943 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a6c004810 0x7f6a6c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:21.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a82ffd640 10 monclient(hunting): picked mon.b con 0x7f6a6c004810 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:23:21.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a82ffd640 1 --2- 172.21.15.37:0/1736430943 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a6c005380 0x7f6a6c00b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:21.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a82ffd640 10 monclient(hunting): picked mon.c con 0x7f6a6c005380 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:23:21.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a82ffd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:21.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a82ffd640 1 -- 172.21.15.37:0/1736430943 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8409b8f0 con 0x7f6a6c0039c0 2024-04-26T00:23:21.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a82ffd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:21.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a82ffd640 1 -- 172.21.15.37:0/1736430943 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6a6c00c2d0 con 0x7f6a6c004810 2024-04-26T00:23:21.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a82ffd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:21.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a82ffd640 1 -- 172.21.15.37:0/1736430943 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f6a6c00c4d0 con 0x7f6a6c005380 2024-04-26T00:23:21.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a82ffd640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:21.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a82ffd640 10 monclient(hunting): _finish_auth 0 2024-04-26T00:23:21.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a88b4a640 1 --2- 172.21.15.37:0/1736430943 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a6c005380 0x7f6a6c00b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:21.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a83fff640 1 -- 172.21.15.37:0/1736430943 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a6c0039c0 msgr2=0x7f6a6c003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:23:21.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a88b4a640 10 monclient(hunting): get_auth_request con 0x7f6a6c005380 auth_method 0 2024-04-26T00:23:21.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a88b4a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:21.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a88b4a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:21.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a88b4a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:23:21.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a83fff640 1 --2- 172.21.15.37:0/1736430943 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a6c0039c0 0x7f6a6c003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:23:21.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a88b4a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:21.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a88b4a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:21.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a88b4a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:23:21.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a837fe640 1 --2- 172.21.15.37:0/1736430943 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a6c004810 0x7f6a6c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:21.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a837fe640 10 monclient(hunting): get_auth_request con 0x7f6a6c004810 auth_method 0 2024-04-26T00:23:21.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:21.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a837fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:21.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a837fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:23:21.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a88b4a640 10 monclient(hunting): handle_auth_done global_id 24298 payload 293 2024-04-26T00:23:21.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a88b4a640 1 -- 172.21.15.37:0/1736430943 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a6c004810 msgr2=0x7f6a6c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:23:21.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a88b4a640 1 --2- 172.21.15.37:0/1736430943 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a6c004810 0x7f6a6c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:21.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a88b4a640 1 -- 172.21.15.37:0/1736430943 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a6c0039c0 msgr2=0x7f6a6c003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:23:21.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a88b4a640 1 --2- 172.21.15.37:0/1736430943 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a6c0039c0 0x7f6a6c003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:21.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a88b4a640 10 monclient: _finish_hunting 0 2024-04-26T00:23:21.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a88b4a640 1 monclient: found mon.c 2024-04-26T00:23:21.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a88b4a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:21.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a88b4a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:21.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a88b4a640 1 -- 172.21.15.37:0/1736430943 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a6c003530 con 0x7f6a6c005380 2024-04-26T00:23:21.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a88b4a640 10 monclient: _finish_auth 0 2024-04-26T00:23:21.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a88b4a640 10 monclient: _check_auth_tickets 2024-04-26T00:23:21.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a88b4a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:21.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a8a5ad640 5 monclient: authenticate success, global_id 24298 2024-04-26T00:23:21.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.518+0000 7f6a8a5ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:23:21.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.522+0000 7f6a88b4a640 1 -- 172.21.15.37:0/1736430943 reap_dead start 2024-04-26T00:23:21.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.522+0000 7f6a88b4a640 1 -- 172.21.15.37:0/1736430943 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a6c005380 msgr2=0x7f6a6c00b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:21.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.522+0000 7f6a88b4a640 1 --2- 172.21.15.37:0/1736430943 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a6c005380 0x7f6a6c00b9f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c009e30 tx=0x7f6a7c01caf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=461e8132e0ef7e9e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:21.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.522+0000 7f6a82ffd640 1 -- 172.21.15.37:0/1736430943 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a7c009a30 con 0x7f6a6c005380 2024-04-26T00:23:21.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.522+0000 7f6a82ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:21.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.522+0000 7f6a82ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:23:21.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.522+0000 7f6a82ffd640 10 monclient: dump: 2024-04-26T00:23:21.509 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:21.510 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:21.510 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:21.510 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:21.510 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:21.510 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:21.510 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:21.510 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:21.510 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:21.510 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:21.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.522+0000 7f6a8a5ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:23:21.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.522+0000 7f6a82ffd640 1 -- 172.21.15.37:0/1736430943 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a7c0a9070 con 0x7f6a6c005380 2024-04-26T00:23:21.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.522+0000 7f6a82ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:23:21.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.522+0000 7f6a8a5ad640 10 monclient: get_monmap_and_config success 2024-04-26T00:23:21.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.522+0000 7f6a8a5ad640 4 set_mon_vals no callback set 2024-04-26T00:23:21.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.522+0000 7f6a82ffd640 1 -- 172.21.15.37:0/1736430943 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a7c0a5a50 con 0x7f6a6c005380 2024-04-26T00:23:21.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.522+0000 7f6a82ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:21.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.522+0000 7f6a82ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:23:21.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.522+0000 7f6a82ffd640 10 monclient: dump: 2024-04-26T00:23:21.511 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:21.511 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:21.511 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:21.511 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:21.511 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:21.511 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:21.511 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:21.511 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:21.511 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:21.511 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:21.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a8a5ad640 10 monclient: shutdown 2024-04-26T00:23:21.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a8a5ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:23:21.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a8a5ad640 1 -- 172.21.15.37:0/1736430943 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a6c005380 msgr2=0x7f6a6c00b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:21.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a8a5ad640 1 --2- 172.21.15.37:0/1736430943 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a6c005380 0x7f6a6c00b9f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c009e30 tx=0x7f6a7c01caf0 comp rx=0 tx=0).stop 2024-04-26T00:23:21.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a8a5ad640 1 -- 172.21.15.37:0/1736430943 shutdown_connections 2024-04-26T00:23:21.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a8a5ad640 1 --2- 172.21.15.37:0/1736430943 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a6c005380 0x7f6a6c00b9f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:21.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a8a5ad640 1 -- 172.21.15.37:0/1736430943 >> 172.21.15.37:0/1736430943 conn(0x7f6a840a64e0 msgr2=0x7f6a840a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:23:21.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a8a5ad640 1 -- 172.21.15.37:0/1736430943 shutdown_connections 2024-04-26T00:23:21.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a8a5ad640 1 -- 172.21.15.37:0/1736430943 wait complete. 2024-04-26T00:23:21.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a8a5ad640 10 monclient: build_initial_monmap 2024-04-26T00:23:21.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a8a5ad640 10 monclient: monmap: 2024-04-26T00:23:21.512 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:23:21.512 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:23:21.512 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:23:21.512 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:23:21.512 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:23:21.512 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:23:21.513 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:23:21.513 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:23:21.513 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:23:21.513 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:21.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a8a5ad640 1 Processor -- start 2024-04-26T00:23:21.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a8a5ad640 1 -- start start 2024-04-26T00:23:21.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a8a5ad640 10 monclient: init 2024-04-26T00:23:21.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a8a5ad640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:21.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a8a5ad640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:23:21.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a8a5ad640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a6c005380 0x7f6a8413f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:21.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a8a5ad640 10 monclient(hunting): picked mon.noname-b con 0x7f6a6c005380 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:23:21.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a8a5ad640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a841382e0 0x7f6a8413f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:21.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a8a5ad640 10 monclient(hunting): picked mon.noname-c con 0x7f6a841382e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:23:21.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a8a5ad640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a841ce6c0 0x7f6a841ceb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:21.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a8a5ad640 10 monclient(hunting): picked mon.noname-a con 0x7f6a841ce6c0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:23:21.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a8a5ad640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:21.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a8a5ad640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f6a84137580 con 0x7f6a841ce6c0 2024-04-26T00:23:21.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a8a5ad640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:21.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a8a5ad640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6a84137400 con 0x7f6a6c005380 2024-04-26T00:23:21.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a8a5ad640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:21.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a8a5ad640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f6a84137700 con 0x7f6a841382e0 2024-04-26T00:23:21.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a8a5ad640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:21.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a8a5ad640 10 monclient(hunting): authenticate will time out at 1182.834024s 2024-04-26T00:23:21.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a88b4a640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a841ce6c0 msgr2=0x7f6a841ceb40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:23:21.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a88b4a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a841ce6c0 0x7f6a841ceb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:23:21.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a837fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a841382e0 0x7f6a8413f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:21.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a837fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a841382e0 0x7f6a8413f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:46810/0 (socket says 172.21.15.37:46810) 2024-04-26T00:23:21.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a837fe640 1 -- 172.21.15.37:0/845213934 learned_addr learned my addr 172.21.15.37:0/845213934 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:23:21.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a837fe640 10 monclient(hunting): get_auth_request con 0x7f6a841382e0 auth_method 0 2024-04-26T00:23:21.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:21.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a837fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:21.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a837fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:21.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a83fff640 1 --2- 172.21.15.37:0/845213934 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a6c005380 0x7f6a8413f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:21.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a83fff640 10 monclient(hunting): get_auth_request con 0x7f6a6c005380 auth_method 0 2024-04-26T00:23:21.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:21.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a83fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:21.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a83fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:21.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:21.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:21.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:21.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a83fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:21.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a83fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:21.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a83fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:21.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a837fe640 10 monclient(hunting): handle_auth_done global_id 34247 payload 824 2024-04-26T00:23:21.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a837fe640 1 -- 172.21.15.37:0/845213934 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a6c005380 msgr2=0x7f6a8413f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:21.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a837fe640 1 --2- 172.21.15.37:0/845213934 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a6c005380 0x7f6a8413f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:21.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a837fe640 1 -- 172.21.15.37:0/845213934 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a841ce6c0 msgr2=0x7f6a841ceb40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:23:21.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a837fe640 1 --2- 172.21.15.37:0/845213934 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a841ce6c0 0x7f6a841ceb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:21.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a837fe640 10 monclient: _finish_hunting 0 2024-04-26T00:23:21.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a837fe640 1 monclient: found mon.noname-c 2024-04-26T00:23:21.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:21.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a837fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:23:21.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.526+0000 7f6a837fe640 1 -- 172.21.15.37:0/845213934 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a84138df0 con 0x7f6a841382e0 2024-04-26T00:23:21.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.530+0000 7f6a837fe640 1 -- 172.21.15.37:0/845213934 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a841382e0 msgr2=0x7f6a8413f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:21.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.530+0000 7f6a837fe640 1 --2- 172.21.15.37:0/845213934 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a841382e0 0x7f6a8413f640 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6a7800d4e0 tx=0x7f6a7800d9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:21.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.530+0000 7f6a80ff9640 1 -- 172.21.15.37:0/845213934 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a78016020 con 0x7f6a841382e0 2024-04-26T00:23:21.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.530+0000 7f6a80ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:21.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.530+0000 7f6a80ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:23:21.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.530+0000 7f6a80ff9640 10 monclient: dump: 2024-04-26T00:23:21.517 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:21.517 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:21.517 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:21.517 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:21.517 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:21.517 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:21.517 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:21.517 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:21.517 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:21.517 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:21.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.530+0000 7f6a80ff9640 10 monclient: _finish_auth 0 2024-04-26T00:23:21.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.530+0000 7f6a80ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:23:21.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.530+0000 7f6a80ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:21.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.530+0000 7f6a80ff9640 1 -- 172.21.15.37:0/845213934 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a78003de0 con 0x7f6a841382e0 2024-04-26T00:23:21.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.530+0000 7f6a80ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:23:21.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.530+0000 7f6a80ff9640 1 -- 172.21.15.37:0/845213934 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a780052f0 con 0x7f6a841382e0 2024-04-26T00:23:21.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.530+0000 7f6a80ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:21.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.530+0000 7f6a80ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:23:21.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.530+0000 7f6a80ff9640 10 monclient: dump: 2024-04-26T00:23:21.518 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:21.518 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:21.518 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:21.518 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:21.518 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:21.518 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:21.518 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:21.518 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:21.518 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:21.518 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:21.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.530+0000 7f6a81ffb640 4 set_mon_vals no callback set 2024-04-26T00:23:21.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.530+0000 7f6a8a5ad640 5 monclient: authenticate success, global_id 34247 2024-04-26T00:23:21.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.530+0000 7f6a8a5ad640 10 monclient: _renew_subs 2024-04-26T00:23:21.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.530+0000 7f6a8a5ad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:21.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.530+0000 7f6a8a5ad640 1 -- 172.21.15.37:0/845213934 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a8413a970 con 0x7f6a841382e0 2024-04-26T00:23:21.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.530+0000 7f6a8a5ad640 10 monclient: _renew_subs 2024-04-26T00:23:21.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.530+0000 7f6a8a5ad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:21.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.530+0000 7f6a8a5ad640 1 -- 172.21.15.37:0/845213934 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a8413a0d0 con 0x7f6a841382e0 2024-04-26T00:23:21.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.530+0000 7f6a8a5ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:21.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.530+0000 7f6a8a5ad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:21.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.530+0000 7f6a80ff9640 1 -- 172.21.15.37:0/845213934 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f6a78019020 con 0x7f6a841382e0 2024-04-26T00:23:21.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.530+0000 7f6a8a5ad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:21.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.530+0000 7f6a8a5ad640 1 -- 172.21.15.37:0/845213934 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a84075910 con 0x7f6a841382e0 2024-04-26T00:23:21.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.530+0000 7f6a80ff9640 1 --2- 172.21.15.37:0/845213934 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6a70035e30 0x7f6a700382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:21.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.530+0000 7f6a80ff9640 1 -- 172.21.15.37:0/845213934 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(146..146 src has 1..146) ==== 3931+0+0 (secure 0 0 0) 0x7f6a78048440 con 0x7f6a841382e0 2024-04-26T00:23:21.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.530+0000 7f6a83fff640 1 --2- 172.21.15.37:0/845213934 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6a70035e30 0x7f6a700382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:21.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.530+0000 7f6a83fff640 10 monclient: get_auth_request con 0x7f6a70035e30 auth_method 0 2024-04-26T00:23:21.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.534+0000 7f6a83fff640 1 --2- 172.21.15.37:0/845213934 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6a70035e30 0x7f6a700382f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6a74001b00 tx=0x7f6a74001a90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:21.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.534+0000 7f6a80ff9640 1 -- 172.21.15.37:0/845213934 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6a78020030 con 0x7f6a841382e0 2024-04-26T00:23:21.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.534+0000 7f6a80ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:21.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.534+0000 7f6a80ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:23:21.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.754+0000 7f6a8a5ad640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:23:21.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.754+0000 7f6a8a5ad640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:23:21.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.754+0000 7f6a8a5ad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:21.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.754+0000 7f6a8a5ad640 1 -- 172.21.15.37:0/845213934 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6a840a0310 con 0x7f6a841382e0 2024-04-26T00:23:21.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.754+0000 7f6a80ff9640 1 -- 172.21.15.37:0/845213934 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f6a78047190 con 0x7f6a841382e0 2024-04-26T00:23:21.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.754+0000 7f6a80ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:23:21.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.754+0000 7f6a80ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:23:21.731 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:23:21.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.754+0000 7f6a8a5ad640 1 -- 172.21.15.37:0/845213934 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6a70035e30 msgr2=0x7f6a700382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:21.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.754+0000 7f6a8a5ad640 1 --2- 172.21.15.37:0/845213934 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6a70035e30 0x7f6a700382f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6a74001b00 tx=0x7f6a74001a90 comp rx=0 tx=0).stop 2024-04-26T00:23:21.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.754+0000 7f6a8a5ad640 10 monclient: shutdown 2024-04-26T00:23:21.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.754+0000 7f6a8a5ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:23:21.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.754+0000 7f6a8a5ad640 1 -- 172.21.15.37:0/845213934 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a841382e0 msgr2=0x7f6a8413f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:21.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.754+0000 7f6a8a5ad640 1 --2- 172.21.15.37:0/845213934 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a841382e0 0x7f6a8413f640 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6a7800d4e0 tx=0x7f6a7800d9a0 comp rx=0 tx=0).stop 2024-04-26T00:23:21.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.754+0000 7f6a8a5ad640 1 -- 172.21.15.37:0/845213934 shutdown_connections 2024-04-26T00:23:21.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.754+0000 7f6a8a5ad640 1 --2- 172.21.15.37:0/845213934 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6a70035e30 0x7f6a700382f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:21.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.754+0000 7f6a8a5ad640 1 --2- 172.21.15.37:0/845213934 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a841ce6c0 0x7f6a841ceb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:21.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.754+0000 7f6a8a5ad640 1 --2- 172.21.15.37:0/845213934 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a841382e0 0x7f6a8413f640 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:21.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.754+0000 7f6a8a5ad640 1 --2- 172.21.15.37:0/845213934 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a6c005380 0x7f6a8413f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:21.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.754+0000 7f6a8a5ad640 1 -- 172.21.15.37:0/845213934 >> 172.21.15.37:0/845213934 conn(0x7f6a840a64e0 msgr2=0x7f6a8409d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:23:21.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.754+0000 7f6a8a5ad640 1 -- 172.21.15.37:0/845213934 shutdown_connections 2024-04-26T00:23:21.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:21.754+0000 7f6a8a5ad640 1 -- 172.21.15.37:0/845213934 wait complete. 2024-04-26T00:23:24.749 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:23:24.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.906+0000 7f90ea39a640 10 monclient: get_monmap_and_config 2024-04-26T00:23:24.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.906+0000 7f90ea39a640 10 monclient: build_initial_monmap 2024-04-26T00:23:24.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.906+0000 7f90ea39a640 10 monclient: monmap: 2024-04-26T00:23:24.884 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:23:24.884 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:24.884 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:23:24.909239+0000 2024-04-26T00:23:24.884 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:23:24.909239+0000 2024-04-26T00:23:24.884 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:23:24.884 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:24.884 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:23:24.884 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:23:24.884 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:23:24.884 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:23:24.884 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:23:24.885 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:24.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.906+0000 7f90ea39a640 1 Processor -- start 2024-04-26T00:23:24.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.906+0000 7f90ea39a640 1 -- start start 2024-04-26T00:23:24.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.906+0000 7f90ea39a640 10 monclient: init 2024-04-26T00:23:24.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.906+0000 7f90ea39a640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:24.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.906+0000 7f90ea39a640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-04-26T00:23:24.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.906+0000 7f90ea39a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f90e4133b20 addr v1:172.21.15.45:6789/0 2024-04-26T00:23:24.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.906+0000 7f90ea39a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f90e4133f00 addr v1:172.21.15.37:6789/0 2024-04-26T00:23:24.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.906+0000 7f90ea39a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f90e413a060 0x7f90e413c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:24.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.906+0000 7f90ea39a640 10 monclient(hunting): picked mon.noname-b con 0x7f90e413a060 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:23:24.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.906+0000 7f90ea39a640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90e409fe20 con 0x7f90e4133f00 2024-04-26T00:23:24.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.906+0000 7f90ea39a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90e4072d00 con 0x7f90e4133b20 2024-04-26T00:23:24.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.906+0000 7f90ea39a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:24.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.906+0000 7f90ea39a640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f90e409e620 con 0x7f90e413a060 2024-04-26T00:23:24.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.906+0000 7f90ea39a640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:24.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.906+0000 7f90ea39a640 10 monclient(hunting): authenticate will time out at 1186.214925s 2024-04-26T00:23:24.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.906+0000 7f90e37fe640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f90e4133f00 legacy=0x7f90e4134350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:23:24.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.906+0000 7f90e8937640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f90e413a060 0x7f90e413c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:24.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.906+0000 7f90e8937640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f90e413a060 0x7f90e413c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:46820/0 (socket says 172.21.15.37:46820) 2024-04-26T00:23:24.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.906+0000 7f90e8937640 1 -- 172.21.15.37:0/1757164823 learned_addr learned my addr 172.21.15.37:0/1757164823 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:23:24.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.906+0000 7f90e8937640 10 monclient(hunting): get_auth_request con 0x7f90e413a060 auth_method 0 2024-04-26T00:23:24.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.906+0000 7f90e8937640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:24.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.906+0000 7f90e8937640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:24.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.906+0000 7f90e8937640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:24.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.906+0000 7f90e8937640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:24.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.906+0000 7f90e8937640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:24.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.906+0000 7f90e8937640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:24.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.906+0000 7f90e8937640 10 monclient(hunting): handle_auth_done global_id 34250 payload 274 2024-04-26T00:23:24.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.906+0000 7f90e8937640 1 -- 172.21.15.37:0/1757164823 >> v1:172.21.15.45:6789/0 conn(0x7f90e4133b20 legacy=0x7f90e409e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:23:24.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.906+0000 7f90e8937640 1 -- 172.21.15.37:0/1757164823 >> v1:172.21.15.37:6789/0 conn(0x7f90e4133f00 legacy=0x7f90e4134350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:23:24.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.906+0000 7f90e8937640 10 monclient: _finish_hunting 0 2024-04-26T00:23:24.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.906+0000 7f90e8937640 1 monclient: found mon.noname-b 2024-04-26T00:23:24.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.906+0000 7f90e8937640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:24.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.906+0000 7f90e8937640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:23:24.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.906+0000 7f90e8937640 1 -- 172.21.15.37:0/1757164823 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90e409ba60 con 0x7f90e413a060 2024-04-26T00:23:24.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e8937640 1 -- 172.21.15.37:0/1757164823 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f90e413a060 msgr2=0x7f90e413c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:24.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e8937640 1 --2- 172.21.15.37:0/1757164823 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f90e413a060 0x7f90e413c450 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f90e409bc30 tx=0x7f90d401c240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b15d0e6ee34bbc35 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:24.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e2ffd640 1 -- 172.21.15.37:0/1757164823 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d40b0070 con 0x7f90e413a060 2024-04-26T00:23:24.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:24.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:23:24.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e2ffd640 10 monclient: dump: 2024-04-26T00:23:24.892 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:24.892 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:24.892 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:24.892 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:24.892 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:24.893 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:24.893 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:24.893 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:24.893 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:24.893 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:24.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e2ffd640 10 monclient: _finish_auth 0 2024-04-26T00:23:24.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e2ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:23:24.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:24.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e2ffd640 1 -- 172.21.15.37:0/1757164823 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90d40ab600 con 0x7f90e413a060 2024-04-26T00:23:24.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e2ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:23:24.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e2ffd640 1 -- 172.21.15.37:0/1757164823 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d40ab9d0 con 0x7f90e413a060 2024-04-26T00:23:24.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90ea39a640 5 monclient: authenticate success, global_id 34250 2024-04-26T00:23:24.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90ea39a640 10 monclient: get_monmap_and_config success 2024-04-26T00:23:24.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90ea39a640 4 set_mon_vals no callback set 2024-04-26T00:23:24.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:24.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:23:24.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e2ffd640 10 monclient: dump: 2024-04-26T00:23:24.894 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:24.894 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:24.894 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:24.894 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:24.894 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:24.894 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:24.895 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:24.895 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:24.895 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:24.895 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:24.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90ea39a640 10 monclient: shutdown 2024-04-26T00:23:24.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90ea39a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:23:24.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90ea39a640 1 -- 172.21.15.37:0/1757164823 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f90e413a060 msgr2=0x7f90e413c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:24.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90ea39a640 1 --2- 172.21.15.37:0/1757164823 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f90e413a060 0x7f90e413c450 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f90e409bc30 tx=0x7f90d401c240 comp rx=0 tx=0).stop 2024-04-26T00:23:24.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90ea39a640 1 -- 172.21.15.37:0/1757164823 shutdown_connections 2024-04-26T00:23:24.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90ea39a640 1 --2- 172.21.15.37:0/1757164823 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f90e413a060 0x7f90e413c450 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:24.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90ea39a640 1 -- 172.21.15.37:0/1757164823 >> 172.21.15.37:0/1757164823 conn(0x7f90e40a1ad0 msgr2=0x7f90e40a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:23:24.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90ea39a640 1 -- 172.21.15.37:0/1757164823 shutdown_connections 2024-04-26T00:23:24.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90ea39a640 1 -- 172.21.15.37:0/1757164823 wait complete. 2024-04-26T00:23:24.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90ea39a640 10 monclient: build_initial_monmap 2024-04-26T00:23:24.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90ea39a640 10 monclient: monmap: 2024-04-26T00:23:24.900 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:23:24.900 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:23:24.901 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:23:24.901 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:23:24.901 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:23:24.901 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:23:24.901 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:23:24.901 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:23:24.901 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:23:24.901 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:24.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90ea39a640 1 Processor -- start 2024-04-26T00:23:24.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90ea39a640 1 -- start start 2024-04-26T00:23:24.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90ea39a640 10 monclient: init 2024-04-26T00:23:24.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90ea39a640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:24.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90ea39a640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:23:24.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90ea39a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90e4133b20 0x7f90e406f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:24.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90ea39a640 10 monclient(hunting): picked mon.noname-b con 0x7f90e4133b20 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:23:24.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90ea39a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f90e4133f00 0x7f90e406fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:24.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90ea39a640 10 monclient(hunting): picked mon.noname-a con 0x7f90e4133f00 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:23:24.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90ea39a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f90e413a060 0x7f90e40700a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:24.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90ea39a640 10 monclient(hunting): picked mon.noname-c con 0x7f90e413a060 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:23:24.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90ea39a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:24.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90ea39a640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f90e40a05f0 con 0x7f90e4133f00 2024-04-26T00:23:24.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90ea39a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:24.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90ea39a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f90e40a0470 con 0x7f90e4133b20 2024-04-26T00:23:24.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90ea39a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:24.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90ea39a640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f90e40a0770 con 0x7f90e413a060 2024-04-26T00:23:24.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90ea39a640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:24.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90ea39a640 10 monclient(hunting): authenticate will time out at 1186.223270s 2024-04-26T00:23:24.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e37fe640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f90e4133f00 msgr2=0x7f90e406fb60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:23:24.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e37fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f90e4133f00 0x7f90e406fb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:23:24.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e8937640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f90e413a060 0x7f90e40700a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:24.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e8937640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f90e413a060 0x7f90e40700a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:46832/0 (socket says 172.21.15.37:46832) 2024-04-26T00:23:24.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e8937640 1 -- 172.21.15.37:0/846791738 learned_addr learned my addr 172.21.15.37:0/846791738 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:23:24.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e8937640 10 monclient(hunting): get_auth_request con 0x7f90e413a060 auth_method 0 2024-04-26T00:23:24.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e8937640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:24.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e8937640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:24.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e8937640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:24.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e8937640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:24.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e8937640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:24.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e8937640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:24.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e3fff640 1 --2- 172.21.15.37:0/846791738 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90e4133b20 0x7f90e406f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:24.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e3fff640 10 monclient(hunting): get_auth_request con 0x7f90e4133b20 auth_method 0 2024-04-26T00:23:24.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:24.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:24.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:24.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e8937640 10 monclient(hunting): handle_auth_done global_id 34253 payload 824 2024-04-26T00:23:24.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e8937640 1 -- 172.21.15.37:0/846791738 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90e4133b20 msgr2=0x7f90e406f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:24.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e8937640 1 --2- 172.21.15.37:0/846791738 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90e4133b20 0x7f90e406f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:24.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e8937640 1 -- 172.21.15.37:0/846791738 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f90e4133f00 msgr2=0x7f90e406fb60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:23:24.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e8937640 1 --2- 172.21.15.37:0/846791738 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f90e4133f00 0x7f90e406fb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:24.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e8937640 10 monclient: _finish_hunting 0 2024-04-26T00:23:24.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e8937640 1 monclient: found mon.noname-c 2024-04-26T00:23:24.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e8937640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:24.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e8937640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:23:24.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.914+0000 7f90e8937640 1 -- 172.21.15.37:0/846791738 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90e40a0000 con 0x7f90e413a060 2024-04-26T00:23:24.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.918+0000 7f90e8937640 1 -- 172.21.15.37:0/846791738 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f90e413a060 msgr2=0x7f90e40700a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:24.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.918+0000 7f90e8937640 1 --2- 172.21.15.37:0/846791738 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f90e413a060 0x7f90e40700a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f90d401a8f0 tx=0x7f90d4018fe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:24.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.918+0000 7f90e0ff9640 1 -- 172.21.15.37:0/846791738 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d40b9020 con 0x7f90e413a060 2024-04-26T00:23:24.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.918+0000 7f90e0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:24.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.918+0000 7f90e0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:23:24.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.918+0000 7f90e0ff9640 10 monclient: dump: 2024-04-26T00:23:24.906 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:24.906 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:24.906 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:24.906 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:24.906 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:24.906 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:24.906 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:24.906 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:24.906 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:24.906 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:24.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.918+0000 7f90e0ff9640 10 monclient: _finish_auth 0 2024-04-26T00:23:24.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.918+0000 7f90e0ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:23:24.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.918+0000 7f90e0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:24.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.918+0000 7f90e0ff9640 1 -- 172.21.15.37:0/846791738 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90d40b1040 con 0x7f90e413a060 2024-04-26T00:23:24.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.918+0000 7f90ea39a640 5 monclient: authenticate success, global_id 34253 2024-04-26T00:23:24.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.918+0000 7f90ea39a640 10 monclient: _renew_subs 2024-04-26T00:23:24.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.918+0000 7f90ea39a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:24.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.918+0000 7f90ea39a640 1 -- 172.21.15.37:0/846791738 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90e412cec0 con 0x7f90e413a060 2024-04-26T00:23:24.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.918+0000 7f90e0ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:23:24.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.918+0000 7f90e0ff9640 1 -- 172.21.15.37:0/846791738 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d40b0050 con 0x7f90e413a060 2024-04-26T00:23:24.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.918+0000 7f90e0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:24.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.918+0000 7f90e0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:23:24.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.918+0000 7f90e0ff9640 10 monclient: dump: 2024-04-26T00:23:24.907 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:24.908 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:24.908 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:24.908 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:24.908 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:24.908 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:24.908 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:24.908 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:24.908 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:24.908 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:24.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.918+0000 7f90e27fc640 4 set_mon_vals no callback set 2024-04-26T00:23:24.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.918+0000 7f90ea39a640 10 monclient: _renew_subs 2024-04-26T00:23:24.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.918+0000 7f90ea39a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:24.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.918+0000 7f90ea39a640 1 -- 172.21.15.37:0/846791738 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f90e40716e0 con 0x7f90e413a060 2024-04-26T00:23:24.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.918+0000 7f90e0ff9640 1 -- 172.21.15.37:0/846791738 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f90d40bc020 con 0x7f90e413a060 2024-04-26T00:23:24.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.918+0000 7f90ea39a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:24.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.918+0000 7f90ea39a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:24.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.918+0000 7f90ea39a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:24.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.918+0000 7f90ea39a640 1 -- 172.21.15.37:0/846791738 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90e4070e40 con 0x7f90e413a060 2024-04-26T00:23:24.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.918+0000 7f90e0ff9640 1 --2- 172.21.15.37:0/846791738 >> v2:172.21.15.45:6812/2379918623 conn(0x7f90c4035e80 0x7f90c4038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:24.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.918+0000 7f90e0ff9640 1 -- 172.21.15.37:0/846791738 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(146..146 src has 1..146) ==== 3931+0+0 (secure 0 0 0) 0x7f90d40e27c0 con 0x7f90e413a060 2024-04-26T00:23:24.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.918+0000 7f90e3fff640 1 --2- 172.21.15.37:0/846791738 >> v2:172.21.15.45:6812/2379918623 conn(0x7f90c4035e80 0x7f90c4038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:24.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.922+0000 7f90e3fff640 10 monclient: get_auth_request con 0x7f90c4035e80 auth_method 0 2024-04-26T00:23:24.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.922+0000 7f90e3fff640 1 --2- 172.21.15.37:0/846791738 >> v2:172.21.15.45:6812/2379918623 conn(0x7f90c4035e80 0x7f90c4038340 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f90dc005530 tx=0x7f90dc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:24.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.926+0000 7f90e0ff9640 1 -- 172.21.15.37:0/846791738 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f90d40c3070 con 0x7f90e413a060 2024-04-26T00:23:24.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.926+0000 7f90e0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:24.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:24.926+0000 7f90e0ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:23:25.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:25.142+0000 7f90ea39a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:23:25.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:25.142+0000 7f90ea39a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:23:25.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:25.142+0000 7f90ea39a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:25.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:25.142+0000 7f90ea39a640 1 -- 172.21.15.37:0/846791738 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f90e4066f40 con 0x7f90e413a060 2024-04-26T00:23:25.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:25.142+0000 7f90e0ff9640 1 -- 172.21.15.37:0/846791738 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f90e40a0000 con 0x7f90e413a060 2024-04-26T00:23:25.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:25.142+0000 7f90e0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:23:25.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:25.142+0000 7f90e0ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:23:25.122 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:23:25.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:25.146+0000 7f90ea39a640 1 -- 172.21.15.37:0/846791738 >> v2:172.21.15.45:6812/2379918623 conn(0x7f90c4035e80 msgr2=0x7f90c4038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:25.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:25.146+0000 7f90ea39a640 1 --2- 172.21.15.37:0/846791738 >> v2:172.21.15.45:6812/2379918623 conn(0x7f90c4035e80 0x7f90c4038340 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f90dc005530 tx=0x7f90dc009040 comp rx=0 tx=0).stop 2024-04-26T00:23:25.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:25.146+0000 7f90ea39a640 10 monclient: shutdown 2024-04-26T00:23:25.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:25.146+0000 7f90ea39a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:23:25.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:25.146+0000 7f90ea39a640 1 -- 172.21.15.37:0/846791738 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f90e413a060 msgr2=0x7f90e40700a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:25.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:25.146+0000 7f90ea39a640 1 --2- 172.21.15.37:0/846791738 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f90e413a060 0x7f90e40700a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f90d401a8f0 tx=0x7f90d4018fe0 comp rx=0 tx=0).stop 2024-04-26T00:23:25.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:25.146+0000 7f90ea39a640 1 -- 172.21.15.37:0/846791738 shutdown_connections 2024-04-26T00:23:25.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:25.146+0000 7f90ea39a640 1 --2- 172.21.15.37:0/846791738 >> v2:172.21.15.45:6812/2379918623 conn(0x7f90c4035e80 0x7f90c4038340 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:25.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:25.146+0000 7f90ea39a640 1 --2- 172.21.15.37:0/846791738 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f90e413a060 0x7f90e40700a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:25.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:25.146+0000 7f90ea39a640 1 --2- 172.21.15.37:0/846791738 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f90e4133f00 0x7f90e406fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:25.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:25.146+0000 7f90ea39a640 1 --2- 172.21.15.37:0/846791738 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90e4133b20 0x7f90e406f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:25.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:25.146+0000 7f90ea39a640 1 -- 172.21.15.37:0/846791738 >> 172.21.15.37:0/846791738 conn(0x7f90e40a1ad0 msgr2=0x7f90e40a3760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:23:25.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:25.146+0000 7f90ea39a640 1 -- 172.21.15.37:0/846791738 shutdown_connections 2024-04-26T00:23:25.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:25.146+0000 7f90ea39a640 1 -- 172.21.15.37:0/846791738 wait complete. 2024-04-26T00:23:28.140 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:23:28.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.294+0000 7f2d6a5df640 10 monclient: get_monmap_and_config 2024-04-26T00:23:28.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.294+0000 7f2d6a5df640 10 monclient: build_initial_monmap 2024-04-26T00:23:28.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.294+0000 7f2d6a5df640 10 monclient: monmap: 2024-04-26T00:23:28.272 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:23:28.272 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:28.272 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:23:28.297494+0000 2024-04-26T00:23:28.272 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:23:28.297494+0000 2024-04-26T00:23:28.272 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:23:28.273 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:28.273 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:23:28.273 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:23:28.273 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:23:28.273 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:23:28.273 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:23:28.273 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:28.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.294+0000 7f2d6a5df640 1 Processor -- start 2024-04-26T00:23:28.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.294+0000 7f2d6a5df640 1 -- start start 2024-04-26T00:23:28.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.294+0000 7f2d6a5df640 10 monclient: init 2024-04-26T00:23:28.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.294+0000 7f2d6a5df640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:28.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.294+0000 7f2d6a5df640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-04-26T00:23:28.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.294+0000 7f2d6a5df640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2d64136e10 0x7f2d6409c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:28.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.294+0000 7f2d6a5df640 10 monclient(hunting): picked mon.noname-b con 0x7f2d64136e10 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:23:28.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.294+0000 7f2d6a5df640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2d641371f0 addr v1:172.21.15.37:6789/0 2024-04-26T00:23:28.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.294+0000 7f2d6a5df640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f2d6413c340 0x7f2d6413e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:28.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.294+0000 7f2d6a5df640 10 monclient(hunting): picked mon.noname-a con 0x7f2d6413c340 addr v2:172.21.15.37:3300/0 2024-04-26T00:23:28.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.294+0000 7f2d6a5df640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d6409b680 con 0x7f2d641371f0 2024-04-26T00:23:28.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.294+0000 7f2d6a5df640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:28.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.294+0000 7f2d6a5df640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f2d6409b8f0 con 0x7f2d6413c340 2024-04-26T00:23:28.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.294+0000 7f2d6a5df640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:28.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.294+0000 7f2d6a5df640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2d6409bbd0 con 0x7f2d64136e10 2024-04-26T00:23:28.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.294+0000 7f2d6a5df640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:28.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.294+0000 7f2d6a5df640 10 monclient(hunting): authenticate will time out at 1189.603812s 2024-04-26T00:23:28.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.294+0000 7f2d5b7fe640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f2d641371f0 legacy=0x7f2d641375d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:23:28.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.294+0000 7f2d63fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2d64136e10 0x7f2d6409c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:28.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.294+0000 7f2d68b7c640 1 -- >> v2:172.21.15.37:3300/0 conn(0x7f2d6413c340 msgr2=0x7f2d6413e730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:23:28.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.298+0000 7f2d68b7c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f2d6413c340 0x7f2d6413e730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:23:28.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.298+0000 7f2d63fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2d64136e10 0x7f2d6409c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:46844/0 (socket says 172.21.15.37:46844) 2024-04-26T00:23:28.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.298+0000 7f2d63fff640 1 -- 172.21.15.37:0/2338345598 learned_addr learned my addr 172.21.15.37:0/2338345598 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:23:28.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.298+0000 7f2d63fff640 10 monclient(hunting): get_auth_request con 0x7f2d64136e10 auth_method 0 2024-04-26T00:23:28.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.298+0000 7f2d63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:28.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.298+0000 7f2d63fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:28.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.298+0000 7f2d63fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:28.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.298+0000 7f2d63fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:28.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.298+0000 7f2d63fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:28.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.298+0000 7f2d63fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:28.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.298+0000 7f2d63fff640 10 monclient(hunting): handle_auth_done global_id 34256 payload 274 2024-04-26T00:23:28.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.298+0000 7f2d63fff640 1 -- 172.21.15.37:0/2338345598 >> v2:172.21.15.37:3300/0 conn(0x7f2d6413c340 msgr2=0x7f2d6413e730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:23:28.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.298+0000 7f2d63fff640 1 --2- 172.21.15.37:0/2338345598 >> v2:172.21.15.37:3300/0 conn(0x7f2d6413c340 0x7f2d6413e730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:28.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.298+0000 7f2d63fff640 1 -- 172.21.15.37:0/2338345598 >> v1:172.21.15.37:6789/0 conn(0x7f2d641371f0 legacy=0x7f2d641375d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:23:28.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.298+0000 7f2d63fff640 10 monclient: _finish_hunting 0 2024-04-26T00:23:28.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.298+0000 7f2d63fff640 1 monclient: found mon.noname-b 2024-04-26T00:23:28.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.298+0000 7f2d63fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:28.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.298+0000 7f2d63fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:23:28.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.298+0000 7f2d63fff640 1 -- 172.21.15.37:0/2338345598 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d640a5d90 con 0x7f2d64136e10 2024-04-26T00:23:28.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d63fff640 1 -- 172.21.15.37:0/2338345598 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2d64136e10 msgr2=0x7f2d6409c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:28.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d63fff640 1 --2- 172.21.15.37:0/2338345598 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2d64136e10 0x7f2d6409c5b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2d540098a0 tx=0x7f2d5401c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50aa441972121536 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:28.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d637fe640 1 -- 172.21.15.37:0/2338345598 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d540b0070 con 0x7f2d64136e10 2024-04-26T00:23:28.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:28.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d637fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:23:28.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d637fe640 10 monclient: dump: 2024-04-26T00:23:28.281 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:28.281 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:28.281 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:28.281 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:28.281 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:28.281 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:28.281 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:28.281 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:28.281 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:28.281 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:28.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d637fe640 10 monclient: _finish_auth 0 2024-04-26T00:23:28.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d637fe640 10 monclient: _check_auth_tickets 2024-04-26T00:23:28.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:28.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d637fe640 1 -- 172.21.15.37:0/2338345598 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d540ab600 con 0x7f2d64136e10 2024-04-26T00:23:28.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d6a5df640 5 monclient: authenticate success, global_id 34256 2024-04-26T00:23:28.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d6a5df640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:23:28.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d637fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:23:28.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d6a5df640 10 monclient: get_monmap_and_config success 2024-04-26T00:23:28.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d637fe640 1 -- 172.21.15.37:0/2338345598 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d540ab9d0 con 0x7f2d64136e10 2024-04-26T00:23:28.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d6a5df640 4 set_mon_vals no callback set 2024-04-26T00:23:28.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:28.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d637fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:23:28.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d637fe640 10 monclient: dump: 2024-04-26T00:23:28.282 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:28.282 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:28.282 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:28.282 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:28.282 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:28.283 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:28.283 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:28.283 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:28.283 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:28.283 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:28.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d6a5df640 10 monclient: shutdown 2024-04-26T00:23:28.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d6a5df640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:23:28.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d6a5df640 1 -- 172.21.15.37:0/2338345598 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2d64136e10 msgr2=0x7f2d6409c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:28.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d6a5df640 1 --2- 172.21.15.37:0/2338345598 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2d64136e10 0x7f2d6409c5b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2d540098a0 tx=0x7f2d5401c3e0 comp rx=0 tx=0).stop 2024-04-26T00:23:28.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d6a5df640 1 -- 172.21.15.37:0/2338345598 shutdown_connections 2024-04-26T00:23:28.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d6a5df640 1 --2- 172.21.15.37:0/2338345598 >> v2:172.21.15.37:3300/0 conn(0x7f2d6413c340 0x7f2d6413e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:28.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d6a5df640 1 --2- 172.21.15.37:0/2338345598 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2d64136e10 0x7f2d6409c5b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:28.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d6a5df640 1 -- 172.21.15.37:0/2338345598 >> 172.21.15.37:0/2338345598 conn(0x7f2d640a64e0 msgr2=0x7f2d640a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:23:28.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d6a5df640 1 -- 172.21.15.37:0/2338345598 shutdown_connections 2024-04-26T00:23:28.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d6a5df640 1 -- 172.21.15.37:0/2338345598 wait complete. 2024-04-26T00:23:28.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d6a5df640 10 monclient: build_initial_monmap 2024-04-26T00:23:28.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d6a5df640 10 monclient: monmap: 2024-04-26T00:23:28.284 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:23:28.284 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:23:28.284 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:23:28.284 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:23:28.284 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:23:28.284 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:23:28.284 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:23:28.284 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:23:28.284 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:23:28.284 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:28.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d6a5df640 1 Processor -- start 2024-04-26T00:23:28.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d6a5df640 1 -- start start 2024-04-26T00:23:28.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d6a5df640 10 monclient: init 2024-04-26T00:23:28.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d6a5df640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:28.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d6a5df640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:23:28.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d6a5df640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d64136e10 0x7f2d64135e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:28.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d6a5df640 10 monclient(hunting): picked mon.noname-b con 0x7f2d64136e10 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:23:28.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d6a5df640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2d641371f0 0x7f2d641363b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:28.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d6a5df640 10 monclient(hunting): picked mon.noname-c con 0x7f2d641371f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:23:28.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d6a5df640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d6413c340 0x7f2d64138d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:28.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d6a5df640 10 monclient(hunting): picked mon.noname-a con 0x7f2d6413c340 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:23:28.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d6a5df640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:28.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d6a5df640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f2d64141820 con 0x7f2d6413c340 2024-04-26T00:23:28.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d6a5df640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:28.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d6a5df640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2d641416a0 con 0x7f2d64136e10 2024-04-26T00:23:28.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d6a5df640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:28.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d6a5df640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2d641419a0 con 0x7f2d641371f0 2024-04-26T00:23:28.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d6a5df640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:28.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d6a5df640 10 monclient(hunting): authenticate will time out at 1189.611101s 2024-04-26T00:23:28.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d68b7c640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d6413c340 msgr2=0x7f2d64138d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:23:28.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d5b7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2d641371f0 0x7f2d641363b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:28.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d5b7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2d641371f0 0x7f2d641363b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:46860/0 (socket says 172.21.15.37:46860) 2024-04-26T00:23:28.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d68b7c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d6413c340 0x7f2d64138d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:23:28.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d5b7fe640 1 -- 172.21.15.37:0/3679308818 learned_addr learned my addr 172.21.15.37:0/3679308818 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:23:28.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d5b7fe640 10 monclient(hunting): get_auth_request con 0x7f2d641371f0 auth_method 0 2024-04-26T00:23:28.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:28.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d5b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:28.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:28.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d5b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:28.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d5b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:28.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d5b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:28.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d63fff640 1 --2- 172.21.15.37:0/3679308818 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d64136e10 0x7f2d64135e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:28.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d63fff640 10 monclient(hunting): get_auth_request con 0x7f2d64136e10 auth_method 0 2024-04-26T00:23:28.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:28.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d63fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:28.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d63fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:28.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d5b7fe640 10 monclient(hunting): handle_auth_done global_id 34259 payload 824 2024-04-26T00:23:28.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d5b7fe640 1 -- 172.21.15.37:0/3679308818 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d64136e10 msgr2=0x7f2d64135e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:28.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d5b7fe640 1 --2- 172.21.15.37:0/3679308818 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d64136e10 0x7f2d64135e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:28.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d5b7fe640 1 -- 172.21.15.37:0/3679308818 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d6413c340 msgr2=0x7f2d64138d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:23:28.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d5b7fe640 1 --2- 172.21.15.37:0/3679308818 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d6413c340 0x7f2d64138d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:28.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d5b7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:23:28.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d5b7fe640 1 monclient: found mon.noname-c 2024-04-26T00:23:28.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d5b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:28.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d5b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:23:28.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.302+0000 7f2d5b7fe640 1 -- 172.21.15.37:0/3679308818 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d64120950 con 0x7f2d641371f0 2024-04-26T00:23:28.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.306+0000 7f2d5b7fe640 1 -- 172.21.15.37:0/3679308818 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2d641371f0 msgr2=0x7f2d641363b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:28.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.306+0000 7f2d5b7fe640 1 --2- 172.21.15.37:0/3679308818 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2d641371f0 0x7f2d641363b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c00b550 tx=0x7f2d4c00ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:28.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.306+0000 7f2d617fa640 1 -- 172.21.15.37:0/3679308818 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d4c005070 con 0x7f2d641371f0 2024-04-26T00:23:28.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.306+0000 7f2d617fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:28.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.306+0000 7f2d617fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:23:28.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.306+0000 7f2d617fa640 10 monclient: dump: 2024-04-26T00:23:28.293 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:28.293 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:28.293 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:28.293 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:28.293 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:28.293 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:28.293 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:28.293 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:28.293 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:28.293 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:28.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.306+0000 7f2d617fa640 10 monclient: _finish_auth 0 2024-04-26T00:23:28.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.306+0000 7f2d617fa640 10 monclient: _check_auth_tickets 2024-04-26T00:23:28.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.306+0000 7f2d617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:28.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.306+0000 7f2d6a5df640 5 monclient: authenticate success, global_id 34259 2024-04-26T00:23:28.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.306+0000 7f2d617fa640 1 -- 172.21.15.37:0/3679308818 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d4c015070 con 0x7f2d641371f0 2024-04-26T00:23:28.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.306+0000 7f2d617fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:23:28.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.306+0000 7f2d6a5df640 10 monclient: _renew_subs 2024-04-26T00:23:28.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.306+0000 7f2d617fa640 1 -- 172.21.15.37:0/3679308818 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d4c005450 con 0x7f2d641371f0 2024-04-26T00:23:28.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.306+0000 7f2d6a5df640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:28.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.306+0000 7f2d6a5df640 1 -- 172.21.15.37:0/3679308818 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d640a8750 con 0x7f2d641371f0 2024-04-26T00:23:28.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.306+0000 7f2d617fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:28.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.306+0000 7f2d617fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:23:28.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.306+0000 7f2d62ffd640 4 set_mon_vals no callback set 2024-04-26T00:23:28.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.306+0000 7f2d617fa640 10 monclient: dump: 2024-04-26T00:23:28.295 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:28.295 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:28.295 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:28.295 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:28.295 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:28.295 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:28.295 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:28.295 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:28.295 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:28.295 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:28.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.306+0000 7f2d6a5df640 10 monclient: _renew_subs 2024-04-26T00:23:28.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.306+0000 7f2d6a5df640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:28.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.306+0000 7f2d6a5df640 1 -- 172.21.15.37:0/3679308818 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d64120670 con 0x7f2d641371f0 2024-04-26T00:23:28.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.306+0000 7f2d6a5df640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:28.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.306+0000 7f2d6a5df640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:28.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.306+0000 7f2d6a5df640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:28.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.306+0000 7f2d6a5df640 1 -- 172.21.15.37:0/3679308818 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d64075910 con 0x7f2d641371f0 2024-04-26T00:23:28.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.306+0000 7f2d617fa640 1 -- 172.21.15.37:0/3679308818 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2d4c005630 con 0x7f2d641371f0 2024-04-26T00:23:28.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.306+0000 7f2d617fa640 1 --2- 172.21.15.37:0/3679308818 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2d44035e80 0x7f2d44038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:28.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.306+0000 7f2d617fa640 1 -- 172.21.15.37:0/3679308818 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(146..146 src has 1..146) ==== 3931+0+0 (secure 0 0 0) 0x7f2d4c048930 con 0x7f2d641371f0 2024-04-26T00:23:28.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.306+0000 7f2d63fff640 1 --2- 172.21.15.37:0/3679308818 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2d44035e80 0x7f2d44038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:28.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.306+0000 7f2d63fff640 10 monclient: get_auth_request con 0x7f2d44035e80 auth_method 0 2024-04-26T00:23:28.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.310+0000 7f2d63fff640 1 --2- 172.21.15.37:0/3679308818 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2d44035e80 0x7f2d44038340 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2d540ab000 tx=0x7f2d54008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:28.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.314+0000 7f2d617fa640 1 -- 172.21.15.37:0/3679308818 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f2d4c00e340 con 0x7f2d641371f0 2024-04-26T00:23:28.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.314+0000 7f2d617fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:28.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.314+0000 7f2d617fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:23:28.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.538+0000 7f2d6a5df640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:23:28.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.538+0000 7f2d6a5df640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:23:28.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.538+0000 7f2d6a5df640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:28.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.538+0000 7f2d6a5df640 1 -- 172.21.15.37:0/3679308818 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2d64066f40 con 0x7f2d641371f0 2024-04-26T00:23:28.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.538+0000 7f2d617fa640 1 -- 172.21.15.37:0/3679308818 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f2d4c010660 con 0x7f2d641371f0 2024-04-26T00:23:28.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.538+0000 7f2d617fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:23:28.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.538+0000 7f2d617fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:23:28.516 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:23:28.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.542+0000 7f2d6a5df640 1 -- 172.21.15.37:0/3679308818 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2d44035e80 msgr2=0x7f2d44038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:28.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.542+0000 7f2d6a5df640 1 --2- 172.21.15.37:0/3679308818 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2d44035e80 0x7f2d44038340 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2d540ab000 tx=0x7f2d54008040 comp rx=0 tx=0).stop 2024-04-26T00:23:28.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.542+0000 7f2d6a5df640 10 monclient: shutdown 2024-04-26T00:23:28.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.542+0000 7f2d6a5df640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:23:28.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.542+0000 7f2d6a5df640 1 -- 172.21.15.37:0/3679308818 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2d641371f0 msgr2=0x7f2d641363b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:28.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.542+0000 7f2d6a5df640 1 --2- 172.21.15.37:0/3679308818 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2d641371f0 0x7f2d641363b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c00b550 tx=0x7f2d4c00ba10 comp rx=0 tx=0).stop 2024-04-26T00:23:28.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.542+0000 7f2d6a5df640 1 -- 172.21.15.37:0/3679308818 shutdown_connections 2024-04-26T00:23:28.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.542+0000 7f2d6a5df640 1 --2- 172.21.15.37:0/3679308818 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2d44035e80 0x7f2d44038340 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:28.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.542+0000 7f2d6a5df640 1 --2- 172.21.15.37:0/3679308818 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d6413c340 0x7f2d64138d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:28.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.542+0000 7f2d6a5df640 1 --2- 172.21.15.37:0/3679308818 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2d641371f0 0x7f2d641363b0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:28.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.542+0000 7f2d6a5df640 1 --2- 172.21.15.37:0/3679308818 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d64136e10 0x7f2d64135e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:28.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.542+0000 7f2d6a5df640 1 -- 172.21.15.37:0/3679308818 >> 172.21.15.37:0/3679308818 conn(0x7f2d640a64e0 msgr2=0x7f2d6413cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:23:28.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.542+0000 7f2d6a5df640 1 -- 172.21.15.37:0/3679308818 shutdown_connections 2024-04-26T00:23:28.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:28.542+0000 7f2d6a5df640 1 -- 172.21.15.37:0/3679308818 wait complete. 2024-04-26T00:23:31.536 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:23:31.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.690+0000 7f0c2a653640 10 monclient: get_monmap_and_config 2024-04-26T00:23:31.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.690+0000 7f0c2a653640 10 monclient: build_initial_monmap 2024-04-26T00:23:31.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.690+0000 7f0c2a653640 10 monclient: monmap: 2024-04-26T00:23:31.668 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:23:31.668 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:31.668 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:23:31.693538+0000 2024-04-26T00:23:31.668 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:23:31.693538+0000 2024-04-26T00:23:31.668 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:23:31.668 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:31.668 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:23:31.668 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:23:31.669 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:23:31.669 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:23:31.669 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:23:31.669 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:31.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.690+0000 7f0c2a653640 1 Processor -- start 2024-04-26T00:23:31.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.690+0000 7f0c2a653640 1 -- start start 2024-04-26T00:23:31.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.690+0000 7f0c2a653640 10 monclient: init 2024-04-26T00:23:31.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.690+0000 7f0c2a653640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:31.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.690+0000 7f0c2a653640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-04-26T00:23:31.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.690+0000 7f0c2a653640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0c240a0260 addr v1:172.21.15.37:6789/0 2024-04-26T00:23:31.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.690+0000 7f0c2a653640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c24137e40 0x7f0c2413a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:31.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.690+0000 7f0c2a653640 10 monclient(hunting): picked mon.noname-b con 0x7f0c24137e40 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:23:31.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.690+0000 7f0c2a653640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0c2413a770 0x7f0c2413cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:31.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.690+0000 7f0c2a653640 10 monclient(hunting): picked mon.noname-c con 0x7f0c2413a770 addr v2:172.21.15.45:3300/0 2024-04-26T00:23:31.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.690+0000 7f0c2a653640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c240a1130 con 0x7f0c240a0260 2024-04-26T00:23:31.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.690+0000 7f0c2a653640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:31.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.690+0000 7f0c2a653640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f0c240a0d50 con 0x7f0c2413a770 2024-04-26T00:23:31.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.690+0000 7f0c2a653640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:31.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.690+0000 7f0c2a653640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0c2409ec10 con 0x7f0c24137e40 2024-04-26T00:23:31.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.690+0000 7f0c2a653640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:31.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.690+0000 7f0c2a653640 10 monclient(hunting): authenticate will time out at 1192.999301s 2024-04-26T00:23:31.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.690+0000 7f0c23fff640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f0c240a0260 legacy=0x7f0c240a0640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:23:31.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.690+0000 7f0c237fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c24137e40 0x7f0c2413a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:31.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.690+0000 7f0c237fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c24137e40 0x7f0c2413a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:44672/0 (socket says 172.21.15.37:44672) 2024-04-26T00:23:31.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.690+0000 7f0c237fe640 1 -- 172.21.15.37:0/3510528771 learned_addr learned my addr 172.21.15.37:0/3510528771 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:23:31.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.690+0000 7f0c237fe640 10 monclient(hunting): get_auth_request con 0x7f0c24137e40 auth_method 0 2024-04-26T00:23:31.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.690+0000 7f0c237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:31.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.690+0000 7f0c237fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:31.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.690+0000 7f0c237fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:31.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.690+0000 7f0c237fe640 1 -- 172.21.15.37:0/3510528771 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c24137e40 msgr2=0x7f0c2413a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:23:31.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.690+0000 7f0c237fe640 1 -- 172.21.15.37:0/3510528771 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c24137e40 msgr2=0x7f0c2413a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:23:31.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.690+0000 7f0c237fe640 1 --2- 172.21.15.37:0/3510528771 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c24137e40 0x7f0c2413a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:23:31.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.690+0000 7f0c237fe640 1 --2- 172.21.15.37:0/3510528771 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c24137e40 0x7f0c2413a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:23:31.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.690+0000 7f0c28bf0640 1 --2- 172.21.15.37:0/3510528771 >> v2:172.21.15.45:3300/0 conn(0x7f0c2413a770 0x7f0c2413cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:31.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.690+0000 7f0c28bf0640 10 monclient(hunting): get_auth_request con 0x7f0c2413a770 auth_method 0 2024-04-26T00:23:31.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.690+0000 7f0c28bf0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:31.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.690+0000 7f0c28bf0640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:31.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.690+0000 7f0c28bf0640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:31.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.694+0000 7f0c28bf0640 1 -- 172.21.15.37:0/3510528771 >> v2:172.21.15.45:3300/0 conn(0x7f0c2413a770 msgr2=0x7f0c2413cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:23:31.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.694+0000 7f0c28bf0640 1 -- 172.21.15.37:0/3510528771 >> v2:172.21.15.45:3300/0 conn(0x7f0c2413a770 msgr2=0x7f0c2413cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:23:31.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.694+0000 7f0c28bf0640 1 --2- 172.21.15.37:0/3510528771 >> v2:172.21.15.45:3300/0 conn(0x7f0c2413a770 0x7f0c2413cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:23:31.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.694+0000 7f0c28bf0640 1 --2- 172.21.15.37:0/3510528771 >> v2:172.21.15.45:3300/0 conn(0x7f0c2413a770 0x7f0c2413cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:23:31.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.894+0000 7f0c23fff640 1 -- 172.21.15.37:0/3510528771 >> v1:172.21.15.37:6789/0 conn(0x7f0c240a0260 legacy=0x7f0c240a0640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:23:31.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.894+0000 7f0c237fe640 1 --2- 172.21.15.37:0/3510528771 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c24137e40 0x7f0c2413a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:31.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.894+0000 7f0c237fe640 10 monclient(hunting): get_auth_request con 0x7f0c24137e40 auth_method 0 2024-04-26T00:23:31.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.894+0000 7f0c237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:31.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.894+0000 7f0c237fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:31.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.894+0000 7f0c237fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:23:31.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.894+0000 7f0c237fe640 1 -- 172.21.15.37:0/3510528771 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c24137e40 msgr2=0x7f0c2413a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:23:31.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.894+0000 7f0c237fe640 1 -- 172.21.15.37:0/3510528771 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c24137e40 msgr2=0x7f0c2413a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:23:31.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.894+0000 7f0c237fe640 1 --2- 172.21.15.37:0/3510528771 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c24137e40 0x7f0c2413a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:23:31.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.894+0000 7f0c237fe640 1 --2- 172.21.15.37:0/3510528771 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c24137e40 0x7f0c2413a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:23:31.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.898+0000 7f0c28bf0640 1 --2- 172.21.15.37:0/3510528771 >> v2:172.21.15.45:3300/0 conn(0x7f0c2413a770 0x7f0c2413cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:31.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.898+0000 7f0c28bf0640 10 monclient(hunting): get_auth_request con 0x7f0c2413a770 auth_method 0 2024-04-26T00:23:31.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.898+0000 7f0c28bf0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:31.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.898+0000 7f0c28bf0640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:31.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.898+0000 7f0c28bf0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:23:31.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.898+0000 7f0c28bf0640 1 -- 172.21.15.37:0/3510528771 >> v2:172.21.15.45:3300/0 conn(0x7f0c2413a770 msgr2=0x7f0c2413cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:23:31.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.898+0000 7f0c28bf0640 1 -- 172.21.15.37:0/3510528771 >> v2:172.21.15.45:3300/0 conn(0x7f0c2413a770 msgr2=0x7f0c2413cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:23:31.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.898+0000 7f0c28bf0640 1 --2- 172.21.15.37:0/3510528771 >> v2:172.21.15.45:3300/0 conn(0x7f0c2413a770 0x7f0c2413cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:23:31.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:31.898+0000 7f0c28bf0640 1 --2- 172.21.15.37:0/3510528771 >> v2:172.21.15.45:3300/0 conn(0x7f0c2413a770 0x7f0c2413cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:23:32.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:32.298+0000 7f0c23fff640 1 -- 172.21.15.37:0/3510528771 >> v1:172.21.15.37:6789/0 conn(0x7f0c240a0260 legacy=0x7f0c240a0640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:23:32.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:32.298+0000 7f0c237fe640 1 --2- 172.21.15.37:0/3510528771 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c24137e40 0x7f0c2413a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:32.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:32.298+0000 7f0c237fe640 10 monclient(hunting): get_auth_request con 0x7f0c24137e40 auth_method 0 2024-04-26T00:23:32.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:32.298+0000 7f0c237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:32.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:32.298+0000 7f0c237fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:32.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:32.298+0000 7f0c237fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:23:32.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:32.298+0000 7f0c237fe640 1 -- 172.21.15.37:0/3510528771 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c24137e40 msgr2=0x7f0c2413a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:23:32.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:32.298+0000 7f0c237fe640 1 -- 172.21.15.37:0/3510528771 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c24137e40 msgr2=0x7f0c2413a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:23:32.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:32.298+0000 7f0c237fe640 1 --2- 172.21.15.37:0/3510528771 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c24137e40 0x7f0c2413a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:23:32.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:32.298+0000 7f0c237fe640 1 --2- 172.21.15.37:0/3510528771 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c24137e40 0x7f0c2413a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:23:32.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:32.302+0000 7f0c28bf0640 1 --2- 172.21.15.37:0/3510528771 >> v2:172.21.15.45:3300/0 conn(0x7f0c2413a770 0x7f0c2413cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:32.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:32.302+0000 7f0c28bf0640 10 monclient(hunting): get_auth_request con 0x7f0c2413a770 auth_method 0 2024-04-26T00:23:32.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:32.302+0000 7f0c28bf0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:32.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:32.302+0000 7f0c28bf0640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:32.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:32.302+0000 7f0c28bf0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:23:32.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:32.302+0000 7f0c28bf0640 1 -- 172.21.15.37:0/3510528771 >> v2:172.21.15.45:3300/0 conn(0x7f0c2413a770 msgr2=0x7f0c2413cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:23:32.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:32.302+0000 7f0c28bf0640 1 -- 172.21.15.37:0/3510528771 >> v2:172.21.15.45:3300/0 conn(0x7f0c2413a770 msgr2=0x7f0c2413cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:23:32.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:32.302+0000 7f0c28bf0640 1 --2- 172.21.15.37:0/3510528771 >> v2:172.21.15.45:3300/0 conn(0x7f0c2413a770 0x7f0c2413cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:23:32.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:32.302+0000 7f0c28bf0640 1 --2- 172.21.15.37:0/3510528771 >> v2:172.21.15.45:3300/0 conn(0x7f0c2413a770 0x7f0c2413cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:23:33.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:33.102+0000 7f0c23fff640 1 -- 172.21.15.37:0/3510528771 >> v1:172.21.15.37:6789/0 conn(0x7f0c240a0260 legacy=0x7f0c240a0640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:23:33.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:33.102+0000 7f0c237fe640 1 --2- 172.21.15.37:0/3510528771 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c24137e40 0x7f0c2413a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:33.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:33.102+0000 7f0c237fe640 10 monclient(hunting): get_auth_request con 0x7f0c24137e40 auth_method 0 2024-04-26T00:23:33.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:33.102+0000 7f0c237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:33.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:33.102+0000 7f0c237fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:33.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:33.102+0000 7f0c237fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:23:33.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:33.102+0000 7f0c237fe640 1 -- 172.21.15.37:0/3510528771 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c24137e40 msgr2=0x7f0c2413a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:23:33.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:33.102+0000 7f0c237fe640 1 -- 172.21.15.37:0/3510528771 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c24137e40 msgr2=0x7f0c2413a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:23:33.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:33.102+0000 7f0c237fe640 1 --2- 172.21.15.37:0/3510528771 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c24137e40 0x7f0c2413a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:23:33.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:33.102+0000 7f0c237fe640 1 --2- 172.21.15.37:0/3510528771 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c24137e40 0x7f0c2413a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:23:33.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:33.106+0000 7f0c28bf0640 1 --2- 172.21.15.37:0/3510528771 >> v2:172.21.15.45:3300/0 conn(0x7f0c2413a770 0x7f0c2413cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:33.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:33.106+0000 7f0c28bf0640 10 monclient(hunting): get_auth_request con 0x7f0c2413a770 auth_method 0 2024-04-26T00:23:33.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:33.106+0000 7f0c28bf0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:33.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:33.106+0000 7f0c28bf0640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:33.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:33.106+0000 7f0c28bf0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:23:33.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:33.106+0000 7f0c28bf0640 1 -- 172.21.15.37:0/3510528771 >> v2:172.21.15.45:3300/0 conn(0x7f0c2413a770 msgr2=0x7f0c2413cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:23:33.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:33.106+0000 7f0c28bf0640 1 -- 172.21.15.37:0/3510528771 >> v2:172.21.15.45:3300/0 conn(0x7f0c2413a770 msgr2=0x7f0c2413cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:23:33.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:33.106+0000 7f0c28bf0640 1 --2- 172.21.15.37:0/3510528771 >> v2:172.21.15.45:3300/0 conn(0x7f0c2413a770 0x7f0c2413cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:23:33.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:33.106+0000 7f0c28bf0640 1 --2- 172.21.15.37:0/3510528771 >> v2:172.21.15.45:3300/0 conn(0x7f0c2413a770 0x7f0c2413cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:23:34.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:34.690+0000 7f0c21ffb640 10 monclient(hunting): tick 2024-04-26T00:23:34.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:34.690+0000 7f0c21ffb640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:23:34.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:34.690+0000 7f0c21ffb640 1 monclient(hunting): continuing hunt 2024-04-26T00:23:34.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:34.690+0000 7f0c21ffb640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:23:34.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:34.690+0000 7f0c21ffb640 1 -- 172.21.15.37:0/3510528771 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c24137e40 msgr2=0x7f0c2413a230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:23:34.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:34.690+0000 7f0c21ffb640 1 --2- 172.21.15.37:0/3510528771 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c24137e40 0x7f0c2413a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:34.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:34.690+0000 7f0c21ffb640 1 -- 172.21.15.37:0/3510528771 >> v2:172.21.15.45:3300/0 conn(0x7f0c2413a770 msgr2=0x7f0c2413cb60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:23:34.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:34.690+0000 7f0c21ffb640 1 --2- 172.21.15.37:0/3510528771 >> v2:172.21.15.45:3300/0 conn(0x7f0c2413a770 0x7f0c2413cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:34.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:34.690+0000 7f0c21ffb640 1 -- 172.21.15.37:0/3510528771 >> v1:172.21.15.37:6789/0 conn(0x7f0c240a0260 legacy=0x7f0c240a0640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:23:34.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:34.690+0000 7f0c21ffb640 10 monclient: _add_conns ranks=[3,1] 2024-04-26T00:23:34.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:34.690+0000 7f0c21ffb640 1 --2- 172.21.15.37:0/3510528771 >> v2:172.21.15.37:3300/0 conn(0x7f0c040036b0 0x7f0c04005b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:34.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:34.690+0000 7f0c21ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f0c040036b0 addr v2:172.21.15.37:3300/0 2024-04-26T00:23:34.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:34.690+0000 7f0c21ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0c04006220 addr v1:172.21.15.45:6789/0 2024-04-26T00:23:34.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:34.690+0000 7f0c21ffb640 1 -- 172.21.15.37:0/3510528771 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c2406cff0 con 0x7f0c04006220 2024-04-26T00:23:34.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:34.690+0000 7f0c21ffb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:34.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:34.690+0000 7f0c21ffb640 1 -- 172.21.15.37:0/3510528771 --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f0c240a0d50 con 0x7f0c040036b0 2024-04-26T00:23:34.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:34.690+0000 7f0c21ffb640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:34.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:34.690+0000 7f0c23fff640 1 -- 172.21.15.37:0/3510528771 >> v2:172.21.15.37:3300/0 conn(0x7f0c040036b0 msgr2=0x7f0c04005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:23:34.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:34.690+0000 7f0c23fff640 1 --2- 172.21.15.37:0/3510528771 >> v2:172.21.15.37:3300/0 conn(0x7f0c040036b0 0x7f0c04005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:23:34.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:34.694+0000 7f0c237fe640 1 -- 172.21.15.37:0/3510528771 >> v1:172.21.15.45:6789/0 conn(0x7f0c04006220 legacy=0x7f0c040086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:34.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:34.894+0000 7f0c23fff640 1 -- 172.21.15.37:0/3510528771 >> v2:172.21.15.37:3300/0 conn(0x7f0c040036b0 msgr2=0x7f0c04005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:23:34.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:34.894+0000 7f0c23fff640 1 --2- 172.21.15.37:0/3510528771 >> v2:172.21.15.37:3300/0 conn(0x7f0c040036b0 0x7f0c04005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:23:35.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.298+0000 7f0c23fff640 1 -- 172.21.15.37:0/3510528771 >> v2:172.21.15.37:3300/0 conn(0x7f0c040036b0 msgr2=0x7f0c04005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:23:35.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.298+0000 7f0c23fff640 1 --2- 172.21.15.37:0/3510528771 >> v2:172.21.15.37:3300/0 conn(0x7f0c040036b0 0x7f0c04005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:23:35.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.970+0000 7f0c22ffd640 1 -- 172.21.15.37:0/3510528771 <== mon.1 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2630083271 0 0) 0x7f0c2406cff0 con 0x7f0c04006220 2024-04-26T00:23:35.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.970+0000 7f0c22ffd640 10 monclient(hunting): _negotiate 2024-04-26T00:23:35.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.970+0000 7f0c22ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:35.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.970+0000 7f0c22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:35.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.970+0000 7f0c22ffd640 10 monclient(hunting): my global_id is 24310 2024-04-26T00:23:35.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c22ffd640 1 -- 172.21.15.37:0/3510528771 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c10003530 con 0x7f0c04006220 2024-04-26T00:23:35.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c22ffd640 1 -- 172.21.15.37:0/3510528771 <== mon.1 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2460104971 0 0) 0x7f0c10003530 con 0x7f0c04006220 2024-04-26T00:23:35.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c22ffd640 1 -- 172.21.15.37:0/3510528771 >> v2:172.21.15.37:3300/0 conn(0x7f0c040036b0 msgr2=0x7f0c04005b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:23:35.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c22ffd640 1 --2- 172.21.15.37:0/3510528771 >> v2:172.21.15.37:3300/0 conn(0x7f0c040036b0 0x7f0c04005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:35.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c22ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:23:35.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c22ffd640 1 monclient: found mon.noname-c-legacy 2024-04-26T00:23:35.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c22ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:35.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c22ffd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.45:6789/0 2024-04-26T00:23:35.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c22ffd640 1 -- 172.21.15.37:0/3510528771 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c0400af70 con 0x7f0c04006220 2024-04-26T00:23:35.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c22ffd640 10 monclient: _finish_auth 0 2024-04-26T00:23:35.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c22ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:23:35.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:35.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c22ffd640 1 -- 172.21.15.37:0/3510528771 <== mon.1 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 866326386 0 0) 0x7f0c0c00f040 con 0x7f0c04006220 2024-04-26T00:23:35.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:35.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c22ffd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-04-26T00:23:35.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c22ffd640 10 monclient: dump: 2024-04-26T00:23:35.955 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:35.955 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:35.955 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:35.955 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:35.955 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:35.955 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:35.955 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:35.955 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:35.955 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:35.955 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:35.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c22ffd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] but i'm connected to v1:172.21.15.45:6789/0, reconnecting 2024-04-26T00:23:35.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c22ffd640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:35.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c22ffd640 1 -- 172.21.15.37:0/3510528771 >> v1:172.21.15.45:6789/0 conn(0x7f0c04006220 legacy=0x7f0c040086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:35.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c22ffd640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:23:35.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c28bf0640 1 -- 172.21.15.37:0/3510528771 reap_dead start 2024-04-26T00:23:35.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c22ffd640 1 --2- 172.21.15.37:0/3510528771 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c100039c0 0x7f0c10003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:35.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c22ffd640 10 monclient(hunting): picked mon.b con 0x7f0c100039c0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:23:35.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c22ffd640 1 --2- 172.21.15.37:0/3510528771 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c10004810 0x7f0c10004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:35.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c22ffd640 10 monclient(hunting): picked mon.c con 0x7f0c10004810 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:23:35.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c22ffd640 1 --2- 172.21.15.37:0/3510528771 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0c100052f0 0x7f0c1000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:35.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c22ffd640 10 monclient(hunting): picked mon.a con 0x7f0c100052f0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:23:35.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c22ffd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:35.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c22ffd640 1 -- 172.21.15.37:0/3510528771 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f0c240a0d50 con 0x7f0c100052f0 2024-04-26T00:23:35.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c22ffd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:35.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c22ffd640 1 -- 172.21.15.37:0/3510528771 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0c1000c240 con 0x7f0c100039c0 2024-04-26T00:23:35.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c22ffd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:35.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c22ffd640 1 -- 172.21.15.37:0/3510528771 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0c1000c440 con 0x7f0c10004810 2024-04-26T00:23:35.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c22ffd640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:35.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c22ffd640 10 monclient(hunting): _finish_auth 0 2024-04-26T00:23:35.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c28bf0640 1 -- 172.21.15.37:0/3510528771 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0c100052f0 msgr2=0x7f0c1000b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:23:35.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c28bf0640 1 --2- 172.21.15.37:0/3510528771 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0c100052f0 0x7f0c1000b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:23:35.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c237fe640 1 --2- 172.21.15.37:0/3510528771 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c10004810 0x7f0c10004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:35.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c23fff640 1 --2- 172.21.15.37:0/3510528771 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c100039c0 0x7f0c10003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:35.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c237fe640 10 monclient(hunting): get_auth_request con 0x7f0c10004810 auth_method 0 2024-04-26T00:23:35.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:35.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c237fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:35.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c237fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:23:35.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c23fff640 10 monclient(hunting): get_auth_request con 0x7f0c100039c0 auth_method 0 2024-04-26T00:23:35.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:35.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c23fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:35.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c23fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:23:35.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:35.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:35.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c237fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:23:35.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:35.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:35.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c23fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:23:35.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c237fe640 10 monclient(hunting): handle_auth_done global_id 24310 payload 293 2024-04-26T00:23:35.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c237fe640 1 -- 172.21.15.37:0/3510528771 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c100039c0 msgr2=0x7f0c10003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:23:35.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c237fe640 1 --2- 172.21.15.37:0/3510528771 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c100039c0 0x7f0c10003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:35.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c237fe640 1 -- 172.21.15.37:0/3510528771 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0c100052f0 msgr2=0x7f0c1000b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:23:35.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c237fe640 1 --2- 172.21.15.37:0/3510528771 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0c100052f0 0x7f0c1000b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:35.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c237fe640 10 monclient: _finish_hunting 0 2024-04-26T00:23:35.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c237fe640 1 monclient: found mon.c 2024-04-26T00:23:35.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:35.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c237fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:35.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c237fe640 1 -- 172.21.15.37:0/3510528771 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c10003530 con 0x7f0c10004810 2024-04-26T00:23:35.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c237fe640 10 monclient: _finish_auth 0 2024-04-26T00:23:35.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c237fe640 10 monclient: _check_auth_tickets 2024-04-26T00:23:35.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:35.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c2a653640 5 monclient: authenticate success, global_id 24310 2024-04-26T00:23:35.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.974+0000 7f0c2a653640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:23:35.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.986+0000 7f0c237fe640 1 -- 172.21.15.37:0/3510528771 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c10004810 msgr2=0x7f0c10004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:35.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.986+0000 7f0c237fe640 1 --2- 172.21.15.37:0/3510528771 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c10004810 0x7f0c10004c90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c006f30 tx=0x7f0c0c020090 comp rx=0 tx=0).ready entity=mon.2 client_cookie=527c5434f708d187 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:35.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.986+0000 7f0c22ffd640 1 -- 172.21.15.37:0/3510528771 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c0af070 con 0x7f0c10004810 2024-04-26T00:23:35.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.986+0000 7f0c22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:35.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.986+0000 7f0c22ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:23:35.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.986+0000 7f0c22ffd640 10 monclient: dump: 2024-04-26T00:23:35.964 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:35.964 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:35.964 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:35.964 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:35.964 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:35.964 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:35.964 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:35.964 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:35.964 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:35.964 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:35.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.986+0000 7f0c2a653640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:23:35.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.986+0000 7f0c22ffd640 1 -- 172.21.15.37:0/3510528771 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c0c0a9d20 con 0x7f0c10004810 2024-04-26T00:23:35.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.986+0000 7f0c22ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:23:35.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.986+0000 7f0c22ffd640 1 -- 172.21.15.37:0/3510528771 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c00cdd0 con 0x7f0c10004810 2024-04-26T00:23:35.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.986+0000 7f0c22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:35.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.986+0000 7f0c22ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:23:35.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.986+0000 7f0c22ffd640 10 monclient: dump: 2024-04-26T00:23:35.965 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:35.965 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:35.965 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:35.966 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:35.966 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:35.966 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:35.966 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:35.966 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:35.966 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:35.966 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:35.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.986+0000 7f0c2a653640 10 monclient: get_monmap_and_config success 2024-04-26T00:23:35.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.986+0000 7f0c2a653640 4 set_mon_vals no callback set 2024-04-26T00:23:35.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.990+0000 7f0c2a653640 10 monclient: shutdown 2024-04-26T00:23:35.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.990+0000 7f0c2a653640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:23:35.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.990+0000 7f0c2a653640 1 -- 172.21.15.37:0/3510528771 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c10004810 msgr2=0x7f0c10004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:35.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.990+0000 7f0c2a653640 1 --2- 172.21.15.37:0/3510528771 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c10004810 0x7f0c10004c90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c006f30 tx=0x7f0c0c020090 comp rx=0 tx=0).stop 2024-04-26T00:23:35.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.990+0000 7f0c2a653640 1 -- 172.21.15.37:0/3510528771 shutdown_connections 2024-04-26T00:23:35.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.990+0000 7f0c2a653640 1 --2- 172.21.15.37:0/3510528771 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0c100052f0 0x7f0c1000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:35.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.990+0000 7f0c2a653640 1 --2- 172.21.15.37:0/3510528771 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c10004810 0x7f0c10004c90 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:35.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.990+0000 7f0c2a653640 1 --2- 172.21.15.37:0/3510528771 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c100039c0 0x7f0c10003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:35.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.990+0000 7f0c2a653640 1 -- 172.21.15.37:0/3510528771 >> 172.21.15.37:0/3510528771 conn(0x7f0c240a1ad0 msgr2=0x7f0c240a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:23:35.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.990+0000 7f0c2a653640 1 -- 172.21.15.37:0/3510528771 shutdown_connections 2024-04-26T00:23:35.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.990+0000 7f0c2a653640 1 -- 172.21.15.37:0/3510528771 wait complete. 2024-04-26T00:23:35.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.990+0000 7f0c2a653640 10 monclient: build_initial_monmap 2024-04-26T00:23:35.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.990+0000 7f0c2a653640 10 monclient: monmap: 2024-04-26T00:23:35.968 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:23:35.968 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:23:35.968 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:23:35.969 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:23:35.969 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:23:35.969 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:23:35.969 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:23:35.969 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:23:35.969 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:23:35.969 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:35.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.990+0000 7f0c2a653640 1 Processor -- start 2024-04-26T00:23:35.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.990+0000 7f0c2a653640 1 -- start start 2024-04-26T00:23:35.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.990+0000 7f0c2a653640 10 monclient: init 2024-04-26T00:23:35.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c2a653640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:35.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c2a653640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:23:35.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c2a653640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0c100039c0 0x7f0c2412f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:35.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c2a653640 10 monclient(hunting): picked mon.noname-a con 0x7f0c100039c0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:23:35.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c2a653640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c10004810 0x7f0c24133330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:35.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c2a653640 10 monclient(hunting): picked mon.noname-b con 0x7f0c10004810 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:23:35.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c2a653640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c100052f0 0x7f0c24133870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:35.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c2a653640 10 monclient(hunting): picked mon.noname-c con 0x7f0c100052f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:23:35.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c2a653640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:35.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c2a653640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f0c2409f2b0 con 0x7f0c100039c0 2024-04-26T00:23:35.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c2a653640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:35.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c2a653640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0c2409f0f0 con 0x7f0c10004810 2024-04-26T00:23:35.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c2a653640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:35.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c2a653640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0c2409f430 con 0x7f0c100052f0 2024-04-26T00:23:35.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c2a653640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:35.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c23fff640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0c100039c0 msgr2=0x7f0c2412f240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:23:35.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c2a653640 10 monclient(hunting): authenticate will time out at 1197.300818s 2024-04-26T00:23:35.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c23fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0c100039c0 0x7f0c2412f240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:23:35.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c28bf0640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c100052f0 0x7f0c24133870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:35.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c28bf0640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c100052f0 0x7f0c24133870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:44694/0 (socket says 172.21.15.37:44694) 2024-04-26T00:23:35.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c28bf0640 1 -- 172.21.15.37:0/1711870384 learned_addr learned my addr 172.21.15.37:0/1711870384 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:23:35.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c28bf0640 10 monclient(hunting): get_auth_request con 0x7f0c100052f0 auth_method 0 2024-04-26T00:23:35.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c28bf0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:35.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c28bf0640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:35.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c28bf0640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:35.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c237fe640 1 --2- 172.21.15.37:0/1711870384 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c10004810 0x7f0c24133330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:35.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c237fe640 10 monclient(hunting): get_auth_request con 0x7f0c10004810 auth_method 0 2024-04-26T00:23:35.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:35.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c237fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:35.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c237fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:35.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c28bf0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:35.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c28bf0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:35.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c28bf0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:35.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:35.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:35.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:35.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c28bf0640 10 monclient(hunting): handle_auth_done global_id 34262 payload 824 2024-04-26T00:23:35.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c28bf0640 1 -- 172.21.15.37:0/1711870384 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c10004810 msgr2=0x7f0c24133330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:35.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c28bf0640 1 --2- 172.21.15.37:0/1711870384 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c10004810 0x7f0c24133330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:35.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c28bf0640 1 -- 172.21.15.37:0/1711870384 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0c100039c0 msgr2=0x7f0c2412f240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:23:35.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c28bf0640 1 --2- 172.21.15.37:0/1711870384 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0c100039c0 0x7f0c2412f240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:35.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c28bf0640 10 monclient: _finish_hunting 0 2024-04-26T00:23:35.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c28bf0640 1 monclient: found mon.noname-c 2024-04-26T00:23:35.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c28bf0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:35.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c28bf0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:23:35.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.994+0000 7f0c28bf0640 1 -- 172.21.15.37:0/1711870384 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c2406f7a0 con 0x7f0c100052f0 2024-04-26T00:23:35.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.998+0000 7f0c28bf0640 1 -- 172.21.15.37:0/1711870384 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c100052f0 msgr2=0x7f0c24133870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:35.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.998+0000 7f0c28bf0640 1 --2- 172.21.15.37:0/1711870384 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c100052f0 0x7f0c24133870 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0c04006190 tx=0x7f0c14014d20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:35.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.998+0000 7f0c20ff9640 1 -- 172.21.15.37:0/1711870384 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c1401b030 con 0x7f0c100052f0 2024-04-26T00:23:35.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.998+0000 7f0c20ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:35.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.998+0000 7f0c20ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:23:35.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.998+0000 7f0c20ff9640 10 monclient: dump: 2024-04-26T00:23:35.978 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:35.978 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:35.978 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:35.978 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:35.978 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:35.978 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:35.978 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:35.978 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:35.978 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:35.978 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:35.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.998+0000 7f0c20ff9640 10 monclient: _finish_auth 0 2024-04-26T00:23:35.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.998+0000 7f0c20ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:23:35.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.998+0000 7f0c20ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:35.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.998+0000 7f0c20ff9640 1 -- 172.21.15.37:0/1711870384 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c14012e20 con 0x7f0c100052f0 2024-04-26T00:23:35.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.998+0000 7f0c2a653640 5 monclient: authenticate success, global_id 34262 2024-04-26T00:23:35.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.998+0000 7f0c2a653640 10 monclient: _renew_subs 2024-04-26T00:23:35.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.998+0000 7f0c2a653640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:35.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.998+0000 7f0c2a653640 1 -- 172.21.15.37:0/1711870384 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c240729d0 con 0x7f0c100052f0 2024-04-26T00:23:35.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.998+0000 7f0c20ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:23:35.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.998+0000 7f0c20ff9640 1 -- 172.21.15.37:0/1711870384 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c1400f040 con 0x7f0c100052f0 2024-04-26T00:23:35.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.998+0000 7f0c20ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:35.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.998+0000 7f0c20ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:23:35.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.998+0000 7f0c20ff9640 10 monclient: dump: 2024-04-26T00:23:35.979 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:35.979 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:35.979 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:35.979 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:35.979 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:35.979 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:35.980 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:35.980 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:35.980 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:35.980 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:35.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.998+0000 7f0c227fc640 4 set_mon_vals no callback set 2024-04-26T00:23:35.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.998+0000 7f0c2a653640 10 monclient: _renew_subs 2024-04-26T00:23:35.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.998+0000 7f0c2a653640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:35.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.998+0000 7f0c2a653640 1 -- 172.21.15.37:0/1711870384 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c2406f4c0 con 0x7f0c100052f0 2024-04-26T00:23:35.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.998+0000 7f0c2a653640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:35.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.998+0000 7f0c2a653640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:35.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.998+0000 7f0c2a653640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:35.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.998+0000 7f0c20ff9640 1 -- 172.21.15.37:0/1711870384 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f0c140042a0 con 0x7f0c100052f0 2024-04-26T00:23:35.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.998+0000 7f0c2a653640 1 -- 172.21.15.37:0/1711870384 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c24066f40 con 0x7f0c100052f0 2024-04-26T00:23:35.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.998+0000 7f0c20ff9640 1 --2- 172.21.15.37:0/1711870384 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0c04036750 0x7f0c04038c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:35.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.998+0000 7f0c20ff9640 1 -- 172.21.15.37:0/1711870384 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(147..147 src has 1..147) ==== 3931+0+0 (secure 0 0 0) 0x7f0c14048cc0 con 0x7f0c100052f0 2024-04-26T00:23:35.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.998+0000 7f0c23fff640 1 --2- 172.21.15.37:0/1711870384 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0c04036750 0x7f0c04038c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:35.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:35.998+0000 7f0c23fff640 10 monclient: get_auth_request con 0x7f0c04036750 auth_method 0 2024-04-26T00:23:35.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.002+0000 7f0c23fff640 1 --2- 172.21.15.37:0/1711870384 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0c04036750 0x7f0c04038c10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0c24134c70 tx=0x7f0c1c003d40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:35.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.002+0000 7f0c20ff9640 1 -- 172.21.15.37:0/1711870384 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f0c14047a10 con 0x7f0c100052f0 2024-04-26T00:23:35.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.002+0000 7f0c20ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:35.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.002+0000 7f0c20ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:23:36.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.234+0000 7f0c2a653640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:23:36.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.234+0000 7f0c2a653640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:23:36.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.234+0000 7f0c2a653640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:36.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.234+0000 7f0c2a653640 1 -- 172.21.15.37:0/1711870384 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0c241358b0 con 0x7f0c100052f0 2024-04-26T00:23:36.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.234+0000 7f0c20ff9640 1 -- 172.21.15.37:0/1711870384 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7f0c14016590 con 0x7f0c100052f0 2024-04-26T00:23:36.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.234+0000 7f0c20ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:23:36.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.234+0000 7f0c20ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:23:36.212 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":32,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:23:36.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.234+0000 7f0c2a653640 1 -- 172.21.15.37:0/1711870384 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0c04036750 msgr2=0x7f0c04038c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:36.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.234+0000 7f0c2a653640 1 --2- 172.21.15.37:0/1711870384 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0c04036750 0x7f0c04038c10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0c24134c70 tx=0x7f0c1c003d40 comp rx=0 tx=0).stop 2024-04-26T00:23:36.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.234+0000 7f0c2a653640 10 monclient: shutdown 2024-04-26T00:23:36.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.238+0000 7f0c2a653640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:23:36.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.238+0000 7f0c2a653640 1 -- 172.21.15.37:0/1711870384 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c100052f0 msgr2=0x7f0c24133870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:36.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.238+0000 7f0c2a653640 1 --2- 172.21.15.37:0/1711870384 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c100052f0 0x7f0c24133870 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0c04006190 tx=0x7f0c14014d20 comp rx=0 tx=0).stop 2024-04-26T00:23:36.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.238+0000 7f0c2a653640 1 -- 172.21.15.37:0/1711870384 shutdown_connections 2024-04-26T00:23:36.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.238+0000 7f0c2a653640 1 --2- 172.21.15.37:0/1711870384 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0c04036750 0x7f0c04038c10 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:36.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.238+0000 7f0c2a653640 1 --2- 172.21.15.37:0/1711870384 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c100052f0 0x7f0c24133870 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:36.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.238+0000 7f0c2a653640 1 --2- 172.21.15.37:0/1711870384 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c10004810 0x7f0c24133330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:36.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.238+0000 7f0c2a653640 1 --2- 172.21.15.37:0/1711870384 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0c100039c0 0x7f0c2412f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:36.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.238+0000 7f0c2a653640 1 -- 172.21.15.37:0/1711870384 >> 172.21.15.37:0/1711870384 conn(0x7f0c240a1ad0 msgr2=0x7f0c240a45f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:23:36.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.238+0000 7f0c2a653640 1 -- 172.21.15.37:0/1711870384 shutdown_connections 2024-04-26T00:23:36.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.238+0000 7f0c2a653640 1 -- 172.21.15.37:0/1711870384 wait complete. 2024-04-26T00:23:36.231 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-26T00:23:36.231 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-04-26T00:23:36.231 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:23:36.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.374+0000 7fcbcf49c640 10 monclient: get_monmap_and_config 2024-04-26T00:23:36.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.374+0000 7fcbcf49c640 10 monclient: build_initial_monmap 2024-04-26T00:23:36.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.374+0000 7fcbcf49c640 10 monclient: monmap: 2024-04-26T00:23:36.351 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:23:36.351 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:36.351 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:23:36.376406+0000 2024-04-26T00:23:36.351 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:23:36.376406+0000 2024-04-26T00:23:36.351 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:23:36.351 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:36.351 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:23:36.351 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:23:36.351 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:23:36.351 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:23:36.351 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:23:36.352 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:36.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.374+0000 7fcbcf49c640 1 Processor -- start 2024-04-26T00:23:36.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.374+0000 7fcbcf49c640 1 -- start start 2024-04-26T00:23:36.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.374+0000 7fcbcf49c640 10 monclient: init 2024-04-26T00:23:36.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.374+0000 7fcbcf49c640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:36.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.374+0000 7fcbcf49c640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-04-26T00:23:36.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.374+0000 7fcbcf49c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fcbc8133f70 0x7fcbc809c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:36.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.374+0000 7fcbcf49c640 10 monclient(hunting): picked mon.noname-a con 0x7fcbc8133f70 addr v2:172.21.15.37:3300/0 2024-04-26T00:23:36.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.374+0000 7fcbcf49c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcbc8134350 addr v1:172.21.15.45:6789/0 2024-04-26T00:23:36.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.374+0000 7fcbcf49c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcbc8134790 addr v1:172.21.15.37:6789/0 2024-04-26T00:23:36.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.374+0000 7fcbcf49c640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbc80a4fa0 con 0x7fcbc8134790 2024-04-26T00:23:36.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.374+0000 7fcbcf49c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbc8137bc0 con 0x7fcbc8134350 2024-04-26T00:23:36.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.374+0000 7fcbcf49c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:36.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.374+0000 7fcbcf49c640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fcbc80a51e0 con 0x7fcbc8133f70 2024-04-26T00:23:36.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.374+0000 7fcbcf49c640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:36.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.374+0000 7fcbcf49c640 10 monclient(hunting): authenticate will time out at 1197.682071s 2024-04-26T00:23:36.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.374+0000 7fcbcda39640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7fcbc8134790 legacy=0x7fcbc81374b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:23:36.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.374+0000 7fcbcd238640 1 -- >> v2:172.21.15.37:3300/0 conn(0x7fcbc8133f70 msgr2=0x7fcbc809c140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:23:36.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.374+0000 7fcbcd238640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fcbc8133f70 0x7fcbc809c140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:23:36.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.374+0000 7fcbcca37640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fcbc8134350 0x7fcbc8136da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.37:44270/0 (socket says 172.21.15.37:44270) 2024-04-26T00:23:36.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.374+0000 7fcbcca37640 1 -- 172.21.15.37:0/543965336 learned_addr learned my addr 172.21.15.37:0/543965336 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:23:36.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbcca37640 1 -- 172.21.15.37:0/543965336 >> v1:172.21.15.45:6789/0 conn(0x7fcbc8134350 legacy=0x7fcbc8136da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:36.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbc7fff640 1 -- 172.21.15.37:0/543965336 <== mon.1 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2188339803 0 0) 0x7fcbc8137bc0 con 0x7fcbc8134350 2024-04-26T00:23:36.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbc7fff640 10 monclient(hunting): _negotiate 2024-04-26T00:23:36.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbc7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:36.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbc7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:36.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbc7fff640 10 monclient(hunting): my global_id is 24316 2024-04-26T00:23:36.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbc7fff640 1 -- 172.21.15.37:0/543965336 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbb0003530 con 0x7fcbc8134350 2024-04-26T00:23:36.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbc7fff640 1 -- 172.21.15.37:0/543965336 <== mon.1 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3502827135 0 0) 0x7fcbb0003530 con 0x7fcbc8134350 2024-04-26T00:23:36.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbc7fff640 1 -- 172.21.15.37:0/543965336 >> v2:172.21.15.37:3300/0 conn(0x7fcbc8133f70 msgr2=0x7fcbc809c140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:23:36.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbc7fff640 1 --2- 172.21.15.37:0/543965336 >> v2:172.21.15.37:3300/0 conn(0x7fcbc8133f70 0x7fcbc809c140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:36.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbc7fff640 1 -- 172.21.15.37:0/543965336 >> v1:172.21.15.37:6789/0 conn(0x7fcbc8134790 legacy=0x7fcbc81374b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:23:36.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbc7fff640 10 monclient: _finish_hunting 0 2024-04-26T00:23:36.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbc7fff640 1 monclient: found mon.noname-c-legacy 2024-04-26T00:23:36.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbc7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:36.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbc7fff640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.45:6789/0 2024-04-26T00:23:36.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbc7fff640 1 -- 172.21.15.37:0/543965336 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbc809b280 con 0x7fcbc8134350 2024-04-26T00:23:36.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbc7fff640 10 monclient: _finish_auth 0 2024-04-26T00:23:36.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbc7fff640 10 monclient: _check_auth_tickets 2024-04-26T00:23:36.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbc7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:36.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbc7fff640 1 -- 172.21.15.37:0/543965336 <== mon.1 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 866326386 0 0) 0x7fcbbc001930 con 0x7fcbc8134350 2024-04-26T00:23:36.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbc7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:36.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbc7fff640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-04-26T00:23:36.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbc7fff640 10 monclient: dump: 2024-04-26T00:23:36.359 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:36.359 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:36.359 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:36.360 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:36.360 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:36.360 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:36.360 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:36.360 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:36.360 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:36.360 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:36.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbc7fff640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] but i'm connected to v1:172.21.15.45:6789/0, reconnecting 2024-04-26T00:23:36.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbc7fff640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:36.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbc7fff640 1 -- 172.21.15.37:0/543965336 >> v1:172.21.15.45:6789/0 conn(0x7fcbc8134350 legacy=0x7fcbc8136da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:36.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbc7fff640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:23:36.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbc7fff640 1 --2- 172.21.15.37:0/543965336 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcbb00039c0 0x7fcbb0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:36.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbc7fff640 10 monclient(hunting): picked mon.b con 0x7fcbb00039c0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:23:36.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbc7fff640 1 --2- 172.21.15.37:0/543965336 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcbb0004810 0x7fcbb0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:36.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbc7fff640 10 monclient(hunting): picked mon.a con 0x7fcbb0004810 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:23:36.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbc7fff640 1 --2- 172.21.15.37:0/543965336 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbb0005380 0x7fcbb000b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:36.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbc7fff640 10 monclient(hunting): picked mon.c con 0x7fcbb0005380 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:23:36.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbc7fff640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:36.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbc7fff640 1 -- 172.21.15.37:0/543965336 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc80a51e0 con 0x7fcbb0004810 2024-04-26T00:23:36.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbc7fff640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:36.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbc7fff640 1 -- 172.21.15.37:0/543965336 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb000c2d0 con 0x7fcbb00039c0 2024-04-26T00:23:36.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbc7fff640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:36.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbc7fff640 1 -- 172.21.15.37:0/543965336 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fcbb000c4d0 con 0x7fcbb0005380 2024-04-26T00:23:36.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbc7fff640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:36.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbc7fff640 10 monclient(hunting): _finish_auth 0 2024-04-26T00:23:36.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbcca37640 1 -- 172.21.15.37:0/543965336 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcbb0004810 msgr2=0x7fcbb0004c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:23:36.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbcca37640 1 --2- 172.21.15.37:0/543965336 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcbb0004810 0x7fcbb0004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:23:36.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbcda39640 1 --2- 172.21.15.37:0/543965336 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbb0005380 0x7fcbb000b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:36.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbcd238640 1 --2- 172.21.15.37:0/543965336 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcbb00039c0 0x7fcbb0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:36.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbcda39640 10 monclient(hunting): get_auth_request con 0x7fcbb0005380 auth_method 0 2024-04-26T00:23:36.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbcda39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:36.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbcda39640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:36.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbcda39640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:23:36.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbcd238640 10 monclient(hunting): get_auth_request con 0x7fcbb00039c0 auth_method 0 2024-04-26T00:23:36.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbcd238640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:36.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbcd238640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:36.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbcd238640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:23:36.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbcda39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:36.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbcda39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:36.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbcda39640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:23:36.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbcd238640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:36.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbcd238640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:36.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbcd238640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:23:36.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbcda39640 10 monclient(hunting): handle_auth_done global_id 24316 payload 293 2024-04-26T00:23:36.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbcda39640 1 -- 172.21.15.37:0/543965336 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcbb00039c0 msgr2=0x7fcbb0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:23:36.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbcda39640 1 --2- 172.21.15.37:0/543965336 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcbb00039c0 0x7fcbb0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:36.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbcda39640 1 -- 172.21.15.37:0/543965336 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcbb0004810 msgr2=0x7fcbb0004c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:23:36.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbcda39640 1 --2- 172.21.15.37:0/543965336 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcbb0004810 0x7fcbb0004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:36.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbcda39640 10 monclient: _finish_hunting 0 2024-04-26T00:23:36.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbcda39640 1 monclient: found mon.c 2024-04-26T00:23:36.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbcda39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:36.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbcda39640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:36.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbcda39640 1 -- 172.21.15.37:0/543965336 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbb0003530 con 0x7fcbb0005380 2024-04-26T00:23:36.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbcda39640 10 monclient: _finish_auth 0 2024-04-26T00:23:36.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbcda39640 10 monclient: _check_auth_tickets 2024-04-26T00:23:36.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbcda39640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:36.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbcf49c640 5 monclient: authenticate success, global_id 24316 2024-04-26T00:23:36.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.378+0000 7fcbcf49c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:23:36.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.382+0000 7fcbcda39640 1 -- 172.21.15.37:0/543965336 reap_dead start 2024-04-26T00:23:36.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.382+0000 7fcbcda39640 1 -- 172.21.15.37:0/543965336 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbb0005380 msgr2=0x7fcbb000b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:36.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.382+0000 7fcbcda39640 1 --2- 172.21.15.37:0/543965336 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbb0005380 0x7fcbb000b9f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fcbb8003990 tx=0x7fcbb801c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96df04bfbbea54b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:36.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.382+0000 7fcbc7fff640 1 -- 172.21.15.37:0/543965336 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb8009520 con 0x7fcbb0005380 2024-04-26T00:23:36.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.382+0000 7fcbc7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:36.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.382+0000 7fcbc7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:23:36.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.382+0000 7fcbc7fff640 10 monclient: dump: 2024-04-26T00:23:36.371 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:36.371 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:36.371 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:36.371 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:36.371 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:36.372 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:36.372 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:36.372 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:36.372 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:36.372 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:36.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.382+0000 7fcbcf49c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:23:36.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.382+0000 7fcbc7fff640 1 -- 172.21.15.37:0/543965336 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbb8003bc0 con 0x7fcbb0005380 2024-04-26T00:23:36.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.382+0000 7fcbc7fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:23:36.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.382+0000 7fcbc7fff640 1 -- 172.21.15.37:0/543965336 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb8005070 con 0x7fcbb0005380 2024-04-26T00:23:36.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.382+0000 7fcbc7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:36.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.382+0000 7fcbc7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:23:36.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.382+0000 7fcbc7fff640 10 monclient: dump: 2024-04-26T00:23:36.372 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:36.372 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:36.372 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:36.373 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:36.373 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:36.373 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:36.373 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:36.373 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:36.373 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:36.373 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:36.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.382+0000 7fcbcf49c640 10 monclient: get_monmap_and_config success 2024-04-26T00:23:36.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.382+0000 7fcbcf49c640 4 set_mon_vals no callback set 2024-04-26T00:23:36.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.382+0000 7fcbcf49c640 10 monclient: shutdown 2024-04-26T00:23:36.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.382+0000 7fcbcf49c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:23:36.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.382+0000 7fcbcf49c640 1 -- 172.21.15.37:0/543965336 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbb0005380 msgr2=0x7fcbb000b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:36.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.382+0000 7fcbcf49c640 1 --2- 172.21.15.37:0/543965336 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbb0005380 0x7fcbb000b9f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fcbb8003990 tx=0x7fcbb801c420 comp rx=0 tx=0).stop 2024-04-26T00:23:36.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.382+0000 7fcbcf49c640 1 -- 172.21.15.37:0/543965336 shutdown_connections 2024-04-26T00:23:36.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.382+0000 7fcbcf49c640 1 --2- 172.21.15.37:0/543965336 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbb0005380 0x7fcbb000b9f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:36.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.382+0000 7fcbcf49c640 1 -- 172.21.15.37:0/543965336 >> 172.21.15.37:0/543965336 conn(0x7fcbc80a5f00 msgr2=0x7fcbc80a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:23:36.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.382+0000 7fcbcf49c640 1 -- 172.21.15.37:0/543965336 shutdown_connections 2024-04-26T00:23:36.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.382+0000 7fcbcf49c640 1 -- 172.21.15.37:0/543965336 wait complete. 2024-04-26T00:23:36.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcf49c640 10 monclient: build_initial_monmap 2024-04-26T00:23:36.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcf49c640 10 monclient: monmap: 2024-04-26T00:23:36.374 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:23:36.374 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:23:36.374 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:23:36.374 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:23:36.374 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:23:36.374 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:23:36.374 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:23:36.374 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:23:36.374 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:23:36.375 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:36.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcf49c640 1 Processor -- start 2024-04-26T00:23:36.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcf49c640 1 -- start start 2024-04-26T00:23:36.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcf49c640 10 monclient: init 2024-04-26T00:23:36.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcf49c640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:36.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcf49c640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:23:36.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcf49c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcbb0005380 0x7fcbc81cecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:36.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcf49c640 10 monclient(hunting): picked mon.noname-b con 0x7fcbb0005380 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:23:36.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcf49c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcbc8133790 0x7fcbc813e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:36.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcf49c640 10 monclient(hunting): picked mon.noname-a con 0x7fcbc8133790 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:23:36.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcf49c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbc813ef10 0x7fcbc81cb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:36.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcf49c640 10 monclient(hunting): picked mon.noname-c con 0x7fcbc813ef10 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:23:36.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcf49c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:36.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcf49c640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc809c4b0 con 0x7fcbc8133790 2024-04-26T00:23:36.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcf49c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:36.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcf49c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc809c330 con 0x7fcbb0005380 2024-04-26T00:23:36.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcf49c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:36.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcf49c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fcbc8136da0 con 0x7fcbc813ef10 2024-04-26T00:23:36.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcf49c640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:36.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcf49c640 10 monclient(hunting): authenticate will time out at 1197.693086s 2024-04-26T00:23:36.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcca37640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcbc8133790 msgr2=0x7fcbc813e9d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:23:36.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcca37640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcbc8133790 0x7fcbc813e9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:23:36.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcda39640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbc813ef10 0x7fcbc81cb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:36.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcda39640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbc813ef10 0x7fcbc81cb7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:44710/0 (socket says 172.21.15.37:44710) 2024-04-26T00:23:36.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcda39640 1 -- 172.21.15.37:0/2239186462 learned_addr learned my addr 172.21.15.37:0/2239186462 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:23:36.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcda39640 10 monclient(hunting): get_auth_request con 0x7fcbc813ef10 auth_method 0 2024-04-26T00:23:36.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcda39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:36.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcda39640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:36.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcda39640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:36.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcda39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:36.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcda39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:36.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcda39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:36.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcd238640 1 --2- 172.21.15.37:0/2239186462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcbb0005380 0x7fcbc81cecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:36.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcd238640 10 monclient(hunting): get_auth_request con 0x7fcbb0005380 auth_method 0 2024-04-26T00:23:36.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcd238640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:36.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcd238640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:36.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcd238640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:36.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcda39640 10 monclient(hunting): handle_auth_done global_id 34265 payload 824 2024-04-26T00:23:36.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcda39640 1 -- 172.21.15.37:0/2239186462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcbb0005380 msgr2=0x7fcbc81cecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:36.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcda39640 1 --2- 172.21.15.37:0/2239186462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcbb0005380 0x7fcbc81cecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:36.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcda39640 1 -- 172.21.15.37:0/2239186462 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcbc8133790 msgr2=0x7fcbc813e9d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:23:36.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcda39640 1 --2- 172.21.15.37:0/2239186462 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcbc8133790 0x7fcbc813e9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:36.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcda39640 10 monclient: _finish_hunting 0 2024-04-26T00:23:36.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcda39640 1 monclient: found mon.noname-c 2024-04-26T00:23:36.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcda39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:36.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcda39640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:23:36.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcda39640 1 -- 172.21.15.37:0/2239186462 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbc81365b0 con 0x7fcbc813ef10 2024-04-26T00:23:36.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcda39640 1 -- 172.21.15.37:0/2239186462 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbc813ef10 msgr2=0x7fcbc81cb7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:36.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.386+0000 7fcbcda39640 1 --2- 172.21.15.37:0/2239186462 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbc813ef10 0x7fcbc81cb7d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fcbb801c900 tx=0x7fcbb0005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:36.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbc5ffb640 1 -- 172.21.15.37:0/2239186462 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb8009500 con 0x7fcbc813ef10 2024-04-26T00:23:36.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbc5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:36.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbc5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:23:36.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbcda39640 1 --2- 172.21.15.37:0/2239186462 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbc813ef10 0x7fcbc81cb7d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fcbb801c900 tx=0x7fcbb0005840 comp rx=0 tx=0).queue_received will delay after 1.71409e+09 on 0x7fcbb80ac460 mon_map magic: 0 2024-04-26T00:23:36.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbc5ffb640 10 monclient: dump: 2024-04-26T00:23:36.378 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:36.378 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:36.378 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:36.378 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:36.378 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:36.378 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:36.379 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:36.379 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:36.379 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:36.379 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:36.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbc5ffb640 10 monclient: _finish_auth 0 2024-04-26T00:23:36.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbc5ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:23:36.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbc5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:36.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbc5ffb640 1 -- 172.21.15.37:0/2239186462 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbb8003d70 con 0x7fcbc813ef10 2024-04-26T00:23:36.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbc5ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:23:36.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbcf49c640 5 monclient: authenticate success, global_id 34265 2024-04-26T00:23:36.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbcf49c640 10 monclient: _renew_subs 2024-04-26T00:23:36.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbcf49c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:36.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbcf49c640 1 -- 172.21.15.37:0/2239186462 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbc81cf900 con 0x7fcbc813ef10 2024-04-26T00:23:36.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbc77fe640 4 set_mon_vals no callback set 2024-04-26T00:23:36.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbcf49c640 10 monclient: _renew_subs 2024-04-26T00:23:36.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbcf49c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:36.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbcf49c640 1 -- 172.21.15.37:0/2239186462 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbc813f650 con 0x7fcbc813ef10 2024-04-26T00:23:36.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbcf49c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:36.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbcf49c640 1 --2- 172.21.15.37:0/2239186462 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbc81cbd10 0x7fcbc81cc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:36.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbcf49c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:36.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbcf49c640 1 -- 172.21.15.37:0/2239186462 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fcbc81381f0 con 0x7fcbc81cbd10 2024-04-26T00:23:36.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbcd238640 1 --2- 172.21.15.37:0/2239186462 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbc81cbd10 0x7fcbc81cc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:36.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbcd238640 10 monclient: get_auth_request con 0x7fcbc81cbd10 auth_method 0 2024-04-26T00:23:36.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbcd238640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:36.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbcd238640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:36.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbcd238640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:36.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbcd238640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:36.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbcd238640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:36.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbc5ffb640 1 -- 172.21.15.37:0/2239186462 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb80ac460 con 0x7fcbc813ef10 2024-04-26T00:23:36.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbcd238640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:36.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbc5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:36.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbc5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:23:36.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbc5ffb640 10 monclient: dump: 2024-04-26T00:23:36.381 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:36.381 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:36.381 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:36.381 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:36.381 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:36.381 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:36.381 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:36.381 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:36.382 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:36.382 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:36.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbc5ffb640 1 -- 172.21.15.37:0/2239186462 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fcbb80ac600 con 0x7fcbc813ef10 2024-04-26T00:23:36.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbcd238640 10 monclient(hunting): handle_auth_done global_id 34268 payload 824 2024-04-26T00:23:36.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbcd238640 1 -- 172.21.15.37:0/2239186462 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcbc81cc630 con 0x7fcbc81cbd10 2024-04-26T00:23:36.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbc5ffb640 1 --2- 172.21.15.37:0/2239186462 >> v2:172.21.15.45:6812/2379918623 conn(0x7fcbb4035e30 0x7fcbb40382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:36.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbcca37640 1 --2- 172.21.15.37:0/2239186462 >> v2:172.21.15.45:6812/2379918623 conn(0x7fcbb4035e30 0x7fcbb40382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:36.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbcca37640 10 monclient: get_auth_request con 0x7fcbb4035e30 auth_method 0 2024-04-26T00:23:36.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbcd238640 1 -- 172.21.15.37:0/2239186462 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbc81cbd10 msgr2=0x7fcbc81cc0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:36.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbcd238640 1 --2- 172.21.15.37:0/2239186462 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbc81cbd10 0x7fcbc81cc0f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcbc000e9a0 tx=0x7fcbc000ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:36.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbc5ffb640 1 -- 172.21.15.37:0/2239186462 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc0018420 con 0x7fcbc81cbd10 2024-04-26T00:23:36.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbc5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:36.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbc5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:23:36.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.390+0000 7fcbc5ffb640 10 monclient: dump: 2024-04-26T00:23:36.383 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:36.383 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:36.383 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:36.383 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:36.383 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:36.383 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:36.383 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:36.383 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:36.383 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:36.383 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:36.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.394+0000 7fcbc5ffb640 1 -- 172.21.15.37:0/2239186462 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcbc0018420 con 0x7fcbc81cbd10 2024-04-26T00:23:36.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.394+0000 7fcbc5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:36.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.394+0000 7fcbc5ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:23:36.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.394+0000 7fcbc5ffb640 1 -- 172.21.15.37:0/2239186462 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbc81cbd10 msgr2=0x7fcbc81cc0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:36.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.394+0000 7fcbc5ffb640 1 --2- 172.21.15.37:0/2239186462 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbc81cbd10 0x7fcbc81cc0f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcbc000e9a0 tx=0x7fcbc000ee60 comp rx=0 tx=0).stop 2024-04-26T00:23:36.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.394+0000 7fcbc5ffb640 1 -- 172.21.15.37:0/2239186462 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbc81cbd10 msgr2=0x7fcbc81cc0f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:23:36.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.394+0000 7fcbc5ffb640 1 --2- 172.21.15.37:0/2239186462 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbc81cbd10 0x7fcbc81cc0f0 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcbc000e9a0 tx=0x7fcbc000ee60 comp rx=0 tx=0).stop 2024-04-26T00:23:36.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.394+0000 7fcbcca37640 1 --2- 172.21.15.37:0/2239186462 >> v2:172.21.15.45:6812/2379918623 conn(0x7fcbb4035e30 0x7fcbb40382f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc000eb0 tx=0x7fcbbc001dd0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:36.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.398+0000 7fcbcf49c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:23:36.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.398+0000 7fcbcf49c640 1 --2- 172.21.15.37:0/2239186462 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbc8137df0 0x7fcbc809d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:36.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.398+0000 7fcbcf49c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:36.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.398+0000 7fcbcf49c640 1 -- 172.21.15.37:0/2239186462 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fcbc8132c30 con 0x7fcbc8137df0 2024-04-26T00:23:36.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.398+0000 7fcbcd238640 1 --2- 172.21.15.37:0/2239186462 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbc8137df0 0x7fcbc809d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:36.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.398+0000 7fcbcd238640 10 monclient: get_auth_request con 0x7fcbc8137df0 auth_method 0 2024-04-26T00:23:36.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.398+0000 7fcbcd238640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:36.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.398+0000 7fcbcd238640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:36.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.398+0000 7fcbcd238640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:36.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.398+0000 7fcbcd238640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:36.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.398+0000 7fcbcd238640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:36.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.398+0000 7fcbcd238640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:36.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.398+0000 7fcbcd238640 10 monclient(hunting): handle_auth_done global_id 34271 payload 824 2024-04-26T00:23:36.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.398+0000 7fcbcd238640 1 -- 172.21.15.37:0/2239186462 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcbc8076040 con 0x7fcbc8137df0 2024-04-26T00:23:36.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.402+0000 7fcbcd238640 1 -- 172.21.15.37:0/2239186462 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbc8137df0 msgr2=0x7fcbc809d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:36.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.402+0000 7fcbcd238640 1 --2- 172.21.15.37:0/2239186462 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbc8137df0 0x7fcbc809d0e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0018000 tx=0x7fcbc0005da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:36.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.402+0000 7fcbc5ffb640 1 -- 172.21.15.37:0/2239186462 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc00187e0 con 0x7fcbc8137df0 2024-04-26T00:23:36.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.402+0000 7fcbc5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:36.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.402+0000 7fcbc5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:23:36.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.402+0000 7fcbc5ffb640 10 monclient: dump: 2024-04-26T00:23:36.386 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:36.387 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:36.387 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:36.387 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:36.387 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:36.387 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:36.387 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:36.387 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:36.387 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:36.387 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:36.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.402+0000 7fcbc5ffb640 1 -- 172.21.15.37:0/2239186462 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4707 (secure 0 0 0) 0x7fcbc001e040 con 0x7fcbc8137df0 2024-04-26T00:23:36.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.402+0000 7fcbc5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:23:36.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.402+0000 7fcbc5ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:23:36.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.402+0000 7fcbc5ffb640 1 -- 172.21.15.37:0/2239186462 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbc8137df0 msgr2=0x7fcbc809d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:36.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.402+0000 7fcbc5ffb640 1 --2- 172.21.15.37:0/2239186462 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbc8137df0 0x7fcbc809d0e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0018000 tx=0x7fcbc0005da0 comp rx=0 tx=0).stop 2024-04-26T00:23:36.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.402+0000 7fcbc5ffb640 1 -- 172.21.15.37:0/2239186462 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbc8137df0 msgr2=0x7fcbc809d0e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:23:36.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.402+0000 7fcbc5ffb640 1 --2- 172.21.15.37:0/2239186462 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbc8137df0 0x7fcbc809d0e0 secure :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0018000 tx=0x7fcbc0005da0 comp rx=0 tx=0).stop 2024-04-26T00:23:36.388 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:23:36.388 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:23:36.388 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:23:36.388 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:23:36.388 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 32, 2024-04-26T00:23:36.388 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 76679, 2024-04-26T00:23:36.388 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:23:36.388 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:23:36.388 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:23:36.388 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:36.389 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:23:36.389 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:23:36.389 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:23:36.389 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:23:36.389 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:23:36.389 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:23:36.389 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:23:36.389 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:23:36.389 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:23:36.389 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:23:36.389 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:23:36.389 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:23:36.389 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:23:36.389 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:23:36.390 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:23:36.390 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:36.390 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:23:36.390 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:23:36.390 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:23:36.390 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:23:36.390 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:23:36.390 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:23:36.390 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:23:36.390 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:23:36.390 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:23:36.390 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:23:36.390 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:23:36.390 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:23:36.391 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:23:36.391 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:36.391 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:36.391 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:23:36.391 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:23:36.391 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:23:36.391 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:23:36.391 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:23:36.391 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:23:36.391 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:23:36.391 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:23:36.391 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:23:36.391 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:23:36.391 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:23:36.392 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:23:36.392 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:23:36.392 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:23:36.392 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:23:36.392 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:23:36.392 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:23:36.392 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:23:36.392 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:23:36.392 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:23:36.392 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:23:36.392 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:23:36.392 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:23:36.392 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:23:36.392 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:23:36.392 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:23:36.392 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:23:36.393 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:23:36.393 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:36.393 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:23:36.393 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:36.393 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:23:36.393 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:36.393 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:23:36.393 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:23:36.393 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:23:36.393 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:23:36.393 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:36.393 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:23:36.393 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:23:36.393 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:36.393 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:36.394 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:36.394 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:23:36.394 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:23:36.394 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:36.394 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:36.394 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:36.394 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:36.394 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:23:36.394 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:23:36.394 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:23:36.394 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:23:36.394 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:23:36.394 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:36.394 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:36.395 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:23:36.395 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:23:36.395 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:23:36.395 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:23:36.395 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:36.395 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:23:36.395 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:23:36.395 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:36.395 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:36.395 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:36.395 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:23:36.395 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:23:36.395 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:36.395 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:36.395 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:36.395 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:36.396 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:23:36.396 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:23:36.396 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:23:36.396 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:23:36.396 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:23:36.396 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:36.396 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:36.396 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:23:36.396 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:23:36.396 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:23:36.396 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:23:36.396 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:36.396 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:23:36.396 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:23:36.397 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:36.397 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:36.397 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:36.397 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:23:36.397 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:23:36.397 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:36.397 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:36.397 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:36.397 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:36.397 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:23:36.397 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:23:36.398 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:23:36.398 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:23:36.398 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:23:36.398 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:36.398 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:36.398 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:36.398 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:23:36.398 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:23:36.398 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:36.398 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:23:36.398 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:23:36.398 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:23:36.398 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:36.398 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:36.398 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:23:36.399 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:36.399 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:23:36.399 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:23:36.399 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:23:36.399 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:36.399 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:36.399 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:23:36.399 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:36.399 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:23:36.399 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:23:36.399 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:23:36.399 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:36.399 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:36.399 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:36.399 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:23:36.400 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:23:36.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.402+0000 7fcbcf49c640 1 -- 172.21.15.37:0/2239186462 >> v2:172.21.15.45:6812/2379918623 conn(0x7fcbb4035e30 msgr2=0x7fcbb40382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:36.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.402+0000 7fcbcf49c640 1 --2- 172.21.15.37:0/2239186462 >> v2:172.21.15.45:6812/2379918623 conn(0x7fcbb4035e30 0x7fcbb40382f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc000eb0 tx=0x7fcbbc001dd0 comp rx=0 tx=0).stop 2024-04-26T00:23:36.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.402+0000 7fcbcf49c640 10 monclient: shutdown 2024-04-26T00:23:36.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.402+0000 7fcbcf49c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:23:36.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.402+0000 7fcbcf49c640 1 -- 172.21.15.37:0/2239186462 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbc813ef10 msgr2=0x7fcbc81cb7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:36.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.402+0000 7fcbcf49c640 1 --2- 172.21.15.37:0/2239186462 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbc813ef10 0x7fcbc81cb7d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fcbb801c900 tx=0x7fcbb0005840 comp rx=0 tx=0).stop 2024-04-26T00:23:36.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.402+0000 7fcbcda39640 1 -- 172.21.15.37:0/2239186462 reap_dead start 2024-04-26T00:23:36.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.402+0000 7fcbcda39640 1 -- 172.21.15.37:0/2239186462 reap_dead start 2024-04-26T00:23:36.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.402+0000 7fcbc5ffb640 1 -- 172.21.15.37:0/2239186462 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(147..147 src has 1..147) ==== 3931+0+0 (unknown 0 0 0) 0x7fcbb80e1cf0 con 0x7fcbc813ef10 2024-04-26T00:23:36.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.402+0000 7fcbcf49c640 1 -- 172.21.15.37:0/2239186462 shutdown_connections 2024-04-26T00:23:36.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.402+0000 7fcbcf49c640 1 -- 172.21.15.37:0/2239186462 >> 172.21.15.37:0/2239186462 conn(0x7fcbc80a5f00 msgr2=0x7fcbc8135ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:23:36.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.402+0000 7fcbcf49c640 1 -- 172.21.15.37:0/2239186462 shutdown_connections 2024-04-26T00:23:36.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.402+0000 7fcbcf49c640 1 -- 172.21.15.37:0/2239186462 wait complete. 2024-04-26T00:23:36.402 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:23:36.402 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-04-26T00:23:36.403 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:23:36.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9faf8640 10 monclient: get_monmap_and_config 2024-04-26T00:23:36.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9faf8640 10 monclient: build_initial_monmap 2024-04-26T00:23:36.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9faf8640 10 monclient: monmap: 2024-04-26T00:23:36.519 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:23:36.519 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:36.519 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:23:36.544237+0000 2024-04-26T00:23:36.519 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:23:36.544237+0000 2024-04-26T00:23:36.519 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:23:36.520 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:36.520 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:23:36.520 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:23:36.520 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:23:36.520 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:23:36.520 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:23:36.520 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:36.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9faf8640 1 Processor -- start 2024-04-26T00:23:36.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9faf8640 1 -- start start 2024-04-26T00:23:36.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9faf8640 10 monclient: init 2024-04-26T00:23:36.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9faf8640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:36.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9faf8640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-04-26T00:23:36.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9faf8640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbe9809e220 0x7fbe9809fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:36.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9faf8640 10 monclient(hunting): picked mon.noname-c con 0x7fbe9809e220 addr v2:172.21.15.45:3300/0 2024-04-26T00:23:36.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9faf8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbe980a14f0 addr v1:172.21.15.37:6789/0 2024-04-26T00:23:36.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9faf8640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fbe9809e600 0x7fbe9813aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:36.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9faf8640 10 monclient(hunting): picked mon.noname-a con 0x7fbe9809e600 addr v2:172.21.15.37:3300/0 2024-04-26T00:23:36.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9faf8640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe980a12b0 con 0x7fbe980a14f0 2024-04-26T00:23:36.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9faf8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:36.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9faf8640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fbe9809da20 con 0x7fbe9809e600 2024-04-26T00:23:36.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9faf8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:36.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9faf8640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fbe9809dd00 con 0x7fbe9809e220 2024-04-26T00:23:36.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9faf8640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:36.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9faf8640 10 monclient(hunting): authenticate will time out at 1197.850648s 2024-04-26T00:23:36.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9d093640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7fbe980a14f0 legacy=0x7fbe980a0190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:23:36.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9e095640 1 -- >> v2:172.21.15.37:3300/0 conn(0x7fbe9809e600 msgr2=0x7fbe9813aa60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:23:36.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9e095640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fbe9809e600 0x7fbe9813aa60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:23:36.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9d894640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbe9809e220 0x7fbe9809fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:36.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9d894640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbe9809e220 0x7fbe9809fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.37:42802/0 (socket says 172.21.15.37:42802) 2024-04-26T00:23:36.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9d894640 1 -- 172.21.15.37:0/332655476 learned_addr learned my addr 172.21.15.37:0/332655476 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:23:36.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9d894640 10 monclient(hunting): get_auth_request con 0x7fbe9809e220 auth_method 0 2024-04-26T00:23:36.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9d894640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:36.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9d894640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:36.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9d894640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:36.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9d894640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:36.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9d894640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:36.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9d894640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:36.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9d894640 10 monclient(hunting): handle_auth_done global_id 24322 payload 274 2024-04-26T00:23:36.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9d894640 1 -- 172.21.15.37:0/332655476 >> v2:172.21.15.37:3300/0 conn(0x7fbe9809e600 msgr2=0x7fbe9813aa60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:23:36.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9d894640 1 --2- 172.21.15.37:0/332655476 >> v2:172.21.15.37:3300/0 conn(0x7fbe9809e600 0x7fbe9813aa60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:36.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9d894640 1 -- 172.21.15.37:0/332655476 >> v1:172.21.15.37:6789/0 conn(0x7fbe980a14f0 legacy=0x7fbe980a0190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:23:36.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9d894640 10 monclient: _finish_hunting 0 2024-04-26T00:23:36.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9d894640 1 monclient: found mon.noname-c 2024-04-26T00:23:36.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9d894640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:36.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9d894640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:23:36.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.542+0000 7fbe9d894640 1 -- 172.21.15.37:0/332655476 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe98073310 con 0x7fbe9809e220 2024-04-26T00:23:36.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9d894640 1 -- 172.21.15.37:0/332655476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe9809e220 msgr2=0x7fbe9809fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:36.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9d894640 1 --2- 172.21.15.37:0/332655476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe9809e220 0x7fbe9809fc50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fbe980764d0 tx=0x7fbe9401d760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab505460a068824f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:36.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9c892640 1 -- 172.21.15.37:0/332655476 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe940b3070 con 0x7fbe9809e220 2024-04-26T00:23:36.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9c892640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:36.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9c892640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:23:36.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9c892640 10 monclient: dump: 2024-04-26T00:23:36.527 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:36.527 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:36.527 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:36.527 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:36.527 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:36.527 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:36.527 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:36.527 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:36.528 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:36.528 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:36.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9c892640 10 monclient: _finish_auth 0 2024-04-26T00:23:36.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9c892640 10 monclient: _check_auth_tickets 2024-04-26T00:23:36.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9c892640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:36.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9faf8640 5 monclient: authenticate success, global_id 24322 2024-04-26T00:23:36.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9faf8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:23:36.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9c892640 1 -- 172.21.15.37:0/332655476 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe980a0ab0 con 0x7fbe9809e220 2024-04-26T00:23:36.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9c892640 10 monclient: handle_config config(0 keys) 2024-04-26T00:23:36.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9faf8640 10 monclient: get_monmap_and_config success 2024-04-26T00:23:36.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9faf8640 4 set_mon_vals no callback set 2024-04-26T00:23:36.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9c892640 1 -- 172.21.15.37:0/332655476 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe9400b040 con 0x7fbe9809e220 2024-04-26T00:23:36.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9c892640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:36.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9c892640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:23:36.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9c892640 10 monclient: dump: 2024-04-26T00:23:36.529 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:36.529 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:36.529 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:36.529 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:36.529 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:36.529 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:36.529 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:36.529 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:36.529 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:36.529 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:36.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9faf8640 10 monclient: shutdown 2024-04-26T00:23:36.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9faf8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:23:36.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9faf8640 1 -- 172.21.15.37:0/332655476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe9809e220 msgr2=0x7fbe9809fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:36.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9faf8640 1 --2- 172.21.15.37:0/332655476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe9809e220 0x7fbe9809fc50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fbe980764d0 tx=0x7fbe9401d760 comp rx=0 tx=0).stop 2024-04-26T00:23:36.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9faf8640 1 -- 172.21.15.37:0/332655476 shutdown_connections 2024-04-26T00:23:36.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9faf8640 1 --2- 172.21.15.37:0/332655476 >> v2:172.21.15.37:3300/0 conn(0x7fbe9809e600 0x7fbe9813aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:36.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9faf8640 1 --2- 172.21.15.37:0/332655476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe9809e220 0x7fbe9809fc50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:36.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9faf8640 1 -- 172.21.15.37:0/332655476 >> 172.21.15.37:0/332655476 conn(0x7fbe980a1d50 msgr2=0x7fbe980a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:23:36.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9faf8640 1 -- 172.21.15.37:0/332655476 shutdown_connections 2024-04-26T00:23:36.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9faf8640 1 -- 172.21.15.37:0/332655476 wait complete. 2024-04-26T00:23:36.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9faf8640 10 monclient: build_initial_monmap 2024-04-26T00:23:36.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9faf8640 10 monclient: monmap: 2024-04-26T00:23:36.536 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:23:36.536 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:23:36.536 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:23:36.536 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:23:36.536 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:23:36.536 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:23:36.536 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:23:36.537 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:23:36.537 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:23:36.537 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:36.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9faf8640 1 Processor -- start 2024-04-26T00:23:36.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9faf8640 1 -- start start 2024-04-26T00:23:36.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9faf8640 10 monclient: init 2024-04-26T00:23:36.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9faf8640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:36.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9faf8640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:23:36.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9faf8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbe9809e220 0x7fbe981cccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:36.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9faf8640 10 monclient(hunting): picked mon.noname-c con 0x7fbe9809e220 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:23:36.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9faf8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe9809e600 0x7fbe98135bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:36.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9faf8640 10 monclient(hunting): picked mon.noname-b con 0x7fbe9809e600 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:23:36.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9faf8640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbe980a14f0 0x7fbe9811c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:36.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9faf8640 10 monclient(hunting): picked mon.noname-a con 0x7fbe980a14f0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:23:36.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9faf8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:36.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9faf8640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fbe9813d3d0 con 0x7fbe980a14f0 2024-04-26T00:23:36.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9faf8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:36.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9faf8640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbe9813d250 con 0x7fbe9809e600 2024-04-26T00:23:36.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9faf8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:36.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9faf8640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fbe9813d550 con 0x7fbe9809e220 2024-04-26T00:23:36.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9faf8640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:36.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9faf8640 10 monclient(hunting): authenticate will time out at 1197.859402s 2024-04-26T00:23:36.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9e095640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbe980a14f0 msgr2=0x7fbe9811c610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:23:36.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9e095640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbe980a14f0 0x7fbe9811c610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:23:36.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9d894640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbe9809e220 0x7fbe981cccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:36.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9d894640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbe9809e220 0x7fbe981cccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:44728/0 (socket says 172.21.15.37:44728) 2024-04-26T00:23:36.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9d894640 1 -- 172.21.15.37:0/1265491100 learned_addr learned my addr 172.21.15.37:0/1265491100 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:23:36.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9d894640 10 monclient(hunting): get_auth_request con 0x7fbe9809e220 auth_method 0 2024-04-26T00:23:36.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9d894640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:36.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9d894640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:36.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9d894640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:36.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9d894640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:36.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9d894640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:36.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9d894640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:36.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9d093640 1 --2- 172.21.15.37:0/1265491100 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe9809e600 0x7fbe98135bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:36.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9d093640 10 monclient(hunting): get_auth_request con 0x7fbe9809e600 auth_method 0 2024-04-26T00:23:36.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9d093640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:36.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9d093640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:36.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9d093640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:36.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9d894640 10 monclient(hunting): handle_auth_done global_id 34274 payload 824 2024-04-26T00:23:36.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9d894640 1 -- 172.21.15.37:0/1265491100 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe9809e600 msgr2=0x7fbe98135bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:36.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9d894640 1 --2- 172.21.15.37:0/1265491100 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe9809e600 0x7fbe98135bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:36.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9d894640 1 -- 172.21.15.37:0/1265491100 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbe980a14f0 msgr2=0x7fbe9811c610 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:23:36.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9d894640 1 --2- 172.21.15.37:0/1265491100 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbe980a14f0 0x7fbe9811c610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:36.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9d894640 10 monclient: _finish_hunting 0 2024-04-26T00:23:36.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9d894640 1 monclient: found mon.noname-c 2024-04-26T00:23:36.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9d894640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:36.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9d894640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:23:36.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.550+0000 7fbe9d894640 1 -- 172.21.15.37:0/1265491100 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe9809c970 con 0x7fbe9809e220 2024-04-26T00:23:36.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.554+0000 7fbe9d894640 1 -- 172.21.15.37:0/1265491100 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbe9809e220 msgr2=0x7fbe981cccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:36.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.554+0000 7fbe9d894640 1 --2- 172.21.15.37:0/1265491100 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbe9809e220 0x7fbe981cccc0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fbe9401b390 tx=0x7fbe9401a500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:36.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.554+0000 7fbe8e7fc640 1 -- 172.21.15.37:0/1265491100 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe940b3040 con 0x7fbe9809e220 2024-04-26T00:23:36.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.554+0000 7fbe8e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:36.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.554+0000 7fbe8e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:23:36.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.554+0000 7fbe8e7fc640 10 monclient: dump: 2024-04-26T00:23:36.540 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:36.540 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:36.541 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:36.541 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:36.541 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:36.541 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:36.541 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:36.541 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:36.541 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:36.541 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:36.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.554+0000 7fbe8e7fc640 10 monclient: _finish_auth 0 2024-04-26T00:23:36.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.554+0000 7fbe8e7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:23:36.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.554+0000 7fbe8e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:36.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.554+0000 7fbe8e7fc640 1 -- 172.21.15.37:0/1265491100 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe940bc070 con 0x7fbe9809e220 2024-04-26T00:23:36.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.554+0000 7fbe9faf8640 5 monclient: authenticate success, global_id 34274 2024-04-26T00:23:36.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.554+0000 7fbe9faf8640 10 monclient: _renew_subs 2024-04-26T00:23:36.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.554+0000 7fbe9faf8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:36.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.554+0000 7fbe9faf8640 1 -- 172.21.15.37:0/1265491100 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe98072af0 con 0x7fbe9809e220 2024-04-26T00:23:36.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.554+0000 7fbe8e7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:23:36.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.554+0000 7fbe8e7fc640 1 -- 172.21.15.37:0/1265491100 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe940aeab0 con 0x7fbe9809e220 2024-04-26T00:23:36.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.554+0000 7fbe8e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:36.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.554+0000 7fbe8e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:23:36.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.554+0000 7fbe8e7fc640 10 monclient: dump: 2024-04-26T00:23:36.542 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:36.542 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:36.543 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:36.543 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:36.543 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:36.543 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:36.543 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:36.543 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:36.543 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:36.543 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:36.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.554+0000 7fbe8ffff640 4 set_mon_vals no callback set 2024-04-26T00:23:36.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.554+0000 7fbe9faf8640 10 monclient: _renew_subs 2024-04-26T00:23:36.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.554+0000 7fbe9faf8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:36.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.554+0000 7fbe9faf8640 1 -- 172.21.15.37:0/1265491100 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe9811fd00 con 0x7fbe9809e220 2024-04-26T00:23:36.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.554+0000 7fbe9faf8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:36.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.554+0000 7fbe9faf8640 1 --2- 172.21.15.37:0/1265491100 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe98136130 0x7fbe9811cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:36.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.554+0000 7fbe9faf8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:36.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.554+0000 7fbe9faf8640 1 -- 172.21.15.37:0/1265491100 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbe980a0ab0 con 0x7fbe98136130 2024-04-26T00:23:36.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.554+0000 7fbe8e7fc640 1 -- 172.21.15.37:0/1265491100 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fbe940aec90 con 0x7fbe9809e220 2024-04-26T00:23:36.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.554+0000 7fbe9d093640 1 --2- 172.21.15.37:0/1265491100 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe98136130 0x7fbe9811cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:36.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.554+0000 7fbe9d093640 10 monclient: get_auth_request con 0x7fbe98136130 auth_method 0 2024-04-26T00:23:36.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.554+0000 7fbe9d093640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:36.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.554+0000 7fbe9d093640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:36.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.554+0000 7fbe9d093640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:36.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.554+0000 7fbe9d093640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:36.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.554+0000 7fbe9d093640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:36.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.554+0000 7fbe9d093640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:36.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.554+0000 7fbe8e7fc640 1 --2- 172.21.15.37:0/1265491100 >> v2:172.21.15.45:6812/2379918623 conn(0x7fbe7c035e30 0x7fbe7c0382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:36.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.554+0000 7fbe8e7fc640 1 -- 172.21.15.37:0/1265491100 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(147..147 src has 1..147) ==== 3931+0+0 (secure 0 0 0) 0x7fbe940e2d60 con 0x7fbe9809e220 2024-04-26T00:23:36.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.554+0000 7fbe9d093640 10 monclient(hunting): handle_auth_done global_id 24328 payload 824 2024-04-26T00:23:36.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.558+0000 7fbe9d093640 1 -- 172.21.15.37:0/1265491100 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbe98120030 con 0x7fbe98136130 2024-04-26T00:23:36.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.558+0000 7fbe9e095640 1 --2- 172.21.15.37:0/1265491100 >> v2:172.21.15.45:6812/2379918623 conn(0x7fbe7c035e30 0x7fbe7c0382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:36.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.558+0000 7fbe9e095640 10 monclient: get_auth_request con 0x7fbe7c035e30 auth_method 0 2024-04-26T00:23:36.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.558+0000 7fbe9d093640 1 -- 172.21.15.37:0/1265491100 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe98136130 msgr2=0x7fbe9811cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:36.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.558+0000 7fbe9d093640 1 --2- 172.21.15.37:0/1265491100 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe98136130 0x7fbe9811cb50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fbe98136a10 tx=0x7fbe8800db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:36.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.558+0000 7fbe8e7fc640 1 -- 172.21.15.37:0/1265491100 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe88003bd0 con 0x7fbe98136130 2024-04-26T00:23:36.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.558+0000 7fbe8e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:36.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.558+0000 7fbe8e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:23:36.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.558+0000 7fbe8e7fc640 10 monclient: dump: 2024-04-26T00:23:36.547 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:36.547 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:36.547 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:36.547 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:36.547 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:36.547 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:36.548 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:36.548 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:36.548 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:36.548 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:36.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.558+0000 7fbe9e095640 1 --2- 172.21.15.37:0/1265491100 >> v2:172.21.15.45:6812/2379918623 conn(0x7fbe7c035e30 0x7fbe7c0382f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fbe90003cc0 tx=0x7fbe900091f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:36.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.558+0000 7fbe8e7fc640 1 -- 172.21.15.37:0/1265491100 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbe88017070 con 0x7fbe98136130 2024-04-26T00:23:36.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.558+0000 7fbe8e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:36.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.558+0000 7fbe8e7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:23:36.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.558+0000 7fbe8e7fc640 1 -- 172.21.15.37:0/1265491100 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe98136130 msgr2=0x7fbe9811cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:36.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.558+0000 7fbe8e7fc640 1 --2- 172.21.15.37:0/1265491100 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe98136130 0x7fbe9811cb50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fbe98136a10 tx=0x7fbe8800db90 comp rx=0 tx=0).stop 2024-04-26T00:23:36.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.558+0000 7fbe8e7fc640 1 -- 172.21.15.37:0/1265491100 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe98136130 msgr2=0x7fbe9811cb50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:23:36.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.558+0000 7fbe8e7fc640 1 --2- 172.21.15.37:0/1265491100 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe98136130 0x7fbe9811cb50 secure :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fbe98136a10 tx=0x7fbe8800db90 comp rx=0 tx=0).stop 2024-04-26T00:23:36.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.566+0000 7fbe9faf8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:23:36.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.566+0000 7fbe9faf8640 1 --2- 172.21.15.37:0/1265491100 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe9811d090 0x7fbe98137e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:36.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.566+0000 7fbe9faf8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:36.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.566+0000 7fbe9faf8640 1 -- 172.21.15.37:0/1265491100 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbe9811d4a0 con 0x7fbe9811d090 2024-04-26T00:23:36.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.566+0000 7fbe9d093640 1 --2- 172.21.15.37:0/1265491100 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe9811d090 0x7fbe98137e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:36.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.566+0000 7fbe9d093640 10 monclient: get_auth_request con 0x7fbe9811d090 auth_method 0 2024-04-26T00:23:36.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.566+0000 7fbe9d093640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:36.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.566+0000 7fbe9d093640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:36.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.566+0000 7fbe9d093640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:36.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.566+0000 7fbe9d093640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:36.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.566+0000 7fbe9d093640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:36.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.566+0000 7fbe9d093640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:36.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.566+0000 7fbe9d093640 10 monclient(hunting): handle_auth_done global_id 24331 payload 824 2024-04-26T00:23:36.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.566+0000 7fbe9d093640 1 -- 172.21.15.37:0/1265491100 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbe9804f510 con 0x7fbe9811d090 2024-04-26T00:23:36.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.570+0000 7fbe9d093640 1 -- 172.21.15.37:0/1265491100 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe9811d090 msgr2=0x7fbe98137e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:36.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.570+0000 7fbe9d093640 1 --2- 172.21.15.37:0/1265491100 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe9811d090 0x7fbe98137e00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbe88016000 tx=0x7fbe880059e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:36.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.570+0000 7fbe8e7fc640 1 -- 172.21.15.37:0/1265491100 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe880054e0 con 0x7fbe9811d090 2024-04-26T00:23:36.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.570+0000 7fbe8e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:36.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.570+0000 7fbe8e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:23:36.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.570+0000 7fbe8e7fc640 10 monclient: dump: 2024-04-26T00:23:36.550 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:36.551 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:36.551 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:36.551 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:36.551 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:36.551 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:36.551 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:36.551 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:36.551 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:36.551 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:36.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.570+0000 7fbe8e7fc640 1 -- 172.21.15.37:0/1265491100 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7fbe880054e0 con 0x7fbe9811d090 2024-04-26T00:23:36.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.570+0000 7fbe8e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:23:36.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.570+0000 7fbe8e7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:23:36.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.570+0000 7fbe8e7fc640 1 -- 172.21.15.37:0/1265491100 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe9811d090 msgr2=0x7fbe98137e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:36.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.570+0000 7fbe8e7fc640 1 --2- 172.21.15.37:0/1265491100 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe9811d090 0x7fbe98137e00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbe88016000 tx=0x7fbe880059e0 comp rx=0 tx=0).stop 2024-04-26T00:23:36.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.570+0000 7fbe8e7fc640 1 -- 172.21.15.37:0/1265491100 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe9811d090 msgr2=0x7fbe98137e00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:23:36.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.570+0000 7fbe8e7fc640 1 --2- 172.21.15.37:0/1265491100 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe9811d090 0x7fbe98137e00 secure :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbe88016000 tx=0x7fbe880059e0 comp rx=0 tx=0).stop 2024-04-26T00:23:36.552 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:23:36.552 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:23:36.552 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:23:36.552 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:23:36.552 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 32, 2024-04-26T00:23:36.552 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 124737, 2024-04-26T00:23:36.552 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:23:36.553 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:23:36.553 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:23:36.553 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:36.553 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:23:36.553 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:23:36.553 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:23:36.553 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:23:36.553 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:23:36.553 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:23:36.553 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:23:36.553 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:23:36.553 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:23:36.553 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:23:36.553 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:23:36.553 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:23:36.554 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:23:36.554 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:23:36.554 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:23:36.554 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:36.554 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:23:36.554 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:23:36.554 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:23:36.554 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:23:36.554 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:23:36.554 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:23:36.554 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:23:36.554 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:23:36.554 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:23:36.555 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:23:36.555 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:23:36.555 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:23:36.555 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:23:36.555 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:36.555 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:36.555 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:23:36.555 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:23:36.555 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:23:36.555 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:23:36.555 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:23:36.555 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:23:36.555 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:23:36.556 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:23:36.556 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:23:36.556 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:23:36.556 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:23:36.556 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:23:36.556 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:23:36.556 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:23:36.556 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:23:36.556 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:23:36.556 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:23:36.556 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:23:36.556 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:23:36.556 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:23:36.556 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:23:36.557 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:23:36.557 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:23:36.557 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:23:36.557 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:23:36.557 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:23:36.557 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:23:36.557 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:23:36.557 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:36.557 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:23:36.557 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:36.557 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:23:36.557 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:36.557 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:23:36.558 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:23:36.558 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:23:36.558 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:23:36.558 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:36.558 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:23:36.558 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:23:36.558 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:36.558 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:36.558 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:36.558 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:23:36.558 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:23:36.558 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:36.558 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:36.559 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:36.559 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:36.559 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:23:36.559 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:23:36.559 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:23:36.559 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:23:36.559 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:23:36.559 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:36.559 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:36.559 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:23:36.559 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:23:36.559 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:23:36.559 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:23:36.560 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:36.560 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:23:36.560 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:23:36.560 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:36.560 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:36.560 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:36.560 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:23:36.560 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:23:36.560 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:36.560 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:36.560 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:36.560 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:36.560 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:23:36.561 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:23:36.561 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:23:36.561 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:23:36.561 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:23:36.561 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:36.561 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:36.561 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:23:36.561 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:23:36.561 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:23:36.561 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:23:36.561 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:36.561 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:23:36.561 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:23:36.561 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:36.562 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:36.562 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:36.562 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:23:36.562 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:23:36.562 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:23:36.562 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:36.562 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:36.562 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:36.562 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:23:36.562 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:23:36.562 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:23:36.562 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:23:36.562 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:23:36.562 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:36.563 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:36.563 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:36.563 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:23:36.563 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:23:36.563 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:36.563 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:23:36.563 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:23:36.563 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:23:36.563 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:36.563 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:36.563 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:23:36.563 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:36.563 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:23:36.563 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:23:36.564 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:23:36.564 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:36.564 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:36.564 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:23:36.564 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:36.564 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:23:36.564 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:23:36.564 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:23:36.564 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:36.564 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:23:36.564 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:23:36.564 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:23:36.564 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:23:36.564 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:23:36.565 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:23:36.565 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:23:36.565 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:23:36.565 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:23:36.565 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:23:36.565 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:23:36.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.570+0000 7fbe9faf8640 1 -- 172.21.15.37:0/1265491100 >> v2:172.21.15.45:6812/2379918623 conn(0x7fbe7c035e30 msgr2=0x7fbe7c0382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:36.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.570+0000 7fbe9faf8640 1 --2- 172.21.15.37:0/1265491100 >> v2:172.21.15.45:6812/2379918623 conn(0x7fbe7c035e30 0x7fbe7c0382f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fbe90003cc0 tx=0x7fbe900091f0 comp rx=0 tx=0).stop 2024-04-26T00:23:36.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.570+0000 7fbe9faf8640 10 monclient: shutdown 2024-04-26T00:23:36.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.570+0000 7fbe9faf8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:23:36.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.570+0000 7fbe9faf8640 1 -- 172.21.15.37:0/1265491100 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbe9809e220 msgr2=0x7fbe981cccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:36.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.570+0000 7fbe9faf8640 1 --2- 172.21.15.37:0/1265491100 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbe9809e220 0x7fbe981cccc0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fbe9401b390 tx=0x7fbe9401a500 comp rx=0 tx=0).stop 2024-04-26T00:23:36.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.570+0000 7fbe9e095640 1 -- 172.21.15.37:0/1265491100 reap_dead start 2024-04-26T00:23:36.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.570+0000 7fbe9e095640 1 -- 172.21.15.37:0/1265491100 reap_dead start 2024-04-26T00:23:36.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.570+0000 7fbe9faf8640 1 -- 172.21.15.37:0/1265491100 shutdown_connections 2024-04-26T00:23:36.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.570+0000 7fbe9faf8640 1 -- 172.21.15.37:0/1265491100 >> 172.21.15.37:0/1265491100 conn(0x7fbe980a1d50 msgr2=0x7fbe980a3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:23:36.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.570+0000 7fbe9faf8640 1 -- 172.21.15.37:0/1265491100 shutdown_connections 2024-04-26T00:23:36.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:36.570+0000 7fbe9faf8640 1 -- 172.21.15.37:0/1265491100 wait complete. 2024-04-26T00:23:36.568 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:23:36.568 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-04-26T00:23:36.909 INFO:tasks.rados.rados.0.smithi045.stdout:628: done (0 left) 2024-04-26T00:23:36.909 INFO:tasks.rados.rados.0.smithi045.stdout:629: delete oid 42 current snap is 70 2024-04-26T00:23:36.925 INFO:tasks.rados.rados.0.smithi045.stdout:629: done (0 left) 2024-04-26T00:23:36.925 INFO:tasks.rados.rados.0.smithi045.stdout:630: read oid 19 snap -1 2024-04-26T00:23:36.925 INFO:tasks.rados.rados.0.smithi045.stdout:630: expect (ObjNum 251 snap 64 seq_num 251) 2024-04-26T00:23:36.925 INFO:tasks.rados.rados.0.smithi045.stdout:631: snap_remove snap 68 2024-04-26T00:23:37.926 INFO:tasks.rados.rados.0.smithi045.stdout:631: done (1 left) 2024-04-26T00:23:37.926 INFO:tasks.rados.rados.0.smithi045.stdout:632: snap_create 2024-04-26T00:23:37.926 INFO:tasks.rados.rados.0.smithi045.stdout:630: expect (ObjNum 251 snap 64 seq_num 251) 2024-04-26T00:23:38.929 INFO:tasks.rados.rados.0.smithi045.stdout:630: done (1 left) 2024-04-26T00:23:38.929 INFO:tasks.rados.rados.0.smithi045.stdout:632: done (0 left) 2024-04-26T00:23:38.930 INFO:tasks.rados.rados.0.smithi045.stdout:633: copy_from oid 8 from oid 28 current snap is 71 2024-04-26T00:23:38.930 INFO:tasks.rados.rados.0.smithi045.stdout:634: delete oid 45 current snap is 71 2024-04-26T00:23:38.938 INFO:tasks.rados.rados.0.smithi045.stdout:634: done (1 left) 2024-04-26T00:23:38.938 INFO:tasks.rados.rados.0.smithi045.stdout:635: read oid 12 snap -1 2024-04-26T00:23:38.939 INFO:tasks.rados.rados.0.smithi045.stdout:635: expect (ObjNum 237 snap 60 seq_num 237) 2024-04-26T00:23:38.939 INFO:tasks.rados.rados.0.smithi045.stdout:636: snap_remove snap 59 2024-04-26T00:23:39.734 INFO:tasks.rados.rados.0.smithi045.stdout:636: done (2 left) 2024-04-26T00:23:39.734 INFO:tasks.rados.rados.0.smithi045.stdout:637: read oid 18 snap -1 2024-04-26T00:23:39.734 INFO:tasks.rados.rados.0.smithi045.stdout:637: expect (ObjNum 190 snap 49 seq_num 190) 2024-04-26T00:23:39.735 INFO:tasks.rados.rados.0.smithi045.stdout:638: rollback oid 20 current snap is 71 2024-04-26T00:23:39.735 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 20 to 70 2024-04-26T00:23:39.736 INFO:tasks.rados.rados.0.smithi045.stdout:639: write oid 4 current snap is 71 2024-04-26T00:23:39.736 INFO:tasks.rados.rados.0.smithi045.stdout:639: seq_num 263 ranges {420241=546489,1373823=544414} 2024-04-26T00:23:39.741 INFO:tasks.rados.rados.0.smithi045.stdout:639: writing smithi04526579-4 from 420241 to 966730 tid 1 2024-04-26T00:23:39.747 INFO:tasks.rados.rados.0.smithi045.stdout:639: writing smithi04526579-4 from 1373823 to 1918237 tid 2 2024-04-26T00:23:39.748 INFO:tasks.rados.rados.0.smithi045.stdout:640: snap_remove snap 61 2024-04-26T00:23:40.745 INFO:tasks.rados.rados.0.smithi045.stdout:640: done (5 left) 2024-04-26T00:23:40.745 INFO:tasks.rados.rados.0.smithi045.stdout:641: read oid 24 snap 56 2024-04-26T00:23:40.746 INFO:tasks.rados.rados.0.smithi045.stdout:641: expect (ObjNum 161 snap 36 seq_num 161) 2024-04-26T00:23:40.746 INFO:tasks.rados.rados.0.smithi045.stdout:635: expect (ObjNum 237 snap 60 seq_num 237) 2024-04-26T00:23:41.042 INFO:tasks.rados.rados.0.smithi045.stdout:633: finishing copy_from racing read to smithi04526579-8 2024-04-26T00:23:41.042 INFO:tasks.rados.rados.0.smithi045.stdout:633: finishing copy_from to smithi04526579-8 2024-04-26T00:23:41.042 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 290 (ObjNum 180 snap 47 seq_num 180) dirty exists 2024-04-26T00:23:41.042 INFO:tasks.rados.rados.0.smithi045.stdout:638: finishing rollback tid 0 to smithi04526579-20 2024-04-26T00:23:41.042 INFO:tasks.rados.rados.0.smithi045.stdout:637: expect (ObjNum 190 snap 49 seq_num 190) 2024-04-26T00:23:41.246 INFO:tasks.rados.rados.0.smithi045.stdout:638: finishing rollback tid 1 to smithi04526579-20 2024-04-26T00:23:41.247 INFO:tasks.rados.rados.0.smithi045.stdout:638: finishing rollback tid 2 to smithi04526579-20 2024-04-26T00:23:41.247 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 222 (ObjNum 249 snap 64 seq_num 249) dirty exists 2024-04-26T00:23:41.247 INFO:tasks.rados.rados.0.smithi045.stdout:639: finishing write tid 1 to smithi04526579-4 2024-04-26T00:23:41.247 INFO:tasks.rados.rados.0.smithi045.stdout:639: finishing write tid 2 to smithi04526579-4 2024-04-26T00:23:41.247 INFO:tasks.rados.rados.0.smithi045.stdout:639: finishing write tid 3 to smithi04526579-4 2024-04-26T00:23:41.247 INFO:tasks.rados.rados.0.smithi045.stdout:639: finishing write tid 4 to smithi04526579-4 2024-04-26T00:23:41.247 INFO:tasks.rados.rados.0.smithi045.stdout:639: oid 4 updating version 0 to 289 2024-04-26T00:23:41.247 INFO:tasks.rados.rados.0.smithi045.stdout:639: oid 4 version 289 is already newer than 288 2024-04-26T00:23:41.247 INFO:tasks.rados.rados.0.smithi045.stdout:639: oid 4 version 289 is already newer than 287 2024-04-26T00:23:41.247 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 289 (ObjNum 263 snap 71 seq_num 263) dirty exists 2024-04-26T00:23:41.247 INFO:tasks.rados.rados.0.smithi045.stdout:639: left oid 4 (ObjNum 263 snap 71 seq_num 263) 2024-04-26T00:23:41.247 INFO:tasks.rados.rados.0.smithi045.stdout:641: expect (ObjNum 161 snap 36 seq_num 161) 2024-04-26T00:23:41.466 INFO:tasks.rados.rados.0.smithi045.stdout:633: done (5 left) 2024-04-26T00:23:41.466 INFO:tasks.rados.rados.0.smithi045.stdout:635: done (4 left) 2024-04-26T00:23:41.466 INFO:tasks.rados.rados.0.smithi045.stdout:637: done (3 left) 2024-04-26T00:23:41.466 INFO:tasks.rados.rados.0.smithi045.stdout:638: done (2 left) 2024-04-26T00:23:41.466 INFO:tasks.rados.rados.0.smithi045.stdout:639: done (1 left) 2024-04-26T00:23:41.466 INFO:tasks.rados.rados.0.smithi045.stdout:641: done (0 left) 2024-04-26T00:23:41.466 INFO:tasks.rados.rados.0.smithi045.stdout:642: snap_create 2024-04-26T00:23:41.962 INFO:tasks.rados.rados.0.smithi045.stdout:642: done (0 left) 2024-04-26T00:23:41.963 INFO:tasks.rados.rados.0.smithi045.stdout:643: snap_create 2024-04-26T00:23:42.966 INFO:tasks.rados.rados.0.smithi045.stdout:643: done (0 left) 2024-04-26T00:23:42.966 INFO:tasks.rados.rados.0.smithi045.stdout:644: rollback oid 3 current snap is 73 2024-04-26T00:23:42.966 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 3 to 66 2024-04-26T00:23:42.966 INFO:tasks.rados.rados.0.smithi045.stdout:645: read oid 30 snap -1 2024-04-26T00:23:42.966 INFO:tasks.rados.rados.0.smithi045.stdout:645: expect (ObjNum 149 snap 32 seq_num 149) 2024-04-26T00:23:42.966 INFO:tasks.rados.rados.0.smithi045.stdout:646: snap_create 2024-04-26T00:23:42.973 INFO:tasks.rados.rados.0.smithi045.stdout:645: expect (ObjNum 149 snap 32 seq_num 149) 2024-04-26T00:23:43.110 INFO:tasks.rados.rados.0.smithi045.stdout:644: finishing rollback tid 0 to smithi04526579-3 2024-04-26T00:23:43.110 INFO:tasks.rados.rados.0.smithi045.stdout:644: finishing rollback tid 1 to smithi04526579-3 2024-04-26T00:23:43.110 INFO:tasks.rados.rados.0.smithi045.stdout:644: finishing rollback tid 2 to smithi04526579-3 2024-04-26T00:23:43.110 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 302 (ObjNum 119 snap 28 seq_num 119) dirty exists 2024-04-26T00:23:43.979 INFO:tasks.rados.rados.0.smithi045.stdout:644: done (2 left) 2024-04-26T00:23:43.979 INFO:tasks.rados.rados.0.smithi045.stdout:645: done (1 left) 2024-04-26T00:23:43.979 INFO:tasks.rados.rados.0.smithi045.stdout:646: done (0 left) 2024-04-26T00:23:43.979 INFO:tasks.rados.rados.0.smithi045.stdout:647: delete oid 31 current snap is 74 2024-04-26T00:23:43.991 INFO:tasks.rados.rados.0.smithi045.stdout:647: done (0 left) 2024-04-26T00:23:43.991 INFO:tasks.rados.rados.0.smithi045.stdout:648: write oid 2 current snap is 74 2024-04-26T00:23:43.991 INFO:tasks.rados.rados.0.smithi045.stdout:648: seq_num 265 ranges {453855=696987,1890787=548083} 2024-04-26T00:23:44.010 INFO:tasks.rados.rados.0.smithi045.stdout:648: writing smithi04526579-2 from 453855 to 1150842 tid 1 2024-04-26T00:23:44.031 INFO:tasks.rados.rados.0.smithi045.stdout:648: writing smithi04526579-2 from 1890787 to 2438870 tid 2 2024-04-26T00:23:44.031 INFO:tasks.rados.rados.0.smithi045.stdout:649: snap_create 2024-04-26T00:23:44.031 INFO:tasks.rados.rados.0.smithi045.stdout:648: finishing write tid 1 to smithi04526579-2 2024-04-26T00:23:44.040 INFO:tasks.rados.rados.0.smithi045.stdout:648: finishing write tid 2 to smithi04526579-2 2024-04-26T00:23:44.041 INFO:tasks.rados.rados.0.smithi045.stdout:648: finishing write tid 3 to smithi04526579-2 2024-04-26T00:23:44.041 INFO:tasks.rados.rados.0.smithi045.stdout:648: finishing write tid 4 to smithi04526579-2 2024-04-26T00:23:44.041 INFO:tasks.rados.rados.0.smithi045.stdout:648: oid 2 updating version 0 to 299 2024-04-26T00:23:44.042 INFO:tasks.rados.rados.0.smithi045.stdout:648: oid 2 version 299 is already newer than 298 2024-04-26T00:23:44.042 INFO:tasks.rados.rados.0.smithi045.stdout:648: oid 2 version 299 is already newer than 297 2024-04-26T00:23:44.042 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 299 (ObjNum 265 snap 74 seq_num 265) dirty exists 2024-04-26T00:23:44.042 INFO:tasks.rados.rados.0.smithi045.stdout:648: left oid 2 (ObjNum 265 snap 74 seq_num 265) 2024-04-26T00:23:44.733 INFO:tasks.rados.rados.0.smithi045.stdout:648: done (1 left) 2024-04-26T00:23:44.733 INFO:tasks.rados.rados.0.smithi045.stdout:649: done (0 left) 2024-04-26T00:23:44.733 INFO:tasks.rados.rados.0.smithi045.stdout:650: write oid 21 current snap is 75 2024-04-26T00:23:44.734 INFO:tasks.rados.rados.0.smithi045.stdout:650: seq_num 266 ranges {470662=772236,1832916=408094} 2024-04-26T00:23:44.753 INFO:tasks.rados.rados.0.smithi045.stdout:650: writing smithi04526579-21 from 470662 to 1242898 tid 1 2024-04-26T00:23:44.771 INFO:tasks.rados.rados.0.smithi045.stdout:650: writing smithi04526579-21 from 1832916 to 2241010 tid 2 2024-04-26T00:23:44.771 INFO:tasks.rados.rados.0.smithi045.stdout:651: write oid 14 current snap is 75 2024-04-26T00:23:44.771 INFO:tasks.rados.rados.0.smithi045.stdout:651: seq_num 267 ranges {487469=563838,1607751=551752} 2024-04-26T00:23:44.787 INFO:tasks.rados.rados.0.smithi045.stdout:651: writing smithi04526579-14 from 487469 to 1051307 tid 1 2024-04-26T00:23:44.801 INFO:tasks.rados.rados.0.smithi045.stdout:651: writing smithi04526579-14 from 1607751 to 2159503 tid 2 2024-04-26T00:23:44.801 INFO:tasks.rados.rados.0.smithi045.stdout:652: snap_create 2024-04-26T00:23:44.801 INFO:tasks.rados.rados.0.smithi045.stdout:650: finishing write tid 1 to smithi04526579-21 2024-04-26T00:23:44.801 INFO:tasks.rados.rados.0.smithi045.stdout:650: finishing write tid 2 to smithi04526579-21 2024-04-26T00:23:44.801 INFO:tasks.rados.rados.0.smithi045.stdout:650: finishing write tid 3 to smithi04526579-21 2024-04-26T00:23:44.801 INFO:tasks.rados.rados.0.smithi045.stdout:650: finishing write tid 4 to smithi04526579-21 2024-04-26T00:23:44.802 INFO:tasks.rados.rados.0.smithi045.stdout:650: oid 21 updating version 0 to 22 2024-04-26T00:23:44.802 INFO:tasks.rados.rados.0.smithi045.stdout:650: oid 21 version 22 is already newer than 21 2024-04-26T00:23:44.802 INFO:tasks.rados.rados.0.smithi045.stdout:650: oid 21 updating version 22 to 23 2024-04-26T00:23:44.802 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 23 (ObjNum 266 snap 75 seq_num 266) dirty exists 2024-04-26T00:23:44.802 INFO:tasks.rados.rados.0.smithi045.stdout:650: left oid 21 (ObjNum 266 snap 75 seq_num 266) 2024-04-26T00:23:44.809 INFO:tasks.rados.rados.0.smithi045.stdout:651: finishing write tid 1 to smithi04526579-14 2024-04-26T00:23:44.810 INFO:tasks.rados.rados.0.smithi045.stdout:651: finishing write tid 2 to smithi04526579-14 2024-04-26T00:23:44.811 INFO:tasks.rados.rados.0.smithi045.stdout:651: finishing write tid 3 to smithi04526579-14 2024-04-26T00:23:44.811 INFO:tasks.rados.rados.0.smithi045.stdout:651: finishing write tid 4 to smithi04526579-14 2024-04-26T00:23:44.812 INFO:tasks.rados.rados.0.smithi045.stdout:651: oid 14 updating version 0 to 127 2024-04-26T00:23:44.812 INFO:tasks.rados.rados.0.smithi045.stdout:651: oid 14 version 127 is already newer than 126 2024-04-26T00:23:44.812 INFO:tasks.rados.rados.0.smithi045.stdout:651: oid 14 updating version 127 to 128 2024-04-26T00:23:44.812 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 128 (ObjNum 267 snap 75 seq_num 267) dirty exists 2024-04-26T00:23:44.812 INFO:tasks.rados.rados.0.smithi045.stdout:651: left oid 14 (ObjNum 267 snap 75 seq_num 267) 2024-04-26T00:23:45.736 INFO:tasks.rados.rados.0.smithi045.stdout:650: done (2 left) 2024-04-26T00:23:45.736 INFO:tasks.rados.rados.0.smithi045.stdout:651: done (1 left) 2024-04-26T00:23:45.737 INFO:tasks.rados.rados.0.smithi045.stdout:652: done (0 left) 2024-04-26T00:23:45.737 INFO:tasks.rados.rados.0.smithi045.stdout:653: read oid 10 snap 45 2024-04-26T00:23:45.737 INFO:tasks.rados.rados.0.smithi045.stdout:653: expect (ObjNum 87 snap 19 seq_num 87) 2024-04-26T00:23:45.737 INFO:tasks.rados.rados.0.smithi045.stdout:654: delete oid 40 current snap is 76 2024-04-26T00:23:45.751 INFO:tasks.rados.rados.0.smithi045.stdout:654: done (1 left) 2024-04-26T00:23:45.751 INFO:tasks.rados.rados.0.smithi045.stdout:655: delete oid 8 current snap is 76 2024-04-26T00:23:45.755 INFO:tasks.rados.rados.0.smithi045.stdout:653: expect (ObjNum 87 snap 19 seq_num 87) 2024-04-26T00:23:45.958 INFO:tasks.rados.rados.0.smithi045.stdout:653: done (1 left) 2024-04-26T00:23:45.959 INFO:tasks.rados.rados.0.smithi045.stdout:655: done (0 left) 2024-04-26T00:23:45.959 INFO:tasks.rados.rados.0.smithi045.stdout:656: snap_remove snap 45 2024-04-26T00:23:46.743 INFO:tasks.rados.rados.0.smithi045.stdout:656: done (0 left) 2024-04-26T00:23:46.744 INFO:tasks.rados.rados.0.smithi045.stdout:657: copy_from oid 19 from oid 41 current snap is 76 2024-04-26T00:23:46.744 INFO:tasks.rados.rados.0.smithi045.stdout:658: write (excl) oid 50 current snap is 76 2024-04-26T00:23:46.744 INFO:tasks.rados.rados.0.smithi045.stdout:658: seq_num 270 ranges {537890=789585,1783197=699079} 2024-04-26T00:23:46.770 INFO:tasks.rados.rados.0.smithi045.stdout:658: writing smithi04526579-50 from 537890 to 1327475 tid 1 2024-04-26T00:23:46.791 INFO:tasks.rados.rados.0.smithi045.stdout:658: writing smithi04526579-50 from 1783197 to 2482276 tid 2 2024-04-26T00:23:46.791 INFO:tasks.rados.rados.0.smithi045.stdout:659: delete oid 5 current snap is 76 2024-04-26T00:23:46.791 INFO:tasks.rados.rados.0.smithi045.stdout:658: finishing write tid 1 to smithi04526579-50 2024-04-26T00:23:46.791 INFO:tasks.rados.rados.0.smithi045.stdout:657: finishing copy_from racing read to smithi04526579-19 2024-04-26T00:23:46.794 INFO:tasks.rados.rados.0.smithi045.stdout:657: finishing copy_from to smithi04526579-19 2024-04-26T00:23:46.795 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 270 (ObjNum 224 snap 59 seq_num 224) dirty exists 2024-04-26T00:23:46.798 INFO:tasks.rados.rados.0.smithi045.stdout:658: finishing write tid 2 to smithi04526579-50 2024-04-26T00:23:46.799 INFO:tasks.rados.rados.0.smithi045.stdout:658: finishing write tid 3 to smithi04526579-50 2024-04-26T00:23:46.800 INFO:tasks.rados.rados.0.smithi045.stdout:658: finishing write tid 4 to smithi04526579-50 2024-04-26T00:23:46.800 INFO:tasks.rados.rados.0.smithi045.stdout:658: oid 50 updating version 0 to 264 2024-04-26T00:23:46.800 INFO:tasks.rados.rados.0.smithi045.stdout:658: oid 50 updating version 264 to 265 2024-04-26T00:23:46.800 INFO:tasks.rados.rados.0.smithi045.stdout:658: oid 50 updating version 265 to 266 2024-04-26T00:23:46.801 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 266 (ObjNum 270 snap 76 seq_num 270) dirty exists 2024-04-26T00:23:46.801 INFO:tasks.rados.rados.0.smithi045.stdout:658: left oid 50 (ObjNum 270 snap 76 seq_num 270) 2024-04-26T00:23:46.802 INFO:tasks.rados.rados.0.smithi045.stdout:657: done (2 left) 2024-04-26T00:23:46.802 INFO:tasks.rados.rados.0.smithi045.stdout:658: done (1 left) 2024-04-26T00:23:46.802 INFO:tasks.rados.rados.0.smithi045.stdout:659: done (0 left) 2024-04-26T00:23:46.802 INFO:tasks.rados.rados.0.smithi045.stdout:660: write (excl) oid 29 current snap is 76 2024-04-26T00:23:46.802 INFO:tasks.rados.rados.0.smithi045.stdout:660: seq_num 272 ranges {571504=540083,1783808=702748} 2024-04-26T00:23:46.817 INFO:tasks.rados.rados.0.smithi045.stdout:660: writing smithi04526579-29 from 571504 to 1111587 tid 1 2024-04-26T00:23:46.837 INFO:tasks.rados.rados.0.smithi045.stdout:660: writing smithi04526579-29 from 1783808 to 2486556 tid 2 2024-04-26T00:23:46.838 INFO:tasks.rados.rados.0.smithi045.stdout:661: read oid 50 snap -1 2024-04-26T00:23:46.838 INFO:tasks.rados.rados.0.smithi045.stdout:661: expect (ObjNum 270 snap 76 seq_num 270) 2024-04-26T00:23:46.838 INFO:tasks.rados.rados.0.smithi045.stdout:662: rollback oid 47 current snap is 76 2024-04-26T00:23:46.838 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 47 to 63 2024-04-26T00:23:46.838 INFO:tasks.rados.rados.0.smithi045.stdout:660: finishing write tid 1 to smithi04526579-29 2024-04-26T00:23:46.838 INFO:tasks.rados.rados.0.smithi045.stdout:663: write oid 36 current snap is 76 2024-04-26T00:23:46.838 INFO:tasks.rados.rados.0.smithi045.stdout:663: seq_num 273 ranges {588311=615332,1842290=562759} 2024-04-26T00:23:46.852 INFO:tasks.rados.rados.0.smithi045.stdout:663: writing smithi04526579-36 from 588311 to 1203643 tid 1 2024-04-26T00:23:46.864 INFO:tasks.rados.rados.0.smithi045.stdout:663: writing smithi04526579-36 from 1842290 to 2405049 tid 2 2024-04-26T00:23:46.864 INFO:tasks.rados.rados.0.smithi045.stdout:664: read oid 11 snap 56 2024-04-26T00:23:46.864 INFO:tasks.rados.rados.0.smithi045.stdout:664: expect (ObjNum 155 snap 33 seq_num 155) 2024-04-26T00:23:46.864 INFO:tasks.rados.rados.0.smithi045.stdout:665: copy_from oid 4 from oid 28 current snap is 76 2024-04-26T00:23:46.864 INFO:tasks.rados.rados.0.smithi045.stdout:662: finishing rollback tid 0 to smithi04526579-47 2024-04-26T00:23:46.864 INFO:tasks.rados.rados.0.smithi045.stdout:661: expect (ObjNum 270 snap 76 seq_num 270) 2024-04-26T00:23:47.070 INFO:tasks.rados.rados.0.smithi045.stdout:660: finishing write tid 2 to smithi04526579-29 2024-04-26T00:23:47.070 INFO:tasks.rados.rados.0.smithi045.stdout:662: finishing rollback tid 1 to smithi04526579-47 2024-04-26T00:23:47.070 INFO:tasks.rados.rados.0.smithi045.stdout:662: finishing rollback tid 2 to smithi04526579-47 2024-04-26T00:23:47.070 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 316 (ObjNum 205 snap 53 seq_num 205) dirty exists 2024-04-26T00:23:47.070 INFO:tasks.rados.rados.0.smithi045.stdout:660: finishing write tid 3 to smithi04526579-29 2024-04-26T00:23:47.070 INFO:tasks.rados.rados.0.smithi045.stdout:660: finishing write tid 4 to smithi04526579-29 2024-04-26T00:23:47.071 INFO:tasks.rados.rados.0.smithi045.stdout:660: oid 29 updating version 0 to 292 2024-04-26T00:23:47.071 INFO:tasks.rados.rados.0.smithi045.stdout:660: oid 29 updating version 292 to 294 2024-04-26T00:23:47.071 INFO:tasks.rados.rados.0.smithi045.stdout:660: oid 29 version 294 is already newer than 293 2024-04-26T00:23:47.071 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 294 (ObjNum 272 snap 76 seq_num 272) dirty exists 2024-04-26T00:23:47.071 INFO:tasks.rados.rados.0.smithi045.stdout:660: left oid 29 (ObjNum 272 snap 76 seq_num 272) 2024-04-26T00:23:47.071 INFO:tasks.rados.rados.0.smithi045.stdout:663: finishing write tid 1 to smithi04526579-36 2024-04-26T00:23:47.071 INFO:tasks.rados.rados.0.smithi045.stdout:663: finishing write tid 2 to smithi04526579-36 2024-04-26T00:23:47.071 INFO:tasks.rados.rados.0.smithi045.stdout:663: finishing write tid 3 to smithi04526579-36 2024-04-26T00:23:47.071 INFO:tasks.rados.rados.0.smithi045.stdout:663: finishing write tid 4 to smithi04526579-36 2024-04-26T00:23:47.071 INFO:tasks.rados.rados.0.smithi045.stdout:663: oid 36 updating version 0 to 151 2024-04-26T00:23:47.071 INFO:tasks.rados.rados.0.smithi045.stdout:663: oid 36 updating version 151 to 152 2024-04-26T00:23:47.071 INFO:tasks.rados.rados.0.smithi045.stdout:663: oid 36 updating version 152 to 153 2024-04-26T00:23:47.072 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 153 (ObjNum 273 snap 76 seq_num 273) dirty exists 2024-04-26T00:23:47.072 INFO:tasks.rados.rados.0.smithi045.stdout:663: left oid 36 (ObjNum 273 snap 76 seq_num 273) 2024-04-26T00:23:47.072 INFO:tasks.rados.rados.0.smithi045.stdout:664: expect (ObjNum 155 snap 33 seq_num 155) 2024-04-26T00:23:47.155 INFO:tasks.rados.rados.0.smithi045.stdout:665: finishing copy_from racing read to smithi04526579-4 2024-04-26T00:23:47.155 INFO:tasks.rados.rados.0.smithi045.stdout:665: finishing copy_from to smithi04526579-4 2024-04-26T00:23:47.155 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 290 (ObjNum 180 snap 47 seq_num 180) dirty exists 2024-04-26T00:23:47.155 INFO:tasks.rados.rados.0.smithi045.stdout:660: done (5 left) 2024-04-26T00:23:47.155 INFO:tasks.rados.rados.0.smithi045.stdout:661: done (4 left) 2024-04-26T00:23:47.156 INFO:tasks.rados.rados.0.smithi045.stdout:662: done (3 left) 2024-04-26T00:23:47.156 INFO:tasks.rados.rados.0.smithi045.stdout:663: done (2 left) 2024-04-26T00:23:47.156 INFO:tasks.rados.rados.0.smithi045.stdout:664: done (1 left) 2024-04-26T00:23:47.156 INFO:tasks.rados.rados.0.smithi045.stdout:665: done (0 left) 2024-04-26T00:23:47.156 INFO:tasks.rados.rados.0.smithi045.stdout:666: delete oid 31 current snap is 76 2024-04-26T00:23:47.159 INFO:tasks.rados.rados.0.smithi045.stdout:666: done (0 left) 2024-04-26T00:23:47.159 INFO:tasks.rados.rados.0.smithi045.stdout:667: snap_create 2024-04-26T00:23:48.473 INFO:tasks.rados.rados.0.smithi045.stdout:667: done (0 left) 2024-04-26T00:23:48.473 INFO:tasks.rados.rados.0.smithi045.stdout:668: write (excl) oid 17 current snap is 77 2024-04-26T00:23:48.473 INFO:tasks.rados.rados.0.smithi045.stdout:668: seq_num 275 ranges {621925=482183,1559254=566428} 2024-04-26T00:23:48.491 INFO:tasks.rados.rados.0.smithi045.stdout:668: writing smithi04526579-17 from 621925 to 1104108 tid 1 2024-04-26T00:23:48.508 INFO:tasks.rados.rados.0.smithi045.stdout:668: writing smithi04526579-17 from 1559254 to 2125682 tid 2 2024-04-26T00:23:48.508 INFO:tasks.rados.rados.0.smithi045.stdout:669: delete oid 25 current snap is 77 2024-04-26T00:23:48.508 INFO:tasks.rados.rados.0.smithi045.stdout:668: finishing write tid 1 to smithi04526579-17 2024-04-26T00:23:48.515 INFO:tasks.rados.rados.0.smithi045.stdout:668: finishing write tid 2 to smithi04526579-17 2024-04-26T00:23:48.515 INFO:tasks.rados.rados.0.smithi045.stdout:668: finishing write tid 3 to smithi04526579-17 2024-04-26T00:23:48.516 INFO:tasks.rados.rados.0.smithi045.stdout:668: finishing write tid 4 to smithi04526579-17 2024-04-26T00:23:48.516 INFO:tasks.rados.rados.0.smithi045.stdout:668: oid 17 updating version 0 to 268 2024-04-26T00:23:48.516 INFO:tasks.rados.rados.0.smithi045.stdout:668: oid 17 version 268 is already newer than 267 2024-04-26T00:23:48.516 INFO:tasks.rados.rados.0.smithi045.stdout:668: oid 17 updating version 268 to 269 2024-04-26T00:23:48.516 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 269 (ObjNum 275 snap 77 seq_num 275) dirty exists 2024-04-26T00:23:48.516 INFO:tasks.rados.rados.0.smithi045.stdout:668: left oid 17 (ObjNum 275 snap 77 seq_num 275) 2024-04-26T00:23:48.517 INFO:tasks.rados.rados.0.smithi045.stdout:668: done (1 left) 2024-04-26T00:23:48.517 INFO:tasks.rados.rados.0.smithi045.stdout:669: done (0 left) 2024-04-26T00:23:48.517 INFO:tasks.rados.rados.0.smithi045.stdout:670: write oid 47 current snap is 77 2024-04-26T00:23:48.517 INFO:tasks.rados.rados.0.smithi045.stdout:670: seq_num 277 ranges {655539=632681,2076218=570097} 2024-04-26T00:23:48.537 INFO:tasks.rados.rados.0.smithi045.stdout:670: writing smithi04526579-47 from 655539 to 1288220 tid 1 2024-04-26T00:23:48.559 INFO:tasks.rados.rados.0.smithi045.stdout:670: writing smithi04526579-47 from 2076218 to 2646315 tid 2 2024-04-26T00:23:48.559 INFO:tasks.rados.rados.0.smithi045.stdout:671: copy_from oid 8 from oid 50 current snap is 77 2024-04-26T00:23:48.559 INFO:tasks.rados.rados.0.smithi045.stdout:670: finishing write tid 1 to smithi04526579-47 2024-04-26T00:23:48.559 INFO:tasks.rados.rados.0.smithi045.stdout:672: delete oid 43 current snap is 77 2024-04-26T00:23:48.566 INFO:tasks.rados.rados.0.smithi045.stdout:672: done (2 left) 2024-04-26T00:23:48.566 INFO:tasks.rados.rados.0.smithi045.stdout:673: write (excl) oid 4 current snap is 77 2024-04-26T00:23:48.567 INFO:tasks.rados.rados.0.smithi045.stdout:673: seq_num 279 ranges {689153=783179,2076829=457413} 2024-04-26T00:23:48.584 INFO:tasks.rados.rados.0.smithi045.stdout:673: writing smithi04526579-4 from 689153 to 1472332 tid 1 2024-04-26T00:23:48.598 INFO:tasks.rados.rados.0.smithi045.stdout:673: writing smithi04526579-4 from 2076829 to 2534242 tid 2 2024-04-26T00:23:48.598 INFO:tasks.rados.rados.0.smithi045.stdout:674: rollback oid 13 current snap is 77 2024-04-26T00:23:48.598 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 13 to 75 2024-04-26T00:23:48.598 INFO:tasks.rados.rados.0.smithi045.stdout:670: finishing write tid 2 to smithi04526579-47 2024-04-26T00:23:48.598 INFO:tasks.rados.rados.0.smithi045.stdout:670: finishing write tid 3 to smithi04526579-47 2024-04-26T00:23:48.599 INFO:tasks.rados.rados.0.smithi045.stdout:670: finishing write tid 4 to smithi04526579-47 2024-04-26T00:23:48.599 INFO:tasks.rados.rados.0.smithi045.stdout:670: oid 47 updating version 0 to 318 2024-04-26T00:23:48.599 INFO:tasks.rados.rados.0.smithi045.stdout:670: oid 47 updating version 318 to 320 2024-04-26T00:23:48.599 INFO:tasks.rados.rados.0.smithi045.stdout:670: oid 47 version 320 is already newer than 319 2024-04-26T00:23:48.599 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 320 (ObjNum 277 snap 77 seq_num 277) dirty exists 2024-04-26T00:23:48.599 INFO:tasks.rados.rados.0.smithi045.stdout:670: left oid 47 (ObjNum 277 snap 77 seq_num 277) 2024-04-26T00:23:48.599 INFO:tasks.rados.rados.0.smithi045.stdout:671: finishing copy_from racing read to smithi04526579-8 2024-04-26T00:23:48.599 INFO:tasks.rados.rados.0.smithi045.stdout:671: finishing copy_from to smithi04526579-8 2024-04-26T00:23:48.599 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 266 (ObjNum 270 snap 76 seq_num 270) dirty exists 2024-04-26T00:23:48.599 INFO:tasks.rados.rados.0.smithi045.stdout:673: finishing write tid 1 to smithi04526579-4 2024-04-26T00:23:48.599 INFO:tasks.rados.rados.0.smithi045.stdout:670: done (3 left) 2024-04-26T00:23:48.599 INFO:tasks.rados.rados.0.smithi045.stdout:671: done (2 left) 2024-04-26T00:23:48.599 INFO:tasks.rados.rados.0.smithi045.stdout:675: rollback oid 31 current snap is 77 2024-04-26T00:23:48.599 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 31 to 51 2024-04-26T00:23:48.599 INFO:tasks.rados.rados.0.smithi045.stdout:676: read oid 25 snap -1 2024-04-26T00:23:48.600 INFO:tasks.rados.rados.0.smithi045.stdout:676: expect deleted 2024-04-26T00:23:48.600 INFO:tasks.rados.rados.0.smithi045.stdout:677: read oid 18 snap -1 2024-04-26T00:23:48.600 INFO:tasks.rados.rados.0.smithi045.stdout:677: expect (ObjNum 190 snap 49 seq_num 190) 2024-04-26T00:23:48.600 INFO:tasks.rados.rados.0.smithi045.stdout:678: snap_remove snap 56 2024-04-26T00:23:49.481 INFO:tasks.rados.rados.0.smithi045.stdout:678: done (5 left) 2024-04-26T00:23:49.481 INFO:tasks.rados.rados.0.smithi045.stdout:679: delete oid 10 current snap is 77 2024-04-26T00:23:49.484 INFO:tasks.rados.rados.0.smithi045.stdout:677: expect (ObjNum 190 snap 49 seq_num 190) 2024-04-26T00:23:49.758 INFO:tasks.rados.rados.0.smithi045.stdout:673: finishing write tid 2 to smithi04526579-4 2024-04-26T00:23:49.759 INFO:tasks.rados.rados.0.smithi045.stdout:673: finishing write tid 3 to smithi04526579-4 2024-04-26T00:23:49.759 INFO:tasks.rados.rados.0.smithi045.stdout:674: finishing rollback tid 0 to smithi04526579-13 2024-04-26T00:23:49.759 INFO:tasks.rados.rados.0.smithi045.stdout:673: finishing write tid 4 to smithi04526579-4 2024-04-26T00:23:49.759 INFO:tasks.rados.rados.0.smithi045.stdout:673: oid 4 updating version 0 to 312 2024-04-26T00:23:49.759 INFO:tasks.rados.rados.0.smithi045.stdout:673: oid 4 updating version 312 to 313 2024-04-26T00:23:49.759 INFO:tasks.rados.rados.0.smithi045.stdout:673: oid 4 updating version 313 to 314 2024-04-26T00:23:49.759 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 314 (ObjNum 279 snap 77 seq_num 279) dirty exists 2024-04-26T00:23:49.759 INFO:tasks.rados.rados.0.smithi045.stdout:673: left oid 4 (ObjNum 279 snap 77 seq_num 279) 2024-04-26T00:23:49.759 INFO:tasks.rados.rados.0.smithi045.stdout:674: finishing rollback tid 1 to smithi04526579-13 2024-04-26T00:23:49.759 INFO:tasks.rados.rados.0.smithi045.stdout:675: finishing rollback tid 1 to smithi04526579-31 2024-04-26T00:23:49.759 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 295 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-04-26T00:23:49.759 INFO:tasks.rados.rados.0.smithi045.stdout:674: finishing rollback tid 2 to smithi04526579-13 2024-04-26T00:23:49.759 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 297 (ObjNum 204 snap 53 seq_num 204) dirty exists 2024-04-26T00:23:49.759 INFO:tasks.rados.rados.0.smithi045.stdout:673: done (5 left) 2024-04-26T00:23:49.760 INFO:tasks.rados.rados.0.smithi045.stdout:674: done (4 left) 2024-04-26T00:23:49.760 INFO:tasks.rados.rados.0.smithi045.stdout:675: done (3 left) 2024-04-26T00:23:49.760 INFO:tasks.rados.rados.0.smithi045.stdout:676: done (2 left) 2024-04-26T00:23:49.760 INFO:tasks.rados.rados.0.smithi045.stdout:677: done (1 left) 2024-04-26T00:23:49.760 INFO:tasks.rados.rados.0.smithi045.stdout:679: done (0 left) 2024-04-26T00:23:49.760 INFO:tasks.rados.rados.0.smithi045.stdout:680: copy_from oid 32 from oid 39 current snap is 77 2024-04-26T00:23:49.760 INFO:tasks.rados.rados.0.smithi045.stdout:681: rollback oid 30 current snap is 77 2024-04-26T00:23:49.760 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 30 to 72 2024-04-26T00:23:49.760 INFO:tasks.rados.rados.0.smithi045.stdout:682: snap_create 2024-04-26T00:23:49.762 INFO:tasks.rados.rados.0.smithi045.stdout:681: finishing rollback tid 0 to smithi04526579-30 2024-04-26T00:23:49.762 INFO:tasks.rados.rados.0.smithi045.stdout:681: finishing rollback tid 1 to smithi04526579-30 2024-04-26T00:23:49.763 INFO:tasks.rados.rados.0.smithi045.stdout:681: finishing rollback tid 2 to smithi04526579-30 2024-04-26T00:23:49.763 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 174 (ObjNum 149 snap 32 seq_num 149) dirty exists 2024-04-26T00:23:49.767 INFO:tasks.rados.rados.0.smithi045.stdout:680: finishing copy_from racing read to smithi04526579-32 2024-04-26T00:23:49.774 INFO:tasks.rados.rados.0.smithi045.stdout:680: finishing copy_from to smithi04526579-32 2024-04-26T00:23:49.774 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 102 (ObjNum 155 snap 33 seq_num 155) dirty exists 2024-04-26T00:23:50.747 INFO:tasks.rados.rados.0.smithi045.stdout:680: done (2 left) 2024-04-26T00:23:50.747 INFO:tasks.rados.rados.0.smithi045.stdout:681: done (1 left) 2024-04-26T00:23:50.747 INFO:tasks.rados.rados.0.smithi045.stdout:682: done (0 left) 2024-04-26T00:23:50.747 INFO:tasks.rados.rados.0.smithi045.stdout:683: rollback oid 21 current snap is 78 2024-04-26T00:23:50.747 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 21 to 69 2024-04-26T00:23:50.747 INFO:tasks.rados.rados.0.smithi045.stdout:684: snap_create 2024-04-26T00:23:50.757 INFO:tasks.rados.rados.0.smithi045.stdout:683: finishing rollback tid 0 to smithi04526579-21 2024-04-26T00:23:50.758 INFO:tasks.rados.rados.0.smithi045.stdout:683: finishing rollback tid 1 to smithi04526579-21 2024-04-26T00:23:50.758 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 26 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-04-26T00:23:51.754 INFO:tasks.rados.rados.0.smithi045.stdout:683: done (1 left) 2024-04-26T00:23:51.754 INFO:tasks.rados.rados.0.smithi045.stdout:684: done (0 left) 2024-04-26T00:23:51.754 INFO:tasks.rados.rados.0.smithi045.stdout:685: read oid 2 snap -1 2024-04-26T00:23:51.754 INFO:tasks.rados.rados.0.smithi045.stdout:685: expect (ObjNum 265 snap 74 seq_num 265) 2024-04-26T00:23:51.754 INFO:tasks.rados.rados.0.smithi045.stdout:686: snap_create 2024-04-26T00:23:51.770 INFO:tasks.rados.rados.0.smithi045.stdout:685: expect (ObjNum 265 snap 74 seq_num 265) 2024-04-26T00:23:52.762 INFO:tasks.rados.rados.0.smithi045.stdout:685: done (1 left) 2024-04-26T00:23:52.762 INFO:tasks.rados.rados.0.smithi045.stdout:686: done (0 left) 2024-04-26T00:23:52.762 INFO:tasks.rados.rados.0.smithi045.stdout:687: write oid 8 current snap is 80 2024-04-26T00:23:52.762 INFO:tasks.rados.rados.0.smithi045.stdout:687: seq_num 281 ranges {722767=533677,1793793=461082} 2024-04-26T00:23:52.782 INFO:tasks.rados.rados.0.smithi045.stdout:687: writing smithi04526579-8 from 722767 to 1256444 tid 1 2024-04-26T00:23:52.799 INFO:tasks.rados.rados.0.smithi045.stdout:687: writing smithi04526579-8 from 1793793 to 2254875 tid 2 2024-04-26T00:23:52.799 INFO:tasks.rados.rados.0.smithi045.stdout:688: delete oid 30 current snap is 80 2024-04-26T00:23:52.799 INFO:tasks.rados.rados.0.smithi045.stdout:687: finishing write tid 1 to smithi04526579-8 2024-04-26T00:23:52.807 INFO:tasks.rados.rados.0.smithi045.stdout:688: done (1 left) 2024-04-26T00:23:52.807 INFO:tasks.rados.rados.0.smithi045.stdout:689: snap_remove snap 55 2024-04-26T00:23:53.772 INFO:tasks.rados.rados.0.smithi045.stdout:689: done (1 left) 2024-04-26T00:23:53.773 INFO:tasks.rados.rados.0.smithi045.stdout:690: snap_create 2024-04-26T00:23:53.773 INFO:tasks.rados.rados.0.smithi045.stdout:687: finishing write tid 2 to smithi04526579-8 2024-04-26T00:23:53.773 INFO:tasks.rados.rados.0.smithi045.stdout:687: finishing write tid 3 to smithi04526579-8 2024-04-26T00:23:53.773 INFO:tasks.rados.rados.0.smithi045.stdout:687: finishing write tid 4 to smithi04526579-8 2024-04-26T00:23:53.773 INFO:tasks.rados.rados.0.smithi045.stdout:687: oid 8 updating version 0 to 298 2024-04-26T00:23:53.773 INFO:tasks.rados.rados.0.smithi045.stdout:687: oid 8 updating version 298 to 299 2024-04-26T00:23:53.773 INFO:tasks.rados.rados.0.smithi045.stdout:687: oid 8 updating version 299 to 300 2024-04-26T00:23:53.773 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 300 (ObjNum 281 snap 80 seq_num 281) dirty exists 2024-04-26T00:23:53.773 INFO:tasks.rados.rados.0.smithi045.stdout:687: left oid 8 (ObjNum 281 snap 80 seq_num 281) 2024-04-26T00:23:54.737 INFO:tasks.rados.rados.0.smithi045.stdout:687: done (1 left) 2024-04-26T00:23:54.737 INFO:tasks.rados.rados.0.smithi045.stdout:690: done (0 left) 2024-04-26T00:23:54.737 INFO:tasks.rados.rados.0.smithi045.stdout:691: write oid 6 current snap is 81 2024-04-26T00:23:54.737 INFO:tasks.rados.rados.0.smithi045.stdout:691: seq_num 283 ranges {756381=400528,1910757=464751} 2024-04-26T00:23:54.756 INFO:tasks.rados.rados.0.smithi045.stdout:691: writing smithi04526579-6 from 756381 to 1156909 tid 1 2024-04-26T00:23:54.776 INFO:tasks.rados.rados.0.smithi045.stdout:691: writing smithi04526579-6 from 1910757 to 2375508 tid 2 2024-04-26T00:23:54.776 INFO:tasks.rados.rados.0.smithi045.stdout:692: write (excl) oid 27 current snap is 81 2024-04-26T00:23:54.776 INFO:tasks.rados.rados.0.smithi045.stdout:691: finishing write tid 1 to smithi04526579-6 2024-04-26T00:23:54.776 INFO:tasks.rados.rados.0.smithi045.stdout:692: seq_num 284 ranges {773188=475777,1969239=724762} 2024-04-26T00:23:54.797 INFO:tasks.rados.rados.0.smithi045.stdout:692: writing smithi04526579-27 from 773188 to 1248965 tid 1 2024-04-26T00:23:54.818 INFO:tasks.rados.rados.0.smithi045.stdout:692: writing smithi04526579-27 from 1969239 to 2694001 tid 2 2024-04-26T00:23:54.818 INFO:tasks.rados.rados.0.smithi045.stdout:693: copy_from oid 12 from oid 28 current snap is 81 2024-04-26T00:23:54.818 INFO:tasks.rados.rados.0.smithi045.stdout:691: finishing write tid 2 to smithi04526579-6 2024-04-26T00:23:54.818 INFO:tasks.rados.rados.0.smithi045.stdout:691: finishing write tid 3 to smithi04526579-6 2024-04-26T00:23:54.818 INFO:tasks.rados.rados.0.smithi045.stdout:691: finishing write tid 4 to smithi04526579-6 2024-04-26T00:23:54.818 INFO:tasks.rados.rados.0.smithi045.stdout:691: oid 6 updating version 0 to 177 2024-04-26T00:23:54.818 INFO:tasks.rados.rados.0.smithi045.stdout:691: oid 6 version 177 is already newer than 176 2024-04-26T00:23:54.818 INFO:tasks.rados.rados.0.smithi045.stdout:691: oid 6 updating version 177 to 178 2024-04-26T00:23:54.818 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 178 (ObjNum 283 snap 81 seq_num 283) dirty exists 2024-04-26T00:23:54.818 INFO:tasks.rados.rados.0.smithi045.stdout:691: left oid 6 (ObjNum 283 snap 81 seq_num 283) 2024-04-26T00:23:54.818 INFO:tasks.rados.rados.0.smithi045.stdout:692: finishing write tid 1 to smithi04526579-27 2024-04-26T00:23:54.818 INFO:tasks.rados.rados.0.smithi045.stdout:691: done (2 left) 2024-04-26T00:23:54.818 INFO:tasks.rados.rados.0.smithi045.stdout:694: snap_remove snap 80 2024-04-26T00:23:55.783 INFO:tasks.rados.rados.0.smithi045.stdout:694: done (2 left) 2024-04-26T00:23:55.783 INFO:tasks.rados.rados.0.smithi045.stdout:695: snap_remove snap 79 2024-04-26T00:23:56.569 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-04-26T00:23:56.569 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-04-26T00:23:56.569 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-26T00:23:56.569 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-26T00:23:56.572 INFO:tasks.ceph.mon.a:Started 2024-04-26T00:23:56.572 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:23:56.572 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:23:56.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.722+0000 7fb35c1a3640 10 monclient: get_monmap_and_config 2024-04-26T00:23:56.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.722+0000 7fb35c1a3640 10 monclient: build_initial_monmap 2024-04-26T00:23:56.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.722+0000 7fb35c1a3640 10 monclient: monmap: 2024-04-26T00:23:56.700 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:23:56.701 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:56.701 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:23:56.726405+0000 2024-04-26T00:23:56.701 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:23:56.726405+0000 2024-04-26T00:23:56.701 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:23:56.701 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:56.701 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:23:56.701 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:23:56.701 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:23:56.701 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:23:56.701 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:23:56.701 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:56.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.722+0000 7fb35c1a3640 1 Processor -- start 2024-04-26T00:23:56.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.722+0000 7fb35c1a3640 1 -- start start 2024-04-26T00:23:56.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.722+0000 7fb35c1a3640 10 monclient: init 2024-04-26T00:23:56.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.722+0000 7fb35c1a3640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:56.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.722+0000 7fb35c1a3640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-04-26T00:23:56.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.722+0000 7fb35c1a3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb3541351c0 0x7fb354134c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:56.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.722+0000 7fb35c1a3640 10 monclient(hunting): picked mon.noname-b con 0x7fb3541351c0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:23:56.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.722+0000 7fb35c1a3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb3541355a0 addr v1:172.21.15.45:6789/0 2024-04-26T00:23:56.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.722+0000 7fb35c1a3640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb3540a2380 0x7fb35413cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:56.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.722+0000 7fb35c1a3640 10 monclient(hunting): picked mon.noname-c con 0x7fb3540a2380 addr v2:172.21.15.45:3300/0 2024-04-26T00:23:56.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.722+0000 7fb35c1a3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb35406b1d0 con 0x7fb3541355a0 2024-04-26T00:23:56.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.722+0000 7fb35c1a3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:56.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.722+0000 7fb35c1a3640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fb35406d310 con 0x7fb3540a2380 2024-04-26T00:23:56.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.722+0000 7fb35c1a3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:56.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.722+0000 7fb35c1a3640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb35409d8e0 con 0x7fb3541351c0 2024-04-26T00:23:56.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.722+0000 7fb35c1a3640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:56.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.722+0000 7fb35c1a3640 10 monclient(hunting): authenticate will time out at 1218.031898s 2024-04-26T00:23:56.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.726+0000 7fb359f3f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb3541351c0 0x7fb354134c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:56.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.726+0000 7fb359f3f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb3541351c0 0x7fb354134c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:53672/0 (socket says 172.21.15.37:53672) 2024-04-26T00:23:56.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.726+0000 7fb359f3f640 1 -- 172.21.15.37:0/676490122 learned_addr learned my addr 172.21.15.37:0/676490122 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:23:56.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.726+0000 7fb359f3f640 10 monclient(hunting): get_auth_request con 0x7fb3541351c0 auth_method 0 2024-04-26T00:23:56.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.726+0000 7fb359f3f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:56.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.726+0000 7fb359f3f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:56.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.726+0000 7fb359f3f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:56.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.726+0000 7fb359f3f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:56.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.726+0000 7fb359f3f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:56.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.726+0000 7fb359f3f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:56.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.726+0000 7fb35a740640 1 --2- 172.21.15.37:0/676490122 >> v2:172.21.15.45:3300/0 conn(0x7fb3540a2380 0x7fb35413cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:56.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.726+0000 7fb35a740640 10 monclient(hunting): get_auth_request con 0x7fb3540a2380 auth_method 0 2024-04-26T00:23:56.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.726+0000 7fb35a740640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:56.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.726+0000 7fb35a740640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:56.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.726+0000 7fb35a740640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:56.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.726+0000 7fb359f3f640 10 monclient(hunting): handle_auth_done global_id 34277 payload 274 2024-04-26T00:23:56.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.726+0000 7fb359f3f640 1 -- 172.21.15.37:0/676490122 >> v2:172.21.15.45:3300/0 conn(0x7fb3540a2380 msgr2=0x7fb35413cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:23:56.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.726+0000 7fb359f3f640 1 --2- 172.21.15.37:0/676490122 >> v2:172.21.15.45:3300/0 conn(0x7fb3540a2380 0x7fb35413cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:56.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.726+0000 7fb359f3f640 1 -- 172.21.15.37:0/676490122 >> v1:172.21.15.45:6789/0 conn(0x7fb3541355a0 legacy=0x7fb3540a1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:23:56.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.726+0000 7fb359f3f640 10 monclient: _finish_hunting 0 2024-04-26T00:23:56.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.726+0000 7fb359f3f640 1 monclient: found mon.noname-b 2024-04-26T00:23:56.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.726+0000 7fb359f3f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:56.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.726+0000 7fb359f3f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:23:56.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.726+0000 7fb359f3f640 1 -- 172.21.15.37:0/676490122 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb35406bb80 con 0x7fb3541351c0 2024-04-26T00:23:56.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb359f3f640 1 -- 172.21.15.37:0/676490122 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb3541351c0 msgr2=0x7fb354134c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:56.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb359f3f640 1 --2- 172.21.15.37:0/676490122 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb3541351c0 0x7fb354134c10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb350009870 tx=0x7fb35001c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a219b76a044cf61b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:56.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb358f3d640 1 -- 172.21.15.37:0/676490122 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3500b0070 con 0x7fb3541351c0 2024-04-26T00:23:56.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb358f3d640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:56.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb358f3d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:23:56.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb358f3d640 10 monclient: dump: 2024-04-26T00:23:56.709 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:56.709 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:56.709 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:56.709 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:56.709 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:56.709 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:56.709 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:56.709 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:56.709 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:56.709 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:56.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb358f3d640 10 monclient: _finish_auth 0 2024-04-26T00:23:56.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb358f3d640 10 monclient: _check_auth_tickets 2024-04-26T00:23:56.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb358f3d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:56.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb358f3d640 1 -- 172.21.15.37:0/676490122 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3500ab600 con 0x7fb3541351c0 2024-04-26T00:23:56.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb358f3d640 10 monclient: handle_config config(0 keys) 2024-04-26T00:23:56.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb358f3d640 1 -- 172.21.15.37:0/676490122 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3500ab9d0 con 0x7fb3541351c0 2024-04-26T00:23:56.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb358f3d640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:56.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb358f3d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:23:56.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb358f3d640 10 monclient: dump: 2024-04-26T00:23:56.710 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:56.710 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:56.710 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:56.710 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:56.710 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:56.710 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:56.710 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:56.710 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:56.710 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:56.710 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:56.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35c1a3640 5 monclient: authenticate success, global_id 34277 2024-04-26T00:23:56.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35c1a3640 10 monclient: get_monmap_and_config success 2024-04-26T00:23:56.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35c1a3640 4 set_mon_vals no callback set 2024-04-26T00:23:56.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35c1a3640 10 monclient: shutdown 2024-04-26T00:23:56.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35c1a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:23:56.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35c1a3640 1 -- 172.21.15.37:0/676490122 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb3541351c0 msgr2=0x7fb354134c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:56.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35c1a3640 1 --2- 172.21.15.37:0/676490122 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb3541351c0 0x7fb354134c10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb350009870 tx=0x7fb35001c3e0 comp rx=0 tx=0).stop 2024-04-26T00:23:56.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35c1a3640 1 -- 172.21.15.37:0/676490122 shutdown_connections 2024-04-26T00:23:56.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35c1a3640 1 --2- 172.21.15.37:0/676490122 >> v2:172.21.15.45:3300/0 conn(0x7fb3540a2380 0x7fb35413cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:56.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35c1a3640 1 --2- 172.21.15.37:0/676490122 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb3541351c0 0x7fb354134c10 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:56.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35c1a3640 1 -- 172.21.15.37:0/676490122 >> 172.21.15.37:0/676490122 conn(0x7fb3540a2d60 msgr2=0x7fb3540a3140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:23:56.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35c1a3640 1 -- 172.21.15.37:0/676490122 shutdown_connections 2024-04-26T00:23:56.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35c1a3640 1 -- 172.21.15.37:0/676490122 wait complete. 2024-04-26T00:23:56.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35c1a3640 10 monclient: build_initial_monmap 2024-04-26T00:23:56.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35c1a3640 10 monclient: monmap: 2024-04-26T00:23:56.712 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:23:56.712 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:23:56.712 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:23:56.712 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:23:56.712 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:23:56.712 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:23:56.712 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:23:56.712 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:23:56.712 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:23:56.712 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:56.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35c1a3640 1 Processor -- start 2024-04-26T00:23:56.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35c1a3640 1 -- start start 2024-04-26T00:23:56.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35c1a3640 10 monclient: init 2024-04-26T00:23:56.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35c1a3640 10 monclient: _reopen_session rank -1 2024-04-26T00:23:56.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35c1a3640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:23:56.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35c1a3640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3540a2380 0x7fb35413b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:56.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35c1a3640 10 monclient(hunting): picked mon.noname-a con 0x7fb3540a2380 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:23:56.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35c1a3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3541351c0 0x7fb354130980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:56.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35c1a3640 10 monclient(hunting): picked mon.noname-b con 0x7fb3541351c0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:23:56.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35c1a3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb3541355a0 0x7fb354130ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:56.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35c1a3640 10 monclient(hunting): picked mon.noname-c con 0x7fb3541355a0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:23:56.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35c1a3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:56.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35c1a3640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb354140040 con 0x7fb3540a2380 2024-04-26T00:23:56.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35c1a3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:56.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35c1a3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb35413fec0 con 0x7fb3541351c0 2024-04-26T00:23:56.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35c1a3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:23:56.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35c1a3640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb3541401c0 con 0x7fb3541355a0 2024-04-26T00:23:56.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35c1a3640 10 monclient(hunting): _renew_subs 2024-04-26T00:23:56.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35c1a3640 10 monclient(hunting): authenticate will time out at 1218.039201s 2024-04-26T00:23:56.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35a740640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb3541355a0 0x7fb354130ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:56.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb359f3f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3540a2380 0x7fb35413b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:56.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb359f3f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3540a2380 0x7fb35413b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51214/0 (socket says 172.21.15.37:51214) 2024-04-26T00:23:56.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35a740640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb3541355a0 0x7fb354130ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:53678/0 (socket says 172.21.15.37:53678) 2024-04-26T00:23:56.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb359f3f640 1 -- 172.21.15.37:0/1655119630 learned_addr learned my addr 172.21.15.37:0/1655119630 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:23:56.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb359f3f640 10 monclient(hunting): get_auth_request con 0x7fb3540a2380 auth_method 0 2024-04-26T00:23:56.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb359f3f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:56.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb359f3f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:56.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb359f3f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:56.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35a740640 10 monclient(hunting): get_auth_request con 0x7fb3541355a0 auth_method 0 2024-04-26T00:23:56.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35a740640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:56.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35a740640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:56.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35a740640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:56.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35a740640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:23:56.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35a740640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:23:56.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb359f3f640 1 -- 172.21.15.37:0/1655119630 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3540a2380 msgr2=0x7fb35413b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:23:56.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb359f3f640 1 -- 172.21.15.37:0/1655119630 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3540a2380 msgr2=0x7fb35413b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:23:56.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb359f3f640 1 --2- 172.21.15.37:0/1655119630 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3540a2380 0x7fb35413b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:23:56.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35a740640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:23:56.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb359f3f640 1 --2- 172.21.15.37:0/1655119630 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3540a2380 0x7fb35413b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:23:56.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35973e640 1 --2- 172.21.15.37:0/1655119630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3541351c0 0x7fb354130980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:56.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35973e640 10 monclient(hunting): get_auth_request con 0x7fb3541351c0 auth_method 0 2024-04-26T00:23:56.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35973e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:23:56.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35973e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:23:56.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35973e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:23:56.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35a740640 10 monclient(hunting): handle_auth_done global_id 34280 payload 824 2024-04-26T00:23:56.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35a740640 1 -- 172.21.15.37:0/1655119630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3541351c0 msgr2=0x7fb354130980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:56.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35a740640 1 --2- 172.21.15.37:0/1655119630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3541351c0 0x7fb354130980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:56.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35a740640 1 -- 172.21.15.37:0/1655119630 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3540a2380 msgr2=0x7fb35413b9c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:23:56.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35a740640 1 --2- 172.21.15.37:0/1655119630 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3540a2380 0x7fb35413b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:56.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35a740640 10 monclient: _finish_hunting 0 2024-04-26T00:23:56.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35a740640 1 monclient: found mon.noname-c 2024-04-26T00:23:56.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35a740640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:23:56.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35a740640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:23:56.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.730+0000 7fb35a740640 1 -- 172.21.15.37:0/1655119630 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb35409fcb0 con 0x7fb3541355a0 2024-04-26T00:23:56.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.734+0000 7fb35a740640 1 -- 172.21.15.37:0/1655119630 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb3541355a0 msgr2=0x7fb354130ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:23:56.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.734+0000 7fb35a740640 1 --2- 172.21.15.37:0/1655119630 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb3541355a0 0x7fb354130ec0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb34c007d90 tx=0x7fb34c00e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:56.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.734+0000 7fb34a7fc640 1 -- 172.21.15.37:0/1655119630 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34c0039a0 con 0x7fb3541355a0 2024-04-26T00:23:56.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.734+0000 7fb34a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:56.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.734+0000 7fb34a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:23:56.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.734+0000 7fb34a7fc640 10 monclient: dump: 2024-04-26T00:23:56.720 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:56.721 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:56.721 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:56.721 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:56.721 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:56.721 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:56.721 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:56.721 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:56.721 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:56.721 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:56.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.734+0000 7fb34a7fc640 10 monclient: _finish_auth 0 2024-04-26T00:23:56.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.734+0000 7fb34a7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:23:56.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.734+0000 7fb34a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:23:56.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.734+0000 7fb34a7fc640 1 -- 172.21.15.37:0/1655119630 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb34c003b60 con 0x7fb3541355a0 2024-04-26T00:23:56.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.734+0000 7fb34a7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:23:56.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.734+0000 7fb34a7fc640 1 -- 172.21.15.37:0/1655119630 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34c004020 con 0x7fb3541355a0 2024-04-26T00:23:56.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.734+0000 7fb35c1a3640 5 monclient: authenticate success, global_id 34280 2024-04-26T00:23:56.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.734+0000 7fb35c1a3640 10 monclient: _renew_subs 2024-04-26T00:23:56.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.734+0000 7fb35c1a3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:56.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.734+0000 7fb35c1a3640 1 -- 172.21.15.37:0/1655119630 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb35409c0d0 con 0x7fb3541355a0 2024-04-26T00:23:56.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.734+0000 7fb34bfff640 4 set_mon_vals no callback set 2024-04-26T00:23:56.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.734+0000 7fb34a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:23:56.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.734+0000 7fb34a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:23:56.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.734+0000 7fb34a7fc640 10 monclient: dump: 2024-04-26T00:23:56.722 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:23:56.722 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:23:56.722 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:56.722 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:23:56.722 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:23:56.722 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:23:56.723 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:23:56.723 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:23:56.723 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:23:56.723 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:23:56.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.734+0000 7fb35c1a3640 10 monclient: _renew_subs 2024-04-26T00:23:56.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.734+0000 7fb35c1a3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:56.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.734+0000 7fb35c1a3640 1 -- 172.21.15.37:0/1655119630 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb35411cd70 con 0x7fb3541355a0 2024-04-26T00:23:56.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.734+0000 7fb34a7fc640 1 -- 172.21.15.37:0/1655119630 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb34c01b020 con 0x7fb3541355a0 2024-04-26T00:23:56.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.734+0000 7fb35c1a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:56.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.734+0000 7fb35c1a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:56.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.734+0000 7fb35c1a3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:56.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.734+0000 7fb35c1a3640 1 -- 172.21.15.37:0/1655119630 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb354066f40 con 0x7fb3541355a0 2024-04-26T00:23:56.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.734+0000 7fb34a7fc640 1 --2- 172.21.15.37:0/1655119630 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb338035ed0 0x7fb338038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:23:56.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.734+0000 7fb34a7fc640 1 -- 172.21.15.37:0/1655119630 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(167..167 src has 1..167) ==== 3943+0+0 (secure 0 0 0) 0x7fb34c048c70 con 0x7fb3541355a0 2024-04-26T00:23:56.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.734+0000 7fb359f3f640 1 --2- 172.21.15.37:0/1655119630 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb338035ed0 0x7fb338038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:23:56.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.734+0000 7fb359f3f640 10 monclient: get_auth_request con 0x7fb338035ed0 auth_method 0 2024-04-26T00:23:56.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.738+0000 7fb359f3f640 1 --2- 172.21.15.37:0/1655119630 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb338035ed0 0x7fb338038390 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb35001a010 tx=0x7fb3500a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:23:56.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.742+0000 7fb34a7fc640 1 -- 172.21.15.37:0/1655119630 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fb35409fcb0 con 0x7fb3541355a0 2024-04-26T00:23:56.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.742+0000 7fb34a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:23:56.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.742+0000 7fb34a7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:23:56.791 INFO:tasks.rados.rados.0.smithi045.stdout:695: done (2 left) 2024-04-26T00:23:56.792 INFO:tasks.rados.rados.0.smithi045.stdout:696: rollback oid 42 current snap is 81 2024-04-26T00:23:56.792 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 42 to 67 2024-04-26T00:23:56.792 INFO:tasks.rados.rados.0.smithi045.stdout:692: finishing write tid 2 to smithi04526579-27 2024-04-26T00:23:56.792 INFO:tasks.rados.rados.0.smithi045.stdout:692: finishing write tid 3 to smithi04526579-27 2024-04-26T00:23:56.792 INFO:tasks.rados.rados.0.smithi045.stdout:692: finishing write tid 4 to smithi04526579-27 2024-04-26T00:23:56.792 INFO:tasks.rados.rados.0.smithi045.stdout:692: oid 27 updating version 0 to 355 2024-04-26T00:23:56.792 INFO:tasks.rados.rados.0.smithi045.stdout:692: oid 27 version 355 is already newer than 353 2024-04-26T00:23:56.792 INFO:tasks.rados.rados.0.smithi045.stdout:692: oid 27 version 355 is already newer than 354 2024-04-26T00:23:56.792 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 355 (ObjNum 284 snap 81 seq_num 284) dirty exists 2024-04-26T00:23:56.792 INFO:tasks.rados.rados.0.smithi045.stdout:692: left oid 27 (ObjNum 284 snap 81 seq_num 284) 2024-04-26T00:23:56.792 INFO:tasks.rados.rados.0.smithi045.stdout:693: finishing copy_from racing read to smithi04526579-12 2024-04-26T00:23:56.792 INFO:tasks.rados.rados.0.smithi045.stdout:693: finishing copy_from to smithi04526579-12 2024-04-26T00:23:56.793 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 290 (ObjNum 180 snap 47 seq_num 180) dirty exists 2024-04-26T00:23:56.793 INFO:tasks.rados.rados.0.smithi045.stdout:692: done (2 left) 2024-04-26T00:23:56.793 INFO:tasks.rados.rados.0.smithi045.stdout:693: done (1 left) 2024-04-26T00:23:56.793 INFO:tasks.rados.rados.0.smithi045.stdout:697: read oid 13 snap -1 2024-04-26T00:23:56.793 INFO:tasks.rados.rados.0.smithi045.stdout:697: expect (ObjNum 204 snap 53 seq_num 204) 2024-04-26T00:23:56.793 INFO:tasks.rados.rados.0.smithi045.stdout:698: read oid 2 snap 75 2024-04-26T00:23:56.793 INFO:tasks.rados.rados.0.smithi045.stdout:698: expect (ObjNum 265 snap 74 seq_num 265) 2024-04-26T00:23:56.793 INFO:tasks.rados.rados.0.smithi045.stdout:699: snap_create 2024-04-26T00:23:56.808 INFO:tasks.rados.rados.0.smithi045.stdout:696: finishing rollback tid 1 to smithi04526579-42 2024-04-26T00:23:56.809 INFO:tasks.rados.rados.0.smithi045.stdout:696: finishing rollback tid 2 to smithi04526579-42 2024-04-26T00:23:56.809 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 357 (ObjNum 114 snap 28 seq_num 114) dirty exists 2024-04-26T00:23:56.813 INFO:tasks.rados.rados.0.smithi045.stdout:698: expect (ObjNum 265 snap 74 seq_num 265) 2024-04-26T00:23:56.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.970+0000 7fb35c1a3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:23:56.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.970+0000 7fb35c1a3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:23:56.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.970+0000 7fb35c1a3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:23:56.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.970+0000 7fb35c1a3640 1 -- 172.21.15.37:0/1655119630 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb3541cf690 con 0x7fb3541355a0 2024-04-26T00:23:56.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.974+0000 7fb34a7fc640 1 -- 172.21.15.37:0/1655119630 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7fb34c0479b0 con 0x7fb3541355a0 2024-04-26T00:23:56.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.974+0000 7fb34a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:23:56.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.974+0000 7fb34a7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:23:56.951 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":32,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:23:56.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.974+0000 7fb35c1a3640 1 -- 172.21.15.37:0/1655119630 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb338035ed0 msgr2=0x7fb338038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:56.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.974+0000 7fb35c1a3640 1 --2- 172.21.15.37:0/1655119630 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb338035ed0 0x7fb338038390 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb35001a010 tx=0x7fb3500a6000 comp rx=0 tx=0).stop 2024-04-26T00:23:56.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.974+0000 7fb35c1a3640 10 monclient: shutdown 2024-04-26T00:23:56.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.974+0000 7fb35c1a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:23:56.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.974+0000 7fb35c1a3640 1 -- 172.21.15.37:0/1655119630 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb3541355a0 msgr2=0x7fb354130ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:23:56.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.974+0000 7fb35c1a3640 1 --2- 172.21.15.37:0/1655119630 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb3541355a0 0x7fb354130ec0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb34c007d90 tx=0x7fb34c00e4b0 comp rx=0 tx=0).stop 2024-04-26T00:23:56.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.974+0000 7fb35c1a3640 1 -- 172.21.15.37:0/1655119630 shutdown_connections 2024-04-26T00:23:56.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.974+0000 7fb35c1a3640 1 --2- 172.21.15.37:0/1655119630 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb338035ed0 0x7fb338038390 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:56.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.974+0000 7fb35c1a3640 1 --2- 172.21.15.37:0/1655119630 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb3541355a0 0x7fb354130ec0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:56.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.974+0000 7fb35c1a3640 1 --2- 172.21.15.37:0/1655119630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3541351c0 0x7fb354130980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:56.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.974+0000 7fb35c1a3640 1 --2- 172.21.15.37:0/1655119630 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3540a2380 0x7fb35413b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:23:56.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.974+0000 7fb35c1a3640 1 -- 172.21.15.37:0/1655119630 >> 172.21.15.37:0/1655119630 conn(0x7fb3540a2d60 msgr2=0x7fb3540a49f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:23:56.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.974+0000 7fb35c1a3640 1 -- 172.21.15.37:0/1655119630 shutdown_connections 2024-04-26T00:23:56.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:23:56.974+0000 7fb35c1a3640 1 -- 172.21.15.37:0/1655119630 wait complete. 2024-04-26T00:23:57.108 INFO:tasks.rados.rados.0.smithi045.stdout:697: expect (ObjNum 204 snap 53 seq_num 204) 2024-04-26T00:23:57.793 INFO:tasks.rados.rados.0.smithi045.stdout:696: done (3 left) 2024-04-26T00:23:57.794 INFO:tasks.rados.rados.0.smithi045.stdout:697: done (2 left) 2024-04-26T00:23:57.794 INFO:tasks.rados.rados.0.smithi045.stdout:698: done (1 left) 2024-04-26T00:23:57.794 INFO:tasks.rados.rados.0.smithi045.stdout:699: done (0 left) 2024-04-26T00:23:57.794 INFO:tasks.rados.rados.0.smithi045.stdout:700: rollback oid 32 current snap is 82 2024-04-26T00:23:57.794 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 32 to 73 2024-04-26T00:23:57.794 INFO:tasks.rados.rados.0.smithi045.stdout:701: read oid 48 snap -1 2024-04-26T00:23:57.794 INFO:tasks.rados.rados.0.smithi045.stdout:701: expect deleted 2024-04-26T00:23:57.794 INFO:tasks.rados.rados.0.smithi045.stdout:702: rollback oid 1 current snap is 82 2024-04-26T00:23:57.794 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 1 to 32 2024-04-26T00:23:57.794 INFO:tasks.rados.rados.0.smithi045.stdout:703: snap_create 2024-04-26T00:23:57.804 INFO:tasks.rados.rados.0.smithi045.stdout:702: finishing rollback tid 0 to smithi04526579-1 2024-04-26T00:23:57.805 INFO:tasks.rados.rados.0.smithi045.stdout:702: finishing rollback tid 1 to smithi04526579-1 2024-04-26T00:23:57.805 INFO:tasks.rados.rados.0.smithi045.stdout:702: finishing rollback tid 2 to smithi04526579-1 2024-04-26T00:23:57.805 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 368 (ObjNum 110 snap 28 seq_num 110) dirty exists 2024-04-26T00:23:57.808 INFO:tasks.rados.rados.0.smithi045.stdout:700: finishing rollback tid 0 to smithi04526579-32 2024-04-26T00:23:57.808 INFO:tasks.rados.rados.0.smithi045.stdout:700: finishing rollback tid 1 to smithi04526579-32 2024-04-26T00:23:57.808 INFO:tasks.rados.rados.0.smithi045.stdout:700: finishing rollback tid 2 to smithi04526579-32 2024-04-26T00:23:57.808 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 360 (ObjNum 242 snap 61 seq_num 242) dirty exists 2024-04-26T00:23:59.028 INFO:tasks.rados.rados.0.smithi045.stdout:700: done (3 left) 2024-04-26T00:23:59.028 INFO:tasks.rados.rados.0.smithi045.stdout:701: done (2 left) 2024-04-26T00:23:59.028 INFO:tasks.rados.rados.0.smithi045.stdout:702: done (1 left) 2024-04-26T00:23:59.028 INFO:tasks.rados.rados.0.smithi045.stdout:703: done (0 left) 2024-04-26T00:23:59.028 INFO:tasks.rados.rados.0.smithi045.stdout:704: copy_from oid 7 from oid 10 current snap is 83 2024-04-26T00:23:59.028 INFO:tasks.rados.rados.0.smithi045.stdout:705: rollback oid 26 current snap is 83 2024-04-26T00:23:59.028 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 26 to 63 2024-04-26T00:23:59.029 INFO:tasks.rados.rados.0.smithi045.stdout:706: rollback oid 19 current snap is 83 2024-04-26T00:23:59.029 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 19 to 77 2024-04-26T00:23:59.029 INFO:tasks.rados.rados.0.smithi045.stdout:707: snap_create 2024-04-26T00:23:59.034 INFO:tasks.rados.rados.0.smithi045.stdout:704: finishing copy_from to smithi04526579-7 2024-04-26T00:23:59.034 INFO:tasks.rados.rados.0.smithi045.stdout:704: got expected ENOENT (src dne) 2024-04-26T00:23:59.041 INFO:tasks.rados.rados.0.smithi045.stdout:704: finishing copy_from racing read to smithi04526579-7 2024-04-26T00:23:59.041 INFO:tasks.rados.rados.0.smithi045.stdout:706: finishing rollback tid 0 to smithi04526579-19 2024-04-26T00:23:59.041 INFO:tasks.rados.rados.0.smithi045.stdout:706: finishing rollback tid 1 to smithi04526579-19 2024-04-26T00:23:59.042 INFO:tasks.rados.rados.0.smithi045.stdout:706: finishing rollback tid 2 to smithi04526579-19 2024-04-26T00:23:59.042 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 363 (ObjNum 224 snap 59 seq_num 224) dirty exists 2024-04-26T00:23:59.045 INFO:tasks.rados.rados.0.smithi045.stdout:705: finishing rollback tid 0 to smithi04526579-26 2024-04-26T00:23:59.046 INFO:tasks.rados.rados.0.smithi045.stdout:705: finishing rollback tid 1 to smithi04526579-26 2024-04-26T00:23:59.047 INFO:tasks.rados.rados.0.smithi045.stdout:705: finishing rollback tid 2 to smithi04526579-26 2024-04-26T00:23:59.047 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 372 (ObjNum 209 snap 53 seq_num 209) dirty exists 2024-04-26T00:23:59.739 INFO:tasks.rados.rados.0.smithi045.stdout:704: done (3 left) 2024-04-26T00:23:59.740 INFO:tasks.rados.rados.0.smithi045.stdout:705: done (2 left) 2024-04-26T00:23:59.740 INFO:tasks.rados.rados.0.smithi045.stdout:706: done (1 left) 2024-04-26T00:23:59.740 INFO:tasks.rados.rados.0.smithi045.stdout:707: done (0 left) 2024-04-26T00:23:59.740 INFO:tasks.rados.rados.0.smithi045.stdout:708: snap_create 2024-04-26T00:23:59.969 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:24:00.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea230c6640 10 monclient: get_monmap_and_config 2024-04-26T00:24:00.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea230c6640 10 monclient: build_initial_monmap 2024-04-26T00:24:00.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea230c6640 10 monclient: monmap: 2024-04-26T00:24:00.095 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:24:00.095 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:00.095 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:24:00.121215+0000 2024-04-26T00:24:00.095 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:24:00.121215+0000 2024-04-26T00:24:00.096 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:24:00.096 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:00.096 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:24:00.096 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:24:00.096 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:24:00.096 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:24:00.096 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:24:00.096 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:00.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea230c6640 1 Processor -- start 2024-04-26T00:24:00.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea230c6640 1 -- start start 2024-04-26T00:24:00.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea230c6640 10 monclient: init 2024-04-26T00:24:00.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea230c6640 10 monclient: _reopen_session rank -1 2024-04-26T00:24:00.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea230c6640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-04-26T00:24:00.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea230c6640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fea1c136da0 0x7fea1c09bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:00.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea230c6640 10 monclient(hunting): picked mon.noname-c con 0x7fea1c136da0 addr v2:172.21.15.45:3300/0 2024-04-26T00:24:00.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea230c6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fea1c09c410 addr v1:172.21.15.45:6789/0 2024-04-26T00:24:00.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea230c6640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fea1c137890 0x7fea1c13e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:00.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea230c6640 10 monclient(hunting): picked mon.noname-b con 0x7fea1c137890 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:24:00.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea230c6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea1c09b4f0 con 0x7fea1c09c410 2024-04-26T00:24:00.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea230c6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:00.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea230c6640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fea1c09bb40 con 0x7fea1c136da0 2024-04-26T00:24:00.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea230c6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:00.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea230c6640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fea1c09b210 con 0x7fea1c137890 2024-04-26T00:24:00.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea230c6640 10 monclient(hunting): _renew_subs 2024-04-26T00:24:00.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea230c6640 10 monclient(hunting): authenticate will time out at 1221.426941s 2024-04-26T00:24:00.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea21663640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fea1c137890 0x7fea1c13e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:00.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea21663640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fea1c137890 0x7fea1c13e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:45904/0 (socket says 172.21.15.37:45904) 2024-04-26T00:24:00.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea21663640 1 -- 172.21.15.37:0/1201385233 learned_addr learned my addr 172.21.15.37:0/1201385233 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:24:00.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea21663640 10 monclient(hunting): get_auth_request con 0x7fea1c137890 auth_method 0 2024-04-26T00:24:00.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea21663640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:00.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea21663640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:00.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea21663640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:00.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea21663640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:00.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea21663640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:00.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea21663640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:00.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea20e62640 1 --2- 172.21.15.37:0/1201385233 >> v2:172.21.15.45:3300/0 conn(0x7fea1c136da0 0x7fea1c09bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:00.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea21663640 10 monclient(hunting): handle_auth_done global_id 34283 payload 274 2024-04-26T00:24:00.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea21663640 1 -- 172.21.15.37:0/1201385233 >> v2:172.21.15.45:3300/0 conn(0x7fea1c136da0 msgr2=0x7fea1c09bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:24:00.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea21663640 1 --2- 172.21.15.37:0/1201385233 >> v2:172.21.15.45:3300/0 conn(0x7fea1c136da0 0x7fea1c09bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:00.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea21663640 1 -- 172.21.15.37:0/1201385233 >> v1:172.21.15.45:6789/0 conn(0x7fea1c09c410 legacy=0x7fea1c137180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:24:00.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea21663640 10 monclient: _finish_hunting 0 2024-04-26T00:24:00.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea21663640 1 monclient: found mon.noname-b 2024-04-26T00:24:00.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea21663640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:24:00.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea21663640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:24:00.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea21663640 1 -- 172.21.15.37:0/1201385233 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea1c09b7d0 con 0x7fea1c137890 2024-04-26T00:24:00.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea20e62640 10 monclient: get_auth_request con 0x7fea1c136da0 auth_method 0 2024-04-26T00:24:00.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.118+0000 7fea20e62640 1 --2- 172.21.15.37:0/1201385233 >> v2:172.21.15.45:3300/0 conn(0x7fea1c136da0 0x7fea1c09bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:24:00.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea21663640 1 -- 172.21.15.37:0/1201385233 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fea1c137890 msgr2=0x7fea1c13e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:00.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea21663640 1 --2- 172.21.15.37:0/1201385233 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fea1c137890 0x7fea1c13e2e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fea14009870 tx=0x7fea1401c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8778b595edfce5e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:00.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea1b7fe640 1 -- 172.21.15.37:0/1201385233 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea140b1070 con 0x7fea1c137890 2024-04-26T00:24:00.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:00.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea1b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:24:00.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea1b7fe640 10 monclient: dump: 2024-04-26T00:24:00.103 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:00.103 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:00.103 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:00.103 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:00.103 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:00.103 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:00.103 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:00.103 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:00.103 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:00.103 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:00.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea1b7fe640 10 monclient: _finish_auth 0 2024-04-26T00:24:00.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea1b7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:24:00.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:24:00.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea1b7fe640 1 -- 172.21.15.37:0/1201385233 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea140ac620 con 0x7fea1c137890 2024-04-26T00:24:00.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea230c6640 5 monclient: authenticate success, global_id 34283 2024-04-26T00:24:00.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea230c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:24:00.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea1b7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:24:00.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea230c6640 10 monclient: get_monmap_and_config success 2024-04-26T00:24:00.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea1b7fe640 1 -- 172.21.15.37:0/1201385233 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea140ac9f0 con 0x7fea1c137890 2024-04-26T00:24:00.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:00.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea230c6640 4 set_mon_vals no callback set 2024-04-26T00:24:00.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea1b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:24:00.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea1b7fe640 10 monclient: dump: 2024-04-26T00:24:00.104 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:00.104 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:00.105 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:00.105 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:00.105 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:00.105 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:00.105 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:00.105 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:00.105 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:00.105 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:00.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea230c6640 10 monclient: shutdown 2024-04-26T00:24:00.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea230c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:24:00.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea230c6640 1 -- 172.21.15.37:0/1201385233 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fea1c137890 msgr2=0x7fea1c13e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:00.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea230c6640 1 --2- 172.21.15.37:0/1201385233 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fea1c137890 0x7fea1c13e2e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fea14009870 tx=0x7fea1401c3e0 comp rx=0 tx=0).stop 2024-04-26T00:24:00.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea230c6640 1 -- 172.21.15.37:0/1201385233 shutdown_connections 2024-04-26T00:24:00.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea230c6640 1 --2- 172.21.15.37:0/1201385233 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fea1c137890 0x7fea1c13e2e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:00.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea230c6640 1 --2- 172.21.15.37:0/1201385233 >> v2:172.21.15.45:3300/0 conn(0x7fea1c136da0 0x7fea1c09bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:00.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea230c6640 1 -- 172.21.15.37:0/1201385233 >> 172.21.15.37:0/1201385233 conn(0x7fea1c0a5f00 msgr2=0x7fea1c0a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:24:00.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea230c6640 1 -- 172.21.15.37:0/1201385233 shutdown_connections 2024-04-26T00:24:00.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea230c6640 1 -- 172.21.15.37:0/1201385233 wait complete. 2024-04-26T00:24:00.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea230c6640 10 monclient: build_initial_monmap 2024-04-26T00:24:00.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea230c6640 10 monclient: monmap: 2024-04-26T00:24:00.109 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:24:00.109 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:24:00.110 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:24:00.110 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:24:00.110 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:24:00.110 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:24:00.110 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:24:00.110 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:24:00.110 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:24:00.110 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:00.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea230c6640 1 Processor -- start 2024-04-26T00:24:00.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea230c6640 1 -- start start 2024-04-26T00:24:00.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea230c6640 10 monclient: init 2024-04-26T00:24:00.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea230c6640 10 monclient: _reopen_session rank -1 2024-04-26T00:24:00.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea230c6640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:24:00.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea230c6640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fea1c09c410 0x7fea1c1ce900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:00.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea230c6640 10 monclient(hunting): picked mon.noname-c con 0x7fea1c09c410 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:24:00.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea230c6640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fea1c136da0 0x7fea1c1cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:00.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea230c6640 10 monclient(hunting): picked mon.noname-a con 0x7fea1c136da0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:24:00.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea230c6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea1c137890 0x7fea1c13acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:00.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea230c6640 10 monclient(hunting): picked mon.noname-b con 0x7fea1c137890 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:24:00.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea230c6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:00.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea230c6640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fea1c1414b0 con 0x7fea1c136da0 2024-04-26T00:24:00.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea230c6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:00.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea230c6640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fea1c141330 con 0x7fea1c137890 2024-04-26T00:24:00.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea230c6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:00.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea230c6640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fea1c141630 con 0x7fea1c09c410 2024-04-26T00:24:00.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea230c6640 10 monclient(hunting): _renew_subs 2024-04-26T00:24:00.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea230c6640 10 monclient(hunting): authenticate will time out at 1221.434652s 2024-04-26T00:24:00.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea20e62640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fea1c09c410 0x7fea1c1ce900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:00.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea20e62640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fea1c09c410 0x7fea1c1ce900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:45908/0 (socket says 172.21.15.37:45908) 2024-04-26T00:24:00.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea20e62640 1 -- 172.21.15.37:0/417565136 learned_addr learned my addr 172.21.15.37:0/417565136 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:24:00.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea20e62640 10 monclient(hunting): get_auth_request con 0x7fea1c09c410 auth_method 0 2024-04-26T00:24:00.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea20e62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:00.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea20e62640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:00.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea20e62640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:00.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea1bfff640 1 --2- 172.21.15.37:0/417565136 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fea1c136da0 0x7fea1c1cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:00.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea1bfff640 10 monclient(hunting): get_auth_request con 0x7fea1c136da0 auth_method 0 2024-04-26T00:24:00.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:00.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea1bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:00.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:00.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea20e62640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:00.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea20e62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:00.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea20e62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:00.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea21663640 1 --2- 172.21.15.37:0/417565136 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea1c137890 0x7fea1c13acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:00.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea21663640 10 monclient(hunting): get_auth_request con 0x7fea1c137890 auth_method 0 2024-04-26T00:24:00.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea21663640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:00.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea21663640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:00.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea21663640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:00.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea1bfff640 1 -- 172.21.15.37:0/417565136 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fea1c136da0 msgr2=0x7fea1c1cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:24:00.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea1bfff640 1 -- 172.21.15.37:0/417565136 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fea1c136da0 msgr2=0x7fea1c1cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:24:00.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea1bfff640 1 --2- 172.21.15.37:0/417565136 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fea1c136da0 0x7fea1c1cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:24:00.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea1bfff640 1 --2- 172.21.15.37:0/417565136 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fea1c136da0 0x7fea1c1cf0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:24:00.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea20e62640 10 monclient(hunting): handle_auth_done global_id 34286 payload 824 2024-04-26T00:24:00.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea20e62640 1 -- 172.21.15.37:0/417565136 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea1c137890 msgr2=0x7fea1c13acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:00.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea20e62640 1 --2- 172.21.15.37:0/417565136 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea1c137890 0x7fea1c13acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:00.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea20e62640 1 -- 172.21.15.37:0/417565136 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fea1c136da0 msgr2=0x7fea1c1cf0d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:24:00.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea20e62640 1 --2- 172.21.15.37:0/417565136 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fea1c136da0 0x7fea1c1cf0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:00.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea20e62640 10 monclient: _finish_hunting 0 2024-04-26T00:24:00.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea20e62640 1 monclient: found mon.noname-c 2024-04-26T00:24:00.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea20e62640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:24:00.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea20e62640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:24:00.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea20e62640 1 -- 172.21.15.37:0/417565136 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea1c1206d0 con 0x7fea1c09c410 2024-04-26T00:24:00.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.126+0000 7fea21663640 1 --2- 172.21.15.37:0/417565136 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea1c137890 0x7fea1c13acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:24:00.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.130+0000 7fea20e62640 1 -- 172.21.15.37:0/417565136 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fea1c09c410 msgr2=0x7fea1c1ce900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:00.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.130+0000 7fea20e62640 1 --2- 172.21.15.37:0/417565136 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fea1c09c410 0x7fea1c1ce900 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fea0c00c7f0 tx=0x7fea0c00ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:00.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.130+0000 7fea197fa640 1 -- 172.21.15.37:0/417565136 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea0c007c40 con 0x7fea1c09c410 2024-04-26T00:24:00.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.130+0000 7fea197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:00.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.130+0000 7fea197fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:24:00.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.130+0000 7fea197fa640 10 monclient: dump: 2024-04-26T00:24:00.114 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:00.114 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:00.114 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:00.114 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:00.114 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:00.114 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:00.115 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:00.115 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:00.115 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:00.115 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:00.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.130+0000 7fea197fa640 10 monclient: _finish_auth 0 2024-04-26T00:24:00.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.130+0000 7fea197fa640 10 monclient: _check_auth_tickets 2024-04-26T00:24:00.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.130+0000 7fea197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:24:00.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.130+0000 7fea197fa640 1 -- 172.21.15.37:0/417565136 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea0c007de0 con 0x7fea1c09c410 2024-04-26T00:24:00.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.130+0000 7fea197fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:24:00.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.130+0000 7fea197fa640 1 -- 172.21.15.37:0/417565136 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea0c011630 con 0x7fea1c09c410 2024-04-26T00:24:00.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.130+0000 7fea197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:00.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.130+0000 7fea197fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:24:00.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.130+0000 7fea197fa640 10 monclient: dump: 2024-04-26T00:24:00.115 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:00.115 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:00.116 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:00.116 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:00.116 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:00.116 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:00.116 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:00.116 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:00.116 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:00.116 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:00.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.130+0000 7fea230c6640 5 monclient: authenticate success, global_id 34286 2024-04-26T00:24:00.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.130+0000 7fea230c6640 10 monclient: _renew_subs 2024-04-26T00:24:00.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.130+0000 7fea1affd640 4 set_mon_vals no callback set 2024-04-26T00:24:00.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.130+0000 7fea230c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:24:00.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.130+0000 7fea230c6640 1 -- 172.21.15.37:0/417565136 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea1c0a8430 con 0x7fea1c09c410 2024-04-26T00:24:00.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.130+0000 7fea230c6640 10 monclient: _renew_subs 2024-04-26T00:24:00.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.130+0000 7fea230c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:24:00.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.130+0000 7fea230c6640 1 -- 172.21.15.37:0/417565136 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fea1c13c850 con 0x7fea1c09c410 2024-04-26T00:24:00.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.130+0000 7fea230c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:24:00.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.130+0000 7fea230c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:24:00.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.130+0000 7fea230c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:24:00.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.130+0000 7fea230c6640 1 -- 172.21.15.37:0/417565136 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea1c066f40 con 0x7fea1c09c410 2024-04-26T00:24:00.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.130+0000 7fea197fa640 1 -- 172.21.15.37:0/417565136 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fea0c0117d0 con 0x7fea1c09c410 2024-04-26T00:24:00.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.130+0000 7fea197fa640 1 --2- 172.21.15.37:0/417565136 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe9fc035e80 0x7fe9fc038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:00.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.130+0000 7fea197fa640 1 -- 172.21.15.37:0/417565136 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(171..171 src has 1..171) ==== 3927+0+0 (secure 0 0 0) 0x7fea0c048cd0 con 0x7fea1c09c410 2024-04-26T00:24:00.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.130+0000 7fea1bfff640 1 --2- 172.21.15.37:0/417565136 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe9fc035e80 0x7fe9fc038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:00.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.130+0000 7fea1bfff640 10 monclient: get_auth_request con 0x7fe9fc035e80 auth_method 0 2024-04-26T00:24:00.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.134+0000 7fea1bfff640 1 --2- 172.21.15.37:0/417565136 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe9fc035e80 0x7fe9fc038340 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fea1c13c4d0 tx=0x7fea04009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:00.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.138+0000 7fea197fa640 1 -- 172.21.15.37:0/417565136 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fea0c015020 con 0x7fea1c09c410 2024-04-26T00:24:00.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.138+0000 7fea197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:24:00.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.138+0000 7fea197fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:24:00.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.358+0000 7fea230c6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:24:00.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.358+0000 7fea230c6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:24:00.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.358+0000 7fea230c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:24:00.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.358+0000 7fea230c6640 1 -- 172.21.15.37:0/417565136 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fea1c1204b0 con 0x7fea1c09c410 2024-04-26T00:24:00.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.358+0000 7fea197fa640 1 -- 172.21.15.37:0/417565136 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7fea0c011da0 con 0x7fea1c09c410 2024-04-26T00:24:00.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.358+0000 7fea197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:24:00.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.358+0000 7fea197fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:24:00.335 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":32,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:24:00.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.358+0000 7fea230c6640 1 -- 172.21.15.37:0/417565136 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe9fc035e80 msgr2=0x7fe9fc038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:00.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.358+0000 7fea230c6640 1 --2- 172.21.15.37:0/417565136 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe9fc035e80 0x7fe9fc038340 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fea1c13c4d0 tx=0x7fea04009040 comp rx=0 tx=0).stop 2024-04-26T00:24:00.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.358+0000 7fea230c6640 10 monclient: shutdown 2024-04-26T00:24:00.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.358+0000 7fea230c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:24:00.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.358+0000 7fea230c6640 1 -- 172.21.15.37:0/417565136 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fea1c09c410 msgr2=0x7fea1c1ce900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:00.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.358+0000 7fea230c6640 1 --2- 172.21.15.37:0/417565136 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fea1c09c410 0x7fea1c1ce900 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fea0c00c7f0 tx=0x7fea0c00ccb0 comp rx=0 tx=0).stop 2024-04-26T00:24:00.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.358+0000 7fea230c6640 1 -- 172.21.15.37:0/417565136 shutdown_connections 2024-04-26T00:24:00.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.358+0000 7fea230c6640 1 --2- 172.21.15.37:0/417565136 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe9fc035e80 0x7fe9fc038340 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:00.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.358+0000 7fea230c6640 1 --2- 172.21.15.37:0/417565136 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea1c137890 0x7fea1c13acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:00.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.358+0000 7fea230c6640 1 --2- 172.21.15.37:0/417565136 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fea1c136da0 0x7fea1c1cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:00.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.358+0000 7fea230c6640 1 --2- 172.21.15.37:0/417565136 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fea1c09c410 0x7fea1c1ce900 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:00.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.358+0000 7fea230c6640 1 -- 172.21.15.37:0/417565136 >> 172.21.15.37:0/417565136 conn(0x7fea1c0a5f00 msgr2=0x7fea1c0a7200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:24:00.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.358+0000 7fea230c6640 1 -- 172.21.15.37:0/417565136 shutdown_connections 2024-04-26T00:24:00.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:00.358+0000 7fea230c6640 1 -- 172.21.15.37:0/417565136 wait complete. 2024-04-26T00:24:00.985 INFO:tasks.rados.rados.0.smithi045.stdout:708: done (0 left) 2024-04-26T00:24:00.985 INFO:tasks.rados.rados.0.smithi045.stdout:709: copy_from oid 30 from oid 15 current snap is 85 2024-04-26T00:24:00.986 INFO:tasks.rados.rados.0.smithi045.stdout:710: delete oid 14 current snap is 85 2024-04-26T00:24:00.998 INFO:tasks.rados.rados.0.smithi045.stdout:710: done (1 left) 2024-04-26T00:24:00.998 INFO:tasks.rados.rados.0.smithi045.stdout:711: read oid 10 snap 82 2024-04-26T00:24:00.998 INFO:tasks.rados.rados.0.smithi045.stdout:711: expect deleted 2024-04-26T00:24:00.998 INFO:tasks.rados.rados.0.smithi045.stdout:712: rollback oid 35 current snap is 85 2024-04-26T00:24:00.998 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 35 to 32 2024-04-26T00:24:00.998 INFO:tasks.rados.rados.0.smithi045.stdout:713: read oid 33 snap -1 2024-04-26T00:24:00.998 INFO:tasks.rados.rados.0.smithi045.stdout:713: expect (ObjNum 246 snap 63 seq_num 246) 2024-04-26T00:24:00.999 INFO:tasks.rados.rados.0.smithi045.stdout:714: read oid 12 snap -1 2024-04-26T00:24:00.999 INFO:tasks.rados.rados.0.smithi045.stdout:714: expect (ObjNum 180 snap 47 seq_num 180) 2024-04-26T00:24:00.999 INFO:tasks.rados.rados.0.smithi045.stdout:715: delete oid 45 current snap is 85 2024-04-26T00:24:01.009 INFO:tasks.rados.rados.0.smithi045.stdout:713: expect (ObjNum 246 snap 63 seq_num 246) 2024-04-26T00:24:01.165 INFO:tasks.rados.rados.0.smithi045.stdout:709: finishing copy_from racing read to smithi04526579-30 2024-04-26T00:24:01.165 INFO:tasks.rados.rados.0.smithi045.stdout:712: finishing rollback tid 0 to smithi04526579-35 2024-04-26T00:24:01.165 INFO:tasks.rados.rados.0.smithi045.stdout:714: expect (ObjNum 180 snap 47 seq_num 180) 2024-04-26T00:24:01.313 INFO:tasks.rados.rados.0.smithi045.stdout:712: finishing rollback tid 1 to smithi04526579-35 2024-04-26T00:24:01.313 INFO:tasks.rados.rados.0.smithi045.stdout:712: finishing rollback tid 2 to smithi04526579-35 2024-04-26T00:24:01.313 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 366 (ObjNum 138 snap 31 seq_num 138) dirty exists 2024-04-26T00:24:01.313 INFO:tasks.rados.rados.0.smithi045.stdout:709: finishing copy_from to smithi04526579-30 2024-04-26T00:24:01.313 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 306 (ObjNum 163 snap 37 seq_num 163) dirty exists 2024-04-26T00:24:01.313 INFO:tasks.rados.rados.0.smithi045.stdout:709: done (5 left) 2024-04-26T00:24:01.313 INFO:tasks.rados.rados.0.smithi045.stdout:711: done (4 left) 2024-04-26T00:24:01.313 INFO:tasks.rados.rados.0.smithi045.stdout:712: done (3 left) 2024-04-26T00:24:01.313 INFO:tasks.rados.rados.0.smithi045.stdout:713: done (2 left) 2024-04-26T00:24:01.313 INFO:tasks.rados.rados.0.smithi045.stdout:714: done (1 left) 2024-04-26T00:24:01.313 INFO:tasks.rados.rados.0.smithi045.stdout:715: done (0 left) 2024-04-26T00:24:01.313 INFO:tasks.rados.rados.0.smithi045.stdout:716: rollback oid 24 current snap is 85 2024-04-26T00:24:01.314 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 24 to 83 2024-04-26T00:24:01.314 INFO:tasks.rados.rados.0.smithi045.stdout:717: rollback oid 20 current snap is 85 2024-04-26T00:24:01.314 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 20 to 74 2024-04-26T00:24:01.314 INFO:tasks.rados.rados.0.smithi045.stdout:718: write (excl) oid 11 current snap is 85 2024-04-26T00:24:01.314 INFO:tasks.rados.rados.0.smithi045.stdout:718: seq_num 287 ranges {423609=701524,1628332=472089,2648987=174622} 2024-04-26T00:24:01.331 INFO:tasks.rados.rados.0.smithi045.stdout:718: writing smithi04526579-11 from 423609 to 1125133 tid 1 2024-04-26T00:24:01.347 INFO:tasks.rados.rados.0.smithi045.stdout:718: writing smithi04526579-11 from 1628332 to 2100421 tid 2 2024-04-26T00:24:01.360 INFO:tasks.rados.rados.0.smithi045.stdout:718: writing smithi04526579-11 from 2648987 to 2823609 tid 3 2024-04-26T00:24:01.360 INFO:tasks.rados.rados.0.smithi045.stdout:719: rollback oid 40 current snap is 85 2024-04-26T00:24:01.360 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 40 to 67 2024-04-26T00:24:01.360 INFO:tasks.rados.rados.0.smithi045.stdout:717: finishing rollback tid 0 to smithi04526579-20 2024-04-26T00:24:01.360 INFO:tasks.rados.rados.0.smithi045.stdout:716: finishing rollback tid 0 to smithi04526579-24 2024-04-26T00:24:01.360 INFO:tasks.rados.rados.0.smithi045.stdout:716: finishing rollback tid 1 to smithi04526579-24 2024-04-26T00:24:01.360 INFO:tasks.rados.rados.0.smithi045.stdout:717: finishing rollback tid 1 to smithi04526579-20 2024-04-26T00:24:01.360 INFO:tasks.rados.rados.0.smithi045.stdout:716: finishing rollback tid 2 to smithi04526579-24 2024-04-26T00:24:01.360 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 470 (ObjNum 201 snap 53 seq_num 201) dirty exists 2024-04-26T00:24:01.360 INFO:tasks.rados.rados.0.smithi045.stdout:717: finishing rollback tid 2 to smithi04526579-20 2024-04-26T00:24:01.360 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 225 (ObjNum 249 snap 64 seq_num 249) dirty exists 2024-04-26T00:24:01.360 INFO:tasks.rados.rados.0.smithi045.stdout:718: finishing write tid 1 to smithi04526579-11 2024-04-26T00:24:01.361 INFO:tasks.rados.rados.0.smithi045.stdout:718: finishing write tid 2 to smithi04526579-11 2024-04-26T00:24:01.361 INFO:tasks.rados.rados.0.smithi045.stdout:716: done (3 left) 2024-04-26T00:24:01.361 INFO:tasks.rados.rados.0.smithi045.stdout:717: done (2 left) 2024-04-26T00:24:01.361 INFO:tasks.rados.rados.0.smithi045.stdout:720: delete oid 18 current snap is 85 2024-04-26T00:24:01.364 INFO:tasks.rados.rados.0.smithi045.stdout:718: finishing write tid 3 to smithi04526579-11 2024-04-26T00:24:01.364 INFO:tasks.rados.rados.0.smithi045.stdout:718: finishing write tid 4 to smithi04526579-11 2024-04-26T00:24:01.365 INFO:tasks.rados.rados.0.smithi045.stdout:718: finishing write tid 5 to smithi04526579-11 2024-04-26T00:24:01.365 INFO:tasks.rados.rados.0.smithi045.stdout:718: oid 11 updating version 0 to 143 2024-04-26T00:24:01.365 INFO:tasks.rados.rados.0.smithi045.stdout:718: oid 11 version 143 is already newer than 140 2024-04-26T00:24:01.365 INFO:tasks.rados.rados.0.smithi045.stdout:718: oid 11 version 143 is already newer than 141 2024-04-26T00:24:01.365 INFO:tasks.rados.rados.0.smithi045.stdout:718: oid 11 version 143 is already newer than 142 2024-04-26T00:24:01.365 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 143 (ObjNum 287 snap 85 seq_num 287) dirty exists 2024-04-26T00:24:01.365 INFO:tasks.rados.rados.0.smithi045.stdout:718: left oid 11 (ObjNum 287 snap 85 seq_num 287) 2024-04-26T00:24:01.365 INFO:tasks.rados.rados.0.smithi045.stdout:719: finishing rollback tid 1 to smithi04526579-40 2024-04-26T00:24:01.366 INFO:tasks.rados.rados.0.smithi045.stdout:719: finishing rollback tid 2 to smithi04526579-40 2024-04-26T00:24:01.366 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 292 (ObjNum 209 snap 53 seq_num 209) dirty exists 2024-04-26T00:24:01.369 INFO:tasks.rados.rados.0.smithi045.stdout:718: done (2 left) 2024-04-26T00:24:01.369 INFO:tasks.rados.rados.0.smithi045.stdout:719: done (1 left) 2024-04-26T00:24:01.369 INFO:tasks.rados.rados.0.smithi045.stdout:720: done (0 left) 2024-04-26T00:24:01.369 INFO:tasks.rados.rados.0.smithi045.stdout:721: delete oid 18 current snap is 85 2024-04-26T00:24:01.372 INFO:tasks.rados.rados.0.smithi045.stdout:721: done (0 left) 2024-04-26T00:24:01.372 INFO:tasks.rados.rados.0.smithi045.stdout:722: delete oid 47 current snap is 85 2024-04-26T00:24:01.381 INFO:tasks.rados.rados.0.smithi045.stdout:722: done (0 left) 2024-04-26T00:24:01.382 INFO:tasks.rados.rados.0.smithi045.stdout:723: write oid 29 current snap is 85 2024-04-26T00:24:01.382 INFO:tasks.rados.rados.0.smithi045.stdout:723: seq_num 291 ranges {490837=718873,1862260=479427,2875032=15805} 2024-04-26T00:24:01.401 INFO:tasks.rados.rados.0.smithi045.stdout:723: writing smithi04526579-29 from 490837 to 1209710 tid 1 2024-04-26T00:24:01.419 INFO:tasks.rados.rados.0.smithi045.stdout:723: writing smithi04526579-29 from 1862260 to 2341687 tid 2 2024-04-26T00:24:01.427 INFO:tasks.rados.rados.0.smithi045.stdout:723: writing smithi04526579-29 from 2875032 to 2890837 tid 3 2024-04-26T00:24:01.427 INFO:tasks.rados.rados.0.smithi045.stdout:724: snap_remove snap 72 2024-04-26T00:24:01.427 INFO:tasks.rados.rados.0.smithi045.stdout:723: finishing write tid 1 to smithi04526579-29 2024-04-26T00:24:01.427 INFO:tasks.rados.rados.0.smithi045.stdout:723: finishing write tid 2 to smithi04526579-29 2024-04-26T00:24:01.992 INFO:tasks.rados.rados.0.smithi045.stdout:724: done (1 left) 2024-04-26T00:24:01.993 INFO:tasks.rados.rados.0.smithi045.stdout:725: read oid 30 snap -1 2024-04-26T00:24:01.993 INFO:tasks.rados.rados.0.smithi045.stdout:725: expect (ObjNum 163 snap 37 seq_num 163) 2024-04-26T00:24:01.993 INFO:tasks.rados.rados.0.smithi045.stdout:726: delete oid 41 current snap is 85 2024-04-26T00:24:01.993 INFO:tasks.rados.rados.0.smithi045.stdout:723: finishing write tid 3 to smithi04526579-29 2024-04-26T00:24:01.993 INFO:tasks.rados.rados.0.smithi045.stdout:723: finishing write tid 4 to smithi04526579-29 2024-04-26T00:24:01.993 INFO:tasks.rados.rados.0.smithi045.stdout:723: finishing write tid 5 to smithi04526579-29 2024-04-26T00:24:01.993 INFO:tasks.rados.rados.0.smithi045.stdout:723: oid 29 updating version 0 to 302 2024-04-26T00:24:01.993 INFO:tasks.rados.rados.0.smithi045.stdout:723: oid 29 updating version 302 to 304 2024-04-26T00:24:01.993 INFO:tasks.rados.rados.0.smithi045.stdout:723: oid 29 version 304 is already newer than 303 2024-04-26T00:24:01.993 INFO:tasks.rados.rados.0.smithi045.stdout:723: oid 29 version 304 is already newer than 301 2024-04-26T00:24:01.993 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 304 (ObjNum 291 snap 85 seq_num 291) dirty exists 2024-04-26T00:24:01.993 INFO:tasks.rados.rados.0.smithi045.stdout:723: left oid 29 (ObjNum 291 snap 85 seq_num 291) 2024-04-26T00:24:02.011 INFO:tasks.rados.rados.0.smithi045.stdout:723: done (2 left) 2024-04-26T00:24:02.011 INFO:tasks.rados.rados.0.smithi045.stdout:726: done (1 left) 2024-04-26T00:24:02.011 INFO:tasks.rados.rados.0.smithi045.stdout:727: read oid 22 snap -1 2024-04-26T00:24:02.011 INFO:tasks.rados.rados.0.smithi045.stdout:727: expect (ObjNum 242 snap 61 seq_num 242) 2024-04-26T00:24:02.011 INFO:tasks.rados.rados.0.smithi045.stdout:728: read oid 12 snap -1 2024-04-26T00:24:02.011 INFO:tasks.rados.rados.0.smithi045.stdout:728: expect (ObjNum 180 snap 47 seq_num 180) 2024-04-26T00:24:02.011 INFO:tasks.rados.rados.0.smithi045.stdout:729: copy_from oid 44 from oid 6 current snap is 85 2024-04-26T00:24:02.012 INFO:tasks.rados.rados.0.smithi045.stdout:730: read oid 27 snap -1 2024-04-26T00:24:02.012 INFO:tasks.rados.rados.0.smithi045.stdout:730: expect (ObjNum 284 snap 81 seq_num 284) 2024-04-26T00:24:02.012 INFO:tasks.rados.rados.0.smithi045.stdout:731: snap_create 2024-04-26T00:24:02.020 INFO:tasks.rados.rados.0.smithi045.stdout:725: expect (ObjNum 163 snap 37 seq_num 163) 2024-04-26T00:24:02.202 INFO:tasks.rados.rados.0.smithi045.stdout:727: expect (ObjNum 242 snap 61 seq_num 242) 2024-04-26T00:24:02.268 INFO:tasks.rados.rados.0.smithi045.stdout:728: expect (ObjNum 180 snap 47 seq_num 180) 2024-04-26T00:24:02.415 INFO:tasks.rados.rados.0.smithi045.stdout:730: expect (ObjNum 284 snap 81 seq_num 284) 2024-04-26T00:24:02.573 INFO:tasks.rados.rados.0.smithi045.stdout:729: finishing copy_from racing read to smithi04526579-44 2024-04-26T00:24:02.573 INFO:tasks.rados.rados.0.smithi045.stdout:729: finishing copy_from to smithi04526579-44 2024-04-26T00:24:02.573 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 178 (ObjNum 283 snap 81 seq_num 283) dirty exists 2024-04-26T00:24:02.999 INFO:tasks.rados.rados.0.smithi045.stdout:725: done (5 left) 2024-04-26T00:24:02.999 INFO:tasks.rados.rados.0.smithi045.stdout:727: done (4 left) 2024-04-26T00:24:02.999 INFO:tasks.rados.rados.0.smithi045.stdout:728: done (3 left) 2024-04-26T00:24:02.999 INFO:tasks.rados.rados.0.smithi045.stdout:729: done (2 left) 2024-04-26T00:24:02.999 INFO:tasks.rados.rados.0.smithi045.stdout:730: done (1 left) 2024-04-26T00:24:02.999 INFO:tasks.rados.rados.0.smithi045.stdout:731: done (0 left) 2024-04-26T00:24:02.999 INFO:tasks.rados.rados.0.smithi045.stdout:732: copy_from oid 45 from oid 2 current snap is 86 2024-04-26T00:24:02.999 INFO:tasks.rados.rados.0.smithi045.stdout:733: write (excl) oid 40 current snap is 86 2024-04-26T00:24:02.999 INFO:tasks.rados.rados.0.smithi045.stdout:733: seq_num 293 ranges {524451=469371,1579224=483096,2529878=394573} 2024-04-26T00:24:03.015 INFO:tasks.rados.rados.0.smithi045.stdout:733: writing smithi04526579-40 from 524451 to 993822 tid 1 2024-04-26T00:24:03.034 INFO:tasks.rados.rados.0.smithi045.stdout:733: writing smithi04526579-40 from 1579224 to 2062320 tid 2 2024-04-26T00:24:03.048 INFO:tasks.rados.rados.0.smithi045.stdout:733: writing smithi04526579-40 from 2529878 to 2924451 tid 3 2024-04-26T00:24:03.049 INFO:tasks.rados.rados.0.smithi045.stdout:734: snap_remove snap 83 2024-04-26T00:24:03.049 INFO:tasks.rados.rados.0.smithi045.stdout:732: finishing copy_from racing read to smithi04526579-45 2024-04-26T00:24:03.049 INFO:tasks.rados.rados.0.smithi045.stdout:733: finishing write tid 1 to smithi04526579-40 2024-04-26T00:24:03.049 INFO:tasks.rados.rados.0.smithi045.stdout:732: finishing copy_from to smithi04526579-45 2024-04-26T00:24:03.049 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 299 (ObjNum 265 snap 74 seq_num 265) dirty exists 2024-04-26T00:24:03.049 INFO:tasks.rados.rados.0.smithi045.stdout:733: finishing write tid 2 to smithi04526579-40 2024-04-26T00:24:03.353 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:24:03.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc2684de640 10 monclient: get_monmap_and_config 2024-04-26T00:24:03.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc2684de640 10 monclient: build_initial_monmap 2024-04-26T00:24:03.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc2684de640 10 monclient: monmap: 2024-04-26T00:24:03.487 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:24:03.487 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:03.487 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:24:03.513208+0000 2024-04-26T00:24:03.488 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:24:03.513208+0000 2024-04-26T00:24:03.488 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:24:03.488 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:03.488 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:24:03.488 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:24:03.488 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:24:03.488 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:24:03.488 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:24:03.488 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:03.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc2684de640 1 Processor -- start 2024-04-26T00:24:03.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc2684de640 1 -- start start 2024-04-26T00:24:03.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc2684de640 10 monclient: init 2024-04-26T00:24:03.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc2684de640 10 monclient: _reopen_session rank -1 2024-04-26T00:24:03.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc2684de640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-04-26T00:24:03.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc2684de640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2600a0260 0x7fc2600a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:03.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc2684de640 10 monclient(hunting): picked mon.noname-b con 0x7fc2600a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:24:03.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc2684de640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc260134b30 addr v1:172.21.15.37:6789/0 2024-04-26T00:24:03.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc2684de640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc260135660 0x7fc26013ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:03.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc2684de640 10 monclient(hunting): picked mon.noname-c con 0x7fc260135660 addr v2:172.21.15.45:3300/0 2024-04-26T00:24:03.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc2684de640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2600a1130 con 0x7fc260134b30 2024-04-26T00:24:03.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc2684de640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:03.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc2684de640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc26009ec10 con 0x7fc260135660 2024-04-26T00:24:03.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc2684de640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:03.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc2684de640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc26009f0f0 con 0x7fc2600a0260 2024-04-26T00:24:03.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc2684de640 10 monclient(hunting): _renew_subs 2024-04-26T00:24:03.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc2684de640 10 monclient(hunting): authenticate will time out at 1224.818937s 2024-04-26T00:24:03.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc26627a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2600a0260 0x7fc2600a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:03.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc26627a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2600a0260 0x7fc2600a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:45924/0 (socket says 172.21.15.37:45924) 2024-04-26T00:24:03.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc265a79640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fc260134b30 0x7fc260134f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:52042/0 (socket says 172.21.15.37:52042) 2024-04-26T00:24:03.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc26627a640 1 -- 172.21.15.37:0/1449993317 learned_addr learned my addr 172.21.15.37:0/1449993317 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:24:03.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc26627a640 10 monclient(hunting): get_auth_request con 0x7fc2600a0260 auth_method 0 2024-04-26T00:24:03.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc26627a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:03.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc26627a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:03.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc26627a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:03.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc26627a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:03.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc26627a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:03.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc26627a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:03.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc266a7b640 1 --2- 172.21.15.37:0/1449993317 >> v2:172.21.15.45:3300/0 conn(0x7fc260135660 0x7fc26013ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:03.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc266a7b640 10 monclient(hunting): get_auth_request con 0x7fc260135660 auth_method 0 2024-04-26T00:24:03.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc266a7b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:03.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc266a7b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:03.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc266a7b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:03.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc26627a640 10 monclient(hunting): handle_auth_done global_id 34289 payload 274 2024-04-26T00:24:03.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc26627a640 1 -- 172.21.15.37:0/1449993317 >> v2:172.21.15.45:3300/0 conn(0x7fc260135660 msgr2=0x7fc26013ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:24:03.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc26627a640 1 --2- 172.21.15.37:0/1449993317 >> v2:172.21.15.45:3300/0 conn(0x7fc260135660 0x7fc26013ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:03.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc26627a640 1 -- 172.21.15.37:0/1449993317 >> v1:172.21.15.37:6789/0 conn(0x7fc260134b30 legacy=0x7fc260134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:24:03.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc26627a640 10 monclient: _finish_hunting 0 2024-04-26T00:24:03.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc26627a640 1 monclient: found mon.noname-b 2024-04-26T00:24:03.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc26627a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:24:03.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc26627a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:24:03.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.509+0000 7fc26627a640 1 -- 172.21.15.37:0/1449993317 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc26006cff0 con 0x7fc2600a0260 2024-04-26T00:24:03.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.513+0000 7fc26627a640 1 -- 172.21.15.37:0/1449993317 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2600a0260 msgr2=0x7fc2600a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:03.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc26627a640 1 --2- 172.21.15.37:0/1449993317 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2600a0260 0x7fc2600a0640 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc25c009870 tx=0x7fc25c01c200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b80f94bd2da0c145 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:03.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc265278640 1 -- 172.21.15.37:0/1449993317 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc25c0b0070 con 0x7fc2600a0260 2024-04-26T00:24:03.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc265278640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:03.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc265278640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:24:03.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc265278640 10 monclient: dump: 2024-04-26T00:24:03.495 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:03.495 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:03.495 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:03.496 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:03.496 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:03.496 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:03.496 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:03.496 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:03.496 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:03.496 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:03.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc265278640 10 monclient: _finish_auth 0 2024-04-26T00:24:03.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc265278640 10 monclient: _check_auth_tickets 2024-04-26T00:24:03.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc265278640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:24:03.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc265278640 1 -- 172.21.15.37:0/1449993317 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc25c0ab600 con 0x7fc2600a0260 2024-04-26T00:24:03.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc2684de640 5 monclient: authenticate success, global_id 34289 2024-04-26T00:24:03.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc2684de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:24:03.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc265278640 10 monclient: handle_config config(0 keys) 2024-04-26T00:24:03.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc2684de640 10 monclient: get_monmap_and_config success 2024-04-26T00:24:03.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc265278640 1 -- 172.21.15.37:0/1449993317 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc25c0ab9d0 con 0x7fc2600a0260 2024-04-26T00:24:03.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc265278640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:03.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc2684de640 4 set_mon_vals no callback set 2024-04-26T00:24:03.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc265278640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:24:03.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc265278640 10 monclient: dump: 2024-04-26T00:24:03.497 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:03.497 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:03.497 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:03.497 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:03.497 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:03.497 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:03.497 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:03.497 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:03.497 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:03.497 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:03.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc2684de640 10 monclient: shutdown 2024-04-26T00:24:03.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc2684de640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:24:03.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc2684de640 1 -- 172.21.15.37:0/1449993317 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2600a0260 msgr2=0x7fc2600a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:03.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc2684de640 1 --2- 172.21.15.37:0/1449993317 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2600a0260 0x7fc2600a0640 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc25c009870 tx=0x7fc25c01c200 comp rx=0 tx=0).stop 2024-04-26T00:24:03.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc2684de640 1 -- 172.21.15.37:0/1449993317 shutdown_connections 2024-04-26T00:24:03.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc2684de640 1 --2- 172.21.15.37:0/1449993317 >> v2:172.21.15.45:3300/0 conn(0x7fc260135660 0x7fc26013ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:03.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc2684de640 1 --2- 172.21.15.37:0/1449993317 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2600a0260 0x7fc2600a0640 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:03.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc2684de640 1 -- 172.21.15.37:0/1449993317 >> 172.21.15.37:0/1449993317 conn(0x7fc2600a1ad0 msgr2=0x7fc2600a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:24:03.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc2684de640 1 -- 172.21.15.37:0/1449993317 shutdown_connections 2024-04-26T00:24:03.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc2684de640 1 -- 172.21.15.37:0/1449993317 wait complete. 2024-04-26T00:24:03.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc2684de640 10 monclient: build_initial_monmap 2024-04-26T00:24:03.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc2684de640 10 monclient: monmap: 2024-04-26T00:24:03.498 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:24:03.498 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:24:03.498 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:24:03.498 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:24:03.498 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:24:03.499 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:24:03.499 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:24:03.499 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:24:03.499 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:24:03.499 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:03.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc2684de640 1 Processor -- start 2024-04-26T00:24:03.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc2684de640 1 -- start start 2024-04-26T00:24:03.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc2684de640 10 monclient: init 2024-04-26T00:24:03.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc2684de640 10 monclient: _reopen_session rank -1 2024-04-26T00:24:03.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc2684de640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:24:03.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc2684de640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2600a0260 0x7fc260070c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:03.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc2684de640 10 monclient(hunting): picked mon.noname-b con 0x7fc2600a0260 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:24:03.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc2684de640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc260134b30 0x7fc260131850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:03.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc2684de640 10 monclient(hunting): picked mon.noname-a con 0x7fc260134b30 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:24:03.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc2684de640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc260135660 0x7fc2601cebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:03.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc2684de640 10 monclient(hunting): picked mon.noname-c con 0x7fc260135660 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:24:03.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc2684de640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:03.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc2684de640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fc2600a0010 con 0x7fc260134b30 2024-04-26T00:24:03.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc2684de640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:03.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc2684de640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc26009fe90 con 0x7fc2600a0260 2024-04-26T00:24:03.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc2684de640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:03.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc2684de640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc26013fbd0 con 0x7fc260135660 2024-04-26T00:24:03.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc2684de640 10 monclient(hunting): _renew_subs 2024-04-26T00:24:03.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc2684de640 10 monclient(hunting): authenticate will time out at 1224.826287s 2024-04-26T00:24:03.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc265a79640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc260134b30 0x7fc260131850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:03.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc265a79640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc260134b30 0x7fc260131850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52600/0 (socket says 172.21.15.37:52600) 2024-04-26T00:24:03.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc265a79640 1 -- 172.21.15.37:0/4211686398 learned_addr learned my addr 172.21.15.37:0/4211686398 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:24:03.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc265a79640 10 monclient(hunting): get_auth_request con 0x7fc260134b30 auth_method 0 2024-04-26T00:24:03.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc265a79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:03.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc265a79640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:03.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc265a79640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:03.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc266a7b640 1 --2- 172.21.15.37:0/4211686398 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc260135660 0x7fc2601cebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:03.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc266a7b640 10 monclient(hunting): get_auth_request con 0x7fc260135660 auth_method 0 2024-04-26T00:24:03.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc266a7b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:03.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc266a7b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:03.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc266a7b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:03.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc265a79640 1 -- 172.21.15.37:0/4211686398 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc260134b30 msgr2=0x7fc260131850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:24:03.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc265a79640 1 -- 172.21.15.37:0/4211686398 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc260134b30 msgr2=0x7fc260131850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:24:03.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc266a7b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:03.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc266a7b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:03.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc265a79640 1 --2- 172.21.15.37:0/4211686398 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc260134b30 0x7fc260131850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:24:03.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc266a7b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:03.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc26627a640 1 --2- 172.21.15.37:0/4211686398 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2600a0260 0x7fc260070c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:03.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc265a79640 1 --2- 172.21.15.37:0/4211686398 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc260134b30 0x7fc260131850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:24:03.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc26627a640 10 monclient(hunting): get_auth_request con 0x7fc2600a0260 auth_method 0 2024-04-26T00:24:03.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc26627a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:03.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc26627a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:03.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc26627a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:03.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc266a7b640 10 monclient(hunting): handle_auth_done global_id 34292 payload 824 2024-04-26T00:24:03.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc266a7b640 1 -- 172.21.15.37:0/4211686398 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2600a0260 msgr2=0x7fc260070c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:03.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc266a7b640 1 --2- 172.21.15.37:0/4211686398 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2600a0260 0x7fc260070c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:03.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc266a7b640 1 -- 172.21.15.37:0/4211686398 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc260134b30 msgr2=0x7fc260131850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:24:03.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc266a7b640 1 --2- 172.21.15.37:0/4211686398 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc260134b30 0x7fc260131850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:03.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc266a7b640 10 monclient: _finish_hunting 0 2024-04-26T00:24:03.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc266a7b640 1 monclient: found mon.noname-c 2024-04-26T00:24:03.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc266a7b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:24:03.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc266a7b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:24:03.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc266a7b640 1 -- 172.21.15.37:0/4211686398 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2600718f0 con 0x7fc260135660 2024-04-26T00:24:03.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.517+0000 7fc26627a640 1 --2- 172.21.15.37:0/4211686398 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2600a0260 0x7fc260070c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:24:03.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.521+0000 7fc266a7b640 1 -- 172.21.15.37:0/4211686398 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc260135660 msgr2=0x7fc2601cebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:03.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.521+0000 7fc266a7b640 1 --2- 172.21.15.37:0/4211686398 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc260135660 0x7fc2601cebe0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc258007dc0 tx=0x7fc25800e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:03.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.521+0000 7fc252ffd640 1 -- 172.21.15.37:0/4211686398 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2580039a0 con 0x7fc260135660 2024-04-26T00:24:03.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.521+0000 7fc252ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:03.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.521+0000 7fc252ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:24:03.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.521+0000 7fc252ffd640 10 monclient: dump: 2024-04-26T00:24:03.506 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:03.506 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:03.506 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:03.506 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:03.506 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:03.506 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:03.506 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:03.506 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:03.506 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:03.506 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:03.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.521+0000 7fc252ffd640 10 monclient: _finish_auth 0 2024-04-26T00:24:03.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.521+0000 7fc252ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:24:03.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.521+0000 7fc252ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:24:03.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.521+0000 7fc2684de640 5 monclient: authenticate success, global_id 34292 2024-04-26T00:24:03.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.521+0000 7fc252ffd640 1 -- 172.21.15.37:0/4211686398 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc258003b60 con 0x7fc260135660 2024-04-26T00:24:03.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.521+0000 7fc252ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:24:03.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.521+0000 7fc252ffd640 1 -- 172.21.15.37:0/4211686398 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc258004020 con 0x7fc260135660 2024-04-26T00:24:03.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.521+0000 7fc252ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:03.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.521+0000 7fc252ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:24:03.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.521+0000 7fc252ffd640 10 monclient: dump: 2024-04-26T00:24:03.507 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:03.507 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:03.507 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:03.507 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:03.507 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:03.507 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:03.507 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:03.507 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:03.507 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:03.508 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:03.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.521+0000 7fc253fff640 4 set_mon_vals no callback set 2024-04-26T00:24:03.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.521+0000 7fc2684de640 10 monclient: _renew_subs 2024-04-26T00:24:03.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.521+0000 7fc2684de640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:24:03.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.521+0000 7fc2684de640 1 -- 172.21.15.37:0/4211686398 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc260134410 con 0x7fc260135660 2024-04-26T00:24:03.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.521+0000 7fc2684de640 10 monclient: _renew_subs 2024-04-26T00:24:03.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.521+0000 7fc2684de640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:24:03.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.521+0000 7fc2684de640 1 -- 172.21.15.37:0/4211686398 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc26011ca70 con 0x7fc260135660 2024-04-26T00:24:03.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.521+0000 7fc2684de640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:24:03.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.521+0000 7fc2684de640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:24:03.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.521+0000 7fc2684de640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:24:03.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.521+0000 7fc2684de640 1 -- 172.21.15.37:0/4211686398 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc260066f40 con 0x7fc260135660 2024-04-26T00:24:03.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.521+0000 7fc252ffd640 1 -- 172.21.15.37:0/4211686398 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc25801b020 con 0x7fc260135660 2024-04-26T00:24:03.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.521+0000 7fc252ffd640 1 --2- 172.21.15.37:0/4211686398 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc244035f00 0x7fc2440383c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:03.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.521+0000 7fc252ffd640 1 -- 172.21.15.37:0/4211686398 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(174..174 src has 1..174) ==== 3915+0+0 (secure 0 0 0) 0x7fc258048d30 con 0x7fc260135660 2024-04-26T00:24:03.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.521+0000 7fc26627a640 1 --2- 172.21.15.37:0/4211686398 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc244035f00 0x7fc2440383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:03.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.521+0000 7fc26627a640 10 monclient: get_auth_request con 0x7fc244035f00 auth_method 0 2024-04-26T00:24:03.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.525+0000 7fc26627a640 1 --2- 172.21.15.37:0/4211686398 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc244035f00 0x7fc2440383c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc26006b640 tx=0x7fc25c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:03.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.525+0000 7fc252ffd640 1 -- 172.21.15.37:0/4211686398 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc25801cc90 con 0x7fc260135660 2024-04-26T00:24:03.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.525+0000 7fc252ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:24:03.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.525+0000 7fc252ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:24:03.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.749+0000 7fc2684de640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:24:03.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.749+0000 7fc2684de640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:24:03.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.749+0000 7fc2684de640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:24:03.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.749+0000 7fc2684de640 1 -- 172.21.15.37:0/4211686398 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc26006b9a0 con 0x7fc260135660 2024-04-26T00:24:03.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.749+0000 7fc252ffd640 1 -- 172.21.15.37:0/4211686398 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fc258014420 con 0x7fc260135660 2024-04-26T00:24:03.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.749+0000 7fc252ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:24:03.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.749+0000 7fc252ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:24:03.729 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":34,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:24:03.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.753+0000 7fc2684de640 1 -- 172.21.15.37:0/4211686398 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc244035f00 msgr2=0x7fc2440383c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:03.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.753+0000 7fc2684de640 1 --2- 172.21.15.37:0/4211686398 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc244035f00 0x7fc2440383c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc26006b640 tx=0x7fc25c008040 comp rx=0 tx=0).stop 2024-04-26T00:24:03.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.753+0000 7fc2684de640 10 monclient: shutdown 2024-04-26T00:24:03.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.753+0000 7fc2684de640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:24:03.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.753+0000 7fc2684de640 1 -- 172.21.15.37:0/4211686398 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc260135660 msgr2=0x7fc2601cebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:03.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.753+0000 7fc2684de640 1 --2- 172.21.15.37:0/4211686398 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc260135660 0x7fc2601cebe0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc258007dc0 tx=0x7fc25800e4b0 comp rx=0 tx=0).stop 2024-04-26T00:24:03.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.753+0000 7fc2684de640 1 -- 172.21.15.37:0/4211686398 shutdown_connections 2024-04-26T00:24:03.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.753+0000 7fc2684de640 1 --2- 172.21.15.37:0/4211686398 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc244035f00 0x7fc2440383c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:03.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.753+0000 7fc2684de640 1 --2- 172.21.15.37:0/4211686398 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc260135660 0x7fc2601cebe0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:03.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.753+0000 7fc2684de640 1 --2- 172.21.15.37:0/4211686398 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc260134b30 0x7fc260131850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:03.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.753+0000 7fc2684de640 1 --2- 172.21.15.37:0/4211686398 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2600a0260 0x7fc260070c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:03.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.753+0000 7fc2684de640 1 -- 172.21.15.37:0/4211686398 >> 172.21.15.37:0/4211686398 conn(0x7fc2600a1ad0 msgr2=0x7fc2600a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:24:03.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.753+0000 7fc2684de640 1 -- 172.21.15.37:0/4211686398 shutdown_connections 2024-04-26T00:24:03.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.753+0000 7fc2684de640 1 -- 172.21.15.37:0/4211686398 wait complete. 2024-04-26T00:24:03.748 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:24:03.748 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:24:03.748 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:24:03.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.897+0000 7f3f74552640 10 monclient: get_monmap_and_config 2024-04-26T00:24:03.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.897+0000 7f3f74552640 10 monclient: build_initial_monmap 2024-04-26T00:24:03.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.897+0000 7f3f74552640 10 monclient: monmap: 2024-04-26T00:24:03.876 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:24:03.876 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:03.876 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:24:03.902168+0000 2024-04-26T00:24:03.877 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:24:03.902168+0000 2024-04-26T00:24:03.877 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:24:03.877 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:03.877 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:24:03.877 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:24:03.877 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:24:03.877 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:24:03.877 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:24:03.877 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:03.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.897+0000 7f3f74552640 1 Processor -- start 2024-04-26T00:24:03.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.897+0000 7f3f74552640 1 -- start start 2024-04-26T00:24:03.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.897+0000 7f3f74552640 10 monclient: init 2024-04-26T00:24:03.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.897+0000 7f3f74552640 10 monclient: _reopen_session rank -1 2024-04-26T00:24:03.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.897+0000 7f3f74552640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-04-26T00:24:03.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.897+0000 7f3f74552640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3f6c130eb0 addr v1:172.21.15.45:6789/0 2024-04-26T00:24:03.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.897+0000 7f3f74552640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f3f6c09ef20 0x7f3f6c138eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:03.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.897+0000 7f3f74552640 10 monclient(hunting): picked mon.noname-a con 0x7f3f6c09ef20 addr v2:172.21.15.37:3300/0 2024-04-26T00:24:03.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.897+0000 7f3f74552640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3f6c1393f0 addr v1:172.21.15.37:6789/0 2024-04-26T00:24:03.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.897+0000 7f3f74552640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f6c09c190 con 0x7f3f6c1393f0 2024-04-26T00:24:03.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.897+0000 7f3f74552640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f6c0a11c0 con 0x7f3f6c130eb0 2024-04-26T00:24:03.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.897+0000 7f3f74552640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:03.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.897+0000 7f3f74552640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f3f6c09df60 con 0x7f3f6c09ef20 2024-04-26T00:24:03.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.897+0000 7f3f74552640 10 monclient(hunting): _renew_subs 2024-04-26T00:24:03.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.897+0000 7f3f74552640 10 monclient(hunting): authenticate will time out at 1225.207836s 2024-04-26T00:24:03.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.897+0000 7f3f72aef640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f3f6c1393f0 0x7f3f6c13b7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:52054/0 (socket says 172.21.15.37:52054) 2024-04-26T00:24:03.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f72aef640 1 -- 172.21.15.37:0/3411959312 learned_addr learned my addr 172.21.15.37:0/3411959312 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:24:03.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f71aed640 1 --2- 172.21.15.37:0/3411959312 >> v2:172.21.15.37:3300/0 conn(0x7f3f6c09ef20 0x7f3f6c138eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:03.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f71aed640 10 monclient(hunting): get_auth_request con 0x7f3f6c09ef20 auth_method 0 2024-04-26T00:24:03.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f71aed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:03.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f71aed640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:03.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f71aed640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:03.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f71aed640 1 -- 172.21.15.37:0/3411959312 >> v2:172.21.15.37:3300/0 conn(0x7f3f6c09ef20 msgr2=0x7f3f6c138eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:24:03.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f71aed640 1 -- 172.21.15.37:0/3411959312 >> v2:172.21.15.37:3300/0 conn(0x7f3f6c09ef20 msgr2=0x7f3f6c138eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:24:03.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f71aed640 1 --2- 172.21.15.37:0/3411959312 >> v2:172.21.15.37:3300/0 conn(0x7f3f6c09ef20 0x7f3f6c138eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:24:03.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f71aed640 1 --2- 172.21.15.37:0/3411959312 >> v2:172.21.15.37:3300/0 conn(0x7f3f6c09ef20 0x7f3f6c138eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:24:03.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f72aef640 1 -- 172.21.15.37:0/3411959312 >> v1:172.21.15.37:6789/0 conn(0x7f3f6c1393f0 legacy=0x7f3f6c13b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:03.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f722ee640 1 -- 172.21.15.37:0/3411959312 >> v1:172.21.15.45:6789/0 conn(0x7f3f6c130eb0 legacy=0x7f3f6c131290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:03.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f712ec640 1 -- 172.21.15.37:0/3411959312 <== mon.1 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1307880918 0 0) 0x7f3f6c0a11c0 con 0x7f3f6c130eb0 2024-04-26T00:24:03.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f712ec640 10 monclient(hunting): _negotiate 2024-04-26T00:24:03.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f712ec640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:03.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f712ec640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:03.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f712ec640 10 monclient(hunting): my global_id is 24349 2024-04-26T00:24:03.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f712ec640 1 -- 172.21.15.37:0/3411959312 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f54003530 con 0x7f3f6c130eb0 2024-04-26T00:24:03.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f712ec640 1 -- 172.21.15.37:0/3411959312 <== mon.1 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2242706435 0 0) 0x7f3f54003530 con 0x7f3f6c130eb0 2024-04-26T00:24:03.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f712ec640 1 -- 172.21.15.37:0/3411959312 >> v2:172.21.15.37:3300/0 conn(0x7f3f6c09ef20 msgr2=0x7f3f6c138eb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:24:03.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f712ec640 1 --2- 172.21.15.37:0/3411959312 >> v2:172.21.15.37:3300/0 conn(0x7f3f6c09ef20 0x7f3f6c138eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:03.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f712ec640 1 -- 172.21.15.37:0/3411959312 >> v1:172.21.15.37:6789/0 conn(0x7f3f6c1393f0 legacy=0x7f3f6c13b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:03.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f712ec640 10 monclient: _finish_hunting 0 2024-04-26T00:24:03.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f712ec640 1 monclient: found mon.noname-c-legacy 2024-04-26T00:24:03.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f712ec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:24:03.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f712ec640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.45:6789/0 2024-04-26T00:24:03.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f712ec640 1 -- 172.21.15.37:0/3411959312 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f6c09dc80 con 0x7f3f6c130eb0 2024-04-26T00:24:03.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f712ec640 10 monclient: _finish_auth 0 2024-04-26T00:24:03.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f712ec640 10 monclient: _check_auth_tickets 2024-04-26T00:24:03.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f712ec640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:24:03.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f712ec640 1 -- 172.21.15.37:0/3411959312 <== mon.1 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 866326386 0 0) 0x7f3f64001d00 con 0x7f3f6c130eb0 2024-04-26T00:24:03.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f712ec640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:03.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f712ec640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-04-26T00:24:03.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f712ec640 10 monclient: dump: 2024-04-26T00:24:03.885 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:03.885 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:03.885 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:03.885 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:03.885 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:03.885 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:03.886 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:03.886 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:03.886 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:03.886 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:03.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f712ec640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] but i'm connected to v1:172.21.15.45:6789/0, reconnecting 2024-04-26T00:24:03.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f712ec640 10 monclient: _reopen_session rank -1 2024-04-26T00:24:03.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f712ec640 1 -- 172.21.15.37:0/3411959312 >> v1:172.21.15.45:6789/0 conn(0x7f3f6c130eb0 legacy=0x7f3f6c131290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:03.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f712ec640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:24:03.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f712ec640 1 --2- 172.21.15.37:0/3411959312 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f54003a90 0x7f3f54003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:03.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f712ec640 10 monclient(hunting): picked mon.a con 0x7f3f54003a90 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:24:03.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f712ec640 1 --2- 172.21.15.37:0/3411959312 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f54004810 0x7f3f54004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:03.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f712ec640 10 monclient(hunting): picked mon.c con 0x7f3f54004810 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:24:03.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f712ec640 1 --2- 172.21.15.37:0/3411959312 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f54005420 0x7f3f5400ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:03.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f712ec640 10 monclient(hunting): picked mon.b con 0x7f3f54005420 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:24:03.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f712ec640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:03.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f712ec640 1 -- 172.21.15.37:0/3411959312 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f3f6c09df60 con 0x7f3f54003a90 2024-04-26T00:24:03.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f712ec640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:03.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f712ec640 1 -- 172.21.15.37:0/3411959312 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3f5400c330 con 0x7f3f54005420 2024-04-26T00:24:03.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f712ec640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:03.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f712ec640 1 -- 172.21.15.37:0/3411959312 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f3f5400c530 con 0x7f3f54004810 2024-04-26T00:24:03.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f712ec640 10 monclient(hunting): _renew_subs 2024-04-26T00:24:03.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f712ec640 10 monclient(hunting): _finish_auth 0 2024-04-26T00:24:03.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f722ee640 1 --2- 172.21.15.37:0/3411959312 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f54003a90 0x7f3f54003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:03.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f71aed640 1 --2- 172.21.15.37:0/3411959312 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f54004810 0x7f3f54004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:03.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f722ee640 10 monclient(hunting): get_auth_request con 0x7f3f54003a90 auth_method 0 2024-04-26T00:24:03.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f722ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:03.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f722ee640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:03.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f722ee640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:24:03.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f71aed640 10 monclient(hunting): get_auth_request con 0x7f3f54004810 auth_method 0 2024-04-26T00:24:03.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f71aed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:03.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f71aed640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:03.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f71aed640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:24:03.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f722ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:03.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f722ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:03.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f722ee640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:24:03.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f71aed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:03.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f71aed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:03.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f72aef640 1 --2- 172.21.15.37:0/3411959312 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f54005420 0x7f3f5400ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:03.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f71aed640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:24:03.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f72aef640 10 monclient(hunting): get_auth_request con 0x7f3f54005420 auth_method 0 2024-04-26T00:24:03.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f72aef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:03.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f72aef640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:03.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f72aef640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:24:03.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f71aed640 10 monclient(hunting): handle_auth_done global_id 24349 payload 293 2024-04-26T00:24:03.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f71aed640 1 -- 172.21.15.37:0/3411959312 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f54005420 msgr2=0x7f3f5400ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:24:03.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f71aed640 1 --2- 172.21.15.37:0/3411959312 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f54005420 0x7f3f5400ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:03.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f71aed640 1 -- 172.21.15.37:0/3411959312 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f54003a90 msgr2=0x7f3f54003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:24:03.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f71aed640 1 --2- 172.21.15.37:0/3411959312 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f54003a90 0x7f3f54003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:03.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f71aed640 10 monclient: _finish_hunting 0 2024-04-26T00:24:03.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f71aed640 1 monclient: found mon.c 2024-04-26T00:24:03.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f71aed640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:24:03.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f71aed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:24:03.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f71aed640 1 -- 172.21.15.37:0/3411959312 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f54003530 con 0x7f3f54004810 2024-04-26T00:24:03.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f71aed640 10 monclient: _finish_auth 0 2024-04-26T00:24:03.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f71aed640 10 monclient: _check_auth_tickets 2024-04-26T00:24:03.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f71aed640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:24:03.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f722ee640 1 --2- 172.21.15.37:0/3411959312 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f54003a90 0x7f3f54003eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:24:03.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f74552640 5 monclient: authenticate success, global_id 24349 2024-04-26T00:24:03.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f74552640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:24:03.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.901+0000 7f3f72aef640 1 -- 172.21.15.37:0/3411959312 reap_dead start 2024-04-26T00:24:03.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f71aed640 1 -- 172.21.15.37:0/3411959312 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f54004810 msgr2=0x7f3f54004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:03.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f71aed640 1 --2- 172.21.15.37:0/3411959312 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f54004810 0x7f3f54004c90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3f60010d50 tx=0x7f3f600207c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f2fa9e9e7bb43e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:03.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f712ec640 1 -- 172.21.15.37:0/3411959312 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f600109d0 con 0x7f3f54004810 2024-04-26T00:24:03.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f712ec640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:03.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f712ec640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:24:03.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f712ec640 10 monclient: dump: 2024-04-26T00:24:03.896 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:03.896 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:03.896 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:03.897 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:03.897 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:03.897 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:03.897 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:03.897 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:03.897 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:03.897 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:03.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f712ec640 1 -- 172.21.15.37:0/3411959312 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f600102d0 con 0x7f3f54004810 2024-04-26T00:24:03.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f74552640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:24:03.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f712ec640 10 monclient: handle_config config(0 keys) 2024-04-26T00:24:03.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f74552640 10 monclient: get_monmap_and_config success 2024-04-26T00:24:03.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f712ec640 1 -- 172.21.15.37:0/3411959312 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f600a9790 con 0x7f3f54004810 2024-04-26T00:24:03.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f712ec640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:03.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f74552640 4 set_mon_vals no callback set 2024-04-26T00:24:03.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f712ec640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:24:03.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f712ec640 10 monclient: dump: 2024-04-26T00:24:03.898 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:03.898 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:03.898 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:03.898 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:03.898 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:03.898 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:03.898 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:03.898 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:03.898 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:03.898 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:03.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f74552640 10 monclient: shutdown 2024-04-26T00:24:03.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f74552640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:24:03.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f74552640 1 -- 172.21.15.37:0/3411959312 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f54004810 msgr2=0x7f3f54004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:03.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f74552640 1 --2- 172.21.15.37:0/3411959312 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f54004810 0x7f3f54004c90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3f60010d50 tx=0x7f3f600207c0 comp rx=0 tx=0).stop 2024-04-26T00:24:03.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f74552640 1 -- 172.21.15.37:0/3411959312 shutdown_connections 2024-04-26T00:24:03.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f74552640 1 --2- 172.21.15.37:0/3411959312 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f54004810 0x7f3f54004c90 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:03.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f74552640 1 -- 172.21.15.37:0/3411959312 >> 172.21.15.37:0/3411959312 conn(0x7f3f6c0a1d50 msgr2=0x7f3f6c0a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:24:03.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f74552640 1 -- 172.21.15.37:0/3411959312 shutdown_connections 2024-04-26T00:24:03.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f74552640 1 -- 172.21.15.37:0/3411959312 wait complete. 2024-04-26T00:24:03.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f74552640 10 monclient: build_initial_monmap 2024-04-26T00:24:03.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f74552640 10 monclient: monmap: 2024-04-26T00:24:03.899 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:24:03.899 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:24:03.899 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:24:03.899 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:24:03.899 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:24:03.900 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:24:03.900 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:24:03.900 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:24:03.900 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:24:03.900 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:03.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f74552640 1 Processor -- start 2024-04-26T00:24:03.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f74552640 1 -- start start 2024-04-26T00:24:03.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f74552640 10 monclient: init 2024-04-26T00:24:03.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f74552640 10 monclient: _reopen_session rank -1 2024-04-26T00:24:03.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f74552640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:24:03.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f74552640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f54004810 0x7f3f6c1cafb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:03.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f74552640 10 monclient(hunting): picked mon.noname-c con 0x7f3f54004810 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:24:03.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f74552640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f6c071500 0x7f3f6c12de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:03.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f74552640 10 monclient(hunting): picked mon.noname-a con 0x7f3f6c071500 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:24:03.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f74552640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f6c1c7a70 0x7f3f6c1c7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:03.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f74552640 10 monclient(hunting): picked mon.noname-b con 0x7f3f6c1c7a70 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:24:03.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f74552640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:03.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f74552640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f3f6c09bdd0 con 0x7f3f6c071500 2024-04-26T00:24:03.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f74552640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:03.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f74552640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3f6c09bc50 con 0x7f3f6c1c7a70 2024-04-26T00:24:03.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f74552640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:03.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f74552640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f3f6c1343a0 con 0x7f3f54004810 2024-04-26T00:24:03.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f74552640 10 monclient(hunting): _renew_subs 2024-04-26T00:24:03.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f74552640 10 monclient(hunting): authenticate will time out at 1225.218945s 2024-04-26T00:24:03.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f722ee640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f54004810 0x7f3f6c1cafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:03.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f71aed640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f6c071500 0x7f3f6c12de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:03.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f722ee640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f54004810 0x7f3f6c1cafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:45952/0 (socket says 172.21.15.37:45952) 2024-04-26T00:24:03.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f71aed640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f6c071500 0x7f3f6c12de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52632/0 (socket says 172.21.15.37:52632) 2024-04-26T00:24:03.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f722ee640 1 -- 172.21.15.37:0/462270175 learned_addr learned my addr 172.21.15.37:0/462270175 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:24:03.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f722ee640 10 monclient(hunting): get_auth_request con 0x7f3f54004810 auth_method 0 2024-04-26T00:24:03.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f722ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:03.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f722ee640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:03.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f722ee640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:03.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f71aed640 10 monclient(hunting): get_auth_request con 0x7f3f6c071500 auth_method 0 2024-04-26T00:24:03.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f71aed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:03.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f71aed640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:03.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f71aed640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:03.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f722ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:03.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f722ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:03.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f722ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:03.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f71aed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:03.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f71aed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:03.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f71aed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:03.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f72aef640 1 --2- 172.21.15.37:0/462270175 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f6c1c7a70 0x7f3f6c1c7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:03.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f72aef640 10 monclient(hunting): get_auth_request con 0x7f3f6c1c7a70 auth_method 0 2024-04-26T00:24:03.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f72aef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:03.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f72aef640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:03.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f72aef640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:03.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f71aed640 10 monclient(hunting): handle_auth_done global_id 54099 payload 824 2024-04-26T00:24:03.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f71aed640 1 -- 172.21.15.37:0/462270175 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f54004810 msgr2=0x7f3f6c1cafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:03.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f71aed640 1 --2- 172.21.15.37:0/462270175 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f54004810 0x7f3f6c1cafb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:03.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f71aed640 1 -- 172.21.15.37:0/462270175 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f6c1c7a70 msgr2=0x7f3f6c1c7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:03.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f71aed640 1 --2- 172.21.15.37:0/462270175 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f6c1c7a70 0x7f3f6c1c7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:03.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f71aed640 10 monclient: _finish_hunting 0 2024-04-26T00:24:03.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f71aed640 1 monclient: found mon.noname-a 2024-04-26T00:24:03.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f71aed640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:24:03.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f71aed640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:24:03.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f71aed640 1 -- 172.21.15.37:0/462270175 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f6c0701f0 con 0x7f3f6c071500 2024-04-26T00:24:03.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.909+0000 7f3f722ee640 1 --2- 172.21.15.37:0/462270175 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f54004810 0x7f3f6c1cafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:24:03.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f71aed640 1 -- 172.21.15.37:0/462270175 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f6c071500 msgr2=0x7f3f6c12de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:03.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f71aed640 1 --2- 172.21.15.37:0/462270175 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f6c071500 0x7f3f6c12de30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3f5400c700 tx=0x7f3f54005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:03.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f6affd640 1 -- 172.21.15.37:0/462270175 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f600c0020 con 0x7f3f6c071500 2024-04-26T00:24:03.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:03.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f6affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:24:03.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f6affd640 10 monclient: dump: 2024-04-26T00:24:03.904 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:03.904 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:03.904 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:03.904 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:03.905 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:03.905 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:03.905 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:03.905 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:03.905 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:03.905 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:03.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f6affd640 10 monclient: _finish_auth 0 2024-04-26T00:24:03.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f6affd640 10 monclient: _check_auth_tickets 2024-04-26T00:24:03.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:24:03.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f6affd640 1 -- 172.21.15.37:0/462270175 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f600a9ce0 con 0x7f3f6c071500 2024-04-26T00:24:03.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f74552640 5 monclient: authenticate success, global_id 54099 2024-04-26T00:24:03.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f74552640 10 monclient: _renew_subs 2024-04-26T00:24:03.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f74552640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:24:03.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f74552640 1 -- 172.21.15.37:0/462270175 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f6c1be4e0 con 0x7f3f6c071500 2024-04-26T00:24:03.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f6affd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:24:03.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f6affd640 1 -- 172.21.15.37:0/462270175 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f60005be0 con 0x7f3f6c071500 2024-04-26T00:24:03.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:03.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f6affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:24:03.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f6affd640 10 monclient: dump: 2024-04-26T00:24:03.906 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:03.906 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:03.906 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:03.906 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:03.906 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:03.906 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:03.906 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:03.906 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:03.906 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:03.907 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:03.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f74552640 10 monclient: _renew_subs 2024-04-26T00:24:03.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f74552640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:24:03.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f74552640 1 -- 172.21.15.37:0/462270175 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f6c0723f0 con 0x7f3f6c071500 2024-04-26T00:24:03.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f70aeb640 4 set_mon_vals no callback set 2024-04-26T00:24:03.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f74552640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:24:03.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f74552640 1 --2- 172.21.15.37:0/462270175 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f6c1c8460 0x7f3f6c136f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:03.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f74552640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:03.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f74552640 1 -- 172.21.15.37:0/462270175 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f3f6c04f510 con 0x7f3f6c1c8460 2024-04-26T00:24:03.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f6affd640 1 -- 172.21.15.37:0/462270175 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f3f6000e4e0 con 0x7f3f6c071500 2024-04-26T00:24:03.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f722ee640 1 --2- 172.21.15.37:0/462270175 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f6c1c8460 0x7f3f6c136f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:03.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f722ee640 10 monclient: get_auth_request con 0x7f3f6c1c8460 auth_method 0 2024-04-26T00:24:03.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f722ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:03.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f722ee640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:03.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f722ee640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:03.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f6affd640 1 --2- 172.21.15.37:0/462270175 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3f58035ed0 0x7f3f58038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:03.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f722ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:03.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f722ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:03.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f722ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:03.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f6affd640 1 -- 172.21.15.37:0/462270175 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 3915+0+0 (secure 0 0 0) 0x7f3f600e6c80 con 0x7f3f6c071500 2024-04-26T00:24:03.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f722ee640 10 monclient(hunting): handle_auth_done global_id 54102 payload 824 2024-04-26T00:24:03.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f722ee640 1 -- 172.21.15.37:0/462270175 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3f6c072630 con 0x7f3f6c1c8460 2024-04-26T00:24:03.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f72aef640 1 --2- 172.21.15.37:0/462270175 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3f58035ed0 0x7f3f58038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:03.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.913+0000 7f3f72aef640 10 monclient: get_auth_request con 0x7f3f58035ed0 auth_method 0 2024-04-26T00:24:03.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.917+0000 7f3f722ee640 1 -- 172.21.15.37:0/462270175 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f6c1c8460 msgr2=0x7f3f6c136f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:03.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.917+0000 7f3f722ee640 1 --2- 172.21.15.37:0/462270175 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f6c1c8460 0x7f3f6c136f80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3f6400d3b0 tx=0x7f3f64007420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:03.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.917+0000 7f3f6affd640 1 -- 172.21.15.37:0/462270175 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f6400d3e0 con 0x7f3f6c1c8460 2024-04-26T00:24:03.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.917+0000 7f3f6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:03.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.917+0000 7f3f6affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:24:03.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.917+0000 7f3f6affd640 10 monclient: dump: 2024-04-26T00:24:03.909 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:03.909 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:03.909 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:03.909 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:03.909 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:03.909 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:03.909 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:03.909 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:03.909 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:03.909 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:03.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.917+0000 7f3f6affd640 1 -- 172.21.15.37:0/462270175 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3f64011a60 con 0x7f3f6c1c8460 2024-04-26T00:24:03.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.917+0000 7f3f6affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:24:03.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.917+0000 7f3f6affd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:24:03.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.917+0000 7f3f6affd640 1 -- 172.21.15.37:0/462270175 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f6c1c8460 msgr2=0x7f3f6c136f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:03.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.917+0000 7f3f6affd640 1 --2- 172.21.15.37:0/462270175 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f6c1c8460 0x7f3f6c136f80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3f6400d3b0 tx=0x7f3f64007420 comp rx=0 tx=0).stop 2024-04-26T00:24:03.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.917+0000 7f3f6affd640 1 -- 172.21.15.37:0/462270175 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f6c1c8460 msgr2=0x7f3f6c136f80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:24:03.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.917+0000 7f3f6affd640 1 --2- 172.21.15.37:0/462270175 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f6c1c8460 0x7f3f6c136f80 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3f6400d3b0 tx=0x7f3f64007420 comp rx=0 tx=0).stop 2024-04-26T00:24:03.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.917+0000 7f3f72aef640 1 --2- 172.21.15.37:0/462270175 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3f58035ed0 0x7f3f58038390 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c0764a0 tx=0x7f3f5c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:03.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.925+0000 7f3f74552640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:24:03.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.925+0000 7f3f74552640 1 --2- 172.21.15.37:0/462270175 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f6c1374c0 0x7f3f6c1378d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:03.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.925+0000 7f3f74552640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:03.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.925+0000 7f3f74552640 1 -- 172.21.15.37:0/462270175 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f3f6c12fc80 con 0x7f3f6c1374c0 2024-04-26T00:24:03.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.925+0000 7f3f722ee640 1 --2- 172.21.15.37:0/462270175 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f6c1374c0 0x7f3f6c1378d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:03.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.925+0000 7f3f722ee640 10 monclient: get_auth_request con 0x7f3f6c1374c0 auth_method 0 2024-04-26T00:24:03.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.925+0000 7f3f722ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:03.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.925+0000 7f3f722ee640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:03.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.925+0000 7f3f722ee640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:03.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.925+0000 7f3f722ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:03.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.925+0000 7f3f722ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:03.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.925+0000 7f3f722ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:03.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.925+0000 7f3f722ee640 10 monclient(hunting): handle_auth_done global_id 54105 payload 824 2024-04-26T00:24:03.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.925+0000 7f3f722ee640 1 -- 172.21.15.37:0/462270175 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3f6c0a0f60 con 0x7f3f6c1374c0 2024-04-26T00:24:03.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.925+0000 7f3f722ee640 1 -- 172.21.15.37:0/462270175 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f6c1374c0 msgr2=0x7f3f6c1378d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:03.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.925+0000 7f3f722ee640 1 --2- 172.21.15.37:0/462270175 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f6c1374c0 0x7f3f6c1378d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3f64001e70 tx=0x7f3f6400a0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:03.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.925+0000 7f3f6affd640 1 -- 172.21.15.37:0/462270175 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f64020990 con 0x7f3f6c1374c0 2024-04-26T00:24:03.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.925+0000 7f3f6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:03.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.925+0000 7f3f6affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:24:03.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.925+0000 7f3f6affd640 10 monclient: dump: 2024-04-26T00:24:03.913 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:03.913 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:03.913 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:03.913 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:03.913 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:03.913 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:03.913 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:03.913 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:03.914 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:03.914 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:03.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.925+0000 7f3f6affd640 1 -- 172.21.15.37:0/462270175 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7f3f64029450 con 0x7f3f6c1374c0 2024-04-26T00:24:03.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.925+0000 7f3f6affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:24:03.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.925+0000 7f3f6affd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:24:03.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.925+0000 7f3f6affd640 1 -- 172.21.15.37:0/462270175 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f6c1374c0 msgr2=0x7f3f6c1378d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:03.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.925+0000 7f3f6affd640 1 --2- 172.21.15.37:0/462270175 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f6c1374c0 0x7f3f6c1378d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3f64001e70 tx=0x7f3f6400a0e0 comp rx=0 tx=0).stop 2024-04-26T00:24:03.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.925+0000 7f3f6affd640 1 -- 172.21.15.37:0/462270175 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f6c1374c0 msgr2=0x7f3f6c1378d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:24:03.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.925+0000 7f3f6affd640 1 --2- 172.21.15.37:0/462270175 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f6c1374c0 0x7f3f6c1378d0 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3f64001e70 tx=0x7f3f6400a0e0 comp rx=0 tx=0).stop 2024-04-26T00:24:03.914 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:24:03.915 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:24:03.915 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:24:03.915 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:24:03.915 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 34, 2024-04-26T00:24:03.915 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 7251, 2024-04-26T00:24:03.915 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:24:03.915 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:24:03.915 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:24:03.915 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:24:03.915 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:03.915 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:24:03.915 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:24:03.915 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:24:03.916 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:24:03.916 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:24:03.916 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:24:03.916 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:24:03.916 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:24:03.916 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:24:03.916 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:24:03.916 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:24:03.916 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:24:03.916 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:24:03.916 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:24:03.916 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:24:03.916 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:03.916 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:24:03.917 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:24:03.917 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:24:03.917 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:24:03.917 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:24:03.917 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:24:03.917 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:24:03.917 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:24:03.917 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:24:03.917 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:24:03.917 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:24:03.917 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:24:03.917 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:24:03.917 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:03.918 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:03.918 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:24:03.918 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:24:03.918 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:24:03.918 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:24:03.918 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:24:03.918 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:24:03.918 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:24:03.918 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:24:03.918 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:24:03.918 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:24:03.918 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:24:03.918 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:24:03.919 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:24:03.919 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:24:03.919 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:24:03.919 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:24:03.919 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:24:03.919 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:24:03.919 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:24:03.919 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:24:03.919 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:24:03.919 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:24:03.919 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:24:03.919 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:24:03.919 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:24:03.919 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:24:03.920 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:24:03.920 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:24:03.920 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:03.920 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:24:03.920 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:03.920 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:24:03.920 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:03.920 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:24:03.920 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:24:03.920 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:24:03.920 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:24:03.920 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:03.920 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:24:03.920 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:24:03.921 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:03.921 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:03.921 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:03.921 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:24:03.921 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:24:03.921 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:03.921 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:03.921 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:03.921 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:03.921 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:24:03.921 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:24:03.921 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:24:03.921 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:24:03.921 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:24:03.921 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:03.922 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:03.922 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:24:03.922 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:24:03.922 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:24:03.922 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:24:03.922 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:03.922 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:24:03.922 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:24:03.922 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:03.922 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:03.922 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:03.922 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:24:03.922 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:24:03.922 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:03.923 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:03.923 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:03.923 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:03.923 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:24:03.923 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:24:03.923 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:24:03.923 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:24:03.923 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:24:03.923 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:03.923 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:03.923 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:24:03.923 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:24:03.923 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:24:03.923 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:24:03.924 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:03.924 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:24:03.924 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:24:03.924 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:03.924 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:03.924 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:03.924 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:24:03.924 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:24:03.924 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:03.924 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:03.924 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:03.924 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:03.924 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:24:03.924 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:24:03.924 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:24:03.925 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:24:03.925 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:24:03.925 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:03.925 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:03.925 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:03.925 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:24:03.925 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:24:03.925 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:03.925 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:24:03.925 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:24:03.925 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:24:03.925 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:03.925 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:03.925 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:24:03.926 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:03.926 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:24:03.926 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:24:03.926 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:24:03.926 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:03.926 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:03.926 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:03.926 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:24:03.926 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:24:03.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.925+0000 7f3f74552640 1 -- 172.21.15.37:0/462270175 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3f58035ed0 msgr2=0x7f3f58038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:03.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.925+0000 7f3f74552640 1 --2- 172.21.15.37:0/462270175 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3f58035ed0 0x7f3f58038390 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c0764a0 tx=0x7f3f5c009040 comp rx=0 tx=0).stop 2024-04-26T00:24:03.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.925+0000 7f3f74552640 10 monclient: shutdown 2024-04-26T00:24:03.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.925+0000 7f3f74552640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:24:03.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.925+0000 7f3f74552640 1 -- 172.21.15.37:0/462270175 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f6c071500 msgr2=0x7f3f6c12de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:03.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.925+0000 7f3f74552640 1 --2- 172.21.15.37:0/462270175 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f6c071500 0x7f3f6c12de30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3f5400c700 tx=0x7f3f54005840 comp rx=0 tx=0).stop 2024-04-26T00:24:03.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.925+0000 7f3f72aef640 1 -- 172.21.15.37:0/462270175 reap_dead start 2024-04-26T00:24:03.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.925+0000 7f3f72aef640 1 -- 172.21.15.37:0/462270175 reap_dead start 2024-04-26T00:24:03.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.925+0000 7f3f74552640 1 -- 172.21.15.37:0/462270175 shutdown_connections 2024-04-26T00:24:03.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.925+0000 7f3f74552640 1 -- 172.21.15.37:0/462270175 >> 172.21.15.37:0/462270175 conn(0x7f3f6c0a1d50 msgr2=0x7f3f6c1301a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:24:03.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.925+0000 7f3f74552640 1 -- 172.21.15.37:0/462270175 shutdown_connections 2024-04-26T00:24:03.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:03.925+0000 7f3f74552640 1 -- 172.21.15.37:0/462270175 wait complete. 2024-04-26T00:24:03.928 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:24:03.929 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:24:03.929 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:24:04.016 INFO:tasks.rados.rados.0.smithi045.stdout:732: done (2 left) 2024-04-26T00:24:04.016 INFO:tasks.rados.rados.0.smithi045.stdout:734: done (1 left) 2024-04-26T00:24:04.016 INFO:tasks.rados.rados.0.smithi045.stdout:735: delete oid 26 current snap is 86 2024-04-26T00:24:04.016 INFO:tasks.rados.rados.0.smithi045.stdout:733: finishing write tid 3 to smithi04526579-40 2024-04-26T00:24:04.016 INFO:tasks.rados.rados.0.smithi045.stdout:733: finishing write tid 4 to smithi04526579-40 2024-04-26T00:24:04.016 INFO:tasks.rados.rados.0.smithi045.stdout:733: finishing write tid 5 to smithi04526579-40 2024-04-26T00:24:04.016 INFO:tasks.rados.rados.0.smithi045.stdout:733: oid 40 updating version 0 to 296 2024-04-26T00:24:04.016 INFO:tasks.rados.rados.0.smithi045.stdout:733: oid 40 version 296 is already newer than 294 2024-04-26T00:24:04.016 INFO:tasks.rados.rados.0.smithi045.stdout:733: oid 40 version 296 is already newer than 295 2024-04-26T00:24:04.017 INFO:tasks.rados.rados.0.smithi045.stdout:733: oid 40 version 296 is already newer than 293 2024-04-26T00:24:04.017 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 296 (ObjNum 293 snap 86 seq_num 293) dirty exists 2024-04-26T00:24:04.017 INFO:tasks.rados.rados.0.smithi045.stdout:733: left oid 40 (ObjNum 293 snap 86 seq_num 293) 2024-04-26T00:24:04.030 INFO:tasks.rados.rados.0.smithi045.stdout:733: done (1 left) 2024-04-26T00:24:04.030 INFO:tasks.rados.rados.0.smithi045.stdout:735: done (0 left) 2024-04-26T00:24:04.031 INFO:tasks.rados.rados.0.smithi045.stdout:736: delete oid 40 current snap is 86 2024-04-26T00:24:04.035 INFO:tasks.rados.rados.0.smithi045.stdout:736: done (0 left) 2024-04-26T00:24:04.035 INFO:tasks.rados.rados.0.smithi045.stdout:737: read oid 7 snap -1 2024-04-26T00:24:04.035 INFO:tasks.rados.rados.0.smithi045.stdout:737: expect deleted 2024-04-26T00:24:04.036 INFO:tasks.rados.rados.0.smithi045.stdout:738: delete oid 45 current snap is 86 2024-04-26T00:24:04.040 INFO:tasks.rados.rados.0.smithi045.stdout:737: done (1 left) 2024-04-26T00:24:04.040 INFO:tasks.rados.rados.0.smithi045.stdout:738: done (0 left) 2024-04-26T00:24:04.040 INFO:tasks.rados.rados.0.smithi045.stdout:739: read oid 22 snap -1 2024-04-26T00:24:04.040 INFO:tasks.rados.rados.0.smithi045.stdout:739: expect (ObjNum 242 snap 61 seq_num 242) 2024-04-26T00:24:04.040 INFO:tasks.rados.rados.0.smithi045.stdout:740: copy_from oid 4 from oid 19 current snap is 86 2024-04-26T00:24:04.040 INFO:tasks.rados.rados.0.smithi045.stdout:741: snap_remove snap 51 2024-04-26T00:24:04.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e675ee640 10 monclient: get_monmap_and_config 2024-04-26T00:24:04.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e675ee640 10 monclient: build_initial_monmap 2024-04-26T00:24:04.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e675ee640 10 monclient: monmap: 2024-04-26T00:24:04.042 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:24:04.043 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:04.043 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:24:04.068566+0000 2024-04-26T00:24:04.043 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:24:04.068566+0000 2024-04-26T00:24:04.043 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:24:04.043 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:04.043 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:24:04.043 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:24:04.043 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:24:04.043 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:24:04.043 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:24:04.043 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:04.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e675ee640 1 Processor -- start 2024-04-26T00:24:04.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e675ee640 1 -- start start 2024-04-26T00:24:04.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e675ee640 10 monclient: init 2024-04-26T00:24:04.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e675ee640 10 monclient: _reopen_session rank -1 2024-04-26T00:24:04.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e675ee640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-04-26T00:24:04.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e675ee640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1e600a3d90 addr v1:172.21.15.37:6789/0 2024-04-26T00:24:04.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e675ee640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1e600a4170 0x7f1e600a45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:04.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e675ee640 10 monclient(hunting): picked mon.noname-c con 0x7f1e600a4170 addr v2:172.21.15.45:3300/0 2024-04-26T00:24:04.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e675ee640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f1e6013c3e0 0x7f1e6013e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:04.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e675ee640 10 monclient(hunting): picked mon.noname-a con 0x7f1e6013c3e0 addr v2:172.21.15.37:3300/0 2024-04-26T00:24:04.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e675ee640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e600a5150 con 0x7f1e600a3d90 2024-04-26T00:24:04.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e675ee640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:04.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e675ee640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f1e6009c650 con 0x7f1e6013c3e0 2024-04-26T00:24:04.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e675ee640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:04.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e675ee640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f1e6009b280 con 0x7f1e600a4170 2024-04-26T00:24:04.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e675ee640 10 monclient(hunting): _renew_subs 2024-04-26T00:24:04.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e675ee640 10 monclient(hunting): authenticate will time out at 1225.374045s 2024-04-26T00:24:04.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e65b8b640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f1e6013c3e0 0x7f1e6013e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:04.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e6538a640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f1e600a3d90 0x7f1e6009bf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:52056/0 (socket says 172.21.15.37:52056) 2024-04-26T00:24:04.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e6538a640 1 -- 172.21.15.37:0/214961716 learned_addr learned my addr 172.21.15.37:0/214961716 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:24:04.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e65b8b640 10 monclient(hunting): get_auth_request con 0x7f1e6013c3e0 auth_method 0 2024-04-26T00:24:04.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e65b8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:04.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e65b8b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:04.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e65b8b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:04.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e65b8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:04.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e65b8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:04.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e65b8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:04.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e64b89640 1 --2- 172.21.15.37:0/214961716 >> v2:172.21.15.45:3300/0 conn(0x7f1e600a4170 0x7f1e600a45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:04.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e64b89640 10 monclient(hunting): get_auth_request con 0x7f1e600a4170 auth_method 0 2024-04-26T00:24:04.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e64b89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:04.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e64b89640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:04.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e64b89640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:04.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e65b8b640 10 monclient(hunting): handle_auth_done global_id 54108 payload 274 2024-04-26T00:24:04.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e65b8b640 1 -- 172.21.15.37:0/214961716 >> v2:172.21.15.45:3300/0 conn(0x7f1e600a4170 msgr2=0x7f1e600a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:24:04.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e65b8b640 1 --2- 172.21.15.37:0/214961716 >> v2:172.21.15.45:3300/0 conn(0x7f1e600a4170 0x7f1e600a45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:04.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e65b8b640 1 -- 172.21.15.37:0/214961716 >> v1:172.21.15.37:6789/0 conn(0x7f1e600a3d90 legacy=0x7f1e6009bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:24:04.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e65b8b640 10 monclient: _finish_hunting 0 2024-04-26T00:24:04.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e65b8b640 1 monclient: found mon.noname-a 2024-04-26T00:24:04.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e65b8b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:24:04.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e65b8b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:24:04.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.065+0000 7f1e65b8b640 1 -- 172.21.15.37:0/214961716 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e6009b560 con 0x7f1e6013c3e0 2024-04-26T00:24:04.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.073+0000 7f1e65b8b640 1 -- 172.21.15.37:0/214961716 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1e6013c3e0 msgr2=0x7f1e6013e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:04.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.073+0000 7f1e65b8b640 1 --2- 172.21.15.37:0/214961716 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1e6013c3e0 0x7f1e6013e7d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1e6009d140 tx=0x7f1e5801c300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2231dca4665259c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:04.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.073+0000 7f1e5ffff640 1 -- 172.21.15.37:0/214961716 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e580b0070 con 0x7f1e6013c3e0 2024-04-26T00:24:04.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.073+0000 7f1e5ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:04.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.073+0000 7f1e5ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:24:04.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.073+0000 7f1e5ffff640 10 monclient: dump: 2024-04-26T00:24:04.051 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:04.051 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:04.051 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:04.052 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:04.052 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:04.052 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:04.052 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:04.052 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:04.052 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:04.052 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:04.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.073+0000 7f1e5ffff640 10 monclient: _finish_auth 0 2024-04-26T00:24:04.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.073+0000 7f1e5ffff640 10 monclient: _check_auth_tickets 2024-04-26T00:24:04.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.073+0000 7f1e5ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:24:04.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.073+0000 7f1e5ffff640 1 -- 172.21.15.37:0/214961716 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e580ab600 con 0x7f1e6013c3e0 2024-04-26T00:24:04.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.073+0000 7f1e675ee640 5 monclient: authenticate success, global_id 54108 2024-04-26T00:24:04.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.073+0000 7f1e675ee640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:24:04.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.073+0000 7f1e5ffff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:24:04.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.073+0000 7f1e675ee640 10 monclient: get_monmap_and_config success 2024-04-26T00:24:04.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.073+0000 7f1e5ffff640 1 -- 172.21.15.37:0/214961716 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e580ab9d0 con 0x7f1e6013c3e0 2024-04-26T00:24:04.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.073+0000 7f1e5ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:04.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.073+0000 7f1e675ee640 4 set_mon_vals no callback set 2024-04-26T00:24:04.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.073+0000 7f1e5ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:24:04.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.073+0000 7f1e5ffff640 10 monclient: dump: 2024-04-26T00:24:04.053 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:04.053 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:04.053 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:04.053 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:04.053 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:04.053 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:04.053 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:04.054 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:04.054 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:04.054 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:04.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.073+0000 7f1e675ee640 10 monclient: shutdown 2024-04-26T00:24:04.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.073+0000 7f1e675ee640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:24:04.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.073+0000 7f1e675ee640 1 -- 172.21.15.37:0/214961716 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1e6013c3e0 msgr2=0x7f1e6013e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:04.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.073+0000 7f1e675ee640 1 --2- 172.21.15.37:0/214961716 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1e6013c3e0 0x7f1e6013e7d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1e6009d140 tx=0x7f1e5801c300 comp rx=0 tx=0).stop 2024-04-26T00:24:04.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.073+0000 7f1e675ee640 1 -- 172.21.15.37:0/214961716 shutdown_connections 2024-04-26T00:24:04.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.073+0000 7f1e675ee640 1 --2- 172.21.15.37:0/214961716 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1e6013c3e0 0x7f1e6013e7d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:04.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.073+0000 7f1e675ee640 1 --2- 172.21.15.37:0/214961716 >> v2:172.21.15.45:3300/0 conn(0x7f1e600a4170 0x7f1e600a45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:04.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.073+0000 7f1e675ee640 1 -- 172.21.15.37:0/214961716 >> 172.21.15.37:0/214961716 conn(0x7f1e600aa0a0 msgr2=0x7f1e600a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:24:04.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.073+0000 7f1e675ee640 1 -- 172.21.15.37:0/214961716 shutdown_connections 2024-04-26T00:24:04.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.073+0000 7f1e675ee640 1 -- 172.21.15.37:0/214961716 wait complete. 2024-04-26T00:24:04.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e675ee640 10 monclient: build_initial_monmap 2024-04-26T00:24:04.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e675ee640 10 monclient: monmap: 2024-04-26T00:24:04.058 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:24:04.058 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:24:04.058 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:24:04.058 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:24:04.058 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:24:04.058 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:24:04.058 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:24:04.058 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:24:04.058 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:24:04.058 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:04.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e675ee640 1 Processor -- start 2024-04-26T00:24:04.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e675ee640 1 -- start start 2024-04-26T00:24:04.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e675ee640 10 monclient: init 2024-04-26T00:24:04.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e675ee640 10 monclient: _reopen_session rank -1 2024-04-26T00:24:04.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e675ee640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:24:04.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e675ee640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1e600a3d90 0x7f1e601ced80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:04.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e675ee640 10 monclient(hunting): picked mon.noname-a con 0x7f1e600a3d90 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:24:04.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e675ee640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e600a4170 0x7f1e601cf550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:04.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e675ee640 10 monclient(hunting): picked mon.noname-b con 0x7f1e600a4170 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:24:04.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e675ee640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1e6013c3e0 0x7f1e60139ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:04.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e675ee640 10 monclient(hunting): picked mon.noname-c con 0x7f1e6013c3e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:24:04.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e675ee640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:04.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e675ee640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f1e601418c0 con 0x7f1e600a3d90 2024-04-26T00:24:04.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e675ee640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:04.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e675ee640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1e60141740 con 0x7f1e600a4170 2024-04-26T00:24:04.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e675ee640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:04.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e675ee640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f1e60141a40 con 0x7f1e6013c3e0 2024-04-26T00:24:04.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e675ee640 10 monclient(hunting): _renew_subs 2024-04-26T00:24:04.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e675ee640 10 monclient(hunting): authenticate will time out at 1225.385508s 2024-04-26T00:24:04.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e6538a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1e600a3d90 0x7f1e601ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:04.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e6538a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1e600a3d90 0x7f1e601ced80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52664/0 (socket says 172.21.15.37:52664) 2024-04-26T00:24:04.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e6538a640 1 -- 172.21.15.37:0/2505182672 learned_addr learned my addr 172.21.15.37:0/2505182672 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:24:04.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e6538a640 10 monclient(hunting): get_auth_request con 0x7f1e600a3d90 auth_method 0 2024-04-26T00:24:04.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e6538a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:04.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e6538a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:04.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e6538a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:04.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e65b8b640 1 --2- 172.21.15.37:0/2505182672 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1e6013c3e0 0x7f1e60139ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:04.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e65b8b640 10 monclient(hunting): get_auth_request con 0x7f1e6013c3e0 auth_method 0 2024-04-26T00:24:04.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e65b8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:04.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e65b8b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:04.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e65b8b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:04.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e6538a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:04.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e6538a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:04.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e6538a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:04.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e65b8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:04.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e65b8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:04.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e65b8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:04.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e64b89640 1 --2- 172.21.15.37:0/2505182672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e600a4170 0x7f1e601cf550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:04.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e64b89640 10 monclient(hunting): get_auth_request con 0x7f1e600a4170 auth_method 0 2024-04-26T00:24:04.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e64b89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:04.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e64b89640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:04.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e64b89640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:04.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e6538a640 10 monclient(hunting): handle_auth_done global_id 54111 payload 824 2024-04-26T00:24:04.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e6538a640 1 -- 172.21.15.37:0/2505182672 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1e6013c3e0 msgr2=0x7f1e60139ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:04.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e6538a640 1 --2- 172.21.15.37:0/2505182672 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1e6013c3e0 0x7f1e60139ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:04.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e6538a640 1 -- 172.21.15.37:0/2505182672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e600a4170 msgr2=0x7f1e601cf550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:04.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e6538a640 1 --2- 172.21.15.37:0/2505182672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1e600a4170 0x7f1e601cf550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:04.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e6538a640 10 monclient: _finish_hunting 0 2024-04-26T00:24:04.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e6538a640 1 monclient: found mon.noname-a 2024-04-26T00:24:04.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e6538a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:24:04.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e6538a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:24:04.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.077+0000 7f1e6538a640 1 -- 172.21.15.37:0/2505182672 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e601218e0 con 0x7f1e600a3d90 2024-04-26T00:24:04.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e6538a640 1 -- 172.21.15.37:0/2505182672 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1e600a3d90 msgr2=0x7f1e601ced80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:04.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e6538a640 1 --2- 172.21.15.37:0/2505182672 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1e600a3d90 0x7f1e601ced80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1e5000b430 tx=0x7f1e5000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:04.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e5dffb640 1 -- 172.21.15.37:0/2505182672 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e50015020 con 0x7f1e600a3d90 2024-04-26T00:24:04.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e5dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:04.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e5dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:24:04.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e5dffb640 10 monclient: dump: 2024-04-26T00:24:04.066 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:04.066 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:04.066 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:04.066 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:04.066 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:04.066 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:04.066 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:04.066 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:04.067 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:04.067 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:04.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e5dffb640 10 monclient: _finish_auth 0 2024-04-26T00:24:04.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e5dffb640 10 monclient: _check_auth_tickets 2024-04-26T00:24:04.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e5dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:24:04.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e5dffb640 1 -- 172.21.15.37:0/2505182672 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e50011a60 con 0x7f1e600a3d90 2024-04-26T00:24:04.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e5dffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:24:04.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e5dffb640 1 -- 172.21.15.37:0/2505182672 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e50011e50 con 0x7f1e600a3d90 2024-04-26T00:24:04.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e5dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:04.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e5dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:24:04.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e5dffb640 10 monclient: dump: 2024-04-26T00:24:04.067 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:04.067 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:04.067 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:04.067 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:04.067 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:04.068 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:04.068 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:04.068 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:04.068 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:04.068 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:04.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e5f7fe640 4 set_mon_vals no callback set 2024-04-26T00:24:04.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e675ee640 5 monclient: authenticate success, global_id 54111 2024-04-26T00:24:04.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e675ee640 10 monclient: _renew_subs 2024-04-26T00:24:04.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e675ee640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:24:04.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e675ee640 1 -- 172.21.15.37:0/2505182672 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e6009ada0 con 0x7f1e600a3d90 2024-04-26T00:24:04.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e675ee640 10 monclient: _renew_subs 2024-04-26T00:24:04.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e675ee640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:24:04.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e675ee640 1 -- 172.21.15.37:0/2505182672 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e60121600 con 0x7f1e600a3d90 2024-04-26T00:24:04.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e675ee640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:24:04.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e675ee640 1 --2- 172.21.15.37:0/2505182672 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1e601cfa90 0x7f1e6013a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:04.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e675ee640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:04.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e675ee640 1 -- 172.21.15.37:0/2505182672 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f1e6004f510 con 0x7f1e601cfa90 2024-04-26T00:24:04.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e5dffb640 1 -- 172.21.15.37:0/2505182672 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f1e50018020 con 0x7f1e600a3d90 2024-04-26T00:24:04.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e64b89640 1 --2- 172.21.15.37:0/2505182672 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1e601cfa90 0x7f1e6013a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:04.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e64b89640 10 monclient: get_auth_request con 0x7f1e601cfa90 auth_method 0 2024-04-26T00:24:04.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e64b89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:04.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e64b89640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:04.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e64b89640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:04.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e64b89640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:04.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e64b89640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:04.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e64b89640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:04.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e5dffb640 1 --2- 172.21.15.37:0/2505182672 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1e4c036070 0x7f1e4c038530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:04.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e5dffb640 1 -- 172.21.15.37:0/2505182672 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(175..175 src has 1..175) ==== 3959+0+0 (secure 0 0 0) 0x7f1e50048a60 con 0x7f1e600a3d90 2024-04-26T00:24:04.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e64b89640 10 monclient(hunting): handle_auth_done global_id 34301 payload 824 2024-04-26T00:24:04.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e64b89640 1 -- 172.21.15.37:0/2505182672 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1e600008e0 con 0x7f1e601cfa90 2024-04-26T00:24:04.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e65b8b640 1 --2- 172.21.15.37:0/2505182672 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1e4c036070 0x7f1e4c038530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:04.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.081+0000 7f1e65b8b640 10 monclient: get_auth_request con 0x7f1e4c036070 auth_method 0 2024-04-26T00:24:04.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.085+0000 7f1e64b89640 1 -- 172.21.15.37:0/2505182672 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1e601cfa90 msgr2=0x7f1e6013a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:04.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.085+0000 7f1e64b89640 1 --2- 172.21.15.37:0/2505182672 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1e601cfa90 0x7f1e6013a530 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1e6009cf40 tx=0x7f1e54007700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:04.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.085+0000 7f1e5dffb640 1 -- 172.21.15.37:0/2505182672 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e54013780 con 0x7f1e601cfa90 2024-04-26T00:24:04.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.085+0000 7f1e5dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:04.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.085+0000 7f1e5dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:24:04.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.085+0000 7f1e5dffb640 10 monclient: dump: 2024-04-26T00:24:04.070 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:04.070 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:04.070 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:04.070 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:04.070 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:04.071 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:04.071 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:04.071 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:04.071 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:04.071 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:04.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.085+0000 7f1e5dffb640 1 -- 172.21.15.37:0/2505182672 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1e54019070 con 0x7f1e601cfa90 2024-04-26T00:24:04.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.085+0000 7f1e5dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:24:04.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.085+0000 7f1e5dffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:24:04.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.085+0000 7f1e5dffb640 1 -- 172.21.15.37:0/2505182672 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1e601cfa90 msgr2=0x7f1e6013a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:04.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.085+0000 7f1e5dffb640 1 --2- 172.21.15.37:0/2505182672 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1e601cfa90 0x7f1e6013a530 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1e6009cf40 tx=0x7f1e54007700 comp rx=0 tx=0).stop 2024-04-26T00:24:04.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.085+0000 7f1e5dffb640 1 -- 172.21.15.37:0/2505182672 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1e601cfa90 msgr2=0x7f1e6013a530 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:24:04.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.085+0000 7f1e5dffb640 1 --2- 172.21.15.37:0/2505182672 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1e601cfa90 0x7f1e6013a530 secure :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1e6009cf40 tx=0x7f1e54007700 comp rx=0 tx=0).stop 2024-04-26T00:24:04.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.085+0000 7f1e65b8b640 1 --2- 172.21.15.37:0/2505182672 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1e4c036070 0x7f1e4c038530 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1e600a57e0 tx=0x7f1e580a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:04.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.089+0000 7f1e675ee640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:24:04.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.089+0000 7f1e675ee640 1 --2- 172.21.15.37:0/2505182672 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1e60121040 0x7f1e6013aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:04.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.089+0000 7f1e675ee640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:04.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.089+0000 7f1e675ee640 1 -- 172.21.15.37:0/2505182672 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f1e60120150 con 0x7f1e60121040 2024-04-26T00:24:04.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.089+0000 7f1e64b89640 1 --2- 172.21.15.37:0/2505182672 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1e60121040 0x7f1e6013aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:04.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.089+0000 7f1e64b89640 10 monclient: get_auth_request con 0x7f1e60121040 auth_method 0 2024-04-26T00:24:04.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.089+0000 7f1e64b89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:04.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.089+0000 7f1e64b89640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:04.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.089+0000 7f1e64b89640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:04.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.089+0000 7f1e64b89640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:04.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.089+0000 7f1e64b89640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:04.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.089+0000 7f1e64b89640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:04.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.089+0000 7f1e64b89640 10 monclient(hunting): handle_auth_done global_id 34304 payload 824 2024-04-26T00:24:04.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.089+0000 7f1e64b89640 1 -- 172.21.15.37:0/2505182672 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1e601da0d0 con 0x7f1e60121040 2024-04-26T00:24:04.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.093+0000 7f1e64b89640 1 -- 172.21.15.37:0/2505182672 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1e60121040 msgr2=0x7f1e6013aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:04.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.093+0000 7f1e64b89640 1 --2- 172.21.15.37:0/2505182672 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1e60121040 0x7f1e6013aa70 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1e54016040 tx=0x7f1e54004a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:04.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.097+0000 7f1e5dffb640 1 -- 172.21.15.37:0/2505182672 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e54013980 con 0x7f1e60121040 2024-04-26T00:24:04.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.097+0000 7f1e5dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:04.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.097+0000 7f1e5dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:24:04.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.097+0000 7f1e5dffb640 10 monclient: dump: 2024-04-26T00:24:04.075 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:04.075 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:04.075 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:04.075 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:04.075 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:04.075 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:04.075 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:04.075 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:04.075 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:04.076 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:04.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.097+0000 7f1e5dffb640 1 -- 172.21.15.37:0/2505182672 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4719 (secure 0 0 0) 0x7f1e54013980 con 0x7f1e60121040 2024-04-26T00:24:04.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.097+0000 7f1e5dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:24:04.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.097+0000 7f1e5dffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:24:04.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.097+0000 7f1e5dffb640 1 -- 172.21.15.37:0/2505182672 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1e60121040 msgr2=0x7f1e6013aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:04.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.097+0000 7f1e5dffb640 1 --2- 172.21.15.37:0/2505182672 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1e60121040 0x7f1e6013aa70 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1e54016040 tx=0x7f1e54004a50 comp rx=0 tx=0).stop 2024-04-26T00:24:04.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.097+0000 7f1e5dffb640 1 -- 172.21.15.37:0/2505182672 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1e60121040 msgr2=0x7f1e6013aa70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:24:04.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.097+0000 7f1e5dffb640 1 --2- 172.21.15.37:0/2505182672 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1e60121040 0x7f1e6013aa70 secure :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1e54016040 tx=0x7f1e54004a50 comp rx=0 tx=0).stop 2024-04-26T00:24:04.076 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:24:04.076 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:24:04.077 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:24:04.077 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:24:04.077 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 34, 2024-04-26T00:24:04.077 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 104375, 2024-04-26T00:24:04.077 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:24:04.077 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:24:04.077 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:24:04.077 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:24:04.077 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:04.077 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 1, 2024-04-26T00:24:04.077 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:24:04.077 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:24:04.077 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:24:04.077 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:24:04.078 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:24:04.078 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:24:04.078 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:24:04.078 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:24:04.078 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:24:04.078 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:24:04.078 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:24:04.078 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:24:04.078 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:24:04.078 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:24:04.078 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:04.078 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:24:04.078 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:24:04.078 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:24:04.079 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:24:04.079 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:24:04.079 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:24:04.079 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:24:04.079 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:24:04.079 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:24:04.079 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:24:04.079 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:24:04.079 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:24:04.079 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:24:04.079 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:04.079 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:04.079 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:24:04.079 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:24:04.079 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:24:04.080 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:24:04.080 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:24:04.080 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:24:04.080 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:24:04.080 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:24:04.080 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:24:04.080 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:24:04.080 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:24:04.080 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:24:04.080 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:24:04.080 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:24:04.080 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:24:04.080 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:24:04.080 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:24:04.081 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:24:04.081 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:24:04.081 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:24:04.081 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:24:04.081 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:24:04.081 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:24:04.081 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:24:04.081 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:24:04.081 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:24:04.081 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:24:04.081 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:24:04.081 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:04.081 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:24:04.081 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:04.082 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:24:04.082 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:04.082 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:24:04.082 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:24:04.082 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:24:04.082 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:24:04.082 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:04.082 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:24:04.082 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:24:04.082 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:04.082 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:04.082 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:04.082 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:24:04.083 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:24:04.083 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:04.083 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:04.083 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:04.083 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:04.083 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:24:04.083 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:24:04.083 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:24:04.083 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:24:04.083 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:24:04.083 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:04.083 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:04.083 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:24:04.083 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:24:04.083 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:24:04.084 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:24:04.084 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:04.084 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:24:04.084 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:24:04.084 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:04.084 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:04.084 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:04.084 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:24:04.084 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:24:04.084 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:04.084 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:04.084 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:04.084 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:04.085 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:24:04.085 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:24:04.085 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:24:04.085 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:24:04.085 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:24:04.085 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:04.085 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:04.085 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:24:04.085 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:24:04.085 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:24:04.085 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:24:04.085 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:04.085 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:24:04.086 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:24:04.086 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:04.086 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:04.086 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:04.086 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:24:04.086 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:24:04.086 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:04.086 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:04.086 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:04.086 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:04.086 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:24:04.086 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:24:04.086 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:24:04.086 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:24:04.086 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:24:04.087 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:04.087 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:04.087 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:04.087 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:24:04.087 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:24:04.087 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:04.087 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:24:04.087 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:24:04.087 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:24:04.087 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:04.087 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:04.087 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:24:04.087 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:04.087 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:24:04.088 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:24:04.088 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:24:04.088 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:04.088 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:04.088 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:24:04.088 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:04.088 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:24:04.088 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:24:04.088 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:24:04.088 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:04.088 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:04.088 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:04.088 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:24:04.088 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:24:04.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.097+0000 7f1e675ee640 1 -- 172.21.15.37:0/2505182672 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1e4c036070 msgr2=0x7f1e4c038530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:04.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.097+0000 7f1e675ee640 1 --2- 172.21.15.37:0/2505182672 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1e4c036070 0x7f1e4c038530 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1e600a57e0 tx=0x7f1e580a6000 comp rx=0 tx=0).stop 2024-04-26T00:24:04.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.097+0000 7f1e675ee640 10 monclient: shutdown 2024-04-26T00:24:04.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.097+0000 7f1e675ee640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:24:04.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.097+0000 7f1e675ee640 1 -- 172.21.15.37:0/2505182672 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1e600a3d90 msgr2=0x7f1e601ced80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:04.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.097+0000 7f1e675ee640 1 --2- 172.21.15.37:0/2505182672 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1e600a3d90 0x7f1e601ced80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1e5000b430 tx=0x7f1e5000b8f0 comp rx=0 tx=0).stop 2024-04-26T00:24:04.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.097+0000 7f1e65b8b640 1 -- 172.21.15.37:0/2505182672 reap_dead start 2024-04-26T00:24:04.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.097+0000 7f1e65b8b640 1 -- 172.21.15.37:0/2505182672 reap_dead start 2024-04-26T00:24:04.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.097+0000 7f1e675ee640 1 -- 172.21.15.37:0/2505182672 shutdown_connections 2024-04-26T00:24:04.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.097+0000 7f1e675ee640 1 -- 172.21.15.37:0/2505182672 >> 172.21.15.37:0/2505182672 conn(0x7f1e600aa0a0 msgr2=0x7f1e600a8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:24:04.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.097+0000 7f1e675ee640 1 -- 172.21.15.37:0/2505182672 shutdown_connections 2024-04-26T00:24:04.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.097+0000 7f1e675ee640 1 -- 172.21.15.37:0/2505182672 wait complete. 2024-04-26T00:24:04.091 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:24:04.091 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:24:04.091 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:24:04.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.217+0000 7f910d8aa640 10 monclient: get_monmap_and_config 2024-04-26T00:24:04.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.217+0000 7f910d8aa640 10 monclient: build_initial_monmap 2024-04-26T00:24:04.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.217+0000 7f910d8aa640 10 monclient: monmap: 2024-04-26T00:24:04.198 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:24:04.198 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:04.198 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:24:04.223749+0000 2024-04-26T00:24:04.198 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:24:04.223749+0000 2024-04-26T00:24:04.198 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:24:04.198 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:04.198 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:24:04.198 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:24:04.198 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:24:04.198 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:24:04.199 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:24:04.199 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:04.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.221+0000 7f910d8aa640 1 Processor -- start 2024-04-26T00:24:04.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.221+0000 7f910d8aa640 1 -- start start 2024-04-26T00:24:04.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.221+0000 7f910d8aa640 10 monclient: init 2024-04-26T00:24:04.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.221+0000 7f910d8aa640 10 monclient: _reopen_session rank -1 2024-04-26T00:24:04.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.221+0000 7f910d8aa640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-04-26T00:24:04.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.221+0000 7f910d8aa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f91080a3d90 addr v1:172.21.15.37:6789/0 2024-04-26T00:24:04.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.221+0000 7f910d8aa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f91080a4170 addr v1:172.21.15.45:6789/0 2024-04-26T00:24:04.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.221+0000 7f910d8aa640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f910813d3f0 0x7f910813f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:04.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.221+0000 7f910d8aa640 10 monclient(hunting): picked mon.noname-a con 0x7f910813d3f0 addr v2:172.21.15.37:3300/0 2024-04-26T00:24:04.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.221+0000 7f910d8aa640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91080a5150 con 0x7f91080a3d90 2024-04-26T00:24:04.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.221+0000 7f910d8aa640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f910809b280 con 0x7f91080a4170 2024-04-26T00:24:04.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.221+0000 7f910d8aa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:04.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.221+0000 7f910d8aa640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f910809c650 con 0x7f910813d3f0 2024-04-26T00:24:04.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.221+0000 7f910d8aa640 10 monclient(hunting): _renew_subs 2024-04-26T00:24:04.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.221+0000 7f910d8aa640 10 monclient(hunting): authenticate will time out at 1225.529377s 2024-04-26T00:24:04.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.221+0000 7f9106ffd640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f91080a3d90 0x7f910809bf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:52070/0 (socket says 172.21.15.37:52070) 2024-04-26T00:24:04.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.221+0000 7f91077fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f910813d3f0 0x7f910813f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:04.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.221+0000 7f9106ffd640 1 -- 172.21.15.37:0/2863263916 learned_addr learned my addr 172.21.15.37:0/2863263916 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:24:04.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.221+0000 7f91077fe640 10 monclient(hunting): get_auth_request con 0x7f910813d3f0 auth_method 0 2024-04-26T00:24:04.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.221+0000 7f91077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:04.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.221+0000 7f91077fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:04.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.221+0000 7f91077fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:04.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.221+0000 7f91077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:04.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.221+0000 7f91077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:04.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.221+0000 7f91077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:04.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.221+0000 7f91077fe640 10 monclient(hunting): handle_auth_done global_id 54114 payload 274 2024-04-26T00:24:04.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.221+0000 7f91077fe640 1 -- 172.21.15.37:0/2863263916 >> v1:172.21.15.45:6789/0 conn(0x7f91080a4170 legacy=0x7f91080a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:24:04.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.221+0000 7f91077fe640 1 -- 172.21.15.37:0/2863263916 >> v1:172.21.15.37:6789/0 conn(0x7f91080a3d90 legacy=0x7f910809bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:24:04.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.221+0000 7f91077fe640 10 monclient: _finish_hunting 0 2024-04-26T00:24:04.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.221+0000 7f91077fe640 1 monclient: found mon.noname-a 2024-04-26T00:24:04.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.221+0000 7f91077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:24:04.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.221+0000 7f91077fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:24:04.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.221+0000 7f91077fe640 1 -- 172.21.15.37:0/2863263916 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f910809b560 con 0x7f910813d3f0 2024-04-26T00:24:04.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.225+0000 7f91077fe640 1 -- 172.21.15.37:0/2863263916 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f910813d3f0 msgr2=0x7f910813f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:04.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.225+0000 7f91077fe640 1 --2- 172.21.15.37:0/2863263916 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f910813d3f0 0x7f910813f7e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9100009d90 tx=0x7f910001c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e847525ba3dc70 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:04.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.225+0000 7f9105ffb640 1 -- 172.21.15.37:0/2863263916 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91000b2070 con 0x7f910813d3f0 2024-04-26T00:24:04.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.225+0000 7f9105ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:04.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.225+0000 7f9105ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:24:04.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.225+0000 7f9105ffb640 10 monclient: dump: 2024-04-26T00:24:04.205 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:04.205 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:04.205 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:04.205 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:04.205 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:04.205 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:04.205 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:04.205 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:04.206 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:04.206 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:04.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.225+0000 7f9105ffb640 10 monclient: _finish_auth 0 2024-04-26T00:24:04.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.225+0000 7f9105ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:24:04.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.225+0000 7f9105ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:24:04.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.225+0000 7f9105ffb640 1 -- 172.21.15.37:0/2863263916 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91000a5860 con 0x7f910813d3f0 2024-04-26T00:24:04.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.225+0000 7f9105ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:24:04.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.225+0000 7f9105ffb640 1 -- 172.21.15.37:0/2863263916 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91000a5c30 con 0x7f910813d3f0 2024-04-26T00:24:04.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.225+0000 7f910d8aa640 5 monclient: authenticate success, global_id 54114 2024-04-26T00:24:04.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.225+0000 7f910d8aa640 10 monclient: get_monmap_and_config success 2024-04-26T00:24:04.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.225+0000 7f910d8aa640 4 set_mon_vals no callback set 2024-04-26T00:24:04.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.225+0000 7f9105ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:04.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.225+0000 7f9105ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:24:04.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.225+0000 7f9105ffb640 10 monclient: dump: 2024-04-26T00:24:04.206 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:04.207 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:04.207 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:04.207 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:04.207 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:04.207 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:04.207 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:04.207 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:04.207 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:04.207 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:04.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.225+0000 7f910d8aa640 10 monclient: shutdown 2024-04-26T00:24:04.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.225+0000 7f910d8aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:24:04.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.225+0000 7f910d8aa640 1 -- 172.21.15.37:0/2863263916 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f910813d3f0 msgr2=0x7f910813f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:04.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.225+0000 7f910d8aa640 1 --2- 172.21.15.37:0/2863263916 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f910813d3f0 0x7f910813f7e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9100009d90 tx=0x7f910001c7b0 comp rx=0 tx=0).stop 2024-04-26T00:24:04.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.225+0000 7f910d8aa640 1 -- 172.21.15.37:0/2863263916 shutdown_connections 2024-04-26T00:24:04.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.225+0000 7f910d8aa640 1 --2- 172.21.15.37:0/2863263916 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f910813d3f0 0x7f910813f7e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:04.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.225+0000 7f910d8aa640 1 -- 172.21.15.37:0/2863263916 >> 172.21.15.37:0/2863263916 conn(0x7f91080aa0a0 msgr2=0x7f91080a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:24:04.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.225+0000 7f910d8aa640 1 -- 172.21.15.37:0/2863263916 shutdown_connections 2024-04-26T00:24:04.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.225+0000 7f910d8aa640 1 -- 172.21.15.37:0/2863263916 wait complete. 2024-04-26T00:24:04.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.225+0000 7f910d8aa640 10 monclient: build_initial_monmap 2024-04-26T00:24:04.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.225+0000 7f910d8aa640 10 monclient: monmap: 2024-04-26T00:24:04.208 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:24:04.208 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:24:04.208 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:24:04.208 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:24:04.208 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:24:04.208 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:24:04.208 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:24:04.208 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:24:04.208 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:24:04.208 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:04.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.225+0000 7f910d8aa640 1 Processor -- start 2024-04-26T00:24:04.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.225+0000 7f910d8aa640 1 -- start start 2024-04-26T00:24:04.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.225+0000 7f910d8aa640 10 monclient: init 2024-04-26T00:24:04.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.225+0000 7f910d8aa640 10 monclient: _reopen_session rank -1 2024-04-26T00:24:04.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.225+0000 7f910d8aa640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:24:04.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.225+0000 7f910d8aa640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91080a3d90 0x7f91081367b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:04.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.225+0000 7f910d8aa640 10 monclient(hunting): picked mon.noname-c con 0x7f91080a3d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:24:04.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.225+0000 7f910d8aa640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91080a4170 0x7f9108136cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:04.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.225+0000 7f910d8aa640 10 monclient(hunting): picked mon.noname-a con 0x7f91080a4170 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:24:04.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.225+0000 7f910d8aa640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f910813d3f0 0x7f9108139d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:04.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f910d8aa640 10 monclient(hunting): picked mon.noname-b con 0x7f910813d3f0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:24:04.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f910d8aa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:04.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f910d8aa640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9108142ef0 con 0x7f91080a4170 2024-04-26T00:24:04.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f910d8aa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:04.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f910d8aa640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9108142d70 con 0x7f910813d3f0 2024-04-26T00:24:04.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f910d8aa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:04.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f910d8aa640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9108143070 con 0x7f91080a3d90 2024-04-26T00:24:04.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f910d8aa640 10 monclient(hunting): _renew_subs 2024-04-26T00:24:04.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f910d8aa640 10 monclient(hunting): authenticate will time out at 1225.536028s 2024-04-26T00:24:04.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f9106ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91080a3d90 0x7f91081367b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:04.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f91067fc640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91080a4170 0x7f9108136cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:04.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f91067fc640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91080a4170 0x7f9108136cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52686/0 (socket says 172.21.15.37:52686) 2024-04-26T00:24:04.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f9106ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91080a3d90 0x7f91081367b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:45984/0 (socket says 172.21.15.37:45984) 2024-04-26T00:24:04.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f91067fc640 1 -- 172.21.15.37:0/3626901031 learned_addr learned my addr 172.21.15.37:0/3626901031 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:24:04.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f91067fc640 10 monclient(hunting): get_auth_request con 0x7f91080a4170 auth_method 0 2024-04-26T00:24:04.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f91067fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:04.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f91067fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:04.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f91067fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:04.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f9106ffd640 10 monclient(hunting): get_auth_request con 0x7f91080a3d90 auth_method 0 2024-04-26T00:24:04.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f9106ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:04.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f9106ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:04.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f9106ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:04.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f91067fc640 1 -- 172.21.15.37:0/3626901031 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91080a4170 msgr2=0x7f9108136cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:24:04.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f91067fc640 1 -- 172.21.15.37:0/3626901031 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91080a4170 msgr2=0x7f9108136cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:24:04.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f91067fc640 1 --2- 172.21.15.37:0/3626901031 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91080a4170 0x7f9108136cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:24:04.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f91067fc640 1 --2- 172.21.15.37:0/3626901031 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91080a4170 0x7f9108136cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:24:04.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f9106ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:04.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f9106ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:04.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f9106ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:04.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f91077fe640 1 --2- 172.21.15.37:0/3626901031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f910813d3f0 0x7f9108139d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:04.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f9106ffd640 10 monclient(hunting): handle_auth_done global_id 34307 payload 824 2024-04-26T00:24:04.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f9106ffd640 1 -- 172.21.15.37:0/3626901031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f910813d3f0 msgr2=0x7f9108139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:04.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f9106ffd640 1 --2- 172.21.15.37:0/3626901031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f910813d3f0 0x7f9108139d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:04.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f9106ffd640 1 -- 172.21.15.37:0/3626901031 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91080a4170 msgr2=0x7f9108136cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:24:04.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f9106ffd640 1 --2- 172.21.15.37:0/3626901031 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91080a4170 0x7f9108136cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:04.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f9106ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:24:04.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f9106ffd640 1 monclient: found mon.noname-c 2024-04-26T00:24:04.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f9106ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:24:04.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f9106ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:24:04.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f9106ffd640 1 -- 172.21.15.37:0/3626901031 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f910809ada0 con 0x7f91080a3d90 2024-04-26T00:24:04.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f91077fe640 10 monclient: get_auth_request con 0x7f910813d3f0 auth_method 0 2024-04-26T00:24:04.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f91077fe640 1 --2- 172.21.15.37:0/3626901031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f910813d3f0 0x7f9108139d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:24:04.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f9106ffd640 1 -- 172.21.15.37:0/3626901031 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91080a3d90 msgr2=0x7f91081367b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:04.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f9106ffd640 1 --2- 172.21.15.37:0/3626901031 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91080a3d90 0x7f91081367b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f90fc00c410 tx=0x7f90fc00c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:04.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f90effff640 1 -- 172.21.15.37:0/3626901031 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90fc017070 con 0x7f91080a3d90 2024-04-26T00:24:04.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f90effff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:04.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f90effff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:24:04.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f90effff640 10 monclient: dump: 2024-04-26T00:24:04.217 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:04.217 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:04.217 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:04.218 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:04.218 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:04.218 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:04.218 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:04.218 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:04.218 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:04.218 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:04.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f90effff640 10 monclient: _finish_auth 0 2024-04-26T00:24:04.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f90effff640 10 monclient: _check_auth_tickets 2024-04-26T00:24:04.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f90effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:24:04.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f90effff640 1 -- 172.21.15.37:0/3626901031 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90fc0070d0 con 0x7f91080a3d90 2024-04-26T00:24:04.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f90effff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:24:04.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f90effff640 1 -- 172.21.15.37:0/3626901031 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90fc0074c0 con 0x7f91080a3d90 2024-04-26T00:24:04.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f90effff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:04.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f90effff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:24:04.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f90effff640 10 monclient: dump: 2024-04-26T00:24:04.219 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:04.219 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:04.219 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:04.219 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:04.219 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:04.219 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:04.219 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:04.219 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:04.219 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:04.219 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:04.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f910d8aa640 5 monclient: authenticate success, global_id 34307 2024-04-26T00:24:04.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f910d8aa640 10 monclient: _renew_subs 2024-04-26T00:24:04.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f910d8aa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:24:04.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f91057fa640 4 set_mon_vals no callback set 2024-04-26T00:24:04.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f910d8aa640 1 -- 172.21.15.37:0/3626901031 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f910813d840 con 0x7f91080a3d90 2024-04-26T00:24:04.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f910d8aa640 10 monclient: _renew_subs 2024-04-26T00:24:04.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f910d8aa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:24:04.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f910d8aa640 1 -- 172.21.15.37:0/3626901031 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f910813c6a0 con 0x7f91080a3d90 2024-04-26T00:24:04.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f910d8aa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:24:04.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f910d8aa640 1 --2- 172.21.15.37:0/3626901031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f910813a260 0x7f910813a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:04.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f910d8aa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:04.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f910d8aa640 1 -- 172.21.15.37:0/3626901031 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f910804f510 con 0x7f910813a260 2024-04-26T00:24:04.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.229+0000 7f90effff640 1 -- 172.21.15.37:0/3626901031 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f90fc01a070 con 0x7f91080a3d90 2024-04-26T00:24:04.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.233+0000 7f91067fc640 1 --2- 172.21.15.37:0/3626901031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f910813a260 0x7f910813a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:04.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.233+0000 7f90effff640 1 --2- 172.21.15.37:0/3626901031 >> v2:172.21.15.45:6812/2379918623 conn(0x7f90f4035f00 0x7f90f40383c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:04.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.233+0000 7f91067fc640 10 monclient: get_auth_request con 0x7f910813a260 auth_method 0 2024-04-26T00:24:04.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.233+0000 7f91067fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:04.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.233+0000 7f91067fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:04.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.233+0000 7f91067fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:04.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.233+0000 7f90effff640 1 -- 172.21.15.37:0/3626901031 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(175..175 src has 1..175) ==== 3959+0+0 (secure 0 0 0) 0x7f90fc047f10 con 0x7f91080a3d90 2024-04-26T00:24:04.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.233+0000 7f91067fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:04.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.233+0000 7f91067fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:04.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.233+0000 7f91067fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:04.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.233+0000 7f91077fe640 1 --2- 172.21.15.37:0/3626901031 >> v2:172.21.15.45:6812/2379918623 conn(0x7f90f4035f00 0x7f90f40383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:04.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.233+0000 7f91077fe640 10 monclient: get_auth_request con 0x7f90f4035f00 auth_method 0 2024-04-26T00:24:04.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.233+0000 7f91067fc640 10 monclient(hunting): handle_auth_done global_id 24361 payload 824 2024-04-26T00:24:04.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.233+0000 7f91067fc640 1 -- 172.21.15.37:0/3626901031 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f91080008e0 con 0x7f910813a260 2024-04-26T00:24:04.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.233+0000 7f91067fc640 1 -- 172.21.15.37:0/3626901031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f910813a260 msgr2=0x7f910813a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:04.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.233+0000 7f91067fc640 1 --2- 172.21.15.37:0/3626901031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f910813a260 0x7f910813a640 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f910813b520 tx=0x7f90f800e980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:04.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.233+0000 7f90effff640 1 -- 172.21.15.37:0/3626901031 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f8007450 con 0x7f910813a260 2024-04-26T00:24:04.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.233+0000 7f90effff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:04.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.233+0000 7f90effff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:24:04.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.233+0000 7f91077fe640 1 --2- 172.21.15.37:0/3626901031 >> v2:172.21.15.45:6812/2379918623 conn(0x7f90f4035f00 0x7f90f40383c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f91000ac000 tx=0x7f91000a7070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:04.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.233+0000 7f90effff640 10 monclient: dump: 2024-04-26T00:24:04.221 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:04.221 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:04.222 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:04.222 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:04.222 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:04.222 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:04.222 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:04.222 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:04.222 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:04.222 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:04.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.237+0000 7f90effff640 1 -- 172.21.15.37:0/3626901031 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f90f8007450 con 0x7f910813a260 2024-04-26T00:24:04.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.237+0000 7f90effff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:24:04.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.237+0000 7f90effff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:24:04.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.237+0000 7f90effff640 1 -- 172.21.15.37:0/3626901031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f910813a260 msgr2=0x7f910813a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:04.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.237+0000 7f90effff640 1 --2- 172.21.15.37:0/3626901031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f910813a260 0x7f910813a640 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f910813b520 tx=0x7f90f800e980 comp rx=0 tx=0).stop 2024-04-26T00:24:04.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.237+0000 7f90effff640 1 -- 172.21.15.37:0/3626901031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f910813a260 msgr2=0x7f910813a640 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:24:04.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.237+0000 7f90effff640 1 --2- 172.21.15.37:0/3626901031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f910813a260 0x7f910813a640 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f910813b520 tx=0x7f90f800e980 comp rx=0 tx=0).stop 2024-04-26T00:24:04.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.241+0000 7f910d8aa640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:24:04.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.241+0000 7f910d8aa640 1 --2- 172.21.15.37:0/3626901031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f910813ab80 0x7f91080a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:04.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.241+0000 7f910d8aa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:04.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.241+0000 7f910d8aa640 1 -- 172.21.15.37:0/3626901031 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f910811fb00 con 0x7f910813ab80 2024-04-26T00:24:04.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.241+0000 7f91067fc640 1 --2- 172.21.15.37:0/3626901031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f910813ab80 0x7f91080a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:04.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.241+0000 7f91067fc640 10 monclient: get_auth_request con 0x7f910813ab80 auth_method 0 2024-04-26T00:24:04.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.241+0000 7f91067fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:04.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.241+0000 7f91067fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:04.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.241+0000 7f91067fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:04.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.241+0000 7f91067fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:04.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.241+0000 7f91067fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:04.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.241+0000 7f91067fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:04.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.241+0000 7f91067fc640 10 monclient(hunting): handle_auth_done global_id 24364 payload 824 2024-04-26T00:24:04.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.241+0000 7f91067fc640 1 -- 172.21.15.37:0/3626901031 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f910813af90 con 0x7f910813ab80 2024-04-26T00:24:04.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.245+0000 7f91067fc640 1 -- 172.21.15.37:0/3626901031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f910813ab80 msgr2=0x7f91080a17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:04.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.245+0000 7f91067fc640 1 --2- 172.21.15.37:0/3626901031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f910813ab80 0x7f91080a17a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f90f800e9c0 tx=0x7f90f8003770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:04.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.245+0000 7f90effff640 1 -- 172.21.15.37:0/3626901031 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f801eb70 con 0x7f910813ab80 2024-04-26T00:24:04.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.245+0000 7f90effff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:04.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.245+0000 7f90effff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:24:04.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.245+0000 7f90effff640 10 monclient: dump: 2024-04-26T00:24:04.225 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:04.225 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:04.225 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:04.226 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:04.226 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:04.226 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:04.226 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:04.226 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:04.226 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:04.226 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:04.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.245+0000 7f90effff640 1 -- 172.21.15.37:0/3626901031 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f90f801eb70 con 0x7f910813ab80 2024-04-26T00:24:04.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.245+0000 7f90effff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:24:04.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.245+0000 7f90effff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:24:04.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.245+0000 7f90effff640 1 -- 172.21.15.37:0/3626901031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f910813ab80 msgr2=0x7f91080a17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:04.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.245+0000 7f90effff640 1 --2- 172.21.15.37:0/3626901031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f910813ab80 0x7f91080a17a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f90f800e9c0 tx=0x7f90f8003770 comp rx=0 tx=0).stop 2024-04-26T00:24:04.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.245+0000 7f90effff640 1 -- 172.21.15.37:0/3626901031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f910813ab80 msgr2=0x7f91080a17a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:24:04.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.245+0000 7f90effff640 1 --2- 172.21.15.37:0/3626901031 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f910813ab80 0x7f91080a17a0 secure :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f90f800e9c0 tx=0x7f90f8003770 comp rx=0 tx=0).stop 2024-04-26T00:24:04.227 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:24:04.227 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:24:04.227 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:24:04.227 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:24:04.227 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 34, 2024-04-26T00:24:04.227 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 152412, 2024-04-26T00:24:04.227 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:24:04.227 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:24:04.227 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:24:04.227 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:24:04.228 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:04.228 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 1, 2024-04-26T00:24:04.228 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:24:04.228 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:24:04.228 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:24:04.228 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:24:04.228 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:24:04.228 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:24:04.228 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:24:04.228 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:24:04.228 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:24:04.228 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:24:04.228 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:24:04.228 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:24:04.228 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:24:04.229 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:24:04.229 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:04.229 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:24:04.229 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:24:04.229 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:24:04.229 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:24:04.229 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:24:04.229 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:24:04.229 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:24:04.229 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:24:04.229 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:24:04.229 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:24:04.229 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:24:04.229 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:24:04.229 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:24:04.229 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:04.229 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:04.230 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:24:04.230 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:24:04.230 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:24:04.230 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:24:04.230 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:24:04.230 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:24:04.230 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:24:04.230 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:24:04.230 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:24:04.230 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:24:04.230 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:24:04.230 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:24:04.230 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:24:04.230 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:24:04.231 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:24:04.231 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:24:04.231 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:24:04.231 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:24:04.231 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:24:04.231 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:24:04.231 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:24:04.231 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:24:04.231 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:24:04.231 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:24:04.231 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:24:04.231 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:24:04.231 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:24:04.231 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:24:04.232 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:04.232 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:24:04.232 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:04.232 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:24:04.232 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:04.232 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:24:04.232 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:24:04.232 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:24:04.232 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:24:04.232 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:04.232 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:24:04.232 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:24:04.232 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:04.232 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:04.232 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:04.232 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:24:04.233 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:24:04.233 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:04.233 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:04.233 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:04.233 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:04.233 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:24:04.233 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:24:04.233 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:24:04.233 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:24:04.233 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:24:04.233 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:04.233 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:04.233 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:24:04.233 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:24:04.233 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:24:04.233 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:24:04.234 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:04.234 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:24:04.234 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:24:04.234 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:04.234 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:04.234 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:04.234 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:24:04.234 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:24:04.234 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:04.234 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:04.234 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:04.234 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:04.234 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:24:04.234 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:24:04.235 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:24:04.235 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:24:04.235 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:24:04.235 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:04.235 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:04.235 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:24:04.235 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:24:04.235 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:24:04.235 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:24:04.235 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:04.236 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:24:04.236 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:24:04.236 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:04.236 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:04.236 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:04.236 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:24:04.236 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:24:04.236 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:04.236 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:04.236 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:04.236 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:04.236 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:24:04.236 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:24:04.236 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:24:04.236 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:24:04.237 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:24:04.237 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:04.237 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:04.237 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:04.237 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:24:04.237 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:24:04.237 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:04.237 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:24:04.237 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:24:04.237 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:24:04.237 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:04.237 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:04.237 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:24:04.237 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:04.237 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:24:04.237 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:24:04.238 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:24:04.238 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:04.238 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:04.238 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:24:04.238 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:04.238 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:24:04.238 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:24:04.238 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:24:04.238 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:04.238 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:04.238 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:24:04.238 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:04.238 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:24:04.238 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:24:04.239 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:24:04.239 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:04.239 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:04.239 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:04.239 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:24:04.239 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:24:04.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.245+0000 7f910d8aa640 1 -- 172.21.15.37:0/3626901031 >> v2:172.21.15.45:6812/2379918623 conn(0x7f90f4035f00 msgr2=0x7f90f40383c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:04.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.245+0000 7f910d8aa640 1 --2- 172.21.15.37:0/3626901031 >> v2:172.21.15.45:6812/2379918623 conn(0x7f90f4035f00 0x7f90f40383c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f91000ac000 tx=0x7f91000a7070 comp rx=0 tx=0).stop 2024-04-26T00:24:04.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.245+0000 7f910d8aa640 10 monclient: shutdown 2024-04-26T00:24:04.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.245+0000 7f910d8aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:24:04.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.245+0000 7f910d8aa640 1 -- 172.21.15.37:0/3626901031 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91080a3d90 msgr2=0x7f91081367b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:04.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.245+0000 7f910d8aa640 1 --2- 172.21.15.37:0/3626901031 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91080a3d90 0x7f91081367b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f90fc00c410 tx=0x7f90fc00c8d0 comp rx=0 tx=0).stop 2024-04-26T00:24:04.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.245+0000 7f91077fe640 1 -- 172.21.15.37:0/3626901031 reap_dead start 2024-04-26T00:24:04.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.245+0000 7f91077fe640 1 -- 172.21.15.37:0/3626901031 reap_dead start 2024-04-26T00:24:04.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.245+0000 7f910d8aa640 1 -- 172.21.15.37:0/3626901031 shutdown_connections 2024-04-26T00:24:04.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.245+0000 7f910d8aa640 1 -- 172.21.15.37:0/3626901031 >> 172.21.15.37:0/3626901031 conn(0x7f91080aa0a0 msgr2=0x7f91080a8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:24:04.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.245+0000 7f910d8aa640 1 -- 172.21.15.37:0/3626901031 shutdown_connections 2024-04-26T00:24:04.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.245+0000 7f910d8aa640 1 -- 172.21.15.37:0/3626901031 wait complete. 2024-04-26T00:24:04.241 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:24:04.241 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-26T00:24:04.241 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-26T00:24:04.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.373+0000 7f1d9eeef640 10 monclient: get_monmap_and_config 2024-04-26T00:24:04.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.373+0000 7f1d9eeef640 10 monclient: build_initial_monmap 2024-04-26T00:24:04.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.373+0000 7f1d9eeef640 10 monclient: monmap: 2024-04-26T00:24:04.352 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:24:04.352 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:04.352 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:24:04.377903+0000 2024-04-26T00:24:04.352 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:24:04.377903+0000 2024-04-26T00:24:04.352 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:24:04.352 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:04.352 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:24:04.352 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:24:04.352 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:24:04.353 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:24:04.353 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:24:04.353 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:04.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.373+0000 7f1d9eeef640 1 Processor -- start 2024-04-26T00:24:04.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.373+0000 7f1d9eeef640 1 -- start start 2024-04-26T00:24:04.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.373+0000 7f1d9eeef640 10 monclient: init 2024-04-26T00:24:04.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.373+0000 7f1d9eeef640 10 monclient: _reopen_session rank -1 2024-04-26T00:24:04.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.373+0000 7f1d9eeef640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-04-26T00:24:04.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.373+0000 7f1d9eeef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1d980a0260 addr v1:172.21.15.45:6789/0 2024-04-26T00:24:04.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.373+0000 7f1d9eeef640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1d98137e40 0x7f1d9813a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:04.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.373+0000 7f1d9eeef640 10 monclient(hunting): picked mon.noname-c con 0x7f1d98137e40 addr v2:172.21.15.45:3300/0 2024-04-26T00:24:04.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.373+0000 7f1d9eeef640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f1d9813a770 0x7f1d9813cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:04.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.373+0000 7f1d9eeef640 10 monclient(hunting): picked mon.noname-a con 0x7f1d9813a770 addr v2:172.21.15.37:3300/0 2024-04-26T00:24:04.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.373+0000 7f1d9eeef640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d980a1130 con 0x7f1d980a0260 2024-04-26T00:24:04.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.373+0000 7f1d9eeef640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:04.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.373+0000 7f1d9eeef640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f1d980a0d50 con 0x7f1d9813a770 2024-04-26T00:24:04.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.373+0000 7f1d9eeef640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:04.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.373+0000 7f1d9eeef640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f1d9809ec10 con 0x7f1d98137e40 2024-04-26T00:24:04.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.373+0000 7f1d9eeef640 10 monclient(hunting): _renew_subs 2024-04-26T00:24:04.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.373+0000 7f1d9eeef640 10 monclient(hunting): authenticate will time out at 1225.683557s 2024-04-26T00:24:04.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.373+0000 7f1d9d48c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f1d9813a770 0x7f1d9813cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:04.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.373+0000 7f1d9d48c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f1d9813a770 0x7f1d9813cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52694/0 (socket says 172.21.15.37:52694) 2024-04-26T00:24:04.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.373+0000 7f1d9d48c640 1 -- 172.21.15.37:0/567010170 learned_addr learned my addr 172.21.15.37:0/567010170 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:24:04.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.373+0000 7f1d9d48c640 10 monclient(hunting): get_auth_request con 0x7f1d9813a770 auth_method 0 2024-04-26T00:24:04.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.373+0000 7f1d9d48c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:04.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.373+0000 7f1d9d48c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:04.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.373+0000 7f1d9d48c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:04.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.373+0000 7f1d97fff640 1 --2- 172.21.15.37:0/567010170 >> v2:172.21.15.45:3300/0 conn(0x7f1d98137e40 0x7f1d9813a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:04.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.377+0000 7f1d97fff640 10 monclient(hunting): get_auth_request con 0x7f1d98137e40 auth_method 0 2024-04-26T00:24:04.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.377+0000 7f1d97fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:04.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.377+0000 7f1d97fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:04.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.377+0000 7f1d97fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:04.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.377+0000 7f1d9d48c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:04.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.377+0000 7f1d9d48c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:04.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.377+0000 7f1d9d48c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:04.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.377+0000 7f1d97fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:04.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.377+0000 7f1d97fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:04.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.377+0000 7f1d97fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:04.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.377+0000 7f1d9d48c640 10 monclient(hunting): handle_auth_done global_id 54117 payload 274 2024-04-26T00:24:04.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.377+0000 7f1d9d48c640 1 -- 172.21.15.37:0/567010170 >> v2:172.21.15.45:3300/0 conn(0x7f1d98137e40 msgr2=0x7f1d9813a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:24:04.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.377+0000 7f1d9d48c640 1 --2- 172.21.15.37:0/567010170 >> v2:172.21.15.45:3300/0 conn(0x7f1d98137e40 0x7f1d9813a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:04.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.377+0000 7f1d9d48c640 1 -- 172.21.15.37:0/567010170 >> v1:172.21.15.45:6789/0 conn(0x7f1d980a0260 legacy=0x7f1d980a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:24:04.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.377+0000 7f1d9d48c640 10 monclient: _finish_hunting 0 2024-04-26T00:24:04.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.377+0000 7f1d9d48c640 1 monclient: found mon.noname-a 2024-04-26T00:24:04.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.377+0000 7f1d9d48c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:24:04.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.377+0000 7f1d9d48c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:24:04.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.377+0000 7f1d9d48c640 1 -- 172.21.15.37:0/567010170 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d9806cff0 con 0x7f1d9813a770 2024-04-26T00:24:04.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9d48c640 1 -- 172.21.15.37:0/567010170 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d9813a770 msgr2=0x7f1d9813cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:04.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9d48c640 1 --2- 172.21.15.37:0/567010170 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d9813a770 0x7f1d9813cb60 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1d9809f690 tx=0x7f1d9001c690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=856d0c4eded40e98 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:04.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d977fe640 1 -- 172.21.15.37:0/567010170 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d900a9460 con 0x7f1d9813a770 2024-04-26T00:24:04.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:04.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d977fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:24:04.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d977fe640 10 monclient: dump: 2024-04-26T00:24:04.359 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:04.359 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:04.360 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:04.360 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:04.360 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:04.360 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:04.360 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:04.360 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:04.360 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:04.360 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:04.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d977fe640 10 monclient: _finish_auth 0 2024-04-26T00:24:04.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d977fe640 10 monclient: _check_auth_tickets 2024-04-26T00:24:04.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:24:04.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9eeef640 5 monclient: authenticate success, global_id 54117 2024-04-26T00:24:04.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9eeef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:24:04.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d977fe640 1 -- 172.21.15.37:0/567010170 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d900ae070 con 0x7f1d9813a770 2024-04-26T00:24:04.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d977fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:24:04.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d977fe640 1 -- 172.21.15.37:0/567010170 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d900a9820 con 0x7f1d9813a770 2024-04-26T00:24:04.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:04.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:24:04.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d977fe640 10 monclient: dump: 2024-04-26T00:24:04.361 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:04.361 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:04.361 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:04.361 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:04.361 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:04.361 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:04.361 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:04.361 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:04.361 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:04.361 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:04.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9eeef640 10 monclient: get_monmap_and_config success 2024-04-26T00:24:04.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9eeef640 4 set_mon_vals no callback set 2024-04-26T00:24:04.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9eeef640 10 monclient: shutdown 2024-04-26T00:24:04.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9eeef640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:24:04.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9eeef640 1 -- 172.21.15.37:0/567010170 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d9813a770 msgr2=0x7f1d9813cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:04.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9eeef640 1 --2- 172.21.15.37:0/567010170 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d9813a770 0x7f1d9813cb60 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1d9809f690 tx=0x7f1d9001c690 comp rx=0 tx=0).stop 2024-04-26T00:24:04.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9eeef640 1 -- 172.21.15.37:0/567010170 shutdown_connections 2024-04-26T00:24:04.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9eeef640 1 --2- 172.21.15.37:0/567010170 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d9813a770 0x7f1d9813cb60 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:04.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9eeef640 1 --2- 172.21.15.37:0/567010170 >> v2:172.21.15.45:3300/0 conn(0x7f1d98137e40 0x7f1d9813a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:04.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9eeef640 1 -- 172.21.15.37:0/567010170 >> 172.21.15.37:0/567010170 conn(0x7f1d980a1ad0 msgr2=0x7f1d980a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:24:04.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9eeef640 1 -- 172.21.15.37:0/567010170 shutdown_connections 2024-04-26T00:24:04.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9eeef640 1 -- 172.21.15.37:0/567010170 wait complete. 2024-04-26T00:24:04.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9eeef640 10 monclient: build_initial_monmap 2024-04-26T00:24:04.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9eeef640 10 monclient: monmap: 2024-04-26T00:24:04.366 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:24:04.366 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:24:04.366 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:24:04.366 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:24:04.366 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:24:04.366 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:24:04.366 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:24:04.367 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:24:04.367 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:24:04.367 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:04.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9eeef640 1 Processor -- start 2024-04-26T00:24:04.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9eeef640 1 -- start start 2024-04-26T00:24:04.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9eeef640 10 monclient: init 2024-04-26T00:24:04.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9eeef640 10 monclient: _reopen_session rank -1 2024-04-26T00:24:04.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9eeef640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:24:04.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9eeef640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d980a0260 0x7f1d981cc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:04.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9eeef640 10 monclient(hunting): picked mon.noname-a con 0x7f1d980a0260 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:24:04.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9eeef640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d98137e40 0x7f1d981368e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:04.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9eeef640 10 monclient(hunting): picked mon.noname-b con 0x7f1d98137e40 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:24:04.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9eeef640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1d9813a770 0x7f1d9811d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:04.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9eeef640 10 monclient(hunting): picked mon.noname-c con 0x7f1d9813a770 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:24:04.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9eeef640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:04.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9eeef640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f1d9809f0f0 con 0x7f1d980a0260 2024-04-26T00:24:04.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9eeef640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:04.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9eeef640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1d980a00a0 con 0x7f1d98137e40 2024-04-26T00:24:04.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9eeef640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:04.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9eeef640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f1d9809f4c0 con 0x7f1d9813a770 2024-04-26T00:24:04.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9eeef640 10 monclient(hunting): _renew_subs 2024-04-26T00:24:04.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9eeef640 10 monclient(hunting): authenticate will time out at 1225.691088s 2024-04-26T00:24:04.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9cc8b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d980a0260 0x7f1d981cc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:04.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9cc8b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d980a0260 0x7f1d981cc7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52696/0 (socket says 172.21.15.37:52696) 2024-04-26T00:24:04.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9cc8b640 1 -- 172.21.15.37:0/1362916264 learned_addr learned my addr 172.21.15.37:0/1362916264 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:24:04.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9cc8b640 10 monclient(hunting): get_auth_request con 0x7f1d980a0260 auth_method 0 2024-04-26T00:24:04.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9cc8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:04.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9cc8b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:04.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9cc8b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:04.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9d48c640 1 --2- 172.21.15.37:0/1362916264 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1d9813a770 0x7f1d9811d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:04.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9d48c640 10 monclient(hunting): get_auth_request con 0x7f1d9813a770 auth_method 0 2024-04-26T00:24:04.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9d48c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:04.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9d48c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:04.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9d48c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:04.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9cc8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:04.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9cc8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:04.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9cc8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:04.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9d48c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:04.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9d48c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:04.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9d48c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:04.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d97fff640 1 --2- 172.21.15.37:0/1362916264 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d98137e40 0x7f1d981368e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:04.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d97fff640 10 monclient(hunting): get_auth_request con 0x7f1d98137e40 auth_method 0 2024-04-26T00:24:04.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d97fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:04.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d97fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:04.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d97fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:04.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9cc8b640 10 monclient(hunting): handle_auth_done global_id 54120 payload 824 2024-04-26T00:24:04.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9cc8b640 1 -- 172.21.15.37:0/1362916264 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1d9813a770 msgr2=0x7f1d9811d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:04.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9cc8b640 1 --2- 172.21.15.37:0/1362916264 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1d9813a770 0x7f1d9811d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:04.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9cc8b640 1 -- 172.21.15.37:0/1362916264 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d98137e40 msgr2=0x7f1d981368e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:04.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9cc8b640 1 --2- 172.21.15.37:0/1362916264 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d98137e40 0x7f1d981368e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:04.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9cc8b640 10 monclient: _finish_hunting 0 2024-04-26T00:24:04.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9cc8b640 1 monclient: found mon.noname-a 2024-04-26T00:24:04.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9cc8b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:24:04.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9cc8b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:24:04.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9cc8b640 1 -- 172.21.15.37:0/1362916264 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d9811ef80 con 0x7f1d980a0260 2024-04-26T00:24:04.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.381+0000 7f1d9d48c640 1 --2- 172.21.15.37:0/1362916264 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1d9813a770 0x7f1d9811d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:24:04.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.385+0000 7f1d9cc8b640 1 -- 172.21.15.37:0/1362916264 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d980a0260 msgr2=0x7f1d981cc7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:04.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.385+0000 7f1d9cc8b640 1 --2- 172.21.15.37:0/1362916264 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d980a0260 0x7f1d981cc7d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c00c450 tx=0x7f1d8c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:04.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.385+0000 7f1d957fa640 1 -- 172.21.15.37:0/1362916264 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d8c012a40 con 0x7f1d980a0260 2024-04-26T00:24:04.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.385+0000 7f1d957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:04.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.385+0000 7f1d957fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:24:04.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.385+0000 7f1d957fa640 10 monclient: dump: 2024-04-26T00:24:04.371 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:04.371 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:04.371 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:04.371 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:04.371 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:04.371 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:04.371 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:04.371 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:04.371 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:04.371 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:04.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.385+0000 7f1d957fa640 10 monclient: _finish_auth 0 2024-04-26T00:24:04.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.385+0000 7f1d957fa640 10 monclient: _check_auth_tickets 2024-04-26T00:24:04.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.385+0000 7f1d957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:24:04.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.385+0000 7f1d957fa640 1 -- 172.21.15.37:0/1362916264 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d8c017070 con 0x7f1d980a0260 2024-04-26T00:24:04.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.385+0000 7f1d957fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:24:04.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.385+0000 7f1d957fa640 1 -- 172.21.15.37:0/1362916264 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d8c012e40 con 0x7f1d980a0260 2024-04-26T00:24:04.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.385+0000 7f1d9eeef640 5 monclient: authenticate success, global_id 54120 2024-04-26T00:24:04.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.385+0000 7f1d9eeef640 10 monclient: _renew_subs 2024-04-26T00:24:04.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.385+0000 7f1d9eeef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:24:04.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.385+0000 7f1d9eeef640 1 -- 172.21.15.37:0/1362916264 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d981cd6c0 con 0x7f1d980a0260 2024-04-26T00:24:04.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.385+0000 7f1d957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:04.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.385+0000 7f1d957fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:24:04.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.385+0000 7f1d967fc640 4 set_mon_vals no callback set 2024-04-26T00:24:04.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.385+0000 7f1d957fa640 10 monclient: dump: 2024-04-26T00:24:04.372 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:04.373 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:04.373 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:04.373 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:04.373 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:04.373 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:04.373 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:04.373 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:04.373 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:04.373 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:04.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.385+0000 7f1d9eeef640 10 monclient: _renew_subs 2024-04-26T00:24:04.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.385+0000 7f1d9eeef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:24:04.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.385+0000 7f1d9eeef640 1 -- 172.21.15.37:0/1362916264 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d98121270 con 0x7f1d980a0260 2024-04-26T00:24:04.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.385+0000 7f1d957fa640 1 -- 172.21.15.37:0/1362916264 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f1d8c01c460 con 0x7f1d980a0260 2024-04-26T00:24:04.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.385+0000 7f1d9eeef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:24:04.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.385+0000 7f1d9eeef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:24:04.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.385+0000 7f1d9eeef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:24:04.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.385+0000 7f1d9eeef640 1 -- 172.21.15.37:0/1362916264 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d98066f40 con 0x7f1d980a0260 2024-04-26T00:24:04.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.385+0000 7f1d957fa640 1 --2- 172.21.15.37:0/1362916264 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1d84035e80 0x7f1d84038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:04.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.385+0000 7f1d957fa640 1 -- 172.21.15.37:0/1362916264 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(175..175 src has 1..175) ==== 3959+0+0 (secure 0 0 0) 0x7f1d8c0488b0 con 0x7f1d980a0260 2024-04-26T00:24:04.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.385+0000 7f1d97fff640 1 --2- 172.21.15.37:0/1362916264 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1d84035e80 0x7f1d84038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:04.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.385+0000 7f1d97fff640 10 monclient: get_auth_request con 0x7f1d84035e80 auth_method 0 2024-04-26T00:24:04.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.389+0000 7f1d97fff640 1 --2- 172.21.15.37:0/1362916264 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1d84035e80 0x7f1d84038340 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1d9811f300 tx=0x7f1d88008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:04.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.393+0000 7f1d957fa640 1 -- 172.21.15.37:0/1362916264 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f1d8c0475e0 con 0x7f1d980a0260 2024-04-26T00:24:04.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.393+0000 7f1d957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:24:04.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.393+0000 7f1d957fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:24:04.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.609+0000 7f1d9eeef640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-26T00:24:04.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.609+0000 7f1d9eeef640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-26T00:24:04.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.609+0000 7f1d9eeef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:24:04.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.609+0000 7f1d9eeef640 1 -- 172.21.15.37:0/1362916264 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f1d98120a30 con 0x7f1d980a0260 2024-04-26T00:24:04.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.613+0000 7f1d957fa640 1 -- 172.21.15.37:0/1362916264 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f1d8c01b410 con 0x7f1d980a0260 2024-04-26T00:24:04.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.613+0000 7f1d957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-26T00:24:04.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.613+0000 7f1d957fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:24:04.592 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.613+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:24:04.593 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.613+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=12,config=2,health=11,kv=10,logm=65} crc {auth=1576550334,config=692619927,health=1096256309,kv=1034996044,logm=553837764}) 2024-04-26T00:24:04.593 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.613+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=11,kv=10,logm=67} crc {auth=2214000349,config=692619927,health=2287149628,kv=1034996044,logm=584198523}) 2024-04-26T00:24:04.593 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.613+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:24:04.593 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.613+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=12,config=2,health=11,kv=10,logm=65} crc {auth=1576550334,config=692619927,health=1096256309,kv=1034996044,logm=553837764}) 2024-04-26T00:24:04.593 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.613+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=12,config=2,health=12,kv=9,logm=65} crc {auth=1576550334,config=692619927,health=1535303502,kv=2885089190,logm=4062459544}) 2024-04-26T00:24:04.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.613+0000 7f1d9eeef640 1 -- 172.21.15.37:0/1362916264 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1d84035e80 msgr2=0x7f1d84038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:04.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.613+0000 7f1d9eeef640 1 --2- 172.21.15.37:0/1362916264 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1d84035e80 0x7f1d84038340 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1d9811f300 tx=0x7f1d88008040 comp rx=0 tx=0).stop 2024-04-26T00:24:04.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.613+0000 7f1d9eeef640 10 monclient: shutdown 2024-04-26T00:24:04.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.613+0000 7f1d9eeef640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:24:04.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.613+0000 7f1d9eeef640 1 -- 172.21.15.37:0/1362916264 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d980a0260 msgr2=0x7f1d981cc7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:04.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.613+0000 7f1d9eeef640 1 --2- 172.21.15.37:0/1362916264 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d980a0260 0x7f1d981cc7d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c00c450 tx=0x7f1d8c00c910 comp rx=0 tx=0).stop 2024-04-26T00:24:04.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.613+0000 7f1d9eeef640 1 -- 172.21.15.37:0/1362916264 shutdown_connections 2024-04-26T00:24:04.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.613+0000 7f1d9eeef640 1 --2- 172.21.15.37:0/1362916264 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1d84035e80 0x7f1d84038340 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:04.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.617+0000 7f1d9eeef640 1 --2- 172.21.15.37:0/1362916264 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1d9813a770 0x7f1d9811d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:04.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.617+0000 7f1d9eeef640 1 --2- 172.21.15.37:0/1362916264 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d98137e40 0x7f1d981368e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:04.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.617+0000 7f1d9eeef640 1 --2- 172.21.15.37:0/1362916264 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d980a0260 0x7f1d981cc7d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:04.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.617+0000 7f1d9eeef640 1 -- 172.21.15.37:0/1362916264 >> 172.21.15.37:0/1362916264 conn(0x7f1d980a1ad0 msgr2=0x7f1d98138610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:24:04.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.617+0000 7f1d9eeef640 1 -- 172.21.15.37:0/1362916264 shutdown_connections 2024-04-26T00:24:04.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:04.617+0000 7f1d9eeef640 1 -- 172.21.15.37:0/1362916264 wait complete. 2024-04-26T00:24:04.595 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.617+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:24:04.596 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.617+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4065812169}) 2024-04-26T00:24:04.596 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.617+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2390544508}) 2024-04-26T00:24:04.596 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.617+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:24:04.596 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.617+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4065812169}) 2024-04-26T00:24:04.596 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.617+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1698268928}) 2024-04-26T00:24:04.597 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.617+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:24:04.597 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.617+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1553110480}) 2024-04-26T00:24:04.597 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.617+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3679310123}) 2024-04-26T00:24:04.597 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.617+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:24:04.597 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.617+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1553110480}) 2024-04-26T00:24:04.597 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.617+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4164139802}) 2024-04-26T00:24:04.599 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.621+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:24:04.599 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.621+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1667464592}) 2024-04-26T00:24:04.599 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.621+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=548988513}) 2024-04-26T00:24:04.599 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.621+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:24:04.599 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.621+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1667464592}) 2024-04-26T00:24:04.599 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.621+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2625991126}) 2024-04-26T00:24:04.601 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.621+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:24:04.601 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.621+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1186403974}) 2024-04-26T00:24:04.601 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.621+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3852342689}) 2024-04-26T00:24:04.601 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.621+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:24:04.601 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.621+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1186403974}) 2024-04-26T00:24:04.601 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.621+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3521301328}) 2024-04-26T00:24:04.603 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.625+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:24:04.603 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.625+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1714789108}) 2024-04-26T00:24:04.603 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.625+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=753665534}) 2024-04-26T00:24:04.603 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.625+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:24:04.603 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.625+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1714789108}) 2024-04-26T00:24:04.604 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.625+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=624358108}) 2024-04-26T00:24:04.606 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.625+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:24:04.606 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.625+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1348936378}) 2024-04-26T00:24:04.606 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.625+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2879089236}) 2024-04-26T00:24:04.606 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.625+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:24:04.606 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.629+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1348936378}) 2024-04-26T00:24:04.606 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.629+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3811853242}) 2024-04-26T00:24:04.608 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.629+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:24:04.608 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.629+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1773419882}) 2024-04-26T00:24:04.608 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.629+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3519282036}) 2024-04-26T00:24:04.608 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.629+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:24:04.608 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.629+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1773419882}) 2024-04-26T00:24:04.608 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.629+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1023411007}) 2024-04-26T00:24:04.609 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-26T00:24:04.610 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.633+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:24:04.610 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.633+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4009525452}) 2024-04-26T00:24:04.610 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.633+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=232046838}) 2024-04-26T00:24:04.610 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.633+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:24:04.610 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.633+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4009525452}) 2024-04-26T00:24:04.610 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.633+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=595359133}) 2024-04-26T00:24:04.612 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.633+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:24:04.612 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.633+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3765032491}) 2024-04-26T00:24:04.612 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.633+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3693209471}) 2024-04-26T00:24:04.613 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.633+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:24:04.613 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.633+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3765032491}) 2024-04-26T00:24:04.613 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.633+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=154152045}) 2024-04-26T00:24:04.615 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.637+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:24:04.615 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.637+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3978061684}) 2024-04-26T00:24:04.615 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.637+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1803732913}) 2024-04-26T00:24:04.615 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.637+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:24:04.615 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.637+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3978061684}) 2024-04-26T00:24:04.616 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.637+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1397911602}) 2024-04-26T00:24:04.618 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.637+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:24:04.618 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.637+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=29,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=15,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=24} crc {logm=2246278267,mdsmap=2840952586,mgr=3332899993,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=3200316698,mon_config_key=4210576201,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=2883343541,osd_snap=75138824}) 2024-04-26T00:24:04.618 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.637+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=29,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=10,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=25} crc {logm=2074037339,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=3506607757,mon_config_key=1720404874,monmap=3140514009,osd_metadata=3306121363,osd_pg_creating=2883343541,osd_snap=1297586828}) 2024-04-26T00:24:04.618 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.637+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:24:04.618 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.641+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=29,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=15,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=24} crc {logm=2246278267,mdsmap=2840952586,mgr=3332899993,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=3200316698,mon_config_key=4210576201,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=2883343541,osd_snap=75138824}) 2024-04-26T00:24:04.618 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.641+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=28,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=24} crc {logm=2256559391,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=804968274,mon_config_key=4061063117,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=2883343541,osd_snap=2486379247}) 2024-04-26T00:24:04.620 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.641+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:24:04.620 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.641+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=2,osdmap=98} crc {osd_snap=1018022996,osdmap=1066826144}) 2024-04-26T00:24:04.620 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.641+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=3,osdmap=97} crc {osd_snap=309009170,osdmap=1719223107}) 2024-04-26T00:24:04.620 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.641+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:24:04.620 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.641+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=2,osdmap=98} crc {osd_snap=1018022996,osdmap=1066826144}) 2024-04-26T00:24:04.620 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.641+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=2,osdmap=98} crc {osd_snap=643204143,osdmap=3668602999}) 2024-04-26T00:24:04.623 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.645+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:24:04.623 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.645+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3561031262}) 2024-04-26T00:24:04.623 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.645+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=101747223}) 2024-04-26T00:24:04.623 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.645+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:24:04.623 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.645+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3561031262}) 2024-04-26T00:24:04.623 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.645+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3665996735}) 2024-04-26T00:24:04.631 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.653+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:24:04.631 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.653+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=69} crc {osdmap=344368267}) 2024-04-26T00:24:04.631 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.653+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=64} crc {osdmap=3353506907}) 2024-04-26T00:24:04.631 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.653+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:24:04.631 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.653+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=69} crc {osdmap=344368267}) 2024-04-26T00:24:04.631 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:24:04.653+0000 7fbc51e51640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=71} crc {osdmap=1336575348}) 2024-04-26T00:24:04.740 INFO:tasks.rados.rados.0.smithi045.stdout:741: done (2 left) 2024-04-26T00:24:04.740 INFO:tasks.rados.rados.0.smithi045.stdout:742: rollback oid 5 current snap is 86 2024-04-26T00:24:04.740 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 5 to 74 2024-04-26T00:24:04.740 INFO:tasks.rados.rados.0.smithi045.stdout:739: expect (ObjNum 242 snap 61 seq_num 242) 2024-04-26T00:24:04.871 INFO:tasks.rados.rados.0.smithi045.stdout:740: finishing copy_from racing read to smithi04526579-4 2024-04-26T00:24:04.871 INFO:tasks.rados.rados.0.smithi045.stdout:740: finishing copy_from to smithi04526579-4 2024-04-26T00:24:04.871 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 363 (ObjNum 224 snap 59 seq_num 224) dirty exists 2024-04-26T00:24:04.871 INFO:tasks.rados.rados.0.smithi045.stdout:742: finishing rollback tid 1 to smithi04526579-5 2024-04-26T00:24:04.871 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 392 (ObjNum 9473 snap 0 seq_num 907322272) dirty dne 2024-04-26T00:24:04.871 INFO:tasks.rados.rados.0.smithi045.stdout:739: done (2 left) 2024-04-26T00:24:04.871 INFO:tasks.rados.rados.0.smithi045.stdout:740: done (1 left) 2024-04-26T00:24:04.871 INFO:tasks.rados.rados.0.smithi045.stdout:742: done (0 left) 2024-04-26T00:24:04.871 INFO:tasks.rados.rados.0.smithi045.stdout:743: rollback oid 33 current snap is 86 2024-04-26T00:24:04.871 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 33 to 82 2024-04-26T00:24:04.871 INFO:tasks.rados.rados.0.smithi045.stdout:744: snap_create 2024-04-26T00:24:04.879 INFO:tasks.rados.rados.0.smithi045.stdout:743: finishing rollback tid 0 to smithi04526579-33 2024-04-26T00:24:04.879 INFO:tasks.rados.rados.0.smithi045.stdout:743: finishing rollback tid 1 to smithi04526579-33 2024-04-26T00:24:04.879 INFO:tasks.rados.rados.0.smithi045.stdout:743: finishing rollback tid 2 to smithi04526579-33 2024-04-26T00:24:04.879 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 500 (ObjNum 246 snap 63 seq_num 246) dirty exists 2024-04-26T00:24:05.610 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:24:05.610 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:24:05.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.765+0000 7fde8269b640 10 monclient: get_monmap_and_config 2024-04-26T00:24:05.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.765+0000 7fde8269b640 10 monclient: build_initial_monmap 2024-04-26T00:24:05.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.765+0000 7fde8269b640 10 monclient: monmap: 2024-04-26T00:24:05.743 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:24:05.743 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:05.743 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:24:05.769089+0000 2024-04-26T00:24:05.743 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:24:05.769089+0000 2024-04-26T00:24:05.743 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:24:05.743 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:05.744 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:24:05.744 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:24:05.744 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:24:05.744 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:24:05.744 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:24:05.744 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:05.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.765+0000 7fde8269b640 1 Processor -- start 2024-04-26T00:24:05.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.765+0000 7fde8269b640 1 -- start start 2024-04-26T00:24:05.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.765+0000 7fde8269b640 10 monclient: init 2024-04-26T00:24:05.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.765+0000 7fde8269b640 10 monclient: _reopen_session rank -1 2024-04-26T00:24:05.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.765+0000 7fde8269b640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-04-26T00:24:05.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.765+0000 7fde8269b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde7c0a1af0 0x7fde7c0a1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:05.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.765+0000 7fde8269b640 10 monclient(hunting): picked mon.noname-b con 0x7fde7c0a1af0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:24:05.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.765+0000 7fde8269b640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fde7c0a2410 0x7fde7c137260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:05.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.765+0000 7fde8269b640 10 monclient(hunting): picked mon.noname-a con 0x7fde7c0a2410 addr v2:172.21.15.37:3300/0 2024-04-26T00:24:05.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.765+0000 7fde8269b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fde7c1377a0 addr v1:172.21.15.45:6789/0 2024-04-26T00:24:05.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.765+0000 7fde8269b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde7c09afa0 con 0x7fde7c1377a0 2024-04-26T00:24:05.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.765+0000 7fde8269b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:05.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.765+0000 7fde8269b640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fde7c09a700 con 0x7fde7c0a2410 2024-04-26T00:24:05.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.765+0000 7fde8269b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:05.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.765+0000 7fde8269b640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fde7c06bf00 con 0x7fde7c0a1af0 2024-04-26T00:24:05.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.765+0000 7fde8269b640 10 monclient(hunting): _renew_subs 2024-04-26T00:24:05.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.765+0000 7fde8269b640 10 monclient(hunting): authenticate will time out at 1227.075770s 2024-04-26T00:24:05.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.765+0000 7fde7bfff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde7c0a1af0 0x7fde7c0a1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:05.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.765+0000 7fde7b7fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fde7c0a2410 0x7fde7c137260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:05.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.765+0000 7fde7bfff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde7c0a1af0 0x7fde7c0a1ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:46010/0 (socket says 172.21.15.37:46010) 2024-04-26T00:24:05.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.765+0000 7fde7b7fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fde7c0a2410 0x7fde7c137260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52698/0 (socket says 172.21.15.37:52698) 2024-04-26T00:24:05.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.769+0000 7fde7bfff640 1 -- 172.21.15.37:0/4236851232 learned_addr learned my addr 172.21.15.37:0/4236851232 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:24:05.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.769+0000 7fde7bfff640 10 monclient(hunting): get_auth_request con 0x7fde7c0a1af0 auth_method 0 2024-04-26T00:24:05.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.769+0000 7fde7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:05.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.769+0000 7fde7bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:05.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.769+0000 7fde7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:05.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.769+0000 7fde7b7fe640 10 monclient(hunting): get_auth_request con 0x7fde7c0a2410 auth_method 0 2024-04-26T00:24:05.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.769+0000 7fde7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:05.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.769+0000 7fde7b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:05.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.769+0000 7fde7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:05.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.769+0000 7fde7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:05.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.769+0000 7fde7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:05.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.769+0000 7fde7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:05.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.769+0000 7fde7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:05.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.769+0000 7fde7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:05.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.769+0000 7fde7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:05.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.769+0000 7fde7bfff640 10 monclient(hunting): handle_auth_done global_id 34313 payload 274 2024-04-26T00:24:05.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.769+0000 7fde7bfff640 1 -- 172.21.15.37:0/4236851232 >> v2:172.21.15.37:3300/0 conn(0x7fde7c0a2410 msgr2=0x7fde7c137260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:24:05.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.769+0000 7fde7bfff640 1 --2- 172.21.15.37:0/4236851232 >> v2:172.21.15.37:3300/0 conn(0x7fde7c0a2410 0x7fde7c137260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:05.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.769+0000 7fde7bfff640 1 -- 172.21.15.37:0/4236851232 >> v1:172.21.15.45:6789/0 conn(0x7fde7c1377a0 legacy=0x7fde7c13e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:24:05.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.769+0000 7fde7bfff640 10 monclient: _finish_hunting 0 2024-04-26T00:24:05.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.769+0000 7fde7bfff640 1 monclient: found mon.noname-b 2024-04-26T00:24:05.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.769+0000 7fde7bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:24:05.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.769+0000 7fde7bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:24:05.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.769+0000 7fde7bfff640 1 -- 172.21.15.37:0/4236851232 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde7c06b410 con 0x7fde7c0a1af0 2024-04-26T00:24:05.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.769+0000 7fde7b7fe640 1 --2- 172.21.15.37:0/4236851232 >> v2:172.21.15.37:3300/0 conn(0x7fde7c0a2410 0x7fde7c137260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:24:05.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde7bfff640 1 -- 172.21.15.37:0/4236851232 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde7c0a1af0 msgr2=0x7fde7c0a1ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:05.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde7bfff640 1 --2- 172.21.15.37:0/4236851232 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde7c0a1af0 0x7fde7c0a1ed0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fde6c009870 tx=0x7fde6c01c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fc2a18166e8a839 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:05.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde7affd640 1 -- 172.21.15.37:0/4236851232 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde6c0b0070 con 0x7fde7c0a1af0 2024-04-26T00:24:05.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:05.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde7affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:24:05.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde7affd640 10 monclient: dump: 2024-04-26T00:24:05.752 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:05.752 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:05.752 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:05.752 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:05.752 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:05.752 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:05.752 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:05.752 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:05.752 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:05.752 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:05.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde7affd640 10 monclient: _finish_auth 0 2024-04-26T00:24:05.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde7affd640 10 monclient: _check_auth_tickets 2024-04-26T00:24:05.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:24:05.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde7affd640 1 -- 172.21.15.37:0/4236851232 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde6c0ab600 con 0x7fde7c0a1af0 2024-04-26T00:24:05.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde7affd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:24:05.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde7affd640 1 -- 172.21.15.37:0/4236851232 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde6c0ab9d0 con 0x7fde7c0a1af0 2024-04-26T00:24:05.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde8269b640 5 monclient: authenticate success, global_id 34313 2024-04-26T00:24:05.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde8269b640 10 monclient: get_monmap_and_config success 2024-04-26T00:24:05.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde8269b640 4 set_mon_vals no callback set 2024-04-26T00:24:05.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:05.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde7affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:24:05.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde7affd640 10 monclient: dump: 2024-04-26T00:24:05.753 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:05.753 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:05.753 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:05.753 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:05.753 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:05.753 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:05.754 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:05.754 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:05.754 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:05.754 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:05.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde8269b640 10 monclient: shutdown 2024-04-26T00:24:05.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde8269b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:24:05.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde8269b640 1 -- 172.21.15.37:0/4236851232 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde7c0a1af0 msgr2=0x7fde7c0a1ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:05.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde8269b640 1 --2- 172.21.15.37:0/4236851232 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde7c0a1af0 0x7fde7c0a1ed0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fde6c009870 tx=0x7fde6c01c440 comp rx=0 tx=0).stop 2024-04-26T00:24:05.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde8269b640 1 -- 172.21.15.37:0/4236851232 shutdown_connections 2024-04-26T00:24:05.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde8269b640 1 --2- 172.21.15.37:0/4236851232 >> v2:172.21.15.37:3300/0 conn(0x7fde7c0a2410 0x7fde7c137260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:05.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde8269b640 1 --2- 172.21.15.37:0/4236851232 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde7c0a1af0 0x7fde7c0a1ed0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:05.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde8269b640 1 -- 172.21.15.37:0/4236851232 >> 172.21.15.37:0/4236851232 conn(0x7fde7c0a2be0 msgr2=0x7fde7c0a2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:24:05.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde8269b640 1 -- 172.21.15.37:0/4236851232 shutdown_connections 2024-04-26T00:24:05.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde8269b640 1 -- 172.21.15.37:0/4236851232 wait complete. 2024-04-26T00:24:05.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde8269b640 10 monclient: build_initial_monmap 2024-04-26T00:24:05.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde8269b640 10 monclient: monmap: 2024-04-26T00:24:05.759 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:24:05.759 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:24:05.759 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:24:05.759 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:24:05.759 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:24:05.759 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:24:05.759 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:24:05.759 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:24:05.759 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:24:05.759 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:05.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde8269b640 1 Processor -- start 2024-04-26T00:24:05.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde8269b640 1 -- start start 2024-04-26T00:24:05.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde8269b640 10 monclient: init 2024-04-26T00:24:05.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde8269b640 10 monclient: _reopen_session rank -1 2024-04-26T00:24:05.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde8269b640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:24:05.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde8269b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde7c0a1af0 0x7fde7c13eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:05.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde8269b640 10 monclient(hunting): picked mon.noname-b con 0x7fde7c0a1af0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:24:05.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde8269b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde7c0a2410 0x7fde7c1366a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:05.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde8269b640 10 monclient(hunting): picked mon.noname-c con 0x7fde7c0a2410 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:24:05.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde8269b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde7c1377a0 0x7fde7c136be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:05.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde8269b640 10 monclient(hunting): picked mon.noname-a con 0x7fde7c1377a0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:24:05.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde8269b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:05.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde8269b640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fde7c0a3930 con 0x7fde7c1377a0 2024-04-26T00:24:05.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde8269b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:05.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde8269b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fde7c0a2910 con 0x7fde7c0a1af0 2024-04-26T00:24:05.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde8269b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:05.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde8269b640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fde7c0a3ab0 con 0x7fde7c0a2410 2024-04-26T00:24:05.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde8269b640 10 monclient(hunting): _renew_subs 2024-04-26T00:24:05.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde8269b640 10 monclient(hunting): authenticate will time out at 1227.083271s 2024-04-26T00:24:05.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde80c38640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde7c1377a0 0x7fde7c136be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:05.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde7b7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde7c0a2410 0x7fde7c1366a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:05.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde80c38640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde7c1377a0 0x7fde7c136be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52710/0 (socket says 172.21.15.37:52710) 2024-04-26T00:24:05.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde80c38640 1 -- 172.21.15.37:0/4109647279 learned_addr learned my addr 172.21.15.37:0/4109647279 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:24:05.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde80c38640 10 monclient(hunting): get_auth_request con 0x7fde7c1377a0 auth_method 0 2024-04-26T00:24:05.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde80c38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:05.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde80c38640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:05.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde80c38640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:05.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde7b7fe640 1 --2- 172.21.15.37:0/4109647279 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde7c0a2410 0x7fde7c1366a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:46016/0 (socket says 172.21.15.37:46016) 2024-04-26T00:24:05.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde7b7fe640 10 monclient(hunting): get_auth_request con 0x7fde7c0a2410 auth_method 0 2024-04-26T00:24:05.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:05.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde7b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:05.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:05.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde80c38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:05.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde80c38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:05.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde80c38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:05.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:05.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:05.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:05.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde80c38640 10 monclient(hunting): handle_auth_done global_id 54126 payload 824 2024-04-26T00:24:05.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde80c38640 1 -- 172.21.15.37:0/4109647279 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde7c0a2410 msgr2=0x7fde7c1366a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:05.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde80c38640 1 --2- 172.21.15.37:0/4109647279 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde7c0a2410 0x7fde7c1366a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:05.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde80c38640 1 -- 172.21.15.37:0/4109647279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde7c0a1af0 msgr2=0x7fde7c13eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:05.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde80c38640 1 --2- 172.21.15.37:0/4109647279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde7c0a1af0 0x7fde7c13eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:05.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde80c38640 10 monclient: _finish_hunting 0 2024-04-26T00:24:05.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde80c38640 1 monclient: found mon.noname-a 2024-04-26T00:24:05.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde80c38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:24:05.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde80c38640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:24:05.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde80c38640 1 -- 172.21.15.37:0/4109647279 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde7c137120 con 0x7fde7c1377a0 2024-04-26T00:24:05.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.773+0000 7fde7b7fe640 1 --2- 172.21.15.37:0/4109647279 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde7c0a2410 0x7fde7c1366a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:24:05.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.777+0000 7fde80c38640 1 -- 172.21.15.37:0/4109647279 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde7c1377a0 msgr2=0x7fde7c136be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:05.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.777+0000 7fde80c38640 1 --2- 172.21.15.37:0/4109647279 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde7c1377a0 0x7fde7c136be0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fde7400c780 tx=0x7fde7400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:05.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.777+0000 7fde78ff9640 1 -- 172.21.15.37:0/4109647279 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde7400ac60 con 0x7fde7c1377a0 2024-04-26T00:24:05.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.777+0000 7fde78ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:05.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.777+0000 7fde78ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:24:05.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.777+0000 7fde78ff9640 10 monclient: dump: 2024-04-26T00:24:05.763 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:05.763 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:05.763 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:05.763 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:05.763 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:05.763 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:05.764 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:05.764 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:05.764 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:05.764 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:05.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.777+0000 7fde78ff9640 10 monclient: _finish_auth 0 2024-04-26T00:24:05.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.777+0000 7fde78ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:24:05.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.777+0000 7fde78ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:24:05.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.777+0000 7fde78ff9640 1 -- 172.21.15.37:0/4109647279 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde7400ae00 con 0x7fde7c1377a0 2024-04-26T00:24:05.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.777+0000 7fde78ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:24:05.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.777+0000 7fde78ff9640 1 -- 172.21.15.37:0/4109647279 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde74004bb0 con 0x7fde7c1377a0 2024-04-26T00:24:05.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.777+0000 7fde78ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:05.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.777+0000 7fde78ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:24:05.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.777+0000 7fde78ff9640 10 monclient: dump: 2024-04-26T00:24:05.764 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:05.764 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:05.765 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:05.765 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:05.765 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:05.765 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:05.765 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:05.765 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:05.765 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:05.765 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:05.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.777+0000 7fde8269b640 5 monclient: authenticate success, global_id 54126 2024-04-26T00:24:05.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.777+0000 7fde8269b640 10 monclient: _renew_subs 2024-04-26T00:24:05.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.777+0000 7fde8269b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:24:05.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.777+0000 7fde8269b640 1 -- 172.21.15.37:0/4109647279 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde7c13e3a0 con 0x7fde7c1377a0 2024-04-26T00:24:05.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.777+0000 7fde7a7fc640 4 set_mon_vals no callback set 2024-04-26T00:24:05.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.777+0000 7fde8269b640 10 monclient: _renew_subs 2024-04-26T00:24:05.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.777+0000 7fde8269b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:24:05.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.777+0000 7fde8269b640 1 -- 172.21.15.37:0/4109647279 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde7c13dde0 con 0x7fde7c1377a0 2024-04-26T00:24:05.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.777+0000 7fde78ff9640 1 -- 172.21.15.37:0/4109647279 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fde74005030 con 0x7fde7c1377a0 2024-04-26T00:24:05.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.777+0000 7fde8269b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:24:05.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.777+0000 7fde8269b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:24:05.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.777+0000 7fde8269b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:24:05.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.777+0000 7fde8269b640 1 -- 172.21.15.37:0/4109647279 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde7c066f40 con 0x7fde7c1377a0 2024-04-26T00:24:05.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.777+0000 7fde78ff9640 1 --2- 172.21.15.37:0/4109647279 >> v2:172.21.15.45:6812/2379918623 conn(0x7fde68035ed0 0x7fde68038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:05.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.777+0000 7fde78ff9640 1 -- 172.21.15.37:0/4109647279 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(177..177 src has 1..177) ==== 3931+0+0 (secure 0 0 0) 0x7fde74048d20 con 0x7fde7c1377a0 2024-04-26T00:24:05.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.781+0000 7fde7bfff640 1 --2- 172.21.15.37:0/4109647279 >> v2:172.21.15.45:6812/2379918623 conn(0x7fde68035ed0 0x7fde68038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:05.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.781+0000 7fde7bfff640 10 monclient: get_auth_request con 0x7fde68035ed0 auth_method 0 2024-04-26T00:24:05.767 INFO:tasks.rados.rados.0.smithi045.stdout:743: done (1 left) 2024-04-26T00:24:05.767 INFO:tasks.rados.rados.0.smithi045.stdout:744: done (0 left) 2024-04-26T00:24:05.767 INFO:tasks.rados.rados.0.smithi045.stdout:745: delete oid 34 current snap is 87 2024-04-26T00:24:05.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.781+0000 7fde7bfff640 1 --2- 172.21.15.37:0/4109647279 >> v2:172.21.15.45:6812/2379918623 conn(0x7fde68035ed0 0x7fde68038390 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fde7c06b360 tx=0x7fde6c004740 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:05.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.785+0000 7fde78ff9640 1 -- 172.21.15.37:0/4109647279 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fde74020070 con 0x7fde7c1377a0 2024-04-26T00:24:05.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.785+0000 7fde78ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:24:05.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:05.785+0000 7fde78ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:24:05.769 INFO:tasks.rados.rados.0.smithi045.stdout:745: done (0 left) 2024-04-26T00:24:05.769 INFO:tasks.rados.rados.0.smithi045.stdout:746: read oid 27 snap -1 2024-04-26T00:24:05.769 INFO:tasks.rados.rados.0.smithi045.stdout:746: expect (ObjNum 284 snap 81 seq_num 284) 2024-04-26T00:24:05.769 INFO:tasks.rados.rados.0.smithi045.stdout:747: copy_from oid 9 from oid 6 current snap is 87 2024-04-26T00:24:05.769 INFO:tasks.rados.rados.0.smithi045.stdout:748: delete oid 2 current snap is 87 2024-04-26T00:24:05.778 INFO:tasks.rados.rados.0.smithi045.stdout:746: expect (ObjNum 284 snap 81 seq_num 284) 2024-04-26T00:24:05.960 INFO:tasks.rados.rados.0.smithi045.stdout:747: finishing copy_from racing read to smithi04526579-9 2024-04-26T00:24:05.961 INFO:tasks.rados.rados.0.smithi045.stdout:747: finishing copy_from to smithi04526579-9 2024-04-26T00:24:05.961 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 178 (ObjNum 283 snap 81 seq_num 283) dirty exists 2024-04-26T00:24:05.961 INFO:tasks.rados.rados.0.smithi045.stdout:746: done (2 left) 2024-04-26T00:24:05.961 INFO:tasks.rados.rados.0.smithi045.stdout:747: done (1 left) 2024-04-26T00:24:05.961 INFO:tasks.rados.rados.0.smithi045.stdout:748: done (0 left) 2024-04-26T00:24:05.961 INFO:tasks.rados.rados.0.smithi045.stdout:749: delete oid 22 current snap is 87 2024-04-26T00:24:05.965 INFO:tasks.rados.rados.0.smithi045.stdout:749: done (0 left) 2024-04-26T00:24:05.965 INFO:tasks.rados.rados.0.smithi045.stdout:750: rollback oid 27 current snap is 87 2024-04-26T00:24:05.965 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 27 to 71 2024-04-26T00:24:05.965 INFO:tasks.rados.rados.0.smithi045.stdout:751: delete oid 15 current snap is 87 2024-04-26T00:24:05.971 INFO:tasks.rados.rados.0.smithi045.stdout:751: done (1 left) 2024-04-26T00:24:05.972 INFO:tasks.rados.rados.0.smithi045.stdout:752: copy_from oid 3 from oid 35 current snap is 87 2024-04-26T00:24:05.972 INFO:tasks.rados.rados.0.smithi045.stdout:753: write (excl) oid 40 current snap is 87 2024-04-26T00:24:05.972 INFO:tasks.rados.rados.0.smithi045.stdout:753: seq_num 301 ranges {658907=787716,1930727=497772,3032909=25998} 2024-04-26T00:24:05.995 INFO:tasks.rados.rados.0.smithi045.stdout:753: writing smithi04526579-40 from 658907 to 1446623 tid 1 2024-04-26T00:24:05.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.021+0000 7fde8269b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:24:05.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.021+0000 7fde8269b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:24:05.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.021+0000 7fde8269b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:24:05.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.021+0000 7fde8269b640 1 -- 172.21.15.37:0/4109647279 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fde7c13cc10 con 0x7fde7c1377a0 2024-04-26T00:24:05.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.021+0000 7fde78ff9640 1 -- 172.21.15.37:0/4109647279 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fde74047a70 con 0x7fde7c1377a0 2024-04-26T00:24:05.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.021+0000 7fde78ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:24:05.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.021+0000 7fde78ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:24:05.999 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":34,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:24:06.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.021+0000 7fde8269b640 1 -- 172.21.15.37:0/4109647279 >> v2:172.21.15.45:6812/2379918623 conn(0x7fde68035ed0 msgr2=0x7fde68038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:06.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.021+0000 7fde8269b640 1 --2- 172.21.15.37:0/4109647279 >> v2:172.21.15.45:6812/2379918623 conn(0x7fde68035ed0 0x7fde68038390 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fde7c06b360 tx=0x7fde6c004740 comp rx=0 tx=0).stop 2024-04-26T00:24:06.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.021+0000 7fde8269b640 10 monclient: shutdown 2024-04-26T00:24:06.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.021+0000 7fde8269b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:24:06.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.021+0000 7fde8269b640 1 -- 172.21.15.37:0/4109647279 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde7c1377a0 msgr2=0x7fde7c136be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:06.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.021+0000 7fde8269b640 1 --2- 172.21.15.37:0/4109647279 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde7c1377a0 0x7fde7c136be0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fde7400c780 tx=0x7fde7400cc40 comp rx=0 tx=0).stop 2024-04-26T00:24:06.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.021+0000 7fde8269b640 1 -- 172.21.15.37:0/4109647279 shutdown_connections 2024-04-26T00:24:06.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.021+0000 7fde8269b640 1 --2- 172.21.15.37:0/4109647279 >> v2:172.21.15.45:6812/2379918623 conn(0x7fde68035ed0 0x7fde68038390 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:06.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.021+0000 7fde8269b640 1 --2- 172.21.15.37:0/4109647279 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde7c1377a0 0x7fde7c136be0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:06.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.021+0000 7fde8269b640 1 --2- 172.21.15.37:0/4109647279 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde7c0a2410 0x7fde7c1366a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:06.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.021+0000 7fde8269b640 1 --2- 172.21.15.37:0/4109647279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde7c0a1af0 0x7fde7c13eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:06.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.021+0000 7fde8269b640 1 -- 172.21.15.37:0/4109647279 >> 172.21.15.37:0/4109647279 conn(0x7fde7c0a2be0 msgr2=0x7fde7c09e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:24:06.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.025+0000 7fde8269b640 1 -- 172.21.15.37:0/4109647279 shutdown_connections 2024-04-26T00:24:06.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.025+0000 7fde8269b640 1 -- 172.21.15.37:0/4109647279 wait complete. 2024-04-26T00:24:06.011 INFO:tasks.rados.rados.0.smithi045.stdout:753: writing smithi04526579-40 from 1930727 to 2428499 tid 2 2024-04-26T00:24:06.019 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:24:06.019 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-26T00:24:06.019 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:24:06.020 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:24:06.022 INFO:tasks.rados.rados.0.smithi045.stdout:753: writing smithi04526579-40 from 3032909 to 3058907 tid 3 2024-04-26T00:24:06.022 INFO:tasks.rados.rados.0.smithi045.stdout:754: write (excl) oid 10 current snap is 87 2024-04-26T00:24:06.022 INFO:tasks.rados.rados.0.smithi045.stdout:754: seq_num 302 ranges {675714=462965,1589209=641430,2943979=131735} 2024-04-26T00:24:06.038 INFO:tasks.rados.rados.0.smithi045.stdout:754: writing smithi04526579-10 from 675714 to 1138679 tid 1 2024-04-26T00:24:06.053 INFO:tasks.rados.rados.0.smithi045.stdout:754: writing smithi04526579-10 from 1589209 to 2230639 tid 2 2024-04-26T00:24:06.064 INFO:tasks.rados.rados.0.smithi045.stdout:754: writing smithi04526579-10 from 2943979 to 3075714 tid 3 2024-04-26T00:24:06.064 INFO:tasks.rados.rados.0.smithi045.stdout:755: write (excl) oid 31 current snap is 87 2024-04-26T00:24:06.064 INFO:tasks.rados.rados.0.smithi045.stdout:755: seq_num 303 ranges {692521=538214,1931338=785088} 2024-04-26T00:24:06.076 INFO:tasks.rados.rados.0.smithi045.stdout:755: writing smithi04526579-31 from 692521 to 1230735 tid 1 2024-04-26T00:24:06.090 INFO:tasks.rados.rados.0.smithi045.stdout:755: writing smithi04526579-31 from 1931338 to 2716426 tid 2 2024-04-26T00:24:06.090 INFO:tasks.rados.rados.0.smithi045.stdout:756: rollback oid 1 current snap is 87 2024-04-26T00:24:06.090 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 1 to 69 2024-04-26T00:24:06.090 INFO:tasks.rados.rados.0.smithi045.stdout:750: finishing rollback tid 0 to smithi04526579-27 2024-04-26T00:24:06.090 INFO:tasks.rados.rados.0.smithi045.stdout:750: finishing rollback tid 1 to smithi04526579-27 2024-04-26T00:24:06.090 INFO:tasks.rados.rados.0.smithi045.stdout:750: finishing rollback tid 2 to smithi04526579-27 2024-04-26T00:24:06.090 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 369 (ObjNum 246 snap 63 seq_num 246) dirty exists 2024-04-26T00:24:06.090 INFO:tasks.rados.rados.0.smithi045.stdout:752: finishing copy_from racing read to smithi04526579-3 2024-04-26T00:24:06.090 INFO:tasks.rados.rados.0.smithi045.stdout:752: finishing copy_from to smithi04526579-3 2024-04-26T00:24:06.091 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 366 (ObjNum 138 snap 31 seq_num 138) dirty exists 2024-04-26T00:24:06.091 INFO:tasks.rados.rados.0.smithi045.stdout:753: finishing write tid 1 to smithi04526579-40 2024-04-26T00:24:06.091 INFO:tasks.rados.rados.0.smithi045.stdout:753: finishing write tid 2 to smithi04526579-40 2024-04-26T00:24:06.091 INFO:tasks.rados.rados.0.smithi045.stdout:753: finishing write tid 3 to smithi04526579-40 2024-04-26T00:24:06.091 INFO:tasks.rados.rados.0.smithi045.stdout:753: finishing write tid 4 to smithi04526579-40 2024-04-26T00:24:06.091 INFO:tasks.rados.rados.0.smithi045.stdout:753: finishing write tid 5 to smithi04526579-40 2024-04-26T00:24:06.091 INFO:tasks.rados.rados.0.smithi045.stdout:753: oid 40 updating version 0 to 298 2024-04-26T00:24:06.091 INFO:tasks.rados.rados.0.smithi045.stdout:753: oid 40 updating version 298 to 299 2024-04-26T00:24:06.091 INFO:tasks.rados.rados.0.smithi045.stdout:753: oid 40 updating version 299 to 301 2024-04-26T00:24:06.091 INFO:tasks.rados.rados.0.smithi045.stdout:753: oid 40 version 301 is already newer than 300 2024-04-26T00:24:06.091 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 301 (ObjNum 301 snap 87 seq_num 301) dirty exists 2024-04-26T00:24:06.091 INFO:tasks.rados.rados.0.smithi045.stdout:753: left oid 40 (ObjNum 301 snap 87 seq_num 301) 2024-04-26T00:24:06.091 INFO:tasks.rados.rados.0.smithi045.stdout:754: finishing write tid 1 to smithi04526579-10 2024-04-26T00:24:06.091 INFO:tasks.rados.rados.0.smithi045.stdout:754: finishing write tid 2 to smithi04526579-10 2024-04-26T00:24:06.092 INFO:tasks.rados.rados.0.smithi045.stdout:754: finishing write tid 3 to smithi04526579-10 2024-04-26T00:24:06.092 INFO:tasks.rados.rados.0.smithi045.stdout:754: finishing write tid 4 to smithi04526579-10 2024-04-26T00:24:06.092 INFO:tasks.rados.rados.0.smithi045.stdout:754: finishing write tid 5 to smithi04526579-10 2024-04-26T00:24:06.092 INFO:tasks.rados.rados.0.smithi045.stdout:754: oid 10 updating version 0 to 300 2024-04-26T00:24:06.092 INFO:tasks.rados.rados.0.smithi045.stdout:754: oid 10 version 300 is already newer than 298 2024-04-26T00:24:06.092 INFO:tasks.rados.rados.0.smithi045.stdout:754: oid 10 updating version 300 to 301 2024-04-26T00:24:06.092 INFO:tasks.rados.rados.0.smithi045.stdout:754: oid 10 version 301 is already newer than 299 2024-04-26T00:24:06.092 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 301 (ObjNum 302 snap 87 seq_num 302) dirty exists 2024-04-26T00:24:06.092 INFO:tasks.rados.rados.0.smithi045.stdout:754: left oid 10 (ObjNum 302 snap 87 seq_num 302) 2024-04-26T00:24:06.092 INFO:tasks.rados.rados.0.smithi045.stdout:755: finishing write tid 1 to smithi04526579-31 2024-04-26T00:24:06.092 INFO:tasks.rados.rados.0.smithi045.stdout:750: done (5 left) 2024-04-26T00:24:06.092 INFO:tasks.rados.rados.0.smithi045.stdout:752: done (4 left) 2024-04-26T00:24:06.092 INFO:tasks.rados.rados.0.smithi045.stdout:753: done (3 left) 2024-04-26T00:24:06.092 INFO:tasks.rados.rados.0.smithi045.stdout:754: done (2 left) 2024-04-26T00:24:06.093 INFO:tasks.rados.rados.0.smithi045.stdout:757: write oid 36 current snap is 87 2024-04-26T00:24:06.093 INFO:tasks.rados.rados.0.smithi045.stdout:757: seq_num 304 ranges {709328=613463,1989820=645099} 2024-04-26T00:24:06.102 INFO:tasks.rados.rados.0.smithi045.stdout:757: writing smithi04526579-36 from 709328 to 1322791 tid 1 2024-04-26T00:24:06.114 INFO:tasks.rados.rados.0.smithi045.stdout:757: writing smithi04526579-36 from 1989820 to 2634919 tid 2 2024-04-26T00:24:06.114 INFO:tasks.rados.rados.0.smithi045.stdout:758: write oid 35 current snap is 87 2024-04-26T00:24:06.115 INFO:tasks.rados.rados.0.smithi045.stdout:758: seq_num 305 ranges {726135=405065,1764655=788757,3026248=99887} 2024-04-26T00:24:06.124 INFO:tasks.rados.rados.0.smithi045.stdout:758: writing smithi04526579-35 from 726135 to 1131200 tid 1 2024-04-26T00:24:06.135 INFO:tasks.rados.rados.0.smithi045.stdout:758: writing smithi04526579-35 from 1764655 to 2553412 tid 2 2024-04-26T00:24:06.140 INFO:tasks.rados.rados.0.smithi045.stdout:758: writing smithi04526579-35 from 3026248 to 3126135 tid 3 2024-04-26T00:24:06.140 INFO:tasks.rados.rados.0.smithi045.stdout:759: read oid 49 snap -1 2024-04-26T00:24:06.140 INFO:tasks.rados.rados.0.smithi045.stdout:759: expect (ObjNum 256 snap 66 seq_num 256) 2024-04-26T00:24:06.140 INFO:tasks.rados.rados.0.smithi045.stdout:760: write (excl) oid 45 current snap is 87 2024-04-26T00:24:06.140 INFO:tasks.rados.rados.0.smithi045.stdout:755: finishing write tid 2 to smithi04526579-31 2024-04-26T00:24:06.140 INFO:tasks.rados.rados.0.smithi045.stdout:760: seq_num 306 ranges {742942=480314,1823137=648768} 2024-04-26T00:24:06.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.165+0000 7f98a29cc640 10 monclient: get_monmap_and_config 2024-04-26T00:24:06.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.165+0000 7f98a29cc640 10 monclient: build_initial_monmap 2024-04-26T00:24:06.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.165+0000 7f98a29cc640 10 monclient: monmap: 2024-04-26T00:24:06.143 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:24:06.143 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:06.143 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:24:06.169463+0000 2024-04-26T00:24:06.143 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:24:06.169463+0000 2024-04-26T00:24:06.144 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:24:06.144 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:06.144 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:24:06.144 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:24:06.144 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:24:06.144 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:24:06.144 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:24:06.144 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:06.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.165+0000 7f98a29cc640 1 Processor -- start 2024-04-26T00:24:06.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.165+0000 7f98a29cc640 1 -- start start 2024-04-26T00:24:06.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.165+0000 7f98a29cc640 10 monclient: init 2024-04-26T00:24:06.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.165+0000 7f98a29cc640 10 monclient: _reopen_session rank -1 2024-04-26T00:24:06.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.165+0000 7f98a29cc640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-04-26T00:24:06.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.165+0000 7f98a29cc640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f989c131cd0 0x7f989c1320b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:06.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.165+0000 7f98a29cc640 10 monclient(hunting): picked mon.noname-a con 0x7f989c131cd0 addr v2:172.21.15.37:3300/0 2024-04-26T00:24:06.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.165+0000 7f98a29cc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f989c073080 0x7f989c134c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:06.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.165+0000 7f98a29cc640 10 monclient(hunting): picked mon.noname-b con 0x7f989c073080 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:24:06.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.165+0000 7f98a29cc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f989c0a1b60 addr v1:172.21.15.37:6789/0 2024-04-26T00:24:06.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.165+0000 7f98a29cc640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f989c071660 con 0x7f989c0a1b60 2024-04-26T00:24:06.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.165+0000 7f98a29cc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:06.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.165+0000 7f98a29cc640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f989c071940 con 0x7f989c131cd0 2024-04-26T00:24:06.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.165+0000 7f98a29cc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:06.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.165+0000 7f98a29cc640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f989c071c20 con 0x7f989c073080 2024-04-26T00:24:06.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.165+0000 7f98a29cc640 10 monclient(hunting): _renew_subs 2024-04-26T00:24:06.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.165+0000 7f98a29cc640 10 monclient(hunting): authenticate will time out at 1227.475841s 2024-04-26T00:24:06.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.165+0000 7f989b7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f989c073080 0x7f989c134c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:06.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.169+0000 7f98a0f69640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f989c0a1b60 0x7f989c135150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:52084/0 (socket says 172.21.15.37:52084) 2024-04-26T00:24:06.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.169+0000 7f98a0f69640 1 -- 172.21.15.37:0/3809026309 learned_addr learned my addr 172.21.15.37:0/3809026309 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:24:06.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.169+0000 7f989b7fe640 1 --2- 172.21.15.37:0/3809026309 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f989c073080 0x7f989c134c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:46034/0 (socket says 172.21.15.37:46034) 2024-04-26T00:24:06.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.169+0000 7f989b7fe640 10 monclient(hunting): get_auth_request con 0x7f989c073080 auth_method 0 2024-04-26T00:24:06.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.169+0000 7f989b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:06.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.169+0000 7f989b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:06.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.169+0000 7f989b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:06.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.169+0000 7f989bfff640 1 --2- 172.21.15.37:0/3809026309 >> v2:172.21.15.37:3300/0 conn(0x7f989c131cd0 0x7f989c1320b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:06.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.169+0000 7f989bfff640 10 monclient(hunting): get_auth_request con 0x7f989c131cd0 auth_method 0 2024-04-26T00:24:06.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.169+0000 7f989bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:06.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.169+0000 7f989bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:06.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.169+0000 7f989bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:06.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.169+0000 7f989b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:06.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.169+0000 7f989b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:06.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.169+0000 7f989b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:06.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.169+0000 7f989bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:06.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.169+0000 7f989bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:06.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.169+0000 7f989bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:06.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.169+0000 7f989bfff640 10 monclient(hunting): handle_auth_done global_id 54129 payload 274 2024-04-26T00:24:06.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.169+0000 7f989bfff640 1 -- 172.21.15.37:0/3809026309 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f989c073080 msgr2=0x7f989c134c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:24:06.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.169+0000 7f989bfff640 1 --2- 172.21.15.37:0/3809026309 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f989c073080 0x7f989c134c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:06.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.169+0000 7f989bfff640 1 -- 172.21.15.37:0/3809026309 >> v1:172.21.15.37:6789/0 conn(0x7f989c0a1b60 legacy=0x7f989c135150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:24:06.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.169+0000 7f989bfff640 10 monclient: _finish_hunting 0 2024-04-26T00:24:06.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.169+0000 7f989bfff640 1 monclient: found mon.noname-a 2024-04-26T00:24:06.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.169+0000 7f989bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:24:06.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.169+0000 7f989bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:24:06.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.169+0000 7f989bfff640 1 -- 172.21.15.37:0/3809026309 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f989c1325f0 con 0x7f989c131cd0 2024-04-26T00:24:06.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.169+0000 7f989b7fe640 1 --2- 172.21.15.37:0/3809026309 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f989c073080 0x7f989c134c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:24:06.150 INFO:tasks.rados.rados.0.smithi045.stdout:760: writing smithi04526579-45 from 742942 to 1223256 tid 1 2024-04-26T00:24:06.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.173+0000 7f989bfff640 1 -- 172.21.15.37:0/3809026309 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f989c131cd0 msgr2=0x7f989c1320b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:06.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.173+0000 7f989bfff640 1 --2- 172.21.15.37:0/3809026309 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f989c131cd0 0x7f989c1320b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f988c0097c0 tx=0x7f988c01c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3183ea66d1152cdd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:06.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.173+0000 7f989affd640 1 -- 172.21.15.37:0/3809026309 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f988c0b2070 con 0x7f989c131cd0 2024-04-26T00:24:06.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.173+0000 7f989affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:06.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.173+0000 7f989affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:24:06.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.173+0000 7f989affd640 10 monclient: dump: 2024-04-26T00:24:06.152 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:06.152 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:06.152 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:06.152 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:06.152 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:06.152 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:06.152 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:06.152 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:06.152 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:06.152 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:06.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.173+0000 7f989affd640 10 monclient: _finish_auth 0 2024-04-26T00:24:06.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.173+0000 7f989affd640 10 monclient: _check_auth_tickets 2024-04-26T00:24:06.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.173+0000 7f989affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:24:06.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.173+0000 7f989affd640 1 -- 172.21.15.37:0/3809026309 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f989c135a70 con 0x7f989c131cd0 2024-04-26T00:24:06.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.173+0000 7f989affd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:24:06.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.173+0000 7f989affd640 1 -- 172.21.15.37:0/3809026309 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f988c0ad860 con 0x7f989c131cd0 2024-04-26T00:24:06.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.173+0000 7f989affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:06.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.173+0000 7f989affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:24:06.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.173+0000 7f989affd640 10 monclient: dump: 2024-04-26T00:24:06.153 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:06.153 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:06.153 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:06.153 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:06.153 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:06.153 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:06.153 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:06.153 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:06.153 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:06.153 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:06.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.173+0000 7f98a29cc640 5 monclient: authenticate success, global_id 54129 2024-04-26T00:24:06.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.173+0000 7f98a29cc640 10 monclient: get_monmap_and_config success 2024-04-26T00:24:06.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.173+0000 7f98a29cc640 4 set_mon_vals no callback set 2024-04-26T00:24:06.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.173+0000 7f98a29cc640 10 monclient: shutdown 2024-04-26T00:24:06.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.173+0000 7f98a29cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:24:06.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.173+0000 7f98a29cc640 1 -- 172.21.15.37:0/3809026309 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f989c131cd0 msgr2=0x7f989c1320b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:06.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.173+0000 7f98a29cc640 1 --2- 172.21.15.37:0/3809026309 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f989c131cd0 0x7f989c1320b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f988c0097c0 tx=0x7f988c01c3b0 comp rx=0 tx=0).stop 2024-04-26T00:24:06.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.173+0000 7f98a29cc640 1 -- 172.21.15.37:0/3809026309 shutdown_connections 2024-04-26T00:24:06.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.173+0000 7f98a29cc640 1 --2- 172.21.15.37:0/3809026309 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f989c073080 0x7f989c134c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:06.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.173+0000 7f98a29cc640 1 --2- 172.21.15.37:0/3809026309 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f989c131cd0 0x7f989c1320b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:06.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.173+0000 7f98a29cc640 1 -- 172.21.15.37:0/3809026309 >> 172.21.15.37:0/3809026309 conn(0x7f989c0a7160 msgr2=0x7f989c0a7540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:24:06.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.173+0000 7f98a29cc640 1 -- 172.21.15.37:0/3809026309 shutdown_connections 2024-04-26T00:24:06.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.173+0000 7f98a29cc640 1 -- 172.21.15.37:0/3809026309 wait complete. 2024-04-26T00:24:06.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.173+0000 7f98a29cc640 10 monclient: build_initial_monmap 2024-04-26T00:24:06.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.173+0000 7f98a29cc640 10 monclient: monmap: 2024-04-26T00:24:06.158 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:24:06.158 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:24:06.159 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:24:06.159 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:24:06.159 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:24:06.159 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:24:06.159 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:24:06.159 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:24:06.159 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:24:06.159 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:06.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f98a29cc640 1 Processor -- start 2024-04-26T00:24:06.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f98a29cc640 1 -- start start 2024-04-26T00:24:06.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f98a29cc640 10 monclient: init 2024-04-26T00:24:06.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f98a29cc640 10 monclient: _reopen_session rank -1 2024-04-26T00:24:06.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f98a29cc640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:24:06.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f98a29cc640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f989c073080 0x7f989c1cd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:06.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f98a29cc640 10 monclient(hunting): picked mon.noname-a con 0x7f989c073080 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:24:06.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f98a29cc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f989c0a1b60 0x7f989c13bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:06.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f98a29cc640 10 monclient(hunting): picked mon.noname-b con 0x7f989c0a1b60 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:24:06.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f98a29cc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f989c131cd0 0x7f989c13c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:06.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f98a29cc640 10 monclient(hunting): picked mon.noname-c con 0x7f989c131cd0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:24:06.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f98a29cc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:06.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f98a29cc640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f989c13fb00 con 0x7f989c073080 2024-04-26T00:24:06.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f98a29cc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:06.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f98a29cc640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f989c13f980 con 0x7f989c0a1b60 2024-04-26T00:24:06.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f98a29cc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:06.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f98a29cc640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f989c13fc80 con 0x7f989c131cd0 2024-04-26T00:24:06.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f98a29cc640 10 monclient(hunting): _renew_subs 2024-04-26T00:24:06.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f98a29cc640 10 monclient(hunting): authenticate will time out at 1227.484708s 2024-04-26T00:24:06.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f989bfff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f989c073080 0x7f989c1cd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:06.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f989bfff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f989c073080 0x7f989c1cd0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52740/0 (socket says 172.21.15.37:52740) 2024-04-26T00:24:06.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f98a0f69640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f989c131cd0 0x7f989c13c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:06.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f989bfff640 1 -- 172.21.15.37:0/755754991 learned_addr learned my addr 172.21.15.37:0/755754991 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:24:06.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f989bfff640 10 monclient(hunting): get_auth_request con 0x7f989c073080 auth_method 0 2024-04-26T00:24:06.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f989bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:06.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f989bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:06.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f989bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:06.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f98a0f69640 10 monclient(hunting): get_auth_request con 0x7f989c131cd0 auth_method 0 2024-04-26T00:24:06.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f98a0f69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:06.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f98a0f69640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:06.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f98a0f69640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:06.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f989bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:06.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f989bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:06.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f989bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:06.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f98a0f69640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:06.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f98a0f69640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:06.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f98a0f69640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:06.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f989b7fe640 1 --2- 172.21.15.37:0/755754991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f989c0a1b60 0x7f989c13bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:06.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f989b7fe640 10 monclient(hunting): get_auth_request con 0x7f989c0a1b60 auth_method 0 2024-04-26T00:24:06.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f989b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:06.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f989b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:06.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f989b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:06.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f989bfff640 10 monclient(hunting): handle_auth_done global_id 54132 payload 824 2024-04-26T00:24:06.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f989bfff640 1 -- 172.21.15.37:0/755754991 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f989c131cd0 msgr2=0x7f989c13c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:06.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f989bfff640 1 --2- 172.21.15.37:0/755754991 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f989c131cd0 0x7f989c13c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:06.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f989bfff640 1 -- 172.21.15.37:0/755754991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f989c0a1b60 msgr2=0x7f989c13bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:06.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f989bfff640 1 --2- 172.21.15.37:0/755754991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f989c0a1b60 0x7f989c13bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:06.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f989bfff640 10 monclient: _finish_hunting 0 2024-04-26T00:24:06.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f989bfff640 1 monclient: found mon.noname-a 2024-04-26T00:24:06.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f989bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:24:06.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f989bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:24:06.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f989bfff640 1 -- 172.21.15.37:0/755754991 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f989c134230 con 0x7f989c073080 2024-04-26T00:24:06.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f989bfff640 1 -- 172.21.15.37:0/755754991 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f989c073080 msgr2=0x7f989c1cd0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:06.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f989bfff640 1 --2- 172.21.15.37:0/755754991 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f989c073080 0x7f989c1cd0f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f988c01cc50 tx=0x7f988c019150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:06.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f9898ff9640 1 -- 172.21.15.37:0/755754991 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f988c0bb070 con 0x7f989c073080 2024-04-26T00:24:06.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f9898ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:06.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f9898ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:24:06.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f9898ff9640 10 monclient: dump: 2024-04-26T00:24:06.163 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:06.163 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:06.163 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:06.163 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:06.163 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:06.164 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:06.164 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:06.164 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:06.164 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:06.164 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:06.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f9898ff9640 10 monclient: _finish_auth 0 2024-04-26T00:24:06.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f9898ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:24:06.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f9898ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:24:06.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f9898ff9640 1 -- 172.21.15.37:0/755754991 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f988c0b63f0 con 0x7f989c073080 2024-04-26T00:24:06.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f98a29cc640 5 monclient: authenticate success, global_id 54132 2024-04-26T00:24:06.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f98a29cc640 10 monclient: _renew_subs 2024-04-26T00:24:06.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f98a29cc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:24:06.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f98a29cc640 1 -- 172.21.15.37:0/755754991 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f989c1387c0 con 0x7f989c073080 2024-04-26T00:24:06.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f9898ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:24:06.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f9898ff9640 1 -- 172.21.15.37:0/755754991 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f988c0b2040 con 0x7f989c073080 2024-04-26T00:24:06.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f9898ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:06.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f9898ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:24:06.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f9898ff9640 10 monclient: dump: 2024-04-26T00:24:06.165 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:06.165 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:06.165 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:06.165 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:06.165 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:06.165 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:06.165 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:06.165 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:06.165 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:06.165 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:06.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f98a29cc640 10 monclient: _renew_subs 2024-04-26T00:24:06.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f98a29cc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:24:06.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f989a7fc640 4 set_mon_vals no callback set 2024-04-26T00:24:06.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.177+0000 7f98a29cc640 1 -- 172.21.15.37:0/755754991 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f989c13a210 con 0x7f989c073080 2024-04-26T00:24:06.167 INFO:tasks.rados.rados.0.smithi045.stdout:760: writing smithi04526579-45 from 1823137 to 2471905 tid 2 2024-04-26T00:24:06.167 INFO:tasks.rados.rados.0.smithi045.stdout:761: read oid 41 snap -1 2024-04-26T00:24:06.167 INFO:tasks.rados.rados.0.smithi045.stdout:761: expect deleted 2024-04-26T00:24:06.167 INFO:tasks.rados.rados.0.smithi045.stdout:762: snap_remove snap 73 2024-04-26T00:24:06.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.181+0000 7f98a29cc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:24:06.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.181+0000 7f98a29cc640 1 --2- 172.21.15.37:0/755754991 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f989c136b20 0x7f989c136f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:06.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.181+0000 7f98a29cc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:06.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.181+0000 7f98a29cc640 1 -- 172.21.15.37:0/755754991 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f989c0708e0 con 0x7f989c136b20 2024-04-26T00:24:06.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.181+0000 7f9898ff9640 1 -- 172.21.15.37:0/755754991 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f988c0be070 con 0x7f989c073080 2024-04-26T00:24:06.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.181+0000 7f989b7fe640 1 --2- 172.21.15.37:0/755754991 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f989c136b20 0x7f989c136f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:06.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.181+0000 7f989b7fe640 10 monclient: get_auth_request con 0x7f989c136b20 auth_method 0 2024-04-26T00:24:06.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.181+0000 7f989b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:06.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.181+0000 7f989b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:06.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.181+0000 7f989b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:06.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.181+0000 7f9898ff9640 1 --2- 172.21.15.37:0/755754991 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9888035e80 0x7f9888038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:06.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.181+0000 7f989b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:06.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.181+0000 7f989b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:06.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.181+0000 7f989b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:06.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.181+0000 7f9898ff9640 1 -- 172.21.15.37:0/755754991 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(177..177 src has 1..177) ==== 3931+0+0 (secure 0 0 0) 0x7f988c0e2070 con 0x7f989c073080 2024-04-26T00:24:06.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.181+0000 7f989b7fe640 10 monclient(hunting): handle_auth_done global_id 54135 payload 824 2024-04-26T00:24:06.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.181+0000 7f989b7fe640 1 -- 172.21.15.37:0/755754991 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f989c0a6280 con 0x7f989c136b20 2024-04-26T00:24:06.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.181+0000 7f98a0f69640 1 --2- 172.21.15.37:0/755754991 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9888035e80 0x7f9888038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:06.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.181+0000 7f98a0f69640 10 monclient: get_auth_request con 0x7f9888035e80 auth_method 0 2024-04-26T00:24:06.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.181+0000 7f989b7fe640 1 -- 172.21.15.37:0/755754991 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f989c136b20 msgr2=0x7f989c136f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:06.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.181+0000 7f989b7fe640 1 --2- 172.21.15.37:0/755754991 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f989c136b20 0x7f989c136f00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f989c1383c0 tx=0x7f9890011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:06.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.185+0000 7f9898ff9640 1 -- 172.21.15.37:0/755754991 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9890015b60 con 0x7f989c136b20 2024-04-26T00:24:06.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.185+0000 7f9898ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:06.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.185+0000 7f9898ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:24:06.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.185+0000 7f9898ff9640 10 monclient: dump: 2024-04-26T00:24:06.172 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:06.172 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:06.172 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:06.172 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:06.172 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:06.172 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:06.172 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:06.172 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:06.172 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:06.172 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:06.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.185+0000 7f9898ff9640 1 -- 172.21.15.37:0/755754991 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9890015b60 con 0x7f989c136b20 2024-04-26T00:24:06.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.185+0000 7f9898ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:24:06.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.185+0000 7f9898ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:24:06.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.185+0000 7f9898ff9640 1 -- 172.21.15.37:0/755754991 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f989c136b20 msgr2=0x7f989c136f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:06.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.185+0000 7f9898ff9640 1 --2- 172.21.15.37:0/755754991 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f989c136b20 0x7f989c136f00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f989c1383c0 tx=0x7f9890011a60 comp rx=0 tx=0).stop 2024-04-26T00:24:06.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.185+0000 7f9898ff9640 1 -- 172.21.15.37:0/755754991 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f989c136b20 msgr2=0x7f989c136f00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:24:06.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.185+0000 7f9898ff9640 1 --2- 172.21.15.37:0/755754991 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f989c136b20 0x7f989c136f00 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f989c1383c0 tx=0x7f9890011a60 comp rx=0 tx=0).stop 2024-04-26T00:24:06.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.185+0000 7f98a0f69640 1 --2- 172.21.15.37:0/755754991 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9888035e80 0x7f9888038340 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f989400ea20 tx=0x7f989400e9b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:06.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.189+0000 7f98a29cc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:24:06.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.189+0000 7f98a29cc640 1 --2- 172.21.15.37:0/755754991 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f989c137440 0x7f989c09b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:06.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.189+0000 7f98a29cc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:06.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.189+0000 7f98a29cc640 1 -- 172.21.15.37:0/755754991 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f989c06e2f0 con 0x7f989c137440 2024-04-26T00:24:06.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.189+0000 7f989b7fe640 1 --2- 172.21.15.37:0/755754991 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f989c137440 0x7f989c09b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:06.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.189+0000 7f989b7fe640 10 monclient: get_auth_request con 0x7f989c137440 auth_method 0 2024-04-26T00:24:06.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.189+0000 7f989b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:06.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.189+0000 7f989b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:06.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.189+0000 7f989b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:06.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.189+0000 7f989b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:06.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.189+0000 7f989b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:06.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.189+0000 7f989b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:06.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.189+0000 7f989b7fe640 10 monclient(hunting): handle_auth_done global_id 54138 payload 824 2024-04-26T00:24:06.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.189+0000 7f989b7fe640 1 -- 172.21.15.37:0/755754991 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f989c04f510 con 0x7f989c137440 2024-04-26T00:24:06.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.193+0000 7f989b7fe640 1 -- 172.21.15.37:0/755754991 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f989c137440 msgr2=0x7f989c09b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:06.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.193+0000 7f989b7fe640 1 --2- 172.21.15.37:0/755754991 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f989c137440 0x7f989c09b060 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f989c1394f0 tx=0x7f9890004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:06.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.193+0000 7f9898ff9640 1 -- 172.21.15.37:0/755754991 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9890006460 con 0x7f989c137440 2024-04-26T00:24:06.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.193+0000 7f9898ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:06.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.193+0000 7f9898ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:24:06.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.193+0000 7f9898ff9640 10 monclient: dump: 2024-04-26T00:24:06.175 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:06.175 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:06.175 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:06.175 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:06.175 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:06.175 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:06.175 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:06.175 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:06.176 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:06.176 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:06.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.193+0000 7f9898ff9640 1 -- 172.21.15.37:0/755754991 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4715 (secure 0 0 0) 0x7f9890005e70 con 0x7f989c137440 2024-04-26T00:24:06.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.193+0000 7f9898ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:24:06.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.193+0000 7f9898ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:24:06.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.193+0000 7f9898ff9640 1 -- 172.21.15.37:0/755754991 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f989c137440 msgr2=0x7f989c09b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:06.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.193+0000 7f9898ff9640 1 --2- 172.21.15.37:0/755754991 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f989c137440 0x7f989c09b060 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f989c1394f0 tx=0x7f9890004740 comp rx=0 tx=0).stop 2024-04-26T00:24:06.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.193+0000 7f9898ff9640 1 -- 172.21.15.37:0/755754991 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f989c137440 msgr2=0x7f989c09b060 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:24:06.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.193+0000 7f9898ff9640 1 --2- 172.21.15.37:0/755754991 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f989c137440 0x7f989c09b060 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f989c1394f0 tx=0x7f9890004740 comp rx=0 tx=0).stop 2024-04-26T00:24:06.176 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:24:06.177 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:24:06.177 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:24:06.177 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:24:06.177 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 34, 2024-04-26T00:24:06.177 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 9519, 2024-04-26T00:24:06.177 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:24:06.177 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:24:06.177 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:24:06.177 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:24:06.177 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:06.177 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 3, 2024-04-26T00:24:06.177 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:24:06.177 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:24:06.177 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:24:06.178 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:24:06.178 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:24:06.178 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:24:06.178 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:24:06.178 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:24:06.178 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:24:06.179 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:24:06.179 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:24:06.179 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:24:06.179 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:24:06.179 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:24:06.179 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:06.179 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:24:06.179 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:24:06.179 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:24:06.180 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:24:06.180 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:24:06.180 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:24:06.180 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:24:06.180 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:24:06.180 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:24:06.180 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:24:06.180 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:24:06.180 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:24:06.180 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:24:06.180 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:06.180 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:06.180 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:24:06.181 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:24:06.181 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:24:06.181 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:24:06.181 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:24:06.181 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:24:06.181 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:24:06.181 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:24:06.181 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:24:06.181 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:24:06.181 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:24:06.181 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:24:06.181 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:24:06.181 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:24:06.181 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:24:06.182 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:24:06.182 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:24:06.182 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:24:06.182 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:24:06.182 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:24:06.182 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:24:06.182 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:24:06.182 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:24:06.182 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:24:06.182 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:24:06.182 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:24:06.182 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:24:06.182 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:24:06.183 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:06.183 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:24:06.183 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:06.183 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:24:06.183 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:06.183 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:24:06.183 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:24:06.183 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:24:06.183 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:24:06.183 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:06.183 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:24:06.183 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:24:06.183 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:06.184 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:06.184 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:06.184 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:24:06.184 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:24:06.184 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:06.184 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:06.184 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:06.184 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:06.184 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:24:06.184 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:24:06.184 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:24:06.184 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:24:06.184 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:24:06.184 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:06.185 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:06.185 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:24:06.185 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:24:06.185 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:24:06.185 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:24:06.185 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:06.185 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:24:06.185 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:24:06.185 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:06.185 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:06.185 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:06.185 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:24:06.185 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:24:06.185 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:06.186 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:06.186 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:06.186 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:06.186 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:24:06.186 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:24:06.186 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:24:06.186 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:24:06.186 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:24:06.186 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:06.186 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:06.186 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:24:06.187 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:24:06.187 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:24:06.187 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:24:06.187 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:06.187 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:24:06.187 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:24:06.187 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:06.187 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:06.187 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:06.187 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:24:06.187 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:24:06.187 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:06.187 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:06.188 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:06.188 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:06.188 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:24:06.188 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:24:06.188 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:24:06.188 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:24:06.188 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:24:06.188 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:06.188 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:06.188 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:06.188 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:24:06.188 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:24:06.188 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:06.188 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:24:06.188 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:24:06.188 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:24:06.189 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:06.189 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:06.189 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:24:06.189 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:06.189 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:24:06.189 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:24:06.189 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:24:06.189 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:06.189 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:06.189 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:24:06.189 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:06.189 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:24:06.189 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:24:06.189 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:24:06.190 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:06.190 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:06.190 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:06.190 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:24:06.190 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:24:06.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.193+0000 7f98a29cc640 1 -- 172.21.15.37:0/755754991 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9888035e80 msgr2=0x7f9888038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:06.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.193+0000 7f98a29cc640 1 --2- 172.21.15.37:0/755754991 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9888035e80 0x7f9888038340 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f989400ea20 tx=0x7f989400e9b0 comp rx=0 tx=0).stop 2024-04-26T00:24:06.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.193+0000 7f98a29cc640 10 monclient: shutdown 2024-04-26T00:24:06.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.193+0000 7f98a29cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:24:06.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.193+0000 7f98a29cc640 1 -- 172.21.15.37:0/755754991 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f989c073080 msgr2=0x7f989c1cd0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:06.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.193+0000 7f98a29cc640 1 --2- 172.21.15.37:0/755754991 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f989c073080 0x7f989c1cd0f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f988c01cc50 tx=0x7f988c019150 comp rx=0 tx=0).stop 2024-04-26T00:24:06.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.193+0000 7f98a0f69640 1 -- 172.21.15.37:0/755754991 reap_dead start 2024-04-26T00:24:06.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.193+0000 7f98a29cc640 1 -- 172.21.15.37:0/755754991 shutdown_connections 2024-04-26T00:24:06.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.193+0000 7f98a29cc640 1 --2- 172.21.15.37:0/755754991 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f989c073080 0x7f989c1cd0f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:06.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.193+0000 7f98a29cc640 1 -- 172.21.15.37:0/755754991 >> 172.21.15.37:0/755754991 conn(0x7f989c0a7160 msgr2=0x7f989c133b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:24:06.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.193+0000 7f98a29cc640 1 -- 172.21.15.37:0/755754991 shutdown_connections 2024-04-26T00:24:06.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.193+0000 7f98a29cc640 1 -- 172.21.15.37:0/755754991 wait complete. 2024-04-26T00:24:06.192 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:24:06.192 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:24:06.192 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:24:06.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.329+0000 7fc3f81f5640 10 monclient: get_monmap_and_config 2024-04-26T00:24:06.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.329+0000 7fc3f81f5640 10 monclient: build_initial_monmap 2024-04-26T00:24:06.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.329+0000 7fc3f81f5640 10 monclient: monmap: 2024-04-26T00:24:06.307 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:24:06.307 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:06.307 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:24:06.332787+0000 2024-04-26T00:24:06.307 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:24:06.332787+0000 2024-04-26T00:24:06.307 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:24:06.307 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:06.307 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:24:06.307 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:24:06.307 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:24:06.308 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:24:06.308 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:24:06.308 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:06.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.329+0000 7fc3f81f5640 1 Processor -- start 2024-04-26T00:24:06.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.329+0000 7fc3f81f5640 1 -- start start 2024-04-26T00:24:06.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.329+0000 7fc3f81f5640 10 monclient: init 2024-04-26T00:24:06.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.329+0000 7fc3f81f5640 10 monclient: _reopen_session rank -1 2024-04-26T00:24:06.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.329+0000 7fc3f81f5640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-04-26T00:24:06.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.329+0000 7fc3f81f5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc3f00a0300 addr v1:172.21.15.45:6789/0 2024-04-26T00:24:06.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.329+0000 7fc3f81f5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc3f009fc50 addr v1:172.21.15.37:6789/0 2024-04-26T00:24:06.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.329+0000 7fc3f81f5640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc3f013b5e0 0x7fc3f013d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:06.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.329+0000 7fc3f81f5640 10 monclient(hunting): picked mon.noname-c con 0x7fc3f013b5e0 addr v2:172.21.15.45:3300/0 2024-04-26T00:24:06.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.329+0000 7fc3f81f5640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3f009c190 con 0x7fc3f009fc50 2024-04-26T00:24:06.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.329+0000 7fc3f81f5640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3f00a1110 con 0x7fc3f00a0300 2024-04-26T00:24:06.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.329+0000 7fc3f81f5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:06.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.329+0000 7fc3f81f5640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc3f00a16d0 con 0x7fc3f013b5e0 2024-04-26T00:24:06.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.329+0000 7fc3f81f5640 10 monclient(hunting): _renew_subs 2024-04-26T00:24:06.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.329+0000 7fc3f81f5640 10 monclient(hunting): authenticate will time out at 1227.638491s 2024-04-26T00:24:06.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.329+0000 7fc3f5790640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fc3f009fc50 0x7fc3f0139ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:52100/0 (socket says 172.21.15.37:52100) 2024-04-26T00:24:06.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.329+0000 7fc3f5790640 1 -- 172.21.15.37:0/414369276 learned_addr learned my addr 172.21.15.37:0/414369276 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:24:06.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.329+0000 7fc3f6792640 1 --2- 172.21.15.37:0/414369276 >> v2:172.21.15.45:3300/0 conn(0x7fc3f013b5e0 0x7fc3f013d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:06.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.329+0000 7fc3f6792640 10 monclient(hunting): get_auth_request con 0x7fc3f013b5e0 auth_method 0 2024-04-26T00:24:06.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.329+0000 7fc3f6792640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:06.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.329+0000 7fc3f6792640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:06.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.329+0000 7fc3f6792640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:06.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.329+0000 7fc3f6792640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:06.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.329+0000 7fc3f6792640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:06.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.329+0000 7fc3f6792640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:06.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.329+0000 7fc3f6792640 10 monclient(hunting): handle_auth_done global_id 24376 payload 274 2024-04-26T00:24:06.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.329+0000 7fc3f6792640 1 -- 172.21.15.37:0/414369276 >> v1:172.21.15.45:6789/0 conn(0x7fc3f00a0300 legacy=0x7fc3f00a06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:24:06.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.329+0000 7fc3f6792640 1 -- 172.21.15.37:0/414369276 >> v1:172.21.15.37:6789/0 conn(0x7fc3f009fc50 legacy=0x7fc3f0139ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:24:06.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.329+0000 7fc3f6792640 10 monclient: _finish_hunting 0 2024-04-26T00:24:06.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.329+0000 7fc3f6792640 1 monclient: found mon.noname-c 2024-04-26T00:24:06.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.329+0000 7fc3f6792640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:24:06.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.329+0000 7fc3f6792640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:24:06.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.329+0000 7fc3f6792640 1 -- 172.21.15.37:0/414369276 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3f009f2a0 con 0x7fc3f013b5e0 2024-04-26T00:24:06.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.337+0000 7fc3f6792640 1 -- 172.21.15.37:0/414369276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3f013b5e0 msgr2=0x7fc3f013d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:06.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.337+0000 7fc3f6792640 1 --2- 172.21.15.37:0/414369276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3f013b5e0 0x7fc3f013d9d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc3f009f4d0 tx=0x7fc3e001c2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91cd909ed66f8c10 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:06.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.337+0000 7fc3f4f8f640 1 -- 172.21.15.37:0/414369276 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e00b0070 con 0x7fc3f013b5e0 2024-04-26T00:24:06.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.337+0000 7fc3f4f8f640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:06.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.337+0000 7fc3f4f8f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:24:06.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.337+0000 7fc3f4f8f640 10 monclient: dump: 2024-04-26T00:24:06.316 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:06.316 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:06.316 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:06.316 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:06.316 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:06.317 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:06.317 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:06.317 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:06.317 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:06.317 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:06.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.337+0000 7fc3f4f8f640 10 monclient: _finish_auth 0 2024-04-26T00:24:06.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.337+0000 7fc3f4f8f640 10 monclient: _check_auth_tickets 2024-04-26T00:24:06.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.337+0000 7fc3f4f8f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:24:06.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.337+0000 7fc3f4f8f640 1 -- 172.21.15.37:0/414369276 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3e00ab600 con 0x7fc3f013b5e0 2024-04-26T00:24:06.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.337+0000 7fc3f4f8f640 10 monclient: handle_config config(0 keys) 2024-04-26T00:24:06.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.337+0000 7fc3f81f5640 5 monclient: authenticate success, global_id 24376 2024-04-26T00:24:06.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.337+0000 7fc3f4f8f640 1 -- 172.21.15.37:0/414369276 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e00ab9d0 con 0x7fc3f013b5e0 2024-04-26T00:24:06.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.337+0000 7fc3f4f8f640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:06.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.337+0000 7fc3f4f8f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:24:06.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.337+0000 7fc3f4f8f640 10 monclient: dump: 2024-04-26T00:24:06.318 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:06.318 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:06.318 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:06.318 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:06.318 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:06.318 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:06.318 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:06.318 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:06.318 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:06.318 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:06.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.337+0000 7fc3f81f5640 10 monclient: get_monmap_and_config success 2024-04-26T00:24:06.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.337+0000 7fc3f81f5640 4 set_mon_vals no callback set 2024-04-26T00:24:06.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.337+0000 7fc3f81f5640 10 monclient: shutdown 2024-04-26T00:24:06.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.337+0000 7fc3f81f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:24:06.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.337+0000 7fc3f81f5640 1 -- 172.21.15.37:0/414369276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3f013b5e0 msgr2=0x7fc3f013d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:06.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.337+0000 7fc3f81f5640 1 --2- 172.21.15.37:0/414369276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3f013b5e0 0x7fc3f013d9d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc3f009f4d0 tx=0x7fc3e001c2e0 comp rx=0 tx=0).stop 2024-04-26T00:24:06.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.337+0000 7fc3f81f5640 1 -- 172.21.15.37:0/414369276 shutdown_connections 2024-04-26T00:24:06.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.337+0000 7fc3f81f5640 1 --2- 172.21.15.37:0/414369276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3f013b5e0 0x7fc3f013d9d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:06.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.337+0000 7fc3f81f5640 1 -- 172.21.15.37:0/414369276 >> 172.21.15.37:0/414369276 conn(0x7fc3f00a1d50 msgr2=0x7fc3f00a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:24:06.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.337+0000 7fc3f81f5640 1 -- 172.21.15.37:0/414369276 shutdown_connections 2024-04-26T00:24:06.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.337+0000 7fc3f81f5640 1 -- 172.21.15.37:0/414369276 wait complete. 2024-04-26T00:24:06.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.337+0000 7fc3f81f5640 10 monclient: build_initial_monmap 2024-04-26T00:24:06.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.337+0000 7fc3f81f5640 10 monclient: monmap: 2024-04-26T00:24:06.324 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:24:06.324 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:24:06.324 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:24:06.324 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:24:06.324 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:24:06.324 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:24:06.324 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:24:06.324 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:24:06.325 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:24:06.325 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:06.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.337+0000 7fc3f81f5640 1 Processor -- start 2024-04-26T00:24:06.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.337+0000 7fc3f81f5640 1 -- start start 2024-04-26T00:24:06.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.337+0000 7fc3f81f5640 10 monclient: init 2024-04-26T00:24:06.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f81f5640 10 monclient: _reopen_session rank -1 2024-04-26T00:24:06.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f81f5640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:24:06.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f81f5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3f009fc50 0x7fc3f01397d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:06.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f81f5640 10 monclient(hunting): picked mon.noname-b con 0x7fc3f009fc50 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:24:06.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f81f5640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc3f00a0300 0x7fc3f0134a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:06.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f81f5640 10 monclient(hunting): picked mon.noname-a con 0x7fc3f00a0300 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:24:06.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f81f5640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc3f013b5e0 0x7fc3f011b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:06.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f81f5640 10 monclient(hunting): picked mon.noname-c con 0x7fc3f013b5e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:24:06.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f81f5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:06.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f81f5640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f0140ac0 con 0x7fc3f00a0300 2024-04-26T00:24:06.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f81f5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:06.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f81f5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f0140940 con 0x7fc3f009fc50 2024-04-26T00:24:06.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f81f5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:06.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f81f5640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc3f0140c40 con 0x7fc3f013b5e0 2024-04-26T00:24:06.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f81f5640 10 monclient(hunting): _renew_subs 2024-04-26T00:24:06.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f81f5640 10 monclient(hunting): authenticate will time out at 1227.648564s 2024-04-26T00:24:06.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f5790640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc3f00a0300 0x7fc3f0134a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:06.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f5790640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc3f00a0300 0x7fc3f0134a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52778/0 (socket says 172.21.15.37:52778) 2024-04-26T00:24:06.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f5790640 1 -- 172.21.15.37:0/3792607985 learned_addr learned my addr 172.21.15.37:0/3792607985 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:24:06.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f5790640 10 monclient(hunting): get_auth_request con 0x7fc3f00a0300 auth_method 0 2024-04-26T00:24:06.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f6792640 1 --2- 172.21.15.37:0/3792607985 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc3f013b5e0 0x7fc3f011b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:06.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f5790640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:06.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f5790640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:06.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f5790640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:06.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f6792640 10 monclient(hunting): get_auth_request con 0x7fc3f013b5e0 auth_method 0 2024-04-26T00:24:06.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f6792640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:06.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f6792640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:06.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f6792640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:06.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f5790640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:06.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f5790640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:06.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f5790640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:06.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f6792640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:06.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f6792640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:06.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f6792640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:06.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f5f91640 1 --2- 172.21.15.37:0/3792607985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3f009fc50 0x7fc3f01397d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:06.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f5f91640 10 monclient(hunting): get_auth_request con 0x7fc3f009fc50 auth_method 0 2024-04-26T00:24:06.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f5f91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:06.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f5f91640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:06.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f5f91640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:06.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f5790640 10 monclient(hunting): handle_auth_done global_id 54141 payload 824 2024-04-26T00:24:06.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f5790640 1 -- 172.21.15.37:0/3792607985 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc3f013b5e0 msgr2=0x7fc3f011b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:06.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f5790640 1 --2- 172.21.15.37:0/3792607985 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc3f013b5e0 0x7fc3f011b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:06.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f5790640 1 -- 172.21.15.37:0/3792607985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3f009fc50 msgr2=0x7fc3f01397d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:06.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f5790640 1 --2- 172.21.15.37:0/3792607985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3f009fc50 0x7fc3f01397d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:06.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f5790640 10 monclient: _finish_hunting 0 2024-04-26T00:24:06.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f5790640 1 monclient: found mon.noname-a 2024-04-26T00:24:06.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f5790640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:24:06.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f5790640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:24:06.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f5790640 1 -- 172.21.15.37:0/3792607985 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3f013a460 con 0x7fc3f00a0300 2024-04-26T00:24:06.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f6792640 1 --2- 172.21.15.37:0/3792607985 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc3f013b5e0 0x7fc3f011b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:24:06.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f5f91640 1 --2- 172.21.15.37:0/3792607985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3f009fc50 0x7fc3f01397d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:24:06.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f5790640 1 -- 172.21.15.37:0/3792607985 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc3f00a0300 msgr2=0x7fc3f0134a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:06.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f5790640 1 --2- 172.21.15.37:0/3792607985 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc3f00a0300 0x7fc3f0134a20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc3e400c4a0 tx=0x7fc3e400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:06.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3ee7fc640 1 -- 172.21.15.37:0/3792607985 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e4017070 con 0x7fc3f00a0300 2024-04-26T00:24:06.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:06.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3ee7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:24:06.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3ee7fc640 10 monclient: dump: 2024-04-26T00:24:06.329 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:06.329 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:06.329 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:06.329 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:06.329 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:06.329 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:06.329 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:06.329 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:06.329 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:06.329 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:06.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3ee7fc640 10 monclient: _finish_auth 0 2024-04-26T00:24:06.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3ee7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:24:06.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:24:06.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3ee7fc640 1 -- 172.21.15.37:0/3792607985 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3e4007070 con 0x7fc3f00a0300 2024-04-26T00:24:06.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3ee7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:24:06.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f81f5640 5 monclient: authenticate success, global_id 54141 2024-04-26T00:24:06.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3ee7fc640 1 -- 172.21.15.37:0/3792607985 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e4007460 con 0x7fc3f00a0300 2024-04-26T00:24:06.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:06.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3ee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:24:06.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3ee7fc640 10 monclient: dump: 2024-04-26T00:24:06.330 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:06.330 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:06.330 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:06.330 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:06.330 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:06.330 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:06.331 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:06.331 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:06.331 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:06.331 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:06.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f81f5640 10 monclient: _renew_subs 2024-04-26T00:24:06.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3ef7fe640 4 set_mon_vals no callback set 2024-04-26T00:24:06.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f81f5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:24:06.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f81f5640 1 -- 172.21.15.37:0/3792607985 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3f009f550 con 0x7fc3f00a0300 2024-04-26T00:24:06.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f81f5640 10 monclient: _renew_subs 2024-04-26T00:24:06.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f81f5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:24:06.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.341+0000 7fc3f81f5640 1 -- 172.21.15.37:0/3792607985 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3f011db00 con 0x7fc3f00a0300 2024-04-26T00:24:06.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.345+0000 7fc3f81f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:24:06.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.345+0000 7fc3f81f5640 1 --2- 172.21.15.37:0/3792607985 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc3f01c5d10 0x7fc3f011bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:06.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.345+0000 7fc3f81f5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:06.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.345+0000 7fc3f81f5640 1 -- 172.21.15.37:0/3792607985 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc3f004f510 con 0x7fc3f01c5d10 2024-04-26T00:24:06.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.345+0000 7fc3ee7fc640 1 -- 172.21.15.37:0/3792607985 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc3e401a070 con 0x7fc3f00a0300 2024-04-26T00:24:06.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.345+0000 7fc3f5f91640 1 --2- 172.21.15.37:0/3792607985 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc3f01c5d10 0x7fc3f011bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:06.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.345+0000 7fc3f5f91640 10 monclient: get_auth_request con 0x7fc3f01c5d10 auth_method 0 2024-04-26T00:24:06.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.345+0000 7fc3f5f91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:06.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.345+0000 7fc3f5f91640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:06.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.345+0000 7fc3f5f91640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:06.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.345+0000 7fc3f5f91640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:06.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.345+0000 7fc3f5f91640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:06.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.345+0000 7fc3f5f91640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:06.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.345+0000 7fc3ee7fc640 1 --2- 172.21.15.37:0/3792607985 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc3dc035e80 0x7fc3dc038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:06.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.345+0000 7fc3ee7fc640 1 -- 172.21.15.37:0/3792607985 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(177..177 src has 1..177) ==== 3931+0+0 (secure 0 0 0) 0x7fc3e4047f10 con 0x7fc3f00a0300 2024-04-26T00:24:06.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.345+0000 7fc3f5f91640 10 monclient(hunting): handle_auth_done global_id 34328 payload 824 2024-04-26T00:24:06.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.345+0000 7fc3f5f91640 1 -- 172.21.15.37:0/3792607985 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc3f009eb60 con 0x7fc3f01c5d10 2024-04-26T00:24:06.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.345+0000 7fc3f6792640 1 --2- 172.21.15.37:0/3792607985 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc3dc035e80 0x7fc3dc038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:06.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.345+0000 7fc3f6792640 10 monclient: get_auth_request con 0x7fc3dc035e80 auth_method 0 2024-04-26T00:24:06.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.345+0000 7fc3f5f91640 1 -- 172.21.15.37:0/3792607985 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc3f01c5d10 msgr2=0x7fc3f011bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:06.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.345+0000 7fc3f5f91640 1 --2- 172.21.15.37:0/3792607985 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc3f01c5d10 0x7fc3f011bde0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc3e800e890 tx=0x7fc3e800ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:06.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.345+0000 7fc3ee7fc640 1 -- 172.21.15.37:0/3792607985 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e800cb50 con 0x7fc3f01c5d10 2024-04-26T00:24:06.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.345+0000 7fc3ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:06.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.345+0000 7fc3ee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:24:06.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.345+0000 7fc3ee7fc640 10 monclient: dump: 2024-04-26T00:24:06.336 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:06.336 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:06.336 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:06.337 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:06.337 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:06.337 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:06.337 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:06.337 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:06.337 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:06.337 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:06.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.345+0000 7fc3ee7fc640 1 -- 172.21.15.37:0/3792607985 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc3e801a070 con 0x7fc3f01c5d10 2024-04-26T00:24:06.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.345+0000 7fc3ee7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:24:06.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.345+0000 7fc3ee7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:24:06.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.345+0000 7fc3ee7fc640 1 -- 172.21.15.37:0/3792607985 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc3f01c5d10 msgr2=0x7fc3f011bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:06.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.345+0000 7fc3ee7fc640 1 --2- 172.21.15.37:0/3792607985 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc3f01c5d10 0x7fc3f011bde0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc3e800e890 tx=0x7fc3e800ed50 comp rx=0 tx=0).stop 2024-04-26T00:24:06.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.345+0000 7fc3ee7fc640 1 -- 172.21.15.37:0/3792607985 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc3f01c5d10 msgr2=0x7fc3f011bde0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:24:06.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.345+0000 7fc3ee7fc640 1 --2- 172.21.15.37:0/3792607985 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc3f01c5d10 0x7fc3f011bde0 secure :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc3e800e890 tx=0x7fc3e800ed50 comp rx=0 tx=0).stop 2024-04-26T00:24:06.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.349+0000 7fc3f6792640 1 --2- 172.21.15.37:0/3792607985 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc3dc035e80 0x7fc3dc038340 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc3e00ab000 tx=0x7fc3e0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:06.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.353+0000 7fc3f81f5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:24:06.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.353+0000 7fc3f81f5640 1 --2- 172.21.15.37:0/3792607985 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc3f011c320 0x7fc3f0135c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:06.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.353+0000 7fc3f81f5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:06.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.353+0000 7fc3f81f5640 1 -- 172.21.15.37:0/3792607985 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc3f011c730 con 0x7fc3f011c320 2024-04-26T00:24:06.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.353+0000 7fc3f5f91640 1 --2- 172.21.15.37:0/3792607985 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc3f011c320 0x7fc3f0135c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:06.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.353+0000 7fc3f5f91640 10 monclient: get_auth_request con 0x7fc3f011c320 auth_method 0 2024-04-26T00:24:06.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.353+0000 7fc3f5f91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:06.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.353+0000 7fc3f5f91640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:06.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.353+0000 7fc3f5f91640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:06.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.353+0000 7fc3f5f91640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:06.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.353+0000 7fc3f5f91640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:06.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.353+0000 7fc3f5f91640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:06.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.353+0000 7fc3f5f91640 10 monclient(hunting): handle_auth_done global_id 34331 payload 824 2024-04-26T00:24:06.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.353+0000 7fc3f5f91640 1 -- 172.21.15.37:0/3792607985 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc3f006ba50 con 0x7fc3f011c320 2024-04-26T00:24:06.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.357+0000 7fc3f5f91640 1 -- 172.21.15.37:0/3792607985 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc3f011c320 msgr2=0x7fc3f0135c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:06.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.357+0000 7fc3f5f91640 1 --2- 172.21.15.37:0/3792607985 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc3f011c320 0x7fc3f0135c70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc3e800e890 tx=0x7fc3e8003af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:06.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.357+0000 7fc3ee7fc640 1 -- 172.21.15.37:0/3792607985 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e8025070 con 0x7fc3f011c320 2024-04-26T00:24:06.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.357+0000 7fc3ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:06.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.357+0000 7fc3ee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:24:06.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.357+0000 7fc3ee7fc640 10 monclient: dump: 2024-04-26T00:24:06.339 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:06.339 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:06.339 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:06.339 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:06.339 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:06.339 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:06.340 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:06.340 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:06.340 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:06.340 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:06.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.357+0000 7fc3ee7fc640 1 -- 172.21.15.37:0/3792607985 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4719 (secure 0 0 0) 0x7fc3e8029450 con 0x7fc3f011c320 2024-04-26T00:24:06.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.357+0000 7fc3ee7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:24:06.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.357+0000 7fc3ee7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:24:06.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.357+0000 7fc3ee7fc640 1 -- 172.21.15.37:0/3792607985 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc3f011c320 msgr2=0x7fc3f0135c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:06.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.357+0000 7fc3ee7fc640 1 --2- 172.21.15.37:0/3792607985 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc3f011c320 0x7fc3f0135c70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc3e800e890 tx=0x7fc3e8003af0 comp rx=0 tx=0).stop 2024-04-26T00:24:06.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.357+0000 7fc3ee7fc640 1 -- 172.21.15.37:0/3792607985 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc3f011c320 msgr2=0x7fc3f0135c70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:24:06.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.357+0000 7fc3ee7fc640 1 --2- 172.21.15.37:0/3792607985 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc3f011c320 0x7fc3f0135c70 secure :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc3e800e890 tx=0x7fc3e8003af0 comp rx=0 tx=0).stop 2024-04-26T00:24:06.340 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:24:06.341 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:24:06.341 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:24:06.341 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:24:06.341 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 34, 2024-04-26T00:24:06.341 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 106635, 2024-04-26T00:24:06.341 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:24:06.341 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:24:06.341 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:24:06.341 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:24:06.341 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:06.341 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 3, 2024-04-26T00:24:06.341 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:24:06.341 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:24:06.342 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:24:06.342 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:24:06.342 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:24:06.342 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:24:06.342 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:24:06.342 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:24:06.342 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:24:06.342 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:24:06.342 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:24:06.342 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:24:06.342 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:24:06.342 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:24:06.342 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:06.342 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:24:06.343 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:24:06.343 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:24:06.343 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:24:06.343 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:24:06.343 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:24:06.343 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:24:06.343 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:24:06.343 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:24:06.343 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:24:06.343 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:24:06.343 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:24:06.343 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:24:06.343 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:06.343 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:06.343 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:24:06.344 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:24:06.344 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:24:06.344 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:24:06.344 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:24:06.344 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:24:06.344 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:24:06.344 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:24:06.344 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:24:06.344 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:24:06.344 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:24:06.344 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:24:06.344 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:24:06.344 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:24:06.344 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:24:06.345 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:24:06.345 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:24:06.345 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:24:06.345 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:24:06.345 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:24:06.345 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:24:06.345 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:24:06.345 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:24:06.345 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:24:06.345 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:24:06.345 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:24:06.345 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:24:06.345 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:24:06.345 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:06.346 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:24:06.346 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:06.346 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:24:06.346 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:06.346 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:24:06.346 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:24:06.346 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:24:06.346 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:24:06.346 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:06.346 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:24:06.346 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:24:06.346 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:06.346 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:06.346 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:06.347 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:24:06.347 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:24:06.347 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:06.347 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:06.347 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:06.347 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:06.347 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:24:06.347 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:24:06.347 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:24:06.347 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:24:06.347 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:24:06.347 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:06.347 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:06.347 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:24:06.348 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:24:06.348 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:24:06.348 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:24:06.348 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:06.348 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:24:06.348 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:24:06.348 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:06.348 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:06.348 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:06.348 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:24:06.348 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:24:06.348 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:06.348 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:06.348 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:06.348 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:06.349 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:24:06.349 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:24:06.349 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:24:06.349 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:24:06.349 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:24:06.349 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:06.349 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:06.349 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:24:06.349 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:24:06.349 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:24:06.349 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:24:06.349 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:06.349 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:24:06.349 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:24:06.350 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:06.350 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:06.350 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:06.350 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:24:06.350 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:24:06.350 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:06.350 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:06.350 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:06.350 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:06.350 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:24:06.350 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:24:06.350 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:24:06.350 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:24:06.351 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:24:06.351 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:06.351 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:06.351 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:06.351 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:24:06.351 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:24:06.351 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:06.351 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:24:06.351 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:24:06.351 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:24:06.351 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:06.351 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:06.351 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:24:06.351 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:06.351 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:24:06.352 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:24:06.352 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:24:06.352 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:06.352 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:06.352 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:24:06.352 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:06.352 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:24:06.352 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:24:06.352 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:24:06.352 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:06.352 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:06.352 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:06.352 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:24:06.352 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:24:06.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.357+0000 7fc3f81f5640 1 -- 172.21.15.37:0/3792607985 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc3dc035e80 msgr2=0x7fc3dc038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:06.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.357+0000 7fc3f81f5640 1 --2- 172.21.15.37:0/3792607985 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc3dc035e80 0x7fc3dc038340 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc3e00ab000 tx=0x7fc3e0008040 comp rx=0 tx=0).stop 2024-04-26T00:24:06.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.357+0000 7fc3f81f5640 10 monclient: shutdown 2024-04-26T00:24:06.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.357+0000 7fc3f81f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:24:06.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.357+0000 7fc3f81f5640 1 -- 172.21.15.37:0/3792607985 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc3f00a0300 msgr2=0x7fc3f0134a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:06.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.357+0000 7fc3f81f5640 1 --2- 172.21.15.37:0/3792607985 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc3f00a0300 0x7fc3f0134a20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc3e400c4a0 tx=0x7fc3e400c960 comp rx=0 tx=0).stop 2024-04-26T00:24:06.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.357+0000 7fc3f6792640 1 -- 172.21.15.37:0/3792607985 reap_dead start 2024-04-26T00:24:06.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.357+0000 7fc3f6792640 1 -- 172.21.15.37:0/3792607985 reap_dead start 2024-04-26T00:24:06.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.357+0000 7fc3f81f5640 1 -- 172.21.15.37:0/3792607985 shutdown_connections 2024-04-26T00:24:06.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.357+0000 7fc3f81f5640 1 -- 172.21.15.37:0/3792607985 >> 172.21.15.37:0/3792607985 conn(0x7fc3f00a1d50 msgr2=0x7fc3f013aed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:24:06.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.357+0000 7fc3f81f5640 1 -- 172.21.15.37:0/3792607985 shutdown_connections 2024-04-26T00:24:06.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.357+0000 7fc3f81f5640 1 -- 172.21.15.37:0/3792607985 wait complete. 2024-04-26T00:24:06.355 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:24:06.355 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:24:06.355 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:24:06.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34f026f640 10 monclient: get_monmap_and_config 2024-04-26T00:24:06.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34f026f640 10 monclient: build_initial_monmap 2024-04-26T00:24:06.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34f026f640 10 monclient: monmap: 2024-04-26T00:24:06.471 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:24:06.471 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:06.471 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:24:06.496599+0000 2024-04-26T00:24:06.471 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:24:06.496599+0000 2024-04-26T00:24:06.471 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:24:06.471 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:06.471 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:24:06.471 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:24:06.471 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:24:06.471 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:24:06.471 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:24:06.471 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:06.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34f026f640 1 Processor -- start 2024-04-26T00:24:06.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34f026f640 1 -- start start 2024-04-26T00:24:06.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34f026f640 10 monclient: init 2024-04-26T00:24:06.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34f026f640 10 monclient: _reopen_session rank -1 2024-04-26T00:24:06.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34f026f640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-04-26T00:24:06.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34f026f640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f34e8136f20 0x7f34e809bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:06.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34f026f640 10 monclient(hunting): picked mon.noname-a con 0x7f34e8136f20 addr v2:172.21.15.37:3300/0 2024-04-26T00:24:06.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34f026f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f34e8137300 addr v1:172.21.15.45:6789/0 2024-04-26T00:24:06.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34f026f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34e813c450 0x7f34e813e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:06.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34f026f640 10 monclient(hunting): picked mon.noname-b con 0x7f34e813c450 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:24:06.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34f026f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34e80a50f0 con 0x7f34e8137300 2024-04-26T00:24:06.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34f026f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:06.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34f026f640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f34e809b330 con 0x7f34e8136f20 2024-04-26T00:24:06.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34f026f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:06.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34f026f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f34e809b610 con 0x7f34e813c450 2024-04-26T00:24:06.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34f026f640 10 monclient(hunting): _renew_subs 2024-04-26T00:24:06.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34f026f640 10 monclient(hunting): authenticate will time out at 1227.802203s 2024-04-26T00:24:06.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34ee00b640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f34e8136f20 0x7f34e809bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:06.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34ee00b640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f34e8136f20 0x7f34e809bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52780/0 (socket says 172.21.15.37:52780) 2024-04-26T00:24:06.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34ee00b640 1 -- 172.21.15.37:0/1458098676 learned_addr learned my addr 172.21.15.37:0/1458098676 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:24:06.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34ee00b640 10 monclient(hunting): get_auth_request con 0x7f34e8136f20 auth_method 0 2024-04-26T00:24:06.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34ee00b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:06.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34ee00b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:06.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34ee00b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:06.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34ee80c640 1 --2- 172.21.15.37:0/1458098676 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34e813c450 0x7f34e813e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:06.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34ee80c640 10 monclient(hunting): get_auth_request con 0x7f34e813c450 auth_method 0 2024-04-26T00:24:06.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34ee80c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:06.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34ee80c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:06.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34ee80c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:06.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34ee00b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:06.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34ee00b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:06.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34ee00b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:06.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34ee80c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:06.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34ee80c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:06.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34ee80c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:06.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34ee80c640 10 monclient(hunting): handle_auth_done global_id 34334 payload 274 2024-04-26T00:24:06.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34ee80c640 1 -- 172.21.15.37:0/1458098676 >> v2:172.21.15.37:3300/0 conn(0x7f34e8136f20 msgr2=0x7f34e809bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:24:06.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34ee80c640 1 --2- 172.21.15.37:0/1458098676 >> v2:172.21.15.37:3300/0 conn(0x7f34e8136f20 0x7f34e809bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:06.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34ee80c640 1 -- 172.21.15.37:0/1458098676 >> v1:172.21.15.45:6789/0 conn(0x7f34e8137300 legacy=0x7f34e81377b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:24:06.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34ee80c640 10 monclient: _finish_hunting 0 2024-04-26T00:24:06.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34ee80c640 1 monclient: found mon.noname-b 2024-04-26T00:24:06.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34ee80c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:24:06.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34ee80c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:24:06.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34ee80c640 1 -- 172.21.15.37:0/1458098676 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34e809b8f0 con 0x7f34e813c450 2024-04-26T00:24:06.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.493+0000 7f34ee00b640 1 --2- 172.21.15.37:0/1458098676 >> v2:172.21.15.37:3300/0 conn(0x7f34e8136f20 0x7f34e809bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:24:06.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.497+0000 7f34ee80c640 1 -- 172.21.15.37:0/1458098676 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34e813c450 msgr2=0x7f34e813e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:06.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.497+0000 7f34ee80c640 1 --2- 172.21.15.37:0/1458098676 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34e813c450 0x7f34e813e840 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f34e0009d90 tx=0x7f34e001c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ff7a2b3f5d88d11 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:06.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.497+0000 7f34ed009640 1 -- 172.21.15.37:0/1458098676 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e00a6710 con 0x7f34e813c450 2024-04-26T00:24:06.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.497+0000 7f34ed009640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:06.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.497+0000 7f34ed009640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:24:06.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.497+0000 7f34ed009640 10 monclient: dump: 2024-04-26T00:24:06.481 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:06.481 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:06.481 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:06.481 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:06.481 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:06.481 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:06.481 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:06.481 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:06.481 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:06.481 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:06.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.497+0000 7f34ed009640 10 monclient: _finish_auth 0 2024-04-26T00:24:06.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.497+0000 7f34ed009640 10 monclient: _check_auth_tickets 2024-04-26T00:24:06.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.497+0000 7f34ed009640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:24:06.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.497+0000 7f34ed009640 1 -- 172.21.15.37:0/1458098676 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34e00a68b0 con 0x7f34e813c450 2024-04-26T00:24:06.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.497+0000 7f34ed009640 10 monclient: handle_config config(0 keys) 2024-04-26T00:24:06.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.497+0000 7f34ed009640 1 -- 172.21.15.37:0/1458098676 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e00a6c80 con 0x7f34e813c450 2024-04-26T00:24:06.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.497+0000 7f34f026f640 5 monclient: authenticate success, global_id 34334 2024-04-26T00:24:06.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.497+0000 7f34f026f640 10 monclient: get_monmap_and_config success 2024-04-26T00:24:06.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.497+0000 7f34f026f640 4 set_mon_vals no callback set 2024-04-26T00:24:06.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.497+0000 7f34ed009640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:06.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.497+0000 7f34ed009640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:24:06.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.497+0000 7f34ed009640 10 monclient: dump: 2024-04-26T00:24:06.482 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:06.482 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:06.482 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:06.483 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:06.483 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:06.483 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:06.483 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:06.483 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:06.483 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:06.483 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:06.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.497+0000 7f34f026f640 10 monclient: shutdown 2024-04-26T00:24:06.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.497+0000 7f34f026f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:24:06.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34f026f640 1 -- 172.21.15.37:0/1458098676 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34e813c450 msgr2=0x7f34e813e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:06.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34f026f640 1 --2- 172.21.15.37:0/1458098676 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34e813c450 0x7f34e813e840 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f34e0009d90 tx=0x7f34e001c880 comp rx=0 tx=0).stop 2024-04-26T00:24:06.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34f026f640 1 -- 172.21.15.37:0/1458098676 shutdown_connections 2024-04-26T00:24:06.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34f026f640 1 --2- 172.21.15.37:0/1458098676 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34e813c450 0x7f34e813e840 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:06.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34f026f640 1 --2- 172.21.15.37:0/1458098676 >> v2:172.21.15.37:3300/0 conn(0x7f34e8136f20 0x7f34e809bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:06.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34f026f640 1 -- 172.21.15.37:0/1458098676 >> 172.21.15.37:0/1458098676 conn(0x7f34e80a61b0 msgr2=0x7f34e80a65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:24:06.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34f026f640 1 -- 172.21.15.37:0/1458098676 shutdown_connections 2024-04-26T00:24:06.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34f026f640 1 -- 172.21.15.37:0/1458098676 wait complete. 2024-04-26T00:24:06.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34f026f640 10 monclient: build_initial_monmap 2024-04-26T00:24:06.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34f026f640 10 monclient: monmap: 2024-04-26T00:24:06.484 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:24:06.484 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:24:06.484 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:24:06.484 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:24:06.484 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:24:06.484 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:24:06.484 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:24:06.484 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:24:06.484 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:24:06.484 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:06.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34f026f640 1 Processor -- start 2024-04-26T00:24:06.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34f026f640 1 -- start start 2024-04-26T00:24:06.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34f026f640 10 monclient: init 2024-04-26T00:24:06.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34f026f640 10 monclient: _reopen_session rank -1 2024-04-26T00:24:06.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34f026f640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:24:06.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34f026f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34e8136f20 0x7f34e81cebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:06.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34f026f640 10 monclient(hunting): picked mon.noname-a con 0x7f34e8136f20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:24:06.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34f026f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34e8137300 0x7f34e8134da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:06.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34f026f640 10 monclient(hunting): picked mon.noname-c con 0x7f34e8137300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:24:06.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34f026f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34e813c450 0x7f34e8137ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:06.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34f026f640 10 monclient(hunting): picked mon.noname-b con 0x7f34e813c450 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:24:06.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34f026f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:06.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34f026f640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f34e8141930 con 0x7f34e8136f20 2024-04-26T00:24:06.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34f026f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:06.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34f026f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f34e81417b0 con 0x7f34e813c450 2024-04-26T00:24:06.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34f026f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:06.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34f026f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f34e8141ab0 con 0x7f34e8137300 2024-04-26T00:24:06.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34f026f640 10 monclient(hunting): _renew_subs 2024-04-26T00:24:06.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34f026f640 10 monclient(hunting): authenticate will time out at 1227.809652s 2024-04-26T00:24:06.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34ed80a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34e8137300 0x7f34e8134da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:06.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34ee00b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34e8136f20 0x7f34e81cebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:06.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34ee00b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34e8136f20 0x7f34e81cebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52796/0 (socket says 172.21.15.37:52796) 2024-04-26T00:24:06.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34ee00b640 1 -- 172.21.15.37:0/1201458730 learned_addr learned my addr 172.21.15.37:0/1201458730 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:24:06.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34ed80a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34e8137300 0x7f34e8134da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:46080/0 (socket says 172.21.15.37:46080) 2024-04-26T00:24:06.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34ed80a640 10 monclient(hunting): get_auth_request con 0x7f34e8137300 auth_method 0 2024-04-26T00:24:06.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34ed80a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:06.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34ed80a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:06.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34ed80a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:06.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34ee00b640 10 monclient(hunting): get_auth_request con 0x7f34e8136f20 auth_method 0 2024-04-26T00:24:06.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34ee00b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:06.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34ee00b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:06.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34ee00b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:06.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34ee00b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:06.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34ee00b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:06.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34ee00b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:06.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34ed80a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:06.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34ed80a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:06.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34ed80a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:06.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34ee80c640 1 --2- 172.21.15.37:0/1201458730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34e813c450 0x7f34e8137ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:06.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34ee80c640 10 monclient(hunting): get_auth_request con 0x7f34e813c450 auth_method 0 2024-04-26T00:24:06.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34ee80c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:06.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34ee80c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:06.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34ee80c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:06.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34ee00b640 10 monclient(hunting): handle_auth_done global_id 54147 payload 824 2024-04-26T00:24:06.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34ee00b640 1 -- 172.21.15.37:0/1201458730 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34e8137300 msgr2=0x7f34e8134da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:06.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34ee00b640 1 --2- 172.21.15.37:0/1201458730 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34e8137300 0x7f34e8134da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:06.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34ee00b640 1 -- 172.21.15.37:0/1201458730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34e813c450 msgr2=0x7f34e8137ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:06.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34ee00b640 1 --2- 172.21.15.37:0/1201458730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34e813c450 0x7f34e8137ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:06.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34ee00b640 10 monclient: _finish_hunting 0 2024-04-26T00:24:06.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34ee00b640 1 monclient: found mon.noname-a 2024-04-26T00:24:06.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34ee00b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:24:06.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34ee00b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:24:06.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.501+0000 7f34ee00b640 1 -- 172.21.15.37:0/1201458730 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34e813b8e0 con 0x7f34e8136f20 2024-04-26T00:24:06.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34ee00b640 1 -- 172.21.15.37:0/1201458730 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34e8136f20 msgr2=0x7f34e81cebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:06.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34ee00b640 1 --2- 172.21.15.37:0/1201458730 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34e8136f20 0x7f34e81cebe0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f34e4009fd0 tx=0x7f34e400ee10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:06.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34deffd640 1 -- 172.21.15.37:0/1201458730 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e400cc10 con 0x7f34e8136f20 2024-04-26T00:24:06.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:06.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34deffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:24:06.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34deffd640 10 monclient: dump: 2024-04-26T00:24:06.492 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:06.492 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:06.492 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:06.492 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:06.492 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:06.492 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:06.492 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:06.492 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:06.492 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:06.492 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:06.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34deffd640 10 monclient: _finish_auth 0 2024-04-26T00:24:06.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34deffd640 10 monclient: _check_auth_tickets 2024-04-26T00:24:06.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:24:06.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34deffd640 1 -- 172.21.15.37:0/1201458730 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34e400cdb0 con 0x7f34e8136f20 2024-04-26T00:24:06.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34deffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:24:06.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34deffd640 1 -- 172.21.15.37:0/1201458730 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e4004250 con 0x7f34e8136f20 2024-04-26T00:24:06.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:06.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34deffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:24:06.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34deffd640 10 monclient: dump: 2024-04-26T00:24:06.493 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:06.493 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:06.493 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:06.493 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:06.493 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:06.493 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:06.493 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:06.494 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:06.494 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:06.494 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:06.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34f026f640 5 monclient: authenticate success, global_id 54147 2024-04-26T00:24:06.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34f026f640 10 monclient: _renew_subs 2024-04-26T00:24:06.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34f026f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:24:06.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34f026f640 1 -- 172.21.15.37:0/1201458730 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34e813bbc0 con 0x7f34e8136f20 2024-04-26T00:24:06.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34dffff640 4 set_mon_vals no callback set 2024-04-26T00:24:06.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34f026f640 10 monclient: _renew_subs 2024-04-26T00:24:06.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34f026f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:24:06.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34f026f640 1 -- 172.21.15.37:0/1201458730 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34e81391a0 con 0x7f34e8136f20 2024-04-26T00:24:06.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34f026f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:24:06.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34f026f640 1 --2- 172.21.15.37:0/1201458730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34e8138bf0 0x7f34e802c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:06.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34f026f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:06.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34f026f640 1 -- 172.21.15.37:0/1201458730 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f34e804f510 con 0x7f34e8138bf0 2024-04-26T00:24:06.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34deffd640 1 -- 172.21.15.37:0/1201458730 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f34e4003620 con 0x7f34e8136f20 2024-04-26T00:24:06.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34deffd640 1 --2- 172.21.15.37:0/1201458730 >> v2:172.21.15.45:6812/2379918623 conn(0x7f34cc035ed0 0x7f34cc038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:06.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34deffd640 1 -- 172.21.15.37:0/1201458730 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(177..177 src has 1..177) ==== 3931+0+0 (secure 0 0 0) 0x7f34e4048b70 con 0x7f34e8136f20 2024-04-26T00:24:06.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34ed80a640 1 --2- 172.21.15.37:0/1201458730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34e8138bf0 0x7f34e802c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:06.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34ed80a640 10 monclient: get_auth_request con 0x7f34e8138bf0 auth_method 0 2024-04-26T00:24:06.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34ed80a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:06.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34ed80a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:06.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34ed80a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:06.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34ed80a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:06.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34ed80a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:06.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34ed80a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:06.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34ee80c640 1 --2- 172.21.15.37:0/1201458730 >> v2:172.21.15.45:6812/2379918623 conn(0x7f34cc035ed0 0x7f34cc038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:06.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34ee80c640 10 monclient: get_auth_request con 0x7f34cc035ed0 auth_method 0 2024-04-26T00:24:06.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34ed80a640 10 monclient(hunting): handle_auth_done global_id 24385 payload 824 2024-04-26T00:24:06.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.505+0000 7f34ed80a640 1 -- 172.21.15.37:0/1201458730 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f34e81398d0 con 0x7f34e8138bf0 2024-04-26T00:24:06.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.509+0000 7f34ed80a640 1 -- 172.21.15.37:0/1201458730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34e8138bf0 msgr2=0x7f34e802c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:06.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.509+0000 7f34ed80a640 1 --2- 172.21.15.37:0/1201458730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34e8138bf0 0x7f34e802c000 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f34d8007fd0 tx=0x7f34d800fc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:06.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.509+0000 7f34ee80c640 1 --2- 172.21.15.37:0/1201458730 >> v2:172.21.15.45:6812/2379918623 conn(0x7f34cc035ed0 0x7f34cc038390 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f34e00ac000 tx=0x7f34e00a7000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:06.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.509+0000 7f34deffd640 1 -- 172.21.15.37:0/1201458730 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d8003b80 con 0x7f34e8138bf0 2024-04-26T00:24:06.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.509+0000 7f34deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:06.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.509+0000 7f34deffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:24:06.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.509+0000 7f34deffd640 10 monclient: dump: 2024-04-26T00:24:06.498 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:06.498 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:06.498 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:06.498 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:06.498 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:06.498 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:06.498 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:06.498 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:06.498 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:06.498 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:06.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.509+0000 7f34deffd640 1 -- 172.21.15.37:0/1201458730 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f34d8019070 con 0x7f34e8138bf0 2024-04-26T00:24:06.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.509+0000 7f34deffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:24:06.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.509+0000 7f34deffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:24:06.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.509+0000 7f34deffd640 1 -- 172.21.15.37:0/1201458730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34e8138bf0 msgr2=0x7f34e802c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:06.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.509+0000 7f34deffd640 1 --2- 172.21.15.37:0/1201458730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34e8138bf0 0x7f34e802c000 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f34d8007fd0 tx=0x7f34d800fc00 comp rx=0 tx=0).stop 2024-04-26T00:24:06.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.509+0000 7f34deffd640 1 -- 172.21.15.37:0/1201458730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34e8138bf0 msgr2=0x7f34e802c000 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:24:06.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.509+0000 7f34deffd640 1 --2- 172.21.15.37:0/1201458730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34e8138bf0 0x7f34e802c000 secure :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f34d8007fd0 tx=0x7f34d800fc00 comp rx=0 tx=0).stop 2024-04-26T00:24:06.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.517+0000 7f34f026f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:24:06.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.517+0000 7f34f026f640 1 --2- 172.21.15.37:0/1201458730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34e802c540 0x7f34e802c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:06.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.517+0000 7f34f026f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:06.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.517+0000 7f34f026f640 1 -- 172.21.15.37:0/1201458730 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f34e809fa40 con 0x7f34e802c540 2024-04-26T00:24:06.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.517+0000 7f34ed80a640 1 --2- 172.21.15.37:0/1201458730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34e802c540 0x7f34e802c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:06.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.517+0000 7f34ed80a640 10 monclient: get_auth_request con 0x7f34e802c540 auth_method 0 2024-04-26T00:24:06.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.517+0000 7f34ed80a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:06.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.517+0000 7f34ed80a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:06.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.517+0000 7f34ed80a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:06.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.517+0000 7f34ed80a640 1 -- 172.21.15.37:0/1201458730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34e802c540 msgr2=0x7f34e802c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:24:06.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.517+0000 7f34ed80a640 1 -- 172.21.15.37:0/1201458730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34e802c540 msgr2=0x7f34e802c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:24:06.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.517+0000 7f34ed80a640 1 --2- 172.21.15.37:0/1201458730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34e802c540 0x7f34e802c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:24:06.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.517+0000 7f34ed80a640 1 --2- 172.21.15.37:0/1201458730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34e802c540 0x7f34e802c950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:24:06.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.721+0000 7f34ed80a640 1 --2- 172.21.15.37:0/1201458730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34e802c540 0x7f34e802c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:06.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.721+0000 7f34ed80a640 10 monclient: get_auth_request con 0x7f34e802c540 auth_method 0 2024-04-26T00:24:06.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.721+0000 7f34ed80a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:06.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.721+0000 7f34ed80a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:06.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.721+0000 7f34ed80a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:24:06.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.721+0000 7f34ed80a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:06.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.721+0000 7f34ed80a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:06.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.721+0000 7f34ed80a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:06.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.721+0000 7f34ed80a640 10 monclient(hunting): handle_auth_done global_id 24388 payload 824 2024-04-26T00:24:06.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.721+0000 7f34ed80a640 1 -- 172.21.15.37:0/1201458730 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f34e8075f10 con 0x7f34e802c540 2024-04-26T00:24:06.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.725+0000 7f34ed80a640 1 -- 172.21.15.37:0/1201458730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34e802c540 msgr2=0x7f34e802c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:06.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.725+0000 7f34ed80a640 1 --2- 172.21.15.37:0/1201458730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34e802c540 0x7f34e802c950 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f34d8007fd0 tx=0x7f34d8003770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:06.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.725+0000 7f34deffd640 1 -- 172.21.15.37:0/1201458730 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d8006c00 con 0x7f34e802c540 2024-04-26T00:24:06.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.725+0000 7f34deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:06.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.725+0000 7f34deffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:24:06.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.725+0000 7f34deffd640 10 monclient: dump: 2024-04-26T00:24:06.703 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:06.703 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:06.703 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:06.703 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:06.703 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:06.703 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:06.703 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:06.703 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:06.703 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:06.703 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:06.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.725+0000 7f34deffd640 1 -- 172.21.15.37:0/1201458730 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f34d8006c00 con 0x7f34e802c540 2024-04-26T00:24:06.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.725+0000 7f34deffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:24:06.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.725+0000 7f34deffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:24:06.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.725+0000 7f34deffd640 1 -- 172.21.15.37:0/1201458730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34e802c540 msgr2=0x7f34e802c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:06.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.725+0000 7f34deffd640 1 --2- 172.21.15.37:0/1201458730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34e802c540 0x7f34e802c950 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f34d8007fd0 tx=0x7f34d8003770 comp rx=0 tx=0).stop 2024-04-26T00:24:06.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.725+0000 7f34deffd640 1 -- 172.21.15.37:0/1201458730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34e802c540 msgr2=0x7f34e802c950 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:24:06.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.725+0000 7f34deffd640 1 --2- 172.21.15.37:0/1201458730 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34e802c540 0x7f34e802c950 secure :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f34d8007fd0 tx=0x7f34d8003770 comp rx=0 tx=0).stop 2024-04-26T00:24:06.705 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:24:06.705 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:24:06.705 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:24:06.705 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:24:06.705 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 34, 2024-04-26T00:24:06.705 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 154892, 2024-04-26T00:24:06.705 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:24:06.705 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:24:06.706 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:24:06.706 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:24:06.706 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:06.706 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 3, 2024-04-26T00:24:06.706 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:24:06.706 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:24:06.706 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:24:06.706 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:24:06.706 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:24:06.706 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:24:06.706 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:24:06.706 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:24:06.706 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:24:06.706 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:24:06.706 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:24:06.707 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:24:06.707 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:24:06.707 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:24:06.707 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:06.707 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:24:06.707 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:24:06.707 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:24:06.707 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:24:06.707 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:24:06.707 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:24:06.707 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:24:06.707 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:24:06.707 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:24:06.707 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:24:06.707 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:24:06.708 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:24:06.708 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:24:06.708 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:06.708 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:06.708 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:24:06.708 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:24:06.708 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:24:06.708 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:24:06.708 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:24:06.708 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:24:06.708 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:24:06.708 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:24:06.709 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:24:06.709 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:24:06.709 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:24:06.709 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:24:06.709 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:24:06.709 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:24:06.709 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:24:06.709 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:24:06.709 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:24:06.709 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:24:06.709 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:24:06.709 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:24:06.710 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:24:06.710 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:24:06.710 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:24:06.710 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:24:06.710 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:24:06.710 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:24:06.710 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:24:06.710 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:24:06.710 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:06.710 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:24:06.710 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:06.710 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:24:06.710 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:06.710 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:24:06.710 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:24:06.711 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:24:06.711 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:24:06.711 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:06.711 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:24:06.711 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:24:06.711 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:06.711 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:06.711 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:06.711 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:24:06.711 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:24:06.711 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:06.711 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:06.711 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:06.711 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:06.711 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:24:06.711 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:24:06.712 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:24:06.712 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:24:06.712 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:24:06.712 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:06.712 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:06.712 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:24:06.712 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:24:06.712 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:24:06.712 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:24:06.712 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:06.712 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:24:06.712 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:24:06.712 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:06.712 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:06.712 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:06.712 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:24:06.712 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:24:06.713 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:06.713 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:06.713 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:06.713 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:06.713 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:24:06.713 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:24:06.713 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:24:06.713 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:24:06.713 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:24:06.713 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:06.713 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:06.713 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:24:06.713 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:24:06.713 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:24:06.713 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:24:06.714 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:06.714 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:24:06.714 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:24:06.714 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:06.714 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:06.714 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:06.714 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:24:06.714 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:24:06.714 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:06.714 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:06.714 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:06.714 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:06.714 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:24:06.714 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:24:06.714 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:24:06.715 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:24:06.715 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:24:06.715 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:06.715 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:06.715 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:06.715 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:24:06.715 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:24:06.715 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:06.715 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:24:06.715 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:24:06.715 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:24:06.715 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:06.715 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:06.715 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:24:06.715 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:06.715 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:24:06.716 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:24:06.716 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:24:06.716 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:06.716 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:06.716 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:24:06.716 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:06.716 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:24:06.716 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:24:06.716 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:24:06.716 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:06.716 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:06.716 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:24:06.716 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:06.716 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:24:06.717 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:24:06.717 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:24:06.717 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:06.717 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:06.717 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:06.717 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:24:06.717 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:24:06.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.725+0000 7f34f026f640 1 -- 172.21.15.37:0/1201458730 >> v2:172.21.15.45:6812/2379918623 conn(0x7f34cc035ed0 msgr2=0x7f34cc038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:06.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.725+0000 7f34f026f640 1 --2- 172.21.15.37:0/1201458730 >> v2:172.21.15.45:6812/2379918623 conn(0x7f34cc035ed0 0x7f34cc038390 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f34e00ac000 tx=0x7f34e00a7000 comp rx=0 tx=0).stop 2024-04-26T00:24:06.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.725+0000 7f34f026f640 10 monclient: shutdown 2024-04-26T00:24:06.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.725+0000 7f34f026f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:24:06.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.725+0000 7f34f026f640 1 -- 172.21.15.37:0/1201458730 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34e8136f20 msgr2=0x7f34e81cebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:06.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.725+0000 7f34f026f640 1 --2- 172.21.15.37:0/1201458730 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34e8136f20 0x7f34e81cebe0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f34e4009fd0 tx=0x7f34e400ee10 comp rx=0 tx=0).stop 2024-04-26T00:24:06.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.725+0000 7f34ee80c640 1 -- 172.21.15.37:0/1201458730 reap_dead start 2024-04-26T00:24:06.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.725+0000 7f34f026f640 1 -- 172.21.15.37:0/1201458730 shutdown_connections 2024-04-26T00:24:06.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.725+0000 7f34f026f640 1 --2- 172.21.15.37:0/1201458730 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34e8136f20 0x7f34e81cebe0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:06.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.725+0000 7f34f026f640 1 -- 172.21.15.37:0/1201458730 >> 172.21.15.37:0/1201458730 conn(0x7f34e80a61b0 msgr2=0x7f34e80a8260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:24:06.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.725+0000 7f34f026f640 1 -- 172.21.15.37:0/1201458730 shutdown_connections 2024-04-26T00:24:06.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:06.725+0000 7f34f026f640 1 -- 172.21.15.37:0/1201458730 wait complete. 2024-04-26T00:24:06.728 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:24:06.729 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-04-26T00:24:06.729 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2024-04-26T00:24:06.729 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-04-26T00:24:06.729 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-04-26T00:24:06.729 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-26T00:24:06.729 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T00:24:06.754 INFO:tasks.ceph.mon.c:Stopped 2024-04-26T00:24:06.754 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-04-26T00:24:06.755 INFO:tasks.ceph.mon.a:Sent signal 19 2024-04-26T00:24:06.755 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2024-04-26T00:24:26.756 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-04-26T00:24:26.757 INFO:tasks.ceph.mon.a:Sent signal 18 2024-04-26T00:24:26.757 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-26T00:24:26.757 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:24:26.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:26.913+0000 7fe50b28e640 10 monclient: get_monmap_and_config 2024-04-26T00:24:26.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:26.913+0000 7fe50b28e640 10 monclient: build_initial_monmap 2024-04-26T00:24:26.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:26.913+0000 7fe50b28e640 10 monclient: monmap: 2024-04-26T00:24:26.893 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:24:26.893 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:26.893 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:24:26.917445+0000 2024-04-26T00:24:26.893 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:24:26.917445+0000 2024-04-26T00:24:26.893 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:24:26.893 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:26.893 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:24:26.893 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:24:26.893 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:24:26.893 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:24:26.893 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:24:26.893 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:26.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:26.913+0000 7fe50b28e640 1 Processor -- start 2024-04-26T00:24:26.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:26.913+0000 7fe50b28e640 1 -- start start 2024-04-26T00:24:26.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:26.913+0000 7fe50b28e640 10 monclient: init 2024-04-26T00:24:26.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:26.913+0000 7fe50b28e640 10 monclient: _reopen_session rank -1 2024-04-26T00:24:26.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:26.913+0000 7fe50b28e640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-04-26T00:24:26.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:26.913+0000 7fe50b28e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe5040a0260 0x7fe5040a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:26.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:26.913+0000 7fe50b28e640 10 monclient(hunting): picked mon.noname-b con 0x7fe5040a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:24:26.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:26.913+0000 7fe50b28e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe504134b30 addr v1:172.21.15.37:6789/0 2024-04-26T00:24:26.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:26.913+0000 7fe50b28e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe504135660 0x7fe50413ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:26.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:26.913+0000 7fe50b28e640 10 monclient(hunting): picked mon.noname-c con 0x7fe504135660 addr v2:172.21.15.45:3300/0 2024-04-26T00:24:26.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:26.913+0000 7fe50b28e640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5040a1130 con 0x7fe504134b30 2024-04-26T00:24:26.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:26.913+0000 7fe50b28e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:26.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:26.913+0000 7fe50b28e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fe50409ec10 con 0x7fe504135660 2024-04-26T00:24:26.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:26.913+0000 7fe50b28e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:26.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:26.913+0000 7fe50b28e640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe50409f0f0 con 0x7fe5040a0260 2024-04-26T00:24:26.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:26.913+0000 7fe50b28e640 10 monclient(hunting): _renew_subs 2024-04-26T00:24:26.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:26.913+0000 7fe50b28e640 10 monclient(hunting): authenticate will time out at 1248.223072s 2024-04-26T00:24:26.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:26.913+0000 7fe50902a640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe5040a0260 msgr2=0x7fe5040a0640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:24:26.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:26.913+0000 7fe50902a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe5040a0260 0x7fe5040a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:24:26.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:26.913+0000 7fe508829640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fe504134b30 0x7fe504134f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:55002/0 (socket says 172.21.15.37:55002) 2024-04-26T00:24:26.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:26.913+0000 7fe508829640 1 -- 172.21.15.37:0/1795356746 learned_addr learned my addr 172.21.15.37:0/1795356746 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:24:26.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:26.913+0000 7fe50982b640 1 --2- 172.21.15.37:0/1795356746 >> v2:172.21.15.45:3300/0 conn(0x7fe504135660 0x7fe50413ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:26.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:26.913+0000 7fe50982b640 10 monclient(hunting): get_auth_request con 0x7fe504135660 auth_method 0 2024-04-26T00:24:26.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:26.913+0000 7fe50982b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:26.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:26.913+0000 7fe50982b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:26.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:26.913+0000 7fe50982b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:26.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:26.913+0000 7fe50982b640 1 -- 172.21.15.37:0/1795356746 >> v2:172.21.15.45:3300/0 conn(0x7fe504135660 msgr2=0x7fe50413ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:24:26.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:26.913+0000 7fe50982b640 1 -- 172.21.15.37:0/1795356746 >> v2:172.21.15.45:3300/0 conn(0x7fe504135660 msgr2=0x7fe50413ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:24:26.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:26.913+0000 7fe50982b640 1 --2- 172.21.15.37:0/1795356746 >> v2:172.21.15.45:3300/0 conn(0x7fe504135660 0x7fe50413ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:24:26.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:26.913+0000 7fe50982b640 1 --2- 172.21.15.37:0/1795356746 >> v2:172.21.15.45:3300/0 conn(0x7fe504135660 0x7fe50413ce70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:24:26.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:26.917+0000 7fe508829640 1 -- 172.21.15.37:0/1795356746 >> v1:172.21.15.37:6789/0 conn(0x7fe504134b30 legacy=0x7fe504134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:27.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:27.117+0000 7fe50902a640 1 -- 172.21.15.37:0/1795356746 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe5040a0260 msgr2=0x7fe5040a0640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:24:27.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:27.117+0000 7fe50902a640 1 --2- 172.21.15.37:0/1795356746 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe5040a0260 0x7fe5040a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:24:27.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:27.117+0000 7fe50982b640 1 --2- 172.21.15.37:0/1795356746 >> v2:172.21.15.45:3300/0 conn(0x7fe504135660 0x7fe50413ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:27.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:27.117+0000 7fe50982b640 10 monclient(hunting): get_auth_request con 0x7fe504135660 auth_method 0 2024-04-26T00:24:27.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:27.117+0000 7fe50982b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:27.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:27.117+0000 7fe50982b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:27.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:27.117+0000 7fe50982b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:24:27.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:27.117+0000 7fe50982b640 1 -- 172.21.15.37:0/1795356746 >> v2:172.21.15.45:3300/0 conn(0x7fe504135660 msgr2=0x7fe50413ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:24:27.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:27.117+0000 7fe50982b640 1 -- 172.21.15.37:0/1795356746 >> v2:172.21.15.45:3300/0 conn(0x7fe504135660 msgr2=0x7fe50413ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:24:27.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:27.117+0000 7fe50982b640 1 --2- 172.21.15.37:0/1795356746 >> v2:172.21.15.45:3300/0 conn(0x7fe504135660 0x7fe50413ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:24:27.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:27.117+0000 7fe50982b640 1 --2- 172.21.15.37:0/1795356746 >> v2:172.21.15.45:3300/0 conn(0x7fe504135660 0x7fe50413ce70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:24:27.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:27.521+0000 7fe50902a640 1 -- 172.21.15.37:0/1795356746 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe5040a0260 msgr2=0x7fe5040a0640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:24:27.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:27.521+0000 7fe50902a640 1 --2- 172.21.15.37:0/1795356746 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe5040a0260 0x7fe5040a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:24:27.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:27.521+0000 7fe50982b640 1 --2- 172.21.15.37:0/1795356746 >> v2:172.21.15.45:3300/0 conn(0x7fe504135660 0x7fe50413ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:27.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:27.521+0000 7fe50982b640 10 monclient(hunting): get_auth_request con 0x7fe504135660 auth_method 0 2024-04-26T00:24:27.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:27.521+0000 7fe50982b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:27.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:27.521+0000 7fe50982b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:27.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:27.521+0000 7fe50982b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:24:27.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:27.521+0000 7fe50982b640 1 -- 172.21.15.37:0/1795356746 >> v2:172.21.15.45:3300/0 conn(0x7fe504135660 msgr2=0x7fe50413ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:24:27.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:27.521+0000 7fe50982b640 1 -- 172.21.15.37:0/1795356746 >> v2:172.21.15.45:3300/0 conn(0x7fe504135660 msgr2=0x7fe50413ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:24:27.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:27.521+0000 7fe50982b640 1 --2- 172.21.15.37:0/1795356746 >> v2:172.21.15.45:3300/0 conn(0x7fe504135660 0x7fe50413ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:24:27.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:27.521+0000 7fe50982b640 1 --2- 172.21.15.37:0/1795356746 >> v2:172.21.15.45:3300/0 conn(0x7fe504135660 0x7fe50413ce70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:24:28.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:28.325+0000 7fe50902a640 1 -- 172.21.15.37:0/1795356746 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe5040a0260 msgr2=0x7fe5040a0640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:24:28.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:28.325+0000 7fe50902a640 1 --2- 172.21.15.37:0/1795356746 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe5040a0260 0x7fe5040a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:24:28.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:28.325+0000 7fe50982b640 1 --2- 172.21.15.37:0/1795356746 >> v2:172.21.15.45:3300/0 conn(0x7fe504135660 0x7fe50413ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:28.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:28.325+0000 7fe50982b640 10 monclient(hunting): get_auth_request con 0x7fe504135660 auth_method 0 2024-04-26T00:24:28.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:28.325+0000 7fe50982b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:28.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:28.325+0000 7fe50982b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:28.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:28.325+0000 7fe50982b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:24:28.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:28.325+0000 7fe50982b640 1 -- 172.21.15.37:0/1795356746 >> v2:172.21.15.45:3300/0 conn(0x7fe504135660 msgr2=0x7fe50413ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:24:28.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:28.325+0000 7fe50982b640 1 -- 172.21.15.37:0/1795356746 >> v2:172.21.15.45:3300/0 conn(0x7fe504135660 msgr2=0x7fe50413ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:24:28.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:28.325+0000 7fe50982b640 1 --2- 172.21.15.37:0/1795356746 >> v2:172.21.15.45:3300/0 conn(0x7fe504135660 0x7fe50413ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:24:28.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:28.325+0000 7fe50982b640 1 --2- 172.21.15.37:0/1795356746 >> v2:172.21.15.45:3300/0 conn(0x7fe504135660 0x7fe50413ce70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:24:29.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:29.917+0000 7fe4faffd640 10 monclient(hunting): tick 2024-04-26T00:24:29.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:29.917+0000 7fe4faffd640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:24:29.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:29.917+0000 7fe4faffd640 1 monclient(hunting): continuing hunt 2024-04-26T00:24:29.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:29.917+0000 7fe4faffd640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:24:29.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:29.917+0000 7fe4faffd640 1 -- 172.21.15.37:0/1795356746 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe5040a0260 msgr2=0x7fe5040a0640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:24:29.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:29.917+0000 7fe4faffd640 1 --2- 172.21.15.37:0/1795356746 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe5040a0260 0x7fe5040a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:29.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:29.917+0000 7fe4faffd640 1 -- 172.21.15.37:0/1795356746 >> v2:172.21.15.45:3300/0 conn(0x7fe504135660 msgr2=0x7fe50413ce70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:24:29.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:29.917+0000 7fe4faffd640 1 --2- 172.21.15.37:0/1795356746 >> v2:172.21.15.45:3300/0 conn(0x7fe504135660 0x7fe50413ce70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:29.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:29.917+0000 7fe4faffd640 1 -- 172.21.15.37:0/1795356746 >> v1:172.21.15.37:6789/0 conn(0x7fe504134b30 legacy=0x7fe504134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:29.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:29.917+0000 7fe4faffd640 10 monclient: _add_conns ranks=[3,1] 2024-04-26T00:24:29.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:29.917+0000 7fe4faffd640 1 --2- 172.21.15.37:0/1795356746 >> v2:172.21.15.37:3300/0 conn(0x7fe4e8003770 0x7fe4e8005bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:29.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:29.917+0000 7fe4faffd640 10 monclient(hunting): picked mon.noname-a con 0x7fe4e8003770 addr v2:172.21.15.37:3300/0 2024-04-26T00:24:29.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:29.917+0000 7fe4faffd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe4e80062c0 addr v1:172.21.15.45:6789/0 2024-04-26T00:24:29.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:29.917+0000 7fe4faffd640 1 -- 172.21.15.37:0/1795356746 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe50406cff0 con 0x7fe4e80062c0 2024-04-26T00:24:29.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:29.917+0000 7fe4faffd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:29.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:29.917+0000 7fe4faffd640 1 -- 172.21.15.37:0/1795356746 --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fe50409f0f0 con 0x7fe4e8003770 2024-04-26T00:24:29.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:29.917+0000 7fe4faffd640 10 monclient(hunting): _renew_subs 2024-04-26T00:24:29.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:29.917+0000 7fe50902a640 1 --2- 172.21.15.37:0/1795356746 >> v2:172.21.15.37:3300/0 conn(0x7fe4e8003770 0x7fe4e8005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:29.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:29.917+0000 7fe50902a640 10 monclient(hunting): get_auth_request con 0x7fe4e8003770 auth_method 0 2024-04-26T00:24:29.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:29.917+0000 7fe50902a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:29.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:29.917+0000 7fe50902a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:29.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:29.917+0000 7fe50902a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:29.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:29.917+0000 7fe50902a640 1 -- 172.21.15.37:0/1795356746 >> v2:172.21.15.37:3300/0 conn(0x7fe4e8003770 msgr2=0x7fe4e8005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:24:29.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:29.917+0000 7fe50902a640 1 -- 172.21.15.37:0/1795356746 >> v2:172.21.15.37:3300/0 conn(0x7fe4e8003770 msgr2=0x7fe4e8005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:24:29.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:29.917+0000 7fe50902a640 1 --2- 172.21.15.37:0/1795356746 >> v2:172.21.15.37:3300/0 conn(0x7fe4e8003770 0x7fe4e8005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:24:29.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:29.917+0000 7fe50902a640 1 --2- 172.21.15.37:0/1795356746 >> v2:172.21.15.37:3300/0 conn(0x7fe4e8003770 0x7fe4e8005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:24:29.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:29.917+0000 7fe508829640 1 -- 172.21.15.37:0/1795356746 >> v1:172.21.15.45:6789/0 conn(0x7fe4e80062c0 legacy=0x7fe4e8008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:30.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:30.121+0000 7fe50902a640 1 --2- 172.21.15.37:0/1795356746 >> v2:172.21.15.37:3300/0 conn(0x7fe4e8003770 0x7fe4e8005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:30.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:30.121+0000 7fe50902a640 10 monclient(hunting): get_auth_request con 0x7fe4e8003770 auth_method 0 2024-04-26T00:24:30.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:30.121+0000 7fe50902a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:30.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:30.121+0000 7fe50902a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:30.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:30.121+0000 7fe50902a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:24:30.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:30.121+0000 7fe50902a640 1 -- 172.21.15.37:0/1795356746 >> v2:172.21.15.37:3300/0 conn(0x7fe4e8003770 msgr2=0x7fe4e8005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:24:30.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:30.121+0000 7fe50902a640 1 -- 172.21.15.37:0/1795356746 >> v2:172.21.15.37:3300/0 conn(0x7fe4e8003770 msgr2=0x7fe4e8005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:24:30.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:30.121+0000 7fe50902a640 1 --2- 172.21.15.37:0/1795356746 >> v2:172.21.15.37:3300/0 conn(0x7fe4e8003770 0x7fe4e8005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:24:30.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:30.121+0000 7fe50902a640 1 --2- 172.21.15.37:0/1795356746 >> v2:172.21.15.37:3300/0 conn(0x7fe4e8003770 0x7fe4e8005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:24:30.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:30.525+0000 7fe50902a640 1 --2- 172.21.15.37:0/1795356746 >> v2:172.21.15.37:3300/0 conn(0x7fe4e8003770 0x7fe4e8005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:30.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:30.525+0000 7fe50902a640 10 monclient(hunting): get_auth_request con 0x7fe4e8003770 auth_method 0 2024-04-26T00:24:30.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:30.525+0000 7fe50902a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:30.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:30.525+0000 7fe50902a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:30.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:30.525+0000 7fe50902a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:24:30.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:30.525+0000 7fe50902a640 1 -- 172.21.15.37:0/1795356746 >> v2:172.21.15.37:3300/0 conn(0x7fe4e8003770 msgr2=0x7fe4e8005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:24:30.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:30.525+0000 7fe50902a640 1 -- 172.21.15.37:0/1795356746 >> v2:172.21.15.37:3300/0 conn(0x7fe4e8003770 msgr2=0x7fe4e8005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:24:30.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:30.525+0000 7fe50902a640 1 --2- 172.21.15.37:0/1795356746 >> v2:172.21.15.37:3300/0 conn(0x7fe4e8003770 0x7fe4e8005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:24:30.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:30.525+0000 7fe50902a640 1 --2- 172.21.15.37:0/1795356746 >> v2:172.21.15.37:3300/0 conn(0x7fe4e8003770 0x7fe4e8005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:24:31.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.329+0000 7fe50902a640 1 --2- 172.21.15.37:0/1795356746 >> v2:172.21.15.37:3300/0 conn(0x7fe4e8003770 0x7fe4e8005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:31.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.329+0000 7fe50902a640 10 monclient(hunting): get_auth_request con 0x7fe4e8003770 auth_method 0 2024-04-26T00:24:31.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.329+0000 7fe50902a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:31.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.329+0000 7fe50902a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:31.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.329+0000 7fe50902a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:24:31.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.329+0000 7fe50902a640 1 -- 172.21.15.37:0/1795356746 >> v2:172.21.15.37:3300/0 conn(0x7fe4e8003770 msgr2=0x7fe4e8005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:24:31.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.329+0000 7fe50902a640 1 -- 172.21.15.37:0/1795356746 >> v2:172.21.15.37:3300/0 conn(0x7fe4e8003770 msgr2=0x7fe4e8005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:24:31.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.329+0000 7fe50902a640 1 --2- 172.21.15.37:0/1795356746 >> v2:172.21.15.37:3300/0 conn(0x7fe4e8003770 0x7fe4e8005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:24:31.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.329+0000 7fe50902a640 1 --2- 172.21.15.37:0/1795356746 >> v2:172.21.15.37:3300/0 conn(0x7fe4e8003770 0x7fe4e8005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:24:31.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.837+0000 7fe4fbfff640 1 -- 172.21.15.37:0/1795356746 <== mon.1 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3213718013 0 0) 0x7fe50406cff0 con 0x7fe4e80062c0 2024-04-26T00:24:31.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.837+0000 7fe4fbfff640 10 monclient(hunting): _negotiate 2024-04-26T00:24:31.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.837+0000 7fe4fbfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:31.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.837+0000 7fe4fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:31.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.837+0000 7fe4fbfff640 10 monclient(hunting): my global_id is 24391 2024-04-26T00:24:31.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.837+0000 7fe4fbfff640 1 -- 172.21.15.37:0/1795356746 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4f0003530 con 0x7fe4e80062c0 2024-04-26T00:24:31.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe4fbfff640 1 -- 172.21.15.37:0/1795356746 <== mon.1 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3674017872 0 0) 0x7fe4f0003530 con 0x7fe4e80062c0 2024-04-26T00:24:31.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe4fbfff640 1 -- 172.21.15.37:0/1795356746 >> v2:172.21.15.37:3300/0 conn(0x7fe4e8003770 msgr2=0x7fe4e8005bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:24:31.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe4fbfff640 1 --2- 172.21.15.37:0/1795356746 >> v2:172.21.15.37:3300/0 conn(0x7fe4e8003770 0x7fe4e8005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:31.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe4fbfff640 10 monclient: _finish_hunting 0 2024-04-26T00:24:31.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe4fbfff640 1 monclient: found mon.noname-c-legacy 2024-04-26T00:24:31.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe4fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:24:31.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe4fbfff640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.45:6789/0 2024-04-26T00:24:31.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe4fbfff640 1 -- 172.21.15.37:0/1795356746 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4e800b050 con 0x7fe4e80062c0 2024-04-26T00:24:31.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe4fbfff640 10 monclient: _finish_auth 0 2024-04-26T00:24:31.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe4fbfff640 10 monclient: _check_auth_tickets 2024-04-26T00:24:31.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe4fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:24:31.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe4fbfff640 1 -- 172.21.15.37:0/1795356746 <== mon.1 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 866326386 0 0) 0x7fe5000018f0 con 0x7fe4e80062c0 2024-04-26T00:24:31.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe4fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:31.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe4fbfff640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-04-26T00:24:31.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe4fbfff640 10 monclient: dump: 2024-04-26T00:24:31.822 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:31.822 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:31.822 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:31.822 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:31.822 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:31.822 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:31.822 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:31.822 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:31.822 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:31.823 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:31.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe4fbfff640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] but i'm connected to v1:172.21.15.45:6789/0, reconnecting 2024-04-26T00:24:31.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe4fbfff640 10 monclient: _reopen_session rank -1 2024-04-26T00:24:31.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe4fbfff640 1 -- 172.21.15.37:0/1795356746 >> v1:172.21.15.45:6789/0 conn(0x7fe4e80062c0 legacy=0x7fe4e8008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:31.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe4fbfff640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:24:31.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe50982b640 1 -- 172.21.15.37:0/1795356746 reap_dead start 2024-04-26T00:24:31.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe4fbfff640 1 --2- 172.21.15.37:0/1795356746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4f00039c0 0x7fe4f0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:31.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe4fbfff640 10 monclient(hunting): picked mon.b con 0x7fe4f00039c0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:24:31.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe4fbfff640 1 --2- 172.21.15.37:0/1795356746 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe4f0004810 0x7fe4f0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:31.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe4fbfff640 10 monclient(hunting): picked mon.c con 0x7fe4f0004810 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:24:31.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe4fbfff640 1 --2- 172.21.15.37:0/1795356746 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4f0005220 0x7fe4f000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:31.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe4fbfff640 10 monclient(hunting): picked mon.a con 0x7fe4f0005220 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:24:31.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe4fbfff640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:31.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe4fbfff640 1 -- 172.21.15.37:0/1795356746 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe50409f0f0 con 0x7fe4f0005220 2024-04-26T00:24:31.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe4fbfff640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:31.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe4fbfff640 1 -- 172.21.15.37:0/1795356746 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe4f000c150 con 0x7fe4f00039c0 2024-04-26T00:24:31.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe4fbfff640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:31.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe4fbfff640 1 -- 172.21.15.37:0/1795356746 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe4f000c350 con 0x7fe4f0004810 2024-04-26T00:24:31.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe4fbfff640 10 monclient(hunting): _renew_subs 2024-04-26T00:24:31.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe4fbfff640 10 monclient(hunting): _finish_auth 0 2024-04-26T00:24:31.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe508829640 1 -- 172.21.15.37:0/1795356746 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe4f0004810 msgr2=0x7fe4f0004c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:24:31.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe508829640 1 --2- 172.21.15.37:0/1795356746 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe4f0004810 0x7fe4f0004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:24:31.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe50982b640 1 --2- 172.21.15.37:0/1795356746 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4f0005220 0x7fe4f000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:31.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe50982b640 10 monclient(hunting): get_auth_request con 0x7fe4f0005220 auth_method 0 2024-04-26T00:24:31.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe50982b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:31.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe50982b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:31.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe50982b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:24:31.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe50902a640 1 --2- 172.21.15.37:0/1795356746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4f00039c0 0x7fe4f0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:31.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe50902a640 10 monclient(hunting): get_auth_request con 0x7fe4f00039c0 auth_method 0 2024-04-26T00:24:31.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe50902a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:31.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe50902a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:31.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe50902a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:24:31.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe50982b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:31.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe50982b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:31.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe50982b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:24:31.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe50902a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:31.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe50902a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:31.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe50902a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:24:31.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe50982b640 10 monclient(hunting): handle_auth_done global_id 24391 payload 293 2024-04-26T00:24:31.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe50982b640 1 -- 172.21.15.37:0/1795356746 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe4f0004810 msgr2=0x7fe4f0004c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:24:31.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe50982b640 1 --2- 172.21.15.37:0/1795356746 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe4f0004810 0x7fe4f0004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:31.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe50982b640 1 -- 172.21.15.37:0/1795356746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4f00039c0 msgr2=0x7fe4f0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:24:31.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe50982b640 1 --2- 172.21.15.37:0/1795356746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4f00039c0 0x7fe4f0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:31.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe50982b640 10 monclient: _finish_hunting 0 2024-04-26T00:24:31.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe50982b640 1 monclient: found mon.a 2024-04-26T00:24:31.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe50982b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:24:31.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe50982b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:24:31.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe50982b640 1 -- 172.21.15.37:0/1795356746 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4f0003530 con 0x7fe4f0005220 2024-04-26T00:24:31.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe50982b640 10 monclient: _finish_auth 0 2024-04-26T00:24:31.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe50982b640 10 monclient: _check_auth_tickets 2024-04-26T00:24:31.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe50982b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:24:31.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe50b28e640 5 monclient: authenticate success, global_id 24391 2024-04-26T00:24:31.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.841+0000 7fe50b28e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:24:31.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.845+0000 7fe50982b640 1 -- 172.21.15.37:0/1795356746 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4f0005220 msgr2=0x7fe4f000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:31.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.845+0000 7fe50982b640 1 --2- 172.21.15.37:0/1795356746 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4f0005220 0x7fe4f000b9d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe5040a1a60 tx=0x7fe4fc01d170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ff211cabb157f1b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:31.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.845+0000 7fe4fbfff640 1 -- 172.21.15.37:0/1795356746 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4fc0b3070 con 0x7fe4f0005220 2024-04-26T00:24:31.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.845+0000 7fe4fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:31.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.845+0000 7fe4fbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:24:31.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.845+0000 7fe4fbfff640 10 monclient: dump: 2024-04-26T00:24:31.828 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:31.828 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:31.828 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:31.828 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:31.828 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:31.828 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:31.828 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:31.828 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:31.829 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:31.829 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:31.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.845+0000 7fe4fbfff640 1 -- 172.21.15.37:0/1795356746 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4fc0ae410 con 0x7fe4f0005220 2024-04-26T00:24:31.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.845+0000 7fe50b28e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:24:31.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.845+0000 7fe4fbfff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:24:31.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.845+0000 7fe50b28e640 10 monclient: get_monmap_and_config success 2024-04-26T00:24:31.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.845+0000 7fe50b28e640 4 set_mon_vals no callback set 2024-04-26T00:24:31.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.845+0000 7fe4fbfff640 1 -- 172.21.15.37:0/1795356746 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4fc003e60 con 0x7fe4f0005220 2024-04-26T00:24:31.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.845+0000 7fe4fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:31.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.845+0000 7fe4fbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:24:31.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.845+0000 7fe4fbfff640 10 monclient: dump: 2024-04-26T00:24:31.829 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:31.829 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:31.829 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:31.829 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:31.830 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:31.830 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:31.830 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:31.830 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:31.830 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:31.830 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:31.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50b28e640 10 monclient: shutdown 2024-04-26T00:24:31.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50b28e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:24:31.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50b28e640 1 -- 172.21.15.37:0/1795356746 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4f0005220 msgr2=0x7fe4f000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:31.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50b28e640 1 --2- 172.21.15.37:0/1795356746 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4f0005220 0x7fe4f000b9d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe5040a1a60 tx=0x7fe4fc01d170 comp rx=0 tx=0).stop 2024-04-26T00:24:31.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50b28e640 1 -- 172.21.15.37:0/1795356746 shutdown_connections 2024-04-26T00:24:31.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50b28e640 1 --2- 172.21.15.37:0/1795356746 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4f0005220 0x7fe4f000b9d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:31.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50b28e640 1 --2- 172.21.15.37:0/1795356746 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe4f0004810 0x7fe4f0004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:31.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50b28e640 1 --2- 172.21.15.37:0/1795356746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4f00039c0 0x7fe4f0003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:31.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50b28e640 1 -- 172.21.15.37:0/1795356746 >> 172.21.15.37:0/1795356746 conn(0x7fe5040a1ad0 msgr2=0x7fe5040a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:24:31.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50b28e640 1 -- 172.21.15.37:0/1795356746 shutdown_connections 2024-04-26T00:24:31.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50b28e640 1 -- 172.21.15.37:0/1795356746 wait complete. 2024-04-26T00:24:31.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50b28e640 10 monclient: build_initial_monmap 2024-04-26T00:24:31.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50b28e640 10 monclient: monmap: 2024-04-26T00:24:31.833 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:24:31.833 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:24:31.833 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:24:31.833 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:24:31.834 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:24:31.834 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:24:31.834 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:24:31.834 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:24:31.834 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:24:31.834 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:31.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50b28e640 1 Processor -- start 2024-04-26T00:24:31.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50b28e640 1 -- start start 2024-04-26T00:24:31.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50b28e640 10 monclient: init 2024-04-26T00:24:31.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50b28e640 10 monclient: _reopen_session rank -1 2024-04-26T00:24:31.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50b28e640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:24:31.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50b28e640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4f00039c0 0x7fe504120f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:31.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50b28e640 10 monclient(hunting): picked mon.noname-a con 0x7fe4f00039c0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:24:31.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50b28e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe4f0004810 0x7fe50409c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:31.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50b28e640 10 monclient(hunting): picked mon.noname-c con 0x7fe4f0004810 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:24:31.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50b28e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4f0005220 0x7fe50409ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:31.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50b28e640 10 monclient(hunting): picked mon.noname-b con 0x7fe4f0005220 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:24:31.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50b28e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:31.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50b28e640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe50409f430 con 0x7fe4f00039c0 2024-04-26T00:24:31.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50b28e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:31.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50b28e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe50409f2b0 con 0x7fe4f0005220 2024-04-26T00:24:31.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50b28e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:31.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50b28e640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe50409fe90 con 0x7fe4f0004810 2024-04-26T00:24:31.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50b28e640 10 monclient(hunting): _renew_subs 2024-04-26T00:24:31.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50b28e640 10 monclient(hunting): authenticate will time out at 1253.159427s 2024-04-26T00:24:31.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe508829640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe4f0004810 msgr2=0x7fe50409c4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:24:31.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe508829640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe4f0004810 0x7fe50409c4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:24:31.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50902a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4f00039c0 0x7fe504120f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:31.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50902a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4f00039c0 0x7fe504120f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58880/0 (socket says 172.21.15.37:58880) 2024-04-26T00:24:31.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50902a640 1 -- 172.21.15.37:0/2158534153 learned_addr learned my addr 172.21.15.37:0/2158534153 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:24:31.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50902a640 10 monclient(hunting): get_auth_request con 0x7fe4f00039c0 auth_method 0 2024-04-26T00:24:31.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50902a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:31.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50902a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:31.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50902a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:31.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50982b640 1 --2- 172.21.15.37:0/2158534153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4f0005220 0x7fe50409ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:31.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50902a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:31.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50902a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:31.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50902a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:31.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50982b640 10 monclient(hunting): get_auth_request con 0x7fe4f0005220 auth_method 0 2024-04-26T00:24:31.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50982b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:31.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50982b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:31.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.849+0000 7fe50982b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:31.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.853+0000 7fe50982b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:31.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.853+0000 7fe50982b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:31.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.853+0000 7fe50982b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:31.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.853+0000 7fe50902a640 10 monclient(hunting): handle_auth_done global_id 54150 payload 824 2024-04-26T00:24:31.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.853+0000 7fe50902a640 1 -- 172.21.15.37:0/2158534153 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe4f0004810 msgr2=0x7fe50409c4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:24:31.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.853+0000 7fe50902a640 1 --2- 172.21.15.37:0/2158534153 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe4f0004810 0x7fe50409c4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:31.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.853+0000 7fe50902a640 1 -- 172.21.15.37:0/2158534153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4f0005220 msgr2=0x7fe50409ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:31.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.853+0000 7fe50902a640 1 --2- 172.21.15.37:0/2158534153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4f0005220 0x7fe50409ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:31.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.853+0000 7fe50902a640 10 monclient: _finish_hunting 0 2024-04-26T00:24:31.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.853+0000 7fe50902a640 1 monclient: found mon.noname-a 2024-04-26T00:24:31.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.853+0000 7fe50902a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:24:31.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.853+0000 7fe50902a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:24:31.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.853+0000 7fe50902a640 1 -- 172.21.15.37:0/2158534153 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe50409d5a0 con 0x7fe4f00039c0 2024-04-26T00:24:31.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.853+0000 7fe50902a640 1 -- 172.21.15.37:0/2158534153 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4f00039c0 msgr2=0x7fe504120f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:31.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.853+0000 7fe50902a640 1 --2- 172.21.15.37:0/2158534153 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4f00039c0 0x7fe504120f70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe4f400ce70 tx=0x7fe4f400a7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:31.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.853+0000 7fe4f9ffb640 1 -- 172.21.15.37:0/2158534153 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f401c070 con 0x7fe4f00039c0 2024-04-26T00:24:31.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.853+0000 7fe4f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:31.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.853+0000 7fe4f9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:24:31.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.853+0000 7fe4f9ffb640 10 monclient: dump: 2024-04-26T00:24:31.840 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:31.840 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:31.840 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:31.840 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:31.840 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:31.840 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:31.840 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:31.840 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:31.840 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:31.840 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:31.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.853+0000 7fe4f9ffb640 10 monclient: _finish_auth 0 2024-04-26T00:24:31.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.853+0000 7fe4f9ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:24:31.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.853+0000 7fe4f9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:24:31.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.853+0000 7fe4f9ffb640 1 -- 172.21.15.37:0/2158534153 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4f400ae60 con 0x7fe4f00039c0 2024-04-26T00:24:31.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.853+0000 7fe4f9ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:24:31.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.853+0000 7fe4f9ffb640 1 -- 172.21.15.37:0/2158534153 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f4018a60 con 0x7fe4f00039c0 2024-04-26T00:24:31.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.853+0000 7fe50b28e640 5 monclient: authenticate success, global_id 54150 2024-04-26T00:24:31.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.853+0000 7fe50b28e640 10 monclient: _renew_subs 2024-04-26T00:24:31.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.853+0000 7fe50b28e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:24:31.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.853+0000 7fe50b28e640 1 -- 172.21.15.37:0/2158534153 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe50409dc20 con 0x7fe4f00039c0 2024-04-26T00:24:31.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.853+0000 7fe4fb7fe640 4 set_mon_vals no callback set 2024-04-26T00:24:31.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.853+0000 7fe4f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:31.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.853+0000 7fe4f9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:24:31.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.853+0000 7fe4f9ffb640 10 monclient: dump: 2024-04-26T00:24:31.842 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:31.842 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:31.842 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:31.842 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:31.842 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:31.842 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:31.842 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:31.842 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:31.842 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:31.842 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:31.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.853+0000 7fe50b28e640 10 monclient: _renew_subs 2024-04-26T00:24:31.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.853+0000 7fe50b28e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:24:31.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.853+0000 7fe50b28e640 1 -- 172.21.15.37:0/2158534153 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe504071730 con 0x7fe4f00039c0 2024-04-26T00:24:31.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.857+0000 7fe50b28e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:24:31.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.857+0000 7fe50b28e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:24:31.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.857+0000 7fe50b28e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:24:31.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.857+0000 7fe50b28e640 1 -- 172.21.15.37:0/2158534153 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe504066f40 con 0x7fe4f00039c0 2024-04-26T00:24:31.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.857+0000 7fe4f9ffb640 1 -- 172.21.15.37:0/2158534153 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe4f401f070 con 0x7fe4f00039c0 2024-04-26T00:24:31.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.857+0000 7fe4f9ffb640 1 --2- 172.21.15.37:0/2158534153 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe4e8036770 0x7fe4e8038c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:31.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.857+0000 7fe4f9ffb640 1 -- 172.21.15.37:0/2158534153 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(177..177 src has 1..177) ==== 3931+0+0 (secure 0 0 0) 0x7fe4f4049e30 con 0x7fe4f00039c0 2024-04-26T00:24:31.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.857+0000 7fe508829640 1 --2- 172.21.15.37:0/2158534153 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe4e8036770 0x7fe4e8038c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:31.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.857+0000 7fe508829640 10 monclient: get_auth_request con 0x7fe4e8036770 auth_method 0 2024-04-26T00:24:31.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.857+0000 7fe508829640 1 --2- 172.21.15.37:0/2158534153 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe4e8036770 0x7fe4e8038c30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe500001ae0 tx=0x7fe500001710 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:31.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.861+0000 7fe4f9ffb640 1 -- 172.21.15.37:0/2158534153 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe4f4048b80 con 0x7fe4f00039c0 2024-04-26T00:24:31.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.861+0000 7fe4f9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:24:31.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:31.861+0000 7fe4f9ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:24:32.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.085+0000 7fe50b28e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:24:32.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.085+0000 7fe50b28e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:24:32.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.085+0000 7fe50b28e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:24:32.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.085+0000 7fe50b28e640 1 -- 172.21.15.37:0/2158534153 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe50406b9a0 con 0x7fe4f00039c0 2024-04-26T00:24:32.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.085+0000 7fe4f9ffb640 1 -- 172.21.15.37:0/2158534153 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7fe4f401fba0 con 0x7fe4f00039c0 2024-04-26T00:24:32.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.085+0000 7fe4f9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:24:32.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.085+0000 7fe4f9ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:24:32.065 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":36,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:24:32.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.085+0000 7fe50b28e640 1 -- 172.21.15.37:0/2158534153 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe4e8036770 msgr2=0x7fe4e8038c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:32.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.085+0000 7fe50b28e640 1 --2- 172.21.15.37:0/2158534153 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe4e8036770 0x7fe4e8038c30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe500001ae0 tx=0x7fe500001710 comp rx=0 tx=0).stop 2024-04-26T00:24:32.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.085+0000 7fe50b28e640 10 monclient: shutdown 2024-04-26T00:24:32.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.085+0000 7fe50b28e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:24:32.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.085+0000 7fe50b28e640 1 -- 172.21.15.37:0/2158534153 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4f00039c0 msgr2=0x7fe504120f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:32.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.085+0000 7fe50b28e640 1 --2- 172.21.15.37:0/2158534153 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4f00039c0 0x7fe504120f70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe4f400ce70 tx=0x7fe4f400a7b0 comp rx=0 tx=0).stop 2024-04-26T00:24:32.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.089+0000 7fe50b28e640 1 -- 172.21.15.37:0/2158534153 shutdown_connections 2024-04-26T00:24:32.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.089+0000 7fe50b28e640 1 --2- 172.21.15.37:0/2158534153 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe4e8036770 0x7fe4e8038c30 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:32.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.089+0000 7fe50b28e640 1 --2- 172.21.15.37:0/2158534153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4f0005220 0x7fe50409ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:32.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.089+0000 7fe50b28e640 1 --2- 172.21.15.37:0/2158534153 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe4f0004810 0x7fe50409c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:32.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.089+0000 7fe50b28e640 1 --2- 172.21.15.37:0/2158534153 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4f00039c0 0x7fe504120f70 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:32.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.089+0000 7fe50b28e640 1 -- 172.21.15.37:0/2158534153 >> 172.21.15.37:0/2158534153 conn(0x7fe5040a1ad0 msgr2=0x7fe504131de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:24:32.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.089+0000 7fe50b28e640 1 -- 172.21.15.37:0/2158534153 shutdown_connections 2024-04-26T00:24:32.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.089+0000 7fe50b28e640 1 -- 172.21.15.37:0/2158534153 wait complete. 2024-04-26T00:24:32.082 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-26T00:24:32.082 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-04-26T00:24:32.082 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:24:32.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.225+0000 7f1b0a405640 10 monclient: get_monmap_and_config 2024-04-26T00:24:32.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.225+0000 7f1b0a405640 10 monclient: build_initial_monmap 2024-04-26T00:24:32.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.225+0000 7f1b0a405640 10 monclient: monmap: 2024-04-26T00:24:32.205 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:24:32.205 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:32.205 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:24:32.229228+0000 2024-04-26T00:24:32.205 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:24:32.229228+0000 2024-04-26T00:24:32.205 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:24:32.205 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:32.205 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:24:32.205 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:24:32.205 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:24:32.205 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:24:32.205 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:24:32.205 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:32.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.225+0000 7f1b0a405640 1 Processor -- start 2024-04-26T00:24:32.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.225+0000 7f1b0a405640 1 -- start start 2024-04-26T00:24:32.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.225+0000 7f1b0a405640 10 monclient: init 2024-04-26T00:24:32.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.225+0000 7f1b0a405640 10 monclient: _reopen_session rank -1 2024-04-26T00:24:32.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.225+0000 7f1b0a405640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-04-26T00:24:32.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.225+0000 7f1b0a405640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1b04136f20 addr v1:172.21.15.45:6789/0 2024-04-26T00:24:32.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.225+0000 7f1b0a405640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1b04137a10 addr v1:172.21.15.37:6789/0 2024-04-26T00:24:32.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.225+0000 7f1b0a405640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1b0409bff0 0x7f1b0413fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:32.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.225+0000 7f1b0a405640 10 monclient(hunting): picked mon.noname-c con 0x7f1b0409bff0 addr v2:172.21.15.45:3300/0 2024-04-26T00:24:32.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.225+0000 7f1b0a405640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b040a50f0 con 0x7f1b04137a10 2024-04-26T00:24:32.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.225+0000 7f1b0a405640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b0409c4a0 con 0x7f1b04136f20 2024-04-26T00:24:32.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.225+0000 7f1b0a405640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:32.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.225+0000 7f1b0a405640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f1b0409b330 con 0x7f1b0409bff0 2024-04-26T00:24:32.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.225+0000 7f1b0a405640 10 monclient(hunting): _renew_subs 2024-04-26T00:24:32.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.225+0000 7f1b0a405640 10 monclient(hunting): authenticate will time out at 1253.535035s 2024-04-26T00:24:32.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.225+0000 7f1b037fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f1b04137a10 0x7f1b0413c450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:37302/0 (socket says 172.21.15.37:37302) 2024-04-26T00:24:32.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.225+0000 7f1b037fe640 1 -- 172.21.15.37:0/1087619325 learned_addr learned my addr 172.21.15.37:0/1087619325 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:24:32.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.225+0000 7f1b089a2640 1 --2- 172.21.15.37:0/1087619325 >> v2:172.21.15.45:3300/0 conn(0x7f1b0409bff0 0x7f1b0413fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:32.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.225+0000 7f1b089a2640 10 monclient(hunting): get_auth_request con 0x7f1b0409bff0 auth_method 0 2024-04-26T00:24:32.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.225+0000 7f1b089a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:32.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.225+0000 7f1b089a2640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:32.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.225+0000 7f1b089a2640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:32.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.225+0000 7f1b089a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:32.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.225+0000 7f1b089a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:32.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.225+0000 7f1b089a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:32.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.229+0000 7f1b089a2640 10 monclient(hunting): handle_auth_done global_id 24397 payload 274 2024-04-26T00:24:32.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.229+0000 7f1b089a2640 1 -- 172.21.15.37:0/1087619325 >> v1:172.21.15.45:6789/0 conn(0x7f1b04136f20 legacy=0x7f1b04137300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:24:32.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.229+0000 7f1b089a2640 1 -- 172.21.15.37:0/1087619325 >> v1:172.21.15.37:6789/0 conn(0x7f1b04137a10 legacy=0x7f1b0413c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:24:32.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.229+0000 7f1b089a2640 10 monclient: _finish_hunting 0 2024-04-26T00:24:32.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.229+0000 7f1b089a2640 1 monclient: found mon.noname-c 2024-04-26T00:24:32.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.229+0000 7f1b089a2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:24:32.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.229+0000 7f1b089a2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:24:32.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.229+0000 7f1b089a2640 1 -- 172.21.15.37:0/1087619325 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b0409b610 con 0x7f1b0409bff0 2024-04-26T00:24:32.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b089a2640 1 -- 172.21.15.37:0/1087619325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b0409bff0 msgr2=0x7f1b0413fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:32.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b089a2640 1 --2- 172.21.15.37:0/1087619325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b0409bff0 0x7f1b0413fb80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1af40097c0 tx=0x7f1af401c340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a97c722fac7281b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:32.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b02ffd640 1 -- 172.21.15.37:0/1087619325 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af40b2070 con 0x7f1b0409bff0 2024-04-26T00:24:32.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:32.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b02ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:24:32.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b02ffd640 10 monclient: dump: 2024-04-26T00:24:32.213 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:32.213 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:32.213 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:32.214 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:32.214 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:32.214 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:32.214 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:32.214 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:32.214 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:32.214 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:32.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b02ffd640 10 monclient: _finish_auth 0 2024-04-26T00:24:32.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b02ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:24:32.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:24:32.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b02ffd640 1 -- 172.21.15.37:0/1087619325 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1af40a9070 con 0x7f1b0409bff0 2024-04-26T00:24:32.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b02ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:24:32.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b02ffd640 1 -- 172.21.15.37:0/1087619325 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af40ad880 con 0x7f1b0409bff0 2024-04-26T00:24:32.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:32.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b02ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:24:32.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b02ffd640 10 monclient: dump: 2024-04-26T00:24:32.215 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:32.215 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:32.215 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:32.215 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:32.215 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:32.215 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:32.215 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:32.215 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:32.216 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:32.216 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:32.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b0a405640 5 monclient: authenticate success, global_id 24397 2024-04-26T00:24:32.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b0a405640 10 monclient: get_monmap_and_config success 2024-04-26T00:24:32.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b0a405640 4 set_mon_vals no callback set 2024-04-26T00:24:32.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b0a405640 10 monclient: shutdown 2024-04-26T00:24:32.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b0a405640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:24:32.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b0a405640 1 -- 172.21.15.37:0/1087619325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b0409bff0 msgr2=0x7f1b0413fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:32.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b0a405640 1 --2- 172.21.15.37:0/1087619325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b0409bff0 0x7f1b0413fb80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1af40097c0 tx=0x7f1af401c340 comp rx=0 tx=0).stop 2024-04-26T00:24:32.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b0a405640 1 -- 172.21.15.37:0/1087619325 shutdown_connections 2024-04-26T00:24:32.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b0a405640 1 --2- 172.21.15.37:0/1087619325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b0409bff0 0x7f1b0413fb80 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:32.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b0a405640 1 -- 172.21.15.37:0/1087619325 >> 172.21.15.37:0/1087619325 conn(0x7f1b040a61b0 msgr2=0x7f1b040a65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:24:32.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b0a405640 1 -- 172.21.15.37:0/1087619325 shutdown_connections 2024-04-26T00:24:32.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b0a405640 1 -- 172.21.15.37:0/1087619325 wait complete. 2024-04-26T00:24:32.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b0a405640 10 monclient: build_initial_monmap 2024-04-26T00:24:32.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b0a405640 10 monclient: monmap: 2024-04-26T00:24:32.223 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:24:32.223 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:24:32.223 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:24:32.223 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:24:32.223 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:24:32.224 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:24:32.224 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:24:32.224 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:24:32.224 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:24:32.224 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:32.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b0a405640 1 Processor -- start 2024-04-26T00:24:32.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b0a405640 1 -- start start 2024-04-26T00:24:32.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b0a405640 10 monclient: init 2024-04-26T00:24:32.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b0a405640 10 monclient: _reopen_session rank -1 2024-04-26T00:24:32.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b0a405640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:24:32.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b0a405640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1b0409bff0 0x7f1b041cee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:32.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b0a405640 10 monclient(hunting): picked mon.noname-c con 0x7f1b0409bff0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:24:32.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b0a405640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1b04136f20 0x7f1b041cf360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:32.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b0a405640 10 monclient(hunting): picked mon.noname-a con 0x7f1b04136f20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:24:32.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b0a405640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b04137a10 0x7f1b041cf8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:32.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b0a405640 10 monclient(hunting): picked mon.noname-b con 0x7f1b04137a10 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:24:32.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b0a405640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:32.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b0a405640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f1b04142a60 con 0x7f1b04136f20 2024-04-26T00:24:32.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.233+0000 7f1b0a405640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:32.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b0a405640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1b041428e0 con 0x7f1b04137a10 2024-04-26T00:24:32.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b0a405640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:32.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b0a405640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f1b04142be0 con 0x7f1b0409bff0 2024-04-26T00:24:32.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b0a405640 10 monclient(hunting): _renew_subs 2024-04-26T00:24:32.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b0a405640 10 monclient(hunting): authenticate will time out at 1253.544023s 2024-04-26T00:24:32.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b03fff640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1b0409bff0 msgr2=0x7f1b041cee20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:24:32.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b03fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1b0409bff0 0x7f1b041cee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:24:32.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b037fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1b04136f20 0x7f1b041cf360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:32.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b037fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1b04136f20 0x7f1b041cf360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58896/0 (socket says 172.21.15.37:58896) 2024-04-26T00:24:32.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b037fe640 1 -- 172.21.15.37:0/2106593233 learned_addr learned my addr 172.21.15.37:0/2106593233 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:24:32.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b037fe640 10 monclient(hunting): get_auth_request con 0x7f1b04136f20 auth_method 0 2024-04-26T00:24:32.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:32.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b037fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:32.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b037fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:32.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:32.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:32.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:32.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b089a2640 1 --2- 172.21.15.37:0/2106593233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b04137a10 0x7f1b041cf8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:32.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b089a2640 10 monclient(hunting): get_auth_request con 0x7f1b04137a10 auth_method 0 2024-04-26T00:24:32.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b089a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:32.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b089a2640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:32.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b089a2640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:32.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b037fe640 10 monclient(hunting): handle_auth_done global_id 54153 payload 824 2024-04-26T00:24:32.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b037fe640 1 -- 172.21.15.37:0/2106593233 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1b0409bff0 msgr2=0x7f1b041cee20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:24:32.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b037fe640 1 --2- 172.21.15.37:0/2106593233 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1b0409bff0 0x7f1b041cee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:32.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b037fe640 1 -- 172.21.15.37:0/2106593233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b04137a10 msgr2=0x7f1b041cf8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:32.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b037fe640 1 --2- 172.21.15.37:0/2106593233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b04137a10 0x7f1b041cf8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:32.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b037fe640 10 monclient: _finish_hunting 0 2024-04-26T00:24:32.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b037fe640 1 monclient: found mon.noname-a 2024-04-26T00:24:32.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:24:32.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b037fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:24:32.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b037fe640 1 -- 172.21.15.37:0/2106593233 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b0413c990 con 0x7f1b04136f20 2024-04-26T00:24:32.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b037fe640 1 -- 172.21.15.37:0/2106593233 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1b04136f20 msgr2=0x7f1b041cf360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:32.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b037fe640 1 --2- 172.21.15.37:0/2106593233 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1b04136f20 0x7f1b041cf360 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1aec00cc10 tx=0x7f1aec00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:32.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b00ff9640 1 -- 172.21.15.37:0/2106593233 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aec012400 con 0x7f1b04136f20 2024-04-26T00:24:32.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b00ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:32.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b00ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:24:32.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b00ff9640 10 monclient: dump: 2024-04-26T00:24:32.228 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:32.228 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:32.228 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:32.228 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:32.228 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:32.228 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:32.228 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:32.228 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:32.228 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:32.228 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:32.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b00ff9640 10 monclient: _finish_auth 0 2024-04-26T00:24:32.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b00ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:24:32.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b00ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:24:32.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b0a405640 5 monclient: authenticate success, global_id 54153 2024-04-26T00:24:32.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b00ff9640 1 -- 172.21.15.37:0/2106593233 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1aec0125e0 con 0x7f1b04136f20 2024-04-26T00:24:32.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b0a405640 10 monclient: _renew_subs 2024-04-26T00:24:32.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b0a405640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:24:32.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b0a405640 1 -- 172.21.15.37:0/2106593233 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b0413ec80 con 0x7f1b04136f20 2024-04-26T00:24:32.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b00ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:24:32.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b00ff9640 1 -- 172.21.15.37:0/2106593233 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aec0129d0 con 0x7f1b04136f20 2024-04-26T00:24:32.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b00ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:32.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b00ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:24:32.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b027fc640 4 set_mon_vals no callback set 2024-04-26T00:24:32.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b00ff9640 10 monclient: dump: 2024-04-26T00:24:32.230 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:32.230 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:32.230 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:32.230 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:32.230 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:32.230 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:32.230 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:32.230 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:32.230 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:32.230 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:32.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b0a405640 10 monclient: _renew_subs 2024-04-26T00:24:32.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b0a405640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:24:32.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b0a405640 1 -- 172.21.15.37:0/2106593233 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b0413fc50 con 0x7f1b04136f20 2024-04-26T00:24:32.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b0a405640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:24:32.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b0a405640 1 --2- 172.21.15.37:0/2106593233 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1b04135540 0x7f1b0402c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:32.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b0a405640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:32.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.237+0000 7f1b0a405640 1 -- 172.21.15.37:0/2106593233 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f1b0413eb00 con 0x7f1b04135540 2024-04-26T00:24:32.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.241+0000 7f1b03fff640 1 --2- 172.21.15.37:0/2106593233 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1b04135540 0x7f1b0402c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:32.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.241+0000 7f1b03fff640 10 monclient: get_auth_request con 0x7f1b04135540 auth_method 0 2024-04-26T00:24:32.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.241+0000 7f1b03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:32.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.241+0000 7f1b03fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:32.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.241+0000 7f1b03fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:32.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.241+0000 7f1b00ff9640 1 -- 172.21.15.37:0/2106593233 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f1aec012bb0 con 0x7f1b04136f20 2024-04-26T00:24:32.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.241+0000 7f1b03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:32.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.241+0000 7f1b03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:32.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.241+0000 7f1b03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:32.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.241+0000 7f1b00ff9640 1 --2- 172.21.15.37:0/2106593233 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1ae4035e30 0x7f1ae40382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:32.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.241+0000 7f1b03fff640 10 monclient(hunting): handle_auth_done global_id 54156 payload 824 2024-04-26T00:24:32.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.241+0000 7f1b00ff9640 1 -- 172.21.15.37:0/2106593233 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(177..177 src has 1..177) ==== 3931+0+0 (secure 0 0 0) 0x7f1aec049210 con 0x7f1b04136f20 2024-04-26T00:24:32.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.241+0000 7f1b03fff640 1 -- 172.21.15.37:0/2106593233 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1b040a87b0 con 0x7f1b04135540 2024-04-26T00:24:32.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.241+0000 7f1b089a2640 1 --2- 172.21.15.37:0/2106593233 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1ae4035e30 0x7f1ae40382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:32.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.241+0000 7f1b089a2640 10 monclient: get_auth_request con 0x7f1ae4035e30 auth_method 0 2024-04-26T00:24:32.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.241+0000 7f1b03fff640 1 -- 172.21.15.37:0/2106593233 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1b04135540 msgr2=0x7f1b0402c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:32.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.241+0000 7f1b03fff640 1 --2- 172.21.15.37:0/2106593233 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1b04135540 0x7f1b0402c000 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1afc007fd0 tx=0x7f1afc00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:32.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.241+0000 7f1b00ff9640 1 -- 172.21.15.37:0/2106593233 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1afc017070 con 0x7f1b04135540 2024-04-26T00:24:32.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.241+0000 7f1b00ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:32.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.241+0000 7f1b00ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:24:32.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.241+0000 7f1b00ff9640 10 monclient: dump: 2024-04-26T00:24:32.232 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:32.233 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:32.233 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:32.233 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:32.233 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:32.233 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:32.233 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:32.233 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:32.233 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:32.233 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:32.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.241+0000 7f1b00ff9640 1 -- 172.21.15.37:0/2106593233 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1afc0057d0 con 0x7f1b04135540 2024-04-26T00:24:32.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.241+0000 7f1b00ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:24:32.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.241+0000 7f1b00ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:24:32.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.241+0000 7f1b00ff9640 1 -- 172.21.15.37:0/2106593233 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1b04135540 msgr2=0x7f1b0402c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:32.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.241+0000 7f1b00ff9640 1 --2- 172.21.15.37:0/2106593233 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1b04135540 0x7f1b0402c000 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1afc007fd0 tx=0x7f1afc00c810 comp rx=0 tx=0).stop 2024-04-26T00:24:32.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.241+0000 7f1b00ff9640 1 -- 172.21.15.37:0/2106593233 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1b04135540 msgr2=0x7f1b0402c000 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:24:32.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.241+0000 7f1b00ff9640 1 --2- 172.21.15.37:0/2106593233 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1b04135540 0x7f1b0402c000 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1afc007fd0 tx=0x7f1afc00c810 comp rx=0 tx=0).stop 2024-04-26T00:24:32.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.241+0000 7f1b089a2640 1 --2- 172.21.15.37:0/2106593233 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1ae4035e30 0x7f1ae40382f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1af40ac000 tx=0x7f1af4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:32.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.249+0000 7f1b0a405640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:24:32.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.249+0000 7f1b0a405640 1 --2- 172.21.15.37:0/2106593233 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1b04135920 0x7f1b0402c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:32.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.249+0000 7f1b0a405640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:32.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.249+0000 7f1b0a405640 1 -- 172.21.15.37:0/2106593233 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f1b0406d960 con 0x7f1b04135920 2024-04-26T00:24:32.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.249+0000 7f1b03fff640 1 --2- 172.21.15.37:0/2106593233 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1b04135920 0x7f1b0402c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:32.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.249+0000 7f1b03fff640 10 monclient: get_auth_request con 0x7f1b04135920 auth_method 0 2024-04-26T00:24:32.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.249+0000 7f1b03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:32.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.249+0000 7f1b03fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:32.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.249+0000 7f1b03fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:32.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.249+0000 7f1b03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:32.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.249+0000 7f1b03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:32.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.249+0000 7f1b03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:32.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.249+0000 7f1b03fff640 10 monclient(hunting): handle_auth_done global_id 54159 payload 824 2024-04-26T00:24:32.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.249+0000 7f1b03fff640 1 -- 172.21.15.37:0/2106593233 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1b0404f510 con 0x7f1b04135920 2024-04-26T00:24:32.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.253+0000 7f1b03fff640 1 -- 172.21.15.37:0/2106593233 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1b04135920 msgr2=0x7f1b0402c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:32.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.253+0000 7f1b03fff640 1 --2- 172.21.15.37:0/2106593233 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1b04135920 0x7f1b0402c540 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1afc007fd0 tx=0x7f1afc0037f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:32.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.253+0000 7f1b00ff9640 1 -- 172.21.15.37:0/2106593233 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1afc017040 con 0x7f1b04135920 2024-04-26T00:24:32.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.253+0000 7f1b00ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:32.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.253+0000 7f1b00ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:24:32.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.253+0000 7f1b00ff9640 10 monclient: dump: 2024-04-26T00:24:32.237 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:32.237 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:32.237 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:32.237 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:32.238 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:32.238 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:32.238 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:32.238 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:32.238 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:32.238 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:32.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.253+0000 7f1b00ff9640 1 -- 172.21.15.37:0/2106593233 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4875 (secure 0 0 0) 0x7f1afc012400 con 0x7f1b04135920 2024-04-26T00:24:32.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.253+0000 7f1b00ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:24:32.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.253+0000 7f1b00ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:24:32.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.253+0000 7f1b00ff9640 1 -- 172.21.15.37:0/2106593233 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1b04135920 msgr2=0x7f1b0402c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:32.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.253+0000 7f1b00ff9640 1 --2- 172.21.15.37:0/2106593233 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1b04135920 0x7f1b0402c540 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1afc007fd0 tx=0x7f1afc0037f0 comp rx=0 tx=0).stop 2024-04-26T00:24:32.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.253+0000 7f1b00ff9640 1 -- 172.21.15.37:0/2106593233 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1b04135920 msgr2=0x7f1b0402c540 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:24:32.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.253+0000 7f1b00ff9640 1 --2- 172.21.15.37:0/2106593233 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1b04135920 0x7f1b0402c540 secure :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1afc007fd0 tx=0x7f1afc0037f0 comp rx=0 tx=0).stop 2024-04-26T00:24:32.239 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:24:32.239 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:24:32.239 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:24:32.239 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:24:32.239 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 36, 2024-04-26T00:24:32.239 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 35579, 2024-04-26T00:24:32.240 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:24:32.240 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:24:32.240 INFO:teuthology.orchestra.run.smithi037.stdout: 1 2024-04-26T00:24:32.240 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:32.240 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:24:32.240 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:24:32.240 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:24:32.240 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:24:32.240 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:24:32.240 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:24:32.240 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:24:32.240 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:24:32.241 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:24:32.241 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:24:32.241 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:24:32.241 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:24:32.241 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:24:32.241 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:24:32.241 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:24:32.241 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:32.241 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:24:32.241 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:24:32.241 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:24:32.241 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:24:32.242 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:24:32.242 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:24:32.242 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:24:32.242 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:24:32.242 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:24:32.242 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:24:32.242 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:24:32.242 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:24:32.242 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:24:32.242 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:32.242 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:32.242 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:24:32.242 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:24:32.243 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:24:32.243 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:24:32.243 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:24:32.243 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:24:32.243 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:24:32.243 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:24:32.243 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:24:32.243 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:24:32.243 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:24:32.243 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:24:32.243 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:24:32.243 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:24:32.243 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:24:32.243 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:24:32.244 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:24:32.244 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:24:32.244 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:24:32.244 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:24:32.244 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:24:32.244 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:24:32.244 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:24:32.244 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:24:32.244 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:24:32.244 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:24:32.244 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:24:32.244 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:24:32.244 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:32.245 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:24:32.245 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:32.245 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:24:32.245 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:32.245 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:24:32.245 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:24:32.245 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:24:32.245 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:24:32.245 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:32.245 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:24:32.245 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:24:32.245 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:32.245 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:32.246 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:32.246 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:24:32.246 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:24:32.246 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:32.246 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:32.246 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:32.246 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:32.246 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:24:32.246 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:24:32.246 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:24:32.246 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:24:32.246 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:24:32.246 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:32.247 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:32.247 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:24:32.247 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:24:32.247 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:24:32.247 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:24:32.247 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:32.247 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:24:32.247 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:24:32.247 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:32.247 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:32.247 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:32.247 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:24:32.247 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:24:32.247 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:32.248 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:32.248 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:32.248 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:32.248 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:24:32.248 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:24:32.248 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:24:32.248 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:24:32.248 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:24:32.248 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:32.248 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:32.248 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:24:32.249 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:24:32.249 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:24:32.249 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:24:32.249 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:32.249 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:24:32.249 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:24:32.249 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:32.249 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:32.249 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:32.249 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:24:32.249 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:24:32.250 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:32.250 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:32.250 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:32.250 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:32.250 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:24:32.250 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:24:32.250 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:24:32.250 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:24:32.250 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:24:32.250 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:32.250 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:32.251 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:32.251 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:24:32.251 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:24:32.251 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:32.251 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:24:32.251 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:24:32.251 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:24:32.251 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:32.251 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:32.251 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:24:32.252 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:32.252 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:24:32.252 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:24:32.252 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:24:32.252 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:32.252 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:32.252 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:24:32.252 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:32.252 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:24:32.252 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:24:32.252 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:24:32.252 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:32.253 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:32.253 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:24:32.253 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:32.253 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:24:32.253 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:24:32.253 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:24:32.253 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:32.253 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:32.253 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:32.253 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:24:32.253 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:24:32.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.253+0000 7f1b0a405640 1 -- 172.21.15.37:0/2106593233 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1ae4035e30 msgr2=0x7f1ae40382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:32.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.253+0000 7f1b0a405640 1 --2- 172.21.15.37:0/2106593233 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1ae4035e30 0x7f1ae40382f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1af40ac000 tx=0x7f1af4008040 comp rx=0 tx=0).stop 2024-04-26T00:24:32.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.253+0000 7f1b0a405640 10 monclient: shutdown 2024-04-26T00:24:32.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.253+0000 7f1b0a405640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:24:32.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.253+0000 7f1b0a405640 1 -- 172.21.15.37:0/2106593233 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1b04136f20 msgr2=0x7f1b041cf360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:32.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.253+0000 7f1b0a405640 1 --2- 172.21.15.37:0/2106593233 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1b04136f20 0x7f1b041cf360 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1aec00cc10 tx=0x7f1aec00a430 comp rx=0 tx=0).stop 2024-04-26T00:24:32.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.253+0000 7f1b089a2640 1 -- 172.21.15.37:0/2106593233 reap_dead start 2024-04-26T00:24:32.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.253+0000 7f1b089a2640 1 -- 172.21.15.37:0/2106593233 reap_dead start 2024-04-26T00:24:32.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.253+0000 7f1b0a405640 1 -- 172.21.15.37:0/2106593233 shutdown_connections 2024-04-26T00:24:32.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.253+0000 7f1b0a405640 1 -- 172.21.15.37:0/2106593233 >> 172.21.15.37:0/2106593233 conn(0x7f1b040a61b0 msgr2=0x7f1b040a08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:24:32.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.253+0000 7f1b0a405640 1 -- 172.21.15.37:0/2106593233 shutdown_connections 2024-04-26T00:24:32.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.253+0000 7f1b0a405640 1 -- 172.21.15.37:0/2106593233 wait complete. 2024-04-26T00:24:32.256 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:24:32.256 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-04-26T00:24:32.256 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:24:32.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa14c92a640 10 monclient: get_monmap_and_config 2024-04-26T00:24:32.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa14c92a640 10 monclient: build_initial_monmap 2024-04-26T00:24:32.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa14c92a640 10 monclient: monmap: 2024-04-26T00:24:32.372 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:24:32.372 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:32.372 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:24:32.396982+0000 2024-04-26T00:24:32.372 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:24:32.396982+0000 2024-04-26T00:24:32.373 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:24:32.373 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:32.373 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:24:32.373 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:24:32.373 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:24:32.373 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:24:32.373 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:24:32.373 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:32.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa14c92a640 1 Processor -- start 2024-04-26T00:24:32.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa14c92a640 1 -- start start 2024-04-26T00:24:32.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa14c92a640 10 monclient: init 2024-04-26T00:24:32.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa14c92a640 10 monclient: _reopen_session rank -1 2024-04-26T00:24:32.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa14c92a640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-04-26T00:24:32.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa14c92a640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa1480a0300 0x7fa14809fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:32.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa14c92a640 10 monclient(hunting): picked mon.noname-c con 0x7fa1480a0300 addr v2:172.21.15.45:3300/0 2024-04-26T00:24:32.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa14c92a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1480a06e0 0x7fa148134ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:32.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa14c92a640 10 monclient(hunting): picked mon.noname-b con 0x7fa1480a06e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:24:32.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa14c92a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa1481350e0 addr v1:172.21.15.37:6789/0 2024-04-26T00:24:32.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa14c92a640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa14809c190 con 0x7fa1481350e0 2024-04-26T00:24:32.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa14c92a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:32.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa14c92a640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fa1480a13f0 con 0x7fa1480a0300 2024-04-26T00:24:32.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa14c92a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:32.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa14c92a640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa1480a1110 con 0x7fa1480a06e0 2024-04-26T00:24:32.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa14c92a640 10 monclient(hunting): _renew_subs 2024-04-26T00:24:32.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa14c92a640 10 monclient(hunting): authenticate will time out at 1253.702603s 2024-04-26T00:24:32.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa145d9b640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1480a06e0 msgr2=0x7fa148134ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:24:32.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa145d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1480a06e0 0x7fa148134ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:24:32.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa146d9d640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fa1481350e0 0x7fa14813bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:37312/0 (socket says 172.21.15.37:37312) 2024-04-26T00:24:32.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa146d9d640 1 -- 172.21.15.37:0/2420639279 learned_addr learned my addr 172.21.15.37:0/2420639279 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:24:32.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa14659c640 1 --2- 172.21.15.37:0/2420639279 >> v2:172.21.15.45:3300/0 conn(0x7fa1480a0300 0x7fa14809fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:32.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa14659c640 10 monclient(hunting): get_auth_request con 0x7fa1480a0300 auth_method 0 2024-04-26T00:24:32.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa14659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:32.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa14659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:32.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa14659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:32.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa14659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:32.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa14659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:32.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa14659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:32.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa14659c640 10 monclient(hunting): handle_auth_done global_id 24403 payload 274 2024-04-26T00:24:32.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa14659c640 1 -- 172.21.15.37:0/2420639279 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1480a06e0 msgr2=0x7fa148134ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:24:32.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa14659c640 1 --2- 172.21.15.37:0/2420639279 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1480a06e0 0x7fa148134ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:32.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa14659c640 1 -- 172.21.15.37:0/2420639279 >> v1:172.21.15.37:6789/0 conn(0x7fa1481350e0 legacy=0x7fa14813bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:24:32.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa14659c640 10 monclient: _finish_hunting 0 2024-04-26T00:24:32.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa14659c640 1 monclient: found mon.noname-c 2024-04-26T00:24:32.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa14659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:24:32.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa14659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:24:32.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.393+0000 7fa14659c640 1 -- 172.21.15.37:0/2420639279 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1480a16d0 con 0x7fa1480a0300 2024-04-26T00:24:32.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14659c640 1 -- 172.21.15.37:0/2420639279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1480a0300 msgr2=0x7fa14809fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:32.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14659c640 1 --2- 172.21.15.37:0/2420639279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1480a0300 0x7fa14809fc50 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa140009d90 tx=0x7fa14001c840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9e1b2fbd9d8106a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:32.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14559a640 1 -- 172.21.15.37:0/2420639279 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1400b2070 con 0x7fa1480a0300 2024-04-26T00:24:32.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:32.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:24:32.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14559a640 10 monclient: dump: 2024-04-26T00:24:32.380 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:32.381 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:32.381 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:32.381 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:32.381 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:32.381 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:32.381 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:32.381 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:32.381 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:32.381 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:32.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14559a640 10 monclient: _finish_auth 0 2024-04-26T00:24:32.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14559a640 10 monclient: _check_auth_tickets 2024-04-26T00:24:32.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:24:32.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14559a640 1 -- 172.21.15.37:0/2420639279 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1400a5890 con 0x7fa1480a0300 2024-04-26T00:24:32.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14c92a640 5 monclient: authenticate success, global_id 24403 2024-04-26T00:24:32.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14c92a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:24:32.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14559a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:24:32.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14c92a640 10 monclient: get_monmap_and_config success 2024-04-26T00:24:32.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14559a640 1 -- 172.21.15.37:0/2420639279 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1400a5c60 con 0x7fa1480a0300 2024-04-26T00:24:32.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:32.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14c92a640 4 set_mon_vals no callback set 2024-04-26T00:24:32.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:24:32.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14559a640 10 monclient: dump: 2024-04-26T00:24:32.382 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:32.382 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:32.382 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:32.382 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:32.383 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:32.383 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:32.383 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:32.383 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:32.383 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:32.383 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:32.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14c92a640 10 monclient: shutdown 2024-04-26T00:24:32.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14c92a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:24:32.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14c92a640 1 -- 172.21.15.37:0/2420639279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1480a0300 msgr2=0x7fa14809fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:32.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14c92a640 1 --2- 172.21.15.37:0/2420639279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1480a0300 0x7fa14809fc50 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa140009d90 tx=0x7fa14001c840 comp rx=0 tx=0).stop 2024-04-26T00:24:32.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14c92a640 1 -- 172.21.15.37:0/2420639279 shutdown_connections 2024-04-26T00:24:32.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14c92a640 1 --2- 172.21.15.37:0/2420639279 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1480a06e0 0x7fa148134ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:32.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14c92a640 1 --2- 172.21.15.37:0/2420639279 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1480a0300 0x7fa14809fc50 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:32.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14c92a640 1 -- 172.21.15.37:0/2420639279 >> 172.21.15.37:0/2420639279 conn(0x7fa1480a1d50 msgr2=0x7fa1480a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:24:32.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14c92a640 1 -- 172.21.15.37:0/2420639279 shutdown_connections 2024-04-26T00:24:32.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14c92a640 1 -- 172.21.15.37:0/2420639279 wait complete. 2024-04-26T00:24:32.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14c92a640 10 monclient: build_initial_monmap 2024-04-26T00:24:32.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14c92a640 10 monclient: monmap: 2024-04-26T00:24:32.389 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:24:32.389 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:24:32.389 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:24:32.389 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:24:32.389 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:24:32.389 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:24:32.389 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:24:32.389 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:24:32.389 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:24:32.389 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:32.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14c92a640 1 Processor -- start 2024-04-26T00:24:32.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14c92a640 1 -- start start 2024-04-26T00:24:32.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14c92a640 10 monclient: init 2024-04-26T00:24:32.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14c92a640 10 monclient: _reopen_session rank -1 2024-04-26T00:24:32.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14c92a640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:24:32.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14c92a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1480a0300 0x7fa1481cb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:32.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14c92a640 10 monclient(hunting): picked mon.noname-a con 0x7fa1480a0300 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:24:32.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14c92a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1480a06e0 0x7fa14813c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:32.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14c92a640 10 monclient(hunting): picked mon.noname-c con 0x7fa1480a06e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:24:32.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14c92a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1481350e0 0x7fa14813c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:32.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14c92a640 10 monclient(hunting): picked mon.noname-b con 0x7fa1481350e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:24:32.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14c92a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:32.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14c92a640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa1480a0c40 con 0x7fa1480a0300 2024-04-26T00:24:32.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14c92a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:32.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14c92a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa1480a0ac0 con 0x7fa1481350e0 2024-04-26T00:24:32.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14c92a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:32.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14c92a640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa14809c750 con 0x7fa1480a06e0 2024-04-26T00:24:32.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14c92a640 10 monclient(hunting): _renew_subs 2024-04-26T00:24:32.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14c92a640 10 monclient(hunting): authenticate will time out at 1253.710957s 2024-04-26T00:24:32.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa145d9b640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1480a06e0 msgr2=0x7fa14813c150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:24:32.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa145d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1480a06e0 0x7fa14813c150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:24:32.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14659c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1480a0300 0x7fa1481cb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:32.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14659c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1480a0300 0x7fa1481cb0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58922/0 (socket says 172.21.15.37:58922) 2024-04-26T00:24:32.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14659c640 1 -- 172.21.15.37:0/766167692 learned_addr learned my addr 172.21.15.37:0/766167692 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:24:32.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14659c640 10 monclient(hunting): get_auth_request con 0x7fa1480a0300 auth_method 0 2024-04-26T00:24:32.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:32.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:32.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:32.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:32.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:32.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:32.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa146d9d640 1 --2- 172.21.15.37:0/766167692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1481350e0 0x7fa14813c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:32.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa146d9d640 10 monclient(hunting): get_auth_request con 0x7fa1481350e0 auth_method 0 2024-04-26T00:24:32.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa146d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:32.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa146d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:32.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa146d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:32.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14659c640 10 monclient(hunting): handle_auth_done global_id 54162 payload 824 2024-04-26T00:24:32.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14659c640 1 -- 172.21.15.37:0/766167692 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1480a06e0 msgr2=0x7fa14813c150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:24:32.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14659c640 1 --2- 172.21.15.37:0/766167692 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1480a06e0 0x7fa14813c150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:32.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14659c640 1 -- 172.21.15.37:0/766167692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1481350e0 msgr2=0x7fa14813c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:32.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14659c640 1 --2- 172.21.15.37:0/766167692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1481350e0 0x7fa14813c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:32.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14659c640 10 monclient: _finish_hunting 0 2024-04-26T00:24:32.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14659c640 1 monclient: found mon.noname-a 2024-04-26T00:24:32.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:24:32.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:24:32.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.401+0000 7fa14659c640 1 -- 172.21.15.37:0/766167692 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa148133860 con 0x7fa1480a0300 2024-04-26T00:24:32.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.405+0000 7fa14659c640 1 -- 172.21.15.37:0/766167692 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1480a0300 msgr2=0x7fa1481cb0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:32.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.405+0000 7fa14659c640 1 --2- 172.21.15.37:0/766167692 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1480a0300 0x7fa1481cb0d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa14001af50 tx=0x7fa140019640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:32.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.405+0000 7fa14659c640 1 --2- 172.21.15.37:0/766167692 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1480a0300 0x7fa1481cb0d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa14001af50 tx=0x7fa140019640 comp rx=0 tx=0).queue_received will delay after 1.71409e+09 on 0x7fa1400bd020 mon_map magic: 0 2024-04-26T00:24:32.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.405+0000 7fa136ffd640 1 -- 172.21.15.37:0/766167692 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1400bd020 con 0x7fa1480a0300 2024-04-26T00:24:32.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.405+0000 7fa136ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:32.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.405+0000 7fa136ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:24:32.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.405+0000 7fa136ffd640 10 monclient: dump: 2024-04-26T00:24:32.393 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:32.393 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:32.393 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:32.393 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:32.393 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:32.394 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:32.394 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:32.394 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:32.394 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:32.394 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:32.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.405+0000 7fa136ffd640 10 monclient: _finish_auth 0 2024-04-26T00:24:32.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.405+0000 7fa136ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:24:32.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.405+0000 7fa136ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:24:32.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.405+0000 7fa136ffd640 1 -- 172.21.15.37:0/766167692 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1400b3040 con 0x7fa1480a0300 2024-04-26T00:24:32.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.405+0000 7fa136ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:24:32.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.405+0000 7fa144d99640 4 set_mon_vals no callback set 2024-04-26T00:24:32.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.405+0000 7fa14c92a640 5 monclient: authenticate success, global_id 54162 2024-04-26T00:24:32.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.405+0000 7fa14c92a640 10 monclient: _renew_subs 2024-04-26T00:24:32.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.405+0000 7fa14c92a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:24:32.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.405+0000 7fa14c92a640 1 -- 172.21.15.37:0/766167692 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa14809f550 con 0x7fa1480a0300 2024-04-26T00:24:32.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.405+0000 7fa14c92a640 10 monclient: _renew_subs 2024-04-26T00:24:32.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.405+0000 7fa14c92a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:24:32.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.405+0000 7fa14c92a640 1 -- 172.21.15.37:0/766167692 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1481cb610 con 0x7fa1480a0300 2024-04-26T00:24:32.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.405+0000 7fa14c92a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:24:32.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.405+0000 7fa14c92a640 1 --2- 172.21.15.37:0/766167692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1481310b0 0x7fa148131490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:32.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.405+0000 7fa14c92a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:32.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.405+0000 7fa14c92a640 1 -- 172.21.15.37:0/766167692 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa14804f510 con 0x7fa1481310b0 2024-04-26T00:24:32.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.405+0000 7fa136ffd640 1 -- 172.21.15.37:0/766167692 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1400b2070 con 0x7fa1480a0300 2024-04-26T00:24:32.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.405+0000 7fa136ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:32.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.405+0000 7fa136ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:24:32.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.405+0000 7fa136ffd640 10 monclient: dump: 2024-04-26T00:24:32.395 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:32.395 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:32.395 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:32.396 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:32.396 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:32.396 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:32.396 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:32.396 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:32.396 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:32.396 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:32.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.405+0000 7fa136ffd640 1 -- 172.21.15.37:0/766167692 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa1400c0020 con 0x7fa1480a0300 2024-04-26T00:24:32.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.405+0000 7fa136ffd640 1 --2- 172.21.15.37:0/766167692 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa128035e60 0x7fa128038320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:32.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.405+0000 7fa145d9b640 1 --2- 172.21.15.37:0/766167692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1481310b0 0x7fa148131490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:32.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.405+0000 7fa145d9b640 10 monclient: get_auth_request con 0x7fa1481310b0 auth_method 0 2024-04-26T00:24:32.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.405+0000 7fa145d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:32.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.405+0000 7fa145d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:32.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.405+0000 7fa145d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:32.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.405+0000 7fa146d9d640 1 --2- 172.21.15.37:0/766167692 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa128035e60 0x7fa128038320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:32.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.409+0000 7fa146d9d640 10 monclient: get_auth_request con 0x7fa128035e60 auth_method 0 2024-04-26T00:24:32.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.409+0000 7fa145d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:32.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.409+0000 7fa145d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:32.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.409+0000 7fa145d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:32.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.409+0000 7fa145d9b640 10 monclient(hunting): handle_auth_done global_id 24409 payload 824 2024-04-26T00:24:32.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.409+0000 7fa145d9b640 1 -- 172.21.15.37:0/766167692 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa148134af0 con 0x7fa1481310b0 2024-04-26T00:24:32.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.409+0000 7fa145d9b640 1 -- 172.21.15.37:0/766167692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1481310b0 msgr2=0x7fa148131490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:32.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.409+0000 7fa145d9b640 1 --2- 172.21.15.37:0/766167692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1481310b0 0x7fa148131490 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa148134640 tx=0x7fa13c00b8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:32.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.409+0000 7fa146d9d640 1 --2- 172.21.15.37:0/766167692 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa128035e60 0x7fa128038320 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa148135d10 tx=0x7fa138009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:32.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.409+0000 7fa136ffd640 1 -- 172.21.15.37:0/766167692 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa13c016070 con 0x7fa1481310b0 2024-04-26T00:24:32.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.409+0000 7fa136ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:32.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.409+0000 7fa136ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:24:32.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.409+0000 7fa136ffd640 10 monclient: dump: 2024-04-26T00:24:32.399 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:32.399 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:32.400 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:32.400 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:32.400 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:32.400 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:32.400 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:32.400 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:32.400 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:32.400 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:32.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.409+0000 7fa136ffd640 1 -- 172.21.15.37:0/766167692 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa13c005270 con 0x7fa1481310b0 2024-04-26T00:24:32.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.409+0000 7fa136ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:24:32.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.409+0000 7fa136ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:24:32.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.409+0000 7fa136ffd640 1 -- 172.21.15.37:0/766167692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1481310b0 msgr2=0x7fa148131490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:32.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.409+0000 7fa136ffd640 1 --2- 172.21.15.37:0/766167692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1481310b0 0x7fa148131490 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa148134640 tx=0x7fa13c00b8e0 comp rx=0 tx=0).stop 2024-04-26T00:24:32.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.409+0000 7fa136ffd640 1 -- 172.21.15.37:0/766167692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1481310b0 msgr2=0x7fa148131490 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:24:32.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.409+0000 7fa136ffd640 1 --2- 172.21.15.37:0/766167692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1481310b0 0x7fa148131490 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:32.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.417+0000 7fa14c92a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:24:32.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.417+0000 7fa14c92a640 1 --2- 172.21.15.37:0/766167692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1481319d0 0x7fa14806e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:24:32.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.417+0000 7fa14c92a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:24:32.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.417+0000 7fa14c92a640 1 -- 172.21.15.37:0/766167692 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa14809eb40 con 0x7fa1481319d0 2024-04-26T00:24:32.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.417+0000 7fa145d9b640 1 --2- 172.21.15.37:0/766167692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1481319d0 0x7fa14806e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:24:32.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.417+0000 7fa145d9b640 10 monclient: get_auth_request con 0x7fa1481319d0 auth_method 0 2024-04-26T00:24:32.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.417+0000 7fa145d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:24:32.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.417+0000 7fa145d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:24:32.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.417+0000 7fa145d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:24:32.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.417+0000 7fa145d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:24:32.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.417+0000 7fa145d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:24:32.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.417+0000 7fa145d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:24:32.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.417+0000 7fa145d9b640 10 monclient(hunting): handle_auth_done global_id 24412 payload 824 2024-04-26T00:24:32.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.417+0000 7fa145d9b640 1 -- 172.21.15.37:0/766167692 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa14806ba50 con 0x7fa1481319d0 2024-04-26T00:24:32.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.421+0000 7fa145d9b640 1 -- 172.21.15.37:0/766167692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1481319d0 msgr2=0x7fa14806e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:24:32.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.421+0000 7fa145d9b640 1 --2- 172.21.15.37:0/766167692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1481319d0 0x7fa14806e270 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa13c013000 tx=0x7fa13c0059e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:24:32.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.421+0000 7fa136ffd640 1 -- 172.21.15.37:0/766167692 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa13c016040 con 0x7fa1481319d0 2024-04-26T00:24:32.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.421+0000 7fa136ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:24:32.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.421+0000 7fa136ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:24:32.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.421+0000 7fa136ffd640 10 monclient: dump: 2024-04-26T00:24:32.403 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:24:32.403 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:24:32.403 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:32.403 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:24:32.403 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:24:32.403 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:24:32.403 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:24:32.404 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:24:32.404 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:24:32.404 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:24:32.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.421+0000 7fa136ffd640 1 -- 172.21.15.37:0/766167692 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4708 (secure 0 0 0) 0x7fa13c021450 con 0x7fa1481319d0 2024-04-26T00:24:32.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.421+0000 7fa136ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:24:32.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.421+0000 7fa136ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:24:32.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.421+0000 7fa136ffd640 1 -- 172.21.15.37:0/766167692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1481319d0 msgr2=0x7fa14806e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:32.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.421+0000 7fa136ffd640 1 --2- 172.21.15.37:0/766167692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1481319d0 0x7fa14806e270 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa13c013000 tx=0x7fa13c0059e0 comp rx=0 tx=0).stop 2024-04-26T00:24:32.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.421+0000 7fa136ffd640 1 -- 172.21.15.37:0/766167692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1481319d0 msgr2=0x7fa14806e270 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:24:32.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.421+0000 7fa136ffd640 1 --2- 172.21.15.37:0/766167692 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1481319d0 0x7fa14806e270 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:24:32.404 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:24:32.405 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:24:32.405 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:24:32.405 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:24:32.405 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 36, 2024-04-26T00:24:32.405 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 180588, 2024-04-26T00:24:32.405 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:24:32.405 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:24:32.405 INFO:teuthology.orchestra.run.smithi037.stdout: 1 2024-04-26T00:24:32.405 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:32.405 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:24:32.405 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:24:32.405 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:24:32.405 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:24:32.405 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:24:32.405 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:24:32.406 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:24:32.406 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:24:32.406 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:24:32.406 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:24:32.406 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:24:32.406 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:24:32.406 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:24:32.406 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:24:32.406 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:24:32.406 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:32.406 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:24:32.406 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:24:32.406 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:24:32.407 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:24:32.407 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:24:32.407 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:24:32.407 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:24:32.407 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:24:32.407 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:24:32.407 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:24:32.407 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:24:32.407 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:24:32.407 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:24:32.407 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:32.408 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:32.408 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:24:32.408 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:24:32.408 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:24:32.408 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:24:32.408 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:24:32.408 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:24:32.408 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:24:32.408 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:24:32.408 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:24:32.408 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:24:32.408 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:24:32.408 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:24:32.408 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:24:32.408 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:24:32.409 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:24:32.409 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:24:32.409 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:24:32.409 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:24:32.409 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:24:32.409 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:24:32.409 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:24:32.409 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:24:32.409 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:24:32.409 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:24:32.409 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:24:32.409 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:24:32.409 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:24:32.409 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:24:32.410 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:32.410 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:24:32.410 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:32.410 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:24:32.410 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:32.410 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:24:32.410 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:24:32.410 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:24:32.410 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:24:32.410 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:32.410 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:24:32.410 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:24:32.410 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:32.411 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:32.411 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:32.411 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:24:32.411 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:24:32.411 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:32.411 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:32.411 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:32.411 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:32.411 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:24:32.411 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:24:32.411 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:24:32.411 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:24:32.411 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:24:32.412 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:32.412 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:32.412 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:24:32.412 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:24:32.412 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:24:32.412 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:24:32.412 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:32.412 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:24:32.412 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:24:32.412 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:32.412 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:32.412 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:32.412 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:24:32.412 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:24:32.412 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:32.413 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:32.413 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:32.413 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:32.413 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:24:32.413 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:24:32.413 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:24:32.413 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:24:32.413 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:24:32.413 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:32.413 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:32.413 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:24:32.413 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:24:32.413 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:24:32.413 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:24:32.413 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:32.414 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:24:32.414 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:24:32.414 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:32.414 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:32.414 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:32.414 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:24:32.414 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:24:32.414 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:24:32.414 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:32.414 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:32.414 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:32.414 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:24:32.414 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:24:32.414 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:24:32.415 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:24:32.415 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:24:32.415 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:32.415 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:32.415 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:32.415 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:24:32.415 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:24:32.415 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:32.415 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:24:32.415 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:24:32.415 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:24:32.415 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:32.415 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:32.415 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:24:32.415 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:32.416 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:24:32.416 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:24:32.416 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:24:32.416 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:32.416 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:24:32.416 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:24:32.416 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:24:32.416 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:24:32.416 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:24:32.416 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:24:32.416 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:24:32.416 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:24:32.416 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:24:32.416 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:24:32.416 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:24:32.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.421+0000 7fa14c92a640 1 -- 172.21.15.37:0/766167692 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa128035e60 msgr2=0x7fa128038320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:32.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.421+0000 7fa14c92a640 1 --2- 172.21.15.37:0/766167692 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa128035e60 0x7fa128038320 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa148135d10 tx=0x7fa138009040 comp rx=0 tx=0).stop 2024-04-26T00:24:32.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.421+0000 7fa14c92a640 10 monclient: shutdown 2024-04-26T00:24:32.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.421+0000 7fa14c92a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:24:32.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.421+0000 7fa14c92a640 1 -- 172.21.15.37:0/766167692 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1480a0300 msgr2=0x7fa1481cb0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:24:32.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.421+0000 7fa14c92a640 1 --2- 172.21.15.37:0/766167692 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1480a0300 0x7fa1481cb0d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa14001af50 tx=0x7fa140019640 comp rx=0 tx=0).stop 2024-04-26T00:24:32.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.421+0000 7fa146d9d640 1 -- 172.21.15.37:0/766167692 reap_dead start 2024-04-26T00:24:32.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.421+0000 7fa136ffd640 1 -- 172.21.15.37:0/766167692 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(177..177 src has 1..177) ==== 3931+0+0 (unknown 0 0 0) 0x7fa1400e3ac0 con 0x7fa1480a0300 2024-04-26T00:24:32.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.421+0000 7fa146d9d640 1 -- 172.21.15.37:0/766167692 reap_dead start 2024-04-26T00:24:32.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.421+0000 7fa14c92a640 1 -- 172.21.15.37:0/766167692 shutdown_connections 2024-04-26T00:24:32.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.421+0000 7fa14c92a640 1 -- 172.21.15.37:0/766167692 >> 172.21.15.37:0/766167692 conn(0x7fa1480a1d50 msgr2=0x7fa14809da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:24:32.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.421+0000 7fa14c92a640 1 -- 172.21.15.37:0/766167692 shutdown_connections 2024-04-26T00:24:32.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:24:32.421+0000 7fa14c92a640 1 -- 172.21.15.37:0/766167692 wait complete. 2024-04-26T00:24:32.419 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:24:32.419 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-04-26T00:24:32.784 INFO:tasks.rados.rados.0.smithi045.stdout:762: done (7 left) 2024-04-26T00:24:32.785 INFO:tasks.rados.rados.0.smithi045.stdout:763: snap_create 2024-04-26T00:24:32.785 INFO:tasks.rados.rados.0.smithi045.stdout:755: finishing write tid 3 to smithi04526579-31 2024-04-26T00:24:32.785 INFO:tasks.rados.rados.0.smithi045.stdout:755: finishing write tid 4 to smithi04526579-31 2024-04-26T00:24:32.785 INFO:tasks.rados.rados.0.smithi045.stdout:755: oid 31 updating version 0 to 367 2024-04-26T00:24:32.785 INFO:tasks.rados.rados.0.smithi045.stdout:755: oid 31 version 367 is already newer than 366 2024-04-26T00:24:32.785 INFO:tasks.rados.rados.0.smithi045.stdout:755: oid 31 version 367 is already newer than 365 2024-04-26T00:24:32.785 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 367 (ObjNum 303 snap 87 seq_num 303) dirty exists 2024-04-26T00:24:32.785 INFO:tasks.rados.rados.0.smithi045.stdout:755: left oid 31 (ObjNum 303 snap 87 seq_num 303) 2024-04-26T00:24:32.785 INFO:tasks.rados.rados.0.smithi045.stdout:756: finishing rollback tid 0 to smithi04526579-1 2024-04-26T00:24:32.785 INFO:tasks.rados.rados.0.smithi045.stdout:756: finishing rollback tid 1 to smithi04526579-1 2024-04-26T00:24:32.785 INFO:tasks.rados.rados.0.smithi045.stdout:756: finishing rollback tid 2 to smithi04526579-1 2024-04-26T00:24:32.785 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 396 (ObjNum 258 snap 68 seq_num 258) dirty exists 2024-04-26T00:24:32.785 INFO:tasks.rados.rados.0.smithi045.stdout:757: finishing write tid 1 to smithi04526579-36 2024-04-26T00:24:32.786 INFO:tasks.rados.rados.0.smithi045.stdout:757: finishing write tid 2 to smithi04526579-36 2024-04-26T00:24:32.786 INFO:tasks.rados.rados.0.smithi045.stdout:757: finishing write tid 3 to smithi04526579-36 2024-04-26T00:24:32.786 INFO:tasks.rados.rados.0.smithi045.stdout:757: finishing write tid 4 to smithi04526579-36 2024-04-26T00:24:32.786 INFO:tasks.rados.rados.0.smithi045.stdout:757: oid 36 updating version 0 to 167 2024-04-26T00:24:32.786 INFO:tasks.rados.rados.0.smithi045.stdout:757: oid 36 version 167 is already newer than 166 2024-04-26T00:24:32.786 INFO:tasks.rados.rados.0.smithi045.stdout:757: oid 36 version 167 is already newer than 165 2024-04-26T00:24:32.786 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 167 (ObjNum 304 snap 87 seq_num 304) dirty exists 2024-04-26T00:24:32.786 INFO:tasks.rados.rados.0.smithi045.stdout:757: left oid 36 (ObjNum 304 snap 87 seq_num 304) 2024-04-26T00:24:32.786 INFO:tasks.rados.rados.0.smithi045.stdout:758: finishing write tid 1 to smithi04526579-35 2024-04-26T00:24:32.786 INFO:tasks.rados.rados.0.smithi045.stdout:758: finishing write tid 2 to smithi04526579-35 2024-04-26T00:24:32.786 INFO:tasks.rados.rados.0.smithi045.stdout:758: finishing write tid 3 to smithi04526579-35 2024-04-26T00:24:32.786 INFO:tasks.rados.rados.0.smithi045.stdout:758: finishing write tid 4 to smithi04526579-35 2024-04-26T00:24:32.786 INFO:tasks.rados.rados.0.smithi045.stdout:758: finishing write tid 5 to smithi04526579-35 2024-04-26T00:24:32.786 INFO:tasks.rados.rados.0.smithi045.stdout:758: oid 35 updating version 0 to 373 2024-04-26T00:24:32.786 INFO:tasks.rados.rados.0.smithi045.stdout:758: oid 35 version 373 is already newer than 371 2024-04-26T00:24:32.787 INFO:tasks.rados.rados.0.smithi045.stdout:758: oid 35 version 373 is already newer than 370 2024-04-26T00:24:32.787 INFO:tasks.rados.rados.0.smithi045.stdout:758: oid 35 version 373 is already newer than 372 2024-04-26T00:24:32.787 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 373 (ObjNum 305 snap 87 seq_num 305) dirty exists 2024-04-26T00:24:32.787 INFO:tasks.rados.rados.0.smithi045.stdout:758: left oid 35 (ObjNum 305 snap 87 seq_num 305) 2024-04-26T00:24:32.787 INFO:tasks.rados.rados.0.smithi045.stdout:759: expect (ObjNum 256 snap 66 seq_num 256) 2024-04-26T00:24:32.933 INFO:tasks.rados.rados.0.smithi045.stdout:760: finishing write tid 1 to smithi04526579-45 2024-04-26T00:24:32.933 INFO:tasks.rados.rados.0.smithi045.stdout:760: finishing write tid 2 to smithi04526579-45 2024-04-26T00:24:32.933 INFO:tasks.rados.rados.0.smithi045.stdout:760: finishing write tid 3 to smithi04526579-45 2024-04-26T00:24:32.933 INFO:tasks.rados.rados.0.smithi045.stdout:760: finishing write tid 4 to smithi04526579-45 2024-04-26T00:24:32.933 INFO:tasks.rados.rados.0.smithi045.stdout:760: oid 45 updating version 0 to 302 2024-04-26T00:24:32.933 INFO:tasks.rados.rados.0.smithi045.stdout:760: oid 45 updating version 302 to 303 2024-04-26T00:24:32.933 INFO:tasks.rados.rados.0.smithi045.stdout:760: oid 45 version 303 is already newer than 301 2024-04-26T00:24:32.933 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 303 (ObjNum 306 snap 87 seq_num 306) dirty exists 2024-04-26T00:24:32.934 INFO:tasks.rados.rados.0.smithi045.stdout:760: left oid 45 (ObjNum 306 snap 87 seq_num 306) 2024-04-26T00:24:33.786 INFO:tasks.rados.rados.0.smithi045.stdout:755: done (7 left) 2024-04-26T00:24:33.786 INFO:tasks.rados.rados.0.smithi045.stdout:756: done (6 left) 2024-04-26T00:24:33.786 INFO:tasks.rados.rados.0.smithi045.stdout:757: done (5 left) 2024-04-26T00:24:33.786 INFO:tasks.rados.rados.0.smithi045.stdout:758: done (4 left) 2024-04-26T00:24:33.786 INFO:tasks.rados.rados.0.smithi045.stdout:759: done (3 left) 2024-04-26T00:24:33.786 INFO:tasks.rados.rados.0.smithi045.stdout:760: done (2 left) 2024-04-26T00:24:33.786 INFO:tasks.rados.rados.0.smithi045.stdout:761: done (1 left) 2024-04-26T00:24:33.786 INFO:tasks.rados.rados.0.smithi045.stdout:763: done (0 left) 2024-04-26T00:24:33.786 INFO:tasks.rados.rados.0.smithi045.stdout:764: snap_create 2024-04-26T00:24:34.790 INFO:tasks.rados.rados.0.smithi045.stdout:764: done (0 left) 2024-04-26T00:24:34.790 INFO:tasks.rados.rados.0.smithi045.stdout:765: delete oid 3 current snap is 89 2024-04-26T00:24:34.798 INFO:tasks.rados.rados.0.smithi045.stdout:765: done (0 left) 2024-04-26T00:24:34.798 INFO:tasks.rados.rados.0.smithi045.stdout:766: read oid 1 snap 78 2024-04-26T00:24:34.798 INFO:tasks.rados.rados.0.smithi045.stdout:766: expect (ObjNum 258 snap 68 seq_num 258) 2024-04-26T00:24:34.798 INFO:tasks.rados.rados.0.smithi045.stdout:767: read oid 35 snap -1 2024-04-26T00:24:34.799 INFO:tasks.rados.rados.0.smithi045.stdout:767: expect (ObjNum 305 snap 87 seq_num 305) 2024-04-26T00:24:34.799 INFO:tasks.rados.rados.0.smithi045.stdout:768: snap_create 2024-04-26T00:24:34.814 INFO:tasks.rados.rados.0.smithi045.stdout:766: expect (ObjNum 258 snap 68 seq_num 258) 2024-04-26T00:24:34.962 INFO:tasks.rados.rados.0.smithi045.stdout:767: expect (ObjNum 305 snap 87 seq_num 305) 2024-04-26T00:24:35.801 INFO:tasks.rados.rados.0.smithi045.stdout:766: done (2 left) 2024-04-26T00:24:35.801 INFO:tasks.rados.rados.0.smithi045.stdout:767: done (1 left) 2024-04-26T00:24:35.801 INFO:tasks.rados.rados.0.smithi045.stdout:768: done (0 left) 2024-04-26T00:24:35.801 INFO:tasks.rados.rados.0.smithi045.stdout:769: write (excl) oid 4 current snap is 90 2024-04-26T00:24:35.801 INFO:tasks.rados.rados.0.smithi045.stdout:769: seq_num 308 ranges {776556=630812,1823748=652437,3108517=68039} 2024-04-26T00:24:35.823 INFO:tasks.rados.rados.0.smithi045.stdout:769: writing smithi04526579-4 from 776556 to 1407368 tid 1 2024-04-26T00:24:35.841 INFO:tasks.rados.rados.0.smithi045.stdout:769: writing smithi04526579-4 from 1823748 to 2476185 tid 2 2024-04-26T00:24:35.852 INFO:tasks.rados.rados.0.smithi045.stdout:769: writing smithi04526579-4 from 3108517 to 3176556 tid 3 2024-04-26T00:24:35.852 INFO:tasks.rados.rados.0.smithi045.stdout:770: copy_from oid 39 from oid 34 current snap is 90 2024-04-26T00:24:35.853 INFO:tasks.rados.rados.0.smithi045.stdout:769: finishing write tid 1 to smithi04526579-4 2024-04-26T00:24:35.853 INFO:tasks.rados.rados.0.smithi045.stdout:769: finishing write tid 2 to smithi04526579-4 2024-04-26T00:24:35.853 INFO:tasks.rados.rados.0.smithi045.stdout:771: write oid 12 current snap is 90 2024-04-26T00:24:35.853 INFO:tasks.rados.rados.0.smithi045.stdout:771: seq_num 309 ranges {793363=706061,2282230=796095} 2024-04-26T00:24:35.870 INFO:tasks.rados.rados.0.smithi045.stdout:771: writing smithi04526579-12 from 793363 to 1499424 tid 1 2024-04-26T00:24:35.886 INFO:tasks.rados.rados.0.smithi045.stdout:771: writing smithi04526579-12 from 2282230 to 3078325 tid 2 2024-04-26T00:24:35.886 INFO:tasks.rados.rados.0.smithi045.stdout:772: delete oid 9 current snap is 90 2024-04-26T00:24:35.886 INFO:tasks.rados.rados.0.smithi045.stdout:769: finishing write tid 3 to smithi04526579-4 2024-04-26T00:24:35.887 INFO:tasks.rados.rados.0.smithi045.stdout:770: finishing copy_from to smithi04526579-39 2024-04-26T00:24:35.887 INFO:tasks.rados.rados.0.smithi045.stdout:770: got expected ENOENT (src dne) 2024-04-26T00:24:35.887 INFO:tasks.rados.rados.0.smithi045.stdout:770: finishing copy_from racing read to smithi04526579-39 2024-04-26T00:24:35.887 INFO:tasks.rados.rados.0.smithi045.stdout:769: finishing write tid 4 to smithi04526579-4 2024-04-26T00:24:35.887 INFO:tasks.rados.rados.0.smithi045.stdout:769: finishing write tid 5 to smithi04526579-4 2024-04-26T00:24:35.887 INFO:tasks.rados.rados.0.smithi045.stdout:769: oid 4 updating version 0 to 368 2024-04-26T00:24:35.887 INFO:tasks.rados.rados.0.smithi045.stdout:769: oid 4 updating version 368 to 371 2024-04-26T00:24:35.887 INFO:tasks.rados.rados.0.smithi045.stdout:769: oid 4 version 371 is already newer than 370 2024-04-26T00:24:35.887 INFO:tasks.rados.rados.0.smithi045.stdout:769: oid 4 version 371 is already newer than 369 2024-04-26T00:24:35.887 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 371 (ObjNum 308 snap 90 seq_num 308) dirty exists 2024-04-26T00:24:35.887 INFO:tasks.rados.rados.0.smithi045.stdout:769: left oid 4 (ObjNum 308 snap 90 seq_num 308) 2024-04-26T00:24:35.887 INFO:tasks.rados.rados.0.smithi045.stdout:771: finishing write tid 1 to smithi04526579-12 2024-04-26T00:24:35.895 INFO:tasks.rados.rados.0.smithi045.stdout:771: finishing write tid 2 to smithi04526579-12 2024-04-26T00:24:35.896 INFO:tasks.rados.rados.0.smithi045.stdout:769: done (3 left) 2024-04-26T00:24:35.896 INFO:tasks.rados.rados.0.smithi045.stdout:770: done (2 left) 2024-04-26T00:24:35.896 INFO:tasks.rados.rados.0.smithi045.stdout:772: done (1 left) 2024-04-26T00:24:35.896 INFO:tasks.rados.rados.0.smithi045.stdout:773: rollback oid 50 current snap is 90 2024-04-26T00:24:35.897 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 50 to 71 2024-04-26T00:24:35.897 INFO:tasks.rados.rados.0.smithi045.stdout:771: finishing write tid 3 to smithi04526579-12 2024-04-26T00:24:35.897 INFO:tasks.rados.rados.0.smithi045.stdout:774: read oid 3 snap 82 2024-04-26T00:24:35.897 INFO:tasks.rados.rados.0.smithi045.stdout:774: expect (ObjNum 119 snap 28 seq_num 119) 2024-04-26T00:24:35.897 INFO:tasks.rados.rados.0.smithi045.stdout:775: read oid 13 snap -1 2024-04-26T00:24:35.897 INFO:tasks.rados.rados.0.smithi045.stdout:775: expect (ObjNum 204 snap 53 seq_num 204) 2024-04-26T00:24:35.897 INFO:tasks.rados.rados.0.smithi045.stdout:776: rollback oid 4 current snap is 90 2024-04-26T00:24:35.897 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 4 to 86 2024-04-26T00:24:35.898 INFO:tasks.rados.rados.0.smithi045.stdout:777: read oid 8 snap 86 2024-04-26T00:24:35.898 INFO:tasks.rados.rados.0.smithi045.stdout:777: expect (ObjNum 281 snap 80 seq_num 281) 2024-04-26T00:24:35.898 INFO:tasks.rados.rados.0.smithi045.stdout:778: read oid 20 snap 71 2024-04-26T00:24:35.898 INFO:tasks.rados.rados.0.smithi045.stdout:778: expect (ObjNum 249 snap 64 seq_num 249) 2024-04-26T00:24:35.898 INFO:tasks.rados.rados.0.smithi045.stdout:779: snap_create 2024-04-26T00:24:35.898 INFO:tasks.rados.rados.0.smithi045.stdout:771: finishing write tid 4 to smithi04526579-12 2024-04-26T00:24:35.898 INFO:tasks.rados.rados.0.smithi045.stdout:771: oid 12 updating version 0 to 302 2024-04-26T00:24:35.898 INFO:tasks.rados.rados.0.smithi045.stdout:771: oid 12 updating version 302 to 304 2024-04-26T00:24:35.898 INFO:tasks.rados.rados.0.smithi045.stdout:771: oid 12 version 304 is already newer than 303 2024-04-26T00:24:35.898 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 304 (ObjNum 309 snap 90 seq_num 309) dirty exists 2024-04-26T00:24:35.898 INFO:tasks.rados.rados.0.smithi045.stdout:771: left oid 12 (ObjNum 309 snap 90 seq_num 309) 2024-04-26T00:24:35.908 INFO:tasks.rados.rados.0.smithi045.stdout:773: finishing rollback tid 0 to smithi04526579-50 2024-04-26T00:24:35.911 INFO:tasks.rados.rados.0.smithi045.stdout:774: expect (ObjNum 119 snap 28 seq_num 119) 2024-04-26T00:24:36.039 INFO:tasks.rados.rados.0.smithi045.stdout:773: finishing rollback tid 1 to smithi04526579-50 2024-04-26T00:24:36.040 INFO:tasks.rados.rados.0.smithi045.stdout:773: finishing rollback tid 2 to smithi04526579-50 2024-04-26T00:24:36.040 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 311 (ObjNum 136 snap 30 seq_num 136) dirty exists 2024-04-26T00:24:36.040 INFO:tasks.rados.rados.0.smithi045.stdout:775: expect (ObjNum 204 snap 53 seq_num 204) 2024-04-26T00:24:36.153 INFO:tasks.rados.rados.0.smithi045.stdout:776: finishing rollback tid 0 to smithi04526579-4 2024-04-26T00:24:36.153 INFO:tasks.rados.rados.0.smithi045.stdout:776: finishing rollback tid 1 to smithi04526579-4 2024-04-26T00:24:36.153 INFO:tasks.rados.rados.0.smithi045.stdout:776: finishing rollback tid 2 to smithi04526579-4 2024-04-26T00:24:36.153 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 375 (ObjNum 224 snap 59 seq_num 224) dirty exists 2024-04-26T00:24:36.153 INFO:tasks.rados.rados.0.smithi045.stdout:778: expect (ObjNum 249 snap 64 seq_num 249) 2024-04-26T00:24:36.283 INFO:tasks.rados.rados.0.smithi045.stdout:777: expect (ObjNum 281 snap 80 seq_num 281) 2024-04-26T00:24:36.802 INFO:tasks.rados.rados.0.smithi045.stdout:771: done (7 left) 2024-04-26T00:24:36.802 INFO:tasks.rados.rados.0.smithi045.stdout:773: done (6 left) 2024-04-26T00:24:36.802 INFO:tasks.rados.rados.0.smithi045.stdout:774: done (5 left) 2024-04-26T00:24:36.802 INFO:tasks.rados.rados.0.smithi045.stdout:775: done (4 left) 2024-04-26T00:24:36.802 INFO:tasks.rados.rados.0.smithi045.stdout:776: done (3 left) 2024-04-26T00:24:36.802 INFO:tasks.rados.rados.0.smithi045.stdout:777: done (2 left) 2024-04-26T00:24:36.803 INFO:tasks.rados.rados.0.smithi045.stdout:778: done (1 left) 2024-04-26T00:24:36.803 INFO:tasks.rados.rados.0.smithi045.stdout:779: done (0 left) 2024-04-26T00:24:36.803 INFO:tasks.rados.rados.0.smithi045.stdout:780: read oid 34 snap 71 2024-04-26T00:24:36.803 INFO:tasks.rados.rados.0.smithi045.stdout:780: expect (ObjNum 173 snap 45 seq_num 173) 2024-04-26T00:24:36.803 INFO:tasks.rados.rados.0.smithi045.stdout:781: delete oid 38 current snap is 91 2024-04-26T00:24:36.815 INFO:tasks.rados.rados.0.smithi045.stdout:781: done (1 left) 2024-04-26T00:24:36.815 INFO:tasks.rados.rados.0.smithi045.stdout:782: snap_create 2024-04-26T00:24:36.827 INFO:tasks.rados.rados.0.smithi045.stdout:780: expect (ObjNum 173 snap 45 seq_num 173) 2024-04-26T00:24:36.965 INFO:tasks.rados.rados.0.smithi045.stdout:780: done (1 left) 2024-04-26T00:24:36.965 INFO:tasks.rados.rados.0.smithi045.stdout:782: done (0 left) 2024-04-26T00:24:36.965 INFO:tasks.rados.rados.0.smithi045.stdout:783: write oid 11 current snap is 92 2024-04-26T00:24:36.965 INFO:tasks.rados.rados.0.smithi045.stdout:783: seq_num 312 ranges {443784=648161,1657676=659775,2818209=425575} 2024-04-26T00:24:36.971 INFO:tasks.rados.rados.0.smithi045.stdout:783: writing smithi04526579-11 from 443784 to 1091945 tid 1 2024-04-26T00:24:36.978 INFO:tasks.rados.rados.0.smithi045.stdout:783: writing smithi04526579-11 from 1657676 to 2317451 tid 2 2024-04-26T00:24:36.985 INFO:tasks.rados.rados.0.smithi045.stdout:783: writing smithi04526579-11 from 2818209 to 3243784 tid 3 2024-04-26T00:24:36.985 INFO:tasks.rados.rados.0.smithi045.stdout:784: rollback oid 33 current snap is 92 2024-04-26T00:24:36.985 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 33 to 75 2024-04-26T00:24:36.985 INFO:tasks.rados.rados.0.smithi045.stdout:785: copy_from oid 45 from oid 35 current snap is 92 2024-04-26T00:24:36.985 INFO:tasks.rados.rados.0.smithi045.stdout:786: rollback oid 26 current snap is 92 2024-04-26T00:24:36.985 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 26 to 74 2024-04-26T00:24:36.985 INFO:tasks.rados.rados.0.smithi045.stdout:787: read oid 13 snap -1 2024-04-26T00:24:36.985 INFO:tasks.rados.rados.0.smithi045.stdout:787: expect (ObjNum 204 snap 53 seq_num 204) 2024-04-26T00:24:36.985 INFO:tasks.rados.rados.0.smithi045.stdout:788: snap_remove snap 81 2024-04-26T00:24:37.860 INFO:tasks.rados.rados.0.smithi045.stdout:788: done (5 left) 2024-04-26T00:24:37.860 INFO:tasks.rados.rados.0.smithi045.stdout:789: read oid 22 snap -1 2024-04-26T00:24:37.860 INFO:tasks.rados.rados.0.smithi045.stdout:789: expect deleted 2024-04-26T00:24:37.861 INFO:tasks.rados.rados.0.smithi045.stdout:790: snap_remove snap 63 2024-04-26T00:24:37.861 INFO:tasks.rados.rados.0.smithi045.stdout:786: finishing rollback tid 1 to smithi04526579-26 2024-04-26T00:24:37.861 INFO:tasks.rados.rados.0.smithi045.stdout:786: finishing rollback tid 2 to smithi04526579-26 2024-04-26T00:24:37.861 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 406 (ObjNum 254 snap 64 seq_num 254) dirty exists 2024-04-26T00:24:37.861 INFO:tasks.rados.rados.0.smithi045.stdout:783: finishing write tid 1 to smithi04526579-11 2024-04-26T00:24:38.866 INFO:tasks.rados.rados.0.smithi045.stdout:786: done (6 left) 2024-04-26T00:24:38.867 INFO:tasks.rados.rados.0.smithi045.stdout:790: done (5 left) 2024-04-26T00:24:38.867 INFO:tasks.rados.rados.0.smithi045.stdout:791: write (excl) oid 46 current snap is 92 2024-04-26T00:24:38.867 INFO:tasks.rados.rados.0.smithi045.stdout:791: seq_num 313 ranges {460591=723410,1716158=403433,2729279=440140} 2024-04-26T00:24:38.887 INFO:tasks.rados.rados.0.smithi045.stdout:791: writing smithi04526579-46 from 460591 to 1184001 tid 1 2024-04-26T00:24:38.904 INFO:tasks.rados.rados.0.smithi045.stdout:791: writing smithi04526579-46 from 1716158 to 2119591 tid 2 2024-04-26T00:24:38.922 INFO:tasks.rados.rados.0.smithi045.stdout:791: writing smithi04526579-46 from 2729279 to 3169419 tid 3 2024-04-26T00:24:38.922 INFO:tasks.rados.rados.0.smithi045.stdout:792: write oid 5 current snap is 92 2024-04-26T00:24:38.922 INFO:tasks.rados.rados.0.smithi045.stdout:783: finishing write tid 2 to smithi04526579-11 2024-04-26T00:24:38.922 INFO:tasks.rados.rados.0.smithi045.stdout:784: finishing rollback tid 0 to smithi04526579-33 2024-04-26T00:24:38.922 INFO:tasks.rados.rados.0.smithi045.stdout:783: finishing write tid 3 to smithi04526579-11 2024-04-26T00:24:38.922 INFO:tasks.rados.rados.0.smithi045.stdout:783: finishing write tid 4 to smithi04526579-11 2024-04-26T00:24:38.922 INFO:tasks.rados.rados.0.smithi045.stdout:783: finishing write tid 5 to smithi04526579-11 2024-04-26T00:24:38.922 INFO:tasks.rados.rados.0.smithi045.stdout:783: oid 11 updating version 0 to 307 2024-04-26T00:24:38.922 INFO:tasks.rados.rados.0.smithi045.stdout:783: oid 11 version 307 is already newer than 306 2024-04-26T00:24:38.922 INFO:tasks.rados.rados.0.smithi045.stdout:783: oid 11 version 307 is already newer than 304 2024-04-26T00:24:38.923 INFO:tasks.rados.rados.0.smithi045.stdout:783: oid 11 version 307 is already newer than 305 2024-04-26T00:24:38.923 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 307 (ObjNum 312 snap 92 seq_num 312) dirty exists 2024-04-26T00:24:38.923 INFO:tasks.rados.rados.0.smithi045.stdout:783: left oid 11 (ObjNum 312 snap 92 seq_num 312) 2024-04-26T00:24:38.923 INFO:tasks.rados.rados.0.smithi045.stdout:787: expect (ObjNum 204 snap 53 seq_num 204) 2024-04-26T00:24:39.080 INFO:tasks.rados.rados.0.smithi045.stdout:784: finishing rollback tid 1 to smithi04526579-33 2024-04-26T00:24:39.080 INFO:tasks.rados.rados.0.smithi045.stdout:784: finishing rollback tid 2 to smithi04526579-33 2024-04-26T00:24:39.080 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 510 (ObjNum 246 snap 63 seq_num 246) dirty exists 2024-04-26T00:24:39.080 INFO:tasks.rados.rados.0.smithi045.stdout:785: finishing copy_from racing read to smithi04526579-45 2024-04-26T00:24:39.080 INFO:tasks.rados.rados.0.smithi045.stdout:785: finishing copy_from to smithi04526579-45 2024-04-26T00:24:39.080 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 373 (ObjNum 305 snap 87 seq_num 305) dirty exists 2024-04-26T00:24:39.080 INFO:tasks.rados.rados.0.smithi045.stdout:792: seq_num 314 ranges {477398=798659,1774640=663444,2873055=404343} 2024-04-26T00:24:39.087 INFO:tasks.rados.rados.0.smithi045.stdout:792: writing smithi04526579-5 from 477398 to 1276057 tid 1 2024-04-26T00:24:39.093 INFO:tasks.rados.rados.0.smithi045.stdout:792: writing smithi04526579-5 from 1774640 to 2438084 tid 2 2024-04-26T00:24:39.098 INFO:tasks.rados.rados.0.smithi045.stdout:792: writing smithi04526579-5 from 2873055 to 3277398 tid 3 2024-04-26T00:24:39.099 INFO:tasks.rados.rados.0.smithi045.stdout:783: done (6 left) 2024-04-26T00:24:39.099 INFO:tasks.rados.rados.0.smithi045.stdout:784: done (5 left) 2024-04-26T00:24:39.099 INFO:tasks.rados.rados.0.smithi045.stdout:785: done (4 left) 2024-04-26T00:24:39.099 INFO:tasks.rados.rados.0.smithi045.stdout:787: done (3 left) 2024-04-26T00:24:39.099 INFO:tasks.rados.rados.0.smithi045.stdout:793: snap_remove snap 74 2024-04-26T00:24:39.882 INFO:tasks.rados.rados.0.smithi045.stdout:793: done (3 left) 2024-04-26T00:24:39.882 INFO:tasks.rados.rados.0.smithi045.stdout:794: snap_remove snap 75 2024-04-26T00:24:40.888 INFO:tasks.rados.rados.0.smithi045.stdout:794: done (3 left) 2024-04-26T00:24:40.888 INFO:tasks.rados.rados.0.smithi045.stdout:795: read oid 2 snap 90 2024-04-26T00:24:40.888 INFO:tasks.rados.rados.0.smithi045.stdout:795: expect deleted 2024-04-26T00:24:40.888 INFO:tasks.rados.rados.0.smithi045.stdout:796: delete oid 47 current snap is 92 2024-04-26T00:24:40.888 INFO:tasks.rados.rados.0.smithi045.stdout:791: finishing write tid 1 to smithi04526579-46 2024-04-26T00:24:40.889 INFO:tasks.rados.rados.0.smithi045.stdout:791: finishing write tid 2 to smithi04526579-46 2024-04-26T00:24:40.889 INFO:tasks.rados.rados.0.smithi045.stdout:791: finishing write tid 3 to smithi04526579-46 2024-04-26T00:24:40.889 INFO:tasks.rados.rados.0.smithi045.stdout:791: finishing write tid 4 to smithi04526579-46 2024-04-26T00:24:40.889 INFO:tasks.rados.rados.0.smithi045.stdout:791: finishing write tid 5 to smithi04526579-46 2024-04-26T00:24:40.889 INFO:tasks.rados.rados.0.smithi045.stdout:791: oid 46 updating version 0 to 179 2024-04-26T00:24:40.889 INFO:tasks.rados.rados.0.smithi045.stdout:791: oid 46 version 179 is already newer than 178 2024-04-26T00:24:40.889 INFO:tasks.rados.rados.0.smithi045.stdout:791: oid 46 updating version 179 to 180 2024-04-26T00:24:40.889 INFO:tasks.rados.rados.0.smithi045.stdout:791: oid 46 version 180 is already newer than 177 2024-04-26T00:24:40.889 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 180 (ObjNum 313 snap 92 seq_num 313) dirty exists 2024-04-26T00:24:40.889 INFO:tasks.rados.rados.0.smithi045.stdout:791: left oid 46 (ObjNum 313 snap 92 seq_num 313) 2024-04-26T00:24:40.889 INFO:tasks.rados.rados.0.smithi045.stdout:792: finishing write tid 1 to smithi04526579-5 2024-04-26T00:24:40.889 INFO:tasks.rados.rados.0.smithi045.stdout:792: finishing write tid 2 to smithi04526579-5 2024-04-26T00:24:40.889 INFO:tasks.rados.rados.0.smithi045.stdout:792: finishing write tid 3 to smithi04526579-5 2024-04-26T00:24:40.889 INFO:tasks.rados.rados.0.smithi045.stdout:792: finishing write tid 4 to smithi04526579-5 2024-04-26T00:24:40.889 INFO:tasks.rados.rados.0.smithi045.stdout:792: finishing write tid 5 to smithi04526579-5 2024-04-26T00:24:40.889 INFO:tasks.rados.rados.0.smithi045.stdout:792: oid 5 updating version 0 to 525 2024-04-26T00:24:40.889 INFO:tasks.rados.rados.0.smithi045.stdout:792: oid 5 updating version 525 to 528 2024-04-26T00:24:40.890 INFO:tasks.rados.rados.0.smithi045.stdout:792: oid 5 version 528 is already newer than 527 2024-04-26T00:24:40.890 INFO:tasks.rados.rados.0.smithi045.stdout:792: oid 5 version 528 is already newer than 526 2024-04-26T00:24:40.890 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 528 (ObjNum 314 snap 92 seq_num 314) dirty exists 2024-04-26T00:24:40.890 INFO:tasks.rados.rados.0.smithi045.stdout:792: left oid 5 (ObjNum 314 snap 92 seq_num 314) 2024-04-26T00:24:40.896 INFO:tasks.rados.rados.0.smithi045.stdout:789: done (4 left) 2024-04-26T00:24:40.896 INFO:tasks.rados.rados.0.smithi045.stdout:791: done (3 left) 2024-04-26T00:24:40.896 INFO:tasks.rados.rados.0.smithi045.stdout:792: done (2 left) 2024-04-26T00:24:40.896 INFO:tasks.rados.rados.0.smithi045.stdout:795: done (1 left) 2024-04-26T00:24:40.896 INFO:tasks.rados.rados.0.smithi045.stdout:796: done (0 left) 2024-04-26T00:24:40.896 INFO:tasks.rados.rados.0.smithi045.stdout:797: snap_create 2024-04-26T00:24:41.898 INFO:tasks.rados.rados.0.smithi045.stdout:797: done (0 left) 2024-04-26T00:24:41.899 INFO:tasks.rados.rados.0.smithi045.stdout:798: write oid 23 current snap is 93 2024-04-26T00:24:41.899 INFO:tasks.rados.rados.0.smithi045.stdout:798: seq_num 316 ranges {511012=549157,1775251=550760,3095195=215817} 2024-04-26T00:24:41.912 INFO:tasks.rados.rados.0.smithi045.stdout:798: writing smithi04526579-23 from 511012 to 1060169 tid 1 2024-04-26T00:24:41.929 INFO:tasks.rados.rados.0.smithi045.stdout:798: writing smithi04526579-23 from 1775251 to 2326011 tid 2 2024-04-26T00:24:41.938 INFO:tasks.rados.rados.0.smithi045.stdout:798: writing smithi04526579-23 from 3095195 to 3311012 tid 3 2024-04-26T00:24:41.938 INFO:tasks.rados.rados.0.smithi045.stdout:799: snap_create 2024-04-26T00:24:41.939 INFO:tasks.rados.rados.0.smithi045.stdout:798: finishing write tid 1 to smithi04526579-23 2024-04-26T00:24:41.939 INFO:tasks.rados.rados.0.smithi045.stdout:798: finishing write tid 2 to smithi04526579-23 2024-04-26T00:24:41.944 INFO:tasks.rados.rados.0.smithi045.stdout:798: finishing write tid 3 to smithi04526579-23 2024-04-26T00:24:41.945 INFO:tasks.rados.rados.0.smithi045.stdout:798: finishing write tid 4 to smithi04526579-23 2024-04-26T00:24:41.945 INFO:tasks.rados.rados.0.smithi045.stdout:798: finishing write tid 5 to smithi04526579-23 2024-04-26T00:24:41.946 INFO:tasks.rados.rados.0.smithi045.stdout:798: oid 23 updating version 0 to 305 2024-04-26T00:24:41.946 INFO:tasks.rados.rados.0.smithi045.stdout:798: oid 23 updating version 305 to 307 2024-04-26T00:24:41.946 INFO:tasks.rados.rados.0.smithi045.stdout:798: oid 23 version 307 is already newer than 306 2024-04-26T00:24:41.946 INFO:tasks.rados.rados.0.smithi045.stdout:798: oid 23 updating version 307 to 308 2024-04-26T00:24:41.946 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 308 (ObjNum 316 snap 93 seq_num 316) dirty exists 2024-04-26T00:24:41.946 INFO:tasks.rados.rados.0.smithi045.stdout:798: left oid 23 (ObjNum 316 snap 93 seq_num 316) 2024-04-26T00:24:42.905 INFO:tasks.rados.rados.0.smithi045.stdout:798: done (1 left) 2024-04-26T00:24:42.905 INFO:tasks.rados.rados.0.smithi045.stdout:799: done (0 left) 2024-04-26T00:24:42.905 INFO:tasks.rados.rados.0.smithi045.stdout:800: write (excl) oid 18 current snap is 94 2024-04-26T00:24:42.905 INFO:tasks.rados.rados.0.smithi045.stdout:800: seq_num 317 ranges {527819=624406,1833733=410771,2722618=601581} 2024-04-26T00:24:42.920 INFO:tasks.rados.rados.0.smithi045.stdout:800: writing smithi04526579-18 from 527819 to 1152225 tid 1 2024-04-26T00:24:42.934 INFO:tasks.rados.rados.0.smithi045.stdout:800: writing smithi04526579-18 from 1833733 to 2244504 tid 2 2024-04-26T00:24:42.948 INFO:tasks.rados.rados.0.smithi045.stdout:800: writing smithi04526579-18 from 2722618 to 3324199 tid 3 2024-04-26T00:24:42.948 INFO:tasks.rados.rados.0.smithi045.stdout:801: delete oid 8 current snap is 94 2024-04-26T00:24:42.949 INFO:tasks.rados.rados.0.smithi045.stdout:800: finishing write tid 1 to smithi04526579-18 2024-04-26T00:24:42.949 INFO:tasks.rados.rados.0.smithi045.stdout:800: finishing write tid 2 to smithi04526579-18 2024-04-26T00:24:42.957 INFO:tasks.rados.rados.0.smithi045.stdout:801: done (1 left) 2024-04-26T00:24:42.957 INFO:tasks.rados.rados.0.smithi045.stdout:802: delete oid 33 current snap is 94 2024-04-26T00:24:42.957 INFO:tasks.rados.rados.0.smithi045.stdout:800: finishing write tid 3 to smithi04526579-18 2024-04-26T00:24:42.957 INFO:tasks.rados.rados.0.smithi045.stdout:800: finishing write tid 4 to smithi04526579-18 2024-04-26T00:24:42.958 INFO:tasks.rados.rados.0.smithi045.stdout:800: finishing write tid 5 to smithi04526579-18 2024-04-26T00:24:42.958 INFO:tasks.rados.rados.0.smithi045.stdout:800: oid 18 updating version 0 to 541 2024-04-26T00:24:42.958 INFO:tasks.rados.rados.0.smithi045.stdout:800: oid 18 updating version 541 to 543 2024-04-26T00:24:42.958 INFO:tasks.rados.rados.0.smithi045.stdout:800: oid 18 updating version 543 to 544 2024-04-26T00:24:42.958 INFO:tasks.rados.rados.0.smithi045.stdout:800: oid 18 version 544 is already newer than 542 2024-04-26T00:24:42.958 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 544 (ObjNum 317 snap 94 seq_num 317) dirty exists 2024-04-26T00:24:42.958 INFO:tasks.rados.rados.0.smithi045.stdout:800: left oid 18 (ObjNum 317 snap 94 seq_num 317) 2024-04-26T00:24:42.963 INFO:tasks.rados.rados.0.smithi045.stdout:800: done (1 left) 2024-04-26T00:24:42.963 INFO:tasks.rados.rados.0.smithi045.stdout:802: done (0 left) 2024-04-26T00:24:42.963 INFO:tasks.rados.rados.0.smithi045.stdout:803: read oid 48 snap -1 2024-04-26T00:24:42.963 INFO:tasks.rados.rados.0.smithi045.stdout:803: expect deleted 2024-04-26T00:24:42.963 INFO:tasks.rados.rados.0.smithi045.stdout:804: snap_create 2024-04-26T00:24:43.919 INFO:tasks.rados.rados.0.smithi045.stdout:803: done (1 left) 2024-04-26T00:24:43.919 INFO:tasks.rados.rados.0.smithi045.stdout:804: done (0 left) 2024-04-26T00:24:43.919 INFO:tasks.rados.rados.0.smithi045.stdout:805: delete oid 22 current snap is 95 2024-04-26T00:24:43.926 INFO:tasks.rados.rados.0.smithi045.stdout:805: done (0 left) 2024-04-26T00:24:43.926 INFO:tasks.rados.rados.0.smithi045.stdout:806: snap_remove snap 76 2024-04-26T00:24:44.931 INFO:tasks.rados.rados.0.smithi045.stdout:806: done (0 left) 2024-04-26T00:24:44.931 INFO:tasks.rados.rados.0.smithi045.stdout:807: write oid 28 current snap is 95 2024-04-26T00:24:44.931 INFO:tasks.rados.rados.0.smithi045.stdout:807: seq_num 321 ranges {595047=641755,1667661=418109,2548663=763022} 2024-04-26T00:24:44.954 INFO:tasks.rados.rados.0.smithi045.stdout:807: writing smithi04526579-28 from 595047 to 1236802 tid 1 2024-04-26T00:24:44.969 INFO:tasks.rados.rados.0.smithi045.stdout:807: writing smithi04526579-28 from 1667661 to 2085770 tid 2 2024-04-26T00:24:44.987 INFO:tasks.rados.rados.0.smithi045.stdout:807: writing smithi04526579-28 from 2548663 to 3311685 tid 3 2024-04-26T00:24:44.987 INFO:tasks.rados.rados.0.smithi045.stdout:808: copy_from oid 11 from oid 45 current snap is 95 2024-04-26T00:24:44.987 INFO:tasks.rados.rados.0.smithi045.stdout:807: finishing write tid 1 to smithi04526579-28 2024-04-26T00:24:44.987 INFO:tasks.rados.rados.0.smithi045.stdout:807: finishing write tid 2 to smithi04526579-28 2024-04-26T00:24:44.987 INFO:tasks.rados.rados.0.smithi045.stdout:809: snap_remove snap 85 2024-04-26T00:24:45.938 INFO:tasks.rados.rados.0.smithi045.stdout:809: done (2 left) 2024-04-26T00:24:45.938 INFO:tasks.rados.rados.0.smithi045.stdout:810: write oid 2 current snap is 95 2024-04-26T00:24:45.938 INFO:tasks.rados.rados.0.smithi045.stdout:810: seq_num 322 ranges {611854=717004,2126143=561767,3259733=152121} 2024-04-26T00:24:45.954 INFO:tasks.rados.rados.0.smithi045.stdout:810: writing smithi04526579-2 from 611854 to 1328858 tid 1 2024-04-26T00:24:45.973 INFO:tasks.rados.rados.0.smithi045.stdout:810: writing smithi04526579-2 from 2126143 to 2687910 tid 2 2024-04-26T00:24:45.983 INFO:tasks.rados.rados.0.smithi045.stdout:810: writing smithi04526579-2 from 3259733 to 3411854 tid 3 2024-04-26T00:24:45.983 INFO:tasks.rados.rados.0.smithi045.stdout:811: read oid 6 snap 93 2024-04-26T00:24:45.983 INFO:tasks.rados.rados.0.smithi045.stdout:811: expect (ObjNum 283 snap 81 seq_num 283) 2024-04-26T00:24:45.983 INFO:tasks.rados.rados.0.smithi045.stdout:812: read oid 41 snap -1 2024-04-26T00:24:45.983 INFO:tasks.rados.rados.0.smithi045.stdout:812: expect deleted 2024-04-26T00:24:45.983 INFO:tasks.rados.rados.0.smithi045.stdout:813: delete oid 33 current snap is 95 2024-04-26T00:24:45.983 INFO:tasks.rados.rados.0.smithi045.stdout:807: finishing write tid 3 to smithi04526579-28 2024-04-26T00:24:45.983 INFO:tasks.rados.rados.0.smithi045.stdout:807: finishing write tid 4 to smithi04526579-28 2024-04-26T00:24:45.984 INFO:tasks.rados.rados.0.smithi045.stdout:807: finishing write tid 5 to smithi04526579-28 2024-04-26T00:24:45.984 INFO:tasks.rados.rados.0.smithi045.stdout:807: oid 28 updating version 0 to 451 2024-04-26T00:24:45.984 INFO:tasks.rados.rados.0.smithi045.stdout:807: oid 28 version 451 is already newer than 450 2024-04-26T00:24:45.984 INFO:tasks.rados.rados.0.smithi045.stdout:807: oid 28 version 451 is already newer than 440 2024-04-26T00:24:45.984 INFO:tasks.rados.rados.0.smithi045.stdout:807: oid 28 version 451 is already newer than 449 2024-04-26T00:24:45.984 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 451 (ObjNum 321 snap 95 seq_num 321) dirty exists 2024-04-26T00:24:45.984 INFO:tasks.rados.rados.0.smithi045.stdout:807: left oid 28 (ObjNum 321 snap 95 seq_num 321) 2024-04-26T00:24:45.984 INFO:tasks.rados.rados.0.smithi045.stdout:808: finishing copy_from racing read to smithi04526579-11 2024-04-26T00:24:45.984 INFO:tasks.rados.rados.0.smithi045.stdout:808: finishing copy_from to smithi04526579-11 2024-04-26T00:24:45.984 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 373 (ObjNum 305 snap 87 seq_num 305) dirty exists 2024-04-26T00:24:45.984 INFO:tasks.rados.rados.0.smithi045.stdout:810: finishing write tid 1 to smithi04526579-2 2024-04-26T00:24:45.984 INFO:tasks.rados.rados.0.smithi045.stdout:810: finishing write tid 2 to smithi04526579-2 2024-04-26T00:24:45.987 INFO:tasks.rados.rados.0.smithi045.stdout:807: done (5 left) 2024-04-26T00:24:45.987 INFO:tasks.rados.rados.0.smithi045.stdout:808: done (4 left) 2024-04-26T00:24:45.987 INFO:tasks.rados.rados.0.smithi045.stdout:813: done (3 left) 2024-04-26T00:24:45.987 INFO:tasks.rados.rados.0.smithi045.stdout:814: write oid 49 current snap is 95 2024-04-26T00:24:45.987 INFO:tasks.rados.rados.0.smithi045.stdout:814: seq_num 324 ranges {645468=467502,1726754=565436,2798226=596838} 2024-04-26T00:24:46.000 INFO:tasks.rados.rados.0.smithi045.stdout:814: writing smithi04526579-49 from 645468 to 1112970 tid 1 2024-04-26T00:24:46.013 INFO:tasks.rados.rados.0.smithi045.stdout:814: writing smithi04526579-49 from 1726754 to 2292190 tid 2 2024-04-26T00:24:46.025 INFO:tasks.rados.rados.0.smithi045.stdout:814: writing smithi04526579-49 from 2798226 to 3395064 tid 3 2024-04-26T00:24:46.025 INFO:tasks.rados.rados.0.smithi045.stdout:815: delete oid 3 current snap is 95 2024-04-26T00:24:46.025 INFO:tasks.rados.rados.0.smithi045.stdout:810: finishing write tid 3 to smithi04526579-2 2024-04-26T00:24:46.025 INFO:tasks.rados.rados.0.smithi045.stdout:810: finishing write tid 4 to smithi04526579-2 2024-04-26T00:24:46.025 INFO:tasks.rados.rados.0.smithi045.stdout:810: finishing write tid 5 to smithi04526579-2 2024-04-26T00:24:46.025 INFO:tasks.rados.rados.0.smithi045.stdout:810: oid 2 updating version 0 to 399 2024-04-26T00:24:46.026 INFO:tasks.rados.rados.0.smithi045.stdout:810: oid 2 updating version 399 to 404 2024-04-26T00:24:46.026 INFO:tasks.rados.rados.0.smithi045.stdout:810: oid 2 version 404 is already newer than 402 2024-04-26T00:24:46.026 INFO:tasks.rados.rados.0.smithi045.stdout:810: oid 2 version 404 is already newer than 403 2024-04-26T00:24:46.026 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 404 (ObjNum 322 snap 95 seq_num 322) dirty exists 2024-04-26T00:24:46.026 INFO:tasks.rados.rados.0.smithi045.stdout:810: left oid 2 (ObjNum 322 snap 95 seq_num 322) 2024-04-26T00:24:46.026 INFO:tasks.rados.rados.0.smithi045.stdout:811: expect (ObjNum 283 snap 81 seq_num 283) 2024-04-26T00:24:46.238 INFO:tasks.rados.rados.0.smithi045.stdout:814: finishing write tid 1 to smithi04526579-49 2024-04-26T00:24:46.238 INFO:tasks.rados.rados.0.smithi045.stdout:814: finishing write tid 2 to smithi04526579-49 2024-04-26T00:24:46.238 INFO:tasks.rados.rados.0.smithi045.stdout:814: finishing write tid 3 to smithi04526579-49 2024-04-26T00:24:46.238 INFO:tasks.rados.rados.0.smithi045.stdout:814: finishing write tid 4 to smithi04526579-49 2024-04-26T00:24:46.238 INFO:tasks.rados.rados.0.smithi045.stdout:814: finishing write tid 5 to smithi04526579-49 2024-04-26T00:24:46.238 INFO:tasks.rados.rados.0.smithi045.stdout:814: oid 49 updating version 0 to 412 2024-04-26T00:24:46.239 INFO:tasks.rados.rados.0.smithi045.stdout:814: oid 49 version 412 is already newer than 410 2024-04-26T00:24:46.239 INFO:tasks.rados.rados.0.smithi045.stdout:814: oid 49 version 412 is already newer than 411 2024-04-26T00:24:46.239 INFO:tasks.rados.rados.0.smithi045.stdout:814: oid 49 version 412 is already newer than 409 2024-04-26T00:24:46.239 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 412 (ObjNum 324 snap 95 seq_num 324) dirty exists 2024-04-26T00:24:46.239 INFO:tasks.rados.rados.0.smithi045.stdout:814: left oid 49 (ObjNum 324 snap 95 seq_num 324) 2024-04-26T00:24:46.239 INFO:tasks.rados.rados.0.smithi045.stdout:810: done (4 left) 2024-04-26T00:24:46.239 INFO:tasks.rados.rados.0.smithi045.stdout:811: done (3 left) 2024-04-26T00:24:46.239 INFO:tasks.rados.rados.0.smithi045.stdout:812: done (2 left) 2024-04-26T00:24:46.239 INFO:tasks.rados.rados.0.smithi045.stdout:814: done (1 left) 2024-04-26T00:24:46.239 INFO:tasks.rados.rados.0.smithi045.stdout:815: done (0 left) 2024-04-26T00:24:46.239 INFO:tasks.rados.rados.0.smithi045.stdout:816: read oid 46 snap -1 2024-04-26T00:24:46.239 INFO:tasks.rados.rados.0.smithi045.stdout:816: expect (ObjNum 313 snap 92 seq_num 313) 2024-04-26T00:24:46.239 INFO:tasks.rados.rados.0.smithi045.stdout:817: rollback oid 4 current snap is 95 2024-04-26T00:24:46.239 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 4 to 86 2024-04-26T00:24:46.240 INFO:tasks.rados.rados.0.smithi045.stdout:818: write oid 48 current snap is 95 2024-04-26T00:24:46.240 INFO:tasks.rados.rados.0.smithi045.stdout:818: seq_num 326 ranges {679082=618000,1843718=569105,2853072=626010} 2024-04-26T00:24:46.250 INFO:tasks.rados.rados.0.smithi045.stdout:818: writing smithi04526579-48 from 679082 to 1297082 tid 1 2024-04-26T00:24:46.261 INFO:tasks.rados.rados.0.smithi045.stdout:818: writing smithi04526579-48 from 1843718 to 2412823 tid 2 2024-04-26T00:24:46.271 INFO:tasks.rados.rados.0.smithi045.stdout:818: writing smithi04526579-48 from 2853072 to 3479082 tid 3 2024-04-26T00:24:46.271 INFO:tasks.rados.rados.0.smithi045.stdout:819: rollback oid 41 current snap is 95 2024-04-26T00:24:46.271 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 41 to 94 2024-04-26T00:24:46.271 INFO:tasks.rados.rados.0.smithi045.stdout:817: finishing rollback tid 0 to smithi04526579-4 2024-04-26T00:24:46.271 INFO:tasks.rados.rados.0.smithi045.stdout:817: finishing rollback tid 1 to smithi04526579-4 2024-04-26T00:24:46.271 INFO:tasks.rados.rados.0.smithi045.stdout:820: write (excl) oid 47 current snap is 95 2024-04-26T00:24:46.271 INFO:tasks.rados.rados.0.smithi045.stdout:820: seq_num 327 ranges {695889=409602,1618553=712763,2996848=499041} 2024-04-26T00:24:46.285 INFO:tasks.rados.rados.0.smithi045.stdout:820: writing smithi04526579-47 from 695889 to 1105491 tid 1 2024-04-26T00:24:46.299 INFO:tasks.rados.rados.0.smithi045.stdout:820: writing smithi04526579-47 from 1618553 to 2331316 tid 2 2024-04-26T00:24:46.310 INFO:tasks.rados.rados.0.smithi045.stdout:820: writing smithi04526579-47 from 2996848 to 3495889 tid 3 2024-04-26T00:24:46.310 INFO:tasks.rados.rados.0.smithi045.stdout:821: rollback oid 10 current snap is 95 2024-04-26T00:24:46.310 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 10 to 32 2024-04-26T00:24:46.310 INFO:tasks.rados.rados.0.smithi045.stdout:816: expect (ObjNum 313 snap 92 seq_num 313) 2024-04-26T00:24:46.608 INFO:tasks.rados.rados.0.smithi045.stdout:817: finishing rollback tid 2 to smithi04526579-4 2024-04-26T00:24:46.608 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 408 (ObjNum 224 snap 59 seq_num 224) dirty exists 2024-04-26T00:24:46.608 INFO:tasks.rados.rados.0.smithi045.stdout:818: finishing write tid 1 to smithi04526579-48 2024-04-26T00:24:46.609 INFO:tasks.rados.rados.0.smithi045.stdout:818: finishing write tid 2 to smithi04526579-48 2024-04-26T00:24:46.609 INFO:tasks.rados.rados.0.smithi045.stdout:819: finishing rollback tid 1 to smithi04526579-41 2024-04-26T00:24:46.609 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 374 (ObjNum 129 snap 0 seq_num 2241619216) dirty dne 2024-04-26T00:24:46.609 INFO:tasks.rados.rados.0.smithi045.stdout:818: finishing write tid 3 to smithi04526579-48 2024-04-26T00:24:46.609 INFO:tasks.rados.rados.0.smithi045.stdout:818: finishing write tid 4 to smithi04526579-48 2024-04-26T00:24:46.609 INFO:tasks.rados.rados.0.smithi045.stdout:818: finishing write tid 5 to smithi04526579-48 2024-04-26T00:24:46.609 INFO:tasks.rados.rados.0.smithi045.stdout:818: oid 48 updating version 0 to 415 2024-04-26T00:24:46.609 INFO:tasks.rados.rados.0.smithi045.stdout:818: oid 48 version 415 is already newer than 414 2024-04-26T00:24:46.609 INFO:tasks.rados.rados.0.smithi045.stdout:818: oid 48 updating version 415 to 417 2024-04-26T00:24:46.609 INFO:tasks.rados.rados.0.smithi045.stdout:818: oid 48 version 417 is already newer than 416 2024-04-26T00:24:46.609 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 417 (ObjNum 326 snap 95 seq_num 326) dirty exists 2024-04-26T00:24:46.609 INFO:tasks.rados.rados.0.smithi045.stdout:818: left oid 48 (ObjNum 326 snap 95 seq_num 326) 2024-04-26T00:24:46.609 INFO:tasks.rados.rados.0.smithi045.stdout:820: finishing write tid 1 to smithi04526579-47 2024-04-26T00:24:46.609 INFO:tasks.rados.rados.0.smithi045.stdout:816: done (5 left) 2024-04-26T00:24:46.610 INFO:tasks.rados.rados.0.smithi045.stdout:817: done (4 left) 2024-04-26T00:24:46.610 INFO:tasks.rados.rados.0.smithi045.stdout:818: done (3 left) 2024-04-26T00:24:46.610 INFO:tasks.rados.rados.0.smithi045.stdout:819: done (2 left) 2024-04-26T00:24:46.610 INFO:tasks.rados.rados.0.smithi045.stdout:822: snap_create 2024-04-26T00:24:46.610 INFO:tasks.rados.rados.0.smithi045.stdout:820: finishing write tid 2 to smithi04526579-47 2024-04-26T00:24:46.610 INFO:tasks.rados.rados.0.smithi045.stdout:820: finishing write tid 3 to smithi04526579-47 2024-04-26T00:24:46.610 INFO:tasks.rados.rados.0.smithi045.stdout:820: finishing write tid 4 to smithi04526579-47 2024-04-26T00:24:46.610 INFO:tasks.rados.rados.0.smithi045.stdout:821: finishing rollback tid 0 to smithi04526579-10 2024-04-26T00:24:46.610 INFO:tasks.rados.rados.0.smithi045.stdout:820: finishing write tid 5 to smithi04526579-47 2024-04-26T00:24:46.610 INFO:tasks.rados.rados.0.smithi045.stdout:820: oid 47 updating version 0 to 421 2024-04-26T00:24:46.610 INFO:tasks.rados.rados.0.smithi045.stdout:820: oid 47 version 421 is already newer than 419 2024-04-26T00:24:46.610 INFO:tasks.rados.rados.0.smithi045.stdout:820: oid 47 version 421 is already newer than 418 2024-04-26T00:24:46.610 INFO:tasks.rados.rados.0.smithi045.stdout:820: oid 47 version 421 is already newer than 420 2024-04-26T00:24:46.610 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 421 (ObjNum 327 snap 95 seq_num 327) dirty exists 2024-04-26T00:24:46.611 INFO:tasks.rados.rados.0.smithi045.stdout:820: left oid 47 (ObjNum 327 snap 95 seq_num 327) 2024-04-26T00:24:46.611 INFO:tasks.rados.rados.0.smithi045.stdout:821: finishing rollback tid 1 to smithi04526579-10 2024-04-26T00:24:46.611 INFO:tasks.rados.rados.0.smithi045.stdout:821: finishing rollback tid 2 to smithi04526579-10 2024-04-26T00:24:46.611 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 341 (ObjNum 87 snap 19 seq_num 87) dirty exists 2024-04-26T00:24:46.854 INFO:tasks.rados.rados.0.smithi045.stdout:820: done (2 left) 2024-04-26T00:24:46.854 INFO:tasks.rados.rados.0.smithi045.stdout:821: done (1 left) 2024-04-26T00:24:46.855 INFO:tasks.rados.rados.0.smithi045.stdout:822: done (0 left) 2024-04-26T00:24:46.855 INFO:tasks.rados.rados.0.smithi045.stdout:823: delete oid 46 current snap is 96 2024-04-26T00:24:46.867 INFO:tasks.rados.rados.0.smithi045.stdout:823: done (0 left) 2024-04-26T00:24:46.867 INFO:tasks.rados.rados.0.smithi045.stdout:824: rollback oid 39 current snap is 96 2024-04-26T00:24:46.867 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 39 to 90 2024-04-26T00:24:46.867 INFO:tasks.rados.rados.0.smithi045.stdout:825: read oid 23 snap -1 2024-04-26T00:24:46.867 INFO:tasks.rados.rados.0.smithi045.stdout:825: expect (ObjNum 316 snap 93 seq_num 316) 2024-04-26T00:24:46.867 INFO:tasks.rados.rados.0.smithi045.stdout:826: write oid 46 current snap is 96 2024-04-26T00:24:46.867 INFO:tasks.rados.rados.0.smithi045.stdout:826: seq_num 329 ranges {729503=560100,2019164=716432,3335341=194162} 2024-04-26T00:24:46.883 INFO:tasks.rados.rados.0.smithi045.stdout:826: writing smithi04526579-46 from 729503 to 1289603 tid 1 2024-04-26T00:24:46.900 INFO:tasks.rados.rados.0.smithi045.stdout:826: writing smithi04526579-46 from 2019164 to 2735596 tid 2 2024-04-26T00:24:46.909 INFO:tasks.rados.rados.0.smithi045.stdout:826: writing smithi04526579-46 from 3335341 to 3529503 tid 3 2024-04-26T00:24:46.909 INFO:tasks.rados.rados.0.smithi045.stdout:827: write (excl) oid 22 current snap is 96 2024-04-26T00:24:46.909 INFO:tasks.rados.rados.0.smithi045.stdout:827: seq_num 330 ranges {746310=635349,2077646=576443,3362764=183546} 2024-04-26T00:24:46.924 INFO:tasks.rados.rados.0.smithi045.stdout:827: writing smithi04526579-22 from 746310 to 1381659 tid 1 2024-04-26T00:24:46.935 INFO:tasks.rados.rados.0.smithi045.stdout:827: writing smithi04526579-22 from 2077646 to 2654089 tid 2 2024-04-26T00:24:46.943 INFO:tasks.rados.rados.0.smithi045.stdout:827: writing smithi04526579-22 from 3362764 to 3546310 tid 3 2024-04-26T00:24:46.943 INFO:tasks.rados.rados.0.smithi045.stdout:828: snap_create 2024-04-26T00:24:46.943 INFO:tasks.rados.rados.0.smithi045.stdout:824: finishing rollback tid 0 to smithi04526579-39 2024-04-26T00:24:46.943 INFO:tasks.rados.rados.0.smithi045.stdout:824: finishing rollback tid 1 to smithi04526579-39 2024-04-26T00:24:46.943 INFO:tasks.rados.rados.0.smithi045.stdout:824: finishing rollback tid 2 to smithi04526579-39 2024-04-26T00:24:46.943 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 260 (ObjNum 155 snap 33 seq_num 155) dirty exists 2024-04-26T00:24:46.943 INFO:tasks.rados.rados.0.smithi045.stdout:825: expect (ObjNum 316 snap 93 seq_num 316) 2024-04-26T00:24:47.199 INFO:tasks.rados.rados.0.smithi045.stdout:826: finishing write tid 1 to smithi04526579-46 2024-04-26T00:24:47.199 INFO:tasks.rados.rados.0.smithi045.stdout:826: finishing write tid 2 to smithi04526579-46 2024-04-26T00:24:47.199 INFO:tasks.rados.rados.0.smithi045.stdout:826: finishing write tid 3 to smithi04526579-46 2024-04-26T00:24:47.199 INFO:tasks.rados.rados.0.smithi045.stdout:826: finishing write tid 4 to smithi04526579-46 2024-04-26T00:24:47.199 INFO:tasks.rados.rados.0.smithi045.stdout:826: finishing write tid 5 to smithi04526579-46 2024-04-26T00:24:47.199 INFO:tasks.rados.rados.0.smithi045.stdout:826: oid 46 updating version 0 to 205 2024-04-26T00:24:47.200 INFO:tasks.rados.rados.0.smithi045.stdout:826: oid 46 updating version 205 to 207 2024-04-26T00:24:47.200 INFO:tasks.rados.rados.0.smithi045.stdout:826: oid 46 version 207 is already newer than 206 2024-04-26T00:24:47.200 INFO:tasks.rados.rados.0.smithi045.stdout:826: oid 46 version 207 is already newer than 204 2024-04-26T00:24:47.200 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 207 (ObjNum 329 snap 96 seq_num 329) dirty exists 2024-04-26T00:24:47.200 INFO:tasks.rados.rados.0.smithi045.stdout:826: left oid 46 (ObjNum 329 snap 96 seq_num 329) 2024-04-26T00:24:47.200 INFO:tasks.rados.rados.0.smithi045.stdout:827: finishing write tid 1 to smithi04526579-22 2024-04-26T00:24:47.200 INFO:tasks.rados.rados.0.smithi045.stdout:827: finishing write tid 2 to smithi04526579-22 2024-04-26T00:24:47.200 INFO:tasks.rados.rados.0.smithi045.stdout:827: finishing write tid 3 to smithi04526579-22 2024-04-26T00:24:47.200 INFO:tasks.rados.rados.0.smithi045.stdout:827: finishing write tid 4 to smithi04526579-22 2024-04-26T00:24:47.200 INFO:tasks.rados.rados.0.smithi045.stdout:827: finishing write tid 5 to smithi04526579-22 2024-04-26T00:24:47.200 INFO:tasks.rados.rados.0.smithi045.stdout:827: oid 22 updating version 0 to 211 2024-04-26T00:24:47.200 INFO:tasks.rados.rados.0.smithi045.stdout:827: oid 22 version 211 is already newer than 208 2024-04-26T00:24:47.200 INFO:tasks.rados.rados.0.smithi045.stdout:827: oid 22 version 211 is already newer than 209 2024-04-26T00:24:47.200 INFO:tasks.rados.rados.0.smithi045.stdout:827: oid 22 version 211 is already newer than 210 2024-04-26T00:24:47.201 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 211 (ObjNum 330 snap 96 seq_num 330) dirty exists 2024-04-26T00:24:47.201 INFO:tasks.rados.rados.0.smithi045.stdout:827: left oid 22 (ObjNum 330 snap 96 seq_num 330) 2024-04-26T00:24:47.948 INFO:tasks.rados.rados.0.smithi045.stdout:824: done (4 left) 2024-04-26T00:24:47.948 INFO:tasks.rados.rados.0.smithi045.stdout:825: done (3 left) 2024-04-26T00:24:47.948 INFO:tasks.rados.rados.0.smithi045.stdout:826: done (2 left) 2024-04-26T00:24:47.948 INFO:tasks.rados.rados.0.smithi045.stdout:827: done (1 left) 2024-04-26T00:24:47.948 INFO:tasks.rados.rados.0.smithi045.stdout:828: done (0 left) 2024-04-26T00:24:47.948 INFO:tasks.rados.rados.0.smithi045.stdout:829: snap_create 2024-04-26T00:24:48.949 INFO:tasks.rados.rados.0.smithi045.stdout:829: done (0 left) 2024-04-26T00:24:48.950 INFO:tasks.rados.rados.0.smithi045.stdout:830: delete oid 19 current snap is 98 2024-04-26T00:24:48.962 INFO:tasks.rados.rados.0.smithi045.stdout:830: done (0 left) 2024-04-26T00:24:48.962 INFO:tasks.rados.rados.0.smithi045.stdout:831: snap_create 2024-04-26T00:24:49.963 INFO:tasks.rados.rados.0.smithi045.stdout:831: done (0 left) 2024-04-26T00:24:49.963 INFO:tasks.rados.rados.0.smithi045.stdout:832: delete oid 3 current snap is 99 2024-04-26T00:24:49.972 INFO:tasks.rados.rados.0.smithi045.stdout:832: done (0 left) 2024-04-26T00:24:49.972 INFO:tasks.rados.rados.0.smithi045.stdout:833: snap_remove snap 88 2024-04-26T00:24:50.966 INFO:tasks.rados.rados.0.smithi045.stdout:833: done (0 left) 2024-04-26T00:24:50.967 INFO:tasks.rados.rados.0.smithi045.stdout:834: snap_remove snap 84 2024-04-26T00:24:52.160 INFO:tasks.rados.rados.0.smithi045.stdout:834: done (0 left) 2024-04-26T00:24:52.160 INFO:tasks.rados.rados.0.smithi045.stdout:835: read oid 45 snap -1 2024-04-26T00:24:52.160 INFO:tasks.rados.rados.0.smithi045.stdout:835: expect (ObjNum 305 snap 87 seq_num 305) 2024-04-26T00:24:52.160 INFO:tasks.rados.rados.0.smithi045.stdout:836: read oid 7 snap -1 2024-04-26T00:24:52.160 INFO:tasks.rados.rados.0.smithi045.stdout:836: expect deleted 2024-04-26T00:24:52.161 INFO:tasks.rados.rados.0.smithi045.stdout:837: snap_remove snap 82 2024-04-26T00:24:52.420 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-04-26T00:24:52.420 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-04-26T00:24:52.420 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-26T00:24:52.421 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-26T00:24:52.424 INFO:tasks.ceph.mon.c:Started 2024-04-26T00:24:52.424 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-04-26T00:24:52.424 INFO:tasks.ceph.mon.a:Sent signal 19 2024-04-26T00:24:52.424 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2024-04-26T00:25:07.263 INFO:tasks.rados.rados.0.smithi045.stdout:837: done (2 left) 2024-04-26T00:25:07.264 INFO:tasks.rados.rados.0.smithi045.stdout:838: write (excl) oid 33 current snap is 99 2024-04-26T00:25:07.264 INFO:tasks.rados.rados.0.smithi045.stdout:838: seq_num 333 ranges {796731=461096,1736739=723770,2928680=668051} 2024-04-26T00:25:07.284 INFO:tasks.rados.rados.0.smithi045.stdout:838: writing smithi04526579-33 from 796731 to 1257827 tid 1 2024-04-26T00:25:07.305 INFO:tasks.rados.rados.0.smithi045.stdout:838: writing smithi04526579-33 from 1736739 to 2460509 tid 2 2024-04-26T00:25:07.322 INFO:tasks.rados.rados.0.smithi045.stdout:838: writing smithi04526579-33 from 2928680 to 3596731 tid 3 2024-04-26T00:25:07.322 INFO:tasks.rados.rados.0.smithi045.stdout:839: delete oid 28 current snap is 99 2024-04-26T00:25:07.322 INFO:tasks.rados.rados.0.smithi045.stdout:835: expect (ObjNum 305 snap 87 seq_num 305) 2024-04-26T00:25:07.542 INFO:tasks.rados.rados.0.smithi045.stdout:838: finishing write tid 1 to smithi04526579-33 2024-04-26T00:25:07.543 INFO:tasks.rados.rados.0.smithi045.stdout:838: finishing write tid 2 to smithi04526579-33 2024-04-26T00:25:07.543 INFO:tasks.rados.rados.0.smithi045.stdout:838: finishing write tid 3 to smithi04526579-33 2024-04-26T00:25:07.543 INFO:tasks.rados.rados.0.smithi045.stdout:838: finishing write tid 4 to smithi04526579-33 2024-04-26T00:25:07.543 INFO:tasks.rados.rados.0.smithi045.stdout:838: finishing write tid 5 to smithi04526579-33 2024-04-26T00:25:07.543 INFO:tasks.rados.rados.0.smithi045.stdout:838: oid 33 updating version 0 to 573 2024-04-26T00:25:07.543 INFO:tasks.rados.rados.0.smithi045.stdout:838: oid 33 version 573 is already newer than 572 2024-04-26T00:25:07.543 INFO:tasks.rados.rados.0.smithi045.stdout:838: oid 33 version 573 is already newer than 571 2024-04-26T00:25:07.543 INFO:tasks.rados.rados.0.smithi045.stdout:838: oid 33 version 573 is already newer than 570 2024-04-26T00:25:07.543 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 573 (ObjNum 333 snap 99 seq_num 333) dirty exists 2024-04-26T00:25:07.543 INFO:tasks.rados.rados.0.smithi045.stdout:838: left oid 33 (ObjNum 333 snap 99 seq_num 333) 2024-04-26T00:25:07.544 INFO:tasks.rados.rados.0.smithi045.stdout:835: done (3 left) 2024-04-26T00:25:07.544 INFO:tasks.rados.rados.0.smithi045.stdout:836: done (2 left) 2024-04-26T00:25:07.544 INFO:tasks.rados.rados.0.smithi045.stdout:838: done (1 left) 2024-04-26T00:25:07.544 INFO:tasks.rados.rados.0.smithi045.stdout:839: done (0 left) 2024-04-26T00:25:07.544 INFO:tasks.rados.rados.0.smithi045.stdout:840: write oid 2 current snap is 99 2024-04-26T00:25:07.544 INFO:tasks.rados.rados.0.smithi045.stdout:840: seq_num 335 ranges {430345=727947,1570056=727439,2699879=469889} 2024-04-26T00:25:07.558 INFO:tasks.rados.rados.0.smithi045.stdout:840: writing smithi04526579-2 from 430345 to 1158292 tid 1 2024-04-26T00:25:07.574 INFO:tasks.rados.rados.0.smithi045.stdout:840: writing smithi04526579-2 from 1570056 to 2297495 tid 2 2024-04-26T00:25:07.586 INFO:tasks.rados.rados.0.smithi045.stdout:840: writing smithi04526579-2 from 2699879 to 3169768 tid 3 2024-04-26T00:25:07.586 INFO:tasks.rados.rados.0.smithi045.stdout:841: snap_create 2024-04-26T00:25:07.587 INFO:tasks.rados.rados.0.smithi045.stdout:840: finishing write tid 1 to smithi04526579-2 2024-04-26T00:25:07.587 INFO:tasks.rados.rados.0.smithi045.stdout:840: finishing write tid 2 to smithi04526579-2 2024-04-26T00:25:07.594 INFO:tasks.rados.rados.0.smithi045.stdout:840: finishing write tid 3 to smithi04526579-2 2024-04-26T00:25:07.594 INFO:tasks.rados.rados.0.smithi045.stdout:840: finishing write tid 4 to smithi04526579-2 2024-04-26T00:25:07.595 INFO:tasks.rados.rados.0.smithi045.stdout:840: finishing write tid 5 to smithi04526579-2 2024-04-26T00:25:07.595 INFO:tasks.rados.rados.0.smithi045.stdout:840: oid 2 updating version 0 to 429 2024-04-26T00:25:07.595 INFO:tasks.rados.rados.0.smithi045.stdout:840: oid 2 version 429 is already newer than 428 2024-04-26T00:25:07.596 INFO:tasks.rados.rados.0.smithi045.stdout:840: oid 2 version 429 is already newer than 426 2024-04-26T00:25:07.596 INFO:tasks.rados.rados.0.smithi045.stdout:840: oid 2 version 429 is already newer than 427 2024-04-26T00:25:07.596 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 429 (ObjNum 335 snap 99 seq_num 335) dirty exists 2024-04-26T00:25:07.596 INFO:tasks.rados.rados.0.smithi045.stdout:840: left oid 2 (ObjNum 335 snap 99 seq_num 335) 2024-04-26T00:25:08.526 INFO:tasks.rados.rados.0.smithi045.stdout:840: done (1 left) 2024-04-26T00:25:08.526 INFO:tasks.rados.rados.0.smithi045.stdout:841: done (0 left) 2024-04-26T00:25:08.526 INFO:tasks.rados.rados.0.smithi045.stdout:842: write oid 29 current snap is 100 2024-04-26T00:25:08.526 INFO:tasks.rados.rados.0.smithi045.stdout:842: seq_num 336 ranges {447152=403196,1512185=587450,2727302=681161} 2024-04-26T00:25:08.538 INFO:tasks.rados.rados.0.smithi045.stdout:842: writing smithi04526579-29 from 447152 to 850348 tid 1 2024-04-26T00:25:08.556 INFO:tasks.rados.rados.0.smithi045.stdout:842: writing smithi04526579-29 from 1512185 to 2099635 tid 2 2024-04-26T00:25:08.575 INFO:tasks.rados.rados.0.smithi045.stdout:842: writing smithi04526579-29 from 2727302 to 3408463 tid 3 2024-04-26T00:25:08.575 INFO:tasks.rados.rados.0.smithi045.stdout:843: copy_from oid 34 from oid 19 current snap is 100 2024-04-26T00:25:08.575 INFO:tasks.rados.rados.0.smithi045.stdout:842: finishing write tid 1 to smithi04526579-29 2024-04-26T00:25:08.575 INFO:tasks.rados.rados.0.smithi045.stdout:842: finishing write tid 2 to smithi04526579-29 2024-04-26T00:25:08.575 INFO:tasks.rados.rados.0.smithi045.stdout:844: read oid 27 snap -1 2024-04-26T00:25:08.575 INFO:tasks.rados.rados.0.smithi045.stdout:844: expect (ObjNum 246 snap 63 seq_num 246) 2024-04-26T00:25:08.575 INFO:tasks.rados.rados.0.smithi045.stdout:845: write oid 13 current snap is 100 2024-04-26T00:25:08.575 INFO:tasks.rados.rados.0.smithi045.stdout:845: seq_num 337 ranges {463959=478445,1570667=731108,3038372=492433} 2024-04-26T00:25:08.586 INFO:tasks.rados.rados.0.smithi045.stdout:845: writing smithi04526579-13 from 463959 to 942404 tid 1 2024-04-26T00:25:08.600 INFO:tasks.rados.rados.0.smithi045.stdout:845: writing smithi04526579-13 from 1570667 to 2301775 tid 2 2024-04-26T00:25:08.611 INFO:tasks.rados.rados.0.smithi045.stdout:845: writing smithi04526579-13 from 3038372 to 3530805 tid 3 2024-04-26T00:25:08.611 INFO:tasks.rados.rados.0.smithi045.stdout:846: copy_from oid 18 from oid 12 current snap is 100 2024-04-26T00:25:08.611 INFO:tasks.rados.rados.0.smithi045.stdout:842: finishing write tid 3 to smithi04526579-29 2024-04-26T00:25:08.611 INFO:tasks.rados.rados.0.smithi045.stdout:842: finishing write tid 4 to smithi04526579-29 2024-04-26T00:25:08.611 INFO:tasks.rados.rados.0.smithi045.stdout:842: finishing write tid 5 to smithi04526579-29 2024-04-26T00:25:08.611 INFO:tasks.rados.rados.0.smithi045.stdout:842: oid 29 updating version 0 to 307 2024-04-26T00:25:08.611 INFO:tasks.rados.rados.0.smithi045.stdout:842: oid 29 updating version 307 to 309 2024-04-26T00:25:08.611 INFO:tasks.rados.rados.0.smithi045.stdout:842: oid 29 version 309 is already newer than 306 2024-04-26T00:25:08.611 INFO:tasks.rados.rados.0.smithi045.stdout:842: oid 29 version 309 is already newer than 308 2024-04-26T00:25:08.611 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 309 (ObjNum 336 snap 100 seq_num 336) dirty exists 2024-04-26T00:25:08.612 INFO:tasks.rados.rados.0.smithi045.stdout:842: left oid 29 (ObjNum 336 snap 100 seq_num 336) 2024-04-26T00:25:08.612 INFO:tasks.rados.rados.0.smithi045.stdout:843: finishing copy_from to smithi04526579-34 2024-04-26T00:25:08.612 INFO:tasks.rados.rados.0.smithi045.stdout:843: got expected ENOENT (src dne) 2024-04-26T00:25:08.612 INFO:tasks.rados.rados.0.smithi045.stdout:843: finishing copy_from racing read to smithi04526579-34 2024-04-26T00:25:08.612 INFO:tasks.rados.rados.0.smithi045.stdout:844: expect (ObjNum 246 snap 63 seq_num 246) 2024-04-26T00:25:08.767 INFO:tasks.rados.rados.0.smithi045.stdout:845: finishing write tid 1 to smithi04526579-13 2024-04-26T00:25:08.768 INFO:tasks.rados.rados.0.smithi045.stdout:845: finishing write tid 2 to smithi04526579-13 2024-04-26T00:25:08.768 INFO:tasks.rados.rados.0.smithi045.stdout:845: finishing write tid 3 to smithi04526579-13 2024-04-26T00:25:08.768 INFO:tasks.rados.rados.0.smithi045.stdout:845: finishing write tid 4 to smithi04526579-13 2024-04-26T00:25:08.768 INFO:tasks.rados.rados.0.smithi045.stdout:845: finishing write tid 5 to smithi04526579-13 2024-04-26T00:25:08.768 INFO:tasks.rados.rados.0.smithi045.stdout:845: oid 13 updating version 0 to 311 2024-04-26T00:25:08.768 INFO:tasks.rados.rados.0.smithi045.stdout:845: oid 13 updating version 311 to 313 2024-04-26T00:25:08.768 INFO:tasks.rados.rados.0.smithi045.stdout:845: oid 13 version 313 is already newer than 310 2024-04-26T00:25:08.768 INFO:tasks.rados.rados.0.smithi045.stdout:845: oid 13 version 313 is already newer than 312 2024-04-26T00:25:08.768 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 313 (ObjNum 337 snap 100 seq_num 337) dirty exists 2024-04-26T00:25:08.768 INFO:tasks.rados.rados.0.smithi045.stdout:845: left oid 13 (ObjNum 337 snap 100 seq_num 337) 2024-04-26T00:25:08.768 INFO:tasks.rados.rados.0.smithi045.stdout:846: finishing copy_from racing read to smithi04526579-18 2024-04-26T00:25:08.769 INFO:tasks.rados.rados.0.smithi045.stdout:846: finishing copy_from to smithi04526579-18 2024-04-26T00:25:08.769 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 304 (ObjNum 309 snap 90 seq_num 309) dirty exists 2024-04-26T00:25:08.769 INFO:tasks.rados.rados.0.smithi045.stdout:842: done (4 left) 2024-04-26T00:25:08.769 INFO:tasks.rados.rados.0.smithi045.stdout:843: done (3 left) 2024-04-26T00:25:08.769 INFO:tasks.rados.rados.0.smithi045.stdout:844: done (2 left) 2024-04-26T00:25:08.769 INFO:tasks.rados.rados.0.smithi045.stdout:845: done (1 left) 2024-04-26T00:25:08.769 INFO:tasks.rados.rados.0.smithi045.stdout:846: done (0 left) 2024-04-26T00:25:08.769 INFO:tasks.rados.rados.0.smithi045.stdout:847: snap_create 2024-04-26T00:25:09.529 INFO:tasks.rados.rados.0.smithi045.stdout:847: done (0 left) 2024-04-26T00:25:09.529 INFO:tasks.rados.rados.0.smithi045.stdout:848: delete oid 38 current snap is 101 2024-04-26T00:25:09.542 INFO:tasks.rados.rados.0.smithi045.stdout:848: done (0 left) 2024-04-26T00:25:09.542 INFO:tasks.rados.rados.0.smithi045.stdout:849: delete oid 12 current snap is 101 2024-04-26T00:25:09.555 INFO:tasks.rados.rados.0.smithi045.stdout:849: done (0 left) 2024-04-26T00:25:09.555 INFO:tasks.rados.rados.0.smithi045.stdout:850: write (excl) oid 5 current snap is 101 2024-04-26T00:25:09.555 INFO:tasks.rados.rados.0.smithi045.stdout:850: seq_num 340 ranges {514380=704192,1629760=478435,2604288=442602} 2024-04-26T00:25:09.575 INFO:tasks.rados.rados.0.smithi045.stdout:850: writing smithi04526579-5 from 514380 to 1218572 tid 1 2024-04-26T00:25:09.589 INFO:tasks.rados.rados.0.smithi045.stdout:850: writing smithi04526579-5 from 1629760 to 2108195 tid 2 2024-04-26T00:25:09.604 INFO:tasks.rados.rados.0.smithi045.stdout:850: writing smithi04526579-5 from 2604288 to 3046890 tid 3 2024-04-26T00:25:09.604 INFO:tasks.rados.rados.0.smithi045.stdout:851: snap_create 2024-04-26T00:25:09.604 INFO:tasks.rados.rados.0.smithi045.stdout:850: finishing write tid 1 to smithi04526579-5 2024-04-26T00:25:09.604 INFO:tasks.rados.rados.0.smithi045.stdout:850: finishing write tid 2 to smithi04526579-5 2024-04-26T00:25:09.612 INFO:tasks.rados.rados.0.smithi045.stdout:850: finishing write tid 3 to smithi04526579-5 2024-04-26T00:25:09.612 INFO:tasks.rados.rados.0.smithi045.stdout:850: finishing write tid 4 to smithi04526579-5 2024-04-26T00:25:09.613 INFO:tasks.rados.rados.0.smithi045.stdout:850: finishing write tid 5 to smithi04526579-5 2024-04-26T00:25:09.613 INFO:tasks.rados.rados.0.smithi045.stdout:850: oid 5 updating version 0 to 580 2024-04-26T00:25:09.613 INFO:tasks.rados.rados.0.smithi045.stdout:850: oid 5 version 580 is already newer than 579 2024-04-26T00:25:09.613 INFO:tasks.rados.rados.0.smithi045.stdout:850: oid 5 version 580 is already newer than 577 2024-04-26T00:25:09.613 INFO:tasks.rados.rados.0.smithi045.stdout:850: oid 5 version 580 is already newer than 578 2024-04-26T00:25:09.613 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 580 (ObjNum 340 snap 101 seq_num 340) dirty exists 2024-04-26T00:25:09.613 INFO:tasks.rados.rados.0.smithi045.stdout:850: left oid 5 (ObjNum 340 snap 101 seq_num 340) 2024-04-26T00:25:10.543 INFO:tasks.rados.rados.0.smithi045.stdout:850: done (1 left) 2024-04-26T00:25:10.543 INFO:tasks.rados.rados.0.smithi045.stdout:851: done (0 left) 2024-04-26T00:25:10.543 INFO:tasks.rados.rados.0.smithi045.stdout:852: snap_remove snap 77 2024-04-26T00:25:11.558 INFO:tasks.rados.rados.0.smithi045.stdout:852: done (0 left) 2024-04-26T00:25:11.558 INFO:tasks.rados.rados.0.smithi045.stdout:853: rollback oid 38 current snap is 102 2024-04-26T00:25:11.559 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 38 to 95 2024-04-26T00:25:11.559 INFO:tasks.rados.rados.0.smithi045.stdout:854: snap_remove snap 90 2024-04-26T00:25:11.861 INFO:tasks.rados.rados.0.smithi045.stdout:854: done (1 left) 2024-04-26T00:25:11.862 INFO:tasks.rados.rados.0.smithi045.stdout:855: snap_remove snap 89 2024-04-26T00:25:12.425 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-04-26T00:25:12.426 INFO:tasks.ceph.mon.a:Sent signal 18 2024-04-26T00:25:12.426 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:25:12.426 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:25:12.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75aec77640 10 monclient: get_monmap_and_config 2024-04-26T00:25:12.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75aec77640 10 monclient: build_initial_monmap 2024-04-26T00:25:12.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75aec77640 10 monclient: monmap: 2024-04-26T00:25:12.554 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:12.554 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:12.554 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:25:12.576773+0000 2024-04-26T00:25:12.554 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:25:12.576773+0000 2024-04-26T00:25:12.554 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:12.554 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:12.554 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:25:12.554 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:25:12.554 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:25:12.554 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:25:12.554 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:25:12.554 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:12.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75aec77640 1 Processor -- start 2024-04-26T00:25:12.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75aec77640 1 -- start start 2024-04-26T00:25:12.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75aec77640 10 monclient: init 2024-04-26T00:25:12.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75aec77640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:12.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75aec77640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-04-26T00:25:12.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75aec77640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f75a8136da0 0x7f75a809bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:12.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75aec77640 10 monclient(hunting): picked mon.noname-c con 0x7f75a8136da0 addr v2:172.21.15.45:3300/0 2024-04-26T00:25:12.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75aec77640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f75a809c410 addr v1:172.21.15.37:6789/0 2024-04-26T00:25:12.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75aec77640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f75a8137890 0x7f75a813e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:12.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75aec77640 10 monclient(hunting): picked mon.noname-a con 0x7f75a8137890 addr v2:172.21.15.37:3300/0 2024-04-26T00:25:12.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75aec77640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75a809b4f0 con 0x7f75a809c410 2024-04-26T00:25:12.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75aec77640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:12.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75aec77640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f75a809bb40 con 0x7f75a8137890 2024-04-26T00:25:12.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75aec77640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:12.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75aec77640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f75a809b210 con 0x7f75a8136da0 2024-04-26T00:25:12.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75aec77640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:12.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75aec77640 10 monclient(hunting): authenticate will time out at 1293.882416s 2024-04-26T00:25:12.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75ad214640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f75a8137890 0x7f75a813e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:12.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75ad214640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f75a8137890 0x7f75a813e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36342/0 (socket says 172.21.15.37:36342) 2024-04-26T00:25:12.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75ad214640 1 -- 172.21.15.37:0/413879754 learned_addr learned my addr 172.21.15.37:0/413879754 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:12.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75ad214640 10 monclient(hunting): get_auth_request con 0x7f75a8137890 auth_method 0 2024-04-26T00:25:12.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75ad214640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:12.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75ad214640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:12.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75ad214640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:12.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75ad214640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:12.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75ad214640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:12.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75ad214640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:12.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75aca13640 1 --2- 172.21.15.37:0/413879754 >> v2:172.21.15.45:3300/0 conn(0x7f75a8136da0 0x7f75a809bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:12.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75aca13640 10 monclient(hunting): get_auth_request con 0x7f75a8136da0 auth_method 0 2024-04-26T00:25:12.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75aca13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:12.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75aca13640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:12.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75aca13640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:12.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75aca13640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:12.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75aca13640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:12.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75aca13640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:12.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75aca13640 10 monclient(hunting): handle_auth_done global_id 24415 payload 274 2024-04-26T00:25:12.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75aca13640 1 -- 172.21.15.37:0/413879754 >> v2:172.21.15.37:3300/0 conn(0x7f75a8137890 msgr2=0x7f75a813e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:12.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75aca13640 1 --2- 172.21.15.37:0/413879754 >> v2:172.21.15.37:3300/0 conn(0x7f75a8137890 0x7f75a813e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:12.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75aca13640 1 -- 172.21.15.37:0/413879754 >> v1:172.21.15.37:6789/0 conn(0x7f75a809c410 legacy=0x7f75a8137180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:12.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75aca13640 10 monclient: _finish_hunting 0 2024-04-26T00:25:12.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75aca13640 1 monclient: found mon.noname-c 2024-04-26T00:25:12.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75aca13640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:12.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75aca13640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:25:12.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.573+0000 7f75aca13640 1 -- 172.21.15.37:0/413879754 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75a809b7d0 con 0x7f75a8136da0 2024-04-26T00:25:12.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.577+0000 7f75aca13640 1 -- 172.21.15.37:0/413879754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75a8136da0 msgr2=0x7f75a809bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:12.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.577+0000 7f75aca13640 1 --2- 172.21.15.37:0/413879754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75a8136da0 0x7f75a809bed0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f75a80a58a0 tx=0x7f759801c300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e30b00e042a5e41 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:12.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75a77fe640 1 -- 172.21.15.37:0/413879754 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75980b0070 con 0x7f75a8136da0 2024-04-26T00:25:12.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:12.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75a77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:25:12.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75a77fe640 10 monclient: dump: 2024-04-26T00:25:12.561 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:12.561 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:12.561 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:12.562 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:12.562 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:12.562 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:12.562 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:12.562 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:12.562 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:12.562 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:12.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75a77fe640 10 monclient: _finish_auth 0 2024-04-26T00:25:12.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75a77fe640 10 monclient: _check_auth_tickets 2024-04-26T00:25:12.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:12.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75a77fe640 1 -- 172.21.15.37:0/413879754 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75980ab600 con 0x7f75a8136da0 2024-04-26T00:25:12.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75a77fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:12.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75a77fe640 1 -- 172.21.15.37:0/413879754 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75980ab9d0 con 0x7f75a8136da0 2024-04-26T00:25:12.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:12.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75a77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:25:12.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75a77fe640 10 monclient: dump: 2024-04-26T00:25:12.563 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:12.563 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:12.563 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:12.563 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:12.563 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:12.563 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:12.563 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:12.563 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:12.563 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:12.563 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:12.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aec77640 5 monclient: authenticate success, global_id 24415 2024-04-26T00:25:12.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aec77640 10 monclient: get_monmap_and_config success 2024-04-26T00:25:12.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aec77640 4 set_mon_vals no callback set 2024-04-26T00:25:12.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aec77640 10 monclient: shutdown 2024-04-26T00:25:12.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aec77640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:12.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aec77640 1 -- 172.21.15.37:0/413879754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75a8136da0 msgr2=0x7f75a809bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:12.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aec77640 1 --2- 172.21.15.37:0/413879754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75a8136da0 0x7f75a809bed0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f75a80a58a0 tx=0x7f759801c300 comp rx=0 tx=0).stop 2024-04-26T00:25:12.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aec77640 1 -- 172.21.15.37:0/413879754 shutdown_connections 2024-04-26T00:25:12.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aec77640 1 --2- 172.21.15.37:0/413879754 >> v2:172.21.15.37:3300/0 conn(0x7f75a8137890 0x7f75a813e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:12.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aec77640 1 --2- 172.21.15.37:0/413879754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75a8136da0 0x7f75a809bed0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:12.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aec77640 1 -- 172.21.15.37:0/413879754 >> 172.21.15.37:0/413879754 conn(0x7f75a80a5f00 msgr2=0x7f75a80a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:12.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aec77640 1 -- 172.21.15.37:0/413879754 shutdown_connections 2024-04-26T00:25:12.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aec77640 1 -- 172.21.15.37:0/413879754 wait complete. 2024-04-26T00:25:12.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aec77640 10 monclient: build_initial_monmap 2024-04-26T00:25:12.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aec77640 10 monclient: monmap: 2024-04-26T00:25:12.570 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:12.570 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:25:12.570 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:25:12.570 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:25:12.570 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:12.570 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:25:12.570 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:25:12.570 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:25:12.570 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:25:12.570 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:12.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aec77640 1 Processor -- start 2024-04-26T00:25:12.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aec77640 1 -- start start 2024-04-26T00:25:12.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aec77640 10 monclient: init 2024-04-26T00:25:12.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aec77640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:12.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aec77640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:25:12.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aec77640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f75a809c410 0x7f75a81ce9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:12.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aec77640 10 monclient(hunting): picked mon.noname-c con 0x7f75a809c410 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:12.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aec77640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75a8136da0 0x7f75a8135ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:12.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aec77640 10 monclient(hunting): picked mon.noname-b con 0x7f75a8136da0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:25:12.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aec77640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f75a8137890 0x7f75a813ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:12.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aec77640 10 monclient(hunting): picked mon.noname-a con 0x7f75a8137890 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:25:12.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aec77640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:12.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aec77640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f75a8141470 con 0x7f75a8137890 2024-04-26T00:25:12.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aec77640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:12.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aec77640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f75a81412f0 con 0x7f75a8136da0 2024-04-26T00:25:12.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aec77640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:12.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aec77640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f75a81415f0 con 0x7f75a809c410 2024-04-26T00:25:12.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aec77640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:12.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aec77640 10 monclient(hunting): authenticate will time out at 1293.891112s 2024-04-26T00:25:12.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aca13640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f75a809c410 0x7f75a81ce9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:12.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75ad214640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f75a8137890 0x7f75a813ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:12.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aca13640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f75a809c410 0x7f75a81ce9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:59680/0 (socket says 172.21.15.37:59680) 2024-04-26T00:25:12.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75ad214640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f75a8137890 0x7f75a813ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36344/0 (socket says 172.21.15.37:36344) 2024-04-26T00:25:12.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aca13640 1 -- 172.21.15.37:0/2872343482 learned_addr learned my addr 172.21.15.37:0/2872343482 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:12.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aca13640 10 monclient(hunting): get_auth_request con 0x7f75a809c410 auth_method 0 2024-04-26T00:25:12.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aca13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:12.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aca13640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:12.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aca13640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:12.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75ad214640 10 monclient(hunting): get_auth_request con 0x7f75a8137890 auth_method 0 2024-04-26T00:25:12.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75ad214640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:12.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75ad214640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:12.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75ad214640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:12.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aca13640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:12.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aca13640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:12.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aca13640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:12.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75ad214640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:12.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75ad214640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:12.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75ad214640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:12.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75a7fff640 1 --2- 172.21.15.37:0/2872343482 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75a8136da0 0x7f75a8135ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:12.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75a7fff640 10 monclient(hunting): get_auth_request con 0x7f75a8136da0 auth_method 0 2024-04-26T00:25:12.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:12.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75a7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:12.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:12.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75ad214640 10 monclient(hunting): handle_auth_done global_id 54168 payload 824 2024-04-26T00:25:12.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75ad214640 1 -- 172.21.15.37:0/2872343482 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f75a809c410 msgr2=0x7f75a81ce9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:12.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75ad214640 1 --2- 172.21.15.37:0/2872343482 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f75a809c410 0x7f75a81ce9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:12.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75ad214640 1 -- 172.21.15.37:0/2872343482 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75a8136da0 msgr2=0x7f75a8135ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:12.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75ad214640 1 --2- 172.21.15.37:0/2872343482 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75a8136da0 0x7f75a8135ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:12.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75ad214640 10 monclient: _finish_hunting 0 2024-04-26T00:25:12.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75ad214640 1 monclient: found mon.noname-a 2024-04-26T00:25:12.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75ad214640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:12.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75ad214640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:25:12.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75ad214640 1 -- 172.21.15.37:0/2872343482 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75a80a8760 con 0x7f75a8137890 2024-04-26T00:25:12.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.581+0000 7f75aca13640 1 --2- 172.21.15.37:0/2872343482 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f75a809c410 0x7f75a81ce9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:25:12.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.585+0000 7f75ad214640 1 -- 172.21.15.37:0/2872343482 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f75a8137890 msgr2=0x7f75a813ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:12.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.585+0000 7f75ad214640 1 --2- 172.21.15.37:0/2872343482 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f75a8137890 0x7f75a813ad60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f75a000efc0 tx=0x7f75a000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:12.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.585+0000 7f75a57fa640 1 -- 172.21.15.37:0/2872343482 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a00039a0 con 0x7f75a8137890 2024-04-26T00:25:12.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.585+0000 7f75a57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:12.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.585+0000 7f75a57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:25:12.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.585+0000 7f75a57fa640 10 monclient: dump: 2024-04-26T00:25:12.575 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:12.575 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:12.575 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:12.575 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:12.575 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:12.575 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:12.575 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:12.575 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:12.575 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:12.575 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:12.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.585+0000 7f75a57fa640 10 monclient: _finish_auth 0 2024-04-26T00:25:12.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.585+0000 7f75a57fa640 10 monclient: _check_auth_tickets 2024-04-26T00:25:12.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.585+0000 7f75a57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:12.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.585+0000 7f75a57fa640 1 -- 172.21.15.37:0/2872343482 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75a0003b40 con 0x7f75a8137890 2024-04-26T00:25:12.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.585+0000 7f75aec77640 5 monclient: authenticate success, global_id 54168 2024-04-26T00:25:12.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.585+0000 7f75aec77640 10 monclient: _renew_subs 2024-04-26T00:25:12.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.585+0000 7f75aec77640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:12.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.585+0000 7f75aec77640 1 -- 172.21.15.37:0/2872343482 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75a813ca00 con 0x7f75a8137890 2024-04-26T00:25:12.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.585+0000 7f75a57fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:12.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.585+0000 7f75a57fa640 1 -- 172.21.15.37:0/2872343482 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a0004030 con 0x7f75a8137890 2024-04-26T00:25:12.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.585+0000 7f75a57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:12.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.585+0000 7f75a57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:12.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.585+0000 7f75a57fa640 10 monclient: dump: 2024-04-26T00:25:12.576 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:12.576 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:12.576 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:12.576 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:12.577 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:12.577 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:12.577 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:12.577 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:12.577 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:12.577 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:12.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.585+0000 7f75a67fc640 4 set_mon_vals no callback set 2024-04-26T00:25:12.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.585+0000 7f75aec77640 10 monclient: _renew_subs 2024-04-26T00:25:12.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.585+0000 7f75aec77640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:12.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.585+0000 7f75aec77640 1 -- 172.21.15.37:0/2872343482 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75a813d2e0 con 0x7f75a8137890 2024-04-26T00:25:12.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.585+0000 7f75aec77640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:12.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.585+0000 7f75aec77640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:12.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.585+0000 7f75aec77640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:12.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.585+0000 7f75aec77640 1 -- 172.21.15.37:0/2872343482 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75a8066f40 con 0x7f75a8137890 2024-04-26T00:25:12.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.585+0000 7f75a57fa640 1 -- 172.21.15.37:0/2872343482 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f75a0012050 con 0x7f75a8137890 2024-04-26T00:25:12.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.585+0000 7f75a57fa640 1 --2- 172.21.15.37:0/2872343482 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7588035e80 0x7f7588038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:12.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.585+0000 7f75a57fa640 1 -- 172.21.15.37:0/2872343482 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(204..204 src has 172..204) ==== 4035+0+0 (secure 0 0 0) 0x7f75a0049110 con 0x7f75a8137890 2024-04-26T00:25:12.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.585+0000 7f75aca13640 1 --2- 172.21.15.37:0/2872343482 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7588035e80 0x7f7588038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:12.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.585+0000 7f75aca13640 10 monclient: get_auth_request con 0x7f7588035e80 auth_method 0 2024-04-26T00:25:12.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.589+0000 7f75aca13640 1 --2- 172.21.15.37:0/2872343482 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7588035e80 0x7f7588038340 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7598000c10 tx=0x7f75980a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:12.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.593+0000 7f75a57fa640 1 -- 172.21.15.37:0/2872343482 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f75a0022020 con 0x7f75a8137890 2024-04-26T00:25:12.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.593+0000 7f75a57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:12.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.593+0000 7f75a57fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:25:12.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.817+0000 7f75aec77640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:25:12.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.817+0000 7f75aec77640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:25:12.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.817+0000 7f75aec77640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:12.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.817+0000 7f75aec77640 1 -- 172.21.15.37:0/2872343482 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f75a8075d50 con 0x7f75a8137890 2024-04-26T00:25:12.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.817+0000 7f75a57fa640 1 -- 172.21.15.37:0/2872343482 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f75a0010040 con 0x7f75a8137890 2024-04-26T00:25:12.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.817+0000 7f75a57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:25:12.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.817+0000 7f75a57fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:25:12.800 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:25:12.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.821+0000 7f75aec77640 1 -- 172.21.15.37:0/2872343482 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7588035e80 msgr2=0x7f7588038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:12.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.821+0000 7f75aec77640 1 --2- 172.21.15.37:0/2872343482 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7588035e80 0x7f7588038340 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7598000c10 tx=0x7f75980a6000 comp rx=0 tx=0).stop 2024-04-26T00:25:12.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.821+0000 7f75aec77640 10 monclient: shutdown 2024-04-26T00:25:12.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.821+0000 7f75aec77640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:12.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.821+0000 7f75aec77640 1 -- 172.21.15.37:0/2872343482 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f75a8137890 msgr2=0x7f75a813ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:12.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.821+0000 7f75aec77640 1 --2- 172.21.15.37:0/2872343482 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f75a8137890 0x7f75a813ad60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f75a000efc0 tx=0x7f75a000c490 comp rx=0 tx=0).stop 2024-04-26T00:25:12.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.821+0000 7f75aec77640 1 -- 172.21.15.37:0/2872343482 shutdown_connections 2024-04-26T00:25:12.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.821+0000 7f75aec77640 1 --2- 172.21.15.37:0/2872343482 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7588035e80 0x7f7588038340 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:12.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.821+0000 7f75aec77640 1 --2- 172.21.15.37:0/2872343482 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f75a8137890 0x7f75a813ad60 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:12.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.821+0000 7f75aec77640 1 --2- 172.21.15.37:0/2872343482 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75a8136da0 0x7f75a8135ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:12.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.821+0000 7f75aec77640 1 --2- 172.21.15.37:0/2872343482 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f75a809c410 0x7f75a81ce9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:12.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.821+0000 7f75aec77640 1 -- 172.21.15.37:0/2872343482 >> 172.21.15.37:0/2872343482 conn(0x7f75a80a5f00 msgr2=0x7f75a80a7200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:12.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.821+0000 7f75aec77640 1 -- 172.21.15.37:0/2872343482 shutdown_connections 2024-04-26T00:25:12.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.821+0000 7f75aec77640 1 -- 172.21.15.37:0/2872343482 wait complete. 2024-04-26T00:25:12.819 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:25:12.819 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:25:12.820 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:25:12.871 INFO:tasks.rados.rados.0.smithi045.stdout:855: done (1 left) 2024-04-26T00:25:12.871 INFO:tasks.rados.rados.0.smithi045.stdout:856: delete oid 26 current snap is 102 2024-04-26T00:25:12.871 INFO:tasks.rados.rados.0.smithi045.stdout:853: finishing rollback tid 1 to smithi04526579-38 2024-04-26T00:25:12.871 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 134 (ObjNum 353 snap 0 seq_num 907320880) dirty dne 2024-04-26T00:25:12.886 INFO:tasks.rados.rados.0.smithi045.stdout:853: done (1 left) 2024-04-26T00:25:12.886 INFO:tasks.rados.rados.0.smithi045.stdout:856: done (0 left) 2024-04-26T00:25:12.886 INFO:tasks.rados.rados.0.smithi045.stdout:857: snap_remove snap 66 2024-04-26T00:25:12.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.965+0000 7f766b50d640 10 monclient: get_monmap_and_config 2024-04-26T00:25:12.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.965+0000 7f766b50d640 10 monclient: build_initial_monmap 2024-04-26T00:25:12.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.965+0000 7f766b50d640 10 monclient: monmap: 2024-04-26T00:25:12.945 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:12.945 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:12.945 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:25:12.968076+0000 2024-04-26T00:25:12.945 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:25:12.968076+0000 2024-04-26T00:25:12.945 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:12.945 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:12.946 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:25:12.946 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:25:12.946 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:25:12.946 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:25:12.946 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:25:12.946 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:12.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.965+0000 7f766b50d640 1 Processor -- start 2024-04-26T00:25:12.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.965+0000 7f766b50d640 1 -- start start 2024-04-26T00:25:12.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.965+0000 7f766b50d640 10 monclient: init 2024-04-26T00:25:12.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.965+0000 7f766b50d640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:12.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.965+0000 7f766b50d640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-04-26T00:25:12.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.965+0000 7f766b50d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f766409c8c0 0x7f766409cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:12.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.965+0000 7f766b50d640 10 monclient(hunting): picked mon.noname-a con 0x7f766409c8c0 addr v2:172.21.15.37:3300/0 2024-04-26T00:25:12.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.965+0000 7f766b50d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7664135040 addr v1:172.21.15.37:6789/0 2024-04-26T00:25:12.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.965+0000 7f766b50d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f76641354c0 addr v1:172.21.15.45:6789/0 2024-04-26T00:25:12.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.965+0000 7f766b50d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76640a2980 con 0x7f7664135040 2024-04-26T00:25:12.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.965+0000 7f766b50d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f766409a150 con 0x7f76641354c0 2024-04-26T00:25:12.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.965+0000 7f766b50d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:12.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.965+0000 7f766b50d640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f766406c030 con 0x7f766409c8c0 2024-04-26T00:25:12.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.965+0000 7f766b50d640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:12.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.965+0000 7f766b50d640 10 monclient(hunting): authenticate will time out at 1294.273697s 2024-04-26T00:25:12.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.965+0000 7f76692a9640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f766409c8c0 0x7f766409cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:12.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.965+0000 7f76692a9640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f766409c8c0 0x7f766409cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36366/0 (socket says 172.21.15.37:36366) 2024-04-26T00:25:12.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.965+0000 7f76692a9640 1 -- 172.21.15.37:0/1648659408 learned_addr learned my addr 172.21.15.37:0/1648659408 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:12.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.965+0000 7f76692a9640 10 monclient(hunting): get_auth_request con 0x7f766409c8c0 auth_method 0 2024-04-26T00:25:12.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.965+0000 7f76692a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:12.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.965+0000 7f76692a9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:12.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.965+0000 7f76692a9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:12.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.965+0000 7f76692a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:12.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.965+0000 7f76692a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:12.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.965+0000 7f76692a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:12.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.969+0000 7f7668aa8640 1 -- 172.21.15.37:0/1648659408 >> v1:172.21.15.37:6789/0 conn(0x7f7664135040 legacy=0x7f76640a1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:12.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.969+0000 7f76692a9640 10 monclient(hunting): handle_auth_done global_id 54171 payload 274 2024-04-26T00:25:12.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.969+0000 7f76692a9640 1 -- 172.21.15.37:0/1648659408 >> v1:172.21.15.45:6789/0 conn(0x7f76641354c0 legacy=0x7f76640a2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:12.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.969+0000 7f76692a9640 1 -- 172.21.15.37:0/1648659408 >> v1:172.21.15.37:6789/0 conn(0x7f7664135040 legacy=0x7f76640a1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:12.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.969+0000 7f76692a9640 10 monclient: _finish_hunting 0 2024-04-26T00:25:12.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.969+0000 7f76692a9640 1 monclient: found mon.noname-a 2024-04-26T00:25:12.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.969+0000 7f76692a9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:12.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.969+0000 7f76692a9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:25:12.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.969+0000 7f76692a9640 1 -- 172.21.15.37:0/1648659408 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f766409fd40 con 0x7f766409c8c0 2024-04-26T00:25:12.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f76692a9640 1 -- 172.21.15.37:0/1648659408 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f766409c8c0 msgr2=0x7f766409cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:12.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f76692a9640 1 --2- 172.21.15.37:0/1648659408 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f766409c8c0 0x7f766409cca0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f765c009ed0 tx=0x7f765c01c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4d0dcb8e555e343 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:12.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f765bfff640 1 -- 172.21.15.37:0/1648659408 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f765c0a5770 con 0x7f766409c8c0 2024-04-26T00:25:12.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f765bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:12.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f765bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:25:12.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f765bfff640 10 monclient: dump: 2024-04-26T00:25:12.953 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:12.954 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:12.954 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:12.954 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:12.954 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:12.954 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:12.954 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:12.954 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:12.954 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:12.954 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:12.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f765bfff640 10 monclient: _finish_auth 0 2024-04-26T00:25:12.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f765bfff640 10 monclient: _check_auth_tickets 2024-04-26T00:25:12.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f765bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:12.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f766b50d640 5 monclient: authenticate success, global_id 54171 2024-04-26T00:25:12.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f765bfff640 1 -- 172.21.15.37:0/1648659408 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f765c0a5910 con 0x7f766409c8c0 2024-04-26T00:25:12.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f766b50d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:25:12.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f765bfff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:12.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f766b50d640 10 monclient: get_monmap_and_config success 2024-04-26T00:25:12.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f766b50d640 4 set_mon_vals no callback set 2024-04-26T00:25:12.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f765bfff640 1 -- 172.21.15.37:0/1648659408 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f765c0a5ce0 con 0x7f766409c8c0 2024-04-26T00:25:12.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f765bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:12.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f765bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:12.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f765bfff640 10 monclient: dump: 2024-04-26T00:25:12.955 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:12.955 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:12.955 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:12.955 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:12.955 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:12.955 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:12.955 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:12.955 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:12.956 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:12.956 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:12.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f766b50d640 10 monclient: shutdown 2024-04-26T00:25:12.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f766b50d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:12.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f766b50d640 1 -- 172.21.15.37:0/1648659408 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f766409c8c0 msgr2=0x7f766409cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:12.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f766b50d640 1 --2- 172.21.15.37:0/1648659408 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f766409c8c0 0x7f766409cca0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f765c009ed0 tx=0x7f765c01c9c0 comp rx=0 tx=0).stop 2024-04-26T00:25:12.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f766b50d640 1 -- 172.21.15.37:0/1648659408 shutdown_connections 2024-04-26T00:25:12.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f766b50d640 1 --2- 172.21.15.37:0/1648659408 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f766409c8c0 0x7f766409cca0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:12.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f766b50d640 1 -- 172.21.15.37:0/1648659408 >> 172.21.15.37:0/1648659408 conn(0x7f76640a2c50 msgr2=0x7f76640a3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:12.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f766b50d640 1 -- 172.21.15.37:0/1648659408 shutdown_connections 2024-04-26T00:25:12.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f766b50d640 1 -- 172.21.15.37:0/1648659408 wait complete. 2024-04-26T00:25:12.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f766b50d640 10 monclient: build_initial_monmap 2024-04-26T00:25:12.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f766b50d640 10 monclient: monmap: 2024-04-26T00:25:12.962 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:12.962 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:25:12.962 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:25:12.962 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:25:12.962 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:12.962 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:25:12.962 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:25:12.962 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:25:12.962 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:25:12.962 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:12.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f766b50d640 1 Processor -- start 2024-04-26T00:25:12.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f766b50d640 1 -- start start 2024-04-26T00:25:12.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f766b50d640 10 monclient: init 2024-04-26T00:25:12.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f766b50d640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:12.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f766b50d640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:25:12.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f766b50d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f766409c8c0 0x7f766412e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:12.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f766b50d640 10 monclient(hunting): picked mon.noname-a con 0x7f766409c8c0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:25:12.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f766b50d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7664135040 0x7f7664099500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:12.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f766b50d640 10 monclient(hunting): picked mon.noname-b con 0x7f7664135040 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:25:12.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f766b50d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f76641354c0 0x7f7664099a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:12.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f766b50d640 10 monclient(hunting): picked mon.noname-c con 0x7f76641354c0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:12.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f766b50d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:12.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f766b50d640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f7664140c20 con 0x7f766409c8c0 2024-04-26T00:25:12.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f766b50d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:12.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f766b50d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7664140aa0 con 0x7f7664135040 2024-04-26T00:25:12.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f766b50d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:12.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f766b50d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f7664140da0 con 0x7f76641354c0 2024-04-26T00:25:12.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f766b50d640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:12.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f766b50d640 10 monclient(hunting): authenticate will time out at 1294.282830s 2024-04-26T00:25:12.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f7669aaa640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f76641354c0 0x7f7664099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:12.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f76692a9640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f766409c8c0 0x7f766412e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:12.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f7669aaa640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f76641354c0 0x7f7664099a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:59686/0 (socket says 172.21.15.37:59686) 2024-04-26T00:25:12.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f7669aaa640 1 -- 172.21.15.37:0/4093254131 learned_addr learned my addr 172.21.15.37:0/4093254131 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:12.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f7669aaa640 10 monclient(hunting): get_auth_request con 0x7f76641354c0 auth_method 0 2024-04-26T00:25:12.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f7669aaa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:12.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f7669aaa640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:12.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f7669aaa640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:12.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f76692a9640 10 monclient(hunting): get_auth_request con 0x7f766409c8c0 auth_method 0 2024-04-26T00:25:12.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f76692a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:12.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f76692a9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:12.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f76692a9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:12.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f7669aaa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:12.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f7669aaa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:12.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f7669aaa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:12.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f7668aa8640 1 --2- 172.21.15.37:0/4093254131 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7664135040 0x7f7664099500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:12.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f76692a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:12.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f76692a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:12.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f76692a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:12.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f7668aa8640 10 monclient(hunting): get_auth_request con 0x7f7664135040 auth_method 0 2024-04-26T00:25:12.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f7668aa8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:12.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f7668aa8640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:12.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f7668aa8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:12.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f7669aaa640 10 monclient(hunting): handle_auth_done global_id 64103 payload 824 2024-04-26T00:25:12.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f7669aaa640 1 -- 172.21.15.37:0/4093254131 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7664135040 msgr2=0x7f7664099500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:12.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f7669aaa640 1 --2- 172.21.15.37:0/4093254131 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7664135040 0x7f7664099500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:12.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f7669aaa640 1 -- 172.21.15.37:0/4093254131 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f766409c8c0 msgr2=0x7f766412e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:12.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f7669aaa640 1 --2- 172.21.15.37:0/4093254131 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f766409c8c0 0x7f766412e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:12.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f7669aaa640 10 monclient: _finish_hunting 0 2024-04-26T00:25:12.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f7669aaa640 1 monclient: found mon.noname-c 2024-04-26T00:25:12.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f7669aaa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:12.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f7669aaa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:25:12.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.973+0000 7f7669aaa640 1 -- 172.21.15.37:0/4093254131 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7664072e30 con 0x7f76641354c0 2024-04-26T00:25:12.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f7669aaa640 1 -- 172.21.15.37:0/4093254131 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f76641354c0 msgr2=0x7f7664099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:12.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f7669aaa640 1 --2- 172.21.15.37:0/4093254131 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f76641354c0 0x7f7664099a40 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f766000c390 tx=0x7f766000c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:12.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f7659ffb640 1 -- 172.21.15.37:0/4093254131 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76600119a0 con 0x7f76641354c0 2024-04-26T00:25:12.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f7659ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:12.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f7659ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:25:12.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f7659ffb640 10 monclient: dump: 2024-04-26T00:25:12.967 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:12.967 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:12.967 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:12.967 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:12.967 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:12.967 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:12.967 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:12.967 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:12.967 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:12.967 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:12.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f7659ffb640 10 monclient: _finish_auth 0 2024-04-26T00:25:12.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f7659ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:25:12.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f7659ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:12.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f766b50d640 5 monclient: authenticate success, global_id 64103 2024-04-26T00:25:12.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f7659ffb640 1 -- 172.21.15.37:0/4093254131 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7660017070 con 0x7f76641354c0 2024-04-26T00:25:12.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f766b50d640 10 monclient: _renew_subs 2024-04-26T00:25:12.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f766b50d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:25:12.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f766b50d640 1 -- 172.21.15.37:0/4093254131 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7664099f80 con 0x7f76641354c0 2024-04-26T00:25:12.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f7659ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:12.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f7659ffb640 1 -- 172.21.15.37:0/4093254131 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7660011d80 con 0x7f76641354c0 2024-04-26T00:25:12.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f7659ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:12.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f7659ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:25:12.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f7659ffb640 10 monclient: dump: 2024-04-26T00:25:12.968 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:12.968 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:12.968 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:12.968 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:12.968 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:12.968 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:12.968 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:12.969 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:12.969 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:12.969 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:12.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f765b7fe640 4 set_mon_vals no callback set 2024-04-26T00:25:12.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f766b50d640 10 monclient: _renew_subs 2024-04-26T00:25:12.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f766b50d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:25:12.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f766b50d640 1 -- 172.21.15.37:0/4093254131 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f766412d620 con 0x7f76641354c0 2024-04-26T00:25:12.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f766b50d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:12.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f766b50d640 1 --2- 172.21.15.37:0/4093254131 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7664072400 0x7f766412b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:12.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f766b50d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:12.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f766b50d640 1 -- 172.21.15.37:0/4093254131 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f76640a5d00 con 0x7f7664072400 2024-04-26T00:25:12.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f76692a9640 1 --2- 172.21.15.37:0/4093254131 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7664072400 0x7f766412b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:12.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f76692a9640 10 monclient: get_auth_request con 0x7f7664072400 auth_method 0 2024-04-26T00:25:12.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f76692a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:12.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f76692a9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:12.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f76692a9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:12.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f76692a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:12.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f76692a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:12.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f76692a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:12.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f7659ffb640 1 -- 172.21.15.37:0/4093254131 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f766001c460 con 0x7f76641354c0 2024-04-26T00:25:12.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f76692a9640 10 monclient(hunting): handle_auth_done global_id 54180 payload 824 2024-04-26T00:25:12.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f76692a9640 1 -- 172.21.15.37:0/4093254131 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f766404f510 con 0x7f7664072400 2024-04-26T00:25:12.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f7659ffb640 1 --2- 172.21.15.37:0/4093254131 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7648035ed0 0x7f7648038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:12.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f7659ffb640 1 -- 172.21.15.37:0/4093254131 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(205..205 src has 172..205) ==== 3975+0+0 (secure 0 0 0) 0x7f76600488a0 con 0x7f76641354c0 2024-04-26T00:25:12.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f7668aa8640 1 --2- 172.21.15.37:0/4093254131 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7648035ed0 0x7f7648038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:12.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.977+0000 7f7668aa8640 10 monclient: get_auth_request con 0x7f7648035ed0 auth_method 0 2024-04-26T00:25:12.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.981+0000 7f76692a9640 1 -- 172.21.15.37:0/4093254131 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7664072400 msgr2=0x7f766412b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:12.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.981+0000 7f76692a9640 1 --2- 172.21.15.37:0/4093254131 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7664072400 0x7f766412b210 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f765c0ae000 tx=0x7f765c019760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:12.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.981+0000 7f7659ffb640 1 -- 172.21.15.37:0/4093254131 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f765c0b6800 con 0x7f7664072400 2024-04-26T00:25:12.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.981+0000 7f7659ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:12.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.981+0000 7f7659ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:12.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.981+0000 7f7659ffb640 10 monclient: dump: 2024-04-26T00:25:12.973 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:12.973 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:12.973 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:12.973 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:12.973 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:12.973 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:12.973 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:12.974 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:12.974 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:12.974 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:12.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.981+0000 7f7659ffb640 1 -- 172.21.15.37:0/4093254131 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f765c0b6800 con 0x7f7664072400 2024-04-26T00:25:12.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.981+0000 7f7659ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:12.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.981+0000 7f7659ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:25:12.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.981+0000 7f7659ffb640 1 -- 172.21.15.37:0/4093254131 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7664072400 msgr2=0x7f766412b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:12.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.981+0000 7f7659ffb640 1 --2- 172.21.15.37:0/4093254131 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7664072400 0x7f766412b210 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f765c0ae000 tx=0x7f765c019760 comp rx=0 tx=0).stop 2024-04-26T00:25:12.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.981+0000 7f7659ffb640 1 -- 172.21.15.37:0/4093254131 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7664072400 msgr2=0x7f766412b210 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:25:12.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.981+0000 7f7659ffb640 1 --2- 172.21.15.37:0/4093254131 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7664072400 0x7f766412b210 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f765c0ae000 tx=0x7f765c019760 comp rx=0 tx=0).stop 2024-04-26T00:25:12.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.981+0000 7f7668aa8640 1 --2- 172.21.15.37:0/4093254131 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7648035ed0 0x7f7648038390 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7664072a30 tx=0x7f7654009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:12.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.989+0000 7f766b50d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:25:12.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.989+0000 7f766b50d640 1 --2- 172.21.15.37:0/4093254131 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f766412b750 0x7f766412bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:12.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.989+0000 7f766b50d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:12.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.989+0000 7f766b50d640 1 -- 172.21.15.37:0/4093254131 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f766412c0a0 con 0x7f766412b750 2024-04-26T00:25:12.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.989+0000 7f76692a9640 1 --2- 172.21.15.37:0/4093254131 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f766412b750 0x7f766412bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:12.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.989+0000 7f76692a9640 10 monclient: get_auth_request con 0x7f766412b750 auth_method 0 2024-04-26T00:25:12.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.989+0000 7f76692a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:12.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.989+0000 7f76692a9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:12.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.989+0000 7f76692a9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:12.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.989+0000 7f76692a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:12.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.989+0000 7f76692a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:12.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.989+0000 7f76692a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:12.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.989+0000 7f76692a9640 10 monclient(hunting): handle_auth_done global_id 54183 payload 824 2024-04-26T00:25:12.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.989+0000 7f76692a9640 1 -- 172.21.15.37:0/4093254131 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f766406c9a0 con 0x7f766412b750 2024-04-26T00:25:12.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.989+0000 7f76692a9640 1 -- 172.21.15.37:0/4093254131 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f766412b750 msgr2=0x7f766412bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:12.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.989+0000 7f76692a9640 1 --2- 172.21.15.37:0/4093254131 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f766412b750 0x7f766412bb60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f765c01a5f0 tx=0x7f765c0a7000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:12.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.989+0000 7f7659ffb640 1 -- 172.21.15.37:0/4093254131 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f765c0c9070 con 0x7f766412b750 2024-04-26T00:25:12.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.989+0000 7f7659ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:12.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.989+0000 7f7659ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:12.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.989+0000 7f7659ffb640 10 monclient: dump: 2024-04-26T00:25:12.976 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:12.976 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:12.976 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:12.976 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:12.976 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:12.976 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:12.976 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:12.976 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:12.976 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:12.976 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:12.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.989+0000 7f7659ffb640 1 -- 172.21.15.37:0/4093254131 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4716 (secure 0 0 0) 0x7f765c0c5290 con 0x7f766412b750 2024-04-26T00:25:12.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.989+0000 7f7659ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:25:12.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.989+0000 7f7659ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:25:12.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.989+0000 7f7659ffb640 1 -- 172.21.15.37:0/4093254131 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f766412b750 msgr2=0x7f766412bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:12.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.989+0000 7f7659ffb640 1 --2- 172.21.15.37:0/4093254131 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f766412b750 0x7f766412bb60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f765c01a5f0 tx=0x7f765c0a7000 comp rx=0 tx=0).stop 2024-04-26T00:25:12.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.989+0000 7f7659ffb640 1 -- 172.21.15.37:0/4093254131 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f766412b750 msgr2=0x7f766412bb60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:25:12.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.989+0000 7f7659ffb640 1 --2- 172.21.15.37:0/4093254131 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f766412b750 0x7f766412bb60 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f765c01a5f0 tx=0x7f765c0a7000 comp rx=0 tx=0).stop 2024-04-26T00:25:12.977 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:25:12.977 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:25:12.977 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:25:12.977 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:25:12.977 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 40, 2024-04-26T00:25:12.978 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 76315, 2024-04-26T00:25:12.978 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:25:12.978 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:25:12.978 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:25:12.978 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:25:12.978 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:12.978 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:25:12.978 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:25:12.978 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:25:12.978 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:25:12.978 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:12.978 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:12.978 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:12.978 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:12.979 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:12.979 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:12.979 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:12.979 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:12.979 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:12.979 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:12.979 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:12.979 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:12.980 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:25:12.980 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:25:12.980 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:12.980 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:12.980 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:12.980 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:12.980 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:12.980 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:12.980 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:12.980 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:12.980 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:12.980 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:12.980 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:12.980 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:12.981 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:12.981 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:25:12.981 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:25:12.981 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:25:12.981 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:25:12.981 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:25:12.981 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:25:12.981 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:25:12.981 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:25:12.981 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:25:12.981 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:25:12.981 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:25:12.981 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:25:12.982 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:25:12.982 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:25:12.982 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:25:12.982 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:25:12.982 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:25:12.982 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:12.982 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:12.982 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:12.982 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:12.982 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:12.982 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:12.982 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:12.982 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:12.982 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:12.982 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:12.983 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:12.983 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:12.983 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:25:12.983 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:12.983 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:25:12.983 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:12.983 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:25:12.983 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:25:12.983 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:12.983 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:12.983 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:12.983 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:12.983 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:25:12.984 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:12.984 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:12.984 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:12.984 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:12.984 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:25:12.984 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:12.984 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:12.984 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:12.984 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:12.984 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:25:12.984 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:25:12.984 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:12.984 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:12.984 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:12.985 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:12.985 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:12.985 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:25:12.985 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:25:12.985 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:12.985 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:12.985 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:12.985 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:12.985 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:25:12.985 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:12.985 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:12.985 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:12.985 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:12.985 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:25:12.985 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:12.986 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:12.986 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:12.986 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:12.986 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:25:12.986 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:25:12.986 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:12.986 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:12.986 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:12.986 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:12.986 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:12.986 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:25:12.986 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:25:12.986 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:12.986 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:12.987 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:12.987 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:12.987 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:25:12.987 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:12.987 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:12.987 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:12.987 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:12.987 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:25:12.987 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:12.987 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:12.987 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:12.987 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:12.987 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:25:12.987 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:25:12.988 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:12.988 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:12.988 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:12.988 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:12.988 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:12.988 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:12.988 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:25:12.988 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:25:12.988 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:12.988 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:12.988 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:12.988 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:25:12.988 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:12.988 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:12.989 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:25:12.989 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:12.989 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:12.989 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:12.989 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:25:12.989 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:12.989 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:12.989 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:25:12.989 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:12.989 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:12.989 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:12.989 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:25:12.989 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:12.989 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:12.990 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:12.990 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:25:12.990 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:25:12.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.993+0000 7f766b50d640 1 -- 172.21.15.37:0/4093254131 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7648035ed0 msgr2=0x7f7648038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:12.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.993+0000 7f766b50d640 1 --2- 172.21.15.37:0/4093254131 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7648035ed0 0x7f7648038390 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7664072a30 tx=0x7f7654009040 comp rx=0 tx=0).stop 2024-04-26T00:25:12.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.993+0000 7f766b50d640 10 monclient: shutdown 2024-04-26T00:25:12.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.993+0000 7f766b50d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:12.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.993+0000 7f766b50d640 1 -- 172.21.15.37:0/4093254131 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f76641354c0 msgr2=0x7f7664099a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:12.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.993+0000 7f766b50d640 1 --2- 172.21.15.37:0/4093254131 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f76641354c0 0x7f7664099a40 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f766000c390 tx=0x7f766000c850 comp rx=0 tx=0).stop 2024-04-26T00:25:12.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.993+0000 7f7669aaa640 1 -- 172.21.15.37:0/4093254131 reap_dead start 2024-04-26T00:25:12.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.993+0000 7f7669aaa640 1 -- 172.21.15.37:0/4093254131 reap_dead start 2024-04-26T00:25:12.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.993+0000 7f766b50d640 1 -- 172.21.15.37:0/4093254131 shutdown_connections 2024-04-26T00:25:12.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.993+0000 7f766b50d640 1 -- 172.21.15.37:0/4093254131 >> 172.21.15.37:0/4093254131 conn(0x7f76640a2c50 msgr2=0x7f76640a3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:12.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.993+0000 7f766b50d640 1 -- 172.21.15.37:0/4093254131 shutdown_connections 2024-04-26T00:25:12.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:12.993+0000 7f766b50d640 1 -- 172.21.15.37:0/4093254131 wait complete. 2024-04-26T00:25:12.994 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:25:12.995 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:25:12.995 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:25:13.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.133+0000 7f3f726c2640 10 monclient: get_monmap_and_config 2024-04-26T00:25:13.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.133+0000 7f3f726c2640 10 monclient: build_initial_monmap 2024-04-26T00:25:13.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.133+0000 7f3f726c2640 10 monclient: monmap: 2024-04-26T00:25:13.113 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:13.113 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:13.113 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:25:13.136221+0000 2024-04-26T00:25:13.114 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:25:13.136221+0000 2024-04-26T00:25:13.114 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:13.114 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:13.114 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:25:13.114 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:25:13.114 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:25:13.114 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:25:13.114 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:25:13.114 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:13.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.133+0000 7f3f726c2640 1 Processor -- start 2024-04-26T00:25:13.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.133+0000 7f3f726c2640 1 -- start start 2024-04-26T00:25:13.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.133+0000 7f3f726c2640 10 monclient: init 2024-04-26T00:25:13.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.133+0000 7f3f726c2640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:13.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.133+0000 7f3f726c2640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-04-26T00:25:13.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.133+0000 7f3f726c2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3f6c0a0300 addr v1:172.21.15.37:6789/0 2024-04-26T00:25:13.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.133+0000 7f3f726c2640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f6c09fc50 0x7f3f6c139ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:13.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.133+0000 7f3f726c2640 10 monclient(hunting): picked mon.noname-b con 0x7f3f6c09fc50 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:13.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.133+0000 7f3f726c2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3f6c13a400 addr v1:172.21.15.45:6789/0 2024-04-26T00:25:13.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.133+0000 7f3f726c2640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f6c09c190 con 0x7f3f6c0a0300 2024-04-26T00:25:13.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.133+0000 7f3f726c2640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f6c0a1110 con 0x7f3f6c13a400 2024-04-26T00:25:13.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.133+0000 7f3f726c2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:13.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.133+0000 7f3f726c2640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f3f6c0a16d0 con 0x7f3f6c09fc50 2024-04-26T00:25:13.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.133+0000 7f3f726c2640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:13.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.133+0000 7f3f726c2640 10 monclient(hunting): authenticate will time out at 1294.441956s 2024-04-26T00:25:13.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.133+0000 7f3f6b7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f6c09fc50 0x7f3f6c139ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:13.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.133+0000 7f3f6bfff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f3f6c0a0300 0x7f3f6c0a06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:57212/0 (socket says 172.21.15.37:57212) 2024-04-26T00:25:13.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.133+0000 7f3f6bfff640 1 -- 172.21.15.37:0/1684506211 learned_addr learned my addr 172.21.15.37:0/1684506211 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:25:13.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.133+0000 7f3f6b7fe640 10 monclient(hunting): get_auth_request con 0x7f3f6c09fc50 auth_method 0 2024-04-26T00:25:13.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.133+0000 7f3f6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:13.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.133+0000 7f3f6b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:13.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.133+0000 7f3f6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:13.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.133+0000 7f3f6b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:13.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.133+0000 7f3f6b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:13.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.133+0000 7f3f6b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:13.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.133+0000 7f3f6b7fe640 10 monclient(hunting): handle_auth_done global_id 64106 payload 274 2024-04-26T00:25:13.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.133+0000 7f3f6b7fe640 1 -- 172.21.15.37:0/1684506211 >> v1:172.21.15.45:6789/0 conn(0x7f3f6c13a400 legacy=0x7f3f6c13c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:13.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.133+0000 7f3f6b7fe640 1 -- 172.21.15.37:0/1684506211 >> v1:172.21.15.37:6789/0 conn(0x7f3f6c0a0300 legacy=0x7f3f6c0a06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:13.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.133+0000 7f3f6b7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:25:13.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.133+0000 7f3f6b7fe640 1 monclient: found mon.noname-b 2024-04-26T00:25:13.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.133+0000 7f3f6b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:13.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.133+0000 7f3f6b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:25:13.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.133+0000 7f3f6b7fe640 1 -- 172.21.15.37:0/1684506211 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f6c09f2a0 con 0x7f3f6c09fc50 2024-04-26T00:25:13.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.137+0000 7f3f6b7fe640 1 -- 172.21.15.37:0/1684506211 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f6c09fc50 msgr2=0x7f3f6c139ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:13.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.137+0000 7f3f6b7fe640 1 --2- 172.21.15.37:0/1684506211 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f6c09fc50 0x7f3f6c139ec0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3f54009820 tx=0x7f3f5401c300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c17ad7ea4a027e97 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:13.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.137+0000 7f3f6affd640 1 -- 172.21.15.37:0/1684506211 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f540b0070 con 0x7f3f6c09fc50 2024-04-26T00:25:13.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.137+0000 7f3f6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:13.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.137+0000 7f3f6affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:25:13.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.137+0000 7f3f6affd640 10 monclient: dump: 2024-04-26T00:25:13.120 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:13.120 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:13.120 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:13.120 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:13.120 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:13.120 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:13.120 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:13.120 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:13.121 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:13.121 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:13.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.137+0000 7f3f6affd640 10 monclient: _finish_auth 0 2024-04-26T00:25:13.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.137+0000 7f3f6affd640 10 monclient: _check_auth_tickets 2024-04-26T00:25:13.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.137+0000 7f3f6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:13.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.137+0000 7f3f6affd640 1 -- 172.21.15.37:0/1684506211 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f540ab600 con 0x7f3f6c09fc50 2024-04-26T00:25:13.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.137+0000 7f3f6affd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:13.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.137+0000 7f3f6affd640 1 -- 172.21.15.37:0/1684506211 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f540a4e60 con 0x7f3f6c09fc50 2024-04-26T00:25:13.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.137+0000 7f3f726c2640 5 monclient: authenticate success, global_id 64106 2024-04-26T00:25:13.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.137+0000 7f3f726c2640 10 monclient: get_monmap_and_config success 2024-04-26T00:25:13.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.137+0000 7f3f726c2640 4 set_mon_vals no callback set 2024-04-26T00:25:13.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.137+0000 7f3f6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:13.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.137+0000 7f3f6affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:25:13.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.137+0000 7f3f6affd640 10 monclient: dump: 2024-04-26T00:25:13.122 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:13.122 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:13.122 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:13.122 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:13.122 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:13.122 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:13.122 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:13.123 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:13.123 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:13.123 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:13.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.137+0000 7f3f726c2640 10 monclient: shutdown 2024-04-26T00:25:13.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.137+0000 7f3f726c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:13.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.137+0000 7f3f726c2640 1 -- 172.21.15.37:0/1684506211 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f6c09fc50 msgr2=0x7f3f6c139ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:13.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.137+0000 7f3f726c2640 1 --2- 172.21.15.37:0/1684506211 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f6c09fc50 0x7f3f6c139ec0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3f54009820 tx=0x7f3f5401c300 comp rx=0 tx=0).stop 2024-04-26T00:25:13.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.137+0000 7f3f726c2640 1 -- 172.21.15.37:0/1684506211 shutdown_connections 2024-04-26T00:25:13.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.137+0000 7f3f726c2640 1 --2- 172.21.15.37:0/1684506211 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f6c09fc50 0x7f3f6c139ec0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:13.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.137+0000 7f3f726c2640 1 -- 172.21.15.37:0/1684506211 >> 172.21.15.37:0/1684506211 conn(0x7f3f6c0a1d50 msgr2=0x7f3f6c0a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:13.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.137+0000 7f3f726c2640 1 -- 172.21.15.37:0/1684506211 shutdown_connections 2024-04-26T00:25:13.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f726c2640 1 -- 172.21.15.37:0/1684506211 wait complete. 2024-04-26T00:25:13.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f726c2640 10 monclient: build_initial_monmap 2024-04-26T00:25:13.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f726c2640 10 monclient: monmap: 2024-04-26T00:25:13.130 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:13.130 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:25:13.130 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:25:13.130 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:25:13.130 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:13.130 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:25:13.130 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:25:13.131 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:25:13.131 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:25:13.131 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:13.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f726c2640 1 Processor -- start 2024-04-26T00:25:13.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f726c2640 1 -- start start 2024-04-26T00:25:13.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f726c2640 10 monclient: init 2024-04-26T00:25:13.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f726c2640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:13.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f726c2640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:25:13.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f726c2640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f6c09fc50 0x7f3f6c134c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:13.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f726c2640 10 monclient(hunting): picked mon.noname-c con 0x7f3f6c09fc50 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:13.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f726c2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f6c0a0300 0x7f3f6c1cef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:13.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f726c2640 10 monclient(hunting): picked mon.noname-b con 0x7f3f6c0a0300 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:25:13.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f726c2640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f6c13a400 0x7f3f6c1cf4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:13.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f726c2640 10 monclient(hunting): picked mon.noname-a con 0x7f3f6c13a400 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:25:13.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f726c2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:13.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f726c2640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f3f6c140b00 con 0x7f3f6c13a400 2024-04-26T00:25:13.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f726c2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:13.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f726c2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3f6c140980 con 0x7f3f6c0a0300 2024-04-26T00:25:13.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f726c2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:13.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f726c2640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f3f6c140c80 con 0x7f3f6c09fc50 2024-04-26T00:25:13.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f726c2640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:13.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f726c2640 10 monclient(hunting): authenticate will time out at 1294.449902s 2024-04-26T00:25:13.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f6bfff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f6c09fc50 0x7f3f6c134c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:13.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f70c5f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f6c13a400 0x7f3f6c1cf4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:13.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f6bfff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f6c09fc50 0x7f3f6c134c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:59714/0 (socket says 172.21.15.37:59714) 2024-04-26T00:25:13.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f6bfff640 1 -- 172.21.15.37:0/3749799377 learned_addr learned my addr 172.21.15.37:0/3749799377 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:13.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f70c5f640 10 monclient(hunting): get_auth_request con 0x7f3f6c13a400 auth_method 0 2024-04-26T00:25:13.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f70c5f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:13.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f70c5f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:13.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f70c5f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:13.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f6bfff640 10 monclient(hunting): get_auth_request con 0x7f3f6c09fc50 auth_method 0 2024-04-26T00:25:13.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:13.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f6bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:13.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:13.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f70c5f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:13.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f70c5f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:13.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f70c5f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:13.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:13.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:13.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:13.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f6b7fe640 1 --2- 172.21.15.37:0/3749799377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f6c0a0300 0x7f3f6c1cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:13.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f6b7fe640 10 monclient(hunting): get_auth_request con 0x7f3f6c0a0300 auth_method 0 2024-04-26T00:25:13.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:13.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f6b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:13.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:13.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f6bfff640 10 monclient(hunting): handle_auth_done global_id 64109 payload 824 2024-04-26T00:25:13.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f6bfff640 1 -- 172.21.15.37:0/3749799377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f6c0a0300 msgr2=0x7f3f6c1cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:13.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f6bfff640 1 --2- 172.21.15.37:0/3749799377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f6c0a0300 0x7f3f6c1cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:13.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f6bfff640 1 -- 172.21.15.37:0/3749799377 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f6c13a400 msgr2=0x7f3f6c1cf4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:13.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f6bfff640 1 --2- 172.21.15.37:0/3749799377 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f6c13a400 0x7f3f6c1cf4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:13.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f6bfff640 10 monclient: _finish_hunting 0 2024-04-26T00:25:13.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f6bfff640 1 monclient: found mon.noname-c 2024-04-26T00:25:13.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:13.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f6bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:25:13.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f6bfff640 1 -- 172.21.15.37:0/3749799377 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f6c071760 con 0x7f3f6c09fc50 2024-04-26T00:25:13.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.141+0000 7f3f70c5f640 1 --2- 172.21.15.37:0/3749799377 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f6c13a400 0x7f3f6c1cf4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:25:13.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f6bfff640 1 -- 172.21.15.37:0/3749799377 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f6c09fc50 msgr2=0x7f3f6c134c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:13.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f6bfff640 1 --2- 172.21.15.37:0/3749799377 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f6c09fc50 0x7f3f6c134c10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c00cc40 tx=0x7f3f5c00a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:13.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f68ff9640 1 -- 172.21.15.37:0/3749799377 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f5c012400 con 0x7f3f6c09fc50 2024-04-26T00:25:13.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f68ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:13.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f68ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:25:13.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f68ff9640 10 monclient: dump: 2024-04-26T00:25:13.135 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:13.135 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:13.135 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:13.136 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:13.136 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:13.136 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:13.136 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:13.136 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:13.136 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:13.136 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:13.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f68ff9640 10 monclient: _finish_auth 0 2024-04-26T00:25:13.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f68ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:25:13.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f68ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:13.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f68ff9640 1 -- 172.21.15.37:0/3749799377 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f5c0125a0 con 0x7f3f6c09fc50 2024-04-26T00:25:13.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f68ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:13.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f68ff9640 1 -- 172.21.15.37:0/3749799377 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f5c012990 con 0x7f3f6c09fc50 2024-04-26T00:25:13.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f726c2640 5 monclient: authenticate success, global_id 64109 2024-04-26T00:25:13.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f726c2640 10 monclient: _renew_subs 2024-04-26T00:25:13.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f726c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:25:13.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f726c2640 1 -- 172.21.15.37:0/3749799377 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f6c11fff0 con 0x7f3f6c09fc50 2024-04-26T00:25:13.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f6a7fc640 4 set_mon_vals no callback set 2024-04-26T00:25:13.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f68ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:13.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f68ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:25:13.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f68ff9640 10 monclient: dump: 2024-04-26T00:25:13.137 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:13.137 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:13.137 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:13.137 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:13.137 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:13.137 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:13.138 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:13.138 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:13.138 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:13.138 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:13.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f726c2640 10 monclient: _renew_subs 2024-04-26T00:25:13.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f726c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:25:13.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f726c2640 1 -- 172.21.15.37:0/3749799377 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f6c09daf0 con 0x7f3f6c09fc50 2024-04-26T00:25:13.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f726c2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:13.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f726c2640 1 --2- 172.21.15.37:0/3749799377 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f6c070d30 0x7f3f6c02d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:13.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f726c2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:13.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f726c2640 1 -- 172.21.15.37:0/3749799377 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f3f6c13aa00 con 0x7f3f6c070d30 2024-04-26T00:25:13.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f68ff9640 1 -- 172.21.15.37:0/3749799377 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f3f5c012bb0 con 0x7f3f6c09fc50 2024-04-26T00:25:13.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f6b7fe640 1 --2- 172.21.15.37:0/3749799377 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f6c070d30 0x7f3f6c02d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:13.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f6b7fe640 10 monclient: get_auth_request con 0x7f3f6c070d30 auth_method 0 2024-04-26T00:25:13.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:13.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f6b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:13.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:13.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f68ff9640 1 --2- 172.21.15.37:0/3749799377 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3f4c035ed0 0x7f3f4c038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:13.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f6b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:13.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f6b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:13.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f6b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:13.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f68ff9640 1 -- 172.21.15.37:0/3749799377 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(205..205 src has 172..205) ==== 3975+0+0 (secure 0 0 0) 0x7f3f5c0491e0 con 0x7f3f6c09fc50 2024-04-26T00:25:13.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f6b7fe640 10 monclient(hunting): handle_auth_done global_id 64112 payload 824 2024-04-26T00:25:13.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f6b7fe640 1 -- 172.21.15.37:0/3749799377 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3f6c04f510 con 0x7f3f6c070d30 2024-04-26T00:25:13.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f70c5f640 1 --2- 172.21.15.37:0/3749799377 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3f4c035ed0 0x7f3f4c038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:13.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.145+0000 7f3f70c5f640 10 monclient: get_auth_request con 0x7f3f4c035ed0 auth_method 0 2024-04-26T00:25:13.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.149+0000 7f3f6b7fe640 1 -- 172.21.15.37:0/3749799377 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f6c070d30 msgr2=0x7f3f6c02d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:13.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.149+0000 7f3f6b7fe640 1 --2- 172.21.15.37:0/3749799377 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f6c070d30 0x7f3f6c02d180 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c0764d0 tx=0x7f3f5401ab40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:13.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.149+0000 7f3f68ff9640 1 -- 172.21.15.37:0/3749799377 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f540ba070 con 0x7f3f6c070d30 2024-04-26T00:25:13.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.149+0000 7f3f68ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:13.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.149+0000 7f3f68ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:25:13.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.149+0000 7f3f68ff9640 10 monclient: dump: 2024-04-26T00:25:13.142 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:13.142 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:13.142 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:13.142 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:13.142 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:13.142 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:13.142 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:13.142 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:13.142 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:13.142 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:13.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.149+0000 7f3f68ff9640 1 -- 172.21.15.37:0/3749799377 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3f540be400 con 0x7f3f6c070d30 2024-04-26T00:25:13.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.149+0000 7f3f68ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:13.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.149+0000 7f3f68ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:25:13.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.149+0000 7f3f70c5f640 1 --2- 172.21.15.37:0/3749799377 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3f4c035ed0 0x7f3f4c038390 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3f64007fd0 tx=0x7f3f64007dc0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:13.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.149+0000 7f3f68ff9640 1 -- 172.21.15.37:0/3749799377 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f6c070d30 msgr2=0x7f3f6c02d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:13.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.149+0000 7f3f68ff9640 1 --2- 172.21.15.37:0/3749799377 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f6c070d30 0x7f3f6c02d180 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c0764d0 tx=0x7f3f5401ab40 comp rx=0 tx=0).stop 2024-04-26T00:25:13.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.149+0000 7f3f68ff9640 1 -- 172.21.15.37:0/3749799377 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f6c070d30 msgr2=0x7f3f6c02d180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:25:13.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.149+0000 7f3f68ff9640 1 --2- 172.21.15.37:0/3749799377 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f6c070d30 0x7f3f6c02d180 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c0764d0 tx=0x7f3f5401ab40 comp rx=0 tx=0).stop 2024-04-26T00:25:13.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.153+0000 7f3f726c2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:25:13.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.153+0000 7f3f726c2640 1 --2- 172.21.15.37:0/3749799377 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f6c135150 0x7f3f6c1cf9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:13.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.153+0000 7f3f726c2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:13.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.153+0000 7f3f726c2640 1 -- 172.21.15.37:0/3749799377 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f3f6c1393b0 con 0x7f3f6c135150 2024-04-26T00:25:13.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.153+0000 7f3f6b7fe640 1 --2- 172.21.15.37:0/3749799377 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f6c135150 0x7f3f6c1cf9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:13.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.153+0000 7f3f6b7fe640 10 monclient: get_auth_request con 0x7f3f6c135150 auth_method 0 2024-04-26T00:25:13.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.153+0000 7f3f6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:13.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.153+0000 7f3f6b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:13.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.153+0000 7f3f6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:13.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.153+0000 7f3f6b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:13.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.153+0000 7f3f6b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:13.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.153+0000 7f3f6b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:13.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.153+0000 7f3f6b7fe640 10 monclient(hunting): handle_auth_done global_id 64115 payload 824 2024-04-26T00:25:13.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.157+0000 7f3f6b7fe640 1 -- 172.21.15.37:0/3749799377 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3f6c120510 con 0x7f3f6c135150 2024-04-26T00:25:13.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.157+0000 7f3f6b7fe640 1 -- 172.21.15.37:0/3749799377 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f6c135150 msgr2=0x7f3f6c1cf9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:13.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.157+0000 7f3f6b7fe640 1 --2- 172.21.15.37:0/3749799377 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f6c135150 0x7f3f6c1cf9e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3f54009740 tx=0x7f3f540190a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:13.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.157+0000 7f3f68ff9640 1 -- 172.21.15.37:0/3749799377 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f540ba040 con 0x7f3f6c135150 2024-04-26T00:25:13.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.157+0000 7f3f68ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:13.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.157+0000 7f3f68ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:25:13.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.157+0000 7f3f68ff9640 10 monclient: dump: 2024-04-26T00:25:13.144 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:13.145 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:13.145 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:13.145 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:13.145 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:13.145 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:13.145 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:13.145 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:13.145 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:13.145 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:13.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.157+0000 7f3f68ff9640 1 -- 172.21.15.37:0/3749799377 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4716 (secure 0 0 0) 0x7f3f540c83c0 con 0x7f3f6c135150 2024-04-26T00:25:13.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.157+0000 7f3f68ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:25:13.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.157+0000 7f3f68ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:25:13.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.157+0000 7f3f68ff9640 1 -- 172.21.15.37:0/3749799377 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f6c135150 msgr2=0x7f3f6c1cf9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:13.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.157+0000 7f3f68ff9640 1 --2- 172.21.15.37:0/3749799377 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f6c135150 0x7f3f6c1cf9e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3f54009740 tx=0x7f3f540190a0 comp rx=0 tx=0).stop 2024-04-26T00:25:13.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.157+0000 7f3f68ff9640 1 -- 172.21.15.37:0/3749799377 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f6c135150 msgr2=0x7f3f6c1cf9e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:25:13.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.157+0000 7f3f68ff9640 1 --2- 172.21.15.37:0/3749799377 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f6c135150 0x7f3f6c1cf9e0 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3f54009740 tx=0x7f3f540190a0 comp rx=0 tx=0).stop 2024-04-26T00:25:13.146 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:25:13.146 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:25:13.146 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:25:13.146 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:25:13.146 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 40, 2024-04-26T00:25:13.146 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 20627, 2024-04-26T00:25:13.147 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:25:13.147 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:25:13.147 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:25:13.147 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:25:13.147 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:13.147 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:25:13.147 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:25:13.147 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:25:13.147 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:25:13.147 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:13.147 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:13.147 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:13.147 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:13.148 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:13.148 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:13.148 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:13.148 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:13.148 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:13.148 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:13.148 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:13.148 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:13.148 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:25:13.148 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:25:13.148 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:13.149 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:13.149 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:13.149 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:13.149 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:13.149 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:13.149 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:13.149 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:13.149 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:13.149 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:13.149 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:13.149 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:13.149 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:13.149 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:25:13.149 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:25:13.150 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:25:13.150 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:25:13.150 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:25:13.150 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:25:13.150 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:25:13.150 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:25:13.150 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:25:13.150 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:25:13.150 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:25:13.150 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:25:13.150 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:25:13.150 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:25:13.151 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:25:13.151 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:25:13.151 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:25:13.151 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:13.151 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:13.151 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:13.151 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:13.151 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:13.151 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:13.151 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:13.151 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:13.151 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:13.151 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:13.152 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:13.152 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:13.152 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:25:13.152 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:13.152 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:25:13.152 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:13.152 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:25:13.152 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:25:13.152 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:13.152 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:13.152 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:13.152 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:13.152 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:25:13.153 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:13.153 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:13.153 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:13.153 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:13.153 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:25:13.153 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:13.153 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:13.153 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:13.153 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:13.153 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:25:13.153 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:25:13.153 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:13.153 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:13.154 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:13.154 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:13.154 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:13.154 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:25:13.154 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:25:13.154 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:13.154 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:13.154 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:13.154 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:13.154 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:25:13.154 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:13.154 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:13.154 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:13.155 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:13.155 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:25:13.155 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:13.155 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:13.155 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:13.155 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:13.155 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:25:13.155 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:25:13.155 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:13.155 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:13.155 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:13.155 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:13.156 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:13.156 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:25:13.156 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:25:13.156 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:13.156 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:13.156 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:13.156 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:13.156 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:25:13.156 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:13.156 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:13.156 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:13.156 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:13.156 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:25:13.156 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:13.157 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:13.157 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:13.157 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:13.157 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:25:13.157 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:25:13.157 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:13.157 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:13.157 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:13.157 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:13.157 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:13.157 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:13.157 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:25:13.157 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:25:13.157 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:13.157 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:13.158 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:13.158 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:25:13.158 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:13.158 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:13.158 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:25:13.158 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:13.158 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:13.158 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:13.158 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:25:13.158 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:13.158 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:13.158 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:25:13.158 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:13.158 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:13.159 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:13.159 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:25:13.159 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:13.159 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:13.159 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:13.159 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:25:13.159 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:25:13.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.157+0000 7f3f726c2640 1 -- 172.21.15.37:0/3749799377 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3f4c035ed0 msgr2=0x7f3f4c038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:13.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.157+0000 7f3f726c2640 1 --2- 172.21.15.37:0/3749799377 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3f4c035ed0 0x7f3f4c038390 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3f64007fd0 tx=0x7f3f64007dc0 comp rx=0 tx=0).stop 2024-04-26T00:25:13.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.157+0000 7f3f726c2640 10 monclient: shutdown 2024-04-26T00:25:13.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.157+0000 7f3f726c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:13.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.157+0000 7f3f726c2640 1 -- 172.21.15.37:0/3749799377 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f6c09fc50 msgr2=0x7f3f6c134c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:13.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.157+0000 7f3f726c2640 1 --2- 172.21.15.37:0/3749799377 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3f6c09fc50 0x7f3f6c134c10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c00cc40 tx=0x7f3f5c00a430 comp rx=0 tx=0).stop 2024-04-26T00:25:13.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.157+0000 7f3f70c5f640 1 -- 172.21.15.37:0/3749799377 reap_dead start 2024-04-26T00:25:13.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.157+0000 7f3f70c5f640 1 -- 172.21.15.37:0/3749799377 reap_dead start 2024-04-26T00:25:13.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.157+0000 7f3f726c2640 1 -- 172.21.15.37:0/3749799377 shutdown_connections 2024-04-26T00:25:13.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.157+0000 7f3f726c2640 1 -- 172.21.15.37:0/3749799377 >> 172.21.15.37:0/3749799377 conn(0x7f3f6c0a1d50 msgr2=0x7f3f6c13c360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:13.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.157+0000 7f3f726c2640 1 -- 172.21.15.37:0/3749799377 shutdown_connections 2024-04-26T00:25:13.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.157+0000 7f3f726c2640 1 -- 172.21.15.37:0/3749799377 wait complete. 2024-04-26T00:25:13.161 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:25:13.161 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:25:13.162 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:25:13.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c6df8640 10 monclient: get_monmap_and_config 2024-04-26T00:25:13.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c6df8640 10 monclient: build_initial_monmap 2024-04-26T00:25:13.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c6df8640 10 monclient: monmap: 2024-04-26T00:25:13.277 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:13.277 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:13.277 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:25:13.300064+0000 2024-04-26T00:25:13.277 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:25:13.300064+0000 2024-04-26T00:25:13.278 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:13.278 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:13.278 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:25:13.278 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:25:13.278 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:25:13.278 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:25:13.278 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:25:13.278 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:13.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c6df8640 1 Processor -- start 2024-04-26T00:25:13.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c6df8640 1 -- start start 2024-04-26T00:25:13.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c6df8640 10 monclient: init 2024-04-26T00:25:13.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c6df8640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:13.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c6df8640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-04-26T00:25:13.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c6df8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61c00a3d90 0x7f61c009bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:13.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c6df8640 10 monclient(hunting): picked mon.noname-b con 0x7f61c00a3d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:13.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c6df8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f61c00a4170 addr v1:172.21.15.37:6789/0 2024-04-26T00:25:13.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c6df8640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f61c013c3e0 0x7f61c013e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:13.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c6df8640 10 monclient(hunting): picked mon.noname-c con 0x7f61c013c3e0 addr v2:172.21.15.45:3300/0 2024-04-26T00:25:13.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c6df8640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61c00a5150 con 0x7f61c00a4170 2024-04-26T00:25:13.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c6df8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:13.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c6df8640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f61c009b280 con 0x7f61c013c3e0 2024-04-26T00:25:13.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c6df8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:13.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c6df8640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f61c009b560 con 0x7f61c00a3d90 2024-04-26T00:25:13.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c6df8640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:13.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c6df8640 10 monclient(hunting): authenticate will time out at 1294.605930s 2024-04-26T00:25:13.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c4b94640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61c00a3d90 0x7f61c009bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:13.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c4b94640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61c00a3d90 0x7f61c009bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:59746/0 (socket says 172.21.15.37:59746) 2024-04-26T00:25:13.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61bffff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f61c00a4170 0x7f61c00a45b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:57226/0 (socket says 172.21.15.37:57226) 2024-04-26T00:25:13.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c4b94640 1 -- 172.21.15.37:0/3410105236 learned_addr learned my addr 172.21.15.37:0/3410105236 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:13.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c4b94640 10 monclient(hunting): get_auth_request con 0x7f61c00a3d90 auth_method 0 2024-04-26T00:25:13.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c4b94640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:13.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c4b94640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:13.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c4b94640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:13.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c4b94640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:13.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c4b94640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:13.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c4b94640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:13.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c5395640 1 --2- 172.21.15.37:0/3410105236 >> v2:172.21.15.45:3300/0 conn(0x7f61c013c3e0 0x7f61c013e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:13.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c5395640 10 monclient(hunting): get_auth_request con 0x7f61c013c3e0 auth_method 0 2024-04-26T00:25:13.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c5395640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:13.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c5395640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:13.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c5395640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:13.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c4b94640 10 monclient(hunting): handle_auth_done global_id 64118 payload 274 2024-04-26T00:25:13.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c4b94640 1 -- 172.21.15.37:0/3410105236 >> v2:172.21.15.45:3300/0 conn(0x7f61c013c3e0 msgr2=0x7f61c013e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:13.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c4b94640 1 --2- 172.21.15.37:0/3410105236 >> v2:172.21.15.45:3300/0 conn(0x7f61c013c3e0 0x7f61c013e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:13.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c4b94640 1 -- 172.21.15.37:0/3410105236 >> v1:172.21.15.37:6789/0 conn(0x7f61c00a4170 legacy=0x7f61c00a45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:13.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c4b94640 10 monclient: _finish_hunting 0 2024-04-26T00:25:13.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c4b94640 1 monclient: found mon.noname-b 2024-04-26T00:25:13.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c4b94640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:13.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c4b94640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:25:13.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.297+0000 7f61c4b94640 1 -- 172.21.15.37:0/3410105236 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61c006dc60 con 0x7f61c00a3d90 2024-04-26T00:25:13.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.301+0000 7f61c4b94640 1 -- 172.21.15.37:0/3410105236 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61c00a3d90 msgr2=0x7f61c009bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:13.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.301+0000 7f61c4b94640 1 --2- 172.21.15.37:0/3410105236 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61c00a3d90 0x7f61c009bf40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f61b8009e50 tx=0x7f61b801c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57dd6a76b37d0c20 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:13.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.301+0000 7f61bf7fe640 1 -- 172.21.15.37:0/3410105236 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b80a5640 con 0x7f61c00a3d90 2024-04-26T00:25:13.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.301+0000 7f61bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:13.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.301+0000 7f61bf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:25:13.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.301+0000 7f61bf7fe640 10 monclient: dump: 2024-04-26T00:25:13.288 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:13.289 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:13.289 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:13.289 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:13.289 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:13.289 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:13.289 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:13.289 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:13.289 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:13.289 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:13.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.301+0000 7f61bf7fe640 10 monclient: _finish_auth 0 2024-04-26T00:25:13.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.301+0000 7f61bf7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:25:13.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.301+0000 7f61bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:13.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.301+0000 7f61bf7fe640 1 -- 172.21.15.37:0/3410105236 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61b80a57e0 con 0x7f61c00a3d90 2024-04-26T00:25:13.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.301+0000 7f61bf7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:13.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.301+0000 7f61bf7fe640 1 -- 172.21.15.37:0/3410105236 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b80a5bb0 con 0x7f61c00a3d90 2024-04-26T00:25:13.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.301+0000 7f61bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:13.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.301+0000 7f61bf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:25:13.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.301+0000 7f61bf7fe640 10 monclient: dump: 2024-04-26T00:25:13.290 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:13.290 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:13.290 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:13.290 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:13.290 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:13.291 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:13.291 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:13.291 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:13.291 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:13.291 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:13.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.301+0000 7f61c6df8640 5 monclient: authenticate success, global_id 64118 2024-04-26T00:25:13.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.301+0000 7f61c6df8640 10 monclient: get_monmap_and_config success 2024-04-26T00:25:13.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.301+0000 7f61c6df8640 4 set_mon_vals no callback set 2024-04-26T00:25:13.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c6df8640 10 monclient: shutdown 2024-04-26T00:25:13.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c6df8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:13.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c6df8640 1 -- 172.21.15.37:0/3410105236 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61c00a3d90 msgr2=0x7f61c009bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:13.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c6df8640 1 --2- 172.21.15.37:0/3410105236 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61c00a3d90 0x7f61c009bf40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f61b8009e50 tx=0x7f61b801c8b0 comp rx=0 tx=0).stop 2024-04-26T00:25:13.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c6df8640 1 -- 172.21.15.37:0/3410105236 shutdown_connections 2024-04-26T00:25:13.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c6df8640 1 --2- 172.21.15.37:0/3410105236 >> v2:172.21.15.45:3300/0 conn(0x7f61c013c3e0 0x7f61c013e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:13.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c6df8640 1 --2- 172.21.15.37:0/3410105236 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61c00a3d90 0x7f61c009bf40 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:13.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c6df8640 1 -- 172.21.15.37:0/3410105236 >> 172.21.15.37:0/3410105236 conn(0x7f61c00aa0a0 msgr2=0x7f61c00a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:13.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c6df8640 1 -- 172.21.15.37:0/3410105236 shutdown_connections 2024-04-26T00:25:13.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c6df8640 1 -- 172.21.15.37:0/3410105236 wait complete. 2024-04-26T00:25:13.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c6df8640 10 monclient: build_initial_monmap 2024-04-26T00:25:13.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c6df8640 10 monclient: monmap: 2024-04-26T00:25:13.292 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:13.292 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:25:13.292 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:25:13.292 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:25:13.292 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:13.292 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:25:13.292 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:25:13.293 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:25:13.293 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:25:13.293 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:13.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c6df8640 1 Processor -- start 2024-04-26T00:25:13.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c6df8640 1 -- start start 2024-04-26T00:25:13.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c6df8640 10 monclient: init 2024-04-26T00:25:13.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c6df8640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:13.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c6df8640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:25:13.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c6df8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61c00a3d90 0x7f61c0139db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:13.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c6df8640 10 monclient(hunting): picked mon.noname-c con 0x7f61c00a3d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:13.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c6df8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61c00a4170 0x7f61c013a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:13.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c6df8640 10 monclient(hunting): picked mon.noname-b con 0x7f61c00a4170 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:25:13.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c6df8640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f61c013c3e0 0x7f61c013a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:13.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c6df8640 10 monclient(hunting): picked mon.noname-a con 0x7f61c013c3e0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:25:13.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c6df8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:13.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c6df8640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f61c0141740 con 0x7f61c013c3e0 2024-04-26T00:25:13.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c6df8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:13.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c6df8640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f61c009c690 con 0x7f61c00a4170 2024-04-26T00:25:13.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c6df8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:13.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c6df8640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f61c01418c0 con 0x7f61c00a3d90 2024-04-26T00:25:13.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c6df8640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:13.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c6df8640 10 monclient(hunting): authenticate will time out at 1294.613674s 2024-04-26T00:25:13.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c4b94640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61c00a3d90 0x7f61c0139db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:13.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c5395640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f61c013c3e0 0x7f61c013a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:13.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c4b94640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61c00a3d90 0x7f61c0139db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:59760/0 (socket says 172.21.15.37:59760) 2024-04-26T00:25:13.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c5395640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f61c013c3e0 0x7f61c013a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36422/0 (socket says 172.21.15.37:36422) 2024-04-26T00:25:13.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c4b94640 1 -- 172.21.15.37:0/71120137 learned_addr learned my addr 172.21.15.37:0/71120137 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:13.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c4b94640 10 monclient(hunting): get_auth_request con 0x7f61c00a3d90 auth_method 0 2024-04-26T00:25:13.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c4b94640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:13.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c4b94640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:13.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c4b94640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:13.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c5395640 10 monclient(hunting): get_auth_request con 0x7f61c013c3e0 auth_method 0 2024-04-26T00:25:13.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c5395640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:13.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c5395640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:13.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c5395640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:13.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c4b94640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:13.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c4b94640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:13.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c4b94640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:13.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c5395640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:13.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c5395640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:13.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c5395640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:13.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61bffff640 1 --2- 172.21.15.37:0/71120137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61c00a4170 0x7f61c013a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:13.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61bffff640 10 monclient(hunting): get_auth_request con 0x7f61c00a4170 auth_method 0 2024-04-26T00:25:13.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:13.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61bffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:13.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61bffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:13.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c4b94640 10 monclient(hunting): handle_auth_done global_id 64121 payload 824 2024-04-26T00:25:13.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c4b94640 1 -- 172.21.15.37:0/71120137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61c00a4170 msgr2=0x7f61c013a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:13.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c4b94640 1 --2- 172.21.15.37:0/71120137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61c00a4170 0x7f61c013a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:13.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c4b94640 1 -- 172.21.15.37:0/71120137 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f61c013c3e0 msgr2=0x7f61c013a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:13.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c4b94640 1 --2- 172.21.15.37:0/71120137 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f61c013c3e0 0x7f61c013a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:13.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c4b94640 10 monclient: _finish_hunting 0 2024-04-26T00:25:13.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c4b94640 1 monclient: found mon.noname-c 2024-04-26T00:25:13.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c4b94640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:13.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c4b94640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:25:13.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c4b94640 1 -- 172.21.15.37:0/71120137 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61c013cc70 con 0x7f61c00a3d90 2024-04-26T00:25:13.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.305+0000 7f61c5395640 1 --2- 172.21.15.37:0/71120137 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f61c013c3e0 0x7f61c013a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:25:13.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61c4b94640 1 -- 172.21.15.37:0/71120137 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61c00a3d90 msgr2=0x7f61c0139db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:13.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61c4b94640 1 --2- 172.21.15.37:0/71120137 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61c00a3d90 0x7f61c0139db0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f61b80af040 tx=0x7f61b8019650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:13.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61bd7fa640 1 -- 172.21.15.37:0/71120137 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b80bc020 con 0x7f61c00a3d90 2024-04-26T00:25:13.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61bd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:13.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61bd7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:25:13.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61bd7fa640 10 monclient: dump: 2024-04-26T00:25:13.300 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:13.300 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:13.301 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:13.301 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:13.301 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:13.301 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:13.301 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:13.301 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:13.301 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:13.301 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:13.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61bd7fa640 10 monclient: _finish_auth 0 2024-04-26T00:25:13.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61bd7fa640 10 monclient: _check_auth_tickets 2024-04-26T00:25:13.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61bd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:13.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61bd7fa640 1 -- 172.21.15.37:0/71120137 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61b80b8a90 con 0x7f61c00a3d90 2024-04-26T00:25:13.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61bd7fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:13.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61bd7fa640 1 -- 172.21.15.37:0/71120137 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b80a5640 con 0x7f61c00a3d90 2024-04-26T00:25:13.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61c6df8640 5 monclient: authenticate success, global_id 64121 2024-04-26T00:25:13.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61c6df8640 10 monclient: _renew_subs 2024-04-26T00:25:13.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61c6df8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:25:13.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61c6df8640 1 -- 172.21.15.37:0/71120137 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61c01379a0 con 0x7f61c00a3d90 2024-04-26T00:25:13.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61bd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:13.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61bd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:25:13.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61bd7fa640 10 monclient: dump: 2024-04-26T00:25:13.302 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:13.302 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:13.302 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:13.302 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:13.303 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:13.303 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:13.303 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:13.303 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:13.303 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:13.303 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:13.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61be7fc640 4 set_mon_vals no callback set 2024-04-26T00:25:13.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61c6df8640 10 monclient: _renew_subs 2024-04-26T00:25:13.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61c6df8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:25:13.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61c6df8640 1 -- 172.21.15.37:0/71120137 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f61c01339d0 con 0x7f61c00a3d90 2024-04-26T00:25:13.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61c6df8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:13.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61c6df8640 1 --2- 172.21.15.37:0/71120137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61c01cebe0 0x7f61c01cefc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:13.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61c6df8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:13.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61c6df8640 1 -- 172.21.15.37:0/71120137 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f61c004f510 con 0x7f61c01cebe0 2024-04-26T00:25:13.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61bd7fa640 1 -- 172.21.15.37:0/71120137 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f61b80bf020 con 0x7f61c00a3d90 2024-04-26T00:25:13.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61bd7fa640 1 --2- 172.21.15.37:0/71120137 >> v2:172.21.15.45:6812/2379918623 conn(0x7f61a0035ed0 0x7f61a0038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:13.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61bd7fa640 1 -- 172.21.15.37:0/71120137 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(205..205 src has 172..205) ==== 3975+0+0 (secure 0 0 0) 0x7f61b80e3b20 con 0x7f61c00a3d90 2024-04-26T00:25:13.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61bffff640 1 --2- 172.21.15.37:0/71120137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61c01cebe0 0x7f61c01cefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:13.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61bffff640 10 monclient: get_auth_request con 0x7f61c01cebe0 auth_method 0 2024-04-26T00:25:13.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:13.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61bffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:13.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61bffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:13.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61c5395640 1 --2- 172.21.15.37:0/71120137 >> v2:172.21.15.45:6812/2379918623 conn(0x7f61a0035ed0 0x7f61a0038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:13.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:13.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:13.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:13.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61c5395640 10 monclient: get_auth_request con 0x7f61a0035ed0 auth_method 0 2024-04-26T00:25:13.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61bffff640 10 monclient(hunting): handle_auth_done global_id 24433 payload 824 2024-04-26T00:25:13.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.309+0000 7f61bffff640 1 -- 172.21.15.37:0/71120137 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f61c0132700 con 0x7f61c01cebe0 2024-04-26T00:25:13.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.313+0000 7f61bffff640 1 -- 172.21.15.37:0/71120137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61c01cebe0 msgr2=0x7f61c01cefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:13.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.313+0000 7f61bffff640 1 --2- 172.21.15.37:0/71120137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61c01cebe0 0x7f61c01cefc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f61c013b650 tx=0x7f61a800e960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:13.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.313+0000 7f61c5395640 1 --2- 172.21.15.37:0/71120137 >> v2:172.21.15.45:6812/2379918623 conn(0x7f61a0035ed0 0x7f61a0038390 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f61c0137840 tx=0x7f61b0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:13.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.313+0000 7f61bd7fa640 1 -- 172.21.15.37:0/71120137 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a8004e30 con 0x7f61c01cebe0 2024-04-26T00:25:13.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.313+0000 7f61bd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:13.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.313+0000 7f61bd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:25:13.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.313+0000 7f61bd7fa640 10 monclient: dump: 2024-04-26T00:25:13.305 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:13.305 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:13.305 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:13.305 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:13.305 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:13.306 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:13.306 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:13.306 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:13.306 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:13.306 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:13.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.313+0000 7f61bd7fa640 1 -- 172.21.15.37:0/71120137 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f61a8018070 con 0x7f61c01cebe0 2024-04-26T00:25:13.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.313+0000 7f61bd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:13.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.313+0000 7f61bd7fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:25:13.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.313+0000 7f61bd7fa640 1 -- 172.21.15.37:0/71120137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61c01cebe0 msgr2=0x7f61c01cefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:13.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.313+0000 7f61bd7fa640 1 --2- 172.21.15.37:0/71120137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61c01cebe0 0x7f61c01cefc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f61c013b650 tx=0x7f61a800e960 comp rx=0 tx=0).stop 2024-04-26T00:25:13.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.313+0000 7f61bd7fa640 1 -- 172.21.15.37:0/71120137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61c01cebe0 msgr2=0x7f61c01cefc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:25:13.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.313+0000 7f61bd7fa640 1 --2- 172.21.15.37:0/71120137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61c01cebe0 0x7f61c01cefc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:13.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.321+0000 7f61c6df8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:25:13.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.321+0000 7f61c6df8640 1 --2- 172.21.15.37:0/71120137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61c01cf500 0x7f61c00a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:13.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.321+0000 7f61c6df8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:13.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.321+0000 7f61c6df8640 1 -- 172.21.15.37:0/71120137 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f61c0132240 con 0x7f61c01cf500 2024-04-26T00:25:13.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.321+0000 7f61bffff640 1 --2- 172.21.15.37:0/71120137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61c01cf500 0x7f61c00a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:13.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.321+0000 7f61bffff640 10 monclient: get_auth_request con 0x7f61c01cf500 auth_method 0 2024-04-26T00:25:13.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.321+0000 7f61bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:13.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.321+0000 7f61bffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:13.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.321+0000 7f61bffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:13.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.321+0000 7f61bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:13.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.321+0000 7f61bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:13.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.321+0000 7f61bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:13.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.321+0000 7f61bffff640 10 monclient(hunting): handle_auth_done global_id 24436 payload 824 2024-04-26T00:25:13.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.321+0000 7f61bffff640 1 -- 172.21.15.37:0/71120137 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f61c0075dc0 con 0x7f61c01cf500 2024-04-26T00:25:13.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.325+0000 7f61bffff640 1 -- 172.21.15.37:0/71120137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61c01cf500 msgr2=0x7f61c00a17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:13.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.325+0000 7f61bffff640 1 --2- 172.21.15.37:0/71120137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61c01cf500 0x7f61c00a17a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f61c013b650 tx=0x7f61a8004940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:13.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.325+0000 7f61bd7fa640 1 -- 172.21.15.37:0/71120137 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a80066f0 con 0x7f61c01cf500 2024-04-26T00:25:13.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.325+0000 7f61bd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:13.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.325+0000 7f61bd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:25:13.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.325+0000 7f61bd7fa640 10 monclient: dump: 2024-04-26T00:25:13.310 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:13.310 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:13.310 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:13.310 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:13.310 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:13.310 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:13.310 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:13.310 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:13.310 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:13.310 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:13.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.325+0000 7f61bd7fa640 1 -- 172.21.15.37:0/71120137 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4719 (secure 0 0 0) 0x7f61a80066f0 con 0x7f61c01cf500 2024-04-26T00:25:13.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.325+0000 7f61bd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:25:13.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.325+0000 7f61bd7fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:25:13.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.325+0000 7f61bd7fa640 1 -- 172.21.15.37:0/71120137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61c01cf500 msgr2=0x7f61c00a17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:13.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.325+0000 7f61bd7fa640 1 --2- 172.21.15.37:0/71120137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61c01cf500 0x7f61c00a17a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f61c013b650 tx=0x7f61a8004940 comp rx=0 tx=0).stop 2024-04-26T00:25:13.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.325+0000 7f61bd7fa640 1 -- 172.21.15.37:0/71120137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61c01cf500 msgr2=0x7f61c00a17a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:25:13.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.325+0000 7f61bd7fa640 1 --2- 172.21.15.37:0/71120137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61c01cf500 0x7f61c00a17a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:13.311 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:25:13.311 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:25:13.311 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:25:13.311 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:25:13.312 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 40, 2024-04-26T00:25:13.312 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 221492, 2024-04-26T00:25:13.312 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:25:13.312 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:25:13.312 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:25:13.312 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:25:13.312 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:13.312 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:25:13.312 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:25:13.312 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:25:13.312 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:25:13.312 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:13.313 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:13.313 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:13.313 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:13.313 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:13.313 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:13.313 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:13.313 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:13.313 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:13.313 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:13.313 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:13.313 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:13.313 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:25:13.313 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:25:13.313 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:13.314 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:13.314 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:13.314 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:13.314 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:13.314 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:13.314 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:13.314 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:13.314 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:13.314 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:13.314 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:13.314 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:13.314 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:13.314 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:25:13.314 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:25:13.315 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:25:13.315 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:25:13.315 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:25:13.315 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:25:13.315 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:25:13.315 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:25:13.315 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:25:13.315 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:25:13.315 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:25:13.315 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:25:13.315 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:25:13.315 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:25:13.315 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:25:13.315 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:25:13.316 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:25:13.316 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:13.316 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:13.316 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:13.316 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:13.316 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:13.316 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:13.316 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:13.316 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:13.316 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:13.316 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:13.316 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:13.317 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:13.317 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:25:13.317 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:13.317 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:25:13.317 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:13.317 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:25:13.317 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:25:13.317 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:13.317 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:13.317 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:13.317 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:13.317 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:25:13.318 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:13.318 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:13.318 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:13.318 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:13.318 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:25:13.318 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:13.318 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:13.318 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:13.318 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:13.318 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:25:13.318 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:25:13.318 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:13.318 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:13.319 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:13.319 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:13.319 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:13.319 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:25:13.319 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:25:13.319 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:13.319 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:13.319 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:13.319 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:13.319 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:25:13.319 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:13.319 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:13.319 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:13.319 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:13.320 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:25:13.320 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:13.320 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:13.320 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:13.320 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:13.320 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:25:13.320 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:25:13.320 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:13.320 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:13.320 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:13.320 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:13.320 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:13.320 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:25:13.321 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:25:13.321 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:13.321 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:13.321 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:13.321 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:13.321 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:25:13.321 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:13.321 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:13.321 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:13.321 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:13.321 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:25:13.321 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:13.321 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:13.321 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:13.322 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:13.322 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:25:13.322 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:25:13.322 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:13.322 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:13.322 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:13.322 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:13.322 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:13.322 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:13.322 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:25:13.322 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:25:13.322 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:13.322 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:13.322 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:13.323 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:25:13.323 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:13.323 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:13.323 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:25:13.323 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:13.323 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:13.323 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:13.323 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:25:13.323 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:13.323 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:13.323 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:25:13.323 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:13.323 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:13.323 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:13.324 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:25:13.324 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:13.324 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:13.324 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:13.324 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:25:13.324 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:25:13.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.325+0000 7f61c6df8640 1 -- 172.21.15.37:0/71120137 >> v2:172.21.15.45:6812/2379918623 conn(0x7f61a0035ed0 msgr2=0x7f61a0038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:13.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.325+0000 7f61c6df8640 1 --2- 172.21.15.37:0/71120137 >> v2:172.21.15.45:6812/2379918623 conn(0x7f61a0035ed0 0x7f61a0038390 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f61c0137840 tx=0x7f61b0008040 comp rx=0 tx=0).stop 2024-04-26T00:25:13.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.325+0000 7f61c6df8640 10 monclient: shutdown 2024-04-26T00:25:13.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.325+0000 7f61c6df8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:13.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.325+0000 7f61c6df8640 1 -- 172.21.15.37:0/71120137 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61c00a3d90 msgr2=0x7f61c0139db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:13.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.325+0000 7f61c6df8640 1 --2- 172.21.15.37:0/71120137 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61c00a3d90 0x7f61c0139db0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f61b80af040 tx=0x7f61b8019650 comp rx=0 tx=0).stop 2024-04-26T00:25:13.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.325+0000 7f61c5395640 1 -- 172.21.15.37:0/71120137 reap_dead start 2024-04-26T00:25:13.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.325+0000 7f61c5395640 1 -- 172.21.15.37:0/71120137 reap_dead start 2024-04-26T00:25:13.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.325+0000 7f61c6df8640 1 -- 172.21.15.37:0/71120137 shutdown_connections 2024-04-26T00:25:13.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.325+0000 7f61c6df8640 1 -- 172.21.15.37:0/71120137 >> 172.21.15.37:0/71120137 conn(0x7f61c00aa0a0 msgr2=0x7f61c00a8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:13.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.325+0000 7f61c6df8640 1 -- 172.21.15.37:0/71120137 shutdown_connections 2024-04-26T00:25:13.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.325+0000 7f61c6df8640 1 -- 172.21.15.37:0/71120137 wait complete. 2024-04-26T00:25:13.326 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:25:13.326 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-26T00:25:13.327 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-26T00:25:13.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.457+0000 7f6ae9ec9640 10 monclient: get_monmap_and_config 2024-04-26T00:25:13.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.457+0000 7f6ae9ec9640 10 monclient: build_initial_monmap 2024-04-26T00:25:13.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.457+0000 7f6ae9ec9640 10 monclient: monmap: 2024-04-26T00:25:13.440 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:13.440 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:13.440 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:25:13.462911+0000 2024-04-26T00:25:13.440 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:25:13.462911+0000 2024-04-26T00:25:13.440 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:13.441 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:13.441 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:25:13.441 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:25:13.441 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:25:13.441 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:25:13.441 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:25:13.441 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:13.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.457+0000 7f6ae9ec9640 1 Processor -- start 2024-04-26T00:25:13.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.457+0000 7f6ae9ec9640 1 -- start start 2024-04-26T00:25:13.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.457+0000 7f6ae9ec9640 10 monclient: init 2024-04-26T00:25:13.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae9ec9640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:13.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae9ec9640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-04-26T00:25:13.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae9ec9640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6ae40a0260 0x7f6ae40a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:13.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae9ec9640 10 monclient(hunting): picked mon.noname-a con 0x7f6ae40a0260 addr v2:172.21.15.37:3300/0 2024-04-26T00:25:13.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae9ec9640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6ae4134b30 0x7f6ae4134f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:13.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae9ec9640 10 monclient(hunting): picked mon.noname-b con 0x7f6ae4134b30 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:13.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae9ec9640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6ae4135490 0x7f6ae413be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:13.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae9ec9640 10 monclient(hunting): picked mon.noname-c con 0x7f6ae4135490 addr v2:172.21.15.45:3300/0 2024-04-26T00:25:13.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae9ec9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:13.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae9ec9640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f6ae409ec10 con 0x7f6ae40a0260 2024-04-26T00:25:13.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae9ec9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:13.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae9ec9640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f6ae409f0f0 con 0x7f6ae4135490 2024-04-26T00:25:13.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae9ec9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:13.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae9ec9640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f6ae40a1130 con 0x7f6ae4134b30 2024-04-26T00:25:13.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae9ec9640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:13.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae9ec9640 10 monclient(hunting): authenticate will time out at 1294.768589s 2024-04-26T00:25:13.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae37fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6ae40a0260 0x7f6ae40a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:13.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae2ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6ae4134b30 0x7f6ae4134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:13.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae37fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6ae40a0260 0x7f6ae40a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36424/0 (socket says 172.21.15.37:36424) 2024-04-26T00:25:13.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae2ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6ae4134b30 0x7f6ae4134f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:59768/0 (socket says 172.21.15.37:59768) 2024-04-26T00:25:13.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae37fe640 1 -- 172.21.15.37:0/2515830197 learned_addr learned my addr 172.21.15.37:0/2515830197 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:13.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae37fe640 10 monclient(hunting): get_auth_request con 0x7f6ae40a0260 auth_method 0 2024-04-26T00:25:13.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:13.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:13.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:13.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae2ffd640 10 monclient(hunting): get_auth_request con 0x7f6ae4134b30 auth_method 0 2024-04-26T00:25:13.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:13.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:13.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:13.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:13.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:13.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:13.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:13.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:13.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae3fff640 1 --2- 172.21.15.37:0/2515830197 >> v2:172.21.15.45:3300/0 conn(0x7f6ae4135490 0x7f6ae413be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:13.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:13.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae3fff640 10 monclient(hunting): get_auth_request con 0x7f6ae4135490 auth_method 0 2024-04-26T00:25:13.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:13.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:13.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:13.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:13.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:13.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:13.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae37fe640 10 monclient(hunting): handle_auth_done global_id 54192 payload 274 2024-04-26T00:25:13.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae37fe640 1 -- 172.21.15.37:0/2515830197 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6ae4134b30 msgr2=0x7f6ae4134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:13.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae37fe640 1 --2- 172.21.15.37:0/2515830197 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6ae4134b30 0x7f6ae4134f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:13.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae37fe640 1 -- 172.21.15.37:0/2515830197 >> v2:172.21.15.45:3300/0 conn(0x7f6ae4135490 msgr2=0x7f6ae413be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:13.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae37fe640 1 --2- 172.21.15.37:0/2515830197 >> v2:172.21.15.45:3300/0 conn(0x7f6ae4135490 0x7f6ae413be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:13.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae37fe640 10 monclient: _finish_hunting 0 2024-04-26T00:25:13.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae37fe640 1 monclient: found mon.noname-a 2024-04-26T00:25:13.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:13.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:25:13.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae37fe640 1 -- 172.21.15.37:0/2515830197 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ae406cff0 con 0x7f6ae40a0260 2024-04-26T00:25:13.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.461+0000 7f6ae2ffd640 1 --2- 172.21.15.37:0/2515830197 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6ae4134b30 0x7f6ae4134f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:25:13.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae37fe640 1 -- 172.21.15.37:0/2515830197 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ae40a0260 msgr2=0x7f6ae40a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:13.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae37fe640 1 --2- 172.21.15.37:0/2515830197 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ae40a0260 0x7f6ae40a0640 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6ae4135bd0 tx=0x7f6adc01c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16b0ac727750c986 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:13.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae27fc640 1 -- 172.21.15.37:0/2515830197 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc0a5530 con 0x7f6ae40a0260 2024-04-26T00:25:13.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:13.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:25:13.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae27fc640 10 monclient: dump: 2024-04-26T00:25:13.454 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:13.454 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:13.454 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:13.454 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:13.454 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:13.454 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:13.454 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:13.454 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:13.454 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:13.454 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:13.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae27fc640 10 monclient: _finish_auth 0 2024-04-26T00:25:13.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae27fc640 10 monclient: _check_auth_tickets 2024-04-26T00:25:13.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:13.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae27fc640 1 -- 172.21.15.37:0/2515830197 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6adc0a56d0 con 0x7f6ae40a0260 2024-04-26T00:25:13.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae9ec9640 5 monclient: authenticate success, global_id 54192 2024-04-26T00:25:13.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae27fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:13.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae9ec9640 10 monclient: get_monmap_and_config success 2024-04-26T00:25:13.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae27fc640 1 -- 172.21.15.37:0/2515830197 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc0a5aa0 con 0x7f6ae40a0260 2024-04-26T00:25:13.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae9ec9640 4 set_mon_vals no callback set 2024-04-26T00:25:13.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:13.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:13.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae27fc640 10 monclient: dump: 2024-04-26T00:25:13.455 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:13.455 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:13.455 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:13.455 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:13.456 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:13.456 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:13.456 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:13.456 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:13.456 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:13.456 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:13.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae9ec9640 10 monclient: shutdown 2024-04-26T00:25:13.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae9ec9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:13.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae9ec9640 1 -- 172.21.15.37:0/2515830197 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ae40a0260 msgr2=0x7f6ae40a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:13.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae9ec9640 1 --2- 172.21.15.37:0/2515830197 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ae40a0260 0x7f6ae40a0640 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6ae4135bd0 tx=0x7f6adc01c8b0 comp rx=0 tx=0).stop 2024-04-26T00:25:13.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae9ec9640 1 -- 172.21.15.37:0/2515830197 shutdown_connections 2024-04-26T00:25:13.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae9ec9640 1 --2- 172.21.15.37:0/2515830197 >> v2:172.21.15.45:3300/0 conn(0x7f6ae4135490 0x7f6ae413be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:13.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae9ec9640 1 --2- 172.21.15.37:0/2515830197 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6ae4134b30 0x7f6ae4134f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:13.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae9ec9640 1 --2- 172.21.15.37:0/2515830197 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ae40a0260 0x7f6ae40a0640 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:13.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae9ec9640 1 -- 172.21.15.37:0/2515830197 >> 172.21.15.37:0/2515830197 conn(0x7f6ae40a1ad0 msgr2=0x7f6ae40a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:13.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae9ec9640 1 -- 172.21.15.37:0/2515830197 shutdown_connections 2024-04-26T00:25:13.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae9ec9640 1 -- 172.21.15.37:0/2515830197 wait complete. 2024-04-26T00:25:13.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae9ec9640 10 monclient: build_initial_monmap 2024-04-26T00:25:13.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae9ec9640 10 monclient: monmap: 2024-04-26T00:25:13.457 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:13.457 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:25:13.457 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:25:13.457 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:25:13.457 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:13.457 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:25:13.457 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:25:13.457 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:25:13.458 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:25:13.458 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:13.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae9ec9640 1 Processor -- start 2024-04-26T00:25:13.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae9ec9640 1 -- start start 2024-04-26T00:25:13.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae9ec9640 10 monclient: init 2024-04-26T00:25:13.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae9ec9640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:13.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae9ec9640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:25:13.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae9ec9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ae40a0260 0x7f6ae41cec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:13.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae9ec9640 10 monclient(hunting): picked mon.noname-b con 0x7f6ae40a0260 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:25:13.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae9ec9640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6ae4134b30 0x7f6ae4131ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:13.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae9ec9640 10 monclient(hunting): picked mon.noname-c con 0x7f6ae4134b30 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:13.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae9ec9640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ae4135490 0x7f6ae4132010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:13.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae9ec9640 10 monclient(hunting): picked mon.noname-a con 0x7f6ae4135490 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:25:13.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae9ec9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:13.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae9ec9640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae409f430 con 0x7f6ae4135490 2024-04-26T00:25:13.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae9ec9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:13.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae9ec9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae409f2b0 con 0x7f6ae40a0260 2024-04-26T00:25:13.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae9ec9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:13.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae9ec9640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f6ae409fe90 con 0x7f6ae4134b30 2024-04-26T00:25:13.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae9ec9640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:13.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae9ec9640 10 monclient(hunting): authenticate will time out at 1294.775661s 2024-04-26T00:25:13.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae3fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ae4135490 0x7f6ae4132010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:13.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae2ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6ae4134b30 0x7f6ae4131ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:13.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae3fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ae4135490 0x7f6ae4132010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36436/0 (socket says 172.21.15.37:36436) 2024-04-26T00:25:13.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae2ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6ae4134b30 0x7f6ae4131ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:59780/0 (socket says 172.21.15.37:59780) 2024-04-26T00:25:13.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae3fff640 1 -- 172.21.15.37:0/2225062992 learned_addr learned my addr 172.21.15.37:0/2225062992 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:13.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae3fff640 10 monclient(hunting): get_auth_request con 0x7f6ae4135490 auth_method 0 2024-04-26T00:25:13.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:13.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:13.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:13.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae2ffd640 10 monclient(hunting): get_auth_request con 0x7f6ae4134b30 auth_method 0 2024-04-26T00:25:13.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:13.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:13.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:13.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:13.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:13.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:13.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:13.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:13.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.465+0000 7f6ae2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:13.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6ae37fe640 1 --2- 172.21.15.37:0/2225062992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ae40a0260 0x7f6ae41cec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:13.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6ae3fff640 10 monclient(hunting): handle_auth_done global_id 54195 payload 824 2024-04-26T00:25:13.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6ae3fff640 1 -- 172.21.15.37:0/2225062992 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6ae4134b30 msgr2=0x7f6ae4131ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:13.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6ae3fff640 1 --2- 172.21.15.37:0/2225062992 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6ae4134b30 0x7f6ae4131ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:13.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6ae3fff640 1 -- 172.21.15.37:0/2225062992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ae40a0260 msgr2=0x7f6ae41cec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:13.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6ae3fff640 1 --2- 172.21.15.37:0/2225062992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ae40a0260 0x7f6ae41cec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:13.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6ae3fff640 10 monclient: _finish_hunting 0 2024-04-26T00:25:13.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6ae3fff640 1 monclient: found mon.noname-a 2024-04-26T00:25:13.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6ae3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:13.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6ae3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:25:13.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6ae3fff640 1 -- 172.21.15.37:0/2225062992 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ae40729d0 con 0x7f6ae4135490 2024-04-26T00:25:13.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6ae37fe640 10 monclient: get_auth_request con 0x7f6ae40a0260 auth_method 0 2024-04-26T00:25:13.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6ae37fe640 1 --2- 172.21.15.37:0/2225062992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ae40a0260 0x7f6ae41cec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:25:13.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6ae3fff640 1 -- 172.21.15.37:0/2225062992 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ae4135490 msgr2=0x7f6ae4132010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:13.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6ae3fff640 1 --2- 172.21.15.37:0/2225062992 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ae4135490 0x7f6ae4132010 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6ad400e930 tx=0x7f6ad400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:13.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6acbfff640 1 -- 172.21.15.37:0/2225062992 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad4017070 con 0x7f6ae4135490 2024-04-26T00:25:13.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6acbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:13.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6acbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:25:13.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6acbfff640 10 monclient: dump: 2024-04-26T00:25:13.462 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:13.462 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:13.462 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:13.462 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:13.462 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:13.462 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:13.462 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:13.462 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:13.462 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:13.462 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:13.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6acbfff640 10 monclient: _finish_auth 0 2024-04-26T00:25:13.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6acbfff640 10 monclient: _check_auth_tickets 2024-04-26T00:25:13.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6acbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:13.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6acbfff640 1 -- 172.21.15.37:0/2225062992 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ad400cbb0 con 0x7f6ae4135490 2024-04-26T00:25:13.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6acbfff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:13.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6acbfff640 1 -- 172.21.15.37:0/2225062992 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad4013470 con 0x7f6ae4135490 2024-04-26T00:25:13.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6ae9ec9640 5 monclient: authenticate success, global_id 54195 2024-04-26T00:25:13.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6ae9ec9640 10 monclient: _renew_subs 2024-04-26T00:25:13.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6ae9ec9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:13.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6ae9ec9640 1 -- 172.21.15.37:0/2225062992 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ae411d1a0 con 0x7f6ae4135490 2024-04-26T00:25:13.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6acbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:13.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6acbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:13.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6ae1ffb640 4 set_mon_vals no callback set 2024-04-26T00:25:13.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6acbfff640 10 monclient: dump: 2024-04-26T00:25:13.463 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:13.464 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:13.464 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:13.464 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:13.464 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:13.464 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:13.464 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:13.464 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:13.464 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:13.464 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:13.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6ae9ec9640 10 monclient: _renew_subs 2024-04-26T00:25:13.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6ae9ec9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:13.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6ae9ec9640 1 -- 172.21.15.37:0/2225062992 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ae411dd00 con 0x7f6ae4135490 2024-04-26T00:25:13.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6ae3fff640 1 --2- 172.21.15.37:0/2225062992 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ae4135490 0x7f6ae4132010 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6ad400e930 tx=0x7f6ad400edf0 comp rx=0 tx=0).queue_received will delay after 1.71409e+09 on 0x7f6ad4048c40 osd_map(205..205 src has 172..205) 2024-04-26T00:25:13.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6ae9ec9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:13.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6ae9ec9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:13.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6ae9ec9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:13.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6acbfff640 1 -- 172.21.15.37:0/2225062992 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f6ad401a070 con 0x7f6ae4135490 2024-04-26T00:25:13.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6ae9ec9640 1 -- 172.21.15.37:0/2225062992 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ae4066f40 con 0x7f6ae4135490 2024-04-26T00:25:13.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.469+0000 7f6acbfff640 1 --2- 172.21.15.37:0/2225062992 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6ad0035eb0 0x7f6ad0038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:13.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.473+0000 7f6ae37fe640 1 --2- 172.21.15.37:0/2225062992 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6ad0035eb0 0x7f6ad0038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:13.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.473+0000 7f6ae37fe640 10 monclient: get_auth_request con 0x7f6ad0035eb0 auth_method 0 2024-04-26T00:25:13.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.473+0000 7f6ae37fe640 1 --2- 172.21.15.37:0/2225062992 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6ad0035eb0 0x7f6ad0038370 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6adc0ab000 tx=0x7f6adc0a7070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:13.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:13.477+0000 7f6acbfff640 1 -- 172.21.15.37:0/2225062992 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(205..205 src has 172..205) ==== 3975+0+0 (secure 0 0 0) 0x7f6ad4048c40 con 0x7f6ae4135490 2024-04-26T00:25:13.882 INFO:tasks.rados.rados.0.smithi045.stdout:857: done (0 left) 2024-04-26T00:25:13.883 INFO:tasks.rados.rados.0.smithi045.stdout:858: copy_from oid 47 from oid 7 current snap is 102 2024-04-26T00:25:13.883 INFO:tasks.rados.rados.0.smithi045.stdout:859: snap_remove snap 67 2024-04-26T00:25:14.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:14.297+0000 7f6acbfff640 1 -- 172.21.15.37:0/2225062992 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6ad4047960 con 0x7f6ae4135490 2024-04-26T00:25:14.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:14.297+0000 7f6acbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:14.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:14.297+0000 7f6acbfff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:25:14.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:14.529+0000 7f6ae9ec9640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-26T00:25:14.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:14.529+0000 7f6ae9ec9640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-26T00:25:14.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:14.529+0000 7f6ae9ec9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:14.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:14.529+0000 7f6ae9ec9640 1 -- 172.21.15.37:0/2225062992 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f6ae406b9a0 con 0x7f6ae4135490 2024-04-26T00:25:14.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:14.533+0000 7f6acbfff640 1 -- 172.21.15.37:0/2225062992 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f6ad401a520 con 0x7f6ae4135490 2024-04-26T00:25:14.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:14.533+0000 7f6acbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-26T00:25:14.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:14.533+0000 7f6acbfff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:25:14.514 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.532+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:14.514 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.532+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=13,config=2,health=11,kv=9,logm=65} crc {auth=2491421479,config=692619927,health=2113251397,kv=2491615610,logm=4232645574}) 2024-04-26T00:25:14.514 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.532+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=1,health=11,kv=7,logm=68} crc {auth=2491421479,config=3678701417,health=1430206638,kv=1880801148,logm=59317421}) 2024-04-26T00:25:14.514 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.532+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:14.514 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.532+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=13,config=2,health=11,kv=9,logm=65} crc {auth=2491421479,config=692619927,health=2113251397,kv=2491615610,logm=4232645574}) 2024-04-26T00:25:14.514 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.532+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=12,config=2,health=11,kv=9,logm=66} crc {auth=3852090976,config=692619927,health=2113251397,kv=789508982,logm=1112282263}) 2024-04-26T00:25:14.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:14.533+0000 7f6ae9ec9640 1 -- 172.21.15.37:0/2225062992 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6ad0035eb0 msgr2=0x7f6ad0038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:14.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:14.533+0000 7f6ae9ec9640 1 --2- 172.21.15.37:0/2225062992 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6ad0035eb0 0x7f6ad0038370 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6adc0ab000 tx=0x7f6adc0a7070 comp rx=0 tx=0).stop 2024-04-26T00:25:14.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:14.533+0000 7f6ae9ec9640 10 monclient: shutdown 2024-04-26T00:25:14.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:14.533+0000 7f6ae9ec9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:14.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:14.533+0000 7f6ae9ec9640 1 -- 172.21.15.37:0/2225062992 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ae4135490 msgr2=0x7f6ae4132010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:14.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:14.533+0000 7f6ae9ec9640 1 --2- 172.21.15.37:0/2225062992 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ae4135490 0x7f6ae4132010 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6ad400e930 tx=0x7f6ad400edf0 comp rx=0 tx=0).stop 2024-04-26T00:25:14.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:14.533+0000 7f6ae9ec9640 1 -- 172.21.15.37:0/2225062992 shutdown_connections 2024-04-26T00:25:14.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:14.533+0000 7f6ae9ec9640 1 --2- 172.21.15.37:0/2225062992 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6ad0035eb0 0x7f6ad0038370 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:14.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:14.533+0000 7f6ae9ec9640 1 --2- 172.21.15.37:0/2225062992 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ae4135490 0x7f6ae4132010 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:14.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:14.533+0000 7f6ae9ec9640 1 --2- 172.21.15.37:0/2225062992 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6ae4134b30 0x7f6ae4131ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:14.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:14.533+0000 7f6ae9ec9640 1 --2- 172.21.15.37:0/2225062992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ae40a0260 0x7f6ae41cec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:14.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:14.533+0000 7f6ae9ec9640 1 -- 172.21.15.37:0/2225062992 >> 172.21.15.37:0/2225062992 conn(0x7f6ae40a1ad0 msgr2=0x7f6ae40a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:14.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:14.533+0000 7f6ae9ec9640 1 -- 172.21.15.37:0/2225062992 shutdown_connections 2024-04-26T00:25:14.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:14.533+0000 7f6ae9ec9640 1 -- 172.21.15.37:0/2225062992 wait complete. 2024-04-26T00:25:14.518 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.536+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:14.518 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.536+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3716578950}) 2024-04-26T00:25:14.518 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.536+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3246735545}) 2024-04-26T00:25:14.518 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.536+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:14.518 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.536+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3716578950}) 2024-04-26T00:25:14.518 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.536+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=245367634}) 2024-04-26T00:25:14.518 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.536+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:14.518 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.536+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=418259420}) 2024-04-26T00:25:14.518 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.536+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3771923295}) 2024-04-26T00:25:14.519 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.536+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:14.519 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.536+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=418259420}) 2024-04-26T00:25:14.519 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.536+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=533670366}) 2024-04-26T00:25:14.520 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.540+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:14.520 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.540+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1718519148}) 2024-04-26T00:25:14.520 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.540+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=490370290}) 2024-04-26T00:25:14.520 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.540+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:14.520 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.540+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1718519148}) 2024-04-26T00:25:14.521 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.540+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3654217392}) 2024-04-26T00:25:14.522 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.540+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:14.522 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.540+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2940901038}) 2024-04-26T00:25:14.522 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.540+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2745378768}) 2024-04-26T00:25:14.522 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.540+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:14.522 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.540+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2940901038}) 2024-04-26T00:25:14.523 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.540+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3421363552}) 2024-04-26T00:25:14.524 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.544+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:14.524 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.544+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=28196716}) 2024-04-26T00:25:14.524 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.544+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2799856118}) 2024-04-26T00:25:14.524 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.544+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:14.524 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.544+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=28196716}) 2024-04-26T00:25:14.524 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.544+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2140288046}) 2024-04-26T00:25:14.526 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.544+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:14.526 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.544+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2143880130}) 2024-04-26T00:25:14.526 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.544+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1672068865}) 2024-04-26T00:25:14.526 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.544+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:14.526 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.544+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2143880130}) 2024-04-26T00:25:14.526 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.544+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=343608684}) 2024-04-26T00:25:14.528 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.548+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:14.528 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.548+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1268989120}) 2024-04-26T00:25:14.528 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.548+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=198550225}) 2024-04-26T00:25:14.528 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.548+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:14.528 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.548+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1268989120}) 2024-04-26T00:25:14.528 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.548+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2829155683}) 2024-04-26T00:25:14.530 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.548+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:14.530 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.548+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=729123260}) 2024-04-26T00:25:14.530 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.548+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4095278419}) 2024-04-26T00:25:14.530 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.548+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:14.530 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.548+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=729123260}) 2024-04-26T00:25:14.530 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.548+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3191005588}) 2024-04-26T00:25:14.535 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-26T00:25:14.860 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.880+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:14.860 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.880+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1498163517}) 2024-04-26T00:25:14.860 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.880+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=891121675}) 2024-04-26T00:25:14.860 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.880+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:14.860 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.880+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1498163517}) 2024-04-26T00:25:14.860 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.880+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=799150127}) 2024-04-26T00:25:14.862 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.880+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:14.862 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.880+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4251268518}) 2024-04-26T00:25:14.862 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.880+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4012084160}) 2024-04-26T00:25:14.862 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.880+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:14.862 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.880+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4251268518}) 2024-04-26T00:25:14.862 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.880+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2665223689}) 2024-04-26T00:25:14.865 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.884+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:14.865 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.884+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2748190853}) 2024-04-26T00:25:14.865 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.884+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1432650988}) 2024-04-26T00:25:14.865 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.884+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:14.865 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.884+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2748190853}) 2024-04-26T00:25:14.865 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.884+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2811952147}) 2024-04-26T00:25:14.867 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.884+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:14.867 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.884+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=937037275}) 2024-04-26T00:25:14.867 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.884+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2315041618}) 2024-04-26T00:25:14.867 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.884+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:14.867 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.884+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=937037275}) 2024-04-26T00:25:14.867 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.884+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2158032240}) 2024-04-26T00:25:14.870 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.888+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:14.870 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.888+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=75,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=1} crc {logm=3037226571,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=466568511,mon_config_key=686720761}) 2024-04-26T00:25:14.870 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.888+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=71,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=6} crc {logm=263294127,mdsmap=2840952586,mgr=3887433460,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=2053329023,mon_config_key=918105881}) 2024-04-26T00:25:14.870 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.888+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:14.870 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.888+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=75,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=1} crc {logm=3037226571,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=466568511,mon_config_key=686720761}) 2024-04-26T00:25:14.870 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.888+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=72,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5} crc {logm=365242683,mdsmap=2840952586,mgr=1214999089,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=2053329023,mon_config_key=4289695955}) 2024-04-26T00:25:14.872 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.892+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:14.873 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.892+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=32,osdmap=48} crc {mon_config_key=686164493,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=2234204405,osd_snap=1987593853,osdmap=4056158456}) 2024-04-26T00:25:14.873 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.892+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=8,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=34,osdmap=49} crc {mon_config_key=1591984771,monmap=3876866194,osd_metadata=839604482,osd_pg_creating=2234204405,osd_snap=1393761665,osdmap=1539992904}) 2024-04-26T00:25:14.873 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.892+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:14.873 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.892+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=32,osdmap=48} crc {mon_config_key=686164493,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=2234204405,osd_snap=1987593853,osdmap=4056158456}) 2024-04-26T00:25:14.873 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.892+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=10,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=31,osdmap=50} crc {mon_config_key=985676101,monmap=3876866194,osd_metadata=1980645665,osd_pg_creating=2234204405,osd_snap=2556890866,osdmap=3671098033}) 2024-04-26T00:25:14.881 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.900+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:14.881 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.900+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=22} crc {osdmap=2168365606}) 2024-04-26T00:25:14.881 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.900+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=22} crc {osdmap=3784840083}) 2024-04-26T00:25:14.881 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.900+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:14.881 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.900+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=22} crc {osdmap=2168365606}) 2024-04-26T00:25:14.881 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:14.900+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=23} crc {osdmap=3780694760}) 2024-04-26T00:25:14.888 INFO:tasks.rados.rados.0.smithi045.stdout:859: done (1 left) 2024-04-26T00:25:14.888 INFO:tasks.rados.rados.0.smithi045.stdout:860: snap_create 2024-04-26T00:25:14.888 INFO:tasks.rados.rados.0.smithi045.stdout:858: finishing copy_from to smithi04526579-47 2024-04-26T00:25:14.888 INFO:tasks.rados.rados.0.smithi045.stdout:858: got expected ENOENT (src dne) 2024-04-26T00:25:14.888 INFO:tasks.rados.rados.0.smithi045.stdout:858: finishing copy_from racing read to smithi04526579-47 2024-04-26T00:25:15.536 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:25:15.536 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:25:15.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.685+0000 7ffbb8a12640 10 monclient: get_monmap_and_config 2024-04-26T00:25:15.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.685+0000 7ffbb8a12640 10 monclient: build_initial_monmap 2024-04-26T00:25:15.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.685+0000 7ffbb8a12640 10 monclient: monmap: 2024-04-26T00:25:15.667 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:15.667 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:15.667 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:25:15.689692+0000 2024-04-26T00:25:15.667 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:25:15.689692+0000 2024-04-26T00:25:15.667 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:15.667 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:15.667 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:25:15.667 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:25:15.667 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:25:15.667 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:25:15.667 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:25:15.668 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:15.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.685+0000 7ffbb8a12640 1 Processor -- start 2024-04-26T00:25:15.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.685+0000 7ffbb8a12640 1 -- start start 2024-04-26T00:25:15.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.685+0000 7ffbb8a12640 10 monclient: init 2024-04-26T00:25:15.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffbb8a12640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:15.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffbb8a12640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-04-26T00:25:15.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffbb8a12640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ffbb4136da0 0x7ffbb409bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:15.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffbb8a12640 10 monclient(hunting): picked mon.noname-c con 0x7ffbb4136da0 addr v2:172.21.15.45:3300/0 2024-04-26T00:25:15.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffbb8a12640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ffbb409c410 0x7ffbb4137180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:15.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffbb8a12640 10 monclient(hunting): picked mon.noname-a con 0x7ffbb409c410 addr v2:172.21.15.37:3300/0 2024-04-26T00:25:15.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffbb8a12640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb41376c0 0x7ffbb4133b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:15.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffbb8a12640 10 monclient(hunting): picked mon.noname-b con 0x7ffbb41376c0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:15.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffbb8a12640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:15.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffbb8a12640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7ffbb409b210 con 0x7ffbb409c410 2024-04-26T00:25:15.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffbb8a12640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:15.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffbb8a12640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7ffbb409b4f0 con 0x7ffbb4136da0 2024-04-26T00:25:15.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffbb8a12640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:15.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffbb8a12640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ffbb409b7d0 con 0x7ffbb41376c0 2024-04-26T00:25:15.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffbb8a12640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:15.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffbb8a12640 10 monclient(hunting): authenticate will time out at 1296.996126s 2024-04-26T00:25:15.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffba9d9b640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ffbb409c410 0x7ffbb4137180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:15.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffbb2d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb41376c0 0x7ffbb4133b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:15.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffba9d9b640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ffbb409c410 0x7ffbb4137180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36454/0 (socket says 172.21.15.37:36454) 2024-04-26T00:25:15.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffba9d9b640 1 -- 172.21.15.37:0/4254749026 learned_addr learned my addr 172.21.15.37:0/4254749026 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:15.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffba9d9b640 10 monclient(hunting): get_auth_request con 0x7ffbb409c410 auth_method 0 2024-04-26T00:25:15.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffba9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:15.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffba9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:15.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffba9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:15.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffbb2d9d640 10 monclient(hunting): get_auth_request con 0x7ffbb41376c0 auth_method 0 2024-04-26T00:25:15.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffbb2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:15.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffbb2d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:15.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffbb2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:15.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffbb2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:15.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffbb2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:15.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffbb2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:15.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffba9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:15.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffba9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:15.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffbb259c640 1 --2- 172.21.15.37:0/4254749026 >> v2:172.21.15.45:3300/0 conn(0x7ffbb4136da0 0x7ffbb409bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:15.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffba9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:15.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffbb259c640 10 monclient(hunting): get_auth_request con 0x7ffbb4136da0 auth_method 0 2024-04-26T00:25:15.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffbb259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:15.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffbb259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:15.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffbb259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:15.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffbb2d9d640 10 monclient(hunting): handle_auth_done global_id 64130 payload 274 2024-04-26T00:25:15.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffbb2d9d640 1 -- 172.21.15.37:0/4254749026 >> v2:172.21.15.45:3300/0 conn(0x7ffbb4136da0 msgr2=0x7ffbb409bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:15.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffbb2d9d640 1 --2- 172.21.15.37:0/4254749026 >> v2:172.21.15.45:3300/0 conn(0x7ffbb4136da0 0x7ffbb409bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:15.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffbb2d9d640 1 -- 172.21.15.37:0/4254749026 >> v2:172.21.15.37:3300/0 conn(0x7ffbb409c410 msgr2=0x7ffbb4137180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:15.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffbb2d9d640 1 --2- 172.21.15.37:0/4254749026 >> v2:172.21.15.37:3300/0 conn(0x7ffbb409c410 0x7ffbb4137180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:15.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffbb2d9d640 10 monclient: _finish_hunting 0 2024-04-26T00:25:15.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffbb2d9d640 1 monclient: found mon.noname-b 2024-04-26T00:25:15.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffbb2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:15.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffbb2d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:25:15.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffbb2d9d640 1 -- 172.21.15.37:0/4254749026 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbb4137b00 con 0x7ffbb41376c0 2024-04-26T00:25:15.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.689+0000 7ffba9d9b640 1 --2- 172.21.15.37:0/4254749026 >> v2:172.21.15.37:3300/0 conn(0x7ffbb409c410 0x7ffbb4137180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:25:15.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb2d9d640 1 -- 172.21.15.37:0/4254749026 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb41376c0 msgr2=0x7ffbb4133b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:15.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb2d9d640 1 --2- 172.21.15.37:0/4254749026 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb41376c0 0x7ffbb4133b90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ffbb40a58a0 tx=0x7ffbac01c530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88b570f05d400ecb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:15.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb1d9b640 1 -- 172.21.15.37:0/4254749026 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbac0a8640 con 0x7ffbb41376c0 2024-04-26T00:25:15.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:15.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:25:15.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb1d9b640 10 monclient: dump: 2024-04-26T00:25:15.679 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:15.679 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:15.679 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:15.679 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:15.679 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:15.679 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:15.679 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:15.680 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:15.680 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:15.680 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:15.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb1d9b640 10 monclient: _finish_auth 0 2024-04-26T00:25:15.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb1d9b640 10 monclient: _check_auth_tickets 2024-04-26T00:25:15.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:15.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb1d9b640 1 -- 172.21.15.37:0/4254749026 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbac0a87e0 con 0x7ffbb41376c0 2024-04-26T00:25:15.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb1d9b640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:15.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb1d9b640 1 -- 172.21.15.37:0/4254749026 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbac0a8bb0 con 0x7ffbb41376c0 2024-04-26T00:25:15.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb8a12640 5 monclient: authenticate success, global_id 64130 2024-04-26T00:25:15.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb8a12640 10 monclient: get_monmap_and_config success 2024-04-26T00:25:15.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb8a12640 4 set_mon_vals no callback set 2024-04-26T00:25:15.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:15.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:25:15.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb1d9b640 10 monclient: dump: 2024-04-26T00:25:15.681 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:15.681 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:15.681 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:15.681 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:15.681 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:15.681 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:15.681 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:15.681 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:15.681 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:15.681 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:15.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb8a12640 10 monclient: shutdown 2024-04-26T00:25:15.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb8a12640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:15.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb8a12640 1 -- 172.21.15.37:0/4254749026 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb41376c0 msgr2=0x7ffbb4133b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:15.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb8a12640 1 --2- 172.21.15.37:0/4254749026 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb41376c0 0x7ffbb4133b90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ffbb40a58a0 tx=0x7ffbac01c530 comp rx=0 tx=0).stop 2024-04-26T00:25:15.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb8a12640 1 -- 172.21.15.37:0/4254749026 shutdown_connections 2024-04-26T00:25:15.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb8a12640 1 --2- 172.21.15.37:0/4254749026 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb41376c0 0x7ffbb4133b90 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:15.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb8a12640 1 --2- 172.21.15.37:0/4254749026 >> v2:172.21.15.37:3300/0 conn(0x7ffbb409c410 0x7ffbb4137180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:15.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb8a12640 1 --2- 172.21.15.37:0/4254749026 >> v2:172.21.15.45:3300/0 conn(0x7ffbb4136da0 0x7ffbb409bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:15.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb8a12640 1 -- 172.21.15.37:0/4254749026 >> 172.21.15.37:0/4254749026 conn(0x7ffbb40a5f00 msgr2=0x7ffbb40a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:15.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb8a12640 1 -- 172.21.15.37:0/4254749026 shutdown_connections 2024-04-26T00:25:15.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb8a12640 1 -- 172.21.15.37:0/4254749026 wait complete. 2024-04-26T00:25:15.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb8a12640 10 monclient: build_initial_monmap 2024-04-26T00:25:15.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb8a12640 10 monclient: monmap: 2024-04-26T00:25:15.682 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:15.682 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:25:15.682 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:25:15.683 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:25:15.683 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:15.683 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:25:15.683 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:25:15.683 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:25:15.683 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:25:15.683 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:15.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb8a12640 1 Processor -- start 2024-04-26T00:25:15.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb8a12640 1 -- start start 2024-04-26T00:25:15.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb8a12640 10 monclient: init 2024-04-26T00:25:15.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb8a12640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:15.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb8a12640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:25:15.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb8a12640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb409c410 0x7ffbb409f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:15.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb8a12640 10 monclient(hunting): picked mon.noname-c con 0x7ffbb409c410 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:15.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb8a12640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffbb4136da0 0x7ffbb41cf930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:15.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb8a12640 10 monclient(hunting): picked mon.noname-a con 0x7ffbb4136da0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:25:15.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb8a12640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffbb41376c0 0x7ffbb41cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:15.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb8a12640 10 monclient(hunting): picked mon.noname-b con 0x7ffbb41376c0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:25:15.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb8a12640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:15.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb8a12640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb4140e90 con 0x7ffbb4136da0 2024-04-26T00:25:15.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb8a12640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:15.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb8a12640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb4140d10 con 0x7ffbb41376c0 2024-04-26T00:25:15.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb8a12640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:15.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb8a12640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ffbb4141010 con 0x7ffbb409c410 2024-04-26T00:25:15.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb8a12640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:15.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb8a12640 10 monclient(hunting): authenticate will time out at 1297.003488s 2024-04-26T00:25:15.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffba9d9b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffbb4136da0 0x7ffbb41cf930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:15.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb259c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb409c410 0x7ffbb409f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:15.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffba9d9b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffbb4136da0 0x7ffbb41cf930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36456/0 (socket says 172.21.15.37:36456) 2024-04-26T00:25:15.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffba9d9b640 1 -- 172.21.15.37:0/3398328327 learned_addr learned my addr 172.21.15.37:0/3398328327 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:15.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffba9d9b640 10 monclient(hunting): get_auth_request con 0x7ffbb4136da0 auth_method 0 2024-04-26T00:25:15.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb259c640 1 --2- 172.21.15.37:0/3398328327 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb409c410 0x7ffbb409f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:59794/0 (socket says 172.21.15.37:59794) 2024-04-26T00:25:15.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffba9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:15.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffba9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:15.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffba9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:15.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb259c640 10 monclient(hunting): get_auth_request con 0x7ffbb409c410 auth_method 0 2024-04-26T00:25:15.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:15.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:15.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:15.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:15.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:15.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:15.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffba9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:15.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffba9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:15.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffba9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:15.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb2d9d640 1 --2- 172.21.15.37:0/3398328327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffbb41376c0 0x7ffbb41cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:15.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb2d9d640 10 monclient(hunting): get_auth_request con 0x7ffbb41376c0 auth_method 0 2024-04-26T00:25:15.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:15.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb2d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:15.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.693+0000 7ffbb2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:15.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffbb259c640 10 monclient(hunting): handle_auth_done global_id 64133 payload 824 2024-04-26T00:25:15.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffbb259c640 1 -- 172.21.15.37:0/3398328327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffbb41376c0 msgr2=0x7ffbb41cfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:15.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffbb259c640 1 --2- 172.21.15.37:0/3398328327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffbb41376c0 0x7ffbb41cfe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:15.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffbb259c640 1 -- 172.21.15.37:0/3398328327 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffbb4136da0 msgr2=0x7ffbb41cf930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:15.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffbb259c640 1 --2- 172.21.15.37:0/3398328327 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffbb4136da0 0x7ffbb41cf930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:15.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffbb259c640 10 monclient: _finish_hunting 0 2024-04-26T00:25:15.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffbb259c640 1 monclient: found mon.noname-c 2024-04-26T00:25:15.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffbb259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:15.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffbb259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:25:15.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffbb259c640 1 -- 172.21.15.37:0/3398328327 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbb409ff80 con 0x7ffbb409c410 2024-04-26T00:25:15.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffba9d9b640 1 --2- 172.21.15.37:0/3398328327 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffbb4136da0 0x7ffbb41cf930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:25:15.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffbb259c640 1 -- 172.21.15.37:0/3398328327 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb409c410 msgr2=0x7ffbb409f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:15.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffbb259c640 1 --2- 172.21.15.37:0/3398328327 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb409c410 0x7ffbb409f430 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ffba400d5a0 tx=0x7ffba400da60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:15.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffbab7fe640 1 -- 172.21.15.37:0/3398328327 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba4012800 con 0x7ffbb409c410 2024-04-26T00:25:15.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffbab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:15.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffbab7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:25:15.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffbab7fe640 10 monclient: dump: 2024-04-26T00:25:15.690 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:15.690 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:15.690 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:15.690 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:15.690 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:15.690 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:15.690 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:15.690 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:15.690 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:15.691 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:15.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffbab7fe640 10 monclient: _finish_auth 0 2024-04-26T00:25:15.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffbab7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:25:15.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffbab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:15.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffbb8a12640 5 monclient: authenticate success, global_id 64133 2024-04-26T00:25:15.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffbb8a12640 10 monclient: _renew_subs 2024-04-26T00:25:15.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffbab7fe640 1 -- 172.21.15.37:0/3398328327 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffba4019070 con 0x7ffbb409c410 2024-04-26T00:25:15.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffbb8a12640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:25:15.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffbb8a12640 1 -- 172.21.15.37:0/3398328327 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbb4139360 con 0x7ffbb409c410 2024-04-26T00:25:15.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffbab7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:15.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffbab7fe640 1 -- 172.21.15.37:0/3398328327 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba4012be0 con 0x7ffbb409c410 2024-04-26T00:25:15.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffbab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:15.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffbab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:25:15.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffbab7fe640 10 monclient: dump: 2024-04-26T00:25:15.692 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:15.692 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:15.692 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:15.692 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:15.692 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:15.692 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:15.692 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:15.692 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:15.692 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:15.692 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:15.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffbb159a640 4 set_mon_vals no callback set 2024-04-26T00:25:15.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffbb8a12640 10 monclient: _renew_subs 2024-04-26T00:25:15.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffbb8a12640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:25:15.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffbb8a12640 1 -- 172.21.15.37:0/3398328327 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbb413da10 con 0x7ffbb409c410 2024-04-26T00:25:15.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffbb8a12640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:15.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffbb8a12640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:15.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffbb8a12640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:25:15.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffbab7fe640 1 -- 172.21.15.37:0/3398328327 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ffba401e460 con 0x7ffbb409c410 2024-04-26T00:25:15.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffbb8a12640 1 -- 172.21.15.37:0/3398328327 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbb4066f40 con 0x7ffbb409c410 2024-04-26T00:25:15.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffbab7fe640 1 --2- 172.21.15.37:0/3398328327 >> v2:172.21.15.45:6812/2379918623 conn(0x7ffb94035e80 0x7ffb94038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:15.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.697+0000 7ffbab7fe640 1 -- 172.21.15.37:0/3398328327 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(207..207 src has 172..207) ==== 3991+0+0 (secure 0 0 0) 0x7ffba4048df0 con 0x7ffbb409c410 2024-04-26T00:25:15.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.701+0000 7ffba9d9b640 1 --2- 172.21.15.37:0/3398328327 >> v2:172.21.15.45:6812/2379918623 conn(0x7ffb94035e80 0x7ffb94038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:15.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.701+0000 7ffba9d9b640 10 monclient: get_auth_request con 0x7ffb94035e80 auth_method 0 2024-04-26T00:25:15.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.701+0000 7ffba9d9b640 1 --2- 172.21.15.37:0/3398328327 >> v2:172.21.15.45:6812/2379918623 conn(0x7ffb94035e80 0x7ffb94038340 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ffbb409ce40 tx=0x7ffb9c009240 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:15.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.705+0000 7ffbab7fe640 1 -- 172.21.15.37:0/3398328327 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ffba404c030 con 0x7ffbb409c410 2024-04-26T00:25:15.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.705+0000 7ffbab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:15.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.705+0000 7ffbab7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:25:15.890 INFO:tasks.rados.rados.0.smithi045.stdout:858: done (1 left) 2024-04-26T00:25:15.890 INFO:tasks.rados.rados.0.smithi045.stdout:860: done (0 left) 2024-04-26T00:25:15.890 INFO:tasks.rados.rados.0.smithi045.stdout:861: read oid 10 snap -1 2024-04-26T00:25:15.890 INFO:tasks.rados.rados.0.smithi045.stdout:861: expect (ObjNum 87 snap 19 seq_num 87) 2024-04-26T00:25:15.891 INFO:tasks.rados.rados.0.smithi045.stdout:862: snap_create 2024-04-26T00:25:15.909 INFO:tasks.rados.rados.0.smithi045.stdout:861: expect (ObjNum 87 snap 19 seq_num 87) 2024-04-26T00:25:15.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.929+0000 7ffbb8a12640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:25:15.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.929+0000 7ffbb8a12640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:25:15.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.929+0000 7ffbb8a12640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:25:15.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.929+0000 7ffbb8a12640 1 -- 172.21.15.37:0/3398328327 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ffbb4075d70 con 0x7ffbb409c410 2024-04-26T00:25:15.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.929+0000 7ffbab7fe640 1 -- 172.21.15.37:0/3398328327 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7ffba401e800 con 0x7ffbb409c410 2024-04-26T00:25:15.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.929+0000 7ffbab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:25:15.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.929+0000 7ffbab7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:25:15.911 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:25:15.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.933+0000 7ffbb8a12640 1 -- 172.21.15.37:0/3398328327 >> v2:172.21.15.45:6812/2379918623 conn(0x7ffb94035e80 msgr2=0x7ffb94038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:15.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.933+0000 7ffbb8a12640 1 --2- 172.21.15.37:0/3398328327 >> v2:172.21.15.45:6812/2379918623 conn(0x7ffb94035e80 0x7ffb94038340 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ffbb409ce40 tx=0x7ffb9c009240 comp rx=0 tx=0).stop 2024-04-26T00:25:15.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.933+0000 7ffbb8a12640 10 monclient: shutdown 2024-04-26T00:25:15.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.933+0000 7ffbb8a12640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:15.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.933+0000 7ffbb8a12640 1 -- 172.21.15.37:0/3398328327 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb409c410 msgr2=0x7ffbb409f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:15.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.933+0000 7ffbb8a12640 1 --2- 172.21.15.37:0/3398328327 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb409c410 0x7ffbb409f430 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ffba400d5a0 tx=0x7ffba400da60 comp rx=0 tx=0).stop 2024-04-26T00:25:15.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.933+0000 7ffbb8a12640 1 -- 172.21.15.37:0/3398328327 shutdown_connections 2024-04-26T00:25:15.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.933+0000 7ffbb8a12640 1 --2- 172.21.15.37:0/3398328327 >> v2:172.21.15.45:6812/2379918623 conn(0x7ffb94035e80 0x7ffb94038340 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:15.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.933+0000 7ffbb8a12640 1 --2- 172.21.15.37:0/3398328327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffbb41376c0 0x7ffbb41cfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:15.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.933+0000 7ffbb8a12640 1 --2- 172.21.15.37:0/3398328327 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffbb4136da0 0x7ffbb41cf930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:15.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.933+0000 7ffbb8a12640 1 --2- 172.21.15.37:0/3398328327 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb409c410 0x7ffbb409f430 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:15.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.933+0000 7ffbb8a12640 1 -- 172.21.15.37:0/3398328327 >> 172.21.15.37:0/3398328327 conn(0x7ffbb40a5f00 msgr2=0x7ffbb41354a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:15.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.933+0000 7ffbb8a12640 1 -- 172.21.15.37:0/3398328327 shutdown_connections 2024-04-26T00:25:15.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:15.933+0000 7ffbb8a12640 1 -- 172.21.15.37:0/3398328327 wait complete. 2024-04-26T00:25:15.930 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:25:15.930 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-26T00:25:15.930 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:25:15.931 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:25:16.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e5bdda640 10 monclient: get_monmap_and_config 2024-04-26T00:25:16.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e5bdda640 10 monclient: build_initial_monmap 2024-04-26T00:25:16.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e5bdda640 10 monclient: monmap: 2024-04-26T00:25:16.054 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:16.054 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:16.054 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:25:16.076999+0000 2024-04-26T00:25:16.054 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:25:16.076999+0000 2024-04-26T00:25:16.055 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:16.055 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:16.055 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:25:16.055 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:25:16.055 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:25:16.055 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:25:16.055 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:25:16.055 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:16.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e5bdda640 1 Processor -- start 2024-04-26T00:25:16.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e5bdda640 1 -- start start 2024-04-26T00:25:16.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e5bdda640 10 monclient: init 2024-04-26T00:25:16.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e5bdda640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:16.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e5bdda640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-04-26T00:25:16.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e5bdda640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f3e540a0300 0x7f3e5409fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:16.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e5bdda640 10 monclient(hunting): picked mon.noname-a con 0x7f3e540a0300 addr v2:172.21.15.37:3300/0 2024-04-26T00:25:16.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e5bdda640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3e540a06e0 addr v1:172.21.15.45:6789/0 2024-04-26T00:25:16.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e5bdda640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3e541352b0 0x7f3e5413cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:16.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e5bdda640 10 monclient(hunting): picked mon.noname-b con 0x7f3e541352b0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:16.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e5bdda640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e5409c190 con 0x7f3e540a06e0 2024-04-26T00:25:16.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e5bdda640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:16.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e5bdda640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f3e540a13f0 con 0x7f3e540a0300 2024-04-26T00:25:16.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e5bdda640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:16.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e5bdda640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f3e540a1110 con 0x7f3e541352b0 2024-04-26T00:25:16.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e5bdda640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:16.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e5bdda640 10 monclient(hunting): authenticate will time out at 1297.382683s 2024-04-26T00:25:16.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e59b76640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f3e540a0300 0x7f3e5409fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:16.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e5a377640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3e541352b0 0x7f3e5413cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:16.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e59b76640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f3e540a0300 0x7f3e5409fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36460/0 (socket says 172.21.15.37:36460) 2024-04-26T00:25:16.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e59b76640 1 -- 172.21.15.37:0/1451140111 learned_addr learned my addr 172.21.15.37:0/1451140111 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:16.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e59b76640 10 monclient(hunting): get_auth_request con 0x7f3e540a0300 auth_method 0 2024-04-26T00:25:16.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e59b76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:16.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e59b76640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:16.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e59b76640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:16.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e5a377640 10 monclient(hunting): get_auth_request con 0x7f3e541352b0 auth_method 0 2024-04-26T00:25:16.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e5a377640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:16.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e5a377640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:16.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e5a377640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:16.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e59b76640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:16.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e59b76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:16.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e59b76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:16.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e5a377640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:16.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e5a377640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:16.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e5a377640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:16.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e59b76640 10 monclient(hunting): handle_auth_done global_id 54204 payload 274 2024-04-26T00:25:16.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e59b76640 1 -- 172.21.15.37:0/1451140111 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3e541352b0 msgr2=0x7f3e5413cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:16.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e59b76640 1 --2- 172.21.15.37:0/1451140111 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3e541352b0 0x7f3e5413cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:16.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e59b76640 1 -- 172.21.15.37:0/1451140111 >> v1:172.21.15.45:6789/0 conn(0x7f3e540a06e0 legacy=0x7f3e54134ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:16.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e59b76640 10 monclient: _finish_hunting 0 2024-04-26T00:25:16.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e59b76640 1 monclient: found mon.noname-a 2024-04-26T00:25:16.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e59b76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:16.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e59b76640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:25:16.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e59b76640 1 -- 172.21.15.37:0/1451140111 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e540a16d0 con 0x7f3e540a0300 2024-04-26T00:25:16.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.073+0000 7f3e5a377640 1 --2- 172.21.15.37:0/1451140111 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3e541352b0 0x7f3e5413cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:25:16.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.077+0000 7f3e59b76640 1 -- 172.21.15.37:0/1451140111 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e540a0300 msgr2=0x7f3e5409fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:16.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.077+0000 7f3e59b76640 1 --2- 172.21.15.37:0/1451140111 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e540a0300 0x7f3e5409fc50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c009e50 tx=0x7f3e4c01c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4783a21b0801fcc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:16.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e58b74640 1 -- 172.21.15.37:0/1451140111 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c0a5710 con 0x7f3e540a0300 2024-04-26T00:25:16.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e58b74640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:16.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e58b74640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:25:16.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e58b74640 10 monclient: dump: 2024-04-26T00:25:16.066 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:16.066 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:16.066 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.066 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.066 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:16.066 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:16.066 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:16.066 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:16.066 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:16.066 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:16.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e58b74640 10 monclient: _finish_auth 0 2024-04-26T00:25:16.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e58b74640 10 monclient: _check_auth_tickets 2024-04-26T00:25:16.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e58b74640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:16.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e58b74640 1 -- 172.21.15.37:0/1451140111 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e4c0a58b0 con 0x7f3e540a0300 2024-04-26T00:25:16.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e5bdda640 5 monclient: authenticate success, global_id 54204 2024-04-26T00:25:16.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e5bdda640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:25:16.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e58b74640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:16.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e5bdda640 10 monclient: get_monmap_and_config success 2024-04-26T00:25:16.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e58b74640 1 -- 172.21.15.37:0/1451140111 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c0a5c80 con 0x7f3e540a0300 2024-04-26T00:25:16.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e5bdda640 4 set_mon_vals no callback set 2024-04-26T00:25:16.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e58b74640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:16.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e58b74640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:16.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e58b74640 10 monclient: dump: 2024-04-26T00:25:16.067 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:16.067 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:16.067 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.067 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.068 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:16.068 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:16.068 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:16.068 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:16.068 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:16.068 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:16.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e5bdda640 10 monclient: shutdown 2024-04-26T00:25:16.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e5bdda640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:16.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e5bdda640 1 -- 172.21.15.37:0/1451140111 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e540a0300 msgr2=0x7f3e5409fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:16.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e5bdda640 1 --2- 172.21.15.37:0/1451140111 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e540a0300 0x7f3e5409fc50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c009e50 tx=0x7f3e4c01c8b0 comp rx=0 tx=0).stop 2024-04-26T00:25:16.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e5bdda640 1 -- 172.21.15.37:0/1451140111 shutdown_connections 2024-04-26T00:25:16.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e5bdda640 1 --2- 172.21.15.37:0/1451140111 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3e541352b0 0x7f3e5413cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:16.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e5bdda640 1 --2- 172.21.15.37:0/1451140111 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e540a0300 0x7f3e5409fc50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:16.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e5bdda640 1 -- 172.21.15.37:0/1451140111 >> 172.21.15.37:0/1451140111 conn(0x7f3e540a1d50 msgr2=0x7f3e540a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:16.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e5bdda640 1 -- 172.21.15.37:0/1451140111 shutdown_connections 2024-04-26T00:25:16.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e5bdda640 1 -- 172.21.15.37:0/1451140111 wait complete. 2024-04-26T00:25:16.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e5bdda640 10 monclient: build_initial_monmap 2024-04-26T00:25:16.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e5bdda640 10 monclient: monmap: 2024-04-26T00:25:16.069 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:16.069 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:25:16.069 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:25:16.069 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:25:16.069 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:16.069 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:25:16.069 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:25:16.069 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:25:16.069 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:25:16.069 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:16.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e5bdda640 1 Processor -- start 2024-04-26T00:25:16.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e5bdda640 1 -- start start 2024-04-26T00:25:16.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e5bdda640 10 monclient: init 2024-04-26T00:25:16.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e5bdda640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:16.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e5bdda640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:25:16.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e5bdda640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e540a0300 0x7f3e5413c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:16.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e5bdda640 10 monclient(hunting): picked mon.noname-a con 0x7f3e540a0300 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:25:16.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e5bdda640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3e540a06e0 0x7f3e5412dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:16.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e5bdda640 10 monclient(hunting): picked mon.noname-c con 0x7f3e540a06e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:16.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e5bdda640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e541352b0 0x7f3e5412e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:16.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e5bdda640 10 monclient(hunting): picked mon.noname-b con 0x7f3e541352b0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:25:16.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e5bdda640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:16.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e5bdda640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f3e5413fdc0 con 0x7f3e540a0300 2024-04-26T00:25:16.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e5bdda640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:16.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e5bdda640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3e5413fc40 con 0x7f3e541352b0 2024-04-26T00:25:16.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e5bdda640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:16.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e5bdda640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f3e5413ff40 con 0x7f3e540a06e0 2024-04-26T00:25:16.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e5bdda640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:16.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e5bdda640 10 monclient(hunting): authenticate will time out at 1297.391151s 2024-04-26T00:25:16.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e59b76640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e540a0300 0x7f3e5413c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:16.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e59375640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3e540a06e0 0x7f3e5412dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:16.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e59b76640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e540a0300 0x7f3e5413c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36476/0 (socket says 172.21.15.37:36476) 2024-04-26T00:25:16.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e59375640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3e540a06e0 0x7f3e5412dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:59802/0 (socket says 172.21.15.37:59802) 2024-04-26T00:25:16.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e59b76640 1 -- 172.21.15.37:0/3459769324 learned_addr learned my addr 172.21.15.37:0/3459769324 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:16.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e59b76640 10 monclient(hunting): get_auth_request con 0x7f3e540a0300 auth_method 0 2024-04-26T00:25:16.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e59b76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:16.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e59b76640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:16.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e59b76640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:16.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e59375640 10 monclient(hunting): get_auth_request con 0x7f3e540a06e0 auth_method 0 2024-04-26T00:25:16.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e59375640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:16.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e59375640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:16.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e59375640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:16.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e59375640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:16.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e59375640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:16.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e59375640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:16.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e59b76640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:16.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e59b76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:16.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e59b76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:16.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e5a377640 1 --2- 172.21.15.37:0/3459769324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e541352b0 0x7f3e5412e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:16.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e5a377640 10 monclient(hunting): get_auth_request con 0x7f3e541352b0 auth_method 0 2024-04-26T00:25:16.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e5a377640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:16.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e5a377640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:16.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e5a377640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:16.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e59375640 10 monclient(hunting): handle_auth_done global_id 64139 payload 824 2024-04-26T00:25:16.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e59375640 1 -- 172.21.15.37:0/3459769324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e541352b0 msgr2=0x7f3e5412e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:16.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e59375640 1 --2- 172.21.15.37:0/3459769324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e541352b0 0x7f3e5412e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:16.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e59375640 1 -- 172.21.15.37:0/3459769324 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e540a0300 msgr2=0x7f3e5413c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:16.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e59375640 1 --2- 172.21.15.37:0/3459769324 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e540a0300 0x7f3e5413c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:16.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e59375640 10 monclient: _finish_hunting 0 2024-04-26T00:25:16.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e59375640 1 monclient: found mon.noname-c 2024-04-26T00:25:16.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e59375640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:16.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e59375640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:25:16.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.081+0000 7f3e59375640 1 -- 172.21.15.37:0/3459769324 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e5412e800 con 0x7f3e540a06e0 2024-04-26T00:25:16.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e59375640 1 -- 172.21.15.37:0/3459769324 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3e540a06e0 msgr2=0x7f3e5412dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:16.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e59375640 1 --2- 172.21.15.37:0/3459769324 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3e540a06e0 0x7f3e5412dd80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3e4800c3e0 tx=0x7f3e4800c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:16.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e527fc640 1 -- 172.21.15.37:0/3459769324 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e480119a0 con 0x7f3e540a06e0 2024-04-26T00:25:16.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:16.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e527fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:25:16.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e527fc640 10 monclient: dump: 2024-04-26T00:25:16.076 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:16.076 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:16.076 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.076 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.076 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:16.076 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:16.076 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:16.076 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:16.076 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:16.076 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:16.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e527fc640 10 monclient: _finish_auth 0 2024-04-26T00:25:16.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e527fc640 10 monclient: _check_auth_tickets 2024-04-26T00:25:16.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:16.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e527fc640 1 -- 172.21.15.37:0/3459769324 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e48017070 con 0x7f3e540a06e0 2024-04-26T00:25:16.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e527fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:16.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e527fc640 1 -- 172.21.15.37:0/3459769324 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e48011d80 con 0x7f3e540a06e0 2024-04-26T00:25:16.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e5bdda640 5 monclient: authenticate success, global_id 64139 2024-04-26T00:25:16.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e5bdda640 10 monclient: _renew_subs 2024-04-26T00:25:16.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e5bdda640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:25:16.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e537fe640 4 set_mon_vals no callback set 2024-04-26T00:25:16.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e5bdda640 1 -- 172.21.15.37:0/3459769324 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e54070ee0 con 0x7f3e540a06e0 2024-04-26T00:25:16.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:16.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e527fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:25:16.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e527fc640 10 monclient: dump: 2024-04-26T00:25:16.077 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:16.077 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:16.077 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.078 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.078 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:16.078 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:16.078 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:16.078 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:16.078 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:16.078 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:16.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e5bdda640 10 monclient: _renew_subs 2024-04-26T00:25:16.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e5bdda640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:25:16.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e5bdda640 1 -- 172.21.15.37:0/3459769324 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e5413bab0 con 0x7f3e540a06e0 2024-04-26T00:25:16.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e5bdda640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:16.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e5bdda640 1 --2- 172.21.15.37:0/3459769324 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e5413b270 0x7f3e541cf210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:16.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e5bdda640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:16.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e5bdda640 1 -- 172.21.15.37:0/3459769324 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f3e5404f510 con 0x7f3e5413b270 2024-04-26T00:25:16.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e59b76640 1 --2- 172.21.15.37:0/3459769324 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e5413b270 0x7f3e541cf210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:16.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e59b76640 10 monclient: get_auth_request con 0x7f3e5413b270 auth_method 0 2024-04-26T00:25:16.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e59b76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:16.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e59b76640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:16.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e59b76640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:16.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e527fc640 1 -- 172.21.15.37:0/3459769324 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f3e4801c460 con 0x7f3e540a06e0 2024-04-26T00:25:16.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e59b76640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:16.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e59b76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:16.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e59b76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:16.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e527fc640 1 --2- 172.21.15.37:0/3459769324 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3e40035ed0 0x7f3e40038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:16.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e59b76640 10 monclient(hunting): handle_auth_done global_id 54210 payload 824 2024-04-26T00:25:16.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e527fc640 1 -- 172.21.15.37:0/3459769324 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(208..208 src has 172..208) ==== 3963+0+0 (secure 0 0 0) 0x7f3e480488b0 con 0x7f3e540a06e0 2024-04-26T00:25:16.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e59b76640 1 -- 172.21.15.37:0/3459769324 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3e5409eb60 con 0x7f3e5413b270 2024-04-26T00:25:16.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e5a377640 1 --2- 172.21.15.37:0/3459769324 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3e40035ed0 0x7f3e40038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:16.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.085+0000 7f3e5a377640 10 monclient: get_auth_request con 0x7f3e40035ed0 auth_method 0 2024-04-26T00:25:16.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.089+0000 7f3e59b76640 1 -- 172.21.15.37:0/3459769324 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e5413b270 msgr2=0x7f3e541cf210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:16.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.089+0000 7f3e59b76640 1 --2- 172.21.15.37:0/3459769324 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e5413b270 0x7f3e541cf210 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c01c6e0 tx=0x7f3e4c019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:16.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.089+0000 7f3e527fc640 1 -- 172.21.15.37:0/3459769324 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c0b6800 con 0x7f3e5413b270 2024-04-26T00:25:16.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.089+0000 7f3e527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:16.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.089+0000 7f3e527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:16.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.089+0000 7f3e527fc640 10 monclient: dump: 2024-04-26T00:25:16.080 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:16.080 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:16.080 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.080 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.081 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:16.081 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:16.081 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:16.081 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:16.081 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:16.081 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:16.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.089+0000 7f3e527fc640 1 -- 172.21.15.37:0/3459769324 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3e4c0b6800 con 0x7f3e5413b270 2024-04-26T00:25:16.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.089+0000 7f3e527fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:16.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.089+0000 7f3e527fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:25:16.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.089+0000 7f3e527fc640 1 -- 172.21.15.37:0/3459769324 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e5413b270 msgr2=0x7f3e541cf210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:16.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.089+0000 7f3e527fc640 1 --2- 172.21.15.37:0/3459769324 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e5413b270 0x7f3e541cf210 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c01c6e0 tx=0x7f3e4c019650 comp rx=0 tx=0).stop 2024-04-26T00:25:16.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.089+0000 7f3e527fc640 1 -- 172.21.15.37:0/3459769324 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e5413b270 msgr2=0x7f3e541cf210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:25:16.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.089+0000 7f3e527fc640 1 --2- 172.21.15.37:0/3459769324 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e5413b270 0x7f3e541cf210 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:16.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.089+0000 7f3e5a377640 1 --2- 172.21.15.37:0/3459769324 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3e40035ed0 0x7f3e40038390 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3e44009580 tx=0x7f3e44008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:16.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.097+0000 7f3e5bdda640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:25:16.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.097+0000 7f3e5bdda640 1 --2- 172.21.15.37:0/3459769324 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e541cf750 0x7f3e541cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:16.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.097+0000 7f3e5bdda640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:16.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.097+0000 7f3e5bdda640 1 -- 172.21.15.37:0/3459769324 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f3e541d00a0 con 0x7f3e541cf750 2024-04-26T00:25:16.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.097+0000 7f3e59b76640 1 --2- 172.21.15.37:0/3459769324 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e541cf750 0x7f3e541cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:16.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.097+0000 7f3e59b76640 10 monclient: get_auth_request con 0x7f3e541cf750 auth_method 0 2024-04-26T00:25:16.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.097+0000 7f3e59b76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:16.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.097+0000 7f3e59b76640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:16.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.097+0000 7f3e59b76640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:16.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.097+0000 7f3e59b76640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:16.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.097+0000 7f3e59b76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:16.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.097+0000 7f3e59b76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:16.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.097+0000 7f3e59b76640 10 monclient(hunting): handle_auth_done global_id 54213 payload 824 2024-04-26T00:25:16.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.097+0000 7f3e59b76640 1 -- 172.21.15.37:0/3459769324 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3e5413b650 con 0x7f3e541cf750 2024-04-26T00:25:16.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.097+0000 7f3e59b76640 1 -- 172.21.15.37:0/3459769324 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e541cf750 msgr2=0x7f3e541cfb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:16.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.097+0000 7f3e59b76640 1 --2- 172.21.15.37:0/3459769324 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e541cf750 0x7f3e541cfb60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c01c8f0 tx=0x7f3e4c0a7000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:16.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.097+0000 7f3e527fc640 1 -- 172.21.15.37:0/3459769324 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c0c5b90 con 0x7f3e541cf750 2024-04-26T00:25:16.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.097+0000 7f3e527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:16.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.097+0000 7f3e527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:16.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.097+0000 7f3e527fc640 10 monclient: dump: 2024-04-26T00:25:16.084 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:16.084 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:16.084 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.084 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.085 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:16.085 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:16.085 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:16.085 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:16.085 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:16.085 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:16.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.097+0000 7f3e527fc640 1 -- 172.21.15.37:0/3459769324 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4716 (secure 0 0 0) 0x7f3e4c0c0440 con 0x7f3e541cf750 2024-04-26T00:25:16.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.097+0000 7f3e527fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:25:16.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.097+0000 7f3e527fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:25:16.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.097+0000 7f3e527fc640 1 -- 172.21.15.37:0/3459769324 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e541cf750 msgr2=0x7f3e541cfb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:16.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.097+0000 7f3e527fc640 1 --2- 172.21.15.37:0/3459769324 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e541cf750 0x7f3e541cfb60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c01c8f0 tx=0x7f3e4c0a7000 comp rx=0 tx=0).stop 2024-04-26T00:25:16.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.097+0000 7f3e527fc640 1 -- 172.21.15.37:0/3459769324 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e541cf750 msgr2=0x7f3e541cfb60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:25:16.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.097+0000 7f3e527fc640 1 --2- 172.21.15.37:0/3459769324 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e541cf750 0x7f3e541cfb60 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:16.086 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:25:16.086 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:25:16.086 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:25:16.086 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:25:16.086 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 40, 2024-04-26T00:25:16.086 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 79423, 2024-04-26T00:25:16.086 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:25:16.086 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:25:16.086 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:25:16.086 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:25:16.086 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:16.086 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 3, 2024-04-26T00:25:16.086 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:25:16.086 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:25:16.087 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:25:16.087 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:16.087 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:16.087 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:16.087 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:16.087 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:16.087 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:16.087 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:16.087 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:16.087 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:16.087 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:16.087 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:16.087 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:16.087 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:25:16.087 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:25:16.087 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:16.088 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:16.088 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:16.088 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:16.088 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:16.088 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:16.088 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:16.088 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:16.088 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:16.088 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:16.088 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:16.088 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:16.088 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:16.088 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:25:16.088 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:25:16.088 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:25:16.088 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:25:16.089 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:25:16.089 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:25:16.089 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:25:16.089 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:25:16.089 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:25:16.089 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:25:16.089 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:25:16.089 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:25:16.089 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:25:16.089 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:25:16.089 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:25:16.089 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:25:16.089 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:25:16.089 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:16.089 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:16.090 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:16.090 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:16.090 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:16.090 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:16.090 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:16.090 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:16.090 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:16.090 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:16.090 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:16.090 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:16.090 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:25:16.090 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:16.091 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:25:16.091 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:16.091 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:25:16.091 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:25:16.091 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:16.091 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:16.091 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:16.091 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:16.091 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:25:16.091 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:16.091 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:16.091 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:16.091 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:16.091 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:25:16.091 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:16.092 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:16.092 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:16.092 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:16.092 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:25:16.092 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:25:16.092 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:16.092 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:16.092 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:16.092 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:16.092 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:16.092 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:25:16.092 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:25:16.092 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:16.092 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:16.093 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:16.093 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:16.093 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:25:16.093 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:16.093 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:16.093 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:16.093 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:16.093 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:25:16.093 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:16.093 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:16.093 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:16.093 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:16.093 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:25:16.093 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:25:16.093 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:16.093 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:16.094 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:16.094 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:16.094 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:16.094 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:25:16.094 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:25:16.094 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:16.094 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:16.094 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:16.094 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:16.094 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:25:16.094 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:16.094 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:16.094 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:16.094 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:16.095 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:25:16.095 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:16.095 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:16.095 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:16.095 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:16.095 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:25:16.095 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:25:16.095 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:16.095 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:16.095 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:16.095 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:16.095 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:16.095 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:16.095 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:25:16.095 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:25:16.096 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:16.096 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:16.096 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:16.096 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:25:16.096 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:16.096 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:16.096 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:25:16.096 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:16.096 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:16.096 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:16.096 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:25:16.096 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:16.096 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:16.096 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:25:16.096 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:16.097 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:16.097 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:16.097 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:25:16.097 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:16.097 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:16.097 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:16.097 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:25:16.097 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:25:16.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.101+0000 7f3e5bdda640 1 -- 172.21.15.37:0/3459769324 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3e40035ed0 msgr2=0x7f3e40038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:16.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.101+0000 7f3e5bdda640 1 --2- 172.21.15.37:0/3459769324 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3e40035ed0 0x7f3e40038390 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3e44009580 tx=0x7f3e44008040 comp rx=0 tx=0).stop 2024-04-26T00:25:16.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.101+0000 7f3e5bdda640 10 monclient: shutdown 2024-04-26T00:25:16.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.101+0000 7f3e5bdda640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:16.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.101+0000 7f3e5bdda640 1 -- 172.21.15.37:0/3459769324 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3e540a06e0 msgr2=0x7f3e5412dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:16.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.101+0000 7f3e5bdda640 1 --2- 172.21.15.37:0/3459769324 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3e540a06e0 0x7f3e5412dd80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3e4800c3e0 tx=0x7f3e4800c8a0 comp rx=0 tx=0).stop 2024-04-26T00:25:16.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.101+0000 7f3e5a377640 1 -- 172.21.15.37:0/3459769324 reap_dead start 2024-04-26T00:25:16.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.101+0000 7f3e5a377640 1 -- 172.21.15.37:0/3459769324 reap_dead start 2024-04-26T00:25:16.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.101+0000 7f3e5bdda640 1 -- 172.21.15.37:0/3459769324 shutdown_connections 2024-04-26T00:25:16.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.101+0000 7f3e5bdda640 1 -- 172.21.15.37:0/3459769324 >> 172.21.15.37:0/3459769324 conn(0x7f3e540a1d50 msgr2=0x7f3e540a3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:16.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.101+0000 7f3e5bdda640 1 -- 172.21.15.37:0/3459769324 shutdown_connections 2024-04-26T00:25:16.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.101+0000 7f3e5bdda640 1 -- 172.21.15.37:0/3459769324 wait complete. 2024-04-26T00:25:16.100 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:25:16.100 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:25:16.100 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:25:16.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.241+0000 7fe310d0d640 10 monclient: get_monmap_and_config 2024-04-26T00:25:16.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.241+0000 7fe310d0d640 10 monclient: build_initial_monmap 2024-04-26T00:25:16.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.241+0000 7fe310d0d640 10 monclient: monmap: 2024-04-26T00:25:16.222 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:16.222 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:16.222 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:25:16.245219+0000 2024-04-26T00:25:16.223 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:25:16.245219+0000 2024-04-26T00:25:16.223 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:16.223 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:16.223 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:25:16.223 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:25:16.223 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:25:16.223 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:25:16.223 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:25:16.223 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:16.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.241+0000 7fe310d0d640 1 Processor -- start 2024-04-26T00:25:16.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.241+0000 7fe310d0d640 1 -- start start 2024-04-26T00:25:16.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.241+0000 7fe310d0d640 10 monclient: init 2024-04-26T00:25:16.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.241+0000 7fe310d0d640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:16.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.241+0000 7fe310d0d640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-04-26T00:25:16.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.241+0000 7fe310d0d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe30c0a2fe0 0x7fe30c136f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:16.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.241+0000 7fe310d0d640 10 monclient(hunting): picked mon.noname-c con 0x7fe30c0a2fe0 addr v2:172.21.15.45:3300/0 2024-04-26T00:25:16.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.241+0000 7fe310d0d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe30c072570 addr v1:172.21.15.37:6789/0 2024-04-26T00:25:16.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.241+0000 7fe310d0d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe30c13c050 addr v1:172.21.15.45:6789/0 2024-04-26T00:25:16.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.241+0000 7fe310d0d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe30c09f010 con 0x7fe30c072570 2024-04-26T00:25:16.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.241+0000 7fe310d0d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe30c137640 con 0x7fe30c13c050 2024-04-26T00:25:16.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.241+0000 7fe310d0d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:16.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.241+0000 7fe310d0d640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fe30c0a33c0 con 0x7fe30c0a2fe0 2024-04-26T00:25:16.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.241+0000 7fe310d0d640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:16.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.241+0000 7fe310d0d640 10 monclient(hunting): authenticate will time out at 1297.551533s 2024-04-26T00:25:16.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.241+0000 7fe309d9b640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fe30c072570 0x7fe30c0729a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:57236/0 (socket says 172.21.15.37:57236) 2024-04-26T00:25:16.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.241+0000 7fe309d9b640 1 -- 172.21.15.37:0/88727310 learned_addr learned my addr 172.21.15.37:0/88727310 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:25:16.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.241+0000 7fe30a59c640 1 --2- 172.21.15.37:0/88727310 >> v2:172.21.15.45:3300/0 conn(0x7fe30c0a2fe0 0x7fe30c136f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:16.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.241+0000 7fe30a59c640 10 monclient(hunting): get_auth_request con 0x7fe30c0a2fe0 auth_method 0 2024-04-26T00:25:16.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.241+0000 7fe30a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:16.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.241+0000 7fe30a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:16.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.241+0000 7fe30a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:16.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:16.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:16.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:16.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe309d9b640 1 -- 172.21.15.37:0/88727310 >> v1:172.21.15.37:6789/0 conn(0x7fe30c072570 legacy=0x7fe30c0729a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:16.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30959a640 1 -- 172.21.15.37:0/88727310 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2906301451 0 0) 0x7fe30c09f010 con 0x7fe30c072570 2024-04-26T00:25:16.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30959a640 10 monclient(hunting): _negotiate 2024-04-26T00:25:16.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30959a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:16.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30959a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:16.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30959a640 10 monclient(hunting): my global_id is 54216 2024-04-26T00:25:16.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30959a640 1 -- 172.21.15.37:0/88727310 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2f4003530 con 0x7fe30c072570 2024-04-26T00:25:16.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30ad9d640 1 -- 172.21.15.37:0/88727310 >> v1:172.21.15.45:6789/0 conn(0x7fe30c13c050 legacy=0x7fe30c13e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:16.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30959a640 1 -- 172.21.15.37:0/88727310 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1359181461 0 0) 0x7fe2f4003530 con 0x7fe30c072570 2024-04-26T00:25:16.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30959a640 1 -- 172.21.15.37:0/88727310 >> v2:172.21.15.45:3300/0 conn(0x7fe30c0a2fe0 msgr2=0x7fe30c136f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:16.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30959a640 1 --2- 172.21.15.37:0/88727310 >> v2:172.21.15.45:3300/0 conn(0x7fe30c0a2fe0 0x7fe30c136f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:16.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30959a640 1 -- 172.21.15.37:0/88727310 >> v1:172.21.15.45:6789/0 conn(0x7fe30c13c050 legacy=0x7fe30c13e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:16.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30959a640 10 monclient: _finish_hunting 0 2024-04-26T00:25:16.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30959a640 1 monclient: found mon.noname-a-legacy 2024-04-26T00:25:16.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:16.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30959a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.37:6789/0 2024-04-26T00:25:16.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30959a640 1 -- 172.21.15.37:0/88727310 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe30c06ea60 con 0x7fe30c072570 2024-04-26T00:25:16.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30959a640 10 monclient: _finish_auth 0 2024-04-26T00:25:16.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30959a640 10 monclient: _check_auth_tickets 2024-04-26T00:25:16.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:16.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30959a640 1 -- 172.21.15.37:0/88727310 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 866326386 0 0) 0x7fe300001930 con 0x7fe30c072570 2024-04-26T00:25:16.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe310d0d640 5 monclient: authenticate success, global_id 54216 2024-04-26T00:25:16.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe310d0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:25:16.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:16.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30959a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-26T00:25:16.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30959a640 10 monclient: dump: 2024-04-26T00:25:16.232 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:16.232 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:16.232 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.232 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.232 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:16.232 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:16.232 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:16.232 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:16.232 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:16.232 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:16.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30959a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] but i'm connected to v1:172.21.15.37:6789/0, reconnecting 2024-04-26T00:25:16.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30959a640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:16.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30959a640 1 -- 172.21.15.37:0/88727310 >> v1:172.21.15.37:6789/0 conn(0x7fe30c072570 legacy=0x7fe30c0729a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:16.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30959a640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:25:16.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30959a640 1 --2- 172.21.15.37:0/88727310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2f4003a90 0x7fe2f4003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:16.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30959a640 10 monclient(hunting): picked mon.b con 0x7fe2f4003a90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:25:16.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30959a640 1 --2- 172.21.15.37:0/88727310 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2f4004810 0x7fe2f4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:16.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30959a640 10 monclient(hunting): picked mon.a con 0x7fe2f4004810 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:25:16.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30959a640 1 --2- 172.21.15.37:0/88727310 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2f40053d0 0x7fe2f400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:16.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30959a640 10 monclient(hunting): picked mon.c con 0x7fe2f40053d0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:16.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30959a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:16.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30959a640 1 -- 172.21.15.37:0/88727310 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe30c0a33c0 con 0x7fe2f4004810 2024-04-26T00:25:16.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30959a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:16.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30959a640 1 -- 172.21.15.37:0/88727310 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f400c270 con 0x7fe2f4003a90 2024-04-26T00:25:16.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30959a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:16.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30959a640 1 -- 172.21.15.37:0/88727310 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe2f400c470 con 0x7fe2f40053d0 2024-04-26T00:25:16.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30959a640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:16.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30959a640 10 monclient(hunting): _finish_auth 0 2024-04-26T00:25:16.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe310d0d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-26T00:25:16.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe309d9b640 1 --2- 172.21.15.37:0/88727310 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2f4004810 0x7fe2f4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:16.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe309d9b640 10 monclient(hunting): get_auth_request con 0x7fe2f4004810 auth_method 0 2024-04-26T00:25:16.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe309d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:16.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe309d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:16.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe309d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:25:16.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe309d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:16.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe309d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:16.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30ad9d640 1 --2- 172.21.15.37:0/88727310 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2f40053d0 0x7fe2f400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:16.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe309d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:25:16.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30ad9d640 10 monclient(hunting): get_auth_request con 0x7fe2f40053d0 auth_method 0 2024-04-26T00:25:16.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:16.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:16.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:25:16.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30a59c640 1 --2- 172.21.15.37:0/88727310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2f4003a90 0x7fe2f4003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:16.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30a59c640 10 monclient(hunting): get_auth_request con 0x7fe2f4003a90 auth_method 0 2024-04-26T00:25:16.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:16.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:16.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:25:16.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:16.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:16.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:25:16.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe309d9b640 10 monclient(hunting): handle_auth_done global_id 54216 payload 293 2024-04-26T00:25:16.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe309d9b640 1 -- 172.21.15.37:0/88727310 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2f40053d0 msgr2=0x7fe2f400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:16.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe309d9b640 1 --2- 172.21.15.37:0/88727310 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2f40053d0 0x7fe2f400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:16.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe309d9b640 1 -- 172.21.15.37:0/88727310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2f4003a90 msgr2=0x7fe2f4003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:16.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe309d9b640 1 --2- 172.21.15.37:0/88727310 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2f4003a90 0x7fe2f4003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:16.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe309d9b640 10 monclient: _finish_hunting 0 2024-04-26T00:25:16.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe309d9b640 1 monclient: found mon.a 2024-04-26T00:25:16.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe309d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:16.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe309d9b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:16.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe309d9b640 1 -- 172.21.15.37:0/88727310 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2f4003530 con 0x7fe2f4004810 2024-04-26T00:25:16.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe309d9b640 10 monclient: _finish_auth 0 2024-04-26T00:25:16.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe309d9b640 10 monclient: _check_auth_tickets 2024-04-26T00:25:16.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe309d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:16.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.245+0000 7fe30ad9d640 1 -- 172.21.15.37:0/88727310 reap_dead start 2024-04-26T00:25:16.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.249+0000 7fe309d9b640 1 -- 172.21.15.37:0/88727310 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2f4004810 msgr2=0x7fe2f4004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:16.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.249+0000 7fe309d9b640 1 --2- 172.21.15.37:0/88727310 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2f4004810 0x7fe2f4004c90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe3000014a0 tx=0x7fe30001ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=124f99b006a7d14f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:16.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.249+0000 7fe30959a640 1 -- 172.21.15.37:0/88727310 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30000c0e0 con 0x7fe2f4004810 2024-04-26T00:25:16.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.249+0000 7fe30959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:16.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.249+0000 7fe30959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:16.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.249+0000 7fe30959a640 10 monclient: dump: 2024-04-26T00:25:16.242 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:16.242 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:16.242 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.242 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.242 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:16.243 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:16.243 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:16.243 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:16.243 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:16.243 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:16.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.249+0000 7fe30959a640 1 -- 172.21.15.37:0/88727310 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3000a9020 con 0x7fe2f4004810 2024-04-26T00:25:16.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.249+0000 7fe310d0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:25:16.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.249+0000 7fe30959a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:16.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.249+0000 7fe310d0d640 10 monclient: get_monmap_and_config success 2024-04-26T00:25:16.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.249+0000 7fe30959a640 1 -- 172.21.15.37:0/88727310 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe300005b00 con 0x7fe2f4004810 2024-04-26T00:25:16.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.249+0000 7fe310d0d640 4 set_mon_vals no callback set 2024-04-26T00:25:16.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.249+0000 7fe30959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:16.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.249+0000 7fe30959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:16.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.249+0000 7fe30959a640 10 monclient: dump: 2024-04-26T00:25:16.244 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:16.244 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:16.244 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.244 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.244 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:16.244 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:16.244 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:16.244 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:16.244 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:16.244 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:16.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.249+0000 7fe310d0d640 10 monclient: shutdown 2024-04-26T00:25:16.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.249+0000 7fe310d0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:16.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.249+0000 7fe310d0d640 1 -- 172.21.15.37:0/88727310 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2f4004810 msgr2=0x7fe2f4004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:16.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.249+0000 7fe310d0d640 1 --2- 172.21.15.37:0/88727310 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2f4004810 0x7fe2f4004c90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe3000014a0 tx=0x7fe30001ccd0 comp rx=0 tx=0).stop 2024-04-26T00:25:16.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.249+0000 7fe310d0d640 1 -- 172.21.15.37:0/88727310 shutdown_connections 2024-04-26T00:25:16.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe310d0d640 1 --2- 172.21.15.37:0/88727310 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2f4004810 0x7fe2f4004c90 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:16.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe310d0d640 1 -- 172.21.15.37:0/88727310 >> 172.21.15.37:0/88727310 conn(0x7fe30c0a24c0 msgr2=0x7fe30c0a28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:16.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe310d0d640 1 -- 172.21.15.37:0/88727310 shutdown_connections 2024-04-26T00:25:16.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe310d0d640 1 -- 172.21.15.37:0/88727310 wait complete. 2024-04-26T00:25:16.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe310d0d640 10 monclient: build_initial_monmap 2024-04-26T00:25:16.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe310d0d640 10 monclient: monmap: 2024-04-26T00:25:16.245 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:16.245 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:25:16.245 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:25:16.245 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:25:16.245 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:16.246 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:25:16.246 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:25:16.246 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:25:16.246 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:25:16.246 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:16.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe310d0d640 1 Processor -- start 2024-04-26T00:25:16.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe310d0d640 1 -- start start 2024-04-26T00:25:16.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe310d0d640 10 monclient: init 2024-04-26T00:25:16.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe310d0d640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:16.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe310d0d640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:25:16.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe310d0d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2f4004810 0x7fe30c11c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:16.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe310d0d640 10 monclient(hunting): picked mon.noname-b con 0x7fe2f4004810 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:25:16.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe310d0d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe30c12fe40 0x7fe30c11c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:16.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe310d0d640 10 monclient(hunting): picked mon.noname-a con 0x7fe30c12fe40 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:25:16.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe310d0d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe30c11ca90 0x7fe30c1cf220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:16.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe310d0d640 10 monclient(hunting): picked mon.noname-c con 0x7fe30c11ca90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:16.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe310d0d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:16.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe310d0d640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe30c0a1760 con 0x7fe30c12fe40 2024-04-26T00:25:16.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe310d0d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:16.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe310d0d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe30c0a15e0 con 0x7fe2f4004810 2024-04-26T00:25:16.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe310d0d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:16.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe310d0d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe30c0a18e0 con 0x7fe30c11ca90 2024-04-26T00:25:16.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe310d0d640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:16.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe310d0d640 10 monclient(hunting): authenticate will time out at 1297.561655s 2024-04-26T00:25:16.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe30ad9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe30c11ca90 0x7fe30c1cf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:16.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe309d9b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe30c12fe40 0x7fe30c11c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:16.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe309d9b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe30c12fe40 0x7fe30c11c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36516/0 (socket says 172.21.15.37:36516) 2024-04-26T00:25:16.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe30ad9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe30c11ca90 0x7fe30c1cf220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:59830/0 (socket says 172.21.15.37:59830) 2024-04-26T00:25:16.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe309d9b640 1 -- 172.21.15.37:0/3511379995 learned_addr learned my addr 172.21.15.37:0/3511379995 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:16.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe309d9b640 10 monclient(hunting): get_auth_request con 0x7fe30c12fe40 auth_method 0 2024-04-26T00:25:16.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe309d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:16.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe309d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:16.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe309d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:16.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe30ad9d640 10 monclient(hunting): get_auth_request con 0x7fe30c11ca90 auth_method 0 2024-04-26T00:25:16.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe30ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:16.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe30ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:16.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe30ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:16.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe309d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:16.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe309d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:16.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe309d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:16.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe30ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:16.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe30ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:16.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe30ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:16.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe30a59c640 1 --2- 172.21.15.37:0/3511379995 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2f4004810 0x7fe30c11c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:16.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe30a59c640 10 monclient(hunting): get_auth_request con 0x7fe2f4004810 auth_method 0 2024-04-26T00:25:16.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe30a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:16.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe30a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:16.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe30a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:16.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe309d9b640 10 monclient(hunting): handle_auth_done global_id 54219 payload 824 2024-04-26T00:25:16.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe309d9b640 1 -- 172.21.15.37:0/3511379995 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe30c11ca90 msgr2=0x7fe30c1cf220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:16.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe309d9b640 1 --2- 172.21.15.37:0/3511379995 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe30c11ca90 0x7fe30c1cf220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:16.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe309d9b640 1 -- 172.21.15.37:0/3511379995 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2f4004810 msgr2=0x7fe30c11c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:16.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe309d9b640 1 --2- 172.21.15.37:0/3511379995 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2f4004810 0x7fe30c11c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:16.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe309d9b640 10 monclient: _finish_hunting 0 2024-04-26T00:25:16.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe309d9b640 1 monclient: found mon.noname-a 2024-04-26T00:25:16.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe309d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:16.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe309d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:25:16.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe309d9b640 1 -- 172.21.15.37:0/3511379995 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe30c11d0b0 con 0x7fe30c12fe40 2024-04-26T00:25:16.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.253+0000 7fe30ad9d640 1 --2- 172.21.15.37:0/3511379995 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe30c11ca90 0x7fe30c1cf220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:25:16.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe309d9b640 1 -- 172.21.15.37:0/3511379995 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe30c12fe40 msgr2=0x7fe30c11c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:16.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe309d9b640 1 --2- 172.21.15.37:0/3511379995 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe30c12fe40 0x7fe30c11c550 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe30001b3d0 tx=0x7fe2f4005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:16.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe2faffd640 1 -- 172.21.15.37:0/3511379995 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3000a5e30 con 0x7fe30c12fe40 2024-04-26T00:25:16.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe2faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:16.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe2faffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:25:16.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe2faffd640 10 monclient: dump: 2024-04-26T00:25:16.250 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:16.250 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:16.250 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.250 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.250 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:16.250 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:16.250 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:16.250 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:16.251 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:16.251 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:16.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe2faffd640 10 monclient: _finish_auth 0 2024-04-26T00:25:16.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe2faffd640 10 monclient: _check_auth_tickets 2024-04-26T00:25:16.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe2faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:16.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe2faffd640 1 -- 172.21.15.37:0/3511379995 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3000047f0 con 0x7fe30c12fe40 2024-04-26T00:25:16.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe2faffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:16.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe2faffd640 1 -- 172.21.15.37:0/3511379995 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe300004ba0 con 0x7fe30c12fe40 2024-04-26T00:25:16.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe310d0d640 5 monclient: authenticate success, global_id 54219 2024-04-26T00:25:16.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe310d0d640 10 monclient: _renew_subs 2024-04-26T00:25:16.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe310d0d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:16.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe310d0d640 1 -- 172.21.15.37:0/3511379995 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe30c0a3be0 con 0x7fe30c12fe40 2024-04-26T00:25:16.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe2fbfff640 4 set_mon_vals no callback set 2024-04-26T00:25:16.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe2faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:16.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe2faffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:16.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe2faffd640 10 monclient: dump: 2024-04-26T00:25:16.252 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:16.252 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:16.252 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.252 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.252 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:16.252 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:16.252 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:16.252 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:16.252 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:16.252 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:16.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe310d0d640 10 monclient: _renew_subs 2024-04-26T00:25:16.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe310d0d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:16.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe310d0d640 1 -- 172.21.15.37:0/3511379995 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe30c11ea90 con 0x7fe30c12fe40 2024-04-26T00:25:16.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe2faffd640 1 -- 172.21.15.37:0/3511379995 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe3000bd060 con 0x7fe30c12fe40 2024-04-26T00:25:16.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe310d0d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:16.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe310d0d640 1 --2- 172.21.15.37:0/3511379995 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe30c1cf760 0x7fe30c1cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:16.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe310d0d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:16.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe310d0d640 1 -- 172.21.15.37:0/3511379995 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe30c11cea0 con 0x7fe30c1cf760 2024-04-26T00:25:16.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe30a59c640 1 --2- 172.21.15.37:0/3511379995 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe30c1cf760 0x7fe30c1cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:16.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe30a59c640 10 monclient: get_auth_request con 0x7fe30c1cf760 auth_method 0 2024-04-26T00:25:16.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe30a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:16.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe30a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:16.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe30a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:16.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe2faffd640 1 --2- 172.21.15.37:0/3511379995 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe2ec035ed0 0x7fe2ec038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:16.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe30a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:16.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe30a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:16.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe30a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:16.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe2faffd640 1 -- 172.21.15.37:0/3511379995 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(208..208 src has 172..208) ==== 3963+0+0 (secure 0 0 0) 0x7fe3000e1ac0 con 0x7fe30c12fe40 2024-04-26T00:25:16.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe30a59c640 10 monclient(hunting): handle_auth_done global_id 64145 payload 824 2024-04-26T00:25:16.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe30a59c640 1 -- 172.21.15.37:0/3511379995 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe30c09fd50 con 0x7fe30c1cf760 2024-04-26T00:25:16.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe30ad9d640 1 --2- 172.21.15.37:0/3511379995 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe2ec035ed0 0x7fe2ec038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:16.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.257+0000 7fe30ad9d640 10 monclient: get_auth_request con 0x7fe2ec035ed0 auth_method 0 2024-04-26T00:25:16.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.261+0000 7fe30a59c640 1 -- 172.21.15.37:0/3511379995 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe30c1cf760 msgr2=0x7fe30c1cfb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:16.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.261+0000 7fe30a59c640 1 --2- 172.21.15.37:0/3511379995 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe30c1cf760 0x7fe30c1cfb40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe30c0a23e0 tx=0x7fe2fc013ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:16.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.261+0000 7fe2faffd640 1 -- 172.21.15.37:0/3511379995 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc01c070 con 0x7fe30c1cf760 2024-04-26T00:25:16.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.261+0000 7fe2faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:16.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.261+0000 7fe2faffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:25:16.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.261+0000 7fe2faffd640 10 monclient: dump: 2024-04-26T00:25:16.254 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:16.254 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:16.254 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.254 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.254 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:16.255 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:16.255 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:16.255 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:16.255 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:16.255 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:16.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.261+0000 7fe2faffd640 1 -- 172.21.15.37:0/3511379995 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe2fc015c90 con 0x7fe30c1cf760 2024-04-26T00:25:16.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.261+0000 7fe2faffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:16.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.261+0000 7fe2faffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:25:16.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.261+0000 7fe2faffd640 1 -- 172.21.15.37:0/3511379995 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe30c1cf760 msgr2=0x7fe30c1cfb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:16.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.261+0000 7fe2faffd640 1 --2- 172.21.15.37:0/3511379995 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe30c1cf760 0x7fe30c1cfb40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe30c0a23e0 tx=0x7fe2fc013ae0 comp rx=0 tx=0).stop 2024-04-26T00:25:16.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.261+0000 7fe2faffd640 1 -- 172.21.15.37:0/3511379995 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe30c1cf760 msgr2=0x7fe30c1cfb40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:25:16.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.261+0000 7fe2faffd640 1 --2- 172.21.15.37:0/3511379995 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe30c1cf760 0x7fe30c1cfb40 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe30c0a23e0 tx=0x7fe2fc013ae0 comp rx=0 tx=0).stop 2024-04-26T00:25:16.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.261+0000 7fe30ad9d640 1 --2- 172.21.15.37:0/3511379995 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe2ec035ed0 0x7fe2ec038390 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe30c0a3580 tx=0x7fe304009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:16.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.265+0000 7fe310d0d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:25:16.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.265+0000 7fe310d0d640 1 --2- 172.21.15.37:0/3511379995 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe30c134df0 0x7fe30c135200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:16.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.265+0000 7fe310d0d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:16.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.265+0000 7fe310d0d640 1 -- 172.21.15.37:0/3511379995 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe30c1d0080 con 0x7fe30c134df0 2024-04-26T00:25:16.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.265+0000 7fe30a59c640 1 --2- 172.21.15.37:0/3511379995 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe30c134df0 0x7fe30c135200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:16.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.265+0000 7fe30a59c640 10 monclient: get_auth_request con 0x7fe30c134df0 auth_method 0 2024-04-26T00:25:16.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.265+0000 7fe30a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:16.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.265+0000 7fe30a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:16.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.265+0000 7fe30a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:16.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.265+0000 7fe30a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:16.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.265+0000 7fe30a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:16.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.265+0000 7fe30a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:16.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.265+0000 7fe30a59c640 10 monclient(hunting): handle_auth_done global_id 64148 payload 824 2024-04-26T00:25:16.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.265+0000 7fe30a59c640 1 -- 172.21.15.37:0/3511379995 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe30c11f150 con 0x7fe30c134df0 2024-04-26T00:25:16.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.269+0000 7fe30a59c640 1 -- 172.21.15.37:0/3511379995 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe30c134df0 msgr2=0x7fe30c135200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:16.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.269+0000 7fe30a59c640 1 --2- 172.21.15.37:0/3511379995 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe30c134df0 0x7fe30c135200 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe30c0a23e0 tx=0x7fe2fc004740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:16.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.269+0000 7fe2faffd640 1 -- 172.21.15.37:0/3511379995 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc0065b0 con 0x7fe30c134df0 2024-04-26T00:25:16.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.269+0000 7fe2faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:16.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.269+0000 7fe2faffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:25:16.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.269+0000 7fe2faffd640 10 monclient: dump: 2024-04-26T00:25:16.259 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:16.259 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:16.259 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.259 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.259 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:16.259 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:16.259 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:16.259 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:16.259 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:16.259 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:16.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.269+0000 7fe2faffd640 1 -- 172.21.15.37:0/3511379995 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4716 (secure 0 0 0) 0x7fe2fc0065b0 con 0x7fe30c134df0 2024-04-26T00:25:16.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.269+0000 7fe2faffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:25:16.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.269+0000 7fe2faffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:25:16.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.269+0000 7fe2faffd640 1 -- 172.21.15.37:0/3511379995 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe30c134df0 msgr2=0x7fe30c135200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:16.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.269+0000 7fe2faffd640 1 --2- 172.21.15.37:0/3511379995 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe30c134df0 0x7fe30c135200 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe30c0a23e0 tx=0x7fe2fc004740 comp rx=0 tx=0).stop 2024-04-26T00:25:16.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.269+0000 7fe2faffd640 1 -- 172.21.15.37:0/3511379995 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe30c134df0 msgr2=0x7fe30c135200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:25:16.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.269+0000 7fe2faffd640 1 --2- 172.21.15.37:0/3511379995 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe30c134df0 0x7fe30c135200 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe30c0a23e0 tx=0x7fe2fc004740 comp rx=0 tx=0).stop 2024-04-26T00:25:16.261 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:25:16.261 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:25:16.261 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:25:16.261 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:25:16.262 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 40, 2024-04-26T00:25:16.262 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 23739, 2024-04-26T00:25:16.262 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:25:16.262 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:25:16.262 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:25:16.262 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:25:16.262 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:16.262 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 3, 2024-04-26T00:25:16.262 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:25:16.262 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:25:16.262 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:25:16.262 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:16.262 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:16.263 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:16.263 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:16.263 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:16.263 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:16.263 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:16.263 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:16.263 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:16.263 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:16.263 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:16.263 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:16.263 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:25:16.263 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:25:16.263 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:16.263 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:16.263 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:16.264 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:16.264 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:16.264 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:16.264 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:16.264 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:16.264 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:16.264 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:16.264 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:16.264 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:16.264 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:16.264 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:25:16.264 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:25:16.264 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:25:16.264 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:25:16.264 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:25:16.265 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:25:16.265 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:25:16.265 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:25:16.265 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:25:16.265 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:25:16.265 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:25:16.265 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:25:16.265 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:25:16.265 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:25:16.265 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:25:16.265 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:25:16.265 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:25:16.265 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:16.265 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:16.265 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:16.266 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:16.266 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:16.266 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:16.266 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:16.266 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:16.266 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:16.266 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:16.266 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:16.266 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:16.266 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:25:16.266 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:16.266 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:25:16.266 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:16.266 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:25:16.267 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:25:16.267 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:16.267 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:16.267 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:16.267 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:16.267 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:25:16.267 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:16.267 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:16.267 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:16.267 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:16.267 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:25:16.267 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:16.267 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:16.267 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:16.268 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:16.268 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:25:16.268 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:25:16.268 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:16.268 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:16.268 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:16.268 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:16.268 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:16.268 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:25:16.268 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:25:16.268 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:16.268 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:16.268 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:16.268 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:16.269 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:25:16.269 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:16.269 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:16.269 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:16.269 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:16.269 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:25:16.269 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:16.269 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:16.269 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:16.269 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:16.269 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:25:16.269 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:25:16.269 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:16.269 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:16.269 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:16.270 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:16.270 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:16.270 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:25:16.270 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:25:16.270 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:16.270 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:16.270 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:16.270 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:16.270 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:25:16.270 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:16.270 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:16.270 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:16.270 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:16.270 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:25:16.271 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:16.271 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:16.271 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:16.271 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:16.271 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:25:16.271 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:25:16.271 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:16.271 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:16.271 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:16.271 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:16.271 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:16.271 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:16.271 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:25:16.271 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:25:16.272 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:16.272 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:16.272 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:16.272 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:25:16.272 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:16.272 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:16.272 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:25:16.272 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:16.272 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:16.272 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:16.272 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:25:16.272 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:16.272 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:16.272 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:25:16.272 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:16.273 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:16.273 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:16.273 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:25:16.273 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:16.273 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:16.273 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:16.273 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:25:16.273 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:25:16.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.269+0000 7fe310d0d640 1 -- 172.21.15.37:0/3511379995 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe2ec035ed0 msgr2=0x7fe2ec038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:16.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.269+0000 7fe310d0d640 1 --2- 172.21.15.37:0/3511379995 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe2ec035ed0 0x7fe2ec038390 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe30c0a3580 tx=0x7fe304009040 comp rx=0 tx=0).stop 2024-04-26T00:25:16.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.269+0000 7fe310d0d640 10 monclient: shutdown 2024-04-26T00:25:16.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.269+0000 7fe310d0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:16.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.269+0000 7fe310d0d640 1 -- 172.21.15.37:0/3511379995 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe30c12fe40 msgr2=0x7fe30c11c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:16.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.269+0000 7fe310d0d640 1 --2- 172.21.15.37:0/3511379995 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe30c12fe40 0x7fe30c11c550 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe30001b3d0 tx=0x7fe2f4005840 comp rx=0 tx=0).stop 2024-04-26T00:25:16.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.269+0000 7fe30ad9d640 1 -- 172.21.15.37:0/3511379995 reap_dead start 2024-04-26T00:25:16.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.269+0000 7fe310d0d640 1 -- 172.21.15.37:0/3511379995 shutdown_connections 2024-04-26T00:25:16.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.269+0000 7fe310d0d640 1 --2- 172.21.15.37:0/3511379995 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe30c12fe40 0x7fe30c11c550 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:16.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.269+0000 7fe310d0d640 1 -- 172.21.15.37:0/3511379995 >> 172.21.15.37:0/3511379995 conn(0x7fe30c0a24c0 msgr2=0x7fe30c130660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:16.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.269+0000 7fe310d0d640 1 -- 172.21.15.37:0/3511379995 shutdown_connections 2024-04-26T00:25:16.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.269+0000 7fe310d0d640 1 -- 172.21.15.37:0/3511379995 wait complete. 2024-04-26T00:25:16.275 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:25:16.275 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:25:16.276 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:25:16.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f2474bae640 10 monclient: get_monmap_and_config 2024-04-26T00:25:16.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f2474bae640 10 monclient: build_initial_monmap 2024-04-26T00:25:16.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f2474bae640 10 monclient: monmap: 2024-04-26T00:25:16.393 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:16.393 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:16.393 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:25:16.416260+0000 2024-04-26T00:25:16.393 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:25:16.416260+0000 2024-04-26T00:25:16.394 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:16.394 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:16.394 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:25:16.394 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:25:16.394 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:25:16.394 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:25:16.394 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:25:16.394 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:16.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f2474bae640 1 Processor -- start 2024-04-26T00:25:16.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f2474bae640 1 -- start start 2024-04-26T00:25:16.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f2474bae640 10 monclient: init 2024-04-26T00:25:16.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f2474bae640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:16.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f2474bae640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-04-26T00:25:16.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f2474bae640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2470136f20 0x7f247009bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:16.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f2474bae640 10 monclient(hunting): picked mon.noname-c con 0x7f2470136f20 addr v2:172.21.15.45:3300/0 2024-04-26T00:25:16.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f2474bae640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2470137300 addr v1:172.21.15.45:6789/0 2024-04-26T00:25:16.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f2474bae640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f247013c450 0x7f247013e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:16.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f2474bae640 10 monclient(hunting): picked mon.noname-a con 0x7f247013c450 addr v2:172.21.15.37:3300/0 2024-04-26T00:25:16.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f2474bae640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24700a50f0 con 0x7f2470137300 2024-04-26T00:25:16.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f2474bae640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:16.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f2474bae640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f247009b330 con 0x7f247013c450 2024-04-26T00:25:16.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f2474bae640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:16.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f2474bae640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f247009b610 con 0x7f2470136f20 2024-04-26T00:25:16.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f2474bae640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:16.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f2474bae640 10 monclient(hunting): authenticate will time out at 1297.722055s 2024-04-26T00:25:16.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f246ed9d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f247013c450 0x7f247013e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:16.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f246ed9d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f247013c450 0x7f247013e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36524/0 (socket says 172.21.15.37:36524) 2024-04-26T00:25:16.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f246ed9d640 1 -- 172.21.15.37:0/1404579708 learned_addr learned my addr 172.21.15.37:0/1404579708 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:16.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f246ed9d640 10 monclient(hunting): get_auth_request con 0x7f247013c450 auth_method 0 2024-04-26T00:25:16.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f246ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:16.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f246ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:16.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f246ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:16.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f246ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:16.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f246ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:16.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f246e59c640 1 --2- 172.21.15.37:0/1404579708 >> v2:172.21.15.45:3300/0 conn(0x7f2470136f20 0x7f247009bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:16.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f246ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:16.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f246e59c640 10 monclient(hunting): get_auth_request con 0x7f2470136f20 auth_method 0 2024-04-26T00:25:16.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f246e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:16.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f246e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:16.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f246e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:16.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f246ed9d640 10 monclient(hunting): handle_auth_done global_id 54222 payload 274 2024-04-26T00:25:16.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f246ed9d640 1 -- 172.21.15.37:0/1404579708 >> v2:172.21.15.45:3300/0 conn(0x7f2470136f20 msgr2=0x7f247009bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:16.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f246ed9d640 1 --2- 172.21.15.37:0/1404579708 >> v2:172.21.15.45:3300/0 conn(0x7f2470136f20 0x7f247009bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:16.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f246ed9d640 1 -- 172.21.15.37:0/1404579708 >> v1:172.21.15.45:6789/0 conn(0x7f2470137300 legacy=0x7f24701377b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:16.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f246ed9d640 10 monclient: _finish_hunting 0 2024-04-26T00:25:16.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f246ed9d640 1 monclient: found mon.noname-a 2024-04-26T00:25:16.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f246ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:16.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f246ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:25:16.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.413+0000 7f246ed9d640 1 -- 172.21.15.37:0/1404579708 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f247009b8f0 con 0x7f247013c450 2024-04-26T00:25:16.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.417+0000 7f246ed9d640 1 -- 172.21.15.37:0/1404579708 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f247013c450 msgr2=0x7f247013e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:16.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.417+0000 7f246ed9d640 1 --2- 172.21.15.37:0/1404579708 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f247013c450 0x7f247013e840 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2470068940 tx=0x7f246801c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a773125449ad28d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:16.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.417+0000 7f246d59a640 1 -- 172.21.15.37:0/1404579708 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24680a57e0 con 0x7f247013c450 2024-04-26T00:25:16.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.417+0000 7f246d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:16.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.417+0000 7f246d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:25:16.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.417+0000 7f246d59a640 10 monclient: dump: 2024-04-26T00:25:16.401 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:16.402 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:16.402 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.402 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.402 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:16.402 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:16.402 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:16.402 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:16.402 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:16.402 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:16.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.417+0000 7f246d59a640 10 monclient: _finish_auth 0 2024-04-26T00:25:16.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.417+0000 7f246d59a640 10 monclient: _check_auth_tickets 2024-04-26T00:25:16.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.417+0000 7f246d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:16.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.417+0000 7f246d59a640 1 -- 172.21.15.37:0/1404579708 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24680a5980 con 0x7f247013c450 2024-04-26T00:25:16.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.417+0000 7f246d59a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:16.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.417+0000 7f246d59a640 1 -- 172.21.15.37:0/1404579708 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24680a5d50 con 0x7f247013c450 2024-04-26T00:25:16.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.417+0000 7f2474bae640 5 monclient: authenticate success, global_id 54222 2024-04-26T00:25:16.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.417+0000 7f2474bae640 10 monclient: get_monmap_and_config success 2024-04-26T00:25:16.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.417+0000 7f2474bae640 4 set_mon_vals no callback set 2024-04-26T00:25:16.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.417+0000 7f246d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:16.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.417+0000 7f246d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:16.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.417+0000 7f246d59a640 10 monclient: dump: 2024-04-26T00:25:16.403 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:16.403 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:16.403 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.403 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.403 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:16.403 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:16.404 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:16.404 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:16.404 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:16.404 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:16.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.417+0000 7f2474bae640 10 monclient: shutdown 2024-04-26T00:25:16.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.417+0000 7f2474bae640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:16.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.417+0000 7f2474bae640 1 -- 172.21.15.37:0/1404579708 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f247013c450 msgr2=0x7f247013e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:16.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.417+0000 7f2474bae640 1 --2- 172.21.15.37:0/1404579708 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f247013c450 0x7f247013e840 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2470068940 tx=0x7f246801c900 comp rx=0 tx=0).stop 2024-04-26T00:25:16.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.417+0000 7f2474bae640 1 -- 172.21.15.37:0/1404579708 shutdown_connections 2024-04-26T00:25:16.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.417+0000 7f2474bae640 1 --2- 172.21.15.37:0/1404579708 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f247013c450 0x7f247013e840 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:16.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.417+0000 7f2474bae640 1 --2- 172.21.15.37:0/1404579708 >> v2:172.21.15.45:3300/0 conn(0x7f2470136f20 0x7f247009bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:16.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.417+0000 7f2474bae640 1 -- 172.21.15.37:0/1404579708 >> 172.21.15.37:0/1404579708 conn(0x7f24700a61b0 msgr2=0x7f24700a65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:16.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.417+0000 7f2474bae640 1 -- 172.21.15.37:0/1404579708 shutdown_connections 2024-04-26T00:25:16.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.417+0000 7f2474bae640 1 -- 172.21.15.37:0/1404579708 wait complete. 2024-04-26T00:25:16.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f2474bae640 10 monclient: build_initial_monmap 2024-04-26T00:25:16.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f2474bae640 10 monclient: monmap: 2024-04-26T00:25:16.405 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:16.405 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:25:16.405 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:25:16.405 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:25:16.405 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:16.405 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:25:16.405 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:25:16.405 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:25:16.405 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:25:16.405 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:16.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f2474bae640 1 Processor -- start 2024-04-26T00:25:16.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f2474bae640 1 -- start start 2024-04-26T00:25:16.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f2474bae640 10 monclient: init 2024-04-26T00:25:16.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f2474bae640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:16.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f2474bae640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:25:16.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f2474bae640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2470136f20 0x7f24701ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:16.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f2474bae640 10 monclient(hunting): picked mon.noname-a con 0x7f2470136f20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:25:16.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f2474bae640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2470137300 0x7f24701350f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:16.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f2474bae640 10 monclient(hunting): picked mon.noname-b con 0x7f2470137300 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:25:16.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f2474bae640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f247013c450 0x7f2470138180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:16.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f2474bae640 10 monclient(hunting): picked mon.noname-c con 0x7f247013c450 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:16.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f2474bae640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:16.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f2474bae640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f2470141930 con 0x7f2470136f20 2024-04-26T00:25:16.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f2474bae640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:16.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f2474bae640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f24701417b0 con 0x7f2470137300 2024-04-26T00:25:16.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f2474bae640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:16.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f2474bae640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2470141ab0 con 0x7f247013c450 2024-04-26T00:25:16.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f2474bae640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:16.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f2474bae640 10 monclient(hunting): authenticate will time out at 1297.729467s 2024-04-26T00:25:16.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f246e59c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2470136f20 0x7f24701ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:16.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f246e59c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2470136f20 0x7f24701ceec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36532/0 (socket says 172.21.15.37:36532) 2024-04-26T00:25:16.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f246e59c640 1 -- 172.21.15.37:0/2205424178 learned_addr learned my addr 172.21.15.37:0/2205424178 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:16.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f246e59c640 10 monclient(hunting): get_auth_request con 0x7f2470136f20 auth_method 0 2024-04-26T00:25:16.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f246e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:16.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f246e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:16.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f246e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:16.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f246ed9d640 1 --2- 172.21.15.37:0/2205424178 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f247013c450 0x7f2470138180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:16.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f246ed9d640 10 monclient(hunting): get_auth_request con 0x7f247013c450 auth_method 0 2024-04-26T00:25:16.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f246ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:16.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f246ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:16.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f246ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:16.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f246ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:16.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f246ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:16.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f246ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:16.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f246e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:16.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f246e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:16.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f246e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:16.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f246dd9b640 1 --2- 172.21.15.37:0/2205424178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2470137300 0x7f24701350f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:16.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f246dd9b640 10 monclient(hunting): get_auth_request con 0x7f2470137300 auth_method 0 2024-04-26T00:25:16.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f246dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:16.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f246dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:16.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f246dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:16.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f246ed9d640 10 monclient(hunting): handle_auth_done global_id 64151 payload 824 2024-04-26T00:25:16.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f246ed9d640 1 -- 172.21.15.37:0/2205424178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2470137300 msgr2=0x7f24701350f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:16.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f246ed9d640 1 --2- 172.21.15.37:0/2205424178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2470137300 0x7f24701350f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:16.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f246ed9d640 1 -- 172.21.15.37:0/2205424178 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2470136f20 msgr2=0x7f24701ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:16.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f246ed9d640 1 --2- 172.21.15.37:0/2205424178 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2470136f20 0x7f24701ceec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:16.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f246ed9d640 10 monclient: _finish_hunting 0 2024-04-26T00:25:16.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f246ed9d640 1 monclient: found mon.noname-c 2024-04-26T00:25:16.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f246ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:16.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f246ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:25:16.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.421+0000 7f246ed9d640 1 -- 172.21.15.37:0/2205424178 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f247009ae30 con 0x7f247013c450 2024-04-26T00:25:16.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f246ed9d640 1 -- 172.21.15.37:0/2205424178 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f247013c450 msgr2=0x7f2470138180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:16.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f246ed9d640 1 --2- 172.21.15.37:0/2205424178 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f247013c450 0x7f2470138180 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f24680ae000 tx=0x7f24680196a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:16.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f245effd640 1 -- 172.21.15.37:0/2205424178 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24680bc020 con 0x7f247013c450 2024-04-26T00:25:16.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f245effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:16.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f245effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:25:16.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f245effd640 10 monclient: dump: 2024-04-26T00:25:16.415 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:16.415 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:16.416 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.416 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.416 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:16.416 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:16.416 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:16.416 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:16.416 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:16.416 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:16.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f245effd640 10 monclient: _finish_auth 0 2024-04-26T00:25:16.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f245effd640 10 monclient: _check_auth_tickets 2024-04-26T00:25:16.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f245effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:16.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f2474bae640 5 monclient: authenticate success, global_id 64151 2024-04-26T00:25:16.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f245effd640 1 -- 172.21.15.37:0/2205424178 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24680b8880 con 0x7f247013c450 2024-04-26T00:25:16.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f245effd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:16.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f245effd640 1 -- 172.21.15.37:0/2205424178 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24680b8c30 con 0x7f247013c450 2024-04-26T00:25:16.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f2474bae640 10 monclient: _renew_subs 2024-04-26T00:25:16.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f2474bae640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:25:16.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f2474bae640 1 -- 172.21.15.37:0/2205424178 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2470135f10 con 0x7f247013c450 2024-04-26T00:25:16.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f245effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:16.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f246cd99640 4 set_mon_vals no callback set 2024-04-26T00:25:16.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f245effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:25:16.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f245effd640 10 monclient: dump: 2024-04-26T00:25:16.417 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:16.417 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:16.417 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.417 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.418 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:16.418 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:16.418 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:16.418 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:16.418 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:16.418 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:16.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f2474bae640 10 monclient: _renew_subs 2024-04-26T00:25:16.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f2474bae640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:25:16.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f2474bae640 1 -- 172.21.15.37:0/2205424178 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2470135630 con 0x7f247013c450 2024-04-26T00:25:16.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f2474bae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:16.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f2474bae640 1 --2- 172.21.15.37:0/2205424178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2470138e80 0x7f247002c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:16.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f2474bae640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:16.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f2474bae640 1 -- 172.21.15.37:0/2205424178 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f247013b820 con 0x7f2470138e80 2024-04-26T00:25:16.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f245effd640 1 -- 172.21.15.37:0/2205424178 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f24680bf020 con 0x7f247013c450 2024-04-26T00:25:16.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f245effd640 1 --2- 172.21.15.37:0/2205424178 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2450035e80 0x7f2450038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:16.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f246e59c640 1 --2- 172.21.15.37:0/2205424178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2470138e80 0x7f247002c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:16.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f246e59c640 10 monclient: get_auth_request con 0x7f2470138e80 auth_method 0 2024-04-26T00:25:16.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f246e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:16.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f246e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:16.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f246e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:16.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f245effd640 1 -- 172.21.15.37:0/2205424178 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(208..208 src has 172..208) ==== 3963+0+0 (secure 0 0 0) 0x7f24680e3b50 con 0x7f247013c450 2024-04-26T00:25:16.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f246e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:16.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f246e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:16.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f246e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:16.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f246dd9b640 1 --2- 172.21.15.37:0/2205424178 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2450035e80 0x7f2450038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:16.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f246dd9b640 10 monclient: get_auth_request con 0x7f2450035e80 auth_method 0 2024-04-26T00:25:16.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f246e59c640 10 monclient(hunting): handle_auth_done global_id 24466 payload 824 2024-04-26T00:25:16.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.425+0000 7f246e59c640 1 -- 172.21.15.37:0/2205424178 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f247013b9c0 con 0x7f2470138e80 2024-04-26T00:25:16.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.429+0000 7f246e59c640 1 -- 172.21.15.37:0/2205424178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2470138e80 msgr2=0x7f247002c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:16.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.429+0000 7f246e59c640 1 --2- 172.21.15.37:0/2205424178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2470138e80 0x7f247002c000 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2460010960 tx=0x7f2460010e20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:16.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.429+0000 7f246dd9b640 1 --2- 172.21.15.37:0/2205424178 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2450035e80 0x7f2450038340 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2470135e60 tx=0x7f2464009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:16.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.429+0000 7f245effd640 1 -- 172.21.15.37:0/2205424178 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2460018020 con 0x7f2470138e80 2024-04-26T00:25:16.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.429+0000 7f245effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:16.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.429+0000 7f245effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:25:16.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.429+0000 7f245effd640 10 monclient: dump: 2024-04-26T00:25:16.420 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:16.420 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:16.420 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.421 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.421 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:16.421 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:16.421 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:16.421 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:16.421 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:16.421 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:16.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.429+0000 7f245effd640 1 -- 172.21.15.37:0/2205424178 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2460004450 con 0x7f2470138e80 2024-04-26T00:25:16.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.429+0000 7f245effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:16.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.429+0000 7f245effd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:25:16.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.429+0000 7f245effd640 1 -- 172.21.15.37:0/2205424178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2470138e80 msgr2=0x7f247002c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:16.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.429+0000 7f245effd640 1 --2- 172.21.15.37:0/2205424178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2470138e80 0x7f247002c000 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2460010960 tx=0x7f2460010e20 comp rx=0 tx=0).stop 2024-04-26T00:25:16.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.429+0000 7f245effd640 1 -- 172.21.15.37:0/2205424178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2470138e80 msgr2=0x7f247002c000 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:25:16.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.429+0000 7f245effd640 1 --2- 172.21.15.37:0/2205424178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2470138e80 0x7f247002c000 secure :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2460010960 tx=0x7f2460010e20 comp rx=0 tx=0).stop 2024-04-26T00:25:16.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.437+0000 7f2474bae640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:25:16.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.437+0000 7f2474bae640 1 --2- 172.21.15.37:0/2205424178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2470139260 0x7f247002c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:16.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.437+0000 7f2474bae640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:16.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.437+0000 7f2474bae640 1 -- 172.21.15.37:0/2205424178 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f247013a520 con 0x7f2470139260 2024-04-26T00:25:16.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.437+0000 7f246e59c640 1 --2- 172.21.15.37:0/2205424178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2470139260 0x7f247002c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:16.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.437+0000 7f246e59c640 10 monclient: get_auth_request con 0x7f2470139260 auth_method 0 2024-04-26T00:25:16.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.437+0000 7f246e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:16.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.437+0000 7f246e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:16.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.437+0000 7f246e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:16.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.437+0000 7f246e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:16.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.437+0000 7f246e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:16.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.437+0000 7f246e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:16.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.437+0000 7f246e59c640 10 monclient(hunting): handle_auth_done global_id 24469 payload 824 2024-04-26T00:25:16.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.437+0000 7f246e59c640 1 -- 172.21.15.37:0/2205424178 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f247004f510 con 0x7f2470139260 2024-04-26T00:25:16.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.441+0000 7f246e59c640 1 -- 172.21.15.37:0/2205424178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2470139260 msgr2=0x7f247002c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:16.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.441+0000 7f246e59c640 1 --2- 172.21.15.37:0/2205424178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2470139260 0x7f247002c540 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2460010960 tx=0x7f2460004740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:16.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.441+0000 7f245effd640 1 -- 172.21.15.37:0/2205424178 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2460018030 con 0x7f2470139260 2024-04-26T00:25:16.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.441+0000 7f245effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:16.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.441+0000 7f245effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:25:16.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.441+0000 7f245effd640 10 monclient: dump: 2024-04-26T00:25:16.425 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:16.425 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:16.425 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.425 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.425 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:16.425 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:16.425 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:16.426 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:16.426 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:16.426 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:16.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.441+0000 7f245effd640 1 -- 172.21.15.37:0/2205424178 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4719 (secure 0 0 0) 0x7f2460003620 con 0x7f2470139260 2024-04-26T00:25:16.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.441+0000 7f245effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:25:16.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.441+0000 7f245effd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:25:16.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.441+0000 7f245effd640 1 -- 172.21.15.37:0/2205424178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2470139260 msgr2=0x7f247002c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:16.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.441+0000 7f245effd640 1 --2- 172.21.15.37:0/2205424178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2470139260 0x7f247002c540 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2460010960 tx=0x7f2460004740 comp rx=0 tx=0).stop 2024-04-26T00:25:16.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.441+0000 7f245effd640 1 -- 172.21.15.37:0/2205424178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2470139260 msgr2=0x7f247002c540 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:25:16.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.441+0000 7f245effd640 1 --2- 172.21.15.37:0/2205424178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2470139260 0x7f247002c540 secure :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2460010960 tx=0x7f2460004740 comp rx=0 tx=0).stop 2024-04-26T00:25:16.427 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:25:16.427 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:25:16.427 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:25:16.427 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:25:16.427 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 40, 2024-04-26T00:25:16.427 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 224608, 2024-04-26T00:25:16.427 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:25:16.427 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:25:16.427 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:25:16.427 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:25:16.428 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:16.428 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 3, 2024-04-26T00:25:16.428 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:25:16.428 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:25:16.428 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:25:16.428 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:16.428 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:16.428 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:16.428 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:16.428 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:16.428 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:16.428 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:16.429 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:16.429 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:16.429 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:16.429 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:16.429 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:16.429 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:25:16.429 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:25:16.429 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:16.429 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:16.429 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:16.429 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:16.429 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:16.429 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:16.430 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:16.430 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:16.430 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:16.430 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:16.430 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:16.430 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:16.430 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:16.430 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:25:16.430 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:25:16.430 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:25:16.430 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:25:16.430 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:25:16.430 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:25:16.431 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:25:16.431 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:25:16.431 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:25:16.431 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:25:16.431 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:25:16.431 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:25:16.431 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:25:16.431 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:25:16.431 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:25:16.431 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:25:16.431 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:25:16.431 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:16.431 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:16.431 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:16.432 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:16.432 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:16.432 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:16.432 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:16.432 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:16.432 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:16.432 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:16.432 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:16.432 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:16.432 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:25:16.432 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:16.432 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:25:16.432 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:16.433 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:25:16.433 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:25:16.433 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:16.433 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:16.433 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:16.433 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:16.433 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:25:16.433 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:16.433 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:16.433 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:16.433 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:16.433 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:25:16.433 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:16.434 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:16.434 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:16.434 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:16.434 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:25:16.434 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:25:16.434 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:16.434 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:16.434 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:16.434 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:16.434 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:16.434 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:25:16.434 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:25:16.434 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:16.434 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:16.435 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:16.435 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:16.435 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:25:16.435 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:16.435 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:16.435 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:16.435 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:16.435 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:25:16.435 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:16.435 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:16.435 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:16.435 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:16.435 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:25:16.435 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:25:16.435 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:16.436 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:16.436 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:16.436 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:16.436 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:16.436 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:25:16.436 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:25:16.436 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:16.436 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:16.436 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:16.436 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:16.436 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:25:16.436 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:16.436 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:16.436 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:16.436 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:16.437 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:25:16.437 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:16.437 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:16.437 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:16.437 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:16.437 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:25:16.437 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:25:16.437 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:16.437 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:16.437 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:16.437 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:16.437 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:16.437 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:16.437 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:25:16.437 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:25:16.438 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:16.438 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:16.438 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:16.438 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:25:16.438 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:16.438 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:16.438 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:25:16.438 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:16.438 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:16.438 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:16.438 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:25:16.438 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:16.439 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:16.439 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:25:16.439 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:16.439 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:16.439 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:16.439 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:25:16.439 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:16.439 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:16.439 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:16.439 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:25:16.439 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:25:16.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.441+0000 7f2474bae640 1 -- 172.21.15.37:0/2205424178 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2450035e80 msgr2=0x7f2450038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:16.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.441+0000 7f2474bae640 1 --2- 172.21.15.37:0/2205424178 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2450035e80 0x7f2450038340 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2470135e60 tx=0x7f2464009040 comp rx=0 tx=0).stop 2024-04-26T00:25:16.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.441+0000 7f2474bae640 10 monclient: shutdown 2024-04-26T00:25:16.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.441+0000 7f2474bae640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:16.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.441+0000 7f2474bae640 1 -- 172.21.15.37:0/2205424178 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f247013c450 msgr2=0x7f2470138180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:16.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.441+0000 7f2474bae640 1 --2- 172.21.15.37:0/2205424178 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f247013c450 0x7f2470138180 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f24680ae000 tx=0x7f24680196a0 comp rx=0 tx=0).stop 2024-04-26T00:25:16.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.441+0000 7f246ed9d640 1 -- 172.21.15.37:0/2205424178 reap_dead start 2024-04-26T00:25:16.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.441+0000 7f246ed9d640 1 -- 172.21.15.37:0/2205424178 reap_dead start 2024-04-26T00:25:16.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.441+0000 7f2474bae640 1 -- 172.21.15.37:0/2205424178 shutdown_connections 2024-04-26T00:25:16.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.441+0000 7f2474bae640 1 -- 172.21.15.37:0/2205424178 >> 172.21.15.37:0/2205424178 conn(0x7f24700a61b0 msgr2=0x7f24700a8280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:16.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.441+0000 7f2474bae640 1 -- 172.21.15.37:0/2205424178 shutdown_connections 2024-04-26T00:25:16.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.441+0000 7f2474bae640 1 -- 172.21.15.37:0/2205424178 wait complete. 2024-04-26T00:25:16.442 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:25:16.442 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-04-26T00:25:16.442 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-26T00:25:16.442 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-04-26T00:25:16.442 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-04-26T00:25:16.442 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-26T00:25:16.442 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T00:25:16.454 INFO:tasks.ceph.mon.c:Stopped 2024-04-26T00:25:16.454 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-26T00:25:16.454 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:25:16.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.637+0000 7f9b4866f640 10 monclient: get_monmap_and_config 2024-04-26T00:25:16.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.637+0000 7f9b4866f640 10 monclient: build_initial_monmap 2024-04-26T00:25:16.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.637+0000 7f9b4866f640 10 monclient: monmap: 2024-04-26T00:25:16.619 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:16.619 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:16.619 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:25:16.641361+0000 2024-04-26T00:25:16.619 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:25:16.641361+0000 2024-04-26T00:25:16.619 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:16.619 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:16.619 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:25:16.619 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:25:16.619 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:25:16.619 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:25:16.619 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:25:16.619 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:16.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.637+0000 7f9b4866f640 1 Processor -- start 2024-04-26T00:25:16.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.637+0000 7f9b4866f640 1 -- start start 2024-04-26T00:25:16.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.637+0000 7f9b4866f640 10 monclient: init 2024-04-26T00:25:16.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.637+0000 7f9b4866f640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:16.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.637+0000 7f9b4866f640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-04-26T00:25:16.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.637+0000 7f9b4866f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9b40133b20 0x7f9b4009e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:16.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.637+0000 7f9b4866f640 10 monclient(hunting): picked mon.noname-b con 0x7f9b40133b20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:16.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.637+0000 7f9b4866f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9b40133f00 addr v1:172.21.15.45:6789/0 2024-04-26T00:25:16.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.637+0000 7f9b4866f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9b40139050 addr v1:172.21.15.37:6789/0 2024-04-26T00:25:16.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.637+0000 7f9b4866f640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b4009fe20 con 0x7f9b40139050 2024-04-26T00:25:16.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.637+0000 7f9b4866f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b40072d00 con 0x7f9b40133f00 2024-04-26T00:25:16.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.637+0000 7f9b4866f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:16.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.637+0000 7f9b4866f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9b4009e620 con 0x7f9b40133b20 2024-04-26T00:25:16.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.637+0000 7f9b4866f640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:16.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.637+0000 7f9b4866f640 10 monclient(hunting): authenticate will time out at 1297.947025s 2024-04-26T00:25:16.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.637+0000 7f9b4640b640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9b40133b20 msgr2=0x7f9b4009e9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:25:16.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.637+0000 7f9b4640b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9b40133b20 0x7f9b4009e9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:25:16.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.637+0000 7f9b46c0c640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f9b40139050 0x7f9b4013b440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:57238/0 (socket says 172.21.15.37:57238) 2024-04-26T00:25:16.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.637+0000 7f9b46c0c640 1 -- 172.21.15.37:0/1780638291 learned_addr learned my addr 172.21.15.37:0/1780638291 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:25:16.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b46c0c640 1 -- 172.21.15.37:0/1780638291 >> v1:172.21.15.37:6789/0 conn(0x7f9b40139050 legacy=0x7f9b4013b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:16.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45409640 1 -- 172.21.15.37:0/1780638291 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3728875551 0 0) 0x7f9b380017e0 con 0x7f9b40139050 2024-04-26T00:25:16.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45409640 10 monclient(hunting): _negotiate 2024-04-26T00:25:16.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45409640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:16.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45409640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:16.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45409640 10 monclient(hunting): my global_id is 54228 2024-04-26T00:25:16.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45409640 1 -- 172.21.15.37:0/1780638291 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b2c003530 con 0x7f9b40139050 2024-04-26T00:25:16.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45c0a640 1 -- 172.21.15.37:0/1780638291 >> v1:172.21.15.45:6789/0 conn(0x7f9b40133f00 legacy=0x7f9b40134320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:16.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45409640 1 -- 172.21.15.37:0/1780638291 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 574593120 0 0) 0x7f9b2c003530 con 0x7f9b40139050 2024-04-26T00:25:16.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45409640 1 -- 172.21.15.37:0/1780638291 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9b40133b20 msgr2=0x7f9b4009e9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:25:16.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45409640 1 --2- 172.21.15.37:0/1780638291 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9b40133b20 0x7f9b4009e9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:16.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45409640 1 -- 172.21.15.37:0/1780638291 >> v1:172.21.15.45:6789/0 conn(0x7f9b40133f00 legacy=0x7f9b40134320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:16.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45409640 10 monclient: _finish_hunting 0 2024-04-26T00:25:16.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45409640 1 monclient: found mon.noname-a-legacy 2024-04-26T00:25:16.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45409640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:16.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45409640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.37:6789/0 2024-04-26T00:25:16.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45409640 1 -- 172.21.15.37:0/1780638291 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b4009ba60 con 0x7f9b40139050 2024-04-26T00:25:16.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45409640 10 monclient: _finish_auth 0 2024-04-26T00:25:16.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45409640 10 monclient: _check_auth_tickets 2024-04-26T00:25:16.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45409640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:16.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45409640 1 -- 172.21.15.37:0/1780638291 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 866326386 0 0) 0x7f9b38001e80 con 0x7f9b40139050 2024-04-26T00:25:16.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45409640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:16.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45409640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-26T00:25:16.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45409640 10 monclient: dump: 2024-04-26T00:25:16.626 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:16.626 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:16.626 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.627 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.627 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:16.627 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:16.627 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:16.627 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:16.627 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:16.627 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:16.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45409640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] but i'm connected to v1:172.21.15.37:6789/0, reconnecting 2024-04-26T00:25:16.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45409640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:16.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45409640 1 -- 172.21.15.37:0/1780638291 >> v1:172.21.15.37:6789/0 conn(0x7f9b40139050 legacy=0x7f9b4013b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:16.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45409640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:25:16.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45409640 1 --2- 172.21.15.37:0/1780638291 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9b2c0039f0 0x7f9b2c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:16.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45409640 10 monclient(hunting): picked mon.a con 0x7f9b2c0039f0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:25:16.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45409640 1 --2- 172.21.15.37:0/1780638291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b2c004810 0x7f9b2c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:16.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45409640 10 monclient(hunting): picked mon.b con 0x7f9b2c004810 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:25:16.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45409640 1 --2- 172.21.15.37:0/1780638291 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9b40133b20 0x7f9b2c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:16.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45409640 10 monclient(hunting): picked mon.c con 0x7f9b40133b20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:16.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45409640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:16.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45409640 1 -- 172.21.15.37:0/1780638291 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4009e620 con 0x7f9b2c0039f0 2024-04-26T00:25:16.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45409640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:16.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45409640 1 -- 172.21.15.37:0/1780638291 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9b2c005650 con 0x7f9b2c004810 2024-04-26T00:25:16.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45409640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:16.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45409640 1 -- 172.21.15.37:0/1780638291 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9b2c00bf40 con 0x7f9b40133b20 2024-04-26T00:25:16.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45409640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:16.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b45409640 10 monclient(hunting): _finish_auth 0 2024-04-26T00:25:16.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b46c0c640 1 -- 172.21.15.37:0/1780638291 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9b40133b20 msgr2=0x7f9b2c00b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:25:16.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b46c0c640 1 --2- 172.21.15.37:0/1780638291 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9b40133b20 0x7f9b2c00b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:25:16.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b4640b640 1 --2- 172.21.15.37:0/1780638291 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9b2c0039f0 0x7f9b2c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:16.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b4640b640 10 monclient(hunting): get_auth_request con 0x7f9b2c0039f0 auth_method 0 2024-04-26T00:25:16.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b4640b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:16.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b4640b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:16.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b4640b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:25:16.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b4640b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:16.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b4640b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:16.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b4640b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:25:16.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b4640b640 10 monclient(hunting): handle_auth_done global_id 54228 payload 293 2024-04-26T00:25:16.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b4640b640 1 -- 172.21.15.37:0/1780638291 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9b40133b20 msgr2=0x7f9b2c00b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:25:16.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b4640b640 1 --2- 172.21.15.37:0/1780638291 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9b40133b20 0x7f9b2c00b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:16.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b4640b640 1 -- 172.21.15.37:0/1780638291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b2c004810 msgr2=0x7f9b2c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:16.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b4640b640 1 --2- 172.21.15.37:0/1780638291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b2c004810 0x7f9b2c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:16.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b4640b640 10 monclient: _finish_hunting 0 2024-04-26T00:25:16.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b4640b640 1 monclient: found mon.a 2024-04-26T00:25:16.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b4640b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:16.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b4640b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:16.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b4640b640 1 -- 172.21.15.37:0/1780638291 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b2c003530 con 0x7f9b2c0039f0 2024-04-26T00:25:16.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b4640b640 10 monclient: _finish_auth 0 2024-04-26T00:25:16.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b4640b640 10 monclient: _check_auth_tickets 2024-04-26T00:25:16.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b4640b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:16.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b4866f640 5 monclient: authenticate success, global_id 54228 2024-04-26T00:25:16.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.641+0000 7f9b4866f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:25:16.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.645+0000 7f9b4640b640 1 -- 172.21.15.37:0/1780638291 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9b2c0039f0 msgr2=0x7f9b2c003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:16.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.645+0000 7f9b4640b640 1 --2- 172.21.15.37:0/1780638291 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9b2c0039f0 0x7f9b2c003e10 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c003990 tx=0x7f9b3c01c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f95cbfcb8af8653e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:16.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.645+0000 7f9b45409640 1 -- 172.21.15.37:0/1780638291 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c009520 con 0x7f9b2c0039f0 2024-04-26T00:25:16.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.645+0000 7f9b45409640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:16.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.645+0000 7f9b45409640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:16.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.645+0000 7f9b45409640 10 monclient: dump: 2024-04-26T00:25:16.636 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:16.636 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:16.636 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.636 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.636 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:16.636 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:16.636 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:16.637 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:16.637 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:16.637 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:16.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.645+0000 7f9b45409640 1 -- 172.21.15.37:0/1780638291 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b3c003bc0 con 0x7f9b2c0039f0 2024-04-26T00:25:16.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.645+0000 7f9b45409640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:16.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.645+0000 7f9b45409640 1 -- 172.21.15.37:0/1780638291 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c005070 con 0x7f9b2c0039f0 2024-04-26T00:25:16.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.645+0000 7f9b45409640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:16.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.645+0000 7f9b45409640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:16.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.645+0000 7f9b45409640 10 monclient: dump: 2024-04-26T00:25:16.637 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:16.637 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:16.637 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.637 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.637 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:16.637 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:16.638 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:16.638 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:16.638 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:16.638 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:16.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.645+0000 7f9b4866f640 10 monclient: get_monmap_and_config success 2024-04-26T00:25:16.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.645+0000 7f9b4866f640 4 set_mon_vals no callback set 2024-04-26T00:25:16.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.645+0000 7f9b4866f640 10 monclient: shutdown 2024-04-26T00:25:16.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.645+0000 7f9b4866f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:16.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.645+0000 7f9b4866f640 1 -- 172.21.15.37:0/1780638291 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9b2c0039f0 msgr2=0x7f9b2c003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:16.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.645+0000 7f9b4866f640 1 --2- 172.21.15.37:0/1780638291 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9b2c0039f0 0x7f9b2c003e10 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c003990 tx=0x7f9b3c01c3d0 comp rx=0 tx=0).stop 2024-04-26T00:25:16.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.645+0000 7f9b46c0c640 1 -- 172.21.15.37:0/1780638291 reap_dead start 2024-04-26T00:25:16.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.645+0000 7f9b4866f640 1 -- 172.21.15.37:0/1780638291 shutdown_connections 2024-04-26T00:25:16.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.645+0000 7f9b4866f640 1 -- 172.21.15.37:0/1780638291 >> 172.21.15.37:0/1780638291 conn(0x7f9b400a1ad0 msgr2=0x7f9b400a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:16.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.645+0000 7f9b4866f640 1 -- 172.21.15.37:0/1780638291 shutdown_connections 2024-04-26T00:25:16.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.645+0000 7f9b4866f640 1 -- 172.21.15.37:0/1780638291 wait complete. 2024-04-26T00:25:16.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b4866f640 10 monclient: build_initial_monmap 2024-04-26T00:25:16.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b4866f640 10 monclient: monmap: 2024-04-26T00:25:16.639 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:16.639 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:25:16.639 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:25:16.639 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:25:16.639 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:16.639 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:25:16.639 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:25:16.639 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:25:16.639 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:25:16.639 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:16.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b4866f640 1 Processor -- start 2024-04-26T00:25:16.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b4866f640 1 -- start start 2024-04-26T00:25:16.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b4866f640 10 monclient: init 2024-04-26T00:25:16.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b4866f640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:16.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b4866f640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:25:16.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b4866f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9b40132c50 0x7f9b40133030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:16.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b4866f640 10 monclient(hunting): picked mon.noname-c con 0x7f9b40132c50 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:16.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b4866f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9b40140190 0x7f9b401cefe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:16.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b4866f640 10 monclient(hunting): picked mon.noname-a con 0x7f9b40140190 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:25:16.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b4866f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b401cf520 0x7f9b401cf980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:16.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b4866f640 10 monclient(hunting): picked mon.noname-b con 0x7f9b401cf520 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:25:16.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b4866f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:16.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b4866f640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9b400a0720 con 0x7f9b40140190 2024-04-26T00:25:16.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b4866f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:16.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b4866f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9b400a05a0 con 0x7f9b401cf520 2024-04-26T00:25:16.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b4866f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:16.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b4866f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9b400a08a0 con 0x7f9b40132c50 2024-04-26T00:25:16.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b4866f640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:16.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b4866f640 10 monclient(hunting): authenticate will time out at 1297.957045s 2024-04-26T00:25:16.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b4640b640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9b40132c50 msgr2=0x7f9b40133030 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:25:16.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b4640b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9b40132c50 0x7f9b40133030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:25:16.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b45c0a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9b40140190 0x7f9b401cefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:16.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b45c0a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9b40140190 0x7f9b401cefe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36576/0 (socket says 172.21.15.37:36576) 2024-04-26T00:25:16.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b45c0a640 1 -- 172.21.15.37:0/584566084 learned_addr learned my addr 172.21.15.37:0/584566084 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:16.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b45c0a640 10 monclient(hunting): get_auth_request con 0x7f9b40140190 auth_method 0 2024-04-26T00:25:16.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b45c0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:16.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b45c0a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:16.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b45c0a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:16.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b45c0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:16.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b45c0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:16.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b45c0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:16.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b46c0c640 1 --2- 172.21.15.37:0/584566084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b401cf520 0x7f9b401cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:16.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b46c0c640 10 monclient(hunting): get_auth_request con 0x7f9b401cf520 auth_method 0 2024-04-26T00:25:16.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b46c0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:16.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b46c0c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:16.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b46c0c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:16.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b45c0a640 10 monclient(hunting): handle_auth_done global_id 54231 payload 824 2024-04-26T00:25:16.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b45c0a640 1 -- 172.21.15.37:0/584566084 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9b40132c50 msgr2=0x7f9b40133030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:25:16.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b45c0a640 1 --2- 172.21.15.37:0/584566084 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9b40132c50 0x7f9b40133030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:16.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b45c0a640 1 -- 172.21.15.37:0/584566084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b401cf520 msgr2=0x7f9b401cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:16.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b45c0a640 1 --2- 172.21.15.37:0/584566084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b401cf520 0x7f9b401cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:16.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b45c0a640 10 monclient: _finish_hunting 0 2024-04-26T00:25:16.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b45c0a640 1 monclient: found mon.noname-a 2024-04-26T00:25:16.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b45c0a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:16.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b45c0a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:25:16.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b45c0a640 1 -- 172.21.15.37:0/584566084 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b400a0000 con 0x7f9b40140190 2024-04-26T00:25:16.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b45c0a640 1 -- 172.21.15.37:0/584566084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9b40140190 msgr2=0x7f9b401cefe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:16.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.649+0000 7f9b45c0a640 1 --2- 172.21.15.37:0/584566084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9b40140190 0x7f9b401cefe0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9b3000c500 tx=0x7f9b3000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:16.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.653+0000 7f9b36ffd640 1 -- 172.21.15.37:0/584566084 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b30017070 con 0x7f9b40140190 2024-04-26T00:25:16.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.653+0000 7f9b36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:16.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.653+0000 7f9b36ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:25:16.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.653+0000 7f9b36ffd640 10 monclient: dump: 2024-04-26T00:25:16.643 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:16.643 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:16.643 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.643 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.643 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:16.643 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:16.643 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:16.643 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:16.643 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:16.643 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:16.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.653+0000 7f9b36ffd640 10 monclient: _finish_auth 0 2024-04-26T00:25:16.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.653+0000 7f9b36ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:25:16.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.653+0000 7f9b36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:16.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.653+0000 7f9b36ffd640 1 -- 172.21.15.37:0/584566084 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b300057d0 con 0x7f9b40140190 2024-04-26T00:25:16.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.653+0000 7f9b4866f640 5 monclient: authenticate success, global_id 54231 2024-04-26T00:25:16.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.653+0000 7f9b4866f640 10 monclient: _renew_subs 2024-04-26T00:25:16.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.653+0000 7f9b4866f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:16.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.653+0000 7f9b4866f640 1 -- 172.21.15.37:0/584566084 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b4009b660 con 0x7f9b40140190 2024-04-26T00:25:16.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.653+0000 7f9b36ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:16.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.653+0000 7f9b36ffd640 1 -- 172.21.15.37:0/584566084 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b30007690 con 0x7f9b40140190 2024-04-26T00:25:16.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.653+0000 7f9b36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:16.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.653+0000 7f9b36ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:16.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.653+0000 7f9b36ffd640 10 monclient: dump: 2024-04-26T00:25:16.644 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:16.644 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:16.644 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.645 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:16.645 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:16.645 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:16.645 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:16.645 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:16.645 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:16.645 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:16.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.653+0000 7f9b4866f640 10 monclient: _renew_subs 2024-04-26T00:25:16.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.653+0000 7f9b4866f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:16.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.653+0000 7f9b44c08640 4 set_mon_vals no callback set 2024-04-26T00:25:16.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.653+0000 7f9b4866f640 1 -- 172.21.15.37:0/584566084 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b4013f1f0 con 0x7f9b40140190 2024-04-26T00:25:16.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.653+0000 7f9b4866f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:16.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.653+0000 7f9b4866f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:16.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.653+0000 7f9b4866f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:16.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.653+0000 7f9b36ffd640 1 -- 172.21.15.37:0/584566084 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9b3001a070 con 0x7f9b40140190 2024-04-26T00:25:16.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.653+0000 7f9b4866f640 1 -- 172.21.15.37:0/584566084 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b40066f40 con 0x7f9b40140190 2024-04-26T00:25:16.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.653+0000 7f9b36ffd640 1 --2- 172.21.15.37:0/584566084 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9b24035e80 0x7f9b24038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:16.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.653+0000 7f9b36ffd640 1 -- 172.21.15.37:0/584566084 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(208..208 src has 172..208) ==== 3963+0+0 (secure 0 0 0) 0x7f9b30047f10 con 0x7f9b40140190 2024-04-26T00:25:16.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.653+0000 7f9b4640b640 1 --2- 172.21.15.37:0/584566084 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9b24035e80 0x7f9b24038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:16.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.653+0000 7f9b4640b640 10 monclient: get_auth_request con 0x7f9b24035e80 auth_method 0 2024-04-26T00:25:16.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.657+0000 7f9b4640b640 1 --2- 172.21.15.37:0/584566084 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9b24035e80 0x7f9b24038340 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c008000 tx=0x7f9b3c005a90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:16.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.657+0000 7f9b36ffd640 1 -- 172.21.15.37:0/584566084 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9b30046c40 con 0x7f9b40140190 2024-04-26T00:25:16.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.657+0000 7f9b36ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:16.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.657+0000 7f9b36ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:25:16.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.885+0000 7f9b4866f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:25:16.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.885+0000 7f9b4866f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:25:16.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.885+0000 7f9b4866f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:16.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.885+0000 7f9b4866f640 1 -- 172.21.15.37:0/584566084 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9b4006b9a0 con 0x7f9b40140190 2024-04-26T00:25:16.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.885+0000 7f9b36ffd640 1 -- 172.21.15.37:0/584566084 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f9b3001abb0 con 0x7f9b40140190 2024-04-26T00:25:16.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.885+0000 7f9b36ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:25:16.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.885+0000 7f9b36ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:25:16.867 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:25:16.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.889+0000 7f9b4866f640 1 -- 172.21.15.37:0/584566084 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9b24035e80 msgr2=0x7f9b24038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:16.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.889+0000 7f9b4866f640 1 --2- 172.21.15.37:0/584566084 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9b24035e80 0x7f9b24038340 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c008000 tx=0x7f9b3c005a90 comp rx=0 tx=0).stop 2024-04-26T00:25:16.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.889+0000 7f9b4866f640 10 monclient: shutdown 2024-04-26T00:25:16.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.889+0000 7f9b4866f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:16.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.889+0000 7f9b4866f640 1 -- 172.21.15.37:0/584566084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9b40140190 msgr2=0x7f9b401cefe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:16.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.889+0000 7f9b4866f640 1 --2- 172.21.15.37:0/584566084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9b40140190 0x7f9b401cefe0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9b3000c500 tx=0x7f9b3000c9c0 comp rx=0 tx=0).stop 2024-04-26T00:25:16.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.889+0000 7f9b4866f640 1 -- 172.21.15.37:0/584566084 shutdown_connections 2024-04-26T00:25:16.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.889+0000 7f9b4866f640 1 --2- 172.21.15.37:0/584566084 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9b24035e80 0x7f9b24038340 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:16.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.889+0000 7f9b4866f640 1 --2- 172.21.15.37:0/584566084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b401cf520 0x7f9b401cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:16.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.889+0000 7f9b4866f640 1 --2- 172.21.15.37:0/584566084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9b40140190 0x7f9b401cefe0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:16.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.889+0000 7f9b4866f640 1 --2- 172.21.15.37:0/584566084 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9b40132c50 0x7f9b40133030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:16.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.889+0000 7f9b4866f640 1 -- 172.21.15.37:0/584566084 >> 172.21.15.37:0/584566084 conn(0x7f9b400a1ad0 msgr2=0x7f9b400a2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:16.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.889+0000 7f9b4866f640 1 -- 172.21.15.37:0/584566084 shutdown_connections 2024-04-26T00:25:16.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:16.889+0000 7f9b4866f640 1 -- 172.21.15.37:0/584566084 wait complete. 2024-04-26T00:25:19.887 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:25:20.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d4544640 10 monclient: get_monmap_and_config 2024-04-26T00:25:20.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d4544640 10 monclient: build_initial_monmap 2024-04-26T00:25:20.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d4544640 10 monclient: monmap: 2024-04-26T00:25:20.019 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:20.019 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:20.019 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:25:20.041389+0000 2024-04-26T00:25:20.019 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:25:20.041389+0000 2024-04-26T00:25:20.019 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:20.019 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:20.019 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:25:20.019 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:25:20.019 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:25:20.019 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:25:20.019 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:25:20.019 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:20.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d4544640 1 Processor -- start 2024-04-26T00:25:20.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d4544640 1 -- start start 2024-04-26T00:25:20.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d4544640 10 monclient: init 2024-04-26T00:25:20.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d4544640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:20.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d4544640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-04-26T00:25:20.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d4544640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb8cc0a0260 addr v1:172.21.15.45:6789/0 2024-04-26T00:25:20.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d4544640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8cc137e40 0x7fb8cc13a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:20.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d4544640 10 monclient(hunting): picked mon.noname-b con 0x7fb8cc137e40 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:20.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d4544640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb8cc13a770 0x7fb8cc13cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:20.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d4544640 10 monclient(hunting): picked mon.noname-a con 0x7fb8cc13a770 addr v2:172.21.15.37:3300/0 2024-04-26T00:25:20.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d4544640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8cc0a1130 con 0x7fb8cc0a0260 2024-04-26T00:25:20.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d4544640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:20.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d4544640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fb8cc0a0d50 con 0x7fb8cc13a770 2024-04-26T00:25:20.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d4544640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:20.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d4544640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb8cc09ec10 con 0x7fb8cc137e40 2024-04-26T00:25:20.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d4544640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:20.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d4544640 10 monclient(hunting): authenticate will time out at 1301.347084s 2024-04-26T00:25:20.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d1adf640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8cc137e40 msgr2=0x7fb8cc13a230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:25:20.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d1adf640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8cc137e40 0x7fb8cc13a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:25:20.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d2ae1640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb8cc13a770 0x7fb8cc13cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:20.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d2ae1640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb8cc13a770 0x7fb8cc13cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:46518/0 (socket says 172.21.15.37:46518) 2024-04-26T00:25:20.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d2ae1640 1 -- 172.21.15.37:0/3303834788 learned_addr learned my addr 172.21.15.37:0/3303834788 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:20.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d2ae1640 10 monclient(hunting): get_auth_request con 0x7fb8cc13a770 auth_method 0 2024-04-26T00:25:20.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d2ae1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:20.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d2ae1640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:20.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d2ae1640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:20.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d2ae1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:20.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d2ae1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:20.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d2ae1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:20.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d2ae1640 10 monclient(hunting): handle_auth_done global_id 54234 payload 274 2024-04-26T00:25:20.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d2ae1640 1 -- 172.21.15.37:0/3303834788 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8cc137e40 msgr2=0x7fb8cc13a230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:25:20.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d2ae1640 1 --2- 172.21.15.37:0/3303834788 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8cc137e40 0x7fb8cc13a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:20.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d2ae1640 1 -- 172.21.15.37:0/3303834788 >> v1:172.21.15.45:6789/0 conn(0x7fb8cc0a0260 legacy=0x7fb8cc0a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:20.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d2ae1640 10 monclient: _finish_hunting 0 2024-04-26T00:25:20.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d2ae1640 1 monclient: found mon.noname-a 2024-04-26T00:25:20.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d2ae1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:20.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d2ae1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:25:20.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.037+0000 7fb8d2ae1640 1 -- 172.21.15.37:0/3303834788 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8cc06cff0 con 0x7fb8cc13a770 2024-04-26T00:25:20.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.041+0000 7fb8d2ae1640 1 -- 172.21.15.37:0/3303834788 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8cc13a770 msgr2=0x7fb8cc13cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:20.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.041+0000 7fb8d2ae1640 1 --2- 172.21.15.37:0/3303834788 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8cc13a770 0x7fb8cc13cb60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb8c4009e50 tx=0x7fb8c401c6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b0a2c7c25b605e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:20.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.041+0000 7fb8d12de640 1 -- 172.21.15.37:0/3303834788 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8c40a9480 con 0x7fb8cc13a770 2024-04-26T00:25:20.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d12de640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:20.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d12de640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:25:20.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d12de640 10 monclient: dump: 2024-04-26T00:25:20.025 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:20.025 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:20.025 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:20.025 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:20.026 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:20.026 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:20.026 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:20.026 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:20.026 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:20.026 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:20.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d12de640 10 monclient: _finish_auth 0 2024-04-26T00:25:20.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d12de640 10 monclient: _check_auth_tickets 2024-04-26T00:25:20.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d12de640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:20.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d4544640 5 monclient: authenticate success, global_id 54234 2024-04-26T00:25:20.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d12de640 1 -- 172.21.15.37:0/3303834788 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8c40ae070 con 0x7fb8cc13a770 2024-04-26T00:25:20.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d4544640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:25:20.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d12de640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:20.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d4544640 10 monclient: get_monmap_and_config success 2024-04-26T00:25:20.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d12de640 1 -- 172.21.15.37:0/3303834788 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8c40a9840 con 0x7fb8cc13a770 2024-04-26T00:25:20.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d4544640 4 set_mon_vals no callback set 2024-04-26T00:25:20.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d12de640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:20.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d12de640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:20.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d12de640 10 monclient: dump: 2024-04-26T00:25:20.027 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:20.027 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:20.027 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:20.028 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:20.028 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:20.028 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:20.028 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:20.028 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:20.028 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:20.028 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:20.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d4544640 10 monclient: shutdown 2024-04-26T00:25:20.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d4544640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:20.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d4544640 1 -- 172.21.15.37:0/3303834788 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8cc13a770 msgr2=0x7fb8cc13cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:20.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d4544640 1 --2- 172.21.15.37:0/3303834788 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8cc13a770 0x7fb8cc13cb60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb8c4009e50 tx=0x7fb8c401c6f0 comp rx=0 tx=0).stop 2024-04-26T00:25:20.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d4544640 1 -- 172.21.15.37:0/3303834788 shutdown_connections 2024-04-26T00:25:20.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d4544640 1 --2- 172.21.15.37:0/3303834788 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8cc13a770 0x7fb8cc13cb60 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:20.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d4544640 1 --2- 172.21.15.37:0/3303834788 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8cc137e40 0x7fb8cc13a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:20.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d4544640 1 -- 172.21.15.37:0/3303834788 >> 172.21.15.37:0/3303834788 conn(0x7fb8cc0a1ad0 msgr2=0x7fb8cc0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:20.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d4544640 1 -- 172.21.15.37:0/3303834788 shutdown_connections 2024-04-26T00:25:20.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d4544640 1 -- 172.21.15.37:0/3303834788 wait complete. 2024-04-26T00:25:20.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d4544640 10 monclient: build_initial_monmap 2024-04-26T00:25:20.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d4544640 10 monclient: monmap: 2024-04-26T00:25:20.035 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:20.035 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:25:20.035 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:25:20.035 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:25:20.035 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:20.035 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:25:20.035 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:25:20.035 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:25:20.035 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:25:20.035 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:20.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d4544640 1 Processor -- start 2024-04-26T00:25:20.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d4544640 1 -- start start 2024-04-26T00:25:20.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d4544640 10 monclient: init 2024-04-26T00:25:20.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d4544640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:20.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d4544640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:25:20.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d4544640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8cc0a0260 0x7fb8cc1cb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:20.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d4544640 10 monclient(hunting): picked mon.noname-b con 0x7fb8cc0a0260 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:25:20.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d4544640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8cc137e40 0x7fb8cc136fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:20.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d4544640 10 monclient(hunting): picked mon.noname-a con 0x7fb8cc137e40 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:25:20.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d4544640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8cc13a770 0x7fb8cc137500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:20.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d4544640 10 monclient(hunting): picked mon.noname-c con 0x7fb8cc13a770 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:20.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d4544640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:20.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d4544640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb8cc09f4c0 con 0x7fb8cc137e40 2024-04-26T00:25:20.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d4544640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:20.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d4544640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb8cc09f0f0 con 0x7fb8cc0a0260 2024-04-26T00:25:20.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d4544640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:20.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d4544640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb8cc13f8c0 con 0x7fb8cc13a770 2024-04-26T00:25:20.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d4544640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:20.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d4544640 10 monclient(hunting): authenticate will time out at 1301.354148s 2024-04-26T00:25:20.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d2ae1640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8cc13a770 msgr2=0x7fb8cc137500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:25:20.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d2ae1640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8cc13a770 0x7fb8cc137500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:25:20.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d1adf640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8cc137e40 0x7fb8cc136fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:20.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d1adf640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8cc137e40 0x7fb8cc136fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:46530/0 (socket says 172.21.15.37:46530) 2024-04-26T00:25:20.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d1adf640 1 -- 172.21.15.37:0/283080156 learned_addr learned my addr 172.21.15.37:0/283080156 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:20.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d1adf640 10 monclient(hunting): get_auth_request con 0x7fb8cc137e40 auth_method 0 2024-04-26T00:25:20.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d1adf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:20.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d1adf640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:20.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d1adf640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:20.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d1adf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:20.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d1adf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:20.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d1adf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:20.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d22e0640 1 --2- 172.21.15.37:0/283080156 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8cc0a0260 0x7fb8cc1cb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:20.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d22e0640 10 monclient(hunting): get_auth_request con 0x7fb8cc0a0260 auth_method 0 2024-04-26T00:25:20.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d22e0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:20.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d22e0640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:20.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d22e0640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:20.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d1adf640 10 monclient(hunting): handle_auth_done global_id 54237 payload 824 2024-04-26T00:25:20.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d1adf640 1 -- 172.21.15.37:0/283080156 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8cc13a770 msgr2=0x7fb8cc137500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:25:20.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d1adf640 1 --2- 172.21.15.37:0/283080156 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8cc13a770 0x7fb8cc137500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:20.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d1adf640 1 -- 172.21.15.37:0/283080156 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8cc0a0260 msgr2=0x7fb8cc1cb9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:20.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d1adf640 1 --2- 172.21.15.37:0/283080156 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8cc0a0260 0x7fb8cc1cb9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:20.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d1adf640 10 monclient: _finish_hunting 0 2024-04-26T00:25:20.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d1adf640 1 monclient: found mon.noname-a 2024-04-26T00:25:20.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d1adf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:20.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d1adf640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:25:20.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.045+0000 7fb8d1adf640 1 -- 172.21.15.37:0/283080156 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8cc123970 con 0x7fb8cc137e40 2024-04-26T00:25:20.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.049+0000 7fb8d1adf640 1 -- 172.21.15.37:0/283080156 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8cc137e40 msgr2=0x7fb8cc136fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:20.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.049+0000 7fb8d1adf640 1 --2- 172.21.15.37:0/283080156 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8cc137e40 0x7fb8cc136fc0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb8c800b550 tx=0x7fb8c800ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:20.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.049+0000 7fb8c2ffd640 1 -- 172.21.15.37:0/283080156 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8c80050f0 con 0x7fb8cc137e40 2024-04-26T00:25:20.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.049+0000 7fb8c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:20.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.049+0000 7fb8c2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:25:20.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.049+0000 7fb8c2ffd640 10 monclient: dump: 2024-04-26T00:25:20.039 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:20.039 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:20.039 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:20.039 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:20.039 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:20.040 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:20.040 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:20.040 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:20.040 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:20.040 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:20.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.049+0000 7fb8c2ffd640 10 monclient: _finish_auth 0 2024-04-26T00:25:20.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.049+0000 7fb8c2ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:25:20.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.049+0000 7fb8c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:20.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.049+0000 7fb8c2ffd640 1 -- 172.21.15.37:0/283080156 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8c8015070 con 0x7fb8cc137e40 2024-04-26T00:25:20.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.049+0000 7fb8c2ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:20.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.049+0000 7fb8c2ffd640 1 -- 172.21.15.37:0/283080156 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8c80054d0 con 0x7fb8cc137e40 2024-04-26T00:25:20.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.049+0000 7fb8d4544640 5 monclient: authenticate success, global_id 54237 2024-04-26T00:25:20.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.049+0000 7fb8d4544640 10 monclient: _renew_subs 2024-04-26T00:25:20.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.049+0000 7fb8d4544640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:20.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.049+0000 7fb8d4544640 1 -- 172.21.15.37:0/283080156 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8cc09e1a0 con 0x7fb8cc137e40 2024-04-26T00:25:20.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.049+0000 7fb8c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:20.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.049+0000 7fb8c2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:20.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.049+0000 7fb8d0add640 4 set_mon_vals no callback set 2024-04-26T00:25:20.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.049+0000 7fb8c2ffd640 10 monclient: dump: 2024-04-26T00:25:20.041 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:20.041 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:20.041 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:20.041 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:20.041 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:20.041 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:20.042 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:20.042 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:20.042 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:20.042 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:20.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.049+0000 7fb8d4544640 10 monclient: _renew_subs 2024-04-26T00:25:20.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.049+0000 7fb8d4544640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:20.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.049+0000 7fb8d4544640 1 -- 172.21.15.37:0/283080156 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8cc1221e0 con 0x7fb8cc137e40 2024-04-26T00:25:20.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.049+0000 7fb8d4544640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:20.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.049+0000 7fb8c2ffd640 1 -- 172.21.15.37:0/283080156 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb8c80056b0 con 0x7fb8cc137e40 2024-04-26T00:25:20.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.049+0000 7fb8d4544640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:20.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.049+0000 7fb8d4544640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:20.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.049+0000 7fb8d4544640 1 -- 172.21.15.37:0/283080156 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8cc066f40 con 0x7fb8cc137e40 2024-04-26T00:25:20.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.049+0000 7fb8c2ffd640 1 --2- 172.21.15.37:0/283080156 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb8b0035e30 0x7fb8b00382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:20.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.049+0000 7fb8c2ffd640 1 -- 172.21.15.37:0/283080156 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(208..208 src has 172..208) ==== 3963+0+0 (secure 0 0 0) 0x7fb8c80489c0 con 0x7fb8cc137e40 2024-04-26T00:25:20.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.049+0000 7fb8d22e0640 1 --2- 172.21.15.37:0/283080156 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb8b0035e30 0x7fb8b00382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:20.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.049+0000 7fb8d22e0640 10 monclient: get_auth_request con 0x7fb8b0035e30 auth_method 0 2024-04-26T00:25:20.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.053+0000 7fb8d22e0640 1 --2- 172.21.15.37:0/283080156 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb8b0035e30 0x7fb8b00382f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc09dfd0 tx=0x7fb8bc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:20.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.053+0000 7fb8c2ffd640 1 -- 172.21.15.37:0/283080156 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fb8c800e340 con 0x7fb8cc137e40 2024-04-26T00:25:20.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.053+0000 7fb8c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:20.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.053+0000 7fb8c2ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:25:20.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.277+0000 7fb8d4544640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:25:20.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.277+0000 7fb8d4544640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:25:20.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.277+0000 7fb8d4544640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:20.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.277+0000 7fb8d4544640 1 -- 172.21.15.37:0/283080156 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb8cc123580 con 0x7fb8cc137e40 2024-04-26T00:25:20.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.277+0000 7fb8c2ffd640 1 -- 172.21.15.37:0/283080156 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fb8c80476f0 con 0x7fb8cc137e40 2024-04-26T00:25:20.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.277+0000 7fb8c2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:25:20.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.277+0000 7fb8c2ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:25:20.258 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:25:20.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.277+0000 7fb8d4544640 1 -- 172.21.15.37:0/283080156 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb8b0035e30 msgr2=0x7fb8b00382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:20.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.277+0000 7fb8d4544640 1 --2- 172.21.15.37:0/283080156 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb8b0035e30 0x7fb8b00382f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc09dfd0 tx=0x7fb8bc009040 comp rx=0 tx=0).stop 2024-04-26T00:25:20.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.277+0000 7fb8d4544640 10 monclient: shutdown 2024-04-26T00:25:20.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.277+0000 7fb8d4544640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:20.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.277+0000 7fb8d4544640 1 -- 172.21.15.37:0/283080156 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8cc137e40 msgr2=0x7fb8cc136fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:20.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.277+0000 7fb8d4544640 1 --2- 172.21.15.37:0/283080156 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8cc137e40 0x7fb8cc136fc0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb8c800b550 tx=0x7fb8c800ba10 comp rx=0 tx=0).stop 2024-04-26T00:25:20.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.277+0000 7fb8d4544640 1 -- 172.21.15.37:0/283080156 shutdown_connections 2024-04-26T00:25:20.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.277+0000 7fb8d4544640 1 --2- 172.21.15.37:0/283080156 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb8b0035e30 0x7fb8b00382f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:20.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.277+0000 7fb8d4544640 1 --2- 172.21.15.37:0/283080156 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8cc13a770 0x7fb8cc137500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:20.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.277+0000 7fb8d4544640 1 --2- 172.21.15.37:0/283080156 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8cc137e40 0x7fb8cc136fc0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:20.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.277+0000 7fb8d4544640 1 --2- 172.21.15.37:0/283080156 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8cc0a0260 0x7fb8cc1cb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:20.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.277+0000 7fb8d4544640 1 -- 172.21.15.37:0/283080156 >> 172.21.15.37:0/283080156 conn(0x7fb8cc0a1ad0 msgr2=0x7fb8cc1388f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:20.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.277+0000 7fb8d4544640 1 -- 172.21.15.37:0/283080156 shutdown_connections 2024-04-26T00:25:20.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:20.277+0000 7fb8d4544640 1 -- 172.21.15.37:0/283080156 wait complete. 2024-04-26T00:25:23.277 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:25:23.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.429+0000 7f7e67d26640 10 monclient: get_monmap_and_config 2024-04-26T00:25:23.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.429+0000 7f7e67d26640 10 monclient: build_initial_monmap 2024-04-26T00:25:23.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.429+0000 7f7e67d26640 10 monclient: monmap: 2024-04-26T00:25:23.411 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:23.412 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:23.412 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:25:23.433722+0000 2024-04-26T00:25:23.412 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:25:23.433722+0000 2024-04-26T00:25:23.412 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:23.412 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:23.412 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:25:23.412 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:25:23.412 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:25:23.412 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:25:23.413 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:25:23.413 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:23.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.429+0000 7f7e67d26640 1 Processor -- start 2024-04-26T00:25:23.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.429+0000 7f7e67d26640 1 -- start start 2024-04-26T00:25:23.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.429+0000 7f7e67d26640 10 monclient: init 2024-04-26T00:25:23.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.429+0000 7f7e67d26640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:23.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.429+0000 7f7e67d26640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-04-26T00:25:23.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.429+0000 7f7e67d26640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f7e60136e10 0x7f7e6009c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:23.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.429+0000 7f7e67d26640 10 monclient(hunting): picked mon.noname-a con 0x7f7e60136e10 addr v2:172.21.15.37:3300/0 2024-04-26T00:25:23.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.429+0000 7f7e67d26640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7e601371f0 addr v1:172.21.15.45:6789/0 2024-04-26T00:25:23.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.429+0000 7f7e67d26640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7e6013c340 0x7f7e6013e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:23.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.429+0000 7f7e67d26640 10 monclient(hunting): picked mon.noname-c con 0x7f7e6013c340 addr v2:172.21.15.45:3300/0 2024-04-26T00:25:23.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.429+0000 7f7e67d26640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e6009b680 con 0x7f7e601371f0 2024-04-26T00:25:23.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.429+0000 7f7e67d26640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:23.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.429+0000 7f7e67d26640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f7e6009b8f0 con 0x7f7e60136e10 2024-04-26T00:25:23.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.429+0000 7f7e67d26640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:23.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.429+0000 7f7e67d26640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f7e6009bbd0 con 0x7f7e6013c340 2024-04-26T00:25:23.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.429+0000 7f7e67d26640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:23.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.429+0000 7f7e67d26640 10 monclient(hunting): authenticate will time out at 1304.739527s 2024-04-26T00:25:23.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.429+0000 7f7e65ac2640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f7e60136e10 0x7f7e6009c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:23.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.429+0000 7f7e65ac2640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f7e60136e10 0x7f7e6009c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:46538/0 (socket says 172.21.15.37:46538) 2024-04-26T00:25:23.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.429+0000 7f7e65ac2640 1 -- 172.21.15.37:0/3052386900 learned_addr learned my addr 172.21.15.37:0/3052386900 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:23.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.429+0000 7f7e65ac2640 10 monclient(hunting): get_auth_request con 0x7f7e60136e10 auth_method 0 2024-04-26T00:25:23.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.429+0000 7f7e65ac2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:23.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.429+0000 7f7e65ac2640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:23.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.429+0000 7f7e65ac2640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:23.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.429+0000 7f7e65ac2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:23.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.429+0000 7f7e65ac2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:23.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.429+0000 7f7e65ac2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:23.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.433+0000 7f7e662c3640 1 --2- 172.21.15.37:0/3052386900 >> v2:172.21.15.45:3300/0 conn(0x7f7e6013c340 0x7f7e6013e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:23.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.433+0000 7f7e65ac2640 10 monclient(hunting): handle_auth_done global_id 54240 payload 274 2024-04-26T00:25:23.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.433+0000 7f7e65ac2640 1 -- 172.21.15.37:0/3052386900 >> v2:172.21.15.45:3300/0 conn(0x7f7e6013c340 msgr2=0x7f7e6013e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:23.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.433+0000 7f7e65ac2640 1 --2- 172.21.15.37:0/3052386900 >> v2:172.21.15.45:3300/0 conn(0x7f7e6013c340 0x7f7e6013e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:23.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.433+0000 7f7e65ac2640 1 -- 172.21.15.37:0/3052386900 >> v1:172.21.15.45:6789/0 conn(0x7f7e601371f0 legacy=0x7f7e60137600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:23.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.433+0000 7f7e65ac2640 10 monclient: _finish_hunting 0 2024-04-26T00:25:23.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.433+0000 7f7e65ac2640 1 monclient: found mon.noname-a 2024-04-26T00:25:23.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.433+0000 7f7e65ac2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:23.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.433+0000 7f7e65ac2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:25:23.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.433+0000 7f7e65ac2640 1 -- 172.21.15.37:0/3052386900 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e600a5d90 con 0x7f7e60136e10 2024-04-26T00:25:23.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.433+0000 7f7e662c3640 10 monclient: get_auth_request con 0x7f7e6013c340 auth_method 0 2024-04-26T00:25:23.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.433+0000 7f7e662c3640 1 --2- 172.21.15.37:0/3052386900 >> v2:172.21.15.45:3300/0 conn(0x7f7e6013c340 0x7f7e6013e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:25:23.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e65ac2640 1 -- 172.21.15.37:0/3052386900 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e60136e10 msgr2=0x7f7e6009c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:23.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e65ac2640 1 --2- 172.21.15.37:0/3052386900 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e60136e10 0x7f7e6009c5b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7e600a9590 tx=0x7f7e5801c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=373f48505a4cd4bd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:23.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e64ac0640 1 -- 172.21.15.37:0/3052386900 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e580b0070 con 0x7f7e60136e10 2024-04-26T00:25:23.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e64ac0640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:23.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e64ac0640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:25:23.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e64ac0640 10 monclient: dump: 2024-04-26T00:25:23.428 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:23.428 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:23.428 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:23.428 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:23.428 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:23.428 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:23.428 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:23.428 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:23.429 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:23.429 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:23.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e64ac0640 10 monclient: _finish_auth 0 2024-04-26T00:25:23.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e64ac0640 10 monclient: _check_auth_tickets 2024-04-26T00:25:23.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e64ac0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:23.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e64ac0640 1 -- 172.21.15.37:0/3052386900 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e580ab600 con 0x7f7e60136e10 2024-04-26T00:25:23.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e64ac0640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:23.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e64ac0640 1 -- 172.21.15.37:0/3052386900 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e580ab9d0 con 0x7f7e60136e10 2024-04-26T00:25:23.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e67d26640 5 monclient: authenticate success, global_id 54240 2024-04-26T00:25:23.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e67d26640 10 monclient: get_monmap_and_config success 2024-04-26T00:25:23.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e67d26640 4 set_mon_vals no callback set 2024-04-26T00:25:23.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e64ac0640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:23.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e64ac0640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:23.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e64ac0640 10 monclient: dump: 2024-04-26T00:25:23.430 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:23.430 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:23.430 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:23.430 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:23.430 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:23.430 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:23.430 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:23.430 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:23.430 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:23.430 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:23.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e67d26640 10 monclient: shutdown 2024-04-26T00:25:23.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e67d26640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:23.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e67d26640 1 -- 172.21.15.37:0/3052386900 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e60136e10 msgr2=0x7f7e6009c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:23.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e67d26640 1 --2- 172.21.15.37:0/3052386900 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e60136e10 0x7f7e6009c5b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7e600a9590 tx=0x7f7e5801c3a0 comp rx=0 tx=0).stop 2024-04-26T00:25:23.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e67d26640 1 -- 172.21.15.37:0/3052386900 shutdown_connections 2024-04-26T00:25:23.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e67d26640 1 --2- 172.21.15.37:0/3052386900 >> v2:172.21.15.45:3300/0 conn(0x7f7e6013c340 0x7f7e6013e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:23.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e67d26640 1 --2- 172.21.15.37:0/3052386900 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e60136e10 0x7f7e6009c5b0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:23.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e67d26640 1 -- 172.21.15.37:0/3052386900 >> 172.21.15.37:0/3052386900 conn(0x7f7e600a64e0 msgr2=0x7f7e600a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:23.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e67d26640 1 -- 172.21.15.37:0/3052386900 shutdown_connections 2024-04-26T00:25:23.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e67d26640 1 -- 172.21.15.37:0/3052386900 wait complete. 2024-04-26T00:25:23.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e67d26640 10 monclient: build_initial_monmap 2024-04-26T00:25:23.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e67d26640 10 monclient: monmap: 2024-04-26T00:25:23.433 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:23.433 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:25:23.434 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:25:23.434 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:25:23.434 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:23.434 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:25:23.434 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:25:23.434 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:25:23.434 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:25:23.434 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:23.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e67d26640 1 Processor -- start 2024-04-26T00:25:23.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e67d26640 1 -- start start 2024-04-26T00:25:23.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e67d26640 10 monclient: init 2024-04-26T00:25:23.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e67d26640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:23.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e67d26640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:25:23.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e67d26640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7e60136e10 0x7f7e601cec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:23.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e67d26640 10 monclient(hunting): picked mon.noname-c con 0x7f7e60136e10 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:23.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e67d26640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e601371f0 0x7f7e60135de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:23.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e67d26640 10 monclient(hunting): picked mon.noname-b con 0x7f7e601371f0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:25:23.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e67d26640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e6013c340 0x7f7e60138bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:23.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e67d26640 10 monclient(hunting): picked mon.noname-a con 0x7f7e6013c340 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:25:23.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e67d26640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:23.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e67d26640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f7e60141820 con 0x7f7e6013c340 2024-04-26T00:25:23.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e67d26640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:23.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e67d26640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7e601416a0 con 0x7f7e601371f0 2024-04-26T00:25:23.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e67d26640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:23.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e67d26640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f7e601419a0 con 0x7f7e60136e10 2024-04-26T00:25:23.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e67d26640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:23.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e67d26640 10 monclient(hunting): authenticate will time out at 1304.746815s 2024-04-26T00:25:23.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e65ac2640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7e60136e10 msgr2=0x7f7e601cec40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:25:23.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e65ac2640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7e60136e10 0x7f7e601cec40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:25:23.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e662c3640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e6013c340 0x7f7e60138bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:23.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e662c3640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e6013c340 0x7f7e60138bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:46552/0 (socket says 172.21.15.37:46552) 2024-04-26T00:25:23.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e662c3640 1 -- 172.21.15.37:0/3756295541 learned_addr learned my addr 172.21.15.37:0/3756295541 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:23.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e662c3640 10 monclient(hunting): get_auth_request con 0x7f7e6013c340 auth_method 0 2024-04-26T00:25:23.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e662c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:23.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e662c3640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:23.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e662c3640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:23.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e662c3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:23.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e662c3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:23.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e662c3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:23.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e652c1640 1 --2- 172.21.15.37:0/3756295541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e601371f0 0x7f7e60135de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:23.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e652c1640 10 monclient(hunting): get_auth_request con 0x7f7e601371f0 auth_method 0 2024-04-26T00:25:23.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e652c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:23.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e652c1640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:23.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e652c1640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:23.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e662c3640 10 monclient(hunting): handle_auth_done global_id 54243 payload 824 2024-04-26T00:25:23.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e662c3640 1 -- 172.21.15.37:0/3756295541 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7e60136e10 msgr2=0x7f7e601cec40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:25:23.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e662c3640 1 --2- 172.21.15.37:0/3756295541 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7e60136e10 0x7f7e601cec40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:23.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e662c3640 1 -- 172.21.15.37:0/3756295541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e601371f0 msgr2=0x7f7e60135de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:23.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e662c3640 1 --2- 172.21.15.37:0/3756295541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e601371f0 0x7f7e60135de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:23.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e662c3640 10 monclient: _finish_hunting 0 2024-04-26T00:25:23.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e662c3640 1 monclient: found mon.noname-a 2024-04-26T00:25:23.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e662c3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:23.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e662c3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:25:23.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.437+0000 7f7e662c3640 1 -- 172.21.15.37:0/3756295541 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e60120490 con 0x7f7e6013c340 2024-04-26T00:25:23.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.441+0000 7f7e662c3640 1 -- 172.21.15.37:0/3756295541 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e6013c340 msgr2=0x7f7e60138bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:23.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.441+0000 7f7e662c3640 1 --2- 172.21.15.37:0/3756295541 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e6013c340 0x7f7e60138bb0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7e5000ce70 tx=0x7f7e5000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:23.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.441+0000 7f7e5e7fc640 1 -- 172.21.15.37:0/3756295541 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e50003ba0 con 0x7f7e6013c340 2024-04-26T00:25:23.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.441+0000 7f7e5e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:23.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.441+0000 7f7e5e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:25:23.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.441+0000 7f7e5e7fc640 10 monclient: dump: 2024-04-26T00:25:23.439 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:23.439 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:23.440 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:23.440 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:23.440 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:23.440 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:23.440 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:23.440 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:23.440 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:23.440 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:23.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.441+0000 7f7e5e7fc640 10 monclient: _finish_auth 0 2024-04-26T00:25:23.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.441+0000 7f7e5e7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:25:23.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.441+0000 7f7e5e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:23.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.441+0000 7f7e5e7fc640 1 -- 172.21.15.37:0/3756295541 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e50003d60 con 0x7f7e6013c340 2024-04-26T00:25:23.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.441+0000 7f7e5e7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:23.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.441+0000 7f7e5e7fc640 1 -- 172.21.15.37:0/3756295541 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e5000da90 con 0x7f7e6013c340 2024-04-26T00:25:23.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.441+0000 7f7e67d26640 5 monclient: authenticate success, global_id 54243 2024-04-26T00:25:23.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.441+0000 7f7e67d26640 10 monclient: _renew_subs 2024-04-26T00:25:23.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.441+0000 7f7e67d26640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:23.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.441+0000 7f7e67d26640 1 -- 172.21.15.37:0/3756295541 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e6013a7b0 con 0x7f7e6013c340 2024-04-26T00:25:23.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.441+0000 7f7e5ffff640 4 set_mon_vals no callback set 2024-04-26T00:25:23.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.441+0000 7f7e5e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:23.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.441+0000 7f7e5e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:23.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.441+0000 7f7e5e7fc640 10 monclient: dump: 2024-04-26T00:25:23.442 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:23.442 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:23.442 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:23.442 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:23.442 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:23.442 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:23.442 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:23.443 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:23.443 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:23.443 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:23.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.441+0000 7f7e67d26640 10 monclient: _renew_subs 2024-04-26T00:25:23.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.441+0000 7f7e67d26640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:23.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.441+0000 7f7e67d26640 1 -- 172.21.15.37:0/3756295541 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e6013bd40 con 0x7f7e6013c340 2024-04-26T00:25:23.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.441+0000 7f7e67d26640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:23.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.441+0000 7f7e67d26640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:23.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.441+0000 7f7e67d26640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:23.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.441+0000 7f7e67d26640 1 -- 172.21.15.37:0/3756295541 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e60066f40 con 0x7f7e6013c340 2024-04-26T00:25:23.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.441+0000 7f7e5e7fc640 1 -- 172.21.15.37:0/3756295541 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f7e5001c020 con 0x7f7e6013c340 2024-04-26T00:25:23.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.441+0000 7f7e5e7fc640 1 --2- 172.21.15.37:0/3756295541 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7e4c035eb0 0x7f7e4c038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:23.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.441+0000 7f7e5e7fc640 1 -- 172.21.15.37:0/3756295541 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(208..208 src has 172..208) ==== 3963+0+0 (secure 0 0 0) 0x7f7e500496c0 con 0x7f7e6013c340 2024-04-26T00:25:23.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.441+0000 7f7e65ac2640 1 --2- 172.21.15.37:0/3756295541 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7e4c035eb0 0x7f7e4c038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:23.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.441+0000 7f7e65ac2640 10 monclient: get_auth_request con 0x7f7e4c035eb0 auth_method 0 2024-04-26T00:25:23.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.445+0000 7f7e65ac2640 1 --2- 172.21.15.37:0/3756295541 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7e4c035eb0 0x7f7e4c038370 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7e60068e60 tx=0x7f7e58008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:23.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.445+0000 7f7e5e7fc640 1 -- 172.21.15.37:0/3756295541 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7e500483f0 con 0x7f7e6013c340 2024-04-26T00:25:23.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.445+0000 7f7e5e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:23.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.445+0000 7f7e5e7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:25:23.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.669+0000 7f7e67d26640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:25:23.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.669+0000 7f7e67d26640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:25:23.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.669+0000 7f7e67d26640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:23.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.669+0000 7f7e67d26640 1 -- 172.21.15.37:0/3756295541 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7e6013b310 con 0x7f7e6013c340 2024-04-26T00:25:23.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.669+0000 7f7e5e7fc640 1 -- 172.21.15.37:0/3756295541 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f7e50015440 con 0x7f7e6013c340 2024-04-26T00:25:23.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.669+0000 7f7e5e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:25:23.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.669+0000 7f7e5e7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:25:23.653 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":40,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:25:23.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.673+0000 7f7e67d26640 1 -- 172.21.15.37:0/3756295541 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7e4c035eb0 msgr2=0x7f7e4c038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:23.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.673+0000 7f7e67d26640 1 --2- 172.21.15.37:0/3756295541 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7e4c035eb0 0x7f7e4c038370 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7e60068e60 tx=0x7f7e58008040 comp rx=0 tx=0).stop 2024-04-26T00:25:23.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.673+0000 7f7e67d26640 10 monclient: shutdown 2024-04-26T00:25:23.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.673+0000 7f7e67d26640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:23.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.673+0000 7f7e67d26640 1 -- 172.21.15.37:0/3756295541 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e6013c340 msgr2=0x7f7e60138bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:23.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.673+0000 7f7e67d26640 1 --2- 172.21.15.37:0/3756295541 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e6013c340 0x7f7e60138bb0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7e5000ce70 tx=0x7f7e5000a710 comp rx=0 tx=0).stop 2024-04-26T00:25:23.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.677+0000 7f7e67d26640 1 -- 172.21.15.37:0/3756295541 shutdown_connections 2024-04-26T00:25:23.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.677+0000 7f7e67d26640 1 --2- 172.21.15.37:0/3756295541 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7e4c035eb0 0x7f7e4c038370 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:23.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.677+0000 7f7e67d26640 1 --2- 172.21.15.37:0/3756295541 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e6013c340 0x7f7e60138bb0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:23.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.677+0000 7f7e67d26640 1 --2- 172.21.15.37:0/3756295541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e601371f0 0x7f7e60135de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:23.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.677+0000 7f7e67d26640 1 --2- 172.21.15.37:0/3756295541 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7e60136e10 0x7f7e601cec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:23.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.677+0000 7f7e67d26640 1 -- 172.21.15.37:0/3756295541 >> 172.21.15.37:0/3756295541 conn(0x7f7e600a64e0 msgr2=0x7f7e6013d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:23.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.677+0000 7f7e67d26640 1 -- 172.21.15.37:0/3756295541 shutdown_connections 2024-04-26T00:25:23.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:23.677+0000 7f7e67d26640 1 -- 172.21.15.37:0/3756295541 wait complete. 2024-04-26T00:25:26.673 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:25:26.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:26.825+0000 7f9bf5bb2640 10 monclient: get_monmap_and_config 2024-04-26T00:25:26.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:26.825+0000 7f9bf5bb2640 10 monclient: build_initial_monmap 2024-04-26T00:25:26.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:26.825+0000 7f9bf5bb2640 10 monclient: monmap: 2024-04-26T00:25:26.807 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:26.807 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:26.807 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:25:26.829104+0000 2024-04-26T00:25:26.807 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:25:26.829104+0000 2024-04-26T00:25:26.807 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:26.807 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:26.807 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:25:26.807 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:25:26.807 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:25:26.807 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:25:26.807 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:25:26.807 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:26.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:26.825+0000 7f9bf5bb2640 1 Processor -- start 2024-04-26T00:25:26.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:26.825+0000 7f9bf5bb2640 1 -- start start 2024-04-26T00:25:26.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:26.825+0000 7f9bf5bb2640 10 monclient: init 2024-04-26T00:25:26.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:26.825+0000 7f9bf5bb2640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:26.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:26.825+0000 7f9bf5bb2640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-04-26T00:25:26.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:26.825+0000 7f9bf5bb2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9bf00a0260 addr v1:172.21.15.37:6789/0 2024-04-26T00:25:26.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:26.825+0000 7f9bf5bb2640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9bf0137e40 0x7f9bf013a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:26.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:26.825+0000 7f9bf5bb2640 10 monclient(hunting): picked mon.noname-c con 0x7f9bf0137e40 addr v2:172.21.15.45:3300/0 2024-04-26T00:25:26.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:26.825+0000 7f9bf5bb2640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9bf013a770 0x7f9bf013cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:26.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:26.825+0000 7f9bf5bb2640 10 monclient(hunting): picked mon.noname-b con 0x7f9bf013a770 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:26.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:26.825+0000 7f9bf5bb2640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9bf00a1130 con 0x7f9bf00a0260 2024-04-26T00:25:26.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:26.825+0000 7f9bf5bb2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:26.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:26.825+0000 7f9bf5bb2640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f9bf00a0d50 con 0x7f9bf0137e40 2024-04-26T00:25:26.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:26.825+0000 7f9bf5bb2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:26.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:26.825+0000 7f9bf5bb2640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9bf009ec10 con 0x7f9bf013a770 2024-04-26T00:25:26.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:26.825+0000 7f9bf5bb2640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:26.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:26.825+0000 7f9bf5bb2640 10 monclient(hunting): authenticate will time out at 1308.134860s 2024-04-26T00:25:26.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:26.825+0000 7f9beffff640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9bf013a770 msgr2=0x7f9bf013cb60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:25:26.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:26.825+0000 7f9beffff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9bf013a770 0x7f9bf013cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:25:26.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:26.825+0000 7f9bef7fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f9bf00a0260 0x7f9bf00a0640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:40238/0 (socket says 172.21.15.37:40238) 2024-04-26T00:25:26.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:26.825+0000 7f9bef7fe640 1 -- 172.21.15.37:0/1008815188 learned_addr learned my addr 172.21.15.37:0/1008815188 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:25:26.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:26.825+0000 7f9beeffd640 1 --2- 172.21.15.37:0/1008815188 >> v2:172.21.15.45:3300/0 conn(0x7f9bf0137e40 0x7f9bf013a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:26.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:26.825+0000 7f9beeffd640 10 monclient(hunting): get_auth_request con 0x7f9bf0137e40 auth_method 0 2024-04-26T00:25:26.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:26.825+0000 7f9beeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:26.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:26.825+0000 7f9beeffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:26.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:26.825+0000 7f9beeffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:26.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:26.825+0000 7f9beeffd640 1 -- 172.21.15.37:0/1008815188 >> v2:172.21.15.45:3300/0 conn(0x7f9bf0137e40 msgr2=0x7f9bf013a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:25:26.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:26.825+0000 7f9beeffd640 1 -- 172.21.15.37:0/1008815188 >> v2:172.21.15.45:3300/0 conn(0x7f9bf0137e40 msgr2=0x7f9bf013a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:25:26.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:26.825+0000 7f9beeffd640 1 --2- 172.21.15.37:0/1008815188 >> v2:172.21.15.45:3300/0 conn(0x7f9bf0137e40 0x7f9bf013a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:25:26.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:26.825+0000 7f9beeffd640 1 --2- 172.21.15.37:0/1008815188 >> v2:172.21.15.45:3300/0 conn(0x7f9bf0137e40 0x7f9bf013a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:25:26.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:26.829+0000 7f9bef7fe640 1 -- 172.21.15.37:0/1008815188 >> v1:172.21.15.37:6789/0 conn(0x7f9bf00a0260 legacy=0x7f9bf00a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:27.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:27.029+0000 7f9beffff640 1 -- 172.21.15.37:0/1008815188 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9bf013a770 msgr2=0x7f9bf013cb60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:25:27.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:27.029+0000 7f9beffff640 1 --2- 172.21.15.37:0/1008815188 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9bf013a770 0x7f9bf013cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:25:27.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:27.029+0000 7f9beeffd640 1 --2- 172.21.15.37:0/1008815188 >> v2:172.21.15.45:3300/0 conn(0x7f9bf0137e40 0x7f9bf013a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:27.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:27.029+0000 7f9beeffd640 10 monclient(hunting): get_auth_request con 0x7f9bf0137e40 auth_method 0 2024-04-26T00:25:27.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:27.029+0000 7f9beeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:27.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:27.029+0000 7f9beeffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:27.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:27.029+0000 7f9beeffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:25:27.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:27.029+0000 7f9beeffd640 1 -- 172.21.15.37:0/1008815188 >> v2:172.21.15.45:3300/0 conn(0x7f9bf0137e40 msgr2=0x7f9bf013a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:25:27.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:27.029+0000 7f9beeffd640 1 -- 172.21.15.37:0/1008815188 >> v2:172.21.15.45:3300/0 conn(0x7f9bf0137e40 msgr2=0x7f9bf013a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:25:27.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:27.029+0000 7f9beeffd640 1 --2- 172.21.15.37:0/1008815188 >> v2:172.21.15.45:3300/0 conn(0x7f9bf0137e40 0x7f9bf013a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:25:27.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:27.029+0000 7f9beeffd640 1 --2- 172.21.15.37:0/1008815188 >> v2:172.21.15.45:3300/0 conn(0x7f9bf0137e40 0x7f9bf013a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:25:27.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:27.433+0000 7f9beffff640 1 -- 172.21.15.37:0/1008815188 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9bf013a770 msgr2=0x7f9bf013cb60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:25:27.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:27.433+0000 7f9beffff640 1 --2- 172.21.15.37:0/1008815188 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9bf013a770 0x7f9bf013cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:25:27.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:27.433+0000 7f9beeffd640 1 --2- 172.21.15.37:0/1008815188 >> v2:172.21.15.45:3300/0 conn(0x7f9bf0137e40 0x7f9bf013a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:27.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:27.433+0000 7f9beeffd640 10 monclient(hunting): get_auth_request con 0x7f9bf0137e40 auth_method 0 2024-04-26T00:25:27.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:27.433+0000 7f9beeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:27.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:27.433+0000 7f9beeffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:27.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:27.433+0000 7f9beeffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:25:27.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:27.433+0000 7f9beeffd640 1 -- 172.21.15.37:0/1008815188 >> v2:172.21.15.45:3300/0 conn(0x7f9bf0137e40 msgr2=0x7f9bf013a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:25:27.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:27.433+0000 7f9beeffd640 1 -- 172.21.15.37:0/1008815188 >> v2:172.21.15.45:3300/0 conn(0x7f9bf0137e40 msgr2=0x7f9bf013a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:25:27.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:27.433+0000 7f9beeffd640 1 --2- 172.21.15.37:0/1008815188 >> v2:172.21.15.45:3300/0 conn(0x7f9bf0137e40 0x7f9bf013a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:25:27.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:27.433+0000 7f9beeffd640 1 --2- 172.21.15.37:0/1008815188 >> v2:172.21.15.45:3300/0 conn(0x7f9bf0137e40 0x7f9bf013a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:25:28.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:28.237+0000 7f9beffff640 1 -- 172.21.15.37:0/1008815188 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9bf013a770 msgr2=0x7f9bf013cb60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:25:28.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:28.237+0000 7f9beffff640 1 --2- 172.21.15.37:0/1008815188 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9bf013a770 0x7f9bf013cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:25:28.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:28.237+0000 7f9beeffd640 1 --2- 172.21.15.37:0/1008815188 >> v2:172.21.15.45:3300/0 conn(0x7f9bf0137e40 0x7f9bf013a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:28.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:28.237+0000 7f9beeffd640 10 monclient(hunting): get_auth_request con 0x7f9bf0137e40 auth_method 0 2024-04-26T00:25:28.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:28.237+0000 7f9beeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:28.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:28.237+0000 7f9beeffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:28.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:28.237+0000 7f9beeffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:25:28.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:28.237+0000 7f9beeffd640 1 -- 172.21.15.37:0/1008815188 >> v2:172.21.15.45:3300/0 conn(0x7f9bf0137e40 msgr2=0x7f9bf013a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:25:28.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:28.237+0000 7f9beeffd640 1 -- 172.21.15.37:0/1008815188 >> v2:172.21.15.45:3300/0 conn(0x7f9bf0137e40 msgr2=0x7f9bf013a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:25:28.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:28.237+0000 7f9beeffd640 1 --2- 172.21.15.37:0/1008815188 >> v2:172.21.15.45:3300/0 conn(0x7f9bf0137e40 0x7f9bf013a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:25:28.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:28.237+0000 7f9beeffd640 1 --2- 172.21.15.37:0/1008815188 >> v2:172.21.15.45:3300/0 conn(0x7f9bf0137e40 0x7f9bf013a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:25:29.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:29.825+0000 7f9bed7fa640 10 monclient(hunting): tick 2024-04-26T00:25:29.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:29.825+0000 7f9bed7fa640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:25:29.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:29.825+0000 7f9bed7fa640 1 monclient(hunting): continuing hunt 2024-04-26T00:25:29.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:29.825+0000 7f9bed7fa640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:25:29.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:29.825+0000 7f9bed7fa640 1 -- 172.21.15.37:0/1008815188 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9bf013a770 msgr2=0x7f9bf013cb60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:25:29.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:29.825+0000 7f9bed7fa640 1 --2- 172.21.15.37:0/1008815188 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9bf013a770 0x7f9bf013cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:29.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:29.825+0000 7f9bed7fa640 1 -- 172.21.15.37:0/1008815188 >> v2:172.21.15.45:3300/0 conn(0x7f9bf0137e40 msgr2=0x7f9bf013a230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:25:29.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:29.825+0000 7f9bed7fa640 1 --2- 172.21.15.37:0/1008815188 >> v2:172.21.15.45:3300/0 conn(0x7f9bf0137e40 0x7f9bf013a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:29.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:29.825+0000 7f9bed7fa640 1 -- 172.21.15.37:0/1008815188 >> v1:172.21.15.37:6789/0 conn(0x7f9bf00a0260 legacy=0x7f9bf00a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:29.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:29.825+0000 7f9bed7fa640 10 monclient: _add_conns ranks=[3,1] 2024-04-26T00:25:29.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:29.825+0000 7f9bed7fa640 1 --2- 172.21.15.37:0/1008815188 >> v2:172.21.15.37:3300/0 conn(0x7f9bd0003770 0x7f9bd0005bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:29.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:29.825+0000 7f9bed7fa640 10 monclient(hunting): picked mon.noname-a con 0x7f9bd0003770 addr v2:172.21.15.37:3300/0 2024-04-26T00:25:29.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:29.825+0000 7f9bed7fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9bd00062c0 addr v1:172.21.15.45:6789/0 2024-04-26T00:25:29.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:29.825+0000 7f9bed7fa640 1 -- 172.21.15.37:0/1008815188 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9bf006cff0 con 0x7f9bd00062c0 2024-04-26T00:25:29.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:29.825+0000 7f9bed7fa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:29.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:29.825+0000 7f9bed7fa640 1 -- 172.21.15.37:0/1008815188 --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f9bf00a0d50 con 0x7f9bd0003770 2024-04-26T00:25:29.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:29.825+0000 7f9bed7fa640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:29.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:29.825+0000 7f9bef7fe640 1 --2- 172.21.15.37:0/1008815188 >> v2:172.21.15.37:3300/0 conn(0x7f9bd0003770 0x7f9bd0005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:29.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:29.825+0000 7f9bef7fe640 10 monclient(hunting): get_auth_request con 0x7f9bd0003770 auth_method 0 2024-04-26T00:25:29.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:29.825+0000 7f9bef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:29.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:29.825+0000 7f9bef7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:29.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:29.825+0000 7f9bef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:29.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:29.825+0000 7f9bef7fe640 1 -- 172.21.15.37:0/1008815188 >> v2:172.21.15.37:3300/0 conn(0x7f9bd0003770 msgr2=0x7f9bd0005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:25:29.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:29.825+0000 7f9bef7fe640 1 -- 172.21.15.37:0/1008815188 >> v2:172.21.15.37:3300/0 conn(0x7f9bd0003770 msgr2=0x7f9bd0005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:25:29.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:29.825+0000 7f9bef7fe640 1 --2- 172.21.15.37:0/1008815188 >> v2:172.21.15.37:3300/0 conn(0x7f9bd0003770 0x7f9bd0005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:25:29.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:29.829+0000 7f9bef7fe640 1 --2- 172.21.15.37:0/1008815188 >> v2:172.21.15.37:3300/0 conn(0x7f9bd0003770 0x7f9bd0005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:25:29.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:29.829+0000 7f9beeffd640 1 -- 172.21.15.37:0/1008815188 >> v1:172.21.15.45:6789/0 conn(0x7f9bd00062c0 legacy=0x7f9bd0008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:30.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.033+0000 7f9bef7fe640 1 --2- 172.21.15.37:0/1008815188 >> v2:172.21.15.37:3300/0 conn(0x7f9bd0003770 0x7f9bd0005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:30.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.033+0000 7f9bef7fe640 10 monclient(hunting): get_auth_request con 0x7f9bd0003770 auth_method 0 2024-04-26T00:25:30.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.033+0000 7f9bef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:30.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.033+0000 7f9bef7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:30.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.033+0000 7f9bef7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:25:30.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.033+0000 7f9bef7fe640 1 -- 172.21.15.37:0/1008815188 >> v2:172.21.15.37:3300/0 conn(0x7f9bd0003770 msgr2=0x7f9bd0005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:25:30.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.033+0000 7f9bef7fe640 1 -- 172.21.15.37:0/1008815188 >> v2:172.21.15.37:3300/0 conn(0x7f9bd0003770 msgr2=0x7f9bd0005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:25:30.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.033+0000 7f9bef7fe640 1 --2- 172.21.15.37:0/1008815188 >> v2:172.21.15.37:3300/0 conn(0x7f9bd0003770 0x7f9bd0005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:25:30.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.033+0000 7f9bef7fe640 1 --2- 172.21.15.37:0/1008815188 >> v2:172.21.15.37:3300/0 conn(0x7f9bd0003770 0x7f9bd0005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:25:30.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.437+0000 7f9bef7fe640 1 --2- 172.21.15.37:0/1008815188 >> v2:172.21.15.37:3300/0 conn(0x7f9bd0003770 0x7f9bd0005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:30.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.437+0000 7f9bef7fe640 10 monclient(hunting): get_auth_request con 0x7f9bd0003770 auth_method 0 2024-04-26T00:25:30.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.437+0000 7f9bef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:30.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.437+0000 7f9bef7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:30.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.437+0000 7f9bef7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:25:30.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.437+0000 7f9bef7fe640 1 -- 172.21.15.37:0/1008815188 >> v2:172.21.15.37:3300/0 conn(0x7f9bd0003770 msgr2=0x7f9bd0005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:25:30.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.437+0000 7f9bef7fe640 1 -- 172.21.15.37:0/1008815188 >> v2:172.21.15.37:3300/0 conn(0x7f9bd0003770 msgr2=0x7f9bd0005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:25:30.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.437+0000 7f9bef7fe640 1 --2- 172.21.15.37:0/1008815188 >> v2:172.21.15.37:3300/0 conn(0x7f9bd0003770 0x7f9bd0005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:25:30.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.437+0000 7f9bef7fe640 1 --2- 172.21.15.37:0/1008815188 >> v2:172.21.15.37:3300/0 conn(0x7f9bd0003770 0x7f9bd0005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:25:30.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bee7fc640 1 -- 172.21.15.37:0/1008815188 <== mon.1 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1603087124 0 0) 0x7f9bf006cff0 con 0x7f9bd00062c0 2024-04-26T00:25:30.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bee7fc640 10 monclient(hunting): _negotiate 2024-04-26T00:25:30.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bee7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:30.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:30.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bee7fc640 10 monclient(hunting): my global_id is 24484 2024-04-26T00:25:30.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bee7fc640 1 -- 172.21.15.37:0/1008815188 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9bdc003530 con 0x7f9bd00062c0 2024-04-26T00:25:30.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bee7fc640 1 -- 172.21.15.37:0/1008815188 <== mon.1 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2727360519 0 0) 0x7f9bdc003530 con 0x7f9bd00062c0 2024-04-26T00:25:30.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bee7fc640 1 -- 172.21.15.37:0/1008815188 >> v2:172.21.15.37:3300/0 conn(0x7f9bd0003770 msgr2=0x7f9bd0005bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:25:30.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bee7fc640 1 --2- 172.21.15.37:0/1008815188 >> v2:172.21.15.37:3300/0 conn(0x7f9bd0003770 0x7f9bd0005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:30.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bee7fc640 10 monclient: _finish_hunting 0 2024-04-26T00:25:30.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bee7fc640 1 monclient: found mon.noname-c-legacy 2024-04-26T00:25:30.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:30.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bee7fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.45:6789/0 2024-04-26T00:25:30.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bee7fc640 1 -- 172.21.15.37:0/1008815188 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bd000b070 con 0x7f9bd00062c0 2024-04-26T00:25:30.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bee7fc640 10 monclient: _finish_auth 0 2024-04-26T00:25:30.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bee7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:25:30.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:30.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bf5bb2640 5 monclient: authenticate success, global_id 24484 2024-04-26T00:25:30.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bee7fc640 1 -- 172.21.15.37:0/1008815188 <== mon.1 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 866326386 0 0) 0x7f9bd800f040 con 0x7f9bd00062c0 2024-04-26T00:25:30.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bf5bb2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:25:30.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:30.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bee7fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-04-26T00:25:30.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bee7fc640 10 monclient: dump: 2024-04-26T00:25:30.970 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:30.970 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:30.970 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:30.970 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:30.971 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:30.971 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:30.971 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:30.971 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:30.971 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:30.971 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:30.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bee7fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] but i'm connected to v1:172.21.15.45:6789/0, reconnecting 2024-04-26T00:25:30.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bee7fc640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:30.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bee7fc640 1 -- 172.21.15.37:0/1008815188 >> v1:172.21.15.45:6789/0 conn(0x7f9bd00062c0 legacy=0x7f9bd0008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:30.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9beffff640 1 -- 172.21.15.37:0/1008815188 reap_dead start 2024-04-26T00:25:30.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bee7fc640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:25:30.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bee7fc640 1 --2- 172.21.15.37:0/1008815188 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9bdc0039c0 0x7f9bdc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:30.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bee7fc640 10 monclient(hunting): picked mon.c con 0x7f9bdc0039c0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:30.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bee7fc640 1 --2- 172.21.15.37:0/1008815188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bdc004810 0x7f9bdc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:30.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bee7fc640 10 monclient(hunting): picked mon.b con 0x7f9bdc004810 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:25:30.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bee7fc640 1 --2- 172.21.15.37:0/1008815188 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bdc005220 0x7f9bdc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:30.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bee7fc640 10 monclient(hunting): picked mon.a con 0x7f9bdc005220 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:25:30.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bee7fc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:30.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bee7fc640 1 -- 172.21.15.37:0/1008815188 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf00a0d50 con 0x7f9bdc005220 2024-04-26T00:25:30.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bee7fc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:30.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bee7fc640 1 -- 172.21.15.37:0/1008815188 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9bdc00c150 con 0x7f9bdc004810 2024-04-26T00:25:30.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bee7fc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:30.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bef7fe640 1 -- 172.21.15.37:0/1008815188 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9bdc0039c0 msgr2=0x7f9bdc003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:25:30.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bee7fc640 1 -- 172.21.15.37:0/1008815188 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9bdc00c350 con 0x7f9bdc0039c0 2024-04-26T00:25:30.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bef7fe640 1 --2- 172.21.15.37:0/1008815188 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9bdc0039c0 0x7f9bdc003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:25:30.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bee7fc640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:30.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bee7fc640 10 monclient(hunting): _finish_auth 0 2024-04-26T00:25:30.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9bf5bb2640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-26T00:25:30.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9beffff640 1 --2- 172.21.15.37:0/1008815188 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bdc005220 0x7f9bdc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:30.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9beffff640 10 monclient(hunting): get_auth_request con 0x7f9bdc005220 auth_method 0 2024-04-26T00:25:30.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9beffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:30.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9beffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:30.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9beffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:25:30.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9beeffd640 1 --2- 172.21.15.37:0/1008815188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bdc004810 0x7f9bdc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:30.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9beeffd640 10 monclient(hunting): get_auth_request con 0x7f9bdc004810 auth_method 0 2024-04-26T00:25:30.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9beeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:30.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9beeffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:30.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9beeffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:25:30.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9beffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:30.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9beffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:30.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9beffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:25:30.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9beeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:30.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9beeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:30.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9beeffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:25:30.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9beffff640 10 monclient(hunting): handle_auth_done global_id 24484 payload 293 2024-04-26T00:25:30.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9beffff640 1 -- 172.21.15.37:0/1008815188 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9bdc0039c0 msgr2=0x7f9bdc003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:25:30.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9beffff640 1 --2- 172.21.15.37:0/1008815188 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9bdc0039c0 0x7f9bdc003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:30.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9beffff640 1 -- 172.21.15.37:0/1008815188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bdc004810 msgr2=0x7f9bdc004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:30.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9beffff640 1 --2- 172.21.15.37:0/1008815188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bdc004810 0x7f9bdc004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:30.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9beffff640 10 monclient: _finish_hunting 0 2024-04-26T00:25:30.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9beffff640 1 monclient: found mon.a 2024-04-26T00:25:30.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9beffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:30.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9beffff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:30.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9beffff640 1 -- 172.21.15.37:0/1008815188 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bdc003530 con 0x7f9bdc005220 2024-04-26T00:25:30.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9beffff640 10 monclient: _finish_auth 0 2024-04-26T00:25:30.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9beffff640 10 monclient: _check_auth_tickets 2024-04-26T00:25:30.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.985+0000 7f9beffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:30.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.997+0000 7f9beffff640 1 -- 172.21.15.37:0/1008815188 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bdc005220 msgr2=0x7f9bdc00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:30.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.997+0000 7f9beffff640 1 --2- 172.21.15.37:0/1008815188 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bdc005220 0x7f9bdc00b9d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0072eb0 tx=0x7f9be001c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=573d713aa8b54c24 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:30.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.997+0000 7f9bee7fc640 1 -- 172.21.15.37:0/1008815188 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be0009930 con 0x7f9bdc005220 2024-04-26T00:25:30.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.997+0000 7f9bee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:30.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.997+0000 7f9bee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:30.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.997+0000 7f9bee7fc640 10 monclient: dump: 2024-04-26T00:25:30.979 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:30.979 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:30.979 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:30.979 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:30.979 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:30.979 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:30.979 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:30.979 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:30.979 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:30.979 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:30.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.997+0000 7f9bf5bb2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:25:30.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.997+0000 7f9bee7fc640 1 -- 172.21.15.37:0/1008815188 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9be00ad070 con 0x7f9bdc005220 2024-04-26T00:25:30.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.997+0000 7f9bee7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:30.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.997+0000 7f9bf5bb2640 10 monclient: get_monmap_and_config success 2024-04-26T00:25:30.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.997+0000 7f9bf5bb2640 4 set_mon_vals no callback set 2024-04-26T00:25:30.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.997+0000 7f9bee7fc640 1 -- 172.21.15.37:0/1008815188 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be00a6720 con 0x7f9bdc005220 2024-04-26T00:25:30.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.997+0000 7f9bee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:30.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.997+0000 7f9bee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:30.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.997+0000 7f9bee7fc640 10 monclient: dump: 2024-04-26T00:25:30.980 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:30.980 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:30.980 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:30.980 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:30.980 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:30.980 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:30.980 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:30.980 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:30.980 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:30.980 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:30.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.997+0000 7f9bf5bb2640 10 monclient: shutdown 2024-04-26T00:25:30.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.997+0000 7f9bf5bb2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:30.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.997+0000 7f9bf5bb2640 1 -- 172.21.15.37:0/1008815188 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bdc005220 msgr2=0x7f9bdc00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:30.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.997+0000 7f9bf5bb2640 1 --2- 172.21.15.37:0/1008815188 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bdc005220 0x7f9bdc00b9d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0072eb0 tx=0x7f9be001c790 comp rx=0 tx=0).stop 2024-04-26T00:25:30.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.997+0000 7f9bf5bb2640 1 -- 172.21.15.37:0/1008815188 shutdown_connections 2024-04-26T00:25:30.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.997+0000 7f9bf5bb2640 1 --2- 172.21.15.37:0/1008815188 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bdc005220 0x7f9bdc00b9d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:30.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.997+0000 7f9bf5bb2640 1 --2- 172.21.15.37:0/1008815188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bdc004810 0x7f9bdc004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:30.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.997+0000 7f9bf5bb2640 1 --2- 172.21.15.37:0/1008815188 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9bdc0039c0 0x7f9bdc003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:30.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.997+0000 7f9bf5bb2640 1 -- 172.21.15.37:0/1008815188 >> 172.21.15.37:0/1008815188 conn(0x7f9bf00a1ad0 msgr2=0x7f9bf00a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:30.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.997+0000 7f9bf5bb2640 1 -- 172.21.15.37:0/1008815188 shutdown_connections 2024-04-26T00:25:30.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.997+0000 7f9bf5bb2640 1 -- 172.21.15.37:0/1008815188 wait complete. 2024-04-26T00:25:30.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.997+0000 7f9bf5bb2640 10 monclient: build_initial_monmap 2024-04-26T00:25:30.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:30.997+0000 7f9bf5bb2640 10 monclient: monmap: 2024-04-26T00:25:30.983 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:30.983 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:25:30.983 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:25:30.983 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:25:30.983 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:30.983 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:25:30.983 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:25:30.983 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:25:30.983 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:25:30.983 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:30.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9bf5bb2640 1 Processor -- start 2024-04-26T00:25:30.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9bf5bb2640 1 -- start start 2024-04-26T00:25:30.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9bf5bb2640 10 monclient: init 2024-04-26T00:25:30.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9bf5bb2640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:30.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9bf5bb2640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:25:30.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9bf5bb2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bdc0039c0 0x7f9bf01346e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:30.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9bf5bb2640 10 monclient(hunting): picked mon.noname-b con 0x7f9bdc0039c0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:25:30.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9bf5bb2640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bdc004810 0x7f9bf011d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:30.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9bf5bb2640 10 monclient(hunting): picked mon.noname-a con 0x7f9bdc004810 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:25:30.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9bf5bb2640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9bdc005220 0x7f9bf011d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:30.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9bf5bb2640 10 monclient(hunting): picked mon.noname-c con 0x7f9bdc005220 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:30.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9bf5bb2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:30.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9bf5bb2640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf009f2b0 con 0x7f9bdc004810 2024-04-26T00:25:30.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9bf5bb2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:30.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9bf5bb2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf009f0f0 con 0x7f9bdc0039c0 2024-04-26T00:25:30.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9bf5bb2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:30.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9bf5bb2640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9bf009f430 con 0x7f9bdc005220 2024-04-26T00:25:30.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9bf5bb2640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:30.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9bf5bb2640 10 monclient(hunting): authenticate will time out at 1312.309526s 2024-04-26T00:25:30.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9beffff640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9bdc005220 msgr2=0x7f9bf011d640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:25:30.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9beffff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9bdc005220 0x7f9bf011d640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:25:30.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9beeffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bdc004810 0x7f9bf011d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:30.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9beeffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bdc004810 0x7f9bf011d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40938/0 (socket says 172.21.15.37:40938) 2024-04-26T00:25:30.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9beeffd640 1 -- 172.21.15.37:0/3501152305 learned_addr learned my addr 172.21.15.37:0/3501152305 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:30.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9beeffd640 10 monclient(hunting): get_auth_request con 0x7f9bdc004810 auth_method 0 2024-04-26T00:25:30.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9beeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:30.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9beeffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:30.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9beeffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:30.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9beeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:30.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9beeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:30.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9bef7fe640 1 --2- 172.21.15.37:0/3501152305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bdc0039c0 0x7f9bf01346e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:30.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9beeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:30.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9bef7fe640 10 monclient(hunting): get_auth_request con 0x7f9bdc0039c0 auth_method 0 2024-04-26T00:25:30.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9bef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:30.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9bef7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:30.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9bef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:30.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9beeffd640 10 monclient(hunting): handle_auth_done global_id 54246 payload 824 2024-04-26T00:25:30.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9beeffd640 1 -- 172.21.15.37:0/3501152305 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9bdc005220 msgr2=0x7f9bf011d640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:25:30.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9beeffd640 1 --2- 172.21.15.37:0/3501152305 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9bdc005220 0x7f9bf011d640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:30.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9beeffd640 1 -- 172.21.15.37:0/3501152305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bdc0039c0 msgr2=0x7f9bf01346e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:30.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9beeffd640 1 --2- 172.21.15.37:0/3501152305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bdc0039c0 0x7f9bf01346e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:30.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9beeffd640 10 monclient: _finish_hunting 0 2024-04-26T00:25:30.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9beeffd640 1 monclient: found mon.noname-a 2024-04-26T00:25:30.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9beeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:30.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9beeffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:25:30.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9beeffd640 1 -- 172.21.15.37:0/3501152305 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bf011eda0 con 0x7f9bdc004810 2024-04-26T00:25:30.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.001+0000 7f9bef7fe640 1 --2- 172.21.15.37:0/3501152305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bdc0039c0 0x7f9bf01346e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:25:30.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.005+0000 7f9beeffd640 1 -- 172.21.15.37:0/3501152305 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bdc004810 msgr2=0x7f9bf011d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:30.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.005+0000 7f9beeffd640 1 --2- 172.21.15.37:0/3501152305 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bdc004810 0x7f9bf011d100 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8018a10 tx=0x7f9bd8018ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:30.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.005+0000 7f9be7fff640 1 -- 172.21.15.37:0/3501152305 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd8004530 con 0x7f9bdc004810 2024-04-26T00:25:30.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.005+0000 7f9be7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:30.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.005+0000 7f9be7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:25:30.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.005+0000 7f9be7fff640 10 monclient: dump: 2024-04-26T00:25:30.992 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:30.992 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:30.992 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:30.992 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:30.992 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:30.993 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:30.993 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:30.993 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:30.993 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:30.993 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:30.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.005+0000 7f9be7fff640 10 monclient: _finish_auth 0 2024-04-26T00:25:30.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.005+0000 7f9be7fff640 10 monclient: _check_auth_tickets 2024-04-26T00:25:30.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.005+0000 7f9be7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:30.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.005+0000 7f9bf5bb2640 5 monclient: authenticate success, global_id 54246 2024-04-26T00:25:30.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.005+0000 7f9bf5bb2640 10 monclient: _renew_subs 2024-04-26T00:25:30.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.005+0000 7f9bf5bb2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:30.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.005+0000 7f9bf5bb2640 1 -- 172.21.15.37:0/3501152305 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bf00729d0 con 0x7f9bdc004810 2024-04-26T00:25:30.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.005+0000 7f9bf5bb2640 10 monclient: _renew_subs 2024-04-26T00:25:30.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.005+0000 7f9bf5bb2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:30.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.005+0000 7f9bf5bb2640 1 -- 172.21.15.37:0/3501152305 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bf011f680 con 0x7f9bdc004810 2024-04-26T00:25:30.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.005+0000 7f9be7fff640 1 -- 172.21.15.37:0/3501152305 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bd8016af0 con 0x7f9bdc004810 2024-04-26T00:25:30.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.005+0000 7f9be7fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:30.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.005+0000 7f9be7fff640 1 -- 172.21.15.37:0/3501152305 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd8003620 con 0x7f9bdc004810 2024-04-26T00:25:30.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.005+0000 7f9be7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:30.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.005+0000 7f9be7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:30.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.005+0000 7f9bedffb640 4 set_mon_vals no callback set 2024-04-26T00:25:30.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.005+0000 7f9be7fff640 10 monclient: dump: 2024-04-26T00:25:30.994 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:30.994 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:30.994 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:30.994 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:30.995 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:30.995 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:30.995 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:30.995 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:30.995 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:30.995 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:30.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.005+0000 7f9bf5bb2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:30.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.005+0000 7f9bf5bb2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:30.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.005+0000 7f9bf5bb2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:30.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.005+0000 7f9bf5bb2640 1 -- 172.21.15.37:0/3501152305 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bd4005090 con 0x7f9bdc004810 2024-04-26T00:25:30.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.005+0000 7f9be7fff640 1 -- 172.21.15.37:0/3501152305 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9bd80037c0 con 0x7f9bdc004810 2024-04-26T00:25:30.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.005+0000 7f9be7fff640 1 --2- 172.21.15.37:0/3501152305 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9bd0036830 0x7f9bd0038cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:30.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.005+0000 7f9be7fff640 1 -- 172.21.15.37:0/3501152305 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(208..208 src has 177..208) ==== 3963+0+0 (secure 0 0 0) 0x7f9bd804c6d0 con 0x7f9bdc004810 2024-04-26T00:25:30.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.005+0000 7f9bef7fe640 1 --2- 172.21.15.37:0/3501152305 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9bd0036830 0x7f9bd0038cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:30.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.005+0000 7f9bef7fe640 10 monclient: get_auth_request con 0x7f9bd0036830 auth_method 0 2024-04-26T00:25:30.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.009+0000 7f9bef7fe640 1 --2- 172.21.15.37:0/3501152305 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9bd0036830 0x7f9bd0038cf0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9be8000f90 tx=0x7f9be8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:30.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.009+0000 7f9be7fff640 1 -- 172.21.15.37:0/3501152305 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9bd802a960 con 0x7f9bdc004810 2024-04-26T00:25:30.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.009+0000 7f9be7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:30.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.009+0000 7f9be7fff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:25:31.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.237+0000 7f9bf5bb2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:25:31.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.237+0000 7f9bf5bb2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:25:31.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.237+0000 7f9bf5bb2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:31.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.237+0000 7f9bf5bb2640 1 -- 172.21.15.37:0/3501152305 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9bd4005380 con 0x7f9bdc004810 2024-04-26T00:25:31.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.237+0000 7f9be7fff640 1 -- 172.21.15.37:0/3501152305 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7f9bd802a440 con 0x7f9bdc004810 2024-04-26T00:25:31.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.237+0000 7f9be7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:25:31.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.237+0000 7f9be7fff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:25:31.220 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":44,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:25:31.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.241+0000 7f9bf5bb2640 1 -- 172.21.15.37:0/3501152305 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9bd0036830 msgr2=0x7f9bd0038cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:31.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.241+0000 7f9bf5bb2640 1 --2- 172.21.15.37:0/3501152305 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9bd0036830 0x7f9bd0038cf0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9be8000f90 tx=0x7f9be8009040 comp rx=0 tx=0).stop 2024-04-26T00:25:31.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.241+0000 7f9bf5bb2640 10 monclient: shutdown 2024-04-26T00:25:31.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.241+0000 7f9bf5bb2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:31.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.241+0000 7f9bf5bb2640 1 -- 172.21.15.37:0/3501152305 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bdc004810 msgr2=0x7f9bf011d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:31.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.241+0000 7f9bf5bb2640 1 --2- 172.21.15.37:0/3501152305 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bdc004810 0x7f9bf011d100 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8018a10 tx=0x7f9bd8018ed0 comp rx=0 tx=0).stop 2024-04-26T00:25:31.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.241+0000 7f9bf5bb2640 1 -- 172.21.15.37:0/3501152305 shutdown_connections 2024-04-26T00:25:31.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.241+0000 7f9bf5bb2640 1 --2- 172.21.15.37:0/3501152305 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9bd0036830 0x7f9bd0038cf0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:31.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.241+0000 7f9bf5bb2640 1 --2- 172.21.15.37:0/3501152305 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9bdc005220 0x7f9bf011d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:31.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.241+0000 7f9bf5bb2640 1 --2- 172.21.15.37:0/3501152305 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bdc004810 0x7f9bf011d100 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:31.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.241+0000 7f9bf5bb2640 1 --2- 172.21.15.37:0/3501152305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bdc0039c0 0x7f9bf01346e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:31.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.241+0000 7f9bf5bb2640 1 -- 172.21.15.37:0/3501152305 >> 172.21.15.37:0/3501152305 conn(0x7f9bf00a1ad0 msgr2=0x7f9bf0139b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:31.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.241+0000 7f9bf5bb2640 1 -- 172.21.15.37:0/3501152305 shutdown_connections 2024-04-26T00:25:31.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.241+0000 7f9bf5bb2640 1 -- 172.21.15.37:0/3501152305 wait complete. 2024-04-26T00:25:31.239 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-26T00:25:31.240 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-04-26T00:25:31.240 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:25:31.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.389+0000 7fc004d2b640 10 monclient: get_monmap_and_config 2024-04-26T00:25:31.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.389+0000 7fc004d2b640 10 monclient: build_initial_monmap 2024-04-26T00:25:31.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.389+0000 7fc004d2b640 10 monclient: monmap: 2024-04-26T00:25:31.370 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:31.370 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:31.370 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:25:31.392240+0000 2024-04-26T00:25:31.370 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:25:31.392240+0000 2024-04-26T00:25:31.370 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:31.370 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:31.371 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:25:31.371 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:25:31.371 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:25:31.371 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:25:31.371 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:25:31.371 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:31.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.389+0000 7fc004d2b640 1 Processor -- start 2024-04-26T00:25:31.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.389+0000 7fc004d2b640 1 -- start start 2024-04-26T00:25:31.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.389+0000 7fc004d2b640 10 monclient: init 2024-04-26T00:25:31.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.389+0000 7fc004d2b640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:31.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.389+0000 7fc004d2b640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-04-26T00:25:31.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.389+0000 7fc004d2b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc0000a3d90 0x7fc00009bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:31.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.389+0000 7fc004d2b640 10 monclient(hunting): picked mon.noname-c con 0x7fc0000a3d90 addr v2:172.21.15.45:3300/0 2024-04-26T00:25:31.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.389+0000 7fc004d2b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc0000a4170 0x7fc0000a45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:31.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.389+0000 7fc004d2b640 10 monclient(hunting): picked mon.noname-b con 0x7fc0000a4170 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:31.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.389+0000 7fc004d2b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc000133b20 addr v1:172.21.15.45:6789/0 2024-04-26T00:25:31.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.389+0000 7fc004d2b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0000a5150 con 0x7fc000133b20 2024-04-26T00:25:31.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.389+0000 7fc004d2b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:31.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.389+0000 7fc004d2b640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc00009b280 con 0x7fc0000a3d90 2024-04-26T00:25:31.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.389+0000 7fc004d2b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:31.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.389+0000 7fc004d2b640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc00009b560 con 0x7fc0000a4170 2024-04-26T00:25:31.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.389+0000 7fc004d2b640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:31.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.389+0000 7fc004d2b640 10 monclient(hunting): authenticate will time out at 1312.697901s 2024-04-26T00:25:31.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.389+0000 7fbffdd9b640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc0000a4170 msgr2=0x7fc0000a45b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:25:31.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.389+0000 7fbffdd9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc0000a4170 0x7fc0000a45b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:25:31.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.389+0000 7fbffe59c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc0000a3d90 0x7fc00009bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:31.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.389+0000 7fbffe59c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc0000a3d90 0x7fc00009bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.37:41460/0 (socket says 172.21.15.37:41460) 2024-04-26T00:25:31.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.389+0000 7fbffe59c640 1 -- 172.21.15.37:0/1547353930 learned_addr learned my addr 172.21.15.37:0/1547353930 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:31.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.389+0000 7fbffe59c640 10 monclient(hunting): get_auth_request con 0x7fc0000a3d90 auth_method 0 2024-04-26T00:25:31.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.389+0000 7fbffe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:31.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.389+0000 7fbffe59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:31.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.389+0000 7fbffe59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:31.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.389+0000 7fbffe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:31.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.389+0000 7fbffe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:31.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.389+0000 7fbffe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:31.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.393+0000 7fbffed9d640 1 -- 172.21.15.37:0/1547353930 >> v1:172.21.15.45:6789/0 conn(0x7fc000133b20 legacy=0x7fc000133f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:31.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.393+0000 7fbffe59c640 10 monclient(hunting): handle_auth_done global_id 24490 payload 274 2024-04-26T00:25:31.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.393+0000 7fbffe59c640 1 -- 172.21.15.37:0/1547353930 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc0000a4170 msgr2=0x7fc0000a45b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:25:31.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.393+0000 7fbffe59c640 1 --2- 172.21.15.37:0/1547353930 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc0000a4170 0x7fc0000a45b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:31.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.393+0000 7fbffe59c640 1 -- 172.21.15.37:0/1547353930 >> v1:172.21.15.45:6789/0 conn(0x7fc000133b20 legacy=0x7fc000133f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:31.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.393+0000 7fbffe59c640 10 monclient: _finish_hunting 0 2024-04-26T00:25:31.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.393+0000 7fbffe59c640 1 monclient: found mon.noname-c 2024-04-26T00:25:31.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.393+0000 7fbffe59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:31.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.393+0000 7fbffe59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:25:31.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.393+0000 7fbffe59c640 1 -- 172.21.15.37:0/1547353930 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0000a4af0 con 0x7fc0000a3d90 2024-04-26T00:25:31.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.397+0000 7fbffe59c640 1 -- 172.21.15.37:0/1547353930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0000a3d90 msgr2=0x7fc00009bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:31.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.397+0000 7fbffe59c640 1 --2- 172.21.15.37:0/1547353930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0000a3d90 0x7fc00009bf40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbff40098f0 tx=0x7fbff401c4c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4729319aa684e56 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:31.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.397+0000 7fbffd59a640 1 -- 172.21.15.37:0/1547353930 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff40b1070 con 0x7fc0000a3d90 2024-04-26T00:25:31.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.397+0000 7fbffd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:31.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.397+0000 7fbffd59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:25:31.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.397+0000 7fbffd59a640 10 monclient: dump: 2024-04-26T00:25:31.380 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:31.380 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:31.380 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:31.380 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:31.380 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:31.380 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:31.380 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:31.380 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:31.380 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:31.380 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:31.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.397+0000 7fbffd59a640 10 monclient: _finish_auth 0 2024-04-26T00:25:31.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.397+0000 7fbffd59a640 10 monclient: _check_auth_tickets 2024-04-26T00:25:31.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.397+0000 7fbffd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:31.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.397+0000 7fbffd59a640 1 -- 172.21.15.37:0/1547353930 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbff40ac630 con 0x7fc0000a3d90 2024-04-26T00:25:31.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.397+0000 7fbffd59a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:31.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.397+0000 7fbffd59a640 1 -- 172.21.15.37:0/1547353930 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff40a8050 con 0x7fc0000a3d90 2024-04-26T00:25:31.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.397+0000 7fc004d2b640 5 monclient: authenticate success, global_id 24490 2024-04-26T00:25:31.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.397+0000 7fc004d2b640 10 monclient: get_monmap_and_config success 2024-04-26T00:25:31.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.397+0000 7fc004d2b640 4 set_mon_vals no callback set 2024-04-26T00:25:31.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.397+0000 7fbffd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:31.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.397+0000 7fbffd59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:25:31.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.397+0000 7fbffd59a640 10 monclient: dump: 2024-04-26T00:25:31.382 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:31.382 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:31.382 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:31.382 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:31.382 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:31.382 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:31.382 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:31.382 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:31.382 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:31.382 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:31.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.397+0000 7fc004d2b640 10 monclient: shutdown 2024-04-26T00:25:31.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.397+0000 7fc004d2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:31.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.397+0000 7fc004d2b640 1 -- 172.21.15.37:0/1547353930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0000a3d90 msgr2=0x7fc00009bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:31.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.397+0000 7fc004d2b640 1 --2- 172.21.15.37:0/1547353930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0000a3d90 0x7fc00009bf40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbff40098f0 tx=0x7fbff401c4c0 comp rx=0 tx=0).stop 2024-04-26T00:25:31.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.397+0000 7fc004d2b640 1 -- 172.21.15.37:0/1547353930 shutdown_connections 2024-04-26T00:25:31.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.397+0000 7fc004d2b640 1 --2- 172.21.15.37:0/1547353930 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc0000a4170 0x7fc0000a45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:31.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.397+0000 7fc004d2b640 1 --2- 172.21.15.37:0/1547353930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0000a3d90 0x7fc00009bf40 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:31.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.397+0000 7fc004d2b640 1 -- 172.21.15.37:0/1547353930 >> 172.21.15.37:0/1547353930 conn(0x7fc0000aa0a0 msgr2=0x7fc0000a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:31.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.397+0000 7fc004d2b640 1 -- 172.21.15.37:0/1547353930 shutdown_connections 2024-04-26T00:25:31.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.397+0000 7fc004d2b640 1 -- 172.21.15.37:0/1547353930 wait complete. 2024-04-26T00:25:31.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.397+0000 7fc004d2b640 10 monclient: build_initial_monmap 2024-04-26T00:25:31.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.397+0000 7fc004d2b640 10 monclient: monmap: 2024-04-26T00:25:31.389 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:31.389 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:25:31.389 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:25:31.389 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:25:31.389 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:31.389 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:25:31.389 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:25:31.389 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:25:31.389 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:25:31.390 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:31.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.397+0000 7fc004d2b640 1 Processor -- start 2024-04-26T00:25:31.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.397+0000 7fc004d2b640 1 -- start start 2024-04-26T00:25:31.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.397+0000 7fc004d2b640 10 monclient: init 2024-04-26T00:25:31.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fc004d2b640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:31.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fc004d2b640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:25:31.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fc004d2b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc0000a3d90 0x7fc0001cf320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:31.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fc004d2b640 10 monclient(hunting): picked mon.noname-c con 0x7fc0000a3d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:31.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fc004d2b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0000a4170 0x7fc00013dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:31.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fc004d2b640 10 monclient(hunting): picked mon.noname-b con 0x7fc0000a4170 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:25:31.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fc004d2b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc000133b20 0x7fc00013e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:31.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fc004d2b640 10 monclient(hunting): picked mon.noname-a con 0x7fc000133b20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:25:31.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fc004d2b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:31.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fc004d2b640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fc000141e30 con 0x7fc000133b20 2024-04-26T00:25:31.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fc004d2b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:31.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fc004d2b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc00009c690 con 0x7fc0000a4170 2024-04-26T00:25:31.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fc004d2b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:31.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fc004d2b640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc000141fb0 con 0x7fc0000a3d90 2024-04-26T00:25:31.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fc004d2b640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:31.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fc004d2b640 10 monclient(hunting): authenticate will time out at 1312.708462s 2024-04-26T00:25:31.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbffe59c640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc0000a3d90 msgr2=0x7fc0001cf320 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:25:31.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbffe59c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc0000a3d90 0x7fc0001cf320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:25:31.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbffed9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc000133b20 0x7fc00013e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:31.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbffed9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc000133b20 0x7fc00013e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40954/0 (socket says 172.21.15.37:40954) 2024-04-26T00:25:31.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbffed9d640 1 -- 172.21.15.37:0/534565343 learned_addr learned my addr 172.21.15.37:0/534565343 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:31.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbffed9d640 10 monclient(hunting): get_auth_request con 0x7fc000133b20 auth_method 0 2024-04-26T00:25:31.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbffed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:31.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbffed9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:31.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbffed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:31.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbffed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:31.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbffed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:31.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbffed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:31.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbffdd9b640 1 --2- 172.21.15.37:0/534565343 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0000a4170 0x7fc00013dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:31.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbffdd9b640 10 monclient(hunting): get_auth_request con 0x7fc0000a4170 auth_method 0 2024-04-26T00:25:31.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbffdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:31.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbffdd9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:31.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbffdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:31.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbffed9d640 10 monclient(hunting): handle_auth_done global_id 54249 payload 824 2024-04-26T00:25:31.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbffed9d640 1 -- 172.21.15.37:0/534565343 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc0000a3d90 msgr2=0x7fc0001cf320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:25:31.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbffed9d640 1 --2- 172.21.15.37:0/534565343 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc0000a3d90 0x7fc0001cf320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:31.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbffed9d640 1 -- 172.21.15.37:0/534565343 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0000a4170 msgr2=0x7fc00013dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:31.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbffed9d640 1 --2- 172.21.15.37:0/534565343 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc0000a4170 0x7fc00013dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:31.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbffed9d640 10 monclient: _finish_hunting 0 2024-04-26T00:25:31.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbffed9d640 1 monclient: found mon.noname-a 2024-04-26T00:25:31.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbffed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:31.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbffed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:25:31.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbffed9d640 1 -- 172.21.15.37:0/534565343 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc00013a500 con 0x7fc000133b20 2024-04-26T00:25:31.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbffed9d640 1 -- 172.21.15.37:0/534565343 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc000133b20 msgr2=0x7fc00013e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:31.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbffed9d640 1 --2- 172.21.15.37:0/534565343 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc000133b20 0x7fc00013e260 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbff800c7a0 tx=0x7fbff800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:31.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbfeeffd640 1 -- 172.21.15.37:0/534565343 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff8017070 con 0x7fc000133b20 2024-04-26T00:25:31.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbfeeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:31.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbfeeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:25:31.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbfeeffd640 10 monclient: dump: 2024-04-26T00:25:31.394 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:31.394 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:31.394 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:31.394 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:31.394 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:31.394 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:31.394 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:31.394 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:31.394 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:31.394 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:31.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbfeeffd640 10 monclient: _finish_auth 0 2024-04-26T00:25:31.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbfeeffd640 10 monclient: _check_auth_tickets 2024-04-26T00:25:31.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbfeeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:31.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fc004d2b640 5 monclient: authenticate success, global_id 54249 2024-04-26T00:25:31.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbfeeffd640 1 -- 172.21.15.37:0/534565343 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbff800a840 con 0x7fc000133b20 2024-04-26T00:25:31.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fc004d2b640 10 monclient: _renew_subs 2024-04-26T00:25:31.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fc004d2b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:31.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fc004d2b640 1 -- 172.21.15.37:0/534565343 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc00013bf50 con 0x7fc000133b20 2024-04-26T00:25:31.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbfeeffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:31.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbfeeffd640 1 -- 172.21.15.37:0/534565343 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff800ac30 con 0x7fc000133b20 2024-04-26T00:25:31.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbfeeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:31.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbfeeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:31.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbfeeffd640 10 monclient: dump: 2024-04-26T00:25:31.395 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:31.395 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:31.395 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:31.395 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:31.396 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:31.396 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:31.396 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:31.396 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:31.396 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:31.396 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:31.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fc004d2b640 10 monclient: _renew_subs 2024-04-26T00:25:31.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fc004d2b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:31.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fbffcd99640 4 set_mon_vals no callback set 2024-04-26T00:25:31.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.401+0000 7fc004d2b640 1 -- 172.21.15.37:0/534565343 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0001cf860 con 0x7fc000133b20 2024-04-26T00:25:31.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.405+0000 7fc004d2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:31.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.405+0000 7fc004d2b640 1 --2- 172.21.15.37:0/534565343 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc0001388a0 0x7fc000138c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:31.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.405+0000 7fc004d2b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:31.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.405+0000 7fc004d2b640 1 -- 172.21.15.37:0/534565343 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fc00013c470 con 0x7fc0001388a0 2024-04-26T00:25:31.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.405+0000 7fbfeeffd640 1 -- 172.21.15.37:0/534565343 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fbff801a070 con 0x7fc000133b20 2024-04-26T00:25:31.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.405+0000 7fbffe59c640 1 --2- 172.21.15.37:0/534565343 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc0001388a0 0x7fc000138c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:31.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.405+0000 7fbffe59c640 10 monclient: get_auth_request con 0x7fc0001388a0 auth_method 0 2024-04-26T00:25:31.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.405+0000 7fbffe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:31.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.405+0000 7fbffe59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:31.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.405+0000 7fbffe59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:31.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.405+0000 7fbffe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:31.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.405+0000 7fbffe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:31.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.405+0000 7fbffe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:31.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.405+0000 7fbfeeffd640 1 --2- 172.21.15.37:0/534565343 >> v2:172.21.15.45:6812/2379918623 conn(0x7fbfe0035e30 0x7fbfe00382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:31.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.405+0000 7fbffe59c640 10 monclient(hunting): handle_auth_done global_id 54252 payload 824 2024-04-26T00:25:31.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.405+0000 7fbffe59c640 1 -- 172.21.15.37:0/534565343 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc00004f510 con 0x7fc0001388a0 2024-04-26T00:25:31.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.405+0000 7fbfeeffd640 1 -- 172.21.15.37:0/534565343 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(208..208 src has 177..208) ==== 3963+0+0 (secure 0 0 0) 0x7fbff80481c0 con 0x7fc000133b20 2024-04-26T00:25:31.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.405+0000 7fbffdd9b640 1 --2- 172.21.15.37:0/534565343 >> v2:172.21.15.45:6812/2379918623 conn(0x7fbfe0035e30 0x7fbfe00382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:31.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.405+0000 7fbffdd9b640 10 monclient: get_auth_request con 0x7fbfe0035e30 auth_method 0 2024-04-26T00:25:31.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.405+0000 7fbffe59c640 1 -- 172.21.15.37:0/534565343 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc0001388a0 msgr2=0x7fc000138c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:31.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.405+0000 7fbffe59c640 1 --2- 172.21.15.37:0/534565343 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc0001388a0 0x7fc000138c80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbff40ae000 tx=0x7fbff4019260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:31.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.405+0000 7fbfeeffd640 1 -- 172.21.15.37:0/534565343 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff40ba020 con 0x7fc0001388a0 2024-04-26T00:25:31.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.405+0000 7fbfeeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:31.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.405+0000 7fbfeeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:31.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.405+0000 7fbfeeffd640 10 monclient: dump: 2024-04-26T00:25:31.398 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:31.398 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:31.398 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:31.398 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:31.398 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:31.398 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:31.399 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:31.399 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:31.399 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:31.399 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:31.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.405+0000 7fbfeeffd640 1 -- 172.21.15.37:0/534565343 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbff40b2040 con 0x7fc0001388a0 2024-04-26T00:25:31.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.405+0000 7fbfeeffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:31.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.405+0000 7fbfeeffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:25:31.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.405+0000 7fbfeeffd640 1 -- 172.21.15.37:0/534565343 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc0001388a0 msgr2=0x7fc000138c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:31.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.405+0000 7fbfeeffd640 1 --2- 172.21.15.37:0/534565343 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc0001388a0 0x7fc000138c80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbff40ae000 tx=0x7fbff4019260 comp rx=0 tx=0).stop 2024-04-26T00:25:31.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.405+0000 7fbfeeffd640 1 -- 172.21.15.37:0/534565343 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc0001388a0 msgr2=0x7fc000138c80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:25:31.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.405+0000 7fbfeeffd640 1 --2- 172.21.15.37:0/534565343 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc0001388a0 0x7fc000138c80 secure :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbff40ae000 tx=0x7fbff4019260 comp rx=0 tx=0).stop 2024-04-26T00:25:31.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.405+0000 7fbffdd9b640 1 --2- 172.21.15.37:0/534565343 >> v2:172.21.15.45:6812/2379918623 conn(0x7fbfe0035e30 0x7fbfe00382f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbff0006fd0 tx=0x7fbff0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:31.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.413+0000 7fc004d2b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:25:31.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.413+0000 7fc004d2b640 1 --2- 172.21.15.37:0/534565343 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc0001391c0 0x7fc000134b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:31.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.413+0000 7fc004d2b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:31.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.413+0000 7fc004d2b640 1 -- 172.21.15.37:0/534565343 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fc000075dc0 con 0x7fc0001391c0 2024-04-26T00:25:31.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.413+0000 7fbffe59c640 1 --2- 172.21.15.37:0/534565343 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc0001391c0 0x7fc000134b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:31.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.413+0000 7fbffe59c640 10 monclient: get_auth_request con 0x7fc0001391c0 auth_method 0 2024-04-26T00:25:31.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.413+0000 7fbffe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:31.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.413+0000 7fbffe59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:31.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.413+0000 7fbffe59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:31.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.413+0000 7fbffe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:31.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.413+0000 7fbffe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:31.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.413+0000 7fbffe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:31.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.413+0000 7fbffe59c640 10 monclient(hunting): handle_auth_done global_id 54255 payload 824 2024-04-26T00:25:31.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.413+0000 7fbffe59c640 1 -- 172.21.15.37:0/534565343 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc00013e7a0 con 0x7fc0001391c0 2024-04-26T00:25:31.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.417+0000 7fbffe59c640 1 -- 172.21.15.37:0/534565343 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc0001391c0 msgr2=0x7fc000134b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:31.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.417+0000 7fbffe59c640 1 --2- 172.21.15.37:0/534565343 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc0001391c0 0x7fc000134b30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fbff401a0f0 tx=0x7fbff4009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:31.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.417+0000 7fbfeeffd640 1 -- 172.21.15.37:0/534565343 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff40ba070 con 0x7fc0001391c0 2024-04-26T00:25:31.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.417+0000 7fbfeeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:31.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.417+0000 7fbfeeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:31.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.417+0000 7fbfeeffd640 10 monclient: dump: 2024-04-26T00:25:31.403 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:31.403 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:31.403 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:31.403 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:31.403 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:31.403 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:31.403 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:31.403 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:31.403 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:31.403 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:31.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.417+0000 7fbfeeffd640 1 -- 172.21.15.37:0/534565343 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4875 (secure 0 0 0) 0x7fbff40ac910 con 0x7fc0001391c0 2024-04-26T00:25:31.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.417+0000 7fbfeeffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:25:31.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.417+0000 7fbfeeffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:25:31.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.417+0000 7fbfeeffd640 1 -- 172.21.15.37:0/534565343 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc0001391c0 msgr2=0x7fc000134b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:31.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.417+0000 7fbfeeffd640 1 --2- 172.21.15.37:0/534565343 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc0001391c0 0x7fc000134b30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fbff401a0f0 tx=0x7fbff4009240 comp rx=0 tx=0).stop 2024-04-26T00:25:31.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.417+0000 7fbfeeffd640 1 -- 172.21.15.37:0/534565343 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc0001391c0 msgr2=0x7fc000134b30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:25:31.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.417+0000 7fbfeeffd640 1 --2- 172.21.15.37:0/534565343 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc0001391c0 0x7fc000134b30 secure :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fbff401a0f0 tx=0x7fbff4009240 comp rx=0 tx=0).stop 2024-04-26T00:25:31.404 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:25:31.404 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:25:31.404 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:25:31.404 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:25:31.405 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 44, 2024-04-26T00:25:31.405 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 94743, 2024-04-26T00:25:31.405 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:25:31.405 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:25:31.405 INFO:teuthology.orchestra.run.smithi037.stdout: 1 2024-04-26T00:25:31.405 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:31.405 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:25:31.405 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:25:31.405 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:25:31.405 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:25:31.405 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:31.405 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:31.405 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:31.405 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:31.406 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:31.406 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:31.406 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:31.406 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:31.406 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:31.406 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:31.406 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:31.406 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:31.406 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:25:31.406 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:25:31.406 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:31.406 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:31.406 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:31.407 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:31.407 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:31.407 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:31.407 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:31.407 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:31.407 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:31.407 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:31.407 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:31.407 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:31.407 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:31.407 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:25:31.407 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:25:31.407 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:25:31.407 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:25:31.408 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:25:31.408 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:25:31.408 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:25:31.408 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:25:31.408 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:25:31.408 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:25:31.408 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:25:31.408 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:25:31.408 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:25:31.408 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:25:31.408 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:25:31.408 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:25:31.408 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:25:31.408 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:31.409 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:31.409 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:31.409 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:31.409 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:31.409 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:31.409 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:31.409 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:31.409 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:31.409 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:31.409 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:31.409 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:31.409 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:25:31.409 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:31.409 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:25:31.410 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:31.410 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:25:31.410 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:25:31.410 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:31.410 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:31.410 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:31.410 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:31.410 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:25:31.410 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:31.410 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:31.410 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:31.410 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:31.410 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:25:31.411 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:31.411 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:31.411 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:31.411 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:31.411 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:25:31.411 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:25:31.411 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:31.411 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:31.411 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:31.411 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:31.411 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:31.411 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:25:31.411 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:25:31.411 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:31.412 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:31.412 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:31.412 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:31.412 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:25:31.412 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:31.412 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:31.412 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:31.412 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:31.412 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:25:31.412 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:31.412 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:31.412 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:31.412 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:31.412 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:25:31.413 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:25:31.413 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:31.413 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:31.413 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:31.413 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:31.413 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:31.413 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:25:31.413 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:25:31.413 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:31.413 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:31.413 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:31.413 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:31.413 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:25:31.413 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:31.414 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:31.414 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:31.414 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:31.414 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:25:31.414 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:31.414 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:31.414 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:31.414 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:31.414 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:25:31.414 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:25:31.414 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:31.414 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:31.414 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:31.414 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:31.415 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:31.415 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:31.415 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:25:31.415 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:25:31.415 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:31.415 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:31.415 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:31.415 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:25:31.415 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:31.415 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:31.415 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:25:31.415 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:31.415 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:31.415 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:31.416 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:25:31.416 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:31.416 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:31.416 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:25:31.416 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:31.416 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:31.416 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:31.416 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:25:31.416 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:31.416 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:31.416 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:25:31.416 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:31.416 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:31.417 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:31.417 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:25:31.417 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:31.417 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:31.417 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:31.417 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:25:31.417 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:25:31.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.417+0000 7fc004d2b640 1 -- 172.21.15.37:0/534565343 >> v2:172.21.15.45:6812/2379918623 conn(0x7fbfe0035e30 msgr2=0x7fbfe00382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:31.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.417+0000 7fc004d2b640 1 --2- 172.21.15.37:0/534565343 >> v2:172.21.15.45:6812/2379918623 conn(0x7fbfe0035e30 0x7fbfe00382f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbff0006fd0 tx=0x7fbff0008040 comp rx=0 tx=0).stop 2024-04-26T00:25:31.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.417+0000 7fc004d2b640 10 monclient: shutdown 2024-04-26T00:25:31.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.417+0000 7fc004d2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:31.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.417+0000 7fc004d2b640 1 -- 172.21.15.37:0/534565343 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc000133b20 msgr2=0x7fc00013e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:31.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.417+0000 7fc004d2b640 1 --2- 172.21.15.37:0/534565343 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc000133b20 0x7fc00013e260 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbff800c7a0 tx=0x7fbff800cc60 comp rx=0 tx=0).stop 2024-04-26T00:25:31.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.417+0000 7fbffed9d640 1 -- 172.21.15.37:0/534565343 reap_dead start 2024-04-26T00:25:31.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.417+0000 7fbffed9d640 1 -- 172.21.15.37:0/534565343 reap_dead start 2024-04-26T00:25:31.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.417+0000 7fc004d2b640 1 -- 172.21.15.37:0/534565343 shutdown_connections 2024-04-26T00:25:31.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.417+0000 7fc004d2b640 1 -- 172.21.15.37:0/534565343 >> 172.21.15.37:0/534565343 conn(0x7fc0000aa0a0 msgr2=0x7fc00009f080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:31.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.417+0000 7fc004d2b640 1 -- 172.21.15.37:0/534565343 shutdown_connections 2024-04-26T00:25:31.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.417+0000 7fc004d2b640 1 -- 172.21.15.37:0/534565343 wait complete. 2024-04-26T00:25:31.419 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:25:31.420 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-04-26T00:25:31.420 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:25:31.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.553+0000 7fd488242640 10 monclient: get_monmap_and_config 2024-04-26T00:25:31.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.553+0000 7fd488242640 10 monclient: build_initial_monmap 2024-04-26T00:25:31.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.553+0000 7fd488242640 10 monclient: monmap: 2024-04-26T00:25:31.535 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:31.535 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:31.535 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:25:31.556942+0000 2024-04-26T00:25:31.535 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:25:31.556942+0000 2024-04-26T00:25:31.535 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:31.535 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:31.535 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:25:31.535 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:25:31.535 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:25:31.535 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:25:31.535 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:25:31.535 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:31.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.553+0000 7fd488242640 1 Processor -- start 2024-04-26T00:25:31.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.553+0000 7fd488242640 1 -- start start 2024-04-26T00:25:31.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.553+0000 7fd488242640 10 monclient: init 2024-04-26T00:25:31.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.553+0000 7fd488242640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:31.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.553+0000 7fd488242640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-04-26T00:25:31.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.553+0000 7fd488242640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd48009e220 0x7fd48009fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:31.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.553+0000 7fd488242640 10 monclient(hunting): picked mon.noname-b con 0x7fd48009e220 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:31.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.553+0000 7fd488242640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd4800a14f0 0x7fd4800a0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:31.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.553+0000 7fd488242640 10 monclient(hunting): picked mon.noname-c con 0x7fd4800a14f0 addr v2:172.21.15.45:3300/0 2024-04-26T00:25:31.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.553+0000 7fd488242640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd48009e600 addr v1:172.21.15.37:6789/0 2024-04-26T00:25:31.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.553+0000 7fd488242640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4800a12b0 con 0x7fd48009e600 2024-04-26T00:25:31.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.553+0000 7fd488242640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:31.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.553+0000 7fd488242640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fd48009cd10 con 0x7fd4800a14f0 2024-04-26T00:25:31.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.553+0000 7fd488242640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:31.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.553+0000 7fd488242640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd48009da20 con 0x7fd48009e220 2024-04-26T00:25:31.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.553+0000 7fd488242640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:31.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.553+0000 7fd488242640 10 monclient(hunting): authenticate will time out at 1312.863330s 2024-04-26T00:25:31.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.553+0000 7fd485fde640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd48009e220 msgr2=0x7fd48009fc50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:25:31.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.553+0000 7fd485fde640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd48009e220 0x7fd48009fc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:25:31.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.553+0000 7fd4867df640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fd48009e600 0x7fd480139a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:46030/0 (socket says 172.21.15.37:46030) 2024-04-26T00:25:31.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.553+0000 7fd4867df640 1 -- 172.21.15.37:0/3411347230 learned_addr learned my addr 172.21.15.37:0/3411347230 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:25:31.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.553+0000 7fd4857dd640 1 --2- 172.21.15.37:0/3411347230 >> v2:172.21.15.45:3300/0 conn(0x7fd4800a14f0 0x7fd4800a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:31.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.553+0000 7fd4857dd640 10 monclient(hunting): get_auth_request con 0x7fd4800a14f0 auth_method 0 2024-04-26T00:25:31.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.553+0000 7fd4857dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:31.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.553+0000 7fd4857dd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:31.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.553+0000 7fd4857dd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:31.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.557+0000 7fd4857dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:31.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.557+0000 7fd4857dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:31.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.557+0000 7fd4857dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:31.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.557+0000 7fd4857dd640 10 monclient(hunting): handle_auth_done global_id 24496 payload 274 2024-04-26T00:25:31.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.557+0000 7fd4857dd640 1 -- 172.21.15.37:0/3411347230 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd48009e220 msgr2=0x7fd48009fc50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:25:31.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.557+0000 7fd4857dd640 1 --2- 172.21.15.37:0/3411347230 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd48009e220 0x7fd48009fc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:31.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.557+0000 7fd4857dd640 1 -- 172.21.15.37:0/3411347230 >> v1:172.21.15.37:6789/0 conn(0x7fd48009e600 legacy=0x7fd480139a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:31.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.557+0000 7fd4857dd640 10 monclient: _finish_hunting 0 2024-04-26T00:25:31.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.557+0000 7fd4857dd640 1 monclient: found mon.noname-c 2024-04-26T00:25:31.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.557+0000 7fd4857dd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:31.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.557+0000 7fd4857dd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:25:31.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.557+0000 7fd4857dd640 1 -- 172.21.15.37:0/3411347230 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd48009dd00 con 0x7fd4800a14f0 2024-04-26T00:25:31.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd4857dd640 1 -- 172.21.15.37:0/3411347230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4800a14f0 msgr2=0x7fd4800a0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:31.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd4857dd640 1 --2- 172.21.15.37:0/3411347230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4800a14f0 0x7fd4800a0190 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd4700097c0 tx=0x7fd47001c340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e78ec8297fb6d827 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:31.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd484fdc640 1 -- 172.21.15.37:0/3411347230 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4700b0070 con 0x7fd4800a14f0 2024-04-26T00:25:31.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd484fdc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:31.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd484fdc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:25:31.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd484fdc640 10 monclient: dump: 2024-04-26T00:25:31.543 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:31.543 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:31.543 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:31.543 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:31.543 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:31.543 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:31.543 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:31.543 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:31.544 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:31.544 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:31.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd484fdc640 10 monclient: _finish_auth 0 2024-04-26T00:25:31.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd484fdc640 10 monclient: _check_auth_tickets 2024-04-26T00:25:31.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd484fdc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:31.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd484fdc640 1 -- 172.21.15.37:0/3411347230 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4700ab600 con 0x7fd4800a14f0 2024-04-26T00:25:31.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd484fdc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:31.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd484fdc640 1 -- 172.21.15.37:0/3411347230 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4700ab9d0 con 0x7fd4800a14f0 2024-04-26T00:25:31.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd484fdc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:31.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd484fdc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:25:31.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd484fdc640 10 monclient: dump: 2024-04-26T00:25:31.544 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:31.544 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:31.545 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:31.545 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:31.545 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:31.545 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:31.545 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:31.545 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:31.545 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:31.545 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:31.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd488242640 5 monclient: authenticate success, global_id 24496 2024-04-26T00:25:31.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd488242640 10 monclient: get_monmap_and_config success 2024-04-26T00:25:31.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd488242640 4 set_mon_vals no callback set 2024-04-26T00:25:31.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd488242640 10 monclient: shutdown 2024-04-26T00:25:31.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd488242640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:31.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd488242640 1 -- 172.21.15.37:0/3411347230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4800a14f0 msgr2=0x7fd4800a0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:31.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd488242640 1 --2- 172.21.15.37:0/3411347230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4800a14f0 0x7fd4800a0190 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd4700097c0 tx=0x7fd47001c340 comp rx=0 tx=0).stop 2024-04-26T00:25:31.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd488242640 1 -- 172.21.15.37:0/3411347230 shutdown_connections 2024-04-26T00:25:31.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd488242640 1 --2- 172.21.15.37:0/3411347230 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4800a14f0 0x7fd4800a0190 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:31.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd488242640 1 --2- 172.21.15.37:0/3411347230 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd48009e220 0x7fd48009fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:31.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd488242640 1 -- 172.21.15.37:0/3411347230 >> 172.21.15.37:0/3411347230 conn(0x7fd4800a1d50 msgr2=0x7fd4800a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:31.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd488242640 1 -- 172.21.15.37:0/3411347230 shutdown_connections 2024-04-26T00:25:31.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd488242640 1 -- 172.21.15.37:0/3411347230 wait complete. 2024-04-26T00:25:31.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd488242640 10 monclient: build_initial_monmap 2024-04-26T00:25:31.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd488242640 10 monclient: monmap: 2024-04-26T00:25:31.551 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:31.551 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:25:31.551 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:25:31.551 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:25:31.552 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:31.552 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:25:31.552 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:25:31.552 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:25:31.552 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:25:31.552 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:31.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd488242640 1 Processor -- start 2024-04-26T00:25:31.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd488242640 1 -- start start 2024-04-26T00:25:31.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd488242640 10 monclient: init 2024-04-26T00:25:31.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd488242640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:31.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd488242640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:25:31.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd488242640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd48009e220 0x7fd4801cf4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:31.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd488242640 10 monclient(hunting): picked mon.noname-a con 0x7fd48009e220 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:25:31.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd488242640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd48009e600 0x7fd480137040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:31.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd488242640 10 monclient(hunting): picked mon.noname-b con 0x7fd48009e600 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:25:31.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd488242640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd480137580 0x7fd48011fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:31.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd488242640 10 monclient(hunting): picked mon.noname-c con 0x7fd480137580 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:31.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd488242640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:31.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd488242640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd48013f990 con 0x7fd48009e220 2024-04-26T00:25:31.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd488242640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:31.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd488242640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd48013f810 con 0x7fd48009e600 2024-04-26T00:25:31.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd488242640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:31.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd488242640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd48013fb10 con 0x7fd480137580 2024-04-26T00:25:31.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd488242640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:31.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd488242640 10 monclient(hunting): authenticate will time out at 1312.871527s 2024-04-26T00:25:31.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd4867df640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd480137580 msgr2=0x7fd48011fb10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:25:31.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd4867df640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd480137580 0x7fd48011fb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:25:31.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd485fde640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd48009e220 0x7fd4801cf4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:31.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd485fde640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd48009e220 0x7fd4801cf4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40984/0 (socket says 172.21.15.37:40984) 2024-04-26T00:25:31.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd485fde640 1 -- 172.21.15.37:0/3862123812 learned_addr learned my addr 172.21.15.37:0/3862123812 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:31.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd485fde640 10 monclient(hunting): get_auth_request con 0x7fd48009e220 auth_method 0 2024-04-26T00:25:31.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd485fde640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:31.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd485fde640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:31.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd485fde640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:31.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd485fde640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:31.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd485fde640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:31.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd485fde640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:31.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd4857dd640 1 --2- 172.21.15.37:0/3862123812 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd48009e600 0x7fd480137040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:31.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.561+0000 7fd4857dd640 10 monclient(hunting): get_auth_request con 0x7fd48009e600 auth_method 0 2024-04-26T00:25:31.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd4857dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:31.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd4857dd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:31.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd4857dd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:31.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd485fde640 10 monclient(hunting): handle_auth_done global_id 54258 payload 824 2024-04-26T00:25:31.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd485fde640 1 -- 172.21.15.37:0/3862123812 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd480137580 msgr2=0x7fd48011fb10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:25:31.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd485fde640 1 --2- 172.21.15.37:0/3862123812 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd480137580 0x7fd48011fb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:31.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd485fde640 1 -- 172.21.15.37:0/3862123812 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd48009e600 msgr2=0x7fd480137040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:31.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd485fde640 1 --2- 172.21.15.37:0/3862123812 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd48009e600 0x7fd480137040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:31.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd485fde640 10 monclient: _finish_hunting 0 2024-04-26T00:25:31.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd485fde640 1 monclient: found mon.noname-a 2024-04-26T00:25:31.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd485fde640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:31.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd485fde640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:25:31.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd485fde640 1 -- 172.21.15.37:0/3862123812 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd480123a10 con 0x7fd48009e220 2024-04-26T00:25:31.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd485fde640 1 -- 172.21.15.37:0/3862123812 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd48009e220 msgr2=0x7fd4801cf4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:31.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd485fde640 1 --2- 172.21.15.37:0/3862123812 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd48009e220 0x7fd4801cf4a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd47c00b550 tx=0x7fd47c00ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:31.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd4767fc640 1 -- 172.21.15.37:0/3862123812 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd47c0050f0 con 0x7fd48009e220 2024-04-26T00:25:31.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd4767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:31.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd4767fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:25:31.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd4767fc640 10 monclient: dump: 2024-04-26T00:25:31.556 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:31.556 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:31.556 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:31.556 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:31.556 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:31.556 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:31.556 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:31.556 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:31.556 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:31.556 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:31.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd4767fc640 10 monclient: _finish_auth 0 2024-04-26T00:25:31.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd4767fc640 10 monclient: _check_auth_tickets 2024-04-26T00:25:31.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd4767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:31.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd4767fc640 1 -- 172.21.15.37:0/3862123812 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd47c015070 con 0x7fd48009e220 2024-04-26T00:25:31.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd4767fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:31.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd4767fc640 1 -- 172.21.15.37:0/3862123812 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd47c0054d0 con 0x7fd48009e220 2024-04-26T00:25:31.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd488242640 5 monclient: authenticate success, global_id 54258 2024-04-26T00:25:31.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd488242640 10 monclient: _renew_subs 2024-04-26T00:25:31.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd488242640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:31.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd488242640 1 -- 172.21.15.37:0/3862123812 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd48009c970 con 0x7fd48009e220 2024-04-26T00:25:31.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd4767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:31.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd4767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:31.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd477fff640 4 set_mon_vals no callback set 2024-04-26T00:25:31.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd4767fc640 10 monclient: dump: 2024-04-26T00:25:31.557 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:31.558 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:31.558 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:31.558 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:31.558 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:31.558 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:31.558 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:31.558 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:31.558 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:31.558 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:31.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd488242640 10 monclient: _renew_subs 2024-04-26T00:25:31.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd488242640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:31.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd488242640 1 -- 172.21.15.37:0/3862123812 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd480121720 con 0x7fd48009e220 2024-04-26T00:25:31.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd488242640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:31.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd488242640 1 --2- 172.21.15.37:0/3862123812 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd480120050 0x7fd480120430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:31.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd488242640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:31.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd488242640 1 -- 172.21.15.37:0/3862123812 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd480120970 con 0x7fd480120050 2024-04-26T00:25:31.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd4767fc640 1 -- 172.21.15.37:0/3862123812 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd47c0056b0 con 0x7fd48009e220 2024-04-26T00:25:31.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd4767fc640 1 --2- 172.21.15.37:0/3862123812 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd464035e60 0x7fd464038320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:31.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd4857dd640 1 --2- 172.21.15.37:0/3862123812 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd480120050 0x7fd480120430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:31.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd4767fc640 1 -- 172.21.15.37:0/3862123812 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(208..208 src has 177..208) ==== 3963+0+0 (secure 0 0 0) 0x7fd47c0489c0 con 0x7fd48009e220 2024-04-26T00:25:31.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd4857dd640 10 monclient: get_auth_request con 0x7fd480120050 auth_method 0 2024-04-26T00:25:31.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd4857dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:31.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd4857dd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:31.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd4857dd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:31.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd4857dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:31.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd4857dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:31.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.565+0000 7fd4857dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:31.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.569+0000 7fd4867df640 1 --2- 172.21.15.37:0/3862123812 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd464035e60 0x7fd464038320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:31.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.569+0000 7fd4867df640 10 monclient: get_auth_request con 0x7fd464035e60 auth_method 0 2024-04-26T00:25:31.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.569+0000 7fd4857dd640 10 monclient(hunting): handle_auth_done global_id 24502 payload 824 2024-04-26T00:25:31.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.569+0000 7fd4857dd640 1 -- 172.21.15.37:0/3862123812 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd48004f510 con 0x7fd480120050 2024-04-26T00:25:31.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.569+0000 7fd4857dd640 1 -- 172.21.15.37:0/3862123812 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd480120050 msgr2=0x7fd480120430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:31.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.569+0000 7fd4857dd640 1 --2- 172.21.15.37:0/3862123812 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd480120050 0x7fd480120430 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd48006d140 tx=0x7fd4700190e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:31.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.569+0000 7fd4867df640 1 --2- 172.21.15.37:0/3862123812 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd464035e60 0x7fd464038320 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd48009ea50 tx=0x7fd478004fe0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:31.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.569+0000 7fd4767fc640 1 -- 172.21.15.37:0/3862123812 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4700bb030 con 0x7fd480120050 2024-04-26T00:25:31.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.569+0000 7fd4767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:31.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.569+0000 7fd4767fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:25:31.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.569+0000 7fd4767fc640 10 monclient: dump: 2024-04-26T00:25:31.562 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:31.562 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:31.562 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:31.562 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:31.562 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:31.562 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:31.562 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:31.562 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:31.562 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:31.562 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:31.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.569+0000 7fd4767fc640 1 -- 172.21.15.37:0/3862123812 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd4700b1040 con 0x7fd480120050 2024-04-26T00:25:31.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.569+0000 7fd4767fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:31.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.569+0000 7fd4767fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:25:31.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.569+0000 7fd4767fc640 1 -- 172.21.15.37:0/3862123812 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd480120050 msgr2=0x7fd480120430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:31.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.569+0000 7fd4767fc640 1 --2- 172.21.15.37:0/3862123812 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd480120050 0x7fd480120430 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd48006d140 tx=0x7fd4700190e0 comp rx=0 tx=0).stop 2024-04-26T00:25:31.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.569+0000 7fd4767fc640 1 -- 172.21.15.37:0/3862123812 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd480120050 msgr2=0x7fd480120430 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:25:31.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.569+0000 7fd4767fc640 1 --2- 172.21.15.37:0/3862123812 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd480120050 0x7fd480120430 secure :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd48006d140 tx=0x7fd4700190e0 comp rx=0 tx=0).stop 2024-04-26T00:25:31.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.577+0000 7fd488242640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:25:31.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.577+0000 7fd488242640 1 --2- 172.21.15.37:0/3862123812 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd48011b8a0 0x7fd48011bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:31.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.577+0000 7fd488242640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:31.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.577+0000 7fd488242640 1 -- 172.21.15.37:0/3862123812 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd480122000 con 0x7fd48011b8a0 2024-04-26T00:25:31.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.577+0000 7fd4857dd640 1 --2- 172.21.15.37:0/3862123812 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd48011b8a0 0x7fd48011bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:31.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.577+0000 7fd4857dd640 10 monclient: get_auth_request con 0x7fd48011b8a0 auth_method 0 2024-04-26T00:25:31.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.577+0000 7fd4857dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:31.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.577+0000 7fd4857dd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:31.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.577+0000 7fd4857dd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:31.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.577+0000 7fd4857dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:31.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.577+0000 7fd4857dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:31.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.577+0000 7fd4857dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:31.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.577+0000 7fd4857dd640 10 monclient(hunting): handle_auth_done global_id 24505 payload 824 2024-04-26T00:25:31.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.577+0000 7fd4857dd640 1 -- 172.21.15.37:0/3862123812 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd480123270 con 0x7fd48011b8a0 2024-04-26T00:25:31.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.581+0000 7fd4857dd640 1 -- 172.21.15.37:0/3862123812 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd48011b8a0 msgr2=0x7fd48011bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:31.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.581+0000 7fd4857dd640 1 --2- 172.21.15.37:0/3862123812 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd48011b8a0 0x7fd48011bcb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd470019f70 tx=0x7fd470009190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:31.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.581+0000 7fd4767fc640 1 -- 172.21.15.37:0/3862123812 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4700bb040 con 0x7fd48011b8a0 2024-04-26T00:25:31.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.581+0000 7fd4767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:31.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.581+0000 7fd4767fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:25:31.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.581+0000 7fd4767fc640 10 monclient: dump: 2024-04-26T00:25:31.565 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:31.565 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:31.565 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:31.565 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:31.565 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:31.566 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:31.566 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:31.566 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:31.566 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:31.566 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:31.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.581+0000 7fd4767fc640 1 -- 172.21.15.37:0/3862123812 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4708 (secure 0 0 0) 0x7fd4700b7500 con 0x7fd48011b8a0 2024-04-26T00:25:31.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.581+0000 7fd4767fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:25:31.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.581+0000 7fd4767fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:25:31.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.581+0000 7fd4767fc640 1 -- 172.21.15.37:0/3862123812 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd48011b8a0 msgr2=0x7fd48011bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:31.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.581+0000 7fd4767fc640 1 --2- 172.21.15.37:0/3862123812 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd48011b8a0 0x7fd48011bcb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd470019f70 tx=0x7fd470009190 comp rx=0 tx=0).stop 2024-04-26T00:25:31.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.581+0000 7fd4767fc640 1 -- 172.21.15.37:0/3862123812 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd48011b8a0 msgr2=0x7fd48011bcb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:25:31.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.581+0000 7fd4767fc640 1 --2- 172.21.15.37:0/3862123812 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd48011b8a0 0x7fd48011bcb0 secure :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd470019f70 tx=0x7fd470009190 comp rx=0 tx=0).stop 2024-04-26T00:25:31.567 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:25:31.567 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:25:31.567 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:25:31.567 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:25:31.567 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 44, 2024-04-26T00:25:31.567 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 239747, 2024-04-26T00:25:31.567 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:25:31.567 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:25:31.567 INFO:teuthology.orchestra.run.smithi037.stdout: 1 2024-04-26T00:25:31.567 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:31.567 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:25:31.567 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:25:31.567 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:25:31.568 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:25:31.568 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:31.568 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:31.568 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:31.568 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:31.568 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:31.568 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:31.568 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:31.568 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:31.568 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:31.568 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:31.568 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:31.568 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:31.568 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:25:31.568 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:25:31.568 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:31.569 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:31.569 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:31.569 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:31.569 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:31.569 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:31.569 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:31.569 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:31.569 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:31.569 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:31.569 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:31.569 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:31.569 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:31.569 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:25:31.570 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:25:31.570 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:25:31.570 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:25:31.570 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:25:31.570 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:25:31.570 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:25:31.570 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:25:31.570 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:25:31.570 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:25:31.570 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:25:31.570 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:25:31.570 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:25:31.570 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:25:31.570 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:25:31.571 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:25:31.571 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:25:31.571 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:31.571 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:31.571 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:31.571 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:31.571 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:31.571 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:31.571 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:31.571 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:31.571 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:31.571 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:31.571 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:31.571 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:31.571 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:25:31.572 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:31.572 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:25:31.572 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:31.572 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:25:31.572 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:25:31.572 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:31.572 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:31.572 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:31.572 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:31.572 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:25:31.572 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:31.572 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:31.572 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:31.573 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:31.573 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:25:31.573 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:31.573 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:31.573 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:31.573 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:31.573 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:25:31.573 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:25:31.573 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:31.573 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:31.573 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:31.573 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:31.573 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:31.573 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:25:31.574 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:25:31.574 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:31.574 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:31.574 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:31.574 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:31.574 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:25:31.574 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:31.574 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:31.574 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:31.574 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:31.574 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:25:31.574 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:31.574 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:31.574 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:31.575 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:31.575 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:25:31.575 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:25:31.575 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:31.575 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:31.575 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:31.575 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:31.575 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:31.575 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:25:31.575 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:25:31.575 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:31.575 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:31.575 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:31.575 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:31.575 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:25:31.576 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:31.576 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:31.576 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:31.576 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:31.576 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:25:31.576 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:31.576 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:31.576 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:31.576 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:31.576 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:25:31.576 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:25:31.576 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:31.576 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:31.576 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:31.576 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:31.576 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:31.577 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:31.577 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:25:31.577 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:25:31.577 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:31.577 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:31.577 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:31.577 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:25:31.577 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:31.577 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:31.577 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:25:31.577 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:31.577 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:31.577 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:31.577 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:25:31.577 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:31.578 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:31.578 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:25:31.578 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:31.578 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:31.578 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:31.578 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:25:31.578 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:31.578 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:31.578 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:31.578 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:25:31.578 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:25:31.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.581+0000 7fd488242640 1 -- 172.21.15.37:0/3862123812 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd464035e60 msgr2=0x7fd464038320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:31.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.581+0000 7fd488242640 1 --2- 172.21.15.37:0/3862123812 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd464035e60 0x7fd464038320 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd48009ea50 tx=0x7fd478004fe0 comp rx=0 tx=0).stop 2024-04-26T00:25:31.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.581+0000 7fd488242640 10 monclient: shutdown 2024-04-26T00:25:31.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.581+0000 7fd488242640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:31.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.581+0000 7fd488242640 1 -- 172.21.15.37:0/3862123812 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd48009e220 msgr2=0x7fd4801cf4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:31.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.581+0000 7fd488242640 1 --2- 172.21.15.37:0/3862123812 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd48009e220 0x7fd4801cf4a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd47c00b550 tx=0x7fd47c00ba10 comp rx=0 tx=0).stop 2024-04-26T00:25:31.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.581+0000 7fd4867df640 1 -- 172.21.15.37:0/3862123812 reap_dead start 2024-04-26T00:25:31.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.581+0000 7fd4867df640 1 -- 172.21.15.37:0/3862123812 reap_dead start 2024-04-26T00:25:31.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.581+0000 7fd488242640 1 -- 172.21.15.37:0/3862123812 shutdown_connections 2024-04-26T00:25:31.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.581+0000 7fd488242640 1 -- 172.21.15.37:0/3862123812 >> 172.21.15.37:0/3862123812 conn(0x7fd4800a1d50 msgr2=0x7fd480072800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:31.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.581+0000 7fd488242640 1 -- 172.21.15.37:0/3862123812 shutdown_connections 2024-04-26T00:25:31.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:31.581+0000 7fd488242640 1 -- 172.21.15.37:0/3862123812 wait complete. 2024-04-26T00:25:31.581 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:25:31.581 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-04-26T00:25:31.954 INFO:tasks.rados.rados.0.smithi045.stdout:861: done (1 left) 2024-04-26T00:25:31.954 INFO:tasks.rados.rados.0.smithi045.stdout:862: done (0 left) 2024-04-26T00:25:31.954 INFO:tasks.rados.rados.0.smithi045.stdout:863: delete oid 50 current snap is 104 2024-04-26T00:25:31.963 INFO:tasks.rados.rados.0.smithi045.stdout:863: done (0 left) 2024-04-26T00:25:31.964 INFO:tasks.rados.rados.0.smithi045.stdout:864: delete oid 47 current snap is 104 2024-04-26T00:25:31.977 INFO:tasks.rados.rados.0.smithi045.stdout:864: done (0 left) 2024-04-26T00:25:31.977 INFO:tasks.rados.rados.0.smithi045.stdout:865: read oid 15 snap -1 2024-04-26T00:25:31.977 INFO:tasks.rados.rados.0.smithi045.stdout:865: expect deleted 2024-04-26T00:25:31.977 INFO:tasks.rados.rados.0.smithi045.stdout:866: write (excl) oid 9 current snap is 104 2024-04-26T00:25:31.977 INFO:tasks.rados.rados.0.smithi045.stdout:866: seq_num 344 ranges {581608=721541,1863688=485773,3113980=604043} 2024-04-26T00:25:31.997 INFO:tasks.rados.rados.0.smithi045.stdout:866: writing smithi04526579-9 from 581608 to 1303149 tid 1 2024-04-26T00:25:32.015 INFO:tasks.rados.rados.0.smithi045.stdout:866: writing smithi04526579-9 from 1863688 to 2349461 tid 2 2024-04-26T00:25:32.034 INFO:tasks.rados.rados.0.smithi045.stdout:866: writing smithi04526579-9 from 3113980 to 3718023 tid 3 2024-04-26T00:25:32.035 INFO:tasks.rados.rados.0.smithi045.stdout:867: read oid 7 snap -1 2024-04-26T00:25:32.035 INFO:tasks.rados.rados.0.smithi045.stdout:867: expect deleted 2024-04-26T00:25:32.035 INFO:tasks.rados.rados.0.smithi045.stdout:868: read oid 42 snap -1 2024-04-26T00:25:32.035 INFO:tasks.rados.rados.0.smithi045.stdout:868: expect (ObjNum 114 snap 28 seq_num 114) 2024-04-26T00:25:32.035 INFO:tasks.rados.rados.0.smithi045.stdout:869: read oid 12 snap 86 2024-04-26T00:25:32.035 INFO:tasks.rados.rados.0.smithi045.stdout:869: expect (ObjNum 180 snap 47 seq_num 180) 2024-04-26T00:25:32.035 INFO:tasks.rados.rados.0.smithi045.stdout:870: copy_from oid 43 from oid 22 current snap is 104 2024-04-26T00:25:32.035 INFO:tasks.rados.rados.0.smithi045.stdout:866: finishing write tid 1 to smithi04526579-9 2024-04-26T00:25:32.035 INFO:tasks.rados.rados.0.smithi045.stdout:866: finishing write tid 2 to smithi04526579-9 2024-04-26T00:25:32.035 INFO:tasks.rados.rados.0.smithi045.stdout:865: done (5 left) 2024-04-26T00:25:32.035 INFO:tasks.rados.rados.0.smithi045.stdout:871: copy_from oid 39 from oid 28 current snap is 104 2024-04-26T00:25:32.035 INFO:tasks.rados.rados.0.smithi045.stdout:872: snap_remove snap 94 2024-04-26T00:25:32.962 INFO:tasks.rados.rados.0.smithi045.stdout:872: done (6 left) 2024-04-26T00:25:32.962 INFO:tasks.rados.rados.0.smithi045.stdout:873: snap_create 2024-04-26T00:25:32.962 INFO:tasks.rados.rados.0.smithi045.stdout:871: finishing copy_from to smithi04526579-39 2024-04-26T00:25:32.963 INFO:tasks.rados.rados.0.smithi045.stdout:871: got expected ENOENT (src dne) 2024-04-26T00:25:32.963 INFO:tasks.rados.rados.0.smithi045.stdout:871: finishing copy_from racing read to smithi04526579-39 2024-04-26T00:25:32.963 INFO:tasks.rados.rados.0.smithi045.stdout:869: expect (ObjNum 180 snap 47 seq_num 180) 2024-04-26T00:25:33.162 INFO:tasks.rados.rados.0.smithi045.stdout:866: finishing write tid 3 to smithi04526579-9 2024-04-26T00:25:33.163 INFO:tasks.rados.rados.0.smithi045.stdout:866: finishing write tid 4 to smithi04526579-9 2024-04-26T00:25:33.163 INFO:tasks.rados.rados.0.smithi045.stdout:866: finishing write tid 5 to smithi04526579-9 2024-04-26T00:25:33.163 INFO:tasks.rados.rados.0.smithi045.stdout:866: oid 9 updating version 0 to 461 2024-04-26T00:25:33.163 INFO:tasks.rados.rados.0.smithi045.stdout:866: oid 9 version 461 is already newer than 460 2024-04-26T00:25:33.163 INFO:tasks.rados.rados.0.smithi045.stdout:866: oid 9 version 461 is already newer than 459 2024-04-26T00:25:33.163 INFO:tasks.rados.rados.0.smithi045.stdout:866: oid 9 version 461 is already newer than 458 2024-04-26T00:25:33.163 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 461 (ObjNum 344 snap 104 seq_num 344) dirty exists 2024-04-26T00:25:33.163 INFO:tasks.rados.rados.0.smithi045.stdout:866: left oid 9 (ObjNum 344 snap 104 seq_num 344) 2024-04-26T00:25:33.163 INFO:tasks.rados.rados.0.smithi045.stdout:868: expect (ObjNum 114 snap 28 seq_num 114) 2024-04-26T00:25:33.292 INFO:tasks.rados.rados.0.smithi045.stdout:870: finishing copy_from racing read to smithi04526579-43 2024-04-26T00:25:33.293 INFO:tasks.rados.rados.0.smithi045.stdout:870: finishing copy_from to smithi04526579-43 2024-04-26T00:25:33.293 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 211 (ObjNum 330 snap 96 seq_num 330) dirty exists 2024-04-26T00:25:33.969 INFO:tasks.rados.rados.0.smithi045.stdout:866: done (6 left) 2024-04-26T00:25:33.970 INFO:tasks.rados.rados.0.smithi045.stdout:867: done (5 left) 2024-04-26T00:25:33.970 INFO:tasks.rados.rados.0.smithi045.stdout:868: done (4 left) 2024-04-26T00:25:33.970 INFO:tasks.rados.rados.0.smithi045.stdout:869: done (3 left) 2024-04-26T00:25:33.970 INFO:tasks.rados.rados.0.smithi045.stdout:870: done (2 left) 2024-04-26T00:25:33.970 INFO:tasks.rados.rados.0.smithi045.stdout:871: done (1 left) 2024-04-26T00:25:33.970 INFO:tasks.rados.rados.0.smithi045.stdout:873: done (0 left) 2024-04-26T00:25:33.970 INFO:tasks.rados.rados.0.smithi045.stdout:874: snap_remove snap 95 2024-04-26T00:25:34.979 INFO:tasks.rados.rados.0.smithi045.stdout:874: done (0 left) 2024-04-26T00:25:34.979 INFO:tasks.rados.rados.0.smithi045.stdout:875: delete oid 19 current snap is 105 2024-04-26T00:25:34.990 INFO:tasks.rados.rados.0.smithi045.stdout:875: done (0 left) 2024-04-26T00:25:34.990 INFO:tasks.rados.rados.0.smithi045.stdout:876: write (excl) oid 40 current snap is 105 2024-04-26T00:25:34.990 INFO:tasks.rados.rados.0.smithi045.stdout:876: seq_num 346 ranges {615222=472039,1580652=489442,2768826=626587} 2024-04-26T00:25:35.009 INFO:tasks.rados.rados.0.smithi045.stdout:876: writing smithi04526579-40 from 615222 to 1087261 tid 1 2024-04-26T00:25:35.025 INFO:tasks.rados.rados.0.smithi045.stdout:876: writing smithi04526579-40 from 1580652 to 2070094 tid 2 2024-04-26T00:25:35.045 INFO:tasks.rados.rados.0.smithi045.stdout:876: writing smithi04526579-40 from 2768826 to 3395413 tid 3 2024-04-26T00:25:35.045 INFO:tasks.rados.rados.0.smithi045.stdout:877: write (excl) oid 24 current snap is 105 2024-04-26T00:25:35.045 INFO:tasks.rados.rados.0.smithi045.stdout:877: seq_num 347 ranges {632029=547288,1639134=749453,2796249=437859,3653524=178505} 2024-04-26T00:25:35.059 INFO:tasks.rados.rados.0.smithi045.stdout:877: writing smithi04526579-24 from 632029 to 1179317 tid 1 2024-04-26T00:25:35.070 INFO:tasks.rados.rados.0.smithi045.stdout:877: writing smithi04526579-24 from 1639134 to 2388587 tid 2 2024-04-26T00:25:35.078 INFO:tasks.rados.rados.0.smithi045.stdout:877: writing smithi04526579-24 from 2796249 to 3234108 tid 3 2024-04-26T00:25:35.082 INFO:tasks.rados.rados.0.smithi045.stdout:877: writing smithi04526579-24 from 3653524 to 3832029 tid 4 2024-04-26T00:25:35.083 INFO:tasks.rados.rados.0.smithi045.stdout:878: write (excl) oid 17 current snap is 105 2024-04-26T00:25:35.083 INFO:tasks.rados.rados.0.smithi045.stdout:878: seq_num 348 ranges {648836=622537,1981263=493111,3107319=741517} 2024-04-26T00:25:35.091 INFO:tasks.rados.rados.0.smithi045.stdout:878: writing smithi04526579-17 from 648836 to 1271373 tid 1 2024-04-26T00:25:35.100 INFO:tasks.rados.rados.0.smithi045.stdout:878: writing smithi04526579-17 from 1981263 to 2474374 tid 2 2024-04-26T00:25:35.109 INFO:tasks.rados.rados.0.smithi045.stdout:878: writing smithi04526579-17 from 3107319 to 3848836 tid 3 2024-04-26T00:25:35.109 INFO:tasks.rados.rados.0.smithi045.stdout:879: read oid 36 snap 100 2024-04-26T00:25:35.109 INFO:tasks.rados.rados.0.smithi045.stdout:879: expect (ObjNum 304 snap 87 seq_num 304) 2024-04-26T00:25:35.109 INFO:tasks.rados.rados.0.smithi045.stdout:880: write (excl) oid 16 current snap is 105 2024-04-26T00:25:35.109 INFO:tasks.rados.rados.0.smithi045.stdout:876: finishing write tid 1 to smithi04526579-40 2024-04-26T00:25:35.109 INFO:tasks.rados.rados.0.smithi045.stdout:880: seq_num 349 ranges {665643=697786,2039745=753122,3534742=330901} 2024-04-26T00:25:35.118 INFO:tasks.rados.rados.0.smithi045.stdout:880: writing smithi04526579-16 from 665643 to 1363429 tid 1 2024-04-26T00:25:35.127 INFO:tasks.rados.rados.0.smithi045.stdout:880: writing smithi04526579-16 from 2039745 to 2792867 tid 2 2024-04-26T00:25:35.135 INFO:tasks.rados.rados.0.smithi045.stdout:880: writing smithi04526579-16 from 3534742 to 3865643 tid 3 2024-04-26T00:25:35.135 INFO:tasks.rados.rados.0.smithi045.stdout:881: read oid 46 snap -1 2024-04-26T00:25:35.135 INFO:tasks.rados.rados.0.smithi045.stdout:881: expect (ObjNum 329 snap 96 seq_num 329) 2024-04-26T00:25:35.135 INFO:tasks.rados.rados.0.smithi045.stdout:882: read oid 5 snap -1 2024-04-26T00:25:35.135 INFO:tasks.rados.rados.0.smithi045.stdout:882: expect (ObjNum 340 snap 101 seq_num 340) 2024-04-26T00:25:35.135 INFO:tasks.rados.rados.0.smithi045.stdout:883: snap_remove snap 103 2024-04-26T00:25:35.988 INFO:tasks.rados.rados.0.smithi045.stdout:883: done (7 left) 2024-04-26T00:25:35.988 INFO:tasks.rados.rados.0.smithi045.stdout:884: write (excl) oid 39 current snap is 105 2024-04-26T00:25:35.988 INFO:tasks.rados.rados.0.smithi045.stdout:876: finishing write tid 2 to smithi04526579-40 2024-04-26T00:25:35.988 INFO:tasks.rados.rados.0.smithi045.stdout:876: finishing write tid 3 to smithi04526579-40 2024-04-26T00:25:35.988 INFO:tasks.rados.rados.0.smithi045.stdout:876: finishing write tid 4 to smithi04526579-40 2024-04-26T00:25:35.988 INFO:tasks.rados.rados.0.smithi045.stdout:876: finishing write tid 5 to smithi04526579-40 2024-04-26T00:25:35.988 INFO:tasks.rados.rados.0.smithi045.stdout:876: oid 40 updating version 0 to 311 2024-04-26T00:25:35.988 INFO:tasks.rados.rados.0.smithi045.stdout:876: oid 40 version 311 is already newer than 310 2024-04-26T00:25:35.988 INFO:tasks.rados.rados.0.smithi045.stdout:876: oid 40 updating version 311 to 313 2024-04-26T00:25:35.988 INFO:tasks.rados.rados.0.smithi045.stdout:876: oid 40 version 313 is already newer than 312 2024-04-26T00:25:35.988 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 313 (ObjNum 346 snap 105 seq_num 346) dirty exists 2024-04-26T00:25:35.989 INFO:tasks.rados.rados.0.smithi045.stdout:876: left oid 40 (ObjNum 346 snap 105 seq_num 346) 2024-04-26T00:25:35.989 INFO:tasks.rados.rados.0.smithi045.stdout:877: finishing write tid 1 to smithi04526579-24 2024-04-26T00:25:35.989 INFO:tasks.rados.rados.0.smithi045.stdout:877: finishing write tid 2 to smithi04526579-24 2024-04-26T00:25:35.989 INFO:tasks.rados.rados.0.smithi045.stdout:877: finishing write tid 3 to smithi04526579-24 2024-04-26T00:25:35.989 INFO:tasks.rados.rados.0.smithi045.stdout:877: finishing write tid 4 to smithi04526579-24 2024-04-26T00:25:35.989 INFO:tasks.rados.rados.0.smithi045.stdout:884: seq_num 350 ranges {682450=489388,1814580=496780,2878518=788028} 2024-04-26T00:25:35.999 INFO:tasks.rados.rados.0.smithi045.stdout:884: writing smithi04526579-39 from 682450 to 1171838 tid 1 2024-04-26T00:25:36.011 INFO:tasks.rados.rados.0.smithi045.stdout:884: writing smithi04526579-39 from 1814580 to 2311360 tid 2 2024-04-26T00:25:36.024 INFO:tasks.rados.rados.0.smithi045.stdout:884: writing smithi04526579-39 from 2878518 to 3666546 tid 3 2024-04-26T00:25:36.024 INFO:tasks.rados.rados.0.smithi045.stdout:876: done (7 left) 2024-04-26T00:25:36.024 INFO:tasks.rados.rados.0.smithi045.stdout:885: rollback oid 50 current snap is 105 2024-04-26T00:25:36.025 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 50 to 99 2024-04-26T00:25:36.025 INFO:tasks.rados.rados.0.smithi045.stdout:877: finishing write tid 5 to smithi04526579-24 2024-04-26T00:25:36.025 INFO:tasks.rados.rados.0.smithi045.stdout:877: finishing write tid 6 to smithi04526579-24 2024-04-26T00:25:36.025 INFO:tasks.rados.rados.0.smithi045.stdout:877: oid 24 updating version 0 to 615 2024-04-26T00:25:36.025 INFO:tasks.rados.rados.0.smithi045.stdout:877: oid 24 version 615 is already newer than 613 2024-04-26T00:25:36.025 INFO:tasks.rados.rados.0.smithi045.stdout:877: oid 24 updating version 615 to 616 2024-04-26T00:25:36.025 INFO:tasks.rados.rados.0.smithi045.stdout:877: oid 24 version 616 is already newer than 614 2024-04-26T00:25:36.025 INFO:tasks.rados.rados.0.smithi045.stdout:877: oid 24 version 616 is already newer than 612 2024-04-26T00:25:36.025 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 616 (ObjNum 347 snap 105 seq_num 347) dirty exists 2024-04-26T00:25:36.025 INFO:tasks.rados.rados.0.smithi045.stdout:877: left oid 24 (ObjNum 347 snap 105 seq_num 347) 2024-04-26T00:25:36.025 INFO:tasks.rados.rados.0.smithi045.stdout:878: finishing write tid 1 to smithi04526579-17 2024-04-26T00:25:36.025 INFO:tasks.rados.rados.0.smithi045.stdout:877: done (7 left) 2024-04-26T00:25:36.025 INFO:tasks.rados.rados.0.smithi045.stdout:886: write (excl) oid 24 current snap is 105 2024-04-26T00:25:36.025 INFO:tasks.rados.rados.0.smithi045.stdout:886: seq_num 351 ranges {699257=564637,1873062=640438,3305941=593316} 2024-04-26T00:25:36.037 INFO:tasks.rados.rados.0.smithi045.stdout:886: writing smithi04526579-24 from 699257 to 1263894 tid 1 2024-04-26T00:25:36.049 INFO:tasks.rados.rados.0.smithi045.stdout:886: writing smithi04526579-24 from 1873062 to 2513500 tid 2 2024-04-26T00:25:36.062 INFO:tasks.rados.rados.0.smithi045.stdout:886: writing smithi04526579-24 from 3305941 to 3899257 tid 3 2024-04-26T00:25:36.062 INFO:tasks.rados.rados.0.smithi045.stdout:887: write (excl) oid 43 current snap is 105 2024-04-26T00:25:36.062 INFO:tasks.rados.rados.0.smithi045.stdout:887: seq_num 352 ranges {716064=639886,1815191=500449,2817011=526925} 2024-04-26T00:25:36.073 INFO:tasks.rados.rados.0.smithi045.stdout:887: writing smithi04526579-43 from 716064 to 1355950 tid 1 2024-04-26T00:25:36.081 INFO:tasks.rados.rados.0.smithi045.stdout:887: writing smithi04526579-43 from 1815191 to 2315640 tid 2 2024-04-26T00:25:36.089 INFO:tasks.rados.rados.0.smithi045.stdout:887: writing smithi04526579-43 from 2817011 to 3343936 tid 3 2024-04-26T00:25:36.090 INFO:tasks.rados.rados.0.smithi045.stdout:888: snap_remove snap 91 2024-04-26T00:25:36.868 INFO:tasks.rados.rados.0.smithi045.stdout:888: done (9 left) 2024-04-26T00:25:36.868 INFO:tasks.rados.rados.0.smithi045.stdout:889: write oid 21 current snap is 105 2024-04-26T00:25:36.868 INFO:tasks.rados.rados.0.smithi045.stdout:878: finishing write tid 2 to smithi04526579-17 2024-04-26T00:25:36.868 INFO:tasks.rados.rados.0.smithi045.stdout:878: finishing write tid 3 to smithi04526579-17 2024-04-26T00:25:36.868 INFO:tasks.rados.rados.0.smithi045.stdout:878: finishing write tid 4 to smithi04526579-17 2024-04-26T00:25:36.868 INFO:tasks.rados.rados.0.smithi045.stdout:878: finishing write tid 5 to smithi04526579-17 2024-04-26T00:25:36.868 INFO:tasks.rados.rados.0.smithi045.stdout:878: oid 17 updating version 0 to 381 2024-04-26T00:25:36.868 INFO:tasks.rados.rados.0.smithi045.stdout:878: oid 17 version 381 is already newer than 380 2024-04-26T00:25:36.868 INFO:tasks.rados.rados.0.smithi045.stdout:878: oid 17 updating version 381 to 382 2024-04-26T00:25:36.868 INFO:tasks.rados.rados.0.smithi045.stdout:878: oid 17 updating version 382 to 383 2024-04-26T00:25:36.868 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 383 (ObjNum 348 snap 105 seq_num 348) dirty exists 2024-04-26T00:25:36.868 INFO:tasks.rados.rados.0.smithi045.stdout:878: left oid 17 (ObjNum 348 snap 105 seq_num 348) 2024-04-26T00:25:36.868 INFO:tasks.rados.rados.0.smithi045.stdout:879: expect (ObjNum 304 snap 87 seq_num 304) 2024-04-26T00:25:37.034 INFO:tasks.rados.rados.0.smithi045.stdout:880: finishing write tid 1 to smithi04526579-16 2024-04-26T00:25:37.034 INFO:tasks.rados.rados.0.smithi045.stdout:880: finishing write tid 2 to smithi04526579-16 2024-04-26T00:25:37.034 INFO:tasks.rados.rados.0.smithi045.stdout:880: finishing write tid 3 to smithi04526579-16 2024-04-26T00:25:37.034 INFO:tasks.rados.rados.0.smithi045.stdout:880: finishing write tid 4 to smithi04526579-16 2024-04-26T00:25:37.034 INFO:tasks.rados.rados.0.smithi045.stdout:880: finishing write tid 5 to smithi04526579-16 2024-04-26T00:25:37.035 INFO:tasks.rados.rados.0.smithi045.stdout:880: oid 16 updating version 0 to 152 2024-04-26T00:25:37.035 INFO:tasks.rados.rados.0.smithi045.stdout:880: oid 16 updating version 152 to 154 2024-04-26T00:25:37.035 INFO:tasks.rados.rados.0.smithi045.stdout:880: oid 16 version 154 is already newer than 153 2024-04-26T00:25:37.035 INFO:tasks.rados.rados.0.smithi045.stdout:880: oid 16 version 154 is already newer than 151 2024-04-26T00:25:37.035 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 154 (ObjNum 349 snap 105 seq_num 349) dirty exists 2024-04-26T00:25:37.035 INFO:tasks.rados.rados.0.smithi045.stdout:880: left oid 16 (ObjNum 349 snap 105 seq_num 349) 2024-04-26T00:25:37.035 INFO:tasks.rados.rados.0.smithi045.stdout:881: expect (ObjNum 329 snap 96 seq_num 329) 2024-04-26T00:25:37.151 INFO:tasks.rados.rados.0.smithi045.stdout:882: expect (ObjNum 340 snap 101 seq_num 340) 2024-04-26T00:25:37.321 INFO:tasks.rados.rados.0.smithi045.stdout:884: finishing write tid 1 to smithi04526579-39 2024-04-26T00:25:37.321 INFO:tasks.rados.rados.0.smithi045.stdout:884: finishing write tid 2 to smithi04526579-39 2024-04-26T00:25:37.322 INFO:tasks.rados.rados.0.smithi045.stdout:884: finishing write tid 3 to smithi04526579-39 2024-04-26T00:25:37.322 INFO:tasks.rados.rados.0.smithi045.stdout:884: finishing write tid 4 to smithi04526579-39 2024-04-26T00:25:37.322 INFO:tasks.rados.rados.0.smithi045.stdout:884: finishing write tid 5 to smithi04526579-39 2024-04-26T00:25:37.322 INFO:tasks.rados.rados.0.smithi045.stdout:884: oid 39 updating version 0 to 307 2024-04-26T00:25:37.322 INFO:tasks.rados.rados.0.smithi045.stdout:884: oid 39 version 307 is already newer than 305 2024-04-26T00:25:37.322 INFO:tasks.rados.rados.0.smithi045.stdout:884: oid 39 version 307 is already newer than 304 2024-04-26T00:25:37.322 INFO:tasks.rados.rados.0.smithi045.stdout:884: oid 39 version 307 is already newer than 306 2024-04-26T00:25:37.322 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 307 (ObjNum 350 snap 105 seq_num 350) dirty exists 2024-04-26T00:25:37.322 INFO:tasks.rados.rados.0.smithi045.stdout:884: left oid 39 (ObjNum 350 snap 105 seq_num 350) 2024-04-26T00:25:37.322 INFO:tasks.rados.rados.0.smithi045.stdout:885: finishing rollback tid 1 to smithi04526579-50 2024-04-26T00:25:37.322 INFO:tasks.rados.rados.0.smithi045.stdout:885: finishing rollback tid 2 to smithi04526579-50 2024-04-26T00:25:37.322 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 385 (ObjNum 136 snap 30 seq_num 136) dirty exists 2024-04-26T00:25:37.322 INFO:tasks.rados.rados.0.smithi045.stdout:886: finishing write tid 1 to smithi04526579-24 2024-04-26T00:25:37.323 INFO:tasks.rados.rados.0.smithi045.stdout:886: finishing write tid 2 to smithi04526579-24 2024-04-26T00:25:37.323 INFO:tasks.rados.rados.0.smithi045.stdout:886: finishing write tid 3 to smithi04526579-24 2024-04-26T00:25:37.323 INFO:tasks.rados.rados.0.smithi045.stdout:886: finishing write tid 4 to smithi04526579-24 2024-04-26T00:25:37.323 INFO:tasks.rados.rados.0.smithi045.stdout:889: seq_num 353 ranges {732871=715135,1873673=644107,3244434=688437} 2024-04-26T00:25:37.340 INFO:tasks.rados.rados.0.smithi045.stdout:889: writing smithi04526579-21 from 732871 to 1448006 tid 1 2024-04-26T00:25:37.354 INFO:tasks.rados.rados.0.smithi045.stdout:889: writing smithi04526579-21 from 1873673 to 2517780 tid 2 2024-04-26T00:25:37.373 INFO:tasks.rados.rados.0.smithi045.stdout:889: writing smithi04526579-21 from 3244434 to 3932871 tid 3 2024-04-26T00:25:37.373 INFO:tasks.rados.rados.0.smithi045.stdout:878: done (9 left) 2024-04-26T00:25:37.374 INFO:tasks.rados.rados.0.smithi045.stdout:879: done (8 left) 2024-04-26T00:25:37.374 INFO:tasks.rados.rados.0.smithi045.stdout:880: done (7 left) 2024-04-26T00:25:37.374 INFO:tasks.rados.rados.0.smithi045.stdout:881: done (6 left) 2024-04-26T00:25:37.374 INFO:tasks.rados.rados.0.smithi045.stdout:882: done (5 left) 2024-04-26T00:25:37.374 INFO:tasks.rados.rados.0.smithi045.stdout:884: done (4 left) 2024-04-26T00:25:37.374 INFO:tasks.rados.rados.0.smithi045.stdout:885: done (3 left) 2024-04-26T00:25:37.374 INFO:tasks.rados.rados.0.smithi045.stdout:890: read oid 23 snap -1 2024-04-26T00:25:37.374 INFO:tasks.rados.rados.0.smithi045.stdout:890: expect (ObjNum 316 snap 93 seq_num 316) 2024-04-26T00:25:37.374 INFO:tasks.rados.rados.0.smithi045.stdout:891: write oid 10 current snap is 105 2024-04-26T00:25:37.374 INFO:tasks.rados.rados.0.smithi045.stdout:886: finishing write tid 5 to smithi04526579-24 2024-04-26T00:25:37.374 INFO:tasks.rados.rados.0.smithi045.stdout:886: oid 24 updating version 0 to 620 2024-04-26T00:25:37.374 INFO:tasks.rados.rados.0.smithi045.stdout:886: oid 24 updating version 620 to 622 2024-04-26T00:25:37.374 INFO:tasks.rados.rados.0.smithi045.stdout:886: oid 24 version 622 is already newer than 619 2024-04-26T00:25:37.374 INFO:tasks.rados.rados.0.smithi045.stdout:886: oid 24 version 622 is already newer than 621 2024-04-26T00:25:37.375 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 622 (ObjNum 351 snap 105 seq_num 351) dirty exists 2024-04-26T00:25:37.375 INFO:tasks.rados.rados.0.smithi045.stdout:886: left oid 24 (ObjNum 351 snap 105 seq_num 351) 2024-04-26T00:25:37.375 INFO:tasks.rados.rados.0.smithi045.stdout:887: finishing write tid 1 to smithi04526579-43 2024-04-26T00:25:37.375 INFO:tasks.rados.rados.0.smithi045.stdout:887: finishing write tid 2 to smithi04526579-43 2024-04-26T00:25:37.375 INFO:tasks.rados.rados.0.smithi045.stdout:887: finishing write tid 3 to smithi04526579-43 2024-04-26T00:25:37.375 INFO:tasks.rados.rados.0.smithi045.stdout:887: finishing write tid 4 to smithi04526579-43 2024-04-26T00:25:37.375 INFO:tasks.rados.rados.0.smithi045.stdout:887: finishing write tid 5 to smithi04526579-43 2024-04-26T00:25:37.375 INFO:tasks.rados.rados.0.smithi045.stdout:887: oid 43 updating version 0 to 393 2024-04-26T00:25:37.375 INFO:tasks.rados.rados.0.smithi045.stdout:887: oid 43 updating version 393 to 394 2024-04-26T00:25:37.375 INFO:tasks.rados.rados.0.smithi045.stdout:887: oid 43 version 394 is already newer than 391 2024-04-26T00:25:37.375 INFO:tasks.rados.rados.0.smithi045.stdout:887: oid 43 version 394 is already newer than 392 2024-04-26T00:25:37.375 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 394 (ObjNum 352 snap 105 seq_num 352) dirty exists 2024-04-26T00:25:37.375 INFO:tasks.rados.rados.0.smithi045.stdout:887: left oid 43 (ObjNum 352 snap 105 seq_num 352) 2024-04-26T00:25:37.375 INFO:tasks.rados.rados.0.smithi045.stdout:889: finishing write tid 1 to smithi04526579-21 2024-04-26T00:25:37.375 INFO:tasks.rados.rados.0.smithi045.stdout:889: finishing write tid 2 to smithi04526579-21 2024-04-26T00:25:37.376 INFO:tasks.rados.rados.0.smithi045.stdout:891: seq_num 354 ranges {749678=790384,2332155=504118,3271857=549469} 2024-04-26T00:25:37.393 INFO:tasks.rados.rados.0.smithi045.stdout:891: writing smithi04526579-10 from 749678 to 1540062 tid 1 2024-04-26T00:25:37.406 INFO:tasks.rados.rados.0.smithi045.stdout:891: writing smithi04526579-10 from 2332155 to 2836273 tid 2 2024-04-26T00:25:37.416 INFO:tasks.rados.rados.0.smithi045.stdout:891: writing smithi04526579-10 from 3271857 to 3821326 tid 3 2024-04-26T00:25:37.416 INFO:tasks.rados.rados.0.smithi045.stdout:886: done (4 left) 2024-04-26T00:25:37.416 INFO:tasks.rados.rados.0.smithi045.stdout:887: done (3 left) 2024-04-26T00:25:37.416 INFO:tasks.rados.rados.0.smithi045.stdout:892: copy_from oid 1 from oid 50 current snap is 105 2024-04-26T00:25:37.416 INFO:tasks.rados.rados.0.smithi045.stdout:889: finishing write tid 3 to smithi04526579-21 2024-04-26T00:25:37.416 INFO:tasks.rados.rados.0.smithi045.stdout:889: finishing write tid 4 to smithi04526579-21 2024-04-26T00:25:37.416 INFO:tasks.rados.rados.0.smithi045.stdout:889: finishing write tid 5 to smithi04526579-21 2024-04-26T00:25:37.416 INFO:tasks.rados.rados.0.smithi045.stdout:889: oid 21 updating version 0 to 33 2024-04-26T00:25:37.416 INFO:tasks.rados.rados.0.smithi045.stdout:889: oid 21 updating version 33 to 35 2024-04-26T00:25:37.416 INFO:tasks.rados.rados.0.smithi045.stdout:889: oid 21 updating version 35 to 36 2024-04-26T00:25:37.417 INFO:tasks.rados.rados.0.smithi045.stdout:889: oid 21 version 36 is already newer than 34 2024-04-26T00:25:37.417 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 36 (ObjNum 353 snap 105 seq_num 353) dirty exists 2024-04-26T00:25:37.417 INFO:tasks.rados.rados.0.smithi045.stdout:889: left oid 21 (ObjNum 353 snap 105 seq_num 353) 2024-04-26T00:25:37.417 INFO:tasks.rados.rados.0.smithi045.stdout:890: expect (ObjNum 316 snap 93 seq_num 316) 2024-04-26T00:25:37.640 INFO:tasks.rados.rados.0.smithi045.stdout:891: finishing write tid 1 to smithi04526579-10 2024-04-26T00:25:37.640 INFO:tasks.rados.rados.0.smithi045.stdout:891: finishing write tid 2 to smithi04526579-10 2024-04-26T00:25:37.640 INFO:tasks.rados.rados.0.smithi045.stdout:891: finishing write tid 3 to smithi04526579-10 2024-04-26T00:25:37.640 INFO:tasks.rados.rados.0.smithi045.stdout:891: finishing write tid 4 to smithi04526579-10 2024-04-26T00:25:37.640 INFO:tasks.rados.rados.0.smithi045.stdout:891: finishing write tid 5 to smithi04526579-10 2024-04-26T00:25:37.640 INFO:tasks.rados.rados.0.smithi045.stdout:891: oid 10 updating version 0 to 404 2024-04-26T00:25:37.640 INFO:tasks.rados.rados.0.smithi045.stdout:891: oid 10 updating version 404 to 405 2024-04-26T00:25:37.640 INFO:tasks.rados.rados.0.smithi045.stdout:891: oid 10 version 405 is already newer than 403 2024-04-26T00:25:37.640 INFO:tasks.rados.rados.0.smithi045.stdout:891: oid 10 version 405 is already newer than 402 2024-04-26T00:25:37.640 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 405 (ObjNum 354 snap 105 seq_num 354) dirty exists 2024-04-26T00:25:37.640 INFO:tasks.rados.rados.0.smithi045.stdout:891: left oid 10 (ObjNum 354 snap 105 seq_num 354) 2024-04-26T00:25:37.640 INFO:tasks.rados.rados.0.smithi045.stdout:892: finishing copy_from racing read to smithi04526579-1 2024-04-26T00:25:37.641 INFO:tasks.rados.rados.0.smithi045.stdout:892: finishing copy_from to smithi04526579-1 2024-04-26T00:25:37.641 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 385 (ObjNum 136 snap 30 seq_num 136) dirty exists 2024-04-26T00:25:37.641 INFO:tasks.rados.rados.0.smithi045.stdout:889: done (3 left) 2024-04-26T00:25:37.641 INFO:tasks.rados.rados.0.smithi045.stdout:890: done (2 left) 2024-04-26T00:25:37.641 INFO:tasks.rados.rados.0.smithi045.stdout:891: done (1 left) 2024-04-26T00:25:37.641 INFO:tasks.rados.rados.0.smithi045.stdout:892: done (0 left) 2024-04-26T00:25:37.641 INFO:tasks.rados.rados.0.smithi045.stdout:893: write (excl) oid 29 current snap is 105 2024-04-26T00:25:37.641 INFO:tasks.rados.rados.0.smithi045.stdout:893: seq_num 355 ranges {766485=465633,1990637=647776,3299280=667205} 2024-04-26T00:25:37.651 INFO:tasks.rados.rados.0.smithi045.stdout:893: writing smithi04526579-29 from 766485 to 1232118 tid 1 2024-04-26T00:25:37.665 INFO:tasks.rados.rados.0.smithi045.stdout:893: writing smithi04526579-29 from 1990637 to 2638413 tid 2 2024-04-26T00:25:37.679 INFO:tasks.rados.rados.0.smithi045.stdout:893: writing smithi04526579-29 from 3299280 to 3966485 tid 3 2024-04-26T00:25:37.679 INFO:tasks.rados.rados.0.smithi045.stdout:894: read oid 36 snap 70 2024-04-26T00:25:37.679 INFO:tasks.rados.rados.0.smithi045.stdout:894: expect deleted 2024-04-26T00:25:37.679 INFO:tasks.rados.rados.0.smithi045.stdout:895: write (excl) oid 20 current snap is 105 2024-04-26T00:25:37.679 INFO:tasks.rados.rados.0.smithi045.stdout:895: seq_num 356 ranges {783292=540882,1932766=507787,3210350=572013} 2024-04-26T00:25:37.692 INFO:tasks.rados.rados.0.smithi045.stdout:895: writing smithi04526579-20 from 783292 to 1324174 tid 1 2024-04-26T00:25:37.703 INFO:tasks.rados.rados.0.smithi045.stdout:895: writing smithi04526579-20 from 1932766 to 2440553 tid 2 2024-04-26T00:25:37.716 INFO:tasks.rados.rados.0.smithi045.stdout:895: writing smithi04526579-20 from 3210350 to 3782363 tid 3 2024-04-26T00:25:37.716 INFO:tasks.rados.rados.0.smithi045.stdout:896: rollback oid 26 current snap is 105 2024-04-26T00:25:37.716 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 26 to 98 2024-04-26T00:25:37.716 INFO:tasks.rados.rados.0.smithi045.stdout:893: finishing write tid 1 to smithi04526579-29 2024-04-26T00:25:37.716 INFO:tasks.rados.rados.0.smithi045.stdout:893: finishing write tid 2 to smithi04526579-29 2024-04-26T00:25:37.717 INFO:tasks.rados.rados.0.smithi045.stdout:897: copy_from oid 48 from oid 49 current snap is 105 2024-04-26T00:25:37.717 INFO:tasks.rados.rados.0.smithi045.stdout:893: finishing write tid 3 to smithi04526579-29 2024-04-26T00:25:37.717 INFO:tasks.rados.rados.0.smithi045.stdout:893: finishing write tid 4 to smithi04526579-29 2024-04-26T00:25:37.717 INFO:tasks.rados.rados.0.smithi045.stdout:893: finishing write tid 5 to smithi04526579-29 2024-04-26T00:25:37.717 INFO:tasks.rados.rados.0.smithi045.stdout:893: oid 29 updating version 0 to 317 2024-04-26T00:25:37.717 INFO:tasks.rados.rados.0.smithi045.stdout:893: oid 29 version 317 is already newer than 315 2024-04-26T00:25:37.717 INFO:tasks.rados.rados.0.smithi045.stdout:893: oid 29 version 317 is already newer than 316 2024-04-26T00:25:37.717 INFO:tasks.rados.rados.0.smithi045.stdout:893: oid 29 version 317 is already newer than 314 2024-04-26T00:25:37.717 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 317 (ObjNum 355 snap 105 seq_num 355) dirty exists 2024-04-26T00:25:37.717 INFO:tasks.rados.rados.0.smithi045.stdout:893: left oid 29 (ObjNum 355 snap 105 seq_num 355) 2024-04-26T00:25:37.717 INFO:tasks.rados.rados.0.smithi045.stdout:895: finishing write tid 1 to smithi04526579-20 2024-04-26T00:25:37.717 INFO:tasks.rados.rados.0.smithi045.stdout:895: finishing write tid 2 to smithi04526579-20 2024-04-26T00:25:37.717 INFO:tasks.rados.rados.0.smithi045.stdout:893: done (4 left) 2024-04-26T00:25:37.717 INFO:tasks.rados.rados.0.smithi045.stdout:894: done (3 left) 2024-04-26T00:25:37.717 INFO:tasks.rados.rados.0.smithi045.stdout:898: read oid 14 snap 97 2024-04-26T00:25:37.718 INFO:tasks.rados.rados.0.smithi045.stdout:898: expect deleted 2024-04-26T00:25:37.718 INFO:tasks.rados.rados.0.smithi045.stdout:899: read oid 4 snap 96 2024-04-26T00:25:37.718 INFO:tasks.rados.rados.0.smithi045.stdout:899: expect (ObjNum 224 snap 59 seq_num 224) 2024-04-26T00:25:37.718 INFO:tasks.rados.rados.0.smithi045.stdout:900: snap_remove snap 102 2024-04-26T00:25:37.998 INFO:tasks.rados.rados.0.smithi045.stdout:900: done (5 left) 2024-04-26T00:25:37.998 INFO:tasks.rados.rados.0.smithi045.stdout:901: snap_create 2024-04-26T00:25:37.998 INFO:tasks.rados.rados.0.smithi045.stdout:896: finishing rollback tid 1 to smithi04526579-26 2024-04-26T00:25:37.999 INFO:tasks.rados.rados.0.smithi045.stdout:896: finishing rollback tid 2 to smithi04526579-26 2024-04-26T00:25:37.999 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 534 (ObjNum 254 snap 64 seq_num 254) dirty exists 2024-04-26T00:25:37.999 INFO:tasks.rados.rados.0.smithi045.stdout:895: finishing write tid 3 to smithi04526579-20 2024-04-26T00:25:37.999 INFO:tasks.rados.rados.0.smithi045.stdout:895: finishing write tid 4 to smithi04526579-20 2024-04-26T00:25:37.999 INFO:tasks.rados.rados.0.smithi045.stdout:895: finishing write tid 5 to smithi04526579-20 2024-04-26T00:25:37.999 INFO:tasks.rados.rados.0.smithi045.stdout:895: oid 20 updating version 0 to 312 2024-04-26T00:25:37.999 INFO:tasks.rados.rados.0.smithi045.stdout:895: oid 20 updating version 312 to 313 2024-04-26T00:25:37.999 INFO:tasks.rados.rados.0.smithi045.stdout:895: oid 20 updating version 313 to 314 2024-04-26T00:25:37.999 INFO:tasks.rados.rados.0.smithi045.stdout:895: oid 20 version 314 is already newer than 311 2024-04-26T00:25:37.999 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 314 (ObjNum 356 snap 105 seq_num 356) dirty exists 2024-04-26T00:25:37.999 INFO:tasks.rados.rados.0.smithi045.stdout:895: left oid 20 (ObjNum 356 snap 105 seq_num 356) 2024-04-26T00:25:37.999 INFO:tasks.rados.rados.0.smithi045.stdout:899: expect (ObjNum 224 snap 59 seq_num 224) 2024-04-26T00:25:38.123 INFO:tasks.rados.rados.0.smithi045.stdout:897: finishing copy_from racing read to smithi04526579-48 2024-04-26T00:25:38.123 INFO:tasks.rados.rados.0.smithi045.stdout:897: finishing copy_from to smithi04526579-48 2024-04-26T00:25:38.123 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 412 (ObjNum 324 snap 95 seq_num 324) dirty exists 2024-04-26T00:25:39.005 INFO:tasks.rados.rados.0.smithi045.stdout:895: done (5 left) 2024-04-26T00:25:39.005 INFO:tasks.rados.rados.0.smithi045.stdout:896: done (4 left) 2024-04-26T00:25:39.005 INFO:tasks.rados.rados.0.smithi045.stdout:897: done (3 left) 2024-04-26T00:25:39.005 INFO:tasks.rados.rados.0.smithi045.stdout:898: done (2 left) 2024-04-26T00:25:39.005 INFO:tasks.rados.rados.0.smithi045.stdout:899: done (1 left) 2024-04-26T00:25:39.005 INFO:tasks.rados.rados.0.smithi045.stdout:901: done (0 left) 2024-04-26T00:25:39.005 INFO:tasks.rados.rados.0.smithi045.stdout:902: read oid 39 snap -1 2024-04-26T00:25:39.006 INFO:tasks.rados.rados.0.smithi045.stdout:902: expect (ObjNum 350 snap 105 seq_num 350) 2024-04-26T00:25:39.006 INFO:tasks.rados.rados.0.smithi045.stdout:903: snap_create 2024-04-26T00:25:39.021 INFO:tasks.rados.rados.0.smithi045.stdout:902: expect (ObjNum 350 snap 105 seq_num 350) 2024-04-26T00:25:40.012 INFO:tasks.rados.rados.0.smithi045.stdout:902: done (1 left) 2024-04-26T00:25:40.013 INFO:tasks.rados.rados.0.smithi045.stdout:903: done (0 left) 2024-04-26T00:25:40.013 INFO:tasks.rados.rados.0.smithi045.stdout:904: write (excl) oid 1 current snap is 107 2024-04-26T00:25:40.013 INFO:tasks.rados.rados.0.smithi045.stdout:904: seq_num 357 ranges {400099=616131,1591248=408851} 2024-04-26T00:25:40.024 INFO:tasks.rados.rados.0.smithi045.stdout:904: writing smithi04526579-1 from 400099 to 1016230 tid 1 2024-04-26T00:25:40.037 INFO:tasks.rados.rados.0.smithi045.stdout:904: writing smithi04526579-1 from 1591248 to 2000099 tid 2 2024-04-26T00:25:40.037 INFO:tasks.rados.rados.0.smithi045.stdout:905: read oid 27 snap -1 2024-04-26T00:25:40.037 INFO:tasks.rados.rados.0.smithi045.stdout:905: expect (ObjNum 246 snap 63 seq_num 246) 2024-04-26T00:25:40.038 INFO:tasks.rados.rados.0.smithi045.stdout:906: read oid 21 snap -1 2024-04-26T00:25:40.038 INFO:tasks.rados.rados.0.smithi045.stdout:906: expect (ObjNum 353 snap 105 seq_num 353) 2024-04-26T00:25:40.038 INFO:tasks.rados.rados.0.smithi045.stdout:907: write oid 33 current snap is 107 2024-04-26T00:25:40.038 INFO:tasks.rados.rados.0.smithi045.stdout:907: seq_num 358 ranges {416906=407733,1366083=511456} 2024-04-26T00:25:40.048 INFO:tasks.rados.rados.0.smithi045.stdout:907: writing smithi04526579-33 from 416906 to 824639 tid 1 2024-04-26T00:25:40.061 INFO:tasks.rados.rados.0.smithi045.stdout:907: writing smithi04526579-33 from 1366083 to 1877539 tid 2 2024-04-26T00:25:40.061 INFO:tasks.rados.rados.0.smithi045.stdout:908: snap_remove snap 105 2024-04-26T00:25:41.016 INFO:tasks.rados.rados.0.smithi045.stdout:908: done (4 left) 2024-04-26T00:25:41.016 INFO:tasks.rados.rados.0.smithi045.stdout:909: write (excl) oid 30 current snap is 107 2024-04-26T00:25:41.016 INFO:tasks.rados.rados.0.smithi045.stdout:909: seq_num 359 ranges {433713=482982,1424565=609148} 2024-04-26T00:25:41.027 INFO:tasks.rados.rados.0.smithi045.stdout:909: writing smithi04526579-30 from 433713 to 916695 tid 1 2024-04-26T00:25:41.040 INFO:tasks.rados.rados.0.smithi045.stdout:909: writing smithi04526579-30 from 1424565 to 2033713 tid 2 2024-04-26T00:25:41.040 INFO:tasks.rados.rados.0.smithi045.stdout:910: snap_remove snap 96 2024-04-26T00:25:41.870 INFO:tasks.rados.rados.0.smithi045.stdout:910: done (5 left) 2024-04-26T00:25:41.871 INFO:tasks.rados.rados.0.smithi045.stdout:911: rollback oid 43 current snap is 107 2024-04-26T00:25:41.871 INFO:tasks.rados.rados.0.smithi045.stdout:904: finishing write tid 1 to smithi04526579-1 2024-04-26T00:25:41.871 INFO:tasks.rados.rados.0.smithi045.stdout:904: finishing write tid 2 to smithi04526579-1 2024-04-26T00:25:41.871 INFO:tasks.rados.rados.0.smithi045.stdout:904: finishing write tid 3 to smithi04526579-1 2024-04-26T00:25:41.871 INFO:tasks.rados.rados.0.smithi045.stdout:904: finishing write tid 4 to smithi04526579-1 2024-04-26T00:25:41.871 INFO:tasks.rados.rados.0.smithi045.stdout:904: oid 1 updating version 0 to 539 2024-04-26T00:25:41.871 INFO:tasks.rados.rados.0.smithi045.stdout:904: oid 1 version 539 is already newer than 538 2024-04-26T00:25:41.871 INFO:tasks.rados.rados.0.smithi045.stdout:904: oid 1 updating version 539 to 540 2024-04-26T00:25:41.871 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 540 (ObjNum 357 snap 107 seq_num 357) dirty exists 2024-04-26T00:25:41.871 INFO:tasks.rados.rados.0.smithi045.stdout:904: left oid 1 (ObjNum 357 snap 107 seq_num 357) 2024-04-26T00:25:41.871 INFO:tasks.rados.rados.0.smithi045.stdout:905: expect (ObjNum 246 snap 63 seq_num 246) 2024-04-26T00:25:41.977 INFO:tasks.rados.rados.0.smithi045.stdout:906: expect (ObjNum 353 snap 105 seq_num 353) 2024-04-26T00:25:42.102 INFO:tasks.rados.rados.0.smithi045.stdout:907: finishing write tid 1 to smithi04526579-33 2024-04-26T00:25:42.103 INFO:tasks.rados.rados.0.smithi045.stdout:907: finishing write tid 2 to smithi04526579-33 2024-04-26T00:25:42.103 INFO:tasks.rados.rados.0.smithi045.stdout:907: finishing write tid 3 to smithi04526579-33 2024-04-26T00:25:42.103 INFO:tasks.rados.rados.0.smithi045.stdout:907: finishing write tid 4 to smithi04526579-33 2024-04-26T00:25:42.103 INFO:tasks.rados.rados.0.smithi045.stdout:907: oid 33 updating version 0 to 630 2024-04-26T00:25:42.103 INFO:tasks.rados.rados.0.smithi045.stdout:907: oid 33 updating version 630 to 632 2024-04-26T00:25:42.103 INFO:tasks.rados.rados.0.smithi045.stdout:907: oid 33 version 632 is already newer than 631 2024-04-26T00:25:42.103 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 632 (ObjNum 358 snap 107 seq_num 358) dirty exists 2024-04-26T00:25:42.103 INFO:tasks.rados.rados.0.smithi045.stdout:907: left oid 33 (ObjNum 358 snap 107 seq_num 358) 2024-04-26T00:25:42.103 INFO:tasks.rados.rados.0.smithi045.stdout:909: finishing write tid 1 to smithi04526579-30 2024-04-26T00:25:42.103 INFO:tasks.rados.rados.0.smithi045.stdout:909: finishing write tid 2 to smithi04526579-30 2024-04-26T00:25:42.103 INFO:tasks.rados.rados.0.smithi045.stdout:909: finishing write tid 3 to smithi04526579-30 2024-04-26T00:25:42.104 INFO:tasks.rados.rados.0.smithi045.stdout:909: finishing write tid 4 to smithi04526579-30 2024-04-26T00:25:42.104 INFO:tasks.rados.rados.0.smithi045.stdout:909: oid 30 updating version 0 to 309 2024-04-26T00:25:42.104 INFO:tasks.rados.rados.0.smithi045.stdout:909: oid 30 version 309 is already newer than 307 2024-04-26T00:25:42.104 INFO:tasks.rados.rados.0.smithi045.stdout:909: oid 30 version 309 is already newer than 308 2024-04-26T00:25:42.104 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 309 (ObjNum 359 snap 107 seq_num 359) dirty exists 2024-04-26T00:25:42.104 INFO:tasks.rados.rados.0.smithi045.stdout:909: left oid 30 (ObjNum 359 snap 107 seq_num 359) 2024-04-26T00:25:42.104 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 43 to 93 2024-04-26T00:25:42.104 INFO:tasks.rados.rados.0.smithi045.stdout:904: done (5 left) 2024-04-26T00:25:42.104 INFO:tasks.rados.rados.0.smithi045.stdout:905: done (4 left) 2024-04-26T00:25:42.104 INFO:tasks.rados.rados.0.smithi045.stdout:906: done (3 left) 2024-04-26T00:25:42.104 INFO:tasks.rados.rados.0.smithi045.stdout:907: done (2 left) 2024-04-26T00:25:42.104 INFO:tasks.rados.rados.0.smithi045.stdout:909: done (1 left) 2024-04-26T00:25:42.104 INFO:tasks.rados.rados.0.smithi045.stdout:912: write (excl) oid 11 current snap is 107 2024-04-26T00:25:42.104 INFO:tasks.rados.rados.0.smithi045.stdout:912: seq_num 360 ranges {450520=558231,1483047=515125} 2024-04-26T00:25:42.108 INFO:tasks.rados.rados.0.smithi045.stdout:912: writing smithi04526579-11 from 450520 to 1008751 tid 1 2024-04-26T00:25:42.114 INFO:tasks.rados.rados.0.smithi045.stdout:912: writing smithi04526579-11 from 1483047 to 1998172 tid 2 2024-04-26T00:25:42.114 INFO:tasks.rados.rados.0.smithi045.stdout:913: write (excl) oid 6 current snap is 107 2024-04-26T00:25:42.114 INFO:tasks.rados.rados.0.smithi045.stdout:913: seq_num 361 ranges {467327=633480,1825176=242151} 2024-04-26T00:25:42.121 INFO:tasks.rados.rados.0.smithi045.stdout:913: writing smithi04526579-6 from 467327 to 1100807 tid 1 2024-04-26T00:25:42.127 INFO:tasks.rados.rados.0.smithi045.stdout:913: writing smithi04526579-6 from 1825176 to 2067327 tid 2 2024-04-26T00:25:42.127 INFO:tasks.rados.rados.0.smithi045.stdout:914: write oid 40 current snap is 107 2024-04-26T00:25:42.127 INFO:tasks.rados.rados.0.smithi045.stdout:911: finishing rollback tid 0 to smithi04526579-43 2024-04-26T00:25:42.128 INFO:tasks.rados.rados.0.smithi045.stdout:914: seq_num 362 ranges {484134=708729,1883658=200476} 2024-04-26T00:25:42.134 INFO:tasks.rados.rados.0.smithi045.stdout:914: writing smithi04526579-40 from 484134 to 1192863 tid 1 2024-04-26T00:25:42.140 INFO:tasks.rados.rados.0.smithi045.stdout:914: writing smithi04526579-40 from 1883658 to 2084134 tid 2 2024-04-26T00:25:42.141 INFO:tasks.rados.rados.0.smithi045.stdout:915: read oid 4 snap -1 2024-04-26T00:25:42.141 INFO:tasks.rados.rados.0.smithi045.stdout:915: expect (ObjNum 224 snap 59 seq_num 224) 2024-04-26T00:25:42.141 INFO:tasks.rados.rados.0.smithi045.stdout:916: snap_create 2024-04-26T00:25:42.141 INFO:tasks.rados.rados.0.smithi045.stdout:911: finishing rollback tid 1 to smithi04526579-43 2024-04-26T00:25:42.141 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 420 (ObjNum 81 snap 0 seq_num 1) dirty dne 2024-04-26T00:25:42.141 INFO:tasks.rados.rados.0.smithi045.stdout:912: finishing write tid 1 to smithi04526579-11 2024-04-26T00:25:42.141 INFO:tasks.rados.rados.0.smithi045.stdout:912: finishing write tid 2 to smithi04526579-11 2024-04-26T00:25:42.141 INFO:tasks.rados.rados.0.smithi045.stdout:912: finishing write tid 3 to smithi04526579-11 2024-04-26T00:25:42.141 INFO:tasks.rados.rados.0.smithi045.stdout:912: finishing write tid 4 to smithi04526579-11 2024-04-26T00:25:42.141 INFO:tasks.rados.rados.0.smithi045.stdout:912: oid 11 updating version 0 to 374 2024-04-26T00:25:42.141 INFO:tasks.rados.rados.0.smithi045.stdout:912: oid 11 updating version 374 to 376 2024-04-26T00:25:42.141 INFO:tasks.rados.rados.0.smithi045.stdout:912: oid 11 version 376 is already newer than 375 2024-04-26T00:25:42.142 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 376 (ObjNum 360 snap 107 seq_num 360) dirty exists 2024-04-26T00:25:42.142 INFO:tasks.rados.rados.0.smithi045.stdout:912: left oid 11 (ObjNum 360 snap 107 seq_num 360) 2024-04-26T00:25:42.142 INFO:tasks.rados.rados.0.smithi045.stdout:913: finishing write tid 1 to smithi04526579-6 2024-04-26T00:25:42.142 INFO:tasks.rados.rados.0.smithi045.stdout:913: finishing write tid 2 to smithi04526579-6 2024-04-26T00:25:42.142 INFO:tasks.rados.rados.0.smithi045.stdout:913: finishing write tid 3 to smithi04526579-6 2024-04-26T00:25:42.142 INFO:tasks.rados.rados.0.smithi045.stdout:913: finishing write tid 4 to smithi04526579-6 2024-04-26T00:25:42.142 INFO:tasks.rados.rados.0.smithi045.stdout:913: oid 6 updating version 0 to 312 2024-04-26T00:25:42.142 INFO:tasks.rados.rados.0.smithi045.stdout:913: oid 6 version 312 is already newer than 311 2024-04-26T00:25:42.142 INFO:tasks.rados.rados.0.smithi045.stdout:913: oid 6 version 312 is already newer than 310 2024-04-26T00:25:42.142 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 312 (ObjNum 361 snap 107 seq_num 361) dirty exists 2024-04-26T00:25:42.142 INFO:tasks.rados.rados.0.smithi045.stdout:913: left oid 6 (ObjNum 361 snap 107 seq_num 361) 2024-04-26T00:25:42.149 INFO:tasks.rados.rados.0.smithi045.stdout:915: expect (ObjNum 224 snap 59 seq_num 224) 2024-04-26T00:25:42.274 INFO:tasks.rados.rados.0.smithi045.stdout:914: finishing write tid 1 to smithi04526579-40 2024-04-26T00:25:42.274 INFO:tasks.rados.rados.0.smithi045.stdout:914: finishing write tid 2 to smithi04526579-40 2024-04-26T00:25:42.274 INFO:tasks.rados.rados.0.smithi045.stdout:914: finishing write tid 3 to smithi04526579-40 2024-04-26T00:25:42.274 INFO:tasks.rados.rados.0.smithi045.stdout:914: finishing write tid 4 to smithi04526579-40 2024-04-26T00:25:42.274 INFO:tasks.rados.rados.0.smithi045.stdout:914: oid 40 updating version 0 to 314 2024-04-26T00:25:42.274 INFO:tasks.rados.rados.0.smithi045.stdout:914: oid 40 updating version 314 to 316 2024-04-26T00:25:42.275 INFO:tasks.rados.rados.0.smithi045.stdout:914: oid 40 version 316 is already newer than 315 2024-04-26T00:25:42.275 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 316 (ObjNum 362 snap 107 seq_num 362) dirty exists 2024-04-26T00:25:42.275 INFO:tasks.rados.rados.0.smithi045.stdout:914: left oid 40 (ObjNum 362 snap 107 seq_num 362) 2024-04-26T00:25:43.039 INFO:tasks.rados.rados.0.smithi045.stdout:911: done (5 left) 2024-04-26T00:25:43.040 INFO:tasks.rados.rados.0.smithi045.stdout:912: done (4 left) 2024-04-26T00:25:43.040 INFO:tasks.rados.rados.0.smithi045.stdout:913: done (3 left) 2024-04-26T00:25:43.040 INFO:tasks.rados.rados.0.smithi045.stdout:914: done (2 left) 2024-04-26T00:25:43.040 INFO:tasks.rados.rados.0.smithi045.stdout:915: done (1 left) 2024-04-26T00:25:43.040 INFO:tasks.rados.rados.0.smithi045.stdout:916: done (0 left) 2024-04-26T00:25:43.040 INFO:tasks.rados.rados.0.smithi045.stdout:917: rollback oid 16 current snap is 108 2024-04-26T00:25:43.040 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 16 to 86 2024-04-26T00:25:43.040 INFO:tasks.rados.rados.0.smithi045.stdout:918: write oid 29 current snap is 108 2024-04-26T00:25:43.040 INFO:tasks.rados.rados.0.smithi045.stdout:918: seq_num 363 ranges {500941=783978,1942140=158801} 2024-04-26T00:25:43.050 INFO:tasks.rados.rados.0.smithi045.stdout:918: writing smithi04526579-29 from 500941 to 1284919 tid 1 2024-04-26T00:25:43.057 INFO:tasks.rados.rados.0.smithi045.stdout:918: writing smithi04526579-29 from 1942140 to 2100941 tid 2 2024-04-26T00:25:43.057 INFO:tasks.rados.rados.0.smithi045.stdout:919: rollback oid 46 current snap is 108 2024-04-26T00:25:43.057 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 46 to 107 2024-04-26T00:25:43.057 INFO:tasks.rados.rados.0.smithi045.stdout:917: finishing rollback tid 0 to smithi04526579-16 2024-04-26T00:25:43.058 INFO:tasks.rados.rados.0.smithi045.stdout:917: finishing rollback tid 1 to smithi04526579-16 2024-04-26T00:25:43.058 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 157 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-04-26T00:25:43.058 INFO:tasks.rados.rados.0.smithi045.stdout:917: done (2 left) 2024-04-26T00:25:43.058 INFO:tasks.rados.rados.0.smithi045.stdout:920: snap_remove snap 106 2024-04-26T00:25:44.043 INFO:tasks.rados.rados.0.smithi045.stdout:920: done (2 left) 2024-04-26T00:25:44.044 INFO:tasks.rados.rados.0.smithi045.stdout:921: read oid 20 snap -1 2024-04-26T00:25:44.044 INFO:tasks.rados.rados.0.smithi045.stdout:921: expect (ObjNum 356 snap 105 seq_num 356) 2024-04-26T00:25:44.044 INFO:tasks.rados.rados.0.smithi045.stdout:922: write (excl) oid 7 current snap is 108 2024-04-26T00:25:44.044 INFO:tasks.rados.rados.0.smithi045.stdout:922: seq_num 364 ranges {517748=459227,1600622=406110} 2024-04-26T00:25:44.059 INFO:tasks.rados.rados.0.smithi045.stdout:922: writing smithi04526579-7 from 517748 to 976975 tid 1 2024-04-26T00:25:44.073 INFO:tasks.rados.rados.0.smithi045.stdout:922: writing smithi04526579-7 from 1600622 to 2006732 tid 2 2024-04-26T00:25:44.073 INFO:tasks.rados.rados.0.smithi045.stdout:923: write oid 34 current snap is 108 2024-04-26T00:25:44.073 INFO:tasks.rados.rados.0.smithi045.stdout:923: seq_num 365 ranges {534555=650829,1659104=475451} 2024-04-26T00:25:44.088 INFO:tasks.rados.rados.0.smithi045.stdout:923: writing smithi04526579-34 from 534555 to 1185384 tid 1 2024-04-26T00:25:44.102 INFO:tasks.rados.rados.0.smithi045.stdout:923: writing smithi04526579-34 from 1659104 to 2134555 tid 2 2024-04-26T00:25:44.102 INFO:tasks.rados.rados.0.smithi045.stdout:924: read oid 28 snap -1 2024-04-26T00:25:44.102 INFO:tasks.rados.rados.0.smithi045.stdout:924: expect deleted 2024-04-26T00:25:44.102 INFO:tasks.rados.rados.0.smithi045.stdout:925: copy_from oid 17 from oid 11 current snap is 108 2024-04-26T00:25:44.102 INFO:tasks.rados.rados.0.smithi045.stdout:918: finishing write tid 1 to smithi04526579-29 2024-04-26T00:25:44.102 INFO:tasks.rados.rados.0.smithi045.stdout:918: finishing write tid 2 to smithi04526579-29 2024-04-26T00:25:44.102 INFO:tasks.rados.rados.0.smithi045.stdout:918: finishing write tid 3 to smithi04526579-29 2024-04-26T00:25:44.102 INFO:tasks.rados.rados.0.smithi045.stdout:919: finishing rollback tid 0 to smithi04526579-46 2024-04-26T00:25:44.102 INFO:tasks.rados.rados.0.smithi045.stdout:919: finishing rollback tid 1 to smithi04526579-46 2024-04-26T00:25:44.102 INFO:tasks.rados.rados.0.smithi045.stdout:918: finishing write tid 4 to smithi04526579-29 2024-04-26T00:25:44.102 INFO:tasks.rados.rados.0.smithi045.stdout:918: oid 29 updating version 0 to 322 2024-04-26T00:25:44.103 INFO:tasks.rados.rados.0.smithi045.stdout:918: oid 29 version 322 is already newer than 321 2024-04-26T00:25:44.103 INFO:tasks.rados.rados.0.smithi045.stdout:918: oid 29 version 322 is already newer than 320 2024-04-26T00:25:44.103 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 322 (ObjNum 363 snap 108 seq_num 363) dirty exists 2024-04-26T00:25:44.103 INFO:tasks.rados.rados.0.smithi045.stdout:918: left oid 29 (ObjNum 363 snap 108 seq_num 363) 2024-04-26T00:25:44.103 INFO:tasks.rados.rados.0.smithi045.stdout:919: finishing rollback tid 2 to smithi04526579-46 2024-04-26T00:25:44.103 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 253 (ObjNum 329 snap 96 seq_num 329) dirty exists 2024-04-26T00:25:44.103 INFO:tasks.rados.rados.0.smithi045.stdout:922: finishing write tid 1 to smithi04526579-7 2024-04-26T00:25:44.103 INFO:tasks.rados.rados.0.smithi045.stdout:921: expect (ObjNum 356 snap 105 seq_num 356) 2024-04-26T00:25:44.380 INFO:tasks.rados.rados.0.smithi045.stdout:922: finishing write tid 2 to smithi04526579-7 2024-04-26T00:25:44.380 INFO:tasks.rados.rados.0.smithi045.stdout:922: finishing write tid 3 to smithi04526579-7 2024-04-26T00:25:44.380 INFO:tasks.rados.rados.0.smithi045.stdout:922: finishing write tid 4 to smithi04526579-7 2024-04-26T00:25:44.380 INFO:tasks.rados.rados.0.smithi045.stdout:922: oid 7 updating version 0 to 645 2024-04-26T00:25:44.380 INFO:tasks.rados.rados.0.smithi045.stdout:922: oid 7 version 645 is already newer than 644 2024-04-26T00:25:44.380 INFO:tasks.rados.rados.0.smithi045.stdout:922: oid 7 version 645 is already newer than 643 2024-04-26T00:25:44.380 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 645 (ObjNum 364 snap 108 seq_num 364) dirty exists 2024-04-26T00:25:44.380 INFO:tasks.rados.rados.0.smithi045.stdout:922: left oid 7 (ObjNum 364 snap 108 seq_num 364) 2024-04-26T00:25:44.380 INFO:tasks.rados.rados.0.smithi045.stdout:923: finishing write tid 1 to smithi04526579-34 2024-04-26T00:25:44.380 INFO:tasks.rados.rados.0.smithi045.stdout:923: finishing write tid 2 to smithi04526579-34 2024-04-26T00:25:44.380 INFO:tasks.rados.rados.0.smithi045.stdout:923: finishing write tid 3 to smithi04526579-34 2024-04-26T00:25:44.380 INFO:tasks.rados.rados.0.smithi045.stdout:923: finishing write tid 4 to smithi04526579-34 2024-04-26T00:25:44.381 INFO:tasks.rados.rados.0.smithi045.stdout:923: oid 34 updating version 0 to 487 2024-04-26T00:25:44.381 INFO:tasks.rados.rados.0.smithi045.stdout:923: oid 34 updating version 487 to 488 2024-04-26T00:25:44.381 INFO:tasks.rados.rados.0.smithi045.stdout:923: oid 34 updating version 488 to 489 2024-04-26T00:25:44.381 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 489 (ObjNum 365 snap 108 seq_num 365) dirty exists 2024-04-26T00:25:44.381 INFO:tasks.rados.rados.0.smithi045.stdout:923: left oid 34 (ObjNum 365 snap 108 seq_num 365) 2024-04-26T00:25:44.381 INFO:tasks.rados.rados.0.smithi045.stdout:925: finishing copy_from racing read to smithi04526579-17 2024-04-26T00:25:44.381 INFO:tasks.rados.rados.0.smithi045.stdout:918: done (6 left) 2024-04-26T00:25:44.381 INFO:tasks.rados.rados.0.smithi045.stdout:919: done (5 left) 2024-04-26T00:25:44.381 INFO:tasks.rados.rados.0.smithi045.stdout:921: done (4 left) 2024-04-26T00:25:44.381 INFO:tasks.rados.rados.0.smithi045.stdout:922: done (3 left) 2024-04-26T00:25:44.381 INFO:tasks.rados.rados.0.smithi045.stdout:923: done (2 left) 2024-04-26T00:25:44.381 INFO:tasks.rados.rados.0.smithi045.stdout:924: done (1 left) 2024-04-26T00:25:44.381 INFO:tasks.rados.rados.0.smithi045.stdout:926: rollback oid 29 current snap is 108 2024-04-26T00:25:44.382 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 29 to 69 2024-04-26T00:25:44.382 INFO:tasks.rados.rados.0.smithi045.stdout:925: finishing copy_from to smithi04526579-17 2024-04-26T00:25:44.382 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 376 (ObjNum 360 snap 107 seq_num 360) dirty exists 2024-04-26T00:25:44.382 INFO:tasks.rados.rados.0.smithi045.stdout:925: done (1 left) 2024-04-26T00:25:44.382 INFO:tasks.rados.rados.0.smithi045.stdout:927: read oid 45 snap -1 2024-04-26T00:25:44.382 INFO:tasks.rados.rados.0.smithi045.stdout:927: expect (ObjNum 305 snap 87 seq_num 305) 2024-04-26T00:25:44.382 INFO:tasks.rados.rados.0.smithi045.stdout:928: delete oid 18 current snap is 108 2024-04-26T00:25:44.386 INFO:tasks.rados.rados.0.smithi045.stdout:927: expect (ObjNum 305 snap 87 seq_num 305) 2024-04-26T00:25:44.619 INFO:tasks.rados.rados.0.smithi045.stdout:926: finishing rollback tid 0 to smithi04526579-29 2024-04-26T00:25:44.620 INFO:tasks.rados.rados.0.smithi045.stdout:926: finishing rollback tid 1 to smithi04526579-29 2024-04-26T00:25:44.620 INFO:tasks.rados.rados.0.smithi045.stdout:926: finishing rollback tid 2 to smithi04526579-29 2024-04-26T00:25:44.620 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 327 (ObjNum 131 snap 30 seq_num 131) dirty exists 2024-04-26T00:25:44.620 INFO:tasks.rados.rados.0.smithi045.stdout:926: done (2 left) 2024-04-26T00:25:44.620 INFO:tasks.rados.rados.0.smithi045.stdout:927: done (1 left) 2024-04-26T00:25:44.620 INFO:tasks.rados.rados.0.smithi045.stdout:928: done (0 left) 2024-04-26T00:25:44.620 INFO:tasks.rados.rados.0.smithi045.stdout:929: read oid 38 snap 78 2024-04-26T00:25:44.620 INFO:tasks.rados.rados.0.smithi045.stdout:929: expect deleted 2024-04-26T00:25:44.620 INFO:tasks.rados.rados.0.smithi045.stdout:930: delete oid 4 current snap is 108 2024-04-26T00:25:44.626 INFO:tasks.rados.rados.0.smithi045.stdout:929: done (1 left) 2024-04-26T00:25:44.626 INFO:tasks.rados.rados.0.smithi045.stdout:930: done (0 left) 2024-04-26T00:25:44.626 INFO:tasks.rados.rados.0.smithi045.stdout:931: write oid 49 current snap is 108 2024-04-26T00:25:44.626 INFO:tasks.rados.rados.0.smithi045.stdout:931: seq_num 368 ranges {584976=476576,1834550=350426} 2024-04-26T00:25:44.643 INFO:tasks.rados.rados.0.smithi045.stdout:931: writing smithi04526579-49 from 584976 to 1061552 tid 1 2024-04-26T00:25:44.661 INFO:tasks.rados.rados.0.smithi045.stdout:931: writing smithi04526579-49 from 1834550 to 2184976 tid 2 2024-04-26T00:25:44.662 INFO:tasks.rados.rados.0.smithi045.stdout:932: snap_create 2024-04-26T00:25:44.662 INFO:tasks.rados.rados.0.smithi045.stdout:931: finishing write tid 1 to smithi04526579-49 2024-04-26T00:25:44.668 INFO:tasks.rados.rados.0.smithi045.stdout:931: finishing write tid 2 to smithi04526579-49 2024-04-26T00:25:44.669 INFO:tasks.rados.rados.0.smithi045.stdout:931: finishing write tid 3 to smithi04526579-49 2024-04-26T00:25:44.669 INFO:tasks.rados.rados.0.smithi045.stdout:931: finishing write tid 4 to smithi04526579-49 2024-04-26T00:25:44.669 INFO:tasks.rados.rados.0.smithi045.stdout:931: oid 49 updating version 0 to 491 2024-04-26T00:25:44.670 INFO:tasks.rados.rados.0.smithi045.stdout:931: oid 49 updating version 491 to 493 2024-04-26T00:25:44.670 INFO:tasks.rados.rados.0.smithi045.stdout:931: oid 49 version 493 is already newer than 492 2024-04-26T00:25:44.670 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 493 (ObjNum 368 snap 108 seq_num 368) dirty exists 2024-04-26T00:25:44.670 INFO:tasks.rados.rados.0.smithi045.stdout:931: left oid 49 (ObjNum 368 snap 108 seq_num 368) 2024-04-26T00:25:45.058 INFO:tasks.rados.rados.0.smithi045.stdout:931: done (1 left) 2024-04-26T00:25:45.059 INFO:tasks.rados.rados.0.smithi045.stdout:932: done (0 left) 2024-04-26T00:25:45.059 INFO:tasks.rados.rados.0.smithi045.stdout:933: rollback oid 36 current snap is 109 2024-04-26T00:25:45.059 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 36 to 98 2024-04-26T00:25:45.059 INFO:tasks.rados.rados.0.smithi045.stdout:934: snap_remove snap 86 2024-04-26T00:25:46.066 INFO:tasks.rados.rados.0.smithi045.stdout:934: done (1 left) 2024-04-26T00:25:46.067 INFO:tasks.rados.rados.0.smithi045.stdout:935: snap_create 2024-04-26T00:25:46.067 INFO:tasks.rados.rados.0.smithi045.stdout:933: finishing rollback tid 0 to smithi04526579-36 2024-04-26T00:25:46.067 INFO:tasks.rados.rados.0.smithi045.stdout:933: finishing rollback tid 1 to smithi04526579-36 2024-04-26T00:25:46.067 INFO:tasks.rados.rados.0.smithi045.stdout:933: finishing rollback tid 2 to smithi04526579-36 2024-04-26T00:25:46.067 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 259 (ObjNum 304 snap 87 seq_num 304) dirty exists 2024-04-26T00:25:46.873 INFO:tasks.rados.rados.0.smithi045.stdout:933: done (1 left) 2024-04-26T00:25:46.873 INFO:tasks.rados.rados.0.smithi045.stdout:935: done (0 left) 2024-04-26T00:25:46.873 INFO:tasks.rados.rados.0.smithi045.stdout:936: snap_remove snap 109 2024-04-26T00:25:47.875 INFO:tasks.rados.rados.0.smithi045.stdout:936: done (0 left) 2024-04-26T00:25:47.875 INFO:tasks.rados.rados.0.smithi045.stdout:937: snap_remove snap 70 2024-04-26T00:25:48.886 INFO:tasks.rados.rados.0.smithi045.stdout:937: done (0 left) 2024-04-26T00:25:48.886 INFO:tasks.rados.rados.0.smithi045.stdout:938: snap_create 2024-04-26T00:25:49.887 INFO:tasks.rados.rados.0.smithi045.stdout:938: done (0 left) 2024-04-26T00:25:49.887 INFO:tasks.rados.rados.0.smithi045.stdout:939: snap_create 2024-04-26T00:25:50.889 INFO:tasks.rados.rados.0.smithi045.stdout:939: done (0 left) 2024-04-26T00:25:50.890 INFO:tasks.rados.rados.0.smithi045.stdout:940: write oid 31 current snap is 112 2024-04-26T00:25:50.890 INFO:tasks.rados.rados.0.smithi045.stdout:940: seq_num 369 ranges {601783=551825,1776679=425104} 2024-04-26T00:25:50.907 INFO:tasks.rados.rados.0.smithi045.stdout:940: writing smithi04526579-31 from 601783 to 1153608 tid 1 2024-04-26T00:25:50.924 INFO:tasks.rados.rados.0.smithi045.stdout:940: writing smithi04526579-31 from 1776679 to 2201783 tid 2 2024-04-26T00:25:50.925 INFO:tasks.rados.rados.0.smithi045.stdout:941: delete oid 11 current snap is 112 2024-04-26T00:25:50.925 INFO:tasks.rados.rados.0.smithi045.stdout:940: finishing write tid 1 to smithi04526579-31 2024-04-26T00:25:50.932 INFO:tasks.rados.rados.0.smithi045.stdout:940: finishing write tid 2 to smithi04526579-31 2024-04-26T00:25:50.933 INFO:tasks.rados.rados.0.smithi045.stdout:940: finishing write tid 3 to smithi04526579-31 2024-04-26T00:25:50.933 INFO:tasks.rados.rados.0.smithi045.stdout:940: finishing write tid 4 to smithi04526579-31 2024-04-26T00:25:50.933 INFO:tasks.rados.rados.0.smithi045.stdout:940: oid 31 updating version 0 to 489 2024-04-26T00:25:50.934 INFO:tasks.rados.rados.0.smithi045.stdout:940: oid 31 version 489 is already newer than 487 2024-04-26T00:25:50.934 INFO:tasks.rados.rados.0.smithi045.stdout:940: oid 31 version 489 is already newer than 488 2024-04-26T00:25:50.934 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 489 (ObjNum 369 snap 112 seq_num 369) dirty exists 2024-04-26T00:25:50.934 INFO:tasks.rados.rados.0.smithi045.stdout:940: left oid 31 (ObjNum 369 snap 112 seq_num 369) 2024-04-26T00:25:50.936 INFO:tasks.rados.rados.0.smithi045.stdout:940: done (1 left) 2024-04-26T00:25:50.936 INFO:tasks.rados.rados.0.smithi045.stdout:941: done (0 left) 2024-04-26T00:25:50.936 INFO:tasks.rados.rados.0.smithi045.stdout:942: read oid 32 snap 107 2024-04-26T00:25:50.936 INFO:tasks.rados.rados.0.smithi045.stdout:942: expect (ObjNum 242 snap 61 seq_num 242) 2024-04-26T00:25:50.936 INFO:tasks.rados.rados.0.smithi045.stdout:943: write (excl) oid 9 current snap is 112 2024-04-26T00:25:50.937 INFO:tasks.rados.rados.0.smithi045.stdout:943: seq_num 371 ranges {635397=702323,1893643=341754} 2024-04-26T00:25:50.958 INFO:tasks.rados.rados.0.smithi045.stdout:943: writing smithi04526579-9 from 635397 to 1337720 tid 1 2024-04-26T00:25:50.972 INFO:tasks.rados.rados.0.smithi045.stdout:943: writing smithi04526579-9 from 1893643 to 2235397 tid 2 2024-04-26T00:25:50.972 INFO:tasks.rados.rados.0.smithi045.stdout:944: read oid 36 snap -1 2024-04-26T00:25:50.972 INFO:tasks.rados.rados.0.smithi045.stdout:944: expect (ObjNum 304 snap 87 seq_num 304) 2024-04-26T00:25:50.972 INFO:tasks.rados.rados.0.smithi045.stdout:945: copy_from oid 10 from oid 44 current snap is 112 2024-04-26T00:25:50.973 INFO:tasks.rados.rados.0.smithi045.stdout:942: expect (ObjNum 242 snap 61 seq_num 242) 2024-04-26T00:25:51.097 INFO:tasks.rados.rados.0.smithi045.stdout:943: finishing write tid 1 to smithi04526579-9 2024-04-26T00:25:51.098 INFO:tasks.rados.rados.0.smithi045.stdout:943: finishing write tid 2 to smithi04526579-9 2024-04-26T00:25:51.098 INFO:tasks.rados.rados.0.smithi045.stdout:943: finishing write tid 3 to smithi04526579-9 2024-04-26T00:25:51.098 INFO:tasks.rados.rados.0.smithi045.stdout:943: finishing write tid 4 to smithi04526579-9 2024-04-26T00:25:51.098 INFO:tasks.rados.rados.0.smithi045.stdout:943: oid 9 updating version 0 to 491 2024-04-26T00:25:51.098 INFO:tasks.rados.rados.0.smithi045.stdout:943: oid 9 updating version 491 to 492 2024-04-26T00:25:51.098 INFO:tasks.rados.rados.0.smithi045.stdout:943: oid 9 updating version 492 to 493 2024-04-26T00:25:51.098 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 493 (ObjNum 371 snap 112 seq_num 371) dirty exists 2024-04-26T00:25:51.098 INFO:tasks.rados.rados.0.smithi045.stdout:943: left oid 9 (ObjNum 371 snap 112 seq_num 371) 2024-04-26T00:25:51.098 INFO:tasks.rados.rados.0.smithi045.stdout:944: expect (ObjNum 304 snap 87 seq_num 304) 2024-04-26T00:25:51.243 INFO:tasks.rados.rados.0.smithi045.stdout:945: finishing copy_from racing read to smithi04526579-10 2024-04-26T00:25:51.243 INFO:tasks.rados.rados.0.smithi045.stdout:945: finishing copy_from to smithi04526579-10 2024-04-26T00:25:51.243 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 178 (ObjNum 283 snap 81 seq_num 283) dirty exists 2024-04-26T00:25:51.243 INFO:tasks.rados.rados.0.smithi045.stdout:942: done (3 left) 2024-04-26T00:25:51.243 INFO:tasks.rados.rados.0.smithi045.stdout:943: done (2 left) 2024-04-26T00:25:51.243 INFO:tasks.rados.rados.0.smithi045.stdout:944: done (1 left) 2024-04-26T00:25:51.243 INFO:tasks.rados.rados.0.smithi045.stdout:945: done (0 left) 2024-04-26T00:25:51.243 INFO:tasks.rados.rados.0.smithi045.stdout:946: snap_create 2024-04-26T00:25:51.582 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-04-26T00:25:51.582 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-04-26T00:25:51.583 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-26T00:25:51.583 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-26T00:25:51.586 INFO:tasks.ceph.mon.c:Started 2024-04-26T00:25:51.586 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:25:51.586 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:25:51.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa783559640 10 monclient: get_monmap_and_config 2024-04-26T00:25:51.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa783559640 10 monclient: build_initial_monmap 2024-04-26T00:25:51.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa783559640 10 monclient: monmap: 2024-04-26T00:25:51.720 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:51.720 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:51.720 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:25:51.741233+0000 2024-04-26T00:25:51.720 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:25:51.741233+0000 2024-04-26T00:25:51.720 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:51.720 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:51.720 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:25:51.720 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:25:51.720 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:25:51.720 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:25:51.721 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:25:51.721 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:51.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa783559640 1 Processor -- start 2024-04-26T00:25:51.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa783559640 1 -- start start 2024-04-26T00:25:51.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa783559640 10 monclient: init 2024-04-26T00:25:51.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa783559640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:51.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa783559640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-04-26T00:25:51.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa783559640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa77c136e10 addr v1:172.21.15.45:6789/0 2024-04-26T00:25:51.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa783559640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa77c137900 0x7fa77c09c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:51.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa783559640 10 monclient(hunting): picked mon.noname-b con 0x7fa77c137900 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:51.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa783559640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa77c13c340 0x7fa77c13e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:51.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa783559640 10 monclient(hunting): picked mon.noname-a con 0x7fa77c13c340 addr v2:172.21.15.37:3300/0 2024-04-26T00:25:51.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa783559640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa77c09b680 con 0x7fa77c136e10 2024-04-26T00:25:51.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa783559640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:51.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa783559640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fa77c09b8f0 con 0x7fa77c13c340 2024-04-26T00:25:51.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa783559640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:51.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa783559640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa77c09bbd0 con 0x7fa77c137900 2024-04-26T00:25:51.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa783559640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:51.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa783559640 10 monclient(hunting): authenticate will time out at 1333.047039s 2024-04-26T00:25:51.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa780af4640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa77c137900 0x7fa77c09c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:51.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa781af6640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa77c13c340 0x7fa77c13e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:51.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa781af6640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa77c13c340 0x7fa77c13e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36530/0 (socket says 172.21.15.37:36530) 2024-04-26T00:25:51.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa781af6640 1 -- 172.21.15.37:0/415453470 learned_addr learned my addr 172.21.15.37:0/415453470 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:51.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa780af4640 10 monclient(hunting): get_auth_request con 0x7fa77c137900 auth_method 0 2024-04-26T00:25:51.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa780af4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:51.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa780af4640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:51.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa780af4640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:51.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa781af6640 10 monclient(hunting): get_auth_request con 0x7fa77c13c340 auth_method 0 2024-04-26T00:25:51.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa781af6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:51.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa781af6640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:51.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa781af6640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:51.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa780af4640 1 -- 172.21.15.37:0/415453470 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa77c137900 msgr2=0x7fa77c09c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:25:51.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa780af4640 1 -- 172.21.15.37:0/415453470 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa77c137900 msgr2=0x7fa77c09c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:25:51.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa781af6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:51.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa781af6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:51.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa780af4640 1 --2- 172.21.15.37:0/415453470 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa77c137900 0x7fa77c09c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:25:51.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa781af6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:51.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa780af4640 1 --2- 172.21.15.37:0/415453470 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa77c137900 0x7fa77c09c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:25:51.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa781af6640 10 monclient(hunting): handle_auth_done global_id 54261 payload 274 2024-04-26T00:25:51.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa781af6640 1 -- 172.21.15.37:0/415453470 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa77c137900 msgr2=0x7fa77c09c5b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:25:51.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa781af6640 1 --2- 172.21.15.37:0/415453470 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa77c137900 0x7fa77c09c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:51.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa781af6640 1 -- 172.21.15.37:0/415453470 >> v1:172.21.15.45:6789/0 conn(0x7fa77c136e10 legacy=0x7fa77c1371f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:51.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa781af6640 10 monclient: _finish_hunting 0 2024-04-26T00:25:51.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa781af6640 1 monclient: found mon.noname-a 2024-04-26T00:25:51.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa781af6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:51.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa781af6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:25:51.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.737+0000 7fa781af6640 1 -- 172.21.15.37:0/415453470 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa77c0a5d90 con 0x7fa77c13c340 2024-04-26T00:25:51.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.741+0000 7fa781af6640 1 -- 172.21.15.37:0/415453470 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa77c13c340 msgr2=0x7fa77c13e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:51.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.741+0000 7fa781af6640 1 --2- 172.21.15.37:0/415453470 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa77c13c340 0x7fa77c13e730 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa774009e80 tx=0x7fa77401c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dda6372120458808 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:51.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.741+0000 7fa773fff640 1 -- 172.21.15.37:0/415453470 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7740a5510 con 0x7fa77c13c340 2024-04-26T00:25:51.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.741+0000 7fa773fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:51.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.741+0000 7fa773fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:25:51.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.741+0000 7fa773fff640 10 monclient: dump: 2024-04-26T00:25:51.728 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:51.728 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:51.728 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:51.728 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:51.728 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:51.728 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:51.728 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:51.728 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:51.728 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:51.728 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:51.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.741+0000 7fa773fff640 10 monclient: _finish_auth 0 2024-04-26T00:25:51.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.741+0000 7fa773fff640 10 monclient: _check_auth_tickets 2024-04-26T00:25:51.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.741+0000 7fa773fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:51.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.741+0000 7fa773fff640 1 -- 172.21.15.37:0/415453470 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7740a56b0 con 0x7fa77c13c340 2024-04-26T00:25:51.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.741+0000 7fa773fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:51.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.741+0000 7fa773fff640 1 -- 172.21.15.37:0/415453470 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7740a5a80 con 0x7fa77c13c340 2024-04-26T00:25:51.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.741+0000 7fa783559640 5 monclient: authenticate success, global_id 54261 2024-04-26T00:25:51.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.741+0000 7fa783559640 10 monclient: get_monmap_and_config success 2024-04-26T00:25:51.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.741+0000 7fa783559640 4 set_mon_vals no callback set 2024-04-26T00:25:51.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.741+0000 7fa773fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:51.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.741+0000 7fa773fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:51.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.741+0000 7fa773fff640 10 monclient: dump: 2024-04-26T00:25:51.729 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:51.729 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:51.729 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:51.729 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:51.729 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:51.730 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:51.730 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:51.730 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:51.730 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:51.730 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:51.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa783559640 10 monclient: shutdown 2024-04-26T00:25:51.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa783559640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:51.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa783559640 1 -- 172.21.15.37:0/415453470 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa77c13c340 msgr2=0x7fa77c13e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:51.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa783559640 1 --2- 172.21.15.37:0/415453470 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa77c13c340 0x7fa77c13e730 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa774009e80 tx=0x7fa77401c890 comp rx=0 tx=0).stop 2024-04-26T00:25:51.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa783559640 1 -- 172.21.15.37:0/415453470 shutdown_connections 2024-04-26T00:25:51.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa783559640 1 --2- 172.21.15.37:0/415453470 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa77c13c340 0x7fa77c13e730 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:51.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa783559640 1 --2- 172.21.15.37:0/415453470 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa77c137900 0x7fa77c09c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:51.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa783559640 1 -- 172.21.15.37:0/415453470 >> 172.21.15.37:0/415453470 conn(0x7fa77c0a64e0 msgr2=0x7fa77c0a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:51.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa783559640 1 -- 172.21.15.37:0/415453470 shutdown_connections 2024-04-26T00:25:51.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa783559640 1 -- 172.21.15.37:0/415453470 wait complete. 2024-04-26T00:25:51.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa783559640 10 monclient: build_initial_monmap 2024-04-26T00:25:51.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa783559640 10 monclient: monmap: 2024-04-26T00:25:51.731 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:51.731 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:25:51.731 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:25:51.731 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:25:51.731 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:51.731 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:25:51.731 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:25:51.731 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:25:51.731 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:25:51.731 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:51.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa783559640 1 Processor -- start 2024-04-26T00:25:51.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa783559640 1 -- start start 2024-04-26T00:25:51.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa783559640 10 monclient: init 2024-04-26T00:25:51.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa783559640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:51.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa783559640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:25:51.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa783559640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa77c136e10 0x7fa77c1cef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:51.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa783559640 10 monclient(hunting): picked mon.noname-c con 0x7fa77c136e10 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:51.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa783559640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa77c137900 0x7fa77c13b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:51.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa783559640 10 monclient(hunting): picked mon.noname-a con 0x7fa77c137900 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:25:51.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa783559640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa77c13c340 0x7fa77c13b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:51.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa783559640 10 monclient(hunting): picked mon.noname-b con 0x7fa77c13c340 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:25:51.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa783559640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:51.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa783559640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa77c141820 con 0x7fa77c137900 2024-04-26T00:25:51.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa783559640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:51.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa783559640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa77c1416a0 con 0x7fa77c13c340 2024-04-26T00:25:51.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa783559640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:51.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa783559640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa77c1419a0 con 0x7fa77c136e10 2024-04-26T00:25:51.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa783559640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:51.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa783559640 10 monclient(hunting): authenticate will time out at 1333.054545s 2024-04-26T00:25:51.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa7812f5640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa77c136e10 0x7fa77c1cef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:51.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa780af4640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa77c137900 0x7fa77c13b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:51.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa7812f5640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa77c136e10 0x7fa77c1cef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51762/0 (socket says 172.21.15.37:51762) 2024-04-26T00:25:51.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa7812f5640 1 -- 172.21.15.37:0/2745274293 learned_addr learned my addr 172.21.15.37:0/2745274293 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:51.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa7812f5640 10 monclient(hunting): get_auth_request con 0x7fa77c136e10 auth_method 0 2024-04-26T00:25:51.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa7812f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:51.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa7812f5640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:51.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa7812f5640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:51.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa780af4640 10 monclient(hunting): get_auth_request con 0x7fa77c137900 auth_method 0 2024-04-26T00:25:51.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa780af4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:51.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa780af4640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:51.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa780af4640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:51.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa7812f5640 1 -- 172.21.15.37:0/2745274293 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa77c136e10 msgr2=0x7fa77c1cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:25:51.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa7812f5640 1 -- 172.21.15.37:0/2745274293 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa77c136e10 msgr2=0x7fa77c1cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:25:51.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa7812f5640 1 --2- 172.21.15.37:0/2745274293 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa77c136e10 0x7fa77c1cef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:25:51.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa780af4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:51.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa780af4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:51.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa780af4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:51.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa7812f5640 1 --2- 172.21.15.37:0/2745274293 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa77c136e10 0x7fa77c1cef40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:25:51.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa781af6640 1 --2- 172.21.15.37:0/2745274293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa77c13c340 0x7fa77c13b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:51.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa781af6640 10 monclient(hunting): get_auth_request con 0x7fa77c13c340 auth_method 0 2024-04-26T00:25:51.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa781af6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:51.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa781af6640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:51.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa781af6640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:51.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa780af4640 10 monclient(hunting): handle_auth_done global_id 54264 payload 824 2024-04-26T00:25:51.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa780af4640 1 -- 172.21.15.37:0/2745274293 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa77c136e10 msgr2=0x7fa77c1cef40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:25:51.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa780af4640 1 --2- 172.21.15.37:0/2745274293 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa77c136e10 0x7fa77c1cef40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:51.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa780af4640 1 -- 172.21.15.37:0/2745274293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa77c13c340 msgr2=0x7fa77c13b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:51.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa780af4640 1 --2- 172.21.15.37:0/2745274293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa77c13c340 0x7fa77c13b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:51.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa780af4640 10 monclient: _finish_hunting 0 2024-04-26T00:25:51.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa780af4640 1 monclient: found mon.noname-a 2024-04-26T00:25:51.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa780af4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:51.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa780af4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:25:51.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.745+0000 7fa780af4640 1 -- 172.21.15.37:0/2745274293 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa77c139ae0 con 0x7fa77c137900 2024-04-26T00:25:51.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.749+0000 7fa780af4640 1 -- 172.21.15.37:0/2745274293 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa77c137900 msgr2=0x7fa77c13b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:51.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.749+0000 7fa780af4640 1 --2- 172.21.15.37:0/2745274293 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa77c137900 0x7fa77c13b1d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa76c00e810 tx=0x7fa76c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:51.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.749+0000 7fa771ffb640 1 -- 172.21.15.37:0/2745274293 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa76c017030 con 0x7fa77c137900 2024-04-26T00:25:51.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.749+0000 7fa771ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:51.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.749+0000 7fa771ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:25:51.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.749+0000 7fa771ffb640 10 monclient: dump: 2024-04-26T00:25:51.740 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:51.740 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:51.740 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:51.740 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:51.740 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:51.741 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:51.741 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:51.741 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:51.741 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:51.741 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:51.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.749+0000 7fa771ffb640 10 monclient: _finish_auth 0 2024-04-26T00:25:51.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.749+0000 7fa771ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:25:51.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.749+0000 7fa771ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:51.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.749+0000 7fa783559640 5 monclient: authenticate success, global_id 54264 2024-04-26T00:25:51.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.749+0000 7fa783559640 10 monclient: _renew_subs 2024-04-26T00:25:51.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.749+0000 7fa771ffb640 1 -- 172.21.15.37:0/2745274293 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa76c004020 con 0x7fa77c137900 2024-04-26T00:25:51.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.749+0000 7fa783559640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:51.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.749+0000 7fa783559640 1 -- 172.21.15.37:0/2745274293 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa77c09aca0 con 0x7fa77c137900 2024-04-26T00:25:51.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.749+0000 7fa771ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:51.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.749+0000 7fa771ffb640 1 -- 172.21.15.37:0/2745274293 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa76c00f040 con 0x7fa77c137900 2024-04-26T00:25:51.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.749+0000 7fa771ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:51.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.749+0000 7fa771ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:51.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.749+0000 7fa771ffb640 10 monclient: dump: 2024-04-26T00:25:51.742 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:51.742 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:51.742 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:51.742 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:51.742 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:51.742 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:51.742 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:51.742 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:51.742 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:51.742 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:51.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.749+0000 7fa7737fe640 4 set_mon_vals no callback set 2024-04-26T00:25:51.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.749+0000 7fa783559640 10 monclient: _renew_subs 2024-04-26T00:25:51.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.749+0000 7fa783559640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:51.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.749+0000 7fa783559640 1 -- 172.21.15.37:0/2745274293 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa77c0a3580 con 0x7fa77c137900 2024-04-26T00:25:51.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.749+0000 7fa783559640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:51.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.749+0000 7fa783559640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:51.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.749+0000 7fa783559640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:51.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.749+0000 7fa783559640 1 -- 172.21.15.37:0/2745274293 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa77c075910 con 0x7fa77c137900 2024-04-26T00:25:51.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.749+0000 7fa771ffb640 1 -- 172.21.15.37:0/2745274293 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa76c01a070 con 0x7fa77c137900 2024-04-26T00:25:51.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.749+0000 7fa771ffb640 1 --2- 172.21.15.37:0/2745274293 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa760035e80 0x7fa760038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:51.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.749+0000 7fa771ffb640 1 -- 172.21.15.37:0/2745274293 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(228..228 src has 177..228) ==== 3963+0+0 (secure 0 0 0) 0x7fa76c048430 con 0x7fa77c137900 2024-04-26T00:25:51.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.749+0000 7fa7812f5640 1 --2- 172.21.15.37:0/2745274293 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa760035e80 0x7fa760038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:51.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.749+0000 7fa7812f5640 10 monclient: get_auth_request con 0x7fa760035e80 auth_method 0 2024-04-26T00:25:51.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.753+0000 7fa7812f5640 1 --2- 172.21.15.37:0/2745274293 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa760035e80 0x7fa760038340 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa778007d20 tx=0x7fa778009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:51.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.753+0000 7fa771ffb640 1 -- 172.21.15.37:0/2745274293 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa76c00c780 con 0x7fa77c137900 2024-04-26T00:25:51.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.753+0000 7fa771ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:51.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.753+0000 7fa771ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:25:51.874 INFO:tasks.rados.rados.0.smithi045.stdout:946: done (0 left) 2024-04-26T00:25:51.874 INFO:tasks.rados.rados.0.smithi045.stdout:947: write oid 5 current snap is 113 2024-04-26T00:25:51.874 INFO:tasks.rados.rados.0.smithi045.stdout:947: seq_num 372 ranges {652204=777572,1952125=300079} 2024-04-26T00:25:51.897 INFO:tasks.rados.rados.0.smithi045.stdout:947: writing smithi04526579-5 from 652204 to 1429776 tid 1 2024-04-26T00:25:51.911 INFO:tasks.rados.rados.0.smithi045.stdout:947: writing smithi04526579-5 from 1952125 to 2252204 tid 2 2024-04-26T00:25:51.911 INFO:tasks.rados.rados.0.smithi045.stdout:948: read oid 47 snap -1 2024-04-26T00:25:51.911 INFO:tasks.rados.rados.0.smithi045.stdout:948: expect deleted 2024-04-26T00:25:51.911 INFO:tasks.rados.rados.0.smithi045.stdout:949: write (excl) oid 43 current snap is 113 2024-04-26T00:25:51.911 INFO:tasks.rados.rados.0.smithi045.stdout:949: seq_num 373 ranges {669011=569174,2010607=258404} 2024-04-26T00:25:51.930 INFO:tasks.rados.rados.0.smithi045.stdout:949: writing smithi04526579-43 from 669011 to 1238185 tid 1 2024-04-26T00:25:51.943 INFO:tasks.rados.rados.0.smithi045.stdout:949: writing smithi04526579-43 from 2010607 to 2269011 tid 2 2024-04-26T00:25:51.943 INFO:tasks.rados.rados.0.smithi045.stdout:950: read oid 44 snap -1 2024-04-26T00:25:51.943 INFO:tasks.rados.rados.0.smithi045.stdout:950: expect (ObjNum 283 snap 81 seq_num 283) 2024-04-26T00:25:51.943 INFO:tasks.rados.rados.0.smithi045.stdout:951: read oid 31 snap -1 2024-04-26T00:25:51.943 INFO:tasks.rados.rados.0.smithi045.stdout:951: expect (ObjNum 369 snap 112 seq_num 369) 2024-04-26T00:25:51.943 INFO:tasks.rados.rados.0.smithi045.stdout:952: copy_from oid 26 from oid 15 current snap is 113 2024-04-26T00:25:51.943 INFO:tasks.rados.rados.0.smithi045.stdout:947: finishing write tid 1 to smithi04526579-5 2024-04-26T00:25:51.943 INFO:tasks.rados.rados.0.smithi045.stdout:947: finishing write tid 2 to smithi04526579-5 2024-04-26T00:25:51.943 INFO:tasks.rados.rados.0.smithi045.stdout:947: finishing write tid 3 to smithi04526579-5 2024-04-26T00:25:51.944 INFO:tasks.rados.rados.0.smithi045.stdout:947: finishing write tid 4 to smithi04526579-5 2024-04-26T00:25:51.944 INFO:tasks.rados.rados.0.smithi045.stdout:947: oid 5 updating version 0 to 660 2024-04-26T00:25:51.944 INFO:tasks.rados.rados.0.smithi045.stdout:947: oid 5 updating version 660 to 661 2024-04-26T00:25:51.944 INFO:tasks.rados.rados.0.smithi045.stdout:947: oid 5 version 661 is already newer than 659 2024-04-26T00:25:51.944 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 661 (ObjNum 372 snap 113 seq_num 372) dirty exists 2024-04-26T00:25:51.944 INFO:tasks.rados.rados.0.smithi045.stdout:947: left oid 5 (ObjNum 372 snap 113 seq_num 372) 2024-04-26T00:25:51.944 INFO:tasks.rados.rados.0.smithi045.stdout:949: finishing write tid 1 to smithi04526579-43 2024-04-26T00:25:51.944 INFO:tasks.rados.rados.0.smithi045.stdout:947: done (5 left) 2024-04-26T00:25:51.944 INFO:tasks.rados.rados.0.smithi045.stdout:948: done (4 left) 2024-04-26T00:25:51.944 INFO:tasks.rados.rados.0.smithi045.stdout:953: write oid 8 current snap is 113 2024-04-26T00:25:51.944 INFO:tasks.rados.rados.0.smithi045.stdout:953: seq_num 374 ranges {685818=644423,2069089=216729} 2024-04-26T00:25:51.957 INFO:tasks.rados.rados.0.smithi045.stdout:953: writing smithi04526579-8 from 685818 to 1330241 tid 1 2024-04-26T00:25:51.967 INFO:tasks.rados.rados.0.smithi045.stdout:953: writing smithi04526579-8 from 2069089 to 2285818 tid 2 2024-04-26T00:25:51.967 INFO:tasks.rados.rados.0.smithi045.stdout:954: rollback oid 18 current snap is 113 2024-04-26T00:25:51.967 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 18 to 107 2024-04-26T00:25:51.967 INFO:tasks.rados.rados.0.smithi045.stdout:952: finishing copy_from to smithi04526579-26 2024-04-26T00:25:51.967 INFO:tasks.rados.rados.0.smithi045.stdout:952: got expected ENOENT (src dne) 2024-04-26T00:25:51.967 INFO:tasks.rados.rados.0.smithi045.stdout:952: finishing copy_from racing read to smithi04526579-26 2024-04-26T00:25:51.967 INFO:tasks.rados.rados.0.smithi045.stdout:949: finishing write tid 2 to smithi04526579-43 2024-04-26T00:25:51.967 INFO:tasks.rados.rados.0.smithi045.stdout:949: finishing write tid 3 to smithi04526579-43 2024-04-26T00:25:51.967 INFO:tasks.rados.rados.0.smithi045.stdout:949: finishing write tid 4 to smithi04526579-43 2024-04-26T00:25:51.967 INFO:tasks.rados.rados.0.smithi045.stdout:949: oid 43 updating version 0 to 438 2024-04-26T00:25:51.967 INFO:tasks.rados.rados.0.smithi045.stdout:949: oid 43 version 438 is already newer than 437 2024-04-26T00:25:51.967 INFO:tasks.rados.rados.0.smithi045.stdout:949: oid 43 updating version 438 to 439 2024-04-26T00:25:51.967 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 439 (ObjNum 373 snap 113 seq_num 373) dirty exists 2024-04-26T00:25:51.968 INFO:tasks.rados.rados.0.smithi045.stdout:949: left oid 43 (ObjNum 373 snap 113 seq_num 373) 2024-04-26T00:25:51.968 INFO:tasks.rados.rados.0.smithi045.stdout:951: expect (ObjNum 369 snap 112 seq_num 369) 2024-04-26T00:25:51.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.993+0000 7fa783559640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:25:51.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.993+0000 7fa783559640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:25:51.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.993+0000 7fa783559640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:51.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.993+0000 7fa783559640 1 -- 172.21.15.37:0/2745274293 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa77c066f40 con 0x7fa77c137900 2024-04-26T00:25:51.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.993+0000 7fa771ffb640 1 -- 172.21.15.37:0/2745274293 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7fa76c0481c0 con 0x7fa77c137900 2024-04-26T00:25:51.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.993+0000 7fa771ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:25:51.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.993+0000 7fa771ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:25:51.976 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":44,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:25:51.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.997+0000 7fa783559640 1 -- 172.21.15.37:0/2745274293 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa760035e80 msgr2=0x7fa760038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:51.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.997+0000 7fa783559640 1 --2- 172.21.15.37:0/2745274293 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa760035e80 0x7fa760038340 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa778007d20 tx=0x7fa778009040 comp rx=0 tx=0).stop 2024-04-26T00:25:51.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.997+0000 7fa783559640 10 monclient: shutdown 2024-04-26T00:25:51.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.997+0000 7fa783559640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:51.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.997+0000 7fa783559640 1 -- 172.21.15.37:0/2745274293 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa77c137900 msgr2=0x7fa77c13b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:51.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.997+0000 7fa783559640 1 --2- 172.21.15.37:0/2745274293 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa77c137900 0x7fa77c13b1d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa76c00e810 tx=0x7fa76c00ecd0 comp rx=0 tx=0).stop 2024-04-26T00:25:51.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.997+0000 7fa783559640 1 -- 172.21.15.37:0/2745274293 shutdown_connections 2024-04-26T00:25:51.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.997+0000 7fa783559640 1 --2- 172.21.15.37:0/2745274293 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa760035e80 0x7fa760038340 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:51.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.997+0000 7fa783559640 1 --2- 172.21.15.37:0/2745274293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa77c13c340 0x7fa77c13b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:51.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.997+0000 7fa783559640 1 --2- 172.21.15.37:0/2745274293 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa77c137900 0x7fa77c13b1d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:51.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.997+0000 7fa783559640 1 --2- 172.21.15.37:0/2745274293 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa77c136e10 0x7fa77c1cef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:51.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.997+0000 7fa783559640 1 -- 172.21.15.37:0/2745274293 >> 172.21.15.37:0/2745274293 conn(0x7fa77c0a64e0 msgr2=0x7fa77c13d030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:51.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.997+0000 7fa783559640 1 -- 172.21.15.37:0/2745274293 shutdown_connections 2024-04-26T00:25:51.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:51.997+0000 7fa783559640 1 -- 172.21.15.37:0/2745274293 wait complete. 2024-04-26T00:25:52.085 INFO:tasks.rados.rados.0.smithi045.stdout:950: expect (ObjNum 283 snap 81 seq_num 283) 2024-04-26T00:25:52.264 INFO:tasks.rados.rados.0.smithi045.stdout:953: finishing write tid 1 to smithi04526579-8 2024-04-26T00:25:52.265 INFO:tasks.rados.rados.0.smithi045.stdout:953: finishing write tid 2 to smithi04526579-8 2024-04-26T00:25:52.265 INFO:tasks.rados.rados.0.smithi045.stdout:953: finishing write tid 3 to smithi04526579-8 2024-04-26T00:25:52.265 INFO:tasks.rados.rados.0.smithi045.stdout:953: finishing write tid 4 to smithi04526579-8 2024-04-26T00:25:52.265 INFO:tasks.rados.rados.0.smithi045.stdout:953: oid 8 updating version 0 to 341 2024-04-26T00:25:52.265 INFO:tasks.rados.rados.0.smithi045.stdout:953: oid 8 updating version 341 to 342 2024-04-26T00:25:52.265 INFO:tasks.rados.rados.0.smithi045.stdout:953: oid 8 version 342 is already newer than 340 2024-04-26T00:25:52.265 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 342 (ObjNum 374 snap 113 seq_num 374) dirty exists 2024-04-26T00:25:52.265 INFO:tasks.rados.rados.0.smithi045.stdout:953: left oid 8 (ObjNum 374 snap 113 seq_num 374) 2024-04-26T00:25:52.265 INFO:tasks.rados.rados.0.smithi045.stdout:954: finishing rollback tid 1 to smithi04526579-18 2024-04-26T00:25:52.265 INFO:tasks.rados.rados.0.smithi045.stdout:954: finishing rollback tid 2 to smithi04526579-18 2024-04-26T00:25:52.265 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 663 (ObjNum 309 snap 90 seq_num 309) dirty exists 2024-04-26T00:25:52.265 INFO:tasks.rados.rados.0.smithi045.stdout:949: done (5 left) 2024-04-26T00:25:52.265 INFO:tasks.rados.rados.0.smithi045.stdout:950: done (4 left) 2024-04-26T00:25:52.266 INFO:tasks.rados.rados.0.smithi045.stdout:951: done (3 left) 2024-04-26T00:25:52.266 INFO:tasks.rados.rados.0.smithi045.stdout:952: done (2 left) 2024-04-26T00:25:52.266 INFO:tasks.rados.rados.0.smithi045.stdout:953: done (1 left) 2024-04-26T00:25:52.266 INFO:tasks.rados.rados.0.smithi045.stdout:954: done (0 left) 2024-04-26T00:25:52.266 INFO:tasks.rados.rados.0.smithi045.stdout:955: read oid 20 snap -1 2024-04-26T00:25:52.266 INFO:tasks.rados.rados.0.smithi045.stdout:955: expect (ObjNum 356 snap 105 seq_num 356) 2024-04-26T00:25:52.266 INFO:tasks.rados.rados.0.smithi045.stdout:956: rollback oid 13 current snap is 113 2024-04-26T00:25:52.266 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 13 to 101 2024-04-26T00:25:52.266 INFO:tasks.rados.rados.0.smithi045.stdout:957: snap_create 2024-04-26T00:25:52.279 INFO:tasks.rados.rados.0.smithi045.stdout:956: finishing rollback tid 0 to smithi04526579-13 2024-04-26T00:25:52.280 INFO:tasks.rados.rados.0.smithi045.stdout:956: finishing rollback tid 1 to smithi04526579-13 2024-04-26T00:25:52.281 INFO:tasks.rados.rados.0.smithi045.stdout:956: finishing rollback tid 2 to smithi04526579-13 2024-04-26T00:25:52.281 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 346 (ObjNum 337 snap 100 seq_num 337) dirty exists 2024-04-26T00:25:52.282 INFO:tasks.rados.rados.0.smithi045.stdout:955: expect (ObjNum 356 snap 105 seq_num 356) 2024-04-26T00:25:52.906 INFO:tasks.rados.rados.0.smithi045.stdout:955: done (2 left) 2024-04-26T00:25:52.906 INFO:tasks.rados.rados.0.smithi045.stdout:956: done (1 left) 2024-04-26T00:25:52.906 INFO:tasks.rados.rados.0.smithi045.stdout:957: done (0 left) 2024-04-26T00:25:52.906 INFO:tasks.rados.rados.0.smithi045.stdout:958: snap_remove snap 100 2024-04-26T00:25:53.915 INFO:tasks.rados.rados.0.smithi045.stdout:958: done (0 left) 2024-04-26T00:25:53.915 INFO:tasks.rados.rados.0.smithi045.stdout:959: copy_from oid 8 from oid 13 current snap is 114 2024-04-26T00:25:53.916 INFO:tasks.rados.rados.0.smithi045.stdout:960: read oid 23 snap -1 2024-04-26T00:25:53.916 INFO:tasks.rados.rados.0.smithi045.stdout:960: expect (ObjNum 316 snap 93 seq_num 316) 2024-04-26T00:25:53.916 INFO:tasks.rados.rados.0.smithi045.stdout:961: rollback oid 39 current snap is 114 2024-04-26T00:25:53.916 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 39 to 69 2024-04-26T00:25:53.916 INFO:tasks.rados.rados.0.smithi045.stdout:962: read oid 33 snap 101 2024-04-26T00:25:53.916 INFO:tasks.rados.rados.0.smithi045.stdout:962: expect (ObjNum 333 snap 99 seq_num 333) 2024-04-26T00:25:53.916 INFO:tasks.rados.rados.0.smithi045.stdout:963: snap_remove snap 87 2024-04-26T00:25:54.918 INFO:tasks.rados.rados.0.smithi045.stdout:963: done (4 left) 2024-04-26T00:25:54.919 INFO:tasks.rados.rados.0.smithi045.stdout:964: read oid 46 snap 104 2024-04-26T00:25:54.919 INFO:tasks.rados.rados.0.smithi045.stdout:964: expect (ObjNum 329 snap 96 seq_num 329) 2024-04-26T00:25:54.919 INFO:tasks.rados.rados.0.smithi045.stdout:965: read oid 24 snap -1 2024-04-26T00:25:54.919 INFO:tasks.rados.rados.0.smithi045.stdout:965: expect (ObjNum 351 snap 105 seq_num 351) 2024-04-26T00:25:54.919 INFO:tasks.rados.rados.0.smithi045.stdout:966: write oid 15 current snap is 114 2024-04-26T00:25:54.920 INFO:tasks.rados.rados.0.smithi045.stdout:966: seq_num 375 ranges {702625=719672,2127571=175054} 2024-04-26T00:25:54.928 INFO:tasks.rados.rados.0.smithi045.stdout:966: writing smithi04526579-15 from 702625 to 1422297 tid 1 2024-04-26T00:25:54.933 INFO:tasks.rados.rados.0.smithi045.stdout:966: writing smithi04526579-15 from 2127571 to 2302625 tid 2 2024-04-26T00:25:54.933 INFO:tasks.rados.rados.0.smithi045.stdout:967: snap_remove snap 107 2024-04-26T00:25:54.996 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:25:55.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e73302640 10 monclient: get_monmap_and_config 2024-04-26T00:25:55.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e73302640 10 monclient: build_initial_monmap 2024-04-26T00:25:55.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e73302640 10 monclient: monmap: 2024-04-26T00:25:55.127 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:55.128 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:55.128 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:25:55.149001+0000 2024-04-26T00:25:55.128 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:25:55.149001+0000 2024-04-26T00:25:55.128 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:55.128 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:55.128 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:25:55.128 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:25:55.128 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:25:55.128 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:25:55.128 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:25:55.128 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:55.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e73302640 1 Processor -- start 2024-04-26T00:25:55.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e73302640 1 -- start start 2024-04-26T00:25:55.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e73302640 10 monclient: init 2024-04-26T00:25:55.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e73302640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:55.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e73302640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-04-26T00:25:55.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e73302640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e6c136da0 0x7f0e6c09bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:55.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e73302640 10 monclient(hunting): picked mon.noname-b con 0x7f0e6c136da0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:55.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e73302640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0e6c09c410 addr v1:172.21.15.45:6789/0 2024-04-26T00:25:55.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e73302640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0e6c137890 0x7f0e6c13e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:55.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e73302640 10 monclient(hunting): picked mon.noname-c con 0x7f0e6c137890 addr v2:172.21.15.45:3300/0 2024-04-26T00:25:55.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e73302640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e6c09b4f0 con 0x7f0e6c09c410 2024-04-26T00:25:55.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e73302640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:55.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e73302640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f0e6c09bb40 con 0x7f0e6c137890 2024-04-26T00:25:55.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e73302640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:55.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e73302640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0e6c09b210 con 0x7f0e6c136da0 2024-04-26T00:25:55.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e73302640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:55.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e73302640 10 monclient(hunting): authenticate will time out at 1336.454740s 2024-04-26T00:25:55.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e7109e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e6c136da0 0x7f0e6c09bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:55.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e7109e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e6c136da0 0x7f0e6c09bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51774/0 (socket says 172.21.15.37:51774) 2024-04-26T00:25:55.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e7109e640 1 -- 172.21.15.37:0/3930503131 learned_addr learned my addr 172.21.15.37:0/3930503131 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:55.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e7109e640 10 monclient(hunting): get_auth_request con 0x7f0e6c136da0 auth_method 0 2024-04-26T00:25:55.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e7109e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:55.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e7109e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:55.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e7109e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:55.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e7109e640 1 -- 172.21.15.37:0/3930503131 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e6c136da0 msgr2=0x7f0e6c09bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:25:55.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e7109e640 1 -- 172.21.15.37:0/3930503131 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e6c136da0 msgr2=0x7f0e6c09bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:25:55.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e7109e640 1 --2- 172.21.15.37:0/3930503131 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e6c136da0 0x7f0e6c09bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:25:55.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e7189f640 1 --2- 172.21.15.37:0/3930503131 >> v2:172.21.15.45:3300/0 conn(0x7f0e6c137890 0x7f0e6c13e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:55.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e7109e640 1 --2- 172.21.15.37:0/3930503131 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e6c136da0 0x7f0e6c09bed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:25:55.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e7189f640 10 monclient(hunting): get_auth_request con 0x7f0e6c137890 auth_method 0 2024-04-26T00:25:55.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e7189f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:55.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e7189f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:55.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e7189f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:55.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e7189f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:55.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e7189f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:55.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e7189f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:55.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e7189f640 10 monclient(hunting): handle_auth_done global_id 24511 payload 274 2024-04-26T00:25:55.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e7189f640 1 -- 172.21.15.37:0/3930503131 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e6c136da0 msgr2=0x7f0e6c09bed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:25:55.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e7189f640 1 --2- 172.21.15.37:0/3930503131 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e6c136da0 0x7f0e6c09bed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:55.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e7189f640 1 -- 172.21.15.37:0/3930503131 >> v1:172.21.15.45:6789/0 conn(0x7f0e6c09c410 legacy=0x7f0e6c137180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:55.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e7189f640 10 monclient: _finish_hunting 0 2024-04-26T00:25:55.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e7189f640 1 monclient: found mon.noname-c 2024-04-26T00:25:55.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e7189f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:55.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e7189f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:25:55.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.145+0000 7f0e7189f640 1 -- 172.21.15.37:0/3930503131 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e6c09b7d0 con 0x7f0e6c137890 2024-04-26T00:25:55.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e7189f640 1 -- 172.21.15.37:0/3930503131 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e6c137890 msgr2=0x7f0e6c13e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:55.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e7189f640 1 --2- 172.21.15.37:0/3930503131 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e6c137890 0x7f0e6c13e2e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0e64009d90 tx=0x7f0e6401c7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db93615c636febf5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:55.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e63fff640 1 -- 172.21.15.37:0/3930503131 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e640b2070 con 0x7f0e6c137890 2024-04-26T00:25:55.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e63fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:55.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e63fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:25:55.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e63fff640 10 monclient: dump: 2024-04-26T00:25:55.137 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:55.137 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:55.137 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:55.137 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:55.137 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:55.137 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:55.137 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:55.137 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:55.137 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:55.137 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:55.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e63fff640 10 monclient: _finish_auth 0 2024-04-26T00:25:55.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e63fff640 10 monclient: _check_auth_tickets 2024-04-26T00:25:55.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:55.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e63fff640 1 -- 172.21.15.37:0/3930503131 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e640a5890 con 0x7f0e6c137890 2024-04-26T00:25:55.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e63fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:55.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e63fff640 1 -- 172.21.15.37:0/3930503131 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e640a5c60 con 0x7f0e6c137890 2024-04-26T00:25:55.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e73302640 5 monclient: authenticate success, global_id 24511 2024-04-26T00:25:55.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e73302640 10 monclient: get_monmap_and_config success 2024-04-26T00:25:55.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e73302640 4 set_mon_vals no callback set 2024-04-26T00:25:55.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e63fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:55.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e63fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:25:55.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e63fff640 10 monclient: dump: 2024-04-26T00:25:55.138 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:55.138 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:55.138 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:55.138 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:55.138 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:55.138 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:55.139 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:55.139 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:55.139 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:55.139 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:55.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e73302640 10 monclient: shutdown 2024-04-26T00:25:55.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e73302640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:55.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e73302640 1 -- 172.21.15.37:0/3930503131 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e6c137890 msgr2=0x7f0e6c13e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:55.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e73302640 1 --2- 172.21.15.37:0/3930503131 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e6c137890 0x7f0e6c13e2e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0e64009d90 tx=0x7f0e6401c7f0 comp rx=0 tx=0).stop 2024-04-26T00:25:55.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e73302640 1 -- 172.21.15.37:0/3930503131 shutdown_connections 2024-04-26T00:25:55.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e73302640 1 --2- 172.21.15.37:0/3930503131 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e6c137890 0x7f0e6c13e2e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:55.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e73302640 1 --2- 172.21.15.37:0/3930503131 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e6c136da0 0x7f0e6c09bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:55.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e73302640 1 -- 172.21.15.37:0/3930503131 >> 172.21.15.37:0/3930503131 conn(0x7f0e6c0a5f00 msgr2=0x7f0e6c0a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:55.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e73302640 1 -- 172.21.15.37:0/3930503131 shutdown_connections 2024-04-26T00:25:55.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e73302640 1 -- 172.21.15.37:0/3930503131 wait complete. 2024-04-26T00:25:55.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e73302640 10 monclient: build_initial_monmap 2024-04-26T00:25:55.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e73302640 10 monclient: monmap: 2024-04-26T00:25:55.140 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:55.140 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:25:55.140 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:25:55.140 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:25:55.140 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:55.140 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:25:55.140 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:25:55.140 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:25:55.140 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:25:55.140 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:55.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e73302640 1 Processor -- start 2024-04-26T00:25:55.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e73302640 1 -- start start 2024-04-26T00:25:55.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e73302640 10 monclient: init 2024-04-26T00:25:55.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e73302640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:55.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e73302640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:25:55.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e73302640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e6c09c410 0x7f0e6c1ce900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:55.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e73302640 10 monclient(hunting): picked mon.noname-b con 0x7f0e6c09c410 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:25:55.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e73302640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0e6c136da0 0x7f0e6c135cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:55.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e73302640 10 monclient(hunting): picked mon.noname-a con 0x7f0e6c136da0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:25:55.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e73302640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e6c137890 0x7f0e6c13ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:55.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e73302640 10 monclient(hunting): picked mon.noname-c con 0x7f0e6c137890 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:55.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e73302640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:55.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e73302640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f0e6c1414b0 con 0x7f0e6c136da0 2024-04-26T00:25:55.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e73302640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:55.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e73302640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0e6c141330 con 0x7f0e6c09c410 2024-04-26T00:25:55.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e73302640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:55.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e73302640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0e6c141630 con 0x7f0e6c137890 2024-04-26T00:25:55.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e73302640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:55.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e73302640 10 monclient(hunting): authenticate will time out at 1336.462952s 2024-04-26T00:25:55.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e7189f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e6c137890 0x7f0e6c13ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:55.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e7089d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0e6c136da0 0x7f0e6c135cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:55.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e7189f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e6c137890 0x7f0e6c13ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51776/0 (socket says 172.21.15.37:51776) 2024-04-26T00:25:55.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e7189f640 1 -- 172.21.15.37:0/3432015461 learned_addr learned my addr 172.21.15.37:0/3432015461 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:55.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e7189f640 10 monclient(hunting): get_auth_request con 0x7f0e6c137890 auth_method 0 2024-04-26T00:25:55.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e7189f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:55.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e7189f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:55.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e7189f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:55.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e7089d640 10 monclient(hunting): get_auth_request con 0x7f0e6c136da0 auth_method 0 2024-04-26T00:25:55.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e7089d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:55.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e7089d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:55.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e7089d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:55.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e7189f640 1 -- 172.21.15.37:0/3432015461 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e6c137890 msgr2=0x7f0e6c13ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:25:55.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e7089d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:55.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e7189f640 1 -- 172.21.15.37:0/3432015461 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e6c137890 msgr2=0x7f0e6c13ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:25:55.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e7089d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:55.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e7189f640 1 --2- 172.21.15.37:0/3432015461 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e6c137890 0x7f0e6c13ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:25:55.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e7089d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:55.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e7189f640 1 --2- 172.21.15.37:0/3432015461 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e6c137890 0x7f0e6c13ab00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:25:55.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e7109e640 1 --2- 172.21.15.37:0/3432015461 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e6c09c410 0x7f0e6c1ce900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:55.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e7109e640 10 monclient(hunting): get_auth_request con 0x7f0e6c09c410 auth_method 0 2024-04-26T00:25:55.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e7109e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:55.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e7109e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:55.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e7109e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:55.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e7089d640 10 monclient(hunting): handle_auth_done global_id 54267 payload 824 2024-04-26T00:25:55.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e7089d640 1 -- 172.21.15.37:0/3432015461 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e6c137890 msgr2=0x7f0e6c13ab00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:25:55.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e7089d640 1 --2- 172.21.15.37:0/3432015461 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e6c137890 0x7f0e6c13ab00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:55.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e7089d640 1 -- 172.21.15.37:0/3432015461 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e6c09c410 msgr2=0x7f0e6c1ce900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:55.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e7089d640 1 --2- 172.21.15.37:0/3432015461 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e6c09c410 0x7f0e6c1ce900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:55.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e7089d640 10 monclient: _finish_hunting 0 2024-04-26T00:25:55.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e7089d640 1 monclient: found mon.noname-a 2024-04-26T00:25:55.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e7089d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:55.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e7089d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:25:55.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.153+0000 7f0e7089d640 1 -- 172.21.15.37:0/3432015461 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e6c13c7a0 con 0x7f0e6c136da0 2024-04-26T00:25:55.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.157+0000 7f0e7089d640 1 -- 172.21.15.37:0/3432015461 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0e6c136da0 msgr2=0x7f0e6c135cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:55.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.157+0000 7f0e7089d640 1 --2- 172.21.15.37:0/3432015461 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0e6c136da0 0x7f0e6c135cb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0e6800c410 tx=0x7f0e6800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:55.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.157+0000 7f0e61ffb640 1 -- 172.21.15.37:0/3432015461 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e680119a0 con 0x7f0e6c136da0 2024-04-26T00:25:55.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.157+0000 7f0e61ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:55.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.157+0000 7f0e61ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:25:55.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.157+0000 7f0e61ffb640 10 monclient: dump: 2024-04-26T00:25:55.148 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:55.148 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:55.148 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:55.148 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:55.148 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:55.148 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:55.148 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:55.148 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:55.148 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:55.148 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:55.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.157+0000 7f0e61ffb640 10 monclient: _finish_auth 0 2024-04-26T00:25:55.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.157+0000 7f0e61ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:25:55.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.157+0000 7f0e61ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:55.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.157+0000 7f0e61ffb640 1 -- 172.21.15.37:0/3432015461 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e68017070 con 0x7f0e6c136da0 2024-04-26T00:25:55.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.157+0000 7f0e61ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:55.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.157+0000 7f0e61ffb640 1 -- 172.21.15.37:0/3432015461 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e68011d80 con 0x7f0e6c136da0 2024-04-26T00:25:55.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.157+0000 7f0e61ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:55.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.157+0000 7f0e61ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:55.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.157+0000 7f0e61ffb640 10 monclient: dump: 2024-04-26T00:25:55.149 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:55.149 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:55.149 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:55.149 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:55.149 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:55.149 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:55.149 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:55.150 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:55.150 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:55.150 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:55.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.157+0000 7f0e73302640 5 monclient: authenticate success, global_id 54267 2024-04-26T00:25:55.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.157+0000 7f0e637fe640 4 set_mon_vals no callback set 2024-04-26T00:25:55.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.157+0000 7f0e73302640 10 monclient: _renew_subs 2024-04-26T00:25:55.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.157+0000 7f0e73302640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:55.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.157+0000 7f0e73302640 1 -- 172.21.15.37:0/3432015461 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e6c09aca0 con 0x7f0e6c136da0 2024-04-26T00:25:55.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.157+0000 7f0e73302640 10 monclient: _renew_subs 2024-04-26T00:25:55.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.157+0000 7f0e73302640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:55.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.157+0000 7f0e73302640 1 -- 172.21.15.37:0/3432015461 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e6c120640 con 0x7f0e6c136da0 2024-04-26T00:25:55.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.157+0000 7f0e73302640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:55.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.157+0000 7f0e73302640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:55.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.157+0000 7f0e73302640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:55.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.157+0000 7f0e61ffb640 1 -- 172.21.15.37:0/3432015461 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f0e6801c460 con 0x7f0e6c136da0 2024-04-26T00:25:55.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.157+0000 7f0e73302640 1 -- 172.21.15.37:0/3432015461 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e6c066f40 con 0x7f0e6c136da0 2024-04-26T00:25:55.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.157+0000 7f0e61ffb640 1 --2- 172.21.15.37:0/3432015461 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0e50035ed0 0x7f0e50038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:55.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.157+0000 7f0e61ffb640 1 -- 172.21.15.37:0/3432015461 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(232..232 src has 177..232) ==== 4007+0+0 (secure 0 0 0) 0x7f0e68048980 con 0x7f0e6c136da0 2024-04-26T00:25:55.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.157+0000 7f0e7109e640 1 --2- 172.21.15.37:0/3432015461 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0e50035ed0 0x7f0e50038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:55.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.157+0000 7f0e7109e640 10 monclient: get_auth_request con 0x7f0e50035ed0 auth_method 0 2024-04-26T00:25:55.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.161+0000 7f0e7109e640 1 --2- 172.21.15.37:0/3432015461 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0e50035ed0 0x7f0e50038390 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c13c440 tx=0x7f0e5c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:55.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.165+0000 7f0e61ffb640 1 -- 172.21.15.37:0/3432015461 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f0e68047680 con 0x7f0e6c136da0 2024-04-26T00:25:55.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.165+0000 7f0e61ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:55.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.165+0000 7f0e61ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:25:55.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.389+0000 7f0e73302640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:25:55.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.389+0000 7f0e73302640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:25:55.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.389+0000 7f0e73302640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:55.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.389+0000 7f0e73302640 1 -- 172.21.15.37:0/3432015461 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0e6c11fa50 con 0x7f0e6c136da0 2024-04-26T00:25:55.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.389+0000 7f0e61ffb640 1 -- 172.21.15.37:0/3432015461 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f0e68012420 con 0x7f0e6c136da0 2024-04-26T00:25:55.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.389+0000 7f0e61ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:25:55.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.389+0000 7f0e61ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:25:55.373 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":46,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:25:55.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.389+0000 7f0e73302640 1 -- 172.21.15.37:0/3432015461 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0e50035ed0 msgr2=0x7f0e50038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:55.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.389+0000 7f0e73302640 1 --2- 172.21.15.37:0/3432015461 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0e50035ed0 0x7f0e50038390 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c13c440 tx=0x7f0e5c008040 comp rx=0 tx=0).stop 2024-04-26T00:25:55.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.389+0000 7f0e73302640 10 monclient: shutdown 2024-04-26T00:25:55.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.389+0000 7f0e73302640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:55.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.389+0000 7f0e73302640 1 -- 172.21.15.37:0/3432015461 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0e6c136da0 msgr2=0x7f0e6c135cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:55.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.389+0000 7f0e73302640 1 --2- 172.21.15.37:0/3432015461 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0e6c136da0 0x7f0e6c135cb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0e6800c410 tx=0x7f0e6800c8d0 comp rx=0 tx=0).stop 2024-04-26T00:25:55.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.393+0000 7f0e73302640 1 -- 172.21.15.37:0/3432015461 shutdown_connections 2024-04-26T00:25:55.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.393+0000 7f0e73302640 1 --2- 172.21.15.37:0/3432015461 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0e50035ed0 0x7f0e50038390 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:55.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.393+0000 7f0e73302640 1 --2- 172.21.15.37:0/3432015461 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e6c137890 0x7f0e6c13ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:55.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.393+0000 7f0e73302640 1 --2- 172.21.15.37:0/3432015461 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0e6c136da0 0x7f0e6c135cb0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:55.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.393+0000 7f0e73302640 1 --2- 172.21.15.37:0/3432015461 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e6c09c410 0x7f0e6c1ce900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:55.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.393+0000 7f0e73302640 1 -- 172.21.15.37:0/3432015461 >> 172.21.15.37:0/3432015461 conn(0x7f0e6c0a5f00 msgr2=0x7f0e6c0a7200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:55.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.393+0000 7f0e73302640 1 -- 172.21.15.37:0/3432015461 shutdown_connections 2024-04-26T00:25:55.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.393+0000 7f0e73302640 1 -- 172.21.15.37:0/3432015461 wait complete. 2024-04-26T00:25:55.391 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:25:55.391 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:25:55.391 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:25:55.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.533+0000 7f5654820640 10 monclient: get_monmap_and_config 2024-04-26T00:25:55.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.533+0000 7f5654820640 10 monclient: build_initial_monmap 2024-04-26T00:25:55.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.533+0000 7f5654820640 10 monclient: monmap: 2024-04-26T00:25:55.518 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:55.518 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:55.519 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:25:55.539741+0000 2024-04-26T00:25:55.519 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:25:55.539741+0000 2024-04-26T00:25:55.519 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:55.519 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:55.519 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:25:55.519 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:25:55.519 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:25:55.519 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:25:55.519 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:25:55.519 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:55.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.537+0000 7f5654820640 1 Processor -- start 2024-04-26T00:25:55.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.537+0000 7f5654820640 1 -- start start 2024-04-26T00:25:55.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.537+0000 7f5654820640 10 monclient: init 2024-04-26T00:25:55.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.537+0000 7f5654820640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:55.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.537+0000 7f5654820640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-04-26T00:25:55.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.537+0000 7f5654820640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f5650130eb0 0x7f565009ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:55.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.537+0000 7f5654820640 10 monclient(hunting): picked mon.noname-a con 0x7f5650130eb0 addr v2:172.21.15.37:3300/0 2024-04-26T00:25:55.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.537+0000 7f5654820640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5650131290 addr v1:172.21.15.45:6789/0 2024-04-26T00:25:55.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.537+0000 7f5654820640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f56501342a0 addr v1:172.21.15.37:6789/0 2024-04-26T00:25:55.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.537+0000 7f5654820640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f565009c190 con 0x7f56501342a0 2024-04-26T00:25:55.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.537+0000 7f5654820640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56500a11c0 con 0x7f5650131290 2024-04-26T00:25:55.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.537+0000 7f5654820640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:55.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.537+0000 7f5654820640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f565009df60 con 0x7f5650130eb0 2024-04-26T00:25:55.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.537+0000 7f5654820640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:55.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.537+0000 7f5654820640 10 monclient(hunting): authenticate will time out at 1336.845354s 2024-04-26T00:25:55.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.537+0000 7f564e59c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f5650130eb0 0x7f565009ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:55.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.537+0000 7f564ed9d640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f56501342a0 0x7f565013bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:39658/0 (socket says 172.21.15.37:39658) 2024-04-26T00:25:55.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.537+0000 7f564ed9d640 1 -- 172.21.15.37:0/4220295963 learned_addr learned my addr 172.21.15.37:0/4220295963 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:25:55.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.537+0000 7f564e59c640 10 monclient(hunting): get_auth_request con 0x7f5650130eb0 auth_method 0 2024-04-26T00:25:55.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.537+0000 7f564e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:55.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.537+0000 7f564e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:55.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.537+0000 7f564e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:55.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.537+0000 7f564e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:55.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.537+0000 7f564e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:55.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.537+0000 7f564e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:55.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.537+0000 7f564e59c640 10 monclient(hunting): handle_auth_done global_id 54270 payload 274 2024-04-26T00:25:55.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.537+0000 7f564e59c640 1 -- 172.21.15.37:0/4220295963 >> v1:172.21.15.45:6789/0 conn(0x7f5650131290 legacy=0x7f5650133b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:55.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.537+0000 7f564e59c640 1 -- 172.21.15.37:0/4220295963 >> v1:172.21.15.37:6789/0 conn(0x7f56501342a0 legacy=0x7f565013bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:55.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.537+0000 7f564e59c640 10 monclient: _finish_hunting 0 2024-04-26T00:25:55.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.537+0000 7f564e59c640 1 monclient: found mon.noname-a 2024-04-26T00:25:55.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.537+0000 7f564e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:55.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.537+0000 7f564e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:25:55.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.537+0000 7f564e59c640 1 -- 172.21.15.37:0/4220295963 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f565009dc80 con 0x7f5650130eb0 2024-04-26T00:25:55.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.541+0000 7f564e59c640 1 -- 172.21.15.37:0/4220295963 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5650130eb0 msgr2=0x7f565009ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:55.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.541+0000 7f564e59c640 1 --2- 172.21.15.37:0/4220295963 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5650130eb0 0x7f565009ef20 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5640009d90 tx=0x7f564001c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8131c60fd0ea10b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:55.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.541+0000 7f564d59a640 1 -- 172.21.15.37:0/4220295963 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56400b2070 con 0x7f5650130eb0 2024-04-26T00:25:55.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.541+0000 7f564d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:55.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.541+0000 7f564d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:25:55.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.541+0000 7f564d59a640 10 monclient: dump: 2024-04-26T00:25:55.526 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:55.526 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:55.526 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:55.526 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:55.526 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:55.526 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:55.526 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:55.526 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:55.527 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:55.527 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:55.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.541+0000 7f564d59a640 10 monclient: _finish_auth 0 2024-04-26T00:25:55.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.541+0000 7f564d59a640 10 monclient: _check_auth_tickets 2024-04-26T00:25:55.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.541+0000 7f564d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:55.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.541+0000 7f564d59a640 1 -- 172.21.15.37:0/4220295963 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56400a5860 con 0x7f5650130eb0 2024-04-26T00:25:55.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.541+0000 7f564d59a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:55.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.541+0000 7f564d59a640 1 -- 172.21.15.37:0/4220295963 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56400a5c30 con 0x7f5650130eb0 2024-04-26T00:25:55.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.541+0000 7f5654820640 5 monclient: authenticate success, global_id 54270 2024-04-26T00:25:55.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.541+0000 7f564d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:55.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.541+0000 7f564d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:55.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.541+0000 7f564d59a640 10 monclient: dump: 2024-04-26T00:25:55.527 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:55.528 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:55.528 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:55.528 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:55.528 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:55.528 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:55.528 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:55.528 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:55.528 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:55.528 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:55.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.541+0000 7f5654820640 10 monclient: get_monmap_and_config success 2024-04-26T00:25:55.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.541+0000 7f5654820640 4 set_mon_vals no callback set 2024-04-26T00:25:55.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.541+0000 7f5654820640 10 monclient: shutdown 2024-04-26T00:25:55.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.541+0000 7f5654820640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:55.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.541+0000 7f5654820640 1 -- 172.21.15.37:0/4220295963 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5650130eb0 msgr2=0x7f565009ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:55.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.541+0000 7f5654820640 1 --2- 172.21.15.37:0/4220295963 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5650130eb0 0x7f565009ef20 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5640009d90 tx=0x7f564001c7b0 comp rx=0 tx=0).stop 2024-04-26T00:25:55.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.541+0000 7f5654820640 1 -- 172.21.15.37:0/4220295963 shutdown_connections 2024-04-26T00:25:55.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.541+0000 7f5654820640 1 --2- 172.21.15.37:0/4220295963 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5650130eb0 0x7f565009ef20 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:55.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.541+0000 7f5654820640 1 -- 172.21.15.37:0/4220295963 >> 172.21.15.37:0/4220295963 conn(0x7f56500a1d50 msgr2=0x7f56500a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:55.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.541+0000 7f5654820640 1 -- 172.21.15.37:0/4220295963 shutdown_connections 2024-04-26T00:25:55.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.541+0000 7f5654820640 1 -- 172.21.15.37:0/4220295963 wait complete. 2024-04-26T00:25:55.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.541+0000 7f5654820640 10 monclient: build_initial_monmap 2024-04-26T00:25:55.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.541+0000 7f5654820640 10 monclient: monmap: 2024-04-26T00:25:55.535 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:55.535 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:25:55.535 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:25:55.535 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:25:55.535 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:55.535 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:25:55.535 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:25:55.535 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:25:55.535 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:25:55.535 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:55.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f5654820640 1 Processor -- start 2024-04-26T00:25:55.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f5654820640 1 -- start start 2024-04-26T00:25:55.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f5654820640 10 monclient: init 2024-04-26T00:25:55.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f5654820640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:55.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f5654820640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:25:55.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f5654820640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5650130eb0 0x7f565013c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:55.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f5654820640 10 monclient(hunting): picked mon.noname-a con 0x7f5650130eb0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:25:55.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f5654820640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5650131290 0x7f56501cf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:55.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f5654820640 10 monclient(hunting): picked mon.noname-b con 0x7f5650131290 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:25:55.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f5654820640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56501342a0 0x7f56501cfc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:55.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f5654820640 10 monclient(hunting): picked mon.noname-c con 0x7f56501342a0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:55.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f5654820640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:55.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f5654820640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f565013ff90 con 0x7f5650130eb0 2024-04-26T00:25:55.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f5654820640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:55.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f5654820640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f565013fe10 con 0x7f5650131290 2024-04-26T00:25:55.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f5654820640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:55.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f5654820640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5650140110 con 0x7f56501342a0 2024-04-26T00:25:55.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f5654820640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:55.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f5654820640 10 monclient(hunting): authenticate will time out at 1336.852786s 2024-04-26T00:25:55.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f564e59c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5650130eb0 0x7f565013c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:55.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f564ed9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56501342a0 0x7f56501cfc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:55.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f564e59c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5650130eb0 0x7f565013c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36590/0 (socket says 172.21.15.37:36590) 2024-04-26T00:25:55.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f564e59c640 1 -- 172.21.15.37:0/1174530093 learned_addr learned my addr 172.21.15.37:0/1174530093 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:55.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f564e59c640 10 monclient(hunting): get_auth_request con 0x7f5650130eb0 auth_method 0 2024-04-26T00:25:55.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f564e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:55.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f564e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:55.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f564e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:55.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f564ed9d640 10 monclient(hunting): get_auth_request con 0x7f56501342a0 auth_method 0 2024-04-26T00:25:55.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f564ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:55.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f564ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:55.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f564ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:55.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f564e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:55.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f564e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:55.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f564e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:55.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f564ed9d640 1 -- 172.21.15.37:0/1174530093 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56501342a0 msgr2=0x7f56501cfc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:25:55.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f564ed9d640 1 -- 172.21.15.37:0/1174530093 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56501342a0 msgr2=0x7f56501cfc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:25:55.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f564ed9d640 1 --2- 172.21.15.37:0/1174530093 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56501342a0 0x7f56501cfc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:25:55.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f564ed9d640 1 --2- 172.21.15.37:0/1174530093 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56501342a0 0x7f56501cfc00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:25:55.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f564dd9b640 1 --2- 172.21.15.37:0/1174530093 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5650131290 0x7f56501cf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:55.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f564dd9b640 10 monclient(hunting): get_auth_request con 0x7f5650131290 auth_method 0 2024-04-26T00:25:55.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f564dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:55.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f564dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:55.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f564dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:55.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f564e59c640 10 monclient(hunting): handle_auth_done global_id 54273 payload 824 2024-04-26T00:25:55.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f564e59c640 1 -- 172.21.15.37:0/1174530093 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56501342a0 msgr2=0x7f56501cfc00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:25:55.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f564e59c640 1 --2- 172.21.15.37:0/1174530093 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56501342a0 0x7f56501cfc00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:55.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f564e59c640 1 -- 172.21.15.37:0/1174530093 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5650131290 msgr2=0x7f56501cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:55.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f564e59c640 1 --2- 172.21.15.37:0/1174530093 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5650131290 0x7f56501cf6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:55.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f564e59c640 10 monclient: _finish_hunting 0 2024-04-26T00:25:55.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f564e59c640 1 monclient: found mon.noname-a 2024-04-26T00:25:55.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f564e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:55.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f564e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:25:55.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f564e59c640 1 -- 172.21.15.37:0/1174530093 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5650133630 con 0x7f5650130eb0 2024-04-26T00:25:55.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f564e59c640 1 -- 172.21.15.37:0/1174530093 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5650130eb0 msgr2=0x7f565013c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:55.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f564e59c640 1 --2- 172.21.15.37:0/1174530093 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5650130eb0 0x7f565013c410 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5640009d90 tx=0x7f5640019550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:55.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f563effd640 1 -- 172.21.15.37:0/1174530093 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56400b2050 con 0x7f5650130eb0 2024-04-26T00:25:55.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f563effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:55.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f563effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:25:55.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f563effd640 10 monclient: dump: 2024-04-26T00:25:55.540 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:55.540 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:55.540 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:55.540 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:55.540 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:55.540 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:55.540 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:55.540 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:55.540 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:55.541 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:55.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f563effd640 10 monclient: _finish_auth 0 2024-04-26T00:25:55.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f563effd640 10 monclient: _check_auth_tickets 2024-04-26T00:25:55.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f563effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:55.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f5654820640 5 monclient: authenticate success, global_id 54273 2024-04-26T00:25:55.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f563effd640 1 -- 172.21.15.37:0/1174530093 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56400be070 con 0x7f5650130eb0 2024-04-26T00:25:55.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f563effd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:55.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f563effd640 1 -- 172.21.15.37:0/1174530093 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56400adc60 con 0x7f5650130eb0 2024-04-26T00:25:55.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f563effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:55.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f563effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:55.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f563effd640 10 monclient: dump: 2024-04-26T00:25:55.541 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:55.541 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:55.541 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:55.542 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:55.542 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:55.542 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:55.542 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:55.542 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:55.542 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:55.542 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:55.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f564cd99640 4 set_mon_vals no callback set 2024-04-26T00:25:55.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f5654820640 10 monclient: _renew_subs 2024-04-26T00:25:55.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f5654820640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:55.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f5654820640 1 -- 172.21.15.37:0/1174530093 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f565013b3f0 con 0x7f5650130eb0 2024-04-26T00:25:55.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f5654820640 10 monclient: _renew_subs 2024-04-26T00:25:55.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f5654820640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:55.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.545+0000 7f5654820640 1 -- 172.21.15.37:0/1174530093 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f565013b9b0 con 0x7f5650130eb0 2024-04-26T00:25:55.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.549+0000 7f5654820640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:55.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.549+0000 7f5654820640 1 --2- 172.21.15.37:0/1174530093 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56501d0140 0x7f565012dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:55.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.549+0000 7f5654820640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:55.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.549+0000 7f5654820640 1 -- 172.21.15.37:0/1174530093 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f56501d0520 con 0x7f56501d0140 2024-04-26T00:25:55.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.549+0000 7f563effd640 1 -- 172.21.15.37:0/1174530093 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f56400c3460 con 0x7f5650130eb0 2024-04-26T00:25:55.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.549+0000 7f564dd9b640 1 --2- 172.21.15.37:0/1174530093 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56501d0140 0x7f565012dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:55.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.549+0000 7f564dd9b640 10 monclient: get_auth_request con 0x7f56501d0140 auth_method 0 2024-04-26T00:25:55.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.549+0000 7f564dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:55.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.549+0000 7f564dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:55.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.549+0000 7f564dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:55.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.549+0000 7f564dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:55.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.549+0000 7f564dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:55.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.549+0000 7f564dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:55.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.549+0000 7f563effd640 1 --2- 172.21.15.37:0/1174530093 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5630035e80 0x7f5630038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:55.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.549+0000 7f563effd640 1 -- 172.21.15.37:0/1174530093 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(232..232 src has 177..232) ==== 4007+0+0 (secure 0 0 0) 0x7f56400e3d40 con 0x7f5650130eb0 2024-04-26T00:25:55.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.549+0000 7f564dd9b640 10 monclient(hunting): handle_auth_done global_id 54276 payload 824 2024-04-26T00:25:55.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.549+0000 7f564dd9b640 1 -- 172.21.15.37:0/1174530093 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f565004f510 con 0x7f56501d0140 2024-04-26T00:25:55.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.549+0000 7f564ed9d640 1 --2- 172.21.15.37:0/1174530093 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5630035e80 0x7f5630038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:55.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.549+0000 7f564ed9d640 10 monclient: get_auth_request con 0x7f5630035e80 auth_method 0 2024-04-26T00:25:55.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.549+0000 7f564dd9b640 1 -- 172.21.15.37:0/1174530093 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56501d0140 msgr2=0x7f565012dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:55.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.549+0000 7f564dd9b640 1 --2- 172.21.15.37:0/1174530093 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56501d0140 0x7f565012dae0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5650072c60 tx=0x7f564400e8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:55.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.549+0000 7f563effd640 1 -- 172.21.15.37:0/1174530093 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5644007450 con 0x7f56501d0140 2024-04-26T00:25:55.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.549+0000 7f563effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:55.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.549+0000 7f563effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:55.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.549+0000 7f563effd640 10 monclient: dump: 2024-04-26T00:25:55.545 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:55.545 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:55.545 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:55.545 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:55.545 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:55.545 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:55.545 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:55.545 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:55.545 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:55.545 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:55.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.549+0000 7f563effd640 1 -- 172.21.15.37:0/1174530093 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5644007450 con 0x7f56501d0140 2024-04-26T00:25:55.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.549+0000 7f563effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:55.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.549+0000 7f563effd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:25:55.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.549+0000 7f563effd640 1 -- 172.21.15.37:0/1174530093 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56501d0140 msgr2=0x7f565012dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:55.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.549+0000 7f563effd640 1 --2- 172.21.15.37:0/1174530093 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56501d0140 0x7f565012dae0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5650072c60 tx=0x7f564400e8b0 comp rx=0 tx=0).stop 2024-04-26T00:25:55.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.549+0000 7f563effd640 1 -- 172.21.15.37:0/1174530093 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56501d0140 msgr2=0x7f565012dae0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:25:55.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.549+0000 7f563effd640 1 --2- 172.21.15.37:0/1174530093 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56501d0140 0x7f565012dae0 secure :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5650072c60 tx=0x7f564400e8b0 comp rx=0 tx=0).stop 2024-04-26T00:25:55.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.553+0000 7f564ed9d640 1 --2- 172.21.15.37:0/1174530093 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5630035e80 0x7f5630038340 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5648007cd0 tx=0x7f5648009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:55.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.557+0000 7f5654820640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:25:55.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.557+0000 7f5654820640 1 --2- 172.21.15.37:0/1174530093 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f565012e020 0x7f565012e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:55.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.557+0000 7f5654820640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:55.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.557+0000 7f5654820640 1 -- 172.21.15.37:0/1174530093 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f565012e940 con 0x7f565012e020 2024-04-26T00:25:55.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.557+0000 7f564dd9b640 1 --2- 172.21.15.37:0/1174530093 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f565012e020 0x7f565012e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:55.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.557+0000 7f564dd9b640 10 monclient: get_auth_request con 0x7f565012e020 auth_method 0 2024-04-26T00:25:55.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.557+0000 7f564dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:55.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.557+0000 7f564dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:55.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.557+0000 7f564dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:55.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.557+0000 7f564dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:55.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.557+0000 7f564dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:55.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.557+0000 7f564dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:55.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.557+0000 7f564dd9b640 10 monclient(hunting): handle_auth_done global_id 54279 payload 824 2024-04-26T00:25:55.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.557+0000 7f564dd9b640 1 -- 172.21.15.37:0/1174530093 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f56500a0f60 con 0x7f565012e020 2024-04-26T00:25:55.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.561+0000 7f564dd9b640 1 -- 172.21.15.37:0/1174530093 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f565012e020 msgr2=0x7f565012e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:55.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.561+0000 7f564dd9b640 1 --2- 172.21.15.37:0/1174530093 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f565012e020 0x7f565012e400 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5650072c60 tx=0x7f5644003770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:55.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.561+0000 7f563effd640 1 -- 172.21.15.37:0/1174530093 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f564401e960 con 0x7f565012e020 2024-04-26T00:25:55.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.561+0000 7f563effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:55.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.561+0000 7f563effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:55.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.561+0000 7f563effd640 10 monclient: dump: 2024-04-26T00:25:55.549 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:55.549 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:55.549 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:55.549 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:55.550 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:55.550 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:55.550 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:55.550 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:55.550 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:55.550 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:55.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.561+0000 7f563effd640 1 -- 172.21.15.37:0/1174530093 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f5644028420 con 0x7f565012e020 2024-04-26T00:25:55.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.561+0000 7f563effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:25:55.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.561+0000 7f563effd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:25:55.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.561+0000 7f563effd640 1 -- 172.21.15.37:0/1174530093 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f565012e020 msgr2=0x7f565012e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:55.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.561+0000 7f563effd640 1 --2- 172.21.15.37:0/1174530093 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f565012e020 0x7f565012e400 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5650072c60 tx=0x7f5644003770 comp rx=0 tx=0).stop 2024-04-26T00:25:55.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.561+0000 7f563effd640 1 -- 172.21.15.37:0/1174530093 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f565012e020 msgr2=0x7f565012e400 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:25:55.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.561+0000 7f563effd640 1 --2- 172.21.15.37:0/1174530093 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f565012e020 0x7f565012e400 secure :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5650072c60 tx=0x7f5644003770 comp rx=0 tx=0).stop 2024-04-26T00:25:55.551 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:25:55.551 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:25:55.551 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:25:55.551 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:25:55.551 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 46, 2024-04-26T00:25:55.551 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 118886, 2024-04-26T00:25:55.551 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:25:55.551 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:25:55.551 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:25:55.552 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:25:55.552 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:55.552 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 2, 2024-04-26T00:25:55.552 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:25:55.552 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:25:55.552 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:25:55.552 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:55.552 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:55.552 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:55.552 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:55.552 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:55.552 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:55.552 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:55.552 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:55.553 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:55.553 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:55.553 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:55.553 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:55.553 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:25:55.553 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:25:55.553 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:55.553 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:55.553 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:55.553 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:55.553 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:55.553 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:55.553 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:55.553 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:55.554 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:55.554 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:55.554 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:55.554 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:55.554 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:55.554 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:25:55.554 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:25:55.554 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:25:55.554 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:25:55.554 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:25:55.554 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:25:55.554 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:25:55.554 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:25:55.555 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:25:55.555 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:25:55.555 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:25:55.555 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:25:55.555 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:25:55.555 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:25:55.555 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:25:55.555 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:25:55.555 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:25:55.555 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:55.555 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:55.555 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:55.555 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:55.556 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:55.556 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:55.556 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:55.556 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:55.556 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:55.556 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:55.556 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:55.556 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:55.556 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:25:55.556 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:55.556 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:25:55.556 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:55.556 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:25:55.557 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:25:55.557 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:55.557 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:55.557 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:55.557 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:55.557 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:25:55.557 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:55.557 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:55.557 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:55.557 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:55.557 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:25:55.557 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:55.557 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:55.557 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:55.558 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:55.558 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:25:55.558 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:25:55.558 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:55.558 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:55.558 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:55.558 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:55.558 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:55.558 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:25:55.558 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:25:55.558 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:55.558 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:55.558 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:55.558 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:55.559 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:25:55.559 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:55.559 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:55.559 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:55.559 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:55.559 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:25:55.559 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:55.559 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:55.559 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:55.559 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:55.559 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:25:55.559 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:25:55.559 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:55.559 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:55.560 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:55.560 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:55.560 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:55.560 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:25:55.560 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:25:55.560 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:55.560 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:55.560 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:55.560 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:55.560 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:25:55.560 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:55.560 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:55.560 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:55.560 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:55.560 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:25:55.561 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:55.561 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:55.561 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:55.561 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:55.561 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:25:55.561 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:25:55.561 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:55.561 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:55.561 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:55.561 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:55.561 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:55.561 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:55.561 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:25:55.561 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:25:55.561 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:55.562 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:55.562 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:55.562 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:25:55.562 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:55.562 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:55.562 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:25:55.562 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:55.562 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:55.562 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:55.562 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:25:55.562 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:55.562 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:55.562 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:25:55.562 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:55.563 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:55.563 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:55.563 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:25:55.563 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:55.563 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:55.563 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:25:55.563 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:55.563 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:55.563 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:55.563 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:25:55.563 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:55.563 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:55.563 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:55.563 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:25:55.564 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:25:55.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.561+0000 7f5654820640 1 -- 172.21.15.37:0/1174530093 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5630035e80 msgr2=0x7f5630038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:55.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.561+0000 7f5654820640 1 --2- 172.21.15.37:0/1174530093 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5630035e80 0x7f5630038340 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5648007cd0 tx=0x7f5648009040 comp rx=0 tx=0).stop 2024-04-26T00:25:55.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.561+0000 7f5654820640 10 monclient: shutdown 2024-04-26T00:25:55.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.561+0000 7f5654820640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:55.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.561+0000 7f5654820640 1 -- 172.21.15.37:0/1174530093 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5650130eb0 msgr2=0x7f565013c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:55.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.561+0000 7f5654820640 1 --2- 172.21.15.37:0/1174530093 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5650130eb0 0x7f565013c410 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5640009d90 tx=0x7f5640019550 comp rx=0 tx=0).stop 2024-04-26T00:25:55.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.561+0000 7f564ed9d640 1 -- 172.21.15.37:0/1174530093 reap_dead start 2024-04-26T00:25:55.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.561+0000 7f564ed9d640 1 -- 172.21.15.37:0/1174530093 reap_dead start 2024-04-26T00:25:55.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.561+0000 7f5654820640 1 -- 172.21.15.37:0/1174530093 shutdown_connections 2024-04-26T00:25:55.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.561+0000 7f5654820640 1 -- 172.21.15.37:0/1174530093 >> 172.21.15.37:0/1174530093 conn(0x7f56500a1d50 msgr2=0x7f565009fb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:55.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.561+0000 7f5654820640 1 -- 172.21.15.37:0/1174530093 shutdown_connections 2024-04-26T00:25:55.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.561+0000 7f5654820640 1 -- 172.21.15.37:0/1174530093 wait complete. 2024-04-26T00:25:55.566 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:25:55.566 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:25:55.566 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:25:55.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.697+0000 7ff03ee87640 10 monclient: get_monmap_and_config 2024-04-26T00:25:55.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.697+0000 7ff03ee87640 10 monclient: build_initial_monmap 2024-04-26T00:25:55.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.697+0000 7ff03ee87640 10 monclient: monmap: 2024-04-26T00:25:55.683 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:55.683 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:55.683 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:25:55.703355+0000 2024-04-26T00:25:55.683 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:25:55.703355+0000 2024-04-26T00:25:55.683 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:55.683 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:55.683 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:25:55.683 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:25:55.683 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:25:55.683 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:25:55.684 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:25:55.684 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:55.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.701+0000 7ff03ee87640 1 Processor -- start 2024-04-26T00:25:55.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.701+0000 7ff03ee87640 1 -- start start 2024-04-26T00:25:55.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.701+0000 7ff03ee87640 10 monclient: init 2024-04-26T00:25:55.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.701+0000 7ff03ee87640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:55.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.701+0000 7ff03ee87640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-04-26T00:25:55.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.701+0000 7ff03ee87640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff0380a0300 0x7ff03809fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:55.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.701+0000 7ff03ee87640 10 monclient(hunting): picked mon.noname-a con 0x7ff0380a0300 addr v2:172.21.15.37:3300/0 2024-04-26T00:25:55.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.701+0000 7ff03ee87640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff0380a06e0 addr v1:172.21.15.45:6789/0 2024-04-26T00:25:55.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.701+0000 7ff03ee87640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff0381352b0 addr v1:172.21.15.37:6789/0 2024-04-26T00:25:55.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.701+0000 7ff03ee87640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff03809c190 con 0x7ff0381352b0 2024-04-26T00:25:55.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.701+0000 7ff03ee87640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0380a1110 con 0x7ff0380a06e0 2024-04-26T00:25:55.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.701+0000 7ff03ee87640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:55.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.701+0000 7ff03ee87640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7ff0380a13f0 con 0x7ff0380a0300 2024-04-26T00:25:55.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.701+0000 7ff03ee87640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:55.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.701+0000 7ff03ee87640 10 monclient(hunting): authenticate will time out at 1337.008989s 2024-04-26T00:25:55.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.701+0000 7ff03cc23640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff0380a0300 0x7ff03809fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:55.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.701+0000 7ff03d424640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7ff0381352b0 0x7ff03813cee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:39674/0 (socket says 172.21.15.37:39674) 2024-04-26T00:25:55.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.701+0000 7ff03d424640 1 -- 172.21.15.37:0/581935847 learned_addr learned my addr 172.21.15.37:0/581935847 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:25:55.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.701+0000 7ff03cc23640 10 monclient(hunting): get_auth_request con 0x7ff0380a0300 auth_method 0 2024-04-26T00:25:55.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.701+0000 7ff03cc23640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:55.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.701+0000 7ff03cc23640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:55.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.701+0000 7ff03cc23640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:55.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.701+0000 7ff03cc23640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:55.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.701+0000 7ff03cc23640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:55.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.701+0000 7ff03cc23640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:55.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.701+0000 7ff03cc23640 10 monclient(hunting): handle_auth_done global_id 54282 payload 274 2024-04-26T00:25:55.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.701+0000 7ff03cc23640 1 -- 172.21.15.37:0/581935847 >> v1:172.21.15.45:6789/0 conn(0x7ff0380a06e0 legacy=0x7ff038134ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:55.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.701+0000 7ff03cc23640 1 -- 172.21.15.37:0/581935847 >> v1:172.21.15.37:6789/0 conn(0x7ff0381352b0 legacy=0x7ff03813cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:55.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.701+0000 7ff03cc23640 10 monclient: _finish_hunting 0 2024-04-26T00:25:55.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.701+0000 7ff03cc23640 1 monclient: found mon.noname-a 2024-04-26T00:25:55.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.701+0000 7ff03cc23640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:55.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.701+0000 7ff03cc23640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:25:55.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.701+0000 7ff03cc23640 1 -- 172.21.15.37:0/581935847 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0380a16d0 con 0x7ff0380a0300 2024-04-26T00:25:55.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.705+0000 7ff03cc23640 1 -- 172.21.15.37:0/581935847 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0380a0300 msgr2=0x7ff03809fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:55.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.705+0000 7ff03cc23640 1 --2- 172.21.15.37:0/581935847 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0380a0300 0x7ff03809fc50 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff02c0097c0 tx=0x7ff02c01c310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fc29e70e5575a8c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:55.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.705+0000 7ff0377fe640 1 -- 172.21.15.37:0/581935847 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff02c0b0070 con 0x7ff0380a0300 2024-04-26T00:25:55.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.705+0000 7ff0377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:55.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.705+0000 7ff0377fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:25:55.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.705+0000 7ff0377fe640 10 monclient: dump: 2024-04-26T00:25:55.691 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:55.691 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:55.691 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:55.691 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:55.691 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:55.691 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:55.691 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:55.691 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:55.691 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:55.691 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:55.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.705+0000 7ff0377fe640 10 monclient: _finish_auth 0 2024-04-26T00:25:55.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.705+0000 7ff0377fe640 10 monclient: _check_auth_tickets 2024-04-26T00:25:55.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.705+0000 7ff0377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:55.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.705+0000 7ff03ee87640 5 monclient: authenticate success, global_id 54282 2024-04-26T00:25:55.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.705+0000 7ff03ee87640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:25:55.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.705+0000 7ff0377fe640 1 -- 172.21.15.37:0/581935847 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff02c0ab600 con 0x7ff0380a0300 2024-04-26T00:25:55.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.705+0000 7ff0377fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:55.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.705+0000 7ff03ee87640 10 monclient: get_monmap_and_config success 2024-04-26T00:25:55.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.705+0000 7ff0377fe640 1 -- 172.21.15.37:0/581935847 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff02c0ab9d0 con 0x7ff0380a0300 2024-04-26T00:25:55.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.705+0000 7ff03ee87640 4 set_mon_vals no callback set 2024-04-26T00:25:55.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.705+0000 7ff0377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:55.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.705+0000 7ff0377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:55.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.705+0000 7ff0377fe640 10 monclient: dump: 2024-04-26T00:25:55.692 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:55.692 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:55.692 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:55.692 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:55.692 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:55.693 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:55.693 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:55.693 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:55.693 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:55.693 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:55.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03ee87640 10 monclient: shutdown 2024-04-26T00:25:55.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03ee87640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:55.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03ee87640 1 -- 172.21.15.37:0/581935847 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0380a0300 msgr2=0x7ff03809fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:55.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03ee87640 1 --2- 172.21.15.37:0/581935847 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0380a0300 0x7ff03809fc50 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff02c0097c0 tx=0x7ff02c01c310 comp rx=0 tx=0).stop 2024-04-26T00:25:55.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03ee87640 1 -- 172.21.15.37:0/581935847 shutdown_connections 2024-04-26T00:25:55.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03ee87640 1 --2- 172.21.15.37:0/581935847 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0380a0300 0x7ff03809fc50 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:55.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03ee87640 1 -- 172.21.15.37:0/581935847 >> 172.21.15.37:0/581935847 conn(0x7ff0380a1d50 msgr2=0x7ff0380a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:55.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03ee87640 1 -- 172.21.15.37:0/581935847 shutdown_connections 2024-04-26T00:25:55.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03ee87640 1 -- 172.21.15.37:0/581935847 wait complete. 2024-04-26T00:25:55.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03ee87640 10 monclient: build_initial_monmap 2024-04-26T00:25:55.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03ee87640 10 monclient: monmap: 2024-04-26T00:25:55.699 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:55.699 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:25:55.699 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:25:55.699 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:25:55.699 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:55.699 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:25:55.699 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:25:55.699 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:25:55.699 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:25:55.699 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:55.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03ee87640 1 Processor -- start 2024-04-26T00:25:55.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03ee87640 1 -- start start 2024-04-26T00:25:55.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03ee87640 10 monclient: init 2024-04-26T00:25:55.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03ee87640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:55.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03ee87640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:25:55.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03ee87640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0380a0300 0x7ff038136c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:55.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03ee87640 10 monclient(hunting): picked mon.noname-b con 0x7ff0380a0300 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:25:55.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03ee87640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0380a06e0 0x7ff0381371a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:55.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03ee87640 10 monclient(hunting): picked mon.noname-c con 0x7ff0380a06e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:55.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03ee87640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0381352b0 0x7ff0381376e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:55.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03ee87640 10 monclient(hunting): picked mon.noname-a con 0x7ff0381352b0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:25:55.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03ee87640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:55.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03ee87640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff038140fa0 con 0x7ff0381352b0 2024-04-26T00:25:55.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03ee87640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:55.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03ee87640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff038140e20 con 0x7ff0380a0300 2024-04-26T00:25:55.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03ee87640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:55.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03ee87640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff038141120 con 0x7ff0380a06e0 2024-04-26T00:25:55.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03ee87640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:55.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03ee87640 10 monclient(hunting): authenticate will time out at 1337.017886s 2024-04-26T00:25:55.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03d424640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0381352b0 0x7ff0381376e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:55.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff037fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0380a06e0 0x7ff0381371a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:55.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03d424640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0381352b0 0x7ff0381376e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36630/0 (socket says 172.21.15.37:36630) 2024-04-26T00:25:55.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03d424640 1 -- 172.21.15.37:0/3560665159 learned_addr learned my addr 172.21.15.37:0/3560665159 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:55.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03d424640 10 monclient(hunting): get_auth_request con 0x7ff0381352b0 auth_method 0 2024-04-26T00:25:55.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03d424640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:55.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03d424640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:55.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03d424640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:55.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff037fff640 10 monclient(hunting): get_auth_request con 0x7ff0380a06e0 auth_method 0 2024-04-26T00:25:55.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff037fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:55.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff037fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:55.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff037fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:55.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03d424640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:55.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03d424640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:55.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03d424640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:55.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff037fff640 1 -- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0380a06e0 msgr2=0x7ff0381371a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:25:55.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff037fff640 1 -- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0380a06e0 msgr2=0x7ff0381371a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:25:55.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff037fff640 1 --2- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0380a06e0 0x7ff0381371a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:25:55.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03cc23640 1 --2- 172.21.15.37:0/3560665159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0380a0300 0x7ff038136c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:55.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff037fff640 1 --2- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0380a06e0 0x7ff0381371a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:25:55.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03cc23640 10 monclient(hunting): get_auth_request con 0x7ff0380a0300 auth_method 0 2024-04-26T00:25:55.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03cc23640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:55.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03cc23640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:55.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03cc23640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:55.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03d424640 10 monclient(hunting): handle_auth_done global_id 54285 payload 824 2024-04-26T00:25:55.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03d424640 1 -- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0380a06e0 msgr2=0x7ff0381371a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:25:55.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03d424640 1 --2- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0380a06e0 0x7ff0381371a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:55.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03d424640 1 -- 172.21.15.37:0/3560665159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0380a0300 msgr2=0x7ff038136c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:55.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03d424640 1 --2- 172.21.15.37:0/3560665159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0380a0300 0x7ff038136c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:55.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03d424640 10 monclient: _finish_hunting 0 2024-04-26T00:25:55.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03d424640 1 monclient: found mon.noname-a 2024-04-26T00:25:55.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03d424640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:55.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03d424640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:25:55.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03d424640 1 -- 172.21.15.37:0/3560665159 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff03813aa50 con 0x7ff0381352b0 2024-04-26T00:25:55.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.709+0000 7ff03cc23640 1 --2- 172.21.15.37:0/3560665159 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0380a0300 0x7ff038136c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:25:55.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff03d424640 1 -- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0381352b0 msgr2=0x7ff0381376e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:55.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff03d424640 1 --2- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0381352b0 0x7ff0381376e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff03000caf0 tx=0x7ff03000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:55.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff0357fa640 1 -- 172.21.15.37:0/3560665159 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff030012400 con 0x7ff0381352b0 2024-04-26T00:25:55.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff0357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:55.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff0357fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:25:55.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff0357fa640 10 monclient: dump: 2024-04-26T00:25:55.703 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:55.703 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:55.703 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:55.704 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:55.704 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:55.704 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:55.704 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:55.704 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:55.704 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:55.704 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:55.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff0357fa640 10 monclient: _finish_auth 0 2024-04-26T00:25:55.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff0357fa640 10 monclient: _check_auth_tickets 2024-04-26T00:25:55.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff0357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:55.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff0357fa640 1 -- 172.21.15.37:0/3560665159 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0300125c0 con 0x7ff0381352b0 2024-04-26T00:25:55.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff0357fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:55.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff0357fa640 1 -- 172.21.15.37:0/3560665159 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0300129b0 con 0x7ff0381352b0 2024-04-26T00:25:55.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff0357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:55.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff0357fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:55.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff0357fa640 10 monclient: dump: 2024-04-26T00:25:55.705 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:55.705 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:55.705 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:55.705 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:55.705 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:55.705 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:55.705 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:55.705 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:55.705 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:55.705 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:55.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff036ffd640 4 set_mon_vals no callback set 2024-04-26T00:25:55.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff03ee87640 5 monclient: authenticate success, global_id 54285 2024-04-26T00:25:55.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff03ee87640 10 monclient: _renew_subs 2024-04-26T00:25:55.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff03ee87640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:55.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff03ee87640 1 -- 172.21.15.37:0/3560665159 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff03806e580 con 0x7ff0381352b0 2024-04-26T00:25:55.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff03ee87640 10 monclient: _renew_subs 2024-04-26T00:25:55.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff03ee87640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:55.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff03ee87640 1 -- 172.21.15.37:0/3560665159 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff03813a1b0 con 0x7ff0381352b0 2024-04-26T00:25:55.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff03ee87640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:55.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff03ee87640 1 --2- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0381cd2b0 0x7ff0381cd690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:55.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff03ee87640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:55.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff03ee87640 1 -- 172.21.15.37:0/3560665159 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff038139380 con 0x7ff0381cd2b0 2024-04-26T00:25:55.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff03cc23640 1 --2- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0381cd2b0 0x7ff0381cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:55.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff03cc23640 10 monclient: get_auth_request con 0x7ff0381cd2b0 auth_method 0 2024-04-26T00:25:55.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff03cc23640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:55.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff03cc23640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:55.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff03cc23640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:55.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff0357fa640 1 -- 172.21.15.37:0/3560665159 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff030012bd0 con 0x7ff0381352b0 2024-04-26T00:25:55.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff03cc23640 1 -- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0381cd2b0 msgr2=0x7ff0381cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:25:55.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff03cc23640 1 -- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0381cd2b0 msgr2=0x7ff0381cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:25:55.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff03cc23640 1 --2- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0381cd2b0 0x7ff0381cd690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:25:55.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff03cc23640 1 --2- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0381cd2b0 0x7ff0381cd690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:25:55.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff0357fa640 1 --2- 172.21.15.37:0/3560665159 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff024035ed0 0x7ff024038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:55.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff0357fa640 1 -- 172.21.15.37:0/3560665159 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(232..232 src has 177..232) ==== 4007+0+0 (secure 0 0 0) 0x7ff030049200 con 0x7ff0381352b0 2024-04-26T00:25:55.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff037fff640 1 --2- 172.21.15.37:0/3560665159 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff024035ed0 0x7ff024038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:55.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.713+0000 7ff037fff640 10 monclient: get_auth_request con 0x7ff024035ed0 auth_method 0 2024-04-26T00:25:55.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.717+0000 7ff037fff640 1 --2- 172.21.15.37:0/3560665159 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff024035ed0 0x7ff024038390 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff0381383e0 tx=0x7ff028009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:55.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.917+0000 7ff03cc23640 1 --2- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0381cd2b0 0x7ff0381cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:55.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.917+0000 7ff03cc23640 10 monclient: get_auth_request con 0x7ff0381cd2b0 auth_method 0 2024-04-26T00:25:55.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.917+0000 7ff03cc23640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:55.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.917+0000 7ff03cc23640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:55.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.917+0000 7ff03cc23640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:25:55.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.917+0000 7ff03cc23640 1 -- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0381cd2b0 msgr2=0x7ff0381cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:25:55.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.917+0000 7ff03cc23640 1 -- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0381cd2b0 msgr2=0x7ff0381cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:25:55.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.917+0000 7ff03cc23640 1 --2- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0381cd2b0 0x7ff0381cd690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:25:55.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:55.917+0000 7ff03cc23640 1 --2- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0381cd2b0 0x7ff0381cd690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:25:55.938 INFO:tasks.rados.rados.0.smithi045.stdout:967: done (7 left) 2024-04-26T00:25:55.939 INFO:tasks.rados.rados.0.smithi045.stdout:968: snap_remove snap 93 2024-04-26T00:25:56.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:56.321+0000 7ff03cc23640 1 --2- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0381cd2b0 0x7ff0381cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:56.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:56.321+0000 7ff03cc23640 10 monclient: get_auth_request con 0x7ff0381cd2b0 auth_method 0 2024-04-26T00:25:56.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:56.321+0000 7ff03cc23640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:56.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:56.321+0000 7ff03cc23640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:56.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:56.321+0000 7ff03cc23640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:25:56.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:56.321+0000 7ff03cc23640 1 -- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0381cd2b0 msgr2=0x7ff0381cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:25:56.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:56.321+0000 7ff03cc23640 1 -- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0381cd2b0 msgr2=0x7ff0381cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:25:56.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:56.321+0000 7ff03cc23640 1 --2- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0381cd2b0 0x7ff0381cd690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:25:56.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:56.321+0000 7ff03cc23640 1 --2- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0381cd2b0 0x7ff0381cd690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:25:56.878 INFO:tasks.rados.rados.0.smithi045.stdout:968: done (7 left) 2024-04-26T00:25:56.879 INFO:tasks.rados.rados.0.smithi045.stdout:969: copy_from oid 3 from oid 42 current snap is 114 2024-04-26T00:25:56.879 INFO:tasks.rados.rados.0.smithi045.stdout:960: expect (ObjNum 316 snap 93 seq_num 316) 2024-04-26T00:25:57.106 INFO:tasks.rados.rados.0.smithi045.stdout:961: finishing rollback tid 0 to smithi04526579-39 2024-04-26T00:25:57.106 INFO:tasks.rados.rados.0.smithi045.stdout:961: finishing rollback tid 1 to smithi04526579-39 2024-04-26T00:25:57.106 INFO:tasks.rados.rados.0.smithi045.stdout:961: finishing rollback tid 2 to smithi04526579-39 2024-04-26T00:25:57.106 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 336 (ObjNum 155 snap 33 seq_num 155) dirty exists 2024-04-26T00:25:57.106 INFO:tasks.rados.rados.0.smithi045.stdout:962: expect (ObjNum 333 snap 99 seq_num 333) 2024-04-26T00:25:57.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.125+0000 7ff03cc23640 1 --2- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0381cd2b0 0x7ff0381cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:57.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.125+0000 7ff03cc23640 10 monclient: get_auth_request con 0x7ff0381cd2b0 auth_method 0 2024-04-26T00:25:57.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.125+0000 7ff03cc23640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:57.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.125+0000 7ff03cc23640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:57.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.125+0000 7ff03cc23640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:25:57.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.125+0000 7ff03cc23640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:57.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.125+0000 7ff03cc23640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:57.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.125+0000 7ff03cc23640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:57.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.125+0000 7ff03cc23640 10 monclient(hunting): handle_auth_done global_id 74099 payload 824 2024-04-26T00:25:57.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.125+0000 7ff03cc23640 1 -- 172.21.15.37:0/3560665159 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff03813a620 con 0x7ff0381cd2b0 2024-04-26T00:25:57.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.125+0000 7ff03cc23640 1 -- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0381cd2b0 msgr2=0x7ff0381cd690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:57.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.129+0000 7ff03cc23640 1 --2- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0381cd2b0 0x7ff0381cd690 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff02c0ad000 tx=0x7ff02c01abb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:57.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.129+0000 7ff03cc23640 1 -- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0381cd2b0 msgr2=0x7ff0381cd690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:25:57.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.129+0000 7ff03cc23640 1 -- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0381cd2b0 msgr2=0x7ff0381cd690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:25:57.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.129+0000 7ff03cc23640 1 --2- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0381cd2b0 0x7ff0381cd690 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff02c0ad000 tx=0x7ff02c01abb0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:25:57.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.129+0000 7ff03cc23640 1 --2- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0381cd2b0 0x7ff0381cd690 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff02c0ad000 tx=0x7ff02c01abb0 comp rx=0 tx=0).stop 2024-04-26T00:25:57.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.129+0000 7ff0357fa640 1 -- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0381cd2b0 msgr2=0x7ff0381cd690 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:25:57.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.129+0000 7ff0357fa640 1 --2- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0381cd2b0 0x7ff0381cd690 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:57.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.129+0000 7ff0357fa640 1 --2- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff02403bcb0 0x7ff02403e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:57.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.129+0000 7ff0357fa640 1 -- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0381cd2b0 msgr2=0x7ff0381cd690 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:25:57.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.129+0000 7ff0357fa640 1 --2- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0381cd2b0 0x7ff0381cd690 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:57.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.129+0000 7ff0357fa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:57.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.129+0000 7ff0357fa640 1 -- 172.21.15.37:0/3560665159 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff02403e6e0 con 0x7ff02403bcb0 2024-04-26T00:25:57.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.129+0000 7ff03cc23640 1 --2- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff02403bcb0 0x7ff02403e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:57.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.129+0000 7ff03cc23640 10 monclient: get_auth_request con 0x7ff02403bcb0 auth_method 0 2024-04-26T00:25:57.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.129+0000 7ff03cc23640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:57.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.129+0000 7ff03cc23640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:57.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.129+0000 7ff03cc23640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:57.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.129+0000 7ff03cc23640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:57.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.129+0000 7ff03cc23640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:57.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.129+0000 7ff03cc23640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:57.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.129+0000 7ff03cc23640 10 monclient(hunting): handle_auth_done global_id 74102 payload 824 2024-04-26T00:25:57.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.129+0000 7ff03cc23640 1 -- 172.21.15.37:0/3560665159 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff03001ce50 con 0x7ff02403bcb0 2024-04-26T00:25:57.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.133+0000 7ff03cc23640 1 -- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff02403bcb0 msgr2=0x7ff02403e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:57.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.133+0000 7ff03cc23640 1 --2- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff02403bcb0 0x7ff02403e0a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff02c01aa20 tx=0x7ff02c019110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:57.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.133+0000 7ff0357fa640 1 -- 172.21.15.37:0/3560665159 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff02c0d0020 con 0x7ff02403bcb0 2024-04-26T00:25:57.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.133+0000 7ff0357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:57.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.133+0000 7ff0357fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:25:57.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.133+0000 7ff0357fa640 10 monclient: dump: 2024-04-26T00:25:57.115 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:57.115 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:57.115 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:57.116 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:57.116 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:57.116 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:57.116 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:57.116 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:57.116 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:57.116 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:57.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.133+0000 7ff0357fa640 1 -- 172.21.15.37:0/3560665159 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff02c0ccb50 con 0x7ff02403bcb0 2024-04-26T00:25:57.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.133+0000 7ff0357fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:57.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.133+0000 7ff0357fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:25:57.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.133+0000 7ff0357fa640 1 -- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff02403bcb0 msgr2=0x7ff02403e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:57.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.133+0000 7ff0357fa640 1 --2- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff02403bcb0 0x7ff02403e0a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff02c01aa20 tx=0x7ff02c019110 comp rx=0 tx=0).stop 2024-04-26T00:25:57.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.133+0000 7ff0357fa640 1 -- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff02403bcb0 msgr2=0x7ff02403e0a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:25:57.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.133+0000 7ff0357fa640 1 --2- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff02403bcb0 0x7ff02403e0a0 secure :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff02c01aa20 tx=0x7ff02c019110 comp rx=0 tx=0).stop 2024-04-26T00:25:57.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.141+0000 7ff03ee87640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:25:57.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.141+0000 7ff03ee87640 1 --2- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0381cdbd0 0x7ff03806ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:57.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.141+0000 7ff03ee87640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:57.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.141+0000 7ff03ee87640 1 -- 172.21.15.37:0/3560665159 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff03804f510 con 0x7ff0381cdbd0 2024-04-26T00:25:57.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.141+0000 7ff03cc23640 1 --2- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0381cdbd0 0x7ff03806ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:57.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.141+0000 7ff03cc23640 10 monclient: get_auth_request con 0x7ff0381cdbd0 auth_method 0 2024-04-26T00:25:57.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.141+0000 7ff03cc23640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:57.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.141+0000 7ff03cc23640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:57.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.141+0000 7ff03cc23640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:57.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.141+0000 7ff03cc23640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:57.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.141+0000 7ff03cc23640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:57.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.141+0000 7ff03cc23640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:57.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.141+0000 7ff03cc23640 10 monclient(hunting): handle_auth_done global_id 74105 payload 824 2024-04-26T00:25:57.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.141+0000 7ff03cc23640 1 -- 172.21.15.37:0/3560665159 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff03809eb20 con 0x7ff0381cdbd0 2024-04-26T00:25:57.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.145+0000 7ff03cc23640 1 -- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0381cdbd0 msgr2=0x7ff03806ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:57.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.145+0000 7ff03cc23640 1 --2- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0381cdbd0 0x7ff03806ff80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff02c019fa0 tx=0x7ff02c0c3000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:57.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.145+0000 7ff0357fa640 1 -- 172.21.15.37:0/3560665159 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff02c0d0040 con 0x7ff0381cdbd0 2024-04-26T00:25:57.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.145+0000 7ff0357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:57.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.145+0000 7ff0357fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:25:57.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.145+0000 7ff0357fa640 10 monclient: dump: 2024-04-26T00:25:57.128 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:57.128 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:57.128 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:57.128 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:57.128 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:57.128 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:57.129 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:57.129 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:57.129 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:57.129 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:57.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.145+0000 7ff0357fa640 1 -- 172.21.15.37:0/3560665159 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7ff02c0d72c0 con 0x7ff0381cdbd0 2024-04-26T00:25:57.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.145+0000 7ff0357fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:25:57.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.145+0000 7ff0357fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:25:57.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.145+0000 7ff0357fa640 1 -- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0381cdbd0 msgr2=0x7ff03806ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:57.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.145+0000 7ff0357fa640 1 --2- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0381cdbd0 0x7ff03806ff80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff02c019fa0 tx=0x7ff02c0c3000 comp rx=0 tx=0).stop 2024-04-26T00:25:57.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.145+0000 7ff0357fa640 1 -- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0381cdbd0 msgr2=0x7ff03806ff80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:25:57.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.145+0000 7ff0357fa640 1 --2- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0381cdbd0 0x7ff03806ff80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:57.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.145+0000 7ff03d424640 1 -- 172.21.15.37:0/3560665159 reap_dead start 2024-04-26T00:25:57.130 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:25:57.130 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:25:57.130 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:25:57.130 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:25:57.130 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 46, 2024-04-26T00:25:57.130 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 5455, 2024-04-26T00:25:57.131 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:25:57.131 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:25:57.131 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:25:57.131 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:25:57.131 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:57.131 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 3, 2024-04-26T00:25:57.131 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:25:57.131 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:25:57.131 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:25:57.131 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:57.131 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:57.131 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:57.131 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:57.131 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:57.132 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:57.132 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:57.132 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:57.132 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:57.132 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:57.132 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:57.132 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:57.132 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:25:57.132 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:25:57.132 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:57.132 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:57.132 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:57.132 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:57.132 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:57.132 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:57.133 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:57.133 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:57.133 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:57.133 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:57.133 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:57.133 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:57.133 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:57.133 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:25:57.133 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:25:57.133 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:25:57.133 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:25:57.133 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:25:57.133 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:25:57.133 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:25:57.133 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:25:57.134 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:25:57.134 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:25:57.134 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:25:57.134 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:25:57.134 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:25:57.134 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:25:57.134 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:25:57.134 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:25:57.134 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:25:57.134 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:57.134 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:57.134 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:57.134 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:57.134 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:57.135 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:57.135 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:57.135 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:57.135 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:57.135 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:57.135 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:57.135 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:57.135 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:25:57.135 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:57.135 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:25:57.135 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:57.135 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:25:57.135 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:25:57.135 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:57.136 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:57.136 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:57.136 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:57.136 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:25:57.136 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:57.136 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:57.136 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:57.136 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:57.136 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:25:57.136 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:57.136 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:57.136 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:57.136 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:57.136 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:25:57.136 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:25:57.136 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:57.137 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:57.137 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:57.137 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:57.137 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:57.137 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:25:57.137 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:25:57.137 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:57.137 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:57.137 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:57.137 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:57.137 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:25:57.137 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:57.137 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:57.137 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:57.137 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:57.138 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:25:57.138 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:57.138 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:57.138 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:57.138 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:57.138 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:25:57.138 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:25:57.138 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:57.138 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:57.138 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:57.138 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:57.138 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:57.138 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:25:57.138 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:25:57.139 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:57.139 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:57.139 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:57.139 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:57.139 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:25:57.139 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:57.139 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:57.139 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:57.139 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:57.139 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:25:57.139 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:57.139 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:57.139 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:57.139 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:57.139 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:25:57.140 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:25:57.140 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:57.140 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:57.140 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:57.140 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:57.140 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:57.140 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:57.140 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:25:57.140 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:25:57.140 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:57.140 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:57.140 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:57.140 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:25:57.140 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:57.140 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:57.141 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:25:57.141 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:57.141 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:57.141 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:57.141 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:25:57.141 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:57.141 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:57.141 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:57.141 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:25:57.141 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:25:57.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.145+0000 7ff03ee87640 1 -- 172.21.15.37:0/3560665159 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff024035ed0 msgr2=0x7ff024038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:57.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.145+0000 7ff03ee87640 1 --2- 172.21.15.37:0/3560665159 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff024035ed0 0x7ff024038390 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff0381383e0 tx=0x7ff028009040 comp rx=0 tx=0).stop 2024-04-26T00:25:57.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.145+0000 7ff03ee87640 10 monclient: shutdown 2024-04-26T00:25:57.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.145+0000 7ff03ee87640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:57.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.145+0000 7ff03ee87640 1 -- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0381352b0 msgr2=0x7ff0381376e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:57.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.145+0000 7ff03ee87640 1 --2- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0381352b0 0x7ff0381376e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff03000caf0 tx=0x7ff03000a060 comp rx=0 tx=0).stop 2024-04-26T00:25:57.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.145+0000 7ff03ee87640 1 -- 172.21.15.37:0/3560665159 shutdown_connections 2024-04-26T00:25:57.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.145+0000 7ff03ee87640 1 --2- 172.21.15.37:0/3560665159 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff024035ed0 0x7ff024038390 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:57.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.145+0000 7ff03ee87640 1 --2- 172.21.15.37:0/3560665159 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0381352b0 0x7ff0381376e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:57.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.145+0000 7ff03ee87640 1 -- 172.21.15.37:0/3560665159 >> 172.21.15.37:0/3560665159 conn(0x7ff0380a1d50 msgr2=0x7ff0380a3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:57.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.145+0000 7ff03ee87640 1 -- 172.21.15.37:0/3560665159 shutdown_connections 2024-04-26T00:25:57.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.145+0000 7ff03ee87640 1 -- 172.21.15.37:0/3560665159 wait complete. 2024-04-26T00:25:57.154 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:25:57.154 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:25:57.154 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:25:57.217 INFO:tasks.rados.rados.0.smithi045.stdout:959: finishing copy_from racing read to smithi04526579-8 2024-04-26T00:25:57.217 INFO:tasks.rados.rados.0.smithi045.stdout:959: finishing copy_from to smithi04526579-8 2024-04-26T00:25:57.217 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 346 (ObjNum 337 snap 100 seq_num 337) dirty exists 2024-04-26T00:25:57.217 INFO:tasks.rados.rados.0.smithi045.stdout:964: expect (ObjNum 329 snap 96 seq_num 329) 2024-04-26T00:25:57.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd78f4640 10 monclient: get_monmap_and_config 2024-04-26T00:25:57.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd78f4640 10 monclient: build_initial_monmap 2024-04-26T00:25:57.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd78f4640 10 monclient: monmap: 2024-04-26T00:25:57.287 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:57.287 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:57.287 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:25:57.308097+0000 2024-04-26T00:25:57.287 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:25:57.308097+0000 2024-04-26T00:25:57.287 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:57.287 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:57.287 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:25:57.287 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:25:57.287 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:25:57.287 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:25:57.287 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:25:57.287 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:57.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd78f4640 1 Processor -- start 2024-04-26T00:25:57.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd78f4640 1 -- start start 2024-04-26T00:25:57.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd78f4640 10 monclient: init 2024-04-26T00:25:57.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd78f4640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:57.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd78f4640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-04-26T00:25:57.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd78f4640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8cd00a0300 0x7f8cd009fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:57.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd78f4640 10 monclient(hunting): picked mon.noname-b con 0x7f8cd00a0300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:57.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd78f4640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f8cd00a06e0 0x7f8cd0134ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:57.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd78f4640 10 monclient(hunting): picked mon.noname-a con 0x7f8cd00a06e0 addr v2:172.21.15.37:3300/0 2024-04-26T00:25:57.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd78f4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8cd01350e0 addr v1:172.21.15.45:6789/0 2024-04-26T00:25:57.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd78f4640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8cd009c190 con 0x7f8cd01350e0 2024-04-26T00:25:57.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd78f4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:57.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd78f4640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f8cd00a13f0 con 0x7f8cd00a06e0 2024-04-26T00:25:57.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd78f4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:57.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd78f4640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8cd00a1110 con 0x7f8cd00a0300 2024-04-26T00:25:57.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd78f4640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:57.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd78f4640 10 monclient(hunting): authenticate will time out at 1338.613771s 2024-04-26T00:25:57.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd4e8f640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f8cd00a06e0 0x7f8cd0134ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:57.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd5690640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8cd00a0300 0x7f8cd009fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:57.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd4e8f640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f8cd00a06e0 0x7f8cd0134ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36646/0 (socket says 172.21.15.37:36646) 2024-04-26T00:25:57.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd4e8f640 1 -- 172.21.15.37:0/2847082809 learned_addr learned my addr 172.21.15.37:0/2847082809 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:57.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd5690640 10 monclient(hunting): get_auth_request con 0x7f8cd00a0300 auth_method 0 2024-04-26T00:25:57.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd5690640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:57.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd5690640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:57.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd5690640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:57.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd4e8f640 10 monclient(hunting): get_auth_request con 0x7f8cd00a06e0 auth_method 0 2024-04-26T00:25:57.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd4e8f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:57.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd4e8f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:57.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd4e8f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:57.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd5690640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:57.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd5690640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:57.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd5690640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:57.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd4e8f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:57.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd4e8f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:57.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd4e8f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:57.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd4e8f640 10 monclient(hunting): handle_auth_done global_id 54288 payload 274 2024-04-26T00:25:57.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd4e8f640 1 -- 172.21.15.37:0/2847082809 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8cd00a0300 msgr2=0x7f8cd009fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:57.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd4e8f640 1 --2- 172.21.15.37:0/2847082809 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8cd00a0300 0x7f8cd009fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:57.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd4e8f640 1 -- 172.21.15.37:0/2847082809 >> v1:172.21.15.45:6789/0 conn(0x7f8cd01350e0 legacy=0x7f8cd013bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:57.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd4e8f640 10 monclient: _finish_hunting 0 2024-04-26T00:25:57.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd4e8f640 1 monclient: found mon.noname-a 2024-04-26T00:25:57.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd4e8f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:57.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd4e8f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:25:57.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd4e8f640 1 -- 172.21.15.37:0/2847082809 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cd00a16d0 con 0x7f8cd00a06e0 2024-04-26T00:25:57.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.305+0000 7f8cd5690640 1 --2- 172.21.15.37:0/2847082809 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8cd00a0300 0x7f8cd009fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:25:57.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.309+0000 7f8cd4e8f640 1 -- 172.21.15.37:0/2847082809 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8cd00a06e0 msgr2=0x7f8cd0134ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:57.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.309+0000 7f8cd4e8f640 1 --2- 172.21.15.37:0/2847082809 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8cd00a06e0 0x7f8cd0134ba0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8cc0009870 tx=0x7f8cc001c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab5d789dddc074a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:57.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.309+0000 7f8cc7fff640 1 -- 172.21.15.37:0/2847082809 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc00b0070 con 0x7f8cd00a06e0 2024-04-26T00:25:57.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.309+0000 7f8cc7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:57.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.309+0000 7f8cc7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:25:57.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.309+0000 7f8cc7fff640 10 monclient: dump: 2024-04-26T00:25:57.294 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:57.294 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:57.295 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:57.295 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:57.295 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:57.295 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:57.295 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:57.295 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:57.295 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:57.295 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:57.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.309+0000 7f8cc7fff640 10 monclient: _finish_auth 0 2024-04-26T00:25:57.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.309+0000 7f8cc7fff640 10 monclient: _check_auth_tickets 2024-04-26T00:25:57.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.309+0000 7f8cc7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:57.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.309+0000 7f8cc7fff640 1 -- 172.21.15.37:0/2847082809 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cc00ab600 con 0x7f8cd00a06e0 2024-04-26T00:25:57.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.309+0000 7f8cd78f4640 5 monclient: authenticate success, global_id 54288 2024-04-26T00:25:57.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.309+0000 7f8cd78f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:25:57.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.309+0000 7f8cc7fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:57.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.309+0000 7f8cd78f4640 10 monclient: get_monmap_and_config success 2024-04-26T00:25:57.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.309+0000 7f8cc7fff640 1 -- 172.21.15.37:0/2847082809 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc00ab9d0 con 0x7f8cd00a06e0 2024-04-26T00:25:57.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.309+0000 7f8cd78f4640 4 set_mon_vals no callback set 2024-04-26T00:25:57.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.309+0000 7f8cc7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:57.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.309+0000 7f8cc7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:57.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.309+0000 7f8cc7fff640 10 monclient: dump: 2024-04-26T00:25:57.296 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:57.296 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:57.296 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:57.296 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:57.296 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:57.296 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:57.296 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:57.296 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:57.296 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:57.297 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:57.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.309+0000 7f8cd78f4640 10 monclient: shutdown 2024-04-26T00:25:57.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.309+0000 7f8cd78f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:57.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.309+0000 7f8cd78f4640 1 -- 172.21.15.37:0/2847082809 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8cd00a06e0 msgr2=0x7f8cd0134ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:57.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.309+0000 7f8cd78f4640 1 --2- 172.21.15.37:0/2847082809 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8cd00a06e0 0x7f8cd0134ba0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8cc0009870 tx=0x7f8cc001c410 comp rx=0 tx=0).stop 2024-04-26T00:25:57.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.309+0000 7f8cd78f4640 1 -- 172.21.15.37:0/2847082809 shutdown_connections 2024-04-26T00:25:57.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.309+0000 7f8cd78f4640 1 --2- 172.21.15.37:0/2847082809 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8cd00a06e0 0x7f8cd0134ba0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:57.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.309+0000 7f8cd78f4640 1 --2- 172.21.15.37:0/2847082809 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8cd00a0300 0x7f8cd009fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:57.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.309+0000 7f8cd78f4640 1 -- 172.21.15.37:0/2847082809 >> 172.21.15.37:0/2847082809 conn(0x7f8cd00a1d50 msgr2=0x7f8cd00a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:57.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.309+0000 7f8cd78f4640 1 -- 172.21.15.37:0/2847082809 shutdown_connections 2024-04-26T00:25:57.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.309+0000 7f8cd78f4640 1 -- 172.21.15.37:0/2847082809 wait complete. 2024-04-26T00:25:57.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd78f4640 10 monclient: build_initial_monmap 2024-04-26T00:25:57.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd78f4640 10 monclient: monmap: 2024-04-26T00:25:57.297 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:57.297 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:25:57.297 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:25:57.297 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:25:57.298 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:57.298 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:25:57.298 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:25:57.298 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:25:57.298 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:25:57.298 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:57.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd78f4640 1 Processor -- start 2024-04-26T00:25:57.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd78f4640 1 -- start start 2024-04-26T00:25:57.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd78f4640 10 monclient: init 2024-04-26T00:25:57.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd78f4640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:57.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd78f4640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:25:57.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd78f4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cd00a0300 0x7f8cd01cc010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:57.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd78f4640 10 monclient(hunting): picked mon.noname-b con 0x7f8cd00a0300 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:25:57.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd78f4640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8cd00a06e0 0x7f8cd0130fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:57.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd78f4640 10 monclient(hunting): picked mon.noname-a con 0x7f8cd00a06e0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:25:57.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd78f4640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8cd01350e0 0x7f8cd01cf550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:57.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd78f4640 10 monclient(hunting): picked mon.noname-c con 0x7f8cd01350e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:57.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd78f4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:57.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd78f4640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f8cd00a0c40 con 0x7f8cd00a06e0 2024-04-26T00:25:57.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd78f4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:57.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd78f4640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8cd00a0ac0 con 0x7f8cd00a0300 2024-04-26T00:25:57.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd78f4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:57.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd78f4640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8cd013fe10 con 0x7f8cd01350e0 2024-04-26T00:25:57.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd78f4640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:57.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd78f4640 10 monclient(hunting): authenticate will time out at 1338.621040s 2024-04-26T00:25:57.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd5e91640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8cd01350e0 0x7f8cd01cf550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:57.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd4e8f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8cd00a06e0 0x7f8cd0130fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:57.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd4e8f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8cd00a06e0 0x7f8cd0130fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36654/0 (socket says 172.21.15.37:36654) 2024-04-26T00:25:57.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd5e91640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8cd01350e0 0x7f8cd01cf550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51848/0 (socket says 172.21.15.37:51848) 2024-04-26T00:25:57.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd4e8f640 1 -- 172.21.15.37:0/1453316683 learned_addr learned my addr 172.21.15.37:0/1453316683 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:57.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd4e8f640 10 monclient(hunting): get_auth_request con 0x7f8cd00a06e0 auth_method 0 2024-04-26T00:25:57.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd4e8f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:57.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd4e8f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:57.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd4e8f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:57.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd5e91640 10 monclient(hunting): get_auth_request con 0x7f8cd01350e0 auth_method 0 2024-04-26T00:25:57.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd5e91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:57.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd5e91640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:57.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd5e91640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:57.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd4e8f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:57.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd4e8f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:57.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd4e8f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:57.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd5e91640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:57.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd5e91640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:57.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd5690640 1 --2- 172.21.15.37:0/1453316683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cd00a0300 0x7f8cd01cc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:57.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd5e91640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:57.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd5690640 10 monclient(hunting): get_auth_request con 0x7f8cd00a0300 auth_method 0 2024-04-26T00:25:57.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd5690640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:57.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd5690640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:57.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd5690640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:57.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd5e91640 10 monclient(hunting): handle_auth_done global_id 74111 payload 824 2024-04-26T00:25:57.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd5e91640 1 -- 172.21.15.37:0/1453316683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cd00a0300 msgr2=0x7f8cd01cc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:57.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd5e91640 1 --2- 172.21.15.37:0/1453316683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cd00a0300 0x7f8cd01cc010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:57.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd5e91640 1 -- 172.21.15.37:0/1453316683 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8cd00a06e0 msgr2=0x7f8cd0130fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:57.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd5e91640 1 --2- 172.21.15.37:0/1453316683 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8cd00a06e0 0x7f8cd0130fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:57.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd5e91640 10 monclient: _finish_hunting 0 2024-04-26T00:25:57.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd5e91640 1 monclient: found mon.noname-c 2024-04-26T00:25:57.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd5e91640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:57.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd5e91640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:25:57.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd5e91640 1 -- 172.21.15.37:0/1453316683 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cd011d670 con 0x7f8cd01350e0 2024-04-26T00:25:57.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd4e8f640 1 --2- 172.21.15.37:0/1453316683 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8cd00a06e0 0x7f8cd0130fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:25:57.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd5e91640 1 -- 172.21.15.37:0/1453316683 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8cd01350e0 msgr2=0x7f8cd01cf550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:57.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.313+0000 7f8cd5e91640 1 --2- 172.21.15.37:0/1453316683 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8cd01350e0 0x7f8cd01cf550 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8cc800c7e0 tx=0x7f8cc800cca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:57.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.317+0000 7f8cc5ffb640 1 -- 172.21.15.37:0/1453316683 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc8016020 con 0x7f8cd01350e0 2024-04-26T00:25:57.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.317+0000 7f8cc5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:57.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.317+0000 7f8cc5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:25:57.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.317+0000 7f8cc5ffb640 10 monclient: dump: 2024-04-26T00:25:57.306 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:57.306 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:57.306 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:57.306 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:57.306 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:57.306 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:57.306 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:57.306 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:57.306 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:57.306 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:57.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.317+0000 7f8cc5ffb640 10 monclient: _finish_auth 0 2024-04-26T00:25:57.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.317+0000 7f8cc5ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:25:57.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.317+0000 7f8cc5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:57.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.317+0000 7f8cc5ffb640 1 -- 172.21.15.37:0/1453316683 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cc800ac60 con 0x7f8cd01350e0 2024-04-26T00:25:57.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.317+0000 7f8cc5ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:57.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.317+0000 7f8cc5ffb640 1 -- 172.21.15.37:0/1453316683 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc8012660 con 0x7f8cd01350e0 2024-04-26T00:25:57.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.317+0000 7f8cd78f4640 5 monclient: authenticate success, global_id 74111 2024-04-26T00:25:57.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.317+0000 7f8cc5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:57.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.317+0000 7f8cc5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:25:57.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.317+0000 7f8cc5ffb640 10 monclient: dump: 2024-04-26T00:25:57.307 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:57.307 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:57.307 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:57.307 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:57.307 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:57.307 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:57.308 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:57.308 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:57.308 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:57.308 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:57.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.317+0000 7f8cc77fe640 4 set_mon_vals no callback set 2024-04-26T00:25:57.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.317+0000 7f8cd78f4640 10 monclient: _renew_subs 2024-04-26T00:25:57.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.317+0000 7f8cd78f4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:25:57.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.317+0000 7f8cd78f4640 1 -- 172.21.15.37:0/1453316683 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cd0132a10 con 0x7f8cd01350e0 2024-04-26T00:25:57.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.317+0000 7f8cd78f4640 10 monclient: _renew_subs 2024-04-26T00:25:57.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.317+0000 7f8cd78f4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:25:57.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.317+0000 7f8cd78f4640 1 -- 172.21.15.37:0/1453316683 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cd011d390 con 0x7f8cd01350e0 2024-04-26T00:25:57.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.317+0000 7f8cd78f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:57.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.317+0000 7f8cd78f4640 1 --2- 172.21.15.37:0/1453316683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cd011c390 0x7f8cd01cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:57.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.317+0000 7f8cd78f4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:57.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.317+0000 7f8cd78f4640 1 -- 172.21.15.37:0/1453316683 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8cd004f510 con 0x7f8cd011c390 2024-04-26T00:25:57.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.317+0000 7f8cc5ffb640 1 -- 172.21.15.37:0/1453316683 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f8cc8019020 con 0x7f8cd01350e0 2024-04-26T00:25:57.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.317+0000 7f8cd5690640 1 --2- 172.21.15.37:0/1453316683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cd011c390 0x7f8cd01cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:57.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.317+0000 7f8cc5ffb640 1 --2- 172.21.15.37:0/1453316683 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8cb4035ed0 0x7f8cb4038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:57.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.317+0000 7f8cd5690640 10 monclient: get_auth_request con 0x7f8cd011c390 auth_method 0 2024-04-26T00:25:57.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.317+0000 7f8cd5690640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:57.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.317+0000 7f8cd5690640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:57.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.317+0000 7f8cd5690640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:57.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.317+0000 7f8cc5ffb640 1 -- 172.21.15.37:0/1453316683 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(234..234 src has 177..234) ==== 4067+0+0 (secure 0 0 0) 0x7f8cc8048a10 con 0x7f8cd01350e0 2024-04-26T00:25:57.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.317+0000 7f8cd5690640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:57.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.317+0000 7f8cd5690640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:57.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.317+0000 7f8cd5690640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:57.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.317+0000 7f8cd4e8f640 1 --2- 172.21.15.37:0/1453316683 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8cb4035ed0 0x7f8cb4038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:57.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.317+0000 7f8cd4e8f640 10 monclient: get_auth_request con 0x7f8cb4035ed0 auth_method 0 2024-04-26T00:25:57.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.317+0000 7f8cd5690640 10 monclient(hunting): handle_auth_done global_id 24526 payload 824 2024-04-26T00:25:57.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.317+0000 7f8cd5690640 1 -- 172.21.15.37:0/1453316683 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8cd009d9a0 con 0x7f8cd011c390 2024-04-26T00:25:57.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.321+0000 7f8cd4e8f640 1 --2- 172.21.15.37:0/1453316683 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8cb4035ed0 0x7f8cb4038390 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8cc001c450 tx=0x7f8cc0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:57.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.321+0000 7f8cd5690640 1 -- 172.21.15.37:0/1453316683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cd011c390 msgr2=0x7f8cd01cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:57.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.321+0000 7f8cd5690640 1 --2- 172.21.15.37:0/1453316683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cd011c390 0x7f8cd01cfa90 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc011600 tx=0x7f8ccc011ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:57.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.321+0000 7f8cc5ffb640 1 -- 172.21.15.37:0/1453316683 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ccc00fc30 con 0x7f8cd011c390 2024-04-26T00:25:57.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.321+0000 7f8cc5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:57.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.321+0000 7f8cc5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:25:57.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.321+0000 7f8cc5ffb640 10 monclient: dump: 2024-04-26T00:25:57.312 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:57.312 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:57.312 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:57.312 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:57.312 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:57.312 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:57.312 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:57.312 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:57.312 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:57.312 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:57.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.321+0000 7f8cc5ffb640 1 -- 172.21.15.37:0/1453316683 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8ccc00fc30 con 0x7f8cd011c390 2024-04-26T00:25:57.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.321+0000 7f8cc5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:57.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.321+0000 7f8cc5ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:25:57.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.321+0000 7f8cc5ffb640 1 -- 172.21.15.37:0/1453316683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cd011c390 msgr2=0x7f8cd01cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:57.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.321+0000 7f8cc5ffb640 1 --2- 172.21.15.37:0/1453316683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cd011c390 0x7f8cd01cfa90 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc011600 tx=0x7f8ccc011ac0 comp rx=0 tx=0).stop 2024-04-26T00:25:57.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.321+0000 7f8cc5ffb640 1 -- 172.21.15.37:0/1453316683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cd011c390 msgr2=0x7f8cd01cfa90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:25:57.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.321+0000 7f8cc5ffb640 1 --2- 172.21.15.37:0/1453316683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cd011c390 0x7f8cd01cfa90 secure :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc011600 tx=0x7f8ccc011ac0 comp rx=0 tx=0).stop 2024-04-26T00:25:57.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.325+0000 7f8cd78f4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:25:57.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.325+0000 7f8cd78f4640 1 --2- 172.21.15.37:0/1453316683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cd01314e0 0x7f8cd01cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:57.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.325+0000 7f8cd78f4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:57.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.325+0000 7f8cd78f4640 1 -- 172.21.15.37:0/1453316683 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8cd00008e0 con 0x7f8cd01314e0 2024-04-26T00:25:57.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.325+0000 7f8cd5690640 1 --2- 172.21.15.37:0/1453316683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cd01314e0 0x7f8cd01cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:57.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.325+0000 7f8cd5690640 10 monclient: get_auth_request con 0x7f8cd01314e0 auth_method 0 2024-04-26T00:25:57.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.325+0000 7f8cd5690640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:57.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.325+0000 7f8cd5690640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:57.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.325+0000 7f8cd5690640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:57.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.329+0000 7f8cd5690640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:57.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.329+0000 7f8cd5690640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:57.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.329+0000 7f8cd5690640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:57.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.329+0000 7f8cd5690640 10 monclient(hunting): handle_auth_done global_id 24529 payload 824 2024-04-26T00:25:57.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.329+0000 7f8cd5690640 1 -- 172.21.15.37:0/1453316683 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8cd01cca90 con 0x7f8cd01314e0 2024-04-26T00:25:57.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.329+0000 7f8cd5690640 1 -- 172.21.15.37:0/1453316683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cd01314e0 msgr2=0x7f8cd01cffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:57.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.329+0000 7f8cd5690640 1 --2- 172.21.15.37:0/1453316683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cd01314e0 0x7f8cd01cffd0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc011600 tx=0x7f8ccc0046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:57.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.329+0000 7f8cc5ffb640 1 -- 172.21.15.37:0/1453316683 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ccc006450 con 0x7f8cd01314e0 2024-04-26T00:25:57.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.329+0000 7f8cc5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:57.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.329+0000 7f8cc5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:25:57.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.329+0000 7f8cc5ffb640 10 monclient: dump: 2024-04-26T00:25:57.315 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:57.315 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:57.315 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:57.315 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:57.315 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:57.315 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:57.315 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:57.315 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:57.315 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:57.316 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:57.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.329+0000 7f8cc5ffb640 1 -- 172.21.15.37:0/1453316683 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4719 (secure 0 0 0) 0x7f8ccc006450 con 0x7f8cd01314e0 2024-04-26T00:25:57.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.329+0000 7f8cc5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:25:57.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.329+0000 7f8cc5ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:25:57.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.329+0000 7f8cc5ffb640 1 -- 172.21.15.37:0/1453316683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cd01314e0 msgr2=0x7f8cd01cffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:57.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.329+0000 7f8cc5ffb640 1 --2- 172.21.15.37:0/1453316683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cd01314e0 0x7f8cd01cffd0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc011600 tx=0x7f8ccc0046d0 comp rx=0 tx=0).stop 2024-04-26T00:25:57.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.329+0000 7f8cc5ffb640 1 -- 172.21.15.37:0/1453316683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cd01314e0 msgr2=0x7f8cd01cffd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:25:57.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.329+0000 7f8cc5ffb640 1 --2- 172.21.15.37:0/1453316683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cd01314e0 0x7f8cd01cffd0 secure :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc011600 tx=0x7f8ccc0046d0 comp rx=0 tx=0).stop 2024-04-26T00:25:57.316 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:25:57.316 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:25:57.316 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:25:57.316 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:25:57.317 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 46, 2024-04-26T00:25:57.317 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 265499, 2024-04-26T00:25:57.317 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:25:57.317 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:25:57.317 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:25:57.317 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:25:57.317 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:57.317 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 4, 2024-04-26T00:25:57.317 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:25:57.317 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:25:57.317 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:25:57.317 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:57.317 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:57.317 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:57.317 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:57.317 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:57.318 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:57.318 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:57.318 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:57.318 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:57.318 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:57.318 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:57.318 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:57.318 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:25:57.318 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:25:57.318 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:57.318 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:57.318 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:57.318 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:57.318 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:57.318 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:57.319 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:57.319 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:57.319 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:57.319 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:57.319 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:57.319 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:57.319 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:57.319 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:25:57.319 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:25:57.319 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:25:57.319 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:25:57.319 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:25:57.319 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:25:57.319 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:25:57.319 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:25:57.319 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:25:57.320 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:25:57.320 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:25:57.320 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:25:57.320 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:25:57.320 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:25:57.320 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:25:57.320 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:25:57.320 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:25:57.320 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:57.320 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:57.320 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:57.320 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:57.320 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:57.320 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:57.320 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:57.321 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:57.321 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:57.321 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:57.321 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:57.321 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:57.321 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:25:57.321 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:57.321 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:25:57.321 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:57.321 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:25:57.321 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:25:57.321 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:57.321 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:57.321 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:57.321 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:57.321 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:25:57.322 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:57.322 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:57.322 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:57.322 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:57.322 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:25:57.322 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:57.322 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:57.322 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:57.322 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:57.322 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:25:57.322 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:25:57.322 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:57.322 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:57.322 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:57.323 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:57.323 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:57.323 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:25:57.323 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:25:57.323 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:57.323 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:57.323 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:57.323 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:57.323 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:25:57.323 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:57.323 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:57.323 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:57.323 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:57.323 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:25:57.323 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:57.324 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:57.324 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:57.324 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:57.324 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:25:57.324 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:25:57.324 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:57.324 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:57.324 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:57.324 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:57.324 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:57.324 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:25:57.324 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:25:57.324 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:57.324 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:57.324 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:57.325 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:57.325 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:25:57.325 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:57.325 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:57.325 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:57.325 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:57.325 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:25:57.325 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:57.325 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:57.325 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:57.325 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:57.325 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:25:57.325 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:25:57.325 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:57.325 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:57.325 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:57.326 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:57.326 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:57.326 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:57.326 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:25:57.326 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:25:57.326 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:57.326 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:57.326 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:57.326 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:25:57.326 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:57.326 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:57.326 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:25:57.326 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:57.326 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:57.326 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:57.327 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:25:57.327 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:57.327 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:57.327 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:25:57.327 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:57.327 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:57.327 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:57.327 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:25:57.327 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:57.327 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:57.327 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:57.327 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:25:57.327 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:25:57.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.333+0000 7f8cd78f4640 1 -- 172.21.15.37:0/1453316683 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8cb4035ed0 msgr2=0x7f8cb4038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:57.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.333+0000 7f8cd78f4640 1 --2- 172.21.15.37:0/1453316683 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8cb4035ed0 0x7f8cb4038390 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8cc001c450 tx=0x7f8cc0008040 comp rx=0 tx=0).stop 2024-04-26T00:25:57.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.333+0000 7f8cd78f4640 10 monclient: shutdown 2024-04-26T00:25:57.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.333+0000 7f8cd78f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:57.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.333+0000 7f8cd78f4640 1 -- 172.21.15.37:0/1453316683 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8cd01350e0 msgr2=0x7f8cd01cf550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:57.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.333+0000 7f8cd78f4640 1 --2- 172.21.15.37:0/1453316683 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8cd01350e0 0x7f8cd01cf550 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8cc800c7e0 tx=0x7f8cc800cca0 comp rx=0 tx=0).stop 2024-04-26T00:25:57.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.333+0000 7f8cd5e91640 1 -- 172.21.15.37:0/1453316683 reap_dead start 2024-04-26T00:25:57.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.333+0000 7f8cd5e91640 1 -- 172.21.15.37:0/1453316683 reap_dead start 2024-04-26T00:25:57.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.333+0000 7f8cd78f4640 1 -- 172.21.15.37:0/1453316683 shutdown_connections 2024-04-26T00:25:57.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.333+0000 7f8cd78f4640 1 -- 172.21.15.37:0/1453316683 >> 172.21.15.37:0/1453316683 conn(0x7f8cd00a1d50 msgr2=0x7f8cd0131c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:57.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.333+0000 7f8cd78f4640 1 -- 172.21.15.37:0/1453316683 shutdown_connections 2024-04-26T00:25:57.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.333+0000 7f8cd78f4640 1 -- 172.21.15.37:0/1453316683 wait complete. 2024-04-26T00:25:57.330 INFO:tasks.rados.rados.0.smithi045.stdout:966: finishing write tid 1 to smithi04526579-15 2024-04-26T00:25:57.330 INFO:tasks.rados.rados.0.smithi045.stdout:966: finishing write tid 2 to smithi04526579-15 2024-04-26T00:25:57.330 INFO:tasks.rados.rados.0.smithi045.stdout:966: finishing write tid 3 to smithi04526579-15 2024-04-26T00:25:57.330 INFO:tasks.rados.rados.0.smithi045.stdout:966: finishing write tid 4 to smithi04526579-15 2024-04-26T00:25:57.330 INFO:tasks.rados.rados.0.smithi045.stdout:966: oid 15 updating version 0 to 572 2024-04-26T00:25:57.330 INFO:tasks.rados.rados.0.smithi045.stdout:966: oid 15 updating version 572 to 573 2024-04-26T00:25:57.330 INFO:tasks.rados.rados.0.smithi045.stdout:966: oid 15 version 573 is already newer than 571 2024-04-26T00:25:57.330 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 573 (ObjNum 375 snap 114 seq_num 375) dirty exists 2024-04-26T00:25:57.330 INFO:tasks.rados.rados.0.smithi045.stdout:966: left oid 15 (ObjNum 375 snap 114 seq_num 375) 2024-04-26T00:25:57.330 INFO:tasks.rados.rados.0.smithi045.stdout:965: expect (ObjNum 351 snap 105 seq_num 351) 2024-04-26T00:25:57.331 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:25:57.331 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-26T00:25:57.332 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-26T00:25:57.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.461+0000 7f02bb318640 10 monclient: get_monmap_and_config 2024-04-26T00:25:57.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.461+0000 7f02bb318640 10 monclient: build_initial_monmap 2024-04-26T00:25:57.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.461+0000 7f02bb318640 10 monclient: monmap: 2024-04-26T00:25:57.446 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:57.446 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:57.446 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:25:57.466845+0000 2024-04-26T00:25:57.446 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:25:57.466845+0000 2024-04-26T00:25:57.446 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:57.446 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:57.446 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:25:57.446 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:25:57.446 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:25:57.447 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:25:57.447 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:25:57.447 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:57.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.461+0000 7f02bb318640 1 Processor -- start 2024-04-26T00:25:57.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.461+0000 7f02bb318640 1 -- start start 2024-04-26T00:25:57.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.461+0000 7f02bb318640 10 monclient: init 2024-04-26T00:25:57.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02bb318640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:57.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02bb318640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-04-26T00:25:57.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02bb318640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f02b40a0260 0x7f02b40a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:57.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02bb318640 10 monclient(hunting): picked mon.noname-a con 0x7f02b40a0260 addr v2:172.21.15.37:3300/0 2024-04-26T00:25:57.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02bb318640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f02b4134b30 addr v1:172.21.15.37:6789/0 2024-04-26T00:25:57.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02bb318640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f02b4135660 0x7f02b413ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:57.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02bb318640 10 monclient(hunting): picked mon.noname-b con 0x7f02b4135660 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:57.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02bb318640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02b40a1130 con 0x7f02b4134b30 2024-04-26T00:25:57.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02bb318640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:57.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02bb318640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f02b409ec10 con 0x7f02b40a0260 2024-04-26T00:25:57.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02bb318640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:57.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02bb318640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f02b409f0f0 con 0x7f02b4135660 2024-04-26T00:25:57.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02bb318640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:57.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02bb318640 10 monclient(hunting): authenticate will time out at 1338.772482s 2024-04-26T00:25:57.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02b98b5640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f02b4135660 0x7f02b413ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:57.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02b90b4640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f02b40a0260 0x7f02b40a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:57.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02b98b5640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f02b4135660 0x7f02b413ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51864/0 (socket says 172.21.15.37:51864) 2024-04-26T00:25:57.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02b98b5640 1 -- 172.21.15.37:0/3980943846 learned_addr learned my addr 172.21.15.37:0/3980943846 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:57.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02b88b3640 1 --1- 172.21.15.37:0/3980943846 >> v1:172.21.15.37:6789/0 conn(0x7f02b4134b30 0x7f02b4134f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:39684/0 (socket says 172.21.15.37:39684) 2024-04-26T00:25:57.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02b98b5640 10 monclient(hunting): get_auth_request con 0x7f02b4135660 auth_method 0 2024-04-26T00:25:57.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02b98b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:57.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02b98b5640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:57.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02b98b5640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:57.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02b90b4640 10 monclient(hunting): get_auth_request con 0x7f02b40a0260 auth_method 0 2024-04-26T00:25:57.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02b90b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:57.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02b90b4640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:57.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02b90b4640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:57.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02b98b5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:57.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02b98b5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:57.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02b98b5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:57.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02b90b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:57.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02b90b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:57.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02b90b4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:57.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02b98b5640 10 monclient(hunting): handle_auth_done global_id 74114 payload 274 2024-04-26T00:25:57.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02b98b5640 1 -- 172.21.15.37:0/3980943846 >> v2:172.21.15.37:3300/0 conn(0x7f02b40a0260 msgr2=0x7f02b40a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:57.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02b98b5640 1 --2- 172.21.15.37:0/3980943846 >> v2:172.21.15.37:3300/0 conn(0x7f02b40a0260 0x7f02b40a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:57.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02b98b5640 1 -- 172.21.15.37:0/3980943846 >> v1:172.21.15.37:6789/0 conn(0x7f02b4134b30 legacy=0x7f02b4134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:57.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02b98b5640 10 monclient: _finish_hunting 0 2024-04-26T00:25:57.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02b98b5640 1 monclient: found mon.noname-b 2024-04-26T00:25:57.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02b98b5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:57.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02b98b5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:25:57.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.465+0000 7f02b98b5640 1 -- 172.21.15.37:0/3980943846 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02b406cff0 con 0x7f02b4135660 2024-04-26T00:25:57.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02b98b5640 1 -- 172.21.15.37:0/3980943846 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f02b4135660 msgr2=0x7f02b413ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:57.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02b98b5640 1 --2- 172.21.15.37:0/3980943846 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f02b4135660 0x7f02b413ce70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f02ac009e50 tx=0x7f02ac01c6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f26aeca2ca13c2d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:57.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02abfff640 1 -- 172.21.15.37:0/3980943846 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ac0a9460 con 0x7f02b4135660 2024-04-26T00:25:57.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:57.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02abfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:25:57.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02abfff640 10 monclient: dump: 2024-04-26T00:25:57.459 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:57.459 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:57.459 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:57.459 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:57.459 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:57.459 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:57.459 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:57.459 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:57.459 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:57.459 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:57.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02abfff640 10 monclient: _finish_auth 0 2024-04-26T00:25:57.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02abfff640 10 monclient: _check_auth_tickets 2024-04-26T00:25:57.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:57.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02abfff640 1 -- 172.21.15.37:0/3980943846 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02ac0ae070 con 0x7f02b4135660 2024-04-26T00:25:57.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02abfff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:57.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02abfff640 1 -- 172.21.15.37:0/3980943846 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ac0a9820 con 0x7f02b4135660 2024-04-26T00:25:57.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:57.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02abfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:25:57.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02abfff640 10 monclient: dump: 2024-04-26T00:25:57.460 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:57.460 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:57.460 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:57.460 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:57.460 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:57.460 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:57.460 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:57.461 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:57.461 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:57.461 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:57.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02bb318640 5 monclient: authenticate success, global_id 74114 2024-04-26T00:25:57.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02bb318640 10 monclient: get_monmap_and_config success 2024-04-26T00:25:57.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02bb318640 4 set_mon_vals no callback set 2024-04-26T00:25:57.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02bb318640 10 monclient: shutdown 2024-04-26T00:25:57.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02bb318640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:57.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02bb318640 1 -- 172.21.15.37:0/3980943846 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f02b4135660 msgr2=0x7f02b413ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:57.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02bb318640 1 --2- 172.21.15.37:0/3980943846 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f02b4135660 0x7f02b413ce70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f02ac009e50 tx=0x7f02ac01c6d0 comp rx=0 tx=0).stop 2024-04-26T00:25:57.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02bb318640 1 -- 172.21.15.37:0/3980943846 shutdown_connections 2024-04-26T00:25:57.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02bb318640 1 --2- 172.21.15.37:0/3980943846 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f02b4135660 0x7f02b413ce70 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:57.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02bb318640 1 --2- 172.21.15.37:0/3980943846 >> v2:172.21.15.37:3300/0 conn(0x7f02b40a0260 0x7f02b40a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:57.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02bb318640 1 -- 172.21.15.37:0/3980943846 >> 172.21.15.37:0/3980943846 conn(0x7f02b40a1ad0 msgr2=0x7f02b40a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:57.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02bb318640 1 -- 172.21.15.37:0/3980943846 shutdown_connections 2024-04-26T00:25:57.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02bb318640 1 -- 172.21.15.37:0/3980943846 wait complete. 2024-04-26T00:25:57.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02bb318640 10 monclient: build_initial_monmap 2024-04-26T00:25:57.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02bb318640 10 monclient: monmap: 2024-04-26T00:25:57.462 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:57.462 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:25:57.462 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:25:57.462 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:25:57.462 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:57.462 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:25:57.462 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:25:57.462 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:25:57.463 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:25:57.463 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:57.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02bb318640 1 Processor -- start 2024-04-26T00:25:57.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02bb318640 1 -- start start 2024-04-26T00:25:57.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02bb318640 10 monclient: init 2024-04-26T00:25:57.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02bb318640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:57.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02bb318640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:25:57.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02bb318640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02b40a0260 0x7f02b41ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:57.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02bb318640 10 monclient(hunting): picked mon.noname-b con 0x7f02b40a0260 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:25:57.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02bb318640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f02b4134b30 0x7f02b41cd530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:57.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02bb318640 10 monclient(hunting): picked mon.noname-c con 0x7f02b4134b30 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:57.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02bb318640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f02b4135660 0x7f02b4139aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:57.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02bb318640 10 monclient(hunting): picked mon.noname-a con 0x7f02b4135660 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:25:57.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02bb318640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:57.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02bb318640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f02b40a0010 con 0x7f02b4135660 2024-04-26T00:25:57.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02bb318640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:57.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02bb318640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f02b409fe90 con 0x7f02b40a0260 2024-04-26T00:25:57.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02bb318640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:57.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02bb318640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f02b413fbd0 con 0x7f02b4134b30 2024-04-26T00:25:57.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02bb318640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:57.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02bb318640 10 monclient(hunting): authenticate will time out at 1338.779555s 2024-04-26T00:25:57.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02b88b3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f02b4134b30 0x7f02b41cd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:57.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02b98b5640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f02b4135660 0x7f02b4139aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:57.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02b88b3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f02b4134b30 0x7f02b41cd530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51866/0 (socket says 172.21.15.37:51866) 2024-04-26T00:25:57.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02b88b3640 1 -- 172.21.15.37:0/2952239091 learned_addr learned my addr 172.21.15.37:0/2952239091 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:57.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02b88b3640 10 monclient(hunting): get_auth_request con 0x7f02b4134b30 auth_method 0 2024-04-26T00:25:57.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02b88b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:57.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02b88b3640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:57.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02b88b3640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:57.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02b98b5640 10 monclient(hunting): get_auth_request con 0x7f02b4135660 auth_method 0 2024-04-26T00:25:57.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02b98b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:57.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02b98b5640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:57.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02b98b5640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:57.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02b98b5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:57.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02b98b5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:57.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02b98b5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:57.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02b88b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:57.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02b88b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:57.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02b88b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:57.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02b90b4640 1 --2- 172.21.15.37:0/2952239091 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02b40a0260 0x7f02b41ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:57.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02b90b4640 10 monclient(hunting): get_auth_request con 0x7f02b40a0260 auth_method 0 2024-04-26T00:25:57.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02b90b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:57.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02b90b4640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:57.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02b90b4640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:57.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.469+0000 7f02b98b5640 10 monclient(hunting): handle_auth_done global_id 54297 payload 824 2024-04-26T00:25:57.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.473+0000 7f02b98b5640 1 -- 172.21.15.37:0/2952239091 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f02b4134b30 msgr2=0x7f02b41cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:57.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.473+0000 7f02b98b5640 1 --2- 172.21.15.37:0/2952239091 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f02b4134b30 0x7f02b41cd530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:57.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.473+0000 7f02b98b5640 1 -- 172.21.15.37:0/2952239091 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02b40a0260 msgr2=0x7f02b41ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:57.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.473+0000 7f02b98b5640 1 --2- 172.21.15.37:0/2952239091 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02b40a0260 0x7f02b41ccd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:57.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.473+0000 7f02b98b5640 10 monclient: _finish_hunting 0 2024-04-26T00:25:57.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.473+0000 7f02b98b5640 1 monclient: found mon.noname-a 2024-04-26T00:25:57.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.473+0000 7f02b98b5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:57.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.473+0000 7f02b98b5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:25:57.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.473+0000 7f02b98b5640 1 -- 172.21.15.37:0/2952239091 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02b411c470 con 0x7f02b4135660 2024-04-26T00:25:57.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.473+0000 7f02b98b5640 1 -- 172.21.15.37:0/2952239091 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f02b4135660 msgr2=0x7f02b4139aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:57.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.473+0000 7f02b98b5640 1 --2- 172.21.15.37:0/2952239091 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f02b4135660 0x7f02b4139aa0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f02b409eb20 tx=0x7f02ac019470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:57.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.473+0000 7f02a9ffb640 1 -- 172.21.15.37:0/2952239091 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ac0b8070 con 0x7f02b4135660 2024-04-26T00:25:57.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.473+0000 7f02a9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:57.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.473+0000 7f02a9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:25:57.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.473+0000 7f02a9ffb640 10 monclient: dump: 2024-04-26T00:25:57.467 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:57.467 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:57.467 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:57.467 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:57.467 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:57.468 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:57.468 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:57.468 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:57.468 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:57.468 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:57.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.473+0000 7f02a9ffb640 10 monclient: _finish_auth 0 2024-04-26T00:25:57.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.473+0000 7f02a9ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:25:57.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.473+0000 7f02a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:57.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.473+0000 7f02a9ffb640 1 -- 172.21.15.37:0/2952239091 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02ac0a9a80 con 0x7f02b4135660 2024-04-26T00:25:57.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.473+0000 7f02bb318640 5 monclient: authenticate success, global_id 54297 2024-04-26T00:25:57.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.473+0000 7f02bb318640 10 monclient: _renew_subs 2024-04-26T00:25:57.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.473+0000 7f02bb318640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:57.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.473+0000 7f02bb318640 1 -- 172.21.15.37:0/2952239091 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02b413b6f0 con 0x7f02b4135660 2024-04-26T00:25:57.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.473+0000 7f02a9ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:57.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.473+0000 7f02a9ffb640 1 -- 172.21.15.37:0/2952239091 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ac0bc410 con 0x7f02b4135660 2024-04-26T00:25:57.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.473+0000 7f02a9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:57.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.473+0000 7f02a9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:57.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.473+0000 7f02a9ffb640 10 monclient: dump: 2024-04-26T00:25:57.469 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:57.469 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:57.469 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:57.469 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:57.469 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:57.469 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:57.469 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:57.469 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:57.469 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:57.469 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:57.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.473+0000 7f02bb318640 10 monclient: _renew_subs 2024-04-26T00:25:57.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.473+0000 7f02bb318640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:57.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.473+0000 7f02ab7fe640 4 set_mon_vals no callback set 2024-04-26T00:25:57.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.473+0000 7f02bb318640 1 -- 172.21.15.37:0/2952239091 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02b411bbd0 con 0x7f02b4135660 2024-04-26T00:25:57.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.473+0000 7f02bb318640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:57.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.473+0000 7f02bb318640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:57.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.473+0000 7f02bb318640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:57.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.473+0000 7f02bb318640 1 -- 172.21.15.37:0/2952239091 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02b4066f40 con 0x7f02b4135660 2024-04-26T00:25:57.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.473+0000 7f02a9ffb640 1 -- 172.21.15.37:0/2952239091 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f02ac0bc5b0 con 0x7f02b4135660 2024-04-26T00:25:57.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.473+0000 7f02a9ffb640 1 --2- 172.21.15.37:0/2952239091 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0298035e80 0x7f0298038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:57.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.473+0000 7f02a9ffb640 1 -- 172.21.15.37:0/2952239091 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(234..234 src has 177..234) ==== 4067+0+0 (secure 0 0 0) 0x7f02ac0e2e20 con 0x7f02b4135660 2024-04-26T00:25:57.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.477+0000 7f02b90b4640 1 --2- 172.21.15.37:0/2952239091 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0298035e80 0x7f0298038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:57.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.477+0000 7f02b90b4640 10 monclient: get_auth_request con 0x7f0298035e80 auth_method 0 2024-04-26T00:25:57.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.477+0000 7f02b90b4640 1 --2- 172.21.15.37:0/2952239091 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0298035e80 0x7f0298038340 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f02b0009a70 tx=0x7f02b0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:57.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.481+0000 7f02a9ffb640 1 -- 172.21.15.37:0/2952239091 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f02b40a1130 con 0x7f02b4135660 2024-04-26T00:25:57.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.481+0000 7f02a9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:57.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.481+0000 7f02a9ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:25:57.550 INFO:tasks.rados.rados.0.smithi045.stdout:969: finishing copy_from racing read to smithi04526579-3 2024-04-26T00:25:57.550 INFO:tasks.rados.rados.0.smithi045.stdout:969: finishing copy_from to smithi04526579-3 2024-04-26T00:25:57.550 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 357 (ObjNum 114 snap 28 seq_num 114) dirty exists 2024-04-26T00:25:57.550 INFO:tasks.rados.rados.0.smithi045.stdout:959: done (7 left) 2024-04-26T00:25:57.550 INFO:tasks.rados.rados.0.smithi045.stdout:960: done (6 left) 2024-04-26T00:25:57.550 INFO:tasks.rados.rados.0.smithi045.stdout:961: done (5 left) 2024-04-26T00:25:57.550 INFO:tasks.rados.rados.0.smithi045.stdout:962: done (4 left) 2024-04-26T00:25:57.551 INFO:tasks.rados.rados.0.smithi045.stdout:964: done (3 left) 2024-04-26T00:25:57.551 INFO:tasks.rados.rados.0.smithi045.stdout:965: done (2 left) 2024-04-26T00:25:57.551 INFO:tasks.rados.rados.0.smithi045.stdout:966: done (1 left) 2024-04-26T00:25:57.551 INFO:tasks.rados.rados.0.smithi045.stdout:969: done (0 left) 2024-04-26T00:25:57.551 INFO:tasks.rados.rados.0.smithi045.stdout:970: rollback oid 38 current snap is 114 2024-04-26T00:25:57.551 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 38 to 99 2024-04-26T00:25:57.551 INFO:tasks.rados.rados.0.smithi045.stdout:971: copy_from oid 43 from oid 42 current snap is 114 2024-04-26T00:25:57.551 INFO:tasks.rados.rados.0.smithi045.stdout:972: snap_create 2024-04-26T00:25:57.553 INFO:tasks.rados.rados.0.smithi045.stdout:970: finishing rollback tid 1 to smithi04526579-38 2024-04-26T00:25:57.553 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 134 (ObjNum 65 snap 0 seq_num 2241614352) dirty dne 2024-04-26T00:25:57.575 INFO:tasks.rados.rados.0.smithi045.stdout:971: finishing copy_from racing read to smithi04526579-43 2024-04-26T00:25:57.575 INFO:tasks.rados.rados.0.smithi045.stdout:971: finishing copy_from to smithi04526579-43 2024-04-26T00:25:57.575 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 357 (ObjNum 114 snap 28 seq_num 114) dirty exists 2024-04-26T00:25:57.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.708+0000 7f02bb318640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-26T00:25:57.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.708+0000 7f02bb318640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-26T00:25:57.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.708+0000 7f02bb318640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:57.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.708+0000 7f02bb318640 1 -- 172.21.15.37:0/2952239091 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f02b4132180 con 0x7f02b4135660 2024-04-26T00:25:57.695 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.711+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:57.695 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.711+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=13,config=2,health=8,kv=9,logm=68} crc {auth=92364277,config=692619927,health=1605311028,kv=789508982,logm=4287311005}) 2024-04-26T00:25:57.695 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.711+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=1,health=12,kv=10,logm=64} crc {auth=3873503191,config=3678701417,health=3837507830,kv=1034996044,logm=4166721178}) 2024-04-26T00:25:57.695 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.711+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:57.695 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.711+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=13,config=2,health=8,kv=9,logm=68} crc {auth=92364277,config=692619927,health=1605311028,kv=789508982,logm=4287311005}) 2024-04-26T00:25:57.695 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.711+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=14,config=2,health=11,kv=8,logm=65} crc {auth=2317308809,config=692619927,health=1446479340,kv=2379020281,logm=1454894530}) 2024-04-26T00:25:57.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.712+0000 7f02a9ffb640 1 -- 172.21.15.37:0/2952239091 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f02ac0a8580 con 0x7f02b4135660 2024-04-26T00:25:57.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.712+0000 7f02a9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-26T00:25:57.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.712+0000 7f02a9ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:25:57.697 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.711+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:57.697 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.711+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=441591828}) 2024-04-26T00:25:57.697 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.711+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1388568509}) 2024-04-26T00:25:57.698 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.711+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:57.698 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.711+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=441591828}) 2024-04-26T00:25:57.698 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.711+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1256591825}) 2024-04-26T00:25:57.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.712+0000 7f02bb318640 1 -- 172.21.15.37:0/2952239091 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0298035e80 msgr2=0x7f0298038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:57.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.712+0000 7f02bb318640 1 --2- 172.21.15.37:0/2952239091 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0298035e80 0x7f0298038340 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f02b0009a70 tx=0x7f02b0008040 comp rx=0 tx=0).stop 2024-04-26T00:25:57.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.712+0000 7f02bb318640 10 monclient: shutdown 2024-04-26T00:25:57.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.712+0000 7f02bb318640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:57.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.712+0000 7f02bb318640 1 -- 172.21.15.37:0/2952239091 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f02b4135660 msgr2=0x7f02b4139aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:57.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.712+0000 7f02bb318640 1 --2- 172.21.15.37:0/2952239091 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f02b4135660 0x7f02b4139aa0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f02b409eb20 tx=0x7f02ac019470 comp rx=0 tx=0).stop 2024-04-26T00:25:57.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.712+0000 7f02bb318640 1 -- 172.21.15.37:0/2952239091 shutdown_connections 2024-04-26T00:25:57.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.712+0000 7f02bb318640 1 --2- 172.21.15.37:0/2952239091 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0298035e80 0x7f0298038340 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:57.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.712+0000 7f02bb318640 1 --2- 172.21.15.37:0/2952239091 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f02b4135660 0x7f02b4139aa0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:57.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.712+0000 7f02bb318640 1 --2- 172.21.15.37:0/2952239091 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f02b4134b30 0x7f02b41cd530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:57.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.712+0000 7f02bb318640 1 --2- 172.21.15.37:0/2952239091 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02b40a0260 0x7f02b41ccd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:57.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.712+0000 7f02bb318640 1 -- 172.21.15.37:0/2952239091 >> 172.21.15.37:0/2952239091 conn(0x7f02b40a1ad0 msgr2=0x7f02b40a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:57.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.712+0000 7f02bb318640 1 -- 172.21.15.37:0/2952239091 shutdown_connections 2024-04-26T00:25:57.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:57.712+0000 7f02bb318640 1 -- 172.21.15.37:0/2952239091 wait complete. 2024-04-26T00:25:57.701 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.715+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:57.701 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.715+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3181602493}) 2024-04-26T00:25:57.701 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.715+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3011284656}) 2024-04-26T00:25:57.701 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.715+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:57.701 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.715+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3181602493}) 2024-04-26T00:25:57.701 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.715+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2923710643}) 2024-04-26T00:25:57.701 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.715+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:57.701 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.715+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=892784107}) 2024-04-26T00:25:57.701 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.715+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4123054702}) 2024-04-26T00:25:57.702 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.715+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:57.702 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.715+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=892784107}) 2024-04-26T00:25:57.702 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.715+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=916156621}) 2024-04-26T00:25:57.702 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.719+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:57.702 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.719+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2223646398}) 2024-04-26T00:25:57.703 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.719+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3821745579}) 2024-04-26T00:25:57.703 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.719+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:57.703 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.719+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2223646398}) 2024-04-26T00:25:57.703 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.719+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3798937728}) 2024-04-26T00:25:57.704 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.719+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:57.704 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.719+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1777079667}) 2024-04-26T00:25:57.704 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.719+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3610791299}) 2024-04-26T00:25:57.704 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.719+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:57.704 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.719+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1777079667}) 2024-04-26T00:25:57.704 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.719+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3622637258}) 2024-04-26T00:25:57.706 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.723+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:57.706 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.723+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1309059027}) 2024-04-26T00:25:57.706 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.723+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3775874597}) 2024-04-26T00:25:57.706 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.723+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:57.706 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.723+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1309059027}) 2024-04-26T00:25:57.706 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.723+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2315713709}) 2024-04-26T00:25:57.708 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.723+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:57.708 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.723+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=272814743}) 2024-04-26T00:25:57.708 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.723+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3670169111}) 2024-04-26T00:25:57.708 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.723+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:57.708 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.723+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=272814743}) 2024-04-26T00:25:57.708 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.723+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1793749789}) 2024-04-26T00:25:57.709 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.727+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:57.709 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.727+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3874356211}) 2024-04-26T00:25:57.709 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.727+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2316329802}) 2024-04-26T00:25:57.709 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.727+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:57.710 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.727+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3874356211}) 2024-04-26T00:25:57.710 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.727+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3461432920}) 2024-04-26T00:25:57.711 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.727+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:57.711 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.727+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2747452415}) 2024-04-26T00:25:57.711 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.727+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2959168315}) 2024-04-26T00:25:57.711 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.727+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:57.712 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.727+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2747452415}) 2024-04-26T00:25:57.712 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.727+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1771483608}) 2024-04-26T00:25:57.713 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.731+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:57.713 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.731+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2254563610}) 2024-04-26T00:25:57.713 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.731+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1125243800}) 2024-04-26T00:25:57.713 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.731+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:57.714 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.731+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2254563610}) 2024-04-26T00:25:57.714 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.731+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2619980857}) 2024-04-26T00:25:57.715 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.731+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:57.715 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.731+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1349879047}) 2024-04-26T00:25:57.716 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.731+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4102391582}) 2024-04-26T00:25:57.716 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.731+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:57.716 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.731+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1349879047}) 2024-04-26T00:25:57.716 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.731+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1645290918}) 2024-04-26T00:25:57.717 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-26T00:25:57.717 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.735+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:57.718 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.735+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3835539507}) 2024-04-26T00:25:57.718 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.735+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2570662101}) 2024-04-26T00:25:57.718 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.735+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:57.718 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.735+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3835539507}) 2024-04-26T00:25:57.718 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.735+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=459156276}) 2024-04-26T00:25:57.719 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.735+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:57.719 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.735+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=105746996}) 2024-04-26T00:25:57.720 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.735+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=378407143}) 2024-04-26T00:25:57.720 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.735+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:57.720 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.735+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=105746996}) 2024-04-26T00:25:57.720 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.735+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3649219952}) 2024-04-26T00:25:57.721 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.739+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:57.721 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.739+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4201521594}) 2024-04-26T00:25:57.721 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.739+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3285479764}) 2024-04-26T00:25:57.721 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.739+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:57.721 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.739+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4201521594}) 2024-04-26T00:25:57.722 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.739+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2841024037}) 2024-04-26T00:25:57.723 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.739+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:57.723 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.739+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=76,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=6,mon_config_key=6} crc {logm=431864702,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgrstat=2532459853,mon_config_key=918105881}) 2024-04-26T00:25:57.723 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.739+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=78,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=1} crc {logm=3026222479,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=4179082600,mon_config_key=2026253307}) 2024-04-26T00:25:57.723 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.739+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:57.723 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.739+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=76,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=6,mon_config_key=6} crc {logm=431864702,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgrstat=2532459853,mon_config_key=918105881}) 2024-04-26T00:25:57.723 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.739+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=75,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=4} crc {logm=880203256,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=4179082599,mon_config_key=1924277172}) 2024-04-26T00:25:57.725 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.743+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:57.725 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.743+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=38,osdmap=47} crc {mon_config_key=225952674,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=999123361,osd_snap=4044181083,osdmap=2039290508}) 2024-04-26T00:25:57.725 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.743+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=40,osdmap=46} crc {mon_config_key=225952674,monmap=3876866194,osd_metadata=839604482,osd_pg_creating=999123361,osd_snap=1026907176,osdmap=4038253013}) 2024-04-26T00:25:57.725 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.743+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:57.725 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.743+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=38,osdmap=47} crc {mon_config_key=225952674,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=999123361,osd_snap=4044181083,osdmap=2039290508}) 2024-04-26T00:25:57.725 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.743+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=4,monmap=1,osd_metadata=6,osd_pg_creating=1,osd_snap=39,osdmap=49} crc {mon_config_key=1489557450,monmap=1228700967,osd_metadata=3306121363,osd_pg_creating=999123361,osd_snap=1763879347,osdmap=1463296039}) 2024-04-26T00:25:57.732 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.747+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:57.732 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.747+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=54} crc {osdmap=2507996673}) 2024-04-26T00:25:57.732 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.747+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=51} crc {osdmap=3630755152}) 2024-04-26T00:25:57.732 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.747+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:25:57.732 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.747+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=54} crc {osdmap=2507996673}) 2024-04-26T00:25:57.732 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:25:57.747+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=53} crc {osdmap=3098593738}) 2024-04-26T00:25:57.944 INFO:tasks.rados.rados.0.smithi045.stdout:970: done (2 left) 2024-04-26T00:25:57.944 INFO:tasks.rados.rados.0.smithi045.stdout:971: done (1 left) 2024-04-26T00:25:57.944 INFO:tasks.rados.rados.0.smithi045.stdout:972: done (0 left) 2024-04-26T00:25:57.944 INFO:tasks.rados.rados.0.smithi045.stdout:973: read oid 9 snap -1 2024-04-26T00:25:57.944 INFO:tasks.rados.rados.0.smithi045.stdout:973: expect (ObjNum 371 snap 112 seq_num 371) 2024-04-26T00:25:57.944 INFO:tasks.rados.rados.0.smithi045.stdout:974: copy_from oid 5 from oid 44 current snap is 115 2024-04-26T00:25:57.944 INFO:tasks.rados.rados.0.smithi045.stdout:975: read oid 26 snap -1 2024-04-26T00:25:57.944 INFO:tasks.rados.rados.0.smithi045.stdout:975: expect (ObjNum 254 snap 64 seq_num 254) 2024-04-26T00:25:57.944 INFO:tasks.rados.rados.0.smithi045.stdout:976: write (excl) oid 15 current snap is 115 2024-04-26T00:25:57.944 INFO:tasks.rados.rados.0.smithi045.stdout:976: seq_num 376 ranges {719432=794921,2186053=133379} 2024-04-26T00:25:57.972 INFO:tasks.rados.rados.0.smithi045.stdout:976: writing smithi04526579-15 from 719432 to 1514353 tid 1 2024-04-26T00:25:57.986 INFO:tasks.rados.rados.0.smithi045.stdout:976: writing smithi04526579-15 from 2186053 to 2319432 tid 2 2024-04-26T00:25:57.986 INFO:tasks.rados.rados.0.smithi045.stdout:977: read oid 33 snap -1 2024-04-26T00:25:57.986 INFO:tasks.rados.rados.0.smithi045.stdout:977: expect (ObjNum 358 snap 107 seq_num 358) 2024-04-26T00:25:57.986 INFO:tasks.rados.rados.0.smithi045.stdout:978: write oid 39 current snap is 115 2024-04-26T00:25:57.986 INFO:tasks.rados.rados.0.smithi045.stdout:978: seq_num 377 ranges {736239=470170,1728182=571782} 2024-04-26T00:25:58.006 INFO:tasks.rados.rados.0.smithi045.stdout:978: writing smithi04526579-39 from 736239 to 1206409 tid 1 2024-04-26T00:25:58.018 INFO:tasks.rados.rados.0.smithi045.stdout:978: writing smithi04526579-39 from 1728182 to 2299964 tid 2 2024-04-26T00:25:58.018 INFO:tasks.rados.rados.0.smithi045.stdout:979: read oid 4 snap -1 2024-04-26T00:25:58.018 INFO:tasks.rados.rados.0.smithi045.stdout:979: expect deleted 2024-04-26T00:25:58.018 INFO:tasks.rados.rados.0.smithi045.stdout:980: delete oid 41 current snap is 115 2024-04-26T00:25:58.018 INFO:tasks.rados.rados.0.smithi045.stdout:973: expect (ObjNum 371 snap 112 seq_num 371) 2024-04-26T00:25:58.177 INFO:tasks.rados.rados.0.smithi045.stdout:975: expect (ObjNum 254 snap 64 seq_num 254) 2024-04-26T00:25:58.344 INFO:tasks.rados.rados.0.smithi045.stdout:974: finishing copy_from racing read to smithi04526579-5 2024-04-26T00:25:58.344 INFO:tasks.rados.rados.0.smithi045.stdout:974: finishing copy_from to smithi04526579-5 2024-04-26T00:25:58.344 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 178 (ObjNum 283 snap 81 seq_num 283) dirty exists 2024-04-26T00:25:58.344 INFO:tasks.rados.rados.0.smithi045.stdout:976: finishing write tid 1 to smithi04526579-15 2024-04-26T00:25:58.344 INFO:tasks.rados.rados.0.smithi045.stdout:976: finishing write tid 2 to smithi04526579-15 2024-04-26T00:25:58.344 INFO:tasks.rados.rados.0.smithi045.stdout:976: finishing write tid 3 to smithi04526579-15 2024-04-26T00:25:58.344 INFO:tasks.rados.rados.0.smithi045.stdout:976: finishing write tid 4 to smithi04526579-15 2024-04-26T00:25:58.344 INFO:tasks.rados.rados.0.smithi045.stdout:976: oid 15 updating version 0 to 587 2024-04-26T00:25:58.344 INFO:tasks.rados.rados.0.smithi045.stdout:976: oid 15 version 587 is already newer than 586 2024-04-26T00:25:58.344 INFO:tasks.rados.rados.0.smithi045.stdout:976: oid 15 version 587 is already newer than 585 2024-04-26T00:25:58.344 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 587 (ObjNum 376 snap 115 seq_num 376) dirty exists 2024-04-26T00:25:58.345 INFO:tasks.rados.rados.0.smithi045.stdout:976: left oid 15 (ObjNum 376 snap 115 seq_num 376) 2024-04-26T00:25:58.345 INFO:tasks.rados.rados.0.smithi045.stdout:977: expect (ObjNum 358 snap 107 seq_num 358) 2024-04-26T00:25:58.437 INFO:tasks.rados.rados.0.smithi045.stdout:978: finishing write tid 1 to smithi04526579-39 2024-04-26T00:25:58.437 INFO:tasks.rados.rados.0.smithi045.stdout:978: finishing write tid 2 to smithi04526579-39 2024-04-26T00:25:58.437 INFO:tasks.rados.rados.0.smithi045.stdout:978: finishing write tid 3 to smithi04526579-39 2024-04-26T00:25:58.437 INFO:tasks.rados.rados.0.smithi045.stdout:978: finishing write tid 4 to smithi04526579-39 2024-04-26T00:25:58.437 INFO:tasks.rados.rados.0.smithi045.stdout:978: oid 39 updating version 0 to 354 2024-04-26T00:25:58.437 INFO:tasks.rados.rados.0.smithi045.stdout:978: oid 39 version 354 is already newer than 353 2024-04-26T00:25:58.437 INFO:tasks.rados.rados.0.smithi045.stdout:978: oid 39 updating version 354 to 355 2024-04-26T00:25:58.437 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 355 (ObjNum 377 snap 115 seq_num 377) dirty exists 2024-04-26T00:25:58.437 INFO:tasks.rados.rados.0.smithi045.stdout:978: left oid 39 (ObjNum 377 snap 115 seq_num 377) 2024-04-26T00:25:58.437 INFO:tasks.rados.rados.0.smithi045.stdout:973: done (7 left) 2024-04-26T00:25:58.437 INFO:tasks.rados.rados.0.smithi045.stdout:974: done (6 left) 2024-04-26T00:25:58.437 INFO:tasks.rados.rados.0.smithi045.stdout:975: done (5 left) 2024-04-26T00:25:58.438 INFO:tasks.rados.rados.0.smithi045.stdout:976: done (4 left) 2024-04-26T00:25:58.438 INFO:tasks.rados.rados.0.smithi045.stdout:977: done (3 left) 2024-04-26T00:25:58.438 INFO:tasks.rados.rados.0.smithi045.stdout:978: done (2 left) 2024-04-26T00:25:58.438 INFO:tasks.rados.rados.0.smithi045.stdout:979: done (1 left) 2024-04-26T00:25:58.438 INFO:tasks.rados.rados.0.smithi045.stdout:980: done (0 left) 2024-04-26T00:25:58.438 INFO:tasks.rados.rados.0.smithi045.stdout:981: write (excl) oid 27 current snap is 115 2024-04-26T00:25:58.438 INFO:tasks.rados.rados.0.smithi045.stdout:981: seq_num 379 ranges {769853=620668,1845146=524707} 2024-04-26T00:25:58.459 INFO:tasks.rados.rados.0.smithi045.stdout:981: writing smithi04526579-27 from 769853 to 1390521 tid 1 2024-04-26T00:25:58.475 INFO:tasks.rados.rados.0.smithi045.stdout:981: writing smithi04526579-27 from 1845146 to 2369853 tid 2 2024-04-26T00:25:58.475 INFO:tasks.rados.rados.0.smithi045.stdout:982: snap_create 2024-04-26T00:25:58.476 INFO:tasks.rados.rados.0.smithi045.stdout:981: finishing write tid 1 to smithi04526579-27 2024-04-26T00:25:58.488 INFO:tasks.rados.rados.0.smithi045.stdout:981: finishing write tid 2 to smithi04526579-27 2024-04-26T00:25:58.489 INFO:tasks.rados.rados.0.smithi045.stdout:981: finishing write tid 3 to smithi04526579-27 2024-04-26T00:25:58.490 INFO:tasks.rados.rados.0.smithi045.stdout:981: finishing write tid 4 to smithi04526579-27 2024-04-26T00:25:58.490 INFO:tasks.rados.rados.0.smithi045.stdout:981: oid 27 updating version 0 to 491 2024-04-26T00:25:58.490 INFO:tasks.rados.rados.0.smithi045.stdout:981: oid 27 updating version 491 to 492 2024-04-26T00:25:58.490 INFO:tasks.rados.rados.0.smithi045.stdout:981: oid 27 version 492 is already newer than 490 2024-04-26T00:25:58.490 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 492 (ObjNum 379 snap 115 seq_num 379) dirty exists 2024-04-26T00:25:58.490 INFO:tasks.rados.rados.0.smithi045.stdout:981: left oid 27 (ObjNum 379 snap 115 seq_num 379) 2024-04-26T00:25:58.717 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:25:58.717 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:25:58.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.864+0000 7fd8983e7640 10 monclient: get_monmap_and_config 2024-04-26T00:25:58.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.864+0000 7fd8983e7640 10 monclient: build_initial_monmap 2024-04-26T00:25:58.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.864+0000 7fd8983e7640 10 monclient: monmap: 2024-04-26T00:25:58.849 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:58.849 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:58.849 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:25:58.869926+0000 2024-04-26T00:25:58.849 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:25:58.869926+0000 2024-04-26T00:25:58.849 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:58.849 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:58.849 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:25:58.849 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:25:58.849 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:25:58.849 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:25:58.849 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:25:58.849 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:58.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.864+0000 7fd8983e7640 1 Processor -- start 2024-04-26T00:25:58.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.864+0000 7fd8983e7640 1 -- start start 2024-04-26T00:25:58.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.864+0000 7fd8983e7640 10 monclient: init 2024-04-26T00:25:58.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.864+0000 7fd8983e7640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:58.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.864+0000 7fd8983e7640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-04-26T00:25:58.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.864+0000 7fd8983e7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd8900a0260 addr v1:172.21.15.45:6789/0 2024-04-26T00:25:58.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.864+0000 7fd8983e7640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd8900a0d50 0x7fd890072e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:58.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.864+0000 7fd8983e7640 10 monclient(hunting): picked mon.noname-c con 0x7fd8900a0d50 addr v2:172.21.15.45:3300/0 2024-04-26T00:25:58.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.864+0000 7fd8983e7640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd8901389e0 0x7fd89013add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:58.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.864+0000 7fd8983e7640 10 monclient(hunting): picked mon.noname-b con 0x7fd8901389e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:58.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.864+0000 7fd8983e7640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8900a12e0 con 0x7fd8900a0260 2024-04-26T00:25:58.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.864+0000 7fd8983e7640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:58.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.864+0000 7fd8983e7640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fd89006d050 con 0x7fd8900a0d50 2024-04-26T00:25:58.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.864+0000 7fd8983e7640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:58.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.864+0000 7fd8983e7640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd89009dcb0 con 0x7fd8901389e0 2024-04-26T00:25:58.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.864+0000 7fd8983e7640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:58.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.864+0000 7fd8983e7640 10 monclient(hunting): authenticate will time out at 1340.175763s 2024-04-26T00:25:58.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.868+0000 7fd895982640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd8900a0d50 0x7fd890072e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:58.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.868+0000 7fd896984640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd8901389e0 0x7fd89013add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:58.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.868+0000 7fd895982640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd8900a0d50 0x7fd890072e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.37:42624/0 (socket says 172.21.15.37:42624) 2024-04-26T00:25:58.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.868+0000 7fd895982640 1 -- 172.21.15.37:0/2679144111 learned_addr learned my addr 172.21.15.37:0/2679144111 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:58.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.868+0000 7fd895982640 10 monclient(hunting): get_auth_request con 0x7fd8900a0d50 auth_method 0 2024-04-26T00:25:58.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.868+0000 7fd895982640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:58.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.868+0000 7fd895982640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:58.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.868+0000 7fd895982640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:58.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.868+0000 7fd896984640 10 monclient(hunting): get_auth_request con 0x7fd8901389e0 auth_method 0 2024-04-26T00:25:58.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.868+0000 7fd896984640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:58.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.868+0000 7fd896984640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:58.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.868+0000 7fd896984640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:58.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.868+0000 7fd896984640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:58.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.868+0000 7fd896984640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:58.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.868+0000 7fd896984640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:58.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.868+0000 7fd895982640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:58.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.868+0000 7fd895982640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:58.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.868+0000 7fd895982640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:58.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.868+0000 7fd896984640 10 monclient(hunting): handle_auth_done global_id 74120 payload 274 2024-04-26T00:25:58.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.868+0000 7fd896984640 1 -- 172.21.15.37:0/2679144111 >> v2:172.21.15.45:3300/0 conn(0x7fd8900a0d50 msgr2=0x7fd890072e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:58.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.868+0000 7fd896984640 1 --2- 172.21.15.37:0/2679144111 >> v2:172.21.15.45:3300/0 conn(0x7fd8900a0d50 0x7fd890072e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:58.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.868+0000 7fd896984640 1 -- 172.21.15.37:0/2679144111 >> v1:172.21.15.45:6789/0 conn(0x7fd8900a0260 legacy=0x7fd8900a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:58.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.868+0000 7fd896984640 10 monclient: _finish_hunting 0 2024-04-26T00:25:58.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.868+0000 7fd896984640 1 monclient: found mon.noname-b 2024-04-26T00:25:58.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.868+0000 7fd896984640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:58.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.868+0000 7fd896984640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:25:58.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.868+0000 7fd896984640 1 -- 172.21.15.37:0/2679144111 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8900a16f0 con 0x7fd8901389e0 2024-04-26T00:25:58.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd896984640 1 -- 172.21.15.37:0/2679144111 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd8901389e0 msgr2=0x7fd89013add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:58.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd896984640 1 --2- 172.21.15.37:0/2679144111 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd8901389e0 0x7fd89013add0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd8880097c0 tx=0x7fd88801c3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a424f4a25be05263 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:58.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd895181640 1 -- 172.21.15.37:0/2679144111 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8880b0070 con 0x7fd8901389e0 2024-04-26T00:25:58.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd895181640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:58.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd895181640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:25:58.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd895181640 10 monclient: dump: 2024-04-26T00:25:58.856 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:58.856 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:58.856 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:58.856 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:58.856 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:58.856 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:58.856 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:58.856 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:58.856 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:58.856 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:58.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd895181640 10 monclient: _finish_auth 0 2024-04-26T00:25:58.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd895181640 10 monclient: _check_auth_tickets 2024-04-26T00:25:58.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd895181640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:58.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd895181640 1 -- 172.21.15.37:0/2679144111 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8880ab600 con 0x7fd8901389e0 2024-04-26T00:25:58.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd8983e7640 5 monclient: authenticate success, global_id 74120 2024-04-26T00:25:58.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd895181640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:58.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd895181640 1 -- 172.21.15.37:0/2679144111 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8880ab9d0 con 0x7fd8901389e0 2024-04-26T00:25:58.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd895181640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:58.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd895181640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:25:58.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd895181640 10 monclient: dump: 2024-04-26T00:25:58.857 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:58.857 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:58.857 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:58.857 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:58.857 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:58.857 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:58.857 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:58.858 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:58.858 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:58.858 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:58.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd8983e7640 10 monclient: get_monmap_and_config success 2024-04-26T00:25:58.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd8983e7640 4 set_mon_vals no callback set 2024-04-26T00:25:58.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd8983e7640 10 monclient: shutdown 2024-04-26T00:25:58.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd8983e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:58.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd8983e7640 1 -- 172.21.15.37:0/2679144111 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd8901389e0 msgr2=0x7fd89013add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:58.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd8983e7640 1 --2- 172.21.15.37:0/2679144111 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd8901389e0 0x7fd89013add0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd8880097c0 tx=0x7fd88801c3b0 comp rx=0 tx=0).stop 2024-04-26T00:25:58.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd8983e7640 1 -- 172.21.15.37:0/2679144111 shutdown_connections 2024-04-26T00:25:58.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd8983e7640 1 --2- 172.21.15.37:0/2679144111 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd8901389e0 0x7fd89013add0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:58.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd8983e7640 1 --2- 172.21.15.37:0/2679144111 >> v2:172.21.15.45:3300/0 conn(0x7fd8900a0d50 0x7fd890072e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:58.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd8983e7640 1 -- 172.21.15.37:0/2679144111 >> 172.21.15.37:0/2679144111 conn(0x7fd8900a1ad0 msgr2=0x7fd8900a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:58.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd8983e7640 1 -- 172.21.15.37:0/2679144111 shutdown_connections 2024-04-26T00:25:58.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd8983e7640 1 -- 172.21.15.37:0/2679144111 wait complete. 2024-04-26T00:25:58.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd8983e7640 10 monclient: build_initial_monmap 2024-04-26T00:25:58.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd8983e7640 10 monclient: monmap: 2024-04-26T00:25:58.863 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:58.864 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:25:58.864 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:25:58.864 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:25:58.864 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:58.864 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:25:58.864 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:25:58.864 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:25:58.864 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:25:58.864 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:58.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd8983e7640 1 Processor -- start 2024-04-26T00:25:58.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd8983e7640 1 -- start start 2024-04-26T00:25:58.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd8983e7640 10 monclient: init 2024-04-26T00:25:58.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd8983e7640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:58.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd8983e7640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:25:58.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd8983e7640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd8900a0260 0x7fd8901cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:58.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd8983e7640 10 monclient(hunting): picked mon.noname-c con 0x7fd8900a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:58.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd8983e7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8900a0d50 0x7fd890136c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:58.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd8983e7640 10 monclient(hunting): picked mon.noname-b con 0x7fd8900a0d50 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:25:58.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd8983e7640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd8901389e0 0x7fd8901ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:58.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd8983e7640 10 monclient(hunting): picked mon.noname-a con 0x7fd8901389e0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:25:58.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd8983e7640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:58.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd8983e7640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd89013d6b0 con 0x7fd8901389e0 2024-04-26T00:25:58.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd8983e7640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:58.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd8983e7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd89013d530 con 0x7fd8900a0d50 2024-04-26T00:25:58.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd8983e7640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:58.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd8983e7640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd89013d830 con 0x7fd8900a0260 2024-04-26T00:25:58.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd8983e7640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:58.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd8983e7640 10 monclient(hunting): authenticate will time out at 1340.183295s 2024-04-26T00:25:58.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd896183640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd8900a0260 0x7fd8901cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:58.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd896183640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd8900a0260 0x7fd8901cb9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51884/0 (socket says 172.21.15.37:51884) 2024-04-26T00:25:58.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd896183640 1 -- 172.21.15.37:0/3528375008 learned_addr learned my addr 172.21.15.37:0/3528375008 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:58.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd896183640 10 monclient(hunting): get_auth_request con 0x7fd8900a0260 auth_method 0 2024-04-26T00:25:58.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd896183640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:58.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd896183640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:58.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd896183640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:58.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd896984640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd8901389e0 0x7fd8901ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:58.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd896984640 10 monclient(hunting): get_auth_request con 0x7fd8901389e0 auth_method 0 2024-04-26T00:25:58.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd896984640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:58.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd896984640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:58.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd896984640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:58.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd896984640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:58.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd896984640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:58.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd896984640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:58.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd896183640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:58.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd896183640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:58.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd896183640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:58.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd895982640 1 --2- 172.21.15.37:0/3528375008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8900a0d50 0x7fd890136c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:58.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd895982640 10 monclient(hunting): get_auth_request con 0x7fd8900a0d50 auth_method 0 2024-04-26T00:25:58.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd895982640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:58.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd895982640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:58.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd895982640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:58.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd896984640 10 monclient(hunting): handle_auth_done global_id 54300 payload 824 2024-04-26T00:25:58.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd896984640 1 -- 172.21.15.37:0/3528375008 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd8900a0260 msgr2=0x7fd8901cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:58.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd896984640 1 --2- 172.21.15.37:0/3528375008 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd8900a0260 0x7fd8901cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:58.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd896984640 1 -- 172.21.15.37:0/3528375008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8900a0d50 msgr2=0x7fd890136c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:58.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd896984640 1 --2- 172.21.15.37:0/3528375008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8900a0d50 0x7fd890136c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:58.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd896984640 10 monclient: _finish_hunting 0 2024-04-26T00:25:58.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd896984640 1 monclient: found mon.noname-a 2024-04-26T00:25:58.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd896984640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:58.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd896984640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:25:58.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.872+0000 7fd896984640 1 -- 172.21.15.37:0/3528375008 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd89013aa90 con 0x7fd8901389e0 2024-04-26T00:25:58.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.876+0000 7fd896984640 1 -- 172.21.15.37:0/3528375008 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd8901389e0 msgr2=0x7fd8901ceee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:58.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.876+0000 7fd896984640 1 --2- 172.21.15.37:0/3528375008 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd8901389e0 0x7fd8901ceee0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd8880ad000 tx=0x7fd888019150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:58.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.876+0000 7fd882ffd640 1 -- 172.21.15.37:0/3528375008 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8880b8020 con 0x7fd8901389e0 2024-04-26T00:25:58.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.876+0000 7fd882ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:58.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.876+0000 7fd882ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:25:58.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.876+0000 7fd882ffd640 10 monclient: dump: 2024-04-26T00:25:58.868 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:58.868 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:58.868 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:58.868 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:58.868 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:58.868 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:58.868 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:58.869 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:58.869 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:58.869 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:58.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.876+0000 7fd882ffd640 10 monclient: _finish_auth 0 2024-04-26T00:25:58.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.876+0000 7fd882ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:25:58.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.876+0000 7fd882ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:58.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.876+0000 7fd882ffd640 1 -- 172.21.15.37:0/3528375008 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8880b4610 con 0x7fd8901389e0 2024-04-26T00:25:58.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.876+0000 7fd8983e7640 5 monclient: authenticate success, global_id 54300 2024-04-26T00:25:58.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.876+0000 7fd882ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:58.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.876+0000 7fd882ffd640 1 -- 172.21.15.37:0/3528375008 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8880b0040 con 0x7fd8901389e0 2024-04-26T00:25:58.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.876+0000 7fd8983e7640 10 monclient: _renew_subs 2024-04-26T00:25:58.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.876+0000 7fd8983e7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:58.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.876+0000 7fd8983e7640 1 -- 172.21.15.37:0/3528375008 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd890124d80 con 0x7fd8901389e0 2024-04-26T00:25:58.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.876+0000 7fd882ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:58.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.876+0000 7fd882ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:58.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.876+0000 7fd882ffd640 10 monclient: dump: 2024-04-26T00:25:58.870 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:58.870 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:58.870 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:58.870 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:58.870 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:58.870 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:58.870 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:58.870 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:58.870 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:58.870 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:58.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.876+0000 7fd894980640 4 set_mon_vals no callback set 2024-04-26T00:25:58.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.876+0000 7fd8983e7640 10 monclient: _renew_subs 2024-04-26T00:25:58.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.876+0000 7fd8983e7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:58.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.876+0000 7fd8983e7640 1 -- 172.21.15.37:0/3528375008 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8901cc420 con 0x7fd8901389e0 2024-04-26T00:25:58.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.876+0000 7fd882ffd640 1 -- 172.21.15.37:0/3528375008 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd8880bb020 con 0x7fd8901389e0 2024-04-26T00:25:58.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.876+0000 7fd8983e7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:58.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.876+0000 7fd8983e7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:58.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.876+0000 7fd8983e7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:58.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.876+0000 7fd8983e7640 1 -- 172.21.15.37:0/3528375008 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd890066f40 con 0x7fd8901389e0 2024-04-26T00:25:58.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.876+0000 7fd882ffd640 1 --2- 172.21.15.37:0/3528375008 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd874035f00 0x7fd8740383c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:58.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.876+0000 7fd882ffd640 1 -- 172.21.15.37:0/3528375008 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(235..235 src has 177..235) ==== 3947+0+0 (secure 0 0 0) 0x7fd8880e2660 con 0x7fd8901389e0 2024-04-26T00:25:58.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.876+0000 7fd896183640 1 --2- 172.21.15.37:0/3528375008 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd874035f00 0x7fd8740383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:58.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.876+0000 7fd896183640 10 monclient: get_auth_request con 0x7fd874035f00 auth_method 0 2024-04-26T00:25:58.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.880+0000 7fd896183640 1 --2- 172.21.15.37:0/3528375008 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd874035f00 0x7fd8740383c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd8840099b0 tx=0x7fd884009110 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:58.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.884+0000 7fd882ffd640 1 -- 172.21.15.37:0/3528375008 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fd8880c2030 con 0x7fd8901389e0 2024-04-26T00:25:58.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.884+0000 7fd882ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:58.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:58.884+0000 7fd882ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:25:58.961 INFO:tasks.rados.rados.0.smithi045.stdout:981: done (1 left) 2024-04-26T00:25:58.961 INFO:tasks.rados.rados.0.smithi045.stdout:982: done (0 left) 2024-04-26T00:25:58.962 INFO:tasks.rados.rados.0.smithi045.stdout:983: snap_create 2024-04-26T00:25:59.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.112+0000 7fd8983e7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:25:59.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.112+0000 7fd8983e7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:25:59.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.112+0000 7fd8983e7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:59.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.112+0000 7fd8983e7640 1 -- 172.21.15.37:0/3528375008 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd8901375c0 con 0x7fd8901389e0 2024-04-26T00:25:59.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.112+0000 7fd882ffd640 1 -- 172.21.15.37:0/3528375008 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fd888016030 con 0x7fd8901389e0 2024-04-26T00:25:59.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.112+0000 7fd882ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:25:59.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.112+0000 7fd882ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:25:59.096 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":46,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:25:59.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.112+0000 7fd8983e7640 1 -- 172.21.15.37:0/3528375008 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd874035f00 msgr2=0x7fd8740383c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:59.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.112+0000 7fd8983e7640 1 --2- 172.21.15.37:0/3528375008 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd874035f00 0x7fd8740383c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd8840099b0 tx=0x7fd884009110 comp rx=0 tx=0).stop 2024-04-26T00:25:59.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.112+0000 7fd8983e7640 10 monclient: shutdown 2024-04-26T00:25:59.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.112+0000 7fd8983e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:59.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.112+0000 7fd8983e7640 1 -- 172.21.15.37:0/3528375008 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd8901389e0 msgr2=0x7fd8901ceee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:59.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.112+0000 7fd8983e7640 1 --2- 172.21.15.37:0/3528375008 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd8901389e0 0x7fd8901ceee0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd8880ad000 tx=0x7fd888019150 comp rx=0 tx=0).stop 2024-04-26T00:25:59.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.112+0000 7fd8983e7640 1 -- 172.21.15.37:0/3528375008 shutdown_connections 2024-04-26T00:25:59.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.112+0000 7fd8983e7640 1 --2- 172.21.15.37:0/3528375008 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd874035f00 0x7fd8740383c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:59.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.112+0000 7fd8983e7640 1 --2- 172.21.15.37:0/3528375008 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd8901389e0 0x7fd8901ceee0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:59.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.112+0000 7fd8983e7640 1 --2- 172.21.15.37:0/3528375008 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd8900a0d50 0x7fd890136c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:59.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.112+0000 7fd8983e7640 1 --2- 172.21.15.37:0/3528375008 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd8900a0260 0x7fd8901cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:59.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.112+0000 7fd8983e7640 1 -- 172.21.15.37:0/3528375008 >> 172.21.15.37:0/3528375008 conn(0x7fd8900a1ad0 msgr2=0x7fd890070e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:59.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.112+0000 7fd8983e7640 1 -- 172.21.15.37:0/3528375008 shutdown_connections 2024-04-26T00:25:59.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.112+0000 7fd8983e7640 1 -- 172.21.15.37:0/3528375008 wait complete. 2024-04-26T00:25:59.115 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:25:59.115 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-26T00:25:59.115 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:25:59.115 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:25:59.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff52446c640 10 monclient: get_monmap_and_config 2024-04-26T00:25:59.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff52446c640 10 monclient: build_initial_monmap 2024-04-26T00:25:59.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff52446c640 10 monclient: monmap: 2024-04-26T00:25:59.232 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:59.232 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:59.232 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:25:59.252971+0000 2024-04-26T00:25:59.232 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:25:59.252971+0000 2024-04-26T00:25:59.232 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:59.232 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:59.232 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:25:59.232 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:25:59.232 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:25:59.232 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:25:59.232 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:25:59.232 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:59.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff52446c640 1 Processor -- start 2024-04-26T00:25:59.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff52446c640 1 -- start start 2024-04-26T00:25:59.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff52446c640 10 monclient: init 2024-04-26T00:25:59.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff52446c640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:59.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff52446c640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-04-26T00:25:59.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff52446c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff51c136f20 0x7ff51c09bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:59.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff52446c640 10 monclient(hunting): picked mon.noname-a con 0x7ff51c136f20 addr v2:172.21.15.37:3300/0 2024-04-26T00:25:59.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff52446c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff51c137300 addr v1:172.21.15.45:6789/0 2024-04-26T00:25:59.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff52446c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff51c13c450 0x7ff51c13e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:59.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff52446c640 10 monclient(hunting): picked mon.noname-b con 0x7ff51c13c450 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:59.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff52446c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff51c0a50f0 con 0x7ff51c137300 2024-04-26T00:25:59.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff52446c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:59.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff52446c640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7ff51c09b330 con 0x7ff51c136f20 2024-04-26T00:25:59.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff52446c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:59.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff52446c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff51c09b610 con 0x7ff51c13c450 2024-04-26T00:25:59.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff52446c640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:59.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff52446c640 10 monclient(hunting): authenticate will time out at 1340.558531s 2024-04-26T00:25:59.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff522208640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff51c136f20 0x7ff51c09bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:59.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff522a09640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff51c13c450 0x7ff51c13e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:59.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff522208640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff51c136f20 0x7ff51c09bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36688/0 (socket says 172.21.15.37:36688) 2024-04-26T00:25:59.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff522208640 1 -- 172.21.15.37:0/2997881813 learned_addr learned my addr 172.21.15.37:0/2997881813 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:59.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff522208640 10 monclient(hunting): get_auth_request con 0x7ff51c136f20 auth_method 0 2024-04-26T00:25:59.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff522208640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:59.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff522208640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:59.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff522208640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:59.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff522a09640 10 monclient(hunting): get_auth_request con 0x7ff51c13c450 auth_method 0 2024-04-26T00:25:59.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff522a09640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:59.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff522a09640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:59.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff522a09640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:59.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff522208640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:59.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff522208640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:59.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff522208640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:59.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff522a09640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:59.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff522a09640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:59.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff522a09640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:59.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff522208640 10 monclient(hunting): handle_auth_done global_id 54303 payload 274 2024-04-26T00:25:59.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff522208640 1 -- 172.21.15.37:0/2997881813 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff51c13c450 msgr2=0x7ff51c13e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:59.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff522208640 1 --2- 172.21.15.37:0/2997881813 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff51c13c450 0x7ff51c13e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:59.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff522208640 1 -- 172.21.15.37:0/2997881813 >> v1:172.21.15.45:6789/0 conn(0x7ff51c137300 legacy=0x7ff51c1377b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:59.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff522208640 10 monclient: _finish_hunting 0 2024-04-26T00:25:59.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff522208640 1 monclient: found mon.noname-a 2024-04-26T00:25:59.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff522208640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:59.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff522208640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:25:59.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff522208640 1 -- 172.21.15.37:0/2997881813 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff51c09b8f0 con 0x7ff51c136f20 2024-04-26T00:25:59.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.248+0000 7ff522a09640 1 --2- 172.21.15.37:0/2997881813 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff51c13c450 0x7ff51c13e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:25:59.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff522208640 1 -- 172.21.15.37:0/2997881813 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff51c136f20 msgr2=0x7ff51c09bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:59.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff522208640 1 --2- 172.21.15.37:0/2997881813 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff51c136f20 0x7ff51c09bff0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff518009870 tx=0x7ff51801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db53b4bbbbb1e3cf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff521206640 1 -- 172.21.15.37:0/2997881813 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5180b0070 con 0x7ff51c136f20 2024-04-26T00:25:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff521206640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff521206640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:25:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff521206640 10 monclient: dump: 2024-04-26T00:25:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:59.245 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff521206640 10 monclient: _finish_auth 0 2024-04-26T00:25:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff521206640 10 monclient: _check_auth_tickets 2024-04-26T00:25:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff521206640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff521206640 1 -- 172.21.15.37:0/2997881813 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5180ab600 con 0x7ff51c136f20 2024-04-26T00:25:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff521206640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff521206640 1 -- 172.21.15.37:0/2997881813 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5180ab9d0 con 0x7ff51c136f20 2024-04-26T00:25:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff52446c640 5 monclient: authenticate success, global_id 54303 2024-04-26T00:25:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff52446c640 10 monclient: get_monmap_and_config success 2024-04-26T00:25:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff52446c640 4 set_mon_vals no callback set 2024-04-26T00:25:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff521206640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff521206640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff521206640 10 monclient: dump: 2024-04-26T00:25:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:59.246 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff52446c640 10 monclient: shutdown 2024-04-26T00:25:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff52446c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff52446c640 1 -- 172.21.15.37:0/2997881813 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff51c136f20 msgr2=0x7ff51c09bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff52446c640 1 --2- 172.21.15.37:0/2997881813 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff51c136f20 0x7ff51c09bff0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff518009870 tx=0x7ff51801c470 comp rx=0 tx=0).stop 2024-04-26T00:25:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff52446c640 1 -- 172.21.15.37:0/2997881813 shutdown_connections 2024-04-26T00:25:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff52446c640 1 --2- 172.21.15.37:0/2997881813 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff51c13c450 0x7ff51c13e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff52446c640 1 --2- 172.21.15.37:0/2997881813 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff51c136f20 0x7ff51c09bff0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff52446c640 1 -- 172.21.15.37:0/2997881813 >> 172.21.15.37:0/2997881813 conn(0x7ff51c0a61b0 msgr2=0x7ff51c0a65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff52446c640 1 -- 172.21.15.37:0/2997881813 shutdown_connections 2024-04-26T00:25:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff52446c640 1 -- 172.21.15.37:0/2997881813 wait complete. 2024-04-26T00:25:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff52446c640 10 monclient: build_initial_monmap 2024-04-26T00:25:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff52446c640 10 monclient: monmap: 2024-04-26T00:25:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:25:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:25:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:25:59.248 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:59.248 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:25:59.248 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:25:59.248 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:25:59.248 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:25:59.248 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:59.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff52446c640 1 Processor -- start 2024-04-26T00:25:59.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff52446c640 1 -- start start 2024-04-26T00:25:59.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff52446c640 10 monclient: init 2024-04-26T00:25:59.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff52446c640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:59.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff52446c640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:25:59.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff52446c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff51c136f20 0x7ff51c134f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:59.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff52446c640 10 monclient(hunting): picked mon.noname-c con 0x7ff51c136f20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:59.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff52446c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff51c137300 0x7ff51c137ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:59.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff52446c640 10 monclient(hunting): picked mon.noname-a con 0x7ff51c137300 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:25:59.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff52446c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff51c13c450 0x7ff51c138430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:59.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff52446c640 10 monclient(hunting): picked mon.noname-b con 0x7ff51c13c450 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:25:59.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff52446c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:59.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff52446c640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff51c141930 con 0x7ff51c137300 2024-04-26T00:25:59.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff52446c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:59.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff52446c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff51c1417b0 con 0x7ff51c13c450 2024-04-26T00:25:59.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff52446c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:59.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff52446c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff51c141ab0 con 0x7ff51c136f20 2024-04-26T00:25:59.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff52446c640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:59.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff52446c640 10 monclient(hunting): authenticate will time out at 1340.567801s 2024-04-26T00:25:59.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff521a07640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff51c137300 0x7ff51c137ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:59.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff522208640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff51c136f20 0x7ff51c134f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:59.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff522208640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff51c136f20 0x7ff51c134f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51906/0 (socket says 172.21.15.37:51906) 2024-04-26T00:25:59.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff522208640 1 -- 172.21.15.37:0/1890677941 learned_addr learned my addr 172.21.15.37:0/1890677941 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:59.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff522208640 10 monclient(hunting): get_auth_request con 0x7ff51c136f20 auth_method 0 2024-04-26T00:25:59.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff522208640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:59.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff522208640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:59.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff522208640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:59.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff521a07640 10 monclient(hunting): get_auth_request con 0x7ff51c137300 auth_method 0 2024-04-26T00:25:59.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff521a07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:59.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff521a07640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:59.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.256+0000 7ff521a07640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:59.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff522208640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:59.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff522208640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:59.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff522208640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:59.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff521a07640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:59.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff521a07640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:59.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff521a07640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:59.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff522a09640 1 --2- 172.21.15.37:0/1890677941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff51c13c450 0x7ff51c138430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:59.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff522a09640 10 monclient(hunting): get_auth_request con 0x7ff51c13c450 auth_method 0 2024-04-26T00:25:59.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff522a09640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:59.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff522a09640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:59.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff522a09640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:59.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff522208640 10 monclient(hunting): handle_auth_done global_id 74129 payload 824 2024-04-26T00:25:59.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff522208640 1 -- 172.21.15.37:0/1890677941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff51c13c450 msgr2=0x7ff51c138430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:59.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff522208640 1 --2- 172.21.15.37:0/1890677941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff51c13c450 0x7ff51c138430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:59.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff522208640 1 -- 172.21.15.37:0/1890677941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff51c137300 msgr2=0x7ff51c137ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:59.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff522208640 1 --2- 172.21.15.37:0/1890677941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff51c137300 0x7ff51c137ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:59.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff522208640 10 monclient: _finish_hunting 0 2024-04-26T00:25:59.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff522208640 1 monclient: found mon.noname-c 2024-04-26T00:25:59.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff522208640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:59.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff522208640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:25:59.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff522208640 1 -- 172.21.15.37:0/1890677941 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff51c13a610 con 0x7ff51c136f20 2024-04-26T00:25:59.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff521a07640 1 --2- 172.21.15.37:0/1890677941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff51c137300 0x7ff51c137ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:25:59.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff522208640 1 -- 172.21.15.37:0/1890677941 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff51c136f20 msgr2=0x7ff51c134f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:59.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff522208640 1 --2- 172.21.15.37:0/1890677941 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff51c136f20 0x7ff51c134f40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff51801cd10 tx=0x7ff518019210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:59.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff512ffd640 1 -- 172.21.15.37:0/1890677941 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5180b0050 con 0x7ff51c136f20 2024-04-26T00:25:59.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff512ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:59.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff512ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:25:59.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff512ffd640 10 monclient: dump: 2024-04-26T00:25:59.255 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:59.255 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:59.255 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.255 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.255 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:59.255 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:59.255 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:59.256 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:59.256 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:59.256 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:59.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff512ffd640 10 monclient: _finish_auth 0 2024-04-26T00:25:59.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff512ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:25:59.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff512ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:59.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff512ffd640 1 -- 172.21.15.37:0/1890677941 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5180ba070 con 0x7ff51c136f20 2024-04-26T00:25:59.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff52446c640 5 monclient: authenticate success, global_id 74129 2024-04-26T00:25:59.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff512ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:59.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff52446c640 10 monclient: _renew_subs 2024-04-26T00:25:59.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff512ffd640 1 -- 172.21.15.37:0/1890677941 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5180b47c0 con 0x7ff51c136f20 2024-04-26T00:25:59.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff52446c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:25:59.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff52446c640 1 -- 172.21.15.37:0/1890677941 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff51c1360a0 con 0x7ff51c136f20 2024-04-26T00:25:59.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff512ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:59.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff520a05640 4 set_mon_vals no callback set 2024-04-26T00:25:59.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff512ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:25:59.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff512ffd640 10 monclient: dump: 2024-04-26T00:25:59.257 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:59.257 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:59.257 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.257 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.257 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:59.257 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:59.257 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:59.257 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:59.257 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:59.257 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:59.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff52446c640 10 monclient: _renew_subs 2024-04-26T00:25:59.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff52446c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:25:59.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff52446c640 1 -- 172.21.15.37:0/1890677941 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff51c135480 con 0x7ff51c136f20 2024-04-26T00:25:59.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff52446c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:59.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff52446c640 1 --2- 172.21.15.37:0/1890677941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff51c139130 0x7ff51c02c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:59.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff52446c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:59.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff52446c640 1 -- 172.21.15.37:0/1890677941 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff51c13a440 con 0x7ff51c139130 2024-04-26T00:25:59.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff512ffd640 1 -- 172.21.15.37:0/1890677941 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff5180b49a0 con 0x7ff51c136f20 2024-04-26T00:25:59.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff521a07640 1 --2- 172.21.15.37:0/1890677941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff51c139130 0x7ff51c02c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:59.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff521a07640 10 monclient: get_auth_request con 0x7ff51c139130 auth_method 0 2024-04-26T00:25:59.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff521a07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:59.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff521a07640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:59.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff521a07640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:59.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff521a07640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:59.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff521a07640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:59.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.260+0000 7ff521a07640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:59.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.264+0000 7ff512ffd640 1 --2- 172.21.15.37:0/1890677941 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff500035ed0 0x7ff500038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:59.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.264+0000 7ff521a07640 10 monclient(hunting): handle_auth_done global_id 54309 payload 824 2024-04-26T00:25:59.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.264+0000 7ff521a07640 1 -- 172.21.15.37:0/1890677941 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff51c09ae30 con 0x7ff51c139130 2024-04-26T00:25:59.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.264+0000 7ff512ffd640 1 -- 172.21.15.37:0/1890677941 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(236..236 src has 177..236) ==== 3947+0+0 (secure 0 0 0) 0x7ff5180e29d0 con 0x7ff51c136f20 2024-04-26T00:25:59.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.264+0000 7ff522a09640 1 --2- 172.21.15.37:0/1890677941 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff500035ed0 0x7ff500038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:59.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.264+0000 7ff522a09640 10 monclient: get_auth_request con 0x7ff500035ed0 auth_method 0 2024-04-26T00:25:59.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.264+0000 7ff521a07640 1 -- 172.21.15.37:0/1890677941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff51c139130 msgr2=0x7ff51c02c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:59.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.264+0000 7ff521a07640 1 --2- 172.21.15.37:0/1890677941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff51c139130 0x7ff51c02c000 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff50c007fd0 tx=0x7ff50c00ff70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:59.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.264+0000 7ff512ffd640 1 -- 172.21.15.37:0/1890677941 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff50c018020 con 0x7ff51c139130 2024-04-26T00:25:59.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.264+0000 7ff512ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:59.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.264+0000 7ff512ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:59.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.264+0000 7ff512ffd640 10 monclient: dump: 2024-04-26T00:25:59.259 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:59.259 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:59.259 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.259 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.259 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:59.260 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:59.260 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:59.260 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:59.260 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:59.260 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:59.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.264+0000 7ff512ffd640 1 -- 172.21.15.37:0/1890677941 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff50c00dd60 con 0x7ff51c139130 2024-04-26T00:25:59.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.264+0000 7ff512ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:59.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.264+0000 7ff512ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:25:59.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.264+0000 7ff512ffd640 1 -- 172.21.15.37:0/1890677941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff51c139130 msgr2=0x7ff51c02c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:59.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.264+0000 7ff512ffd640 1 --2- 172.21.15.37:0/1890677941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff51c139130 0x7ff51c02c000 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff50c007fd0 tx=0x7ff50c00ff70 comp rx=0 tx=0).stop 2024-04-26T00:25:59.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.264+0000 7ff512ffd640 1 -- 172.21.15.37:0/1890677941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff51c139130 msgr2=0x7ff51c02c000 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:25:59.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.264+0000 7ff512ffd640 1 --2- 172.21.15.37:0/1890677941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff51c139130 0x7ff51c02c000 secure :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff50c007fd0 tx=0x7ff50c00ff70 comp rx=0 tx=0).stop 2024-04-26T00:25:59.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.264+0000 7ff522a09640 1 --2- 172.21.15.37:0/1890677941 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff500035ed0 0x7ff500038390 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff514007520 tx=0x7ff51400b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:59.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.272+0000 7ff52446c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:25:59.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.272+0000 7ff52446c640 1 --2- 172.21.15.37:0/1890677941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff51c1ccc60 0x7ff51c02c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:59.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.272+0000 7ff52446c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:59.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.272+0000 7ff52446c640 1 -- 172.21.15.37:0/1890677941 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff51c04f510 con 0x7ff51c1ccc60 2024-04-26T00:25:59.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.272+0000 7ff521a07640 1 --2- 172.21.15.37:0/1890677941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff51c1ccc60 0x7ff51c02c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:59.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.272+0000 7ff521a07640 10 monclient: get_auth_request con 0x7ff51c1ccc60 auth_method 0 2024-04-26T00:25:59.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.272+0000 7ff521a07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:59.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.272+0000 7ff521a07640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:59.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.272+0000 7ff521a07640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:59.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.272+0000 7ff521a07640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:59.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.272+0000 7ff521a07640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:59.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.272+0000 7ff521a07640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:59.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.272+0000 7ff521a07640 10 monclient(hunting): handle_auth_done global_id 54312 payload 824 2024-04-26T00:25:59.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.272+0000 7ff521a07640 1 -- 172.21.15.37:0/1890677941 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff51c09fa40 con 0x7ff51c1ccc60 2024-04-26T00:25:59.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.272+0000 7ff521a07640 1 -- 172.21.15.37:0/1890677941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff51c1ccc60 msgr2=0x7ff51c02c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:59.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.272+0000 7ff521a07640 1 --2- 172.21.15.37:0/1890677941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff51c1ccc60 0x7ff51c02c540 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff50c007fd0 tx=0x7ff50c003700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:59.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.272+0000 7ff512ffd640 1 -- 172.21.15.37:0/1890677941 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff50c018030 con 0x7ff51c1ccc60 2024-04-26T00:25:59.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.276+0000 7ff512ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:59.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.276+0000 7ff512ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:59.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.276+0000 7ff512ffd640 10 monclient: dump: 2024-04-26T00:25:59.263 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:59.263 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:59.263 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.263 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.264 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:59.264 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:59.264 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:59.264 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:59.264 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:59.264 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:59.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.276+0000 7ff512ffd640 1 -- 172.21.15.37:0/1890677941 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7ff50c014b80 con 0x7ff51c1ccc60 2024-04-26T00:25:59.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.276+0000 7ff512ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:25:59.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.276+0000 7ff512ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:25:59.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.276+0000 7ff512ffd640 1 -- 172.21.15.37:0/1890677941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff51c1ccc60 msgr2=0x7ff51c02c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:59.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.276+0000 7ff512ffd640 1 --2- 172.21.15.37:0/1890677941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff51c1ccc60 0x7ff51c02c540 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff50c007fd0 tx=0x7ff50c003700 comp rx=0 tx=0).stop 2024-04-26T00:25:59.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.276+0000 7ff512ffd640 1 -- 172.21.15.37:0/1890677941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff51c1ccc60 msgr2=0x7ff51c02c540 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:25:59.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.276+0000 7ff512ffd640 1 --2- 172.21.15.37:0/1890677941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff51c1ccc60 0x7ff51c02c540 secure :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff50c007fd0 tx=0x7ff50c003700 comp rx=0 tx=0).stop 2024-04-26T00:25:59.265 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:25:59.265 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:25:59.265 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:25:59.265 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:25:59.265 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 46, 2024-04-26T00:25:59.265 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 122602, 2024-04-26T00:25:59.265 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:25:59.265 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:25:59.265 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:25:59.265 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:25:59.265 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:59.265 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 6, 2024-04-26T00:25:59.266 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:25:59.266 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:25:59.266 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:25:59.266 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:59.266 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:59.266 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:59.266 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:59.266 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:59.266 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:59.266 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:59.266 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:59.266 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:59.266 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:59.266 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:59.266 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:59.267 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:25:59.267 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:25:59.267 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:59.267 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:59.267 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:59.267 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:59.267 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:59.267 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:59.267 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:59.267 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:59.267 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:59.267 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:59.267 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:59.267 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:59.267 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:59.268 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:25:59.268 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:25:59.268 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:25:59.268 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:25:59.268 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:25:59.268 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:25:59.268 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:25:59.268 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:25:59.268 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:25:59.268 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:25:59.268 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:25:59.268 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:25:59.268 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:25:59.268 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:25:59.269 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:25:59.269 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:25:59.269 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:25:59.269 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:59.269 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:59.269 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:59.269 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:59.269 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:59.269 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:59.269 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:59.269 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:59.269 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:59.269 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:59.269 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:59.270 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:59.270 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:25:59.270 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:59.270 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:25:59.270 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:59.270 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:25:59.270 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:25:59.270 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:59.270 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:59.270 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:59.270 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:59.270 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:25:59.270 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:59.270 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:59.270 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:59.271 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:59.271 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:25:59.271 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:59.271 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:59.271 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:59.271 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:59.271 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:25:59.271 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:25:59.271 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:59.271 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:59.271 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:59.271 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:59.271 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:59.271 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:25:59.271 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:25:59.272 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:59.272 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:59.272 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:59.272 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:59.272 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:25:59.272 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:59.272 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:59.272 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:59.272 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:59.272 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:25:59.272 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:59.272 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:59.272 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:59.272 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:59.272 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:25:59.273 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:25:59.273 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:59.273 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:59.273 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:59.273 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:59.273 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:59.273 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:25:59.273 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:25:59.273 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:59.273 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:59.273 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:59.273 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:59.273 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:25:59.273 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:59.273 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:59.274 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:59.274 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:59.274 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:25:59.274 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:59.274 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:59.274 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:59.274 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:59.274 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:25:59.274 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:25:59.274 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:59.274 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:59.274 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:59.274 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:59.274 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:59.275 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:59.275 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:25:59.275 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:25:59.275 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:59.275 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:59.275 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:59.275 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:25:59.275 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:59.275 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:59.275 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:25:59.275 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:59.275 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:59.275 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:59.275 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:25:59.275 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:59.276 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:59.276 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:25:59.276 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:59.276 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:59.276 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:59.276 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:25:59.276 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:59.276 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:59.276 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:25:59.276 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:59.276 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:59.276 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:59.276 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:25:59.276 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:59.276 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:59.277 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:59.277 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:25:59.277 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:25:59.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.276+0000 7ff52446c640 1 -- 172.21.15.37:0/1890677941 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff500035ed0 msgr2=0x7ff500038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:59.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.276+0000 7ff52446c640 1 --2- 172.21.15.37:0/1890677941 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff500035ed0 0x7ff500038390 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff514007520 tx=0x7ff51400b040 comp rx=0 tx=0).stop 2024-04-26T00:25:59.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.276+0000 7ff52446c640 10 monclient: shutdown 2024-04-26T00:25:59.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.276+0000 7ff52446c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:59.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.276+0000 7ff52446c640 1 -- 172.21.15.37:0/1890677941 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff51c136f20 msgr2=0x7ff51c134f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:59.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.276+0000 7ff52446c640 1 --2- 172.21.15.37:0/1890677941 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff51c136f20 0x7ff51c134f40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff51801cd10 tx=0x7ff518019210 comp rx=0 tx=0).stop 2024-04-26T00:25:59.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.276+0000 7ff522a09640 1 -- 172.21.15.37:0/1890677941 reap_dead start 2024-04-26T00:25:59.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.276+0000 7ff522a09640 1 -- 172.21.15.37:0/1890677941 reap_dead start 2024-04-26T00:25:59.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.276+0000 7ff52446c640 1 -- 172.21.15.37:0/1890677941 shutdown_connections 2024-04-26T00:25:59.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.276+0000 7ff52446c640 1 -- 172.21.15.37:0/1890677941 >> 172.21.15.37:0/1890677941 conn(0x7ff51c0a61b0 msgr2=0x7ff51c0a8230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:59.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.276+0000 7ff52446c640 1 -- 172.21.15.37:0/1890677941 shutdown_connections 2024-04-26T00:25:59.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.276+0000 7ff52446c640 1 -- 172.21.15.37:0/1890677941 wait complete. 2024-04-26T00:25:59.279 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:25:59.279 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:25:59.279 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:25:59.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19be138640 10 monclient: get_monmap_and_config 2024-04-26T00:25:59.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19be138640 10 monclient: build_initial_monmap 2024-04-26T00:25:59.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19be138640 10 monclient: monmap: 2024-04-26T00:25:59.395 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:59.395 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:59.395 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:25:59.416279+0000 2024-04-26T00:25:59.395 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:25:59.416279+0000 2024-04-26T00:25:59.395 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:59.395 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:59.395 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:25:59.395 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:25:59.395 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:25:59.396 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:25:59.396 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:25:59.396 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19be138640 1 Processor -- start 2024-04-26T00:25:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19be138640 1 -- start start 2024-04-26T00:25:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19be138640 10 monclient: init 2024-04-26T00:25:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19be138640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19be138640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-04-26T00:25:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19be138640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f19b8130eb0 0x7f19b809ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19be138640 10 monclient(hunting): picked mon.noname-a con 0x7f19b8130eb0 addr v2:172.21.15.37:3300/0 2024-04-26T00:25:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19be138640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19b8131290 0x7f19b8133b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19be138640 10 monclient(hunting): picked mon.noname-b con 0x7f19b8131290 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19be138640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f19b81340d0 0x7f19b8134580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:59.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19be138640 10 monclient(hunting): picked mon.noname-c con 0x7f19b81340d0 addr v2:172.21.15.45:3300/0 2024-04-26T00:25:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19be138640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19be138640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f19b809df60 con 0x7f19b8130eb0 2024-04-26T00:25:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19be138640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19be138640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f19b809c190 con 0x7f19b81340d0 2024-04-26T00:25:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19be138640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19be138640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f19b80a11c0 con 0x7f19b8131290 2024-04-26T00:25:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19be138640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19be138640 10 monclient(hunting): authenticate will time out at 1340.721914s 2024-04-26T00:25:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19b6ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19b8131290 0x7f19b8133b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19b6ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19b8131290 0x7f19b8133b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51910/0 (socket says 172.21.15.37:51910) 2024-04-26T00:25:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19b6ffd640 1 -- 172.21.15.37:0/1861114393 learned_addr learned my addr 172.21.15.37:0/1861114393 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19b6ffd640 10 monclient(hunting): get_auth_request con 0x7f19b8131290 auth_method 0 2024-04-26T00:25:59.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19b6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19b77fe640 1 --2- 172.21.15.37:0/1861114393 >> v2:172.21.15.37:3300/0 conn(0x7f19b8130eb0 0x7f19b809ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19b77fe640 10 monclient(hunting): get_auth_request con 0x7f19b8130eb0 auth_method 0 2024-04-26T00:25:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19b77fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:59.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19b7fff640 1 --2- 172.21.15.37:0/1861114393 >> v2:172.21.15.45:3300/0 conn(0x7f19b81340d0 0x7f19b8134580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:59.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19b6ffd640 10 monclient(hunting): handle_auth_done global_id 74132 payload 274 2024-04-26T00:25:59.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19b6ffd640 1 -- 172.21.15.37:0/1861114393 >> v2:172.21.15.45:3300/0 conn(0x7f19b81340d0 msgr2=0x7f19b8134580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:59.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19b6ffd640 1 --2- 172.21.15.37:0/1861114393 >> v2:172.21.15.45:3300/0 conn(0x7f19b81340d0 0x7f19b8134580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:59.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19b6ffd640 1 -- 172.21.15.37:0/1861114393 >> v2:172.21.15.37:3300/0 conn(0x7f19b8130eb0 msgr2=0x7f19b809ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:59.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19b6ffd640 1 --2- 172.21.15.37:0/1861114393 >> v2:172.21.15.37:3300/0 conn(0x7f19b8130eb0 0x7f19b809ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:59.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19b6ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:25:59.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19b6ffd640 1 monclient: found mon.noname-b 2024-04-26T00:25:59.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:59.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19b6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:25:59.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19b6ffd640 1 -- 172.21.15.37:0/1861114393 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19b809dc80 con 0x7f19b8131290 2024-04-26T00:25:59.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19b7fff640 10 monclient: get_auth_request con 0x7f19b81340d0 auth_method 0 2024-04-26T00:25:59.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.412+0000 7f19b7fff640 1 --2- 172.21.15.37:0/1861114393 >> v2:172.21.15.45:3300/0 conn(0x7f19b81340d0 0x7f19b8134580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:25:59.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.416+0000 7f19b6ffd640 1 -- 172.21.15.37:0/1861114393 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19b8131290 msgr2=0x7f19b8133b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:59.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.416+0000 7f19b6ffd640 1 --2- 172.21.15.37:0/1861114393 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19b8131290 0x7f19b8133b90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f19b806d2b0 tx=0x7f19a801eab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73e9c8d8235c5594 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:59.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.416+0000 7f19b67fc640 1 -- 172.21.15.37:0/1861114393 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a80b4070 con 0x7f19b8131290 2024-04-26T00:25:59.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.416+0000 7f19b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:59.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.416+0000 7f19b67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:25:59.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.416+0000 7f19b67fc640 10 monclient: dump: 2024-04-26T00:25:59.405 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:59.405 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:59.405 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.405 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.406 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:59.406 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:59.406 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:59.406 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:59.406 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:59.406 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:59.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.416+0000 7f19b67fc640 10 monclient: _finish_auth 0 2024-04-26T00:25:59.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.416+0000 7f19b67fc640 10 monclient: _check_auth_tickets 2024-04-26T00:25:59.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.416+0000 7f19b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:59.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.416+0000 7f19b67fc640 1 -- 172.21.15.37:0/1861114393 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19a80a7950 con 0x7f19b8131290 2024-04-26T00:25:59.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.416+0000 7f19b67fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:59.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.416+0000 7f19b67fc640 1 -- 172.21.15.37:0/1861114393 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a80a7d20 con 0x7f19b8131290 2024-04-26T00:25:59.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.416+0000 7f19be138640 5 monclient: authenticate success, global_id 74132 2024-04-26T00:25:59.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.416+0000 7f19be138640 10 monclient: get_monmap_and_config success 2024-04-26T00:25:59.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.416+0000 7f19be138640 4 set_mon_vals no callback set 2024-04-26T00:25:59.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.416+0000 7f19b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:59.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.416+0000 7f19b67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:25:59.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.416+0000 7f19b67fc640 10 monclient: dump: 2024-04-26T00:25:59.407 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:59.407 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:59.407 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.407 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.407 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:59.407 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:59.407 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:59.408 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:59.408 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:59.408 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:59.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.416+0000 7f19be138640 10 monclient: shutdown 2024-04-26T00:25:59.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.416+0000 7f19be138640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:59.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.416+0000 7f19be138640 1 -- 172.21.15.37:0/1861114393 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19b8131290 msgr2=0x7f19b8133b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:59.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.416+0000 7f19be138640 1 --2- 172.21.15.37:0/1861114393 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19b8131290 0x7f19b8133b90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f19b806d2b0 tx=0x7f19a801eab0 comp rx=0 tx=0).stop 2024-04-26T00:25:59.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.416+0000 7f19be138640 1 -- 172.21.15.37:0/1861114393 shutdown_connections 2024-04-26T00:25:59.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.416+0000 7f19be138640 1 --2- 172.21.15.37:0/1861114393 >> v2:172.21.15.45:3300/0 conn(0x7f19b81340d0 0x7f19b8134580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:59.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.416+0000 7f19be138640 1 --2- 172.21.15.37:0/1861114393 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19b8131290 0x7f19b8133b90 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:59.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.416+0000 7f19be138640 1 --2- 172.21.15.37:0/1861114393 >> v2:172.21.15.37:3300/0 conn(0x7f19b8130eb0 0x7f19b809ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:59.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.416+0000 7f19be138640 1 -- 172.21.15.37:0/1861114393 >> 172.21.15.37:0/1861114393 conn(0x7f19b80a1d50 msgr2=0x7f19b80a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:59.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.416+0000 7f19be138640 1 -- 172.21.15.37:0/1861114393 shutdown_connections 2024-04-26T00:25:59.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.416+0000 7f19be138640 1 -- 172.21.15.37:0/1861114393 wait complete. 2024-04-26T00:25:59.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19be138640 10 monclient: build_initial_monmap 2024-04-26T00:25:59.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19be138640 10 monclient: monmap: 2024-04-26T00:25:59.409 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:59.409 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:25:59.409 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:25:59.409 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:25:59.409 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:59.409 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:25:59.409 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:25:59.409 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:25:59.409 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:25:59.409 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:59.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19be138640 1 Processor -- start 2024-04-26T00:25:59.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19be138640 1 -- start start 2024-04-26T00:25:59.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19be138640 10 monclient: init 2024-04-26T00:25:59.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19be138640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:59.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19be138640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:25:59.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19be138640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19b8130eb0 0x7f19b81cace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:59.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19be138640 10 monclient(hunting): picked mon.noname-a con 0x7f19b8130eb0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:25:59.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19be138640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19b8131290 0x7f19b8139de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:59.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19be138640 10 monclient(hunting): picked mon.noname-b con 0x7f19b8131290 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:25:59.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19be138640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19b81340d0 0x7f19b813a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:59.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19be138640 10 monclient(hunting): picked mon.noname-c con 0x7f19b81340d0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:59.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19be138640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:59.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19be138640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f19b80a0460 con 0x7f19b8130eb0 2024-04-26T00:25:59.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19be138640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:59.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19be138640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f19b8131880 con 0x7f19b8131290 2024-04-26T00:25:59.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19be138640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:59.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19be138640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f19b80a05e0 con 0x7f19b81340d0 2024-04-26T00:25:59.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19be138640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:59.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19be138640 10 monclient(hunting): authenticate will time out at 1340.729188s 2024-04-26T00:25:59.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19b77fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19b8130eb0 0x7f19b81cace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:59.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19b77fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19b8130eb0 0x7f19b81cace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36734/0 (socket says 172.21.15.37:36734) 2024-04-26T00:25:59.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19b77fe640 1 -- 172.21.15.37:0/2690625535 learned_addr learned my addr 172.21.15.37:0/2690625535 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:59.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19b77fe640 10 monclient(hunting): get_auth_request con 0x7f19b8130eb0 auth_method 0 2024-04-26T00:25:59.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:59.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19b77fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:59.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:59.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19b7fff640 1 --2- 172.21.15.37:0/2690625535 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19b81340d0 0x7f19b813a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:59.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:59.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:59.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:59.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19b7fff640 10 monclient(hunting): get_auth_request con 0x7f19b81340d0 auth_method 0 2024-04-26T00:25:59.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:59.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19b7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:59.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:59.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19b6ffd640 1 --2- 172.21.15.37:0/2690625535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19b8131290 0x7f19b8139de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:59.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19b6ffd640 10 monclient(hunting): get_auth_request con 0x7f19b8131290 auth_method 0 2024-04-26T00:25:59.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:59.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19b6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:59.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:59.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:59.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:59.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:59.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19b77fe640 10 monclient(hunting): handle_auth_done global_id 54318 payload 824 2024-04-26T00:25:59.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19b77fe640 1 -- 172.21.15.37:0/2690625535 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19b81340d0 msgr2=0x7f19b813a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:59.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19b77fe640 1 --2- 172.21.15.37:0/2690625535 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19b81340d0 0x7f19b813a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:59.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19b77fe640 1 -- 172.21.15.37:0/2690625535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19b8131290 msgr2=0x7f19b8139de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:59.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19b77fe640 1 --2- 172.21.15.37:0/2690625535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19b8131290 0x7f19b8139de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:59.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19b77fe640 10 monclient: _finish_hunting 0 2024-04-26T00:25:59.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19b77fe640 1 monclient: found mon.noname-a 2024-04-26T00:25:59.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:59.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19b77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:25:59.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19b77fe640 1 -- 172.21.15.37:0/2690625535 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19b80a1470 con 0x7f19b8130eb0 2024-04-26T00:25:59.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.420+0000 7f19b6ffd640 1 --2- 172.21.15.37:0/2690625535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19b8131290 0x7f19b8139de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:25:59.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19b77fe640 1 -- 172.21.15.37:0/2690625535 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19b8130eb0 msgr2=0x7f19b81cace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:59.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19b77fe640 1 --2- 172.21.15.37:0/2690625535 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19b8130eb0 0x7f19b81cace0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f19a000e960 tx=0x7f19a000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:59.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19affff640 1 -- 172.21.15.37:0/2690625535 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a0017070 con 0x7f19b8130eb0 2024-04-26T00:25:59.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19affff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:59.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19affff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:25:59.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19affff640 10 monclient: dump: 2024-04-26T00:25:59.416 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:59.416 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:59.416 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.417 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.417 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:59.417 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:59.417 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:59.417 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:59.417 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:59.417 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:59.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19affff640 10 monclient: _finish_auth 0 2024-04-26T00:25:59.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19affff640 10 monclient: _check_auth_tickets 2024-04-26T00:25:59.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:59.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19affff640 1 -- 172.21.15.37:0/2690625535 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19a000cc10 con 0x7f19b8130eb0 2024-04-26T00:25:59.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19affff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:59.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19affff640 1 -- 172.21.15.37:0/2690625535 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a0013470 con 0x7f19b8130eb0 2024-04-26T00:25:59.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19affff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:59.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19affff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:59.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19affff640 10 monclient: dump: 2024-04-26T00:25:59.418 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:59.418 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:59.418 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.418 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.418 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:59.418 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:59.418 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:59.418 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:59.418 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:59.418 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:59.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19be138640 5 monclient: authenticate success, global_id 54318 2024-04-26T00:25:59.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19be138640 10 monclient: _renew_subs 2024-04-26T00:25:59.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19be138640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:59.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19b5ffb640 4 set_mon_vals no callback set 2024-04-26T00:25:59.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19be138640 1 -- 172.21.15.37:0/2690625535 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19b8133160 con 0x7f19b8130eb0 2024-04-26T00:25:59.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19be138640 10 monclient: _renew_subs 2024-04-26T00:25:59.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19be138640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:59.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19be138640 1 -- 172.21.15.37:0/2690625535 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19b8136660 con 0x7f19b8130eb0 2024-04-26T00:25:59.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19be138640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:59.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19be138640 1 --2- 172.21.15.37:0/2690625535 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19b80711e0 0x7f19b802c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:59.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19be138640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:59.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19be138640 1 -- 172.21.15.37:0/2690625535 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f19b804f510 con 0x7f19b80711e0 2024-04-26T00:25:59.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19b6ffd640 1 --2- 172.21.15.37:0/2690625535 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19b80711e0 0x7f19b802c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:59.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19b6ffd640 10 monclient: get_auth_request con 0x7f19b80711e0 auth_method 0 2024-04-26T00:25:59.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:59.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19b6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:59.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:59.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19affff640 1 -- 172.21.15.37:0/2690625535 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f19a001a070 con 0x7f19b8130eb0 2024-04-26T00:25:59.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:59.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:59.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:59.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19b6ffd640 10 monclient(hunting): handle_auth_done global_id 74138 payload 824 2024-04-26T00:25:59.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19b6ffd640 1 -- 172.21.15.37:0/2690625535 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f19b80a0f60 con 0x7f19b80711e0 2024-04-26T00:25:59.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19affff640 1 --2- 172.21.15.37:0/2690625535 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1998035eb0 0x7f1998038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:59.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19affff640 1 -- 172.21.15.37:0/2690625535 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(236..236 src has 177..236) ==== 3947+0+0 (secure 0 0 0) 0x7f19a0048b40 con 0x7f19b8130eb0 2024-04-26T00:25:59.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19b7fff640 1 --2- 172.21.15.37:0/2690625535 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1998035eb0 0x7f1998038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:59.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19b7fff640 10 monclient: get_auth_request con 0x7f1998035eb0 auth_method 0 2024-04-26T00:25:59.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19b6ffd640 1 -- 172.21.15.37:0/2690625535 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19b80711e0 msgr2=0x7f19b802c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:59.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.424+0000 7f19b6ffd640 1 --2- 172.21.15.37:0/2690625535 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19b80711e0 0x7f19b802c000 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f19a80b1000 tx=0x7f19a801b850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:59.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.428+0000 7f19affff640 1 -- 172.21.15.37:0/2690625535 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a80be020 con 0x7f19b80711e0 2024-04-26T00:25:59.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.428+0000 7f19affff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:59.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.428+0000 7f19affff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:25:59.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.428+0000 7f19affff640 10 monclient: dump: 2024-04-26T00:25:59.421 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:59.421 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:59.421 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.421 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.421 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:59.421 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:59.421 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:59.421 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:59.421 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:59.421 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:59.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.428+0000 7f19affff640 1 -- 172.21.15.37:0/2690625535 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f19a80b9660 con 0x7f19b80711e0 2024-04-26T00:25:59.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.428+0000 7f19affff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:59.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.428+0000 7f19affff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:25:59.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.428+0000 7f19affff640 1 -- 172.21.15.37:0/2690625535 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19b80711e0 msgr2=0x7f19b802c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:59.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.428+0000 7f19affff640 1 --2- 172.21.15.37:0/2690625535 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19b80711e0 0x7f19b802c000 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f19a80b1000 tx=0x7f19a801b850 comp rx=0 tx=0).stop 2024-04-26T00:25:59.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.428+0000 7f19affff640 1 -- 172.21.15.37:0/2690625535 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19b80711e0 msgr2=0x7f19b802c000 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:25:59.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.428+0000 7f19affff640 1 --2- 172.21.15.37:0/2690625535 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19b80711e0 0x7f19b802c000 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f19a80b1000 tx=0x7f19a801b850 comp rx=0 tx=0).stop 2024-04-26T00:25:59.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.428+0000 7f19b7fff640 1 --2- 172.21.15.37:0/2690625535 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1998035eb0 0x7f1998038370 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f19b8136430 tx=0x7f19b0007780 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:59.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.432+0000 7f19be138640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:25:59.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.432+0000 7f19be138640 1 --2- 172.21.15.37:0/2690625535 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19b80715c0 0x7f19b802c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:59.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.432+0000 7f19be138640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:59.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.432+0000 7f19be138640 1 -- 172.21.15.37:0/2690625535 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f19b8072670 con 0x7f19b80715c0 2024-04-26T00:25:59.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.432+0000 7f19b6ffd640 1 --2- 172.21.15.37:0/2690625535 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19b80715c0 0x7f19b802c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:59.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.432+0000 7f19b6ffd640 10 monclient: get_auth_request con 0x7f19b80715c0 auth_method 0 2024-04-26T00:25:59.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.432+0000 7f19b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:59.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.432+0000 7f19b6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:59.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.432+0000 7f19b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:59.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.432+0000 7f19b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:59.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.432+0000 7f19b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:59.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.432+0000 7f19b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:59.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.432+0000 7f19b6ffd640 10 monclient(hunting): handle_auth_done global_id 74141 payload 824 2024-04-26T00:25:59.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.432+0000 7f19b6ffd640 1 -- 172.21.15.37:0/2690625535 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f19b812e2d0 con 0x7f19b80715c0 2024-04-26T00:25:59.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.436+0000 7f19b6ffd640 1 -- 172.21.15.37:0/2690625535 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19b80715c0 msgr2=0x7f19b802c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:59.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.436+0000 7f19b6ffd640 1 --2- 172.21.15.37:0/2690625535 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19b80715c0 0x7f19b802c540 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f19a801d160 tx=0x7f19a80a9000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:59.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.436+0000 7f19affff640 1 -- 172.21.15.37:0/2690625535 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a80be030 con 0x7f19b80715c0 2024-04-26T00:25:59.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.436+0000 7f19affff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:59.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.436+0000 7f19affff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:25:59.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.436+0000 7f19affff640 10 monclient: dump: 2024-04-26T00:25:59.425 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:59.425 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:59.425 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.425 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.425 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:59.425 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:59.426 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:59.426 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:59.426 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:59.426 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:59.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.436+0000 7f19affff640 1 -- 172.21.15.37:0/2690625535 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7f19a80b9e30 con 0x7f19b80715c0 2024-04-26T00:25:59.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.436+0000 7f19affff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:25:59.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.436+0000 7f19affff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:25:59.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.436+0000 7f19affff640 1 -- 172.21.15.37:0/2690625535 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19b80715c0 msgr2=0x7f19b802c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:59.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.436+0000 7f19affff640 1 --2- 172.21.15.37:0/2690625535 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19b80715c0 0x7f19b802c540 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f19a801d160 tx=0x7f19a80a9000 comp rx=0 tx=0).stop 2024-04-26T00:25:59.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.436+0000 7f19affff640 1 -- 172.21.15.37:0/2690625535 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19b80715c0 msgr2=0x7f19b802c540 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:25:59.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.436+0000 7f19affff640 1 --2- 172.21.15.37:0/2690625535 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19b80715c0 0x7f19b802c540 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f19a801d160 tx=0x7f19a80a9000 comp rx=0 tx=0).stop 2024-04-26T00:25:59.427 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:25:59.427 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:25:59.427 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:25:59.427 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:25:59.427 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 46, 2024-04-26T00:25:59.427 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 7747, 2024-04-26T00:25:59.427 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:25:59.427 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:25:59.427 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:25:59.427 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:25:59.427 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:59.427 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 6, 2024-04-26T00:25:59.428 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:25:59.428 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:25:59.428 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:25:59.428 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:59.428 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:59.428 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:59.428 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:59.428 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:59.428 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:59.428 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:59.428 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:59.428 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:59.428 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:59.428 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:59.429 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:59.429 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:25:59.429 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:25:59.429 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:59.429 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:59.429 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:59.429 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:59.429 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:59.429 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:59.429 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:59.429 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:59.429 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:59.429 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:59.430 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:59.430 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:59.430 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:59.430 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:25:59.430 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:25:59.430 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:25:59.430 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:25:59.430 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:25:59.430 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:25:59.430 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:25:59.430 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:25:59.430 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:25:59.430 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:25:59.431 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:25:59.431 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:25:59.431 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:25:59.431 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:25:59.431 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:25:59.431 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:25:59.431 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:25:59.431 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:59.431 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:59.431 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:59.431 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:59.431 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:59.431 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:59.431 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:59.432 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:59.432 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:59.432 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:59.432 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:59.432 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:59.432 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:25:59.432 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:59.432 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:25:59.432 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:59.432 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:25:59.432 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:25:59.432 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:59.432 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:59.432 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:59.433 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:59.433 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:25:59.433 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:59.433 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:59.433 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:59.433 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:59.433 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:25:59.433 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:59.433 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:59.433 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:59.433 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:59.433 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:25:59.433 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:25:59.433 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:59.434 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:59.434 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:59.434 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:59.434 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:59.434 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:25:59.434 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:25:59.434 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:59.434 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:59.434 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:59.434 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:59.434 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:25:59.434 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:59.434 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:59.434 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:59.435 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:59.435 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:25:59.435 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:59.435 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:59.435 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:59.435 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:59.435 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:25:59.435 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:25:59.435 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:59.435 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:59.435 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:59.435 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:59.435 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:59.435 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:25:59.435 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:25:59.436 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:59.436 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:59.436 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:59.436 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:59.436 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:25:59.436 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:59.436 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:59.436 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:59.436 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:59.436 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:25:59.436 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:59.436 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:59.436 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:59.436 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:59.436 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:25:59.437 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:25:59.437 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:59.437 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:59.437 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:59.437 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:59.437 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:59.437 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:59.437 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:25:59.437 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:25:59.437 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:59.437 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:59.437 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:59.437 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:25:59.438 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:59.438 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:59.438 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:25:59.438 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:59.438 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:59.438 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:59.438 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:25:59.438 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:59.438 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:59.438 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:59.438 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:25:59.438 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:25:59.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.436+0000 7f19be138640 1 -- 172.21.15.37:0/2690625535 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1998035eb0 msgr2=0x7f1998038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:59.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.436+0000 7f19be138640 1 --2- 172.21.15.37:0/2690625535 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1998035eb0 0x7f1998038370 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f19b8136430 tx=0x7f19b0007780 comp rx=0 tx=0).stop 2024-04-26T00:25:59.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.436+0000 7f19be138640 10 monclient: shutdown 2024-04-26T00:25:59.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.436+0000 7f19be138640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:59.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.436+0000 7f19be138640 1 -- 172.21.15.37:0/2690625535 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19b8130eb0 msgr2=0x7f19b81cace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:59.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.436+0000 7f19be138640 1 --2- 172.21.15.37:0/2690625535 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19b8130eb0 0x7f19b81cace0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f19a000e960 tx=0x7f19a000ee20 comp rx=0 tx=0).stop 2024-04-26T00:25:59.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.436+0000 7f19b7fff640 1 -- 172.21.15.37:0/2690625535 reap_dead start 2024-04-26T00:25:59.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.436+0000 7f19b7fff640 1 -- 172.21.15.37:0/2690625535 reap_dead start 2024-04-26T00:25:59.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.436+0000 7f19be138640 1 -- 172.21.15.37:0/2690625535 shutdown_connections 2024-04-26T00:25:59.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.436+0000 7f19be138640 1 -- 172.21.15.37:0/2690625535 >> 172.21.15.37:0/2690625535 conn(0x7f19b80a1d50 msgr2=0x7f19b8131db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:59.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.436+0000 7f19be138640 1 -- 172.21.15.37:0/2690625535 shutdown_connections 2024-04-26T00:25:59.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.436+0000 7f19be138640 1 -- 172.21.15.37:0/2690625535 wait complete. 2024-04-26T00:25:59.441 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:25:59.441 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:25:59.441 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:25:59.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.568+0000 7f62327b8640 10 monclient: get_monmap_and_config 2024-04-26T00:25:59.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.568+0000 7f62327b8640 10 monclient: build_initial_monmap 2024-04-26T00:25:59.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.568+0000 7f62327b8640 10 monclient: monmap: 2024-04-26T00:25:59.554 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:59.554 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:59.554 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:25:59.575498+0000 2024-04-26T00:25:59.554 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:25:59.575498+0000 2024-04-26T00:25:59.554 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:59.555 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:59.555 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:25:59.555 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:25:59.555 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:25:59.555 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:25:59.555 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:25:59.555 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:59.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.572+0000 7f62327b8640 1 Processor -- start 2024-04-26T00:25:59.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.572+0000 7f62327b8640 1 -- start start 2024-04-26T00:25:59.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.572+0000 7f62327b8640 10 monclient: init 2024-04-26T00:25:59.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.572+0000 7f62327b8640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:59.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.572+0000 7f62327b8640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-04-26T00:25:59.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.572+0000 7f62327b8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f622c0a3d90 addr v1:172.21.15.37:6789/0 2024-04-26T00:25:59.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.572+0000 7f62327b8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f622c0a4170 addr v1:172.21.15.45:6789/0 2024-04-26T00:25:59.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.572+0000 7f62327b8640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f622c13d3f0 0x7f622c13f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:59.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.572+0000 7f62327b8640 10 monclient(hunting): picked mon.noname-a con 0x7f622c13d3f0 addr v2:172.21.15.37:3300/0 2024-04-26T00:25:59.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.572+0000 7f62327b8640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f622c0a5150 con 0x7f622c0a3d90 2024-04-26T00:25:59.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.572+0000 7f62327b8640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f622c09b280 con 0x7f622c0a4170 2024-04-26T00:25:59.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.572+0000 7f62327b8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:59.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.572+0000 7f62327b8640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f622c09c650 con 0x7f622c13d3f0 2024-04-26T00:25:59.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.572+0000 7f62327b8640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:59.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.572+0000 7f62327b8640 10 monclient(hunting): authenticate will time out at 1340.881161s 2024-04-26T00:25:59.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.572+0000 7f6230d55640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f622c13d3f0 0x7f622c13f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:59.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.572+0000 7f622bfff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f622c0a3d90 0x7f622c09bf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:39690/0 (socket says 172.21.15.37:39690) 2024-04-26T00:25:59.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.572+0000 7f622bfff640 1 -- 172.21.15.37:0/366799132 learned_addr learned my addr 172.21.15.37:0/366799132 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:25:59.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.572+0000 7f6230d55640 10 monclient(hunting): get_auth_request con 0x7f622c13d3f0 auth_method 0 2024-04-26T00:25:59.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.572+0000 7f6230d55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:59.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.572+0000 7f6230d55640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:59.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.572+0000 7f6230d55640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:59.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.572+0000 7f6230d55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:59.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.572+0000 7f6230d55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:59.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.572+0000 7f6230d55640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:59.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.572+0000 7f6230d55640 10 monclient(hunting): handle_auth_done global_id 54321 payload 274 2024-04-26T00:25:59.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.572+0000 7f6230d55640 1 -- 172.21.15.37:0/366799132 >> v1:172.21.15.45:6789/0 conn(0x7f622c0a4170 legacy=0x7f622c0a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:59.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.572+0000 7f6230d55640 1 -- 172.21.15.37:0/366799132 >> v1:172.21.15.37:6789/0 conn(0x7f622c0a3d90 legacy=0x7f622c09bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:59.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.572+0000 7f6230d55640 10 monclient: _finish_hunting 0 2024-04-26T00:25:59.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.572+0000 7f6230d55640 1 monclient: found mon.noname-a 2024-04-26T00:25:59.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.572+0000 7f6230d55640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:59.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.572+0000 7f6230d55640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:25:59.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.572+0000 7f6230d55640 1 -- 172.21.15.37:0/366799132 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f622c09b560 con 0x7f622c13d3f0 2024-04-26T00:25:59.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.576+0000 7f6230d55640 1 -- 172.21.15.37:0/366799132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f622c13d3f0 msgr2=0x7f622c13f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:59.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.576+0000 7f6230d55640 1 --2- 172.21.15.37:0/366799132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f622c13d3f0 0x7f622c13f7e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6224009d90 tx=0x7f622401c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbdd7ba5444e8e4c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:59.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.576+0000 7f622affd640 1 -- 172.21.15.37:0/366799132 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62240b2070 con 0x7f622c13d3f0 2024-04-26T00:25:59.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.576+0000 7f622affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:59.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.576+0000 7f622affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:25:59.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.576+0000 7f622affd640 10 monclient: dump: 2024-04-26T00:25:59.562 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:59.562 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:59.562 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.562 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.562 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:59.562 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:59.562 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:59.562 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:59.562 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:59.562 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:59.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.576+0000 7f622affd640 10 monclient: _finish_auth 0 2024-04-26T00:25:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.576+0000 7f622affd640 10 monclient: _check_auth_tickets 2024-04-26T00:25:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.576+0000 7f622affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.576+0000 7f622affd640 1 -- 172.21.15.37:0/366799132 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62240a5860 con 0x7f622c13d3f0 2024-04-26T00:25:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.576+0000 7f622affd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.576+0000 7f622affd640 1 -- 172.21.15.37:0/366799132 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62240a5c30 con 0x7f622c13d3f0 2024-04-26T00:25:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.576+0000 7f62327b8640 5 monclient: authenticate success, global_id 54321 2024-04-26T00:25:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.576+0000 7f62327b8640 10 monclient: get_monmap_and_config success 2024-04-26T00:25:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.576+0000 7f62327b8640 4 set_mon_vals no callback set 2024-04-26T00:25:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.576+0000 7f622affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.576+0000 7f622affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.576+0000 7f622affd640 10 monclient: dump: 2024-04-26T00:25:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:59.563 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:59.564 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.576+0000 7f62327b8640 10 monclient: shutdown 2024-04-26T00:25:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.576+0000 7f62327b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.576+0000 7f62327b8640 1 -- 172.21.15.37:0/366799132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f622c13d3f0 msgr2=0x7f622c13f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.576+0000 7f62327b8640 1 --2- 172.21.15.37:0/366799132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f622c13d3f0 0x7f622c13f7e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6224009d90 tx=0x7f622401c7b0 comp rx=0 tx=0).stop 2024-04-26T00:25:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.576+0000 7f62327b8640 1 -- 172.21.15.37:0/366799132 shutdown_connections 2024-04-26T00:25:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.576+0000 7f62327b8640 1 --2- 172.21.15.37:0/366799132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f622c13d3f0 0x7f622c13f7e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.576+0000 7f62327b8640 1 -- 172.21.15.37:0/366799132 >> 172.21.15.37:0/366799132 conn(0x7f622c0aa0a0 msgr2=0x7f622c0a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:59.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.576+0000 7f62327b8640 1 -- 172.21.15.37:0/366799132 shutdown_connections 2024-04-26T00:25:59.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.576+0000 7f62327b8640 1 -- 172.21.15.37:0/366799132 wait complete. 2024-04-26T00:25:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.576+0000 7f62327b8640 10 monclient: build_initial_monmap 2024-04-26T00:25:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.576+0000 7f62327b8640 10 monclient: monmap: 2024-04-26T00:25:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:25:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:25:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:25:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:25:59.570 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:25:59.570 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:25:59.570 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:25:59.570 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:59.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.576+0000 7f62327b8640 1 Processor -- start 2024-04-26T00:25:59.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f62327b8640 1 -- start start 2024-04-26T00:25:59.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f62327b8640 10 monclient: init 2024-04-26T00:25:59.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f62327b8640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:59.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f62327b8640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:25:59.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f62327b8640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f622c0a3d90 0x7f622c1367b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:59.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f62327b8640 10 monclient(hunting): picked mon.noname-a con 0x7f622c0a3d90 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:25:59.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f62327b8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f622c0a4170 0x7f622c136cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:59.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f62327b8640 10 monclient(hunting): picked mon.noname-c con 0x7f622c0a4170 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:59.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f62327b8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f622c13d3f0 0x7f622c139d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:59.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f62327b8640 10 monclient(hunting): picked mon.noname-b con 0x7f622c13d3f0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:25:59.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f62327b8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:59.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f62327b8640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f622c142ef0 con 0x7f622c0a3d90 2024-04-26T00:25:59.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f62327b8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:59.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f62327b8640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f622c142d70 con 0x7f622c13d3f0 2024-04-26T00:25:59.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f62327b8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:59.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f62327b8640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f622c143070 con 0x7f622c0a4170 2024-04-26T00:25:59.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f62327b8640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:59.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f62327b8640 10 monclient(hunting): authenticate will time out at 1340.888528s 2024-04-26T00:25:59.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f622b7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f622c0a4170 0x7f622c136cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:59.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f622bfff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f622c0a3d90 0x7f622c1367b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:59.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f622b7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f622c0a4170 0x7f622c136cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51940/0 (socket says 172.21.15.37:51940) 2024-04-26T00:25:59.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f622b7fe640 1 -- 172.21.15.37:0/2242441451 learned_addr learned my addr 172.21.15.37:0/2242441451 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:59.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f622b7fe640 10 monclient(hunting): get_auth_request con 0x7f622c0a4170 auth_method 0 2024-04-26T00:25:59.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f622b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:59.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f622b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:59.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f622b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:59.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f622bfff640 10 monclient(hunting): get_auth_request con 0x7f622c0a3d90 auth_method 0 2024-04-26T00:25:59.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f622bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:59.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f622bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:59.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f622bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:59.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f622bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:59.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f622bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:59.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f622bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:59.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f622b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:59.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f622b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:59.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f622b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:59.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f6230d55640 1 --2- 172.21.15.37:0/2242441451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f622c13d3f0 0x7f622c139d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:59.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f6230d55640 10 monclient(hunting): get_auth_request con 0x7f622c13d3f0 auth_method 0 2024-04-26T00:25:59.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f6230d55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:59.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f6230d55640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:59.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f6230d55640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:59.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f622bfff640 10 monclient(hunting): handle_auth_done global_id 54324 payload 824 2024-04-26T00:25:59.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f622bfff640 1 -- 172.21.15.37:0/2242441451 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f622c0a4170 msgr2=0x7f622c136cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:59.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f622bfff640 1 --2- 172.21.15.37:0/2242441451 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f622c0a4170 0x7f622c136cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:59.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f622bfff640 1 -- 172.21.15.37:0/2242441451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f622c13d3f0 msgr2=0x7f622c139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:59.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f622bfff640 1 --2- 172.21.15.37:0/2242441451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f622c13d3f0 0x7f622c139d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:59.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f622bfff640 10 monclient: _finish_hunting 0 2024-04-26T00:25:59.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f622bfff640 1 monclient: found mon.noname-a 2024-04-26T00:25:59.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f622bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:59.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f622bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:25:59.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f622bfff640 1 -- 172.21.15.37:0/2242441451 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f622c09ada0 con 0x7f622c0a3d90 2024-04-26T00:25:59.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f622bfff640 1 -- 172.21.15.37:0/2242441451 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f622c0a3d90 msgr2=0x7f622c1367b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:59.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f622bfff640 1 --2- 172.21.15.37:0/2242441451 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f622c0a3d90 0x7f622c1367b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f621c00c440 tx=0x7f621c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:59.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f6228ff9640 1 -- 172.21.15.37:0/2242441451 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f621c017070 con 0x7f622c0a3d90 2024-04-26T00:25:59.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f6228ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:59.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f6228ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:25:59.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f6228ff9640 10 monclient: dump: 2024-04-26T00:25:59.574 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:59.574 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:59.574 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.574 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.574 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:59.574 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:59.574 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:59.574 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:59.574 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:59.575 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:59.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f6228ff9640 10 monclient: _finish_auth 0 2024-04-26T00:25:59.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f6228ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:25:59.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f6228ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:59.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f62327b8640 5 monclient: authenticate success, global_id 54324 2024-04-26T00:25:59.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f6228ff9640 1 -- 172.21.15.37:0/2242441451 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f621c0070d0 con 0x7f622c0a3d90 2024-04-26T00:25:59.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f62327b8640 10 monclient: _renew_subs 2024-04-26T00:25:59.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f62327b8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:59.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f62327b8640 1 -- 172.21.15.37:0/2242441451 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f622c13d840 con 0x7f622c0a3d90 2024-04-26T00:25:59.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f6228ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:59.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f6228ff9640 1 -- 172.21.15.37:0/2242441451 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f621c0074c0 con 0x7f622c0a3d90 2024-04-26T00:25:59.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f6228ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:59.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f6228ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:59.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f6228ff9640 10 monclient: dump: 2024-04-26T00:25:59.575 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:59.576 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:59.576 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.576 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.576 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:59.576 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:59.576 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:59.576 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:59.576 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:59.576 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:59.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f62327b8640 10 monclient: _renew_subs 2024-04-26T00:25:59.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f62327b8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:59.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f62327b8640 1 -- 172.21.15.37:0/2242441451 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f622c13c6a0 con 0x7f622c0a3d90 2024-04-26T00:25:59.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.580+0000 7f622a7fc640 4 set_mon_vals no callback set 2024-04-26T00:25:59.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.584+0000 7f62327b8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:59.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.584+0000 7f62327b8640 1 --2- 172.21.15.37:0/2242441451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f622c13a260 0x7f622c13a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:59.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.584+0000 7f62327b8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:59.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.584+0000 7f62327b8640 1 -- 172.21.15.37:0/2242441451 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f622c04f510 con 0x7f622c13a260 2024-04-26T00:25:59.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.584+0000 7f6228ff9640 1 -- 172.21.15.37:0/2242441451 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f621c01a070 con 0x7f622c0a3d90 2024-04-26T00:25:59.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.584+0000 7f622b7fe640 1 --2- 172.21.15.37:0/2242441451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f622c13a260 0x7f622c13a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:59.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.584+0000 7f622b7fe640 10 monclient: get_auth_request con 0x7f622c13a260 auth_method 0 2024-04-26T00:25:59.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.584+0000 7f622b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:59.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.584+0000 7f622b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:59.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.584+0000 7f622b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:59.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.584+0000 7f6228ff9640 1 --2- 172.21.15.37:0/2242441451 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6218035f00 0x7f62180383c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:59.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.584+0000 7f6228ff9640 1 -- 172.21.15.37:0/2242441451 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(236..236 src has 177..236) ==== 3947+0+0 (secure 0 0 0) 0x7f621c047f40 con 0x7f622c0a3d90 2024-04-26T00:25:59.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.584+0000 7f622b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:59.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.584+0000 7f622b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:59.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.584+0000 7f622b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:59.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.584+0000 7f6230d55640 1 --2- 172.21.15.37:0/2242441451 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6218035f00 0x7f62180383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:59.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.584+0000 7f6230d55640 10 monclient: get_auth_request con 0x7f6218035f00 auth_method 0 2024-04-26T00:25:59.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.584+0000 7f622b7fe640 10 monclient(hunting): handle_auth_done global_id 24550 payload 824 2024-04-26T00:25:59.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.584+0000 7f622b7fe640 1 -- 172.21.15.37:0/2242441451 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f622c0008e0 con 0x7f622c13a260 2024-04-26T00:25:59.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.584+0000 7f622b7fe640 1 -- 172.21.15.37:0/2242441451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f622c13a260 msgr2=0x7f622c13a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:59.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.584+0000 7f622b7fe640 1 --2- 172.21.15.37:0/2242441451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f622c13a260 0x7f622c13a640 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f622c13b520 tx=0x7f622000fc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:59.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.588+0000 7f6228ff9640 1 -- 172.21.15.37:0/2242441451 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6220018030 con 0x7f622c13a260 2024-04-26T00:25:59.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.588+0000 7f6228ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:59.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.588+0000 7f6228ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:25:59.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.588+0000 7f6230d55640 1 --2- 172.21.15.37:0/2242441451 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6218035f00 0x7f62180383c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f62240ac000 tx=0x7f62240a7070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:59.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.588+0000 7f6228ff9640 10 monclient: dump: 2024-04-26T00:25:59.580 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:59.580 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:59.580 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.580 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.580 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:59.580 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:59.580 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:59.580 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:59.580 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:59.581 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:59.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.588+0000 7f6228ff9640 1 -- 172.21.15.37:0/2242441451 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f622000dbc0 con 0x7f622c13a260 2024-04-26T00:25:59.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.588+0000 7f6228ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:59.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.588+0000 7f6228ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:25:59.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.588+0000 7f6228ff9640 1 -- 172.21.15.37:0/2242441451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f622c13a260 msgr2=0x7f622c13a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:59.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.588+0000 7f6228ff9640 1 --2- 172.21.15.37:0/2242441451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f622c13a260 0x7f622c13a640 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f622c13b520 tx=0x7f622000fc00 comp rx=0 tx=0).stop 2024-04-26T00:25:59.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.588+0000 7f6228ff9640 1 -- 172.21.15.37:0/2242441451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f622c13a260 msgr2=0x7f622c13a640 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:25:59.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.588+0000 7f6228ff9640 1 --2- 172.21.15.37:0/2242441451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f622c13a260 0x7f622c13a640 secure :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f622c13b520 tx=0x7f622000fc00 comp rx=0 tx=0).stop 2024-04-26T00:25:59.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.592+0000 7f62327b8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:25:59.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.592+0000 7f62327b8640 1 --2- 172.21.15.37:0/2242441451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f622c13ab80 0x7f622c0a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:59.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.592+0000 7f62327b8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:59.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.592+0000 7f62327b8640 1 -- 172.21.15.37:0/2242441451 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f622c11fb00 con 0x7f622c13ab80 2024-04-26T00:25:59.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.592+0000 7f622b7fe640 1 --2- 172.21.15.37:0/2242441451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f622c13ab80 0x7f622c0a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:59.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.592+0000 7f622b7fe640 10 monclient: get_auth_request con 0x7f622c13ab80 auth_method 0 2024-04-26T00:25:59.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.592+0000 7f622b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:59.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.592+0000 7f622b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:59.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.592+0000 7f622b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:59.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.592+0000 7f622b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:59.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.592+0000 7f622b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:59.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.592+0000 7f622b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:59.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.592+0000 7f622b7fe640 10 monclient(hunting): handle_auth_done global_id 24553 payload 824 2024-04-26T00:25:59.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.596+0000 7f622b7fe640 1 -- 172.21.15.37:0/2242441451 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f622c13af90 con 0x7f622c13ab80 2024-04-26T00:25:59.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.596+0000 7f622b7fe640 1 -- 172.21.15.37:0/2242441451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f622c13ab80 msgr2=0x7f622c0a17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:59.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.596+0000 7f622b7fe640 1 --2- 172.21.15.37:0/2242441451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f622c13ab80 0x7f622c0a17a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6220018000 tx=0x7f6220007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:59.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.596+0000 7f6228ff9640 1 -- 172.21.15.37:0/2242441451 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f622001dd80 con 0x7f622c13ab80 2024-04-26T00:25:59.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.596+0000 7f6228ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:59.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.596+0000 7f6228ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:25:59.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.596+0000 7f6228ff9640 10 monclient: dump: 2024-04-26T00:25:59.584 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:59.584 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:59.584 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.584 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.584 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:59.584 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:59.584 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:59.584 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:59.584 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:59.584 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:59.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.596+0000 7f6228ff9640 1 -- 172.21.15.37:0/2242441451 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4719 (secure 0 0 0) 0x7f6220003fe0 con 0x7f622c13ab80 2024-04-26T00:25:59.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.596+0000 7f6228ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:25:59.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.596+0000 7f6228ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:25:59.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.596+0000 7f6228ff9640 1 -- 172.21.15.37:0/2242441451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f622c13ab80 msgr2=0x7f622c0a17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:59.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.596+0000 7f6228ff9640 1 --2- 172.21.15.37:0/2242441451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f622c13ab80 0x7f622c0a17a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6220018000 tx=0x7f6220007f90 comp rx=0 tx=0).stop 2024-04-26T00:25:59.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.596+0000 7f6228ff9640 1 -- 172.21.15.37:0/2242441451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f622c13ab80 msgr2=0x7f622c0a17a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:25:59.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.596+0000 7f6228ff9640 1 --2- 172.21.15.37:0/2242441451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f622c13ab80 0x7f622c0a17a0 secure :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6220018000 tx=0x7f6220007f90 comp rx=0 tx=0).stop 2024-04-26T00:25:59.585 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:25:59.585 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:25:59.585 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:25:59.585 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:25:59.585 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 46, 2024-04-26T00:25:59.585 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 267767, 2024-04-26T00:25:59.586 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:25:59.586 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:25:59.586 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:25:59.586 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:25:59.586 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:59.586 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 6, 2024-04-26T00:25:59.586 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:25:59.586 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:25:59.586 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:25:59.586 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:59.586 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:59.586 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:59.586 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:59.586 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:59.587 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:59.587 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:59.587 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:59.587 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:59.587 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:59.587 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:59.587 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:59.587 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:25:59.587 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:25:59.587 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:59.587 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:59.587 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:59.588 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:59.588 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:59.588 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:59.588 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:59.588 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:59.588 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:59.588 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:59.588 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:59.588 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:59.588 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:59.588 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:25:59.588 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:25:59.589 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:25:59.589 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:25:59.589 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:25:59.589 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:25:59.589 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:25:59.589 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:25:59.589 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:25:59.589 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:25:59.589 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:25:59.589 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:25:59.589 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:25:59.589 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:25:59.589 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:25:59.590 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:25:59.590 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:25:59.590 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:25:59.590 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:25:59.590 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:25:59.590 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:25:59.590 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:25:59.590 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:25:59.590 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:25:59.590 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:25:59.590 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:25:59.590 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:25:59.591 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:25:59.591 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:59.591 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:25:59.591 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:59.591 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:25:59.591 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:59.591 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:25:59.591 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:25:59.591 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:59.591 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:59.591 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:59.591 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:59.592 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:25:59.592 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:59.592 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:59.592 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:59.592 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:59.592 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:25:59.592 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:59.592 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:59.592 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:59.592 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:59.592 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:25:59.592 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:25:59.592 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:59.593 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:59.593 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:59.593 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:59.593 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:59.593 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:25:59.593 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:25:59.593 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:59.593 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:59.593 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:59.593 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:59.593 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:25:59.593 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:59.594 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:59.594 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:59.594 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:59.594 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:25:59.594 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:59.594 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:59.594 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:59.594 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:59.594 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:25:59.594 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:25:59.594 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:59.594 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:59.594 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:59.594 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:59.595 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:59.595 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:25:59.595 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:25:59.595 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:25:59.595 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:25:59.595 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:59.595 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:25:59.595 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:25:59.595 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:59.595 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:59.595 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:59.595 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:25:59.595 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:25:59.595 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:25:59.595 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:59.596 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:59.596 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:59.596 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:25:59.596 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:25:59.596 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:25:59.596 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:25:59.596 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:25:59.596 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:59.596 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:59.596 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:59.596 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:25:59.596 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:25:59.596 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:59.596 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:59.596 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:59.597 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:25:59.597 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:59.597 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:59.597 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:25:59.597 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:59.597 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:59.597 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:59.597 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:25:59.597 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:59.597 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:25:59.597 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:25:59.597 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:25:59.597 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:25:59.597 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:25:59.597 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:25:59.598 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:25:59.598 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:25:59.598 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:25:59.598 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:25:59.598 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:25:59.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.600+0000 7f62327b8640 1 -- 172.21.15.37:0/2242441451 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6218035f00 msgr2=0x7f62180383c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:59.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.600+0000 7f62327b8640 1 --2- 172.21.15.37:0/2242441451 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6218035f00 0x7f62180383c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f62240ac000 tx=0x7f62240a7070 comp rx=0 tx=0).stop 2024-04-26T00:25:59.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.600+0000 7f62327b8640 10 monclient: shutdown 2024-04-26T00:25:59.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.600+0000 7f62327b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:59.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.600+0000 7f62327b8640 1 -- 172.21.15.37:0/2242441451 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f622c0a3d90 msgr2=0x7f622c1367b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:59.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.600+0000 7f62327b8640 1 --2- 172.21.15.37:0/2242441451 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f622c0a3d90 0x7f622c1367b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f621c00c440 tx=0x7f621c00c900 comp rx=0 tx=0).stop 2024-04-26T00:25:59.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.600+0000 7f6230d55640 1 -- 172.21.15.37:0/2242441451 reap_dead start 2024-04-26T00:25:59.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.600+0000 7f6230d55640 1 -- 172.21.15.37:0/2242441451 reap_dead start 2024-04-26T00:25:59.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.600+0000 7f62327b8640 1 -- 172.21.15.37:0/2242441451 shutdown_connections 2024-04-26T00:25:59.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.600+0000 7f62327b8640 1 -- 172.21.15.37:0/2242441451 >> 172.21.15.37:0/2242441451 conn(0x7f622c0aa0a0 msgr2=0x7f622c0a8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:59.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.600+0000 7f62327b8640 1 -- 172.21.15.37:0/2242441451 shutdown_connections 2024-04-26T00:25:59.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.600+0000 7f62327b8640 1 -- 172.21.15.37:0/2242441451 wait complete. 2024-04-26T00:25:59.600 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:25:59.600 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-04-26T00:25:59.600 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-26T00:25:59.600 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-04-26T00:25:59.601 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-04-26T00:25:59.601 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-26T00:25:59.601 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T00:25:59.611 INFO:tasks.ceph.mon.c:Stopped 2024-04-26T00:25:59.611 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-26T00:25:59.611 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:25:59.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f698d534640 10 monclient: get_monmap_and_config 2024-04-26T00:25:59.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f698d534640 10 monclient: build_initial_monmap 2024-04-26T00:25:59.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f698d534640 10 monclient: monmap: 2024-04-26T00:25:59.769 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:59.769 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:59.769 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:25:59.789662+0000 2024-04-26T00:25:59.769 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:25:59.789662+0000 2024-04-26T00:25:59.769 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:59.769 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:59.769 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:25:59.769 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:25:59.769 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:25:59.769 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:25:59.769 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:25:59.769 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:59.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f698d534640 1 Processor -- start 2024-04-26T00:25:59.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f698d534640 1 -- start start 2024-04-26T00:25:59.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f698d534640 10 monclient: init 2024-04-26T00:25:59.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f698d534640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:59.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f698d534640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-04-26T00:25:59.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f698d534640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f69880a1010 addr v1:172.21.15.45:6789/0 2024-04-26T00:25:59.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f698d534640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f69880a13f0 0x7f698813be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:59.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f698d534640 10 monclient(hunting): picked mon.noname-c con 0x7f69880a13f0 addr v2:172.21.15.45:3300/0 2024-04-26T00:25:59.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f698d534640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f698813c390 0x7f698813e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:59.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f698d534640 10 monclient(hunting): picked mon.noname-a con 0x7f698813c390 addr v2:172.21.15.37:3300/0 2024-04-26T00:25:59.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f698d534640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69880a36f0 con 0x7f69880a1010 2024-04-26T00:25:59.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f698d534640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:59.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f698d534640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f698809e8a0 con 0x7f698813c390 2024-04-26T00:25:59.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f698d534640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:59.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f698d534640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f69880a3b30 con 0x7f69880a13f0 2024-04-26T00:25:59.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f698d534640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:59.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f698d534640 10 monclient(hunting): authenticate will time out at 1341.095317s 2024-04-26T00:25:59.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f69877fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f698813c390 0x7f698813e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:59.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f69877fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f698813c390 0x7f698813e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44774/0 (socket says 172.21.15.37:44774) 2024-04-26T00:25:59.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f69877fe640 1 -- 172.21.15.37:0/1365467103 learned_addr learned my addr 172.21.15.37:0/1365467103 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:59.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f69877fe640 10 monclient(hunting): get_auth_request con 0x7f698813c390 auth_method 0 2024-04-26T00:25:59.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f69877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:59.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f69877fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:59.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f69877fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:59.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f69877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:59.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f69877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:59.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f69877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:59.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f69867fc640 1 --2- 172.21.15.37:0/1365467103 >> v2:172.21.15.45:3300/0 conn(0x7f69880a13f0 0x7f698813be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:59.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f69867fc640 10 monclient(hunting): get_auth_request con 0x7f69880a13f0 auth_method 0 2024-04-26T00:25:59.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f69867fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:59.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f69867fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:59.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f69867fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:59.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f69877fe640 10 monclient(hunting): handle_auth_done global_id 54327 payload 274 2024-04-26T00:25:59.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f69877fe640 1 -- 172.21.15.37:0/1365467103 >> v2:172.21.15.45:3300/0 conn(0x7f69880a13f0 msgr2=0x7f698813be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:59.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f69877fe640 1 --2- 172.21.15.37:0/1365467103 >> v2:172.21.15.45:3300/0 conn(0x7f69880a13f0 0x7f698813be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:59.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f69877fe640 1 -- 172.21.15.37:0/1365467103 >> v1:172.21.15.45:6789/0 conn(0x7f69880a1010 legacy=0x7f698809f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:25:59.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f69877fe640 10 monclient: _finish_hunting 0 2024-04-26T00:25:59.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f69877fe640 1 monclient: found mon.noname-a 2024-04-26T00:25:59.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f69877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:59.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f69877fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:25:59.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.784+0000 7f69877fe640 1 -- 172.21.15.37:0/1365467103 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69880a3190 con 0x7f698813c390 2024-04-26T00:25:59.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f69877fe640 1 -- 172.21.15.37:0/1365467103 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f698813c390 msgr2=0x7f698813e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:59.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f69877fe640 1 --2- 172.21.15.37:0/1365467103 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f698813c390 0x7f698813e780 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f698809dba0 tx=0x7f698001c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9836ac8e7a1d345e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:59.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f6985ffb640 1 -- 172.21.15.37:0/1365467103 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69800a5820 con 0x7f698813c390 2024-04-26T00:25:59.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f6985ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:59.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f6985ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:25:59.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f6985ffb640 10 monclient: dump: 2024-04-26T00:25:59.777 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:59.777 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:59.777 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.777 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.777 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:59.777 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:59.777 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:59.777 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:59.777 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:59.777 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:59.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f6985ffb640 10 monclient: _finish_auth 0 2024-04-26T00:25:59.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f6985ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:25:59.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f6985ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:59.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f6985ffb640 1 -- 172.21.15.37:0/1365467103 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69800a59c0 con 0x7f698813c390 2024-04-26T00:25:59.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f6985ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:59.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f6985ffb640 1 -- 172.21.15.37:0/1365467103 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69800a5d90 con 0x7f698813c390 2024-04-26T00:25:59.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f698d534640 5 monclient: authenticate success, global_id 54327 2024-04-26T00:25:59.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f698d534640 10 monclient: get_monmap_and_config success 2024-04-26T00:25:59.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f698d534640 4 set_mon_vals no callback set 2024-04-26T00:25:59.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f6985ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:59.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f6985ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:59.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f6985ffb640 10 monclient: dump: 2024-04-26T00:25:59.778 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:59.778 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:59.778 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.778 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.779 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:59.779 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:59.779 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:59.779 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:59.779 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:59.779 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:59.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f698d534640 10 monclient: shutdown 2024-04-26T00:25:59.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f698d534640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:25:59.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f698d534640 1 -- 172.21.15.37:0/1365467103 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f698813c390 msgr2=0x7f698813e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:59.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f698d534640 1 --2- 172.21.15.37:0/1365467103 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f698813c390 0x7f698813e780 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f698809dba0 tx=0x7f698001c850 comp rx=0 tx=0).stop 2024-04-26T00:25:59.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f698d534640 1 -- 172.21.15.37:0/1365467103 shutdown_connections 2024-04-26T00:25:59.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f698d534640 1 --2- 172.21.15.37:0/1365467103 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f698813c390 0x7f698813e780 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:59.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f698d534640 1 --2- 172.21.15.37:0/1365467103 >> v2:172.21.15.45:3300/0 conn(0x7f69880a13f0 0x7f698813be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:59.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f698d534640 1 -- 172.21.15.37:0/1365467103 >> 172.21.15.37:0/1365467103 conn(0x7f69880a4170 msgr2=0x7f69880a4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:25:59.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f698d534640 1 -- 172.21.15.37:0/1365467103 shutdown_connections 2024-04-26T00:25:59.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f698d534640 1 -- 172.21.15.37:0/1365467103 wait complete. 2024-04-26T00:25:59.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f698d534640 10 monclient: build_initial_monmap 2024-04-26T00:25:59.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f698d534640 10 monclient: monmap: 2024-04-26T00:25:59.780 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:25:59.780 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:25:59.780 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:25:59.780 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:25:59.780 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:25:59.780 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:25:59.780 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:25:59.780 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:25:59.780 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:25:59.781 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:59.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f698d534640 1 Processor -- start 2024-04-26T00:25:59.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f698d534640 1 -- start start 2024-04-26T00:25:59.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f698d534640 10 monclient: init 2024-04-26T00:25:59.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f698d534640 10 monclient: _reopen_session rank -1 2024-04-26T00:25:59.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f698d534640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:25:59.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f698d534640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69880a1010 0x7f69881ceb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:59.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f698d534640 10 monclient(hunting): picked mon.noname-b con 0x7f69880a1010 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:25:59.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f698d534640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69880a13f0 0x7f698813b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:59.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f698d534640 10 monclient(hunting): picked mon.noname-c con 0x7f69880a13f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:25:59.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f698d534640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f698813c390 0x7f6988121f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:59.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f698d534640 10 monclient(hunting): picked mon.noname-a con 0x7f698813c390 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:25:59.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f698d534640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:59.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f698d534640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f6988141870 con 0x7f698813c390 2024-04-26T00:25:59.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f698d534640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:59.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f698d534640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f69881416f0 con 0x7f69880a1010 2024-04-26T00:25:59.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f698d534640 10 monclient(hunting): start opening mon connection 2024-04-26T00:25:59.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f698d534640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f69881419f0 con 0x7f69880a13f0 2024-04-26T00:25:59.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f698d534640 10 monclient(hunting): _renew_subs 2024-04-26T00:25:59.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f698d534640 10 monclient(hunting): authenticate will time out at 1341.102608s 2024-04-26T00:25:59.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f69867fc640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69880a13f0 msgr2=0x7f698813b450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:25:59.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f69867fc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69880a13f0 0x7f698813b450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:25:59.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f69877fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f698813c390 0x7f6988121f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:59.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f69877fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f698813c390 0x7f6988121f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44778/0 (socket says 172.21.15.37:44778) 2024-04-26T00:25:59.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f69877fe640 1 -- 172.21.15.37:0/1044030171 learned_addr learned my addr 172.21.15.37:0/1044030171 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:25:59.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f69877fe640 10 monclient(hunting): get_auth_request con 0x7f698813c390 auth_method 0 2024-04-26T00:25:59.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f69877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:59.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f69877fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:59.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f69877fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:59.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f69877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:25:59.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f69877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:25:59.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f69877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:25:59.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f6986ffd640 1 --2- 172.21.15.37:0/1044030171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69880a1010 0x7f69881ceb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:59.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f6986ffd640 10 monclient(hunting): get_auth_request con 0x7f69880a1010 auth_method 0 2024-04-26T00:25:59.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f6986ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:25:59.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f6986ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:25:59.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f6986ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:25:59.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f69877fe640 10 monclient(hunting): handle_auth_done global_id 54330 payload 824 2024-04-26T00:25:59.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f69877fe640 1 -- 172.21.15.37:0/1044030171 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69880a13f0 msgr2=0x7f698813b450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:25:59.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f69877fe640 1 --2- 172.21.15.37:0/1044030171 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69880a13f0 0x7f698813b450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:59.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f69877fe640 1 -- 172.21.15.37:0/1044030171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69880a1010 msgr2=0x7f69881ceb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:25:59.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f69877fe640 1 --2- 172.21.15.37:0/1044030171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69880a1010 0x7f69881ceb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:25:59.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f69877fe640 10 monclient: _finish_hunting 0 2024-04-26T00:25:59.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f69877fe640 1 monclient: found mon.noname-a 2024-04-26T00:25:59.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f69877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:25:59.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f69877fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:25:59.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.792+0000 7f69877fe640 1 -- 172.21.15.37:0/1044030171 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6988125b00 con 0x7f698813c390 2024-04-26T00:25:59.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.796+0000 7f69877fe640 1 -- 172.21.15.37:0/1044030171 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f698813c390 msgr2=0x7f6988121f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:25:59.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.796+0000 7f69877fe640 1 --2- 172.21.15.37:0/1044030171 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f698813c390 0x7f6988121f60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f69800af040 tx=0x7f69800195f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:59.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.796+0000 7f696f7fe640 1 -- 172.21.15.37:0/1044030171 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69800bc020 con 0x7f698813c390 2024-04-26T00:25:59.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.796+0000 7f696f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:59.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.796+0000 7f696f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:25:59.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.796+0000 7f696f7fe640 10 monclient: dump: 2024-04-26T00:25:59.788 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:59.788 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:59.788 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.788 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.788 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:59.788 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:59.788 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:59.788 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:59.788 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:59.789 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:59.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.796+0000 7f696f7fe640 10 monclient: _finish_auth 0 2024-04-26T00:25:59.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.796+0000 7f696f7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:25:59.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.796+0000 7f696f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:25:59.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.796+0000 7f696f7fe640 1 -- 172.21.15.37:0/1044030171 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69800b8a90 con 0x7f698813c390 2024-04-26T00:25:59.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.796+0000 7f696f7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:25:59.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.796+0000 7f696f7fe640 1 -- 172.21.15.37:0/1044030171 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69800073f0 con 0x7f698813c390 2024-04-26T00:25:59.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.796+0000 7f696f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:25:59.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.796+0000 7f696f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:25:59.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.796+0000 7f696f7fe640 10 monclient: dump: 2024-04-26T00:25:59.789 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:25:59.789 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:25:59.789 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.789 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:25:59.790 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:25:59.790 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:25:59.790 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:25:59.790 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:25:59.790 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:25:59.790 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:25:59.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.796+0000 7f698d534640 5 monclient: authenticate success, global_id 54330 2024-04-26T00:25:59.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.796+0000 7f698d534640 10 monclient: _renew_subs 2024-04-26T00:25:59.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.796+0000 7f69857fa640 4 set_mon_vals no callback set 2024-04-26T00:25:59.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.796+0000 7f698d534640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:59.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.796+0000 7f698d534640 1 -- 172.21.15.37:0/1044030171 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6988125de0 con 0x7f698813c390 2024-04-26T00:25:59.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.796+0000 7f698d534640 10 monclient: _renew_subs 2024-04-26T00:25:59.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.796+0000 7f698d534640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:59.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.796+0000 7f698d534640 1 -- 172.21.15.37:0/1044030171 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69880a5370 con 0x7f698813c390 2024-04-26T00:25:59.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.796+0000 7f698d534640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:59.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.796+0000 7f698d534640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:59.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.796+0000 7f698d534640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:25:59.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.796+0000 7f696f7fe640 1 -- 172.21.15.37:0/1044030171 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f69800bf020 con 0x7f698813c390 2024-04-26T00:25:59.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.796+0000 7f698d534640 1 -- 172.21.15.37:0/1044030171 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6988066f40 con 0x7f698813c390 2024-04-26T00:25:59.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.796+0000 7f696f7fe640 1 --2- 172.21.15.37:0/1044030171 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6974035e80 0x7f6974038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:25:59.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.796+0000 7f696f7fe640 1 -- 172.21.15.37:0/1044030171 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(236..236 src has 177..236) ==== 3947+0+0 (secure 0 0 0) 0x7f69800e3a60 con 0x7f698813c390 2024-04-26T00:25:59.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.796+0000 7f6986ffd640 1 --2- 172.21.15.37:0/1044030171 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6974035e80 0x7f6974038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:25:59.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.796+0000 7f6986ffd640 10 monclient: get_auth_request con 0x7f6974035e80 auth_method 0 2024-04-26T00:25:59.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.800+0000 7f6986ffd640 1 --2- 172.21.15.37:0/1044030171 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6974035e80 0x7f6974038340 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6978004e30 tx=0x7f6978009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:25:59.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.800+0000 7f696f7fe640 1 -- 172.21.15.37:0/1044030171 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f69800e5e20 con 0x7f698813c390 2024-04-26T00:25:59.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.800+0000 7f696f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:25:59.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:25:59.800+0000 7f696f7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:26:00.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:00.020+0000 7f698d534640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:26:00.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:00.020+0000 7f698d534640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:26:00.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:00.020+0000 7f698d534640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:26:00.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:00.020+0000 7f698d534640 1 -- 172.21.15.37:0/1044030171 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f69880a5e20 con 0x7f698813c390 2024-04-26T00:26:00.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:00.020+0000 7f696f7fe640 1 -- 172.21.15.37:0/1044030171 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f69800c6070 con 0x7f698813c390 2024-04-26T00:26:00.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:00.020+0000 7f696f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:26:00.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:00.020+0000 7f696f7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:26:00.005 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":46,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:26:00.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:00.024+0000 7f698d534640 1 -- 172.21.15.37:0/1044030171 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6974035e80 msgr2=0x7f6974038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:00.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:00.024+0000 7f698d534640 1 --2- 172.21.15.37:0/1044030171 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6974035e80 0x7f6974038340 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6978004e30 tx=0x7f6978009040 comp rx=0 tx=0).stop 2024-04-26T00:26:00.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:00.024+0000 7f698d534640 10 monclient: shutdown 2024-04-26T00:26:00.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:00.024+0000 7f698d534640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:00.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:00.024+0000 7f698d534640 1 -- 172.21.15.37:0/1044030171 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f698813c390 msgr2=0x7f6988121f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:00.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:00.024+0000 7f698d534640 1 --2- 172.21.15.37:0/1044030171 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f698813c390 0x7f6988121f60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f69800af040 tx=0x7f69800195f0 comp rx=0 tx=0).stop 2024-04-26T00:26:00.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:00.024+0000 7f698d534640 1 -- 172.21.15.37:0/1044030171 shutdown_connections 2024-04-26T00:26:00.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:00.024+0000 7f698d534640 1 --2- 172.21.15.37:0/1044030171 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6974035e80 0x7f6974038340 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:00.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:00.024+0000 7f698d534640 1 --2- 172.21.15.37:0/1044030171 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f698813c390 0x7f6988121f60 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:00.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:00.024+0000 7f698d534640 1 --2- 172.21.15.37:0/1044030171 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69880a13f0 0x7f698813b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:00.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:00.024+0000 7f698d534640 1 --2- 172.21.15.37:0/1044030171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69880a1010 0x7f69881ceb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:00.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:00.024+0000 7f698d534640 1 -- 172.21.15.37:0/1044030171 >> 172.21.15.37:0/1044030171 conn(0x7f69880a4170 msgr2=0x7f698806c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:00.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:00.024+0000 7f698d534640 1 -- 172.21.15.37:0/1044030171 shutdown_connections 2024-04-26T00:26:00.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:00.024+0000 7f698d534640 1 -- 172.21.15.37:0/1044030171 wait complete. 2024-04-26T00:26:03.024 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:26:03.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.172+0000 7fa6d4ef0640 10 monclient: get_monmap_and_config 2024-04-26T00:26:03.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.172+0000 7fa6d4ef0640 10 monclient: build_initial_monmap 2024-04-26T00:26:03.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.172+0000 7fa6d4ef0640 10 monclient: monmap: 2024-04-26T00:26:03.156 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:03.156 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:03.156 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:26:03.177181+0000 2024-04-26T00:26:03.157 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:26:03.177181+0000 2024-04-26T00:26:03.157 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:03.157 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:03.157 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:26:03.157 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:26:03.157 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:26:03.157 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:26:03.157 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:26:03.157 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:03.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.172+0000 7fa6d4ef0640 1 Processor -- start 2024-04-26T00:26:03.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.172+0000 7fa6d4ef0640 1 -- start start 2024-04-26T00:26:03.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.172+0000 7fa6d4ef0640 10 monclient: init 2024-04-26T00:26:03.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.172+0000 7fa6d4ef0640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:03.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.172+0000 7fa6d4ef0640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-04-26T00:26:03.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.172+0000 7fa6d4ef0640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa6d00a0260 0x7fa6d00a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:03.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.172+0000 7fa6d4ef0640 10 monclient(hunting): picked mon.noname-c con 0x7fa6d00a0260 addr v2:172.21.15.45:3300/0 2024-04-26T00:26:03.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.172+0000 7fa6d4ef0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa6d0134b30 addr v1:172.21.15.37:6789/0 2024-04-26T00:26:03.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.172+0000 7fa6d4ef0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa6d0135660 addr v1:172.21.15.45:6789/0 2024-04-26T00:26:03.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.172+0000 7fa6d4ef0640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6d00a1130 con 0x7fa6d0134b30 2024-04-26T00:26:03.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.172+0000 7fa6d4ef0640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6d006cff0 con 0x7fa6d0135660 2024-04-26T00:26:03.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.172+0000 7fa6d4ef0640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:03.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.172+0000 7fa6d4ef0640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fa6d009ec10 con 0x7fa6d00a0260 2024-04-26T00:26:03.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.172+0000 7fa6d4ef0640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:03.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.172+0000 7fa6d4ef0640 10 monclient(hunting): authenticate will time out at 1344.482873s 2024-04-26T00:26:03.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.172+0000 7fa6cdd9b640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fa6d0134b30 0x7fa6d0134f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:55622/0 (socket says 172.21.15.37:55622) 2024-04-26T00:26:03.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.172+0000 7fa6cdd9b640 1 -- 172.21.15.37:0/3888030858 learned_addr learned my addr 172.21.15.37:0/3888030858 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:26:03.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.172+0000 7fa6ce59c640 1 --2- 172.21.15.37:0/3888030858 >> v2:172.21.15.45:3300/0 conn(0x7fa6d00a0260 0x7fa6d00a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:03.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.172+0000 7fa6ce59c640 10 monclient(hunting): get_auth_request con 0x7fa6d00a0260 auth_method 0 2024-04-26T00:26:03.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.172+0000 7fa6ce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:03.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.172+0000 7fa6ce59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:03.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.172+0000 7fa6ce59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:03.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.172+0000 7fa6ce59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:03.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.172+0000 7fa6ce59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:03.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.172+0000 7fa6ce59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:03.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.172+0000 7fa6ce59c640 10 monclient(hunting): handle_auth_done global_id 24562 payload 274 2024-04-26T00:26:03.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.172+0000 7fa6ce59c640 1 -- 172.21.15.37:0/3888030858 >> v1:172.21.15.45:6789/0 conn(0x7fa6d0135660 legacy=0x7fa6d013ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:26:03.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.172+0000 7fa6ce59c640 1 -- 172.21.15.37:0/3888030858 >> v1:172.21.15.37:6789/0 conn(0x7fa6d0134b30 legacy=0x7fa6d0134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:26:03.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.172+0000 7fa6ce59c640 10 monclient: _finish_hunting 0 2024-04-26T00:26:03.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.172+0000 7fa6ce59c640 1 monclient: found mon.noname-c 2024-04-26T00:26:03.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.172+0000 7fa6ce59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:03.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.172+0000 7fa6ce59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:26:03.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.172+0000 7fa6ce59c640 1 -- 172.21.15.37:0/3888030858 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6d009bc50 con 0x7fa6d00a0260 2024-04-26T00:26:03.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6ce59c640 1 -- 172.21.15.37:0/3888030858 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6d00a0260 msgr2=0x7fa6d00a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:03.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6ce59c640 1 --2- 172.21.15.37:0/3888030858 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6d00a0260 0x7fa6d00a0640 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa6c40097c0 tx=0x7fa6c401c100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b9d2cdf29671e26 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:03.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6cd59a640 1 -- 172.21.15.37:0/3888030858 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c40b1070 con 0x7fa6d00a0260 2024-04-26T00:26:03.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6cd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:03.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6cd59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:26:03.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6cd59a640 10 monclient: dump: 2024-04-26T00:26:03.164 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:03.164 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:03.164 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:03.164 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:03.164 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:03.164 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:03.165 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:03.165 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:03.165 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:03.165 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:03.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6cd59a640 10 monclient: _finish_auth 0 2024-04-26T00:26:03.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6cd59a640 10 monclient: _check_auth_tickets 2024-04-26T00:26:03.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6cd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:03.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6d4ef0640 5 monclient: authenticate success, global_id 24562 2024-04-26T00:26:03.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6d4ef0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:26:03.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6cd59a640 1 -- 172.21.15.37:0/3888030858 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6c40ac440 con 0x7fa6d00a0260 2024-04-26T00:26:03.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6cd59a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:03.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6d4ef0640 10 monclient: get_monmap_and_config success 2024-04-26T00:26:03.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6d4ef0640 4 set_mon_vals no callback set 2024-04-26T00:26:03.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6cd59a640 1 -- 172.21.15.37:0/3888030858 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c40ac810 con 0x7fa6d00a0260 2024-04-26T00:26:03.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6cd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:03.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6cd59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:26:03.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6cd59a640 10 monclient: dump: 2024-04-26T00:26:03.166 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:03.166 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:03.166 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:03.166 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:03.166 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:03.166 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:03.166 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:03.166 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:03.167 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:03.167 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:03.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6d4ef0640 10 monclient: shutdown 2024-04-26T00:26:03.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6d4ef0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:03.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6d4ef0640 1 -- 172.21.15.37:0/3888030858 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6d00a0260 msgr2=0x7fa6d00a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:03.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6d4ef0640 1 --2- 172.21.15.37:0/3888030858 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6d00a0260 0x7fa6d00a0640 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa6c40097c0 tx=0x7fa6c401c100 comp rx=0 tx=0).stop 2024-04-26T00:26:03.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6d4ef0640 1 -- 172.21.15.37:0/3888030858 shutdown_connections 2024-04-26T00:26:03.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6d4ef0640 1 --2- 172.21.15.37:0/3888030858 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6d00a0260 0x7fa6d00a0640 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:03.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6d4ef0640 1 -- 172.21.15.37:0/3888030858 >> 172.21.15.37:0/3888030858 conn(0x7fa6d00a1ad0 msgr2=0x7fa6d00a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:03.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6d4ef0640 1 -- 172.21.15.37:0/3888030858 shutdown_connections 2024-04-26T00:26:03.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6d4ef0640 1 -- 172.21.15.37:0/3888030858 wait complete. 2024-04-26T00:26:03.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6d4ef0640 10 monclient: build_initial_monmap 2024-04-26T00:26:03.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6d4ef0640 10 monclient: monmap: 2024-04-26T00:26:03.171 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:03.171 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:26:03.171 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:26:03.171 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:26:03.171 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:03.171 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:26:03.172 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:26:03.172 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:26:03.172 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:26:03.172 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:03.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6d4ef0640 1 Processor -- start 2024-04-26T00:26:03.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6d4ef0640 1 -- start start 2024-04-26T00:26:03.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6d4ef0640 10 monclient: init 2024-04-26T00:26:03.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6d4ef0640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:03.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6d4ef0640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:26:03.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6d4ef0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6d00a0260 0x7fa6d0130f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:03.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6d4ef0640 10 monclient(hunting): picked mon.noname-b con 0x7fa6d00a0260 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:26:03.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6d4ef0640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6d0134b30 0x7fa6d0131460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:03.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6d4ef0640 10 monclient(hunting): picked mon.noname-a con 0x7fa6d0134b30 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:26:03.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6d4ef0640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa6d0135660 0x7fa6d0139510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:03.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6d4ef0640 10 monclient(hunting): picked mon.noname-c con 0x7fa6d0135660 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:26:03.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6d4ef0640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:03.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6d4ef0640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d009f4c0 con 0x7fa6d0134b30 2024-04-26T00:26:03.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6d4ef0640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:03.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6d4ef0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d009f0f0 con 0x7fa6d00a0260 2024-04-26T00:26:03.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6d4ef0640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:03.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6d4ef0640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa6d0140db0 con 0x7fa6d0135660 2024-04-26T00:26:03.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6d4ef0640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:03.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6d4ef0640 10 monclient(hunting): authenticate will time out at 1344.490953s 2024-04-26T00:26:03.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6ced9d640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa6d0135660 msgr2=0x7fa6d0139510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:26:03.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6cdd9b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6d0134b30 0x7fa6d0131460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:03.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6ced9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa6d0135660 0x7fa6d0139510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:26:03.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6cdd9b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6d0134b30 0x7fa6d0131460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44786/0 (socket says 172.21.15.37:44786) 2024-04-26T00:26:03.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6cdd9b640 1 -- 172.21.15.37:0/2353765811 learned_addr learned my addr 172.21.15.37:0/2353765811 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:26:03.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6cdd9b640 10 monclient(hunting): get_auth_request con 0x7fa6d0134b30 auth_method 0 2024-04-26T00:26:03.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6cdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:03.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6cdd9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:03.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:03.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6cdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:03.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6cdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:03.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6cdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:03.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6ce59c640 1 --2- 172.21.15.37:0/2353765811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6d00a0260 0x7fa6d0130f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:03.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6ce59c640 10 monclient(hunting): get_auth_request con 0x7fa6d00a0260 auth_method 0 2024-04-26T00:26:03.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6ce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:03.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6ce59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:03.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6ce59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:03.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6ce59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:03.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6ce59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:03.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6ce59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:03.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6cdd9b640 10 monclient(hunting): handle_auth_done global_id 54333 payload 824 2024-04-26T00:26:03.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6cdd9b640 1 -- 172.21.15.37:0/2353765811 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa6d0135660 msgr2=0x7fa6d0139510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:26:03.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6cdd9b640 1 --2- 172.21.15.37:0/2353765811 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa6d0135660 0x7fa6d0139510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:03.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6cdd9b640 1 -- 172.21.15.37:0/2353765811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6d00a0260 msgr2=0x7fa6d0130f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:03.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6cdd9b640 1 --2- 172.21.15.37:0/2353765811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6d00a0260 0x7fa6d0130f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:03.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6cdd9b640 10 monclient: _finish_hunting 0 2024-04-26T00:26:03.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6cdd9b640 1 monclient: found mon.noname-a 2024-04-26T00:26:03.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6cdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:03.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6cdd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:26:03.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.180+0000 7fa6cdd9b640 1 -- 172.21.15.37:0/2353765811 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6d01348c0 con 0x7fa6d0134b30 2024-04-26T00:26:03.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.184+0000 7fa6cdd9b640 1 -- 172.21.15.37:0/2353765811 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6d0134b30 msgr2=0x7fa6d0131460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:03.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.184+0000 7fa6cdd9b640 1 --2- 172.21.15.37:0/2353765811 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6d0134b30 0x7fa6d0131460 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa6c000c470 tx=0x7fa6c000c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:03.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.184+0000 7fa6beffd640 1 -- 172.21.15.37:0/2353765811 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c0007070 con 0x7fa6d0134b30 2024-04-26T00:26:03.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.184+0000 7fa6beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:03.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.184+0000 7fa6beffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:26:03.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.184+0000 7fa6beffd640 10 monclient: dump: 2024-04-26T00:26:03.176 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:03.176 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:03.176 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:03.176 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:03.176 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:03.176 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:03.176 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:03.177 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:03.177 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:03.177 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:03.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.184+0000 7fa6beffd640 10 monclient: _finish_auth 0 2024-04-26T00:26:03.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.184+0000 7fa6beffd640 10 monclient: _check_auth_tickets 2024-04-26T00:26:03.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.184+0000 7fa6beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:03.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.184+0000 7fa6beffd640 1 -- 172.21.15.37:0/2353765811 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6c0007210 con 0x7fa6d0134b30 2024-04-26T00:26:03.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.184+0000 7fa6beffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:03.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.184+0000 7fa6beffd640 1 -- 172.21.15.37:0/2353765811 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c0007600 con 0x7fa6d0134b30 2024-04-26T00:26:03.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.184+0000 7fa6d4ef0640 5 monclient: authenticate success, global_id 54333 2024-04-26T00:26:03.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.184+0000 7fa6d4ef0640 10 monclient: _renew_subs 2024-04-26T00:26:03.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.184+0000 7fa6d4ef0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:26:03.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.184+0000 7fa6d4ef0640 1 -- 172.21.15.37:0/2353765811 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6d013d450 con 0x7fa6d0134b30 2024-04-26T00:26:03.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.184+0000 7fa6beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:03.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.184+0000 7fa6ccd99640 4 set_mon_vals no callback set 2024-04-26T00:26:03.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.184+0000 7fa6beffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:26:03.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.184+0000 7fa6beffd640 10 monclient: dump: 2024-04-26T00:26:03.178 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:03.178 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:03.178 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:03.178 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:03.178 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:03.178 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:03.178 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:03.178 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:03.178 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:03.178 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:03.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.184+0000 7fa6d4ef0640 10 monclient: _renew_subs 2024-04-26T00:26:03.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.184+0000 7fa6d4ef0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:26:03.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.184+0000 7fa6d4ef0640 1 -- 172.21.15.37:0/2353765811 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6d013ce90 con 0x7fa6d0134b30 2024-04-26T00:26:03.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.184+0000 7fa6d4ef0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:03.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.184+0000 7fa6d4ef0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:03.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.184+0000 7fa6d4ef0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:26:03.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.184+0000 7fa6beffd640 1 -- 172.21.15.37:0/2353765811 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa6c001b400 con 0x7fa6d0134b30 2024-04-26T00:26:03.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.184+0000 7fa6d4ef0640 1 -- 172.21.15.37:0/2353765811 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6d0066f40 con 0x7fa6d0134b30 2024-04-26T00:26:03.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.184+0000 7fa6beffd640 1 --2- 172.21.15.37:0/2353765811 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa6b0035e60 0x7fa6b0038320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:03.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.184+0000 7fa6beffd640 1 -- 172.21.15.37:0/2353765811 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(236..236 src has 177..236) ==== 3947+0+0 (secure 0 0 0) 0x7fa6c0048750 con 0x7fa6d0134b30 2024-04-26T00:26:03.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.184+0000 7fa6ce59c640 1 --2- 172.21.15.37:0/2353765811 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa6b0035e60 0x7fa6b0038320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:03.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.184+0000 7fa6ce59c640 10 monclient: get_auth_request con 0x7fa6b0035e60 auth_method 0 2024-04-26T00:26:03.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.188+0000 7fa6ce59c640 1 --2- 172.21.15.37:0/2353765811 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa6b0035e60 0x7fa6b0038320 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa6c40ae000 tx=0x7fa6c4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:03.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.192+0000 7fa6beffd640 1 -- 172.21.15.37:0/2353765811 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa6d01348c0 con 0x7fa6d0134b30 2024-04-26T00:26:03.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.192+0000 7fa6beffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:03.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.192+0000 7fa6beffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:26:03.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.404+0000 7fa6d4ef0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:26:03.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.404+0000 7fa6d4ef0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:26:03.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.404+0000 7fa6d4ef0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:26:03.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.404+0000 7fa6d4ef0640 1 -- 172.21.15.37:0/2353765811 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa6d00008e0 con 0x7fa6d0134b30 2024-04-26T00:26:03.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.408+0000 7fa6beffd640 1 -- 172.21.15.37:0/2353765811 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fa6c0047490 con 0x7fa6d0134b30 2024-04-26T00:26:03.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.408+0000 7fa6beffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:26:03.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.408+0000 7fa6beffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:26:03.391 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":46,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:26:03.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.408+0000 7fa6d4ef0640 1 -- 172.21.15.37:0/2353765811 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa6b0035e60 msgr2=0x7fa6b0038320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:03.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.408+0000 7fa6d4ef0640 1 --2- 172.21.15.37:0/2353765811 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa6b0035e60 0x7fa6b0038320 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa6c40ae000 tx=0x7fa6c4008040 comp rx=0 tx=0).stop 2024-04-26T00:26:03.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.408+0000 7fa6d4ef0640 10 monclient: shutdown 2024-04-26T00:26:03.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.408+0000 7fa6d4ef0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:03.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.408+0000 7fa6d4ef0640 1 -- 172.21.15.37:0/2353765811 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6d0134b30 msgr2=0x7fa6d0131460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:03.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.408+0000 7fa6d4ef0640 1 --2- 172.21.15.37:0/2353765811 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6d0134b30 0x7fa6d0131460 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa6c000c470 tx=0x7fa6c000c930 comp rx=0 tx=0).stop 2024-04-26T00:26:03.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.408+0000 7fa6d4ef0640 1 -- 172.21.15.37:0/2353765811 shutdown_connections 2024-04-26T00:26:03.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.408+0000 7fa6d4ef0640 1 --2- 172.21.15.37:0/2353765811 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa6b0035e60 0x7fa6b0038320 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:03.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.408+0000 7fa6d4ef0640 1 --2- 172.21.15.37:0/2353765811 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa6d0135660 0x7fa6d0139510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:03.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.408+0000 7fa6d4ef0640 1 --2- 172.21.15.37:0/2353765811 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6d0134b30 0x7fa6d0131460 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:03.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.408+0000 7fa6d4ef0640 1 --2- 172.21.15.37:0/2353765811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6d00a0260 0x7fa6d0130f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:03.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.408+0000 7fa6d4ef0640 1 -- 172.21.15.37:0/2353765811 >> 172.21.15.37:0/2353765811 conn(0x7fa6d00a1ad0 msgr2=0x7fa6d00a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:03.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.408+0000 7fa6d4ef0640 1 -- 172.21.15.37:0/2353765811 shutdown_connections 2024-04-26T00:26:03.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:03.408+0000 7fa6d4ef0640 1 -- 172.21.15.37:0/2353765811 wait complete. 2024-04-26T00:26:06.408 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:26:06.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.556+0000 7f4f4b458640 10 monclient: get_monmap_and_config 2024-04-26T00:26:06.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.556+0000 7f4f4b458640 10 monclient: build_initial_monmap 2024-04-26T00:26:06.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.556+0000 7f4f4b458640 10 monclient: monmap: 2024-04-26T00:26:06.541 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:06.541 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:06.541 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:26:06.561449+0000 2024-04-26T00:26:06.541 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:26:06.561449+0000 2024-04-26T00:26:06.541 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:06.541 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:06.541 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:26:06.541 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:26:06.541 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:26:06.542 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:26:06.542 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:26:06.542 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:06.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.556+0000 7f4f4b458640 1 Processor -- start 2024-04-26T00:26:06.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.556+0000 7f4f4b458640 1 -- start start 2024-04-26T00:26:06.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.556+0000 7f4f4b458640 10 monclient: init 2024-04-26T00:26:06.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.556+0000 7f4f4b458640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:06.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.556+0000 7f4f4b458640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-04-26T00:26:06.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.556+0000 7f4f4b458640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4f44136e10 addr v1:172.21.15.45:6789/0 2024-04-26T00:26:06.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.556+0000 7f4f4b458640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4f44137900 0x7f4f4409c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:06.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.556+0000 7f4f4b458640 10 monclient(hunting): picked mon.noname-c con 0x7f4f44137900 addr v2:172.21.15.45:3300/0 2024-04-26T00:26:06.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.556+0000 7f4f4b458640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4f4413c340 addr v1:172.21.15.37:6789/0 2024-04-26T00:26:06.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.556+0000 7f4f4b458640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f4409b680 con 0x7f4f4413c340 2024-04-26T00:26:06.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.556+0000 7f4f4b458640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f4409bbd0 con 0x7f4f44136e10 2024-04-26T00:26:06.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.556+0000 7f4f4b458640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:06.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.556+0000 7f4f4b458640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f4f4409b8f0 con 0x7f4f44137900 2024-04-26T00:26:06.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.556+0000 7f4f4b458640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:06.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.556+0000 7f4f4b458640 10 monclient(hunting): authenticate will time out at 1347.867839s 2024-04-26T00:26:06.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.560+0000 7f4f499f5640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f4f4413c340 0x7f4f4413e730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:55630/0 (socket says 172.21.15.37:55630) 2024-04-26T00:26:06.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.560+0000 7f4f499f5640 1 -- 172.21.15.37:0/336089379 learned_addr learned my addr 172.21.15.37:0/336089379 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:26:06.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.560+0000 7f4f489f3640 1 --2- 172.21.15.37:0/336089379 >> v2:172.21.15.45:3300/0 conn(0x7f4f44137900 0x7f4f4409c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:06.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.560+0000 7f4f489f3640 10 monclient(hunting): get_auth_request con 0x7f4f44137900 auth_method 0 2024-04-26T00:26:06.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.560+0000 7f4f489f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:06.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.560+0000 7f4f489f3640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:06.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.560+0000 7f4f489f3640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:06.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.560+0000 7f4f489f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:06.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.560+0000 7f4f489f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:06.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.560+0000 7f4f489f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:06.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.560+0000 7f4f489f3640 10 monclient(hunting): handle_auth_done global_id 24568 payload 274 2024-04-26T00:26:06.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.560+0000 7f4f489f3640 1 -- 172.21.15.37:0/336089379 >> v1:172.21.15.45:6789/0 conn(0x7f4f44136e10 legacy=0x7f4f441371f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:26:06.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.560+0000 7f4f489f3640 1 -- 172.21.15.37:0/336089379 >> v1:172.21.15.37:6789/0 conn(0x7f4f4413c340 legacy=0x7f4f4413e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:26:06.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.560+0000 7f4f489f3640 10 monclient: _finish_hunting 0 2024-04-26T00:26:06.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.560+0000 7f4f489f3640 1 monclient: found mon.noname-c 2024-04-26T00:26:06.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.560+0000 7f4f489f3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:06.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.560+0000 7f4f489f3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:26:06.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.560+0000 7f4f489f3640 1 -- 172.21.15.37:0/336089379 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f440a5d90 con 0x7f4f44137900 2024-04-26T00:26:06.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.564+0000 7f4f489f3640 1 -- 172.21.15.37:0/336089379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f44137900 msgr2=0x7f4f4409c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:06.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.564+0000 7f4f489f3640 1 --2- 172.21.15.37:0/336089379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f44137900 0x7f4f4409c5b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4f340097c0 tx=0x7f4f3401c2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14348e1c3bea001e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:06.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.564+0000 7f4f3bfff640 1 -- 172.21.15.37:0/336089379 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f340b0070 con 0x7f4f44137900 2024-04-26T00:26:06.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.564+0000 7f4f3bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:06.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.564+0000 7f4f3bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:26:06.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.564+0000 7f4f3bfff640 10 monclient: dump: 2024-04-26T00:26:06.549 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:06.550 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:06.550 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:06.550 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:06.550 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:06.550 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:06.550 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:06.550 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:06.550 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:06.550 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:06.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.564+0000 7f4f3bfff640 10 monclient: _finish_auth 0 2024-04-26T00:26:06.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.564+0000 7f4f3bfff640 10 monclient: _check_auth_tickets 2024-04-26T00:26:06.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.564+0000 7f4f3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:06.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.564+0000 7f4f3bfff640 1 -- 172.21.15.37:0/336089379 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f340ab600 con 0x7f4f44137900 2024-04-26T00:26:06.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.564+0000 7f4f3bfff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:06.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.564+0000 7f4f3bfff640 1 -- 172.21.15.37:0/336089379 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f340ab9d0 con 0x7f4f44137900 2024-04-26T00:26:06.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.564+0000 7f4f3bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:06.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.564+0000 7f4f3bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:26:06.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.564+0000 7f4f3bfff640 10 monclient: dump: 2024-04-26T00:26:06.551 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:06.551 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:06.551 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:06.551 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:06.551 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:06.552 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:06.552 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:06.552 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:06.552 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:06.552 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:06.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.564+0000 7f4f4b458640 5 monclient: authenticate success, global_id 24568 2024-04-26T00:26:06.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.564+0000 7f4f4b458640 10 monclient: get_monmap_and_config success 2024-04-26T00:26:06.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.564+0000 7f4f4b458640 4 set_mon_vals no callback set 2024-04-26T00:26:06.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.564+0000 7f4f4b458640 10 monclient: shutdown 2024-04-26T00:26:06.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.564+0000 7f4f4b458640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:06.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.564+0000 7f4f4b458640 1 -- 172.21.15.37:0/336089379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f44137900 msgr2=0x7f4f4409c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:06.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.564+0000 7f4f4b458640 1 --2- 172.21.15.37:0/336089379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f44137900 0x7f4f4409c5b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4f340097c0 tx=0x7f4f3401c2e0 comp rx=0 tx=0).stop 2024-04-26T00:26:06.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.564+0000 7f4f4b458640 1 -- 172.21.15.37:0/336089379 shutdown_connections 2024-04-26T00:26:06.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.564+0000 7f4f4b458640 1 --2- 172.21.15.37:0/336089379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f44137900 0x7f4f4409c5b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:06.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.564+0000 7f4f4b458640 1 -- 172.21.15.37:0/336089379 >> 172.21.15.37:0/336089379 conn(0x7f4f440a64e0 msgr2=0x7f4f440a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:06.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.564+0000 7f4f4b458640 1 -- 172.21.15.37:0/336089379 shutdown_connections 2024-04-26T00:26:06.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.564+0000 7f4f4b458640 1 -- 172.21.15.37:0/336089379 wait complete. 2024-04-26T00:26:06.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f4b458640 10 monclient: build_initial_monmap 2024-04-26T00:26:06.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f4b458640 10 monclient: monmap: 2024-04-26T00:26:06.558 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:06.558 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:26:06.558 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:26:06.558 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:26:06.558 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:06.558 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:26:06.558 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:26:06.559 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:26:06.559 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:26:06.559 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:06.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f4b458640 1 Processor -- start 2024-04-26T00:26:06.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f4b458640 1 -- start start 2024-04-26T00:26:06.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f4b458640 10 monclient: init 2024-04-26T00:26:06.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f4b458640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:06.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f4b458640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:26:06.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f4b458640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f44136e10 0x7f4f441ced70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:06.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f4b458640 10 monclient(hunting): picked mon.noname-a con 0x7f4f44136e10 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:26:06.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f4b458640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f44137900 0x7f4f44135d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:06.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f4b458640 10 monclient(hunting): picked mon.noname-b con 0x7f4f44137900 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:26:06.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f4b458640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f4413c340 0x7f4f44137e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:06.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f4b458640 10 monclient(hunting): picked mon.noname-c con 0x7f4f4413c340 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:26:06.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f4b458640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:06.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f4b458640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f4f44142a00 con 0x7f4f44136e10 2024-04-26T00:26:06.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f4b458640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:06.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f4b458640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4f44142880 con 0x7f4f44137900 2024-04-26T00:26:06.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f4b458640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:06.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f4b458640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4f44142b80 con 0x7f4f4413c340 2024-04-26T00:26:06.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f4b458640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:06.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f4b458640 10 monclient(hunting): authenticate will time out at 1347.877328s 2024-04-26T00:26:06.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f499f5640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f4413c340 msgr2=0x7f4f44137e20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:26:06.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f499f5640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f4413c340 0x7f4f44137e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:26:06.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f491f4640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f44136e10 0x7f4f441ced70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:06.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f491f4640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f44136e10 0x7f4f441ced70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44792/0 (socket says 172.21.15.37:44792) 2024-04-26T00:26:06.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f491f4640 1 -- 172.21.15.37:0/4282969285 learned_addr learned my addr 172.21.15.37:0/4282969285 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:26:06.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f491f4640 10 monclient(hunting): get_auth_request con 0x7f4f44136e10 auth_method 0 2024-04-26T00:26:06.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f491f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:06.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f491f4640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:06.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f491f4640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:06.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f491f4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:06.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f491f4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:06.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f491f4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:06.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f489f3640 1 --2- 172.21.15.37:0/4282969285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f44137900 0x7f4f44135d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:06.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f489f3640 10 monclient(hunting): get_auth_request con 0x7f4f44137900 auth_method 0 2024-04-26T00:26:06.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f489f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:06.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f489f3640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:06.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f489f3640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:06.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f491f4640 10 monclient(hunting): handle_auth_done global_id 54336 payload 824 2024-04-26T00:26:06.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f491f4640 1 -- 172.21.15.37:0/4282969285 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f4413c340 msgr2=0x7f4f44137e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:26:06.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f491f4640 1 --2- 172.21.15.37:0/4282969285 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f4413c340 0x7f4f44137e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:06.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f491f4640 1 -- 172.21.15.37:0/4282969285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f44137900 msgr2=0x7f4f44135d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:06.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f491f4640 1 --2- 172.21.15.37:0/4282969285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f44137900 0x7f4f44135d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:06.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f491f4640 10 monclient: _finish_hunting 0 2024-04-26T00:26:06.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f491f4640 1 monclient: found mon.noname-a 2024-04-26T00:26:06.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f491f4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:06.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f491f4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:26:06.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.568+0000 7f4f491f4640 1 -- 172.21.15.37:0/4282969285 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f4413b260 con 0x7f4f44136e10 2024-04-26T00:26:06.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.572+0000 7f4f491f4640 1 -- 172.21.15.37:0/4282969285 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f44136e10 msgr2=0x7f4f441ced70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:06.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.572+0000 7f4f491f4640 1 --2- 172.21.15.37:0/4282969285 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f44136e10 0x7f4f441ced70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4f4000c480 tx=0x7f4f4000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:06.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.572+0000 7f4f39ffb640 1 -- 172.21.15.37:0/4282969285 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f40012a40 con 0x7f4f44136e10 2024-04-26T00:26:06.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.572+0000 7f4f39ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:06.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.572+0000 7f4f39ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:26:06.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.572+0000 7f4f39ffb640 10 monclient: dump: 2024-04-26T00:26:06.563 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:06.563 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:06.563 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:06.563 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:06.563 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:06.563 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:06.563 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:06.563 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:06.563 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:06.563 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:06.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.572+0000 7f4f39ffb640 10 monclient: _finish_auth 0 2024-04-26T00:26:06.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.572+0000 7f4f39ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:26:06.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.572+0000 7f4f39ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:06.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.572+0000 7f4f4b458640 5 monclient: authenticate success, global_id 54336 2024-04-26T00:26:06.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.572+0000 7f4f4b458640 10 monclient: _renew_subs 2024-04-26T00:26:06.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.572+0000 7f4f4b458640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:26:06.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.572+0000 7f4f4b458640 1 -- 172.21.15.37:0/4282969285 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f44139810 con 0x7f4f44136e10 2024-04-26T00:26:06.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.572+0000 7f4f4b458640 10 monclient: _renew_subs 2024-04-26T00:26:06.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.572+0000 7f4f4b458640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:26:06.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.572+0000 7f4f4b458640 1 -- 172.21.15.37:0/4282969285 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f441cfbf0 con 0x7f4f44136e10 2024-04-26T00:26:06.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.572+0000 7f4f39ffb640 1 -- 172.21.15.37:0/4282969285 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f40017070 con 0x7f4f44136e10 2024-04-26T00:26:06.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.572+0000 7f4f39ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:06.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.572+0000 7f4f39ffb640 1 -- 172.21.15.37:0/4282969285 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f40012e40 con 0x7f4f44136e10 2024-04-26T00:26:06.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.572+0000 7f4f39ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:06.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.572+0000 7f4f39ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:26:06.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.572+0000 7f4f39ffb640 10 monclient: dump: 2024-04-26T00:26:06.564 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:06.564 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:06.564 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:06.565 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:06.565 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:06.565 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:06.565 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:06.565 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:06.565 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:06.565 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:06.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.572+0000 7f4f3b7fe640 4 set_mon_vals no callback set 2024-04-26T00:26:06.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.572+0000 7f4f4b458640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:06.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.572+0000 7f4f4b458640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:06.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.572+0000 7f4f4b458640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:26:06.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.572+0000 7f4f4b458640 1 -- 172.21.15.37:0/4282969285 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f44075910 con 0x7f4f44136e10 2024-04-26T00:26:06.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.572+0000 7f4f39ffb640 1 -- 172.21.15.37:0/4282969285 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f4f4001c460 con 0x7f4f44136e10 2024-04-26T00:26:06.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.572+0000 7f4f39ffb640 1 --2- 172.21.15.37:0/4282969285 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4f28035e30 0x7f4f280382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:06.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.572+0000 7f4f39ffb640 1 -- 172.21.15.37:0/4282969285 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(236..236 src has 177..236) ==== 3947+0+0 (secure 0 0 0) 0x7f4f400488b0 con 0x7f4f44136e10 2024-04-26T00:26:06.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.572+0000 7f4f489f3640 1 --2- 172.21.15.37:0/4282969285 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4f28035e30 0x7f4f280382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:06.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.572+0000 7f4f489f3640 10 monclient: get_auth_request con 0x7f4f28035e30 auth_method 0 2024-04-26T00:26:06.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.576+0000 7f4f489f3640 1 --2- 172.21.15.37:0/4282969285 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4f28035e30 0x7f4f280382f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4f340ab000 tx=0x7f4f34008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:06.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.580+0000 7f4f39ffb640 1 -- 172.21.15.37:0/4282969285 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f4f4413b260 con 0x7f4f44136e10 2024-04-26T00:26:06.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.580+0000 7f4f39ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:06.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.580+0000 7f4f39ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:26:06.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.808+0000 7f4f4b458640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:26:06.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.808+0000 7f4f4b458640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:26:06.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.808+0000 7f4f4b458640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:26:06.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.808+0000 7f4f4b458640 1 -- 172.21.15.37:0/4282969285 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4f44066f40 con 0x7f4f44136e10 2024-04-26T00:26:06.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.808+0000 7f4f39ffb640 1 -- 172.21.15.37:0/4282969285 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f4f4001b3c0 con 0x7f4f44136e10 2024-04-26T00:26:06.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.808+0000 7f4f39ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:26:06.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.808+0000 7f4f39ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:26:06.793 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":46,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:26:06.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.812+0000 7f4f4b458640 1 -- 172.21.15.37:0/4282969285 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4f28035e30 msgr2=0x7f4f280382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:06.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.812+0000 7f4f4b458640 1 --2- 172.21.15.37:0/4282969285 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4f28035e30 0x7f4f280382f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4f340ab000 tx=0x7f4f34008040 comp rx=0 tx=0).stop 2024-04-26T00:26:06.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.812+0000 7f4f4b458640 10 monclient: shutdown 2024-04-26T00:26:06.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.812+0000 7f4f4b458640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:06.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.812+0000 7f4f4b458640 1 -- 172.21.15.37:0/4282969285 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f44136e10 msgr2=0x7f4f441ced70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:06.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.812+0000 7f4f4b458640 1 --2- 172.21.15.37:0/4282969285 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f44136e10 0x7f4f441ced70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4f4000c480 tx=0x7f4f4000c940 comp rx=0 tx=0).stop 2024-04-26T00:26:06.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.812+0000 7f4f4b458640 1 -- 172.21.15.37:0/4282969285 shutdown_connections 2024-04-26T00:26:06.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.812+0000 7f4f4b458640 1 --2- 172.21.15.37:0/4282969285 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4f28035e30 0x7f4f280382f0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:06.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.812+0000 7f4f4b458640 1 --2- 172.21.15.37:0/4282969285 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f4413c340 0x7f4f44137e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:06.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.812+0000 7f4f4b458640 1 --2- 172.21.15.37:0/4282969285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f44137900 0x7f4f44135d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:06.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.812+0000 7f4f4b458640 1 --2- 172.21.15.37:0/4282969285 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f44136e10 0x7f4f441ced70 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:06.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.812+0000 7f4f4b458640 1 -- 172.21.15.37:0/4282969285 >> 172.21.15.37:0/4282969285 conn(0x7f4f440a64e0 msgr2=0x7f4f440a73e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:06.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.812+0000 7f4f4b458640 1 -- 172.21.15.37:0/4282969285 shutdown_connections 2024-04-26T00:26:06.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:06.812+0000 7f4f4b458640 1 -- 172.21.15.37:0/4282969285 wait complete. 2024-04-26T00:26:09.814 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:26:09.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:09.960+0000 7fc12a126640 10 monclient: get_monmap_and_config 2024-04-26T00:26:09.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:09.960+0000 7fc12a126640 10 monclient: build_initial_monmap 2024-04-26T00:26:09.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:09.960+0000 7fc12a126640 10 monclient: monmap: 2024-04-26T00:26:09.945 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:09.945 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:09.945 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:26:09.965452+0000 2024-04-26T00:26:09.946 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:26:09.965452+0000 2024-04-26T00:26:09.946 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:09.946 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:09.946 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:26:09.946 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:26:09.947 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:26:09.947 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:26:09.947 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:26:09.947 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:09.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:09.960+0000 7fc12a126640 1 Processor -- start 2024-04-26T00:26:09.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:09.960+0000 7fc12a126640 1 -- start start 2024-04-26T00:26:09.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:09.960+0000 7fc12a126640 10 monclient: init 2024-04-26T00:26:09.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:09.960+0000 7fc12a126640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:09.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:09.960+0000 7fc12a126640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-04-26T00:26:09.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:09.960+0000 7fc12a126640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc124136da0 addr v1:172.21.15.45:6789/0 2024-04-26T00:26:09.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:09.960+0000 7fc12a126640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc124137180 0x7fc1241375c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:09.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:09.960+0000 7fc12a126640 10 monclient(hunting): picked mon.noname-c con 0x7fc124137180 addr v2:172.21.15.45:3300/0 2024-04-26T00:26:09.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:09.960+0000 7fc12a126640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc12413c2d0 addr v1:172.21.15.37:6789/0 2024-04-26T00:26:09.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:09.960+0000 7fc12a126640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc12409b4f0 con 0x7fc12413c2d0 2024-04-26T00:26:09.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:09.960+0000 7fc12a126640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc12409b7d0 con 0x7fc124136da0 2024-04-26T00:26:09.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:09.960+0000 7fc12a126640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:09.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:09.960+0000 7fc12a126640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc12409b210 con 0x7fc124137180 2024-04-26T00:26:09.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:09.960+0000 7fc12a126640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:09.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:09.960+0000 7fc12a126640 10 monclient(hunting): authenticate will time out at 1351.271300s 2024-04-26T00:26:09.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:09.960+0000 7fc123fff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fc12413c2d0 0x7fc12413e6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:49884/0 (socket says 172.21.15.37:49884) 2024-04-26T00:26:09.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:09.960+0000 7fc123fff640 1 -- 172.21.15.37:0/3219831037 learned_addr learned my addr 172.21.15.37:0/3219831037 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:26:09.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:09.960+0000 7fc122ffd640 1 --2- 172.21.15.37:0/3219831037 >> v2:172.21.15.45:3300/0 conn(0x7fc124137180 0x7fc1241375c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:09.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:09.960+0000 7fc122ffd640 10 monclient(hunting): get_auth_request con 0x7fc124137180 auth_method 0 2024-04-26T00:26:09.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:09.960+0000 7fc122ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:09.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:09.960+0000 7fc122ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:09.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:09.960+0000 7fc122ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:09.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:09.960+0000 7fc122ffd640 1 -- 172.21.15.37:0/3219831037 >> v2:172.21.15.45:3300/0 conn(0x7fc124137180 msgr2=0x7fc1241375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:26:09.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:09.960+0000 7fc122ffd640 1 -- 172.21.15.37:0/3219831037 >> v2:172.21.15.45:3300/0 conn(0x7fc124137180 msgr2=0x7fc1241375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:26:09.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:09.960+0000 7fc122ffd640 1 --2- 172.21.15.37:0/3219831037 >> v2:172.21.15.45:3300/0 conn(0x7fc124137180 0x7fc1241375c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:26:09.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:09.960+0000 7fc122ffd640 1 --2- 172.21.15.37:0/3219831037 >> v2:172.21.15.45:3300/0 conn(0x7fc124137180 0x7fc1241375c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:26:09.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:09.964+0000 7fc123fff640 1 -- 172.21.15.37:0/3219831037 >> v1:172.21.15.37:6789/0 conn(0x7fc12413c2d0 legacy=0x7fc12413e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:09.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:09.964+0000 7fc1237fe640 1 -- 172.21.15.37:0/3219831037 >> v1:172.21.15.45:6789/0 conn(0x7fc124136da0 legacy=0x7fc12409bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:10.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:10.164+0000 7fc122ffd640 1 --2- 172.21.15.37:0/3219831037 >> v2:172.21.15.45:3300/0 conn(0x7fc124137180 0x7fc1241375c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:10.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:10.164+0000 7fc122ffd640 10 monclient(hunting): get_auth_request con 0x7fc124137180 auth_method 0 2024-04-26T00:26:10.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:10.164+0000 7fc122ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:10.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:10.164+0000 7fc122ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:10.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:10.164+0000 7fc122ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:26:10.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:10.164+0000 7fc122ffd640 1 -- 172.21.15.37:0/3219831037 >> v2:172.21.15.45:3300/0 conn(0x7fc124137180 msgr2=0x7fc1241375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:26:10.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:10.164+0000 7fc122ffd640 1 -- 172.21.15.37:0/3219831037 >> v2:172.21.15.45:3300/0 conn(0x7fc124137180 msgr2=0x7fc1241375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:26:10.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:10.164+0000 7fc122ffd640 1 --2- 172.21.15.37:0/3219831037 >> v2:172.21.15.45:3300/0 conn(0x7fc124137180 0x7fc1241375c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:26:10.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:10.164+0000 7fc122ffd640 1 --2- 172.21.15.37:0/3219831037 >> v2:172.21.15.45:3300/0 conn(0x7fc124137180 0x7fc1241375c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:26:10.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:10.568+0000 7fc122ffd640 1 --2- 172.21.15.37:0/3219831037 >> v2:172.21.15.45:3300/0 conn(0x7fc124137180 0x7fc1241375c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:10.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:10.568+0000 7fc122ffd640 10 monclient(hunting): get_auth_request con 0x7fc124137180 auth_method 0 2024-04-26T00:26:10.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:10.568+0000 7fc122ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:10.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:10.568+0000 7fc122ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:10.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:10.568+0000 7fc122ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:26:10.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:10.568+0000 7fc122ffd640 1 -- 172.21.15.37:0/3219831037 >> v2:172.21.15.45:3300/0 conn(0x7fc124137180 msgr2=0x7fc1241375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:26:10.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:10.568+0000 7fc122ffd640 1 -- 172.21.15.37:0/3219831037 >> v2:172.21.15.45:3300/0 conn(0x7fc124137180 msgr2=0x7fc1241375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:26:10.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:10.568+0000 7fc122ffd640 1 --2- 172.21.15.37:0/3219831037 >> v2:172.21.15.45:3300/0 conn(0x7fc124137180 0x7fc1241375c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:26:10.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:10.568+0000 7fc122ffd640 1 --2- 172.21.15.37:0/3219831037 >> v2:172.21.15.45:3300/0 conn(0x7fc124137180 0x7fc1241375c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:26:11.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:11.372+0000 7fc122ffd640 1 --2- 172.21.15.37:0/3219831037 >> v2:172.21.15.45:3300/0 conn(0x7fc124137180 0x7fc1241375c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:11.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:11.372+0000 7fc122ffd640 10 monclient(hunting): get_auth_request con 0x7fc124137180 auth_method 0 2024-04-26T00:26:11.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:11.372+0000 7fc122ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:11.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:11.372+0000 7fc122ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:11.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:11.372+0000 7fc122ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:26:11.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:11.372+0000 7fc122ffd640 1 -- 172.21.15.37:0/3219831037 >> v2:172.21.15.45:3300/0 conn(0x7fc124137180 msgr2=0x7fc1241375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:26:11.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:11.372+0000 7fc122ffd640 1 -- 172.21.15.37:0/3219831037 >> v2:172.21.15.45:3300/0 conn(0x7fc124137180 msgr2=0x7fc1241375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:26:11.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:11.372+0000 7fc122ffd640 1 --2- 172.21.15.37:0/3219831037 >> v2:172.21.15.45:3300/0 conn(0x7fc124137180 0x7fc1241375c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:26:11.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:11.372+0000 7fc122ffd640 1 --2- 172.21.15.37:0/3219831037 >> v2:172.21.15.45:3300/0 conn(0x7fc124137180 0x7fc1241375c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:26:12.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:12.960+0000 7fc1217fa640 10 monclient(hunting): tick 2024-04-26T00:26:12.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:12.960+0000 7fc1217fa640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:26:12.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:12.960+0000 7fc1217fa640 1 monclient(hunting): continuing hunt 2024-04-26T00:26:12.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:12.960+0000 7fc1217fa640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:26:12.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:12.960+0000 7fc1217fa640 1 -- 172.21.15.37:0/3219831037 >> v2:172.21.15.45:3300/0 conn(0x7fc124137180 msgr2=0x7fc1241375c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:26:12.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:12.960+0000 7fc1217fa640 1 --2- 172.21.15.37:0/3219831037 >> v2:172.21.15.45:3300/0 conn(0x7fc124137180 0x7fc1241375c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:12.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:12.960+0000 7fc1217fa640 1 -- 172.21.15.37:0/3219831037 >> v1:172.21.15.45:6789/0 conn(0x7fc124136da0 legacy=0x7fc12409bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:12.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:12.960+0000 7fc1217fa640 1 -- 172.21.15.37:0/3219831037 >> v1:172.21.15.37:6789/0 conn(0x7fc12413c2d0 legacy=0x7fc12413e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:12.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:12.960+0000 7fc1217fa640 10 monclient: _add_conns ranks=[2,3] 2024-04-26T00:26:12.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:12.960+0000 7fc1217fa640 1 --2- 172.21.15.37:0/3219831037 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc110003840 0x7fc110005c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:12.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:12.960+0000 7fc1217fa640 10 monclient(hunting): picked mon.noname-b con 0x7fc110003840 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:26:12.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:12.960+0000 7fc1217fa640 1 --2- 172.21.15.37:0/3219831037 >> v2:172.21.15.37:3300/0 conn(0x7fc1100063a0 0x7fc110008860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:12.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:12.960+0000 7fc1217fa640 10 monclient(hunting): picked mon.noname-a con 0x7fc1100063a0 addr v2:172.21.15.37:3300/0 2024-04-26T00:26:12.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:12.960+0000 7fc1217fa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:12.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:12.960+0000 7fc1217fa640 1 -- 172.21.15.37:0/3219831037 --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fc12409b210 con 0x7fc1100063a0 2024-04-26T00:26:12.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:12.960+0000 7fc1217fa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:12.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:12.960+0000 7fc1217fa640 1 -- 172.21.15.37:0/3219831037 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc110008f20 con 0x7fc110003840 2024-04-26T00:26:12.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:12.960+0000 7fc1217fa640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:12.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:12.960+0000 7fc1237fe640 1 -- 172.21.15.37:0/3219831037 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc110003840 msgr2=0x7fc110005c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:26:12.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:12.960+0000 7fc1237fe640 1 --2- 172.21.15.37:0/3219831037 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc110003840 0x7fc110005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:26:12.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:12.960+0000 7fc122ffd640 1 --2- 172.21.15.37:0/3219831037 >> v2:172.21.15.37:3300/0 conn(0x7fc1100063a0 0x7fc110008860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:12.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:12.960+0000 7fc122ffd640 10 monclient(hunting): get_auth_request con 0x7fc1100063a0 auth_method 0 2024-04-26T00:26:12.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:12.960+0000 7fc122ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:12.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:12.960+0000 7fc122ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:12.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:12.960+0000 7fc122ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:12.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:12.964+0000 7fc122ffd640 1 -- 172.21.15.37:0/3219831037 >> v2:172.21.15.37:3300/0 conn(0x7fc1100063a0 msgr2=0x7fc110008860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:26:12.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:12.964+0000 7fc122ffd640 1 -- 172.21.15.37:0/3219831037 >> v2:172.21.15.37:3300/0 conn(0x7fc1100063a0 msgr2=0x7fc110008860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:26:12.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:12.964+0000 7fc122ffd640 1 --2- 172.21.15.37:0/3219831037 >> v2:172.21.15.37:3300/0 conn(0x7fc1100063a0 0x7fc110008860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:26:12.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:12.964+0000 7fc122ffd640 1 --2- 172.21.15.37:0/3219831037 >> v2:172.21.15.37:3300/0 conn(0x7fc1100063a0 0x7fc110008860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:26:13.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:13.164+0000 7fc1237fe640 1 -- 172.21.15.37:0/3219831037 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc110003840 msgr2=0x7fc110005c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:26:13.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:13.164+0000 7fc1237fe640 1 --2- 172.21.15.37:0/3219831037 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc110003840 0x7fc110005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:26:13.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:13.168+0000 7fc122ffd640 1 --2- 172.21.15.37:0/3219831037 >> v2:172.21.15.37:3300/0 conn(0x7fc1100063a0 0x7fc110008860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:13.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:13.168+0000 7fc122ffd640 10 monclient(hunting): get_auth_request con 0x7fc1100063a0 auth_method 0 2024-04-26T00:26:13.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:13.168+0000 7fc122ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:13.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:13.168+0000 7fc122ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:13.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:13.168+0000 7fc122ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:26:13.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:13.168+0000 7fc122ffd640 1 -- 172.21.15.37:0/3219831037 >> v2:172.21.15.37:3300/0 conn(0x7fc1100063a0 msgr2=0x7fc110008860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:26:13.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:13.168+0000 7fc122ffd640 1 -- 172.21.15.37:0/3219831037 >> v2:172.21.15.37:3300/0 conn(0x7fc1100063a0 msgr2=0x7fc110008860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:26:13.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:13.168+0000 7fc122ffd640 1 --2- 172.21.15.37:0/3219831037 >> v2:172.21.15.37:3300/0 conn(0x7fc1100063a0 0x7fc110008860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:26:13.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:13.168+0000 7fc122ffd640 1 --2- 172.21.15.37:0/3219831037 >> v2:172.21.15.37:3300/0 conn(0x7fc1100063a0 0x7fc110008860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:26:13.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:13.568+0000 7fc1237fe640 1 -- 172.21.15.37:0/3219831037 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc110003840 msgr2=0x7fc110005c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:26:13.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:13.568+0000 7fc1237fe640 1 --2- 172.21.15.37:0/3219831037 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc110003840 0x7fc110005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:26:13.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:13.572+0000 7fc122ffd640 1 --2- 172.21.15.37:0/3219831037 >> v2:172.21.15.37:3300/0 conn(0x7fc1100063a0 0x7fc110008860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:13.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:13.572+0000 7fc122ffd640 10 monclient(hunting): get_auth_request con 0x7fc1100063a0 auth_method 0 2024-04-26T00:26:13.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:13.572+0000 7fc122ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:13.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:13.572+0000 7fc122ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:13.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:13.572+0000 7fc122ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:26:13.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:13.572+0000 7fc122ffd640 1 -- 172.21.15.37:0/3219831037 >> v2:172.21.15.37:3300/0 conn(0x7fc1100063a0 msgr2=0x7fc110008860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:26:13.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:13.572+0000 7fc122ffd640 1 -- 172.21.15.37:0/3219831037 >> v2:172.21.15.37:3300/0 conn(0x7fc1100063a0 msgr2=0x7fc110008860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:26:13.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:13.572+0000 7fc122ffd640 1 --2- 172.21.15.37:0/3219831037 >> v2:172.21.15.37:3300/0 conn(0x7fc1100063a0 0x7fc110008860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:26:13.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:13.572+0000 7fc122ffd640 1 --2- 172.21.15.37:0/3219831037 >> v2:172.21.15.37:3300/0 conn(0x7fc1100063a0 0x7fc110008860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:26:14.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.372+0000 7fc1237fe640 1 -- 172.21.15.37:0/3219831037 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc110003840 msgr2=0x7fc110005c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:26:14.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.372+0000 7fc1237fe640 1 --2- 172.21.15.37:0/3219831037 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc110003840 0x7fc110005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:26:14.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.376+0000 7fc122ffd640 1 --2- 172.21.15.37:0/3219831037 >> v2:172.21.15.37:3300/0 conn(0x7fc1100063a0 0x7fc110008860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:14.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.376+0000 7fc122ffd640 10 monclient(hunting): get_auth_request con 0x7fc1100063a0 auth_method 0 2024-04-26T00:26:14.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.376+0000 7fc122ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:14.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.376+0000 7fc122ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:14.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.376+0000 7fc122ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:26:14.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.376+0000 7fc122ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:14.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.376+0000 7fc122ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:14.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.376+0000 7fc122ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:14.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.376+0000 7fc122ffd640 10 monclient(hunting): handle_auth_done global_id 54339 payload 274 2024-04-26T00:26:14.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.376+0000 7fc122ffd640 1 -- 172.21.15.37:0/3219831037 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc110003840 msgr2=0x7fc110005c70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:26:14.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.376+0000 7fc122ffd640 1 --2- 172.21.15.37:0/3219831037 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc110003840 0x7fc110005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:14.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.376+0000 7fc122ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:26:14.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.376+0000 7fc122ffd640 1 monclient: found mon.noname-a 2024-04-26T00:26:14.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.376+0000 7fc122ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:14.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.376+0000 7fc122ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:26:14.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.376+0000 7fc122ffd640 1 -- 172.21.15.37:0/3219831037 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc124137b00 con 0x7fc1100063a0 2024-04-26T00:26:14.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.384+0000 7fc122ffd640 1 -- 172.21.15.37:0/3219831037 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc1100063a0 msgr2=0x7fc110008860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:14.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.384+0000 7fc122ffd640 1 --2- 172.21.15.37:0/3219831037 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc1100063a0 0x7fc110008860 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc118000ee0 tx=0x7fc11801d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f91eea8254f462e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:14.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.384+0000 7fc1227fc640 1 -- 172.21.15.37:0/3219831037 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc11800f040 con 0x7fc1100063a0 2024-04-26T00:26:14.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.384+0000 7fc1227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:14.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.384+0000 7fc1227fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:26:14.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.384+0000 7fc1227fc640 10 monclient: dump: 2024-04-26T00:26:14.373 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:14.373 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:14.373 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:14.373 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:14.373 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:14.374 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:14.374 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:14.374 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:14.374 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:14.375 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:14.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.384+0000 7fc1227fc640 10 monclient: _finish_auth 0 2024-04-26T00:26:14.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.384+0000 7fc1227fc640 10 monclient: _check_auth_tickets 2024-04-26T00:26:14.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.384+0000 7fc1227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:14.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.384+0000 7fc1227fc640 1 -- 172.21.15.37:0/3219831037 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1180a6a30 con 0x7fc1100063a0 2024-04-26T00:26:14.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.384+0000 7fc1227fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:14.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.384+0000 7fc1227fc640 1 -- 172.21.15.37:0/3219831037 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1180a6de0 con 0x7fc1100063a0 2024-04-26T00:26:14.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.384+0000 7fc12a126640 5 monclient: authenticate success, global_id 54339 2024-04-26T00:26:14.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.384+0000 7fc12a126640 10 monclient: get_monmap_and_config success 2024-04-26T00:26:14.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.384+0000 7fc12a126640 4 set_mon_vals no callback set 2024-04-26T00:26:14.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.384+0000 7fc1227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:14.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.384+0000 7fc1227fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:26:14.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.384+0000 7fc1227fc640 10 monclient: dump: 2024-04-26T00:26:14.377 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:14.377 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:14.377 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:14.377 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:14.378 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:14.378 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:14.378 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:14.378 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:14.378 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:14.379 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:14.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.384+0000 7fc12a126640 10 monclient: shutdown 2024-04-26T00:26:14.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.384+0000 7fc12a126640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:14.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.384+0000 7fc12a126640 1 -- 172.21.15.37:0/3219831037 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc1100063a0 msgr2=0x7fc110008860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:14.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.384+0000 7fc12a126640 1 --2- 172.21.15.37:0/3219831037 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc1100063a0 0x7fc110008860 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc118000ee0 tx=0x7fc11801d9d0 comp rx=0 tx=0).stop 2024-04-26T00:26:14.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.384+0000 7fc123fff640 1 -- 172.21.15.37:0/3219831037 reap_dead start 2024-04-26T00:26:14.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.384+0000 7fc12a126640 1 -- 172.21.15.37:0/3219831037 shutdown_connections 2024-04-26T00:26:14.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.384+0000 7fc12a126640 1 --2- 172.21.15.37:0/3219831037 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc1100063a0 0x7fc110008860 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:14.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.384+0000 7fc12a126640 1 -- 172.21.15.37:0/3219831037 >> 172.21.15.37:0/3219831037 conn(0x7fc1240a5f00 msgr2=0x7fc1240a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:14.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.384+0000 7fc12a126640 1 -- 172.21.15.37:0/3219831037 shutdown_connections 2024-04-26T00:26:14.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.384+0000 7fc12a126640 1 -- 172.21.15.37:0/3219831037 wait complete. 2024-04-26T00:26:14.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc12a126640 10 monclient: build_initial_monmap 2024-04-26T00:26:14.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc12a126640 10 monclient: monmap: 2024-04-26T00:26:14.381 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:14.381 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:26:14.381 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:26:14.381 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:26:14.381 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:14.382 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:26:14.382 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:26:14.382 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:26:14.382 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:26:14.382 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:14.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc12a126640 1 Processor -- start 2024-04-26T00:26:14.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc12a126640 1 -- start start 2024-04-26T00:26:14.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc12a126640 10 monclient: init 2024-04-26T00:26:14.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc12a126640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:14.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc12a126640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:26:14.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc12a126640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc1100063a0 0x7fc1241cf300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:14.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc12a126640 10 monclient(hunting): picked mon.noname-c con 0x7fc1100063a0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:26:14.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc12a126640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1241cf840 0x7fc124137d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:14.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc12a126640 10 monclient(hunting): picked mon.noname-b con 0x7fc1241cf840 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:26:14.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc12a126640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc1241382d0 0x7fc124138760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:14.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc12a126640 10 monclient(hunting): picked mon.noname-a con 0x7fc1241382d0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:26:14.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc12a126640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:14.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc12a126640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fc124142990 con 0x7fc1241382d0 2024-04-26T00:26:14.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc12a126640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:14.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc12a126640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc124142810 con 0x7fc1241cf840 2024-04-26T00:26:14.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc12a126640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:14.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc12a126640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc124142b10 con 0x7fc1100063a0 2024-04-26T00:26:14.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc12a126640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:14.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc1237fe640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc1100063a0 msgr2=0x7fc1241cf300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:26:14.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc12a126640 10 monclient(hunting): authenticate will time out at 1355.698893s 2024-04-26T00:26:14.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc1237fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc1100063a0 0x7fc1241cf300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:26:14.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc123fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc1241382d0 0x7fc124138760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:14.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc123fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc1241382d0 0x7fc124138760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52218/0 (socket says 172.21.15.37:52218) 2024-04-26T00:26:14.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc123fff640 1 -- 172.21.15.37:0/1175701006 learned_addr learned my addr 172.21.15.37:0/1175701006 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:26:14.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc123fff640 10 monclient(hunting): get_auth_request con 0x7fc1241382d0 auth_method 0 2024-04-26T00:26:14.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc123fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:14.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc123fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:14.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc123fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:14.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc123fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:14.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc123fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:14.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc123fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:14.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc122ffd640 1 --2- 172.21.15.37:0/1175701006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1241cf840 0x7fc124137d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:14.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc122ffd640 10 monclient(hunting): get_auth_request con 0x7fc1241cf840 auth_method 0 2024-04-26T00:26:14.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc122ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:14.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc122ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:14.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc122ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:14.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc122ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:14.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc122ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:14.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc122ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:14.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc123fff640 10 monclient(hunting): handle_auth_done global_id 54342 payload 824 2024-04-26T00:26:14.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc123fff640 1 -- 172.21.15.37:0/1175701006 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc1100063a0 msgr2=0x7fc1241cf300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:26:14.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc123fff640 1 --2- 172.21.15.37:0/1175701006 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc1100063a0 0x7fc1241cf300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:14.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc123fff640 1 -- 172.21.15.37:0/1175701006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1241cf840 msgr2=0x7fc124137d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:14.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc123fff640 1 --2- 172.21.15.37:0/1175701006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1241cf840 0x7fc124137d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:14.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc123fff640 10 monclient: _finish_hunting 0 2024-04-26T00:26:14.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc123fff640 1 monclient: found mon.noname-a 2024-04-26T00:26:14.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc123fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:14.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc123fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:26:14.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.388+0000 7fc123fff640 1 -- 172.21.15.37:0/1175701006 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc12413b890 con 0x7fc1241382d0 2024-04-26T00:26:14.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.392+0000 7fc123fff640 1 -- 172.21.15.37:0/1175701006 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc1241382d0 msgr2=0x7fc124138760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:14.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.392+0000 7fc123fff640 1 --2- 172.21.15.37:0/1175701006 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc1241382d0 0x7fc124138760 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc124076070 tx=0x7fc11c00ce10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:14.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.392+0000 7fc10bfff640 1 -- 172.21.15.37:0/1175701006 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc11c017070 con 0x7fc1241382d0 2024-04-26T00:26:14.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.392+0000 7fc10bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:14.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.392+0000 7fc10bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:26:14.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.392+0000 7fc10bfff640 10 monclient: dump: 2024-04-26T00:26:14.403 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:14.403 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:14.403 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:14.404 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:14.404 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:14.404 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:14.404 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:14.404 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:14.404 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:14.405 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:14.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.392+0000 7fc10bfff640 10 monclient: _finish_auth 0 2024-04-26T00:26:14.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.392+0000 7fc10bfff640 10 monclient: _check_auth_tickets 2024-04-26T00:26:14.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.392+0000 7fc10bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:14.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.392+0000 7fc10bfff640 1 -- 172.21.15.37:0/1175701006 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc11c00ac70 con 0x7fc1241382d0 2024-04-26T00:26:14.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.392+0000 7fc10bfff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:14.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.392+0000 7fc10bfff640 1 -- 172.21.15.37:0/1175701006 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc11c012400 con 0x7fc1241382d0 2024-04-26T00:26:14.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.392+0000 7fc10bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:14.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.392+0000 7fc10bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:26:14.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.392+0000 7fc10bfff640 10 monclient: dump: 2024-04-26T00:26:14.406 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:14.407 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:14.407 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:14.407 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:14.407 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:14.407 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:14.407 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:14.408 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:14.408 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:14.408 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:14.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.392+0000 7fc12a126640 5 monclient: authenticate success, global_id 54342 2024-04-26T00:26:14.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.392+0000 7fc121ffb640 4 set_mon_vals no callback set 2024-04-26T00:26:14.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.392+0000 7fc12a126640 10 monclient: _renew_subs 2024-04-26T00:26:14.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.392+0000 7fc12a126640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:26:14.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.392+0000 7fc12a126640 1 -- 172.21.15.37:0/1175701006 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc124139e80 con 0x7fc1241382d0 2024-04-26T00:26:14.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.392+0000 7fc12a126640 10 monclient: _renew_subs 2024-04-26T00:26:14.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.392+0000 7fc12a126640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:26:14.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.392+0000 7fc12a126640 1 -- 172.21.15.37:0/1175701006 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1241c7b50 con 0x7fc1241382d0 2024-04-26T00:26:14.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.392+0000 7fc12a126640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:14.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.392+0000 7fc12a126640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:14.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.392+0000 7fc12a126640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:26:14.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.392+0000 7fc10bfff640 1 -- 172.21.15.37:0/1175701006 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc11c01a070 con 0x7fc1241382d0 2024-04-26T00:26:14.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.392+0000 7fc12a126640 1 -- 172.21.15.37:0/1175701006 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc124066f40 con 0x7fc1241382d0 2024-04-26T00:26:14.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.396+0000 7fc10bfff640 1 --2- 172.21.15.37:0/1175701006 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc110036e40 0x7fc110039300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:14.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.396+0000 7fc10bfff640 1 -- 172.21.15.37:0/1175701006 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(236..236 src has 177..236) ==== 3947+0+0 (secure 0 0 0) 0x7fc11c048450 con 0x7fc1241382d0 2024-04-26T00:26:14.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.396+0000 7fc1237fe640 1 --2- 172.21.15.37:0/1175701006 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc110036e40 0x7fc110039300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:14.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.396+0000 7fc1237fe640 10 monclient: get_auth_request con 0x7fc110036e40 auth_method 0 2024-04-26T00:26:14.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.396+0000 7fc1237fe640 1 --2- 172.21.15.37:0/1175701006 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc110036e40 0x7fc110039300 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc1140014c0 tx=0x7fc1140018d0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:14.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.400+0000 7fc10bfff640 1 -- 172.21.15.37:0/1175701006 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc11c01a360 con 0x7fc1241382d0 2024-04-26T00:26:14.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.400+0000 7fc10bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:14.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.400+0000 7fc10bfff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:26:14.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.628+0000 7fc12a126640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:26:14.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.628+0000 7fc12a126640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:26:14.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.628+0000 7fc12a126640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:26:14.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.628+0000 7fc12a126640 1 -- 172.21.15.37:0/1175701006 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc1240a84c0 con 0x7fc1241382d0 2024-04-26T00:26:14.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.628+0000 7fc10bfff640 1 -- 172.21.15.37:0/1175701006 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7fc11c047190 con 0x7fc1241382d0 2024-04-26T00:26:14.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.628+0000 7fc10bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:26:14.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.628+0000 7fc10bfff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:26:14.615 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":50,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:26:14.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.632+0000 7fc12a126640 1 -- 172.21.15.37:0/1175701006 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc110036e40 msgr2=0x7fc110039300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:14.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.632+0000 7fc12a126640 1 --2- 172.21.15.37:0/1175701006 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc110036e40 0x7fc110039300 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc1140014c0 tx=0x7fc1140018d0 comp rx=0 tx=0).stop 2024-04-26T00:26:14.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.632+0000 7fc12a126640 10 monclient: shutdown 2024-04-26T00:26:14.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.632+0000 7fc12a126640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:14.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.632+0000 7fc12a126640 1 -- 172.21.15.37:0/1175701006 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc1241382d0 msgr2=0x7fc124138760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:14.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.632+0000 7fc12a126640 1 --2- 172.21.15.37:0/1175701006 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc1241382d0 0x7fc124138760 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc124076070 tx=0x7fc11c00ce10 comp rx=0 tx=0).stop 2024-04-26T00:26:14.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.632+0000 7fc12a126640 1 -- 172.21.15.37:0/1175701006 shutdown_connections 2024-04-26T00:26:14.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.632+0000 7fc12a126640 1 --2- 172.21.15.37:0/1175701006 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc110036e40 0x7fc110039300 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:14.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.632+0000 7fc12a126640 1 --2- 172.21.15.37:0/1175701006 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc1241382d0 0x7fc124138760 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:14.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.632+0000 7fc12a126640 1 --2- 172.21.15.37:0/1175701006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1241cf840 0x7fc124137d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:14.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.632+0000 7fc12a126640 1 --2- 172.21.15.37:0/1175701006 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc1100063a0 0x7fc1241cf300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:14.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.632+0000 7fc12a126640 1 -- 172.21.15.37:0/1175701006 >> 172.21.15.37:0/1175701006 conn(0x7fc1240a5f00 msgr2=0x7fc12413e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:14.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.632+0000 7fc12a126640 1 -- 172.21.15.37:0/1175701006 shutdown_connections 2024-04-26T00:26:14.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.632+0000 7fc12a126640 1 -- 172.21.15.37:0/1175701006 wait complete. 2024-04-26T00:26:14.634 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-26T00:26:14.635 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-04-26T00:26:14.635 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:26:14.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.776+0000 7f56d1e82640 10 monclient: get_monmap_and_config 2024-04-26T00:26:14.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.776+0000 7f56d1e82640 10 monclient: build_initial_monmap 2024-04-26T00:26:14.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.776+0000 7f56d1e82640 10 monclient: monmap: 2024-04-26T00:26:14.763 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:14.763 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:14.763 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:26:14.783807+0000 2024-04-26T00:26:14.763 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:26:14.783807+0000 2024-04-26T00:26:14.763 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:14.764 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:14.764 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:26:14.764 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:26:14.764 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:26:14.764 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:26:14.764 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:26:14.764 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:14.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.780+0000 7f56d1e82640 1 Processor -- start 2024-04-26T00:26:14.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.780+0000 7f56d1e82640 1 -- start start 2024-04-26T00:26:14.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.780+0000 7f56d1e82640 10 monclient: init 2024-04-26T00:26:14.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.780+0000 7f56d1e82640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:14.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.780+0000 7f56d1e82640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-04-26T00:26:14.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.780+0000 7f56d1e82640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f56cc0a0300 0x7f56cc09fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:14.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.780+0000 7f56d1e82640 10 monclient(hunting): picked mon.noname-a con 0x7f56cc0a0300 addr v2:172.21.15.37:3300/0 2024-04-26T00:26:14.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.780+0000 7f56d1e82640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f56cc0a06e0 addr v1:172.21.15.37:6789/0 2024-04-26T00:26:14.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.780+0000 7f56d1e82640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f56cc1352b0 0x7f56cc13cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:14.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.780+0000 7f56d1e82640 10 monclient(hunting): picked mon.noname-c con 0x7f56cc1352b0 addr v2:172.21.15.45:3300/0 2024-04-26T00:26:14.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.780+0000 7f56d1e82640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56cc09c190 con 0x7f56cc0a06e0 2024-04-26T00:26:14.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.780+0000 7f56d1e82640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:14.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.780+0000 7f56d1e82640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f56cc0a13f0 con 0x7f56cc0a0300 2024-04-26T00:26:14.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.780+0000 7f56d1e82640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:14.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.780+0000 7f56d1e82640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f56cc0a1110 con 0x7f56cc1352b0 2024-04-26T00:26:14.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.780+0000 7f56d1e82640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:14.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.780+0000 7f56d1e82640 10 monclient(hunting): authenticate will time out at 1356.089455s 2024-04-26T00:26:14.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.780+0000 7f56cb7fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f56cc0a0300 0x7f56cc09fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:14.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.780+0000 7f56cb7fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f56cc0a0300 0x7f56cc09fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52220/0 (socket says 172.21.15.37:52220) 2024-04-26T00:26:14.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.780+0000 7f56cb7fe640 1 -- 172.21.15.37:0/913224087 learned_addr learned my addr 172.21.15.37:0/913224087 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:26:14.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.780+0000 7f56cb7fe640 10 monclient(hunting): get_auth_request con 0x7f56cc0a0300 auth_method 0 2024-04-26T00:26:14.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.780+0000 7f56cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:14.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.780+0000 7f56cb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:14.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.780+0000 7f56cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:14.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.780+0000 7f56cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:14.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.780+0000 7f56cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:14.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.780+0000 7f56cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:14.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.780+0000 7f56cb7fe640 10 monclient(hunting): handle_auth_done global_id 54345 payload 274 2024-04-26T00:26:14.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.780+0000 7f56cbfff640 1 --2- 172.21.15.37:0/913224087 >> v2:172.21.15.45:3300/0 conn(0x7f56cc1352b0 0x7f56cc13cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:14.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.780+0000 7f56cb7fe640 1 -- 172.21.15.37:0/913224087 >> v2:172.21.15.45:3300/0 conn(0x7f56cc1352b0 msgr2=0x7f56cc13cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:26:14.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.780+0000 7f56cb7fe640 1 --2- 172.21.15.37:0/913224087 >> v2:172.21.15.45:3300/0 conn(0x7f56cc1352b0 0x7f56cc13cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:14.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.780+0000 7f56cb7fe640 1 -- 172.21.15.37:0/913224087 >> v1:172.21.15.37:6789/0 conn(0x7f56cc0a06e0 legacy=0x7f56cc134ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:26:14.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.780+0000 7f56cb7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:26:14.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.780+0000 7f56cb7fe640 1 monclient: found mon.noname-a 2024-04-26T00:26:14.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.780+0000 7f56cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:14.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.780+0000 7f56cb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:26:14.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.780+0000 7f56cb7fe640 1 -- 172.21.15.37:0/913224087 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56cc0a16d0 con 0x7f56cc0a0300 2024-04-26T00:26:14.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.784+0000 7f56cb7fe640 1 -- 172.21.15.37:0/913224087 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56cc0a0300 msgr2=0x7f56cc09fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:14.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.784+0000 7f56cb7fe640 1 --2- 172.21.15.37:0/913224087 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56cc0a0300 0x7f56cc09fc50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f56cc06de30 tx=0x7f56bc01c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aba4f4a706e2ea87 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:14.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.784+0000 7f56ca7fc640 1 -- 172.21.15.37:0/913224087 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56bc0a5710 con 0x7f56cc0a0300 2024-04-26T00:26:14.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.784+0000 7f56ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:14.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.784+0000 7f56ca7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:26:14.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.784+0000 7f56ca7fc640 10 monclient: dump: 2024-04-26T00:26:14.771 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:14.771 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:14.771 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:14.771 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:14.771 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:14.771 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:14.771 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:14.771 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:14.771 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:14.771 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:14.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.784+0000 7f56ca7fc640 10 monclient: _finish_auth 0 2024-04-26T00:26:14.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.784+0000 7f56ca7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:26:14.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.784+0000 7f56ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:14.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.784+0000 7f56ca7fc640 1 -- 172.21.15.37:0/913224087 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56bc0a58b0 con 0x7f56cc0a0300 2024-04-26T00:26:14.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.784+0000 7f56ca7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:14.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.784+0000 7f56ca7fc640 1 -- 172.21.15.37:0/913224087 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56bc0a5c80 con 0x7f56cc0a0300 2024-04-26T00:26:14.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.784+0000 7f56d1e82640 5 monclient: authenticate success, global_id 54345 2024-04-26T00:26:14.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.784+0000 7f56d1e82640 10 monclient: get_monmap_and_config success 2024-04-26T00:26:14.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.784+0000 7f56d1e82640 4 set_mon_vals no callback set 2024-04-26T00:26:14.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.784+0000 7f56ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:14.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.784+0000 7f56ca7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:26:14.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.784+0000 7f56ca7fc640 10 monclient: dump: 2024-04-26T00:26:14.772 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:14.772 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:14.772 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:14.772 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:14.772 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:14.772 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:14.773 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:14.773 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:14.773 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:14.773 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:14.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.784+0000 7f56d1e82640 10 monclient: shutdown 2024-04-26T00:26:14.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.784+0000 7f56d1e82640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:14.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.784+0000 7f56d1e82640 1 -- 172.21.15.37:0/913224087 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56cc0a0300 msgr2=0x7f56cc09fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:14.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.784+0000 7f56d1e82640 1 --2- 172.21.15.37:0/913224087 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56cc0a0300 0x7f56cc09fc50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f56cc06de30 tx=0x7f56bc01c870 comp rx=0 tx=0).stop 2024-04-26T00:26:14.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.784+0000 7f56d1e82640 1 -- 172.21.15.37:0/913224087 shutdown_connections 2024-04-26T00:26:14.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.784+0000 7f56d1e82640 1 --2- 172.21.15.37:0/913224087 >> v2:172.21.15.45:3300/0 conn(0x7f56cc1352b0 0x7f56cc13cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:14.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.784+0000 7f56d1e82640 1 --2- 172.21.15.37:0/913224087 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56cc0a0300 0x7f56cc09fc50 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:14.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.784+0000 7f56d1e82640 1 -- 172.21.15.37:0/913224087 >> 172.21.15.37:0/913224087 conn(0x7f56cc0a1d50 msgr2=0x7f56cc0a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:14.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.784+0000 7f56d1e82640 1 -- 172.21.15.37:0/913224087 shutdown_connections 2024-04-26T00:26:14.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.784+0000 7f56d1e82640 1 -- 172.21.15.37:0/913224087 wait complete. 2024-04-26T00:26:14.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.784+0000 7f56d1e82640 10 monclient: build_initial_monmap 2024-04-26T00:26:14.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.784+0000 7f56d1e82640 10 monclient: monmap: 2024-04-26T00:26:14.774 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:14.774 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:26:14.774 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:26:14.774 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:26:14.774 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:14.774 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:26:14.774 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:26:14.774 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:26:14.774 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:26:14.774 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:14.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56d1e82640 1 Processor -- start 2024-04-26T00:26:14.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56d1e82640 1 -- start start 2024-04-26T00:26:14.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56d1e82640 10 monclient: init 2024-04-26T00:26:14.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56d1e82640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:14.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56d1e82640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:26:14.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56d1e82640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56cc0a0300 0x7f56cc1cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:14.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56d1e82640 10 monclient(hunting): picked mon.noname-c con 0x7f56cc0a0300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:26:14.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56d1e82640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56cc0a06e0 0x7f56cc133dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:14.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56d1e82640 10 monclient(hunting): picked mon.noname-b con 0x7f56cc0a06e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:26:14.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56d1e82640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56cc1352b0 0x7f56cc134300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:14.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56d1e82640 10 monclient(hunting): picked mon.noname-a con 0x7f56cc1352b0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:26:14.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56d1e82640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:14.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56d1e82640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f56cc13fdc0 con 0x7f56cc1352b0 2024-04-26T00:26:14.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56cb7fe640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56cc0a0300 msgr2=0x7f56cc1cf3f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:26:14.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56d1e82640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:14.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56d1e82640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f56cc13fc40 con 0x7f56cc0a06e0 2024-04-26T00:26:14.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56d1e82640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:14.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56d1e82640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f56cc13ff40 con 0x7f56cc0a0300 2024-04-26T00:26:14.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56cb7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56cc0a0300 0x7f56cc1cf3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:26:14.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56d1e82640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:14.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56d1e82640 10 monclient(hunting): authenticate will time out at 1356.096679s 2024-04-26T00:26:14.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56cbfff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56cc1352b0 0x7f56cc134300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:14.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56cbfff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56cc1352b0 0x7f56cc134300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52228/0 (socket says 172.21.15.37:52228) 2024-04-26T00:26:14.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56cbfff640 1 -- 172.21.15.37:0/796584941 learned_addr learned my addr 172.21.15.37:0/796584941 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:26:14.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56cbfff640 10 monclient(hunting): get_auth_request con 0x7f56cc1352b0 auth_method 0 2024-04-26T00:26:14.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:14.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56cbfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:14.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:14.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:14.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:14.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:14.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56caffd640 1 --2- 172.21.15.37:0/796584941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56cc0a06e0 0x7f56cc133dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:14.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56caffd640 10 monclient(hunting): get_auth_request con 0x7f56cc0a06e0 auth_method 0 2024-04-26T00:26:14.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:14.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56caffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:14.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56caffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:14.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56cbfff640 10 monclient(hunting): handle_auth_done global_id 54348 payload 824 2024-04-26T00:26:14.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56cbfff640 1 -- 172.21.15.37:0/796584941 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56cc0a0300 msgr2=0x7f56cc1cf3f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:26:14.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56cbfff640 1 --2- 172.21.15.37:0/796584941 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56cc0a0300 0x7f56cc1cf3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:14.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56cbfff640 1 -- 172.21.15.37:0/796584941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56cc0a06e0 msgr2=0x7f56cc133dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:14.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56cbfff640 1 --2- 172.21.15.37:0/796584941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56cc0a06e0 0x7f56cc133dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:14.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56cbfff640 10 monclient: _finish_hunting 0 2024-04-26T00:26:14.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56cbfff640 1 monclient: found mon.noname-a 2024-04-26T00:26:14.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:14.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56cbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:26:14.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56cbfff640 1 -- 172.21.15.37:0/796584941 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56cc11fef0 con 0x7f56cc1352b0 2024-04-26T00:26:14.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56cbfff640 1 -- 172.21.15.37:0/796584941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56cc1352b0 msgr2=0x7f56cc134300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:14.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56cbfff640 1 --2- 172.21.15.37:0/796584941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56cc1352b0 0x7f56cc134300 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f56c400c7f0 tx=0x7f56c400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:14.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56b3fff640 1 -- 172.21.15.37:0/796584941 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56c4015070 con 0x7f56cc1352b0 2024-04-26T00:26:14.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:14.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56b3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:26:14.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56b3fff640 10 monclient: dump: 2024-04-26T00:26:14.783 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:14.783 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:14.783 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:14.783 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:14.783 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:14.783 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:14.783 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:14.783 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:14.783 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:14.783 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:14.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56b3fff640 10 monclient: _finish_auth 0 2024-04-26T00:26:14.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56b3fff640 10 monclient: _check_auth_tickets 2024-04-26T00:26:14.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:14.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56b3fff640 1 -- 172.21.15.37:0/796584941 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56c4004220 con 0x7f56cc1352b0 2024-04-26T00:26:14.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56b3fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:14.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56b3fff640 1 -- 172.21.15.37:0/796584941 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56c4007830 con 0x7f56cc1352b0 2024-04-26T00:26:14.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:14.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56b3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:26:14.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56b3fff640 10 monclient: dump: 2024-04-26T00:26:14.784 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:14.784 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:14.784 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:14.784 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:14.784 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:14.784 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:14.784 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:14.785 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:14.785 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:14.785 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:14.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56c9ffb640 4 set_mon_vals no callback set 2024-04-26T00:26:14.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56d1e82640 5 monclient: authenticate success, global_id 54348 2024-04-26T00:26:14.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56d1e82640 10 monclient: _renew_subs 2024-04-26T00:26:14.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56d1e82640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:26:14.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56d1e82640 1 -- 172.21.15.37:0/796584941 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56cc06e5b0 con 0x7f56cc1352b0 2024-04-26T00:26:14.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56d1e82640 10 monclient: _renew_subs 2024-04-26T00:26:14.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56d1e82640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:26:14.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.788+0000 7f56d1e82640 1 -- 172.21.15.37:0/796584941 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56cc11f2c0 con 0x7f56cc1352b0 2024-04-26T00:26:14.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.792+0000 7f56d1e82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:14.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.792+0000 7f56d1e82640 1 --2- 172.21.15.37:0/796584941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56cc11c800 0x7f56cc11cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:14.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.792+0000 7f56d1e82640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:14.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.792+0000 7f56d1e82640 1 -- 172.21.15.37:0/796584941 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f56cc04f510 con 0x7f56cc11c800 2024-04-26T00:26:14.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.792+0000 7f56cb7fe640 1 --2- 172.21.15.37:0/796584941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56cc11c800 0x7f56cc11cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:14.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.792+0000 7f56cb7fe640 10 monclient: get_auth_request con 0x7f56cc11c800 auth_method 0 2024-04-26T00:26:14.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.792+0000 7f56cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:14.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.792+0000 7f56cb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:14.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.792+0000 7f56cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:14.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.792+0000 7f56b3fff640 1 -- 172.21.15.37:0/796584941 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f56c4007a50 con 0x7f56cc1352b0 2024-04-26T00:26:14.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.792+0000 7f56cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:14.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.792+0000 7f56cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:14.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.792+0000 7f56cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:14.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.792+0000 7f56b3fff640 1 --2- 172.21.15.37:0/796584941 >> v2:172.21.15.45:6812/2379918623 conn(0x7f56b8035e80 0x7f56b8038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:14.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.792+0000 7f56cb7fe640 10 monclient(hunting): handle_auth_done global_id 54351 payload 824 2024-04-26T00:26:14.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.792+0000 7f56cb7fe640 1 -- 172.21.15.37:0/796584941 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f56cc09eb60 con 0x7f56cc11c800 2024-04-26T00:26:14.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.792+0000 7f56b3fff640 1 -- 172.21.15.37:0/796584941 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(236..236 src has 177..236) ==== 3947+0+0 (secure 0 0 0) 0x7f56c40489b0 con 0x7f56cc1352b0 2024-04-26T00:26:14.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.792+0000 7f56caffd640 1 --2- 172.21.15.37:0/796584941 >> v2:172.21.15.45:6812/2379918623 conn(0x7f56b8035e80 0x7f56b8038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:14.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.792+0000 7f56caffd640 10 monclient: get_auth_request con 0x7f56b8035e80 auth_method 0 2024-04-26T00:26:14.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.792+0000 7f56cb7fe640 1 -- 172.21.15.37:0/796584941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56cc11c800 msgr2=0x7f56cc11cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:14.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.792+0000 7f56cb7fe640 1 --2- 172.21.15.37:0/796584941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56cc11c800 0x7f56cc11cbe0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f56bc0ae000 tx=0x7f56bc019610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:14.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.792+0000 7f56b3fff640 1 -- 172.21.15.37:0/796584941 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56bc0bd020 con 0x7f56cc11c800 2024-04-26T00:26:14.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.792+0000 7f56b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:14.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.792+0000 7f56b3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:26:14.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.792+0000 7f56b3fff640 10 monclient: dump: 2024-04-26T00:26:14.787 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:14.787 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:14.787 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:14.787 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:14.787 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:14.789 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:14.789 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:14.789 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:14.789 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:14.789 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:14.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.792+0000 7f56b3fff640 1 -- 172.21.15.37:0/796584941 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f56bc009af0 con 0x7f56cc11c800 2024-04-26T00:26:14.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.792+0000 7f56b3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:14.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.792+0000 7f56b3fff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:26:14.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.792+0000 7f56b3fff640 1 -- 172.21.15.37:0/796584941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56cc11c800 msgr2=0x7f56cc11cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:14.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.792+0000 7f56b3fff640 1 --2- 172.21.15.37:0/796584941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56cc11c800 0x7f56cc11cbe0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f56bc0ae000 tx=0x7f56bc019610 comp rx=0 tx=0).stop 2024-04-26T00:26:14.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.792+0000 7f56b3fff640 1 -- 172.21.15.37:0/796584941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56cc11c800 msgr2=0x7f56cc11cbe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:26:14.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.792+0000 7f56b3fff640 1 --2- 172.21.15.37:0/796584941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56cc11c800 0x7f56cc11cbe0 secure :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f56bc0ae000 tx=0x7f56bc019610 comp rx=0 tx=0).stop 2024-04-26T00:26:14.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.796+0000 7f56caffd640 1 --2- 172.21.15.37:0/796584941 >> v2:172.21.15.45:6812/2379918623 conn(0x7f56b8035e80 0x7f56b8038340 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f56cc071c10 tx=0x7f56c0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:14.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.800+0000 7f56d1e82640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:26:14.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.800+0000 7f56d1e82640 1 --2- 172.21.15.37:0/796584941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56cc11d120 0x7f56cc13a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:14.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.800+0000 7f56d1e82640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:14.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.800+0000 7f56d1e82640 1 -- 172.21.15.37:0/796584941 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f56cc1207f0 con 0x7f56cc11d120 2024-04-26T00:26:14.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.800+0000 7f56cb7fe640 1 --2- 172.21.15.37:0/796584941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56cc11d120 0x7f56cc13a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:14.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.800+0000 7f56cb7fe640 10 monclient: get_auth_request con 0x7f56cc11d120 auth_method 0 2024-04-26T00:26:14.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.800+0000 7f56cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:14.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.800+0000 7f56cb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:14.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.800+0000 7f56cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:14.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.800+0000 7f56cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:14.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.800+0000 7f56cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:14.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.800+0000 7f56cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:14.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.800+0000 7f56cb7fe640 10 monclient(hunting): handle_auth_done global_id 54354 payload 824 2024-04-26T00:26:14.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.800+0000 7f56cb7fe640 1 -- 172.21.15.37:0/796584941 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f56cc06ba50 con 0x7f56cc11d120 2024-04-26T00:26:14.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.804+0000 7f56cb7fe640 1 -- 172.21.15.37:0/796584941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56cc11d120 msgr2=0x7f56cc13a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:14.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.804+0000 7f56cb7fe640 1 --2- 172.21.15.37:0/796584941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56cc11d120 0x7f56cc13a0a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f56bc01af20 tx=0x7f56bc009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:14.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.804+0000 7f56b3fff640 1 -- 172.21.15.37:0/796584941 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56bc0bd040 con 0x7f56cc11d120 2024-04-26T00:26:14.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.804+0000 7f56b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:14.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.804+0000 7f56b3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:26:14.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.804+0000 7f56b3fff640 10 monclient: dump: 2024-04-26T00:26:14.793 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:14.793 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:14.793 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:14.793 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:14.793 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:14.793 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:14.793 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:14.794 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:14.794 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:14.794 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:14.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.804+0000 7f56b3fff640 1 -- 172.21.15.37:0/796584941 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f56bc0c4220 con 0x7f56cc11d120 2024-04-26T00:26:14.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.804+0000 7f56b3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:26:14.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.804+0000 7f56b3fff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:26:14.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.804+0000 7f56b3fff640 1 -- 172.21.15.37:0/796584941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56cc11d120 msgr2=0x7f56cc13a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:14.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.804+0000 7f56b3fff640 1 --2- 172.21.15.37:0/796584941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56cc11d120 0x7f56cc13a0a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f56bc01af20 tx=0x7f56bc009810 comp rx=0 tx=0).stop 2024-04-26T00:26:14.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.804+0000 7f56b3fff640 1 -- 172.21.15.37:0/796584941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56cc11d120 msgr2=0x7f56cc13a0a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:26:14.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.804+0000 7f56b3fff640 1 --2- 172.21.15.37:0/796584941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56cc11d120 0x7f56cc13a0a0 secure :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f56bc01af20 tx=0x7f56bc009810 comp rx=0 tx=0).stop 2024-04-26T00:26:14.794 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:26:14.795 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:26:14.795 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:26:14.795 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:26:14.795 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 50, 2024-04-26T00:26:14.795 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 138130, 2024-04-26T00:26:14.795 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:26:14.795 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:26:14.795 INFO:teuthology.orchestra.run.smithi037.stdout: 1 2024-04-26T00:26:14.795 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:14.795 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:26:14.795 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:26:14.795 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:26:14.795 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:26:14.795 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:26:14.795 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:26:14.796 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:26:14.796 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:26:14.796 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:26:14.796 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:26:14.796 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:26:14.796 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:26:14.796 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:26:14.796 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:26:14.796 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:26:14.796 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:14.796 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:26:14.796 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:26:14.796 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:26:14.796 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:26:14.796 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:26:14.796 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:26:14.796 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:26:14.797 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:26:14.797 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:26:14.797 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:26:14.797 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:26:14.797 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:26:14.797 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:26:14.797 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:14.797 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:14.797 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:26:14.797 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:26:14.797 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:26:14.797 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:26:14.797 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:26:14.797 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:26:14.797 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:26:14.797 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:26:14.798 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:26:14.798 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:26:14.798 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:26:14.798 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:26:14.798 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:26:14.798 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:26:14.798 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:26:14.798 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:26:14.798 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:26:14.798 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:26:14.798 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:26:14.798 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:26:14.798 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:26:14.798 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:26:14.799 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:26:14.799 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:26:14.799 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:26:14.799 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:26:14.799 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:26:14.799 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:26:14.799 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:14.799 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:26:14.799 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:14.799 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:26:14.799 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:14.799 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:26:14.799 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:26:14.799 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:26:14.799 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:26:14.799 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:14.800 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:26:14.800 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:26:14.800 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:14.800 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:14.800 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:14.800 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:26:14.800 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:26:14.800 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:14.800 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:14.800 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:14.800 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:14.800 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:26:14.800 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:26:14.800 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:26:14.800 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:26:14.800 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:26:14.801 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:14.801 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:14.801 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:26:14.801 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:26:14.801 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:26:14.801 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:26:14.801 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:14.801 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:26:14.801 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:26:14.801 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:14.801 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:14.801 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:14.801 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:26:14.802 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:26:14.802 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:14.802 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:14.802 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:14.802 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:14.802 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:26:14.802 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:26:14.802 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:26:14.802 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:26:14.802 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:26:14.802 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:14.802 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:14.802 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:26:14.802 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:26:14.803 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:26:14.803 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:26:14.803 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:14.803 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:26:14.803 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:26:14.803 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:14.803 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:14.803 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:14.803 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:26:14.803 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:26:14.803 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:14.803 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:14.803 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:14.803 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:14.803 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:26:14.803 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:26:14.804 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:26:14.804 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:26:14.804 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:26:14.804 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:14.804 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:14.804 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:14.804 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:26:14.804 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:26:14.804 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:14.804 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:26:14.804 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:26:14.804 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:26:14.804 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:14.804 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:14.805 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:26:14.805 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:14.805 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:26:14.805 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:26:14.805 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:26:14.805 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:14.805 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:14.805 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:26:14.805 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:14.805 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:26:14.805 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:26:14.805 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:26:14.805 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:14.805 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:14.805 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:26:14.806 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:14.806 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:26:14.806 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:26:14.806 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:26:14.806 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:14.806 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:14.806 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:14.806 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:26:14.806 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:26:14.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.804+0000 7f56d1e82640 1 -- 172.21.15.37:0/796584941 >> v2:172.21.15.45:6812/2379918623 conn(0x7f56b8035e80 msgr2=0x7f56b8038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:14.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.804+0000 7f56d1e82640 1 --2- 172.21.15.37:0/796584941 >> v2:172.21.15.45:6812/2379918623 conn(0x7f56b8035e80 0x7f56b8038340 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f56cc071c10 tx=0x7f56c0009040 comp rx=0 tx=0).stop 2024-04-26T00:26:14.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.804+0000 7f56d1e82640 10 monclient: shutdown 2024-04-26T00:26:14.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.804+0000 7f56d1e82640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:14.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.804+0000 7f56d1e82640 1 -- 172.21.15.37:0/796584941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56cc1352b0 msgr2=0x7f56cc134300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:14.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.804+0000 7f56d1e82640 1 --2- 172.21.15.37:0/796584941 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56cc1352b0 0x7f56cc134300 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f56c400c7f0 tx=0x7f56c400ccb0 comp rx=0 tx=0).stop 2024-04-26T00:26:14.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.804+0000 7f56cbfff640 1 -- 172.21.15.37:0/796584941 reap_dead start 2024-04-26T00:26:14.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.804+0000 7f56cbfff640 1 -- 172.21.15.37:0/796584941 reap_dead start 2024-04-26T00:26:14.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.804+0000 7f56d1e82640 1 -- 172.21.15.37:0/796584941 shutdown_connections 2024-04-26T00:26:14.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.804+0000 7f56d1e82640 1 -- 172.21.15.37:0/796584941 >> 172.21.15.37:0/796584941 conn(0x7f56cc0a1d50 msgr2=0x7f56cc0a4870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:14.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.804+0000 7f56d1e82640 1 -- 172.21.15.37:0/796584941 shutdown_connections 2024-04-26T00:26:14.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.804+0000 7f56d1e82640 1 -- 172.21.15.37:0/796584941 wait complete. 2024-04-26T00:26:14.809 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:26:14.809 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-04-26T00:26:14.809 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:26:14.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.936+0000 7f342ffe3640 10 monclient: get_monmap_and_config 2024-04-26T00:26:14.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.936+0000 7f342ffe3640 10 monclient: build_initial_monmap 2024-04-26T00:26:14.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.936+0000 7f342ffe3640 10 monclient: monmap: 2024-04-26T00:26:14.923 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:14.923 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:14.923 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:26:14.943455+0000 2024-04-26T00:26:14.923 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:26:14.943455+0000 2024-04-26T00:26:14.923 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:14.923 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:14.923 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:26:14.923 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:26:14.923 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:26:14.923 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:26:14.924 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:26:14.924 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:14.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.940+0000 7f342ffe3640 1 Processor -- start 2024-04-26T00:26:14.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.940+0000 7f342ffe3640 1 -- start start 2024-04-26T00:26:14.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.940+0000 7f342ffe3640 10 monclient: init 2024-04-26T00:26:14.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.940+0000 7f342ffe3640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:14.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.940+0000 7f342ffe3640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-04-26T00:26:14.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.940+0000 7f342ffe3640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f34280a3d90 0x7f342809bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:14.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.940+0000 7f342ffe3640 10 monclient(hunting): picked mon.noname-c con 0x7f34280a3d90 addr v2:172.21.15.45:3300/0 2024-04-26T00:26:14.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.940+0000 7f342ffe3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f34280a4170 addr v1:172.21.15.45:6789/0 2024-04-26T00:26:14.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.940+0000 7f342ffe3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f342813c3e0 0x7f342813e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:14.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.940+0000 7f342ffe3640 10 monclient(hunting): picked mon.noname-b con 0x7f342813c3e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:26:14.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.940+0000 7f342ffe3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34280a5150 con 0x7f34280a4170 2024-04-26T00:26:14.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.940+0000 7f342ffe3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:14.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.940+0000 7f342ffe3640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f342809b280 con 0x7f34280a3d90 2024-04-26T00:26:14.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.940+0000 7f342ffe3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:14.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.940+0000 7f342ffe3640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f342809b560 con 0x7f342813c3e0 2024-04-26T00:26:14.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.940+0000 7f342ffe3640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:14.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.940+0000 7f342ffe3640 10 monclient(hunting): authenticate will time out at 1356.249078s 2024-04-26T00:26:14.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.940+0000 7f342e580640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f342813c3e0 msgr2=0x7f342813e7d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:26:14.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.940+0000 7f342e580640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f342813c3e0 0x7f342813e7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:26:14.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.940+0000 7f342dd7f640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f34280a3d90 0x7f342809bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:14.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.940+0000 7f342dd7f640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f34280a3d90 0x7f342809bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.37:52934/0 (socket says 172.21.15.37:52934) 2024-04-26T00:26:14.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.940+0000 7f342dd7f640 1 -- 172.21.15.37:0/2583773177 learned_addr learned my addr 172.21.15.37:0/2583773177 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:26:14.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.940+0000 7f342dd7f640 10 monclient(hunting): get_auth_request con 0x7f34280a3d90 auth_method 0 2024-04-26T00:26:14.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.940+0000 7f342d57e640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f34280a4170 0x7f34280a45e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.37:55898/0 (socket says 172.21.15.37:55898) 2024-04-26T00:26:14.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.940+0000 7f342dd7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:14.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.940+0000 7f342dd7f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:14.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.940+0000 7f342dd7f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:14.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.940+0000 7f342dd7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:14.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.940+0000 7f342dd7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:14.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.940+0000 7f342dd7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:14.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.940+0000 7f342dd7f640 10 monclient(hunting): handle_auth_done global_id 24580 payload 274 2024-04-26T00:26:14.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.940+0000 7f342dd7f640 1 -- 172.21.15.37:0/2583773177 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f342813c3e0 msgr2=0x7f342813e7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:26:14.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.940+0000 7f342dd7f640 1 --2- 172.21.15.37:0/2583773177 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f342813c3e0 0x7f342813e7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:14.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.940+0000 7f342dd7f640 1 -- 172.21.15.37:0/2583773177 >> v1:172.21.15.45:6789/0 conn(0x7f34280a4170 legacy=0x7f34280a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:26:14.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.940+0000 7f342dd7f640 10 monclient: _finish_hunting 0 2024-04-26T00:26:14.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.940+0000 7f342dd7f640 1 monclient: found mon.noname-c 2024-04-26T00:26:14.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.940+0000 7f342dd7f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:14.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.940+0000 7f342dd7f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:26:14.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.940+0000 7f342dd7f640 1 -- 172.21.15.37:0/2583773177 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f342806dc60 con 0x7f34280a3d90 2024-04-26T00:26:14.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.944+0000 7f342dd7f640 1 -- 172.21.15.37:0/2583773177 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34280a3d90 msgr2=0x7f342809bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:14.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.944+0000 7f342dd7f640 1 --2- 172.21.15.37:0/2583773177 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34280a3d90 0x7f342809bf40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3424009870 tx=0x7f342401c400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc458ca68316a9f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:14.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.944+0000 7f342cd7d640 1 -- 172.21.15.37:0/2583773177 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34240b0070 con 0x7f34280a3d90 2024-04-26T00:26:14.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.944+0000 7f342cd7d640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:14.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.944+0000 7f342cd7d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:26:14.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.944+0000 7f342cd7d640 10 monclient: dump: 2024-04-26T00:26:14.931 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:14.931 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:14.931 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:14.931 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:14.931 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:14.931 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:14.931 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:14.931 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:14.932 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:14.932 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:14.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.944+0000 7f342cd7d640 10 monclient: _finish_auth 0 2024-04-26T00:26:14.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.944+0000 7f342cd7d640 10 monclient: _check_auth_tickets 2024-04-26T00:26:14.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.944+0000 7f342cd7d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:14.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.944+0000 7f342cd7d640 1 -- 172.21.15.37:0/2583773177 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34240ab600 con 0x7f34280a3d90 2024-04-26T00:26:14.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.944+0000 7f342cd7d640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:14.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.944+0000 7f342cd7d640 1 -- 172.21.15.37:0/2583773177 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34240ab9d0 con 0x7f34280a3d90 2024-04-26T00:26:14.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.944+0000 7f342cd7d640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:14.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.944+0000 7f342cd7d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:26:14.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.944+0000 7f342cd7d640 10 monclient: dump: 2024-04-26T00:26:14.932 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:14.932 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:14.932 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:14.932 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:14.933 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:14.933 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:14.933 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:14.933 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:14.933 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:14.933 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:14.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.944+0000 7f342ffe3640 5 monclient: authenticate success, global_id 24580 2024-04-26T00:26:14.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.944+0000 7f342ffe3640 10 monclient: get_monmap_and_config success 2024-04-26T00:26:14.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.944+0000 7f342ffe3640 4 set_mon_vals no callback set 2024-04-26T00:26:14.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.944+0000 7f342ffe3640 10 monclient: shutdown 2024-04-26T00:26:14.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.944+0000 7f342ffe3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:14.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.944+0000 7f342ffe3640 1 -- 172.21.15.37:0/2583773177 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34280a3d90 msgr2=0x7f342809bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:14.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.944+0000 7f342ffe3640 1 --2- 172.21.15.37:0/2583773177 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34280a3d90 0x7f342809bf40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3424009870 tx=0x7f342401c400 comp rx=0 tx=0).stop 2024-04-26T00:26:14.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.944+0000 7f342ffe3640 1 -- 172.21.15.37:0/2583773177 shutdown_connections 2024-04-26T00:26:14.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.944+0000 7f342ffe3640 1 --2- 172.21.15.37:0/2583773177 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f342813c3e0 0x7f342813e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:14.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.944+0000 7f342ffe3640 1 --2- 172.21.15.37:0/2583773177 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34280a3d90 0x7f342809bf40 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:14.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.944+0000 7f342ffe3640 1 -- 172.21.15.37:0/2583773177 >> 172.21.15.37:0/2583773177 conn(0x7f34280aa0a0 msgr2=0x7f34280a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:14.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.944+0000 7f342ffe3640 1 -- 172.21.15.37:0/2583773177 shutdown_connections 2024-04-26T00:26:14.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.944+0000 7f342ffe3640 1 -- 172.21.15.37:0/2583773177 wait complete. 2024-04-26T00:26:14.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342ffe3640 10 monclient: build_initial_monmap 2024-04-26T00:26:14.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342ffe3640 10 monclient: monmap: 2024-04-26T00:26:14.939 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:14.939 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:26:14.939 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:26:14.939 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:26:14.939 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:14.939 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:26:14.939 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:26:14.939 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:26:14.939 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:26:14.939 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:14.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342ffe3640 1 Processor -- start 2024-04-26T00:26:14.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342ffe3640 1 -- start start 2024-04-26T00:26:14.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342ffe3640 10 monclient: init 2024-04-26T00:26:14.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342ffe3640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:14.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342ffe3640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:26:14.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342ffe3640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34280a3d90 0x7f34281cef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:14.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342ffe3640 10 monclient(hunting): picked mon.noname-a con 0x7f34280a3d90 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:26:14.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342ffe3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34280a4170 0x7f34281371a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:14.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342ffe3640 10 monclient(hunting): picked mon.noname-c con 0x7f34280a4170 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:26:14.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342ffe3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f342813c3e0 0x7f34281cb980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:14.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342ffe3640 10 monclient(hunting): picked mon.noname-b con 0x7f342813c3e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:26:14.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342ffe3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:14.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342ffe3640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f3428141740 con 0x7f34280a3d90 2024-04-26T00:26:14.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342ffe3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:14.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342ffe3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f342809c690 con 0x7f342813c3e0 2024-04-26T00:26:14.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342ffe3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:14.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342ffe3640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f34281418c0 con 0x7f34280a4170 2024-04-26T00:26:14.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342ffe3640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:14.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342ffe3640 10 monclient(hunting): authenticate will time out at 1356.257229s 2024-04-26T00:26:14.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342d57e640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34280a4170 msgr2=0x7f34281371a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:26:14.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342d57e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34280a4170 0x7f34281371a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:26:14.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342dd7f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34280a3d90 0x7f34281cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:14.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342dd7f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34280a3d90 0x7f34281cef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52262/0 (socket says 172.21.15.37:52262) 2024-04-26T00:26:14.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342dd7f640 1 -- 172.21.15.37:0/756639411 learned_addr learned my addr 172.21.15.37:0/756639411 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:26:14.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342dd7f640 10 monclient(hunting): get_auth_request con 0x7f34280a3d90 auth_method 0 2024-04-26T00:26:14.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342dd7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:14.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342dd7f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:14.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342dd7f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:14.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342dd7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:14.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342dd7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:14.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342dd7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:14.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342e580640 1 --2- 172.21.15.37:0/756639411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f342813c3e0 0x7f34281cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:14.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342e580640 10 monclient(hunting): get_auth_request con 0x7f342813c3e0 auth_method 0 2024-04-26T00:26:14.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342e580640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:14.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342e580640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:14.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342e580640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:14.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342dd7f640 10 monclient(hunting): handle_auth_done global_id 54357 payload 824 2024-04-26T00:26:14.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342dd7f640 1 -- 172.21.15.37:0/756639411 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34280a4170 msgr2=0x7f34281371a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:26:14.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342dd7f640 1 --2- 172.21.15.37:0/756639411 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34280a4170 0x7f34281371a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:14.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342dd7f640 1 -- 172.21.15.37:0/756639411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f342813c3e0 msgr2=0x7f34281cb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:14.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342dd7f640 1 --2- 172.21.15.37:0/756639411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f342813c3e0 0x7f34281cb980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:14.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342dd7f640 10 monclient: _finish_hunting 0 2024-04-26T00:26:14.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342dd7f640 1 monclient: found mon.noname-a 2024-04-26T00:26:14.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342dd7f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:14.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342dd7f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:26:14.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.948+0000 7f342dd7f640 1 -- 172.21.15.37:0/756639411 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f342813cea0 con 0x7f34280a3d90 2024-04-26T00:26:14.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f342dd7f640 1 -- 172.21.15.37:0/756639411 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34280a3d90 msgr2=0x7f34281cef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:14.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f342dd7f640 1 --2- 172.21.15.37:0/756639411 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34280a3d90 0x7f34281cef00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f342401aab0 tx=0x7f34240191a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:14.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f341e7fc640 1 -- 172.21.15.37:0/756639411 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34240b9020 con 0x7f34280a3d90 2024-04-26T00:26:14.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f341e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:14.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f341e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:26:14.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f341e7fc640 10 monclient: dump: 2024-04-26T00:26:14.943 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:14.943 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:14.943 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:14.943 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:14.943 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:14.943 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:14.943 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:14.943 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:14.943 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:14.943 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:14.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f341e7fc640 10 monclient: _finish_auth 0 2024-04-26T00:26:14.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f341e7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:26:14.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f341e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:14.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f341e7fc640 1 -- 172.21.15.37:0/756639411 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34240b1040 con 0x7f34280a3d90 2024-04-26T00:26:14.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f342ffe3640 5 monclient: authenticate success, global_id 54357 2024-04-26T00:26:14.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f342ffe3640 10 monclient: _renew_subs 2024-04-26T00:26:14.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f342ffe3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:26:14.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f342ffe3640 1 -- 172.21.15.37:0/756639411 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3428121390 con 0x7f34280a3d90 2024-04-26T00:26:14.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f341e7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:14.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f341e7fc640 1 -- 172.21.15.37:0/756639411 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34240b0050 con 0x7f34280a3d90 2024-04-26T00:26:14.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f341e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:14.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f341e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:26:14.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f341e7fc640 10 monclient: dump: 2024-04-26T00:26:14.944 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:14.944 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:14.944 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:14.944 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:14.944 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:14.944 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:14.944 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:14.945 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:14.945 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:14.945 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:14.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f341ffff640 4 set_mon_vals no callback set 2024-04-26T00:26:14.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f342ffe3640 10 monclient: _renew_subs 2024-04-26T00:26:14.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f342ffe3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:26:14.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f342ffe3640 1 -- 172.21.15.37:0/756639411 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f342811fa70 con 0x7f34280a3d90 2024-04-26T00:26:14.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f342ffe3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:14.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f342ffe3640 1 --2- 172.21.15.37:0/756639411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34281206c0 0x7f34281cbec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:14.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f342ffe3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:14.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f342ffe3640 1 -- 172.21.15.37:0/756639411 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f34280a45c0 con 0x7f34281206c0 2024-04-26T00:26:14.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f341e7fc640 1 -- 172.21.15.37:0/756639411 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f34240bc020 con 0x7f34280a3d90 2024-04-26T00:26:14.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f341e7fc640 1 --2- 172.21.15.37:0/756639411 >> v2:172.21.15.45:6812/2379918623 conn(0x7f340c035e30 0x7f340c0382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:14.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f342d57e640 1 --2- 172.21.15.37:0/756639411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34281206c0 0x7f34281cbec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:14.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f341e7fc640 1 -- 172.21.15.37:0/756639411 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(236..236 src has 177..236) ==== 3947+0+0 (secure 0 0 0) 0x7f34240e2ae0 con 0x7f34280a3d90 2024-04-26T00:26:14.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f342d57e640 10 monclient: get_auth_request con 0x7f34281206c0 auth_method 0 2024-04-26T00:26:14.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f342d57e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:14.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f342d57e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:14.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f342d57e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:14.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f342d57e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:14.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f342d57e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:14.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f342d57e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:14.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f342e580640 1 --2- 172.21.15.37:0/756639411 >> v2:172.21.15.45:6812/2379918623 conn(0x7f340c035e30 0x7f340c0382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:14.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f342e580640 10 monclient: get_auth_request con 0x7f340c035e30 auth_method 0 2024-04-26T00:26:14.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f342d57e640 10 monclient(hunting): handle_auth_done global_id 24586 payload 824 2024-04-26T00:26:14.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.952+0000 7f342d57e640 1 -- 172.21.15.37:0/756639411 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f34281cf980 con 0x7f34281206c0 2024-04-26T00:26:14.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.956+0000 7f342d57e640 1 -- 172.21.15.37:0/756639411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34281206c0 msgr2=0x7f34281cbec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:14.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.956+0000 7f342d57e640 1 --2- 172.21.15.37:0/756639411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34281206c0 0x7f34281cbec0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f34281cff70 tx=0x7f341800c7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:14.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.956+0000 7f342e580640 1 --2- 172.21.15.37:0/756639411 >> v2:172.21.15.45:6812/2379918623 conn(0x7f340c035e30 0x7f340c0382f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3420003cc0 tx=0x7f3420009430 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:14.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.956+0000 7f341e7fc640 1 -- 172.21.15.37:0/756639411 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3418017070 con 0x7f34281206c0 2024-04-26T00:26:14.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.956+0000 7f341e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:14.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.956+0000 7f341e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:26:14.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.956+0000 7f341e7fc640 10 monclient: dump: 2024-04-26T00:26:14.948 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:14.948 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:14.948 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:14.948 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:14.948 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:14.948 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:14.949 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:14.949 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:14.949 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:14.949 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:14.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.956+0000 7f341e7fc640 1 -- 172.21.15.37:0/756639411 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3418005800 con 0x7f34281206c0 2024-04-26T00:26:14.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.956+0000 7f341e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:14.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.956+0000 7f341e7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:26:14.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.956+0000 7f341e7fc640 1 -- 172.21.15.37:0/756639411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34281206c0 msgr2=0x7f34281cbec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:14.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.956+0000 7f341e7fc640 1 --2- 172.21.15.37:0/756639411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34281206c0 0x7f34281cbec0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f34281cff70 tx=0x7f341800c7f0 comp rx=0 tx=0).stop 2024-04-26T00:26:14.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.956+0000 7f341e7fc640 1 -- 172.21.15.37:0/756639411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34281206c0 msgr2=0x7f34281cbec0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:26:14.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.956+0000 7f341e7fc640 1 --2- 172.21.15.37:0/756639411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34281206c0 0x7f34281cbec0 secure :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f34281cff70 tx=0x7f341800c7f0 comp rx=0 tx=0).stop 2024-04-26T00:26:14.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.964+0000 7f342ffe3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:26:14.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.964+0000 7f342ffe3640 1 --2- 172.21.15.37:0/756639411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34281376e0 0x7f34281cc400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:14.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.964+0000 7f342ffe3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:14.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.964+0000 7f342ffe3640 1 -- 172.21.15.37:0/756639411 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3428121d50 con 0x7f34281376e0 2024-04-26T00:26:14.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.964+0000 7f342d57e640 1 --2- 172.21.15.37:0/756639411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34281376e0 0x7f34281cc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:14.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.964+0000 7f342d57e640 10 monclient: get_auth_request con 0x7f34281376e0 auth_method 0 2024-04-26T00:26:14.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.964+0000 7f342d57e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:14.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.964+0000 7f342d57e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:14.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.964+0000 7f342d57e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:14.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.964+0000 7f342d57e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:14.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.964+0000 7f342d57e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:14.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.964+0000 7f342d57e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:14.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.964+0000 7f342d57e640 10 monclient(hunting): handle_auth_done global_id 24589 payload 824 2024-04-26T00:26:14.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.964+0000 7f342d57e640 1 -- 172.21.15.37:0/756639411 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f342804f510 con 0x7f34281376e0 2024-04-26T00:26:14.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.964+0000 7f342d57e640 1 -- 172.21.15.37:0/756639411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34281376e0 msgr2=0x7f34281cc400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:14.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.964+0000 7f342d57e640 1 --2- 172.21.15.37:0/756639411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34281376e0 0x7f34281cc400 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f342809cf40 tx=0x7f3418004990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:14.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.968+0000 7f341e7fc640 1 -- 172.21.15.37:0/756639411 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3418017070 con 0x7f34281376e0 2024-04-26T00:26:14.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.968+0000 7f341e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:14.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.968+0000 7f341e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:26:14.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.968+0000 7f341e7fc640 10 monclient: dump: 2024-04-26T00:26:14.952 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:14.952 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:14.952 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:14.952 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:14.952 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:14.952 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:14.952 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:14.952 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:14.953 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:14.953 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:14.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.968+0000 7f341e7fc640 1 -- 172.21.15.37:0/756639411 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4708 (secure 0 0 0) 0x7f34180192d0 con 0x7f34281376e0 2024-04-26T00:26:14.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.968+0000 7f341e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:26:14.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.968+0000 7f341e7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:26:14.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.968+0000 7f341e7fc640 1 -- 172.21.15.37:0/756639411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34281376e0 msgr2=0x7f34281cc400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:14.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.968+0000 7f341e7fc640 1 --2- 172.21.15.37:0/756639411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34281376e0 0x7f34281cc400 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f342809cf40 tx=0x7f3418004990 comp rx=0 tx=0).stop 2024-04-26T00:26:14.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.968+0000 7f341e7fc640 1 -- 172.21.15.37:0/756639411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34281376e0 msgr2=0x7f34281cc400 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:26:14.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.968+0000 7f341e7fc640 1 --2- 172.21.15.37:0/756639411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f34281376e0 0x7f34281cc400 secure :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f342809cf40 tx=0x7f3418004990 comp rx=0 tx=0).stop 2024-04-26T00:26:14.953 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:26:14.953 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:26:14.953 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:26:14.954 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:26:14.954 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 50, 2024-04-26T00:26:14.954 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 283135, 2024-04-26T00:26:14.954 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:26:14.954 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:26:14.954 INFO:teuthology.orchestra.run.smithi037.stdout: 1 2024-04-26T00:26:14.954 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:14.954 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:26:14.954 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:26:14.954 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:26:14.954 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:26:14.954 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:26:14.955 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:26:14.955 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:26:14.955 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:26:14.955 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:26:14.955 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:26:14.955 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:26:14.955 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:26:14.955 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:26:14.955 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:26:14.955 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:26:14.955 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:14.955 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:26:14.955 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:26:14.956 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:26:14.956 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:26:14.956 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:26:14.956 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:26:14.956 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:26:14.956 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:26:14.956 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:26:14.956 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:26:14.956 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:26:14.956 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:26:14.956 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:26:14.956 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:14.956 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:14.956 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:26:14.957 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:26:14.957 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:26:14.957 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:26:14.957 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:26:14.957 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:26:14.957 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:26:14.957 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:26:14.957 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:26:14.957 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:26:14.957 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:26:14.957 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:26:14.957 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:26:14.957 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:26:14.957 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:26:14.957 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:26:14.957 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:26:14.958 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:26:14.958 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:26:14.958 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:26:14.958 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:26:14.958 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:26:14.958 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:26:14.958 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:26:14.958 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:26:14.958 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:26:14.958 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:26:14.958 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:26:14.958 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:14.958 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:26:14.958 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:14.959 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:26:14.959 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:14.959 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:26:14.959 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:26:14.959 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:26:14.959 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:26:14.959 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:14.959 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:26:14.959 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:26:14.959 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:14.959 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:14.959 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:14.959 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:26:14.959 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:26:14.959 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:14.959 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:14.960 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:14.960 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:14.960 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:26:14.960 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:26:14.960 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:26:14.960 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:26:14.960 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:26:14.960 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:14.960 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:14.960 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:26:14.960 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:26:14.960 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:26:14.960 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:26:14.960 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:14.960 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:26:14.961 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:26:14.961 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:14.961 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:14.961 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:14.961 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:26:14.961 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:26:14.961 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:14.961 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:14.961 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:14.961 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:14.961 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:26:14.961 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:26:14.961 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:26:14.961 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:26:14.961 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:26:14.962 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:14.962 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:14.962 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:26:14.962 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:26:14.962 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:26:14.962 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:26:14.962 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:14.962 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:26:14.962 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:26:14.962 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:14.962 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:14.962 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:14.962 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:26:14.962 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:26:14.962 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:14.963 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:14.963 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:14.963 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:14.963 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:26:14.963 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:26:14.963 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:26:14.963 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:26:14.963 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:26:14.963 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:14.963 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:14.963 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:14.963 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:26:14.963 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:26:14.963 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:14.963 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:26:14.963 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:26:14.964 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:26:14.964 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:14.964 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:14.964 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:26:14.964 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:14.964 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:26:14.964 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:26:14.964 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:26:14.964 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:14.964 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:14.964 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:26:14.964 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:14.964 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:26:14.964 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:26:14.964 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:26:14.965 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:14.965 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:14.965 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:14.965 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:26:14.965 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:26:14.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.968+0000 7f342ffe3640 1 -- 172.21.15.37:0/756639411 >> v2:172.21.15.45:6812/2379918623 conn(0x7f340c035e30 msgr2=0x7f340c0382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:14.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.968+0000 7f342ffe3640 1 --2- 172.21.15.37:0/756639411 >> v2:172.21.15.45:6812/2379918623 conn(0x7f340c035e30 0x7f340c0382f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3420003cc0 tx=0x7f3420009430 comp rx=0 tx=0).stop 2024-04-26T00:26:14.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.968+0000 7f342ffe3640 10 monclient: shutdown 2024-04-26T00:26:14.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.968+0000 7f342ffe3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:14.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.968+0000 7f342ffe3640 1 -- 172.21.15.37:0/756639411 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34280a3d90 msgr2=0x7f34281cef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:14.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.968+0000 7f342ffe3640 1 --2- 172.21.15.37:0/756639411 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34280a3d90 0x7f34281cef00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f342401aab0 tx=0x7f34240191a0 comp rx=0 tx=0).stop 2024-04-26T00:26:14.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.968+0000 7f342e580640 1 -- 172.21.15.37:0/756639411 reap_dead start 2024-04-26T00:26:14.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.968+0000 7f342e580640 1 -- 172.21.15.37:0/756639411 reap_dead start 2024-04-26T00:26:14.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.968+0000 7f342ffe3640 1 -- 172.21.15.37:0/756639411 shutdown_connections 2024-04-26T00:26:14.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.968+0000 7f342ffe3640 1 -- 172.21.15.37:0/756639411 >> 172.21.15.37:0/756639411 conn(0x7f34280aa0a0 msgr2=0x7f34280a8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:14.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.968+0000 7f342ffe3640 1 -- 172.21.15.37:0/756639411 shutdown_connections 2024-04-26T00:26:14.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:14.968+0000 7f342ffe3640 1 -- 172.21.15.37:0/756639411 wait complete. 2024-04-26T00:26:14.968 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:26:14.968 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-04-26T00:26:15.016 INFO:tasks.rados.rados.0.smithi045.stdout:983: done (0 left) 2024-04-26T00:26:15.016 INFO:tasks.rados.rados.0.smithi045.stdout:984: read oid 23 snap -1 2024-04-26T00:26:15.016 INFO:tasks.rados.rados.0.smithi045.stdout:984: expect (ObjNum 316 snap 93 seq_num 316) 2024-04-26T00:26:15.016 INFO:tasks.rados.rados.0.smithi045.stdout:985: read oid 24 snap -1 2024-04-26T00:26:15.016 INFO:tasks.rados.rados.0.smithi045.stdout:985: expect (ObjNum 351 snap 105 seq_num 351) 2024-04-26T00:26:15.016 INFO:tasks.rados.rados.0.smithi045.stdout:986: read oid 49 snap -1 2024-04-26T00:26:15.017 INFO:tasks.rados.rados.0.smithi045.stdout:986: expect (ObjNum 368 snap 108 seq_num 368) 2024-04-26T00:26:15.017 INFO:tasks.rados.rados.0.smithi045.stdout:987: delete oid 39 current snap is 117 2024-04-26T00:26:15.020 INFO:tasks.rados.rados.0.smithi045.stdout:987: done (3 left) 2024-04-26T00:26:15.020 INFO:tasks.rados.rados.0.smithi045.stdout:988: read oid 37 snap 108 2024-04-26T00:26:15.020 INFO:tasks.rados.rados.0.smithi045.stdout:988: expect (ObjNum 215 snap 55 seq_num 215) 2024-04-26T00:26:15.020 INFO:tasks.rados.rados.0.smithi045.stdout:989: delete oid 5 current snap is 117 2024-04-26T00:26:15.027 INFO:tasks.rados.rados.0.smithi045.stdout:984: expect (ObjNum 316 snap 93 seq_num 316) 2024-04-26T00:26:15.293 INFO:tasks.rados.rados.0.smithi045.stdout:986: expect (ObjNum 368 snap 108 seq_num 368) 2024-04-26T00:26:15.441 INFO:tasks.rados.rados.0.smithi045.stdout:985: expect (ObjNum 351 snap 105 seq_num 351) 2024-04-26T00:26:15.666 INFO:tasks.rados.rados.0.smithi045.stdout:988: expect (ObjNum 215 snap 55 seq_num 215) 2024-04-26T00:26:15.780 INFO:tasks.rados.rados.0.smithi045.stdout:984: done (4 left) 2024-04-26T00:26:15.780 INFO:tasks.rados.rados.0.smithi045.stdout:985: done (3 left) 2024-04-26T00:26:15.780 INFO:tasks.rados.rados.0.smithi045.stdout:986: done (2 left) 2024-04-26T00:26:15.780 INFO:tasks.rados.rados.0.smithi045.stdout:988: done (1 left) 2024-04-26T00:26:15.780 INFO:tasks.rados.rados.0.smithi045.stdout:989: done (0 left) 2024-04-26T00:26:15.781 INFO:tasks.rados.rados.0.smithi045.stdout:990: write (excl) oid 47 current snap is 117 2024-04-26T00:26:15.781 INFO:tasks.rados.rados.0.smithi045.stdout:990: seq_num 382 ranges {420274=562768,1620592=439131} 2024-04-26T00:26:15.796 INFO:tasks.rados.rados.0.smithi045.stdout:990: writing smithi04526579-47 from 420274 to 983042 tid 1 2024-04-26T00:26:15.814 INFO:tasks.rados.rados.0.smithi045.stdout:990: writing smithi04526579-47 from 1620592 to 2059723 tid 2 2024-04-26T00:26:15.814 INFO:tasks.rados.rados.0.smithi045.stdout:991: write (excl) oid 41 current snap is 117 2024-04-26T00:26:15.814 INFO:tasks.rados.rados.0.smithi045.stdout:990: finishing write tid 1 to smithi04526579-47 2024-04-26T00:26:15.814 INFO:tasks.rados.rados.0.smithi045.stdout:991: seq_num 383 ranges {437081=638017,1679074=582789} 2024-04-26T00:26:15.832 INFO:tasks.rados.rados.0.smithi045.stdout:991: writing smithi04526579-41 from 437081 to 1075098 tid 1 2024-04-26T00:26:15.851 INFO:tasks.rados.rados.0.smithi045.stdout:991: writing smithi04526579-41 from 1679074 to 2261863 tid 2 2024-04-26T00:26:15.851 INFO:tasks.rados.rados.0.smithi045.stdout:992: copy_from oid 32 from oid 50 current snap is 117 2024-04-26T00:26:15.851 INFO:tasks.rados.rados.0.smithi045.stdout:990: finishing write tid 2 to smithi04526579-47 2024-04-26T00:26:15.851 INFO:tasks.rados.rados.0.smithi045.stdout:990: finishing write tid 3 to smithi04526579-47 2024-04-26T00:26:15.851 INFO:tasks.rados.rados.0.smithi045.stdout:990: finishing write tid 4 to smithi04526579-47 2024-04-26T00:26:15.851 INFO:tasks.rados.rados.0.smithi045.stdout:990: oid 47 updating version 0 to 524 2024-04-26T00:26:15.851 INFO:tasks.rados.rados.0.smithi045.stdout:990: oid 47 updating version 524 to 525 2024-04-26T00:26:15.851 INFO:tasks.rados.rados.0.smithi045.stdout:990: oid 47 version 525 is already newer than 523 2024-04-26T00:26:15.851 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 525 (ObjNum 382 snap 117 seq_num 382) dirty exists 2024-04-26T00:26:15.852 INFO:tasks.rados.rados.0.smithi045.stdout:990: left oid 47 (ObjNum 382 snap 117 seq_num 382) 2024-04-26T00:26:15.852 INFO:tasks.rados.rados.0.smithi045.stdout:991: finishing write tid 1 to smithi04526579-41 2024-04-26T00:26:15.852 INFO:tasks.rados.rados.0.smithi045.stdout:990: done (2 left) 2024-04-26T00:26:15.852 INFO:tasks.rados.rados.0.smithi045.stdout:993: delete oid 16 current snap is 117 2024-04-26T00:26:15.854 INFO:tasks.rados.rados.0.smithi045.stdout:993: done (2 left) 2024-04-26T00:26:15.854 INFO:tasks.rados.rados.0.smithi045.stdout:994: read oid 5 snap 99 2024-04-26T00:26:15.854 INFO:tasks.rados.rados.0.smithi045.stdout:994: expect (ObjNum 314 snap 92 seq_num 314) 2024-04-26T00:26:15.854 INFO:tasks.rados.rados.0.smithi045.stdout:995: copy_from oid 3 from oid 36 current snap is 117 2024-04-26T00:26:15.854 INFO:tasks.rados.rados.0.smithi045.stdout:996: delete oid 24 current snap is 117 2024-04-26T00:26:15.856 INFO:tasks.rados.rados.0.smithi045.stdout:991: finishing write tid 2 to smithi04526579-41 2024-04-26T00:26:15.858 INFO:tasks.rados.rados.0.smithi045.stdout:991: finishing write tid 3 to smithi04526579-41 2024-04-26T00:26:15.858 INFO:tasks.rados.rados.0.smithi045.stdout:991: finishing write tid 4 to smithi04526579-41 2024-04-26T00:26:15.858 INFO:tasks.rados.rados.0.smithi045.stdout:991: oid 41 updating version 0 to 589 2024-04-26T00:26:15.858 INFO:tasks.rados.rados.0.smithi045.stdout:991: oid 41 updating version 589 to 590 2024-04-26T00:26:15.858 INFO:tasks.rados.rados.0.smithi045.stdout:991: oid 41 updating version 590 to 591 2024-04-26T00:26:15.858 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 591 (ObjNum 383 snap 117 seq_num 383) dirty exists 2024-04-26T00:26:15.858 INFO:tasks.rados.rados.0.smithi045.stdout:991: left oid 41 (ObjNum 383 snap 117 seq_num 383) 2024-04-26T00:26:15.873 INFO:tasks.rados.rados.0.smithi045.stdout:992: finishing copy_from racing read to smithi04526579-32 2024-04-26T00:26:15.874 INFO:tasks.rados.rados.0.smithi045.stdout:994: expect (ObjNum 314 snap 92 seq_num 314) 2024-04-26T00:26:16.035 INFO:tasks.rados.rados.0.smithi045.stdout:992: finishing copy_from to smithi04526579-32 2024-04-26T00:26:16.035 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 385 (ObjNum 136 snap 30 seq_num 136) dirty exists 2024-04-26T00:26:16.035 INFO:tasks.rados.rados.0.smithi045.stdout:995: finishing copy_from racing read to smithi04526579-3 2024-04-26T00:26:16.036 INFO:tasks.rados.rados.0.smithi045.stdout:995: finishing copy_from to smithi04526579-3 2024-04-26T00:26:16.036 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 259 (ObjNum 304 snap 87 seq_num 304) dirty exists 2024-04-26T00:26:16.036 INFO:tasks.rados.rados.0.smithi045.stdout:991: done (4 left) 2024-04-26T00:26:16.036 INFO:tasks.rados.rados.0.smithi045.stdout:992: done (3 left) 2024-04-26T00:26:16.036 INFO:tasks.rados.rados.0.smithi045.stdout:994: done (2 left) 2024-04-26T00:26:16.036 INFO:tasks.rados.rados.0.smithi045.stdout:995: done (1 left) 2024-04-26T00:26:16.036 INFO:tasks.rados.rados.0.smithi045.stdout:996: done (0 left) 2024-04-26T00:26:16.036 INFO:tasks.rados.rados.0.smithi045.stdout:997: write oid 45 current snap is 117 2024-04-26T00:26:16.036 INFO:tasks.rados.rados.0.smithi045.stdout:997: seq_num 386 ranges {487502=463764,1738167=446469} 2024-04-26T00:26:16.048 INFO:tasks.rados.rados.0.smithi045.stdout:997: writing smithi04526579-45 from 487502 to 951266 tid 1 2024-04-26T00:26:16.066 INFO:tasks.rados.rados.0.smithi045.stdout:997: writing smithi04526579-45 from 1738167 to 2184636 tid 2 2024-04-26T00:26:16.067 INFO:tasks.rados.rados.0.smithi045.stdout:998: copy_from oid 43 from oid 30 current snap is 117 2024-04-26T00:26:16.067 INFO:tasks.rados.rados.0.smithi045.stdout:997: finishing write tid 1 to smithi04526579-45 2024-04-26T00:26:16.067 INFO:tasks.rados.rados.0.smithi045.stdout:999: write oid 40 current snap is 117 2024-04-26T00:26:16.067 INFO:tasks.rados.rados.0.smithi045.stdout:999: seq_num 387 ranges {504309=539013,1796649=590127} 2024-04-26T00:26:16.082 INFO:tasks.rados.rados.0.smithi045.stdout:999: writing smithi04526579-40 from 504309 to 1043322 tid 1 2024-04-26T00:26:16.099 INFO:tasks.rados.rados.0.smithi045.stdout:999: writing smithi04526579-40 from 1796649 to 2386776 tid 2 2024-04-26T00:26:16.099 INFO:tasks.rados.rados.0.smithi045.stdout:1000: read oid 35 snap -1 2024-04-26T00:26:16.100 INFO:tasks.rados.rados.0.smithi045.stdout:1000: expect (ObjNum 305 snap 87 seq_num 305) 2024-04-26T00:26:16.100 INFO:tasks.rados.rados.0.smithi045.stdout:1001: write oid 27 current snap is 117 2024-04-26T00:26:16.100 INFO:tasks.rados.rados.0.smithi045.stdout:997: finishing write tid 2 to smithi04526579-45 2024-04-26T00:26:16.100 INFO:tasks.rados.rados.0.smithi045.stdout:997: finishing write tid 3 to smithi04526579-45 2024-04-26T00:26:16.100 INFO:tasks.rados.rados.0.smithi045.stdout:997: finishing write tid 4 to smithi04526579-45 2024-04-26T00:26:16.100 INFO:tasks.rados.rados.0.smithi045.stdout:997: oid 45 updating version 0 to 378 2024-04-26T00:26:16.100 INFO:tasks.rados.rados.0.smithi045.stdout:997: oid 45 updating version 378 to 379 2024-04-26T00:26:16.100 INFO:tasks.rados.rados.0.smithi045.stdout:997: oid 45 updating version 379 to 380 2024-04-26T00:26:16.100 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 380 (ObjNum 386 snap 117 seq_num 386) dirty exists 2024-04-26T00:26:16.100 INFO:tasks.rados.rados.0.smithi045.stdout:997: left oid 45 (ObjNum 386 snap 117 seq_num 386) 2024-04-26T00:26:16.100 INFO:tasks.rados.rados.0.smithi045.stdout:998: finishing copy_from racing read to smithi04526579-43 2024-04-26T00:26:16.100 INFO:tasks.rados.rados.0.smithi045.stdout:998: finishing copy_from to smithi04526579-43 2024-04-26T00:26:16.100 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 309 (ObjNum 359 snap 107 seq_num 359) dirty exists 2024-04-26T00:26:16.100 INFO:tasks.rados.rados.0.smithi045.stdout:999: finishing write tid 1 to smithi04526579-40 2024-04-26T00:26:16.100 INFO:tasks.rados.rados.0.smithi045.stdout:1001: seq_num 388 ranges {521116=730615,1971484=549632} 2024-04-26T00:26:16.112 INFO:tasks.rados.rados.0.smithi045.stdout:1001: writing smithi04526579-27 from 521116 to 1251731 tid 1 2024-04-26T00:26:16.125 INFO:tasks.rados.rados.0.smithi045.stdout:1001: writing smithi04526579-27 from 1971484 to 2521116 tid 2 2024-04-26T00:26:16.125 INFO:tasks.rados.rados.0.smithi045.stdout:997: done (4 left) 2024-04-26T00:26:16.125 INFO:tasks.rados.rados.0.smithi045.stdout:998: done (3 left) 2024-04-26T00:26:16.125 INFO:tasks.rados.rados.0.smithi045.stdout:1002: copy_from oid 6 from oid 49 current snap is 117 2024-04-26T00:26:16.125 INFO:tasks.rados.rados.0.smithi045.stdout:999: finishing write tid 2 to smithi04526579-40 2024-04-26T00:26:16.125 INFO:tasks.rados.rados.0.smithi045.stdout:999: finishing write tid 3 to smithi04526579-40 2024-04-26T00:26:16.125 INFO:tasks.rados.rados.0.smithi045.stdout:999: finishing write tid 4 to smithi04526579-40 2024-04-26T00:26:16.125 INFO:tasks.rados.rados.0.smithi045.stdout:999: oid 40 updating version 0 to 325 2024-04-26T00:26:16.125 INFO:tasks.rados.rados.0.smithi045.stdout:999: oid 40 version 325 is already newer than 324 2024-04-26T00:26:16.125 INFO:tasks.rados.rados.0.smithi045.stdout:999: oid 40 version 325 is already newer than 323 2024-04-26T00:26:16.125 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 325 (ObjNum 387 snap 117 seq_num 387) dirty exists 2024-04-26T00:26:16.125 INFO:tasks.rados.rados.0.smithi045.stdout:999: left oid 40 (ObjNum 387 snap 117 seq_num 387) 2024-04-26T00:26:16.126 INFO:tasks.rados.rados.0.smithi045.stdout:1000: expect (ObjNum 305 snap 87 seq_num 305) 2024-04-26T00:26:16.345 INFO:tasks.rados.rados.0.smithi045.stdout:1001: finishing write tid 1 to smithi04526579-27 2024-04-26T00:26:16.345 INFO:tasks.rados.rados.0.smithi045.stdout:1001: finishing write tid 2 to smithi04526579-27 2024-04-26T00:26:16.345 INFO:tasks.rados.rados.0.smithi045.stdout:1001: finishing write tid 3 to smithi04526579-27 2024-04-26T00:26:16.345 INFO:tasks.rados.rados.0.smithi045.stdout:1001: finishing write tid 4 to smithi04526579-27 2024-04-26T00:26:16.345 INFO:tasks.rados.rados.0.smithi045.stdout:1001: oid 27 updating version 0 to 497 2024-04-26T00:26:16.345 INFO:tasks.rados.rados.0.smithi045.stdout:1001: oid 27 version 497 is already newer than 496 2024-04-26T00:26:16.346 INFO:tasks.rados.rados.0.smithi045.stdout:1001: oid 27 updating version 497 to 498 2024-04-26T00:26:16.346 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 498 (ObjNum 388 snap 117 seq_num 388) dirty exists 2024-04-26T00:26:16.346 INFO:tasks.rados.rados.0.smithi045.stdout:1001: left oid 27 (ObjNum 388 snap 117 seq_num 388) 2024-04-26T00:26:16.346 INFO:tasks.rados.rados.0.smithi045.stdout:1002: finishing copy_from racing read to smithi04526579-6 2024-04-26T00:26:16.346 INFO:tasks.rados.rados.0.smithi045.stdout:1002: finishing copy_from to smithi04526579-6 2024-04-26T00:26:16.346 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 493 (ObjNum 368 snap 108 seq_num 368) dirty exists 2024-04-26T00:26:16.346 INFO:tasks.rados.rados.0.smithi045.stdout:999: done (3 left) 2024-04-26T00:26:16.346 INFO:tasks.rados.rados.0.smithi045.stdout:1000: done (2 left) 2024-04-26T00:26:16.346 INFO:tasks.rados.rados.0.smithi045.stdout:1001: done (1 left) 2024-04-26T00:26:16.347 INFO:tasks.rados.rados.0.smithi045.stdout:1002: done (0 left) 2024-04-26T00:26:16.347 INFO:tasks.rados.rados.0.smithi045.stdout:1003: rollback oid 25 current snap is 117 2024-04-26T00:26:16.347 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 25 to 97 2024-04-26T00:26:16.347 INFO:tasks.rados.rados.0.smithi045.stdout:1004: rollback oid 6 current snap is 117 2024-04-26T00:26:16.347 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 6 to 110 2024-04-26T00:26:16.347 INFO:tasks.rados.rados.0.smithi045.stdout:1005: read oid 44 snap -1 2024-04-26T00:26:16.347 INFO:tasks.rados.rados.0.smithi045.stdout:1005: expect (ObjNum 283 snap 81 seq_num 283) 2024-04-26T00:26:16.347 INFO:tasks.rados.rados.0.smithi045.stdout:1006: read oid 49 snap 78 2024-04-26T00:26:16.347 INFO:tasks.rados.rados.0.smithi045.stdout:1006: expect (ObjNum 256 snap 66 seq_num 256) 2024-04-26T00:26:16.347 INFO:tasks.rados.rados.0.smithi045.stdout:1007: write oid 47 current snap is 117 2024-04-26T00:26:16.347 INFO:tasks.rados.rados.0.smithi045.stdout:1007: seq_num 389 ranges {537923=405864,1513613=593796} 2024-04-26T00:26:16.355 INFO:tasks.rados.rados.0.smithi045.stdout:1007: writing smithi04526579-47 from 537923 to 943787 tid 1 2024-04-26T00:26:16.368 INFO:tasks.rados.rados.0.smithi045.stdout:1007: writing smithi04526579-47 from 1513613 to 2107409 tid 2 2024-04-26T00:26:16.368 INFO:tasks.rados.rados.0.smithi045.stdout:1008: snap_create 2024-04-26T00:26:16.368 INFO:tasks.rados.rados.0.smithi045.stdout:1003: finishing rollback tid 1 to smithi04526579-25 2024-04-26T00:26:16.368 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 290 (ObjNum 161 snap 0 seq_num 1476942341) dirty dne 2024-04-26T00:26:16.368 INFO:tasks.rados.rados.0.smithi045.stdout:1004: finishing rollback tid 0 to smithi04526579-6 2024-04-26T00:26:16.368 INFO:tasks.rados.rados.0.smithi045.stdout:1004: finishing rollback tid 1 to smithi04526579-6 2024-04-26T00:26:16.368 INFO:tasks.rados.rados.0.smithi045.stdout:1004: finishing rollback tid 2 to smithi04526579-6 2024-04-26T00:26:16.368 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 496 (ObjNum 361 snap 107 seq_num 361) dirty exists 2024-04-26T00:26:16.369 INFO:tasks.rados.rados.0.smithi045.stdout:1005: expect (ObjNum 283 snap 81 seq_num 283) 2024-04-26T00:26:16.572 INFO:tasks.rados.rados.0.smithi045.stdout:1006: expect (ObjNum 256 snap 66 seq_num 256) 2024-04-26T00:26:16.672 INFO:tasks.rados.rados.0.smithi045.stdout:1007: finishing write tid 1 to smithi04526579-47 2024-04-26T00:26:16.672 INFO:tasks.rados.rados.0.smithi045.stdout:1007: finishing write tid 2 to smithi04526579-47 2024-04-26T00:26:16.672 INFO:tasks.rados.rados.0.smithi045.stdout:1007: finishing write tid 3 to smithi04526579-47 2024-04-26T00:26:16.672 INFO:tasks.rados.rados.0.smithi045.stdout:1007: finishing write tid 4 to smithi04526579-47 2024-04-26T00:26:16.672 INFO:tasks.rados.rados.0.smithi045.stdout:1007: oid 47 updating version 0 to 528 2024-04-26T00:26:16.672 INFO:tasks.rados.rados.0.smithi045.stdout:1007: oid 47 updating version 528 to 529 2024-04-26T00:26:16.672 INFO:tasks.rados.rados.0.smithi045.stdout:1007: oid 47 updating version 529 to 530 2024-04-26T00:26:16.672 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 530 (ObjNum 389 snap 117 seq_num 389) dirty exists 2024-04-26T00:26:16.672 INFO:tasks.rados.rados.0.smithi045.stdout:1007: left oid 47 (ObjNum 389 snap 117 seq_num 389) 2024-04-26T00:26:16.879 INFO:tasks.rados.rados.0.smithi045.stdout:1003: done (5 left) 2024-04-26T00:26:16.879 INFO:tasks.rados.rados.0.smithi045.stdout:1004: done (4 left) 2024-04-26T00:26:16.879 INFO:tasks.rados.rados.0.smithi045.stdout:1005: done (3 left) 2024-04-26T00:26:16.879 INFO:tasks.rados.rados.0.smithi045.stdout:1006: done (2 left) 2024-04-26T00:26:16.879 INFO:tasks.rados.rados.0.smithi045.stdout:1007: done (1 left) 2024-04-26T00:26:16.879 INFO:tasks.rados.rados.0.smithi045.stdout:1008: done (0 left) 2024-04-26T00:26:16.880 INFO:tasks.rados.rados.0.smithi045.stdout:1009: snap_remove snap 98 2024-04-26T00:26:17.883 INFO:tasks.rados.rados.0.smithi045.stdout:1009: done (0 left) 2024-04-26T00:26:17.884 INFO:tasks.rados.rados.0.smithi045.stdout:1010: read oid 32 snap -1 2024-04-26T00:26:17.884 INFO:tasks.rados.rados.0.smithi045.stdout:1010: expect (ObjNum 136 snap 30 seq_num 136) 2024-04-26T00:26:17.884 INFO:tasks.rados.rados.0.smithi045.stdout:1011: write (excl) oid 36 current snap is 118 2024-04-26T00:26:17.884 INFO:tasks.rados.rados.0.smithi045.stdout:1011: seq_num 390 ranges {554730=481113,1572095=737454} 2024-04-26T00:26:17.900 INFO:tasks.rados.rados.0.smithi045.stdout:1011: writing smithi04526579-36 from 554730 to 1035843 tid 1 2024-04-26T00:26:17.921 INFO:tasks.rados.rados.0.smithi045.stdout:1011: writing smithi04526579-36 from 1572095 to 2309549 tid 2 2024-04-26T00:26:17.921 INFO:tasks.rados.rados.0.smithi045.stdout:1012: snap_create 2024-04-26T00:26:17.922 INFO:tasks.rados.rados.0.smithi045.stdout:1010: expect (ObjNum 136 snap 30 seq_num 136) 2024-04-26T00:26:18.064 INFO:tasks.rados.rados.0.smithi045.stdout:1011: finishing write tid 1 to smithi04526579-36 2024-04-26T00:26:18.064 INFO:tasks.rados.rados.0.smithi045.stdout:1011: finishing write tid 2 to smithi04526579-36 2024-04-26T00:26:18.064 INFO:tasks.rados.rados.0.smithi045.stdout:1011: finishing write tid 3 to smithi04526579-36 2024-04-26T00:26:18.064 INFO:tasks.rados.rados.0.smithi045.stdout:1011: finishing write tid 4 to smithi04526579-36 2024-04-26T00:26:18.064 INFO:tasks.rados.rados.0.smithi045.stdout:1011: oid 36 updating version 0 to 293 2024-04-26T00:26:18.064 INFO:tasks.rados.rados.0.smithi045.stdout:1011: oid 36 version 293 is already newer than 292 2024-04-26T00:26:18.064 INFO:tasks.rados.rados.0.smithi045.stdout:1011: oid 36 version 293 is already newer than 291 2024-04-26T00:26:18.064 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 293 (ObjNum 390 snap 118 seq_num 390) dirty exists 2024-04-26T00:26:18.065 INFO:tasks.rados.rados.0.smithi045.stdout:1011: left oid 36 (ObjNum 390 snap 118 seq_num 390) 2024-04-26T00:26:18.885 INFO:tasks.rados.rados.0.smithi045.stdout:1010: done (2 left) 2024-04-26T00:26:18.885 INFO:tasks.rados.rados.0.smithi045.stdout:1011: done (1 left) 2024-04-26T00:26:18.885 INFO:tasks.rados.rados.0.smithi045.stdout:1012: done (0 left) 2024-04-26T00:26:18.885 INFO:tasks.rados.rados.0.smithi045.stdout:1013: read oid 41 snap 99 2024-04-26T00:26:18.885 INFO:tasks.rados.rados.0.smithi045.stdout:1013: expect deleted 2024-04-26T00:26:18.886 INFO:tasks.rados.rados.0.smithi045.stdout:1014: rollback oid 36 current snap is 119 2024-04-26T00:26:18.886 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 36 to 112 2024-04-26T00:26:18.886 INFO:tasks.rados.rados.0.smithi045.stdout:1015: snap_create 2024-04-26T00:26:18.899 INFO:tasks.rados.rados.0.smithi045.stdout:1014: finishing rollback tid 0 to smithi04526579-36 2024-04-26T00:26:18.901 INFO:tasks.rados.rados.0.smithi045.stdout:1014: finishing rollback tid 1 to smithi04526579-36 2024-04-26T00:26:18.902 INFO:tasks.rados.rados.0.smithi045.stdout:1014: finishing rollback tid 2 to smithi04526579-36 2024-04-26T00:26:18.902 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 297 (ObjNum 304 snap 87 seq_num 304) dirty exists 2024-04-26T00:26:19.889 INFO:tasks.rados.rados.0.smithi045.stdout:1013: done (2 left) 2024-04-26T00:26:19.889 INFO:tasks.rados.rados.0.smithi045.stdout:1014: done (1 left) 2024-04-26T00:26:19.889 INFO:tasks.rados.rados.0.smithi045.stdout:1015: done (0 left) 2024-04-26T00:26:19.889 INFO:tasks.rados.rados.0.smithi045.stdout:1016: read oid 31 snap 97 2024-04-26T00:26:19.889 INFO:tasks.rados.rados.0.smithi045.stdout:1016: expect (ObjNum 303 snap 87 seq_num 303) 2024-04-26T00:26:19.889 INFO:tasks.rados.rados.0.smithi045.stdout:1017: copy_from oid 3 from oid 47 current snap is 120 2024-04-26T00:26:19.889 INFO:tasks.rados.rados.0.smithi045.stdout:1018: copy_from oid 7 from oid 42 current snap is 120 2024-04-26T00:26:19.889 INFO:tasks.rados.rados.0.smithi045.stdout:1019: copy_from oid 12 from oid 15 current snap is 120 2024-04-26T00:26:19.890 INFO:tasks.rados.rados.0.smithi045.stdout:1020: copy_from oid 48 from oid 30 current snap is 120 2024-04-26T00:26:19.890 INFO:tasks.rados.rados.0.smithi045.stdout:1021: read oid 39 snap 104 2024-04-26T00:26:19.890 INFO:tasks.rados.rados.0.smithi045.stdout:1021: expect (ObjNum 155 snap 33 seq_num 155) 2024-04-26T00:26:19.890 INFO:tasks.rados.rados.0.smithi045.stdout:1022: read oid 28 snap -1 2024-04-26T00:26:19.891 INFO:tasks.rados.rados.0.smithi045.stdout:1022: expect deleted 2024-04-26T00:26:19.891 INFO:tasks.rados.rados.0.smithi045.stdout:1023: read oid 9 snap -1 2024-04-26T00:26:19.891 INFO:tasks.rados.rados.0.smithi045.stdout:1023: expect (ObjNum 371 snap 112 seq_num 371) 2024-04-26T00:26:19.891 INFO:tasks.rados.rados.0.smithi045.stdout:1024: read oid 41 snap 69 2024-04-26T00:26:19.891 INFO:tasks.rados.rados.0.smithi045.stdout:1024: expect (ObjNum 224 snap 59 seq_num 224) 2024-04-26T00:26:19.891 INFO:tasks.rados.rados.0.smithi045.stdout:1025: write oid 38 current snap is 120 2024-04-26T00:26:19.891 INFO:tasks.rados.rados.0.smithi045.stdout:1025: seq_num 391 ranges {571537=556362,1630577=597465} 2024-04-26T00:26:19.915 INFO:tasks.rados.rados.0.smithi045.stdout:1025: writing smithi04526579-38 from 571537 to 1127899 tid 1 2024-04-26T00:26:19.941 INFO:tasks.rados.rados.0.smithi045.stdout:1025: writing smithi04526579-38 from 1630577 to 2228042 tid 2 2024-04-26T00:26:19.941 INFO:tasks.rados.rados.0.smithi045.stdout:1026: rollback oid 26 current snap is 120 2024-04-26T00:26:19.941 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 26 to 119 2024-04-26T00:26:19.941 INFO:tasks.rados.rados.0.smithi045.stdout:1024: expect (ObjNum 224 snap 59 seq_num 224) 2024-04-26T00:26:20.112 INFO:tasks.rados.rados.0.smithi045.stdout:1019: finishing copy_from racing read to smithi04526579-12 2024-04-26T00:26:20.112 INFO:tasks.rados.rados.0.smithi045.stdout:1016: expect (ObjNum 303 snap 87 seq_num 303) 2024-04-26T00:26:20.213 INFO:tasks.rados.rados.0.smithi045.stdout:1019: finishing copy_from to smithi04526579-12 2024-04-26T00:26:20.213 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 587 (ObjNum 376 snap 115 seq_num 376) dirty exists 2024-04-26T00:26:20.213 INFO:tasks.rados.rados.0.smithi045.stdout:1021: expect (ObjNum 155 snap 33 seq_num 155) 2024-04-26T00:26:20.298 INFO:tasks.rados.rados.0.smithi045.stdout:1023: expect (ObjNum 371 snap 112 seq_num 371) 2024-04-26T00:26:20.401 INFO:tasks.rados.rados.0.smithi045.stdout:1025: finishing write tid 1 to smithi04526579-38 2024-04-26T00:26:20.401 INFO:tasks.rados.rados.0.smithi045.stdout:1018: finishing copy_from racing read to smithi04526579-7 2024-04-26T00:26:20.401 INFO:tasks.rados.rados.0.smithi045.stdout:1018: finishing copy_from to smithi04526579-7 2024-04-26T00:26:20.401 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 357 (ObjNum 114 snap 28 seq_num 114) dirty exists 2024-04-26T00:26:20.402 INFO:tasks.rados.rados.0.smithi045.stdout:1020: finishing copy_from racing read to smithi04526579-48 2024-04-26T00:26:20.402 INFO:tasks.rados.rados.0.smithi045.stdout:1017: finishing copy_from racing read to smithi04526579-3 2024-04-26T00:26:20.402 INFO:tasks.rados.rados.0.smithi045.stdout:1016: done (10 left) 2024-04-26T00:26:20.402 INFO:tasks.rados.rados.0.smithi045.stdout:1018: done (9 left) 2024-04-26T00:26:20.402 INFO:tasks.rados.rados.0.smithi045.stdout:1019: done (8 left) 2024-04-26T00:26:20.402 INFO:tasks.rados.rados.0.smithi045.stdout:1021: done (7 left) 2024-04-26T00:26:20.402 INFO:tasks.rados.rados.0.smithi045.stdout:1022: done (6 left) 2024-04-26T00:26:20.402 INFO:tasks.rados.rados.0.smithi045.stdout:1023: done (5 left) 2024-04-26T00:26:20.402 INFO:tasks.rados.rados.0.smithi045.stdout:1024: done (4 left) 2024-04-26T00:26:20.402 INFO:tasks.rados.rados.0.smithi045.stdout:1027: snap_remove snap 111 2024-04-26T00:26:20.904 INFO:tasks.rados.rados.0.smithi045.stdout:1027: done (4 left) 2024-04-26T00:26:20.904 INFO:tasks.rados.rados.0.smithi045.stdout:1028: write oid 40 current snap is 120 2024-04-26T00:26:20.904 INFO:tasks.rados.rados.0.smithi045.stdout:1017: finishing copy_from to smithi04526579-3 2024-04-26T00:26:20.904 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 530 (ObjNum 389 snap 117 seq_num 389) dirty exists 2024-04-26T00:26:20.904 INFO:tasks.rados.rados.0.smithi045.stdout:1025: finishing write tid 2 to smithi04526579-38 2024-04-26T00:26:20.904 INFO:tasks.rados.rados.0.smithi045.stdout:1028: seq_num 392 ranges {588344=631611,1689059=741123} 2024-04-26T00:26:20.925 INFO:tasks.rados.rados.0.smithi045.stdout:1028: writing smithi04526579-40 from 588344 to 1219955 tid 1 2024-04-26T00:26:20.941 INFO:tasks.rados.rados.0.smithi045.stdout:1028: writing smithi04526579-40 from 1689059 to 2430182 tid 2 2024-04-26T00:26:20.941 INFO:tasks.rados.rados.0.smithi045.stdout:1017: done (4 left) 2024-04-26T00:26:20.941 INFO:tasks.rados.rados.0.smithi045.stdout:1029: write oid 1 current snap is 120 2024-04-26T00:26:20.941 INFO:tasks.rados.rados.0.smithi045.stdout:1029: seq_num 393 ranges {605151=706860,2031188=573963} 2024-04-26T00:26:20.958 INFO:tasks.rados.rados.0.smithi045.stdout:1029: writing smithi04526579-1 from 605151 to 1312011 tid 1 2024-04-26T00:26:20.973 INFO:tasks.rados.rados.0.smithi045.stdout:1029: writing smithi04526579-1 from 2031188 to 2605151 tid 2 2024-04-26T00:26:20.973 INFO:tasks.rados.rados.0.smithi045.stdout:1030: read oid 13 snap 114 2024-04-26T00:26:20.973 INFO:tasks.rados.rados.0.smithi045.stdout:1030: expect (ObjNum 337 snap 100 seq_num 337) 2024-04-26T00:26:20.973 INFO:tasks.rados.rados.0.smithi045.stdout:1031: read oid 22 snap 118 2024-04-26T00:26:20.973 INFO:tasks.rados.rados.0.smithi045.stdout:1031: expect (ObjNum 330 snap 96 seq_num 330) 2024-04-26T00:26:20.973 INFO:tasks.rados.rados.0.smithi045.stdout:1032: snap_remove snap 112 2024-04-26T00:26:21.883 INFO:tasks.rados.rados.0.smithi045.stdout:1032: done (7 left) 2024-04-26T00:26:21.883 INFO:tasks.rados.rados.0.smithi045.stdout:1033: rollback oid 33 current snap is 120 2024-04-26T00:26:21.883 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 33 to 32 2024-04-26T00:26:21.883 INFO:tasks.rados.rados.0.smithi045.stdout:1020: finishing copy_from to smithi04526579-48 2024-04-26T00:26:21.883 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 309 (ObjNum 359 snap 107 seq_num 359) dirty exists 2024-04-26T00:26:21.883 INFO:tasks.rados.rados.0.smithi045.stdout:1025: finishing write tid 3 to smithi04526579-38 2024-04-26T00:26:21.883 INFO:tasks.rados.rados.0.smithi045.stdout:1025: finishing write tid 4 to smithi04526579-38 2024-04-26T00:26:21.883 INFO:tasks.rados.rados.0.smithi045.stdout:1025: oid 38 updating version 0 to 364 2024-04-26T00:26:21.883 INFO:tasks.rados.rados.0.smithi045.stdout:1025: oid 38 version 364 is already newer than 363 2024-04-26T00:26:21.883 INFO:tasks.rados.rados.0.smithi045.stdout:1025: oid 38 version 364 is already newer than 362 2024-04-26T00:26:21.883 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 364 (ObjNum 391 snap 120 seq_num 391) dirty exists 2024-04-26T00:26:21.884 INFO:tasks.rados.rados.0.smithi045.stdout:1025: left oid 38 (ObjNum 391 snap 120 seq_num 391) 2024-04-26T00:26:21.884 INFO:tasks.rados.rados.0.smithi045.stdout:1026: finishing rollback tid 0 to smithi04526579-26 2024-04-26T00:26:21.884 INFO:tasks.rados.rados.0.smithi045.stdout:1026: finishing rollback tid 1 to smithi04526579-26 2024-04-26T00:26:21.884 INFO:tasks.rados.rados.0.smithi045.stdout:1026: finishing rollback tid 2 to smithi04526579-26 2024-04-26T00:26:21.884 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 601 (ObjNum 254 snap 64 seq_num 254) dirty exists 2024-04-26T00:26:21.884 INFO:tasks.rados.rados.0.smithi045.stdout:1028: finishing write tid 1 to smithi04526579-40 2024-04-26T00:26:21.884 INFO:tasks.rados.rados.0.smithi045.stdout:1028: finishing write tid 2 to smithi04526579-40 2024-04-26T00:26:21.884 INFO:tasks.rados.rados.0.smithi045.stdout:1028: finishing write tid 3 to smithi04526579-40 2024-04-26T00:26:21.884 INFO:tasks.rados.rados.0.smithi045.stdout:1028: finishing write tid 4 to smithi04526579-40 2024-04-26T00:26:21.884 INFO:tasks.rados.rados.0.smithi045.stdout:1028: oid 40 updating version 0 to 590 2024-04-26T00:26:21.884 INFO:tasks.rados.rados.0.smithi045.stdout:1028: oid 40 version 590 is already newer than 589 2024-04-26T00:26:21.884 INFO:tasks.rados.rados.0.smithi045.stdout:1028: oid 40 version 590 is already newer than 588 2024-04-26T00:26:21.884 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 590 (ObjNum 392 snap 120 seq_num 392) dirty exists 2024-04-26T00:26:21.884 INFO:tasks.rados.rados.0.smithi045.stdout:1028: left oid 40 (ObjNum 392 snap 120 seq_num 392) 2024-04-26T00:26:21.884 INFO:tasks.rados.rados.0.smithi045.stdout:1029: finishing write tid 1 to smithi04526579-1 2024-04-26T00:26:21.884 INFO:tasks.rados.rados.0.smithi045.stdout:1031: expect (ObjNum 330 snap 96 seq_num 330) 2024-04-26T00:26:22.008 INFO:tasks.rados.rados.0.smithi045.stdout:1029: finishing write tid 2 to smithi04526579-1 2024-04-26T00:26:22.008 INFO:tasks.rados.rados.0.smithi045.stdout:1029: finishing write tid 3 to smithi04526579-1 2024-04-26T00:26:22.008 INFO:tasks.rados.rados.0.smithi045.stdout:1029: finishing write tid 4 to smithi04526579-1 2024-04-26T00:26:22.008 INFO:tasks.rados.rados.0.smithi045.stdout:1029: oid 1 updating version 0 to 606 2024-04-26T00:26:22.008 INFO:tasks.rados.rados.0.smithi045.stdout:1029: oid 1 updating version 606 to 607 2024-04-26T00:26:22.008 INFO:tasks.rados.rados.0.smithi045.stdout:1029: oid 1 version 607 is already newer than 605 2024-04-26T00:26:22.008 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 607 (ObjNum 393 snap 120 seq_num 393) dirty exists 2024-04-26T00:26:22.008 INFO:tasks.rados.rados.0.smithi045.stdout:1029: left oid 1 (ObjNum 393 snap 120 seq_num 393) 2024-04-26T00:26:22.008 INFO:tasks.rados.rados.0.smithi045.stdout:1030: expect (ObjNum 337 snap 100 seq_num 337) 2024-04-26T00:26:22.179 INFO:tasks.rados.rados.0.smithi045.stdout:1033: finishing rollback tid 0 to smithi04526579-33 2024-04-26T00:26:22.179 INFO:tasks.rados.rados.0.smithi045.stdout:1033: finishing rollback tid 1 to smithi04526579-33 2024-04-26T00:26:22.179 INFO:tasks.rados.rados.0.smithi045.stdout:1033: finishing rollback tid 2 to smithi04526579-33 2024-04-26T00:26:22.179 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 710 (ObjNum 150 snap 32 seq_num 150) dirty exists 2024-04-26T00:26:22.179 INFO:tasks.rados.rados.0.smithi045.stdout:1020: done (7 left) 2024-04-26T00:26:22.179 INFO:tasks.rados.rados.0.smithi045.stdout:1025: done (6 left) 2024-04-26T00:26:22.179 INFO:tasks.rados.rados.0.smithi045.stdout:1026: done (5 left) 2024-04-26T00:26:22.179 INFO:tasks.rados.rados.0.smithi045.stdout:1028: done (4 left) 2024-04-26T00:26:22.180 INFO:tasks.rados.rados.0.smithi045.stdout:1029: done (3 left) 2024-04-26T00:26:22.180 INFO:tasks.rados.rados.0.smithi045.stdout:1030: done (2 left) 2024-04-26T00:26:22.180 INFO:tasks.rados.rados.0.smithi045.stdout:1031: done (1 left) 2024-04-26T00:26:22.180 INFO:tasks.rados.rados.0.smithi045.stdout:1033: done (0 left) 2024-04-26T00:26:22.180 INFO:tasks.rados.rados.0.smithi045.stdout:1034: write oid 41 current snap is 120 2024-04-26T00:26:22.180 INFO:tasks.rados.rados.0.smithi045.stdout:1034: seq_num 394 ranges {621958=782109,2089670=532288} 2024-04-26T00:26:22.202 INFO:tasks.rados.rados.0.smithi045.stdout:1034: writing smithi04526579-41 from 621958 to 1404067 tid 1 2024-04-26T00:26:22.223 INFO:tasks.rados.rados.0.smithi045.stdout:1034: writing smithi04526579-41 from 2089670 to 2621958 tid 2 2024-04-26T00:26:22.223 INFO:tasks.rados.rados.0.smithi045.stdout:1035: snap_remove snap 114 2024-04-26T00:26:22.924 INFO:tasks.rados.rados.0.smithi045.stdout:1035: done (1 left) 2024-04-26T00:26:22.924 INFO:tasks.rados.rados.0.smithi045.stdout:1036: write oid 16 current snap is 120 2024-04-26T00:26:22.924 INFO:tasks.rados.rados.0.smithi045.stdout:1036: seq_num 395 ranges {638765=457358,1748152=604803} 2024-04-26T00:26:22.940 INFO:tasks.rados.rados.0.smithi045.stdout:1036: writing smithi04526579-16 from 638765 to 1096123 tid 1 2024-04-26T00:26:22.960 INFO:tasks.rados.rados.0.smithi045.stdout:1036: writing smithi04526579-16 from 1748152 to 2352955 tid 2 2024-04-26T00:26:22.960 INFO:tasks.rados.rados.0.smithi045.stdout:1037: delete oid 19 current snap is 120 2024-04-26T00:26:22.960 INFO:tasks.rados.rados.0.smithi045.stdout:1034: finishing write tid 1 to smithi04526579-41 2024-04-26T00:26:22.960 INFO:tasks.rados.rados.0.smithi045.stdout:1034: finishing write tid 2 to smithi04526579-41 2024-04-26T00:26:22.960 INFO:tasks.rados.rados.0.smithi045.stdout:1034: finishing write tid 3 to smithi04526579-41 2024-04-26T00:26:22.960 INFO:tasks.rados.rados.0.smithi045.stdout:1034: finishing write tid 4 to smithi04526579-41 2024-04-26T00:26:22.960 INFO:tasks.rados.rados.0.smithi045.stdout:1034: oid 41 updating version 0 to 613 2024-04-26T00:26:22.960 INFO:tasks.rados.rados.0.smithi045.stdout:1034: oid 41 updating version 613 to 614 2024-04-26T00:26:22.960 INFO:tasks.rados.rados.0.smithi045.stdout:1034: oid 41 updating version 614 to 615 2024-04-26T00:26:22.961 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 615 (ObjNum 394 snap 120 seq_num 394) dirty exists 2024-04-26T00:26:22.961 INFO:tasks.rados.rados.0.smithi045.stdout:1034: left oid 41 (ObjNum 394 snap 120 seq_num 394) 2024-04-26T00:26:22.961 INFO:tasks.rados.rados.0.smithi045.stdout:1036: finishing write tid 1 to smithi04526579-16 2024-04-26T00:26:22.964 INFO:tasks.rados.rados.0.smithi045.stdout:1034: done (2 left) 2024-04-26T00:26:22.964 INFO:tasks.rados.rados.0.smithi045.stdout:1037: done (1 left) 2024-04-26T00:26:22.965 INFO:tasks.rados.rados.0.smithi045.stdout:1038: write (excl) oid 2 current snap is 120 2024-04-26T00:26:22.965 INFO:tasks.rados.rados.0.smithi045.stdout:1038: seq_num 397 ranges {672379=724209,1865116=608472} 2024-04-26T00:26:22.985 INFO:tasks.rados.rados.0.smithi045.stdout:1038: writing smithi04526579-2 from 672379 to 1396588 tid 1 2024-04-26T00:26:23.001 INFO:tasks.rados.rados.0.smithi045.stdout:1038: writing smithi04526579-2 from 1865116 to 2473588 tid 2 2024-04-26T00:26:23.001 INFO:tasks.rados.rados.0.smithi045.stdout:1039: copy_from oid 18 from oid 11 current snap is 120 2024-04-26T00:26:23.001 INFO:tasks.rados.rados.0.smithi045.stdout:1036: finishing write tid 2 to smithi04526579-16 2024-04-26T00:26:23.002 INFO:tasks.rados.rados.0.smithi045.stdout:1040: write (excl) oid 40 current snap is 120 2024-04-26T00:26:23.002 INFO:tasks.rados.rados.0.smithi045.stdout:1036: finishing write tid 3 to smithi04526579-16 2024-04-26T00:26:23.002 INFO:tasks.rados.rados.0.smithi045.stdout:1040: seq_num 398 ranges {689186=799458,1923598=752130} 2024-04-26T00:26:23.018 INFO:tasks.rados.rados.0.smithi045.stdout:1040: writing smithi04526579-40 from 689186 to 1488644 tid 1 2024-04-26T00:26:23.031 INFO:tasks.rados.rados.0.smithi045.stdout:1040: writing smithi04526579-40 from 1923598 to 2675728 tid 2 2024-04-26T00:26:23.031 INFO:tasks.rados.rados.0.smithi045.stdout:1041: read oid 5 snap -1 2024-04-26T00:26:23.031 INFO:tasks.rados.rados.0.smithi045.stdout:1041: expect deleted 2024-04-26T00:26:23.031 INFO:tasks.rados.rados.0.smithi045.stdout:1042: snap_create 2024-04-26T00:26:23.031 INFO:tasks.rados.rados.0.smithi045.stdout:1036: finishing write tid 4 to smithi04526579-16 2024-04-26T00:26:23.031 INFO:tasks.rados.rados.0.smithi045.stdout:1036: oid 16 updating version 0 to 165 2024-04-26T00:26:23.031 INFO:tasks.rados.rados.0.smithi045.stdout:1036: oid 16 updating version 165 to 167 2024-04-26T00:26:23.031 INFO:tasks.rados.rados.0.smithi045.stdout:1036: oid 16 version 167 is already newer than 166 2024-04-26T00:26:23.031 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 167 (ObjNum 395 snap 120 seq_num 395) dirty exists 2024-04-26T00:26:23.031 INFO:tasks.rados.rados.0.smithi045.stdout:1036: left oid 16 (ObjNum 395 snap 120 seq_num 395) 2024-04-26T00:26:23.031 INFO:tasks.rados.rados.0.smithi045.stdout:1038: finishing write tid 1 to smithi04526579-2 2024-04-26T00:26:23.031 INFO:tasks.rados.rados.0.smithi045.stdout:1039: finishing copy_from to smithi04526579-18 2024-04-26T00:26:23.032 INFO:tasks.rados.rados.0.smithi045.stdout:1039: got expected ENOENT (src dne) 2024-04-26T00:26:23.032 INFO:tasks.rados.rados.0.smithi045.stdout:1039: finishing copy_from racing read to smithi04526579-18 2024-04-26T00:26:23.032 INFO:tasks.rados.rados.0.smithi045.stdout:1038: finishing write tid 2 to smithi04526579-2 2024-04-26T00:26:23.032 INFO:tasks.rados.rados.0.smithi045.stdout:1038: finishing write tid 3 to smithi04526579-2 2024-04-26T00:26:23.032 INFO:tasks.rados.rados.0.smithi045.stdout:1038: finishing write tid 4 to smithi04526579-2 2024-04-26T00:26:23.032 INFO:tasks.rados.rados.0.smithi045.stdout:1038: oid 2 updating version 0 to 527 2024-04-26T00:26:23.032 INFO:tasks.rados.rados.0.smithi045.stdout:1038: oid 2 version 527 is already newer than 526 2024-04-26T00:26:23.032 INFO:tasks.rados.rados.0.smithi045.stdout:1038: oid 2 version 527 is already newer than 525 2024-04-26T00:26:23.032 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 527 (ObjNum 397 snap 120 seq_num 397) dirty exists 2024-04-26T00:26:23.032 INFO:tasks.rados.rados.0.smithi045.stdout:1038: left oid 2 (ObjNum 397 snap 120 seq_num 397) 2024-04-26T00:26:23.032 INFO:tasks.rados.rados.0.smithi045.stdout:1040: finishing write tid 1 to smithi04526579-40 2024-04-26T00:26:23.039 INFO:tasks.rados.rados.0.smithi045.stdout:1040: finishing write tid 2 to smithi04526579-40 2024-04-26T00:26:23.039 INFO:tasks.rados.rados.0.smithi045.stdout:1040: finishing write tid 3 to smithi04526579-40 2024-04-26T00:26:23.040 INFO:tasks.rados.rados.0.smithi045.stdout:1040: finishing write tid 4 to smithi04526579-40 2024-04-26T00:26:23.040 INFO:tasks.rados.rados.0.smithi045.stdout:1040: oid 40 updating version 0 to 593 2024-04-26T00:26:23.040 INFO:tasks.rados.rados.0.smithi045.stdout:1040: oid 40 version 593 is already newer than 592 2024-04-26T00:26:23.040 INFO:tasks.rados.rados.0.smithi045.stdout:1040: oid 40 version 593 is already newer than 591 2024-04-26T00:26:23.040 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 593 (ObjNum 398 snap 120 seq_num 398) dirty exists 2024-04-26T00:26:23.040 INFO:tasks.rados.rados.0.smithi045.stdout:1040: left oid 40 (ObjNum 398 snap 120 seq_num 398) 2024-04-26T00:26:23.939 INFO:tasks.rados.rados.0.smithi045.stdout:1036: done (5 left) 2024-04-26T00:26:23.939 INFO:tasks.rados.rados.0.smithi045.stdout:1038: done (4 left) 2024-04-26T00:26:23.939 INFO:tasks.rados.rados.0.smithi045.stdout:1039: done (3 left) 2024-04-26T00:26:23.939 INFO:tasks.rados.rados.0.smithi045.stdout:1040: done (2 left) 2024-04-26T00:26:23.939 INFO:tasks.rados.rados.0.smithi045.stdout:1041: done (1 left) 2024-04-26T00:26:23.939 INFO:tasks.rados.rados.0.smithi045.stdout:1042: done (0 left) 2024-04-26T00:26:23.939 INFO:tasks.rados.rados.0.smithi045.stdout:1043: snap_remove snap 119 2024-04-26T00:26:24.947 INFO:tasks.rados.rados.0.smithi045.stdout:1043: done (0 left) 2024-04-26T00:26:24.947 INFO:tasks.rados.rados.0.smithi045.stdout:1044: snap_remove snap 110 2024-04-26T00:26:25.956 INFO:tasks.rados.rados.0.smithi045.stdout:1044: done (0 left) 2024-04-26T00:26:25.956 INFO:tasks.rados.rados.0.smithi045.stdout:1045: snap_create 2024-04-26T00:26:26.885 INFO:tasks.rados.rados.0.smithi045.stdout:1045: done (0 left) 2024-04-26T00:26:26.885 INFO:tasks.rados.rados.0.smithi045.stdout:1046: rollback oid 49 current snap is 122 2024-04-26T00:26:26.885 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 49 to 92 2024-04-26T00:26:26.885 INFO:tasks.rados.rados.0.smithi045.stdout:1047: copy_from oid 28 from oid 12 current snap is 122 2024-04-26T00:26:26.885 INFO:tasks.rados.rados.0.smithi045.stdout:1048: copy_from oid 37 from oid 32 current snap is 122 2024-04-26T00:26:26.885 INFO:tasks.rados.rados.0.smithi045.stdout:1049: read oid 30 snap 32 2024-04-26T00:26:26.885 INFO:tasks.rados.rados.0.smithi045.stdout:1049: expect (ObjNum 149 snap 32 seq_num 149) 2024-04-26T00:26:26.886 INFO:tasks.rados.rados.0.smithi045.stdout:1050: write (excl) oid 34 current snap is 122 2024-04-26T00:26:26.886 INFO:tasks.rados.rados.0.smithi045.stdout:1050: seq_num 399 ranges {705993=474707,1582080=612141} 2024-04-26T00:26:26.904 INFO:tasks.rados.rados.0.smithi045.stdout:1050: writing smithi04526579-34 from 705993 to 1180700 tid 1 2024-04-26T00:26:26.920 INFO:tasks.rados.rados.0.smithi045.stdout:1050: writing smithi04526579-34 from 1582080 to 2194221 tid 2 2024-04-26T00:26:26.920 INFO:tasks.rados.rados.0.smithi045.stdout:1051: write oid 5 current snap is 122 2024-04-26T00:26:26.920 INFO:tasks.rados.rados.0.smithi045.stdout:1051: seq_num 400 ranges {722800=549956,2040562=682238} 2024-04-26T00:26:26.937 INFO:tasks.rados.rados.0.smithi045.stdout:1051: writing smithi04526579-5 from 722800 to 1272756 tid 1 2024-04-26T00:26:26.954 INFO:tasks.rados.rados.0.smithi045.stdout:1051: writing smithi04526579-5 from 2040562 to 2722800 tid 2 2024-04-26T00:26:26.955 INFO:tasks.rados.rados.0.smithi045.stdout:1052: write oid 11 current snap is 122 2024-04-26T00:26:26.955 INFO:tasks.rados.rados.0.smithi045.stdout:1052: seq_num 401 ranges {739607=625205,1982691=499457} 2024-04-26T00:26:26.966 INFO:tasks.rados.rados.0.smithi045.stdout:1052: writing smithi04526579-11 from 739607 to 1364812 tid 1 2024-04-26T00:26:26.976 INFO:tasks.rados.rados.0.smithi045.stdout:1052: writing smithi04526579-11 from 1982691 to 2482148 tid 2 2024-04-26T00:26:26.976 INFO:tasks.rados.rados.0.smithi045.stdout:1053: read oid 17 snap -1 2024-04-26T00:26:26.977 INFO:tasks.rados.rados.0.smithi045.stdout:1053: expect (ObjNum 360 snap 107 seq_num 360) 2024-04-26T00:26:26.977 INFO:tasks.rados.rados.0.smithi045.stdout:1054: write (excl) oid 4 current snap is 122 2024-04-26T00:26:26.977 INFO:tasks.rados.rados.0.smithi045.stdout:1054: seq_num 402 ranges {756414=700454,2041173=715241} 2024-04-26T00:26:26.987 INFO:tasks.rados.rados.0.smithi045.stdout:1054: writing smithi04526579-4 from 756414 to 1456868 tid 1 2024-04-26T00:26:26.996 INFO:tasks.rados.rados.0.smithi045.stdout:1054: writing smithi04526579-4 from 2041173 to 2756414 tid 2 2024-04-26T00:26:26.996 INFO:tasks.rados.rados.0.smithi045.stdout:1055: write (excl) oid 33 current snap is 122 2024-04-26T00:26:26.996 INFO:tasks.rados.rados.0.smithi045.stdout:1055: seq_num 403 ranges {773221=492056,1816008=503126} 2024-04-26T00:26:27.004 INFO:tasks.rados.rados.0.smithi045.stdout:1055: writing smithi04526579-33 from 773221 to 1265277 tid 1 2024-04-26T00:26:27.010 INFO:tasks.rados.rados.0.smithi045.stdout:1055: writing smithi04526579-33 from 1816008 to 2319134 tid 2 2024-04-26T00:26:27.010 INFO:tasks.rados.rados.0.smithi045.stdout:1056: read oid 10 snap -1 2024-04-26T00:26:27.010 INFO:tasks.rados.rados.0.smithi045.stdout:1056: expect (ObjNum 283 snap 81 seq_num 283) 2024-04-26T00:26:27.010 INFO:tasks.rados.rados.0.smithi045.stdout:1057: copy_from oid 21 from oid 1 current snap is 122 2024-04-26T00:26:27.010 INFO:tasks.rados.rados.0.smithi045.stdout:1049: expect (ObjNum 149 snap 32 seq_num 149) 2024-04-26T00:26:27.143 INFO:tasks.rados.rados.0.smithi045.stdout:1046: finishing rollback tid 0 to smithi04526579-49 2024-04-26T00:26:27.144 INFO:tasks.rados.rados.0.smithi045.stdout:1046: finishing rollback tid 1 to smithi04526579-49 2024-04-26T00:26:27.144 INFO:tasks.rados.rados.0.smithi045.stdout:1046: finishing rollback tid 2 to smithi04526579-49 2024-04-26T00:26:27.144 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 558 (ObjNum 256 snap 66 seq_num 256) dirty exists 2024-04-26T00:26:27.144 INFO:tasks.rados.rados.0.smithi045.stdout:1047: finishing copy_from racing read to smithi04526579-28 2024-04-26T00:26:27.144 INFO:tasks.rados.rados.0.smithi045.stdout:1047: finishing copy_from to smithi04526579-28 2024-04-26T00:26:27.144 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 587 (ObjNum 376 snap 115 seq_num 376) dirty exists 2024-04-26T00:26:27.144 INFO:tasks.rados.rados.0.smithi045.stdout:1048: finishing copy_from racing read to smithi04526579-37 2024-04-26T00:26:27.144 INFO:tasks.rados.rados.0.smithi045.stdout:1048: finishing copy_from to smithi04526579-37 2024-04-26T00:26:27.144 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 385 (ObjNum 136 snap 30 seq_num 136) dirty exists 2024-04-26T00:26:27.144 INFO:tasks.rados.rados.0.smithi045.stdout:1050: finishing write tid 1 to smithi04526579-34 2024-04-26T00:26:27.144 INFO:tasks.rados.rados.0.smithi045.stdout:1050: finishing write tid 2 to smithi04526579-34 2024-04-26T00:26:27.144 INFO:tasks.rados.rados.0.smithi045.stdout:1050: finishing write tid 3 to smithi04526579-34 2024-04-26T00:26:27.144 INFO:tasks.rados.rados.0.smithi045.stdout:1050: finishing write tid 4 to smithi04526579-34 2024-04-26T00:26:27.145 INFO:tasks.rados.rados.0.smithi045.stdout:1050: oid 34 updating version 0 to 561 2024-04-26T00:26:27.145 INFO:tasks.rados.rados.0.smithi045.stdout:1050: oid 34 version 561 is already newer than 560 2024-04-26T00:26:27.145 INFO:tasks.rados.rados.0.smithi045.stdout:1050: oid 34 updating version 561 to 562 2024-04-26T00:26:27.145 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 562 (ObjNum 399 snap 122 seq_num 399) dirty exists 2024-04-26T00:26:27.145 INFO:tasks.rados.rados.0.smithi045.stdout:1050: left oid 34 (ObjNum 399 snap 122 seq_num 399) 2024-04-26T00:26:27.145 INFO:tasks.rados.rados.0.smithi045.stdout:1051: finishing write tid 1 to smithi04526579-5 2024-04-26T00:26:27.145 INFO:tasks.rados.rados.0.smithi045.stdout:1051: finishing write tid 2 to smithi04526579-5 2024-04-26T00:26:27.145 INFO:tasks.rados.rados.0.smithi045.stdout:1051: finishing write tid 3 to smithi04526579-5 2024-04-26T00:26:27.145 INFO:tasks.rados.rados.0.smithi045.stdout:1051: finishing write tid 4 to smithi04526579-5 2024-04-26T00:26:27.145 INFO:tasks.rados.rados.0.smithi045.stdout:1051: oid 5 updating version 0 to 740 2024-04-26T00:26:27.145 INFO:tasks.rados.rados.0.smithi045.stdout:1051: oid 5 updating version 740 to 741 2024-04-26T00:26:27.145 INFO:tasks.rados.rados.0.smithi045.stdout:1051: oid 5 version 741 is already newer than 739 2024-04-26T00:26:27.145 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 741 (ObjNum 400 snap 122 seq_num 400) dirty exists 2024-04-26T00:26:27.145 INFO:tasks.rados.rados.0.smithi045.stdout:1051: left oid 5 (ObjNum 400 snap 122 seq_num 400) 2024-04-26T00:26:27.145 INFO:tasks.rados.rados.0.smithi045.stdout:1052: finishing write tid 1 to smithi04526579-11 2024-04-26T00:26:27.146 INFO:tasks.rados.rados.0.smithi045.stdout:1053: expect (ObjNum 360 snap 107 seq_num 360) 2024-04-26T00:26:27.260 INFO:tasks.rados.rados.0.smithi045.stdout:1052: finishing write tid 2 to smithi04526579-11 2024-04-26T00:26:27.260 INFO:tasks.rados.rados.0.smithi045.stdout:1052: finishing write tid 3 to smithi04526579-11 2024-04-26T00:26:27.260 INFO:tasks.rados.rados.0.smithi045.stdout:1052: finishing write tid 4 to smithi04526579-11 2024-04-26T00:26:27.260 INFO:tasks.rados.rados.0.smithi045.stdout:1052: oid 11 updating version 0 to 382 2024-04-26T00:26:27.260 INFO:tasks.rados.rados.0.smithi045.stdout:1052: oid 11 version 382 is already newer than 381 2024-04-26T00:26:27.260 INFO:tasks.rados.rados.0.smithi045.stdout:1052: oid 11 updating version 382 to 383 2024-04-26T00:26:27.261 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 383 (ObjNum 401 snap 122 seq_num 401) dirty exists 2024-04-26T00:26:27.261 INFO:tasks.rados.rados.0.smithi045.stdout:1052: left oid 11 (ObjNum 401 snap 122 seq_num 401) 2024-04-26T00:26:27.261 INFO:tasks.rados.rados.0.smithi045.stdout:1054: finishing write tid 1 to smithi04526579-4 2024-04-26T00:26:27.261 INFO:tasks.rados.rados.0.smithi045.stdout:1054: finishing write tid 2 to smithi04526579-4 2024-04-26T00:26:27.261 INFO:tasks.rados.rados.0.smithi045.stdout:1054: finishing write tid 3 to smithi04526579-4 2024-04-26T00:26:27.261 INFO:tasks.rados.rados.0.smithi045.stdout:1054: finishing write tid 4 to smithi04526579-4 2024-04-26T00:26:27.261 INFO:tasks.rados.rados.0.smithi045.stdout:1054: oid 4 updating version 0 to 537 2024-04-26T00:26:27.261 INFO:tasks.rados.rados.0.smithi045.stdout:1054: oid 4 updating version 537 to 538 2024-04-26T00:26:27.261 INFO:tasks.rados.rados.0.smithi045.stdout:1054: oid 4 version 538 is already newer than 536 2024-04-26T00:26:27.261 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 538 (ObjNum 402 snap 122 seq_num 402) dirty exists 2024-04-26T00:26:27.261 INFO:tasks.rados.rados.0.smithi045.stdout:1054: left oid 4 (ObjNum 402 snap 122 seq_num 402) 2024-04-26T00:26:27.261 INFO:tasks.rados.rados.0.smithi045.stdout:1055: finishing write tid 1 to smithi04526579-33 2024-04-26T00:26:27.261 INFO:tasks.rados.rados.0.smithi045.stdout:1055: finishing write tid 2 to smithi04526579-33 2024-04-26T00:26:27.261 INFO:tasks.rados.rados.0.smithi045.stdout:1056: expect (ObjNum 283 snap 81 seq_num 283) 2024-04-26T00:26:27.440 INFO:tasks.rados.rados.0.smithi045.stdout:1055: finishing write tid 3 to smithi04526579-33 2024-04-26T00:26:27.440 INFO:tasks.rados.rados.0.smithi045.stdout:1055: finishing write tid 4 to smithi04526579-33 2024-04-26T00:26:27.440 INFO:tasks.rados.rados.0.smithi045.stdout:1055: oid 33 updating version 0 to 743 2024-04-26T00:26:27.440 INFO:tasks.rados.rados.0.smithi045.stdout:1055: oid 33 updating version 743 to 745 2024-04-26T00:26:27.441 INFO:tasks.rados.rados.0.smithi045.stdout:1055: oid 33 version 745 is already newer than 744 2024-04-26T00:26:27.441 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 745 (ObjNum 403 snap 122 seq_num 403) dirty exists 2024-04-26T00:26:27.441 INFO:tasks.rados.rados.0.smithi045.stdout:1055: left oid 33 (ObjNum 403 snap 122 seq_num 403) 2024-04-26T00:26:27.441 INFO:tasks.rados.rados.0.smithi045.stdout:1057: finishing copy_from racing read to smithi04526579-21 2024-04-26T00:26:27.441 INFO:tasks.rados.rados.0.smithi045.stdout:1057: finishing copy_from to smithi04526579-21 2024-04-26T00:26:27.441 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 607 (ObjNum 393 snap 120 seq_num 393) dirty exists 2024-04-26T00:26:27.441 INFO:tasks.rados.rados.0.smithi045.stdout:1046: done (11 left) 2024-04-26T00:26:27.441 INFO:tasks.rados.rados.0.smithi045.stdout:1047: done (10 left) 2024-04-26T00:26:27.441 INFO:tasks.rados.rados.0.smithi045.stdout:1048: done (9 left) 2024-04-26T00:26:27.441 INFO:tasks.rados.rados.0.smithi045.stdout:1049: done (8 left) 2024-04-26T00:26:27.441 INFO:tasks.rados.rados.0.smithi045.stdout:1050: done (7 left) 2024-04-26T00:26:27.441 INFO:tasks.rados.rados.0.smithi045.stdout:1051: done (6 left) 2024-04-26T00:26:27.441 INFO:tasks.rados.rados.0.smithi045.stdout:1052: done (5 left) 2024-04-26T00:26:27.441 INFO:tasks.rados.rados.0.smithi045.stdout:1053: done (4 left) 2024-04-26T00:26:27.441 INFO:tasks.rados.rados.0.smithi045.stdout:1054: done (3 left) 2024-04-26T00:26:27.442 INFO:tasks.rados.rados.0.smithi045.stdout:1055: done (2 left) 2024-04-26T00:26:27.442 INFO:tasks.rados.rados.0.smithi045.stdout:1056: done (1 left) 2024-04-26T00:26:27.442 INFO:tasks.rados.rados.0.smithi045.stdout:1057: done (0 left) 2024-04-26T00:26:27.442 INFO:tasks.rados.rados.0.smithi045.stdout:1058: rollback oid 47 current snap is 122 2024-04-26T00:26:27.442 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 47 to 99 2024-04-26T00:26:27.442 INFO:tasks.rados.rados.0.smithi045.stdout:1059: copy_from oid 21 from oid 26 current snap is 122 2024-04-26T00:26:27.442 INFO:tasks.rados.rados.0.smithi045.stdout:1060: snap_create 2024-04-26T00:26:27.449 INFO:tasks.rados.rados.0.smithi045.stdout:1058: finishing rollback tid 0 to smithi04526579-47 2024-04-26T00:26:27.449 INFO:tasks.rados.rados.0.smithi045.stdout:1058: finishing rollback tid 1 to smithi04526579-47 2024-04-26T00:26:27.449 INFO:tasks.rados.rados.0.smithi045.stdout:1058: finishing rollback tid 2 to smithi04526579-47 2024-04-26T00:26:27.449 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 566 (ObjNum 327 snap 95 seq_num 327) dirty exists 2024-04-26T00:26:27.456 INFO:tasks.rados.rados.0.smithi045.stdout:1059: finishing copy_from racing read to smithi04526579-21 2024-04-26T00:26:27.459 INFO:tasks.rados.rados.0.smithi045.stdout:1059: finishing copy_from to smithi04526579-21 2024-04-26T00:26:27.459 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 601 (ObjNum 254 snap 64 seq_num 254) dirty exists 2024-04-26T00:26:27.964 INFO:tasks.rados.rados.0.smithi045.stdout:1058: done (2 left) 2024-04-26T00:26:27.964 INFO:tasks.rados.rados.0.smithi045.stdout:1059: done (1 left) 2024-04-26T00:26:27.964 INFO:tasks.rados.rados.0.smithi045.stdout:1060: done (0 left) 2024-04-26T00:26:27.964 INFO:tasks.rados.rados.0.smithi045.stdout:1061: copy_from oid 14 from oid 26 current snap is 123 2024-04-26T00:26:27.964 INFO:tasks.rados.rados.0.smithi045.stdout:1062: snap_remove snap 97 2024-04-26T00:26:28.973 INFO:tasks.rados.rados.0.smithi045.stdout:1062: done (1 left) 2024-04-26T00:26:28.973 INFO:tasks.rados.rados.0.smithi045.stdout:1063: copy_from oid 49 from oid 44 current snap is 123 2024-04-26T00:26:28.973 INFO:tasks.rados.rados.0.smithi045.stdout:1061: finishing copy_from racing read to smithi04526579-14 2024-04-26T00:26:28.973 INFO:tasks.rados.rados.0.smithi045.stdout:1061: finishing copy_from to smithi04526579-14 2024-04-26T00:26:28.974 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 601 (ObjNum 254 snap 64 seq_num 254) dirty exists 2024-04-26T00:26:28.974 INFO:tasks.rados.rados.0.smithi045.stdout:1061: done (1 left) 2024-04-26T00:26:28.974 INFO:tasks.rados.rados.0.smithi045.stdout:1064: rollback oid 12 current snap is 123 2024-04-26T00:26:28.974 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 12 to 121 2024-04-26T00:26:28.974 INFO:tasks.rados.rados.0.smithi045.stdout:1065: snap_create 2024-04-26T00:26:29.002 INFO:tasks.rados.rados.0.smithi045.stdout:1064: finishing rollback tid 0 to smithi04526579-12 2024-04-26T00:26:29.002 INFO:tasks.rados.rados.0.smithi045.stdout:1064: finishing rollback tid 1 to smithi04526579-12 2024-04-26T00:26:29.003 INFO:tasks.rados.rados.0.smithi045.stdout:1064: finishing rollback tid 2 to smithi04526579-12 2024-04-26T00:26:29.003 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 596 (ObjNum 376 snap 115 seq_num 376) dirty exists 2024-04-26T00:26:29.010 INFO:tasks.rados.rados.0.smithi045.stdout:1063: finishing copy_from racing read to smithi04526579-49 2024-04-26T00:26:29.016 INFO:tasks.rados.rados.0.smithi045.stdout:1063: finishing copy_from to smithi04526579-49 2024-04-26T00:26:29.016 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 178 (ObjNum 283 snap 81 seq_num 283) dirty exists 2024-04-26T00:26:29.981 INFO:tasks.rados.rados.0.smithi045.stdout:1063: done (2 left) 2024-04-26T00:26:29.981 INFO:tasks.rados.rados.0.smithi045.stdout:1064: done (1 left) 2024-04-26T00:26:29.981 INFO:tasks.rados.rados.0.smithi045.stdout:1065: done (0 left) 2024-04-26T00:26:29.982 INFO:tasks.rados.rados.0.smithi045.stdout:1066: read oid 17 snap -1 2024-04-26T00:26:29.982 INFO:tasks.rados.rados.0.smithi045.stdout:1066: expect (ObjNum 360 snap 107 seq_num 360) 2024-04-26T00:26:29.982 INFO:tasks.rados.rados.0.smithi045.stdout:1067: copy_from oid 49 from oid 23 current snap is 124 2024-04-26T00:26:29.982 INFO:tasks.rados.rados.0.smithi045.stdout:1068: read oid 43 snap -1 2024-04-26T00:26:29.982 INFO:tasks.rados.rados.0.smithi045.stdout:1068: expect (ObjNum 359 snap 107 seq_num 359) 2024-04-26T00:26:29.982 INFO:tasks.rados.rados.0.smithi045.stdout:1069: write oid 2 current snap is 124 2024-04-26T00:26:29.982 INFO:tasks.rados.rados.0.smithi045.stdout:1069: seq_num 404 ranges {790028=567305,1874490=763137} 2024-04-26T00:26:29.998 INFO:tasks.rados.rados.0.smithi045.stdout:1069: writing smithi04526579-2 from 790028 to 1357333 tid 1 2024-04-26T00:26:30.012 INFO:tasks.rados.rados.0.smithi045.stdout:1069: writing smithi04526579-2 from 1874490 to 2637627 tid 2 2024-04-26T00:26:30.012 INFO:tasks.rados.rados.0.smithi045.stdout:1070: read oid 1 snap -1 2024-04-26T00:26:30.012 INFO:tasks.rados.rados.0.smithi045.stdout:1070: expect (ObjNum 393 snap 120 seq_num 393) 2024-04-26T00:26:30.012 INFO:tasks.rados.rados.0.smithi045.stdout:1071: snap_create 2024-04-26T00:26:30.013 INFO:tasks.rados.rados.0.smithi045.stdout:1066: expect (ObjNum 360 snap 107 seq_num 360) 2024-04-26T00:26:30.165 INFO:tasks.rados.rados.0.smithi045.stdout:1068: expect (ObjNum 359 snap 107 seq_num 359) 2024-04-26T00:26:30.247 INFO:tasks.rados.rados.0.smithi045.stdout:1067: finishing copy_from racing read to smithi04526579-49 2024-04-26T00:26:30.247 INFO:tasks.rados.rados.0.smithi045.stdout:1070: expect (ObjNum 393 snap 120 seq_num 393) 2024-04-26T00:26:30.383 INFO:tasks.rados.rados.0.smithi045.stdout:1067: finishing copy_from to smithi04526579-49 2024-04-26T00:26:30.383 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 308 (ObjNum 316 snap 93 seq_num 316) dirty exists 2024-04-26T00:26:30.383 INFO:tasks.rados.rados.0.smithi045.stdout:1069: finishing write tid 1 to smithi04526579-2 2024-04-26T00:26:30.383 INFO:tasks.rados.rados.0.smithi045.stdout:1069: finishing write tid 2 to smithi04526579-2 2024-04-26T00:26:30.383 INFO:tasks.rados.rados.0.smithi045.stdout:1069: finishing write tid 3 to smithi04526579-2 2024-04-26T00:26:30.384 INFO:tasks.rados.rados.0.smithi045.stdout:1069: finishing write tid 4 to smithi04526579-2 2024-04-26T00:26:30.384 INFO:tasks.rados.rados.0.smithi045.stdout:1069: oid 2 updating version 0 to 546 2024-04-26T00:26:30.384 INFO:tasks.rados.rados.0.smithi045.stdout:1069: oid 2 updating version 546 to 547 2024-04-26T00:26:30.384 INFO:tasks.rados.rados.0.smithi045.stdout:1069: oid 2 updating version 547 to 548 2024-04-26T00:26:30.384 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 548 (ObjNum 404 snap 124 seq_num 404) dirty exists 2024-04-26T00:26:30.384 INFO:tasks.rados.rados.0.smithi045.stdout:1069: left oid 2 (ObjNum 404 snap 124 seq_num 404) 2024-04-26T00:26:30.983 INFO:tasks.rados.rados.0.smithi045.stdout:1066: done (5 left) 2024-04-26T00:26:30.984 INFO:tasks.rados.rados.0.smithi045.stdout:1067: done (4 left) 2024-04-26T00:26:30.984 INFO:tasks.rados.rados.0.smithi045.stdout:1068: done (3 left) 2024-04-26T00:26:30.984 INFO:tasks.rados.rados.0.smithi045.stdout:1069: done (2 left) 2024-04-26T00:26:30.984 INFO:tasks.rados.rados.0.smithi045.stdout:1070: done (1 left) 2024-04-26T00:26:30.984 INFO:tasks.rados.rados.0.smithi045.stdout:1071: done (0 left) 2024-04-26T00:26:30.984 INFO:tasks.rados.rados.0.smithi045.stdout:1072: rollback oid 3 current snap is 125 2024-04-26T00:26:30.984 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 3 to 122 2024-04-26T00:26:30.984 INFO:tasks.rados.rados.0.smithi045.stdout:1073: snap_remove snap 123 2024-04-26T00:26:31.887 INFO:tasks.rados.rados.0.smithi045.stdout:1073: done (1 left) 2024-04-26T00:26:31.887 INFO:tasks.rados.rados.0.smithi045.stdout:1074: snap_remove snap 99 2024-04-26T00:26:32.890 INFO:tasks.rados.rados.0.smithi045.stdout:1074: done (1 left) 2024-04-26T00:26:32.890 INFO:tasks.rados.rados.0.smithi045.stdout:1075: write oid 33 current snap is 125 2024-04-26T00:26:32.890 INFO:tasks.rados.rados.0.smithi045.stdout:1075: seq_num 405 ranges {406835=642554,1532972=506795,2772312=34523} 2024-04-26T00:26:32.911 INFO:tasks.rados.rados.0.smithi045.stdout:1075: writing smithi04526579-33 from 406835 to 1049389 tid 1 2024-04-26T00:26:32.928 INFO:tasks.rados.rados.0.smithi045.stdout:1075: writing smithi04526579-33 from 1532972 to 2039767 tid 2 2024-04-26T00:26:32.941 INFO:tasks.rados.rados.0.smithi045.stdout:1075: writing smithi04526579-33 from 2772312 to 2806835 tid 3 2024-04-26T00:26:32.941 INFO:tasks.rados.rados.0.smithi045.stdout:1076: write oid 4 current snap is 125 2024-04-26T00:26:32.941 INFO:tasks.rados.rados.0.smithi045.stdout:1072: finishing rollback tid 0 to smithi04526579-3 2024-04-26T00:26:32.941 INFO:tasks.rados.rados.0.smithi045.stdout:1072: finishing rollback tid 1 to smithi04526579-3 2024-04-26T00:26:32.941 INFO:tasks.rados.rados.0.smithi045.stdout:1072: finishing rollback tid 2 to smithi04526579-3 2024-04-26T00:26:32.941 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 580 (ObjNum 389 snap 117 seq_num 389) dirty exists 2024-04-26T00:26:32.941 INFO:tasks.rados.rados.0.smithi045.stdout:1075: finishing write tid 1 to smithi04526579-33 2024-04-26T00:26:32.941 INFO:tasks.rados.rados.0.smithi045.stdout:1075: finishing write tid 2 to smithi04526579-33 2024-04-26T00:26:32.941 INFO:tasks.rados.rados.0.smithi045.stdout:1076: seq_num 406 ranges {423642=717803,1875101=766806} 2024-04-26T00:26:32.957 INFO:tasks.rados.rados.0.smithi045.stdout:1076: writing smithi04526579-4 from 423642 to 1141445 tid 1 2024-04-26T00:26:32.977 INFO:tasks.rados.rados.0.smithi045.stdout:1076: writing smithi04526579-4 from 1875101 to 2641907 tid 2 2024-04-26T00:26:32.977 INFO:tasks.rados.rados.0.smithi045.stdout:1072: done (2 left) 2024-04-26T00:26:32.977 INFO:tasks.rados.rados.0.smithi045.stdout:1077: snap_remove snap 104 2024-04-26T00:26:33.891 INFO:tasks.rados.rados.0.smithi045.stdout:1077: done (2 left) 2024-04-26T00:26:33.891 INFO:tasks.rados.rados.0.smithi045.stdout:1078: snap_remove snap 115 2024-04-26T00:26:34.894 INFO:tasks.rados.rados.0.smithi045.stdout:1078: done (2 left) 2024-04-26T00:26:34.895 INFO:tasks.rados.rados.0.smithi045.stdout:1079: snap_remove snap 116 2024-04-26T00:26:34.968 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-04-26T00:26:34.969 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-04-26T00:26:34.969 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-26T00:26:34.969 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-26T00:26:34.972 INFO:tasks.ceph.mon.c:Started 2024-04-26T00:26:34.972 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:26:34.972 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:26:35.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.104+0000 7f232ebab640 10 monclient: get_monmap_and_config 2024-04-26T00:26:35.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.104+0000 7f232ebab640 10 monclient: build_initial_monmap 2024-04-26T00:26:35.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.104+0000 7f232ebab640 10 monclient: monmap: 2024-04-26T00:26:35.092 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:35.092 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:35.092 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:26:35.111399+0000 2024-04-26T00:26:35.092 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:26:35.111399+0000 2024-04-26T00:26:35.092 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:35.092 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:35.092 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:26:35.092 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:26:35.092 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:26:35.092 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:26:35.092 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:26:35.092 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:35.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f232ebab640 1 Processor -- start 2024-04-26T00:26:35.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f232ebab640 1 -- start start 2024-04-26T00:26:35.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f232ebab640 10 monclient: init 2024-04-26T00:26:35.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f232ebab640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:35.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f232ebab640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-04-26T00:26:35.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f232ebab640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2328136e10 0x7f232809c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:35.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f232ebab640 10 monclient(hunting): picked mon.noname-c con 0x7f2328136e10 addr v2:172.21.15.45:3300/0 2024-04-26T00:26:35.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f232ebab640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f23281371f0 0x7f2328137600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:35.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f232ebab640 10 monclient(hunting): picked mon.noname-a con 0x7f23281371f0 addr v2:172.21.15.37:3300/0 2024-04-26T00:26:35.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f232ebab640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2328135d90 0x7f2328136240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:35.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f232ebab640 10 monclient(hunting): picked mon.noname-b con 0x7f2328135d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:26:35.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f232ebab640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:35.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f232ebab640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f232809b8f0 con 0x7f23281371f0 2024-04-26T00:26:35.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f232ebab640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:35.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f232ebab640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f232809b680 con 0x7f2328136e10 2024-04-26T00:26:35.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f232ebab640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:35.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f232ebab640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f232809bbd0 con 0x7f2328135d90 2024-04-26T00:26:35.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f232ebab640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:35.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f232ebab640 10 monclient(hunting): authenticate will time out at 1376.416963s 2024-04-26T00:26:35.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f232d148640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2328135d90 0x7f2328136240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:35.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f2327fff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f23281371f0 0x7f2328137600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:35.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f232d148640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2328135d90 0x7f2328136240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:59680/0 (socket says 172.21.15.37:59680) 2024-04-26T00:26:35.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f2327fff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f23281371f0 0x7f2328137600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50918/0 (socket says 172.21.15.37:50918) 2024-04-26T00:26:35.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f232d148640 1 -- 172.21.15.37:0/2254073317 learned_addr learned my addr 172.21.15.37:0/2254073317 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:26:35.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f232d148640 10 monclient(hunting): get_auth_request con 0x7f2328135d90 auth_method 0 2024-04-26T00:26:35.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f232d148640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:35.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f232d148640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:35.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f232d148640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:35.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f2327fff640 10 monclient(hunting): get_auth_request con 0x7f23281371f0 auth_method 0 2024-04-26T00:26:35.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f2327fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:35.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f2327fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:35.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f2327fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:35.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f2327fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:35.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f2327fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:35.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f232d148640 1 -- 172.21.15.37:0/2254073317 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2328135d90 msgr2=0x7f2328136240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:26:35.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f232d148640 1 -- 172.21.15.37:0/2254073317 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2328135d90 msgr2=0x7f2328136240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:26:35.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f232d148640 1 --2- 172.21.15.37:0/2254073317 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2328135d90 0x7f2328136240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:26:35.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f2327fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:35.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f232d148640 1 --2- 172.21.15.37:0/2254073317 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2328135d90 0x7f2328136240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:26:35.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f232c947640 1 --2- 172.21.15.37:0/2254073317 >> v2:172.21.15.45:3300/0 conn(0x7f2328136e10 0x7f232809c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:35.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f232c947640 10 monclient(hunting): get_auth_request con 0x7f2328136e10 auth_method 0 2024-04-26T00:26:35.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f232c947640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:35.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f232c947640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:35.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f232c947640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:35.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f2327fff640 10 monclient(hunting): handle_auth_done global_id 54360 payload 274 2024-04-26T00:26:35.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f2327fff640 1 -- 172.21.15.37:0/2254073317 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2328135d90 msgr2=0x7f2328136240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:26:35.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f2327fff640 1 --2- 172.21.15.37:0/2254073317 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2328135d90 0x7f2328136240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:35.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f2327fff640 1 -- 172.21.15.37:0/2254073317 >> v2:172.21.15.45:3300/0 conn(0x7f2328136e10 msgr2=0x7f232809c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:26:35.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f2327fff640 1 --2- 172.21.15.37:0/2254073317 >> v2:172.21.15.45:3300/0 conn(0x7f2328136e10 0x7f232809c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:35.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f2327fff640 10 monclient: _finish_hunting 0 2024-04-26T00:26:35.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f2327fff640 1 monclient: found mon.noname-a 2024-04-26T00:26:35.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f2327fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:35.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f2327fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:26:35.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.108+0000 7f2327fff640 1 -- 172.21.15.37:0/2254073317 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2328137b40 con 0x7f23281371f0 2024-04-26T00:26:35.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.112+0000 7f2327fff640 1 -- 172.21.15.37:0/2254073317 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23281371f0 msgr2=0x7f2328137600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:35.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.112+0000 7f2327fff640 1 --2- 172.21.15.37:0/2254073317 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23281371f0 0x7f2328137600 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f23280a5b00 tx=0x7f231c01c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1db2accd04133a5f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:35.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.112+0000 7f23277fe640 1 -- 172.21.15.37:0/2254073317 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f231c0b0070 con 0x7f23281371f0 2024-04-26T00:26:35.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.112+0000 7f23277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:35.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.112+0000 7f23277fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:26:35.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.112+0000 7f23277fe640 10 monclient: dump: 2024-04-26T00:26:35.104 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:35.104 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:35.104 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:35.104 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:35.104 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:35.104 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:35.104 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:35.104 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:35.104 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:35.104 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:35.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.112+0000 7f23277fe640 10 monclient: _finish_auth 0 2024-04-26T00:26:35.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.112+0000 7f23277fe640 10 monclient: _check_auth_tickets 2024-04-26T00:26:35.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.112+0000 7f23277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:35.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.112+0000 7f23277fe640 1 -- 172.21.15.37:0/2254073317 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f231c0ab600 con 0x7f23281371f0 2024-04-26T00:26:35.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.112+0000 7f23277fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:35.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.112+0000 7f23277fe640 1 -- 172.21.15.37:0/2254073317 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f231c0ab9d0 con 0x7f23281371f0 2024-04-26T00:26:35.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.112+0000 7f23277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:35.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.112+0000 7f23277fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:26:35.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.112+0000 7f23277fe640 10 monclient: dump: 2024-04-26T00:26:35.105 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:35.105 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:35.105 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:35.105 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:35.105 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:35.105 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:35.106 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:35.106 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:35.106 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:35.106 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:35.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.112+0000 7f232ebab640 5 monclient: authenticate success, global_id 54360 2024-04-26T00:26:35.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.112+0000 7f232ebab640 10 monclient: get_monmap_and_config success 2024-04-26T00:26:35.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.112+0000 7f232ebab640 4 set_mon_vals no callback set 2024-04-26T00:26:35.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.112+0000 7f232ebab640 10 monclient: shutdown 2024-04-26T00:26:35.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.112+0000 7f232ebab640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:35.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.112+0000 7f232ebab640 1 -- 172.21.15.37:0/2254073317 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23281371f0 msgr2=0x7f2328137600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:35.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.112+0000 7f232ebab640 1 --2- 172.21.15.37:0/2254073317 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23281371f0 0x7f2328137600 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f23280a5b00 tx=0x7f231c01c320 comp rx=0 tx=0).stop 2024-04-26T00:26:35.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.112+0000 7f232ebab640 1 -- 172.21.15.37:0/2254073317 shutdown_connections 2024-04-26T00:26:35.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.112+0000 7f232ebab640 1 --2- 172.21.15.37:0/2254073317 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2328135d90 0x7f2328136240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:35.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.112+0000 7f232ebab640 1 --2- 172.21.15.37:0/2254073317 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23281371f0 0x7f2328137600 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:35.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.112+0000 7f232ebab640 1 --2- 172.21.15.37:0/2254073317 >> v2:172.21.15.45:3300/0 conn(0x7f2328136e10 0x7f232809c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:35.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.112+0000 7f232ebab640 1 -- 172.21.15.37:0/2254073317 >> 172.21.15.37:0/2254073317 conn(0x7f23280a64e0 msgr2=0x7f23280a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:35.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.112+0000 7f232ebab640 1 -- 172.21.15.37:0/2254073317 shutdown_connections 2024-04-26T00:26:35.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.112+0000 7f232ebab640 1 -- 172.21.15.37:0/2254073317 wait complete. 2024-04-26T00:26:35.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.112+0000 7f232ebab640 10 monclient: build_initial_monmap 2024-04-26T00:26:35.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.112+0000 7f232ebab640 10 monclient: monmap: 2024-04-26T00:26:35.107 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:35.107 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:26:35.107 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:26:35.107 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:26:35.107 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:35.107 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:26:35.107 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:26:35.107 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:26:35.108 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:26:35.108 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:35.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.112+0000 7f232ebab640 1 Processor -- start 2024-04-26T00:26:35.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.112+0000 7f232ebab640 1 -- start start 2024-04-26T00:26:35.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.112+0000 7f232ebab640 10 monclient: init 2024-04-26T00:26:35.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f232ebab640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:35.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f232ebab640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:26:35.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f232ebab640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2328135d90 0x7f23281cf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:35.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f232ebab640 10 monclient(hunting): picked mon.noname-b con 0x7f2328135d90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:26:35.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f232ebab640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2328136e10 0x7f2328134e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:35.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f232ebab640 10 monclient(hunting): picked mon.noname-a con 0x7f2328136e10 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:26:35.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f232ebab640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f23281371f0 0x7f232813ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:35.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f232ebab640 10 monclient(hunting): picked mon.noname-c con 0x7f23281371f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:26:35.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f232ebab640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:35.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f232ebab640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f2328140cf0 con 0x7f2328136e10 2024-04-26T00:26:35.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f232ebab640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:35.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f232ebab640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2328140b70 con 0x7f2328135d90 2024-04-26T00:26:35.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f232ebab640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:35.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f232ebab640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2328140e70 con 0x7f23281371f0 2024-04-26T00:26:35.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f232ebab640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:35.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f232ebab640 10 monclient(hunting): authenticate will time out at 1376.424419s 2024-04-26T00:26:35.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f2327fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2328136e10 0x7f2328134e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:35.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f232d148640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f23281371f0 0x7f232813ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:35.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f2327fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2328136e10 0x7f2328134e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50926/0 (socket says 172.21.15.37:50926) 2024-04-26T00:26:35.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f2327fff640 1 -- 172.21.15.37:0/3599604160 learned_addr learned my addr 172.21.15.37:0/3599604160 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:26:35.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f2327fff640 10 monclient(hunting): get_auth_request con 0x7f2328136e10 auth_method 0 2024-04-26T00:26:35.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f2327fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:35.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f2327fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:35.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f2327fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:35.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f232d148640 10 monclient(hunting): get_auth_request con 0x7f23281371f0 auth_method 0 2024-04-26T00:26:35.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f232d148640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:35.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f232d148640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:35.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f232d148640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:35.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f2327fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:35.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f2327fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:35.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f2327fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:35.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f232d148640 1 -- 172.21.15.37:0/3599604160 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f23281371f0 msgr2=0x7f232813ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:26:35.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f232d148640 1 -- 172.21.15.37:0/3599604160 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f23281371f0 msgr2=0x7f232813ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:26:35.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f232d148640 1 --2- 172.21.15.37:0/3599604160 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f23281371f0 0x7f232813ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:26:35.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f232c947640 1 --2- 172.21.15.37:0/3599604160 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2328135d90 0x7f23281cf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:35.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f232d148640 1 --2- 172.21.15.37:0/3599604160 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f23281371f0 0x7f232813ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:26:35.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f232c947640 10 monclient(hunting): get_auth_request con 0x7f2328135d90 auth_method 0 2024-04-26T00:26:35.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f232c947640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:35.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f232c947640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:35.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f232c947640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:35.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f2327fff640 10 monclient(hunting): handle_auth_done global_id 54363 payload 824 2024-04-26T00:26:35.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f2327fff640 1 -- 172.21.15.37:0/3599604160 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f23281371f0 msgr2=0x7f232813ac30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:26:35.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f2327fff640 1 --2- 172.21.15.37:0/3599604160 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f23281371f0 0x7f232813ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:35.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f2327fff640 1 -- 172.21.15.37:0/3599604160 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2328135d90 msgr2=0x7f23281cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:35.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f2327fff640 1 --2- 172.21.15.37:0/3599604160 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2328135d90 0x7f23281cf1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:35.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f2327fff640 10 monclient: _finish_hunting 0 2024-04-26T00:26:35.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f2327fff640 1 monclient: found mon.noname-a 2024-04-26T00:26:35.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f2327fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:35.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f2327fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:26:35.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f2327fff640 1 -- 172.21.15.37:0/3599604160 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2328120820 con 0x7f2328136e10 2024-04-26T00:26:35.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f2327fff640 1 -- 172.21.15.37:0/3599604160 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2328136e10 msgr2=0x7f2328134e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:35.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f2327fff640 1 --2- 172.21.15.37:0/3599604160 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2328136e10 0x7f2328134e80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f231c01cbc0 tx=0x7f231c0190c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:35.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f23257fa640 1 -- 172.21.15.37:0/3599604160 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f231c0ba070 con 0x7f2328136e10 2024-04-26T00:26:35.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f23257fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:35.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f23257fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:26:35.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f23257fa640 10 monclient: dump: 2024-04-26T00:26:35.112 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:35.112 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:35.112 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:35.112 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:35.112 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:35.112 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:35.112 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:35.112 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:35.112 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:35.112 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:35.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f23257fa640 10 monclient: _finish_auth 0 2024-04-26T00:26:35.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f23257fa640 10 monclient: _check_auth_tickets 2024-04-26T00:26:35.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f23257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:35.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f23257fa640 1 -- 172.21.15.37:0/3599604160 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f231c0b5890 con 0x7f2328136e10 2024-04-26T00:26:35.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f232ebab640 5 monclient: authenticate success, global_id 54363 2024-04-26T00:26:35.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f232ebab640 10 monclient: _renew_subs 2024-04-26T00:26:35.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f232ebab640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:26:35.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f232ebab640 1 -- 172.21.15.37:0/3599604160 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23280a3430 con 0x7f2328136e10 2024-04-26T00:26:35.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f23257fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:35.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f23257fa640 1 -- 172.21.15.37:0/3599604160 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f231c0b0040 con 0x7f2328136e10 2024-04-26T00:26:35.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f23257fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:35.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f23257fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:26:35.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f23257fa640 10 monclient: dump: 2024-04-26T00:26:35.113 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:35.113 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:35.114 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:35.114 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:35.114 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:35.114 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:35.114 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:35.114 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:35.114 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:35.114 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:35.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f232ebab640 10 monclient: _renew_subs 2024-04-26T00:26:35.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f232ebab640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:26:35.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f232ebab640 1 -- 172.21.15.37:0/3599604160 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f232811ff80 con 0x7f2328136e10 2024-04-26T00:26:35.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.116+0000 7f2326ffd640 4 set_mon_vals no callback set 2024-04-26T00:26:35.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.120+0000 7f232ebab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:35.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.120+0000 7f232ebab640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:35.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.120+0000 7f232ebab640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:26:35.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.120+0000 7f23257fa640 1 -- 172.21.15.37:0/3599604160 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f231c0b3070 con 0x7f2328136e10 2024-04-26T00:26:35.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.120+0000 7f232ebab640 1 -- 172.21.15.37:0/3599604160 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2328075910 con 0x7f2328136e10 2024-04-26T00:26:35.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.120+0000 7f23257fa640 1 --2- 172.21.15.37:0/3599604160 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2314035ed0 0x7f2314038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:35.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.120+0000 7f23257fa640 1 -- 172.21.15.37:0/3599604160 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(256..256 src has 177..256) ==== 4023+0+0 (secure 0 0 0) 0x7f231c0e2c90 con 0x7f2328136e10 2024-04-26T00:26:35.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.120+0000 7f232c947640 1 --2- 172.21.15.37:0/3599604160 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2314035ed0 0x7f2314038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:35.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.120+0000 7f232c947640 10 monclient: get_auth_request con 0x7f2314035ed0 auth_method 0 2024-04-26T00:26:35.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.120+0000 7f232c947640 1 --2- 172.21.15.37:0/3599604160 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2314035ed0 0x7f2314038390 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f231800b950 tx=0x7f2318008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:35.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.124+0000 7f23257fa640 1 -- 172.21.15.37:0/3599604160 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f231c0e9080 con 0x7f2328136e10 2024-04-26T00:26:35.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.124+0000 7f23257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:35.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.124+0000 7f23257fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:26:35.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.352+0000 7f232ebab640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:26:35.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.352+0000 7f232ebab640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:26:35.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.352+0000 7f232ebab640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:26:35.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.352+0000 7f232ebab640 1 -- 172.21.15.37:0/3599604160 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f23281d0020 con 0x7f2328136e10 2024-04-26T00:26:35.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.356+0000 7f23257fa640 1 -- 172.21.15.37:0/3599604160 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f231c016030 con 0x7f2328136e10 2024-04-26T00:26:35.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.356+0000 7f23257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:26:35.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.356+0000 7f23257fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:26:35.340 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":50,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:26:35.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.356+0000 7f232ebab640 1 -- 172.21.15.37:0/3599604160 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2314035ed0 msgr2=0x7f2314038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:35.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.356+0000 7f232ebab640 1 --2- 172.21.15.37:0/3599604160 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2314035ed0 0x7f2314038390 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f231800b950 tx=0x7f2318008040 comp rx=0 tx=0).stop 2024-04-26T00:26:35.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.356+0000 7f232ebab640 10 monclient: shutdown 2024-04-26T00:26:35.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.356+0000 7f232ebab640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:35.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.356+0000 7f232ebab640 1 -- 172.21.15.37:0/3599604160 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2328136e10 msgr2=0x7f2328134e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:35.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.356+0000 7f232ebab640 1 --2- 172.21.15.37:0/3599604160 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2328136e10 0x7f2328134e80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f231c01cbc0 tx=0x7f231c0190c0 comp rx=0 tx=0).stop 2024-04-26T00:26:35.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.356+0000 7f232ebab640 1 -- 172.21.15.37:0/3599604160 shutdown_connections 2024-04-26T00:26:35.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.356+0000 7f232ebab640 1 --2- 172.21.15.37:0/3599604160 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2314035ed0 0x7f2314038390 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:35.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.356+0000 7f232ebab640 1 --2- 172.21.15.37:0/3599604160 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f23281371f0 0x7f232813ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:35.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.356+0000 7f232ebab640 1 --2- 172.21.15.37:0/3599604160 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2328136e10 0x7f2328134e80 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:35.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.356+0000 7f232ebab640 1 --2- 172.21.15.37:0/3599604160 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2328135d90 0x7f23281cf1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:35.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.356+0000 7f232ebab640 1 -- 172.21.15.37:0/3599604160 >> 172.21.15.37:0/3599604160 conn(0x7f23280a64e0 msgr2=0x7f232809d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:35.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.356+0000 7f232ebab640 1 -- 172.21.15.37:0/3599604160 shutdown_connections 2024-04-26T00:26:35.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:35.356+0000 7f232ebab640 1 -- 172.21.15.37:0/3599604160 wait complete. 2024-04-26T00:26:35.897 INFO:tasks.rados.rados.0.smithi045.stdout:1079: done (2 left) 2024-04-26T00:26:35.897 INFO:tasks.rados.rados.0.smithi045.stdout:1080: delete oid 44 current snap is 125 2024-04-26T00:26:35.897 INFO:tasks.rados.rados.0.smithi045.stdout:1075: finishing write tid 3 to smithi04526579-33 2024-04-26T00:26:35.897 INFO:tasks.rados.rados.0.smithi045.stdout:1075: finishing write tid 4 to smithi04526579-33 2024-04-26T00:26:35.898 INFO:tasks.rados.rados.0.smithi045.stdout:1075: finishing write tid 5 to smithi04526579-33 2024-04-26T00:26:35.898 INFO:tasks.rados.rados.0.smithi045.stdout:1075: oid 33 updating version 0 to 763 2024-04-26T00:26:35.898 INFO:tasks.rados.rados.0.smithi045.stdout:1075: oid 33 updating version 763 to 768 2024-04-26T00:26:35.898 INFO:tasks.rados.rados.0.smithi045.stdout:1075: oid 33 version 768 is already newer than 766 2024-04-26T00:26:35.898 INFO:tasks.rados.rados.0.smithi045.stdout:1075: oid 33 version 768 is already newer than 767 2024-04-26T00:26:35.898 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 768 (ObjNum 405 snap 125 seq_num 405) dirty exists 2024-04-26T00:26:35.898 INFO:tasks.rados.rados.0.smithi045.stdout:1075: left oid 33 (ObjNum 405 snap 125 seq_num 405) 2024-04-26T00:26:35.898 INFO:tasks.rados.rados.0.smithi045.stdout:1076: finishing write tid 1 to smithi04526579-4 2024-04-26T00:26:35.898 INFO:tasks.rados.rados.0.smithi045.stdout:1076: finishing write tid 2 to smithi04526579-4 2024-04-26T00:26:35.898 INFO:tasks.rados.rados.0.smithi045.stdout:1076: finishing write tid 3 to smithi04526579-4 2024-04-26T00:26:35.898 INFO:tasks.rados.rados.0.smithi045.stdout:1076: finishing write tid 4 to smithi04526579-4 2024-04-26T00:26:35.898 INFO:tasks.rados.rados.0.smithi045.stdout:1076: oid 4 updating version 0 to 558 2024-04-26T00:26:35.898 INFO:tasks.rados.rados.0.smithi045.stdout:1076: oid 4 updating version 558 to 560 2024-04-26T00:26:35.899 INFO:tasks.rados.rados.0.smithi045.stdout:1076: oid 4 version 560 is already newer than 559 2024-04-26T00:26:35.899 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 560 (ObjNum 406 snap 125 seq_num 406) dirty exists 2024-04-26T00:26:35.899 INFO:tasks.rados.rados.0.smithi045.stdout:1076: left oid 4 (ObjNum 406 snap 125 seq_num 406) 2024-04-26T00:26:35.916 INFO:tasks.rados.rados.0.smithi045.stdout:1075: done (2 left) 2024-04-26T00:26:35.916 INFO:tasks.rados.rados.0.smithi045.stdout:1076: done (1 left) 2024-04-26T00:26:35.917 INFO:tasks.rados.rados.0.smithi045.stdout:1080: done (0 left) 2024-04-26T00:26:35.917 INFO:tasks.rados.rados.0.smithi045.stdout:1081: copy_from oid 35 from oid 31 current snap is 125 2024-04-26T00:26:35.917 INFO:tasks.rados.rados.0.smithi045.stdout:1082: write oid 32 current snap is 125 2024-04-26T00:26:35.917 INFO:tasks.rados.rados.0.smithi045.stdout:1082: seq_num 408 ranges {457256=468301,1592065=770475,2854581=2675} 2024-04-26T00:26:35.931 INFO:tasks.rados.rados.0.smithi045.stdout:1082: writing smithi04526579-32 from 457256 to 925557 tid 1 2024-04-26T00:26:35.952 INFO:tasks.rados.rados.0.smithi045.stdout:1082: writing smithi04526579-32 from 1592065 to 2362540 tid 2 2024-04-26T00:26:35.960 INFO:tasks.rados.rados.0.smithi045.stdout:1082: writing smithi04526579-32 from 2854581 to 2857256 tid 3 2024-04-26T00:26:35.960 INFO:tasks.rados.rados.0.smithi045.stdout:1083: delete oid 36 current snap is 125 2024-04-26T00:26:35.960 INFO:tasks.rados.rados.0.smithi045.stdout:1081: finishing copy_from racing read to smithi04526579-35 2024-04-26T00:26:35.960 INFO:tasks.rados.rados.0.smithi045.stdout:1081: finishing copy_from to smithi04526579-35 2024-04-26T00:26:35.961 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 489 (ObjNum 369 snap 112 seq_num 369) dirty exists 2024-04-26T00:26:35.961 INFO:tasks.rados.rados.0.smithi045.stdout:1082: finishing write tid 1 to smithi04526579-32 2024-04-26T00:26:35.964 INFO:tasks.rados.rados.0.smithi045.stdout:1082: finishing write tid 2 to smithi04526579-32 2024-04-26T00:26:35.965 INFO:tasks.rados.rados.0.smithi045.stdout:1082: finishing write tid 3 to smithi04526579-32 2024-04-26T00:26:35.966 INFO:tasks.rados.rados.0.smithi045.stdout:1082: finishing write tid 4 to smithi04526579-32 2024-04-26T00:26:35.966 INFO:tasks.rados.rados.0.smithi045.stdout:1081: done (2 left) 2024-04-26T00:26:35.967 INFO:tasks.rados.rados.0.smithi045.stdout:1083: done (1 left) 2024-04-26T00:26:35.967 INFO:tasks.rados.rados.0.smithi045.stdout:1084: write (excl) oid 26 current snap is 125 2024-04-26T00:26:35.967 INFO:tasks.rados.rados.0.smithi045.stdout:1084: seq_num 410 ranges {490870=618799,1592676=774144,2793074=97796} 2024-04-26T00:26:35.985 INFO:tasks.rados.rados.0.smithi045.stdout:1084: writing smithi04526579-26 from 490870 to 1109669 tid 1 2024-04-26T00:26:36.004 INFO:tasks.rados.rados.0.smithi045.stdout:1084: writing smithi04526579-26 from 1592676 to 2366820 tid 2 2024-04-26T00:26:36.011 INFO:tasks.rados.rados.0.smithi045.stdout:1084: writing smithi04526579-26 from 2793074 to 2890870 tid 3 2024-04-26T00:26:36.011 INFO:tasks.rados.rados.0.smithi045.stdout:1085: read oid 35 snap -1 2024-04-26T00:26:36.011 INFO:tasks.rados.rados.0.smithi045.stdout:1085: expect (ObjNum 369 snap 112 seq_num 369) 2024-04-26T00:26:36.011 INFO:tasks.rados.rados.0.smithi045.stdout:1086: snap_remove snap 32 2024-04-26T00:26:36.889 INFO:tasks.rados.rados.0.smithi045.stdout:1086: done (3 left) 2024-04-26T00:26:36.889 INFO:tasks.rados.rados.0.smithi045.stdout:1087: read oid 46 snap -1 2024-04-26T00:26:36.889 INFO:tasks.rados.rados.0.smithi045.stdout:1087: expect (ObjNum 329 snap 96 seq_num 329) 2024-04-26T00:26:36.889 INFO:tasks.rados.rados.0.smithi045.stdout:1088: rollback oid 37 current snap is 125 2024-04-26T00:26:36.890 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 37 to 118 2024-04-26T00:26:36.890 INFO:tasks.rados.rados.0.smithi045.stdout:1082: finishing write tid 5 to smithi04526579-32 2024-04-26T00:26:36.890 INFO:tasks.rados.rados.0.smithi045.stdout:1082: oid 32 updating version 0 to 545 2024-04-26T00:26:36.890 INFO:tasks.rados.rados.0.smithi045.stdout:1082: oid 32 updating version 545 to 548 2024-04-26T00:26:36.890 INFO:tasks.rados.rados.0.smithi045.stdout:1082: oid 32 version 548 is already newer than 547 2024-04-26T00:26:36.890 INFO:tasks.rados.rados.0.smithi045.stdout:1082: oid 32 version 548 is already newer than 546 2024-04-26T00:26:36.890 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 548 (ObjNum 408 snap 125 seq_num 408) dirty exists 2024-04-26T00:26:36.890 INFO:tasks.rados.rados.0.smithi045.stdout:1082: left oid 32 (ObjNum 408 snap 125 seq_num 408) 2024-04-26T00:26:36.890 INFO:tasks.rados.rados.0.smithi045.stdout:1084: finishing write tid 1 to smithi04526579-26 2024-04-26T00:26:36.890 INFO:tasks.rados.rados.0.smithi045.stdout:1084: finishing write tid 2 to smithi04526579-26 2024-04-26T00:26:36.890 INFO:tasks.rados.rados.0.smithi045.stdout:1084: finishing write tid 3 to smithi04526579-26 2024-04-26T00:26:36.890 INFO:tasks.rados.rados.0.smithi045.stdout:1084: finishing write tid 4 to smithi04526579-26 2024-04-26T00:26:36.891 INFO:tasks.rados.rados.0.smithi045.stdout:1084: finishing write tid 5 to smithi04526579-26 2024-04-26T00:26:36.891 INFO:tasks.rados.rados.0.smithi045.stdout:1084: oid 26 updating version 0 to 656 2024-04-26T00:26:36.891 INFO:tasks.rados.rados.0.smithi045.stdout:1084: oid 26 updating version 656 to 657 2024-04-26T00:26:36.891 INFO:tasks.rados.rados.0.smithi045.stdout:1084: oid 26 version 657 is already newer than 654 2024-04-26T00:26:36.891 INFO:tasks.rados.rados.0.smithi045.stdout:1084: oid 26 version 657 is already newer than 655 2024-04-26T00:26:36.891 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 657 (ObjNum 410 snap 125 seq_num 410) dirty exists 2024-04-26T00:26:36.891 INFO:tasks.rados.rados.0.smithi045.stdout:1084: left oid 26 (ObjNum 410 snap 125 seq_num 410) 2024-04-26T00:26:36.891 INFO:tasks.rados.rados.0.smithi045.stdout:1085: expect (ObjNum 369 snap 112 seq_num 369) 2024-04-26T00:26:37.044 INFO:tasks.rados.rados.0.smithi045.stdout:1087: expect (ObjNum 329 snap 96 seq_num 329) 2024-04-26T00:26:37.161 INFO:tasks.rados.rados.0.smithi045.stdout:1088: finishing rollback tid 0 to smithi04526579-37 2024-04-26T00:26:37.161 INFO:tasks.rados.rados.0.smithi045.stdout:1088: finishing rollback tid 1 to smithi04526579-37 2024-04-26T00:26:37.161 INFO:tasks.rados.rados.0.smithi045.stdout:1088: finishing rollback tid 2 to smithi04526579-37 2024-04-26T00:26:37.161 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 802 (ObjNum 215 snap 55 seq_num 215) dirty exists 2024-04-26T00:26:37.161 INFO:tasks.rados.rados.0.smithi045.stdout:1082: done (4 left) 2024-04-26T00:26:37.161 INFO:tasks.rados.rados.0.smithi045.stdout:1084: done (3 left) 2024-04-26T00:26:37.161 INFO:tasks.rados.rados.0.smithi045.stdout:1085: done (2 left) 2024-04-26T00:26:37.161 INFO:tasks.rados.rados.0.smithi045.stdout:1087: done (1 left) 2024-04-26T00:26:37.162 INFO:tasks.rados.rados.0.smithi045.stdout:1088: done (0 left) 2024-04-26T00:26:37.162 INFO:tasks.rados.rados.0.smithi045.stdout:1089: read oid 23 snap -1 2024-04-26T00:26:37.162 INFO:tasks.rados.rados.0.smithi045.stdout:1089: expect (ObjNum 316 snap 93 seq_num 316) 2024-04-26T00:26:37.162 INFO:tasks.rados.rados.0.smithi045.stdout:1090: snap_create 2024-04-26T00:26:37.167 INFO:tasks.rados.rados.0.smithi045.stdout:1089: expect (ObjNum 316 snap 93 seq_num 316) 2024-04-26T00:26:37.937 INFO:tasks.rados.rados.0.smithi045.stdout:1089: done (1 left) 2024-04-26T00:26:37.937 INFO:tasks.rados.rados.0.smithi045.stdout:1090: done (0 left) 2024-04-26T00:26:37.937 INFO:tasks.rados.rados.0.smithi045.stdout:1091: write (excl) oid 24 current snap is 126 2024-04-26T00:26:37.937 INFO:tasks.rados.rados.0.smithi045.stdout:1091: seq_num 411 ranges {507677=410401,1367511=517802,2420497=487180} 2024-04-26T00:26:37.951 INFO:tasks.rados.rados.0.smithi045.stdout:1091: writing smithi04526579-24 from 507677 to 918078 tid 1 2024-04-26T00:26:37.967 INFO:tasks.rados.rados.0.smithi045.stdout:1091: writing smithi04526579-24 from 1367511 to 1885313 tid 2 2024-04-26T00:26:37.984 INFO:tasks.rados.rados.0.smithi045.stdout:1091: writing smithi04526579-24 from 2420497 to 2907677 tid 3 2024-04-26T00:26:37.984 INFO:tasks.rados.rados.0.smithi045.stdout:1092: delete oid 22 current snap is 126 2024-04-26T00:26:37.984 INFO:tasks.rados.rados.0.smithi045.stdout:1091: finishing write tid 1 to smithi04526579-24 2024-04-26T00:26:37.984 INFO:tasks.rados.rados.0.smithi045.stdout:1091: finishing write tid 2 to smithi04526579-24 2024-04-26T00:26:37.991 INFO:tasks.rados.rados.0.smithi045.stdout:1091: finishing write tid 3 to smithi04526579-24 2024-04-26T00:26:37.992 INFO:tasks.rados.rados.0.smithi045.stdout:1091: finishing write tid 4 to smithi04526579-24 2024-04-26T00:26:37.992 INFO:tasks.rados.rados.0.smithi045.stdout:1091: finishing write tid 5 to smithi04526579-24 2024-04-26T00:26:37.992 INFO:tasks.rados.rados.0.smithi045.stdout:1091: oid 24 updating version 0 to 817 2024-04-26T00:26:37.993 INFO:tasks.rados.rados.0.smithi045.stdout:1091: oid 24 version 817 is already newer than 816 2024-04-26T00:26:37.993 INFO:tasks.rados.rados.0.smithi045.stdout:1091: oid 24 updating version 817 to 818 2024-04-26T00:26:37.993 INFO:tasks.rados.rados.0.smithi045.stdout:1091: oid 24 version 818 is already newer than 815 2024-04-26T00:26:37.993 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 818 (ObjNum 411 snap 126 seq_num 411) dirty exists 2024-04-26T00:26:37.993 INFO:tasks.rados.rados.0.smithi045.stdout:1091: left oid 24 (ObjNum 411 snap 126 seq_num 411) 2024-04-26T00:26:37.997 INFO:tasks.rados.rados.0.smithi045.stdout:1091: done (1 left) 2024-04-26T00:26:37.997 INFO:tasks.rados.rados.0.smithi045.stdout:1092: done (0 left) 2024-04-26T00:26:37.997 INFO:tasks.rados.rados.0.smithi045.stdout:1093: rollback oid 22 current snap is 126 2024-04-26T00:26:37.997 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 22 to 117 2024-04-26T00:26:37.997 INFO:tasks.rados.rados.0.smithi045.stdout:1094: rollback oid 36 current snap is 126 2024-04-26T00:26:37.998 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 36 to 108 2024-04-26T00:26:37.998 INFO:tasks.rados.rados.0.smithi045.stdout:1095: copy_from oid 12 from oid 11 current snap is 126 2024-04-26T00:26:37.998 INFO:tasks.rados.rados.0.smithi045.stdout:1096: snap_create 2024-04-26T00:26:38.006 INFO:tasks.rados.rados.0.smithi045.stdout:1093: finishing rollback tid 1 to smithi04526579-22 2024-04-26T00:26:38.006 INFO:tasks.rados.rados.0.smithi045.stdout:1093: finishing rollback tid 2 to smithi04526579-22 2024-04-26T00:26:38.006 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 331 (ObjNum 330 snap 96 seq_num 330) dirty exists 2024-04-26T00:26:38.007 INFO:tasks.rados.rados.0.smithi045.stdout:1094: finishing rollback tid 1 to smithi04526579-36 2024-04-26T00:26:38.008 INFO:tasks.rados.rados.0.smithi045.stdout:1094: finishing rollback tid 2 to smithi04526579-36 2024-04-26T00:26:38.008 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 333 (ObjNum 304 snap 87 seq_num 304) dirty exists 2024-04-26T00:26:38.015 INFO:tasks.rados.rados.0.smithi045.stdout:1095: finishing copy_from racing read to smithi04526579-12 2024-04-26T00:26:38.016 INFO:tasks.rados.rados.0.smithi045.stdout:1095: finishing copy_from to smithi04526579-12 2024-04-26T00:26:38.016 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 383 (ObjNum 401 snap 122 seq_num 401) dirty exists 2024-04-26T00:26:38.360 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:26:38.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.508+0000 7f992f8f1640 10 monclient: get_monmap_and_config 2024-04-26T00:26:38.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.508+0000 7f992f8f1640 10 monclient: build_initial_monmap 2024-04-26T00:26:38.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.508+0000 7f992f8f1640 10 monclient: monmap: 2024-04-26T00:26:38.493 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:38.493 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:38.493 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:26:38.512947+0000 2024-04-26T00:26:38.493 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:26:38.512947+0000 2024-04-26T00:26:38.494 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:38.494 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:38.494 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:26:38.494 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:26:38.494 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:26:38.494 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:26:38.494 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:26:38.494 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:38.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.508+0000 7f992f8f1640 1 Processor -- start 2024-04-26T00:26:38.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.508+0000 7f992f8f1640 1 -- start start 2024-04-26T00:26:38.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.508+0000 7f992f8f1640 10 monclient: init 2024-04-26T00:26:38.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.508+0000 7f992f8f1640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:38.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.508+0000 7f992f8f1640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-04-26T00:26:38.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.508+0000 7f992f8f1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9928133b20 addr v1:172.21.15.37:6789/0 2024-04-26T00:26:38.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.508+0000 7f992f8f1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9928133f00 addr v1:172.21.15.45:6789/0 2024-04-26T00:26:38.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.508+0000 7f992f8f1640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f992813a060 0x7f992813c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:38.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.508+0000 7f992f8f1640 10 monclient(hunting): picked mon.noname-a con 0x7f992813a060 addr v2:172.21.15.37:3300/0 2024-04-26T00:26:38.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.508+0000 7f992f8f1640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f992809fe20 con 0x7f9928133b20 2024-04-26T00:26:38.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.508+0000 7f992f8f1640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9928072d00 con 0x7f9928133f00 2024-04-26T00:26:38.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.508+0000 7f992f8f1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:38.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.508+0000 7f992f8f1640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f992809e620 con 0x7f992813a060 2024-04-26T00:26:38.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.508+0000 7f992f8f1640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:38.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.508+0000 7f992f8f1640 10 monclient(hunting): authenticate will time out at 1379.819418s 2024-04-26T00:26:38.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.508+0000 7f992de8e640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f992813a060 0x7f992813c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:38.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.508+0000 7f992de8e640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f992813a060 0x7f992813c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50972/0 (socket says 172.21.15.37:50972) 2024-04-26T00:26:38.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.508+0000 7f992de8e640 1 -- 172.21.15.37:0/1429387326 learned_addr learned my addr 172.21.15.37:0/1429387326 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:26:38.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.508+0000 7f992de8e640 10 monclient(hunting): get_auth_request con 0x7f992813a060 auth_method 0 2024-04-26T00:26:38.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.508+0000 7f992de8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:38.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.508+0000 7f992de8e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:38.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.508+0000 7f992de8e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:38.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.508+0000 7f992de8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:38.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.508+0000 7f992de8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:38.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.508+0000 7f992de8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:38.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.508+0000 7f992de8e640 10 monclient(hunting): handle_auth_done global_id 54366 payload 274 2024-04-26T00:26:38.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.508+0000 7f992de8e640 1 -- 172.21.15.37:0/1429387326 >> v1:172.21.15.45:6789/0 conn(0x7f9928133f00 legacy=0x7f9928134350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:26:38.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.508+0000 7f992de8e640 1 -- 172.21.15.37:0/1429387326 >> v1:172.21.15.37:6789/0 conn(0x7f9928133b20 legacy=0x7f992809e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:26:38.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.508+0000 7f992de8e640 10 monclient: _finish_hunting 0 2024-04-26T00:26:38.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.508+0000 7f992de8e640 1 monclient: found mon.noname-a 2024-04-26T00:26:38.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.508+0000 7f992de8e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:38.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.508+0000 7f992de8e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:26:38.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.508+0000 7f992de8e640 1 -- 172.21.15.37:0/1429387326 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f992809ba60 con 0x7f992813a060 2024-04-26T00:26:38.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992de8e640 1 -- 172.21.15.37:0/1429387326 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f992813a060 msgr2=0x7f992813c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:38.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992de8e640 1 --2- 172.21.15.37:0/1429387326 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f992813a060 0x7f992813c450 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9920009e50 tx=0x7f992001c690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=700d37fd277a6150 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:38.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f991ffff640 1 -- 172.21.15.37:0/1429387326 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99200ac020 con 0x7f992813a060 2024-04-26T00:26:38.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f991ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:38.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f991ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:26:38.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f991ffff640 10 monclient: dump: 2024-04-26T00:26:38.501 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:38.501 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:38.502 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:38.502 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:38.502 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:38.502 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:38.502 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:38.502 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:38.502 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:38.502 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:38.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f991ffff640 10 monclient: _finish_auth 0 2024-04-26T00:26:38.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f991ffff640 10 monclient: _check_auth_tickets 2024-04-26T00:26:38.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f991ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:38.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f991ffff640 1 -- 172.21.15.37:0/1429387326 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99200a6470 con 0x7f992813a060 2024-04-26T00:26:38.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f991ffff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:38.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f991ffff640 1 -- 172.21.15.37:0/1429387326 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99200a6820 con 0x7f992813a060 2024-04-26T00:26:38.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992f8f1640 5 monclient: authenticate success, global_id 54366 2024-04-26T00:26:38.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992f8f1640 10 monclient: get_monmap_and_config success 2024-04-26T00:26:38.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992f8f1640 4 set_mon_vals no callback set 2024-04-26T00:26:38.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f991ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:38.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f991ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:26:38.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f991ffff640 10 monclient: dump: 2024-04-26T00:26:38.503 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:38.503 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:38.503 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:38.503 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:38.503 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:38.503 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:38.503 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:38.503 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:38.504 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:38.504 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:38.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992f8f1640 10 monclient: shutdown 2024-04-26T00:26:38.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992f8f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:38.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992f8f1640 1 -- 172.21.15.37:0/1429387326 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f992813a060 msgr2=0x7f992813c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:38.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992f8f1640 1 --2- 172.21.15.37:0/1429387326 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f992813a060 0x7f992813c450 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9920009e50 tx=0x7f992001c690 comp rx=0 tx=0).stop 2024-04-26T00:26:38.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992f8f1640 1 -- 172.21.15.37:0/1429387326 shutdown_connections 2024-04-26T00:26:38.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992f8f1640 1 --2- 172.21.15.37:0/1429387326 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f992813a060 0x7f992813c450 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:38.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992f8f1640 1 -- 172.21.15.37:0/1429387326 >> 172.21.15.37:0/1429387326 conn(0x7f99280a1ad0 msgr2=0x7f99280a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:38.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992f8f1640 1 -- 172.21.15.37:0/1429387326 shutdown_connections 2024-04-26T00:26:38.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992f8f1640 1 -- 172.21.15.37:0/1429387326 wait complete. 2024-04-26T00:26:38.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992f8f1640 10 monclient: build_initial_monmap 2024-04-26T00:26:38.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992f8f1640 10 monclient: monmap: 2024-04-26T00:26:38.509 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:38.509 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:26:38.509 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:26:38.509 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:26:38.509 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:38.509 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:26:38.509 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:26:38.509 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:26:38.509 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:26:38.509 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:38.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992f8f1640 1 Processor -- start 2024-04-26T00:26:38.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992f8f1640 1 -- start start 2024-04-26T00:26:38.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992f8f1640 10 monclient: init 2024-04-26T00:26:38.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992f8f1640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:38.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992f8f1640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:26:38.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992f8f1640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9928133b20 0x7f99280a0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:38.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992f8f1640 10 monclient(hunting): picked mon.noname-c con 0x7f9928133b20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:26:38.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992f8f1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9928133f00 0x7f9928134ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:38.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992f8f1640 10 monclient(hunting): picked mon.noname-b con 0x7f9928133f00 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:26:38.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992f8f1640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f992813a060 0x7f9928134ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:38.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992f8f1640 10 monclient(hunting): picked mon.noname-a con 0x7f992813a060 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:26:38.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992f8f1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:38.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992f8f1640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f992813f9b0 con 0x7f992813a060 2024-04-26T00:26:38.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992f8f1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:38.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992f8f1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f992813f830 con 0x7f9928133f00 2024-04-26T00:26:38.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992f8f1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:38.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992f8f1640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f992813fb30 con 0x7f9928133b20 2024-04-26T00:26:38.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992f8f1640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:38.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992f8f1640 10 monclient(hunting): authenticate will time out at 1379.827474s 2024-04-26T00:26:38.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992d68d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9928133b20 0x7f99280a0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:38.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992d68d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9928133b20 0x7f99280a0260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:59706/0 (socket says 172.21.15.37:59706) 2024-04-26T00:26:38.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992d68d640 1 -- 172.21.15.37:0/3585539988 learned_addr learned my addr 172.21.15.37:0/3585539988 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:26:38.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992de8e640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f992813a060 0x7f9928134ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:38.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992d68d640 10 monclient(hunting): get_auth_request con 0x7f9928133b20 auth_method 0 2024-04-26T00:26:38.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992d68d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:38.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992d68d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:38.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992d68d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:38.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992de8e640 10 monclient(hunting): get_auth_request con 0x7f992813a060 auth_method 0 2024-04-26T00:26:38.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992de8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:38.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992de8e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:38.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992de8e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:38.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992d68d640 1 -- 172.21.15.37:0/3585539988 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9928133b20 msgr2=0x7f99280a0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:26:38.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992d68d640 1 -- 172.21.15.37:0/3585539988 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9928133b20 msgr2=0x7f99280a0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:26:38.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992d68d640 1 --2- 172.21.15.37:0/3585539988 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9928133b20 0x7f99280a0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:26:38.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992de8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:38.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992de8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:38.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992d68d640 1 --2- 172.21.15.37:0/3585539988 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9928133b20 0x7f99280a0260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:26:38.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992de8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:38.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992ce8c640 1 --2- 172.21.15.37:0/3585539988 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9928133f00 0x7f9928134ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:38.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992ce8c640 10 monclient(hunting): get_auth_request con 0x7f9928133f00 auth_method 0 2024-04-26T00:26:38.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992ce8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:38.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992ce8c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:38.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.516+0000 7f992ce8c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:38.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.520+0000 7f992de8e640 10 monclient(hunting): handle_auth_done global_id 54369 payload 824 2024-04-26T00:26:38.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.520+0000 7f992de8e640 1 -- 172.21.15.37:0/3585539988 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9928133b20 msgr2=0x7f99280a0260 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:26:38.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.520+0000 7f992de8e640 1 --2- 172.21.15.37:0/3585539988 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9928133b20 0x7f99280a0260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:38.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.520+0000 7f992de8e640 1 -- 172.21.15.37:0/3585539988 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9928133f00 msgr2=0x7f9928134ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:38.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.520+0000 7f992de8e640 1 --2- 172.21.15.37:0/3585539988 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9928133f00 0x7f9928134ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:38.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.520+0000 7f992de8e640 10 monclient: _finish_hunting 0 2024-04-26T00:26:38.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.520+0000 7f992de8e640 1 monclient: found mon.noname-a 2024-04-26T00:26:38.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.520+0000 7f992de8e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:38.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.520+0000 7f992de8e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:26:38.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.520+0000 7f992de8e640 1 -- 172.21.15.37:0/3585539988 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9928137760 con 0x7f992813a060 2024-04-26T00:26:38.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.520+0000 7f992de8e640 1 -- 172.21.15.37:0/3585539988 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f992813a060 msgr2=0x7f9928134ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:38.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.520+0000 7f992de8e640 1 --2- 172.21.15.37:0/3585539988 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f992813a060 0x7f9928134ff0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f99200aa000 tx=0x7f9920019430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:38.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.520+0000 7f991dffb640 1 -- 172.21.15.37:0/3585539988 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99200a7860 con 0x7f992813a060 2024-04-26T00:26:38.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.520+0000 7f991dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:38.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.520+0000 7f991dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:26:38.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.520+0000 7f991dffb640 10 monclient: dump: 2024-04-26T00:26:38.514 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:38.514 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:38.514 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:38.514 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:38.514 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:38.514 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:38.514 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:38.514 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:38.514 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:38.514 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:38.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.520+0000 7f991dffb640 10 monclient: _finish_auth 0 2024-04-26T00:26:38.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.520+0000 7f991dffb640 10 monclient: _check_auth_tickets 2024-04-26T00:26:38.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.520+0000 7f991dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:38.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.520+0000 7f991dffb640 1 -- 172.21.15.37:0/3585539988 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99200a7a20 con 0x7f992813a060 2024-04-26T00:26:38.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.520+0000 7f992f8f1640 5 monclient: authenticate success, global_id 54369 2024-04-26T00:26:38.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.520+0000 7f992f8f1640 10 monclient: _renew_subs 2024-04-26T00:26:38.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.520+0000 7f992f8f1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:26:38.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.520+0000 7f992f8f1640 1 -- 172.21.15.37:0/3585539988 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9928137480 con 0x7f992813a060 2024-04-26T00:26:38.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.520+0000 7f991dffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:38.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.520+0000 7f991dffb640 1 -- 172.21.15.37:0/3585539988 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99200a7e10 con 0x7f992813a060 2024-04-26T00:26:38.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.520+0000 7f991dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:38.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.520+0000 7f991dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:26:38.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.520+0000 7f991dffb640 10 monclient: dump: 2024-04-26T00:26:38.516 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:38.516 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:38.516 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:38.516 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:38.516 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:38.516 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:38.516 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:38.516 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:38.516 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:38.516 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:38.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.520+0000 7f991f7fe640 4 set_mon_vals no callback set 2024-04-26T00:26:38.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.520+0000 7f992f8f1640 10 monclient: _renew_subs 2024-04-26T00:26:38.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.520+0000 7f992f8f1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:26:38.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.520+0000 7f992f8f1640 1 -- 172.21.15.37:0/3585539988 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9928135fd0 con 0x7f992813a060 2024-04-26T00:26:38.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.520+0000 7f991dffb640 1 -- 172.21.15.37:0/3585539988 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f99200af020 con 0x7f992813a060 2024-04-26T00:26:38.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.520+0000 7f992f8f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:38.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.520+0000 7f991dffb640 1 --2- 172.21.15.37:0/3585539988 >> v2:172.21.15.45:6812/2379918623 conn(0x7f990c035eb0 0x7f990c038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:38.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.520+0000 7f992f8f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:38.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.520+0000 7f992f8f1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:26:38.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.520+0000 7f992f8f1640 1 -- 172.21.15.37:0/3585539988 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9928066f40 con 0x7f992813a060 2024-04-26T00:26:38.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.520+0000 7f991dffb640 1 -- 172.21.15.37:0/3585539988 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(259..259 src has 177..259) ==== 3963+0+0 (secure 0 0 0) 0x7f99200e31a0 con 0x7f992813a060 2024-04-26T00:26:38.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.524+0000 7f992d68d640 1 --2- 172.21.15.37:0/3585539988 >> v2:172.21.15.45:6812/2379918623 conn(0x7f990c035eb0 0x7f990c038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:38.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.524+0000 7f992d68d640 10 monclient: get_auth_request con 0x7f990c035eb0 auth_method 0 2024-04-26T00:26:38.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.524+0000 7f992d68d640 1 --2- 172.21.15.37:0/3585539988 >> v2:172.21.15.45:6812/2379918623 conn(0x7f990c035eb0 0x7f990c038370 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9924007d20 tx=0x7f9924009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:38.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.528+0000 7f991dffb640 1 -- 172.21.15.37:0/3585539988 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f992809fe20 con 0x7f992813a060 2024-04-26T00:26:38.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.528+0000 7f991dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:38.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.528+0000 7f991dffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:26:38.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.748+0000 7f992f8f1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:26:38.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.748+0000 7f992f8f1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:26:38.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.748+0000 7f992f8f1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:26:38.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.748+0000 7f992f8f1640 1 -- 172.21.15.37:0/3585539988 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9928139d20 con 0x7f992813a060 2024-04-26T00:26:38.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.748+0000 7f991dffb640 1 -- 172.21.15.37:0/3585539988 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f99200bae50 con 0x7f992813a060 2024-04-26T00:26:38.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.748+0000 7f991dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:26:38.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.748+0000 7f991dffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:26:38.736 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":52,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:26:38.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.752+0000 7f992f8f1640 1 -- 172.21.15.37:0/3585539988 >> v2:172.21.15.45:6812/2379918623 conn(0x7f990c035eb0 msgr2=0x7f990c038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:38.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.752+0000 7f992f8f1640 1 --2- 172.21.15.37:0/3585539988 >> v2:172.21.15.45:6812/2379918623 conn(0x7f990c035eb0 0x7f990c038370 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9924007d20 tx=0x7f9924009040 comp rx=0 tx=0).stop 2024-04-26T00:26:38.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.752+0000 7f992f8f1640 10 monclient: shutdown 2024-04-26T00:26:38.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.752+0000 7f992f8f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:38.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.752+0000 7f992f8f1640 1 -- 172.21.15.37:0/3585539988 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f992813a060 msgr2=0x7f9928134ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:38.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.752+0000 7f992f8f1640 1 --2- 172.21.15.37:0/3585539988 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f992813a060 0x7f9928134ff0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f99200aa000 tx=0x7f9920019430 comp rx=0 tx=0).stop 2024-04-26T00:26:38.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.752+0000 7f992f8f1640 1 -- 172.21.15.37:0/3585539988 shutdown_connections 2024-04-26T00:26:38.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.752+0000 7f992f8f1640 1 --2- 172.21.15.37:0/3585539988 >> v2:172.21.15.45:6812/2379918623 conn(0x7f990c035eb0 0x7f990c038370 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:38.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.752+0000 7f992f8f1640 1 --2- 172.21.15.37:0/3585539988 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f992813a060 0x7f9928134ff0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:38.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.752+0000 7f992f8f1640 1 --2- 172.21.15.37:0/3585539988 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9928133f00 0x7f9928134ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:38.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.752+0000 7f992f8f1640 1 --2- 172.21.15.37:0/3585539988 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9928133b20 0x7f99280a0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:38.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.752+0000 7f992f8f1640 1 -- 172.21.15.37:0/3585539988 >> 172.21.15.37:0/3585539988 conn(0x7f99280a1ad0 msgr2=0x7f99280a3670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:38.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.752+0000 7f992f8f1640 1 -- 172.21.15.37:0/3585539988 shutdown_connections 2024-04-26T00:26:38.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.752+0000 7f992f8f1640 1 -- 172.21.15.37:0/3585539988 wait complete. 2024-04-26T00:26:38.755 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:26:38.755 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:26:38.755 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:26:38.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10b54b6640 10 monclient: get_monmap_and_config 2024-04-26T00:26:38.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10b54b6640 10 monclient: build_initial_monmap 2024-04-26T00:26:38.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10b54b6640 10 monclient: monmap: 2024-04-26T00:26:38.881 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:38.882 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:38.882 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:26:38.901022+0000 2024-04-26T00:26:38.882 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:26:38.901022+0000 2024-04-26T00:26:38.882 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:38.882 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:38.882 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:26:38.882 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:26:38.882 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:26:38.882 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:26:38.882 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:26:38.882 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:38.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10b54b6640 1 Processor -- start 2024-04-26T00:26:38.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10b54b6640 1 -- start start 2024-04-26T00:26:38.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10b54b6640 10 monclient: init 2024-04-26T00:26:38.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10b54b6640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:38.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10b54b6640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-04-26T00:26:38.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10b54b6640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f10b00a3d90 0x7f10b009bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:38.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10b54b6640 10 monclient(hunting): picked mon.noname-a con 0x7f10b00a3d90 addr v2:172.21.15.37:3300/0 2024-04-26T00:26:38.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10b54b6640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f10b00a4170 0x7f10b00a45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:38.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10b54b6640 10 monclient(hunting): picked mon.noname-b con 0x7f10b00a4170 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:26:38.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10b54b6640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f10b0133b20 0x7f10b0133f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:38.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10b54b6640 10 monclient(hunting): picked mon.noname-c con 0x7f10b0133b20 addr v2:172.21.15.45:3300/0 2024-04-26T00:26:38.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10b54b6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:38.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10b54b6640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f10b00a5150 con 0x7f10b00a3d90 2024-04-26T00:26:38.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10b54b6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:38.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10b54b6640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f10b009b280 con 0x7f10b0133b20 2024-04-26T00:26:38.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10b54b6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:38.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10b54b6640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f10b009b560 con 0x7f10b00a4170 2024-04-26T00:26:38.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10b54b6640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:38.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10b54b6640 10 monclient(hunting): authenticate will time out at 1380.207277s 2024-04-26T00:26:38.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10aeffd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f10b00a3d90 0x7f10b009bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:38.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10a7fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f10b00a4170 0x7f10b00a45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:38.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10aeffd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f10b00a3d90 0x7f10b009bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50988/0 (socket says 172.21.15.37:50988) 2024-04-26T00:26:38.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10aeffd640 1 -- 172.21.15.37:0/1988379252 learned_addr learned my addr 172.21.15.37:0/1988379252 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:26:38.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10aeffd640 10 monclient(hunting): get_auth_request con 0x7f10b00a3d90 auth_method 0 2024-04-26T00:26:38.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:38.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10aeffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:38.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:38.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10a7fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f10b00a4170 0x7f10b00a45b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:59716/0 (socket says 172.21.15.37:59716) 2024-04-26T00:26:38.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10a7fff640 10 monclient(hunting): get_auth_request con 0x7f10b00a4170 auth_method 0 2024-04-26T00:26:38.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:38.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10a7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:38.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:38.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:38.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:38.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10a7fff640 1 -- 172.21.15.37:0/1988379252 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f10b00a4170 msgr2=0x7f10b00a45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:26:38.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10a7fff640 1 -- 172.21.15.37:0/1988379252 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f10b00a4170 msgr2=0x7f10b00a45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:26:38.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10a7fff640 1 --2- 172.21.15.37:0/1988379252 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f10b00a4170 0x7f10b00a45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:26:38.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:38.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10a7fff640 1 --2- 172.21.15.37:0/1988379252 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f10b00a4170 0x7f10b00a45b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:26:38.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10af7fe640 1 --2- 172.21.15.37:0/1988379252 >> v2:172.21.15.45:3300/0 conn(0x7f10b0133b20 0x7f10b0133f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:38.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10aeffd640 1 -- 172.21.15.37:0/1988379252 >> v2:172.21.15.37:3300/0 conn(0x7f10b00a3d90 msgr2=0x7f10b009bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:26:38.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10aeffd640 1 -- 172.21.15.37:0/1988379252 >> v2:172.21.15.37:3300/0 conn(0x7f10b00a3d90 msgr2=0x7f10b009bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:26:38.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10aeffd640 1 --2- 172.21.15.37:0/1988379252 >> v2:172.21.15.37:3300/0 conn(0x7f10b00a3d90 0x7f10b009bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:26:38.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10af7fe640 10 monclient(hunting): get_auth_request con 0x7f10b0133b20 auth_method 0 2024-04-26T00:26:38.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:38.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10af7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:38.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:38.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.896+0000 7f10aeffd640 1 --2- 172.21.15.37:0/1988379252 >> v2:172.21.15.37:3300/0 conn(0x7f10b00a3d90 0x7f10b009bf40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:26:38.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.900+0000 7f10af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:38.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.900+0000 7f10af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:38.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.900+0000 7f10af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:38.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.900+0000 7f10af7fe640 10 monclient(hunting): handle_auth_done global_id 24601 payload 274 2024-04-26T00:26:38.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.900+0000 7f10af7fe640 1 -- 172.21.15.37:0/1988379252 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f10b00a4170 msgr2=0x7f10b00a45b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:26:38.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.900+0000 7f10af7fe640 1 --2- 172.21.15.37:0/1988379252 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f10b00a4170 0x7f10b00a45b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:38.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.900+0000 7f10af7fe640 1 -- 172.21.15.37:0/1988379252 >> v2:172.21.15.37:3300/0 conn(0x7f10b00a3d90 msgr2=0x7f10b009bf40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:26:38.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.900+0000 7f10af7fe640 1 --2- 172.21.15.37:0/1988379252 >> v2:172.21.15.37:3300/0 conn(0x7f10b00a3d90 0x7f10b009bf40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:38.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.900+0000 7f10af7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:26:38.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.900+0000 7f10af7fe640 1 monclient: found mon.noname-c 2024-04-26T00:26:38.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.900+0000 7f10af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:38.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.900+0000 7f10af7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:26:38.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.900+0000 7f10af7fe640 1 -- 172.21.15.37:0/1988379252 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10b00a4af0 con 0x7f10b0133b20 2024-04-26T00:26:38.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10af7fe640 1 -- 172.21.15.37:0/1988379252 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10b0133b20 msgr2=0x7f10b0133f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:38.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10af7fe640 1 --2- 172.21.15.37:0/1988379252 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10b0133b20 0x7f10b0133f90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f10b0076100 tx=0x7f10a801c7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50b70e2501285f16 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:38.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10ae7fc640 1 -- 172.21.15.37:0/1988379252 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a80ac070 con 0x7f10b0133b20 2024-04-26T00:26:38.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:38.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10ae7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:26:38.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10ae7fc640 10 monclient: dump: 2024-04-26T00:26:38.891 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:38.891 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:38.891 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:38.891 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:38.891 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:38.891 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:38.891 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:38.891 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:38.891 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:38.892 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:38.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10ae7fc640 10 monclient: _finish_auth 0 2024-04-26T00:26:38.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10ae7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:26:38.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:38.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10b54b6640 5 monclient: authenticate success, global_id 24601 2024-04-26T00:26:38.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10ae7fc640 1 -- 172.21.15.37:0/1988379252 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10a80a6680 con 0x7f10b0133b20 2024-04-26T00:26:38.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10b54b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:26:38.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10ae7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:38.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10b54b6640 10 monclient: get_monmap_and_config success 2024-04-26T00:26:38.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10b54b6640 4 set_mon_vals no callback set 2024-04-26T00:26:38.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10ae7fc640 1 -- 172.21.15.37:0/1988379252 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a80a6a50 con 0x7f10b0133b20 2024-04-26T00:26:38.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:38.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10ae7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:26:38.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10ae7fc640 10 monclient: dump: 2024-04-26T00:26:38.892 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:38.893 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:38.893 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:38.893 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:38.893 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:38.893 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:38.893 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:38.893 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:38.893 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:38.893 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:38.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10b54b6640 10 monclient: shutdown 2024-04-26T00:26:38.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10b54b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:38.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10b54b6640 1 -- 172.21.15.37:0/1988379252 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10b0133b20 msgr2=0x7f10b0133f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:38.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10b54b6640 1 --2- 172.21.15.37:0/1988379252 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10b0133b20 0x7f10b0133f90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f10b0076100 tx=0x7f10a801c7f0 comp rx=0 tx=0).stop 2024-04-26T00:26:38.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10b54b6640 1 -- 172.21.15.37:0/1988379252 shutdown_connections 2024-04-26T00:26:38.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10b54b6640 1 --2- 172.21.15.37:0/1988379252 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10b0133b20 0x7f10b0133f90 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:38.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10b54b6640 1 --2- 172.21.15.37:0/1988379252 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f10b00a4170 0x7f10b00a45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:38.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10b54b6640 1 --2- 172.21.15.37:0/1988379252 >> v2:172.21.15.37:3300/0 conn(0x7f10b00a3d90 0x7f10b009bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:38.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10b54b6640 1 -- 172.21.15.37:0/1988379252 >> 172.21.15.37:0/1988379252 conn(0x7f10b00aa0a0 msgr2=0x7f10b00a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:38.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10b54b6640 1 -- 172.21.15.37:0/1988379252 shutdown_connections 2024-04-26T00:26:38.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10b54b6640 1 -- 172.21.15.37:0/1988379252 wait complete. 2024-04-26T00:26:38.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10b54b6640 10 monclient: build_initial_monmap 2024-04-26T00:26:38.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10b54b6640 10 monclient: monmap: 2024-04-26T00:26:38.894 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:38.894 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:26:38.894 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:26:38.894 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:26:38.894 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:38.894 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:26:38.894 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:26:38.894 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:26:38.895 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:26:38.895 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:38.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10b54b6640 1 Processor -- start 2024-04-26T00:26:38.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10b54b6640 1 -- start start 2024-04-26T00:26:38.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10b54b6640 10 monclient: init 2024-04-26T00:26:38.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10b54b6640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:38.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10b54b6640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:26:38.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10b54b6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10b00a3d90 0x7f10b01cf180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:38.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10b54b6640 10 monclient(hunting): picked mon.noname-b con 0x7f10b00a3d90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:26:38.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10b54b6640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10b00a4170 0x7f10b013d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:38.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10b54b6640 10 monclient(hunting): picked mon.noname-a con 0x7f10b00a4170 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:26:38.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10b54b6640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f10b0133b20 0x7f10b013d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:38.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10b54b6640 10 monclient(hunting): picked mon.noname-c con 0x7f10b0133b20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:26:38.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10b54b6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:38.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10b54b6640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f10b009b8d0 con 0x7f10b00a4170 2024-04-26T00:26:38.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10b54b6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:38.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10b54b6640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f10b009c690 con 0x7f10b00a3d90 2024-04-26T00:26:38.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10b54b6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:38.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10b54b6640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f10b0140c10 con 0x7f10b0133b20 2024-04-26T00:26:38.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10b54b6640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:38.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10b54b6640 10 monclient(hunting): authenticate will time out at 1380.215804s 2024-04-26T00:26:38.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10af7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f10b0133b20 0x7f10b013d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:38.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10a7fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10b00a4170 0x7f10b013d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:38.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10a7fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10b00a4170 0x7f10b013d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51002/0 (socket says 172.21.15.37:51002) 2024-04-26T00:26:38.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.904+0000 7f10af7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f10b0133b20 0x7f10b013d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:59724/0 (socket says 172.21.15.37:59724) 2024-04-26T00:26:38.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10a7fff640 1 -- 172.21.15.37:0/41445009 learned_addr learned my addr 172.21.15.37:0/41445009 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:26:38.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10a7fff640 10 monclient(hunting): get_auth_request con 0x7f10b00a4170 auth_method 0 2024-04-26T00:26:38.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:38.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10a7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:38.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:38.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10af7fe640 10 monclient(hunting): get_auth_request con 0x7f10b0133b20 auth_method 0 2024-04-26T00:26:38.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:38.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10af7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:38.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:38.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:38.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:38.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10af7fe640 1 -- 172.21.15.37:0/41445009 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f10b0133b20 msgr2=0x7f10b013d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:26:38.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:38.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10af7fe640 1 -- 172.21.15.37:0/41445009 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f10b0133b20 msgr2=0x7f10b013d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:26:38.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10af7fe640 1 --2- 172.21.15.37:0/41445009 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f10b0133b20 0x7f10b013d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:26:38.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10af7fe640 1 --2- 172.21.15.37:0/41445009 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f10b0133b20 0x7f10b013d670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:26:38.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10aeffd640 1 --2- 172.21.15.37:0/41445009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10b00a3d90 0x7f10b01cf180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:38.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10aeffd640 10 monclient(hunting): get_auth_request con 0x7f10b00a3d90 auth_method 0 2024-04-26T00:26:38.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:38.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10aeffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:38.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:38.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10a7fff640 10 monclient(hunting): handle_auth_done global_id 54375 payload 824 2024-04-26T00:26:38.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10a7fff640 1 -- 172.21.15.37:0/41445009 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f10b0133b20 msgr2=0x7f10b013d670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:26:38.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10a7fff640 1 --2- 172.21.15.37:0/41445009 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f10b0133b20 0x7f10b013d670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:38.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10a7fff640 1 -- 172.21.15.37:0/41445009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10b00a3d90 msgr2=0x7f10b01cf180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:38.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10a7fff640 1 --2- 172.21.15.37:0/41445009 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10b00a3d90 0x7f10b01cf180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:38.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10a7fff640 10 monclient: _finish_hunting 0 2024-04-26T00:26:38.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10a7fff640 1 monclient: found mon.noname-a 2024-04-26T00:26:38.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:38.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10a7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:26:38.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10a7fff640 1 -- 172.21.15.37:0/41445009 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10b01391c0 con 0x7f10b00a4170 2024-04-26T00:26:38.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10a7fff640 1 -- 172.21.15.37:0/41445009 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10b00a4170 msgr2=0x7f10b013d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:38.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10a7fff640 1 --2- 172.21.15.37:0/41445009 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10b00a4170 0x7f10b013d130 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f109800e7e0 tx=0x7f109800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:38.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10a77fe640 1 -- 172.21.15.37:0/41445009 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1098017020 con 0x7f10b00a4170 2024-04-26T00:26:38.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:38.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10a77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:26:38.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10a77fe640 10 monclient: dump: 2024-04-26T00:26:38.905 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:38.905 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:38.905 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:38.905 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:38.905 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:38.905 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:38.905 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:38.905 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:38.905 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:38.905 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:38.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10a77fe640 10 monclient: _finish_auth 0 2024-04-26T00:26:38.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10a77fe640 10 monclient: _check_auth_tickets 2024-04-26T00:26:38.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:38.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10b54b6640 5 monclient: authenticate success, global_id 54375 2024-04-26T00:26:38.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10b54b6640 10 monclient: _renew_subs 2024-04-26T00:26:38.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10a77fe640 1 -- 172.21.15.37:0/41445009 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f109800c8f0 con 0x7f10b00a4170 2024-04-26T00:26:38.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10b54b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:26:38.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10b54b6640 1 -- 172.21.15.37:0/41445009 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10b009ada0 con 0x7f10b00a4170 2024-04-26T00:26:38.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10a77fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:38.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10a77fe640 1 -- 172.21.15.37:0/41445009 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f109800f040 con 0x7f10b00a4170 2024-04-26T00:26:38.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:38.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10a77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:26:38.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10a77fe640 10 monclient: dump: 2024-04-26T00:26:38.906 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:38.906 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:38.907 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:38.907 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:38.907 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:38.907 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:38.907 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:38.907 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:38.907 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:38.907 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:38.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10b54b6640 10 monclient: _renew_subs 2024-04-26T00:26:38.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10ad7fa640 4 set_mon_vals no callback set 2024-04-26T00:26:38.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10b54b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:26:38.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10b54b6640 1 -- 172.21.15.37:0/41445009 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10b013a400 con 0x7f10b00a4170 2024-04-26T00:26:38.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10b54b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:38.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10b54b6640 1 --2- 172.21.15.37:0/41445009 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10b013a5d0 0x7f10b0136bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:38.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10b54b6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:38.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10b54b6640 1 -- 172.21.15.37:0/41445009 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f10b004f510 con 0x7f10b013a5d0 2024-04-26T00:26:38.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10aeffd640 1 --2- 172.21.15.37:0/41445009 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10b013a5d0 0x7f10b0136bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:38.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10aeffd640 10 monclient: get_auth_request con 0x7f10b013a5d0 auth_method 0 2024-04-26T00:26:38.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:38.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10aeffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:38.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:38.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10a77fe640 1 -- 172.21.15.37:0/41445009 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f109801a020 con 0x7f10b00a4170 2024-04-26T00:26:38.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:38.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:38.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:38.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.908+0000 7f10a77fe640 1 --2- 172.21.15.37:0/41445009 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1090035eb0 0x7f1090038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:38.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.912+0000 7f10a77fe640 1 -- 172.21.15.37:0/41445009 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(259..259 src has 177..259) ==== 3963+0+0 (secure 0 0 0) 0x7f1098048890 con 0x7f10b00a4170 2024-04-26T00:26:38.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.912+0000 7f10aeffd640 10 monclient(hunting): handle_auth_done global_id 54378 payload 824 2024-04-26T00:26:38.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.912+0000 7f10aeffd640 1 -- 172.21.15.37:0/41445009 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f10b0138f40 con 0x7f10b013a5d0 2024-04-26T00:26:38.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.912+0000 7f10af7fe640 1 --2- 172.21.15.37:0/41445009 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1090035eb0 0x7f1090038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:38.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.912+0000 7f10af7fe640 10 monclient: get_auth_request con 0x7f1090035eb0 auth_method 0 2024-04-26T00:26:38.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.912+0000 7f10aeffd640 1 -- 172.21.15.37:0/41445009 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10b013a5d0 msgr2=0x7f10b0136bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:38.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.912+0000 7f10aeffd640 1 --2- 172.21.15.37:0/41445009 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10b013a5d0 0x7f10b0136bf0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f10a0011560 tx=0x7f10a0011a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:38.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.912+0000 7f10a77fe640 1 -- 172.21.15.37:0/41445009 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a0015a80 con 0x7f10b013a5d0 2024-04-26T00:26:38.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.912+0000 7f10a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:38.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.912+0000 7f10a77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:26:38.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.912+0000 7f10a77fe640 10 monclient: dump: 2024-04-26T00:26:38.909 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:38.909 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:38.909 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:38.909 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:38.909 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:38.909 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:38.909 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:38.909 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:38.909 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:38.909 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:38.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.912+0000 7f10a77fe640 1 -- 172.21.15.37:0/41445009 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f10a0015a80 con 0x7f10b013a5d0 2024-04-26T00:26:38.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.912+0000 7f10a77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:38.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.912+0000 7f10a77fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:26:38.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.912+0000 7f10a77fe640 1 -- 172.21.15.37:0/41445009 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10b013a5d0 msgr2=0x7f10b0136bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:38.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.912+0000 7f10a77fe640 1 --2- 172.21.15.37:0/41445009 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10b013a5d0 0x7f10b0136bf0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f10a0011560 tx=0x7f10a0011a20 comp rx=0 tx=0).stop 2024-04-26T00:26:38.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.912+0000 7f10a77fe640 1 -- 172.21.15.37:0/41445009 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10b013a5d0 msgr2=0x7f10b0136bf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:26:38.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.912+0000 7f10a77fe640 1 --2- 172.21.15.37:0/41445009 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10b013a5d0 0x7f10b0136bf0 secure :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f10a0011560 tx=0x7f10a0011a20 comp rx=0 tx=0).stop 2024-04-26T00:26:38.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.912+0000 7f10af7fe640 1 --2- 172.21.15.37:0/41445009 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1090035eb0 0x7f1090038370 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f10a80ab000 tx=0x7f10a80a7000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:38.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.920+0000 7f10b54b6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:26:38.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.920+0000 7f10b54b6640 1 --2- 172.21.15.37:0/41445009 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10b0137130 0x7f10b0137540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:38.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.920+0000 7f10b54b6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:38.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.920+0000 7f10b54b6640 1 -- 172.21.15.37:0/41445009 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f10b0137a80 con 0x7f10b0137130 2024-04-26T00:26:38.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.920+0000 7f10aeffd640 1 --2- 172.21.15.37:0/41445009 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10b0137130 0x7f10b0137540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:38.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.920+0000 7f10aeffd640 10 monclient: get_auth_request con 0x7f10b0137130 auth_method 0 2024-04-26T00:26:38.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.920+0000 7f10aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:38.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.920+0000 7f10aeffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:38.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.920+0000 7f10aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:38.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.920+0000 7f10aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:38.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.920+0000 7f10aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:38.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.920+0000 7f10aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:38.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.920+0000 7f10aeffd640 10 monclient(hunting): handle_auth_done global_id 54381 payload 824 2024-04-26T00:26:38.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.920+0000 7f10aeffd640 1 -- 172.21.15.37:0/41445009 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f10b0075dc0 con 0x7f10b0137130 2024-04-26T00:26:38.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.920+0000 7f10aeffd640 1 -- 172.21.15.37:0/41445009 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10b0137130 msgr2=0x7f10b0137540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:38.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.920+0000 7f10aeffd640 1 --2- 172.21.15.37:0/41445009 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10b0137130 0x7f10b0137540 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f10a0011560 tx=0x7f10a00046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:38.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.924+0000 7f10a77fe640 1 -- 172.21.15.37:0/41445009 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a0006450 con 0x7f10b0137130 2024-04-26T00:26:38.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.924+0000 7f10a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:38.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.924+0000 7f10a77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:26:38.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.924+0000 7f10a77fe640 10 monclient: dump: 2024-04-26T00:26:38.913 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:38.913 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:38.913 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:38.913 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:38.913 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:38.914 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:38.914 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:38.914 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:38.914 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:38.914 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:38.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.924+0000 7f10a77fe640 1 -- 172.21.15.37:0/41445009 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f10a0006450 con 0x7f10b0137130 2024-04-26T00:26:38.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.924+0000 7f10a77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:26:38.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.924+0000 7f10a77fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:26:38.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.924+0000 7f10a77fe640 1 -- 172.21.15.37:0/41445009 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10b0137130 msgr2=0x7f10b0137540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:38.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.924+0000 7f10a77fe640 1 --2- 172.21.15.37:0/41445009 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10b0137130 0x7f10b0137540 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f10a0011560 tx=0x7f10a00046d0 comp rx=0 tx=0).stop 2024-04-26T00:26:38.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.924+0000 7f10a77fe640 1 -- 172.21.15.37:0/41445009 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10b0137130 msgr2=0x7f10b0137540 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:26:38.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.924+0000 7f10a77fe640 1 --2- 172.21.15.37:0/41445009 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10b0137130 0x7f10b0137540 secure :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f10a0011560 tx=0x7f10a00046d0 comp rx=0 tx=0).stop 2024-04-26T00:26:38.915 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:26:38.915 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:26:38.915 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:26:38.915 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:26:38.915 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 52, 2024-04-26T00:26:38.915 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 162250, 2024-04-26T00:26:38.915 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:26:38.915 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:26:38.915 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:26:38.915 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:26:38.916 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:38.916 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 2, 2024-04-26T00:26:38.916 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:26:38.916 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:26:38.916 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:26:38.916 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:26:38.916 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:26:38.916 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:26:38.916 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:26:38.916 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:26:38.916 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:26:38.916 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:26:38.916 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:26:38.916 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:26:38.916 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:26:38.917 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:26:38.917 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:38.917 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:26:38.917 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:26:38.917 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:26:38.917 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:26:38.917 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:26:38.917 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:26:38.917 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:26:38.917 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:26:38.917 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:26:38.918 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:26:38.918 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:26:38.918 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:26:38.918 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:26:38.918 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:38.918 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:38.918 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:26:38.918 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:26:38.918 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:26:38.918 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:26:38.918 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:26:38.919 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:26:38.919 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:26:38.919 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:26:38.919 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:26:38.919 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:26:38.919 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:26:38.919 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:26:38.919 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:26:38.919 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:26:38.919 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:26:38.919 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:26:38.919 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:26:38.919 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:26:38.919 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:26:38.920 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:26:38.920 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:26:38.920 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:26:38.920 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:26:38.920 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:26:38.920 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:26:38.920 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:26:38.920 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:26:38.920 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:26:38.920 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:38.920 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:26:38.920 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:38.921 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:26:38.921 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:38.921 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:26:38.921 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:26:38.921 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:26:38.921 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:26:38.921 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:38.921 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:26:38.921 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:26:38.921 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:38.921 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:38.921 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:38.921 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:26:38.921 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:26:38.922 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:38.922 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:38.922 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:38.922 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:38.922 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:26:38.922 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:26:38.922 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:26:38.922 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:26:38.922 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:26:38.922 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:38.925 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:38.925 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:26:38.925 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:26:38.926 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:26:38.926 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:26:38.926 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:38.926 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:26:38.926 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:26:38.926 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:38.926 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:38.926 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:38.926 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:26:38.926 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:26:38.926 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:38.926 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:38.926 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:38.926 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:38.927 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:26:38.927 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:26:38.927 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:26:38.927 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:26:38.927 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:26:38.927 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:38.927 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:38.927 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:26:38.927 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:26:38.927 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:26:38.927 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:26:38.927 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:38.927 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:26:38.927 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:26:38.927 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:38.928 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:38.928 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:38.928 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:26:38.928 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:26:38.928 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:38.928 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:38.928 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:38.928 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:38.928 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:26:38.928 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:26:38.928 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:26:38.928 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:26:38.928 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:26:38.928 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:38.928 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:38.929 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:38.929 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:26:38.929 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:26:38.929 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:38.929 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:26:38.929 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:26:38.929 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:26:38.929 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:38.929 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:38.929 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:26:38.929 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:38.929 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:26:38.929 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:26:38.929 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:26:38.930 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:38.930 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:38.930 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:26:38.930 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:38.930 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:26:38.930 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:26:38.930 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:26:38.930 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:38.930 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:38.930 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:26:38.930 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:38.930 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:26:38.930 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:26:38.930 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:26:38.931 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:38.931 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:38.931 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:38.931 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:26:38.931 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:26:38.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.924+0000 7f10b54b6640 1 -- 172.21.15.37:0/41445009 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1090035eb0 msgr2=0x7f1090038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:38.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.924+0000 7f10b54b6640 1 --2- 172.21.15.37:0/41445009 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1090035eb0 0x7f1090038370 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f10a80ab000 tx=0x7f10a80a7000 comp rx=0 tx=0).stop 2024-04-26T00:26:38.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.924+0000 7f10b54b6640 10 monclient: shutdown 2024-04-26T00:26:38.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.924+0000 7f10b54b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:38.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.924+0000 7f10b54b6640 1 -- 172.21.15.37:0/41445009 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10b00a4170 msgr2=0x7f10b013d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:38.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.924+0000 7f10b54b6640 1 --2- 172.21.15.37:0/41445009 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10b00a4170 0x7f10b013d130 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f109800e7e0 tx=0x7f109800eca0 comp rx=0 tx=0).stop 2024-04-26T00:26:38.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.924+0000 7f10af7fe640 1 -- 172.21.15.37:0/41445009 reap_dead start 2024-04-26T00:26:38.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.924+0000 7f10af7fe640 1 -- 172.21.15.37:0/41445009 reap_dead start 2024-04-26T00:26:38.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.924+0000 7f10b54b6640 1 -- 172.21.15.37:0/41445009 shutdown_connections 2024-04-26T00:26:38.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.924+0000 7f10b54b6640 1 -- 172.21.15.37:0/41445009 >> 172.21.15.37:0/41445009 conn(0x7f10b00aa0a0 msgr2=0x7f10b00a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:38.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.924+0000 7f10b54b6640 1 -- 172.21.15.37:0/41445009 shutdown_connections 2024-04-26T00:26:38.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:38.924+0000 7f10b54b6640 1 -- 172.21.15.37:0/41445009 wait complete. 2024-04-26T00:26:38.934 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:26:38.934 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:26:38.934 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:26:38.939 INFO:tasks.rados.rados.0.smithi045.stdout:1093: done (3 left) 2024-04-26T00:26:38.939 INFO:tasks.rados.rados.0.smithi045.stdout:1094: done (2 left) 2024-04-26T00:26:38.939 INFO:tasks.rados.rados.0.smithi045.stdout:1095: done (1 left) 2024-04-26T00:26:38.939 INFO:tasks.rados.rados.0.smithi045.stdout:1096: done (0 left) 2024-04-26T00:26:38.939 INFO:tasks.rados.rados.0.smithi045.stdout:1097: write oid 47 current snap is 127 2024-04-26T00:26:38.939 INFO:tasks.rados.rados.0.smithi045.stdout:1097: seq_num 413 ranges {541291=560899,1884475=521471} 2024-04-26T00:26:38.955 INFO:tasks.rados.rados.0.smithi045.stdout:1097: writing smithi04526579-47 from 541291 to 1102190 tid 1 2024-04-26T00:26:38.973 INFO:tasks.rados.rados.0.smithi045.stdout:1097: writing smithi04526579-47 from 1884475 to 2405946 tid 2 2024-04-26T00:26:38.973 INFO:tasks.rados.rados.0.smithi045.stdout:1098: snap_create 2024-04-26T00:26:38.973 INFO:tasks.rados.rados.0.smithi045.stdout:1097: finishing write tid 1 to smithi04526579-47 2024-04-26T00:26:38.979 INFO:tasks.rados.rados.0.smithi045.stdout:1097: finishing write tid 2 to smithi04526579-47 2024-04-26T00:26:38.980 INFO:tasks.rados.rados.0.smithi045.stdout:1097: finishing write tid 3 to smithi04526579-47 2024-04-26T00:26:38.981 INFO:tasks.rados.rados.0.smithi045.stdout:1097: finishing write tid 4 to smithi04526579-47 2024-04-26T00:26:38.981 INFO:tasks.rados.rados.0.smithi045.stdout:1097: oid 47 updating version 0 to 622 2024-04-26T00:26:38.981 INFO:tasks.rados.rados.0.smithi045.stdout:1097: oid 47 updating version 622 to 623 2024-04-26T00:26:38.981 INFO:tasks.rados.rados.0.smithi045.stdout:1097: oid 47 updating version 623 to 624 2024-04-26T00:26:38.981 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 624 (ObjNum 413 snap 127 seq_num 413) dirty exists 2024-04-26T00:26:38.981 INFO:tasks.rados.rados.0.smithi045.stdout:1097: left oid 47 (ObjNum 413 snap 127 seq_num 413) 2024-04-26T00:26:39.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad4402640 10 monclient: get_monmap_and_config 2024-04-26T00:26:39.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad4402640 10 monclient: build_initial_monmap 2024-04-26T00:26:39.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad4402640 10 monclient: monmap: 2024-04-26T00:26:39.053 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:39.053 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:39.053 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:26:39.072486+0000 2024-04-26T00:26:39.053 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:26:39.072486+0000 2024-04-26T00:26:39.053 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:39.053 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:39.053 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:26:39.053 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:26:39.053 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:26:39.053 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:26:39.054 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:26:39.054 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:39.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad4402640 1 Processor -- start 2024-04-26T00:26:39.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad4402640 1 -- start start 2024-04-26T00:26:39.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad4402640 10 monclient: init 2024-04-26T00:26:39.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad4402640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:39.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad4402640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-04-26T00:26:39.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad4402640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc0726b0 0x7f4acc072a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:39.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad4402640 10 monclient(hunting): picked mon.noname-b con 0x7f4acc0726b0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:26:39.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad4402640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f4acc0a0110 0x7f4acc136dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:39.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad4402640 10 monclient(hunting): picked mon.noname-a con 0x7f4acc0a0110 addr v2:172.21.15.37:3300/0 2024-04-26T00:26:39.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad4402640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4acc137310 addr v1:172.21.15.37:6789/0 2024-04-26T00:26:39.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad4402640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4acc0a1470 con 0x7f4acc137310 2024-04-26T00:26:39.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad4402640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:39.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad4402640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f4acc0a1bd0 con 0x7f4acc0a0110 2024-04-26T00:26:39.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad4402640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:39.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad4402640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4acc0a0eb0 con 0x7f4acc0726b0 2024-04-26T00:26:39.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad4402640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:39.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad4402640 10 monclient(hunting): authenticate will time out at 1380.378171s 2024-04-26T00:26:39.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad199d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f4acc0a0110 0x7f4acc136dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:39.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad199d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f4acc0a0110 0x7f4acc136dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51026/0 (socket says 172.21.15.37:51026) 2024-04-26T00:26:39.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad199d640 1 -- 172.21.15.37:0/699741244 learned_addr learned my addr 172.21.15.37:0/699741244 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:26:39.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad199d640 10 monclient(hunting): get_auth_request con 0x7f4acc0a0110 auth_method 0 2024-04-26T00:26:39.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad199d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:39.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad199d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:39.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad199d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:39.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad219e640 1 --2- 172.21.15.37:0/699741244 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc0726b0 0x7f4acc072a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:39.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad219e640 10 monclient(hunting): get_auth_request con 0x7f4acc0726b0 auth_method 0 2024-04-26T00:26:39.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad219e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:39.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad219e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:39.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad219e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:39.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad199d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:39.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad199d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:39.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad199d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:39.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad219e640 1 -- 172.21.15.37:0/699741244 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc0726b0 msgr2=0x7f4acc072a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:26:39.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad219e640 1 -- 172.21.15.37:0/699741244 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc0726b0 msgr2=0x7f4acc072a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:26:39.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad219e640 1 --2- 172.21.15.37:0/699741244 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc0726b0 0x7f4acc072a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:26:39.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad219e640 1 --2- 172.21.15.37:0/699741244 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc0726b0 0x7f4acc072a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:26:39.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad199d640 10 monclient(hunting): handle_auth_done global_id 54384 payload 274 2024-04-26T00:26:39.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad199d640 1 -- 172.21.15.37:0/699741244 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc0726b0 msgr2=0x7f4acc072a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:26:39.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad199d640 1 --2- 172.21.15.37:0/699741244 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc0726b0 0x7f4acc072a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:39.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad199d640 1 -- 172.21.15.37:0/699741244 >> v1:172.21.15.37:6789/0 conn(0x7f4acc137310 legacy=0x7f4acc13e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:26:39.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad199d640 10 monclient: _finish_hunting 0 2024-04-26T00:26:39.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad199d640 1 monclient: found mon.noname-a 2024-04-26T00:26:39.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad199d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:39.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad199d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:26:39.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.068+0000 7f4ad199d640 1 -- 172.21.15.37:0/699741244 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4acc0a0bd0 con 0x7f4acc0a0110 2024-04-26T00:26:39.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.072+0000 7f4ad199d640 1 -- 172.21.15.37:0/699741244 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4acc0a0110 msgr2=0x7f4acc136dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:39.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.072+0000 7f4ad199d640 1 --2- 172.21.15.37:0/699741244 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4acc0a0110 0x7f4acc136dd0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4ac00098a0 tx=0x7f4ac001c1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=955251c049a387db server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:39.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.072+0000 7f4ad119c640 1 -- 172.21.15.37:0/699741244 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac00b0070 con 0x7f4acc0a0110 2024-04-26T00:26:39.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.072+0000 7f4ad119c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:39.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.072+0000 7f4ad119c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:26:39.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.072+0000 7f4ad119c640 10 monclient: dump: 2024-04-26T00:26:39.062 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:39.062 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:39.062 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:39.062 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:39.062 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:39.062 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:39.062 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:39.062 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:39.062 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:39.063 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:39.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad119c640 10 monclient: _finish_auth 0 2024-04-26T00:26:39.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad119c640 10 monclient: _check_auth_tickets 2024-04-26T00:26:39.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad119c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:39.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad4402640 5 monclient: authenticate success, global_id 54384 2024-04-26T00:26:39.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad4402640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:26:39.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad119c640 1 -- 172.21.15.37:0/699741244 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ac00ab600 con 0x7f4acc0a0110 2024-04-26T00:26:39.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad119c640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:39.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad119c640 1 -- 172.21.15.37:0/699741244 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac00ab9d0 con 0x7f4acc0a0110 2024-04-26T00:26:39.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad119c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:39.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad119c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:26:39.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad119c640 10 monclient: dump: 2024-04-26T00:26:39.064 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:39.064 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:39.064 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:39.064 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:39.064 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:39.064 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:39.064 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:39.064 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:39.064 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:39.064 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:39.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad4402640 10 monclient: get_monmap_and_config success 2024-04-26T00:26:39.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad4402640 4 set_mon_vals no callback set 2024-04-26T00:26:39.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad4402640 10 monclient: shutdown 2024-04-26T00:26:39.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad4402640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:39.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad4402640 1 -- 172.21.15.37:0/699741244 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4acc0a0110 msgr2=0x7f4acc136dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:39.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad4402640 1 --2- 172.21.15.37:0/699741244 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4acc0a0110 0x7f4acc136dd0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4ac00098a0 tx=0x7f4ac001c1b0 comp rx=0 tx=0).stop 2024-04-26T00:26:39.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad4402640 1 -- 172.21.15.37:0/699741244 shutdown_connections 2024-04-26T00:26:39.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad4402640 1 --2- 172.21.15.37:0/699741244 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4acc0a0110 0x7f4acc136dd0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:39.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad4402640 1 --2- 172.21.15.37:0/699741244 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc0726b0 0x7f4acc072a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:39.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad4402640 1 -- 172.21.15.37:0/699741244 >> 172.21.15.37:0/699741244 conn(0x7f4acc0a1d50 msgr2=0x7f4acc0a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:39.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad4402640 1 -- 172.21.15.37:0/699741244 shutdown_connections 2024-04-26T00:26:39.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad4402640 1 -- 172.21.15.37:0/699741244 wait complete. 2024-04-26T00:26:39.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad4402640 10 monclient: build_initial_monmap 2024-04-26T00:26:39.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad4402640 10 monclient: monmap: 2024-04-26T00:26:39.065 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:39.065 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:26:39.065 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:26:39.066 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:26:39.066 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:39.066 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:26:39.066 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:26:39.066 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:26:39.066 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:26:39.066 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:39.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad4402640 1 Processor -- start 2024-04-26T00:26:39.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad4402640 1 -- start start 2024-04-26T00:26:39.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad4402640 10 monclient: init 2024-04-26T00:26:39.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad4402640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:39.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad4402640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:26:39.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad4402640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc0726b0 0x7f4acc1cf200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:39.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad4402640 10 monclient(hunting): picked mon.noname-c con 0x7f4acc0726b0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:26:39.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad4402640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4acc0a0110 0x7f4acc1cf740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:39.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad4402640 10 monclient(hunting): picked mon.noname-a con 0x7f4acc0a0110 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:26:39.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad4402640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4acc137310 0x7f4acc1cfc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:39.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad4402640 10 monclient(hunting): picked mon.noname-b con 0x7f4acc137310 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:26:39.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad4402640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:39.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad4402640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f4acc09bc50 con 0x7f4acc0a0110 2024-04-26T00:26:39.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad4402640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:39.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad4402640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4acc0730e0 con 0x7f4acc137310 2024-04-26T00:26:39.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad4402640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:39.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad4402640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4acc09bdd0 con 0x7f4acc0726b0 2024-04-26T00:26:39.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad4402640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:39.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad4402640 10 monclient(hunting): authenticate will time out at 1380.386462s 2024-04-26T00:26:39.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad219e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc0726b0 0x7f4acc1cf200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:39.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad219e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc0726b0 0x7f4acc1cf200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:59738/0 (socket says 172.21.15.37:59738) 2024-04-26T00:26:39.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad199d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4acc0a0110 0x7f4acc1cf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:39.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad219e640 1 -- 172.21.15.37:0/1175315334 learned_addr learned my addr 172.21.15.37:0/1175315334 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:26:39.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad219e640 10 monclient(hunting): get_auth_request con 0x7f4acc0726b0 auth_method 0 2024-04-26T00:26:39.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad219e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:39.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad219e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:39.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad219e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:39.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad199d640 10 monclient(hunting): get_auth_request con 0x7f4acc0a0110 auth_method 0 2024-04-26T00:26:39.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad199d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:39.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad199d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:39.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad199d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:39.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad199d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:39.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad199d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:39.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad219e640 1 -- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc0726b0 msgr2=0x7f4acc1cf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:26:39.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad199d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:39.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad219e640 1 -- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc0726b0 msgr2=0x7f4acc1cf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:26:39.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad219e640 1 --2- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc0726b0 0x7f4acc1cf200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:26:39.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad219e640 1 --2- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc0726b0 0x7f4acc1cf200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:26:39.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad299f640 1 --2- 172.21.15.37:0/1175315334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4acc137310 0x7f4acc1cfc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:39.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad199d640 10 monclient(hunting): handle_auth_done global_id 54387 payload 824 2024-04-26T00:26:39.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad199d640 1 -- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc0726b0 msgr2=0x7f4acc1cf200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:26:39.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad199d640 1 --2- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc0726b0 0x7f4acc1cf200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:39.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad199d640 1 -- 172.21.15.37:0/1175315334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4acc137310 msgr2=0x7f4acc1cfc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:39.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad199d640 1 --2- 172.21.15.37:0/1175315334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4acc137310 0x7f4acc1cfc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:39.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad199d640 10 monclient: _finish_hunting 0 2024-04-26T00:26:39.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad199d640 1 monclient: found mon.noname-a 2024-04-26T00:26:39.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad199d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:39.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad199d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:26:39.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad199d640 1 -- 172.21.15.37:0/1175315334 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4acc0a05d0 con 0x7f4acc0a0110 2024-04-26T00:26:39.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad299f640 10 monclient: get_auth_request con 0x7f4acc137310 auth_method 0 2024-04-26T00:26:39.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.076+0000 7f4ad299f640 1 --2- 172.21.15.37:0/1175315334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4acc137310 0x7f4acc1cfc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:26:39.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4ad199d640 1 -- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4acc0a0110 msgr2=0x7f4acc1cf740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:39.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4ad199d640 1 --2- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4acc0a0110 0x7f4acc1cf740 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4acc06d050 tx=0x7f4ac0018f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:39.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4acaffd640 1 -- 172.21.15.37:0/1175315334 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac00b0040 con 0x7f4acc0a0110 2024-04-26T00:26:39.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4acaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:39.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4acaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:26:39.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4acaffd640 10 monclient: dump: 2024-04-26T00:26:39.074 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:39.074 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:39.074 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:39.074 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:39.074 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:39.074 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:39.074 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:39.074 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:39.074 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:39.075 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:39.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4acaffd640 10 monclient: _finish_auth 0 2024-04-26T00:26:39.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4acaffd640 10 monclient: _check_auth_tickets 2024-04-26T00:26:39.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4acaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:39.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4ad4402640 5 monclient: authenticate success, global_id 54387 2024-04-26T00:26:39.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4ad4402640 10 monclient: _renew_subs 2024-04-26T00:26:39.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4ad4402640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:26:39.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4acaffd640 1 -- 172.21.15.37:0/1175315334 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ac00aa5b0 con 0x7f4acc0a0110 2024-04-26T00:26:39.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4ad4402640 1 -- 172.21.15.37:0/1175315334 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4acc13d650 con 0x7f4acc0a0110 2024-04-26T00:26:39.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4acaffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:39.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4acaffd640 1 -- 172.21.15.37:0/1175315334 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac00aa9c0 con 0x7f4acc0a0110 2024-04-26T00:26:39.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4acaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:39.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4acaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:26:39.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4acaffd640 10 monclient: dump: 2024-04-26T00:26:39.076 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:39.076 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:39.076 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:39.076 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:39.076 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:39.076 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:39.076 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:39.076 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:39.076 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:39.076 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:39.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4ad099b640 4 set_mon_vals no callback set 2024-04-26T00:26:39.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4ad4402640 10 monclient: _renew_subs 2024-04-26T00:26:39.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4ad4402640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:26:39.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4ad4402640 1 -- 172.21.15.37:0/1175315334 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4acc13e8b0 con 0x7f4acc0a0110 2024-04-26T00:26:39.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4ad4402640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:39.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4ad4402640 1 --2- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc12f9e0 0x7f4acc12fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:39.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4ad4402640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:39.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4ad4402640 1 -- 172.21.15.37:0/1175315334 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4acc13bfd0 con 0x7f4acc12f9e0 2024-04-26T00:26:39.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4ad219e640 1 --2- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc12f9e0 0x7f4acc12fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:39.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4ad219e640 10 monclient: get_auth_request con 0x7f4acc12f9e0 auth_method 0 2024-04-26T00:26:39.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4ad219e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:39.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4ad219e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:39.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4ad219e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:39.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4ad219e640 1 -- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc12f9e0 msgr2=0x7f4acc12fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:26:39.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4ad219e640 1 -- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc12f9e0 msgr2=0x7f4acc12fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:26:39.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4ad219e640 1 --2- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc12f9e0 0x7f4acc12fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:26:39.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4acaffd640 1 -- 172.21.15.37:0/1175315334 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f4ac00aab60 con 0x7f4acc0a0110 2024-04-26T00:26:39.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4ad219e640 1 --2- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc12f9e0 0x7f4acc12fdc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:26:39.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4acaffd640 1 --2- 172.21.15.37:0/1175315334 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4ab8035e80 0x7f4ab8038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:39.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4acaffd640 1 -- 172.21.15.37:0/1175315334 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(260..260 src has 177..260) ==== 3963+0+0 (secure 0 0 0) 0x7f4ac00e2690 con 0x7f4acc0a0110 2024-04-26T00:26:39.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4ad299f640 1 --2- 172.21.15.37:0/1175315334 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4ab8035e80 0x7f4ab8038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:39.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.080+0000 7f4ad299f640 10 monclient: get_auth_request con 0x7f4ab8035e80 auth_method 0 2024-04-26T00:26:39.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.084+0000 7f4ad299f640 1 --2- 172.21.15.37:0/1175315334 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4ab8035e80 0x7f4ab8038340 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4acc13c370 tx=0x7f4ac4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:39.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.284+0000 7f4ad219e640 1 --2- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc12f9e0 0x7f4acc12fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:39.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.284+0000 7f4ad219e640 10 monclient: get_auth_request con 0x7f4acc12f9e0 auth_method 0 2024-04-26T00:26:39.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.284+0000 7f4ad219e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:39.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.284+0000 7f4ad219e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:39.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.284+0000 7f4ad219e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:26:39.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.284+0000 7f4ad219e640 1 -- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc12f9e0 msgr2=0x7f4acc12fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:26:39.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.284+0000 7f4ad219e640 1 -- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc12f9e0 msgr2=0x7f4acc12fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:26:39.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.284+0000 7f4ad219e640 1 --2- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc12f9e0 0x7f4acc12fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:26:39.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.284+0000 7f4ad219e640 1 --2- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc12f9e0 0x7f4acc12fdc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:26:39.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.688+0000 7f4ad219e640 1 --2- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc12f9e0 0x7f4acc12fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:39.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.688+0000 7f4ad219e640 10 monclient: get_auth_request con 0x7f4acc12f9e0 auth_method 0 2024-04-26T00:26:39.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.688+0000 7f4ad219e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:39.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.688+0000 7f4ad219e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:39.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.688+0000 7f4ad219e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:26:39.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.688+0000 7f4ad219e640 1 -- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc12f9e0 msgr2=0x7f4acc12fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:26:39.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.688+0000 7f4ad219e640 1 -- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc12f9e0 msgr2=0x7f4acc12fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:26:39.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.688+0000 7f4ad219e640 1 --2- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc12f9e0 0x7f4acc12fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:26:39.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:39.688+0000 7f4ad219e640 1 --2- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc12f9e0 0x7f4acc12fdc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:26:40.248 INFO:tasks.rados.rados.0.smithi045.stdout:1097: done (1 left) 2024-04-26T00:26:40.248 INFO:tasks.rados.rados.0.smithi045.stdout:1098: done (0 left) 2024-04-26T00:26:40.248 INFO:tasks.rados.rados.0.smithi045.stdout:1099: copy_from oid 43 from oid 46 current snap is 128 2024-04-26T00:26:40.248 INFO:tasks.rados.rados.0.smithi045.stdout:1100: read oid 45 snap -1 2024-04-26T00:26:40.248 INFO:tasks.rados.rados.0.smithi045.stdout:1100: expect (ObjNum 386 snap 117 seq_num 386) 2024-04-26T00:26:40.248 INFO:tasks.rados.rados.0.smithi045.stdout:1101: snap_remove snap 121 2024-04-26T00:26:40.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.492+0000 7f4ad219e640 1 --2- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc12f9e0 0x7f4acc12fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:40.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.492+0000 7f4ad219e640 10 monclient: get_auth_request con 0x7f4acc12f9e0 auth_method 0 2024-04-26T00:26:40.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.492+0000 7f4ad219e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:40.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.492+0000 7f4ad219e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:40.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.492+0000 7f4ad219e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:26:40.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.492+0000 7f4ad219e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:40.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.492+0000 7f4ad219e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:40.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.492+0000 7f4ad219e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:40.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.492+0000 7f4ad219e640 10 monclient(hunting): handle_auth_done global_id 84101 payload 824 2024-04-26T00:26:40.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.492+0000 7f4ad219e640 1 -- 172.21.15.37:0/1175315334 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4acc13d490 con 0x7f4acc12f9e0 2024-04-26T00:26:40.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.492+0000 7f4ad219e640 1 -- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc12f9e0 msgr2=0x7f4acc12fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:40.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.492+0000 7f4ad219e640 1 --2- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc12f9e0 0x7f4acc12fdc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4abc007f90 tx=0x7f4abc0166b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:40.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.496+0000 7f4acaffd640 1 -- 172.21.15.37:0/1175315334 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4abc01d030 con 0x7f4acc12f9e0 2024-04-26T00:26:40.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.496+0000 7f4acaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:40.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.496+0000 7f4acaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:26:40.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.496+0000 7f4acaffd640 10 monclient: dump: 2024-04-26T00:26:40.480 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:40.480 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:40.480 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:40.481 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:40.481 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:40.481 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:40.481 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:40.481 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:40.481 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:40.481 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:40.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.496+0000 7f4acaffd640 1 -- 172.21.15.37:0/1175315334 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4abc019670 con 0x7f4acc12f9e0 2024-04-26T00:26:40.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.496+0000 7f4acaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:40.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.496+0000 7f4acaffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:26:40.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.496+0000 7f4acaffd640 1 -- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc12f9e0 msgr2=0x7f4acc12fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:40.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.496+0000 7f4acaffd640 1 --2- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc12f9e0 0x7f4acc12fdc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4abc007f90 tx=0x7f4abc0166b0 comp rx=0 tx=0).stop 2024-04-26T00:26:40.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.496+0000 7f4acaffd640 1 -- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc12f9e0 msgr2=0x7f4acc12fdc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:26:40.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.496+0000 7f4acaffd640 1 --2- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc12f9e0 0x7f4acc12fdc0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:40.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.504+0000 7f4ad4402640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:26:40.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.504+0000 7f4ad4402640 1 --2- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc130300 0x7f4acc1340f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:40.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.504+0000 7f4ad4402640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:40.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.504+0000 7f4ad4402640 1 -- 172.21.15.37:0/1175315334 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4acc0a3b40 con 0x7f4acc130300 2024-04-26T00:26:40.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.504+0000 7f4ad219e640 1 --2- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc130300 0x7f4acc1340f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:40.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.504+0000 7f4ad219e640 10 monclient: get_auth_request con 0x7f4acc130300 auth_method 0 2024-04-26T00:26:40.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.504+0000 7f4ad219e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:40.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.504+0000 7f4ad219e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:40.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.504+0000 7f4ad219e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:40.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.504+0000 7f4ad219e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:40.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.504+0000 7f4ad219e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:40.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.504+0000 7f4ad219e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:40.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.504+0000 7f4ad219e640 10 monclient(hunting): handle_auth_done global_id 84104 payload 824 2024-04-26T00:26:40.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.504+0000 7f4ad219e640 1 -- 172.21.15.37:0/1175315334 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4acc13d090 con 0x7f4acc130300 2024-04-26T00:26:40.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.504+0000 7f4ad219e640 1 -- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc130300 msgr2=0x7f4acc1340f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:40.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.504+0000 7f4ad219e640 1 --2- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc130300 0x7f4acc1340f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4abc017010 tx=0x7f4abc004ac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:40.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.504+0000 7f4acaffd640 1 -- 172.21.15.37:0/1175315334 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4abc009b60 con 0x7f4acc130300 2024-04-26T00:26:40.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.508+0000 7f4acaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:40.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.508+0000 7f4acaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:26:40.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.508+0000 7f4acaffd640 10 monclient: dump: 2024-04-26T00:26:40.492 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:40.492 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:40.492 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:40.492 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:40.492 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:40.492 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:40.492 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:40.493 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:40.493 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:40.493 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:40.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.508+0000 7f4acaffd640 1 -- 172.21.15.37:0/1175315334 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7f4abc02e410 con 0x7f4acc130300 2024-04-26T00:26:40.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.508+0000 7f4acaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:26:40.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.508+0000 7f4acaffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:26:40.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.508+0000 7f4acaffd640 1 -- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc130300 msgr2=0x7f4acc1340f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:40.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.508+0000 7f4acaffd640 1 --2- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc130300 0x7f4acc1340f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4abc017010 tx=0x7f4abc004ac0 comp rx=0 tx=0).stop 2024-04-26T00:26:40.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.508+0000 7f4acaffd640 1 -- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc130300 msgr2=0x7f4acc1340f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:26:40.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.508+0000 7f4acaffd640 1 --2- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4acc130300 0x7f4acc1340f0 secure :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4abc017010 tx=0x7f4abc004ac0 comp rx=0 tx=0).stop 2024-04-26T00:26:40.494 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:26:40.494 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:26:40.494 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:26:40.494 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:26:40.494 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 52, 2024-04-26T00:26:40.494 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 5443, 2024-04-26T00:26:40.494 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:26:40.495 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:26:40.495 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:26:40.495 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:26:40.495 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:40.495 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 3, 2024-04-26T00:26:40.495 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:26:40.495 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:26:40.495 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:26:40.495 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:26:40.495 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:26:40.495 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:26:40.495 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:26:40.495 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:26:40.495 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:26:40.495 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:26:40.496 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:26:40.496 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:26:40.496 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:26:40.496 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:26:40.496 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:40.496 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:26:40.496 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:26:40.496 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:26:40.496 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:26:40.496 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:26:40.496 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:26:40.496 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:26:40.496 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:26:40.496 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:26:40.496 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:26:40.497 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:26:40.497 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:26:40.497 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:26:40.497 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:40.497 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:40.497 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:26:40.497 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:26:40.497 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:26:40.497 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:26:40.497 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:26:40.497 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:26:40.497 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:26:40.497 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:26:40.497 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:26:40.498 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:26:40.498 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:26:40.498 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:26:40.498 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:26:40.498 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:26:40.498 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:26:40.498 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:26:40.498 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:26:40.498 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:26:40.498 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:26:40.498 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:26:40.498 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:26:40.498 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:26:40.498 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:26:40.499 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:26:40.499 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:26:40.499 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:26:40.499 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:26:40.499 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:26:40.499 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:40.499 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:26:40.499 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:40.499 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:26:40.499 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:40.499 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:26:40.499 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:26:40.499 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:26:40.499 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:26:40.499 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:40.500 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:26:40.500 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:26:40.500 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:40.500 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:40.500 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:40.500 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:26:40.500 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:26:40.500 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:40.500 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:40.500 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:40.500 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:40.500 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:26:40.500 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:26:40.500 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:26:40.500 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:26:40.501 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:26:40.501 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:40.501 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:40.501 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:26:40.501 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:26:40.501 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:26:40.501 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:26:40.501 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:40.501 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:26:40.501 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:26:40.501 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:40.501 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:40.501 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:40.501 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:26:40.501 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:26:40.502 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:40.502 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:40.502 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:40.502 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:40.502 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:26:40.502 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:26:40.502 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:26:40.502 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:26:40.502 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:26:40.502 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:40.502 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:40.502 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:26:40.502 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:26:40.502 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:26:40.502 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:26:40.503 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:40.503 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:26:40.503 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:26:40.503 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:40.503 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:40.503 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:40.503 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:26:40.503 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:26:40.503 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:40.503 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:40.503 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:40.503 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:40.503 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:26:40.503 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:26:40.503 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:26:40.504 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:26:40.504 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:26:40.504 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:40.504 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:40.504 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:40.504 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:26:40.504 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:26:40.504 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:40.504 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:26:40.504 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:26:40.504 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:26:40.504 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:40.504 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:40.504 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:26:40.504 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:40.505 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:26:40.505 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:26:40.505 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:26:40.505 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:40.505 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:40.505 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:40.505 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:26:40.505 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:26:40.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.508+0000 7f4ad4402640 1 -- 172.21.15.37:0/1175315334 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4ab8035e80 msgr2=0x7f4ab8038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:40.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.508+0000 7f4ad4402640 1 --2- 172.21.15.37:0/1175315334 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4ab8035e80 0x7f4ab8038340 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4acc13c370 tx=0x7f4ac4009040 comp rx=0 tx=0).stop 2024-04-26T00:26:40.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.508+0000 7f4ad4402640 10 monclient: shutdown 2024-04-26T00:26:40.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.508+0000 7f4ad4402640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:40.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.508+0000 7f4ad4402640 1 -- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4acc0a0110 msgr2=0x7f4acc1cf740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:40.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.508+0000 7f4ad4402640 1 --2- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4acc0a0110 0x7f4acc1cf740 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4acc06d050 tx=0x7f4ac0018f50 comp rx=0 tx=0).stop 2024-04-26T00:26:40.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.508+0000 7f4ad299f640 1 -- 172.21.15.37:0/1175315334 reap_dead start 2024-04-26T00:26:40.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.508+0000 7f4ad4402640 1 -- 172.21.15.37:0/1175315334 shutdown_connections 2024-04-26T00:26:40.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.508+0000 7f4ad4402640 1 --2- 172.21.15.37:0/1175315334 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4acc0a0110 0x7f4acc1cf740 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:40.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.508+0000 7f4ad4402640 1 -- 172.21.15.37:0/1175315334 >> 172.21.15.37:0/1175315334 conn(0x7f4acc0a1d50 msgr2=0x7f4acc0a4ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:40.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.508+0000 7f4ad4402640 1 -- 172.21.15.37:0/1175315334 shutdown_connections 2024-04-26T00:26:40.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.508+0000 7f4ad4402640 1 -- 172.21.15.37:0/1175315334 wait complete. 2024-04-26T00:26:40.519 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:26:40.519 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:26:40.519 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:26:40.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.664+0000 7f3c06544640 10 monclient: get_monmap_and_config 2024-04-26T00:26:40.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.664+0000 7f3c06544640 10 monclient: build_initial_monmap 2024-04-26T00:26:40.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.664+0000 7f3c06544640 10 monclient: monmap: 2024-04-26T00:26:40.649 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:40.649 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:40.649 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:26:40.668873+0000 2024-04-26T00:26:40.649 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:26:40.668873+0000 2024-04-26T00:26:40.650 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:40.650 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:40.650 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:26:40.650 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:26:40.650 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:26:40.650 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:26:40.650 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:26:40.650 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:40.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.664+0000 7f3c06544640 1 Processor -- start 2024-04-26T00:26:40.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.664+0000 7f3c06544640 1 -- start start 2024-04-26T00:26:40.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.664+0000 7f3c06544640 10 monclient: init 2024-04-26T00:26:40.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.664+0000 7f3c06544640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:40.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.664+0000 7f3c06544640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-04-26T00:26:40.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.664+0000 7f3c06544640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3c00136f20 addr v1:172.21.15.45:6789/0 2024-04-26T00:26:40.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.664+0000 7f3c06544640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3c00137a10 0x7f3c0009bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:40.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.664+0000 7f3c06544640 10 monclient(hunting): picked mon.noname-c con 0x7f3c00137a10 addr v2:172.21.15.45:3300/0 2024-04-26T00:26:40.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.664+0000 7f3c06544640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3c0013c450 addr v1:172.21.15.37:6789/0 2024-04-26T00:26:40.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.664+0000 7f3c06544640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c000a50f0 con 0x7f3c0013c450 2024-04-26T00:26:40.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.664+0000 7f3c06544640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c0009b330 con 0x7f3c00136f20 2024-04-26T00:26:40.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.664+0000 7f3c06544640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:40.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.664+0000 7f3c06544640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f3c0009b610 con 0x7f3c00137a10 2024-04-26T00:26:40.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.664+0000 7f3c06544640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:40.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.664+0000 7f3c06544640 10 monclient(hunting): authenticate will time out at 1381.974556s 2024-04-26T00:26:40.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.664+0000 7f3c04ae1640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f3c0013c450 0x7f3c0013e840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:33282/0 (socket says 172.21.15.37:33282) 2024-04-26T00:26:40.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.664+0000 7f3c04ae1640 1 -- 172.21.15.37:0/4112334575 learned_addr learned my addr 172.21.15.37:0/4112334575 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:26:40.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.664+0000 7f3bff7fe640 1 --2- 172.21.15.37:0/4112334575 >> v2:172.21.15.45:3300/0 conn(0x7f3c00137a10 0x7f3c0009bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:40.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.664+0000 7f3bff7fe640 10 monclient(hunting): get_auth_request con 0x7f3c00137a10 auth_method 0 2024-04-26T00:26:40.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.664+0000 7f3bff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:40.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.664+0000 7f3bff7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:40.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.664+0000 7f3bff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:40.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.664+0000 7f3bff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:40.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.664+0000 7f3bff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:40.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.664+0000 7f3bff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:40.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.664+0000 7f3bff7fe640 10 monclient(hunting): handle_auth_done global_id 24607 payload 274 2024-04-26T00:26:40.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.664+0000 7f3bff7fe640 1 -- 172.21.15.37:0/4112334575 >> v1:172.21.15.45:6789/0 conn(0x7f3c00136f20 legacy=0x7f3c00137300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:26:40.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.664+0000 7f3bff7fe640 1 -- 172.21.15.37:0/4112334575 >> v1:172.21.15.37:6789/0 conn(0x7f3c0013c450 legacy=0x7f3c0013e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:26:40.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.664+0000 7f3bff7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:26:40.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.664+0000 7f3bff7fe640 1 monclient: found mon.noname-c 2024-04-26T00:26:40.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.664+0000 7f3bff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:40.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.664+0000 7f3bff7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:26:40.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.664+0000 7f3bff7fe640 1 -- 172.21.15.37:0/4112334575 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c0009b8f0 con 0x7f3c00137a10 2024-04-26T00:26:40.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3bff7fe640 1 -- 172.21.15.37:0/4112334575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c00137a10 msgr2=0x7f3c0009bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:40.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3bff7fe640 1 --2- 172.21.15.37:0/4112334575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c00137a10 0x7f3c0009bff0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3be80097c0 tx=0x7f3be801c370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41139528d4fe208e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:40.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3bfeffd640 1 -- 172.21.15.37:0/4112334575 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3be80b2070 con 0x7f3c00137a10 2024-04-26T00:26:40.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3bfeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:40.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3bfeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:26:40.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3bfeffd640 10 monclient: dump: 2024-04-26T00:26:40.657 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:40.657 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:40.657 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:40.657 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:40.657 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:40.657 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:40.657 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:40.658 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:40.658 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:40.658 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:40.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3bfeffd640 10 monclient: _finish_auth 0 2024-04-26T00:26:40.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3bfeffd640 10 monclient: _check_auth_tickets 2024-04-26T00:26:40.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3bfeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:40.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3bfeffd640 1 -- 172.21.15.37:0/4112334575 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3be80ad640 con 0x7f3c00137a10 2024-04-26T00:26:40.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3bfeffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:40.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3bfeffd640 1 -- 172.21.15.37:0/4112334575 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3be80ada10 con 0x7f3c00137a10 2024-04-26T00:26:40.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3c06544640 5 monclient: authenticate success, global_id 24607 2024-04-26T00:26:40.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3c06544640 10 monclient: get_monmap_and_config success 2024-04-26T00:26:40.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3c06544640 4 set_mon_vals no callback set 2024-04-26T00:26:40.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3bfeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:40.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3bfeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:26:40.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3bfeffd640 10 monclient: dump: 2024-04-26T00:26:40.659 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:40.659 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:40.659 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:40.659 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:40.659 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:40.659 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:40.659 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:40.659 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:40.659 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:40.659 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:40.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3c06544640 10 monclient: shutdown 2024-04-26T00:26:40.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3c06544640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:40.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3c06544640 1 -- 172.21.15.37:0/4112334575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c00137a10 msgr2=0x7f3c0009bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:40.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3c06544640 1 --2- 172.21.15.37:0/4112334575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c00137a10 0x7f3c0009bff0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3be80097c0 tx=0x7f3be801c370 comp rx=0 tx=0).stop 2024-04-26T00:26:40.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3c06544640 1 -- 172.21.15.37:0/4112334575 shutdown_connections 2024-04-26T00:26:40.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3c06544640 1 --2- 172.21.15.37:0/4112334575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c00137a10 0x7f3c0009bff0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:40.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3c06544640 1 -- 172.21.15.37:0/4112334575 >> 172.21.15.37:0/4112334575 conn(0x7f3c000a61b0 msgr2=0x7f3c000a65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:40.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3c06544640 1 -- 172.21.15.37:0/4112334575 shutdown_connections 2024-04-26T00:26:40.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3c06544640 1 -- 172.21.15.37:0/4112334575 wait complete. 2024-04-26T00:26:40.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3c06544640 10 monclient: build_initial_monmap 2024-04-26T00:26:40.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3c06544640 10 monclient: monmap: 2024-04-26T00:26:40.665 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:40.665 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:26:40.666 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:26:40.666 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:26:40.666 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:40.666 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:26:40.666 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:26:40.666 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:26:40.666 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:26:40.666 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:40.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3c06544640 1 Processor -- start 2024-04-26T00:26:40.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3c06544640 1 -- start start 2024-04-26T00:26:40.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3c06544640 10 monclient: init 2024-04-26T00:26:40.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3c06544640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:40.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3c06544640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:26:40.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3c06544640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c00136f20 0x7f3c001cef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:40.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3c06544640 10 monclient(hunting): picked mon.noname-a con 0x7f3c00136f20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:26:40.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3c06544640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c00137a10 0x7f3c001cf4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:40.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3c06544640 10 monclient(hunting): picked mon.noname-b con 0x7f3c00137a10 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:26:40.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3c06544640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3c0013c450 0x7f3c001cf9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:40.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3c06544640 10 monclient(hunting): picked mon.noname-c con 0x7f3c0013c450 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:26:40.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3c06544640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:40.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3c06544640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f3c00142b10 con 0x7f3c00136f20 2024-04-26T00:26:40.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3c06544640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:40.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3c06544640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3c00142990 con 0x7f3c00137a10 2024-04-26T00:26:40.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3c06544640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:40.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3c06544640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f3c00142c90 con 0x7f3c0013c450 2024-04-26T00:26:40.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3c06544640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:40.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3c06544640 10 monclient(hunting): authenticate will time out at 1381.983753s 2024-04-26T00:26:40.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3c04ae1640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3c0013c450 0x7f3c001cf9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:40.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.672+0000 7f3bfffff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c00136f20 0x7f3c001cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:40.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3c04ae1640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3c0013c450 msgr2=0x7f3c001cf9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:26:40.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3c04ae1640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3c0013c450 msgr2=0x7f3c001cf9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:26:40.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bfffff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c00136f20 0x7f3c001cef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50014/0 (socket says 172.21.15.37:50014) 2024-04-26T00:26:40.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3c04ae1640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3c0013c450 0x7f3c001cf9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:26:40.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bfffff640 1 -- 172.21.15.37:0/916900095 learned_addr learned my addr 172.21.15.37:0/916900095 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:26:40.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bfffff640 10 monclient(hunting): get_auth_request con 0x7f3c00136f20 auth_method 0 2024-04-26T00:26:40.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bfffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:40.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bfffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:40.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bfffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:40.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3c04ae1640 1 --2- 172.21.15.37:0/916900095 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3c0013c450 0x7f3c001cf9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:26:40.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bfffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:40.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bfffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:40.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bfffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:40.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bff7fe640 1 --2- 172.21.15.37:0/916900095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c00137a10 0x7f3c001cf4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:40.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bff7fe640 10 monclient(hunting): get_auth_request con 0x7f3c00137a10 auth_method 0 2024-04-26T00:26:40.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:40.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bff7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:40.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:40.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bfffff640 10 monclient(hunting): handle_auth_done global_id 54390 payload 824 2024-04-26T00:26:40.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bfffff640 1 -- 172.21.15.37:0/916900095 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3c0013c450 msgr2=0x7f3c001cf9e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:26:40.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bfffff640 1 --2- 172.21.15.37:0/916900095 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3c0013c450 0x7f3c001cf9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:40.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bfffff640 1 -- 172.21.15.37:0/916900095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c00137a10 msgr2=0x7f3c001cf4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:40.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bfffff640 1 --2- 172.21.15.37:0/916900095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c00137a10 0x7f3c001cf4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:40.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bfffff640 10 monclient: _finish_hunting 0 2024-04-26T00:26:40.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bfffff640 1 monclient: found mon.noname-a 2024-04-26T00:26:40.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bfffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:40.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bfffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:26:40.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bfffff640 1 -- 172.21.15.37:0/916900095 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c000a3610 con 0x7f3c00136f20 2024-04-26T00:26:40.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bff7fe640 1 --2- 172.21.15.37:0/916900095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c00137a10 0x7f3c001cf4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:26:40.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bfffff640 1 -- 172.21.15.37:0/916900095 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c00136f20 msgr2=0x7f3c001cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:40.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bfffff640 1 --2- 172.21.15.37:0/916900095 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c00136f20 0x7f3c001cef60 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3bf000c7c0 tx=0x7f3bf000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:40.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bfcff9640 1 -- 172.21.15.37:0/916900095 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf0016020 con 0x7f3c00136f20 2024-04-26T00:26:40.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bfcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:40.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bfcff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:26:40.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bfcff9640 10 monclient: dump: 2024-04-26T00:26:40.670 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:40.670 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:40.670 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:40.670 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:40.670 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:40.670 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:40.670 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:40.670 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:40.671 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:40.671 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:40.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bfcff9640 10 monclient: _finish_auth 0 2024-04-26T00:26:40.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bfcff9640 10 monclient: _check_auth_tickets 2024-04-26T00:26:40.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bfcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:40.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bfcff9640 1 -- 172.21.15.37:0/916900095 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bf000ac60 con 0x7f3c00136f20 2024-04-26T00:26:40.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3c06544640 5 monclient: authenticate success, global_id 54390 2024-04-26T00:26:40.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3c06544640 10 monclient: _renew_subs 2024-04-26T00:26:40.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3c06544640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:26:40.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3c06544640 1 -- 172.21.15.37:0/916900095 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c0013ad50 con 0x7f3c00136f20 2024-04-26T00:26:40.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bfcff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:40.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bfcff9640 1 -- 172.21.15.37:0/916900095 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf0012660 con 0x7f3c00136f20 2024-04-26T00:26:40.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bfcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:40.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bfcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:26:40.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bfcff9640 10 monclient: dump: 2024-04-26T00:26:40.672 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:40.672 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:40.672 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:40.672 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:40.672 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:40.672 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:40.672 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:40.672 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:40.672 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:40.672 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:40.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bfdffb640 4 set_mon_vals no callback set 2024-04-26T00:26:40.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3c06544640 10 monclient: _renew_subs 2024-04-26T00:26:40.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3c06544640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:26:40.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3c06544640 1 -- 172.21.15.37:0/916900095 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c0013a790 con 0x7f3c00136f20 2024-04-26T00:26:40.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3c06544640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:40.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3c06544640 1 --2- 172.21.15.37:0/916900095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c0013bbe0 0x7f3c001354d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:40.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3c06544640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:40.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3c06544640 1 -- 172.21.15.37:0/916900095 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3c00139680 con 0x7f3c0013bbe0 2024-04-26T00:26:40.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.676+0000 7f3bfcff9640 1 -- 172.21.15.37:0/916900095 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f3bf0019020 con 0x7f3c00136f20 2024-04-26T00:26:40.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.680+0000 7f3bfcff9640 1 --2- 172.21.15.37:0/916900095 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3be0035e80 0x7f3be0038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:40.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.680+0000 7f3bfcff9640 1 -- 172.21.15.37:0/916900095 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(261..261 src has 177..261) ==== 3963+0+0 (secure 0 0 0) 0x7f3bf0048c70 con 0x7f3c00136f20 2024-04-26T00:26:40.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.680+0000 7f3bff7fe640 1 --2- 172.21.15.37:0/916900095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c0013bbe0 0x7f3c001354d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:40.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.680+0000 7f3bff7fe640 10 monclient: get_auth_request con 0x7f3c0013bbe0 auth_method 0 2024-04-26T00:26:40.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.680+0000 7f3bff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:40.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.680+0000 7f3bff7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:40.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.680+0000 7f3bff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:40.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.680+0000 7f3c04ae1640 1 --2- 172.21.15.37:0/916900095 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3be0035e80 0x7f3be0038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:40.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.680+0000 7f3c04ae1640 10 monclient: get_auth_request con 0x7f3be0035e80 auth_method 0 2024-04-26T00:26:40.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.680+0000 7f3bff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:40.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.680+0000 7f3bff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:40.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.680+0000 7f3bff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:40.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.680+0000 7f3bff7fe640 10 monclient(hunting): handle_auth_done global_id 24613 payload 824 2024-04-26T00:26:40.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.680+0000 7f3bff7fe640 1 -- 172.21.15.37:0/916900095 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3c0004f510 con 0x7f3c0013bbe0 2024-04-26T00:26:40.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.680+0000 7f3c04ae1640 1 --2- 172.21.15.37:0/916900095 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3be0035e80 0x7f3be0038340 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3c00136ef0 tx=0x7f3bf8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:40.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.680+0000 7f3bff7fe640 1 -- 172.21.15.37:0/916900095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c0013bbe0 msgr2=0x7f3c001354d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:40.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.680+0000 7f3bff7fe640 1 --2- 172.21.15.37:0/916900095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c0013bbe0 0x7f3c001354d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3c0009bbd0 tx=0x7f3be8019110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:40.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.680+0000 7f3bfcff9640 1 -- 172.21.15.37:0/916900095 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3be80b2040 con 0x7f3c0013bbe0 2024-04-26T00:26:40.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.680+0000 7f3bfcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:40.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.680+0000 7f3bfcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:26:40.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.680+0000 7f3bfcff9640 10 monclient: dump: 2024-04-26T00:26:40.676 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:40.676 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:40.676 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:40.676 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:40.676 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:40.676 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:40.676 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:40.676 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:40.676 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:40.676 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:40.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.684+0000 7f3bfcff9640 1 -- 172.21.15.37:0/916900095 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3be80b7650 con 0x7f3c0013bbe0 2024-04-26T00:26:40.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.684+0000 7f3bfcff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:40.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.684+0000 7f3bfcff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:26:40.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.684+0000 7f3bfcff9640 1 -- 172.21.15.37:0/916900095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c0013bbe0 msgr2=0x7f3c001354d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:40.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.684+0000 7f3bfcff9640 1 --2- 172.21.15.37:0/916900095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c0013bbe0 0x7f3c001354d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3c0009bbd0 tx=0x7f3be8019110 comp rx=0 tx=0).stop 2024-04-26T00:26:40.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.684+0000 7f3bfcff9640 1 -- 172.21.15.37:0/916900095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c0013bbe0 msgr2=0x7f3c001354d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:26:40.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.684+0000 7f3bfcff9640 1 --2- 172.21.15.37:0/916900095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c0013bbe0 0x7f3c001354d0 secure :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3c0009bbd0 tx=0x7f3be8019110 comp rx=0 tx=0).stop 2024-04-26T00:26:40.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.688+0000 7f3c06544640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:26:40.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.688+0000 7f3c06544640 1 --2- 172.21.15.37:0/916900095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c0002c000 0x7f3c0002c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:40.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.688+0000 7f3c06544640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:40.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.688+0000 7f3c06544640 1 -- 172.21.15.37:0/916900095 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3c0013df40 con 0x7f3c0002c000 2024-04-26T00:26:40.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.688+0000 7f3bff7fe640 1 --2- 172.21.15.37:0/916900095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c0002c000 0x7f3c0002c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:40.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.688+0000 7f3bff7fe640 10 monclient: get_auth_request con 0x7f3c0002c000 auth_method 0 2024-04-26T00:26:40.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.688+0000 7f3bff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:40.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.688+0000 7f3bff7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:40.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.688+0000 7f3bff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:40.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.688+0000 7f3bff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:40.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.688+0000 7f3bff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:40.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.688+0000 7f3bff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:40.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.688+0000 7f3bff7fe640 10 monclient(hunting): handle_auth_done global_id 24616 payload 824 2024-04-26T00:26:40.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.688+0000 7f3bff7fe640 1 -- 172.21.15.37:0/916900095 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3c00075f10 con 0x7f3c0002c000 2024-04-26T00:26:40.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.692+0000 7f3bff7fe640 1 -- 172.21.15.37:0/916900095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c0002c000 msgr2=0x7f3c0002c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:40.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.692+0000 7f3bff7fe640 1 --2- 172.21.15.37:0/916900095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c0002c000 0x7f3c0002c3e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3be80097c0 tx=0x7f3be80a7000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:40.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.692+0000 7f3bfcff9640 1 -- 172.21.15.37:0/916900095 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3be80b2040 con 0x7f3c0002c000 2024-04-26T00:26:40.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.692+0000 7f3bfcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:40.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.692+0000 7f3bfcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:26:40.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.692+0000 7f3bfcff9640 10 monclient: dump: 2024-04-26T00:26:40.679 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:40.679 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:40.679 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:40.679 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:40.679 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:40.679 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:40.680 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:40.680 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:40.680 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:40.680 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:40.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.692+0000 7f3bfcff9640 1 -- 172.21.15.37:0/916900095 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4719 (secure 0 0 0) 0x7f3be80c4570 con 0x7f3c0002c000 2024-04-26T00:26:40.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.692+0000 7f3bfcff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:26:40.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.692+0000 7f3bfcff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:26:40.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.692+0000 7f3bfcff9640 1 -- 172.21.15.37:0/916900095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c0002c000 msgr2=0x7f3c0002c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:40.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.692+0000 7f3bfcff9640 1 --2- 172.21.15.37:0/916900095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c0002c000 0x7f3c0002c3e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3be80097c0 tx=0x7f3be80a7000 comp rx=0 tx=0).stop 2024-04-26T00:26:40.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.692+0000 7f3bfcff9640 1 -- 172.21.15.37:0/916900095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c0002c000 msgr2=0x7f3c0002c3e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:26:40.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.692+0000 7f3bfcff9640 1 --2- 172.21.15.37:0/916900095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3c0002c000 0x7f3c0002c3e0 secure :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3be80097c0 tx=0x7f3be80a7000 comp rx=0 tx=0).stop 2024-04-26T00:26:40.681 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:26:40.681 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:26:40.681 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:26:40.681 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:26:40.681 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 52, 2024-04-26T00:26:40.681 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 308863, 2024-04-26T00:26:40.681 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:26:40.682 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:26:40.682 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:26:40.682 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:26:40.682 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:40.682 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 4, 2024-04-26T00:26:40.682 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:26:40.682 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:26:40.682 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:26:40.682 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:26:40.682 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:26:40.682 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:26:40.682 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:26:40.682 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:26:40.683 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:26:40.683 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:26:40.683 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:26:40.683 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:26:40.683 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:26:40.683 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:26:40.683 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:40.683 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:26:40.683 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:26:40.683 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:26:40.683 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:26:40.683 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:26:40.683 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:26:40.683 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:26:40.683 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:26:40.684 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:26:40.684 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:26:40.684 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:26:40.684 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:26:40.684 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:26:40.684 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:40.684 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:40.684 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:26:40.684 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:26:40.684 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:26:40.684 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:26:40.684 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:26:40.684 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:26:40.684 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:26:40.684 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:26:40.685 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:26:40.685 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:26:40.685 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:26:40.685 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:26:40.685 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:26:40.685 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:26:40.685 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:26:40.685 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:26:40.685 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:26:40.685 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:26:40.685 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:26:40.685 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:26:40.685 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:26:40.685 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:26:40.685 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:26:40.686 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:26:40.686 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:26:40.686 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:26:40.686 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:26:40.686 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:26:40.686 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:40.686 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:26:40.686 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:40.686 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:26:40.686 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:40.686 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:26:40.686 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:26:40.686 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:26:40.686 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:26:40.687 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:40.687 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:26:40.687 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:26:40.687 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:40.687 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:40.687 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:40.687 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:26:40.687 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:26:40.687 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:40.687 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:40.687 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:40.687 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:40.687 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:26:40.687 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:26:40.687 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:26:40.688 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:26:40.688 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:26:40.688 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:40.688 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:40.688 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:26:40.688 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:26:40.688 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:26:40.688 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:26:40.688 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:40.688 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:26:40.688 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:26:40.688 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:40.688 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:40.688 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:40.689 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:26:40.689 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:26:40.689 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:40.689 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:40.689 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:40.689 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:40.689 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:26:40.689 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:26:40.689 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:26:40.689 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:26:40.689 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:26:40.689 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:40.689 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:40.689 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:26:40.689 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:26:40.690 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:26:40.690 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:26:40.690 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:40.690 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:26:40.690 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:26:40.690 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:40.690 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:40.690 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:40.690 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:26:40.690 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:26:40.690 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:40.690 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:40.690 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:40.690 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:40.691 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:26:40.691 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:26:40.691 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:26:40.691 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:26:40.691 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:26:40.691 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:40.691 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:40.691 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:40.691 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:26:40.691 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:26:40.691 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:40.691 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:26:40.691 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:26:40.691 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:26:40.692 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:40.692 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:40.692 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:26:40.692 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:40.692 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:26:40.692 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:26:40.692 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:26:40.692 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:40.692 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:40.692 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:26:40.692 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:40.692 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:26:40.692 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:26:40.692 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:26:40.692 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:40.693 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:40.693 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:40.693 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:26:40.693 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:26:40.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.696+0000 7f3c06544640 1 -- 172.21.15.37:0/916900095 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3be0035e80 msgr2=0x7f3be0038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:40.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.696+0000 7f3c06544640 1 --2- 172.21.15.37:0/916900095 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3be0035e80 0x7f3be0038340 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3c00136ef0 tx=0x7f3bf8009040 comp rx=0 tx=0).stop 2024-04-26T00:26:40.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.696+0000 7f3c06544640 10 monclient: shutdown 2024-04-26T00:26:40.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.696+0000 7f3c06544640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:40.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.696+0000 7f3c06544640 1 -- 172.21.15.37:0/916900095 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c00136f20 msgr2=0x7f3c001cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:40.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.696+0000 7f3c06544640 1 --2- 172.21.15.37:0/916900095 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c00136f20 0x7f3c001cef60 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3bf000c7c0 tx=0x7f3bf000cc80 comp rx=0 tx=0).stop 2024-04-26T00:26:40.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.696+0000 7f3c04ae1640 1 -- 172.21.15.37:0/916900095 reap_dead start 2024-04-26T00:26:40.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.696+0000 7f3c04ae1640 1 -- 172.21.15.37:0/916900095 reap_dead start 2024-04-26T00:26:40.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.696+0000 7f3c06544640 1 -- 172.21.15.37:0/916900095 shutdown_connections 2024-04-26T00:26:40.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.696+0000 7f3c06544640 1 -- 172.21.15.37:0/916900095 >> 172.21.15.37:0/916900095 conn(0x7f3c000a61b0 msgr2=0x7f3c000a82a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:40.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.696+0000 7f3c06544640 1 -- 172.21.15.37:0/916900095 shutdown_connections 2024-04-26T00:26:40.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.696+0000 7f3c06544640 1 -- 172.21.15.37:0/916900095 wait complete. 2024-04-26T00:26:40.697 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:26:40.697 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-26T00:26:40.697 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-26T00:26:40.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.832+0000 7f145418f640 10 monclient: get_monmap_and_config 2024-04-26T00:26:40.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.832+0000 7f145418f640 10 monclient: build_initial_monmap 2024-04-26T00:26:40.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.832+0000 7f145418f640 10 monclient: monmap: 2024-04-26T00:26:40.820 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:40.820 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:40.820 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:26:40.839111+0000 2024-04-26T00:26:40.820 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:26:40.839111+0000 2024-04-26T00:26:40.820 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:40.820 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:40.820 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:26:40.820 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:26:40.820 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:26:40.820 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:26:40.820 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:26:40.820 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:40.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.836+0000 7f145418f640 1 Processor -- start 2024-04-26T00:26:40.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.836+0000 7f145418f640 1 -- start start 2024-04-26T00:26:40.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.836+0000 7f145418f640 10 monclient: init 2024-04-26T00:26:40.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.836+0000 7f145418f640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:40.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.836+0000 7f145418f640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-04-26T00:26:40.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.836+0000 7f145418f640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f144c0a0260 0x7f144c0a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:40.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.836+0000 7f145418f640 10 monclient(hunting): picked mon.noname-c con 0x7f144c0a0260 addr v2:172.21.15.45:3300/0 2024-04-26T00:26:40.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.836+0000 7f145418f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f144c134b30 addr v1:172.21.15.45:6789/0 2024-04-26T00:26:40.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.836+0000 7f145418f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f144c135660 addr v1:172.21.15.37:6789/0 2024-04-26T00:26:40.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.836+0000 7f145418f640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f144c0a1130 con 0x7f144c135660 2024-04-26T00:26:40.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.836+0000 7f145418f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f144c06cff0 con 0x7f144c134b30 2024-04-26T00:26:40.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.836+0000 7f145418f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:40.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.836+0000 7f145418f640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f144c09ec10 con 0x7f144c0a0260 2024-04-26T00:26:40.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.836+0000 7f145418f640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:40.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.836+0000 7f145418f640 10 monclient(hunting): authenticate will time out at 1382.144858s 2024-04-26T00:26:40.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.836+0000 7f145272c640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f144c135660 0x7f144c13ce70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:33288/0 (socket says 172.21.15.37:33288) 2024-04-26T00:26:40.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.836+0000 7f145272c640 1 -- 172.21.15.37:0/3143721728 learned_addr learned my addr 172.21.15.37:0/3143721728 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:26:40.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.836+0000 7f1451f2b640 1 --2- 172.21.15.37:0/3143721728 >> v2:172.21.15.45:3300/0 conn(0x7f144c0a0260 0x7f144c0a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:40.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.836+0000 7f1451f2b640 10 monclient(hunting): get_auth_request con 0x7f144c0a0260 auth_method 0 2024-04-26T00:26:40.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.836+0000 7f1451f2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:40.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.836+0000 7f1451f2b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:40.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.836+0000 7f1451f2b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:40.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.836+0000 7f1451f2b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:40.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.836+0000 7f1451f2b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:40.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.836+0000 7f1451f2b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:40.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.836+0000 7f1451f2b640 10 monclient(hunting): handle_auth_done global_id 24619 payload 274 2024-04-26T00:26:40.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.836+0000 7f1451f2b640 1 -- 172.21.15.37:0/3143721728 >> v1:172.21.15.45:6789/0 conn(0x7f144c134b30 legacy=0x7f144c134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:26:40.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.836+0000 7f1451f2b640 1 -- 172.21.15.37:0/3143721728 >> v1:172.21.15.37:6789/0 conn(0x7f144c135660 legacy=0x7f144c13ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:26:40.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.836+0000 7f1451f2b640 10 monclient: _finish_hunting 0 2024-04-26T00:26:40.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.836+0000 7f1451f2b640 1 monclient: found mon.noname-c 2024-04-26T00:26:40.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.836+0000 7f1451f2b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:40.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.836+0000 7f1451f2b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:26:40.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.836+0000 7f1451f2b640 1 -- 172.21.15.37:0/3143721728 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f144c09bc50 con 0x7f144c0a0260 2024-04-26T00:26:40.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.840+0000 7f1451f2b640 1 -- 172.21.15.37:0/3143721728 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f144c0a0260 msgr2=0x7f144c0a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:40.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.840+0000 7f1451f2b640 1 --2- 172.21.15.37:0/3143721728 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f144c0a0260 0x7f144c0a0640 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f14440097c0 tx=0x7f144401c100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6af18ff0867ff38 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:40.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f1450f29640 1 -- 172.21.15.37:0/3143721728 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14440b1070 con 0x7f144c0a0260 2024-04-26T00:26:40.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f1450f29640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:40.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f1450f29640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:26:40.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f1450f29640 10 monclient: dump: 2024-04-26T00:26:40.829 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:40.829 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:40.829 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:40.829 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:40.829 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:40.829 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:40.829 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:40.830 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:40.830 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:40.830 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:40.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f1450f29640 10 monclient: _finish_auth 0 2024-04-26T00:26:40.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f1450f29640 10 monclient: _check_auth_tickets 2024-04-26T00:26:40.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f1450f29640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:40.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f1450f29640 1 -- 172.21.15.37:0/3143721728 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14440ac440 con 0x7f144c0a0260 2024-04-26T00:26:40.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145418f640 5 monclient: authenticate success, global_id 24619 2024-04-26T00:26:40.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145418f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:26:40.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f1450f29640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:40.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145418f640 10 monclient: get_monmap_and_config success 2024-04-26T00:26:40.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145418f640 4 set_mon_vals no callback set 2024-04-26T00:26:40.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f1450f29640 1 -- 172.21.15.37:0/3143721728 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14440ac810 con 0x7f144c0a0260 2024-04-26T00:26:40.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f1450f29640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:40.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f1450f29640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:26:40.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f1450f29640 10 monclient: dump: 2024-04-26T00:26:40.831 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:40.831 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:40.831 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:40.831 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:40.831 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:40.831 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:40.831 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:40.831 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:40.831 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:40.831 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:40.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145418f640 10 monclient: shutdown 2024-04-26T00:26:40.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145418f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:40.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145418f640 1 -- 172.21.15.37:0/3143721728 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f144c0a0260 msgr2=0x7f144c0a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:40.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145418f640 1 --2- 172.21.15.37:0/3143721728 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f144c0a0260 0x7f144c0a0640 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f14440097c0 tx=0x7f144401c100 comp rx=0 tx=0).stop 2024-04-26T00:26:40.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145418f640 1 -- 172.21.15.37:0/3143721728 shutdown_connections 2024-04-26T00:26:40.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145418f640 1 --2- 172.21.15.37:0/3143721728 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f144c0a0260 0x7f144c0a0640 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:40.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145418f640 1 -- 172.21.15.37:0/3143721728 >> 172.21.15.37:0/3143721728 conn(0x7f144c0a1ad0 msgr2=0x7f144c0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:40.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145418f640 1 -- 172.21.15.37:0/3143721728 shutdown_connections 2024-04-26T00:26:40.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145418f640 1 -- 172.21.15.37:0/3143721728 wait complete. 2024-04-26T00:26:40.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145418f640 10 monclient: build_initial_monmap 2024-04-26T00:26:40.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145418f640 10 monclient: monmap: 2024-04-26T00:26:40.837 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:40.837 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:26:40.837 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:26:40.837 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:26:40.837 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:40.837 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:26:40.837 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:26:40.837 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:26:40.837 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:26:40.837 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:40.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145418f640 1 Processor -- start 2024-04-26T00:26:40.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145418f640 1 -- start start 2024-04-26T00:26:40.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145418f640 10 monclient: init 2024-04-26T00:26:40.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145418f640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:40.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145418f640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:26:40.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145418f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f144c0a0260 0x7f144c131940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:40.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145418f640 10 monclient(hunting): picked mon.noname-a con 0x7f144c0a0260 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:26:40.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145418f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f144c134b30 0x7f144c09c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:40.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145418f640 10 monclient(hunting): picked mon.noname-c con 0x7f144c134b30 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:26:40.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145418f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f144c135660 0x7f144c09ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:40.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145418f640 10 monclient(hunting): picked mon.noname-b con 0x7f144c135660 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:26:40.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145418f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:40.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145418f640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f144c09f4c0 con 0x7f144c0a0260 2024-04-26T00:26:40.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145418f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:40.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145418f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f144c09f0f0 con 0x7f144c135660 2024-04-26T00:26:40.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145418f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:40.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145418f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f144c140db0 con 0x7f144c134b30 2024-04-26T00:26:40.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145418f640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:40.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145418f640 10 monclient(hunting): authenticate will time out at 1382.155357s 2024-04-26T00:26:40.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145172a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f144c134b30 0x7f144c09c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:40.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f1451f2b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f144c0a0260 0x7f144c131940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:40.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145172a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f144c134b30 0x7f144c09c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:58632/0 (socket says 172.21.15.37:58632) 2024-04-26T00:26:40.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145172a640 1 -- 172.21.15.37:0/3480884997 learned_addr learned my addr 172.21.15.37:0/3480884997 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:26:40.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145172a640 10 monclient(hunting): get_auth_request con 0x7f144c134b30 auth_method 0 2024-04-26T00:26:40.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145172a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:40.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145172a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:40.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145172a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:40.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f1451f2b640 10 monclient(hunting): get_auth_request con 0x7f144c0a0260 auth_method 0 2024-04-26T00:26:40.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f1451f2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:40.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f1451f2b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:40.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f1451f2b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:40.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145172a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:40.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145172a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:40.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145172a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:40.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f1451f2b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:40.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f1451f2b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:40.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f1451f2b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:40.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145272c640 1 --2- 172.21.15.37:0/3480884997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f144c135660 0x7f144c09ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:40.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145172a640 10 monclient(hunting): handle_auth_done global_id 84107 payload 824 2024-04-26T00:26:40.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145172a640 1 -- 172.21.15.37:0/3480884997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f144c135660 msgr2=0x7f144c09ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:40.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145172a640 1 --2- 172.21.15.37:0/3480884997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f144c135660 0x7f144c09ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:40.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145172a640 1 -- 172.21.15.37:0/3480884997 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f144c0a0260 msgr2=0x7f144c131940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:40.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145172a640 1 --2- 172.21.15.37:0/3480884997 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f144c0a0260 0x7f144c131940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:40.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145172a640 10 monclient: _finish_hunting 0 2024-04-26T00:26:40.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145172a640 1 monclient: found mon.noname-c 2024-04-26T00:26:40.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145172a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:40.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145172a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:26:40.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.844+0000 7f145172a640 1 -- 172.21.15.37:0/3480884997 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f144c134870 con 0x7f144c134b30 2024-04-26T00:26:40.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.848+0000 7f145172a640 1 -- 172.21.15.37:0/3480884997 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f144c134b30 msgr2=0x7f144c09c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:40.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.848+0000 7f145172a640 1 --2- 172.21.15.37:0/3480884997 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f144c134b30 0x7f144c09c4d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f144000c7c0 tx=0x7f144000cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:40.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.848+0000 7f144a7fc640 1 -- 172.21.15.37:0/3480884997 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1440016020 con 0x7f144c134b30 2024-04-26T00:26:40.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.848+0000 7f144a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:40.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.848+0000 7f144a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:26:40.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.848+0000 7f144a7fc640 10 monclient: dump: 2024-04-26T00:26:40.841 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:40.841 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:40.841 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:40.841 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:40.841 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:40.841 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:40.841 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:40.841 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:40.841 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:40.841 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:40.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.848+0000 7f144a7fc640 10 monclient: _finish_auth 0 2024-04-26T00:26:40.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.848+0000 7f144a7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:26:40.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.848+0000 7f144a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:40.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.848+0000 7f144a7fc640 1 -- 172.21.15.37:0/3480884997 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f144000ac60 con 0x7f144c134b30 2024-04-26T00:26:40.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.848+0000 7f144a7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:40.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.848+0000 7f144a7fc640 1 -- 172.21.15.37:0/3480884997 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1440012660 con 0x7f144c134b30 2024-04-26T00:26:40.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.848+0000 7f145418f640 5 monclient: authenticate success, global_id 84107 2024-04-26T00:26:40.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.848+0000 7f145418f640 10 monclient: _renew_subs 2024-04-26T00:26:40.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.848+0000 7f145418f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:26:40.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.848+0000 7f145418f640 1 -- 172.21.15.37:0/3480884997 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f144c09e0a0 con 0x7f144c134b30 2024-04-26T00:26:40.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.848+0000 7f144a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:40.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.848+0000 7f144a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:26:40.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.848+0000 7f144a7fc640 10 monclient: dump: 2024-04-26T00:26:40.842 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:40.843 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:40.843 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:40.843 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:40.843 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:40.843 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:40.843 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:40.843 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:40.843 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:40.843 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:40.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.848+0000 7f144b7fe640 4 set_mon_vals no callback set 2024-04-26T00:26:40.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.848+0000 7f145418f640 10 monclient: _renew_subs 2024-04-26T00:26:40.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.848+0000 7f145418f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:26:40.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.848+0000 7f145418f640 1 -- 172.21.15.37:0/3480884997 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f144c11ca40 con 0x7f144c134b30 2024-04-26T00:26:40.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.848+0000 7f145418f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:40.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.848+0000 7f145418f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:40.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.848+0000 7f145418f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:26:40.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.848+0000 7f145418f640 1 -- 172.21.15.37:0/3480884997 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f144c066f40 con 0x7f144c134b30 2024-04-26T00:26:40.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.848+0000 7f144a7fc640 1 -- 172.21.15.37:0/3480884997 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f1440019020 con 0x7f144c134b30 2024-04-26T00:26:40.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.848+0000 7f144a7fc640 1 --2- 172.21.15.37:0/3480884997 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1438035e80 0x7f1438038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:40.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.848+0000 7f144a7fc640 1 -- 172.21.15.37:0/3480884997 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(261..261 src has 177..261) ==== 3963+0+0 (secure 0 0 0) 0x7f1440048b50 con 0x7f144c134b30 2024-04-26T00:26:40.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.852+0000 7f1451f2b640 1 --2- 172.21.15.37:0/3480884997 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1438035e80 0x7f1438038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:40.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.852+0000 7f1451f2b640 10 monclient: get_auth_request con 0x7f1438035e80 auth_method 0 2024-04-26T00:26:40.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.852+0000 7f1451f2b640 1 --2- 172.21.15.37:0/3480884997 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1438035e80 0x7f1438038340 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f14440ab000 tx=0x7f14440a6070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:40.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.856+0000 7f144a7fc640 1 -- 172.21.15.37:0/3480884997 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f1440020030 con 0x7f144c134b30 2024-04-26T00:26:40.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.856+0000 7f144a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:40.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:40.856+0000 7f144a7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:26:41.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:41.076+0000 7f145418f640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-26T00:26:41.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:41.076+0000 7f145418f640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-26T00:26:41.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:41.076+0000 7f145418f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:26:41.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:41.076+0000 7f145418f640 1 -- 172.21.15.37:0/3480884997 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f144c11be90 con 0x7f144c134b30 2024-04-26T00:26:41.067 INFO:tasks.rados.rados.0.smithi045.stdout:1101: done (2 left) 2024-04-26T00:26:41.067 INFO:tasks.rados.rados.0.smithi045.stdout:1102: snap_create 2024-04-26T00:26:41.067 INFO:tasks.rados.rados.0.smithi045.stdout:1100: expect (ObjNum 386 snap 117 seq_num 386) 2024-04-26T00:26:41.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:41.092+0000 7f144a7fc640 1 -- 172.21.15.37:0/3480884997 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f1440048e20 con 0x7f144c134b30 2024-04-26T00:26:41.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:41.092+0000 7f144a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-26T00:26:41.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:41.092+0000 7f144a7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:26:41.077 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.091+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.078 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.091+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=15,config=2,health=12,kv=10,logm=61} crc {auth=551674344,config=692619927,health=1068221574,kv=1034996044,logm=2184112393}) 2024-04-26T00:26:41.078 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.091+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=12,kv=10,logm=62} crc {auth=461226654,config=692619927,health=1068221574,kv=1034996044,logm=944685960}) 2024-04-26T00:26:41.078 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.091+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.078 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.091+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=15,config=2,health=12,kv=10,logm=61} crc {auth=551674344,config=692619927,health=1068221574,kv=1034996044,logm=2184112393}) 2024-04-26T00:26:41.078 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.091+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=14,config=2,health=12,kv=10,logm=62} crc {auth=45008549,config=692619927,health=1068221574,kv=2259169355,logm=1829375140}) 2024-04-26T00:26:41.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:41.096+0000 7f145418f640 1 -- 172.21.15.37:0/3480884997 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1438035e80 msgr2=0x7f1438038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:41.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:41.096+0000 7f145418f640 1 --2- 172.21.15.37:0/3480884997 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1438035e80 0x7f1438038340 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f14440ab000 tx=0x7f14440a6070 comp rx=0 tx=0).stop 2024-04-26T00:26:41.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:41.096+0000 7f145418f640 10 monclient: shutdown 2024-04-26T00:26:41.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:41.096+0000 7f145418f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:41.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:41.096+0000 7f145418f640 1 -- 172.21.15.37:0/3480884997 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f144c134b30 msgr2=0x7f144c09c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:41.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:41.096+0000 7f145418f640 1 --2- 172.21.15.37:0/3480884997 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f144c134b30 0x7f144c09c4d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f144000c7c0 tx=0x7f144000cc80 comp rx=0 tx=0).stop 2024-04-26T00:26:41.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:41.096+0000 7f145418f640 1 -- 172.21.15.37:0/3480884997 shutdown_connections 2024-04-26T00:26:41.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:41.096+0000 7f145418f640 1 --2- 172.21.15.37:0/3480884997 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1438035e80 0x7f1438038340 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:41.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:41.096+0000 7f145418f640 1 --2- 172.21.15.37:0/3480884997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f144c135660 0x7f144c09ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:41.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:41.096+0000 7f145418f640 1 --2- 172.21.15.37:0/3480884997 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f144c134b30 0x7f144c09c4d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:41.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:41.096+0000 7f145418f640 1 --2- 172.21.15.37:0/3480884997 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f144c0a0260 0x7f144c131940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:41.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:41.096+0000 7f145418f640 1 -- 172.21.15.37:0/3480884997 >> 172.21.15.37:0/3480884997 conn(0x7f144c0a1ad0 msgr2=0x7f144c0a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:41.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:41.096+0000 7f145418f640 1 -- 172.21.15.37:0/3480884997 shutdown_connections 2024-04-26T00:26:41.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:41.096+0000 7f145418f640 1 -- 172.21.15.37:0/3480884997 wait complete. 2024-04-26T00:26:41.084 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.095+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.084 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.095+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2315834564}) 2024-04-26T00:26:41.084 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.095+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4081431799}) 2024-04-26T00:26:41.084 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.095+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.084 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.095+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2315834564}) 2024-04-26T00:26:41.084 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.095+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1314220739}) 2024-04-26T00:26:41.087 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.099+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.087 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.099+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1476315517}) 2024-04-26T00:26:41.087 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.099+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2640368717}) 2024-04-26T00:26:41.087 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.099+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.087 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.099+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1476315517}) 2024-04-26T00:26:41.087 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.099+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=673227768}) 2024-04-26T00:26:41.087 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.099+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.087 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.099+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4213060385}) 2024-04-26T00:26:41.087 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.099+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3541094688}) 2024-04-26T00:26:41.087 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.099+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.087 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.099+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4213060385}) 2024-04-26T00:26:41.087 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.099+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2162409787}) 2024-04-26T00:26:41.088 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.103+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.088 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.103+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1484184543}) 2024-04-26T00:26:41.088 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.103+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3013535901}) 2024-04-26T00:26:41.088 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.103+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.088 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.103+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1484184543}) 2024-04-26T00:26:41.088 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.103+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3023384233}) 2024-04-26T00:26:41.090 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.103+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.090 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.103+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=835219834}) 2024-04-26T00:26:41.090 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.103+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2010648025}) 2024-04-26T00:26:41.090 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.103+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.090 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.103+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=835219834}) 2024-04-26T00:26:41.090 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.103+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=634772991}) 2024-04-26T00:26:41.091 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.107+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.091 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.107+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3098796618}) 2024-04-26T00:26:41.091 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.107+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3480113438}) 2024-04-26T00:26:41.092 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.107+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.092 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.107+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3098796618}) 2024-04-26T00:26:41.092 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.107+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3567296596}) 2024-04-26T00:26:41.093 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.107+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.093 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.107+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3327343429}) 2024-04-26T00:26:41.093 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.107+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2177115006}) 2024-04-26T00:26:41.094 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.107+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.094 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.107+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3327343429}) 2024-04-26T00:26:41.094 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.107+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1175444027}) 2024-04-26T00:26:41.095 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.111+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.095 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.111+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3077671572}) 2024-04-26T00:26:41.095 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.111+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1495381664}) 2024-04-26T00:26:41.095 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.111+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.096 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.111+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3077671572}) 2024-04-26T00:26:41.096 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.111+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2993752756}) 2024-04-26T00:26:41.097 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.111+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.097 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.111+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3995725948}) 2024-04-26T00:26:41.097 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.111+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3300988349}) 2024-04-26T00:26:41.097 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.111+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.097 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.111+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3995725948}) 2024-04-26T00:26:41.097 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.111+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4169561649}) 2024-04-26T00:26:41.099 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.115+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.099 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.115+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2770705191}) 2024-04-26T00:26:41.099 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.115+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1595180043}) 2024-04-26T00:26:41.099 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.115+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.099 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.115+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2770705191}) 2024-04-26T00:26:41.099 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.115+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1268516638}) 2024-04-26T00:26:41.100 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-26T00:26:41.101 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.115+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.101 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.115+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=479521422}) 2024-04-26T00:26:41.101 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.115+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=712121851}) 2024-04-26T00:26:41.101 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.115+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.101 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.115+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=479521422}) 2024-04-26T00:26:41.101 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.115+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2968752059}) 2024-04-26T00:26:41.103 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.119+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.103 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.119+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3819552439}) 2024-04-26T00:26:41.103 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.119+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4272121611}) 2024-04-26T00:26:41.103 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.119+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.103 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.119+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3819552439}) 2024-04-26T00:26:41.103 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.119+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3321430935}) 2024-04-26T00:26:41.105 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.119+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.105 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.119+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=669253838}) 2024-04-26T00:26:41.105 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.119+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4217019370}) 2024-04-26T00:26:41.105 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.119+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.105 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.119+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=669253838}) 2024-04-26T00:26:41.105 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.119+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=944654780}) 2024-04-26T00:26:41.106 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.123+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.107 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.123+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1193248651}) 2024-04-26T00:26:41.107 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.123+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3303320219}) 2024-04-26T00:26:41.107 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.123+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.107 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.123+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1193248651}) 2024-04-26T00:26:41.107 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.123+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3635462028}) 2024-04-26T00:26:41.108 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.123+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.108 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.123+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4210071576}) 2024-04-26T00:26:41.109 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.123+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2441575257}) 2024-04-26T00:26:41.109 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.123+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.109 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.123+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4210071576}) 2024-04-26T00:26:41.109 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.123+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4199653057}) 2024-04-26T00:26:41.110 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.123+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.110 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.123+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3387502195}) 2024-04-26T00:26:41.111 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.123+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2168028489}) 2024-04-26T00:26:41.111 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.123+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.111 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.127+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3387502195}) 2024-04-26T00:26:41.111 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.127+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=222709758}) 2024-04-26T00:26:41.112 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.127+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.112 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.127+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1869911296}) 2024-04-26T00:26:41.112 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.127+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2256029803}) 2024-04-26T00:26:41.113 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.127+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.113 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.127+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1869911296}) 2024-04-26T00:26:41.113 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.127+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3479551149}) 2024-04-26T00:26:41.115 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.131+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.115 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.131+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=20,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=35} crc {logm=642749922,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=785475189,mon_config_key=4061063117,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=3440010605,osd_snap=3114378435}) 2024-04-26T00:26:41.115 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.131+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=19,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=36} crc {logm=341169522,mdsmap=2840952586,mgr=1022427549,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=3878743617,mon_config_key=4061063117,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=3440010605,osd_snap=3890873887}) 2024-04-26T00:26:41.115 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.131+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.115 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.131+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=20,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=35} crc {logm=642749922,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=785475189,mon_config_key=4061063117,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=3440010605,osd_snap=3114378435}) 2024-04-26T00:26:41.115 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.131+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=19,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=36} crc {logm=2909287744,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=2453740193,mon_config_key=1720404874,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=3440010605,osd_snap=2011372645}) 2024-04-26T00:26:41.117 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.131+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.117 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.131+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=7,osdmap=93} crc {osd_snap=25439448,osdmap=1374808622}) 2024-04-26T00:26:41.117 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.131+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=6,osdmap=94} crc {osd_snap=2744362595,osdmap=1976645637}) 2024-04-26T00:26:41.117 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.131+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.117 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.131+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=7,osdmap=93} crc {osd_snap=25439448,osdmap=1374808622}) 2024-04-26T00:26:41.117 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.131+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=7,osdmap=93} crc {osd_snap=25439448,osdmap=4079918012}) 2024-04-26T00:26:41.126 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.139+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.126 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.139+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=48} crc {osdmap=1097506955}) 2024-04-26T00:26:41.126 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.139+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=50} crc {osdmap=3291271538}) 2024-04-26T00:26:41.126 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.139+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:26:41.127 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.139+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=48} crc {osdmap=1097506955}) 2024-04-26T00:26:41.127 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:26:41.139+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=49} crc {osdmap=3352583784}) 2024-04-26T00:26:41.320 INFO:tasks.rados.rados.0.smithi045.stdout:1099: finishing copy_from racing read to smithi04526579-43 2024-04-26T00:26:41.320 INFO:tasks.rados.rados.0.smithi045.stdout:1099: finishing copy_from to smithi04526579-43 2024-04-26T00:26:41.320 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 253 (ObjNum 329 snap 96 seq_num 329) dirty exists 2024-04-26T00:26:41.891 INFO:tasks.rados.rados.0.smithi045.stdout:1099: done (2 left) 2024-04-26T00:26:41.891 INFO:tasks.rados.rados.0.smithi045.stdout:1100: done (1 left) 2024-04-26T00:26:41.891 INFO:tasks.rados.rados.0.smithi045.stdout:1102: done (0 left) 2024-04-26T00:26:41.891 INFO:tasks.rados.rados.0.smithi045.stdout:1103: write (excl) oid 17 current snap is 129 2024-04-26T00:26:41.891 INFO:tasks.rados.rados.0.smithi045.stdout:1103: seq_num 414 ranges {558098=636148,1826604=665129} 2024-04-26T00:26:41.910 INFO:tasks.rados.rados.0.smithi045.stdout:1103: writing smithi04526579-17 from 558098 to 1194246 tid 1 2024-04-26T00:26:41.932 INFO:tasks.rados.rados.0.smithi045.stdout:1103: writing smithi04526579-17 from 1826604 to 2491733 tid 2 2024-04-26T00:26:41.932 INFO:tasks.rados.rados.0.smithi045.stdout:1104: snap_create 2024-04-26T00:26:41.932 INFO:tasks.rados.rados.0.smithi045.stdout:1103: finishing write tid 1 to smithi04526579-17 2024-04-26T00:26:41.937 INFO:tasks.rados.rados.0.smithi045.stdout:1103: finishing write tid 2 to smithi04526579-17 2024-04-26T00:26:41.937 INFO:tasks.rados.rados.0.smithi045.stdout:1103: finishing write tid 3 to smithi04526579-17 2024-04-26T00:26:41.937 INFO:tasks.rados.rados.0.smithi045.stdout:1103: finishing write tid 4 to smithi04526579-17 2024-04-26T00:26:41.938 INFO:tasks.rados.rados.0.smithi045.stdout:1103: oid 17 updating version 0 to 515 2024-04-26T00:26:41.938 INFO:tasks.rados.rados.0.smithi045.stdout:1103: oid 17 version 515 is already newer than 513 2024-04-26T00:26:41.938 INFO:tasks.rados.rados.0.smithi045.stdout:1103: oid 17 version 515 is already newer than 514 2024-04-26T00:26:41.938 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 515 (ObjNum 414 snap 129 seq_num 414) dirty exists 2024-04-26T00:26:41.938 INFO:tasks.rados.rados.0.smithi045.stdout:1103: left oid 17 (ObjNum 414 snap 129 seq_num 414) 2024-04-26T00:26:42.101 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:26:42.101 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:26:42.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.248+0000 7f809b8e1640 10 monclient: get_monmap_and_config 2024-04-26T00:26:42.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.248+0000 7f809b8e1640 10 monclient: build_initial_monmap 2024-04-26T00:26:42.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.248+0000 7f809b8e1640 10 monclient: monmap: 2024-04-26T00:26:42.234 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:42.234 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:42.234 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:26:42.253247+0000 2024-04-26T00:26:42.234 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:26:42.253247+0000 2024-04-26T00:26:42.234 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:42.234 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:42.234 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:26:42.234 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:26:42.234 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:26:42.234 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:26:42.234 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:26:42.235 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:42.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.248+0000 7f809b8e1640 1 Processor -- start 2024-04-26T00:26:42.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.248+0000 7f809b8e1640 1 -- start start 2024-04-26T00:26:42.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.248+0000 7f809b8e1640 10 monclient: init 2024-04-26T00:26:42.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.248+0000 7f809b8e1640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:42.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.248+0000 7f809b8e1640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-04-26T00:26:42.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.248+0000 7f809b8e1640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8094135090 0x7f8094135470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:42.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.248+0000 7f809b8e1640 10 monclient(hunting): picked mon.noname-b con 0x7f8094135090 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:26:42.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.248+0000 7f809b8e1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f809406ead0 addr v1:172.21.15.37:6789/0 2024-04-26T00:26:42.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.248+0000 7f809b8e1640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f80940a1bc0 0x7f809413ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:42.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.248+0000 7f809b8e1640 10 monclient(hunting): picked mon.noname-a con 0x7f80940a1bc0 addr v2:172.21.15.37:3300/0 2024-04-26T00:26:42.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.248+0000 7f809b8e1640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80941359b0 con 0x7f809406ead0 2024-04-26T00:26:42.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.248+0000 7f809b8e1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:42.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.248+0000 7f809b8e1640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f809406e610 con 0x7f80940a1bc0 2024-04-26T00:26:42.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.248+0000 7f809b8e1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:42.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.248+0000 7f809b8e1640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8094071b00 con 0x7f8094135090 2024-04-26T00:26:42.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.248+0000 7f809b8e1640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:42.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.248+0000 7f809b8e1640 10 monclient(hunting): authenticate will time out at 1383.559916s 2024-04-26T00:26:42.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.252+0000 7f8098e7c640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f809406ead0 0x7f80940a14b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:33298/0 (socket says 172.21.15.37:33298) 2024-04-26T00:26:42.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.252+0000 7f8098e7c640 1 -- 172.21.15.37:0/4206642156 learned_addr learned my addr 172.21.15.37:0/4206642156 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:26:42.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.252+0000 7f809967d640 1 --2- 172.21.15.37:0/4206642156 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8094135090 0x7f8094135470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:42.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.252+0000 7f8099e7e640 1 --2- 172.21.15.37:0/4206642156 >> v2:172.21.15.37:3300/0 conn(0x7f80940a1bc0 0x7f809413ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:42.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.252+0000 7f8099e7e640 10 monclient(hunting): get_auth_request con 0x7f80940a1bc0 auth_method 0 2024-04-26T00:26:42.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.252+0000 7f8099e7e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:42.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.252+0000 7f8099e7e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:42.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.252+0000 7f8099e7e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:42.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.252+0000 7f809967d640 10 monclient(hunting): get_auth_request con 0x7f8094135090 auth_method 0 2024-04-26T00:26:42.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.252+0000 7f809967d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:42.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.252+0000 7f809967d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:42.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.252+0000 7f809967d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:42.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.252+0000 7f8099e7e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:42.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.252+0000 7f8099e7e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:42.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.252+0000 7f8099e7e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:42.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.252+0000 7f809967d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:42.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.252+0000 7f809967d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:42.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.252+0000 7f809967d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:42.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.252+0000 7f8099e7e640 10 monclient(hunting): handle_auth_done global_id 54396 payload 274 2024-04-26T00:26:42.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.252+0000 7f8099e7e640 1 -- 172.21.15.37:0/4206642156 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8094135090 msgr2=0x7f8094135470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:26:42.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.252+0000 7f8099e7e640 1 --2- 172.21.15.37:0/4206642156 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8094135090 0x7f8094135470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:42.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.252+0000 7f8099e7e640 1 -- 172.21.15.37:0/4206642156 >> v1:172.21.15.37:6789/0 conn(0x7f809406ead0 legacy=0x7f80940a14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:26:42.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.252+0000 7f8099e7e640 10 monclient: _finish_hunting 0 2024-04-26T00:26:42.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.252+0000 7f8099e7e640 1 monclient: found mon.noname-a 2024-04-26T00:26:42.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.252+0000 7f8099e7e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:42.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.252+0000 7f8099e7e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:26:42.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.252+0000 7f8099e7e640 1 -- 172.21.15.37:0/4206642156 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8094071de0 con 0x7f80940a1bc0 2024-04-26T00:26:42.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.256+0000 7f8099e7e640 1 -- 172.21.15.37:0/4206642156 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80940a1bc0 msgr2=0x7f809413ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:42.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.256+0000 7f8099e7e640 1 --2- 172.21.15.37:0/4206642156 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80940a1bc0 0x7f809413ceb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8094072d60 tx=0x7f808c01c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=867f9a65ace6e758 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:42.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.256+0000 7f808bfff640 1 -- 172.21.15.37:0/4206642156 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f808c0b2070 con 0x7f80940a1bc0 2024-04-26T00:26:42.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.256+0000 7f808bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:42.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.256+0000 7f808bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:26:42.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.256+0000 7f808bfff640 10 monclient: dump: 2024-04-26T00:26:42.243 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:42.243 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:42.243 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.243 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.243 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:42.243 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:42.243 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:42.243 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:42.243 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:42.243 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:42.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.256+0000 7f808bfff640 10 monclient: _finish_auth 0 2024-04-26T00:26:42.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.256+0000 7f808bfff640 10 monclient: _check_auth_tickets 2024-04-26T00:26:42.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.256+0000 7f808bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:42.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.256+0000 7f808bfff640 1 -- 172.21.15.37:0/4206642156 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f808c0a5740 con 0x7f80940a1bc0 2024-04-26T00:26:42.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.256+0000 7f808bfff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:42.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.256+0000 7f808bfff640 1 -- 172.21.15.37:0/4206642156 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f808c0a5b10 con 0x7f80940a1bc0 2024-04-26T00:26:42.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.256+0000 7f808bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:42.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.256+0000 7f808bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:26:42.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.256+0000 7f808bfff640 10 monclient: dump: 2024-04-26T00:26:42.244 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:42.244 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:42.244 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.244 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.244 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:42.244 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:42.244 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:42.245 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:42.245 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:42.245 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:42.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.256+0000 7f809b8e1640 5 monclient: authenticate success, global_id 54396 2024-04-26T00:26:42.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.256+0000 7f809b8e1640 10 monclient: get_monmap_and_config success 2024-04-26T00:26:42.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.256+0000 7f809b8e1640 4 set_mon_vals no callback set 2024-04-26T00:26:42.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.256+0000 7f809b8e1640 10 monclient: shutdown 2024-04-26T00:26:42.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.256+0000 7f809b8e1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:42.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.256+0000 7f809b8e1640 1 -- 172.21.15.37:0/4206642156 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80940a1bc0 msgr2=0x7f809413ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:42.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.256+0000 7f809b8e1640 1 --2- 172.21.15.37:0/4206642156 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80940a1bc0 0x7f809413ceb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8094072d60 tx=0x7f808c01c7b0 comp rx=0 tx=0).stop 2024-04-26T00:26:42.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.256+0000 7f809b8e1640 1 -- 172.21.15.37:0/4206642156 shutdown_connections 2024-04-26T00:26:42.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.256+0000 7f809b8e1640 1 --2- 172.21.15.37:0/4206642156 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80940a1bc0 0x7f809413ceb0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:42.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.256+0000 7f809b8e1640 1 --2- 172.21.15.37:0/4206642156 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8094135090 0x7f8094135470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:42.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.256+0000 7f809b8e1640 1 -- 172.21.15.37:0/4206642156 >> 172.21.15.37:0/4206642156 conn(0x7f80940a6250 msgr2=0x7f80940a2610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:42.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.256+0000 7f809b8e1640 1 -- 172.21.15.37:0/4206642156 shutdown_connections 2024-04-26T00:26:42.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.256+0000 7f809b8e1640 1 -- 172.21.15.37:0/4206642156 wait complete. 2024-04-26T00:26:42.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.256+0000 7f809b8e1640 10 monclient: build_initial_monmap 2024-04-26T00:26:42.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.256+0000 7f809b8e1640 10 monclient: monmap: 2024-04-26T00:26:42.251 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:42.251 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:26:42.251 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:26:42.251 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:26:42.251 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:42.251 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:26:42.251 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:26:42.251 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:26:42.252 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:26:42.252 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:42.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.256+0000 7f809b8e1640 1 Processor -- start 2024-04-26T00:26:42.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.256+0000 7f809b8e1640 1 -- start start 2024-04-26T00:26:42.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.256+0000 7f809b8e1640 10 monclient: init 2024-04-26T00:26:42.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.256+0000 7f809b8e1640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:42.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f809b8e1640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:26:42.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f809b8e1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f809406ead0 0x7f80941cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:42.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f809b8e1640 10 monclient(hunting): picked mon.noname-b con 0x7f809406ead0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:26:42.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f809b8e1640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80940a1bc0 0x7f809413c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:42.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f809b8e1640 10 monclient(hunting): picked mon.noname-a con 0x7f80940a1bc0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:26:42.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f809b8e1640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8094135090 0x7f809413c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:42.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f809b8e1640 10 monclient(hunting): picked mon.noname-c con 0x7f8094135090 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:26:42.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f809b8e1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:42.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f809b8e1640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f809413fd90 con 0x7f80940a1bc0 2024-04-26T00:26:42.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f809b8e1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:42.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f809b8e1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f809413fc10 con 0x7f809406ead0 2024-04-26T00:26:42.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f809b8e1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:42.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f809b8e1640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f809413ff10 con 0x7f8094135090 2024-04-26T00:26:42.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f809b8e1640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:42.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f809b8e1640 10 monclient(hunting): authenticate will time out at 1383.568150s 2024-04-26T00:26:42.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8098e7c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80940a1bc0 0x7f809413c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:42.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8099e7e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8094135090 0x7f809413c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:42.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8098e7c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80940a1bc0 0x7f809413c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50056/0 (socket says 172.21.15.37:50056) 2024-04-26T00:26:42.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8098e7c640 1 -- 172.21.15.37:0/2860299372 learned_addr learned my addr 172.21.15.37:0/2860299372 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:26:42.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8098e7c640 10 monclient(hunting): get_auth_request con 0x7f80940a1bc0 auth_method 0 2024-04-26T00:26:42.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8098e7c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:42.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8098e7c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:42.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8098e7c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:42.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8099e7e640 10 monclient(hunting): get_auth_request con 0x7f8094135090 auth_method 0 2024-04-26T00:26:42.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8099e7e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:42.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8099e7e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:42.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8099e7e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:42.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8098e7c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:42.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8098e7c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:42.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8098e7c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:42.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8099e7e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:42.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8099e7e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:42.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8099e7e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:42.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f809967d640 1 --2- 172.21.15.37:0/2860299372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f809406ead0 0x7f80941cd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:42.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f809967d640 10 monclient(hunting): get_auth_request con 0x7f809406ead0 auth_method 0 2024-04-26T00:26:42.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f809967d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:42.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f809967d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:42.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f809967d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:42.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8098e7c640 10 monclient(hunting): handle_auth_done global_id 54399 payload 824 2024-04-26T00:26:42.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8098e7c640 1 -- 172.21.15.37:0/2860299372 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8094135090 msgr2=0x7f809413c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:42.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8098e7c640 1 --2- 172.21.15.37:0/2860299372 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8094135090 0x7f809413c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:42.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8098e7c640 1 -- 172.21.15.37:0/2860299372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f809406ead0 msgr2=0x7f80941cd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:42.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8098e7c640 1 --2- 172.21.15.37:0/2860299372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f809406ead0 0x7f80941cd2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:42.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8098e7c640 10 monclient: _finish_hunting 0 2024-04-26T00:26:42.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8098e7c640 1 monclient: found mon.noname-a 2024-04-26T00:26:42.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8098e7c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:42.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8098e7c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:26:42.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8098e7c640 1 -- 172.21.15.37:0/2860299372 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f809406ef90 con 0x7f80940a1bc0 2024-04-26T00:26:42.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8098e7c640 1 -- 172.21.15.37:0/2860299372 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80940a1bc0 msgr2=0x7f809413c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:42.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8098e7c640 1 --2- 172.21.15.37:0/2860299372 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80940a1bc0 0x7f809413c150 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f809000b460 tx=0x7f809000b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:42.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8089ffb640 1 -- 172.21.15.37:0/2860299372 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8090015020 con 0x7f80940a1bc0 2024-04-26T00:26:42.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8089ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:42.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8089ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:26:42.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8089ffb640 10 monclient: dump: 2024-04-26T00:26:42.257 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:42.257 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:42.257 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.257 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.257 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:42.257 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:42.257 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:42.257 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:42.257 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:42.257 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:42.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8089ffb640 10 monclient: _finish_auth 0 2024-04-26T00:26:42.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8089ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:26:42.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8089ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:42.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8089ffb640 1 -- 172.21.15.37:0/2860299372 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8090011ac0 con 0x7f80940a1bc0 2024-04-26T00:26:42.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8089ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:42.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8089ffb640 1 -- 172.21.15.37:0/2860299372 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80900103e0 con 0x7f80940a1bc0 2024-04-26T00:26:42.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8089ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:42.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8089ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:26:42.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f8089ffb640 10 monclient: dump: 2024-04-26T00:26:42.258 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:42.258 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:42.258 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.259 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.259 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:42.259 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:42.259 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:42.259 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:42.259 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:42.259 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:42.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f808b7fe640 4 set_mon_vals no callback set 2024-04-26T00:26:42.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f809b8e1640 5 monclient: authenticate success, global_id 54399 2024-04-26T00:26:42.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f809b8e1640 10 monclient: _renew_subs 2024-04-26T00:26:42.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f809b8e1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:26:42.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f809b8e1640 1 -- 172.21.15.37:0/2860299372 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8094138250 con 0x7f80940a1bc0 2024-04-26T00:26:42.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f809b8e1640 10 monclient: _renew_subs 2024-04-26T00:26:42.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f809b8e1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:26:42.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.260+0000 7f809b8e1640 1 -- 172.21.15.37:0/2860299372 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8094137fd0 con 0x7f80940a1bc0 2024-04-26T00:26:42.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.264+0000 7f8089ffb640 1 -- 172.21.15.37:0/2860299372 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f8090018020 con 0x7f80940a1bc0 2024-04-26T00:26:42.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.264+0000 7f809b8e1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:42.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.264+0000 7f809b8e1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:42.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.264+0000 7f809b8e1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:26:42.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.264+0000 7f809b8e1640 1 -- 172.21.15.37:0/2860299372 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80941ce000 con 0x7f80940a1bc0 2024-04-26T00:26:42.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.264+0000 7f8089ffb640 1 --2- 172.21.15.37:0/2860299372 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8078035e80 0x7f8078038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:42.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.264+0000 7f8089ffb640 1 -- 172.21.15.37:0/2860299372 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(263..263 src has 177..263) ==== 3979+0+0 (secure 0 0 0) 0x7f8090048c80 con 0x7f80940a1bc0 2024-04-26T00:26:42.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.264+0000 7f809967d640 1 --2- 172.21.15.37:0/2860299372 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8078035e80 0x7f8078038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:42.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.264+0000 7f809967d640 10 monclient: get_auth_request con 0x7f8078035e80 auth_method 0 2024-04-26T00:26:42.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.264+0000 7f809967d640 1 --2- 172.21.15.37:0/2860299372 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8078035e80 0x7f8078038340 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8084009a70 tx=0x7f8084008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:42.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.268+0000 7f8089ffb640 1 -- 172.21.15.37:0/2860299372 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f80941ce000 con 0x7f80940a1bc0 2024-04-26T00:26:42.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.268+0000 7f8089ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:42.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.268+0000 7f8089ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:26:42.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.492+0000 7f809b8e1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:26:42.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.492+0000 7f809b8e1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:26:42.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.492+0000 7f809b8e1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:26:42.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.492+0000 7f809b8e1640 1 -- 172.21.15.37:0/2860299372 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8094066f40 con 0x7f80940a1bc0 2024-04-26T00:26:42.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.492+0000 7f8089ffb640 1 -- 172.21.15.37:0/2860299372 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f8090039020 con 0x7f80940a1bc0 2024-04-26T00:26:42.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.492+0000 7f8089ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:26:42.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.492+0000 7f8089ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:26:42.479 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":52,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:26:42.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.496+0000 7f809b8e1640 1 -- 172.21.15.37:0/2860299372 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8078035e80 msgr2=0x7f8078038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:42.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.496+0000 7f809b8e1640 1 --2- 172.21.15.37:0/2860299372 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8078035e80 0x7f8078038340 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8084009a70 tx=0x7f8084008040 comp rx=0 tx=0).stop 2024-04-26T00:26:42.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.496+0000 7f809b8e1640 10 monclient: shutdown 2024-04-26T00:26:42.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.496+0000 7f809b8e1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:42.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.496+0000 7f809b8e1640 1 -- 172.21.15.37:0/2860299372 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80940a1bc0 msgr2=0x7f809413c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:42.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.496+0000 7f809b8e1640 1 --2- 172.21.15.37:0/2860299372 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80940a1bc0 0x7f809413c150 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f809000b460 tx=0x7f809000b920 comp rx=0 tx=0).stop 2024-04-26T00:26:42.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.496+0000 7f809b8e1640 1 -- 172.21.15.37:0/2860299372 shutdown_connections 2024-04-26T00:26:42.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.496+0000 7f809b8e1640 1 --2- 172.21.15.37:0/2860299372 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8078035e80 0x7f8078038340 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:42.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.496+0000 7f809b8e1640 1 --2- 172.21.15.37:0/2860299372 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8094135090 0x7f809413c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:42.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.496+0000 7f809b8e1640 1 --2- 172.21.15.37:0/2860299372 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80940a1bc0 0x7f809413c150 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:42.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.496+0000 7f809b8e1640 1 --2- 172.21.15.37:0/2860299372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f809406ead0 0x7f80941cd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:42.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.496+0000 7f809b8e1640 1 -- 172.21.15.37:0/2860299372 >> 172.21.15.37:0/2860299372 conn(0x7f80940a6250 msgr2=0x7f8094131690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:42.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.496+0000 7f809b8e1640 1 -- 172.21.15.37:0/2860299372 shutdown_connections 2024-04-26T00:26:42.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.496+0000 7f809b8e1640 1 -- 172.21.15.37:0/2860299372 wait complete. 2024-04-26T00:26:42.499 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:26:42.499 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-26T00:26:42.499 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:26:42.499 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:26:42.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb863da640 10 monclient: get_monmap_and_config 2024-04-26T00:26:42.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb863da640 10 monclient: build_initial_monmap 2024-04-26T00:26:42.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb863da640 10 monclient: monmap: 2024-04-26T00:26:42.625 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:42.625 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:42.625 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:26:42.644489+0000 2024-04-26T00:26:42.625 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:26:42.644489+0000 2024-04-26T00:26:42.625 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:42.625 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:42.625 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:26:42.626 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:26:42.626 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:26:42.626 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:26:42.626 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:26:42.626 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:42.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb863da640 1 Processor -- start 2024-04-26T00:26:42.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb863da640 1 -- start start 2024-04-26T00:26:42.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb863da640 10 monclient: init 2024-04-26T00:26:42.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb863da640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:42.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb863da640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-04-26T00:26:42.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb863da640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbb800a0300 0x7fbb8009fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:42.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb863da640 10 monclient(hunting): picked mon.noname-c con 0x7fbb800a0300 addr v2:172.21.15.45:3300/0 2024-04-26T00:26:42.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb863da640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fbb800a06e0 0x7fbb80134ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:42.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb863da640 10 monclient(hunting): picked mon.noname-a con 0x7fbb800a06e0 addr v2:172.21.15.37:3300/0 2024-04-26T00:26:42.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb863da640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb801350e0 0x7fbb80135550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:42.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb863da640 10 monclient(hunting): picked mon.noname-b con 0x7fbb801350e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:26:42.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb863da640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:42.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb863da640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fbb800a13f0 con 0x7fbb800a06e0 2024-04-26T00:26:42.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb863da640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:42.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb863da640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fbb8009c190 con 0x7fbb800a0300 2024-04-26T00:26:42.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb863da640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:42.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb863da640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fbb800a1110 con 0x7fbb801350e0 2024-04-26T00:26:42.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb863da640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:42.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb863da640 10 monclient(hunting): authenticate will time out at 1383.950313s 2024-04-26T00:26:42.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb7f7fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fbb800a06e0 0x7fbb80134ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:42.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb84977640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb801350e0 0x7fbb80135550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:42.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb7f7fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fbb800a06e0 0x7fbb80134ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50066/0 (socket says 172.21.15.37:50066) 2024-04-26T00:26:42.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb7f7fe640 1 -- 172.21.15.37:0/766475679 learned_addr learned my addr 172.21.15.37:0/766475679 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:26:42.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb7f7fe640 10 monclient(hunting): get_auth_request con 0x7fbb800a06e0 auth_method 0 2024-04-26T00:26:42.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb7f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:42.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb7f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:42.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:42.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb84977640 10 monclient(hunting): get_auth_request con 0x7fbb801350e0 auth_method 0 2024-04-26T00:26:42.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb84977640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:42.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb84977640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:42.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb84977640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:42.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb7f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:42.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb7f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:42.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb7f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:42.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb84977640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:42.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb84977640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:42.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb84977640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:42.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb7ffff640 1 --2- 172.21.15.37:0/766475679 >> v2:172.21.15.45:3300/0 conn(0x7fbb800a0300 0x7fbb8009fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:42.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb7ffff640 10 monclient(hunting): get_auth_request con 0x7fbb800a0300 auth_method 0 2024-04-26T00:26:42.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:42.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb7ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:42.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:42.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb7f7fe640 10 monclient(hunting): handle_auth_done global_id 54402 payload 274 2024-04-26T00:26:42.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb7f7fe640 1 -- 172.21.15.37:0/766475679 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb801350e0 msgr2=0x7fbb80135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:26:42.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb7f7fe640 1 --2- 172.21.15.37:0/766475679 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb801350e0 0x7fbb80135550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:42.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb7f7fe640 1 -- 172.21.15.37:0/766475679 >> v2:172.21.15.45:3300/0 conn(0x7fbb800a0300 msgr2=0x7fbb8009fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:26:42.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb7f7fe640 1 --2- 172.21.15.37:0/766475679 >> v2:172.21.15.45:3300/0 conn(0x7fbb800a0300 0x7fbb8009fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:42.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb7f7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:26:42.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb7f7fe640 1 monclient: found mon.noname-a 2024-04-26T00:26:42.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb7f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:42.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb7f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:26:42.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.640+0000 7fbb7f7fe640 1 -- 172.21.15.37:0/766475679 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb800a16d0 con 0x7fbb800a06e0 2024-04-26T00:26:42.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.644+0000 7fbb7f7fe640 1 -- 172.21.15.37:0/766475679 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb800a06e0 msgr2=0x7fbb80134ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:42.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.644+0000 7fbb7f7fe640 1 --2- 172.21.15.37:0/766475679 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb800a06e0 0x7fbb80134ba0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbb800764d0 tx=0x7fbb6801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b3b6b1de6ff06e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:42.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.644+0000 7fbb7effd640 1 -- 172.21.15.37:0/766475679 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb680b0070 con 0x7fbb800a06e0 2024-04-26T00:26:42.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.644+0000 7fbb7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:42.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.644+0000 7fbb7effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:26:42.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.644+0000 7fbb7effd640 10 monclient: dump: 2024-04-26T00:26:42.639 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:42.640 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:42.640 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.640 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.640 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:42.640 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:42.640 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:42.640 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:42.640 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:42.640 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:42.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.644+0000 7fbb7effd640 10 monclient: _finish_auth 0 2024-04-26T00:26:42.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.644+0000 7fbb7effd640 10 monclient: _check_auth_tickets 2024-04-26T00:26:42.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.644+0000 7fbb7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:42.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.644+0000 7fbb863da640 5 monclient: authenticate success, global_id 54402 2024-04-26T00:26:42.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.644+0000 7fbb863da640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:26:42.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.644+0000 7fbb7effd640 1 -- 172.21.15.37:0/766475679 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb680ab600 con 0x7fbb800a06e0 2024-04-26T00:26:42.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.644+0000 7fbb7effd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:42.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.644+0000 7fbb863da640 10 monclient: get_monmap_and_config success 2024-04-26T00:26:42.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.644+0000 7fbb7effd640 1 -- 172.21.15.37:0/766475679 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb680ab9d0 con 0x7fbb800a06e0 2024-04-26T00:26:42.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.644+0000 7fbb863da640 4 set_mon_vals no callback set 2024-04-26T00:26:42.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.644+0000 7fbb7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:42.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.644+0000 7fbb7effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:26:42.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.644+0000 7fbb7effd640 10 monclient: dump: 2024-04-26T00:26:42.641 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:42.641 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:42.641 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.641 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.641 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:42.642 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:42.642 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:42.642 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:42.642 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:42.642 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:42.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.644+0000 7fbb863da640 10 monclient: shutdown 2024-04-26T00:26:42.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.644+0000 7fbb863da640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:42.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.644+0000 7fbb863da640 1 -- 172.21.15.37:0/766475679 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb800a06e0 msgr2=0x7fbb80134ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:42.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.644+0000 7fbb863da640 1 --2- 172.21.15.37:0/766475679 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb800a06e0 0x7fbb80134ba0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbb800764d0 tx=0x7fbb6801c3e0 comp rx=0 tx=0).stop 2024-04-26T00:26:42.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb863da640 1 -- 172.21.15.37:0/766475679 shutdown_connections 2024-04-26T00:26:42.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb863da640 1 --2- 172.21.15.37:0/766475679 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb801350e0 0x7fbb80135550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:42.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb863da640 1 --2- 172.21.15.37:0/766475679 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb800a06e0 0x7fbb80134ba0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:42.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb863da640 1 --2- 172.21.15.37:0/766475679 >> v2:172.21.15.45:3300/0 conn(0x7fbb800a0300 0x7fbb8009fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:42.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb863da640 1 -- 172.21.15.37:0/766475679 >> 172.21.15.37:0/766475679 conn(0x7fbb800a1d50 msgr2=0x7fbb800a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:42.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb863da640 1 -- 172.21.15.37:0/766475679 shutdown_connections 2024-04-26T00:26:42.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb863da640 1 -- 172.21.15.37:0/766475679 wait complete. 2024-04-26T00:26:42.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb863da640 10 monclient: build_initial_monmap 2024-04-26T00:26:42.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb863da640 10 monclient: monmap: 2024-04-26T00:26:42.643 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:42.643 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:26:42.643 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:26:42.644 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:26:42.644 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:42.644 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:26:42.644 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:26:42.644 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:26:42.644 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:26:42.644 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:42.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb863da640 1 Processor -- start 2024-04-26T00:26:42.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb863da640 1 -- start start 2024-04-26T00:26:42.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb863da640 10 monclient: init 2024-04-26T00:26:42.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb863da640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:42.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb863da640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:26:42.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb863da640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb800a0300 0x7fbb801cf460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:42.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb863da640 10 monclient(hunting): picked mon.noname-b con 0x7fbb800a0300 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:26:42.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb863da640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb800a06e0 0x7fbb8012fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:42.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb863da640 10 monclient(hunting): picked mon.noname-a con 0x7fbb800a06e0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:26:42.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb863da640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb801350e0 0x7fbb80130300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:42.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb863da640 10 monclient(hunting): picked mon.noname-c con 0x7fbb801350e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:26:42.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb863da640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:42.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb863da640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fbb8013e6f0 con 0x7fbb800a06e0 2024-04-26T00:26:42.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb863da640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:42.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb863da640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbb8013e870 con 0x7fbb800a0300 2024-04-26T00:26:42.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb863da640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:42.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb863da640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fbb8013e9f0 con 0x7fbb801350e0 2024-04-26T00:26:42.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb863da640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:42.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb863da640 10 monclient(hunting): authenticate will time out at 1383.958106s 2024-04-26T00:26:42.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb7f7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb800a06e0 0x7fbb8012fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:42.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb84977640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb801350e0 0x7fbb80130300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:42.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb7f7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb800a06e0 0x7fbb8012fdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50078/0 (socket says 172.21.15.37:50078) 2024-04-26T00:26:42.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb7f7fe640 1 -- 172.21.15.37:0/3976753029 learned_addr learned my addr 172.21.15.37:0/3976753029 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:26:42.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb7f7fe640 10 monclient(hunting): get_auth_request con 0x7fbb800a06e0 auth_method 0 2024-04-26T00:26:42.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb7f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:42.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb7f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:42.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:42.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb84977640 10 monclient(hunting): get_auth_request con 0x7fbb801350e0 auth_method 0 2024-04-26T00:26:42.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb84977640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:42.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb84977640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:42.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb84977640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:42.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb7f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:42.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb7f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:42.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb7f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:42.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb84977640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:42.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb84977640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:42.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb84977640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:42.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb7f7fe640 10 monclient(hunting): handle_auth_done global_id 54405 payload 824 2024-04-26T00:26:42.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb7ffff640 1 --2- 172.21.15.37:0/3976753029 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb800a0300 0x7fbb801cf460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:42.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb7f7fe640 1 -- 172.21.15.37:0/3976753029 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb801350e0 msgr2=0x7fbb80130300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:42.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb7f7fe640 1 --2- 172.21.15.37:0/3976753029 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb801350e0 0x7fbb80130300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:42.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb7f7fe640 1 -- 172.21.15.37:0/3976753029 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb800a0300 msgr2=0x7fbb801cf460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:42.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb7f7fe640 1 --2- 172.21.15.37:0/3976753029 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb800a0300 0x7fbb801cf460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:42.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb7f7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:26:42.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb7f7fe640 1 monclient: found mon.noname-a 2024-04-26T00:26:42.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb7f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:42.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb7f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:26:42.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.648+0000 7fbb7f7fe640 1 -- 172.21.15.37:0/3976753029 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb80071b30 con 0x7fbb800a06e0 2024-04-26T00:26:42.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb7f7fe640 1 -- 172.21.15.37:0/3976753029 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb800a06e0 msgr2=0x7fbb8012fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:42.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb7f7fe640 1 --2- 172.21.15.37:0/3976753029 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb800a06e0 0x7fbb8012fdc0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fbb6801aa90 tx=0x7fbb68019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:42.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb7cff9640 1 -- 172.21.15.37:0/3976753029 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb680b0040 con 0x7fbb800a06e0 2024-04-26T00:26:42.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb7cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:42.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb7cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:26:42.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb7cff9640 10 monclient: dump: 2024-04-26T00:26:42.649 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:42.649 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:42.649 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.649 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.649 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:42.649 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:42.649 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:42.649 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:42.649 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:42.649 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:42.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb7cff9640 10 monclient: _finish_auth 0 2024-04-26T00:26:42.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb7cff9640 10 monclient: _check_auth_tickets 2024-04-26T00:26:42.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb7cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:42.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb7cff9640 1 -- 172.21.15.37:0/3976753029 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb680aa5b0 con 0x7fbb800a06e0 2024-04-26T00:26:42.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb863da640 5 monclient: authenticate success, global_id 54405 2024-04-26T00:26:42.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb863da640 10 monclient: _renew_subs 2024-04-26T00:26:42.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb863da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:26:42.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb863da640 1 -- 172.21.15.37:0/3976753029 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb8011d7a0 con 0x7fbb800a06e0 2024-04-26T00:26:42.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb7cff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:42.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb7cff9640 1 -- 172.21.15.37:0/3976753029 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb680aa9a0 con 0x7fbb800a06e0 2024-04-26T00:26:42.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb7cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:42.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb7cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:26:42.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb7cff9640 10 monclient: dump: 2024-04-26T00:26:42.650 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:42.650 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:42.651 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.651 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.651 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:42.651 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:42.651 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:42.651 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:42.651 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:42.651 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:42.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb7e7fc640 4 set_mon_vals no callback set 2024-04-26T00:26:42.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb863da640 10 monclient: _renew_subs 2024-04-26T00:26:42.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb863da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:26:42.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb863da640 1 -- 172.21.15.37:0/3976753029 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb80130840 con 0x7fbb800a06e0 2024-04-26T00:26:42.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb863da640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:42.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb863da640 1 --2- 172.21.15.37:0/3976753029 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb8006fee0 0x7fbb800702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:42.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb863da640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:42.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb863da640 1 -- 172.21.15.37:0/3976753029 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fbb800a0cd0 con 0x7fbb8006fee0 2024-04-26T00:26:42.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb7ffff640 1 --2- 172.21.15.37:0/3976753029 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb8006fee0 0x7fbb800702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:42.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb7ffff640 10 monclient: get_auth_request con 0x7fbb8006fee0 auth_method 0 2024-04-26T00:26:42.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:42.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb7ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:42.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:42.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb7cff9640 1 -- 172.21.15.37:0/3976753029 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fbb680aab40 con 0x7fbb800a06e0 2024-04-26T00:26:42.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb7ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:42.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb7ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:42.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb7ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:42.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb7ffff640 10 monclient(hunting): handle_auth_done global_id 54408 payload 824 2024-04-26T00:26:42.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb7ffff640 1 -- 172.21.15.37:0/3976753029 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbb8009b710 con 0x7fbb8006fee0 2024-04-26T00:26:42.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb7cff9640 1 --2- 172.21.15.37:0/3976753029 >> v2:172.21.15.45:6812/2379918623 conn(0x7fbb60035eb0 0x7fbb60038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:42.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb7cff9640 1 -- 172.21.15.37:0/3976753029 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(263..263 src has 177..263) ==== 3979+0+0 (secure 0 0 0) 0x7fbb680e2660 con 0x7fbb800a06e0 2024-04-26T00:26:42.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb84977640 1 -- 172.21.15.37:0/3976753029 >> v2:172.21.15.45:6812/2379918623 conn(0x7fbb60035eb0 msgr2=0x7fbb60038370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:26:42.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb84977640 1 -- 172.21.15.37:0/3976753029 >> v2:172.21.15.45:6812/2379918623 conn(0x7fbb60035eb0 msgr2=0x7fbb60038370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:26:42.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb84977640 1 --2- 172.21.15.37:0/3976753029 >> v2:172.21.15.45:6812/2379918623 conn(0x7fbb60035eb0 0x7fbb60038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2024-04-26T00:26:42.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.652+0000 7fbb84977640 1 --2- 172.21.15.37:0/3976753029 >> v2:172.21.15.45:6812/2379918623 conn(0x7fbb60035eb0 0x7fbb60038370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:26:42.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.656+0000 7fbb7ffff640 1 -- 172.21.15.37:0/3976753029 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb8006fee0 msgr2=0x7fbb800702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:42.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.656+0000 7fbb7ffff640 1 --2- 172.21.15.37:0/3976753029 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb8006fee0 0x7fbb800702c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fbb78007d70 tx=0x7fbb7800f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:42.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.656+0000 7fbb7cff9640 1 -- 172.21.15.37:0/3976753029 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb78018020 con 0x7fbb8006fee0 2024-04-26T00:26:42.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.656+0000 7fbb7cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:42.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.656+0000 7fbb7cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:26:42.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.656+0000 7fbb7cff9640 10 monclient: dump: 2024-04-26T00:26:42.656 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:42.656 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:42.656 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.656 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.656 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:42.656 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:42.656 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:42.656 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:42.656 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:42.656 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:42.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.656+0000 7fbb7cff9640 1 -- 172.21.15.37:0/3976753029 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbb78014800 con 0x7fbb8006fee0 2024-04-26T00:26:42.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.656+0000 7fbb7cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:42.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.656+0000 7fbb7cff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:26:42.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.656+0000 7fbb7cff9640 1 -- 172.21.15.37:0/3976753029 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb8006fee0 msgr2=0x7fbb800702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:42.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.656+0000 7fbb7cff9640 1 --2- 172.21.15.37:0/3976753029 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb8006fee0 0x7fbb800702c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fbb78007d70 tx=0x7fbb7800f520 comp rx=0 tx=0).stop 2024-04-26T00:26:42.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.656+0000 7fbb7cff9640 1 -- 172.21.15.37:0/3976753029 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb8006fee0 msgr2=0x7fbb800702c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:26:42.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.656+0000 7fbb7cff9640 1 --2- 172.21.15.37:0/3976753029 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb8006fee0 0x7fbb800702c0 secure :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fbb78007d70 tx=0x7fbb7800f520 comp rx=0 tx=0).stop 2024-04-26T00:26:42.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.660+0000 7fbb863da640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:26:42.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.660+0000 7fbb863da640 1 --2- 172.21.15.37:0/3976753029 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb80070800 0x7fbb8009ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:42.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.660+0000 7fbb863da640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:42.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.660+0000 7fbb863da640 1 -- 172.21.15.37:0/3976753029 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fbb801cf9a0 con 0x7fbb80070800 2024-04-26T00:26:42.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.660+0000 7fbb7ffff640 1 --2- 172.21.15.37:0/3976753029 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb80070800 0x7fbb8009ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:42.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.660+0000 7fbb7ffff640 10 monclient: get_auth_request con 0x7fbb80070800 auth_method 0 2024-04-26T00:26:42.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.660+0000 7fbb7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:42.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.660+0000 7fbb7ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:42.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.660+0000 7fbb7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:42.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.660+0000 7fbb7ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:42.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.660+0000 7fbb7ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:42.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.660+0000 7fbb7ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:42.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.664+0000 7fbb7ffff640 10 monclient(hunting): handle_auth_done global_id 54411 payload 824 2024-04-26T00:26:42.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.664+0000 7fbb7ffff640 1 -- 172.21.15.37:0/3976753029 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbb8011cf30 con 0x7fbb80070800 2024-04-26T00:26:42.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.664+0000 7fbb7ffff640 1 -- 172.21.15.37:0/3976753029 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb80070800 msgr2=0x7fbb8009ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:42.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.664+0000 7fbb7ffff640 1 --2- 172.21.15.37:0/3976753029 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb80070800 0x7fbb8009ccb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbb78007d70 tx=0x7fbb78004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:42.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.664+0000 7fbb7cff9640 1 -- 172.21.15.37:0/3976753029 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb78018030 con 0x7fbb80070800 2024-04-26T00:26:42.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.664+0000 7fbb7cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:42.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.664+0000 7fbb7cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:26:42.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.664+0000 7fbb7cff9640 10 monclient: dump: 2024-04-26T00:26:42.659 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:42.659 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:42.659 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.659 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.659 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:42.659 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:42.659 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:42.659 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:42.659 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:42.659 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:42.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.664+0000 7fbb7cff9640 1 -- 172.21.15.37:0/3976753029 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7fbb780283c0 con 0x7fbb80070800 2024-04-26T00:26:42.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.664+0000 7fbb7cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:26:42.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.664+0000 7fbb7cff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:26:42.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.664+0000 7fbb7cff9640 1 -- 172.21.15.37:0/3976753029 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb80070800 msgr2=0x7fbb8009ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:42.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.664+0000 7fbb7cff9640 1 --2- 172.21.15.37:0/3976753029 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb80070800 0x7fbb8009ccb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbb78007d70 tx=0x7fbb78004a50 comp rx=0 tx=0).stop 2024-04-26T00:26:42.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.664+0000 7fbb7cff9640 1 -- 172.21.15.37:0/3976753029 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb80070800 msgr2=0x7fbb8009ccb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:26:42.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.664+0000 7fbb7cff9640 1 --2- 172.21.15.37:0/3976753029 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb80070800 0x7fbb8009ccb0 secure :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbb78007d70 tx=0x7fbb78004a50 comp rx=0 tx=0).stop 2024-04-26T00:26:42.660 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:26:42.660 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:26:42.660 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:26:42.660 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:26:42.660 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 52, 2024-04-26T00:26:42.661 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 165990, 2024-04-26T00:26:42.661 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:26:42.661 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:26:42.661 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:26:42.661 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:26:42.661 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:42.661 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 6, 2024-04-26T00:26:42.661 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:26:42.661 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:26:42.661 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:26:42.661 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:26:42.661 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:26:42.661 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:26:42.661 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:26:42.662 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:26:42.662 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:26:42.662 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:26:42.662 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:26:42.662 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:26:42.662 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:26:42.662 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:26:42.662 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:42.662 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:26:42.662 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:26:42.662 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:26:42.662 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:26:42.662 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:26:42.662 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:26:42.663 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:26:42.663 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:26:42.663 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:26:42.663 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:26:42.663 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:26:42.663 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:26:42.663 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:26:42.663 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:42.663 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:42.663 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:26:42.663 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:26:42.663 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:26:42.663 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:26:42.663 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:26:42.664 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:26:42.664 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:26:42.664 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:26:42.664 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:26:42.664 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:26:42.664 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:26:42.664 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:26:42.664 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:26:42.664 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:26:42.664 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:26:42.664 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:26:42.664 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:26:42.664 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:26:42.664 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:26:42.665 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:26:42.665 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:26:42.665 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:26:42.665 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:26:42.665 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:26:42.665 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:26:42.665 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:26:42.665 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:26:42.665 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:26:42.665 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:42.665 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:26:42.665 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:42.665 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:26:42.665 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:42.666 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:26:42.666 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:26:42.666 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:26:42.666 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:26:42.666 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:42.666 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:26:42.666 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:26:42.666 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:42.666 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:42.666 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:42.666 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:26:42.666 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:26:42.666 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:42.666 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:42.667 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:42.667 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:42.667 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:26:42.667 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:26:42.667 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:26:42.667 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:26:42.667 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:26:42.667 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:42.667 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:42.667 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:26:42.667 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:26:42.667 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:26:42.667 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:26:42.667 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:42.668 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:26:42.668 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:26:42.668 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:42.668 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:42.668 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:42.668 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:26:42.668 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:26:42.668 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:42.668 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:42.668 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:42.668 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:42.668 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:26:42.668 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:26:42.668 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:26:42.669 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:26:42.669 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:26:42.669 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:42.669 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:42.669 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:26:42.669 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:26:42.669 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:26:42.669 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:26:42.669 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:42.669 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:26:42.669 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:26:42.669 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:42.669 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:42.669 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:42.670 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:26:42.670 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:26:42.670 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:42.670 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:42.670 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:42.670 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:42.670 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:26:42.670 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:26:42.670 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:26:42.670 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:26:42.670 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:26:42.670 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:42.670 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:42.670 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:42.671 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:26:42.671 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:26:42.671 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:42.671 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:26:42.671 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:26:42.671 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:26:42.671 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:42.671 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:42.671 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:26:42.671 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:42.671 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:26:42.671 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:26:42.671 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:26:42.671 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:42.672 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:42.672 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:26:42.672 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:42.672 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:26:42.672 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:26:42.672 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:26:42.672 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:42.672 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:42.672 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:26:42.672 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:42.672 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:26:42.672 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:26:42.672 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:26:42.672 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:42.673 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:42.673 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:42.673 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:26:42.673 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:26:42.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.664+0000 7fbb863da640 1 -- 172.21.15.37:0/3976753029 >> v2:172.21.15.45:6812/2379918623 conn(0x7fbb60035eb0 msgr2=0x7fbb60038370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:26:42.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.664+0000 7fbb863da640 1 --2- 172.21.15.37:0/3976753029 >> v2:172.21.15.45:6812/2379918623 conn(0x7fbb60035eb0 0x7fbb60038370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:42.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.664+0000 7fbb863da640 10 monclient: shutdown 2024-04-26T00:26:42.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.664+0000 7fbb863da640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:42.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.664+0000 7fbb863da640 1 -- 172.21.15.37:0/3976753029 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb800a06e0 msgr2=0x7fbb8012fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:42.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.664+0000 7fbb863da640 1 --2- 172.21.15.37:0/3976753029 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb800a06e0 0x7fbb8012fdc0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fbb6801aa90 tx=0x7fbb68019180 comp rx=0 tx=0).stop 2024-04-26T00:26:42.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.664+0000 7fbb84977640 1 -- 172.21.15.37:0/3976753029 reap_dead start 2024-04-26T00:26:42.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.664+0000 7fbb84977640 1 -- 172.21.15.37:0/3976753029 reap_dead start 2024-04-26T00:26:42.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.664+0000 7fbb863da640 1 -- 172.21.15.37:0/3976753029 shutdown_connections 2024-04-26T00:26:42.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.664+0000 7fbb863da640 1 -- 172.21.15.37:0/3976753029 >> 172.21.15.37:0/3976753029 conn(0x7fbb800a1d50 msgr2=0x7fbb800a3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:42.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.664+0000 7fbb863da640 1 -- 172.21.15.37:0/3976753029 shutdown_connections 2024-04-26T00:26:42.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.664+0000 7fbb863da640 1 -- 172.21.15.37:0/3976753029 wait complete. 2024-04-26T00:26:42.675 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:26:42.675 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:26:42.675 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:26:42.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.804+0000 7fe086f53640 10 monclient: get_monmap_and_config 2024-04-26T00:26:42.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.804+0000 7fe086f53640 10 monclient: build_initial_monmap 2024-04-26T00:26:42.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe086f53640 10 monclient: monmap: 2024-04-26T00:26:42.792 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:42.792 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:42.793 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:26:42.811889+0000 2024-04-26T00:26:42.793 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:26:42.811889+0000 2024-04-26T00:26:42.793 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:42.793 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:42.793 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:26:42.793 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:26:42.793 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:26:42.793 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:26:42.793 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:26:42.793 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:42.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe086f53640 1 Processor -- start 2024-04-26T00:26:42.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe086f53640 1 -- start start 2024-04-26T00:26:42.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe086f53640 10 monclient: init 2024-04-26T00:26:42.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe086f53640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:42.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe086f53640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-04-26T00:26:42.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe086f53640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe0800a0300 0x7fe08009fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:42.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe086f53640 10 monclient(hunting): picked mon.noname-c con 0x7fe0800a0300 addr v2:172.21.15.45:3300/0 2024-04-26T00:26:42.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe086f53640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fe0800a06e0 0x7fe080134ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:42.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe086f53640 10 monclient(hunting): picked mon.noname-a con 0x7fe0800a06e0 addr v2:172.21.15.37:3300/0 2024-04-26T00:26:42.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe086f53640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe0801350e0 addr v1:172.21.15.37:6789/0 2024-04-26T00:26:42.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe086f53640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe08009c190 con 0x7fe0801350e0 2024-04-26T00:26:42.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe086f53640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:42.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe086f53640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fe0800a13f0 con 0x7fe0800a06e0 2024-04-26T00:26:42.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe086f53640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:42.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe086f53640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fe0800a1110 con 0x7fe0800a0300 2024-04-26T00:26:42.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe086f53640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:42.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe086f53640 10 monclient(hunting): authenticate will time out at 1384.117541s 2024-04-26T00:26:42.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe07ffff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fe0800a06e0 0x7fe080134ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:42.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe07ffff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fe0800a06e0 0x7fe080134ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50102/0 (socket says 172.21.15.37:50102) 2024-04-26T00:26:42.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe0854f0640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fe0801350e0 0x7fe08013bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:33310/0 (socket says 172.21.15.37:33310) 2024-04-26T00:26:42.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe07ffff640 1 -- 172.21.15.37:0/1851975000 learned_addr learned my addr 172.21.15.37:0/1851975000 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:26:42.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe07ffff640 10 monclient(hunting): get_auth_request con 0x7fe0800a06e0 auth_method 0 2024-04-26T00:26:42.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe07ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:42.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe07ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:42.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe07ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:42.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe07ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:42.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe07ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:42.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe07ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:42.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe084cef640 1 --2- 172.21.15.37:0/1851975000 >> v2:172.21.15.45:3300/0 conn(0x7fe0800a0300 0x7fe08009fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:42.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe084cef640 10 monclient(hunting): get_auth_request con 0x7fe0800a0300 auth_method 0 2024-04-26T00:26:42.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe084cef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:42.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe084cef640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:42.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe084cef640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:42.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe07ffff640 10 monclient(hunting): handle_auth_done global_id 54414 payload 274 2024-04-26T00:26:42.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe07ffff640 1 -- 172.21.15.37:0/1851975000 >> v2:172.21.15.45:3300/0 conn(0x7fe0800a0300 msgr2=0x7fe08009fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:26:42.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe07ffff640 1 --2- 172.21.15.37:0/1851975000 >> v2:172.21.15.45:3300/0 conn(0x7fe0800a0300 0x7fe08009fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:42.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe07ffff640 1 -- 172.21.15.37:0/1851975000 >> v1:172.21.15.37:6789/0 conn(0x7fe0801350e0 legacy=0x7fe08013bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:26:42.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe07ffff640 10 monclient: _finish_hunting 0 2024-04-26T00:26:42.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe07ffff640 1 monclient: found mon.noname-a 2024-04-26T00:26:42.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe07ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:42.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe07ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:26:42.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.808+0000 7fe07ffff640 1 -- 172.21.15.37:0/1851975000 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0800a16d0 con 0x7fe0800a06e0 2024-04-26T00:26:42.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.812+0000 7fe07ffff640 1 -- 172.21.15.37:0/1851975000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0800a06e0 msgr2=0x7fe080134ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:42.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.812+0000 7fe07ffff640 1 --2- 172.21.15.37:0/1851975000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0800a06e0 0x7fe080134ba0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe0800764d0 tx=0x7fe07401c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35c8ed00dd5cbfc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:42.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.812+0000 7fe07f7fe640 1 -- 172.21.15.37:0/1851975000 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0740b0070 con 0x7fe0800a06e0 2024-04-26T00:26:42.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.812+0000 7fe07f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:42.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.812+0000 7fe07f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:26:42.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.812+0000 7fe07f7fe640 10 monclient: dump: 2024-04-26T00:26:42.800 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:42.801 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:42.801 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.801 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.801 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:42.801 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:42.801 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:42.801 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:42.801 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:42.801 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:42.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.812+0000 7fe07f7fe640 10 monclient: _finish_auth 0 2024-04-26T00:26:42.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.812+0000 7fe07f7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:26:42.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.812+0000 7fe07f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:42.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.812+0000 7fe07f7fe640 1 -- 172.21.15.37:0/1851975000 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0740ab600 con 0x7fe0800a06e0 2024-04-26T00:26:42.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.812+0000 7fe086f53640 5 monclient: authenticate success, global_id 54414 2024-04-26T00:26:42.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.812+0000 7fe086f53640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:26:42.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.812+0000 7fe07f7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:42.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.812+0000 7fe086f53640 10 monclient: get_monmap_and_config success 2024-04-26T00:26:42.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.812+0000 7fe086f53640 4 set_mon_vals no callback set 2024-04-26T00:26:42.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.812+0000 7fe07f7fe640 1 -- 172.21.15.37:0/1851975000 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0740ab9d0 con 0x7fe0800a06e0 2024-04-26T00:26:42.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.812+0000 7fe07f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:42.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.812+0000 7fe07f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:26:42.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.812+0000 7fe07f7fe640 10 monclient: dump: 2024-04-26T00:26:42.802 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:42.802 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:42.802 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.802 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.802 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:42.802 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:42.802 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:42.803 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:42.803 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:42.803 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:42.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.812+0000 7fe086f53640 10 monclient: shutdown 2024-04-26T00:26:42.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.812+0000 7fe086f53640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:42.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.812+0000 7fe086f53640 1 -- 172.21.15.37:0/1851975000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0800a06e0 msgr2=0x7fe080134ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:42.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.812+0000 7fe086f53640 1 --2- 172.21.15.37:0/1851975000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0800a06e0 0x7fe080134ba0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe0800764d0 tx=0x7fe07401c3d0 comp rx=0 tx=0).stop 2024-04-26T00:26:42.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.812+0000 7fe086f53640 1 -- 172.21.15.37:0/1851975000 shutdown_connections 2024-04-26T00:26:42.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.812+0000 7fe086f53640 1 --2- 172.21.15.37:0/1851975000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0800a06e0 0x7fe080134ba0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:42.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.812+0000 7fe086f53640 1 --2- 172.21.15.37:0/1851975000 >> v2:172.21.15.45:3300/0 conn(0x7fe0800a0300 0x7fe08009fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:42.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.812+0000 7fe086f53640 1 -- 172.21.15.37:0/1851975000 >> 172.21.15.37:0/1851975000 conn(0x7fe0800a1d50 msgr2=0x7fe0800a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:42.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.812+0000 7fe086f53640 1 -- 172.21.15.37:0/1851975000 shutdown_connections 2024-04-26T00:26:42.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.812+0000 7fe086f53640 1 -- 172.21.15.37:0/1851975000 wait complete. 2024-04-26T00:26:42.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.812+0000 7fe086f53640 10 monclient: build_initial_monmap 2024-04-26T00:26:42.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.812+0000 7fe086f53640 10 monclient: monmap: 2024-04-26T00:26:42.804 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:42.804 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:26:42.804 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:26:42.804 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:26:42.804 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:42.804 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:26:42.804 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:26:42.804 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:26:42.804 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:26:42.804 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:42.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe086f53640 1 Processor -- start 2024-04-26T00:26:42.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe086f53640 1 -- start start 2024-04-26T00:26:42.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe086f53640 10 monclient: init 2024-04-26T00:26:42.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe086f53640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:42.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe086f53640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:26:42.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe086f53640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0800a0300 0x7fe0801cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:42.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe086f53640 10 monclient(hunting): picked mon.noname-a con 0x7fe0800a0300 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:26:42.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe086f53640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe0800a06e0 0x7fe0801310a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:42.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe086f53640 10 monclient(hunting): picked mon.noname-c con 0x7fe0800a06e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:26:42.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe086f53640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0801350e0 0x7fe0801cf610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:42.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe086f53640 10 monclient(hunting): picked mon.noname-b con 0x7fe0801350e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:26:42.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe086f53640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:42.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe086f53640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe0800a0c40 con 0x7fe0800a0300 2024-04-26T00:26:42.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe086f53640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:42.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe086f53640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe0800a0ac0 con 0x7fe0801350e0 2024-04-26T00:26:42.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe086f53640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:42.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe086f53640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe08013fe10 con 0x7fe0800a06e0 2024-04-26T00:26:42.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe086f53640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:42.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe086f53640 10 monclient(hunting): authenticate will time out at 1384.124743s 2024-04-26T00:26:42.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe084cef640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0800a0300 0x7fe0801cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:42.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe084cef640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0800a0300 0x7fe0801cc0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50108/0 (socket says 172.21.15.37:50108) 2024-04-26T00:26:42.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe084cef640 1 -- 172.21.15.37:0/82709754 learned_addr learned my addr 172.21.15.37:0/82709754 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:26:42.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe084cef640 10 monclient(hunting): get_auth_request con 0x7fe0800a0300 auth_method 0 2024-04-26T00:26:42.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe084cef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:42.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe084cef640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:42.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe084cef640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:42.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe07ffff640 1 --2- 172.21.15.37:0/82709754 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe0800a06e0 0x7fe0801310a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:42.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe084cef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:42.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe084cef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:42.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe084cef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:42.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe07ffff640 10 monclient(hunting): get_auth_request con 0x7fe0800a06e0 auth_method 0 2024-04-26T00:26:42.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe07ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:42.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe07ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:42.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe07ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:42.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe084cef640 10 monclient(hunting): handle_auth_done global_id 54417 payload 824 2024-04-26T00:26:42.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe084cef640 1 -- 172.21.15.37:0/82709754 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe0800a06e0 msgr2=0x7fe0801310a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:42.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe0854f0640 1 --2- 172.21.15.37:0/82709754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0801350e0 0x7fe0801cf610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:42.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe084cef640 1 --2- 172.21.15.37:0/82709754 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe0800a06e0 0x7fe0801310a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:42.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe084cef640 1 -- 172.21.15.37:0/82709754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0801350e0 msgr2=0x7fe0801cf610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:42.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe084cef640 1 --2- 172.21.15.37:0/82709754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0801350e0 0x7fe0801cf610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:42.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe084cef640 10 monclient: _finish_hunting 0 2024-04-26T00:26:42.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe084cef640 1 monclient: found mon.noname-a 2024-04-26T00:26:42.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe084cef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:42.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe084cef640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:26:42.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe084cef640 1 -- 172.21.15.37:0/82709754 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe08011c040 con 0x7fe0800a0300 2024-04-26T00:26:42.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe0854f0640 10 monclient: get_auth_request con 0x7fe0801350e0 auth_method 0 2024-04-26T00:26:42.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe07ffff640 1 --2- 172.21.15.37:0/82709754 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe0800a06e0 0x7fe0801310a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:26:42.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe0854f0640 1 --2- 172.21.15.37:0/82709754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0801350e0 0x7fe0801cf610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:26:42.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe084cef640 1 -- 172.21.15.37:0/82709754 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0800a0300 msgr2=0x7fe0801cc0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:42.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe084cef640 1 --2- 172.21.15.37:0/82709754 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0800a0300 0x7fe0801cc0b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe07000d800 tx=0x7fe07000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:42.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe07d7fa640 1 -- 172.21.15.37:0/82709754 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe07000b8d0 con 0x7fe0800a0300 2024-04-26T00:26:42.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe07d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:42.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe07d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:26:42.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe07d7fa640 10 monclient: dump: 2024-04-26T00:26:42.812 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:42.812 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:42.812 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.813 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.813 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:42.813 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:42.813 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:42.813 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:42.813 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:42.813 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:42.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe07d7fa640 10 monclient: _finish_auth 0 2024-04-26T00:26:42.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe07d7fa640 10 monclient: _check_auth_tickets 2024-04-26T00:26:42.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe07d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:42.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe07d7fa640 1 -- 172.21.15.37:0/82709754 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe070019070 con 0x7fe0800a0300 2024-04-26T00:26:42.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe07d7fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:42.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe07d7fa640 1 -- 172.21.15.37:0/82709754 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe07000bc70 con 0x7fe0800a0300 2024-04-26T00:26:42.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe086f53640 5 monclient: authenticate success, global_id 54417 2024-04-26T00:26:42.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe086f53640 10 monclient: _renew_subs 2024-04-26T00:26:42.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe086f53640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:26:42.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe086f53640 1 -- 172.21.15.37:0/82709754 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe08009f550 con 0x7fe0800a0300 2024-04-26T00:26:42.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe07d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:42.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe07d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:26:42.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe07d7fa640 10 monclient: dump: 2024-04-26T00:26:42.814 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:42.814 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:42.814 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.814 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.814 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:42.814 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:42.814 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:42.814 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:42.815 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:42.815 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:42.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe07effd640 4 set_mon_vals no callback set 2024-04-26T00:26:42.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe086f53640 10 monclient: _renew_subs 2024-04-26T00:26:42.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe086f53640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:26:42.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.816+0000 7fe086f53640 1 -- 172.21.15.37:0/82709754 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe08011da50 con 0x7fe0800a0300 2024-04-26T00:26:42.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.820+0000 7fe086f53640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:42.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.820+0000 7fe086f53640 1 --2- 172.21.15.37:0/82709754 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe0801315e0 0x7fe0801cfb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:42.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.820+0000 7fe086f53640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:42.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.820+0000 7fe086f53640 1 -- 172.21.15.37:0/82709754 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe08011d8c0 con 0x7fe0801315e0 2024-04-26T00:26:42.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.820+0000 7fe07d7fa640 1 -- 172.21.15.37:0/82709754 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe07001e460 con 0x7fe0800a0300 2024-04-26T00:26:42.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.820+0000 7fe07ffff640 1 --2- 172.21.15.37:0/82709754 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe0801315e0 0x7fe0801cfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:42.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.820+0000 7fe07ffff640 10 monclient: get_auth_request con 0x7fe0801315e0 auth_method 0 2024-04-26T00:26:42.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.820+0000 7fe07ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:42.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.820+0000 7fe07ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:42.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.820+0000 7fe07ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:42.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.820+0000 7fe07ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:42.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.820+0000 7fe07ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:42.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.820+0000 7fe07ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:42.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.820+0000 7fe07d7fa640 1 --2- 172.21.15.37:0/82709754 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe06c035f00 0x7fe06c0383c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:42.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.820+0000 7fe07ffff640 10 monclient(hunting): handle_auth_done global_id 84125 payload 824 2024-04-26T00:26:42.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.820+0000 7fe07ffff640 1 -- 172.21.15.37:0/82709754 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe08011d620 con 0x7fe0801315e0 2024-04-26T00:26:42.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.820+0000 7fe07d7fa640 1 -- 172.21.15.37:0/82709754 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(263..263 src has 177..263) ==== 3979+0+0 (secure 0 0 0) 0x7fe070049030 con 0x7fe0800a0300 2024-04-26T00:26:42.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.820+0000 7fe0854f0640 1 --2- 172.21.15.37:0/82709754 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe06c035f00 0x7fe06c0383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:42.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.820+0000 7fe0854f0640 10 monclient: get_auth_request con 0x7fe06c035f00 auth_method 0 2024-04-26T00:26:42.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.820+0000 7fe07ffff640 1 -- 172.21.15.37:0/82709754 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe0801315e0 msgr2=0x7fe0801cfb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:42.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.820+0000 7fe07ffff640 1 --2- 172.21.15.37:0/82709754 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe0801315e0 0x7fe0801cfb50 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe08006de30 tx=0x7fe0740191d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:42.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.820+0000 7fe07d7fa640 1 -- 172.21.15.37:0/82709754 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0740b0040 con 0x7fe0801315e0 2024-04-26T00:26:42.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.820+0000 7fe07d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:42.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.820+0000 7fe07d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:26:42.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.820+0000 7fe07d7fa640 10 monclient: dump: 2024-04-26T00:26:42.817 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:42.817 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:42.817 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.817 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.817 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:42.817 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:42.817 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:42.817 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:42.817 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:42.817 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:42.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.820+0000 7fe07d7fa640 1 -- 172.21.15.37:0/82709754 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe0740b5680 con 0x7fe0801315e0 2024-04-26T00:26:42.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.820+0000 7fe07d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:42.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.820+0000 7fe07d7fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:26:42.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.820+0000 7fe07d7fa640 1 -- 172.21.15.37:0/82709754 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe0801315e0 msgr2=0x7fe0801cfb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:42.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.820+0000 7fe07d7fa640 1 --2- 172.21.15.37:0/82709754 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe0801315e0 0x7fe0801cfb50 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe08006de30 tx=0x7fe0740191d0 comp rx=0 tx=0).stop 2024-04-26T00:26:42.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.820+0000 7fe07d7fa640 1 -- 172.21.15.37:0/82709754 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe0801315e0 msgr2=0x7fe0801cfb50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:26:42.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.820+0000 7fe07d7fa640 1 --2- 172.21.15.37:0/82709754 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe0801315e0 0x7fe0801cfb50 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe08006de30 tx=0x7fe0740191d0 comp rx=0 tx=0).stop 2024-04-26T00:26:42.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.824+0000 7fe0854f0640 1 --2- 172.21.15.37:0/82709754 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe06c035f00 0x7fe06c0383c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe08011bef0 tx=0x7fe078009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:42.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.828+0000 7fe086f53640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:26:42.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.828+0000 7fe086f53640 1 --2- 172.21.15.37:0/82709754 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe08011d1b0 0x7fe0801d0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:42.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.828+0000 7fe086f53640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:42.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.828+0000 7fe086f53640 1 -- 172.21.15.37:0/82709754 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe08004f510 con 0x7fe08011d1b0 2024-04-26T00:26:42.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.828+0000 7fe07ffff640 1 --2- 172.21.15.37:0/82709754 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe08011d1b0 0x7fe0801d0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:42.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.828+0000 7fe07ffff640 10 monclient: get_auth_request con 0x7fe08011d1b0 auth_method 0 2024-04-26T00:26:42.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.828+0000 7fe07ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:42.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.828+0000 7fe07ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:42.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.828+0000 7fe07ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:42.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.828+0000 7fe07ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:42.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.828+0000 7fe07ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:42.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.828+0000 7fe07ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:42.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.828+0000 7fe07ffff640 10 monclient(hunting): handle_auth_done global_id 84128 payload 824 2024-04-26T00:26:42.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.828+0000 7fe07ffff640 1 -- 172.21.15.37:0/82709754 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe08013d3d0 con 0x7fe08011d1b0 2024-04-26T00:26:42.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.832+0000 7fe07ffff640 1 -- 172.21.15.37:0/82709754 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe08011d1b0 msgr2=0x7fe0801d0090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:42.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.832+0000 7fe07ffff640 1 --2- 172.21.15.37:0/82709754 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe08011d1b0 0x7fe0801d0090 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe07401aae0 tx=0x7fe0740a8000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:42.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.832+0000 7fe07d7fa640 1 -- 172.21.15.37:0/82709754 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0740b0040 con 0x7fe08011d1b0 2024-04-26T00:26:42.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.832+0000 7fe07d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:42.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.832+0000 7fe07d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:26:42.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.832+0000 7fe07d7fa640 10 monclient: dump: 2024-04-26T00:26:42.821 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:42.821 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:42.821 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.821 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.821 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:42.821 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:42.821 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:42.821 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:42.821 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:42.821 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:42.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.832+0000 7fe07d7fa640 1 -- 172.21.15.37:0/82709754 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7fe0740bfe50 con 0x7fe08011d1b0 2024-04-26T00:26:42.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.832+0000 7fe07d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:26:42.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.832+0000 7fe07d7fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:26:42.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.832+0000 7fe07d7fa640 1 -- 172.21.15.37:0/82709754 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe08011d1b0 msgr2=0x7fe0801d0090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:42.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.832+0000 7fe07d7fa640 1 --2- 172.21.15.37:0/82709754 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe08011d1b0 0x7fe0801d0090 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe07401aae0 tx=0x7fe0740a8000 comp rx=0 tx=0).stop 2024-04-26T00:26:42.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.832+0000 7fe07d7fa640 1 -- 172.21.15.37:0/82709754 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe08011d1b0 msgr2=0x7fe0801d0090 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:26:42.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.832+0000 7fe07d7fa640 1 --2- 172.21.15.37:0/82709754 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe08011d1b0 0x7fe0801d0090 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:42.822 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:26:42.822 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:26:42.822 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:26:42.823 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:26:42.823 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 52, 2024-04-26T00:26:42.823 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 7767, 2024-04-26T00:26:42.823 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:26:42.823 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:26:42.823 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:26:42.823 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:26:42.823 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:42.823 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 6, 2024-04-26T00:26:42.823 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:26:42.823 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:26:42.823 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:26:42.823 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:26:42.823 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:26:42.824 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:26:42.824 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:26:42.824 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:26:42.824 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:26:42.824 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:26:42.824 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:26:42.824 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:26:42.824 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:26:42.824 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:26:42.824 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:42.824 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:26:42.824 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:26:42.824 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:26:42.824 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:26:42.824 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:26:42.825 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:26:42.825 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:26:42.825 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:26:42.825 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:26:42.825 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:26:42.825 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:26:42.825 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:26:42.825 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:26:42.825 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:42.825 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:42.825 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:26:42.825 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:26:42.825 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:26:42.825 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:26:42.825 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:26:42.826 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:26:42.826 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:26:42.826 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:26:42.826 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:26:42.826 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:26:42.826 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:26:42.826 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:26:42.826 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:26:42.826 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:26:42.826 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:26:42.826 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:26:42.826 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:26:42.826 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:26:42.826 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:26:42.827 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:26:42.827 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:26:42.827 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:26:42.827 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:26:42.827 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:26:42.827 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:26:42.827 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:26:42.827 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:26:42.827 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:26:42.827 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:42.827 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:26:42.827 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:42.827 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:26:42.827 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:42.827 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:26:42.828 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:26:42.828 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:26:42.828 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:26:42.828 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:42.828 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:26:42.828 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:26:42.828 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:42.828 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:42.828 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:42.828 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:26:42.828 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:26:42.828 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:42.828 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:42.828 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:42.828 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:42.829 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:26:42.829 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:26:42.829 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:26:42.829 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:26:42.829 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:26:42.829 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:42.829 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:42.829 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:26:42.829 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:26:42.829 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:26:42.829 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:26:42.829 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:42.829 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:26:42.829 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:26:42.829 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:42.830 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:42.830 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:42.830 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:26:42.830 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:26:42.830 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:42.830 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:42.830 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:42.830 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:42.830 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:26:42.830 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:26:42.830 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:26:42.830 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:26:42.830 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:26:42.830 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:42.831 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:42.831 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:26:42.831 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:26:42.831 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:26:42.831 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:26:42.831 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:42.831 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:26:42.831 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:26:42.831 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:42.831 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:42.831 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:42.831 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:26:42.831 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:26:42.831 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:42.831 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:42.832 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:42.832 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:42.832 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:26:42.832 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:26:42.832 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:26:42.832 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:26:42.832 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:26:42.832 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:42.832 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:42.832 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:42.832 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:26:42.832 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:26:42.832 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:42.832 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:26:42.832 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:26:42.833 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:26:42.833 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:42.833 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:42.833 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:26:42.833 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:42.833 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:26:42.833 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:26:42.833 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:26:42.833 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:42.833 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:42.833 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:42.833 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:26:42.833 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:26:42.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.832+0000 7fe086f53640 1 -- 172.21.15.37:0/82709754 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe06c035f00 msgr2=0x7fe06c0383c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:42.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.832+0000 7fe086f53640 1 --2- 172.21.15.37:0/82709754 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe06c035f00 0x7fe06c0383c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe08011bef0 tx=0x7fe078009040 comp rx=0 tx=0).stop 2024-04-26T00:26:42.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.832+0000 7fe086f53640 10 monclient: shutdown 2024-04-26T00:26:42.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.832+0000 7fe086f53640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:42.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.832+0000 7fe086f53640 1 -- 172.21.15.37:0/82709754 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0800a0300 msgr2=0x7fe0801cc0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:42.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.832+0000 7fe086f53640 1 --2- 172.21.15.37:0/82709754 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0800a0300 0x7fe0801cc0b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe07000d800 tx=0x7fe07000dcc0 comp rx=0 tx=0).stop 2024-04-26T00:26:42.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.832+0000 7fe0854f0640 1 -- 172.21.15.37:0/82709754 reap_dead start 2024-04-26T00:26:42.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.832+0000 7fe0854f0640 1 -- 172.21.15.37:0/82709754 reap_dead start 2024-04-26T00:26:42.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.832+0000 7fe086f53640 1 -- 172.21.15.37:0/82709754 shutdown_connections 2024-04-26T00:26:42.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.832+0000 7fe086f53640 1 -- 172.21.15.37:0/82709754 >> 172.21.15.37:0/82709754 conn(0x7fe0800a1d50 msgr2=0x7fe080131c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:42.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.832+0000 7fe086f53640 1 -- 172.21.15.37:0/82709754 shutdown_connections 2024-04-26T00:26:42.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.832+0000 7fe086f53640 1 -- 172.21.15.37:0/82709754 wait complete. 2024-04-26T00:26:42.836 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:26:42.836 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:26:42.836 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:26:42.900 INFO:tasks.rados.rados.0.smithi045.stdout:1103: done (1 left) 2024-04-26T00:26:42.900 INFO:tasks.rados.rados.0.smithi045.stdout:1104: done (0 left) 2024-04-26T00:26:42.900 INFO:tasks.rados.rados.0.smithi045.stdout:1105: write (excl) oid 34 current snap is 130 2024-04-26T00:26:42.900 INFO:tasks.rados.rados.0.smithi045.stdout:1105: seq_num 415 ranges {574905=711397,1885086=525140,2930189=44716} 2024-04-26T00:26:42.921 INFO:tasks.rados.rados.0.smithi045.stdout:1105: writing smithi04526579-34 from 574905 to 1286302 tid 1 2024-04-26T00:26:42.939 INFO:tasks.rados.rados.0.smithi045.stdout:1105: writing smithi04526579-34 from 1885086 to 2410226 tid 2 2024-04-26T00:26:42.949 INFO:tasks.rados.rados.0.smithi045.stdout:1105: writing smithi04526579-34 from 2930189 to 2974905 tid 3 2024-04-26T00:26:42.949 INFO:tasks.rados.rados.0.smithi045.stdout:1106: write (excl) oid 47 current snap is 130 2024-04-26T00:26:42.949 INFO:tasks.rados.rados.0.smithi045.stdout:1105: finishing write tid 1 to smithi04526579-34 2024-04-26T00:26:42.949 INFO:tasks.rados.rados.0.smithi045.stdout:1105: finishing write tid 2 to smithi04526579-34 2024-04-26T00:26:42.949 INFO:tasks.rados.rados.0.smithi045.stdout:1106: seq_num 416 ranges {591712=786646,1943568=668798} 2024-04-26T00:26:42.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fb9203640 10 monclient: get_monmap_and_config 2024-04-26T00:26:42.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fb9203640 10 monclient: build_initial_monmap 2024-04-26T00:26:42.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fb9203640 10 monclient: monmap: 2024-04-26T00:26:42.953 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:42.953 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:42.953 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:26:42.972963+0000 2024-04-26T00:26:42.953 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:26:42.972963+0000 2024-04-26T00:26:42.953 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:42.954 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:42.954 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:26:42.954 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:26:42.954 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:26:42.954 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:26:42.954 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:26:42.954 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:42.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fb9203640 1 Processor -- start 2024-04-26T00:26:42.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fb9203640 1 -- start start 2024-04-26T00:26:42.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fb9203640 10 monclient: init 2024-04-26T00:26:42.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fb9203640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:42.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fb9203640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-04-26T00:26:42.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fb9203640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9fb4130eb0 0x7f9fb409ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:42.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fb9203640 10 monclient(hunting): picked mon.noname-b con 0x7f9fb4130eb0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:26:42.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fb9203640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9fb4131290 0x7f9fb4133b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:42.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fb9203640 10 monclient(hunting): picked mon.noname-a con 0x7f9fb4131290 addr v2:172.21.15.37:3300/0 2024-04-26T00:26:42.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fb9203640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9fb41340d0 addr v1:172.21.15.45:6789/0 2024-04-26T00:26:42.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fb9203640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fb409c190 con 0x7f9fb41340d0 2024-04-26T00:26:42.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fb9203640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:42.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fb9203640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f9fb409df60 con 0x7f9fb4131290 2024-04-26T00:26:42.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fb9203640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:42.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fb9203640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9fb40a11c0 con 0x7f9fb4130eb0 2024-04-26T00:26:42.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fb9203640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:42.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fb9203640 10 monclient(hunting): authenticate will time out at 1384.278682s 2024-04-26T00:26:42.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fb2d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9fb4130eb0 0x7f9fb409ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:42.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fb2d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9fb4130eb0 0x7f9fb409ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:58718/0 (socket says 172.21.15.37:58718) 2024-04-26T00:26:42.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fb2d9d640 1 -- 172.21.15.37:0/3551484958 learned_addr learned my addr 172.21.15.37:0/3551484958 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:26:42.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fb2d9d640 10 monclient(hunting): get_auth_request con 0x7f9fb4130eb0 auth_method 0 2024-04-26T00:26:42.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fb2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:42.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fb2d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:42.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fb2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:42.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fabfff640 1 --2- 172.21.15.37:0/3551484958 >> v2:172.21.15.37:3300/0 conn(0x7f9fb4131290 0x7f9fb4133b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:42.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fb2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:42.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fb2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:42.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fb2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:42.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fabfff640 10 monclient(hunting): get_auth_request con 0x7f9fb4131290 auth_method 0 2024-04-26T00:26:42.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:42.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fabfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:42.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fabfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:42.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fabfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:42.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fabfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:42.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fabfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:42.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fb2d9d640 10 monclient(hunting): handle_auth_done global_id 84131 payload 274 2024-04-26T00:26:42.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fb2d9d640 1 -- 172.21.15.37:0/3551484958 >> v2:172.21.15.37:3300/0 conn(0x7f9fb4131290 msgr2=0x7f9fb4133b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:26:42.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fb2d9d640 1 --2- 172.21.15.37:0/3551484958 >> v2:172.21.15.37:3300/0 conn(0x7f9fb4131290 0x7f9fb4133b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:42.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fb2d9d640 1 -- 172.21.15.37:0/3551484958 >> v1:172.21.15.45:6789/0 conn(0x7f9fb41340d0 legacy=0x7f9fb413aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:26:42.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fb2d9d640 10 monclient: _finish_hunting 0 2024-04-26T00:26:42.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fb2d9d640 1 monclient: found mon.noname-b 2024-04-26T00:26:42.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fb2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:42.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fb2d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:26:42.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.968+0000 7f9fb2d9d640 1 -- 172.21.15.37:0/3551484958 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fb409dc80 con 0x7f9fb4130eb0 2024-04-26T00:26:42.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.972+0000 7f9fb2d9d640 1 -- 172.21.15.37:0/3551484958 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9fb4130eb0 msgr2=0x7f9fb409ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:42.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.972+0000 7f9fb2d9d640 1 --2- 172.21.15.37:0/3551484958 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9fb4130eb0 0x7f9fb409ef20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9fa400bb10 tx=0x7f9fa401eab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8550c66adf0c2fc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:42.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.972+0000 7f9fb259c640 1 -- 172.21.15.37:0/3551484958 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa40b4070 con 0x7f9fb4130eb0 2024-04-26T00:26:42.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.972+0000 7f9fb259c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:42.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.972+0000 7f9fb259c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:26:42.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.972+0000 7f9fb259c640 10 monclient: dump: 2024-04-26T00:26:42.962 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:42.962 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:42.962 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.962 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.962 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:42.962 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:42.962 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:42.962 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:42.962 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:42.962 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:42.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.972+0000 7f9fb259c640 10 monclient: _finish_auth 0 2024-04-26T00:26:42.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.972+0000 7f9fb259c640 10 monclient: _check_auth_tickets 2024-04-26T00:26:42.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.972+0000 7f9fb259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:42.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.972+0000 7f9fb259c640 1 -- 172.21.15.37:0/3551484958 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fa40a7950 con 0x7f9fb4130eb0 2024-04-26T00:26:42.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.972+0000 7f9fb259c640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:42.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.972+0000 7f9fb259c640 1 -- 172.21.15.37:0/3551484958 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa40a7d20 con 0x7f9fb4130eb0 2024-04-26T00:26:42.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.972+0000 7f9fb9203640 5 monclient: authenticate success, global_id 84131 2024-04-26T00:26:42.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.972+0000 7f9fb9203640 10 monclient: get_monmap_and_config success 2024-04-26T00:26:42.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.972+0000 7f9fb9203640 4 set_mon_vals no callback set 2024-04-26T00:26:42.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.972+0000 7f9fb259c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:42.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.972+0000 7f9fb259c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:26:42.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.972+0000 7f9fb259c640 10 monclient: dump: 2024-04-26T00:26:42.963 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:42.963 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:42.963 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.963 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.963 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:42.964 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:42.964 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:42.964 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:42.964 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:42.964 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:42.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb9203640 10 monclient: shutdown 2024-04-26T00:26:42.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb9203640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:42.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb9203640 1 -- 172.21.15.37:0/3551484958 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9fb4130eb0 msgr2=0x7f9fb409ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:42.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb9203640 1 --2- 172.21.15.37:0/3551484958 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9fb4130eb0 0x7f9fb409ef20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9fa400bb10 tx=0x7f9fa401eab0 comp rx=0 tx=0).stop 2024-04-26T00:26:42.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb9203640 1 -- 172.21.15.37:0/3551484958 shutdown_connections 2024-04-26T00:26:42.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb9203640 1 --2- 172.21.15.37:0/3551484958 >> v2:172.21.15.37:3300/0 conn(0x7f9fb4131290 0x7f9fb4133b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:42.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb9203640 1 --2- 172.21.15.37:0/3551484958 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9fb4130eb0 0x7f9fb409ef20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:42.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb9203640 1 -- 172.21.15.37:0/3551484958 >> 172.21.15.37:0/3551484958 conn(0x7f9fb40a1d50 msgr2=0x7f9fb40a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:42.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb9203640 1 -- 172.21.15.37:0/3551484958 shutdown_connections 2024-04-26T00:26:42.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb9203640 1 -- 172.21.15.37:0/3551484958 wait complete. 2024-04-26T00:26:42.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb9203640 10 monclient: build_initial_monmap 2024-04-26T00:26:42.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb9203640 10 monclient: monmap: 2024-04-26T00:26:42.965 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:42.965 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:26:42.965 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:26:42.965 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:26:42.965 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:42.965 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:26:42.965 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:26:42.965 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:26:42.965 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:26:42.965 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:42.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb9203640 1 Processor -- start 2024-04-26T00:26:42.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb9203640 1 -- start start 2024-04-26T00:26:42.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb9203640 10 monclient: init 2024-04-26T00:26:42.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb9203640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:42.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb9203640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:26:42.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb9203640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9fb4130eb0 0x7f9fb411b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:42.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb9203640 10 monclient(hunting): picked mon.noname-a con 0x7f9fb4130eb0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:26:42.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb9203640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fb4131290 0x7f9fb4139d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:42.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb9203640 10 monclient(hunting): picked mon.noname-b con 0x7f9fb4131290 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:26:42.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb9203640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9fb41340d0 0x7f9fb41cf5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:42.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb9203640 10 monclient(hunting): picked mon.noname-c con 0x7f9fb41340d0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:26:42.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb9203640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:42.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb9203640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb41317f0 con 0x7f9fb4130eb0 2024-04-26T00:26:42.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb9203640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:42.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb9203640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb4131670 con 0x7f9fb4131290 2024-04-26T00:26:42.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb9203640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:42.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb9203640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9fb40a0460 con 0x7f9fb41340d0 2024-04-26T00:26:42.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb9203640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:42.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb9203640 10 monclient(hunting): authenticate will time out at 1384.286081s 2024-04-26T00:26:42.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb2d9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9fb4130eb0 0x7f9fb411b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:42.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb359e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9fb41340d0 0x7f9fb41cf5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:42.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb2d9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9fb4130eb0 0x7f9fb411b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50122/0 (socket says 172.21.15.37:50122) 2024-04-26T00:26:42.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb359e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9fb41340d0 0x7f9fb41cf5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:58722/0 (socket says 172.21.15.37:58722) 2024-04-26T00:26:42.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb2d9d640 1 -- 172.21.15.37:0/4017338247 learned_addr learned my addr 172.21.15.37:0/4017338247 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:26:42.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb2d9d640 10 monclient(hunting): get_auth_request con 0x7f9fb4130eb0 auth_method 0 2024-04-26T00:26:42.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:42.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb2d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:42.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:42.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb359e640 10 monclient(hunting): get_auth_request con 0x7f9fb41340d0 auth_method 0 2024-04-26T00:26:42.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:42.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb359e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:42.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb359e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:42.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:42.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:42.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:42.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:42.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:42.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:42.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fabfff640 1 --2- 172.21.15.37:0/4017338247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fb4131290 0x7f9fb4139d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:42.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fabfff640 10 monclient(hunting): get_auth_request con 0x7f9fb4131290 auth_method 0 2024-04-26T00:26:42.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:42.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fabfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:42.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fabfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:42.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb359e640 10 monclient(hunting): handle_auth_done global_id 84134 payload 824 2024-04-26T00:26:42.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb359e640 1 -- 172.21.15.37:0/4017338247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fb4131290 msgr2=0x7f9fb4139d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:42.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb359e640 1 --2- 172.21.15.37:0/4017338247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fb4131290 0x7f9fb4139d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:42.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb359e640 1 -- 172.21.15.37:0/4017338247 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9fb4130eb0 msgr2=0x7f9fb411b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:42.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb359e640 1 --2- 172.21.15.37:0/4017338247 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9fb4130eb0 0x7f9fb411b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:42.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb359e640 10 monclient: _finish_hunting 0 2024-04-26T00:26:42.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb359e640 1 monclient: found mon.noname-c 2024-04-26T00:26:42.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:42.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb359e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:26:42.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb359e640 1 -- 172.21.15.37:0/4017338247 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fb413c1b0 con 0x7f9fb41340d0 2024-04-26T00:26:42.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.976+0000 7f9fb2d9d640 1 --2- 172.21.15.37:0/4017338247 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9fb4130eb0 0x7f9fb411b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:26:42.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fb359e640 1 -- 172.21.15.37:0/4017338247 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9fb41340d0 msgr2=0x7f9fb41cf5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:42.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fb359e640 1 --2- 172.21.15.37:0/4017338247 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9fb41340d0 0x7f9fb41cf5a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9fac00c7b0 tx=0x7f9fac00cc70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:42.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fab7fe640 1 -- 172.21.15.37:0/4017338247 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fac016020 con 0x7f9fb41340d0 2024-04-26T00:26:42.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:42.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fab7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:26:42.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fab7fe640 10 monclient: dump: 2024-04-26T00:26:42.974 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:42.974 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:42.974 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.974 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.974 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:42.974 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:42.974 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:42.974 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:42.974 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:42.975 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:42.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fab7fe640 10 monclient: _finish_auth 0 2024-04-26T00:26:42.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fab7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:26:42.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:42.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fb9203640 5 monclient: authenticate success, global_id 84134 2024-04-26T00:26:42.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fab7fe640 1 -- 172.21.15.37:0/4017338247 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fac00ac60 con 0x7f9fb41340d0 2024-04-26T00:26:42.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fb9203640 10 monclient: _renew_subs 2024-04-26T00:26:42.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fb9203640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:26:42.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fb9203640 1 -- 172.21.15.37:0/4017338247 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fb413a890 con 0x7f9fb41340d0 2024-04-26T00:26:42.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fab7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:42.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fab7fe640 1 -- 172.21.15.37:0/4017338247 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fac00ce60 con 0x7f9fb41340d0 2024-04-26T00:26:42.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:42.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:26:42.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fab7fe640 10 monclient: dump: 2024-04-26T00:26:42.976 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:42.976 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:42.976 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.976 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.976 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:42.976 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:42.976 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:42.976 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:42.976 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:42.976 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:42.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fb1d9b640 4 set_mon_vals no callback set 2024-04-26T00:26:42.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fb9203640 10 monclient: _renew_subs 2024-04-26T00:26:42.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fb9203640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:26:42.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fb9203640 1 -- 172.21.15.37:0/4017338247 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fb411c4a0 con 0x7f9fb41340d0 2024-04-26T00:26:42.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fb9203640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:42.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fb9203640 1 --2- 172.21.15.37:0/4017338247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fb41cfae0 0x7f9fb41cfec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:42.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fb9203640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:42.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fb9203640 1 -- 172.21.15.37:0/4017338247 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb411bf70 con 0x7f9fb41cfae0 2024-04-26T00:26:42.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fab7fe640 1 -- 172.21.15.37:0/4017338247 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9fac019020 con 0x7f9fb41340d0 2024-04-26T00:26:42.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fb2d9d640 1 --2- 172.21.15.37:0/4017338247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fb41cfae0 0x7f9fb41cfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:42.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fab7fe640 1 --2- 172.21.15.37:0/4017338247 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9f94035e80 0x7f9f94038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:42.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fb2d9d640 10 monclient: get_auth_request con 0x7f9fb41cfae0 auth_method 0 2024-04-26T00:26:42.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fb2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:42.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fb2d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:42.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fb2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:42.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fab7fe640 1 -- 172.21.15.37:0/4017338247 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(264..264 src has 177..264) ==== 3979+0+0 (secure 0 0 0) 0x7f9fac048a10 con 0x7f9fb41340d0 2024-04-26T00:26:42.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fb2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:42.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fb2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:42.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fb2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:42.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fabfff640 1 --2- 172.21.15.37:0/4017338247 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9f94035e80 0x7f9f94038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:42.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fabfff640 10 monclient: get_auth_request con 0x7f9f94035e80 auth_method 0 2024-04-26T00:26:42.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fb2d9d640 10 monclient(hunting): handle_auth_done global_id 24634 payload 824 2024-04-26T00:26:42.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.980+0000 7f9fb2d9d640 1 -- 172.21.15.37:0/4017338247 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9fb41d0400 con 0x7f9fb41cfae0 2024-04-26T00:26:42.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.984+0000 7f9fabfff640 1 --2- 172.21.15.37:0/4017338247 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9f94035e80 0x7f9f94038340 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c009af0 tx=0x7f9f9c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:42.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.984+0000 7f9fb2d9d640 1 -- 172.21.15.37:0/4017338247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fb41cfae0 msgr2=0x7f9fb41cfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:42.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.984+0000 7f9fb2d9d640 1 --2- 172.21.15.37:0/4017338247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fb41cfae0 0x7f9fb41cfec0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9fa401d160 tx=0x7f9fa401d2f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:42.979 INFO:tasks.rados.rados.0.smithi045.stdout:1106: writing smithi04526579-47 from 591712 to 1378358 tid 1 2024-04-26T00:26:42.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.984+0000 7f9fab7fe640 1 -- 172.21.15.37:0/4017338247 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa40b4040 con 0x7f9fb41cfae0 2024-04-26T00:26:42.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.984+0000 7f9fab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:42.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.984+0000 7f9fab7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:26:42.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.984+0000 7f9fab7fe640 10 monclient: dump: 2024-04-26T00:26:42.980 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:42.980 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:42.980 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.980 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.980 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:42.980 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:42.981 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:42.981 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:42.981 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:42.981 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:42.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.984+0000 7f9fab7fe640 1 -- 172.21.15.37:0/4017338247 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9fa40af710 con 0x7f9fb41cfae0 2024-04-26T00:26:42.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.984+0000 7f9fab7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:42.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.984+0000 7f9fab7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:26:42.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.984+0000 7f9fab7fe640 1 -- 172.21.15.37:0/4017338247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fb41cfae0 msgr2=0x7f9fb41cfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:42.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.984+0000 7f9fab7fe640 1 --2- 172.21.15.37:0/4017338247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fb41cfae0 0x7f9fb41cfec0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9fa401d160 tx=0x7f9fa401d2f0 comp rx=0 tx=0).stop 2024-04-26T00:26:42.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.984+0000 7f9fab7fe640 1 -- 172.21.15.37:0/4017338247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fb41cfae0 msgr2=0x7f9fb41cfec0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:26:42.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.984+0000 7f9fab7fe640 1 --2- 172.21.15.37:0/4017338247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fb41cfae0 0x7f9fb41cfec0 secure :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9fa401d160 tx=0x7f9fa401d2f0 comp rx=0 tx=0).stop 2024-04-26T00:26:42.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.992+0000 7f9fb9203640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:26:42.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.992+0000 7f9fb9203640 1 --2- 172.21.15.37:0/4017338247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fb406db60 0x7f9fb406df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:42.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.992+0000 7f9fb9203640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:42.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.992+0000 7f9fb9203640 1 -- 172.21.15.37:0/4017338247 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb404f510 con 0x7f9fb406db60 2024-04-26T00:26:42.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.992+0000 7f9fb2d9d640 1 --2- 172.21.15.37:0/4017338247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fb406db60 0x7f9fb406df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:42.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.992+0000 7f9fb2d9d640 10 monclient: get_auth_request con 0x7f9fb406db60 auth_method 0 2024-04-26T00:26:42.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.992+0000 7f9fb2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:42.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.992+0000 7f9fb2d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:42.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.992+0000 7f9fb2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:42.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.992+0000 7f9fb2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:42.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.992+0000 7f9fb2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:42.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.992+0000 7f9fb2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:42.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.992+0000 7f9fb2d9d640 10 monclient(hunting): handle_auth_done global_id 24637 payload 824 2024-04-26T00:26:42.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.992+0000 7f9fb2d9d640 1 -- 172.21.15.37:0/4017338247 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9fb40a0f60 con 0x7f9fb406db60 2024-04-26T00:26:42.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.996+0000 7f9fb2d9d640 1 -- 172.21.15.37:0/4017338247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fb406db60 msgr2=0x7f9fb406df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:42.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.996+0000 7f9fb2d9d640 1 --2- 172.21.15.37:0/4017338247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fb406db60 0x7f9fb406df70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9fa401eaf0 tx=0x7f9fa401b850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:42.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.996+0000 7f9fab7fe640 1 -- 172.21.15.37:0/4017338247 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa40c2b30 con 0x7f9fb406db60 2024-04-26T00:26:42.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.996+0000 7f9fab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:42.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.996+0000 7f9fab7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:26:42.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.996+0000 7f9fab7fe640 10 monclient: dump: 2024-04-26T00:26:42.984 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:42.984 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:42.984 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.984 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:42.984 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:42.984 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:42.984 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:42.984 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:42.984 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:42.984 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:42.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.996+0000 7f9fab7fe640 1 -- 172.21.15.37:0/4017338247 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4719 (secure 0 0 0) 0x7f9fa40c2b30 con 0x7f9fb406db60 2024-04-26T00:26:42.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.996+0000 7f9fab7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:26:42.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.996+0000 7f9fab7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:26:42.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.996+0000 7f9fab7fe640 1 -- 172.21.15.37:0/4017338247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fb406db60 msgr2=0x7f9fb406df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:42.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.996+0000 7f9fab7fe640 1 --2- 172.21.15.37:0/4017338247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fb406db60 0x7f9fb406df70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9fa401eaf0 tx=0x7f9fa401b850 comp rx=0 tx=0).stop 2024-04-26T00:26:42.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.996+0000 7f9fab7fe640 1 -- 172.21.15.37:0/4017338247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fb406db60 msgr2=0x7f9fb406df70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:26:42.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.996+0000 7f9fab7fe640 1 --2- 172.21.15.37:0/4017338247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9fb406db60 0x7f9fb406df70 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:42.985 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:26:42.985 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:26:42.985 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:26:42.985 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:26:42.985 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 52, 2024-04-26T00:26:42.985 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 311163, 2024-04-26T00:26:42.985 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:26:42.985 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:26:42.985 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:26:42.986 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:26:42.986 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:42.986 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 6, 2024-04-26T00:26:42.986 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:26:42.986 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:26:42.986 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:26:42.986 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:26:42.986 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:26:42.986 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:26:42.986 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:26:42.986 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:26:42.986 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:26:42.986 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:26:42.987 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:26:42.987 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:26:42.987 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:26:42.987 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:26:42.987 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:42.987 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:26:42.987 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:26:42.987 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:26:42.987 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:26:42.987 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:26:42.987 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:26:42.987 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:26:42.987 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:26:42.987 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:26:42.988 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:26:42.988 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:26:42.988 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:26:42.988 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:26:42.988 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:42.988 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:42.988 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:26:42.988 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:26:42.988 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:26:42.988 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:26:42.988 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:26:42.988 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:26:42.988 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:26:42.988 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:26:42.988 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:26:42.989 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:26:42.989 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:26:42.989 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:26:42.989 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:26:42.989 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:26:42.989 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:26:42.989 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:26:42.989 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:26:42.989 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:26:42.989 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:26:42.989 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:26:42.989 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:26:42.989 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:26:42.989 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:26:42.989 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:26:42.990 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:26:42.990 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:26:42.990 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:26:42.990 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:26:42.990 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:42.990 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:26:42.990 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:42.990 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:26:42.990 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:42.990 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:26:42.990 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:26:42.990 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:26:42.990 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:26:42.990 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:42.991 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:26:42.991 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:26:42.991 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:42.991 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:42.991 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:42.991 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:26:42.991 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:26:42.991 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:42.991 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:42.991 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:42.991 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:42.991 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:26:42.991 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:26:42.991 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:26:42.991 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:26:42.992 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:26:42.992 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:42.992 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:42.992 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:26:42.992 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:26:42.992 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:26:42.992 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:26:42.992 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:42.992 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:26:42.992 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:26:42.992 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:42.992 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:42.992 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:42.992 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:26:42.993 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:26:42.993 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:42.993 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:42.993 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:42.993 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:42.993 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:26:42.993 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:26:42.993 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:26:42.993 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:26:42.993 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:26:42.993 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:42.993 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:42.993 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:26:42.993 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:26:42.993 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:26:42.994 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:26:42.994 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:42.994 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:26:42.994 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:26:42.994 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:42.994 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:42.994 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:42.994 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:26:42.994 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:26:42.994 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:42.994 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:42.994 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:42.994 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:42.994 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:26:42.994 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:26:42.995 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:26:42.995 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:26:42.995 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:26:42.995 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:42.995 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:42.995 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:42.995 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:26:42.995 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:26:42.995 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:42.995 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:26:42.995 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:26:42.995 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:26:42.995 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:42.995 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:42.996 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:26:42.996 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:42.996 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:26:42.996 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:26:42.996 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:26:42.996 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:42.996 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:42.996 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:26:42.996 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:42.996 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:26:42.996 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:26:42.996 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:26:42.996 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:42.996 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:42.996 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:42.997 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:26:42.997 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:26:42.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.996+0000 7f9fb9203640 1 -- 172.21.15.37:0/4017338247 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9f94035e80 msgr2=0x7f9f94038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:42.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.996+0000 7f9fb9203640 1 --2- 172.21.15.37:0/4017338247 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9f94035e80 0x7f9f94038340 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c009af0 tx=0x7f9f9c008040 comp rx=0 tx=0).stop 2024-04-26T00:26:42.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.996+0000 7f9fb9203640 10 monclient: shutdown 2024-04-26T00:26:42.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.996+0000 7f9fb9203640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:42.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.996+0000 7f9fb9203640 1 -- 172.21.15.37:0/4017338247 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9fb41340d0 msgr2=0x7f9fb41cf5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:42.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.996+0000 7f9fb9203640 1 --2- 172.21.15.37:0/4017338247 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9fb41340d0 0x7f9fb41cf5a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9fac00c7b0 tx=0x7f9fac00cc70 comp rx=0 tx=0).stop 2024-04-26T00:26:42.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.996+0000 7f9fb359e640 1 -- 172.21.15.37:0/4017338247 reap_dead start 2024-04-26T00:26:42.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.996+0000 7f9fb359e640 1 -- 172.21.15.37:0/4017338247 reap_dead start 2024-04-26T00:26:42.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.996+0000 7f9fb9203640 1 -- 172.21.15.37:0/4017338247 shutdown_connections 2024-04-26T00:26:42.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.996+0000 7f9fb9203640 1 -- 172.21.15.37:0/4017338247 >> 172.21.15.37:0/4017338247 conn(0x7f9fb40a1d50 msgr2=0x7f9fb409cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:42.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.996+0000 7f9fb9203640 1 -- 172.21.15.37:0/4017338247 shutdown_connections 2024-04-26T00:26:42.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:42.996+0000 7f9fb9203640 1 -- 172.21.15.37:0/4017338247 wait complete. 2024-04-26T00:26:42.999 INFO:tasks.rados.rados.0.smithi045.stdout:1106: writing smithi04526579-47 from 1943568 to 2612366 tid 2 2024-04-26T00:26:42.999 INFO:tasks.rados.rados.0.smithi045.stdout:1107: delete oid 1 current snap is 130 2024-04-26T00:26:42.999 INFO:tasks.rados.rados.0.smithi045.stdout:1105: finishing write tid 3 to smithi04526579-34 2024-04-26T00:26:42.999 INFO:tasks.rados.rados.0.smithi045.stdout:1105: finishing write tid 4 to smithi04526579-34 2024-04-26T00:26:43.000 INFO:tasks.rados.rados.0.smithi045.stdout:1105: finishing write tid 5 to smithi04526579-34 2024-04-26T00:26:43.000 INFO:tasks.rados.rados.0.smithi045.stdout:1105: oid 34 updating version 0 to 634 2024-04-26T00:26:43.000 INFO:tasks.rados.rados.0.smithi045.stdout:1105: oid 34 updating version 634 to 637 2024-04-26T00:26:43.000 INFO:tasks.rados.rados.0.smithi045.stdout:1105: oid 34 version 637 is already newer than 636 2024-04-26T00:26:43.000 INFO:tasks.rados.rados.0.smithi045.stdout:1105: oid 34 version 637 is already newer than 635 2024-04-26T00:26:43.000 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 637 (ObjNum 415 snap 130 seq_num 415) dirty exists 2024-04-26T00:26:43.000 INFO:tasks.rados.rados.0.smithi045.stdout:1105: left oid 34 (ObjNum 415 snap 130 seq_num 415) 2024-04-26T00:26:43.000 INFO:tasks.rados.rados.0.smithi045.stdout:1106: finishing write tid 1 to smithi04526579-47 2024-04-26T00:26:43.000 INFO:tasks.rados.rados.0.smithi045.stdout:1106: finishing write tid 2 to smithi04526579-47 2024-04-26T00:26:43.000 INFO:tasks.rados.rados.0.smithi045.stdout:1106: finishing write tid 3 to smithi04526579-47 2024-04-26T00:26:43.000 INFO:tasks.rados.rados.0.smithi045.stdout:1106: finishing write tid 4 to smithi04526579-47 2024-04-26T00:26:43.000 INFO:tasks.rados.rados.0.smithi045.stdout:1106: oid 47 updating version 0 to 640 2024-04-26T00:26:43.001 INFO:tasks.rados.rados.0.smithi045.stdout:1106: oid 47 version 640 is already newer than 639 2024-04-26T00:26:43.001 INFO:tasks.rados.rados.0.smithi045.stdout:1106: oid 47 updating version 640 to 641 2024-04-26T00:26:43.001 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 641 (ObjNum 416 snap 130 seq_num 416) dirty exists 2024-04-26T00:26:43.001 INFO:tasks.rados.rados.0.smithi045.stdout:1106: left oid 47 (ObjNum 416 snap 130 seq_num 416) 2024-04-26T00:26:43.002 INFO:tasks.rados.rados.0.smithi045.stdout:1105: done (2 left) 2024-04-26T00:26:43.002 INFO:tasks.rados.rados.0.smithi045.stdout:1106: done (1 left) 2024-04-26T00:26:43.002 INFO:tasks.rados.rados.0.smithi045.stdout:1107: done (0 left) 2024-04-26T00:26:43.002 INFO:tasks.rados.rados.0.smithi045.stdout:1108: write (excl) oid 42 current snap is 130 2024-04-26T00:26:43.002 INFO:tasks.rados.rados.0.smithi045.stdout:1108: seq_num 418 ranges {625326=537144,1944179=672467} 2024-04-26T00:26:43.002 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:26:43.003 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-04-26T00:26:43.003 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-26T00:26:43.003 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-04-26T00:26:43.003 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-04-26T00:26:43.004 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-26T00:26:43.004 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T00:26:43.015 INFO:tasks.rados.rados.0.smithi045.stdout:1108: writing smithi04526579-42 from 625326 to 1162470 tid 1 2024-04-26T00:26:43.016 INFO:tasks.ceph.mon.a:Stopped 2024-04-26T00:26:43.016 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-26T00:26:43.016 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:26:43.033 INFO:tasks.rados.rados.0.smithi045.stdout:1108: writing smithi04526579-42 from 1944179 to 2616646 tid 2 2024-04-26T00:26:43.033 INFO:tasks.rados.rados.0.smithi045.stdout:1109: snap_create 2024-04-26T00:26:43.033 INFO:tasks.rados.rados.0.smithi045.stdout:1108: finishing write tid 1 to smithi04526579-42 2024-04-26T00:26:43.037 INFO:tasks.rados.rados.0.smithi045.stdout:1108: finishing write tid 2 to smithi04526579-42 2024-04-26T00:26:43.037 INFO:tasks.rados.rados.0.smithi045.stdout:1108: finishing write tid 3 to smithi04526579-42 2024-04-26T00:26:43.038 INFO:tasks.rados.rados.0.smithi045.stdout:1108: finishing write tid 4 to smithi04526579-42 2024-04-26T00:26:43.038 INFO:tasks.rados.rados.0.smithi045.stdout:1108: oid 42 updating version 0 to 566 2024-04-26T00:26:43.038 INFO:tasks.rados.rados.0.smithi045.stdout:1108: oid 42 version 566 is already newer than 565 2024-04-26T00:26:43.038 INFO:tasks.rados.rados.0.smithi045.stdout:1108: oid 42 version 566 is already newer than 564 2024-04-26T00:26:43.038 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 566 (ObjNum 418 snap 130 seq_num 418) dirty exists 2024-04-26T00:26:43.038 INFO:tasks.rados.rados.0.smithi045.stdout:1108: left oid 42 (ObjNum 418 snap 130 seq_num 418) 2024-04-26T00:26:43.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b390fa640 10 monclient: get_monmap_and_config 2024-04-26T00:26:43.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b390fa640 10 monclient: build_initial_monmap 2024-04-26T00:26:43.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b390fa640 10 monclient: monmap: 2024-04-26T00:26:43.182 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:43.182 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:43.182 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:26:43.201075+0000 2024-04-26T00:26:43.182 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:26:43.201075+0000 2024-04-26T00:26:43.182 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:43.182 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:43.182 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:26:43.182 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:26:43.182 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:26:43.182 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:26:43.182 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:26:43.182 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:43.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b390fa640 1 Processor -- start 2024-04-26T00:26:43.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b390fa640 1 -- start start 2024-04-26T00:26:43.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b390fa640 10 monclient: init 2024-04-26T00:26:43.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b390fa640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:43.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b390fa640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-04-26T00:26:43.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b390fa640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2b340a0260 0x7f2b34072e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:43.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b390fa640 10 monclient(hunting): picked mon.noname-b con 0x7f2b340a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:26:43.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b390fa640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2b340a0640 0x7f2b340a0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:43.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b390fa640 10 monclient(hunting): picked mon.noname-c con 0x7f2b340a0640 addr v2:172.21.15.45:3300/0 2024-04-26T00:26:43.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b390fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2b341337a0 addr v1:172.21.15.37:6789/0 2024-04-26T00:26:43.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b390fa640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b340a12e0 con 0x7f2b341337a0 2024-04-26T00:26:43.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b390fa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:43.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b390fa640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f2b3406d050 con 0x7f2b340a0640 2024-04-26T00:26:43.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b390fa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:43.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b390fa640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2b3409dcb0 con 0x7f2b340a0260 2024-04-26T00:26:43.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b390fa640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:43.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b390fa640 10 monclient(hunting): authenticate will time out at 1384.506560s 2024-04-26T00:26:43.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b3359e640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f2b341337a0 legacy=0x7f2b34133c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:26:43.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b32d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2b340a0260 0x7f2b34072e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:43.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b32d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2b340a0260 0x7f2b34072e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:58762/0 (socket says 172.21.15.37:58762) 2024-04-26T00:26:43.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b32d9d640 1 -- 172.21.15.37:0/1334837682 learned_addr learned my addr 172.21.15.37:0/1334837682 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:26:43.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b32d9d640 10 monclient(hunting): get_auth_request con 0x7f2b340a0260 auth_method 0 2024-04-26T00:26:43.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b32d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:43.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b32d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:43.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b32d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:43.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b32d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:43.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b32d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:43.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b32d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:43.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b3259c640 1 --2- 172.21.15.37:0/1334837682 >> v2:172.21.15.45:3300/0 conn(0x7f2b340a0640 0x7f2b340a0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:43.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b3259c640 10 monclient(hunting): get_auth_request con 0x7f2b340a0640 auth_method 0 2024-04-26T00:26:43.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b3259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:43.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b3259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:43.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b3259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:43.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b32d9d640 10 monclient(hunting): handle_auth_done global_id 84137 payload 274 2024-04-26T00:26:43.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b32d9d640 1 -- 172.21.15.37:0/1334837682 >> v2:172.21.15.45:3300/0 conn(0x7f2b340a0640 msgr2=0x7f2b340a0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:26:43.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b32d9d640 1 --2- 172.21.15.37:0/1334837682 >> v2:172.21.15.45:3300/0 conn(0x7f2b340a0640 0x7f2b340a0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:43.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b32d9d640 1 -- 172.21.15.37:0/1334837682 >> v1:172.21.15.37:6789/0 conn(0x7f2b341337a0 legacy=0x7f2b34133c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:26:43.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b32d9d640 10 monclient: _finish_hunting 0 2024-04-26T00:26:43.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b32d9d640 1 monclient: found mon.noname-b 2024-04-26T00:26:43.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b32d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:43.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b32d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:26:43.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.196+0000 7f2b32d9d640 1 -- 172.21.15.37:0/1334837682 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b340a16f0 con 0x7f2b340a0260 2024-04-26T00:26:43.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.200+0000 7f2b32d9d640 1 -- 172.21.15.37:0/1334837682 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2b340a0260 msgr2=0x7f2b34072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:43.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.200+0000 7f2b32d9d640 1 --2- 172.21.15.37:0/1334837682 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2b340a0260 0x7f2b34072e10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2b24006fd0 tx=0x7f2b2401e900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a397c747f4df3ba server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:43.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.200+0000 7f2b31d9b640 1 -- 172.21.15.37:0/1334837682 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b240ad070 con 0x7f2b340a0260 2024-04-26T00:26:43.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.200+0000 7f2b31d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:43.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.200+0000 7f2b31d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:26:43.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.200+0000 7f2b31d9b640 10 monclient: dump: 2024-04-26T00:26:43.190 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:43.190 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:43.190 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:43.190 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:43.190 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:43.190 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:43.190 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:43.190 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:43.190 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:43.190 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:43.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.200+0000 7f2b31d9b640 10 monclient: _finish_auth 0 2024-04-26T00:26:43.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.200+0000 7f2b31d9b640 10 monclient: _check_auth_tickets 2024-04-26T00:26:43.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.200+0000 7f2b31d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:43.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.200+0000 7f2b31d9b640 1 -- 172.21.15.37:0/1334837682 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b24004230 con 0x7f2b340a0260 2024-04-26T00:26:43.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.200+0000 7f2b31d9b640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:43.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.200+0000 7f2b31d9b640 1 -- 172.21.15.37:0/1334837682 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b24003620 con 0x7f2b340a0260 2024-04-26T00:26:43.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.200+0000 7f2b390fa640 5 monclient: authenticate success, global_id 84137 2024-04-26T00:26:43.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.200+0000 7f2b390fa640 10 monclient: get_monmap_and_config success 2024-04-26T00:26:43.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.200+0000 7f2b390fa640 4 set_mon_vals no callback set 2024-04-26T00:26:43.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.200+0000 7f2b31d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:43.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.200+0000 7f2b31d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:26:43.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.200+0000 7f2b31d9b640 10 monclient: dump: 2024-04-26T00:26:43.191 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:43.191 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:43.191 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:43.191 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:43.191 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:43.191 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:43.191 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:43.191 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:43.192 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:43.192 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:43.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b390fa640 10 monclient: shutdown 2024-04-26T00:26:43.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b390fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:43.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b390fa640 1 -- 172.21.15.37:0/1334837682 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2b340a0260 msgr2=0x7f2b34072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:43.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b390fa640 1 --2- 172.21.15.37:0/1334837682 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2b340a0260 0x7f2b34072e10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2b24006fd0 tx=0x7f2b2401e900 comp rx=0 tx=0).stop 2024-04-26T00:26:43.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b390fa640 1 -- 172.21.15.37:0/1334837682 shutdown_connections 2024-04-26T00:26:43.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b390fa640 1 --2- 172.21.15.37:0/1334837682 >> v2:172.21.15.45:3300/0 conn(0x7f2b340a0640 0x7f2b340a0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:43.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b390fa640 1 --2- 172.21.15.37:0/1334837682 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2b340a0260 0x7f2b34072e10 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:43.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b390fa640 1 -- 172.21.15.37:0/1334837682 >> 172.21.15.37:0/1334837682 conn(0x7f2b340a1ad0 msgr2=0x7f2b340a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:43.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b390fa640 1 -- 172.21.15.37:0/1334837682 shutdown_connections 2024-04-26T00:26:43.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b390fa640 1 -- 172.21.15.37:0/1334837682 wait complete. 2024-04-26T00:26:43.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b390fa640 10 monclient: build_initial_monmap 2024-04-26T00:26:43.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b390fa640 10 monclient: monmap: 2024-04-26T00:26:43.192 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:43.193 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:26:43.193 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:26:43.193 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:26:43.193 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:43.193 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:26:43.193 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:26:43.193 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:26:43.193 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:26:43.193 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:43.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b390fa640 1 Processor -- start 2024-04-26T00:26:43.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b390fa640 1 -- start start 2024-04-26T00:26:43.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b390fa640 10 monclient: init 2024-04-26T00:26:43.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b390fa640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:43.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b390fa640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:26:43.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b390fa640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b340a0260 0x7f2b341cf050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:43.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b390fa640 10 monclient(hunting): picked mon.noname-b con 0x7f2b340a0260 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:26:43.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b390fa640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2b340a0640 0x7f2b34132980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:43.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b390fa640 10 monclient(hunting): picked mon.noname-c con 0x7f2b340a0640 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:26:43.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b390fa640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2b341337a0 0x7f2b341393b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:43.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b390fa640 10 monclient(hunting): picked mon.noname-a con 0x7f2b341337a0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:26:43.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b390fa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:43.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b390fa640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f2b3413db90 con 0x7f2b341337a0 2024-04-26T00:26:43.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b390fa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:43.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b390fa640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2b3413da10 con 0x7f2b340a0260 2024-04-26T00:26:43.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b390fa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:43.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b390fa640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2b3413dd10 con 0x7f2b340a0640 2024-04-26T00:26:43.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b390fa640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:43.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b390fa640 10 monclient(hunting): authenticate will time out at 1384.514027s 2024-04-26T00:26:43.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b3359e640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2b341337a0 msgr2=0x7f2b341393b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:26:43.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b3359e640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2b341337a0 0x7f2b341393b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:26:43.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b3259c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2b340a0640 0x7f2b34132980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:43.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b3259c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2b340a0640 0x7f2b34132980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:58764/0 (socket says 172.21.15.37:58764) 2024-04-26T00:26:43.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b3259c640 1 -- 172.21.15.37:0/1081114103 learned_addr learned my addr 172.21.15.37:0/1081114103 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:26:43.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b3259c640 10 monclient(hunting): get_auth_request con 0x7f2b340a0640 auth_method 0 2024-04-26T00:26:43.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b3259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:43.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b3259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:43.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b3259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:43.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b3259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:43.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b3259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:43.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b3259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:43.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b32d9d640 1 --2- 172.21.15.37:0/1081114103 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b340a0260 0x7f2b341cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:43.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b32d9d640 10 monclient(hunting): get_auth_request con 0x7f2b340a0260 auth_method 0 2024-04-26T00:26:43.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b32d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:43.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b32d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:43.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b32d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:43.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b3259c640 10 monclient(hunting): handle_auth_done global_id 84140 payload 824 2024-04-26T00:26:43.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b3259c640 1 -- 172.21.15.37:0/1081114103 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b340a0260 msgr2=0x7f2b341cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:43.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b3259c640 1 --2- 172.21.15.37:0/1081114103 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b340a0260 0x7f2b341cf050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:43.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b3259c640 1 -- 172.21.15.37:0/1081114103 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2b341337a0 msgr2=0x7f2b341393b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:26:43.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b3259c640 1 --2- 172.21.15.37:0/1081114103 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2b341337a0 0x7f2b341393b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:43.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b3259c640 10 monclient: _finish_hunting 0 2024-04-26T00:26:43.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b3259c640 1 monclient: found mon.noname-c 2024-04-26T00:26:43.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b3259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:43.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b3259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:26:43.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.204+0000 7f2b3259c640 1 -- 172.21.15.37:0/1081114103 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b34072270 con 0x7f2b340a0640 2024-04-26T00:26:43.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.208+0000 7f2b3259c640 1 -- 172.21.15.37:0/1081114103 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2b340a0640 msgr2=0x7f2b34132980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:43.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.208+0000 7f2b3259c640 1 --2- 172.21.15.37:0/1081114103 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2b340a0640 0x7f2b34132980 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2b2800d800 tx=0x7f2b2800dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:43.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.208+0000 7f2b1b7fe640 1 -- 172.21.15.37:0/1081114103 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b2800b8d0 con 0x7f2b340a0640 2024-04-26T00:26:43.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.208+0000 7f2b1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:43.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.208+0000 7f2b1b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:26:43.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.208+0000 7f2b1b7fe640 10 monclient: dump: 2024-04-26T00:26:43.200 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:43.200 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:43.200 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:43.200 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:43.200 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:43.200 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:43.200 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:43.200 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:43.200 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:43.200 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:43.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.208+0000 7f2b1b7fe640 10 monclient: _finish_auth 0 2024-04-26T00:26:43.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.208+0000 7f2b1b7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:26:43.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.208+0000 7f2b1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:43.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.208+0000 7f2b1b7fe640 1 -- 172.21.15.37:0/1081114103 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b28019070 con 0x7f2b340a0640 2024-04-26T00:26:43.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.208+0000 7f2b390fa640 5 monclient: authenticate success, global_id 84140 2024-04-26T00:26:43.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.208+0000 7f2b390fa640 10 monclient: _renew_subs 2024-04-26T00:26:43.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.208+0000 7f2b390fa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:26:43.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.208+0000 7f2b390fa640 1 -- 172.21.15.37:0/1081114103 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b3413b000 con 0x7f2b340a0640 2024-04-26T00:26:43.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.208+0000 7f2b1b7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:43.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.208+0000 7f2b1b7fe640 1 -- 172.21.15.37:0/1081114103 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b2800bc70 con 0x7f2b340a0640 2024-04-26T00:26:43.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.208+0000 7f2b1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:43.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.208+0000 7f2b1b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:26:43.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.208+0000 7f2b1b7fe640 10 monclient: dump: 2024-04-26T00:26:43.201 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:43.201 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:43.201 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:43.201 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:43.201 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:43.202 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:43.202 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:43.202 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:43.202 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:43.202 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:43.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.208+0000 7f2b390fa640 10 monclient: _renew_subs 2024-04-26T00:26:43.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.208+0000 7f2b390fa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:26:43.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.208+0000 7f2b390fa640 1 -- 172.21.15.37:0/1081114103 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b3411fbc0 con 0x7f2b340a0640 2024-04-26T00:26:43.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.208+0000 7f2b30d99640 4 set_mon_vals no callback set 2024-04-26T00:26:43.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.208+0000 7f2b390fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:43.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.208+0000 7f2b390fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:43.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.208+0000 7f2b390fa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:26:43.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.208+0000 7f2b1b7fe640 1 -- 172.21.15.37:0/1081114103 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2b2801e460 con 0x7f2b340a0640 2024-04-26T00:26:43.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.208+0000 7f2b390fa640 1 -- 172.21.15.37:0/1081114103 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b34066f40 con 0x7f2b340a0640 2024-04-26T00:26:43.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.208+0000 7f2b1b7fe640 1 --2- 172.21.15.37:0/1081114103 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2b20035e30 0x7f2b200382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:43.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.208+0000 7f2b1b7fe640 1 -- 172.21.15.37:0/1081114103 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(264..264 src has 177..264) ==== 3979+0+0 (secure 0 0 0) 0x7f2b28048ff0 con 0x7f2b340a0640 2024-04-26T00:26:43.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.208+0000 7f2b32d9d640 1 --2- 172.21.15.37:0/1081114103 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2b20035e30 0x7f2b200382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:43.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.208+0000 7f2b32d9d640 10 monclient: get_auth_request con 0x7f2b20035e30 auth_method 0 2024-04-26T00:26:43.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.212+0000 7f2b1b7fe640 1 -- 172.21.15.37:0/1081114103 DispatchQueue::entry inject delay of 0.002000 2024-04-26T00:26:43.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.212+0000 7f2b32d9d640 1 --2- 172.21.15.37:0/1081114103 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2b20035e30 0x7f2b200382f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2b24004000 tx=0x7f2b240a9070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:43.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.212+0000 7f2b1b7fe640 1 -- 172.21.15.37:0/1081114103 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f2b2800be10 con 0x7f2b340a0640 2024-04-26T00:26:43.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.212+0000 7f2b1b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:43.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.212+0000 7f2b1b7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:26:43.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.436+0000 7f2b390fa640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:26:43.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.436+0000 7f2b390fa640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:26:43.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.436+0000 7f2b390fa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:26:43.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.436+0000 7f2b390fa640 1 -- 172.21.15.37:0/1081114103 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2b341cfed0 con 0x7f2b340a0640 2024-04-26T00:26:43.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.436+0000 7f2b1b7fe640 1 -- 172.21.15.37:0/1081114103 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f2b2801d580 con 0x7f2b340a0640 2024-04-26T00:26:43.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.436+0000 7f2b1b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:26:43.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.436+0000 7f2b1b7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:26:43.424 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":52,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:26:43.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.440+0000 7f2b390fa640 1 -- 172.21.15.37:0/1081114103 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2b20035e30 msgr2=0x7f2b200382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:43.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.440+0000 7f2b390fa640 1 --2- 172.21.15.37:0/1081114103 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2b20035e30 0x7f2b200382f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2b24004000 tx=0x7f2b240a9070 comp rx=0 tx=0).stop 2024-04-26T00:26:43.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.440+0000 7f2b390fa640 10 monclient: shutdown 2024-04-26T00:26:43.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.440+0000 7f2b390fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:43.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.440+0000 7f2b390fa640 1 -- 172.21.15.37:0/1081114103 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2b340a0640 msgr2=0x7f2b34132980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:43.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.440+0000 7f2b390fa640 1 --2- 172.21.15.37:0/1081114103 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2b340a0640 0x7f2b34132980 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2b2800d800 tx=0x7f2b2800dcc0 comp rx=0 tx=0).stop 2024-04-26T00:26:43.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.440+0000 7f2b390fa640 1 -- 172.21.15.37:0/1081114103 shutdown_connections 2024-04-26T00:26:43.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.440+0000 7f2b390fa640 1 --2- 172.21.15.37:0/1081114103 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2b20035e30 0x7f2b200382f0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:43.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.440+0000 7f2b390fa640 1 --2- 172.21.15.37:0/1081114103 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2b341337a0 0x7f2b341393b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:43.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.440+0000 7f2b390fa640 1 --2- 172.21.15.37:0/1081114103 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2b340a0640 0x7f2b34132980 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:43.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.440+0000 7f2b390fa640 1 --2- 172.21.15.37:0/1081114103 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b340a0260 0x7f2b341cf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:43.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.440+0000 7f2b390fa640 1 -- 172.21.15.37:0/1081114103 >> 172.21.15.37:0/1081114103 conn(0x7f2b340a1ad0 msgr2=0x7f2b34070890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:43.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.440+0000 7f2b390fa640 1 -- 172.21.15.37:0/1081114103 shutdown_connections 2024-04-26T00:26:43.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:43.440+0000 7f2b390fa640 1 -- 172.21.15.37:0/1081114103 wait complete. 2024-04-26T00:26:46.444 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:26:46.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.660+0000 7f01459d0640 10 monclient: get_monmap_and_config 2024-04-26T00:26:46.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.660+0000 7f01459d0640 10 monclient: build_initial_monmap 2024-04-26T00:26:46.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.660+0000 7f01459d0640 10 monclient: monmap: 2024-04-26T00:26:46.646 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:46.646 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:46.646 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:26:46.665647+0000 2024-04-26T00:26:46.647 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:26:46.665647+0000 2024-04-26T00:26:46.647 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:46.647 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:46.647 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:26:46.647 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:26:46.647 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:26:46.647 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:26:46.647 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:26:46.647 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:46.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.660+0000 7f01459d0640 1 Processor -- start 2024-04-26T00:26:46.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.660+0000 7f01459d0640 1 -- start start 2024-04-26T00:26:46.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.660+0000 7f01459d0640 10 monclient: init 2024-04-26T00:26:46.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.660+0000 7f01459d0640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:46.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.660+0000 7f01459d0640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-04-26T00:26:46.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.660+0000 7f01459d0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0140133b20 addr v1:172.21.15.45:6789/0 2024-04-26T00:26:46.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.660+0000 7f01459d0640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0140133f00 0x7f0140134350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:46.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.660+0000 7f01459d0640 10 monclient(hunting): picked mon.noname-b con 0x7f0140133f00 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:26:46.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.660+0000 7f01459d0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0140139050 addr v1:172.21.15.37:6789/0 2024-04-26T00:26:46.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.660+0000 7f01459d0640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f014009fe20 con 0x7f0140139050 2024-04-26T00:26:46.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.660+0000 7f01459d0640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0140072d00 con 0x7f0140133b20 2024-04-26T00:26:46.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.660+0000 7f01459d0640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:46.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.660+0000 7f01459d0640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f014009e620 con 0x7f0140133f00 2024-04-26T00:26:46.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.660+0000 7f01459d0640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:46.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.660+0000 7f01459d0640 10 monclient(hunting): authenticate will time out at 1387.971390s 2024-04-26T00:26:46.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.660+0000 7f013f7fe640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f0140139050 legacy=0x7f014013b440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:26:46.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.660+0000 7f013e7fc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0140133f00 0x7f0140134350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:46.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.660+0000 7f013e7fc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0140133f00 0x7f0140134350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:58774/0 (socket says 172.21.15.37:58774) 2024-04-26T00:26:46.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.660+0000 7f013e7fc640 1 -- 172.21.15.37:0/2638387214 learned_addr learned my addr 172.21.15.37:0/2638387214 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:26:46.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.660+0000 7f013e7fc640 10 monclient(hunting): get_auth_request con 0x7f0140133f00 auth_method 0 2024-04-26T00:26:46.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.660+0000 7f013e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:46.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.660+0000 7f013e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:46.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.660+0000 7f013e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:46.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.660+0000 7f013e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:46.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.660+0000 7f013e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:46.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.660+0000 7f013e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:46.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.660+0000 7f013e7fc640 10 monclient(hunting): handle_auth_done global_id 84143 payload 274 2024-04-26T00:26:46.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.660+0000 7f013e7fc640 1 -- 172.21.15.37:0/2638387214 >> v1:172.21.15.45:6789/0 conn(0x7f0140133b20 legacy=0x7f014009e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:26:46.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.660+0000 7f013e7fc640 1 -- 172.21.15.37:0/2638387214 >> v1:172.21.15.37:6789/0 conn(0x7f0140139050 legacy=0x7f014013b440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:26:46.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.660+0000 7f013e7fc640 10 monclient: _finish_hunting 0 2024-04-26T00:26:46.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.660+0000 7f013e7fc640 1 monclient: found mon.noname-b 2024-04-26T00:26:46.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.660+0000 7f013e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:46.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.660+0000 7f013e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:26:46.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.660+0000 7f013e7fc640 1 -- 172.21.15.37:0/2638387214 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f014009ba60 con 0x7f0140133f00 2024-04-26T00:26:46.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013e7fc640 1 -- 172.21.15.37:0/2638387214 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0140133f00 msgr2=0x7f0140134350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:46.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013e7fc640 1 --2- 172.21.15.37:0/2638387214 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0140133f00 0x7f0140134350 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f01340098a0 tx=0x7f013401c1e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3a35b4884f30d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:46.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013dffb640 1 -- 172.21.15.37:0/2638387214 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01340b0070 con 0x7f0140133f00 2024-04-26T00:26:46.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:46.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:26:46.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013dffb640 10 monclient: dump: 2024-04-26T00:26:46.654 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:46.654 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:46.654 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:46.654 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:46.654 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:46.654 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:46.654 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:46.654 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:46.654 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:46.654 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:46.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013dffb640 10 monclient: _finish_auth 0 2024-04-26T00:26:46.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013dffb640 10 monclient: _check_auth_tickets 2024-04-26T00:26:46.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:46.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013dffb640 1 -- 172.21.15.37:0/2638387214 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01340ab600 con 0x7f0140133f00 2024-04-26T00:26:46.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f01459d0640 5 monclient: authenticate success, global_id 84143 2024-04-26T00:26:46.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f01459d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:26:46.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013dffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:46.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f01459d0640 10 monclient: get_monmap_and_config success 2024-04-26T00:26:46.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f01459d0640 4 set_mon_vals no callback set 2024-04-26T00:26:46.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013dffb640 1 -- 172.21.15.37:0/2638387214 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01340ab9d0 con 0x7f0140133f00 2024-04-26T00:26:46.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:46.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:26:46.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013dffb640 10 monclient: dump: 2024-04-26T00:26:46.655 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:46.655 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:46.655 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:46.655 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:46.655 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:46.655 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:46.655 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:46.655 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:46.656 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:46.656 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:46.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f01459d0640 10 monclient: shutdown 2024-04-26T00:26:46.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f01459d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:46.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f01459d0640 1 -- 172.21.15.37:0/2638387214 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0140133f00 msgr2=0x7f0140134350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:46.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f01459d0640 1 --2- 172.21.15.37:0/2638387214 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0140133f00 0x7f0140134350 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f01340098a0 tx=0x7f013401c1e0 comp rx=0 tx=0).stop 2024-04-26T00:26:46.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f01459d0640 1 -- 172.21.15.37:0/2638387214 shutdown_connections 2024-04-26T00:26:46.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f01459d0640 1 --2- 172.21.15.37:0/2638387214 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0140133f00 0x7f0140134350 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:46.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f01459d0640 1 -- 172.21.15.37:0/2638387214 >> 172.21.15.37:0/2638387214 conn(0x7f01400a1ad0 msgr2=0x7f01400a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:46.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f01459d0640 1 -- 172.21.15.37:0/2638387214 shutdown_connections 2024-04-26T00:26:46.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f01459d0640 1 -- 172.21.15.37:0/2638387214 wait complete. 2024-04-26T00:26:46.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f01459d0640 10 monclient: build_initial_monmap 2024-04-26T00:26:46.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f01459d0640 10 monclient: monmap: 2024-04-26T00:26:46.661 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:46.661 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:26:46.661 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:26:46.661 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:26:46.661 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:46.661 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:26:46.661 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:26:46.661 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:26:46.661 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:26:46.661 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:46.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f01459d0640 1 Processor -- start 2024-04-26T00:26:46.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f01459d0640 1 -- start start 2024-04-26T00:26:46.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f01459d0640 10 monclient: init 2024-04-26T00:26:46.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f01459d0640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:46.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f01459d0640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:26:46.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f01459d0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0140133b20 0x7f0140072250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:46.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f01459d0640 10 monclient(hunting): picked mon.noname-b con 0x7f0140133b20 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:26:46.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f01459d0640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0140133f00 0x7f0140132170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:46.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f01459d0640 10 monclient(hunting): picked mon.noname-c con 0x7f0140133f00 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:26:46.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f01459d0640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0140139050 0x7f014011b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:46.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f01459d0640 10 monclient(hunting): picked mon.noname-a con 0x7f0140139050 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:26:46.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f01459d0640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:46.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f01459d0640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f014013f950 con 0x7f0140139050 2024-04-26T00:26:46.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f01459d0640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:46.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f01459d0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f014013f7d0 con 0x7f0140133b20 2024-04-26T00:26:46.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f01459d0640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:46.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f01459d0640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f014013fad0 con 0x7f0140133f00 2024-04-26T00:26:46.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f01459d0640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:46.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f01459d0640 10 monclient(hunting): authenticate will time out at 1387.978668s 2024-04-26T00:26:46.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013f7fe640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0140139050 msgr2=0x7f014011b620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:26:46.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013f7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0140139050 0x7f014011b620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:26:46.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013e7fc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0140133f00 0x7f0140132170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:46.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013e7fc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0140133f00 0x7f0140132170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:58788/0 (socket says 172.21.15.37:58788) 2024-04-26T00:26:46.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013e7fc640 1 -- 172.21.15.37:0/1648469430 learned_addr learned my addr 172.21.15.37:0/1648469430 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:26:46.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013e7fc640 10 monclient(hunting): get_auth_request con 0x7f0140133f00 auth_method 0 2024-04-26T00:26:46.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:46.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:46.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:46.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:46.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:46.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:46.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013effd640 1 --2- 172.21.15.37:0/1648469430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0140133b20 0x7f0140072250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:46.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013effd640 10 monclient(hunting): get_auth_request con 0x7f0140133b20 auth_method 0 2024-04-26T00:26:46.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:46.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:46.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:46.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013e7fc640 10 monclient(hunting): handle_auth_done global_id 84146 payload 824 2024-04-26T00:26:46.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013e7fc640 1 -- 172.21.15.37:0/1648469430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0140133b20 msgr2=0x7f0140072250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:46.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013e7fc640 1 --2- 172.21.15.37:0/1648469430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0140133b20 0x7f0140072250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:46.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013e7fc640 1 -- 172.21.15.37:0/1648469430 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0140139050 msgr2=0x7f014011b620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:26:46.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013e7fc640 1 --2- 172.21.15.37:0/1648469430 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0140139050 0x7f014011b620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:46.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013e7fc640 10 monclient: _finish_hunting 0 2024-04-26T00:26:46.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013e7fc640 1 monclient: found mon.noname-c 2024-04-26T00:26:46.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:46.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013e7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:26:46.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.668+0000 7f013e7fc640 1 -- 172.21.15.37:0/1648469430 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f014011eb20 con 0x7f0140133f00 2024-04-26T00:26:46.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.672+0000 7f013e7fc640 1 -- 172.21.15.37:0/1648469430 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0140133f00 msgr2=0x7f0140132170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:46.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.672+0000 7f013e7fc640 1 --2- 172.21.15.37:0/1648469430 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0140133f00 0x7f0140132170 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f01340098a0 tx=0x7f0134018f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:46.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.672+0000 7f013e7fc640 1 --2- 172.21.15.37:0/1648469430 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0140133f00 0x7f0140132170 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f01340098a0 tx=0x7f0134018f80 comp rx=0 tx=0).queue_received will delay after 1.71409e+09 on 0x7f01340ba070 config(0 keys) 2024-04-26T00:26:46.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.672+0000 7f0127fff640 1 -- 172.21.15.37:0/1648469430 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01340b0050 con 0x7f0140133f00 2024-04-26T00:26:46.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.672+0000 7f0127fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:46.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.672+0000 7f0127fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:26:46.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.672+0000 7f0127fff640 10 monclient: dump: 2024-04-26T00:26:46.665 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:46.665 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:46.665 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:46.665 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:46.665 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:46.665 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:46.665 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:46.666 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:46.666 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:46.666 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:46.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.672+0000 7f0127fff640 10 monclient: _finish_auth 0 2024-04-26T00:26:46.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.672+0000 7f0127fff640 10 monclient: _check_auth_tickets 2024-04-26T00:26:46.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.672+0000 7f0127fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:46.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.672+0000 7f0127fff640 1 -- 172.21.15.37:0/1648469430 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01340ba070 con 0x7f0140133f00 2024-04-26T00:26:46.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.672+0000 7f0127fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:46.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.672+0000 7f01459d0640 5 monclient: authenticate success, global_id 84146 2024-04-26T00:26:46.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.672+0000 7f01459d0640 10 monclient: _renew_subs 2024-04-26T00:26:46.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.672+0000 7f01459d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:26:46.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.672+0000 7f01459d0640 1 -- 172.21.15.37:0/1648469430 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f014011c7c0 con 0x7f0140133f00 2024-04-26T00:26:46.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.672+0000 7f01459d0640 10 monclient: _renew_subs 2024-04-26T00:26:46.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.672+0000 7f01459d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:26:46.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.672+0000 7f013d7fa640 4 set_mon_vals no callback set 2024-04-26T00:26:46.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.672+0000 7f01459d0640 1 -- 172.21.15.37:0/1648469430 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f014011d540 con 0x7f0140133f00 2024-04-26T00:26:46.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.672+0000 7f0127fff640 1 -- 172.21.15.37:0/1648469430 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01340b4a00 con 0x7f0140133f00 2024-04-26T00:26:46.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.672+0000 7f0127fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:46.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.672+0000 7f0127fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:26:46.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.672+0000 7f0127fff640 10 monclient: dump: 2024-04-26T00:26:46.667 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:46.667 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:46.667 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:46.667 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:46.667 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:46.667 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:46.667 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:46.667 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:46.667 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:46.667 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:46.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.672+0000 7f0127fff640 1 -- 172.21.15.37:0/1648469430 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f01340b4c20 con 0x7f0140133f00 2024-04-26T00:26:46.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.672+0000 7f0127fff640 1 --2- 172.21.15.37:0/1648469430 >> v2:172.21.15.45:6812/2379918623 conn(0x7f012c035e30 0x7f012c0382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:46.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.672+0000 7f01459d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:46.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.672+0000 7f01459d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:46.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.672+0000 7f01459d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:26:46.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.672+0000 7f01459d0640 1 -- 172.21.15.37:0/1648469430 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0140066f40 con 0x7f0140133f00 2024-04-26T00:26:46.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.672+0000 7f013effd640 1 --2- 172.21.15.37:0/1648469430 >> v2:172.21.15.45:6812/2379918623 conn(0x7f012c035e30 0x7f012c0382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:46.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.672+0000 7f013effd640 10 monclient: get_auth_request con 0x7f012c035e30 auth_method 0 2024-04-26T00:26:46.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.676+0000 7f013effd640 1 --2- 172.21.15.37:0/1648469430 >> v2:172.21.15.45:6812/2379918623 conn(0x7f012c035e30 0x7f012c0382f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0140072a90 tx=0x7f0130009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:46.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:46.676+0000 7f0127fff640 1 -- 172.21.15.37:0/1648469430 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(264..264 src has 177..264) ==== 3979+0+0 (secure 0 0 0) 0x7f01340e2d30 con 0x7f0140133f00 2024-04-26T00:26:47.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:47.300+0000 7f0127fff640 1 -- 172.21.15.37:0/1648469430 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f014009fe20 con 0x7f0140133f00 2024-04-26T00:26:47.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:47.300+0000 7f0127fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:47.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:47.300+0000 7f0127fff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:26:47.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:47.532+0000 7f01459d0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:26:47.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:47.532+0000 7f01459d0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:26:47.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:47.532+0000 7f01459d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:26:47.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:47.532+0000 7f01459d0640 1 -- 172.21.15.37:0/1648469430 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f014013c460 con 0x7f0140133f00 2024-04-26T00:26:47.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:47.532+0000 7f0127fff640 1 -- 172.21.15.37:0/1648469430 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f01340aa3a0 con 0x7f0140133f00 2024-04-26T00:26:47.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:47.536+0000 7f0127fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:26:47.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:47.536+0000 7f0127fff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:26:47.520 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":52,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:26:47.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:47.536+0000 7f01459d0640 1 -- 172.21.15.37:0/1648469430 >> v2:172.21.15.45:6812/2379918623 conn(0x7f012c035e30 msgr2=0x7f012c0382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:47.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:47.536+0000 7f01459d0640 1 --2- 172.21.15.37:0/1648469430 >> v2:172.21.15.45:6812/2379918623 conn(0x7f012c035e30 0x7f012c0382f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0140072a90 tx=0x7f0130009040 comp rx=0 tx=0).stop 2024-04-26T00:26:47.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:47.536+0000 7f01459d0640 10 monclient: shutdown 2024-04-26T00:26:47.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:47.536+0000 7f01459d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:47.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:47.536+0000 7f01459d0640 1 -- 172.21.15.37:0/1648469430 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0140133f00 msgr2=0x7f0140132170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:47.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:47.536+0000 7f01459d0640 1 --2- 172.21.15.37:0/1648469430 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0140133f00 0x7f0140132170 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f01340098a0 tx=0x7f0134018f80 comp rx=0 tx=0).stop 2024-04-26T00:26:47.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:47.536+0000 7f01459d0640 1 -- 172.21.15.37:0/1648469430 shutdown_connections 2024-04-26T00:26:47.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:47.536+0000 7f01459d0640 1 --2- 172.21.15.37:0/1648469430 >> v2:172.21.15.45:6812/2379918623 conn(0x7f012c035e30 0x7f012c0382f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:47.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:47.536+0000 7f01459d0640 1 --2- 172.21.15.37:0/1648469430 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0140139050 0x7f014011b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:47.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:47.536+0000 7f01459d0640 1 --2- 172.21.15.37:0/1648469430 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0140133f00 0x7f0140132170 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:47.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:47.536+0000 7f01459d0640 1 --2- 172.21.15.37:0/1648469430 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0140133b20 0x7f0140072250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:47.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:47.536+0000 7f01459d0640 1 -- 172.21.15.37:0/1648469430 >> 172.21.15.37:0/1648469430 conn(0x7f01400a1ad0 msgr2=0x7f014013b440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:47.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:47.536+0000 7f01459d0640 1 -- 172.21.15.37:0/1648469430 shutdown_connections 2024-04-26T00:26:47.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:47.536+0000 7f01459d0640 1 -- 172.21.15.37:0/1648469430 wait complete. 2024-04-26T00:26:50.540 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:26:50.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.688+0000 7f41856c4640 10 monclient: get_monmap_and_config 2024-04-26T00:26:50.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.688+0000 7f41856c4640 10 monclient: build_initial_monmap 2024-04-26T00:26:50.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.688+0000 7f41856c4640 10 monclient: monmap: 2024-04-26T00:26:50.674 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:50.674 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:50.674 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:26:50.693195+0000 2024-04-26T00:26:50.674 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:26:50.693195+0000 2024-04-26T00:26:50.674 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:50.674 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:50.675 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:26:50.675 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:26:50.675 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:26:50.675 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:26:50.675 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:26:50.675 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:50.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.688+0000 7f41856c4640 1 Processor -- start 2024-04-26T00:26:50.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.688+0000 7f41856c4640 1 -- start start 2024-04-26T00:26:50.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.688+0000 7f41856c4640 10 monclient: init 2024-04-26T00:26:50.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.688+0000 7f41856c4640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:50.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.688+0000 7f41856c4640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-04-26T00:26:50.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.688+0000 7f41856c4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f41800a0260 addr v1:172.21.15.37:6789/0 2024-04-26T00:26:50.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.688+0000 7f41856c4640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4180137e40 0x7f418013a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:50.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.688+0000 7f41856c4640 10 monclient(hunting): picked mon.noname-c con 0x7f4180137e40 addr v2:172.21.15.45:3300/0 2024-04-26T00:26:50.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.688+0000 7f41856c4640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f418013a770 0x7f418013cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:50.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.688+0000 7f41856c4640 10 monclient(hunting): picked mon.noname-a con 0x7f418013a770 addr v2:172.21.15.37:3300/0 2024-04-26T00:26:50.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.688+0000 7f41856c4640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41800a1130 con 0x7f41800a0260 2024-04-26T00:26:50.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.688+0000 7f41856c4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:50.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.688+0000 7f41856c4640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f41800a0d50 con 0x7f418013a770 2024-04-26T00:26:50.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.688+0000 7f41856c4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:50.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.688+0000 7f41856c4640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f418009ec10 con 0x7f4180137e40 2024-04-26T00:26:50.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.688+0000 7f41856c4640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:50.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.688+0000 7f41856c4640 10 monclient(hunting): authenticate will time out at 1391.998937s 2024-04-26T00:26:50.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.688+0000 7f417effd640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f41800a0260 legacy=0x7f41800a0640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:26:50.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.688+0000 7f417f7fe640 1 -- >> v2:172.21.15.37:3300/0 conn(0x7f418013a770 msgr2=0x7f418013cb60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:26:50.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.688+0000 7f417f7fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f418013a770 0x7f418013cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:26:50.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.688+0000 7f41767fc640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4180137e40 0x7f418013a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:50.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.688+0000 7f41767fc640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4180137e40 0x7f418013a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.37:42842/0 (socket says 172.21.15.37:42842) 2024-04-26T00:26:50.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.688+0000 7f41767fc640 1 -- 172.21.15.37:0/3202643298 learned_addr learned my addr 172.21.15.37:0/3202643298 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:26:50.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.688+0000 7f41767fc640 10 monclient(hunting): get_auth_request con 0x7f4180137e40 auth_method 0 2024-04-26T00:26:50.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.688+0000 7f41767fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:50.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.688+0000 7f41767fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:50.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.688+0000 7f41767fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:50.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.688+0000 7f41767fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:50.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.688+0000 7f41767fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:50.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.688+0000 7f41767fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:50.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.692+0000 7f41767fc640 10 monclient(hunting): handle_auth_done global_id 24649 payload 274 2024-04-26T00:26:50.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.692+0000 7f41767fc640 1 -- 172.21.15.37:0/3202643298 >> v2:172.21.15.37:3300/0 conn(0x7f418013a770 msgr2=0x7f418013cb60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:26:50.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.692+0000 7f41767fc640 1 --2- 172.21.15.37:0/3202643298 >> v2:172.21.15.37:3300/0 conn(0x7f418013a770 0x7f418013cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:50.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.692+0000 7f41767fc640 1 -- 172.21.15.37:0/3202643298 >> v1:172.21.15.37:6789/0 conn(0x7f41800a0260 legacy=0x7f41800a0640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:26:50.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.692+0000 7f41767fc640 10 monclient: _finish_hunting 0 2024-04-26T00:26:50.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.692+0000 7f41767fc640 1 monclient: found mon.noname-c 2024-04-26T00:26:50.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.692+0000 7f41767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:50.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.692+0000 7f41767fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:26:50.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.692+0000 7f41767fc640 1 -- 172.21.15.37:0/3202643298 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f418006cff0 con 0x7f4180137e40 2024-04-26T00:26:50.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f41767fc640 1 -- 172.21.15.37:0/3202643298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4180137e40 msgr2=0x7f418013a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:50.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f41767fc640 1 --2- 172.21.15.37:0/3202643298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4180137e40 0x7f418013a230 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f418009f690 tx=0x7f416801c1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2a168cdfb4d1e17 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:50.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f417e7fc640 1 -- 172.21.15.37:0/3202643298 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41680b0070 con 0x7f4180137e40 2024-04-26T00:26:50.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f417e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:50.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f417e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:26:50.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f417e7fc640 10 monclient: dump: 2024-04-26T00:26:50.683 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:50.683 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:50.683 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:50.683 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:50.683 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:50.684 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:50.684 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:50.684 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:50.684 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:50.684 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:50.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f417e7fc640 10 monclient: _finish_auth 0 2024-04-26T00:26:50.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f417e7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:26:50.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f417e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:50.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f417e7fc640 1 -- 172.21.15.37:0/3202643298 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41680ab600 con 0x7f4180137e40 2024-04-26T00:26:50.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f417e7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:50.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f41856c4640 5 monclient: authenticate success, global_id 24649 2024-04-26T00:26:50.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f41856c4640 10 monclient: get_monmap_and_config success 2024-04-26T00:26:50.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f41856c4640 4 set_mon_vals no callback set 2024-04-26T00:26:50.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f417e7fc640 1 -- 172.21.15.37:0/3202643298 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41680ab9d0 con 0x7f4180137e40 2024-04-26T00:26:50.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f417e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:50.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f417e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:26:50.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f417e7fc640 10 monclient: dump: 2024-04-26T00:26:50.685 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:50.685 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:50.685 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:50.685 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:50.685 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:50.685 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:50.685 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:50.685 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:50.686 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:50.686 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:50.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f41856c4640 10 monclient: shutdown 2024-04-26T00:26:50.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f41856c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:50.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f41856c4640 1 -- 172.21.15.37:0/3202643298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4180137e40 msgr2=0x7f418013a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:50.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f41856c4640 1 --2- 172.21.15.37:0/3202643298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4180137e40 0x7f418013a230 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f418009f690 tx=0x7f416801c1e0 comp rx=0 tx=0).stop 2024-04-26T00:26:50.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f41856c4640 1 -- 172.21.15.37:0/3202643298 shutdown_connections 2024-04-26T00:26:50.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f41856c4640 1 --2- 172.21.15.37:0/3202643298 >> v2:172.21.15.37:3300/0 conn(0x7f418013a770 0x7f418013cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:50.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f41856c4640 1 --2- 172.21.15.37:0/3202643298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4180137e40 0x7f418013a230 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:50.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f41856c4640 1 -- 172.21.15.37:0/3202643298 >> 172.21.15.37:0/3202643298 conn(0x7f41800a1ad0 msgr2=0x7f41800a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:50.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f41856c4640 1 -- 172.21.15.37:0/3202643298 shutdown_connections 2024-04-26T00:26:50.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f41856c4640 1 -- 172.21.15.37:0/3202643298 wait complete. 2024-04-26T00:26:50.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f41856c4640 10 monclient: build_initial_monmap 2024-04-26T00:26:50.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f41856c4640 10 monclient: monmap: 2024-04-26T00:26:50.687 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:50.687 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:26:50.687 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:26:50.687 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:26:50.687 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:50.687 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:26:50.687 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:26:50.687 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:26:50.687 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:26:50.687 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:50.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f41856c4640 1 Processor -- start 2024-04-26T00:26:50.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f41856c4640 1 -- start start 2024-04-26T00:26:50.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f41856c4640 10 monclient: init 2024-04-26T00:26:50.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f41856c4640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:50.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f41856c4640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:26:50.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f41856c4640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41800a0260 0x7f41801cef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:50.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f41856c4640 10 monclient(hunting): picked mon.noname-c con 0x7f41800a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:26:50.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f41856c4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4180137e40 0x7f4180134f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:50.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f41856c4640 10 monclient(hunting): picked mon.noname-b con 0x7f4180137e40 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:26:50.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f41856c4640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f418013a770 0x7f41801354a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:50.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f41856c4640 10 monclient(hunting): picked mon.noname-a con 0x7f418013a770 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:26:50.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f41856c4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:50.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f41856c4640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f418009f0f0 con 0x7f418013a770 2024-04-26T00:26:50.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f41856c4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:50.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f41856c4640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f41800a00e0 con 0x7f4180137e40 2024-04-26T00:26:50.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f41856c4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:50.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f41856c4640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f418009f4c0 con 0x7f41800a0260 2024-04-26T00:26:50.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f41856c4640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:50.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f41856c4640 10 monclient(hunting): authenticate will time out at 1392.007637s 2024-04-26T00:26:50.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f417f7fe640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f418013a770 msgr2=0x7f41801354a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:26:50.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f417f7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f418013a770 0x7f41801354a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:26:50.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f417effd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41800a0260 0x7f41801cef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:50.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f417effd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41800a0260 0x7f41801cef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:44426/0 (socket says 172.21.15.37:44426) 2024-04-26T00:26:50.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f417effd640 1 -- 172.21.15.37:0/3328262868 learned_addr learned my addr 172.21.15.37:0/3328262868 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:26:50.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f417effd640 10 monclient(hunting): get_auth_request con 0x7f41800a0260 auth_method 0 2024-04-26T00:26:50.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f417effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:50.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f417effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:50.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f417effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:50.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f417effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:50.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f417effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:50.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.696+0000 7f417effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:50.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f41767fc640 1 --2- 172.21.15.37:0/3328262868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4180137e40 0x7f4180134f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:50.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f41767fc640 10 monclient(hunting): get_auth_request con 0x7f4180137e40 auth_method 0 2024-04-26T00:26:50.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f41767fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:50.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f41767fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:50.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f41767fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:50.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f417effd640 10 monclient(hunting): handle_auth_done global_id 84149 payload 824 2024-04-26T00:26:50.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f417effd640 1 -- 172.21.15.37:0/3328262868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4180137e40 msgr2=0x7f4180134f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:50.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f417effd640 1 --2- 172.21.15.37:0/3328262868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4180137e40 0x7f4180134f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:50.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f417effd640 1 -- 172.21.15.37:0/3328262868 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f418013a770 msgr2=0x7f41801354a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:26:50.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f417effd640 1 --2- 172.21.15.37:0/3328262868 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f418013a770 0x7f41801354a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:50.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f417effd640 10 monclient: _finish_hunting 0 2024-04-26T00:26:50.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f417effd640 1 monclient: found mon.noname-c 2024-04-26T00:26:50.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f417effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:50.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f417effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:26:50.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f417effd640 1 -- 172.21.15.37:0/3328262868 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f418009e080 con 0x7f41800a0260 2024-04-26T00:26:50.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f417effd640 1 -- 172.21.15.37:0/3328262868 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41800a0260 msgr2=0x7f41801cef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:50.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f417effd640 1 --2- 172.21.15.37:0/3328262868 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41800a0260 0x7f41801cef40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f417000b590 tx=0x7f417000ba50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:50.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f4177fff640 1 -- 172.21.15.37:0/3328262868 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4170013020 con 0x7f41800a0260 2024-04-26T00:26:50.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f4177fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:50.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f4177fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:26:50.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f4177fff640 10 monclient: dump: 2024-04-26T00:26:50.696 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:50.696 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:50.696 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:50.697 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:50.697 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:50.697 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:50.697 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:50.697 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:50.697 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:50.697 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:50.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f4177fff640 10 monclient: _finish_auth 0 2024-04-26T00:26:50.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f4177fff640 10 monclient: _check_auth_tickets 2024-04-26T00:26:50.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f4177fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:50.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f4177fff640 1 -- 172.21.15.37:0/3328262868 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4170005070 con 0x7f41800a0260 2024-04-26T00:26:50.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f41856c4640 5 monclient: authenticate success, global_id 84149 2024-04-26T00:26:50.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f41856c4640 10 monclient: _renew_subs 2024-04-26T00:26:50.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f41856c4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:26:50.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f41856c4640 1 -- 172.21.15.37:0/3328262868 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41801204c0 con 0x7f41800a0260 2024-04-26T00:26:50.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f4177fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:50.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f4177fff640 1 -- 172.21.15.37:0/3328262868 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4170005460 con 0x7f41800a0260 2024-04-26T00:26:50.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f4177fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:50.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f4177fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:26:50.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f4177fff640 10 monclient: dump: 2024-04-26T00:26:50.698 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:50.698 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:50.698 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:50.699 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:50.699 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:50.699 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:50.699 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:50.699 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:50.699 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:50.699 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:50.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f417dffb640 4 set_mon_vals no callback set 2024-04-26T00:26:50.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f41856c4640 10 monclient: _renew_subs 2024-04-26T00:26:50.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f41856c4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:26:50.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f41856c4640 1 -- 172.21.15.37:0/3328262868 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4180121070 con 0x7f41800a0260 2024-04-26T00:26:50.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f41856c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:50.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f41856c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:50.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f41856c4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:26:50.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f41856c4640 1 -- 172.21.15.37:0/3328262868 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4180066f40 con 0x7f41800a0260 2024-04-26T00:26:50.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f4177fff640 1 -- 172.21.15.37:0/3328262868 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f4170016020 con 0x7f41800a0260 2024-04-26T00:26:50.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f4177fff640 1 --2- 172.21.15.37:0/3328262868 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4160035e80 0x7f4160038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:50.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.700+0000 7f4177fff640 1 -- 172.21.15.37:0/3328262868 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(264..264 src has 177..264) ==== 3979+0+0 (secure 0 0 0) 0x7f4170048930 con 0x7f41800a0260 2024-04-26T00:26:50.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.704+0000 7f41767fc640 1 --2- 172.21.15.37:0/3328262868 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4160035e80 0x7f4160038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:50.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.704+0000 7f41767fc640 10 monclient: get_auth_request con 0x7f4160035e80 auth_method 0 2024-04-26T00:26:50.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.704+0000 7f41767fc640 1 --2- 172.21.15.37:0/3328262868 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4160035e80 0x7f4160038340 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f41680ad000 tx=0x7f41680a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:50.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.708+0000 7f4177fff640 1 -- 172.21.15.37:0/3328262868 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f417001d020 con 0x7f41800a0260 2024-04-26T00:26:50.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.708+0000 7f4177fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:50.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.708+0000 7f4177fff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:26:50.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.924+0000 7f41856c4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:26:50.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.924+0000 7f41856c4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:26:50.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.924+0000 7f41856c4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:26:50.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.924+0000 7f41856c4640 1 -- 172.21.15.37:0/3328262868 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4180121350 con 0x7f41800a0260 2024-04-26T00:26:50.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.928+0000 7f4177fff640 1 -- 172.21.15.37:0/3328262868 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f4170018dc0 con 0x7f41800a0260 2024-04-26T00:26:50.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.928+0000 7f4177fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:26:50.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.928+0000 7f4177fff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:26:50.913 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":52,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:26:50.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.928+0000 7f41856c4640 1 -- 172.21.15.37:0/3328262868 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4160035e80 msgr2=0x7f4160038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:50.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.928+0000 7f41856c4640 1 --2- 172.21.15.37:0/3328262868 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4160035e80 0x7f4160038340 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f41680ad000 tx=0x7f41680a6000 comp rx=0 tx=0).stop 2024-04-26T00:26:50.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.928+0000 7f41856c4640 10 monclient: shutdown 2024-04-26T00:26:50.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.928+0000 7f41856c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:50.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.928+0000 7f41856c4640 1 -- 172.21.15.37:0/3328262868 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41800a0260 msgr2=0x7f41801cef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:50.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.928+0000 7f41856c4640 1 --2- 172.21.15.37:0/3328262868 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41800a0260 0x7f41801cef40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f417000b590 tx=0x7f417000ba50 comp rx=0 tx=0).stop 2024-04-26T00:26:50.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.928+0000 7f41856c4640 1 -- 172.21.15.37:0/3328262868 shutdown_connections 2024-04-26T00:26:50.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.928+0000 7f41856c4640 1 --2- 172.21.15.37:0/3328262868 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4160035e80 0x7f4160038340 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:50.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.928+0000 7f41856c4640 1 --2- 172.21.15.37:0/3328262868 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f418013a770 0x7f41801354a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:50.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.928+0000 7f41856c4640 1 --2- 172.21.15.37:0/3328262868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4180137e40 0x7f4180134f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:50.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.928+0000 7f41856c4640 1 --2- 172.21.15.37:0/3328262868 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41800a0260 0x7f41801cef40 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:50.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.928+0000 7f41856c4640 1 -- 172.21.15.37:0/3328262868 >> 172.21.15.37:0/3328262868 conn(0x7f41800a1ad0 msgr2=0x7f41800a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:50.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.928+0000 7f41856c4640 1 -- 172.21.15.37:0/3328262868 shutdown_connections 2024-04-26T00:26:50.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:50.928+0000 7f41856c4640 1 -- 172.21.15.37:0/3328262868 wait complete. 2024-04-26T00:26:53.931 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:26:54.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.076+0000 7f7143552640 10 monclient: get_monmap_and_config 2024-04-26T00:26:54.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.076+0000 7f7143552640 10 monclient: build_initial_monmap 2024-04-26T00:26:54.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.076+0000 7f7143552640 10 monclient: monmap: 2024-04-26T00:26:54.064 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:54.064 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:54.064 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:26:54.082648+0000 2024-04-26T00:26:54.064 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:26:54.082648+0000 2024-04-26T00:26:54.064 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:54.064 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:54.064 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:26:54.064 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:26:54.064 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:26:54.064 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:26:54.064 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:26:54.064 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:54.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.076+0000 7f7143552640 1 Processor -- start 2024-04-26T00:26:54.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.076+0000 7f7143552640 1 -- start start 2024-04-26T00:26:54.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.076+0000 7f7143552640 10 monclient: init 2024-04-26T00:26:54.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.080+0000 7f7143552640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:54.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.080+0000 7f7143552640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-04-26T00:26:54.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.080+0000 7f7143552640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f713c072e10 0x7f713c1337a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:54.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.080+0000 7f7143552640 10 monclient(hunting): picked mon.noname-a con 0x7f713c072e10 addr v2:172.21.15.37:3300/0 2024-04-26T00:26:54.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.080+0000 7f7143552640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f713c133ce0 0x7f713c1340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:54.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.080+0000 7f7143552640 10 monclient(hunting): picked mon.noname-b con 0x7f713c133ce0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:26:54.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.080+0000 7f7143552640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f713c09f9d0 addr v1:172.21.15.37:6789/0 2024-04-26T00:26:54.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.080+0000 7f7143552640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f713c0a12e0 con 0x7f713c09f9d0 2024-04-26T00:26:54.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.080+0000 7f7143552640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:54.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.080+0000 7f7143552640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f713c0731f0 con 0x7f713c072e10 2024-04-26T00:26:54.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.080+0000 7f7143552640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:54.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.080+0000 7f7143552640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f713c134600 con 0x7f713c133ce0 2024-04-26T00:26:54.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.080+0000 7f7143552640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:54.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.080+0000 7f7143552640 10 monclient(hunting): authenticate will time out at 1395.388639s 2024-04-26T00:26:54.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.080+0000 7f71412ee640 1 -- >> v2:172.21.15.37:3300/0 conn(0x7f713c072e10 msgr2=0x7f713c1337a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:26:54.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.080+0000 7f7141aef640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f713c09f9d0 legacy=0x7f713c09fe40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:26:54.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.080+0000 7f71412ee640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f713c072e10 0x7f713c1337a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:26:54.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.080+0000 7f7140aed640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f713c133ce0 0x7f713c1340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:54.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.080+0000 7f7140aed640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f713c133ce0 0x7f713c1340c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:44438/0 (socket says 172.21.15.37:44438) 2024-04-26T00:26:54.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.080+0000 7f7140aed640 1 -- 172.21.15.37:0/3852824649 learned_addr learned my addr 172.21.15.37:0/3852824649 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:26:54.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.080+0000 7f7140aed640 10 monclient(hunting): get_auth_request con 0x7f713c133ce0 auth_method 0 2024-04-26T00:26:54.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.080+0000 7f7140aed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:54.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.080+0000 7f7140aed640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:54.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.080+0000 7f7140aed640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:54.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.080+0000 7f7140aed640 1 -- 172.21.15.37:0/3852824649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f713c133ce0 msgr2=0x7f713c1340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:26:54.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.080+0000 7f7140aed640 1 -- 172.21.15.37:0/3852824649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f713c133ce0 msgr2=0x7f713c1340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:26:54.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.080+0000 7f7140aed640 1 --2- 172.21.15.37:0/3852824649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f713c133ce0 0x7f713c1340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:26:54.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.080+0000 7f7140aed640 1 --2- 172.21.15.37:0/3852824649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f713c133ce0 0x7f713c1340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:26:54.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.284+0000 7f71412ee640 1 -- 172.21.15.37:0/3852824649 >> v2:172.21.15.37:3300/0 conn(0x7f713c072e10 msgr2=0x7f713c1337a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:26:54.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.284+0000 7f71412ee640 1 --2- 172.21.15.37:0/3852824649 >> v2:172.21.15.37:3300/0 conn(0x7f713c072e10 0x7f713c1337a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:26:54.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.284+0000 7f7141aef640 1 -- 172.21.15.37:0/3852824649 >> v1:172.21.15.37:6789/0 conn(0x7f713c09f9d0 legacy=0x7f713c09fe40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:26:54.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.284+0000 7f7140aed640 1 --2- 172.21.15.37:0/3852824649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f713c133ce0 0x7f713c1340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:54.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.284+0000 7f7140aed640 10 monclient(hunting): get_auth_request con 0x7f713c133ce0 auth_method 0 2024-04-26T00:26:54.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.284+0000 7f7140aed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:54.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.284+0000 7f7140aed640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:54.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.284+0000 7f7140aed640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:26:54.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.284+0000 7f7140aed640 1 -- 172.21.15.37:0/3852824649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f713c133ce0 msgr2=0x7f713c1340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:26:54.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.284+0000 7f7140aed640 1 -- 172.21.15.37:0/3852824649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f713c133ce0 msgr2=0x7f713c1340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:26:54.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.284+0000 7f7140aed640 1 --2- 172.21.15.37:0/3852824649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f713c133ce0 0x7f713c1340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:26:54.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.284+0000 7f7140aed640 1 --2- 172.21.15.37:0/3852824649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f713c133ce0 0x7f713c1340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:26:54.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.688+0000 7f71412ee640 1 -- 172.21.15.37:0/3852824649 >> v2:172.21.15.37:3300/0 conn(0x7f713c072e10 msgr2=0x7f713c1337a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:26:54.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.688+0000 7f7141aef640 1 -- 172.21.15.37:0/3852824649 >> v1:172.21.15.37:6789/0 conn(0x7f713c09f9d0 legacy=0x7f713c09fe40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:26:54.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.688+0000 7f71412ee640 1 --2- 172.21.15.37:0/3852824649 >> v2:172.21.15.37:3300/0 conn(0x7f713c072e10 0x7f713c1337a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:26:54.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.688+0000 7f7140aed640 1 --2- 172.21.15.37:0/3852824649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f713c133ce0 0x7f713c1340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:54.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.688+0000 7f7140aed640 10 monclient(hunting): get_auth_request con 0x7f713c133ce0 auth_method 0 2024-04-26T00:26:54.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.688+0000 7f7140aed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:54.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.688+0000 7f7140aed640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:54.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.688+0000 7f7140aed640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:26:54.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.688+0000 7f7140aed640 1 -- 172.21.15.37:0/3852824649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f713c133ce0 msgr2=0x7f713c1340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:26:54.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.688+0000 7f7140aed640 1 -- 172.21.15.37:0/3852824649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f713c133ce0 msgr2=0x7f713c1340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:26:54.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.688+0000 7f7140aed640 1 --2- 172.21.15.37:0/3852824649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f713c133ce0 0x7f713c1340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:26:54.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:54.688+0000 7f7140aed640 1 --2- 172.21.15.37:0/3852824649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f713c133ce0 0x7f713c1340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:26:55.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:55.492+0000 7f7141aef640 1 -- 172.21.15.37:0/3852824649 >> v1:172.21.15.37:6789/0 conn(0x7f713c09f9d0 legacy=0x7f713c09fe40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:26:55.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:55.492+0000 7f71412ee640 1 -- 172.21.15.37:0/3852824649 >> v2:172.21.15.37:3300/0 conn(0x7f713c072e10 msgr2=0x7f713c1337a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:26:55.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:55.492+0000 7f71412ee640 1 --2- 172.21.15.37:0/3852824649 >> v2:172.21.15.37:3300/0 conn(0x7f713c072e10 0x7f713c1337a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:26:55.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:55.492+0000 7f7140aed640 1 --2- 172.21.15.37:0/3852824649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f713c133ce0 0x7f713c1340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:55.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:55.492+0000 7f7140aed640 10 monclient(hunting): get_auth_request con 0x7f713c133ce0 auth_method 0 2024-04-26T00:26:55.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:55.492+0000 7f7140aed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:55.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:55.492+0000 7f7140aed640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:55.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:55.492+0000 7f7140aed640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:26:55.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:55.492+0000 7f7140aed640 1 -- 172.21.15.37:0/3852824649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f713c133ce0 msgr2=0x7f713c1340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:26:55.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:55.492+0000 7f7140aed640 1 -- 172.21.15.37:0/3852824649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f713c133ce0 msgr2=0x7f713c1340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:26:55.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:55.492+0000 7f7140aed640 1 --2- 172.21.15.37:0/3852824649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f713c133ce0 0x7f713c1340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:26:55.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:55.492+0000 7f7140aed640 1 --2- 172.21.15.37:0/3852824649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f713c133ce0 0x7f713c1340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:26:57.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.084+0000 7f7132ffd640 10 monclient(hunting): tick 2024-04-26T00:26:57.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.084+0000 7f7132ffd640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:26:57.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.084+0000 7f7132ffd640 1 monclient(hunting): continuing hunt 2024-04-26T00:26:57.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.084+0000 7f7132ffd640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:26:57.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.084+0000 7f7132ffd640 1 -- 172.21.15.37:0/3852824649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f713c133ce0 msgr2=0x7f713c1340c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:26:57.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.084+0000 7f7132ffd640 1 --2- 172.21.15.37:0/3852824649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f713c133ce0 0x7f713c1340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:57.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.084+0000 7f7132ffd640 1 -- 172.21.15.37:0/3852824649 >> v2:172.21.15.37:3300/0 conn(0x7f713c072e10 msgr2=0x7f713c1337a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:26:57.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.084+0000 7f7132ffd640 1 --2- 172.21.15.37:0/3852824649 >> v2:172.21.15.37:3300/0 conn(0x7f713c072e10 0x7f713c1337a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:57.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.084+0000 7f7132ffd640 1 -- 172.21.15.37:0/3852824649 >> v1:172.21.15.37:6789/0 conn(0x7f713c09f9d0 legacy=0x7f713c09fe40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:26:57.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.084+0000 7f7132ffd640 10 monclient: _add_conns ranks=[1,4] 2024-04-26T00:26:57.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.084+0000 7f7132ffd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f71200058c0 addr v1:172.21.15.45:6789/0 2024-04-26T00:26:57.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.084+0000 7f7132ffd640 1 --2- 172.21.15.37:0/3852824649 >> v2:172.21.15.45:3300/0 conn(0x7f7120009570 0x7f712000ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:57.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.084+0000 7f7132ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f7120009570 addr v2:172.21.15.45:3300/0 2024-04-26T00:26:57.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.084+0000 7f7132ffd640 1 -- 172.21.15.37:0/3852824649 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f713c06d050 con 0x7f71200058c0 2024-04-26T00:26:57.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.084+0000 7f7132ffd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:57.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.084+0000 7f7132ffd640 1 -- 172.21.15.37:0/3852824649 --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f713c0731f0 con 0x7f7120009570 2024-04-26T00:26:57.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.084+0000 7f7132ffd640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:57.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.084+0000 7f7140aed640 1 --2- 172.21.15.37:0/3852824649 >> v2:172.21.15.45:3300/0 conn(0x7f7120009570 0x7f712000ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:57.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.084+0000 7f7140aed640 10 monclient(hunting): get_auth_request con 0x7f7120009570 auth_method 0 2024-04-26T00:26:57.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.084+0000 7f7140aed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:57.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.084+0000 7f7140aed640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:57.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.084+0000 7f7140aed640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:57.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.084+0000 7f7140aed640 1 -- 172.21.15.37:0/3852824649 >> v2:172.21.15.45:3300/0 conn(0x7f7120009570 msgr2=0x7f712000ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:26:57.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.084+0000 7f7140aed640 1 -- 172.21.15.37:0/3852824649 >> v2:172.21.15.45:3300/0 conn(0x7f7120009570 msgr2=0x7f712000ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:26:57.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.084+0000 7f7140aed640 1 --2- 172.21.15.37:0/3852824649 >> v2:172.21.15.45:3300/0 conn(0x7f7120009570 0x7f712000ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:26:57.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.084+0000 7f7140aed640 1 --2- 172.21.15.37:0/3852824649 >> v2:172.21.15.45:3300/0 conn(0x7f7120009570 0x7f712000ba30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:26:57.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.084+0000 7f71412ee640 1 -- 172.21.15.37:0/3852824649 >> v1:172.21.15.45:6789/0 conn(0x7f71200058c0 legacy=0x7f7120007d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:57.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.288+0000 7f7140aed640 1 --2- 172.21.15.37:0/3852824649 >> v2:172.21.15.45:3300/0 conn(0x7f7120009570 0x7f712000ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:57.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.288+0000 7f7140aed640 10 monclient(hunting): get_auth_request con 0x7f7120009570 auth_method 0 2024-04-26T00:26:57.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.288+0000 7f7140aed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:57.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.288+0000 7f7140aed640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:57.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.288+0000 7f7140aed640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:26:57.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.288+0000 7f7140aed640 1 -- 172.21.15.37:0/3852824649 >> v2:172.21.15.45:3300/0 conn(0x7f7120009570 msgr2=0x7f712000ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:26:57.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.288+0000 7f7140aed640 1 -- 172.21.15.37:0/3852824649 >> v2:172.21.15.45:3300/0 conn(0x7f7120009570 msgr2=0x7f712000ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:26:57.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.288+0000 7f7140aed640 1 --2- 172.21.15.37:0/3852824649 >> v2:172.21.15.45:3300/0 conn(0x7f7120009570 0x7f712000ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:26:57.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.288+0000 7f7140aed640 1 --2- 172.21.15.37:0/3852824649 >> v2:172.21.15.45:3300/0 conn(0x7f7120009570 0x7f712000ba30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:26:57.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.692+0000 7f7140aed640 1 --2- 172.21.15.37:0/3852824649 >> v2:172.21.15.45:3300/0 conn(0x7f7120009570 0x7f712000ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:57.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.692+0000 7f7140aed640 10 monclient(hunting): get_auth_request con 0x7f7120009570 auth_method 0 2024-04-26T00:26:57.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.692+0000 7f7140aed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:57.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.692+0000 7f7140aed640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:57.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.692+0000 7f7140aed640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:26:57.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.692+0000 7f7140aed640 1 -- 172.21.15.37:0/3852824649 >> v2:172.21.15.45:3300/0 conn(0x7f7120009570 msgr2=0x7f712000ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:26:57.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.692+0000 7f7140aed640 1 -- 172.21.15.37:0/3852824649 >> v2:172.21.15.45:3300/0 conn(0x7f7120009570 msgr2=0x7f712000ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:26:57.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.692+0000 7f7140aed640 1 --2- 172.21.15.37:0/3852824649 >> v2:172.21.15.45:3300/0 conn(0x7f7120009570 0x7f712000ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:26:57.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:57.692+0000 7f7140aed640 1 --2- 172.21.15.37:0/3852824649 >> v2:172.21.15.45:3300/0 conn(0x7f7120009570 0x7f712000ba30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:26:58.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.496+0000 7f7140aed640 1 --2- 172.21.15.37:0/3852824649 >> v2:172.21.15.45:3300/0 conn(0x7f7120009570 0x7f712000ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:58.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.496+0000 7f7140aed640 10 monclient(hunting): get_auth_request con 0x7f7120009570 auth_method 0 2024-04-26T00:26:58.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.496+0000 7f7140aed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:58.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.496+0000 7f7140aed640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:58.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.496+0000 7f7140aed640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:26:58.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.496+0000 7f7140aed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:58.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.496+0000 7f7140aed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:58.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.496+0000 7f7140aed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:58.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.496+0000 7f7140aed640 10 monclient(hunting): handle_auth_done global_id 24655 payload 274 2024-04-26T00:26:58.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.496+0000 7f7140aed640 1 -- 172.21.15.37:0/3852824649 >> v1:172.21.15.45:6789/0 conn(0x7f71200058c0 legacy=0x7f7120007d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:58.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.496+0000 7f7140aed640 10 monclient: _finish_hunting 0 2024-04-26T00:26:58.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.496+0000 7f7140aed640 1 monclient: found mon.noname-c 2024-04-26T00:26:58.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.496+0000 7f7140aed640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:58.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.496+0000 7f7140aed640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:26:58.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.496+0000 7f7140aed640 1 -- 172.21.15.37:0/3852824649 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f713c0a12e0 con 0x7f7120009570 2024-04-26T00:26:58.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.504+0000 7f7140aed640 1 -- 172.21.15.37:0/3852824649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7120009570 msgr2=0x7f712000ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:58.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.504+0000 7f7140aed640 1 --2- 172.21.15.37:0/3852824649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7120009570 0x7f712000ba30 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7138009100 tx=0x7f713801de80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=713d0a01f7c5a6fa server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:58.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.536+0000 7f7133fff640 1 -- 172.21.15.37:0/3852824649 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71380ac070 con 0x7f7120009570 2024-04-26T00:26:58.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.536+0000 7f7133fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:58.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.536+0000 7f7133fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:26:58.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.536+0000 7f7133fff640 10 monclient: dump: 2024-04-26T00:26:58.521 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:58.521 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:58.521 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:58.521 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:58.521 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:58.521 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:58.522 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:58.522 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:58.522 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:58.522 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:58.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.536+0000 7f7133fff640 10 monclient: _finish_auth 0 2024-04-26T00:26:58.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.536+0000 7f7133fff640 10 monclient: _check_auth_tickets 2024-04-26T00:26:58.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.536+0000 7f7133fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:58.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.536+0000 7f7143552640 5 monclient: authenticate success, global_id 24655 2024-04-26T00:26:58.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.536+0000 7f7143552640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:26:58.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.536+0000 7f7133fff640 1 -- 172.21.15.37:0/3852824649 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71380aa070 con 0x7f7120009570 2024-04-26T00:26:58.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.536+0000 7f7133fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:58.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.536+0000 7f7133fff640 1 -- 172.21.15.37:0/3852824649 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f713800f040 con 0x7f7120009570 2024-04-26T00:26:58.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.536+0000 7f7133fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:58.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.536+0000 7f7133fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:26:58.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.536+0000 7f7133fff640 10 monclient: dump: 2024-04-26T00:26:58.523 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:58.523 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:58.523 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:58.523 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:58.523 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:58.524 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:58.524 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:58.524 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:58.524 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:58.524 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:58.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.536+0000 7f7143552640 10 monclient: get_monmap_and_config success 2024-04-26T00:26:58.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.536+0000 7f7143552640 4 set_mon_vals no callback set 2024-04-26T00:26:58.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.536+0000 7f7143552640 10 monclient: shutdown 2024-04-26T00:26:58.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.536+0000 7f7143552640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:58.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.536+0000 7f7143552640 1 -- 172.21.15.37:0/3852824649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7120009570 msgr2=0x7f712000ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:58.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.536+0000 7f7143552640 1 --2- 172.21.15.37:0/3852824649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7120009570 0x7f712000ba30 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7138009100 tx=0x7f713801de80 comp rx=0 tx=0).stop 2024-04-26T00:26:58.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.536+0000 7f7141aef640 1 -- 172.21.15.37:0/3852824649 reap_dead start 2024-04-26T00:26:58.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.536+0000 7f7143552640 1 -- 172.21.15.37:0/3852824649 shutdown_connections 2024-04-26T00:26:58.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.536+0000 7f7143552640 1 --2- 172.21.15.37:0/3852824649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7120009570 0x7f712000ba30 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:58.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.536+0000 7f7143552640 1 -- 172.21.15.37:0/3852824649 >> 172.21.15.37:0/3852824649 conn(0x7f713c0a1ad0 msgr2=0x7f713c0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:58.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.536+0000 7f7143552640 1 -- 172.21.15.37:0/3852824649 shutdown_connections 2024-04-26T00:26:58.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.536+0000 7f7143552640 1 -- 172.21.15.37:0/3852824649 wait complete. 2024-04-26T00:26:58.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.540+0000 7f7143552640 10 monclient: build_initial_monmap 2024-04-26T00:26:58.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.540+0000 7f7143552640 10 monclient: monmap: 2024-04-26T00:26:58.526 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:58.526 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:26:58.526 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:26:58.526 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:26:58.526 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:58.526 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:26:58.526 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:26:58.526 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:26:58.527 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:26:58.527 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:58.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.540+0000 7f7143552640 1 Processor -- start 2024-04-26T00:26:58.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.540+0000 7f7143552640 1 -- start start 2024-04-26T00:26:58.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.540+0000 7f7143552640 10 monclient: init 2024-04-26T00:26:58.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.540+0000 7f7143552640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:58.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.540+0000 7f7143552640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:26:58.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.540+0000 7f7143552640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7120009570 0x7f713c1cefc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:58.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.540+0000 7f7143552640 10 monclient(hunting): picked mon.noname-a con 0x7f7120009570 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:26:58.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.540+0000 7f7143552640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f713c12fa90 0x7f713c12ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:58.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.540+0000 7f7143552640 10 monclient(hunting): picked mon.noname-c con 0x7f713c12fa90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:26:58.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.540+0000 7f7143552640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f713c1396c0 0x7f713c139b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:58.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.540+0000 7f7143552640 10 monclient(hunting): picked mon.noname-b con 0x7f713c1396c0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:26:58.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.540+0000 7f7143552640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:58.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.540+0000 7f7143552640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f713c09bdd0 con 0x7f7120009570 2024-04-26T00:26:58.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.540+0000 7f7143552640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:58.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.540+0000 7f7143552640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f713c09bc50 con 0x7f713c1396c0 2024-04-26T00:26:58.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.540+0000 7f7143552640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:58.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.540+0000 7f7143552640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f713c0a0d60 con 0x7f713c12fa90 2024-04-26T00:26:58.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.540+0000 7f7143552640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:58.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.540+0000 7f7143552640 10 monclient(hunting): authenticate will time out at 1399.851747s 2024-04-26T00:26:58.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.540+0000 7f71412ee640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7120009570 msgr2=0x7f713c1cefc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:26:58.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.540+0000 7f71412ee640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7120009570 0x7f713c1cefc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:26:58.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.540+0000 7f7140aed640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f713c12fa90 0x7f713c12ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:58.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7141aef640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f713c1396c0 0x7f713c139b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:58.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7140aed640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f713c12fa90 0x7f713c12ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:44494/0 (socket says 172.21.15.37:44494) 2024-04-26T00:26:58.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7141aef640 10 monclient(hunting): get_auth_request con 0x7f713c1396c0 auth_method 0 2024-04-26T00:26:58.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7140aed640 1 -- 172.21.15.37:0/3199175079 learned_addr learned my addr 172.21.15.37:0/3199175079 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:26:58.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7141aef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:58.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7141aef640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:58.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7141aef640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:58.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7140aed640 10 monclient(hunting): get_auth_request con 0x7f713c12fa90 auth_method 0 2024-04-26T00:26:58.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7140aed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:58.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7140aed640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:58.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7140aed640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:58.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7141aef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:58.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7141aef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:58.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7141aef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:58.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7140aed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:58.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7140aed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:58.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7140aed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:58.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7140aed640 10 monclient(hunting): handle_auth_done global_id 84152 payload 824 2024-04-26T00:26:58.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7140aed640 1 -- 172.21.15.37:0/3199175079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f713c1396c0 msgr2=0x7f713c139b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:58.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7140aed640 1 --2- 172.21.15.37:0/3199175079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f713c1396c0 0x7f713c139b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:58.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7140aed640 1 -- 172.21.15.37:0/3199175079 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7120009570 msgr2=0x7f713c1cefc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:26:58.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7140aed640 1 --2- 172.21.15.37:0/3199175079 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7120009570 0x7f713c1cefc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:58.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7140aed640 10 monclient: _finish_hunting 0 2024-04-26T00:26:58.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7140aed640 1 monclient: found mon.noname-c 2024-04-26T00:26:58.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7140aed640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:58.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7140aed640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:26:58.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7140aed640 1 -- 172.21.15.37:0/3199175079 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f713c11d170 con 0x7f713c12fa90 2024-04-26T00:26:58.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7140aed640 1 -- 172.21.15.37:0/3199175079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f713c12fa90 msgr2=0x7f713c12ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:58.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7140aed640 1 --2- 172.21.15.37:0/3199175079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f713c12fa90 0x7f713c12ff10 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f71380089c0 tx=0x7f7138008b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:58.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7131ffb640 1 -- 172.21.15.37:0/3199175079 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71380ac040 con 0x7f713c12fa90 2024-04-26T00:26:58.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7131ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:58.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7131ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:26:58.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7131ffb640 10 monclient: dump: 2024-04-26T00:26:58.535 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:58.535 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:58.535 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:58.536 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:58.536 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:58.536 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:58.536 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:58.536 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:58.536 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:58.536 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:58.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7131ffb640 10 monclient: _finish_auth 0 2024-04-26T00:26:58.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7131ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:26:58.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7131ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:58.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7143552640 5 monclient: authenticate success, global_id 84152 2024-04-26T00:26:58.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7143552640 10 monclient: _renew_subs 2024-04-26T00:26:58.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7143552640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:26:58.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7131ffb640 1 -- 172.21.15.37:0/3199175079 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71380b1460 con 0x7f713c12fa90 2024-04-26T00:26:58.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7143552640 1 -- 172.21.15.37:0/3199175079 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f713c11bcc0 con 0x7f713c12fa90 2024-04-26T00:26:58.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7131ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:58.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7131ffb640 1 -- 172.21.15.37:0/3199175079 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71380b1850 con 0x7f713c12fa90 2024-04-26T00:26:58.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7131ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:58.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7131ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:26:58.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7131ffb640 10 monclient: dump: 2024-04-26T00:26:58.537 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:58.537 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:58.537 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:58.537 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:58.537 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:58.537 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:58.537 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:58.537 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:58.537 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:58.537 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:58.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f71337fe640 4 set_mon_vals no callback set 2024-04-26T00:26:58.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7143552640 10 monclient: _renew_subs 2024-04-26T00:26:58.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7143552640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:26:58.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.544+0000 7f7143552640 1 -- 172.21.15.37:0/3199175079 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f713c11ce90 con 0x7f713c12fa90 2024-04-26T00:26:58.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.548+0000 7f7143552640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:58.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.548+0000 7f7143552640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:58.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.548+0000 7f7143552640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:26:58.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.548+0000 7f7131ffb640 1 -- 172.21.15.37:0/3199175079 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f71380c1060 con 0x7f713c12fa90 2024-04-26T00:26:58.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.548+0000 7f7143552640 1 -- 172.21.15.37:0/3199175079 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f713c066f40 con 0x7f713c12fa90 2024-04-26T00:26:58.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.548+0000 7f7131ffb640 1 --2- 172.21.15.37:0/3199175079 >> v2:172.21.15.45:6812/2379918623 conn(0x7f71200369d0 0x7f7120038e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:58.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.548+0000 7f7131ffb640 1 -- 172.21.15.37:0/3199175079 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(264..264 src has 177..264) ==== 3979+0+0 (secure 0 0 0) 0x7f71380e34b0 con 0x7f713c12fa90 2024-04-26T00:26:58.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.548+0000 7f71412ee640 1 --2- 172.21.15.37:0/3199175079 >> v2:172.21.15.45:6812/2379918623 conn(0x7f71200369d0 0x7f7120038e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:58.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.548+0000 7f71412ee640 10 monclient: get_auth_request con 0x7f71200369d0 auth_method 0 2024-04-26T00:26:58.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.552+0000 7f71412ee640 1 --2- 172.21.15.37:0/3199175079 >> v2:172.21.15.45:6812/2379918623 conn(0x7f71200369d0 0x7f7120038e90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f712c003990 tx=0x7f712c009060 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:58.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.556+0000 7f7131ffb640 1 -- 172.21.15.37:0/3199175079 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f713800a9b0 con 0x7f713c12fa90 2024-04-26T00:26:58.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.556+0000 7f7131ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:58.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.556+0000 7f7131ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:26:58.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.800+0000 7f7143552640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:26:58.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.800+0000 7f7143552640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:26:58.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.800+0000 7f7143552640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:26:58.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.800+0000 7f7143552640 1 -- 172.21.15.37:0/3199175079 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f713c09fe90 con 0x7f713c12fa90 2024-04-26T00:26:58.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.800+0000 7f7131ffb640 1 -- 172.21.15.37:0/3199175079 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7f71380b1dd0 con 0x7f713c12fa90 2024-04-26T00:26:58.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.800+0000 7f7131ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:26:58.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.800+0000 7f7131ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:26:58.788 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":56,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:26:58.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.804+0000 7f7143552640 1 -- 172.21.15.37:0/3199175079 >> v2:172.21.15.45:6812/2379918623 conn(0x7f71200369d0 msgr2=0x7f7120038e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:58.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.804+0000 7f7143552640 1 --2- 172.21.15.37:0/3199175079 >> v2:172.21.15.45:6812/2379918623 conn(0x7f71200369d0 0x7f7120038e90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f712c003990 tx=0x7f712c009060 comp rx=0 tx=0).stop 2024-04-26T00:26:58.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.804+0000 7f7143552640 10 monclient: shutdown 2024-04-26T00:26:58.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.804+0000 7f7143552640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:58.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.804+0000 7f7143552640 1 -- 172.21.15.37:0/3199175079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f713c12fa90 msgr2=0x7f713c12ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:58.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.804+0000 7f7143552640 1 --2- 172.21.15.37:0/3199175079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f713c12fa90 0x7f713c12ff10 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f71380089c0 tx=0x7f7138008b50 comp rx=0 tx=0).stop 2024-04-26T00:26:58.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.804+0000 7f7143552640 1 -- 172.21.15.37:0/3199175079 shutdown_connections 2024-04-26T00:26:58.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.804+0000 7f7143552640 1 --2- 172.21.15.37:0/3199175079 >> v2:172.21.15.45:6812/2379918623 conn(0x7f71200369d0 0x7f7120038e90 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:58.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.804+0000 7f7143552640 1 --2- 172.21.15.37:0/3199175079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f713c1396c0 0x7f713c139b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:58.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.804+0000 7f7143552640 1 --2- 172.21.15.37:0/3199175079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f713c12fa90 0x7f713c12ff10 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:58.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.804+0000 7f7143552640 1 --2- 172.21.15.37:0/3199175079 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7120009570 0x7f713c1cefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:58.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.804+0000 7f7143552640 1 -- 172.21.15.37:0/3199175079 >> 172.21.15.37:0/3199175079 conn(0x7f713c0a1ad0 msgr2=0x7f713c132900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:58.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.804+0000 7f7143552640 1 -- 172.21.15.37:0/3199175079 shutdown_connections 2024-04-26T00:26:58.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.804+0000 7f7143552640 1 -- 172.21.15.37:0/3199175079 wait complete. 2024-04-26T00:26:58.808 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-26T00:26:58.808 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-04-26T00:26:58.808 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:26:58.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f832bb32640 10 monclient: get_monmap_and_config 2024-04-26T00:26:58.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f832bb32640 10 monclient: build_initial_monmap 2024-04-26T00:26:58.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f832bb32640 10 monclient: monmap: 2024-04-26T00:26:58.937 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:58.937 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:58.937 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:26:58.956086+0000 2024-04-26T00:26:58.937 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:26:58.956086+0000 2024-04-26T00:26:58.937 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:58.938 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:58.938 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:26:58.938 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:26:58.938 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:26:58.938 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:26:58.938 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:26:58.938 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:58.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f832bb32640 1 Processor -- start 2024-04-26T00:26:58.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f832bb32640 1 -- start start 2024-04-26T00:26:58.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f832bb32640 10 monclient: init 2024-04-26T00:26:58.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f832bb32640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:58.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f832bb32640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-04-26T00:26:58.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f832bb32640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f83240a3d90 0x7f832409bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:58.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f832bb32640 10 monclient(hunting): picked mon.noname-b con 0x7f83240a3d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:26:58.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f832bb32640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f83240a4170 0x7f83240a45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:58.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f832bb32640 10 monclient(hunting): picked mon.noname-a con 0x7f83240a4170 addr v2:172.21.15.37:3300/0 2024-04-26T00:26:58.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f832bb32640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8324133b20 addr v1:172.21.15.45:6789/0 2024-04-26T00:26:58.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f832bb32640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83240a5150 con 0x7f8324133b20 2024-04-26T00:26:58.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f832bb32640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:58.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f832bb32640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f832409b280 con 0x7f83240a4170 2024-04-26T00:26:58.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f832bb32640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:58.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f832bb32640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f832409b560 con 0x7f83240a3d90 2024-04-26T00:26:58.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f832bb32640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:58.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f832bb32640 10 monclient(hunting): authenticate will time out at 1400.261869s 2024-04-26T00:26:58.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f83290cd640 1 -- >> v2:172.21.15.37:3300/0 conn(0x7f83240a4170 msgr2=0x7f83240a45b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:26:58.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f83290cd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f83240a4170 0x7f83240a45b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:26:58.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f83298ce640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f83240a3d90 0x7f832409bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:58.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f83298ce640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f83240a3d90 0x7f832409bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:44498/0 (socket says 172.21.15.37:44498) 2024-04-26T00:26:58.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f83298ce640 1 -- 172.21.15.37:0/2929387007 learned_addr learned my addr 172.21.15.37:0/2929387007 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:26:58.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f83298ce640 10 monclient(hunting): get_auth_request con 0x7f83240a3d90 auth_method 0 2024-04-26T00:26:58.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f83298ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:58.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f83298ce640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:58.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f83298ce640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:58.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f83298ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:58.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f83298ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:58.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f83298ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:58.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f83298ce640 10 monclient(hunting): handle_auth_done global_id 84155 payload 274 2024-04-26T00:26:58.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f83298ce640 1 -- 172.21.15.37:0/2929387007 >> v2:172.21.15.37:3300/0 conn(0x7f83240a4170 msgr2=0x7f83240a45b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:26:58.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f83298ce640 1 --2- 172.21.15.37:0/2929387007 >> v2:172.21.15.37:3300/0 conn(0x7f83240a4170 0x7f83240a45b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:58.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f83298ce640 1 -- 172.21.15.37:0/2929387007 >> v1:172.21.15.45:6789/0 conn(0x7f8324133b20 legacy=0x7f8324133f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:26:58.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f83298ce640 10 monclient: _finish_hunting 0 2024-04-26T00:26:58.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f83298ce640 1 monclient: found mon.noname-b 2024-04-26T00:26:58.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f83298ce640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:58.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f83298ce640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:26:58.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.952+0000 7f83298ce640 1 -- 172.21.15.37:0/2929387007 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83240a4af0 con 0x7f83240a3d90 2024-04-26T00:26:58.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.956+0000 7f83298ce640 1 -- 172.21.15.37:0/2929387007 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f83240a3d90 msgr2=0x7f832409bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:58.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.956+0000 7f83298ce640 1 --2- 172.21.15.37:0/2929387007 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f83240a3d90 0x7f832409bf40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8314009e50 tx=0x7f831401c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c0af658b73c25bd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:58.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.956+0000 7f83288cc640 1 -- 172.21.15.37:0/2929387007 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83140a5660 con 0x7f83240a3d90 2024-04-26T00:26:58.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.956+0000 7f83288cc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:58.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.956+0000 7f83288cc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:26:58.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.956+0000 7f83288cc640 10 monclient: dump: 2024-04-26T00:26:58.945 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:58.945 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:58.945 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:58.945 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:58.945 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:58.946 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:58.946 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:58.946 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:58.946 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:58.946 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:58.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.956+0000 7f83288cc640 10 monclient: _finish_auth 0 2024-04-26T00:26:58.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.956+0000 7f83288cc640 10 monclient: _check_auth_tickets 2024-04-26T00:26:58.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.956+0000 7f83288cc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:58.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.956+0000 7f832bb32640 5 monclient: authenticate success, global_id 84155 2024-04-26T00:26:58.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.956+0000 7f83288cc640 1 -- 172.21.15.37:0/2929387007 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83140a5800 con 0x7f83240a3d90 2024-04-26T00:26:58.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.956+0000 7f83288cc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:58.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.956+0000 7f832bb32640 10 monclient: get_monmap_and_config success 2024-04-26T00:26:58.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.956+0000 7f83288cc640 1 -- 172.21.15.37:0/2929387007 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83140a5bd0 con 0x7f83240a3d90 2024-04-26T00:26:58.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.956+0000 7f832bb32640 4 set_mon_vals no callback set 2024-04-26T00:26:58.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.956+0000 7f83288cc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:58.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.956+0000 7f83288cc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:26:58.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.956+0000 7f83288cc640 10 monclient: dump: 2024-04-26T00:26:58.947 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:58.947 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:58.947 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:58.947 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:58.947 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:58.947 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:58.947 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:58.947 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:58.947 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:58.947 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:58.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.956+0000 7f832bb32640 10 monclient: shutdown 2024-04-26T00:26:58.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.956+0000 7f832bb32640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:58.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.956+0000 7f832bb32640 1 -- 172.21.15.37:0/2929387007 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f83240a3d90 msgr2=0x7f832409bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:58.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.956+0000 7f832bb32640 1 --2- 172.21.15.37:0/2929387007 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f83240a3d90 0x7f832409bf40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8314009e50 tx=0x7f831401c8d0 comp rx=0 tx=0).stop 2024-04-26T00:26:58.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.956+0000 7f832bb32640 1 -- 172.21.15.37:0/2929387007 shutdown_connections 2024-04-26T00:26:58.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.956+0000 7f832bb32640 1 --2- 172.21.15.37:0/2929387007 >> v2:172.21.15.37:3300/0 conn(0x7f83240a4170 0x7f83240a45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:58.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.956+0000 7f832bb32640 1 --2- 172.21.15.37:0/2929387007 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f83240a3d90 0x7f832409bf40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:58.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.956+0000 7f832bb32640 1 -- 172.21.15.37:0/2929387007 >> 172.21.15.37:0/2929387007 conn(0x7f83240aa0a0 msgr2=0x7f83240a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:58.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.956+0000 7f832bb32640 1 -- 172.21.15.37:0/2929387007 shutdown_connections 2024-04-26T00:26:58.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.956+0000 7f832bb32640 1 -- 172.21.15.37:0/2929387007 wait complete. 2024-04-26T00:26:58.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832bb32640 10 monclient: build_initial_monmap 2024-04-26T00:26:58.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832bb32640 10 monclient: monmap: 2024-04-26T00:26:58.948 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:58.948 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:26:58.948 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:26:58.949 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:26:58.949 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:58.949 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:26:58.949 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:26:58.949 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:26:58.949 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:26:58.949 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:58.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832bb32640 1 Processor -- start 2024-04-26T00:26:58.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832bb32640 1 -- start start 2024-04-26T00:26:58.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832bb32640 10 monclient: init 2024-04-26T00:26:58.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832bb32640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:58.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832bb32640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:26:58.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832bb32640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83240a3d90 0x7f83241cf530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:58.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832bb32640 10 monclient(hunting): picked mon.noname-b con 0x7f83240a3d90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:26:58.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832bb32640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83240a4170 0x7f832413e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:58.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832bb32640 10 monclient(hunting): picked mon.noname-a con 0x7f83240a4170 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:26:58.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832bb32640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8324133b20 0x7f832413e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:58.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832bb32640 10 monclient(hunting): picked mon.noname-c con 0x7f8324133b20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:26:58.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832bb32640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:58.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832bb32640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f8324141e30 con 0x7f83240a4170 2024-04-26T00:26:58.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832bb32640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:58.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832bb32640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f832409c690 con 0x7f83240a3d90 2024-04-26T00:26:58.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832bb32640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:58.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832bb32640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8324141fb0 con 0x7f8324133b20 2024-04-26T00:26:58.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832bb32640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:58.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832bb32640 10 monclient(hunting): authenticate will time out at 1400.269103s 2024-04-26T00:26:58.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f83290cd640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83240a4170 msgr2=0x7f832413e110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:26:58.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f83290cd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83240a4170 0x7f832413e110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:26:58.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832a0cf640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8324133b20 0x7f832413e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:58.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832a0cf640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8324133b20 0x7f832413e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:44506/0 (socket says 172.21.15.37:44506) 2024-04-26T00:26:58.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832a0cf640 1 -- 172.21.15.37:0/3703436759 learned_addr learned my addr 172.21.15.37:0/3703436759 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:26:58.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832a0cf640 10 monclient(hunting): get_auth_request con 0x7f8324133b20 auth_method 0 2024-04-26T00:26:58.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832a0cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:58.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832a0cf640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:58.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832a0cf640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:58.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832a0cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:58.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832a0cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:58.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832a0cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:58.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832a0cf640 10 monclient(hunting): handle_auth_done global_id 84158 payload 824 2024-04-26T00:26:58.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f83298ce640 1 --2- 172.21.15.37:0/3703436759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83240a3d90 0x7f83241cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:58.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832a0cf640 1 -- 172.21.15.37:0/3703436759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83240a3d90 msgr2=0x7f83241cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:58.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832a0cf640 1 --2- 172.21.15.37:0/3703436759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f83240a3d90 0x7f83241cf530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:58.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832a0cf640 1 -- 172.21.15.37:0/3703436759 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83240a4170 msgr2=0x7f832413e110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:26:58.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832a0cf640 1 --2- 172.21.15.37:0/3703436759 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83240a4170 0x7f832413e110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:58.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832a0cf640 10 monclient: _finish_hunting 0 2024-04-26T00:26:58.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832a0cf640 1 monclient: found mon.noname-c 2024-04-26T00:26:58.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832a0cf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:58.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832a0cf640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:26:58.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832a0cf640 1 -- 172.21.15.37:0/3703436759 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f832409ada0 con 0x7f8324133b20 2024-04-26T00:26:58.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832a0cf640 1 -- 172.21.15.37:0/3703436759 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8324133b20 msgr2=0x7f832413e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:58.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.960+0000 7f832a0cf640 1 --2- 172.21.15.37:0/3703436759 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8324133b20 0x7f832413e650 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f831c00c3c0 tx=0x7f831c00c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:58.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f83227fc640 1 -- 172.21.15.37:0/3703436759 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f831c0119a0 con 0x7f8324133b20 2024-04-26T00:26:58.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f83227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:58.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f83227fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:26:58.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f83227fc640 10 monclient: dump: 2024-04-26T00:26:58.957 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:58.957 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:58.957 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:58.957 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:58.957 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:58.957 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:58.957 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:58.957 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:58.957 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:58.957 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:58.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f83227fc640 10 monclient: _finish_auth 0 2024-04-26T00:26:58.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f83227fc640 10 monclient: _check_auth_tickets 2024-04-26T00:26:58.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f83227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:58.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f83227fc640 1 -- 172.21.15.37:0/3703436759 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f831c017070 con 0x7f8324133b20 2024-04-26T00:26:58.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f832bb32640 5 monclient: authenticate success, global_id 84158 2024-04-26T00:26:58.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f832bb32640 10 monclient: _renew_subs 2024-04-26T00:26:58.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f832bb32640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:26:58.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f832bb32640 1 -- 172.21.15.37:0/3703436759 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f832413cbe0 con 0x7f8324133b20 2024-04-26T00:26:58.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f83227fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:58.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f83227fc640 1 -- 172.21.15.37:0/3703436759 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f831c011d80 con 0x7f8324133b20 2024-04-26T00:26:58.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f83227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:58.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f83227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:26:58.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f83227fc640 10 monclient: dump: 2024-04-26T00:26:58.959 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:58.959 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:58.959 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:58.959 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:58.959 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:58.959 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:58.959 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:58.959 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:58.959 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:58.959 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:58.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f8323fff640 4 set_mon_vals no callback set 2024-04-26T00:26:58.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f832bb32640 10 monclient: _renew_subs 2024-04-26T00:26:58.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f832bb32640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:26:58.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f832bb32640 1 -- 172.21.15.37:0/3703436759 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f832413a8f0 con 0x7f8324133b20 2024-04-26T00:26:58.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f832bb32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:58.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f832bb32640 1 --2- 172.21.15.37:0/3703436759 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f832413eb90 0x7f8324138c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:58.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f832bb32640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:58.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f832bb32640 1 -- 172.21.15.37:0/3703436759 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f832413c580 con 0x7f832413eb90 2024-04-26T00:26:58.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f83298ce640 1 --2- 172.21.15.37:0/3703436759 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f832413eb90 0x7f8324138c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:58.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f83298ce640 10 monclient: get_auth_request con 0x7f832413eb90 auth_method 0 2024-04-26T00:26:58.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f83298ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:58.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f83298ce640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:58.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f83298ce640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:58.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f83227fc640 1 -- 172.21.15.37:0/3703436759 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f831c01c460 con 0x7f8324133b20 2024-04-26T00:26:58.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f83298ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:58.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f83298ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:58.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f83298ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:58.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f83227fc640 1 --2- 172.21.15.37:0/3703436759 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8310035e80 0x7f8310038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:58.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f83298ce640 10 monclient(hunting): handle_auth_done global_id 84161 payload 824 2024-04-26T00:26:58.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f83227fc640 1 -- 172.21.15.37:0/3703436759 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(264..264 src has 177..264) ==== 3979+0+0 (secure 0 0 0) 0x7f831c048930 con 0x7f8324133b20 2024-04-26T00:26:58.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f83298ce640 1 -- 172.21.15.37:0/3703436759 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f832404f510 con 0x7f832413eb90 2024-04-26T00:26:58.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f83290cd640 1 --2- 172.21.15.37:0/3703436759 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8310035e80 0x7f8310038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:58.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f83290cd640 10 monclient: get_auth_request con 0x7f8310035e80 auth_method 0 2024-04-26T00:26:58.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f83298ce640 1 -- 172.21.15.37:0/3703436759 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f832413eb90 msgr2=0x7f8324138c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:58.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f83298ce640 1 --2- 172.21.15.37:0/3703436759 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f832413eb90 0x7f8324138c50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f83140ae000 tx=0x7f8314019670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:58.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f83227fc640 1 -- 172.21.15.37:0/3703436759 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83140bd020 con 0x7f832413eb90 2024-04-26T00:26:58.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f83227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:58.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f83227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:26:58.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.964+0000 7f83227fc640 10 monclient: dump: 2024-04-26T00:26:58.961 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:58.961 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:58.961 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:58.961 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:58.962 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:58.962 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:58.962 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:58.962 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:58.962 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:58.962 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:58.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.968+0000 7f83227fc640 1 -- 172.21.15.37:0/3703436759 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f83140b9670 con 0x7f832413eb90 2024-04-26T00:26:58.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.968+0000 7f83227fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:58.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.968+0000 7f83227fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:26:58.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.968+0000 7f83227fc640 1 -- 172.21.15.37:0/3703436759 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f832413eb90 msgr2=0x7f8324138c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:58.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.968+0000 7f83227fc640 1 --2- 172.21.15.37:0/3703436759 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f832413eb90 0x7f8324138c50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f83140ae000 tx=0x7f8314019670 comp rx=0 tx=0).stop 2024-04-26T00:26:58.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.968+0000 7f83227fc640 1 -- 172.21.15.37:0/3703436759 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f832413eb90 msgr2=0x7f8324138c50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:26:58.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.968+0000 7f83227fc640 1 --2- 172.21.15.37:0/3703436759 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f832413eb90 0x7f8324138c50 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f83140ae000 tx=0x7f8314019670 comp rx=0 tx=0).stop 2024-04-26T00:26:58.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.968+0000 7f83290cd640 1 --2- 172.21.15.37:0/3703436759 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8310035e80 0x7f8310038340 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8318006fd0 tx=0x7f8318006d40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:58.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.972+0000 7f832bb32640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:26:58.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.972+0000 7f832bb32640 1 --2- 172.21.15.37:0/3703436759 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8324139190 0x7f83241395a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:58.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.972+0000 7f832bb32640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:58.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.972+0000 7f832bb32640 1 -- 172.21.15.37:0/3703436759 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8324139ae0 con 0x7f8324139190 2024-04-26T00:26:58.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.972+0000 7f83298ce640 1 --2- 172.21.15.37:0/3703436759 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8324139190 0x7f83241395a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:58.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.972+0000 7f83298ce640 10 monclient: get_auth_request con 0x7f8324139190 auth_method 0 2024-04-26T00:26:58.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.972+0000 7f83298ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:58.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.972+0000 7f83298ce640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:58.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.972+0000 7f83298ce640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:58.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.972+0000 7f83298ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:58.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.972+0000 7f83298ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:58.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.972+0000 7f83298ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:58.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.972+0000 7f83298ce640 10 monclient(hunting): handle_auth_done global_id 84164 payload 824 2024-04-26T00:26:58.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.972+0000 7f83298ce640 1 -- 172.21.15.37:0/3703436759 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f832413c960 con 0x7f8324139190 2024-04-26T00:26:58.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.976+0000 7f83298ce640 1 -- 172.21.15.37:0/3703436759 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8324139190 msgr2=0x7f83241395a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:58.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.976+0000 7f83298ce640 1 --2- 172.21.15.37:0/3703436759 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8324139190 0x7f83241395a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8314009e50 tx=0x7f83140a7000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:58.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.976+0000 7f83227fc640 1 -- 172.21.15.37:0/3703436759 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83140bd030 con 0x7f8324139190 2024-04-26T00:26:58.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.976+0000 7f83227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:58.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.976+0000 7f83227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:26:58.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.976+0000 7f83227fc640 10 monclient: dump: 2024-04-26T00:26:58.965 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:58.966 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:58.966 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:58.966 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:58.966 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:58.966 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:58.966 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:58.966 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:58.966 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:58.966 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:58.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.976+0000 7f83227fc640 1 -- 172.21.15.37:0/3703436759 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4705 (secure 0 0 0) 0x7f83140c4860 con 0x7f8324139190 2024-04-26T00:26:58.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.976+0000 7f83227fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:26:58.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.976+0000 7f83227fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:26:58.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.976+0000 7f83227fc640 1 -- 172.21.15.37:0/3703436759 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8324139190 msgr2=0x7f83241395a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:58.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.976+0000 7f83227fc640 1 --2- 172.21.15.37:0/3703436759 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8324139190 0x7f83241395a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8314009e50 tx=0x7f83140a7000 comp rx=0 tx=0).stop 2024-04-26T00:26:58.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.976+0000 7f83227fc640 1 -- 172.21.15.37:0/3703436759 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8324139190 msgr2=0x7f83241395a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:26:58.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.976+0000 7f83227fc640 1 --2- 172.21.15.37:0/3703436759 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8324139190 0x7f83241395a0 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8314009e50 tx=0x7f83140a7000 comp rx=0 tx=0).stop 2024-04-26T00:26:58.967 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:26:58.967 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:26:58.967 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:26:58.967 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:26:58.967 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 56, 2024-04-26T00:26:58.967 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 23911, 2024-04-26T00:26:58.967 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:26:58.968 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:26:58.968 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:26:58.968 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:58.968 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 1, 2024-04-26T00:26:58.968 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:26:58.968 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:26:58.968 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:26:58.968 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:26:58.968 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:26:58.968 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:26:58.968 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:26:58.968 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:26:58.968 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:26:58.968 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:26:58.969 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:26:58.969 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:26:58.969 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:26:58.969 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:26:58.969 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:58.969 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:26:58.969 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:26:58.969 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:26:58.969 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:26:58.969 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:26:58.969 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:26:58.969 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:26:58.969 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:26:58.969 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:26:58.969 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:26:58.970 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:26:58.970 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:26:58.970 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:26:58.970 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:58.970 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:58.970 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:26:58.970 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:26:58.970 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:26:58.970 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:26:58.970 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:26:58.970 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:26:58.970 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:26:58.970 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:26:58.970 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:26:58.971 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:26:58.971 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:26:58.971 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:26:58.971 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:26:58.971 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:26:58.971 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:26:58.971 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:26:58.971 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:26:58.971 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:26:58.971 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:26:58.971 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:26:58.971 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:26:58.971 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:26:58.971 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:26:58.972 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:26:58.972 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:26:58.972 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:26:58.972 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:26:58.972 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:26:58.972 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:58.972 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:26:58.972 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:58.972 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:26:58.972 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:58.972 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:26:58.972 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:26:58.972 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:26:58.972 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:26:58.972 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:58.973 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:26:58.973 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:26:58.973 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:58.973 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:58.973 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:58.973 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:26:58.973 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:26:58.973 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:58.973 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:58.973 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:58.973 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:58.973 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:26:58.973 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:26:58.973 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:26:58.974 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:26:58.974 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:26:58.974 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:58.974 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:58.974 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:26:58.974 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:26:58.974 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:26:58.974 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:26:58.974 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:58.974 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:26:58.974 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:26:58.974 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:58.974 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:58.974 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:58.975 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:26:58.975 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:26:58.975 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:58.975 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:58.975 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:58.975 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:58.975 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:26:58.975 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:26:58.975 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:26:58.975 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:26:58.975 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:26:58.975 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:58.975 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:58.975 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:26:58.975 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:26:58.976 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:26:58.976 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:26:58.976 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:58.976 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:26:58.976 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:26:58.976 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:58.976 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:58.976 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:58.976 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:26:58.976 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:26:58.976 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:58.976 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:58.976 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:58.976 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:58.977 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:26:58.977 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:26:58.977 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:26:58.977 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:26:58.977 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:26:58.977 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:58.977 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:58.977 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:58.977 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:26:58.977 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:26:58.977 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:58.977 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:26:58.977 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:26:58.977 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:26:58.977 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:58.978 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:58.978 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:26:58.978 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:58.978 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:26:58.978 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:26:58.978 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:26:58.978 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:58.978 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:58.978 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:26:58.978 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:58.978 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:26:58.978 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:26:58.978 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:26:58.978 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:58.979 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:58.979 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:58.979 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:26:58.979 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:26:58.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.976+0000 7f832bb32640 1 -- 172.21.15.37:0/3703436759 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8310035e80 msgr2=0x7f8310038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:58.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.976+0000 7f832bb32640 1 --2- 172.21.15.37:0/3703436759 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8310035e80 0x7f8310038340 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8318006fd0 tx=0x7f8318006d40 comp rx=0 tx=0).stop 2024-04-26T00:26:58.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.976+0000 7f832bb32640 10 monclient: shutdown 2024-04-26T00:26:58.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.976+0000 7f832bb32640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:58.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.976+0000 7f832bb32640 1 -- 172.21.15.37:0/3703436759 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8324133b20 msgr2=0x7f832413e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:58.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.976+0000 7f832bb32640 1 --2- 172.21.15.37:0/3703436759 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8324133b20 0x7f832413e650 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f831c00c3c0 tx=0x7f831c00c880 comp rx=0 tx=0).stop 2024-04-26T00:26:58.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.976+0000 7f832a0cf640 1 -- 172.21.15.37:0/3703436759 reap_dead start 2024-04-26T00:26:58.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.976+0000 7f832a0cf640 1 -- 172.21.15.37:0/3703436759 reap_dead start 2024-04-26T00:26:58.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.976+0000 7f832bb32640 1 -- 172.21.15.37:0/3703436759 shutdown_connections 2024-04-26T00:26:58.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.976+0000 7f832bb32640 1 -- 172.21.15.37:0/3703436759 >> 172.21.15.37:0/3703436759 conn(0x7f83240aa0a0 msgr2=0x7f83240a1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:58.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.976+0000 7f832bb32640 1 -- 172.21.15.37:0/3703436759 shutdown_connections 2024-04-26T00:26:58.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:58.976+0000 7f832bb32640 1 -- 172.21.15.37:0/3703436759 wait complete. 2024-04-26T00:26:58.981 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:26:58.981 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-04-26T00:26:58.981 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:26:59.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3ef29e2640 10 monclient: get_monmap_and_config 2024-04-26T00:26:59.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3ef29e2640 10 monclient: build_initial_monmap 2024-04-26T00:26:59.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3ef29e2640 10 monclient: monmap: 2024-04-26T00:26:59.098 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:59.098 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:59.098 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:26:59.116504+0000 2024-04-26T00:26:59.098 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:26:59.116504+0000 2024-04-26T00:26:59.098 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:59.098 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:59.098 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:26:59.098 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:26:59.098 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:26:59.098 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:26:59.098 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:26:59.098 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:59.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3ef29e2640 1 Processor -- start 2024-04-26T00:26:59.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3ef29e2640 1 -- start start 2024-04-26T00:26:59.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3ef29e2640 10 monclient: init 2024-04-26T00:26:59.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3ef29e2640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:59.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3ef29e2640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-04-26T00:26:59.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3ef29e2640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f3eec13b6e0 0x7f3eec09de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:59.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3ef29e2640 10 monclient(hunting): picked mon.noname-a con 0x7f3eec13b6e0 addr v2:172.21.15.37:3300/0 2024-04-26T00:26:59.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3ef29e2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3eec13bac0 addr v1:172.21.15.37:6789/0 2024-04-26T00:26:59.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3ef29e2640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3eec0a0360 0x7f3eec1434f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:59.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3ef29e2640 10 monclient(hunting): picked mon.noname-c con 0x7f3eec0a0360 addr v2:172.21.15.45:3300/0 2024-04-26T00:26:59.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3ef29e2640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3eec0732a0 con 0x7f3eec13bac0 2024-04-26T00:26:59.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3ef29e2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:59.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3ef29e2640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f3eec09cb90 con 0x7f3eec13b6e0 2024-04-26T00:26:59.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3ef29e2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:59.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3ef29e2640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f3eec09c140 con 0x7f3eec0a0360 2024-04-26T00:26:59.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3ef29e2640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:59.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3ef29e2640 10 monclient(hunting): authenticate will time out at 1400.422549s 2024-04-26T00:26:59.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3eebfff640 1 -- >> v2:172.21.15.37:3300/0 conn(0x7f3eec13b6e0 msgr2=0x7f3eec09de30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:26:59.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3eebfff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f3eec13b6e0 0x7f3eec09de30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:26:59.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3eeb7fe640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f3eec13bac0 legacy=0x7f3eec09fc50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:26:59.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3ef0f7f640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3eec0a0360 0x7f3eec1434f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:59.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3ef0f7f640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3eec0a0360 0x7f3eec1434f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.37:42918/0 (socket says 172.21.15.37:42918) 2024-04-26T00:26:59.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3ef0f7f640 1 -- 172.21.15.37:0/3429492655 learned_addr learned my addr 172.21.15.37:0/3429492655 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:26:59.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3ef0f7f640 10 monclient(hunting): get_auth_request con 0x7f3eec0a0360 auth_method 0 2024-04-26T00:26:59.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3ef0f7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:59.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3ef0f7f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:59.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3ef0f7f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:59.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3ef0f7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:59.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3ef0f7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:59.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3ef0f7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:59.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3ef0f7f640 10 monclient(hunting): handle_auth_done global_id 24661 payload 274 2024-04-26T00:26:59.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3ef0f7f640 1 -- 172.21.15.37:0/3429492655 >> v2:172.21.15.37:3300/0 conn(0x7f3eec13b6e0 msgr2=0x7f3eec09de30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:26:59.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3ef0f7f640 1 --2- 172.21.15.37:0/3429492655 >> v2:172.21.15.37:3300/0 conn(0x7f3eec13b6e0 0x7f3eec09de30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:59.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3ef0f7f640 1 -- 172.21.15.37:0/3429492655 >> v1:172.21.15.37:6789/0 conn(0x7f3eec13bac0 legacy=0x7f3eec09fc50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:26:59.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3ef0f7f640 10 monclient: _finish_hunting 0 2024-04-26T00:26:59.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3ef0f7f640 1 monclient: found mon.noname-c 2024-04-26T00:26:59.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3ef0f7f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:59.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3ef0f7f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:26:59.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.112+0000 7f3ef0f7f640 1 -- 172.21.15.37:0/3429492655 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3eec0a10c0 con 0x7f3eec0a0360 2024-04-26T00:26:59.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.120+0000 7f3ef0f7f640 1 -- 172.21.15.37:0/3429492655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3eec0a0360 msgr2=0x7f3eec1434f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:59.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.120+0000 7f3ef0f7f640 1 --2- 172.21.15.37:0/3429492655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3eec0a0360 0x7f3eec1434f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f3eec09f8e0 tx=0x7f3ee401c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f05e403ab620491a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:59.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.120+0000 7f3eeaffd640 1 -- 172.21.15.37:0/3429492655 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee40b2070 con 0x7f3eec0a0360 2024-04-26T00:26:59.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.120+0000 7f3eeaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:59.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.120+0000 7f3eeaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:26:59.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.120+0000 7f3eeaffd640 10 monclient: dump: 2024-04-26T00:26:59.107 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:59.107 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:59.107 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:59.107 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:59.107 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:59.107 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:59.107 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:59.107 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:59.108 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:59.108 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:59.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.120+0000 7f3eeaffd640 10 monclient: _finish_auth 0 2024-04-26T00:26:59.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.120+0000 7f3eeaffd640 10 monclient: _check_auth_tickets 2024-04-26T00:26:59.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.120+0000 7f3eeaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:59.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.120+0000 7f3eeaffd640 1 -- 172.21.15.37:0/3429492655 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ee40a9070 con 0x7f3eec0a0360 2024-04-26T00:26:59.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.120+0000 7f3eeaffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:59.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.120+0000 7f3eeaffd640 1 -- 172.21.15.37:0/3429492655 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee40ad880 con 0x7f3eec0a0360 2024-04-26T00:26:59.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.120+0000 7f3eeaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:59.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.120+0000 7f3eeaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:26:59.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.120+0000 7f3eeaffd640 10 monclient: dump: 2024-04-26T00:26:59.108 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:59.108 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:59.108 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:59.108 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:59.109 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:59.109 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:59.109 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:59.109 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:59.109 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:59.109 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:59.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.120+0000 7f3ef29e2640 5 monclient: authenticate success, global_id 24661 2024-04-26T00:26:59.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.120+0000 7f3ef29e2640 10 monclient: get_monmap_and_config success 2024-04-26T00:26:59.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.120+0000 7f3ef29e2640 4 set_mon_vals no callback set 2024-04-26T00:26:59.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.120+0000 7f3ef29e2640 10 monclient: shutdown 2024-04-26T00:26:59.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.120+0000 7f3ef29e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:59.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.120+0000 7f3ef29e2640 1 -- 172.21.15.37:0/3429492655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3eec0a0360 msgr2=0x7f3eec1434f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:59.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.120+0000 7f3ef29e2640 1 --2- 172.21.15.37:0/3429492655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3eec0a0360 0x7f3eec1434f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f3eec09f8e0 tx=0x7f3ee401c450 comp rx=0 tx=0).stop 2024-04-26T00:26:59.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.120+0000 7f3ef29e2640 1 -- 172.21.15.37:0/3429492655 shutdown_connections 2024-04-26T00:26:59.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.120+0000 7f3ef29e2640 1 --2- 172.21.15.37:0/3429492655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3eec0a0360 0x7f3eec1434f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:59.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.120+0000 7f3ef29e2640 1 --2- 172.21.15.37:0/3429492655 >> v2:172.21.15.37:3300/0 conn(0x7f3eec13b6e0 0x7f3eec09de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:59.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.120+0000 7f3ef29e2640 1 -- 172.21.15.37:0/3429492655 >> 172.21.15.37:0/3429492655 conn(0x7f3eec0a1d50 msgr2=0x7f3eec0a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:59.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.120+0000 7f3ef29e2640 1 -- 172.21.15.37:0/3429492655 shutdown_connections 2024-04-26T00:26:59.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.120+0000 7f3ef29e2640 1 -- 172.21.15.37:0/3429492655 wait complete. 2024-04-26T00:26:59.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.120+0000 7f3ef29e2640 10 monclient: build_initial_monmap 2024-04-26T00:26:59.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.120+0000 7f3ef29e2640 10 monclient: monmap: 2024-04-26T00:26:59.115 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:26:59.115 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:26:59.115 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:26:59.115 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:26:59.115 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:26:59.115 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:26:59.115 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:26:59.115 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:26:59.115 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:26:59.115 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:59.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef29e2640 1 Processor -- start 2024-04-26T00:26:59.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef29e2640 1 -- start start 2024-04-26T00:26:59.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef29e2640 10 monclient: init 2024-04-26T00:26:59.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef29e2640 10 monclient: _reopen_session rank -1 2024-04-26T00:26:59.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef29e2640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:26:59.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef29e2640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3eec0a0360 0x7f3eec1d3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:59.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef29e2640 10 monclient(hunting): picked mon.noname-a con 0x7f3eec0a0360 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:26:59.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef29e2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3eec13b6e0 0x7f3eec13f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:59.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef29e2640 10 monclient(hunting): picked mon.noname-b con 0x7f3eec13b6e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:26:59.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef29e2640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3eec13bac0 0x7f3eec13f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:59.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef29e2640 10 monclient(hunting): picked mon.noname-c con 0x7f3eec13bac0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:26:59.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef29e2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:59.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef29e2640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f3eec13c0e0 con 0x7f3eec0a0360 2024-04-26T00:26:59.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef29e2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:59.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef29e2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3eec09be60 con 0x7f3eec13b6e0 2024-04-26T00:26:59.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef29e2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:59.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef29e2640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f3eec146460 con 0x7f3eec13bac0 2024-04-26T00:26:59.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3eebfff640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3eec0a0360 msgr2=0x7f3eec1d3c80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:26:59.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef29e2640 10 monclient(hunting): _renew_subs 2024-04-26T00:26:59.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef29e2640 10 monclient(hunting): authenticate will time out at 1400.432658s 2024-04-26T00:26:59.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3eebfff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3eec0a0360 0x7f3eec1d3c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:26:59.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef0f7f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3eec13bac0 0x7f3eec13f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:59.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef0f7f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3eec13bac0 0x7f3eec13f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:44532/0 (socket says 172.21.15.37:44532) 2024-04-26T00:26:59.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef0f7f640 1 -- 172.21.15.37:0/1029742174 learned_addr learned my addr 172.21.15.37:0/1029742174 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:26:59.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef0f7f640 10 monclient(hunting): get_auth_request con 0x7f3eec13bac0 auth_method 0 2024-04-26T00:26:59.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef0f7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:59.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef0f7f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:59.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef0f7f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:59.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef0f7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:59.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef0f7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:59.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef0f7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:59.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3eeb7fe640 1 --2- 172.21.15.37:0/1029742174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3eec13b6e0 0x7f3eec13f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:59.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef0f7f640 10 monclient(hunting): handle_auth_done global_id 84167 payload 824 2024-04-26T00:26:59.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef0f7f640 1 -- 172.21.15.37:0/1029742174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3eec13b6e0 msgr2=0x7f3eec13f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:59.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef0f7f640 1 --2- 172.21.15.37:0/1029742174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3eec13b6e0 0x7f3eec13f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:59.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef0f7f640 1 -- 172.21.15.37:0/1029742174 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3eec0a0360 msgr2=0x7f3eec1d3c80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:26:59.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef0f7f640 1 --2- 172.21.15.37:0/1029742174 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3eec0a0360 0x7f3eec1d3c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:59.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef0f7f640 10 monclient: _finish_hunting 0 2024-04-26T00:26:59.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef0f7f640 1 monclient: found mon.noname-c 2024-04-26T00:26:59.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef0f7f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:26:59.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef0f7f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:26:59.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef0f7f640 1 -- 172.21.15.37:0/1029742174 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3eec1429b0 con 0x7f3eec13bac0 2024-04-26T00:26:59.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef0f7f640 1 -- 172.21.15.37:0/1029742174 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3eec13bac0 msgr2=0x7f3eec13f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:59.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef0f7f640 1 --2- 172.21.15.37:0/1029742174 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3eec13bac0 0x7f3eec13f5c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3ee4019060 tx=0x7f3ee40191f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:59.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ee8ff9640 1 -- 172.21.15.37:0/1029742174 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee40bb020 con 0x7f3eec13bac0 2024-04-26T00:26:59.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ee8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:59.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ee8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:26:59.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ee8ff9640 10 monclient: dump: 2024-04-26T00:26:59.119 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:59.119 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:59.119 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:59.119 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:59.119 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:59.119 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:59.119 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:59.119 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:59.119 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:59.119 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:59.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ee8ff9640 10 monclient: _finish_auth 0 2024-04-26T00:26:59.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ee8ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:26:59.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ee8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:26:59.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef29e2640 5 monclient: authenticate success, global_id 84167 2024-04-26T00:26:59.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef29e2640 10 monclient: _renew_subs 2024-04-26T00:26:59.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ee8ff9640 1 -- 172.21.15.37:0/1029742174 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ee40b3040 con 0x7f3eec13bac0 2024-04-26T00:26:59.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef29e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:26:59.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef29e2640 1 -- 172.21.15.37:0/1029742174 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3eec142c90 con 0x7f3eec13bac0 2024-04-26T00:26:59.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ee8ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:26:59.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ee8ff9640 1 -- 172.21.15.37:0/1029742174 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee40b2050 con 0x7f3eec13bac0 2024-04-26T00:26:59.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ee8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:59.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ee8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:26:59.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ee8ff9640 10 monclient: dump: 2024-04-26T00:26:59.121 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:59.121 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:59.121 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:59.121 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:59.121 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:59.121 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:59.121 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:59.121 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:59.121 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:59.121 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:59.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3eea7fc640 4 set_mon_vals no callback set 2024-04-26T00:26:59.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef29e2640 10 monclient: _renew_subs 2024-04-26T00:26:59.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef29e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:26:59.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.124+0000 7f3ef29e2640 1 -- 172.21.15.37:0/1029742174 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3eec1399d0 con 0x7f3eec13bac0 2024-04-26T00:26:59.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.128+0000 7f3ef29e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:59.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.128+0000 7f3ef29e2640 1 --2- 172.21.15.37:0/1029742174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3eec1d41c0 0x7f3eec13fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:59.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.128+0000 7f3ef29e2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:59.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.128+0000 7f3ef29e2640 1 -- 172.21.15.37:0/1029742174 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3eec04f510 con 0x7f3eec1d41c0 2024-04-26T00:26:59.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.128+0000 7f3ee8ff9640 1 -- 172.21.15.37:0/1029742174 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f3ee40be020 con 0x7f3eec13bac0 2024-04-26T00:26:59.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.128+0000 7f3ee8ff9640 1 --2- 172.21.15.37:0/1029742174 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3ecc035e30 0x7f3ecc0382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:59.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.128+0000 7f3eebfff640 1 --2- 172.21.15.37:0/1029742174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3eec1d41c0 0x7f3eec13fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:59.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.128+0000 7f3eebfff640 10 monclient: get_auth_request con 0x7f3eec1d41c0 auth_method 0 2024-04-26T00:26:59.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.128+0000 7f3ee8ff9640 1 -- 172.21.15.37:0/1029742174 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(264..264 src has 177..264) ==== 3979+0+0 (secure 0 0 0) 0x7f3ee40e2730 con 0x7f3eec13bac0 2024-04-26T00:26:59.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.128+0000 7f3eebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:59.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.128+0000 7f3eebfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:59.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.128+0000 7f3eebfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:59.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.128+0000 7f3eebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:59.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.128+0000 7f3eebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:59.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.128+0000 7f3eebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:59.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.128+0000 7f3eeb7fe640 1 --2- 172.21.15.37:0/1029742174 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3ecc035e30 0x7f3ecc0382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:59.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.128+0000 7f3eeb7fe640 10 monclient: get_auth_request con 0x7f3ecc035e30 auth_method 0 2024-04-26T00:26:59.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.128+0000 7f3eebfff640 10 monclient(hunting): handle_auth_done global_id 24664 payload 824 2024-04-26T00:26:59.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.128+0000 7f3eebfff640 1 -- 172.21.15.37:0/1029742174 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3eec0a0e60 con 0x7f3eec1d41c0 2024-04-26T00:26:59.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.128+0000 7f3eebfff640 1 -- 172.21.15.37:0/1029742174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3eec1d41c0 msgr2=0x7f3eec13fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:59.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.128+0000 7f3eebfff640 1 --2- 172.21.15.37:0/1029742174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3eec1d41c0 0x7f3eec13fb00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3ed400b450 tx=0x7f3ed400b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:59.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.132+0000 7f3eeb7fe640 1 --2- 172.21.15.37:0/1029742174 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3ecc035e30 0x7f3ecc0382f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3edc003d10 tx=0x7f3edc007240 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:59.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.132+0000 7f3ee8ff9640 1 -- 172.21.15.37:0/1029742174 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed4005270 con 0x7f3eec1d41c0 2024-04-26T00:26:59.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.132+0000 7f3ee8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:59.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.132+0000 7f3ee8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:26:59.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.132+0000 7f3ee8ff9640 10 monclient: dump: 2024-04-26T00:26:59.125 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:59.125 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:59.125 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:59.125 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:59.125 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:59.125 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:59.125 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:59.126 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:59.126 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:59.126 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:59.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.132+0000 7f3ee8ff9640 1 -- 172.21.15.37:0/1029742174 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3ed4015070 con 0x7f3eec1d41c0 2024-04-26T00:26:59.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.132+0000 7f3ee8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:26:59.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.132+0000 7f3ee8ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:26:59.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.132+0000 7f3ee8ff9640 1 -- 172.21.15.37:0/1029742174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3eec1d41c0 msgr2=0x7f3eec13fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:59.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.132+0000 7f3ee8ff9640 1 --2- 172.21.15.37:0/1029742174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3eec1d41c0 0x7f3eec13fb00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3ed400b450 tx=0x7f3ed400b910 comp rx=0 tx=0).stop 2024-04-26T00:26:59.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.132+0000 7f3ee8ff9640 1 -- 172.21.15.37:0/1029742174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3eec1d41c0 msgr2=0x7f3eec13fb00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:26:59.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.132+0000 7f3ee8ff9640 1 --2- 172.21.15.37:0/1029742174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3eec1d41c0 0x7f3eec13fb00 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:26:59.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.136+0000 7f3ef29e2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:26:59.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.136+0000 7f3ef29e2640 1 --2- 172.21.15.37:0/1029742174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3eec1cb4a0 0x7f3eec140040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:26:59.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.136+0000 7f3ef29e2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:26:59.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.136+0000 7f3ef29e2640 1 -- 172.21.15.37:0/1029742174 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3eec140c80 con 0x7f3eec1cb4a0 2024-04-26T00:26:59.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.136+0000 7f3eebfff640 1 --2- 172.21.15.37:0/1029742174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3eec1cb4a0 0x7f3eec140040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:26:59.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.136+0000 7f3eebfff640 10 monclient: get_auth_request con 0x7f3eec1cb4a0 auth_method 0 2024-04-26T00:26:59.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.136+0000 7f3eebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:26:59.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.136+0000 7f3eebfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:26:59.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.136+0000 7f3eebfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:26:59.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.140+0000 7f3eebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:26:59.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.140+0000 7f3eebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:26:59.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.140+0000 7f3eebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:26:59.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.140+0000 7f3eebfff640 10 monclient(hunting): handle_auth_done global_id 24667 payload 824 2024-04-26T00:26:59.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.140+0000 7f3eebfff640 1 -- 172.21.15.37:0/1029742174 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3eec06ba50 con 0x7f3eec1cb4a0 2024-04-26T00:26:59.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.140+0000 7f3eebfff640 1 -- 172.21.15.37:0/1029742174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3eec1cb4a0 msgr2=0x7f3eec140040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:26:59.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.140+0000 7f3eebfff640 1 --2- 172.21.15.37:0/1029742174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3eec1cb4a0 0x7f3eec140040 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3ed400b450 tx=0x7f3ed40059e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:26:59.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.140+0000 7f3ee8ff9640 1 -- 172.21.15.37:0/1029742174 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed4010690 con 0x7f3eec1cb4a0 2024-04-26T00:26:59.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.140+0000 7f3ee8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:26:59.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.140+0000 7f3ee8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:26:59.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.140+0000 7f3ee8ff9640 10 monclient: dump: 2024-04-26T00:26:59.129 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:26:59.129 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:26:59.129 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:59.129 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:26:59.129 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:26:59.129 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:26:59.129 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:26:59.129 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:26:59.130 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:26:59.130 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:26:59.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.140+0000 7f3ee8ff9640 1 -- 172.21.15.37:0/1029742174 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4878 (secure 0 0 0) 0x7f3ed4010690 con 0x7f3eec1cb4a0 2024-04-26T00:26:59.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.140+0000 7f3ee8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:26:59.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.140+0000 7f3ee8ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:26:59.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.140+0000 7f3ee8ff9640 1 -- 172.21.15.37:0/1029742174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3eec1cb4a0 msgr2=0x7f3eec140040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:59.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.140+0000 7f3ee8ff9640 1 --2- 172.21.15.37:0/1029742174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3eec1cb4a0 0x7f3eec140040 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3ed400b450 tx=0x7f3ed40059e0 comp rx=0 tx=0).stop 2024-04-26T00:26:59.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.140+0000 7f3ee8ff9640 1 -- 172.21.15.37:0/1029742174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3eec1cb4a0 msgr2=0x7f3eec140040 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:26:59.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.140+0000 7f3ee8ff9640 1 --2- 172.21.15.37:0/1029742174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3eec1cb4a0 0x7f3eec140040 secure :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3ed400b450 tx=0x7f3ed40059e0 comp rx=0 tx=0).stop 2024-04-26T00:26:59.130 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:26:59.130 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:26:59.131 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:26:59.131 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:26:59.131 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 56, 2024-04-26T00:26:59.131 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 327310, 2024-04-26T00:26:59.131 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:26:59.131 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:26:59.131 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:26:59.131 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:59.131 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 1, 2024-04-26T00:26:59.131 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:26:59.131 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:26:59.131 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:26:59.131 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:26:59.131 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:26:59.132 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:26:59.132 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:26:59.132 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:26:59.132 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:26:59.132 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:26:59.132 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:26:59.132 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:26:59.132 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:26:59.132 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:26:59.132 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:59.132 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:26:59.132 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:26:59.132 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:26:59.133 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:26:59.133 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:26:59.133 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:26:59.133 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:26:59.133 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:26:59.133 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:26:59.133 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:26:59.133 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:26:59.133 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:26:59.133 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:26:59.133 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:59.133 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:59.133 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:26:59.133 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:26:59.134 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:26:59.134 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:26:59.134 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:26:59.134 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:26:59.134 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:26:59.134 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:26:59.134 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:26:59.134 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:26:59.134 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:26:59.134 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:26:59.134 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:26:59.134 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:26:59.134 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:26:59.135 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:26:59.135 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:26:59.135 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:26:59.135 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:26:59.135 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:26:59.135 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:26:59.135 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:26:59.135 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:26:59.135 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:26:59.135 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:26:59.135 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:26:59.135 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:26:59.135 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:26:59.135 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:59.136 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:26:59.136 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:59.136 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:26:59.136 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:59.136 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:26:59.136 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:26:59.136 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:26:59.136 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:26:59.136 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:59.136 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:26:59.136 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:26:59.136 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:59.136 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:59.136 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:59.137 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:26:59.137 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:26:59.137 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:59.137 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:59.137 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:59.137 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:59.137 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:26:59.137 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:26:59.137 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:26:59.137 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:26:59.137 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:26:59.137 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:59.137 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:59.137 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:26:59.137 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:26:59.138 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:26:59.138 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:26:59.138 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:59.138 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:26:59.138 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:26:59.138 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:59.138 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:59.138 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:59.138 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:26:59.138 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:26:59.138 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:59.138 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:59.138 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:59.138 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:59.138 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:26:59.139 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:26:59.139 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:26:59.139 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:26:59.139 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:26:59.139 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:59.139 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:59.139 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:26:59.139 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:26:59.139 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:26:59.139 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:26:59.139 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:59.139 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:26:59.139 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:26:59.139 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:59.140 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:59.140 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:59.140 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:26:59.140 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:26:59.140 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:26:59.140 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:59.140 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:59.140 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:59.140 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:26:59.140 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:26:59.140 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:26:59.140 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:26:59.140 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:26:59.140 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:59.140 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:59.141 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:59.141 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:26:59.141 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:26:59.141 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:59.141 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:26:59.141 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:26:59.141 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:26:59.141 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:59.141 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:59.141 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:26:59.141 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:59.141 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:26:59.141 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:26:59.141 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:26:59.141 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:59.141 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:59.142 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:26:59.142 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:59.142 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:26:59.142 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:26:59.142 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:26:59.142 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:59.142 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:26:59.142 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:26:59.142 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:26:59.142 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:26:59.142 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:26:59.142 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:26:59.142 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:26:59.143 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:26:59.143 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:26:59.143 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:26:59.143 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:26:59.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.144+0000 7f3ef29e2640 1 -- 172.21.15.37:0/1029742174 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3ecc035e30 msgr2=0x7f3ecc0382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:59.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.144+0000 7f3ef29e2640 1 --2- 172.21.15.37:0/1029742174 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3ecc035e30 0x7f3ecc0382f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3edc003d10 tx=0x7f3edc007240 comp rx=0 tx=0).stop 2024-04-26T00:26:59.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.144+0000 7f3ef29e2640 10 monclient: shutdown 2024-04-26T00:26:59.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.144+0000 7f3ef29e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:26:59.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.144+0000 7f3ef29e2640 1 -- 172.21.15.37:0/1029742174 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3eec13bac0 msgr2=0x7f3eec13f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:26:59.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.144+0000 7f3ef29e2640 1 --2- 172.21.15.37:0/1029742174 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3eec13bac0 0x7f3eec13f5c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3ee4019060 tx=0x7f3ee40191f0 comp rx=0 tx=0).stop 2024-04-26T00:26:59.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.144+0000 7f3ef0f7f640 1 -- 172.21.15.37:0/1029742174 reap_dead start 2024-04-26T00:26:59.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.144+0000 7f3ef0f7f640 1 -- 172.21.15.37:0/1029742174 reap_dead start 2024-04-26T00:26:59.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.144+0000 7f3ef29e2640 1 -- 172.21.15.37:0/1029742174 shutdown_connections 2024-04-26T00:26:59.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.144+0000 7f3ef29e2640 1 -- 172.21.15.37:0/1029742174 >> 172.21.15.37:0/1029742174 conn(0x7f3eec0a1d50 msgr2=0x7f3eec0a39e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:26:59.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.144+0000 7f3ef29e2640 1 -- 172.21.15.37:0/1029742174 shutdown_connections 2024-04-26T00:26:59.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:26:59.144+0000 7f3ef29e2640 1 -- 172.21.15.37:0/1029742174 wait complete. 2024-04-26T00:26:59.147 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:26:59.147 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-04-26T00:26:59.507 INFO:tasks.rados.rados.0.smithi045.stdout:1108: done (1 left) 2024-04-26T00:26:59.507 INFO:tasks.rados.rados.0.smithi045.stdout:1109: done (0 left) 2024-04-26T00:26:59.507 INFO:tasks.rados.rados.0.smithi045.stdout:1110: snap_remove snap 120 2024-04-26T00:27:00.508 INFO:tasks.rados.rados.0.smithi045.stdout:1110: done (0 left) 2024-04-26T00:27:00.508 INFO:tasks.rados.rados.0.smithi045.stdout:1111: read oid 7 snap 129 2024-04-26T00:27:00.508 INFO:tasks.rados.rados.0.smithi045.stdout:1111: expect (ObjNum 114 snap 28 seq_num 114) 2024-04-26T00:27:00.508 INFO:tasks.rados.rados.0.smithi045.stdout:1112: read oid 23 snap 129 2024-04-26T00:27:00.508 INFO:tasks.rados.rados.0.smithi045.stdout:1112: expect (ObjNum 316 snap 93 seq_num 316) 2024-04-26T00:27:00.508 INFO:tasks.rados.rados.0.smithi045.stdout:1113: write (excl) oid 17 current snap is 131 2024-04-26T00:27:00.508 INFO:tasks.rados.rados.0.smithi045.stdout:1113: seq_num 419 ranges {642133=728746,2119014=532478} 2024-04-26T00:27:00.529 INFO:tasks.rados.rados.0.smithi045.stdout:1113: writing smithi04526579-17 from 642133 to 1370879 tid 1 2024-04-26T00:27:00.549 INFO:tasks.rados.rados.0.smithi045.stdout:1113: writing smithi04526579-17 from 2119014 to 2651492 tid 2 2024-04-26T00:27:00.550 INFO:tasks.rados.rados.0.smithi045.stdout:1114: rollback oid 18 current snap is 131 2024-04-26T00:27:00.550 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 18 to 69 2024-04-26T00:27:00.550 INFO:tasks.rados.rados.0.smithi045.stdout:1112: expect (ObjNum 316 snap 93 seq_num 316) 2024-04-26T00:27:00.826 INFO:tasks.rados.rados.0.smithi045.stdout:1111: expect (ObjNum 114 snap 28 seq_num 114) 2024-04-26T00:27:00.955 INFO:tasks.rados.rados.0.smithi045.stdout:1113: finishing write tid 1 to smithi04526579-17 2024-04-26T00:27:00.956 INFO:tasks.rados.rados.0.smithi045.stdout:1111: done (3 left) 2024-04-26T00:27:00.956 INFO:tasks.rados.rados.0.smithi045.stdout:1112: done (2 left) 2024-04-26T00:27:00.956 INFO:tasks.rados.rados.0.smithi045.stdout:1115: write oid 27 current snap is 131 2024-04-26T00:27:00.956 INFO:tasks.rados.rados.0.smithi045.stdout:1113: finishing write tid 2 to smithi04526579-17 2024-04-26T00:27:00.956 INFO:tasks.rados.rados.0.smithi045.stdout:1113: finishing write tid 3 to smithi04526579-17 2024-04-26T00:27:00.956 INFO:tasks.rados.rados.0.smithi045.stdout:1113: finishing write tid 4 to smithi04526579-17 2024-04-26T00:27:00.956 INFO:tasks.rados.rados.0.smithi045.stdout:1113: oid 17 updating version 0 to 523 2024-04-26T00:27:00.956 INFO:tasks.rados.rados.0.smithi045.stdout:1113: oid 17 version 523 is already newer than 521 2024-04-26T00:27:00.956 INFO:tasks.rados.rados.0.smithi045.stdout:1113: oid 17 version 523 is already newer than 522 2024-04-26T00:27:00.956 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 523 (ObjNum 419 snap 131 seq_num 419) dirty exists 2024-04-26T00:27:00.956 INFO:tasks.rados.rados.0.smithi045.stdout:1113: left oid 17 (ObjNum 419 snap 131 seq_num 419) 2024-04-26T00:27:00.956 INFO:tasks.rados.rados.0.smithi045.stdout:1114: finishing rollback tid 0 to smithi04526579-18 2024-04-26T00:27:00.956 INFO:tasks.rados.rados.0.smithi045.stdout:1115: seq_num 420 ranges {658940=403995,1777496=676136,2950951=107989} 2024-04-26T00:27:00.961 INFO:tasks.rados.rados.0.smithi045.stdout:1115: writing smithi04526579-27 from 658940 to 1062935 tid 1 2024-04-26T00:27:00.969 INFO:tasks.rados.rados.0.smithi045.stdout:1115: writing smithi04526579-27 from 1777496 to 2453632 tid 2 2024-04-26T00:27:00.973 INFO:tasks.rados.rados.0.smithi045.stdout:1115: writing smithi04526579-27 from 2950951 to 3058940 tid 3 2024-04-26T00:27:00.973 INFO:tasks.rados.rados.0.smithi045.stdout:1113: done (2 left) 2024-04-26T00:27:00.973 INFO:tasks.rados.rados.0.smithi045.stdout:1114: finishing rollback tid 1 to smithi04526579-18 2024-04-26T00:27:00.973 INFO:tasks.rados.rados.0.smithi045.stdout:1114: finishing rollback tid 2 to smithi04526579-18 2024-04-26T00:27:00.973 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 830 (ObjNum 190 snap 49 seq_num 190) dirty exists 2024-04-26T00:27:00.973 INFO:tasks.rados.rados.0.smithi045.stdout:1116: read oid 33 snap -1 2024-04-26T00:27:00.973 INFO:tasks.rados.rados.0.smithi045.stdout:1116: expect (ObjNum 405 snap 125 seq_num 405) 2024-04-26T00:27:00.973 INFO:tasks.rados.rados.0.smithi045.stdout:1114: done (2 left) 2024-04-26T00:27:00.973 INFO:tasks.rados.rados.0.smithi045.stdout:1117: read oid 38 snap -1 2024-04-26T00:27:00.974 INFO:tasks.rados.rados.0.smithi045.stdout:1117: expect (ObjNum 391 snap 120 seq_num 391) 2024-04-26T00:27:00.974 INFO:tasks.rados.rados.0.smithi045.stdout:1118: copy_from oid 50 from oid 18 current snap is 131 2024-04-26T00:27:00.974 INFO:tasks.rados.rados.0.smithi045.stdout:1119: rollback oid 28 current snap is 131 2024-04-26T00:27:00.974 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 28 to 130 2024-04-26T00:27:00.974 INFO:tasks.rados.rados.0.smithi045.stdout:1120: write oid 26 current snap is 131 2024-04-26T00:27:00.974 INFO:tasks.rados.rados.0.smithi045.stdout:1120: seq_num 421 ranges {675747=479244,1835978=536147} 2024-04-26T00:27:00.980 INFO:tasks.rados.rados.0.smithi045.stdout:1120: writing smithi04526579-26 from 675747 to 1154991 tid 1 2024-04-26T00:27:00.988 INFO:tasks.rados.rados.0.smithi045.stdout:1120: writing smithi04526579-26 from 1835978 to 2372125 tid 2 2024-04-26T00:27:00.988 INFO:tasks.rados.rados.0.smithi045.stdout:1121: write (excl) oid 41 current snap is 131 2024-04-26T00:27:00.988 INFO:tasks.rados.rados.0.smithi045.stdout:1115: finishing write tid 1 to smithi04526579-27 2024-04-26T00:27:00.988 INFO:tasks.rados.rados.0.smithi045.stdout:1119: finishing rollback tid 0 to smithi04526579-28 2024-04-26T00:27:00.988 INFO:tasks.rados.rados.0.smithi045.stdout:1119: finishing rollback tid 1 to smithi04526579-28 2024-04-26T00:27:00.989 INFO:tasks.rados.rados.0.smithi045.stdout:1119: finishing rollback tid 2 to smithi04526579-28 2024-04-26T00:27:00.989 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 679 (ObjNum 376 snap 115 seq_num 376) dirty exists 2024-04-26T00:27:00.989 INFO:tasks.rados.rados.0.smithi045.stdout:1115: finishing write tid 2 to smithi04526579-27 2024-04-26T00:27:00.989 INFO:tasks.rados.rados.0.smithi045.stdout:1115: finishing write tid 3 to smithi04526579-27 2024-04-26T00:27:00.989 INFO:tasks.rados.rados.0.smithi045.stdout:1115: finishing write tid 4 to smithi04526579-27 2024-04-26T00:27:00.989 INFO:tasks.rados.rados.0.smithi045.stdout:1121: seq_num 422 ranges {692554=554493,1778107=679805,2889444=203110} 2024-04-26T00:27:00.998 INFO:tasks.rados.rados.0.smithi045.stdout:1121: writing smithi04526579-41 from 692554 to 1247047 tid 1 2024-04-26T00:27:01.008 INFO:tasks.rados.rados.0.smithi045.stdout:1121: writing smithi04526579-41 from 1778107 to 2457912 tid 2 2024-04-26T00:27:01.012 INFO:tasks.rados.rados.0.smithi045.stdout:1121: writing smithi04526579-41 from 2889444 to 3092554 tid 3 2024-04-26T00:27:01.013 INFO:tasks.rados.rados.0.smithi045.stdout:1119: done (6 left) 2024-04-26T00:27:01.013 INFO:tasks.rados.rados.0.smithi045.stdout:1122: delete oid 24 current snap is 131 2024-04-26T00:27:01.013 INFO:tasks.rados.rados.0.smithi045.stdout:1117: expect (ObjNum 391 snap 120 seq_num 391) 2024-04-26T00:27:01.099 INFO:tasks.rados.rados.0.smithi045.stdout:1115: finishing write tid 5 to smithi04526579-27 2024-04-26T00:27:01.099 INFO:tasks.rados.rados.0.smithi045.stdout:1115: oid 27 updating version 0 to 577 2024-04-26T00:27:01.099 INFO:tasks.rados.rados.0.smithi045.stdout:1115: oid 27 version 577 is already newer than 574 2024-04-26T00:27:01.099 INFO:tasks.rados.rados.0.smithi045.stdout:1115: oid 27 version 577 is already newer than 575 2024-04-26T00:27:01.099 INFO:tasks.rados.rados.0.smithi045.stdout:1115: oid 27 version 577 is already newer than 576 2024-04-26T00:27:01.099 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 577 (ObjNum 420 snap 131 seq_num 420) dirty exists 2024-04-26T00:27:01.100 INFO:tasks.rados.rados.0.smithi045.stdout:1115: left oid 27 (ObjNum 420 snap 131 seq_num 420) 2024-04-26T00:27:01.100 INFO:tasks.rados.rados.0.smithi045.stdout:1120: finishing write tid 1 to smithi04526579-26 2024-04-26T00:27:01.100 INFO:tasks.rados.rados.0.smithi045.stdout:1116: expect (ObjNum 405 snap 125 seq_num 405) 2024-04-26T00:27:01.286 INFO:tasks.rados.rados.0.smithi045.stdout:1120: finishing write tid 2 to smithi04526579-26 2024-04-26T00:27:01.287 INFO:tasks.rados.rados.0.smithi045.stdout:1120: finishing write tid 3 to smithi04526579-26 2024-04-26T00:27:01.287 INFO:tasks.rados.rados.0.smithi045.stdout:1120: finishing write tid 4 to smithi04526579-26 2024-04-26T00:27:01.287 INFO:tasks.rados.rados.0.smithi045.stdout:1120: oid 26 updating version 0 to 681 2024-04-26T00:27:01.287 INFO:tasks.rados.rados.0.smithi045.stdout:1120: oid 26 updating version 681 to 683 2024-04-26T00:27:01.287 INFO:tasks.rados.rados.0.smithi045.stdout:1120: oid 26 version 683 is already newer than 682 2024-04-26T00:27:01.287 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 683 (ObjNum 421 snap 131 seq_num 421) dirty exists 2024-04-26T00:27:01.287 INFO:tasks.rados.rados.0.smithi045.stdout:1120: left oid 26 (ObjNum 421 snap 131 seq_num 421) 2024-04-26T00:27:01.287 INFO:tasks.rados.rados.0.smithi045.stdout:1118: finishing copy_from racing read to smithi04526579-50 2024-04-26T00:27:01.287 INFO:tasks.rados.rados.0.smithi045.stdout:1121: finishing write tid 1 to smithi04526579-41 2024-04-26T00:27:01.287 INFO:tasks.rados.rados.0.smithi045.stdout:1118: finishing copy_from to smithi04526579-50 2024-04-26T00:27:01.287 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 830 (ObjNum 190 snap 49 seq_num 190) dirty exists 2024-04-26T00:27:01.287 INFO:tasks.rados.rados.0.smithi045.stdout:1121: finishing write tid 2 to smithi04526579-41 2024-04-26T00:27:01.287 INFO:tasks.rados.rados.0.smithi045.stdout:1121: finishing write tid 3 to smithi04526579-41 2024-04-26T00:27:01.287 INFO:tasks.rados.rados.0.smithi045.stdout:1121: finishing write tid 4 to smithi04526579-41 2024-04-26T00:27:01.288 INFO:tasks.rados.rados.0.smithi045.stdout:1121: finishing write tid 5 to smithi04526579-41 2024-04-26T00:27:01.288 INFO:tasks.rados.rados.0.smithi045.stdout:1121: oid 41 updating version 0 to 685 2024-04-26T00:27:01.288 INFO:tasks.rados.rados.0.smithi045.stdout:1121: oid 41 updating version 685 to 686 2024-04-26T00:27:01.288 INFO:tasks.rados.rados.0.smithi045.stdout:1121: oid 41 updating version 686 to 687 2024-04-26T00:27:01.288 INFO:tasks.rados.rados.0.smithi045.stdout:1121: oid 41 updating version 687 to 688 2024-04-26T00:27:01.288 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 688 (ObjNum 422 snap 131 seq_num 422) dirty exists 2024-04-26T00:27:01.288 INFO:tasks.rados.rados.0.smithi045.stdout:1121: left oid 41 (ObjNum 422 snap 131 seq_num 422) 2024-04-26T00:27:01.288 INFO:tasks.rados.rados.0.smithi045.stdout:1115: done (6 left) 2024-04-26T00:27:01.288 INFO:tasks.rados.rados.0.smithi045.stdout:1116: done (5 left) 2024-04-26T00:27:01.288 INFO:tasks.rados.rados.0.smithi045.stdout:1117: done (4 left) 2024-04-26T00:27:01.288 INFO:tasks.rados.rados.0.smithi045.stdout:1118: done (3 left) 2024-04-26T00:27:01.288 INFO:tasks.rados.rados.0.smithi045.stdout:1120: done (2 left) 2024-04-26T00:27:01.288 INFO:tasks.rados.rados.0.smithi045.stdout:1121: done (1 left) 2024-04-26T00:27:01.288 INFO:tasks.rados.rados.0.smithi045.stdout:1122: done (0 left) 2024-04-26T00:27:01.288 INFO:tasks.rados.rados.0.smithi045.stdout:1123: delete oid 18 current snap is 131 2024-04-26T00:27:01.291 INFO:tasks.rados.rados.0.smithi045.stdout:1123: done (0 left) 2024-04-26T00:27:01.291 INFO:tasks.rados.rados.0.smithi045.stdout:1124: copy_from oid 39 from oid 43 current snap is 131 2024-04-26T00:27:01.292 INFO:tasks.rados.rados.0.smithi045.stdout:1125: snap_remove snap 130 2024-04-26T00:27:01.516 INFO:tasks.rados.rados.0.smithi045.stdout:1125: done (1 left) 2024-04-26T00:27:01.516 INFO:tasks.rados.rados.0.smithi045.stdout:1126: snap_remove snap 122 2024-04-26T00:27:01.894 INFO:tasks.rados.rados.0.smithi045.stdout:1126: done (1 left) 2024-04-26T00:27:01.894 INFO:tasks.rados.rados.0.smithi045.stdout:1127: snap_remove snap 117 2024-04-26T00:27:02.898 INFO:tasks.rados.rados.0.smithi045.stdout:1127: done (1 left) 2024-04-26T00:27:02.898 INFO:tasks.rados.rados.0.smithi045.stdout:1128: read oid 26 snap -1 2024-04-26T00:27:02.898 INFO:tasks.rados.rados.0.smithi045.stdout:1128: expect (ObjNum 421 snap 131 seq_num 421) 2024-04-26T00:27:02.898 INFO:tasks.rados.rados.0.smithi045.stdout:1129: delete oid 9 current snap is 131 2024-04-26T00:27:02.898 INFO:tasks.rados.rados.0.smithi045.stdout:1124: finishing copy_from racing read to smithi04526579-39 2024-04-26T00:27:02.898 INFO:tasks.rados.rados.0.smithi045.stdout:1124: finishing copy_from to smithi04526579-39 2024-04-26T00:27:02.898 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 253 (ObjNum 329 snap 96 seq_num 329) dirty exists 2024-04-26T00:27:02.912 INFO:tasks.rados.rados.0.smithi045.stdout:1128: expect (ObjNum 421 snap 131 seq_num 421) 2024-04-26T00:27:03.170 INFO:tasks.rados.rados.0.smithi045.stdout:1124: done (2 left) 2024-04-26T00:27:03.170 INFO:tasks.rados.rados.0.smithi045.stdout:1128: done (1 left) 2024-04-26T00:27:03.170 INFO:tasks.rados.rados.0.smithi045.stdout:1129: done (0 left) 2024-04-26T00:27:03.170 INFO:tasks.rados.rados.0.smithi045.stdout:1130: delete oid 40 current snap is 131 2024-04-26T00:27:03.176 INFO:tasks.rados.rados.0.smithi045.stdout:1130: done (0 left) 2024-04-26T00:27:03.176 INFO:tasks.rados.rados.0.smithi045.stdout:1131: snap_create 2024-04-26T00:27:03.900 INFO:tasks.rados.rados.0.smithi045.stdout:1131: done (0 left) 2024-04-26T00:27:03.901 INFO:tasks.rados.rados.0.smithi045.stdout:1132: delete oid 42 current snap is 132 2024-04-26T00:27:03.909 INFO:tasks.rados.rados.0.smithi045.stdout:1132: done (0 left) 2024-04-26T00:27:03.909 INFO:tasks.rados.rados.0.smithi045.stdout:1133: snap_create 2024-04-26T00:27:04.915 INFO:tasks.rados.rados.0.smithi045.stdout:1133: done (0 left) 2024-04-26T00:27:04.915 INFO:tasks.rados.rados.0.smithi045.stdout:1134: read oid 29 snap -1 2024-04-26T00:27:04.915 INFO:tasks.rados.rados.0.smithi045.stdout:1134: expect (ObjNum 131 snap 30 seq_num 131) 2024-04-26T00:27:04.915 INFO:tasks.rados.rados.0.smithi045.stdout:1135: rollback oid 50 current snap is 133 2024-04-26T00:27:04.915 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 50 to 127 2024-04-26T00:27:04.915 INFO:tasks.rados.rados.0.smithi045.stdout:1136: copy_from oid 8 from oid 13 current snap is 133 2024-04-26T00:27:04.915 INFO:tasks.rados.rados.0.smithi045.stdout:1137: snap_remove snap 132 2024-04-26T00:27:05.924 INFO:tasks.rados.rados.0.smithi045.stdout:1137: done (3 left) 2024-04-26T00:27:05.924 INFO:tasks.rados.rados.0.smithi045.stdout:1138: rollback oid 19 current snap is 133 2024-04-26T00:27:05.924 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 19 to 92 2024-04-26T00:27:05.924 INFO:tasks.rados.rados.0.smithi045.stdout:1135: finishing rollback tid 0 to smithi04526579-50 2024-04-26T00:27:05.924 INFO:tasks.rados.rados.0.smithi045.stdout:1135: finishing rollback tid 1 to smithi04526579-50 2024-04-26T00:27:05.924 INFO:tasks.rados.rados.0.smithi045.stdout:1135: finishing rollback tid 2 to smithi04526579-50 2024-04-26T00:27:05.924 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 833 (ObjNum 136 snap 30 seq_num 136) dirty exists 2024-04-26T00:27:05.924 INFO:tasks.rados.rados.0.smithi045.stdout:1134: expect (ObjNum 131 snap 30 seq_num 131) 2024-04-26T00:27:06.079 INFO:tasks.rados.rados.0.smithi045.stdout:1136: finishing copy_from racing read to smithi04526579-8 2024-04-26T00:27:06.080 INFO:tasks.rados.rados.0.smithi045.stdout:1136: finishing copy_from to smithi04526579-8 2024-04-26T00:27:06.080 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 346 (ObjNum 337 snap 100 seq_num 337) dirty exists 2024-04-26T00:27:06.080 INFO:tasks.rados.rados.0.smithi045.stdout:1138: finishing rollback tid 1 to smithi04526579-19 2024-04-26T00:27:06.080 INFO:tasks.rados.rados.0.smithi045.stdout:1138: finishing rollback tid 2 to smithi04526579-19 2024-04-26T00:27:06.080 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 599 (ObjNum 224 snap 59 seq_num 224) dirty exists 2024-04-26T00:27:06.080 INFO:tasks.rados.rados.0.smithi045.stdout:1134: done (3 left) 2024-04-26T00:27:06.080 INFO:tasks.rados.rados.0.smithi045.stdout:1135: done (2 left) 2024-04-26T00:27:06.080 INFO:tasks.rados.rados.0.smithi045.stdout:1136: done (1 left) 2024-04-26T00:27:06.080 INFO:tasks.rados.rados.0.smithi045.stdout:1138: done (0 left) 2024-04-26T00:27:06.080 INFO:tasks.rados.rados.0.smithi045.stdout:1139: delete oid 29 current snap is 133 2024-04-26T00:27:06.086 INFO:tasks.rados.rados.0.smithi045.stdout:1139: done (0 left) 2024-04-26T00:27:06.086 INFO:tasks.rados.rados.0.smithi045.stdout:1140: snap_remove snap 92 2024-04-26T00:27:06.897 INFO:tasks.rados.rados.0.smithi045.stdout:1140: done (0 left) 2024-04-26T00:27:06.897 INFO:tasks.rados.rados.0.smithi045.stdout:1141: delete oid 36 current snap is 133 2024-04-26T00:27:06.920 INFO:tasks.rados.rados.0.smithi045.stdout:1141: done (0 left) 2024-04-26T00:27:06.920 INFO:tasks.rados.rados.0.smithi045.stdout:1142: copy_from oid 7 from oid 48 current snap is 133 2024-04-26T00:27:06.920 INFO:tasks.rados.rados.0.smithi045.stdout:1143: read oid 5 snap -1 2024-04-26T00:27:06.920 INFO:tasks.rados.rados.0.smithi045.stdout:1143: expect (ObjNum 400 snap 122 seq_num 400) 2024-04-26T00:27:06.920 INFO:tasks.rados.rados.0.smithi045.stdout:1144: snap_create 2024-04-26T00:27:06.948 INFO:tasks.rados.rados.0.smithi045.stdout:1142: finishing copy_from racing read to smithi04526579-7 2024-04-26T00:27:06.953 INFO:tasks.rados.rados.0.smithi045.stdout:1143: expect (ObjNum 400 snap 122 seq_num 400) 2024-04-26T00:27:07.065 INFO:tasks.rados.rados.0.smithi045.stdout:1142: finishing copy_from to smithi04526579-7 2024-04-26T00:27:07.066 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 309 (ObjNum 359 snap 107 seq_num 359) dirty exists 2024-04-26T00:27:07.907 INFO:tasks.rados.rados.0.smithi045.stdout:1142: done (2 left) 2024-04-26T00:27:07.907 INFO:tasks.rados.rados.0.smithi045.stdout:1143: done (1 left) 2024-04-26T00:27:07.907 INFO:tasks.rados.rados.0.smithi045.stdout:1144: done (0 left) 2024-04-26T00:27:07.907 INFO:tasks.rados.rados.0.smithi045.stdout:1145: copy_from oid 23 from oid 44 current snap is 134 2024-04-26T00:27:07.908 INFO:tasks.rados.rados.0.smithi045.stdout:1146: write (excl) oid 16 current snap is 134 2024-04-26T00:27:07.908 INFO:tasks.rados.rados.0.smithi045.stdout:1146: seq_num 430 ranges {427010=472838,1329610=694481,2708828=518182} 2024-04-26T00:27:07.922 INFO:tasks.rados.rados.0.smithi045.stdout:1146: writing smithi04526579-16 from 427010 to 899848 tid 1 2024-04-26T00:27:07.940 INFO:tasks.rados.rados.0.smithi045.stdout:1146: writing smithi04526579-16 from 1329610 to 2024091 tid 2 2024-04-26T00:27:07.959 INFO:tasks.rados.rados.0.smithi045.stdout:1146: writing smithi04526579-16 from 2708828 to 3227010 tid 3 2024-04-26T00:27:07.959 INFO:tasks.rados.rados.0.smithi045.stdout:1147: snap_create 2024-04-26T00:27:07.959 INFO:tasks.rados.rados.0.smithi045.stdout:1145: finishing copy_from to smithi04526579-23 2024-04-26T00:27:07.959 INFO:tasks.rados.rados.0.smithi045.stdout:1145: got expected ENOENT (src dne) 2024-04-26T00:27:07.960 INFO:tasks.rados.rados.0.smithi045.stdout:1145: finishing copy_from racing read to smithi04526579-23 2024-04-26T00:27:07.960 INFO:tasks.rados.rados.0.smithi045.stdout:1146: finishing write tid 1 to smithi04526579-16 2024-04-26T00:27:07.960 INFO:tasks.rados.rados.0.smithi045.stdout:1146: finishing write tid 2 to smithi04526579-16 2024-04-26T00:27:07.967 INFO:tasks.rados.rados.0.smithi045.stdout:1146: finishing write tid 3 to smithi04526579-16 2024-04-26T00:27:07.967 INFO:tasks.rados.rados.0.smithi045.stdout:1146: finishing write tid 4 to smithi04526579-16 2024-04-26T00:27:07.968 INFO:tasks.rados.rados.0.smithi045.stdout:1146: finishing write tid 5 to smithi04526579-16 2024-04-26T00:27:07.968 INFO:tasks.rados.rados.0.smithi045.stdout:1146: oid 16 updating version 0 to 604 2024-04-26T00:27:07.968 INFO:tasks.rados.rados.0.smithi045.stdout:1146: oid 16 version 604 is already newer than 603 2024-04-26T00:27:07.968 INFO:tasks.rados.rados.0.smithi045.stdout:1146: oid 16 version 604 is already newer than 602 2024-04-26T00:27:07.968 INFO:tasks.rados.rados.0.smithi045.stdout:1146: oid 16 updating version 604 to 605 2024-04-26T00:27:07.968 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 605 (ObjNum 430 snap 134 seq_num 430) dirty exists 2024-04-26T00:27:07.968 INFO:tasks.rados.rados.0.smithi045.stdout:1146: left oid 16 (ObjNum 430 snap 134 seq_num 430) 2024-04-26T00:27:08.940 INFO:tasks.rados.rados.0.smithi045.stdout:1145: done (2 left) 2024-04-26T00:27:08.940 INFO:tasks.rados.rados.0.smithi045.stdout:1146: done (1 left) 2024-04-26T00:27:08.940 INFO:tasks.rados.rados.0.smithi045.stdout:1147: done (0 left) 2024-04-26T00:27:08.940 INFO:tasks.rados.rados.0.smithi045.stdout:1148: snap_create 2024-04-26T00:27:09.949 INFO:tasks.rados.rados.0.smithi045.stdout:1148: done (0 left) 2024-04-26T00:27:09.949 INFO:tasks.rados.rados.0.smithi045.stdout:1149: read oid 3 snap -1 2024-04-26T00:27:09.949 INFO:tasks.rados.rados.0.smithi045.stdout:1149: expect (ObjNum 389 snap 117 seq_num 389) 2024-04-26T00:27:09.949 INFO:tasks.rados.rados.0.smithi045.stdout:1150: write (excl) oid 35 current snap is 136 2024-04-26T00:27:09.949 INFO:tasks.rados.rados.0.smithi045.stdout:1150: seq_num 431 ranges {443817=548087,1788092=438139,3019898=223919} 2024-04-26T00:27:09.965 INFO:tasks.rados.rados.0.smithi045.stdout:1150: writing smithi04526579-35 from 443817 to 991904 tid 1 2024-04-26T00:27:09.985 INFO:tasks.rados.rados.0.smithi045.stdout:1150: writing smithi04526579-35 from 1788092 to 2226231 tid 2 2024-04-26T00:27:10.001 INFO:tasks.rados.rados.0.smithi045.stdout:1150: writing smithi04526579-35 from 3019898 to 3243817 tid 3 2024-04-26T00:27:10.001 INFO:tasks.rados.rados.0.smithi045.stdout:1151: write oid 29 current snap is 136 2024-04-26T00:27:10.002 INFO:tasks.rados.rados.0.smithi045.stdout:1151: seq_num 432 ranges {460624=623336,1846574=698150,3163674=96950} 2024-04-26T00:27:10.015 INFO:tasks.rados.rados.0.smithi045.stdout:1151: writing smithi04526579-29 from 460624 to 1083960 tid 1 2024-04-26T00:27:10.030 INFO:tasks.rados.rados.0.smithi045.stdout:1151: writing smithi04526579-29 from 1846574 to 2544724 tid 2 2024-04-26T00:27:10.037 INFO:tasks.rados.rados.0.smithi045.stdout:1151: writing smithi04526579-29 from 3163674 to 3260624 tid 3 2024-04-26T00:27:10.037 INFO:tasks.rados.rados.0.smithi045.stdout:1152: read oid 48 snap 124 2024-04-26T00:27:10.037 INFO:tasks.rados.rados.0.smithi045.stdout:1152: expect (ObjNum 359 snap 107 seq_num 359) 2024-04-26T00:27:10.038 INFO:tasks.rados.rados.0.smithi045.stdout:1153: write (excl) oid 14 current snap is 136 2024-04-26T00:27:10.038 INFO:tasks.rados.rados.0.smithi045.stdout:1149: expect (ObjNum 389 snap 117 seq_num 389) 2024-04-26T00:27:10.219 INFO:tasks.rados.rados.0.smithi045.stdout:1150: finishing write tid 1 to smithi04526579-35 2024-04-26T00:27:10.220 INFO:tasks.rados.rados.0.smithi045.stdout:1150: finishing write tid 2 to smithi04526579-35 2024-04-26T00:27:10.220 INFO:tasks.rados.rados.0.smithi045.stdout:1150: finishing write tid 3 to smithi04526579-35 2024-04-26T00:27:10.220 INFO:tasks.rados.rados.0.smithi045.stdout:1150: finishing write tid 4 to smithi04526579-35 2024-04-26T00:27:10.220 INFO:tasks.rados.rados.0.smithi045.stdout:1150: finishing write tid 5 to smithi04526579-35 2024-04-26T00:27:10.220 INFO:tasks.rados.rados.0.smithi045.stdout:1150: oid 35 updating version 0 to 612 2024-04-26T00:27:10.220 INFO:tasks.rados.rados.0.smithi045.stdout:1150: oid 35 updating version 612 to 614 2024-04-26T00:27:10.220 INFO:tasks.rados.rados.0.smithi045.stdout:1150: oid 35 version 614 is already newer than 611 2024-04-26T00:27:10.220 INFO:tasks.rados.rados.0.smithi045.stdout:1150: oid 35 version 614 is already newer than 613 2024-04-26T00:27:10.220 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 614 (ObjNum 431 snap 136 seq_num 431) dirty exists 2024-04-26T00:27:10.220 INFO:tasks.rados.rados.0.smithi045.stdout:1150: left oid 35 (ObjNum 431 snap 136 seq_num 431) 2024-04-26T00:27:10.220 INFO:tasks.rados.rados.0.smithi045.stdout:1151: finishing write tid 1 to smithi04526579-29 2024-04-26T00:27:10.220 INFO:tasks.rados.rados.0.smithi045.stdout:1151: finishing write tid 2 to smithi04526579-29 2024-04-26T00:27:10.220 INFO:tasks.rados.rados.0.smithi045.stdout:1151: finishing write tid 3 to smithi04526579-29 2024-04-26T00:27:10.221 INFO:tasks.rados.rados.0.smithi045.stdout:1151: finishing write tid 4 to smithi04526579-29 2024-04-26T00:27:10.221 INFO:tasks.rados.rados.0.smithi045.stdout:1152: expect (ObjNum 359 snap 107 seq_num 359) 2024-04-26T00:27:10.302 INFO:tasks.rados.rados.0.smithi045.stdout:1151: finishing write tid 5 to smithi04526579-29 2024-04-26T00:27:10.302 INFO:tasks.rados.rados.0.smithi045.stdout:1151: oid 29 updating version 0 to 408 2024-04-26T00:27:10.302 INFO:tasks.rados.rados.0.smithi045.stdout:1151: oid 29 updating version 408 to 409 2024-04-26T00:27:10.302 INFO:tasks.rados.rados.0.smithi045.stdout:1151: oid 29 updating version 409 to 410 2024-04-26T00:27:10.302 INFO:tasks.rados.rados.0.smithi045.stdout:1151: oid 29 version 410 is already newer than 407 2024-04-26T00:27:10.302 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 410 (ObjNum 432 snap 136 seq_num 432) dirty exists 2024-04-26T00:27:10.302 INFO:tasks.rados.rados.0.smithi045.stdout:1151: left oid 29 (ObjNum 432 snap 136 seq_num 432) 2024-04-26T00:27:10.303 INFO:tasks.rados.rados.0.smithi045.stdout:1153: seq_num 433 ranges {477431=698585,1905056=441808,3074744=202687} 2024-04-26T00:27:10.317 INFO:tasks.rados.rados.0.smithi045.stdout:1153: writing smithi04526579-14 from 477431 to 1176016 tid 1 2024-04-26T00:27:10.333 INFO:tasks.rados.rados.0.smithi045.stdout:1153: writing smithi04526579-14 from 1905056 to 2346864 tid 2 2024-04-26T00:27:10.345 INFO:tasks.rados.rados.0.smithi045.stdout:1153: writing smithi04526579-14 from 3074744 to 3277431 tid 3 2024-04-26T00:27:10.345 INFO:tasks.rados.rados.0.smithi045.stdout:1149: done (4 left) 2024-04-26T00:27:10.346 INFO:tasks.rados.rados.0.smithi045.stdout:1150: done (3 left) 2024-04-26T00:27:10.346 INFO:tasks.rados.rados.0.smithi045.stdout:1151: done (2 left) 2024-04-26T00:27:10.346 INFO:tasks.rados.rados.0.smithi045.stdout:1152: done (1 left) 2024-04-26T00:27:10.346 INFO:tasks.rados.rados.0.smithi045.stdout:1154: write oid 42 current snap is 136 2024-04-26T00:27:10.346 INFO:tasks.rados.rados.0.smithi045.stdout:1153: finishing write tid 1 to smithi04526579-14 2024-04-26T00:27:10.346 INFO:tasks.rados.rados.0.smithi045.stdout:1153: finishing write tid 2 to smithi04526579-14 2024-04-26T00:27:10.346 INFO:tasks.rados.rados.0.smithi045.stdout:1154: seq_num 434 ranges {494238=490187,1563538=701819,2818520=475718} 2024-04-26T00:27:10.358 INFO:tasks.rados.rados.0.smithi045.stdout:1154: writing smithi04526579-42 from 494238 to 984425 tid 1 2024-04-26T00:27:10.374 INFO:tasks.rados.rados.0.smithi045.stdout:1154: writing smithi04526579-42 from 1563538 to 2265357 tid 2 2024-04-26T00:27:10.386 INFO:tasks.rados.rados.0.smithi045.stdout:1154: writing smithi04526579-42 from 2818520 to 3294238 tid 3 2024-04-26T00:27:10.386 INFO:tasks.rados.rados.0.smithi045.stdout:1155: read oid 29 snap -1 2024-04-26T00:27:10.386 INFO:tasks.rados.rados.0.smithi045.stdout:1155: expect (ObjNum 432 snap 136 seq_num 432) 2024-04-26T00:27:10.386 INFO:tasks.rados.rados.0.smithi045.stdout:1156: snap_remove snap 101 2024-04-26T00:27:10.952 INFO:tasks.rados.rados.0.smithi045.stdout:1156: done (3 left) 2024-04-26T00:27:10.952 INFO:tasks.rados.rados.0.smithi045.stdout:1157: read oid 47 snap -1 2024-04-26T00:27:10.952 INFO:tasks.rados.rados.0.smithi045.stdout:1157: expect (ObjNum 416 snap 130 seq_num 416) 2024-04-26T00:27:10.952 INFO:tasks.rados.rados.0.smithi045.stdout:1158: snap_remove snap 131 2024-04-26T00:27:11.900 INFO:tasks.rados.rados.0.smithi045.stdout:1158: done (4 left) 2024-04-26T00:27:11.900 INFO:tasks.rados.rados.0.smithi045.stdout:1159: read oid 46 snap -1 2024-04-26T00:27:11.900 INFO:tasks.rados.rados.0.smithi045.stdout:1159: expect (ObjNum 329 snap 96 seq_num 329) 2024-04-26T00:27:11.900 INFO:tasks.rados.rados.0.smithi045.stdout:1160: read oid 5 snap -1 2024-04-26T00:27:11.901 INFO:tasks.rados.rados.0.smithi045.stdout:1160: expect (ObjNum 400 snap 122 seq_num 400) 2024-04-26T00:27:11.901 INFO:tasks.rados.rados.0.smithi045.stdout:1161: copy_from oid 11 from oid 19 current snap is 136 2024-04-26T00:27:11.901 INFO:tasks.rados.rados.0.smithi045.stdout:1153: finishing write tid 3 to smithi04526579-14 2024-04-26T00:27:11.901 INFO:tasks.rados.rados.0.smithi045.stdout:1153: finishing write tid 4 to smithi04526579-14 2024-04-26T00:27:11.901 INFO:tasks.rados.rados.0.smithi045.stdout:1153: finishing write tid 5 to smithi04526579-14 2024-04-26T00:27:11.901 INFO:tasks.rados.rados.0.smithi045.stdout:1153: oid 14 updating version 0 to 607 2024-04-26T00:27:11.901 INFO:tasks.rados.rados.0.smithi045.stdout:1153: oid 14 updating version 607 to 608 2024-04-26T00:27:11.901 INFO:tasks.rados.rados.0.smithi045.stdout:1153: oid 14 version 608 is already newer than 606 2024-04-26T00:27:11.901 INFO:tasks.rados.rados.0.smithi045.stdout:1153: oid 14 updating version 608 to 609 2024-04-26T00:27:11.901 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 609 (ObjNum 433 snap 136 seq_num 433) dirty exists 2024-04-26T00:27:11.901 INFO:tasks.rados.rados.0.smithi045.stdout:1153: left oid 14 (ObjNum 433 snap 136 seq_num 433) 2024-04-26T00:27:11.901 INFO:tasks.rados.rados.0.smithi045.stdout:1153: done (6 left) 2024-04-26T00:27:11.901 INFO:tasks.rados.rados.0.smithi045.stdout:1162: snap_create 2024-04-26T00:27:11.901 INFO:tasks.rados.rados.0.smithi045.stdout:1154: finishing write tid 1 to smithi04526579-42 2024-04-26T00:27:11.902 INFO:tasks.rados.rados.0.smithi045.stdout:1154: finishing write tid 2 to smithi04526579-42 2024-04-26T00:27:11.902 INFO:tasks.rados.rados.0.smithi045.stdout:1154: finishing write tid 3 to smithi04526579-42 2024-04-26T00:27:11.902 INFO:tasks.rados.rados.0.smithi045.stdout:1154: finishing write tid 4 to smithi04526579-42 2024-04-26T00:27:11.902 INFO:tasks.rados.rados.0.smithi045.stdout:1154: finishing write tid 5 to smithi04526579-42 2024-04-26T00:27:11.902 INFO:tasks.rados.rados.0.smithi045.stdout:1154: oid 42 updating version 0 to 615 2024-04-26T00:27:11.902 INFO:tasks.rados.rados.0.smithi045.stdout:1154: oid 42 updating version 615 to 618 2024-04-26T00:27:11.902 INFO:tasks.rados.rados.0.smithi045.stdout:1154: oid 42 version 618 is already newer than 617 2024-04-26T00:27:11.902 INFO:tasks.rados.rados.0.smithi045.stdout:1154: oid 42 version 618 is already newer than 616 2024-04-26T00:27:11.902 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 618 (ObjNum 434 snap 136 seq_num 434) dirty exists 2024-04-26T00:27:11.902 INFO:tasks.rados.rados.0.smithi045.stdout:1154: left oid 42 (ObjNum 434 snap 136 seq_num 434) 2024-04-26T00:27:11.902 INFO:tasks.rados.rados.0.smithi045.stdout:1155: expect (ObjNum 432 snap 136 seq_num 432) 2024-04-26T00:27:12.041 INFO:tasks.rados.rados.0.smithi045.stdout:1157: expect (ObjNum 416 snap 130 seq_num 416) 2024-04-26T00:27:12.213 INFO:tasks.rados.rados.0.smithi045.stdout:1159: expect (ObjNum 329 snap 96 seq_num 329) 2024-04-26T00:27:12.330 INFO:tasks.rados.rados.0.smithi045.stdout:1160: expect (ObjNum 400 snap 122 seq_num 400) 2024-04-26T00:27:12.419 INFO:tasks.rados.rados.0.smithi045.stdout:1161: finishing copy_from racing read to smithi04526579-11 2024-04-26T00:27:12.420 INFO:tasks.rados.rados.0.smithi045.stdout:1161: finishing copy_from to smithi04526579-11 2024-04-26T00:27:12.420 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 599 (ObjNum 224 snap 59 seq_num 224) dirty exists 2024-04-26T00:27:12.901 INFO:tasks.rados.rados.0.smithi045.stdout:1154: done (6 left) 2024-04-26T00:27:12.901 INFO:tasks.rados.rados.0.smithi045.stdout:1155: done (5 left) 2024-04-26T00:27:12.902 INFO:tasks.rados.rados.0.smithi045.stdout:1157: done (4 left) 2024-04-26T00:27:12.902 INFO:tasks.rados.rados.0.smithi045.stdout:1159: done (3 left) 2024-04-26T00:27:12.902 INFO:tasks.rados.rados.0.smithi045.stdout:1160: done (2 left) 2024-04-26T00:27:12.902 INFO:tasks.rados.rados.0.smithi045.stdout:1161: done (1 left) 2024-04-26T00:27:12.902 INFO:tasks.rados.rados.0.smithi045.stdout:1162: done (0 left) 2024-04-26T00:27:12.902 INFO:tasks.rados.rados.0.smithi045.stdout:1163: snap_remove snap 78 2024-04-26T00:27:13.904 INFO:tasks.rados.rados.0.smithi045.stdout:1163: done (0 left) 2024-04-26T00:27:13.904 INFO:tasks.rados.rados.0.smithi045.stdout:1164: delete oid 41 current snap is 137 2024-04-26T00:27:13.921 INFO:tasks.rados.rados.0.smithi045.stdout:1164: done (0 left) 2024-04-26T00:27:13.921 INFO:tasks.rados.rados.0.smithi045.stdout:1165: snap_create 2024-04-26T00:27:14.911 INFO:tasks.rados.rados.0.smithi045.stdout:1165: done (0 left) 2024-04-26T00:27:14.911 INFO:tasks.rados.rados.0.smithi045.stdout:1166: rollback oid 34 current snap is 138 2024-04-26T00:27:14.911 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 34 to 134 2024-04-26T00:27:14.911 INFO:tasks.rados.rados.0.smithi045.stdout:1167: write oid 28 current snap is 138 2024-04-26T00:27:14.911 INFO:tasks.rados.rados.0.smithi045.stdout:1167: seq_num 436 ranges {527852=640685,1680502=589135,2757013=570839} 2024-04-26T00:27:14.931 INFO:tasks.rados.rados.0.smithi045.stdout:1167: writing smithi04526579-28 from 527852 to 1168537 tid 1 2024-04-26T00:27:14.950 INFO:tasks.rados.rados.0.smithi045.stdout:1167: writing smithi04526579-28 from 1680502 to 2269637 tid 2 2024-04-26T00:27:14.966 INFO:tasks.rados.rados.0.smithi045.stdout:1167: writing smithi04526579-28 from 2757013 to 3327852 tid 3 2024-04-26T00:27:14.967 INFO:tasks.rados.rados.0.smithi045.stdout:1168: write (excl) oid 35 current snap is 138 2024-04-26T00:27:14.967 INFO:tasks.rados.rados.0.smithi045.stdout:1166: finishing rollback tid 0 to smithi04526579-34 2024-04-26T00:27:14.967 INFO:tasks.rados.rados.0.smithi045.stdout:1166: finishing rollback tid 1 to smithi04526579-34 2024-04-26T00:27:14.967 INFO:tasks.rados.rados.0.smithi045.stdout:1166: finishing rollback tid 2 to smithi04526579-34 2024-04-26T00:27:14.967 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 687 (ObjNum 415 snap 130 seq_num 415) dirty exists 2024-04-26T00:27:14.967 INFO:tasks.rados.rados.0.smithi045.stdout:1167: finishing write tid 1 to smithi04526579-28 2024-04-26T00:27:14.967 INFO:tasks.rados.rados.0.smithi045.stdout:1168: seq_num 437 ranges {544659=715934,1738984=449146,2784436=560223} 2024-04-26T00:27:14.983 INFO:tasks.rados.rados.0.smithi045.stdout:1168: writing smithi04526579-35 from 544659 to 1260593 tid 1 2024-04-26T00:27:14.995 INFO:tasks.rados.rados.0.smithi045.stdout:1168: writing smithi04526579-35 from 1738984 to 2188130 tid 2 2024-04-26T00:27:15.009 INFO:tasks.rados.rados.0.smithi045.stdout:1168: writing smithi04526579-35 from 2784436 to 3344659 tid 3 2024-04-26T00:27:15.009 INFO:tasks.rados.rados.0.smithi045.stdout:1166: done (2 left) 2024-04-26T00:27:15.009 INFO:tasks.rados.rados.0.smithi045.stdout:1169: copy_from oid 2 from oid 20 current snap is 138 2024-04-26T00:27:15.009 INFO:tasks.rados.rados.0.smithi045.stdout:1167: finishing write tid 2 to smithi04526579-28 2024-04-26T00:27:15.009 INFO:tasks.rados.rados.0.smithi045.stdout:1167: finishing write tid 3 to smithi04526579-28 2024-04-26T00:27:15.009 INFO:tasks.rados.rados.0.smithi045.stdout:1167: finishing write tid 4 to smithi04526579-28 2024-04-26T00:27:15.009 INFO:tasks.rados.rados.0.smithi045.stdout:1167: finishing write tid 5 to smithi04526579-28 2024-04-26T00:27:15.009 INFO:tasks.rados.rados.0.smithi045.stdout:1167: oid 28 updating version 0 to 733 2024-04-26T00:27:15.010 INFO:tasks.rados.rados.0.smithi045.stdout:1167: oid 28 updating version 733 to 735 2024-04-26T00:27:15.010 INFO:tasks.rados.rados.0.smithi045.stdout:1167: oid 28 version 735 is already newer than 732 2024-04-26T00:27:15.010 INFO:tasks.rados.rados.0.smithi045.stdout:1167: oid 28 version 735 is already newer than 734 2024-04-26T00:27:15.010 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 735 (ObjNum 436 snap 138 seq_num 436) dirty exists 2024-04-26T00:27:15.010 INFO:tasks.rados.rados.0.smithi045.stdout:1167: left oid 28 (ObjNum 436 snap 138 seq_num 436) 2024-04-26T00:27:15.010 INFO:tasks.rados.rados.0.smithi045.stdout:1168: finishing write tid 1 to smithi04526579-35 2024-04-26T00:27:15.010 INFO:tasks.rados.rados.0.smithi045.stdout:1167: done (2 left) 2024-04-26T00:27:15.010 INFO:tasks.rados.rados.0.smithi045.stdout:1170: delete oid 36 current snap is 138 2024-04-26T00:27:15.010 INFO:tasks.rados.rados.0.smithi045.stdout:1168: finishing write tid 2 to smithi04526579-35 2024-04-26T00:27:15.018 INFO:tasks.rados.rados.0.smithi045.stdout:1168: finishing write tid 3 to smithi04526579-35 2024-04-26T00:27:15.019 INFO:tasks.rados.rados.0.smithi045.stdout:1168: finishing write tid 4 to smithi04526579-35 2024-04-26T00:27:15.019 INFO:tasks.rados.rados.0.smithi045.stdout:1170: done (2 left) 2024-04-26T00:27:15.019 INFO:tasks.rados.rados.0.smithi045.stdout:1171: write oid 26 current snap is 138 2024-04-26T00:27:15.019 INFO:tasks.rados.rados.0.smithi045.stdout:1171: seq_num 439 ranges {578273=466432,1739595=452815,2722929=655344} 2024-04-26T00:27:15.031 INFO:tasks.rados.rados.0.smithi045.stdout:1171: writing smithi04526579-26 from 578273 to 1044705 tid 1 2024-04-26T00:27:15.045 INFO:tasks.rados.rados.0.smithi045.stdout:1171: writing smithi04526579-26 from 1739595 to 2192410 tid 2 2024-04-26T00:27:15.058 INFO:tasks.rados.rados.0.smithi045.stdout:1171: writing smithi04526579-26 from 2722929 to 3378273 tid 3 2024-04-26T00:27:15.059 INFO:tasks.rados.rados.0.smithi045.stdout:1172: copy_from oid 7 from oid 24 current snap is 138 2024-04-26T00:27:15.059 INFO:tasks.rados.rados.0.smithi045.stdout:1168: finishing write tid 5 to smithi04526579-35 2024-04-26T00:27:15.059 INFO:tasks.rados.rados.0.smithi045.stdout:1168: oid 35 updating version 0 to 641 2024-04-26T00:27:15.059 INFO:tasks.rados.rados.0.smithi045.stdout:1168: oid 35 updating version 641 to 643 2024-04-26T00:27:15.059 INFO:tasks.rados.rados.0.smithi045.stdout:1168: oid 35 version 643 is already newer than 642 2024-04-26T00:27:15.059 INFO:tasks.rados.rados.0.smithi045.stdout:1168: oid 35 version 643 is already newer than 640 2024-04-26T00:27:15.059 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 643 (ObjNum 437 snap 138 seq_num 437) dirty exists 2024-04-26T00:27:15.059 INFO:tasks.rados.rados.0.smithi045.stdout:1168: left oid 35 (ObjNum 437 snap 138 seq_num 437) 2024-04-26T00:27:15.059 INFO:tasks.rados.rados.0.smithi045.stdout:1168: done (3 left) 2024-04-26T00:27:15.059 INFO:tasks.rados.rados.0.smithi045.stdout:1173: copy_from oid 46 from oid 36 current snap is 138 2024-04-26T00:27:15.059 INFO:tasks.rados.rados.0.smithi045.stdout:1171: finishing write tid 1 to smithi04526579-26 2024-04-26T00:27:15.059 INFO:tasks.rados.rados.0.smithi045.stdout:1169: finishing copy_from racing read to smithi04526579-2 2024-04-26T00:27:15.060 INFO:tasks.rados.rados.0.smithi045.stdout:1169: finishing copy_from to smithi04526579-2 2024-04-26T00:27:15.060 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 314 (ObjNum 356 snap 105 seq_num 356) dirty exists 2024-04-26T00:27:15.060 INFO:tasks.rados.rados.0.smithi045.stdout:1171: finishing write tid 2 to smithi04526579-26 2024-04-26T00:27:15.060 INFO:tasks.rados.rados.0.smithi045.stdout:1169: done (3 left) 2024-04-26T00:27:15.060 INFO:tasks.rados.rados.0.smithi045.stdout:1174: copy_from oid 22 from oid 12 current snap is 138 2024-04-26T00:27:15.060 INFO:tasks.rados.rados.0.smithi045.stdout:1175: write (excl) oid 9 current snap is 138 2024-04-26T00:27:15.060 INFO:tasks.rados.rados.0.smithi045.stdout:1175: seq_num 440 ranges {595080=541681,1798077=596473,3150352=244728} 2024-04-26T00:27:15.070 INFO:tasks.rados.rados.0.smithi045.stdout:1175: writing smithi04526579-9 from 595080 to 1136761 tid 1 2024-04-26T00:27:15.082 INFO:tasks.rados.rados.0.smithi045.stdout:1175: writing smithi04526579-9 from 1798077 to 2394550 tid 2 2024-04-26T00:27:15.092 INFO:tasks.rados.rados.0.smithi045.stdout:1175: writing smithi04526579-9 from 3150352 to 3395080 tid 3 2024-04-26T00:27:15.092 INFO:tasks.rados.rados.0.smithi045.stdout:1176: copy_from oid 5 from oid 19 current snap is 138 2024-04-26T00:27:15.092 INFO:tasks.rados.rados.0.smithi045.stdout:1172: finishing copy_from to smithi04526579-7 2024-04-26T00:27:15.092 INFO:tasks.rados.rados.0.smithi045.stdout:1172: got expected ENOENT (src dne) 2024-04-26T00:27:15.092 INFO:tasks.rados.rados.0.smithi045.stdout:1177: write (excl) oid 47 current snap is 138 2024-04-26T00:27:15.092 INFO:tasks.rados.rados.0.smithi045.stdout:1172: finishing copy_from racing read to smithi04526579-7 2024-04-26T00:27:15.092 INFO:tasks.rados.rados.0.smithi045.stdout:1173: finishing copy_from to smithi04526579-46 2024-04-26T00:27:15.092 INFO:tasks.rados.rados.0.smithi045.stdout:1173: got expected ENOENT (src dne) 2024-04-26T00:27:15.092 INFO:tasks.rados.rados.0.smithi045.stdout:1173: finishing copy_from racing read to smithi04526579-46 2024-04-26T00:27:15.092 INFO:tasks.rados.rados.0.smithi045.stdout:1171: finishing write tid 3 to smithi04526579-26 2024-04-26T00:27:15.092 INFO:tasks.rados.rados.0.smithi045.stdout:1171: finishing write tid 4 to smithi04526579-26 2024-04-26T00:27:15.092 INFO:tasks.rados.rados.0.smithi045.stdout:1171: finishing write tid 5 to smithi04526579-26 2024-04-26T00:27:15.093 INFO:tasks.rados.rados.0.smithi045.stdout:1171: oid 26 updating version 0 to 740 2024-04-26T00:27:15.093 INFO:tasks.rados.rados.0.smithi045.stdout:1171: oid 26 version 740 is already newer than 739 2024-04-26T00:27:15.093 INFO:tasks.rados.rados.0.smithi045.stdout:1171: oid 26 version 740 is already newer than 737 2024-04-26T00:27:15.093 INFO:tasks.rados.rados.0.smithi045.stdout:1171: oid 26 version 740 is already newer than 738 2024-04-26T00:27:15.093 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 740 (ObjNum 439 snap 138 seq_num 439) dirty exists 2024-04-26T00:27:15.093 INFO:tasks.rados.rados.0.smithi045.stdout:1171: left oid 26 (ObjNum 439 snap 138 seq_num 439) 2024-04-26T00:27:15.093 INFO:tasks.rados.rados.0.smithi045.stdout:1175: finishing write tid 1 to smithi04526579-9 2024-04-26T00:27:15.093 INFO:tasks.rados.rados.0.smithi045.stdout:1174: finishing copy_from racing read to smithi04526579-22 2024-04-26T00:27:15.093 INFO:tasks.rados.rados.0.smithi045.stdout:1177: seq_num 441 ranges {611887=733283,1972912=456484,2894128=517759} 2024-04-26T00:27:15.103 INFO:tasks.rados.rados.0.smithi045.stdout:1177: writing smithi04526579-47 from 611887 to 1345170 tid 1 2024-04-26T00:27:15.113 INFO:tasks.rados.rados.0.smithi045.stdout:1177: writing smithi04526579-47 from 1972912 to 2429396 tid 2 2024-04-26T00:27:15.121 INFO:tasks.rados.rados.0.smithi045.stdout:1177: writing smithi04526579-47 from 2894128 to 3411887 tid 3 2024-04-26T00:27:15.121 INFO:tasks.rados.rados.0.smithi045.stdout:1171: done (6 left) 2024-04-26T00:27:15.121 INFO:tasks.rados.rados.0.smithi045.stdout:1172: done (5 left) 2024-04-26T00:27:15.121 INFO:tasks.rados.rados.0.smithi045.stdout:1173: done (4 left) 2024-04-26T00:27:15.121 INFO:tasks.rados.rados.0.smithi045.stdout:1178: copy_from oid 16 from oid 4 current snap is 138 2024-04-26T00:27:15.121 INFO:tasks.rados.rados.0.smithi045.stdout:1175: finishing write tid 2 to smithi04526579-9 2024-04-26T00:27:15.122 INFO:tasks.rados.rados.0.smithi045.stdout:1179: delete oid 37 current snap is 138 2024-04-26T00:27:15.122 INFO:tasks.rados.rados.0.smithi045.stdout:1174: finishing copy_from to smithi04526579-22 2024-04-26T00:27:15.122 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 383 (ObjNum 401 snap 122 seq_num 401) dirty exists 2024-04-26T00:27:15.122 INFO:tasks.rados.rados.0.smithi045.stdout:1175: finishing write tid 3 to smithi04526579-9 2024-04-26T00:27:15.122 INFO:tasks.rados.rados.0.smithi045.stdout:1175: finishing write tid 4 to smithi04526579-9 2024-04-26T00:27:15.122 INFO:tasks.rados.rados.0.smithi045.stdout:1175: finishing write tid 5 to smithi04526579-9 2024-04-26T00:27:15.122 INFO:tasks.rados.rados.0.smithi045.stdout:1175: oid 9 updating version 0 to 613 2024-04-26T00:27:15.122 INFO:tasks.rados.rados.0.smithi045.stdout:1175: oid 9 updating version 613 to 614 2024-04-26T00:27:15.122 INFO:tasks.rados.rados.0.smithi045.stdout:1175: oid 9 version 614 is already newer than 612 2024-04-26T00:27:15.122 INFO:tasks.rados.rados.0.smithi045.stdout:1175: oid 9 version 614 is already newer than 611 2024-04-26T00:27:15.122 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 614 (ObjNum 440 snap 138 seq_num 440) dirty exists 2024-04-26T00:27:15.122 INFO:tasks.rados.rados.0.smithi045.stdout:1175: left oid 9 (ObjNum 440 snap 138 seq_num 440) 2024-04-26T00:27:15.122 INFO:tasks.rados.rados.0.smithi045.stdout:1176: finishing copy_from racing read to smithi04526579-5 2024-04-26T00:27:15.122 INFO:tasks.rados.rados.0.smithi045.stdout:1177: finishing write tid 1 to smithi04526579-47 2024-04-26T00:27:15.123 INFO:tasks.rados.rados.0.smithi045.stdout:1177: finishing write tid 2 to smithi04526579-47 2024-04-26T00:27:15.126 INFO:tasks.rados.rados.0.smithi045.stdout:1176: finishing copy_from to smithi04526579-5 2024-04-26T00:27:15.126 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 599 (ObjNum 224 snap 59 seq_num 224) dirty exists 2024-04-26T00:27:15.130 INFO:tasks.rados.rados.0.smithi045.stdout:1177: finishing write tid 3 to smithi04526579-47 2024-04-26T00:27:15.130 INFO:tasks.rados.rados.0.smithi045.stdout:1174: done (5 left) 2024-04-26T00:27:15.131 INFO:tasks.rados.rados.0.smithi045.stdout:1175: done (4 left) 2024-04-26T00:27:15.131 INFO:tasks.rados.rados.0.smithi045.stdout:1176: done (3 left) 2024-04-26T00:27:15.131 INFO:tasks.rados.rados.0.smithi045.stdout:1179: done (2 left) 2024-04-26T00:27:15.131 INFO:tasks.rados.rados.0.smithi045.stdout:1180: write oid 19 current snap is 138 2024-04-26T00:27:15.131 INFO:tasks.rados.rados.0.smithi045.stdout:1180: seq_num 443 ranges {645501=483781,1573523=460153,2548974=763384} 2024-04-26T00:27:15.138 INFO:tasks.rados.rados.0.smithi045.stdout:1180: writing smithi04526579-19 from 645501 to 1129282 tid 1 2024-04-26T00:27:15.145 INFO:tasks.rados.rados.0.smithi045.stdout:1180: writing smithi04526579-19 from 1573523 to 2033676 tid 2 2024-04-26T00:27:15.155 INFO:tasks.rados.rados.0.smithi045.stdout:1180: writing smithi04526579-19 from 2548974 to 3312358 tid 3 2024-04-26T00:27:15.155 INFO:tasks.rados.rados.0.smithi045.stdout:1181: copy_from oid 46 from oid 45 current snap is 138 2024-04-26T00:27:15.155 INFO:tasks.rados.rados.0.smithi045.stdout:1177: finishing write tid 4 to smithi04526579-47 2024-04-26T00:27:15.155 INFO:tasks.rados.rados.0.smithi045.stdout:1182: snap_remove snap 129 2024-04-26T00:27:15.920 INFO:tasks.rados.rados.0.smithi045.stdout:1182: done (4 left) 2024-04-26T00:27:15.920 INFO:tasks.rados.rados.0.smithi045.stdout:1183: snap_remove snap 136 2024-04-26T00:27:16.901 INFO:tasks.rados.rados.0.smithi045.stdout:1183: done (4 left) 2024-04-26T00:27:16.902 INFO:tasks.rados.rados.0.smithi045.stdout:1184: snap_remove snap 135 2024-04-26T00:27:17.905 INFO:tasks.rados.rados.0.smithi045.stdout:1184: done (4 left) 2024-04-26T00:27:17.905 INFO:tasks.rados.rados.0.smithi045.stdout:1185: snap_remove snap 113 2024-04-26T00:27:18.907 INFO:tasks.rados.rados.0.smithi045.stdout:1185: done (4 left) 2024-04-26T00:27:18.907 INFO:tasks.rados.rados.0.smithi045.stdout:1186: read oid 1 snap -1 2024-04-26T00:27:18.907 INFO:tasks.rados.rados.0.smithi045.stdout:1186: expect deleted 2024-04-26T00:27:18.907 INFO:tasks.rados.rados.0.smithi045.stdout:1187: snap_remove snap 71 2024-04-26T00:27:19.147 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-04-26T00:27:19.147 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-04-26T00:27:19.148 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-26T00:27:19.148 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-26T00:27:19.151 INFO:tasks.ceph.mon.a:Started 2024-04-26T00:27:19.151 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:27:19.151 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:27:19.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.288+0000 7f5af24b9640 10 monclient: get_monmap_and_config 2024-04-26T00:27:19.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.288+0000 7f5af24b9640 10 monclient: build_initial_monmap 2024-04-26T00:27:19.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.288+0000 7f5af24b9640 10 monclient: monmap: 2024-04-26T00:27:19.276 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:27:19.276 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:19.276 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:27:19.294336+0000 2024-04-26T00:27:19.276 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:27:19.294336+0000 2024-04-26T00:27:19.276 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:27:19.277 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:19.277 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:27:19.277 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:27:19.277 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:27:19.277 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:27:19.277 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:27:19.277 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:19.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.288+0000 7f5af24b9640 1 Processor -- start 2024-04-26T00:27:19.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.288+0000 7f5af24b9640 1 -- start start 2024-04-26T00:27:19.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.288+0000 7f5af24b9640 10 monclient: init 2024-04-26T00:27:19.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.288+0000 7f5af24b9640 10 monclient: _reopen_session rank -1 2024-04-26T00:27:19.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.288+0000 7f5af24b9640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-04-26T00:27:19.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.288+0000 7f5af24b9640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f5aec0a0260 0x7f5aec0a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:19.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.288+0000 7f5af24b9640 10 monclient(hunting): picked mon.noname-a con 0x7f5aec0a0260 addr v2:172.21.15.37:3300/0 2024-04-26T00:27:19.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.288+0000 7f5af24b9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5aec134b30 addr v1:172.21.15.37:6789/0 2024-04-26T00:27:19.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.288+0000 7f5af24b9640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5aec135660 0x7f5aec13ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:19.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.288+0000 7f5af24b9640 10 monclient(hunting): picked mon.noname-c con 0x7f5aec135660 addr v2:172.21.15.45:3300/0 2024-04-26T00:27:19.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.288+0000 7f5af24b9640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5aec0a1130 con 0x7f5aec134b30 2024-04-26T00:27:19.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.288+0000 7f5af24b9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:19.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.288+0000 7f5af24b9640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f5aec09ec10 con 0x7f5aec0a0260 2024-04-26T00:27:19.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.288+0000 7f5af24b9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:19.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.288+0000 7f5af24b9640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f5aec09f0f0 con 0x7f5aec135660 2024-04-26T00:27:19.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.288+0000 7f5af24b9640 10 monclient(hunting): _renew_subs 2024-04-26T00:27:19.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.288+0000 7f5af24b9640 10 monclient(hunting): authenticate will time out at 1420.599717s 2024-04-26T00:27:19.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.288+0000 7f5aebfff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f5aec0a0260 0x7f5aec0a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:19.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.288+0000 7f5aebfff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f5aec0a0260 0x7f5aec0a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:57960/0 (socket says 172.21.15.37:57960) 2024-04-26T00:27:19.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.288+0000 7f5aebfff640 1 -- 172.21.15.37:0/2801354512 learned_addr learned my addr 172.21.15.37:0/2801354512 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:27:19.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.288+0000 7f5aeb7fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f5aec134b30 0x7f5aec134f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:59714/0 (socket says 172.21.15.37:59714) 2024-04-26T00:27:19.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.288+0000 7f5aebfff640 10 monclient(hunting): get_auth_request con 0x7f5aec0a0260 auth_method 0 2024-04-26T00:27:19.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.288+0000 7f5aebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:19.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.288+0000 7f5aebfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:19.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.288+0000 7f5aebfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:19.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.292+0000 7f5aebfff640 1 -- 172.21.15.37:0/2801354512 >> v2:172.21.15.37:3300/0 conn(0x7f5aec0a0260 msgr2=0x7f5aec0a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:27:19.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.292+0000 7f5aebfff640 1 -- 172.21.15.37:0/2801354512 >> v2:172.21.15.37:3300/0 conn(0x7f5aec0a0260 msgr2=0x7f5aec0a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:27:19.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.292+0000 7f5aebfff640 1 --2- 172.21.15.37:0/2801354512 >> v2:172.21.15.37:3300/0 conn(0x7f5aec0a0260 0x7f5aec0a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:27:19.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.292+0000 7f5aebfff640 1 --2- 172.21.15.37:0/2801354512 >> v2:172.21.15.37:3300/0 conn(0x7f5aec0a0260 0x7f5aec0a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:27:19.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.292+0000 7f5af0a56640 1 --2- 172.21.15.37:0/2801354512 >> v2:172.21.15.45:3300/0 conn(0x7f5aec135660 0x7f5aec13ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:19.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.292+0000 7f5af0a56640 10 monclient(hunting): get_auth_request con 0x7f5aec135660 auth_method 0 2024-04-26T00:27:19.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.292+0000 7f5af0a56640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:19.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.292+0000 7f5af0a56640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:19.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.292+0000 7f5af0a56640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:19.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.292+0000 7f5af0a56640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:19.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.292+0000 7f5af0a56640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:19.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.292+0000 7f5af0a56640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:19.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.292+0000 7f5af0a56640 10 monclient(hunting): handle_auth_done global_id 24670 payload 274 2024-04-26T00:27:19.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.292+0000 7f5af0a56640 1 -- 172.21.15.37:0/2801354512 >> v2:172.21.15.37:3300/0 conn(0x7f5aec0a0260 msgr2=0x7f5aec0a0640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:27:19.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.292+0000 7f5af0a56640 1 --2- 172.21.15.37:0/2801354512 >> v2:172.21.15.37:3300/0 conn(0x7f5aec0a0260 0x7f5aec0a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:19.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.292+0000 7f5af0a56640 1 -- 172.21.15.37:0/2801354512 >> v1:172.21.15.37:6789/0 conn(0x7f5aec134b30 legacy=0x7f5aec134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:27:19.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.292+0000 7f5af0a56640 10 monclient: _finish_hunting 0 2024-04-26T00:27:19.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.292+0000 7f5af0a56640 1 monclient: found mon.noname-c 2024-04-26T00:27:19.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.292+0000 7f5af0a56640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:27:19.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.292+0000 7f5af0a56640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:27:19.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.292+0000 7f5af0a56640 1 -- 172.21.15.37:0/2801354512 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aec06cff0 con 0x7f5aec135660 2024-04-26T00:27:19.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.296+0000 7f5af0a56640 1 -- 172.21.15.37:0/2801354512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5aec135660 msgr2=0x7f5aec13ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:19.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.296+0000 7f5af0a56640 1 --2- 172.21.15.37:0/2801354512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5aec135660 0x7f5aec13ce70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5aec0a1a60 tx=0x7f5ae401c5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=998f5fc64176abf0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:19.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.296+0000 7f5aeaffd640 1 -- 172.21.15.37:0/2801354512 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae40a6040 con 0x7f5aec135660 2024-04-26T00:27:19.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.296+0000 7f5aeaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:19.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.296+0000 7f5aeaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:27:19.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.296+0000 7f5aeaffd640 10 monclient: dump: 2024-04-26T00:27:19.284 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:19.284 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:19.284 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:19.284 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:19.284 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:19.285 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:19.285 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:19.285 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:19.285 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:19.285 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:19.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.296+0000 7f5aeaffd640 10 monclient: _finish_auth 0 2024-04-26T00:27:19.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.296+0000 7f5aeaffd640 10 monclient: _check_auth_tickets 2024-04-26T00:27:19.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.296+0000 7f5aeaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:27:19.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.296+0000 7f5aeaffd640 1 -- 172.21.15.37:0/2801354512 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ae40af070 con 0x7f5aec135660 2024-04-26T00:27:19.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.296+0000 7f5aeaffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:27:19.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.296+0000 7f5aeaffd640 1 -- 172.21.15.37:0/2801354512 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae40aa860 con 0x7f5aec135660 2024-04-26T00:27:19.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.296+0000 7f5af24b9640 5 monclient: authenticate success, global_id 24670 2024-04-26T00:27:19.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.296+0000 7f5af24b9640 10 monclient: get_monmap_and_config success 2024-04-26T00:27:19.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.296+0000 7f5af24b9640 4 set_mon_vals no callback set 2024-04-26T00:27:19.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.296+0000 7f5aeaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:19.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.296+0000 7f5aeaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:27:19.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.296+0000 7f5aeaffd640 10 monclient: dump: 2024-04-26T00:27:19.286 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:19.286 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:19.286 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:19.286 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:19.286 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:19.286 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:19.286 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:19.286 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:19.286 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:19.286 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:19.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.296+0000 7f5af24b9640 10 monclient: shutdown 2024-04-26T00:27:19.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.296+0000 7f5af24b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:27:19.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.296+0000 7f5af24b9640 1 -- 172.21.15.37:0/2801354512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5aec135660 msgr2=0x7f5aec13ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:19.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.296+0000 7f5af24b9640 1 --2- 172.21.15.37:0/2801354512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5aec135660 0x7f5aec13ce70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5aec0a1a60 tx=0x7f5ae401c5d0 comp rx=0 tx=0).stop 2024-04-26T00:27:19.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.296+0000 7f5af24b9640 1 -- 172.21.15.37:0/2801354512 shutdown_connections 2024-04-26T00:27:19.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.296+0000 7f5af24b9640 1 --2- 172.21.15.37:0/2801354512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5aec135660 0x7f5aec13ce70 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:19.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.296+0000 7f5af24b9640 1 --2- 172.21.15.37:0/2801354512 >> v2:172.21.15.37:3300/0 conn(0x7f5aec0a0260 0x7f5aec0a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:19.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.296+0000 7f5af24b9640 1 -- 172.21.15.37:0/2801354512 >> 172.21.15.37:0/2801354512 conn(0x7f5aec0a1ad0 msgr2=0x7f5aec0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:27:19.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.296+0000 7f5af24b9640 1 -- 172.21.15.37:0/2801354512 shutdown_connections 2024-04-26T00:27:19.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.296+0000 7f5af24b9640 1 -- 172.21.15.37:0/2801354512 wait complete. 2024-04-26T00:27:19.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.296+0000 7f5af24b9640 10 monclient: build_initial_monmap 2024-04-26T00:27:19.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.296+0000 7f5af24b9640 10 monclient: monmap: 2024-04-26T00:27:19.292 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:27:19.292 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:27:19.292 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:27:19.292 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:27:19.292 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:27:19.292 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:27:19.293 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:27:19.293 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:27:19.293 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:27:19.293 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:19.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5af24b9640 1 Processor -- start 2024-04-26T00:27:19.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5af24b9640 1 -- start start 2024-04-26T00:27:19.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5af24b9640 10 monclient: init 2024-04-26T00:27:19.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5af24b9640 10 monclient: _reopen_session rank -1 2024-04-26T00:27:19.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5af24b9640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:27:19.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5af24b9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5aec0a0260 0x7f5aec1cccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:19.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5af24b9640 10 monclient(hunting): picked mon.noname-b con 0x7f5aec0a0260 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:27:19.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5af24b9640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5aec134b30 0x7f5aec131ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:19.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5af24b9640 10 monclient(hunting): picked mon.noname-a con 0x7f5aec134b30 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:27:19.293 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5af24b9640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5aec135660 0x7f5aec1399a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:19.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5af24b9640 10 monclient(hunting): picked mon.noname-c con 0x7f5aec135660 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:27:19.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5af24b9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:19.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5af24b9640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f5aec0a0010 con 0x7f5aec134b30 2024-04-26T00:27:19.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5af24b9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:19.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5af24b9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5aec09fe90 con 0x7f5aec0a0260 2024-04-26T00:27:19.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5af24b9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:19.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5af24b9640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5aec13fbd0 con 0x7f5aec135660 2024-04-26T00:27:19.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5af24b9640 10 monclient(hunting): _renew_subs 2024-04-26T00:27:19.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5af24b9640 10 monclient(hunting): authenticate will time out at 1420.608980s 2024-04-26T00:27:19.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5aeb7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5aec134b30 0x7f5aec131ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:19.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5af0a56640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5aec135660 0x7f5aec1399a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:19.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5aeb7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5aec134b30 0x7f5aec131ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:57962/0 (socket says 172.21.15.37:57962) 2024-04-26T00:27:19.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5aeb7fe640 1 -- 172.21.15.37:0/1288165472 learned_addr learned my addr 172.21.15.37:0/1288165472 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:27:19.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5aeb7fe640 10 monclient(hunting): get_auth_request con 0x7f5aec134b30 auth_method 0 2024-04-26T00:27:19.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5aeb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:19.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5aeb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:19.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5aeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:19.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5af0a56640 10 monclient(hunting): get_auth_request con 0x7f5aec135660 auth_method 0 2024-04-26T00:27:19.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5af0a56640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:19.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5af0a56640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:19.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5af0a56640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:19.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5aeb7fe640 1 -- 172.21.15.37:0/1288165472 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5aec134b30 msgr2=0x7f5aec131ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:27:19.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5aeb7fe640 1 -- 172.21.15.37:0/1288165472 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5aec134b30 msgr2=0x7f5aec131ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:27:19.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5aeb7fe640 1 --2- 172.21.15.37:0/1288165472 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5aec134b30 0x7f5aec131ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:27:19.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5af0a56640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:19.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5af0a56640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:19.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5af0a56640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:19.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5aeb7fe640 1 --2- 172.21.15.37:0/1288165472 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5aec134b30 0x7f5aec131ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:27:19.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5aebfff640 1 --2- 172.21.15.37:0/1288165472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5aec0a0260 0x7f5aec1cccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:19.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5aebfff640 10 monclient(hunting): get_auth_request con 0x7f5aec0a0260 auth_method 0 2024-04-26T00:27:19.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5aebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:19.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5aebfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:19.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5aebfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:19.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5af0a56640 10 monclient(hunting): handle_auth_done global_id 84170 payload 824 2024-04-26T00:27:19.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5af0a56640 1 -- 172.21.15.37:0/1288165472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5aec0a0260 msgr2=0x7f5aec1cccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:19.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5af0a56640 1 --2- 172.21.15.37:0/1288165472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5aec0a0260 0x7f5aec1cccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:19.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5af0a56640 1 -- 172.21.15.37:0/1288165472 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5aec134b30 msgr2=0x7f5aec131ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:27:19.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5af0a56640 1 --2- 172.21.15.37:0/1288165472 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5aec134b30 0x7f5aec131ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:19.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5af0a56640 10 monclient: _finish_hunting 0 2024-04-26T00:27:19.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5af0a56640 1 monclient: found mon.noname-c 2024-04-26T00:27:19.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5af0a56640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:27:19.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5af0a56640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:27:19.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5af0a56640 1 -- 172.21.15.37:0/1288165472 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aec11b7a0 con 0x7f5aec135660 2024-04-26T00:27:19.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5af0a56640 1 -- 172.21.15.37:0/1288165472 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5aec135660 msgr2=0x7f5aec1399a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:19.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5af0a56640 1 --2- 172.21.15.37:0/1288165472 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5aec135660 0x7f5aec1399a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5aec09f690 tx=0x7f5ae4019370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:19.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.300+0000 7f5ae8ff9640 1 -- 172.21.15.37:0/1288165472 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae40aad70 con 0x7f5aec135660 2024-04-26T00:27:19.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.304+0000 7f5ae8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:19.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.304+0000 7f5ae8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:27:19.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.304+0000 7f5ae8ff9640 10 monclient: dump: 2024-04-26T00:27:19.297 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:19.297 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:19.297 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:19.297 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:19.297 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:19.298 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:19.298 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:19.298 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:19.298 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:19.298 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:19.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.304+0000 7f5ae8ff9640 10 monclient: _finish_auth 0 2024-04-26T00:27:19.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.304+0000 7f5ae8ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:27:19.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.304+0000 7f5ae8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:27:19.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.304+0000 7f5af24b9640 5 monclient: authenticate success, global_id 84170 2024-04-26T00:27:19.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.304+0000 7f5ae8ff9640 1 -- 172.21.15.37:0/1288165472 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ae40bd3f0 con 0x7f5aec135660 2024-04-26T00:27:19.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.304+0000 7f5ae8ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:27:19.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.304+0000 7f5ae8ff9640 1 -- 172.21.15.37:0/1288165472 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae40bd7e0 con 0x7f5aec135660 2024-04-26T00:27:19.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.304+0000 7f5af24b9640 10 monclient: _renew_subs 2024-04-26T00:27:19.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.304+0000 7f5af24b9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:27:19.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.304+0000 7f5af24b9640 1 -- 172.21.15.37:0/1288165472 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5aec134770 con 0x7f5aec135660 2024-04-26T00:27:19.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.304+0000 7f5ae9ffb640 4 set_mon_vals no callback set 2024-04-26T00:27:19.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.304+0000 7f5ae8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:19.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.304+0000 7f5ae8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:27:19.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.304+0000 7f5ae8ff9640 10 monclient: dump: 2024-04-26T00:27:19.299 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:19.299 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:19.299 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:19.299 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:19.299 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:19.299 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:19.299 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:19.299 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:19.299 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:19.300 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:19.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.304+0000 7f5af24b9640 10 monclient: _renew_subs 2024-04-26T00:27:19.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.304+0000 7f5af24b9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:27:19.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.304+0000 7f5af24b9640 1 -- 172.21.15.37:0/1288165472 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5aec13c1a0 con 0x7f5aec135660 2024-04-26T00:27:19.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.304+0000 7f5ae8ff9640 1 -- 172.21.15.37:0/1288165472 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f5ae40bd980 con 0x7f5aec135660 2024-04-26T00:27:19.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.304+0000 7f5af24b9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:19.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.304+0000 7f5af24b9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:19.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.304+0000 7f5af24b9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:27:19.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.304+0000 7f5af24b9640 1 -- 172.21.15.37:0/1288165472 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5aec066f40 con 0x7f5aec135660 2024-04-26T00:27:19.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.304+0000 7f5ae8ff9640 1 --2- 172.21.15.37:0/1288165472 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5ad8035e80 0x7f5ad8038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:19.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.304+0000 7f5ae8ff9640 1 -- 172.21.15.37:0/1288165472 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(285..285 src has 177..285) ==== 4007+0+0 (secure 0 0 0) 0x7f5ae40e27c0 con 0x7f5aec135660 2024-04-26T00:27:19.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.304+0000 7f5aebfff640 1 --2- 172.21.15.37:0/1288165472 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5ad8035e80 0x7f5ad8038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:19.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.304+0000 7f5aebfff640 10 monclient: get_auth_request con 0x7f5ad8035e80 auth_method 0 2024-04-26T00:27:19.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.308+0000 7f5aebfff640 1 --2- 172.21.15.37:0/1288165472 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5ad8035e80 0x7f5ad8038340 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5adc009100 tx=0x7f5adc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:19.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.308+0000 7f5ae8ff9640 1 -- 172.21.15.37:0/1288165472 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5ae40c2070 con 0x7f5aec135660 2024-04-26T00:27:19.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.308+0000 7f5ae8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:19.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.308+0000 7f5ae8ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:27:19.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.528+0000 7f5af24b9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:27:19.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.528+0000 7f5af24b9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:27:19.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.528+0000 7f5af24b9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:27:19.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.528+0000 7f5af24b9640 1 -- 172.21.15.37:0/1288165472 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5aec133580 con 0x7f5aec135660 2024-04-26T00:27:19.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.528+0000 7f5ae8ff9640 1 -- 172.21.15.37:0/1288165472 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f5aec0a1130 con 0x7f5aec135660 2024-04-26T00:27:19.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.532+0000 7f5ae8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:27:19.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.532+0000 7f5ae8ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:27:19.518 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":56,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:27:19.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.532+0000 7f5af24b9640 1 -- 172.21.15.37:0/1288165472 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5ad8035e80 msgr2=0x7f5ad8038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:19.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.532+0000 7f5af24b9640 1 --2- 172.21.15.37:0/1288165472 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5ad8035e80 0x7f5ad8038340 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5adc009100 tx=0x7f5adc008040 comp rx=0 tx=0).stop 2024-04-26T00:27:19.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.532+0000 7f5af24b9640 10 monclient: shutdown 2024-04-26T00:27:19.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.532+0000 7f5af24b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:27:19.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.532+0000 7f5af24b9640 1 -- 172.21.15.37:0/1288165472 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5aec135660 msgr2=0x7f5aec1399a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:19.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.532+0000 7f5af24b9640 1 --2- 172.21.15.37:0/1288165472 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5aec135660 0x7f5aec1399a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5aec09f690 tx=0x7f5ae4019370 comp rx=0 tx=0).stop 2024-04-26T00:27:19.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.532+0000 7f5af24b9640 1 -- 172.21.15.37:0/1288165472 shutdown_connections 2024-04-26T00:27:19.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.532+0000 7f5af24b9640 1 --2- 172.21.15.37:0/1288165472 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5ad8035e80 0x7f5ad8038340 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:19.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.532+0000 7f5af24b9640 1 --2- 172.21.15.37:0/1288165472 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5aec135660 0x7f5aec1399a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:19.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.532+0000 7f5af24b9640 1 --2- 172.21.15.37:0/1288165472 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5aec134b30 0x7f5aec131ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:19.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.532+0000 7f5af24b9640 1 --2- 172.21.15.37:0/1288165472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5aec0a0260 0x7f5aec1cccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:19.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.532+0000 7f5af24b9640 1 -- 172.21.15.37:0/1288165472 >> 172.21.15.37:0/1288165472 conn(0x7f5aec0a1ad0 msgr2=0x7f5aec0a3d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:27:19.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.532+0000 7f5af24b9640 1 -- 172.21.15.37:0/1288165472 shutdown_connections 2024-04-26T00:27:19.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:19.532+0000 7f5af24b9640 1 -- 172.21.15.37:0/1288165472 wait complete. 2024-04-26T00:27:19.909 INFO:tasks.rados.rados.0.smithi045.stdout:1187: done (5 left) 2024-04-26T00:27:19.910 INFO:tasks.rados.rados.0.smithi045.stdout:1188: read oid 38 snap -1 2024-04-26T00:27:19.910 INFO:tasks.rados.rados.0.smithi045.stdout:1188: expect (ObjNum 391 snap 120 seq_num 391) 2024-04-26T00:27:19.910 INFO:tasks.rados.rados.0.smithi045.stdout:1189: read oid 7 snap -1 2024-04-26T00:27:19.910 INFO:tasks.rados.rados.0.smithi045.stdout:1189: expect (ObjNum 359 snap 107 seq_num 359) 2024-04-26T00:27:19.910 INFO:tasks.rados.rados.0.smithi045.stdout:1190: write (excl) oid 26 current snap is 138 2024-04-26T00:27:19.910 INFO:tasks.rados.rados.0.smithi045.stdout:1190: seq_num 444 ranges {662308=559030,1632005=603811,2860044=602264} 2024-04-26T00:27:19.930 INFO:tasks.rados.rados.0.smithi045.stdout:1190: writing smithi04526579-26 from 662308 to 1221338 tid 1 2024-04-26T00:27:19.947 INFO:tasks.rados.rados.0.smithi045.stdout:1190: writing smithi04526579-26 from 1632005 to 2235816 tid 2 2024-04-26T00:27:19.965 INFO:tasks.rados.rados.0.smithi045.stdout:1190: writing smithi04526579-26 from 2860044 to 3462308 tid 3 2024-04-26T00:27:19.965 INFO:tasks.rados.rados.0.smithi045.stdout:1191: rollback oid 20 current snap is 138 2024-04-26T00:27:19.966 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 20 to 134 2024-04-26T00:27:19.966 INFO:tasks.rados.rados.0.smithi045.stdout:1177: finishing write tid 5 to smithi04526579-47 2024-04-26T00:27:19.966 INFO:tasks.rados.rados.0.smithi045.stdout:1177: oid 47 updating version 0 to 692 2024-04-26T00:27:19.966 INFO:tasks.rados.rados.0.smithi045.stdout:1177: oid 47 version 692 is already newer than 691 2024-04-26T00:27:19.966 INFO:tasks.rados.rados.0.smithi045.stdout:1177: oid 47 version 692 is already newer than 689 2024-04-26T00:27:19.966 INFO:tasks.rados.rados.0.smithi045.stdout:1177: oid 47 version 692 is already newer than 690 2024-04-26T00:27:19.966 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 692 (ObjNum 441 snap 138 seq_num 441) dirty exists 2024-04-26T00:27:19.966 INFO:tasks.rados.rados.0.smithi045.stdout:1177: left oid 47 (ObjNum 441 snap 138 seq_num 441) 2024-04-26T00:27:19.966 INFO:tasks.rados.rados.0.smithi045.stdout:1178: finishing copy_from racing read to smithi04526579-16 2024-04-26T00:27:19.966 INFO:tasks.rados.rados.0.smithi045.stdout:1180: finishing write tid 1 to smithi04526579-19 2024-04-26T00:27:19.966 INFO:tasks.rados.rados.0.smithi045.stdout:1177: done (8 left) 2024-04-26T00:27:19.966 INFO:tasks.rados.rados.0.smithi045.stdout:1192: rollback oid 18 current snap is 138 2024-04-26T00:27:19.967 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 18 to 134 2024-04-26T00:27:19.967 INFO:tasks.rados.rados.0.smithi045.stdout:1178: finishing copy_from to smithi04526579-16 2024-04-26T00:27:19.967 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 560 (ObjNum 406 snap 125 seq_num 406) dirty exists 2024-04-26T00:27:19.967 INFO:tasks.rados.rados.0.smithi045.stdout:1180: finishing write tid 2 to smithi04526579-19 2024-04-26T00:27:19.967 INFO:tasks.rados.rados.0.smithi045.stdout:1180: finishing write tid 3 to smithi04526579-19 2024-04-26T00:27:19.967 INFO:tasks.rados.rados.0.smithi045.stdout:1180: finishing write tid 4 to smithi04526579-19 2024-04-26T00:27:19.967 INFO:tasks.rados.rados.0.smithi045.stdout:1180: finishing write tid 5 to smithi04526579-19 2024-04-26T00:27:19.967 INFO:tasks.rados.rados.0.smithi045.stdout:1180: oid 19 updating version 0 to 645 2024-04-26T00:27:19.967 INFO:tasks.rados.rados.0.smithi045.stdout:1180: oid 19 updating version 645 to 648 2024-04-26T00:27:19.967 INFO:tasks.rados.rados.0.smithi045.stdout:1180: oid 19 version 648 is already newer than 647 2024-04-26T00:27:19.967 INFO:tasks.rados.rados.0.smithi045.stdout:1180: oid 19 version 648 is already newer than 646 2024-04-26T00:27:19.967 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 648 (ObjNum 443 snap 138 seq_num 443) dirty exists 2024-04-26T00:27:19.967 INFO:tasks.rados.rados.0.smithi045.stdout:1180: left oid 19 (ObjNum 443 snap 138 seq_num 443) 2024-04-26T00:27:19.967 INFO:tasks.rados.rados.0.smithi045.stdout:1181: finishing copy_from racing read to smithi04526579-46 2024-04-26T00:27:19.967 INFO:tasks.rados.rados.0.smithi045.stdout:1181: finishing copy_from to smithi04526579-46 2024-04-26T00:27:19.968 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 380 (ObjNum 386 snap 117 seq_num 386) dirty exists 2024-04-26T00:27:19.968 INFO:tasks.rados.rados.0.smithi045.stdout:1189: expect (ObjNum 359 snap 107 seq_num 359) 2024-04-26T00:27:20.094 INFO:tasks.rados.rados.0.smithi045.stdout:1188: expect (ObjNum 391 snap 120 seq_num 391) 2024-04-26T00:27:20.177 INFO:tasks.rados.rados.0.smithi045.stdout:1190: finishing write tid 1 to smithi04526579-26 2024-04-26T00:27:20.178 INFO:tasks.rados.rados.0.smithi045.stdout:1190: finishing write tid 2 to smithi04526579-26 2024-04-26T00:27:20.178 INFO:tasks.rados.rados.0.smithi045.stdout:1178: done (8 left) 2024-04-26T00:27:20.178 INFO:tasks.rados.rados.0.smithi045.stdout:1180: done (7 left) 2024-04-26T00:27:20.178 INFO:tasks.rados.rados.0.smithi045.stdout:1181: done (6 left) 2024-04-26T00:27:20.178 INFO:tasks.rados.rados.0.smithi045.stdout:1186: done (5 left) 2024-04-26T00:27:20.178 INFO:tasks.rados.rados.0.smithi045.stdout:1188: done (4 left) 2024-04-26T00:27:20.178 INFO:tasks.rados.rados.0.smithi045.stdout:1189: done (3 left) 2024-04-26T00:27:20.178 INFO:tasks.rados.rados.0.smithi045.stdout:1193: read oid 38 snap -1 2024-04-26T00:27:20.178 INFO:tasks.rados.rados.0.smithi045.stdout:1193: expect (ObjNum 391 snap 120 seq_num 391) 2024-04-26T00:27:20.179 INFO:tasks.rados.rados.0.smithi045.stdout:1194: read oid 33 snap -1 2024-04-26T00:27:20.179 INFO:tasks.rados.rados.0.smithi045.stdout:1194: expect (ObjNum 405 snap 125 seq_num 405) 2024-04-26T00:27:20.179 INFO:tasks.rados.rados.0.smithi045.stdout:1195: snap_create 2024-04-26T00:27:20.179 INFO:tasks.rados.rados.0.smithi045.stdout:1192: finishing rollback tid 1 to smithi04526579-18 2024-04-26T00:27:20.179 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 833 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-04-26T00:27:20.179 INFO:tasks.rados.rados.0.smithi045.stdout:1190: finishing write tid 3 to smithi04526579-26 2024-04-26T00:27:20.179 INFO:tasks.rados.rados.0.smithi045.stdout:1190: finishing write tid 4 to smithi04526579-26 2024-04-26T00:27:20.179 INFO:tasks.rados.rados.0.smithi045.stdout:1190: finishing write tid 5 to smithi04526579-26 2024-04-26T00:27:20.179 INFO:tasks.rados.rados.0.smithi045.stdout:1190: oid 26 updating version 0 to 777 2024-04-26T00:27:20.179 INFO:tasks.rados.rados.0.smithi045.stdout:1190: oid 26 version 777 is already newer than 774 2024-04-26T00:27:20.179 INFO:tasks.rados.rados.0.smithi045.stdout:1190: oid 26 updating version 777 to 778 2024-04-26T00:27:20.179 INFO:tasks.rados.rados.0.smithi045.stdout:1190: oid 26 version 778 is already newer than 765 2024-04-26T00:27:20.180 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 778 (ObjNum 444 snap 138 seq_num 444) dirty exists 2024-04-26T00:27:20.180 INFO:tasks.rados.rados.0.smithi045.stdout:1190: left oid 26 (ObjNum 444 snap 138 seq_num 444) 2024-04-26T00:27:20.180 INFO:tasks.rados.rados.0.smithi045.stdout:1191: finishing rollback tid 0 to smithi04526579-20 2024-04-26T00:27:20.180 INFO:tasks.rados.rados.0.smithi045.stdout:1191: finishing rollback tid 1 to smithi04526579-20 2024-04-26T00:27:20.180 INFO:tasks.rados.rados.0.smithi045.stdout:1191: finishing rollback tid 2 to smithi04526579-20 2024-04-26T00:27:20.180 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 437 (ObjNum 356 snap 105 seq_num 356) dirty exists 2024-04-26T00:27:20.190 INFO:tasks.rados.rados.0.smithi045.stdout:1194: expect (ObjNum 405 snap 125 seq_num 405) 2024-04-26T00:27:20.384 INFO:tasks.rados.rados.0.smithi045.stdout:1193: expect (ObjNum 391 snap 120 seq_num 391) 2024-04-26T00:27:20.941 INFO:tasks.rados.rados.0.smithi045.stdout:1190: done (5 left) 2024-04-26T00:27:20.941 INFO:tasks.rados.rados.0.smithi045.stdout:1191: done (4 left) 2024-04-26T00:27:20.941 INFO:tasks.rados.rados.0.smithi045.stdout:1192: done (3 left) 2024-04-26T00:27:20.941 INFO:tasks.rados.rados.0.smithi045.stdout:1193: done (2 left) 2024-04-26T00:27:20.941 INFO:tasks.rados.rados.0.smithi045.stdout:1194: done (1 left) 2024-04-26T00:27:20.941 INFO:tasks.rados.rados.0.smithi045.stdout:1195: done (0 left) 2024-04-26T00:27:20.941 INFO:tasks.rados.rados.0.smithi045.stdout:1196: read oid 26 snap 137 2024-04-26T00:27:20.941 INFO:tasks.rados.rados.0.smithi045.stdout:1196: expect (ObjNum 421 snap 131 seq_num 421) 2024-04-26T00:27:20.942 INFO:tasks.rados.rados.0.smithi045.stdout:1197: write (excl) oid 33 current snap is 139 2024-04-26T00:27:20.942 INFO:tasks.rados.rados.0.smithi045.stdout:1197: seq_num 445 ranges {679115=634279,2090487=463822,3003820=475295} 2024-04-26T00:27:20.963 INFO:tasks.rados.rados.0.smithi045.stdout:1197: writing smithi04526579-33 from 679115 to 1313394 tid 1 2024-04-26T00:27:20.984 INFO:tasks.rados.rados.0.smithi045.stdout:1197: writing smithi04526579-33 from 2090487 to 2554309 tid 2 2024-04-26T00:27:20.998 INFO:tasks.rados.rados.0.smithi045.stdout:1197: writing smithi04526579-33 from 3003820 to 3479115 tid 3 2024-04-26T00:27:20.998 INFO:tasks.rados.rados.0.smithi045.stdout:1198: read oid 28 snap -1 2024-04-26T00:27:20.998 INFO:tasks.rados.rados.0.smithi045.stdout:1198: expect (ObjNum 436 snap 138 seq_num 436) 2024-04-26T00:27:20.998 INFO:tasks.rados.rados.0.smithi045.stdout:1199: write (excl) oid 31 current snap is 139 2024-04-26T00:27:20.998 INFO:tasks.rados.rados.0.smithi045.stdout:1199: seq_num 446 ranges {695922=709528,2032616=607480,3198537=297385} 2024-04-26T00:27:21.015 INFO:tasks.rados.rados.0.smithi045.stdout:1199: writing smithi04526579-31 from 695922 to 1405450 tid 1 2024-04-26T00:27:21.026 INFO:tasks.rados.rados.0.smithi045.stdout:1199: writing smithi04526579-31 from 2032616 to 2640096 tid 2 2024-04-26T00:27:21.033 INFO:tasks.rados.rados.0.smithi045.stdout:1199: writing smithi04526579-31 from 3198537 to 3495922 tid 3 2024-04-26T00:27:21.033 INFO:tasks.rados.rados.0.smithi045.stdout:1200: write oid 30 current snap is 139 2024-04-26T00:27:21.034 INFO:tasks.rados.rados.0.smithi045.stdout:1200: seq_num 447 ranges {712729=784777,2091098=467491,3342313=170416} 2024-04-26T00:27:21.045 INFO:tasks.rados.rados.0.smithi045.stdout:1200: writing smithi04526579-30 from 712729 to 1497506 tid 1 2024-04-26T00:27:21.052 INFO:tasks.rados.rados.0.smithi045.stdout:1200: writing smithi04526579-30 from 2091098 to 2558589 tid 2 2024-04-26T00:27:21.059 INFO:tasks.rados.rados.0.smithi045.stdout:1200: writing smithi04526579-30 from 3342313 to 3512729 tid 3 2024-04-26T00:27:21.059 INFO:tasks.rados.rados.0.smithi045.stdout:1201: delete oid 23 current snap is 139 2024-04-26T00:27:21.059 INFO:tasks.rados.rados.0.smithi045.stdout:1196: expect (ObjNum 421 snap 131 seq_num 421) 2024-04-26T00:27:21.321 INFO:tasks.rados.rados.0.smithi045.stdout:1197: finishing write tid 1 to smithi04526579-33 2024-04-26T00:27:21.321 INFO:tasks.rados.rados.0.smithi045.stdout:1197: finishing write tid 2 to smithi04526579-33 2024-04-26T00:27:21.321 INFO:tasks.rados.rados.0.smithi045.stdout:1197: finishing write tid 3 to smithi04526579-33 2024-04-26T00:27:21.321 INFO:tasks.rados.rados.0.smithi045.stdout:1197: finishing write tid 4 to smithi04526579-33 2024-04-26T00:27:21.321 INFO:tasks.rados.rados.0.smithi045.stdout:1197: finishing write tid 5 to smithi04526579-33 2024-04-26T00:27:21.321 INFO:tasks.rados.rados.0.smithi045.stdout:1197: oid 33 updating version 0 to 926 2024-04-26T00:27:21.322 INFO:tasks.rados.rados.0.smithi045.stdout:1197: oid 33 updating version 926 to 929 2024-04-26T00:27:21.322 INFO:tasks.rados.rados.0.smithi045.stdout:1197: oid 33 version 929 is already newer than 927 2024-04-26T00:27:21.322 INFO:tasks.rados.rados.0.smithi045.stdout:1197: oid 33 version 929 is already newer than 928 2024-04-26T00:27:21.322 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 929 (ObjNum 445 snap 139 seq_num 445) dirty exists 2024-04-26T00:27:21.322 INFO:tasks.rados.rados.0.smithi045.stdout:1197: left oid 33 (ObjNum 445 snap 139 seq_num 445) 2024-04-26T00:27:21.322 INFO:tasks.rados.rados.0.smithi045.stdout:1198: expect (ObjNum 436 snap 138 seq_num 436) 2024-04-26T00:27:21.461 INFO:tasks.rados.rados.0.smithi045.stdout:1199: finishing write tid 1 to smithi04526579-31 2024-04-26T00:27:21.461 INFO:tasks.rados.rados.0.smithi045.stdout:1199: finishing write tid 2 to smithi04526579-31 2024-04-26T00:27:21.461 INFO:tasks.rados.rados.0.smithi045.stdout:1199: finishing write tid 3 to smithi04526579-31 2024-04-26T00:27:21.461 INFO:tasks.rados.rados.0.smithi045.stdout:1199: finishing write tid 4 to smithi04526579-31 2024-04-26T00:27:21.461 INFO:tasks.rados.rados.0.smithi045.stdout:1199: finishing write tid 5 to smithi04526579-31 2024-04-26T00:27:21.461 INFO:tasks.rados.rados.0.smithi045.stdout:1199: oid 31 updating version 0 to 637 2024-04-26T00:27:21.461 INFO:tasks.rados.rados.0.smithi045.stdout:1199: oid 31 updating version 637 to 639 2024-04-26T00:27:21.462 INFO:tasks.rados.rados.0.smithi045.stdout:1199: oid 31 version 639 is already newer than 638 2024-04-26T00:27:21.462 INFO:tasks.rados.rados.0.smithi045.stdout:1199: oid 31 version 639 is already newer than 636 2024-04-26T00:27:21.462 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 639 (ObjNum 446 snap 139 seq_num 446) dirty exists 2024-04-26T00:27:21.462 INFO:tasks.rados.rados.0.smithi045.stdout:1199: left oid 31 (ObjNum 446 snap 139 seq_num 446) 2024-04-26T00:27:21.462 INFO:tasks.rados.rados.0.smithi045.stdout:1200: finishing write tid 1 to smithi04526579-30 2024-04-26T00:27:21.462 INFO:tasks.rados.rados.0.smithi045.stdout:1200: finishing write tid 2 to smithi04526579-30 2024-04-26T00:27:21.462 INFO:tasks.rados.rados.0.smithi045.stdout:1200: finishing write tid 3 to smithi04526579-30 2024-04-26T00:27:21.462 INFO:tasks.rados.rados.0.smithi045.stdout:1200: finishing write tid 4 to smithi04526579-30 2024-04-26T00:27:21.462 INFO:tasks.rados.rados.0.smithi045.stdout:1200: finishing write tid 5 to smithi04526579-30 2024-04-26T00:27:21.462 INFO:tasks.rados.rados.0.smithi045.stdout:1200: oid 30 updating version 0 to 499 2024-04-26T00:27:21.462 INFO:tasks.rados.rados.0.smithi045.stdout:1200: oid 30 version 499 is already newer than 498 2024-04-26T00:27:21.462 INFO:tasks.rados.rados.0.smithi045.stdout:1200: oid 30 updating version 499 to 500 2024-04-26T00:27:21.462 INFO:tasks.rados.rados.0.smithi045.stdout:1200: oid 30 version 500 is already newer than 497 2024-04-26T00:27:21.463 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 500 (ObjNum 447 snap 139 seq_num 447) dirty exists 2024-04-26T00:27:21.463 INFO:tasks.rados.rados.0.smithi045.stdout:1200: left oid 30 (ObjNum 447 snap 139 seq_num 447) 2024-04-26T00:27:21.463 INFO:tasks.rados.rados.0.smithi045.stdout:1196: done (5 left) 2024-04-26T00:27:21.463 INFO:tasks.rados.rados.0.smithi045.stdout:1197: done (4 left) 2024-04-26T00:27:21.463 INFO:tasks.rados.rados.0.smithi045.stdout:1198: done (3 left) 2024-04-26T00:27:21.463 INFO:tasks.rados.rados.0.smithi045.stdout:1199: done (2 left) 2024-04-26T00:27:21.463 INFO:tasks.rados.rados.0.smithi045.stdout:1200: done (1 left) 2024-04-26T00:27:21.463 INFO:tasks.rados.rados.0.smithi045.stdout:1201: done (0 left) 2024-04-26T00:27:21.463 INFO:tasks.rados.rados.0.smithi045.stdout:1202: write oid 26 current snap is 139 2024-04-26T00:27:21.463 INFO:tasks.rados.rados.0.smithi045.stdout:1202: seq_num 449 ranges {746343=651628,1924415=754807,3397159=149184} 2024-04-26T00:27:21.472 INFO:tasks.rados.rados.0.smithi045.stdout:1202: writing smithi04526579-26 from 746343 to 1397971 tid 1 2024-04-26T00:27:21.483 INFO:tasks.rados.rados.0.smithi045.stdout:1202: writing smithi04526579-26 from 1924415 to 2679222 tid 2 2024-04-26T00:27:21.490 INFO:tasks.rados.rados.0.smithi045.stdout:1202: writing smithi04526579-26 from 3397159 to 3546343 tid 3 2024-04-26T00:27:21.490 INFO:tasks.rados.rados.0.smithi045.stdout:1203: rollback oid 35 current snap is 139 2024-04-26T00:27:21.490 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 35 to 128 2024-04-26T00:27:21.490 INFO:tasks.rados.rados.0.smithi045.stdout:1202: finishing write tid 1 to smithi04526579-26 2024-04-26T00:27:21.491 INFO:tasks.rados.rados.0.smithi045.stdout:1204: snap_create 2024-04-26T00:27:21.493 INFO:tasks.rados.rados.0.smithi045.stdout:1202: finishing write tid 2 to smithi04526579-26 2024-04-26T00:27:21.494 INFO:tasks.rados.rados.0.smithi045.stdout:1202: finishing write tid 3 to smithi04526579-26 2024-04-26T00:27:21.495 INFO:tasks.rados.rados.0.smithi045.stdout:1202: finishing write tid 4 to smithi04526579-26 2024-04-26T00:27:21.495 INFO:tasks.rados.rados.0.smithi045.stdout:1202: finishing write tid 5 to smithi04526579-26 2024-04-26T00:27:21.495 INFO:tasks.rados.rados.0.smithi045.stdout:1202: oid 26 updating version 0 to 781 2024-04-26T00:27:21.496 INFO:tasks.rados.rados.0.smithi045.stdout:1202: oid 26 updating version 781 to 782 2024-04-26T00:27:21.496 INFO:tasks.rados.rados.0.smithi045.stdout:1202: oid 26 updating version 782 to 783 2024-04-26T00:27:21.496 INFO:tasks.rados.rados.0.smithi045.stdout:1202: oid 26 version 783 is already newer than 780 2024-04-26T00:27:21.496 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 783 (ObjNum 449 snap 139 seq_num 449) dirty exists 2024-04-26T00:27:21.496 INFO:tasks.rados.rados.0.smithi045.stdout:1202: left oid 26 (ObjNum 449 snap 139 seq_num 449) 2024-04-26T00:27:21.502 INFO:tasks.rados.rados.0.smithi045.stdout:1203: finishing rollback tid 0 to smithi04526579-35 2024-04-26T00:27:21.502 INFO:tasks.rados.rados.0.smithi045.stdout:1203: finishing rollback tid 1 to smithi04526579-35 2024-04-26T00:27:21.502 INFO:tasks.rados.rados.0.smithi045.stdout:1203: finishing rollback tid 2 to smithi04526579-35 2024-04-26T00:27:21.502 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 682 (ObjNum 369 snap 112 seq_num 369) dirty exists 2024-04-26T00:27:21.902 INFO:tasks.rados.rados.0.smithi045.stdout:1202: done (2 left) 2024-04-26T00:27:21.902 INFO:tasks.rados.rados.0.smithi045.stdout:1203: done (1 left) 2024-04-26T00:27:21.902 INFO:tasks.rados.rados.0.smithi045.stdout:1204: done (0 left) 2024-04-26T00:27:21.902 INFO:tasks.rados.rados.0.smithi045.stdout:1205: delete oid 28 current snap is 140 2024-04-26T00:27:21.911 INFO:tasks.rados.rados.0.smithi045.stdout:1205: done (0 left) 2024-04-26T00:27:21.911 INFO:tasks.rados.rados.0.smithi045.stdout:1206: write (excl) oid 10 current snap is 140 2024-04-26T00:27:21.911 INFO:tasks.rados.rados.0.smithi045.stdout:1206: seq_num 451 ranges {779957=402126,1925026=758476,3335652=244305} 2024-04-26T00:27:21.929 INFO:tasks.rados.rados.0.smithi045.stdout:1206: writing smithi04526579-10 from 779957 to 1182083 tid 1 2024-04-26T00:27:21.952 INFO:tasks.rados.rados.0.smithi045.stdout:1206: writing smithi04526579-10 from 1925026 to 2683502 tid 2 2024-04-26T00:27:21.965 INFO:tasks.rados.rados.0.smithi045.stdout:1206: writing smithi04526579-10 from 3335652 to 3579957 tid 3 2024-04-26T00:27:21.965 INFO:tasks.rados.rados.0.smithi045.stdout:1207: snap_create 2024-04-26T00:27:21.965 INFO:tasks.rados.rados.0.smithi045.stdout:1206: finishing write tid 1 to smithi04526579-10 2024-04-26T00:27:21.965 INFO:tasks.rados.rados.0.smithi045.stdout:1206: finishing write tid 2 to smithi04526579-10 2024-04-26T00:27:21.970 INFO:tasks.rados.rados.0.smithi045.stdout:1206: finishing write tid 3 to smithi04526579-10 2024-04-26T00:27:21.970 INFO:tasks.rados.rados.0.smithi045.stdout:1206: finishing write tid 4 to smithi04526579-10 2024-04-26T00:27:21.970 INFO:tasks.rados.rados.0.smithi045.stdout:1206: finishing write tid 5 to smithi04526579-10 2024-04-26T00:27:21.970 INFO:tasks.rados.rados.0.smithi045.stdout:1206: oid 10 updating version 0 to 835 2024-04-26T00:27:21.970 INFO:tasks.rados.rados.0.smithi045.stdout:1206: oid 10 updating version 835 to 836 2024-04-26T00:27:21.971 INFO:tasks.rados.rados.0.smithi045.stdout:1206: oid 10 version 836 is already newer than 834 2024-04-26T00:27:21.971 INFO:tasks.rados.rados.0.smithi045.stdout:1206: oid 10 updating version 836 to 837 2024-04-26T00:27:21.971 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 837 (ObjNum 451 snap 140 seq_num 451) dirty exists 2024-04-26T00:27:21.971 INFO:tasks.rados.rados.0.smithi045.stdout:1206: left oid 10 (ObjNum 451 snap 140 seq_num 451) 2024-04-26T00:27:22.538 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:27:22.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.683+0000 7f771cc3f640 10 monclient: get_monmap_and_config 2024-04-26T00:27:22.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.683+0000 7f771cc3f640 10 monclient: build_initial_monmap 2024-04-26T00:27:22.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.683+0000 7f771cc3f640 10 monclient: monmap: 2024-04-26T00:27:22.672 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:27:22.672 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:22.672 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:27:22.689552+0000 2024-04-26T00:27:22.672 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:27:22.689552+0000 2024-04-26T00:27:22.672 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:27:22.672 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:22.672 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:27:22.672 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:27:22.672 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:27:22.672 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:27:22.672 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:27:22.672 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:22.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.683+0000 7f771cc3f640 1 Processor -- start 2024-04-26T00:27:22.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.683+0000 7f771cc3f640 1 -- start start 2024-04-26T00:27:22.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.683+0000 7f771cc3f640 10 monclient: init 2024-04-26T00:27:22.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.683+0000 7f771cc3f640 10 monclient: _reopen_session rank -1 2024-04-26T00:27:22.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.683+0000 7f771cc3f640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-04-26T00:27:22.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.683+0000 7f771cc3f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7718136da0 0x7f771809bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:22.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.683+0000 7f771cc3f640 10 monclient(hunting): picked mon.noname-b con 0x7f7718136da0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:27:22.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.683+0000 7f771cc3f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f771809c410 addr v1:172.21.15.45:6789/0 2024-04-26T00:27:22.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.683+0000 7f771cc3f640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7718137890 0x7f771813e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:22.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.683+0000 7f771cc3f640 10 monclient(hunting): picked mon.noname-c con 0x7f7718137890 addr v2:172.21.15.45:3300/0 2024-04-26T00:27:22.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.683+0000 7f771cc3f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f771809b4f0 con 0x7f771809c410 2024-04-26T00:27:22.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.683+0000 7f771cc3f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:22.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.683+0000 7f771cc3f640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f771809bb40 con 0x7f7718137890 2024-04-26T00:27:22.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.683+0000 7f771cc3f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:22.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.683+0000 7f771cc3f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f771809b210 con 0x7f7718136da0 2024-04-26T00:27:22.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.683+0000 7f771cc3f640 10 monclient(hunting): _renew_subs 2024-04-26T00:27:22.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.683+0000 7f771cc3f640 10 monclient(hunting): authenticate will time out at 1423.994888s 2024-04-26T00:27:22.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.683+0000 7f771659c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7718136da0 0x7f771809bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:22.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.683+0000 7f771659c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7718136da0 0x7f771809bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:34336/0 (socket says 172.21.15.37:34336) 2024-04-26T00:27:22.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.683+0000 7f771659c640 1 -- 172.21.15.37:0/2853526744 learned_addr learned my addr 172.21.15.37:0/2853526744 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:27:22.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.683+0000 7f771659c640 10 monclient(hunting): get_auth_request con 0x7f7718136da0 auth_method 0 2024-04-26T00:27:22.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.683+0000 7f771659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:22.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.683+0000 7f771659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:22.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.683+0000 7f771659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:22.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.683+0000 7f771659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:22.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.683+0000 7f771659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:22.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.683+0000 7f771659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:22.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.683+0000 7f7716d9d640 1 --2- 172.21.15.37:0/2853526744 >> v2:172.21.15.45:3300/0 conn(0x7f7718137890 0x7f771813e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:22.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.683+0000 7f771659c640 10 monclient(hunting): handle_auth_done global_id 84173 payload 274 2024-04-26T00:27:22.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.683+0000 7f771659c640 1 -- 172.21.15.37:0/2853526744 >> v2:172.21.15.45:3300/0 conn(0x7f7718137890 msgr2=0x7f771813e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:27:22.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.683+0000 7f771659c640 1 --2- 172.21.15.37:0/2853526744 >> v2:172.21.15.45:3300/0 conn(0x7f7718137890 0x7f771813e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:22.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.683+0000 7f771659c640 1 -- 172.21.15.37:0/2853526744 >> v1:172.21.15.45:6789/0 conn(0x7f771809c410 legacy=0x7f7718137180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:27:22.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.683+0000 7f771659c640 10 monclient: _finish_hunting 0 2024-04-26T00:27:22.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.683+0000 7f771659c640 1 monclient: found mon.noname-b 2024-04-26T00:27:22.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.683+0000 7f771659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:27:22.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.683+0000 7f771659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:27:22.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.683+0000 7f771659c640 1 -- 172.21.15.37:0/2853526744 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f771809b7d0 con 0x7f7718136da0 2024-04-26T00:27:22.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.687+0000 7f771659c640 1 -- 172.21.15.37:0/2853526744 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7718136da0 msgr2=0x7f771809bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:22.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.687+0000 7f771659c640 1 --2- 172.21.15.37:0/2853526744 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7718136da0 0x7f771809bed0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7708009870 tx=0x7f770801c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b061beaf00eb6e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:22.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.687+0000 7f771559a640 1 -- 172.21.15.37:0/2853526744 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77080b1070 con 0x7f7718136da0 2024-04-26T00:27:22.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.687+0000 7f771559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:22.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.687+0000 7f771559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:27:22.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.687+0000 7f771559a640 10 monclient: dump: 2024-04-26T00:27:22.679 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:22.679 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:22.679 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:22.679 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:22.680 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:22.680 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:22.680 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:22.680 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:22.680 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:22.680 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:22.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.687+0000 7f771559a640 10 monclient: _finish_auth 0 2024-04-26T00:27:22.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.687+0000 7f771559a640 10 monclient: _check_auth_tickets 2024-04-26T00:27:22.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.687+0000 7f771559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:27:22.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.687+0000 7f771559a640 1 -- 172.21.15.37:0/2853526744 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77080ac620 con 0x7f7718136da0 2024-04-26T00:27:22.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.687+0000 7f771cc3f640 5 monclient: authenticate success, global_id 84173 2024-04-26T00:27:22.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.687+0000 7f771cc3f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:27:22.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.687+0000 7f771559a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:27:22.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.687+0000 7f771cc3f640 10 monclient: get_monmap_and_config success 2024-04-26T00:27:22.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.687+0000 7f771559a640 1 -- 172.21.15.37:0/2853526744 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77080ac9f0 con 0x7f7718136da0 2024-04-26T00:27:22.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.687+0000 7f771cc3f640 4 set_mon_vals no callback set 2024-04-26T00:27:22.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.687+0000 7f771559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:22.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.687+0000 7f771559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:27:22.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.687+0000 7f771559a640 10 monclient: dump: 2024-04-26T00:27:22.681 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:22.681 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:22.681 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:22.681 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:22.681 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:22.681 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:22.681 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:22.681 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:22.682 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:22.682 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:22.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f771cc3f640 10 monclient: shutdown 2024-04-26T00:27:22.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f771cc3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:27:22.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f771cc3f640 1 -- 172.21.15.37:0/2853526744 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7718136da0 msgr2=0x7f771809bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:22.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f771cc3f640 1 --2- 172.21.15.37:0/2853526744 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7718136da0 0x7f771809bed0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7708009870 tx=0x7f770801c3e0 comp rx=0 tx=0).stop 2024-04-26T00:27:22.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f771cc3f640 1 -- 172.21.15.37:0/2853526744 shutdown_connections 2024-04-26T00:27:22.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f771cc3f640 1 --2- 172.21.15.37:0/2853526744 >> v2:172.21.15.45:3300/0 conn(0x7f7718137890 0x7f771813e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:22.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f771cc3f640 1 --2- 172.21.15.37:0/2853526744 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7718136da0 0x7f771809bed0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:22.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f771cc3f640 1 -- 172.21.15.37:0/2853526744 >> 172.21.15.37:0/2853526744 conn(0x7f77180a5f00 msgr2=0x7f77180a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:27:22.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f771cc3f640 1 -- 172.21.15.37:0/2853526744 shutdown_connections 2024-04-26T00:27:22.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f771cc3f640 1 -- 172.21.15.37:0/2853526744 wait complete. 2024-04-26T00:27:22.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f771cc3f640 10 monclient: build_initial_monmap 2024-04-26T00:27:22.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f771cc3f640 10 monclient: monmap: 2024-04-26T00:27:22.683 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:27:22.683 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:27:22.683 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:27:22.683 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:27:22.683 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:27:22.683 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:27:22.683 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:27:22.683 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:27:22.683 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:27:22.683 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:22.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f771cc3f640 1 Processor -- start 2024-04-26T00:27:22.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f771cc3f640 1 -- start start 2024-04-26T00:27:22.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f771cc3f640 10 monclient: init 2024-04-26T00:27:22.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f771cc3f640 10 monclient: _reopen_session rank -1 2024-04-26T00:27:22.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f771cc3f640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:27:22.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f771cc3f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f771809c410 0x7f77181cee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:22.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f771cc3f640 10 monclient(hunting): picked mon.noname-a con 0x7f771809c410 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:27:22.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f771cc3f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7718136da0 0x7f771813d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:22.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f771cc3f640 10 monclient(hunting): picked mon.noname-c con 0x7f7718136da0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:27:22.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f771cc3f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7718137890 0x7f771813d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:22.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f771cc3f640 10 monclient(hunting): picked mon.noname-b con 0x7f7718137890 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:27:22.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f771cc3f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:22.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f771cc3f640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f7718141430 con 0x7f771809c410 2024-04-26T00:27:22.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f771cc3f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:22.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f771cc3f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f77181412b0 con 0x7f7718137890 2024-04-26T00:27:22.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f771cc3f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:22.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f771cc3f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f77181415b0 con 0x7f7718136da0 2024-04-26T00:27:22.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f771cc3f640 10 monclient(hunting): _renew_subs 2024-04-26T00:27:22.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f771cc3f640 10 monclient(hunting): authenticate will time out at 1424.002058s 2024-04-26T00:27:22.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f7715d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7718136da0 0x7f771813d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:22.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f7715d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7718136da0 0x7f771813d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:34340/0 (socket says 172.21.15.37:34340) 2024-04-26T00:27:22.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f7715d9b640 1 -- 172.21.15.37:0/1128391283 learned_addr learned my addr 172.21.15.37:0/1128391283 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:27:22.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f7715d9b640 10 monclient(hunting): get_auth_request con 0x7f7718136da0 auth_method 0 2024-04-26T00:27:22.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f7715d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:22.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f7715d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:22.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f7715d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:22.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f7715d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:22.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f7715d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:22.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f771659c640 1 --2- 172.21.15.37:0/1128391283 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f771809c410 0x7f77181cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:22.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f7715d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:22.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f771659c640 10 monclient(hunting): get_auth_request con 0x7f771809c410 auth_method 0 2024-04-26T00:27:22.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f771659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:22.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f771659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:22.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f771659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:22.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f771659c640 1 -- 172.21.15.37:0/1128391283 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f771809c410 msgr2=0x7f77181cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:27:22.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f771659c640 1 -- 172.21.15.37:0/1128391283 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f771809c410 msgr2=0x7f77181cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:27:22.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f7716d9d640 1 --2- 172.21.15.37:0/1128391283 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7718137890 0x7f771813d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:22.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f771659c640 1 --2- 172.21.15.37:0/1128391283 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f771809c410 0x7f77181cee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:27:22.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f771659c640 1 --2- 172.21.15.37:0/1128391283 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f771809c410 0x7f77181cee80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:27:22.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f7715d9b640 10 monclient(hunting): handle_auth_done global_id 84176 payload 824 2024-04-26T00:27:22.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f7715d9b640 1 -- 172.21.15.37:0/1128391283 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7718137890 msgr2=0x7f771813d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:22.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f7715d9b640 1 --2- 172.21.15.37:0/1128391283 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7718137890 0x7f771813d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:22.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f7715d9b640 1 -- 172.21.15.37:0/1128391283 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f771809c410 msgr2=0x7f77181cee80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:27:22.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f7715d9b640 1 --2- 172.21.15.37:0/1128391283 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f771809c410 0x7f77181cee80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:22.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f7715d9b640 10 monclient: _finish_hunting 0 2024-04-26T00:27:22.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f7715d9b640 1 monclient: found mon.noname-c 2024-04-26T00:27:22.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f7715d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:27:22.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f7715d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:27:22.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f7715d9b640 1 -- 172.21.15.37:0/1128391283 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77180a2c60 con 0x7f7718136da0 2024-04-26T00:27:22.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f7716d9d640 10 monclient: get_auth_request con 0x7f7718137890 auth_method 0 2024-04-26T00:27:22.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.691+0000 7f7716d9d640 1 --2- 172.21.15.37:0/1128391283 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7718137890 0x7f771813d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:27:22.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.695+0000 7f7715d9b640 1 -- 172.21.15.37:0/1128391283 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7718136da0 msgr2=0x7f771813d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:22.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.695+0000 7f7715d9b640 1 --2- 172.21.15.37:0/1128391283 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7718136da0 0x7f771813d130 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f770000c7d0 tx=0x7f770000cc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:22.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.695+0000 7f770effd640 1 -- 172.21.15.37:0/1128391283 DispatchQueue::entry inject delay of 0.002000 2024-04-26T00:27:22.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.695+0000 7f770effd640 1 -- 172.21.15.37:0/1128391283 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7700017070 con 0x7f7718136da0 2024-04-26T00:27:22.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.695+0000 7f770effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:22.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.695+0000 7f770effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:27:22.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.695+0000 7f770effd640 10 monclient: dump: 2024-04-26T00:27:22.691 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:22.691 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:22.691 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:22.691 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:22.692 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:22.692 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:22.692 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:22.692 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:22.692 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:22.692 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:22.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.695+0000 7f770effd640 10 monclient: _finish_auth 0 2024-04-26T00:27:22.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.695+0000 7f770effd640 10 monclient: _check_auth_tickets 2024-04-26T00:27:22.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.695+0000 7f770effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:27:22.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.695+0000 7f770effd640 1 -- 172.21.15.37:0/1128391283 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7700005070 con 0x7f7718136da0 2024-04-26T00:27:22.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.695+0000 7f771cc3f640 5 monclient: authenticate success, global_id 84176 2024-04-26T00:27:22.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.695+0000 7f771cc3f640 10 monclient: _renew_subs 2024-04-26T00:27:22.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.695+0000 7f771cc3f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:27:22.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.695+0000 7f771cc3f640 1 -- 172.21.15.37:0/1128391283 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f771813dbb0 con 0x7f7718136da0 2024-04-26T00:27:22.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.695+0000 7f770effd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:27:22.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.695+0000 7f770effd640 1 -- 172.21.15.37:0/1128391283 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7700005460 con 0x7f7718136da0 2024-04-26T00:27:22.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.695+0000 7f770effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:22.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.695+0000 7f770effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:27:22.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.695+0000 7f770effd640 10 monclient: dump: 2024-04-26T00:27:22.693 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:22.693 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:22.693 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:22.693 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:22.693 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:22.693 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:22.693 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:22.693 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:22.693 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:22.693 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:22.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.695+0000 7f771cc3f640 10 monclient: _renew_subs 2024-04-26T00:27:22.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.695+0000 7f771cc3f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:27:22.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.695+0000 7f771cc3f640 1 -- 172.21.15.37:0/1128391283 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f771813a470 con 0x7f7718136da0 2024-04-26T00:27:22.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.695+0000 7f7714d99640 4 set_mon_vals no callback set 2024-04-26T00:27:22.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.695+0000 7f771cc3f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:22.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.695+0000 7f771cc3f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:22.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.695+0000 7f771cc3f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:27:22.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.695+0000 7f771cc3f640 1 -- 172.21.15.37:0/1128391283 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f771813a1f0 con 0x7f7718136da0 2024-04-26T00:27:22.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.699+0000 7f770effd640 1 -- 172.21.15.37:0/1128391283 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f770001a070 con 0x7f7718136da0 2024-04-26T00:27:22.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.699+0000 7f770effd640 1 --2- 172.21.15.37:0/1128391283 >> v2:172.21.15.45:6812/2379918623 conn(0x7f76f8035ed0 0x7f76f8038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:22.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.699+0000 7f770effd640 1 -- 172.21.15.37:0/1128391283 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(288..288 src has 177..288) ==== 4007+0+0 (secure 0 0 0) 0x7f7700048270 con 0x7f7718136da0 2024-04-26T00:27:22.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.699+0000 7f771659c640 1 --2- 172.21.15.37:0/1128391283 >> v2:172.21.15.45:6812/2379918623 conn(0x7f76f8035ed0 0x7f76f8038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:22.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.699+0000 7f771659c640 10 monclient: get_auth_request con 0x7f76f8035ed0 auth_method 0 2024-04-26T00:27:22.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.699+0000 7f771659c640 1 --2- 172.21.15.37:0/1128391283 >> v2:172.21.15.45:6812/2379918623 conn(0x7f76f8035ed0 0x7f76f8038390 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f771806b910 tx=0x7f77080a7000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:22.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.703+0000 7f770effd640 1 -- 172.21.15.37:0/1128391283 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f771813dbb0 con 0x7f7718136da0 2024-04-26T00:27:22.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.703+0000 7f770effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:22.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.703+0000 7f770effd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:27:22.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.923+0000 7f771cc3f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:27:22.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.923+0000 7f771cc3f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:27:22.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.923+0000 7f771cc3f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:27:22.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.923+0000 7f771cc3f640 1 -- 172.21.15.37:0/1128391283 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f771809aca0 con 0x7f7718136da0 2024-04-26T00:27:22.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.923+0000 7f770effd640 1 -- 172.21.15.37:0/1128391283 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f770000a640 con 0x7f7718136da0 2024-04-26T00:27:22.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.923+0000 7f770effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:27:22.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.923+0000 7f770effd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:27:22.913 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":56,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:27:22.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.927+0000 7f771cc3f640 1 -- 172.21.15.37:0/1128391283 >> v2:172.21.15.45:6812/2379918623 conn(0x7f76f8035ed0 msgr2=0x7f76f8038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:22.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.927+0000 7f771cc3f640 1 --2- 172.21.15.37:0/1128391283 >> v2:172.21.15.45:6812/2379918623 conn(0x7f76f8035ed0 0x7f76f8038390 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f771806b910 tx=0x7f77080a7000 comp rx=0 tx=0).stop 2024-04-26T00:27:22.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.927+0000 7f771cc3f640 10 monclient: shutdown 2024-04-26T00:27:22.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.927+0000 7f771cc3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:27:22.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.927+0000 7f771cc3f640 1 -- 172.21.15.37:0/1128391283 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7718136da0 msgr2=0x7f771813d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:22.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.927+0000 7f771cc3f640 1 --2- 172.21.15.37:0/1128391283 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7718136da0 0x7f771813d130 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f770000c7d0 tx=0x7f770000cc90 comp rx=0 tx=0).stop 2024-04-26T00:27:22.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.927+0000 7f771cc3f640 1 -- 172.21.15.37:0/1128391283 shutdown_connections 2024-04-26T00:27:22.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.927+0000 7f771cc3f640 1 --2- 172.21.15.37:0/1128391283 >> v2:172.21.15.45:6812/2379918623 conn(0x7f76f8035ed0 0x7f76f8038390 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:22.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.927+0000 7f771cc3f640 1 --2- 172.21.15.37:0/1128391283 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7718137890 0x7f771813d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:22.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.927+0000 7f771cc3f640 1 --2- 172.21.15.37:0/1128391283 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7718136da0 0x7f771813d130 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:22.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.927+0000 7f771cc3f640 1 --2- 172.21.15.37:0/1128391283 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f771809c410 0x7f77181cee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:22.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.927+0000 7f771cc3f640 1 -- 172.21.15.37:0/1128391283 >> 172.21.15.37:0/1128391283 conn(0x7f77180a5f00 msgr2=0x7f7718135db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:27:22.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.927+0000 7f771cc3f640 1 -- 172.21.15.37:0/1128391283 shutdown_connections 2024-04-26T00:27:22.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:22.927+0000 7f771cc3f640 1 -- 172.21.15.37:0/1128391283 wait complete. 2024-04-26T00:27:22.959 INFO:tasks.rados.rados.0.smithi045.stdout:1206: done (1 left) 2024-04-26T00:27:22.959 INFO:tasks.rados.rados.0.smithi045.stdout:1207: done (0 left) 2024-04-26T00:27:22.959 INFO:tasks.rados.rados.0.smithi045.stdout:1208: delete oid 26 current snap is 141 2024-04-26T00:27:22.973 INFO:tasks.rados.rados.0.smithi045.stdout:1208: done (0 left) 2024-04-26T00:27:22.973 INFO:tasks.rados.rados.0.smithi045.stdout:1209: write oid 23 current snap is 141 2024-04-26T00:27:22.973 INFO:tasks.rados.rados.0.smithi045.stdout:1209: seq_num 453 ranges {413571=552624,1641990=762145,2990498=425163} 2024-04-26T00:27:22.989 INFO:tasks.rados.rados.0.smithi045.stdout:1209: writing smithi04526579-23 from 413571 to 966195 tid 1 2024-04-26T00:27:23.013 INFO:tasks.rados.rados.0.smithi045.stdout:1209: writing smithi04526579-23 from 1641990 to 2404135 tid 2 2024-04-26T00:27:23.028 INFO:tasks.rados.rados.0.smithi045.stdout:1209: writing smithi04526579-23 from 2990498 to 3415661 tid 3 2024-04-26T00:27:23.028 INFO:tasks.rados.rados.0.smithi045.stdout:1210: write (excl) oid 11 current snap is 141 2024-04-26T00:27:23.028 INFO:tasks.rados.rados.0.smithi045.stdout:1210: seq_num 454 ranges {430378=627873,1700472=622156,3017921=612457} 2024-04-26T00:27:23.040 INFO:tasks.rados.rados.0.smithi045.stdout:1210: writing smithi04526579-11 from 430378 to 1058251 tid 1 2024-04-26T00:27:23.053 INFO:tasks.rados.rados.0.smithi045.stdout:1210: writing smithi04526579-11 from 1700472 to 2322628 tid 2 2024-04-26T00:27:23.065 INFO:tasks.rados.rados.0.smithi045.stdout:1210: writing smithi04526579-11 from 3017921 to 3630378 tid 3 2024-04-26T00:27:23.065 INFO:tasks.rados.rados.0.smithi045.stdout:1211: rollback oid 43 current snap is 141 2024-04-26T00:27:23.065 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 43 to 138 2024-04-26T00:27:23.065 INFO:tasks.rados.rados.0.smithi045.stdout:1209: finishing write tid 1 to smithi04526579-23 2024-04-26T00:27:23.065 INFO:tasks.rados.rados.0.smithi045.stdout:1209: finishing write tid 2 to smithi04526579-23 2024-04-26T00:27:23.065 INFO:tasks.rados.rados.0.smithi045.stdout:1209: finishing write tid 3 to smithi04526579-23 2024-04-26T00:27:23.065 INFO:tasks.rados.rados.0.smithi045.stdout:1209: finishing write tid 4 to smithi04526579-23 2024-04-26T00:27:23.065 INFO:tasks.rados.rados.0.smithi045.stdout:1209: finishing write tid 5 to smithi04526579-23 2024-04-26T00:27:23.066 INFO:tasks.rados.rados.0.smithi045.stdout:1209: oid 23 updating version 0 to 600 2024-04-26T00:27:23.066 INFO:tasks.rados.rados.0.smithi045.stdout:1209: oid 23 updating version 600 to 601 2024-04-26T00:27:23.066 INFO:tasks.rados.rados.0.smithi045.stdout:1209: oid 23 updating version 601 to 602 2024-04-26T00:27:23.066 INFO:tasks.rados.rados.0.smithi045.stdout:1209: oid 23 version 602 is already newer than 599 2024-04-26T00:27:23.066 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 602 (ObjNum 453 snap 141 seq_num 453) dirty exists 2024-04-26T00:27:23.066 INFO:tasks.rados.rados.0.smithi045.stdout:1209: left oid 23 (ObjNum 453 snap 141 seq_num 453) 2024-04-26T00:27:23.066 INFO:tasks.rados.rados.0.smithi045.stdout:1210: finishing write tid 1 to smithi04526579-11 2024-04-26T00:27:23.066 INFO:tasks.rados.rados.0.smithi045.stdout:1209: done (2 left) 2024-04-26T00:27:23.066 INFO:tasks.rados.rados.0.smithi045.stdout:1212: read oid 21 snap -1 2024-04-26T00:27:23.066 INFO:tasks.rados.rados.0.smithi045.stdout:1212: expect (ObjNum 254 snap 64 seq_num 254) 2024-04-26T00:27:23.066 INFO:tasks.rados.rados.0.smithi045.stdout:1213: write oid 28 current snap is 141 2024-04-26T00:27:23.066 INFO:tasks.rados.rados.0.smithi045.stdout:1213: seq_num 455 ranges {447185=703122,1642601=765814,2928991=447707} 2024-04-26T00:27:23.074 INFO:tasks.rados.rados.0.smithi045.stdout:1213: writing smithi04526579-28 from 447185 to 1150307 tid 1 2024-04-26T00:27:23.084 INFO:tasks.rados.rados.0.smithi045.stdout:1213: writing smithi04526579-28 from 1642601 to 2408415 tid 2 2024-04-26T00:27:23.093 INFO:tasks.rados.rados.0.smithi045.stdout:1213: writing smithi04526579-28 from 2928991 to 3376698 tid 3 2024-04-26T00:27:23.093 INFO:tasks.rados.rados.0.smithi045.stdout:1214: rollback oid 16 current snap is 141 2024-04-26T00:27:23.093 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 16 to 125 2024-04-26T00:27:23.093 INFO:tasks.rados.rados.0.smithi045.stdout:1210: finishing write tid 2 to smithi04526579-11 2024-04-26T00:27:23.093 INFO:tasks.rados.rados.0.smithi045.stdout:1210: finishing write tid 3 to smithi04526579-11 2024-04-26T00:27:23.093 INFO:tasks.rados.rados.0.smithi045.stdout:1210: finishing write tid 4 to smithi04526579-11 2024-04-26T00:27:23.093 INFO:tasks.rados.rados.0.smithi045.stdout:1210: finishing write tid 5 to smithi04526579-11 2024-04-26T00:27:23.093 INFO:tasks.rados.rados.0.smithi045.stdout:1210: oid 11 updating version 0 to 600 2024-04-26T00:27:23.094 INFO:tasks.rados.rados.0.smithi045.stdout:1210: oid 11 updating version 600 to 602 2024-04-26T00:27:23.094 INFO:tasks.rados.rados.0.smithi045.stdout:1210: oid 11 updating version 602 to 603 2024-04-26T00:27:23.094 INFO:tasks.rados.rados.0.smithi045.stdout:1210: oid 11 version 603 is already newer than 601 2024-04-26T00:27:23.094 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 603 (ObjNum 454 snap 141 seq_num 454) dirty exists 2024-04-26T00:27:23.094 INFO:tasks.rados.rados.0.smithi045.stdout:1210: left oid 11 (ObjNum 454 snap 141 seq_num 454) 2024-04-26T00:27:23.094 INFO:tasks.rados.rados.0.smithi045.stdout:1211: finishing rollback tid 0 to smithi04526579-43 2024-04-26T00:27:23.094 INFO:tasks.rados.rados.0.smithi045.stdout:1210: done (4 left) 2024-04-26T00:27:23.094 INFO:tasks.rados.rados.0.smithi045.stdout:1215: read oid 48 snap 124 2024-04-26T00:27:23.094 INFO:tasks.rados.rados.0.smithi045.stdout:1215: expect (ObjNum 359 snap 107 seq_num 359) 2024-04-26T00:27:23.094 INFO:tasks.rados.rados.0.smithi045.stdout:1216: delete oid 37 current snap is 141 2024-04-26T00:27:23.094 INFO:tasks.rados.rados.0.smithi045.stdout:1211: finishing rollback tid 1 to smithi04526579-43 2024-04-26T00:27:23.094 INFO:tasks.rados.rados.0.smithi045.stdout:1211: finishing rollback tid 2 to smithi04526579-43 2024-04-26T00:27:23.094 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 840 (ObjNum 329 snap 96 seq_num 329) dirty exists 2024-04-26T00:27:23.094 INFO:tasks.rados.rados.0.smithi045.stdout:1213: finishing write tid 1 to smithi04526579-28 2024-04-26T00:27:23.094 INFO:tasks.rados.rados.0.smithi045.stdout:1212: expect (ObjNum 254 snap 64 seq_num 254) 2024-04-26T00:27:23.314 INFO:tasks.rados.rados.0.smithi045.stdout:1213: finishing write tid 2 to smithi04526579-28 2024-04-26T00:27:23.314 INFO:tasks.rados.rados.0.smithi045.stdout:1213: finishing write tid 3 to smithi04526579-28 2024-04-26T00:27:23.314 INFO:tasks.rados.rados.0.smithi045.stdout:1213: finishing write tid 4 to smithi04526579-28 2024-04-26T00:27:23.314 INFO:tasks.rados.rados.0.smithi045.stdout:1213: finishing write tid 5 to smithi04526579-28 2024-04-26T00:27:23.314 INFO:tasks.rados.rados.0.smithi045.stdout:1213: oid 28 updating version 0 to 789 2024-04-26T00:27:23.314 INFO:tasks.rados.rados.0.smithi045.stdout:1213: oid 28 updating version 789 to 790 2024-04-26T00:27:23.315 INFO:tasks.rados.rados.0.smithi045.stdout:1213: oid 28 updating version 790 to 791 2024-04-26T00:27:23.315 INFO:tasks.rados.rados.0.smithi045.stdout:1213: oid 28 version 791 is already newer than 788 2024-04-26T00:27:23.315 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 791 (ObjNum 455 snap 141 seq_num 455) dirty exists 2024-04-26T00:27:23.315 INFO:tasks.rados.rados.0.smithi045.stdout:1213: left oid 28 (ObjNum 455 snap 141 seq_num 455) 2024-04-26T00:27:23.315 INFO:tasks.rados.rados.0.smithi045.stdout:1215: expect (ObjNum 359 snap 107 seq_num 359) 2024-04-26T00:27:23.396 INFO:tasks.rados.rados.0.smithi045.stdout:1214: finishing rollback tid 0 to smithi04526579-16 2024-04-26T00:27:23.396 INFO:tasks.rados.rados.0.smithi045.stdout:1214: finishing rollback tid 1 to smithi04526579-16 2024-04-26T00:27:23.396 INFO:tasks.rados.rados.0.smithi045.stdout:1214: finishing rollback tid 2 to smithi04526579-16 2024-04-26T00:27:23.397 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 612 (ObjNum 395 snap 120 seq_num 395) dirty exists 2024-04-26T00:27:23.397 INFO:tasks.rados.rados.0.smithi045.stdout:1211: done (5 left) 2024-04-26T00:27:23.397 INFO:tasks.rados.rados.0.smithi045.stdout:1212: done (4 left) 2024-04-26T00:27:23.397 INFO:tasks.rados.rados.0.smithi045.stdout:1213: done (3 left) 2024-04-26T00:27:23.397 INFO:tasks.rados.rados.0.smithi045.stdout:1214: done (2 left) 2024-04-26T00:27:23.397 INFO:tasks.rados.rados.0.smithi045.stdout:1215: done (1 left) 2024-04-26T00:27:23.397 INFO:tasks.rados.rados.0.smithi045.stdout:1216: done (0 left) 2024-04-26T00:27:23.397 INFO:tasks.rados.rados.0.smithi045.stdout:1217: read oid 1 snap 124 2024-04-26T00:27:23.397 INFO:tasks.rados.rados.0.smithi045.stdout:1217: expect (ObjNum 393 snap 120 seq_num 393) 2024-04-26T00:27:23.397 INFO:tasks.rados.rados.0.smithi045.stdout:1218: snap_create 2024-04-26T00:27:23.406 INFO:tasks.rados.rados.0.smithi045.stdout:1217: expect (ObjNum 393 snap 120 seq_num 393) 2024-04-26T00:27:23.969 INFO:tasks.rados.rados.0.smithi045.stdout:1217: done (1 left) 2024-04-26T00:27:23.969 INFO:tasks.rados.rados.0.smithi045.stdout:1218: done (0 left) 2024-04-26T00:27:23.969 INFO:tasks.rados.rados.0.smithi045.stdout:1219: write (excl) oid 6 current snap is 142 2024-04-26T00:27:23.969 INFO:tasks.rados.rados.0.smithi045.stdout:1219: seq_num 457 ranges {480799=569973,1475918=769483,2700190=586604} 2024-04-26T00:27:23.985 INFO:tasks.rados.rados.0.smithi045.stdout:1219: writing smithi04526579-6 from 480799 to 1050772 tid 1 2024-04-26T00:27:24.004 INFO:tasks.rados.rados.0.smithi045.stdout:1219: writing smithi04526579-6 from 1475918 to 2245401 tid 2 2024-04-26T00:27:24.020 INFO:tasks.rados.rados.0.smithi045.stdout:1219: writing smithi04526579-6 from 2700190 to 3286794 tid 3 2024-04-26T00:27:24.020 INFO:tasks.rados.rados.0.smithi045.stdout:1220: copy_from oid 45 from oid 50 current snap is 142 2024-04-26T00:27:24.021 INFO:tasks.rados.rados.0.smithi045.stdout:1219: finishing write tid 1 to smithi04526579-6 2024-04-26T00:27:24.021 INFO:tasks.rados.rados.0.smithi045.stdout:1219: finishing write tid 2 to smithi04526579-6 2024-04-26T00:27:24.021 INFO:tasks.rados.rados.0.smithi045.stdout:1221: snap_remove snap 139 2024-04-26T00:27:24.972 INFO:tasks.rados.rados.0.smithi045.stdout:1221: done (2 left) 2024-04-26T00:27:24.972 INFO:tasks.rados.rados.0.smithi045.stdout:1222: snap_remove snap 69 2024-04-26T00:27:25.932 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:27:25.989 INFO:tasks.rados.rados.0.smithi045.stdout:1222: done (2 left) 2024-04-26T00:27:25.989 INFO:tasks.rados.rados.0.smithi045.stdout:1223: copy_from oid 38 from oid 20 current snap is 142 2024-04-26T00:27:25.989 INFO:tasks.rados.rados.0.smithi045.stdout:1219: finishing write tid 3 to smithi04526579-6 2024-04-26T00:27:25.989 INFO:tasks.rados.rados.0.smithi045.stdout:1219: finishing write tid 4 to smithi04526579-6 2024-04-26T00:27:25.989 INFO:tasks.rados.rados.0.smithi045.stdout:1219: finishing write tid 5 to smithi04526579-6 2024-04-26T00:27:25.989 INFO:tasks.rados.rados.0.smithi045.stdout:1219: oid 6 updating version 0 to 504 2024-04-26T00:27:25.989 INFO:tasks.rados.rados.0.smithi045.stdout:1219: oid 6 version 504 is already newer than 501 2024-04-26T00:27:25.989 INFO:tasks.rados.rados.0.smithi045.stdout:1219: oid 6 version 504 is already newer than 503 2024-04-26T00:27:25.989 INFO:tasks.rados.rados.0.smithi045.stdout:1219: oid 6 version 504 is already newer than 502 2024-04-26T00:27:25.989 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 504 (ObjNum 457 snap 142 seq_num 457) dirty exists 2024-04-26T00:27:25.989 INFO:tasks.rados.rados.0.smithi045.stdout:1219: left oid 6 (ObjNum 457 snap 142 seq_num 457) 2024-04-26T00:27:25.990 INFO:tasks.rados.rados.0.smithi045.stdout:1220: finishing copy_from racing read to smithi04526579-45 2024-04-26T00:27:25.990 INFO:tasks.rados.rados.0.smithi045.stdout:1220: finishing copy_from to smithi04526579-45 2024-04-26T00:27:25.990 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 833 (ObjNum 136 snap 30 seq_num 136) dirty exists 2024-04-26T00:27:25.990 INFO:tasks.rados.rados.0.smithi045.stdout:1219: done (2 left) 2024-04-26T00:27:25.990 INFO:tasks.rados.rados.0.smithi045.stdout:1220: done (1 left) 2024-04-26T00:27:25.990 INFO:tasks.rados.rados.0.smithi045.stdout:1224: snap_create 2024-04-26T00:27:26.028 INFO:tasks.rados.rados.0.smithi045.stdout:1223: finishing copy_from racing read to smithi04526579-38 2024-04-26T00:27:26.036 INFO:tasks.rados.rados.0.smithi045.stdout:1223: finishing copy_from to smithi04526579-38 2024-04-26T00:27:26.036 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 437 (ObjNum 356 snap 105 seq_num 356) dirty exists 2024-04-26T00:27:26.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.095+0000 7f4179e3c640 10 monclient: get_monmap_and_config 2024-04-26T00:27:26.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.095+0000 7f4179e3c640 10 monclient: build_initial_monmap 2024-04-26T00:27:26.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.095+0000 7f4179e3c640 10 monclient: monmap: 2024-04-26T00:27:26.083 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:27:26.083 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:26.083 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:27:26.100919+0000 2024-04-26T00:27:26.083 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:27:26.100919+0000 2024-04-26T00:27:26.083 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:27:26.083 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:26.083 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:27:26.083 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:27:26.084 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:27:26.084 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:27:26.084 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:27:26.084 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:26.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.095+0000 7f4179e3c640 1 Processor -- start 2024-04-26T00:27:26.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.095+0000 7f4179e3c640 1 -- start start 2024-04-26T00:27:26.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.095+0000 7f4179e3c640 10 monclient: init 2024-04-26T00:27:26.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.095+0000 7f4179e3c640 10 monclient: _reopen_session rank -1 2024-04-26T00:27:26.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.095+0000 7f4179e3c640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-04-26T00:27:26.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.095+0000 7f4179e3c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4174136da0 addr v1:172.21.15.37:6789/0 2024-04-26T00:27:26.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.095+0000 7f4179e3c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4174137180 addr v1:172.21.15.45:6789/0 2024-04-26T00:27:26.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.095+0000 7f4179e3c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f417413d2e0 0x7f417413f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:26.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.095+0000 7f4179e3c640 10 monclient(hunting): picked mon.noname-c con 0x7f417413d2e0 addr v2:172.21.15.45:3300/0 2024-04-26T00:27:26.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.095+0000 7f4179e3c640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f417409b4f0 con 0x7f4174136da0 2024-04-26T00:27:26.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.095+0000 7f4179e3c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f417409b7d0 con 0x7f4174137180 2024-04-26T00:27:26.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.095+0000 7f4179e3c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:26.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.095+0000 7f4179e3c640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f417409bb70 con 0x7f417413d2e0 2024-04-26T00:27:26.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.095+0000 7f4179e3c640 10 monclient(hunting): _renew_subs 2024-04-26T00:27:26.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.095+0000 7f4179e3c640 10 monclient(hunting): authenticate will time out at 1427.406407s 2024-04-26T00:27:26.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.095+0000 7f41737fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f4174136da0 0x7f417409bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:33986/0 (socket says 172.21.15.37:33986) 2024-04-26T00:27:26.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.095+0000 7f41737fe640 1 -- 172.21.15.37:0/2398271370 learned_addr learned my addr 172.21.15.37:0/2398271370 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:27:26.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.095+0000 7f4173fff640 1 --2- 172.21.15.37:0/2398271370 >> v2:172.21.15.45:3300/0 conn(0x7f417413d2e0 0x7f417413f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:26.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.095+0000 7f4173fff640 10 monclient(hunting): get_auth_request con 0x7f417413d2e0 auth_method 0 2024-04-26T00:27:26.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.095+0000 7f4173fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:26.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.095+0000 7f4173fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:26.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.095+0000 7f4173fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:26.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.095+0000 7f4173fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:26.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.095+0000 7f4173fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:26.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.095+0000 7f4173fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:26.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.095+0000 7f4173fff640 10 monclient(hunting): handle_auth_done global_id 24676 payload 274 2024-04-26T00:27:26.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.095+0000 7f4173fff640 1 -- 172.21.15.37:0/2398271370 >> v1:172.21.15.45:6789/0 conn(0x7f4174137180 legacy=0x7f41741375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:27:26.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.095+0000 7f4173fff640 1 -- 172.21.15.37:0/2398271370 >> v1:172.21.15.37:6789/0 conn(0x7f4174136da0 legacy=0x7f417409bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:27:26.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.095+0000 7f4173fff640 10 monclient: _finish_hunting 0 2024-04-26T00:27:26.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.095+0000 7f4173fff640 1 monclient: found mon.noname-c 2024-04-26T00:27:26.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.095+0000 7f4173fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:27:26.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.095+0000 7f4173fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:27:26.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.095+0000 7f4173fff640 1 -- 172.21.15.37:0/2398271370 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f417409c5e0 con 0x7f417413d2e0 2024-04-26T00:27:26.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4173fff640 1 -- 172.21.15.37:0/2398271370 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f417413d2e0 msgr2=0x7f417413f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:26.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4173fff640 1 --2- 172.21.15.37:0/2398271370 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f417413d2e0 0x7f417413f6d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f416c009d90 tx=0x7f416c01c7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cca655cec600db99 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:26.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f41727fc640 1 -- 172.21.15.37:0/2398271370 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416c0ac070 con 0x7f417413d2e0 2024-04-26T00:27:26.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f41727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:26.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f41727fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:27:26.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f41727fc640 10 monclient: dump: 2024-04-26T00:27:26.091 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:26.091 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:26.091 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:26.091 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:26.091 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:26.091 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:26.091 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:26.091 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:26.091 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:26.091 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:26.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f41727fc640 10 monclient: _finish_auth 0 2024-04-26T00:27:26.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f41727fc640 10 monclient: _check_auth_tickets 2024-04-26T00:27:26.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f41727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:27:26.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f41727fc640 1 -- 172.21.15.37:0/2398271370 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f416c0a6760 con 0x7f417413d2e0 2024-04-26T00:27:26.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f41727fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:27:26.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f41727fc640 1 -- 172.21.15.37:0/2398271370 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416c0a6b30 con 0x7f417413d2e0 2024-04-26T00:27:26.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f41727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:26.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f41727fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:27:26.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f41727fc640 10 monclient: dump: 2024-04-26T00:27:26.092 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:26.092 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:26.092 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:26.092 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:26.092 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:26.092 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:26.092 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:26.093 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:26.093 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:26.093 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:26.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4179e3c640 5 monclient: authenticate success, global_id 24676 2024-04-26T00:27:26.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4179e3c640 10 monclient: get_monmap_and_config success 2024-04-26T00:27:26.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4179e3c640 4 set_mon_vals no callback set 2024-04-26T00:27:26.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4179e3c640 10 monclient: shutdown 2024-04-26T00:27:26.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4179e3c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:27:26.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4179e3c640 1 -- 172.21.15.37:0/2398271370 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f417413d2e0 msgr2=0x7f417413f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:26.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4179e3c640 1 --2- 172.21.15.37:0/2398271370 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f417413d2e0 0x7f417413f6d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f416c009d90 tx=0x7f416c01c7b0 comp rx=0 tx=0).stop 2024-04-26T00:27:26.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4179e3c640 1 -- 172.21.15.37:0/2398271370 shutdown_connections 2024-04-26T00:27:26.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4179e3c640 1 --2- 172.21.15.37:0/2398271370 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f417413d2e0 0x7f417413f6d0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:26.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4179e3c640 1 -- 172.21.15.37:0/2398271370 >> 172.21.15.37:0/2398271370 conn(0x7f41740a5f00 msgr2=0x7f41740a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:27:26.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4179e3c640 1 -- 172.21.15.37:0/2398271370 shutdown_connections 2024-04-26T00:27:26.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4179e3c640 1 -- 172.21.15.37:0/2398271370 wait complete. 2024-04-26T00:27:26.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4179e3c640 10 monclient: build_initial_monmap 2024-04-26T00:27:26.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4179e3c640 10 monclient: monmap: 2024-04-26T00:27:26.099 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:27:26.099 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:27:26.099 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:27:26.099 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:27:26.099 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:27:26.099 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:27:26.099 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:27:26.099 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:27:26.099 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:27:26.099 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:26.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4179e3c640 1 Processor -- start 2024-04-26T00:27:26.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4179e3c640 1 -- start start 2024-04-26T00:27:26.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4179e3c640 10 monclient: init 2024-04-26T00:27:26.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4179e3c640 10 monclient: _reopen_session rank -1 2024-04-26T00:27:26.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4179e3c640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:27:26.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4179e3c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4174136da0 0x7f4174135bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:26.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4179e3c640 10 monclient(hunting): picked mon.noname-b con 0x7f4174136da0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:27:26.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4179e3c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4174137180 0x7f4174138940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:26.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4179e3c640 10 monclient(hunting): picked mon.noname-c con 0x7f4174137180 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:27:26.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4179e3c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f417413d2e0 0x7f4174138e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:26.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4179e3c640 10 monclient(hunting): picked mon.noname-a con 0x7f417413d2e0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:27:26.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4179e3c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:26.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4179e3c640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f4174142e20 con 0x7f417413d2e0 2024-04-26T00:27:26.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4179e3c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:26.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4179e3c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4174142ca0 con 0x7f4174136da0 2024-04-26T00:27:26.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4179e3c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:26.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4179e3c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4174142fa0 con 0x7f4174137180 2024-04-26T00:27:26.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4179e3c640 10 monclient(hunting): _renew_subs 2024-04-26T00:27:26.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4179e3c640 10 monclient(hunting): authenticate will time out at 1427.415159s 2024-04-26T00:27:26.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4172ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4174137180 0x7f4174138940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:26.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4173fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f417413d2e0 0x7f4174138e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:26.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4173fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f417413d2e0 0x7f4174138e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:37468/0 (socket says 172.21.15.37:37468) 2024-04-26T00:27:26.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4173fff640 1 -- 172.21.15.37:0/3941325129 learned_addr learned my addr 172.21.15.37:0/3941325129 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:27:26.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4172ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4174137180 0x7f4174138940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:34370/0 (socket says 172.21.15.37:34370) 2024-04-26T00:27:26.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4173fff640 10 monclient(hunting): get_auth_request con 0x7f417413d2e0 auth_method 0 2024-04-26T00:27:26.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4173fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:26.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4173fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:26.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4173fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:26.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4172ffd640 10 monclient(hunting): get_auth_request con 0x7f4174137180 auth_method 0 2024-04-26T00:27:26.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4172ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:26.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4172ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:26.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4172ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:26.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4172ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:26.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4172ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:26.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4173fff640 1 -- 172.21.15.37:0/3941325129 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f417413d2e0 msgr2=0x7f4174138e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:27:26.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4173fff640 1 -- 172.21.15.37:0/3941325129 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f417413d2e0 msgr2=0x7f4174138e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:27:26.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4173fff640 1 --2- 172.21.15.37:0/3941325129 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f417413d2e0 0x7f4174138e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:27:26.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4172ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:26.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4173fff640 1 --2- 172.21.15.37:0/3941325129 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f417413d2e0 0x7f4174138e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:27:26.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f41737fe640 1 --2- 172.21.15.37:0/3941325129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4174136da0 0x7f4174135bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:26.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f41737fe640 10 monclient(hunting): get_auth_request con 0x7f4174136da0 auth_method 0 2024-04-26T00:27:26.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f41737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:26.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f41737fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:26.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f41737fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:26.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4172ffd640 10 monclient(hunting): handle_auth_done global_id 84179 payload 824 2024-04-26T00:27:26.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4172ffd640 1 -- 172.21.15.37:0/3941325129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4174136da0 msgr2=0x7f4174135bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:26.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4172ffd640 1 --2- 172.21.15.37:0/3941325129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4174136da0 0x7f4174135bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:26.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4172ffd640 1 -- 172.21.15.37:0/3941325129 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f417413d2e0 msgr2=0x7f4174138e80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:27:26.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4172ffd640 1 --2- 172.21.15.37:0/3941325129 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f417413d2e0 0x7f4174138e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:26.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4172ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:27:26.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4172ffd640 1 monclient: found mon.noname-c 2024-04-26T00:27:26.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4172ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:27:26.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4172ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:27:26.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.103+0000 7f4172ffd640 1 -- 172.21.15.37:0/3941325129 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41741cff80 con 0x7f4174137180 2024-04-26T00:27:26.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.107+0000 7f4172ffd640 1 -- 172.21.15.37:0/3941325129 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4174137180 msgr2=0x7f4174138940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:26.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.107+0000 7f4172ffd640 1 --2- 172.21.15.37:0/3941325129 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4174137180 0x7f4174138940 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f416800c410 tx=0x7f416800c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:26.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.107+0000 7f415bfff640 1 -- 172.21.15.37:0/3941325129 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4168017070 con 0x7f4174137180 2024-04-26T00:27:26.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.107+0000 7f415bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:26.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.107+0000 7f415bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:27:26.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.107+0000 7f415bfff640 10 monclient: dump: 2024-04-26T00:27:26.104 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:26.104 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:26.104 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:26.104 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:26.104 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:26.104 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:26.104 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:26.105 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:26.105 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:26.105 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:26.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.107+0000 7f415bfff640 10 monclient: _finish_auth 0 2024-04-26T00:27:26.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.107+0000 7f415bfff640 10 monclient: _check_auth_tickets 2024-04-26T00:27:26.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.107+0000 7f415bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:27:26.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.107+0000 7f415bfff640 1 -- 172.21.15.37:0/3941325129 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41680069e0 con 0x7f4174137180 2024-04-26T00:27:26.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.107+0000 7f4179e3c640 5 monclient: authenticate success, global_id 84179 2024-04-26T00:27:26.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.107+0000 7f4179e3c640 10 monclient: _renew_subs 2024-04-26T00:27:26.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.107+0000 7f4179e3c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:27:26.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.107+0000 7f4179e3c640 1 -- 172.21.15.37:0/3941325129 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f417413c880 con 0x7f4174137180 2024-04-26T00:27:26.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.107+0000 7f415bfff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:27:26.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.107+0000 7f415bfff640 1 -- 172.21.15.37:0/3941325129 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4168006dd0 con 0x7f4174137180 2024-04-26T00:27:26.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.107+0000 7f415bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:26.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.107+0000 7f415bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:27:26.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.107+0000 7f415bfff640 10 monclient: dump: 2024-04-26T00:27:26.106 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:26.106 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:26.106 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:26.106 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:26.106 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:26.106 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:26.106 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:26.106 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:26.106 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:26.106 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:26.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.107+0000 7f4171ffb640 4 set_mon_vals no callback set 2024-04-26T00:27:26.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.107+0000 7f4179e3c640 10 monclient: _renew_subs 2024-04-26T00:27:26.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.107+0000 7f4179e3c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:27:26.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.107+0000 7f4179e3c640 1 -- 172.21.15.37:0/3941325129 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f417413c5a0 con 0x7f4174137180 2024-04-26T00:27:26.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.107+0000 7f4179e3c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:26.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.107+0000 7f4179e3c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:26.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.107+0000 7f4179e3c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:27:26.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.107+0000 7f4179e3c640 1 -- 172.21.15.37:0/3941325129 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4174066f40 con 0x7f4174137180 2024-04-26T00:27:26.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.107+0000 7f415bfff640 1 -- 172.21.15.37:0/3941325129 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f416801a070 con 0x7f4174137180 2024-04-26T00:27:26.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.107+0000 7f415bfff640 1 --2- 172.21.15.37:0/3941325129 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4160035e80 0x7f4160038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:26.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.107+0000 7f415bfff640 1 -- 172.21.15.37:0/3941325129 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(292..292 src has 177..292) ==== 4039+0+0 (secure 0 0 0) 0x7f4168047f40 con 0x7f4174137180 2024-04-26T00:27:26.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.107+0000 7f41737fe640 1 --2- 172.21.15.37:0/3941325129 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4160035e80 0x7f4160038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:26.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.111+0000 7f41737fe640 10 monclient: get_auth_request con 0x7f4160035e80 auth_method 0 2024-04-26T00:27:26.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.111+0000 7f41737fe640 1 --2- 172.21.15.37:0/3941325129 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4160035e80 0x7f4160038340 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4164007fd0 tx=0x7f4164009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:26.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.115+0000 7f415bfff640 1 -- 172.21.15.37:0/3941325129 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f41741cff80 con 0x7f4174137180 2024-04-26T00:27:26.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.115+0000 7f415bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:26.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.115+0000 7f415bfff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:27:26.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.343+0000 7f4179e3c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:27:26.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.343+0000 7f4179e3c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:27:26.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.343+0000 7f4179e3c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:27:26.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.343+0000 7f4179e3c640 1 -- 172.21.15.37:0/3941325129 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f417413ae70 con 0x7f4174137180 2024-04-26T00:27:26.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.347+0000 7f415bfff640 1 -- 172.21.15.37:0/3941325129 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f416801abf0 con 0x7f4174137180 2024-04-26T00:27:26.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.347+0000 7f415bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:27:26.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.347+0000 7f415bfff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:27:26.335 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":58,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:27:26.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.347+0000 7f4179e3c640 1 -- 172.21.15.37:0/3941325129 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4160035e80 msgr2=0x7f4160038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:26.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.347+0000 7f4179e3c640 1 --2- 172.21.15.37:0/3941325129 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4160035e80 0x7f4160038340 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4164007fd0 tx=0x7f4164009040 comp rx=0 tx=0).stop 2024-04-26T00:27:26.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.347+0000 7f4179e3c640 10 monclient: shutdown 2024-04-26T00:27:26.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.347+0000 7f4179e3c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:27:26.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.347+0000 7f4179e3c640 1 -- 172.21.15.37:0/3941325129 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4174137180 msgr2=0x7f4174138940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:26.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.347+0000 7f4179e3c640 1 --2- 172.21.15.37:0/3941325129 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4174137180 0x7f4174138940 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f416800c410 tx=0x7f416800c8d0 comp rx=0 tx=0).stop 2024-04-26T00:27:26.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.347+0000 7f4179e3c640 1 -- 172.21.15.37:0/3941325129 shutdown_connections 2024-04-26T00:27:26.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.347+0000 7f4179e3c640 1 --2- 172.21.15.37:0/3941325129 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4160035e80 0x7f4160038340 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:26.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.347+0000 7f4179e3c640 1 --2- 172.21.15.37:0/3941325129 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f417413d2e0 0x7f4174138e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:26.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.347+0000 7f4179e3c640 1 --2- 172.21.15.37:0/3941325129 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4174137180 0x7f4174138940 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:26.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.347+0000 7f4179e3c640 1 --2- 172.21.15.37:0/3941325129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4174136da0 0x7f4174135bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:26.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.347+0000 7f4179e3c640 1 -- 172.21.15.37:0/3941325129 >> 172.21.15.37:0/3941325129 conn(0x7f41740a5f00 msgr2=0x7f417413df50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:27:26.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.347+0000 7f4179e3c640 1 -- 172.21.15.37:0/3941325129 shutdown_connections 2024-04-26T00:27:26.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.347+0000 7f4179e3c640 1 -- 172.21.15.37:0/3941325129 wait complete. 2024-04-26T00:27:26.353 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:27:26.353 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:27:26.354 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:27:26.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.483+0000 7f00a9bd2640 10 monclient: get_monmap_and_config 2024-04-26T00:27:26.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.483+0000 7f00a9bd2640 10 monclient: build_initial_monmap 2024-04-26T00:27:26.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.483+0000 7f00a9bd2640 10 monclient: monmap: 2024-04-26T00:27:26.473 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:27:26.473 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:26.473 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:27:26.490580+0000 2024-04-26T00:27:26.473 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:27:26.490580+0000 2024-04-26T00:27:26.473 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:27:26.473 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:26.473 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:27:26.473 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:27:26.473 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:27:26.473 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:27:26.473 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:27:26.473 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:26.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.483+0000 7f00a9bd2640 1 Processor -- start 2024-04-26T00:27:26.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.483+0000 7f00a9bd2640 1 -- start start 2024-04-26T00:27:26.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.483+0000 7f00a9bd2640 10 monclient: init 2024-04-26T00:27:26.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a9bd2640 10 monclient: _reopen_session rank -1 2024-04-26T00:27:26.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a9bd2640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-04-26T00:27:26.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a9bd2640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f00a40a0300 0x7f00a409fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:26.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a9bd2640 10 monclient(hunting): picked mon.noname-b con 0x7f00a40a0300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:27:26.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a9bd2640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f00a40a06e0 0x7f00a4134ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:26.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a9bd2640 10 monclient(hunting): picked mon.noname-a con 0x7f00a40a06e0 addr v2:172.21.15.37:3300/0 2024-04-26T00:27:26.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a9bd2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f00a41350e0 addr v1:172.21.15.37:6789/0 2024-04-26T00:27:26.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a9bd2640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00a409c190 con 0x7f00a41350e0 2024-04-26T00:27:26.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a9bd2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:26.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a9bd2640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f00a40a13f0 con 0x7f00a40a06e0 2024-04-26T00:27:26.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a9bd2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:26.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a9bd2640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f00a40a1110 con 0x7f00a40a0300 2024-04-26T00:27:26.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a9bd2640 10 monclient(hunting): _renew_subs 2024-04-26T00:27:26.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a9bd2640 10 monclient(hunting): authenticate will time out at 1427.796494s 2024-04-26T00:27:26.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a2ffd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f00a40a06e0 0x7f00a4134ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:26.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a37fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f00a40a0300 0x7f00a409fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:26.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a3fff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f00a41350e0 0x7f00a413bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:33990/0 (socket says 172.21.15.37:33990) 2024-04-26T00:27:26.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a3fff640 1 -- 172.21.15.37:0/2301086556 learned_addr learned my addr 172.21.15.37:0/2301086556 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:27:26.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a2ffd640 10 monclient(hunting): get_auth_request con 0x7f00a40a06e0 auth_method 0 2024-04-26T00:27:26.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:26.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:26.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:26.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a37fe640 10 monclient(hunting): get_auth_request con 0x7f00a40a0300 auth_method 0 2024-04-26T00:27:26.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:26.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:26.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:26.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a2ffd640 1 -- 172.21.15.37:0/2301086556 >> v2:172.21.15.37:3300/0 conn(0x7f00a40a06e0 msgr2=0x7f00a4134ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:27:26.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a2ffd640 1 -- 172.21.15.37:0/2301086556 >> v2:172.21.15.37:3300/0 conn(0x7f00a40a06e0 msgr2=0x7f00a4134ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:27:26.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a2ffd640 1 --2- 172.21.15.37:0/2301086556 >> v2:172.21.15.37:3300/0 conn(0x7f00a40a06e0 0x7f00a4134ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:27:26.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:26.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:26.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a2ffd640 1 --2- 172.21.15.37:0/2301086556 >> v2:172.21.15.37:3300/0 conn(0x7f00a40a06e0 0x7f00a4134ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:27:26.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:26.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a37fe640 10 monclient(hunting): handle_auth_done global_id 84182 payload 274 2024-04-26T00:27:26.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a37fe640 1 -- 172.21.15.37:0/2301086556 >> v2:172.21.15.37:3300/0 conn(0x7f00a40a06e0 msgr2=0x7f00a4134ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:27:26.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a37fe640 1 --2- 172.21.15.37:0/2301086556 >> v2:172.21.15.37:3300/0 conn(0x7f00a40a06e0 0x7f00a4134ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:26.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a37fe640 1 -- 172.21.15.37:0/2301086556 >> v1:172.21.15.37:6789/0 conn(0x7f00a41350e0 legacy=0x7f00a413bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:27:26.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a37fe640 10 monclient: _finish_hunting 0 2024-04-26T00:27:26.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a37fe640 1 monclient: found mon.noname-b 2024-04-26T00:27:26.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:27:26.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:27:26.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.487+0000 7f00a37fe640 1 -- 172.21.15.37:0/2301086556 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00a40a16d0 con 0x7f00a40a0300 2024-04-26T00:27:26.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a37fe640 1 -- 172.21.15.37:0/2301086556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f00a40a0300 msgr2=0x7f00a409fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:26.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a37fe640 1 --2- 172.21.15.37:0/2301086556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f00a40a0300 0x7f00a409fc50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f009c009d90 tx=0x7f009c01c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2bc3f3c9ec9953a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:26.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a27fc640 1 -- 172.21.15.37:0/2301086556 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009c0b2070 con 0x7f00a40a0300 2024-04-26T00:27:26.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:26.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:27:26.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a27fc640 10 monclient: dump: 2024-04-26T00:27:26.481 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:26.481 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:26.481 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:26.481 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:26.481 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:26.481 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:26.481 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:26.481 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:26.482 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:26.482 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:26.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a27fc640 10 monclient: _finish_auth 0 2024-04-26T00:27:26.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a27fc640 10 monclient: _check_auth_tickets 2024-04-26T00:27:26.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:27:26.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a27fc640 1 -- 172.21.15.37:0/2301086556 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f009c0a5790 con 0x7f00a40a0300 2024-04-26T00:27:26.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a27fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:27:26.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a27fc640 1 -- 172.21.15.37:0/2301086556 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009c0a5b60 con 0x7f00a40a0300 2024-04-26T00:27:26.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a9bd2640 5 monclient: authenticate success, global_id 84182 2024-04-26T00:27:26.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a9bd2640 10 monclient: get_monmap_and_config success 2024-04-26T00:27:26.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a9bd2640 4 set_mon_vals no callback set 2024-04-26T00:27:26.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:26.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:27:26.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a27fc640 10 monclient: dump: 2024-04-26T00:27:26.483 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:26.483 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:26.483 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:26.483 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:26.483 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:26.484 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:26.484 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:26.484 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:26.484 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:26.484 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:26.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a9bd2640 10 monclient: shutdown 2024-04-26T00:27:26.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a9bd2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:27:26.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a9bd2640 1 -- 172.21.15.37:0/2301086556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f00a40a0300 msgr2=0x7f00a409fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:26.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a9bd2640 1 --2- 172.21.15.37:0/2301086556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f00a40a0300 0x7f00a409fc50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f009c009d90 tx=0x7f009c01c820 comp rx=0 tx=0).stop 2024-04-26T00:27:26.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a9bd2640 1 -- 172.21.15.37:0/2301086556 shutdown_connections 2024-04-26T00:27:26.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a9bd2640 1 --2- 172.21.15.37:0/2301086556 >> v2:172.21.15.37:3300/0 conn(0x7f00a40a06e0 0x7f00a4134ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:26.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a9bd2640 1 --2- 172.21.15.37:0/2301086556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f00a40a0300 0x7f00a409fc50 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:26.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a9bd2640 1 -- 172.21.15.37:0/2301086556 >> 172.21.15.37:0/2301086556 conn(0x7f00a40a1d50 msgr2=0x7f00a40a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:27:26.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a9bd2640 1 -- 172.21.15.37:0/2301086556 shutdown_connections 2024-04-26T00:27:26.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a9bd2640 1 -- 172.21.15.37:0/2301086556 wait complete. 2024-04-26T00:27:26.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a9bd2640 10 monclient: build_initial_monmap 2024-04-26T00:27:26.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a9bd2640 10 monclient: monmap: 2024-04-26T00:27:26.485 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:27:26.485 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:27:26.485 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:27:26.485 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:27:26.485 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:27:26.485 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:27:26.485 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:27:26.485 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:27:26.485 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:27:26.485 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:26.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a9bd2640 1 Processor -- start 2024-04-26T00:27:26.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a9bd2640 1 -- start start 2024-04-26T00:27:26.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a9bd2640 10 monclient: init 2024-04-26T00:27:26.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a9bd2640 10 monclient: _reopen_session rank -1 2024-04-26T00:27:26.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a9bd2640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:27:26.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a9bd2640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f00a40a0300 0x7f00a41cd020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:26.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a9bd2640 10 monclient(hunting): picked mon.noname-c con 0x7f00a40a0300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:27:26.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a9bd2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f00a40a06e0 0x7f00a413bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:26.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a9bd2640 10 monclient(hunting): picked mon.noname-b con 0x7f00a40a06e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:27:26.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a9bd2640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a41350e0 0x7f00a4135c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:26.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a9bd2640 10 monclient(hunting): picked mon.noname-a con 0x7f00a41350e0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:27:26.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a9bd2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:26.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a9bd2640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f00a40a0c40 con 0x7f00a41350e0 2024-04-26T00:27:26.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a9bd2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:26.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a9bd2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f00a40a0ac0 con 0x7f00a40a06e0 2024-04-26T00:27:26.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a9bd2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:26.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a9bd2640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f00a413fe10 con 0x7f00a40a0300 2024-04-26T00:27:26.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a9bd2640 10 monclient(hunting): _renew_subs 2024-04-26T00:27:26.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a9bd2640 10 monclient(hunting): authenticate will time out at 1427.803827s 2024-04-26T00:27:26.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a37fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f00a40a0300 0x7f00a41cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:26.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a3fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a41350e0 0x7f00a4135c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:26.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a37fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f00a40a0300 0x7f00a41cd020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:34386/0 (socket says 172.21.15.37:34386) 2024-04-26T00:27:26.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a37fe640 1 -- 172.21.15.37:0/3210261000 learned_addr learned my addr 172.21.15.37:0/3210261000 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:27:26.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a3fff640 10 monclient(hunting): get_auth_request con 0x7f00a41350e0 auth_method 0 2024-04-26T00:27:26.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.491+0000 7f00a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:26.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:26.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:26.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a37fe640 10 monclient(hunting): get_auth_request con 0x7f00a40a0300 auth_method 0 2024-04-26T00:27:26.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:26.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:26.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:26.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a3fff640 1 -- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a41350e0 msgr2=0x7f00a4135c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:27:26.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a3fff640 1 -- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a41350e0 msgr2=0x7f00a4135c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:27:26.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a3fff640 1 --2- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a41350e0 0x7f00a4135c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:27:26.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:26.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:26.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:26.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a3fff640 1 --2- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a41350e0 0x7f00a4135c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:27:26.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a2ffd640 1 --2- 172.21.15.37:0/3210261000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f00a40a06e0 0x7f00a413bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:26.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a37fe640 10 monclient(hunting): handle_auth_done global_id 84185 payload 824 2024-04-26T00:27:26.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a37fe640 1 -- 172.21.15.37:0/3210261000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f00a40a06e0 msgr2=0x7f00a413bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:26.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a37fe640 1 --2- 172.21.15.37:0/3210261000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f00a40a06e0 0x7f00a413bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:26.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a37fe640 1 -- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a41350e0 msgr2=0x7f00a4135c60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:27:26.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a37fe640 1 --2- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a41350e0 0x7f00a4135c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:26.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a37fe640 10 monclient: _finish_hunting 0 2024-04-26T00:27:26.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a37fe640 1 monclient: found mon.noname-c 2024-04-26T00:27:26.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:27:26.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:27:26.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a37fe640 1 -- 172.21.15.37:0/3210261000 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00a41397c0 con 0x7f00a40a0300 2024-04-26T00:27:26.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a2ffd640 10 monclient: get_auth_request con 0x7f00a40a06e0 auth_method 0 2024-04-26T00:27:26.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a2ffd640 1 --2- 172.21.15.37:0/3210261000 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f00a40a06e0 0x7f00a413bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:27:26.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a37fe640 1 -- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f00a40a0300 msgr2=0x7f00a41cd020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:26.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a37fe640 1 --2- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f00a40a0300 0x7f00a41cd020 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f009c0b0040 tx=0x7f009c019620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:26.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f009bfff640 1 -- 172.21.15.37:0/3210261000 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009c0b2050 con 0x7f00a40a0300 2024-04-26T00:27:26.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f009bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:26.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f009bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:27:26.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f009bfff640 10 monclient: dump: 2024-04-26T00:27:26.494 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:26.494 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:26.494 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:26.494 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:26.494 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:26.494 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:26.494 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:26.494 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:26.495 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:26.495 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:26.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f009bfff640 10 monclient: _finish_auth 0 2024-04-26T00:27:26.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f009bfff640 10 monclient: _check_auth_tickets 2024-04-26T00:27:26.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f009bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:27:26.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f009bfff640 1 -- 172.21.15.37:0/3210261000 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f009c0be070 con 0x7f00a40a0300 2024-04-26T00:27:26.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f009bfff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:27:26.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f009bfff640 1 -- 172.21.15.37:0/3210261000 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009c0adc60 con 0x7f00a40a0300 2024-04-26T00:27:26.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a9bd2640 5 monclient: authenticate success, global_id 84185 2024-04-26T00:27:26.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a9bd2640 10 monclient: _renew_subs 2024-04-26T00:27:26.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a9bd2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:27:26.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a9bd2640 1 -- 172.21.15.37:0/3210261000 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00a41cd560 con 0x7f00a40a0300 2024-04-26T00:27:26.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f009bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:26.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f009bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:27:26.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a1ffb640 4 set_mon_vals no callback set 2024-04-26T00:27:26.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f009bfff640 10 monclient: dump: 2024-04-26T00:27:26.496 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:26.496 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:26.496 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:26.496 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:26.496 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:26.496 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:26.496 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:26.496 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:26.496 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:26.496 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:26.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a9bd2640 10 monclient: _renew_subs 2024-04-26T00:27:26.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a9bd2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:27:26.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a9bd2640 1 -- 172.21.15.37:0/3210261000 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f00a409f550 con 0x7f00a40a0300 2024-04-26T00:27:26.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a9bd2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:26.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a9bd2640 1 --2- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a413c500 0x7f00a41361a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:26.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a9bd2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:26.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a9bd2640 1 -- 172.21.15.37:0/3210261000 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f00a41377b0 con 0x7f00a413c500 2024-04-26T00:27:26.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a2ffd640 1 --2- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a413c500 0x7f00a41361a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:26.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a2ffd640 10 monclient: get_auth_request con 0x7f00a413c500 auth_method 0 2024-04-26T00:27:26.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:26.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:26.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:26.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f009bfff640 1 -- 172.21.15.37:0/3210261000 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f009c0a5610 con 0x7f00a40a0300 2024-04-26T00:27:26.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a2ffd640 1 -- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a413c500 msgr2=0x7f00a41361a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:27:26.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a2ffd640 1 -- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a413c500 msgr2=0x7f00a41361a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:27:26.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a2ffd640 1 --2- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a413c500 0x7f00a41361a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:27:26.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.495+0000 7f00a2ffd640 1 --2- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a413c500 0x7f00a41361a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:27:26.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.499+0000 7f009bfff640 1 --2- 172.21.15.37:0/3210261000 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0084035ed0 0x7f0084038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:26.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.499+0000 7f009bfff640 1 -- 172.21.15.37:0/3210261000 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(292..292 src has 177..292) ==== 4039+0+0 (secure 0 0 0) 0x7f009c0e36f0 con 0x7f00a40a0300 2024-04-26T00:27:26.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.499+0000 7f00a3fff640 1 --2- 172.21.15.37:0/3210261000 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0084035ed0 0x7f0084038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:26.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.499+0000 7f00a3fff640 10 monclient: get_auth_request con 0x7f0084035ed0 auth_method 0 2024-04-26T00:27:26.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.499+0000 7f00a3fff640 1 --2- 172.21.15.37:0/3210261000 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0084035ed0 0x7f0084038390 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f00a413d050 tx=0x7f0094009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:26.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.699+0000 7f00a2ffd640 1 --2- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a413c500 0x7f00a41361a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:26.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.699+0000 7f00a2ffd640 10 monclient: get_auth_request con 0x7f00a413c500 auth_method 0 2024-04-26T00:27:26.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.699+0000 7f00a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:26.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.699+0000 7f00a2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:26.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.699+0000 7f00a2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:27:26.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.699+0000 7f00a2ffd640 1 -- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a413c500 msgr2=0x7f00a41361a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:27:26.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.699+0000 7f00a2ffd640 1 -- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a413c500 msgr2=0x7f00a41361a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:27:26.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.699+0000 7f00a2ffd640 1 --2- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a413c500 0x7f00a41361a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:27:26.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:26.699+0000 7f00a2ffd640 1 --2- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a413c500 0x7f00a41361a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:27:26.905 INFO:tasks.rados.rados.0.smithi045.stdout:1223: done (1 left) 2024-04-26T00:27:26.905 INFO:tasks.rados.rados.0.smithi045.stdout:1224: done (0 left) 2024-04-26T00:27:26.905 INFO:tasks.rados.rados.0.smithi045.stdout:1225: write oid 9 current snap is 143 2024-04-26T00:27:26.905 INFO:tasks.rados.rados.0.smithi045.stdout:1225: seq_num 458 ranges {497606=645222,1934400=629494,3243966=453640} 2024-04-26T00:27:26.924 INFO:tasks.rados.rados.0.smithi045.stdout:1225: writing smithi04526579-9 from 497606 to 1142828 tid 1 2024-04-26T00:27:26.947 INFO:tasks.rados.rados.0.smithi045.stdout:1225: writing smithi04526579-9 from 1934400 to 2563894 tid 2 2024-04-26T00:27:26.964 INFO:tasks.rados.rados.0.smithi045.stdout:1225: writing smithi04526579-9 from 3243966 to 3697606 tid 3 2024-04-26T00:27:26.964 INFO:tasks.rados.rados.0.smithi045.stdout:1226: read oid 31 snap -1 2024-04-26T00:27:26.964 INFO:tasks.rados.rados.0.smithi045.stdout:1226: expect (ObjNum 446 snap 139 seq_num 446) 2024-04-26T00:27:26.964 INFO:tasks.rados.rados.0.smithi045.stdout:1227: write oid 26 current snap is 143 2024-04-26T00:27:26.964 INFO:tasks.rados.rados.0.smithi045.stdout:1225: finishing write tid 1 to smithi04526579-9 2024-04-26T00:27:26.964 INFO:tasks.rados.rados.0.smithi045.stdout:1225: finishing write tid 2 to smithi04526579-9 2024-04-26T00:27:26.964 INFO:tasks.rados.rados.0.smithi045.stdout:1227: seq_num 459 ranges {514413=720471,1876529=773152,3438683=275730} 2024-04-26T00:27:26.981 INFO:tasks.rados.rados.0.smithi045.stdout:1227: writing smithi04526579-26 from 514413 to 1234884 tid 1 2024-04-26T00:27:27.000 INFO:tasks.rados.rados.0.smithi045.stdout:1227: writing smithi04526579-26 from 1876529 to 2649681 tid 2 2024-04-26T00:27:27.012 INFO:tasks.rados.rados.0.smithi045.stdout:1227: writing smithi04526579-26 from 3438683 to 3714413 tid 3 2024-04-26T00:27:27.013 INFO:tasks.rados.rados.0.smithi045.stdout:1228: snap_create 2024-04-26T00:27:27.013 INFO:tasks.rados.rados.0.smithi045.stdout:1225: finishing write tid 3 to smithi04526579-9 2024-04-26T00:27:27.013 INFO:tasks.rados.rados.0.smithi045.stdout:1225: finishing write tid 4 to smithi04526579-9 2024-04-26T00:27:27.013 INFO:tasks.rados.rados.0.smithi045.stdout:1225: finishing write tid 5 to smithi04526579-9 2024-04-26T00:27:27.013 INFO:tasks.rados.rados.0.smithi045.stdout:1225: oid 9 updating version 0 to 652 2024-04-26T00:27:27.013 INFO:tasks.rados.rados.0.smithi045.stdout:1225: oid 9 version 652 is already newer than 650 2024-04-26T00:27:27.013 INFO:tasks.rados.rados.0.smithi045.stdout:1225: oid 9 version 652 is already newer than 651 2024-04-26T00:27:27.013 INFO:tasks.rados.rados.0.smithi045.stdout:1225: oid 9 version 652 is already newer than 649 2024-04-26T00:27:27.013 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 652 (ObjNum 458 snap 143 seq_num 458) dirty exists 2024-04-26T00:27:27.013 INFO:tasks.rados.rados.0.smithi045.stdout:1225: left oid 9 (ObjNum 458 snap 143 seq_num 458) 2024-04-26T00:27:27.013 INFO:tasks.rados.rados.0.smithi045.stdout:1226: expect (ObjNum 446 snap 139 seq_num 446) 2024-04-26T00:27:27.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:27.103+0000 7f00a2ffd640 1 --2- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a413c500 0x7f00a41361a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:27.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:27.103+0000 7f00a2ffd640 10 monclient: get_auth_request con 0x7f00a413c500 auth_method 0 2024-04-26T00:27:27.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:27.103+0000 7f00a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:27.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:27.103+0000 7f00a2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:27.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:27.103+0000 7f00a2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:27:27.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:27.103+0000 7f00a2ffd640 1 -- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a413c500 msgr2=0x7f00a41361a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:27:27.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:27.103+0000 7f00a2ffd640 1 -- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a413c500 msgr2=0x7f00a41361a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:27:27.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:27.103+0000 7f00a2ffd640 1 --2- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a413c500 0x7f00a41361a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:27:27.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:27.103+0000 7f00a2ffd640 1 --2- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a413c500 0x7f00a41361a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:27:27.205 INFO:tasks.rados.rados.0.smithi045.stdout:1227: finishing write tid 1 to smithi04526579-26 2024-04-26T00:27:27.205 INFO:tasks.rados.rados.0.smithi045.stdout:1227: finishing write tid 2 to smithi04526579-26 2024-04-26T00:27:27.205 INFO:tasks.rados.rados.0.smithi045.stdout:1227: finishing write tid 3 to smithi04526579-26 2024-04-26T00:27:27.205 INFO:tasks.rados.rados.0.smithi045.stdout:1227: finishing write tid 4 to smithi04526579-26 2024-04-26T00:27:27.206 INFO:tasks.rados.rados.0.smithi045.stdout:1227: finishing write tid 5 to smithi04526579-26 2024-04-26T00:27:27.206 INFO:tasks.rados.rados.0.smithi045.stdout:1227: oid 26 updating version 0 to 806 2024-04-26T00:27:27.206 INFO:tasks.rados.rados.0.smithi045.stdout:1227: oid 26 updating version 806 to 807 2024-04-26T00:27:27.206 INFO:tasks.rados.rados.0.smithi045.stdout:1227: oid 26 updating version 807 to 809 2024-04-26T00:27:27.206 INFO:tasks.rados.rados.0.smithi045.stdout:1227: oid 26 version 809 is already newer than 808 2024-04-26T00:27:27.206 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 809 (ObjNum 459 snap 143 seq_num 459) dirty exists 2024-04-26T00:27:27.206 INFO:tasks.rados.rados.0.smithi045.stdout:1227: left oid 26 (ObjNum 459 snap 143 seq_num 459) 2024-04-26T00:27:27.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:27.907+0000 7f00a2ffd640 1 --2- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a413c500 0x7f00a41361a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:27.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:27.907+0000 7f00a2ffd640 10 monclient: get_auth_request con 0x7f00a413c500 auth_method 0 2024-04-26T00:27:27.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:27.907+0000 7f00a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:27.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:27.907+0000 7f00a2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:27.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:27.907+0000 7f00a2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:27:27.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:27.907+0000 7f00a2ffd640 1 -- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a413c500 msgr2=0x7f00a41361a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:27:27.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:27.907+0000 7f00a2ffd640 1 -- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a413c500 msgr2=0x7f00a41361a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:27:27.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:27.907+0000 7f00a2ffd640 1 --2- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a413c500 0x7f00a41361a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:27:27.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:27.907+0000 7f00a2ffd640 1 --2- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a413c500 0x7f00a41361a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:27:28.000 INFO:tasks.rados.rados.0.smithi045.stdout:1225: done (3 left) 2024-04-26T00:27:28.000 INFO:tasks.rados.rados.0.smithi045.stdout:1226: done (2 left) 2024-04-26T00:27:28.000 INFO:tasks.rados.rados.0.smithi045.stdout:1227: done (1 left) 2024-04-26T00:27:28.001 INFO:tasks.rados.rados.0.smithi045.stdout:1228: done (0 left) 2024-04-26T00:27:28.001 INFO:tasks.rados.rados.0.smithi045.stdout:1229: snap_create 2024-04-26T00:27:29.010 INFO:tasks.rados.rados.0.smithi045.stdout:1229: done (0 left) 2024-04-26T00:27:29.010 INFO:tasks.rados.rados.0.smithi045.stdout:1230: write (excl) oid 7 current snap is 145 2024-04-26T00:27:29.010 INFO:tasks.rados.rados.0.smithi045.stdout:1230: seq_num 460 ranges {531220=795720,1935011=516810,3066106=420420} 2024-04-26T00:27:29.026 INFO:tasks.rados.rados.0.smithi045.stdout:1230: writing smithi04526579-7 from 531220 to 1326940 tid 1 2024-04-26T00:27:29.040 INFO:tasks.rados.rados.0.smithi045.stdout:1230: writing smithi04526579-7 from 1935011 to 2451821 tid 2 2024-04-26T00:27:29.054 INFO:tasks.rados.rados.0.smithi045.stdout:1230: writing smithi04526579-7 from 3066106 to 3486526 tid 3 2024-04-26T00:27:29.054 INFO:tasks.rados.rados.0.smithi045.stdout:1231: read oid 26 snap -1 2024-04-26T00:27:29.054 INFO:tasks.rados.rados.0.smithi045.stdout:1231: expect (ObjNum 459 snap 143 seq_num 459) 2024-04-26T00:27:29.054 INFO:tasks.rados.rados.0.smithi045.stdout:1232: read oid 36 snap -1 2024-04-26T00:27:29.054 INFO:tasks.rados.rados.0.smithi045.stdout:1232: expect deleted 2024-04-26T00:27:29.054 INFO:tasks.rados.rados.0.smithi045.stdout:1233: read oid 39 snap -1 2024-04-26T00:27:29.054 INFO:tasks.rados.rados.0.smithi045.stdout:1233: expect (ObjNum 329 snap 96 seq_num 329) 2024-04-26T00:27:29.055 INFO:tasks.rados.rados.0.smithi045.stdout:1234: snap_remove snap 125 2024-04-26T00:27:29.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.495+0000 7f009affd640 10 monclient: tick 2024-04-26T00:27:29.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.495+0000 7f009affd640 10 monclient: _check_auth_tickets 2024-04-26T00:27:29.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.495+0000 7f009affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:27:29.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.495+0000 7f009affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:27:29.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.511+0000 7f00a2ffd640 1 --2- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a413c500 0x7f00a41361a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:29.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.511+0000 7f00a2ffd640 10 monclient: get_auth_request con 0x7f00a413c500 auth_method 0 2024-04-26T00:27:29.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.511+0000 7f00a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:29.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.511+0000 7f00a2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:29.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.511+0000 7f00a2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:27:29.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.511+0000 7f00a2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:29.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.511+0000 7f00a2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:29.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.511+0000 7f00a2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:29.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.511+0000 7f00a2ffd640 10 monclient(hunting): handle_auth_done global_id 94098 payload 824 2024-04-26T00:27:29.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.511+0000 7f00a2ffd640 1 -- 172.21.15.37:0/3210261000 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f00a404f510 con 0x7f00a413c500 2024-04-26T00:27:29.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.511+0000 7f00a2ffd640 1 -- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a413c500 msgr2=0x7f00a41361a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:29.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.511+0000 7f00a2ffd640 1 --2- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a413c500 0x7f00a41361a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f00a41373b0 tx=0x7f008c017e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:29.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.515+0000 7f009bfff640 1 -- 172.21.15.37:0/3210261000 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008c013bf0 con 0x7f00a413c500 2024-04-26T00:27:29.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.515+0000 7f009bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:29.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.515+0000 7f009bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:27:29.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.515+0000 7f009bfff640 10 monclient: dump: 2024-04-26T00:27:29.502 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:29.502 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:29.502 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:29.502 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:29.503 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:29.503 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:29.503 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:29.503 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:29.503 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:29.503 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:29.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.515+0000 7f009bfff640 1 -- 172.21.15.37:0/3210261000 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f008c013bf0 con 0x7f00a413c500 2024-04-26T00:27:29.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.515+0000 7f009bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:29.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.515+0000 7f009bfff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:27:29.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.515+0000 7f009bfff640 1 -- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a413c500 msgr2=0x7f00a41361a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:29.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.515+0000 7f009bfff640 1 --2- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a413c500 0x7f00a41361a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f00a41373b0 tx=0x7f008c017e10 comp rx=0 tx=0).stop 2024-04-26T00:27:29.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.515+0000 7f009bfff640 1 -- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a413c500 msgr2=0x7f00a41361a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:27:29.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.515+0000 7f009bfff640 1 --2- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a413c500 0x7f00a41361a0 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f00a41373b0 tx=0x7f008c017e10 comp rx=0 tx=0).stop 2024-04-26T00:27:29.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.523+0000 7f00a9bd2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:27:29.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.523+0000 7f00a9bd2640 1 --2- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a41366e0 0x7f00a40705b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:29.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.523+0000 7f00a9bd2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:29.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.523+0000 7f00a9bd2640 1 -- 172.21.15.37:0/3210261000 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f00a4136af0 con 0x7f00a41366e0 2024-04-26T00:27:29.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.523+0000 7f00a2ffd640 1 --2- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a41366e0 0x7f00a40705b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:29.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.523+0000 7f00a2ffd640 10 monclient: get_auth_request con 0x7f00a41366e0 auth_method 0 2024-04-26T00:27:29.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.523+0000 7f00a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:29.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.523+0000 7f00a2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:29.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.523+0000 7f00a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:29.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.523+0000 7f00a2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:29.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.523+0000 7f00a2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:29.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.523+0000 7f00a2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:29.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.523+0000 7f00a2ffd640 10 monclient(hunting): handle_auth_done global_id 94101 payload 824 2024-04-26T00:27:29.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.523+0000 7f00a2ffd640 1 -- 172.21.15.37:0/3210261000 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f00a409eb40 con 0x7f00a41366e0 2024-04-26T00:27:29.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.531+0000 7f00a2ffd640 1 -- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a41366e0 msgr2=0x7f00a40705b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:29.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.531+0000 7f00a2ffd640 1 --2- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a41366e0 0x7f00a40705b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f008c020000 tx=0x7f008c0046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:29.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.531+0000 7f009bfff640 1 -- 172.21.15.37:0/3210261000 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008c003a30 con 0x7f00a41366e0 2024-04-26T00:27:29.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.531+0000 7f009bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:29.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.531+0000 7f009bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:27:29.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.531+0000 7f009bfff640 10 monclient: dump: 2024-04-26T00:27:29.516 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:29.516 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:29.516 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:29.516 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:29.516 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:29.516 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:29.516 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:29.516 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:29.516 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:29.516 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:29.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.531+0000 7f009bfff640 1 -- 172.21.15.37:0/3210261000 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7f008c003a30 con 0x7f00a41366e0 2024-04-26T00:27:29.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.531+0000 7f009bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:27:29.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.531+0000 7f009bfff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:27:29.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.531+0000 7f009bfff640 1 -- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a41366e0 msgr2=0x7f00a40705b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:29.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.531+0000 7f009bfff640 1 --2- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a41366e0 0x7f00a40705b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f008c020000 tx=0x7f008c0046d0 comp rx=0 tx=0).stop 2024-04-26T00:27:29.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.531+0000 7f009bfff640 1 -- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a41366e0 msgr2=0x7f00a40705b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:27:29.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.531+0000 7f009bfff640 1 --2- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00a41366e0 0x7f00a40705b0 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f008c020000 tx=0x7f008c0046d0 comp rx=0 tx=0).stop 2024-04-26T00:27:29.517 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:27:29.517 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:27:29.517 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:27:29.517 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:27:29.518 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 58, 2024-04-26T00:27:29.518 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 10291, 2024-04-26T00:27:29.518 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:27:29.518 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:27:29.518 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:27:29.518 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:27:29.518 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:29.518 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 3, 2024-04-26T00:27:29.518 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:27:29.519 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:27:29.519 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:27:29.519 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:27:29.519 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:27:29.519 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:27:29.519 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:27:29.519 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:27:29.519 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:27:29.519 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:27:29.519 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:27:29.519 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:27:29.519 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:27:29.519 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:27:29.519 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:29.520 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:27:29.520 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:27:29.520 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:27:29.520 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:27:29.520 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:27:29.520 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:27:29.520 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:27:29.520 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:27:29.520 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:27:29.520 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:27:29.520 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:27:29.520 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:27:29.520 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:27:29.520 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:29.521 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:29.521 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:27:29.521 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:27:29.521 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:27:29.521 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:27:29.521 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:27:29.521 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:27:29.521 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:27:29.521 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:27:29.521 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:27:29.521 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:27:29.521 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:27:29.521 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:27:29.521 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:27:29.522 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:27:29.522 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:27:29.522 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:27:29.522 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:27:29.522 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:27:29.522 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:27:29.522 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:27:29.522 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:27:29.522 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:27:29.522 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:27:29.522 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:27:29.522 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:27:29.522 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:27:29.522 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:27:29.523 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:27:29.523 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:29.523 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:27:29.523 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:29.523 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:27:29.523 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:29.523 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:27:29.523 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:27:29.523 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:27:29.523 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:27:29.523 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:29.523 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:27:29.523 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:27:29.523 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:29.523 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:29.524 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:29.524 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:27:29.524 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:27:29.524 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:29.524 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:29.524 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:29.524 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:29.524 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:27:29.524 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:27:29.524 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:27:29.524 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:27:29.524 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:27:29.524 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:29.524 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:29.524 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:27:29.525 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:27:29.525 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:27:29.525 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:27:29.525 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:29.525 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:27:29.525 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:27:29.525 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:29.525 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:29.525 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:29.525 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:27:29.525 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:27:29.525 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:29.525 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:29.525 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:29.526 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:29.526 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:27:29.526 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:27:29.526 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:27:29.526 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:27:29.526 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:27:29.526 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:29.526 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:29.526 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:27:29.526 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:27:29.526 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:27:29.526 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:27:29.526 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:29.527 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:27:29.527 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:27:29.527 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:29.527 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:29.527 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:29.527 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:27:29.527 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:27:29.527 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:29.527 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:29.527 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:29.527 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:29.527 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:27:29.527 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:27:29.527 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:27:29.527 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:27:29.528 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:27:29.528 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:29.528 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:29.528 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:29.528 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:27:29.528 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:27:29.528 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:29.528 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:27:29.528 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:27:29.528 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:27:29.528 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:29.528 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:29.528 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:27:29.529 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:29.529 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:27:29.529 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:27:29.529 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:27:29.529 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:29.529 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:29.529 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:29.529 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:27:29.529 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:27:29.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.531+0000 7f00a9bd2640 1 -- 172.21.15.37:0/3210261000 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0084035ed0 msgr2=0x7f0084038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:29.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.531+0000 7f00a9bd2640 1 --2- 172.21.15.37:0/3210261000 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0084035ed0 0x7f0084038390 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f00a413d050 tx=0x7f0094009040 comp rx=0 tx=0).stop 2024-04-26T00:27:29.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.531+0000 7f00a9bd2640 10 monclient: shutdown 2024-04-26T00:27:29.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.531+0000 7f00a9bd2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:27:29.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.531+0000 7f00a9bd2640 1 -- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f00a40a0300 msgr2=0x7f00a41cd020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:29.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.531+0000 7f00a9bd2640 1 --2- 172.21.15.37:0/3210261000 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f00a40a0300 0x7f00a41cd020 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f009c0b0040 tx=0x7f009c019620 comp rx=0 tx=0).stop 2024-04-26T00:27:29.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.531+0000 7f00a3fff640 1 -- 172.21.15.37:0/3210261000 reap_dead start 2024-04-26T00:27:29.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.531+0000 7f00a3fff640 1 -- 172.21.15.37:0/3210261000 reap_dead start 2024-04-26T00:27:29.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.531+0000 7f00a9bd2640 1 -- 172.21.15.37:0/3210261000 shutdown_connections 2024-04-26T00:27:29.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.531+0000 7f00a9bd2640 1 -- 172.21.15.37:0/3210261000 >> 172.21.15.37:0/3210261000 conn(0x7f00a40a1d50 msgr2=0x7f00a4130bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:27:29.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.531+0000 7f00a9bd2640 1 -- 172.21.15.37:0/3210261000 shutdown_connections 2024-04-26T00:27:29.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.531+0000 7f00a9bd2640 1 -- 172.21.15.37:0/3210261000 wait complete. 2024-04-26T00:27:29.537 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:27:29.537 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:27:29.538 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:27:29.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.675+0000 7fc2cff6e640 10 monclient: get_monmap_and_config 2024-04-26T00:27:29.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.675+0000 7fc2cff6e640 10 monclient: build_initial_monmap 2024-04-26T00:27:29.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.675+0000 7fc2cff6e640 10 monclient: monmap: 2024-04-26T00:27:29.663 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:27:29.663 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:29.663 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:27:29.680603+0000 2024-04-26T00:27:29.663 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:27:29.680603+0000 2024-04-26T00:27:29.663 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:27:29.663 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:29.663 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:27:29.663 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:27:29.663 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:27:29.663 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:27:29.663 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:27:29.664 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:29.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.675+0000 7fc2cff6e640 1 Processor -- start 2024-04-26T00:27:29.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.675+0000 7fc2cff6e640 1 -- start start 2024-04-26T00:27:29.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.675+0000 7fc2cff6e640 10 monclient: init 2024-04-26T00:27:29.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.675+0000 7fc2cff6e640 10 monclient: _reopen_session rank -1 2024-04-26T00:27:29.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.675+0000 7fc2cff6e640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-04-26T00:27:29.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.675+0000 7fc2cff6e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2c80a0300 0x7fc2c809fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:29.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.675+0000 7fc2cff6e640 10 monclient(hunting): picked mon.noname-b con 0x7fc2c80a0300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:27:29.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.675+0000 7fc2cff6e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc2c80a06e0 addr v1:172.21.15.37:6789/0 2024-04-26T00:27:29.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.675+0000 7fc2cff6e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc2c81352b0 addr v1:172.21.15.45:6789/0 2024-04-26T00:27:29.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.675+0000 7fc2cff6e640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2c809c190 con 0x7fc2c80a06e0 2024-04-26T00:27:29.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.675+0000 7fc2cff6e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2c80a1110 con 0x7fc2c81352b0 2024-04-26T00:27:29.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.675+0000 7fc2cff6e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:29.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.675+0000 7fc2cff6e640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc2c80a13f0 con 0x7fc2c80a0300 2024-04-26T00:27:29.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.675+0000 7fc2cff6e640 10 monclient(hunting): _renew_subs 2024-04-26T00:27:29.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.675+0000 7fc2cff6e640 10 monclient(hunting): authenticate will time out at 1430.986466s 2024-04-26T00:27:29.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.675+0000 7fc2cdd0a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2c80a0300 0x7fc2c809fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:29.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.675+0000 7fc2cdd0a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2c80a0300 0x7fc2c809fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:34388/0 (socket says 172.21.15.37:34388) 2024-04-26T00:27:29.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.675+0000 7fc2cdd0a640 1 -- 172.21.15.37:0/3349937061 learned_addr learned my addr 172.21.15.37:0/3349937061 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:27:29.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.675+0000 7fc2cdd0a640 10 monclient(hunting): get_auth_request con 0x7fc2c80a0300 auth_method 0 2024-04-26T00:27:29.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.675+0000 7fc2cdd0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:29.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.675+0000 7fc2cdd0a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:29.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.675+0000 7fc2cdd0a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:29.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.675+0000 7fc2cd509640 1 --1- 172.21.15.37:0/3349937061 >> v1:172.21.15.37:6789/0 conn(0x7fc2c80a06e0 0x7fc2c8134ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:33996/0 (socket says 172.21.15.37:33996) 2024-04-26T00:27:29.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.675+0000 7fc2cdd0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:29.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.675+0000 7fc2cdd0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:29.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.675+0000 7fc2cdd0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:29.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.675+0000 7fc2cdd0a640 10 monclient(hunting): handle_auth_done global_id 84188 payload 274 2024-04-26T00:27:29.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.675+0000 7fc2cdd0a640 1 -- 172.21.15.37:0/3349937061 >> v1:172.21.15.45:6789/0 conn(0x7fc2c81352b0 legacy=0x7fc2c813cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:27:29.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.675+0000 7fc2cdd0a640 1 -- 172.21.15.37:0/3349937061 >> v1:172.21.15.37:6789/0 conn(0x7fc2c80a06e0 legacy=0x7fc2c8134ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:27:29.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.675+0000 7fc2cdd0a640 10 monclient: _finish_hunting 0 2024-04-26T00:27:29.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.675+0000 7fc2cdd0a640 1 monclient: found mon.noname-b 2024-04-26T00:27:29.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.675+0000 7fc2cdd0a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:27:29.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.675+0000 7fc2cdd0a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:27:29.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.675+0000 7fc2cdd0a640 1 -- 172.21.15.37:0/3349937061 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2c80a16d0 con 0x7fc2c80a0300 2024-04-26T00:27:29.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.679+0000 7fc2cdd0a640 1 -- 172.21.15.37:0/3349937061 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2c80a0300 msgr2=0x7fc2c809fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:29.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.679+0000 7fc2cdd0a640 1 --2- 172.21.15.37:0/3349937061 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2c80a0300 0x7fc2c809fc50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc2c40098a0 tx=0x7fc2c401c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc8d09e3ccd7a891 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:29.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.679+0000 7fc2ccd08640 1 -- 172.21.15.37:0/3349937061 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c40b2070 con 0x7fc2c80a0300 2024-04-26T00:27:29.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.679+0000 7fc2ccd08640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:29.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.679+0000 7fc2ccd08640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:27:29.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.679+0000 7fc2ccd08640 10 monclient: dump: 2024-04-26T00:27:29.669 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:29.669 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:29.670 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:29.670 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:29.670 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:29.670 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:29.670 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:29.670 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:29.670 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:29.670 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:29.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.679+0000 7fc2ccd08640 10 monclient: _finish_auth 0 2024-04-26T00:27:29.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.679+0000 7fc2ccd08640 10 monclient: _check_auth_tickets 2024-04-26T00:27:29.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.679+0000 7fc2ccd08640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:27:29.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.679+0000 7fc2ccd08640 1 -- 172.21.15.37:0/3349937061 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2c40a9070 con 0x7fc2c80a0300 2024-04-26T00:27:29.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.679+0000 7fc2cff6e640 5 monclient: authenticate success, global_id 84188 2024-04-26T00:27:29.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.679+0000 7fc2cff6e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:27:29.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.679+0000 7fc2ccd08640 10 monclient: handle_config config(0 keys) 2024-04-26T00:27:29.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.679+0000 7fc2cff6e640 10 monclient: get_monmap_and_config success 2024-04-26T00:27:29.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.679+0000 7fc2cff6e640 4 set_mon_vals no callback set 2024-04-26T00:27:29.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.679+0000 7fc2ccd08640 1 -- 172.21.15.37:0/3349937061 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c40ad880 con 0x7fc2c80a0300 2024-04-26T00:27:29.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.679+0000 7fc2ccd08640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:29.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.679+0000 7fc2ccd08640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:27:29.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.679+0000 7fc2ccd08640 10 monclient: dump: 2024-04-26T00:27:29.671 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:29.671 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:29.671 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:29.671 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:29.671 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:29.671 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:29.671 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:29.672 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:29.672 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:29.672 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:29.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.679+0000 7fc2cff6e640 10 monclient: shutdown 2024-04-26T00:27:29.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.679+0000 7fc2cff6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:27:29.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.679+0000 7fc2cff6e640 1 -- 172.21.15.37:0/3349937061 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2c80a0300 msgr2=0x7fc2c809fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:29.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.679+0000 7fc2cff6e640 1 --2- 172.21.15.37:0/3349937061 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2c80a0300 0x7fc2c809fc50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc2c40098a0 tx=0x7fc2c401c3c0 comp rx=0 tx=0).stop 2024-04-26T00:27:29.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cff6e640 1 -- 172.21.15.37:0/3349937061 shutdown_connections 2024-04-26T00:27:29.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cff6e640 1 --2- 172.21.15.37:0/3349937061 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2c80a0300 0x7fc2c809fc50 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:29.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cff6e640 1 -- 172.21.15.37:0/3349937061 >> 172.21.15.37:0/3349937061 conn(0x7fc2c80a1d50 msgr2=0x7fc2c80a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:27:29.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cff6e640 1 -- 172.21.15.37:0/3349937061 shutdown_connections 2024-04-26T00:27:29.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cff6e640 1 -- 172.21.15.37:0/3349937061 wait complete. 2024-04-26T00:27:29.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cff6e640 10 monclient: build_initial_monmap 2024-04-26T00:27:29.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cff6e640 10 monclient: monmap: 2024-04-26T00:27:29.678 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:27:29.678 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:27:29.678 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:27:29.678 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:27:29.678 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:27:29.678 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:27:29.678 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:27:29.678 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:27:29.678 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:27:29.678 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:29.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cff6e640 1 Processor -- start 2024-04-26T00:27:29.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cff6e640 1 -- start start 2024-04-26T00:27:29.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cff6e640 10 monclient: init 2024-04-26T00:27:29.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cff6e640 10 monclient: _reopen_session rank -1 2024-04-26T00:27:29.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cff6e640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:27:29.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cff6e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2c80a0300 0x7fc2c812ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:29.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cff6e640 10 monclient(hunting): picked mon.noname-c con 0x7fc2c80a0300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:27:29.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cff6e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2c80a06e0 0x7fc2c812d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:29.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cff6e640 10 monclient(hunting): picked mon.noname-b con 0x7fc2c80a06e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:27:29.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cff6e640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc2c81352b0 0x7fc2c8135c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:29.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cff6e640 10 monclient(hunting): picked mon.noname-a con 0x7fc2c81352b0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:27:29.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cff6e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:29.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cff6e640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c8140fa0 con 0x7fc2c81352b0 2024-04-26T00:27:29.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cff6e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:29.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cff6e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc2c8140e20 con 0x7fc2c80a06e0 2024-04-26T00:27:29.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cff6e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:29.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cff6e640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc2c8141120 con 0x7fc2c80a0300 2024-04-26T00:27:29.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cff6e640 10 monclient(hunting): _renew_subs 2024-04-26T00:27:29.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cff6e640 10 monclient(hunting): authenticate will time out at 1430.993725s 2024-04-26T00:27:29.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cdd0a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2c80a0300 0x7fc2c812ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:29.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cdd0a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2c80a0300 0x7fc2c812ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:34400/0 (socket says 172.21.15.37:34400) 2024-04-26T00:27:29.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cdd0a640 1 -- 172.21.15.37:0/2668729067 learned_addr learned my addr 172.21.15.37:0/2668729067 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:27:29.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cdd0a640 10 monclient(hunting): get_auth_request con 0x7fc2c80a0300 auth_method 0 2024-04-26T00:27:29.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2ce50b640 1 --2- 172.21.15.37:0/2668729067 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc2c81352b0 0x7fc2c8135c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:29.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cdd0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:29.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cdd0a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:29.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cdd0a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:29.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2ce50b640 10 monclient(hunting): get_auth_request con 0x7fc2c81352b0 auth_method 0 2024-04-26T00:27:29.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2ce50b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:29.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2ce50b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:29.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2ce50b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:29.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cdd0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:29.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cdd0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:29.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cdd0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:29.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2ce50b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:29.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2ce50b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:29.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2ce50b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:29.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cd509640 1 --2- 172.21.15.37:0/2668729067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2c80a06e0 0x7fc2c812d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:29.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cd509640 10 monclient(hunting): get_auth_request con 0x7fc2c80a06e0 auth_method 0 2024-04-26T00:27:29.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cd509640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:29.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cd509640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:29.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cd509640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:29.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cdd0a640 10 monclient(hunting): handle_auth_done global_id 84191 payload 824 2024-04-26T00:27:29.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cdd0a640 1 -- 172.21.15.37:0/2668729067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2c80a06e0 msgr2=0x7fc2c812d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:29.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cdd0a640 1 --2- 172.21.15.37:0/2668729067 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2c80a06e0 0x7fc2c812d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:29.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cdd0a640 1 -- 172.21.15.37:0/2668729067 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc2c81352b0 msgr2=0x7fc2c8135c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:29.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cdd0a640 1 --2- 172.21.15.37:0/2668729067 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc2c81352b0 0x7fc2c8135c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:29.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cdd0a640 10 monclient: _finish_hunting 0 2024-04-26T00:27:29.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cdd0a640 1 monclient: found mon.noname-c 2024-04-26T00:27:29.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cdd0a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:27:29.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cdd0a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:27:29.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.683+0000 7fc2cdd0a640 1 -- 172.21.15.37:0/2668729067 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2c81349e0 con 0x7fc2c80a0300 2024-04-26T00:27:29.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2cdd0a640 1 -- 172.21.15.37:0/2668729067 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2c80a0300 msgr2=0x7fc2c812ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:29.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2cdd0a640 1 --2- 172.21.15.37:0/2668729067 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2c80a0300 0x7fc2c812ce00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc2c401cc60 tx=0x7fc2c401ac00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:29.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2be7fc640 1 -- 172.21.15.37:0/2668729067 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c40bc020 con 0x7fc2c80a0300 2024-04-26T00:27:29.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2be7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:29.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2be7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:27:29.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2be7fc640 10 monclient: dump: 2024-04-26T00:27:29.683 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:29.683 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:29.683 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:29.683 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:29.683 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:29.683 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:29.683 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:29.683 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:29.683 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:29.683 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:29.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2be7fc640 10 monclient: _finish_auth 0 2024-04-26T00:27:29.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2be7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:27:29.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:27:29.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2be7fc640 1 -- 172.21.15.37:0/2668729067 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2c40b8420 con 0x7fc2c80a0300 2024-04-26T00:27:29.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2be7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:27:29.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2be7fc640 1 -- 172.21.15.37:0/2668729067 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c40b2040 con 0x7fc2c80a0300 2024-04-26T00:27:29.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2be7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:29.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2be7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:27:29.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2be7fc640 10 monclient: dump: 2024-04-26T00:27:29.684 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:29.684 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:29.684 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:29.684 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:29.684 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:29.684 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:29.684 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:29.685 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:29.685 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:29.685 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:29.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2cff6e640 5 monclient: authenticate success, global_id 84191 2024-04-26T00:27:29.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2cff6e640 10 monclient: _renew_subs 2024-04-26T00:27:29.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2bffff640 4 set_mon_vals no callback set 2024-04-26T00:27:29.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2cff6e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:27:29.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2cff6e640 1 -- 172.21.15.37:0/2668729067 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2c8137e50 con 0x7fc2c80a0300 2024-04-26T00:27:29.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2cff6e640 10 monclient: _renew_subs 2024-04-26T00:27:29.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2cff6e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:27:29.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2cff6e640 1 -- 172.21.15.37:0/2668729067 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2c8133fb0 con 0x7fc2c80a0300 2024-04-26T00:27:29.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2cff6e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:29.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2cff6e640 1 --2- 172.21.15.37:0/2668729067 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2c81390f0 0x7fc2c81361a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:29.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2cff6e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:29.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2cff6e640 1 -- 172.21.15.37:0/2668729067 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc2c806e560 con 0x7fc2c81390f0 2024-04-26T00:27:29.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2be7fc640 1 -- 172.21.15.37:0/2668729067 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc2c40bf020 con 0x7fc2c80a0300 2024-04-26T00:27:29.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2cd509640 1 --2- 172.21.15.37:0/2668729067 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2c81390f0 0x7fc2c81361a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:29.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2cd509640 10 monclient: get_auth_request con 0x7fc2c81390f0 auth_method 0 2024-04-26T00:27:29.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2cd509640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:29.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2cd509640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:29.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2cd509640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:29.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2cd509640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:29.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2cd509640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:29.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2cd509640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:29.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2be7fc640 1 --2- 172.21.15.37:0/2668729067 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc2ac035eb0 0x7fc2ac038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:29.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2be7fc640 1 -- 172.21.15.37:0/2668729067 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(295..295 src has 177..295) ==== 3995+0+0 (secure 0 0 0) 0x7fc2c40e2ed0 con 0x7fc2c80a0300 2024-04-26T00:27:29.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2cd509640 10 monclient(hunting): handle_auth_done global_id 84194 payload 824 2024-04-26T00:27:29.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2cd509640 1 -- 172.21.15.37:0/2668729067 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc2c804f510 con 0x7fc2c81390f0 2024-04-26T00:27:29.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2ce50b640 1 --2- 172.21.15.37:0/2668729067 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc2ac035eb0 0x7fc2ac038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:29.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.687+0000 7fc2ce50b640 10 monclient: get_auth_request con 0x7fc2ac035eb0 auth_method 0 2024-04-26T00:27:29.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.691+0000 7fc2cd509640 1 -- 172.21.15.37:0/2668729067 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2c81390f0 msgr2=0x7fc2c81361a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:29.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.691+0000 7fc2cd509640 1 --2- 172.21.15.37:0/2668729067 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2c81390f0 0x7fc2c81361a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc2b800e4d0 tx=0x7fc2b800e990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:29.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.691+0000 7fc2be7fc640 1 -- 172.21.15.37:0/2668729067 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2b8004e30 con 0x7fc2c81390f0 2024-04-26T00:27:29.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.691+0000 7fc2be7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:29.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.691+0000 7fc2be7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:27:29.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.691+0000 7fc2be7fc640 10 monclient: dump: 2024-04-26T00:27:29.690 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:29.690 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:29.690 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:29.690 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:29.690 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:29.690 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:29.690 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:29.690 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:29.690 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:29.690 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:29.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.691+0000 7fc2be7fc640 1 -- 172.21.15.37:0/2668729067 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc2b8018070 con 0x7fc2c81390f0 2024-04-26T00:27:29.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.691+0000 7fc2be7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:29.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.691+0000 7fc2be7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:27:29.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.691+0000 7fc2be7fc640 1 -- 172.21.15.37:0/2668729067 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2c81390f0 msgr2=0x7fc2c81361a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:29.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.691+0000 7fc2be7fc640 1 --2- 172.21.15.37:0/2668729067 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2c81390f0 0x7fc2c81361a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc2b800e4d0 tx=0x7fc2b800e990 comp rx=0 tx=0).stop 2024-04-26T00:27:29.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.691+0000 7fc2be7fc640 1 -- 172.21.15.37:0/2668729067 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2c81390f0 msgr2=0x7fc2c81361a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:27:29.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.691+0000 7fc2be7fc640 1 --2- 172.21.15.37:0/2668729067 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2c81390f0 0x7fc2c81361a0 secure :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc2b800e4d0 tx=0x7fc2b800e990 comp rx=0 tx=0).stop 2024-04-26T00:27:29.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.691+0000 7fc2ce50b640 1 --2- 172.21.15.37:0/2668729067 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc2ac035eb0 0x7fc2ac038370 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fc2c0005600 tx=0x7fc2c0003d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:29.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.695+0000 7fc2cff6e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:27:29.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.695+0000 7fc2cff6e640 1 --2- 172.21.15.37:0/2668729067 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2c81366e0 0x7fc2c806fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:29.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.695+0000 7fc2cff6e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:29.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.695+0000 7fc2cff6e640 1 -- 172.21.15.37:0/2668729067 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc2c8136ac0 con 0x7fc2c81366e0 2024-04-26T00:27:29.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.695+0000 7fc2cd509640 1 --2- 172.21.15.37:0/2668729067 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2c81366e0 0x7fc2c806fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:29.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.695+0000 7fc2cd509640 10 monclient: get_auth_request con 0x7fc2c81366e0 auth_method 0 2024-04-26T00:27:29.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.695+0000 7fc2cd509640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:29.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.695+0000 7fc2cd509640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:29.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.695+0000 7fc2cd509640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:29.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.695+0000 7fc2cd509640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:29.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.695+0000 7fc2cd509640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:29.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.695+0000 7fc2cd509640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:29.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.695+0000 7fc2cd509640 10 monclient(hunting): handle_auth_done global_id 84197 payload 824 2024-04-26T00:27:29.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.695+0000 7fc2cd509640 1 -- 172.21.15.37:0/2668729067 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc2c809eb20 con 0x7fc2c81366e0 2024-04-26T00:27:29.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.699+0000 7fc2cd509640 1 -- 172.21.15.37:0/2668729067 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2c81366e0 msgr2=0x7fc2c806fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:29.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.699+0000 7fc2cd509640 1 --2- 172.21.15.37:0/2668729067 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2c81366e0 0x7fc2c806fff0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8009010 tx=0x7fc2b8004940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:29.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.699+0000 7fc2be7fc640 1 -- 172.21.15.37:0/2668729067 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2b80064e0 con 0x7fc2c81366e0 2024-04-26T00:27:29.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.699+0000 7fc2be7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:29.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.699+0000 7fc2be7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:27:29.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.699+0000 7fc2be7fc640 10 monclient: dump: 2024-04-26T00:27:29.692 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:29.692 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:29.692 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:29.693 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:29.693 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:29.693 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:29.693 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:29.693 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:29.693 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:29.693 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:29.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.699+0000 7fc2be7fc640 1 -- 172.21.15.37:0/2668729067 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4716 (secure 0 0 0) 0x7fc2b80064e0 con 0x7fc2c81366e0 2024-04-26T00:27:29.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.699+0000 7fc2be7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:27:29.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.699+0000 7fc2be7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:27:29.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.699+0000 7fc2be7fc640 1 -- 172.21.15.37:0/2668729067 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2c81366e0 msgr2=0x7fc2c806fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:29.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.699+0000 7fc2be7fc640 1 --2- 172.21.15.37:0/2668729067 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2c81366e0 0x7fc2c806fff0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8009010 tx=0x7fc2b8004940 comp rx=0 tx=0).stop 2024-04-26T00:27:29.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.699+0000 7fc2be7fc640 1 -- 172.21.15.37:0/2668729067 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2c81366e0 msgr2=0x7fc2c806fff0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:27:29.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.699+0000 7fc2be7fc640 1 --2- 172.21.15.37:0/2668729067 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2c81366e0 0x7fc2c806fff0 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8009010 tx=0x7fc2b8004940 comp rx=0 tx=0).stop 2024-04-26T00:27:29.694 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:27:29.694 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:27:29.694 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:27:29.694 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:27:29.694 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 58, 2024-04-26T00:27:29.694 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 54635, 2024-04-26T00:27:29.694 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:27:29.694 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:27:29.694 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:27:29.695 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:27:29.695 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:29.695 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 4, 2024-04-26T00:27:29.695 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:27:29.695 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:27:29.695 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:27:29.695 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:27:29.695 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:27:29.695 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:27:29.695 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:27:29.696 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:27:29.696 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:27:29.696 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:27:29.696 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:27:29.696 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:27:29.696 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:27:29.696 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:27:29.696 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:29.696 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:27:29.696 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:27:29.696 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:27:29.696 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:27:29.696 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:27:29.696 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:27:29.697 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:27:29.697 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:27:29.697 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:27:29.697 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:27:29.697 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:27:29.697 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:27:29.697 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:27:29.697 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:29.697 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:29.697 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:27:29.697 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:27:29.697 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:27:29.697 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:27:29.697 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:27:29.697 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:27:29.698 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:27:29.698 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:27:29.698 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:27:29.698 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:27:29.698 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:27:29.698 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:27:29.698 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:27:29.698 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:27:29.698 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:27:29.698 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:27:29.698 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:27:29.698 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:27:29.698 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:27:29.699 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:27:29.699 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:27:29.699 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:27:29.699 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:27:29.699 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:27:29.699 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:27:29.699 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:27:29.699 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:27:29.699 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:27:29.699 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:29.699 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:27:29.699 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:29.699 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:27:29.699 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:29.699 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:27:29.700 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:27:29.700 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:27:29.700 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:27:29.700 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:29.700 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:27:29.700 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:27:29.700 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:29.700 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:29.700 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:29.700 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:27:29.700 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:27:29.700 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:29.700 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:29.700 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:29.701 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:29.701 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:27:29.701 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:27:29.701 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:27:29.701 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:27:29.701 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:27:29.701 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:29.701 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:29.701 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:27:29.701 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:27:29.701 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:27:29.701 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:27:29.701 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:29.701 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:27:29.702 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:27:29.702 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:29.702 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:29.702 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:29.702 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:27:29.702 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:27:29.702 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:29.702 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:29.702 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:29.702 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:29.702 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:27:29.702 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:27:29.702 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:27:29.702 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:27:29.703 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:27:29.703 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:29.703 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:29.703 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:27:29.703 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:27:29.703 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:27:29.703 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:27:29.703 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:29.703 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:27:29.703 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:27:29.703 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:29.703 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:29.703 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:29.703 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:27:29.703 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:27:29.704 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:29.704 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:29.704 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:29.704 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:29.704 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:27:29.704 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:27:29.704 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:27:29.704 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:27:29.704 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:27:29.704 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:29.704 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:29.704 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:29.704 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:27:29.704 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:27:29.705 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:29.705 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:27:29.705 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:27:29.705 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:27:29.705 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:29.705 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:29.705 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:27:29.705 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:29.705 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:27:29.705 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:27:29.705 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:27:29.705 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:29.705 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:29.705 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:27:29.706 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:29.706 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:27:29.706 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:27:29.706 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:27:29.706 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:29.706 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:29.706 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:29.706 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:27:29.706 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:27:29.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.699+0000 7fc2cff6e640 1 -- 172.21.15.37:0/2668729067 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc2ac035eb0 msgr2=0x7fc2ac038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:29.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.699+0000 7fc2cff6e640 1 --2- 172.21.15.37:0/2668729067 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc2ac035eb0 0x7fc2ac038370 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fc2c0005600 tx=0x7fc2c0003d50 comp rx=0 tx=0).stop 2024-04-26T00:27:29.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.699+0000 7fc2cff6e640 10 monclient: shutdown 2024-04-26T00:27:29.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.699+0000 7fc2cff6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:27:29.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.699+0000 7fc2cff6e640 1 -- 172.21.15.37:0/2668729067 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2c80a0300 msgr2=0x7fc2c812ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:29.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.699+0000 7fc2cff6e640 1 --2- 172.21.15.37:0/2668729067 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2c80a0300 0x7fc2c812ce00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc2c401cc60 tx=0x7fc2c401ac00 comp rx=0 tx=0).stop 2024-04-26T00:27:29.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.699+0000 7fc2ce50b640 1 -- 172.21.15.37:0/2668729067 reap_dead start 2024-04-26T00:27:29.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.699+0000 7fc2ce50b640 1 -- 172.21.15.37:0/2668729067 reap_dead start 2024-04-26T00:27:29.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.699+0000 7fc2cff6e640 1 -- 172.21.15.37:0/2668729067 shutdown_connections 2024-04-26T00:27:29.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.699+0000 7fc2cff6e640 1 -- 172.21.15.37:0/2668729067 >> 172.21.15.37:0/2668729067 conn(0x7fc2c80a1d50 msgr2=0x7fc2c80a3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:27:29.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.699+0000 7fc2cff6e640 1 -- 172.21.15.37:0/2668729067 shutdown_connections 2024-04-26T00:27:29.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.699+0000 7fc2cff6e640 1 -- 172.21.15.37:0/2668729067 wait complete. 2024-04-26T00:27:29.709 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:27:29.709 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:27:29.709 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:27:29.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.843+0000 7f2e9cbcd640 10 monclient: get_monmap_and_config 2024-04-26T00:27:29.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.843+0000 7f2e9cbcd640 10 monclient: build_initial_monmap 2024-04-26T00:27:29.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.843+0000 7f2e9cbcd640 10 monclient: monmap: 2024-04-26T00:27:29.833 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:27:29.833 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:29.833 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:27:29.850996+0000 2024-04-26T00:27:29.833 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:27:29.850996+0000 2024-04-26T00:27:29.833 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:27:29.834 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:29.834 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:27:29.834 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:27:29.834 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:27:29.834 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:27:29.834 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:27:29.834 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:29.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.843+0000 7f2e9cbcd640 1 Processor -- start 2024-04-26T00:27:29.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e9cbcd640 1 -- start start 2024-04-26T00:27:29.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e9cbcd640 10 monclient: init 2024-04-26T00:27:29.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e9cbcd640 10 monclient: _reopen_session rank -1 2024-04-26T00:27:29.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e9cbcd640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-04-26T00:27:29.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e9cbcd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2e98136f20 addr v1:172.21.15.37:6789/0 2024-04-26T00:27:29.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e9cbcd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2e98137a10 0x7f2e9809bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:29.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e9cbcd640 10 monclient(hunting): picked mon.noname-c con 0x7f2e98137a10 addr v2:172.21.15.45:3300/0 2024-04-26T00:27:29.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e9cbcd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2e9813c450 0x7f2e9813e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:29.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e9cbcd640 10 monclient(hunting): picked mon.noname-b con 0x7f2e9813c450 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:27:29.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e9cbcd640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e980a50f0 con 0x7f2e98136f20 2024-04-26T00:27:29.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e9cbcd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:29.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e9cbcd640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f2e9809b330 con 0x7f2e98137a10 2024-04-26T00:27:29.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e9cbcd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:29.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e9cbcd640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2e9809b610 con 0x7f2e9813c450 2024-04-26T00:27:29.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e9cbcd640 10 monclient(hunting): _renew_subs 2024-04-26T00:27:29.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e9cbcd640 10 monclient(hunting): authenticate will time out at 1431.156840s 2024-04-26T00:27:29.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e9659c640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f2e98136f20 0x7f2e98137300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:44854/0 (socket says 172.21.15.37:44854) 2024-04-26T00:27:29.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e9659c640 1 -- 172.21.15.37:0/2641010957 learned_addr learned my addr 172.21.15.37:0/2641010957 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:27:29.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e96d9d640 1 --2- 172.21.15.37:0/2641010957 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2e9813c450 0x7f2e9813e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:29.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e96d9d640 10 monclient(hunting): get_auth_request con 0x7f2e9813c450 auth_method 0 2024-04-26T00:27:29.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e96d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:29.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e96d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:29.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e96d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:29.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e96d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:29.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e96d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:29.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e8dd9b640 1 --2- 172.21.15.37:0/2641010957 >> v2:172.21.15.45:3300/0 conn(0x7f2e98137a10 0x7f2e9809bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:29.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e96d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:29.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e8dd9b640 10 monclient(hunting): get_auth_request con 0x7f2e98137a10 auth_method 0 2024-04-26T00:27:29.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e8dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:29.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e8dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:29.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e8dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:29.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e96d9d640 10 monclient(hunting): handle_auth_done global_id 84200 payload 274 2024-04-26T00:27:29.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e96d9d640 1 -- 172.21.15.37:0/2641010957 >> v2:172.21.15.45:3300/0 conn(0x7f2e98137a10 msgr2=0x7f2e9809bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:27:29.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e96d9d640 1 --2- 172.21.15.37:0/2641010957 >> v2:172.21.15.45:3300/0 conn(0x7f2e98137a10 0x7f2e9809bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:29.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e96d9d640 1 -- 172.21.15.37:0/2641010957 >> v1:172.21.15.37:6789/0 conn(0x7f2e98136f20 legacy=0x7f2e98137300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:27:29.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e96d9d640 10 monclient: _finish_hunting 0 2024-04-26T00:27:29.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e96d9d640 1 monclient: found mon.noname-b 2024-04-26T00:27:29.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e96d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:27:29.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e96d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:27:29.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e96d9d640 1 -- 172.21.15.37:0/2641010957 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e9809b8f0 con 0x7f2e9813c450 2024-04-26T00:27:29.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.847+0000 7f2e8dd9b640 1 --2- 172.21.15.37:0/2641010957 >> v2:172.21.15.45:3300/0 conn(0x7f2e98137a10 0x7f2e9809bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:27:29.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.851+0000 7f2e96d9d640 1 -- 172.21.15.37:0/2641010957 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2e9813c450 msgr2=0x7f2e9813e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:29.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.851+0000 7f2e96d9d640 1 --2- 172.21.15.37:0/2641010957 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2e9813c450 0x7f2e9813e840 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2e90009d90 tx=0x7f2e9001c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9babfeaf830dd874 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:29.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.851+0000 7f2e95d9b640 1 -- 172.21.15.37:0/2641010957 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e900a6710 con 0x7f2e9813c450 2024-04-26T00:27:29.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.851+0000 7f2e95d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:29.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.851+0000 7f2e95d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:27:29.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.851+0000 7f2e95d9b640 10 monclient: dump: 2024-04-26T00:27:29.849 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:29.849 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:29.849 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:29.849 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:29.849 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:29.849 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:29.849 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:29.849 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:29.849 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:29.849 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:29.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.851+0000 7f2e95d9b640 10 monclient: _finish_auth 0 2024-04-26T00:27:29.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.851+0000 7f2e95d9b640 10 monclient: _check_auth_tickets 2024-04-26T00:27:29.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.851+0000 7f2e95d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:27:29.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.851+0000 7f2e95d9b640 1 -- 172.21.15.37:0/2641010957 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e900a68b0 con 0x7f2e9813c450 2024-04-26T00:27:29.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.851+0000 7f2e95d9b640 10 monclient: handle_config config(0 keys) 2024-04-26T00:27:29.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.851+0000 7f2e95d9b640 1 -- 172.21.15.37:0/2641010957 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e900a6c80 con 0x7f2e9813c450 2024-04-26T00:27:29.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.851+0000 7f2e9cbcd640 5 monclient: authenticate success, global_id 84200 2024-04-26T00:27:29.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.851+0000 7f2e9cbcd640 10 monclient: get_monmap_and_config success 2024-04-26T00:27:29.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.851+0000 7f2e9cbcd640 4 set_mon_vals no callback set 2024-04-26T00:27:29.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.851+0000 7f2e95d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:29.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.851+0000 7f2e95d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:27:29.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.851+0000 7f2e95d9b640 10 monclient: dump: 2024-04-26T00:27:29.850 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:29.851 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:29.851 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:29.851 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:29.851 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:29.851 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:29.851 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:29.851 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:29.851 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:29.851 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:29.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.851+0000 7f2e9cbcd640 10 monclient: shutdown 2024-04-26T00:27:29.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.851+0000 7f2e9cbcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:27:29.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.851+0000 7f2e9cbcd640 1 -- 172.21.15.37:0/2641010957 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2e9813c450 msgr2=0x7f2e9813e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:29.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.851+0000 7f2e9cbcd640 1 --2- 172.21.15.37:0/2641010957 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2e9813c450 0x7f2e9813e840 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2e90009d90 tx=0x7f2e9001c880 comp rx=0 tx=0).stop 2024-04-26T00:27:29.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.851+0000 7f2e9cbcd640 1 -- 172.21.15.37:0/2641010957 shutdown_connections 2024-04-26T00:27:29.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.851+0000 7f2e9cbcd640 1 --2- 172.21.15.37:0/2641010957 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2e9813c450 0x7f2e9813e840 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:29.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.851+0000 7f2e9cbcd640 1 --2- 172.21.15.37:0/2641010957 >> v2:172.21.15.45:3300/0 conn(0x7f2e98137a10 0x7f2e9809bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:29.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.851+0000 7f2e9cbcd640 1 -- 172.21.15.37:0/2641010957 >> 172.21.15.37:0/2641010957 conn(0x7f2e980a61b0 msgr2=0x7f2e980a65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:27:29.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.851+0000 7f2e9cbcd640 1 -- 172.21.15.37:0/2641010957 shutdown_connections 2024-04-26T00:27:29.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.851+0000 7f2e9cbcd640 1 -- 172.21.15.37:0/2641010957 wait complete. 2024-04-26T00:27:29.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.851+0000 7f2e9cbcd640 10 monclient: build_initial_monmap 2024-04-26T00:27:29.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.851+0000 7f2e9cbcd640 10 monclient: monmap: 2024-04-26T00:27:29.852 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:27:29.852 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:27:29.852 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:27:29.852 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:27:29.852 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:27:29.852 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:27:29.852 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:27:29.852 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:27:29.852 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:27:29.853 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:29.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e9cbcd640 1 Processor -- start 2024-04-26T00:27:29.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e9cbcd640 1 -- start start 2024-04-26T00:27:29.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e9cbcd640 10 monclient: init 2024-04-26T00:27:29.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e9cbcd640 10 monclient: _reopen_session rank -1 2024-04-26T00:27:29.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e9cbcd640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:27:29.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e9cbcd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2e98136f20 0x7f2e9813a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:29.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e9cbcd640 10 monclient(hunting): picked mon.noname-c con 0x7f2e98136f20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:27:29.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e9cbcd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2e98137a10 0x7f2e98138670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:29.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e9cbcd640 10 monclient(hunting): picked mon.noname-a con 0x7f2e98137a10 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:27:29.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e9cbcd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e9813c450 0x7f2e98135240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:29.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e9cbcd640 10 monclient(hunting): picked mon.noname-b con 0x7f2e9813c450 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:27:29.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e9cbcd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:29.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e9cbcd640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f2e98141930 con 0x7f2e98137a10 2024-04-26T00:27:29.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e9cbcd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:29.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e9cbcd640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2e981417b0 con 0x7f2e9813c450 2024-04-26T00:27:29.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e9cbcd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:29.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e9cbcd640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2e98141ab0 con 0x7f2e98136f20 2024-04-26T00:27:29.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e9cbcd640 10 monclient(hunting): _renew_subs 2024-04-26T00:27:29.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e9cbcd640 10 monclient(hunting): authenticate will time out at 1431.164830s 2024-04-26T00:27:29.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e9659c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2e98136f20 0x7f2e9813a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:29.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e8dd9b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2e98137a10 0x7f2e98138670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:29.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e9659c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2e98136f20 0x7f2e9813a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:49968/0 (socket says 172.21.15.37:49968) 2024-04-26T00:27:29.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e8dd9b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2e98137a10 0x7f2e98138670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50280/0 (socket says 172.21.15.37:50280) 2024-04-26T00:27:29.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e9659c640 1 -- 172.21.15.37:0/1756173652 learned_addr learned my addr 172.21.15.37:0/1756173652 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:27:29.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e9659c640 10 monclient(hunting): get_auth_request con 0x7f2e98136f20 auth_method 0 2024-04-26T00:27:29.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e9659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:29.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e9659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:29.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e9659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:29.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e8dd9b640 10 monclient(hunting): get_auth_request con 0x7f2e98137a10 auth_method 0 2024-04-26T00:27:29.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e8dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:29.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e8dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:29.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e8dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:29.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e9659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:29.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e9659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:29.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e9659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:29.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e8dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:29.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e8dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:29.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e96d9d640 1 --2- 172.21.15.37:0/1756173652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e9813c450 0x7f2e98135240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:29.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e8dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:29.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e96d9d640 10 monclient(hunting): get_auth_request con 0x7f2e9813c450 auth_method 0 2024-04-26T00:27:29.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e96d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:29.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e96d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:29.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e96d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:29.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e96d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:29.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e96d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:29.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e96d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:29.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e8dd9b640 10 monclient(hunting): handle_auth_done global_id 94107 payload 824 2024-04-26T00:27:29.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e8dd9b640 1 -- 172.21.15.37:0/1756173652 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2e98136f20 msgr2=0x7f2e9813a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:29.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e8dd9b640 1 --2- 172.21.15.37:0/1756173652 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2e98136f20 0x7f2e9813a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:29.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e8dd9b640 1 -- 172.21.15.37:0/1756173652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e9813c450 msgr2=0x7f2e98135240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:29.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e8dd9b640 1 --2- 172.21.15.37:0/1756173652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e9813c450 0x7f2e98135240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:29.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e8dd9b640 10 monclient: _finish_hunting 0 2024-04-26T00:27:29.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e8dd9b640 1 monclient: found mon.noname-a 2024-04-26T00:27:29.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e8dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:27:29.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e8dd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:27:29.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e8dd9b640 1 -- 172.21.15.37:0/1756173652 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e9813b6f0 con 0x7f2e98137a10 2024-04-26T00:27:29.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e9659c640 1 --2- 172.21.15.37:0/1756173652 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2e98136f20 0x7f2e9813a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:27:29.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e8dd9b640 1 -- 172.21.15.37:0/1756173652 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2e98137a10 msgr2=0x7f2e98138670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:29.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e8dd9b640 1 --2- 172.21.15.37:0/1756173652 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2e98137a10 0x7f2e98138670 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2e8000e7e0 tx=0x7f2e8000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:29.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e8f7fe640 1 -- 172.21.15.37:0/1756173652 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e80017070 con 0x7f2e98137a10 2024-04-26T00:27:29.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e8f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:29.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e8f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:27:29.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e8f7fe640 10 monclient: dump: 2024-04-26T00:27:29.857 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:29.857 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:29.857 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:29.857 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:29.857 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:29.857 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:29.857 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:29.857 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:29.857 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:29.857 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:29.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e8f7fe640 10 monclient: _finish_auth 0 2024-04-26T00:27:29.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e8f7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:27:29.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:27:29.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e9cbcd640 5 monclient: authenticate success, global_id 94107 2024-04-26T00:27:29.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e8f7fe640 1 -- 172.21.15.37:0/1756173652 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e8000c8d0 con 0x7f2e98137a10 2024-04-26T00:27:29.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.855+0000 7f2e8f7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:27:29.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.859+0000 7f2e9cbcd640 10 monclient: _renew_subs 2024-04-26T00:27:29.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.859+0000 7f2e8f7fe640 1 -- 172.21.15.37:0/1756173652 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e8000f040 con 0x7f2e98137a10 2024-04-26T00:27:29.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.859+0000 7f2e9cbcd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:27:29.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.859+0000 7f2e9cbcd640 1 -- 172.21.15.37:0/1756173652 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e98135f30 con 0x7f2e98137a10 2024-04-26T00:27:29.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.859+0000 7f2e8f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:29.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.859+0000 7f2e9559a640 4 set_mon_vals no callback set 2024-04-26T00:27:29.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.859+0000 7f2e8f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:27:29.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.859+0000 7f2e8f7fe640 10 monclient: dump: 2024-04-26T00:27:29.859 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:29.859 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:29.859 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:29.859 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:29.859 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:29.859 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:29.859 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:29.859 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:29.859 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:29.859 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:29.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.859+0000 7f2e9cbcd640 10 monclient: _renew_subs 2024-04-26T00:27:29.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.859+0000 7f2e9cbcd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:27:29.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.859+0000 7f2e9cbcd640 1 -- 172.21.15.37:0/1756173652 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e98139ca0 con 0x7f2e98137a10 2024-04-26T00:27:29.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.859+0000 7f2e9cbcd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:29.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.859+0000 7f2e9cbcd640 1 --2- 172.21.15.37:0/1756173652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e98135780 0x7f2e981cf750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:29.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.859+0000 7f2e9cbcd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:29.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.859+0000 7f2e9cbcd640 1 -- 172.21.15.37:0/1756173652 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2e98138bb0 con 0x7f2e98135780 2024-04-26T00:27:29.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.859+0000 7f2e8f7fe640 1 -- 172.21.15.37:0/1756173652 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2e8001a070 con 0x7f2e98137a10 2024-04-26T00:27:29.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.859+0000 7f2e9659c640 1 --2- 172.21.15.37:0/1756173652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e98135780 0x7f2e981cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:29.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.859+0000 7f2e8f7fe640 1 --2- 172.21.15.37:0/1756173652 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2e78035ed0 0x7f2e78038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:29.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.859+0000 7f2e9659c640 10 monclient: get_auth_request con 0x7f2e98135780 auth_method 0 2024-04-26T00:27:29.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.859+0000 7f2e9659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:29.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.859+0000 7f2e9659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:29.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.859+0000 7f2e9659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:29.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.859+0000 7f2e8f7fe640 1 -- 172.21.15.37:0/1756173652 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(295..295 src has 177..295) ==== 3995+0+0 (secure 0 0 0) 0x7f2e80048630 con 0x7f2e98137a10 2024-04-26T00:27:29.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.859+0000 7f2e9659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:29.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.859+0000 7f2e9659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:29.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.859+0000 7f2e9659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:29.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.859+0000 7f2e96d9d640 1 --2- 172.21.15.37:0/1756173652 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2e78035ed0 0x7f2e78038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:29.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.859+0000 7f2e96d9d640 10 monclient: get_auth_request con 0x7f2e78035ed0 auth_method 0 2024-04-26T00:27:29.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.859+0000 7f2e9659c640 10 monclient(hunting): handle_auth_done global_id 24691 payload 824 2024-04-26T00:27:29.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.859+0000 7f2e9659c640 1 -- 172.21.15.37:0/1756173652 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2e9813bae0 con 0x7f2e98135780 2024-04-26T00:27:29.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.863+0000 7f2e9659c640 1 -- 172.21.15.37:0/1756173652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e98135780 msgr2=0x7f2e981cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:29.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.863+0000 7f2e9659c640 1 --2- 172.21.15.37:0/1756173652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e98135780 0x7f2e981cf750 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2e880072a0 tx=0x7f2e88007760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:29.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.863+0000 7f2e96d9d640 1 --2- 172.21.15.37:0/1756173652 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2e78035ed0 0x7f2e78038390 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2e98068940 tx=0x7f2e900a7000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:29.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.863+0000 7f2e8f7fe640 1 -- 172.21.15.37:0/1756173652 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e8800fde0 con 0x7f2e98135780 2024-04-26T00:27:29.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.863+0000 7f2e8f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:29.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.863+0000 7f2e8f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:27:29.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.863+0000 7f2e8f7fe640 10 monclient: dump: 2024-04-26T00:27:29.864 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:29.864 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:29.864 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:29.864 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:29.864 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:29.864 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:29.864 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:29.864 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:29.864 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:29.864 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:29.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.863+0000 7f2e8f7fe640 1 -- 172.21.15.37:0/1756173652 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2e8800fde0 con 0x7f2e98135780 2024-04-26T00:27:29.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.863+0000 7f2e8f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:29.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.863+0000 7f2e8f7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:27:29.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.863+0000 7f2e8f7fe640 1 -- 172.21.15.37:0/1756173652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e98135780 msgr2=0x7f2e981cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:29.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.863+0000 7f2e8f7fe640 1 --2- 172.21.15.37:0/1756173652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e98135780 0x7f2e981cf750 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2e880072a0 tx=0x7f2e88007760 comp rx=0 tx=0).stop 2024-04-26T00:27:29.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.863+0000 7f2e8f7fe640 1 -- 172.21.15.37:0/1756173652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e98135780 msgr2=0x7f2e981cf750 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:27:29.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.863+0000 7f2e8f7fe640 1 --2- 172.21.15.37:0/1756173652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e98135780 0x7f2e981cf750 secure :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2e880072a0 tx=0x7f2e88007760 comp rx=0 tx=0).stop 2024-04-26T00:27:29.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.867+0000 7f2e9cbcd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:27:29.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.867+0000 7f2e9cbcd640 1 --2- 172.21.15.37:0/1756173652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e9802c000 0x7f2e9802c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:29.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.867+0000 7f2e9cbcd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:29.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.867+0000 7f2e9cbcd640 1 -- 172.21.15.37:0/1756173652 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2e9804f510 con 0x7f2e9802c000 2024-04-26T00:27:29.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.871+0000 7f2e9659c640 1 --2- 172.21.15.37:0/1756173652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e9802c000 0x7f2e9802c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:29.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.871+0000 7f2e9659c640 10 monclient: get_auth_request con 0x7f2e9802c000 auth_method 0 2024-04-26T00:27:29.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.871+0000 7f2e9659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:29.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.871+0000 7f2e9659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:29.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.871+0000 7f2e9659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:29.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.871+0000 7f2e9659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:29.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.871+0000 7f2e9659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:29.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.871+0000 7f2e9659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:29.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.871+0000 7f2e9659c640 10 monclient(hunting): handle_auth_done global_id 24694 payload 824 2024-04-26T00:27:29.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.871+0000 7f2e9659c640 1 -- 172.21.15.37:0/1756173652 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2e980a3910 con 0x7f2e9802c000 2024-04-26T00:27:29.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.871+0000 7f2e9659c640 1 -- 172.21.15.37:0/1756173652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e9802c000 msgr2=0x7f2e9802c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:29.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.871+0000 7f2e9659c640 1 --2- 172.21.15.37:0/1756173652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e9802c000 0x7f2e9802c410 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f2e88017000 tx=0x7f2e88004950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:29.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.871+0000 7f2e8f7fe640 1 -- 172.21.15.37:0/1756173652 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e8801ee40 con 0x7f2e9802c000 2024-04-26T00:27:29.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.871+0000 7f2e8f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:29.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.871+0000 7f2e8f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:27:29.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.871+0000 7f2e8f7fe640 10 monclient: dump: 2024-04-26T00:27:29.866 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:29.866 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:29.866 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:29.867 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:29.867 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:29.867 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:29.867 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:29.867 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:29.867 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:29.867 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:29.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.875+0000 7f2e8f7fe640 1 -- 172.21.15.37:0/1756173652 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4889 (secure 0 0 0) 0x7f2e88006240 con 0x7f2e9802c000 2024-04-26T00:27:29.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.875+0000 7f2e8f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:27:29.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.875+0000 7f2e8f7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:27:29.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.875+0000 7f2e8f7fe640 1 -- 172.21.15.37:0/1756173652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e9802c000 msgr2=0x7f2e9802c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:29.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.875+0000 7f2e8f7fe640 1 --2- 172.21.15.37:0/1756173652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e9802c000 0x7f2e9802c410 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f2e88017000 tx=0x7f2e88004950 comp rx=0 tx=0).stop 2024-04-26T00:27:29.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.875+0000 7f2e8f7fe640 1 -- 172.21.15.37:0/1756173652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e9802c000 msgr2=0x7f2e9802c410 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:27:29.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.875+0000 7f2e8f7fe640 1 --2- 172.21.15.37:0/1756173652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e9802c000 0x7f2e9802c410 secure :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f2e88017000 tx=0x7f2e88004950 comp rx=0 tx=0).stop 2024-04-26T00:27:29.868 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:27:29.868 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:27:29.868 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:27:29.868 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:27:29.868 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 58, 2024-04-26T00:27:29.868 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 358042, 2024-04-26T00:27:29.868 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:27:29.868 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:27:29.868 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:27:29.868 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:27:29.868 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:29.869 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 4, 2024-04-26T00:27:29.869 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:27:29.869 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:27:29.869 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:27:29.869 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:27:29.869 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:27:29.869 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:27:29.869 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:27:29.869 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:27:29.869 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:27:29.869 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:27:29.869 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:27:29.869 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:27:29.869 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:27:29.870 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:27:29.870 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:29.870 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:27:29.870 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:27:29.870 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:27:29.870 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:27:29.870 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:27:29.870 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:27:29.870 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:27:29.870 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:27:29.870 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:27:29.870 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:27:29.870 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:27:29.870 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:27:29.871 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:27:29.871 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:29.871 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:29.871 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:27:29.871 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:27:29.872 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:27:29.872 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:27:29.872 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:27:29.872 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:27:29.872 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:27:29.872 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:27:29.872 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:27:29.872 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:27:29.872 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:27:29.872 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:27:29.872 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:27:29.872 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:27:29.872 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:27:29.873 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:27:29.873 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:27:29.873 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:27:29.873 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:27:29.873 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:27:29.873 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:27:29.873 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:27:29.873 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:27:29.873 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:27:29.873 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:27:29.873 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:27:29.873 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:27:29.873 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:27:29.873 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:29.874 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:27:29.874 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:29.874 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:27:29.874 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:29.874 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:27:29.874 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:27:29.874 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:27:29.874 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:27:29.874 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:29.874 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:27:29.874 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:27:29.874 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:29.874 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:29.874 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:29.875 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:27:29.875 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:27:29.875 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:29.875 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:29.875 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:29.875 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:29.875 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:27:29.875 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:27:29.875 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:27:29.875 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:27:29.875 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:27:29.875 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:29.875 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:29.875 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:27:29.876 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:27:29.876 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:27:29.876 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:27:29.876 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:29.876 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:27:29.876 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:27:29.876 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:29.876 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:29.876 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:29.876 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:27:29.876 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:27:29.876 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:29.876 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:29.876 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:29.876 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:29.877 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:27:29.877 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:27:29.877 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:27:29.877 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:27:29.877 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:27:29.877 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:29.877 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:29.877 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:27:29.877 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:27:29.877 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:27:29.877 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:27:29.877 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:29.877 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:27:29.877 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:27:29.877 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:29.878 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:29.878 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:29.878 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:27:29.878 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:27:29.878 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:29.878 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:29.878 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:29.878 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:29.878 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:27:29.878 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:27:29.878 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:27:29.878 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:27:29.878 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:27:29.878 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:29.879 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:29.879 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:29.879 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:27:29.879 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:27:29.879 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:29.879 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:27:29.879 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:27:29.879 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:27:29.879 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:29.879 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:29.879 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:27:29.879 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:29.879 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:27:29.879 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:27:29.879 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:27:29.880 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:29.880 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:29.880 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:27:29.880 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:29.880 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:27:29.880 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:27:29.880 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:27:29.880 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:29.880 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:29.880 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:27:29.880 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:29.880 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:27:29.880 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:27:29.880 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:27:29.881 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:29.881 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:29.881 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:29.881 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:27:29.881 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:27:29.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.875+0000 7f2e9cbcd640 1 -- 172.21.15.37:0/1756173652 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2e78035ed0 msgr2=0x7f2e78038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:29.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.875+0000 7f2e9cbcd640 1 --2- 172.21.15.37:0/1756173652 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2e78035ed0 0x7f2e78038390 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2e98068940 tx=0x7f2e900a7000 comp rx=0 tx=0).stop 2024-04-26T00:27:29.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.875+0000 7f2e9cbcd640 10 monclient: shutdown 2024-04-26T00:27:29.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.875+0000 7f2e9cbcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:27:29.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.875+0000 7f2e9cbcd640 1 -- 172.21.15.37:0/1756173652 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2e98137a10 msgr2=0x7f2e98138670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:29.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.875+0000 7f2e9cbcd640 1 --2- 172.21.15.37:0/1756173652 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2e98137a10 0x7f2e98138670 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2e8000e7e0 tx=0x7f2e8000eca0 comp rx=0 tx=0).stop 2024-04-26T00:27:29.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.875+0000 7f2e96d9d640 1 -- 172.21.15.37:0/1756173652 reap_dead start 2024-04-26T00:27:29.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.875+0000 7f2e96d9d640 1 -- 172.21.15.37:0/1756173652 reap_dead start 2024-04-26T00:27:29.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.875+0000 7f2e9cbcd640 1 -- 172.21.15.37:0/1756173652 shutdown_connections 2024-04-26T00:27:29.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.875+0000 7f2e9cbcd640 1 -- 172.21.15.37:0/1756173652 >> 172.21.15.37:0/1756173652 conn(0x7f2e980a61b0 msgr2=0x7f2e980a81a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:27:29.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.875+0000 7f2e9cbcd640 1 -- 172.21.15.37:0/1756173652 shutdown_connections 2024-04-26T00:27:29.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:29.875+0000 7f2e9cbcd640 1 -- 172.21.15.37:0/1756173652 wait complete. 2024-04-26T00:27:29.883 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:27:29.883 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-26T00:27:29.883 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-26T00:27:29.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.007+0000 7fe303b4c640 10 monclient: get_monmap_and_config 2024-04-26T00:27:29.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.007+0000 7fe303b4c640 10 monclient: build_initial_monmap 2024-04-26T00:27:29.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.007+0000 7fe303b4c640 10 monclient: monmap: 2024-04-26T00:27:29.997 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:27:29.997 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:29.997 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:27:30.014805+0000 2024-04-26T00:27:29.997 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:27:30.014805+0000 2024-04-26T00:27:29.997 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:27:29.997 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:29.998 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:27:29.998 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:27:29.998 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:27:29.998 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:27:29.998 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:27:29.998 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:30.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.007+0000 7fe303b4c640 1 Processor -- start 2024-04-26T00:27:30.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.007+0000 7fe303b4c640 1 -- start start 2024-04-26T00:27:30.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.007+0000 7fe303b4c640 10 monclient: init 2024-04-26T00:27:30.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe303b4c640 10 monclient: _reopen_session rank -1 2024-04-26T00:27:30.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe303b4c640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-04-26T00:27:30.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe303b4c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe2fc0a0260 addr v1:172.21.15.45:6789/0 2024-04-26T00:27:30.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe303b4c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2fc137e40 0x7fe2fc13a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:30.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe303b4c640 10 monclient(hunting): picked mon.noname-b con 0x7fe2fc137e40 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:27:30.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe303b4c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe2fc13a770 0x7fe2fc13cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:30.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe303b4c640 10 monclient(hunting): picked mon.noname-c con 0x7fe2fc13a770 addr v2:172.21.15.45:3300/0 2024-04-26T00:27:30.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe303b4c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2fc0a1130 con 0x7fe2fc0a0260 2024-04-26T00:27:30.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe303b4c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:30.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe303b4c640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fe2fc0a0d50 con 0x7fe2fc13a770 2024-04-26T00:27:30.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe303b4c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:30.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe303b4c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe2fc09ec10 con 0x7fe2fc137e40 2024-04-26T00:27:30.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe303b4c640 10 monclient(hunting): _renew_subs 2024-04-26T00:27:30.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe303b4c640 10 monclient(hunting): authenticate will time out at 1431.320673s 2024-04-26T00:27:30.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe3010e7640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2fc137e40 0x7fe2fc13a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:30.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe3010e7640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2fc137e40 0x7fe2fc13a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:49970/0 (socket says 172.21.15.37:49970) 2024-04-26T00:27:30.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe3010e7640 1 -- 172.21.15.37:0/871717488 learned_addr learned my addr 172.21.15.37:0/871717488 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:27:30.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe3010e7640 10 monclient(hunting): get_auth_request con 0x7fe2fc137e40 auth_method 0 2024-04-26T00:27:30.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe3010e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:30.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe3010e7640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:30.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe3010e7640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:30.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe3020e9640 1 --2- 172.21.15.37:0/871717488 >> v2:172.21.15.45:3300/0 conn(0x7fe2fc13a770 0x7fe2fc13cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:30.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe3010e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:30.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe3010e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:30.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe3010e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:30.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe3020e9640 10 monclient(hunting): get_auth_request con 0x7fe2fc13a770 auth_method 0 2024-04-26T00:27:30.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe3020e9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:30.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe3020e9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:30.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe3020e9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:30.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe3020e9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:30.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe3020e9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:30.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe3020e9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:30.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe3010e7640 10 monclient(hunting): handle_auth_done global_id 84206 payload 274 2024-04-26T00:27:30.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe3010e7640 1 -- 172.21.15.37:0/871717488 >> v2:172.21.15.45:3300/0 conn(0x7fe2fc13a770 msgr2=0x7fe2fc13cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:27:30.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe3010e7640 1 --2- 172.21.15.37:0/871717488 >> v2:172.21.15.45:3300/0 conn(0x7fe2fc13a770 0x7fe2fc13cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:30.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe3010e7640 1 -- 172.21.15.37:0/871717488 >> v1:172.21.15.45:6789/0 conn(0x7fe2fc0a0260 legacy=0x7fe2fc0a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:27:30.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe3010e7640 10 monclient: _finish_hunting 0 2024-04-26T00:27:30.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe3010e7640 1 monclient: found mon.noname-b 2024-04-26T00:27:30.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe3010e7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:27:30.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe3010e7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:27:30.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.011+0000 7fe3010e7640 1 -- 172.21.15.37:0/871717488 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2fc06cff0 con 0x7fe2fc137e40 2024-04-26T00:27:30.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.015+0000 7fe3010e7640 1 -- 172.21.15.37:0/871717488 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2fc137e40 msgr2=0x7fe2fc13a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:30.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.015+0000 7fe3010e7640 1 --2- 172.21.15.37:0/871717488 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2fc137e40 0x7fe2fc13a230 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec009870 tx=0x7fe2ec01c200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34504a5f35451711 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:30.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.015+0000 7fe3008e6640 1 -- 172.21.15.37:0/871717488 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec0b0070 con 0x7fe2fc137e40 2024-04-26T00:27:30.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.015+0000 7fe3008e6640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:30.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.015+0000 7fe3008e6640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:27:30.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.015+0000 7fe3008e6640 10 monclient: dump: 2024-04-26T00:27:30.006 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:30.006 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:30.006 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:30.006 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:30.006 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:30.006 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:30.006 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:30.006 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:30.006 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:30.006 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:30.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.015+0000 7fe3008e6640 10 monclient: _finish_auth 0 2024-04-26T00:27:30.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.015+0000 7fe3008e6640 10 monclient: _check_auth_tickets 2024-04-26T00:27:30.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.015+0000 7fe3008e6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:27:30.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.015+0000 7fe303b4c640 5 monclient: authenticate success, global_id 84206 2024-04-26T00:27:30.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.015+0000 7fe3008e6640 1 -- 172.21.15.37:0/871717488 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2ec0ab600 con 0x7fe2fc137e40 2024-04-26T00:27:30.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.015+0000 7fe3008e6640 10 monclient: handle_config config(0 keys) 2024-04-26T00:27:30.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.015+0000 7fe303b4c640 10 monclient: get_monmap_and_config success 2024-04-26T00:27:30.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.015+0000 7fe3008e6640 1 -- 172.21.15.37:0/871717488 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec0ab9d0 con 0x7fe2fc137e40 2024-04-26T00:27:30.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.015+0000 7fe303b4c640 4 set_mon_vals no callback set 2024-04-26T00:27:30.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.015+0000 7fe3008e6640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:30.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.015+0000 7fe3008e6640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:27:30.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.015+0000 7fe3008e6640 10 monclient: dump: 2024-04-26T00:27:30.007 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:30.007 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:30.007 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:30.007 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:30.007 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:30.007 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:30.008 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:30.008 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:30.008 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:30.008 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:30.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.015+0000 7fe303b4c640 10 monclient: shutdown 2024-04-26T00:27:30.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.015+0000 7fe303b4c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:27:30.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.015+0000 7fe303b4c640 1 -- 172.21.15.37:0/871717488 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2fc137e40 msgr2=0x7fe2fc13a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:30.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.015+0000 7fe303b4c640 1 --2- 172.21.15.37:0/871717488 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2fc137e40 0x7fe2fc13a230 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec009870 tx=0x7fe2ec01c200 comp rx=0 tx=0).stop 2024-04-26T00:27:30.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.015+0000 7fe303b4c640 1 -- 172.21.15.37:0/871717488 shutdown_connections 2024-04-26T00:27:30.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.015+0000 7fe303b4c640 1 --2- 172.21.15.37:0/871717488 >> v2:172.21.15.45:3300/0 conn(0x7fe2fc13a770 0x7fe2fc13cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:30.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.015+0000 7fe303b4c640 1 --2- 172.21.15.37:0/871717488 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2fc137e40 0x7fe2fc13a230 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:30.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.015+0000 7fe303b4c640 1 -- 172.21.15.37:0/871717488 >> 172.21.15.37:0/871717488 conn(0x7fe2fc0a1ad0 msgr2=0x7fe2fc0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:27:30.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.015+0000 7fe303b4c640 1 -- 172.21.15.37:0/871717488 shutdown_connections 2024-04-26T00:27:30.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.015+0000 7fe303b4c640 1 -- 172.21.15.37:0/871717488 wait complete. 2024-04-26T00:27:30.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.015+0000 7fe303b4c640 10 monclient: build_initial_monmap 2024-04-26T00:27:30.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.015+0000 7fe303b4c640 10 monclient: monmap: 2024-04-26T00:27:30.009 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:27:30.009 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:27:30.009 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:27:30.009 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:27:30.009 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:27:30.009 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:27:30.009 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:27:30.009 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:27:30.009 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:27:30.009 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:30.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.015+0000 7fe303b4c640 1 Processor -- start 2024-04-26T00:27:30.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.015+0000 7fe303b4c640 1 -- start start 2024-04-26T00:27:30.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.015+0000 7fe303b4c640 10 monclient: init 2024-04-26T00:27:30.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.015+0000 7fe303b4c640 10 monclient: _reopen_session rank -1 2024-04-26T00:27:30.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.015+0000 7fe303b4c640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:27:30.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe303b4c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2fc0a0260 0x7fe2fc1cec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:30.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe303b4c640 10 monclient(hunting): picked mon.noname-a con 0x7fe2fc0a0260 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:27:30.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe303b4c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2fc137e40 0x7fe2fc134d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:30.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe303b4c640 10 monclient(hunting): picked mon.noname-b con 0x7fe2fc137e40 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:27:30.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe303b4c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2fc13a770 0x7fe2fc135260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:30.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe303b4c640 10 monclient(hunting): picked mon.noname-c con 0x7fe2fc13a770 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:27:30.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe303b4c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:30.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe303b4c640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe2fc09f4c0 con 0x7fe2fc0a0260 2024-04-26T00:27:30.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe303b4c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:30.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe303b4c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe2fc09f0f0 con 0x7fe2fc137e40 2024-04-26T00:27:30.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe303b4c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:30.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe303b4c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe2fc13f8c0 con 0x7fe2fc13a770 2024-04-26T00:27:30.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe303b4c640 10 monclient(hunting): _renew_subs 2024-04-26T00:27:30.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe303b4c640 10 monclient(hunting): authenticate will time out at 1431.328177s 2024-04-26T00:27:30.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe3018e8640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2fc0a0260 0x7fe2fc1cec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:30.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe3020e9640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2fc13a770 0x7fe2fc135260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:30.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe3020e9640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2fc13a770 0x7fe2fc135260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:49974/0 (socket says 172.21.15.37:49974) 2024-04-26T00:27:30.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe3018e8640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2fc0a0260 0x7fe2fc1cec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50292/0 (socket says 172.21.15.37:50292) 2024-04-26T00:27:30.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe3020e9640 1 -- 172.21.15.37:0/1299782527 learned_addr learned my addr 172.21.15.37:0/1299782527 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:27:30.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe3020e9640 10 monclient(hunting): get_auth_request con 0x7fe2fc13a770 auth_method 0 2024-04-26T00:27:30.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe3020e9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:30.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe3020e9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:30.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe3020e9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:30.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe3018e8640 10 monclient(hunting): get_auth_request con 0x7fe2fc0a0260 auth_method 0 2024-04-26T00:27:30.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe3018e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:30.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe3018e8640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:30.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe3018e8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:30.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe3018e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:30.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe3018e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:30.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe3018e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:30.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe3010e7640 1 --2- 172.21.15.37:0/1299782527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2fc137e40 0x7fe2fc134d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:30.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe3020e9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:30.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe3020e9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:30.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe3020e9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:30.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe3010e7640 10 monclient(hunting): get_auth_request con 0x7fe2fc137e40 auth_method 0 2024-04-26T00:27:30.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe3010e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:30.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe3010e7640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:30.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe3010e7640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:30.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe3018e8640 10 monclient(hunting): handle_auth_done global_id 94110 payload 824 2024-04-26T00:27:30.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe3018e8640 1 -- 172.21.15.37:0/1299782527 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2fc13a770 msgr2=0x7fe2fc135260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:30.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe3018e8640 1 --2- 172.21.15.37:0/1299782527 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2fc13a770 0x7fe2fc135260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:30.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe3018e8640 1 -- 172.21.15.37:0/1299782527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2fc137e40 msgr2=0x7fe2fc134d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:30.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe3018e8640 1 --2- 172.21.15.37:0/1299782527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2fc137e40 0x7fe2fc134d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:30.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe3018e8640 10 monclient: _finish_hunting 0 2024-04-26T00:27:30.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe3018e8640 1 monclient: found mon.noname-a 2024-04-26T00:27:30.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe3018e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:27:30.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe3018e8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:27:30.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe3018e8640 1 -- 172.21.15.37:0/1299782527 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2fc071bb0 con 0x7fe2fc0a0260 2024-04-26T00:27:30.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe3020e9640 1 --2- 172.21.15.37:0/1299782527 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2fc13a770 0x7fe2fc135260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:27:30.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe3018e8640 1 -- 172.21.15.37:0/1299782527 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2fc0a0260 msgr2=0x7fe2fc1cec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:30.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe3018e8640 1 --2- 172.21.15.37:0/1299782527 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2fc0a0260 0x7fe2fc1cec60 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe2f800c3e0 tx=0x7fe2f800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:30.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe2f27fc640 1 -- 172.21.15.37:0/1299782527 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f80119a0 con 0x7fe2fc0a0260 2024-04-26T00:27:30.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe2f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:30.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe2f27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:27:30.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe2f27fc640 10 monclient: dump: 2024-04-26T00:27:30.017 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:30.017 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:30.017 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:30.017 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:30.017 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:30.018 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:30.018 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:30.018 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:30.018 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:30.018 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:30.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe2f27fc640 10 monclient: _finish_auth 0 2024-04-26T00:27:30.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe2f27fc640 10 monclient: _check_auth_tickets 2024-04-26T00:27:30.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe2f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:27:30.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe2f27fc640 1 -- 172.21.15.37:0/1299782527 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2f8017070 con 0x7fe2fc0a0260 2024-04-26T00:27:30.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe303b4c640 5 monclient: authenticate success, global_id 94110 2024-04-26T00:27:30.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe303b4c640 10 monclient: _renew_subs 2024-04-26T00:27:30.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe303b4c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:27:30.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe303b4c640 1 -- 172.21.15.37:0/1299782527 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2fc1216f0 con 0x7fe2fc0a0260 2024-04-26T00:27:30.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe2f27fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:27:30.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe2f27fc640 1 -- 172.21.15.37:0/1299782527 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f8011d80 con 0x7fe2fc0a0260 2024-04-26T00:27:30.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe2f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:30.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe2f27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:27:30.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe2f27fc640 10 monclient: dump: 2024-04-26T00:27:30.019 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:30.019 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:30.019 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:30.019 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:30.019 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:30.019 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:30.019 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:30.019 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:30.019 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:30.019 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:30.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe2f3fff640 4 set_mon_vals no callback set 2024-04-26T00:27:30.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe303b4c640 10 monclient: _renew_subs 2024-04-26T00:27:30.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe303b4c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:27:30.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.019+0000 7fe303b4c640 1 -- 172.21.15.37:0/1299782527 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2fc06ff60 con 0x7fe2fc0a0260 2024-04-26T00:27:30.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.023+0000 7fe2f27fc640 1 -- 172.21.15.37:0/1299782527 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe2f801c460 con 0x7fe2fc0a0260 2024-04-26T00:27:30.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.023+0000 7fe2f27fc640 1 --2- 172.21.15.37:0/1299782527 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe2e0035ed0 0x7fe2e0038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:30.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.023+0000 7fe2f27fc640 1 -- 172.21.15.37:0/1299782527 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(295..295 src has 177..295) ==== 3995+0+0 (secure 0 0 0) 0x7fe2f8048870 con 0x7fe2fc0a0260 2024-04-26T00:27:30.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.023+0000 7fe303b4c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:30.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.023+0000 7fe303b4c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:30.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.023+0000 7fe303b4c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:27:30.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.023+0000 7fe303b4c640 1 -- 172.21.15.37:0/1299782527 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2fc066f40 con 0x7fe2fc0a0260 2024-04-26T00:27:30.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.023+0000 7fe3010e7640 1 --2- 172.21.15.37:0/1299782527 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe2e0035ed0 0x7fe2e0038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:30.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.023+0000 7fe3010e7640 10 monclient: get_auth_request con 0x7fe2e0035ed0 auth_method 0 2024-04-26T00:27:30.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.023+0000 7fe3010e7640 1 --2- 172.21.15.37:0/1299782527 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe2e0035ed0 0x7fe2e0038390 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc0a2ad0 tx=0x7fe2ec008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:30.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.027+0000 7fe2f27fc640 1 -- 172.21.15.37:0/1299782527 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe2f8012420 con 0x7fe2fc0a0260 2024-04-26T00:27:30.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.027+0000 7fe2f27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:30.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.027+0000 7fe2f27fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:27:30.024 INFO:tasks.rados.rados.0.smithi045.stdout:1234: done (4 left) 2024-04-26T00:27:30.025 INFO:tasks.rados.rados.0.smithi045.stdout:1235: rollback oid 28 current snap is 145 2024-04-26T00:27:30.025 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 28 to 133 2024-04-26T00:27:30.025 INFO:tasks.rados.rados.0.smithi045.stdout:1230: finishing write tid 1 to smithi04526579-7 2024-04-26T00:27:30.025 INFO:tasks.rados.rados.0.smithi045.stdout:1230: finishing write tid 2 to smithi04526579-7 2024-04-26T00:27:30.025 INFO:tasks.rados.rados.0.smithi045.stdout:1230: finishing write tid 3 to smithi04526579-7 2024-04-26T00:27:30.025 INFO:tasks.rados.rados.0.smithi045.stdout:1230: finishing write tid 4 to smithi04526579-7 2024-04-26T00:27:30.025 INFO:tasks.rados.rados.0.smithi045.stdout:1230: finishing write tid 5 to smithi04526579-7 2024-04-26T00:27:30.025 INFO:tasks.rados.rados.0.smithi045.stdout:1230: oid 7 updating version 0 to 942 2024-04-26T00:27:30.025 INFO:tasks.rados.rados.0.smithi045.stdout:1230: oid 7 updating version 942 to 943 2024-04-26T00:27:30.025 INFO:tasks.rados.rados.0.smithi045.stdout:1230: oid 7 version 943 is already newer than 941 2024-04-26T00:27:30.025 INFO:tasks.rados.rados.0.smithi045.stdout:1230: oid 7 version 943 is already newer than 940 2024-04-26T00:27:30.025 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 943 (ObjNum 460 snap 145 seq_num 460) dirty exists 2024-04-26T00:27:30.025 INFO:tasks.rados.rados.0.smithi045.stdout:1230: left oid 7 (ObjNum 460 snap 145 seq_num 460) 2024-04-26T00:27:30.025 INFO:tasks.rados.rados.0.smithi045.stdout:1233: expect (ObjNum 329 snap 96 seq_num 329) 2024-04-26T00:27:30.151 INFO:tasks.rados.rados.0.smithi045.stdout:1231: expect (ObjNum 459 snap 143 seq_num 459) 2024-04-26T00:27:30.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.263+0000 7fe303b4c640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-26T00:27:30.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.263+0000 7fe303b4c640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-26T00:27:30.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.263+0000 7fe303b4c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:27:30.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.263+0000 7fe303b4c640 1 -- 172.21.15.37:0/1299782527 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fe2fc121090 con 0x7fe2fc0a0260 2024-04-26T00:27:30.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.267+0000 7fe2f27fc640 1 -- 172.21.15.37:0/1299782527 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fe2f801cbc0 con 0x7fe2fc0a0260 2024-04-26T00:27:30.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.267+0000 7fe2f27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-26T00:27:30.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.267+0000 7fe2f27fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:27:30.255 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.266+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.255 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.266+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=16,config=2,health=12,kv=9,logm=61} crc {auth=3939197676,config=692619927,health=1970285960,kv=789508982,logm=3920907016}) 2024-04-26T00:27:30.255 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.266+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=12,kv=10,logm=60} crc {auth=3939197676,config=692619927,health=1970285961,kv=1034996044,logm=2090161484}) 2024-04-26T00:27:30.255 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.266+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.255 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.266+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=16,config=2,health=12,kv=9,logm=61} crc {auth=3939197676,config=692619927,health=1970285960,kv=789508982,logm=3920907016}) 2024-04-26T00:27:30.255 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.266+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=16,config=2,health=11,kv=8,logm=63} crc {auth=3939197676,config=692619927,health=273366162,kv=914734275,logm=136259334}) 2024-04-26T00:27:30.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.267+0000 7fe303b4c640 1 -- 172.21.15.37:0/1299782527 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe2e0035ed0 msgr2=0x7fe2e0038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:30.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.267+0000 7fe303b4c640 1 --2- 172.21.15.37:0/1299782527 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe2e0035ed0 0x7fe2e0038390 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc0a2ad0 tx=0x7fe2ec008040 comp rx=0 tx=0).stop 2024-04-26T00:27:30.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.267+0000 7fe303b4c640 10 monclient: shutdown 2024-04-26T00:27:30.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.267+0000 7fe303b4c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:27:30.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.267+0000 7fe303b4c640 1 -- 172.21.15.37:0/1299782527 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2fc0a0260 msgr2=0x7fe2fc1cec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:30.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.267+0000 7fe303b4c640 1 --2- 172.21.15.37:0/1299782527 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2fc0a0260 0x7fe2fc1cec60 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe2f800c3e0 tx=0x7fe2f800c8a0 comp rx=0 tx=0).stop 2024-04-26T00:27:30.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.267+0000 7fe303b4c640 1 -- 172.21.15.37:0/1299782527 shutdown_connections 2024-04-26T00:27:30.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.267+0000 7fe303b4c640 1 --2- 172.21.15.37:0/1299782527 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe2e0035ed0 0x7fe2e0038390 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:30.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.267+0000 7fe303b4c640 1 --2- 172.21.15.37:0/1299782527 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2fc13a770 0x7fe2fc135260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:30.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.267+0000 7fe303b4c640 1 --2- 172.21.15.37:0/1299782527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2fc137e40 0x7fe2fc134d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:30.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.267+0000 7fe303b4c640 1 --2- 172.21.15.37:0/1299782527 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2fc0a0260 0x7fe2fc1cec60 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:30.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.267+0000 7fe303b4c640 1 -- 172.21.15.37:0/1299782527 >> 172.21.15.37:0/1299782527 conn(0x7fe2fc0a1ad0 msgr2=0x7fe2fc0a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:27:30.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.267+0000 7fe303b4c640 1 -- 172.21.15.37:0/1299782527 shutdown_connections 2024-04-26T00:27:30.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:30.267+0000 7fe303b4c640 1 -- 172.21.15.37:0/1299782527 wait complete. 2024-04-26T00:27:30.258 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.270+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.258 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.270+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3546971800}) 2024-04-26T00:27:30.258 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.270+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2822895529}) 2024-04-26T00:27:30.258 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.270+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.258 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.270+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3546971800}) 2024-04-26T00:27:30.258 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.270+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=664890190}) 2024-04-26T00:27:30.260 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.274+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.261 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.274+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3397208971}) 2024-04-26T00:27:30.261 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.274+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4002312401}) 2024-04-26T00:27:30.261 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.274+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.261 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.274+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3397208971}) 2024-04-26T00:27:30.261 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.274+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4225487826}) 2024-04-26T00:27:30.269 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.282+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.269 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.282+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2480866159}) 2024-04-26T00:27:30.269 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.282+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=767707441}) 2024-04-26T00:27:30.269 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.282+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.269 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.282+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2480866159}) 2024-04-26T00:27:30.269 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.282+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3957395946}) 2024-04-26T00:27:30.271 INFO:tasks.rados.rados.0.smithi045.stdout:1235: finishing rollback tid 0 to smithi04526579-28 2024-04-26T00:27:30.271 INFO:tasks.rados.rados.0.smithi045.stdout:1235: finishing rollback tid 1 to smithi04526579-28 2024-04-26T00:27:30.271 INFO:tasks.rados.rados.0.smithi045.stdout:1235: finishing rollback tid 2 to smithi04526579-28 2024-04-26T00:27:30.271 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 817 (ObjNum 376 snap 115 seq_num 376) dirty exists 2024-04-26T00:27:30.271 INFO:tasks.rados.rados.0.smithi045.stdout:1230: done (4 left) 2024-04-26T00:27:30.271 INFO:tasks.rados.rados.0.smithi045.stdout:1231: done (3 left) 2024-04-26T00:27:30.271 INFO:tasks.rados.rados.0.smithi045.stdout:1232: done (2 left) 2024-04-26T00:27:30.271 INFO:tasks.rados.rados.0.smithi045.stdout:1233: done (1 left) 2024-04-26T00:27:30.271 INFO:tasks.rados.rados.0.smithi045.stdout:1235: done (0 left) 2024-04-26T00:27:30.271 INFO:tasks.rados.rados.0.smithi045.stdout:1236: copy_from oid 47 from oid 39 current snap is 145 2024-04-26T00:27:30.271 INFO:tasks.rados.rados.0.smithi045.stdout:1237: delete oid 1 current snap is 145 2024-04-26T00:27:30.272 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.282+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.272 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.282+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1583407887}) 2024-04-26T00:27:30.272 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.282+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1444436912}) 2024-04-26T00:27:30.272 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.282+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.272 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.282+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1583407887}) 2024-04-26T00:27:30.273 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.282+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2910440606}) 2024-04-26T00:27:30.273 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.286+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.273 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.286+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=268518372}) 2024-04-26T00:27:30.273 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.286+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2819724300}) 2024-04-26T00:27:30.274 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.286+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.274 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.286+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=268518372}) 2024-04-26T00:27:30.274 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.286+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=588900958}) 2024-04-26T00:27:30.275 INFO:tasks.rados.rados.0.smithi045.stdout:1237: done (1 left) 2024-04-26T00:27:30.275 INFO:tasks.rados.rados.0.smithi045.stdout:1238: write (excl) oid 9 current snap is 145 2024-04-26T00:27:30.275 INFO:tasks.rados.rados.0.smithi045.stdout:1238: seq_num 462 ranges {564834=546218,1651975=520479,2720952=559317} 2024-04-26T00:27:30.275 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-26T00:27:30.276 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.286+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.276 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.286+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3314255097}) 2024-04-26T00:27:30.276 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.286+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=427586882}) 2024-04-26T00:27:30.276 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.286+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.277 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.286+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3314255097}) 2024-04-26T00:27:30.277 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.286+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=774707467}) 2024-04-26T00:27:30.278 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.290+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.279 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.290+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4220671860}) 2024-04-26T00:27:30.279 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.290+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2463780038}) 2024-04-26T00:27:30.279 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.290+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.279 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.290+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4220671860}) 2024-04-26T00:27:30.279 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.290+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=395186462}) 2024-04-26T00:27:30.281 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.294+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.281 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.294+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=35030713}) 2024-04-26T00:27:30.281 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.294+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2766093321}) 2024-04-26T00:27:30.281 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.294+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.281 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.294+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=35030713}) 2024-04-26T00:27:30.281 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.294+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1465065344}) 2024-04-26T00:27:30.283 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.294+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.283 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.294+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=877079977}) 2024-04-26T00:27:30.283 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.294+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=393956482}) 2024-04-26T00:27:30.283 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.294+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.283 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.294+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=877079977}) 2024-04-26T00:27:30.283 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.294+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2866020735}) 2024-04-26T00:27:30.285 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.298+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.285 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.298+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1101788301}) 2024-04-26T00:27:30.285 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.298+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2601674114}) 2024-04-26T00:27:30.285 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.298+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.285 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.298+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1101788301}) 2024-04-26T00:27:30.285 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.298+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1263766988}) 2024-04-26T00:27:30.287 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.298+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.287 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.298+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3745868096}) 2024-04-26T00:27:30.287 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.298+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2399339507}) 2024-04-26T00:27:30.287 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.298+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.288 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.298+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3745868096}) 2024-04-26T00:27:30.288 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.298+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=270724176}) 2024-04-26T00:27:30.289 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.302+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.289 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.302+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1824362714}) 2024-04-26T00:27:30.289 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.302+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=796609324}) 2024-04-26T00:27:30.290 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.302+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.290 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.302+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1824362714}) 2024-04-26T00:27:30.290 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.302+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2577845793}) 2024-04-26T00:27:30.291 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.302+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.292 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.302+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=451563562}) 2024-04-26T00:27:30.292 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.302+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3640314888}) 2024-04-26T00:27:30.292 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.302+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.292 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.302+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=451563562}) 2024-04-26T00:27:30.292 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.302+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1355449394}) 2024-04-26T00:27:30.294 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.306+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.294 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.306+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=904270499}) 2024-04-26T00:27:30.294 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.306+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1642661694}) 2024-04-26T00:27:30.294 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.306+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.294 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.306+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=904270499}) 2024-04-26T00:27:30.294 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.306+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4191127780}) 2024-04-26T00:27:30.294 INFO:tasks.rados.rados.0.smithi045.stdout:1238: writing smithi04526579-9 from 564834 to 1111052 tid 1 2024-04-26T00:27:30.296 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.306+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.296 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.306+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1808390192}) 2024-04-26T00:27:30.296 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.306+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3500215062}) 2024-04-26T00:27:30.296 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.306+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.296 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.306+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1808390192}) 2024-04-26T00:27:30.296 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.306+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2660966629}) 2024-04-26T00:27:30.298 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.310+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.299 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.310+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4165247042}) 2024-04-26T00:27:30.299 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.310+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3011232534}) 2024-04-26T00:27:30.299 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.310+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.299 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.310+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4165247042}) 2024-04-26T00:27:30.299 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.310+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=374151090}) 2024-04-26T00:27:30.300 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.310+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.300 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.310+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3189306112}) 2024-04-26T00:27:30.300 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.310+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3301123108}) 2024-04-26T00:27:30.301 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.310+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.301 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.310+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3189306112}) 2024-04-26T00:27:30.301 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.310+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1306158271}) 2024-04-26T00:27:30.302 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.314+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.302 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.314+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2768538363}) 2024-04-26T00:27:30.302 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.314+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1261697784}) 2024-04-26T00:27:30.302 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.314+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.302 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.314+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2768538363}) 2024-04-26T00:27:30.302 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.314+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=132142519}) 2024-04-26T00:27:30.304 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.314+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.304 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.314+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3568848493}) 2024-04-26T00:27:30.304 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.314+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1062646379}) 2024-04-26T00:27:30.304 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.314+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.304 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.314+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3568848493}) 2024-04-26T00:27:30.304 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.314+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=246926794}) 2024-04-26T00:27:30.306 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.318+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.306 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.318+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=74,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=14} crc {logm=1701370121,mdsmap=2840952586,mgr=3019493264,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=3743573713}) 2024-04-26T00:27:30.306 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.318+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=75,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12} crc {logm=2278502308,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=3129004580}) 2024-04-26T00:27:30.306 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.318+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.306 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.318+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=74,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=14} crc {logm=1701370121,mdsmap=2840952586,mgr=3019493264,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=3743573713}) 2024-04-26T00:27:30.306 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.318+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=74,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13} crc {logm=1225854231,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=1257188647}) 2024-04-26T00:27:30.308 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.318+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.308 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.318+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=45,osdmap=34} crc {mon_config_key=4061063117,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=3035946785,osd_snap=3100578397,osdmap=1119893784}) 2024-04-26T00:27:30.308 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.318+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=41,osdmap=39} crc {mon_config_key=1720404874,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=3035946785,osd_snap=3500713054,osdmap=3618381587}) 2024-04-26T00:27:30.308 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.318+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.308 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.318+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=45,osdmap=34} crc {mon_config_key=4061063117,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=3035946785,osd_snap=3100578397,osdmap=1119893784}) 2024-04-26T00:27:30.308 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.322+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=42,osdmap=37} crc {mon_config_key=4061063117,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=3035946785,osd_snap=195643388,osdmap=3342621932}) 2024-04-26T00:27:30.310 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.322+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.310 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.322+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2525507757}) 2024-04-26T00:27:30.310 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.322+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1518400125}) 2024-04-26T00:27:30.310 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.322+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.311 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.322+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2525507757}) 2024-04-26T00:27:30.311 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.322+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4010584211}) 2024-04-26T00:27:30.312 INFO:tasks.rados.rados.0.smithi045.stdout:1238: writing smithi04526579-9 from 1651975 to 2172454 tid 2 2024-04-26T00:27:30.321 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.330+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.321 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.330+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=56} crc {osdmap=2349833064}) 2024-04-26T00:27:30.322 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.330+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=54} crc {osdmap=2736220209}) 2024-04-26T00:27:30.322 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.330+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:27:30.322 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.330+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=56} crc {osdmap=2349833064}) 2024-04-26T00:27:30.322 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:27:30.330+0000 7fc4c04ad640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=58} crc {osdmap=2474546678}) 2024-04-26T00:27:30.329 INFO:tasks.rados.rados.0.smithi045.stdout:1238: writing smithi04526579-9 from 2720952 to 3280269 tid 3 2024-04-26T00:27:30.329 INFO:tasks.rados.rados.0.smithi045.stdout:1239: write (excl) oid 34 current snap is 145 2024-04-26T00:27:30.329 INFO:tasks.rados.rados.0.smithi045.stdout:1239: seq_num 463 ranges {581641=621467,1994104=780490,3432022=349619} 2024-04-26T00:27:30.343 INFO:tasks.rados.rados.0.smithi045.stdout:1239: writing smithi04526579-34 from 581641 to 1203108 tid 1 2024-04-26T00:27:30.357 INFO:tasks.rados.rados.0.smithi045.stdout:1239: writing smithi04526579-34 from 1994104 to 2774594 tid 2 2024-04-26T00:27:30.366 INFO:tasks.rados.rados.0.smithi045.stdout:1239: writing smithi04526579-34 from 3432022 to 3781641 tid 3 2024-04-26T00:27:30.366 INFO:tasks.rados.rados.0.smithi045.stdout:1240: write oid 5 current snap is 145 2024-04-26T00:27:30.367 INFO:tasks.rados.rados.0.smithi045.stdout:1240: seq_num 464 ranges {598448=413069,1768939=524148,2775798=581861,3781488=16960} 2024-04-26T00:27:30.374 INFO:tasks.rados.rados.0.smithi045.stdout:1240: writing smithi04526579-5 from 598448 to 1011517 tid 1 2024-04-26T00:27:30.383 INFO:tasks.rados.rados.0.smithi045.stdout:1240: writing smithi04526579-5 from 1768939 to 2293087 tid 2 2024-04-26T00:27:30.390 INFO:tasks.rados.rados.0.smithi045.stdout:1240: writing smithi04526579-5 from 2775798 to 3357659 tid 3 2024-04-26T00:27:30.393 INFO:tasks.rados.rados.0.smithi045.stdout:1240: writing smithi04526579-5 from 3781488 to 3798448 tid 4 2024-04-26T00:27:30.393 INFO:tasks.rados.rados.0.smithi045.stdout:1241: read oid 10 snap -1 2024-04-26T00:27:30.393 INFO:tasks.rados.rados.0.smithi045.stdout:1241: expect (ObjNum 451 snap 140 seq_num 451) 2024-04-26T00:27:30.394 INFO:tasks.rados.rados.0.smithi045.stdout:1242: read oid 20 snap 137 2024-04-26T00:27:30.394 INFO:tasks.rados.rados.0.smithi045.stdout:1242: expect (ObjNum 356 snap 105 seq_num 356) 2024-04-26T00:27:30.394 INFO:tasks.rados.rados.0.smithi045.stdout:1243: read oid 42 snap -1 2024-04-26T00:27:30.394 INFO:tasks.rados.rados.0.smithi045.stdout:1243: expect (ObjNum 434 snap 136 seq_num 434) 2024-04-26T00:27:30.394 INFO:tasks.rados.rados.0.smithi045.stdout:1244: read oid 23 snap 143 2024-04-26T00:27:30.394 INFO:tasks.rados.rados.0.smithi045.stdout:1244: expect (ObjNum 453 snap 141 seq_num 453) 2024-04-26T00:27:30.394 INFO:tasks.rados.rados.0.smithi045.stdout:1245: delete oid 3 current snap is 145 2024-04-26T00:27:30.394 INFO:tasks.rados.rados.0.smithi045.stdout:1236: finishing copy_from racing read to smithi04526579-47 2024-04-26T00:27:30.394 INFO:tasks.rados.rados.0.smithi045.stdout:1236: finishing copy_from to smithi04526579-47 2024-04-26T00:27:30.394 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 253 (ObjNum 329 snap 96 seq_num 329) dirty exists 2024-04-26T00:27:30.394 INFO:tasks.rados.rados.0.smithi045.stdout:1238: finishing write tid 1 to smithi04526579-9 2024-04-26T00:27:30.394 INFO:tasks.rados.rados.0.smithi045.stdout:1238: finishing write tid 2 to smithi04526579-9 2024-04-26T00:27:30.395 INFO:tasks.rados.rados.0.smithi045.stdout:1238: finishing write tid 3 to smithi04526579-9 2024-04-26T00:27:30.395 INFO:tasks.rados.rados.0.smithi045.stdout:1238: finishing write tid 4 to smithi04526579-9 2024-04-26T00:27:30.395 INFO:tasks.rados.rados.0.smithi045.stdout:1238: finishing write tid 5 to smithi04526579-9 2024-04-26T00:27:30.395 INFO:tasks.rados.rados.0.smithi045.stdout:1238: oid 9 updating version 0 to 660 2024-04-26T00:27:30.395 INFO:tasks.rados.rados.0.smithi045.stdout:1238: oid 9 updating version 660 to 661 2024-04-26T00:27:30.395 INFO:tasks.rados.rados.0.smithi045.stdout:1238: oid 9 updating version 661 to 663 2024-04-26T00:27:30.395 INFO:tasks.rados.rados.0.smithi045.stdout:1238: oid 9 version 663 is already newer than 662 2024-04-26T00:27:30.395 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 663 (ObjNum 462 snap 145 seq_num 462) dirty exists 2024-04-26T00:27:30.395 INFO:tasks.rados.rados.0.smithi045.stdout:1238: left oid 9 (ObjNum 462 snap 145 seq_num 462) 2024-04-26T00:27:30.395 INFO:tasks.rados.rados.0.smithi045.stdout:1239: finishing write tid 1 to smithi04526579-34 2024-04-26T00:27:30.396 INFO:tasks.rados.rados.0.smithi045.stdout:1239: finishing write tid 2 to smithi04526579-34 2024-04-26T00:27:30.396 INFO:tasks.rados.rados.0.smithi045.stdout:1239: finishing write tid 3 to smithi04526579-34 2024-04-26T00:27:30.396 INFO:tasks.rados.rados.0.smithi045.stdout:1239: finishing write tid 4 to smithi04526579-34 2024-04-26T00:27:30.396 INFO:tasks.rados.rados.0.smithi045.stdout:1239: finishing write tid 5 to smithi04526579-34 2024-04-26T00:27:30.396 INFO:tasks.rados.rados.0.smithi045.stdout:1239: oid 34 updating version 0 to 735 2024-04-26T00:27:30.396 INFO:tasks.rados.rados.0.smithi045.stdout:1239: oid 34 version 735 is already newer than 734 2024-04-26T00:27:30.396 INFO:tasks.rados.rados.0.smithi045.stdout:1239: oid 34 updating version 735 to 737 2024-04-26T00:27:30.397 INFO:tasks.rados.rados.0.smithi045.stdout:1239: oid 34 version 737 is already newer than 736 2024-04-26T00:27:30.397 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 737 (ObjNum 463 snap 145 seq_num 463) dirty exists 2024-04-26T00:27:30.397 INFO:tasks.rados.rados.0.smithi045.stdout:1239: left oid 34 (ObjNum 463 snap 145 seq_num 463) 2024-04-26T00:27:30.397 INFO:tasks.rados.rados.0.smithi045.stdout:1240: finishing write tid 1 to smithi04526579-5 2024-04-26T00:27:30.397 INFO:tasks.rados.rados.0.smithi045.stdout:1240: finishing write tid 2 to smithi04526579-5 2024-04-26T00:27:30.400 INFO:tasks.rados.rados.0.smithi045.stdout:1236: done (8 left) 2024-04-26T00:27:30.400 INFO:tasks.rados.rados.0.smithi045.stdout:1238: done (7 left) 2024-04-26T00:27:30.400 INFO:tasks.rados.rados.0.smithi045.stdout:1239: done (6 left) 2024-04-26T00:27:30.400 INFO:tasks.rados.rados.0.smithi045.stdout:1245: done (5 left) 2024-04-26T00:27:30.401 INFO:tasks.rados.rados.0.smithi045.stdout:1246: write (excl) oid 31 current snap is 145 2024-04-26T00:27:30.401 INFO:tasks.rados.rados.0.smithi045.stdout:1246: seq_num 466 ranges {632062=563567,1885903=527817,2830644=720758} 2024-04-26T00:27:30.411 INFO:tasks.rados.rados.0.smithi045.stdout:1246: writing smithi04526579-31 from 632062 to 1195629 tid 1 2024-04-26T00:27:30.420 INFO:tasks.rados.rados.0.smithi045.stdout:1246: writing smithi04526579-31 from 1885903 to 2413720 tid 2 2024-04-26T00:27:30.429 INFO:tasks.rados.rados.0.smithi045.stdout:1246: writing smithi04526579-31 from 2830644 to 3551402 tid 3 2024-04-26T00:27:30.429 INFO:tasks.rados.rados.0.smithi045.stdout:1247: delete oid 17 current snap is 145 2024-04-26T00:27:30.429 INFO:tasks.rados.rados.0.smithi045.stdout:1240: finishing write tid 3 to smithi04526579-5 2024-04-26T00:27:30.429 INFO:tasks.rados.rados.0.smithi045.stdout:1240: finishing write tid 4 to smithi04526579-5 2024-04-26T00:27:30.429 INFO:tasks.rados.rados.0.smithi045.stdout:1240: finishing write tid 5 to smithi04526579-5 2024-04-26T00:27:30.429 INFO:tasks.rados.rados.0.smithi045.stdout:1240: finishing write tid 6 to smithi04526579-5 2024-04-26T00:27:30.429 INFO:tasks.rados.rados.0.smithi045.stdout:1240: oid 5 updating version 0 to 959 2024-04-26T00:27:30.430 INFO:tasks.rados.rados.0.smithi045.stdout:1240: oid 5 version 959 is already newer than 957 2024-04-26T00:27:30.430 INFO:tasks.rados.rados.0.smithi045.stdout:1240: oid 5 version 959 is already newer than 958 2024-04-26T00:27:30.430 INFO:tasks.rados.rados.0.smithi045.stdout:1240: oid 5 version 959 is already newer than 955 2024-04-26T00:27:30.430 INFO:tasks.rados.rados.0.smithi045.stdout:1240: oid 5 version 959 is already newer than 956 2024-04-26T00:27:30.430 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 959 (ObjNum 464 snap 145 seq_num 464) dirty exists 2024-04-26T00:27:30.430 INFO:tasks.rados.rados.0.smithi045.stdout:1240: left oid 5 (ObjNum 464 snap 145 seq_num 464) 2024-04-26T00:27:30.430 INFO:tasks.rados.rados.0.smithi045.stdout:1241: expect (ObjNum 451 snap 140 seq_num 451) 2024-04-26T00:27:30.739 INFO:tasks.rados.rados.0.smithi045.stdout:1242: expect (ObjNum 356 snap 105 seq_num 356) 2024-04-26T00:27:30.958 INFO:tasks.rados.rados.0.smithi045.stdout:1244: expect (ObjNum 453 snap 141 seq_num 453) 2024-04-26T00:27:31.074 INFO:tasks.rados.rados.0.smithi045.stdout:1243: expect (ObjNum 434 snap 136 seq_num 434) 2024-04-26T00:27:31.179 INFO:tasks.rados.rados.0.smithi045.stdout:1246: finishing write tid 1 to smithi04526579-31 2024-04-26T00:27:31.179 INFO:tasks.rados.rados.0.smithi045.stdout:1246: finishing write tid 2 to smithi04526579-31 2024-04-26T00:27:31.179 INFO:tasks.rados.rados.0.smithi045.stdout:1246: finishing write tid 3 to smithi04526579-31 2024-04-26T00:27:31.179 INFO:tasks.rados.rados.0.smithi045.stdout:1246: finishing write tid 4 to smithi04526579-31 2024-04-26T00:27:31.179 INFO:tasks.rados.rados.0.smithi045.stdout:1246: finishing write tid 5 to smithi04526579-31 2024-04-26T00:27:31.180 INFO:tasks.rados.rados.0.smithi045.stdout:1246: oid 31 updating version 0 to 666 2024-04-26T00:27:31.180 INFO:tasks.rados.rados.0.smithi045.stdout:1246: oid 31 updating version 666 to 668 2024-04-26T00:27:31.180 INFO:tasks.rados.rados.0.smithi045.stdout:1246: oid 31 version 668 is already newer than 665 2024-04-26T00:27:31.180 INFO:tasks.rados.rados.0.smithi045.stdout:1246: oid 31 version 668 is already newer than 667 2024-04-26T00:27:31.180 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 668 (ObjNum 466 snap 145 seq_num 466) dirty exists 2024-04-26T00:27:31.180 INFO:tasks.rados.rados.0.smithi045.stdout:1246: left oid 31 (ObjNum 466 snap 145 seq_num 466) 2024-04-26T00:27:31.180 INFO:tasks.rados.rados.0.smithi045.stdout:1240: done (6 left) 2024-04-26T00:27:31.180 INFO:tasks.rados.rados.0.smithi045.stdout:1241: done (5 left) 2024-04-26T00:27:31.180 INFO:tasks.rados.rados.0.smithi045.stdout:1242: done (4 left) 2024-04-26T00:27:31.180 INFO:tasks.rados.rados.0.smithi045.stdout:1243: done (3 left) 2024-04-26T00:27:31.180 INFO:tasks.rados.rados.0.smithi045.stdout:1244: done (2 left) 2024-04-26T00:27:31.180 INFO:tasks.rados.rados.0.smithi045.stdout:1246: done (1 left) 2024-04-26T00:27:31.180 INFO:tasks.rados.rados.0.smithi045.stdout:1247: done (0 left) 2024-04-26T00:27:31.180 INFO:tasks.rados.rados.0.smithi045.stdout:1248: snap_create 2024-04-26T00:27:31.276 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:27:31.276 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:27:31.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed38a01640 10 monclient: get_monmap_and_config 2024-04-26T00:27:31.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed38a01640 10 monclient: build_initial_monmap 2024-04-26T00:27:31.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed38a01640 10 monclient: monmap: 2024-04-26T00:27:31.408 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:27:31.408 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:31.408 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:27:31.425479+0000 2024-04-26T00:27:31.408 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:27:31.425479+0000 2024-04-26T00:27:31.408 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:27:31.408 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:31.408 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:27:31.408 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:27:31.408 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:27:31.408 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:27:31.408 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:27:31.408 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:31.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed38a01640 1 Processor -- start 2024-04-26T00:27:31.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed38a01640 1 -- start start 2024-04-26T00:27:31.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed38a01640 10 monclient: init 2024-04-26T00:27:31.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed38a01640 10 monclient: _reopen_session rank -1 2024-04-26T00:27:31.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed38a01640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-04-26T00:27:31.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed38a01640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed340a0260 0x7fed340a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:31.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed38a01640 10 monclient(hunting): picked mon.noname-b con 0x7fed340a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:27:31.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed38a01640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fed34134b30 0x7fed34134f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:31.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed38a01640 10 monclient(hunting): picked mon.noname-c con 0x7fed34134b30 addr v2:172.21.15.45:3300/0 2024-04-26T00:27:31.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed38a01640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fed34135490 0x7fed3413be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:31.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed38a01640 10 monclient(hunting): picked mon.noname-a con 0x7fed34135490 addr v2:172.21.15.37:3300/0 2024-04-26T00:27:31.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed38a01640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:31.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed38a01640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fed3409ec10 con 0x7fed34135490 2024-04-26T00:27:31.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed38a01640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:31.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed38a01640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fed3409f0f0 con 0x7fed34134b30 2024-04-26T00:27:31.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed38a01640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:31.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed38a01640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fed340a1130 con 0x7fed340a0260 2024-04-26T00:27:31.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed38a01640 10 monclient(hunting): _renew_subs 2024-04-26T00:27:31.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed38a01640 10 monclient(hunting): authenticate will time out at 1432.731187s 2024-04-26T00:27:31.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed3259c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed340a0260 0x7fed340a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:31.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed3259c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed340a0260 0x7fed340a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:49978/0 (socket says 172.21.15.37:49978) 2024-04-26T00:27:31.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed3259c640 1 -- 172.21.15.37:0/3994183132 learned_addr learned my addr 172.21.15.37:0/3994183132 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:27:31.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed3259c640 10 monclient(hunting): get_auth_request con 0x7fed340a0260 auth_method 0 2024-04-26T00:27:31.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed3259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:31.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed3259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:31.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed3259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:31.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed32d9d640 1 --2- 172.21.15.37:0/3994183132 >> v2:172.21.15.37:3300/0 conn(0x7fed34135490 0x7fed3413be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:31.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed32d9d640 10 monclient(hunting): get_auth_request con 0x7fed34135490 auth_method 0 2024-04-26T00:27:31.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed32d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:31.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed32d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:31.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed32d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:31.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed3259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:31.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed3259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:31.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed3259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:31.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed32d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:31.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed32d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:31.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed32d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:31.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed31d9b640 1 --2- 172.21.15.37:0/3994183132 >> v2:172.21.15.45:3300/0 conn(0x7fed34134b30 0x7fed34134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:31.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed3259c640 10 monclient(hunting): handle_auth_done global_id 84212 payload 274 2024-04-26T00:27:31.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed3259c640 1 -- 172.21.15.37:0/3994183132 >> v2:172.21.15.45:3300/0 conn(0x7fed34134b30 msgr2=0x7fed34134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:27:31.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed3259c640 1 --2- 172.21.15.37:0/3994183132 >> v2:172.21.15.45:3300/0 conn(0x7fed34134b30 0x7fed34134f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:31.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed3259c640 1 -- 172.21.15.37:0/3994183132 >> v2:172.21.15.37:3300/0 conn(0x7fed34135490 msgr2=0x7fed3413be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:27:31.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed3259c640 1 --2- 172.21.15.37:0/3994183132 >> v2:172.21.15.37:3300/0 conn(0x7fed34135490 0x7fed3413be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:31.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed3259c640 10 monclient: _finish_hunting 0 2024-04-26T00:27:31.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed3259c640 1 monclient: found mon.noname-b 2024-04-26T00:27:31.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed3259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:27:31.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed3259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:27:31.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed3259c640 1 -- 172.21.15.37:0/3994183132 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed3406cff0 con 0x7fed340a0260 2024-04-26T00:27:31.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed31d9b640 10 monclient: get_auth_request con 0x7fed34134b30 auth_method 0 2024-04-26T00:27:31.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed31d9b640 1 --2- 172.21.15.37:0/3994183132 >> v2:172.21.15.45:3300/0 conn(0x7fed34134b30 0x7fed34134f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:27:31.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.419+0000 7fed32d9d640 1 --2- 172.21.15.37:0/3994183132 >> v2:172.21.15.37:3300/0 conn(0x7fed34135490 0x7fed3413be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:27:31.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.423+0000 7fed3259c640 1 -- 172.21.15.37:0/3994183132 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed340a0260 msgr2=0x7fed340a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:31.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.423+0000 7fed3259c640 1 --2- 172.21.15.37:0/3994183132 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed340a0260 0x7fed340a0640 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fed3406b640 tx=0x7fed2401c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a7314fa8902a9e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:31.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed3159a640 1 -- 172.21.15.37:0/3994183132 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed240b0070 con 0x7fed340a0260 2024-04-26T00:27:31.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed3159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:31.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed3159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:27:31.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed3159a640 10 monclient: dump: 2024-04-26T00:27:31.419 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:31.419 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:31.419 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:31.419 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:31.419 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:31.419 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:31.419 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:31.419 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:31.419 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:31.419 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:31.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed3159a640 10 monclient: _finish_auth 0 2024-04-26T00:27:31.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed3159a640 10 monclient: _check_auth_tickets 2024-04-26T00:27:31.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed3159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:27:31.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed3159a640 1 -- 172.21.15.37:0/3994183132 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed240ab600 con 0x7fed340a0260 2024-04-26T00:27:31.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed38a01640 5 monclient: authenticate success, global_id 84212 2024-04-26T00:27:31.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed38a01640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:27:31.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed3159a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:27:31.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed38a01640 10 monclient: get_monmap_and_config success 2024-04-26T00:27:31.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed3159a640 1 -- 172.21.15.37:0/3994183132 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed240ab9d0 con 0x7fed340a0260 2024-04-26T00:27:31.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed38a01640 4 set_mon_vals no callback set 2024-04-26T00:27:31.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed3159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:31.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed3159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:27:31.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed3159a640 10 monclient: dump: 2024-04-26T00:27:31.420 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:31.420 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:31.420 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:31.420 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:31.421 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:31.421 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:31.421 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:31.421 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:31.421 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:31.421 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:31.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed38a01640 10 monclient: shutdown 2024-04-26T00:27:31.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed38a01640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:27:31.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed38a01640 1 -- 172.21.15.37:0/3994183132 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed340a0260 msgr2=0x7fed340a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:31.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed38a01640 1 --2- 172.21.15.37:0/3994183132 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed340a0260 0x7fed340a0640 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fed3406b640 tx=0x7fed2401c3e0 comp rx=0 tx=0).stop 2024-04-26T00:27:31.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed38a01640 1 -- 172.21.15.37:0/3994183132 shutdown_connections 2024-04-26T00:27:31.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed38a01640 1 --2- 172.21.15.37:0/3994183132 >> v2:172.21.15.45:3300/0 conn(0x7fed34134b30 0x7fed34134f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:31.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed38a01640 1 --2- 172.21.15.37:0/3994183132 >> v2:172.21.15.37:3300/0 conn(0x7fed34135490 0x7fed3413be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:31.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed38a01640 1 --2- 172.21.15.37:0/3994183132 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed340a0260 0x7fed340a0640 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:31.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed38a01640 1 -- 172.21.15.37:0/3994183132 >> 172.21.15.37:0/3994183132 conn(0x7fed340a1ad0 msgr2=0x7fed340a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:27:31.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed38a01640 1 -- 172.21.15.37:0/3994183132 shutdown_connections 2024-04-26T00:27:31.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed38a01640 1 -- 172.21.15.37:0/3994183132 wait complete. 2024-04-26T00:27:31.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed38a01640 10 monclient: build_initial_monmap 2024-04-26T00:27:31.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed38a01640 10 monclient: monmap: 2024-04-26T00:27:31.422 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:27:31.422 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:27:31.422 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:27:31.422 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:27:31.422 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:27:31.422 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:27:31.422 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:27:31.422 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:27:31.422 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:27:31.422 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:31.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed38a01640 1 Processor -- start 2024-04-26T00:27:31.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed38a01640 1 -- start start 2024-04-26T00:27:31.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed38a01640 10 monclient: init 2024-04-26T00:27:31.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed38a01640 10 monclient: _reopen_session rank -1 2024-04-26T00:27:31.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed38a01640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:27:31.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed38a01640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed340a0260 0x7fed341cf030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:31.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed38a01640 10 monclient(hunting): picked mon.noname-c con 0x7fed340a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:27:31.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed38a01640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed34134b30 0x7fed34131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:31.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed38a01640 10 monclient(hunting): picked mon.noname-a con 0x7fed34134b30 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:27:31.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed38a01640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed34135490 0x7fed341cbb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:31.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed38a01640 10 monclient(hunting): picked mon.noname-b con 0x7fed34135490 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:27:31.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed38a01640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:31.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed38a01640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fed3409f430 con 0x7fed34134b30 2024-04-26T00:27:31.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed38a01640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:31.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed38a01640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fed3409f2b0 con 0x7fed34135490 2024-04-26T00:27:31.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed38a01640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:31.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed38a01640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fed3409fe90 con 0x7fed340a0260 2024-04-26T00:27:31.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed38a01640 10 monclient(hunting): _renew_subs 2024-04-26T00:27:31.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed38a01640 10 monclient(hunting): authenticate will time out at 1432.738421s 2024-04-26T00:27:31.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed3259c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed340a0260 0x7fed341cf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:31.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed31d9b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed34134b30 0x7fed34131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:31.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed3259c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed340a0260 0x7fed341cf030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:49982/0 (socket says 172.21.15.37:49982) 2024-04-26T00:27:31.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed3259c640 1 -- 172.21.15.37:0/2729756186 learned_addr learned my addr 172.21.15.37:0/2729756186 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:27:31.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed3259c640 10 monclient(hunting): get_auth_request con 0x7fed340a0260 auth_method 0 2024-04-26T00:27:31.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed3259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:31.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed3259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:31.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed3259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:31.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed31d9b640 10 monclient(hunting): get_auth_request con 0x7fed34134b30 auth_method 0 2024-04-26T00:27:31.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed31d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:31.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed31d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:31.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed31d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:31.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed3259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:31.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed3259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:31.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed3259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:31.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed31d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:31.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed31d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:31.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed31d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:31.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed32d9d640 1 --2- 172.21.15.37:0/2729756186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed34135490 0x7fed341cbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:31.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed3259c640 10 monclient(hunting): handle_auth_done global_id 84215 payload 824 2024-04-26T00:27:31.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed3259c640 1 -- 172.21.15.37:0/2729756186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed34135490 msgr2=0x7fed341cbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:31.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed3259c640 1 --2- 172.21.15.37:0/2729756186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed34135490 0x7fed341cbb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:31.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed3259c640 1 -- 172.21.15.37:0/2729756186 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed34134b30 msgr2=0x7fed34131ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:31.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed3259c640 1 --2- 172.21.15.37:0/2729756186 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed34134b30 0x7fed34131ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:31.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed3259c640 10 monclient: _finish_hunting 0 2024-04-26T00:27:31.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed3259c640 1 monclient: found mon.noname-c 2024-04-26T00:27:31.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed3259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:27:31.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed3259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:27:31.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed3259c640 1 -- 172.21.15.37:0/2729756186 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed341cfa90 con 0x7fed340a0260 2024-04-26T00:27:31.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed32d9d640 10 monclient: get_auth_request con 0x7fed34135490 auth_method 0 2024-04-26T00:27:31.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.427+0000 7fed32d9d640 1 --2- 172.21.15.37:0/2729756186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed34135490 0x7fed341cbb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:27:31.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.431+0000 7fed3259c640 1 -- 172.21.15.37:0/2729756186 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed340a0260 msgr2=0x7fed341cf030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:31.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.431+0000 7fed3259c640 1 --2- 172.21.15.37:0/2729756186 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed340a0260 0x7fed341cf030 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fed2401a010 tx=0x7fed24019180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:31.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.431+0000 7fed22ffd640 1 -- 172.21.15.37:0/2729756186 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed240b0070 con 0x7fed340a0260 2024-04-26T00:27:31.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.431+0000 7fed22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:31.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.431+0000 7fed22ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:27:31.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.431+0000 7fed22ffd640 10 monclient: dump: 2024-04-26T00:27:31.428 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:31.428 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:31.428 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:31.428 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:31.428 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:31.429 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:31.429 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:31.429 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:31.429 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:31.429 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:31.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.431+0000 7fed22ffd640 10 monclient: _finish_auth 0 2024-04-26T00:27:31.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.431+0000 7fed22ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:27:31.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.431+0000 7fed22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:27:31.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.431+0000 7fed22ffd640 1 -- 172.21.15.37:0/2729756186 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed240b1040 con 0x7fed340a0260 2024-04-26T00:27:31.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.431+0000 7fed38a01640 5 monclient: authenticate success, global_id 84215 2024-04-26T00:27:31.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.431+0000 7fed38a01640 10 monclient: _renew_subs 2024-04-26T00:27:31.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.431+0000 7fed38a01640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:27:31.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.431+0000 7fed38a01640 1 -- 172.21.15.37:0/2729756186 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed3411d5e0 con 0x7fed340a0260 2024-04-26T00:27:31.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.431+0000 7fed22ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:27:31.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.431+0000 7fed22ffd640 1 -- 172.21.15.37:0/2729756186 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed240abe60 con 0x7fed340a0260 2024-04-26T00:27:31.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.431+0000 7fed22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:31.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.431+0000 7fed22ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:27:31.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.431+0000 7fed22ffd640 10 monclient: dump: 2024-04-26T00:27:31.430 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:31.430 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:31.430 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:31.430 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:31.430 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:31.430 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:31.430 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:31.430 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:31.430 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:31.430 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:31.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.431+0000 7fed23fff640 4 set_mon_vals no callback set 2024-04-26T00:27:31.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.431+0000 7fed38a01640 10 monclient: _renew_subs 2024-04-26T00:27:31.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.431+0000 7fed38a01640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:27:31.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.431+0000 7fed38a01640 1 -- 172.21.15.37:0/2729756186 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fed3411e360 con 0x7fed340a0260 2024-04-26T00:27:31.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.431+0000 7fed38a01640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:31.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.431+0000 7fed38a01640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:31.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.431+0000 7fed38a01640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:27:31.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.431+0000 7fed38a01640 1 -- 172.21.15.37:0/2729756186 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed3411e530 con 0x7fed340a0260 2024-04-26T00:27:31.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.431+0000 7fed22ffd640 1 -- 172.21.15.37:0/2729756186 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fed240aa9e0 con 0x7fed340a0260 2024-04-26T00:27:31.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.431+0000 7fed22ffd640 1 --2- 172.21.15.37:0/2729756186 >> v2:172.21.15.45:6812/2379918623 conn(0x7fed14035ed0 0x7fed14038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:31.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.431+0000 7fed22ffd640 1 -- 172.21.15.37:0/2729756186 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(296..296 src has 177..296) ==== 4039+0+0 (secure 0 0 0) 0x7fed240e3380 con 0x7fed340a0260 2024-04-26T00:27:31.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.431+0000 7fed31d9b640 1 --2- 172.21.15.37:0/2729756186 >> v2:172.21.15.45:6812/2379918623 conn(0x7fed14035ed0 0x7fed14038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:31.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.431+0000 7fed31d9b640 10 monclient: get_auth_request con 0x7fed14035ed0 auth_method 0 2024-04-26T00:27:31.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.435+0000 7fed31d9b640 1 --2- 172.21.15.37:0/2729756186 >> v2:172.21.15.45:6812/2379918623 conn(0x7fed14035ed0 0x7fed14038390 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fed3411ef60 tx=0x7fed28008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:31.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.435+0000 7fed22ffd640 1 -- 172.21.15.37:0/2729756186 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fed341cfa90 con 0x7fed340a0260 2024-04-26T00:27:31.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.435+0000 7fed22ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:31.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.435+0000 7fed22ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:27:31.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.659+0000 7fed38a01640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:27:31.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.659+0000 7fed38a01640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:27:31.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.659+0000 7fed38a01640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:27:31.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.659+0000 7fed38a01640 1 -- 172.21.15.37:0/2729756186 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fed34066f40 con 0x7fed340a0260 2024-04-26T00:27:31.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.659+0000 7fed22ffd640 1 -- 172.21.15.37:0/2729756186 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fed240160f0 con 0x7fed340a0260 2024-04-26T00:27:31.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.659+0000 7fed22ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:27:31.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.659+0000 7fed22ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:27:31.648 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":58,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:27:31.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.663+0000 7fed38a01640 1 -- 172.21.15.37:0/2729756186 >> v2:172.21.15.45:6812/2379918623 conn(0x7fed14035ed0 msgr2=0x7fed14038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:31.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.663+0000 7fed38a01640 1 --2- 172.21.15.37:0/2729756186 >> v2:172.21.15.45:6812/2379918623 conn(0x7fed14035ed0 0x7fed14038390 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fed3411ef60 tx=0x7fed28008040 comp rx=0 tx=0).stop 2024-04-26T00:27:31.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.663+0000 7fed38a01640 10 monclient: shutdown 2024-04-26T00:27:31.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.663+0000 7fed38a01640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:27:31.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.663+0000 7fed38a01640 1 -- 172.21.15.37:0/2729756186 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed340a0260 msgr2=0x7fed341cf030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:31.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.663+0000 7fed38a01640 1 --2- 172.21.15.37:0/2729756186 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed340a0260 0x7fed341cf030 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fed2401a010 tx=0x7fed24019180 comp rx=0 tx=0).stop 2024-04-26T00:27:31.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.663+0000 7fed38a01640 1 -- 172.21.15.37:0/2729756186 shutdown_connections 2024-04-26T00:27:31.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.663+0000 7fed38a01640 1 --2- 172.21.15.37:0/2729756186 >> v2:172.21.15.45:6812/2379918623 conn(0x7fed14035ed0 0x7fed14038390 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:31.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.663+0000 7fed38a01640 1 --2- 172.21.15.37:0/2729756186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed34135490 0x7fed341cbb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:31.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.663+0000 7fed38a01640 1 --2- 172.21.15.37:0/2729756186 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed34134b30 0x7fed34131ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:31.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.663+0000 7fed38a01640 1 --2- 172.21.15.37:0/2729756186 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed340a0260 0x7fed341cf030 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:31.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.663+0000 7fed38a01640 1 -- 172.21.15.37:0/2729756186 >> 172.21.15.37:0/2729756186 conn(0x7fed340a1ad0 msgr2=0x7fed340a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:27:31.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.663+0000 7fed38a01640 1 -- 172.21.15.37:0/2729756186 shutdown_connections 2024-04-26T00:27:31.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.663+0000 7fed38a01640 1 -- 172.21.15.37:0/2729756186 wait complete. 2024-04-26T00:27:31.666 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:27:31.667 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-26T00:27:31.667 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:27:31.667 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:27:31.755 INFO:tasks.rados.rados.0.smithi045.stdout:1248: done (0 left) 2024-04-26T00:27:31.755 INFO:tasks.rados.rados.0.smithi045.stdout:1249: copy_from oid 42 from oid 24 current snap is 146 2024-04-26T00:27:31.755 INFO:tasks.rados.rados.0.smithi045.stdout:1250: write (excl) oid 39 current snap is 146 2024-04-26T00:27:31.755 INFO:tasks.rados.rados.0.smithi045.stdout:1250: seq_num 468 ranges {665676=714065,1886514=531486,3169137=696539} 2024-04-26T00:27:31.777 INFO:tasks.rados.rados.0.smithi045.stdout:1250: writing smithi04526579-39 from 665676 to 1379741 tid 1 2024-04-26T00:27:31.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.795+0000 7fe007476640 10 monclient: get_monmap_and_config 2024-04-26T00:27:31.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.795+0000 7fe007476640 10 monclient: build_initial_monmap 2024-04-26T00:27:31.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.795+0000 7fe007476640 10 monclient: monmap: 2024-04-26T00:27:31.785 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:27:31.785 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:31.786 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:27:31.802590+0000 2024-04-26T00:27:31.786 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:27:31.802590+0000 2024-04-26T00:27:31.786 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:27:31.786 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:31.786 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:27:31.786 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:27:31.786 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:27:31.786 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:27:31.787 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:27:31.787 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:31.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.795+0000 7fe007476640 1 Processor -- start 2024-04-26T00:27:31.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.795+0000 7fe007476640 1 -- start start 2024-04-26T00:27:31.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.795+0000 7fe007476640 10 monclient: init 2024-04-26T00:27:31.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe007476640 10 monclient: _reopen_session rank -1 2024-04-26T00:27:31.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe007476640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-04-26T00:27:31.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe007476640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe000136f20 0x7fe00009bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:31.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe007476640 10 monclient(hunting): picked mon.noname-c con 0x7fe000136f20 addr v2:172.21.15.45:3300/0 2024-04-26T00:27:31.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe007476640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fe000137300 0x7fe0001377b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:31.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe007476640 10 monclient(hunting): picked mon.noname-a con 0x7fe000137300 addr v2:172.21.15.37:3300/0 2024-04-26T00:27:31.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe007476640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe000133c70 0x7fe000134120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:31.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe007476640 10 monclient(hunting): picked mon.noname-b con 0x7fe000133c70 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:27:31.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe007476640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:31.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe007476640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fe0000a50f0 con 0x7fe000137300 2024-04-26T00:27:31.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe007476640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:31.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe007476640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fe00009b330 con 0x7fe000136f20 2024-04-26T00:27:31.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe007476640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:31.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe007476640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe00009b610 con 0x7fe000133c70 2024-04-26T00:27:31.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe007476640 10 monclient(hunting): _renew_subs 2024-04-26T00:27:31.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe007476640 10 monclient(hunting): authenticate will time out at 1433.108179s 2024-04-26T00:27:31.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe005a13640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe000133c70 0x7fe000134120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:31.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe004a11640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fe000137300 0x7fe0001377b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:31.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe005a13640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe000133c70 0x7fe000134120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:49984/0 (socket says 172.21.15.37:49984) 2024-04-26T00:27:31.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe004a11640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fe000137300 0x7fe0001377b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50314/0 (socket says 172.21.15.37:50314) 2024-04-26T00:27:31.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe005a13640 1 -- 172.21.15.37:0/1620204722 learned_addr learned my addr 172.21.15.37:0/1620204722 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:27:31.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe005a13640 10 monclient(hunting): get_auth_request con 0x7fe000133c70 auth_method 0 2024-04-26T00:27:31.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe005a13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:31.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe005a13640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:31.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe005a13640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:31.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe004a11640 10 monclient(hunting): get_auth_request con 0x7fe000137300 auth_method 0 2024-04-26T00:27:31.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe004a11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:31.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe004a11640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:31.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe004a11640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:31.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe005a13640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:31.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe005a13640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:31.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe005a13640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:31.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe004a11640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:31.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe004a11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:31.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe004a11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:31.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe005212640 1 --2- 172.21.15.37:0/1620204722 >> v2:172.21.15.45:3300/0 conn(0x7fe000136f20 0x7fe00009bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:31.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe005212640 10 monclient(hunting): get_auth_request con 0x7fe000136f20 auth_method 0 2024-04-26T00:27:31.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe005212640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:31.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe005212640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:31.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe005212640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:31.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe005a13640 10 monclient(hunting): handle_auth_done global_id 84218 payload 274 2024-04-26T00:27:31.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe005a13640 1 -- 172.21.15.37:0/1620204722 >> v2:172.21.15.45:3300/0 conn(0x7fe000136f20 msgr2=0x7fe00009bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:27:31.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe005a13640 1 --2- 172.21.15.37:0/1620204722 >> v2:172.21.15.45:3300/0 conn(0x7fe000136f20 0x7fe00009bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:31.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe005a13640 1 -- 172.21.15.37:0/1620204722 >> v2:172.21.15.37:3300/0 conn(0x7fe000137300 msgr2=0x7fe0001377b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:27:31.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe005a13640 1 --2- 172.21.15.37:0/1620204722 >> v2:172.21.15.37:3300/0 conn(0x7fe000137300 0x7fe0001377b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:31.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe005a13640 10 monclient: _finish_hunting 0 2024-04-26T00:27:31.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe005a13640 1 monclient: found mon.noname-b 2024-04-26T00:27:31.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe005a13640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:27:31.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe005a13640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:27:31.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.799+0000 7fe005a13640 1 -- 172.21.15.37:0/1620204722 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe00009b8f0 con 0x7fe000133c70 2024-04-26T00:27:31.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe005a13640 1 -- 172.21.15.37:0/1620204722 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe000133c70 msgr2=0x7fe000134120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:31.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe005a13640 1 --2- 172.21.15.37:0/1620204722 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe000133c70 0x7fe000134120 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe000068940 tx=0x7fdff801c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=deafb53b488b4a9c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:31.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fdff7fff640 1 -- 172.21.15.37:0/1620204722 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff80a5640 con 0x7fe000133c70 2024-04-26T00:27:31.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fdff7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:31.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fdff7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:27:31.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fdff7fff640 10 monclient: dump: 2024-04-26T00:27:31.799 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:31.799 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:31.799 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:31.799 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:31.799 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:31.799 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:31.799 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:31.799 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:31.799 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:31.799 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:31.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fdff7fff640 10 monclient: _finish_auth 0 2024-04-26T00:27:31.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fdff7fff640 10 monclient: _check_auth_tickets 2024-04-26T00:27:31.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fdff7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:27:31.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fdff7fff640 1 -- 172.21.15.37:0/1620204722 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdff80a57e0 con 0x7fe000133c70 2024-04-26T00:27:31.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fdff7fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:27:31.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fdff7fff640 1 -- 172.21.15.37:0/1620204722 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff80a5bb0 con 0x7fe000133c70 2024-04-26T00:27:31.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe007476640 5 monclient: authenticate success, global_id 84218 2024-04-26T00:27:31.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe007476640 10 monclient: get_monmap_and_config success 2024-04-26T00:27:31.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe007476640 4 set_mon_vals no callback set 2024-04-26T00:27:31.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fdff7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:31.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fdff7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:27:31.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fdff7fff640 10 monclient: dump: 2024-04-26T00:27:31.801 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:31.801 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:31.801 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:31.801 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:31.801 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:31.801 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:31.801 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:31.801 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:31.802 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:31.802 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:31.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe007476640 10 monclient: shutdown 2024-04-26T00:27:31.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe007476640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:27:31.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe007476640 1 -- 172.21.15.37:0/1620204722 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe000133c70 msgr2=0x7fe000134120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:31.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe007476640 1 --2- 172.21.15.37:0/1620204722 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe000133c70 0x7fe000134120 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe000068940 tx=0x7fdff801c8b0 comp rx=0 tx=0).stop 2024-04-26T00:27:31.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe007476640 1 -- 172.21.15.37:0/1620204722 shutdown_connections 2024-04-26T00:27:31.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe007476640 1 --2- 172.21.15.37:0/1620204722 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe000133c70 0x7fe000134120 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:31.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe007476640 1 --2- 172.21.15.37:0/1620204722 >> v2:172.21.15.37:3300/0 conn(0x7fe000137300 0x7fe0001377b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:31.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe007476640 1 --2- 172.21.15.37:0/1620204722 >> v2:172.21.15.45:3300/0 conn(0x7fe000136f20 0x7fe00009bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:31.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe007476640 1 -- 172.21.15.37:0/1620204722 >> 172.21.15.37:0/1620204722 conn(0x7fe0000a61b0 msgr2=0x7fe0000a65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:27:31.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe007476640 1 -- 172.21.15.37:0/1620204722 shutdown_connections 2024-04-26T00:27:31.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe007476640 1 -- 172.21.15.37:0/1620204722 wait complete. 2024-04-26T00:27:31.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe007476640 10 monclient: build_initial_monmap 2024-04-26T00:27:31.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe007476640 10 monclient: monmap: 2024-04-26T00:27:31.803 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:27:31.803 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:27:31.803 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:27:31.803 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:27:31.803 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:27:31.803 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:27:31.803 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:27:31.803 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:27:31.803 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:27:31.803 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:31.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe007476640 1 Processor -- start 2024-04-26T00:27:31.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe007476640 1 -- start start 2024-04-26T00:27:31.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe007476640 10 monclient: init 2024-04-26T00:27:31.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe007476640 10 monclient: _reopen_session rank -1 2024-04-26T00:27:31.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe007476640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:27:31.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe007476640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe000133c70 0x7fe0001ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:31.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe007476640 10 monclient(hunting): picked mon.noname-b con 0x7fe000133c70 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:27:31.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe007476640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe000136f20 0x7fe0001cf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:31.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe007476640 10 monclient(hunting): picked mon.noname-a con 0x7fe000136f20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:27:31.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe007476640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe000137300 0x7fe0001cf920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:31.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe007476640 10 monclient(hunting): picked mon.noname-c con 0x7fe000137300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:27:31.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe007476640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:31.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe007476640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe00009ac60 con 0x7fe000136f20 2024-04-26T00:27:31.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe007476640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:31.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe007476640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe00009aae0 con 0x7fe000133c70 2024-04-26T00:27:31.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe007476640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:31.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe007476640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe000140c80 con 0x7fe000137300 2024-04-26T00:27:31.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe007476640 10 monclient(hunting): _renew_subs 2024-04-26T00:27:31.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe007476640 10 monclient(hunting): authenticate will time out at 1433.115072s 2024-04-26T00:27:31.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe005a13640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe000137300 0x7fe0001cf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:31.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe005a13640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe000137300 0x7fe0001cf920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:49994/0 (socket says 172.21.15.37:49994) 2024-04-26T00:27:31.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe005a13640 1 -- 172.21.15.37:0/3076637590 learned_addr learned my addr 172.21.15.37:0/3076637590 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:27:31.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe005a13640 10 monclient(hunting): get_auth_request con 0x7fe000137300 auth_method 0 2024-04-26T00:27:31.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe005a13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:31.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe005a13640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:31.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe005a13640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:31.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe004a11640 1 --2- 172.21.15.37:0/3076637590 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe000136f20 0x7fe0001cf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:31.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe005a13640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:31.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe005a13640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:31.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe005a13640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:31.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe004a11640 10 monclient(hunting): get_auth_request con 0x7fe000136f20 auth_method 0 2024-04-26T00:27:31.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe004a11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:31.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe004a11640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:31.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe004a11640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:31.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe005212640 1 --2- 172.21.15.37:0/3076637590 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe000133c70 0x7fe0001ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:31.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe004a11640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:31.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe004a11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:31.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe004a11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:31.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe005a13640 10 monclient(hunting): handle_auth_done global_id 84221 payload 824 2024-04-26T00:27:31.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe005a13640 1 -- 172.21.15.37:0/3076637590 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe000133c70 msgr2=0x7fe0001ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:31.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe005a13640 1 --2- 172.21.15.37:0/3076637590 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe000133c70 0x7fe0001ceea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:31.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe005a13640 1 -- 172.21.15.37:0/3076637590 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe000136f20 msgr2=0x7fe0001cf3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:31.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe005a13640 1 --2- 172.21.15.37:0/3076637590 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe000136f20 0x7fe0001cf3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:31.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe005a13640 10 monclient: _finish_hunting 0 2024-04-26T00:27:31.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe005a13640 1 monclient: found mon.noname-c 2024-04-26T00:27:31.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe005a13640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:27:31.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe005a13640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:27:31.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe005a13640 1 -- 172.21.15.37:0/3076637590 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe00013a800 con 0x7fe000137300 2024-04-26T00:27:31.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe005212640 10 monclient: get_auth_request con 0x7fe000133c70 auth_method 0 2024-04-26T00:27:31.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.803+0000 7fe005212640 1 --2- 172.21.15.37:0/3076637590 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe000133c70 0x7fe0001ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:27:31.808 INFO:tasks.rados.rados.0.smithi045.stdout:1250: writing smithi04526579-39 from 1886514 to 2418000 tid 2 2024-04-26T00:27:31.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fe005a13640 1 -- 172.21.15.37:0/3076637590 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe000137300 msgr2=0x7fe0001cf920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:31.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fe005a13640 1 --2- 172.21.15.37:0/3076637590 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe000137300 0x7fe0001cf920 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fdff801af60 tx=0x7fdff8019650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:31.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fdff5ffb640 1 -- 172.21.15.37:0/3076637590 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff80bc020 con 0x7fe000137300 2024-04-26T00:27:31.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fdff5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:31.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fdff5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:27:31.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fdff5ffb640 10 monclient: dump: 2024-04-26T00:27:31.812 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:31.812 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:31.812 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:31.812 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:31.812 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:31.812 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:31.812 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:31.812 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:31.813 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:31.813 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:31.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fdff5ffb640 10 monclient: _finish_auth 0 2024-04-26T00:27:31.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fdff5ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:27:31.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fdff5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:27:31.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fdff5ffb640 1 -- 172.21.15.37:0/3076637590 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdff80b8880 con 0x7fe000137300 2024-04-26T00:27:31.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fe007476640 5 monclient: authenticate success, global_id 84221 2024-04-26T00:27:31.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fe007476640 10 monclient: _renew_subs 2024-04-26T00:27:31.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fe007476640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:27:31.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fe007476640 1 -- 172.21.15.37:0/3076637590 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe00013caf0 con 0x7fe000137300 2024-04-26T00:27:31.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fdff5ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:27:31.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fdff5ffb640 1 -- 172.21.15.37:0/3076637590 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff80a5640 con 0x7fe000137300 2024-04-26T00:27:31.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fdff5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:31.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fdff5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:27:31.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fdff5ffb640 10 monclient: dump: 2024-04-26T00:27:31.814 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:31.814 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:31.814 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:31.814 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:31.814 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:31.814 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:31.814 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:31.814 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:31.814 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:31.814 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:31.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fe007476640 10 monclient: _renew_subs 2024-04-26T00:27:31.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fe007476640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:27:31.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fe007476640 1 -- 172.21.15.37:0/3076637590 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe000138b60 con 0x7fe000137300 2024-04-26T00:27:31.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fdff77fe640 4 set_mon_vals no callback set 2024-04-26T00:27:31.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fe007476640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:31.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fdff5ffb640 1 -- 172.21.15.37:0/3076637590 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fdff80bf020 con 0x7fe000137300 2024-04-26T00:27:31.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fe007476640 1 --2- 172.21.15.37:0/3076637590 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe000132c60 0x7fe000133040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:31.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fe007476640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:31.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fe007476640 1 -- 172.21.15.37:0/3076637590 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe000136260 con 0x7fe000132c60 2024-04-26T00:27:31.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fe005212640 1 --2- 172.21.15.37:0/3076637590 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe000132c60 0x7fe000133040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:31.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fe005212640 10 monclient: get_auth_request con 0x7fe000132c60 auth_method 0 2024-04-26T00:27:31.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fdff5ffb640 1 --2- 172.21.15.37:0/3076637590 >> v2:172.21.15.45:6812/2379918623 conn(0x7fdfe4035f00 0x7fdfe40383c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:31.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fe005212640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:31.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fe005212640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:31.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fe005212640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:31.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fdff5ffb640 1 -- 172.21.15.37:0/3076637590 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(297..297 src has 177..297) ==== 4011+0+0 (secure 0 0 0) 0x7fdff80e38c0 con 0x7fe000137300 2024-04-26T00:27:31.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fe005212640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:31.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fe005212640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:31.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fe005212640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:31.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fe005212640 10 monclient(hunting): handle_auth_done global_id 94125 payload 824 2024-04-26T00:27:31.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fe005212640 1 -- 172.21.15.37:0/3076637590 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe00009ae30 con 0x7fe000132c60 2024-04-26T00:27:31.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fe004a11640 1 --2- 172.21.15.37:0/3076637590 >> v2:172.21.15.45:6812/2379918623 conn(0x7fdfe4035f00 0x7fdfe40383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:31.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.807+0000 7fe004a11640 10 monclient: get_auth_request con 0x7fdfe4035f00 auth_method 0 2024-04-26T00:27:31.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.811+0000 7fe005212640 1 -- 172.21.15.37:0/3076637590 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe000132c60 msgr2=0x7fe000133040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:31.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.811+0000 7fe005212640 1 --2- 172.21.15.37:0/3076637590 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe000132c60 0x7fe000133040 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdffc009d70 tx=0x7fdffc00c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:31.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.811+0000 7fdff5ffb640 1 -- 172.21.15.37:0/3076637590 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdffc015d80 con 0x7fe000132c60 2024-04-26T00:27:31.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.811+0000 7fdff5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:31.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.811+0000 7fdff5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:27:31.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.811+0000 7fdff5ffb640 10 monclient: dump: 2024-04-26T00:27:31.817 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:31.817 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:31.817 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:31.817 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:31.817 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:31.817 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:31.817 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:31.817 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:31.817 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:31.817 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:31.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.811+0000 7fdff5ffb640 1 -- 172.21.15.37:0/3076637590 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdffc015d80 con 0x7fe000132c60 2024-04-26T00:27:31.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.811+0000 7fdff5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:31.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.811+0000 7fdff5ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:27:31.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.811+0000 7fdff5ffb640 1 -- 172.21.15.37:0/3076637590 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe000132c60 msgr2=0x7fe000133040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:31.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.811+0000 7fdff5ffb640 1 --2- 172.21.15.37:0/3076637590 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe000132c60 0x7fe000133040 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdffc009d70 tx=0x7fdffc00c4c0 comp rx=0 tx=0).stop 2024-04-26T00:27:31.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.811+0000 7fdff5ffb640 1 -- 172.21.15.37:0/3076637590 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe000132c60 msgr2=0x7fe000133040 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:27:31.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.811+0000 7fdff5ffb640 1 --2- 172.21.15.37:0/3076637590 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe000132c60 0x7fe000133040 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:31.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.811+0000 7fe004a11640 1 --2- 172.21.15.37:0/3076637590 >> v2:172.21.15.45:6812/2379918623 conn(0x7fdfe4035f00 0x7fdfe40383c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe00013a400 tx=0x7fdff0009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:31.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.819+0000 7fe007476640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:27:31.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.819+0000 7fe007476640 1 --2- 172.21.15.37:0/3076637590 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe000133580 0x7fe0000a08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:31.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.819+0000 7fe007476640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:31.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.819+0000 7fe007476640 1 -- 172.21.15.37:0/3076637590 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe00004f510 con 0x7fe000133580 2024-04-26T00:27:31.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.819+0000 7fe005212640 1 --2- 172.21.15.37:0/3076637590 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe000133580 0x7fe0000a08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:31.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.819+0000 7fe005212640 10 monclient: get_auth_request con 0x7fe000133580 auth_method 0 2024-04-26T00:27:31.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.819+0000 7fe005212640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:31.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.819+0000 7fe005212640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:31.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.819+0000 7fe005212640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:31.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.819+0000 7fe005212640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:31.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.819+0000 7fe005212640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:31.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.819+0000 7fe005212640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:31.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.819+0000 7fe005212640 10 monclient(hunting): handle_auth_done global_id 94128 payload 824 2024-04-26T00:27:31.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.819+0000 7fe005212640 1 -- 172.21.15.37:0/3076637590 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe000075f10 con 0x7fe000133580 2024-04-26T00:27:31.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.819+0000 7fe005212640 1 -- 172.21.15.37:0/3076637590 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe000133580 msgr2=0x7fe0000a08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:31.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.819+0000 7fe005212640 1 --2- 172.21.15.37:0/3076637590 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe000133580 0x7fe0000a08e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fdffc009d70 tx=0x7fdffc0046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:31.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.819+0000 7fdff5ffb640 1 -- 172.21.15.37:0/3076637590 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdffc003a30 con 0x7fe000133580 2024-04-26T00:27:31.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.819+0000 7fdff5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:31.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.819+0000 7fdff5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:27:31.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.819+0000 7fdff5ffb640 10 monclient: dump: 2024-04-26T00:27:31.821 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:31.822 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:31.822 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:31.822 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:31.822 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:31.822 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:31.822 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:31.822 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:31.822 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:31.822 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:31.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.819+0000 7fdff5ffb640 1 -- 172.21.15.37:0/3076637590 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7fdffc0252b0 con 0x7fe000133580 2024-04-26T00:27:31.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.819+0000 7fdff5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:27:31.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.819+0000 7fdff5ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:27:31.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.819+0000 7fdff5ffb640 1 -- 172.21.15.37:0/3076637590 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe000133580 msgr2=0x7fe0000a08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:31.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.819+0000 7fdff5ffb640 1 --2- 172.21.15.37:0/3076637590 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe000133580 0x7fe0000a08e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fdffc009d70 tx=0x7fdffc0046d0 comp rx=0 tx=0).stop 2024-04-26T00:27:31.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.819+0000 7fdff5ffb640 1 -- 172.21.15.37:0/3076637590 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe000133580 msgr2=0x7fe0000a08e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:27:31.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.819+0000 7fdff5ffb640 1 --2- 172.21.15.37:0/3076637590 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe000133580 0x7fe0000a08e0 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fdffc009d70 tx=0x7fdffc0046d0 comp rx=0 tx=0).stop 2024-04-26T00:27:31.823 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:27:31.823 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:27:31.823 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:27:31.824 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:27:31.824 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 58, 2024-04-26T00:27:31.824 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 12579, 2024-04-26T00:27:31.824 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:27:31.824 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:27:31.824 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:27:31.824 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:27:31.824 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:31.824 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 6, 2024-04-26T00:27:31.825 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:27:31.825 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:27:31.825 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:27:31.825 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:27:31.825 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:27:31.825 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:27:31.826 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:27:31.826 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:27:31.826 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:27:31.826 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:27:31.826 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:27:31.826 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:27:31.826 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:27:31.826 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:27:31.827 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:31.827 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:27:31.827 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:27:31.827 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:27:31.827 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:27:31.827 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:27:31.827 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:27:31.827 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:27:31.827 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:27:31.827 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:27:31.827 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:27:31.828 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:27:31.828 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:27:31.828 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:27:31.828 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:31.828 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:31.828 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:27:31.828 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:27:31.828 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:27:31.828 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:27:31.828 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:27:31.828 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:27:31.828 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:27:31.828 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:27:31.828 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:27:31.829 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:27:31.829 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:27:31.829 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:27:31.829 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:27:31.829 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:27:31.829 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:27:31.829 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:27:31.829 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:27:31.829 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:27:31.829 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:27:31.829 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:27:31.829 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:27:31.829 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:27:31.830 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:27:31.830 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:27:31.830 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:27:31.830 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:27:31.830 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:27:31.830 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:27:31.830 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:31.830 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:27:31.830 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:31.831 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:27:31.831 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:31.831 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:27:31.831 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:27:31.831 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:27:31.831 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:27:31.831 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:31.831 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:27:31.831 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:27:31.831 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:31.831 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:31.831 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:31.831 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:27:31.832 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:27:31.832 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:31.832 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:31.832 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:31.832 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:31.832 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:27:31.832 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:27:31.832 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:27:31.832 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:27:31.832 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:27:31.832 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:31.833 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:31.833 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:27:31.833 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:27:31.833 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:27:31.833 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:27:31.833 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:31.833 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:27:31.833 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:27:31.833 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:31.833 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:31.833 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:31.833 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:27:31.834 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:27:31.834 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:31.834 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:31.834 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:31.834 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:31.834 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:27:31.834 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:27:31.834 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:27:31.834 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:27:31.834 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:27:31.834 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:31.834 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:31.834 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:27:31.835 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:27:31.835 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:27:31.835 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:27:31.835 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:31.835 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:27:31.835 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:27:31.835 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:31.835 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:31.835 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:31.835 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:27:31.835 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:27:31.835 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:31.835 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:31.835 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:31.836 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:31.836 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:27:31.836 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:27:31.836 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:27:31.836 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:27:31.836 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:27:31.836 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:31.836 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:31.836 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:31.836 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:27:31.836 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:27:31.836 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:31.836 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:27:31.836 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:27:31.837 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:27:31.837 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:31.837 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:31.837 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:27:31.837 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:31.837 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:27:31.837 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:27:31.837 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:27:31.837 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:31.837 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:31.837 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:31.837 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:27:31.837 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:27:31.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.819+0000 7fe007476640 1 -- 172.21.15.37:0/3076637590 >> v2:172.21.15.45:6812/2379918623 conn(0x7fdfe4035f00 msgr2=0x7fdfe40383c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:31.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.819+0000 7fe007476640 1 --2- 172.21.15.37:0/3076637590 >> v2:172.21.15.45:6812/2379918623 conn(0x7fdfe4035f00 0x7fdfe40383c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe00013a400 tx=0x7fdff0009190 comp rx=0 tx=0).stop 2024-04-26T00:27:31.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.819+0000 7fe007476640 10 monclient: shutdown 2024-04-26T00:27:31.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.819+0000 7fe007476640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:27:31.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.819+0000 7fe007476640 1 -- 172.21.15.37:0/3076637590 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe000137300 msgr2=0x7fe0001cf920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:31.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.819+0000 7fe007476640 1 --2- 172.21.15.37:0/3076637590 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe000137300 0x7fe0001cf920 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fdff801af60 tx=0x7fdff8019650 comp rx=0 tx=0).stop 2024-04-26T00:27:31.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.819+0000 7fe005a13640 1 -- 172.21.15.37:0/3076637590 reap_dead start 2024-04-26T00:27:31.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.819+0000 7fe005a13640 1 -- 172.21.15.37:0/3076637590 reap_dead start 2024-04-26T00:27:31.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.819+0000 7fe007476640 1 -- 172.21.15.37:0/3076637590 shutdown_connections 2024-04-26T00:27:31.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.819+0000 7fe007476640 1 -- 172.21.15.37:0/3076637590 >> 172.21.15.37:0/3076637590 conn(0x7fe0000a61b0 msgr2=0x7fe000135720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:27:31.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.819+0000 7fe007476640 1 -- 172.21.15.37:0/3076637590 shutdown_connections 2024-04-26T00:27:31.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.819+0000 7fe007476640 1 -- 172.21.15.37:0/3076637590 wait complete. 2024-04-26T00:27:31.840 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:27:31.840 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:27:31.841 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:27:31.841 INFO:tasks.rados.rados.0.smithi045.stdout:1250: writing smithi04526579-39 from 3169137 to 3865676 tid 3 2024-04-26T00:27:31.841 INFO:tasks.rados.rados.0.smithi045.stdout:1251: write (excl) oid 5 current snap is 146 2024-04-26T00:27:31.841 INFO:tasks.rados.rados.0.smithi045.stdout:1251: seq_num 469 ranges {682483=789314,1944996=791497,3196560=554574} 2024-04-26T00:27:31.841 INFO:tasks.rados.rados.0.smithi045.stdout:1251: writing smithi04526579-5 from 682483 to 1471797 tid 1 2024-04-26T00:27:31.842 INFO:tasks.rados.rados.0.smithi045.stdout:1251: writing smithi04526579-5 from 1944996 to 2736493 tid 2 2024-04-26T00:27:31.851 INFO:tasks.rados.rados.0.smithi045.stdout:1251: writing smithi04526579-5 from 3196560 to 3751134 tid 3 2024-04-26T00:27:31.851 INFO:tasks.rados.rados.0.smithi045.stdout:1252: read oid 50 snap -1 2024-04-26T00:27:31.851 INFO:tasks.rados.rados.0.smithi045.stdout:1252: expect (ObjNum 136 snap 30 seq_num 136) 2024-04-26T00:27:31.851 INFO:tasks.rados.rados.0.smithi045.stdout:1253: copy_from oid 32 from oid 8 current snap is 146 2024-04-26T00:27:31.851 INFO:tasks.rados.rados.0.smithi045.stdout:1249: finishing copy_from to smithi04526579-42 2024-04-26T00:27:31.851 INFO:tasks.rados.rados.0.smithi045.stdout:1249: got expected ENOENT (src dne) 2024-04-26T00:27:31.851 INFO:tasks.rados.rados.0.smithi045.stdout:1249: finishing copy_from racing read to smithi04526579-42 2024-04-26T00:27:31.851 INFO:tasks.rados.rados.0.smithi045.stdout:1250: finishing write tid 1 to smithi04526579-39 2024-04-26T00:27:31.851 INFO:tasks.rados.rados.0.smithi045.stdout:1250: finishing write tid 2 to smithi04526579-39 2024-04-26T00:27:31.852 INFO:tasks.rados.rados.0.smithi045.stdout:1250: finishing write tid 3 to smithi04526579-39 2024-04-26T00:27:31.852 INFO:tasks.rados.rados.0.smithi045.stdout:1250: finishing write tid 4 to smithi04526579-39 2024-04-26T00:27:31.852 INFO:tasks.rados.rados.0.smithi045.stdout:1250: finishing write tid 5 to smithi04526579-39 2024-04-26T00:27:31.852 INFO:tasks.rados.rados.0.smithi045.stdout:1250: oid 39 updating version 0 to 454 2024-04-26T00:27:31.852 INFO:tasks.rados.rados.0.smithi045.stdout:1250: oid 39 version 454 is already newer than 451 2024-04-26T00:27:31.852 INFO:tasks.rados.rados.0.smithi045.stdout:1250: oid 39 version 454 is already newer than 453 2024-04-26T00:27:31.852 INFO:tasks.rados.rados.0.smithi045.stdout:1250: oid 39 version 454 is already newer than 452 2024-04-26T00:27:31.852 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 454 (ObjNum 468 snap 146 seq_num 468) dirty exists 2024-04-26T00:27:31.852 INFO:tasks.rados.rados.0.smithi045.stdout:1250: left oid 39 (ObjNum 468 snap 146 seq_num 468) 2024-04-26T00:27:31.852 INFO:tasks.rados.rados.0.smithi045.stdout:1251: finishing write tid 1 to smithi04526579-5 2024-04-26T00:27:31.852 INFO:tasks.rados.rados.0.smithi045.stdout:1251: finishing write tid 2 to smithi04526579-5 2024-04-26T00:27:31.852 INFO:tasks.rados.rados.0.smithi045.stdout:1249: done (4 left) 2024-04-26T00:27:31.852 INFO:tasks.rados.rados.0.smithi045.stdout:1250: done (3 left) 2024-04-26T00:27:31.852 INFO:tasks.rados.rados.0.smithi045.stdout:1254: snap_create 2024-04-26T00:27:31.854 INFO:tasks.rados.rados.0.smithi045.stdout:1251: finishing write tid 3 to smithi04526579-5 2024-04-26T00:27:31.855 INFO:tasks.rados.rados.0.smithi045.stdout:1251: finishing write tid 4 to smithi04526579-5 2024-04-26T00:27:31.855 INFO:tasks.rados.rados.0.smithi045.stdout:1251: finishing write tid 5 to smithi04526579-5 2024-04-26T00:27:31.855 INFO:tasks.rados.rados.0.smithi045.stdout:1251: oid 5 updating version 0 to 962 2024-04-26T00:27:31.855 INFO:tasks.rados.rados.0.smithi045.stdout:1251: oid 5 updating version 962 to 964 2024-04-26T00:27:31.855 INFO:tasks.rados.rados.0.smithi045.stdout:1251: oid 5 version 964 is already newer than 961 2024-04-26T00:27:31.855 INFO:tasks.rados.rados.0.smithi045.stdout:1251: oid 5 version 964 is already newer than 963 2024-04-26T00:27:31.855 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 964 (ObjNum 469 snap 146 seq_num 469) dirty exists 2024-04-26T00:27:31.855 INFO:tasks.rados.rados.0.smithi045.stdout:1251: left oid 5 (ObjNum 469 snap 146 seq_num 469) 2024-04-26T00:27:31.857 INFO:tasks.rados.rados.0.smithi045.stdout:1252: expect (ObjNum 136 snap 30 seq_num 136) 2024-04-26T00:27:31.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc314c83640 10 monclient: get_monmap_and_config 2024-04-26T00:27:31.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc314c83640 10 monclient: build_initial_monmap 2024-04-26T00:27:31.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc314c83640 10 monclient: monmap: 2024-04-26T00:27:31.959 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:27:31.959 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:31.959 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:27:31.976555+0000 2024-04-26T00:27:31.959 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:27:31.976555+0000 2024-04-26T00:27:31.959 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:27:31.959 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:31.959 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:27:31.959 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:27:31.959 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:27:31.960 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:27:31.960 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:27:31.960 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:31.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc314c83640 1 Processor -- start 2024-04-26T00:27:31.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc314c83640 1 -- start start 2024-04-26T00:27:31.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc314c83640 10 monclient: init 2024-04-26T00:27:31.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc314c83640 10 monclient: _reopen_session rank -1 2024-04-26T00:27:31.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc314c83640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-04-26T00:27:31.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc314c83640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc310136f20 0x7fc31009bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:31.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc314c83640 10 monclient(hunting): picked mon.noname-a con 0x7fc310136f20 addr v2:172.21.15.37:3300/0 2024-04-26T00:27:31.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc314c83640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc310137300 0x7fc3101377b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:31.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc314c83640 10 monclient(hunting): picked mon.noname-c con 0x7fc310137300 addr v2:172.21.15.45:3300/0 2024-04-26T00:27:31.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc314c83640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc310133c70 addr v1:172.21.15.45:6789/0 2024-04-26T00:27:31.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc314c83640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3100a50f0 con 0x7fc310133c70 2024-04-26T00:27:31.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc314c83640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:31.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc314c83640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fc31009b330 con 0x7fc310136f20 2024-04-26T00:27:31.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc314c83640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:31.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc314c83640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc31009b610 con 0x7fc310137300 2024-04-26T00:27:31.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc314c83640 10 monclient(hunting): _renew_subs 2024-04-26T00:27:31.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc314c83640 10 monclient(hunting): authenticate will time out at 1433.282206s 2024-04-26T00:27:31.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc30e59c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc310136f20 0x7fc31009bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:31.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc30e59c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc310136f20 0x7fc31009bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50344/0 (socket says 172.21.15.37:50344) 2024-04-26T00:27:31.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc30e59c640 1 -- 172.21.15.37:0/343753429 learned_addr learned my addr 172.21.15.37:0/343753429 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:27:31.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc30e59c640 10 monclient(hunting): get_auth_request con 0x7fc310136f20 auth_method 0 2024-04-26T00:27:31.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc30e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:31.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc30e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:31.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc30e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:31.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc30e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:31.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc30e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:31.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc30e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:31.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc30dd9b640 1 --2- 172.21.15.37:0/343753429 >> v2:172.21.15.45:3300/0 conn(0x7fc310137300 0x7fc3101377b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:31.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc30dd9b640 10 monclient(hunting): get_auth_request con 0x7fc310137300 auth_method 0 2024-04-26T00:27:31.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc30dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:31.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc30dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:31.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc30dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:31.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc30e59c640 10 monclient(hunting): handle_auth_done global_id 94131 payload 274 2024-04-26T00:27:31.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc30e59c640 1 -- 172.21.15.37:0/343753429 >> v2:172.21.15.45:3300/0 conn(0x7fc310137300 msgr2=0x7fc3101377b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:27:31.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc30e59c640 1 --2- 172.21.15.37:0/343753429 >> v2:172.21.15.45:3300/0 conn(0x7fc310137300 0x7fc3101377b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:31.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc30e59c640 1 -- 172.21.15.37:0/343753429 >> v1:172.21.15.45:6789/0 conn(0x7fc310133c70 legacy=0x7fc310134120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:27:31.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc30e59c640 10 monclient: _finish_hunting 0 2024-04-26T00:27:31.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc30e59c640 1 monclient: found mon.noname-a 2024-04-26T00:27:31.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc30e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:27:31.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc30e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:27:31.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc30e59c640 1 -- 172.21.15.37:0/343753429 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc31009b8f0 con 0x7fc310136f20 2024-04-26T00:27:31.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc30e59c640 1 -- 172.21.15.37:0/343753429 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc310136f20 msgr2=0x7fc31009bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:31.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc30e59c640 1 --2- 172.21.15.37:0/343753429 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc310136f20 0x7fc31009bff0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc3100a5ec0 tx=0x7fc2f801c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fff3b667b2e5a41 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:31.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc30d59a640 1 -- 172.21.15.37:0/343753429 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f80b0070 con 0x7fc310136f20 2024-04-26T00:27:31.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc30d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:31.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc30d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:27:31.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc30d59a640 10 monclient: dump: 2024-04-26T00:27:31.973 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:31.973 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:31.973 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:31.973 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:31.973 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:31.973 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:31.973 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:31.973 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:31.974 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:31.974 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:31.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc30d59a640 10 monclient: _finish_auth 0 2024-04-26T00:27:31.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc30d59a640 10 monclient: _check_auth_tickets 2024-04-26T00:27:31.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc30d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:27:31.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc30d59a640 1 -- 172.21.15.37:0/343753429 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2f80ab600 con 0x7fc310136f20 2024-04-26T00:27:31.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc30d59a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:27:31.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc30d59a640 1 -- 172.21.15.37:0/343753429 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f80ab9d0 con 0x7fc310136f20 2024-04-26T00:27:31.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc30d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:31.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc30d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:27:31.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc30d59a640 10 monclient: dump: 2024-04-26T00:27:31.975 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:31.975 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:31.975 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:31.975 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:31.975 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:31.975 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:31.975 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:31.975 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:31.975 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:31.975 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:31.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc314c83640 5 monclient: authenticate success, global_id 94131 2024-04-26T00:27:31.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc314c83640 10 monclient: get_monmap_and_config success 2024-04-26T00:27:31.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc314c83640 4 set_mon_vals no callback set 2024-04-26T00:27:31.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc314c83640 10 monclient: shutdown 2024-04-26T00:27:31.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc314c83640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:27:31.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc314c83640 1 -- 172.21.15.37:0/343753429 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc310136f20 msgr2=0x7fc31009bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:31.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc314c83640 1 --2- 172.21.15.37:0/343753429 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc310136f20 0x7fc31009bff0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc3100a5ec0 tx=0x7fc2f801c430 comp rx=0 tx=0).stop 2024-04-26T00:27:31.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc314c83640 1 -- 172.21.15.37:0/343753429 shutdown_connections 2024-04-26T00:27:31.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc314c83640 1 --2- 172.21.15.37:0/343753429 >> v2:172.21.15.45:3300/0 conn(0x7fc310137300 0x7fc3101377b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:31.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.975+0000 7fc314c83640 1 --2- 172.21.15.37:0/343753429 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc310136f20 0x7fc31009bff0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:31.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc314c83640 1 -- 172.21.15.37:0/343753429 >> 172.21.15.37:0/343753429 conn(0x7fc3100a61b0 msgr2=0x7fc3100a65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:27:31.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc314c83640 1 -- 172.21.15.37:0/343753429 shutdown_connections 2024-04-26T00:27:31.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc314c83640 1 -- 172.21.15.37:0/343753429 wait complete. 2024-04-26T00:27:31.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc314c83640 10 monclient: build_initial_monmap 2024-04-26T00:27:31.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc314c83640 10 monclient: monmap: 2024-04-26T00:27:31.977 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:27:31.977 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:27:31.977 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:27:31.977 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:27:31.977 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:27:31.977 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:27:31.977 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:27:31.977 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:27:31.978 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:27:31.978 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:31.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc314c83640 1 Processor -- start 2024-04-26T00:27:31.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc314c83640 1 -- start start 2024-04-26T00:27:31.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc314c83640 10 monclient: init 2024-04-26T00:27:31.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc314c83640 10 monclient: _reopen_session rank -1 2024-04-26T00:27:31.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc314c83640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:27:31.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc314c83640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc310133c70 0x7fc3101cf030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:31.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc314c83640 10 monclient(hunting): picked mon.noname-b con 0x7fc310133c70 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:27:31.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc314c83640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc310136f20 0x7fc31013e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:31.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc314c83640 10 monclient(hunting): picked mon.noname-a con 0x7fc310136f20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:27:31.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc314c83640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc310137300 0x7fc31013e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:31.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc314c83640 10 monclient(hunting): picked mon.noname-c con 0x7fc310137300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:27:31.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc314c83640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:31.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc314c83640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fc310141e10 con 0x7fc310136f20 2024-04-26T00:27:31.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc314c83640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:31.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc314c83640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc310141c90 con 0x7fc310133c70 2024-04-26T00:27:31.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc314c83640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:31.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc314c83640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc310141f90 con 0x7fc310137300 2024-04-26T00:27:31.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc314c83640 10 monclient(hunting): _renew_subs 2024-04-26T00:27:31.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc314c83640 10 monclient(hunting): authenticate will time out at 1433.289419s 2024-04-26T00:27:31.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc30ed9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc310137300 0x7fc31013e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:31.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc30ed9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc310137300 0x7fc31013e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:50002/0 (socket says 172.21.15.37:50002) 2024-04-26T00:27:31.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc30dd9b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc310136f20 0x7fc31013e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:31.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc30ed9d640 1 -- 172.21.15.37:0/4121320186 learned_addr learned my addr 172.21.15.37:0/4121320186 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:27:31.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc30ed9d640 10 monclient(hunting): get_auth_request con 0x7fc310137300 auth_method 0 2024-04-26T00:27:31.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc30ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:31.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc30ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:31.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc30ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:31.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc30dd9b640 10 monclient(hunting): get_auth_request con 0x7fc310136f20 auth_method 0 2024-04-26T00:27:31.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc30dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:31.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc30dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:31.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc30dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:31.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc30ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:31.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc30ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:31.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc30ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:31.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc30dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:31.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc30dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:31.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc30e59c640 1 --2- 172.21.15.37:0/4121320186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc310133c70 0x7fc3101cf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:31.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc30dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:31.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc30e59c640 10 monclient(hunting): get_auth_request con 0x7fc310133c70 auth_method 0 2024-04-26T00:27:31.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc30e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:31.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc30e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:31.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc30e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:31.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc30ed9d640 10 monclient(hunting): handle_auth_done global_id 84224 payload 824 2024-04-26T00:27:31.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc30ed9d640 1 -- 172.21.15.37:0/4121320186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc310133c70 msgr2=0x7fc3101cf030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:31.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc30ed9d640 1 --2- 172.21.15.37:0/4121320186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc310133c70 0x7fc3101cf030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:31.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc30ed9d640 1 -- 172.21.15.37:0/4121320186 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc310136f20 msgr2=0x7fc31013e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:31.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc30ed9d640 1 --2- 172.21.15.37:0/4121320186 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc310136f20 0x7fc31013e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:31.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc30ed9d640 10 monclient: _finish_hunting 0 2024-04-26T00:27:31.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc30ed9d640 1 monclient: found mon.noname-c 2024-04-26T00:27:31.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc30ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:27:31.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc30ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:27:31.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.979+0000 7fc30ed9d640 1 -- 172.21.15.37:0/4121320186 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc31013a800 con 0x7fc310137300 2024-04-26T00:27:31.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc30ed9d640 1 -- 172.21.15.37:0/4121320186 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc310137300 msgr2=0x7fc31013e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:31.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc30ed9d640 1 --2- 172.21.15.37:0/4121320186 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc310137300 0x7fc31013e790 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc30800c780 tx=0x7fc30800cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:31.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc306ffd640 1 -- 172.21.15.37:0/4121320186 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30800ac60 con 0x7fc310137300 2024-04-26T00:27:31.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc306ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:31.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc306ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:27:31.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc306ffd640 10 monclient: dump: 2024-04-26T00:27:31.983 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:31.983 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:31.983 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:31.983 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:31.983 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:31.983 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:31.983 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:31.983 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:31.984 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:31.984 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:31.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc306ffd640 10 monclient: _finish_auth 0 2024-04-26T00:27:31.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc306ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:27:31.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc306ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:27:31.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc306ffd640 1 -- 172.21.15.37:0/4121320186 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc30800ae00 con 0x7fc310137300 2024-04-26T00:27:31.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc314c83640 5 monclient: authenticate success, global_id 84224 2024-04-26T00:27:31.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc314c83640 10 monclient: _renew_subs 2024-04-26T00:27:31.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc314c83640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:27:31.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc314c83640 1 -- 172.21.15.37:0/4121320186 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc31013caf0 con 0x7fc310137300 2024-04-26T00:27:31.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc306ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:27:31.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc306ffd640 1 -- 172.21.15.37:0/4121320186 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc308004bb0 con 0x7fc310137300 2024-04-26T00:27:31.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc306ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:31.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc306ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:27:31.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc306ffd640 10 monclient: dump: 2024-04-26T00:27:31.985 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:31.985 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:31.985 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:31.985 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:31.985 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:31.985 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:31.985 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:31.985 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:31.985 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:31.986 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:31.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc30cd99640 4 set_mon_vals no callback set 2024-04-26T00:27:31.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc314c83640 10 monclient: _renew_subs 2024-04-26T00:27:31.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc314c83640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:27:31.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc314c83640 1 -- 172.21.15.37:0/4121320186 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc310135890 con 0x7fc310137300 2024-04-26T00:27:31.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc314c83640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:31.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc314c83640 1 --2- 172.21.15.37:0/4121320186 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc310138b60 0x7fc310138f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:31.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc314c83640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:31.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc314c83640 1 -- 172.21.15.37:0/4121320186 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc31004f510 con 0x7fc310138b60 2024-04-26T00:27:31.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc30e59c640 1 --2- 172.21.15.37:0/4121320186 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc310138b60 0x7fc310138f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:31.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc30e59c640 10 monclient: get_auth_request con 0x7fc310138b60 auth_method 0 2024-04-26T00:27:31.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc30e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:31.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc30e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:31.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc30e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:31.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc30e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:31.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc30e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:31.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc306ffd640 1 -- 172.21.15.37:0/4121320186 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc308005030 con 0x7fc310137300 2024-04-26T00:27:31.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc30e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:31.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc30e59c640 10 monclient(hunting): handle_auth_done global_id 84227 payload 824 2024-04-26T00:27:31.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc30e59c640 1 -- 172.21.15.37:0/4121320186 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc31013b0e0 con 0x7fc310138b60 2024-04-26T00:27:31.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc306ffd640 1 --2- 172.21.15.37:0/4121320186 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc2f0035ed0 0x7fc2f0038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:31.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc306ffd640 1 -- 172.21.15.37:0/4121320186 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(298..298 src has 177..298) ==== 4011+0+0 (secure 0 0 0) 0x7fc308049040 con 0x7fc310137300 2024-04-26T00:27:31.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc30dd9b640 1 --2- 172.21.15.37:0/4121320186 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc2f0035ed0 0x7fc2f0038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:31.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc30dd9b640 10 monclient: get_auth_request con 0x7fc2f0035ed0 auth_method 0 2024-04-26T00:27:31.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc30e59c640 1 -- 172.21.15.37:0/4121320186 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc310138b60 msgr2=0x7fc310138f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:31.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc30e59c640 1 --2- 172.21.15.37:0/4121320186 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc310138b60 0x7fc310138f40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc3100a5530 tx=0x7fc2f80191d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:31.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc306ffd640 1 -- 172.21.15.37:0/4121320186 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f80ba020 con 0x7fc310138b60 2024-04-26T00:27:31.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc306ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:31.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc306ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:27:31.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc306ffd640 10 monclient: dump: 2024-04-26T00:27:31.991 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:31.991 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:31.991 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:31.992 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:31.992 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:31.992 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:31.992 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:31.992 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:31.992 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:31.992 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:31.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc306ffd640 1 -- 172.21.15.37:0/4121320186 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc2f80b6470 con 0x7fc310138b60 2024-04-26T00:27:31.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc306ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:31.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc306ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:27:31.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc306ffd640 1 -- 172.21.15.37:0/4121320186 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc310138b60 msgr2=0x7fc310138f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:31.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc306ffd640 1 --2- 172.21.15.37:0/4121320186 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc310138b60 0x7fc310138f40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc3100a5530 tx=0x7fc2f80191d0 comp rx=0 tx=0).stop 2024-04-26T00:27:31.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc306ffd640 1 -- 172.21.15.37:0/4121320186 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc310138b60 msgr2=0x7fc310138f40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:27:31.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc306ffd640 1 --2- 172.21.15.37:0/4121320186 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc310138b60 0x7fc310138f40 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc3100a5530 tx=0x7fc2f80191d0 comp rx=0 tx=0).stop 2024-04-26T00:27:31.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.987+0000 7fc30dd9b640 1 --2- 172.21.15.37:0/4121320186 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc2f0035ed0 0x7fc2f0038390 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc31013c180 tx=0x7fc300008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:31.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.991+0000 7fc314c83640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:27:31.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.991+0000 7fc314c83640 1 --2- 172.21.15.37:0/4121320186 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc310139480 0x7fc31009d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:31.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.991+0000 7fc314c83640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:31.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.991+0000 7fc314c83640 1 -- 172.21.15.37:0/4121320186 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc310075f10 con 0x7fc310139480 2024-04-26T00:27:31.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.991+0000 7fc30e59c640 1 --2- 172.21.15.37:0/4121320186 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc310139480 0x7fc31009d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:31.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.991+0000 7fc30e59c640 10 monclient: get_auth_request con 0x7fc310139480 auth_method 0 2024-04-26T00:27:31.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.991+0000 7fc30e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:31.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.991+0000 7fc30e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:31.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.991+0000 7fc30e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:31.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.991+0000 7fc30e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:31.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.991+0000 7fc30e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:31.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.991+0000 7fc30e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:31.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.991+0000 7fc30e59c640 10 monclient(hunting): handle_auth_done global_id 84230 payload 824 2024-04-26T00:27:31.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.991+0000 7fc30e59c640 1 -- 172.21.15.37:0/4121320186 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc310139860 con 0x7fc310139480 2024-04-26T00:27:31.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.995+0000 7fc30e59c640 1 -- 172.21.15.37:0/4121320186 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc310139480 msgr2=0x7fc31009d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:31.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.995+0000 7fc30e59c640 1 --2- 172.21.15.37:0/4121320186 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc310139480 0x7fc31009d320 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8019040 tx=0x7fc2f80a8000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:31.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.995+0000 7fc306ffd640 1 -- 172.21.15.37:0/4121320186 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f80ba030 con 0x7fc310139480 2024-04-26T00:27:31.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.995+0000 7fc306ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:31.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.995+0000 7fc306ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:27:31.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.995+0000 7fc306ffd640 10 monclient: dump: 2024-04-26T00:27:31.995 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:31.995 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:31.995 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:31.995 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:31.995 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:31.995 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:31.995 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:31.995 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:31.995 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:31.995 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:31.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.995+0000 7fc306ffd640 1 -- 172.21.15.37:0/4121320186 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4716 (secure 0 0 0) 0x7fc2f80b6d00 con 0x7fc310139480 2024-04-26T00:27:31.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.995+0000 7fc306ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:27:31.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.995+0000 7fc306ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:27:31.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.995+0000 7fc306ffd640 1 -- 172.21.15.37:0/4121320186 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc310139480 msgr2=0x7fc31009d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:31.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.995+0000 7fc306ffd640 1 --2- 172.21.15.37:0/4121320186 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc310139480 0x7fc31009d320 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8019040 tx=0x7fc2f80a8000 comp rx=0 tx=0).stop 2024-04-26T00:27:31.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.995+0000 7fc306ffd640 1 -- 172.21.15.37:0/4121320186 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc310139480 msgr2=0x7fc31009d320 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:27:31.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.995+0000 7fc306ffd640 1 --2- 172.21.15.37:0/4121320186 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc310139480 0x7fc31009d320 secure :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8019040 tx=0x7fc2f80a8000 comp rx=0 tx=0).stop 2024-04-26T00:27:31.996 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:27:31.996 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:27:31.997 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:27:31.997 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:27:31.997 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 58, 2024-04-26T00:27:31.997 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 56931, 2024-04-26T00:27:31.997 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:27:31.997 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:27:31.997 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:27:31.997 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:27:31.997 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:31.997 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 6, 2024-04-26T00:27:31.997 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:27:31.998 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:27:31.998 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:27:31.998 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:27:31.998 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:27:31.998 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:27:31.998 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:27:31.998 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:27:31.998 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:27:31.998 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:27:31.998 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:27:31.998 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:27:31.998 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:27:31.998 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:27:31.999 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:31.999 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:27:31.999 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:27:31.999 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:27:31.999 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:27:31.999 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:27:31.999 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:27:31.999 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:27:31.999 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:27:31.999 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:27:31.999 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:27:31.999 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:27:32.000 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:27:32.000 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:27:32.000 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:32.000 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:32.000 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:27:32.000 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:27:32.000 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:27:32.000 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:27:32.000 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:27:32.000 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:27:32.000 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:27:32.000 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:27:32.000 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:27:32.001 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:27:32.001 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:27:32.001 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:27:32.001 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:27:32.001 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:27:32.001 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:27:32.001 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:27:32.001 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:27:32.001 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:27:32.001 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:27:32.001 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:27:32.001 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:27:32.002 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:27:32.002 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:27:32.002 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:27:32.002 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:27:32.002 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:27:32.002 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:27:32.002 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:27:32.002 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:32.002 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:27:32.002 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:32.002 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:27:32.002 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:32.003 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:27:32.003 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:27:32.003 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:27:32.003 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:27:32.003 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:32.003 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:27:32.003 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:27:32.003 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:32.003 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:32.003 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:32.003 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:27:32.003 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:27:32.004 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:32.004 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:32.004 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:32.004 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:32.004 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:27:32.004 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:27:32.004 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:27:32.004 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:27:32.004 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:27:32.004 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:32.004 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:32.004 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:27:32.005 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:27:32.005 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:27:32.005 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:27:32.005 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:32.005 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:27:32.005 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:27:32.005 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:32.005 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:32.005 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:32.005 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:27:32.005 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:27:32.006 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:32.006 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:32.006 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:32.006 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:32.006 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:27:32.006 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:27:32.006 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:27:32.006 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:27:32.006 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:27:32.006 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:32.006 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:32.007 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:27:32.007 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:27:32.007 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:27:32.007 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:27:32.007 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:32.007 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:27:32.007 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:27:32.007 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:32.007 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:32.007 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:32.007 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:27:32.007 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:27:32.007 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:32.007 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:32.008 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:32.008 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:32.008 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:27:32.008 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:27:32.008 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:27:32.008 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:27:32.008 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:27:32.008 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:32.008 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:32.008 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:32.008 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:27:32.008 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:27:32.009 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:32.009 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:27:32.009 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:27:32.009 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:27:32.009 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:32.009 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:32.009 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:27:32.009 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:32.009 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:27:32.009 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:27:32.009 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:27:32.009 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:32.010 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:32.010 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:27:32.010 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:32.010 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:27:32.010 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:27:32.010 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:27:32.010 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:32.010 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:32.010 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:32.010 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:27:32.010 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:27:32.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.995+0000 7fc314c83640 1 -- 172.21.15.37:0/4121320186 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc2f0035ed0 msgr2=0x7fc2f0038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:32.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.995+0000 7fc314c83640 1 --2- 172.21.15.37:0/4121320186 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc2f0035ed0 0x7fc2f0038390 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc31013c180 tx=0x7fc300008040 comp rx=0 tx=0).stop 2024-04-26T00:27:32.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.995+0000 7fc314c83640 10 monclient: shutdown 2024-04-26T00:27:32.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.995+0000 7fc314c83640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:27:32.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.995+0000 7fc314c83640 1 -- 172.21.15.37:0/4121320186 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc310137300 msgr2=0x7fc31013e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:32.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.995+0000 7fc314c83640 1 --2- 172.21.15.37:0/4121320186 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc310137300 0x7fc31013e790 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc30800c780 tx=0x7fc30800cc40 comp rx=0 tx=0).stop 2024-04-26T00:27:32.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.995+0000 7fc30ed9d640 1 -- 172.21.15.37:0/4121320186 reap_dead start 2024-04-26T00:27:32.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.995+0000 7fc314c83640 1 -- 172.21.15.37:0/4121320186 shutdown_connections 2024-04-26T00:27:32.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.995+0000 7fc314c83640 1 --2- 172.21.15.37:0/4121320186 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc310137300 0x7fc31013e790 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:32.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.995+0000 7fc314c83640 1 -- 172.21.15.37:0/4121320186 >> 172.21.15.37:0/4121320186 conn(0x7fc3100a61b0 msgr2=0x7fc3100a7b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:27:32.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.995+0000 7fc314c83640 1 -- 172.21.15.37:0/4121320186 shutdown_connections 2024-04-26T00:27:32.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:31.995+0000 7fc314c83640 1 -- 172.21.15.37:0/4121320186 wait complete. 2024-04-26T00:27:32.013 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:27:32.014 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:27:32.014 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:27:32.035 INFO:tasks.rados.rados.0.smithi045.stdout:1253: finishing copy_from racing read to smithi04526579-32 2024-04-26T00:27:32.035 INFO:tasks.rados.rados.0.smithi045.stdout:1253: finishing copy_from to smithi04526579-32 2024-04-26T00:27:32.035 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 346 (ObjNum 337 snap 100 seq_num 337) dirty exists 2024-04-26T00:27:32.035 INFO:tasks.rados.rados.0.smithi045.stdout:1251: done (3 left) 2024-04-26T00:27:32.035 INFO:tasks.rados.rados.0.smithi045.stdout:1252: done (2 left) 2024-04-26T00:27:32.036 INFO:tasks.rados.rados.0.smithi045.stdout:1253: done (1 left) 2024-04-26T00:27:32.036 INFO:tasks.rados.rados.0.smithi045.stdout:1254: done (0 left) 2024-04-26T00:27:32.036 INFO:tasks.rados.rados.0.smithi045.stdout:1255: snap_remove snap 137 2024-04-26T00:27:32.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.139+0000 7f96b5f63640 10 monclient: get_monmap_and_config 2024-04-26T00:27:32.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.139+0000 7f96b5f63640 10 monclient: build_initial_monmap 2024-04-26T00:27:32.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.139+0000 7f96b5f63640 10 monclient: monmap: 2024-04-26T00:27:32.125 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:27:32.125 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:32.126 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:27:32.142982+0000 2024-04-26T00:27:32.126 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:27:32.142982+0000 2024-04-26T00:27:32.126 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:27:32.126 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:32.126 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:27:32.126 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:27:32.126 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:27:32.126 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:27:32.126 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:27:32.126 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:32.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.139+0000 7f96b5f63640 1 Processor -- start 2024-04-26T00:27:32.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.139+0000 7f96b5f63640 1 -- start start 2024-04-26T00:27:32.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.139+0000 7f96b5f63640 10 monclient: init 2024-04-26T00:27:32.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.143+0000 7f96b5f63640 10 monclient: _reopen_session rank -1 2024-04-26T00:27:32.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.143+0000 7f96b5f63640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-04-26T00:27:32.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.143+0000 7f96b5f63640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f96b00726b0 0x7f96b0072a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:32.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.143+0000 7f96b5f63640 10 monclient(hunting): picked mon.noname-c con 0x7f96b00726b0 addr v2:172.21.15.45:3300/0 2024-04-26T00:27:32.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.143+0000 7f96b5f63640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f96b00a0110 addr v1:172.21.15.45:6789/0 2024-04-26T00:27:32.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.143+0000 7f96b5f63640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f96b01374e0 addr v1:172.21.15.37:6789/0 2024-04-26T00:27:32.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.143+0000 7f96b5f63640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96b00a1470 con 0x7f96b01374e0 2024-04-26T00:27:32.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.143+0000 7f96b5f63640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96b00a0bd0 con 0x7f96b00a0110 2024-04-26T00:27:32.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.143+0000 7f96b5f63640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:32.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.143+0000 7f96b5f63640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f96b00a1bd0 con 0x7f96b00726b0 2024-04-26T00:27:32.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.143+0000 7f96b5f63640 10 monclient(hunting): _renew_subs 2024-04-26T00:27:32.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.143+0000 7f96b5f63640 10 monclient(hunting): authenticate will time out at 1433.448527s 2024-04-26T00:27:32.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.143+0000 7f96affff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f96b01374e0 0x7f96b013f110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:44860/0 (socket says 172.21.15.37:44860) 2024-04-26T00:27:32.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.143+0000 7f96affff640 1 -- 172.21.15.37:0/3724912764 learned_addr learned my addr 172.21.15.37:0/3724912764 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:27:32.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.143+0000 7f96af7fe640 1 --2- 172.21.15.37:0/3724912764 >> v2:172.21.15.45:3300/0 conn(0x7f96b00726b0 0x7f96b0072a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:32.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.143+0000 7f96af7fe640 10 monclient(hunting): get_auth_request con 0x7f96b00726b0 auth_method 0 2024-04-26T00:27:32.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.143+0000 7f96af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:32.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.143+0000 7f96af7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:32.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.143+0000 7f96af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:32.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.143+0000 7f96af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:32.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.143+0000 7f96af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:32.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.143+0000 7f96af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:32.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.143+0000 7f96af7fe640 10 monclient(hunting): handle_auth_done global_id 24712 payload 274 2024-04-26T00:27:32.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.143+0000 7f96af7fe640 1 -- 172.21.15.37:0/3724912764 >> v1:172.21.15.45:6789/0 conn(0x7f96b00a0110 legacy=0x7f96b0136dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:27:32.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.143+0000 7f96af7fe640 1 -- 172.21.15.37:0/3724912764 >> v1:172.21.15.37:6789/0 conn(0x7f96b01374e0 legacy=0x7f96b013f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:27:32.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.143+0000 7f96af7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:27:32.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.143+0000 7f96af7fe640 1 monclient: found mon.noname-c 2024-04-26T00:27:32.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.143+0000 7f96af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:27:32.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.143+0000 7f96af7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:27:32.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.143+0000 7f96af7fe640 1 -- 172.21.15.37:0/3724912764 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96b00a1190 con 0x7f96b00726b0 2024-04-26T00:27:32.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96af7fe640 1 -- 172.21.15.37:0/3724912764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b00726b0 msgr2=0x7f96b0072a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:32.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96af7fe640 1 --2- 172.21.15.37:0/3724912764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b00726b0 0x7f96b0072a90 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f96a00097c0 tx=0x7f96a001c100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81d90cf31a6067fa server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:32.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96aeffd640 1 -- 172.21.15.37:0/3724912764 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a00b1070 con 0x7f96b00726b0 2024-04-26T00:27:32.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:32.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96aeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:27:32.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96aeffd640 10 monclient: dump: 2024-04-26T00:27:32.136 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:32.136 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:32.136 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:32.136 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:32.136 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:32.136 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:32.136 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:32.136 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:32.137 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:32.137 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:32.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96aeffd640 10 monclient: _finish_auth 0 2024-04-26T00:27:32.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96aeffd640 10 monclient: _check_auth_tickets 2024-04-26T00:27:32.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:27:32.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96aeffd640 1 -- 172.21.15.37:0/3724912764 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96a00ac440 con 0x7f96b00726b0 2024-04-26T00:27:32.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96aeffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:27:32.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96aeffd640 1 -- 172.21.15.37:0/3724912764 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a00ac810 con 0x7f96b00726b0 2024-04-26T00:27:32.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96b5f63640 5 monclient: authenticate success, global_id 24712 2024-04-26T00:27:32.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96b5f63640 10 monclient: get_monmap_and_config success 2024-04-26T00:27:32.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96b5f63640 4 set_mon_vals no callback set 2024-04-26T00:27:32.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:32.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96aeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:27:32.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96aeffd640 10 monclient: dump: 2024-04-26T00:27:32.137 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:32.137 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:32.138 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:32.138 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:32.138 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:32.138 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:32.138 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:32.138 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:32.138 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:32.138 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:32.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96b5f63640 10 monclient: shutdown 2024-04-26T00:27:32.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96b5f63640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:27:32.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96b5f63640 1 -- 172.21.15.37:0/3724912764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b00726b0 msgr2=0x7f96b0072a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:32.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96b5f63640 1 --2- 172.21.15.37:0/3724912764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b00726b0 0x7f96b0072a90 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f96a00097c0 tx=0x7f96a001c100 comp rx=0 tx=0).stop 2024-04-26T00:27:32.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96b5f63640 1 -- 172.21.15.37:0/3724912764 shutdown_connections 2024-04-26T00:27:32.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96b5f63640 1 --2- 172.21.15.37:0/3724912764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b00726b0 0x7f96b0072a90 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:32.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96b5f63640 1 -- 172.21.15.37:0/3724912764 >> 172.21.15.37:0/3724912764 conn(0x7f96b00a1d50 msgr2=0x7f96b00a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:27:32.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96b5f63640 1 -- 172.21.15.37:0/3724912764 shutdown_connections 2024-04-26T00:27:32.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96b5f63640 1 -- 172.21.15.37:0/3724912764 wait complete. 2024-04-26T00:27:32.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96b5f63640 10 monclient: build_initial_monmap 2024-04-26T00:27:32.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96b5f63640 10 monclient: monmap: 2024-04-26T00:27:32.144 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:27:32.144 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:27:32.144 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:27:32.144 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:27:32.144 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:27:32.144 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:27:32.144 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:27:32.144 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:27:32.145 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:27:32.145 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:32.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96b5f63640 1 Processor -- start 2024-04-26T00:27:32.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96b5f63640 1 -- start start 2024-04-26T00:27:32.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96b5f63640 10 monclient: init 2024-04-26T00:27:32.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96b5f63640 10 monclient: _reopen_session rank -1 2024-04-26T00:27:32.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96b5f63640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:27:32.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96b5f63640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f96b00726b0 0x7f96b013eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:32.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96b5f63640 10 monclient(hunting): picked mon.noname-a con 0x7f96b00726b0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:27:32.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96b5f63640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b00a0110 0x7f96b013a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:32.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96b5f63640 10 monclient(hunting): picked mon.noname-b con 0x7f96b00a0110 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:27:32.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96b5f63640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f96b01374e0 0x7f96b013ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:32.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96b5f63640 10 monclient(hunting): picked mon.noname-c con 0x7f96b01374e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:27:32.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96b5f63640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:32.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96b5f63640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f96b01435e0 con 0x7f96b00726b0 2024-04-26T00:27:32.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96b5f63640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:32.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96b5f63640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f96b0143460 con 0x7f96b00a0110 2024-04-26T00:27:32.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96b5f63640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:32.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96b5f63640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f96b0143760 con 0x7f96b01374e0 2024-04-26T00:27:32.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96b5f63640 10 monclient(hunting): _renew_subs 2024-04-26T00:27:32.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96b5f63640 10 monclient(hunting): authenticate will time out at 1433.460474s 2024-04-26T00:27:32.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96af7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f96b00726b0 0x7f96b013eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:32.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96af7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f96b00726b0 0x7f96b013eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50362/0 (socket says 172.21.15.37:50362) 2024-04-26T00:27:32.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96af7fe640 1 -- 172.21.15.37:0/1983836382 learned_addr learned my addr 172.21.15.37:0/1983836382 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:27:32.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96affff640 1 --2- 172.21.15.37:0/1983836382 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f96b01374e0 0x7f96b013ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:32.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96af7fe640 10 monclient(hunting): get_auth_request con 0x7f96b00726b0 auth_method 0 2024-04-26T00:27:32.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:32.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96af7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:32.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:32.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96affff640 10 monclient(hunting): get_auth_request con 0x7f96b01374e0 auth_method 0 2024-04-26T00:27:32.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:32.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96affff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:32.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96affff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:32.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:32.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:32.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:32.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:32.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:32.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:32.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96a7fff640 1 --2- 172.21.15.37:0/1983836382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b00a0110 0x7f96b013a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:32.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96a7fff640 10 monclient(hunting): get_auth_request con 0x7f96b00a0110 auth_method 0 2024-04-26T00:27:32.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:32.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96a7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:32.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:32.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96affff640 10 monclient(hunting): handle_auth_done global_id 84233 payload 824 2024-04-26T00:27:32.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96affff640 1 -- 172.21.15.37:0/1983836382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b00a0110 msgr2=0x7f96b013a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:32.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96affff640 1 --2- 172.21.15.37:0/1983836382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b00a0110 0x7f96b013a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:32.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96affff640 1 -- 172.21.15.37:0/1983836382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f96b00726b0 msgr2=0x7f96b013eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:32.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96affff640 1 --2- 172.21.15.37:0/1983836382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f96b00726b0 0x7f96b013eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:32.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96affff640 10 monclient: _finish_hunting 0 2024-04-26T00:27:32.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96affff640 1 monclient: found mon.noname-c 2024-04-26T00:27:32.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:27:32.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96affff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:27:32.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96affff640 1 -- 172.21.15.37:0/1983836382 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96b013e620 con 0x7f96b01374e0 2024-04-26T00:27:32.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96affff640 1 -- 172.21.15.37:0/1983836382 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f96b01374e0 msgr2=0x7f96b013ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:32.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96affff640 1 --2- 172.21.15.37:0/1983836382 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f96b01374e0 0x7f96b013ac20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f96a800cb20 tx=0x7f96a800cf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:32.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96acff9640 1 -- 172.21.15.37:0/1983836382 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a8017070 con 0x7f96b01374e0 2024-04-26T00:27:32.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96acff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:32.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96acff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:27:32.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96acff9640 10 monclient: dump: 2024-04-26T00:27:32.149 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:32.149 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:32.149 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:32.149 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:32.149 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:32.149 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:32.149 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:32.149 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:32.149 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:32.149 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:32.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96acff9640 10 monclient: _finish_auth 0 2024-04-26T00:27:32.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96acff9640 10 monclient: _check_auth_tickets 2024-04-26T00:27:32.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96acff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:27:32.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96acff9640 1 -- 172.21.15.37:0/1983836382 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96a800ade0 con 0x7f96b01374e0 2024-04-26T00:27:32.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96b5f63640 5 monclient: authenticate success, global_id 84233 2024-04-26T00:27:32.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96b5f63640 10 monclient: _renew_subs 2024-04-26T00:27:32.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96b5f63640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:27:32.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96b5f63640 1 -- 172.21.15.37:0/1983836382 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96b013c330 con 0x7f96b01374e0 2024-04-26T00:27:32.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96acff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:27:32.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96acff9640 1 -- 172.21.15.37:0/1983836382 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a8012640 con 0x7f96b01374e0 2024-04-26T00:27:32.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96acff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:32.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96acff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:27:32.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96acff9640 10 monclient: dump: 2024-04-26T00:27:32.150 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:32.151 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:32.151 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:32.151 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:32.151 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:32.151 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:32.151 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:32.151 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:32.151 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:32.151 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:32.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96adffb640 4 set_mon_vals no callback set 2024-04-26T00:27:32.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96b5f63640 10 monclient: _renew_subs 2024-04-26T00:27:32.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96b5f63640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:27:32.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.151+0000 7f96b5f63640 1 -- 172.21.15.37:0/1983836382 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f96b013cc10 con 0x7f96b01374e0 2024-04-26T00:27:32.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.155+0000 7f96b5f63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:32.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.155+0000 7f96b5f63640 1 --2- 172.21.15.37:0/1983836382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b013b160 0x7f96b01312d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:32.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.155+0000 7f96b5f63640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:32.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.155+0000 7f96b5f63640 1 -- 172.21.15.37:0/1983836382 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f96b013f2e0 con 0x7f96b013b160 2024-04-26T00:27:32.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.155+0000 7f96acff9640 1 -- 172.21.15.37:0/1983836382 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f96a801a070 con 0x7f96b01374e0 2024-04-26T00:27:32.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.155+0000 7f96acff9640 1 --2- 172.21.15.37:0/1983836382 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9690035e80 0x7f9690038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:32.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.155+0000 7f96acff9640 1 -- 172.21.15.37:0/1983836382 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(298..298 src has 177..298) ==== 4011+0+0 (secure 0 0 0) 0x7f96a80488c0 con 0x7f96b01374e0 2024-04-26T00:27:32.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.155+0000 7f96af7fe640 1 --2- 172.21.15.37:0/1983836382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b013b160 0x7f96b01312d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:32.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.155+0000 7f96af7fe640 10 monclient: get_auth_request con 0x7f96b013b160 auth_method 0 2024-04-26T00:27:32.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.155+0000 7f96af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:32.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.155+0000 7f96af7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:32.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.155+0000 7f96af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:32.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.155+0000 7f96af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:32.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.155+0000 7f96af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:32.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.155+0000 7f96af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:32.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.155+0000 7f96a7fff640 1 --2- 172.21.15.37:0/1983836382 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9690035e80 0x7f9690038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:32.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.155+0000 7f96a7fff640 10 monclient: get_auth_request con 0x7f9690035e80 auth_method 0 2024-04-26T00:27:32.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.155+0000 7f96af7fe640 10 monclient(hunting): handle_auth_done global_id 24718 payload 824 2024-04-26T00:27:32.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.155+0000 7f96af7fe640 1 -- 172.21.15.37:0/1983836382 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f96b013b540 con 0x7f96b013b160 2024-04-26T00:27:32.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.155+0000 7f96af7fe640 1 -- 172.21.15.37:0/1983836382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b013b160 msgr2=0x7f96b01312d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:32.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.155+0000 7f96af7fe640 1 --2- 172.21.15.37:0/1983836382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b013b160 0x7f96b01312d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f96a0019d30 tx=0x7f96a0018ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:32.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.155+0000 7f96a7fff640 1 --2- 172.21.15.37:0/1983836382 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9690035e80 0x7f9690038340 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f96b013bf30 tx=0x7f9698009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:32.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.155+0000 7f96acff9640 1 -- 172.21.15.37:0/1983836382 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a00bb030 con 0x7f96b013b160 2024-04-26T00:27:32.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.155+0000 7f96acff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:32.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.155+0000 7f96acff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:27:32.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.155+0000 7f96acff9640 10 monclient: dump: 2024-04-26T00:27:32.155 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:32.155 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:32.155 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:32.155 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:32.155 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:32.155 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:32.155 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:32.155 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:32.155 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:32.156 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:32.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.159+0000 7f96acff9640 1 -- 172.21.15.37:0/1983836382 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f96a00ace40 con 0x7f96b013b160 2024-04-26T00:27:32.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.159+0000 7f96acff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:32.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.159+0000 7f96acff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:27:32.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.159+0000 7f96acff9640 1 -- 172.21.15.37:0/1983836382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b013b160 msgr2=0x7f96b01312d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:32.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.159+0000 7f96acff9640 1 --2- 172.21.15.37:0/1983836382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b013b160 0x7f96b01312d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f96a0019d30 tx=0x7f96a0018ea0 comp rx=0 tx=0).stop 2024-04-26T00:27:32.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.159+0000 7f96acff9640 1 -- 172.21.15.37:0/1983836382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b013b160 msgr2=0x7f96b01312d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:27:32.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.159+0000 7f96acff9640 1 --2- 172.21.15.37:0/1983836382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b013b160 0x7f96b01312d0 secure :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f96a0019d30 tx=0x7f96a0018ea0 comp rx=0 tx=0).stop 2024-04-26T00:27:32.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.167+0000 7f96b5f63640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:27:32.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.167+0000 7f96b5f63640 1 --2- 172.21.15.37:0/1983836382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b002c000 0x7f96b002c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:32.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.167+0000 7f96b5f63640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:32.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.167+0000 7f96b5f63640 1 -- 172.21.15.37:0/1983836382 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f96b004f510 con 0x7f96b002c000 2024-04-26T00:27:32.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.167+0000 7f96af7fe640 1 --2- 172.21.15.37:0/1983836382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b002c000 0x7f96b002c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:32.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.167+0000 7f96af7fe640 10 monclient: get_auth_request con 0x7f96b002c000 auth_method 0 2024-04-26T00:27:32.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.167+0000 7f96af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:32.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.167+0000 7f96af7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:32.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.167+0000 7f96af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:32.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.167+0000 7f96af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:32.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.167+0000 7f96af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:32.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.167+0000 7f96af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:32.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.167+0000 7f96af7fe640 10 monclient(hunting): handle_auth_done global_id 24721 payload 824 2024-04-26T00:27:32.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.167+0000 7f96af7fe640 1 -- 172.21.15.37:0/1983836382 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f96b0073280 con 0x7f96b002c000 2024-04-26T00:27:32.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.167+0000 7f96af7fe640 1 -- 172.21.15.37:0/1983836382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b002c000 msgr2=0x7f96b002c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:32.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.167+0000 7f96af7fe640 1 --2- 172.21.15.37:0/1983836382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b002c000 0x7f96b002c410 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f96a0000c10 tx=0x7f96a00a6000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:32.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.167+0000 7f96acff9640 1 -- 172.21.15.37:0/1983836382 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a00bb040 con 0x7f96b002c000 2024-04-26T00:27:32.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.167+0000 7f96acff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:32.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.167+0000 7f96acff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:27:32.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.167+0000 7f96acff9640 10 monclient: dump: 2024-04-26T00:27:32.158 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:32.158 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:32.158 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:32.159 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:32.159 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:32.159 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:32.159 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:32.159 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:32.159 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:32.159 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:32.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.167+0000 7f96acff9640 1 -- 172.21.15.37:0/1983836382 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4889 (secure 0 0 0) 0x7f96a00c0830 con 0x7f96b002c000 2024-04-26T00:27:32.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.167+0000 7f96acff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:27:32.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.167+0000 7f96acff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:27:32.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.167+0000 7f96acff9640 1 -- 172.21.15.37:0/1983836382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b002c000 msgr2=0x7f96b002c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:32.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.167+0000 7f96acff9640 1 --2- 172.21.15.37:0/1983836382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b002c000 0x7f96b002c410 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f96a0000c10 tx=0x7f96a00a6000 comp rx=0 tx=0).stop 2024-04-26T00:27:32.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.167+0000 7f96acff9640 1 -- 172.21.15.37:0/1983836382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b002c000 msgr2=0x7f96b002c410 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:27:32.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.167+0000 7f96acff9640 1 --2- 172.21.15.37:0/1983836382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96b002c000 0x7f96b002c410 secure :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f96a0000c10 tx=0x7f96a00a6000 comp rx=0 tx=0).stop 2024-04-26T00:27:32.160 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:27:32.160 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:27:32.160 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:27:32.160 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:27:32.160 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 58, 2024-04-26T00:27:32.160 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 360338, 2024-04-26T00:27:32.160 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:27:32.160 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:27:32.160 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:27:32.160 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:27:32.160 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:32.161 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 6, 2024-04-26T00:27:32.161 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:27:32.161 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:27:32.161 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:27:32.161 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:27:32.161 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:27:32.161 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:27:32.161 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:27:32.161 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:27:32.161 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:27:32.161 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:27:32.161 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:27:32.161 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:27:32.161 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:27:32.162 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:27:32.162 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:32.162 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:27:32.162 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:27:32.162 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:27:32.162 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:27:32.162 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:27:32.162 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:27:32.162 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:27:32.162 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:27:32.162 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:27:32.162 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:27:32.162 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:27:32.162 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:27:32.163 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:27:32.163 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:32.163 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:32.163 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:27:32.163 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:27:32.163 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:27:32.163 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:27:32.163 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:27:32.163 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:27:32.163 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:27:32.163 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:27:32.163 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:27:32.163 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:27:32.163 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:27:32.164 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:27:32.164 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:27:32.164 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:27:32.164 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:27:32.164 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:27:32.164 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:27:32.164 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:27:32.164 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:27:32.164 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:27:32.164 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:27:32.164 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:27:32.164 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:27:32.164 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:27:32.164 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:27:32.164 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:27:32.165 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:27:32.165 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:27:32.165 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:32.165 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:27:32.165 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:32.165 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:27:32.165 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:32.165 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:27:32.165 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:27:32.165 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:27:32.165 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:27:32.165 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:32.165 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:27:32.165 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:27:32.165 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:32.166 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:32.166 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:32.166 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:27:32.166 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:27:32.166 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:32.166 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:32.166 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:32.166 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:32.166 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:27:32.166 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:27:32.166 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:27:32.166 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:27:32.166 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:27:32.166 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:32.167 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:32.167 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:27:32.167 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:27:32.167 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:27:32.167 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:27:32.167 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:32.167 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:27:32.167 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:27:32.167 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:32.167 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:32.167 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:32.167 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:27:32.167 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:27:32.167 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:32.167 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:32.168 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:32.168 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:32.168 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:27:32.168 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:27:32.168 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:27:32.168 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:27:32.168 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:27:32.168 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:32.168 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:32.168 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:27:32.168 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:27:32.168 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:27:32.168 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:27:32.168 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:32.168 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:27:32.169 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:27:32.169 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:32.169 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:32.169 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:32.169 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:27:32.169 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:27:32.169 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:32.169 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:32.169 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:32.169 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:32.169 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:27:32.169 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:27:32.169 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:27:32.169 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:27:32.170 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:27:32.170 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:32.170 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:32.170 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:32.170 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:27:32.170 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:27:32.170 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:32.170 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:27:32.170 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:27:32.170 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:27:32.170 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:32.170 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:32.170 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:27:32.170 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:32.170 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:27:32.171 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:27:32.171 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:27:32.171 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:32.171 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:32.171 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:27:32.171 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:32.171 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:27:32.171 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:27:32.171 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:27:32.171 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:32.171 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:32.171 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:27:32.171 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:32.171 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:27:32.171 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:27:32.172 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:27:32.172 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:32.172 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:32.172 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:32.172 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:27:32.172 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:27:32.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.167+0000 7f96b5f63640 1 -- 172.21.15.37:0/1983836382 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9690035e80 msgr2=0x7f9690038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:32.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.167+0000 7f96b5f63640 1 --2- 172.21.15.37:0/1983836382 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9690035e80 0x7f9690038340 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f96b013bf30 tx=0x7f9698009040 comp rx=0 tx=0).stop 2024-04-26T00:27:32.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.167+0000 7f96b5f63640 10 monclient: shutdown 2024-04-26T00:27:32.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.167+0000 7f96b5f63640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:27:32.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.167+0000 7f96b5f63640 1 -- 172.21.15.37:0/1983836382 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f96b01374e0 msgr2=0x7f96b013ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:32.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.167+0000 7f96b5f63640 1 --2- 172.21.15.37:0/1983836382 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f96b01374e0 0x7f96b013ac20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f96a800cb20 tx=0x7f96a800cf20 comp rx=0 tx=0).stop 2024-04-26T00:27:32.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.167+0000 7f96affff640 1 -- 172.21.15.37:0/1983836382 reap_dead start 2024-04-26T00:27:32.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.167+0000 7f96b5f63640 1 -- 172.21.15.37:0/1983836382 shutdown_connections 2024-04-26T00:27:32.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.167+0000 7f96b5f63640 1 --2- 172.21.15.37:0/1983836382 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f96b01374e0 0x7f96b013ac20 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:32.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.167+0000 7f96b5f63640 1 -- 172.21.15.37:0/1983836382 >> 172.21.15.37:0/1983836382 conn(0x7f96b00a1d50 msgr2=0x7f96b009d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:27:32.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.167+0000 7f96b5f63640 1 -- 172.21.15.37:0/1983836382 shutdown_connections 2024-04-26T00:27:32.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.171+0000 7f96b5f63640 1 -- 172.21.15.37:0/1983836382 wait complete. 2024-04-26T00:27:32.176 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:27:32.176 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-04-26T00:27:32.176 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-26T00:27:32.176 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-04-26T00:27:32.177 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-04-26T00:27:32.177 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-26T00:27:32.177 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T00:27:32.199 INFO:tasks.ceph.mon.b:Stopped 2024-04-26T00:27:32.200 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-26T00:27:32.200 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:27:32.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc733706640 10 monclient: get_monmap_and_config 2024-04-26T00:27:32.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc733706640 10 monclient: build_initial_monmap 2024-04-26T00:27:32.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc733706640 10 monclient: monmap: 2024-04-26T00:27:32.311 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:27:32.311 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:32.311 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:27:32.328950+0000 2024-04-26T00:27:32.311 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:27:32.328950+0000 2024-04-26T00:27:32.311 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:27:32.311 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:32.312 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:27:32.312 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:27:32.312 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:27:32.312 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:27:32.312 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:27:32.312 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:32.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc733706640 1 Processor -- start 2024-04-26T00:27:32.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc733706640 1 -- start start 2024-04-26T00:27:32.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc733706640 10 monclient: init 2024-04-26T00:27:32.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc733706640 10 monclient: _reopen_session rank -1 2024-04-26T00:27:32.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc733706640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-04-26T00:27:32.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc733706640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc72c136da0 0x7fc72c09bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:32.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc733706640 10 monclient(hunting): picked mon.noname-b con 0x7fc72c136da0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:27:32.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc733706640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc72c09c410 0x7fc72c137180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:32.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc733706640 10 monclient(hunting): picked mon.noname-c con 0x7fc72c09c410 addr v2:172.21.15.45:3300/0 2024-04-26T00:27:32.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc733706640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc72c1376c0 addr v1:172.21.15.45:6789/0 2024-04-26T00:27:32.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc733706640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc72c09b4f0 con 0x7fc72c1376c0 2024-04-26T00:27:32.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc733706640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:32.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc733706640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc72c09b210 con 0x7fc72c09c410 2024-04-26T00:27:32.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc733706640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:32.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc733706640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc72c09b7d0 con 0x7fc72c136da0 2024-04-26T00:27:32.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc733706640 10 monclient(hunting): _renew_subs 2024-04-26T00:27:32.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc733706640 10 monclient(hunting): authenticate will time out at 1433.634562s 2024-04-26T00:27:32.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc7314a2640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc72c136da0 0x7fc72c09bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:32.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc731ca3640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7fc72c1376c0 legacy=0x7fc72c133b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:27:32.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc730ca1640 1 -- >> v2:172.21.15.45:3300/0 conn(0x7fc72c09c410 msgr2=0x7fc72c137180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:27:32.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc7314a2640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc72c136da0 0x7fc72c09bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:50042/0 (socket says 172.21.15.37:50042) 2024-04-26T00:27:32.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc730ca1640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc72c09c410 0x7fc72c137180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:27:32.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc7314a2640 1 -- 172.21.15.37:0/4167742885 learned_addr learned my addr 172.21.15.37:0/4167742885 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:27:32.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc7314a2640 10 monclient(hunting): get_auth_request con 0x7fc72c136da0 auth_method 0 2024-04-26T00:27:32.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc7314a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:32.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc7314a2640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:32.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc7314a2640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:32.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc7314a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:32.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc7314a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:32.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc7314a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:32.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc7314a2640 10 monclient(hunting): handle_auth_done global_id 84236 payload 274 2024-04-26T00:27:32.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc7314a2640 1 -- 172.21.15.37:0/4167742885 >> v2:172.21.15.45:3300/0 conn(0x7fc72c09c410 msgr2=0x7fc72c137180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:27:32.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc7314a2640 1 --2- 172.21.15.37:0/4167742885 >> v2:172.21.15.45:3300/0 conn(0x7fc72c09c410 0x7fc72c137180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:32.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc7314a2640 1 -- 172.21.15.37:0/4167742885 >> v1:172.21.15.45:6789/0 conn(0x7fc72c1376c0 legacy=0x7fc72c133b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:27:32.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc7314a2640 10 monclient: _finish_hunting 0 2024-04-26T00:27:32.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc7314a2640 1 monclient: found mon.noname-b 2024-04-26T00:27:32.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc7314a2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:27:32.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc7314a2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:27:32.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.327+0000 7fc7314a2640 1 -- 172.21.15.37:0/4167742885 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc72c137b30 con 0x7fc72c136da0 2024-04-26T00:27:32.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.331+0000 7fc7314a2640 1 -- 172.21.15.37:0/4167742885 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc72c136da0 msgr2=0x7fc72c09bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:32.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.331+0000 7fc7314a2640 1 --2- 172.21.15.37:0/4167742885 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc72c136da0 0x7fc72c09bed0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc728009870 tx=0x7fc72801c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfb22635130f3dca server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:32.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.331+0000 7fc723fff640 1 -- 172.21.15.37:0/4167742885 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7280b0070 con 0x7fc72c136da0 2024-04-26T00:27:32.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.331+0000 7fc723fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:32.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.331+0000 7fc723fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:27:32.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.331+0000 7fc723fff640 10 monclient: dump: 2024-04-26T00:27:32.318 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:32.318 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:32.318 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:32.318 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:32.318 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:32.318 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:32.318 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:32.318 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:32.318 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:32.318 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:32.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.331+0000 7fc723fff640 10 monclient: _finish_auth 0 2024-04-26T00:27:32.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.331+0000 7fc723fff640 10 monclient: _check_auth_tickets 2024-04-26T00:27:32.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.331+0000 7fc723fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:27:32.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.331+0000 7fc723fff640 1 -- 172.21.15.37:0/4167742885 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7280ab600 con 0x7fc72c136da0 2024-04-26T00:27:32.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.331+0000 7fc733706640 5 monclient: authenticate success, global_id 84236 2024-04-26T00:27:32.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.331+0000 7fc733706640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:27:32.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.331+0000 7fc723fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:27:32.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.331+0000 7fc733706640 10 monclient: get_monmap_and_config success 2024-04-26T00:27:32.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.331+0000 7fc723fff640 1 -- 172.21.15.37:0/4167742885 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7280ab9d0 con 0x7fc72c136da0 2024-04-26T00:27:32.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.331+0000 7fc723fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:32.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.331+0000 7fc733706640 4 set_mon_vals no callback set 2024-04-26T00:27:32.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.331+0000 7fc723fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:27:32.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.331+0000 7fc723fff640 10 monclient: dump: 2024-04-26T00:27:32.319 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:32.319 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:32.320 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:32.320 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:32.320 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:32.320 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:32.320 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:32.320 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:32.320 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:32.320 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:32.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc733706640 10 monclient: shutdown 2024-04-26T00:27:32.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc733706640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:27:32.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc733706640 1 -- 172.21.15.37:0/4167742885 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc72c136da0 msgr2=0x7fc72c09bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:32.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc733706640 1 --2- 172.21.15.37:0/4167742885 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc72c136da0 0x7fc72c09bed0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc728009870 tx=0x7fc72801c400 comp rx=0 tx=0).stop 2024-04-26T00:27:32.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc733706640 1 -- 172.21.15.37:0/4167742885 shutdown_connections 2024-04-26T00:27:32.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc733706640 1 --2- 172.21.15.37:0/4167742885 >> v2:172.21.15.45:3300/0 conn(0x7fc72c09c410 0x7fc72c137180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:32.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc733706640 1 --2- 172.21.15.37:0/4167742885 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc72c136da0 0x7fc72c09bed0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:32.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc733706640 1 -- 172.21.15.37:0/4167742885 >> 172.21.15.37:0/4167742885 conn(0x7fc72c0a5f00 msgr2=0x7fc72c0a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:27:32.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc733706640 1 -- 172.21.15.37:0/4167742885 shutdown_connections 2024-04-26T00:27:32.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc733706640 1 -- 172.21.15.37:0/4167742885 wait complete. 2024-04-26T00:27:32.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc733706640 10 monclient: build_initial_monmap 2024-04-26T00:27:32.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc733706640 10 monclient: monmap: 2024-04-26T00:27:32.325 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:27:32.325 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:27:32.325 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:27:32.325 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:27:32.325 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:27:32.325 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:27:32.325 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:27:32.325 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:27:32.325 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:27:32.325 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:32.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc733706640 1 Processor -- start 2024-04-26T00:27:32.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc733706640 1 -- start start 2024-04-26T00:27:32.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc733706640 10 monclient: init 2024-04-26T00:27:32.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc733706640 10 monclient: _reopen_session rank -1 2024-04-26T00:27:32.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc733706640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:27:32.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc733706640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc72c09c410 0x7fc72c1cf470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:32.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc733706640 10 monclient(hunting): picked mon.noname-b con 0x7fc72c09c410 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:27:32.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc733706640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc72c136da0 0x7fc72c1cf9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:32.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc733706640 10 monclient(hunting): picked mon.noname-a con 0x7fc72c136da0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:27:32.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc733706640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc72c1376c0 0x7fc72c1cfef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:32.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc733706640 10 monclient(hunting): picked mon.noname-c con 0x7fc72c1376c0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:27:32.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc733706640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:32.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc733706640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fc72c141ea0 con 0x7fc72c136da0 2024-04-26T00:27:32.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc733706640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:32.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc733706640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc72c141d20 con 0x7fc72c09c410 2024-04-26T00:27:32.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc733706640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:32.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc733706640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc72c142020 con 0x7fc72c1376c0 2024-04-26T00:27:32.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc733706640 10 monclient(hunting): _renew_subs 2024-04-26T00:27:32.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc733706640 10 monclient(hunting): authenticate will time out at 1433.641772s 2024-04-26T00:27:32.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc731ca3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc72c1376c0 0x7fc72c1cfef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:32.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc730ca1640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc72c136da0 0x7fc72c1cf9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:32.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc7314a2640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc72c09c410 msgr2=0x7fc72c1cf470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:27:32.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc7314a2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc72c09c410 0x7fc72c1cf470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:27:32.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc730ca1640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc72c136da0 0x7fc72c1cf9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50378/0 (socket says 172.21.15.37:50378) 2024-04-26T00:27:32.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc730ca1640 1 -- 172.21.15.37:0/3058147773 learned_addr learned my addr 172.21.15.37:0/3058147773 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:27:32.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc731ca3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc72c1376c0 0x7fc72c1cfef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:50058/0 (socket says 172.21.15.37:50058) 2024-04-26T00:27:32.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc730ca1640 10 monclient(hunting): get_auth_request con 0x7fc72c136da0 auth_method 0 2024-04-26T00:27:32.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc730ca1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:32.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc730ca1640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:32.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc730ca1640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:32.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc731ca3640 10 monclient(hunting): get_auth_request con 0x7fc72c1376c0 auth_method 0 2024-04-26T00:27:32.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc731ca3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:32.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc731ca3640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:32.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc731ca3640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:32.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc730ca1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:32.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc730ca1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:32.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc730ca1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:32.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc731ca3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:32.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc731ca3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:32.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc731ca3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:32.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc730ca1640 10 monclient(hunting): handle_auth_done global_id 94140 payload 824 2024-04-26T00:27:32.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc730ca1640 1 -- 172.21.15.37:0/3058147773 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc72c1376c0 msgr2=0x7fc72c1cfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:32.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc730ca1640 1 --2- 172.21.15.37:0/3058147773 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc72c1376c0 0x7fc72c1cfef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:32.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc730ca1640 1 -- 172.21.15.37:0/3058147773 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc72c09c410 msgr2=0x7fc72c1cf470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:27:32.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc730ca1640 1 --2- 172.21.15.37:0/3058147773 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc72c09c410 0x7fc72c1cf470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:32.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc730ca1640 10 monclient: _finish_hunting 0 2024-04-26T00:27:32.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc730ca1640 1 monclient: found mon.noname-a 2024-04-26T00:27:32.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc730ca1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:27:32.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc730ca1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:27:32.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.335+0000 7fc730ca1640 1 -- 172.21.15.37:0/3058147773 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc72c09e760 con 0x7fc72c136da0 2024-04-26T00:27:32.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.339+0000 7fc730ca1640 1 -- 172.21.15.37:0/3058147773 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc72c136da0 msgr2=0x7fc72c1cf9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:32.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.339+0000 7fc730ca1640 1 --2- 172.21.15.37:0/3058147773 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc72c136da0 0x7fc72c1cf9b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc71c00b550 tx=0x7fc71c00ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:32.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.339+0000 7fc721ffb640 1 -- 172.21.15.37:0/3058147773 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc71c003d10 con 0x7fc72c136da0 2024-04-26T00:27:32.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.339+0000 7fc721ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:32.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.339+0000 7fc721ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:27:32.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.339+0000 7fc721ffb640 10 monclient: dump: 2024-04-26T00:27:32.329 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:32.329 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:32.329 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:32.329 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:32.329 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:32.329 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:32.329 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:32.330 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:32.330 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:32.330 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:32.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.339+0000 7fc721ffb640 10 monclient: _finish_auth 0 2024-04-26T00:27:32.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.339+0000 7fc721ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:27:32.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.339+0000 7fc721ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:27:32.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.339+0000 7fc733706640 5 monclient: authenticate success, global_id 94140 2024-04-26T00:27:32.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.339+0000 7fc721ffb640 1 -- 172.21.15.37:0/3058147773 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc71c015070 con 0x7fc72c136da0 2024-04-26T00:27:32.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.339+0000 7fc721ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:27:32.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.339+0000 7fc721ffb640 1 -- 172.21.15.37:0/3058147773 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc71c005330 con 0x7fc72c136da0 2024-04-26T00:27:32.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.339+0000 7fc721ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:32.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.339+0000 7fc721ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:27:32.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.339+0000 7fc7237fe640 4 set_mon_vals no callback set 2024-04-26T00:27:32.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.339+0000 7fc721ffb640 10 monclient: dump: 2024-04-26T00:27:32.330 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:32.331 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:32.331 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:32.331 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:32.331 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:32.331 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:32.331 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:32.331 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:32.331 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:32.331 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:32.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.339+0000 7fc733706640 10 monclient: _renew_subs 2024-04-26T00:27:32.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.339+0000 7fc733706640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:27:32.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.339+0000 7fc733706640 1 -- 172.21.15.37:0/3058147773 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc72c135da0 con 0x7fc72c136da0 2024-04-26T00:27:32.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.339+0000 7fc733706640 10 monclient: _renew_subs 2024-04-26T00:27:32.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.339+0000 7fc733706640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:27:32.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.339+0000 7fc733706640 1 -- 172.21.15.37:0/3058147773 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc72c09ff80 con 0x7fc72c136da0 2024-04-26T00:27:32.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.339+0000 7fc733706640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:32.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.339+0000 7fc733706640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:32.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.339+0000 7fc733706640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:27:32.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.339+0000 7fc733706640 1 -- 172.21.15.37:0/3058147773 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc72c066f40 con 0x7fc72c136da0 2024-04-26T00:27:32.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.339+0000 7fc721ffb640 1 -- 172.21.15.37:0/3058147773 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc71c005510 con 0x7fc72c136da0 2024-04-26T00:27:32.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.339+0000 7fc721ffb640 1 --2- 172.21.15.37:0/3058147773 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc710035e80 0x7fc710038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:32.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.339+0000 7fc721ffb640 1 -- 172.21.15.37:0/3058147773 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(298..298 src has 177..298) ==== 4011+0+0 (secure 0 0 0) 0x7fc71c048520 con 0x7fc72c136da0 2024-04-26T00:27:32.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.339+0000 7fc7314a2640 1 --2- 172.21.15.37:0/3058147773 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc710035e80 0x7fc710038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:32.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.339+0000 7fc7314a2640 10 monclient: get_auth_request con 0x7fc710035e80 auth_method 0 2024-04-26T00:27:32.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.343+0000 7fc7314a2640 1 --2- 172.21.15.37:0/3058147773 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc710035e80 0x7fc710038340 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc7280ab000 tx=0x7fc728008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:32.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.343+0000 7fc721ffb640 1 -- 172.21.15.37:0/3058147773 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc71c01a640 con 0x7fc72c136da0 2024-04-26T00:27:32.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.343+0000 7fc721ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:32.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.343+0000 7fc721ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:27:32.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.555+0000 7fc733706640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:27:32.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.555+0000 7fc733706640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:27:32.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.555+0000 7fc733706640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:27:32.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.555+0000 7fc733706640 1 -- 172.21.15.37:0/3058147773 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc72c09ee20 con 0x7fc72c136da0 2024-04-26T00:27:32.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.559+0000 7fc721ffb640 1 -- 172.21.15.37:0/3058147773 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fc71c048280 con 0x7fc72c136da0 2024-04-26T00:27:32.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.559+0000 7fc721ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:27:32.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.559+0000 7fc721ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:27:32.543 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":58,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:27:32.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.559+0000 7fc733706640 1 -- 172.21.15.37:0/3058147773 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc710035e80 msgr2=0x7fc710038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:32.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.559+0000 7fc733706640 1 --2- 172.21.15.37:0/3058147773 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc710035e80 0x7fc710038340 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc7280ab000 tx=0x7fc728008040 comp rx=0 tx=0).stop 2024-04-26T00:27:32.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.559+0000 7fc733706640 10 monclient: shutdown 2024-04-26T00:27:32.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.559+0000 7fc733706640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:27:32.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.559+0000 7fc733706640 1 -- 172.21.15.37:0/3058147773 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc72c136da0 msgr2=0x7fc72c1cf9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:32.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.559+0000 7fc733706640 1 --2- 172.21.15.37:0/3058147773 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc72c136da0 0x7fc72c1cf9b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc71c00b550 tx=0x7fc71c00ba10 comp rx=0 tx=0).stop 2024-04-26T00:27:32.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.559+0000 7fc733706640 1 -- 172.21.15.37:0/3058147773 shutdown_connections 2024-04-26T00:27:32.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.559+0000 7fc733706640 1 --2- 172.21.15.37:0/3058147773 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc710035e80 0x7fc710038340 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:32.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.559+0000 7fc733706640 1 --2- 172.21.15.37:0/3058147773 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc72c1376c0 0x7fc72c1cfef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:32.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.559+0000 7fc733706640 1 --2- 172.21.15.37:0/3058147773 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc72c136da0 0x7fc72c1cf9b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:32.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.559+0000 7fc733706640 1 --2- 172.21.15.37:0/3058147773 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc72c09c410 0x7fc72c1cf470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:32.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.559+0000 7fc733706640 1 -- 172.21.15.37:0/3058147773 >> 172.21.15.37:0/3058147773 conn(0x7fc72c0a5f00 msgr2=0x7fc72c12fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:27:32.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.559+0000 7fc733706640 1 -- 172.21.15.37:0/3058147773 shutdown_connections 2024-04-26T00:27:32.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:32.559+0000 7fc733706640 1 -- 172.21.15.37:0/3058147773 wait complete. 2024-04-26T00:27:35.562 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:27:35.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aae506640 10 monclient: get_monmap_and_config 2024-04-26T00:27:35.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aae506640 10 monclient: build_initial_monmap 2024-04-26T00:27:35.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aae506640 10 monclient: monmap: 2024-04-26T00:27:35.696 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:27:35.696 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:35.696 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:27:35.713198+0000 2024-04-26T00:27:35.696 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:27:35.713198+0000 2024-04-26T00:27:35.696 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:27:35.696 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:35.696 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:27:35.696 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:27:35.696 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:27:35.696 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:27:35.696 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:27:35.696 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:35.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aae506640 1 Processor -- start 2024-04-26T00:27:35.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aae506640 1 -- start start 2024-04-26T00:27:35.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aae506640 10 monclient: init 2024-04-26T00:27:35.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aae506640 10 monclient: _reopen_session rank -1 2024-04-26T00:27:35.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aae506640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-04-26T00:27:35.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aae506640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9aa8136da0 0x7f9aa809bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:35.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aae506640 10 monclient(hunting): picked mon.noname-a con 0x7f9aa8136da0 addr v2:172.21.15.37:3300/0 2024-04-26T00:27:35.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aae506640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9aa809c410 0x7f9aa8137180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:35.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aae506640 10 monclient(hunting): picked mon.noname-c con 0x7f9aa809c410 addr v2:172.21.15.45:3300/0 2024-04-26T00:27:35.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aae506640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9aa81376c0 addr v1:172.21.15.45:6789/0 2024-04-26T00:27:35.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aae506640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9aa809b4f0 con 0x7f9aa81376c0 2024-04-26T00:27:35.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aae506640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:35.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aae506640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f9aa809b210 con 0x7f9aa8136da0 2024-04-26T00:27:35.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aae506640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:35.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aae506640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f9aa809b7d0 con 0x7f9aa809c410 2024-04-26T00:27:35.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aae506640 10 monclient(hunting): _renew_subs 2024-04-26T00:27:35.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aae506640 10 monclient(hunting): authenticate will time out at 1437.019055s 2024-04-26T00:27:35.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aa7fff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9aa8136da0 0x7f9aa809bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:35.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aacaa3640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f9aa81376c0 legacy=0x7f9aa8133b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:27:35.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aa77fe640 1 -- >> v2:172.21.15.45:3300/0 conn(0x7f9aa809c410 msgr2=0x7f9aa8137180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:27:35.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aa77fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9aa809c410 0x7f9aa8137180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:27:35.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aa7fff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9aa8136da0 0x7f9aa809bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50392/0 (socket says 172.21.15.37:50392) 2024-04-26T00:27:35.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aa7fff640 1 -- 172.21.15.37:0/1133376800 learned_addr learned my addr 172.21.15.37:0/1133376800 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:27:35.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aa7fff640 10 monclient(hunting): get_auth_request con 0x7f9aa8136da0 auth_method 0 2024-04-26T00:27:35.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aa7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:35.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aa7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:35.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aa7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:35.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aa7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:35.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aa7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:35.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aa7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:35.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aa7fff640 10 monclient(hunting): handle_auth_done global_id 94143 payload 274 2024-04-26T00:27:35.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aa7fff640 1 -- 172.21.15.37:0/1133376800 >> v2:172.21.15.45:3300/0 conn(0x7f9aa809c410 msgr2=0x7f9aa8137180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:27:35.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aa7fff640 1 --2- 172.21.15.37:0/1133376800 >> v2:172.21.15.45:3300/0 conn(0x7f9aa809c410 0x7f9aa8137180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:35.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aa7fff640 1 -- 172.21.15.37:0/1133376800 >> v1:172.21.15.45:6789/0 conn(0x7f9aa81376c0 legacy=0x7f9aa8133b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:27:35.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aa7fff640 10 monclient: _finish_hunting 0 2024-04-26T00:27:35.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aa7fff640 1 monclient: found mon.noname-a 2024-04-26T00:27:35.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aa7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:27:35.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aa7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:27:35.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.711+0000 7f9aa7fff640 1 -- 172.21.15.37:0/1133376800 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aa8137b30 con 0x7f9aa8136da0 2024-04-26T00:27:35.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa7fff640 1 -- 172.21.15.37:0/1133376800 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9aa8136da0 msgr2=0x7f9aa809bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:35.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa7fff640 1 --2- 172.21.15.37:0/1133376800 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9aa8136da0 0x7f9aa809bed0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9aa80a05c0 tx=0x7f9a9801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35a7d56f64bc7b11 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:35.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa6ffd640 1 -- 172.21.15.37:0/1133376800 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a980b0070 con 0x7f9aa8136da0 2024-04-26T00:27:35.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:35.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:27:35.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa6ffd640 10 monclient: dump: 2024-04-26T00:27:35.705 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:35.705 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:35.705 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:35.705 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:35.705 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:35.705 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:35.705 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:35.705 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:35.705 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:35.705 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:35.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa6ffd640 10 monclient: _finish_auth 0 2024-04-26T00:27:35.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa6ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:27:35.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:27:35.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aae506640 5 monclient: authenticate success, global_id 94143 2024-04-26T00:27:35.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aae506640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:27:35.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa6ffd640 1 -- 172.21.15.37:0/1133376800 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a980ab400 con 0x7f9aa8136da0 2024-04-26T00:27:35.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa6ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:27:35.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aae506640 10 monclient: get_monmap_and_config success 2024-04-26T00:27:35.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aae506640 4 set_mon_vals no callback set 2024-04-26T00:27:35.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa6ffd640 1 -- 172.21.15.37:0/1133376800 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a980ab7d0 con 0x7f9aa8136da0 2024-04-26T00:27:35.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:35.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:27:35.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa6ffd640 10 monclient: dump: 2024-04-26T00:27:35.706 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:35.706 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:35.706 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:35.706 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:35.707 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:35.707 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:35.707 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:35.707 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:35.707 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:35.707 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:35.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aae506640 10 monclient: shutdown 2024-04-26T00:27:35.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aae506640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:27:35.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aae506640 1 -- 172.21.15.37:0/1133376800 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9aa8136da0 msgr2=0x7f9aa809bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:35.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aae506640 1 --2- 172.21.15.37:0/1133376800 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9aa8136da0 0x7f9aa809bed0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9aa80a05c0 tx=0x7f9a9801c450 comp rx=0 tx=0).stop 2024-04-26T00:27:35.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aae506640 1 -- 172.21.15.37:0/1133376800 shutdown_connections 2024-04-26T00:27:35.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aae506640 1 --2- 172.21.15.37:0/1133376800 >> v2:172.21.15.45:3300/0 conn(0x7f9aa809c410 0x7f9aa8137180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:35.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aae506640 1 --2- 172.21.15.37:0/1133376800 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9aa8136da0 0x7f9aa809bed0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:35.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aae506640 1 -- 172.21.15.37:0/1133376800 >> 172.21.15.37:0/1133376800 conn(0x7f9aa80a5f00 msgr2=0x7f9aa80a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:27:35.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aae506640 1 -- 172.21.15.37:0/1133376800 shutdown_connections 2024-04-26T00:27:35.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aae506640 1 -- 172.21.15.37:0/1133376800 wait complete. 2024-04-26T00:27:35.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aae506640 10 monclient: build_initial_monmap 2024-04-26T00:27:35.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aae506640 10 monclient: monmap: 2024-04-26T00:27:35.708 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:27:35.708 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:27:35.708 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:27:35.708 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:27:35.708 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:27:35.708 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:27:35.708 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:27:35.708 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:27:35.708 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:27:35.708 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:35.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aae506640 1 Processor -- start 2024-04-26T00:27:35.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aae506640 1 -- start start 2024-04-26T00:27:35.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aae506640 10 monclient: init 2024-04-26T00:27:35.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aae506640 10 monclient: _reopen_session rank -1 2024-04-26T00:27:35.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aae506640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:27:35.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aae506640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9aa809c410 0x7f9aa81cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:35.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aae506640 10 monclient(hunting): picked mon.noname-c con 0x7f9aa809c410 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:27:35.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aae506640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9aa8136da0 0x7f9aa81d0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:35.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aae506640 10 monclient(hunting): picked mon.noname-a con 0x7f9aa8136da0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:27:35.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aae506640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9aa81376c0 0x7f9aa81cc260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:35.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aae506640 10 monclient(hunting): picked mon.noname-b con 0x7f9aa81376c0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:27:35.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aae506640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:35.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aae506640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa8141ee0 con 0x7f9aa8136da0 2024-04-26T00:27:35.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aae506640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:35.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aae506640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa8141d60 con 0x7f9aa81376c0 2024-04-26T00:27:35.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aae506640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:35.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aae506640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9aa8142060 con 0x7f9aa809c410 2024-04-26T00:27:35.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aae506640 10 monclient(hunting): _renew_subs 2024-04-26T00:27:35.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aae506640 10 monclient(hunting): authenticate will time out at 1437.027322s 2024-04-26T00:27:35.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa77fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9aa8136da0 0x7f9aa81d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:35.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa7fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9aa809c410 0x7f9aa81cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:35.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aacaa3640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9aa81376c0 msgr2=0x7f9aa81cc260 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:27:35.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aacaa3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9aa81376c0 0x7f9aa81cc260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:27:35.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa7fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9aa809c410 0x7f9aa81cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:50062/0 (socket says 172.21.15.37:50062) 2024-04-26T00:27:35.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa77fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9aa8136da0 0x7f9aa81d0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50394/0 (socket says 172.21.15.37:50394) 2024-04-26T00:27:35.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa7fff640 1 -- 172.21.15.37:0/2908442612 learned_addr learned my addr 172.21.15.37:0/2908442612 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:27:35.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa7fff640 10 monclient(hunting): get_auth_request con 0x7f9aa809c410 auth_method 0 2024-04-26T00:27:35.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:35.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:35.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:35.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa77fe640 10 monclient(hunting): get_auth_request con 0x7f9aa8136da0 auth_method 0 2024-04-26T00:27:35.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:35.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa77fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:35.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:35.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:35.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:35.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:35.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:35.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:35.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:35.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa7fff640 10 monclient(hunting): handle_auth_done global_id 84242 payload 824 2024-04-26T00:27:35.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa7fff640 1 -- 172.21.15.37:0/2908442612 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9aa81376c0 msgr2=0x7f9aa81cc260 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:27:35.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa7fff640 1 --2- 172.21.15.37:0/2908442612 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9aa81376c0 0x7f9aa81cc260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:35.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa7fff640 1 -- 172.21.15.37:0/2908442612 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9aa8136da0 msgr2=0x7f9aa81d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:35.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa7fff640 1 --2- 172.21.15.37:0/2908442612 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9aa8136da0 0x7f9aa81d0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:35.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa7fff640 10 monclient: _finish_hunting 0 2024-04-26T00:27:35.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa7fff640 1 monclient: found mon.noname-c 2024-04-26T00:27:35.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:27:35.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:27:35.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa7fff640 1 -- 172.21.15.37:0/2908442612 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aa809d4f0 con 0x7f9aa809c410 2024-04-26T00:27:35.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.719+0000 7f9aa77fe640 1 --2- 172.21.15.37:0/2908442612 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9aa8136da0 0x7f9aa81d0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:27:35.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.723+0000 7f9aa7fff640 1 -- 172.21.15.37:0/2908442612 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9aa809c410 msgr2=0x7f9aa81cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:35.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.723+0000 7f9aa7fff640 1 --2- 172.21.15.37:0/2908442612 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9aa809c410 0x7f9aa81cfb20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9a98019060 tx=0x7f9a980191f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:35.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.723+0000 7f9aa4ff9640 1 -- 172.21.15.37:0/2908442612 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a980b9020 con 0x7f9aa809c410 2024-04-26T00:27:35.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.723+0000 7f9aa4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:35.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.723+0000 7f9aa4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:27:35.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.723+0000 7f9aa4ff9640 10 monclient: dump: 2024-04-26T00:27:35.716 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:35.716 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:35.716 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:35.717 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:35.717 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:35.717 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:35.717 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:35.717 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:35.717 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:35.717 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:35.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.723+0000 7f9aa4ff9640 10 monclient: _finish_auth 0 2024-04-26T00:27:35.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.723+0000 7f9aa4ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:27:35.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.723+0000 7f9aa4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:27:35.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.723+0000 7f9aa4ff9640 1 -- 172.21.15.37:0/2908442612 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a980b1040 con 0x7f9aa809c410 2024-04-26T00:27:35.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.723+0000 7f9aae506640 5 monclient: authenticate success, global_id 84242 2024-04-26T00:27:35.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.723+0000 7f9aae506640 10 monclient: _renew_subs 2024-04-26T00:27:35.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.723+0000 7f9aae506640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:27:35.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.723+0000 7f9aae506640 1 -- 172.21.15.37:0/2908442612 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9aa809f340 con 0x7f9aa809c410 2024-04-26T00:27:35.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.723+0000 7f9aa4ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:27:35.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.723+0000 7f9aa4ff9640 1 -- 172.21.15.37:0/2908442612 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a980b0050 con 0x7f9aa809c410 2024-04-26T00:27:35.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.723+0000 7f9aa4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:35.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.723+0000 7f9aa4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:27:35.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.723+0000 7f9aa4ff9640 10 monclient: dump: 2024-04-26T00:27:35.718 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:35.718 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:35.718 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:35.718 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:35.718 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:35.719 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:35.719 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:35.719 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:35.719 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:35.719 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:35.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.723+0000 7f9aa67fc640 4 set_mon_vals no callback set 2024-04-26T00:27:35.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.723+0000 7f9aae506640 10 monclient: _renew_subs 2024-04-26T00:27:35.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.723+0000 7f9aae506640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:27:35.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.723+0000 7f9aae506640 1 -- 172.21.15.37:0/2908442612 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9aa809f060 con 0x7f9aa809c410 2024-04-26T00:27:35.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.723+0000 7f9aa4ff9640 1 -- 172.21.15.37:0/2908442612 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9a980bc020 con 0x7f9aa809c410 2024-04-26T00:27:35.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.723+0000 7f9aae506640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:35.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.723+0000 7f9aae506640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:35.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.723+0000 7f9aae506640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:27:35.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.723+0000 7f9aae506640 1 -- 172.21.15.37:0/2908442612 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9aa8066f40 con 0x7f9aa809c410 2024-04-26T00:27:35.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.723+0000 7f9aa4ff9640 1 --2- 172.21.15.37:0/2908442612 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9a88035e30 0x7f9a880382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:35.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.723+0000 7f9aa4ff9640 1 -- 172.21.15.37:0/2908442612 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(298..298 src has 177..298) ==== 4011+0+0 (secure 0 0 0) 0x7f9a980e2990 con 0x7f9aa809c410 2024-04-26T00:27:35.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.727+0000 7f9aa77fe640 1 --2- 172.21.15.37:0/2908442612 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9a88035e30 0x7f9a880382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:35.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.727+0000 7f9aa77fe640 10 monclient: get_auth_request con 0x7f9a88035e30 auth_method 0 2024-04-26T00:27:35.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.727+0000 7f9aa77fe640 1 --2- 172.21.15.37:0/2908442612 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9a88035e30 0x7f9a880382f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9aa809db40 tx=0x7f9a9000d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:35.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.731+0000 7f9aa4ff9640 1 -- 172.21.15.37:0/2908442612 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9a980c3030 con 0x7f9aa809c410 2024-04-26T00:27:35.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.731+0000 7f9aa4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:35.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.731+0000 7f9aa4ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:27:35.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.951+0000 7f9aae506640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:27:35.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.951+0000 7f9aae506640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:27:35.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.951+0000 7f9aae506640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:27:35.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.951+0000 7f9aae506640 1 -- 172.21.15.37:0/2908442612 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9aa80a81f0 con 0x7f9aa809c410 2024-04-26T00:27:35.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.951+0000 7f9aa4ff9640 1 -- 172.21.15.37:0/2908442612 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f9a980160d0 con 0x7f9aa809c410 2024-04-26T00:27:35.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.951+0000 7f9aa4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:27:35.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.951+0000 7f9aa4ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:27:35.938 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":58,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:27:35.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.955+0000 7f9aae506640 1 -- 172.21.15.37:0/2908442612 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9a88035e30 msgr2=0x7f9a880382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:35.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.955+0000 7f9aae506640 1 --2- 172.21.15.37:0/2908442612 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9a88035e30 0x7f9a880382f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9aa809db40 tx=0x7f9a9000d040 comp rx=0 tx=0).stop 2024-04-26T00:27:35.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.955+0000 7f9aae506640 10 monclient: shutdown 2024-04-26T00:27:35.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.955+0000 7f9aae506640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:27:35.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.955+0000 7f9aae506640 1 -- 172.21.15.37:0/2908442612 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9aa809c410 msgr2=0x7f9aa81cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:35.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.955+0000 7f9aae506640 1 --2- 172.21.15.37:0/2908442612 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9aa809c410 0x7f9aa81cfb20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9a98019060 tx=0x7f9a980191f0 comp rx=0 tx=0).stop 2024-04-26T00:27:35.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.955+0000 7f9aae506640 1 -- 172.21.15.37:0/2908442612 shutdown_connections 2024-04-26T00:27:35.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.955+0000 7f9aae506640 1 --2- 172.21.15.37:0/2908442612 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9a88035e30 0x7f9a880382f0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:35.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.955+0000 7f9aae506640 1 --2- 172.21.15.37:0/2908442612 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9aa81376c0 0x7f9aa81cc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:35.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.955+0000 7f9aae506640 1 --2- 172.21.15.37:0/2908442612 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9aa8136da0 0x7f9aa81d0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:35.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.955+0000 7f9aae506640 1 --2- 172.21.15.37:0/2908442612 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9aa809c410 0x7f9aa81cfb20 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:35.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.955+0000 7f9aae506640 1 -- 172.21.15.37:0/2908442612 >> 172.21.15.37:0/2908442612 conn(0x7f9aa80a5f00 msgr2=0x7f9aa812fdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:27:35.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.955+0000 7f9aae506640 1 -- 172.21.15.37:0/2908442612 shutdown_connections 2024-04-26T00:27:35.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:35.955+0000 7f9aae506640 1 -- 172.21.15.37:0/2908442612 wait complete. 2024-04-26T00:27:38.957 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:27:39.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f5239350640 10 monclient: get_monmap_and_config 2024-04-26T00:27:39.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f5239350640 10 monclient: build_initial_monmap 2024-04-26T00:27:39.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f5239350640 10 monclient: monmap: 2024-04-26T00:27:39.088 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:27:39.088 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:39.088 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:27:39.105370+0000 2024-04-26T00:27:39.088 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:27:39.105370+0000 2024-04-26T00:27:39.088 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:27:39.089 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:39.089 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:27:39.089 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:27:39.089 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:27:39.089 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:27:39.089 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:27:39.089 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:39.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f5239350640 1 Processor -- start 2024-04-26T00:27:39.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f5239350640 1 -- start start 2024-04-26T00:27:39.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f5239350640 10 monclient: init 2024-04-26T00:27:39.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f5239350640 10 monclient: _reopen_session rank -1 2024-04-26T00:27:39.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f5239350640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-04-26T00:27:39.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f5239350640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5234071fc0 addr v1:172.21.15.37:6789/0 2024-04-26T00:27:39.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f5239350640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52340723a0 0x7f52340727e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:39.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f5239350640 10 monclient(hunting): picked mon.noname-b con 0x7f52340723a0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:27:39.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f5239350640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f523413c340 0x7f523413e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:39.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f5239350640 10 monclient(hunting): picked mon.noname-a con 0x7f523413c340 addr v2:172.21.15.37:3300/0 2024-04-26T00:27:39.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f5239350640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5234073200 con 0x7f5234071fc0 2024-04-26T00:27:39.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f5239350640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:39.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f5239350640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f52340a0c20 con 0x7f523413c340 2024-04-26T00:27:39.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f5239350640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:39.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f5239350640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f52340a0f00 con 0x7f52340723a0 2024-04-26T00:27:39.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f5239350640 10 monclient(hunting): _renew_subs 2024-04-26T00:27:39.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f5239350640 10 monclient(hunting): authenticate will time out at 1440.411004s 2024-04-26T00:27:39.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f52337fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f523413c340 0x7f523413e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:39.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f52327fc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52340723a0 0x7f52340727e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:39.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f52337fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f523413c340 0x7f523413e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50398/0 (socket says 172.21.15.37:50398) 2024-04-26T00:27:39.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f52337fe640 1 -- 172.21.15.37:0/2455530157 learned_addr learned my addr 172.21.15.37:0/2455530157 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:27:39.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f52327fc640 10 monclient(hunting): get_auth_request con 0x7f52340723a0 auth_method 0 2024-04-26T00:27:39.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f52327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:39.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f52327fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:39.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f52327fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:39.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f52337fe640 10 monclient(hunting): get_auth_request con 0x7f523413c340 auth_method 0 2024-04-26T00:27:39.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f52337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:39.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f52337fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:39.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f52337fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:39.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f52327fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:39.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f52327fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:39.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f52327fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:39.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f52337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:39.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f52337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:39.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f52337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:39.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f52327fc640 10 monclient(hunting): handle_auth_done global_id 84245 payload 274 2024-04-26T00:27:39.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f52327fc640 1 -- 172.21.15.37:0/2455530157 >> v2:172.21.15.37:3300/0 conn(0x7f523413c340 msgr2=0x7f523413e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:27:39.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f52327fc640 1 --2- 172.21.15.37:0/2455530157 >> v2:172.21.15.37:3300/0 conn(0x7f523413c340 0x7f523413e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:39.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f52327fc640 1 -- 172.21.15.37:0/2455530157 >> v1:172.21.15.37:6789/0 conn(0x7f5234071fc0 legacy=0x7f5234137240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:27:39.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f52327fc640 10 monclient: _finish_hunting 0 2024-04-26T00:27:39.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f52327fc640 1 monclient: found mon.noname-b 2024-04-26T00:27:39.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f52327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:27:39.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f52327fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:27:39.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.103+0000 7f52327fc640 1 -- 172.21.15.37:0/2455530157 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5234137950 con 0x7f52340723a0 2024-04-26T00:27:39.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f52327fc640 1 -- 172.21.15.37:0/2455530157 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52340723a0 msgr2=0x7f52340727e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:39.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f52327fc640 1 --2- 172.21.15.37:0/2455530157 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52340723a0 0x7f52340727e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f52240097c0 tx=0x7f522401c320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fd92d80d7108c9c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:39.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5231ffb640 1 -- 172.21.15.37:0/2455530157 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52240b0070 con 0x7f52340723a0 2024-04-26T00:27:39.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5231ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:39.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5231ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:27:39.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5231ffb640 10 monclient: dump: 2024-04-26T00:27:39.100 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:39.101 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:39.101 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:39.101 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:39.101 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:39.101 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:39.101 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:39.101 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:39.101 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:39.101 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:39.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5231ffb640 10 monclient: _finish_auth 0 2024-04-26T00:27:39.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5231ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:27:39.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5231ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:27:39.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5239350640 5 monclient: authenticate success, global_id 84245 2024-04-26T00:27:39.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5239350640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:27:39.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5231ffb640 1 -- 172.21.15.37:0/2455530157 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52240ab600 con 0x7f52340723a0 2024-04-26T00:27:39.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5231ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:27:39.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5231ffb640 1 -- 172.21.15.37:0/2455530157 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52240ab9d0 con 0x7f52340723a0 2024-04-26T00:27:39.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5239350640 10 monclient: get_monmap_and_config success 2024-04-26T00:27:39.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5239350640 4 set_mon_vals no callback set 2024-04-26T00:27:39.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5231ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:39.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5231ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:27:39.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5231ffb640 10 monclient: dump: 2024-04-26T00:27:39.103 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:39.103 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:39.103 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:39.103 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:39.103 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:39.103 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:39.103 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:39.104 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:39.104 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:39.104 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:39.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5239350640 10 monclient: shutdown 2024-04-26T00:27:39.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5239350640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:27:39.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5239350640 1 -- 172.21.15.37:0/2455530157 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52340723a0 msgr2=0x7f52340727e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:39.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5239350640 1 --2- 172.21.15.37:0/2455530157 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52340723a0 0x7f52340727e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f52240097c0 tx=0x7f522401c320 comp rx=0 tx=0).stop 2024-04-26T00:27:39.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5239350640 1 -- 172.21.15.37:0/2455530157 shutdown_connections 2024-04-26T00:27:39.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5239350640 1 --2- 172.21.15.37:0/2455530157 >> v2:172.21.15.37:3300/0 conn(0x7f523413c340 0x7f523413e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:39.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5239350640 1 --2- 172.21.15.37:0/2455530157 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f52340723a0 0x7f52340727e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:39.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5239350640 1 -- 172.21.15.37:0/2455530157 >> 172.21.15.37:0/2455530157 conn(0x7f52340a1ad0 msgr2=0x7f52340a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:27:39.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5239350640 1 -- 172.21.15.37:0/2455530157 shutdown_connections 2024-04-26T00:27:39.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5239350640 1 -- 172.21.15.37:0/2455530157 wait complete. 2024-04-26T00:27:39.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5239350640 10 monclient: build_initial_monmap 2024-04-26T00:27:39.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5239350640 10 monclient: monmap: 2024-04-26T00:27:39.105 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:27:39.105 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:27:39.105 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:27:39.105 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:27:39.105 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:27:39.106 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:27:39.106 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:27:39.106 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:27:39.106 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:27:39.106 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:39.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5239350640 1 Processor -- start 2024-04-26T00:27:39.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5239350640 1 -- start start 2024-04-26T00:27:39.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5239350640 10 monclient: init 2024-04-26T00:27:39.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5239350640 10 monclient: _reopen_session rank -1 2024-04-26T00:27:39.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5239350640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:27:39.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5239350640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5234071fc0 0x7f52341cec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:39.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5239350640 10 monclient(hunting): picked mon.noname-c con 0x7f5234071fc0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:27:39.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5239350640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52340723a0 0x7f52341cf440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:39.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5239350640 10 monclient(hunting): picked mon.noname-b con 0x7f52340723a0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:27:39.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5239350640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f523413c340 0x7f5234137f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:39.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5239350640 10 monclient(hunting): picked mon.noname-a con 0x7f523413c340 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:27:39.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5239350640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:39.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5239350640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f5234141820 con 0x7f523413c340 2024-04-26T00:27:39.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5239350640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:39.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5239350640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f52341416a0 con 0x7f52340723a0 2024-04-26T00:27:39.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5239350640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:39.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5239350640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f52341419a0 con 0x7f5234071fc0 2024-04-26T00:27:39.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5239350640 10 monclient(hunting): _renew_subs 2024-04-26T00:27:39.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5239350640 10 monclient(hunting): authenticate will time out at 1440.419727s 2024-04-26T00:27:39.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5232ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5234071fc0 0x7f52341cec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:39.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f52337fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f523413c340 0x7f5234137f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:39.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f52327fc640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52340723a0 msgr2=0x7f52341cf440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:27:39.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5232ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5234071fc0 0x7f52341cec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:50076/0 (socket says 172.21.15.37:50076) 2024-04-26T00:27:39.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5232ffd640 1 -- 172.21.15.37:0/2422247711 learned_addr learned my addr 172.21.15.37:0/2422247711 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:27:39.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5232ffd640 10 monclient(hunting): get_auth_request con 0x7f5234071fc0 auth_method 0 2024-04-26T00:27:39.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5232ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:39.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5232ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:39.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f5232ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:39.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f52327fc640 1 --2- 172.21.15.37:0/2422247711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52340723a0 0x7f52341cf440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:27:39.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f52337fe640 10 monclient(hunting): get_auth_request con 0x7f523413c340 auth_method 0 2024-04-26T00:27:39.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f52337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:39.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f52337fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:39.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.111+0000 7f52337fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:39.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f5232ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:39.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f5232ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:39.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f5232ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:39.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f52337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:39.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f52337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:39.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f52337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:39.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f5232ffd640 10 monclient(hunting): handle_auth_done global_id 84248 payload 824 2024-04-26T00:27:39.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f5232ffd640 1 -- 172.21.15.37:0/2422247711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52340723a0 msgr2=0x7f52341cf440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:27:39.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f5232ffd640 1 --2- 172.21.15.37:0/2422247711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52340723a0 0x7f52341cf440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:39.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f5232ffd640 1 -- 172.21.15.37:0/2422247711 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f523413c340 msgr2=0x7f5234137f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:39.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f5232ffd640 1 --2- 172.21.15.37:0/2422247711 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f523413c340 0x7f5234137f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:39.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f5232ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:27:39.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f5232ffd640 1 monclient: found mon.noname-c 2024-04-26T00:27:39.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f5232ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:27:39.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f5232ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:27:39.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f5232ffd640 1 -- 172.21.15.37:0/2422247711 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5234134c10 con 0x7f5234071fc0 2024-04-26T00:27:39.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f52337fe640 1 --2- 172.21.15.37:0/2422247711 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f523413c340 0x7f5234137f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:27:39.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f5232ffd640 1 -- 172.21.15.37:0/2422247711 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5234071fc0 msgr2=0x7f52341cec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:39.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f5232ffd640 1 --2- 172.21.15.37:0/2422247711 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5234071fc0 0x7f52341cec70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f521c00c450 tx=0x7f521c00c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:39.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f522b7fe640 1 -- 172.21.15.37:0/2422247711 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f521c012a40 con 0x7f5234071fc0 2024-04-26T00:27:39.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f522b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:39.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f522b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:27:39.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f522b7fe640 10 monclient: dump: 2024-04-26T00:27:39.113 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:39.113 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:39.113 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:39.113 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:39.113 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:39.113 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:39.114 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:39.114 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:39.114 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:39.114 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:39.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f522b7fe640 10 monclient: _finish_auth 0 2024-04-26T00:27:39.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f522b7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:27:39.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f522b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:27:39.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f5239350640 5 monclient: authenticate success, global_id 84248 2024-04-26T00:27:39.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f5239350640 10 monclient: _renew_subs 2024-04-26T00:27:39.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f522b7fe640 1 -- 172.21.15.37:0/2422247711 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f521c017070 con 0x7f5234071fc0 2024-04-26T00:27:39.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f5239350640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:27:39.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f5239350640 1 -- 172.21.15.37:0/2422247711 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f523413ba00 con 0x7f5234071fc0 2024-04-26T00:27:39.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f522b7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:27:39.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f522b7fe640 1 -- 172.21.15.37:0/2422247711 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f521c012e40 con 0x7f5234071fc0 2024-04-26T00:27:39.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f522b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:39.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f522b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:27:39.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f522b7fe640 10 monclient: dump: 2024-04-26T00:27:39.115 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:39.115 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:39.115 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:39.115 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:39.115 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:39.115 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:39.115 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:39.116 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:39.116 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:39.116 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:39.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f52317fa640 4 set_mon_vals no callback set 2024-04-26T00:27:39.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f5239350640 10 monclient: _renew_subs 2024-04-26T00:27:39.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f5239350640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:27:39.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f5239350640 1 -- 172.21.15.37:0/2422247711 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5234130d90 con 0x7f5234071fc0 2024-04-26T00:27:39.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f5239350640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:39.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f5239350640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:39.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f5239350640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:27:39.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f522b7fe640 1 -- 172.21.15.37:0/2422247711 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f521c01c460 con 0x7f5234071fc0 2024-04-26T00:27:39.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f5239350640 1 -- 172.21.15.37:0/2422247711 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5234066f40 con 0x7f5234071fc0 2024-04-26T00:27:39.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f522b7fe640 1 --2- 172.21.15.37:0/2422247711 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5214035e60 0x7f5214038320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:39.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.115+0000 7f522b7fe640 1 -- 172.21.15.37:0/2422247711 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(298..298 src has 177..298) ==== 4011+0+0 (secure 0 0 0) 0x7f521c0488f0 con 0x7f5234071fc0 2024-04-26T00:27:39.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.119+0000 7f52327fc640 1 --2- 172.21.15.37:0/2422247711 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5214035e60 0x7f5214038320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:39.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.119+0000 7f52327fc640 10 monclient: get_auth_request con 0x7f5214035e60 auth_method 0 2024-04-26T00:27:39.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.119+0000 7f52327fc640 1 --2- 172.21.15.37:0/2422247711 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5214035e60 0x7f5214038320 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f52340a2ad0 tx=0x7f5224008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:39.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.123+0000 7f522b7fe640 1 -- 172.21.15.37:0/2422247711 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5234134c10 con 0x7f5234071fc0 2024-04-26T00:27:39.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.123+0000 7f522b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:39.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.123+0000 7f522b7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:27:39.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.343+0000 7f5239350640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:27:39.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.343+0000 7f5239350640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:27:39.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.343+0000 7f5239350640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:27:39.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.343+0000 7f5239350640 1 -- 172.21.15.37:0/2422247711 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f523406b9a0 con 0x7f5234071fc0 2024-04-26T00:27:39.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.343+0000 7f522b7fe640 1 -- 172.21.15.37:0/2422247711 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f521c0475f0 con 0x7f5234071fc0 2024-04-26T00:27:39.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.343+0000 7f522b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:27:39.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.343+0000 7f522b7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:27:39.328 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":58,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"b","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:27:39.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.343+0000 7f5239350640 1 -- 172.21.15.37:0/2422247711 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5214035e60 msgr2=0x7f5214038320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:39.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.343+0000 7f5239350640 1 --2- 172.21.15.37:0/2422247711 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5214035e60 0x7f5214038320 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f52340a2ad0 tx=0x7f5224008040 comp rx=0 tx=0).stop 2024-04-26T00:27:39.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.343+0000 7f5239350640 10 monclient: shutdown 2024-04-26T00:27:39.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.343+0000 7f5239350640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:27:39.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.343+0000 7f5239350640 1 -- 172.21.15.37:0/2422247711 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5234071fc0 msgr2=0x7f52341cec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:39.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.343+0000 7f5239350640 1 --2- 172.21.15.37:0/2422247711 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5234071fc0 0x7f52341cec70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f521c00c450 tx=0x7f521c00c910 comp rx=0 tx=0).stop 2024-04-26T00:27:39.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.347+0000 7f5239350640 1 -- 172.21.15.37:0/2422247711 shutdown_connections 2024-04-26T00:27:39.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.347+0000 7f5239350640 1 --2- 172.21.15.37:0/2422247711 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5214035e60 0x7f5214038320 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:39.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.347+0000 7f5239350640 1 --2- 172.21.15.37:0/2422247711 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f523413c340 0x7f5234137f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:39.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.347+0000 7f5239350640 1 --2- 172.21.15.37:0/2422247711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f52340723a0 0x7f52341cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:39.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.347+0000 7f5239350640 1 --2- 172.21.15.37:0/2422247711 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5234071fc0 0x7f52341cec70 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:39.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.347+0000 7f5239350640 1 -- 172.21.15.37:0/2422247711 >> 172.21.15.37:0/2422247711 conn(0x7f52340a1ad0 msgr2=0x7f5234134e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:27:39.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.347+0000 7f5239350640 1 -- 172.21.15.37:0/2422247711 shutdown_connections 2024-04-26T00:27:39.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:39.347+0000 7f5239350640 1 -- 172.21.15.37:0/2422247711 wait complete. 2024-04-26T00:27:42.349 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:27:42.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:42.495+0000 7fe3a2ab5640 10 monclient: get_monmap_and_config 2024-04-26T00:27:42.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:42.495+0000 7fe3a2ab5640 10 monclient: build_initial_monmap 2024-04-26T00:27:42.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:42.495+0000 7fe3a2ab5640 10 monclient: monmap: 2024-04-26T00:27:42.480 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:27:42.480 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:42.480 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:27:42.497375+0000 2024-04-26T00:27:42.480 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:27:42.497375+0000 2024-04-26T00:27:42.480 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:27:42.480 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:42.481 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:27:42.481 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:27:42.481 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:27:42.481 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:27:42.481 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:27:42.481 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:42.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:42.495+0000 7fe3a2ab5640 1 Processor -- start 2024-04-26T00:27:42.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:42.495+0000 7fe3a2ab5640 1 -- start start 2024-04-26T00:27:42.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:42.495+0000 7fe3a2ab5640 10 monclient: init 2024-04-26T00:27:42.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:42.495+0000 7fe3a2ab5640 10 monclient: _reopen_session rank -1 2024-04-26T00:27:42.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:42.495+0000 7fe3a2ab5640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-04-26T00:27:42.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:42.495+0000 7fe3a2ab5640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe39c071fc0 0x7fe39c137240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:42.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:42.495+0000 7fe3a2ab5640 10 monclient(hunting): picked mon.noname-c con 0x7fe39c071fc0 addr v2:172.21.15.45:3300/0 2024-04-26T00:27:42.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:42.495+0000 7fe3a2ab5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe39c137780 addr v1:172.21.15.37:6789/0 2024-04-26T00:27:42.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:42.495+0000 7fe3a2ab5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe39c072ab0 addr v1:172.21.15.45:6789/0 2024-04-26T00:27:42.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:42.495+0000 7fe3a2ab5640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe39c073200 con 0x7fe39c137780 2024-04-26T00:27:42.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:42.495+0000 7fe3a2ab5640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe39c0a0c20 con 0x7fe39c072ab0 2024-04-26T00:27:42.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:42.495+0000 7fe3a2ab5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:42.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:42.495+0000 7fe3a2ab5640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fe39c0a0f00 con 0x7fe39c071fc0 2024-04-26T00:27:42.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:42.495+0000 7fe3a2ab5640 10 monclient(hunting): _renew_subs 2024-04-26T00:27:42.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:42.495+0000 7fe3a2ab5640 10 monclient(hunting): authenticate will time out at 1443.803125s 2024-04-26T00:27:42.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:42.495+0000 7fe3a1052640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7fe39c072ab0 legacy=0x7fe39c13e350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:27:42.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:42.495+0000 7fe3a0851640 1 -- >> v2:172.21.15.45:3300/0 conn(0x7fe39c071fc0 msgr2=0x7fe39c137240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:27:42.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:42.495+0000 7fe3a0851640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe39c071fc0 0x7fe39c137240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:27:42.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:42.495+0000 7fe39bfff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fe39c137780 0x7fe39c0723a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:50070/0 (socket says 172.21.15.37:50070) 2024-04-26T00:27:42.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:42.495+0000 7fe39bfff640 1 -- 172.21.15.37:0/3899793128 learned_addr learned my addr 172.21.15.37:0/3899793128 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:27:42.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:42.499+0000 7fe39bfff640 1 -- 172.21.15.37:0/3899793128 >> v1:172.21.15.37:6789/0 conn(0x7fe39c137780 legacy=0x7fe39c0723a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:42.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:42.699+0000 7fe3a1052640 1 -- 172.21.15.37:0/3899793128 >> v1:172.21.15.45:6789/0 conn(0x7fe39c072ab0 legacy=0x7fe39c13e350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:27:42.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:42.699+0000 7fe3a0851640 1 -- 172.21.15.37:0/3899793128 >> v2:172.21.15.45:3300/0 conn(0x7fe39c071fc0 msgr2=0x7fe39c137240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:27:42.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:42.699+0000 7fe3a0851640 1 --2- 172.21.15.37:0/3899793128 >> v2:172.21.15.45:3300/0 conn(0x7fe39c071fc0 0x7fe39c137240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:27:43.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:43.103+0000 7fe3a1052640 1 -- 172.21.15.37:0/3899793128 >> v1:172.21.15.45:6789/0 conn(0x7fe39c072ab0 legacy=0x7fe39c13e350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:27:43.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:43.103+0000 7fe3a0851640 1 -- 172.21.15.37:0/3899793128 >> v2:172.21.15.45:3300/0 conn(0x7fe39c071fc0 msgr2=0x7fe39c137240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:27:43.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:43.103+0000 7fe3a0851640 1 --2- 172.21.15.37:0/3899793128 >> v2:172.21.15.45:3300/0 conn(0x7fe39c071fc0 0x7fe39c137240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:27:43.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:43.907+0000 7fe3a1052640 1 -- 172.21.15.37:0/3899793128 >> v1:172.21.15.45:6789/0 conn(0x7fe39c072ab0 legacy=0x7fe39c13e350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:27:43.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:43.907+0000 7fe3a0851640 1 -- 172.21.15.37:0/3899793128 >> v2:172.21.15.45:3300/0 conn(0x7fe39c071fc0 msgr2=0x7fe39c137240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:27:43.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:43.907+0000 7fe3a0851640 1 --2- 172.21.15.37:0/3899793128 >> v2:172.21.15.45:3300/0 conn(0x7fe39c071fc0 0x7fe39c137240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:27:45.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.495+0000 7fe39a7fc640 10 monclient(hunting): tick 2024-04-26T00:27:45.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.495+0000 7fe39a7fc640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:27:45.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.495+0000 7fe39a7fc640 1 monclient(hunting): continuing hunt 2024-04-26T00:27:45.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.495+0000 7fe39a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:27:45.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.495+0000 7fe39a7fc640 1 -- 172.21.15.37:0/3899793128 >> v2:172.21.15.45:3300/0 conn(0x7fe39c071fc0 msgr2=0x7fe39c137240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:27:45.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.495+0000 7fe39a7fc640 1 --2- 172.21.15.37:0/3899793128 >> v2:172.21.15.45:3300/0 conn(0x7fe39c071fc0 0x7fe39c137240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:45.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.495+0000 7fe39a7fc640 1 -- 172.21.15.37:0/3899793128 >> v1:172.21.15.45:6789/0 conn(0x7fe39c072ab0 legacy=0x7fe39c13e350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:27:45.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.495+0000 7fe39a7fc640 1 -- 172.21.15.37:0/3899793128 >> v1:172.21.15.37:6789/0 conn(0x7fe39c137780 legacy=0x7fe39c0723a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:45.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.495+0000 7fe39a7fc640 10 monclient: _add_conns ranks=[3,2] 2024-04-26T00:27:45.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.495+0000 7fe39a7fc640 1 --2- 172.21.15.37:0/3899793128 >> v2:172.21.15.37:3300/0 conn(0x7fe388003750 0x7fe388005bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:45.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.495+0000 7fe39a7fc640 10 monclient(hunting): picked mon.noname-a con 0x7fe388003750 addr v2:172.21.15.37:3300/0 2024-04-26T00:27:45.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.495+0000 7fe39a7fc640 1 --2- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 0x7fe388008720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:45.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.495+0000 7fe39a7fc640 10 monclient(hunting): picked mon.noname-b con 0x7fe388006260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:27:45.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.495+0000 7fe39a7fc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:45.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.495+0000 7fe39a7fc640 1 -- 172.21.15.37:0/3899793128 --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fe39c0a0f00 con 0x7fe388003750 2024-04-26T00:27:45.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.495+0000 7fe39a7fc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:45.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.495+0000 7fe39a7fc640 1 -- 172.21.15.37:0/3899793128 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe388008e30 con 0x7fe388006260 2024-04-26T00:27:45.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.495+0000 7fe39a7fc640 10 monclient(hunting): _renew_subs 2024-04-26T00:27:45.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.495+0000 7fe3a0851640 1 --2- 172.21.15.37:0/3899793128 >> v2:172.21.15.37:3300/0 conn(0x7fe388003750 0x7fe388005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:45.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.495+0000 7fe39bfff640 1 --2- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 0x7fe388008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:45.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.495+0000 7fe3a0851640 10 monclient(hunting): get_auth_request con 0x7fe388003750 auth_method 0 2024-04-26T00:27:45.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.495+0000 7fe3a0851640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:45.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.495+0000 7fe3a0851640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:45.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.495+0000 7fe3a0851640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:45.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.495+0000 7fe39bfff640 10 monclient(hunting): get_auth_request con 0x7fe388006260 auth_method 0 2024-04-26T00:27:45.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.495+0000 7fe39bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:45.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.495+0000 7fe39bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:45.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.495+0000 7fe39bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:45.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.499+0000 7fe3a0851640 1 -- 172.21.15.37:0/3899793128 >> v2:172.21.15.37:3300/0 conn(0x7fe388003750 msgr2=0x7fe388005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:27:45.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.499+0000 7fe3a0851640 1 -- 172.21.15.37:0/3899793128 >> v2:172.21.15.37:3300/0 conn(0x7fe388003750 msgr2=0x7fe388005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:27:45.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.499+0000 7fe39bfff640 1 -- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 msgr2=0x7fe388008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:27:45.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.499+0000 7fe3a0851640 1 --2- 172.21.15.37:0/3899793128 >> v2:172.21.15.37:3300/0 conn(0x7fe388003750 0x7fe388005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:27:45.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.499+0000 7fe39bfff640 1 -- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 msgr2=0x7fe388008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:27:45.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.499+0000 7fe39bfff640 1 --2- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 0x7fe388008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:27:45.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.499+0000 7fe3a0851640 1 --2- 172.21.15.37:0/3899793128 >> v2:172.21.15.37:3300/0 conn(0x7fe388003750 0x7fe388005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:27:45.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.499+0000 7fe39bfff640 1 --2- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 0x7fe388008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:27:45.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.703+0000 7fe39bfff640 1 --2- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 0x7fe388008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:45.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.703+0000 7fe39bfff640 10 monclient(hunting): get_auth_request con 0x7fe388006260 auth_method 0 2024-04-26T00:27:45.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.703+0000 7fe39bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:45.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.703+0000 7fe39bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:45.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.703+0000 7fe39bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:27:45.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.703+0000 7fe3a0851640 1 --2- 172.21.15.37:0/3899793128 >> v2:172.21.15.37:3300/0 conn(0x7fe388003750 0x7fe388005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:45.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.703+0000 7fe3a0851640 10 monclient(hunting): get_auth_request con 0x7fe388003750 auth_method 0 2024-04-26T00:27:45.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.703+0000 7fe3a0851640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:45.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.703+0000 7fe3a0851640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:45.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.703+0000 7fe3a0851640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:27:45.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.703+0000 7fe39bfff640 1 -- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 msgr2=0x7fe388008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:27:45.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.703+0000 7fe39bfff640 1 -- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 msgr2=0x7fe388008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:27:45.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.703+0000 7fe39bfff640 1 --2- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 0x7fe388008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:27:45.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.703+0000 7fe3a0851640 1 -- 172.21.15.37:0/3899793128 >> v2:172.21.15.37:3300/0 conn(0x7fe388003750 msgr2=0x7fe388005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:27:45.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.703+0000 7fe3a0851640 1 -- 172.21.15.37:0/3899793128 >> v2:172.21.15.37:3300/0 conn(0x7fe388003750 msgr2=0x7fe388005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:27:45.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.703+0000 7fe3a0851640 1 --2- 172.21.15.37:0/3899793128 >> v2:172.21.15.37:3300/0 conn(0x7fe388003750 0x7fe388005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:27:45.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.703+0000 7fe39bfff640 1 --2- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 0x7fe388008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:27:45.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:45.703+0000 7fe3a0851640 1 --2- 172.21.15.37:0/3899793128 >> v2:172.21.15.37:3300/0 conn(0x7fe388003750 0x7fe388005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:27:46.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:46.107+0000 7fe39bfff640 1 --2- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 0x7fe388008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:46.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:46.107+0000 7fe3a0851640 1 --2- 172.21.15.37:0/3899793128 >> v2:172.21.15.37:3300/0 conn(0x7fe388003750 0x7fe388005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:46.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:46.107+0000 7fe3a0851640 10 monclient(hunting): get_auth_request con 0x7fe388003750 auth_method 0 2024-04-26T00:27:46.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:46.107+0000 7fe3a0851640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:46.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:46.107+0000 7fe3a0851640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:46.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:46.107+0000 7fe3a0851640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:27:46.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:46.107+0000 7fe39bfff640 10 monclient(hunting): get_auth_request con 0x7fe388006260 auth_method 0 2024-04-26T00:27:46.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:46.107+0000 7fe39bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:46.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:46.107+0000 7fe39bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:46.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:46.107+0000 7fe39bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:27:46.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:46.107+0000 7fe3a0851640 1 -- 172.21.15.37:0/3899793128 >> v2:172.21.15.37:3300/0 conn(0x7fe388003750 msgr2=0x7fe388005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:27:46.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:46.107+0000 7fe3a0851640 1 -- 172.21.15.37:0/3899793128 >> v2:172.21.15.37:3300/0 conn(0x7fe388003750 msgr2=0x7fe388005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:27:46.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:46.107+0000 7fe3a0851640 1 --2- 172.21.15.37:0/3899793128 >> v2:172.21.15.37:3300/0 conn(0x7fe388003750 0x7fe388005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:27:46.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:46.107+0000 7fe39bfff640 1 -- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 msgr2=0x7fe388008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:27:46.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:46.107+0000 7fe39bfff640 1 -- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 msgr2=0x7fe388008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:27:46.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:46.107+0000 7fe39bfff640 1 --2- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 0x7fe388008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:27:46.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:46.107+0000 7fe3a0851640 1 --2- 172.21.15.37:0/3899793128 >> v2:172.21.15.37:3300/0 conn(0x7fe388003750 0x7fe388005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:27:46.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:46.107+0000 7fe39bfff640 1 --2- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 0x7fe388008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:27:46.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:46.911+0000 7fe39bfff640 1 --2- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 0x7fe388008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:46.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:46.911+0000 7fe39bfff640 10 monclient(hunting): get_auth_request con 0x7fe388006260 auth_method 0 2024-04-26T00:27:46.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:46.911+0000 7fe39bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:46.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:46.911+0000 7fe39bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:46.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:46.911+0000 7fe39bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:27:46.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:46.911+0000 7fe3a0851640 1 --2- 172.21.15.37:0/3899793128 >> v2:172.21.15.37:3300/0 conn(0x7fe388003750 0x7fe388005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:46.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:46.911+0000 7fe3a0851640 1 -- 172.21.15.37:0/3899793128 >> v2:172.21.15.37:3300/0 conn(0x7fe388003750 msgr2=0x7fe388005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:27:46.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:46.911+0000 7fe3a0851640 1 -- 172.21.15.37:0/3899793128 >> v2:172.21.15.37:3300/0 conn(0x7fe388003750 msgr2=0x7fe388005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:27:46.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:46.911+0000 7fe3a0851640 1 --2- 172.21.15.37:0/3899793128 >> v2:172.21.15.37:3300/0 conn(0x7fe388003750 0x7fe388005bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:27:46.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:46.911+0000 7fe39bfff640 1 -- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 msgr2=0x7fe388008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:27:46.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:46.911+0000 7fe3a0851640 1 --2- 172.21.15.37:0/3899793128 >> v2:172.21.15.37:3300/0 conn(0x7fe388003750 0x7fe388005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:27:46.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:46.911+0000 7fe39bfff640 1 -- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 msgr2=0x7fe388008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:27:46.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:46.911+0000 7fe39bfff640 1 --2- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 0x7fe388008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:27:46.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:46.911+0000 7fe39bfff640 1 --2- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 0x7fe388008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:27:48.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.495+0000 7fe39a7fc640 10 monclient(hunting): tick 2024-04-26T00:27:48.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.495+0000 7fe39a7fc640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:27:48.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.495+0000 7fe39a7fc640 1 monclient(hunting): continuing hunt 2024-04-26T00:27:48.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.495+0000 7fe39a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:27:48.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.495+0000 7fe39a7fc640 1 -- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 msgr2=0x7fe388008720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:27:48.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.495+0000 7fe39a7fc640 1 --2- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 0x7fe388008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:48.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.495+0000 7fe39a7fc640 1 -- 172.21.15.37:0/3899793128 >> v2:172.21.15.37:3300/0 conn(0x7fe388003750 msgr2=0x7fe388005bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:27:48.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.495+0000 7fe39a7fc640 1 --2- 172.21.15.37:0/3899793128 >> v2:172.21.15.37:3300/0 conn(0x7fe388003750 0x7fe388005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:48.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.495+0000 7fe39a7fc640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-04-26T00:27:48.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.495+0000 7fe3a1052640 1 -- 172.21.15.37:0/3899793128 reap_dead start 2024-04-26T00:27:48.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.495+0000 7fe39a7fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe39c137780 addr v1:172.21.15.37:6789/0 2024-04-26T00:27:48.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.495+0000 7fe39a7fc640 1 --2- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 0x7fe388008700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:48.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.495+0000 7fe39a7fc640 10 monclient(hunting): picked mon.noname-b con 0x7fe388006260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:27:48.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.495+0000 7fe39a7fc640 1 --2- 172.21.15.37:0/3899793128 >> v2:172.21.15.45:3300/0 conn(0x7fe388003b80 0x7fe38800c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:48.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.495+0000 7fe39a7fc640 10 monclient(hunting): picked mon.noname-c con 0x7fe388003b80 addr v2:172.21.15.45:3300/0 2024-04-26T00:27:48.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.495+0000 7fe39a7fc640 1 -- 172.21.15.37:0/3899793128 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe39c09bc50 con 0x7fe39c137780 2024-04-26T00:27:48.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.495+0000 7fe39a7fc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:48.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.495+0000 7fe39a7fc640 1 -- 172.21.15.37:0/3899793128 --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fe39c0a0f00 con 0x7fe388003b80 2024-04-26T00:27:48.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.495+0000 7fe39a7fc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:48.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.495+0000 7fe39a7fc640 1 -- 172.21.15.37:0/3899793128 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe388008e30 con 0x7fe388006260 2024-04-26T00:27:48.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.495+0000 7fe39a7fc640 10 monclient(hunting): _renew_subs 2024-04-26T00:27:48.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.499+0000 7fe3a1052640 1 -- 172.21.15.37:0/3899793128 >> v2:172.21.15.45:3300/0 conn(0x7fe388003b80 msgr2=0x7fe38800c960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:27:48.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.499+0000 7fe3a1052640 1 --2- 172.21.15.37:0/3899793128 >> v2:172.21.15.45:3300/0 conn(0x7fe388003b80 0x7fe38800c960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:27:48.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.499+0000 7fe39bfff640 1 --2- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 0x7fe388008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:48.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.499+0000 7fe39bfff640 10 monclient(hunting): get_auth_request con 0x7fe388006260 auth_method 0 2024-04-26T00:27:48.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.499+0000 7fe39bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:48.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.499+0000 7fe39bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:48.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.499+0000 7fe39bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:48.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.499+0000 7fe39bfff640 1 -- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 msgr2=0x7fe388008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:27:48.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.499+0000 7fe39bfff640 1 -- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 msgr2=0x7fe388008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:27:48.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.499+0000 7fe39bfff640 1 --2- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 0x7fe388008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:27:48.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.499+0000 7fe39bfff640 1 --2- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 0x7fe388008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:27:48.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.499+0000 7fe3a0851640 1 -- 172.21.15.37:0/3899793128 >> v1:172.21.15.37:6789/0 conn(0x7fe39c137780 legacy=0x7fe38800b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:48.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.703+0000 7fe3a1052640 1 -- 172.21.15.37:0/3899793128 >> v2:172.21.15.45:3300/0 conn(0x7fe388003b80 msgr2=0x7fe38800c960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:27:48.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.703+0000 7fe3a1052640 1 --2- 172.21.15.37:0/3899793128 >> v2:172.21.15.45:3300/0 conn(0x7fe388003b80 0x7fe38800c960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:27:48.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.703+0000 7fe39bfff640 1 --2- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 0x7fe388008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:48.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.703+0000 7fe39bfff640 10 monclient(hunting): get_auth_request con 0x7fe388006260 auth_method 0 2024-04-26T00:27:48.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.703+0000 7fe39bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:48.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.703+0000 7fe39bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:48.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.703+0000 7fe39bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:27:48.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.703+0000 7fe39bfff640 1 -- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 msgr2=0x7fe388008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:27:48.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.703+0000 7fe39bfff640 1 -- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 msgr2=0x7fe388008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:27:48.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.703+0000 7fe39bfff640 1 --2- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 0x7fe388008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:27:48.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:48.703+0000 7fe39bfff640 1 --2- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 0x7fe388008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:27:49.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:49.107+0000 7fe3a1052640 1 -- 172.21.15.37:0/3899793128 >> v2:172.21.15.45:3300/0 conn(0x7fe388003b80 msgr2=0x7fe38800c960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:27:49.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:49.107+0000 7fe3a1052640 1 --2- 172.21.15.37:0/3899793128 >> v2:172.21.15.45:3300/0 conn(0x7fe388003b80 0x7fe38800c960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:27:49.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:49.107+0000 7fe39bfff640 1 --2- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 0x7fe388008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:49.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:49.107+0000 7fe39bfff640 10 monclient(hunting): get_auth_request con 0x7fe388006260 auth_method 0 2024-04-26T00:27:49.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:49.107+0000 7fe39bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:49.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:49.107+0000 7fe39bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:49.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:49.107+0000 7fe39bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:27:49.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:49.107+0000 7fe39bfff640 1 -- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 msgr2=0x7fe388008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:27:49.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:49.107+0000 7fe39bfff640 1 -- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 msgr2=0x7fe388008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:27:49.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:49.107+0000 7fe39bfff640 1 --2- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 0x7fe388008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:27:49.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:49.107+0000 7fe39bfff640 1 --2- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 0x7fe388008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:27:49.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:49.911+0000 7fe3a1052640 1 -- 172.21.15.37:0/3899793128 >> v2:172.21.15.45:3300/0 conn(0x7fe388003b80 msgr2=0x7fe38800c960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:27:49.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:49.911+0000 7fe3a1052640 1 --2- 172.21.15.37:0/3899793128 >> v2:172.21.15.45:3300/0 conn(0x7fe388003b80 0x7fe38800c960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:27:49.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:49.911+0000 7fe39bfff640 1 --2- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 0x7fe388008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:49.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:49.911+0000 7fe39bfff640 10 monclient(hunting): get_auth_request con 0x7fe388006260 auth_method 0 2024-04-26T00:27:49.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:49.911+0000 7fe39bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:49.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:49.911+0000 7fe39bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:49.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:49.911+0000 7fe39bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:27:49.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:49.911+0000 7fe39bfff640 1 -- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 msgr2=0x7fe388008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:27:49.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:49.911+0000 7fe39bfff640 1 -- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 msgr2=0x7fe388008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:27:49.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:49.911+0000 7fe39bfff640 1 --2- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 0x7fe388008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:27:49.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:49.911+0000 7fe39bfff640 1 --2- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 0x7fe388008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:27:51.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:51.499+0000 7fe39a7fc640 10 monclient(hunting): tick 2024-04-26T00:27:51.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:51.499+0000 7fe39a7fc640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:27:51.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:51.499+0000 7fe39a7fc640 1 monclient(hunting): continuing hunt 2024-04-26T00:27:51.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:51.499+0000 7fe39a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:27:51.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:51.499+0000 7fe39a7fc640 1 -- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 msgr2=0x7fe388008700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:27:51.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:51.499+0000 7fe39a7fc640 1 --2- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe388006260 0x7fe388008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:51.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:51.499+0000 7fe39a7fc640 1 -- 172.21.15.37:0/3899793128 >> v2:172.21.15.45:3300/0 conn(0x7fe388003b80 msgr2=0x7fe38800c960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:27:51.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:51.499+0000 7fe39a7fc640 1 --2- 172.21.15.37:0/3899793128 >> v2:172.21.15.45:3300/0 conn(0x7fe388003b80 0x7fe38800c960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:51.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:51.499+0000 7fe39a7fc640 1 -- 172.21.15.37:0/3899793128 >> v1:172.21.15.37:6789/0 conn(0x7fe39c137780 legacy=0x7fe38800b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:51.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:51.499+0000 7fe39a7fc640 10 monclient: _add_conns ranks=[3,1] 2024-04-26T00:27:51.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:51.499+0000 7fe39a7fc640 1 --2- 172.21.15.37:0/3899793128 >> v2:172.21.15.37:3300/0 conn(0x7fe38800cf50 0x7fe38800d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:51.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:51.499+0000 7fe39a7fc640 10 monclient(hunting): picked mon.noname-a con 0x7fe38800cf50 addr v2:172.21.15.37:3300/0 2024-04-26T00:27:51.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:51.499+0000 7fe39a7fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe38800da30 addr v1:172.21.15.45:6789/0 2024-04-26T00:27:51.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:51.499+0000 7fe39a7fc640 1 -- 172.21.15.37:0/3899793128 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe39c0a0c20 con 0x7fe38800da30 2024-04-26T00:27:51.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:51.499+0000 7fe39a7fc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:51.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:51.499+0000 7fe39a7fc640 1 -- 172.21.15.37:0/3899793128 --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fe39c0a0f00 con 0x7fe38800cf50 2024-04-26T00:27:51.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:51.499+0000 7fe39a7fc640 10 monclient(hunting): _renew_subs 2024-04-26T00:27:51.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:51.499+0000 7fe39bfff640 1 -- 172.21.15.37:0/3899793128 >> v1:172.21.15.45:6789/0 conn(0x7fe38800da30 legacy=0x7fe388011f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:27:51.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:51.499+0000 7fe3a0851640 1 --2- 172.21.15.37:0/3899793128 >> v2:172.21.15.37:3300/0 conn(0x7fe38800cf50 0x7fe38800d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:51.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:51.499+0000 7fe3a0851640 10 monclient(hunting): get_auth_request con 0x7fe38800cf50 auth_method 0 2024-04-26T00:27:51.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:51.499+0000 7fe3a0851640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:51.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:51.499+0000 7fe3a0851640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:51.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:51.499+0000 7fe3a0851640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:51.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:51.499+0000 7fe3a0851640 1 -- 172.21.15.37:0/3899793128 >> v2:172.21.15.37:3300/0 conn(0x7fe38800cf50 msgr2=0x7fe38800d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:27:51.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:51.499+0000 7fe3a0851640 1 -- 172.21.15.37:0/3899793128 >> v2:172.21.15.37:3300/0 conn(0x7fe38800cf50 msgr2=0x7fe38800d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:27:51.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:51.499+0000 7fe3a0851640 1 --2- 172.21.15.37:0/3899793128 >> v2:172.21.15.37:3300/0 conn(0x7fe38800cf50 0x7fe38800d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:27:51.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:51.499+0000 7fe3a0851640 1 --2- 172.21.15.37:0/3899793128 >> v2:172.21.15.37:3300/0 conn(0x7fe38800cf50 0x7fe38800d360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:27:51.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:51.703+0000 7fe39bfff640 1 -- 172.21.15.37:0/3899793128 >> v1:172.21.15.45:6789/0 conn(0x7fe38800da30 legacy=0x7fe388011f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:27:51.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:51.703+0000 7fe3a0851640 1 --2- 172.21.15.37:0/3899793128 >> v2:172.21.15.37:3300/0 conn(0x7fe38800cf50 0x7fe38800d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:51.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:51.703+0000 7fe3a0851640 10 monclient(hunting): get_auth_request con 0x7fe38800cf50 auth_method 0 2024-04-26T00:27:51.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:51.703+0000 7fe3a0851640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:51.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:51.703+0000 7fe3a0851640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:51.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:51.703+0000 7fe3a0851640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:27:51.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:51.703+0000 7fe3a0851640 1 -- 172.21.15.37:0/3899793128 >> v2:172.21.15.37:3300/0 conn(0x7fe38800cf50 msgr2=0x7fe38800d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:27:51.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:51.703+0000 7fe3a0851640 1 -- 172.21.15.37:0/3899793128 >> v2:172.21.15.37:3300/0 conn(0x7fe38800cf50 msgr2=0x7fe38800d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:27:51.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:51.703+0000 7fe3a0851640 1 --2- 172.21.15.37:0/3899793128 >> v2:172.21.15.37:3300/0 conn(0x7fe38800cf50 0x7fe38800d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:27:51.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:51.703+0000 7fe3a0851640 1 --2- 172.21.15.37:0/3899793128 >> v2:172.21.15.37:3300/0 conn(0x7fe38800cf50 0x7fe38800d360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:27:52.037 INFO:tasks.rados.rados.0.smithi045.stdout:1255: done (0 left) 2024-04-26T00:27:52.037 INFO:tasks.rados.rados.0.smithi045.stdout:1256: copy_from oid 46 from oid 14 current snap is 147 2024-04-26T00:27:52.037 INFO:tasks.rados.rados.0.smithi045.stdout:1257: write (excl) oid 50 current snap is 147 2024-04-26T00:27:52.038 INFO:tasks.rados.rados.0.smithi045.stdout:1257: seq_num 470 ranges {699290=464563,1603478=535155,2823983=765846} 2024-04-26T00:27:52.051 INFO:tasks.rados.rados.0.smithi045.stdout:1257: writing smithi04526579-50 from 699290 to 1163853 tid 1 2024-04-26T00:27:52.063 INFO:tasks.rados.rados.0.smithi045.stdout:1257: writing smithi04526579-50 from 1603478 to 2138633 tid 2 2024-04-26T00:27:52.082 INFO:tasks.rados.rados.0.smithi045.stdout:1257: writing smithi04526579-50 from 2823983 to 3589829 tid 3 2024-04-26T00:27:52.082 INFO:tasks.rados.rados.0.smithi045.stdout:1258: read oid 9 snap -1 2024-04-26T00:27:52.082 INFO:tasks.rados.rados.0.smithi045.stdout:1258: expect (ObjNum 462 snap 145 seq_num 462) 2024-04-26T00:27:52.082 INFO:tasks.rados.rados.0.smithi045.stdout:1259: read oid 31 snap -1 2024-04-26T00:27:52.082 INFO:tasks.rados.rados.0.smithi045.stdout:1259: expect (ObjNum 466 snap 145 seq_num 466) 2024-04-26T00:27:52.082 INFO:tasks.rados.rados.0.smithi045.stdout:1260: snap_remove snap 124 2024-04-26T00:27:52.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.107+0000 7fe39bfff640 1 -- 172.21.15.37:0/3899793128 >> v1:172.21.15.45:6789/0 conn(0x7fe38800da30 legacy=0x7fe388011f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:27:52.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.107+0000 7fe3a0851640 1 --2- 172.21.15.37:0/3899793128 >> v2:172.21.15.37:3300/0 conn(0x7fe38800cf50 0x7fe38800d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:52.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.107+0000 7fe3a0851640 10 monclient(hunting): get_auth_request con 0x7fe38800cf50 auth_method 0 2024-04-26T00:27:52.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.107+0000 7fe3a0851640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:52.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.107+0000 7fe3a0851640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:52.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.107+0000 7fe3a0851640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:27:52.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.107+0000 7fe3a0851640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:52.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.107+0000 7fe3a0851640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:52.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.107+0000 7fe3a0851640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:52.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.107+0000 7fe3a0851640 10 monclient(hunting): handle_auth_done global_id 94155 payload 274 2024-04-26T00:27:52.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.107+0000 7fe3a0851640 1 -- 172.21.15.37:0/3899793128 >> v1:172.21.15.45:6789/0 conn(0x7fe38800da30 legacy=0x7fe388011f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:27:52.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.107+0000 7fe3a0851640 10 monclient: _finish_hunting 0 2024-04-26T00:27:52.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.107+0000 7fe3a0851640 1 monclient: found mon.noname-a 2024-04-26T00:27:52.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.107+0000 7fe3a0851640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:27:52.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.107+0000 7fe3a0851640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:27:52.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.107+0000 7fe3a0851640 1 -- 172.21.15.37:0/3899793128 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3880147c0 con 0x7fe38800cf50 2024-04-26T00:27:52.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.115+0000 7fe3a0851640 1 -- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe38800cf50 msgr2=0x7fe38800d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:52.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.115+0000 7fe3a0851640 1 --2- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe38800cf50 0x7fe38800d360 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe38c0038c0 tx=0x7fe38c01da70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=470dc56695ffd29c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:52.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.115+0000 7fe39b7fe640 1 -- 172.21.15.37:0/3899793128 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe38c0ac070 con 0x7fe38800cf50 2024-04-26T00:27:52.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.115+0000 7fe39b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:52.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.115+0000 7fe39b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:27:52.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.115+0000 7fe39b7fe640 10 monclient: dump: 2024-04-26T00:27:52.101 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:52.101 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:52.101 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:52.101 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:52.101 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:52.101 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:52.101 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:52.101 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:52.102 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:52.102 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:52.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.115+0000 7fe39b7fe640 10 monclient: _finish_auth 0 2024-04-26T00:27:52.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.115+0000 7fe39b7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:27:52.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.115+0000 7fe39b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:27:52.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.115+0000 7fe39b7fe640 1 -- 172.21.15.37:0/3899793128 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe38c0a6930 con 0x7fe38800cf50 2024-04-26T00:27:52.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.115+0000 7fe3a2ab5640 5 monclient: authenticate success, global_id 94155 2024-04-26T00:27:52.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.115+0000 7fe39b7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:27:52.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.115+0000 7fe3a2ab5640 10 monclient: get_monmap_and_config success 2024-04-26T00:27:52.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.115+0000 7fe3a2ab5640 4 set_mon_vals no callback set 2024-04-26T00:27:52.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.115+0000 7fe39b7fe640 1 -- 172.21.15.37:0/3899793128 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe38c0a6d00 con 0x7fe38800cf50 2024-04-26T00:27:52.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.115+0000 7fe39b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:52.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.115+0000 7fe39b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:27:52.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.115+0000 7fe39b7fe640 10 monclient: dump: 2024-04-26T00:27:52.103 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:52.103 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:52.103 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:52.103 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:52.103 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:52.103 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:52.103 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:52.103 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:52.103 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:52.103 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:52.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.115+0000 7fe3a2ab5640 10 monclient: shutdown 2024-04-26T00:27:52.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.115+0000 7fe3a2ab5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:27:52.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.115+0000 7fe3a2ab5640 1 -- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe38800cf50 msgr2=0x7fe38800d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:52.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.115+0000 7fe3a2ab5640 1 --2- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe38800cf50 0x7fe38800d360 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe38c0038c0 tx=0x7fe38c01da70 comp rx=0 tx=0).stop 2024-04-26T00:27:52.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.115+0000 7fe3a1052640 1 -- 172.21.15.37:0/3899793128 reap_dead start 2024-04-26T00:27:52.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.115+0000 7fe3a2ab5640 1 -- 172.21.15.37:0/3899793128 shutdown_connections 2024-04-26T00:27:52.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.115+0000 7fe3a2ab5640 1 --2- 172.21.15.37:0/3899793128 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe38800cf50 0x7fe38800d360 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:52.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.115+0000 7fe3a2ab5640 1 -- 172.21.15.37:0/3899793128 >> 172.21.15.37:0/3899793128 conn(0x7fe39c0a1ad0 msgr2=0x7fe39c0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:27:52.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.115+0000 7fe3a2ab5640 1 -- 172.21.15.37:0/3899793128 shutdown_connections 2024-04-26T00:27:52.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.115+0000 7fe3a2ab5640 1 -- 172.21.15.37:0/3899793128 wait complete. 2024-04-26T00:27:52.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a2ab5640 10 monclient: build_initial_monmap 2024-04-26T00:27:52.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a2ab5640 10 monclient: monmap: 2024-04-26T00:27:52.106 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:27:52.106 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:27:52.106 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:27:52.106 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:27:52.106 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:27:52.106 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:27:52.106 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:27:52.106 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:27:52.106 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:27:52.106 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:52.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a2ab5640 1 Processor -- start 2024-04-26T00:27:52.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a2ab5640 1 -- start start 2024-04-26T00:27:52.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a2ab5640 10 monclient: init 2024-04-26T00:27:52.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a2ab5640 10 monclient: _reopen_session rank -1 2024-04-26T00:27:52.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a2ab5640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:27:52.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a2ab5640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe38800cf50 0x7fe39c1cf620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:52.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a2ab5640 10 monclient(hunting): picked mon.noname-a con 0x7fe38800cf50 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:27:52.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a2ab5640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe39c12f6a0 0x7fe39c12fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:52.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a2ab5640 10 monclient(hunting): picked mon.noname-c con 0x7fe39c12f6a0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:27:52.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a2ab5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe39c130020 0x7fe39c11e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:52.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a2ab5640 10 monclient(hunting): picked mon.noname-b con 0x7fe39c130020 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:27:52.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a2ab5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:52.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a2ab5640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe39c131350 con 0x7fe38800cf50 2024-04-26T00:27:52.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a2ab5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:52.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a2ab5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe39c1311d0 con 0x7fe39c130020 2024-04-26T00:27:52.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a2ab5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:52.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a2ab5640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe39c1314d0 con 0x7fe39c12f6a0 2024-04-26T00:27:52.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a2ab5640 10 monclient(hunting): _renew_subs 2024-04-26T00:27:52.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a2ab5640 10 monclient(hunting): authenticate will time out at 1453.425940s 2024-04-26T00:27:52.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a0851640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe38800cf50 0x7fe39c1cf620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:52.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a0851640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe38800cf50 0x7fe39c1cf620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:38558/0 (socket says 172.21.15.37:38558) 2024-04-26T00:27:52.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a0851640 1 -- 172.21.15.37:0/1427556756 learned_addr learned my addr 172.21.15.37:0/1427556756 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:27:52.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a0851640 10 monclient(hunting): get_auth_request con 0x7fe38800cf50 auth_method 0 2024-04-26T00:27:52.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a0851640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:52.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a0851640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:52.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a0851640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:52.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a0851640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:52.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a0851640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:52.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe39bfff640 1 --2- 172.21.15.37:0/1427556756 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe39c12f6a0 0x7fe39c12fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:52.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a0851640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:52.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe39bfff640 10 monclient(hunting): get_auth_request con 0x7fe39c12f6a0 auth_method 0 2024-04-26T00:27:52.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe39bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:52.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe39bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:52.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe39bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:52.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a1052640 1 -- 172.21.15.37:0/1427556756 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe39c130020 msgr2=0x7fe39c11e1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:27:52.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a1052640 1 --2- 172.21.15.37:0/1427556756 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe39c130020 0x7fe39c11e1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:27:52.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe39bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:52.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe39bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:52.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe39bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:52.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a0851640 10 monclient(hunting): handle_auth_done global_id 94158 payload 824 2024-04-26T00:27:52.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a0851640 1 -- 172.21.15.37:0/1427556756 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe39c12f6a0 msgr2=0x7fe39c12fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:52.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a0851640 1 --2- 172.21.15.37:0/1427556756 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe39c12f6a0 0x7fe39c12fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:52.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a0851640 1 -- 172.21.15.37:0/1427556756 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe39c130020 msgr2=0x7fe39c11e1a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:27:52.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a0851640 1 --2- 172.21.15.37:0/1427556756 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe39c130020 0x7fe39c11e1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:52.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a0851640 10 monclient: _finish_hunting 0 2024-04-26T00:27:52.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a0851640 1 monclient: found mon.noname-a 2024-04-26T00:27:52.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a0851640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:27:52.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a0851640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:27:52.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.119+0000 7fe3a0851640 1 -- 172.21.15.37:0/1427556756 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe39c120b40 con 0x7fe38800cf50 2024-04-26T00:27:52.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.123+0000 7fe3a0851640 1 -- 172.21.15.37:0/1427556756 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe38800cf50 msgr2=0x7fe39c1cf620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:52.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.123+0000 7fe3a0851640 1 --2- 172.21.15.37:0/1427556756 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe38800cf50 0x7fe39c1cf620 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe38c01c120 tx=0x7fe38c01a810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:52.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.123+0000 7fe3997fa640 1 -- 172.21.15.37:0/1427556756 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe38c0bf020 con 0x7fe38800cf50 2024-04-26T00:27:52.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.123+0000 7fe3997fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:52.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.123+0000 7fe3997fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:27:52.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.123+0000 7fe3997fa640 10 monclient: dump: 2024-04-26T00:27:52.112 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:52.112 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:52.112 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:52.112 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:52.112 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:52.112 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:52.112 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:52.112 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:52.112 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:52.112 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:52.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.123+0000 7fe3997fa640 10 monclient: _finish_auth 0 2024-04-26T00:27:52.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.123+0000 7fe3997fa640 10 monclient: _check_auth_tickets 2024-04-26T00:27:52.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.123+0000 7fe3997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:27:52.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.123+0000 7fe3997fa640 1 -- 172.21.15.37:0/1427556756 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe38c0bba90 con 0x7fe38800cf50 2024-04-26T00:27:52.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.123+0000 7fe3a2ab5640 5 monclient: authenticate success, global_id 94158 2024-04-26T00:27:52.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.123+0000 7fe3997fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:27:52.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.123+0000 7fe3997fa640 1 -- 172.21.15.37:0/1427556756 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe38c0ac040 con 0x7fe38800cf50 2024-04-26T00:27:52.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.123+0000 7fe3a2ab5640 10 monclient: _renew_subs 2024-04-26T00:27:52.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.123+0000 7fe3a2ab5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:27:52.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.123+0000 7fe3a2ab5640 1 -- 172.21.15.37:0/1427556756 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe39c11fdc0 con 0x7fe38800cf50 2024-04-26T00:27:52.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.123+0000 7fe3997fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:52.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.123+0000 7fe3997fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:27:52.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.123+0000 7fe3997fa640 10 monclient: dump: 2024-04-26T00:27:52.113 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:52.113 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:52.113 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:52.113 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:52.113 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:52.113 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:52.113 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:52.113 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:52.114 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:52.114 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:52.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.123+0000 7fe39affd640 4 set_mon_vals no callback set 2024-04-26T00:27:52.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.123+0000 7fe3a2ab5640 10 monclient: _renew_subs 2024-04-26T00:27:52.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.123+0000 7fe3a2ab5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:27:52.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.123+0000 7fe3a2ab5640 1 -- 172.21.15.37:0/1427556756 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe39c11e6e0 con 0x7fe38800cf50 2024-04-26T00:27:52.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.123+0000 7fe3997fa640 1 -- 172.21.15.37:0/1427556756 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe38c0c2020 con 0x7fe38800cf50 2024-04-26T00:27:52.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.123+0000 7fe3a2ab5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:52.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.123+0000 7fe3a2ab5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:52.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.123+0000 7fe3a2ab5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:27:52.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.123+0000 7fe3997fa640 1 --2- 172.21.15.37:0/1427556756 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe388037bb0 0x7fe38803a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:52.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.123+0000 7fe3a2ab5640 1 -- 172.21.15.37:0/1427556756 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe39c066f40 con 0x7fe38800cf50 2024-04-26T00:27:52.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.123+0000 7fe3997fa640 1 -- 172.21.15.37:0/1427556756 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(299..299 src has 177..299) ==== 3927+0+0 (secure 0 0 0) 0x7fe38c0e48b0 con 0x7fe38800cf50 2024-04-26T00:27:52.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.127+0000 7fe39bfff640 1 --2- 172.21.15.37:0/1427556756 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe388037bb0 0x7fe38803a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:52.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.127+0000 7fe39bfff640 10 monclient: get_auth_request con 0x7fe388037bb0 auth_method 0 2024-04-26T00:27:52.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.131+0000 7fe39bfff640 1 --2- 172.21.15.37:0/1427556756 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe388037bb0 0x7fe38803a070 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe390009010 tx=0x7fe390009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:52.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.131+0000 7fe3997fa640 1 -- 172.21.15.37:0/1427556756 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe38c0c9030 con 0x7fe38800cf50 2024-04-26T00:27:52.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.131+0000 7fe3997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:52.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.131+0000 7fe3997fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:27:52.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.355+0000 7fe3a2ab5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:27:52.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.355+0000 7fe3a2ab5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:27:52.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.355+0000 7fe3a2ab5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:27:52.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.355+0000 7fe3a2ab5640 1 -- 172.21.15.37:0/1427556756 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe39c133f20 con 0x7fe38800cf50 2024-04-26T00:27:52.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.355+0000 7fe3997fa640 1 -- 172.21.15.37:0/1427556756 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7fe38c017a30 con 0x7fe38800cf50 2024-04-26T00:27:52.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.355+0000 7fe3997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:27:52.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.355+0000 7fe3997fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:27:52.340 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":62,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:27:52.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.355+0000 7fe3a2ab5640 1 -- 172.21.15.37:0/1427556756 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe388037bb0 msgr2=0x7fe38803a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:52.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.355+0000 7fe3a2ab5640 1 --2- 172.21.15.37:0/1427556756 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe388037bb0 0x7fe38803a070 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe390009010 tx=0x7fe390009040 comp rx=0 tx=0).stop 2024-04-26T00:27:52.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.355+0000 7fe3a2ab5640 10 monclient: shutdown 2024-04-26T00:27:52.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.355+0000 7fe3a2ab5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:27:52.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.355+0000 7fe3a2ab5640 1 -- 172.21.15.37:0/1427556756 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe38800cf50 msgr2=0x7fe39c1cf620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:52.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.355+0000 7fe3a2ab5640 1 --2- 172.21.15.37:0/1427556756 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe38800cf50 0x7fe39c1cf620 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe38c01c120 tx=0x7fe38c01a810 comp rx=0 tx=0).stop 2024-04-26T00:27:52.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.355+0000 7fe3a2ab5640 1 -- 172.21.15.37:0/1427556756 shutdown_connections 2024-04-26T00:27:52.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.355+0000 7fe3a2ab5640 1 --2- 172.21.15.37:0/1427556756 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe388037bb0 0x7fe38803a070 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:52.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.355+0000 7fe3a2ab5640 1 --2- 172.21.15.37:0/1427556756 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe39c130020 0x7fe39c11e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:52.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.355+0000 7fe3a2ab5640 1 --2- 172.21.15.37:0/1427556756 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe39c12f6a0 0x7fe39c12fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:52.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.355+0000 7fe3a2ab5640 1 --2- 172.21.15.37:0/1427556756 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe38800cf50 0x7fe39c1cf620 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:52.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.355+0000 7fe3a2ab5640 1 -- 172.21.15.37:0/1427556756 >> 172.21.15.37:0/1427556756 conn(0x7fe39c0a1ad0 msgr2=0x7fe39c0a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:27:52.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.355+0000 7fe3a2ab5640 1 -- 172.21.15.37:0/1427556756 shutdown_connections 2024-04-26T00:27:52.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.355+0000 7fe3a2ab5640 1 -- 172.21.15.37:0/1427556756 wait complete. 2024-04-26T00:27:52.359 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-26T00:27:52.360 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-04-26T00:27:52.360 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:27:52.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f28147c1640 10 monclient: get_monmap_and_config 2024-04-26T00:27:52.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f28147c1640 10 monclient: build_initial_monmap 2024-04-26T00:27:52.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f28147c1640 10 monclient: monmap: 2024-04-26T00:27:52.483 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:27:52.483 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:52.484 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:27:52.500219+0000 2024-04-26T00:27:52.484 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:27:52.500219+0000 2024-04-26T00:27:52.484 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:27:52.484 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:52.484 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:27:52.484 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:27:52.484 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:27:52.484 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:27:52.484 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:27:52.484 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:52.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f28147c1640 1 Processor -- start 2024-04-26T00:27:52.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f28147c1640 1 -- start start 2024-04-26T00:27:52.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f28147c1640 10 monclient: init 2024-04-26T00:27:52.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f28147c1640 10 monclient: _reopen_session rank -1 2024-04-26T00:27:52.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f28147c1640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-04-26T00:27:52.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f28147c1640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f280c0a11e0 0x7f280c09f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:52.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f28147c1640 10 monclient(hunting): picked mon.noname-a con 0x7f280c0a11e0 addr v2:172.21.15.37:3300/0 2024-04-26T00:27:52.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f28147c1640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f280c06c2e0 0x7f280c06c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:52.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f28147c1640 10 monclient(hunting): picked mon.noname-c con 0x7f280c06c2e0 addr v2:172.21.15.45:3300/0 2024-04-26T00:27:52.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f28147c1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f280c06cc80 addr v1:172.21.15.45:6789/0 2024-04-26T00:27:52.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f28147c1640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f280c0a3780 con 0x7f280c06cc80 2024-04-26T00:27:52.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f28147c1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:52.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f28147c1640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f280c073370 con 0x7f280c0a11e0 2024-04-26T00:27:52.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f28147c1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:52.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f28147c1640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f280c0a39c0 con 0x7f280c06c2e0 2024-04-26T00:27:52.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f28147c1640 10 monclient(hunting): _renew_subs 2024-04-26T00:27:52.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f28147c1640 10 monclient(hunting): authenticate will time out at 1453.805874s 2024-04-26T00:27:52.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f281255d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f280c0a11e0 0x7f280c09f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:52.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f2812d5e640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f280c06cc80 legacy=0x7f280c13de60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:27:52.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f2811d5c640 1 -- >> v2:172.21.15.45:3300/0 conn(0x7f280c06c2e0 msgr2=0x7f280c06c740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:27:52.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f2811d5c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f280c06c2e0 0x7f280c06c740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:27:52.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f281255d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f280c0a11e0 0x7f280c09f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:38572/0 (socket says 172.21.15.37:38572) 2024-04-26T00:27:52.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f281255d640 1 -- 172.21.15.37:0/3540707912 learned_addr learned my addr 172.21.15.37:0/3540707912 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:27:52.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f281255d640 10 monclient(hunting): get_auth_request con 0x7f280c0a11e0 auth_method 0 2024-04-26T00:27:52.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f281255d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:52.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f281255d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:52.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f281255d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:52.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f281255d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:52.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f281255d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:52.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f281255d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:52.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f281255d640 10 monclient(hunting): handle_auth_done global_id 94161 payload 274 2024-04-26T00:27:52.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f281255d640 1 -- 172.21.15.37:0/3540707912 >> v2:172.21.15.45:3300/0 conn(0x7f280c06c2e0 msgr2=0x7f280c06c740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:27:52.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f281255d640 1 --2- 172.21.15.37:0/3540707912 >> v2:172.21.15.45:3300/0 conn(0x7f280c06c2e0 0x7f280c06c740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:52.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f281255d640 1 -- 172.21.15.37:0/3540707912 >> v1:172.21.15.45:6789/0 conn(0x7f280c06cc80 legacy=0x7f280c13de60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:27:52.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f281255d640 10 monclient: _finish_hunting 0 2024-04-26T00:27:52.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f281255d640 1 monclient: found mon.noname-a 2024-04-26T00:27:52.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f281255d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:27:52.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f281255d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:27:52.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.499+0000 7f281255d640 1 -- 172.21.15.37:0/3540707912 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f280c09e770 con 0x7f280c0a11e0 2024-04-26T00:27:52.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.503+0000 7f281255d640 1 -- 172.21.15.37:0/3540707912 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f280c0a11e0 msgr2=0x7f280c09f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:52.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.503+0000 7f281255d640 1 --2- 172.21.15.37:0/3540707912 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f280c0a11e0 0x7f280c09f9b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f280c0a4bf0 tx=0x7f280401c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7edf2de0e0ff306c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:52.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.503+0000 7f281155b640 1 -- 172.21.15.37:0/3540707912 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28040a5660 con 0x7f280c0a11e0 2024-04-26T00:27:52.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.503+0000 7f281155b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:52.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.503+0000 7f281155b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:27:52.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.503+0000 7f281155b640 10 monclient: dump: 2024-04-26T00:27:52.491 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:52.491 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:52.491 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:52.491 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:52.491 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:52.491 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:52.491 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:52.491 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:52.491 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:52.491 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:52.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.503+0000 7f281155b640 10 monclient: _finish_auth 0 2024-04-26T00:27:52.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.503+0000 7f281155b640 10 monclient: _check_auth_tickets 2024-04-26T00:27:52.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.503+0000 7f281155b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:27:52.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.503+0000 7f28147c1640 5 monclient: authenticate success, global_id 94161 2024-04-26T00:27:52.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.503+0000 7f281155b640 1 -- 172.21.15.37:0/3540707912 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28040a5800 con 0x7f280c0a11e0 2024-04-26T00:27:52.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.503+0000 7f28147c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:27:52.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.503+0000 7f281155b640 10 monclient: handle_config config(0 keys) 2024-04-26T00:27:52.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.503+0000 7f28147c1640 10 monclient: get_monmap_and_config success 2024-04-26T00:27:52.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.503+0000 7f28147c1640 4 set_mon_vals no callback set 2024-04-26T00:27:52.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.503+0000 7f281155b640 1 -- 172.21.15.37:0/3540707912 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28040a5bd0 con 0x7f280c0a11e0 2024-04-26T00:27:52.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.503+0000 7f281155b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:52.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.503+0000 7f281155b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:27:52.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.503+0000 7f281155b640 10 monclient: dump: 2024-04-26T00:27:52.492 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:52.492 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:52.492 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:52.492 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:52.492 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:52.492 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:52.492 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:52.493 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:52.493 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:52.493 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:52.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.503+0000 7f28147c1640 10 monclient: shutdown 2024-04-26T00:27:52.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.503+0000 7f28147c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:27:52.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.503+0000 7f28147c1640 1 -- 172.21.15.37:0/3540707912 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f280c0a11e0 msgr2=0x7f280c09f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:52.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.503+0000 7f28147c1640 1 --2- 172.21.15.37:0/3540707912 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f280c0a11e0 0x7f280c09f9b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f280c0a4bf0 tx=0x7f280401c890 comp rx=0 tx=0).stop 2024-04-26T00:27:52.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.503+0000 7f28147c1640 1 -- 172.21.15.37:0/3540707912 shutdown_connections 2024-04-26T00:27:52.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.503+0000 7f28147c1640 1 --2- 172.21.15.37:0/3540707912 >> v2:172.21.15.45:3300/0 conn(0x7f280c06c2e0 0x7f280c06c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:52.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.503+0000 7f28147c1640 1 --2- 172.21.15.37:0/3540707912 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f280c0a11e0 0x7f280c09f9b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:52.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.503+0000 7f28147c1640 1 -- 172.21.15.37:0/3540707912 >> 172.21.15.37:0/3540707912 conn(0x7f280c0a7de0 msgr2=0x7f280c0a3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:27:52.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.503+0000 7f28147c1640 1 -- 172.21.15.37:0/3540707912 shutdown_connections 2024-04-26T00:27:52.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.503+0000 7f28147c1640 1 -- 172.21.15.37:0/3540707912 wait complete. 2024-04-26T00:27:52.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f28147c1640 10 monclient: build_initial_monmap 2024-04-26T00:27:52.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f28147c1640 10 monclient: monmap: 2024-04-26T00:27:52.498 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:27:52.498 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:27:52.498 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:27:52.498 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:27:52.498 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:27:52.499 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:27:52.499 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:27:52.499 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:27:52.499 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:27:52.499 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:52.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f28147c1640 1 Processor -- start 2024-04-26T00:27:52.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f28147c1640 1 -- start start 2024-04-26T00:27:52.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f28147c1640 10 monclient: init 2024-04-26T00:27:52.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f28147c1640 10 monclient: _reopen_session rank -1 2024-04-26T00:27:52.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f28147c1640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:27:52.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f28147c1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f280c06c2e0 0x7f280c1cee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:52.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f28147c1640 10 monclient(hunting): picked mon.noname-b con 0x7f280c06c2e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:27:52.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f28147c1640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f280c06cc80 0x7f280c137e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:52.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f28147c1640 10 monclient(hunting): picked mon.noname-a con 0x7f280c06cc80 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:27:52.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f28147c1640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f280c0a11e0 0x7f280c13c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:52.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f28147c1640 10 monclient(hunting): picked mon.noname-c con 0x7f280c0a11e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:27:52.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f28147c1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:52.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f28147c1640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f280c0a3060 con 0x7f280c06cc80 2024-04-26T00:27:52.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f28147c1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:52.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f28147c1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f280c0a2ee0 con 0x7f280c06c2e0 2024-04-26T00:27:52.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f28147c1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:52.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f28147c1640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f280c0a15c0 con 0x7f280c0a11e0 2024-04-26T00:27:52.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f28147c1640 10 monclient(hunting): _renew_subs 2024-04-26T00:27:52.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f28147c1640 10 monclient(hunting): authenticate will time out at 1453.813624s 2024-04-26T00:27:52.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f2811d5c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f280c06cc80 0x7f280c137e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:52.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f281255d640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f280c06c2e0 msgr2=0x7f280c1cee40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:27:52.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f281255d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f280c06c2e0 0x7f280c1cee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:27:52.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f2811d5c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f280c06cc80 0x7f280c137e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:38578/0 (socket says 172.21.15.37:38578) 2024-04-26T00:27:52.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f2811d5c640 1 -- 172.21.15.37:0/2960193226 learned_addr learned my addr 172.21.15.37:0/2960193226 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:27:52.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f2811d5c640 10 monclient(hunting): get_auth_request con 0x7f280c06cc80 auth_method 0 2024-04-26T00:27:52.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f2811d5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:52.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f2811d5c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:52.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f2811d5c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:52.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f2812d5e640 1 --2- 172.21.15.37:0/2960193226 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f280c0a11e0 0x7f280c13c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:52.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f2812d5e640 10 monclient(hunting): get_auth_request con 0x7f280c0a11e0 auth_method 0 2024-04-26T00:27:52.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f2812d5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:52.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f2812d5e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:52.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f2812d5e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:52.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f2811d5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:52.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f2811d5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:52.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f2811d5c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:52.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f2812d5e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:52.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f2812d5e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:52.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f2812d5e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:52.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f2811d5c640 10 monclient(hunting): handle_auth_done global_id 94164 payload 824 2024-04-26T00:27:52.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f2811d5c640 1 -- 172.21.15.37:0/2960193226 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f280c0a11e0 msgr2=0x7f280c13c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:52.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f2811d5c640 1 --2- 172.21.15.37:0/2960193226 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f280c0a11e0 0x7f280c13c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:52.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f2811d5c640 1 -- 172.21.15.37:0/2960193226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f280c06c2e0 msgr2=0x7f280c1cee40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:27:52.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f2811d5c640 1 --2- 172.21.15.37:0/2960193226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f280c06c2e0 0x7f280c1cee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:52.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f2811d5c640 10 monclient: _finish_hunting 0 2024-04-26T00:27:52.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f2811d5c640 1 monclient: found mon.noname-a 2024-04-26T00:27:52.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f2811d5c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:27:52.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f2811d5c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:27:52.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.507+0000 7f2811d5c640 1 -- 172.21.15.37:0/2960193226 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f280c13e8a0 con 0x7f280c06cc80 2024-04-26T00:27:52.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f2811d5c640 1 -- 172.21.15.37:0/2960193226 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f280c06cc80 msgr2=0x7f280c137e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:52.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f2811d5c640 1 --2- 172.21.15.37:0/2960193226 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f280c06cc80 0x7f280c137e60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f27fc00b580 tx=0x7f27fc00ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:52.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f2802ffd640 1 -- 172.21.15.37:0/2960193226 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27fc016070 con 0x7f280c06cc80 2024-04-26T00:27:52.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f2802ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:52.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f2802ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:27:52.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f2802ffd640 10 monclient: dump: 2024-04-26T00:27:52.503 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:52.503 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:52.503 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:52.503 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:52.503 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:52.503 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:52.503 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:52.503 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:52.503 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:52.503 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:52.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f2802ffd640 10 monclient: _finish_auth 0 2024-04-26T00:27:52.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f2802ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:27:52.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f2802ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:27:52.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f2802ffd640 1 -- 172.21.15.37:0/2960193226 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27fc0050d0 con 0x7f280c06cc80 2024-04-26T00:27:52.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f2802ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:27:52.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f2802ffd640 1 -- 172.21.15.37:0/2960193226 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27fc0054c0 con 0x7f280c06cc80 2024-04-26T00:27:52.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f2802ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:52.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f2802ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:27:52.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f2802ffd640 10 monclient: dump: 2024-04-26T00:27:52.504 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:52.504 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:52.504 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:52.504 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:52.504 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:52.504 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:52.504 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:52.504 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:52.504 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:52.505 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:52.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f2810d5a640 4 set_mon_vals no callback set 2024-04-26T00:27:52.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f28147c1640 5 monclient: authenticate success, global_id 94164 2024-04-26T00:27:52.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f28147c1640 10 monclient: _renew_subs 2024-04-26T00:27:52.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f28147c1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:27:52.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f28147c1640 1 -- 172.21.15.37:0/2960193226 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f280c1cfd50 con 0x7f280c06cc80 2024-04-26T00:27:52.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f28147c1640 10 monclient: _renew_subs 2024-04-26T00:27:52.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f28147c1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:27:52.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f28147c1640 1 -- 172.21.15.37:0/2960193226 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f280c121f50 con 0x7f280c06cc80 2024-04-26T00:27:52.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f28147c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:52.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f28147c1640 1 --2- 172.21.15.37:0/2960193226 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f280c13c8e0 0x7f280c13ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:52.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f28147c1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:52.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f28147c1640 1 -- 172.21.15.37:0/2960193226 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f280c13d200 con 0x7f280c13c8e0 2024-04-26T00:27:52.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f281255d640 1 --2- 172.21.15.37:0/2960193226 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f280c13c8e0 0x7f280c13ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:52.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f281255d640 10 monclient: get_auth_request con 0x7f280c13c8e0 auth_method 0 2024-04-26T00:27:52.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f281255d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:52.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f281255d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:52.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f281255d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:52.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f2802ffd640 1 -- 172.21.15.37:0/2960193226 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f27fc00f3e0 con 0x7f280c06cc80 2024-04-26T00:27:52.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f281255d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:52.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f281255d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:52.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f281255d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:52.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f2802ffd640 1 --2- 172.21.15.37:0/2960193226 >> v2:172.21.15.45:6812/2379918623 conn(0x7f27f0035e30 0x7f27f00382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:52.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f2802ffd640 1 -- 172.21.15.37:0/2960193226 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(299..299 src has 177..299) ==== 3927+0+0 (secure 0 0 0) 0x7f27fc047c80 con 0x7f280c06cc80 2024-04-26T00:27:52.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f281255d640 10 monclient(hunting): handle_auth_done global_id 94167 payload 824 2024-04-26T00:27:52.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f281255d640 1 -- 172.21.15.37:0/2960193226 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f280c04f510 con 0x7f280c13c8e0 2024-04-26T00:27:52.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f2812d5e640 1 --2- 172.21.15.37:0/2960193226 >> v2:172.21.15.45:6812/2379918623 conn(0x7f27f0035e30 0x7f27f00382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:52.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f2812d5e640 10 monclient: get_auth_request con 0x7f27f0035e30 auth_method 0 2024-04-26T00:27:52.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f2812d5e640 1 -- 172.21.15.37:0/2960193226 >> v2:172.21.15.45:6812/2379918623 conn(0x7f27f0035e30 msgr2=0x7f27f00382f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:27:52.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f2812d5e640 1 -- 172.21.15.37:0/2960193226 >> v2:172.21.15.45:6812/2379918623 conn(0x7f27f0035e30 msgr2=0x7f27f00382f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:27:52.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f2812d5e640 1 --2- 172.21.15.37:0/2960193226 >> v2:172.21.15.45:6812/2379918623 conn(0x7f27f0035e30 0x7f27f00382f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:27:52.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.511+0000 7f2812d5e640 1 --2- 172.21.15.37:0/2960193226 >> v2:172.21.15.45:6812/2379918623 conn(0x7f27f0035e30 0x7f27f00382f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:27:52.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.515+0000 7f281255d640 1 -- 172.21.15.37:0/2960193226 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f280c13c8e0 msgr2=0x7f280c13ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:52.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.515+0000 7f2802ffd640 1 -- 172.21.15.37:0/2960193226 DispatchQueue::entry inject delay of 0.002000 2024-04-26T00:27:52.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.515+0000 7f281255d640 1 --2- 172.21.15.37:0/2960193226 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f280c13c8e0 0x7f280c13ccc0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f280c0a3d20 tx=0x7f2804019630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:52.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.515+0000 7f2802ffd640 1 -- 172.21.15.37:0/2960193226 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28040bc020 con 0x7f280c13c8e0 2024-04-26T00:27:52.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.515+0000 7f2802ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:52.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.515+0000 7f2802ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:27:52.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.515+0000 7f2802ffd640 10 monclient: dump: 2024-04-26T00:27:52.508 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:52.509 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:52.509 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:52.509 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:52.509 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:52.509 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:52.509 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:52.509 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:52.509 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:52.509 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:52.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.515+0000 7f2802ffd640 1 -- 172.21.15.37:0/2960193226 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f28040b7660 con 0x7f280c13c8e0 2024-04-26T00:27:52.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.515+0000 7f2802ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:52.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.515+0000 7f2802ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:27:52.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.515+0000 7f2802ffd640 1 -- 172.21.15.37:0/2960193226 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f280c13c8e0 msgr2=0x7f280c13ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:52.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.515+0000 7f2802ffd640 1 --2- 172.21.15.37:0/2960193226 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f280c13c8e0 0x7f280c13ccc0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f280c0a3d20 tx=0x7f2804019630 comp rx=0 tx=0).stop 2024-04-26T00:27:52.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.515+0000 7f2802ffd640 1 -- 172.21.15.37:0/2960193226 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f280c13c8e0 msgr2=0x7f280c13ccc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:27:52.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.515+0000 7f2802ffd640 1 --2- 172.21.15.37:0/2960193226 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f280c13c8e0 0x7f280c13ccc0 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f280c0a3d20 tx=0x7f2804019630 comp rx=0 tx=0).stop 2024-04-26T00:27:52.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.523+0000 7f28147c1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:27:52.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.523+0000 7f28147c1640 1 --2- 172.21.15.37:0/2960193226 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f280c09b0a0 0x7f280c09b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:52.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.523+0000 7f28147c1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:52.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.523+0000 7f28147c1640 1 -- 172.21.15.37:0/2960193226 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f280c13f2d0 con 0x7f280c09b0a0 2024-04-26T00:27:52.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.523+0000 7f281255d640 1 --2- 172.21.15.37:0/2960193226 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f280c09b0a0 0x7f280c09b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:52.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.523+0000 7f281255d640 10 monclient: get_auth_request con 0x7f280c09b0a0 auth_method 0 2024-04-26T00:27:52.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.523+0000 7f281255d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:52.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.523+0000 7f281255d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:52.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.523+0000 7f281255d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:52.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.523+0000 7f281255d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:52.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.523+0000 7f281255d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:52.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.523+0000 7f281255d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:52.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.523+0000 7f281255d640 10 monclient(hunting): handle_auth_done global_id 94170 payload 824 2024-04-26T00:27:52.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.523+0000 7f281255d640 1 -- 172.21.15.37:0/2960193226 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f280c122310 con 0x7f280c09b0a0 2024-04-26T00:27:52.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.523+0000 7f281255d640 1 -- 172.21.15.37:0/2960193226 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f280c09b0a0 msgr2=0x7f280c09b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:52.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.523+0000 7f281255d640 1 --2- 172.21.15.37:0/2960193226 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f280c09b0a0 0x7f280c09b480 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f280401af40 tx=0x7f2804009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:52.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.527+0000 7f2802ffd640 1 -- 172.21.15.37:0/2960193226 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28040bc050 con 0x7f280c09b0a0 2024-04-26T00:27:52.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.527+0000 7f2802ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:52.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.527+0000 7f2802ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:27:52.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.527+0000 7f2802ffd640 10 monclient: dump: 2024-04-26T00:27:52.512 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:52.512 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:52.512 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:52.512 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:52.512 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:52.512 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:52.512 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:52.512 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:52.512 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:52.512 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:52.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.527+0000 7f2802ffd640 1 -- 172.21.15.37:0/2960193226 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4707 (secure 0 0 0) 0x7f28040b7a10 con 0x7f280c09b0a0 2024-04-26T00:27:52.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.527+0000 7f2802ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:27:52.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.527+0000 7f2802ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:27:52.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.527+0000 7f2802ffd640 1 -- 172.21.15.37:0/2960193226 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f280c09b0a0 msgr2=0x7f280c09b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:52.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.527+0000 7f2802ffd640 1 --2- 172.21.15.37:0/2960193226 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f280c09b0a0 0x7f280c09b480 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f280401af40 tx=0x7f2804009810 comp rx=0 tx=0).stop 2024-04-26T00:27:52.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.527+0000 7f2802ffd640 1 -- 172.21.15.37:0/2960193226 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f280c09b0a0 msgr2=0x7f280c09b480 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:27:52.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.527+0000 7f2802ffd640 1 --2- 172.21.15.37:0/2960193226 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f280c09b0a0 0x7f280c09b480 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f280401af40 tx=0x7f2804009810 comp rx=0 tx=0).stop 2024-04-26T00:27:52.513 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:27:52.513 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:27:52.513 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:27:52.513 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:27:52.513 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 62, 2024-04-26T00:27:52.513 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 33287, 2024-04-26T00:27:52.514 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:27:52.514 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:27:52.514 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:27:52.514 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:52.514 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:27:52.514 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:27:52.514 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:27:52.514 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:27:52.514 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:27:52.514 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:27:52.514 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:27:52.514 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:27:52.514 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:27:52.514 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:27:52.515 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:27:52.515 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:27:52.515 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:27:52.515 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:27:52.515 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:27:52.515 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:52.515 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:27:52.515 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:27:52.515 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:27:52.515 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:27:52.515 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:27:52.515 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:27:52.515 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:27:52.515 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:27:52.516 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:27:52.516 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:27:52.516 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:27:52.516 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:27:52.516 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:27:52.516 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:52.516 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:52.516 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:27:52.516 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:27:52.517 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:27:52.517 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:27:52.517 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:27:52.517 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:27:52.517 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:27:52.517 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:27:52.517 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:27:52.517 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:27:52.517 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:27:52.517 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:27:52.518 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:27:52.518 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:27:52.518 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:27:52.518 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:27:52.518 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:27:52.518 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:27:52.518 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:27:52.518 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:27:52.518 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:27:52.518 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:27:52.518 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:27:52.518 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:27:52.518 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:27:52.519 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:27:52.519 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:27:52.519 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:27:52.519 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:52.519 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:27:52.519 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:52.519 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:27:52.519 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:52.519 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:27:52.519 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:27:52.519 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:27:52.519 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:27:52.519 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:52.519 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:27:52.519 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:27:52.520 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:52.520 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:52.520 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:52.520 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:27:52.521 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:27:52.521 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:52.521 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:52.521 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:52.521 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:52.521 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:27:52.521 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:27:52.521 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:27:52.521 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:27:52.521 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:27:52.521 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:52.521 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:52.521 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:27:52.521 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:27:52.522 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:27:52.522 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:27:52.522 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:52.522 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:27:52.522 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:27:52.522 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:52.522 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:52.522 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:52.522 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:27:52.522 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:27:52.522 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:52.522 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:52.522 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:52.522 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:52.523 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:27:52.523 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:27:52.523 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:27:52.523 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:27:52.523 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:27:52.523 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:52.524 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:52.524 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:27:52.524 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:27:52.524 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:27:52.524 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:27:52.524 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:52.524 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:27:52.524 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:27:52.524 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:52.524 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:52.524 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:52.524 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:27:52.525 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:27:52.525 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:52.525 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:52.525 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:52.525 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:52.525 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:27:52.525 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:27:52.525 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:27:52.525 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:27:52.525 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:27:52.525 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:52.525 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:52.525 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:52.525 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:27:52.525 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:27:52.526 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:52.526 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:27:52.526 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:27:52.526 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:27:52.526 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:52.526 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:52.526 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:27:52.526 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:52.526 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:27:52.526 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:27:52.526 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:27:52.526 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:52.526 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:52.526 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:27:52.527 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:52.527 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:27:52.527 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:27:52.527 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:27:52.527 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:52.527 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:52.527 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:52.527 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:27:52.527 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:27:52.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.527+0000 7f28147c1640 1 -- 172.21.15.37:0/2960193226 >> v2:172.21.15.45:6812/2379918623 conn(0x7f27f0035e30 msgr2=0x7f27f00382f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:27:52.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.527+0000 7f28147c1640 1 --2- 172.21.15.37:0/2960193226 >> v2:172.21.15.45:6812/2379918623 conn(0x7f27f0035e30 0x7f27f00382f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:52.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.527+0000 7f28147c1640 10 monclient: shutdown 2024-04-26T00:27:52.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.527+0000 7f28147c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:27:52.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.527+0000 7f28147c1640 1 -- 172.21.15.37:0/2960193226 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f280c06cc80 msgr2=0x7f280c137e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:52.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.527+0000 7f2812d5e640 1 -- 172.21.15.37:0/2960193226 reap_dead start 2024-04-26T00:27:52.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.527+0000 7f28147c1640 1 --2- 172.21.15.37:0/2960193226 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f280c06cc80 0x7f280c137e60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f27fc00b580 tx=0x7f27fc00ba40 comp rx=0 tx=0).stop 2024-04-26T00:27:52.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.527+0000 7f28147c1640 1 -- 172.21.15.37:0/2960193226 shutdown_connections 2024-04-26T00:27:52.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.527+0000 7f28147c1640 1 --2- 172.21.15.37:0/2960193226 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f280c06cc80 0x7f280c137e60 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:52.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.527+0000 7f28147c1640 1 -- 172.21.15.37:0/2960193226 >> 172.21.15.37:0/2960193226 conn(0x7f280c0a7de0 msgr2=0x7f280c0a0140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:27:52.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.527+0000 7f28147c1640 1 -- 172.21.15.37:0/2960193226 shutdown_connections 2024-04-26T00:27:52.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.527+0000 7f28147c1640 1 -- 172.21.15.37:0/2960193226 wait complete. 2024-04-26T00:27:52.531 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:27:52.531 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-04-26T00:27:52.531 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:27:52.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.667+0000 7f91fbf62640 10 monclient: get_monmap_and_config 2024-04-26T00:27:52.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.667+0000 7f91fbf62640 10 monclient: build_initial_monmap 2024-04-26T00:27:52.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.667+0000 7f91fbf62640 10 monclient: monmap: 2024-04-26T00:27:52.655 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:27:52.655 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:52.655 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:27:52.671839+0000 2024-04-26T00:27:52.655 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:27:52.671839+0000 2024-04-26T00:27:52.655 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:27:52.655 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:52.655 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:27:52.655 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:27:52.655 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:27:52.655 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:27:52.656 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:27:52.656 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:52.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.671+0000 7f91fbf62640 1 Processor -- start 2024-04-26T00:27:52.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.671+0000 7f91fbf62640 1 -- start start 2024-04-26T00:27:52.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.671+0000 7f91fbf62640 10 monclient: init 2024-04-26T00:27:52.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.671+0000 7f91fbf62640 10 monclient: _reopen_session rank -1 2024-04-26T00:27:52.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.671+0000 7f91fbf62640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-04-26T00:27:52.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.671+0000 7f91fbf62640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91f40a0300 0x7f91f409fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:52.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.671+0000 7f91fbf62640 10 monclient(hunting): picked mon.noname-b con 0x7f91f40a0300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:27:52.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.671+0000 7f91fbf62640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f91f40a06e0 addr v1:172.21.15.37:6789/0 2024-04-26T00:27:52.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.671+0000 7f91fbf62640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f91f41352b0 addr v1:172.21.15.45:6789/0 2024-04-26T00:27:52.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.671+0000 7f91fbf62640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91f409c190 con 0x7f91f40a06e0 2024-04-26T00:27:52.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.671+0000 7f91fbf62640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91f40a1110 con 0x7f91f41352b0 2024-04-26T00:27:52.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.671+0000 7f91fbf62640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:52.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.671+0000 7f91fbf62640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f91f40a13f0 con 0x7f91f40a0300 2024-04-26T00:27:52.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.671+0000 7f91fbf62640 10 monclient(hunting): _renew_subs 2024-04-26T00:27:52.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.671+0000 7f91fbf62640 10 monclient(hunting): authenticate will time out at 1453.977556s 2024-04-26T00:27:52.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.671+0000 7f91f94fd640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f91f40a06e0 0x7f91f4134ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:58488/0 (socket says 172.21.15.37:58488) 2024-04-26T00:27:52.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.671+0000 7f91f94fd640 1 -- 172.21.15.37:0/3612968874 learned_addr learned my addr 172.21.15.37:0/3612968874 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:27:52.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.671+0000 7f91f9cfe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91f40a0300 0x7f91f409fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:52.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.671+0000 7f91fa4ff640 1 -- 172.21.15.37:0/3612968874 >> v1:172.21.15.45:6789/0 conn(0x7f91f41352b0 legacy=0x7f91f413cee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:27:52.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.671+0000 7f91f9cfe640 10 monclient(hunting): get_auth_request con 0x7f91f40a0300 auth_method 0 2024-04-26T00:27:52.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.671+0000 7f91f9cfe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:52.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.671+0000 7f91f9cfe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:52.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.671+0000 7f91f9cfe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:52.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.671+0000 7f91f9cfe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:52.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.671+0000 7f91f9cfe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:52.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.671+0000 7f91f9cfe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:52.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.671+0000 7f91f9cfe640 10 monclient(hunting): handle_auth_done global_id 84257 payload 274 2024-04-26T00:27:52.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.671+0000 7f91f9cfe640 1 -- 172.21.15.37:0/3612968874 >> v1:172.21.15.45:6789/0 conn(0x7f91f41352b0 legacy=0x7f91f413cee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:27:52.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.671+0000 7f91f9cfe640 1 -- 172.21.15.37:0/3612968874 >> v1:172.21.15.37:6789/0 conn(0x7f91f40a06e0 legacy=0x7f91f4134ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:27:52.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.671+0000 7f91f9cfe640 10 monclient: _finish_hunting 0 2024-04-26T00:27:52.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.671+0000 7f91f9cfe640 1 monclient: found mon.noname-b 2024-04-26T00:27:52.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.671+0000 7f91f9cfe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:27:52.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.671+0000 7f91f9cfe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:27:52.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.671+0000 7f91f9cfe640 1 -- 172.21.15.37:0/3612968874 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91f40a16d0 con 0x7f91f40a0300 2024-04-26T00:27:52.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.675+0000 7f91f9cfe640 1 -- 172.21.15.37:0/3612968874 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91f40a0300 msgr2=0x7f91f409fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:52.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.675+0000 7f91f9cfe640 1 --2- 172.21.15.37:0/3612968874 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91f40a0300 0x7f91f409fc50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f91f00097f0 tx=0x7f91f001c320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1f698af816dab26 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:52.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.675+0000 7f91f8cfc640 1 -- 172.21.15.37:0/3612968874 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91f00b0070 con 0x7f91f40a0300 2024-04-26T00:27:52.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.675+0000 7f91f8cfc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:52.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.675+0000 7f91f8cfc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:27:52.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.675+0000 7f91f8cfc640 10 monclient: dump: 2024-04-26T00:27:52.662 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:52.662 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:52.663 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:52.663 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:52.663 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:52.663 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:52.663 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:52.663 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:52.663 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:52.663 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:52.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.675+0000 7f91f8cfc640 10 monclient: _finish_auth 0 2024-04-26T00:27:52.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.675+0000 7f91f8cfc640 10 monclient: _check_auth_tickets 2024-04-26T00:27:52.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.675+0000 7f91f8cfc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:27:52.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.675+0000 7f91fbf62640 5 monclient: authenticate success, global_id 84257 2024-04-26T00:27:52.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.675+0000 7f91f8cfc640 1 -- 172.21.15.37:0/3612968874 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91f00ab600 con 0x7f91f40a0300 2024-04-26T00:27:52.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.675+0000 7f91fbf62640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:27:52.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.675+0000 7f91f8cfc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:27:52.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.675+0000 7f91fbf62640 10 monclient: get_monmap_and_config success 2024-04-26T00:27:52.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.675+0000 7f91fbf62640 4 set_mon_vals no callback set 2024-04-26T00:27:52.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.675+0000 7f91f8cfc640 1 -- 172.21.15.37:0/3612968874 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91f00ab9d0 con 0x7f91f40a0300 2024-04-26T00:27:52.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.675+0000 7f91f8cfc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:52.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.675+0000 7f91f8cfc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:27:52.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.675+0000 7f91f8cfc640 10 monclient: dump: 2024-04-26T00:27:52.664 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:52.664 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:52.664 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:52.664 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:52.664 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:52.664 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:52.664 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:52.665 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:52.665 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:52.665 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:52.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.675+0000 7f91fbf62640 10 monclient: shutdown 2024-04-26T00:27:52.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.675+0000 7f91fbf62640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:27:52.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.675+0000 7f91fbf62640 1 -- 172.21.15.37:0/3612968874 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91f40a0300 msgr2=0x7f91f409fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:52.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.675+0000 7f91fbf62640 1 --2- 172.21.15.37:0/3612968874 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91f40a0300 0x7f91f409fc50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f91f00097f0 tx=0x7f91f001c320 comp rx=0 tx=0).stop 2024-04-26T00:27:52.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.675+0000 7f91fbf62640 1 -- 172.21.15.37:0/3612968874 shutdown_connections 2024-04-26T00:27:52.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.675+0000 7f91fbf62640 1 --2- 172.21.15.37:0/3612968874 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91f40a0300 0x7f91f409fc50 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:52.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.675+0000 7f91fbf62640 1 -- 172.21.15.37:0/3612968874 >> 172.21.15.37:0/3612968874 conn(0x7f91f40a1d50 msgr2=0x7f91f40a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:27:52.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.675+0000 7f91fbf62640 1 -- 172.21.15.37:0/3612968874 shutdown_connections 2024-04-26T00:27:52.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.675+0000 7f91fbf62640 1 -- 172.21.15.37:0/3612968874 wait complete. 2024-04-26T00:27:52.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91fbf62640 10 monclient: build_initial_monmap 2024-04-26T00:27:52.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91fbf62640 10 monclient: monmap: 2024-04-26T00:27:52.672 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:27:52.672 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:27:52.672 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:27:52.672 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:27:52.672 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:27:52.672 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:27:52.672 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:27:52.672 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:27:52.672 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:27:52.672 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:52.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91fbf62640 1 Processor -- start 2024-04-26T00:27:52.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91fbf62640 1 -- start start 2024-04-26T00:27:52.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91fbf62640 10 monclient: init 2024-04-26T00:27:52.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91fbf62640 10 monclient: _reopen_session rank -1 2024-04-26T00:27:52.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91fbf62640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:27:52.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91fbf62640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91f40a0300 0x7f91f41cf5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:52.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91fbf62640 10 monclient(hunting): picked mon.noname-b con 0x7f91f40a0300 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:27:52.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91fbf62640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91f40a06e0 0x7f91f4130c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:52.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91fbf62640 10 monclient(hunting): picked mon.noname-c con 0x7f91f40a06e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:27:52.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91fbf62640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91f41352b0 0x7f91f406ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:52.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91fbf62640 10 monclient(hunting): picked mon.noname-a con 0x7f91f41352b0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:27:52.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91fbf62640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:52.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91fbf62640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f91f4140fa0 con 0x7f91f41352b0 2024-04-26T00:27:52.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91fbf62640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:52.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91fbf62640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f91f4140e20 con 0x7f91f40a0300 2024-04-26T00:27:52.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91fbf62640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:52.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91fbf62640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f91f4141120 con 0x7f91f40a06e0 2024-04-26T00:27:52.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91fbf62640 10 monclient(hunting): _renew_subs 2024-04-26T00:27:52.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91fbf62640 10 monclient(hunting): authenticate will time out at 1453.985030s 2024-04-26T00:27:52.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91fa4ff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91f41352b0 0x7f91f406ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:52.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91f9cfe640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91f40a0300 msgr2=0x7f91f41cf5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:27:52.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91fa4ff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91f41352b0 0x7f91f406ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:38610/0 (socket says 172.21.15.37:38610) 2024-04-26T00:27:52.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91fa4ff640 1 -- 172.21.15.37:0/1830902568 learned_addr learned my addr 172.21.15.37:0/1830902568 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:27:52.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91fa4ff640 10 monclient(hunting): get_auth_request con 0x7f91f41352b0 auth_method 0 2024-04-26T00:27:52.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91f9cfe640 1 --2- 172.21.15.37:0/1830902568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91f40a0300 0x7f91f41cf5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:27:52.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91fa4ff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:52.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91fa4ff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:52.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91fa4ff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:52.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91f94fd640 1 --2- 172.21.15.37:0/1830902568 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91f40a06e0 0x7f91f4130c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:52.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91f94fd640 10 monclient(hunting): get_auth_request con 0x7f91f40a06e0 auth_method 0 2024-04-26T00:27:52.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91f94fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:52.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91f94fd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:52.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91f94fd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:52.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91f94fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:52.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91f94fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:52.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91f94fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:52.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91fa4ff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:52.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91fa4ff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:52.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91fa4ff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:52.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91f94fd640 10 monclient(hunting): handle_auth_done global_id 84260 payload 824 2024-04-26T00:27:52.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91f94fd640 1 -- 172.21.15.37:0/1830902568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91f40a0300 msgr2=0x7f91f41cf5c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:27:52.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91f94fd640 1 --2- 172.21.15.37:0/1830902568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91f40a0300 0x7f91f41cf5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:52.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91f94fd640 1 -- 172.21.15.37:0/1830902568 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91f41352b0 msgr2=0x7f91f406ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:52.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91f94fd640 1 --2- 172.21.15.37:0/1830902568 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91f41352b0 0x7f91f406ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:52.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91f94fd640 10 monclient: _finish_hunting 0 2024-04-26T00:27:52.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91f94fd640 1 monclient: found mon.noname-c 2024-04-26T00:27:52.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91f94fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:27:52.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91f94fd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:27:52.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91f94fd640 1 -- 172.21.15.37:0/1830902568 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91f409f550 con 0x7f91f40a06e0 2024-04-26T00:27:52.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91fa4ff640 1 --2- 172.21.15.37:0/1830902568 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91f41352b0 0x7f91f406ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:27:52.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91f94fd640 1 -- 172.21.15.37:0/1830902568 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91f40a06e0 msgr2=0x7f91f4130c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:52.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.679+0000 7f91f94fd640 1 --2- 172.21.15.37:0/1830902568 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91f40a06e0 0x7f91f4130c10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f91e400c440 tx=0x7f91e400c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:52.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91ea7fc640 1 -- 172.21.15.37:0/1830902568 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91e4017070 con 0x7f91f40a06e0 2024-04-26T00:27:52.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:52.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91ea7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:27:52.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91ea7fc640 10 monclient: dump: 2024-04-26T00:27:52.677 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:52.677 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:52.677 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:52.677 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:52.677 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:52.677 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:52.677 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:52.677 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:52.677 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:52.677 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:52.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91ea7fc640 10 monclient: _finish_auth 0 2024-04-26T00:27:52.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91ea7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:27:52.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:27:52.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91ea7fc640 1 -- 172.21.15.37:0/1830902568 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91e40070d0 con 0x7f91f40a06e0 2024-04-26T00:27:52.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91fbf62640 5 monclient: authenticate success, global_id 84260 2024-04-26T00:27:52.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91fbf62640 10 monclient: _renew_subs 2024-04-26T00:27:52.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91fbf62640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:27:52.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91fbf62640 1 -- 172.21.15.37:0/1830902568 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91f4070aa0 con 0x7f91f40a06e0 2024-04-26T00:27:52.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91ea7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:27:52.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91ea7fc640 1 -- 172.21.15.37:0/1830902568 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91e40074c0 con 0x7f91f40a06e0 2024-04-26T00:27:52.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:52.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91ea7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:27:52.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91ea7fc640 10 monclient: dump: 2024-04-26T00:27:52.678 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:52.678 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:52.679 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:52.679 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:52.679 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:52.679 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:52.679 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:52.679 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:52.679 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:52.679 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:52.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91ebfff640 4 set_mon_vals no callback set 2024-04-26T00:27:52.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91fbf62640 10 monclient: _renew_subs 2024-04-26T00:27:52.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91fbf62640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:27:52.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91fbf62640 1 -- 172.21.15.37:0/1830902568 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f91f4131920 con 0x7f91f40a06e0 2024-04-26T00:27:52.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91fbf62640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:52.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91fbf62640 1 --2- 172.21.15.37:0/1830902568 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91f4131150 0x7f91f406fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:52.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91fbf62640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:52.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91fbf62640 1 -- 172.21.15.37:0/1830902568 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f91f411c550 con 0x7f91f4131150 2024-04-26T00:27:52.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91f9cfe640 1 --2- 172.21.15.37:0/1830902568 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91f4131150 0x7f91f406fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:52.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91f9cfe640 10 monclient: get_auth_request con 0x7f91f4131150 auth_method 0 2024-04-26T00:27:52.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91f9cfe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:52.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91f9cfe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:52.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91f9cfe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:52.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91ea7fc640 1 -- 172.21.15.37:0/1830902568 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f91e401a070 con 0x7f91f40a06e0 2024-04-26T00:27:52.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91f9cfe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:52.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91f9cfe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:52.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91f9cfe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:52.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91ea7fc640 1 --2- 172.21.15.37:0/1830902568 >> v2:172.21.15.45:6812/2379918623 conn(0x7f91d8035e30 0x7f91d80382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:52.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91f9cfe640 10 monclient(hunting): handle_auth_done global_id 84263 payload 824 2024-04-26T00:27:52.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91ea7fc640 1 -- 172.21.15.37:0/1830902568 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(299..299 src has 177..299) ==== 3927+0+0 (secure 0 0 0) 0x7f91e4047f30 con 0x7f91f40a06e0 2024-04-26T00:27:52.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91f9cfe640 1 -- 172.21.15.37:0/1830902568 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f91f404f510 con 0x7f91f4131150 2024-04-26T00:27:52.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91fa4ff640 1 --2- 172.21.15.37:0/1830902568 >> v2:172.21.15.45:6812/2379918623 conn(0x7f91d8035e30 0x7f91d80382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:52.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91fa4ff640 10 monclient: get_auth_request con 0x7f91d8035e30 auth_method 0 2024-04-26T00:27:52.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91f9cfe640 1 -- 172.21.15.37:0/1830902568 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91f4131150 msgr2=0x7f91f406fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:52.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91f9cfe640 1 --2- 172.21.15.37:0/1830902568 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91f4131150 0x7f91f406fb00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f91f406d330 tx=0x7f91f001ab60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:52.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91ea7fc640 1 -- 172.21.15.37:0/1830902568 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91f00bb020 con 0x7f91f4131150 2024-04-26T00:27:52.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:52.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91ea7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:27:52.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.683+0000 7f91ea7fc640 10 monclient: dump: 2024-04-26T00:27:52.681 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:52.682 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:52.682 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:52.682 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:52.682 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:52.682 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:52.682 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:52.682 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:52.682 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:52.682 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:52.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.687+0000 7f91ea7fc640 1 -- 172.21.15.37:0/1830902568 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f91f00b1040 con 0x7f91f4131150 2024-04-26T00:27:52.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.687+0000 7f91ea7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:27:52.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.687+0000 7f91ea7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:27:52.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.687+0000 7f91ea7fc640 1 -- 172.21.15.37:0/1830902568 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91f4131150 msgr2=0x7f91f406fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:52.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.687+0000 7f91ea7fc640 1 --2- 172.21.15.37:0/1830902568 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91f4131150 0x7f91f406fb00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f91f406d330 tx=0x7f91f001ab60 comp rx=0 tx=0).stop 2024-04-26T00:27:52.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.687+0000 7f91ea7fc640 1 -- 172.21.15.37:0/1830902568 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91f4131150 msgr2=0x7f91f406fb00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:27:52.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.687+0000 7f91ea7fc640 1 --2- 172.21.15.37:0/1830902568 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91f4131150 0x7f91f406fb00 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f91f406d330 tx=0x7f91f001ab60 comp rx=0 tx=0).stop 2024-04-26T00:27:52.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.687+0000 7f91fa4ff640 1 --2- 172.21.15.37:0/1830902568 >> v2:172.21.15.45:6812/2379918623 conn(0x7f91d8035e30 0x7f91d80382f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f91ec009fd0 tx=0x7f91ec00d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:52.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.691+0000 7f91fbf62640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:27:52.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.691+0000 7f91fbf62640 1 --2- 172.21.15.37:0/1830902568 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91f4070040 0x7f91f402c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:27:52.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.691+0000 7f91fbf62640 10 monclient(hunting): start opening mon connection 2024-04-26T00:27:52.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.691+0000 7f91fbf62640 1 -- 172.21.15.37:0/1830902568 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f91f413b6d0 con 0x7f91f4070040 2024-04-26T00:27:52.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.691+0000 7f91f9cfe640 1 --2- 172.21.15.37:0/1830902568 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91f4070040 0x7f91f402c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:27:52.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.691+0000 7f91f9cfe640 10 monclient: get_auth_request con 0x7f91f4070040 auth_method 0 2024-04-26T00:27:52.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.691+0000 7f91f9cfe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:27:52.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.691+0000 7f91f9cfe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:27:52.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.691+0000 7f91f9cfe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:27:52.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.691+0000 7f91f9cfe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:27:52.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.691+0000 7f91f9cfe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:27:52.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.691+0000 7f91f9cfe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:27:52.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.691+0000 7f91f9cfe640 10 monclient(hunting): handle_auth_done global_id 84266 payload 824 2024-04-26T00:27:52.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.691+0000 7f91f9cfe640 1 -- 172.21.15.37:0/1830902568 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f91f409db30 con 0x7f91f4070040 2024-04-26T00:27:52.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.695+0000 7f91f9cfe640 1 -- 172.21.15.37:0/1830902568 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91f4070040 msgr2=0x7f91f402c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:27:52.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.695+0000 7f91f9cfe640 1 --2- 172.21.15.37:0/1830902568 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91f4070040 0x7f91f402c000 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f91f001c150 tx=0x7f91f00190c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:27:52.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.695+0000 7f91ea7fc640 1 -- 172.21.15.37:0/1830902568 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91f00bb040 con 0x7f91f4070040 2024-04-26T00:27:52.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.695+0000 7f91ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:27:52.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.695+0000 7f91ea7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:27:52.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.695+0000 7f91ea7fc640 10 monclient: dump: 2024-04-26T00:27:52.686 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:27:52.686 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:27:52.686 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:52.686 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:27:52.686 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:27:52.686 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:27:52.687 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:27:52.687 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:27:52.687 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:27:52.687 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:27:52.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.695+0000 7f91ea7fc640 1 -- 172.21.15.37:0/1830902568 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4875 (secure 0 0 0) 0x7f91f0004230 con 0x7f91f4070040 2024-04-26T00:27:52.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.695+0000 7f91ea7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:27:52.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.695+0000 7f91ea7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:27:52.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.695+0000 7f91ea7fc640 1 -- 172.21.15.37:0/1830902568 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91f4070040 msgr2=0x7f91f402c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:52.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.695+0000 7f91ea7fc640 1 --2- 172.21.15.37:0/1830902568 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91f4070040 0x7f91f402c000 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f91f001c150 tx=0x7f91f00190c0 comp rx=0 tx=0).stop 2024-04-26T00:27:52.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.695+0000 7f91ea7fc640 1 -- 172.21.15.37:0/1830902568 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91f4070040 msgr2=0x7f91f402c000 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:27:52.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.695+0000 7f91ea7fc640 1 --2- 172.21.15.37:0/1830902568 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91f4070040 0x7f91f402c000 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f91f001c150 tx=0x7f91f00190c0 comp rx=0 tx=0).stop 2024-04-26T00:27:52.687 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:27:52.688 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:27:52.688 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:27:52.688 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:27:52.688 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 62, 2024-04-26T00:27:52.688 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 77630, 2024-04-26T00:27:52.688 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:27:52.688 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:27:52.688 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:27:52.688 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:52.688 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:27:52.688 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:27:52.688 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:27:52.688 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:27:52.689 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:27:52.689 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:27:52.689 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:27:52.689 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:27:52.689 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:27:52.689 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:27:52.689 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:27:52.689 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:27:52.689 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:27:52.689 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:27:52.689 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:27:52.689 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:52.689 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:27:52.690 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:27:52.690 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:27:52.690 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:27:52.690 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:27:52.690 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:27:52.690 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:27:52.690 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:27:52.690 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:27:52.690 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:27:52.690 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:27:52.690 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:27:52.691 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:27:52.691 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:52.691 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:52.691 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:27:52.691 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:27:52.691 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:27:52.691 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:27:52.691 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:27:52.691 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:27:52.691 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:27:52.691 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:27:52.691 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:27:52.691 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:27:52.692 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:27:52.692 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:27:52.692 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:27:52.692 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:27:52.692 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:27:52.692 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:27:52.692 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:27:52.692 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:27:52.692 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:27:52.692 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:27:52.693 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:27:52.693 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:27:52.693 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:27:52.693 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:27:52.693 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:27:52.693 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:27:52.693 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:27:52.693 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:27:52.693 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:52.693 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:27:52.693 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:52.694 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:27:52.694 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:52.694 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:27:52.694 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:27:52.694 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:27:52.694 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:27:52.694 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:52.694 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:27:52.694 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:27:52.694 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:52.694 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:52.694 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:52.694 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:27:52.695 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:27:52.695 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:52.695 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:52.695 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:52.695 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:52.695 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:27:52.695 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:27:52.695 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:27:52.695 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:27:52.695 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:27:52.695 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:52.695 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:52.695 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:27:52.695 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:27:52.695 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:27:52.696 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:27:52.696 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:52.696 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:27:52.696 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:27:52.696 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:52.696 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:52.696 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:52.696 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:27:52.696 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:27:52.696 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:52.696 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:52.696 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:52.696 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:52.696 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:27:52.697 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:27:52.697 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:27:52.697 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:27:52.697 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:27:52.697 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:52.697 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:52.697 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:27:52.697 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:27:52.697 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:27:52.697 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:27:52.697 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:52.697 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:27:52.697 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:27:52.697 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:52.698 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:52.698 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:52.698 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:27:52.698 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:27:52.698 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:27:52.698 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:52.698 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:52.698 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:52.698 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:27:52.698 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:27:52.698 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:27:52.698 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:27:52.698 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:27:52.698 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:52.699 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:52.699 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:52.699 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:27:52.699 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:27:52.699 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:52.699 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:27:52.699 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:27:52.699 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:27:52.699 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:52.699 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:52.699 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:27:52.699 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:52.699 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:27:52.699 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:27:52.699 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:27:52.700 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:52.700 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:52.700 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:27:52.700 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:52.700 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:27:52.700 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:27:52.700 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:27:52.700 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:52.700 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:27:52.700 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:27:52.700 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:27:52.700 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:27:52.700 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:27:52.700 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:27:52.701 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:27:52.701 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:27:52.701 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:27:52.701 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:27:52.701 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:27:52.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.695+0000 7f91fbf62640 1 -- 172.21.15.37:0/1830902568 >> v2:172.21.15.45:6812/2379918623 conn(0x7f91d8035e30 msgr2=0x7f91d80382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:52.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.695+0000 7f91fbf62640 1 --2- 172.21.15.37:0/1830902568 >> v2:172.21.15.45:6812/2379918623 conn(0x7f91d8035e30 0x7f91d80382f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f91ec009fd0 tx=0x7f91ec00d040 comp rx=0 tx=0).stop 2024-04-26T00:27:52.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.695+0000 7f91fbf62640 10 monclient: shutdown 2024-04-26T00:27:52.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.695+0000 7f91fbf62640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:27:52.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.695+0000 7f91fbf62640 1 -- 172.21.15.37:0/1830902568 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91f40a06e0 msgr2=0x7f91f4130c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:27:52.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.695+0000 7f91fbf62640 1 --2- 172.21.15.37:0/1830902568 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91f40a06e0 0x7f91f4130c10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f91e400c440 tx=0x7f91e400c900 comp rx=0 tx=0).stop 2024-04-26T00:27:52.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.695+0000 7f91fa4ff640 1 -- 172.21.15.37:0/1830902568 reap_dead start 2024-04-26T00:27:52.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.695+0000 7f91fbf62640 1 -- 172.21.15.37:0/1830902568 shutdown_connections 2024-04-26T00:27:52.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.695+0000 7f91fbf62640 1 --2- 172.21.15.37:0/1830902568 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91f40a06e0 0x7f91f4130c10 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:27:52.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.695+0000 7f91fbf62640 1 -- 172.21.15.37:0/1830902568 >> 172.21.15.37:0/1830902568 conn(0x7f91f40a1d50 msgr2=0x7f91f4133c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:27:52.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.695+0000 7f91fbf62640 1 -- 172.21.15.37:0/1830902568 shutdown_connections 2024-04-26T00:27:52.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:27:52.695+0000 7f91fbf62640 1 -- 172.21.15.37:0/1830902568 wait complete. 2024-04-26T00:27:52.703 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:27:52.703 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-04-26T00:27:53.056 INFO:tasks.rados.rados.0.smithi045.stdout:1260: done (4 left) 2024-04-26T00:27:53.056 INFO:tasks.rados.rados.0.smithi045.stdout:1261: read oid 2 snap 126 2024-04-26T00:27:53.056 INFO:tasks.rados.rados.0.smithi045.stdout:1261: expect (ObjNum 404 snap 124 seq_num 404) 2024-04-26T00:27:53.057 INFO:tasks.rados.rados.0.smithi045.stdout:1262: write (excl) oid 33 current snap is 147 2024-04-26T00:27:53.057 INFO:tasks.rados.rados.0.smithi045.stdout:1262: seq_num 471 ranges {716097=539812,1945607=795166,3535053=381044} 2024-04-26T00:27:53.071 INFO:tasks.rados.rados.0.smithi045.stdout:1262: writing smithi04526579-33 from 716097 to 1255909 tid 1 2024-04-26T00:27:53.091 INFO:tasks.rados.rados.0.smithi045.stdout:1262: writing smithi04526579-33 from 1945607 to 2740773 tid 2 2024-04-26T00:27:53.106 INFO:tasks.rados.rados.0.smithi045.stdout:1262: writing smithi04526579-33 from 3535053 to 3916097 tid 3 2024-04-26T00:27:53.106 INFO:tasks.rados.rados.0.smithi045.stdout:1263: rollback oid 4 current snap is 147 2024-04-26T00:27:53.107 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 4 to 126 2024-04-26T00:27:53.107 INFO:tasks.rados.rados.0.smithi045.stdout:1256: finishing copy_from racing read to smithi04526579-46 2024-04-26T00:27:53.107 INFO:tasks.rados.rados.0.smithi045.stdout:1257: finishing write tid 1 to smithi04526579-50 2024-04-26T00:27:53.107 INFO:tasks.rados.rados.0.smithi045.stdout:1257: finishing write tid 2 to smithi04526579-50 2024-04-26T00:27:53.107 INFO:tasks.rados.rados.0.smithi045.stdout:1257: finishing write tid 3 to smithi04526579-50 2024-04-26T00:27:53.107 INFO:tasks.rados.rados.0.smithi045.stdout:1256: finishing copy_from to smithi04526579-46 2024-04-26T00:27:53.107 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 609 (ObjNum 433 snap 136 seq_num 433) dirty exists 2024-04-26T00:27:53.107 INFO:tasks.rados.rados.0.smithi045.stdout:1257: finishing write tid 4 to smithi04526579-50 2024-04-26T00:27:53.107 INFO:tasks.rados.rados.0.smithi045.stdout:1257: finishing write tid 5 to smithi04526579-50 2024-04-26T00:27:53.107 INFO:tasks.rados.rados.0.smithi045.stdout:1257: oid 50 updating version 0 to 843 2024-04-26T00:27:53.107 INFO:tasks.rados.rados.0.smithi045.stdout:1257: oid 50 version 843 is already newer than 842 2024-04-26T00:27:53.107 INFO:tasks.rados.rados.0.smithi045.stdout:1257: oid 50 updating version 843 to 845 2024-04-26T00:27:53.108 INFO:tasks.rados.rados.0.smithi045.stdout:1257: oid 50 version 845 is already newer than 844 2024-04-26T00:27:53.108 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 845 (ObjNum 470 snap 147 seq_num 470) dirty exists 2024-04-26T00:27:53.108 INFO:tasks.rados.rados.0.smithi045.stdout:1257: left oid 50 (ObjNum 470 snap 147 seq_num 470) 2024-04-26T00:27:53.108 INFO:tasks.rados.rados.0.smithi045.stdout:1258: expect (ObjNum 462 snap 145 seq_num 462) 2024-04-26T00:27:53.319 INFO:tasks.rados.rados.0.smithi045.stdout:1259: expect (ObjNum 466 snap 145 seq_num 466) 2024-04-26T00:27:53.540 INFO:tasks.rados.rados.0.smithi045.stdout:1261: expect (ObjNum 404 snap 124 seq_num 404) 2024-04-26T00:27:53.742 INFO:tasks.rados.rados.0.smithi045.stdout:1262: finishing write tid 1 to smithi04526579-33 2024-04-26T00:27:53.742 INFO:tasks.rados.rados.0.smithi045.stdout:1262: finishing write tid 2 to smithi04526579-33 2024-04-26T00:27:53.742 INFO:tasks.rados.rados.0.smithi045.stdout:1262: finishing write tid 3 to smithi04526579-33 2024-04-26T00:27:53.742 INFO:tasks.rados.rados.0.smithi045.stdout:1262: finishing write tid 4 to smithi04526579-33 2024-04-26T00:27:53.743 INFO:tasks.rados.rados.0.smithi045.stdout:1262: finishing write tid 5 to smithi04526579-33 2024-04-26T00:27:53.743 INFO:tasks.rados.rados.0.smithi045.stdout:1262: oid 33 updating version 0 to 985 2024-04-26T00:27:53.743 INFO:tasks.rados.rados.0.smithi045.stdout:1262: oid 33 version 985 is already newer than 978 2024-04-26T00:27:53.743 INFO:tasks.rados.rados.0.smithi045.stdout:1262: oid 33 version 985 is already newer than 984 2024-04-26T00:27:53.743 INFO:tasks.rados.rados.0.smithi045.stdout:1262: oid 33 version 985 is already newer than 979 2024-04-26T00:27:53.743 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 985 (ObjNum 471 snap 147 seq_num 471) dirty exists 2024-04-26T00:27:53.743 INFO:tasks.rados.rados.0.smithi045.stdout:1262: left oid 33 (ObjNum 471 snap 147 seq_num 471) 2024-04-26T00:27:53.743 INFO:tasks.rados.rados.0.smithi045.stdout:1263: finishing rollback tid 0 to smithi04526579-4 2024-04-26T00:27:53.743 INFO:tasks.rados.rados.0.smithi045.stdout:1263: finishing rollback tid 1 to smithi04526579-4 2024-04-26T00:27:53.743 INFO:tasks.rados.rados.0.smithi045.stdout:1263: finishing rollback tid 2 to smithi04526579-4 2024-04-26T00:27:53.743 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 684 (ObjNum 406 snap 125 seq_num 406) dirty exists 2024-04-26T00:27:53.743 INFO:tasks.rados.rados.0.smithi045.stdout:1256: done (6 left) 2024-04-26T00:27:53.743 INFO:tasks.rados.rados.0.smithi045.stdout:1257: done (5 left) 2024-04-26T00:27:53.743 INFO:tasks.rados.rados.0.smithi045.stdout:1258: done (4 left) 2024-04-26T00:27:53.744 INFO:tasks.rados.rados.0.smithi045.stdout:1259: done (3 left) 2024-04-26T00:27:53.744 INFO:tasks.rados.rados.0.smithi045.stdout:1261: done (2 left) 2024-04-26T00:27:53.744 INFO:tasks.rados.rados.0.smithi045.stdout:1262: done (1 left) 2024-04-26T00:27:53.744 INFO:tasks.rados.rados.0.smithi045.stdout:1263: done (0 left) 2024-04-26T00:27:53.744 INFO:tasks.rados.rados.0.smithi045.stdout:1264: copy_from oid 7 from oid 44 current snap is 147 2024-04-26T00:27:53.744 INFO:tasks.rados.rados.0.smithi045.stdout:1265: write (excl) oid 9 current snap is 147 2024-04-26T00:27:53.744 INFO:tasks.rados.rados.0.smithi045.stdout:1265: seq_num 472 ranges {732904=731414,2120442=538824,3278829=504743} 2024-04-26T00:27:53.766 INFO:tasks.rados.rados.0.smithi045.stdout:1265: writing smithi04526579-9 from 732904 to 1464318 tid 1 2024-04-26T00:27:53.785 INFO:tasks.rados.rados.0.smithi045.stdout:1265: writing smithi04526579-9 from 2120442 to 2659266 tid 2 2024-04-26T00:27:53.802 INFO:tasks.rados.rados.0.smithi045.stdout:1265: writing smithi04526579-9 from 3278829 to 3783572 tid 3 2024-04-26T00:27:53.802 INFO:tasks.rados.rados.0.smithi045.stdout:1266: snap_remove snap 134 2024-04-26T00:27:54.064 INFO:tasks.rados.rados.0.smithi045.stdout:1266: done (2 left) 2024-04-26T00:27:54.065 INFO:tasks.rados.rados.0.smithi045.stdout:1267: read oid 33 snap -1 2024-04-26T00:27:54.065 INFO:tasks.rados.rados.0.smithi045.stdout:1267: expect (ObjNum 471 snap 147 seq_num 471) 2024-04-26T00:27:54.065 INFO:tasks.rados.rados.0.smithi045.stdout:1268: copy_from oid 30 from oid 3 current snap is 147 2024-04-26T00:27:54.065 INFO:tasks.rados.rados.0.smithi045.stdout:1264: finishing copy_from to smithi04526579-7 2024-04-26T00:27:54.065 INFO:tasks.rados.rados.0.smithi045.stdout:1264: got expected ENOENT (src dne) 2024-04-26T00:27:54.065 INFO:tasks.rados.rados.0.smithi045.stdout:1264: finishing copy_from racing read to smithi04526579-7 2024-04-26T00:27:54.065 INFO:tasks.rados.rados.0.smithi045.stdout:1265: finishing write tid 1 to smithi04526579-9 2024-04-26T00:27:54.065 INFO:tasks.rados.rados.0.smithi045.stdout:1265: finishing write tid 2 to smithi04526579-9 2024-04-26T00:27:54.065 INFO:tasks.rados.rados.0.smithi045.stdout:1265: finishing write tid 3 to smithi04526579-9 2024-04-26T00:27:54.065 INFO:tasks.rados.rados.0.smithi045.stdout:1265: finishing write tid 4 to smithi04526579-9 2024-04-26T00:27:54.065 INFO:tasks.rados.rados.0.smithi045.stdout:1265: finishing write tid 5 to smithi04526579-9 2024-04-26T00:27:54.065 INFO:tasks.rados.rados.0.smithi045.stdout:1265: oid 9 updating version 0 to 688 2024-04-26T00:27:54.065 INFO:tasks.rados.rados.0.smithi045.stdout:1265: oid 9 version 688 is already newer than 687 2024-04-26T00:27:54.065 INFO:tasks.rados.rados.0.smithi045.stdout:1265: oid 9 updating version 688 to 689 2024-04-26T00:27:54.066 INFO:tasks.rados.rados.0.smithi045.stdout:1265: oid 9 version 689 is already newer than 686 2024-04-26T00:27:54.066 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 689 (ObjNum 472 snap 147 seq_num 472) dirty exists 2024-04-26T00:27:54.066 INFO:tasks.rados.rados.0.smithi045.stdout:1265: left oid 9 (ObjNum 472 snap 147 seq_num 472) 2024-04-26T00:27:54.066 INFO:tasks.rados.rados.0.smithi045.stdout:1264: done (3 left) 2024-04-26T00:27:54.066 INFO:tasks.rados.rados.0.smithi045.stdout:1265: done (2 left) 2024-04-26T00:27:54.066 INFO:tasks.rados.rados.0.smithi045.stdout:1269: write oid 24 current snap is 147 2024-04-26T00:27:54.066 INFO:tasks.rados.rados.0.smithi045.stdout:1269: seq_num 473 ranges {749711=406663,1778924=682482,3189899=716015} 2024-04-26T00:27:54.079 INFO:tasks.rados.rados.0.smithi045.stdout:1269: writing smithi04526579-24 from 749711 to 1156374 tid 1 2024-04-26T00:27:54.097 INFO:tasks.rados.rados.0.smithi045.stdout:1269: writing smithi04526579-24 from 1778924 to 2461406 tid 2 2024-04-26T00:27:54.115 INFO:tasks.rados.rados.0.smithi045.stdout:1269: writing smithi04526579-24 from 3189899 to 3905914 tid 3 2024-04-26T00:27:54.115 INFO:tasks.rados.rados.0.smithi045.stdout:1270: read oid 31 snap 118 2024-04-26T00:27:54.115 INFO:tasks.rados.rados.0.smithi045.stdout:1270: expect (ObjNum 369 snap 112 seq_num 369) 2024-04-26T00:27:54.115 INFO:tasks.rados.rados.0.smithi045.stdout:1271: write oid 9 current snap is 147 2024-04-26T00:27:54.116 INFO:tasks.rados.rados.0.smithi045.stdout:1271: seq_num 474 ranges {766518=481912,1837406=542493,2933675=527287,3876584=89934} 2024-04-26T00:27:54.129 INFO:tasks.rados.rados.0.smithi045.stdout:1271: writing smithi04526579-9 from 766518 to 1248430 tid 1 2024-04-26T00:27:54.139 INFO:tasks.rados.rados.0.smithi045.stdout:1271: writing smithi04526579-9 from 1837406 to 2379899 tid 2 2024-04-26T00:27:54.148 INFO:tasks.rados.rados.0.smithi045.stdout:1271: writing smithi04526579-9 from 2933675 to 3460962 tid 3 2024-04-26T00:27:54.152 INFO:tasks.rados.rados.0.smithi045.stdout:1271: writing smithi04526579-9 from 3876584 to 3966518 tid 4 2024-04-26T00:27:54.152 INFO:tasks.rados.rados.0.smithi045.stdout:1272: snap_create 2024-04-26T00:27:54.152 INFO:tasks.rados.rados.0.smithi045.stdout:1268: finishing copy_from to smithi04526579-30 2024-04-26T00:27:54.152 INFO:tasks.rados.rados.0.smithi045.stdout:1268: got expected ENOENT (src dne) 2024-04-26T00:27:54.152 INFO:tasks.rados.rados.0.smithi045.stdout:1268: finishing copy_from racing read to smithi04526579-30 2024-04-26T00:27:54.152 INFO:tasks.rados.rados.0.smithi045.stdout:1267: expect (ObjNum 471 snap 147 seq_num 471) 2024-04-26T00:27:54.487 INFO:tasks.rados.rados.0.smithi045.stdout:1269: finishing write tid 1 to smithi04526579-24 2024-04-26T00:27:54.487 INFO:tasks.rados.rados.0.smithi045.stdout:1269: finishing write tid 2 to smithi04526579-24 2024-04-26T00:27:54.487 INFO:tasks.rados.rados.0.smithi045.stdout:1269: finishing write tid 3 to smithi04526579-24 2024-04-26T00:27:54.487 INFO:tasks.rados.rados.0.smithi045.stdout:1269: finishing write tid 4 to smithi04526579-24 2024-04-26T00:27:54.487 INFO:tasks.rados.rados.0.smithi045.stdout:1269: finishing write tid 5 to smithi04526579-24 2024-04-26T00:27:54.488 INFO:tasks.rados.rados.0.smithi045.stdout:1269: oid 24 updating version 0 to 999 2024-04-26T00:27:54.488 INFO:tasks.rados.rados.0.smithi045.stdout:1269: oid 24 version 999 is already newer than 988 2024-04-26T00:27:54.488 INFO:tasks.rados.rados.0.smithi045.stdout:1269: oid 24 version 999 is already newer than 993 2024-04-26T00:27:54.488 INFO:tasks.rados.rados.0.smithi045.stdout:1269: oid 24 version 999 is already newer than 998 2024-04-26T00:27:54.488 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 999 (ObjNum 473 snap 147 seq_num 473) dirty exists 2024-04-26T00:27:54.488 INFO:tasks.rados.rados.0.smithi045.stdout:1269: left oid 24 (ObjNum 473 snap 147 seq_num 473) 2024-04-26T00:27:54.488 INFO:tasks.rados.rados.0.smithi045.stdout:1270: expect (ObjNum 369 snap 112 seq_num 369) 2024-04-26T00:27:54.579 INFO:tasks.rados.rados.0.smithi045.stdout:1271: finishing write tid 1 to smithi04526579-9 2024-04-26T00:27:54.579 INFO:tasks.rados.rados.0.smithi045.stdout:1271: finishing write tid 2 to smithi04526579-9 2024-04-26T00:27:54.579 INFO:tasks.rados.rados.0.smithi045.stdout:1271: finishing write tid 3 to smithi04526579-9 2024-04-26T00:27:54.579 INFO:tasks.rados.rados.0.smithi045.stdout:1271: finishing write tid 4 to smithi04526579-9 2024-04-26T00:27:54.579 INFO:tasks.rados.rados.0.smithi045.stdout:1271: finishing write tid 5 to smithi04526579-9 2024-04-26T00:27:54.579 INFO:tasks.rados.rados.0.smithi045.stdout:1271: finishing write tid 6 to smithi04526579-9 2024-04-26T00:27:54.579 INFO:tasks.rados.rados.0.smithi045.stdout:1271: oid 9 updating version 0 to 700 2024-04-26T00:27:54.579 INFO:tasks.rados.rados.0.smithi045.stdout:1271: oid 9 version 700 is already newer than 696 2024-04-26T00:27:54.579 INFO:tasks.rados.rados.0.smithi045.stdout:1271: oid 9 version 700 is already newer than 699 2024-04-26T00:27:54.579 INFO:tasks.rados.rados.0.smithi045.stdout:1271: oid 9 version 700 is already newer than 698 2024-04-26T00:27:54.580 INFO:tasks.rados.rados.0.smithi045.stdout:1271: oid 9 version 700 is already newer than 697 2024-04-26T00:27:54.580 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 700 (ObjNum 474 snap 147 seq_num 474) dirty exists 2024-04-26T00:27:54.580 INFO:tasks.rados.rados.0.smithi045.stdout:1271: left oid 9 (ObjNum 474 snap 147 seq_num 474) 2024-04-26T00:27:55.070 INFO:tasks.rados.rados.0.smithi045.stdout:1267: done (5 left) 2024-04-26T00:27:55.071 INFO:tasks.rados.rados.0.smithi045.stdout:1268: done (4 left) 2024-04-26T00:27:55.071 INFO:tasks.rados.rados.0.smithi045.stdout:1269: done (3 left) 2024-04-26T00:27:55.071 INFO:tasks.rados.rados.0.smithi045.stdout:1270: done (2 left) 2024-04-26T00:27:55.071 INFO:tasks.rados.rados.0.smithi045.stdout:1271: done (1 left) 2024-04-26T00:27:55.071 INFO:tasks.rados.rados.0.smithi045.stdout:1272: done (0 left) 2024-04-26T00:27:55.071 INFO:tasks.rados.rados.0.smithi045.stdout:1273: write oid 45 current snap is 148 2024-04-26T00:27:55.071 INFO:tasks.rados.rados.0.smithi045.stdout:1273: seq_num 475 ranges {783325=557161,1779535=686151,3244745=454912} 2024-04-26T00:27:55.080 INFO:tasks.rados.rados.0.smithi045.stdout:1273: writing smithi04526579-45 from 783325 to 1340486 tid 1 2024-04-26T00:27:55.088 INFO:tasks.rados.rados.0.smithi045.stdout:1273: writing smithi04526579-45 from 1779535 to 2465686 tid 2 2024-04-26T00:27:55.097 INFO:tasks.rados.rados.0.smithi045.stdout:1273: writing smithi04526579-45 from 3244745 to 3699657 tid 3 2024-04-26T00:27:55.097 INFO:tasks.rados.rados.0.smithi045.stdout:1274: delete oid 8 current snap is 148 2024-04-26T00:27:55.097 INFO:tasks.rados.rados.0.smithi045.stdout:1273: finishing write tid 1 to smithi04526579-45 2024-04-26T00:27:55.097 INFO:tasks.rados.rados.0.smithi045.stdout:1273: finishing write tid 2 to smithi04526579-45 2024-04-26T00:27:55.103 INFO:tasks.rados.rados.0.smithi045.stdout:1273: finishing write tid 3 to smithi04526579-45 2024-04-26T00:27:55.103 INFO:tasks.rados.rados.0.smithi045.stdout:1273: finishing write tid 4 to smithi04526579-45 2024-04-26T00:27:55.103 INFO:tasks.rados.rados.0.smithi045.stdout:1273: finishing write tid 5 to smithi04526579-45 2024-04-26T00:27:55.104 INFO:tasks.rados.rados.0.smithi045.stdout:1273: oid 45 updating version 0 to 836 2024-04-26T00:27:55.104 INFO:tasks.rados.rados.0.smithi045.stdout:1273: oid 45 version 836 is already newer than 835 2024-04-26T00:27:55.104 INFO:tasks.rados.rados.0.smithi045.stdout:1273: oid 45 updating version 836 to 837 2024-04-26T00:27:55.104 INFO:tasks.rados.rados.0.smithi045.stdout:1273: oid 45 version 837 is already newer than 834 2024-04-26T00:27:55.104 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 837 (ObjNum 475 snap 148 seq_num 475) dirty exists 2024-04-26T00:27:55.104 INFO:tasks.rados.rados.0.smithi045.stdout:1273: left oid 45 (ObjNum 475 snap 148 seq_num 475) 2024-04-26T00:27:55.108 INFO:tasks.rados.rados.0.smithi045.stdout:1273: done (1 left) 2024-04-26T00:27:55.108 INFO:tasks.rados.rados.0.smithi045.stdout:1274: done (0 left) 2024-04-26T00:27:55.108 INFO:tasks.rados.rados.0.smithi045.stdout:1275: read oid 15 snap -1 2024-04-26T00:27:55.108 INFO:tasks.rados.rados.0.smithi045.stdout:1275: expect (ObjNum 376 snap 115 seq_num 376) 2024-04-26T00:27:55.108 INFO:tasks.rados.rados.0.smithi045.stdout:1276: write oid 12 current snap is 148 2024-04-26T00:27:55.108 INFO:tasks.rados.rados.0.smithi045.stdout:1276: seq_num 477 ranges {416939=707659,1896499=120440} 2024-04-26T00:27:55.116 INFO:tasks.rados.rados.0.smithi045.stdout:1276: writing smithi04526579-12 from 416939 to 1124598 tid 1 2024-04-26T00:27:55.124 INFO:tasks.rados.rados.0.smithi045.stdout:1276: writing smithi04526579-12 from 1896499 to 2016939 tid 2 2024-04-26T00:27:55.124 INFO:tasks.rados.rados.0.smithi045.stdout:1277: snap_create 2024-04-26T00:27:55.124 INFO:tasks.rados.rados.0.smithi045.stdout:1275: expect (ObjNum 376 snap 115 seq_num 376) 2024-04-26T00:27:55.257 INFO:tasks.rados.rados.0.smithi045.stdout:1276: finishing write tid 1 to smithi04526579-12 2024-04-26T00:27:55.257 INFO:tasks.rados.rados.0.smithi045.stdout:1276: finishing write tid 2 to smithi04526579-12 2024-04-26T00:27:55.257 INFO:tasks.rados.rados.0.smithi045.stdout:1276: finishing write tid 3 to smithi04526579-12 2024-04-26T00:27:55.257 INFO:tasks.rados.rados.0.smithi045.stdout:1276: finishing write tid 4 to smithi04526579-12 2024-04-26T00:27:55.257 INFO:tasks.rados.rados.0.smithi045.stdout:1276: oid 12 updating version 0 to 603 2024-04-26T00:27:55.257 INFO:tasks.rados.rados.0.smithi045.stdout:1276: oid 12 updating version 603 to 604 2024-04-26T00:27:55.257 INFO:tasks.rados.rados.0.smithi045.stdout:1276: oid 12 updating version 604 to 605 2024-04-26T00:27:55.257 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 605 (ObjNum 477 snap 148 seq_num 477) dirty exists 2024-04-26T00:27:55.258 INFO:tasks.rados.rados.0.smithi045.stdout:1276: left oid 12 (ObjNum 477 snap 148 seq_num 477) 2024-04-26T00:27:56.082 INFO:tasks.rados.rados.0.smithi045.stdout:1275: done (2 left) 2024-04-26T00:27:56.083 INFO:tasks.rados.rados.0.smithi045.stdout:1276: done (1 left) 2024-04-26T00:27:56.083 INFO:tasks.rados.rados.0.smithi045.stdout:1277: done (0 left) 2024-04-26T00:27:56.083 INFO:tasks.rados.rados.0.smithi045.stdout:1278: read oid 50 snap -1 2024-04-26T00:27:56.083 INFO:tasks.rados.rados.0.smithi045.stdout:1278: expect (ObjNum 470 snap 147 seq_num 470) 2024-04-26T00:27:56.083 INFO:tasks.rados.rados.0.smithi045.stdout:1279: read oid 43 snap -1 2024-04-26T00:27:56.083 INFO:tasks.rados.rados.0.smithi045.stdout:1279: expect (ObjNum 329 snap 96 seq_num 329) 2024-04-26T00:27:56.083 INFO:tasks.rados.rados.0.smithi045.stdout:1280: read oid 16 snap -1 2024-04-26T00:27:56.083 INFO:tasks.rados.rados.0.smithi045.stdout:1280: expect (ObjNum 395 snap 120 seq_num 395) 2024-04-26T00:27:56.083 INFO:tasks.rados.rados.0.smithi045.stdout:1281: write (excl) oid 4 current snap is 149 2024-04-26T00:27:56.083 INFO:tasks.rados.rados.0.smithi045.stdout:1281: seq_num 478 ranges {433746=782908,1954981=78765} 2024-04-26T00:27:56.095 INFO:tasks.rados.rados.0.smithi045.stdout:1281: writing smithi04526579-4 from 433746 to 1216654 tid 1 2024-04-26T00:27:56.104 INFO:tasks.rados.rados.0.smithi045.stdout:1281: writing smithi04526579-4 from 1954981 to 2033746 tid 2 2024-04-26T00:27:56.104 INFO:tasks.rados.rados.0.smithi045.stdout:1282: snap_create 2024-04-26T00:27:56.104 INFO:tasks.rados.rados.0.smithi045.stdout:1280: expect (ObjNum 395 snap 120 seq_num 395) 2024-04-26T00:27:56.200 INFO:tasks.rados.rados.0.smithi045.stdout:1278: expect (ObjNum 470 snap 147 seq_num 470) 2024-04-26T00:27:56.425 INFO:tasks.rados.rados.0.smithi045.stdout:1279: expect (ObjNum 329 snap 96 seq_num 329) 2024-04-26T00:27:56.542 INFO:tasks.rados.rados.0.smithi045.stdout:1281: finishing write tid 1 to smithi04526579-4 2024-04-26T00:27:56.542 INFO:tasks.rados.rados.0.smithi045.stdout:1281: finishing write tid 2 to smithi04526579-4 2024-04-26T00:27:56.542 INFO:tasks.rados.rados.0.smithi045.stdout:1281: finishing write tid 3 to smithi04526579-4 2024-04-26T00:27:56.542 INFO:tasks.rados.rados.0.smithi045.stdout:1281: finishing write tid 4 to smithi04526579-4 2024-04-26T00:27:56.542 INFO:tasks.rados.rados.0.smithi045.stdout:1281: oid 4 updating version 0 to 702 2024-04-26T00:27:56.542 INFO:tasks.rados.rados.0.smithi045.stdout:1281: oid 4 updating version 702 to 703 2024-04-26T00:27:56.542 INFO:tasks.rados.rados.0.smithi045.stdout:1281: oid 4 updating version 703 to 704 2024-04-26T00:27:56.542 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 704 (ObjNum 478 snap 149 seq_num 478) dirty exists 2024-04-26T00:27:56.542 INFO:tasks.rados.rados.0.smithi045.stdout:1281: left oid 4 (ObjNum 478 snap 149 seq_num 478) 2024-04-26T00:27:57.095 INFO:tasks.rados.rados.0.smithi045.stdout:1278: done (4 left) 2024-04-26T00:27:57.095 INFO:tasks.rados.rados.0.smithi045.stdout:1279: done (3 left) 2024-04-26T00:27:57.096 INFO:tasks.rados.rados.0.smithi045.stdout:1280: done (2 left) 2024-04-26T00:27:57.096 INFO:tasks.rados.rados.0.smithi045.stdout:1281: done (1 left) 2024-04-26T00:27:57.096 INFO:tasks.rados.rados.0.smithi045.stdout:1282: done (0 left) 2024-04-26T00:27:57.096 INFO:tasks.rados.rados.0.smithi045.stdout:1283: read oid 6 snap 144 2024-04-26T00:27:57.096 INFO:tasks.rados.rados.0.smithi045.stdout:1283: expect (ObjNum 457 snap 142 seq_num 457) 2024-04-26T00:27:57.096 INFO:tasks.rados.rados.0.smithi045.stdout:1284: read oid 13 snap 141 2024-04-26T00:27:57.096 INFO:tasks.rados.rados.0.smithi045.stdout:1284: expect (ObjNum 337 snap 100 seq_num 337) 2024-04-26T00:27:57.096 INFO:tasks.rados.rados.0.smithi045.stdout:1285: copy_from oid 42 from oid 38 current snap is 150 2024-04-26T00:27:57.096 INFO:tasks.rados.rados.0.smithi045.stdout:1286: rollback oid 19 current snap is 150 2024-04-26T00:27:57.096 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 19 to 142 2024-04-26T00:27:57.096 INFO:tasks.rados.rados.0.smithi045.stdout:1287: write (excl) oid 31 current snap is 150 2024-04-26T00:27:57.096 INFO:tasks.rados.rados.0.smithi045.stdout:1287: seq_num 479 ranges {450553=574510,1613463=437090} 2024-04-26T00:27:57.107 INFO:tasks.rados.rados.0.smithi045.stdout:1287: writing smithi04526579-31 from 450553 to 1025063 tid 1 2024-04-26T00:27:57.118 INFO:tasks.rados.rados.0.smithi045.stdout:1287: writing smithi04526579-31 from 1613463 to 2050553 tid 2 2024-04-26T00:27:57.118 INFO:tasks.rados.rados.0.smithi045.stdout:1288: snap_remove snap 145 2024-04-26T00:27:58.110 INFO:tasks.rados.rados.0.smithi045.stdout:1288: done (5 left) 2024-04-26T00:27:58.110 INFO:tasks.rados.rados.0.smithi045.stdout:1289: delete oid 40 current snap is 150 2024-04-26T00:27:58.110 INFO:tasks.rados.rados.0.smithi045.stdout:1284: expect (ObjNum 337 snap 100 seq_num 337) 2024-04-26T00:27:58.303 INFO:tasks.rados.rados.0.smithi045.stdout:1286: finishing rollback tid 0 to smithi04526579-19 2024-04-26T00:27:58.303 INFO:tasks.rados.rados.0.smithi045.stdout:1286: finishing rollback tid 1 to smithi04526579-19 2024-04-26T00:27:58.303 INFO:tasks.rados.rados.0.smithi045.stdout:1283: expect (ObjNum 457 snap 142 seq_num 457) 2024-04-26T00:27:58.532 INFO:tasks.rados.rados.0.smithi045.stdout:1286: finishing rollback tid 2 to smithi04526579-19 2024-04-26T00:27:58.533 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 724 (ObjNum 443 snap 138 seq_num 443) dirty exists 2024-04-26T00:27:58.533 INFO:tasks.rados.rados.0.smithi045.stdout:1287: finishing write tid 1 to smithi04526579-31 2024-04-26T00:27:58.533 INFO:tasks.rados.rados.0.smithi045.stdout:1285: finishing copy_from racing read to smithi04526579-42 2024-04-26T00:27:58.533 INFO:tasks.rados.rados.0.smithi045.stdout:1287: finishing write tid 2 to smithi04526579-31 2024-04-26T00:27:58.533 INFO:tasks.rados.rados.0.smithi045.stdout:1287: finishing write tid 3 to smithi04526579-31 2024-04-26T00:27:58.533 INFO:tasks.rados.rados.0.smithi045.stdout:1287: finishing write tid 4 to smithi04526579-31 2024-04-26T00:27:58.533 INFO:tasks.rados.rados.0.smithi045.stdout:1287: oid 31 updating version 0 to 707 2024-04-26T00:27:58.533 INFO:tasks.rados.rados.0.smithi045.stdout:1287: oid 31 version 707 is already newer than 706 2024-04-26T00:27:58.533 INFO:tasks.rados.rados.0.smithi045.stdout:1287: oid 31 updating version 707 to 708 2024-04-26T00:27:58.533 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 708 (ObjNum 479 snap 150 seq_num 479) dirty exists 2024-04-26T00:27:58.533 INFO:tasks.rados.rados.0.smithi045.stdout:1287: left oid 31 (ObjNum 479 snap 150 seq_num 479) 2024-04-26T00:27:58.534 INFO:tasks.rados.rados.0.smithi045.stdout:1285: finishing copy_from to smithi04526579-42 2024-04-26T00:27:58.534 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 437 (ObjNum 356 snap 105 seq_num 356) dirty exists 2024-04-26T00:27:58.534 INFO:tasks.rados.rados.0.smithi045.stdout:1283: done (5 left) 2024-04-26T00:27:58.534 INFO:tasks.rados.rados.0.smithi045.stdout:1284: done (4 left) 2024-04-26T00:27:58.534 INFO:tasks.rados.rados.0.smithi045.stdout:1285: done (3 left) 2024-04-26T00:27:58.534 INFO:tasks.rados.rados.0.smithi045.stdout:1286: done (2 left) 2024-04-26T00:27:58.534 INFO:tasks.rados.rados.0.smithi045.stdout:1287: done (1 left) 2024-04-26T00:27:58.534 INFO:tasks.rados.rados.0.smithi045.stdout:1289: done (0 left) 2024-04-26T00:27:58.534 INFO:tasks.rados.rados.0.smithi045.stdout:1290: write (excl) oid 35 current snap is 150 2024-04-26T00:27:58.534 INFO:tasks.rados.rados.0.smithi045.stdout:1290: seq_num 481 ranges {484167=725008,1730427=353740} 2024-04-26T00:27:58.552 INFO:tasks.rados.rados.0.smithi045.stdout:1290: writing smithi04526579-35 from 484167 to 1209175 tid 1 2024-04-26T00:27:58.566 INFO:tasks.rados.rados.0.smithi045.stdout:1290: writing smithi04526579-35 from 1730427 to 2084167 tid 2 2024-04-26T00:27:58.567 INFO:tasks.rados.rados.0.smithi045.stdout:1291: write oid 27 current snap is 150 2024-04-26T00:27:58.567 INFO:tasks.rados.rados.0.smithi045.stdout:1290: finishing write tid 1 to smithi04526579-35 2024-04-26T00:27:58.567 INFO:tasks.rados.rados.0.smithi045.stdout:1291: seq_num 482 ranges {500974=400257,1388909=557169} 2024-04-26T00:27:58.577 INFO:tasks.rados.rados.0.smithi045.stdout:1291: writing smithi04526579-27 from 500974 to 901231 tid 1 2024-04-26T00:27:58.588 INFO:tasks.rados.rados.0.smithi045.stdout:1291: writing smithi04526579-27 from 1388909 to 1946078 tid 2 2024-04-26T00:27:58.588 INFO:tasks.rados.rados.0.smithi045.stdout:1292: write (excl) oid 17 current snap is 150 2024-04-26T00:27:58.589 INFO:tasks.rados.rados.0.smithi045.stdout:1292: seq_num 483 ranges {517781=475506,1731038=386743} 2024-04-26T00:27:58.594 INFO:tasks.rados.rados.0.smithi045.stdout:1292: writing smithi04526579-17 from 517781 to 993287 tid 1 2024-04-26T00:27:58.602 INFO:tasks.rados.rados.0.smithi045.stdout:1292: writing smithi04526579-17 from 1731038 to 2117781 tid 2 2024-04-26T00:27:58.602 INFO:tasks.rados.rados.0.smithi045.stdout:1293: copy_from oid 19 from oid 1 current snap is 150 2024-04-26T00:27:58.602 INFO:tasks.rados.rados.0.smithi045.stdout:1290: finishing write tid 2 to smithi04526579-35 2024-04-26T00:27:58.602 INFO:tasks.rados.rados.0.smithi045.stdout:1290: finishing write tid 3 to smithi04526579-35 2024-04-26T00:27:58.602 INFO:tasks.rados.rados.0.smithi045.stdout:1290: finishing write tid 4 to smithi04526579-35 2024-04-26T00:27:58.602 INFO:tasks.rados.rados.0.smithi045.stdout:1290: oid 35 updating version 0 to 735 2024-04-26T00:27:58.602 INFO:tasks.rados.rados.0.smithi045.stdout:1290: oid 35 updating version 735 to 736 2024-04-26T00:27:58.602 INFO:tasks.rados.rados.0.smithi045.stdout:1290: oid 35 version 736 is already newer than 734 2024-04-26T00:27:58.602 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 736 (ObjNum 481 snap 150 seq_num 481) dirty exists 2024-04-26T00:27:58.602 INFO:tasks.rados.rados.0.smithi045.stdout:1290: left oid 35 (ObjNum 481 snap 150 seq_num 481) 2024-04-26T00:27:58.602 INFO:tasks.rados.rados.0.smithi045.stdout:1291: finishing write tid 1 to smithi04526579-27 2024-04-26T00:27:58.602 INFO:tasks.rados.rados.0.smithi045.stdout:1291: finishing write tid 2 to smithi04526579-27 2024-04-26T00:27:58.602 INFO:tasks.rados.rados.0.smithi045.stdout:1290: done (3 left) 2024-04-26T00:27:58.603 INFO:tasks.rados.rados.0.smithi045.stdout:1294: snap_remove snap 141 2024-04-26T00:27:59.110 INFO:tasks.rados.rados.0.smithi045.stdout:1294: done (3 left) 2024-04-26T00:27:59.110 INFO:tasks.rados.rados.0.smithi045.stdout:1295: snap_create 2024-04-26T00:27:59.110 INFO:tasks.rados.rados.0.smithi045.stdout:1291: finishing write tid 3 to smithi04526579-27 2024-04-26T00:27:59.110 INFO:tasks.rados.rados.0.smithi045.stdout:1291: finishing write tid 4 to smithi04526579-27 2024-04-26T00:27:59.110 INFO:tasks.rados.rados.0.smithi045.stdout:1291: oid 27 updating version 0 to 740 2024-04-26T00:27:59.111 INFO:tasks.rados.rados.0.smithi045.stdout:1291: oid 27 version 740 is already newer than 739 2024-04-26T00:27:59.111 INFO:tasks.rados.rados.0.smithi045.stdout:1291: oid 27 version 740 is already newer than 738 2024-04-26T00:27:59.111 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 740 (ObjNum 482 snap 150 seq_num 482) dirty exists 2024-04-26T00:27:59.111 INFO:tasks.rados.rados.0.smithi045.stdout:1291: left oid 27 (ObjNum 482 snap 150 seq_num 482) 2024-04-26T00:27:59.111 INFO:tasks.rados.rados.0.smithi045.stdout:1292: finishing write tid 1 to smithi04526579-17 2024-04-26T00:27:59.111 INFO:tasks.rados.rados.0.smithi045.stdout:1293: finishing copy_from to smithi04526579-19 2024-04-26T00:27:59.111 INFO:tasks.rados.rados.0.smithi045.stdout:1293: got expected ENOENT (src dne) 2024-04-26T00:27:59.111 INFO:tasks.rados.rados.0.smithi045.stdout:1293: finishing copy_from racing read to smithi04526579-19 2024-04-26T00:27:59.111 INFO:tasks.rados.rados.0.smithi045.stdout:1292: finishing write tid 2 to smithi04526579-17 2024-04-26T00:27:59.111 INFO:tasks.rados.rados.0.smithi045.stdout:1292: finishing write tid 3 to smithi04526579-17 2024-04-26T00:27:59.111 INFO:tasks.rados.rados.0.smithi045.stdout:1292: finishing write tid 4 to smithi04526579-17 2024-04-26T00:27:59.111 INFO:tasks.rados.rados.0.smithi045.stdout:1292: oid 17 updating version 0 to 847 2024-04-26T00:27:59.111 INFO:tasks.rados.rados.0.smithi045.stdout:1292: oid 17 version 847 is already newer than 846 2024-04-26T00:27:59.111 INFO:tasks.rados.rados.0.smithi045.stdout:1292: oid 17 updating version 847 to 848 2024-04-26T00:27:59.112 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 848 (ObjNum 483 snap 150 seq_num 483) dirty exists 2024-04-26T00:27:59.112 INFO:tasks.rados.rados.0.smithi045.stdout:1292: left oid 17 (ObjNum 483 snap 150 seq_num 483) 2024-04-26T00:27:59.245 INFO:tasks.rados.rados.0.smithi045.stdout:1291: done (3 left) 2024-04-26T00:27:59.246 INFO:tasks.rados.rados.0.smithi045.stdout:1292: done (2 left) 2024-04-26T00:27:59.246 INFO:tasks.rados.rados.0.smithi045.stdout:1293: done (1 left) 2024-04-26T00:27:59.246 INFO:tasks.rados.rados.0.smithi045.stdout:1295: done (0 left) 2024-04-26T00:27:59.246 INFO:tasks.rados.rados.0.smithi045.stdout:1296: snap_create 2024-04-26T00:28:00.252 INFO:tasks.rados.rados.0.smithi045.stdout:1296: done (0 left) 2024-04-26T00:28:00.252 INFO:tasks.rados.rados.0.smithi045.stdout:1297: delete oid 25 current snap is 152 2024-04-26T00:28:00.260 INFO:tasks.rados.rados.0.smithi045.stdout:1297: done (0 left) 2024-04-26T00:28:00.260 INFO:tasks.rados.rados.0.smithi045.stdout:1298: write oid 29 current snap is 152 2024-04-26T00:28:00.260 INFO:tasks.rados.rados.0.smithi045.stdout:1298: seq_num 485 ranges {551395=626004,1848002=303393} 2024-04-26T00:28:00.279 INFO:tasks.rados.rados.0.smithi045.stdout:1298: writing smithi04526579-29 from 551395 to 1177399 tid 1 2024-04-26T00:28:00.297 INFO:tasks.rados.rados.0.smithi045.stdout:1298: writing smithi04526579-29 from 1848002 to 2151395 tid 2 2024-04-26T00:28:00.297 INFO:tasks.rados.rados.0.smithi045.stdout:1299: delete oid 7 current snap is 152 2024-04-26T00:28:00.297 INFO:tasks.rados.rados.0.smithi045.stdout:1298: finishing write tid 1 to smithi04526579-29 2024-04-26T00:28:00.304 INFO:tasks.rados.rados.0.smithi045.stdout:1298: finishing write tid 2 to smithi04526579-29 2024-04-26T00:28:00.305 INFO:tasks.rados.rados.0.smithi045.stdout:1298: finishing write tid 3 to smithi04526579-29 2024-04-26T00:28:00.305 INFO:tasks.rados.rados.0.smithi045.stdout:1298: finishing write tid 4 to smithi04526579-29 2024-04-26T00:28:00.306 INFO:tasks.rados.rados.0.smithi045.stdout:1298: oid 29 updating version 0 to 462 2024-04-26T00:28:00.306 INFO:tasks.rados.rados.0.smithi045.stdout:1298: oid 29 version 462 is already newer than 460 2024-04-26T00:28:00.306 INFO:tasks.rados.rados.0.smithi045.stdout:1298: oid 29 version 462 is already newer than 461 2024-04-26T00:28:00.306 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 462 (ObjNum 485 snap 152 seq_num 485) dirty exists 2024-04-26T00:28:00.306 INFO:tasks.rados.rados.0.smithi045.stdout:1298: left oid 29 (ObjNum 485 snap 152 seq_num 485) 2024-04-26T00:28:00.308 INFO:tasks.rados.rados.0.smithi045.stdout:1298: done (1 left) 2024-04-26T00:28:00.308 INFO:tasks.rados.rados.0.smithi045.stdout:1299: done (0 left) 2024-04-26T00:28:00.308 INFO:tasks.rados.rados.0.smithi045.stdout:1300: snap_create 2024-04-26T00:28:01.254 INFO:tasks.rados.rados.0.smithi045.stdout:1300: done (0 left) 2024-04-26T00:28:01.255 INFO:tasks.rados.rados.0.smithi045.stdout:1301: write (excl) oid 11 current snap is 153 2024-04-26T00:28:01.255 INFO:tasks.rados.rados.0.smithi045.stdout:1301: seq_num 487 ranges {585009=492855,1564966=620043} 2024-04-26T00:28:01.272 INFO:tasks.rados.rados.0.smithi045.stdout:1301: writing smithi04526579-11 from 585009 to 1077864 tid 1 2024-04-26T00:28:01.292 INFO:tasks.rados.rados.0.smithi045.stdout:1301: writing smithi04526579-11 from 1564966 to 2185009 tid 2 2024-04-26T00:28:01.292 INFO:tasks.rados.rados.0.smithi045.stdout:1302: read oid 18 snap -1 2024-04-26T00:28:01.292 INFO:tasks.rados.rados.0.smithi045.stdout:1302: expect deleted 2024-04-26T00:28:01.292 INFO:tasks.rados.rados.0.smithi045.stdout:1303: read oid 30 snap -1 2024-04-26T00:28:01.292 INFO:tasks.rados.rados.0.smithi045.stdout:1303: expect (ObjNum 447 snap 139 seq_num 447) 2024-04-26T00:28:01.292 INFO:tasks.rados.rados.0.smithi045.stdout:1304: snap_remove snap 152 2024-04-26T00:28:02.263 INFO:tasks.rados.rados.0.smithi045.stdout:1304: done (3 left) 2024-04-26T00:28:02.263 INFO:tasks.rados.rados.0.smithi045.stdout:1305: read oid 10 snap 142 2024-04-26T00:28:02.263 INFO:tasks.rados.rados.0.smithi045.stdout:1305: expect (ObjNum 451 snap 140 seq_num 451) 2024-04-26T00:28:02.263 INFO:tasks.rados.rados.0.smithi045.stdout:1301: finishing write tid 1 to smithi04526579-11 2024-04-26T00:28:02.264 INFO:tasks.rados.rados.0.smithi045.stdout:1301: finishing write tid 2 to smithi04526579-11 2024-04-26T00:28:02.264 INFO:tasks.rados.rados.0.smithi045.stdout:1301: finishing write tid 3 to smithi04526579-11 2024-04-26T00:28:02.264 INFO:tasks.rados.rados.0.smithi045.stdout:1301: finishing write tid 4 to smithi04526579-11 2024-04-26T00:28:02.264 INFO:tasks.rados.rados.0.smithi045.stdout:1301: oid 11 updating version 0 to 838 2024-04-26T00:28:02.264 INFO:tasks.rados.rados.0.smithi045.stdout:1301: oid 11 updating version 838 to 839 2024-04-26T00:28:02.264 INFO:tasks.rados.rados.0.smithi045.stdout:1301: oid 11 updating version 839 to 840 2024-04-26T00:28:02.264 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 840 (ObjNum 487 snap 153 seq_num 487) dirty exists 2024-04-26T00:28:02.264 INFO:tasks.rados.rados.0.smithi045.stdout:1301: left oid 11 (ObjNum 487 snap 153 seq_num 487) 2024-04-26T00:28:02.264 INFO:tasks.rados.rados.0.smithi045.stdout:1303: expect (ObjNum 447 snap 139 seq_num 447) 2024-04-26T00:28:02.451 INFO:tasks.rados.rados.0.smithi045.stdout:1305: expect (ObjNum 451 snap 140 seq_num 451) 2024-04-26T00:28:02.706 INFO:tasks.rados.rados.0.smithi045.stdout:1301: done (3 left) 2024-04-26T00:28:02.706 INFO:tasks.rados.rados.0.smithi045.stdout:1302: done (2 left) 2024-04-26T00:28:02.706 INFO:tasks.rados.rados.0.smithi045.stdout:1303: done (1 left) 2024-04-26T00:28:02.706 INFO:tasks.rados.rados.0.smithi045.stdout:1305: done (0 left) 2024-04-26T00:28:02.706 INFO:tasks.rados.rados.0.smithi045.stdout:1306: delete oid 33 current snap is 153 2024-04-26T00:28:02.715 INFO:tasks.rados.rados.0.smithi045.stdout:1306: done (0 left) 2024-04-26T00:28:02.715 INFO:tasks.rados.rados.0.smithi045.stdout:1307: write oid 33 current snap is 153 2024-04-26T00:28:02.715 INFO:tasks.rados.rados.0.smithi045.stdout:1307: seq_num 489 ranges {618623=643353,1681930=536693} 2024-04-26T00:28:02.735 INFO:tasks.rados.rados.0.smithi045.stdout:1307: writing smithi04526579-33 from 618623 to 1261976 tid 1 2024-04-26T00:28:02.751 INFO:tasks.rados.rados.0.smithi045.stdout:1307: writing smithi04526579-33 from 1681930 to 2218623 tid 2 2024-04-26T00:28:02.751 INFO:tasks.rados.rados.0.smithi045.stdout:1308: read oid 10 snap -1 2024-04-26T00:28:02.751 INFO:tasks.rados.rados.0.smithi045.stdout:1308: expect (ObjNum 451 snap 140 seq_num 451) 2024-04-26T00:28:02.751 INFO:tasks.rados.rados.0.smithi045.stdout:1309: write oid 37 current snap is 153 2024-04-26T00:28:02.751 INFO:tasks.rados.rados.0.smithi045.stdout:1307: finishing write tid 1 to smithi04526579-33 2024-04-26T00:28:02.752 INFO:tasks.rados.rados.0.smithi045.stdout:1309: seq_num 490 ranges {635430=718602,2140412=95018} 2024-04-26T00:28:02.773 INFO:tasks.rados.rados.0.smithi045.stdout:1309: writing smithi04526579-37 from 635430 to 1354032 tid 1 2024-04-26T00:28:02.787 INFO:tasks.rados.rados.0.smithi045.stdout:1309: writing smithi04526579-37 from 2140412 to 2235430 tid 2 2024-04-26T00:28:02.787 INFO:tasks.rados.rados.0.smithi045.stdout:1310: delete oid 22 current snap is 153 2024-04-26T00:28:02.787 INFO:tasks.rados.rados.0.smithi045.stdout:1307: finishing write tid 2 to smithi04526579-33 2024-04-26T00:28:02.787 INFO:tasks.rados.rados.0.smithi045.stdout:1307: finishing write tid 3 to smithi04526579-33 2024-04-26T00:28:02.787 INFO:tasks.rados.rados.0.smithi045.stdout:1308: expect (ObjNum 451 snap 140 seq_num 451) 2024-04-26T00:28:03.045 INFO:tasks.rados.rados.0.smithi045.stdout:1307: finishing write tid 4 to smithi04526579-33 2024-04-26T00:28:03.045 INFO:tasks.rados.rados.0.smithi045.stdout:1307: oid 33 updating version 0 to 1015 2024-04-26T00:28:03.045 INFO:tasks.rados.rados.0.smithi045.stdout:1307: oid 33 updating version 1015 to 1017 2024-04-26T00:28:03.045 INFO:tasks.rados.rados.0.smithi045.stdout:1307: oid 33 version 1017 is already newer than 1016 2024-04-26T00:28:03.045 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 1017 (ObjNum 489 snap 153 seq_num 489) dirty exists 2024-04-26T00:28:03.045 INFO:tasks.rados.rados.0.smithi045.stdout:1307: left oid 33 (ObjNum 489 snap 153 seq_num 489) 2024-04-26T00:28:03.045 INFO:tasks.rados.rados.0.smithi045.stdout:1309: finishing write tid 1 to smithi04526579-37 2024-04-26T00:28:03.045 INFO:tasks.rados.rados.0.smithi045.stdout:1309: finishing write tid 2 to smithi04526579-37 2024-04-26T00:28:03.045 INFO:tasks.rados.rados.0.smithi045.stdout:1309: finishing write tid 3 to smithi04526579-37 2024-04-26T00:28:03.046 INFO:tasks.rados.rados.0.smithi045.stdout:1309: finishing write tid 4 to smithi04526579-37 2024-04-26T00:28:03.046 INFO:tasks.rados.rados.0.smithi045.stdout:1309: oid 37 updating version 0 to 1018 2024-04-26T00:28:03.046 INFO:tasks.rados.rados.0.smithi045.stdout:1309: oid 37 updating version 1018 to 1019 2024-04-26T00:28:03.046 INFO:tasks.rados.rados.0.smithi045.stdout:1309: oid 37 updating version 1019 to 1020 2024-04-26T00:28:03.046 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 1020 (ObjNum 490 snap 153 seq_num 490) dirty exists 2024-04-26T00:28:03.046 INFO:tasks.rados.rados.0.smithi045.stdout:1309: left oid 37 (ObjNum 490 snap 153 seq_num 490) 2024-04-26T00:28:03.046 INFO:tasks.rados.rados.0.smithi045.stdout:1307: done (3 left) 2024-04-26T00:28:03.046 INFO:tasks.rados.rados.0.smithi045.stdout:1308: done (2 left) 2024-04-26T00:28:03.046 INFO:tasks.rados.rados.0.smithi045.stdout:1309: done (1 left) 2024-04-26T00:28:03.046 INFO:tasks.rados.rados.0.smithi045.stdout:1310: done (0 left) 2024-04-26T00:28:03.046 INFO:tasks.rados.rados.0.smithi045.stdout:1311: write oid 49 current snap is 153 2024-04-26T00:28:03.046 INFO:tasks.rados.rados.0.smithi045.stdout:1311: seq_num 492 ranges {669044=469100,1741023=459161} 2024-04-26T00:28:03.062 INFO:tasks.rados.rados.0.smithi045.stdout:1311: writing smithi04526579-49 from 669044 to 1138144 tid 1 2024-04-26T00:28:03.078 INFO:tasks.rados.rados.0.smithi045.stdout:1311: writing smithi04526579-49 from 1741023 to 2200184 tid 2 2024-04-26T00:28:03.079 INFO:tasks.rados.rados.0.smithi045.stdout:1312: snap_remove snap 144 2024-04-26T00:28:03.558 INFO:tasks.rados.rados.0.smithi045.stdout:1312: done (1 left) 2024-04-26T00:28:03.558 INFO:tasks.rados.rados.0.smithi045.stdout:1313: snap_create 2024-04-26T00:28:03.558 INFO:tasks.rados.rados.0.smithi045.stdout:1311: finishing write tid 1 to smithi04526579-49 2024-04-26T00:28:03.558 INFO:tasks.rados.rados.0.smithi045.stdout:1311: finishing write tid 2 to smithi04526579-49 2024-04-26T00:28:03.558 INFO:tasks.rados.rados.0.smithi045.stdout:1311: finishing write tid 3 to smithi04526579-49 2024-04-26T00:28:03.558 INFO:tasks.rados.rados.0.smithi045.stdout:1311: finishing write tid 4 to smithi04526579-49 2024-04-26T00:28:03.558 INFO:tasks.rados.rados.0.smithi045.stdout:1311: oid 49 updating version 0 to 767 2024-04-26T00:28:03.558 INFO:tasks.rados.rados.0.smithi045.stdout:1311: oid 49 version 767 is already newer than 766 2024-04-26T00:28:03.558 INFO:tasks.rados.rados.0.smithi045.stdout:1311: oid 49 version 767 is already newer than 765 2024-04-26T00:28:03.559 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 767 (ObjNum 492 snap 153 seq_num 492) dirty exists 2024-04-26T00:28:03.559 INFO:tasks.rados.rados.0.smithi045.stdout:1311: left oid 49 (ObjNum 492 snap 153 seq_num 492) 2024-04-26T00:28:04.249 INFO:tasks.rados.rados.0.smithi045.stdout:1311: done (1 left) 2024-04-26T00:28:04.249 INFO:tasks.rados.rados.0.smithi045.stdout:1313: done (0 left) 2024-04-26T00:28:04.249 INFO:tasks.rados.rados.0.smithi045.stdout:1314: copy_from oid 4 from oid 9 current snap is 154 2024-04-26T00:28:04.249 INFO:tasks.rados.rados.0.smithi045.stdout:1315: read oid 12 snap -1 2024-04-26T00:28:04.249 INFO:tasks.rados.rados.0.smithi045.stdout:1315: expect (ObjNum 477 snap 148 seq_num 477) 2024-04-26T00:28:04.249 INFO:tasks.rados.rados.0.smithi045.stdout:1316: write (excl) oid 45 current snap is 154 2024-04-26T00:28:04.249 INFO:tasks.rados.rados.0.smithi045.stdout:1316: seq_num 493 ranges {685851=544349,1799505=486346} 2024-04-26T00:28:04.269 INFO:tasks.rados.rados.0.smithi045.stdout:1316: writing smithi04526579-45 from 685851 to 1230200 tid 1 2024-04-26T00:28:04.287 INFO:tasks.rados.rados.0.smithi045.stdout:1316: writing smithi04526579-45 from 1799505 to 2285851 tid 2 2024-04-26T00:28:04.287 INFO:tasks.rados.rados.0.smithi045.stdout:1317: delete oid 25 current snap is 154 2024-04-26T00:28:04.287 INFO:tasks.rados.rados.0.smithi045.stdout:1315: expect (ObjNum 477 snap 148 seq_num 477) 2024-04-26T00:28:04.433 INFO:tasks.rados.rados.0.smithi045.stdout:1316: finishing write tid 1 to smithi04526579-45 2024-04-26T00:28:04.433 INFO:tasks.rados.rados.0.smithi045.stdout:1314: finishing copy_from racing read to smithi04526579-4 2024-04-26T00:28:04.433 INFO:tasks.rados.rados.0.smithi045.stdout:1316: finishing write tid 2 to smithi04526579-45 2024-04-26T00:28:04.433 INFO:tasks.rados.rados.0.smithi045.stdout:1316: finishing write tid 3 to smithi04526579-45 2024-04-26T00:28:04.433 INFO:tasks.rados.rados.0.smithi045.stdout:1314: finishing copy_from to smithi04526579-4 2024-04-26T00:28:04.434 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 700 (ObjNum 474 snap 147 seq_num 474) dirty exists 2024-04-26T00:28:04.434 INFO:tasks.rados.rados.0.smithi045.stdout:1316: finishing write tid 4 to smithi04526579-45 2024-04-26T00:28:04.434 INFO:tasks.rados.rados.0.smithi045.stdout:1316: oid 45 updating version 0 to 841 2024-04-26T00:28:04.434 INFO:tasks.rados.rados.0.smithi045.stdout:1316: oid 45 updating version 841 to 842 2024-04-26T00:28:04.434 INFO:tasks.rados.rados.0.smithi045.stdout:1316: oid 45 updating version 842 to 843 2024-04-26T00:28:04.434 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 843 (ObjNum 493 snap 154 seq_num 493) dirty exists 2024-04-26T00:28:04.434 INFO:tasks.rados.rados.0.smithi045.stdout:1316: left oid 45 (ObjNum 493 snap 154 seq_num 493) 2024-04-26T00:28:04.434 INFO:tasks.rados.rados.0.smithi045.stdout:1314: done (3 left) 2024-04-26T00:28:04.434 INFO:tasks.rados.rados.0.smithi045.stdout:1315: done (2 left) 2024-04-26T00:28:04.434 INFO:tasks.rados.rados.0.smithi045.stdout:1316: done (1 left) 2024-04-26T00:28:04.434 INFO:tasks.rados.rados.0.smithi045.stdout:1317: done (0 left) 2024-04-26T00:28:04.434 INFO:tasks.rados.rados.0.smithi045.stdout:1318: copy_from oid 15 from oid 37 current snap is 154 2024-04-26T00:28:04.434 INFO:tasks.rados.rados.0.smithi045.stdout:1319: delete oid 47 current snap is 154 2024-04-26T00:28:04.446 INFO:tasks.rados.rados.0.smithi045.stdout:1319: done (1 left) 2024-04-26T00:28:04.446 INFO:tasks.rados.rados.0.smithi045.stdout:1320: copy_from oid 9 from oid 23 current snap is 154 2024-04-26T00:28:04.446 INFO:tasks.rados.rados.0.smithi045.stdout:1321: copy_from oid 24 from oid 18 current snap is 154 2024-04-26T00:28:04.446 INFO:tasks.rados.rados.0.smithi045.stdout:1322: delete oid 7 current snap is 154 2024-04-26T00:28:04.449 INFO:tasks.rados.rados.0.smithi045.stdout:1321: finishing copy_from to smithi04526579-24 2024-04-26T00:28:04.449 INFO:tasks.rados.rados.0.smithi045.stdout:1321: got expected ENOENT (src dne) 2024-04-26T00:28:04.449 INFO:tasks.rados.rados.0.smithi045.stdout:1321: finishing copy_from racing read to smithi04526579-24 2024-04-26T00:28:04.454 INFO:tasks.rados.rados.0.smithi045.stdout:1321: done (3 left) 2024-04-26T00:28:04.454 INFO:tasks.rados.rados.0.smithi045.stdout:1322: done (2 left) 2024-04-26T00:28:04.454 INFO:tasks.rados.rados.0.smithi045.stdout:1323: rollback oid 47 current snap is 154 2024-04-26T00:28:04.454 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 47 to 128 2024-04-26T00:28:04.454 INFO:tasks.rados.rados.0.smithi045.stdout:1324: copy_from oid 5 from oid 12 current snap is 154 2024-04-26T00:28:04.454 INFO:tasks.rados.rados.0.smithi045.stdout:1325: write (excl) oid 20 current snap is 154 2024-04-26T00:28:04.454 INFO:tasks.rados.rados.0.smithi045.stdout:1325: seq_num 497 ranges {753079=561698,2033433=319646} 2024-04-26T00:28:04.475 INFO:tasks.rados.rados.0.smithi045.stdout:1325: writing smithi04526579-20 from 753079 to 1314777 tid 1 2024-04-26T00:28:04.494 INFO:tasks.rados.rados.0.smithi045.stdout:1325: writing smithi04526579-20 from 2033433 to 2353079 tid 2 2024-04-26T00:28:04.494 INFO:tasks.rados.rados.0.smithi045.stdout:1326: copy_from oid 36 from oid 27 current snap is 154 2024-04-26T00:28:04.494 INFO:tasks.rados.rados.0.smithi045.stdout:1318: finishing copy_from racing read to smithi04526579-15 2024-04-26T00:28:04.494 INFO:tasks.rados.rados.0.smithi045.stdout:1318: finishing copy_from to smithi04526579-15 2024-04-26T00:28:04.494 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 1020 (ObjNum 490 snap 153 seq_num 490) dirty exists 2024-04-26T00:28:04.494 INFO:tasks.rados.rados.0.smithi045.stdout:1323: finishing rollback tid 1 to smithi04526579-47 2024-04-26T00:28:04.494 INFO:tasks.rados.rados.0.smithi045.stdout:1323: finishing rollback tid 2 to smithi04526579-47 2024-04-26T00:28:04.494 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 779 (ObjNum 413 snap 127 seq_num 413) dirty exists 2024-04-26T00:28:04.495 INFO:tasks.rados.rados.0.smithi045.stdout:1320: finishing copy_from racing read to smithi04526579-9 2024-04-26T00:28:04.495 INFO:tasks.rados.rados.0.smithi045.stdout:1320: finishing copy_from to smithi04526579-9 2024-04-26T00:28:04.495 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 602 (ObjNum 453 snap 141 seq_num 453) dirty exists 2024-04-26T00:28:04.495 INFO:tasks.rados.rados.0.smithi045.stdout:1324: finishing copy_from racing read to smithi04526579-5 2024-04-26T00:28:04.495 INFO:tasks.rados.rados.0.smithi045.stdout:1325: finishing write tid 1 to smithi04526579-20 2024-04-26T00:28:04.495 INFO:tasks.rados.rados.0.smithi045.stdout:1324: finishing copy_from to smithi04526579-5 2024-04-26T00:28:04.495 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 605 (ObjNum 477 snap 148 seq_num 477) dirty exists 2024-04-26T00:28:04.495 INFO:tasks.rados.rados.0.smithi045.stdout:1318: done (5 left) 2024-04-26T00:28:04.495 INFO:tasks.rados.rados.0.smithi045.stdout:1320: done (4 left) 2024-04-26T00:28:04.495 INFO:tasks.rados.rados.0.smithi045.stdout:1323: done (3 left) 2024-04-26T00:28:04.495 INFO:tasks.rados.rados.0.smithi045.stdout:1324: done (2 left) 2024-04-26T00:28:04.496 INFO:tasks.rados.rados.0.smithi045.stdout:1327: rollback oid 39 current snap is 154 2024-04-26T00:28:04.496 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 39 to 127 2024-04-26T00:28:04.496 INFO:tasks.rados.rados.0.smithi045.stdout:1328: delete oid 25 current snap is 154 2024-04-26T00:28:04.498 INFO:tasks.rados.rados.0.smithi045.stdout:1328: done (3 left) 2024-04-26T00:28:04.498 INFO:tasks.rados.rados.0.smithi045.stdout:1329: delete oid 28 current snap is 154 2024-04-26T00:28:04.508 INFO:tasks.rados.rados.0.smithi045.stdout:1325: finishing write tid 2 to smithi04526579-20 2024-04-26T00:28:04.508 INFO:tasks.rados.rados.0.smithi045.stdout:1329: done (3 left) 2024-04-26T00:28:04.508 INFO:tasks.rados.rados.0.smithi045.stdout:1330: write oid 2 current snap is 154 2024-04-26T00:28:04.508 INFO:tasks.rados.rados.0.smithi045.stdout:1330: seq_num 500 ranges {403500=787445,1692526=473837} 2024-04-26T00:28:04.524 INFO:tasks.rados.rados.0.smithi045.stdout:1330: writing smithi04526579-2 from 403500 to 1190945 tid 1 2024-04-26T00:28:04.537 INFO:tasks.rados.rados.0.smithi045.stdout:1330: writing smithi04526579-2 from 1692526 to 2166363 tid 2 2024-04-26T00:28:04.537 INFO:tasks.rados.rados.0.smithi045.stdout:1331: delete oid 47 current snap is 154 2024-04-26T00:28:04.537 INFO:tasks.rados.rados.0.smithi045.stdout:1325: finishing write tid 3 to smithi04526579-20 2024-04-26T00:28:04.537 INFO:tasks.rados.rados.0.smithi045.stdout:1325: finishing write tid 4 to smithi04526579-20 2024-04-26T00:28:04.537 INFO:tasks.rados.rados.0.smithi045.stdout:1325: oid 20 updating version 0 to 482 2024-04-26T00:28:04.537 INFO:tasks.rados.rados.0.smithi045.stdout:1325: oid 20 updating version 482 to 484 2024-04-26T00:28:04.537 INFO:tasks.rados.rados.0.smithi045.stdout:1325: oid 20 version 484 is already newer than 483 2024-04-26T00:28:04.537 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 484 (ObjNum 497 snap 154 seq_num 497) dirty exists 2024-04-26T00:28:04.537 INFO:tasks.rados.rados.0.smithi045.stdout:1325: left oid 20 (ObjNum 497 snap 154 seq_num 497) 2024-04-26T00:28:04.537 INFO:tasks.rados.rados.0.smithi045.stdout:1327: finishing rollback tid 0 to smithi04526579-39 2024-04-26T00:28:04.538 INFO:tasks.rados.rados.0.smithi045.stdout:1326: finishing copy_from racing read to smithi04526579-36 2024-04-26T00:28:04.538 INFO:tasks.rados.rados.0.smithi045.stdout:1327: finishing rollback tid 1 to smithi04526579-39 2024-04-26T00:28:04.538 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 487 (ObjNum 129 snap 0 seq_num 2242045488) dirty dne 2024-04-26T00:28:04.538 INFO:tasks.rados.rados.0.smithi045.stdout:1326: finishing copy_from to smithi04526579-36 2024-04-26T00:28:04.538 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 740 (ObjNum 482 snap 150 seq_num 482) dirty exists 2024-04-26T00:28:04.538 INFO:tasks.rados.rados.0.smithi045.stdout:1330: finishing write tid 1 to smithi04526579-2 2024-04-26T00:28:04.541 INFO:tasks.rados.rados.0.smithi045.stdout:1325: done (4 left) 2024-04-26T00:28:04.541 INFO:tasks.rados.rados.0.smithi045.stdout:1326: done (3 left) 2024-04-26T00:28:04.541 INFO:tasks.rados.rados.0.smithi045.stdout:1327: done (2 left) 2024-04-26T00:28:04.541 INFO:tasks.rados.rados.0.smithi045.stdout:1331: done (1 left) 2024-04-26T00:28:04.541 INFO:tasks.rados.rados.0.smithi045.stdout:1332: write oid 15 current snap is 154 2024-04-26T00:28:04.541 INFO:tasks.rados.rados.0.smithi045.stdout:1332: seq_num 502 ranges {437114=654296,1525843=477506} 2024-04-26T00:28:04.548 INFO:tasks.rados.rados.0.smithi045.stdout:1332: writing smithi04526579-15 from 437114 to 1091410 tid 1 2024-04-26T00:28:04.556 INFO:tasks.rados.rados.0.smithi045.stdout:1332: writing smithi04526579-15 from 1525843 to 2003349 tid 2 2024-04-26T00:28:04.556 INFO:tasks.rados.rados.0.smithi045.stdout:1333: snap_create 2024-04-26T00:28:04.556 INFO:tasks.rados.rados.0.smithi045.stdout:1330: finishing write tid 2 to smithi04526579-2 2024-04-26T00:28:04.556 INFO:tasks.rados.rados.0.smithi045.stdout:1330: finishing write tid 3 to smithi04526579-2 2024-04-26T00:28:04.556 INFO:tasks.rados.rados.0.smithi045.stdout:1330: finishing write tid 4 to smithi04526579-2 2024-04-26T00:28:04.556 INFO:tasks.rados.rados.0.smithi045.stdout:1330: oid 2 updating version 0 to 734 2024-04-26T00:28:04.556 INFO:tasks.rados.rados.0.smithi045.stdout:1330: oid 2 version 734 is already newer than 732 2024-04-26T00:28:04.556 INFO:tasks.rados.rados.0.smithi045.stdout:1330: oid 2 version 734 is already newer than 733 2024-04-26T00:28:04.556 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 734 (ObjNum 500 snap 154 seq_num 500) dirty exists 2024-04-26T00:28:04.556 INFO:tasks.rados.rados.0.smithi045.stdout:1330: left oid 2 (ObjNum 500 snap 154 seq_num 500) 2024-04-26T00:28:04.557 INFO:tasks.rados.rados.0.smithi045.stdout:1332: finishing write tid 1 to smithi04526579-15 2024-04-26T00:28:04.561 INFO:tasks.rados.rados.0.smithi045.stdout:1332: finishing write tid 2 to smithi04526579-15 2024-04-26T00:28:04.561 INFO:tasks.rados.rados.0.smithi045.stdout:1332: finishing write tid 3 to smithi04526579-15 2024-04-26T00:28:04.562 INFO:tasks.rados.rados.0.smithi045.stdout:1332: finishing write tid 4 to smithi04526579-15 2024-04-26T00:28:04.562 INFO:tasks.rados.rados.0.smithi045.stdout:1332: oid 15 updating version 0 to 1024 2024-04-26T00:28:04.562 INFO:tasks.rados.rados.0.smithi045.stdout:1332: oid 15 version 1024 is already newer than 1022 2024-04-26T00:28:04.562 INFO:tasks.rados.rados.0.smithi045.stdout:1332: oid 15 version 1024 is already newer than 1023 2024-04-26T00:28:04.562 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 1024 (ObjNum 502 snap 154 seq_num 502) dirty exists 2024-04-26T00:28:04.562 INFO:tasks.rados.rados.0.smithi045.stdout:1332: left oid 15 (ObjNum 502 snap 154 seq_num 502) 2024-04-26T00:28:05.641 INFO:tasks.rados.rados.0.smithi045.stdout:1330: done (2 left) 2024-04-26T00:28:05.641 INFO:tasks.rados.rados.0.smithi045.stdout:1332: done (1 left) 2024-04-26T00:28:05.641 INFO:tasks.rados.rados.0.smithi045.stdout:1333: done (0 left) 2024-04-26T00:28:05.641 INFO:tasks.rados.rados.0.smithi045.stdout:1334: snap_remove snap 148 2024-04-26T00:28:06.646 INFO:tasks.rados.rados.0.smithi045.stdout:1334: done (0 left) 2024-04-26T00:28:06.646 INFO:tasks.rados.rados.0.smithi045.stdout:1335: delete oid 21 current snap is 155 2024-04-26T00:28:06.660 INFO:tasks.rados.rados.0.smithi045.stdout:1335: done (0 left) 2024-04-26T00:28:06.660 INFO:tasks.rados.rados.0.smithi045.stdout:1336: rollback oid 33 current snap is 155 2024-04-26T00:28:06.660 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 33 to 143 2024-04-26T00:28:06.660 INFO:tasks.rados.rados.0.smithi045.stdout:1337: snap_remove snap 128 2024-04-26T00:28:07.674 INFO:tasks.rados.rados.0.smithi045.stdout:1337: done (1 left) 2024-04-26T00:28:07.675 INFO:tasks.rados.rados.0.smithi045.stdout:1338: write (excl) oid 16 current snap is 155 2024-04-26T00:28:07.675 INFO:tasks.rados.rados.0.smithi045.stdout:1338: seq_num 504 ranges {470728=404794,1526454=481175} 2024-04-26T00:28:07.691 INFO:tasks.rados.rados.0.smithi045.stdout:1338: writing smithi04526579-16 from 470728 to 875522 tid 1 2024-04-26T00:28:07.712 INFO:tasks.rados.rados.0.smithi045.stdout:1338: writing smithi04526579-16 from 1526454 to 2007629 tid 2 2024-04-26T00:28:07.712 INFO:tasks.rados.rados.0.smithi045.stdout:1339: rollback oid 37 current snap is 155 2024-04-26T00:28:07.712 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 37 to 127 2024-04-26T00:28:07.712 INFO:tasks.rados.rados.0.smithi045.stdout:1336: finishing rollback tid 0 to smithi04526579-33 2024-04-26T00:28:07.712 INFO:tasks.rados.rados.0.smithi045.stdout:1336: finishing rollback tid 1 to smithi04526579-33 2024-04-26T00:28:07.712 INFO:tasks.rados.rados.0.smithi045.stdout:1336: finishing rollback tid 2 to smithi04526579-33 2024-04-26T00:28:07.712 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 1039 (ObjNum 445 snap 139 seq_num 445) dirty exists 2024-04-26T00:28:07.712 INFO:tasks.rados.rados.0.smithi045.stdout:1338: finishing write tid 1 to smithi04526579-16 2024-04-26T00:28:07.712 INFO:tasks.rados.rados.0.smithi045.stdout:1336: done (2 left) 2024-04-26T00:28:07.712 INFO:tasks.rados.rados.0.smithi045.stdout:1340: rollback oid 46 current snap is 155 2024-04-26T00:28:07.713 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 46 to 133 2024-04-26T00:28:07.713 INFO:tasks.rados.rados.0.smithi045.stdout:1341: rollback oid 7 current snap is 155 2024-04-26T00:28:07.713 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 7 to 143 2024-04-26T00:28:07.713 INFO:tasks.rados.rados.0.smithi045.stdout:1342: copy_from oid 35 from oid 41 current snap is 155 2024-04-26T00:28:07.713 INFO:tasks.rados.rados.0.smithi045.stdout:1343: write oid 24 current snap is 155 2024-04-26T00:28:07.713 INFO:tasks.rados.rados.0.smithi045.stdout:1343: seq_num 505 ranges {487535=480043,1584936=624833} 2024-04-26T00:28:07.728 INFO:tasks.rados.rados.0.smithi045.stdout:1343: writing smithi04526579-24 from 487535 to 967578 tid 1 2024-04-26T00:28:07.745 INFO:tasks.rados.rados.0.smithi045.stdout:1343: writing smithi04526579-24 from 1584936 to 2209769 tid 2 2024-04-26T00:28:07.745 INFO:tasks.rados.rados.0.smithi045.stdout:1344: read oid 49 snap 149 2024-04-26T00:28:07.745 INFO:tasks.rados.rados.0.smithi045.stdout:1344: expect (ObjNum 316 snap 93 seq_num 316) 2024-04-26T00:28:07.745 INFO:tasks.rados.rados.0.smithi045.stdout:1345: copy_from oid 45 from oid 36 current snap is 155 2024-04-26T00:28:07.745 INFO:tasks.rados.rados.0.smithi045.stdout:1342: finishing copy_from to smithi04526579-35 2024-04-26T00:28:07.745 INFO:tasks.rados.rados.0.smithi045.stdout:1342: got expected ENOENT (src dne) 2024-04-26T00:28:07.745 INFO:tasks.rados.rados.0.smithi045.stdout:1342: finishing copy_from racing read to smithi04526579-35 2024-04-26T00:28:07.745 INFO:tasks.rados.rados.0.smithi045.stdout:1340: finishing rollback tid 0 to smithi04526579-46 2024-04-26T00:28:07.745 INFO:tasks.rados.rados.0.smithi045.stdout:1338: finishing write tid 2 to smithi04526579-16 2024-04-26T00:28:07.745 INFO:tasks.rados.rados.0.smithi045.stdout:1338: finishing write tid 3 to smithi04526579-16 2024-04-26T00:28:07.745 INFO:tasks.rados.rados.0.smithi045.stdout:1338: finishing write tid 4 to smithi04526579-16 2024-04-26T00:28:07.746 INFO:tasks.rados.rados.0.smithi045.stdout:1338: oid 16 updating version 0 to 613 2024-04-26T00:28:07.746 INFO:tasks.rados.rados.0.smithi045.stdout:1338: oid 16 updating version 613 to 615 2024-04-26T00:28:07.746 INFO:tasks.rados.rados.0.smithi045.stdout:1338: oid 16 version 615 is already newer than 614 2024-04-26T00:28:07.746 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 615 (ObjNum 504 snap 155 seq_num 504) dirty exists 2024-04-26T00:28:07.746 INFO:tasks.rados.rados.0.smithi045.stdout:1338: left oid 16 (ObjNum 504 snap 155 seq_num 504) 2024-04-26T00:28:07.746 INFO:tasks.rados.rados.0.smithi045.stdout:1340: finishing rollback tid 1 to smithi04526579-46 2024-04-26T00:28:07.746 INFO:tasks.rados.rados.0.smithi045.stdout:1339: finishing rollback tid 0 to smithi04526579-37 2024-04-26T00:28:07.746 INFO:tasks.rados.rados.0.smithi045.stdout:1340: finishing rollback tid 2 to smithi04526579-46 2024-04-26T00:28:07.746 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 743 (ObjNum 329 snap 96 seq_num 329) dirty exists 2024-04-26T00:28:07.746 INFO:tasks.rados.rados.0.smithi045.stdout:1339: finishing rollback tid 1 to smithi04526579-37 2024-04-26T00:28:07.746 INFO:tasks.rados.rados.0.smithi045.stdout:1339: finishing rollback tid 2 to smithi04526579-37 2024-04-26T00:28:07.746 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 1053 (ObjNum 215 snap 55 seq_num 215) dirty exists 2024-04-26T00:28:07.746 INFO:tasks.rados.rados.0.smithi045.stdout:1341: finishing rollback tid 1 to smithi04526579-7 2024-04-26T00:28:07.746 INFO:tasks.rados.rados.0.smithi045.stdout:1341: finishing rollback tid 2 to smithi04526579-7 2024-04-26T00:28:07.747 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 1055 (ObjNum 359 snap 107 seq_num 359) dirty exists 2024-04-26T00:28:07.747 INFO:tasks.rados.rados.0.smithi045.stdout:1343: finishing write tid 1 to smithi04526579-24 2024-04-26T00:28:07.747 INFO:tasks.rados.rados.0.smithi045.stdout:1338: done (7 left) 2024-04-26T00:28:07.747 INFO:tasks.rados.rados.0.smithi045.stdout:1339: done (6 left) 2024-04-26T00:28:07.747 INFO:tasks.rados.rados.0.smithi045.stdout:1340: done (5 left) 2024-04-26T00:28:07.747 INFO:tasks.rados.rados.0.smithi045.stdout:1341: done (4 left) 2024-04-26T00:28:07.747 INFO:tasks.rados.rados.0.smithi045.stdout:1342: done (3 left) 2024-04-26T00:28:07.747 INFO:tasks.rados.rados.0.smithi045.stdout:1346: read oid 30 snap -1 2024-04-26T00:28:07.747 INFO:tasks.rados.rados.0.smithi045.stdout:1346: expect (ObjNum 447 snap 139 seq_num 447) 2024-04-26T00:28:07.747 INFO:tasks.rados.rados.0.smithi045.stdout:1347: rollback oid 42 current snap is 155 2024-04-26T00:28:07.747 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 42 to 140 2024-04-26T00:28:07.747 INFO:tasks.rados.rados.0.smithi045.stdout:1348: read oid 22 snap -1 2024-04-26T00:28:07.747 INFO:tasks.rados.rados.0.smithi045.stdout:1348: expect deleted 2024-04-26T00:28:07.747 INFO:tasks.rados.rados.0.smithi045.stdout:1349: snap_remove snap 151 2024-04-26T00:28:08.682 INFO:tasks.rados.rados.0.smithi045.stdout:1349: done (6 left) 2024-04-26T00:28:08.682 INFO:tasks.rados.rados.0.smithi045.stdout:1350: read oid 38 snap -1 2024-04-26T00:28:08.682 INFO:tasks.rados.rados.0.smithi045.stdout:1350: expect (ObjNum 356 snap 105 seq_num 356) 2024-04-26T00:28:08.682 INFO:tasks.rados.rados.0.smithi045.stdout:1343: finishing write tid 2 to smithi04526579-24 2024-04-26T00:28:08.682 INFO:tasks.rados.rados.0.smithi045.stdout:1347: finishing rollback tid 0 to smithi04526579-42 2024-04-26T00:28:08.683 INFO:tasks.rados.rados.0.smithi045.stdout:1347: finishing rollback tid 1 to smithi04526579-42 2024-04-26T00:28:08.683 INFO:tasks.rados.rados.0.smithi045.stdout:1346: expect (ObjNum 447 snap 139 seq_num 447) 2024-04-26T00:28:08.881 INFO:tasks.rados.rados.0.smithi045.stdout:1347: finishing rollback tid 2 to smithi04526579-42 2024-04-26T00:28:08.881 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 780 (ObjNum 434 snap 136 seq_num 434) dirty exists 2024-04-26T00:28:08.881 INFO:tasks.rados.rados.0.smithi045.stdout:1343: finishing write tid 3 to smithi04526579-24 2024-04-26T00:28:08.881 INFO:tasks.rados.rados.0.smithi045.stdout:1343: finishing write tid 4 to smithi04526579-24 2024-04-26T00:28:08.881 INFO:tasks.rados.rados.0.smithi045.stdout:1343: oid 24 updating version 0 to 1061 2024-04-26T00:28:08.881 INFO:tasks.rados.rados.0.smithi045.stdout:1343: oid 24 updating version 1061 to 1063 2024-04-26T00:28:08.882 INFO:tasks.rados.rados.0.smithi045.stdout:1343: oid 24 version 1063 is already newer than 1062 2024-04-26T00:28:08.882 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 1063 (ObjNum 505 snap 155 seq_num 505) dirty exists 2024-04-26T00:28:08.882 INFO:tasks.rados.rados.0.smithi045.stdout:1343: left oid 24 (ObjNum 505 snap 155 seq_num 505) 2024-04-26T00:28:08.882 INFO:tasks.rados.rados.0.smithi045.stdout:1344: expect (ObjNum 316 snap 93 seq_num 316) 2024-04-26T00:28:09.105 INFO:tasks.rados.rados.0.smithi045.stdout:1345: finishing copy_from racing read to smithi04526579-45 2024-04-26T00:28:09.105 INFO:tasks.rados.rados.0.smithi045.stdout:1345: finishing copy_from to smithi04526579-45 2024-04-26T00:28:09.105 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 740 (ObjNum 482 snap 150 seq_num 482) dirty exists 2024-04-26T00:28:09.105 INFO:tasks.rados.rados.0.smithi045.stdout:1343: done (6 left) 2024-04-26T00:28:09.105 INFO:tasks.rados.rados.0.smithi045.stdout:1344: done (5 left) 2024-04-26T00:28:09.105 INFO:tasks.rados.rados.0.smithi045.stdout:1345: done (4 left) 2024-04-26T00:28:09.105 INFO:tasks.rados.rados.0.smithi045.stdout:1346: done (3 left) 2024-04-26T00:28:09.105 INFO:tasks.rados.rados.0.smithi045.stdout:1347: done (2 left) 2024-04-26T00:28:09.105 INFO:tasks.rados.rados.0.smithi045.stdout:1348: done (1 left) 2024-04-26T00:28:09.105 INFO:tasks.rados.rados.0.smithi045.stdout:1351: write oid 19 current snap is 155 2024-04-26T00:28:09.105 INFO:tasks.rados.rados.0.smithi045.stdout:1351: seq_num 506 ranges {504342=555292,1643418=484844} 2024-04-26T00:28:09.122 INFO:tasks.rados.rados.0.smithi045.stdout:1351: writing smithi04526579-19 from 504342 to 1059634 tid 1 2024-04-26T00:28:09.140 INFO:tasks.rados.rados.0.smithi045.stdout:1351: writing smithi04526579-19 from 1643418 to 2128262 tid 2 2024-04-26T00:28:09.140 INFO:tasks.rados.rados.0.smithi045.stdout:1352: snap_create 2024-04-26T00:28:09.140 INFO:tasks.rados.rados.0.smithi045.stdout:1350: expect (ObjNum 356 snap 105 seq_num 356) 2024-04-26T00:28:09.371 INFO:tasks.rados.rados.0.smithi045.stdout:1351: finishing write tid 1 to smithi04526579-19 2024-04-26T00:28:09.371 INFO:tasks.rados.rados.0.smithi045.stdout:1351: finishing write tid 2 to smithi04526579-19 2024-04-26T00:28:09.372 INFO:tasks.rados.rados.0.smithi045.stdout:1351: finishing write tid 3 to smithi04526579-19 2024-04-26T00:28:09.372 INFO:tasks.rados.rados.0.smithi045.stdout:1351: finishing write tid 4 to smithi04526579-19 2024-04-26T00:28:09.372 INFO:tasks.rados.rados.0.smithi045.stdout:1351: oid 19 updating version 0 to 784 2024-04-26T00:28:09.372 INFO:tasks.rados.rados.0.smithi045.stdout:1351: oid 19 updating version 784 to 785 2024-04-26T00:28:09.372 INFO:tasks.rados.rados.0.smithi045.stdout:1351: oid 19 updating version 785 to 786 2024-04-26T00:28:09.372 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 786 (ObjNum 506 snap 155 seq_num 506) dirty exists 2024-04-26T00:28:09.372 INFO:tasks.rados.rados.0.smithi045.stdout:1351: left oid 19 (ObjNum 506 snap 155 seq_num 506) 2024-04-26T00:28:09.372 INFO:tasks.rados.rados.0.smithi045.stdout:1350: done (2 left) 2024-04-26T00:28:09.372 INFO:tasks.rados.rados.0.smithi045.stdout:1351: done (1 left) 2024-04-26T00:28:09.372 INFO:tasks.rados.rados.0.smithi045.stdout:1352: done (0 left) 2024-04-26T00:28:09.372 INFO:tasks.rados.rados.0.smithi045.stdout:1353: snap_remove snap 142 2024-04-26T00:28:10.260 INFO:tasks.rados.rados.0.smithi045.stdout:1353: done (0 left) 2024-04-26T00:28:10.260 INFO:tasks.rados.rados.0.smithi045.stdout:1354: read oid 40 snap -1 2024-04-26T00:28:10.260 INFO:tasks.rados.rados.0.smithi045.stdout:1354: expect deleted 2024-04-26T00:28:10.260 INFO:tasks.rados.rados.0.smithi045.stdout:1355: write oid 25 current snap is 156 2024-04-26T00:28:10.260 INFO:tasks.rados.rados.0.smithi045.stdout:1355: seq_num 507 ranges {521149=630541,1701900=628502} 2024-04-26T00:28:10.282 INFO:tasks.rados.rados.0.smithi045.stdout:1355: writing smithi04526579-25 from 521149 to 1151690 tid 1 2024-04-26T00:28:10.301 INFO:tasks.rados.rados.0.smithi045.stdout:1355: writing smithi04526579-25 from 1701900 to 2330402 tid 2 2024-04-26T00:28:10.301 INFO:tasks.rados.rados.0.smithi045.stdout:1356: snap_remove snap 138 2024-04-26T00:28:11.264 INFO:tasks.rados.rados.0.smithi045.stdout:1356: done (2 left) 2024-04-26T00:28:11.265 INFO:tasks.rados.rados.0.smithi045.stdout:1357: snap_create 2024-04-26T00:28:11.265 INFO:tasks.rados.rados.0.smithi045.stdout:1355: finishing write tid 1 to smithi04526579-25 2024-04-26T00:28:11.265 INFO:tasks.rados.rados.0.smithi045.stdout:1355: finishing write tid 2 to smithi04526579-25 2024-04-26T00:28:11.265 INFO:tasks.rados.rados.0.smithi045.stdout:1355: finishing write tid 3 to smithi04526579-25 2024-04-26T00:28:11.265 INFO:tasks.rados.rados.0.smithi045.stdout:1355: finishing write tid 4 to smithi04526579-25 2024-04-26T00:28:11.265 INFO:tasks.rados.rados.0.smithi045.stdout:1355: oid 25 updating version 0 to 1080 2024-04-26T00:28:11.265 INFO:tasks.rados.rados.0.smithi045.stdout:1355: oid 25 version 1080 is already newer than 1079 2024-04-26T00:28:11.265 INFO:tasks.rados.rados.0.smithi045.stdout:1355: oid 25 version 1080 is already newer than 1076 2024-04-26T00:28:11.265 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 1080 (ObjNum 507 snap 156 seq_num 507) dirty exists 2024-04-26T00:28:11.265 INFO:tasks.rados.rados.0.smithi045.stdout:1355: left oid 25 (ObjNum 507 snap 156 seq_num 507) 2024-04-26T00:28:12.266 INFO:tasks.rados.rados.0.smithi045.stdout:1354: done (2 left) 2024-04-26T00:28:12.266 INFO:tasks.rados.rados.0.smithi045.stdout:1355: done (1 left) 2024-04-26T00:28:12.266 INFO:tasks.rados.rados.0.smithi045.stdout:1357: done (0 left) 2024-04-26T00:28:12.266 INFO:tasks.rados.rados.0.smithi045.stdout:1358: write (excl) oid 39 current snap is 157 2024-04-26T00:28:12.266 INFO:tasks.rados.rados.0.smithi045.stdout:1358: seq_num 508 ranges {537956=705790,1644029=772160} 2024-04-26T00:28:12.286 INFO:tasks.rados.rados.0.smithi045.stdout:1358: writing smithi04526579-39 from 537956 to 1243746 tid 1 2024-04-26T00:28:12.306 INFO:tasks.rados.rados.0.smithi045.stdout:1358: writing smithi04526579-39 from 1644029 to 2416189 tid 2 2024-04-26T00:28:12.306 INFO:tasks.rados.rados.0.smithi045.stdout:1359: read oid 48 snap -1 2024-04-26T00:28:12.306 INFO:tasks.rados.rados.0.smithi045.stdout:1359: expect (ObjNum 359 snap 107 seq_num 359) 2024-04-26T00:28:12.306 INFO:tasks.rados.rados.0.smithi045.stdout:1360: write (excl) oid 8 current snap is 157 2024-04-26T00:28:12.306 INFO:tasks.rados.rados.0.smithi045.stdout:1358: finishing write tid 1 to smithi04526579-39 2024-04-26T00:28:12.306 INFO:tasks.rados.rados.0.smithi045.stdout:1360: seq_num 509 ranges {554763=781039,2102511=452252} 2024-04-26T00:28:12.329 INFO:tasks.rados.rados.0.smithi045.stdout:1360: writing smithi04526579-8 from 554763 to 1335802 tid 1 2024-04-26T00:28:12.346 INFO:tasks.rados.rados.0.smithi045.stdout:1360: writing smithi04526579-8 from 2102511 to 2554763 tid 2 2024-04-26T00:28:12.347 INFO:tasks.rados.rados.0.smithi045.stdout:1361: read oid 19 snap -1 2024-04-26T00:28:12.347 INFO:tasks.rados.rados.0.smithi045.stdout:1361: expect (ObjNum 506 snap 155 seq_num 506) 2024-04-26T00:28:12.347 INFO:tasks.rados.rados.0.smithi045.stdout:1362: write oid 2 current snap is 157 2024-04-26T00:28:12.347 INFO:tasks.rados.rados.0.smithi045.stdout:1358: finishing write tid 2 to smithi04526579-39 2024-04-26T00:28:12.347 INFO:tasks.rados.rados.0.smithi045.stdout:1358: finishing write tid 3 to smithi04526579-39 2024-04-26T00:28:12.347 INFO:tasks.rados.rados.0.smithi045.stdout:1358: finishing write tid 4 to smithi04526579-39 2024-04-26T00:28:12.347 INFO:tasks.rados.rados.0.smithi045.stdout:1358: oid 39 updating version 0 to 511 2024-04-26T00:28:12.347 INFO:tasks.rados.rados.0.smithi045.stdout:1358: oid 39 version 511 is already newer than 510 2024-04-26T00:28:12.347 INFO:tasks.rados.rados.0.smithi045.stdout:1358: oid 39 updating version 511 to 512 2024-04-26T00:28:12.347 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 512 (ObjNum 508 snap 157 seq_num 508) dirty exists 2024-04-26T00:28:12.347 INFO:tasks.rados.rados.0.smithi045.stdout:1358: left oid 39 (ObjNum 508 snap 157 seq_num 508) 2024-04-26T00:28:12.347 INFO:tasks.rados.rados.0.smithi045.stdout:1362: seq_num 510 ranges {571570=572641,1877346=694224} 2024-04-26T00:28:12.362 INFO:tasks.rados.rados.0.smithi045.stdout:1362: writing smithi04526579-2 from 571570 to 1144211 tid 1 2024-04-26T00:28:12.379 INFO:tasks.rados.rados.0.smithi045.stdout:1362: writing smithi04526579-2 from 1877346 to 2571570 tid 2 2024-04-26T00:28:12.379 INFO:tasks.rados.rados.0.smithi045.stdout:1358: done (4 left) 2024-04-26T00:28:12.379 INFO:tasks.rados.rados.0.smithi045.stdout:1363: read oid 39 snap 108 2024-04-26T00:28:12.379 INFO:tasks.rados.rados.0.smithi045.stdout:1363: expect (ObjNum 350 snap 105 seq_num 350) 2024-04-26T00:28:12.380 INFO:tasks.rados.rados.0.smithi045.stdout:1364: read oid 41 snap -1 2024-04-26T00:28:12.380 INFO:tasks.rados.rados.0.smithi045.stdout:1364: expect deleted 2024-04-26T00:28:12.380 INFO:tasks.rados.rados.0.smithi045.stdout:1365: copy_from oid 3 from oid 21 current snap is 157 2024-04-26T00:28:12.380 INFO:tasks.rados.rados.0.smithi045.stdout:1359: expect (ObjNum 359 snap 107 seq_num 359) 2024-04-26T00:28:12.520 INFO:tasks.rados.rados.0.smithi045.stdout:1360: finishing write tid 1 to smithi04526579-8 2024-04-26T00:28:12.520 INFO:tasks.rados.rados.0.smithi045.stdout:1360: finishing write tid 2 to smithi04526579-8 2024-04-26T00:28:12.520 INFO:tasks.rados.rados.0.smithi045.stdout:1360: finishing write tid 3 to smithi04526579-8 2024-04-26T00:28:12.520 INFO:tasks.rados.rados.0.smithi045.stdout:1360: finishing write tid 4 to smithi04526579-8 2024-04-26T00:28:12.520 INFO:tasks.rados.rados.0.smithi045.stdout:1360: oid 8 updating version 0 to 485 2024-04-26T00:28:12.520 INFO:tasks.rados.rados.0.smithi045.stdout:1360: oid 8 version 485 is already newer than 483 2024-04-26T00:28:12.520 INFO:tasks.rados.rados.0.smithi045.stdout:1360: oid 8 version 485 is already newer than 484 2024-04-26T00:28:12.520 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 485 (ObjNum 509 snap 157 seq_num 509) dirty exists 2024-04-26T00:28:12.521 INFO:tasks.rados.rados.0.smithi045.stdout:1360: left oid 8 (ObjNum 509 snap 157 seq_num 509) 2024-04-26T00:28:12.521 INFO:tasks.rados.rados.0.smithi045.stdout:1361: expect (ObjNum 506 snap 155 seq_num 506) 2024-04-26T00:28:12.648 INFO:tasks.rados.rados.0.smithi045.stdout:1362: finishing write tid 1 to smithi04526579-2 2024-04-26T00:28:12.648 INFO:tasks.rados.rados.0.smithi045.stdout:1365: finishing copy_from to smithi04526579-3 2024-04-26T00:28:12.648 INFO:tasks.rados.rados.0.smithi045.stdout:1365: got expected ENOENT (src dne) 2024-04-26T00:28:12.648 INFO:tasks.rados.rados.0.smithi045.stdout:1365: finishing copy_from racing read to smithi04526579-3 2024-04-26T00:28:12.648 INFO:tasks.rados.rados.0.smithi045.stdout:1362: finishing write tid 2 to smithi04526579-2 2024-04-26T00:28:12.648 INFO:tasks.rados.rados.0.smithi045.stdout:1362: finishing write tid 3 to smithi04526579-2 2024-04-26T00:28:12.648 INFO:tasks.rados.rados.0.smithi045.stdout:1362: finishing write tid 4 to smithi04526579-2 2024-04-26T00:28:12.648 INFO:tasks.rados.rados.0.smithi045.stdout:1362: oid 2 updating version 0 to 775 2024-04-26T00:28:12.648 INFO:tasks.rados.rados.0.smithi045.stdout:1362: oid 2 updating version 775 to 776 2024-04-26T00:28:12.648 INFO:tasks.rados.rados.0.smithi045.stdout:1362: oid 2 version 776 is already newer than 774 2024-04-26T00:28:12.649 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 776 (ObjNum 510 snap 157 seq_num 510) dirty exists 2024-04-26T00:28:12.649 INFO:tasks.rados.rados.0.smithi045.stdout:1362: left oid 2 (ObjNum 510 snap 157 seq_num 510) 2024-04-26T00:28:12.649 INFO:tasks.rados.rados.0.smithi045.stdout:1363: expect (ObjNum 350 snap 105 seq_num 350) 2024-04-26T00:28:12.705 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-04-26T00:28:12.705 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-04-26T00:28:12.705 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-26T00:28:12.705 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-26T00:28:12.708 INFO:tasks.ceph.mon.b:Started 2024-04-26T00:28:12.708 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:28:12.708 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:28:12.830 INFO:tasks.rados.rados.0.smithi045.stdout:1359: done (6 left) 2024-04-26T00:28:12.830 INFO:tasks.rados.rados.0.smithi045.stdout:1360: done (5 left) 2024-04-26T00:28:12.830 INFO:tasks.rados.rados.0.smithi045.stdout:1361: done (4 left) 2024-04-26T00:28:12.830 INFO:tasks.rados.rados.0.smithi045.stdout:1362: done (3 left) 2024-04-26T00:28:12.830 INFO:tasks.rados.rados.0.smithi045.stdout:1363: done (2 left) 2024-04-26T00:28:12.830 INFO:tasks.rados.rados.0.smithi045.stdout:1364: done (1 left) 2024-04-26T00:28:12.830 INFO:tasks.rados.rados.0.smithi045.stdout:1365: done (0 left) 2024-04-26T00:28:12.830 INFO:tasks.rados.rados.0.smithi045.stdout:1366: write oid 15 current snap is 157 2024-04-26T00:28:12.830 INFO:tasks.rados.rados.0.smithi045.stdout:1366: seq_num 511 ranges {588377=647890,1935828=635840} 2024-04-26T00:28:12.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.855+0000 7f18501c9640 10 monclient: get_monmap_and_config 2024-04-26T00:28:12.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.855+0000 7f18501c9640 10 monclient: build_initial_monmap 2024-04-26T00:28:12.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.855+0000 7f18501c9640 10 monclient: monmap: 2024-04-26T00:28:12.841 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:28:12.841 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:12.841 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:28:12.857636+0000 2024-04-26T00:28:12.841 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:28:12.857636+0000 2024-04-26T00:28:12.841 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:28:12.842 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:12.842 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:28:12.842 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:28:12.842 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:28:12.842 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:28:12.842 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:28:12.842 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:12.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.855+0000 7f18501c9640 1 Processor -- start 2024-04-26T00:28:12.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.855+0000 7f18501c9640 1 -- start start 2024-04-26T00:28:12.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.855+0000 7f18501c9640 10 monclient: init 2024-04-26T00:28:12.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.855+0000 7f18501c9640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:12.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.855+0000 7f18501c9640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-04-26T00:28:12.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.855+0000 7f18501c9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f18480a0260 addr v1:172.21.15.45:6789/0 2024-04-26T00:28:12.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.855+0000 7f18501c9640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1848137e40 0x7f184813a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:12.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.855+0000 7f18501c9640 10 monclient(hunting): picked mon.noname-b con 0x7f1848137e40 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:28:12.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.855+0000 7f18501c9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f184813a770 addr v1:172.21.15.37:6789/0 2024-04-26T00:28:12.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.855+0000 7f18501c9640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18480a1130 con 0x7f184813a770 2024-04-26T00:28:12.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.855+0000 7f18501c9640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f184806cff0 con 0x7f18480a0260 2024-04-26T00:28:12.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.855+0000 7f18501c9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:12.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.855+0000 7f18501c9640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f18480a0d50 con 0x7f1848137e40 2024-04-26T00:28:12.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.855+0000 7f18501c9640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:12.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.855+0000 7f18501c9640 10 monclient(hunting): authenticate will time out at 1474.163524s 2024-04-26T00:28:12.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.855+0000 7f184d764640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1848137e40 0x7f184813a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:12.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.855+0000 7f184d764640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1848137e40 0x7f184813a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:58090/0 (socket says 172.21.15.37:58090) 2024-04-26T00:28:12.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.855+0000 7f184d764640 1 -- 172.21.15.37:0/3892311229 learned_addr learned my addr 172.21.15.37:0/3892311229 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:28:12.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.855+0000 7f184d764640 10 monclient(hunting): get_auth_request con 0x7f1848137e40 auth_method 0 2024-04-26T00:28:12.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.855+0000 7f184d764640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:12.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.855+0000 7f184d764640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:12.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.855+0000 7f184d764640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:12.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.855+0000 7f184e766640 1 --1- 172.21.15.37:0/3892311229 >> v1:172.21.15.37:6789/0 conn(0x7f184813a770 0x7f184813cb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:47080/0 (socket says 172.21.15.37:47080) 2024-04-26T00:28:12.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.855+0000 7f184d764640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:12.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.855+0000 7f184d764640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:12.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.855+0000 7f184d764640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:12.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.859+0000 7f184d764640 10 monclient(hunting): handle_auth_done global_id 84269 payload 274 2024-04-26T00:28:12.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.859+0000 7f184d764640 1 -- 172.21.15.37:0/3892311229 >> v1:172.21.15.45:6789/0 conn(0x7f18480a0260 legacy=0x7f18480a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:28:12.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.859+0000 7f184d764640 1 -- 172.21.15.37:0/3892311229 >> v1:172.21.15.37:6789/0 conn(0x7f184813a770 legacy=0x7f184813cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:28:12.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.859+0000 7f184d764640 10 monclient: _finish_hunting 0 2024-04-26T00:28:12.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.859+0000 7f184d764640 1 monclient: found mon.noname-b 2024-04-26T00:28:12.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.859+0000 7f184d764640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:12.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.859+0000 7f184d764640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:28:12.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.859+0000 7f184d764640 1 -- 172.21.15.37:0/3892311229 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f184809bc50 con 0x7f1848137e40 2024-04-26T00:28:12.847 INFO:tasks.rados.rados.0.smithi045.stdout:1366: writing smithi04526579-15 from 588377 to 1236267 tid 1 2024-04-26T00:28:12.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184d764640 1 -- 172.21.15.37:0/3892311229 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1848137e40 msgr2=0x7f184813a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:12.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184d764640 1 --2- 172.21.15.37:0/3892311229 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1848137e40 0x7f184813a230 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f183c0098a0 tx=0x7f183c01c1e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=231f9f2f5599b7e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:12.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184cf63640 1 -- 172.21.15.37:0/3892311229 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f183c0b0070 con 0x7f1848137e40 2024-04-26T00:28:12.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184cf63640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:12.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184cf63640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:28:12.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184cf63640 10 monclient: dump: 2024-04-26T00:28:12.849 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:12.849 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:12.849 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:12.849 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:12.849 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:12.849 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:12.849 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:12.849 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:12.849 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:12.849 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:12.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184cf63640 10 monclient: _finish_auth 0 2024-04-26T00:28:12.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184cf63640 10 monclient: _check_auth_tickets 2024-04-26T00:28:12.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184cf63640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:12.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184cf63640 1 -- 172.21.15.37:0/3892311229 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f183c0ab600 con 0x7f1848137e40 2024-04-26T00:28:12.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184cf63640 10 monclient: handle_config config(0 keys) 2024-04-26T00:28:12.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184cf63640 1 -- 172.21.15.37:0/3892311229 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f183c0ab9d0 con 0x7f1848137e40 2024-04-26T00:28:12.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f18501c9640 5 monclient: authenticate success, global_id 84269 2024-04-26T00:28:12.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f18501c9640 10 monclient: get_monmap_and_config success 2024-04-26T00:28:12.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f18501c9640 4 set_mon_vals no callback set 2024-04-26T00:28:12.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184cf63640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:12.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184cf63640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:28:12.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184cf63640 10 monclient: dump: 2024-04-26T00:28:12.850 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:12.850 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:12.850 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:12.850 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:12.850 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:12.851 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:12.851 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:12.851 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:12.851 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:12.851 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:12.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f18501c9640 10 monclient: shutdown 2024-04-26T00:28:12.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f18501c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:28:12.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f18501c9640 1 -- 172.21.15.37:0/3892311229 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1848137e40 msgr2=0x7f184813a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:12.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f18501c9640 1 --2- 172.21.15.37:0/3892311229 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1848137e40 0x7f184813a230 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f183c0098a0 tx=0x7f183c01c1e0 comp rx=0 tx=0).stop 2024-04-26T00:28:12.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f18501c9640 1 -- 172.21.15.37:0/3892311229 shutdown_connections 2024-04-26T00:28:12.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f18501c9640 1 --2- 172.21.15.37:0/3892311229 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1848137e40 0x7f184813a230 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:12.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f18501c9640 1 -- 172.21.15.37:0/3892311229 >> 172.21.15.37:0/3892311229 conn(0x7f18480a1ad0 msgr2=0x7f18480a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:28:12.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f18501c9640 1 -- 172.21.15.37:0/3892311229 shutdown_connections 2024-04-26T00:28:12.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f18501c9640 1 -- 172.21.15.37:0/3892311229 wait complete. 2024-04-26T00:28:12.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f18501c9640 10 monclient: build_initial_monmap 2024-04-26T00:28:12.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f18501c9640 10 monclient: monmap: 2024-04-26T00:28:12.857 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:28:12.857 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:28:12.857 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:28:12.857 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:28:12.857 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:28:12.857 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:28:12.857 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:28:12.857 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:28:12.857 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:28:12.858 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:12.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f18501c9640 1 Processor -- start 2024-04-26T00:28:12.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f18501c9640 1 -- start start 2024-04-26T00:28:12.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f18501c9640 10 monclient: init 2024-04-26T00:28:12.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f18501c9640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:12.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f18501c9640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:28:12.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f18501c9640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f18480a0260 0x7f184811b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:12.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f18501c9640 10 monclient(hunting): picked mon.noname-c con 0x7f18480a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:28:12.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f18501c9640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1848137e40 0x7f18480712d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:12.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f18501c9640 10 monclient(hunting): picked mon.noname-a con 0x7f1848137e40 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:28:12.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f18501c9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f184813a770 0x7f1848134b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:12.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f18501c9640 10 monclient(hunting): picked mon.noname-b con 0x7f184813a770 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:28:12.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f18501c9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:12.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f18501c9640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f184809f0f0 con 0x7f1848137e40 2024-04-26T00:28:12.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f18501c9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:12.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f18501c9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f184809f4c0 con 0x7f184813a770 2024-04-26T00:28:12.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f18501c9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:12.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f18501c9640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f18480a00a0 con 0x7f18480a0260 2024-04-26T00:28:12.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f18501c9640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:12.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f18501c9640 10 monclient(hunting): authenticate will time out at 1474.170758s 2024-04-26T00:28:12.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184d764640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1848137e40 0x7f18480712d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:12.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184df65640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f18480a0260 0x7f184811b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:12.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184d764640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1848137e40 0x7f18480712d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52284/0 (socket says 172.21.15.37:52284) 2024-04-26T00:28:12.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184df65640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f18480a0260 0x7f184811b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:58098/0 (socket says 172.21.15.37:58098) 2024-04-26T00:28:12.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184d764640 1 -- 172.21.15.37:0/2358837620 learned_addr learned my addr 172.21.15.37:0/2358837620 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:28:12.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184d764640 10 monclient(hunting): get_auth_request con 0x7f1848137e40 auth_method 0 2024-04-26T00:28:12.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184d764640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:12.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184d764640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:12.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184d764640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:12.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184df65640 10 monclient(hunting): get_auth_request con 0x7f18480a0260 auth_method 0 2024-04-26T00:28:12.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184df65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:12.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184df65640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:12.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184df65640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:12.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184d764640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:12.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184d764640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:12.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184d764640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:12.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184e766640 1 --2- 172.21.15.37:0/2358837620 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f184813a770 0x7f1848134b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:12.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184df65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:12.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184df65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:12.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184df65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:12.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184e766640 10 monclient(hunting): get_auth_request con 0x7f184813a770 auth_method 0 2024-04-26T00:28:12.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184e766640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:12.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184e766640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:12.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184e766640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:12.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184d764640 10 monclient(hunting): handle_auth_done global_id 94176 payload 824 2024-04-26T00:28:12.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184d764640 1 -- 172.21.15.37:0/2358837620 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f18480a0260 msgr2=0x7f184811b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:12.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184d764640 1 --2- 172.21.15.37:0/2358837620 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f18480a0260 0x7f184811b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:12.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184d764640 1 -- 172.21.15.37:0/2358837620 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f184813a770 msgr2=0x7f1848134b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:12.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184d764640 1 --2- 172.21.15.37:0/2358837620 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f184813a770 0x7f1848134b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:12.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184d764640 10 monclient: _finish_hunting 0 2024-04-26T00:28:12.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184d764640 1 monclient: found mon.noname-a 2024-04-26T00:28:12.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184d764640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:12.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184d764640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:28:12.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.863+0000 7f184d764640 1 -- 172.21.15.37:0/2358837620 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1848071f10 con 0x7f1848137e40 2024-04-26T00:28:12.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.867+0000 7f184d764640 1 -- 172.21.15.37:0/2358837620 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1848137e40 msgr2=0x7f18480712d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:12.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.867+0000 7f184d764640 1 --2- 172.21.15.37:0/2358837620 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1848137e40 0x7f18480712d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f183c01a890 tx=0x7f183c018f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:12.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.867+0000 7f18467fc640 1 -- 172.21.15.37:0/2358837620 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f183c0b9020 con 0x7f1848137e40 2024-04-26T00:28:12.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.867+0000 7f18467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:12.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.867+0000 7f18467fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:28:12.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.867+0000 7f18467fc640 10 monclient: dump: 2024-04-26T00:28:12.862 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:12.862 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:12.862 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:12.862 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:12.862 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:12.862 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:12.862 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:12.862 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:12.863 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:12.863 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:12.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.867+0000 7f18467fc640 10 monclient: _finish_auth 0 2024-04-26T00:28:12.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.867+0000 7f18467fc640 10 monclient: _check_auth_tickets 2024-04-26T00:28:12.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.867+0000 7f18467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:12.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.867+0000 7f18467fc640 1 -- 172.21.15.37:0/2358837620 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f183c0b1040 con 0x7f1848137e40 2024-04-26T00:28:12.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.867+0000 7f18501c9640 5 monclient: authenticate success, global_id 94176 2024-04-26T00:28:12.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.867+0000 7f18467fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:28:12.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.867+0000 7f18467fc640 1 -- 172.21.15.37:0/2358837620 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f183c0b0050 con 0x7f1848137e40 2024-04-26T00:28:12.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.867+0000 7f18467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:12.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.867+0000 7f18467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:28:12.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.867+0000 7f18467fc640 10 monclient: dump: 2024-04-26T00:28:12.863 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:12.863 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:12.863 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:12.864 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:12.864 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:12.864 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:12.864 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:12.864 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:12.864 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:12.864 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:12.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.867+0000 7f18501c9640 10 monclient: _renew_subs 2024-04-26T00:28:12.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.867+0000 7f18501c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:28:12.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.867+0000 7f18501c9640 1 -- 172.21.15.37:0/2358837620 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f184811c590 con 0x7f1848137e40 2024-04-26T00:28:12.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.867+0000 7f1847fff640 4 set_mon_vals no callback set 2024-04-26T00:28:12.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.867+0000 7f18501c9640 10 monclient: _renew_subs 2024-04-26T00:28:12.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.867+0000 7f18501c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:28:12.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.867+0000 7f18501c9640 1 -- 172.21.15.37:0/2358837620 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f184811dd30 con 0x7f1848137e40 2024-04-26T00:28:12.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.867+0000 7f18501c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:12.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.867+0000 7f18501c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:12.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.867+0000 7f18501c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:28:12.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.867+0000 7f18467fc640 1 -- 172.21.15.37:0/2358837620 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f183c0bc020 con 0x7f1848137e40 2024-04-26T00:28:12.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.867+0000 7f18501c9640 1 -- 172.21.15.37:0/2358837620 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1848066f40 con 0x7f1848137e40 2024-04-26T00:28:12.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.867+0000 7f18467fc640 1 --2- 172.21.15.37:0/2358837620 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1834035e80 0x7f1834038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:12.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.867+0000 7f18467fc640 1 -- 172.21.15.37:0/2358837620 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(320..320 src has 177..320) ==== 3963+0+0 (secure 0 0 0) 0x7f183c0e2960 con 0x7f1848137e40 2024-04-26T00:28:12.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.867+0000 7f184df65640 1 --2- 172.21.15.37:0/2358837620 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1834035e80 0x7f1834038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:12.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.867+0000 7f184df65640 10 monclient: get_auth_request con 0x7f1834035e80 auth_method 0 2024-04-26T00:28:12.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.871+0000 7f184df65640 1 --2- 172.21.15.37:0/2358837620 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1834035e80 0x7f1834038340 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1840005680 tx=0x7f1840003dd0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:12.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.875+0000 7f18467fc640 1 -- 172.21.15.37:0/2358837620 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f183c0c3030 con 0x7f1848137e40 2024-04-26T00:28:12.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.875+0000 7f18467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:12.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:12.875+0000 7f18467fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:28:12.867 INFO:tasks.rados.rados.0.smithi045.stdout:1366: writing smithi04526579-15 from 1935828 to 2571668 tid 2 2024-04-26T00:28:12.867 INFO:tasks.rados.rados.0.smithi045.stdout:1367: read oid 36 snap -1 2024-04-26T00:28:12.867 INFO:tasks.rados.rados.0.smithi045.stdout:1367: expect (ObjNum 482 snap 150 seq_num 482) 2024-04-26T00:28:12.867 INFO:tasks.rados.rados.0.smithi045.stdout:1368: rollback oid 47 current snap is 157 2024-04-26T00:28:12.867 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 47 to 150 2024-04-26T00:28:12.867 INFO:tasks.rados.rados.0.smithi045.stdout:1366: finishing write tid 1 to smithi04526579-15 2024-04-26T00:28:12.867 INFO:tasks.rados.rados.0.smithi045.stdout:1369: rollback oid 46 current snap is 157 2024-04-26T00:28:12.867 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 46 to 126 2024-04-26T00:28:12.867 INFO:tasks.rados.rados.0.smithi045.stdout:1370: write (excl) oid 32 current snap is 157 2024-04-26T00:28:12.867 INFO:tasks.rados.rados.0.smithi045.stdout:1370: seq_num 512 ranges {605184=723139,1877957=727227} 2024-04-26T00:28:12.870 INFO:tasks.rados.rados.0.smithi045.stdout:1370: writing smithi04526579-32 from 605184 to 1328323 tid 1 2024-04-26T00:28:12.880 INFO:tasks.rados.rados.0.smithi045.stdout:1370: writing smithi04526579-32 from 1877957 to 2605184 tid 2 2024-04-26T00:28:12.880 INFO:tasks.rados.rados.0.smithi045.stdout:1371: rollback oid 22 current snap is 157 2024-04-26T00:28:12.880 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 22 to 127 2024-04-26T00:28:12.880 INFO:tasks.rados.rados.0.smithi045.stdout:1368: finishing rollback tid 1 to smithi04526579-47 2024-04-26T00:28:12.880 INFO:tasks.rados.rados.0.smithi045.stdout:1367: expect (ObjNum 482 snap 150 seq_num 482) 2024-04-26T00:28:13.021 INFO:tasks.rados.rados.0.smithi045.stdout:1368: finishing rollback tid 2 to smithi04526579-47 2024-04-26T00:28:13.021 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 814 (ObjNum 329 snap 96 seq_num 329) dirty exists 2024-04-26T00:28:13.022 INFO:tasks.rados.rados.0.smithi045.stdout:1366: finishing write tid 2 to smithi04526579-15 2024-04-26T00:28:13.022 INFO:tasks.rados.rados.0.smithi045.stdout:1366: finishing write tid 3 to smithi04526579-15 2024-04-26T00:28:13.022 INFO:tasks.rados.rados.0.smithi045.stdout:1366: finishing write tid 4 to smithi04526579-15 2024-04-26T00:28:13.022 INFO:tasks.rados.rados.0.smithi045.stdout:1366: oid 15 updating version 0 to 1026 2024-04-26T00:28:13.022 INFO:tasks.rados.rados.0.smithi045.stdout:1366: oid 15 version 1026 is already newer than 1025 2024-04-26T00:28:13.022 INFO:tasks.rados.rados.0.smithi045.stdout:1366: oid 15 updating version 1026 to 1027 2024-04-26T00:28:13.022 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 1027 (ObjNum 511 snap 157 seq_num 511) dirty exists 2024-04-26T00:28:13.022 INFO:tasks.rados.rados.0.smithi045.stdout:1366: left oid 15 (ObjNum 511 snap 157 seq_num 511) 2024-04-26T00:28:13.022 INFO:tasks.rados.rados.0.smithi045.stdout:1369: finishing rollback tid 0 to smithi04526579-46 2024-04-26T00:28:13.022 INFO:tasks.rados.rados.0.smithi045.stdout:1369: finishing rollback tid 1 to smithi04526579-46 2024-04-26T00:28:13.022 INFO:tasks.rados.rados.0.smithi045.stdout:1369: finishing rollback tid 2 to smithi04526579-46 2024-04-26T00:28:13.022 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 746 (ObjNum 329 snap 96 seq_num 329) dirty exists 2024-04-26T00:28:13.022 INFO:tasks.rados.rados.0.smithi045.stdout:1370: finishing write tid 1 to smithi04526579-32 2024-04-26T00:28:13.023 INFO:tasks.rados.rados.0.smithi045.stdout:1366: done (5 left) 2024-04-26T00:28:13.023 INFO:tasks.rados.rados.0.smithi045.stdout:1367: done (4 left) 2024-04-26T00:28:13.023 INFO:tasks.rados.rados.0.smithi045.stdout:1368: done (3 left) 2024-04-26T00:28:13.023 INFO:tasks.rados.rados.0.smithi045.stdout:1369: done (2 left) 2024-04-26T00:28:13.023 INFO:tasks.rados.rados.0.smithi045.stdout:1372: copy_from oid 17 from oid 5 current snap is 157 2024-04-26T00:28:13.023 INFO:tasks.rados.rados.0.smithi045.stdout:1371: finishing rollback tid 1 to smithi04526579-22 2024-04-26T00:28:13.023 INFO:tasks.rados.rados.0.smithi045.stdout:1371: finishing rollback tid 2 to smithi04526579-22 2024-04-26T00:28:13.023 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 748 (ObjNum 330 snap 96 seq_num 330) dirty exists 2024-04-26T00:28:13.023 INFO:tasks.rados.rados.0.smithi045.stdout:1370: finishing write tid 2 to smithi04526579-32 2024-04-26T00:28:13.023 INFO:tasks.rados.rados.0.smithi045.stdout:1370: finishing write tid 3 to smithi04526579-32 2024-04-26T00:28:13.023 INFO:tasks.rados.rados.0.smithi045.stdout:1370: finishing write tid 4 to smithi04526579-32 2024-04-26T00:28:13.023 INFO:tasks.rados.rados.0.smithi045.stdout:1370: oid 32 updating version 0 to 809 2024-04-26T00:28:13.023 INFO:tasks.rados.rados.0.smithi045.stdout:1370: oid 32 version 809 is already newer than 808 2024-04-26T00:28:13.023 INFO:tasks.rados.rados.0.smithi045.stdout:1370: oid 32 updating version 809 to 810 2024-04-26T00:28:13.023 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 810 (ObjNum 512 snap 157 seq_num 512) dirty exists 2024-04-26T00:28:13.024 INFO:tasks.rados.rados.0.smithi045.stdout:1370: left oid 32 (ObjNum 512 snap 157 seq_num 512) 2024-04-26T00:28:13.024 INFO:tasks.rados.rados.0.smithi045.stdout:1370: done (2 left) 2024-04-26T00:28:13.024 INFO:tasks.rados.rados.0.smithi045.stdout:1371: done (1 left) 2024-04-26T00:28:13.024 INFO:tasks.rados.rados.0.smithi045.stdout:1373: read oid 26 snap 143 2024-04-26T00:28:13.024 INFO:tasks.rados.rados.0.smithi045.stdout:1373: expect (ObjNum 459 snap 143 seq_num 459) 2024-04-26T00:28:13.024 INFO:tasks.rados.rados.0.smithi045.stdout:1374: write (excl) oid 47 current snap is 157 2024-04-26T00:28:13.024 INFO:tasks.rados.rados.0.smithi045.stdout:1374: seq_num 513 ranges {621991=798388,1936439=639509} 2024-04-26T00:28:13.034 INFO:tasks.rados.rados.0.smithi045.stdout:1374: writing smithi04526579-47 from 621991 to 1420379 tid 1 2024-04-26T00:28:13.046 INFO:tasks.rados.rados.0.smithi045.stdout:1374: writing smithi04526579-47 from 1936439 to 2575948 tid 2 2024-04-26T00:28:13.047 INFO:tasks.rados.rados.0.smithi045.stdout:1375: delete oid 31 current snap is 157 2024-04-26T00:28:13.047 INFO:tasks.rados.rados.0.smithi045.stdout:1373: expect (ObjNum 459 snap 143 seq_num 459) 2024-04-26T00:28:13.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:13.103+0000 7f18501c9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:28:13.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:13.103+0000 7f18501c9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:28:13.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:13.103+0000 7f18501c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:28:13.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:13.103+0000 7f18501c9640 1 -- 172.21.15.37:0/2358837620 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f184811e0d0 con 0x7f1848137e40 2024-04-26T00:28:13.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:13.103+0000 7f18467fc640 1 -- 172.21.15.37:0/2358837620 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f18480a1130 con 0x7f1848137e40 2024-04-26T00:28:13.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:13.103+0000 7f18467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:28:13.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:13.103+0000 7f18467fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:28:13.091 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":62,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:28:13.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:13.107+0000 7f18501c9640 1 -- 172.21.15.37:0/2358837620 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1834035e80 msgr2=0x7f1834038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:13.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:13.107+0000 7f18501c9640 1 --2- 172.21.15.37:0/2358837620 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1834035e80 0x7f1834038340 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1840005680 tx=0x7f1840003dd0 comp rx=0 tx=0).stop 2024-04-26T00:28:13.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:13.107+0000 7f18501c9640 10 monclient: shutdown 2024-04-26T00:28:13.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:13.107+0000 7f18501c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:28:13.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:13.107+0000 7f18501c9640 1 -- 172.21.15.37:0/2358837620 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1848137e40 msgr2=0x7f18480712d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:13.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:13.107+0000 7f18501c9640 1 --2- 172.21.15.37:0/2358837620 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1848137e40 0x7f18480712d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f183c01a890 tx=0x7f183c018f80 comp rx=0 tx=0).stop 2024-04-26T00:28:13.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:13.107+0000 7f18501c9640 1 -- 172.21.15.37:0/2358837620 shutdown_connections 2024-04-26T00:28:13.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:13.107+0000 7f18501c9640 1 --2- 172.21.15.37:0/2358837620 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1834035e80 0x7f1834038340 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:13.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:13.107+0000 7f18501c9640 1 --2- 172.21.15.37:0/2358837620 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f184813a770 0x7f1848134b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:13.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:13.107+0000 7f18501c9640 1 --2- 172.21.15.37:0/2358837620 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1848137e40 0x7f18480712d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:13.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:13.107+0000 7f18501c9640 1 --2- 172.21.15.37:0/2358837620 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f18480a0260 0x7f184811b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:13.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:13.107+0000 7f18501c9640 1 -- 172.21.15.37:0/2358837620 >> 172.21.15.37:0/2358837620 conn(0x7f18480a1ad0 msgr2=0x7f1848136440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:28:13.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:13.107+0000 7f18501c9640 1 -- 172.21.15.37:0/2358837620 shutdown_connections 2024-04-26T00:28:13.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:13.107+0000 7f18501c9640 1 -- 172.21.15.37:0/2358837620 wait complete. 2024-04-26T00:28:13.224 INFO:tasks.rados.rados.0.smithi045.stdout:1374: finishing write tid 1 to smithi04526579-47 2024-04-26T00:28:13.224 INFO:tasks.rados.rados.0.smithi045.stdout:1372: finishing copy_from racing read to smithi04526579-17 2024-04-26T00:28:13.224 INFO:tasks.rados.rados.0.smithi045.stdout:1372: finishing copy_from to smithi04526579-17 2024-04-26T00:28:13.224 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 605 (ObjNum 477 snap 148 seq_num 477) dirty exists 2024-04-26T00:28:13.224 INFO:tasks.rados.rados.0.smithi045.stdout:1374: finishing write tid 2 to smithi04526579-47 2024-04-26T00:28:13.224 INFO:tasks.rados.rados.0.smithi045.stdout:1374: finishing write tid 3 to smithi04526579-47 2024-04-26T00:28:13.225 INFO:tasks.rados.rados.0.smithi045.stdout:1374: finishing write tid 4 to smithi04526579-47 2024-04-26T00:28:13.225 INFO:tasks.rados.rados.0.smithi045.stdout:1374: oid 47 updating version 0 to 817 2024-04-26T00:28:13.225 INFO:tasks.rados.rados.0.smithi045.stdout:1374: oid 47 version 817 is already newer than 816 2024-04-26T00:28:13.225 INFO:tasks.rados.rados.0.smithi045.stdout:1374: oid 47 version 817 is already newer than 815 2024-04-26T00:28:13.225 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 817 (ObjNum 513 snap 157 seq_num 513) dirty exists 2024-04-26T00:28:13.225 INFO:tasks.rados.rados.0.smithi045.stdout:1374: left oid 47 (ObjNum 513 snap 157 seq_num 513) 2024-04-26T00:28:13.225 INFO:tasks.rados.rados.0.smithi045.stdout:1372: done (3 left) 2024-04-26T00:28:13.225 INFO:tasks.rados.rados.0.smithi045.stdout:1373: done (2 left) 2024-04-26T00:28:13.225 INFO:tasks.rados.rados.0.smithi045.stdout:1374: done (1 left) 2024-04-26T00:28:13.225 INFO:tasks.rados.rados.0.smithi045.stdout:1375: done (0 left) 2024-04-26T00:28:13.225 INFO:tasks.rados.rados.0.smithi045.stdout:1376: read oid 20 snap -1 2024-04-26T00:28:13.225 INFO:tasks.rados.rados.0.smithi045.stdout:1376: expect (ObjNum 497 snap 154 seq_num 497) 2024-04-26T00:28:13.225 INFO:tasks.rados.rados.0.smithi045.stdout:1377: read oid 12 snap -1 2024-04-26T00:28:13.225 INFO:tasks.rados.rados.0.smithi045.stdout:1377: expect (ObjNum 477 snap 148 seq_num 477) 2024-04-26T00:28:13.225 INFO:tasks.rados.rados.0.smithi045.stdout:1378: write oid 16 current snap is 157 2024-04-26T00:28:13.226 INFO:tasks.rados.rados.0.smithi045.stdout:1378: seq_num 515 ranges {655605=548886,1653403=643178} 2024-04-26T00:28:13.239 INFO:tasks.rados.rados.0.smithi045.stdout:1378: writing smithi04526579-16 from 655605 to 1204491 tid 1 2024-04-26T00:28:13.253 INFO:tasks.rados.rados.0.smithi045.stdout:1378: writing smithi04526579-16 from 1653403 to 2296581 tid 2 2024-04-26T00:28:13.253 INFO:tasks.rados.rados.0.smithi045.stdout:1379: rollback oid 11 current snap is 157 2024-04-26T00:28:13.253 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 11 to 118 2024-04-26T00:28:13.253 INFO:tasks.rados.rados.0.smithi045.stdout:1377: expect (ObjNum 477 snap 148 seq_num 477) 2024-04-26T00:28:13.340 INFO:tasks.rados.rados.0.smithi045.stdout:1376: expect (ObjNum 497 snap 154 seq_num 497) 2024-04-26T00:28:13.506 INFO:tasks.rados.rados.0.smithi045.stdout:1378: finishing write tid 1 to smithi04526579-16 2024-04-26T00:28:13.507 INFO:tasks.rados.rados.0.smithi045.stdout:1379: finishing rollback tid 0 to smithi04526579-11 2024-04-26T00:28:13.507 INFO:tasks.rados.rados.0.smithi045.stdout:1378: finishing write tid 2 to smithi04526579-16 2024-04-26T00:28:13.507 INFO:tasks.rados.rados.0.smithi045.stdout:1379: finishing rollback tid 1 to smithi04526579-11 2024-04-26T00:28:13.507 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 845 (ObjNum 129 snap 0 seq_num 2240327904) dirty dne 2024-04-26T00:28:13.507 INFO:tasks.rados.rados.0.smithi045.stdout:1378: finishing write tid 3 to smithi04526579-16 2024-04-26T00:28:13.507 INFO:tasks.rados.rados.0.smithi045.stdout:1378: finishing write tid 4 to smithi04526579-16 2024-04-26T00:28:13.507 INFO:tasks.rados.rados.0.smithi045.stdout:1378: oid 16 updating version 0 to 618 2024-04-26T00:28:13.507 INFO:tasks.rados.rados.0.smithi045.stdout:1378: oid 16 version 618 is already newer than 617 2024-04-26T00:28:13.507 INFO:tasks.rados.rados.0.smithi045.stdout:1378: oid 16 version 618 is already newer than 616 2024-04-26T00:28:13.507 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 618 (ObjNum 515 snap 157 seq_num 515) dirty exists 2024-04-26T00:28:13.508 INFO:tasks.rados.rados.0.smithi045.stdout:1378: left oid 16 (ObjNum 515 snap 157 seq_num 515) 2024-04-26T00:28:13.508 INFO:tasks.rados.rados.0.smithi045.stdout:1376: done (3 left) 2024-04-26T00:28:13.508 INFO:tasks.rados.rados.0.smithi045.stdout:1377: done (2 left) 2024-04-26T00:28:13.508 INFO:tasks.rados.rados.0.smithi045.stdout:1378: done (1 left) 2024-04-26T00:28:13.508 INFO:tasks.rados.rados.0.smithi045.stdout:1379: done (0 left) 2024-04-26T00:28:13.508 INFO:tasks.rados.rados.0.smithi045.stdout:1380: rollback oid 10 current snap is 157 2024-04-26T00:28:13.508 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 10 to 156 2024-04-26T00:28:13.508 INFO:tasks.rados.rados.0.smithi045.stdout:1381: snap_create 2024-04-26T00:28:13.515 INFO:tasks.rados.rados.0.smithi045.stdout:1380: finishing rollback tid 0 to smithi04526579-10 2024-04-26T00:28:13.515 INFO:tasks.rados.rados.0.smithi045.stdout:1380: finishing rollback tid 1 to smithi04526579-10 2024-04-26T00:28:13.515 INFO:tasks.rados.rados.0.smithi045.stdout:1380: finishing rollback tid 2 to smithi04526579-10 2024-04-26T00:28:13.515 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 851 (ObjNum 451 snap 140 seq_num 451) dirty exists 2024-04-26T00:28:14.253 INFO:tasks.rados.rados.0.smithi045.stdout:1380: done (1 left) 2024-04-26T00:28:14.253 INFO:tasks.rados.rados.0.smithi045.stdout:1381: done (0 left) 2024-04-26T00:28:14.253 INFO:tasks.rados.rados.0.smithi045.stdout:1382: read oid 23 snap -1 2024-04-26T00:28:14.254 INFO:tasks.rados.rados.0.smithi045.stdout:1382: expect (ObjNum 453 snap 141 seq_num 453) 2024-04-26T00:28:14.254 INFO:tasks.rados.rados.0.smithi045.stdout:1383: write (excl) oid 35 current snap is 158 2024-04-26T00:28:14.254 INFO:tasks.rados.rados.0.smithi045.stdout:1383: seq_num 516 ranges {672412=624135,1995532=676880} 2024-04-26T00:28:14.275 INFO:tasks.rados.rados.0.smithi045.stdout:1383: writing smithi04526579-35 from 672412 to 1296547 tid 1 2024-04-26T00:28:14.298 INFO:tasks.rados.rados.0.smithi045.stdout:1383: writing smithi04526579-35 from 1995532 to 2672412 tid 2 2024-04-26T00:28:14.298 INFO:tasks.rados.rados.0.smithi045.stdout:1384: delete oid 31 current snap is 158 2024-04-26T00:28:14.298 INFO:tasks.rados.rados.0.smithi045.stdout:1382: expect (ObjNum 453 snap 141 seq_num 453) 2024-04-26T00:28:14.455 INFO:tasks.rados.rados.0.smithi045.stdout:1383: finishing write tid 1 to smithi04526579-35 2024-04-26T00:28:14.455 INFO:tasks.rados.rados.0.smithi045.stdout:1383: finishing write tid 2 to smithi04526579-35 2024-04-26T00:28:14.455 INFO:tasks.rados.rados.0.smithi045.stdout:1383: finishing write tid 3 to smithi04526579-35 2024-04-26T00:28:14.455 INFO:tasks.rados.rados.0.smithi045.stdout:1383: finishing write tid 4 to smithi04526579-35 2024-04-26T00:28:14.455 INFO:tasks.rados.rados.0.smithi045.stdout:1383: oid 35 updating version 0 to 814 2024-04-26T00:28:14.455 INFO:tasks.rados.rados.0.smithi045.stdout:1383: oid 35 version 814 is already newer than 812 2024-04-26T00:28:14.455 INFO:tasks.rados.rados.0.smithi045.stdout:1383: oid 35 version 814 is already newer than 813 2024-04-26T00:28:14.455 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 814 (ObjNum 516 snap 158 seq_num 516) dirty exists 2024-04-26T00:28:14.456 INFO:tasks.rados.rados.0.smithi045.stdout:1383: left oid 35 (ObjNum 516 snap 158 seq_num 516) 2024-04-26T00:28:14.456 INFO:tasks.rados.rados.0.smithi045.stdout:1382: done (2 left) 2024-04-26T00:28:14.456 INFO:tasks.rados.rados.0.smithi045.stdout:1383: done (1 left) 2024-04-26T00:28:14.456 INFO:tasks.rados.rados.0.smithi045.stdout:1384: done (0 left) 2024-04-26T00:28:14.456 INFO:tasks.rados.rados.0.smithi045.stdout:1385: write (excl) oid 32 current snap is 158 2024-04-26T00:28:14.456 INFO:tasks.rados.rados.0.smithi045.stdout:1385: seq_num 518 ranges {706026=490986,1828849=790505} 2024-04-26T00:28:14.474 INFO:tasks.rados.rados.0.smithi045.stdout:1385: writing smithi04526579-32 from 706026 to 1197012 tid 1 2024-04-26T00:28:14.489 INFO:tasks.rados.rados.0.smithi045.stdout:1385: writing smithi04526579-32 from 1828849 to 2619354 tid 2 2024-04-26T00:28:14.490 INFO:tasks.rados.rados.0.smithi045.stdout:1386: copy_from oid 16 from oid 28 current snap is 158 2024-04-26T00:28:14.490 INFO:tasks.rados.rados.0.smithi045.stdout:1385: finishing write tid 1 to smithi04526579-32 2024-04-26T00:28:14.490 INFO:tasks.rados.rados.0.smithi045.stdout:1387: write (excl) oid 37 current snap is 158 2024-04-26T00:28:14.490 INFO:tasks.rados.rados.0.smithi045.stdout:1387: seq_num 519 ranges {722833=566235,1887331=650516} 2024-04-26T00:28:14.496 INFO:tasks.rados.rados.0.smithi045.stdout:1387: writing smithi04526579-37 from 722833 to 1289068 tid 1 2024-04-26T00:28:14.505 INFO:tasks.rados.rados.0.smithi045.stdout:1387: writing smithi04526579-37 from 1887331 to 2537847 tid 2 2024-04-26T00:28:14.505 INFO:tasks.rados.rados.0.smithi045.stdout:1388: snap_create 2024-04-26T00:28:14.505 INFO:tasks.rados.rados.0.smithi045.stdout:1386: finishing copy_from to smithi04526579-16 2024-04-26T00:28:14.505 INFO:tasks.rados.rados.0.smithi045.stdout:1386: got expected ENOENT (src dne) 2024-04-26T00:28:14.506 INFO:tasks.rados.rados.0.smithi045.stdout:1386: finishing copy_from racing read to smithi04526579-16 2024-04-26T00:28:14.506 INFO:tasks.rados.rados.0.smithi045.stdout:1385: finishing write tid 2 to smithi04526579-32 2024-04-26T00:28:14.506 INFO:tasks.rados.rados.0.smithi045.stdout:1385: finishing write tid 3 to smithi04526579-32 2024-04-26T00:28:14.506 INFO:tasks.rados.rados.0.smithi045.stdout:1385: finishing write tid 4 to smithi04526579-32 2024-04-26T00:28:14.506 INFO:tasks.rados.rados.0.smithi045.stdout:1385: oid 32 updating version 0 to 817 2024-04-26T00:28:14.506 INFO:tasks.rados.rados.0.smithi045.stdout:1385: oid 32 version 817 is already newer than 816 2024-04-26T00:28:14.506 INFO:tasks.rados.rados.0.smithi045.stdout:1385: oid 32 updating version 817 to 818 2024-04-26T00:28:14.506 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 818 (ObjNum 518 snap 158 seq_num 518) dirty exists 2024-04-26T00:28:14.506 INFO:tasks.rados.rados.0.smithi045.stdout:1385: left oid 32 (ObjNum 518 snap 158 seq_num 518) 2024-04-26T00:28:14.507 INFO:tasks.rados.rados.0.smithi045.stdout:1387: finishing write tid 1 to smithi04526579-37 2024-04-26T00:28:14.515 INFO:tasks.rados.rados.0.smithi045.stdout:1387: finishing write tid 2 to smithi04526579-37 2024-04-26T00:28:14.516 INFO:tasks.rados.rados.0.smithi045.stdout:1387: finishing write tid 3 to smithi04526579-37 2024-04-26T00:28:14.517 INFO:tasks.rados.rados.0.smithi045.stdout:1387: finishing write tid 4 to smithi04526579-37 2024-04-26T00:28:14.517 INFO:tasks.rados.rados.0.smithi045.stdout:1387: oid 37 updating version 0 to 1088 2024-04-26T00:28:14.517 INFO:tasks.rados.rados.0.smithi045.stdout:1387: oid 37 updating version 1088 to 1089 2024-04-26T00:28:14.517 INFO:tasks.rados.rados.0.smithi045.stdout:1387: oid 37 updating version 1089 to 1090 2024-04-26T00:28:14.517 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 1090 (ObjNum 519 snap 158 seq_num 519) dirty exists 2024-04-26T00:28:14.517 INFO:tasks.rados.rados.0.smithi045.stdout:1387: left oid 37 (ObjNum 519 snap 158 seq_num 519) 2024-04-26T00:28:15.300 INFO:tasks.rados.rados.0.smithi045.stdout:1385: done (3 left) 2024-04-26T00:28:15.300 INFO:tasks.rados.rados.0.smithi045.stdout:1386: done (2 left) 2024-04-26T00:28:15.300 INFO:tasks.rados.rados.0.smithi045.stdout:1387: done (1 left) 2024-04-26T00:28:15.300 INFO:tasks.rados.rados.0.smithi045.stdout:1388: done (0 left) 2024-04-26T00:28:15.301 INFO:tasks.rados.rados.0.smithi045.stdout:1389: copy_from oid 18 from oid 28 current snap is 159 2024-04-26T00:28:15.301 INFO:tasks.rados.rados.0.smithi045.stdout:1390: snap_create 2024-04-26T00:28:15.306 INFO:tasks.rados.rados.0.smithi045.stdout:1389: finishing copy_from to smithi04526579-18 2024-04-26T00:28:15.306 INFO:tasks.rados.rados.0.smithi045.stdout:1389: got expected ENOENT (src dne) 2024-04-26T00:28:15.306 INFO:tasks.rados.rados.0.smithi045.stdout:1389: finishing copy_from racing read to smithi04526579-18 2024-04-26T00:28:16.110 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:28:16.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.255+0000 7ff4be8a3640 10 monclient: get_monmap_and_config 2024-04-26T00:28:16.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.255+0000 7ff4be8a3640 10 monclient: build_initial_monmap 2024-04-26T00:28:16.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.255+0000 7ff4be8a3640 10 monclient: monmap: 2024-04-26T00:28:16.242 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:28:16.242 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:16.242 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:28:16.257940+0000 2024-04-26T00:28:16.242 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:28:16.257940+0000 2024-04-26T00:28:16.242 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:28:16.242 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:16.242 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:28:16.242 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:28:16.242 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:28:16.242 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:28:16.242 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:28:16.243 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:16.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.255+0000 7ff4be8a3640 1 Processor -- start 2024-04-26T00:28:16.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.255+0000 7ff4be8a3640 1 -- start start 2024-04-26T00:28:16.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.255+0000 7ff4be8a3640 10 monclient: init 2024-04-26T00:28:16.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.255+0000 7ff4be8a3640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:16.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.255+0000 7ff4be8a3640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-04-26T00:28:16.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.255+0000 7ff4be8a3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff4b8133b20 addr v1:172.21.15.37:6789/0 2024-04-26T00:28:16.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.255+0000 7ff4be8a3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff4b8133f00 0x7ff4b8134350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:16.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.255+0000 7ff4be8a3640 10 monclient(hunting): picked mon.noname-b con 0x7ff4b8133f00 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:28:16.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.255+0000 7ff4be8a3640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff4b8139050 0x7ff4b813b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:16.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.255+0000 7ff4be8a3640 10 monclient(hunting): picked mon.noname-a con 0x7ff4b8139050 addr v2:172.21.15.37:3300/0 2024-04-26T00:28:16.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.255+0000 7ff4be8a3640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4b809fe20 con 0x7ff4b8133b20 2024-04-26T00:28:16.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.255+0000 7ff4be8a3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:16.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.255+0000 7ff4be8a3640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7ff4b809e620 con 0x7ff4b8139050 2024-04-26T00:28:16.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.255+0000 7ff4be8a3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:16.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.255+0000 7ff4be8a3640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff4b8072d00 con 0x7ff4b8133f00 2024-04-26T00:28:16.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.255+0000 7ff4be8a3640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:16.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.255+0000 7ff4be8a3640 10 monclient(hunting): authenticate will time out at 1477.563660s 2024-04-26T00:28:16.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.255+0000 7ff4b77fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff4b8133f00 0x7ff4b8134350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:16.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.255+0000 7ff4b77fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff4b8133f00 0x7ff4b8134350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:58130/0 (socket says 172.21.15.37:58130) 2024-04-26T00:28:16.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.255+0000 7ff4b77fe640 1 -- 172.21.15.37:0/1117129013 learned_addr learned my addr 172.21.15.37:0/1117129013 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:28:16.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.255+0000 7ff4b77fe640 10 monclient(hunting): get_auth_request con 0x7ff4b8133f00 auth_method 0 2024-04-26T00:28:16.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.255+0000 7ff4b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:16.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.255+0000 7ff4b77fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:16.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.255+0000 7ff4b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:16.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.255+0000 7ff4bce40640 1 --2- 172.21.15.37:0/1117129013 >> v2:172.21.15.37:3300/0 conn(0x7ff4b8139050 0x7ff4b813b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:16.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.255+0000 7ff4bce40640 10 monclient(hunting): get_auth_request con 0x7ff4b8139050 auth_method 0 2024-04-26T00:28:16.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.255+0000 7ff4bce40640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:16.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.255+0000 7ff4bce40640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:16.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.255+0000 7ff4bce40640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:16.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.259+0000 7ff4b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:16.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.259+0000 7ff4b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:16.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.259+0000 7ff4b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:16.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.259+0000 7ff4bce40640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:16.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.259+0000 7ff4bce40640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:16.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.259+0000 7ff4bce40640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:16.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.259+0000 7ff4b77fe640 10 monclient(hunting): handle_auth_done global_id 84275 payload 274 2024-04-26T00:28:16.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.259+0000 7ff4b77fe640 1 -- 172.21.15.37:0/1117129013 >> v2:172.21.15.37:3300/0 conn(0x7ff4b8139050 msgr2=0x7ff4b813b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:28:16.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.259+0000 7ff4b77fe640 1 --2- 172.21.15.37:0/1117129013 >> v2:172.21.15.37:3300/0 conn(0x7ff4b8139050 0x7ff4b813b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:16.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.259+0000 7ff4b77fe640 1 -- 172.21.15.37:0/1117129013 >> v1:172.21.15.37:6789/0 conn(0x7ff4b8133b20 legacy=0x7ff4b809e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:28:16.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.259+0000 7ff4b77fe640 10 monclient: _finish_hunting 0 2024-04-26T00:28:16.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.259+0000 7ff4b77fe640 1 monclient: found mon.noname-b 2024-04-26T00:28:16.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.259+0000 7ff4b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:16.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.259+0000 7ff4b77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:28:16.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.259+0000 7ff4b77fe640 1 -- 172.21.15.37:0/1117129013 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4b809ba60 con 0x7ff4b8133f00 2024-04-26T00:28:16.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4b77fe640 1 -- 172.21.15.37:0/1117129013 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff4b8133f00 msgr2=0x7ff4b8134350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:16.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4b77fe640 1 --2- 172.21.15.37:0/1117129013 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff4b8133f00 0x7ff4b8134350 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff4a000bb10 tx=0x7ff4a001e8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5cf825fe5b2d9d07 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:16.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4b6ffd640 1 -- 172.21.15.37:0/1117129013 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a00a86f0 con 0x7ff4b8133f00 2024-04-26T00:28:16.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:16.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4b6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:28:16.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4b6ffd640 10 monclient: dump: 2024-04-26T00:28:16.250 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:16.250 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:16.250 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:16.250 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:16.250 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:16.250 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:16.250 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:16.250 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:16.250 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:16.251 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:16.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4b6ffd640 10 monclient: _finish_auth 0 2024-04-26T00:28:16.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4b6ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:28:16.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:16.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4b6ffd640 1 -- 172.21.15.37:0/1117129013 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4a00a8890 con 0x7ff4b8133f00 2024-04-26T00:28:16.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4be8a3640 5 monclient: authenticate success, global_id 84275 2024-04-26T00:28:16.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4be8a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:28:16.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4b6ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:28:16.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4be8a3640 10 monclient: get_monmap_and_config success 2024-04-26T00:28:16.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4be8a3640 4 set_mon_vals no callback set 2024-04-26T00:28:16.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4b6ffd640 1 -- 172.21.15.37:0/1117129013 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a00a8c60 con 0x7ff4b8133f00 2024-04-26T00:28:16.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:16.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:28:16.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4b6ffd640 10 monclient: dump: 2024-04-26T00:28:16.252 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:16.252 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:16.252 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:16.252 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:16.252 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:16.252 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:16.252 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:16.252 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:16.252 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:16.252 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:16.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4be8a3640 10 monclient: shutdown 2024-04-26T00:28:16.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4be8a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:28:16.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4be8a3640 1 -- 172.21.15.37:0/1117129013 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff4b8133f00 msgr2=0x7ff4b8134350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:16.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4be8a3640 1 --2- 172.21.15.37:0/1117129013 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff4b8133f00 0x7ff4b8134350 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff4a000bb10 tx=0x7ff4a001e8d0 comp rx=0 tx=0).stop 2024-04-26T00:28:16.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4be8a3640 1 -- 172.21.15.37:0/1117129013 shutdown_connections 2024-04-26T00:28:16.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4be8a3640 1 --2- 172.21.15.37:0/1117129013 >> v2:172.21.15.37:3300/0 conn(0x7ff4b8139050 0x7ff4b813b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:16.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4be8a3640 1 --2- 172.21.15.37:0/1117129013 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff4b8133f00 0x7ff4b8134350 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:16.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4be8a3640 1 -- 172.21.15.37:0/1117129013 >> 172.21.15.37:0/1117129013 conn(0x7ff4b80a1ad0 msgr2=0x7ff4b80a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:28:16.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4be8a3640 1 -- 172.21.15.37:0/1117129013 shutdown_connections 2024-04-26T00:28:16.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4be8a3640 1 -- 172.21.15.37:0/1117129013 wait complete. 2024-04-26T00:28:16.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4be8a3640 10 monclient: build_initial_monmap 2024-04-26T00:28:16.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4be8a3640 10 monclient: monmap: 2024-04-26T00:28:16.253 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:28:16.253 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:28:16.253 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:28:16.253 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:28:16.253 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:28:16.253 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:28:16.253 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:28:16.254 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:28:16.254 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:28:16.254 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:16.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4be8a3640 1 Processor -- start 2024-04-26T00:28:16.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4be8a3640 1 -- start start 2024-04-26T00:28:16.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4be8a3640 10 monclient: init 2024-04-26T00:28:16.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4be8a3640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:16.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4be8a3640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:28:16.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4be8a3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff4b8133b20 0x7ff4b8071400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:16.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4be8a3640 10 monclient(hunting): picked mon.noname-c con 0x7ff4b8133b20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:28:16.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4be8a3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4b8133f00 0x7ff4b81cf2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:16.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4be8a3640 10 monclient(hunting): picked mon.noname-b con 0x7ff4b8133f00 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:28:16.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4be8a3640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff4b8139050 0x7ff4b8130f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:16.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4be8a3640 10 monclient(hunting): picked mon.noname-a con 0x7ff4b8139050 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:28:16.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4be8a3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:16.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4be8a3640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b80a0bf0 con 0x7ff4b8139050 2024-04-26T00:28:16.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4be8a3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:16.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4be8a3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b80a0a70 con 0x7ff4b8133f00 2024-04-26T00:28:16.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4be8a3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:16.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4be8a3640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff4b80a0d70 con 0x7ff4b8133b20 2024-04-26T00:28:16.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4be8a3640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:16.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4be8a3640 10 monclient(hunting): authenticate will time out at 1477.571351s 2024-04-26T00:28:16.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4bce40640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff4b8139050 0x7ff4b8130f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:16.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4bce40640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff4b8139050 0x7ff4b8130f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52302/0 (socket says 172.21.15.37:52302) 2024-04-26T00:28:16.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4bce40640 1 -- 172.21.15.37:0/3150797965 learned_addr learned my addr 172.21.15.37:0/3150797965 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:28:16.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4bce40640 10 monclient(hunting): get_auth_request con 0x7ff4b8139050 auth_method 0 2024-04-26T00:28:16.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4bce40640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:16.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4bce40640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:16.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4bce40640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:16.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4b7fff640 1 --2- 172.21.15.37:0/3150797965 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff4b8133b20 0x7ff4b8071400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:16.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4bce40640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:16.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4bce40640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:16.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4bce40640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:16.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4b7fff640 10 monclient(hunting): get_auth_request con 0x7ff4b8133b20 auth_method 0 2024-04-26T00:28:16.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:16.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4b7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:16.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:16.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:16.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:16.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:16.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4bce40640 10 monclient(hunting): handle_auth_done global_id 94182 payload 824 2024-04-26T00:28:16.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4bce40640 1 -- 172.21.15.37:0/3150797965 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff4b8133b20 msgr2=0x7ff4b8071400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:16.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4bce40640 1 --2- 172.21.15.37:0/3150797965 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff4b8133b20 0x7ff4b8071400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:16.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4bce40640 1 -- 172.21.15.37:0/3150797965 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4b8133f00 msgr2=0x7ff4b81cf2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:16.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4b77fe640 1 --2- 172.21.15.37:0/3150797965 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4b8133f00 0x7ff4b81cf2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:16.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4bce40640 1 --2- 172.21.15.37:0/3150797965 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4b8133f00 0x7ff4b81cf2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:16.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4bce40640 10 monclient: _finish_hunting 0 2024-04-26T00:28:16.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4bce40640 1 monclient: found mon.noname-a 2024-04-26T00:28:16.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4bce40640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:16.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4bce40640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:28:16.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.263+0000 7ff4bce40640 1 -- 172.21.15.37:0/3150797965 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4b80708a0 con 0x7ff4b8139050 2024-04-26T00:28:16.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.267+0000 7ff4bce40640 1 -- 172.21.15.37:0/3150797965 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff4b8139050 msgr2=0x7ff4b8130f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:16.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.267+0000 7ff4bce40640 1 --2- 172.21.15.37:0/3150797965 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff4b8139050 0x7ff4b8130f20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff4a800e880 tx=0x7ff4a800ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:16.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.267+0000 7ff4b4ff9640 1 -- 172.21.15.37:0/3150797965 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a8017070 con 0x7ff4b8139050 2024-04-26T00:28:16.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.267+0000 7ff4b4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:16.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.267+0000 7ff4b4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:28:16.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.267+0000 7ff4b4ff9640 10 monclient: dump: 2024-04-26T00:28:16.261 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:16.261 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:16.261 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:16.261 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:16.261 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:16.261 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:16.261 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:16.262 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:16.262 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:16.262 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:16.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.267+0000 7ff4b4ff9640 10 monclient: _finish_auth 0 2024-04-26T00:28:16.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.267+0000 7ff4b4ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:28:16.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.267+0000 7ff4b4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:16.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.267+0000 7ff4b4ff9640 1 -- 172.21.15.37:0/3150797965 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4a800caf0 con 0x7ff4b8139050 2024-04-26T00:28:16.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.267+0000 7ff4be8a3640 5 monclient: authenticate success, global_id 94182 2024-04-26T00:28:16.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.267+0000 7ff4be8a3640 10 monclient: _renew_subs 2024-04-26T00:28:16.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.267+0000 7ff4be8a3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:28:16.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.267+0000 7ff4be8a3640 1 -- 172.21.15.37:0/3150797965 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4b80a3a60 con 0x7ff4b8139050 2024-04-26T00:28:16.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.267+0000 7ff4b4ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:28:16.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.267+0000 7ff4b4ff9640 1 -- 172.21.15.37:0/3150797965 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a8013470 con 0x7ff4b8139050 2024-04-26T00:28:16.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.267+0000 7ff4b4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:16.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.267+0000 7ff4b4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:28:16.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.267+0000 7ff4b4ff9640 10 monclient: dump: 2024-04-26T00:28:16.263 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:16.263 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:16.263 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:16.264 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:16.264 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:16.264 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:16.264 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:16.264 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:16.264 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:16.264 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:16.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.267+0000 7ff4be8a3640 10 monclient: _renew_subs 2024-04-26T00:28:16.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.267+0000 7ff4b5ffb640 4 set_mon_vals no callback set 2024-04-26T00:28:16.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.267+0000 7ff4be8a3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:28:16.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.267+0000 7ff4be8a3640 1 -- 172.21.15.37:0/3150797965 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4b81cf810 con 0x7ff4b8139050 2024-04-26T00:28:16.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.267+0000 7ff4be8a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:16.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.267+0000 7ff4be8a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:16.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.267+0000 7ff4be8a3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:28:16.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.267+0000 7ff4be8a3640 1 -- 172.21.15.37:0/3150797965 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4b8070130 con 0x7ff4b8139050 2024-04-26T00:28:16.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.267+0000 7ff4b4ff9640 1 -- 172.21.15.37:0/3150797965 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff4a801a070 con 0x7ff4b8139050 2024-04-26T00:28:16.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.267+0000 7ff4b4ff9640 1 --2- 172.21.15.37:0/3150797965 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff498035ed0 0x7ff498038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:16.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.267+0000 7ff4b4ff9640 1 -- 172.21.15.37:0/3150797965 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(322..322 src has 177..322) ==== 3915+0+0 (secure 0 0 0) 0x7ff4a8048b10 con 0x7ff4b8139050 2024-04-26T00:28:16.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.267+0000 7ff4b7fff640 1 --2- 172.21.15.37:0/3150797965 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff498035ed0 0x7ff498038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:16.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.271+0000 7ff4b7fff640 10 monclient: get_auth_request con 0x7ff498035ed0 auth_method 0 2024-04-26T00:28:16.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.271+0000 7ff4b7fff640 1 --2- 172.21.15.37:0/3150797965 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff498035ed0 0x7ff498038390 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff4b0005680 tx=0x7ff4b0003dd0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:16.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.275+0000 7ff4b4ff9640 1 -- 172.21.15.37:0/3150797965 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff4a801a520 con 0x7ff4b8139050 2024-04-26T00:28:16.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.275+0000 7ff4b4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:16.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.275+0000 7ff4b4ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:28:16.299 INFO:tasks.rados.rados.0.smithi045.stdout:1389: done (1 left) 2024-04-26T00:28:16.299 INFO:tasks.rados.rados.0.smithi045.stdout:1390: done (0 left) 2024-04-26T00:28:16.299 INFO:tasks.rados.rados.0.smithi045.stdout:1391: read oid 4 snap -1 2024-04-26T00:28:16.299 INFO:tasks.rados.rados.0.smithi045.stdout:1391: expect (ObjNum 474 snap 147 seq_num 474) 2024-04-26T00:28:16.299 INFO:tasks.rados.rados.0.smithi045.stdout:1392: write (excl) oid 11 current snap is 160 2024-04-26T00:28:16.299 INFO:tasks.rados.rados.0.smithi045.stdout:1392: seq_num 520 ranges {739640=641484,1829460=794174} 2024-04-26T00:28:16.316 INFO:tasks.rados.rados.0.smithi045.stdout:1392: writing smithi04526579-11 from 739640 to 1381124 tid 1 2024-04-26T00:28:16.337 INFO:tasks.rados.rados.0.smithi045.stdout:1392: writing smithi04526579-11 from 1829460 to 2623634 tid 2 2024-04-26T00:28:16.337 INFO:tasks.rados.rados.0.smithi045.stdout:1393: snap_create 2024-04-26T00:28:16.337 INFO:tasks.rados.rados.0.smithi045.stdout:1392: finishing write tid 1 to smithi04526579-11 2024-04-26T00:28:16.337 INFO:tasks.rados.rados.0.smithi045.stdout:1391: expect (ObjNum 474 snap 147 seq_num 474) 2024-04-26T00:28:16.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.507+0000 7ff4be8a3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:28:16.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.507+0000 7ff4be8a3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:28:16.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.507+0000 7ff4be8a3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:28:16.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.507+0000 7ff4be8a3640 1 -- 172.21.15.37:0/3150797965 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff4b8066f40 con 0x7ff4b8139050 2024-04-26T00:28:16.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.507+0000 7ff4b4ff9640 1 -- 172.21.15.37:0/3150797965 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7ff4a8013650 con 0x7ff4b8139050 2024-04-26T00:28:16.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.507+0000 7ff4b4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:28:16.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.507+0000 7ff4b4ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:28:16.494 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":62,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:28:16.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.511+0000 7ff4be8a3640 1 -- 172.21.15.37:0/3150797965 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff498035ed0 msgr2=0x7ff498038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:16.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.511+0000 7ff4be8a3640 1 --2- 172.21.15.37:0/3150797965 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff498035ed0 0x7ff498038390 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff4b0005680 tx=0x7ff4b0003dd0 comp rx=0 tx=0).stop 2024-04-26T00:28:16.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.511+0000 7ff4be8a3640 10 monclient: shutdown 2024-04-26T00:28:16.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.511+0000 7ff4be8a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:28:16.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.511+0000 7ff4be8a3640 1 -- 172.21.15.37:0/3150797965 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff4b8139050 msgr2=0x7ff4b8130f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:16.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.511+0000 7ff4be8a3640 1 --2- 172.21.15.37:0/3150797965 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff4b8139050 0x7ff4b8130f20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff4a800e880 tx=0x7ff4a800ed40 comp rx=0 tx=0).stop 2024-04-26T00:28:16.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.511+0000 7ff4be8a3640 1 -- 172.21.15.37:0/3150797965 shutdown_connections 2024-04-26T00:28:16.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.511+0000 7ff4be8a3640 1 --2- 172.21.15.37:0/3150797965 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff498035ed0 0x7ff498038390 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:16.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.511+0000 7ff4be8a3640 1 --2- 172.21.15.37:0/3150797965 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff4b8139050 0x7ff4b8130f20 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:16.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.511+0000 7ff4be8a3640 1 --2- 172.21.15.37:0/3150797965 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff4b8133f00 0x7ff4b81cf2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:16.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.511+0000 7ff4be8a3640 1 --2- 172.21.15.37:0/3150797965 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff4b8133b20 0x7ff4b8071400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:16.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.511+0000 7ff4be8a3640 1 -- 172.21.15.37:0/3150797965 >> 172.21.15.37:0/3150797965 conn(0x7ff4b80a1ad0 msgr2=0x7ff4b81321d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:28:16.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.511+0000 7ff4be8a3640 1 -- 172.21.15.37:0/3150797965 shutdown_connections 2024-04-26T00:28:16.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:16.511+0000 7ff4be8a3640 1 -- 172.21.15.37:0/3150797965 wait complete. 2024-04-26T00:28:16.645 INFO:tasks.rados.rados.0.smithi045.stdout:1392: finishing write tid 2 to smithi04526579-11 2024-04-26T00:28:16.645 INFO:tasks.rados.rados.0.smithi045.stdout:1392: finishing write tid 3 to smithi04526579-11 2024-04-26T00:28:16.645 INFO:tasks.rados.rados.0.smithi045.stdout:1392: finishing write tid 4 to smithi04526579-11 2024-04-26T00:28:16.645 INFO:tasks.rados.rados.0.smithi045.stdout:1392: oid 11 updating version 0 to 848 2024-04-26T00:28:16.645 INFO:tasks.rados.rados.0.smithi045.stdout:1392: oid 11 version 848 is already newer than 847 2024-04-26T00:28:16.645 INFO:tasks.rados.rados.0.smithi045.stdout:1392: oid 11 version 848 is already newer than 846 2024-04-26T00:28:16.645 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 848 (ObjNum 520 snap 160 seq_num 520) dirty exists 2024-04-26T00:28:16.645 INFO:tasks.rados.rados.0.smithi045.stdout:1392: left oid 11 (ObjNum 520 snap 160 seq_num 520) 2024-04-26T00:28:17.562 INFO:tasks.rados.rados.0.smithi045.stdout:1391: done (2 left) 2024-04-26T00:28:17.562 INFO:tasks.rados.rados.0.smithi045.stdout:1392: done (1 left) 2024-04-26T00:28:17.563 INFO:tasks.rados.rados.0.smithi045.stdout:1393: done (0 left) 2024-04-26T00:28:17.563 INFO:tasks.rados.rados.0.smithi045.stdout:1394: delete oid 17 current snap is 161 2024-04-26T00:28:17.578 INFO:tasks.rados.rados.0.smithi045.stdout:1394: done (0 left) 2024-04-26T00:28:17.578 INFO:tasks.rados.rados.0.smithi045.stdout:1395: read oid 44 snap -1 2024-04-26T00:28:17.578 INFO:tasks.rados.rados.0.smithi045.stdout:1395: expect deleted 2024-04-26T00:28:17.578 INFO:tasks.rados.rados.0.smithi045.stdout:1396: snap_remove snap 157 2024-04-26T00:28:18.565 INFO:tasks.rados.rados.0.smithi045.stdout:1396: done (1 left) 2024-04-26T00:28:18.565 INFO:tasks.rados.rados.0.smithi045.stdout:1397: delete oid 8 current snap is 161 2024-04-26T00:28:18.581 INFO:tasks.rados.rados.0.smithi045.stdout:1395: done (1 left) 2024-04-26T00:28:18.582 INFO:tasks.rados.rados.0.smithi045.stdout:1397: done (0 left) 2024-04-26T00:28:18.582 INFO:tasks.rados.rados.0.smithi045.stdout:1398: snap_create 2024-04-26T00:28:19.256 INFO:tasks.rados.rados.0.smithi045.stdout:1398: done (0 left) 2024-04-26T00:28:19.256 INFO:tasks.rados.rados.0.smithi045.stdout:1399: copy_from oid 30 from oid 45 current snap is 162 2024-04-26T00:28:19.256 INFO:tasks.rados.rados.0.smithi045.stdout:1400: delete oid 29 current snap is 162 2024-04-26T00:28:19.268 INFO:tasks.rados.rados.0.smithi045.stdout:1400: done (1 left) 2024-04-26T00:28:19.268 INFO:tasks.rados.rados.0.smithi045.stdout:1401: snap_remove snap 133 2024-04-26T00:28:19.514 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:28:19.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.659+0000 7f0ea2a55640 10 monclient: get_monmap_and_config 2024-04-26T00:28:19.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.659+0000 7f0ea2a55640 10 monclient: build_initial_monmap 2024-04-26T00:28:19.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.659+0000 7f0ea2a55640 10 monclient: monmap: 2024-04-26T00:28:19.647 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:28:19.647 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:19.647 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:28:19.662503+0000 2024-04-26T00:28:19.647 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:28:19.662503+0000 2024-04-26T00:28:19.647 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:28:19.647 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:19.647 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:28:19.647 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:28:19.647 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:28:19.647 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:28:19.647 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:28:19.648 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:19.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.659+0000 7f0ea2a55640 1 Processor -- start 2024-04-26T00:28:19.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0ea2a55640 1 -- start start 2024-04-26T00:28:19.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0ea2a55640 10 monclient: init 2024-04-26T00:28:19.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0ea2a55640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:19.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0ea2a55640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-04-26T00:28:19.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0ea2a55640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0e9c09c7d0 0x7f0e9c130570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:19.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0ea2a55640 10 monclient(hunting): picked mon.noname-c con 0x7f0e9c09c7d0 addr v2:172.21.15.45:3300/0 2024-04-26T00:28:19.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0ea2a55640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f0e9c130ab0 0x7f0e9c130f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:19.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0ea2a55640 10 monclient(hunting): picked mon.noname-a con 0x7f0e9c130ab0 addr v2:172.21.15.37:3300/0 2024-04-26T00:28:19.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0ea2a55640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e9c133a90 0x7f0e9c133ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:19.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0ea2a55640 10 monclient(hunting): picked mon.noname-b con 0x7f0e9c133a90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:28:19.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0ea2a55640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:19.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0ea2a55640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f0e9c09cbb0 con 0x7f0e9c130ab0 2024-04-26T00:28:19.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0ea2a55640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:19.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0ea2a55640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f0e9c09ba10 con 0x7f0e9c09c7d0 2024-04-26T00:28:19.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0ea2a55640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:19.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0ea2a55640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0e9c09bcf0 con 0x7f0e9c133a90 2024-04-26T00:28:19.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0ea2a55640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:19.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0ea2a55640 10 monclient(hunting): authenticate will time out at 1480.968985s 2024-04-26T00:28:19.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0e93fff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f0e9c130ab0 0x7f0e9c130f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:19.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0e93fff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f0e9c130ab0 0x7f0e9c130f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52312/0 (socket says 172.21.15.37:52312) 2024-04-26T00:28:19.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0e93fff640 1 -- 172.21.15.37:0/1103107217 learned_addr learned my addr 172.21.15.37:0/1103107217 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:28:19.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0e93fff640 10 monclient(hunting): get_auth_request con 0x7f0e9c130ab0 auth_method 0 2024-04-26T00:28:19.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0e93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:19.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0e93fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:19.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0e93fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:19.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0ea0ff2640 1 --2- 172.21.15.37:0/1103107217 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e9c133a90 0x7f0e9c133ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:19.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0ea0ff2640 10 monclient(hunting): get_auth_request con 0x7f0e9c133a90 auth_method 0 2024-04-26T00:28:19.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0ea0ff2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:19.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0ea0ff2640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:19.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0ea0ff2640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:19.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0e9bfff640 1 --2- 172.21.15.37:0/1103107217 >> v2:172.21.15.45:3300/0 conn(0x7f0e9c09c7d0 0x7f0e9c130570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:19.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0e93fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:19.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0e93fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:19.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0e93fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:19.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0e9bfff640 10 monclient(hunting): get_auth_request con 0x7f0e9c09c7d0 auth_method 0 2024-04-26T00:28:19.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0e9bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:19.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0e9bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:19.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0e9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:19.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0ea0ff2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:19.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0ea0ff2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:19.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0ea0ff2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:19.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0e93fff640 10 monclient(hunting): handle_auth_done global_id 94185 payload 274 2024-04-26T00:28:19.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0e9bfff640 1 -- 172.21.15.37:0/1103107217 >> v2:172.21.15.45:3300/0 conn(0x7f0e9c09c7d0 msgr2=0x7f0e9c130570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:28:19.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0e9bfff640 1 -- 172.21.15.37:0/1103107217 >> v2:172.21.15.45:3300/0 conn(0x7f0e9c09c7d0 msgr2=0x7f0e9c130570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:28:19.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0e9bfff640 1 --2- 172.21.15.37:0/1103107217 >> v2:172.21.15.45:3300/0 conn(0x7f0e9c09c7d0 0x7f0e9c130570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:28:19.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0e93fff640 1 -- 172.21.15.37:0/1103107217 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e9c133a90 msgr2=0x7f0e9c133ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:28:19.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0e93fff640 1 --2- 172.21.15.37:0/1103107217 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e9c133a90 0x7f0e9c133ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:19.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0e9bfff640 1 --2- 172.21.15.37:0/1103107217 >> v2:172.21.15.45:3300/0 conn(0x7f0e9c09c7d0 0x7f0e9c130570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:28:19.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0e93fff640 1 -- 172.21.15.37:0/1103107217 >> v2:172.21.15.45:3300/0 conn(0x7f0e9c09c7d0 msgr2=0x7f0e9c130570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:28:19.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0e93fff640 1 --2- 172.21.15.37:0/1103107217 >> v2:172.21.15.45:3300/0 conn(0x7f0e9c09c7d0 0x7f0e9c130570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:19.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0e93fff640 10 monclient: _finish_hunting 0 2024-04-26T00:28:19.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0e93fff640 1 monclient: found mon.noname-a 2024-04-26T00:28:19.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0e93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:19.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0e93fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:28:19.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.663+0000 7f0e93fff640 1 -- 172.21.15.37:0/1103107217 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e9c09bfd0 con 0x7f0e9c130ab0 2024-04-26T00:28:19.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.667+0000 7f0e93fff640 1 -- 172.21.15.37:0/1103107217 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0e9c130ab0 msgr2=0x7f0e9c130f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:19.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.667+0000 7f0e93fff640 1 --2- 172.21.15.37:0/1103107217 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0e9c130ab0 0x7f0e9c130f40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c0a2430 tx=0x7f0e8401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29929c0c8a3c2f0e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:19.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.667+0000 7f0e9b7fe640 1 -- 172.21.15.37:0/1103107217 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e840b0070 con 0x7f0e9c130ab0 2024-04-26T00:28:19.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.667+0000 7f0e9b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:19.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.667+0000 7f0e9b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:28:19.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.667+0000 7f0e9b7fe640 10 monclient: dump: 2024-04-26T00:28:19.662 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:19.662 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:19.662 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:19.662 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:19.662 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:19.662 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:19.662 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:19.662 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:19.662 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:19.662 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:19.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.667+0000 7f0e9b7fe640 10 monclient: _finish_auth 0 2024-04-26T00:28:19.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.667+0000 7f0e9b7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:28:19.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.667+0000 7f0e9b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:19.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.667+0000 7f0e9b7fe640 1 -- 172.21.15.37:0/1103107217 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e840ab600 con 0x7f0e9c130ab0 2024-04-26T00:28:19.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.667+0000 7f0e9b7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:28:19.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.667+0000 7f0e9b7fe640 1 -- 172.21.15.37:0/1103107217 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e840ab9d0 con 0x7f0e9c130ab0 2024-04-26T00:28:19.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.667+0000 7f0ea2a55640 5 monclient: authenticate success, global_id 94185 2024-04-26T00:28:19.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.667+0000 7f0ea2a55640 10 monclient: get_monmap_and_config success 2024-04-26T00:28:19.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.667+0000 7f0ea2a55640 4 set_mon_vals no callback set 2024-04-26T00:28:19.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.667+0000 7f0e9b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:19.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.667+0000 7f0e9b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:28:19.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.667+0000 7f0e9b7fe640 10 monclient: dump: 2024-04-26T00:28:19.663 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:19.663 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:19.663 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:19.663 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:19.663 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:19.664 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.667+0000 7f0ea2a55640 10 monclient: shutdown 2024-04-26T00:28:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.667+0000 7f0ea2a55640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:28:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.667+0000 7f0ea2a55640 1 -- 172.21.15.37:0/1103107217 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0e9c130ab0 msgr2=0x7f0e9c130f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.667+0000 7f0ea2a55640 1 --2- 172.21.15.37:0/1103107217 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0e9c130ab0 0x7f0e9c130f40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c0a2430 tx=0x7f0e8401c3e0 comp rx=0 tx=0).stop 2024-04-26T00:28:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.667+0000 7f0ea2a55640 1 -- 172.21.15.37:0/1103107217 shutdown_connections 2024-04-26T00:28:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.667+0000 7f0ea2a55640 1 --2- 172.21.15.37:0/1103107217 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e9c133a90 0x7f0e9c133ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.667+0000 7f0ea2a55640 1 --2- 172.21.15.37:0/1103107217 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0e9c130ab0 0x7f0e9c130f40 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.667+0000 7f0ea2a55640 1 --2- 172.21.15.37:0/1103107217 >> v2:172.21.15.45:3300/0 conn(0x7f0e9c09c7d0 0x7f0e9c130570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.667+0000 7f0ea2a55640 1 -- 172.21.15.37:0/1103107217 >> 172.21.15.37:0/1103107217 conn(0x7f0e9c0a6450 msgr2=0x7f0e9c0a6860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:28:19.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.667+0000 7f0ea2a55640 1 -- 172.21.15.37:0/1103107217 shutdown_connections 2024-04-26T00:28:19.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.667+0000 7f0ea2a55640 1 -- 172.21.15.37:0/1103107217 wait complete. 2024-04-26T00:28:19.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.667+0000 7f0ea2a55640 10 monclient: build_initial_monmap 2024-04-26T00:28:19.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.667+0000 7f0ea2a55640 10 monclient: monmap: 2024-04-26T00:28:19.665 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:28:19.665 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:28:19.665 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:28:19.665 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:28:19.665 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:28:19.665 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:28:19.665 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:28:19.665 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:28:19.665 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:28:19.665 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:19.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.667+0000 7f0ea2a55640 1 Processor -- start 2024-04-26T00:28:19.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.667+0000 7f0ea2a55640 1 -- start start 2024-04-26T00:28:19.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.667+0000 7f0ea2a55640 10 monclient: init 2024-04-26T00:28:19.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0ea2a55640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:19.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0ea2a55640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:28:19.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0ea2a55640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0e9c09c7d0 0x7f0e9c1cd5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:19.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0ea2a55640 10 monclient(hunting): picked mon.noname-a con 0x7f0e9c09c7d0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:28:19.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0ea2a55640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e9c130ab0 0x7f0e9c072b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:19.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0ea2a55640 10 monclient(hunting): picked mon.noname-b con 0x7f0e9c130ab0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:28:19.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0ea2a55640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e9c133a90 0x7f0e9c073070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:19.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0ea2a55640 10 monclient(hunting): picked mon.noname-c con 0x7f0e9c133a90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:28:19.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0ea2a55640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:19.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0ea2a55640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f0e9c06bdb0 con 0x7f0e9c09c7d0 2024-04-26T00:28:19.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0ea2a55640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:19.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0ea2a55640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0e9c06bc30 con 0x7f0e9c130ab0 2024-04-26T00:28:19.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0ea2a55640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:19.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0ea2a55640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0e9c06bf30 con 0x7f0e9c133a90 2024-04-26T00:28:19.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0ea2a55640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:19.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0ea2a55640 10 monclient(hunting): authenticate will time out at 1480.976431s 2024-04-26T00:28:19.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0e9bfff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0e9c09c7d0 0x7f0e9c1cd5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:19.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0ea0ff2640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e9c133a90 0x7f0e9c073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:19.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0e9bfff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0e9c09c7d0 0x7f0e9c1cd5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52318/0 (socket says 172.21.15.37:52318) 2024-04-26T00:28:19.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0e9bfff640 1 -- 172.21.15.37:0/1300818443 learned_addr learned my addr 172.21.15.37:0/1300818443 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:28:19.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0e9bfff640 10 monclient(hunting): get_auth_request con 0x7f0e9c09c7d0 auth_method 0 2024-04-26T00:28:19.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0e9bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:19.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0e9bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:19.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0e9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:19.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0ea0ff2640 10 monclient(hunting): get_auth_request con 0x7f0e9c133a90 auth_method 0 2024-04-26T00:28:19.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0ea0ff2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:19.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0ea0ff2640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:19.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0ea0ff2640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:19.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0e9bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:19.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0e9bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:19.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0e9bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:19.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0ea0ff2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:19.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0ea0ff2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:19.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0ea0ff2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:19.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0e9bfff640 10 monclient(hunting): handle_auth_done global_id 94188 payload 824 2024-04-26T00:28:19.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0e93fff640 1 --2- 172.21.15.37:0/1300818443 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e9c130ab0 0x7f0e9c072b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:19.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0e9bfff640 1 -- 172.21.15.37:0/1300818443 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e9c133a90 msgr2=0x7f0e9c073070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:19.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0e9bfff640 1 --2- 172.21.15.37:0/1300818443 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e9c133a90 0x7f0e9c073070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:19.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0e9bfff640 1 -- 172.21.15.37:0/1300818443 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e9c130ab0 msgr2=0x7f0e9c072b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:19.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0e9bfff640 1 --2- 172.21.15.37:0/1300818443 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e9c130ab0 0x7f0e9c072b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:19.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0e9bfff640 10 monclient: _finish_hunting 0 2024-04-26T00:28:19.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0e9bfff640 1 monclient: found mon.noname-a 2024-04-26T00:28:19.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0e9bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:19.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0e9bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:28:19.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0e9bfff640 1 -- 172.21.15.37:0/1300818443 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e9c09aec0 con 0x7f0e9c09c7d0 2024-04-26T00:28:19.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0e9bfff640 1 -- 172.21.15.37:0/1300818443 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0e9c09c7d0 msgr2=0x7f0e9c1cd5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:19.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0e9bfff640 1 --2- 172.21.15.37:0/1300818443 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0e9c09c7d0 0x7f0e9c1cd5c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c00e810 tx=0x7f0e8c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:19.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0e997fa640 1 -- 172.21.15.37:0/1300818443 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c017020 con 0x7f0e9c09c7d0 2024-04-26T00:28:19.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0e997fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:19.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0e997fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:28:19.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0e997fa640 10 monclient: dump: 2024-04-26T00:28:19.671 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:19.672 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:19.672 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:19.672 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:19.672 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:19.672 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:19.672 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:19.672 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:19.672 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:19.672 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:19.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0e997fa640 10 monclient: _finish_auth 0 2024-04-26T00:28:19.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0e997fa640 10 monclient: _check_auth_tickets 2024-04-26T00:28:19.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0e997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:19.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0e997fa640 1 -- 172.21.15.37:0/1300818443 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e8c00c910 con 0x7f0e9c09c7d0 2024-04-26T00:28:19.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0e997fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:28:19.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0e997fa640 1 -- 172.21.15.37:0/1300818443 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c00f040 con 0x7f0e9c09c7d0 2024-04-26T00:28:19.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0ea2a55640 5 monclient: authenticate success, global_id 94188 2024-04-26T00:28:19.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0ea2a55640 10 monclient: _renew_subs 2024-04-26T00:28:19.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0ea2a55640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:28:19.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0ea2a55640 1 -- 172.21.15.37:0/1300818443 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e9c0a53b0 con 0x7f0e9c09c7d0 2024-04-26T00:28:19.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0e997fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:19.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0e997fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:28:19.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0e9affd640 4 set_mon_vals no callback set 2024-04-26T00:28:19.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0e997fa640 10 monclient: dump: 2024-04-26T00:28:19.673 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:19.673 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:19.673 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:19.673 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:19.674 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:19.674 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:19.674 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:19.674 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:19.674 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:19.674 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:19.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0ea2a55640 10 monclient: _renew_subs 2024-04-26T00:28:19.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0ea2a55640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:28:19.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.671+0000 7f0ea2a55640 1 -- 172.21.15.37:0/1300818443 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e9c0705f0 con 0x7f0e9c09c7d0 2024-04-26T00:28:19.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.675+0000 7f0ea2a55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:19.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.675+0000 7f0ea2a55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:19.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.675+0000 7f0ea2a55640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:28:19.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.675+0000 7f0ea2a55640 1 -- 172.21.15.37:0/1300818443 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e9c066f40 con 0x7f0e9c09c7d0 2024-04-26T00:28:19.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.675+0000 7f0e997fa640 1 -- 172.21.15.37:0/1300818443 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f0e8c01a020 con 0x7f0e9c09c7d0 2024-04-26T00:28:19.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.675+0000 7f0e997fa640 1 --2- 172.21.15.37:0/1300818443 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0e7c035e80 0x7f0e7c038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:19.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.675+0000 7f0e997fa640 1 -- 172.21.15.37:0/1300818443 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(326..326 src has 177..326) ==== 3943+0+0 (secure 0 0 0) 0x7f0e8c0485c0 con 0x7f0e9c09c7d0 2024-04-26T00:28:19.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.675+0000 7f0e93fff640 1 --2- 172.21.15.37:0/1300818443 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0e7c035e80 0x7f0e7c038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:19.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.675+0000 7f0e93fff640 10 monclient: get_auth_request con 0x7f0e7c035e80 auth_method 0 2024-04-26T00:28:19.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.679+0000 7f0e93fff640 1 --2- 172.21.15.37:0/1300818443 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0e7c035e80 0x7f0e7c038340 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0e840ab000 tx=0x7f0e840a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:19.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.679+0000 7f0e997fa640 1 -- 172.21.15.37:0/1300818443 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f0e8c021030 con 0x7f0e9c09c7d0 2024-04-26T00:28:19.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.679+0000 7f0e997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:19.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.679+0000 7f0e997fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:28:19.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.903+0000 7f0ea2a55640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:28:19.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.903+0000 7f0ea2a55640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:28:19.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.903+0000 7f0ea2a55640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:28:19.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.903+0000 7f0ea2a55640 1 -- 172.21.15.37:0/1300818443 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0e9c09b540 con 0x7f0e9c09c7d0 2024-04-26T00:28:19.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.903+0000 7f0e997fa640 1 -- 172.21.15.37:0/1300818443 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f0e8c023300 con 0x7f0e9c09c7d0 2024-04-26T00:28:19.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.903+0000 7f0e997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:28:19.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.903+0000 7f0e997fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:28:19.893 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":62,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":27,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:28:19.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.907+0000 7f0ea2a55640 1 -- 172.21.15.37:0/1300818443 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0e7c035e80 msgr2=0x7f0e7c038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:19.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.907+0000 7f0ea2a55640 1 --2- 172.21.15.37:0/1300818443 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0e7c035e80 0x7f0e7c038340 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0e840ab000 tx=0x7f0e840a6000 comp rx=0 tx=0).stop 2024-04-26T00:28:19.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.907+0000 7f0ea2a55640 10 monclient: shutdown 2024-04-26T00:28:19.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.907+0000 7f0ea2a55640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:28:19.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.907+0000 7f0ea2a55640 1 -- 172.21.15.37:0/1300818443 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0e9c09c7d0 msgr2=0x7f0e9c1cd5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:19.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.907+0000 7f0ea2a55640 1 --2- 172.21.15.37:0/1300818443 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0e9c09c7d0 0x7f0e9c1cd5c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c00e810 tx=0x7f0e8c00ecd0 comp rx=0 tx=0).stop 2024-04-26T00:28:19.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.907+0000 7f0ea2a55640 1 -- 172.21.15.37:0/1300818443 shutdown_connections 2024-04-26T00:28:19.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.907+0000 7f0ea2a55640 1 --2- 172.21.15.37:0/1300818443 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0e7c035e80 0x7f0e7c038340 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:19.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.907+0000 7f0ea2a55640 1 --2- 172.21.15.37:0/1300818443 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0e9c133a90 0x7f0e9c073070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:19.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.907+0000 7f0ea2a55640 1 --2- 172.21.15.37:0/1300818443 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e9c130ab0 0x7f0e9c072b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:19.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.907+0000 7f0ea2a55640 1 --2- 172.21.15.37:0/1300818443 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0e9c09c7d0 0x7f0e9c1cd5c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:19.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.907+0000 7f0ea2a55640 1 -- 172.21.15.37:0/1300818443 >> 172.21.15.37:0/1300818443 conn(0x7f0e9c0a6450 msgr2=0x7f0e9c0a4080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:28:19.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.907+0000 7f0ea2a55640 1 -- 172.21.15.37:0/1300818443 shutdown_connections 2024-04-26T00:28:19.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:19.907+0000 7f0ea2a55640 1 -- 172.21.15.37:0/1300818443 wait complete. 2024-04-26T00:28:20.272 INFO:tasks.rados.rados.0.smithi045.stdout:1401: done (1 left) 2024-04-26T00:28:20.272 INFO:tasks.rados.rados.0.smithi045.stdout:1402: read oid 47 snap -1 2024-04-26T00:28:20.272 INFO:tasks.rados.rados.0.smithi045.stdout:1402: expect (ObjNum 513 snap 157 seq_num 513) 2024-04-26T00:28:20.272 INFO:tasks.rados.rados.0.smithi045.stdout:1403: write oid 1 current snap is 162 2024-04-26T00:28:20.272 INFO:tasks.rados.rados.0.smithi045.stdout:1399: finishing copy_from racing read to smithi04526579-30 2024-04-26T00:28:20.272 INFO:tasks.rados.rados.0.smithi045.stdout:1399: finishing copy_from to smithi04526579-30 2024-04-26T00:28:20.272 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 740 (ObjNum 482 snap 150 seq_num 482) dirty exists 2024-04-26T00:28:20.273 INFO:tasks.rados.rados.0.smithi045.stdout:1403: seq_num 524 ranges {406868=542480,1547035=401512,2690354=116514} 2024-04-26T00:28:20.291 INFO:tasks.rados.rados.0.smithi045.stdout:1403: writing smithi04526579-1 from 406868 to 949348 tid 1 2024-04-26T00:28:20.310 INFO:tasks.rados.rados.0.smithi045.stdout:1403: writing smithi04526579-1 from 1547035 to 1948547 tid 2 2024-04-26T00:28:20.325 INFO:tasks.rados.rados.0.smithi045.stdout:1403: writing smithi04526579-1 from 2690354 to 2806868 tid 3 2024-04-26T00:28:20.325 INFO:tasks.rados.rados.0.smithi045.stdout:1399: done (2 left) 2024-04-26T00:28:20.325 INFO:tasks.rados.rados.0.smithi045.stdout:1404: rollback oid 11 current snap is 162 2024-04-26T00:28:20.325 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 11 to 159 2024-04-26T00:28:20.325 INFO:tasks.rados.rados.0.smithi045.stdout:1402: expect (ObjNum 513 snap 157 seq_num 513) 2024-04-26T00:28:20.496 INFO:tasks.rados.rados.0.smithi045.stdout:1403: finishing write tid 1 to smithi04526579-1 2024-04-26T00:28:20.496 INFO:tasks.rados.rados.0.smithi045.stdout:1403: finishing write tid 2 to smithi04526579-1 2024-04-26T00:28:20.496 INFO:tasks.rados.rados.0.smithi045.stdout:1403: finishing write tid 3 to smithi04526579-1 2024-04-26T00:28:20.496 INFO:tasks.rados.rados.0.smithi045.stdout:1403: finishing write tid 4 to smithi04526579-1 2024-04-26T00:28:20.496 INFO:tasks.rados.rados.0.smithi045.stdout:1403: finishing write tid 5 to smithi04526579-1 2024-04-26T00:28:20.496 INFO:tasks.rados.rados.0.smithi045.stdout:1403: oid 1 updating version 0 to 1030 2024-04-26T00:28:20.496 INFO:tasks.rados.rados.0.smithi045.stdout:1403: oid 1 version 1030 is already newer than 1029 2024-04-26T00:28:20.496 INFO:tasks.rados.rados.0.smithi045.stdout:1403: oid 1 updating version 1030 to 1031 2024-04-26T00:28:20.496 INFO:tasks.rados.rados.0.smithi045.stdout:1403: oid 1 version 1031 is already newer than 1028 2024-04-26T00:28:20.496 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 1031 (ObjNum 524 snap 162 seq_num 524) dirty exists 2024-04-26T00:28:20.496 INFO:tasks.rados.rados.0.smithi045.stdout:1403: left oid 1 (ObjNum 524 snap 162 seq_num 524) 2024-04-26T00:28:20.497 INFO:tasks.rados.rados.0.smithi045.stdout:1404: finishing rollback tid 0 to smithi04526579-11 2024-04-26T00:28:20.497 INFO:tasks.rados.rados.0.smithi045.stdout:1404: finishing rollback tid 1 to smithi04526579-11 2024-04-26T00:28:20.497 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 850 (ObjNum 81 snap 0 seq_num 2242122512) dirty dne 2024-04-26T00:28:20.497 INFO:tasks.rados.rados.0.smithi045.stdout:1402: done (2 left) 2024-04-26T00:28:20.497 INFO:tasks.rados.rados.0.smithi045.stdout:1403: done (1 left) 2024-04-26T00:28:20.497 INFO:tasks.rados.rados.0.smithi045.stdout:1404: done (0 left) 2024-04-26T00:28:20.497 INFO:tasks.rados.rados.0.smithi045.stdout:1405: read oid 17 snap -1 2024-04-26T00:28:20.497 INFO:tasks.rados.rados.0.smithi045.stdout:1405: expect deleted 2024-04-26T00:28:20.497 INFO:tasks.rados.rados.0.smithi045.stdout:1406: delete oid 29 current snap is 162 2024-04-26T00:28:20.499 INFO:tasks.rados.rados.0.smithi045.stdout:1405: done (1 left) 2024-04-26T00:28:20.499 INFO:tasks.rados.rados.0.smithi045.stdout:1406: done (0 left) 2024-04-26T00:28:20.499 INFO:tasks.rados.rados.0.smithi045.stdout:1407: snap_create 2024-04-26T00:28:21.275 INFO:tasks.rados.rados.0.smithi045.stdout:1407: done (0 left) 2024-04-26T00:28:21.275 INFO:tasks.rados.rados.0.smithi045.stdout:1408: rollback oid 7 current snap is 163 2024-04-26T00:28:21.275 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 7 to 126 2024-04-26T00:28:21.275 INFO:tasks.rados.rados.0.smithi045.stdout:1409: read oid 47 snap 158 2024-04-26T00:28:21.276 INFO:tasks.rados.rados.0.smithi045.stdout:1409: expect (ObjNum 513 snap 157 seq_num 513) 2024-04-26T00:28:21.276 INFO:tasks.rados.rados.0.smithi045.stdout:1410: read oid 6 snap 162 2024-04-26T00:28:21.276 INFO:tasks.rados.rados.0.smithi045.stdout:1410: expect (ObjNum 457 snap 142 seq_num 457) 2024-04-26T00:28:21.276 INFO:tasks.rados.rados.0.smithi045.stdout:1411: write oid 32 current snap is 163 2024-04-26T00:28:21.276 INFO:tasks.rados.rados.0.smithi045.stdout:1411: seq_num 526 ranges {440482=409331,1380352=405181,2461553=378929} 2024-04-26T00:28:21.288 INFO:tasks.rados.rados.0.smithi045.stdout:1411: writing smithi04526579-32 from 440482 to 849813 tid 1 2024-04-26T00:28:21.299 INFO:tasks.rados.rados.0.smithi045.stdout:1411: writing smithi04526579-32 from 1380352 to 1785533 tid 2 2024-04-26T00:28:21.310 INFO:tasks.rados.rados.0.smithi045.stdout:1411: writing smithi04526579-32 from 2461553 to 2840482 tid 3 2024-04-26T00:28:21.310 INFO:tasks.rados.rados.0.smithi045.stdout:1412: read oid 39 snap 150 2024-04-26T00:28:21.313 INFO:tasks.rados.rados.0.smithi045.stdout:1412: expect (ObjNum 468 snap 146 seq_num 468) 2024-04-26T00:28:21.313 INFO:tasks.rados.rados.0.smithi045.stdout:1413: read oid 16 snap -1 2024-04-26T00:28:21.313 INFO:tasks.rados.rados.0.smithi045.stdout:1413: expect (ObjNum 515 snap 157 seq_num 515) 2024-04-26T00:28:21.313 INFO:tasks.rados.rados.0.smithi045.stdout:1414: snap_create 2024-04-26T00:28:21.313 INFO:tasks.rados.rados.0.smithi045.stdout:1408: finishing rollback tid 0 to smithi04526579-7 2024-04-26T00:28:21.313 INFO:tasks.rados.rados.0.smithi045.stdout:1408: finishing rollback tid 1 to smithi04526579-7 2024-04-26T00:28:21.314 INFO:tasks.rados.rados.0.smithi045.stdout:1408: finishing rollback tid 2 to smithi04526579-7 2024-04-26T00:28:21.314 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 1104 (ObjNum 114 snap 28 seq_num 114) dirty exists 2024-04-26T00:28:21.314 INFO:tasks.rados.rados.0.smithi045.stdout:1409: expect (ObjNum 513 snap 157 seq_num 513) 2024-04-26T00:28:21.485 INFO:tasks.rados.rados.0.smithi045.stdout:1410: expect (ObjNum 457 snap 142 seq_num 457) 2024-04-26T00:28:21.715 INFO:tasks.rados.rados.0.smithi045.stdout:1411: finishing write tid 1 to smithi04526579-32 2024-04-26T00:28:21.716 INFO:tasks.rados.rados.0.smithi045.stdout:1411: finishing write tid 2 to smithi04526579-32 2024-04-26T00:28:21.716 INFO:tasks.rados.rados.0.smithi045.stdout:1411: finishing write tid 3 to smithi04526579-32 2024-04-26T00:28:21.716 INFO:tasks.rados.rados.0.smithi045.stdout:1411: finishing write tid 4 to smithi04526579-32 2024-04-26T00:28:21.716 INFO:tasks.rados.rados.0.smithi045.stdout:1411: finishing write tid 5 to smithi04526579-32 2024-04-26T00:28:21.716 INFO:tasks.rados.rados.0.smithi045.stdout:1411: oid 32 updating version 0 to 834 2024-04-26T00:28:21.716 INFO:tasks.rados.rados.0.smithi045.stdout:1411: oid 32 version 834 is already newer than 832 2024-04-26T00:28:21.716 INFO:tasks.rados.rados.0.smithi045.stdout:1411: oid 32 updating version 834 to 835 2024-04-26T00:28:21.716 INFO:tasks.rados.rados.0.smithi045.stdout:1411: oid 32 version 835 is already newer than 833 2024-04-26T00:28:21.716 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 835 (ObjNum 526 snap 163 seq_num 526) dirty exists 2024-04-26T00:28:21.717 INFO:tasks.rados.rados.0.smithi045.stdout:1411: left oid 32 (ObjNum 526 snap 163 seq_num 526) 2024-04-26T00:28:21.717 INFO:tasks.rados.rados.0.smithi045.stdout:1413: expect (ObjNum 515 snap 157 seq_num 515) 2024-04-26T00:28:21.878 INFO:tasks.rados.rados.0.smithi045.stdout:1412: expect (ObjNum 468 snap 146 seq_num 468) 2024-04-26T00:28:22.284 INFO:tasks.rados.rados.0.smithi045.stdout:1408: done (6 left) 2024-04-26T00:28:22.284 INFO:tasks.rados.rados.0.smithi045.stdout:1409: done (5 left) 2024-04-26T00:28:22.284 INFO:tasks.rados.rados.0.smithi045.stdout:1410: done (4 left) 2024-04-26T00:28:22.284 INFO:tasks.rados.rados.0.smithi045.stdout:1411: done (3 left) 2024-04-26T00:28:22.284 INFO:tasks.rados.rados.0.smithi045.stdout:1412: done (2 left) 2024-04-26T00:28:22.284 INFO:tasks.rados.rados.0.smithi045.stdout:1413: done (1 left) 2024-04-26T00:28:22.284 INFO:tasks.rados.rados.0.smithi045.stdout:1414: done (0 left) 2024-04-26T00:28:22.284 INFO:tasks.rados.rados.0.smithi045.stdout:1415: write oid 20 current snap is 164 2024-04-26T00:28:22.284 INFO:tasks.rados.rados.0.smithi045.stdout:1415: seq_num 527 ranges {457289=484580,1438834=548839,2772623=84666} 2024-04-26T00:28:22.297 INFO:tasks.rados.rados.0.smithi045.stdout:1415: writing smithi04526579-20 from 457289 to 941869 tid 1 2024-04-26T00:28:22.311 INFO:tasks.rados.rados.0.smithi045.stdout:1415: writing smithi04526579-20 from 1438834 to 1987673 tid 2 2024-04-26T00:28:22.321 INFO:tasks.rados.rados.0.smithi045.stdout:1415: writing smithi04526579-20 from 2772623 to 2857289 tid 3 2024-04-26T00:28:22.322 INFO:tasks.rados.rados.0.smithi045.stdout:1416: rollback oid 22 current snap is 164 2024-04-26T00:28:22.322 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 22 to 154 2024-04-26T00:28:22.322 INFO:tasks.rados.rados.0.smithi045.stdout:1415: finishing write tid 1 to smithi04526579-20 2024-04-26T00:28:22.322 INFO:tasks.rados.rados.0.smithi045.stdout:1417: delete oid 15 current snap is 164 2024-04-26T00:28:22.322 INFO:tasks.rados.rados.0.smithi045.stdout:1415: finishing write tid 2 to smithi04526579-20 2024-04-26T00:28:22.328 INFO:tasks.rados.rados.0.smithi045.stdout:1415: finishing write tid 3 to smithi04526579-20 2024-04-26T00:28:22.328 INFO:tasks.rados.rados.0.smithi045.stdout:1415: finishing write tid 4 to smithi04526579-20 2024-04-26T00:28:22.328 INFO:tasks.rados.rados.0.smithi045.stdout:1415: finishing write tid 5 to smithi04526579-20 2024-04-26T00:28:22.328 INFO:tasks.rados.rados.0.smithi045.stdout:1415: oid 20 updating version 0 to 522 2024-04-26T00:28:22.329 INFO:tasks.rados.rados.0.smithi045.stdout:1415: oid 20 updating version 522 to 523 2024-04-26T00:28:22.329 INFO:tasks.rados.rados.0.smithi045.stdout:1415: oid 20 version 523 is already newer than 521 2024-04-26T00:28:22.329 INFO:tasks.rados.rados.0.smithi045.stdout:1415: oid 20 version 523 is already newer than 520 2024-04-26T00:28:22.329 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 523 (ObjNum 527 snap 164 seq_num 527) dirty exists 2024-04-26T00:28:22.329 INFO:tasks.rados.rados.0.smithi045.stdout:1415: left oid 20 (ObjNum 527 snap 164 seq_num 527) 2024-04-26T00:28:22.329 INFO:tasks.rados.rados.0.smithi045.stdout:1416: finishing rollback tid 0 to smithi04526579-22 2024-04-26T00:28:22.329 INFO:tasks.rados.rados.0.smithi045.stdout:1416: finishing rollback tid 1 to smithi04526579-22 2024-04-26T00:28:22.330 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 750 (ObjNum 129 snap 0 seq_num 2242029424) dirty dne 2024-04-26T00:28:22.331 INFO:tasks.rados.rados.0.smithi045.stdout:1415: done (2 left) 2024-04-26T00:28:22.331 INFO:tasks.rados.rados.0.smithi045.stdout:1416: done (1 left) 2024-04-26T00:28:22.331 INFO:tasks.rados.rados.0.smithi045.stdout:1417: done (0 left) 2024-04-26T00:28:22.331 INFO:tasks.rados.rados.0.smithi045.stdout:1418: delete oid 10 current snap is 164 2024-04-26T00:28:22.337 INFO:tasks.rados.rados.0.smithi045.stdout:1418: done (0 left) 2024-04-26T00:28:22.338 INFO:tasks.rados.rados.0.smithi045.stdout:1419: snap_remove snap 155 2024-04-26T00:28:22.911 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:28:23.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9af7aa640 10 monclient: get_monmap_and_config 2024-04-26T00:28:23.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9af7aa640 10 monclient: build_initial_monmap 2024-04-26T00:28:23.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9af7aa640 10 monclient: monmap: 2024-04-26T00:28:23.045 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:28:23.045 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:23.045 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:28:23.060788+0000 2024-04-26T00:28:23.045 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:28:23.060788+0000 2024-04-26T00:28:23.045 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:28:23.045 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:23.045 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:28:23.045 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:28:23.045 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:28:23.045 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:28:23.046 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:28:23.046 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:23.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9af7aa640 1 Processor -- start 2024-04-26T00:28:23.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9af7aa640 1 -- start start 2024-04-26T00:28:23.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9af7aa640 10 monclient: init 2024-04-26T00:28:23.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9af7aa640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:23.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9af7aa640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-04-26T00:28:23.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9af7aa640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc9a80a0260 0x7fc9a80a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:23.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9af7aa640 10 monclient(hunting): picked mon.noname-a con 0x7fc9a80a0260 addr v2:172.21.15.37:3300/0 2024-04-26T00:28:23.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9af7aa640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc9a8134b30 0x7fc9a8134f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:23.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9af7aa640 10 monclient(hunting): picked mon.noname-c con 0x7fc9a8134b30 addr v2:172.21.15.45:3300/0 2024-04-26T00:28:23.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9af7aa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc9a8135490 addr v1:172.21.15.45:6789/0 2024-04-26T00:28:23.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9af7aa640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9a80a1130 con 0x7fc9a8135490 2024-04-26T00:28:23.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9af7aa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:23.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9af7aa640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fc9a809ec10 con 0x7fc9a80a0260 2024-04-26T00:28:23.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9af7aa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:23.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9af7aa640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc9a809f0f0 con 0x7fc9a8134b30 2024-04-26T00:28:23.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9af7aa640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:23.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9af7aa640 10 monclient(hunting): authenticate will time out at 1484.366719s 2024-04-26T00:28:23.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9ad546640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc9a80a0260 0x7fc9a80a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:23.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9ad546640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc9a80a0260 0x7fc9a80a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:56228/0 (socket says 172.21.15.37:56228) 2024-04-26T00:28:23.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9ad546640 1 -- 172.21.15.37:0/4285721573 learned_addr learned my addr 172.21.15.37:0/4285721573 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:28:23.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9ad546640 10 monclient(hunting): get_auth_request con 0x7fc9a80a0260 auth_method 0 2024-04-26T00:28:23.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9ad546640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:23.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9ad546640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:23.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9ad546640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:23.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9ad546640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:23.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9ad546640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:23.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9ad546640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:23.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9acd45640 1 --2- 172.21.15.37:0/4285721573 >> v2:172.21.15.45:3300/0 conn(0x7fc9a8134b30 0x7fc9a8134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:23.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9ad546640 10 monclient(hunting): handle_auth_done global_id 94191 payload 274 2024-04-26T00:28:23.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9ad546640 1 -- 172.21.15.37:0/4285721573 >> v2:172.21.15.45:3300/0 conn(0x7fc9a8134b30 msgr2=0x7fc9a8134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:28:23.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9ad546640 1 --2- 172.21.15.37:0/4285721573 >> v2:172.21.15.45:3300/0 conn(0x7fc9a8134b30 0x7fc9a8134f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:23.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9ad546640 1 -- 172.21.15.37:0/4285721573 >> v1:172.21.15.45:6789/0 conn(0x7fc9a8135490 legacy=0x7fc9a813be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:28:23.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9ad546640 10 monclient: _finish_hunting 0 2024-04-26T00:28:23.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9ad546640 1 monclient: found mon.noname-a 2024-04-26T00:28:23.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9ad546640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:23.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9ad546640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:28:23.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9ad546640 1 -- 172.21.15.37:0/4285721573 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9a806cff0 con 0x7fc9a80a0260 2024-04-26T00:28:23.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9acd45640 10 monclient: get_auth_request con 0x7fc9a8134b30 auth_method 0 2024-04-26T00:28:23.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.059+0000 7fc9acd45640 1 --2- 172.21.15.37:0/4285721573 >> v2:172.21.15.45:3300/0 conn(0x7fc9a8134b30 0x7fc9a8134f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:28:23.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.063+0000 7fc9ad546640 1 -- 172.21.15.37:0/4285721573 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc9a80a0260 msgr2=0x7fc9a80a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:23.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.063+0000 7fc9ad546640 1 --2- 172.21.15.37:0/4285721573 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc9a80a0260 0x7fc9a80a0640 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc9a809f690 tx=0x7fc99c01c1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efbec4686baa07c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:23.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.063+0000 7fc9a7fff640 1 -- 172.21.15.37:0/4285721573 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc99c0b0070 con 0x7fc9a80a0260 2024-04-26T00:28:23.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.063+0000 7fc9a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:23.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.063+0000 7fc9a7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:28:23.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.063+0000 7fc9a7fff640 10 monclient: dump: 2024-04-26T00:28:23.052 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:23.053 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:23.053 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:23.053 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:23.053 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:23.053 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:23.053 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:23.053 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:23.053 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:23.053 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:23.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.063+0000 7fc9a7fff640 10 monclient: _finish_auth 0 2024-04-26T00:28:23.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.063+0000 7fc9a7fff640 10 monclient: _check_auth_tickets 2024-04-26T00:28:23.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.063+0000 7fc9a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:23.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.063+0000 7fc9a7fff640 1 -- 172.21.15.37:0/4285721573 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc99c0ab600 con 0x7fc9a80a0260 2024-04-26T00:28:23.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.063+0000 7fc9a7fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:28:23.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.063+0000 7fc9a7fff640 1 -- 172.21.15.37:0/4285721573 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc99c0ab9d0 con 0x7fc9a80a0260 2024-04-26T00:28:23.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.063+0000 7fc9af7aa640 5 monclient: authenticate success, global_id 94191 2024-04-26T00:28:23.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.063+0000 7fc9af7aa640 10 monclient: get_monmap_and_config success 2024-04-26T00:28:23.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.063+0000 7fc9af7aa640 4 set_mon_vals no callback set 2024-04-26T00:28:23.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.063+0000 7fc9a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:23.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.063+0000 7fc9a7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:28:23.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.063+0000 7fc9a7fff640 10 monclient: dump: 2024-04-26T00:28:23.054 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:23.054 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:23.054 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:23.054 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:23.054 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:23.054 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:23.054 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:23.055 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:23.055 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:23.055 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:23.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9af7aa640 10 monclient: shutdown 2024-04-26T00:28:23.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9af7aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:28:23.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9af7aa640 1 -- 172.21.15.37:0/4285721573 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc9a80a0260 msgr2=0x7fc9a80a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:23.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9af7aa640 1 --2- 172.21.15.37:0/4285721573 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc9a80a0260 0x7fc9a80a0640 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc9a809f690 tx=0x7fc99c01c1c0 comp rx=0 tx=0).stop 2024-04-26T00:28:23.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9af7aa640 1 -- 172.21.15.37:0/4285721573 shutdown_connections 2024-04-26T00:28:23.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9af7aa640 1 --2- 172.21.15.37:0/4285721573 >> v2:172.21.15.45:3300/0 conn(0x7fc9a8134b30 0x7fc9a8134f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:23.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9af7aa640 1 --2- 172.21.15.37:0/4285721573 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc9a80a0260 0x7fc9a80a0640 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:23.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9af7aa640 1 -- 172.21.15.37:0/4285721573 >> 172.21.15.37:0/4285721573 conn(0x7fc9a80a1ad0 msgr2=0x7fc9a80a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:28:23.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9af7aa640 1 -- 172.21.15.37:0/4285721573 shutdown_connections 2024-04-26T00:28:23.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9af7aa640 1 -- 172.21.15.37:0/4285721573 wait complete. 2024-04-26T00:28:23.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9af7aa640 10 monclient: build_initial_monmap 2024-04-26T00:28:23.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9af7aa640 10 monclient: monmap: 2024-04-26T00:28:23.065 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:28:23.065 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:28:23.065 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:28:23.065 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:28:23.065 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:28:23.065 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:28:23.065 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:28:23.065 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:28:23.065 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:28:23.066 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:23.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9af7aa640 1 Processor -- start 2024-04-26T00:28:23.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9af7aa640 1 -- start start 2024-04-26T00:28:23.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9af7aa640 10 monclient: init 2024-04-26T00:28:23.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9af7aa640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:23.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9af7aa640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:28:23.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9af7aa640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc9a80a0260 0x7fc9a81323a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:23.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9af7aa640 10 monclient(hunting): picked mon.noname-a con 0x7fc9a80a0260 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:28:23.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9af7aa640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9a8134b30 0x7fc9a813c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:23.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9af7aa640 10 monclient(hunting): picked mon.noname-b con 0x7fc9a8134b30 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:28:23.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9af7aa640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc9a8135490 0x7fc9a809c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:23.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9af7aa640 10 monclient(hunting): picked mon.noname-c con 0x7fc9a8135490 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:28:23.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9af7aa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:23.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9af7aa640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a809f430 con 0x7fc9a80a0260 2024-04-26T00:28:23.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9af7aa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:23.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9af7aa640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a809f2b0 con 0x7fc9a8134b30 2024-04-26T00:28:23.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9af7aa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:23.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9af7aa640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc9a809fe90 con 0x7fc9a8135490 2024-04-26T00:28:23.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9af7aa640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:23.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9af7aa640 10 monclient(hunting): authenticate will time out at 1484.373959s 2024-04-26T00:28:23.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9add47640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc9a8135490 0x7fc9a809c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:23.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9ad546640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc9a80a0260 0x7fc9a81323a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:23.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9add47640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc9a8135490 0x7fc9a809c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:35456/0 (socket says 172.21.15.37:35456) 2024-04-26T00:28:23.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9add47640 1 -- 172.21.15.37:0/1536808800 learned_addr learned my addr 172.21.15.37:0/1536808800 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:28:23.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9add47640 10 monclient(hunting): get_auth_request con 0x7fc9a8135490 auth_method 0 2024-04-26T00:28:23.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9add47640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:23.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9add47640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:23.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9add47640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:23.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9ad546640 10 monclient(hunting): get_auth_request con 0x7fc9a80a0260 auth_method 0 2024-04-26T00:28:23.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9ad546640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:23.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9ad546640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:23.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9ad546640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:23.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9add47640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:23.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9add47640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:23.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9add47640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:23.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9ad546640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:23.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9ad546640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:23.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9ad546640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:23.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9acd45640 1 --2- 172.21.15.37:0/1536808800 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9a8134b30 0x7fc9a813c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:23.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9acd45640 10 monclient(hunting): get_auth_request con 0x7fc9a8134b30 auth_method 0 2024-04-26T00:28:23.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9acd45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:23.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9acd45640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:23.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9acd45640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:23.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9ad546640 10 monclient(hunting): handle_auth_done global_id 94194 payload 824 2024-04-26T00:28:23.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9ad546640 1 -- 172.21.15.37:0/1536808800 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc9a8135490 msgr2=0x7fc9a809c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:23.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9ad546640 1 --2- 172.21.15.37:0/1536808800 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc9a8135490 0x7fc9a809c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:23.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9ad546640 1 -- 172.21.15.37:0/1536808800 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9a8134b30 msgr2=0x7fc9a813c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:23.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9ad546640 1 --2- 172.21.15.37:0/1536808800 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9a8134b30 0x7fc9a813c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:23.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9ad546640 10 monclient: _finish_hunting 0 2024-04-26T00:28:23.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9ad546640 1 monclient: found mon.noname-a 2024-04-26T00:28:23.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9ad546640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:23.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9ad546640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:28:23.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9ad546640 1 -- 172.21.15.37:0/1536808800 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9a813c800 con 0x7fc9a80a0260 2024-04-26T00:28:23.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.067+0000 7fc9add47640 1 --2- 172.21.15.37:0/1536808800 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc9a8135490 0x7fc9a809c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:28:23.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.071+0000 7fc9ad546640 1 -- 172.21.15.37:0/1536808800 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc9a80a0260 msgr2=0x7fc9a81323a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:23.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.071+0000 7fc9ad546640 1 --2- 172.21.15.37:0/1536808800 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc9a80a0260 0x7fc9a81323a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc99c0ad000 tx=0x7fc99c018f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:23.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.071+0000 7fc9a5ffb640 1 -- 172.21.15.37:0/1536808800 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc99c0ba070 con 0x7fc9a80a0260 2024-04-26T00:28:23.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.071+0000 7fc9a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:23.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.071+0000 7fc9a5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:28:23.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.071+0000 7fc9a5ffb640 10 monclient: dump: 2024-04-26T00:28:23.073 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:23.073 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:23.073 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:23.073 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:23.073 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:23.073 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:23.073 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:23.073 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:23.073 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:23.073 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:23.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.071+0000 7fc9a5ffb640 10 monclient: _finish_auth 0 2024-04-26T00:28:23.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.071+0000 7fc9a5ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:28:23.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.071+0000 7fc9a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:23.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.071+0000 7fc9a5ffb640 1 -- 172.21.15.37:0/1536808800 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc99c0b1040 con 0x7fc9a80a0260 2024-04-26T00:28:23.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.071+0000 7fc9a5ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:28:23.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.071+0000 7fc9a5ffb640 1 -- 172.21.15.37:0/1536808800 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc99c0b0070 con 0x7fc9a80a0260 2024-04-26T00:28:23.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.071+0000 7fc9a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:23.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.071+0000 7fc9a5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:28:23.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.071+0000 7fc9a5ffb640 10 monclient: dump: 2024-04-26T00:28:23.074 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:23.074 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:23.074 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:23.074 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:23.074 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:23.074 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:23.074 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:23.074 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:23.074 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:23.075 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:23.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.071+0000 7fc9af7aa640 5 monclient: authenticate success, global_id 94194 2024-04-26T00:28:23.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.071+0000 7fc9a77fe640 4 set_mon_vals no callback set 2024-04-26T00:28:23.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.071+0000 7fc9af7aa640 10 monclient: _renew_subs 2024-04-26T00:28:23.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.071+0000 7fc9af7aa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:28:23.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.071+0000 7fc9af7aa640 1 -- 172.21.15.37:0/1536808800 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9a80729d0 con 0x7fc9a80a0260 2024-04-26T00:28:23.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.071+0000 7fc9af7aa640 10 monclient: _renew_subs 2024-04-26T00:28:23.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.071+0000 7fc9af7aa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:28:23.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.071+0000 7fc9af7aa640 1 -- 172.21.15.37:0/1536808800 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9a809e100 con 0x7fc9a80a0260 2024-04-26T00:28:23.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.071+0000 7fc9af7aa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:23.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.071+0000 7fc9af7aa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:23.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.071+0000 7fc9af7aa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:28:23.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.071+0000 7fc9af7aa640 1 -- 172.21.15.37:0/1536808800 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9a8066f40 con 0x7fc9a80a0260 2024-04-26T00:28:23.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.071+0000 7fc9a5ffb640 1 -- 172.21.15.37:0/1536808800 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc99c0aa3a0 con 0x7fc9a80a0260 2024-04-26T00:28:23.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.071+0000 7fc9a5ffb640 1 --2- 172.21.15.37:0/1536808800 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc98c035e80 0x7fc98c038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:23.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.071+0000 7fc9a5ffb640 1 -- 172.21.15.37:0/1536808800 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(329..329 src has 177..329) ==== 3915+0+0 (secure 0 0 0) 0x7fc99c0e1c60 con 0x7fc9a80a0260 2024-04-26T00:28:23.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.071+0000 7fc9acd45640 1 --2- 172.21.15.37:0/1536808800 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc98c035e80 0x7fc98c038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:23.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.071+0000 7fc9acd45640 10 monclient: get_auth_request con 0x7fc98c035e80 auth_method 0 2024-04-26T00:28:23.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.075+0000 7fc9acd45640 1 --2- 172.21.15.37:0/1536808800 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc98c035e80 0x7fc98c038340 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc9940075b0 tx=0x7fc994007540 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:23.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.075+0000 7fc9a5ffb640 1 -- 172.21.15.37:0/1536808800 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc9a80a1130 con 0x7fc9a80a0260 2024-04-26T00:28:23.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.075+0000 7fc9a5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:23.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.075+0000 7fc9a5ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:28:23.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.299+0000 7fc9af7aa640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:28:23.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.299+0000 7fc9af7aa640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:28:23.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.299+0000 7fc9af7aa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:28:23.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.299+0000 7fc9af7aa640 1 -- 172.21.15.37:0/1536808800 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc9a806b9a0 con 0x7fc9a80a0260 2024-04-26T00:28:23.290 INFO:tasks.rados.rados.0.smithi045.stdout:1419: done (0 left) 2024-04-26T00:28:23.290 INFO:tasks.rados.rados.0.smithi045.stdout:1420: rollback oid 28 current snap is 164 2024-04-26T00:28:23.291 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 28 to 163 2024-04-26T00:28:23.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.303+0000 7fc9a5ffb640 1 -- 172.21.15.37:0/1536808800 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7fc99c0b4520 con 0x7fc9a80a0260 2024-04-26T00:28:23.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.303+0000 7fc9a5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:28:23.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.303+0000 7fc9a5ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:28:23.291 INFO:tasks.rados.rados.0.smithi045.stdout:1421: read oid 12 snap -1 2024-04-26T00:28:23.291 INFO:tasks.rados.rados.0.smithi045.stdout:1421: expect (ObjNum 477 snap 148 seq_num 477) 2024-04-26T00:28:23.292 INFO:tasks.rados.rados.0.smithi045.stdout:1422: snap_create 2024-04-26T00:28:23.292 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":62,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":31,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:28:23.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.311+0000 7fc9af7aa640 1 -- 172.21.15.37:0/1536808800 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc98c035e80 msgr2=0x7fc98c038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:23.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.311+0000 7fc9af7aa640 1 --2- 172.21.15.37:0/1536808800 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc98c035e80 0x7fc98c038340 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc9940075b0 tx=0x7fc994007540 comp rx=0 tx=0).stop 2024-04-26T00:28:23.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.311+0000 7fc9af7aa640 10 monclient: shutdown 2024-04-26T00:28:23.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.311+0000 7fc9af7aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:28:23.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.311+0000 7fc9af7aa640 1 -- 172.21.15.37:0/1536808800 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc9a80a0260 msgr2=0x7fc9a81323a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:23.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.311+0000 7fc9af7aa640 1 --2- 172.21.15.37:0/1536808800 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc9a80a0260 0x7fc9a81323a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc99c0ad000 tx=0x7fc99c018f60 comp rx=0 tx=0).stop 2024-04-26T00:28:23.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.311+0000 7fc9af7aa640 1 -- 172.21.15.37:0/1536808800 shutdown_connections 2024-04-26T00:28:23.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.311+0000 7fc9af7aa640 1 --2- 172.21.15.37:0/1536808800 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc98c035e80 0x7fc98c038340 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:23.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.311+0000 7fc9af7aa640 1 --2- 172.21.15.37:0/1536808800 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc9a8135490 0x7fc9a809c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:23.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.311+0000 7fc9af7aa640 1 --2- 172.21.15.37:0/1536808800 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc9a8134b30 0x7fc9a813c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:23.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.311+0000 7fc9af7aa640 1 --2- 172.21.15.37:0/1536808800 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc9a80a0260 0x7fc9a81323a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:23.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.311+0000 7fc9af7aa640 1 -- 172.21.15.37:0/1536808800 >> 172.21.15.37:0/1536808800 conn(0x7fc9a80a1ad0 msgr2=0x7fc9a80a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:28:23.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.311+0000 7fc9af7aa640 1 -- 172.21.15.37:0/1536808800 shutdown_connections 2024-04-26T00:28:23.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:23.311+0000 7fc9af7aa640 1 -- 172.21.15.37:0/1536808800 wait complete. 2024-04-26T00:28:23.304 INFO:tasks.rados.rados.0.smithi045.stdout:1420: finishing rollback tid 1 to smithi04526579-28 2024-04-26T00:28:23.304 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 1021 (ObjNum 97 snap 0 seq_num 2241979616) dirty dne 2024-04-26T00:28:23.306 INFO:tasks.rados.rados.0.smithi045.stdout:1421: expect (ObjNum 477 snap 148 seq_num 477) 2024-04-26T00:28:24.259 INFO:tasks.rados.rados.0.smithi045.stdout:1420: done (2 left) 2024-04-26T00:28:24.260 INFO:tasks.rados.rados.0.smithi045.stdout:1421: done (1 left) 2024-04-26T00:28:24.260 INFO:tasks.rados.rados.0.smithi045.stdout:1422: done (0 left) 2024-04-26T00:28:24.260 INFO:tasks.rados.rados.0.smithi045.stdout:1423: write (excl) oid 17 current snap is 165 2024-04-26T00:28:24.260 INFO:tasks.rados.rados.0.smithi045.stdout:1423: seq_num 530 ranges {507710=710327,1897927=412519,2854892=52818} 2024-04-26T00:28:24.279 INFO:tasks.rados.rados.0.smithi045.stdout:1423: writing smithi04526579-17 from 507710 to 1218037 tid 1 2024-04-26T00:28:24.298 INFO:tasks.rados.rados.0.smithi045.stdout:1423: writing smithi04526579-17 from 1897927 to 2310446 tid 2 2024-04-26T00:28:24.308 INFO:tasks.rados.rados.0.smithi045.stdout:1423: writing smithi04526579-17 from 2854892 to 2907710 tid 3 2024-04-26T00:28:24.308 INFO:tasks.rados.rados.0.smithi045.stdout:1424: write (excl) oid 27 current snap is 165 2024-04-26T00:28:24.308 INFO:tasks.rados.rados.0.smithi045.stdout:1424: seq_num 531 ranges {524517=785576,1956409=556177} 2024-04-26T00:28:24.325 INFO:tasks.rados.rados.0.smithi045.stdout:1424: writing smithi04526579-27 from 524517 to 1310093 tid 1 2024-04-26T00:28:24.341 INFO:tasks.rados.rados.0.smithi045.stdout:1424: writing smithi04526579-27 from 1956409 to 2512586 tid 2 2024-04-26T00:28:24.341 INFO:tasks.rados.rados.0.smithi045.stdout:1425: read oid 41 snap -1 2024-04-26T00:28:24.341 INFO:tasks.rados.rados.0.smithi045.stdout:1425: expect deleted 2024-04-26T00:28:24.341 INFO:tasks.rados.rados.0.smithi045.stdout:1426: read oid 39 snap -1 2024-04-26T00:28:24.341 INFO:tasks.rados.rados.0.smithi045.stdout:1426: expect (ObjNum 508 snap 157 seq_num 508) 2024-04-26T00:28:24.341 INFO:tasks.rados.rados.0.smithi045.stdout:1427: write oid 3 current snap is 165 2024-04-26T00:28:24.342 INFO:tasks.rados.rados.0.smithi045.stdout:1423: finishing write tid 1 to smithi04526579-17 2024-04-26T00:28:24.342 INFO:tasks.rados.rados.0.smithi045.stdout:1427: seq_num 532 ranges {541324=460825,1498538=416188,2393385=547939} 2024-04-26T00:28:24.347 INFO:tasks.rados.rados.0.smithi045.stdout:1427: writing smithi04526579-3 from 541324 to 1002149 tid 1 2024-04-26T00:28:24.356 INFO:tasks.rados.rados.0.smithi045.stdout:1427: writing smithi04526579-3 from 1498538 to 1914726 tid 2 2024-04-26T00:28:24.366 INFO:tasks.rados.rados.0.smithi045.stdout:1427: writing smithi04526579-3 from 2393385 to 2941324 tid 3 2024-04-26T00:28:24.366 INFO:tasks.rados.rados.0.smithi045.stdout:1428: rollback oid 46 current snap is 165 2024-04-26T00:28:24.366 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 46 to 140 2024-04-26T00:28:24.367 INFO:tasks.rados.rados.0.smithi045.stdout:1423: finishing write tid 2 to smithi04526579-17 2024-04-26T00:28:24.367 INFO:tasks.rados.rados.0.smithi045.stdout:1423: finishing write tid 3 to smithi04526579-17 2024-04-26T00:28:24.367 INFO:tasks.rados.rados.0.smithi045.stdout:1423: finishing write tid 4 to smithi04526579-17 2024-04-26T00:28:24.367 INFO:tasks.rados.rados.0.smithi045.stdout:1423: finishing write tid 5 to smithi04526579-17 2024-04-26T00:28:24.367 INFO:tasks.rados.rados.0.smithi045.stdout:1423: oid 17 updating version 0 to 857 2024-04-26T00:28:24.367 INFO:tasks.rados.rados.0.smithi045.stdout:1423: oid 17 version 857 is already newer than 855 2024-04-26T00:28:24.367 INFO:tasks.rados.rados.0.smithi045.stdout:1423: oid 17 version 857 is already newer than 856 2024-04-26T00:28:24.367 INFO:tasks.rados.rados.0.smithi045.stdout:1423: oid 17 version 857 is already newer than 854 2024-04-26T00:28:24.367 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 857 (ObjNum 530 snap 165 seq_num 530) dirty exists 2024-04-26T00:28:24.367 INFO:tasks.rados.rados.0.smithi045.stdout:1423: left oid 17 (ObjNum 530 snap 165 seq_num 530) 2024-04-26T00:28:24.367 INFO:tasks.rados.rados.0.smithi045.stdout:1424: finishing write tid 1 to smithi04526579-27 2024-04-26T00:28:24.367 INFO:tasks.rados.rados.0.smithi045.stdout:1424: finishing write tid 2 to smithi04526579-27 2024-04-26T00:28:24.367 INFO:tasks.rados.rados.0.smithi045.stdout:1424: finishing write tid 3 to smithi04526579-27 2024-04-26T00:28:24.367 INFO:tasks.rados.rados.0.smithi045.stdout:1423: done (5 left) 2024-04-26T00:28:24.368 INFO:tasks.rados.rados.0.smithi045.stdout:1425: done (4 left) 2024-04-26T00:28:24.368 INFO:tasks.rados.rados.0.smithi045.stdout:1429: write (excl) oid 9 current snap is 165 2024-04-26T00:28:24.368 INFO:tasks.rados.rados.0.smithi045.stdout:1429: seq_num 533 ranges {558131=652427,1673373=559846,2820808=137323} 2024-04-26T00:28:24.378 INFO:tasks.rados.rados.0.smithi045.stdout:1429: writing smithi04526579-9 from 558131 to 1210558 tid 1 2024-04-26T00:28:24.388 INFO:tasks.rados.rados.0.smithi045.stdout:1429: writing smithi04526579-9 from 1673373 to 2233219 tid 2 2024-04-26T00:28:24.395 INFO:tasks.rados.rados.0.smithi045.stdout:1429: writing smithi04526579-9 from 2820808 to 2958131 tid 3 2024-04-26T00:28:24.395 INFO:tasks.rados.rados.0.smithi045.stdout:1430: snap_remove snap 156 2024-04-26T00:28:25.305 INFO:tasks.rados.rados.0.smithi045.stdout:1430: done (5 left) 2024-04-26T00:28:25.305 INFO:tasks.rados.rados.0.smithi045.stdout:1431: write (excl) oid 19 current snap is 165 2024-04-26T00:28:25.306 INFO:tasks.rados.rados.0.smithi045.stdout:1431: seq_num 534 ranges {574938=727676,1731855=703504,2848231=126707} 2024-04-26T00:28:25.316 INFO:tasks.rados.rados.0.smithi045.stdout:1431: writing smithi04526579-19 from 574938 to 1302614 tid 1 2024-04-26T00:28:25.327 INFO:tasks.rados.rados.0.smithi045.stdout:1431: writing smithi04526579-19 from 1731855 to 2435359 tid 2 2024-04-26T00:28:25.332 INFO:tasks.rados.rados.0.smithi045.stdout:1431: writing smithi04526579-19 from 2848231 to 2974938 tid 3 2024-04-26T00:28:25.332 INFO:tasks.rados.rados.0.smithi045.stdout:1432: copy_from oid 30 from oid 17 current snap is 165 2024-04-26T00:28:25.332 INFO:tasks.rados.rados.0.smithi045.stdout:1424: finishing write tid 4 to smithi04526579-27 2024-04-26T00:28:25.332 INFO:tasks.rados.rados.0.smithi045.stdout:1424: oid 27 updating version 0 to 841 2024-04-26T00:28:25.332 INFO:tasks.rados.rados.0.smithi045.stdout:1424: oid 27 updating version 841 to 842 2024-04-26T00:28:25.332 INFO:tasks.rados.rados.0.smithi045.stdout:1424: oid 27 updating version 842 to 843 2024-04-26T00:28:25.332 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 843 (ObjNum 531 snap 165 seq_num 531) dirty exists 2024-04-26T00:28:25.332 INFO:tasks.rados.rados.0.smithi045.stdout:1424: left oid 27 (ObjNum 531 snap 165 seq_num 531) 2024-04-26T00:28:25.333 INFO:tasks.rados.rados.0.smithi045.stdout:1426: expect (ObjNum 508 snap 157 seq_num 508) 2024-04-26T00:28:25.466 INFO:tasks.rados.rados.0.smithi045.stdout:1427: finishing write tid 1 to smithi04526579-3 2024-04-26T00:28:25.466 INFO:tasks.rados.rados.0.smithi045.stdout:1427: finishing write tid 2 to smithi04526579-3 2024-04-26T00:28:25.466 INFO:tasks.rados.rados.0.smithi045.stdout:1427: finishing write tid 3 to smithi04526579-3 2024-04-26T00:28:25.466 INFO:tasks.rados.rados.0.smithi045.stdout:1428: finishing rollback tid 0 to smithi04526579-46 2024-04-26T00:28:25.466 INFO:tasks.rados.rados.0.smithi045.stdout:1427: finishing write tid 4 to smithi04526579-3 2024-04-26T00:28:25.466 INFO:tasks.rados.rados.0.smithi045.stdout:1428: finishing rollback tid 1 to smithi04526579-46 2024-04-26T00:28:25.466 INFO:tasks.rados.rados.0.smithi045.stdout:1427: finishing write tid 5 to smithi04526579-3 2024-04-26T00:28:25.466 INFO:tasks.rados.rados.0.smithi045.stdout:1427: oid 3 updating version 0 to 826 2024-04-26T00:28:25.466 INFO:tasks.rados.rados.0.smithi045.stdout:1427: oid 3 updating version 826 to 827 2024-04-26T00:28:25.467 INFO:tasks.rados.rados.0.smithi045.stdout:1427: oid 3 updating version 827 to 829 2024-04-26T00:28:25.467 INFO:tasks.rados.rados.0.smithi045.stdout:1427: oid 3 version 829 is already newer than 828 2024-04-26T00:28:25.467 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 829 (ObjNum 532 snap 165 seq_num 532) dirty exists 2024-04-26T00:28:25.467 INFO:tasks.rados.rados.0.smithi045.stdout:1427: left oid 3 (ObjNum 532 snap 165 seq_num 532) 2024-04-26T00:28:25.467 INFO:tasks.rados.rados.0.smithi045.stdout:1428: finishing rollback tid 2 to smithi04526579-46 2024-04-26T00:28:25.467 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 753 (ObjNum 386 snap 117 seq_num 386) dirty exists 2024-04-26T00:28:25.467 INFO:tasks.rados.rados.0.smithi045.stdout:1429: finishing write tid 1 to smithi04526579-9 2024-04-26T00:28:25.467 INFO:tasks.rados.rados.0.smithi045.stdout:1429: finishing write tid 2 to smithi04526579-9 2024-04-26T00:28:25.467 INFO:tasks.rados.rados.0.smithi045.stdout:1429: finishing write tid 3 to smithi04526579-9 2024-04-26T00:28:25.467 INFO:tasks.rados.rados.0.smithi045.stdout:1429: finishing write tid 4 to smithi04526579-9 2024-04-26T00:28:25.467 INFO:tasks.rados.rados.0.smithi045.stdout:1429: finishing write tid 5 to smithi04526579-9 2024-04-26T00:28:25.467 INFO:tasks.rados.rados.0.smithi045.stdout:1429: oid 9 updating version 0 to 792 2024-04-26T00:28:25.467 INFO:tasks.rados.rados.0.smithi045.stdout:1429: oid 9 updating version 792 to 794 2024-04-26T00:28:25.467 INFO:tasks.rados.rados.0.smithi045.stdout:1429: oid 9 version 794 is already newer than 793 2024-04-26T00:28:25.468 INFO:tasks.rados.rados.0.smithi045.stdout:1429: oid 9 version 794 is already newer than 791 2024-04-26T00:28:25.468 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 794 (ObjNum 533 snap 165 seq_num 533) dirty exists 2024-04-26T00:28:25.468 INFO:tasks.rados.rados.0.smithi045.stdout:1429: left oid 9 (ObjNum 533 snap 165 seq_num 533) 2024-04-26T00:28:25.468 INFO:tasks.rados.rados.0.smithi045.stdout:1431: finishing write tid 1 to smithi04526579-19 2024-04-26T00:28:25.468 INFO:tasks.rados.rados.0.smithi045.stdout:1431: finishing write tid 2 to smithi04526579-19 2024-04-26T00:28:25.468 INFO:tasks.rados.rados.0.smithi045.stdout:1431: finishing write tid 3 to smithi04526579-19 2024-04-26T00:28:25.468 INFO:tasks.rados.rados.0.smithi045.stdout:1431: finishing write tid 4 to smithi04526579-19 2024-04-26T00:28:25.468 INFO:tasks.rados.rados.0.smithi045.stdout:1431: finishing write tid 5 to smithi04526579-19 2024-04-26T00:28:25.468 INFO:tasks.rados.rados.0.smithi045.stdout:1431: oid 19 updating version 0 to 849 2024-04-26T00:28:25.468 INFO:tasks.rados.rados.0.smithi045.stdout:1431: oid 19 updating version 849 to 852 2024-04-26T00:28:25.468 INFO:tasks.rados.rados.0.smithi045.stdout:1431: oid 19 version 852 is already newer than 850 2024-04-26T00:28:25.468 INFO:tasks.rados.rados.0.smithi045.stdout:1431: oid 19 version 852 is already newer than 851 2024-04-26T00:28:25.468 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 852 (ObjNum 534 snap 165 seq_num 534) dirty exists 2024-04-26T00:28:25.468 INFO:tasks.rados.rados.0.smithi045.stdout:1431: left oid 19 (ObjNum 534 snap 165 seq_num 534) 2024-04-26T00:28:25.469 INFO:tasks.rados.rados.0.smithi045.stdout:1432: finishing copy_from racing read to smithi04526579-30 2024-04-26T00:28:25.469 INFO:tasks.rados.rados.0.smithi045.stdout:1432: finishing copy_from to smithi04526579-30 2024-04-26T00:28:25.469 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 857 (ObjNum 530 snap 165 seq_num 530) dirty exists 2024-04-26T00:28:25.469 INFO:tasks.rados.rados.0.smithi045.stdout:1424: done (6 left) 2024-04-26T00:28:25.469 INFO:tasks.rados.rados.0.smithi045.stdout:1426: done (5 left) 2024-04-26T00:28:25.469 INFO:tasks.rados.rados.0.smithi045.stdout:1427: done (4 left) 2024-04-26T00:28:25.469 INFO:tasks.rados.rados.0.smithi045.stdout:1428: done (3 left) 2024-04-26T00:28:25.469 INFO:tasks.rados.rados.0.smithi045.stdout:1429: done (2 left) 2024-04-26T00:28:25.469 INFO:tasks.rados.rados.0.smithi045.stdout:1431: done (1 left) 2024-04-26T00:28:25.469 INFO:tasks.rados.rados.0.smithi045.stdout:1432: done (0 left) 2024-04-26T00:28:25.469 INFO:tasks.rados.rados.0.smithi045.stdout:1433: snap_create 2024-04-26T00:28:26.317 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:28:26.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.463+0000 7f58ed49b640 10 monclient: get_monmap_and_config 2024-04-26T00:28:26.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.463+0000 7f58ed49b640 10 monclient: build_initial_monmap 2024-04-26T00:28:26.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.463+0000 7f58ed49b640 10 monclient: monmap: 2024-04-26T00:28:26.449 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:28:26.449 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:26.450 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:28:26.465165+0000 2024-04-26T00:28:26.450 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:28:26.465165+0000 2024-04-26T00:28:26.450 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:28:26.450 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:26.450 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:28:26.450 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:28:26.450 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:28:26.450 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:28:26.450 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:28:26.450 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:26.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.463+0000 7f58ed49b640 1 Processor -- start 2024-04-26T00:28:26.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.463+0000 7f58ed49b640 1 -- start start 2024-04-26T00:28:26.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.463+0000 7f58ed49b640 10 monclient: init 2024-04-26T00:28:26.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.463+0000 7f58ed49b640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:26.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.463+0000 7f58ed49b640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-04-26T00:28:26.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.463+0000 7f58ed49b640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f58e8133b20 0x7f58e809e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:26.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.463+0000 7f58ed49b640 10 monclient(hunting): picked mon.noname-a con 0x7f58e8133b20 addr v2:172.21.15.37:3300/0 2024-04-26T00:28:26.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.463+0000 7f58ed49b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f58e8133f00 0x7f58e8134350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:26.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.463+0000 7f58ed49b640 10 monclient(hunting): picked mon.noname-c con 0x7f58e8133f00 addr v2:172.21.15.45:3300/0 2024-04-26T00:28:26.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.463+0000 7f58ed49b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f58e8130770 addr v1:172.21.15.37:6789/0 2024-04-26T00:28:26.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.463+0000 7f58ed49b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58e809fe20 con 0x7f58e8130770 2024-04-26T00:28:26.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.463+0000 7f58ed49b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:26.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.463+0000 7f58ed49b640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f58e8072d40 con 0x7f58e8133b20 2024-04-26T00:28:26.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.463+0000 7f58ed49b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:26.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.463+0000 7f58ed49b640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f58e809ba60 con 0x7f58e8133f00 2024-04-26T00:28:26.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.463+0000 7f58ed49b640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:26.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.463+0000 7f58ed49b640 10 monclient(hunting): authenticate will time out at 1487.771380s 2024-04-26T00:28:26.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.463+0000 7f58e6ffd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f58e8133b20 0x7f58e809e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:26.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.463+0000 7f58e77fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f58e8130770 0x7f58e8130b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:57114/0 (socket says 172.21.15.37:57114) 2024-04-26T00:28:26.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.463+0000 7f58e77fe640 1 -- 172.21.15.37:0/2847531440 learned_addr learned my addr 172.21.15.37:0/2847531440 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:28:26.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.463+0000 7f58e6ffd640 10 monclient(hunting): get_auth_request con 0x7f58e8133b20 auth_method 0 2024-04-26T00:28:26.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.463+0000 7f58e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:26.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.463+0000 7f58e6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:26.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.463+0000 7f58e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:26.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.463+0000 7f58e67fc640 1 --2- 172.21.15.37:0/2847531440 >> v2:172.21.15.45:3300/0 conn(0x7f58e8133f00 0x7f58e8134350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:26.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.463+0000 7f58e67fc640 10 monclient(hunting): get_auth_request con 0x7f58e8133f00 auth_method 0 2024-04-26T00:28:26.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.463+0000 7f58e67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:26.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.463+0000 7f58e67fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:26.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.463+0000 7f58e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:26.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.467+0000 7f58e67fc640 1 -- 172.21.15.37:0/2847531440 >> v2:172.21.15.45:3300/0 conn(0x7f58e8133f00 msgr2=0x7f58e8134350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:28:26.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.467+0000 7f58e67fc640 1 -- 172.21.15.37:0/2847531440 >> v2:172.21.15.45:3300/0 conn(0x7f58e8133f00 msgr2=0x7f58e8134350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:28:26.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.467+0000 7f58e67fc640 1 --2- 172.21.15.37:0/2847531440 >> v2:172.21.15.45:3300/0 conn(0x7f58e8133f00 0x7f58e8134350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:28:26.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.467+0000 7f58e67fc640 1 --2- 172.21.15.37:0/2847531440 >> v2:172.21.15.45:3300/0 conn(0x7f58e8133f00 0x7f58e8134350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:28:26.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.467+0000 7f58e77fe640 1 -- 172.21.15.37:0/2847531440 >> v1:172.21.15.37:6789/0 conn(0x7f58e8130770 legacy=0x7f58e8130b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:26.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.467+0000 7f58e6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:26.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.467+0000 7f58e6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:26.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.467+0000 7f58e6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:26.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.467+0000 7f58e6ffd640 10 monclient(hunting): handle_auth_done global_id 94197 payload 274 2024-04-26T00:28:26.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.467+0000 7f58e6ffd640 1 -- 172.21.15.37:0/2847531440 >> v2:172.21.15.45:3300/0 conn(0x7f58e8133f00 msgr2=0x7f58e8134350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:28:26.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.467+0000 7f58e6ffd640 1 --2- 172.21.15.37:0/2847531440 >> v2:172.21.15.45:3300/0 conn(0x7f58e8133f00 0x7f58e8134350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:26.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.467+0000 7f58e6ffd640 1 -- 172.21.15.37:0/2847531440 >> v1:172.21.15.37:6789/0 conn(0x7f58e8130770 legacy=0x7f58e8130b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:26.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.467+0000 7f58e6ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:28:26.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.467+0000 7f58e6ffd640 1 monclient: found mon.noname-a 2024-04-26T00:28:26.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.467+0000 7f58e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:26.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.467+0000 7f58e6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:28:26.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.467+0000 7f58e6ffd640 1 -- 172.21.15.37:0/2847531440 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58e809dd10 con 0x7f58e8133b20 2024-04-26T00:28:26.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.471+0000 7f58e6ffd640 1 -- 172.21.15.37:0/2847531440 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f58e8133b20 msgr2=0x7f58e809e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:26.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.471+0000 7f58e6ffd640 1 --2- 172.21.15.37:0/2847531440 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f58e8133b20 0x7f58e809e9d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f58e809f670 tx=0x7f58d801c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bfd3719ea64fc59 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:26.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e5ffb640 1 -- 172.21.15.37:0/2847531440 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d80b2070 con 0x7f58e8133b20 2024-04-26T00:28:26.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:26.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:28:26.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e5ffb640 10 monclient: dump: 2024-04-26T00:28:26.461 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:26.461 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:26.461 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:26.461 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:26.461 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:26.461 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:26.461 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:26.461 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:26.461 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:26.461 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:26.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e5ffb640 10 monclient: _finish_auth 0 2024-04-26T00:28:26.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e5ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:28:26.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:26.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58ed49b640 5 monclient: authenticate success, global_id 94197 2024-04-26T00:28:26.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58ed49b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:28:26.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e5ffb640 1 -- 172.21.15.37:0/2847531440 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58d80a57c0 con 0x7f58e8133b20 2024-04-26T00:28:26.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e5ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:28:26.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e5ffb640 1 -- 172.21.15.37:0/2847531440 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d80a5b90 con 0x7f58e8133b20 2024-04-26T00:28:26.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:26.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:28:26.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e5ffb640 10 monclient: dump: 2024-04-26T00:28:26.462 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:26.462 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:26.462 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:26.463 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:26.463 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:26.463 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:26.463 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:26.463 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:26.463 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:26.463 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:26.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58ed49b640 10 monclient: get_monmap_and_config success 2024-04-26T00:28:26.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58ed49b640 4 set_mon_vals no callback set 2024-04-26T00:28:26.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58ed49b640 10 monclient: shutdown 2024-04-26T00:28:26.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58ed49b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:28:26.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58ed49b640 1 -- 172.21.15.37:0/2847531440 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f58e8133b20 msgr2=0x7f58e809e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:26.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58ed49b640 1 --2- 172.21.15.37:0/2847531440 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f58e8133b20 0x7f58e809e9d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f58e809f670 tx=0x7f58d801c900 comp rx=0 tx=0).stop 2024-04-26T00:28:26.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58ed49b640 1 -- 172.21.15.37:0/2847531440 shutdown_connections 2024-04-26T00:28:26.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58ed49b640 1 --2- 172.21.15.37:0/2847531440 >> v2:172.21.15.45:3300/0 conn(0x7f58e8133f00 0x7f58e8134350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:26.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58ed49b640 1 --2- 172.21.15.37:0/2847531440 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f58e8133b20 0x7f58e809e9d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:26.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58ed49b640 1 -- 172.21.15.37:0/2847531440 >> 172.21.15.37:0/2847531440 conn(0x7f58e80a1ad0 msgr2=0x7f58e80a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:28:26.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58ed49b640 1 -- 172.21.15.37:0/2847531440 shutdown_connections 2024-04-26T00:28:26.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58ed49b640 1 -- 172.21.15.37:0/2847531440 wait complete. 2024-04-26T00:28:26.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58ed49b640 10 monclient: build_initial_monmap 2024-04-26T00:28:26.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58ed49b640 10 monclient: monmap: 2024-04-26T00:28:26.469 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:28:26.469 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:28:26.469 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:28:26.469 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:28:26.469 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:28:26.469 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:28:26.469 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:28:26.469 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:28:26.469 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:28:26.470 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:26.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58ed49b640 1 Processor -- start 2024-04-26T00:28:26.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58ed49b640 1 -- start start 2024-04-26T00:28:26.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58ed49b640 10 monclient: init 2024-04-26T00:28:26.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58ed49b640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:26.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58ed49b640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:28:26.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58ed49b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f58e8130770 0x7f58e812fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:26.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58ed49b640 10 monclient(hunting): picked mon.noname-a con 0x7f58e8130770 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:28:26.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58ed49b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58e8133b20 0x7f58e81cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:26.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58ed49b640 10 monclient(hunting): picked mon.noname-b con 0x7f58e8133b20 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:28:26.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58ed49b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f58e8133f00 0x7f58e81cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:26.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58ed49b640 10 monclient(hunting): picked mon.noname-c con 0x7f58e8133f00 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:28:26.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58ed49b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:26.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58ed49b640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f58e80a03e0 con 0x7f58e8130770 2024-04-26T00:28:26.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58ed49b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:26.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58ed49b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f58e80a0260 con 0x7f58e8133b20 2024-04-26T00:28:26.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58ed49b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:26.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58ed49b640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f58e80a0560 con 0x7f58e8133f00 2024-04-26T00:28:26.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58ed49b640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:26.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58ed49b640 10 monclient(hunting): authenticate will time out at 1487.783141s 2024-04-26T00:28:26.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e6ffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f58e8130770 0x7f58e812fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:26.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e6ffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f58e8130770 0x7f58e812fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:56262/0 (socket says 172.21.15.37:56262) 2024-04-26T00:28:26.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e6ffd640 1 -- 172.21.15.37:0/3587320505 learned_addr learned my addr 172.21.15.37:0/3587320505 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:28:26.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e6ffd640 10 monclient(hunting): get_auth_request con 0x7f58e8130770 auth_method 0 2024-04-26T00:28:26.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:26.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:26.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:26.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e77fe640 1 --2- 172.21.15.37:0/3587320505 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f58e8133f00 0x7f58e81cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:26.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e77fe640 10 monclient(hunting): get_auth_request con 0x7f58e8133f00 auth_method 0 2024-04-26T00:28:26.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:26.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e77fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:26.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:26.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:26.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:26.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:26.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:26.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:26.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:26.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e67fc640 1 --2- 172.21.15.37:0/3587320505 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58e8133b20 0x7f58e81cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:26.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e67fc640 10 monclient(hunting): get_auth_request con 0x7f58e8133b20 auth_method 0 2024-04-26T00:28:26.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:26.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e67fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:26.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:26.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e6ffd640 10 monclient(hunting): handle_auth_done global_id 94203 payload 824 2024-04-26T00:28:26.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e6ffd640 1 -- 172.21.15.37:0/3587320505 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f58e8133f00 msgr2=0x7f58e81cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:26.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e6ffd640 1 --2- 172.21.15.37:0/3587320505 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f58e8133f00 0x7f58e81cbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:26.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e6ffd640 1 -- 172.21.15.37:0/3587320505 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58e8133b20 msgr2=0x7f58e81cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:26.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e67fc640 1 -- 172.21.15.37:0/3587320505 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58e8133b20 msgr2=0x7f58e81cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:28:26.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e67fc640 1 -- 172.21.15.37:0/3587320505 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58e8133b20 msgr2=0x7f58e81cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:28:26.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e67fc640 1 --2- 172.21.15.37:0/3587320505 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58e8133b20 0x7f58e81cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:28:26.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e67fc640 1 --2- 172.21.15.37:0/3587320505 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58e8133b20 0x7f58e81cb9a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:28:26.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e6ffd640 1 --2- 172.21.15.37:0/3587320505 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58e8133b20 0x7f58e81cb9a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:26.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e6ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:28:26.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e6ffd640 1 monclient: found mon.noname-a 2024-04-26T00:28:26.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:26.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:28:26.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e6ffd640 1 -- 172.21.15.37:0/3587320505 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58e8130ce0 con 0x7f58e8130770 2024-04-26T00:28:26.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.475+0000 7f58e77fe640 1 --2- 172.21.15.37:0/3587320505 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f58e8133f00 0x7f58e81cbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:28:26.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.479+0000 7f58e6ffd640 1 -- 172.21.15.37:0/3587320505 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f58e8130770 msgr2=0x7f58e812fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:26.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.479+0000 7f58e6ffd640 1 --2- 172.21.15.37:0/3587320505 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f58e8130770 0x7f58e812fe80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f58d8019510 tx=0x7f58d80196a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:26.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.479+0000 7f58cf7fe640 1 -- 172.21.15.37:0/3587320505 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d80b2050 con 0x7f58e8130770 2024-04-26T00:28:26.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.479+0000 7f58cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:26.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.479+0000 7f58cf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:28:26.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.479+0000 7f58cf7fe640 10 monclient: dump: 2024-04-26T00:28:26.477 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:26.477 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:26.477 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:26.477 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:26.477 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:26.477 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:26.477 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:26.477 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:26.477 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:26.477 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:26.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.479+0000 7f58cf7fe640 10 monclient: _finish_auth 0 2024-04-26T00:28:26.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.479+0000 7f58cf7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:28:26.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.479+0000 7f58cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:26.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.479+0000 7f58cf7fe640 1 -- 172.21.15.37:0/3587320505 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58d80be070 con 0x7f58e8130770 2024-04-26T00:28:26.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.479+0000 7f58cf7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:28:26.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.479+0000 7f58cf7fe640 1 -- 172.21.15.37:0/3587320505 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d80adc80 con 0x7f58e8130770 2024-04-26T00:28:26.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.479+0000 7f58ed49b640 5 monclient: authenticate success, global_id 94203 2024-04-26T00:28:26.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.479+0000 7f58ed49b640 10 monclient: _renew_subs 2024-04-26T00:28:26.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.479+0000 7f58ed49b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:28:26.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.479+0000 7f58ed49b640 1 -- 172.21.15.37:0/3587320505 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58e811be30 con 0x7f58e8130770 2024-04-26T00:28:26.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.479+0000 7f58cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:26.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.479+0000 7f58cf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:28:26.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.479+0000 7f58e57fa640 4 set_mon_vals no callback set 2024-04-26T00:28:26.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.479+0000 7f58cf7fe640 10 monclient: dump: 2024-04-26T00:28:26.479 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:26.479 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:26.479 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:26.479 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:26.479 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:26.479 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:26.479 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:26.479 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:26.479 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:26.479 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:26.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.479+0000 7f58ed49b640 10 monclient: _renew_subs 2024-04-26T00:28:26.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.479+0000 7f58ed49b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:28:26.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.479+0000 7f58ed49b640 1 -- 172.21.15.37:0/3587320505 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58e8130590 con 0x7f58e8130770 2024-04-26T00:28:26.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.479+0000 7f58ed49b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:26.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.479+0000 7f58ed49b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:26.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.479+0000 7f58ed49b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:28:26.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.479+0000 7f58e6ffd640 1 --2- 172.21.15.37:0/3587320505 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f58e8130770 0x7f58e812fe80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f58d8019510 tx=0x7f58d80196a0 comp rx=0 tx=0).queue_received will delay after 1.71409e+09 on 0x7f58d80c3460 mgrmap(e 5) 2024-04-26T00:28:26.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.479+0000 7f58ed49b640 1 -- 172.21.15.37:0/3587320505 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58e8066f40 con 0x7f58e8130770 2024-04-26T00:28:26.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.479+0000 7f58cf7fe640 1 -- 172.21.15.37:0/3587320505 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f58d80c3460 con 0x7f58e8130770 2024-04-26T00:28:26.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.479+0000 7f58cf7fe640 1 --2- 172.21.15.37:0/3587320505 >> v2:172.21.15.45:6812/2379918623 conn(0x7f58d4035eb0 0x7f58d4038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:26.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.483+0000 7f58e67fc640 1 --2- 172.21.15.37:0/3587320505 >> v2:172.21.15.45:6812/2379918623 conn(0x7f58d4035eb0 0x7f58d4038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:26.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.483+0000 7f58e67fc640 10 monclient: get_auth_request con 0x7f58d4035eb0 auth_method 0 2024-04-26T00:28:26.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.483+0000 7f58e67fc640 1 --2- 172.21.15.37:0/3587320505 >> v2:172.21.15.45:6812/2379918623 conn(0x7f58d4035eb0 0x7f58d4038370 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f58dc006aa0 tx=0x7f58dc006a30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:26.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:26.487+0000 7f58cf7fe640 1 -- 172.21.15.37:0/3587320505 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(332..332 src has 177..332) ==== 3943+0+0 (secure 0 0 0) 0x7f58d80e3d00 con 0x7f58e8130770 2024-04-26T00:28:26.851 INFO:tasks.rados.rados.0.smithi045.stdout:1433: done (0 left) 2024-04-26T00:28:26.851 INFO:tasks.rados.rados.0.smithi045.stdout:1434: copy_from oid 20 from oid 17 current snap is 166 2024-04-26T00:28:26.851 INFO:tasks.rados.rados.0.smithi045.stdout:1435: snap_create 2024-04-26T00:28:26.883 INFO:tasks.rados.rados.0.smithi045.stdout:1434: finishing copy_from racing read to smithi04526579-20 2024-04-26T00:28:26.885 INFO:tasks.rados.rados.0.smithi045.stdout:1434: finishing copy_from to smithi04526579-20 2024-04-26T00:28:26.886 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 857 (ObjNum 530 snap 165 seq_num 530) dirty exists 2024-04-26T00:28:27.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:27.307+0000 7f58cf7fe640 1 -- 172.21.15.37:0/3587320505 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f58e8130590 con 0x7f58e8130770 2024-04-26T00:28:27.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:27.307+0000 7f58cf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:27.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:27.307+0000 7f58cf7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:28:27.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:27.543+0000 7f58ed49b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:28:27.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:27.543+0000 7f58ed49b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:28:27.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:27.543+0000 7f58ed49b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:28:27.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:27.543+0000 7f58ed49b640 1 -- 172.21.15.37:0/3587320505 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f58e811d1c0 con 0x7f58e8130770 2024-04-26T00:28:27.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:27.543+0000 7f58cf7fe640 1 -- 172.21.15.37:0/3587320505 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f58d8016580 con 0x7f58e8130770 2024-04-26T00:28:27.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:27.543+0000 7f58cf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:28:27.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:27.543+0000 7f58cf7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:28:27.530 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":62,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":35,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:28:27.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:27.543+0000 7f58ed49b640 1 -- 172.21.15.37:0/3587320505 >> v2:172.21.15.45:6812/2379918623 conn(0x7f58d4035eb0 msgr2=0x7f58d4038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:27.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:27.543+0000 7f58ed49b640 1 --2- 172.21.15.37:0/3587320505 >> v2:172.21.15.45:6812/2379918623 conn(0x7f58d4035eb0 0x7f58d4038370 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f58dc006aa0 tx=0x7f58dc006a30 comp rx=0 tx=0).stop 2024-04-26T00:28:27.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:27.543+0000 7f58ed49b640 10 monclient: shutdown 2024-04-26T00:28:27.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:27.543+0000 7f58ed49b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:28:27.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:27.543+0000 7f58ed49b640 1 -- 172.21.15.37:0/3587320505 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f58e8130770 msgr2=0x7f58e812fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:27.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:27.543+0000 7f58ed49b640 1 --2- 172.21.15.37:0/3587320505 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f58e8130770 0x7f58e812fe80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f58d8019510 tx=0x7f58d80196a0 comp rx=0 tx=0).stop 2024-04-26T00:28:27.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:27.543+0000 7f58ed49b640 1 -- 172.21.15.37:0/3587320505 shutdown_connections 2024-04-26T00:28:27.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:27.543+0000 7f58ed49b640 1 --2- 172.21.15.37:0/3587320505 >> v2:172.21.15.45:6812/2379918623 conn(0x7f58d4035eb0 0x7f58d4038370 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:27.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:27.543+0000 7f58ed49b640 1 --2- 172.21.15.37:0/3587320505 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f58e8133f00 0x7f58e81cbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:27.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:27.543+0000 7f58ed49b640 1 --2- 172.21.15.37:0/3587320505 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58e8133b20 0x7f58e81cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:27.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:27.543+0000 7f58ed49b640 1 --2- 172.21.15.37:0/3587320505 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f58e8130770 0x7f58e812fe80 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:27.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:27.543+0000 7f58ed49b640 1 -- 172.21.15.37:0/3587320505 >> 172.21.15.37:0/3587320505 conn(0x7f58e80a1ad0 msgr2=0x7f58e809c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:28:27.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:27.543+0000 7f58ed49b640 1 -- 172.21.15.37:0/3587320505 shutdown_connections 2024-04-26T00:28:27.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:27.543+0000 7f58ed49b640 1 -- 172.21.15.37:0/3587320505 wait complete. 2024-04-26T00:28:27.865 INFO:tasks.rados.rados.0.smithi045.stdout:1434: done (1 left) 2024-04-26T00:28:27.866 INFO:tasks.rados.rados.0.smithi045.stdout:1435: done (0 left) 2024-04-26T00:28:27.866 INFO:tasks.rados.rados.0.smithi045.stdout:1436: read oid 27 snap -1 2024-04-26T00:28:27.866 INFO:tasks.rados.rados.0.smithi045.stdout:1436: expect (ObjNum 531 snap 165 seq_num 531) 2024-04-26T00:28:27.866 INFO:tasks.rados.rados.0.smithi045.stdout:1437: read oid 43 snap 160 2024-04-26T00:28:27.866 INFO:tasks.rados.rados.0.smithi045.stdout:1437: expect (ObjNum 329 snap 96 seq_num 329) 2024-04-26T00:28:27.866 INFO:tasks.rados.rados.0.smithi045.stdout:1438: delete oid 47 current snap is 167 2024-04-26T00:28:27.880 INFO:tasks.rados.rados.0.smithi045.stdout:1438: done (2 left) 2024-04-26T00:28:27.880 INFO:tasks.rados.rados.0.smithi045.stdout:1439: rollback oid 7 current snap is 167 2024-04-26T00:28:27.880 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 7 to 165 2024-04-26T00:28:27.880 INFO:tasks.rados.rados.0.smithi045.stdout:1440: write oid 41 current snap is 167 2024-04-26T00:28:27.880 INFO:tasks.rados.rados.0.smithi045.stdout:1440: seq_num 536 ranges {608552=478174,1848819=707173} 2024-04-26T00:28:27.898 INFO:tasks.rados.rados.0.smithi045.stdout:1440: writing smithi04526579-41 from 608552 to 1086726 tid 1 2024-04-26T00:28:27.922 INFO:tasks.rados.rados.0.smithi045.stdout:1440: writing smithi04526579-41 from 1848819 to 2555992 tid 2 2024-04-26T00:28:27.923 INFO:tasks.rados.rados.0.smithi045.stdout:1441: delete oid 15 current snap is 167 2024-04-26T00:28:27.923 INFO:tasks.rados.rados.0.smithi045.stdout:1437: expect (ObjNum 329 snap 96 seq_num 329) 2024-04-26T00:28:28.095 INFO:tasks.rados.rados.0.smithi045.stdout:1436: expect (ObjNum 531 snap 165 seq_num 531) 2024-04-26T00:28:28.281 INFO:tasks.rados.rados.0.smithi045.stdout:1439: finishing rollback tid 0 to smithi04526579-7 2024-04-26T00:28:28.282 INFO:tasks.rados.rados.0.smithi045.stdout:1439: finishing rollback tid 1 to smithi04526579-7 2024-04-26T00:28:28.282 INFO:tasks.rados.rados.0.smithi045.stdout:1439: finishing rollback tid 2 to smithi04526579-7 2024-04-26T00:28:28.282 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 1116 (ObjNum 114 snap 28 seq_num 114) dirty exists 2024-04-26T00:28:28.282 INFO:tasks.rados.rados.0.smithi045.stdout:1440: finishing write tid 1 to smithi04526579-41 2024-04-26T00:28:28.282 INFO:tasks.rados.rados.0.smithi045.stdout:1440: finishing write tid 2 to smithi04526579-41 2024-04-26T00:28:28.282 INFO:tasks.rados.rados.0.smithi045.stdout:1440: finishing write tid 3 to smithi04526579-41 2024-04-26T00:28:28.282 INFO:tasks.rados.rados.0.smithi045.stdout:1440: finishing write tid 4 to smithi04526579-41 2024-04-26T00:28:28.282 INFO:tasks.rados.rados.0.smithi045.stdout:1440: oid 41 updating version 0 to 1035 2024-04-26T00:28:28.282 INFO:tasks.rados.rados.0.smithi045.stdout:1440: oid 41 version 1035 is already newer than 1034 2024-04-26T00:28:28.282 INFO:tasks.rados.rados.0.smithi045.stdout:1440: oid 41 version 1035 is already newer than 1033 2024-04-26T00:28:28.282 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 1035 (ObjNum 536 snap 167 seq_num 536) dirty exists 2024-04-26T00:28:28.282 INFO:tasks.rados.rados.0.smithi045.stdout:1440: left oid 41 (ObjNum 536 snap 167 seq_num 536) 2024-04-26T00:28:28.283 INFO:tasks.rados.rados.0.smithi045.stdout:1436: done (4 left) 2024-04-26T00:28:28.283 INFO:tasks.rados.rados.0.smithi045.stdout:1437: done (3 left) 2024-04-26T00:28:28.283 INFO:tasks.rados.rados.0.smithi045.stdout:1439: done (2 left) 2024-04-26T00:28:28.283 INFO:tasks.rados.rados.0.smithi045.stdout:1440: done (1 left) 2024-04-26T00:28:28.283 INFO:tasks.rados.rados.0.smithi045.stdout:1441: done (0 left) 2024-04-26T00:28:28.283 INFO:tasks.rados.rados.0.smithi045.stdout:1442: snap_remove snap 166 2024-04-26T00:28:28.874 INFO:tasks.rados.rados.0.smithi045.stdout:1442: done (0 left) 2024-04-26T00:28:28.874 INFO:tasks.rados.rados.0.smithi045.stdout:1443: read oid 16 snap -1 2024-04-26T00:28:28.874 INFO:tasks.rados.rados.0.smithi045.stdout:1443: expect (ObjNum 515 snap 157 seq_num 515) 2024-04-26T00:28:28.874 INFO:tasks.rados.rados.0.smithi045.stdout:1444: delete oid 46 current snap is 167 2024-04-26T00:28:28.883 INFO:tasks.rados.rados.0.smithi045.stdout:1444: done (1 left) 2024-04-26T00:28:28.883 INFO:tasks.rados.rados.0.smithi045.stdout:1445: write oid 37 current snap is 167 2024-04-26T00:28:28.883 INFO:tasks.rados.rados.0.smithi045.stdout:1445: seq_num 539 ranges {658973=703921,1907912=570853,2985346=73627} 2024-04-26T00:28:28.905 INFO:tasks.rados.rados.0.smithi045.stdout:1445: writing smithi04526579-37 from 658973 to 1362894 tid 1 2024-04-26T00:28:28.924 INFO:tasks.rados.rados.0.smithi045.stdout:1445: writing smithi04526579-37 from 1907912 to 2478765 tid 2 2024-04-26T00:28:28.933 INFO:tasks.rados.rados.0.smithi045.stdout:1445: writing smithi04526579-37 from 2985346 to 3058973 tid 3 2024-04-26T00:28:28.933 INFO:tasks.rados.rados.0.smithi045.stdout:1446: read oid 4 snap -1 2024-04-26T00:28:28.933 INFO:tasks.rados.rados.0.smithi045.stdout:1446: expect (ObjNum 474 snap 147 seq_num 474) 2024-04-26T00:28:28.933 INFO:tasks.rados.rados.0.smithi045.stdout:1447: delete oid 2 current snap is 167 2024-04-26T00:28:28.933 INFO:tasks.rados.rados.0.smithi045.stdout:1443: expect (ObjNum 515 snap 157 seq_num 515) 2024-04-26T00:28:29.099 INFO:tasks.rados.rados.0.smithi045.stdout:1445: finishing write tid 1 to smithi04526579-37 2024-04-26T00:28:29.099 INFO:tasks.rados.rados.0.smithi045.stdout:1445: finishing write tid 2 to smithi04526579-37 2024-04-26T00:28:29.099 INFO:tasks.rados.rados.0.smithi045.stdout:1446: expect (ObjNum 474 snap 147 seq_num 474) 2024-04-26T00:28:29.349 INFO:tasks.rados.rados.0.smithi045.stdout:1445: finishing write tid 3 to smithi04526579-37 2024-04-26T00:28:29.350 INFO:tasks.rados.rados.0.smithi045.stdout:1445: finishing write tid 4 to smithi04526579-37 2024-04-26T00:28:29.350 INFO:tasks.rados.rados.0.smithi045.stdout:1445: finishing write tid 5 to smithi04526579-37 2024-04-26T00:28:29.350 INFO:tasks.rados.rados.0.smithi045.stdout:1445: oid 37 updating version 0 to 1120 2024-04-26T00:28:29.350 INFO:tasks.rados.rados.0.smithi045.stdout:1445: oid 37 updating version 1120 to 1121 2024-04-26T00:28:29.350 INFO:tasks.rados.rados.0.smithi045.stdout:1445: oid 37 updating version 1121 to 1123 2024-04-26T00:28:29.350 INFO:tasks.rados.rados.0.smithi045.stdout:1445: oid 37 version 1123 is already newer than 1122 2024-04-26T00:28:29.350 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 1123 (ObjNum 539 snap 167 seq_num 539) dirty exists 2024-04-26T00:28:29.350 INFO:tasks.rados.rados.0.smithi045.stdout:1445: left oid 37 (ObjNum 539 snap 167 seq_num 539) 2024-04-26T00:28:29.350 INFO:tasks.rados.rados.0.smithi045.stdout:1443: done (3 left) 2024-04-26T00:28:29.350 INFO:tasks.rados.rados.0.smithi045.stdout:1445: done (2 left) 2024-04-26T00:28:29.350 INFO:tasks.rados.rados.0.smithi045.stdout:1446: done (1 left) 2024-04-26T00:28:29.350 INFO:tasks.rados.rados.0.smithi045.stdout:1447: done (0 left) 2024-04-26T00:28:29.350 INFO:tasks.rados.rados.0.smithi045.stdout:1448: rollback oid 50 current snap is 167 2024-04-26T00:28:29.350 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 50 to 164 2024-04-26T00:28:29.351 INFO:tasks.rados.rados.0.smithi045.stdout:1449: read oid 23 snap -1 2024-04-26T00:28:29.351 INFO:tasks.rados.rados.0.smithi045.stdout:1449: expect (ObjNum 453 snap 141 seq_num 453) 2024-04-26T00:28:29.351 INFO:tasks.rados.rados.0.smithi045.stdout:1450: snap_remove snap 147 2024-04-26T00:28:29.899 INFO:tasks.rados.rados.0.smithi045.stdout:1450: done (2 left) 2024-04-26T00:28:29.900 INFO:tasks.rados.rados.0.smithi045.stdout:1451: write (excl) oid 38 current snap is 167 2024-04-26T00:28:29.900 INFO:tasks.rados.rados.0.smithi045.stdout:1451: seq_num 541 ranges {692587=570772,2024876=574522,3040192=52395} 2024-04-26T00:28:29.922 INFO:tasks.rados.rados.0.smithi045.stdout:1451: writing smithi04526579-38 from 692587 to 1263359 tid 1 2024-04-26T00:28:29.944 INFO:tasks.rados.rados.0.smithi045.stdout:1451: writing smithi04526579-38 from 2024876 to 2599398 tid 2 2024-04-26T00:28:29.952 INFO:tasks.rados.rados.0.smithi045.stdout:1451: writing smithi04526579-38 from 3040192 to 3092587 tid 3 2024-04-26T00:28:29.952 INFO:tasks.rados.rados.0.smithi045.stdout:1452: read oid 27 snap -1 2024-04-26T00:28:29.952 INFO:tasks.rados.rados.0.smithi045.stdout:1452: expect (ObjNum 531 snap 165 seq_num 531) 2024-04-26T00:28:29.952 INFO:tasks.rados.rados.0.smithi045.stdout:1453: write (excl) oid 20 current snap is 167 2024-04-26T00:28:29.952 INFO:tasks.rados.rados.0.smithi045.stdout:1453: seq_num 542 ranges {709394=646021,2083358=718180} 2024-04-26T00:28:29.971 INFO:tasks.rados.rados.0.smithi045.stdout:1453: writing smithi04526579-20 from 709394 to 1355415 tid 1 2024-04-26T00:28:29.991 INFO:tasks.rados.rados.0.smithi045.stdout:1453: writing smithi04526579-20 from 2083358 to 2801538 tid 2 2024-04-26T00:28:29.991 INFO:tasks.rados.rados.0.smithi045.stdout:1454: write oid 49 current snap is 167 2024-04-26T00:28:29.991 INFO:tasks.rados.rados.0.smithi045.stdout:1454: seq_num 543 ranges {726201=721270,2141840=578191} 2024-04-26T00:28:30.005 INFO:tasks.rados.rados.0.smithi045.stdout:1454: writing smithi04526579-49 from 726201 to 1447471 tid 1 2024-04-26T00:28:30.017 INFO:tasks.rados.rados.0.smithi045.stdout:1454: writing smithi04526579-49 from 2141840 to 2720031 tid 2 2024-04-26T00:28:30.017 INFO:tasks.rados.rados.0.smithi045.stdout:1455: copy_from oid 24 from oid 31 current snap is 167 2024-04-26T00:28:30.017 INFO:tasks.rados.rados.0.smithi045.stdout:1449: expect (ObjNum 453 snap 141 seq_num 453) 2024-04-26T00:28:30.192 INFO:tasks.rados.rados.0.smithi045.stdout:1448: finishing rollback tid 0 to smithi04526579-50 2024-04-26T00:28:30.192 INFO:tasks.rados.rados.0.smithi045.stdout:1448: finishing rollback tid 1 to smithi04526579-50 2024-04-26T00:28:30.192 INFO:tasks.rados.rados.0.smithi045.stdout:1448: finishing rollback tid 2 to smithi04526579-50 2024-04-26T00:28:30.193 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 860 (ObjNum 470 snap 147 seq_num 470) dirty exists 2024-04-26T00:28:30.193 INFO:tasks.rados.rados.0.smithi045.stdout:1451: finishing write tid 1 to smithi04526579-38 2024-04-26T00:28:30.193 INFO:tasks.rados.rados.0.smithi045.stdout:1451: finishing write tid 2 to smithi04526579-38 2024-04-26T00:28:30.193 INFO:tasks.rados.rados.0.smithi045.stdout:1451: finishing write tid 3 to smithi04526579-38 2024-04-26T00:28:30.193 INFO:tasks.rados.rados.0.smithi045.stdout:1452: expect (ObjNum 531 snap 165 seq_num 531) 2024-04-26T00:28:30.378 INFO:tasks.rados.rados.0.smithi045.stdout:1451: finishing write tid 4 to smithi04526579-38 2024-04-26T00:28:30.378 INFO:tasks.rados.rados.0.smithi045.stdout:1451: finishing write tid 5 to smithi04526579-38 2024-04-26T00:28:30.378 INFO:tasks.rados.rados.0.smithi045.stdout:1451: oid 38 updating version 0 to 859 2024-04-26T00:28:30.378 INFO:tasks.rados.rados.0.smithi045.stdout:1451: oid 38 updating version 859 to 861 2024-04-26T00:28:30.378 INFO:tasks.rados.rados.0.smithi045.stdout:1451: oid 38 version 861 is already newer than 858 2024-04-26T00:28:30.378 INFO:tasks.rados.rados.0.smithi045.stdout:1451: oid 38 version 861 is already newer than 860 2024-04-26T00:28:30.379 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 861 (ObjNum 541 snap 167 seq_num 541) dirty exists 2024-04-26T00:28:30.379 INFO:tasks.rados.rados.0.smithi045.stdout:1451: left oid 38 (ObjNum 541 snap 167 seq_num 541) 2024-04-26T00:28:30.379 INFO:tasks.rados.rados.0.smithi045.stdout:1453: finishing write tid 1 to smithi04526579-20 2024-04-26T00:28:30.379 INFO:tasks.rados.rados.0.smithi045.stdout:1453: finishing write tid 2 to smithi04526579-20 2024-04-26T00:28:30.379 INFO:tasks.rados.rados.0.smithi045.stdout:1453: finishing write tid 3 to smithi04526579-20 2024-04-26T00:28:30.379 INFO:tasks.rados.rados.0.smithi045.stdout:1453: finishing write tid 4 to smithi04526579-20 2024-04-26T00:28:30.379 INFO:tasks.rados.rados.0.smithi045.stdout:1453: oid 20 updating version 0 to 863 2024-04-26T00:28:30.379 INFO:tasks.rados.rados.0.smithi045.stdout:1453: oid 20 version 863 is already newer than 862 2024-04-26T00:28:30.379 INFO:tasks.rados.rados.0.smithi045.stdout:1453: oid 20 updating version 863 to 864 2024-04-26T00:28:30.379 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 864 (ObjNum 542 snap 167 seq_num 542) dirty exists 2024-04-26T00:28:30.379 INFO:tasks.rados.rados.0.smithi045.stdout:1453: left oid 20 (ObjNum 542 snap 167 seq_num 542) 2024-04-26T00:28:30.379 INFO:tasks.rados.rados.0.smithi045.stdout:1454: finishing write tid 1 to smithi04526579-49 2024-04-26T00:28:30.379 INFO:tasks.rados.rados.0.smithi045.stdout:1455: finishing copy_from to smithi04526579-24 2024-04-26T00:28:30.380 INFO:tasks.rados.rados.0.smithi045.stdout:1455: got expected ENOENT (src dne) 2024-04-26T00:28:30.380 INFO:tasks.rados.rados.0.smithi045.stdout:1455: finishing copy_from racing read to smithi04526579-24 2024-04-26T00:28:30.380 INFO:tasks.rados.rados.0.smithi045.stdout:1454: finishing write tid 2 to smithi04526579-49 2024-04-26T00:28:30.380 INFO:tasks.rados.rados.0.smithi045.stdout:1454: finishing write tid 3 to smithi04526579-49 2024-04-26T00:28:30.380 INFO:tasks.rados.rados.0.smithi045.stdout:1454: finishing write tid 4 to smithi04526579-49 2024-04-26T00:28:30.380 INFO:tasks.rados.rados.0.smithi045.stdout:1454: oid 49 updating version 0 to 839 2024-04-26T00:28:30.380 INFO:tasks.rados.rados.0.smithi045.stdout:1454: oid 49 updating version 839 to 840 2024-04-26T00:28:30.380 INFO:tasks.rados.rados.0.smithi045.stdout:1454: oid 49 updating version 840 to 841 2024-04-26T00:28:30.380 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 841 (ObjNum 543 snap 167 seq_num 543) dirty exists 2024-04-26T00:28:30.380 INFO:tasks.rados.rados.0.smithi045.stdout:1454: left oid 49 (ObjNum 543 snap 167 seq_num 543) 2024-04-26T00:28:30.380 INFO:tasks.rados.rados.0.smithi045.stdout:1448: done (6 left) 2024-04-26T00:28:30.380 INFO:tasks.rados.rados.0.smithi045.stdout:1449: done (5 left) 2024-04-26T00:28:30.380 INFO:tasks.rados.rados.0.smithi045.stdout:1451: done (4 left) 2024-04-26T00:28:30.380 INFO:tasks.rados.rados.0.smithi045.stdout:1452: done (3 left) 2024-04-26T00:28:30.380 INFO:tasks.rados.rados.0.smithi045.stdout:1453: done (2 left) 2024-04-26T00:28:30.381 INFO:tasks.rados.rados.0.smithi045.stdout:1454: done (1 left) 2024-04-26T00:28:30.381 INFO:tasks.rados.rados.0.smithi045.stdout:1455: done (0 left) 2024-04-26T00:28:30.381 INFO:tasks.rados.rados.0.smithi045.stdout:1456: snap_create 2024-04-26T00:28:30.549 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:28:30.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.695+0000 7f5eb61ea640 10 monclient: get_monmap_and_config 2024-04-26T00:28:30.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.695+0000 7f5eb61ea640 10 monclient: build_initial_monmap 2024-04-26T00:28:30.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.695+0000 7f5eb61ea640 10 monclient: monmap: 2024-04-26T00:28:30.682 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:28:30.682 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:30.682 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:28:30.697189+0000 2024-04-26T00:28:30.682 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:28:30.697189+0000 2024-04-26T00:28:30.682 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:28:30.682 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:30.682 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:28:30.682 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:28:30.682 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:28:30.682 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:28:30.682 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:28:30.682 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:30.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.695+0000 7f5eb61ea640 1 Processor -- start 2024-04-26T00:28:30.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.695+0000 7f5eb61ea640 1 -- start start 2024-04-26T00:28:30.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.695+0000 7f5eb61ea640 10 monclient: init 2024-04-26T00:28:30.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.695+0000 7f5eb61ea640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:30.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.695+0000 7f5eb61ea640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-04-26T00:28:30.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.695+0000 7f5eb61ea640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f5eb0071fc0 0x7f5eb0137240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:30.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.695+0000 7f5eb61ea640 10 monclient(hunting): picked mon.noname-a con 0x7f5eb0071fc0 addr v2:172.21.15.37:3300/0 2024-04-26T00:28:30.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.695+0000 7f5eb61ea640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5eb0137780 addr v1:172.21.15.45:6789/0 2024-04-26T00:28:30.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.695+0000 7f5eb61ea640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5eb0072ab0 addr v1:172.21.15.37:6789/0 2024-04-26T00:28:30.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.695+0000 7f5eb61ea640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5eb0073200 con 0x7f5eb0072ab0 2024-04-26T00:28:30.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.695+0000 7f5eb61ea640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5eb00a0c20 con 0x7f5eb0137780 2024-04-26T00:28:30.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.695+0000 7f5eb61ea640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:30.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.695+0000 7f5eb61ea640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f5eb00a0f00 con 0x7f5eb0071fc0 2024-04-26T00:28:30.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.695+0000 7f5eb61ea640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:30.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.695+0000 7f5eb61ea640 10 monclient(hunting): authenticate will time out at 1492.002907s 2024-04-26T00:28:30.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.695+0000 7f5eaf7fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f5eb0071fc0 0x7f5eb0137240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:30.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.695+0000 7f5eaffff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f5eb0072ab0 0x7f5eb013e350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:34660/0 (socket says 172.21.15.37:34660) 2024-04-26T00:28:30.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.695+0000 7f5eaffff640 1 -- 172.21.15.37:0/1904486994 learned_addr learned my addr 172.21.15.37:0/1904486994 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:28:30.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.695+0000 7f5eaf7fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f5eb0071fc0 0x7f5eb0137240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:49726/0 (socket says 172.21.15.37:49726) 2024-04-26T00:28:30.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.695+0000 7f5eaf7fe640 10 monclient(hunting): get_auth_request con 0x7f5eb0071fc0 auth_method 0 2024-04-26T00:28:30.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.695+0000 7f5eaf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:30.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.695+0000 7f5eaf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:30.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.695+0000 7f5eaf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:30.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.699+0000 7f5eaffff640 1 -- 172.21.15.37:0/1904486994 >> v1:172.21.15.37:6789/0 conn(0x7f5eb0072ab0 legacy=0x7f5eb013e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:30.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.699+0000 7f5eaf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:30.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.699+0000 7f5eaf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:30.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.699+0000 7f5eaf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:30.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.699+0000 7f5eaf7fe640 10 monclient(hunting): handle_auth_done global_id 94206 payload 274 2024-04-26T00:28:30.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.699+0000 7f5eaf7fe640 1 -- 172.21.15.37:0/1904486994 >> v1:172.21.15.45:6789/0 conn(0x7f5eb0137780 legacy=0x7f5eb00723a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:28:30.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.699+0000 7f5eaf7fe640 1 -- 172.21.15.37:0/1904486994 >> v1:172.21.15.37:6789/0 conn(0x7f5eb0072ab0 legacy=0x7f5eb013e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:30.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.699+0000 7f5eaf7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:28:30.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.699+0000 7f5eaf7fe640 1 monclient: found mon.noname-a 2024-04-26T00:28:30.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.699+0000 7f5eaf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:30.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.699+0000 7f5eaf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:28:30.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.699+0000 7f5eaf7fe640 1 -- 172.21.15.37:0/1904486994 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5eb009bc50 con 0x7f5eb0071fc0 2024-04-26T00:28:30.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.703+0000 7f5eaf7fe640 1 -- 172.21.15.37:0/1904486994 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5eb0071fc0 msgr2=0x7f5eb0137240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:30.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.703+0000 7f5eaf7fe640 1 --2- 172.21.15.37:0/1904486994 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5eb0071fc0 0x7f5eb0137240 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5e98009840 tx=0x7f5e9801c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d5f1b06c852d537 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:30.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.703+0000 7f5eae7fc640 1 -- 172.21.15.37:0/1904486994 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e980b0070 con 0x7f5eb0071fc0 2024-04-26T00:28:30.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.703+0000 7f5eae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:30.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.703+0000 7f5eae7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:28:30.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.703+0000 7f5eae7fc640 10 monclient: dump: 2024-04-26T00:28:30.690 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:30.690 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:30.690 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:30.691 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:30.691 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:30.691 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:30.691 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:30.691 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:30.691 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:30.691 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:30.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.703+0000 7f5eae7fc640 10 monclient: _finish_auth 0 2024-04-26T00:28:30.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.703+0000 7f5eae7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:28:30.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.703+0000 7f5eae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:30.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.703+0000 7f5eae7fc640 1 -- 172.21.15.37:0/1904486994 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e980a97f0 con 0x7f5eb0071fc0 2024-04-26T00:28:30.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.703+0000 7f5eae7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:28:30.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.703+0000 7f5eae7fc640 1 -- 172.21.15.37:0/1904486994 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e980a9bc0 con 0x7f5eb0071fc0 2024-04-26T00:28:30.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.703+0000 7f5eb61ea640 5 monclient: authenticate success, global_id 94206 2024-04-26T00:28:30.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.703+0000 7f5eb61ea640 10 monclient: get_monmap_and_config success 2024-04-26T00:28:30.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.703+0000 7f5eb61ea640 4 set_mon_vals no callback set 2024-04-26T00:28:30.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.703+0000 7f5eae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:30.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.703+0000 7f5eae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:28:30.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.703+0000 7f5eae7fc640 10 monclient: dump: 2024-04-26T00:28:30.692 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:30.692 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:30.692 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:30.692 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:30.692 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:30.692 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:30.692 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:30.692 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:30.692 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:30.693 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:30.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.703+0000 7f5eb61ea640 10 monclient: shutdown 2024-04-26T00:28:30.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.703+0000 7f5eb61ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:28:30.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.703+0000 7f5eb61ea640 1 -- 172.21.15.37:0/1904486994 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5eb0071fc0 msgr2=0x7f5eb0137240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:30.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.703+0000 7f5eb61ea640 1 --2- 172.21.15.37:0/1904486994 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5eb0071fc0 0x7f5eb0137240 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5e98009840 tx=0x7f5e9801c3a0 comp rx=0 tx=0).stop 2024-04-26T00:28:30.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.703+0000 7f5eb61ea640 1 -- 172.21.15.37:0/1904486994 shutdown_connections 2024-04-26T00:28:30.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.703+0000 7f5eb61ea640 1 --2- 172.21.15.37:0/1904486994 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5eb0071fc0 0x7f5eb0137240 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:30.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.703+0000 7f5eb61ea640 1 -- 172.21.15.37:0/1904486994 >> 172.21.15.37:0/1904486994 conn(0x7f5eb00a1ad0 msgr2=0x7f5eb00a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:28:30.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.703+0000 7f5eb61ea640 1 -- 172.21.15.37:0/1904486994 shutdown_connections 2024-04-26T00:28:30.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.703+0000 7f5eb61ea640 1 -- 172.21.15.37:0/1904486994 wait complete. 2024-04-26T00:28:30.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.703+0000 7f5eb61ea640 10 monclient: build_initial_monmap 2024-04-26T00:28:30.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.703+0000 7f5eb61ea640 10 monclient: monmap: 2024-04-26T00:28:30.698 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:28:30.698 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:28:30.698 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:28:30.698 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:28:30.698 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:28:30.698 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:28:30.698 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:28:30.698 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:28:30.698 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:28:30.699 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:30.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.703+0000 7f5eb61ea640 1 Processor -- start 2024-04-26T00:28:30.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.703+0000 7f5eb61ea640 1 -- start start 2024-04-26T00:28:30.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.703+0000 7f5eb61ea640 10 monclient: init 2024-04-26T00:28:30.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eb61ea640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:30.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eb61ea640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:28:30.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eb61ea640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5eb0071fc0 0x7f5eb01ced40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:30.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eb61ea640 10 monclient(hunting): picked mon.noname-b con 0x7f5eb0071fc0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:28:30.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eb61ea640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5eb0072ab0 0x7f5eb01cf280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:30.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eb61ea640 10 monclient(hunting): picked mon.noname-a con 0x7f5eb0072ab0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:28:30.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eb61ea640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5eb0137780 0x7f5eb01cf7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:30.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eb61ea640 10 monclient(hunting): picked mon.noname-c con 0x7f5eb0137780 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:28:30.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eb61ea640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:30.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eb61ea640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f5eb0142a30 con 0x7f5eb0072ab0 2024-04-26T00:28:30.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eb61ea640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:30.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eb61ea640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5eb01428b0 con 0x7f5eb0071fc0 2024-04-26T00:28:30.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eb61ea640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:30.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eb61ea640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5eb0142bb0 con 0x7f5eb0137780 2024-04-26T00:28:30.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eb61ea640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:30.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eb61ea640 10 monclient(hunting): authenticate will time out at 1492.012503s 2024-04-26T00:28:30.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eaeffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5eb0072ab0 0x7f5eb01cf280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:30.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eaffff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5eb0137780 0x7f5eb01cf7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:30.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eaeffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5eb0072ab0 0x7f5eb01cf280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:49728/0 (socket says 172.21.15.37:49728) 2024-04-26T00:28:30.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eaffff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5eb0137780 0x7f5eb01cf7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:48566/0 (socket says 172.21.15.37:48566) 2024-04-26T00:28:30.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eaeffd640 1 -- 172.21.15.37:0/3019279921 learned_addr learned my addr 172.21.15.37:0/3019279921 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:28:30.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eaeffd640 10 monclient(hunting): get_auth_request con 0x7f5eb0072ab0 auth_method 0 2024-04-26T00:28:30.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eaeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:30.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eaeffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:30.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eaeffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:30.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eaffff640 10 monclient(hunting): get_auth_request con 0x7f5eb0137780 auth_method 0 2024-04-26T00:28:30.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eaffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:30.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eaffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:30.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eaffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:30.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eaffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:30.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eaffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:30.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eaffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:30.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eaeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:30.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eaeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:30.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eaf7fe640 1 --2- 172.21.15.37:0/3019279921 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5eb0071fc0 0x7f5eb01ced40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:30.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eaeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:30.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eaf7fe640 10 monclient(hunting): get_auth_request con 0x7f5eb0071fc0 auth_method 0 2024-04-26T00:28:30.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eaf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:30.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eaf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:30.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eaf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:30.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eaf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:30.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eaf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:30.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eaf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:30.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eaffff640 10 monclient(hunting): handle_auth_done global_id 84293 payload 824 2024-04-26T00:28:30.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eaffff640 1 -- 172.21.15.37:0/3019279921 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5eb0071fc0 msgr2=0x7f5eb01ced40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:30.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eaffff640 1 --2- 172.21.15.37:0/3019279921 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5eb0071fc0 0x7f5eb01ced40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:30.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eaffff640 1 -- 172.21.15.37:0/3019279921 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5eb0072ab0 msgr2=0x7f5eb01cf280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:30.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eaffff640 1 --2- 172.21.15.37:0/3019279921 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5eb0072ab0 0x7f5eb01cf280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:30.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eaffff640 10 monclient: _finish_hunting 0 2024-04-26T00:28:30.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eaffff640 1 monclient: found mon.noname-c 2024-04-26T00:28:30.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eaffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:30.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eaffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:28:30.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eaffff640 1 -- 172.21.15.37:0/3019279921 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5eb013b660 con 0x7f5eb0137780 2024-04-26T00:28:30.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eaeffd640 1 --2- 172.21.15.37:0/3019279921 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5eb0072ab0 0x7f5eb01cf280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:28:30.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eaffff640 1 -- 172.21.15.37:0/3019279921 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5eb0137780 msgr2=0x7f5eb01cf7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:30.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eaffff640 1 --2- 172.21.15.37:0/3019279921 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5eb0137780 0x7f5eb01cf7c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5eb0072fa0 tx=0x7f5ea800a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:30.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5ea7fff640 1 -- 172.21.15.37:0/3019279921 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ea8017070 con 0x7f5eb0137780 2024-04-26T00:28:30.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5ea7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:30.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5ea7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:28:30.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5ea7fff640 10 monclient: dump: 2024-04-26T00:28:30.703 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:30.703 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:30.703 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:30.703 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:30.703 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:30.703 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:30.703 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:30.703 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:30.704 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:30.704 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:30.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5ea7fff640 10 monclient: _finish_auth 0 2024-04-26T00:28:30.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5ea7fff640 10 monclient: _check_auth_tickets 2024-04-26T00:28:30.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5ea7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:30.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5ea7fff640 1 -- 172.21.15.37:0/3019279921 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ea800ad40 con 0x7f5eb0137780 2024-04-26T00:28:30.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5ea7fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:28:30.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5ea7fff640 1 -- 172.21.15.37:0/3019279921 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ea8012640 con 0x7f5eb0137780 2024-04-26T00:28:30.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eb61ea640 5 monclient: authenticate success, global_id 84293 2024-04-26T00:28:30.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eb61ea640 10 monclient: _renew_subs 2024-04-26T00:28:30.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eadffb640 4 set_mon_vals no callback set 2024-04-26T00:28:30.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eb61ea640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:28:30.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eb61ea640 1 -- 172.21.15.37:0/3019279921 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5eb013d0b0 con 0x7f5eb0137780 2024-04-26T00:28:30.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5ea7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:30.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5ea7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:28:30.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5ea7fff640 10 monclient: dump: 2024-04-26T00:28:30.705 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:30.705 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:30.705 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:30.705 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:30.705 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:30.705 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:30.705 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:30.705 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:30.705 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:30.705 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:30.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eb61ea640 10 monclient: _renew_subs 2024-04-26T00:28:30.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eb61ea640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:28:30.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.707+0000 7f5eb61ea640 1 -- 172.21.15.37:0/3019279921 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5eb013c3e0 con 0x7f5eb0137780 2024-04-26T00:28:30.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.711+0000 7f5eb61ea640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:30.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.711+0000 7f5eb61ea640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:30.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.711+0000 7f5eb61ea640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:28:30.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.711+0000 7f5eb61ea640 1 -- 172.21.15.37:0/3019279921 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5eb0066f40 con 0x7f5eb0137780 2024-04-26T00:28:30.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.711+0000 7f5ea7fff640 1 -- 172.21.15.37:0/3019279921 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f5ea801a070 con 0x7f5eb0137780 2024-04-26T00:28:30.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.711+0000 7f5ea7fff640 1 --2- 172.21.15.37:0/3019279921 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5e90035ed0 0x7f5e90038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:30.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.711+0000 7f5ea7fff640 1 -- 172.21.15.37:0/3019279921 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(336..336 src has 177..336) ==== 3975+0+0 (secure 0 0 0) 0x7f5ea8048730 con 0x7f5eb0137780 2024-04-26T00:28:30.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.711+0000 7f5eaf7fe640 1 --2- 172.21.15.37:0/3019279921 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5e90035ed0 0x7f5e90038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:30.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.711+0000 7f5eaf7fe640 10 monclient: get_auth_request con 0x7f5e90035ed0 auth_method 0 2024-04-26T00:28:30.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.715+0000 7f5eaf7fe640 1 --2- 172.21.15.37:0/3019279921 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5e90035ed0 0x7f5e90038390 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5eb00a13a0 tx=0x7f5e980a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:30.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.715+0000 7f5ea7fff640 1 -- 172.21.15.37:0/3019279921 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5ea8007920 con 0x7f5eb0137780 2024-04-26T00:28:30.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.715+0000 7f5ea7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:30.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.715+0000 7f5ea7fff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:28:30.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.939+0000 7f5eb61ea640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:28:30.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.939+0000 7f5eb61ea640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:28:30.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.939+0000 7f5eb61ea640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:28:30.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.939+0000 7f5eb61ea640 1 -- 172.21.15.37:0/3019279921 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5eb006b9a0 con 0x7f5eb0137780 2024-04-26T00:28:30.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.939+0000 7f5ea7fff640 1 -- 172.21.15.37:0/3019279921 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f5ea8007400 con 0x7f5eb0137780 2024-04-26T00:28:30.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.939+0000 7f5ea7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:28:30.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.939+0000 7f5ea7fff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:28:30.928 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":64,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:28:30.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.943+0000 7f5eb61ea640 1 -- 172.21.15.37:0/3019279921 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5e90035ed0 msgr2=0x7f5e90038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:30.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.943+0000 7f5eb61ea640 1 --2- 172.21.15.37:0/3019279921 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5e90035ed0 0x7f5e90038390 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5eb00a13a0 tx=0x7f5e980a6000 comp rx=0 tx=0).stop 2024-04-26T00:28:30.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.943+0000 7f5eb61ea640 10 monclient: shutdown 2024-04-26T00:28:30.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.943+0000 7f5eb61ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:28:30.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.943+0000 7f5eb61ea640 1 -- 172.21.15.37:0/3019279921 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5eb0137780 msgr2=0x7f5eb01cf7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:30.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.943+0000 7f5eb61ea640 1 --2- 172.21.15.37:0/3019279921 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5eb0137780 0x7f5eb01cf7c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5eb0072fa0 tx=0x7f5ea800a430 comp rx=0 tx=0).stop 2024-04-26T00:28:30.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.943+0000 7f5eb61ea640 1 -- 172.21.15.37:0/3019279921 shutdown_connections 2024-04-26T00:28:30.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.943+0000 7f5eb61ea640 1 --2- 172.21.15.37:0/3019279921 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5e90035ed0 0x7f5e90038390 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:30.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.943+0000 7f5eb61ea640 1 --2- 172.21.15.37:0/3019279921 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5eb0137780 0x7f5eb01cf7c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:30.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.943+0000 7f5eb61ea640 1 --2- 172.21.15.37:0/3019279921 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5eb0072ab0 0x7f5eb01cf280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:30.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.943+0000 7f5eb61ea640 1 --2- 172.21.15.37:0/3019279921 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5eb0071fc0 0x7f5eb01ced40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:30.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.943+0000 7f5eb61ea640 1 -- 172.21.15.37:0/3019279921 >> 172.21.15.37:0/3019279921 conn(0x7f5eb00a1ad0 msgr2=0x7f5eb00a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:28:30.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.943+0000 7f5eb61ea640 1 -- 172.21.15.37:0/3019279921 shutdown_connections 2024-04-26T00:28:30.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:30.943+0000 7f5eb61ea640 1 -- 172.21.15.37:0/3019279921 wait complete. 2024-04-26T00:28:30.947 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:28:30.947 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:28:30.947 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:28:31.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad51bba640 10 monclient: get_monmap_and_config 2024-04-26T00:28:31.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad51bba640 10 monclient: build_initial_monmap 2024-04-26T00:28:31.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad51bba640 10 monclient: monmap: 2024-04-26T00:28:31.073 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:28:31.073 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:31.074 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:28:31.088764+0000 2024-04-26T00:28:31.074 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:28:31.088764+0000 2024-04-26T00:28:31.074 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:28:31.074 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:31.074 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:28:31.074 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:28:31.074 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:28:31.074 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:28:31.075 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:28:31.075 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:31.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad51bba640 1 Processor -- start 2024-04-26T00:28:31.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad51bba640 1 -- start start 2024-04-26T00:28:31.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad51bba640 10 monclient: init 2024-04-26T00:28:31.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad51bba640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:31.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad51bba640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-04-26T00:28:31.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad51bba640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fad4c130eb0 0x7fad4c09ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:31.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad51bba640 10 monclient(hunting): picked mon.noname-b con 0x7fad4c130eb0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:28:31.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad51bba640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fad4c131290 0x7fad4c133b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:31.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad51bba640 10 monclient(hunting): picked mon.noname-a con 0x7fad4c131290 addr v2:172.21.15.37:3300/0 2024-04-26T00:28:31.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad51bba640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fad4c1340d0 addr v1:172.21.15.37:6789/0 2024-04-26T00:28:31.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad51bba640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad4c09c190 con 0x7fad4c1340d0 2024-04-26T00:28:31.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad51bba640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:31.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad51bba640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fad4c09df60 con 0x7fad4c131290 2024-04-26T00:28:31.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad51bba640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:31.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad51bba640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fad4c0a11c0 con 0x7fad4c130eb0 2024-04-26T00:28:31.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad51bba640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:31.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad51bba640 10 monclient(hunting): authenticate will time out at 1492.394677s 2024-04-26T00:28:31.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad4affd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fad4c131290 0x7fad4c133b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:31.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad4b7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fad4c130eb0 0x7fad4c09ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:31.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad4bfff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fad4c1340d0 0x7fad4c13aec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:34670/0 (socket says 172.21.15.37:34670) 2024-04-26T00:28:31.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad4affd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fad4c131290 0x7fad4c133b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:49740/0 (socket says 172.21.15.37:49740) 2024-04-26T00:28:31.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad4bfff640 1 -- 172.21.15.37:0/925530756 learned_addr learned my addr 172.21.15.37:0/925530756 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:28:31.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad4b7fe640 10 monclient(hunting): get_auth_request con 0x7fad4c130eb0 auth_method 0 2024-04-26T00:28:31.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:31.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad4b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:31.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:31.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad4affd640 10 monclient(hunting): get_auth_request con 0x7fad4c131290 auth_method 0 2024-04-26T00:28:31.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad4affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:31.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad4affd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:31.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad4affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:31.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:31.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:31.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:31.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad4affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:31.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad4affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:31.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad4affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:31.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad4affd640 10 monclient(hunting): handle_auth_done global_id 94215 payload 274 2024-04-26T00:28:31.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad4affd640 1 -- 172.21.15.37:0/925530756 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fad4c130eb0 msgr2=0x7fad4c09ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:28:31.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad4affd640 1 --2- 172.21.15.37:0/925530756 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fad4c130eb0 0x7fad4c09ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:31.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad4affd640 1 -- 172.21.15.37:0/925530756 >> v1:172.21.15.37:6789/0 conn(0x7fad4c1340d0 legacy=0x7fad4c13aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:28:31.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad4affd640 10 monclient: _finish_hunting 0 2024-04-26T00:28:31.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad4affd640 1 monclient: found mon.noname-a 2024-04-26T00:28:31.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad4affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:31.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad4affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:28:31.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad4affd640 1 -- 172.21.15.37:0/925530756 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad4c09dc80 con 0x7fad4c131290 2024-04-26T00:28:31.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.087+0000 7fad4b7fe640 1 --2- 172.21.15.37:0/925530756 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fad4c130eb0 0x7fad4c09ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:28:31.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.091+0000 7fad4affd640 1 -- 172.21.15.37:0/925530756 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fad4c131290 msgr2=0x7fad4c133b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:31.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.091+0000 7fad4affd640 1 --2- 172.21.15.37:0/925530756 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fad4c131290 0x7fad4c133b90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fad340097c0 tx=0x7fad3401c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80788eb2d3e1ed1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:31.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.091+0000 7fad4a7fc640 1 -- 172.21.15.37:0/925530756 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad340b0070 con 0x7fad4c131290 2024-04-26T00:28:31.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:31.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:28:31.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4a7fc640 10 monclient: dump: 2024-04-26T00:28:31.092 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:31.092 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:31.092 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.092 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.092 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:31.092 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:31.092 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:31.092 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:31.092 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:31.092 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:31.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4a7fc640 10 monclient: _finish_auth 0 2024-04-26T00:28:31.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4a7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:28:31.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:31.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4a7fc640 1 -- 172.21.15.37:0/925530756 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad340ab600 con 0x7fad4c131290 2024-04-26T00:28:31.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad51bba640 5 monclient: authenticate success, global_id 94215 2024-04-26T00:28:31.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad51bba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:28:31.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4a7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:28:31.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad51bba640 10 monclient: get_monmap_and_config success 2024-04-26T00:28:31.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad51bba640 4 set_mon_vals no callback set 2024-04-26T00:28:31.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4a7fc640 1 -- 172.21.15.37:0/925530756 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad340ab9d0 con 0x7fad4c131290 2024-04-26T00:28:31.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:31.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:28:31.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4a7fc640 10 monclient: dump: 2024-04-26T00:28:31.093 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:31.093 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:31.093 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.094 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.094 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:31.094 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:31.094 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:31.094 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:31.094 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:31.094 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:31.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad51bba640 10 monclient: shutdown 2024-04-26T00:28:31.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad51bba640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:28:31.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad51bba640 1 -- 172.21.15.37:0/925530756 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fad4c131290 msgr2=0x7fad4c133b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:31.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad51bba640 1 --2- 172.21.15.37:0/925530756 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fad4c131290 0x7fad4c133b90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fad340097c0 tx=0x7fad3401c320 comp rx=0 tx=0).stop 2024-04-26T00:28:31.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad51bba640 1 -- 172.21.15.37:0/925530756 shutdown_connections 2024-04-26T00:28:31.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad51bba640 1 --2- 172.21.15.37:0/925530756 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fad4c131290 0x7fad4c133b90 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:31.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad51bba640 1 --2- 172.21.15.37:0/925530756 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fad4c130eb0 0x7fad4c09ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:31.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad51bba640 1 -- 172.21.15.37:0/925530756 >> 172.21.15.37:0/925530756 conn(0x7fad4c0a1d50 msgr2=0x7fad4c0a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:28:31.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad51bba640 1 -- 172.21.15.37:0/925530756 shutdown_connections 2024-04-26T00:28:31.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad51bba640 1 -- 172.21.15.37:0/925530756 wait complete. 2024-04-26T00:28:31.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad51bba640 10 monclient: build_initial_monmap 2024-04-26T00:28:31.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad51bba640 10 monclient: monmap: 2024-04-26T00:28:31.095 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:28:31.095 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:28:31.095 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:28:31.095 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:28:31.095 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:28:31.095 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:28:31.095 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:28:31.095 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:28:31.096 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:28:31.096 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:31.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad51bba640 1 Processor -- start 2024-04-26T00:28:31.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad51bba640 1 -- start start 2024-04-26T00:28:31.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad51bba640 10 monclient: init 2024-04-26T00:28:31.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad51bba640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:31.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad51bba640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:28:31.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad51bba640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fad4c130eb0 0x7fad4c135400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:31.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad51bba640 10 monclient(hunting): picked mon.noname-c con 0x7fad4c130eb0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:28:31.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad51bba640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad4c131290 0x7fad4c135940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:31.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad51bba640 10 monclient(hunting): picked mon.noname-b con 0x7fad4c131290 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:28:31.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad51bba640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fad4c1340d0 0x7fad4c135e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:31.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad51bba640 10 monclient(hunting): picked mon.noname-a con 0x7fad4c1340d0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:28:31.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad51bba640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:31.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad51bba640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fad4c1317f0 con 0x7fad4c1340d0 2024-04-26T00:28:31.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad51bba640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:31.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad51bba640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fad4c131670 con 0x7fad4c131290 2024-04-26T00:28:31.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad51bba640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:31.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad51bba640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fad4c13ee00 con 0x7fad4c130eb0 2024-04-26T00:28:31.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad51bba640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:31.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad51bba640 10 monclient(hunting): authenticate will time out at 1492.402238s 2024-04-26T00:28:31.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4bfff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fad4c1340d0 0x7fad4c135e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:31.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4b7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fad4c130eb0 0x7fad4c135400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:31.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4bfff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fad4c1340d0 0x7fad4c135e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:49748/0 (socket says 172.21.15.37:49748) 2024-04-26T00:28:31.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4bfff640 1 -- 172.21.15.37:0/3339682289 learned_addr learned my addr 172.21.15.37:0/3339682289 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:28:31.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4bfff640 10 monclient(hunting): get_auth_request con 0x7fad4c1340d0 auth_method 0 2024-04-26T00:28:31.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:31.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:31.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4b7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fad4c130eb0 0x7fad4c135400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:48584/0 (socket says 172.21.15.37:48584) 2024-04-26T00:28:31.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:31.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4b7fe640 10 monclient(hunting): get_auth_request con 0x7fad4c130eb0 auth_method 0 2024-04-26T00:28:31.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:31.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:31.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:31.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:31.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:31.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:31.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:31.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:31.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:31.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4affd640 1 --2- 172.21.15.37:0/3339682289 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad4c131290 0x7fad4c135940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:31.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4b7fe640 10 monclient(hunting): handle_auth_done global_id 84299 payload 824 2024-04-26T00:28:31.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4b7fe640 1 -- 172.21.15.37:0/3339682289 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad4c131290 msgr2=0x7fad4c135940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:31.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4b7fe640 1 --2- 172.21.15.37:0/3339682289 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad4c131290 0x7fad4c135940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:31.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4b7fe640 1 -- 172.21.15.37:0/3339682289 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fad4c1340d0 msgr2=0x7fad4c135e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:31.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4b7fe640 1 --2- 172.21.15.37:0/3339682289 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fad4c1340d0 0x7fad4c135e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:31.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4b7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:28:31.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4b7fe640 1 monclient: found mon.noname-c 2024-04-26T00:28:31.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:31.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:28:31.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4b7fe640 1 -- 172.21.15.37:0/3339682289 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad4c138b70 con 0x7fad4c130eb0 2024-04-26T00:28:31.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.095+0000 7fad4bfff640 1 --2- 172.21.15.37:0/3339682289 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fad4c1340d0 0x7fad4c135e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:28:31.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad4b7fe640 1 -- 172.21.15.37:0/3339682289 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fad4c130eb0 msgr2=0x7fad4c135400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:31.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad4b7fe640 1 --2- 172.21.15.37:0/3339682289 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fad4c130eb0 0x7fad4c135400 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fad3c009e60 tx=0x7fad3c00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:31.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad43fff640 1 -- 172.21.15.37:0/3339682289 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad3c018020 con 0x7fad4c130eb0 2024-04-26T00:28:31.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad43fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:31.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad43fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:28:31.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad43fff640 10 monclient: dump: 2024-04-26T00:28:31.100 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:31.100 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:31.100 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.100 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.100 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:31.100 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:31.100 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:31.101 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:31.101 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:31.101 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:31.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad43fff640 10 monclient: _finish_auth 0 2024-04-26T00:28:31.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad43fff640 10 monclient: _check_auth_tickets 2024-04-26T00:28:31.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:31.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad43fff640 1 -- 172.21.15.37:0/3339682289 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad3c014890 con 0x7fad4c130eb0 2024-04-26T00:28:31.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad51bba640 5 monclient: authenticate success, global_id 84299 2024-04-26T00:28:31.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad51bba640 10 monclient: _renew_subs 2024-04-26T00:28:31.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad51bba640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:28:31.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad51bba640 1 -- 172.21.15.37:0/3339682289 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad4c138e50 con 0x7fad4c130eb0 2024-04-26T00:28:31.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad43fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:28:31.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad43fff640 1 -- 172.21.15.37:0/3339682289 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad3c014c80 con 0x7fad4c130eb0 2024-04-26T00:28:31.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad43fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:31.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad43fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:28:31.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad43fff640 10 monclient: dump: 2024-04-26T00:28:31.103 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:31.103 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:31.104 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.104 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.104 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:31.104 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:31.104 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:31.104 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:31.105 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:31.105 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:31.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad51bba640 10 monclient: _renew_subs 2024-04-26T00:28:31.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad49ffb640 4 set_mon_vals no callback set 2024-04-26T00:28:31.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad51bba640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:28:31.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad51bba640 1 -- 172.21.15.37:0/3339682289 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fad4c137720 con 0x7fad4c130eb0 2024-04-26T00:28:31.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad51bba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:31.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad51bba640 1 --2- 172.21.15.37:0/3339682289 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fad4c0a3050 0x7fad4c0a3430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:31.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad51bba640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:31.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad51bba640 1 -- 172.21.15.37:0/3339682289 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fad4c138890 con 0x7fad4c0a3050 2024-04-26T00:28:31.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad4affd640 1 --2- 172.21.15.37:0/3339682289 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fad4c0a3050 0x7fad4c0a3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:31.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad4affd640 10 monclient: get_auth_request con 0x7fad4c0a3050 auth_method 0 2024-04-26T00:28:31.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad4affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:31.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad4affd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:31.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad4affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:31.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad43fff640 1 -- 172.21.15.37:0/3339682289 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fad3c01b020 con 0x7fad4c130eb0 2024-04-26T00:28:31.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad4affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:31.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad4affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:31.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad4affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:31.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad4affd640 10 monclient(hunting): handle_auth_done global_id 94221 payload 824 2024-04-26T00:28:31.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad43fff640 1 --2- 172.21.15.37:0/3339682289 >> v2:172.21.15.45:6812/2379918623 conn(0x7fad2c035e80 0x7fad2c038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:31.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad4affd640 1 -- 172.21.15.37:0/3339682289 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fad4c04f510 con 0x7fad4c0a3050 2024-04-26T00:28:31.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad43fff640 1 -- 172.21.15.37:0/3339682289 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(336..336 src has 177..336) ==== 3975+0+0 (secure 0 0 0) 0x7fad3c049860 con 0x7fad4c130eb0 2024-04-26T00:28:31.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad4bfff640 1 --2- 172.21.15.37:0/3339682289 >> v2:172.21.15.45:6812/2379918623 conn(0x7fad2c035e80 0x7fad2c038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:31.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.099+0000 7fad4bfff640 10 monclient: get_auth_request con 0x7fad2c035e80 auth_method 0 2024-04-26T00:28:31.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.103+0000 7fad4affd640 1 -- 172.21.15.37:0/3339682289 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fad4c0a3050 msgr2=0x7fad4c0a3430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:31.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.103+0000 7fad4affd640 1 --2- 172.21.15.37:0/3339682289 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fad4c0a3050 0x7fad4c0a3430 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fad4c06d2b0 tx=0x7fad340190c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:31.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.103+0000 7fad43fff640 1 -- 172.21.15.37:0/3339682289 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad340b9070 con 0x7fad4c0a3050 2024-04-26T00:28:31.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.103+0000 7fad43fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:31.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.103+0000 7fad43fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:28:31.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.103+0000 7fad43fff640 10 monclient: dump: 2024-04-26T00:28:31.107 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:31.107 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:31.107 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.107 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.108 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:31.108 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:31.108 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:31.108 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:31.108 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:31.108 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:31.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.103+0000 7fad43fff640 1 -- 172.21.15.37:0/3339682289 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fad340abe00 con 0x7fad4c0a3050 2024-04-26T00:28:31.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.103+0000 7fad43fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:31.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.103+0000 7fad43fff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:28:31.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.103+0000 7fad43fff640 1 -- 172.21.15.37:0/3339682289 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fad4c0a3050 msgr2=0x7fad4c0a3430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:31.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.103+0000 7fad43fff640 1 --2- 172.21.15.37:0/3339682289 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fad4c0a3050 0x7fad4c0a3430 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fad4c06d2b0 tx=0x7fad340190c0 comp rx=0 tx=0).stop 2024-04-26T00:28:31.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.103+0000 7fad43fff640 1 -- 172.21.15.37:0/3339682289 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fad4c0a3050 msgr2=0x7fad4c0a3430 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:28:31.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.103+0000 7fad43fff640 1 --2- 172.21.15.37:0/3339682289 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fad4c0a3050 0x7fad4c0a3430 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fad4c06d2b0 tx=0x7fad340190c0 comp rx=0 tx=0).stop 2024-04-26T00:28:31.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.103+0000 7fad4bfff640 1 --2- 172.21.15.37:0/3339682289 >> v2:172.21.15.45:6812/2379918623 conn(0x7fad2c035e80 0x7fad2c038340 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fad44007d60 tx=0x7fad44007cb0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:31.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.107+0000 7fad51bba640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:28:31.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.107+0000 7fad51bba640 1 --2- 172.21.15.37:0/3339682289 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fad4c0a3970 0x7fad4c1d9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:31.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.107+0000 7fad51bba640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:31.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.107+0000 7fad51bba640 1 -- 172.21.15.37:0/3339682289 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fad4c133d30 con 0x7fad4c0a3970 2024-04-26T00:28:31.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.107+0000 7fad4affd640 1 --2- 172.21.15.37:0/3339682289 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fad4c0a3970 0x7fad4c1d9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:31.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.107+0000 7fad4affd640 10 monclient: get_auth_request con 0x7fad4c0a3970 auth_method 0 2024-04-26T00:28:31.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.107+0000 7fad4affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:31.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.107+0000 7fad4affd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:31.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.107+0000 7fad4affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:31.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.107+0000 7fad4affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:31.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.107+0000 7fad4affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:31.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.107+0000 7fad4affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:31.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.107+0000 7fad4affd640 10 monclient(hunting): handle_auth_done global_id 94224 payload 824 2024-04-26T00:28:31.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.107+0000 7fad4affd640 1 -- 172.21.15.37:0/3339682289 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fad4c130940 con 0x7fad4c0a3970 2024-04-26T00:28:31.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.111+0000 7fad4affd640 1 -- 172.21.15.37:0/3339682289 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fad4c0a3970 msgr2=0x7fad4c1d9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:31.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.111+0000 7fad4affd640 1 --2- 172.21.15.37:0/3339682289 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fad4c0a3970 0x7fad4c1d9ff0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fad340097c0 tx=0x7fad340a8000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:31.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.111+0000 7fad43fff640 1 -- 172.21.15.37:0/3339682289 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad340b9050 con 0x7fad4c0a3970 2024-04-26T00:28:31.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.111+0000 7fad43fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:31.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.111+0000 7fad43fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:28:31.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.111+0000 7fad43fff640 10 monclient: dump: 2024-04-26T00:28:31.112 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:31.112 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:31.112 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.112 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.112 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:31.112 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:31.112 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:31.113 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:31.113 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:31.113 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:31.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.111+0000 7fad43fff640 1 -- 172.21.15.37:0/3339682289 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4718 (secure 0 0 0) 0x7fad340b5480 con 0x7fad4c0a3970 2024-04-26T00:28:31.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.111+0000 7fad43fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:28:31.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.111+0000 7fad43fff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:28:31.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.111+0000 7fad43fff640 1 -- 172.21.15.37:0/3339682289 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fad4c0a3970 msgr2=0x7fad4c1d9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:31.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.111+0000 7fad43fff640 1 --2- 172.21.15.37:0/3339682289 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fad4c0a3970 0x7fad4c1d9ff0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fad340097c0 tx=0x7fad340a8000 comp rx=0 tx=0).stop 2024-04-26T00:28:31.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.111+0000 7fad43fff640 1 -- 172.21.15.37:0/3339682289 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fad4c0a3970 msgr2=0x7fad4c1d9ff0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:28:31.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.111+0000 7fad43fff640 1 --2- 172.21.15.37:0/3339682289 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fad4c0a3970 0x7fad4c1d9ff0 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fad340097c0 tx=0x7fad340a8000 comp rx=0 tx=0).stop 2024-04-26T00:28:31.114 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:28:31.114 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:28:31.114 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:28:31.114 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:28:31.114 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 64, 2024-04-26T00:28:31.114 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 71870, 2024-04-26T00:28:31.114 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:28:31.114 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:28:31.114 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:28:31.114 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:28:31.114 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:31.114 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 3, 2024-04-26T00:28:31.115 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:28:31.115 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:28:31.115 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:28:31.115 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:28:31.115 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:28:31.115 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:28:31.115 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:28:31.115 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:28:31.115 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:28:31.116 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:28:31.116 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:28:31.116 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:28:31.116 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:28:31.116 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:28:31.116 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:31.116 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:28:31.116 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:28:31.116 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:28:31.116 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:28:31.116 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:28:31.116 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:28:31.116 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:28:31.116 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:28:31.117 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:28:31.117 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:28:31.117 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:28:31.117 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:28:31.117 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:28:31.117 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:31.117 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:31.117 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:28:31.117 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:28:31.117 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:28:31.117 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:28:31.117 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:28:31.117 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:28:31.117 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:28:31.118 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:28:31.118 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:28:31.118 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:28:31.118 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:28:31.118 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:28:31.118 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:28:31.118 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:28:31.118 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:28:31.118 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:28:31.118 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:28:31.118 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:28:31.118 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:28:31.118 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:28:31.119 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:28:31.119 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:28:31.119 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:28:31.119 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:28:31.119 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:28:31.119 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:28:31.119 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:28:31.119 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:28:31.119 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:31.119 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:28:31.119 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:31.119 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:28:31.119 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:31.119 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:28:31.120 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:28:31.120 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:28:31.120 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:28:31.120 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:31.120 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:28:31.120 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:28:31.120 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:31.120 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:31.120 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:31.120 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:28:31.120 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:28:31.120 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:31.120 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:31.120 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:31.121 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:31.121 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:28:31.121 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:28:31.121 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:28:31.121 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:28:31.121 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:28:31.121 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:31.121 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:31.121 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:28:31.121 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:28:31.121 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:28:31.122 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:28:31.122 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:31.122 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:28:31.122 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:28:31.122 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:31.122 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:31.122 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:31.122 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:28:31.122 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:28:31.122 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:31.122 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:31.122 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:31.122 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:31.123 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:28:31.123 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:28:31.123 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:28:31.123 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:28:31.123 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:28:31.123 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:31.123 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:31.123 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:28:31.123 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:28:31.123 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:28:31.123 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:28:31.123 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:31.123 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:28:31.124 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:28:31.124 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:31.124 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:31.124 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:31.124 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:28:31.124 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:28:31.124 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:31.124 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:31.124 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:31.124 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:31.124 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:28:31.124 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:28:31.124 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:28:31.124 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:28:31.125 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:28:31.125 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:31.125 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:31.125 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:31.125 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:28:31.125 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:28:31.125 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:31.125 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:28:31.125 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:28:31.125 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:28:31.125 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:31.125 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:31.125 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:28:31.126 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:31.126 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:28:31.126 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:28:31.126 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:28:31.126 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:31.126 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:31.126 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:28:31.126 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:31.126 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:28:31.126 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:28:31.126 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:28:31.126 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:31.126 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:31.127 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:31.127 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:28:31.127 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:28:31.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.111+0000 7fad51bba640 1 -- 172.21.15.37:0/3339682289 >> v2:172.21.15.45:6812/2379918623 conn(0x7fad2c035e80 msgr2=0x7fad2c038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:31.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.111+0000 7fad51bba640 1 --2- 172.21.15.37:0/3339682289 >> v2:172.21.15.45:6812/2379918623 conn(0x7fad2c035e80 0x7fad2c038340 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fad44007d60 tx=0x7fad44007cb0 comp rx=0 tx=0).stop 2024-04-26T00:28:31.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.111+0000 7fad51bba640 10 monclient: shutdown 2024-04-26T00:28:31.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.111+0000 7fad51bba640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:28:31.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.111+0000 7fad51bba640 1 -- 172.21.15.37:0/3339682289 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fad4c130eb0 msgr2=0x7fad4c135400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:31.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.111+0000 7fad51bba640 1 --2- 172.21.15.37:0/3339682289 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fad4c130eb0 0x7fad4c135400 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fad3c009e60 tx=0x7fad3c00c490 comp rx=0 tx=0).stop 2024-04-26T00:28:31.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.111+0000 7fad4bfff640 1 -- 172.21.15.37:0/3339682289 reap_dead start 2024-04-26T00:28:31.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.111+0000 7fad4bfff640 1 -- 172.21.15.37:0/3339682289 reap_dead start 2024-04-26T00:28:31.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.111+0000 7fad51bba640 1 -- 172.21.15.37:0/3339682289 shutdown_connections 2024-04-26T00:28:31.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.111+0000 7fad51bba640 1 -- 172.21.15.37:0/3339682289 >> 172.21.15.37:0/3339682289 conn(0x7fad4c0a1d50 msgr2=0x7fad4c12fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:28:31.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.111+0000 7fad51bba640 1 -- 172.21.15.37:0/3339682289 shutdown_connections 2024-04-26T00:28:31.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.111+0000 7fad51bba640 1 -- 172.21.15.37:0/3339682289 wait complete. 2024-04-26T00:28:31.129 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:28:31.129 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:28:31.130 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:28:31.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.255+0000 7fd47823c640 10 monclient: get_monmap_and_config 2024-04-26T00:28:31.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.255+0000 7fd47823c640 10 monclient: build_initial_monmap 2024-04-26T00:28:31.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.255+0000 7fd47823c640 10 monclient: monmap: 2024-04-26T00:28:31.242 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:28:31.242 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:31.242 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:28:31.257264+0000 2024-04-26T00:28:31.242 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:28:31.257264+0000 2024-04-26T00:28:31.242 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:28:31.242 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:31.242 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:28:31.242 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:28:31.242 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:28:31.242 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:28:31.242 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:28:31.242 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:31.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.255+0000 7fd47823c640 1 Processor -- start 2024-04-26T00:28:31.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.255+0000 7fd47823c640 1 -- start start 2024-04-26T00:28:31.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.255+0000 7fd47823c640 10 monclient: init 2024-04-26T00:28:31.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.255+0000 7fd47823c640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:31.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.255+0000 7fd47823c640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-04-26T00:28:31.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.255+0000 7fd47823c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd4700a0300 0x7fd47009fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:31.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.255+0000 7fd47823c640 10 monclient(hunting): picked mon.noname-a con 0x7fd4700a0300 addr v2:172.21.15.37:3300/0 2024-04-26T00:28:31.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.255+0000 7fd47823c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd4700a06e0 addr v1:172.21.15.45:6789/0 2024-04-26T00:28:31.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.255+0000 7fd47823c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd4701352b0 addr v1:172.21.15.37:6789/0 2024-04-26T00:28:31.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.255+0000 7fd47823c640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd47009c190 con 0x7fd4701352b0 2024-04-26T00:28:31.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.255+0000 7fd47823c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4700a1110 con 0x7fd4700a06e0 2024-04-26T00:28:31.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.255+0000 7fd47823c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:31.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.255+0000 7fd47823c640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fd4700a13f0 con 0x7fd4700a0300 2024-04-26T00:28:31.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.255+0000 7fd47823c640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:31.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.255+0000 7fd47823c640 10 monclient(hunting): authenticate will time out at 1492.562934s 2024-04-26T00:28:31.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.255+0000 7fd475fd8640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd4700a0300 0x7fd47009fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:31.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.255+0000 7fd475fd8640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd4700a0300 0x7fd47009fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:49762/0 (socket says 172.21.15.37:49762) 2024-04-26T00:28:31.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.255+0000 7fd475fd8640 1 -- 172.21.15.37:0/617573693 learned_addr learned my addr 172.21.15.37:0/617573693 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:28:31.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.255+0000 7fd4767d9640 1 --1- 172.21.15.37:0/617573693 >> v1:172.21.15.37:6789/0 conn(0x7fd4701352b0 0x7fd47013cee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:34674/0 (socket says 172.21.15.37:34674) 2024-04-26T00:28:31.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.255+0000 7fd475fd8640 10 monclient(hunting): get_auth_request con 0x7fd4700a0300 auth_method 0 2024-04-26T00:28:31.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.255+0000 7fd475fd8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:31.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.255+0000 7fd475fd8640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:31.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.255+0000 7fd475fd8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:31.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.255+0000 7fd475fd8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:31.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.255+0000 7fd475fd8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:31.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.255+0000 7fd475fd8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:31.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.255+0000 7fd475fd8640 10 monclient(hunting): handle_auth_done global_id 94227 payload 274 2024-04-26T00:28:31.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.255+0000 7fd475fd8640 1 -- 172.21.15.37:0/617573693 >> v1:172.21.15.45:6789/0 conn(0x7fd4700a06e0 legacy=0x7fd470134ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:28:31.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.255+0000 7fd475fd8640 1 -- 172.21.15.37:0/617573693 >> v1:172.21.15.37:6789/0 conn(0x7fd4701352b0 legacy=0x7fd47013cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:28:31.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.255+0000 7fd475fd8640 10 monclient: _finish_hunting 0 2024-04-26T00:28:31.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.255+0000 7fd475fd8640 1 monclient: found mon.noname-a 2024-04-26T00:28:31.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.255+0000 7fd475fd8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:31.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.255+0000 7fd475fd8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:28:31.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.255+0000 7fd475fd8640 1 -- 172.21.15.37:0/617573693 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4700a16d0 con 0x7fd4700a0300 2024-04-26T00:28:31.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.259+0000 7fd475fd8640 1 -- 172.21.15.37:0/617573693 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd4700a0300 msgr2=0x7fd47009fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:31.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.259+0000 7fd475fd8640 1 --2- 172.21.15.37:0/617573693 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd4700a0300 0x7fd47009fc50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd468009e50 tx=0x7fd46801c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=387420fbbe733a0d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:31.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.259+0000 7fd474fd6640 1 -- 172.21.15.37:0/617573693 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4680a5670 con 0x7fd4700a0300 2024-04-26T00:28:31.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd474fd6640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:31.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd474fd6640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:28:31.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd474fd6640 10 monclient: dump: 2024-04-26T00:28:31.252 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:31.252 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:31.252 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.252 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.253 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:31.253 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:31.253 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:31.253 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:31.253 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:31.253 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:31.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd474fd6640 10 monclient: _finish_auth 0 2024-04-26T00:28:31.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd474fd6640 10 monclient: _check_auth_tickets 2024-04-26T00:28:31.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd474fd6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:31.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd474fd6640 1 -- 172.21.15.37:0/617573693 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4680a5810 con 0x7fd4700a0300 2024-04-26T00:28:31.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd474fd6640 10 monclient: handle_config config(0 keys) 2024-04-26T00:28:31.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd474fd6640 1 -- 172.21.15.37:0/617573693 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4680a5be0 con 0x7fd4700a0300 2024-04-26T00:28:31.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd47823c640 5 monclient: authenticate success, global_id 94227 2024-04-26T00:28:31.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd47823c640 10 monclient: get_monmap_and_config success 2024-04-26T00:28:31.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd47823c640 4 set_mon_vals no callback set 2024-04-26T00:28:31.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd474fd6640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:31.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd474fd6640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:28:31.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd474fd6640 10 monclient: dump: 2024-04-26T00:28:31.255 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:31.255 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:31.256 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.256 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.256 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:31.256 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:31.256 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:31.256 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:31.256 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:31.256 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:31.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd47823c640 10 monclient: shutdown 2024-04-26T00:28:31.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd47823c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:28:31.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd47823c640 1 -- 172.21.15.37:0/617573693 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd4700a0300 msgr2=0x7fd47009fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:31.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd47823c640 1 --2- 172.21.15.37:0/617573693 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd4700a0300 0x7fd47009fc50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd468009e50 tx=0x7fd46801c8a0 comp rx=0 tx=0).stop 2024-04-26T00:28:31.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd47823c640 1 -- 172.21.15.37:0/617573693 shutdown_connections 2024-04-26T00:28:31.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd47823c640 1 --2- 172.21.15.37:0/617573693 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd4700a0300 0x7fd47009fc50 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:31.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd47823c640 1 -- 172.21.15.37:0/617573693 >> 172.21.15.37:0/617573693 conn(0x7fd4700a1d50 msgr2=0x7fd4700a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:28:31.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd47823c640 1 -- 172.21.15.37:0/617573693 shutdown_connections 2024-04-26T00:28:31.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd47823c640 1 -- 172.21.15.37:0/617573693 wait complete. 2024-04-26T00:28:31.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd47823c640 10 monclient: build_initial_monmap 2024-04-26T00:28:31.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd47823c640 10 monclient: monmap: 2024-04-26T00:28:31.257 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:28:31.257 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:28:31.257 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:28:31.257 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:28:31.258 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:28:31.258 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:28:31.258 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:28:31.258 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:28:31.258 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:28:31.258 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:31.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd47823c640 1 Processor -- start 2024-04-26T00:28:31.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd47823c640 1 -- start start 2024-04-26T00:28:31.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd47823c640 10 monclient: init 2024-04-26T00:28:31.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd47823c640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:31.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd47823c640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:28:31.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd47823c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd4700a0300 0x7fd4701cf450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:31.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd47823c640 10 monclient(hunting): picked mon.noname-c con 0x7fd4700a0300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:28:31.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd47823c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4700a06e0 0x7fd470130b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:31.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd47823c640 10 monclient(hunting): picked mon.noname-b con 0x7fd4700a06e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:28:31.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd47823c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd4701352b0 0x7fd47006ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:31.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd47823c640 10 monclient(hunting): picked mon.noname-a con 0x7fd4701352b0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:28:31.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd47823c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:31.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd47823c640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd470140fa0 con 0x7fd4701352b0 2024-04-26T00:28:31.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd47823c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:31.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd47823c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd470140e20 con 0x7fd4700a06e0 2024-04-26T00:28:31.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd47823c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:31.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd47823c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd470141120 con 0x7fd4700a0300 2024-04-26T00:28:31.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd47823c640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:31.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd47823c640 10 monclient(hunting): authenticate will time out at 1492.570300s 2024-04-26T00:28:31.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd475fd8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd4700a0300 0x7fd4701cf450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:31.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd4767d9640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd4701352b0 0x7fd47006ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:31.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd475fd8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd4700a0300 0x7fd4701cf450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:48588/0 (socket says 172.21.15.37:48588) 2024-04-26T00:28:31.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd475fd8640 1 -- 172.21.15.37:0/3173967173 learned_addr learned my addr 172.21.15.37:0/3173967173 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:28:31.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd475fd8640 10 monclient(hunting): get_auth_request con 0x7fd4700a0300 auth_method 0 2024-04-26T00:28:31.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd475fd8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:31.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd475fd8640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:31.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd475fd8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:31.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd4767d9640 10 monclient(hunting): get_auth_request con 0x7fd4701352b0 auth_method 0 2024-04-26T00:28:31.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd4767d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:31.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd4767d9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:31.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd4767d9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:31.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd475fd8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:31.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd475fd8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:31.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd475fd8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:31.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd4767d9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:31.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd4767d9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:31.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd4767d9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:31.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd4757d7640 1 --2- 172.21.15.37:0/3173967173 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4700a06e0 0x7fd470130b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:31.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd4757d7640 10 monclient(hunting): get_auth_request con 0x7fd4700a06e0 auth_method 0 2024-04-26T00:28:31.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd4757d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:31.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd4757d7640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:31.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd4757d7640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:31.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd475fd8640 10 monclient(hunting): handle_auth_done global_id 84302 payload 824 2024-04-26T00:28:31.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd475fd8640 1 -- 172.21.15.37:0/3173967173 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4700a06e0 msgr2=0x7fd470130b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:31.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd475fd8640 1 --2- 172.21.15.37:0/3173967173 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4700a06e0 0x7fd470130b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:31.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd475fd8640 1 -- 172.21.15.37:0/3173967173 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd4701352b0 msgr2=0x7fd47006ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:31.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd475fd8640 1 --2- 172.21.15.37:0/3173967173 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd4701352b0 0x7fd47006ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:31.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd475fd8640 10 monclient: _finish_hunting 0 2024-04-26T00:28:31.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd475fd8640 1 monclient: found mon.noname-c 2024-04-26T00:28:31.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd475fd8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:31.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd475fd8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:28:31.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.263+0000 7fd475fd8640 1 -- 172.21.15.37:0/3173967173 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4701d0360 con 0x7fd4700a0300 2024-04-26T00:28:31.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd475fd8640 1 -- 172.21.15.37:0/3173967173 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd4700a0300 msgr2=0x7fd4701cf450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:31.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd475fd8640 1 --2- 172.21.15.37:0/3173967173 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd4700a0300 0x7fd4701cf450 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd46801a530 tx=0x7fd4680196a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:31.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd4667fc640 1 -- 172.21.15.37:0/3173967173 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4680bc020 con 0x7fd4700a0300 2024-04-26T00:28:31.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd4667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:31.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd4667fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:28:31.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd4667fc640 10 monclient: dump: 2024-04-26T00:28:31.270 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:31.270 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:31.270 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.270 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.270 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:31.270 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:31.271 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:31.271 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:31.271 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:31.271 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:31.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd4667fc640 10 monclient: _finish_auth 0 2024-04-26T00:28:31.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd4667fc640 10 monclient: _check_auth_tickets 2024-04-26T00:28:31.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd4667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:31.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd4667fc640 1 -- 172.21.15.37:0/3173967173 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4680b8880 con 0x7fd4700a0300 2024-04-26T00:28:31.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd4667fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:28:31.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd4667fc640 1 -- 172.21.15.37:0/3173967173 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4680b8c70 con 0x7fd4700a0300 2024-04-26T00:28:31.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd4667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:31.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd4667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:28:31.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd4667fc640 10 monclient: dump: 2024-04-26T00:28:31.272 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:31.272 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:31.272 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.272 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.273 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:31.273 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:31.273 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:31.273 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:31.273 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:31.273 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:31.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd467fff640 4 set_mon_vals no callback set 2024-04-26T00:28:31.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd47823c640 5 monclient: authenticate success, global_id 84302 2024-04-26T00:28:31.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd47823c640 10 monclient: _renew_subs 2024-04-26T00:28:31.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd47823c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:28:31.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd47823c640 1 -- 172.21.15.37:0/3173967173 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd470070930 con 0x7fd4700a0300 2024-04-26T00:28:31.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd47823c640 10 monclient: _renew_subs 2024-04-26T00:28:31.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd47823c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:28:31.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd47823c640 1 -- 172.21.15.37:0/3173967173 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4701319d0 con 0x7fd4700a0300 2024-04-26T00:28:31.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd47823c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:31.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd47823c640 1 --2- 172.21.15.37:0/3173967173 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd470131050 0x7fd47002c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:31.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd4667fc640 1 -- 172.21.15.37:0/3173967173 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd4680bf020 con 0x7fd4700a0300 2024-04-26T00:28:31.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd47823c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:31.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd47823c640 1 -- 172.21.15.37:0/3173967173 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd47011c410 con 0x7fd470131050 2024-04-26T00:28:31.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd4667fc640 1 --2- 172.21.15.37:0/3173967173 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd454035ed0 0x7fd454038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:31.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd4757d7640 1 --2- 172.21.15.37:0/3173967173 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd470131050 0x7fd47002c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:31.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd4667fc640 1 -- 172.21.15.37:0/3173967173 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(336..336 src has 177..336) ==== 3975+0+0 (secure 0 0 0) 0x7fd4680e3af0 con 0x7fd4700a0300 2024-04-26T00:28:31.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd4757d7640 10 monclient: get_auth_request con 0x7fd470131050 auth_method 0 2024-04-26T00:28:31.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd4757d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:31.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd4757d7640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:31.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd4757d7640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:31.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd4757d7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:31.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd4757d7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:31.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd4757d7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:31.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd4757d7640 10 monclient(hunting): handle_auth_done global_id 84305 payload 824 2024-04-26T00:28:31.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd4757d7640 1 -- 172.21.15.37:0/3173967173 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd47011c230 con 0x7fd470131050 2024-04-26T00:28:31.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd4767d9640 1 --2- 172.21.15.37:0/3173967173 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd454035ed0 0x7fd454038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:31.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.267+0000 7fd4767d9640 10 monclient: get_auth_request con 0x7fd454035ed0 auth_method 0 2024-04-26T00:28:31.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.271+0000 7fd4757d7640 1 -- 172.21.15.37:0/3173967173 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd470131050 msgr2=0x7fd47002c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:31.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.271+0000 7fd4757d7640 1 --2- 172.21.15.37:0/3173967173 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd470131050 0x7fd47002c000 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd46000e420 tx=0x7fd46000e8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:31.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.271+0000 7fd4667fc640 1 -- 172.21.15.37:0/3173967173 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd460007450 con 0x7fd470131050 2024-04-26T00:28:31.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.271+0000 7fd4667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:31.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.271+0000 7fd4667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:28:31.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.271+0000 7fd4667fc640 10 monclient: dump: 2024-04-26T00:28:31.277 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:31.277 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:31.277 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.277 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.277 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:31.278 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:31.278 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:31.278 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:31.278 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:31.278 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:31.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.271+0000 7fd4667fc640 1 -- 172.21.15.37:0/3173967173 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd460007450 con 0x7fd470131050 2024-04-26T00:28:31.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.271+0000 7fd4667fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:31.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.271+0000 7fd4667fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:28:31.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.271+0000 7fd4667fc640 1 -- 172.21.15.37:0/3173967173 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd470131050 msgr2=0x7fd47002c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:31.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.271+0000 7fd4667fc640 1 --2- 172.21.15.37:0/3173967173 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd470131050 0x7fd47002c000 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd46000e420 tx=0x7fd46000e8e0 comp rx=0 tx=0).stop 2024-04-26T00:28:31.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.271+0000 7fd4667fc640 1 -- 172.21.15.37:0/3173967173 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd470131050 msgr2=0x7fd47002c000 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:28:31.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.271+0000 7fd4667fc640 1 --2- 172.21.15.37:0/3173967173 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd470131050 0x7fd47002c000 secure :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd46000e420 tx=0x7fd46000e8e0 comp rx=0 tx=0).stop 2024-04-26T00:28:31.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.271+0000 7fd4767d9640 1 --2- 172.21.15.37:0/3173967173 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd454035ed0 0x7fd454038390 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd46c007240 tx=0x7fd46c007df0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:31.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.279+0000 7fd47823c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:28:31.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.279+0000 7fd47823c640 1 --2- 172.21.15.37:0/3173967173 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd47002c540 0x7fd47002c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:31.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.279+0000 7fd47823c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:31.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.279+0000 7fd47823c640 1 -- 172.21.15.37:0/3173967173 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd47013b790 con 0x7fd47002c540 2024-04-26T00:28:31.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.279+0000 7fd4757d7640 1 --2- 172.21.15.37:0/3173967173 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd47002c540 0x7fd47002c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:31.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.279+0000 7fd4757d7640 10 monclient: get_auth_request con 0x7fd47002c540 auth_method 0 2024-04-26T00:28:31.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.279+0000 7fd4757d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:31.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.279+0000 7fd4757d7640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:31.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.279+0000 7fd4757d7640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:31.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.279+0000 7fd4757d7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:31.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.279+0000 7fd4757d7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:31.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.279+0000 7fd4757d7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:31.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.279+0000 7fd4757d7640 10 monclient(hunting): handle_auth_done global_id 84308 payload 824 2024-04-26T00:28:31.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.279+0000 7fd4757d7640 1 -- 172.21.15.37:0/3173967173 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd47009eb20 con 0x7fd47002c540 2024-04-26T00:28:31.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.279+0000 7fd4757d7640 1 -- 172.21.15.37:0/3173967173 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd47002c540 msgr2=0x7fd47002c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:31.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.279+0000 7fd4757d7640 1 --2- 172.21.15.37:0/3173967173 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd47002c540 0x7fd47002c950 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd46000e420 tx=0x7fd460003770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:31.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.279+0000 7fd4667fc640 1 -- 172.21.15.37:0/3173967173 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd46001eb70 con 0x7fd47002c540 2024-04-26T00:28:31.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.279+0000 7fd4667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:31.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.279+0000 7fd4667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:28:31.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.279+0000 7fd4667fc640 10 monclient: dump: 2024-04-26T00:28:31.280 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:31.280 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:31.280 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.280 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.281 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:31.281 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:31.281 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:31.281 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:31.281 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:31.281 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:31.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.279+0000 7fd4667fc640 1 -- 172.21.15.37:0/3173967173 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7fd46001eb70 con 0x7fd47002c540 2024-04-26T00:28:31.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.279+0000 7fd4667fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:28:31.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.279+0000 7fd4667fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:28:31.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.279+0000 7fd4667fc640 1 -- 172.21.15.37:0/3173967173 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd47002c540 msgr2=0x7fd47002c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:31.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.279+0000 7fd4667fc640 1 --2- 172.21.15.37:0/3173967173 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd47002c540 0x7fd47002c950 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd46000e420 tx=0x7fd460003770 comp rx=0 tx=0).stop 2024-04-26T00:28:31.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.279+0000 7fd4667fc640 1 -- 172.21.15.37:0/3173967173 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd47002c540 msgr2=0x7fd47002c950 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:28:31.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.279+0000 7fd4667fc640 1 --2- 172.21.15.37:0/3173967173 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd47002c540 0x7fd47002c950 secure :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd46000e420 tx=0x7fd460003770 comp rx=0 tx=0).stop 2024-04-26T00:28:31.282 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:28:31.282 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:28:31.282 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:28:31.282 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:28:31.282 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 64, 2024-04-26T00:28:31.282 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 116214, 2024-04-26T00:28:31.282 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:28:31.282 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:28:31.282 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:28:31.282 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:28:31.283 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:31.283 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 3, 2024-04-26T00:28:31.283 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:28:31.283 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:28:31.283 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:28:31.283 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:28:31.283 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:28:31.283 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:28:31.283 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:28:31.283 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:28:31.283 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:28:31.283 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:28:31.283 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:28:31.283 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:28:31.284 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:28:31.284 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:28:31.284 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:31.284 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:28:31.284 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:28:31.284 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:28:31.284 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:28:31.284 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:28:31.284 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:28:31.284 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:28:31.284 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:28:31.284 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:28:31.284 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:28:31.284 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:28:31.285 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:28:31.285 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:28:31.285 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:31.285 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:31.285 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:28:31.285 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:28:31.285 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:28:31.285 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:28:31.285 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:28:31.285 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:28:31.286 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:28:31.286 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:28:31.286 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:28:31.286 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:28:31.286 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:28:31.286 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:28:31.286 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:28:31.286 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:28:31.286 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:28:31.286 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:28:31.286 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:28:31.286 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:28:31.286 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:28:31.287 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:28:31.287 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:28:31.287 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:28:31.287 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:28:31.287 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:28:31.287 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:28:31.287 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:28:31.287 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:28:31.287 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:28:31.287 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:31.287 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:28:31.287 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:31.287 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:28:31.287 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:31.287 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:28:31.288 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:28:31.288 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:28:31.288 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:28:31.288 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:31.288 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:28:31.288 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:28:31.288 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:31.288 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:31.288 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:31.288 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:28:31.288 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:28:31.288 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:31.288 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:31.288 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:31.289 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:31.289 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:28:31.289 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:28:31.289 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:28:31.289 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:28:31.289 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:28:31.289 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:31.289 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:31.289 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:28:31.289 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:28:31.289 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:28:31.289 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:28:31.289 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:31.289 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:28:31.290 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:28:31.290 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:31.290 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:31.290 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:31.290 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:28:31.290 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:28:31.290 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:31.290 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:31.290 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:31.290 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:31.290 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:28:31.290 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:28:31.290 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:28:31.291 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:28:31.291 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:28:31.291 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:31.291 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:31.291 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:28:31.291 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:28:31.291 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:28:31.291 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:28:31.291 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:31.291 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:28:31.291 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:28:31.291 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:31.291 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:31.291 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:31.291 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:28:31.292 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:28:31.292 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:31.292 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:31.292 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:31.292 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:31.292 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:28:31.292 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:28:31.292 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:28:31.292 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:28:31.292 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:28:31.292 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:31.292 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:31.292 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:31.292 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:28:31.293 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:28:31.293 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:31.293 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:28:31.293 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:28:31.293 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:28:31.293 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:31.293 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:31.293 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:28:31.293 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:31.293 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:28:31.293 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:28:31.293 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:28:31.293 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:31.293 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:31.294 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:28:31.294 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:31.294 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:28:31.294 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:28:31.294 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:28:31.294 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:31.294 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:31.294 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:28:31.294 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:31.294 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:28:31.294 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:28:31.294 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:28:31.294 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:31.294 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:31.295 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:31.295 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:28:31.295 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:28:31.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.279+0000 7fd47823c640 1 -- 172.21.15.37:0/3173967173 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd454035ed0 msgr2=0x7fd454038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:31.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.279+0000 7fd47823c640 1 --2- 172.21.15.37:0/3173967173 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd454035ed0 0x7fd454038390 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd46c007240 tx=0x7fd46c007df0 comp rx=0 tx=0).stop 2024-04-26T00:28:31.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.279+0000 7fd47823c640 10 monclient: shutdown 2024-04-26T00:28:31.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.279+0000 7fd47823c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:28:31.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.279+0000 7fd47823c640 1 -- 172.21.15.37:0/3173967173 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd4700a0300 msgr2=0x7fd4701cf450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:31.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.279+0000 7fd47823c640 1 --2- 172.21.15.37:0/3173967173 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd4700a0300 0x7fd4701cf450 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd46801a530 tx=0x7fd4680196a0 comp rx=0 tx=0).stop 2024-04-26T00:28:31.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.279+0000 7fd4767d9640 1 -- 172.21.15.37:0/3173967173 reap_dead start 2024-04-26T00:28:31.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.279+0000 7fd4767d9640 1 -- 172.21.15.37:0/3173967173 reap_dead start 2024-04-26T00:28:31.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.279+0000 7fd47823c640 1 -- 172.21.15.37:0/3173967173 shutdown_connections 2024-04-26T00:28:31.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.279+0000 7fd47823c640 1 -- 172.21.15.37:0/3173967173 >> 172.21.15.37:0/3173967173 conn(0x7fd4700a1d50 msgr2=0x7fd470133c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:28:31.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.279+0000 7fd47823c640 1 -- 172.21.15.37:0/3173967173 shutdown_connections 2024-04-26T00:28:31.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.279+0000 7fd47823c640 1 -- 172.21.15.37:0/3173967173 wait complete. 2024-04-26T00:28:31.297 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:28:31.297 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:28:31.297 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:28:31.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a30521640 10 monclient: get_monmap_and_config 2024-04-26T00:28:31.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a30521640 10 monclient: build_initial_monmap 2024-04-26T00:28:31.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a30521640 10 monclient: monmap: 2024-04-26T00:28:31.414 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:28:31.414 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:31.414 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:28:31.429374+0000 2024-04-26T00:28:31.414 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:28:31.429374+0000 2024-04-26T00:28:31.414 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:28:31.414 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:31.414 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:28:31.414 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:28:31.414 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:28:31.414 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:28:31.414 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:28:31.414 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:31.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a30521640 1 Processor -- start 2024-04-26T00:28:31.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a30521640 1 -- start start 2024-04-26T00:28:31.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a30521640 10 monclient: init 2024-04-26T00:28:31.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a30521640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:31.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a30521640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-04-26T00:28:31.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a30521640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2a2809ee60 addr v1:172.21.15.37:6789/0 2024-04-26T00:28:31.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a30521640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2a2809f240 0x7f2a2813c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:31.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a30521640 10 monclient(hunting): picked mon.noname-c con 0x7f2a2809f240 addr v2:172.21.15.45:3300/0 2024-04-26T00:28:31.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a30521640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f2a2809f6c0 0x7f2a2813e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:31.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a30521640 10 monclient(hunting): picked mon.noname-a con 0x7f2a2809f6c0 addr v2:172.21.15.37:3300/0 2024-04-26T00:28:31.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a30521640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a2809d090 con 0x7f2a2809ee60 2024-04-26T00:28:31.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a30521640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:31.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a30521640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f2a28134440 con 0x7f2a2809f6c0 2024-04-26T00:28:31.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a30521640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:31.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a30521640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f2a2809d370 con 0x7f2a2809f240 2024-04-26T00:28:31.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a30521640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:31.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a30521640 10 monclient(hunting): authenticate will time out at 1492.734958s 2024-04-26T00:28:31.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a2eabe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f2a2809f6c0 0x7f2a2813e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:31.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a2eabe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f2a2809f6c0 0x7f2a2813e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:49780/0 (socket says 172.21.15.37:49780) 2024-04-26T00:28:31.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a2eabe640 1 -- 172.21.15.37:0/2560842798 learned_addr learned my addr 172.21.15.37:0/2560842798 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:28:31.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a2eabe640 10 monclient(hunting): get_auth_request con 0x7f2a2809f6c0 auth_method 0 2024-04-26T00:28:31.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a2eabe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:31.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a2eabe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:31.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a2eabe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:31.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a2dabc640 1 --2- 172.21.15.37:0/2560842798 >> v2:172.21.15.45:3300/0 conn(0x7f2a2809f240 0x7f2a2813c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:31.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a2eabe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:31.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a2eabe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:31.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a2eabe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:31.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a2dabc640 10 monclient(hunting): get_auth_request con 0x7f2a2809f240 auth_method 0 2024-04-26T00:28:31.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a2dabc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:31.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a2dabc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:31.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a2dabc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:31.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a2eabe640 10 monclient(hunting): handle_auth_done global_id 94233 payload 274 2024-04-26T00:28:31.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a2eabe640 1 -- 172.21.15.37:0/2560842798 >> v2:172.21.15.45:3300/0 conn(0x7f2a2809f240 msgr2=0x7f2a2813c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:28:31.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a2eabe640 1 --2- 172.21.15.37:0/2560842798 >> v2:172.21.15.45:3300/0 conn(0x7f2a2809f240 0x7f2a2813c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:31.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a2eabe640 1 -- 172.21.15.37:0/2560842798 >> v1:172.21.15.37:6789/0 conn(0x7f2a2809ee60 legacy=0x7f2a28133d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:28:31.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a2eabe640 10 monclient: _finish_hunting 0 2024-04-26T00:28:31.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a2eabe640 1 monclient: found mon.noname-a 2024-04-26T00:28:31.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a2eabe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:31.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a2eabe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:28:31.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a2eabe640 1 -- 172.21.15.37:0/2560842798 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a2809d650 con 0x7f2a2809f6c0 2024-04-26T00:28:31.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.427+0000 7f2a2dabc640 1 --2- 172.21.15.37:0/2560842798 >> v2:172.21.15.45:3300/0 conn(0x7f2a2809f240 0x7f2a2813c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:28:31.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.431+0000 7f2a2eabe640 1 -- 172.21.15.37:0/2560842798 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a2809f6c0 msgr2=0x7f2a2813e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:31.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.431+0000 7f2a2eabe640 1 --2- 172.21.15.37:0/2560842798 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a2809f6c0 0x7f2a2813e650 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2a280a1f70 tx=0x7f2a2001c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc72db1fbf59d4a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:31.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2d2bb640 1 -- 172.21.15.37:0/2560842798 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a200a5640 con 0x7f2a2809f6c0 2024-04-26T00:28:31.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2d2bb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:31.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2d2bb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:28:31.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2d2bb640 10 monclient: dump: 2024-04-26T00:28:31.422 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:31.422 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:31.422 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.422 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.422 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:31.422 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:31.422 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:31.422 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:31.422 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:31.422 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:31.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2d2bb640 10 monclient: _finish_auth 0 2024-04-26T00:28:31.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2d2bb640 10 monclient: _check_auth_tickets 2024-04-26T00:28:31.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2d2bb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:31.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a30521640 5 monclient: authenticate success, global_id 94233 2024-04-26T00:28:31.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a30521640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:28:31.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2d2bb640 1 -- 172.21.15.37:0/2560842798 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a200a57e0 con 0x7f2a2809f6c0 2024-04-26T00:28:31.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2d2bb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:28:31.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2d2bb640 1 -- 172.21.15.37:0/2560842798 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a200a5bb0 con 0x7f2a2809f6c0 2024-04-26T00:28:31.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a30521640 10 monclient: get_monmap_and_config success 2024-04-26T00:28:31.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a30521640 4 set_mon_vals no callback set 2024-04-26T00:28:31.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2d2bb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:31.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2d2bb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:28:31.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2d2bb640 10 monclient: dump: 2024-04-26T00:28:31.423 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:31.423 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:31.424 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.424 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.424 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:31.424 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:31.424 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:31.424 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:31.424 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:31.424 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:31.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a30521640 10 monclient: shutdown 2024-04-26T00:28:31.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a30521640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:28:31.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a30521640 1 -- 172.21.15.37:0/2560842798 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a2809f6c0 msgr2=0x7f2a2813e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:31.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a30521640 1 --2- 172.21.15.37:0/2560842798 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a2809f6c0 0x7f2a2813e650 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2a280a1f70 tx=0x7f2a2001c870 comp rx=0 tx=0).stop 2024-04-26T00:28:31.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a30521640 1 -- 172.21.15.37:0/2560842798 shutdown_connections 2024-04-26T00:28:31.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a30521640 1 --2- 172.21.15.37:0/2560842798 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a2809f6c0 0x7f2a2813e650 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:31.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a30521640 1 --2- 172.21.15.37:0/2560842798 >> v2:172.21.15.45:3300/0 conn(0x7f2a2809f240 0x7f2a2813c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:31.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a30521640 1 -- 172.21.15.37:0/2560842798 >> 172.21.15.37:0/2560842798 conn(0x7f2a280a5710 msgr2=0x7f2a280a5b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:28:31.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a30521640 1 -- 172.21.15.37:0/2560842798 shutdown_connections 2024-04-26T00:28:31.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a30521640 1 -- 172.21.15.37:0/2560842798 wait complete. 2024-04-26T00:28:31.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a30521640 10 monclient: build_initial_monmap 2024-04-26T00:28:31.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a30521640 10 monclient: monmap: 2024-04-26T00:28:31.425 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:28:31.425 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:28:31.425 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:28:31.425 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:28:31.425 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:28:31.425 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:28:31.425 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:28:31.425 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:28:31.425 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:28:31.426 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:31.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a30521640 1 Processor -- start 2024-04-26T00:28:31.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a30521640 1 -- start start 2024-04-26T00:28:31.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a30521640 10 monclient: init 2024-04-26T00:28:31.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a30521640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:31.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a30521640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:28:31.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a30521640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a2809ee60 0x7f2a281cbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:31.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a30521640 10 monclient(hunting): picked mon.noname-b con 0x7f2a2809ee60 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:28:31.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a30521640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a2809f240 0x7f2a2813d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:31.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a30521640 10 monclient(hunting): picked mon.noname-c con 0x7f2a2809f240 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:28:31.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a30521640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a2809f6c0 0x7f2a281cf130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:31.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a30521640 10 monclient(hunting): picked mon.noname-a con 0x7f2a2809f6c0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:28:31.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a30521640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:31.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a30521640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f2a28141780 con 0x7f2a2809f6c0 2024-04-26T00:28:31.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a30521640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:31.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a30521640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2a28141600 con 0x7f2a2809ee60 2024-04-26T00:28:31.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a30521640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:31.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a30521640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2a28141900 con 0x7f2a2809f240 2024-04-26T00:28:31.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a30521640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:31.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a30521640 10 monclient(hunting): authenticate will time out at 1492.742590s 2024-04-26T00:28:31.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2dabc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a2809f240 0x7f2a2813d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:31.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2dabc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a2809f240 0x7f2a2813d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:48608/0 (socket says 172.21.15.37:48608) 2024-04-26T00:28:31.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2dabc640 1 -- 172.21.15.37:0/2031092494 learned_addr learned my addr 172.21.15.37:0/2031092494 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:28:31.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2dabc640 10 monclient(hunting): get_auth_request con 0x7f2a2809f240 auth_method 0 2024-04-26T00:28:31.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2dabc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:31.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2dabc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:31.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2dabc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:31.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2eabe640 1 --2- 172.21.15.37:0/2031092494 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a2809f6c0 0x7f2a281cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:31.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2eabe640 10 monclient(hunting): get_auth_request con 0x7f2a2809f6c0 auth_method 0 2024-04-26T00:28:31.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2eabe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:31.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2eabe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:31.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2eabe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:31.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2dabc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:31.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2dabc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:31.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2dabc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:31.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2eabe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:31.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2eabe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:31.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2eabe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:31.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2e2bd640 1 --2- 172.21.15.37:0/2031092494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a2809ee60 0x7f2a281cbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:31.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2e2bd640 10 monclient(hunting): get_auth_request con 0x7f2a2809ee60 auth_method 0 2024-04-26T00:28:31.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2e2bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:31.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2e2bd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:31.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2e2bd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:31.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2dabc640 10 monclient(hunting): handle_auth_done global_id 84311 payload 824 2024-04-26T00:28:31.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2dabc640 1 -- 172.21.15.37:0/2031092494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a2809ee60 msgr2=0x7f2a281cbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:31.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2dabc640 1 --2- 172.21.15.37:0/2031092494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a2809ee60 0x7f2a281cbbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:31.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2dabc640 1 -- 172.21.15.37:0/2031092494 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a2809f6c0 msgr2=0x7f2a281cf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:31.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2dabc640 1 --2- 172.21.15.37:0/2031092494 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a2809f6c0 0x7f2a281cf130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:31.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2dabc640 10 monclient: _finish_hunting 0 2024-04-26T00:28:31.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2dabc640 1 monclient: found mon.noname-c 2024-04-26T00:28:31.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2dabc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:31.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2dabc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:28:31.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2dabc640 1 -- 172.21.15.37:0/2031092494 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a2812fb90 con 0x7f2a2809f240 2024-04-26T00:28:31.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.435+0000 7f2a2eabe640 1 --2- 172.21.15.37:0/2031092494 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a2809f6c0 0x7f2a281cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:28:31.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a2dabc640 1 -- 172.21.15.37:0/2031092494 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a2809f240 msgr2=0x7f2a2813d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:31.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a2dabc640 1 --2- 172.21.15.37:0/2031092494 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a2809f240 0x7f2a2813d210 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2a2400e810 tx=0x7f2a2400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:31.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a1affd640 1 -- 172.21.15.37:0/2031092494 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a24017070 con 0x7f2a2809f240 2024-04-26T00:28:31.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a1affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:31.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a1affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:28:31.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a1affd640 10 monclient: dump: 2024-04-26T00:28:31.435 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:31.435 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:31.435 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.435 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.435 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:31.435 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:31.435 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:31.435 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:31.436 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:31.436 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:31.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a1affd640 10 monclient: _finish_auth 0 2024-04-26T00:28:31.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a1affd640 10 monclient: _check_auth_tickets 2024-04-26T00:28:31.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:31.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a1affd640 1 -- 172.21.15.37:0/2031092494 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a2400cae0 con 0x7f2a2809f240 2024-04-26T00:28:31.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a30521640 5 monclient: authenticate success, global_id 84311 2024-04-26T00:28:31.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a1affd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:28:31.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a1affd640 1 -- 172.21.15.37:0/2031092494 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a2400f040 con 0x7f2a2809f240 2024-04-26T00:28:31.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a30521640 10 monclient: _renew_subs 2024-04-26T00:28:31.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a30521640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:28:31.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a30521640 1 -- 172.21.15.37:0/2031092494 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a280a4a90 con 0x7f2a2809f240 2024-04-26T00:28:31.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a1affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:31.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a1affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:28:31.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a1affd640 10 monclient: dump: 2024-04-26T00:28:31.437 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:31.437 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:31.437 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.437 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.437 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:31.437 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:31.437 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:31.437 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:31.437 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:31.437 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:31.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a30521640 10 monclient: _renew_subs 2024-04-26T00:28:31.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a30521640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:28:31.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a30521640 1 -- 172.21.15.37:0/2031092494 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a281315e0 con 0x7f2a2809f240 2024-04-26T00:28:31.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a2caba640 4 set_mon_vals no callback set 2024-04-26T00:28:31.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a30521640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:31.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a30521640 1 --2- 172.21.15.37:0/2031092494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a10002890 0x7f2a10004d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:31.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a30521640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:31.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a30521640 1 -- 172.21.15.37:0/2031092494 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2a10007cc0 con 0x7f2a10002890 2024-04-26T00:28:31.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a1affd640 1 -- 172.21.15.37:0/2031092494 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2a2401a070 con 0x7f2a2809f240 2024-04-26T00:28:31.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a1affd640 1 --2- 172.21.15.37:0/2031092494 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2a0c035ed0 0x7f2a0c038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:31.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a1affd640 1 -- 172.21.15.37:0/2031092494 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(336..336 src has 177..336) ==== 3975+0+0 (secure 0 0 0) 0x7f2a24048720 con 0x7f2a2809f240 2024-04-26T00:28:31.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a2e2bd640 1 --2- 172.21.15.37:0/2031092494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a10002890 0x7f2a10004d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:31.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a2e2bd640 10 monclient: get_auth_request con 0x7f2a10002890 auth_method 0 2024-04-26T00:28:31.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a2e2bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:31.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a2e2bd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:31.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a2e2bd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:31.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a2e2bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:31.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a2e2bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:31.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a2eabe640 1 --2- 172.21.15.37:0/2031092494 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2a0c035ed0 0x7f2a0c038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:31.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a2e2bd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:31.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a2eabe640 10 monclient: get_auth_request con 0x7f2a0c035ed0 auth_method 0 2024-04-26T00:28:31.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a2e2bd640 10 monclient(hunting): handle_auth_done global_id 104110 payload 824 2024-04-26T00:28:31.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.439+0000 7f2a2e2bd640 1 -- 172.21.15.37:0/2031092494 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2a10007ec0 con 0x7f2a10002890 2024-04-26T00:28:31.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.443+0000 7f2a2e2bd640 1 -- 172.21.15.37:0/2031092494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a10002890 msgr2=0x7f2a10004d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:31.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.443+0000 7f2a2e2bd640 1 --2- 172.21.15.37:0/2031092494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a10002890 0x7f2a10004d30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c00e4d0 tx=0x7f2a1c00e990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:31.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.443+0000 7f2a2eabe640 1 --2- 172.21.15.37:0/2031092494 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2a0c035ed0 0x7f2a0c038390 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2a280685c0 tx=0x7f2a200a7070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:31.440 INFO:tasks.rados.rados.0.smithi045.stdout:1456: done (0 left) 2024-04-26T00:28:31.440 INFO:tasks.rados.rados.0.smithi045.stdout:1457: rollback oid 46 current snap is 168 2024-04-26T00:28:31.440 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 46 to 159 2024-04-26T00:28:31.440 INFO:tasks.rados.rados.0.smithi045.stdout:1458: read oid 48 snap 154 2024-04-26T00:28:31.440 INFO:tasks.rados.rados.0.smithi045.stdout:1458: expect (ObjNum 359 snap 107 seq_num 359) 2024-04-26T00:28:31.441 INFO:tasks.rados.rados.0.smithi045.stdout:1459: write oid 2 current snap is 168 2024-04-26T00:28:31.441 INFO:tasks.rados.rados.0.smithi045.stdout:1459: seq_num 544 ranges {743008=796519,2200322=721849} 2024-04-26T00:28:31.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.447+0000 7f2a1affd640 1 -- 172.21.15.37:0/2031092494 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a1c007450 con 0x7f2a10002890 2024-04-26T00:28:31.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.447+0000 7f2a1affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:31.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.447+0000 7f2a1affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:28:31.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.447+0000 7f2a1affd640 10 monclient: dump: 2024-04-26T00:28:31.441 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:31.441 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:31.441 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.441 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.441 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:31.441 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:31.442 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:31.442 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:31.442 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:31.442 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:31.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.447+0000 7f2a1affd640 1 -- 172.21.15.37:0/2031092494 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2a1c007450 con 0x7f2a10002890 2024-04-26T00:28:31.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.447+0000 7f2a1affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:31.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.447+0000 7f2a1affd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:28:31.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.447+0000 7f2a1affd640 1 -- 172.21.15.37:0/2031092494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a10002890 msgr2=0x7f2a10004d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:31.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.447+0000 7f2a1affd640 1 --2- 172.21.15.37:0/2031092494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a10002890 0x7f2a10004d30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c00e4d0 tx=0x7f2a1c00e990 comp rx=0 tx=0).stop 2024-04-26T00:28:31.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.447+0000 7f2a1affd640 1 -- 172.21.15.37:0/2031092494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a10002890 msgr2=0x7f2a10004d30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:28:31.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.447+0000 7f2a1affd640 1 --2- 172.21.15.37:0/2031092494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a10002890 0x7f2a10004d30 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c00e4d0 tx=0x7f2a1c00e990 comp rx=0 tx=0).stop 2024-04-26T00:28:31.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.455+0000 7f2a30521640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:28:31.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.455+0000 7f2a30521640 1 --2- 172.21.15.37:0/2031092494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a10006ce0 0x7f2a10007170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:31.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.455+0000 7f2a30521640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:31.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.455+0000 7f2a30521640 1 -- 172.21.15.37:0/2031092494 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2a1000a1e0 con 0x7f2a10006ce0 2024-04-26T00:28:31.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.455+0000 7f2a2e2bd640 1 --2- 172.21.15.37:0/2031092494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a10006ce0 0x7f2a10007170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:31.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.455+0000 7f2a2e2bd640 10 monclient: get_auth_request con 0x7f2a10006ce0 auth_method 0 2024-04-26T00:28:31.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.455+0000 7f2a2e2bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:31.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.455+0000 7f2a2e2bd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:31.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.455+0000 7f2a2e2bd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:31.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.455+0000 7f2a2e2bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:31.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.455+0000 7f2a2e2bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:31.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.455+0000 7f2a2e2bd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:31.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.455+0000 7f2a2e2bd640 10 monclient(hunting): handle_auth_done global_id 104113 payload 824 2024-04-26T00:28:31.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.455+0000 7f2a2e2bd640 1 -- 172.21.15.37:0/2031092494 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2a100076b0 con 0x7f2a10006ce0 2024-04-26T00:28:31.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.459+0000 7f2a2e2bd640 1 -- 172.21.15.37:0/2031092494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a10006ce0 msgr2=0x7f2a10007170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:31.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.459+0000 7f2a2e2bd640 1 --2- 172.21.15.37:0/2031092494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a10006ce0 0x7f2a10007170 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c00e4d0 tx=0x7f2a1c003770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:31.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.459+0000 7f2a1affd640 1 -- 172.21.15.37:0/2031092494 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a1c01e960 con 0x7f2a10006ce0 2024-04-26T00:28:31.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.459+0000 7f2a1affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:31.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.459+0000 7f2a1affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:28:31.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.459+0000 7f2a1affd640 10 monclient: dump: 2024-04-26T00:28:31.446 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:31.446 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:31.446 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.446 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.447 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:31.447 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:31.447 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:31.447 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:31.447 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:31.447 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:31.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.459+0000 7f2a1affd640 1 -- 172.21.15.37:0/2031092494 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7f2a1c028460 con 0x7f2a10006ce0 2024-04-26T00:28:31.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.459+0000 7f2a1affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:28:31.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.459+0000 7f2a1affd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:28:31.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.459+0000 7f2a1affd640 1 -- 172.21.15.37:0/2031092494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a10006ce0 msgr2=0x7f2a10007170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:31.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.459+0000 7f2a1affd640 1 --2- 172.21.15.37:0/2031092494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a10006ce0 0x7f2a10007170 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c00e4d0 tx=0x7f2a1c003770 comp rx=0 tx=0).stop 2024-04-26T00:28:31.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.459+0000 7f2a1affd640 1 -- 172.21.15.37:0/2031092494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a10006ce0 msgr2=0x7f2a10007170 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:28:31.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.459+0000 7f2a1affd640 1 --2- 172.21.15.37:0/2031092494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a10006ce0 0x7f2a10007170 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c00e4d0 tx=0x7f2a1c003770 comp rx=0 tx=0).stop 2024-04-26T00:28:31.448 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:28:31.448 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:28:31.448 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:28:31.449 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:28:31.449 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 64, 2024-04-26T00:28:31.449 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 18663, 2024-04-26T00:28:31.449 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:28:31.449 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:28:31.449 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:28:31.449 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:28:31.449 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:31.449 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 3, 2024-04-26T00:28:31.449 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:28:31.449 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:28:31.449 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:28:31.449 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:28:31.450 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:28:31.450 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:28:31.450 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:28:31.450 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:28:31.450 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:28:31.450 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:28:31.450 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:28:31.450 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:28:31.450 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:28:31.450 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:28:31.450 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:31.450 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:28:31.450 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:28:31.450 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:28:31.451 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:28:31.451 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:28:31.451 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:28:31.451 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:28:31.451 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:28:31.451 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:28:31.451 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:28:31.451 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:28:31.451 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:28:31.451 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:28:31.451 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:31.451 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:31.451 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:28:31.451 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:28:31.451 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:28:31.452 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:28:31.452 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:28:31.452 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:28:31.452 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:28:31.452 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:28:31.452 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:28:31.452 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:28:31.452 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:28:31.452 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:28:31.452 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:28:31.452 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:28:31.452 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:28:31.452 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:28:31.452 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:28:31.452 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:28:31.453 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:28:31.453 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:28:31.453 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:28:31.453 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:28:31.453 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:28:31.453 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:28:31.453 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:28:31.453 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:28:31.453 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:28:31.453 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:28:31.453 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:31.453 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:28:31.453 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:31.453 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:28:31.454 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:31.454 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:28:31.454 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:28:31.454 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:28:31.454 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:28:31.454 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:31.454 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:28:31.454 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:28:31.454 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:31.454 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:31.454 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:31.454 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:28:31.454 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:28:31.454 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:31.454 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:31.455 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:31.455 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:31.455 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:28:31.455 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:28:31.455 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:28:31.455 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:28:31.455 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:28:31.455 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:31.455 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:31.455 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:28:31.455 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:28:31.455 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:28:31.455 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:28:31.455 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:31.456 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:28:31.456 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:28:31.456 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:31.456 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:31.456 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:31.456 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:28:31.456 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:28:31.456 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:31.456 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:31.456 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:31.456 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:31.456 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:28:31.456 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:28:31.456 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:28:31.456 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:28:31.457 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:28:31.457 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:31.457 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:31.457 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:28:31.457 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:28:31.457 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:28:31.457 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:28:31.457 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:31.457 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:28:31.457 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:28:31.457 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:31.457 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:31.457 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:31.457 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:28:31.457 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:28:31.458 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:31.458 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:31.458 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:31.458 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:31.458 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:28:31.458 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:28:31.458 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:28:31.458 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:28:31.458 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:28:31.458 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:31.458 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:31.458 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:31.458 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:28:31.458 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:28:31.459 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:31.459 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:28:31.459 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:28:31.459 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:28:31.459 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:31.459 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:31.459 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:28:31.459 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:31.459 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:28:31.459 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:28:31.459 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:28:31.459 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:31.459 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:31.459 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:31.460 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:28:31.460 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:28:31.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.459+0000 7f2a30521640 1 -- 172.21.15.37:0/2031092494 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2a0c035ed0 msgr2=0x7f2a0c038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:31.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.459+0000 7f2a30521640 1 --2- 172.21.15.37:0/2031092494 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2a0c035ed0 0x7f2a0c038390 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2a280685c0 tx=0x7f2a200a7070 comp rx=0 tx=0).stop 2024-04-26T00:28:31.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.459+0000 7f2a30521640 10 monclient: shutdown 2024-04-26T00:28:31.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.459+0000 7f2a30521640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:28:31.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.459+0000 7f2a30521640 1 -- 172.21.15.37:0/2031092494 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a2809f240 msgr2=0x7f2a2813d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:31.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.459+0000 7f2a30521640 1 --2- 172.21.15.37:0/2031092494 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a2809f240 0x7f2a2813d210 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2a2400e810 tx=0x7f2a2400ecd0 comp rx=0 tx=0).stop 2024-04-26T00:28:31.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.459+0000 7f2a2eabe640 1 -- 172.21.15.37:0/2031092494 reap_dead start 2024-04-26T00:28:31.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.459+0000 7f2a2eabe640 1 -- 172.21.15.37:0/2031092494 reap_dead start 2024-04-26T00:28:31.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.459+0000 7f2a30521640 1 -- 172.21.15.37:0/2031092494 shutdown_connections 2024-04-26T00:28:31.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.459+0000 7f2a30521640 1 -- 172.21.15.37:0/2031092494 >> 172.21.15.37:0/2031092494 conn(0x7f2a280a5710 msgr2=0x7f2a2813de80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:28:31.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.459+0000 7f2a30521640 1 -- 172.21.15.37:0/2031092494 shutdown_connections 2024-04-26T00:28:31.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.459+0000 7f2a30521640 1 -- 172.21.15.37:0/2031092494 wait complete. 2024-04-26T00:28:31.462 INFO:tasks.rados.rados.0.smithi045.stdout:1459: writing smithi04526579-2 from 743008 to 1539527 tid 1 2024-04-26T00:28:31.466 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:28:31.466 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-26T00:28:31.466 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-26T00:28:31.467 INFO:tasks.rados.rados.0.smithi045.stdout:1459: writing smithi04526579-2 from 2200322 to 2922171 tid 2 2024-04-26T00:28:31.467 INFO:tasks.rados.rados.0.smithi045.stdout:1460: delete oid 35 current snap is 168 2024-04-26T00:28:31.467 INFO:tasks.rados.rados.0.smithi045.stdout:1457: finishing rollback tid 1 to smithi04526579-46 2024-04-26T00:28:31.467 INFO:tasks.rados.rados.0.smithi045.stdout:1458: expect (ObjNum 359 snap 107 seq_num 359) 2024-04-26T00:28:31.549 INFO:tasks.rados.rados.0.smithi045.stdout:1457: finishing rollback tid 2 to smithi04526579-46 2024-04-26T00:28:31.549 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 756 (ObjNum 329 snap 96 seq_num 329) dirty exists 2024-04-26T00:28:31.550 INFO:tasks.rados.rados.0.smithi045.stdout:1459: finishing write tid 1 to smithi04526579-2 2024-04-26T00:28:31.550 INFO:tasks.rados.rados.0.smithi045.stdout:1459: finishing write tid 2 to smithi04526579-2 2024-04-26T00:28:31.550 INFO:tasks.rados.rados.0.smithi045.stdout:1459: finishing write tid 3 to smithi04526579-2 2024-04-26T00:28:31.550 INFO:tasks.rados.rados.0.smithi045.stdout:1459: finishing write tid 4 to smithi04526579-2 2024-04-26T00:28:31.550 INFO:tasks.rados.rados.0.smithi045.stdout:1459: oid 2 updating version 0 to 813 2024-04-26T00:28:31.550 INFO:tasks.rados.rados.0.smithi045.stdout:1459: oid 2 version 813 is already newer than 812 2024-04-26T00:28:31.550 INFO:tasks.rados.rados.0.smithi045.stdout:1459: oid 2 version 813 is already newer than 811 2024-04-26T00:28:31.550 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 813 (ObjNum 544 snap 168 seq_num 544) dirty exists 2024-04-26T00:28:31.550 INFO:tasks.rados.rados.0.smithi045.stdout:1459: left oid 2 (ObjNum 544 snap 168 seq_num 544) 2024-04-26T00:28:31.550 INFO:tasks.rados.rados.0.smithi045.stdout:1457: done (3 left) 2024-04-26T00:28:31.550 INFO:tasks.rados.rados.0.smithi045.stdout:1458: done (2 left) 2024-04-26T00:28:31.550 INFO:tasks.rados.rados.0.smithi045.stdout:1459: done (1 left) 2024-04-26T00:28:31.550 INFO:tasks.rados.rados.0.smithi045.stdout:1460: done (0 left) 2024-04-26T00:28:31.550 INFO:tasks.rados.rados.0.smithi045.stdout:1461: write (excl) oid 44 current snap is 168 2024-04-26T00:28:31.551 INFO:tasks.rados.rados.0.smithi045.stdout:1461: seq_num 546 ranges {776622=547017,1800933=725518,2944601=232021} 2024-04-26T00:28:31.569 INFO:tasks.rados.rados.0.smithi045.stdout:1461: writing smithi04526579-44 from 776622 to 1323639 tid 1 2024-04-26T00:28:31.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.591+0000 7f0d9edef640 10 monclient: get_monmap_and_config 2024-04-26T00:28:31.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.591+0000 7f0d9edef640 10 monclient: build_initial_monmap 2024-04-26T00:28:31.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.591+0000 7f0d9edef640 10 monclient: monmap: 2024-04-26T00:28:31.580 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:28:31.580 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:31.580 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:28:31.595594+0000 2024-04-26T00:28:31.580 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:28:31.595594+0000 2024-04-26T00:28:31.580 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:28:31.580 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:31.580 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:28:31.580 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:28:31.580 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:28:31.580 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:28:31.580 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:28:31.580 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:31.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.595+0000 7f0d9edef640 1 Processor -- start 2024-04-26T00:28:31.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.595+0000 7f0d9edef640 1 -- start start 2024-04-26T00:28:31.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.595+0000 7f0d9edef640 10 monclient: init 2024-04-26T00:28:31.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.595+0000 7f0d9edef640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:31.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.595+0000 7f0d9edef640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-04-26T00:28:31.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.595+0000 7f0d9edef640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f0d98135090 0x7f0d98135470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:31.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.595+0000 7f0d9edef640 10 monclient(hunting): picked mon.noname-a con 0x7f0d98135090 addr v2:172.21.15.37:3300/0 2024-04-26T00:28:31.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.595+0000 7f0d9edef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0d9806ead0 addr v1:172.21.15.37:6789/0 2024-04-26T00:28:31.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.595+0000 7f0d9edef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0d980a1bc0 addr v1:172.21.15.45:6789/0 2024-04-26T00:28:31.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.595+0000 7f0d9edef640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d981359b0 con 0x7f0d9806ead0 2024-04-26T00:28:31.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.595+0000 7f0d9edef640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d9806e610 con 0x7f0d980a1bc0 2024-04-26T00:28:31.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.595+0000 7f0d9edef640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:31.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.595+0000 7f0d9edef640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f0d98071b00 con 0x7f0d98135090 2024-04-26T00:28:31.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.595+0000 7f0d9edef640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:31.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.595+0000 7f0d9edef640 10 monclient(hunting): authenticate will time out at 1492.901319s 2024-04-26T00:28:31.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.595+0000 7f0d9cb8b640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f0d98135090 0x7f0d98135470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:31.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.595+0000 7f0d9cb8b640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f0d98135090 0x7f0d98135470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:49804/0 (socket says 172.21.15.37:49804) 2024-04-26T00:28:31.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.595+0000 7f0d9cb8b640 1 -- 172.21.15.37:0/177248441 learned_addr learned my addr 172.21.15.37:0/177248441 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:28:31.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.595+0000 7f0d9cb8b640 10 monclient(hunting): get_auth_request con 0x7f0d98135090 auth_method 0 2024-04-26T00:28:31.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.595+0000 7f0d97fff640 1 --1- 172.21.15.37:0/177248441 >> v1:172.21.15.37:6789/0 conn(0x7f0d9806ead0 0x7f0d980a14b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:34700/0 (socket says 172.21.15.37:34700) 2024-04-26T00:28:31.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.595+0000 7f0d9cb8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:31.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.595+0000 7f0d9cb8b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:31.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.595+0000 7f0d9cb8b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:31.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.595+0000 7f0d9cb8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:31.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.595+0000 7f0d9cb8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:31.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.595+0000 7f0d9cb8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:31.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.595+0000 7f0d9cb8b640 10 monclient(hunting): handle_auth_done global_id 94239 payload 274 2024-04-26T00:28:31.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.595+0000 7f0d9cb8b640 1 -- 172.21.15.37:0/177248441 >> v1:172.21.15.45:6789/0 conn(0x7f0d980a1bc0 legacy=0x7f0d9813ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:28:31.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.595+0000 7f0d9cb8b640 1 -- 172.21.15.37:0/177248441 >> v1:172.21.15.37:6789/0 conn(0x7f0d9806ead0 legacy=0x7f0d980a14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:28:31.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.595+0000 7f0d9cb8b640 10 monclient: _finish_hunting 0 2024-04-26T00:28:31.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.595+0000 7f0d9cb8b640 1 monclient: found mon.noname-a 2024-04-26T00:28:31.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.595+0000 7f0d9cb8b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:31.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.595+0000 7f0d9cb8b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:28:31.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.595+0000 7f0d9cb8b640 1 -- 172.21.15.37:0/177248441 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d98071de0 con 0x7f0d98135090 2024-04-26T00:28:31.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.599+0000 7f0d9cb8b640 1 -- 172.21.15.37:0/177248441 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d98135090 msgr2=0x7f0d98135470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:31.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.599+0000 7f0d9cb8b640 1 --2- 172.21.15.37:0/177248441 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d98135090 0x7f0d98135470 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0d980a3620 tx=0x7f0d8801c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e2d823562cb6159 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:31.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.599+0000 7f0d977fe640 1 -- 172.21.15.37:0/177248441 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d880b0070 con 0x7f0d98135090 2024-04-26T00:28:31.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.599+0000 7f0d977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:31.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.599+0000 7f0d977fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:28:31.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.599+0000 7f0d977fe640 10 monclient: dump: 2024-04-26T00:28:31.587 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:31.587 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:31.587 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.587 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.587 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:31.587 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:31.587 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:31.588 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:31.588 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:31.588 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:31.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.599+0000 7f0d977fe640 10 monclient: _finish_auth 0 2024-04-26T00:28:31.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.599+0000 7f0d977fe640 10 monclient: _check_auth_tickets 2024-04-26T00:28:31.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.599+0000 7f0d977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:31.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.599+0000 7f0d977fe640 1 -- 172.21.15.37:0/177248441 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d880ab600 con 0x7f0d98135090 2024-04-26T00:28:31.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.599+0000 7f0d9edef640 5 monclient: authenticate success, global_id 94239 2024-04-26T00:28:31.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.599+0000 7f0d9edef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:28:31.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.599+0000 7f0d977fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:28:31.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.599+0000 7f0d9edef640 10 monclient: get_monmap_and_config success 2024-04-26T00:28:31.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.599+0000 7f0d977fe640 1 -- 172.21.15.37:0/177248441 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d880ab9d0 con 0x7f0d98135090 2024-04-26T00:28:31.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.599+0000 7f0d977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:31.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.599+0000 7f0d9edef640 4 set_mon_vals no callback set 2024-04-26T00:28:31.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.599+0000 7f0d977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:28:31.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.599+0000 7f0d977fe640 10 monclient: dump: 2024-04-26T00:28:31.589 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:31.589 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:31.589 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.589 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.589 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:31.589 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:31.589 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:31.589 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:31.589 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:31.589 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:31.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.599+0000 7f0d9edef640 10 monclient: shutdown 2024-04-26T00:28:31.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.599+0000 7f0d9edef640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:28:31.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.599+0000 7f0d9edef640 1 -- 172.21.15.37:0/177248441 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d98135090 msgr2=0x7f0d98135470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:31.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.599+0000 7f0d9edef640 1 --2- 172.21.15.37:0/177248441 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d98135090 0x7f0d98135470 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0d980a3620 tx=0x7f0d8801c380 comp rx=0 tx=0).stop 2024-04-26T00:28:31.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.599+0000 7f0d9edef640 1 -- 172.21.15.37:0/177248441 shutdown_connections 2024-04-26T00:28:31.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.599+0000 7f0d9edef640 1 --2- 172.21.15.37:0/177248441 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d98135090 0x7f0d98135470 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:31.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.599+0000 7f0d9edef640 1 -- 172.21.15.37:0/177248441 >> 172.21.15.37:0/177248441 conn(0x7f0d980a6250 msgr2=0x7f0d980a2610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:28:31.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.599+0000 7f0d9edef640 1 -- 172.21.15.37:0/177248441 shutdown_connections 2024-04-26T00:28:31.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.599+0000 7f0d9edef640 1 -- 172.21.15.37:0/177248441 wait complete. 2024-04-26T00:28:31.591 INFO:tasks.rados.rados.0.smithi045.stdout:1461: writing smithi04526579-44 from 1800933 to 2526451 tid 2 2024-04-26T00:28:31.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.599+0000 7f0d9edef640 10 monclient: build_initial_monmap 2024-04-26T00:28:31.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.599+0000 7f0d9edef640 10 monclient: monmap: 2024-04-26T00:28:31.594 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:28:31.594 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:28:31.595 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:28:31.595 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:28:31.595 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:28:31.595 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:28:31.595 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:28:31.595 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:28:31.595 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:28:31.595 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:31.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.599+0000 7f0d9edef640 1 Processor -- start 2024-04-26T00:28:31.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9edef640 1 -- start start 2024-04-26T00:28:31.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9edef640 10 monclient: init 2024-04-26T00:28:31.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9edef640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:31.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9edef640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:28:31.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9edef640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d9806ead0 0x7f0d981cf450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:31.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9edef640 10 monclient(hunting): picked mon.noname-b con 0x7f0d9806ead0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:28:31.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9edef640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d980a1bc0 0x7f0d9813d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:31.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9edef640 10 monclient(hunting): picked mon.noname-a con 0x7f0d980a1bc0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:28:31.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9edef640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d98135090 0x7f0d9813d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:31.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9edef640 10 monclient(hunting): picked mon.noname-c con 0x7f0d98135090 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:28:31.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9edef640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:31.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9edef640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f0d98140f70 con 0x7f0d980a1bc0 2024-04-26T00:28:31.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9edef640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:31.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9edef640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0d98140df0 con 0x7f0d9806ead0 2024-04-26T00:28:31.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9edef640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:31.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9edef640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0d981410f0 con 0x7f0d98135090 2024-04-26T00:28:31.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9edef640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:31.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9edef640 10 monclient(hunting): authenticate will time out at 1492.908685s 2024-04-26T00:28:31.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d97fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d980a1bc0 0x7f0d9813d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:31.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9d38c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d98135090 0x7f0d9813d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:31.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d97fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d980a1bc0 0x7f0d9813d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:49818/0 (socket says 172.21.15.37:49818) 2024-04-26T00:28:31.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d97fff640 1 -- 172.21.15.37:0/108620690 learned_addr learned my addr 172.21.15.37:0/108620690 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:28:31.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d97fff640 10 monclient(hunting): get_auth_request con 0x7f0d980a1bc0 auth_method 0 2024-04-26T00:28:31.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d97fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:31.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d97fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:31.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d97fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:31.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9d38c640 10 monclient(hunting): get_auth_request con 0x7f0d98135090 auth_method 0 2024-04-26T00:28:31.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9d38c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:31.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9d38c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:31.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9d38c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:31.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d97fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:31.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d97fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:31.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9cb8b640 1 --2- 172.21.15.37:0/108620690 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d9806ead0 0x7f0d981cf450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:31.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d97fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:31.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9cb8b640 10 monclient(hunting): get_auth_request con 0x7f0d9806ead0 auth_method 0 2024-04-26T00:28:31.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9cb8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:31.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9cb8b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:31.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9cb8b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:31.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9d38c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:31.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9d38c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:31.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9d38c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:31.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9cb8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:31.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9cb8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:31.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9cb8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:31.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9d38c640 10 monclient(hunting): handle_auth_done global_id 84314 payload 824 2024-04-26T00:28:31.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9d38c640 1 -- 172.21.15.37:0/108620690 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d9806ead0 msgr2=0x7f0d981cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:31.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9d38c640 1 --2- 172.21.15.37:0/108620690 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d9806ead0 0x7f0d981cf450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:31.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9d38c640 1 -- 172.21.15.37:0/108620690 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d980a1bc0 msgr2=0x7f0d9813d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:31.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9d38c640 1 --2- 172.21.15.37:0/108620690 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d980a1bc0 0x7f0d9813d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:31.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9d38c640 10 monclient: _finish_hunting 0 2024-04-26T00:28:31.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9d38c640 1 monclient: found mon.noname-c 2024-04-26T00:28:31.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9d38c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:31.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9d38c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:28:31.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9d38c640 1 -- 172.21.15.37:0/108620690 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d9813ba10 con 0x7f0d98135090 2024-04-26T00:28:31.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9d38c640 1 -- 172.21.15.37:0/108620690 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d98135090 msgr2=0x7f0d9813d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:31.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.603+0000 7f0d9d38c640 1 --2- 172.21.15.37:0/108620690 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d98135090 0x7f0d9813d6b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0d9000c780 tx=0x7f0d9000cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:31.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.607+0000 7f0d957fa640 1 -- 172.21.15.37:0/108620690 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d90016020 con 0x7f0d98135090 2024-04-26T00:28:31.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.607+0000 7f0d957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:31.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.607+0000 7f0d957fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:28:31.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.607+0000 7f0d957fa640 10 monclient: dump: 2024-04-26T00:28:31.599 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:31.599 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:31.600 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.600 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.600 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:31.600 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:31.600 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:31.600 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:31.600 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:31.600 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:31.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.607+0000 7f0d957fa640 10 monclient: _finish_auth 0 2024-04-26T00:28:31.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.607+0000 7f0d957fa640 10 monclient: _check_auth_tickets 2024-04-26T00:28:31.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.607+0000 7f0d957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:31.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.607+0000 7f0d957fa640 1 -- 172.21.15.37:0/108620690 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d9000abe0 con 0x7f0d98135090 2024-04-26T00:28:31.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.607+0000 7f0d957fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:28:31.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.607+0000 7f0d957fa640 1 -- 172.21.15.37:0/108620690 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d9000ce30 con 0x7f0d98135090 2024-04-26T00:28:31.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.607+0000 7f0d9edef640 5 monclient: authenticate success, global_id 84314 2024-04-26T00:28:31.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.607+0000 7f0d9edef640 10 monclient: _renew_subs 2024-04-26T00:28:31.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.607+0000 7f0d9edef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:28:31.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.607+0000 7f0d9edef640 1 -- 172.21.15.37:0/108620690 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d9806ef90 con 0x7f0d98135090 2024-04-26T00:28:31.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.607+0000 7f0d957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:31.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.607+0000 7f0d957fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:28:31.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.607+0000 7f0d96ffd640 4 set_mon_vals no callback set 2024-04-26T00:28:31.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.607+0000 7f0d957fa640 10 monclient: dump: 2024-04-26T00:28:31.601 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:31.601 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:31.601 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.601 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:31.601 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:31.601 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:31.601 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:31.601 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:31.602 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:31.602 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:31.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.607+0000 7f0d9edef640 10 monclient: _renew_subs 2024-04-26T00:28:31.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.607+0000 7f0d9edef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:28:31.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.607+0000 7f0d9edef640 1 -- 172.21.15.37:0/108620690 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d9813b170 con 0x7f0d98135090 2024-04-26T00:28:31.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.607+0000 7f0d957fa640 1 -- 172.21.15.37:0/108620690 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f0d90019020 con 0x7f0d98135090 2024-04-26T00:28:31.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.607+0000 7f0d9edef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:31.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.607+0000 7f0d9edef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:31.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.607+0000 7f0d9edef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:28:31.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.607+0000 7f0d9edef640 1 -- 172.21.15.37:0/108620690 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d9813dcb0 con 0x7f0d98135090 2024-04-26T00:28:31.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.607+0000 7f0d957fa640 1 --2- 172.21.15.37:0/108620690 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0d84035ed0 0x7f0d84038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:31.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.607+0000 7f0d957fa640 1 -- 172.21.15.37:0/108620690 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(337..337 src has 177..337) ==== 3947+0+0 (secure 0 0 0) 0x7f0d90048b90 con 0x7f0d98135090 2024-04-26T00:28:31.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.607+0000 7f0d9cb8b640 1 --2- 172.21.15.37:0/108620690 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0d84035ed0 0x7f0d84038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:31.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.607+0000 7f0d9cb8b640 10 monclient: get_auth_request con 0x7f0d84035ed0 auth_method 0 2024-04-26T00:28:31.603 INFO:tasks.rados.rados.0.smithi045.stdout:1461: writing smithi04526579-44 from 2944601 to 3176622 tid 3 2024-04-26T00:28:31.603 INFO:tasks.rados.rados.0.smithi045.stdout:1462: snap_remove snap 165 2024-04-26T00:28:31.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.611+0000 7f0d9cb8b640 1 --2- 172.21.15.37:0/108620690 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0d84035ed0 0x7f0d84038390 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0d880ab000 tx=0x7f0d880a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:31.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.611+0000 7f0d957fa640 1 -- 172.21.15.37:0/108620690 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f0d90020020 con 0x7f0d98135090 2024-04-26T00:28:31.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.611+0000 7f0d957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:31.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.611+0000 7f0d957fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:28:31.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.831+0000 7f0d9edef640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-26T00:28:31.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.831+0000 7f0d9edef640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-26T00:28:31.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.831+0000 7f0d9edef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:28:31.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.831+0000 7f0d9edef640 1 -- 172.21.15.37:0/108620690 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f0d98066f40 con 0x7f0d98135090 2024-04-26T00:28:31.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.835+0000 7f0d957fa640 1 -- 172.21.15.37:0/108620690 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f0d900478d0 con 0x7f0d98135090 2024-04-26T00:28:31.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.835+0000 7f0d957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-26T00:28:31.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.835+0000 7f0d957fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:28:31.823 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.835+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.823 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.835+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=18,config=2,health=12,kv=10,logm=58} crc {auth=2891081687,config=692619927,health=2925548024,kv=1034996044,logm=48444461}) 2024-04-26T00:28:31.823 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.835+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=17,config=2,health=11,kv=10,logm=60} crc {auth=974207460,config=692619927,health=2718910994,kv=1034996044,logm=3999829032}) 2024-04-26T00:28:31.823 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.835+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.823 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.835+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=18,config=2,health=12,kv=10,logm=58} crc {auth=2891081687,config=692619927,health=2925548024,kv=1034996044,logm=48444461}) 2024-04-26T00:28:31.823 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.835+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=16,config=1,health=10,kv=10,logm=63} crc {auth=922728460,config=3678701417,health=1149494872,kv=1034996044,logm=2947973618}) 2024-04-26T00:28:31.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.839+0000 7f0d9edef640 1 -- 172.21.15.37:0/108620690 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0d84035ed0 msgr2=0x7f0d84038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:31.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.839+0000 7f0d9edef640 1 --2- 172.21.15.37:0/108620690 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0d84035ed0 0x7f0d84038390 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0d880ab000 tx=0x7f0d880a6000 comp rx=0 tx=0).stop 2024-04-26T00:28:31.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.839+0000 7f0d9edef640 10 monclient: shutdown 2024-04-26T00:28:31.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.839+0000 7f0d9edef640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:28:31.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.839+0000 7f0d9edef640 1 -- 172.21.15.37:0/108620690 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d98135090 msgr2=0x7f0d9813d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:31.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.839+0000 7f0d9edef640 1 --2- 172.21.15.37:0/108620690 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d98135090 0x7f0d9813d6b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0d9000c780 tx=0x7f0d9000cc40 comp rx=0 tx=0).stop 2024-04-26T00:28:31.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.839+0000 7f0d9edef640 1 -- 172.21.15.37:0/108620690 shutdown_connections 2024-04-26T00:28:31.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.839+0000 7f0d9edef640 1 --2- 172.21.15.37:0/108620690 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0d84035ed0 0x7f0d84038390 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:31.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.839+0000 7f0d9edef640 1 --2- 172.21.15.37:0/108620690 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d98135090 0x7f0d9813d6b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:31.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.839+0000 7f0d9edef640 1 --2- 172.21.15.37:0/108620690 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d980a1bc0 0x7f0d9813d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:31.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.839+0000 7f0d9edef640 1 --2- 172.21.15.37:0/108620690 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0d9806ead0 0x7f0d981cf450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:31.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.839+0000 7f0d9edef640 1 -- 172.21.15.37:0/108620690 >> 172.21.15.37:0/108620690 conn(0x7f0d980a6250 msgr2=0x7f0d9813deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:28:31.826 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.839+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.826 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.839+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2038995863}) 2024-04-26T00:28:31.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.839+0000 7f0d9edef640 1 -- 172.21.15.37:0/108620690 shutdown_connections 2024-04-26T00:28:31.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:31.839+0000 7f0d9edef640 1 -- 172.21.15.37:0/108620690 wait complete. 2024-04-26T00:28:31.827 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.839+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=53512484}) 2024-04-26T00:28:31.827 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.839+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.827 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.839+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2038995863}) 2024-04-26T00:28:31.827 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.839+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1721636142}) 2024-04-26T00:28:31.828 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.839+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.828 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.839+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3664963493}) 2024-04-26T00:28:31.828 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.839+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3722653592}) 2024-04-26T00:28:31.828 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.839+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.828 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.839+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3664963493}) 2024-04-26T00:28:31.828 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.839+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3460922878}) 2024-04-26T00:28:31.829 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.843+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.830 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.843+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3909242039}) 2024-04-26T00:28:31.830 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.843+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1002839860}) 2024-04-26T00:28:31.830 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.843+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.830 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.843+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3909242039}) 2024-04-26T00:28:31.830 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.843+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2629523504}) 2024-04-26T00:28:31.832 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.843+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.832 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.843+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=656585925}) 2024-04-26T00:28:31.832 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.843+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1136248660}) 2024-04-26T00:28:31.832 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.843+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.832 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.843+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=656585925}) 2024-04-26T00:28:31.832 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.843+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4202399199}) 2024-04-26T00:28:31.833 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.847+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.834 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.847+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=859602890}) 2024-04-26T00:28:31.834 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.847+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1892621020}) 2024-04-26T00:28:31.834 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.847+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.834 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.847+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=859602890}) 2024-04-26T00:28:31.834 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.847+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=18206643}) 2024-04-26T00:28:31.836 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.851+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.836 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.851+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=150958289}) 2024-04-26T00:28:31.836 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.851+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2029687963}) 2024-04-26T00:28:31.836 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.851+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.836 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.851+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=150958289}) 2024-04-26T00:28:31.836 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.851+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1176194428}) 2024-04-26T00:28:31.838 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.851+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.838 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.851+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=589840658}) 2024-04-26T00:28:31.838 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.851+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=502401410}) 2024-04-26T00:28:31.838 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.851+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.838 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.851+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=589840658}) 2024-04-26T00:28:31.838 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.851+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3680797765}) 2024-04-26T00:28:31.840 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.855+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.840 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.855+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3821150156}) 2024-04-26T00:28:31.840 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.855+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1157881095}) 2024-04-26T00:28:31.840 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.855+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.840 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.855+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3821150156}) 2024-04-26T00:28:31.841 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.855+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1974747756}) 2024-04-26T00:28:31.843 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.855+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.843 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.855+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2236445058}) 2024-04-26T00:28:31.843 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.855+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3968216855}) 2024-04-26T00:28:31.843 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.855+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.843 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.855+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2236445058}) 2024-04-26T00:28:31.843 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.855+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4209374117}) 2024-04-26T00:28:31.844 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-26T00:28:31.845 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.859+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.845 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.859+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3147047431}) 2024-04-26T00:28:31.845 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.859+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1318077117}) 2024-04-26T00:28:31.845 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.859+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.845 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.859+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3147047431}) 2024-04-26T00:28:31.845 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.859+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2705124228}) 2024-04-26T00:28:31.848 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.859+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.848 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.859+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2706414890}) 2024-04-26T00:28:31.848 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.859+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1501585979}) 2024-04-26T00:28:31.848 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.859+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.848 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.859+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2706414890}) 2024-04-26T00:28:31.848 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.859+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2113855293}) 2024-04-26T00:28:31.850 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.863+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.850 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.863+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2100326460}) 2024-04-26T00:28:31.850 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.863+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=785529397}) 2024-04-26T00:28:31.850 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.863+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.850 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.863+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2100326460}) 2024-04-26T00:28:31.851 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.863+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3148374235}) 2024-04-26T00:28:31.852 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.867+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.852 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.867+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2637358144}) 2024-04-26T00:28:31.852 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.867+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1922911451}) 2024-04-26T00:28:31.852 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.867+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.852 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.867+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2637358144}) 2024-04-26T00:28:31.852 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.867+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=943433639}) 2024-04-26T00:28:31.854 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.867+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.855 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.867+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3712648238}) 2024-04-26T00:28:31.855 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.867+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2721628277}) 2024-04-26T00:28:31.855 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.867+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.855 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.867+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3712648238}) 2024-04-26T00:28:31.855 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.867+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2410010795}) 2024-04-26T00:28:31.857 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.871+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.857 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.871+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2725989248}) 2024-04-26T00:28:31.857 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.871+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=229955289}) 2024-04-26T00:28:31.857 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.871+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.857 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.871+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2725989248}) 2024-04-26T00:28:31.857 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.871+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2944361924}) 2024-04-26T00:28:31.859 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.871+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.859 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.871+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3131184854}) 2024-04-26T00:28:31.859 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.871+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2436112478}) 2024-04-26T00:28:31.859 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.871+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.859 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.871+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3131184854}) 2024-04-26T00:28:31.859 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.871+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3563334860}) 2024-04-26T00:28:31.861 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.875+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.861 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.875+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2035593589}) 2024-04-26T00:28:31.862 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.875+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=728331939}) 2024-04-26T00:28:31.862 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.875+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.862 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.875+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2035593589}) 2024-04-26T00:28:31.862 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.875+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1070882663}) 2024-04-26T00:28:31.864 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.875+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.864 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.875+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2485976348}) 2024-04-26T00:28:31.864 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.875+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2299640120}) 2024-04-26T00:28:31.864 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.875+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.864 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.879+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2485976348}) 2024-04-26T00:28:31.864 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.879+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3010649402}) 2024-04-26T00:28:31.866 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.879+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.866 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.879+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4170640661}) 2024-04-26T00:28:31.866 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.879+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3376540800}) 2024-04-26T00:28:31.866 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.879+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.866 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.879+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4170640661}) 2024-04-26T00:28:31.867 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.879+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1597727605}) 2024-04-26T00:28:31.868 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.883+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.869 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.883+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4053410046}) 2024-04-26T00:28:31.869 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.883+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2717320074}) 2024-04-26T00:28:31.869 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.883+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.869 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.883+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4053410046}) 2024-04-26T00:28:31.869 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.883+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=42579774}) 2024-04-26T00:28:31.871 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.883+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.871 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.883+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2874467047}) 2024-04-26T00:28:31.871 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.883+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=942779091}) 2024-04-26T00:28:31.871 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.883+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.871 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.883+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2874467047}) 2024-04-26T00:28:31.872 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.883+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2400865028}) 2024-04-26T00:28:31.873 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.887+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.874 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.887+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4105222085}) 2024-04-26T00:28:31.874 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.887+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1946785977}) 2024-04-26T00:28:31.874 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.887+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.874 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.887+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4105222085}) 2024-04-26T00:28:31.874 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.887+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2767088883}) 2024-04-26T00:28:31.876 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.891+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.877 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.891+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=68,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11} crc {logm=2429815997,mdsmap=2840952586,mgr=1717321805,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=1864888201,mon_config_key=4061063117}) 2024-04-26T00:28:31.877 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.891+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=66,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=10} crc {logm=3808143443,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339439,mgr_metadata=3686404438,mgrstat=2519452505,mon_config_key=2198044097}) 2024-04-26T00:28:31.877 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.891+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.877 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.891+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=68,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11} crc {logm=2429815997,mdsmap=2840952586,mgr=1717321805,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=1864888201,mon_config_key=4061063117}) 2024-04-26T00:28:31.877 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.891+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=66,mdsmap=3,mgr=5,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=10,monmap=3} crc {logm=377806766,mdsmap=2840952586,mgr=3906413757,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=3991024243,mon_config_key=1484518886,monmap=3876866194}) 2024-04-26T00:28:31.879 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.891+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.879 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.891+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {monmap=3,osd_metadata=6,osd_snap=55,osdmap=36} crc {monmap=3876866194,osd_metadata=3306121363,osd_snap=2536205502,osdmap=746647561}) 2024-04-26T00:28:31.879 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.891+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=55,osdmap=35} crc {monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=34098387,osd_snap=150644056,osdmap=1974565236}) 2024-04-26T00:28:31.879 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.891+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.879 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.891+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {monmap=3,osd_metadata=6,osd_snap=55,osdmap=36} crc {monmap=3876866194,osd_metadata=3306121363,osd_snap=2536205502,osdmap=746647561}) 2024-04-26T00:28:31.879 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.891+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=56,osdmap=35} crc {monmap=3876866194,osd_metadata=1980645665,osd_pg_creating=34098387,osd_snap=4072323019,osdmap=3743729323}) 2024-04-26T00:28:31.881 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.895+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.881 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.895+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1257876482}) 2024-04-26T00:28:31.882 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.895+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3165293515}) 2024-04-26T00:28:31.882 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.895+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.882 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.895+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1257876482}) 2024-04-26T00:28:31.882 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.895+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1260016122}) 2024-04-26T00:28:31.886 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.899+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.886 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.899+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4051883428}) 2024-04-26T00:28:31.886 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.899+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=587441699}) 2024-04-26T00:28:31.886 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.899+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:28:31.886 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.899+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4051883428}) 2024-04-26T00:28:31.886 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:28:31.899+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=868083000}) 2024-04-26T00:28:32.844 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:28:32.845 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:28:32.903 INFO:tasks.rados.rados.0.smithi045.stdout:1462: done (1 left) 2024-04-26T00:28:32.903 INFO:tasks.rados.rados.0.smithi045.stdout:1463: copy_from oid 24 from oid 19 current snap is 168 2024-04-26T00:28:32.903 INFO:tasks.rados.rados.0.smithi045.stdout:1461: finishing write tid 1 to smithi04526579-44 2024-04-26T00:28:32.903 INFO:tasks.rados.rados.0.smithi045.stdout:1461: finishing write tid 2 to smithi04526579-44 2024-04-26T00:28:32.903 INFO:tasks.rados.rados.0.smithi045.stdout:1461: finishing write tid 3 to smithi04526579-44 2024-04-26T00:28:32.903 INFO:tasks.rados.rados.0.smithi045.stdout:1461: finishing write tid 4 to smithi04526579-44 2024-04-26T00:28:32.903 INFO:tasks.rados.rados.0.smithi045.stdout:1461: finishing write tid 5 to smithi04526579-44 2024-04-26T00:28:32.903 INFO:tasks.rados.rados.0.smithi045.stdout:1461: oid 44 updating version 0 to 868 2024-04-26T00:28:32.904 INFO:tasks.rados.rados.0.smithi045.stdout:1461: oid 44 version 868 is already newer than 865 2024-04-26T00:28:32.904 INFO:tasks.rados.rados.0.smithi045.stdout:1461: oid 44 version 868 is already newer than 866 2024-04-26T00:28:32.904 INFO:tasks.rados.rados.0.smithi045.stdout:1461: oid 44 version 868 is already newer than 867 2024-04-26T00:28:32.904 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 868 (ObjNum 546 snap 168 seq_num 546) dirty exists 2024-04-26T00:28:32.904 INFO:tasks.rados.rados.0.smithi045.stdout:1461: left oid 44 (ObjNum 546 snap 168 seq_num 546) 2024-04-26T00:28:32.904 INFO:tasks.rados.rados.0.smithi045.stdout:1461: done (1 left) 2024-04-26T00:28:32.904 INFO:tasks.rados.rados.0.smithi045.stdout:1464: delete oid 50 current snap is 168 2024-04-26T00:28:32.922 INFO:tasks.rados.rados.0.smithi045.stdout:1464: done (1 left) 2024-04-26T00:28:32.923 INFO:tasks.rados.rados.0.smithi045.stdout:1465: rollback oid 41 current snap is 168 2024-04-26T00:28:32.923 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 41 to 127 2024-04-26T00:28:32.923 INFO:tasks.rados.rados.0.smithi045.stdout:1466: write oid 50 current snap is 168 2024-04-26T00:28:32.923 INFO:tasks.rados.rados.0.smithi045.stdout:1466: seq_num 548 ranges {410236=413868,1234250=729187,2715800=423063} 2024-04-26T00:28:32.935 INFO:tasks.rados.rados.0.smithi045.stdout:1466: writing smithi04526579-50 from 410236 to 824104 tid 1 2024-04-26T00:28:32.943 INFO:tasks.rados.rados.0.smithi045.stdout:1466: writing smithi04526579-50 from 1234250 to 1963437 tid 2 2024-04-26T00:28:32.952 INFO:tasks.rados.rados.0.smithi045.stdout:1466: writing smithi04526579-50 from 2715800 to 3138863 tid 3 2024-04-26T00:28:32.952 INFO:tasks.rados.rados.0.smithi045.stdout:1467: rollback oid 15 current snap is 168 2024-04-26T00:28:32.952 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 15 to 127 2024-04-26T00:28:32.952 INFO:tasks.rados.rados.0.smithi045.stdout:1465: finishing rollback tid 0 to smithi04526579-41 2024-04-26T00:28:32.952 INFO:tasks.rados.rados.0.smithi045.stdout:1463: finishing copy_from racing read to smithi04526579-24 2024-04-26T00:28:32.952 INFO:tasks.rados.rados.0.smithi045.stdout:1465: finishing rollback tid 1 to smithi04526579-41 2024-04-26T00:28:32.953 INFO:tasks.rados.rados.0.smithi045.stdout:1465: finishing rollback tid 2 to smithi04526579-41 2024-04-26T00:28:32.953 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 1038 (ObjNum 394 snap 120 seq_num 394) dirty exists 2024-04-26T00:28:32.953 INFO:tasks.rados.rados.0.smithi045.stdout:1463: finishing copy_from to smithi04526579-24 2024-04-26T00:28:32.953 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 852 (ObjNum 534 snap 165 seq_num 534) dirty exists 2024-04-26T00:28:32.953 INFO:tasks.rados.rados.0.smithi045.stdout:1466: finishing write tid 1 to smithi04526579-50 2024-04-26T00:28:32.953 INFO:tasks.rados.rados.0.smithi045.stdout:1466: finishing write tid 2 to smithi04526579-50 2024-04-26T00:28:32.953 INFO:tasks.rados.rados.0.smithi045.stdout:1463: done (3 left) 2024-04-26T00:28:32.953 INFO:tasks.rados.rados.0.smithi045.stdout:1465: done (2 left) 2024-04-26T00:28:32.953 INFO:tasks.rados.rados.0.smithi045.stdout:1468: delete oid 13 current snap is 168 2024-04-26T00:28:32.957 INFO:tasks.rados.rados.0.smithi045.stdout:1466: finishing write tid 3 to smithi04526579-50 2024-04-26T00:28:32.958 INFO:tasks.rados.rados.0.smithi045.stdout:1467: finishing rollback tid 1 to smithi04526579-15 2024-04-26T00:28:32.958 INFO:tasks.rados.rados.0.smithi045.stdout:1468: done (2 left) 2024-04-26T00:28:32.958 INFO:tasks.rados.rados.0.smithi045.stdout:1469: read oid 32 snap -1 2024-04-26T00:28:32.958 INFO:tasks.rados.rados.0.smithi045.stdout:1469: expect (ObjNum 526 snap 163 seq_num 526) 2024-04-26T00:28:32.958 INFO:tasks.rados.rados.0.smithi045.stdout:1470: delete oid 36 current snap is 168 2024-04-26T00:28:32.960 INFO:tasks.rados.rados.0.smithi045.stdout:1467: finishing rollback tid 2 to smithi04526579-15 2024-04-26T00:28:32.960 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 1040 (ObjNum 376 snap 115 seq_num 376) dirty exists 2024-04-26T00:28:32.961 INFO:tasks.rados.rados.0.smithi045.stdout:1466: finishing write tid 4 to smithi04526579-50 2024-04-26T00:28:32.961 INFO:tasks.rados.rados.0.smithi045.stdout:1466: finishing write tid 5 to smithi04526579-50 2024-04-26T00:28:32.961 INFO:tasks.rados.rados.0.smithi045.stdout:1466: oid 50 updating version 0 to 864 2024-04-26T00:28:32.961 INFO:tasks.rados.rados.0.smithi045.stdout:1466: oid 50 updating version 864 to 865 2024-04-26T00:28:32.961 INFO:tasks.rados.rados.0.smithi045.stdout:1466: oid 50 version 865 is already newer than 863 2024-04-26T00:28:32.961 INFO:tasks.rados.rados.0.smithi045.stdout:1466: oid 50 version 865 is already newer than 862 2024-04-26T00:28:32.961 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 865 (ObjNum 548 snap 168 seq_num 548) dirty exists 2024-04-26T00:28:32.961 INFO:tasks.rados.rados.0.smithi045.stdout:1466: left oid 50 (ObjNum 548 snap 168 seq_num 548) 2024-04-26T00:28:32.963 INFO:tasks.rados.rados.0.smithi045.stdout:1466: done (3 left) 2024-04-26T00:28:32.963 INFO:tasks.rados.rados.0.smithi045.stdout:1467: done (2 left) 2024-04-26T00:28:32.963 INFO:tasks.rados.rados.0.smithi045.stdout:1470: done (1 left) 2024-04-26T00:28:32.963 INFO:tasks.rados.rados.0.smithi045.stdout:1471: write (excl) oid 43 current snap is 168 2024-04-26T00:28:32.963 INFO:tasks.rados.rados.0.smithi045.stdout:1471: seq_num 551 ranges {460657=639615,1693343=476514,2681716=578941} 2024-04-26T00:28:32.973 INFO:tasks.rados.rados.0.smithi045.stdout:1471: writing smithi04526579-43 from 460657 to 1100272 tid 1 2024-04-26T00:28:32.982 INFO:tasks.rados.rados.0.smithi045.stdout:1471: writing smithi04526579-43 from 1693343 to 2169857 tid 2 2024-04-26T00:28:32.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.003+0000 7f97b48a1640 10 monclient: get_monmap_and_config 2024-04-26T00:28:32.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.003+0000 7f97b48a1640 10 monclient: build_initial_monmap 2024-04-26T00:28:32.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.003+0000 7f97b48a1640 10 monclient: monmap: 2024-04-26T00:28:32.988 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:28:32.988 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:32.988 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:28:33.004177+0000 2024-04-26T00:28:32.988 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:28:33.004177+0000 2024-04-26T00:28:32.989 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:28:32.989 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:32.989 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:28:32.989 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:28:32.989 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:28:32.989 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:28:32.989 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:28:32.989 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:32.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.003+0000 7f97b48a1640 1 Processor -- start 2024-04-26T00:28:32.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.003+0000 7f97b48a1640 1 -- start start 2024-04-26T00:28:32.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.003+0000 7f97b48a1640 10 monclient: init 2024-04-26T00:28:32.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.003+0000 7f97b48a1640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:32.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.003+0000 7f97b48a1640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-04-26T00:28:32.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.003+0000 7f97b48a1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f97b00a0260 addr v1:172.21.15.45:6789/0 2024-04-26T00:28:32.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.003+0000 7f97b48a1640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f97b00a0d50 0x7f97b0072e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:32.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.003+0000 7f97b48a1640 10 monclient(hunting): picked mon.noname-a con 0x7f97b00a0d50 addr v2:172.21.15.37:3300/0 2024-04-26T00:28:32.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.003+0000 7f97b48a1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f97b01389e0 addr v1:172.21.15.37:6789/0 2024-04-26T00:28:32.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.003+0000 7f97b48a1640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97b00a12e0 con 0x7f97b01389e0 2024-04-26T00:28:32.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.003+0000 7f97b48a1640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97b006d050 con 0x7f97b00a0260 2024-04-26T00:28:32.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.003+0000 7f97b48a1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:32.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.003+0000 7f97b48a1640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f97b009dcb0 con 0x7f97b00a0d50 2024-04-26T00:28:32.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.003+0000 7f97b48a1640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:32.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.003+0000 7f97b48a1640 10 monclient(hunting): authenticate will time out at 1494.310004s 2024-04-26T00:28:32.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.003+0000 7f97a7fff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f97b00a0d50 0x7f97b0072e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:32.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.003+0000 7f97a7fff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f97b00a0d50 0x7f97b0072e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:49834/0 (socket says 172.21.15.37:49834) 2024-04-26T00:28:32.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.003+0000 7f97a7fff640 1 -- 172.21.15.37:0/2150440617 learned_addr learned my addr 172.21.15.37:0/2150440617 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:28:32.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.003+0000 7f97a7fff640 10 monclient(hunting): get_auth_request con 0x7f97b00a0d50 auth_method 0 2024-04-26T00:28:32.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.003+0000 7f97a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:32.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.003+0000 7f97a7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:32.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.003+0000 7f97a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:32.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.003+0000 7f97aed9d640 1 --1- 172.21.15.37:0/2150440617 >> v1:172.21.15.37:6789/0 conn(0x7f97b01389e0 0x7f97b013add0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:34702/0 (socket says 172.21.15.37:34702) 2024-04-26T00:28:32.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.003+0000 7f97a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:32.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.003+0000 7f97a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:32.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.003+0000 7f97a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:32.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.003+0000 7f97a7fff640 10 monclient(hunting): handle_auth_done global_id 94245 payload 274 2024-04-26T00:28:32.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.003+0000 7f97a7fff640 1 -- 172.21.15.37:0/2150440617 >> v1:172.21.15.45:6789/0 conn(0x7f97b00a0260 legacy=0x7f97b00a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:28:32.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.003+0000 7f97a7fff640 1 -- 172.21.15.37:0/2150440617 >> v1:172.21.15.37:6789/0 conn(0x7f97b01389e0 legacy=0x7f97b013add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:28:32.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.003+0000 7f97a7fff640 10 monclient: _finish_hunting 0 2024-04-26T00:28:32.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.003+0000 7f97a7fff640 1 monclient: found mon.noname-a 2024-04-26T00:28:32.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.003+0000 7f97a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:32.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.003+0000 7f97a7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:28:32.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.003+0000 7f97a7fff640 1 -- 172.21.15.37:0/2150440617 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97b00a16f0 con 0x7f97b00a0d50 2024-04-26T00:28:32.994 INFO:tasks.rados.rados.0.smithi045.stdout:1471: writing smithi04526579-43 from 2681716 to 3260657 tid 3 2024-04-26T00:28:32.994 INFO:tasks.rados.rados.0.smithi045.stdout:1472: delete oid 47 current snap is 168 2024-04-26T00:28:32.994 INFO:tasks.rados.rados.0.smithi045.stdout:1469: expect (ObjNum 526 snap 163 seq_num 526) 2024-04-26T00:28:33.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.007+0000 7f97a7fff640 1 -- 172.21.15.37:0/2150440617 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f97b00a0d50 msgr2=0x7f97b0072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:33.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.007+0000 7f97a7fff640 1 --2- 172.21.15.37:0/2150440617 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f97b00a0d50 0x7f97b0072e10 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f97a0009870 tx=0x7f97a001d540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb771ed30538a5a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:33.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.007+0000 7f97add9b640 1 -- 172.21.15.37:0/2150440617 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a00b1070 con 0x7f97b00a0d50 2024-04-26T00:28:33.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.007+0000 7f97add9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:33.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.007+0000 7f97add9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:28:33.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.007+0000 7f97add9b640 10 monclient: dump: 2024-04-26T00:28:33.003 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:33.003 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:33.003 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.003 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.003 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:33.003 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:33.003 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:33.003 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:33.003 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:33.003 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:33.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.007+0000 7f97add9b640 10 monclient: _finish_auth 0 2024-04-26T00:28:33.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.007+0000 7f97add9b640 10 monclient: _check_auth_tickets 2024-04-26T00:28:33.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.007+0000 7f97add9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:33.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.007+0000 7f97add9b640 1 -- 172.21.15.37:0/2150440617 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97a00ac600 con 0x7f97b00a0d50 2024-04-26T00:28:33.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.007+0000 7f97add9b640 10 monclient: handle_config config(0 keys) 2024-04-26T00:28:33.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.007+0000 7f97add9b640 1 -- 172.21.15.37:0/2150440617 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a00ac9d0 con 0x7f97b00a0d50 2024-04-26T00:28:33.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.007+0000 7f97b48a1640 5 monclient: authenticate success, global_id 94245 2024-04-26T00:28:33.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.007+0000 7f97b48a1640 10 monclient: get_monmap_and_config success 2024-04-26T00:28:33.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.007+0000 7f97b48a1640 4 set_mon_vals no callback set 2024-04-26T00:28:33.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.007+0000 7f97add9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:33.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.007+0000 7f97add9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:28:33.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.007+0000 7f97add9b640 10 monclient: dump: 2024-04-26T00:28:33.004 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:33.004 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:33.004 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.004 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.005 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:33.005 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:33.005 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:33.005 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:33.005 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:33.005 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:33.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.007+0000 7f97b48a1640 10 monclient: shutdown 2024-04-26T00:28:33.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.007+0000 7f97b48a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:28:33.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.007+0000 7f97b48a1640 1 -- 172.21.15.37:0/2150440617 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f97b00a0d50 msgr2=0x7f97b0072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:33.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.007+0000 7f97b48a1640 1 --2- 172.21.15.37:0/2150440617 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f97b00a0d50 0x7f97b0072e10 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f97a0009870 tx=0x7f97a001d540 comp rx=0 tx=0).stop 2024-04-26T00:28:33.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.007+0000 7f97b48a1640 1 -- 172.21.15.37:0/2150440617 shutdown_connections 2024-04-26T00:28:33.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.007+0000 7f97b48a1640 1 --2- 172.21.15.37:0/2150440617 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f97b00a0d50 0x7f97b0072e10 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:33.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.007+0000 7f97b48a1640 1 -- 172.21.15.37:0/2150440617 >> 172.21.15.37:0/2150440617 conn(0x7f97b00a1ad0 msgr2=0x7f97b00a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:28:33.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.007+0000 7f97b48a1640 1 -- 172.21.15.37:0/2150440617 shutdown_connections 2024-04-26T00:28:33.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.007+0000 7f97b48a1640 1 -- 172.21.15.37:0/2150440617 wait complete. 2024-04-26T00:28:33.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97b48a1640 10 monclient: build_initial_monmap 2024-04-26T00:28:33.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97b48a1640 10 monclient: monmap: 2024-04-26T00:28:33.006 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:28:33.006 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:28:33.006 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:28:33.006 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:28:33.006 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:28:33.006 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:28:33.006 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:28:33.006 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:28:33.006 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:28:33.006 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:33.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97b48a1640 1 Processor -- start 2024-04-26T00:28:33.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97b48a1640 1 -- start start 2024-04-26T00:28:33.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97b48a1640 10 monclient: init 2024-04-26T00:28:33.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97b48a1640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:33.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97b48a1640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:28:33.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97b48a1640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97b00a0260 0x7f97b01377c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:33.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97b48a1640 10 monclient(hunting): picked mon.noname-c con 0x7f97b00a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:28:33.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97b48a1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97b00a0d50 0x7f97b011d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:33.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97b48a1640 10 monclient(hunting): picked mon.noname-b con 0x7f97b00a0d50 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:28:33.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97b48a1640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f97b01389e0 0x7f97b011d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:33.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97b48a1640 10 monclient(hunting): picked mon.noname-a con 0x7f97b01389e0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:28:33.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97b48a1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:33.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97b48a1640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f97b0140aa0 con 0x7f97b01389e0 2024-04-26T00:28:33.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97b48a1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:33.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97b48a1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f97b0140920 con 0x7f97b00a0d50 2024-04-26T00:28:33.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97b48a1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:33.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97b48a1640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f97b0140c20 con 0x7f97b00a0260 2024-04-26T00:28:33.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97b48a1640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:33.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97b48a1640 10 monclient(hunting): authenticate will time out at 1494.317185s 2024-04-26T00:28:33.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97aed9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f97b01389e0 0x7f97b011d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:33.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97ae59c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97b00a0260 0x7f97b01377c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:33.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97aed9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f97b01389e0 0x7f97b011d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:49846/0 (socket says 172.21.15.37:49846) 2024-04-26T00:28:33.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97aed9d640 1 -- 172.21.15.37:0/2105250794 learned_addr learned my addr 172.21.15.37:0/2105250794 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:28:33.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97aed9d640 10 monclient(hunting): get_auth_request con 0x7f97b01389e0 auth_method 0 2024-04-26T00:28:33.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97aed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:33.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97aed9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:33.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97aed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:33.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97ae59c640 10 monclient(hunting): get_auth_request con 0x7f97b00a0260 auth_method 0 2024-04-26T00:28:33.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97ae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:33.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97ae59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:33.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97ae59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:33.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97aed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:33.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97aed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:33.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97aed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:33.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97ae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:33.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97ae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:33.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97ae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:33.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97a7fff640 1 --2- 172.21.15.37:0/2105250794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97b00a0d50 0x7f97b011d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:33.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97a7fff640 10 monclient(hunting): get_auth_request con 0x7f97b00a0d50 auth_method 0 2024-04-26T00:28:33.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:33.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97a7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:33.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:33.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:33.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:33.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:33.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97ae59c640 10 monclient(hunting): handle_auth_done global_id 84317 payload 824 2024-04-26T00:28:33.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97ae59c640 1 -- 172.21.15.37:0/2105250794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97b00a0d50 msgr2=0x7f97b011d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:33.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97ae59c640 1 --2- 172.21.15.37:0/2105250794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97b00a0d50 0x7f97b011d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:33.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97ae59c640 1 -- 172.21.15.37:0/2105250794 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f97b01389e0 msgr2=0x7f97b011d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:33.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97ae59c640 1 --2- 172.21.15.37:0/2105250794 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f97b01389e0 0x7f97b011d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:33.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97ae59c640 10 monclient: _finish_hunting 0 2024-04-26T00:28:33.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97ae59c640 1 monclient: found mon.noname-c 2024-04-26T00:28:33.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97ae59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:33.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97ae59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:28:33.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97ae59c640 1 -- 172.21.15.37:0/2105250794 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97b009ca50 con 0x7f97b00a0260 2024-04-26T00:28:33.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.011+0000 7f97aed9d640 1 --2- 172.21.15.37:0/2105250794 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f97b01389e0 0x7f97b011d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:28:33.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.015+0000 7f97ae59c640 1 -- 172.21.15.37:0/2105250794 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97b00a0260 msgr2=0x7f97b01377c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:33.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.015+0000 7f97ae59c640 1 --2- 172.21.15.37:0/2105250794 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97b00a0260 0x7f97b01377c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f979800c410 tx=0x7f979800c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:33.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.015+0000 7f97a6ffd640 1 -- 172.21.15.37:0/2105250794 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9798017070 con 0x7f97b00a0260 2024-04-26T00:28:33.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.015+0000 7f97a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:33.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.015+0000 7f97a6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:28:33.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.015+0000 7f97a6ffd640 10 monclient: dump: 2024-04-26T00:28:33.011 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:33.011 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:33.011 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.011 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.011 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:33.011 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:33.011 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:33.011 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:33.011 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:33.011 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:33.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.015+0000 7f97a6ffd640 10 monclient: _finish_auth 0 2024-04-26T00:28:33.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.015+0000 7f97a6ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:28:33.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.015+0000 7f97a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:33.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.015+0000 7f97a6ffd640 1 -- 172.21.15.37:0/2105250794 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97980069e0 con 0x7f97b00a0260 2024-04-26T00:28:33.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.015+0000 7f97b48a1640 5 monclient: authenticate success, global_id 84317 2024-04-26T00:28:33.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.015+0000 7f97b48a1640 10 monclient: _renew_subs 2024-04-26T00:28:33.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.015+0000 7f97b48a1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:28:33.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.015+0000 7f97b48a1640 1 -- 172.21.15.37:0/2105250794 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97b0120e60 con 0x7f97b00a0260 2024-04-26T00:28:33.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.015+0000 7f97a6ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:28:33.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.015+0000 7f97a6ffd640 1 -- 172.21.15.37:0/2105250794 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9798006dd0 con 0x7f97b00a0260 2024-04-26T00:28:33.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.015+0000 7f97a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:33.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.015+0000 7f97a6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:28:33.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.015+0000 7f97a6ffd640 10 monclient: dump: 2024-04-26T00:28:33.012 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:33.013 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:33.013 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.013 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.013 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:33.013 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:33.013 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:33.013 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:33.013 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:33.013 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:33.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.015+0000 7f97ad59a640 4 set_mon_vals no callback set 2024-04-26T00:28:33.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.015+0000 7f97b48a1640 10 monclient: _renew_subs 2024-04-26T00:28:33.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.015+0000 7f97b48a1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:28:33.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.015+0000 7f97b48a1640 1 -- 172.21.15.37:0/2105250794 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f97b01cdae0 con 0x7f97b00a0260 2024-04-26T00:28:33.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.015+0000 7f97b48a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:33.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.015+0000 7f97b48a1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:33.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.015+0000 7f97b48a1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:28:33.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.015+0000 7f97b48a1640 1 -- 172.21.15.37:0/2105250794 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97b0066f40 con 0x7f97b00a0260 2024-04-26T00:28:33.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.015+0000 7f97a6ffd640 1 -- 172.21.15.37:0/2105250794 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f979801a070 con 0x7f97b00a0260 2024-04-26T00:28:33.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.015+0000 7f97a6ffd640 1 --2- 172.21.15.37:0/2105250794 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9790035e80 0x7f9790038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:33.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.015+0000 7f97a6ffd640 1 -- 172.21.15.37:0/2105250794 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(338..338 src has 177..338) ==== 3975+0+0 (secure 0 0 0) 0x7f9798047e90 con 0x7f97b00a0260 2024-04-26T00:28:33.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.015+0000 7f97a7fff640 1 --2- 172.21.15.37:0/2105250794 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9790035e80 0x7f9790038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:33.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.015+0000 7f97a7fff640 10 monclient: get_auth_request con 0x7f9790035e80 auth_method 0 2024-04-26T00:28:33.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.019+0000 7f97a7fff640 1 --2- 172.21.15.37:0/2105250794 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9790035e80 0x7f9790038340 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f97b006bce0 tx=0x7f97a0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:33.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.019+0000 7f97a6ffd640 1 -- 172.21.15.37:0/2105250794 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f97b009ca50 con 0x7f97b00a0260 2024-04-26T00:28:33.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.019+0000 7f97a6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:33.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.019+0000 7f97a6ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:28:33.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.243+0000 7f97b48a1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:28:33.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.243+0000 7f97b48a1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:28:33.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.243+0000 7f97b48a1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:28:33.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.243+0000 7f97b48a1640 1 -- 172.21.15.37:0/2105250794 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f97b013bac0 con 0x7f97b00a0260 2024-04-26T00:28:33.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.243+0000 7f97a6ffd640 1 -- 172.21.15.37:0/2105250794 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f979801ac60 con 0x7f97b00a0260 2024-04-26T00:28:33.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.243+0000 7f97a6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:28:33.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.243+0000 7f97a6ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:28:33.231 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":64,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:28:33.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.247+0000 7f97b48a1640 1 -- 172.21.15.37:0/2105250794 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9790035e80 msgr2=0x7f9790038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:33.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.247+0000 7f97b48a1640 1 --2- 172.21.15.37:0/2105250794 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9790035e80 0x7f9790038340 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f97b006bce0 tx=0x7f97a0008040 comp rx=0 tx=0).stop 2024-04-26T00:28:33.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.247+0000 7f97b48a1640 10 monclient: shutdown 2024-04-26T00:28:33.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.247+0000 7f97b48a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:28:33.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.247+0000 7f97b48a1640 1 -- 172.21.15.37:0/2105250794 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97b00a0260 msgr2=0x7f97b01377c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:33.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.247+0000 7f97b48a1640 1 --2- 172.21.15.37:0/2105250794 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97b00a0260 0x7f97b01377c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f979800c410 tx=0x7f979800c8d0 comp rx=0 tx=0).stop 2024-04-26T00:28:33.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.247+0000 7f97b48a1640 1 -- 172.21.15.37:0/2105250794 shutdown_connections 2024-04-26T00:28:33.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.247+0000 7f97b48a1640 1 --2- 172.21.15.37:0/2105250794 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9790035e80 0x7f9790038340 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:33.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.247+0000 7f97b48a1640 1 --2- 172.21.15.37:0/2105250794 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f97b01389e0 0x7f97b011d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:33.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.247+0000 7f97b48a1640 1 --2- 172.21.15.37:0/2105250794 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97b00a0d50 0x7f97b011d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:33.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.247+0000 7f97b48a1640 1 --2- 172.21.15.37:0/2105250794 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97b00a0260 0x7f97b01377c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:33.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.247+0000 7f97b48a1640 1 -- 172.21.15.37:0/2105250794 >> 172.21.15.37:0/2105250794 conn(0x7f97b00a1ad0 msgr2=0x7f97b013bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:28:33.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.247+0000 7f97b48a1640 1 -- 172.21.15.37:0/2105250794 shutdown_connections 2024-04-26T00:28:33.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.247+0000 7f97b48a1640 1 -- 172.21.15.37:0/2105250794 wait complete. 2024-04-26T00:28:33.248 INFO:tasks.rados.rados.0.smithi045.stdout:1471: finishing write tid 1 to smithi04526579-43 2024-04-26T00:28:33.249 INFO:tasks.rados.rados.0.smithi045.stdout:1471: finishing write tid 2 to smithi04526579-43 2024-04-26T00:28:33.249 INFO:tasks.rados.rados.0.smithi045.stdout:1471: finishing write tid 3 to smithi04526579-43 2024-04-26T00:28:33.249 INFO:tasks.rados.rados.0.smithi045.stdout:1471: finishing write tid 4 to smithi04526579-43 2024-04-26T00:28:33.249 INFO:tasks.rados.rados.0.smithi045.stdout:1471: finishing write tid 5 to smithi04526579-43 2024-04-26T00:28:33.249 INFO:tasks.rados.rados.0.smithi045.stdout:1471: oid 43 updating version 0 to 868 2024-04-26T00:28:33.249 INFO:tasks.rados.rados.0.smithi045.stdout:1471: oid 43 updating version 868 to 869 2024-04-26T00:28:33.249 INFO:tasks.rados.rados.0.smithi045.stdout:1471: oid 43 version 869 is already newer than 866 2024-04-26T00:28:33.249 INFO:tasks.rados.rados.0.smithi045.stdout:1471: oid 43 version 869 is already newer than 867 2024-04-26T00:28:33.250 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 869 (ObjNum 551 snap 168 seq_num 551) dirty exists 2024-04-26T00:28:33.250 INFO:tasks.rados.rados.0.smithi045.stdout:1471: left oid 43 (ObjNum 551 snap 168 seq_num 551) 2024-04-26T00:28:33.250 INFO:tasks.rados.rados.0.smithi045.stdout:1469: done (2 left) 2024-04-26T00:28:33.250 INFO:tasks.rados.rados.0.smithi045.stdout:1471: done (1 left) 2024-04-26T00:28:33.250 INFO:tasks.rados.rados.0.smithi045.stdout:1472: done (0 left) 2024-04-26T00:28:33.250 INFO:tasks.rados.rados.0.smithi045.stdout:1473: write (excl) oid 35 current snap is 168 2024-04-26T00:28:33.250 INFO:tasks.rados.rados.0.smithi045.stdout:1473: seq_num 553 ranges {494271=790113,1693954=480183,2620209=512129} 2024-04-26T00:28:33.251 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:28:33.251 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-26T00:28:33.251 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:28:33.251 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:28:33.260 INFO:tasks.rados.rados.0.smithi045.stdout:1473: writing smithi04526579-35 from 494271 to 1284384 tid 1 2024-04-26T00:28:33.268 INFO:tasks.rados.rados.0.smithi045.stdout:1473: writing smithi04526579-35 from 1693954 to 2174137 tid 2 2024-04-26T00:28:33.277 INFO:tasks.rados.rados.0.smithi045.stdout:1473: writing smithi04526579-35 from 2620209 to 3132338 tid 3 2024-04-26T00:28:33.277 INFO:tasks.rados.rados.0.smithi045.stdout:1474: write (excl) oid 25 current snap is 168 2024-04-26T00:28:33.278 INFO:tasks.rados.rados.0.smithi045.stdout:1474: seq_num 554 ranges {511078=465362,1752436=740194,3163985=147093} 2024-04-26T00:28:33.286 INFO:tasks.rados.rados.0.smithi045.stdout:1474: writing smithi04526579-25 from 511078 to 976440 tid 1 2024-04-26T00:28:33.301 INFO:tasks.rados.rados.0.smithi045.stdout:1474: writing smithi04526579-25 from 1752436 to 2492630 tid 2 2024-04-26T00:28:33.310 INFO:tasks.rados.rados.0.smithi045.stdout:1474: writing smithi04526579-25 from 3163985 to 3311078 tid 3 2024-04-26T00:28:33.310 INFO:tasks.rados.rados.0.smithi045.stdout:1475: copy_from oid 45 from oid 44 current snap is 168 2024-04-26T00:28:33.310 INFO:tasks.rados.rados.0.smithi045.stdout:1473: finishing write tid 1 to smithi04526579-35 2024-04-26T00:28:33.310 INFO:tasks.rados.rados.0.smithi045.stdout:1473: finishing write tid 2 to smithi04526579-35 2024-04-26T00:28:33.310 INFO:tasks.rados.rados.0.smithi045.stdout:1473: finishing write tid 3 to smithi04526579-35 2024-04-26T00:28:33.311 INFO:tasks.rados.rados.0.smithi045.stdout:1473: finishing write tid 4 to smithi04526579-35 2024-04-26T00:28:33.311 INFO:tasks.rados.rados.0.smithi045.stdout:1473: finishing write tid 5 to smithi04526579-35 2024-04-26T00:28:33.311 INFO:tasks.rados.rados.0.smithi045.stdout:1473: oid 35 updating version 0 to 871 2024-04-26T00:28:33.311 INFO:tasks.rados.rados.0.smithi045.stdout:1473: oid 35 version 871 is already newer than 869 2024-04-26T00:28:33.311 INFO:tasks.rados.rados.0.smithi045.stdout:1473: oid 35 updating version 871 to 872 2024-04-26T00:28:33.311 INFO:tasks.rados.rados.0.smithi045.stdout:1473: oid 35 version 872 is already newer than 870 2024-04-26T00:28:33.311 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 872 (ObjNum 553 snap 168 seq_num 553) dirty exists 2024-04-26T00:28:33.312 INFO:tasks.rados.rados.0.smithi045.stdout:1473: left oid 35 (ObjNum 553 snap 168 seq_num 553) 2024-04-26T00:28:33.312 INFO:tasks.rados.rados.0.smithi045.stdout:1474: finishing write tid 1 to smithi04526579-25 2024-04-26T00:28:33.312 INFO:tasks.rados.rados.0.smithi045.stdout:1474: finishing write tid 2 to smithi04526579-25 2024-04-26T00:28:33.312 INFO:tasks.rados.rados.0.smithi045.stdout:1473: done (2 left) 2024-04-26T00:28:33.312 INFO:tasks.rados.rados.0.smithi045.stdout:1476: rollback oid 22 current snap is 168 2024-04-26T00:28:33.312 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 22 to 159 2024-04-26T00:28:33.313 INFO:tasks.rados.rados.0.smithi045.stdout:1477: snap_create 2024-04-26T00:28:33.314 INFO:tasks.rados.rados.0.smithi045.stdout:1474: finishing write tid 3 to smithi04526579-25 2024-04-26T00:28:33.314 INFO:tasks.rados.rados.0.smithi045.stdout:1474: finishing write tid 4 to smithi04526579-25 2024-04-26T00:28:33.315 INFO:tasks.rados.rados.0.smithi045.stdout:1474: finishing write tid 5 to smithi04526579-25 2024-04-26T00:28:33.315 INFO:tasks.rados.rados.0.smithi045.stdout:1474: oid 25 updating version 0 to 1142 2024-04-26T00:28:33.315 INFO:tasks.rados.rados.0.smithi045.stdout:1474: oid 25 version 1142 is already newer than 1140 2024-04-26T00:28:33.315 INFO:tasks.rados.rados.0.smithi045.stdout:1474: oid 25 version 1142 is already newer than 1141 2024-04-26T00:28:33.315 INFO:tasks.rados.rados.0.smithi045.stdout:1474: oid 25 version 1142 is already newer than 1139 2024-04-26T00:28:33.315 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 1142 (ObjNum 554 snap 168 seq_num 554) dirty exists 2024-04-26T00:28:33.315 INFO:tasks.rados.rados.0.smithi045.stdout:1474: left oid 25 (ObjNum 554 snap 168 seq_num 554) 2024-04-26T00:28:33.315 INFO:tasks.rados.rados.0.smithi045.stdout:1476: finishing rollback tid 1 to smithi04526579-22 2024-04-26T00:28:33.316 INFO:tasks.rados.rados.0.smithi045.stdout:1476: finishing rollback tid 2 to smithi04526579-22 2024-04-26T00:28:33.316 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 759 (ObjNum 330 snap 96 seq_num 330) dirty exists 2024-04-26T00:28:33.330 INFO:tasks.rados.rados.0.smithi045.stdout:1475: finishing copy_from racing read to smithi04526579-45 2024-04-26T00:28:33.330 INFO:tasks.rados.rados.0.smithi045.stdout:1475: finishing copy_from to smithi04526579-45 2024-04-26T00:28:33.330 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 868 (ObjNum 546 snap 168 seq_num 546) dirty exists 2024-04-26T00:28:33.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f8129770640 10 monclient: get_monmap_and_config 2024-04-26T00:28:33.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f8129770640 10 monclient: build_initial_monmap 2024-04-26T00:28:33.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f8129770640 10 monclient: monmap: 2024-04-26T00:28:33.381 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:28:33.382 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:33.382 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:28:33.397101+0000 2024-04-26T00:28:33.382 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:28:33.397101+0000 2024-04-26T00:28:33.382 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:28:33.382 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:33.382 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:28:33.382 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:28:33.382 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:28:33.382 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:28:33.383 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:28:33.383 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:33.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f8129770640 1 Processor -- start 2024-04-26T00:28:33.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f8129770640 1 -- start start 2024-04-26T00:28:33.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f8129770640 10 monclient: init 2024-04-26T00:28:33.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f8129770640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:33.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f8129770640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-04-26T00:28:33.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f8129770640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81240a3d90 0x7f812409bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:33.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f8129770640 10 monclient(hunting): picked mon.noname-b con 0x7f81240a3d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:28:33.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f8129770640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f81240a4170 0x7f81240a45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:33.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f8129770640 10 monclient(hunting): picked mon.noname-c con 0x7f81240a4170 addr v2:172.21.15.45:3300/0 2024-04-26T00:28:33.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f8129770640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8124133b20 addr v1:172.21.15.45:6789/0 2024-04-26T00:28:33.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f8129770640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81240a5150 con 0x7f8124133b20 2024-04-26T00:28:33.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f8129770640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:33.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f8129770640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f812409b280 con 0x7f81240a4170 2024-04-26T00:28:33.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f8129770640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:33.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f8129770640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f812409b560 con 0x7f81240a3d90 2024-04-26T00:28:33.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f8129770640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:33.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f8129770640 10 monclient(hunting): authenticate will time out at 1494.703094s 2024-04-26T00:28:33.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f8122ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81240a3d90 0x7f812409bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:33.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f8122ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81240a3d90 0x7f812409bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:48624/0 (socket says 172.21.15.37:48624) 2024-04-26T00:28:33.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f8122ffd640 1 -- 172.21.15.37:0/1423884829 learned_addr learned my addr 172.21.15.37:0/1423884829 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:28:33.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f8122ffd640 10 monclient(hunting): get_auth_request con 0x7f81240a3d90 auth_method 0 2024-04-26T00:28:33.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f8122ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:33.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f8122ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:33.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f8122ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:33.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f8122ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:33.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f8122ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:33.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f8122ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:33.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f81227fc640 1 --2- 172.21.15.37:0/1423884829 >> v2:172.21.15.45:3300/0 conn(0x7f81240a4170 0x7f81240a45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:33.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f81227fc640 10 monclient(hunting): get_auth_request con 0x7f81240a4170 auth_method 0 2024-04-26T00:28:33.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f81227fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:33.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f81227fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:33.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f81227fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:33.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f8122ffd640 10 monclient(hunting): handle_auth_done global_id 84320 payload 274 2024-04-26T00:28:33.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f8122ffd640 1 -- 172.21.15.37:0/1423884829 >> v2:172.21.15.45:3300/0 conn(0x7f81240a4170 msgr2=0x7f81240a45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:28:33.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f8122ffd640 1 --2- 172.21.15.37:0/1423884829 >> v2:172.21.15.45:3300/0 conn(0x7f81240a4170 0x7f81240a45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:33.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f8122ffd640 1 -- 172.21.15.37:0/1423884829 >> v1:172.21.15.45:6789/0 conn(0x7f8124133b20 legacy=0x7f8124133f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:28:33.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f8122ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:28:33.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f8122ffd640 1 monclient: found mon.noname-b 2024-04-26T00:28:33.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f8122ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:33.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f8122ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:28:33.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.395+0000 7f8122ffd640 1 -- 172.21.15.37:0/1423884829 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81240a4af0 con 0x7f81240a3d90 2024-04-26T00:28:33.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.399+0000 7f8122ffd640 1 -- 172.21.15.37:0/1423884829 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81240a3d90 msgr2=0x7f812409bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:33.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.399+0000 7f8122ffd640 1 --2- 172.21.15.37:0/1423884829 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81240a3d90 0x7f812409bf40 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f811c009870 tx=0x7f811c01c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a97ea160cb4a1f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:33.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8121ffb640 1 -- 172.21.15.37:0/1423884829 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f811c0b0070 con 0x7f81240a3d90 2024-04-26T00:28:33.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8121ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:33.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8121ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:28:33.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8121ffb640 10 monclient: dump: 2024-04-26T00:28:33.396 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:33.396 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:33.396 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.396 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.396 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:33.396 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:33.397 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:33.397 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:33.397 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:33.397 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:33.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8121ffb640 10 monclient: _finish_auth 0 2024-04-26T00:28:33.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8121ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:28:33.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8121ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:33.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8129770640 5 monclient: authenticate success, global_id 84320 2024-04-26T00:28:33.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8129770640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:28:33.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8121ffb640 1 -- 172.21.15.37:0/1423884829 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f811c0ab600 con 0x7f81240a3d90 2024-04-26T00:28:33.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8121ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:28:33.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8121ffb640 1 -- 172.21.15.37:0/1423884829 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f811c0ab9d0 con 0x7f81240a3d90 2024-04-26T00:28:33.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8121ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:33.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8121ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:28:33.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8121ffb640 10 monclient: dump: 2024-04-26T00:28:33.397 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:33.398 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:33.398 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.398 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.398 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:33.398 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:33.398 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:33.398 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:33.398 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:33.398 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:33.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8129770640 10 monclient: get_monmap_and_config success 2024-04-26T00:28:33.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8129770640 4 set_mon_vals no callback set 2024-04-26T00:28:33.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8129770640 10 monclient: shutdown 2024-04-26T00:28:33.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8129770640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:28:33.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8129770640 1 -- 172.21.15.37:0/1423884829 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81240a3d90 msgr2=0x7f812409bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:33.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8129770640 1 --2- 172.21.15.37:0/1423884829 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81240a3d90 0x7f812409bf40 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f811c009870 tx=0x7f811c01c3e0 comp rx=0 tx=0).stop 2024-04-26T00:28:33.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8129770640 1 -- 172.21.15.37:0/1423884829 shutdown_connections 2024-04-26T00:28:33.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8129770640 1 --2- 172.21.15.37:0/1423884829 >> v2:172.21.15.45:3300/0 conn(0x7f81240a4170 0x7f81240a45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:33.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8129770640 1 --2- 172.21.15.37:0/1423884829 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81240a3d90 0x7f812409bf40 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:33.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8129770640 1 -- 172.21.15.37:0/1423884829 >> 172.21.15.37:0/1423884829 conn(0x7f81240aa0a0 msgr2=0x7f81240a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:28:33.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8129770640 1 -- 172.21.15.37:0/1423884829 shutdown_connections 2024-04-26T00:28:33.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8129770640 1 -- 172.21.15.37:0/1423884829 wait complete. 2024-04-26T00:28:33.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8129770640 10 monclient: build_initial_monmap 2024-04-26T00:28:33.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8129770640 10 monclient: monmap: 2024-04-26T00:28:33.399 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:28:33.399 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:28:33.399 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:28:33.399 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:28:33.399 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:28:33.399 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:28:33.400 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:28:33.400 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:28:33.400 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:28:33.400 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:33.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8129770640 1 Processor -- start 2024-04-26T00:28:33.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8129770640 1 -- start start 2024-04-26T00:28:33.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8129770640 10 monclient: init 2024-04-26T00:28:33.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8129770640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:33.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8129770640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:28:33.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8129770640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81240a3d90 0x7f81241cf1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:33.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8129770640 10 monclient(hunting): picked mon.noname-a con 0x7f81240a3d90 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:28:33.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8129770640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81240a4170 0x7f812413e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:33.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8129770640 10 monclient(hunting): picked mon.noname-b con 0x7f81240a4170 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:28:33.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8129770640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8124133b20 0x7f812413e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:33.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8129770640 10 monclient(hunting): picked mon.noname-c con 0x7f8124133b20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:28:33.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8129770640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:33.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8129770640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f8124141e30 con 0x7f81240a3d90 2024-04-26T00:28:33.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8129770640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:33.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8129770640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f812409c690 con 0x7f81240a4170 2024-04-26T00:28:33.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8129770640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:33.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8129770640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8124141fb0 con 0x7f8124133b20 2024-04-26T00:28:33.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8129770640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:33.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8129770640 10 monclient(hunting): authenticate will time out at 1494.710628s 2024-04-26T00:28:33.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8122ffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81240a3d90 0x7f81241cf1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:33.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f81237fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8124133b20 0x7f812413e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:33.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8122ffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81240a3d90 0x7f81241cf1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:49852/0 (socket says 172.21.15.37:49852) 2024-04-26T00:28:33.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8122ffd640 1 -- 172.21.15.37:0/3690126012 learned_addr learned my addr 172.21.15.37:0/3690126012 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:28:33.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f81237fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8124133b20 0x7f812413e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:48630/0 (socket says 172.21.15.37:48630) 2024-04-26T00:28:33.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8122ffd640 10 monclient(hunting): get_auth_request con 0x7f81240a3d90 auth_method 0 2024-04-26T00:28:33.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8122ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:33.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8122ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:33.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8122ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:33.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f81237fe640 10 monclient(hunting): get_auth_request con 0x7f8124133b20 auth_method 0 2024-04-26T00:28:33.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f81237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:33.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f81237fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:33.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f81237fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:33.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8122ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:33.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8122ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:33.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8122ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:33.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f81237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:33.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f81237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:33.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f81237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:33.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f81227fc640 1 --2- 172.21.15.37:0/3690126012 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81240a4170 0x7f812413e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:33.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8122ffd640 10 monclient(hunting): handle_auth_done global_id 94251 payload 824 2024-04-26T00:28:33.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8122ffd640 1 -- 172.21.15.37:0/3690126012 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8124133b20 msgr2=0x7f812413e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:33.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8122ffd640 1 --2- 172.21.15.37:0/3690126012 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8124133b20 0x7f812413e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:33.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8122ffd640 1 -- 172.21.15.37:0/3690126012 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81240a4170 msgr2=0x7f812413e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:33.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8122ffd640 1 --2- 172.21.15.37:0/3690126012 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81240a4170 0x7f812413e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:33.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8122ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:28:33.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8122ffd640 1 monclient: found mon.noname-a 2024-04-26T00:28:33.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8122ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:33.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8122ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:28:33.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f8122ffd640 1 -- 172.21.15.37:0/3690126012 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f812413ccb0 con 0x7f81240a3d90 2024-04-26T00:28:33.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f81227fc640 10 monclient: get_auth_request con 0x7f81240a4170 auth_method 0 2024-04-26T00:28:33.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f81227fc640 1 --2- 172.21.15.37:0/3690126012 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81240a4170 0x7f812413e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:28:33.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.403+0000 7f81237fe640 1 --2- 172.21.15.37:0/3690126012 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8124133b20 0x7f812413e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:28:33.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f8122ffd640 1 -- 172.21.15.37:0/3690126012 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81240a3d90 msgr2=0x7f81241cf1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:33.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f8122ffd640 1 --2- 172.21.15.37:0/3690126012 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81240a3d90 0x7f81241cf1c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f811c0ae040 tx=0x7f811c019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:33.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f811b7fe640 1 -- 172.21.15.37:0/3690126012 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f811c0b0040 con 0x7f81240a3d90 2024-04-26T00:28:33.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f811b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:33.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f811b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:28:33.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f811b7fe640 10 monclient: dump: 2024-04-26T00:28:33.407 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:33.407 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:33.408 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.408 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.408 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:33.408 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:33.408 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:33.408 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:33.408 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:33.408 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:33.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f811b7fe640 10 monclient: _finish_auth 0 2024-04-26T00:28:33.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f811b7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:28:33.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f811b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:33.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f811b7fe640 1 -- 172.21.15.37:0/3690126012 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f811c0b9070 con 0x7f81240a3d90 2024-04-26T00:28:33.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f811b7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:28:33.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f811b7fe640 1 -- 172.21.15.37:0/3690126012 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f811c0b37f0 con 0x7f81240a3d90 2024-04-26T00:28:33.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f811b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:33.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f811b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:28:33.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f811b7fe640 10 monclient: dump: 2024-04-26T00:28:33.409 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:33.409 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:33.409 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.409 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.409 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:33.409 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:33.409 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:33.409 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:33.409 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:33.409 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:33.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f8129770640 5 monclient: authenticate success, global_id 94251 2024-04-26T00:28:33.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f8129770640 10 monclient: _renew_subs 2024-04-26T00:28:33.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f81217fa640 4 set_mon_vals no callback set 2024-04-26T00:28:33.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f8129770640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:28:33.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f8129770640 1 -- 172.21.15.37:0/3690126012 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f812409ada0 con 0x7f81240a3d90 2024-04-26T00:28:33.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f8129770640 10 monclient: _renew_subs 2024-04-26T00:28:33.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f8129770640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:28:33.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f8129770640 1 -- 172.21.15.37:0/3690126012 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81241cf8e0 con 0x7f81240a3d90 2024-04-26T00:28:33.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f8129770640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:33.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f8129770640 1 --2- 172.21.15.37:0/3690126012 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8124138d20 0x7f8124139100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:33.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f8129770640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:33.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f8129770640 1 -- 172.21.15.37:0/3690126012 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f812404f510 con 0x7f8124138d20 2024-04-26T00:28:33.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f81227fc640 1 --2- 172.21.15.37:0/3690126012 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8124138d20 0x7f8124139100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:33.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f81227fc640 10 monclient: get_auth_request con 0x7f8124138d20 auth_method 0 2024-04-26T00:28:33.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f81227fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:33.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f81227fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:33.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f81227fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:33.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f811b7fe640 1 -- 172.21.15.37:0/3690126012 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f811c0b3990 con 0x7f81240a3d90 2024-04-26T00:28:33.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f81227fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:33.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f81227fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:33.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f81227fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:33.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f81227fc640 10 monclient(hunting): handle_auth_done global_id 94254 payload 824 2024-04-26T00:28:33.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f81227fc640 1 -- 172.21.15.37:0/3690126012 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f812413a9c0 con 0x7f8124138d20 2024-04-26T00:28:33.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f811b7fe640 1 --2- 172.21.15.37:0/3690126012 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8104035ed0 0x7f8104038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:33.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f811b7fe640 1 -- 172.21.15.37:0/3690126012 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(338..338 src has 177..338) ==== 3975+0+0 (secure 0 0 0) 0x7f811c0e2660 con 0x7f81240a3d90 2024-04-26T00:28:33.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f81227fc640 1 -- 172.21.15.37:0/3690126012 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8124138d20 msgr2=0x7f8124139100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:28:33.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f81227fc640 1 -- 172.21.15.37:0/3690126012 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8124138d20 msgr2=0x7f8124139100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:28:33.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f81227fc640 1 --2- 172.21.15.37:0/3690126012 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8124138d20 0x7f8124139100 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f810c009ec0 tx=0x7f810c00f520 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:28:33.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f81227fc640 1 --2- 172.21.15.37:0/3690126012 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8124138d20 0x7f8124139100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:28:33.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f81237fe640 1 --2- 172.21.15.37:0/3690126012 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8104035ed0 0x7f8104038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:33.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.407+0000 7f81237fe640 10 monclient: get_auth_request con 0x7f8104035ed0 auth_method 0 2024-04-26T00:28:33.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.411+0000 7f81237fe640 1 --2- 172.21.15.37:0/3690126012 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8104035ed0 0x7f8104038390 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f812413b910 tx=0x7f8114007dc0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:33.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.611+0000 7f81227fc640 1 --2- 172.21.15.37:0/3690126012 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8124138d20 0x7f8124139100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:33.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.611+0000 7f81227fc640 10 monclient: get_auth_request con 0x7f8124138d20 auth_method 0 2024-04-26T00:28:33.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.611+0000 7f81227fc640 10 monclient: get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:33.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.611+0000 7f81227fc640 10 monclient: _init_auth method 2 2024-04-26T00:28:33.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.611+0000 7f81227fc640 10 monclient: _init_auth already have auth, reseting 2024-04-26T00:28:33.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.611+0000 7f81227fc640 10 monclient: handle_auth_reply_more payload 9 2024-04-26T00:28:33.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.611+0000 7f81227fc640 10 monclient: handle_auth_reply_more payload_len 9 2024-04-26T00:28:33.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.611+0000 7f81227fc640 10 monclient: handle_auth_reply_more responding with 132 bytes 2024-04-26T00:28:33.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.611+0000 7f81227fc640 10 monclient: handle_auth_done global_id 94254 payload 293 2024-04-26T00:28:33.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.611+0000 7f81227fc640 1 -- 172.21.15.37:0/3690126012 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8124138d20 msgr2=0x7f8124139100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:33.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.611+0000 7f81227fc640 1 --2- 172.21.15.37:0/3690126012 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8124138d20 0x7f8124139100 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f810c007fd0 tx=0x7f810c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:33.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.611+0000 7f811b7fe640 1 -- 172.21.15.37:0/3690126012 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f810c0039a0 con 0x7f8124138d20 2024-04-26T00:28:33.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.611+0000 7f811b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:33.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.611+0000 7f811b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:28:33.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.611+0000 7f811b7fe640 10 monclient: dump: 2024-04-26T00:28:33.600 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:33.600 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:33.600 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.600 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.600 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:33.600 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:33.600 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:33.600 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:33.600 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:33.600 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:33.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.615+0000 7f811b7fe640 1 -- 172.21.15.37:0/3690126012 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f810c003ee0 con 0x7f8124138d20 2024-04-26T00:28:33.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.615+0000 7f811b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:33.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.615+0000 7f811b7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:28:33.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.615+0000 7f811b7fe640 1 -- 172.21.15.37:0/3690126012 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8124138d20 msgr2=0x7f8124139100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:33.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.615+0000 7f811b7fe640 1 --2- 172.21.15.37:0/3690126012 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8124138d20 0x7f8124139100 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f810c007fd0 tx=0x7f810c00b040 comp rx=0 tx=0).stop 2024-04-26T00:28:33.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.615+0000 7f811b7fe640 1 -- 172.21.15.37:0/3690126012 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8124138d20 msgr2=0x7f8124139100 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:28:33.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.615+0000 7f811b7fe640 1 --2- 172.21.15.37:0/3690126012 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8124138d20 0x7f8124139100 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:33.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.619+0000 7f8129770640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:28:33.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.619+0000 7f8129770640 1 --2- 172.21.15.37:0/3690126012 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8124139640 0x7f8124134b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:33.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.623+0000 7f8129770640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:33.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.623+0000 7f8129770640 1 -- 172.21.15.37:0/3690126012 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f812413b2a0 con 0x7f8124139640 2024-04-26T00:28:33.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.623+0000 7f81227fc640 1 --2- 172.21.15.37:0/3690126012 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8124139640 0x7f8124134b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:33.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.623+0000 7f81227fc640 10 monclient: get_auth_request con 0x7f8124139640 auth_method 0 2024-04-26T00:28:33.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.623+0000 7f81227fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:33.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.623+0000 7f81227fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:33.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.623+0000 7f81227fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:33.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.623+0000 7f81227fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:33.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.623+0000 7f81227fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:33.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.623+0000 7f81227fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:33.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.623+0000 7f81227fc640 10 monclient(hunting): handle_auth_done global_id 94257 payload 824 2024-04-26T00:28:33.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.623+0000 7f81227fc640 1 -- 172.21.15.37:0/3690126012 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8124075dc0 con 0x7f8124139640 2024-04-26T00:28:33.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.623+0000 7f81227fc640 1 -- 172.21.15.37:0/3690126012 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8124139640 msgr2=0x7f8124134b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:33.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.623+0000 7f81227fc640 1 --2- 172.21.15.37:0/3690126012 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8124139640 0x7f8124134b30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f810c01d000 tx=0x7f810c004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:33.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.627+0000 7f811b7fe640 1 -- 172.21.15.37:0/3690126012 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f810c023770 con 0x7f8124139640 2024-04-26T00:28:33.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.627+0000 7f811b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:33.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.627+0000 7f811b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:28:33.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.627+0000 7f811b7fe640 10 monclient: dump: 2024-04-26T00:28:33.617 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:33.617 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:33.618 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.618 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.618 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:33.618 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:33.618 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:33.618 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:33.618 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:33.619 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:33.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.627+0000 7f811b7fe640 1 -- 172.21.15.37:0/3690126012 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4718 (secure 0 0 0) 0x7f810c004890 con 0x7f8124139640 2024-04-26T00:28:33.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.627+0000 7f811b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:28:33.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.627+0000 7f811b7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:28:33.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.627+0000 7f811b7fe640 1 -- 172.21.15.37:0/3690126012 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8124139640 msgr2=0x7f8124134b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:33.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.627+0000 7f811b7fe640 1 --2- 172.21.15.37:0/3690126012 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8124139640 0x7f8124134b30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f810c01d000 tx=0x7f810c004a50 comp rx=0 tx=0).stop 2024-04-26T00:28:33.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.627+0000 7f811b7fe640 1 -- 172.21.15.37:0/3690126012 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8124139640 msgr2=0x7f8124134b30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:28:33.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.627+0000 7f811b7fe640 1 --2- 172.21.15.37:0/3690126012 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8124139640 0x7f8124134b30 secure :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f810c01d000 tx=0x7f810c004a50 comp rx=0 tx=0).stop 2024-04-26T00:28:33.620 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:28:33.620 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:28:33.621 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:28:33.621 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:28:33.621 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 64, 2024-04-26T00:28:33.621 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 74386, 2024-04-26T00:28:33.621 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:28:33.621 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:28:33.622 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:28:33.622 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:28:33.622 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:33.622 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 5, 2024-04-26T00:28:33.622 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:28:33.622 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:28:33.623 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:28:33.623 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:28:33.623 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:28:33.623 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:28:33.623 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:28:33.623 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:28:33.623 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:28:33.624 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:28:33.624 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:28:33.624 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:28:33.624 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:28:33.624 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:28:33.624 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:33.625 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:28:33.625 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:28:33.625 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:28:33.625 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:28:33.625 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:28:33.625 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:28:33.625 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:28:33.626 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:28:33.626 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:28:33.626 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:28:33.626 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:28:33.626 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:28:33.626 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:28:33.627 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:33.627 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:33.627 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:28:33.627 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:28:33.627 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:28:33.627 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:28:33.628 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:28:33.628 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:28:33.628 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:28:33.628 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:28:33.628 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:28:33.628 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:28:33.628 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:28:33.629 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:28:33.629 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:28:33.629 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:28:33.629 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:28:33.629 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:28:33.629 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:28:33.630 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:28:33.630 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:28:33.630 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:28:33.630 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:28:33.630 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:28:33.630 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:28:33.630 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:28:33.631 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:28:33.631 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:28:33.631 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:28:33.631 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:28:33.631 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:33.631 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:28:33.632 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:33.632 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:28:33.632 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:33.632 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:28:33.632 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:28:33.632 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:28:33.633 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:28:33.633 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:33.633 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:28:33.633 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:28:33.633 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:33.633 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:33.633 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:33.634 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:28:33.634 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:28:33.634 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:33.634 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:33.634 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:33.634 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:33.635 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:28:33.635 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:28:33.635 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:28:33.635 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:28:33.635 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:28:33.635 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:33.635 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:33.636 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:28:33.636 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:28:33.636 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:28:33.636 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:28:33.636 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:33.636 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:28:33.637 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:28:33.637 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:33.637 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:33.637 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:33.637 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:28:33.638 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:28:33.638 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:33.638 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:33.638 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:33.638 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:33.638 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:28:33.639 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:28:33.639 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:28:33.639 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:28:33.639 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:28:33.639 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:33.639 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:33.640 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:28:33.640 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:28:33.640 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:28:33.640 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:28:33.640 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:33.640 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:28:33.640 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:28:33.641 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:33.641 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:33.641 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:33.641 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:28:33.641 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:28:33.641 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:33.642 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:33.642 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:33.642 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:33.642 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:28:33.642 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:28:33.642 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:28:33.643 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:28:33.643 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:28:33.643 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:33.643 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:33.643 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:33.643 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:28:33.643 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:28:33.644 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:33.644 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:28:33.644 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:28:33.644 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:28:33.644 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:33.644 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:33.645 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:28:33.645 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:33.645 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:28:33.645 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:28:33.645 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:28:33.645 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:33.645 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:33.646 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:28:33.646 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:33.646 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:28:33.646 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:28:33.646 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:28:33.647 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:33.647 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:33.647 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:33.647 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:28:33.648 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:28:33.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.627+0000 7f8129770640 1 -- 172.21.15.37:0/3690126012 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8104035ed0 msgr2=0x7f8104038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:33.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.627+0000 7f8129770640 1 --2- 172.21.15.37:0/3690126012 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8104035ed0 0x7f8104038390 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f812413b910 tx=0x7f8114007dc0 comp rx=0 tx=0).stop 2024-04-26T00:28:33.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.627+0000 7f8129770640 10 monclient: shutdown 2024-04-26T00:28:33.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.627+0000 7f8129770640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:28:33.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.627+0000 7f8129770640 1 -- 172.21.15.37:0/3690126012 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81240a3d90 msgr2=0x7f81241cf1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:33.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.627+0000 7f8129770640 1 --2- 172.21.15.37:0/3690126012 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81240a3d90 0x7f81241cf1c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f811c0ae040 tx=0x7f811c019180 comp rx=0 tx=0).stop 2024-04-26T00:28:33.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.627+0000 7f81237fe640 1 -- 172.21.15.37:0/3690126012 reap_dead start 2024-04-26T00:28:33.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.627+0000 7f81237fe640 1 -- 172.21.15.37:0/3690126012 reap_dead start 2024-04-26T00:28:33.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.627+0000 7f8129770640 1 -- 172.21.15.37:0/3690126012 shutdown_connections 2024-04-26T00:28:33.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.627+0000 7f8129770640 1 -- 172.21.15.37:0/3690126012 >> 172.21.15.37:0/3690126012 conn(0x7f81240aa0a0 msgr2=0x7f81240a2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:28:33.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.627+0000 7f8129770640 1 -- 172.21.15.37:0/3690126012 shutdown_connections 2024-04-26T00:28:33.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.627+0000 7f8129770640 1 -- 172.21.15.37:0/3690126012 wait complete. 2024-04-26T00:28:33.653 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:28:33.653 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:28:33.653 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:28:33.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.799+0000 7f37d5885640 10 monclient: get_monmap_and_config 2024-04-26T00:28:33.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.799+0000 7f37d5885640 10 monclient: build_initial_monmap 2024-04-26T00:28:33.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.799+0000 7f37d5885640 10 monclient: monmap: 2024-04-26T00:28:33.784 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:28:33.785 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:33.785 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:28:33.800022+0000 2024-04-26T00:28:33.785 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:28:33.800022+0000 2024-04-26T00:28:33.785 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:28:33.785 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:33.785 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:28:33.785 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:28:33.785 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:28:33.785 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:28:33.785 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:28:33.785 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:33.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.799+0000 7f37d5885640 1 Processor -- start 2024-04-26T00:28:33.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.799+0000 7f37d5885640 1 -- start start 2024-04-26T00:28:33.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.799+0000 7f37d5885640 10 monclient: init 2024-04-26T00:28:33.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.799+0000 7f37d5885640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:33.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.799+0000 7f37d5885640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-04-26T00:28:33.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.799+0000 7f37d5885640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f37d00a3d90 addr v1:172.21.15.45:6789/0 2024-04-26T00:28:33.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.799+0000 7f37d5885640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f37d00a4170 0x7f37d00a45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:33.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.799+0000 7f37d5885640 10 monclient(hunting): picked mon.noname-a con 0x7f37d00a4170 addr v2:172.21.15.37:3300/0 2024-04-26T00:28:33.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.799+0000 7f37d5885640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f37d013c3e0 0x7f37d013e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:33.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.799+0000 7f37d5885640 10 monclient(hunting): picked mon.noname-c con 0x7f37d013c3e0 addr v2:172.21.15.45:3300/0 2024-04-26T00:28:33.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.799+0000 7f37d5885640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37d00a5150 con 0x7f37d00a3d90 2024-04-26T00:28:33.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.799+0000 7f37d5885640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:33.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.799+0000 7f37d5885640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f37d009c650 con 0x7f37d00a4170 2024-04-26T00:28:33.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.799+0000 7f37d5885640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:33.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.799+0000 7f37d5885640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f37d009b280 con 0x7f37d013c3e0 2024-04-26T00:28:33.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.799+0000 7f37d5885640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:33.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.799+0000 7f37d5885640 10 monclient(hunting): authenticate will time out at 1495.105832s 2024-04-26T00:28:33.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.799+0000 7f37ce7fc640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f37d00a4170 0x7f37d00a45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:33.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.799+0000 7f37ce7fc640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f37d00a4170 0x7f37d00a45e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:49890/0 (socket says 172.21.15.37:49890) 2024-04-26T00:28:33.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.799+0000 7f37ce7fc640 1 -- 172.21.15.37:0/3045268455 learned_addr learned my addr 172.21.15.37:0/3045268455 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:28:33.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.799+0000 7f37ce7fc640 10 monclient(hunting): get_auth_request con 0x7f37d00a4170 auth_method 0 2024-04-26T00:28:33.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.799+0000 7f37ce7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:33.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.799+0000 7f37ce7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:33.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.799+0000 7f37ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:33.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.799+0000 7f37ce7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:33.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.799+0000 7f37ce7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:33.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.799+0000 7f37ce7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:33.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.799+0000 7f37cf7fe640 1 --2- 172.21.15.37:0/3045268455 >> v2:172.21.15.45:3300/0 conn(0x7f37d013c3e0 0x7f37d013e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:33.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.799+0000 7f37ce7fc640 10 monclient(hunting): handle_auth_done global_id 94260 payload 274 2024-04-26T00:28:33.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.799+0000 7f37ce7fc640 1 -- 172.21.15.37:0/3045268455 >> v2:172.21.15.45:3300/0 conn(0x7f37d013c3e0 msgr2=0x7f37d013e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:28:33.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.799+0000 7f37ce7fc640 1 --2- 172.21.15.37:0/3045268455 >> v2:172.21.15.45:3300/0 conn(0x7f37d013c3e0 0x7f37d013e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:33.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.799+0000 7f37ce7fc640 1 -- 172.21.15.37:0/3045268455 >> v1:172.21.15.45:6789/0 conn(0x7f37d00a3d90 legacy=0x7f37d009bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:28:33.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.799+0000 7f37ce7fc640 10 monclient: _finish_hunting 0 2024-04-26T00:28:33.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.799+0000 7f37ce7fc640 1 monclient: found mon.noname-a 2024-04-26T00:28:33.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.799+0000 7f37ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:33.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.799+0000 7f37ce7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:28:33.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.799+0000 7f37ce7fc640 1 -- 172.21.15.37:0/3045268455 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37d009b560 con 0x7f37d00a4170 2024-04-26T00:28:33.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.803+0000 7f37ce7fc640 1 -- 172.21.15.37:0/3045268455 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f37d00a4170 msgr2=0x7f37d00a45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:33.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.803+0000 7f37ce7fc640 1 --2- 172.21.15.37:0/3045268455 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f37d00a4170 0x7f37d00a45e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f37d00a6e00 tx=0x7f37b801c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e186dea334f25cf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:33.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.803+0000 7f37cdffb640 1 -- 172.21.15.37:0/3045268455 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b80b0070 con 0x7f37d00a4170 2024-04-26T00:28:33.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.803+0000 7f37cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:33.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.803+0000 7f37cdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:28:33.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.803+0000 7f37cdffb640 10 monclient: dump: 2024-04-26T00:28:33.800 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:33.800 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:33.800 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.800 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.800 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:33.800 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:33.800 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:33.800 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:33.801 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:33.801 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:33.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.803+0000 7f37cdffb640 10 monclient: _finish_auth 0 2024-04-26T00:28:33.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.803+0000 7f37cdffb640 10 monclient: _check_auth_tickets 2024-04-26T00:28:33.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.803+0000 7f37cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:33.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.803+0000 7f37d5885640 5 monclient: authenticate success, global_id 94260 2024-04-26T00:28:33.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.803+0000 7f37cdffb640 1 -- 172.21.15.37:0/3045268455 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37b80ab600 con 0x7f37d00a4170 2024-04-26T00:28:33.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.803+0000 7f37d5885640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:28:33.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.803+0000 7f37cdffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:28:33.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.803+0000 7f37d5885640 10 monclient: get_monmap_and_config success 2024-04-26T00:28:33.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.803+0000 7f37d5885640 4 set_mon_vals no callback set 2024-04-26T00:28:33.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.803+0000 7f37cdffb640 1 -- 172.21.15.37:0/3045268455 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b80ab9d0 con 0x7f37d00a4170 2024-04-26T00:28:33.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.803+0000 7f37cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:33.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.803+0000 7f37cdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:28:33.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.803+0000 7f37cdffb640 10 monclient: dump: 2024-04-26T00:28:33.802 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:33.802 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:33.803 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.803 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.803 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:33.803 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:33.803 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:33.803 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:33.803 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:33.803 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:33.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.803+0000 7f37d5885640 10 monclient: shutdown 2024-04-26T00:28:33.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.803+0000 7f37d5885640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:28:33.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.803+0000 7f37d5885640 1 -- 172.21.15.37:0/3045268455 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f37d00a4170 msgr2=0x7f37d00a45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:33.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.803+0000 7f37d5885640 1 --2- 172.21.15.37:0/3045268455 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f37d00a4170 0x7f37d00a45e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f37d00a6e00 tx=0x7f37b801c3a0 comp rx=0 tx=0).stop 2024-04-26T00:28:33.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.803+0000 7f37d5885640 1 -- 172.21.15.37:0/3045268455 shutdown_connections 2024-04-26T00:28:33.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.803+0000 7f37d5885640 1 --2- 172.21.15.37:0/3045268455 >> v2:172.21.15.45:3300/0 conn(0x7f37d013c3e0 0x7f37d013e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:33.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.803+0000 7f37d5885640 1 --2- 172.21.15.37:0/3045268455 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f37d00a4170 0x7f37d00a45e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:33.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.803+0000 7f37d5885640 1 -- 172.21.15.37:0/3045268455 >> 172.21.15.37:0/3045268455 conn(0x7f37d00aa0a0 msgr2=0x7f37d00a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:28:33.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.803+0000 7f37d5885640 1 -- 172.21.15.37:0/3045268455 shutdown_connections 2024-04-26T00:28:33.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.803+0000 7f37d5885640 1 -- 172.21.15.37:0/3045268455 wait complete. 2024-04-26T00:28:33.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37d5885640 10 monclient: build_initial_monmap 2024-04-26T00:28:33.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37d5885640 10 monclient: monmap: 2024-04-26T00:28:33.805 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:28:33.805 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:28:33.805 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:28:33.805 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:28:33.806 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:28:33.806 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:28:33.806 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:28:33.806 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:28:33.806 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:28:33.806 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:33.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37d5885640 1 Processor -- start 2024-04-26T00:28:33.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37d5885640 1 -- start start 2024-04-26T00:28:33.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37d5885640 10 monclient: init 2024-04-26T00:28:33.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37d5885640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:33.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37d5885640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:28:33.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37d5885640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f37d00a3d90 0x7f37d01cee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:33.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37d5885640 10 monclient(hunting): picked mon.noname-a con 0x7f37d00a3d90 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:28:33.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37d5885640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37d00a4170 0x7f37d0136fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:33.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37d5885640 10 monclient(hunting): picked mon.noname-c con 0x7f37d00a4170 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:28:33.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37d5885640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37d013c3e0 0x7f37d013a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:33.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37d5885640 10 monclient(hunting): picked mon.noname-b con 0x7f37d013c3e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:28:33.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37d5885640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:33.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37d5885640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f37d01418c0 con 0x7f37d00a3d90 2024-04-26T00:28:33.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37d5885640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:33.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37d5885640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f37d0141740 con 0x7f37d013c3e0 2024-04-26T00:28:33.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37d5885640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:33.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37d5885640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f37d0141a40 con 0x7f37d00a4170 2024-04-26T00:28:33.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37d5885640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:33.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37d5885640 10 monclient(hunting): authenticate will time out at 1495.113010s 2024-04-26T00:28:33.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37ceffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f37d00a3d90 0x7f37d01cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:33.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37ce7fc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37d00a4170 0x7f37d0136fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:33.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37ceffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f37d00a3d90 0x7f37d01cee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:49902/0 (socket says 172.21.15.37:49902) 2024-04-26T00:28:33.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37ceffd640 1 -- 172.21.15.37:0/3290313328 learned_addr learned my addr 172.21.15.37:0/3290313328 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:28:33.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37ceffd640 10 monclient(hunting): get_auth_request con 0x7f37d00a3d90 auth_method 0 2024-04-26T00:28:33.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37ceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:33.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37ceffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:33.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:33.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37ce7fc640 10 monclient(hunting): get_auth_request con 0x7f37d00a4170 auth_method 0 2024-04-26T00:28:33.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37ce7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:33.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37ce7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:33.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:33.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37ceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:33.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37ceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:33.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37ceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:33.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37ce7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:33.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37ce7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:33.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37ce7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:33.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37cf7fe640 1 --2- 172.21.15.37:0/3290313328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37d013c3e0 0x7f37d013a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:33.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37ceffd640 10 monclient(hunting): handle_auth_done global_id 94263 payload 824 2024-04-26T00:28:33.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37ceffd640 1 -- 172.21.15.37:0/3290313328 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37d00a4170 msgr2=0x7f37d0136fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:33.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37ceffd640 1 --2- 172.21.15.37:0/3290313328 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37d00a4170 0x7f37d0136fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:33.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37ceffd640 1 -- 172.21.15.37:0/3290313328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37d013c3e0 msgr2=0x7f37d013a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:33.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37ceffd640 1 --2- 172.21.15.37:0/3290313328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37d013c3e0 0x7f37d013a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:33.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37ceffd640 10 monclient: _finish_hunting 0 2024-04-26T00:28:33.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37ceffd640 1 monclient: found mon.noname-a 2024-04-26T00:28:33.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:33.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37ceffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:28:33.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37ceffd640 1 -- 172.21.15.37:0/3290313328 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37d0121a50 con 0x7f37d00a3d90 2024-04-26T00:28:33.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37cf7fe640 10 monclient: get_auth_request con 0x7f37d013c3e0 auth_method 0 2024-04-26T00:28:33.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37cf7fe640 1 --2- 172.21.15.37:0/3290313328 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37d013c3e0 0x7f37d013a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:28:33.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37ceffd640 1 -- 172.21.15.37:0/3290313328 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f37d00a3d90 msgr2=0x7f37d01cee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:33.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37ceffd640 1 --2- 172.21.15.37:0/3290313328 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f37d00a3d90 0x7f37d01cee80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f37c000c7c0 tx=0x7f37c000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:33.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37c7fff640 1 -- 172.21.15.37:0/3290313328 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c0016020 con 0x7f37d00a3d90 2024-04-26T00:28:33.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.807+0000 7f37c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:33.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37c7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:28:33.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37c7fff640 10 monclient: dump: 2024-04-26T00:28:33.811 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:33.811 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:33.811 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.811 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.811 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:33.811 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:33.812 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:33.812 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:33.812 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:33.812 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:33.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37c7fff640 10 monclient: _finish_auth 0 2024-04-26T00:28:33.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37c7fff640 10 monclient: _check_auth_tickets 2024-04-26T00:28:33.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:33.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37c7fff640 1 -- 172.21.15.37:0/3290313328 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37c000ac60 con 0x7f37d00a3d90 2024-04-26T00:28:33.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37d5885640 5 monclient: authenticate success, global_id 94263 2024-04-26T00:28:33.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37d5885640 10 monclient: _renew_subs 2024-04-26T00:28:33.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37d5885640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:28:33.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37d5885640 1 -- 172.21.15.37:0/3290313328 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37d009ada0 con 0x7f37d00a3d90 2024-04-26T00:28:33.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37c7fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:28:33.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37c7fff640 1 -- 172.21.15.37:0/3290313328 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c0012660 con 0x7f37d00a3d90 2024-04-26T00:28:33.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:33.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37c7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:28:33.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37c7fff640 10 monclient: dump: 2024-04-26T00:28:33.813 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:33.813 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:33.813 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.813 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.813 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:33.813 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:33.813 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:33.813 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:33.813 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:33.813 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:33.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37cd7fa640 4 set_mon_vals no callback set 2024-04-26T00:28:33.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37d5885640 10 monclient: _renew_subs 2024-04-26T00:28:33.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37d5885640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:28:33.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37d5885640 1 -- 172.21.15.37:0/3290313328 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37d0120660 con 0x7f37d00a3d90 2024-04-26T00:28:33.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37d5885640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:33.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37d5885640 1 --2- 172.21.15.37:0/3290313328 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37d0137500 0x7f37d013a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:33.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37d5885640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:33.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37d5885640 1 -- 172.21.15.37:0/3290313328 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f37d004f510 con 0x7f37d0137500 2024-04-26T00:28:33.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37ce7fc640 1 --2- 172.21.15.37:0/3290313328 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37d0137500 0x7f37d013a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:33.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37ce7fc640 10 monclient: get_auth_request con 0x7f37d0137500 auth_method 0 2024-04-26T00:28:33.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37ce7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:33.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37ce7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:33.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:33.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37c7fff640 1 -- 172.21.15.37:0/3290313328 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f37c0019020 con 0x7f37d00a3d90 2024-04-26T00:28:33.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37ce7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:33.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37ce7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:33.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37ce7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:33.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37c7fff640 1 --2- 172.21.15.37:0/3290313328 >> v2:172.21.15.45:6812/2379918623 conn(0x7f37b0035e80 0x7f37b0038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:33.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37ce7fc640 10 monclient(hunting): handle_auth_done global_id 84329 payload 824 2024-04-26T00:28:33.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37ce7fc640 1 -- 172.21.15.37:0/3290313328 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f37d0121830 con 0x7f37d0137500 2024-04-26T00:28:33.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37c7fff640 1 -- 172.21.15.37:0/3290313328 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(338..338 src has 177..338) ==== 3975+0+0 (secure 0 0 0) 0x7f37c0048b40 con 0x7f37d00a3d90 2024-04-26T00:28:33.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37cf7fe640 1 --2- 172.21.15.37:0/3290313328 >> v2:172.21.15.45:6812/2379918623 conn(0x7f37b0035e80 0x7f37b0038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:33.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37cf7fe640 10 monclient: get_auth_request con 0x7f37b0035e80 auth_method 0 2024-04-26T00:28:33.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37ce7fc640 1 -- 172.21.15.37:0/3290313328 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37d0137500 msgr2=0x7f37d013a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:33.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37ce7fc640 1 --2- 172.21.15.37:0/3290313328 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37d0137500 0x7f37d013a650 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f37d00a57e0 tx=0x7f37b8019140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:33.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37c7fff640 1 -- 172.21.15.37:0/3290313328 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b80ba070 con 0x7f37d0137500 2024-04-26T00:28:33.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:33.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37c7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:28:33.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.811+0000 7f37c7fff640 10 monclient: dump: 2024-04-26T00:28:33.820 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:33.821 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:33.821 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.821 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.821 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:33.821 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:33.821 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:33.822 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:33.822 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:33.822 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:33.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.815+0000 7f37c7fff640 1 -- 172.21.15.37:0/3290313328 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f37b80b1040 con 0x7f37d0137500 2024-04-26T00:28:33.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.815+0000 7f37c7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:33.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.815+0000 7f37c7fff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:28:33.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.815+0000 7f37c7fff640 1 -- 172.21.15.37:0/3290313328 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37d0137500 msgr2=0x7f37d013a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:33.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.815+0000 7f37c7fff640 1 --2- 172.21.15.37:0/3290313328 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37d0137500 0x7f37d013a650 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f37d00a57e0 tx=0x7f37b8019140 comp rx=0 tx=0).stop 2024-04-26T00:28:33.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.815+0000 7f37c7fff640 1 -- 172.21.15.37:0/3290313328 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37d0137500 msgr2=0x7f37d013a650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:28:33.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.815+0000 7f37c7fff640 1 --2- 172.21.15.37:0/3290313328 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37d0137500 0x7f37d013a650 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:33.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.815+0000 7f37cf7fe640 1 --2- 172.21.15.37:0/3290313328 >> v2:172.21.15.45:6812/2379918623 conn(0x7f37b0035e80 0x7f37b0038340 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f37c8006fd0 tx=0x7f37c8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:33.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.819+0000 7f37d5885640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:28:33.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.819+0000 7f37d5885640 1 --2- 172.21.15.37:0/3290313328 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37d013ab90 0x7f37d009e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:33.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.819+0000 7f37d5885640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:33.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.819+0000 7f37d5885640 1 -- 172.21.15.37:0/3290313328 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f37d013af70 con 0x7f37d013ab90 2024-04-26T00:28:33.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.819+0000 7f37ce7fc640 1 --2- 172.21.15.37:0/3290313328 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37d013ab90 0x7f37d009e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:33.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.819+0000 7f37ce7fc640 10 monclient: get_auth_request con 0x7f37d013ab90 auth_method 0 2024-04-26T00:28:33.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.819+0000 7f37ce7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:33.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.819+0000 7f37ce7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:33.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.819+0000 7f37ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:33.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.819+0000 7f37ce7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:33.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.819+0000 7f37ce7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:33.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.819+0000 7f37ce7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:33.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.819+0000 7f37ce7fc640 10 monclient(hunting): handle_auth_done global_id 84332 payload 824 2024-04-26T00:28:33.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.819+0000 7f37ce7fc640 1 -- 172.21.15.37:0/3290313328 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f37d0120ed0 con 0x7f37d013ab90 2024-04-26T00:28:33.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.823+0000 7f37ce7fc640 1 -- 172.21.15.37:0/3290313328 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37d013ab90 msgr2=0x7f37d009e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:33.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.823+0000 7f37ce7fc640 1 --2- 172.21.15.37:0/3290313328 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37d013ab90 0x7f37d009e080 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f37b801aa50 tx=0x7f37b8009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:33.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.823+0000 7f37c7fff640 1 -- 172.21.15.37:0/3290313328 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b80b0040 con 0x7f37d013ab90 2024-04-26T00:28:33.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.823+0000 7f37c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:33.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.823+0000 7f37c7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:28:33.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.823+0000 7f37c7fff640 10 monclient: dump: 2024-04-26T00:28:33.824 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:33.824 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:33.824 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.824 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.824 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:33.824 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:33.824 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:33.825 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:33.825 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:33.825 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:33.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.823+0000 7f37c7fff640 1 -- 172.21.15.37:0/3290313328 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f37b80bfd20 con 0x7f37d013ab90 2024-04-26T00:28:33.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.823+0000 7f37c7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:28:33.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.823+0000 7f37c7fff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:28:33.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.823+0000 7f37c7fff640 1 -- 172.21.15.37:0/3290313328 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37d013ab90 msgr2=0x7f37d009e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:33.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.823+0000 7f37c7fff640 1 --2- 172.21.15.37:0/3290313328 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37d013ab90 0x7f37d009e080 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f37b801aa50 tx=0x7f37b8009240 comp rx=0 tx=0).stop 2024-04-26T00:28:33.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.823+0000 7f37c7fff640 1 -- 172.21.15.37:0/3290313328 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37d013ab90 msgr2=0x7f37d009e080 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:28:33.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.823+0000 7f37c7fff640 1 --2- 172.21.15.37:0/3290313328 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37d013ab90 0x7f37d009e080 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f37b801aa50 tx=0x7f37b8009240 comp rx=0 tx=0).stop 2024-04-26T00:28:33.826 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:28:33.826 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:28:33.826 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:28:33.826 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:28:33.826 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 64, 2024-04-26T00:28:33.826 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 118758, 2024-04-26T00:28:33.826 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:28:33.826 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:28:33.826 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:28:33.826 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:28:33.826 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:33.827 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 6, 2024-04-26T00:28:33.827 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:28:33.827 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:28:33.827 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:28:33.827 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:28:33.827 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:28:33.827 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:28:33.827 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:28:33.827 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:28:33.827 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:28:33.827 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:28:33.827 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:28:33.827 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:28:33.827 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:28:33.828 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:28:33.828 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:33.828 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:28:33.828 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:28:33.828 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:28:33.828 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:28:33.828 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:28:33.828 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:28:33.828 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:28:33.828 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:28:33.828 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:28:33.828 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:28:33.828 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:28:33.829 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:28:33.829 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:28:33.829 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:33.829 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:33.829 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:28:33.829 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:28:33.829 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:28:33.829 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:28:33.829 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:28:33.829 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:28:33.829 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:28:33.829 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:28:33.829 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:28:33.830 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:28:33.830 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:28:33.830 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:28:33.830 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:28:33.830 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:28:33.830 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:28:33.830 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:28:33.830 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:28:33.830 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:28:33.830 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:28:33.830 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:28:33.831 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:28:33.831 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:28:33.831 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:28:33.831 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:28:33.831 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:28:33.831 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:28:33.831 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:28:33.831 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:28:33.831 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:33.831 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:28:33.831 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:33.831 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:28:33.831 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:33.832 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:28:33.832 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:28:33.832 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:28:33.832 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:28:33.832 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:33.832 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:28:33.832 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:28:33.832 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:33.832 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:33.832 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:33.832 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:28:33.832 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:28:33.832 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:33.832 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:33.833 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:33.833 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:33.833 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:28:33.833 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:28:33.833 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:28:33.833 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:28:33.833 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:28:33.833 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:33.833 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:33.833 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:28:33.833 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:28:33.833 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:28:33.833 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:28:33.833 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:33.833 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:28:33.834 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:28:33.834 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:33.834 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:33.834 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:33.834 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:28:33.834 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:28:33.834 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:33.834 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:33.834 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:33.834 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:33.834 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:28:33.835 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:28:33.835 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:28:33.835 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:28:33.835 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:28:33.835 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:33.835 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:33.835 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:28:33.835 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:28:33.835 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:28:33.835 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:28:33.836 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:33.836 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:28:33.836 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:28:33.836 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:33.836 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:33.836 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:33.836 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:28:33.836 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:28:33.836 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:33.836 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:33.836 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:33.836 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:33.836 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:28:33.836 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:28:33.837 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:28:33.837 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:28:33.837 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:28:33.837 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:33.837 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:33.837 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:33.837 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:28:33.837 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:28:33.837 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:33.837 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:28:33.837 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:28:33.837 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:28:33.837 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:33.837 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:33.837 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:28:33.838 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:33.838 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:28:33.838 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:28:33.838 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:28:33.838 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:33.838 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:33.838 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:28:33.838 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:33.838 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:28:33.838 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:28:33.838 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:28:33.838 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:33.838 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:33.838 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:28:33.839 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:33.839 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:28:33.839 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:28:33.839 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:28:33.839 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:33.839 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:33.839 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:33.839 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:28:33.839 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:28:33.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.823+0000 7f37d5885640 1 -- 172.21.15.37:0/3290313328 >> v2:172.21.15.45:6812/2379918623 conn(0x7f37b0035e80 msgr2=0x7f37b0038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:33.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.823+0000 7f37d5885640 1 --2- 172.21.15.37:0/3290313328 >> v2:172.21.15.45:6812/2379918623 conn(0x7f37b0035e80 0x7f37b0038340 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f37c8006fd0 tx=0x7f37c8008040 comp rx=0 tx=0).stop 2024-04-26T00:28:33.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.823+0000 7f37d5885640 10 monclient: shutdown 2024-04-26T00:28:33.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.823+0000 7f37d5885640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:28:33.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.823+0000 7f37d5885640 1 -- 172.21.15.37:0/3290313328 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f37d00a3d90 msgr2=0x7f37d01cee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:33.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.823+0000 7f37d5885640 1 --2- 172.21.15.37:0/3290313328 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f37d00a3d90 0x7f37d01cee80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f37c000c7c0 tx=0x7f37c000cc80 comp rx=0 tx=0).stop 2024-04-26T00:28:33.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.823+0000 7f37cf7fe640 1 -- 172.21.15.37:0/3290313328 reap_dead start 2024-04-26T00:28:33.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.823+0000 7f37cf7fe640 1 -- 172.21.15.37:0/3290313328 reap_dead start 2024-04-26T00:28:33.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.823+0000 7f37d5885640 1 -- 172.21.15.37:0/3290313328 shutdown_connections 2024-04-26T00:28:33.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.823+0000 7f37d5885640 1 -- 172.21.15.37:0/3290313328 >> 172.21.15.37:0/3290313328 conn(0x7f37d00aa0a0 msgr2=0x7f37d00a8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:28:33.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.823+0000 7f37d5885640 1 -- 172.21.15.37:0/3290313328 shutdown_connections 2024-04-26T00:28:33.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.823+0000 7f37d5885640 1 -- 172.21.15.37:0/3290313328 wait complete. 2024-04-26T00:28:33.842 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:28:33.842 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:28:33.842 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:28:33.924 INFO:tasks.rados.rados.0.smithi045.stdout:1474: done (3 left) 2024-04-26T00:28:33.924 INFO:tasks.rados.rados.0.smithi045.stdout:1475: done (2 left) 2024-04-26T00:28:33.924 INFO:tasks.rados.rados.0.smithi045.stdout:1476: done (1 left) 2024-04-26T00:28:33.924 INFO:tasks.rados.rados.0.smithi045.stdout:1477: done (0 left) 2024-04-26T00:28:33.924 INFO:tasks.rados.rados.0.smithi045.stdout:1478: snap_create 2024-04-26T00:28:33.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02d2427640 10 monclient: get_monmap_and_config 2024-04-26T00:28:33.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02d2427640 10 monclient: build_initial_monmap 2024-04-26T00:28:33.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02d2427640 10 monclient: monmap: 2024-04-26T00:28:33.960 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:28:33.960 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:33.960 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:28:33.975915+0000 2024-04-26T00:28:33.961 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:28:33.975915+0000 2024-04-26T00:28:33.961 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:28:33.961 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:33.961 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:28:33.961 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:28:33.961 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:28:33.961 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:28:33.961 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:28:33.961 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:33.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02d2427640 1 Processor -- start 2024-04-26T00:28:33.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02d2427640 1 -- start start 2024-04-26T00:28:33.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02d2427640 10 monclient: init 2024-04-26T00:28:33.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02d2427640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:33.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02d2427640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-04-26T00:28:33.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02d2427640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f02cc0a3d90 0x7f02cc09bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:33.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02d2427640 10 monclient(hunting): picked mon.noname-c con 0x7f02cc0a3d90 addr v2:172.21.15.45:3300/0 2024-04-26T00:28:33.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02d2427640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f02cc0a4170 addr v1:172.21.15.37:6789/0 2024-04-26T00:28:33.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02d2427640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f02cc13c3e0 0x7f02cc13e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:33.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02d2427640 10 monclient(hunting): picked mon.noname-a con 0x7f02cc13c3e0 addr v2:172.21.15.37:3300/0 2024-04-26T00:28:33.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02d2427640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02cc0a5150 con 0x7f02cc0a4170 2024-04-26T00:28:33.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02d2427640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:33.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02d2427640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f02cc09b280 con 0x7f02cc13c3e0 2024-04-26T00:28:33.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02d2427640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:33.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02d2427640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f02cc09b560 con 0x7f02cc0a3d90 2024-04-26T00:28:33.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02d2427640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:33.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02d2427640 10 monclient(hunting): authenticate will time out at 1495.281520s 2024-04-26T00:28:33.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02d09c4640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f02cc13c3e0 0x7f02cc13e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:33.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02d09c4640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f02cc13c3e0 0x7f02cc13e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:49920/0 (socket says 172.21.15.37:49920) 2024-04-26T00:28:33.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02d09c4640 1 -- 172.21.15.37:0/290902772 learned_addr learned my addr 172.21.15.37:0/290902772 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:28:33.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02d09c4640 10 monclient(hunting): get_auth_request con 0x7f02cc13c3e0 auth_method 0 2024-04-26T00:28:33.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02d09c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:33.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02d09c4640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:33.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02d09c4640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:33.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02d09c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:33.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02d09c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:33.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02d09c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:33.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02cbfff640 1 --2- 172.21.15.37:0/290902772 >> v2:172.21.15.45:3300/0 conn(0x7f02cc0a3d90 0x7f02cc09bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:33.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02cbfff640 10 monclient(hunting): get_auth_request con 0x7f02cc0a3d90 auth_method 0 2024-04-26T00:28:33.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:33.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02cbfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:33.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:33.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:33.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:33.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:33.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02cbfff640 10 monclient(hunting): handle_auth_done global_id 104125 payload 274 2024-04-26T00:28:33.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02cbfff640 1 -- 172.21.15.37:0/290902772 >> v2:172.21.15.37:3300/0 conn(0x7f02cc13c3e0 msgr2=0x7f02cc13e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:28:33.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02cbfff640 1 --2- 172.21.15.37:0/290902772 >> v2:172.21.15.37:3300/0 conn(0x7f02cc13c3e0 0x7f02cc13e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:33.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02cbfff640 1 -- 172.21.15.37:0/290902772 >> v1:172.21.15.37:6789/0 conn(0x7f02cc0a4170 legacy=0x7f02cc0a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:28:33.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02cbfff640 10 monclient: _finish_hunting 0 2024-04-26T00:28:33.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02cbfff640 1 monclient: found mon.noname-c 2024-04-26T00:28:33.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:33.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02cbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:28:33.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.974+0000 7f02cbfff640 1 -- 172.21.15.37:0/290902772 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02cc06dc60 con 0x7f02cc0a3d90 2024-04-26T00:28:33.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.978+0000 7f02cbfff640 1 -- 172.21.15.37:0/290902772 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02cc0a3d90 msgr2=0x7f02cc09bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:33.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.978+0000 7f02cbfff640 1 --2- 172.21.15.37:0/290902772 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02cc0a3d90 0x7f02cc09bf40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f02cc0a57e0 tx=0x7f02bc01c300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79ef1a7999abda96 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:33.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.978+0000 7f02caffd640 1 -- 172.21.15.37:0/290902772 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02bc0b0070 con 0x7f02cc0a3d90 2024-04-26T00:28:33.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.978+0000 7f02caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:33.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.978+0000 7f02caffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:28:33.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.978+0000 7f02caffd640 10 monclient: dump: 2024-04-26T00:28:33.970 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:33.970 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:33.970 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.970 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.970 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:33.970 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:33.970 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:33.970 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:33.970 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:33.970 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:33.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.978+0000 7f02caffd640 10 monclient: _finish_auth 0 2024-04-26T00:28:33.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.978+0000 7f02caffd640 10 monclient: _check_auth_tickets 2024-04-26T00:28:33.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.978+0000 7f02caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:33.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.978+0000 7f02caffd640 1 -- 172.21.15.37:0/290902772 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02bc0ab600 con 0x7f02cc0a3d90 2024-04-26T00:28:33.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.978+0000 7f02caffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:28:33.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.978+0000 7f02caffd640 1 -- 172.21.15.37:0/290902772 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02bc0ab9d0 con 0x7f02cc0a3d90 2024-04-26T00:28:33.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.978+0000 7f02caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:33.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.978+0000 7f02caffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:28:33.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.978+0000 7f02caffd640 10 monclient: dump: 2024-04-26T00:28:33.971 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:33.971 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:33.971 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.971 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.971 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:33.971 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:33.971 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:33.971 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:33.971 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:33.972 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:33.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.978+0000 7f02d2427640 5 monclient: authenticate success, global_id 104125 2024-04-26T00:28:33.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.978+0000 7f02d2427640 10 monclient: get_monmap_and_config success 2024-04-26T00:28:33.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.978+0000 7f02d2427640 4 set_mon_vals no callback set 2024-04-26T00:28:33.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.978+0000 7f02d2427640 10 monclient: shutdown 2024-04-26T00:28:33.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.978+0000 7f02d2427640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:28:33.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.978+0000 7f02d2427640 1 -- 172.21.15.37:0/290902772 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02cc0a3d90 msgr2=0x7f02cc09bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:33.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02d2427640 1 --2- 172.21.15.37:0/290902772 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02cc0a3d90 0x7f02cc09bf40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f02cc0a57e0 tx=0x7f02bc01c300 comp rx=0 tx=0).stop 2024-04-26T00:28:33.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02d2427640 1 -- 172.21.15.37:0/290902772 shutdown_connections 2024-04-26T00:28:33.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02d2427640 1 --2- 172.21.15.37:0/290902772 >> v2:172.21.15.37:3300/0 conn(0x7f02cc13c3e0 0x7f02cc13e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:33.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02d2427640 1 --2- 172.21.15.37:0/290902772 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02cc0a3d90 0x7f02cc09bf40 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:33.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02d2427640 1 -- 172.21.15.37:0/290902772 >> 172.21.15.37:0/290902772 conn(0x7f02cc0aa0a0 msgr2=0x7f02cc0a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:28:33.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02d2427640 1 -- 172.21.15.37:0/290902772 shutdown_connections 2024-04-26T00:28:33.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02d2427640 1 -- 172.21.15.37:0/290902772 wait complete. 2024-04-26T00:28:33.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02d2427640 10 monclient: build_initial_monmap 2024-04-26T00:28:33.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02d2427640 10 monclient: monmap: 2024-04-26T00:28:33.973 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:28:33.973 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:28:33.973 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:28:33.973 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:28:33.973 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:28:33.973 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:28:33.973 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:28:33.973 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:28:33.973 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:28:33.973 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:33.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02d2427640 1 Processor -- start 2024-04-26T00:28:33.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02d2427640 1 -- start start 2024-04-26T00:28:33.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02d2427640 10 monclient: init 2024-04-26T00:28:33.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02d2427640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:33.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02d2427640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:28:33.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02d2427640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02cc0a3d90 0x7f02cc1ceb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:33.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02d2427640 10 monclient(hunting): picked mon.noname-b con 0x7f02cc0a3d90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:28:33.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02d2427640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f02cc0a4170 0x7f02cc136f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:33.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02d2427640 10 monclient(hunting): picked mon.noname-a con 0x7f02cc0a4170 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:28:33.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02d2427640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f02cc13c3e0 0x7f02cc139ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:33.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02d2427640 10 monclient(hunting): picked mon.noname-c con 0x7f02cc13c3e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:28:33.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02d2427640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:33.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02d2427640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f02cc141740 con 0x7f02cc0a4170 2024-04-26T00:28:33.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02d2427640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:33.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02d2427640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f02cc09c690 con 0x7f02cc0a3d90 2024-04-26T00:28:33.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02d2427640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:33.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02d2427640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f02cc1418c0 con 0x7f02cc13c3e0 2024-04-26T00:28:33.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02d2427640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:33.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02d2427640 10 monclient(hunting): authenticate will time out at 1495.290149s 2024-04-26T00:28:33.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02cb7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f02cc0a4170 0x7f02cc136f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:33.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02d09c4640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f02cc13c3e0 0x7f02cc139ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:33.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02cb7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f02cc0a4170 0x7f02cc136f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:49932/0 (socket says 172.21.15.37:49932) 2024-04-26T00:28:33.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02cb7fe640 1 -- 172.21.15.37:0/4217744637 learned_addr learned my addr 172.21.15.37:0/4217744637 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:28:33.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02cb7fe640 10 monclient(hunting): get_auth_request con 0x7f02cc0a4170 auth_method 0 2024-04-26T00:28:33.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:33.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02cb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:33.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:33.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02d09c4640 10 monclient(hunting): get_auth_request con 0x7f02cc13c3e0 auth_method 0 2024-04-26T00:28:33.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02d09c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:33.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02d09c4640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:33.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02d09c4640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:33.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:33.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:33.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:33.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02d09c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:33.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02d09c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:33.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02d09c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:33.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02cbfff640 1 --2- 172.21.15.37:0/4217744637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02cc0a3d90 0x7f02cc1ceb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:33.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02cbfff640 10 monclient(hunting): get_auth_request con 0x7f02cc0a3d90 auth_method 0 2024-04-26T00:28:33.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:33.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02cbfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:33.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:33.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02cb7fe640 10 monclient(hunting): handle_auth_done global_id 94269 payload 824 2024-04-26T00:28:33.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02cb7fe640 1 -- 172.21.15.37:0/4217744637 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f02cc13c3e0 msgr2=0x7f02cc139ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:33.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02cb7fe640 1 --2- 172.21.15.37:0/4217744637 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f02cc13c3e0 0x7f02cc139ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:33.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02cb7fe640 1 -- 172.21.15.37:0/4217744637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02cc0a3d90 msgr2=0x7f02cc1ceb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:33.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02cb7fe640 1 --2- 172.21.15.37:0/4217744637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02cc0a3d90 0x7f02cc1ceb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:33.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02cb7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:28:33.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02cb7fe640 1 monclient: found mon.noname-a 2024-04-26T00:28:33.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:33.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02cb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:28:33.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02cb7fe640 1 -- 172.21.15.37:0/4217744637 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02cc09ada0 con 0x7f02cc0a4170 2024-04-26T00:28:33.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.982+0000 7f02cbfff640 1 --2- 172.21.15.37:0/4217744637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02cc0a3d90 0x7f02cc1ceb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:28:33.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02cb7fe640 1 -- 172.21.15.37:0/4217744637 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f02cc0a4170 msgr2=0x7f02cc136f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:33.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02cb7fe640 1 --2- 172.21.15.37:0/4217744637 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f02cc0a4170 0x7f02cc136f50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f02c000c480 tx=0x7f02c000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:33.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02c8ff9640 1 -- 172.21.15.37:0/4217744637 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c0012a40 con 0x7f02cc0a4170 2024-04-26T00:28:33.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02c8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:33.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02c8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:28:33.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02c8ff9640 10 monclient: dump: 2024-04-26T00:28:33.986 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:33.986 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:33.986 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.986 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.986 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:33.987 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:33.987 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:33.987 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:33.987 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:33.987 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:33.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02c8ff9640 10 monclient: _finish_auth 0 2024-04-26T00:28:33.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02c8ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:28:33.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02c8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:33.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02d2427640 5 monclient: authenticate success, global_id 94269 2024-04-26T00:28:33.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02c8ff9640 1 -- 172.21.15.37:0/4217744637 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02c0017070 con 0x7f02cc0a4170 2024-04-26T00:28:33.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02d2427640 10 monclient: _renew_subs 2024-04-26T00:28:33.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02d2427640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:28:33.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02d2427640 1 -- 172.21.15.37:0/4217744637 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02cc13cc10 con 0x7f02cc0a4170 2024-04-26T00:28:33.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02c8ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:28:33.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02c8ff9640 1 -- 172.21.15.37:0/4217744637 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c0012e40 con 0x7f02cc0a4170 2024-04-26T00:28:33.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02c8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:33.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02c8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:28:33.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02c8ff9640 10 monclient: dump: 2024-04-26T00:28:33.989 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:33.989 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:33.989 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.989 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.989 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:33.990 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:33.990 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:33.990 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:33.990 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:33.990 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:33.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02ca7fc640 4 set_mon_vals no callback set 2024-04-26T00:28:33.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02d2427640 10 monclient: _renew_subs 2024-04-26T00:28:33.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02d2427640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:28:33.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02d2427640 1 -- 172.21.15.37:0/4217744637 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02cc11fed0 con 0x7f02cc0a4170 2024-04-26T00:28:33.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02d2427640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:33.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02d2427640 1 --2- 172.21.15.37:0/4217744637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02cc137490 0x7f02cc13a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:33.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02d2427640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:33.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02d2427640 1 -- 172.21.15.37:0/4217744637 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f02cc04f510 con 0x7f02cc137490 2024-04-26T00:28:33.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02c8ff9640 1 -- 172.21.15.37:0/4217744637 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f02c001c460 con 0x7f02cc0a4170 2024-04-26T00:28:33.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02cbfff640 1 --2- 172.21.15.37:0/4217744637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02cc137490 0x7f02cc13a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:33.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02cbfff640 10 monclient: get_auth_request con 0x7f02cc137490 auth_method 0 2024-04-26T00:28:33.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:33.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02cbfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:33.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:33.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02c8ff9640 1 --2- 172.21.15.37:0/4217744637 >> v2:172.21.15.45:6812/2379918623 conn(0x7f02b8035e80 0x7f02b8038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:33.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02c8ff9640 1 -- 172.21.15.37:0/4217744637 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(339..339 src has 177..339) ==== 3947+0+0 (secure 0 0 0) 0x7f02c00488c0 con 0x7f02cc0a4170 2024-04-26T00:28:33.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:33.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:33.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:33.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02d09c4640 1 --2- 172.21.15.37:0/4217744637 >> v2:172.21.15.45:6812/2379918623 conn(0x7f02b8035e80 0x7f02b8038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:33.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02d09c4640 10 monclient: get_auth_request con 0x7f02b8035e80 auth_method 0 2024-04-26T00:28:33.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02cbfff640 10 monclient(hunting): handle_auth_done global_id 104131 payload 824 2024-04-26T00:28:33.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.986+0000 7f02cbfff640 1 -- 172.21.15.37:0/4217744637 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f02cc0008e0 con 0x7f02cc137490 2024-04-26T00:28:33.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.990+0000 7f02cbfff640 1 -- 172.21.15.37:0/4217744637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02cc137490 msgr2=0x7f02cc13a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:33.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.990+0000 7f02cbfff640 1 --2- 172.21.15.37:0/4217744637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02cc137490 0x7f02cc13a420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f02cc09d140 tx=0x7f02bc01ab40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:33.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.990+0000 7f02c8ff9640 1 -- 172.21.15.37:0/4217744637 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02bc0b0040 con 0x7f02cc137490 2024-04-26T00:28:33.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.990+0000 7f02c8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:33.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.990+0000 7f02c8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:28:33.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.990+0000 7f02c8ff9640 10 monclient: dump: 2024-04-26T00:28:33.994 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:33.994 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:33.994 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.994 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.994 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:33.994 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:33.995 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:33.995 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:33.995 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:33.995 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:33.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.990+0000 7f02d09c4640 1 --2- 172.21.15.37:0/4217744637 >> v2:172.21.15.45:6812/2379918623 conn(0x7f02b8035e80 0x7f02b8038340 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f02cc137c70 tx=0x7f02c400b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:33.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.990+0000 7f02c8ff9640 1 -- 172.21.15.37:0/4217744637 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f02bc0b6670 con 0x7f02cc137490 2024-04-26T00:28:33.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.990+0000 7f02c8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:33.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.990+0000 7f02c8ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:28:33.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.990+0000 7f02c8ff9640 1 -- 172.21.15.37:0/4217744637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02cc137490 msgr2=0x7f02cc13a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:33.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.990+0000 7f02c8ff9640 1 --2- 172.21.15.37:0/4217744637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02cc137490 0x7f02cc13a420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f02cc09d140 tx=0x7f02bc01ab40 comp rx=0 tx=0).stop 2024-04-26T00:28:33.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.990+0000 7f02c8ff9640 1 -- 172.21.15.37:0/4217744637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02cc137490 msgr2=0x7f02cc13a420 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:28:33.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.990+0000 7f02c8ff9640 1 --2- 172.21.15.37:0/4217744637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02cc137490 0x7f02cc13a420 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f02cc09d140 tx=0x7f02bc01ab40 comp rx=0 tx=0).stop 2024-04-26T00:28:33.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.998+0000 7f02d2427640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:28:33.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.998+0000 7f02d2427640 1 --2- 172.21.15.37:0/4217744637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02cc13a960 0x7f02cc09e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:33.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.998+0000 7f02d2427640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:33.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.998+0000 7f02d2427640 1 -- 172.21.15.37:0/4217744637 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f02cc13ad70 con 0x7f02cc13a960 2024-04-26T00:28:33.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.998+0000 7f02cbfff640 1 --2- 172.21.15.37:0/4217744637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02cc13a960 0x7f02cc09e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:33.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.998+0000 7f02cbfff640 10 monclient: get_auth_request con 0x7f02cc13a960 auth_method 0 2024-04-26T00:28:33.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.998+0000 7f02cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:33.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.998+0000 7f02cbfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:33.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.998+0000 7f02cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:33.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.998+0000 7f02cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:33.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.998+0000 7f02cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:33.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.998+0000 7f02cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:33.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.998+0000 7f02cbfff640 10 monclient(hunting): handle_auth_done global_id 104134 payload 824 2024-04-26T00:28:33.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:33.998+0000 7f02cbfff640 1 -- 172.21.15.37:0/4217744637 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f02cc1cf960 con 0x7f02cc13a960 2024-04-26T00:28:33.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.002+0000 7f02cbfff640 1 -- 172.21.15.37:0/4217744637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02cc13a960 msgr2=0x7f02cc09e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:33.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.002+0000 7f02cbfff640 1 --2- 172.21.15.37:0/4217744637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02cc13a960 0x7f02cc09e180 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f02bc019f30 tx=0x7f02bc0190a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:33.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.002+0000 7f02c8ff9640 1 -- 172.21.15.37:0/4217744637 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02bc0b0040 con 0x7f02cc13a960 2024-04-26T00:28:33.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.002+0000 7f02c8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:33.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.002+0000 7f02c8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:28:33.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.002+0000 7f02c8ff9640 10 monclient: dump: 2024-04-26T00:28:33.999 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:33.999 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:33.999 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.999 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:33.999 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:33.999 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:33.999 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:33.999 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:34.000 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:34.000 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:34.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.002+0000 7f02c8ff9640 1 -- 172.21.15.37:0/4217744637 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7f02bc0bfd70 con 0x7f02cc13a960 2024-04-26T00:28:34.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.002+0000 7f02c8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:28:34.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.002+0000 7f02c8ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:28:34.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.002+0000 7f02c8ff9640 1 -- 172.21.15.37:0/4217744637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02cc13a960 msgr2=0x7f02cc09e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:34.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.002+0000 7f02c8ff9640 1 --2- 172.21.15.37:0/4217744637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02cc13a960 0x7f02cc09e180 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f02bc019f30 tx=0x7f02bc0190a0 comp rx=0 tx=0).stop 2024-04-26T00:28:34.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.002+0000 7f02c8ff9640 1 -- 172.21.15.37:0/4217744637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02cc13a960 msgr2=0x7f02cc09e180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:28:34.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.002+0000 7f02c8ff9640 1 --2- 172.21.15.37:0/4217744637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02cc13a960 0x7f02cc09e180 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f02bc019f30 tx=0x7f02bc0190a0 comp rx=0 tx=0).stop 2024-04-26T00:28:34.000 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:28:34.000 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:28:34.001 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:28:34.001 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:28:34.001 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 64, 2024-04-26T00:28:34.001 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 21203, 2024-04-26T00:28:34.001 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:28:34.001 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:28:34.001 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:28:34.001 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:28:34.001 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:34.001 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 6, 2024-04-26T00:28:34.001 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:28:34.001 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:28:34.001 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:28:34.001 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:28:34.002 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:28:34.002 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:28:34.002 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:28:34.002 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:28:34.002 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:28:34.002 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:28:34.002 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:28:34.002 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:28:34.002 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:28:34.002 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:28:34.002 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:34.002 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:28:34.002 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:28:34.002 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:28:34.003 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:28:34.003 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:28:34.003 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:28:34.003 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:28:34.003 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:28:34.003 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:28:34.003 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:28:34.003 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:28:34.003 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:28:34.003 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:28:34.003 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:34.003 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:34.003 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:28:34.004 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:28:34.004 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:28:34.004 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:28:34.004 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:28:34.004 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:28:34.004 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:28:34.004 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:28:34.004 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:28:34.004 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:28:34.004 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:28:34.004 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:28:34.004 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:28:34.004 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:28:34.004 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:28:34.004 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:28:34.005 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:28:34.005 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:28:34.005 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:28:34.005 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:28:34.005 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:28:34.005 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:28:34.005 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:28:34.005 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:28:34.005 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:28:34.005 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:28:34.005 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:28:34.005 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:28:34.005 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:34.005 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:28:34.005 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:34.006 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:28:34.006 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:34.006 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:28:34.006 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:28:34.006 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:28:34.006 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:28:34.006 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:34.006 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:28:34.006 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:28:34.006 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:34.006 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:34.006 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:34.006 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:28:34.006 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:28:34.007 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:34.007 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:34.007 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:34.007 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:34.007 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:28:34.007 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:28:34.007 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:28:34.007 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:28:34.007 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:28:34.007 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:34.007 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:34.007 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:28:34.007 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:28:34.007 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:28:34.008 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:28:34.008 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:34.008 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:28:34.008 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:28:34.008 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:34.008 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:34.008 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:34.008 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:28:34.008 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:28:34.008 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:34.009 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:34.009 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:34.009 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:34.009 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:28:34.009 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:28:34.009 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:28:34.009 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:28:34.009 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:28:34.009 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:34.009 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:34.009 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:28:34.009 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:28:34.009 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:28:34.010 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:28:34.010 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:34.010 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:28:34.010 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:28:34.010 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:34.010 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:34.010 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:34.010 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:28:34.010 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:28:34.010 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:34.010 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:34.010 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:34.010 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:34.011 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:28:34.011 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:28:34.011 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:28:34.011 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:28:34.011 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:28:34.011 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:34.011 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:34.011 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:34.011 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:28:34.011 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:28:34.011 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:34.011 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:28:34.011 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:28:34.011 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:28:34.012 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:34.012 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:34.012 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:28:34.012 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:34.012 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:28:34.012 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:28:34.012 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:28:34.012 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:34.012 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:34.012 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:34.013 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:28:34.013 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:28:34.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.002+0000 7f02d2427640 1 -- 172.21.15.37:0/4217744637 >> v2:172.21.15.45:6812/2379918623 conn(0x7f02b8035e80 msgr2=0x7f02b8038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:34.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.002+0000 7f02d2427640 1 --2- 172.21.15.37:0/4217744637 >> v2:172.21.15.45:6812/2379918623 conn(0x7f02b8035e80 0x7f02b8038340 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f02cc137c70 tx=0x7f02c400b040 comp rx=0 tx=0).stop 2024-04-26T00:28:34.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.002+0000 7f02d2427640 10 monclient: shutdown 2024-04-26T00:28:34.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.002+0000 7f02d2427640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:28:34.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.002+0000 7f02d2427640 1 -- 172.21.15.37:0/4217744637 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f02cc0a4170 msgr2=0x7f02cc136f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:34.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.002+0000 7f02d2427640 1 --2- 172.21.15.37:0/4217744637 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f02cc0a4170 0x7f02cc136f50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f02c000c480 tx=0x7f02c000c940 comp rx=0 tx=0).stop 2024-04-26T00:28:34.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.002+0000 7f02d09c4640 1 -- 172.21.15.37:0/4217744637 reap_dead start 2024-04-26T00:28:34.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.002+0000 7f02d09c4640 1 -- 172.21.15.37:0/4217744637 reap_dead start 2024-04-26T00:28:34.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.002+0000 7f02d2427640 1 -- 172.21.15.37:0/4217744637 shutdown_connections 2024-04-26T00:28:34.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.002+0000 7f02d2427640 1 -- 172.21.15.37:0/4217744637 >> 172.21.15.37:0/4217744637 conn(0x7f02cc0aa0a0 msgr2=0x7f02cc0a8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:28:34.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.002+0000 7f02d2427640 1 -- 172.21.15.37:0/4217744637 shutdown_connections 2024-04-26T00:28:34.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.002+0000 7f02d2427640 1 -- 172.21.15.37:0/4217744637 wait complete. 2024-04-26T00:28:34.015 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:28:34.016 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-04-26T00:28:34.016 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-26T00:28:34.016 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-04-26T00:28:34.016 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-04-26T00:28:34.016 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-26T00:28:34.016 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T00:28:34.036 INFO:tasks.ceph.mon.b:Stopped 2024-04-26T00:28:34.036 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-26T00:28:34.036 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:28:34.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.162+0000 7f6882cbb640 10 monclient: get_monmap_and_config 2024-04-26T00:28:34.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.162+0000 7f6882cbb640 10 monclient: build_initial_monmap 2024-04-26T00:28:34.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.162+0000 7f6882cbb640 10 monclient: monmap: 2024-04-26T00:28:34.151 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:28:34.151 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:34.151 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:28:34.166103+0000 2024-04-26T00:28:34.151 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:28:34.166103+0000 2024-04-26T00:28:34.151 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:28:34.151 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:34.151 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:28:34.151 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:28:34.151 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:28:34.151 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:28:34.151 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:28:34.151 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:34.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.162+0000 7f6882cbb640 1 Processor -- start 2024-04-26T00:28:34.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.162+0000 7f6882cbb640 1 -- start start 2024-04-26T00:28:34.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.162+0000 7f6882cbb640 10 monclient: init 2024-04-26T00:28:34.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.162+0000 7f6882cbb640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:34.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.162+0000 7f6882cbb640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-04-26T00:28:34.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.162+0000 7f6882cbb640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f687c0a0260 0x7f687c0a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:34.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.162+0000 7f6882cbb640 10 monclient(hunting): picked mon.noname-a con 0x7f687c0a0260 addr v2:172.21.15.37:3300/0 2024-04-26T00:28:34.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.162+0000 7f6882cbb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f687c134b30 addr v1:172.21.15.37:6789/0 2024-04-26T00:28:34.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.162+0000 7f6882cbb640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f687c135660 0x7f687c13ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:34.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.162+0000 7f6882cbb640 10 monclient(hunting): picked mon.noname-b con 0x7f687c135660 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:28:34.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.162+0000 7f6882cbb640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f687c0a1130 con 0x7f687c134b30 2024-04-26T00:28:34.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.162+0000 7f6882cbb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:34.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.162+0000 7f6882cbb640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f687c09ec10 con 0x7f687c0a0260 2024-04-26T00:28:34.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.162+0000 7f6882cbb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:34.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.162+0000 7f6882cbb640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f687c09f0f0 con 0x7f687c135660 2024-04-26T00:28:34.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.162+0000 7f6882cbb640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:34.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.162+0000 7f6882cbb640 10 monclient(hunting): authenticate will time out at 1495.471935s 2024-04-26T00:28:34.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.166+0000 7f6880a57640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f687c0a0260 0x7f687c0a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:34.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.166+0000 7f6880a57640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f687c0a0260 0x7f687c0a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:49942/0 (socket says 172.21.15.37:49942) 2024-04-26T00:28:34.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.166+0000 7f6880a57640 1 -- 172.21.15.37:0/2862079569 learned_addr learned my addr 172.21.15.37:0/2862079569 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:28:34.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.166+0000 7f6881258640 1 --2- 172.21.15.37:0/2862079569 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f687c135660 0x7f687c13ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:34.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.166+0000 7f6880a57640 10 monclient(hunting): get_auth_request con 0x7f687c0a0260 auth_method 0 2024-04-26T00:28:34.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.166+0000 7f6880a57640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:34.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.166+0000 7f6880a57640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:34.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.166+0000 7f6880a57640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:34.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.166+0000 7f6881258640 10 monclient(hunting): get_auth_request con 0x7f687c135660 auth_method 0 2024-04-26T00:28:34.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.166+0000 7f6881258640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:34.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.166+0000 7f6881258640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:34.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.166+0000 7f6881258640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:34.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.166+0000 7f6880a57640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:34.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.166+0000 7f6880a57640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:34.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.166+0000 7f6880a57640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:34.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.166+0000 7f6881258640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:34.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.166+0000 7f6881258640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:34.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.166+0000 7f6881258640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:34.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.166+0000 7f6881258640 10 monclient(hunting): handle_auth_done global_id 84338 payload 274 2024-04-26T00:28:34.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.166+0000 7f6881258640 1 -- 172.21.15.37:0/2862079569 >> v2:172.21.15.37:3300/0 conn(0x7f687c0a0260 msgr2=0x7f687c0a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:28:34.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.166+0000 7f6881258640 1 --2- 172.21.15.37:0/2862079569 >> v2:172.21.15.37:3300/0 conn(0x7f687c0a0260 0x7f687c0a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:34.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.166+0000 7f6881258640 1 -- 172.21.15.37:0/2862079569 >> v1:172.21.15.37:6789/0 conn(0x7f687c134b30 legacy=0x7f687c134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:28:34.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.166+0000 7f6881258640 10 monclient: _finish_hunting 0 2024-04-26T00:28:34.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.166+0000 7f6881258640 1 monclient: found mon.noname-b 2024-04-26T00:28:34.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.166+0000 7f6881258640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:34.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.166+0000 7f6881258640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:28:34.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.166+0000 7f6881258640 1 -- 172.21.15.37:0/2862079569 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f687c06cff0 con 0x7f687c135660 2024-04-26T00:28:34.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6881258640 1 -- 172.21.15.37:0/2862079569 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f687c135660 msgr2=0x7f687c13ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:34.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6881258640 1 --2- 172.21.15.37:0/2862079569 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f687c135660 0x7f687c13ce70 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6874009d90 tx=0x7f687401c610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c468248e10db4d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:34.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f687b7fe640 1 -- 172.21.15.37:0/2862079569 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68740a6040 con 0x7f687c135660 2024-04-26T00:28:34.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f687b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:34.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f687b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:28:34.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f687b7fe640 10 monclient: dump: 2024-04-26T00:28:34.164 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:34.164 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:34.164 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:34.164 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:34.164 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:34.165 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:34.165 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:34.165 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:34.165 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:34.165 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:34.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f687b7fe640 10 monclient: _finish_auth 0 2024-04-26T00:28:34.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f687b7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:28:34.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f687b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:34.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6882cbb640 5 monclient: authenticate success, global_id 84338 2024-04-26T00:28:34.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f687b7fe640 1 -- 172.21.15.37:0/2862079569 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68740af070 con 0x7f687c135660 2024-04-26T00:28:34.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6882cbb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:28:34.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f687b7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:28:34.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6882cbb640 10 monclient: get_monmap_and_config success 2024-04-26T00:28:34.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6882cbb640 4 set_mon_vals no callback set 2024-04-26T00:28:34.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f687b7fe640 1 -- 172.21.15.37:0/2862079569 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68740aa860 con 0x7f687c135660 2024-04-26T00:28:34.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f687b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:34.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f687b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:28:34.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f687b7fe640 10 monclient: dump: 2024-04-26T00:28:34.167 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:34.167 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:34.167 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:34.167 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:34.167 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:34.167 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:34.167 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:34.167 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:34.167 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:34.167 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:34.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6882cbb640 10 monclient: shutdown 2024-04-26T00:28:34.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6882cbb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:28:34.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6882cbb640 1 -- 172.21.15.37:0/2862079569 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f687c135660 msgr2=0x7f687c13ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6882cbb640 1 --2- 172.21.15.37:0/2862079569 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f687c135660 0x7f687c13ce70 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6874009d90 tx=0x7f687401c610 comp rx=0 tx=0).stop 2024-04-26T00:28:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6882cbb640 1 -- 172.21.15.37:0/2862079569 shutdown_connections 2024-04-26T00:28:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6882cbb640 1 --2- 172.21.15.37:0/2862079569 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f687c135660 0x7f687c13ce70 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6882cbb640 1 --2- 172.21.15.37:0/2862079569 >> v2:172.21.15.37:3300/0 conn(0x7f687c0a0260 0x7f687c0a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6882cbb640 1 -- 172.21.15.37:0/2862079569 >> 172.21.15.37:0/2862079569 conn(0x7f687c0a1ad0 msgr2=0x7f687c0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:28:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6882cbb640 1 -- 172.21.15.37:0/2862079569 shutdown_connections 2024-04-26T00:28:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6882cbb640 1 -- 172.21.15.37:0/2862079569 wait complete. 2024-04-26T00:28:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6882cbb640 10 monclient: build_initial_monmap 2024-04-26T00:28:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6882cbb640 10 monclient: monmap: 2024-04-26T00:28:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:28:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:28:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:28:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:28:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:28:34.169 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:28:34.169 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:28:34.169 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:28:34.169 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:28:34.169 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:34.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6882cbb640 1 Processor -- start 2024-04-26T00:28:34.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6882cbb640 1 -- start start 2024-04-26T00:28:34.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6882cbb640 10 monclient: init 2024-04-26T00:28:34.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6882cbb640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:34.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6882cbb640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:28:34.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6882cbb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f687c0a0260 0x7f687c132220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:34.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6882cbb640 10 monclient(hunting): picked mon.noname-b con 0x7f687c0a0260 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:28:34.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6882cbb640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f687c134b30 0x7f687c09c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:34.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6882cbb640 10 monclient(hunting): picked mon.noname-a con 0x7f687c134b30 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:28:34.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6882cbb640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f687c135660 0x7f687c09ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:34.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6882cbb640 10 monclient(hunting): picked mon.noname-c con 0x7f687c135660 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:28:34.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6882cbb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:34.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6882cbb640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f687c0a0010 con 0x7f687c134b30 2024-04-26T00:28:34.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6882cbb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:34.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6882cbb640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f687c09fe90 con 0x7f687c0a0260 2024-04-26T00:28:34.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6882cbb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:34.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6882cbb640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f687c13fbd0 con 0x7f687c135660 2024-04-26T00:28:34.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6882cbb640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:34.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6882cbb640 10 monclient(hunting): authenticate will time out at 1495.479022s 2024-04-26T00:28:34.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6881258640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f687c135660 0x7f687c09ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:34.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f687bfff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f687c134b30 0x7f687c09c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:34.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6881258640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f687c135660 0x7f687c09ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:48696/0 (socket says 172.21.15.37:48696) 2024-04-26T00:28:34.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6881258640 1 -- 172.21.15.37:0/700108911 learned_addr learned my addr 172.21.15.37:0/700108911 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:28:34.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6881258640 10 monclient(hunting): get_auth_request con 0x7f687c135660 auth_method 0 2024-04-26T00:28:34.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6881258640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:34.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6881258640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:34.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6881258640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:34.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6880a57640 1 -- 172.21.15.37:0/700108911 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f687c0a0260 msgr2=0x7f687c132220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:28:34.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f687bfff640 10 monclient(hunting): get_auth_request con 0x7f687c134b30 auth_method 0 2024-04-26T00:28:34.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f687bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:34.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f687bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:34.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f687bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:34.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6880a57640 1 --2- 172.21.15.37:0/700108911 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f687c0a0260 0x7f687c132220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:28:34.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6881258640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:34.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6881258640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:34.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6881258640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:34.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f687bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:34.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f687bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:34.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f687bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:34.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6881258640 10 monclient(hunting): handle_auth_done global_id 84341 payload 824 2024-04-26T00:28:34.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6881258640 1 -- 172.21.15.37:0/700108911 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f687c0a0260 msgr2=0x7f687c132220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:28:34.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6881258640 1 --2- 172.21.15.37:0/700108911 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f687c0a0260 0x7f687c132220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:34.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6881258640 1 -- 172.21.15.37:0/700108911 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f687c134b30 msgr2=0x7f687c09c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:34.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6881258640 1 --2- 172.21.15.37:0/700108911 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f687c134b30 0x7f687c09c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:34.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6881258640 10 monclient: _finish_hunting 0 2024-04-26T00:28:34.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6881258640 1 monclient: found mon.noname-c 2024-04-26T00:28:34.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6881258640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:34.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6881258640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:28:34.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f6881258640 1 -- 172.21.15.37:0/700108911 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f687c13b2c0 con 0x7f687c135660 2024-04-26T00:28:34.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.170+0000 7f687bfff640 1 --2- 172.21.15.37:0/700108911 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f687c134b30 0x7f687c09c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:28:34.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.174+0000 7f6881258640 1 -- 172.21.15.37:0/700108911 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f687c135660 msgr2=0x7f687c09ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:34.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.174+0000 7f6881258640 1 --2- 172.21.15.37:0/700108911 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f687c135660 0x7f687c09ca10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f687401acc0 tx=0x7f68740193b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:34.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.174+0000 7f68797fa640 1 -- 172.21.15.37:0/700108911 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68740aab30 con 0x7f687c135660 2024-04-26T00:28:34.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.174+0000 7f68797fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:34.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.174+0000 7f68797fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:28:34.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.174+0000 7f68797fa640 10 monclient: dump: 2024-04-26T00:28:34.173 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:34.173 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:34.173 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:34.173 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:34.173 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:34.173 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:34.173 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:34.173 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:34.174 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:34.174 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:34.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.174+0000 7f68797fa640 10 monclient: _finish_auth 0 2024-04-26T00:28:34.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.174+0000 7f68797fa640 10 monclient: _check_auth_tickets 2024-04-26T00:28:34.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.174+0000 7f68797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:34.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.174+0000 7f68797fa640 1 -- 172.21.15.37:0/700108911 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68740aacd0 con 0x7f687c135660 2024-04-26T00:28:34.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.174+0000 7f6882cbb640 5 monclient: authenticate success, global_id 84341 2024-04-26T00:28:34.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.174+0000 7f6882cbb640 10 monclient: _renew_subs 2024-04-26T00:28:34.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.174+0000 7f6882cbb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:28:34.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.174+0000 7f6882cbb640 1 -- 172.21.15.37:0/700108911 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f687c13a0f0 con 0x7f687c135660 2024-04-26T00:28:34.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.174+0000 7f68797fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:28:34.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.174+0000 7f68797fa640 1 -- 172.21.15.37:0/700108911 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68740be690 con 0x7f687c135660 2024-04-26T00:28:34.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.174+0000 7f68797fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:34.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.174+0000 7f68797fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:28:34.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.174+0000 7f68797fa640 10 monclient: dump: 2024-04-26T00:28:34.175 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:34.175 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:34.175 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:34.175 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:34.175 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:34.175 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:34.175 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:34.176 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:34.176 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:34.176 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:34.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.174+0000 7f6882cbb640 10 monclient: _renew_subs 2024-04-26T00:28:34.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.174+0000 7f6882cbb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:28:34.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.174+0000 7f687a7fc640 4 set_mon_vals no callback set 2024-04-26T00:28:34.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.174+0000 7f6882cbb640 1 -- 172.21.15.37:0/700108911 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f687c13b5a0 con 0x7f687c135660 2024-04-26T00:28:34.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.174+0000 7f6882cbb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:34.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.174+0000 7f6882cbb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:34.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.174+0000 7f6882cbb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:28:34.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.174+0000 7f68797fa640 1 -- 172.21.15.37:0/700108911 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f68740039a0 con 0x7f687c135660 2024-04-26T00:28:34.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.174+0000 7f6882cbb640 1 -- 172.21.15.37:0/700108911 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f687c066f40 con 0x7f687c135660 2024-04-26T00:28:34.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.174+0000 7f68797fa640 1 --2- 172.21.15.37:0/700108911 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6868035e80 0x7f6868038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:34.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.174+0000 7f68797fa640 1 -- 172.21.15.37:0/700108911 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(339..339 src has 177..339) ==== 3947+0+0 (secure 0 0 0) 0x7f68740e3500 con 0x7f687c135660 2024-04-26T00:28:34.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.174+0000 7f6880a57640 1 --2- 172.21.15.37:0/700108911 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6868035e80 0x7f6868038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:34.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.174+0000 7f6880a57640 10 monclient: get_auth_request con 0x7f6868035e80 auth_method 0 2024-04-26T00:28:34.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.178+0000 7f6880a57640 1 --2- 172.21.15.37:0/700108911 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6868035e80 0x7f6868038340 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f686c006c30 tx=0x7f686c009240 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:34.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.182+0000 7f68797fa640 1 -- 172.21.15.37:0/700108911 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f687c0a1130 con 0x7f687c135660 2024-04-26T00:28:34.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.182+0000 7f68797fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:34.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.182+0000 7f68797fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:28:34.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.402+0000 7f6882cbb640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:28:34.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.402+0000 7f6882cbb640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:28:34.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.402+0000 7f6882cbb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:28:34.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.402+0000 7f6882cbb640 1 -- 172.21.15.37:0/700108911 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f687c134840 con 0x7f687c135660 2024-04-26T00:28:34.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.402+0000 7f68797fa640 1 -- 172.21.15.37:0/700108911 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f68740e56e0 con 0x7f687c135660 2024-04-26T00:28:34.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.402+0000 7f68797fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:28:34.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.402+0000 7f68797fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:28:34.391 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":64,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:28:34.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.406+0000 7f6882cbb640 1 -- 172.21.15.37:0/700108911 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6868035e80 msgr2=0x7f6868038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:34.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.406+0000 7f6882cbb640 1 --2- 172.21.15.37:0/700108911 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6868035e80 0x7f6868038340 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f686c006c30 tx=0x7f686c009240 comp rx=0 tx=0).stop 2024-04-26T00:28:34.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.406+0000 7f6882cbb640 10 monclient: shutdown 2024-04-26T00:28:34.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.406+0000 7f6882cbb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:28:34.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.406+0000 7f6882cbb640 1 -- 172.21.15.37:0/700108911 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f687c135660 msgr2=0x7f687c09ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:34.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.406+0000 7f6882cbb640 1 --2- 172.21.15.37:0/700108911 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f687c135660 0x7f687c09ca10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f687401acc0 tx=0x7f68740193b0 comp rx=0 tx=0).stop 2024-04-26T00:28:34.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.406+0000 7f6882cbb640 1 -- 172.21.15.37:0/700108911 shutdown_connections 2024-04-26T00:28:34.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.406+0000 7f6882cbb640 1 --2- 172.21.15.37:0/700108911 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6868035e80 0x7f6868038340 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:34.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.406+0000 7f6882cbb640 1 --2- 172.21.15.37:0/700108911 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f687c135660 0x7f687c09ca10 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:34.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.406+0000 7f6882cbb640 1 --2- 172.21.15.37:0/700108911 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f687c134b30 0x7f687c09c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:34.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.406+0000 7f6882cbb640 1 --2- 172.21.15.37:0/700108911 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f687c0a0260 0x7f687c132220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:34.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.406+0000 7f6882cbb640 1 -- 172.21.15.37:0/700108911 >> 172.21.15.37:0/700108911 conn(0x7f687c0a1ad0 msgr2=0x7f687c0a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:28:34.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.406+0000 7f6882cbb640 1 -- 172.21.15.37:0/700108911 shutdown_connections 2024-04-26T00:28:34.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:34.406+0000 7f6882cbb640 1 -- 172.21.15.37:0/700108911 wait complete. 2024-04-26T00:28:37.409 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:28:37.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe1359e640 10 monclient: get_monmap_and_config 2024-04-26T00:28:37.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe1359e640 10 monclient: build_initial_monmap 2024-04-26T00:28:37.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe1359e640 10 monclient: monmap: 2024-04-26T00:28:37.539 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:28:37.539 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:37.539 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:28:37.553575+0000 2024-04-26T00:28:37.539 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:28:37.553575+0000 2024-04-26T00:28:37.539 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:28:37.539 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:37.539 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:28:37.540 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:28:37.540 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:28:37.540 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:28:37.540 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:28:37.540 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:37.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe1359e640 1 Processor -- start 2024-04-26T00:28:37.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe1359e640 1 -- start start 2024-04-26T00:28:37.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe1359e640 10 monclient: init 2024-04-26T00:28:37.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe1359e640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:37.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe1359e640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-04-26T00:28:37.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe1359e640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7efe1409fa40 0x7efe1409fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:37.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe1359e640 10 monclient(hunting): picked mon.noname-a con 0x7efe1409fa40 addr v2:172.21.15.37:3300/0 2024-04-26T00:28:37.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe1359e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efe140a0360 0x7efe140a0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:37.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe1359e640 10 monclient(hunting): picked mon.noname-b con 0x7efe140a0360 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:28:37.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe1359e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efe140a0cb0 addr v1:172.21.15.37:6789/0 2024-04-26T00:28:37.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe1359e640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe140a18d0 con 0x7efe140a0cb0 2024-04-26T00:28:37.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe1359e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:37.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe1359e640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7efe140a1140 con 0x7efe1409fa40 2024-04-26T00:28:37.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe1359e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:37.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe1359e640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7efe1409d650 con 0x7efe140a0360 2024-04-26T00:28:37.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe1359e640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:37.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe1359e640 10 monclient(hunting): authenticate will time out at 1498.859450s 2024-04-26T00:28:37.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe11d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efe140a0360 0x7efe140a0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:37.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe1259c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7efe1409fa40 0x7efe1409fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:37.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe11d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efe140a0360 0x7efe140a0770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:48706/0 (socket says 172.21.15.37:48706) 2024-04-26T00:28:37.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe11d9b640 1 -- 172.21.15.37:0/3128646269 learned_addr learned my addr 172.21.15.37:0/3128646269 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:28:37.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe11d9b640 10 monclient(hunting): get_auth_request con 0x7efe140a0360 auth_method 0 2024-04-26T00:28:37.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe11d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:37.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe1259c640 1 --2- 172.21.15.37:0/3128646269 >> v2:172.21.15.37:3300/0 conn(0x7efe1409fa40 0x7efe1409fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:49960/0 (socket says 172.21.15.37:49960) 2024-04-26T00:28:37.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe11d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:37.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe11d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:37.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe1259c640 10 monclient(hunting): get_auth_request con 0x7efe1409fa40 auth_method 0 2024-04-26T00:28:37.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe1259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:37.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe1259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:37.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe1259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:37.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe11d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:37.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe11d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:37.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe11d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:37.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe11d9b640 10 monclient(hunting): handle_auth_done global_id 84344 payload 274 2024-04-26T00:28:37.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe11d9b640 1 -- 172.21.15.37:0/3128646269 >> v2:172.21.15.37:3300/0 conn(0x7efe1409fa40 msgr2=0x7efe1409fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:28:37.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe11d9b640 1 --2- 172.21.15.37:0/3128646269 >> v2:172.21.15.37:3300/0 conn(0x7efe1409fa40 0x7efe1409fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:37.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe11d9b640 1 -- 172.21.15.37:0/3128646269 >> v1:172.21.15.37:6789/0 conn(0x7efe140a0cb0 legacy=0x7efe1413a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:28:37.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe11d9b640 10 monclient: _finish_hunting 0 2024-04-26T00:28:37.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe11d9b640 1 monclient: found mon.noname-b 2024-04-26T00:28:37.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe11d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:37.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe11d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:28:37.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.550+0000 7efe11d9b640 1 -- 172.21.15.37:0/3128646269 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe1409d370 con 0x7efe140a0360 2024-04-26T00:28:37.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe11d9b640 1 -- 172.21.15.37:0/3128646269 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efe140a0360 msgr2=0x7efe140a0770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:37.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe11d9b640 1 --2- 172.21.15.37:0/3128646269 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efe140a0360 0x7efe140a0770 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7efe08009870 tx=0x7efe0801c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4a0112c5f1226f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:37.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1159a640 1 -- 172.21.15.37:0/3128646269 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe080b1070 con 0x7efe140a0360 2024-04-26T00:28:37.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:37.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:28:37.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1159a640 10 monclient: dump: 2024-04-26T00:28:37.558 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:37.558 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:37.558 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:37.558 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:37.558 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:37.558 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:37.558 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:37.558 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:37.558 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:37.558 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:37.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1159a640 10 monclient: _finish_auth 0 2024-04-26T00:28:37.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1159a640 10 monclient: _check_auth_tickets 2024-04-26T00:28:37.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:37.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1159a640 1 -- 172.21.15.37:0/3128646269 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe080ac600 con 0x7efe140a0360 2024-04-26T00:28:37.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1159a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:28:37.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1159a640 1 -- 172.21.15.37:0/3128646269 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe080ac9d0 con 0x7efe140a0360 2024-04-26T00:28:37.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:37.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:28:37.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1159a640 10 monclient: dump: 2024-04-26T00:28:37.559 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:37.559 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:37.559 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:37.560 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:37.560 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:37.560 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:37.560 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:37.560 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:37.560 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:37.560 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:37.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1359e640 5 monclient: authenticate success, global_id 84344 2024-04-26T00:28:37.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1359e640 10 monclient: get_monmap_and_config success 2024-04-26T00:28:37.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1359e640 4 set_mon_vals no callback set 2024-04-26T00:28:37.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1359e640 10 monclient: shutdown 2024-04-26T00:28:37.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:28:37.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1359e640 1 -- 172.21.15.37:0/3128646269 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efe140a0360 msgr2=0x7efe140a0770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:37.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1359e640 1 --2- 172.21.15.37:0/3128646269 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efe140a0360 0x7efe140a0770 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7efe08009870 tx=0x7efe0801c440 comp rx=0 tx=0).stop 2024-04-26T00:28:37.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1359e640 1 -- 172.21.15.37:0/3128646269 shutdown_connections 2024-04-26T00:28:37.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1359e640 1 --2- 172.21.15.37:0/3128646269 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efe140a0360 0x7efe140a0770 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:37.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1359e640 1 --2- 172.21.15.37:0/3128646269 >> v2:172.21.15.37:3300/0 conn(0x7efe1409fa40 0x7efe1409fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:37.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1359e640 1 -- 172.21.15.37:0/3128646269 >> 172.21.15.37:0/3128646269 conn(0x7efe140a1b40 msgr2=0x7efe140a1f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:28:37.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1359e640 1 -- 172.21.15.37:0/3128646269 shutdown_connections 2024-04-26T00:28:37.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1359e640 1 -- 172.21.15.37:0/3128646269 wait complete. 2024-04-26T00:28:37.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1359e640 10 monclient: build_initial_monmap 2024-04-26T00:28:37.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1359e640 10 monclient: monmap: 2024-04-26T00:28:37.561 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:28:37.561 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:28:37.561 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:28:37.561 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:28:37.561 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:28:37.561 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:28:37.562 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:28:37.562 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:28:37.562 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:28:37.562 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:37.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1359e640 1 Processor -- start 2024-04-26T00:28:37.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1359e640 1 -- start start 2024-04-26T00:28:37.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1359e640 10 monclient: init 2024-04-26T00:28:37.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1359e640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:37.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1359e640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:28:37.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1359e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efe1409fa40 0x7efe141cca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:37.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1359e640 10 monclient(hunting): picked mon.noname-c con 0x7efe1409fa40 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:28:37.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1359e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe140a0360 0x7efe14136750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:37.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1359e640 10 monclient(hunting): picked mon.noname-b con 0x7efe140a0360 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:28:37.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1359e640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efe140a0cb0 0x7efe1411c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:37.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1359e640 10 monclient(hunting): picked mon.noname-a con 0x7efe140a0cb0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:28:37.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1359e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:37.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1359e640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7efe1409f4f0 con 0x7efe140a0cb0 2024-04-26T00:28:37.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1359e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:37.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1359e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7efe1409f370 con 0x7efe140a0360 2024-04-26T00:28:37.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1359e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:37.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1359e640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7efe1413e890 con 0x7efe1409fa40 2024-04-26T00:28:37.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1359e640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:37.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1359e640 10 monclient(hunting): authenticate will time out at 1498.866559s 2024-04-26T00:28:37.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1259c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efe1409fa40 0x7efe141cca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:37.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1259c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efe1409fa40 0x7efe141cca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:48720/0 (socket says 172.21.15.37:48720) 2024-04-26T00:28:37.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1259c640 1 -- 172.21.15.37:0/3682034316 learned_addr learned my addr 172.21.15.37:0/3682034316 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:28:37.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1259c640 10 monclient(hunting): get_auth_request con 0x7efe1409fa40 auth_method 0 2024-04-26T00:28:37.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:37.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:37.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:37.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe12d9d640 1 --2- 172.21.15.37:0/3682034316 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efe140a0cb0 0x7efe1411c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:37.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe11d9b640 1 -- 172.21.15.37:0/3682034316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe140a0360 msgr2=0x7efe14136750 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:28:37.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe11d9b640 1 --2- 172.21.15.37:0/3682034316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe140a0360 0x7efe14136750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:28:37.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe12d9d640 10 monclient(hunting): get_auth_request con 0x7efe140a0cb0 auth_method 0 2024-04-26T00:28:37.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe12d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:37.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe12d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:37.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe12d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:37.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe12d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:37.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe12d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:37.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe12d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:37.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:37.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:37.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:37.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe12d9d640 10 monclient(hunting): handle_auth_done global_id 94281 payload 824 2024-04-26T00:28:37.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe12d9d640 1 -- 172.21.15.37:0/3682034316 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efe1409fa40 msgr2=0x7efe141cca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:37.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe12d9d640 1 --2- 172.21.15.37:0/3682034316 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efe1409fa40 0x7efe141cca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:37.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe12d9d640 1 -- 172.21.15.37:0/3682034316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe140a0360 msgr2=0x7efe14136750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:28:37.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe12d9d640 1 --2- 172.21.15.37:0/3682034316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe140a0360 0x7efe14136750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:37.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe12d9d640 10 monclient: _finish_hunting 0 2024-04-26T00:28:37.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe12d9d640 1 monclient: found mon.noname-a 2024-04-26T00:28:37.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe12d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:37.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe12d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:28:37.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe12d9d640 1 -- 172.21.15.37:0/3682034316 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe1411fe10 con 0x7efe140a0cb0 2024-04-26T00:28:37.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.558+0000 7efe1259c640 1 --2- 172.21.15.37:0/3682034316 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efe1409fa40 0x7efe141cca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:28:37.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.562+0000 7efe12d9d640 1 -- 172.21.15.37:0/3682034316 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efe140a0cb0 msgr2=0x7efe1411c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:37.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.562+0000 7efe12d9d640 1 --2- 172.21.15.37:0/3682034316 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efe140a0cb0 0x7efe1411c160 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7efe0c00c780 tx=0x7efe0c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:37.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.562+0000 7efe02ffd640 1 -- 172.21.15.37:0/3682034316 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe0c016020 con 0x7efe140a0cb0 2024-04-26T00:28:37.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.562+0000 7efe02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:37.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.562+0000 7efe02ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:28:37.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.562+0000 7efe02ffd640 10 monclient: dump: 2024-04-26T00:28:37.567 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:37.567 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:37.567 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:37.567 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:37.567 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:37.567 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:37.567 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:37.568 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:37.568 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:37.568 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:37.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.562+0000 7efe02ffd640 10 monclient: _finish_auth 0 2024-04-26T00:28:37.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.562+0000 7efe02ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:28:37.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.562+0000 7efe02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:37.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.562+0000 7efe02ffd640 1 -- 172.21.15.37:0/3682034316 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe0c00ac00 con 0x7efe140a0cb0 2024-04-26T00:28:37.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.562+0000 7efe02ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:28:37.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.562+0000 7efe02ffd640 1 -- 172.21.15.37:0/3682034316 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe0c00ce30 con 0x7efe140a0cb0 2024-04-26T00:28:37.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.562+0000 7efe1359e640 5 monclient: authenticate success, global_id 94281 2024-04-26T00:28:37.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.562+0000 7efe1359e640 10 monclient: _renew_subs 2024-04-26T00:28:37.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.562+0000 7efe1359e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:28:37.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.562+0000 7efe1359e640 1 -- 172.21.15.37:0/3682034316 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe1409c820 con 0x7efe140a0cb0 2024-04-26T00:28:37.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.562+0000 7efe10d99640 4 set_mon_vals no callback set 2024-04-26T00:28:37.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.562+0000 7efe02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:37.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.562+0000 7efe02ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:28:37.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.562+0000 7efe02ffd640 10 monclient: dump: 2024-04-26T00:28:37.569 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:37.569 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:37.569 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:37.569 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:37.569 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:37.569 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:37.569 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:37.569 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:37.569 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:37.569 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:37.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.562+0000 7efe1359e640 10 monclient: _renew_subs 2024-04-26T00:28:37.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.562+0000 7efe1359e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:28:37.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.562+0000 7efe1359e640 1 -- 172.21.15.37:0/3682034316 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe141200f0 con 0x7efe140a0cb0 2024-04-26T00:28:37.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.562+0000 7efe1359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:37.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.562+0000 7efe1359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:37.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.562+0000 7efe1359e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:28:37.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.562+0000 7efe02ffd640 1 -- 172.21.15.37:0/3682034316 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7efe0c019020 con 0x7efe140a0cb0 2024-04-26T00:28:37.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.562+0000 7efe1359e640 1 -- 172.21.15.37:0/3682034316 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe14066ed0 con 0x7efe140a0cb0 2024-04-26T00:28:37.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.562+0000 7efe02ffd640 1 --2- 172.21.15.37:0/3682034316 >> v2:172.21.15.45:6812/2379918623 conn(0x7efdf4035e20 0x7efdf40382e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:37.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.562+0000 7efe02ffd640 1 -- 172.21.15.37:0/3682034316 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(339..339 src has 177..339) ==== 3947+0+0 (secure 0 0 0) 0x7efe0c048730 con 0x7efe140a0cb0 2024-04-26T00:28:37.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.562+0000 7efe1259c640 1 --2- 172.21.15.37:0/3682034316 >> v2:172.21.15.45:6812/2379918623 conn(0x7efdf4035e20 0x7efdf40382e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:37.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.562+0000 7efe1259c640 10 monclient: get_auth_request con 0x7efdf4035e20 auth_method 0 2024-04-26T00:28:37.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.566+0000 7efe1259c640 1 --2- 172.21.15.37:0/3682034316 >> v2:172.21.15.45:6812/2379918623 conn(0x7efdf4035e20 0x7efdf40382e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7efe04007f00 tx=0x7efe04008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:37.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.566+0000 7efe02ffd640 1 -- 172.21.15.37:0/3682034316 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7efe0c020070 con 0x7efe140a0cb0 2024-04-26T00:28:37.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.566+0000 7efe02ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:37.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.566+0000 7efe02ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:28:37.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.782+0000 7efe1359e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:28:37.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.782+0000 7efe1359e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:28:37.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.782+0000 7efe1359e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:28:37.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.782+0000 7efe1359e640 1 -- 172.21.15.37:0/3682034316 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7efe14136c90 con 0x7efe140a0cb0 2024-04-26T00:28:37.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.782+0000 7efe02ffd640 1 -- 172.21.15.37:0/3682034316 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7efe0c047470 con 0x7efe140a0cb0 2024-04-26T00:28:37.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.782+0000 7efe02ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:28:37.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.782+0000 7efe02ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:28:37.772 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":64,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:28:37.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.786+0000 7efe1359e640 1 -- 172.21.15.37:0/3682034316 >> v2:172.21.15.45:6812/2379918623 conn(0x7efdf4035e20 msgr2=0x7efdf40382e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:37.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.786+0000 7efe1359e640 1 --2- 172.21.15.37:0/3682034316 >> v2:172.21.15.45:6812/2379918623 conn(0x7efdf4035e20 0x7efdf40382e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7efe04007f00 tx=0x7efe04008040 comp rx=0 tx=0).stop 2024-04-26T00:28:37.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.786+0000 7efe1359e640 10 monclient: shutdown 2024-04-26T00:28:37.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.786+0000 7efe1359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:28:37.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.786+0000 7efe1359e640 1 -- 172.21.15.37:0/3682034316 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efe140a0cb0 msgr2=0x7efe1411c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:37.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.786+0000 7efe1359e640 1 --2- 172.21.15.37:0/3682034316 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efe140a0cb0 0x7efe1411c160 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7efe0c00c780 tx=0x7efe0c00cc40 comp rx=0 tx=0).stop 2024-04-26T00:28:37.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.786+0000 7efe1359e640 1 -- 172.21.15.37:0/3682034316 shutdown_connections 2024-04-26T00:28:37.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.786+0000 7efe1359e640 1 --2- 172.21.15.37:0/3682034316 >> v2:172.21.15.45:6812/2379918623 conn(0x7efdf4035e20 0x7efdf40382e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:37.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.786+0000 7efe1359e640 1 --2- 172.21.15.37:0/3682034316 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efe140a0cb0 0x7efe1411c160 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:37.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.786+0000 7efe1359e640 1 --2- 172.21.15.37:0/3682034316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe140a0360 0x7efe14136750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:37.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.786+0000 7efe1359e640 1 --2- 172.21.15.37:0/3682034316 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efe1409fa40 0x7efe141cca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:37.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.786+0000 7efe1359e640 1 -- 172.21.15.37:0/3682034316 >> 172.21.15.37:0/3682034316 conn(0x7efe140a1b40 msgr2=0x7efe14071140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:28:37.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.786+0000 7efe1359e640 1 -- 172.21.15.37:0/3682034316 shutdown_connections 2024-04-26T00:28:37.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:37.786+0000 7efe1359e640 1 -- 172.21.15.37:0/3682034316 wait complete. 2024-04-26T00:28:40.789 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:28:40.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d93b7640 10 monclient: get_monmap_and_config 2024-04-26T00:28:40.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d93b7640 10 monclient: build_initial_monmap 2024-04-26T00:28:40.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d93b7640 10 monclient: monmap: 2024-04-26T00:28:40.922 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:28:40.922 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:40.922 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:28:40.937362+0000 2024-04-26T00:28:40.922 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:28:40.937362+0000 2024-04-26T00:28:40.922 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:28:40.922 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:40.923 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:28:40.923 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:28:40.923 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:28:40.923 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:28:40.923 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:28:40.923 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:40.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d93b7640 1 Processor -- start 2024-04-26T00:28:40.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d93b7640 1 -- start start 2024-04-26T00:28:40.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d93b7640 10 monclient: init 2024-04-26T00:28:40.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d93b7640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:40.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d93b7640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-04-26T00:28:40.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d93b7640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f96d40a0260 0x7f96d40a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:40.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d93b7640 10 monclient(hunting): picked mon.noname-b con 0x7f96d40a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:28:40.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d93b7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f96d4134b30 addr v1:172.21.15.37:6789/0 2024-04-26T00:28:40.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d93b7640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f96d4135660 0x7f96d413ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:40.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d93b7640 10 monclient(hunting): picked mon.noname-c con 0x7f96d4135660 addr v2:172.21.15.45:3300/0 2024-04-26T00:28:40.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d93b7640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96d40a1130 con 0x7f96d4134b30 2024-04-26T00:28:40.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d93b7640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:40.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d93b7640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f96d409ec10 con 0x7f96d4135660 2024-04-26T00:28:40.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d93b7640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:40.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d93b7640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f96d409f0f0 con 0x7f96d40a0260 2024-04-26T00:28:40.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d93b7640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:40.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d93b7640 10 monclient(hunting): authenticate will time out at 1502.243025s 2024-04-26T00:28:40.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d27fc640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f96d4134b30 0x7f96d4134f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:52848/0 (socket says 172.21.15.37:52848) 2024-04-26T00:28:40.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d27fc640 1 -- 172.21.15.37:0/4091294615 learned_addr learned my addr 172.21.15.37:0/4091294615 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:28:40.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d2ffd640 1 --2- 172.21.15.37:0/4091294615 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f96d40a0260 0x7f96d40a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:40.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d37fe640 1 -- 172.21.15.37:0/4091294615 >> v2:172.21.15.45:3300/0 conn(0x7f96d4135660 msgr2=0x7f96d413ce70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:28:40.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d2ffd640 10 monclient(hunting): get_auth_request con 0x7f96d40a0260 auth_method 0 2024-04-26T00:28:40.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:40.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:40.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:40.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d37fe640 1 --2- 172.21.15.37:0/4091294615 >> v2:172.21.15.45:3300/0 conn(0x7f96d4135660 0x7f96d413ce70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:28:40.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:40.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:40.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:40.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d2ffd640 10 monclient(hunting): handle_auth_done global_id 84350 payload 274 2024-04-26T00:28:40.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d2ffd640 1 -- 172.21.15.37:0/4091294615 >> v2:172.21.15.45:3300/0 conn(0x7f96d4135660 msgr2=0x7f96d413ce70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:28:40.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d2ffd640 1 --2- 172.21.15.37:0/4091294615 >> v2:172.21.15.45:3300/0 conn(0x7f96d4135660 0x7f96d413ce70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:40.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d2ffd640 1 -- 172.21.15.37:0/4091294615 >> v1:172.21.15.37:6789/0 conn(0x7f96d4134b30 legacy=0x7f96d4134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:28:40.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d2ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:28:40.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d2ffd640 1 monclient: found mon.noname-b 2024-04-26T00:28:40.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:40.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:28:40.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.934+0000 7f96d2ffd640 1 -- 172.21.15.37:0/4091294615 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96d406cff0 con 0x7f96d40a0260 2024-04-26T00:28:40.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d2ffd640 1 -- 172.21.15.37:0/4091294615 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f96d40a0260 msgr2=0x7f96d40a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:40.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d2ffd640 1 --2- 172.21.15.37:0/4091294615 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f96d40a0260 0x7f96d40a0640 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f96c40097c0 tx=0x7f96c401c160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76fa549e7dab539a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:40.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d1ffb640 1 -- 172.21.15.37:0/4091294615 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96c40b1070 con 0x7f96d40a0260 2024-04-26T00:28:40.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:40.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:28:40.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d1ffb640 10 monclient: dump: 2024-04-26T00:28:40.943 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:40.943 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:40.943 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:40.943 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:40.943 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:40.943 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:40.944 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:40.944 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:40.944 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:40.944 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:40.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d1ffb640 10 monclient: _finish_auth 0 2024-04-26T00:28:40.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d1ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:28:40.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:40.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d1ffb640 1 -- 172.21.15.37:0/4091294615 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96c40ac440 con 0x7f96d40a0260 2024-04-26T00:28:40.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d1ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:28:40.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d1ffb640 1 -- 172.21.15.37:0/4091294615 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96c40ac810 con 0x7f96d40a0260 2024-04-26T00:28:40.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d93b7640 5 monclient: authenticate success, global_id 84350 2024-04-26T00:28:40.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d93b7640 10 monclient: get_monmap_and_config success 2024-04-26T00:28:40.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d93b7640 4 set_mon_vals no callback set 2024-04-26T00:28:40.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:40.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:28:40.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d1ffb640 10 monclient: dump: 2024-04-26T00:28:40.946 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:40.946 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:40.947 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:40.947 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:40.947 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:40.947 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:40.947 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:40.947 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:40.948 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:40.948 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:40.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d93b7640 10 monclient: shutdown 2024-04-26T00:28:40.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d93b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:28:40.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d93b7640 1 -- 172.21.15.37:0/4091294615 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f96d40a0260 msgr2=0x7f96d40a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:40.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d93b7640 1 --2- 172.21.15.37:0/4091294615 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f96d40a0260 0x7f96d40a0640 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f96c40097c0 tx=0x7f96c401c160 comp rx=0 tx=0).stop 2024-04-26T00:28:40.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d93b7640 1 -- 172.21.15.37:0/4091294615 shutdown_connections 2024-04-26T00:28:40.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d93b7640 1 --2- 172.21.15.37:0/4091294615 >> v2:172.21.15.45:3300/0 conn(0x7f96d4135660 0x7f96d413ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:40.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d93b7640 1 --2- 172.21.15.37:0/4091294615 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f96d40a0260 0x7f96d40a0640 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:40.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d93b7640 1 -- 172.21.15.37:0/4091294615 >> 172.21.15.37:0/4091294615 conn(0x7f96d40a1ad0 msgr2=0x7f96d40a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:28:40.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d93b7640 1 -- 172.21.15.37:0/4091294615 shutdown_connections 2024-04-26T00:28:40.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d93b7640 1 -- 172.21.15.37:0/4091294615 wait complete. 2024-04-26T00:28:40.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d93b7640 10 monclient: build_initial_monmap 2024-04-26T00:28:40.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d93b7640 10 monclient: monmap: 2024-04-26T00:28:40.950 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:28:40.950 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:28:40.950 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:28:40.951 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:28:40.951 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:28:40.951 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:28:40.951 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:28:40.951 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:28:40.952 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:28:40.952 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:40.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d93b7640 1 Processor -- start 2024-04-26T00:28:40.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d93b7640 1 -- start start 2024-04-26T00:28:40.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d93b7640 10 monclient: init 2024-04-26T00:28:40.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d93b7640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:40.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d93b7640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:28:40.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d93b7640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f96d40a0260 0x7f96d41ccc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:40.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d93b7640 10 monclient(hunting): picked mon.noname-a con 0x7f96d40a0260 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:28:40.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d93b7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96d4134b30 0x7f96d41319b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:40.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d93b7640 10 monclient(hunting): picked mon.noname-b con 0x7f96d4134b30 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:28:40.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d93b7640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f96d4135660 0x7f96d4139af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:40.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d93b7640 10 monclient(hunting): picked mon.noname-c con 0x7f96d4135660 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:28:40.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d93b7640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:40.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d93b7640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f96d40a0010 con 0x7f96d40a0260 2024-04-26T00:28:40.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d93b7640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:40.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d93b7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f96d409fe90 con 0x7f96d4134b30 2024-04-26T00:28:40.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d93b7640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:40.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d93b7640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f96d413fbd0 con 0x7f96d4135660 2024-04-26T00:28:40.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d93b7640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:40.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d93b7640 10 monclient(hunting): authenticate will time out at 1502.250472s 2024-04-26T00:28:40.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d2ffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f96d40a0260 0x7f96d41ccc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:40.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d2ffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f96d40a0260 0x7f96d41ccc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:59302/0 (socket says 172.21.15.37:59302) 2024-04-26T00:28:40.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d2ffd640 1 -- 172.21.15.37:0/3066176277 learned_addr learned my addr 172.21.15.37:0/3066176277 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:28:40.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d2ffd640 10 monclient(hunting): get_auth_request con 0x7f96d40a0260 auth_method 0 2024-04-26T00:28:40.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d27fc640 1 -- 172.21.15.37:0/3066176277 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96d4134b30 msgr2=0x7f96d41319b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:28:40.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:40.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:40.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:40.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d27fc640 1 --2- 172.21.15.37:0/3066176277 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96d4134b30 0x7f96d41319b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:28:40.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:40.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:40.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d37fe640 1 --2- 172.21.15.37:0/3066176277 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f96d4135660 0x7f96d4139af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:40.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:40.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d37fe640 10 monclient(hunting): get_auth_request con 0x7f96d4135660 auth_method 0 2024-04-26T00:28:40.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:40.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:40.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:40.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:40.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:40.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:40.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d2ffd640 10 monclient(hunting): handle_auth_done global_id 94284 payload 824 2024-04-26T00:28:40.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d2ffd640 1 -- 172.21.15.37:0/3066176277 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f96d4135660 msgr2=0x7f96d4139af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:40.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d2ffd640 1 --2- 172.21.15.37:0/3066176277 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f96d4135660 0x7f96d4139af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:40.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d2ffd640 1 -- 172.21.15.37:0/3066176277 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96d4134b30 msgr2=0x7f96d41319b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:28:40.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d2ffd640 1 --2- 172.21.15.37:0/3066176277 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96d4134b30 0x7f96d41319b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:40.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d2ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:28:40.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d2ffd640 1 monclient: found mon.noname-a 2024-04-26T00:28:40.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:40.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:28:40.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.942+0000 7f96d2ffd640 1 -- 172.21.15.37:0/3066176277 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96d411c340 con 0x7f96d40a0260 2024-04-26T00:28:40.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.946+0000 7f96d2ffd640 1 -- 172.21.15.37:0/3066176277 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f96d40a0260 msgr2=0x7f96d41ccc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:40.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.946+0000 7f96d2ffd640 1 --2- 172.21.15.37:0/3066176277 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f96d40a0260 0x7f96d41ccc50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f96c40af040 tx=0x7f96c4018f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:40.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.946+0000 7f96cb7fe640 1 -- 172.21.15.37:0/3066176277 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96c40b9070 con 0x7f96d40a0260 2024-04-26T00:28:40.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.946+0000 7f96cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:40.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.946+0000 7f96cb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:28:40.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.946+0000 7f96cb7fe640 10 monclient: dump: 2024-04-26T00:28:40.958 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:40.958 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:40.958 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:40.958 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:40.958 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:40.958 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:40.958 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:40.958 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:40.958 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:40.958 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:40.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.946+0000 7f96cb7fe640 10 monclient: _finish_auth 0 2024-04-26T00:28:40.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.946+0000 7f96cb7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:28:40.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.946+0000 7f96cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:40.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.946+0000 7f96cb7fe640 1 -- 172.21.15.37:0/3066176277 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96c40b5820 con 0x7f96d40a0260 2024-04-26T00:28:40.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.946+0000 7f96d93b7640 5 monclient: authenticate success, global_id 94284 2024-04-26T00:28:40.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.946+0000 7f96d93b7640 10 monclient: _renew_subs 2024-04-26T00:28:40.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.946+0000 7f96d93b7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:28:40.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.946+0000 7f96d93b7640 1 -- 172.21.15.37:0/3066176277 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96d40729d0 con 0x7f96d40a0260 2024-04-26T00:28:40.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.946+0000 7f96cb7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:28:40.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.946+0000 7f96cb7fe640 1 -- 172.21.15.37:0/3066176277 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96c40b1040 con 0x7f96d40a0260 2024-04-26T00:28:40.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.946+0000 7f96cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:40.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.946+0000 7f96cb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:28:40.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.946+0000 7f96cb7fe640 10 monclient: dump: 2024-04-26T00:28:40.959 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:40.959 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:40.960 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:40.960 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:40.960 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:40.960 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:40.960 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:40.960 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:40.960 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:40.960 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:40.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.946+0000 7f96d93b7640 10 monclient: _renew_subs 2024-04-26T00:28:40.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.946+0000 7f96d93b7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:28:40.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.946+0000 7f96d93b7640 1 -- 172.21.15.37:0/3066176277 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96d411c060 con 0x7f96d40a0260 2024-04-26T00:28:40.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.946+0000 7f96d17fa640 4 set_mon_vals no callback set 2024-04-26T00:28:40.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.946+0000 7f96d93b7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:40.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.946+0000 7f96d93b7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:40.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.946+0000 7f96d93b7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:28:40.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.946+0000 7f96cb7fe640 1 -- 172.21.15.37:0/3066176277 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f96c40bc070 con 0x7f96d40a0260 2024-04-26T00:28:40.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.946+0000 7f96d93b7640 1 -- 172.21.15.37:0/3066176277 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96d4066f40 con 0x7f96d40a0260 2024-04-26T00:28:40.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.946+0000 7f96cb7fe640 1 --2- 172.21.15.37:0/3066176277 >> v2:172.21.15.45:6812/2379918623 conn(0x7f96b4035e30 0x7f96b40382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:40.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.946+0000 7f96cb7fe640 1 -- 172.21.15.37:0/3066176277 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(339..339 src has 177..339) ==== 3947+0+0 (secure 0 0 0) 0x7f96c40e2450 con 0x7f96d40a0260 2024-04-26T00:28:40.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.946+0000 7f96d27fc640 1 --2- 172.21.15.37:0/3066176277 >> v2:172.21.15.45:6812/2379918623 conn(0x7f96b4035e30 0x7f96b40382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:40.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.946+0000 7f96d27fc640 10 monclient: get_auth_request con 0x7f96b4035e30 auth_method 0 2024-04-26T00:28:40.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.950+0000 7f96d27fc640 1 --2- 172.21.15.37:0/3066176277 >> v2:172.21.15.45:6812/2379918623 conn(0x7f96b4035e30 0x7f96b40382f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f96d413c140 tx=0x7f96bc007b30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:40.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.950+0000 7f96cb7fe640 1 -- 172.21.15.37:0/3066176277 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f96c40c3070 con 0x7f96d40a0260 2024-04-26T00:28:40.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.950+0000 7f96cb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:40.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:40.950+0000 7f96cb7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:28:41.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:41.170+0000 7f96d93b7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:28:41.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:41.170+0000 7f96d93b7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:28:41.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:41.170+0000 7f96d93b7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:28:41.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:41.170+0000 7f96d93b7640 1 -- 172.21.15.37:0/3066176277 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f96d413bee0 con 0x7f96d40a0260 2024-04-26T00:28:41.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:41.170+0000 7f96cb7fe640 1 -- 172.21.15.37:0/3066176277 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f96d40a1130 con 0x7f96d40a0260 2024-04-26T00:28:41.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:41.170+0000 7f96cb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:28:41.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:41.170+0000 7f96cb7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:28:41.160 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":64,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:28:41.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:41.174+0000 7f96d93b7640 1 -- 172.21.15.37:0/3066176277 >> v2:172.21.15.45:6812/2379918623 conn(0x7f96b4035e30 msgr2=0x7f96b40382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:41.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:41.174+0000 7f96d93b7640 1 --2- 172.21.15.37:0/3066176277 >> v2:172.21.15.45:6812/2379918623 conn(0x7f96b4035e30 0x7f96b40382f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f96d413c140 tx=0x7f96bc007b30 comp rx=0 tx=0).stop 2024-04-26T00:28:41.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:41.174+0000 7f96d93b7640 10 monclient: shutdown 2024-04-26T00:28:41.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:41.174+0000 7f96d93b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:28:41.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:41.174+0000 7f96d93b7640 1 -- 172.21.15.37:0/3066176277 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f96d40a0260 msgr2=0x7f96d41ccc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:41.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:41.174+0000 7f96d93b7640 1 --2- 172.21.15.37:0/3066176277 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f96d40a0260 0x7f96d41ccc50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f96c40af040 tx=0x7f96c4018f00 comp rx=0 tx=0).stop 2024-04-26T00:28:41.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:41.174+0000 7f96d93b7640 1 -- 172.21.15.37:0/3066176277 shutdown_connections 2024-04-26T00:28:41.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:41.174+0000 7f96d93b7640 1 --2- 172.21.15.37:0/3066176277 >> v2:172.21.15.45:6812/2379918623 conn(0x7f96b4035e30 0x7f96b40382f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:41.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:41.174+0000 7f96d93b7640 1 --2- 172.21.15.37:0/3066176277 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f96d4135660 0x7f96d4139af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:41.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:41.174+0000 7f96d93b7640 1 --2- 172.21.15.37:0/3066176277 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96d4134b30 0x7f96d41319b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:41.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:41.174+0000 7f96d93b7640 1 --2- 172.21.15.37:0/3066176277 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f96d40a0260 0x7f96d41ccc50 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:41.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:41.174+0000 7f96d93b7640 1 -- 172.21.15.37:0/3066176277 >> 172.21.15.37:0/3066176277 conn(0x7f96d40a1ad0 msgr2=0x7f96d4132df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:28:41.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:41.174+0000 7f96d93b7640 1 -- 172.21.15.37:0/3066176277 shutdown_connections 2024-04-26T00:28:41.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:41.174+0000 7f96d93b7640 1 -- 172.21.15.37:0/3066176277 wait complete. 2024-04-26T00:28:44.177 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:28:44.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.322+0000 7f0b19a3e640 10 monclient: get_monmap_and_config 2024-04-26T00:28:44.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.322+0000 7f0b19a3e640 10 monclient: build_initial_monmap 2024-04-26T00:28:44.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.322+0000 7f0b19a3e640 10 monclient: monmap: 2024-04-26T00:28:44.310 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:28:44.310 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:44.310 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:28:44.325386+0000 2024-04-26T00:28:44.310 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:28:44.325386+0000 2024-04-26T00:28:44.310 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:28:44.310 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:44.311 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:28:44.311 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:28:44.311 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:28:44.311 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:28:44.311 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:28:44.311 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:44.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.322+0000 7f0b19a3e640 1 Processor -- start 2024-04-26T00:28:44.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.322+0000 7f0b19a3e640 1 -- start start 2024-04-26T00:28:44.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.322+0000 7f0b19a3e640 10 monclient: init 2024-04-26T00:28:44.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.322+0000 7f0b19a3e640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:44.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.322+0000 7f0b19a3e640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-04-26T00:28:44.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.322+0000 7f0b19a3e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0b14131b70 addr v1:172.21.15.45:6789/0 2024-04-26T00:28:44.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.322+0000 7f0b19a3e640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f0b1406eba0 0x7f0b14139e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:44.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.322+0000 7f0b19a3e640 10 monclient(hunting): picked mon.noname-a con 0x7f0b1406eba0 addr v2:172.21.15.37:3300/0 2024-04-26T00:28:44.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.322+0000 7f0b19a3e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0b1413a380 0x7f0b1413c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:44.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.322+0000 7f0b19a3e640 10 monclient(hunting): picked mon.noname-c con 0x7f0b1413a380 addr v2:172.21.15.45:3300/0 2024-04-26T00:28:44.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.322+0000 7f0b19a3e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b14072170 con 0x7f0b14131b70 2024-04-26T00:28:44.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.322+0000 7f0b19a3e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:44.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.322+0000 7f0b19a3e640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f0b1406e790 con 0x7f0b1406eba0 2024-04-26T00:28:44.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.322+0000 7f0b19a3e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:44.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.322+0000 7f0b19a3e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f0b1409ff10 con 0x7f0b1413a380 2024-04-26T00:28:44.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.322+0000 7f0b19a3e640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:44.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.322+0000 7f0b19a3e640 10 monclient(hunting): authenticate will time out at 1505.631252s 2024-04-26T00:28:44.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.322+0000 7f0b12ffd640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f0b14131b70 legacy=0x7f0b14131f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:28:44.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.322+0000 7f0b127fc640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f0b1406eba0 0x7f0b14139e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:44.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.322+0000 7f0b137fe640 1 -- >> v2:172.21.15.45:3300/0 conn(0x7f0b1413a380 msgr2=0x7f0b1413c770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:28:44.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.322+0000 7f0b127fc640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f0b1406eba0 0x7f0b14139e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:59308/0 (socket says 172.21.15.37:59308) 2024-04-26T00:28:44.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.322+0000 7f0b127fc640 1 -- 172.21.15.37:0/3386379567 learned_addr learned my addr 172.21.15.37:0/3386379567 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:28:44.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.322+0000 7f0b127fc640 10 monclient(hunting): get_auth_request con 0x7f0b1406eba0 auth_method 0 2024-04-26T00:28:44.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.322+0000 7f0b127fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:44.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.322+0000 7f0b127fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:44.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.322+0000 7f0b127fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:44.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.322+0000 7f0b137fe640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0b1413a380 0x7f0b1413c770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:28:44.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.322+0000 7f0b127fc640 1 -- 172.21.15.37:0/3386379567 >> v2:172.21.15.37:3300/0 conn(0x7f0b1406eba0 msgr2=0x7f0b14139e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:28:44.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.322+0000 7f0b127fc640 1 -- 172.21.15.37:0/3386379567 >> v2:172.21.15.37:3300/0 conn(0x7f0b1406eba0 msgr2=0x7f0b14139e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:28:44.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.322+0000 7f0b127fc640 1 --2- 172.21.15.37:0/3386379567 >> v2:172.21.15.37:3300/0 conn(0x7f0b1406eba0 0x7f0b14139e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:28:44.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.322+0000 7f0b127fc640 1 --2- 172.21.15.37:0/3386379567 >> v2:172.21.15.37:3300/0 conn(0x7f0b1406eba0 0x7f0b14139e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:28:44.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.526+0000 7f0b12ffd640 1 -- 172.21.15.37:0/3386379567 >> v1:172.21.15.45:6789/0 conn(0x7f0b14131b70 legacy=0x7f0b14131f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:28:44.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.526+0000 7f0b137fe640 1 -- 172.21.15.37:0/3386379567 >> v2:172.21.15.45:3300/0 conn(0x7f0b1413a380 msgr2=0x7f0b1413c770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:28:44.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.526+0000 7f0b137fe640 1 --2- 172.21.15.37:0/3386379567 >> v2:172.21.15.45:3300/0 conn(0x7f0b1413a380 0x7f0b1413c770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:28:44.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.526+0000 7f0b127fc640 1 --2- 172.21.15.37:0/3386379567 >> v2:172.21.15.37:3300/0 conn(0x7f0b1406eba0 0x7f0b14139e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:44.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.526+0000 7f0b127fc640 10 monclient(hunting): get_auth_request con 0x7f0b1406eba0 auth_method 0 2024-04-26T00:28:44.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.526+0000 7f0b127fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:44.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.526+0000 7f0b127fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:44.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.526+0000 7f0b127fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:28:44.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.526+0000 7f0b127fc640 1 -- 172.21.15.37:0/3386379567 >> v2:172.21.15.37:3300/0 conn(0x7f0b1406eba0 msgr2=0x7f0b14139e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:28:44.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.526+0000 7f0b127fc640 1 -- 172.21.15.37:0/3386379567 >> v2:172.21.15.37:3300/0 conn(0x7f0b1406eba0 msgr2=0x7f0b14139e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:28:44.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.526+0000 7f0b127fc640 1 --2- 172.21.15.37:0/3386379567 >> v2:172.21.15.37:3300/0 conn(0x7f0b1406eba0 0x7f0b14139e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:28:44.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.526+0000 7f0b127fc640 1 --2- 172.21.15.37:0/3386379567 >> v2:172.21.15.37:3300/0 conn(0x7f0b1406eba0 0x7f0b14139e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:28:44.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.930+0000 7f0b137fe640 1 -- 172.21.15.37:0/3386379567 >> v2:172.21.15.45:3300/0 conn(0x7f0b1413a380 msgr2=0x7f0b1413c770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:28:44.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.930+0000 7f0b12ffd640 1 -- 172.21.15.37:0/3386379567 >> v1:172.21.15.45:6789/0 conn(0x7f0b14131b70 legacy=0x7f0b14131f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:28:44.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.930+0000 7f0b137fe640 1 --2- 172.21.15.37:0/3386379567 >> v2:172.21.15.45:3300/0 conn(0x7f0b1413a380 0x7f0b1413c770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:28:44.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.930+0000 7f0b127fc640 1 --2- 172.21.15.37:0/3386379567 >> v2:172.21.15.37:3300/0 conn(0x7f0b1406eba0 0x7f0b14139e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:44.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.930+0000 7f0b127fc640 10 monclient(hunting): get_auth_request con 0x7f0b1406eba0 auth_method 0 2024-04-26T00:28:44.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.930+0000 7f0b127fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:44.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.930+0000 7f0b127fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:44.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.930+0000 7f0b127fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:28:44.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.930+0000 7f0b127fc640 1 -- 172.21.15.37:0/3386379567 >> v2:172.21.15.37:3300/0 conn(0x7f0b1406eba0 msgr2=0x7f0b14139e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:28:44.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.930+0000 7f0b127fc640 1 -- 172.21.15.37:0/3386379567 >> v2:172.21.15.37:3300/0 conn(0x7f0b1406eba0 msgr2=0x7f0b14139e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:28:44.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.930+0000 7f0b127fc640 1 --2- 172.21.15.37:0/3386379567 >> v2:172.21.15.37:3300/0 conn(0x7f0b1406eba0 0x7f0b14139e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:28:44.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:44.930+0000 7f0b127fc640 1 --2- 172.21.15.37:0/3386379567 >> v2:172.21.15.37:3300/0 conn(0x7f0b1406eba0 0x7f0b14139e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:28:45.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:45.734+0000 7f0b12ffd640 1 -- 172.21.15.37:0/3386379567 >> v1:172.21.15.45:6789/0 conn(0x7f0b14131b70 legacy=0x7f0b14131f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:28:45.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:45.734+0000 7f0b137fe640 1 -- 172.21.15.37:0/3386379567 >> v2:172.21.15.45:3300/0 conn(0x7f0b1413a380 msgr2=0x7f0b1413c770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:28:45.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:45.734+0000 7f0b137fe640 1 --2- 172.21.15.37:0/3386379567 >> v2:172.21.15.45:3300/0 conn(0x7f0b1413a380 0x7f0b1413c770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:28:45.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:45.734+0000 7f0b127fc640 1 --2- 172.21.15.37:0/3386379567 >> v2:172.21.15.37:3300/0 conn(0x7f0b1406eba0 0x7f0b14139e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:45.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:45.734+0000 7f0b127fc640 10 monclient(hunting): get_auth_request con 0x7f0b1406eba0 auth_method 0 2024-04-26T00:28:45.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:45.734+0000 7f0b127fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:45.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:45.734+0000 7f0b127fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:45.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:45.734+0000 7f0b127fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:28:45.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:45.734+0000 7f0b127fc640 1 -- 172.21.15.37:0/3386379567 >> v2:172.21.15.37:3300/0 conn(0x7f0b1406eba0 msgr2=0x7f0b14139e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:28:45.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:45.734+0000 7f0b127fc640 1 -- 172.21.15.37:0/3386379567 >> v2:172.21.15.37:3300/0 conn(0x7f0b1406eba0 msgr2=0x7f0b14139e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:28:45.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:45.734+0000 7f0b127fc640 1 --2- 172.21.15.37:0/3386379567 >> v2:172.21.15.37:3300/0 conn(0x7f0b1406eba0 0x7f0b14139e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:28:45.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:45.734+0000 7f0b127fc640 1 --2- 172.21.15.37:0/3386379567 >> v2:172.21.15.37:3300/0 conn(0x7f0b1406eba0 0x7f0b14139e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:28:47.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.326+0000 7f0b10ff9640 10 monclient(hunting): tick 2024-04-26T00:28:47.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.326+0000 7f0b10ff9640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:28:47.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.326+0000 7f0b10ff9640 1 monclient(hunting): continuing hunt 2024-04-26T00:28:47.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.326+0000 7f0b10ff9640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:28:47.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.326+0000 7f0b10ff9640 1 -- 172.21.15.37:0/3386379567 >> v2:172.21.15.45:3300/0 conn(0x7f0b1413a380 msgr2=0x7f0b1413c770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:28:47.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.326+0000 7f0b10ff9640 1 --2- 172.21.15.37:0/3386379567 >> v2:172.21.15.45:3300/0 conn(0x7f0b1413a380 0x7f0b1413c770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:47.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.326+0000 7f0b10ff9640 1 -- 172.21.15.37:0/3386379567 >> v2:172.21.15.37:3300/0 conn(0x7f0b1406eba0 msgr2=0x7f0b14139e40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:28:47.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.326+0000 7f0b10ff9640 1 --2- 172.21.15.37:0/3386379567 >> v2:172.21.15.37:3300/0 conn(0x7f0b1406eba0 0x7f0b14139e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:47.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.326+0000 7f0b10ff9640 1 -- 172.21.15.37:0/3386379567 >> v1:172.21.15.45:6789/0 conn(0x7f0b14131b70 legacy=0x7f0b14131f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:28:47.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.326+0000 7f0b10ff9640 10 monclient: _add_conns ranks=[0,2] 2024-04-26T00:28:47.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.326+0000 7f0b10ff9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0af4003680 addr v1:172.21.15.37:6789/0 2024-04-26T00:28:47.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.326+0000 7f0b10ff9640 1 --2- 172.21.15.37:0/3386379567 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0af40072d0 0x7f0af4009790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:47.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.326+0000 7f0b10ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f0af40072d0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:28:47.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.326+0000 7f0b10ff9640 1 -- 172.21.15.37:0/3386379567 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b14071bb0 con 0x7f0af4003680 2024-04-26T00:28:47.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.326+0000 7f0b10ff9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:47.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.326+0000 7f0b10ff9640 1 -- 172.21.15.37:0/3386379567 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0b1409ff10 con 0x7f0af40072d0 2024-04-26T00:28:47.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.326+0000 7f0b10ff9640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:47.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.326+0000 7f0b127fc640 1 --2- 172.21.15.37:0/3386379567 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0af40072d0 0x7f0af4009790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:47.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.326+0000 7f0b127fc640 10 monclient(hunting): get_auth_request con 0x7f0af40072d0 auth_method 0 2024-04-26T00:28:47.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.326+0000 7f0b127fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:47.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.326+0000 7f0b127fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:47.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.326+0000 7f0b127fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:47.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.326+0000 7f0b127fc640 1 -- 172.21.15.37:0/3386379567 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0af40072d0 msgr2=0x7f0af4009790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:28:47.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.326+0000 7f0b127fc640 1 -- 172.21.15.37:0/3386379567 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0af40072d0 msgr2=0x7f0af4009790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:28:47.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.326+0000 7f0b127fc640 1 --2- 172.21.15.37:0/3386379567 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0af40072d0 0x7f0af4009790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:28:47.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.326+0000 7f0b127fc640 1 --2- 172.21.15.37:0/3386379567 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0af40072d0 0x7f0af4009790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:28:47.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.326+0000 7f0b12ffd640 1 -- 172.21.15.37:0/3386379567 >> v1:172.21.15.37:6789/0 conn(0x7f0af4003680 legacy=0x7f0af4005b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:47.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.530+0000 7f0b127fc640 1 --2- 172.21.15.37:0/3386379567 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0af40072d0 0x7f0af4009790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:47.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.530+0000 7f0b127fc640 10 monclient(hunting): get_auth_request con 0x7f0af40072d0 auth_method 0 2024-04-26T00:28:47.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.530+0000 7f0b127fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:47.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.530+0000 7f0b127fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:47.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.530+0000 7f0b127fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:28:47.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.530+0000 7f0b127fc640 1 -- 172.21.15.37:0/3386379567 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0af40072d0 msgr2=0x7f0af4009790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:28:47.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.530+0000 7f0b127fc640 1 -- 172.21.15.37:0/3386379567 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0af40072d0 msgr2=0x7f0af4009790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:28:47.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.530+0000 7f0b127fc640 1 --2- 172.21.15.37:0/3386379567 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0af40072d0 0x7f0af4009790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:28:47.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.530+0000 7f0b127fc640 1 --2- 172.21.15.37:0/3386379567 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0af40072d0 0x7f0af4009790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:28:47.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.934+0000 7f0b127fc640 1 --2- 172.21.15.37:0/3386379567 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0af40072d0 0x7f0af4009790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:47.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.934+0000 7f0b127fc640 10 monclient(hunting): get_auth_request con 0x7f0af40072d0 auth_method 0 2024-04-26T00:28:47.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.934+0000 7f0b127fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:47.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.934+0000 7f0b127fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:47.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.934+0000 7f0b127fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:28:47.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.934+0000 7f0b127fc640 1 -- 172.21.15.37:0/3386379567 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0af40072d0 msgr2=0x7f0af4009790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:28:47.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.934+0000 7f0b127fc640 1 -- 172.21.15.37:0/3386379567 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0af40072d0 msgr2=0x7f0af4009790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:28:47.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.934+0000 7f0b127fc640 1 --2- 172.21.15.37:0/3386379567 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0af40072d0 0x7f0af4009790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:28:47.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:47.934+0000 7f0b127fc640 1 --2- 172.21.15.37:0/3386379567 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0af40072d0 0x7f0af4009790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:28:48.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:48.738+0000 7f0b127fc640 1 --2- 172.21.15.37:0/3386379567 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0af40072d0 0x7f0af4009790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:48.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:48.738+0000 7f0b127fc640 10 monclient(hunting): get_auth_request con 0x7f0af40072d0 auth_method 0 2024-04-26T00:28:48.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:48.738+0000 7f0b127fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:48.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:48.738+0000 7f0b127fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:48.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:48.738+0000 7f0b127fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:28:48.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:48.738+0000 7f0b127fc640 1 -- 172.21.15.37:0/3386379567 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0af40072d0 msgr2=0x7f0af4009790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:28:48.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:48.738+0000 7f0b127fc640 1 -- 172.21.15.37:0/3386379567 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0af40072d0 msgr2=0x7f0af4009790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:28:48.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:48.738+0000 7f0b127fc640 1 --2- 172.21.15.37:0/3386379567 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0af40072d0 0x7f0af4009790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:28:48.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:48.738+0000 7f0b127fc640 1 --2- 172.21.15.37:0/3386379567 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0af40072d0 0x7f0af4009790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:28:49.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.198+0000 7f0b11ffb640 1 -- 172.21.15.37:0/3386379567 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 693918059 0 0) 0x7f0b14071bb0 con 0x7f0af4003680 2024-04-26T00:28:49.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.198+0000 7f0b11ffb640 10 monclient(hunting): _negotiate 2024-04-26T00:28:49.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.198+0000 7f0b11ffb640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:49.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.198+0000 7f0b11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:49.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.198+0000 7f0b11ffb640 10 monclient(hunting): my global_id is 94287 2024-04-26T00:28:49.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.198+0000 7f0b11ffb640 1 -- 172.21.15.37:0/3386379567 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b00003530 con 0x7f0af4003680 2024-04-26T00:28:49.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b11ffb640 1 -- 172.21.15.37:0/3386379567 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1027973653 0 0) 0x7f0b00003530 con 0x7f0af4003680 2024-04-26T00:28:49.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b11ffb640 1 -- 172.21.15.37:0/3386379567 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0af40072d0 msgr2=0x7f0af4009790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:28:49.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b11ffb640 1 --2- 172.21.15.37:0/3386379567 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0af40072d0 0x7f0af4009790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:49.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b11ffb640 10 monclient: _finish_hunting 0 2024-04-26T00:28:49.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b11ffb640 1 monclient: found mon.noname-a-legacy 2024-04-26T00:28:49.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b11ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:49.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b11ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.37:6789/0 2024-04-26T00:28:49.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b11ffb640 1 -- 172.21.15.37:0/3386379567 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b14072170 con 0x7f0af4003680 2024-04-26T00:28:49.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b11ffb640 10 monclient: _finish_auth 0 2024-04-26T00:28:49.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b11ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:28:49.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:49.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b11ffb640 1 -- 172.21.15.37:0/3386379567 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 866326386 0 0) 0x7f0afc004400 con 0x7f0af4003680 2024-04-26T00:28:49.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b11ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:49.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b11ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-26T00:28:49.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b11ffb640 10 monclient: dump: 2024-04-26T00:28:49.192 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:49.192 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:49.192 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:49.193 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:49.193 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:49.193 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:49.193 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:49.193 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:49.193 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:49.193 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:49.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b11ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] but i'm connected to v1:172.21.15.37:6789/0, reconnecting 2024-04-26T00:28:49.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b11ffb640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:49.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b11ffb640 1 -- 172.21.15.37:0/3386379567 >> v1:172.21.15.37:6789/0 conn(0x7f0af4003680 legacy=0x7f0af4005b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:49.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b11ffb640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:28:49.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b137fe640 1 -- 172.21.15.37:0/3386379567 reap_dead start 2024-04-26T00:28:49.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b11ffb640 1 --2- 172.21.15.37:0/3386379567 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0af40072d0 0x7f0b000039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:49.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b11ffb640 10 monclient(hunting): picked mon.c con 0x7f0af40072d0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:28:49.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b11ffb640 1 --2- 172.21.15.37:0/3386379567 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0b00003fd0 0x7f0b00004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:49.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b11ffb640 10 monclient(hunting): picked mon.a con 0x7f0b00003fd0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:28:49.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b11ffb640 1 --2- 172.21.15.37:0/3386379567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b00004d50 0x7f0b000051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:49.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b11ffb640 10 monclient(hunting): picked mon.b con 0x7f0b00004d50 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:28:49.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b11ffb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:49.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b11ffb640 1 -- 172.21.15.37:0/3386379567 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f0b1409ff10 con 0x7f0b00003fd0 2024-04-26T00:28:49.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b11ffb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:49.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b11ffb640 1 -- 172.21.15.37:0/3386379567 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0b0000bbc0 con 0x7f0b00004d50 2024-04-26T00:28:49.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b11ffb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:49.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b11ffb640 1 -- 172.21.15.37:0/3386379567 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0b0000bdc0 con 0x7f0af40072d0 2024-04-26T00:28:49.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b11ffb640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:49.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b11ffb640 10 monclient(hunting): _finish_auth 0 2024-04-26T00:28:49.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b127fc640 1 --2- 172.21.15.37:0/3386379567 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0b00003fd0 0x7f0b00004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:49.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b12ffd640 1 --2- 172.21.15.37:0/3386379567 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0af40072d0 0x7f0b000039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:49.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b137fe640 1 -- 172.21.15.37:0/3386379567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b00004d50 msgr2=0x7f0b000051d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:28:49.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b12ffd640 10 monclient(hunting): get_auth_request con 0x7f0af40072d0 auth_method 0 2024-04-26T00:28:49.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b12ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:49.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b137fe640 1 --2- 172.21.15.37:0/3386379567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b00004d50 0x7f0b000051d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:28:49.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b12ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:49.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b12ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:28:49.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b127fc640 10 monclient(hunting): get_auth_request con 0x7f0b00003fd0 auth_method 0 2024-04-26T00:28:49.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b127fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:49.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b127fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:49.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b127fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:28:49.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b12ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:49.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b12ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:49.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b12ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:28:49.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b127fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:49.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b127fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:49.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b127fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:28:49.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b12ffd640 10 monclient(hunting): handle_auth_done global_id 94287 payload 293 2024-04-26T00:28:49.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b12ffd640 1 -- 172.21.15.37:0/3386379567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b00004d50 msgr2=0x7f0b000051d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:28:49.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b12ffd640 1 --2- 172.21.15.37:0/3386379567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b00004d50 0x7f0b000051d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:49.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b12ffd640 1 -- 172.21.15.37:0/3386379567 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0b00003fd0 msgr2=0x7f0b00004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:28:49.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b12ffd640 1 --2- 172.21.15.37:0/3386379567 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0b00003fd0 0x7f0b00004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:49.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b12ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:28:49.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b12ffd640 1 monclient: found mon.c 2024-04-26T00:28:49.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:49.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b12ffd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:28:49.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b12ffd640 1 -- 172.21.15.37:0/3386379567 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b00003530 con 0x7f0af40072d0 2024-04-26T00:28:49.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b12ffd640 10 monclient: _finish_auth 0 2024-04-26T00:28:49.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b12ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:28:49.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:49.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b19a3e640 5 monclient: authenticate success, global_id 94287 2024-04-26T00:28:49.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.202+0000 7f0b19a3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:28:49.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.210+0000 7f0b12ffd640 1 -- 172.21.15.37:0/3386379567 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0af40072d0 msgr2=0x7f0b000039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:49.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.210+0000 7f0b12ffd640 1 --2- 172.21.15.37:0/3386379567 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0af40072d0 0x7f0b000039c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0afc003940 tx=0x7f0afc01cd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8796310f0cb2054 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:49.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.210+0000 7f0b11ffb640 1 -- 172.21.15.37:0/3386379567 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0afc00c0e0 con 0x7f0af40072d0 2024-04-26T00:28:49.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.210+0000 7f0b11ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:49.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.210+0000 7f0b11ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:28:49.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.210+0000 7f0b11ffb640 10 monclient: dump: 2024-04-26T00:28:49.204 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:49.205 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:49.205 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:49.205 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:49.205 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:49.205 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:49.205 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:49.205 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:49.206 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:49.206 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:49.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.210+0000 7f0b11ffb640 1 -- 172.21.15.37:0/3386379567 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0afc0a9070 con 0x7f0af40072d0 2024-04-26T00:28:49.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.210+0000 7f0b19a3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:28:49.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.210+0000 7f0b11ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:28:49.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.210+0000 7f0b19a3e640 10 monclient: get_monmap_and_config success 2024-04-26T00:28:49.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.210+0000 7f0b19a3e640 4 set_mon_vals no callback set 2024-04-26T00:28:49.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.210+0000 7f0b11ffb640 1 -- 172.21.15.37:0/3386379567 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0afc0064b0 con 0x7f0af40072d0 2024-04-26T00:28:49.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.210+0000 7f0b11ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:49.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.210+0000 7f0b11ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:28:49.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.210+0000 7f0b11ffb640 10 monclient: dump: 2024-04-26T00:28:49.207 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:49.207 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:49.207 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:49.207 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:49.207 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:49.207 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:49.207 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:49.207 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:49.207 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:49.207 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:49.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.214+0000 7f0b19a3e640 10 monclient: shutdown 2024-04-26T00:28:49.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.214+0000 7f0b19a3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:28:49.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.214+0000 7f0b19a3e640 1 -- 172.21.15.37:0/3386379567 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0af40072d0 msgr2=0x7f0b000039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:49.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.214+0000 7f0b19a3e640 1 --2- 172.21.15.37:0/3386379567 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0af40072d0 0x7f0b000039c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0afc003940 tx=0x7f0afc01cd10 comp rx=0 tx=0).stop 2024-04-26T00:28:49.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.214+0000 7f0b19a3e640 1 -- 172.21.15.37:0/3386379567 shutdown_connections 2024-04-26T00:28:49.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.214+0000 7f0b19a3e640 1 --2- 172.21.15.37:0/3386379567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b00004d50 0x7f0b000051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:49.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.214+0000 7f0b19a3e640 1 --2- 172.21.15.37:0/3386379567 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0b00003fd0 0x7f0b00004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:49.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.214+0000 7f0b19a3e640 1 --2- 172.21.15.37:0/3386379567 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0af40072d0 0x7f0b000039c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:49.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.214+0000 7f0b19a3e640 1 -- 172.21.15.37:0/3386379567 >> 172.21.15.37:0/3386379567 conn(0x7f0b140a24c0 msgr2=0x7f0b140a28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:28:49.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.214+0000 7f0b19a3e640 1 -- 172.21.15.37:0/3386379567 shutdown_connections 2024-04-26T00:28:49.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.214+0000 7f0b19a3e640 1 -- 172.21.15.37:0/3386379567 wait complete. 2024-04-26T00:28:49.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.214+0000 7f0b19a3e640 10 monclient: build_initial_monmap 2024-04-26T00:28:49.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.214+0000 7f0b19a3e640 10 monclient: monmap: 2024-04-26T00:28:49.208 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:28:49.208 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:28:49.208 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:28:49.208 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:28:49.208 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:28:49.208 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:28:49.209 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:28:49.209 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:28:49.209 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:28:49.209 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:49.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b19a3e640 1 Processor -- start 2024-04-26T00:28:49.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b19a3e640 1 -- start start 2024-04-26T00:28:49.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b19a3e640 10 monclient: init 2024-04-26T00:28:49.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b19a3e640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:49.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b19a3e640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:28:49.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b19a3e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0af40072d0 0x7f0b141cca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:49.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b19a3e640 10 monclient(hunting): picked mon.noname-c con 0x7f0af40072d0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:28:49.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b19a3e640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0b00003fd0 0x7f0b1412e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:49.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b19a3e640 10 monclient(hunting): picked mon.noname-a con 0x7f0b00003fd0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:28:49.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b19a3e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b00004d50 0x7f0b1412ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:49.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b19a3e640 10 monclient(hunting): picked mon.noname-b con 0x7f0b00004d50 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:28:49.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b19a3e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:49.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b19a3e640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f0b14131740 con 0x7f0b00003fd0 2024-04-26T00:28:49.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b19a3e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:49.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b19a3e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0b141315c0 con 0x7f0b00004d50 2024-04-26T00:28:49.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b19a3e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:49.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b19a3e640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0b140a1be0 con 0x7f0af40072d0 2024-04-26T00:28:49.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b19a3e640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:49.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b19a3e640 10 monclient(hunting): authenticate will time out at 1510.526026s 2024-04-26T00:28:49.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b12ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0af40072d0 0x7f0b141cca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:49.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b137fe640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b00004d50 msgr2=0x7f0b1412ed30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:28:49.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b137fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b00004d50 0x7f0b1412ed30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:28:49.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b127fc640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0b00003fd0 0x7f0b1412e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:49.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b12ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0af40072d0 0x7f0b141cca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:58558/0 (socket says 172.21.15.37:58558) 2024-04-26T00:28:49.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b12ffd640 1 -- 172.21.15.37:0/2714591515 learned_addr learned my addr 172.21.15.37:0/2714591515 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:28:49.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b12ffd640 10 monclient(hunting): get_auth_request con 0x7f0af40072d0 auth_method 0 2024-04-26T00:28:49.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b12ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:49.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b12ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:49.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:49.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b127fc640 10 monclient(hunting): get_auth_request con 0x7f0b00003fd0 auth_method 0 2024-04-26T00:28:49.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b127fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:49.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b127fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:49.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b127fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:49.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b127fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:49.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b127fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:49.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b127fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:49.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b12ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:49.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b12ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:49.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b12ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:49.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b12ffd640 10 monclient(hunting): handle_auth_done global_id 84356 payload 824 2024-04-26T00:28:49.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b12ffd640 1 -- 172.21.15.37:0/2714591515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b00004d50 msgr2=0x7f0b1412ed30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:28:49.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b12ffd640 1 --2- 172.21.15.37:0/2714591515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b00004d50 0x7f0b1412ed30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:49.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b12ffd640 1 -- 172.21.15.37:0/2714591515 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0b00003fd0 msgr2=0x7f0b1412e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:49.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b12ffd640 1 --2- 172.21.15.37:0/2714591515 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0b00003fd0 0x7f0b1412e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:49.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b12ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:28:49.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b12ffd640 1 monclient: found mon.noname-c 2024-04-26T00:28:49.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:49.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b12ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:28:49.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.218+0000 7f0b12ffd640 1 -- 172.21.15.37:0/2714591515 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b141ccf80 con 0x7f0af40072d0 2024-04-26T00:28:49.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.222+0000 7f0b12ffd640 1 -- 172.21.15.37:0/2714591515 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0af40072d0 msgr2=0x7f0b141cca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:49.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.222+0000 7f0b12ffd640 1 --2- 172.21.15.37:0/2714591515 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0af40072d0 0x7f0b141cca40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0afc01b410 tx=0x7f0b00005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:49.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.222+0000 7f0b0bfff640 1 -- 172.21.15.37:0/2714591515 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0afc0a5d70 con 0x7f0af40072d0 2024-04-26T00:28:49.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.222+0000 7f0b0bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:49.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.222+0000 7f0b0bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:28:49.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.222+0000 7f0b0bfff640 10 monclient: dump: 2024-04-26T00:28:49.218 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:49.218 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:49.219 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:49.219 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:49.219 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:49.219 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:49.219 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:49.219 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:49.219 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:49.219 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:49.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.222+0000 7f0b0bfff640 10 monclient: _finish_auth 0 2024-04-26T00:28:49.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.222+0000 7f0b0bfff640 10 monclient: _check_auth_tickets 2024-04-26T00:28:49.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.222+0000 7f0b0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:49.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.222+0000 7f0b0bfff640 1 -- 172.21.15.37:0/2714591515 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0afc004200 con 0x7f0af40072d0 2024-04-26T00:28:49.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.222+0000 7f0b0bfff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:28:49.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.222+0000 7f0b0bfff640 1 -- 172.21.15.37:0/2714591515 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0afc006290 con 0x7f0af40072d0 2024-04-26T00:28:49.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.222+0000 7f0b0bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:49.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.222+0000 7f0b0bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:28:49.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.222+0000 7f0b0bfff640 10 monclient: dump: 2024-04-26T00:28:49.220 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:49.220 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:49.220 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:49.220 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:49.220 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:49.220 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:49.220 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:49.220 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:49.220 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:49.220 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:49.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.222+0000 7f0b19a3e640 5 monclient: authenticate success, global_id 84356 2024-04-26T00:28:49.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.222+0000 7f0b19a3e640 10 monclient: _renew_subs 2024-04-26T00:28:49.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.222+0000 7f0b19a3e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:28:49.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.222+0000 7f0b19a3e640 1 -- 172.21.15.37:0/2714591515 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b141cd210 con 0x7f0af40072d0 2024-04-26T00:28:49.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.222+0000 7f0b117fa640 4 set_mon_vals no callback set 2024-04-26T00:28:49.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.222+0000 7f0b19a3e640 10 monclient: _renew_subs 2024-04-26T00:28:49.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.222+0000 7f0b19a3e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:28:49.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.222+0000 7f0b19a3e640 1 -- 172.21.15.37:0/2714591515 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b1409dfd0 con 0x7f0af40072d0 2024-04-26T00:28:49.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.222+0000 7f0b0bfff640 1 -- 172.21.15.37:0/2714591515 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f0afc0046e0 con 0x7f0af40072d0 2024-04-26T00:28:49.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.222+0000 7f0b0bfff640 1 --2- 172.21.15.37:0/2714591515 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0af4036520 0x7f0af40389e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:49.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.222+0000 7f0b0bfff640 1 -- 172.21.15.37:0/2714591515 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(340..340 src has 177..340) ==== 3975+0+0 (secure 0 0 0) 0x7f0afc0e22c0 con 0x7f0af40072d0 2024-04-26T00:28:49.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.222+0000 7f0b19a3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:49.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.222+0000 7f0b19a3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:49.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.222+0000 7f0b19a3e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:28:49.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.222+0000 7f0b19a3e640 1 -- 172.21.15.37:0/2714591515 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b14066f40 con 0x7f0af40072d0 2024-04-26T00:28:49.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.222+0000 7f0b127fc640 1 --2- 172.21.15.37:0/2714591515 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0af4036520 0x7f0af40389e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:49.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.222+0000 7f0b127fc640 10 monclient: get_auth_request con 0x7f0af4036520 auth_method 0 2024-04-26T00:28:49.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.226+0000 7f0b127fc640 1 --2- 172.21.15.37:0/2714591515 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0af4036520 0x7f0af40389e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0b14130360 tx=0x7f0b04008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:49.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.230+0000 7f0b0bfff640 1 -- 172.21.15.37:0/2714591515 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f0afc016a70 con 0x7f0af40072d0 2024-04-26T00:28:49.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.230+0000 7f0b0bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:49.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.230+0000 7f0b0bfff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:28:49.385 INFO:tasks.rados.rados.0.smithi045.stdout:1478: done (0 left) 2024-04-26T00:28:49.385 INFO:tasks.rados.rados.0.smithi045.stdout:1479: snap_create 2024-04-26T00:28:49.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.474+0000 7f0b19a3e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:28:49.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.474+0000 7f0b19a3e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:28:49.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.474+0000 7f0b19a3e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:28:49.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.474+0000 7f0b19a3e640 1 -- 172.21.15.37:0/2714591515 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0b1406c1e0 con 0x7f0af40072d0 2024-04-26T00:28:49.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.474+0000 7f0b0bfff640 1 -- 172.21.15.37:0/2714591515 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7f0afc0043b0 con 0x7f0af40072d0 2024-04-26T00:28:49.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.474+0000 7f0b0bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:28:49.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.474+0000 7f0b0bfff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:28:49.464 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":68,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:28:49.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.478+0000 7f0b19a3e640 1 -- 172.21.15.37:0/2714591515 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0af4036520 msgr2=0x7f0af40389e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:49.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.478+0000 7f0b19a3e640 1 --2- 172.21.15.37:0/2714591515 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0af4036520 0x7f0af40389e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0b14130360 tx=0x7f0b04008040 comp rx=0 tx=0).stop 2024-04-26T00:28:49.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.478+0000 7f0b19a3e640 10 monclient: shutdown 2024-04-26T00:28:49.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.478+0000 7f0b19a3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:28:49.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.478+0000 7f0b19a3e640 1 -- 172.21.15.37:0/2714591515 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0af40072d0 msgr2=0x7f0b141cca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:49.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.478+0000 7f0b19a3e640 1 --2- 172.21.15.37:0/2714591515 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0af40072d0 0x7f0b141cca40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0afc01b410 tx=0x7f0b00005840 comp rx=0 tx=0).stop 2024-04-26T00:28:49.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.478+0000 7f0b19a3e640 1 -- 172.21.15.37:0/2714591515 shutdown_connections 2024-04-26T00:28:49.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.478+0000 7f0b19a3e640 1 --2- 172.21.15.37:0/2714591515 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0af4036520 0x7f0af40389e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:49.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.478+0000 7f0b19a3e640 1 --2- 172.21.15.37:0/2714591515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b00004d50 0x7f0b1412ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:49.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.478+0000 7f0b19a3e640 1 --2- 172.21.15.37:0/2714591515 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0b00003fd0 0x7f0b1412e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:49.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.478+0000 7f0b19a3e640 1 --2- 172.21.15.37:0/2714591515 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0af40072d0 0x7f0b141cca40 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:49.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.478+0000 7f0b19a3e640 1 -- 172.21.15.37:0/2714591515 >> 172.21.15.37:0/2714591515 conn(0x7f0b140a24c0 msgr2=0x7f0b140a4750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:28:49.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.478+0000 7f0b19a3e640 1 -- 172.21.15.37:0/2714591515 shutdown_connections 2024-04-26T00:28:49.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.478+0000 7f0b19a3e640 1 -- 172.21.15.37:0/2714591515 wait complete. 2024-04-26T00:28:49.483 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-26T00:28:49.483 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-04-26T00:28:49.484 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:28:49.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d08ce5640 10 monclient: get_monmap_and_config 2024-04-26T00:28:49.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d08ce5640 10 monclient: build_initial_monmap 2024-04-26T00:28:49.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d08ce5640 10 monclient: monmap: 2024-04-26T00:28:49.602 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:28:49.602 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:49.602 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:28:49.616910+0000 2024-04-26T00:28:49.602 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:28:49.616910+0000 2024-04-26T00:28:49.602 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:28:49.602 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:49.602 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:28:49.602 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:28:49.602 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:28:49.605 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:28:49.605 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:28:49.606 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:49.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d08ce5640 1 Processor -- start 2024-04-26T00:28:49.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d08ce5640 1 -- start start 2024-04-26T00:28:49.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d08ce5640 10 monclient: init 2024-04-26T00:28:49.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d08ce5640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:49.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d08ce5640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-04-26T00:28:49.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d08ce5640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f2d04136f20 0x7f2d0409bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:49.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d08ce5640 10 monclient(hunting): picked mon.noname-a con 0x7f2d04136f20 addr v2:172.21.15.37:3300/0 2024-04-26T00:28:49.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d08ce5640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2d04137300 0x7f2d041377b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:49.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d08ce5640 10 monclient(hunting): picked mon.noname-c con 0x7f2d04137300 addr v2:172.21.15.45:3300/0 2024-04-26T00:28:49.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d08ce5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2d04133c70 addr v1:172.21.15.45:6789/0 2024-04-26T00:28:49.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d08ce5640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d040a50f0 con 0x7f2d04133c70 2024-04-26T00:28:49.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d08ce5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:49.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d08ce5640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f2d0409b330 con 0x7f2d04136f20 2024-04-26T00:28:49.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d08ce5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:49.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d08ce5640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f2d0409b610 con 0x7f2d04137300 2024-04-26T00:28:49.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d08ce5640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:49.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d08ce5640 10 monclient(hunting): authenticate will time out at 1510.922531s 2024-04-26T00:28:49.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d0259c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f2d04136f20 0x7f2d0409bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:49.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d01d9b640 1 -- >> v2:172.21.15.45:3300/0 conn(0x7f2d04137300 msgr2=0x7f2d041377b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:28:49.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d01d9b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2d04137300 0x7f2d041377b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:28:49.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d0259c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f2d04136f20 0x7f2d0409bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:59354/0 (socket says 172.21.15.37:59354) 2024-04-26T00:28:49.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d0259c640 1 -- 172.21.15.37:0/1362473790 learned_addr learned my addr 172.21.15.37:0/1362473790 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:28:49.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d0259c640 10 monclient(hunting): get_auth_request con 0x7f2d04136f20 auth_method 0 2024-04-26T00:28:49.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d0259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:49.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d0259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:49.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d0259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:49.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d02d9d640 1 -- 172.21.15.37:0/1362473790 >> v1:172.21.15.45:6789/0 conn(0x7f2d04133c70 legacy=0x7f2d04134120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:28:49.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d0259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:49.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d0259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:49.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d0159a640 1 -- 172.21.15.37:0/1362473790 DispatchQueue::entry inject delay of 0.002000 2024-04-26T00:28:49.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d0259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:49.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d0259c640 10 monclient(hunting): handle_auth_done global_id 94293 payload 274 2024-04-26T00:28:49.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d0259c640 1 -- 172.21.15.37:0/1362473790 >> v2:172.21.15.45:3300/0 conn(0x7f2d04137300 msgr2=0x7f2d041377b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:28:49.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d0259c640 1 --2- 172.21.15.37:0/1362473790 >> v2:172.21.15.45:3300/0 conn(0x7f2d04137300 0x7f2d041377b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:49.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d0259c640 1 -- 172.21.15.37:0/1362473790 >> v1:172.21.15.45:6789/0 conn(0x7f2d04133c70 legacy=0x7f2d04134120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:28:49.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d0259c640 10 monclient: _finish_hunting 0 2024-04-26T00:28:49.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d0259c640 1 monclient: found mon.noname-a 2024-04-26T00:28:49.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d0259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:49.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d0259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:28:49.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.614+0000 7f2d0259c640 1 -- 172.21.15.37:0/1362473790 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d0409b8f0 con 0x7f2d04136f20 2024-04-26T00:28:49.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.618+0000 7f2d0259c640 1 -- 172.21.15.37:0/1362473790 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d04136f20 msgr2=0x7f2d0409bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:49.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.618+0000 7f2d0259c640 1 --2- 172.21.15.37:0/1362473790 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d04136f20 0x7f2d0409bff0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2d040a5ec0 tx=0x7f2cf401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4d4e72eb7dbaddc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:49.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.618+0000 7f2d0159a640 1 -- 172.21.15.37:0/1362473790 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf40b2070 con 0x7f2d04136f20 2024-04-26T00:28:49.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.618+0000 7f2d0159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:49.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.618+0000 7f2d0159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:28:49.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.618+0000 7f2d0159a640 10 monclient: dump: 2024-04-26T00:28:49.616 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:49.616 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:49.616 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:49.616 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:49.616 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:49.616 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:49.616 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:49.616 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:49.616 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:49.616 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:49.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.618+0000 7f2d0159a640 10 monclient: _finish_auth 0 2024-04-26T00:28:49.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.618+0000 7f2d0159a640 10 monclient: _check_auth_tickets 2024-04-26T00:28:49.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.618+0000 7f2d0159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:49.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.618+0000 7f2d08ce5640 5 monclient: authenticate success, global_id 94293 2024-04-26T00:28:49.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.618+0000 7f2d08ce5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:28:49.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.618+0000 7f2d0159a640 1 -- 172.21.15.37:0/1362473790 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cf40a9070 con 0x7f2d04136f20 2024-04-26T00:28:49.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.618+0000 7f2d0159a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:28:49.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.618+0000 7f2d08ce5640 10 monclient: get_monmap_and_config success 2024-04-26T00:28:49.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.618+0000 7f2d0159a640 1 -- 172.21.15.37:0/1362473790 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf40ad880 con 0x7f2d04136f20 2024-04-26T00:28:49.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.618+0000 7f2d08ce5640 4 set_mon_vals no callback set 2024-04-26T00:28:49.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.618+0000 7f2d0159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:49.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.618+0000 7f2d0159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:28:49.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.618+0000 7f2d0159a640 10 monclient: dump: 2024-04-26T00:28:49.617 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:49.617 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:49.617 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:49.617 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:49.618 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:49.618 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:49.618 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:49.618 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:49.618 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:49.618 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:49.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d08ce5640 10 monclient: shutdown 2024-04-26T00:28:49.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d08ce5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:28:49.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d08ce5640 1 -- 172.21.15.37:0/1362473790 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d04136f20 msgr2=0x7f2d0409bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:49.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d08ce5640 1 --2- 172.21.15.37:0/1362473790 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d04136f20 0x7f2d0409bff0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2d040a5ec0 tx=0x7f2cf401c450 comp rx=0 tx=0).stop 2024-04-26T00:28:49.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d08ce5640 1 -- 172.21.15.37:0/1362473790 shutdown_connections 2024-04-26T00:28:49.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d08ce5640 1 --2- 172.21.15.37:0/1362473790 >> v2:172.21.15.45:3300/0 conn(0x7f2d04137300 0x7f2d041377b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:49.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d08ce5640 1 --2- 172.21.15.37:0/1362473790 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d04136f20 0x7f2d0409bff0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:49.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d08ce5640 1 -- 172.21.15.37:0/1362473790 >> 172.21.15.37:0/1362473790 conn(0x7f2d040a61b0 msgr2=0x7f2d040a65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:28:49.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d08ce5640 1 -- 172.21.15.37:0/1362473790 shutdown_connections 2024-04-26T00:28:49.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d08ce5640 1 -- 172.21.15.37:0/1362473790 wait complete. 2024-04-26T00:28:49.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d08ce5640 10 monclient: build_initial_monmap 2024-04-26T00:28:49.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d08ce5640 10 monclient: monmap: 2024-04-26T00:28:49.619 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:28:49.619 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:28:49.619 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:28:49.619 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:28:49.619 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:28:49.619 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:28:49.619 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:28:49.619 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:28:49.619 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:28:49.620 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:49.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d08ce5640 1 Processor -- start 2024-04-26T00:28:49.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d08ce5640 1 -- start start 2024-04-26T00:28:49.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d08ce5640 10 monclient: init 2024-04-26T00:28:49.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d08ce5640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:49.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d08ce5640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:28:49.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d08ce5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d04133c70 0x7f2d041cf490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:49.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d08ce5640 10 monclient(hunting): picked mon.noname-b con 0x7f2d04133c70 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:28:49.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d08ce5640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2d04136f20 0x7f2d0413e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:49.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d08ce5640 10 monclient(hunting): picked mon.noname-c con 0x7f2d04136f20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:28:49.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d08ce5640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d04137300 0x7f2d0413e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:49.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d08ce5640 10 monclient(hunting): picked mon.noname-a con 0x7f2d04137300 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:28:49.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d08ce5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:49.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d08ce5640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f2d04141e10 con 0x7f2d04137300 2024-04-26T00:28:49.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d08ce5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:49.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d08ce5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2d04141c90 con 0x7f2d04133c70 2024-04-26T00:28:49.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d08ce5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:49.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d08ce5640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2d04141f90 con 0x7f2d04136f20 2024-04-26T00:28:49.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d08ce5640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:49.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d08ce5640 10 monclient(hunting): authenticate will time out at 1510.929766s 2024-04-26T00:28:49.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d0259c640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d04133c70 msgr2=0x7f2d041cf490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:28:49.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d02d9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d04137300 0x7f2d0413e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:49.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d0259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d04133c70 0x7f2d041cf490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:28:49.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d02d9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d04137300 0x7f2d0413e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:59366/0 (socket says 172.21.15.37:59366) 2024-04-26T00:28:49.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d02d9d640 1 -- 172.21.15.37:0/3321054007 learned_addr learned my addr 172.21.15.37:0/3321054007 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:28:49.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d02d9d640 10 monclient(hunting): get_auth_request con 0x7f2d04137300 auth_method 0 2024-04-26T00:28:49.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d02d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:49.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d02d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:49.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d02d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:49.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d01d9b640 1 --2- 172.21.15.37:0/3321054007 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2d04136f20 0x7f2d0413e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:49.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d02d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:49.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d02d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:49.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d02d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:49.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d01d9b640 10 monclient(hunting): get_auth_request con 0x7f2d04136f20 auth_method 0 2024-04-26T00:28:49.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d01d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:49.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d01d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:49.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d01d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:49.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d02d9d640 10 monclient(hunting): handle_auth_done global_id 94296 payload 824 2024-04-26T00:28:49.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d02d9d640 1 -- 172.21.15.37:0/3321054007 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2d04136f20 msgr2=0x7f2d0413e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:49.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d02d9d640 1 --2- 172.21.15.37:0/3321054007 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2d04136f20 0x7f2d0413e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:49.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d02d9d640 1 -- 172.21.15.37:0/3321054007 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d04133c70 msgr2=0x7f2d041cf490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:28:49.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d02d9d640 1 --2- 172.21.15.37:0/3321054007 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2d04133c70 0x7f2d041cf490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:49.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d02d9d640 10 monclient: _finish_hunting 0 2024-04-26T00:28:49.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d02d9d640 1 monclient: found mon.noname-a 2024-04-26T00:28:49.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d02d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:49.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d02d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:28:49.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d02d9d640 1 -- 172.21.15.37:0/3321054007 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d0413cb70 con 0x7f2d04137300 2024-04-26T00:28:49.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.622+0000 7f2d01d9b640 1 --2- 172.21.15.37:0/3321054007 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2d04136f20 0x7f2d0413e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:28:49.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2d02d9d640 1 -- 172.21.15.37:0/3321054007 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d04137300 msgr2=0x7f2d0413e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:49.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2d02d9d640 1 --2- 172.21.15.37:0/3321054007 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d04137300 0x7f2d0413e7a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc00be50 tx=0x7f2cfc00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:49.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2cf2ffd640 1 -- 172.21.15.37:0/3321054007 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc016630 con 0x7f2d04137300 2024-04-26T00:28:49.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2cf2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:49.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2cf2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:28:49.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2cf2ffd640 10 monclient: dump: 2024-04-26T00:28:49.623 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:49.623 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:49.623 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:49.624 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:49.624 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:49.624 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:49.624 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:49.624 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:49.624 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:49.624 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:49.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2cf2ffd640 10 monclient: _finish_auth 0 2024-04-26T00:28:49.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2cf2ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:28:49.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2cf2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:49.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2cf2ffd640 1 -- 172.21.15.37:0/3321054007 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cfc010030 con 0x7f2d04137300 2024-04-26T00:28:49.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2cf2ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:28:49.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2cf2ffd640 1 -- 172.21.15.37:0/3321054007 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc016a10 con 0x7f2d04137300 2024-04-26T00:28:49.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2d08ce5640 5 monclient: authenticate success, global_id 94296 2024-04-26T00:28:49.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2d08ce5640 10 monclient: _renew_subs 2024-04-26T00:28:49.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2d08ce5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:28:49.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2d08ce5640 1 -- 172.21.15.37:0/3321054007 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d0413b160 con 0x7f2d04137300 2024-04-26T00:28:49.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2cf2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:49.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2cf2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:28:49.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2d00d99640 4 set_mon_vals no callback set 2024-04-26T00:28:49.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2cf2ffd640 10 monclient: dump: 2024-04-26T00:28:49.625 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:49.625 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:49.625 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:49.625 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:49.625 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:49.625 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:49.625 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:49.625 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:49.626 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:49.626 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:49.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2d08ce5640 10 monclient: _renew_subs 2024-04-26T00:28:49.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2d08ce5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:28:49.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2d08ce5640 1 -- 172.21.15.37:0/3321054007 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d040a9190 con 0x7f2d04137300 2024-04-26T00:28:49.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2d08ce5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:49.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2d08ce5640 1 --2- 172.21.15.37:0/3321054007 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d04138be0 0x7f2d04138fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:49.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2d08ce5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:49.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2d08ce5640 1 -- 172.21.15.37:0/3321054007 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f2d0404f510 con 0x7f2d04138be0 2024-04-26T00:28:49.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2d0259c640 1 --2- 172.21.15.37:0/3321054007 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d04138be0 0x7f2d04138fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:49.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2d0259c640 10 monclient: get_auth_request con 0x7f2d04138be0 auth_method 0 2024-04-26T00:28:49.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2d0259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:49.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2d0259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:49.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2d0259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:49.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2d0259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:49.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2d0259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:49.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2d0259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:49.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2d0259c640 10 monclient(hunting): handle_auth_done global_id 94299 payload 824 2024-04-26T00:28:49.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2cf2ffd640 1 -- 172.21.15.37:0/3321054007 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2cfc0153e0 con 0x7f2d04137300 2024-04-26T00:28:49.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2d0259c640 1 -- 172.21.15.37:0/3321054007 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2d0413a880 con 0x7f2d04138be0 2024-04-26T00:28:49.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2cf2ffd640 1 --2- 172.21.15.37:0/3321054007 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2ce4035e80 0x7f2ce4038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:49.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2cf2ffd640 1 -- 172.21.15.37:0/3321054007 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(341..341 src has 177..341) ==== 3899+0+0 (secure 0 0 0) 0x7f2cfc049670 con 0x7f2d04137300 2024-04-26T00:28:49.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2d01d9b640 1 --2- 172.21.15.37:0/3321054007 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2ce4035e80 0x7f2ce4038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:49.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.626+0000 7f2d01d9b640 10 monclient: get_auth_request con 0x7f2ce4035e80 auth_method 0 2024-04-26T00:28:49.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.630+0000 7f2d0259c640 1 -- 172.21.15.37:0/3321054007 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d04138be0 msgr2=0x7f2d04138fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:49.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.630+0000 7f2d0259c640 1 --2- 172.21.15.37:0/3321054007 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d04138be0 0x7f2d04138fc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2cf40af000 tx=0x7f2cf401ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:49.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.630+0000 7f2cf2ffd640 1 -- 172.21.15.37:0/3321054007 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf40bc070 con 0x7f2d04138be0 2024-04-26T00:28:49.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.630+0000 7f2cf2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:49.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.630+0000 7f2cf2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:28:49.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.630+0000 7f2cf2ffd640 10 monclient: dump: 2024-04-26T00:28:49.630 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:49.630 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:49.630 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:49.630 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:49.630 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:49.630 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:49.631 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:49.631 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:49.631 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:49.631 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:49.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.630+0000 7f2cf2ffd640 1 -- 172.21.15.37:0/3321054007 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2cf40ad870 con 0x7f2d04138be0 2024-04-26T00:28:49.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.630+0000 7f2cf2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:49.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.630+0000 7f2cf2ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:28:49.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.630+0000 7f2cf2ffd640 1 -- 172.21.15.37:0/3321054007 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d04138be0 msgr2=0x7f2d04138fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:49.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.630+0000 7f2cf2ffd640 1 --2- 172.21.15.37:0/3321054007 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d04138be0 0x7f2d04138fc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2cf40af000 tx=0x7f2cf401ac90 comp rx=0 tx=0).stop 2024-04-26T00:28:49.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.630+0000 7f2cf2ffd640 1 -- 172.21.15.37:0/3321054007 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d04138be0 msgr2=0x7f2d04138fc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:28:49.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.630+0000 7f2cf2ffd640 1 --2- 172.21.15.37:0/3321054007 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d04138be0 0x7f2d04138fc0 secure :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2cf40af000 tx=0x7f2cf401ac90 comp rx=0 tx=0).stop 2024-04-26T00:28:49.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.630+0000 7f2d01d9b640 1 --2- 172.21.15.37:0/3321054007 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2ce4035e80 0x7f2ce4038340 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2d040a6bf0 tx=0x7f2cf8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:49.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.634+0000 7f2d08ce5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:28:49.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.634+0000 7f2d08ce5640 1 --2- 172.21.15.37:0/3321054007 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d04139500 0x7f2d040a08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:49.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.634+0000 7f2d08ce5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:49.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.634+0000 7f2d08ce5640 1 -- 172.21.15.37:0/3321054007 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f2d04075f10 con 0x7f2d04139500 2024-04-26T00:28:49.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.634+0000 7f2d0259c640 1 --2- 172.21.15.37:0/3321054007 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d04139500 0x7f2d040a08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:49.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.634+0000 7f2d0259c640 10 monclient: get_auth_request con 0x7f2d04139500 auth_method 0 2024-04-26T00:28:49.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.634+0000 7f2d0259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:49.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.634+0000 7f2d0259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:49.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.634+0000 7f2d0259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:49.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.634+0000 7f2d0259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:49.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.634+0000 7f2d0259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:49.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.634+0000 7f2d0259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:49.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.634+0000 7f2d0259c640 10 monclient(hunting): handle_auth_done global_id 94302 payload 824 2024-04-26T00:28:49.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.634+0000 7f2d0259c640 1 -- 172.21.15.37:0/3321054007 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2d041398e0 con 0x7f2d04139500 2024-04-26T00:28:49.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.638+0000 7f2d0259c640 1 -- 172.21.15.37:0/3321054007 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d04139500 msgr2=0x7f2d040a08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:49.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.638+0000 7f2d0259c640 1 --2- 172.21.15.37:0/3321054007 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d04139500 0x7f2d040a08e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2cf401ab00 tx=0x7f2cf40191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:49.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.638+0000 7f2cf2ffd640 1 -- 172.21.15.37:0/3321054007 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf40bc040 con 0x7f2d04139500 2024-04-26T00:28:49.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.638+0000 7f2cf2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:49.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.638+0000 7f2cf2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:28:49.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.638+0000 7f2cf2ffd640 10 monclient: dump: 2024-04-26T00:28:49.633 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:49.633 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:49.633 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:49.633 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:49.633 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:49.633 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:49.633 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:49.633 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:49.633 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:49.633 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:49.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.638+0000 7f2cf2ffd640 1 -- 172.21.15.37:0/3321054007 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4707 (secure 0 0 0) 0x7f2cf40c1a70 con 0x7f2d04139500 2024-04-26T00:28:49.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.638+0000 7f2cf2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:28:49.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.638+0000 7f2cf2ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:28:49.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.638+0000 7f2cf2ffd640 1 -- 172.21.15.37:0/3321054007 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d04139500 msgr2=0x7f2d040a08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:49.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.638+0000 7f2cf2ffd640 1 --2- 172.21.15.37:0/3321054007 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d04139500 0x7f2d040a08e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2cf401ab00 tx=0x7f2cf40191f0 comp rx=0 tx=0).stop 2024-04-26T00:28:49.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.638+0000 7f2cf2ffd640 1 -- 172.21.15.37:0/3321054007 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d04139500 msgr2=0x7f2d040a08e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:28:49.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.638+0000 7f2cf2ffd640 1 --2- 172.21.15.37:0/3321054007 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d04139500 0x7f2d040a08e0 secure :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2cf401ab00 tx=0x7f2cf40191f0 comp rx=0 tx=0).stop 2024-04-26T00:28:49.634 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:28:49.634 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:28:49.634 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:28:49.634 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:28:49.634 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 68, 2024-04-26T00:28:49.635 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 90398, 2024-04-26T00:28:49.635 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:28:49.635 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:28:49.635 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:28:49.635 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:49.635 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:28:49.635 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:28:49.635 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:28:49.635 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:28:49.635 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:28:49.635 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:28:49.635 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:28:49.635 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:28:49.636 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:28:49.636 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:28:49.636 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:28:49.636 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:28:49.636 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:28:49.636 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:28:49.636 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:28:49.636 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:49.636 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:28:49.636 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:28:49.636 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:28:49.636 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:28:49.636 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:28:49.636 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:28:49.636 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:28:49.637 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:28:49.637 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:28:49.637 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:28:49.637 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:28:49.637 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:28:49.637 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:28:49.637 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:49.637 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:49.637 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:28:49.637 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:28:49.637 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:28:49.637 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:28:49.637 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:28:49.637 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:28:49.638 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:28:49.638 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:28:49.638 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:28:49.638 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:28:49.638 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:28:49.638 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:28:49.638 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:28:49.638 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:28:49.638 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:28:49.638 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:28:49.638 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:28:49.638 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:28:49.638 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:28:49.638 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:28:49.638 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:28:49.639 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:28:49.639 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:28:49.639 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:28:49.639 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:28:49.639 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:28:49.639 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:28:49.639 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:28:49.639 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:49.639 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:28:49.639 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:49.639 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:28:49.639 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:49.639 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:28:49.639 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:28:49.640 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:28:49.640 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:28:49.640 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:49.640 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:28:49.640 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:28:49.640 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:49.640 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:49.640 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:49.640 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:28:49.640 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:28:49.640 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:49.640 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:49.640 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:49.640 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:49.641 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:28:49.641 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:28:49.641 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:28:49.641 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:28:49.641 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:28:49.641 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:49.641 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:49.641 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:28:49.641 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:28:49.641 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:28:49.641 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:28:49.641 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:49.641 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:28:49.641 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:28:49.641 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:49.642 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:49.642 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:49.642 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:28:49.642 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:28:49.642 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:49.642 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:49.642 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:49.642 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:49.642 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:28:49.642 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:28:49.642 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:28:49.642 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:28:49.642 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:28:49.642 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:49.643 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:49.643 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:28:49.643 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:28:49.643 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:28:49.643 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:28:49.643 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:49.643 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:28:49.643 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:28:49.643 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:49.643 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:49.643 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:49.643 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:28:49.643 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:28:49.643 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:49.644 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:49.644 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:49.644 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:49.644 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:28:49.644 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:28:49.644 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:28:49.644 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:28:49.644 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:28:49.644 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:49.644 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:49.644 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:49.644 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:28:49.644 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:28:49.644 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:49.644 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:28:49.645 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:28:49.645 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:28:49.645 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:49.645 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:49.645 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:28:49.645 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:49.645 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:28:49.645 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:28:49.645 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:28:49.645 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:49.645 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:49.645 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:28:49.645 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:49.645 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:28:49.645 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:28:49.646 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:28:49.646 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:49.646 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:49.646 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:49.646 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:28:49.646 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:28:49.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.638+0000 7f2d08ce5640 1 -- 172.21.15.37:0/3321054007 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2ce4035e80 msgr2=0x7f2ce4038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:49.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.638+0000 7f2d08ce5640 1 --2- 172.21.15.37:0/3321054007 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2ce4035e80 0x7f2ce4038340 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2d040a6bf0 tx=0x7f2cf8008040 comp rx=0 tx=0).stop 2024-04-26T00:28:49.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.638+0000 7f2d08ce5640 10 monclient: shutdown 2024-04-26T00:28:49.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.638+0000 7f2d08ce5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:28:49.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.638+0000 7f2d08ce5640 1 -- 172.21.15.37:0/3321054007 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d04137300 msgr2=0x7f2d0413e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:49.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.638+0000 7f2d08ce5640 1 --2- 172.21.15.37:0/3321054007 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2d04137300 0x7f2d0413e7a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc00be50 tx=0x7f2cfc00be80 comp rx=0 tx=0).stop 2024-04-26T00:28:49.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.638+0000 7f2d02d9d640 1 -- 172.21.15.37:0/3321054007 reap_dead start 2024-04-26T00:28:49.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.638+0000 7f2d02d9d640 1 -- 172.21.15.37:0/3321054007 reap_dead start 2024-04-26T00:28:49.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.638+0000 7f2d08ce5640 1 -- 172.21.15.37:0/3321054007 shutdown_connections 2024-04-26T00:28:49.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.638+0000 7f2d08ce5640 1 -- 172.21.15.37:0/3321054007 >> 172.21.15.37:0/3321054007 conn(0x7f2d040a61b0 msgr2=0x7f2d04135a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:28:49.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.638+0000 7f2d08ce5640 1 -- 172.21.15.37:0/3321054007 shutdown_connections 2024-04-26T00:28:49.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.638+0000 7f2d08ce5640 1 -- 172.21.15.37:0/3321054007 wait complete. 2024-04-26T00:28:49.648 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:28:49.649 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-04-26T00:28:49.649 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:28:49.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505facd640 10 monclient: get_monmap_and_config 2024-04-26T00:28:49.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505facd640 10 monclient: build_initial_monmap 2024-04-26T00:28:49.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505facd640 10 monclient: monmap: 2024-04-26T00:28:49.768 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:28:49.768 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:49.768 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:28:49.780056+0000 2024-04-26T00:28:49.768 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:28:49.780056+0000 2024-04-26T00:28:49.768 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:28:49.768 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:49.769 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:28:49.769 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:28:49.769 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:28:49.769 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:28:49.769 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:28:49.769 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:49.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505facd640 1 Processor -- start 2024-04-26T00:28:49.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505facd640 1 -- start start 2024-04-26T00:28:49.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505facd640 10 monclient: init 2024-04-26T00:28:49.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505facd640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:49.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505facd640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-04-26T00:28:49.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505facd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f50580a3d90 0x7f505809bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:49.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505facd640 10 monclient(hunting): picked mon.noname-a con 0x7f50580a3d90 addr v2:172.21.15.37:3300/0 2024-04-26T00:28:49.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505facd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f50580a4170 0x7f50580a45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:49.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505facd640 10 monclient(hunting): picked mon.noname-b con 0x7f50580a4170 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:28:49.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505facd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5058133b20 addr v1:172.21.15.45:6789/0 2024-04-26T00:28:49.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505facd640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50580a5150 con 0x7f5058133b20 2024-04-26T00:28:49.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505facd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:49.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505facd640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f505809b280 con 0x7f50580a3d90 2024-04-26T00:28:49.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505facd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:49.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505facd640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f505809b560 con 0x7f50580a4170 2024-04-26T00:28:49.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505facd640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:49.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505facd640 10 monclient(hunting): authenticate will time out at 1511.085668s 2024-04-26T00:28:49.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505e06a640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f5058133b20 legacy=0x7f5058133f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:28:49.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505d068640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f50580a4170 0x7f50580a45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:49.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505d869640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f50580a3d90 0x7f505809bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:49.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505d068640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f50580a4170 0x7f50580a45b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51182/0 (socket says 172.21.15.37:51182) 2024-04-26T00:28:49.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505d068640 1 -- 172.21.15.37:0/1369990722 learned_addr learned my addr 172.21.15.37:0/1369990722 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:28:49.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505d869640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f50580a3d90 0x7f505809bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:49666/0 (socket says 172.21.15.37:49666) 2024-04-26T00:28:49.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505d068640 10 monclient(hunting): get_auth_request con 0x7f50580a4170 auth_method 0 2024-04-26T00:28:49.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505d068640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:49.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505d068640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:49.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505d068640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:49.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505d869640 10 monclient(hunting): get_auth_request con 0x7f50580a3d90 auth_method 0 2024-04-26T00:28:49.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505d869640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:49.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505d869640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:49.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505d869640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:49.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505d068640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:49.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505d068640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:49.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505d068640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:49.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505d869640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:49.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505d869640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:49.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505d869640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:49.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505d068640 10 monclient(hunting): handle_auth_done global_id 84362 payload 274 2024-04-26T00:28:49.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505d068640 1 -- 172.21.15.37:0/1369990722 >> v2:172.21.15.37:3300/0 conn(0x7f50580a3d90 msgr2=0x7f505809bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:28:49.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505d068640 1 --2- 172.21.15.37:0/1369990722 >> v2:172.21.15.37:3300/0 conn(0x7f50580a3d90 0x7f505809bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:49.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505d068640 1 -- 172.21.15.37:0/1369990722 >> v1:172.21.15.45:6789/0 conn(0x7f5058133b20 legacy=0x7f5058133f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:28:49.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505d068640 10 monclient: _finish_hunting 0 2024-04-26T00:28:49.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505d068640 1 monclient: found mon.noname-b 2024-04-26T00:28:49.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505d068640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:49.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505d068640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:28:49.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.778+0000 7f505d068640 1 -- 172.21.15.37:0/1369990722 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50580a4af0 con 0x7f50580a4170 2024-04-26T00:28:49.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.782+0000 7f505d068640 1 -- 172.21.15.37:0/1369990722 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f50580a4170 msgr2=0x7f50580a45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:49.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.782+0000 7f505d068640 1 --2- 172.21.15.37:0/1369990722 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f50580a4170 0x7f50580a45b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5048009870 tx=0x7f504801c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca4622445bd25546 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:49.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.782+0000 7f505c867640 1 -- 172.21.15.37:0/1369990722 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50480b0070 con 0x7f50580a4170 2024-04-26T00:28:49.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.782+0000 7f505c867640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:49.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.782+0000 7f505c867640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:28:49.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.782+0000 7f505c867640 10 monclient: dump: 2024-04-26T00:28:49.776 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:49.776 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:49.776 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:49.776 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:49.776 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:49.776 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:49.776 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:49.776 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:49.777 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:49.777 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:49.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.782+0000 7f505c867640 10 monclient: _finish_auth 0 2024-04-26T00:28:49.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.782+0000 7f505c867640 10 monclient: _check_auth_tickets 2024-04-26T00:28:49.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.782+0000 7f505c867640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:49.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.782+0000 7f505c867640 1 -- 172.21.15.37:0/1369990722 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50480ab600 con 0x7f50580a4170 2024-04-26T00:28:49.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.782+0000 7f505facd640 5 monclient: authenticate success, global_id 84362 2024-04-26T00:28:49.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.782+0000 7f505facd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:28:49.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.782+0000 7f505c867640 10 monclient: handle_config config(0 keys) 2024-04-26T00:28:49.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.782+0000 7f505c867640 1 -- 172.21.15.37:0/1369990722 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50480ab9d0 con 0x7f50580a4170 2024-04-26T00:28:49.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.782+0000 7f505c867640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:49.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.782+0000 7f505c867640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:28:49.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.782+0000 7f505c867640 10 monclient: dump: 2024-04-26T00:28:49.777 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:49.778 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:49.778 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:49.778 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:49.778 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:49.778 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:49.778 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:49.778 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:49.778 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:49.778 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:49.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.782+0000 7f505facd640 10 monclient: get_monmap_and_config success 2024-04-26T00:28:49.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.782+0000 7f505facd640 4 set_mon_vals no callback set 2024-04-26T00:28:49.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.782+0000 7f505facd640 10 monclient: shutdown 2024-04-26T00:28:49.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.782+0000 7f505facd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:28:49.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.782+0000 7f505facd640 1 -- 172.21.15.37:0/1369990722 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f50580a4170 msgr2=0x7f50580a45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:49.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.782+0000 7f505facd640 1 --2- 172.21.15.37:0/1369990722 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f50580a4170 0x7f50580a45b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5048009870 tx=0x7f504801c3e0 comp rx=0 tx=0).stop 2024-04-26T00:28:49.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.782+0000 7f505facd640 1 -- 172.21.15.37:0/1369990722 shutdown_connections 2024-04-26T00:28:49.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.782+0000 7f505facd640 1 --2- 172.21.15.37:0/1369990722 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f50580a4170 0x7f50580a45b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:49.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.782+0000 7f505facd640 1 --2- 172.21.15.37:0/1369990722 >> v2:172.21.15.37:3300/0 conn(0x7f50580a3d90 0x7f505809bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:49.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.782+0000 7f505facd640 1 -- 172.21.15.37:0/1369990722 >> 172.21.15.37:0/1369990722 conn(0x7f50580aa0a0 msgr2=0x7f50580a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:28:49.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.782+0000 7f505facd640 1 -- 172.21.15.37:0/1369990722 shutdown_connections 2024-04-26T00:28:49.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.782+0000 7f505facd640 1 -- 172.21.15.37:0/1369990722 wait complete. 2024-04-26T00:28:49.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505facd640 10 monclient: build_initial_monmap 2024-04-26T00:28:49.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505facd640 10 monclient: monmap: 2024-04-26T00:28:49.779 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:28:49.779 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:28:49.779 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:28:49.779 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:28:49.780 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:28:49.780 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:28:49.780 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:28:49.780 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:28:49.780 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:28:49.780 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:49.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505facd640 1 Processor -- start 2024-04-26T00:28:49.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505facd640 1 -- start start 2024-04-26T00:28:49.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505facd640 10 monclient: init 2024-04-26T00:28:49.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505facd640 10 monclient: _reopen_session rank -1 2024-04-26T00:28:49.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505facd640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:28:49.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505facd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f50580a3d90 0x7f50581cf400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:49.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505facd640 10 monclient(hunting): picked mon.noname-a con 0x7f50580a3d90 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:28:49.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505facd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50580a4170 0x7f505813e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:49.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505facd640 10 monclient(hunting): picked mon.noname-b con 0x7f50580a4170 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:28:49.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505facd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5058133b20 0x7f505813e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:49.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505facd640 10 monclient(hunting): picked mon.noname-c con 0x7f5058133b20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:28:49.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505facd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:49.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505facd640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f5058141e30 con 0x7f50580a3d90 2024-04-26T00:28:49.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505facd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:49.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505facd640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f505809c690 con 0x7f50580a4170 2024-04-26T00:28:49.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505facd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:49.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505facd640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5058141fb0 con 0x7f5058133b20 2024-04-26T00:28:49.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505facd640 10 monclient(hunting): _renew_subs 2024-04-26T00:28:49.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505facd640 10 monclient(hunting): authenticate will time out at 1511.093078s 2024-04-26T00:28:49.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505d869640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f50580a3d90 0x7f50581cf400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:49.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505e06a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5058133b20 0x7f505813e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:49.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505d869640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f50580a3d90 0x7f50581cf400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:49670/0 (socket says 172.21.15.37:49670) 2024-04-26T00:28:49.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505d869640 1 -- 172.21.15.37:0/1313785284 learned_addr learned my addr 172.21.15.37:0/1313785284 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:28:49.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505d869640 10 monclient(hunting): get_auth_request con 0x7f50580a3d90 auth_method 0 2024-04-26T00:28:49.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505e06a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5058133b20 0x7f505813e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51190/0 (socket says 172.21.15.37:51190) 2024-04-26T00:28:49.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505d869640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:49.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505d869640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:49.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505d869640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:49.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505e06a640 10 monclient(hunting): get_auth_request con 0x7f5058133b20 auth_method 0 2024-04-26T00:28:49.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505d068640 1 -- 172.21.15.37:0/1313785284 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50580a4170 msgr2=0x7f505813e310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:28:49.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505e06a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:49.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505e06a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:49.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505e06a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:49.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505d068640 1 --2- 172.21.15.37:0/1313785284 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50580a4170 0x7f505813e310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:28:49.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505d869640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:49.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505d869640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:49.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505d869640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:49.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505e06a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:49.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505e06a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:49.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505e06a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:49.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505d869640 10 monclient(hunting): handle_auth_done global_id 94308 payload 824 2024-04-26T00:28:49.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505d869640 1 -- 172.21.15.37:0/1313785284 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5058133b20 msgr2=0x7f505813e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:49.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505d869640 1 --2- 172.21.15.37:0/1313785284 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5058133b20 0x7f505813e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:49.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505d869640 1 -- 172.21.15.37:0/1313785284 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50580a4170 msgr2=0x7f505813e310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:28:49.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505d869640 1 --2- 172.21.15.37:0/1313785284 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50580a4170 0x7f505813e310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:28:49.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505d869640 10 monclient: _finish_hunting 0 2024-04-26T00:28:49.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505d869640 1 monclient: found mon.noname-a 2024-04-26T00:28:49.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505d869640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:28:49.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505d869640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:28:49.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505d869640 1 -- 172.21.15.37:0/1313785284 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f505813cbb0 con 0x7f50580a3d90 2024-04-26T00:28:49.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505e06a640 1 --2- 172.21.15.37:0/1313785284 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5058133b20 0x7f505813e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:28:49.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505d869640 1 -- 172.21.15.37:0/1313785284 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f50580a3d90 msgr2=0x7f50581cf400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:49.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.786+0000 7f505d869640 1 --2- 172.21.15.37:0/1313785284 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f50580a3d90 0x7f50581cf400 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f505400efc0 tx=0x7f505400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:49.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f504e7fc640 1 -- 172.21.15.37:0/1313785284 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5054018020 con 0x7f50580a3d90 2024-04-26T00:28:49.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f504e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:49.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f504e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:28:49.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f504e7fc640 10 monclient: dump: 2024-04-26T00:28:49.788 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:49.788 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:49.788 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:49.788 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:49.788 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:49.788 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:49.788 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:49.788 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:49.788 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:49.788 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:49.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f504e7fc640 10 monclient: _finish_auth 0 2024-04-26T00:28:49.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f504e7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:28:49.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f504e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:28:49.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f504e7fc640 1 -- 172.21.15.37:0/1313785284 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5054014680 con 0x7f50580a3d90 2024-04-26T00:28:49.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f505facd640 5 monclient: authenticate success, global_id 94308 2024-04-26T00:28:49.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f505facd640 10 monclient: _renew_subs 2024-04-26T00:28:49.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f505facd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:28:49.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f505facd640 1 -- 172.21.15.37:0/1313785284 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f505813b1a0 con 0x7f50580a3d90 2024-04-26T00:28:49.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f504e7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:28:49.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f504e7fc640 1 -- 172.21.15.37:0/1313785284 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5054014a50 con 0x7f50580a3d90 2024-04-26T00:28:49.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f504e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:49.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f504e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:28:49.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f504e7fc640 10 monclient: dump: 2024-04-26T00:28:49.789 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:49.789 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:49.789 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:49.789 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:49.790 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:49.790 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:49.790 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:49.790 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:49.790 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:49.790 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:49.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f505facd640 10 monclient: _renew_subs 2024-04-26T00:28:49.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f505facd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:28:49.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f504ffff640 4 set_mon_vals no callback set 2024-04-26T00:28:49.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f505facd640 1 -- 172.21.15.37:0/1313785284 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50580a0350 con 0x7f50580a3d90 2024-04-26T00:28:49.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f505facd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:49.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f505facd640 1 --2- 172.21.15.37:0/1313785284 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5058138c20 0x7f5058139000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:49.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f505facd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:49.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f505facd640 1 -- 172.21.15.37:0/1313785284 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f505804f510 con 0x7f5058138c20 2024-04-26T00:28:49.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f504e7fc640 1 -- 172.21.15.37:0/1313785284 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f505401b020 con 0x7f50580a3d90 2024-04-26T00:28:49.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f505d068640 1 --2- 172.21.15.37:0/1313785284 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5058138c20 0x7f5058139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:49.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f505d068640 10 monclient: get_auth_request con 0x7f5058138c20 auth_method 0 2024-04-26T00:28:49.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f505d068640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:49.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f505d068640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:49.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f505d068640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:49.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f505d068640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:49.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f505d068640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:49.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f505d068640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:49.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f504e7fc640 1 --2- 172.21.15.37:0/1313785284 >> v2:172.21.15.45:6812/2379918623 conn(0x7f503c035eb0 0x7f503c038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:49.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f504e7fc640 1 -- 172.21.15.37:0/1313785284 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(341..341 src has 177..341) ==== 3899+0+0 (secure 0 0 0) 0x7f5054049620 con 0x7f50580a3d90 2024-04-26T00:28:49.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f505d068640 10 monclient(hunting): handle_auth_done global_id 84368 payload 824 2024-04-26T00:28:49.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f505d068640 1 -- 172.21.15.37:0/1313785284 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f505813a8c0 con 0x7f5058138c20 2024-04-26T00:28:49.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f505e06a640 1 --2- 172.21.15.37:0/1313785284 >> v2:172.21.15.45:6812/2379918623 conn(0x7f503c035eb0 0x7f503c038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:49.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f505e06a640 10 monclient: get_auth_request con 0x7f503c035eb0 auth_method 0 2024-04-26T00:28:49.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f505d068640 1 -- 172.21.15.37:0/1313785284 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5058138c20 msgr2=0x7f5058139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:49.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f505d068640 1 --2- 172.21.15.37:0/1313785284 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5058138c20 0x7f5058139000 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f50480ad000 tx=0x7f504801ac20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:49.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f504e7fc640 1 -- 172.21.15.37:0/1313785284 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50480ba020 con 0x7f5058138c20 2024-04-26T00:28:49.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f504e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:49.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f504e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:28:49.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.790+0000 7f504e7fc640 10 monclient: dump: 2024-04-26T00:28:49.792 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:49.792 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:49.792 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:49.792 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:49.792 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:49.792 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:49.792 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:49.793 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:49.793 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:49.793 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:49.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.794+0000 7f504e7fc640 1 -- 172.21.15.37:0/1313785284 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f50480b6630 con 0x7f5058138c20 2024-04-26T00:28:49.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.794+0000 7f504e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:28:49.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.794+0000 7f504e7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:28:49.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.794+0000 7f504e7fc640 1 -- 172.21.15.37:0/1313785284 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5058138c20 msgr2=0x7f5058139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:49.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.794+0000 7f504e7fc640 1 --2- 172.21.15.37:0/1313785284 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5058138c20 0x7f5058139000 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f50480ad000 tx=0x7f504801ac20 comp rx=0 tx=0).stop 2024-04-26T00:28:49.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.794+0000 7f504e7fc640 1 -- 172.21.15.37:0/1313785284 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5058138c20 msgr2=0x7f5058139000 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:28:49.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.794+0000 7f504e7fc640 1 --2- 172.21.15.37:0/1313785284 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5058138c20 0x7f5058139000 secure :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f50480ad000 tx=0x7f504801ac20 comp rx=0 tx=0).stop 2024-04-26T00:28:49.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.794+0000 7f505e06a640 1 --2- 172.21.15.37:0/1313785284 >> v2:172.21.15.45:6812/2379918623 conn(0x7f503c035eb0 0x7f503c038370 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5050003850 tx=0x7f50500090e0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:49.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.798+0000 7f505facd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:28:49.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.798+0000 7f505facd640 1 --2- 172.21.15.37:0/1313785284 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5058139540 0x7f5058134b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:28:49.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.798+0000 7f505facd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:28:49.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.798+0000 7f505facd640 1 -- 172.21.15.37:0/1313785284 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5058075dc0 con 0x7f5058139540 2024-04-26T00:28:49.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.798+0000 7f505d068640 1 --2- 172.21.15.37:0/1313785284 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5058139540 0x7f5058134b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:28:49.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.798+0000 7f505d068640 10 monclient: get_auth_request con 0x7f5058139540 auth_method 0 2024-04-26T00:28:49.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.798+0000 7f505d068640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:28:49.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.798+0000 7f505d068640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:28:49.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.798+0000 7f505d068640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:28:49.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.798+0000 7f505d068640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:28:49.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.798+0000 7f505d068640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:28:49.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.798+0000 7f505d068640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:28:49.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.798+0000 7f505d068640 10 monclient(hunting): handle_auth_done global_id 84371 payload 824 2024-04-26T00:28:49.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.798+0000 7f505d068640 1 -- 172.21.15.37:0/1313785284 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5058139970 con 0x7f5058139540 2024-04-26T00:28:49.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.802+0000 7f505d068640 1 -- 172.21.15.37:0/1313785284 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5058139540 msgr2=0x7f5058134b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:28:49.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.802+0000 7f505d068640 1 --2- 172.21.15.37:0/1313785284 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5058139540 0x7f5058134b30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5048009870 tx=0x7f5048019180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:28:49.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.802+0000 7f504e7fc640 1 -- 172.21.15.37:0/1313785284 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50480c6070 con 0x7f5058139540 2024-04-26T00:28:49.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.802+0000 7f504e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:28:49.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.802+0000 7f504e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:28:49.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.802+0000 7f504e7fc640 10 monclient: dump: 2024-04-26T00:28:49.796 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:28:49.796 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:28:49.796 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:49.796 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:28:49.796 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:28:49.796 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:28:49.796 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:28:49.796 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:28:49.796 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:28:49.796 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:28:49.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.802+0000 7f504e7fc640 1 -- 172.21.15.37:0/1313785284 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f50480c1c70 con 0x7f5058139540 2024-04-26T00:28:49.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.802+0000 7f504e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:28:49.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.802+0000 7f504e7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:28:49.797 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:28:49.797 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:28:49.797 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:28:49.797 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:28:49.797 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 68, 2024-04-26T00:28:49.797 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 134738, 2024-04-26T00:28:49.797 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:28:49.797 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:28:49.797 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:28:49.797 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:49.798 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:28:49.798 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:28:49.798 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:28:49.798 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:28:49.798 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:28:49.798 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:28:49.798 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:28:49.798 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:28:49.798 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:28:49.798 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:28:49.798 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:28:49.798 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:28:49.798 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:28:49.799 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:28:49.799 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:28:49.799 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:49.799 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:28:49.799 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:28:49.799 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:28:49.799 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:28:49.799 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:28:49.799 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:28:49.799 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:28:49.799 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:28:49.799 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:28:49.799 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:28:49.800 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:28:49.800 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:28:49.800 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:28:49.800 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:49.800 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:49.800 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:28:49.800 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:28:49.800 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:28:49.800 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:28:49.800 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:28:49.800 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:28:49.800 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:28:49.800 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:28:49.800 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:28:49.800 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:28:49.801 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:28:49.801 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:28:49.801 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:28:49.801 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:28:49.801 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:28:49.801 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:28:49.801 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:28:49.801 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:28:49.801 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:28:49.801 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:28:49.801 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:28:49.801 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:28:49.801 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:28:49.801 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:28:49.802 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:28:49.802 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:28:49.802 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:28:49.802 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:28:49.802 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:49.802 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:28:49.802 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:49.802 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:28:49.802 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:49.802 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:28:49.802 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:28:49.802 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:28:49.802 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:28:49.802 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:49.803 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:28:49.803 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:28:49.803 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:49.803 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:49.803 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:49.803 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:28:49.803 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:28:49.803 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:49.803 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:49.803 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:49.803 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:49.803 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:28:49.803 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:28:49.803 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:28:49.804 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:28:49.804 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:28:49.804 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:49.804 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:49.804 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:28:49.804 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:28:49.804 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:28:49.804 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:28:49.804 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:49.804 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:28:49.804 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:28:49.804 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:49.804 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:49.804 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:49.805 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:28:49.805 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:28:49.805 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:49.805 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:49.805 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:49.805 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:49.805 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:28:49.805 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:28:49.805 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:28:49.805 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:28:49.805 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:28:49.805 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:49.805 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:49.805 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:28:49.806 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:28:49.806 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:28:49.806 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:28:49.806 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:49.806 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:28:49.806 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:28:49.806 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:49.806 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:49.806 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:49.806 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:28:49.806 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:28:49.806 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:28:49.806 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:49.806 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:49.806 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:49.807 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:28:49.807 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:28:49.807 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:28:49.807 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:28:49.807 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:28:49.807 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:49.807 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:49.807 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:49.807 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:28:49.807 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:28:49.807 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:49.807 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:28:49.807 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:28:49.807 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:28:49.808 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:49.808 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:49.808 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:28:49.808 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:49.808 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:28:49.808 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:28:49.808 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:28:49.808 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:49.808 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:49.808 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:28:49.808 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:49.808 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:28:49.808 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:28:49.808 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:28:49.809 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:49.809 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:28:49.809 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:28:49.809 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:28:49.809 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:28:49.809 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:28:49.809 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:28:49.809 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:28:49.809 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:28:49.809 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:28:49.809 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:28:49.809 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:28:49.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.802+0000 7f504e7fc640 1 -- 172.21.15.37:0/1313785284 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5058139540 msgr2=0x7f5058134b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:49.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.802+0000 7f504e7fc640 1 --2- 172.21.15.37:0/1313785284 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5058139540 0x7f5058134b30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5048009870 tx=0x7f5048019180 comp rx=0 tx=0).stop 2024-04-26T00:28:49.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.802+0000 7f504e7fc640 1 -- 172.21.15.37:0/1313785284 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5058139540 msgr2=0x7f5058134b30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:28:49.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.802+0000 7f504e7fc640 1 --2- 172.21.15.37:0/1313785284 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5058139540 0x7f5058134b30 secure :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5048009870 tx=0x7f5048019180 comp rx=0 tx=0).stop 2024-04-26T00:28:49.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.802+0000 7f505facd640 1 -- 172.21.15.37:0/1313785284 >> v2:172.21.15.45:6812/2379918623 conn(0x7f503c035eb0 msgr2=0x7f503c038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:49.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.802+0000 7f505facd640 1 --2- 172.21.15.37:0/1313785284 >> v2:172.21.15.45:6812/2379918623 conn(0x7f503c035eb0 0x7f503c038370 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5050003850 tx=0x7f50500090e0 comp rx=0 tx=0).stop 2024-04-26T00:28:49.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.802+0000 7f505facd640 10 monclient: shutdown 2024-04-26T00:28:49.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.802+0000 7f505facd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:28:49.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.802+0000 7f505facd640 1 -- 172.21.15.37:0/1313785284 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f50580a3d90 msgr2=0x7f50581cf400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:28:49.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.802+0000 7f505facd640 1 --2- 172.21.15.37:0/1313785284 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f50580a3d90 0x7f50581cf400 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f505400efc0 tx=0x7f505400c490 comp rx=0 tx=0).stop 2024-04-26T00:28:49.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.802+0000 7f505e06a640 1 -- 172.21.15.37:0/1313785284 reap_dead start 2024-04-26T00:28:49.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.802+0000 7f505e06a640 1 -- 172.21.15.37:0/1313785284 reap_dead start 2024-04-26T00:28:49.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.802+0000 7f505facd640 1 -- 172.21.15.37:0/1313785284 shutdown_connections 2024-04-26T00:28:49.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.802+0000 7f505facd640 1 -- 172.21.15.37:0/1313785284 >> 172.21.15.37:0/1313785284 conn(0x7f50580aa0a0 msgr2=0x7f50580a2400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:28:49.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.802+0000 7f505facd640 1 -- 172.21.15.37:0/1313785284 shutdown_connections 2024-04-26T00:28:49.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:28:49.802+0000 7f505facd640 1 -- 172.21.15.37:0/1313785284 wait complete. 2024-04-26T00:28:49.812 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:28:49.812 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-04-26T00:28:50.692 INFO:tasks.rados.rados.0.smithi045.stdout:1479: done (0 left) 2024-04-26T00:28:50.731 INFO:tasks.rados.rados.0.smithi045.stdout:1480: snap_remove snap 150 2024-04-26T00:28:51.695 INFO:tasks.rados.rados.0.smithi045.stdout:1480: done (0 left) 2024-04-26T00:28:51.695 INFO:tasks.rados.rados.0.smithi045.stdout:1481: copy_from oid 28 from oid 35 current snap is 171 2024-04-26T00:28:51.697 INFO:tasks.rados.rados.0.smithi045.stdout:1482: read oid 30 snap 146 2024-04-26T00:28:51.697 INFO:tasks.rados.rados.0.smithi045.stdout:1482: expect (ObjNum 447 snap 139 seq_num 447) 2024-04-26T00:28:51.697 INFO:tasks.rados.rados.0.smithi045.stdout:1483: read oid 24 snap -1 2024-04-26T00:28:51.697 INFO:tasks.rados.rados.0.smithi045.stdout:1483: expect (ObjNum 534 snap 165 seq_num 534) 2024-04-26T00:28:51.697 INFO:tasks.rados.rados.0.smithi045.stdout:1484: read oid 13 snap 167 2024-04-26T00:28:51.697 INFO:tasks.rados.rados.0.smithi045.stdout:1484: expect (ObjNum 337 snap 100 seq_num 337) 2024-04-26T00:28:51.697 INFO:tasks.rados.rados.0.smithi045.stdout:1485: read oid 25 snap -1 2024-04-26T00:28:51.697 INFO:tasks.rados.rados.0.smithi045.stdout:1485: expect (ObjNum 554 snap 168 seq_num 554) 2024-04-26T00:28:51.697 INFO:tasks.rados.rados.0.smithi045.stdout:1486: write oid 45 current snap is 171 2024-04-26T00:28:51.697 INFO:tasks.rados.rados.0.smithi045.stdout:1486: seq_num 555 ranges {527885=656964,1927271=483852,3191408=136477} 2024-04-26T00:28:51.706 INFO:tasks.rados.rados.0.smithi045.stdout:1486: writing smithi04526579-45 from 527885 to 1184849 tid 1 2024-04-26T00:28:51.717 INFO:tasks.rados.rados.0.smithi045.stdout:1486: writing smithi04526579-45 from 1927271 to 2411123 tid 2 2024-04-26T00:28:51.726 INFO:tasks.rados.rados.0.smithi045.stdout:1486: writing smithi04526579-45 from 3191408 to 3327885 tid 3 2024-04-26T00:28:51.726 INFO:tasks.rados.rados.0.smithi045.stdout:1487: copy_from oid 41 from oid 21 current snap is 171 2024-04-26T00:28:51.726 INFO:tasks.rados.rados.0.smithi045.stdout:1482: expect (ObjNum 447 snap 139 seq_num 447) 2024-04-26T00:28:51.933 INFO:tasks.rados.rados.0.smithi045.stdout:1487: finishing copy_from to smithi04526579-41 2024-04-26T00:28:51.933 INFO:tasks.rados.rados.0.smithi045.stdout:1487: got expected ENOENT (src dne) 2024-04-26T00:28:51.933 INFO:tasks.rados.rados.0.smithi045.stdout:1487: finishing copy_from racing read to smithi04526579-41 2024-04-26T00:28:51.933 INFO:tasks.rados.rados.0.smithi045.stdout:1486: finishing write tid 1 to smithi04526579-45 2024-04-26T00:28:51.933 INFO:tasks.rados.rados.0.smithi045.stdout:1486: finishing write tid 2 to smithi04526579-45 2024-04-26T00:28:51.933 INFO:tasks.rados.rados.0.smithi045.stdout:1486: finishing write tid 3 to smithi04526579-45 2024-04-26T00:28:51.933 INFO:tasks.rados.rados.0.smithi045.stdout:1486: finishing write tid 4 to smithi04526579-45 2024-04-26T00:28:51.933 INFO:tasks.rados.rados.0.smithi045.stdout:1486: finishing write tid 5 to smithi04526579-45 2024-04-26T00:28:51.934 INFO:tasks.rados.rados.0.smithi045.stdout:1486: oid 45 updating version 0 to 871 2024-04-26T00:28:51.934 INFO:tasks.rados.rados.0.smithi045.stdout:1486: oid 45 version 871 is already newer than 869 2024-04-26T00:28:51.934 INFO:tasks.rados.rados.0.smithi045.stdout:1486: oid 45 updating version 871 to 872 2024-04-26T00:28:51.934 INFO:tasks.rados.rados.0.smithi045.stdout:1486: oid 45 version 872 is already newer than 870 2024-04-26T00:28:51.934 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 872 (ObjNum 555 snap 171 seq_num 555) dirty exists 2024-04-26T00:28:51.934 INFO:tasks.rados.rados.0.smithi045.stdout:1486: left oid 45 (ObjNum 555 snap 171 seq_num 555) 2024-04-26T00:28:51.934 INFO:tasks.rados.rados.0.smithi045.stdout:1484: expect (ObjNum 337 snap 100 seq_num 337) 2024-04-26T00:28:52.103 INFO:tasks.rados.rados.0.smithi045.stdout:1483: expect (ObjNum 534 snap 165 seq_num 534) 2024-04-26T00:28:52.265 INFO:tasks.rados.rados.0.smithi045.stdout:1481: finishing copy_from racing read to smithi04526579-28 2024-04-26T00:28:52.265 INFO:tasks.rados.rados.0.smithi045.stdout:1485: expect (ObjNum 554 snap 168 seq_num 554) 2024-04-26T00:28:52.414 INFO:tasks.rados.rados.0.smithi045.stdout:1481: finishing copy_from to smithi04526579-28 2024-04-26T00:28:52.414 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 872 (ObjNum 553 snap 168 seq_num 553) dirty exists 2024-04-26T00:28:52.414 INFO:tasks.rados.rados.0.smithi045.stdout:1481: done (6 left) 2024-04-26T00:28:52.414 INFO:tasks.rados.rados.0.smithi045.stdout:1482: done (5 left) 2024-04-26T00:28:52.414 INFO:tasks.rados.rados.0.smithi045.stdout:1483: done (4 left) 2024-04-26T00:28:52.414 INFO:tasks.rados.rados.0.smithi045.stdout:1484: done (3 left) 2024-04-26T00:28:52.414 INFO:tasks.rados.rados.0.smithi045.stdout:1485: done (2 left) 2024-04-26T00:28:52.414 INFO:tasks.rados.rados.0.smithi045.stdout:1486: done (1 left) 2024-04-26T00:28:52.414 INFO:tasks.rados.rados.0.smithi045.stdout:1487: done (0 left) 2024-04-26T00:28:52.414 INFO:tasks.rados.rados.0.smithi045.stdout:1488: snap_create 2024-04-26T00:28:53.174 INFO:tasks.rados.rados.0.smithi045.stdout:1488: done (0 left) 2024-04-26T00:28:53.174 INFO:tasks.rados.rados.0.smithi045.stdout:1489: read oid 13 snap -1 2024-04-26T00:28:53.174 INFO:tasks.rados.rados.0.smithi045.stdout:1489: expect deleted 2024-04-26T00:28:53.174 INFO:tasks.rados.rados.0.smithi045.stdout:1490: snap_create 2024-04-26T00:28:54.187 INFO:tasks.rados.rados.0.smithi045.stdout:1489: done (1 left) 2024-04-26T00:28:54.187 INFO:tasks.rados.rados.0.smithi045.stdout:1490: done (0 left) 2024-04-26T00:28:54.187 INFO:tasks.rados.rados.0.smithi045.stdout:1491: write (excl) oid 37 current snap is 173 2024-04-26T00:28:54.187 INFO:tasks.rados.rados.0.smithi045.stdout:1491: seq_num 556 ranges {544692=732213,1985753=743863,3335184=9508} 2024-04-26T00:28:54.207 INFO:tasks.rados.rados.0.smithi045.stdout:1491: writing smithi04526579-37 from 544692 to 1276905 tid 1 2024-04-26T00:28:54.231 INFO:tasks.rados.rados.0.smithi045.stdout:1491: writing smithi04526579-37 from 1985753 to 2729616 tid 2 2024-04-26T00:28:54.241 INFO:tasks.rados.rados.0.smithi045.stdout:1491: writing smithi04526579-37 from 3335184 to 3344692 tid 3 2024-04-26T00:28:54.241 INFO:tasks.rados.rados.0.smithi045.stdout:1492: copy_from oid 6 from oid 41 current snap is 173 2024-04-26T00:28:54.241 INFO:tasks.rados.rados.0.smithi045.stdout:1491: finishing write tid 1 to smithi04526579-37 2024-04-26T00:28:54.241 INFO:tasks.rados.rados.0.smithi045.stdout:1491: finishing write tid 2 to smithi04526579-37 2024-04-26T00:28:54.241 INFO:tasks.rados.rados.0.smithi045.stdout:1493: snap_remove snap 159 2024-04-26T00:28:54.268 INFO:tasks.rados.rados.0.smithi045.stdout:1493: done (2 left) 2024-04-26T00:28:54.268 INFO:tasks.rados.rados.0.smithi045.stdout:1494: write oid 42 current snap is 173 2024-04-26T00:28:54.268 INFO:tasks.rados.rados.0.smithi045.stdout:1494: seq_num 557 ranges {561499=407462,1527882=487521,2729901=557217} 2024-04-26T00:28:54.284 INFO:tasks.rados.rados.0.smithi045.stdout:1494: writing smithi04526579-42 from 561499 to 968961 tid 1 2024-04-26T00:28:54.302 INFO:tasks.rados.rados.0.smithi045.stdout:1494: writing smithi04526579-42 from 1527882 to 2015403 tid 2 2024-04-26T00:28:54.319 INFO:tasks.rados.rados.0.smithi045.stdout:1494: writing smithi04526579-42 from 2729901 to 3287118 tid 3 2024-04-26T00:28:54.319 INFO:tasks.rados.rados.0.smithi045.stdout:1495: read oid 36 snap -1 2024-04-26T00:28:54.319 INFO:tasks.rados.rados.0.smithi045.stdout:1495: expect deleted 2024-04-26T00:28:54.319 INFO:tasks.rados.rados.0.smithi045.stdout:1496: write (excl) oid 23 current snap is 173 2024-04-26T00:28:54.319 INFO:tasks.rados.rados.0.smithi045.stdout:1496: seq_num 558 ranges {578306=482711,1586364=631179,2757324=484842} 2024-04-26T00:28:54.330 INFO:tasks.rados.rados.0.smithi045.stdout:1496: writing smithi04526579-23 from 578306 to 1061017 tid 1 2024-04-26T00:28:54.340 INFO:tasks.rados.rados.0.smithi045.stdout:1496: writing smithi04526579-23 from 1586364 to 2217543 tid 2 2024-04-26T00:28:54.349 INFO:tasks.rados.rados.0.smithi045.stdout:1496: writing smithi04526579-23 from 2757324 to 3242166 tid 3 2024-04-26T00:28:54.349 INFO:tasks.rados.rados.0.smithi045.stdout:1497: copy_from oid 31 from oid 16 current snap is 173 2024-04-26T00:28:54.349 INFO:tasks.rados.rados.0.smithi045.stdout:1491: finishing write tid 3 to smithi04526579-37 2024-04-26T00:28:54.349 INFO:tasks.rados.rados.0.smithi045.stdout:1491: finishing write tid 4 to smithi04526579-37 2024-04-26T00:28:54.350 INFO:tasks.rados.rados.0.smithi045.stdout:1491: finishing write tid 5 to smithi04526579-37 2024-04-26T00:28:54.350 INFO:tasks.rados.rados.0.smithi045.stdout:1491: oid 37 updating version 0 to 1153 2024-04-26T00:28:54.350 INFO:tasks.rados.rados.0.smithi045.stdout:1491: oid 37 version 1153 is already newer than 1152 2024-04-26T00:28:54.350 INFO:tasks.rados.rados.0.smithi045.stdout:1491: oid 37 updating version 1153 to 1155 2024-04-26T00:28:54.350 INFO:tasks.rados.rados.0.smithi045.stdout:1491: oid 37 version 1155 is already newer than 1154 2024-04-26T00:28:54.350 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 1155 (ObjNum 556 snap 173 seq_num 556) dirty exists 2024-04-26T00:28:54.350 INFO:tasks.rados.rados.0.smithi045.stdout:1491: left oid 37 (ObjNum 556 snap 173 seq_num 556) 2024-04-26T00:28:54.350 INFO:tasks.rados.rados.0.smithi045.stdout:1492: finishing copy_from racing read to smithi04526579-6 2024-04-26T00:28:54.350 INFO:tasks.rados.rados.0.smithi045.stdout:1492: finishing copy_from to smithi04526579-6 2024-04-26T00:28:54.350 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 1038 (ObjNum 394 snap 120 seq_num 394) dirty exists 2024-04-26T00:28:54.350 INFO:tasks.rados.rados.0.smithi045.stdout:1491: done (5 left) 2024-04-26T00:28:54.350 INFO:tasks.rados.rados.0.smithi045.stdout:1492: done (4 left) 2024-04-26T00:28:54.350 INFO:tasks.rados.rados.0.smithi045.stdout:1498: snap_create 2024-04-26T00:28:54.351 INFO:tasks.rados.rados.0.smithi045.stdout:1494: finishing write tid 1 to smithi04526579-42 2024-04-26T00:28:54.351 INFO:tasks.rados.rados.0.smithi045.stdout:1494: finishing write tid 2 to smithi04526579-42 2024-04-26T00:28:54.351 INFO:tasks.rados.rados.0.smithi045.stdout:1494: finishing write tid 3 to smithi04526579-42 2024-04-26T00:28:54.351 INFO:tasks.rados.rados.0.smithi045.stdout:1494: finishing write tid 4 to smithi04526579-42 2024-04-26T00:28:54.351 INFO:tasks.rados.rados.0.smithi045.stdout:1494: finishing write tid 5 to smithi04526579-42 2024-04-26T00:28:54.351 INFO:tasks.rados.rados.0.smithi045.stdout:1494: oid 42 updating version 0 to 894 2024-04-26T00:28:54.351 INFO:tasks.rados.rados.0.smithi045.stdout:1494: oid 42 version 894 is already newer than 893 2024-04-26T00:28:54.351 INFO:tasks.rados.rados.0.smithi045.stdout:1494: oid 42 version 894 is already newer than 888 2024-04-26T00:28:54.351 INFO:tasks.rados.rados.0.smithi045.stdout:1494: oid 42 updating version 894 to 895 2024-04-26T00:28:54.351 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 895 (ObjNum 557 snap 173 seq_num 557) dirty exists 2024-04-26T00:28:54.351 INFO:tasks.rados.rados.0.smithi045.stdout:1494: left oid 42 (ObjNum 557 snap 173 seq_num 557) 2024-04-26T00:28:54.351 INFO:tasks.rados.rados.0.smithi045.stdout:1496: finishing write tid 1 to smithi04526579-23 2024-04-26T00:28:54.351 INFO:tasks.rados.rados.0.smithi045.stdout:1496: finishing write tid 2 to smithi04526579-23 2024-04-26T00:28:54.355 INFO:tasks.rados.rados.0.smithi045.stdout:1496: finishing write tid 3 to smithi04526579-23 2024-04-26T00:28:54.356 INFO:tasks.rados.rados.0.smithi045.stdout:1496: finishing write tid 4 to smithi04526579-23 2024-04-26T00:28:54.357 INFO:tasks.rados.rados.0.smithi045.stdout:1496: finishing write tid 5 to smithi04526579-23 2024-04-26T00:28:54.357 INFO:tasks.rados.rados.0.smithi045.stdout:1496: oid 23 updating version 0 to 606 2024-04-26T00:28:54.357 INFO:tasks.rados.rados.0.smithi045.stdout:1496: oid 23 updating version 606 to 609 2024-04-26T00:28:54.357 INFO:tasks.rados.rados.0.smithi045.stdout:1496: oid 23 version 609 is already newer than 607 2024-04-26T00:28:54.357 INFO:tasks.rados.rados.0.smithi045.stdout:1496: oid 23 version 609 is already newer than 608 2024-04-26T00:28:54.357 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 609 (ObjNum 558 snap 173 seq_num 558) dirty exists 2024-04-26T00:28:54.357 INFO:tasks.rados.rados.0.smithi045.stdout:1496: left oid 23 (ObjNum 558 snap 173 seq_num 558) 2024-04-26T00:28:54.363 INFO:tasks.rados.rados.0.smithi045.stdout:1497: finishing copy_from racing read to smithi04526579-31 2024-04-26T00:28:54.366 INFO:tasks.rados.rados.0.smithi045.stdout:1497: finishing copy_from to smithi04526579-31 2024-04-26T00:28:54.366 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 618 (ObjNum 515 snap 157 seq_num 515) dirty exists 2024-04-26T00:28:55.271 INFO:tasks.rados.rados.0.smithi045.stdout:1494: done (4 left) 2024-04-26T00:28:55.271 INFO:tasks.rados.rados.0.smithi045.stdout:1495: done (3 left) 2024-04-26T00:28:55.272 INFO:tasks.rados.rados.0.smithi045.stdout:1496: done (2 left) 2024-04-26T00:28:55.272 INFO:tasks.rados.rados.0.smithi045.stdout:1497: done (1 left) 2024-04-26T00:28:55.272 INFO:tasks.rados.rados.0.smithi045.stdout:1498: done (0 left) 2024-04-26T00:28:55.272 INFO:tasks.rados.rados.0.smithi045.stdout:1499: read oid 38 snap 143 2024-04-26T00:28:55.272 INFO:tasks.rados.rados.0.smithi045.stdout:1499: expect (ObjNum 356 snap 105 seq_num 356) 2024-04-26T00:28:55.272 INFO:tasks.rados.rados.0.smithi045.stdout:1500: write oid 12 current snap is 174 2024-04-26T00:28:55.272 INFO:tasks.rados.rados.0.smithi045.stdout:1500: seq_num 559 ranges {595113=557960,1644846=491190,2784747=610366} 2024-04-26T00:28:55.280 INFO:tasks.rados.rados.0.smithi045.stdout:1500: writing smithi04526579-12 from 595113 to 1153073 tid 1 2024-04-26T00:28:55.289 INFO:tasks.rados.rados.0.smithi045.stdout:1500: writing smithi04526579-12 from 1644846 to 2136036 tid 2 2024-04-26T00:28:55.299 INFO:tasks.rados.rados.0.smithi045.stdout:1500: writing smithi04526579-12 from 2784747 to 3395113 tid 3 2024-04-26T00:28:55.300 INFO:tasks.rados.rados.0.smithi045.stdout:1501: snap_remove snap 108 2024-04-26T00:28:56.281 INFO:tasks.rados.rados.0.smithi045.stdout:1501: done (2 left) 2024-04-26T00:28:56.282 INFO:tasks.rados.rados.0.smithi045.stdout:1502: read oid 25 snap -1 2024-04-26T00:28:56.282 INFO:tasks.rados.rados.0.smithi045.stdout:1502: expect (ObjNum 554 snap 168 seq_num 554) 2024-04-26T00:28:56.282 INFO:tasks.rados.rados.0.smithi045.stdout:1503: copy_from oid 8 from oid 43 current snap is 174 2024-04-26T00:28:56.282 INFO:tasks.rados.rados.0.smithi045.stdout:1499: expect (ObjNum 356 snap 105 seq_num 356) 2024-04-26T00:28:56.562 INFO:tasks.rados.rados.0.smithi045.stdout:1500: finishing write tid 1 to smithi04526579-12 2024-04-26T00:28:56.563 INFO:tasks.rados.rados.0.smithi045.stdout:1500: finishing write tid 2 to smithi04526579-12 2024-04-26T00:28:56.563 INFO:tasks.rados.rados.0.smithi045.stdout:1500: finishing write tid 3 to smithi04526579-12 2024-04-26T00:28:56.563 INFO:tasks.rados.rados.0.smithi045.stdout:1500: finishing write tid 4 to smithi04526579-12 2024-04-26T00:28:56.563 INFO:tasks.rados.rados.0.smithi045.stdout:1500: finishing write tid 5 to smithi04526579-12 2024-04-26T00:28:56.563 INFO:tasks.rados.rados.0.smithi045.stdout:1500: oid 12 updating version 0 to 613 2024-04-26T00:28:56.563 INFO:tasks.rados.rados.0.smithi045.stdout:1500: oid 12 version 613 is already newer than 612 2024-04-26T00:28:56.563 INFO:tasks.rados.rados.0.smithi045.stdout:1500: oid 12 version 613 is already newer than 611 2024-04-26T00:28:56.563 INFO:tasks.rados.rados.0.smithi045.stdout:1500: oid 12 version 613 is already newer than 610 2024-04-26T00:28:56.563 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 613 (ObjNum 559 snap 174 seq_num 559) dirty exists 2024-04-26T00:28:56.563 INFO:tasks.rados.rados.0.smithi045.stdout:1500: left oid 12 (ObjNum 559 snap 174 seq_num 559) 2024-04-26T00:28:56.563 INFO:tasks.rados.rados.0.smithi045.stdout:1502: expect (ObjNum 554 snap 168 seq_num 554) 2024-04-26T00:28:56.712 INFO:tasks.rados.rados.0.smithi045.stdout:1503: finishing copy_from racing read to smithi04526579-8 2024-04-26T00:28:56.712 INFO:tasks.rados.rados.0.smithi045.stdout:1503: finishing copy_from to smithi04526579-8 2024-04-26T00:28:56.712 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 869 (ObjNum 551 snap 168 seq_num 551) dirty exists 2024-04-26T00:28:56.712 INFO:tasks.rados.rados.0.smithi045.stdout:1499: done (3 left) 2024-04-26T00:28:56.712 INFO:tasks.rados.rados.0.smithi045.stdout:1500: done (2 left) 2024-04-26T00:28:56.712 INFO:tasks.rados.rados.0.smithi045.stdout:1502: done (1 left) 2024-04-26T00:28:56.713 INFO:tasks.rados.rados.0.smithi045.stdout:1503: done (0 left) 2024-04-26T00:28:56.713 INFO:tasks.rados.rados.0.smithi045.stdout:1504: snap_remove snap 158 2024-04-26T00:28:57.279 INFO:tasks.rados.rados.0.smithi045.stdout:1504: done (0 left) 2024-04-26T00:28:57.279 INFO:tasks.rados.rados.0.smithi045.stdout:1505: read oid 3 snap -1 2024-04-26T00:28:57.279 INFO:tasks.rados.rados.0.smithi045.stdout:1505: expect (ObjNum 532 snap 165 seq_num 532) 2024-04-26T00:28:57.279 INFO:tasks.rados.rados.0.smithi045.stdout:1506: copy_from oid 21 from oid 44 current snap is 174 2024-04-26T00:28:57.279 INFO:tasks.rados.rados.0.smithi045.stdout:1507: read oid 26 snap -1 2024-04-26T00:28:57.279 INFO:tasks.rados.rados.0.smithi045.stdout:1507: expect (ObjNum 459 snap 143 seq_num 459) 2024-04-26T00:28:57.279 INFO:tasks.rados.rados.0.smithi045.stdout:1508: rollback oid 18 current snap is 174 2024-04-26T00:28:57.279 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 18 to 169 2024-04-26T00:28:57.279 INFO:tasks.rados.rados.0.smithi045.stdout:1509: write (excl) oid 17 current snap is 174 2024-04-26T00:28:57.279 INFO:tasks.rados.rados.0.smithi045.stdout:1509: seq_num 560 ranges {611920=633209,1703328=634848,2812170=507386} 2024-04-26T00:28:57.302 INFO:tasks.rados.rados.0.smithi045.stdout:1509: writing smithi04526579-17 from 611920 to 1245129 tid 1 2024-04-26T00:28:57.322 INFO:tasks.rados.rados.0.smithi045.stdout:1509: writing smithi04526579-17 from 1703328 to 2338176 tid 2 2024-04-26T00:28:57.336 INFO:tasks.rados.rados.0.smithi045.stdout:1509: writing smithi04526579-17 from 2812170 to 3319556 tid 3 2024-04-26T00:28:57.337 INFO:tasks.rados.rados.0.smithi045.stdout:1510: read oid 24 snap -1 2024-04-26T00:28:57.337 INFO:tasks.rados.rados.0.smithi045.stdout:1510: expect (ObjNum 534 snap 165 seq_num 534) 2024-04-26T00:28:57.337 INFO:tasks.rados.rados.0.smithi045.stdout:1511: snap_create 2024-04-26T00:28:57.337 INFO:tasks.rados.rados.0.smithi045.stdout:1505: expect (ObjNum 532 snap 165 seq_num 532) 2024-04-26T00:28:57.451 INFO:tasks.rados.rados.0.smithi045.stdout:1507: expect (ObjNum 459 snap 143 seq_num 459) 2024-04-26T00:28:57.573 INFO:tasks.rados.rados.0.smithi045.stdout:1508: finishing rollback tid 1 to smithi04526579-18 2024-04-26T00:28:57.574 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 833 (ObjNum 929 snap 0 seq_num 2241842544) dirty dne 2024-04-26T00:28:57.574 INFO:tasks.rados.rados.0.smithi045.stdout:1506: finishing copy_from racing read to smithi04526579-21 2024-04-26T00:28:57.574 INFO:tasks.rados.rados.0.smithi045.stdout:1509: finishing write tid 1 to smithi04526579-17 2024-04-26T00:28:57.574 INFO:tasks.rados.rados.0.smithi045.stdout:1506: finishing copy_from to smithi04526579-21 2024-04-26T00:28:57.574 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 868 (ObjNum 546 snap 168 seq_num 546) dirty exists 2024-04-26T00:28:57.574 INFO:tasks.rados.rados.0.smithi045.stdout:1509: finishing write tid 2 to smithi04526579-17 2024-04-26T00:28:57.574 INFO:tasks.rados.rados.0.smithi045.stdout:1509: finishing write tid 3 to smithi04526579-17 2024-04-26T00:28:57.574 INFO:tasks.rados.rados.0.smithi045.stdout:1509: finishing write tid 4 to smithi04526579-17 2024-04-26T00:28:57.575 INFO:tasks.rados.rados.0.smithi045.stdout:1509: finishing write tid 5 to smithi04526579-17 2024-04-26T00:28:57.575 INFO:tasks.rados.rados.0.smithi045.stdout:1509: oid 17 updating version 0 to 873 2024-04-26T00:28:57.575 INFO:tasks.rados.rados.0.smithi045.stdout:1509: oid 17 version 873 is already newer than 870 2024-04-26T00:28:57.575 INFO:tasks.rados.rados.0.smithi045.stdout:1509: oid 17 version 873 is already newer than 871 2024-04-26T00:28:57.575 INFO:tasks.rados.rados.0.smithi045.stdout:1509: oid 17 version 873 is already newer than 872 2024-04-26T00:28:57.575 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 873 (ObjNum 560 snap 174 seq_num 560) dirty exists 2024-04-26T00:28:57.575 INFO:tasks.rados.rados.0.smithi045.stdout:1509: left oid 17 (ObjNum 560 snap 174 seq_num 560) 2024-04-26T00:28:57.576 INFO:tasks.rados.rados.0.smithi045.stdout:1510: expect (ObjNum 534 snap 165 seq_num 534) 2024-04-26T00:28:58.297 INFO:tasks.rados.rados.0.smithi045.stdout:1505: done (6 left) 2024-04-26T00:28:58.297 INFO:tasks.rados.rados.0.smithi045.stdout:1506: done (5 left) 2024-04-26T00:28:58.298 INFO:tasks.rados.rados.0.smithi045.stdout:1507: done (4 left) 2024-04-26T00:28:58.298 INFO:tasks.rados.rados.0.smithi045.stdout:1508: done (3 left) 2024-04-26T00:28:58.298 INFO:tasks.rados.rados.0.smithi045.stdout:1509: done (2 left) 2024-04-26T00:28:58.298 INFO:tasks.rados.rados.0.smithi045.stdout:1510: done (1 left) 2024-04-26T00:28:58.298 INFO:tasks.rados.rados.0.smithi045.stdout:1511: done (0 left) 2024-04-26T00:28:58.298 INFO:tasks.rados.rados.0.smithi045.stdout:1512: write oid 1 current snap is 175 2024-04-26T00:28:58.298 INFO:tasks.rados.rados.0.smithi045.stdout:1512: seq_num 561 ranges {628727=708458,2045457=494859,3123240=305487} 2024-04-26T00:28:58.314 INFO:tasks.rados.rados.0.smithi045.stdout:1512: writing smithi04526579-1 from 628727 to 1337185 tid 1 2024-04-26T00:28:58.330 INFO:tasks.rados.rados.0.smithi045.stdout:1512: writing smithi04526579-1 from 2045457 to 2540316 tid 2 2024-04-26T00:28:58.342 INFO:tasks.rados.rados.0.smithi045.stdout:1512: writing smithi04526579-1 from 3123240 to 3428727 tid 3 2024-04-26T00:28:58.342 INFO:tasks.rados.rados.0.smithi045.stdout:1513: rollback oid 12 current snap is 175 2024-04-26T00:28:58.342 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 12 to 140 2024-04-26T00:28:58.342 INFO:tasks.rados.rados.0.smithi045.stdout:1512: finishing write tid 1 to smithi04526579-1 2024-04-26T00:28:58.342 INFO:tasks.rados.rados.0.smithi045.stdout:1512: finishing write tid 2 to smithi04526579-1 2024-04-26T00:28:58.342 INFO:tasks.rados.rados.0.smithi045.stdout:1514: write oid 33 current snap is 175 2024-04-26T00:28:58.342 INFO:tasks.rados.rados.0.smithi045.stdout:1514: seq_num 562 ranges {645534=783707,2103939=638517,3150663=294871} 2024-04-26T00:28:58.358 INFO:tasks.rados.rados.0.smithi045.stdout:1514: writing smithi04526579-33 from 645534 to 1429241 tid 1 2024-04-26T00:28:58.370 INFO:tasks.rados.rados.0.smithi045.stdout:1514: writing smithi04526579-33 from 2103939 to 2742456 tid 2 2024-04-26T00:28:58.376 INFO:tasks.rados.rados.0.smithi045.stdout:1514: writing smithi04526579-33 from 3150663 to 3445534 tid 3 2024-04-26T00:28:58.376 INFO:tasks.rados.rados.0.smithi045.stdout:1515: snap_remove snap 167 2024-04-26T00:28:59.268 INFO:tasks.rados.rados.0.smithi045.stdout:1515: done (3 left) 2024-04-26T00:28:59.268 INFO:tasks.rados.rados.0.smithi045.stdout:1516: rollback oid 45 current snap is 175 2024-04-26T00:28:59.268 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 45 to 143 2024-04-26T00:28:59.268 INFO:tasks.rados.rados.0.smithi045.stdout:1512: finishing write tid 3 to smithi04526579-1 2024-04-26T00:28:59.268 INFO:tasks.rados.rados.0.smithi045.stdout:1512: finishing write tid 4 to smithi04526579-1 2024-04-26T00:28:59.268 INFO:tasks.rados.rados.0.smithi045.stdout:1512: finishing write tid 5 to smithi04526579-1 2024-04-26T00:28:59.268 INFO:tasks.rados.rados.0.smithi045.stdout:1512: oid 1 updating version 0 to 1042 2024-04-26T00:28:59.269 INFO:tasks.rados.rados.0.smithi045.stdout:1512: oid 1 updating version 1042 to 1044 2024-04-26T00:28:59.269 INFO:tasks.rados.rados.0.smithi045.stdout:1512: oid 1 version 1044 is already newer than 1043 2024-04-26T00:28:59.269 INFO:tasks.rados.rados.0.smithi045.stdout:1512: oid 1 version 1044 is already newer than 1041 2024-04-26T00:28:59.269 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 1044 (ObjNum 561 snap 175 seq_num 561) dirty exists 2024-04-26T00:28:59.269 INFO:tasks.rados.rados.0.smithi045.stdout:1512: left oid 1 (ObjNum 561 snap 175 seq_num 561) 2024-04-26T00:28:59.269 INFO:tasks.rados.rados.0.smithi045.stdout:1513: finishing rollback tid 0 to smithi04526579-12 2024-04-26T00:28:59.269 INFO:tasks.rados.rados.0.smithi045.stdout:1513: finishing rollback tid 1 to smithi04526579-12 2024-04-26T00:28:59.270 INFO:tasks.rados.rados.0.smithi045.stdout:1513: finishing rollback tid 2 to smithi04526579-12 2024-04-26T00:28:59.270 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 616 (ObjNum 401 snap 122 seq_num 401) dirty exists 2024-04-26T00:28:59.270 INFO:tasks.rados.rados.0.smithi045.stdout:1514: finishing write tid 1 to smithi04526579-33 2024-04-26T00:28:59.270 INFO:tasks.rados.rados.0.smithi045.stdout:1514: finishing write tid 2 to smithi04526579-33 2024-04-26T00:28:59.270 INFO:tasks.rados.rados.0.smithi045.stdout:1514: finishing write tid 3 to smithi04526579-33 2024-04-26T00:28:59.270 INFO:tasks.rados.rados.0.smithi045.stdout:1514: finishing write tid 4 to smithi04526579-33 2024-04-26T00:28:59.270 INFO:tasks.rados.rados.0.smithi045.stdout:1514: finishing write tid 5 to smithi04526579-33 2024-04-26T00:28:59.270 INFO:tasks.rados.rados.0.smithi045.stdout:1514: oid 33 updating version 0 to 1187 2024-04-26T00:28:59.270 INFO:tasks.rados.rados.0.smithi045.stdout:1514: oid 33 version 1187 is already newer than 1184 2024-04-26T00:28:59.271 INFO:tasks.rados.rados.0.smithi045.stdout:1514: oid 33 version 1187 is already newer than 1185 2024-04-26T00:28:59.271 INFO:tasks.rados.rados.0.smithi045.stdout:1514: oid 33 version 1187 is already newer than 1186 2024-04-26T00:28:59.271 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 1187 (ObjNum 562 snap 175 seq_num 562) dirty exists 2024-04-26T00:28:59.271 INFO:tasks.rados.rados.0.smithi045.stdout:1514: left oid 33 (ObjNum 562 snap 175 seq_num 562) 2024-04-26T00:28:59.271 INFO:tasks.rados.rados.0.smithi045.stdout:1512: done (3 left) 2024-04-26T00:28:59.271 INFO:tasks.rados.rados.0.smithi045.stdout:1513: done (2 left) 2024-04-26T00:28:59.271 INFO:tasks.rados.rados.0.smithi045.stdout:1514: done (1 left) 2024-04-26T00:28:59.271 INFO:tasks.rados.rados.0.smithi045.stdout:1517: snap_remove snap 168 2024-04-26T00:29:00.270 INFO:tasks.rados.rados.0.smithi045.stdout:1517: done (1 left) 2024-04-26T00:29:00.270 INFO:tasks.rados.rados.0.smithi045.stdout:1518: read oid 23 snap 172 2024-04-26T00:29:00.270 INFO:tasks.rados.rados.0.smithi045.stdout:1518: expect (ObjNum 453 snap 141 seq_num 453) 2024-04-26T00:29:00.270 INFO:tasks.rados.rados.0.smithi045.stdout:1519: snap_remove snap 173 2024-04-26T00:29:00.271 INFO:tasks.rados.rados.0.smithi045.stdout:1516: finishing rollback tid 0 to smithi04526579-45 2024-04-26T00:29:00.271 INFO:tasks.rados.rados.0.smithi045.stdout:1516: finishing rollback tid 1 to smithi04526579-45 2024-04-26T00:29:00.271 INFO:tasks.rados.rados.0.smithi045.stdout:1516: finishing rollback tid 2 to smithi04526579-45 2024-04-26T00:29:00.271 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 875 (ObjNum 136 snap 30 seq_num 136) dirty exists 2024-04-26T00:29:01.278 INFO:tasks.rados.rados.0.smithi045.stdout:1516: done (2 left) 2024-04-26T00:29:01.278 INFO:tasks.rados.rados.0.smithi045.stdout:1519: done (1 left) 2024-04-26T00:29:01.278 INFO:tasks.rados.rados.0.smithi045.stdout:1520: write (excl) oid 9 current snap is 175 2024-04-26T00:29:01.278 INFO:tasks.rados.rados.0.smithi045.stdout:1520: seq_num 563 ranges {662341=575309,1878774=498528,2894439=457555} 2024-04-26T00:29:01.290 INFO:tasks.rados.rados.0.smithi045.stdout:1520: writing smithi04526579-9 from 662341 to 1237650 tid 1 2024-04-26T00:29:01.302 INFO:tasks.rados.rados.0.smithi045.stdout:1520: writing smithi04526579-9 from 1878774 to 2377302 tid 2 2024-04-26T00:29:01.312 INFO:tasks.rados.rados.0.smithi045.stdout:1520: writing smithi04526579-9 from 2894439 to 3351994 tid 3 2024-04-26T00:29:01.312 INFO:tasks.rados.rados.0.smithi045.stdout:1521: snap_create 2024-04-26T00:29:01.312 INFO:tasks.rados.rados.0.smithi045.stdout:1518: expect (ObjNum 453 snap 141 seq_num 453) 2024-04-26T00:29:01.489 INFO:tasks.rados.rados.0.smithi045.stdout:1520: finishing write tid 1 to smithi04526579-9 2024-04-26T00:29:01.489 INFO:tasks.rados.rados.0.smithi045.stdout:1520: finishing write tid 2 to smithi04526579-9 2024-04-26T00:29:01.489 INFO:tasks.rados.rados.0.smithi045.stdout:1520: finishing write tid 3 to smithi04526579-9 2024-04-26T00:29:01.489 INFO:tasks.rados.rados.0.smithi045.stdout:1520: finishing write tid 4 to smithi04526579-9 2024-04-26T00:29:01.489 INFO:tasks.rados.rados.0.smithi045.stdout:1520: finishing write tid 5 to smithi04526579-9 2024-04-26T00:29:01.489 INFO:tasks.rados.rados.0.smithi045.stdout:1520: oid 9 updating version 0 to 843 2024-04-26T00:29:01.489 INFO:tasks.rados.rados.0.smithi045.stdout:1520: oid 9 version 843 is already newer than 842 2024-04-26T00:29:01.489 INFO:tasks.rados.rados.0.smithi045.stdout:1520: oid 9 version 843 is already newer than 840 2024-04-26T00:29:01.489 INFO:tasks.rados.rados.0.smithi045.stdout:1520: oid 9 version 843 is already newer than 841 2024-04-26T00:29:01.489 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 843 (ObjNum 563 snap 175 seq_num 563) dirty exists 2024-04-26T00:29:01.490 INFO:tasks.rados.rados.0.smithi045.stdout:1520: left oid 9 (ObjNum 563 snap 175 seq_num 563) 2024-04-26T00:29:02.286 INFO:tasks.rados.rados.0.smithi045.stdout:1518: done (2 left) 2024-04-26T00:29:02.286 INFO:tasks.rados.rados.0.smithi045.stdout:1520: done (1 left) 2024-04-26T00:29:02.286 INFO:tasks.rados.rados.0.smithi045.stdout:1521: done (0 left) 2024-04-26T00:29:02.287 INFO:tasks.rados.rados.0.smithi045.stdout:1522: rollback oid 41 current snap is 176 2024-04-26T00:29:02.287 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 41 to 174 2024-04-26T00:29:02.287 INFO:tasks.rados.rados.0.smithi045.stdout:1523: delete oid 24 current snap is 176 2024-04-26T00:29:02.297 INFO:tasks.rados.rados.0.smithi045.stdout:1523: done (1 left) 2024-04-26T00:29:02.297 INFO:tasks.rados.rados.0.smithi045.stdout:1524: copy_from oid 43 from oid 35 current snap is 176 2024-04-26T00:29:02.298 INFO:tasks.rados.rados.0.smithi045.stdout:1525: read oid 17 snap -1 2024-04-26T00:29:02.298 INFO:tasks.rados.rados.0.smithi045.stdout:1525: expect (ObjNum 560 snap 174 seq_num 560) 2024-04-26T00:29:02.298 INFO:tasks.rados.rados.0.smithi045.stdout:1526: copy_from oid 30 from oid 20 current snap is 176 2024-04-26T00:29:02.298 INFO:tasks.rados.rados.0.smithi045.stdout:1527: read oid 22 snap -1 2024-04-26T00:29:02.299 INFO:tasks.rados.rados.0.smithi045.stdout:1527: expect (ObjNum 330 snap 96 seq_num 330) 2024-04-26T00:29:02.299 INFO:tasks.rados.rados.0.smithi045.stdout:1528: rollback oid 1 current snap is 176 2024-04-26T00:29:02.299 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 1 to 163 2024-04-26T00:29:02.299 INFO:tasks.rados.rados.0.smithi045.stdout:1529: delete oid 45 current snap is 176 2024-04-26T00:29:02.309 INFO:tasks.rados.rados.0.smithi045.stdout:1527: expect (ObjNum 330 snap 96 seq_num 330) 2024-04-26T00:29:02.475 INFO:tasks.rados.rados.0.smithi045.stdout:1522: finishing rollback tid 0 to smithi04526579-41 2024-04-26T00:29:02.475 INFO:tasks.rados.rados.0.smithi045.stdout:1522: finishing rollback tid 1 to smithi04526579-41 2024-04-26T00:29:02.475 INFO:tasks.rados.rados.0.smithi045.stdout:1522: finishing rollback tid 2 to smithi04526579-41 2024-04-26T00:29:02.475 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 1047 (ObjNum 394 snap 120 seq_num 394) dirty exists 2024-04-26T00:29:02.475 INFO:tasks.rados.rados.0.smithi045.stdout:1528: finishing rollback tid 0 to smithi04526579-1 2024-04-26T00:29:02.475 INFO:tasks.rados.rados.0.smithi045.stdout:1528: finishing rollback tid 1 to smithi04526579-1 2024-04-26T00:29:02.475 INFO:tasks.rados.rados.0.smithi045.stdout:1528: finishing rollback tid 2 to smithi04526579-1 2024-04-26T00:29:02.476 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 1050 (ObjNum 524 snap 162 seq_num 524) dirty exists 2024-04-26T00:29:02.476 INFO:tasks.rados.rados.0.smithi045.stdout:1524: finishing copy_from racing read to smithi04526579-43 2024-04-26T00:29:02.476 INFO:tasks.rados.rados.0.smithi045.stdout:1525: expect (ObjNum 560 snap 174 seq_num 560) 2024-04-26T00:29:02.638 INFO:tasks.rados.rados.0.smithi045.stdout:1524: finishing copy_from to smithi04526579-43 2024-04-26T00:29:02.638 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 872 (ObjNum 553 snap 168 seq_num 553) dirty exists 2024-04-26T00:29:02.638 INFO:tasks.rados.rados.0.smithi045.stdout:1526: finishing copy_from racing read to smithi04526579-30 2024-04-26T00:29:02.639 INFO:tasks.rados.rados.0.smithi045.stdout:1526: finishing copy_from to smithi04526579-30 2024-04-26T00:29:02.639 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 864 (ObjNum 542 snap 167 seq_num 542) dirty exists 2024-04-26T00:29:02.639 INFO:tasks.rados.rados.0.smithi045.stdout:1522: done (6 left) 2024-04-26T00:29:02.639 INFO:tasks.rados.rados.0.smithi045.stdout:1524: done (5 left) 2024-04-26T00:29:02.639 INFO:tasks.rados.rados.0.smithi045.stdout:1525: done (4 left) 2024-04-26T00:29:02.639 INFO:tasks.rados.rados.0.smithi045.stdout:1526: done (3 left) 2024-04-26T00:29:02.639 INFO:tasks.rados.rados.0.smithi045.stdout:1527: done (2 left) 2024-04-26T00:29:02.639 INFO:tasks.rados.rados.0.smithi045.stdout:1528: done (1 left) 2024-04-26T00:29:02.639 INFO:tasks.rados.rados.0.smithi045.stdout:1529: done (0 left) 2024-04-26T00:29:02.639 INFO:tasks.rados.rados.0.smithi045.stdout:1530: copy_from oid 20 from oid 6 current snap is 176 2024-04-26T00:29:02.639 INFO:tasks.rados.rados.0.smithi045.stdout:1531: rollback oid 9 current snap is 176 2024-04-26T00:29:02.639 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 9 to 143 2024-04-26T00:29:02.639 INFO:tasks.rados.rados.0.smithi045.stdout:1532: write (excl) oid 1 current snap is 176 2024-04-26T00:29:02.639 INFO:tasks.rados.rados.0.smithi045.stdout:1532: seq_num 566 ranges {712762=401056,1537867=645855,2860355=652407} 2024-04-26T00:29:02.657 INFO:tasks.rados.rados.0.smithi045.stdout:1532: writing smithi04526579-1 from 712762 to 1113818 tid 1 2024-04-26T00:29:02.674 INFO:tasks.rados.rados.0.smithi045.stdout:1532: writing smithi04526579-1 from 1537867 to 2183722 tid 2 2024-04-26T00:29:02.696 INFO:tasks.rados.rados.0.smithi045.stdout:1532: writing smithi04526579-1 from 2860355 to 3512762 tid 3 2024-04-26T00:29:02.696 INFO:tasks.rados.rados.0.smithi045.stdout:1533: read oid 25 snap -1 2024-04-26T00:29:02.696 INFO:tasks.rados.rados.0.smithi045.stdout:1533: expect (ObjNum 554 snap 168 seq_num 554) 2024-04-26T00:29:02.696 INFO:tasks.rados.rados.0.smithi045.stdout:1534: delete oid 22 current snap is 176 2024-04-26T00:29:02.696 INFO:tasks.rados.rados.0.smithi045.stdout:1531: finishing rollback tid 0 to smithi04526579-9 2024-04-26T00:29:02.696 INFO:tasks.rados.rados.0.smithi045.stdout:1531: finishing rollback tid 1 to smithi04526579-9 2024-04-26T00:29:02.696 INFO:tasks.rados.rados.0.smithi045.stdout:1531: finishing rollback tid 2 to smithi04526579-9 2024-04-26T00:29:02.696 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 847 (ObjNum 458 snap 143 seq_num 458) dirty exists 2024-04-26T00:29:02.697 INFO:tasks.rados.rados.0.smithi045.stdout:1530: finishing copy_from racing read to smithi04526579-20 2024-04-26T00:29:02.697 INFO:tasks.rados.rados.0.smithi045.stdout:1532: finishing write tid 1 to smithi04526579-1 2024-04-26T00:29:02.697 INFO:tasks.rados.rados.0.smithi045.stdout:1530: finishing copy_from to smithi04526579-20 2024-04-26T00:29:02.697 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 1038 (ObjNum 394 snap 120 seq_num 394) dirty exists 2024-04-26T00:29:02.697 INFO:tasks.rados.rados.0.smithi045.stdout:1532: finishing write tid 2 to smithi04526579-1 2024-04-26T00:29:02.704 INFO:tasks.rados.rados.0.smithi045.stdout:1532: finishing write tid 3 to smithi04526579-1 2024-04-26T00:29:02.704 INFO:tasks.rados.rados.0.smithi045.stdout:1532: finishing write tid 4 to smithi04526579-1 2024-04-26T00:29:02.705 INFO:tasks.rados.rados.0.smithi045.stdout:1532: finishing write tid 5 to smithi04526579-1 2024-04-26T00:29:02.705 INFO:tasks.rados.rados.0.smithi045.stdout:1532: oid 1 updating version 0 to 1054 2024-04-26T00:29:02.706 INFO:tasks.rados.rados.0.smithi045.stdout:1532: oid 1 version 1054 is already newer than 1051 2024-04-26T00:29:02.706 INFO:tasks.rados.rados.0.smithi045.stdout:1532: oid 1 version 1054 is already newer than 1053 2024-04-26T00:29:02.706 INFO:tasks.rados.rados.0.smithi045.stdout:1532: oid 1 version 1054 is already newer than 1052 2024-04-26T00:29:02.706 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 1054 (ObjNum 566 snap 176 seq_num 566) dirty exists 2024-04-26T00:29:02.706 INFO:tasks.rados.rados.0.smithi045.stdout:1532: left oid 1 (ObjNum 566 snap 176 seq_num 566) 2024-04-26T00:29:02.706 INFO:tasks.rados.rados.0.smithi045.stdout:1530: done (4 left) 2024-04-26T00:29:02.706 INFO:tasks.rados.rados.0.smithi045.stdout:1531: done (3 left) 2024-04-26T00:29:02.706 INFO:tasks.rados.rados.0.smithi045.stdout:1532: done (2 left) 2024-04-26T00:29:02.706 INFO:tasks.rados.rados.0.smithi045.stdout:1534: done (1 left) 2024-04-26T00:29:02.706 INFO:tasks.rados.rados.0.smithi045.stdout:1535: read oid 2 snap -1 2024-04-26T00:29:02.706 INFO:tasks.rados.rados.0.smithi045.stdout:1535: expect (ObjNum 544 snap 168 seq_num 544) 2024-04-26T00:29:02.706 INFO:tasks.rados.rados.0.smithi045.stdout:1536: rollback oid 31 current snap is 176 2024-04-26T00:29:02.706 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 31 to 171 2024-04-26T00:29:02.707 INFO:tasks.rados.rados.0.smithi045.stdout:1537: read oid 1 snap -1 2024-04-26T00:29:02.707 INFO:tasks.rados.rados.0.smithi045.stdout:1537: expect (ObjNum 566 snap 176 seq_num 566) 2024-04-26T00:29:02.707 INFO:tasks.rados.rados.0.smithi045.stdout:1538: read oid 41 snap -1 2024-04-26T00:29:02.707 INFO:tasks.rados.rados.0.smithi045.stdout:1538: expect (ObjNum 394 snap 120 seq_num 394) 2024-04-26T00:29:02.707 INFO:tasks.rados.rados.0.smithi045.stdout:1539: read oid 28 snap -1 2024-04-26T00:29:02.707 INFO:tasks.rados.rados.0.smithi045.stdout:1539: expect (ObjNum 553 snap 168 seq_num 553) 2024-04-26T00:29:02.707 INFO:tasks.rados.rados.0.smithi045.stdout:1540: snap_remove snap 174 2024-04-26T00:29:03.289 INFO:tasks.rados.rados.0.smithi045.stdout:1540: done (6 left) 2024-04-26T00:29:03.289 INFO:tasks.rados.rados.0.smithi045.stdout:1541: snap_remove snap 126 2024-04-26T00:29:04.274 INFO:tasks.rados.rados.0.smithi045.stdout:1541: done (6 left) 2024-04-26T00:29:04.275 INFO:tasks.rados.rados.0.smithi045.stdout:1542: read oid 18 snap 153 2024-04-26T00:29:04.275 INFO:tasks.rados.rados.0.smithi045.stdout:1542: expect deleted 2024-04-26T00:29:04.275 INFO:tasks.rados.rados.0.smithi045.stdout:1543: write oid 35 current snap is 176 2024-04-26T00:29:04.275 INFO:tasks.rados.rados.0.smithi045.stdout:1533: expect (ObjNum 554 snap 168 seq_num 554) 2024-04-26T00:29:04.482 INFO:tasks.rados.rados.0.smithi045.stdout:1537: expect (ObjNum 566 snap 176 seq_num 566) 2024-04-26T00:29:04.640 INFO:tasks.rados.rados.0.smithi045.stdout:1538: expect (ObjNum 394 snap 120 seq_num 394) 2024-04-26T00:29:04.752 INFO:tasks.rados.rados.0.smithi045.stdout:1535: expect (ObjNum 544 snap 168 seq_num 544) 2024-04-26T00:29:04.853 INFO:tasks.rados.rados.0.smithi045.stdout:1536: finishing rollback tid 0 to smithi04526579-31 2024-04-26T00:29:04.853 INFO:tasks.rados.rados.0.smithi045.stdout:1536: finishing rollback tid 1 to smithi04526579-31 2024-04-26T00:29:04.853 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 850 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-04-26T00:29:04.853 INFO:tasks.rados.rados.0.smithi045.stdout:1539: expect (ObjNum 553 snap 168 seq_num 553) 2024-04-26T00:29:04.956 INFO:tasks.rados.rados.0.smithi045.stdout:1543: seq_num 568 ranges {746376=551554,2054831=649524,3315201=231175} 2024-04-26T00:29:04.963 INFO:tasks.rados.rados.0.smithi045.stdout:1543: writing smithi04526579-35 from 746376 to 1297930 tid 1 2024-04-26T00:29:04.971 INFO:tasks.rados.rados.0.smithi045.stdout:1543: writing smithi04526579-35 from 2054831 to 2704355 tid 2 2024-04-26T00:29:04.976 INFO:tasks.rados.rados.0.smithi045.stdout:1543: writing smithi04526579-35 from 3315201 to 3546376 tid 3 2024-04-26T00:29:04.976 INFO:tasks.rados.rados.0.smithi045.stdout:1533: done (7 left) 2024-04-26T00:29:04.977 INFO:tasks.rados.rados.0.smithi045.stdout:1535: done (6 left) 2024-04-26T00:29:04.977 INFO:tasks.rados.rados.0.smithi045.stdout:1536: done (5 left) 2024-04-26T00:29:04.977 INFO:tasks.rados.rados.0.smithi045.stdout:1537: done (4 left) 2024-04-26T00:29:04.977 INFO:tasks.rados.rados.0.smithi045.stdout:1538: done (3 left) 2024-04-26T00:29:04.977 INFO:tasks.rados.rados.0.smithi045.stdout:1539: done (2 left) 2024-04-26T00:29:04.977 INFO:tasks.rados.rados.0.smithi045.stdout:1542: done (1 left) 2024-04-26T00:29:04.977 INFO:tasks.rados.rados.0.smithi045.stdout:1544: read oid 8 snap -1 2024-04-26T00:29:04.977 INFO:tasks.rados.rados.0.smithi045.stdout:1544: expect (ObjNum 551 snap 168 seq_num 551) 2024-04-26T00:29:04.977 INFO:tasks.rados.rados.0.smithi045.stdout:1545: write oid 45 current snap is 176 2024-04-26T00:29:04.977 INFO:tasks.rados.rados.0.smithi045.stdout:1545: seq_num 569 ranges {763183=626803,1996960=793182,3226271=336912} 2024-04-26T00:29:04.984 INFO:tasks.rados.rados.0.smithi045.stdout:1545: writing smithi04526579-45 from 763183 to 1389986 tid 1 2024-04-26T00:29:04.994 INFO:tasks.rados.rados.0.smithi045.stdout:1545: writing smithi04526579-45 from 1996960 to 2790142 tid 2 2024-04-26T00:29:05.001 INFO:tasks.rados.rados.0.smithi045.stdout:1545: writing smithi04526579-45 from 3226271 to 3563183 tid 3 2024-04-26T00:29:05.001 INFO:tasks.rados.rados.0.smithi045.stdout:1546: write oid 11 current snap is 176 2024-04-26T00:29:05.001 INFO:tasks.rados.rados.0.smithi045.stdout:1546: seq_num 570 ranges {779990=702052,2055442=653193,3253694=326296} 2024-04-26T00:29:05.012 INFO:tasks.rados.rados.0.smithi045.stdout:1546: writing smithi04526579-11 from 779990 to 1482042 tid 1 2024-04-26T00:29:05.020 INFO:tasks.rados.rados.0.smithi045.stdout:1546: writing smithi04526579-11 from 2055442 to 2708635 tid 2 2024-04-26T00:29:05.026 INFO:tasks.rados.rados.0.smithi045.stdout:1546: writing smithi04526579-11 from 3253694 to 3579990 tid 3 2024-04-26T00:29:05.026 INFO:tasks.rados.rados.0.smithi045.stdout:1547: delete oid 18 current snap is 176 2024-04-26T00:29:05.026 INFO:tasks.rados.rados.0.smithi045.stdout:1543: finishing write tid 1 to smithi04526579-35 2024-04-26T00:29:05.027 INFO:tasks.rados.rados.0.smithi045.stdout:1543: finishing write tid 2 to smithi04526579-35 2024-04-26T00:29:05.027 INFO:tasks.rados.rados.0.smithi045.stdout:1543: finishing write tid 3 to smithi04526579-35 2024-04-26T00:29:05.027 INFO:tasks.rados.rados.0.smithi045.stdout:1543: finishing write tid 4 to smithi04526579-35 2024-04-26T00:29:05.027 INFO:tasks.rados.rados.0.smithi045.stdout:1545: finishing write tid 1 to smithi04526579-45 2024-04-26T00:29:05.027 INFO:tasks.rados.rados.0.smithi045.stdout:1543: finishing write tid 5 to smithi04526579-35 2024-04-26T00:29:05.027 INFO:tasks.rados.rados.0.smithi045.stdout:1543: oid 35 updating version 0 to 929 2024-04-26T00:29:05.027 INFO:tasks.rados.rados.0.smithi045.stdout:1543: oid 35 updating version 929 to 931 2024-04-26T00:29:05.027 INFO:tasks.rados.rados.0.smithi045.stdout:1543: oid 35 updating version 931 to 932 2024-04-26T00:29:05.027 INFO:tasks.rados.rados.0.smithi045.stdout:1543: oid 35 version 932 is already newer than 930 2024-04-26T00:29:05.027 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 932 (ObjNum 568 snap 176 seq_num 568) dirty exists 2024-04-26T00:29:05.027 INFO:tasks.rados.rados.0.smithi045.stdout:1543: left oid 35 (ObjNum 568 snap 176 seq_num 568) 2024-04-26T00:29:05.027 INFO:tasks.rados.rados.0.smithi045.stdout:1544: expect (ObjNum 551 snap 168 seq_num 551) 2024-04-26T00:29:05.199 INFO:tasks.rados.rados.0.smithi045.stdout:1545: finishing write tid 2 to smithi04526579-45 2024-04-26T00:29:05.199 INFO:tasks.rados.rados.0.smithi045.stdout:1545: finishing write tid 3 to smithi04526579-45 2024-04-26T00:29:05.199 INFO:tasks.rados.rados.0.smithi045.stdout:1545: finishing write tid 4 to smithi04526579-45 2024-04-26T00:29:05.199 INFO:tasks.rados.rados.0.smithi045.stdout:1545: finishing write tid 5 to smithi04526579-45 2024-04-26T00:29:05.199 INFO:tasks.rados.rados.0.smithi045.stdout:1545: oid 45 updating version 0 to 878 2024-04-26T00:29:05.199 INFO:tasks.rados.rados.0.smithi045.stdout:1545: oid 45 updating version 878 to 879 2024-04-26T00:29:05.199 INFO:tasks.rados.rados.0.smithi045.stdout:1545: oid 45 updating version 879 to 880 2024-04-26T00:29:05.199 INFO:tasks.rados.rados.0.smithi045.stdout:1545: oid 45 version 880 is already newer than 877 2024-04-26T00:29:05.199 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 880 (ObjNum 569 snap 176 seq_num 569) dirty exists 2024-04-26T00:29:05.199 INFO:tasks.rados.rados.0.smithi045.stdout:1545: left oid 45 (ObjNum 569 snap 176 seq_num 569) 2024-04-26T00:29:05.200 INFO:tasks.rados.rados.0.smithi045.stdout:1546: finishing write tid 1 to smithi04526579-11 2024-04-26T00:29:05.200 INFO:tasks.rados.rados.0.smithi045.stdout:1546: finishing write tid 2 to smithi04526579-11 2024-04-26T00:29:05.200 INFO:tasks.rados.rados.0.smithi045.stdout:1546: finishing write tid 3 to smithi04526579-11 2024-04-26T00:29:05.200 INFO:tasks.rados.rados.0.smithi045.stdout:1546: finishing write tid 4 to smithi04526579-11 2024-04-26T00:29:05.200 INFO:tasks.rados.rados.0.smithi045.stdout:1546: finishing write tid 5 to smithi04526579-11 2024-04-26T00:29:05.200 INFO:tasks.rados.rados.0.smithi045.stdout:1546: oid 11 updating version 0 to 884 2024-04-26T00:29:05.200 INFO:tasks.rados.rados.0.smithi045.stdout:1546: oid 11 version 884 is already newer than 881 2024-04-26T00:29:05.200 INFO:tasks.rados.rados.0.smithi045.stdout:1546: oid 11 version 884 is already newer than 882 2024-04-26T00:29:05.200 INFO:tasks.rados.rados.0.smithi045.stdout:1546: oid 11 version 884 is already newer than 883 2024-04-26T00:29:05.200 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 884 (ObjNum 570 snap 176 seq_num 570) dirty exists 2024-04-26T00:29:05.200 INFO:tasks.rados.rados.0.smithi045.stdout:1546: left oid 11 (ObjNum 570 snap 176 seq_num 570) 2024-04-26T00:29:05.200 INFO:tasks.rados.rados.0.smithi045.stdout:1543: done (4 left) 2024-04-26T00:29:05.200 INFO:tasks.rados.rados.0.smithi045.stdout:1544: done (3 left) 2024-04-26T00:29:05.200 INFO:tasks.rados.rados.0.smithi045.stdout:1545: done (2 left) 2024-04-26T00:29:05.201 INFO:tasks.rados.rados.0.smithi045.stdout:1546: done (1 left) 2024-04-26T00:29:05.201 INFO:tasks.rados.rados.0.smithi045.stdout:1547: done (0 left) 2024-04-26T00:29:05.201 INFO:tasks.rados.rados.0.smithi045.stdout:1548: write (excl) oid 25 current snap is 176 2024-04-26T00:29:05.201 INFO:tasks.rados.rados.0.smithi045.stdout:1548: seq_num 572 ranges {413604=568903,1488759=656862,2624893=591709} 2024-04-26T00:29:05.205 INFO:tasks.rados.rados.0.smithi045.stdout:1548: writing smithi04526579-25 from 413604 to 982507 tid 1 2024-04-26T00:29:05.213 INFO:tasks.rados.rados.0.smithi045.stdout:1548: writing smithi04526579-25 from 1488759 to 2145621 tid 2 2024-04-26T00:29:05.220 INFO:tasks.rados.rados.0.smithi045.stdout:1548: writing smithi04526579-25 from 2624893 to 3216602 tid 3 2024-04-26T00:29:05.220 INFO:tasks.rados.rados.0.smithi045.stdout:1549: rollback oid 37 current snap is 176 2024-04-26T00:29:05.220 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 37 to 153 2024-04-26T00:29:05.220 INFO:tasks.rados.rados.0.smithi045.stdout:1550: write (excl) oid 10 current snap is 176 2024-04-26T00:29:05.220 INFO:tasks.rados.rados.0.smithi045.stdout:1550: seq_num 573 ranges {430411=644152,1830888=400520,2935963=402981} 2024-04-26T00:29:05.226 INFO:tasks.rados.rados.0.smithi045.stdout:1550: writing smithi04526579-10 from 430411 to 1074563 tid 1 2024-04-26T00:29:05.234 INFO:tasks.rados.rados.0.smithi045.stdout:1550: writing smithi04526579-10 from 1830888 to 2231408 tid 2 2024-04-26T00:29:05.241 INFO:tasks.rados.rados.0.smithi045.stdout:1550: writing smithi04526579-10 from 2935963 to 3338944 tid 3 2024-04-26T00:29:05.241 INFO:tasks.rados.rados.0.smithi045.stdout:1551: write (excl) oid 6 current snap is 176 2024-04-26T00:29:05.241 INFO:tasks.rados.rados.0.smithi045.stdout:1551: seq_num 574 ranges {447218=719401,1889370=660531,2963386=614253} 2024-04-26T00:29:05.249 INFO:tasks.rados.rados.0.smithi045.stdout:1551: writing smithi04526579-6 from 447218 to 1166619 tid 1 2024-04-26T00:29:05.258 INFO:tasks.rados.rados.0.smithi045.stdout:1551: writing smithi04526579-6 from 1889370 to 2549901 tid 2 2024-04-26T00:29:05.265 INFO:tasks.rados.rados.0.smithi045.stdout:1551: writing smithi04526579-6 from 2963386 to 3577639 tid 3 2024-04-26T00:29:05.265 INFO:tasks.rados.rados.0.smithi045.stdout:1552: write oid 19 current snap is 176 2024-04-26T00:29:05.265 INFO:tasks.rados.rados.0.smithi045.stdout:1548: finishing write tid 1 to smithi04526579-25 2024-04-26T00:29:05.265 INFO:tasks.rados.rados.0.smithi045.stdout:1552: seq_num 575 ranges {464025=794650,1947852=404189,2990809=425525} 2024-04-26T00:29:05.273 INFO:tasks.rados.rados.0.smithi045.stdout:1552: writing smithi04526579-19 from 464025 to 1258675 tid 1 2024-04-26T00:29:05.279 INFO:tasks.rados.rados.0.smithi045.stdout:1552: writing smithi04526579-19 from 1947852 to 2352041 tid 2 2024-04-26T00:29:05.286 INFO:tasks.rados.rados.0.smithi045.stdout:1552: writing smithi04526579-19 from 2990809 to 3416334 tid 3 2024-04-26T00:29:05.286 INFO:tasks.rados.rados.0.smithi045.stdout:1553: snap_remove snap 163 2024-04-26T00:29:05.321 INFO:tasks.rados.rados.0.smithi045.stdout:1553: done (5 left) 2024-04-26T00:29:05.321 INFO:tasks.rados.rados.0.smithi045.stdout:1554: rollback oid 45 current snap is 176 2024-04-26T00:29:05.322 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 45 to 169 2024-04-26T00:29:05.322 INFO:tasks.rados.rados.0.smithi045.stdout:1548: finishing write tid 2 to smithi04526579-25 2024-04-26T00:29:05.322 INFO:tasks.rados.rados.0.smithi045.stdout:1550: finishing write tid 1 to smithi04526579-10 2024-04-26T00:29:05.322 INFO:tasks.rados.rados.0.smithi045.stdout:1550: finishing write tid 2 to smithi04526579-10 2024-04-26T00:29:05.322 INFO:tasks.rados.rados.0.smithi045.stdout:1548: finishing write tid 3 to smithi04526579-25 2024-04-26T00:29:05.322 INFO:tasks.rados.rados.0.smithi045.stdout:1548: finishing write tid 4 to smithi04526579-25 2024-04-26T00:29:05.322 INFO:tasks.rados.rados.0.smithi045.stdout:1548: finishing write tid 5 to smithi04526579-25 2024-04-26T00:29:05.322 INFO:tasks.rados.rados.0.smithi045.stdout:1548: oid 25 updating version 0 to 1225 2024-04-26T00:29:05.322 INFO:tasks.rados.rados.0.smithi045.stdout:1548: oid 25 version 1225 is already newer than 1224 2024-04-26T00:29:05.322 INFO:tasks.rados.rados.0.smithi045.stdout:1548: oid 25 version 1225 is already newer than 1223 2024-04-26T00:29:05.322 INFO:tasks.rados.rados.0.smithi045.stdout:1548: oid 25 version 1225 is already newer than 1222 2024-04-26T00:29:05.322 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 1225 (ObjNum 572 snap 176 seq_num 572) dirty exists 2024-04-26T00:29:05.322 INFO:tasks.rados.rados.0.smithi045.stdout:1548: left oid 25 (ObjNum 572 snap 176 seq_num 572) 2024-04-26T00:29:05.322 INFO:tasks.rados.rados.0.smithi045.stdout:1549: finishing rollback tid 0 to smithi04526579-37 2024-04-26T00:29:05.323 INFO:tasks.rados.rados.0.smithi045.stdout:1549: finishing rollback tid 1 to smithi04526579-37 2024-04-26T00:29:05.323 INFO:tasks.rados.rados.0.smithi045.stdout:1549: finishing rollback tid 2 to smithi04526579-37 2024-04-26T00:29:05.323 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 1229 (ObjNum 490 snap 153 seq_num 490) dirty exists 2024-04-26T00:29:05.323 INFO:tasks.rados.rados.0.smithi045.stdout:1550: finishing write tid 3 to smithi04526579-10 2024-04-26T00:29:05.323 INFO:tasks.rados.rados.0.smithi045.stdout:1550: finishing write tid 4 to smithi04526579-10 2024-04-26T00:29:05.323 INFO:tasks.rados.rados.0.smithi045.stdout:1550: finishing write tid 5 to smithi04526579-10 2024-04-26T00:29:05.323 INFO:tasks.rados.rados.0.smithi045.stdout:1550: oid 10 updating version 0 to 874 2024-04-26T00:29:05.323 INFO:tasks.rados.rados.0.smithi045.stdout:1550: oid 10 updating version 874 to 877 2024-04-26T00:29:05.323 INFO:tasks.rados.rados.0.smithi045.stdout:1550: oid 10 version 877 is already newer than 875 2024-04-26T00:29:05.323 INFO:tasks.rados.rados.0.smithi045.stdout:1550: oid 10 version 877 is already newer than 876 2024-04-26T00:29:05.323 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 877 (ObjNum 573 snap 176 seq_num 573) dirty exists 2024-04-26T00:29:05.323 INFO:tasks.rados.rados.0.smithi045.stdout:1550: left oid 10 (ObjNum 573 snap 176 seq_num 573) 2024-04-26T00:29:05.323 INFO:tasks.rados.rados.0.smithi045.stdout:1551: finishing write tid 1 to smithi04526579-6 2024-04-26T00:29:05.323 INFO:tasks.rados.rados.0.smithi045.stdout:1551: finishing write tid 2 to smithi04526579-6 2024-04-26T00:29:05.324 INFO:tasks.rados.rados.0.smithi045.stdout:1551: finishing write tid 3 to smithi04526579-6 2024-04-26T00:29:05.324 INFO:tasks.rados.rados.0.smithi045.stdout:1551: finishing write tid 4 to smithi04526579-6 2024-04-26T00:29:05.324 INFO:tasks.rados.rados.0.smithi045.stdout:1551: finishing write tid 5 to smithi04526579-6 2024-04-26T00:29:05.324 INFO:tasks.rados.rados.0.smithi045.stdout:1551: oid 6 updating version 0 to 1042 2024-04-26T00:29:05.324 INFO:tasks.rados.rados.0.smithi045.stdout:1551: oid 6 version 1042 is already newer than 1041 2024-04-26T00:29:05.324 INFO:tasks.rados.rados.0.smithi045.stdout:1551: oid 6 version 1042 is already newer than 1039 2024-04-26T00:29:05.324 INFO:tasks.rados.rados.0.smithi045.stdout:1551: oid 6 version 1042 is already newer than 1040 2024-04-26T00:29:05.324 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 1042 (ObjNum 574 snap 176 seq_num 574) dirty exists 2024-04-26T00:29:05.324 INFO:tasks.rados.rados.0.smithi045.stdout:1551: left oid 6 (ObjNum 574 snap 176 seq_num 574) 2024-04-26T00:29:05.324 INFO:tasks.rados.rados.0.smithi045.stdout:1552: finishing write tid 1 to smithi04526579-19 2024-04-26T00:29:05.324 INFO:tasks.rados.rados.0.smithi045.stdout:1552: finishing write tid 2 to smithi04526579-19 2024-04-26T00:29:05.324 INFO:tasks.rados.rados.0.smithi045.stdout:1548: done (5 left) 2024-04-26T00:29:05.324 INFO:tasks.rados.rados.0.smithi045.stdout:1549: done (4 left) 2024-04-26T00:29:05.324 INFO:tasks.rados.rados.0.smithi045.stdout:1550: done (3 left) 2024-04-26T00:29:05.324 INFO:tasks.rados.rados.0.smithi045.stdout:1551: done (2 left) 2024-04-26T00:29:05.325 INFO:tasks.rados.rados.0.smithi045.stdout:1555: snap_create 2024-04-26T00:29:05.325 INFO:tasks.rados.rados.0.smithi045.stdout:1552: finishing write tid 3 to smithi04526579-19 2024-04-26T00:29:05.325 INFO:tasks.rados.rados.0.smithi045.stdout:1552: finishing write tid 4 to smithi04526579-19 2024-04-26T00:29:05.325 INFO:tasks.rados.rados.0.smithi045.stdout:1552: finishing write tid 5 to smithi04526579-19 2024-04-26T00:29:05.325 INFO:tasks.rados.rados.0.smithi045.stdout:1552: oid 19 updating version 0 to 934 2024-04-26T00:29:05.325 INFO:tasks.rados.rados.0.smithi045.stdout:1552: oid 19 updating version 934 to 935 2024-04-26T00:29:05.325 INFO:tasks.rados.rados.0.smithi045.stdout:1552: oid 19 updating version 935 to 937 2024-04-26T00:29:05.325 INFO:tasks.rados.rados.0.smithi045.stdout:1552: oid 19 version 937 is already newer than 936 2024-04-26T00:29:05.325 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 937 (ObjNum 575 snap 176 seq_num 575) dirty exists 2024-04-26T00:29:05.325 INFO:tasks.rados.rados.0.smithi045.stdout:1552: left oid 19 (ObjNum 575 snap 176 seq_num 575) 2024-04-26T00:29:05.332 INFO:tasks.rados.rados.0.smithi045.stdout:1554: finishing rollback tid 0 to smithi04526579-45 2024-04-26T00:29:05.336 INFO:tasks.rados.rados.0.smithi045.stdout:1554: finishing rollback tid 1 to smithi04526579-45 2024-04-26T00:29:05.336 INFO:tasks.rados.rados.0.smithi045.stdout:1554: finishing rollback tid 2 to smithi04526579-45 2024-04-26T00:29:05.336 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 887 (ObjNum 546 snap 168 seq_num 546) dirty exists 2024-04-26T00:29:06.327 INFO:tasks.rados.rados.0.smithi045.stdout:1552: done (2 left) 2024-04-26T00:29:06.328 INFO:tasks.rados.rados.0.smithi045.stdout:1554: done (1 left) 2024-04-26T00:29:06.328 INFO:tasks.rados.rados.0.smithi045.stdout:1555: done (0 left) 2024-04-26T00:29:06.328 INFO:tasks.rados.rados.0.smithi045.stdout:1556: read oid 35 snap -1 2024-04-26T00:29:06.328 INFO:tasks.rados.rados.0.smithi045.stdout:1556: expect (ObjNum 568 snap 176 seq_num 568) 2024-04-26T00:29:06.328 INFO:tasks.rados.rados.0.smithi045.stdout:1557: snap_create 2024-04-26T00:29:06.347 INFO:tasks.rados.rados.0.smithi045.stdout:1556: expect (ObjNum 568 snap 176 seq_num 568) 2024-04-26T00:29:07.337 INFO:tasks.rados.rados.0.smithi045.stdout:1556: done (1 left) 2024-04-26T00:29:07.337 INFO:tasks.rados.rados.0.smithi045.stdout:1557: done (0 left) 2024-04-26T00:29:07.337 INFO:tasks.rados.rados.0.smithi045.stdout:1558: delete oid 5 current snap is 178 2024-04-26T00:29:07.347 INFO:tasks.rados.rados.0.smithi045.stdout:1558: done (0 left) 2024-04-26T00:29:07.347 INFO:tasks.rados.rados.0.smithi045.stdout:1559: read oid 23 snap -1 2024-04-26T00:29:07.347 INFO:tasks.rados.rados.0.smithi045.stdout:1559: expect (ObjNum 558 snap 173 seq_num 558) 2024-04-26T00:29:07.347 INFO:tasks.rados.rados.0.smithi045.stdout:1560: snap_create 2024-04-26T00:29:07.364 INFO:tasks.rados.rados.0.smithi045.stdout:1559: expect (ObjNum 558 snap 173 seq_num 558) 2024-04-26T00:29:08.348 INFO:tasks.rados.rados.0.smithi045.stdout:1559: done (1 left) 2024-04-26T00:29:08.349 INFO:tasks.rados.rados.0.smithi045.stdout:1560: done (0 left) 2024-04-26T00:29:08.349 INFO:tasks.rados.rados.0.smithi045.stdout:1561: write (excl) oid 36 current snap is 179 2024-04-26T00:29:08.349 INFO:tasks.rados.rados.0.smithi045.stdout:1561: seq_num 577 ranges {497639=545148,1548463=407858,2529302=564422,3528143=169496} 2024-04-26T00:29:08.365 INFO:tasks.rados.rados.0.smithi045.stdout:1561: writing smithi04526579-36 from 497639 to 1042787 tid 1 2024-04-26T00:29:08.380 INFO:tasks.rados.rados.0.smithi045.stdout:1561: writing smithi04526579-36 from 1548463 to 1956321 tid 2 2024-04-26T00:29:08.399 INFO:tasks.rados.rados.0.smithi045.stdout:1561: writing smithi04526579-36 from 2529302 to 3093724 tid 3 2024-04-26T00:29:08.406 INFO:tasks.rados.rados.0.smithi045.stdout:1561: writing smithi04526579-36 from 3528143 to 3697639 tid 4 2024-04-26T00:29:08.406 INFO:tasks.rados.rados.0.smithi045.stdout:1562: copy_from oid 33 from oid 42 current snap is 179 2024-04-26T00:29:08.406 INFO:tasks.rados.rados.0.smithi045.stdout:1561: finishing write tid 1 to smithi04526579-36 2024-04-26T00:29:08.406 INFO:tasks.rados.rados.0.smithi045.stdout:1561: finishing write tid 2 to smithi04526579-36 2024-04-26T00:29:08.407 INFO:tasks.rados.rados.0.smithi045.stdout:1561: finishing write tid 3 to smithi04526579-36 2024-04-26T00:29:08.407 INFO:tasks.rados.rados.0.smithi045.stdout:1563: delete oid 24 current snap is 179 2024-04-26T00:29:08.411 INFO:tasks.rados.rados.0.smithi045.stdout:1561: finishing write tid 4 to smithi04526579-36 2024-04-26T00:29:08.412 INFO:tasks.rados.rados.0.smithi045.stdout:1561: finishing write tid 5 to smithi04526579-36 2024-04-26T00:29:08.412 INFO:tasks.rados.rados.0.smithi045.stdout:1561: finishing write tid 6 to smithi04526579-36 2024-04-26T00:29:08.412 INFO:tasks.rados.rados.0.smithi045.stdout:1561: oid 36 updating version 0 to 764 2024-04-26T00:29:08.412 INFO:tasks.rados.rados.0.smithi045.stdout:1561: oid 36 version 764 is already newer than 763 2024-04-26T00:29:08.412 INFO:tasks.rados.rados.0.smithi045.stdout:1561: oid 36 updating version 764 to 765 2024-04-26T00:29:08.412 INFO:tasks.rados.rados.0.smithi045.stdout:1561: oid 36 version 765 is already newer than 762 2024-04-26T00:29:08.412 INFO:tasks.rados.rados.0.smithi045.stdout:1561: oid 36 version 765 is already newer than 761 2024-04-26T00:29:08.412 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 765 (ObjNum 577 snap 179 seq_num 577) dirty exists 2024-04-26T00:29:08.412 INFO:tasks.rados.rados.0.smithi045.stdout:1561: left oid 36 (ObjNum 577 snap 179 seq_num 577) 2024-04-26T00:29:08.416 INFO:tasks.rados.rados.0.smithi045.stdout:1561: done (2 left) 2024-04-26T00:29:08.416 INFO:tasks.rados.rados.0.smithi045.stdout:1563: done (1 left) 2024-04-26T00:29:08.416 INFO:tasks.rados.rados.0.smithi045.stdout:1564: snap_create 2024-04-26T00:29:08.442 INFO:tasks.rados.rados.0.smithi045.stdout:1562: finishing copy_from racing read to smithi04526579-33 2024-04-26T00:29:08.445 INFO:tasks.rados.rados.0.smithi045.stdout:1562: finishing copy_from to smithi04526579-33 2024-04-26T00:29:08.445 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 895 (ObjNum 557 snap 173 seq_num 557) dirty exists 2024-04-26T00:29:09.278 INFO:tasks.rados.rados.0.smithi045.stdout:1562: done (1 left) 2024-04-26T00:29:09.279 INFO:tasks.rados.rados.0.smithi045.stdout:1564: done (0 left) 2024-04-26T00:29:09.279 INFO:tasks.rados.rados.0.smithi045.stdout:1565: rollback oid 50 current snap is 180 2024-04-26T00:29:09.279 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 50 to 146 2024-04-26T00:29:09.279 INFO:tasks.rados.rados.0.smithi045.stdout:1566: snap_remove snap 127 2024-04-26T00:29:09.813 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-04-26T00:29:09.813 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-04-26T00:29:09.813 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-26T00:29:09.814 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-26T00:29:09.816 INFO:tasks.ceph.mon.b:Started 2024-04-26T00:29:09.817 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:29:09.817 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:29:09.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.962+0000 7fce837b1640 10 monclient: get_monmap_and_config 2024-04-26T00:29:09.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.962+0000 7fce837b1640 10 monclient: build_initial_monmap 2024-04-26T00:29:09.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.962+0000 7fce837b1640 10 monclient: monmap: 2024-04-26T00:29:09.951 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:29:09.951 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:09.951 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:29:09.964917+0000 2024-04-26T00:29:09.951 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:29:09.964917+0000 2024-04-26T00:29:09.951 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:29:09.951 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:09.951 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:29:09.951 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:29:09.951 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:29:09.951 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:29:09.951 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:29:09.951 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:09.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.962+0000 7fce837b1640 1 Processor -- start 2024-04-26T00:29:09.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.962+0000 7fce837b1640 1 -- start start 2024-04-26T00:29:09.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.962+0000 7fce837b1640 10 monclient: init 2024-04-26T00:29:09.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.962+0000 7fce837b1640 10 monclient: _reopen_session rank -1 2024-04-26T00:29:09.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.962+0000 7fce837b1640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-04-26T00:29:09.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.962+0000 7fce837b1640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce7c0a0260 0x7fce7c0a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:09.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.962+0000 7fce837b1640 10 monclient(hunting): picked mon.noname-b con 0x7fce7c0a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:29:09.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.962+0000 7fce837b1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fce7c134b30 addr v1:172.21.15.45:6789/0 2024-04-26T00:29:09.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.962+0000 7fce837b1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fce7c135660 addr v1:172.21.15.37:6789/0 2024-04-26T00:29:09.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.962+0000 7fce837b1640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce7c0a1130 con 0x7fce7c135660 2024-04-26T00:29:09.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.962+0000 7fce837b1640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce7c06cff0 con 0x7fce7c134b30 2024-04-26T00:29:09.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.962+0000 7fce837b1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:09.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.962+0000 7fce837b1640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fce7c09ec10 con 0x7fce7c0a0260 2024-04-26T00:29:09.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.962+0000 7fce837b1640 10 monclient(hunting): _renew_subs 2024-04-26T00:29:09.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.962+0000 7fce837b1640 10 monclient(hunting): authenticate will time out at 1531.270654s 2024-04-26T00:29:09.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.962+0000 7fce8154d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce7c0a0260 0x7fce7c0a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:09.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.962+0000 7fce8154d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce7c0a0260 0x7fce7c0a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:41256/0 (socket says 172.21.15.37:41256) 2024-04-26T00:29:09.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.962+0000 7fce8154d640 1 -- 172.21.15.37:0/2574231354 learned_addr learned my addr 172.21.15.37:0/2574231354 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:29:09.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.962+0000 7fce8154d640 10 monclient(hunting): get_auth_request con 0x7fce7c0a0260 auth_method 0 2024-04-26T00:29:09.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.962+0000 7fce8154d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:09.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.962+0000 7fce8154d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:09.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.962+0000 7fce8154d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:09.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.962+0000 7fce8154d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:09.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.962+0000 7fce8154d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:09.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.962+0000 7fce8154d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:09.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.962+0000 7fce8154d640 10 monclient(hunting): handle_auth_done global_id 84374 payload 274 2024-04-26T00:29:09.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.962+0000 7fce8154d640 1 -- 172.21.15.37:0/2574231354 >> v1:172.21.15.45:6789/0 conn(0x7fce7c134b30 legacy=0x7fce7c134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:29:09.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.962+0000 7fce8154d640 1 -- 172.21.15.37:0/2574231354 >> v1:172.21.15.37:6789/0 conn(0x7fce7c135660 legacy=0x7fce7c13ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:29:09.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.962+0000 7fce8154d640 10 monclient: _finish_hunting 0 2024-04-26T00:29:09.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.962+0000 7fce8154d640 1 monclient: found mon.noname-b 2024-04-26T00:29:09.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.962+0000 7fce8154d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:29:09.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.962+0000 7fce8154d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:29:09.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.962+0000 7fce8154d640 1 -- 172.21.15.37:0/2574231354 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce7c09bc50 con 0x7fce7c0a0260 2024-04-26T00:29:09.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.966+0000 7fce8154d640 1 -- 172.21.15.37:0/2574231354 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce7c0a0260 msgr2=0x7fce7c0a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:09.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.966+0000 7fce8154d640 1 --2- 172.21.15.37:0/2574231354 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce7c0a0260 0x7fce7c0a0640 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fce780098a0 tx=0x7fce7801c1e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54d58a414b7ad90d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:09.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.966+0000 7fce73fff640 1 -- 172.21.15.37:0/2574231354 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce780b0070 con 0x7fce7c0a0260 2024-04-26T00:29:09.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.966+0000 7fce73fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:09.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.966+0000 7fce73fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:29:09.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.966+0000 7fce73fff640 10 monclient: dump: 2024-04-26T00:29:09.958 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:09.958 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:09.958 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:09.958 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:09.958 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:09.958 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:09.958 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:09.958 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:09.959 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:09.959 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:09.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.966+0000 7fce73fff640 10 monclient: _finish_auth 0 2024-04-26T00:29:09.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.966+0000 7fce73fff640 10 monclient: _check_auth_tickets 2024-04-26T00:29:09.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.966+0000 7fce73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:29:09.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.966+0000 7fce73fff640 1 -- 172.21.15.37:0/2574231354 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce780ab600 con 0x7fce7c0a0260 2024-04-26T00:29:09.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.966+0000 7fce837b1640 5 monclient: authenticate success, global_id 84374 2024-04-26T00:29:09.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.966+0000 7fce837b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:29:09.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.966+0000 7fce73fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:29:09.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.966+0000 7fce837b1640 10 monclient: get_monmap_and_config success 2024-04-26T00:29:09.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.966+0000 7fce73fff640 1 -- 172.21.15.37:0/2574231354 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce780ab9d0 con 0x7fce7c0a0260 2024-04-26T00:29:09.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.966+0000 7fce837b1640 4 set_mon_vals no callback set 2024-04-26T00:29:09.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.966+0000 7fce73fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:09.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.966+0000 7fce73fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:29:09.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.966+0000 7fce73fff640 10 monclient: dump: 2024-04-26T00:29:09.960 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:09.960 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:09.960 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:09.960 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:09.960 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:09.960 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:09.960 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:09.960 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:09.960 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:09.960 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:09.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce837b1640 10 monclient: shutdown 2024-04-26T00:29:09.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce837b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:29:09.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce837b1640 1 -- 172.21.15.37:0/2574231354 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce7c0a0260 msgr2=0x7fce7c0a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:09.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce837b1640 1 --2- 172.21.15.37:0/2574231354 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce7c0a0260 0x7fce7c0a0640 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fce780098a0 tx=0x7fce7801c1e0 comp rx=0 tx=0).stop 2024-04-26T00:29:09.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce837b1640 1 -- 172.21.15.37:0/2574231354 shutdown_connections 2024-04-26T00:29:09.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce837b1640 1 --2- 172.21.15.37:0/2574231354 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce7c0a0260 0x7fce7c0a0640 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:09.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce837b1640 1 -- 172.21.15.37:0/2574231354 >> 172.21.15.37:0/2574231354 conn(0x7fce7c0a1ad0 msgr2=0x7fce7c0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:29:09.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce837b1640 1 -- 172.21.15.37:0/2574231354 shutdown_connections 2024-04-26T00:29:09.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce837b1640 1 -- 172.21.15.37:0/2574231354 wait complete. 2024-04-26T00:29:09.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce837b1640 10 monclient: build_initial_monmap 2024-04-26T00:29:09.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce837b1640 10 monclient: monmap: 2024-04-26T00:29:09.965 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:29:09.965 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:29:09.965 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:29:09.965 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:29:09.965 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:29:09.965 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:29:09.965 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:29:09.965 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:29:09.966 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:29:09.966 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:09.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce837b1640 1 Processor -- start 2024-04-26T00:29:09.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce837b1640 1 -- start start 2024-04-26T00:29:09.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce837b1640 10 monclient: init 2024-04-26T00:29:09.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce837b1640 10 monclient: _reopen_session rank -1 2024-04-26T00:29:09.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce837b1640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:29:09.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce837b1640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce7c0a0260 0x7fce7c09c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:09.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce837b1640 10 monclient(hunting): picked mon.noname-c con 0x7fce7c0a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:29:09.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce837b1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce7c134b30 0x7fce7c09ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:09.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce837b1640 10 monclient(hunting): picked mon.noname-b con 0x7fce7c134b30 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:29:09.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce837b1640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce7c135660 0x7fce7c09cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:09.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce837b1640 10 monclient(hunting): picked mon.noname-a con 0x7fce7c135660 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:29:09.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce837b1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:09.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce837b1640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fce7c09f4c0 con 0x7fce7c135660 2024-04-26T00:29:09.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce837b1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:09.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce837b1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fce7c09f0f0 con 0x7fce7c134b30 2024-04-26T00:29:09.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce837b1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:09.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce837b1640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fce7c140db0 con 0x7fce7c0a0260 2024-04-26T00:29:09.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce837b1640 10 monclient(hunting): _renew_subs 2024-04-26T00:29:09.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce837b1640 10 monclient(hunting): authenticate will time out at 1531.277784s 2024-04-26T00:29:09.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce81d4e640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce7c135660 0x7fce7c09cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:09.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce8154d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce7c0a0260 0x7fce7c09c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:09.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce81d4e640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce7c135660 0x7fce7c09cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33756/0 (socket says 172.21.15.37:33756) 2024-04-26T00:29:09.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce81d4e640 1 -- 172.21.15.37:0/140127888 learned_addr learned my addr 172.21.15.37:0/140127888 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:29:09.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce81d4e640 10 monclient(hunting): get_auth_request con 0x7fce7c135660 auth_method 0 2024-04-26T00:29:09.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce81d4e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:09.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce81d4e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:09.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce81d4e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:09.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce8154d640 10 monclient(hunting): get_auth_request con 0x7fce7c0a0260 auth_method 0 2024-04-26T00:29:09.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce8154d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:09.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce8154d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:09.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce8154d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:09.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce81d4e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:09.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce81d4e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:09.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce81d4e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:09.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce8154d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:09.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce8154d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:09.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce8154d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:09.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce80d4c640 1 --2- 172.21.15.37:0/140127888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce7c134b30 0x7fce7c09ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:09.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce81d4e640 10 monclient(hunting): handle_auth_done global_id 94311 payload 824 2024-04-26T00:29:09.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce81d4e640 1 -- 172.21.15.37:0/140127888 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce7c0a0260 msgr2=0x7fce7c09c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:09.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce81d4e640 1 --2- 172.21.15.37:0/140127888 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce7c0a0260 0x7fce7c09c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:09.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce81d4e640 1 -- 172.21.15.37:0/140127888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce7c134b30 msgr2=0x7fce7c09ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:09.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce81d4e640 1 --2- 172.21.15.37:0/140127888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce7c134b30 0x7fce7c09ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:09.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce81d4e640 10 monclient: _finish_hunting 0 2024-04-26T00:29:09.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce81d4e640 1 monclient: found mon.noname-a 2024-04-26T00:29:09.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce81d4e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:29:09.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce81d4e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:29:09.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce81d4e640 1 -- 172.21.15.37:0/140127888 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce7c0729d0 con 0x7fce7c135660 2024-04-26T00:29:09.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce80d4c640 10 monclient: get_auth_request con 0x7fce7c134b30 auth_method 0 2024-04-26T00:29:09.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce8154d640 1 --2- 172.21.15.37:0/140127888 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce7c0a0260 0x7fce7c09c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:29:09.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.970+0000 7fce80d4c640 1 --2- 172.21.15.37:0/140127888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce7c134b30 0x7fce7c09ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:29:09.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.974+0000 7fce81d4e640 1 -- 172.21.15.37:0/140127888 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce7c135660 msgr2=0x7fce7c09cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:09.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.974+0000 7fce81d4e640 1 --2- 172.21.15.37:0/140127888 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce7c135660 0x7fce7c09cf80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fce7400c780 tx=0x7fce7400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:09.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.974+0000 7fce71ffb640 1 -- 172.21.15.37:0/140127888 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce7400ac00 con 0x7fce7c135660 2024-04-26T00:29:09.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.974+0000 7fce71ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:09.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.974+0000 7fce71ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:29:09.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.974+0000 7fce71ffb640 10 monclient: dump: 2024-04-26T00:29:09.970 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:09.970 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:09.970 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:09.970 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:09.970 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:09.970 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:09.970 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:09.970 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:09.971 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:09.971 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:09.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.974+0000 7fce71ffb640 10 monclient: _finish_auth 0 2024-04-26T00:29:09.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.974+0000 7fce71ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:29:09.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.974+0000 7fce71ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:29:09.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.974+0000 7fce71ffb640 1 -- 172.21.15.37:0/140127888 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce7400ada0 con 0x7fce7c135660 2024-04-26T00:29:09.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.974+0000 7fce71ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:29:09.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.974+0000 7fce71ffb640 1 -- 172.21.15.37:0/140127888 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce74004bb0 con 0x7fce7c135660 2024-04-26T00:29:09.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.974+0000 7fce71ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:09.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.974+0000 7fce71ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:29:09.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.974+0000 7fce71ffb640 10 monclient: dump: 2024-04-26T00:29:09.971 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:09.971 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:09.971 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:09.972 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:09.972 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:09.972 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:09.972 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:09.972 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:09.972 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:09.972 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:09.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.974+0000 7fce72ffd640 4 set_mon_vals no callback set 2024-04-26T00:29:09.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.974+0000 7fce837b1640 5 monclient: authenticate success, global_id 94311 2024-04-26T00:29:09.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.974+0000 7fce837b1640 10 monclient: _renew_subs 2024-04-26T00:29:09.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.974+0000 7fce837b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:29:09.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.974+0000 7fce837b1640 1 -- 172.21.15.37:0/140127888 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce7c09e1a0 con 0x7fce7c135660 2024-04-26T00:29:09.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.974+0000 7fce837b1640 10 monclient: _renew_subs 2024-04-26T00:29:09.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.974+0000 7fce837b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:29:09.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.974+0000 7fce837b1640 1 -- 172.21.15.37:0/140127888 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce7c13a2f0 con 0x7fce7c135660 2024-04-26T00:29:09.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.974+0000 7fce837b1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:29:09.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.974+0000 7fce837b1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:29:09.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.974+0000 7fce837b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:29:09.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.974+0000 7fce71ffb640 1 -- 172.21.15.37:0/140127888 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fce74005030 con 0x7fce7c135660 2024-04-26T00:29:09.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.974+0000 7fce837b1640 1 -- 172.21.15.37:0/140127888 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce7c066f40 con 0x7fce7c135660 2024-04-26T00:29:09.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.974+0000 7fce71ffb640 1 --2- 172.21.15.37:0/140127888 >> v2:172.21.15.45:6812/2379918623 conn(0x7fce60035e80 0x7fce60038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:09.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.974+0000 7fce71ffb640 1 -- 172.21.15.37:0/140127888 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(361..361 src has 177..361) ==== 3959+0+0 (secure 0 0 0) 0x7fce74048de0 con 0x7fce7c135660 2024-04-26T00:29:09.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.974+0000 7fce8154d640 1 --2- 172.21.15.37:0/140127888 >> v2:172.21.15.45:6812/2379918623 conn(0x7fce60035e80 0x7fce60038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:09.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.974+0000 7fce8154d640 10 monclient: get_auth_request con 0x7fce60035e80 auth_method 0 2024-04-26T00:29:09.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.978+0000 7fce8154d640 1 --2- 172.21.15.37:0/140127888 >> v2:172.21.15.45:6812/2379918623 conn(0x7fce60035e80 0x7fce60038340 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fce7c135bd0 tx=0x7fce780a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:09.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.978+0000 7fce71ffb640 1 -- 172.21.15.37:0/140127888 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fce74020070 con 0x7fce7c135660 2024-04-26T00:29:09.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.978+0000 7fce71ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:29:09.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:09.978+0000 7fce71ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:29:10.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:10.202+0000 7fce837b1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:29:10.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:10.202+0000 7fce837b1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:29:10.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:10.202+0000 7fce837b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:29:10.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:10.202+0000 7fce837b1640 1 -- 172.21.15.37:0/140127888 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fce7c06b9a0 con 0x7fce7c135660 2024-04-26T00:29:10.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:10.202+0000 7fce71ffb640 1 -- 172.21.15.37:0/140127888 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7fce74004e10 con 0x7fce7c135660 2024-04-26T00:29:10.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:10.202+0000 7fce71ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:29:10.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:10.202+0000 7fce71ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:29:10.192 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":68,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:29:10.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:10.206+0000 7fce837b1640 1 -- 172.21.15.37:0/140127888 >> v2:172.21.15.45:6812/2379918623 conn(0x7fce60035e80 msgr2=0x7fce60038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:10.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:10.206+0000 7fce837b1640 1 --2- 172.21.15.37:0/140127888 >> v2:172.21.15.45:6812/2379918623 conn(0x7fce60035e80 0x7fce60038340 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fce7c135bd0 tx=0x7fce780a6000 comp rx=0 tx=0).stop 2024-04-26T00:29:10.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:10.206+0000 7fce837b1640 10 monclient: shutdown 2024-04-26T00:29:10.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:10.206+0000 7fce837b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:29:10.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:10.206+0000 7fce837b1640 1 -- 172.21.15.37:0/140127888 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce7c135660 msgr2=0x7fce7c09cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:10.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:10.206+0000 7fce837b1640 1 --2- 172.21.15.37:0/140127888 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce7c135660 0x7fce7c09cf80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fce7400c780 tx=0x7fce7400cc40 comp rx=0 tx=0).stop 2024-04-26T00:29:10.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:10.206+0000 7fce837b1640 1 -- 172.21.15.37:0/140127888 shutdown_connections 2024-04-26T00:29:10.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:10.206+0000 7fce837b1640 1 --2- 172.21.15.37:0/140127888 >> v2:172.21.15.45:6812/2379918623 conn(0x7fce60035e80 0x7fce60038340 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:10.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:10.206+0000 7fce837b1640 1 --2- 172.21.15.37:0/140127888 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce7c135660 0x7fce7c09cf80 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:10.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:10.206+0000 7fce837b1640 1 --2- 172.21.15.37:0/140127888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce7c134b30 0x7fce7c09ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:10.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:10.206+0000 7fce837b1640 1 --2- 172.21.15.37:0/140127888 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce7c0a0260 0x7fce7c09c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:10.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:10.206+0000 7fce837b1640 1 -- 172.21.15.37:0/140127888 >> 172.21.15.37:0/140127888 conn(0x7fce7c0a1ad0 msgr2=0x7fce7c0a3d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:29:10.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:10.206+0000 7fce837b1640 1 -- 172.21.15.37:0/140127888 shutdown_connections 2024-04-26T00:29:10.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:10.206+0000 7fce837b1640 1 -- 172.21.15.37:0/140127888 wait complete. 2024-04-26T00:29:10.274 INFO:tasks.rados.rados.0.smithi045.stdout:1566: done (1 left) 2024-04-26T00:29:10.274 INFO:tasks.rados.rados.0.smithi045.stdout:1567: write (excl) oid 18 current snap is 180 2024-04-26T00:29:10.274 INFO:tasks.rados.rados.0.smithi045.stdout:1567: seq_num 579 ranges {531253=411999,1381780=411527,2300501=586966,3376974=354279} 2024-04-26T00:29:10.293 INFO:tasks.rados.rados.0.smithi045.stdout:1567: writing smithi04526579-18 from 531253 to 943252 tid 1 2024-04-26T00:29:10.309 INFO:tasks.rados.rados.0.smithi045.stdout:1567: writing smithi04526579-18 from 1381780 to 1793307 tid 2 2024-04-26T00:29:10.326 INFO:tasks.rados.rados.0.smithi045.stdout:1567: writing smithi04526579-18 from 2300501 to 2887467 tid 3 2024-04-26T00:29:10.337 INFO:tasks.rados.rados.0.smithi045.stdout:1567: writing smithi04526579-18 from 3376974 to 3731253 tid 4 2024-04-26T00:29:10.338 INFO:tasks.rados.rados.0.smithi045.stdout:1568: delete oid 15 current snap is 180 2024-04-26T00:29:10.338 INFO:tasks.rados.rados.0.smithi045.stdout:1565: finishing rollback tid 0 to smithi04526579-50 2024-04-26T00:29:10.338 INFO:tasks.rados.rados.0.smithi045.stdout:1565: finishing rollback tid 1 to smithi04526579-50 2024-04-26T00:29:10.338 INFO:tasks.rados.rados.0.smithi045.stdout:1565: finishing rollback tid 2 to smithi04526579-50 2024-04-26T00:29:10.338 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 880 (ObjNum 136 snap 30 seq_num 136) dirty exists 2024-04-26T00:29:10.338 INFO:tasks.rados.rados.0.smithi045.stdout:1567: finishing write tid 1 to smithi04526579-18 2024-04-26T00:29:10.339 INFO:tasks.rados.rados.0.smithi045.stdout:1567: finishing write tid 2 to smithi04526579-18 2024-04-26T00:29:10.344 INFO:tasks.rados.rados.0.smithi045.stdout:1567: finishing write tid 3 to smithi04526579-18 2024-04-26T00:29:10.345 INFO:tasks.rados.rados.0.smithi045.stdout:1565: done (2 left) 2024-04-26T00:29:10.345 INFO:tasks.rados.rados.0.smithi045.stdout:1568: done (1 left) 2024-04-26T00:29:10.345 INFO:tasks.rados.rados.0.smithi045.stdout:1569: rollback oid 14 current snap is 180 2024-04-26T00:29:10.345 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 14 to 149 2024-04-26T00:29:10.345 INFO:tasks.rados.rados.0.smithi045.stdout:1567: finishing write tid 4 to smithi04526579-18 2024-04-26T00:29:10.345 INFO:tasks.rados.rados.0.smithi045.stdout:1570: copy_from oid 39 from oid 33 current snap is 180 2024-04-26T00:29:10.345 INFO:tasks.rados.rados.0.smithi045.stdout:1571: delete oid 19 current snap is 180 2024-04-26T00:29:10.346 INFO:tasks.rados.rados.0.smithi045.stdout:1567: finishing write tid 5 to smithi04526579-18 2024-04-26T00:29:10.347 INFO:tasks.rados.rados.0.smithi045.stdout:1567: finishing write tid 6 to smithi04526579-18 2024-04-26T00:29:10.347 INFO:tasks.rados.rados.0.smithi045.stdout:1567: oid 18 updating version 0 to 1259 2024-04-26T00:29:10.347 INFO:tasks.rados.rados.0.smithi045.stdout:1567: oid 18 updating version 1259 to 1260 2024-04-26T00:29:10.347 INFO:tasks.rados.rados.0.smithi045.stdout:1567: oid 18 updating version 1260 to 1261 2024-04-26T00:29:10.347 INFO:tasks.rados.rados.0.smithi045.stdout:1567: oid 18 version 1261 is already newer than 1254 2024-04-26T00:29:10.347 INFO:tasks.rados.rados.0.smithi045.stdout:1567: oid 18 version 1261 is already newer than 1249 2024-04-26T00:29:10.347 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 1261 (ObjNum 579 snap 180 seq_num 579) dirty exists 2024-04-26T00:29:10.347 INFO:tasks.rados.rados.0.smithi045.stdout:1567: left oid 18 (ObjNum 579 snap 180 seq_num 579) 2024-04-26T00:29:10.356 INFO:tasks.rados.rados.0.smithi045.stdout:1567: done (3 left) 2024-04-26T00:29:10.357 INFO:tasks.rados.rados.0.smithi045.stdout:1571: done (2 left) 2024-04-26T00:29:10.357 INFO:tasks.rados.rados.0.smithi045.stdout:1572: write oid 28 current snap is 180 2024-04-26T00:29:10.357 INFO:tasks.rados.rados.0.smithi045.stdout:1569: finishing rollback tid 0 to smithi04526579-14 2024-04-26T00:29:10.357 INFO:tasks.rados.rados.0.smithi045.stdout:1572: seq_num 582 ranges {581674=637746,1840873=558854,3066417=537135} 2024-04-26T00:29:10.376 INFO:tasks.rados.rados.0.smithi045.stdout:1572: writing smithi04526579-28 from 581674 to 1219420 tid 1 2024-04-26T00:29:10.395 INFO:tasks.rados.rados.0.smithi045.stdout:1572: writing smithi04526579-28 from 1840873 to 2399727 tid 2 2024-04-26T00:29:10.410 INFO:tasks.rados.rados.0.smithi045.stdout:1572: writing smithi04526579-28 from 3066417 to 3603552 tid 3 2024-04-26T00:29:10.410 INFO:tasks.rados.rados.0.smithi045.stdout:1573: rollback oid 50 current snap is 180 2024-04-26T00:29:10.410 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 50 to 154 2024-04-26T00:29:10.410 INFO:tasks.rados.rados.0.smithi045.stdout:1569: finishing rollback tid 1 to smithi04526579-14 2024-04-26T00:29:10.410 INFO:tasks.rados.rados.0.smithi045.stdout:1569: finishing rollback tid 2 to smithi04526579-14 2024-04-26T00:29:10.410 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 621 (ObjNum 433 snap 136 seq_num 433) dirty exists 2024-04-26T00:29:10.410 INFO:tasks.rados.rados.0.smithi045.stdout:1570: finishing copy_from racing read to smithi04526579-39 2024-04-26T00:29:10.411 INFO:tasks.rados.rados.0.smithi045.stdout:1570: finishing copy_from to smithi04526579-39 2024-04-26T00:29:10.411 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 895 (ObjNum 557 snap 173 seq_num 557) dirty exists 2024-04-26T00:29:10.411 INFO:tasks.rados.rados.0.smithi045.stdout:1572: finishing write tid 1 to smithi04526579-28 2024-04-26T00:29:10.411 INFO:tasks.rados.rados.0.smithi045.stdout:1572: finishing write tid 2 to smithi04526579-28 2024-04-26T00:29:10.411 INFO:tasks.rados.rados.0.smithi045.stdout:1569: done (3 left) 2024-04-26T00:29:10.411 INFO:tasks.rados.rados.0.smithi045.stdout:1570: done (2 left) 2024-04-26T00:29:10.411 INFO:tasks.rados.rados.0.smithi045.stdout:1574: delete oid 30 current snap is 180 2024-04-26T00:29:10.423 INFO:tasks.rados.rados.0.smithi045.stdout:1572: finishing write tid 3 to smithi04526579-28 2024-04-26T00:29:10.424 INFO:tasks.rados.rados.0.smithi045.stdout:1573: finishing rollback tid 0 to smithi04526579-50 2024-04-26T00:29:10.424 INFO:tasks.rados.rados.0.smithi045.stdout:1572: finishing write tid 4 to smithi04526579-28 2024-04-26T00:29:10.424 INFO:tasks.rados.rados.0.smithi045.stdout:1573: finishing rollback tid 1 to smithi04526579-50 2024-04-26T00:29:10.424 INFO:tasks.rados.rados.0.smithi045.stdout:1572: finishing write tid 5 to smithi04526579-28 2024-04-26T00:29:10.424 INFO:tasks.rados.rados.0.smithi045.stdout:1572: oid 28 updating version 0 to 1058 2024-04-26T00:29:10.424 INFO:tasks.rados.rados.0.smithi045.stdout:1572: oid 28 version 1058 is already newer than 1056 2024-04-26T00:29:10.424 INFO:tasks.rados.rados.0.smithi045.stdout:1572: oid 28 version 1058 is already newer than 1057 2024-04-26T00:29:10.424 INFO:tasks.rados.rados.0.smithi045.stdout:1572: oid 28 updating version 1058 to 1059 2024-04-26T00:29:10.424 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 1059 (ObjNum 582 snap 180 seq_num 582) dirty exists 2024-04-26T00:29:10.425 INFO:tasks.rados.rados.0.smithi045.stdout:1572: left oid 28 (ObjNum 582 snap 180 seq_num 582) 2024-04-26T00:29:10.425 INFO:tasks.rados.rados.0.smithi045.stdout:1573: finishing rollback tid 2 to smithi04526579-50 2024-04-26T00:29:10.425 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 883 (ObjNum 470 snap 147 seq_num 470) dirty exists 2024-04-26T00:29:10.425 INFO:tasks.rados.rados.0.smithi045.stdout:1572: done (2 left) 2024-04-26T00:29:10.425 INFO:tasks.rados.rados.0.smithi045.stdout:1573: done (1 left) 2024-04-26T00:29:10.425 INFO:tasks.rados.rados.0.smithi045.stdout:1574: done (0 left) 2024-04-26T00:29:10.425 INFO:tasks.rados.rados.0.smithi045.stdout:1575: snap_create 2024-04-26T00:29:11.362 INFO:tasks.rados.rados.0.smithi045.stdout:1575: done (0 left) 2024-04-26T00:29:11.363 INFO:tasks.rados.rados.0.smithi045.stdout:1576: delete oid 49 current snap is 181 2024-04-26T00:29:11.373 INFO:tasks.rados.rados.0.smithi045.stdout:1576: done (0 left) 2024-04-26T00:29:11.373 INFO:tasks.rados.rados.0.smithi045.stdout:1577: snap_create 2024-04-26T00:29:12.365 INFO:tasks.rados.rados.0.smithi045.stdout:1577: done (0 left) 2024-04-26T00:29:12.365 INFO:tasks.rados.rados.0.smithi045.stdout:1578: delete oid 11 current snap is 182 2024-04-26T00:29:12.373 INFO:tasks.rados.rados.0.smithi045.stdout:1578: done (0 left) 2024-04-26T00:29:12.373 INFO:tasks.rados.rados.0.smithi045.stdout:1579: write oid 39 current snap is 182 2024-04-26T00:29:12.373 INFO:tasks.rados.rados.0.smithi045.stdout:1579: seq_num 586 ranges {648902=655095,2074801=566192,3176109=672793} 2024-04-26T00:29:12.394 INFO:tasks.rados.rados.0.smithi045.stdout:1579: writing smithi04526579-39 from 648902 to 1303997 tid 1 2024-04-26T00:29:12.416 INFO:tasks.rados.rados.0.smithi045.stdout:1579: writing smithi04526579-39 from 2074801 to 2640993 tid 2 2024-04-26T00:29:12.435 INFO:tasks.rados.rados.0.smithi045.stdout:1579: writing smithi04526579-39 from 3176109 to 3848902 tid 3 2024-04-26T00:29:12.435 INFO:tasks.rados.rados.0.smithi045.stdout:1580: snap_remove snap 149 2024-04-26T00:29:13.212 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:29:13.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.354+0000 7f5517cd3640 10 monclient: get_monmap_and_config 2024-04-26T00:29:13.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.354+0000 7f5517cd3640 10 monclient: build_initial_monmap 2024-04-26T00:29:13.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.354+0000 7f5517cd3640 10 monclient: monmap: 2024-04-26T00:29:13.344 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:29:13.344 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:13.344 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:29:13.358409+0000 2024-04-26T00:29:13.344 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:29:13.358409+0000 2024-04-26T00:29:13.344 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:29:13.344 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:13.344 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:29:13.344 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:29:13.345 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:29:13.345 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:29:13.345 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:29:13.345 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:13.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.354+0000 7f5517cd3640 1 Processor -- start 2024-04-26T00:29:13.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.354+0000 7f5517cd3640 1 -- start start 2024-04-26T00:29:13.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.354+0000 7f5517cd3640 10 monclient: init 2024-04-26T00:29:13.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5517cd3640 10 monclient: _reopen_session rank -1 2024-04-26T00:29:13.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5517cd3640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-04-26T00:29:13.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5517cd3640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f5510136e10 0x7f551009c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:13.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5517cd3640 10 monclient(hunting): picked mon.noname-a con 0x7f5510136e10 addr v2:172.21.15.37:3300/0 2024-04-26T00:29:13.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5517cd3640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f55101371f0 0x7f5510137600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:13.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5517cd3640 10 monclient(hunting): picked mon.noname-c con 0x7f55101371f0 addr v2:172.21.15.45:3300/0 2024-04-26T00:29:13.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5517cd3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5510135d90 0x7f5510136240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:13.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5517cd3640 10 monclient(hunting): picked mon.noname-b con 0x7f5510135d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:29:13.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5517cd3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:13.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5517cd3640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f551009b8f0 con 0x7f5510136e10 2024-04-26T00:29:13.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5517cd3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:13.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5517cd3640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f551009b680 con 0x7f55101371f0 2024-04-26T00:29:13.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5517cd3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:13.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5517cd3640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f551009bbd0 con 0x7f5510135d90 2024-04-26T00:29:13.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5517cd3640 10 monclient(hunting): _renew_subs 2024-04-26T00:29:13.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5517cd3640 10 monclient(hunting): authenticate will time out at 1534.664349s 2024-04-26T00:29:13.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5515a6f640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f5510136e10 0x7f551009c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:13.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5516270640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5510135d90 0x7f5510136240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:13.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5515a6f640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f5510136e10 0x7f551009c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33772/0 (socket says 172.21.15.37:33772) 2024-04-26T00:29:13.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5515a6f640 1 -- 172.21.15.37:0/3542993161 learned_addr learned my addr 172.21.15.37:0/3542993161 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:29:13.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5515a6f640 10 monclient(hunting): get_auth_request con 0x7f5510136e10 auth_method 0 2024-04-26T00:29:13.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5515a6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:13.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5515a6f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:13.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5515a6f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:13.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5516270640 10 monclient(hunting): get_auth_request con 0x7f5510135d90 auth_method 0 2024-04-26T00:29:13.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5516270640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:13.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5516270640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:13.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5516270640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:13.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5515a6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:13.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5515a6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:13.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5515a6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:13.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5516270640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:13.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5516270640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:13.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5516270640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:13.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f551526e640 1 --2- 172.21.15.37:0/3542993161 >> v2:172.21.15.45:3300/0 conn(0x7f55101371f0 0x7f5510137600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:13.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f551526e640 10 monclient(hunting): get_auth_request con 0x7f55101371f0 auth_method 0 2024-04-26T00:29:13.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f551526e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:13.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f551526e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:13.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f551526e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:13.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5515a6f640 10 monclient(hunting): handle_auth_done global_id 94314 payload 274 2024-04-26T00:29:13.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5515a6f640 1 -- 172.21.15.37:0/3542993161 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5510135d90 msgr2=0x7f5510136240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:29:13.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5515a6f640 1 --2- 172.21.15.37:0/3542993161 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5510135d90 0x7f5510136240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:13.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5515a6f640 1 -- 172.21.15.37:0/3542993161 >> v2:172.21.15.45:3300/0 conn(0x7f55101371f0 msgr2=0x7f5510137600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:29:13.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5515a6f640 1 --2- 172.21.15.37:0/3542993161 >> v2:172.21.15.45:3300/0 conn(0x7f55101371f0 0x7f5510137600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:13.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5515a6f640 10 monclient: _finish_hunting 0 2024-04-26T00:29:13.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5515a6f640 1 monclient: found mon.noname-a 2024-04-26T00:29:13.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5515a6f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:29:13.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5515a6f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:29:13.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5515a6f640 1 -- 172.21.15.37:0/3542993161 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5510137b40 con 0x7f5510136e10 2024-04-26T00:29:13.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.358+0000 7f5516270640 1 --2- 172.21.15.37:0/3542993161 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5510135d90 0x7f5510136240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:29:13.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5515a6f640 1 -- 172.21.15.37:0/3542993161 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5510136e10 msgr2=0x7f551009c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:13.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5515a6f640 1 --2- 172.21.15.37:0/3542993161 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5510136e10 0x7f551009c5b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5510068e60 tx=0x7f550001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ac76d9beabd3246 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:13.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5514a6d640 1 -- 172.21.15.37:0/3542993161 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55000b1070 con 0x7f5510136e10 2024-04-26T00:29:13.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5514a6d640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:13.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5514a6d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:29:13.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5514a6d640 10 monclient: dump: 2024-04-26T00:29:13.358 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:13.358 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:13.358 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:13.358 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:13.358 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:13.358 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:13.358 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:13.358 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:13.358 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:13.359 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:13.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5514a6d640 10 monclient: _finish_auth 0 2024-04-26T00:29:13.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5514a6d640 10 monclient: _check_auth_tickets 2024-04-26T00:29:13.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5514a6d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:29:13.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5517cd3640 5 monclient: authenticate success, global_id 94314 2024-04-26T00:29:13.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5517cd3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:29:13.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5514a6d640 1 -- 172.21.15.37:0/3542993161 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55000ac620 con 0x7f5510136e10 2024-04-26T00:29:13.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5514a6d640 10 monclient: handle_config config(0 keys) 2024-04-26T00:29:13.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5514a6d640 1 -- 172.21.15.37:0/3542993161 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55000ac9f0 con 0x7f5510136e10 2024-04-26T00:29:13.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5514a6d640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:13.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5514a6d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:29:13.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5514a6d640 10 monclient: dump: 2024-04-26T00:29:13.359 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:13.359 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:13.359 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:13.360 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:13.360 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:13.360 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:13.360 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:13.360 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:13.360 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:13.360 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:13.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5517cd3640 10 monclient: get_monmap_and_config success 2024-04-26T00:29:13.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5517cd3640 4 set_mon_vals no callback set 2024-04-26T00:29:13.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5517cd3640 10 monclient: shutdown 2024-04-26T00:29:13.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5517cd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:29:13.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5517cd3640 1 -- 172.21.15.37:0/3542993161 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5510136e10 msgr2=0x7f551009c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:13.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5517cd3640 1 --2- 172.21.15.37:0/3542993161 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5510136e10 0x7f551009c5b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5510068e60 tx=0x7f550001c3e0 comp rx=0 tx=0).stop 2024-04-26T00:29:13.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5517cd3640 1 -- 172.21.15.37:0/3542993161 shutdown_connections 2024-04-26T00:29:13.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5517cd3640 1 --2- 172.21.15.37:0/3542993161 >> v2:172.21.15.45:3300/0 conn(0x7f55101371f0 0x7f5510137600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:13.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5517cd3640 1 --2- 172.21.15.37:0/3542993161 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5510135d90 0x7f5510136240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:13.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5517cd3640 1 --2- 172.21.15.37:0/3542993161 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5510136e10 0x7f551009c5b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:13.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5517cd3640 1 -- 172.21.15.37:0/3542993161 >> 172.21.15.37:0/3542993161 conn(0x7f55100a64e0 msgr2=0x7f55100a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:29:13.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5517cd3640 1 -- 172.21.15.37:0/3542993161 shutdown_connections 2024-04-26T00:29:13.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5517cd3640 1 -- 172.21.15.37:0/3542993161 wait complete. 2024-04-26T00:29:13.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5517cd3640 10 monclient: build_initial_monmap 2024-04-26T00:29:13.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5517cd3640 10 monclient: monmap: 2024-04-26T00:29:13.362 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:29:13.362 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:29:13.362 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:29:13.363 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:29:13.363 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:29:13.363 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:29:13.363 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:29:13.363 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:29:13.363 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:29:13.363 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:13.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5517cd3640 1 Processor -- start 2024-04-26T00:29:13.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5517cd3640 1 -- start start 2024-04-26T00:29:13.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5517cd3640 10 monclient: init 2024-04-26T00:29:13.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5517cd3640 10 monclient: _reopen_session rank -1 2024-04-26T00:29:13.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5517cd3640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:29:13.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5517cd3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5510135d90 0x7f55101cf090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:13.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5517cd3640 10 monclient(hunting): picked mon.noname-c con 0x7f5510135d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:29:13.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5517cd3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5510136e10 0x7f55101cf5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:13.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5517cd3640 10 monclient(hunting): picked mon.noname-b con 0x7f5510136e10 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:29:13.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5517cd3640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55101371f0 0x7f55101cfb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:13.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5517cd3640 10 monclient(hunting): picked mon.noname-a con 0x7f55101371f0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:29:13.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5517cd3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:13.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5517cd3640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f5510140cf0 con 0x7f55101371f0 2024-04-26T00:29:13.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5517cd3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:13.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5517cd3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5510140b70 con 0x7f5510136e10 2024-04-26T00:29:13.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5517cd3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:13.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5517cd3640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5510140e70 con 0x7f5510135d90 2024-04-26T00:29:13.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5517cd3640 10 monclient(hunting): _renew_subs 2024-04-26T00:29:13.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5517cd3640 10 monclient(hunting): authenticate will time out at 1534.671843s 2024-04-26T00:29:13.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5515a6f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5510135d90 0x7f55101cf090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:13.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5515a6f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5510135d90 0x7f55101cf090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:41282/0 (socket says 172.21.15.37:41282) 2024-04-26T00:29:13.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5515a6f640 1 -- 172.21.15.37:0/3619343755 learned_addr learned my addr 172.21.15.37:0/3619343755 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:29:13.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.362+0000 7f5515a6f640 10 monclient(hunting): get_auth_request con 0x7f5510135d90 auth_method 0 2024-04-26T00:29:13.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f5515a6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:13.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f5515a6f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:13.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f5515a6f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:13.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f5516270640 1 --2- 172.21.15.37:0/3619343755 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55101371f0 0x7f55101cfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:13.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f5516270640 10 monclient(hunting): get_auth_request con 0x7f55101371f0 auth_method 0 2024-04-26T00:29:13.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f5516270640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:13.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f5516270640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:13.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f5516270640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:13.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f5515a6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:13.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f5515a6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:13.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f5515a6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:13.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f5516270640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:13.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f5516270640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:13.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f5516270640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:13.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f551526e640 1 --2- 172.21.15.37:0/3619343755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5510136e10 0x7f55101cf5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:13.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f551526e640 10 monclient(hunting): get_auth_request con 0x7f5510136e10 auth_method 0 2024-04-26T00:29:13.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f551526e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:13.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f551526e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:13.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f551526e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:13.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f5515a6f640 10 monclient(hunting): handle_auth_done global_id 84383 payload 824 2024-04-26T00:29:13.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f5515a6f640 1 -- 172.21.15.37:0/3619343755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5510136e10 msgr2=0x7f55101cf5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:13.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f5515a6f640 1 --2- 172.21.15.37:0/3619343755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5510136e10 0x7f55101cf5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:13.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f5515a6f640 1 -- 172.21.15.37:0/3619343755 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55101371f0 msgr2=0x7f55101cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:13.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f5515a6f640 1 --2- 172.21.15.37:0/3619343755 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55101371f0 0x7f55101cfb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:13.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f5515a6f640 10 monclient: _finish_hunting 0 2024-04-26T00:29:13.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f5515a6f640 1 monclient: found mon.noname-c 2024-04-26T00:29:13.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f5515a6f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:29:13.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f5515a6f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:29:13.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f5515a6f640 1 -- 172.21.15.37:0/3619343755 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f551013c690 con 0x7f5510135d90 2024-04-26T00:29:13.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f5515a6f640 1 -- 172.21.15.37:0/3619343755 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5510135d90 msgr2=0x7f55101cf090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:13.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f5515a6f640 1 --2- 172.21.15.37:0/3619343755 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5510135d90 0x7f55101cf090 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f550001cc80 tx=0x7f550001ac20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:13.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f550e7fc640 1 -- 172.21.15.37:0/3619343755 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55000b1040 con 0x7f5510135d90 2024-04-26T00:29:13.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f550e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:13.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f550e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:29:13.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f550e7fc640 10 monclient: dump: 2024-04-26T00:29:13.368 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:13.368 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:13.368 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:13.368 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:13.368 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:13.368 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:13.368 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:13.368 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:13.368 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:13.369 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:13.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f550e7fc640 10 monclient: _finish_auth 0 2024-04-26T00:29:13.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f550e7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:29:13.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f550e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:29:13.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f550e7fc640 1 -- 172.21.15.37:0/3619343755 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55000ac420 con 0x7f5510135d90 2024-04-26T00:29:13.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f550e7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:29:13.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f550e7fc640 1 -- 172.21.15.37:0/3619343755 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55000b7690 con 0x7f5510135d90 2024-04-26T00:29:13.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f550e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:13.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f550e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:29:13.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f550e7fc640 10 monclient: dump: 2024-04-26T00:29:13.370 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:13.370 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:13.370 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:13.370 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:13.370 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:13.370 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:13.370 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:13.370 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:13.370 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:13.370 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:13.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f5517cd3640 5 monclient: authenticate success, global_id 84383 2024-04-26T00:29:13.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f5517cd3640 10 monclient: _renew_subs 2024-04-26T00:29:13.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f550f7fe640 4 set_mon_vals no callback set 2024-04-26T00:29:13.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f5517cd3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:29:13.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f5517cd3640 1 -- 172.21.15.37:0/3619343755 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f551013c970 con 0x7f5510135d90 2024-04-26T00:29:13.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f5517cd3640 10 monclient: _renew_subs 2024-04-26T00:29:13.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f5517cd3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:29:13.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f5517cd3640 1 -- 172.21.15.37:0/3619343755 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f55100a3910 con 0x7f5510135d90 2024-04-26T00:29:13.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f5517cd3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:29:13.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f5517cd3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:29:13.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f5517cd3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:29:13.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f550e7fc640 1 -- 172.21.15.37:0/3619343755 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f55000be030 con 0x7f5510135d90 2024-04-26T00:29:13.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f5517cd3640 1 -- 172.21.15.37:0/3619343755 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5510066f40 con 0x7f5510135d90 2024-04-26T00:29:13.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.366+0000 7f550e7fc640 1 --2- 172.21.15.37:0/3619343755 >> v2:172.21.15.45:6812/2379918623 conn(0x7f54fc035e80 0x7f54fc038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:13.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.370+0000 7f550e7fc640 1 -- 172.21.15.37:0/3619343755 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(364..364 src has 177..364) ==== 3915+0+0 (secure 0 0 0) 0x7f55000e2070 con 0x7f5510135d90 2024-04-26T00:29:13.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.370+0000 7f551526e640 1 --2- 172.21.15.37:0/3619343755 >> v2:172.21.15.45:6812/2379918623 conn(0x7f54fc035e80 0x7f54fc038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:13.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.370+0000 7f551526e640 10 monclient: get_auth_request con 0x7f54fc035e80 auth_method 0 2024-04-26T00:29:13.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.370+0000 7f551526e640 1 --2- 172.21.15.37:0/3619343755 >> v2:172.21.15.45:6812/2379918623 conn(0x7f54fc035e80 0x7f54fc038340 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5504003fd0 tx=0x7f5504003ce0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:13.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.374+0000 7f550e7fc640 1 -- 172.21.15.37:0/3619343755 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f55000160f0 con 0x7f5510135d90 2024-04-26T00:29:13.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.374+0000 7f550e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:29:13.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.374+0000 7f550e7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:29:13.374 INFO:tasks.rados.rados.0.smithi045.stdout:1580: done (1 left) 2024-04-26T00:29:13.374 INFO:tasks.rados.rados.0.smithi045.stdout:1581: write (excl) oid 22 current snap is 182 2024-04-26T00:29:13.374 INFO:tasks.rados.rados.0.smithi045.stdout:1579: finishing write tid 1 to smithi04526579-39 2024-04-26T00:29:13.374 INFO:tasks.rados.rados.0.smithi045.stdout:1581: seq_num 587 ranges {665709=730344,2133283=426203,3203532=509848} 2024-04-26T00:29:13.393 INFO:tasks.rados.rados.0.smithi045.stdout:1581: writing smithi04526579-22 from 665709 to 1396053 tid 1 2024-04-26T00:29:13.405 INFO:tasks.rados.rados.0.smithi045.stdout:1581: writing smithi04526579-22 from 2133283 to 2559486 tid 2 2024-04-26T00:29:13.415 INFO:tasks.rados.rados.0.smithi045.stdout:1581: writing smithi04526579-22 from 3203532 to 3713380 tid 3 2024-04-26T00:29:13.415 INFO:tasks.rados.rados.0.smithi045.stdout:1582: delete oid 24 current snap is 182 2024-04-26T00:29:13.415 INFO:tasks.rados.rados.0.smithi045.stdout:1579: finishing write tid 2 to smithi04526579-39 2024-04-26T00:29:13.415 INFO:tasks.rados.rados.0.smithi045.stdout:1579: finishing write tid 3 to smithi04526579-39 2024-04-26T00:29:13.415 INFO:tasks.rados.rados.0.smithi045.stdout:1579: finishing write tid 4 to smithi04526579-39 2024-04-26T00:29:13.415 INFO:tasks.rados.rados.0.smithi045.stdout:1579: finishing write tid 5 to smithi04526579-39 2024-04-26T00:29:13.415 INFO:tasks.rados.rados.0.smithi045.stdout:1579: oid 39 updating version 0 to 1041 2024-04-26T00:29:13.415 INFO:tasks.rados.rados.0.smithi045.stdout:1579: oid 39 updating version 1041 to 1042 2024-04-26T00:29:13.415 INFO:tasks.rados.rados.0.smithi045.stdout:1579: oid 39 version 1042 is already newer than 1040 2024-04-26T00:29:13.415 INFO:tasks.rados.rados.0.smithi045.stdout:1579: oid 39 version 1042 is already newer than 1039 2024-04-26T00:29:13.415 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 1042 (ObjNum 586 snap 182 seq_num 586) dirty exists 2024-04-26T00:29:13.415 INFO:tasks.rados.rados.0.smithi045.stdout:1579: left oid 39 (ObjNum 586 snap 182 seq_num 586) 2024-04-26T00:29:13.415 INFO:tasks.rados.rados.0.smithi045.stdout:1581: finishing write tid 1 to smithi04526579-22 2024-04-26T00:29:13.416 INFO:tasks.rados.rados.0.smithi045.stdout:1581: finishing write tid 2 to smithi04526579-22 2024-04-26T00:29:13.417 INFO:tasks.rados.rados.0.smithi045.stdout:1579: done (2 left) 2024-04-26T00:29:13.418 INFO:tasks.rados.rados.0.smithi045.stdout:1582: done (1 left) 2024-04-26T00:29:13.418 INFO:tasks.rados.rados.0.smithi045.stdout:1583: rollback oid 14 current snap is 182 2024-04-26T00:29:13.418 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 14 to 175 2024-04-26T00:29:13.418 INFO:tasks.rados.rados.0.smithi045.stdout:1584: snap_create 2024-04-26T00:29:13.419 INFO:tasks.rados.rados.0.smithi045.stdout:1581: finishing write tid 3 to smithi04526579-22 2024-04-26T00:29:13.419 INFO:tasks.rados.rados.0.smithi045.stdout:1581: finishing write tid 4 to smithi04526579-22 2024-04-26T00:29:13.419 INFO:tasks.rados.rados.0.smithi045.stdout:1581: finishing write tid 5 to smithi04526579-22 2024-04-26T00:29:13.419 INFO:tasks.rados.rados.0.smithi045.stdout:1581: oid 22 updating version 0 to 768 2024-04-26T00:29:13.419 INFO:tasks.rados.rados.0.smithi045.stdout:1581: oid 22 version 768 is already newer than 766 2024-04-26T00:29:13.419 INFO:tasks.rados.rados.0.smithi045.stdout:1581: oid 22 version 768 is already newer than 767 2024-04-26T00:29:13.420 INFO:tasks.rados.rados.0.smithi045.stdout:1581: oid 22 updating version 768 to 769 2024-04-26T00:29:13.420 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 769 (ObjNum 587 snap 182 seq_num 587) dirty exists 2024-04-26T00:29:13.420 INFO:tasks.rados.rados.0.smithi045.stdout:1581: left oid 22 (ObjNum 587 snap 182 seq_num 587) 2024-04-26T00:29:13.422 INFO:tasks.rados.rados.0.smithi045.stdout:1583: finishing rollback tid 0 to smithi04526579-14 2024-04-26T00:29:13.422 INFO:tasks.rados.rados.0.smithi045.stdout:1583: finishing rollback tid 1 to smithi04526579-14 2024-04-26T00:29:13.422 INFO:tasks.rados.rados.0.smithi045.stdout:1583: finishing rollback tid 2 to smithi04526579-14 2024-04-26T00:29:13.423 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 624 (ObjNum 433 snap 136 seq_num 433) dirty exists 2024-04-26T00:29:13.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.614+0000 7f5517cd3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:29:13.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.614+0000 7f5517cd3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:29:13.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.614+0000 7f5517cd3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:29:13.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.614+0000 7f5517cd3640 1 -- 172.21.15.37:0/3619343755 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f551013a680 con 0x7f5510135d90 2024-04-26T00:29:13.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.614+0000 7f550e7fc640 1 -- 172.21.15.37:0/3619343755 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f55000c0410 con 0x7f5510135d90 2024-04-26T00:29:13.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.614+0000 7f550e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:29:13.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.614+0000 7f550e7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:29:13.604 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":68,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:29:13.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.618+0000 7f5517cd3640 1 -- 172.21.15.37:0/3619343755 >> v2:172.21.15.45:6812/2379918623 conn(0x7f54fc035e80 msgr2=0x7f54fc038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:13.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.618+0000 7f5517cd3640 1 --2- 172.21.15.37:0/3619343755 >> v2:172.21.15.45:6812/2379918623 conn(0x7f54fc035e80 0x7f54fc038340 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5504003fd0 tx=0x7f5504003ce0 comp rx=0 tx=0).stop 2024-04-26T00:29:13.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.618+0000 7f5517cd3640 10 monclient: shutdown 2024-04-26T00:29:13.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.618+0000 7f5517cd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:29:13.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.618+0000 7f5517cd3640 1 -- 172.21.15.37:0/3619343755 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5510135d90 msgr2=0x7f55101cf090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:13.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.618+0000 7f5517cd3640 1 --2- 172.21.15.37:0/3619343755 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5510135d90 0x7f55101cf090 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f550001cc80 tx=0x7f550001ac20 comp rx=0 tx=0).stop 2024-04-26T00:29:13.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.618+0000 7f5517cd3640 1 -- 172.21.15.37:0/3619343755 shutdown_connections 2024-04-26T00:29:13.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.618+0000 7f5517cd3640 1 --2- 172.21.15.37:0/3619343755 >> v2:172.21.15.45:6812/2379918623 conn(0x7f54fc035e80 0x7f54fc038340 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:13.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.618+0000 7f5517cd3640 1 --2- 172.21.15.37:0/3619343755 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55101371f0 0x7f55101cfb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:13.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.618+0000 7f5517cd3640 1 --2- 172.21.15.37:0/3619343755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5510136e10 0x7f55101cf5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:13.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.618+0000 7f5517cd3640 1 --2- 172.21.15.37:0/3619343755 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5510135d90 0x7f55101cf090 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:13.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.618+0000 7f5517cd3640 1 -- 172.21.15.37:0/3619343755 >> 172.21.15.37:0/3619343755 conn(0x7f55100a64e0 msgr2=0x7f55100a73e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:29:13.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.618+0000 7f5517cd3640 1 -- 172.21.15.37:0/3619343755 shutdown_connections 2024-04-26T00:29:13.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:13.618+0000 7f5517cd3640 1 -- 172.21.15.37:0/3619343755 wait complete. 2024-04-26T00:29:14.280 INFO:tasks.rados.rados.0.smithi045.stdout:1581: done (2 left) 2024-04-26T00:29:14.280 INFO:tasks.rados.rados.0.smithi045.stdout:1583: done (1 left) 2024-04-26T00:29:14.280 INFO:tasks.rados.rados.0.smithi045.stdout:1584: done (0 left) 2024-04-26T00:29:14.280 INFO:tasks.rados.rados.0.smithi045.stdout:1585: write (excl) oid 14 current snap is 183 2024-04-26T00:29:14.280 INFO:tasks.rados.rados.0.smithi045.stdout:1585: seq_num 589 ranges {699323=480842,1850247=429872,2858378=532392} 2024-04-26T00:29:14.293 INFO:tasks.rados.rados.0.smithi045.stdout:1585: writing smithi04526579-14 from 699323 to 1180165 tid 1 2024-04-26T00:29:14.306 INFO:tasks.rados.rados.0.smithi045.stdout:1585: writing smithi04526579-14 from 1850247 to 2280119 tid 2 2024-04-26T00:29:14.319 INFO:tasks.rados.rados.0.smithi045.stdout:1585: writing smithi04526579-14 from 2858378 to 3390770 tid 3 2024-04-26T00:29:14.319 INFO:tasks.rados.rados.0.smithi045.stdout:1586: delete oid 26 current snap is 183 2024-04-26T00:29:14.319 INFO:tasks.rados.rados.0.smithi045.stdout:1585: finishing write tid 1 to smithi04526579-14 2024-04-26T00:29:14.319 INFO:tasks.rados.rados.0.smithi045.stdout:1585: finishing write tid 2 to smithi04526579-14 2024-04-26T00:29:14.326 INFO:tasks.rados.rados.0.smithi045.stdout:1585: finishing write tid 3 to smithi04526579-14 2024-04-26T00:29:14.326 INFO:tasks.rados.rados.0.smithi045.stdout:1585: finishing write tid 4 to smithi04526579-14 2024-04-26T00:29:14.327 INFO:tasks.rados.rados.0.smithi045.stdout:1585: finishing write tid 5 to smithi04526579-14 2024-04-26T00:29:14.328 INFO:tasks.rados.rados.0.smithi045.stdout:1585: oid 14 updating version 0 to 628 2024-04-26T00:29:14.328 INFO:tasks.rados.rados.0.smithi045.stdout:1585: oid 14 version 628 is already newer than 627 2024-04-26T00:29:14.328 INFO:tasks.rados.rados.0.smithi045.stdout:1585: oid 14 version 628 is already newer than 626 2024-04-26T00:29:14.328 INFO:tasks.rados.rados.0.smithi045.stdout:1585: oid 14 version 628 is already newer than 625 2024-04-26T00:29:14.328 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 628 (ObjNum 589 snap 183 seq_num 589) dirty exists 2024-04-26T00:29:14.328 INFO:tasks.rados.rados.0.smithi045.stdout:1585: left oid 14 (ObjNum 589 snap 183 seq_num 589) 2024-04-26T00:29:14.331 INFO:tasks.rados.rados.0.smithi045.stdout:1585: done (1 left) 2024-04-26T00:29:14.331 INFO:tasks.rados.rados.0.smithi045.stdout:1586: done (0 left) 2024-04-26T00:29:14.331 INFO:tasks.rados.rados.0.smithi045.stdout:1587: rollback oid 37 current snap is 183 2024-04-26T00:29:14.331 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 37 to 171 2024-04-26T00:29:14.331 INFO:tasks.rados.rados.0.smithi045.stdout:1588: write (excl) oid 32 current snap is 183 2024-04-26T00:29:14.331 INFO:tasks.rados.rados.0.smithi045.stdout:1588: seq_num 591 ranges {732937=631340,1850858=433541,2796871=671289,3888195=44742} 2024-04-26T00:29:14.343 INFO:tasks.rados.rados.0.smithi045.stdout:1588: writing smithi04526579-32 from 732937 to 1364277 tid 1 2024-04-26T00:29:14.350 INFO:tasks.rados.rados.0.smithi045.stdout:1588: writing smithi04526579-32 from 1850858 to 2284399 tid 2 2024-04-26T00:29:14.360 INFO:tasks.rados.rados.0.smithi045.stdout:1588: writing smithi04526579-32 from 2796871 to 3468160 tid 3 2024-04-26T00:29:14.365 INFO:tasks.rados.rados.0.smithi045.stdout:1588: writing smithi04526579-32 from 3888195 to 3932937 tid 4 2024-04-26T00:29:14.365 INFO:tasks.rados.rados.0.smithi045.stdout:1589: read oid 36 snap -1 2024-04-26T00:29:14.365 INFO:tasks.rados.rados.0.smithi045.stdout:1589: expect (ObjNum 577 snap 179 seq_num 577) 2024-04-26T00:29:14.365 INFO:tasks.rados.rados.0.smithi045.stdout:1590: read oid 6 snap -1 2024-04-26T00:29:14.365 INFO:tasks.rados.rados.0.smithi045.stdout:1590: expect (ObjNum 574 snap 176 seq_num 574) 2024-04-26T00:29:14.365 INFO:tasks.rados.rados.0.smithi045.stdout:1591: write oid 7 current snap is 183 2024-04-26T00:29:14.365 INFO:tasks.rados.rados.0.smithi045.stdout:1591: seq_num 592 ranges {749744=706589,1909340=577199,3224294=482561} 2024-04-26T00:29:14.379 INFO:tasks.rados.rados.0.smithi045.stdout:1591: writing smithi04526579-7 from 749744 to 1456333 tid 1 2024-04-26T00:29:14.388 INFO:tasks.rados.rados.0.smithi045.stdout:1591: writing smithi04526579-7 from 1909340 to 2486539 tid 2 2024-04-26T00:29:14.399 INFO:tasks.rados.rados.0.smithi045.stdout:1591: writing smithi04526579-7 from 3224294 to 3706855 tid 3 2024-04-26T00:29:14.399 INFO:tasks.rados.rados.0.smithi045.stdout:1592: rollback oid 19 current snap is 183 2024-04-26T00:29:14.399 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 19 to 172 2024-04-26T00:29:14.399 INFO:tasks.rados.rados.0.smithi045.stdout:1587: finishing rollback tid 0 to smithi04526579-37 2024-04-26T00:29:14.399 INFO:tasks.rados.rados.0.smithi045.stdout:1587: finishing rollback tid 1 to smithi04526579-37 2024-04-26T00:29:14.399 INFO:tasks.rados.rados.0.smithi045.stdout:1587: finishing rollback tid 2 to smithi04526579-37 2024-04-26T00:29:14.399 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 1274 (ObjNum 539 snap 167 seq_num 539) dirty exists 2024-04-26T00:29:14.399 INFO:tasks.rados.rados.0.smithi045.stdout:1588: finishing write tid 1 to smithi04526579-32 2024-04-26T00:29:14.399 INFO:tasks.rados.rados.0.smithi045.stdout:1588: finishing write tid 2 to smithi04526579-32 2024-04-26T00:29:14.399 INFO:tasks.rados.rados.0.smithi045.stdout:1587: done (5 left) 2024-04-26T00:29:14.399 INFO:tasks.rados.rados.0.smithi045.stdout:1593: read oid 27 snap -1 2024-04-26T00:29:14.399 INFO:tasks.rados.rados.0.smithi045.stdout:1593: expect (ObjNum 531 snap 165 seq_num 531) 2024-04-26T00:29:14.400 INFO:tasks.rados.rados.0.smithi045.stdout:1594: rollback oid 16 current snap is 183 2024-04-26T00:29:14.400 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 16 to 180 2024-04-26T00:29:14.400 INFO:tasks.rados.rados.0.smithi045.stdout:1588: finishing write tid 3 to smithi04526579-32 2024-04-26T00:29:14.400 INFO:tasks.rados.rados.0.smithi045.stdout:1588: finishing write tid 4 to smithi04526579-32 2024-04-26T00:29:14.400 INFO:tasks.rados.rados.0.smithi045.stdout:1588: finishing write tid 5 to smithi04526579-32 2024-04-26T00:29:14.400 INFO:tasks.rados.rados.0.smithi045.stdout:1588: finishing write tid 6 to smithi04526579-32 2024-04-26T00:29:14.400 INFO:tasks.rados.rados.0.smithi045.stdout:1588: oid 32 updating version 0 to 970 2024-04-26T00:29:14.400 INFO:tasks.rados.rados.0.smithi045.stdout:1588: oid 32 version 970 is already newer than 967 2024-04-26T00:29:14.400 INFO:tasks.rados.rados.0.smithi045.stdout:1588: oid 32 version 970 is already newer than 969 2024-04-26T00:29:14.400 INFO:tasks.rados.rados.0.smithi045.stdout:1588: oid 32 version 970 is already newer than 968 2024-04-26T00:29:14.400 INFO:tasks.rados.rados.0.smithi045.stdout:1588: oid 32 updating version 970 to 971 2024-04-26T00:29:14.400 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 971 (ObjNum 591 snap 183 seq_num 591) dirty exists 2024-04-26T00:29:14.400 INFO:tasks.rados.rados.0.smithi045.stdout:1588: left oid 32 (ObjNum 591 snap 183 seq_num 591) 2024-04-26T00:29:14.400 INFO:tasks.rados.rados.0.smithi045.stdout:1588: done (6 left) 2024-04-26T00:29:14.401 INFO:tasks.rados.rados.0.smithi045.stdout:1595: snap_create 2024-04-26T00:29:14.401 INFO:tasks.rados.rados.0.smithi045.stdout:1589: expect (ObjNum 577 snap 179 seq_num 577) 2024-04-26T00:29:14.572 INFO:tasks.rados.rados.0.smithi045.stdout:1590: expect (ObjNum 574 snap 176 seq_num 574) 2024-04-26T00:29:14.772 INFO:tasks.rados.rados.0.smithi045.stdout:1591: finishing write tid 1 to smithi04526579-7 2024-04-26T00:29:14.772 INFO:tasks.rados.rados.0.smithi045.stdout:1591: finishing write tid 2 to smithi04526579-7 2024-04-26T00:29:14.772 INFO:tasks.rados.rados.0.smithi045.stdout:1594: finishing rollback tid 0 to smithi04526579-16 2024-04-26T00:29:14.772 INFO:tasks.rados.rados.0.smithi045.stdout:1591: finishing write tid 3 to smithi04526579-7 2024-04-26T00:29:14.772 INFO:tasks.rados.rados.0.smithi045.stdout:1591: finishing write tid 4 to smithi04526579-7 2024-04-26T00:29:14.772 INFO:tasks.rados.rados.0.smithi045.stdout:1591: finishing write tid 5 to smithi04526579-7 2024-04-26T00:29:14.772 INFO:tasks.rados.rados.0.smithi045.stdout:1591: oid 7 updating version 0 to 1277 2024-04-26T00:29:14.772 INFO:tasks.rados.rados.0.smithi045.stdout:1591: oid 7 updating version 1277 to 1279 2024-04-26T00:29:14.772 INFO:tasks.rados.rados.0.smithi045.stdout:1591: oid 7 version 1279 is already newer than 1278 2024-04-26T00:29:14.772 INFO:tasks.rados.rados.0.smithi045.stdout:1591: oid 7 version 1279 is already newer than 1276 2024-04-26T00:29:14.772 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 1279 (ObjNum 592 snap 183 seq_num 592) dirty exists 2024-04-26T00:29:14.773 INFO:tasks.rados.rados.0.smithi045.stdout:1591: left oid 7 (ObjNum 592 snap 183 seq_num 592) 2024-04-26T00:29:14.773 INFO:tasks.rados.rados.0.smithi045.stdout:1594: finishing rollback tid 1 to smithi04526579-16 2024-04-26T00:29:14.773 INFO:tasks.rados.rados.0.smithi045.stdout:1594: finishing rollback tid 2 to smithi04526579-16 2024-04-26T00:29:14.773 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 631 (ObjNum 515 snap 157 seq_num 515) dirty exists 2024-04-26T00:29:14.773 INFO:tasks.rados.rados.0.smithi045.stdout:1592: finishing rollback tid 1 to smithi04526579-19 2024-04-26T00:29:14.773 INFO:tasks.rados.rados.0.smithi045.stdout:1592: finishing rollback tid 2 to smithi04526579-19 2024-04-26T00:29:14.773 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 973 (ObjNum 534 snap 165 seq_num 534) dirty exists 2024-04-26T00:29:14.773 INFO:tasks.rados.rados.0.smithi045.stdout:1593: expect (ObjNum 531 snap 165 seq_num 531) 2024-04-26T00:29:15.395 INFO:tasks.rados.rados.0.smithi045.stdout:1589: done (6 left) 2024-04-26T00:29:15.395 INFO:tasks.rados.rados.0.smithi045.stdout:1590: done (5 left) 2024-04-26T00:29:15.395 INFO:tasks.rados.rados.0.smithi045.stdout:1591: done (4 left) 2024-04-26T00:29:15.395 INFO:tasks.rados.rados.0.smithi045.stdout:1592: done (3 left) 2024-04-26T00:29:15.395 INFO:tasks.rados.rados.0.smithi045.stdout:1593: done (2 left) 2024-04-26T00:29:15.395 INFO:tasks.rados.rados.0.smithi045.stdout:1594: done (1 left) 2024-04-26T00:29:15.395 INFO:tasks.rados.rados.0.smithi045.stdout:1595: done (0 left) 2024-04-26T00:29:15.395 INFO:tasks.rados.rados.0.smithi045.stdout:1596: write (excl) oid 1 current snap is 184 2024-04-26T00:29:15.396 INFO:tasks.rados.rados.0.smithi045.stdout:1596: seq_num 593 ranges {766551=498191,1684175=720857,2851717=693833} 2024-04-26T00:29:15.404 INFO:tasks.rados.rados.0.smithi045.stdout:1596: writing smithi04526579-1 from 766551 to 1264742 tid 1 2024-04-26T00:29:15.413 INFO:tasks.rados.rados.0.smithi045.stdout:1596: writing smithi04526579-1 from 1684175 to 2405032 tid 2 2024-04-26T00:29:15.423 INFO:tasks.rados.rados.0.smithi045.stdout:1596: writing smithi04526579-1 from 2851717 to 3545550 tid 3 2024-04-26T00:29:15.423 INFO:tasks.rados.rados.0.smithi045.stdout:1597: rollback oid 35 current snap is 184 2024-04-26T00:29:15.423 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 35 to 170 2024-04-26T00:29:15.423 INFO:tasks.rados.rados.0.smithi045.stdout:1596: finishing write tid 1 to smithi04526579-1 2024-04-26T00:29:15.423 INFO:tasks.rados.rados.0.smithi045.stdout:1598: delete oid 50 current snap is 184 2024-04-26T00:29:15.424 INFO:tasks.rados.rados.0.smithi045.stdout:1596: finishing write tid 2 to smithi04526579-1 2024-04-26T00:29:15.432 INFO:tasks.rados.rados.0.smithi045.stdout:1598: done (2 left) 2024-04-26T00:29:15.432 INFO:tasks.rados.rados.0.smithi045.stdout:1599: read oid 37 snap -1 2024-04-26T00:29:15.432 INFO:tasks.rados.rados.0.smithi045.stdout:1599: expect (ObjNum 539 snap 167 seq_num 539) 2024-04-26T00:29:15.432 INFO:tasks.rados.rados.0.smithi045.stdout:1600: copy_from oid 14 from oid 19 current snap is 184 2024-04-26T00:29:15.432 INFO:tasks.rados.rados.0.smithi045.stdout:1601: write (excl) oid 30 current snap is 184 2024-04-26T00:29:15.432 INFO:tasks.rados.rados.0.smithi045.stdout:1601: seq_num 595 ranges {400165=648689,1684786=315379} 2024-04-26T00:29:15.441 INFO:tasks.rados.rados.0.smithi045.stdout:1601: writing smithi04526579-30 from 400165 to 1048854 tid 1 2024-04-26T00:29:15.450 INFO:tasks.rados.rados.0.smithi045.stdout:1601: writing smithi04526579-30 from 1684786 to 2000165 tid 2 2024-04-26T00:29:15.450 INFO:tasks.rados.rados.0.smithi045.stdout:1602: delete oid 48 current snap is 184 2024-04-26T00:29:15.450 INFO:tasks.rados.rados.0.smithi045.stdout:1596: finishing write tid 3 to smithi04526579-1 2024-04-26T00:29:15.450 INFO:tasks.rados.rados.0.smithi045.stdout:1596: finishing write tid 4 to smithi04526579-1 2024-04-26T00:29:15.450 INFO:tasks.rados.rados.0.smithi045.stdout:1596: finishing write tid 5 to smithi04526579-1 2024-04-26T00:29:15.450 INFO:tasks.rados.rados.0.smithi045.stdout:1596: oid 1 updating version 0 to 1063 2024-04-26T00:29:15.450 INFO:tasks.rados.rados.0.smithi045.stdout:1596: oid 1 version 1063 is already newer than 1061 2024-04-26T00:29:15.450 INFO:tasks.rados.rados.0.smithi045.stdout:1596: oid 1 version 1063 is already newer than 1062 2024-04-26T00:29:15.450 INFO:tasks.rados.rados.0.smithi045.stdout:1596: oid 1 updating version 1063 to 1064 2024-04-26T00:29:15.450 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 1064 (ObjNum 593 snap 184 seq_num 593) dirty exists 2024-04-26T00:29:15.450 INFO:tasks.rados.rados.0.smithi045.stdout:1596: left oid 1 (ObjNum 593 snap 184 seq_num 593) 2024-04-26T00:29:15.450 INFO:tasks.rados.rados.0.smithi045.stdout:1597: finishing rollback tid 0 to smithi04526579-35 2024-04-26T00:29:15.451 INFO:tasks.rados.rados.0.smithi045.stdout:1597: finishing rollback tid 1 to smithi04526579-35 2024-04-26T00:29:15.451 INFO:tasks.rados.rados.0.smithi045.stdout:1597: finishing rollback tid 2 to smithi04526579-35 2024-04-26T00:29:15.451 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 977 (ObjNum 553 snap 168 seq_num 553) dirty exists 2024-04-26T00:29:15.455 INFO:tasks.rados.rados.0.smithi045.stdout:1599: expect (ObjNum 539 snap 167 seq_num 539) 2024-04-26T00:29:15.668 INFO:tasks.rados.rados.0.smithi045.stdout:1601: finishing write tid 1 to smithi04526579-30 2024-04-26T00:29:15.668 INFO:tasks.rados.rados.0.smithi045.stdout:1601: finishing write tid 2 to smithi04526579-30 2024-04-26T00:29:15.668 INFO:tasks.rados.rados.0.smithi045.stdout:1601: finishing write tid 3 to smithi04526579-30 2024-04-26T00:29:15.668 INFO:tasks.rados.rados.0.smithi045.stdout:1601: finishing write tid 4 to smithi04526579-30 2024-04-26T00:29:15.668 INFO:tasks.rados.rados.0.smithi045.stdout:1601: oid 30 updating version 0 to 1044 2024-04-26T00:29:15.668 INFO:tasks.rados.rados.0.smithi045.stdout:1601: oid 30 updating version 1044 to 1046 2024-04-26T00:29:15.668 INFO:tasks.rados.rados.0.smithi045.stdout:1601: oid 30 version 1046 is already newer than 1045 2024-04-26T00:29:15.668 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 1046 (ObjNum 595 snap 184 seq_num 595) dirty exists 2024-04-26T00:29:15.669 INFO:tasks.rados.rados.0.smithi045.stdout:1601: left oid 30 (ObjNum 595 snap 184 seq_num 595) 2024-04-26T00:29:15.669 INFO:tasks.rados.rados.0.smithi045.stdout:1600: finishing copy_from racing read to smithi04526579-14 2024-04-26T00:29:15.669 INFO:tasks.rados.rados.0.smithi045.stdout:1600: finishing copy_from to smithi04526579-14 2024-04-26T00:29:15.669 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 973 (ObjNum 534 snap 165 seq_num 534) dirty exists 2024-04-26T00:29:15.669 INFO:tasks.rados.rados.0.smithi045.stdout:1596: done (5 left) 2024-04-26T00:29:15.669 INFO:tasks.rados.rados.0.smithi045.stdout:1597: done (4 left) 2024-04-26T00:29:15.669 INFO:tasks.rados.rados.0.smithi045.stdout:1599: done (3 left) 2024-04-26T00:29:15.669 INFO:tasks.rados.rados.0.smithi045.stdout:1600: done (2 left) 2024-04-26T00:29:15.669 INFO:tasks.rados.rados.0.smithi045.stdout:1601: done (1 left) 2024-04-26T00:29:15.669 INFO:tasks.rados.rados.0.smithi045.stdout:1602: done (0 left) 2024-04-26T00:29:15.669 INFO:tasks.rados.rados.0.smithi045.stdout:1603: read oid 15 snap -1 2024-04-26T00:29:15.669 INFO:tasks.rados.rados.0.smithi045.stdout:1603: expect deleted 2024-04-26T00:29:15.669 INFO:tasks.rados.rados.0.smithi045.stdout:1604: write oid 38 current snap is 184 2024-04-26T00:29:15.669 INFO:tasks.rados.rados.0.smithi045.stdout:1604: seq_num 597 ranges {433779=799187,1801750=232029} 2024-04-26T00:29:15.679 INFO:tasks.rados.rados.0.smithi045.stdout:1604: writing smithi04526579-38 from 433779 to 1232966 tid 1 2024-04-26T00:29:15.687 INFO:tasks.rados.rados.0.smithi045.stdout:1604: writing smithi04526579-38 from 1801750 to 2033779 tid 2 2024-04-26T00:29:15.687 INFO:tasks.rados.rados.0.smithi045.stdout:1605: delete oid 49 current snap is 184 2024-04-26T00:29:15.692 INFO:tasks.rados.rados.0.smithi045.stdout:1603: done (2 left) 2024-04-26T00:29:15.693 INFO:tasks.rados.rados.0.smithi045.stdout:1605: done (1 left) 2024-04-26T00:29:15.693 INFO:tasks.rados.rados.0.smithi045.stdout:1606: snap_remove snap 177 2024-04-26T00:29:16.406 INFO:tasks.rados.rados.0.smithi045.stdout:1606: done (1 left) 2024-04-26T00:29:16.406 INFO:tasks.rados.rados.0.smithi045.stdout:1607: write oid 42 current snap is 184 2024-04-26T00:29:16.406 INFO:tasks.rados.rados.0.smithi045.stdout:1607: seq_num 599 ranges {467393=549685,1802361=265032} 2024-04-26T00:29:16.419 INFO:tasks.rados.rados.0.smithi045.stdout:1607: writing smithi04526579-42 from 467393 to 1017078 tid 1 2024-04-26T00:29:16.433 INFO:tasks.rados.rados.0.smithi045.stdout:1607: writing smithi04526579-42 from 1802361 to 2067393 tid 2 2024-04-26T00:29:16.433 INFO:tasks.rados.rados.0.smithi045.stdout:1608: snap_create 2024-04-26T00:29:16.433 INFO:tasks.rados.rados.0.smithi045.stdout:1604: finishing write tid 1 to smithi04526579-38 2024-04-26T00:29:16.433 INFO:tasks.rados.rados.0.smithi045.stdout:1604: finishing write tid 2 to smithi04526579-38 2024-04-26T00:29:16.433 INFO:tasks.rados.rados.0.smithi045.stdout:1604: finishing write tid 3 to smithi04526579-38 2024-04-26T00:29:16.433 INFO:tasks.rados.rados.0.smithi045.stdout:1604: finishing write tid 4 to smithi04526579-38 2024-04-26T00:29:16.434 INFO:tasks.rados.rados.0.smithi045.stdout:1604: oid 38 updating version 0 to 1044 2024-04-26T00:29:16.434 INFO:tasks.rados.rados.0.smithi045.stdout:1604: oid 38 updating version 1044 to 1045 2024-04-26T00:29:16.434 INFO:tasks.rados.rados.0.smithi045.stdout:1604: oid 38 version 1045 is already newer than 1043 2024-04-26T00:29:16.434 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 1045 (ObjNum 597 snap 184 seq_num 597) dirty exists 2024-04-26T00:29:16.434 INFO:tasks.rados.rados.0.smithi045.stdout:1604: left oid 38 (ObjNum 597 snap 184 seq_num 597) 2024-04-26T00:29:16.441 INFO:tasks.rados.rados.0.smithi045.stdout:1607: finishing write tid 1 to smithi04526579-42 2024-04-26T00:29:16.442 INFO:tasks.rados.rados.0.smithi045.stdout:1607: finishing write tid 2 to smithi04526579-42 2024-04-26T00:29:16.443 INFO:tasks.rados.rados.0.smithi045.stdout:1607: finishing write tid 3 to smithi04526579-42 2024-04-26T00:29:16.444 INFO:tasks.rados.rados.0.smithi045.stdout:1607: finishing write tid 4 to smithi04526579-42 2024-04-26T00:29:16.444 INFO:tasks.rados.rados.0.smithi045.stdout:1607: oid 42 updating version 0 to 986 2024-04-26T00:29:16.444 INFO:tasks.rados.rados.0.smithi045.stdout:1607: oid 42 version 986 is already newer than 985 2024-04-26T00:29:16.444 INFO:tasks.rados.rados.0.smithi045.stdout:1607: oid 42 updating version 986 to 987 2024-04-26T00:29:16.444 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 987 (ObjNum 599 snap 184 seq_num 599) dirty exists 2024-04-26T00:29:16.444 INFO:tasks.rados.rados.0.smithi045.stdout:1607: left oid 42 (ObjNum 599 snap 184 seq_num 599) 2024-04-26T00:29:16.624 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:29:16.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.766+0000 7fa183a29640 10 monclient: get_monmap_and_config 2024-04-26T00:29:16.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.766+0000 7fa183a29640 10 monclient: build_initial_monmap 2024-04-26T00:29:16.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.766+0000 7fa183a29640 10 monclient: monmap: 2024-04-26T00:29:16.756 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:29:16.756 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:16.756 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:29:16.769983+0000 2024-04-26T00:29:16.756 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:29:16.769983+0000 2024-04-26T00:29:16.756 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:29:16.757 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:16.757 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:29:16.757 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:29:16.757 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:29:16.757 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:29:16.757 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:29:16.757 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:16.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.766+0000 7fa183a29640 1 Processor -- start 2024-04-26T00:29:16.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.766+0000 7fa183a29640 1 -- start start 2024-04-26T00:29:16.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.766+0000 7fa183a29640 10 monclient: init 2024-04-26T00:29:16.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.770+0000 7fa183a29640 10 monclient: _reopen_session rank -1 2024-04-26T00:29:16.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.770+0000 7fa183a29640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-04-26T00:29:16.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.770+0000 7fa183a29640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa17c133b20 addr v1:172.21.15.37:6789/0 2024-04-26T00:29:16.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.770+0000 7fa183a29640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa17c133f00 addr v1:172.21.15.45:6789/0 2024-04-26T00:29:16.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.770+0000 7fa183a29640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa17c13a060 0x7fa17c13c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:16.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.770+0000 7fa183a29640 10 monclient(hunting): picked mon.noname-a con 0x7fa17c13a060 addr v2:172.21.15.37:3300/0 2024-04-26T00:29:16.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.770+0000 7fa183a29640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa17c09fe20 con 0x7fa17c133b20 2024-04-26T00:29:16.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.770+0000 7fa183a29640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa17c072d00 con 0x7fa17c133f00 2024-04-26T00:29:16.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.770+0000 7fa183a29640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:16.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.770+0000 7fa183a29640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fa17c09e620 con 0x7fa17c13a060 2024-04-26T00:29:16.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.770+0000 7fa183a29640 10 monclient(hunting): _renew_subs 2024-04-26T00:29:16.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.770+0000 7fa183a29640 10 monclient(hunting): authenticate will time out at 1538.076409s 2024-04-26T00:29:16.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.770+0000 7fa1817c5640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fa17c133b20 0x7fa17c09e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:46778/0 (socket says 172.21.15.37:46778) 2024-04-26T00:29:16.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.770+0000 7fa1817c5640 1 -- 172.21.15.37:0/1502730577 learned_addr learned my addr 172.21.15.37:0/1502730577 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:29:16.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.770+0000 7fa181fc6640 1 --2- 172.21.15.37:0/1502730577 >> v2:172.21.15.37:3300/0 conn(0x7fa17c13a060 0x7fa17c13c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:16.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.770+0000 7fa181fc6640 10 monclient(hunting): get_auth_request con 0x7fa17c13a060 auth_method 0 2024-04-26T00:29:16.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.770+0000 7fa181fc6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:16.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.770+0000 7fa181fc6640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:16.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.770+0000 7fa181fc6640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:16.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.770+0000 7fa181fc6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:16.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.770+0000 7fa181fc6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:16.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.770+0000 7fa181fc6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:16.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.770+0000 7fa181fc6640 10 monclient(hunting): handle_auth_done global_id 94320 payload 274 2024-04-26T00:29:16.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.770+0000 7fa181fc6640 1 -- 172.21.15.37:0/1502730577 >> v1:172.21.15.45:6789/0 conn(0x7fa17c133f00 legacy=0x7fa17c134350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:29:16.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.770+0000 7fa181fc6640 1 -- 172.21.15.37:0/1502730577 >> v1:172.21.15.37:6789/0 conn(0x7fa17c133b20 legacy=0x7fa17c09e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:29:16.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.770+0000 7fa181fc6640 10 monclient: _finish_hunting 0 2024-04-26T00:29:16.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.770+0000 7fa181fc6640 1 monclient: found mon.noname-a 2024-04-26T00:29:16.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.770+0000 7fa181fc6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:29:16.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.770+0000 7fa181fc6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:29:16.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.770+0000 7fa181fc6640 1 -- 172.21.15.37:0/1502730577 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa17c09ba60 con 0x7fa17c13a060 2024-04-26T00:29:16.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa181fc6640 1 -- 172.21.15.37:0/1502730577 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa17c13a060 msgr2=0x7fa17c13c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:16.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa181fc6640 1 --2- 172.21.15.37:0/1502730577 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa17c13a060 0x7fa17c13c450 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa174009d90 tx=0x7fa17401c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49266cf7cedf77ac server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:16.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa173fff640 1 -- 172.21.15.37:0/1502730577 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1740a86f0 con 0x7fa17c13a060 2024-04-26T00:29:16.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa173fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:16.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa173fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:29:16.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa173fff640 10 monclient: dump: 2024-04-26T00:29:16.764 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:16.764 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:16.764 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:16.764 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:16.764 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:16.764 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:16.765 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:16.765 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:16.765 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:16.765 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:16.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa173fff640 10 monclient: _finish_auth 0 2024-04-26T00:29:16.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa173fff640 10 monclient: _check_auth_tickets 2024-04-26T00:29:16.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa173fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:29:16.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa173fff640 1 -- 172.21.15.37:0/1502730577 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1740a8890 con 0x7fa17c13a060 2024-04-26T00:29:16.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa173fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:29:16.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa173fff640 1 -- 172.21.15.37:0/1502730577 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1740a8c60 con 0x7fa17c13a060 2024-04-26T00:29:16.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa183a29640 5 monclient: authenticate success, global_id 94320 2024-04-26T00:29:16.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa183a29640 10 monclient: get_monmap_and_config success 2024-04-26T00:29:16.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa183a29640 4 set_mon_vals no callback set 2024-04-26T00:29:16.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa173fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:16.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa173fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:29:16.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa173fff640 10 monclient: dump: 2024-04-26T00:29:16.766 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:16.766 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:16.766 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:16.766 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:16.766 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:16.766 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:16.766 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:16.766 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:16.766 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:16.766 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:16.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa183a29640 10 monclient: shutdown 2024-04-26T00:29:16.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa183a29640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:29:16.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa183a29640 1 -- 172.21.15.37:0/1502730577 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa17c13a060 msgr2=0x7fa17c13c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:16.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa183a29640 1 --2- 172.21.15.37:0/1502730577 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa17c13a060 0x7fa17c13c450 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa174009d90 tx=0x7fa17401c5d0 comp rx=0 tx=0).stop 2024-04-26T00:29:16.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa183a29640 1 -- 172.21.15.37:0/1502730577 shutdown_connections 2024-04-26T00:29:16.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa183a29640 1 --2- 172.21.15.37:0/1502730577 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa17c13a060 0x7fa17c13c450 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:16.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa183a29640 1 -- 172.21.15.37:0/1502730577 >> 172.21.15.37:0/1502730577 conn(0x7fa17c0a1ad0 msgr2=0x7fa17c0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:29:16.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa183a29640 1 -- 172.21.15.37:0/1502730577 shutdown_connections 2024-04-26T00:29:16.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa183a29640 1 -- 172.21.15.37:0/1502730577 wait complete. 2024-04-26T00:29:16.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa183a29640 10 monclient: build_initial_monmap 2024-04-26T00:29:16.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa183a29640 10 monclient: monmap: 2024-04-26T00:29:16.771 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:29:16.771 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:29:16.772 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:29:16.772 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:29:16.772 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:29:16.772 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:29:16.772 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:29:16.772 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:29:16.772 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:29:16.772 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:16.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa183a29640 1 Processor -- start 2024-04-26T00:29:16.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa183a29640 1 -- start start 2024-04-26T00:29:16.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa183a29640 10 monclient: init 2024-04-26T00:29:16.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa183a29640 10 monclient: _reopen_session rank -1 2024-04-26T00:29:16.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa183a29640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:29:16.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa183a29640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa17c133b20 0x7fa17c070ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:16.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa183a29640 10 monclient(hunting): picked mon.noname-c con 0x7fa17c133b20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:29:16.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa183a29640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa17c133f00 0x7fa17c11b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:16.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa183a29640 10 monclient(hunting): picked mon.noname-b con 0x7fa17c133f00 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:29:16.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa183a29640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa17c13a060 0x7fa17c0712e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:16.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa183a29640 10 monclient(hunting): picked mon.noname-a con 0x7fa17c13a060 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:29:16.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa183a29640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:16.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa183a29640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa17c13f9b0 con 0x7fa17c13a060 2024-04-26T00:29:16.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa183a29640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:16.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa183a29640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa17c13f830 con 0x7fa17c133f00 2024-04-26T00:29:16.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa183a29640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:16.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa183a29640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa17c13fb30 con 0x7fa17c133b20 2024-04-26T00:29:16.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa183a29640 10 monclient(hunting): _renew_subs 2024-04-26T00:29:16.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.774+0000 7fa183a29640 10 monclient(hunting): authenticate will time out at 1538.083948s 2024-04-26T00:29:16.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa1817c5640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa17c133b20 0x7fa17c070ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:16.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa181fc6640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa17c13a060 0x7fa17c0712e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:16.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa1817c5640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa17c133b20 0x7fa17c070ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:41302/0 (socket says 172.21.15.37:41302) 2024-04-26T00:29:16.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa181fc6640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa17c13a060 0x7fa17c0712e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33808/0 (socket says 172.21.15.37:33808) 2024-04-26T00:29:16.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa1817c5640 1 -- 172.21.15.37:0/585793289 learned_addr learned my addr 172.21.15.37:0/585793289 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:29:16.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa1817c5640 10 monclient(hunting): get_auth_request con 0x7fa17c133b20 auth_method 0 2024-04-26T00:29:16.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa1817c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:16.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa1817c5640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:16.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa1817c5640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:16.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa181fc6640 10 monclient(hunting): get_auth_request con 0x7fa17c13a060 auth_method 0 2024-04-26T00:29:16.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa181fc6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:16.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa181fc6640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:16.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa181fc6640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:16.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa181fc6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:16.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa181fc6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:16.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa181fc6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:16.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa1817c5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:16.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa1817c5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:16.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa1817c5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:16.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa180fc4640 1 --2- 172.21.15.37:0/585793289 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa17c133f00 0x7fa17c11b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:16.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa180fc4640 10 monclient(hunting): get_auth_request con 0x7fa17c133f00 auth_method 0 2024-04-26T00:29:16.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa180fc4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:16.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa180fc4640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:16.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa180fc4640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:16.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa181fc6640 10 monclient(hunting): handle_auth_done global_id 94323 payload 824 2024-04-26T00:29:16.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa181fc6640 1 -- 172.21.15.37:0/585793289 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa17c133b20 msgr2=0x7fa17c070ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:16.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa181fc6640 1 --2- 172.21.15.37:0/585793289 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa17c133b20 0x7fa17c070ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:16.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa181fc6640 1 -- 172.21.15.37:0/585793289 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa17c133f00 msgr2=0x7fa17c11b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:16.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa181fc6640 1 --2- 172.21.15.37:0/585793289 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa17c133f00 0x7fa17c11b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:16.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa181fc6640 10 monclient: _finish_hunting 0 2024-04-26T00:29:16.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa181fc6640 1 monclient: found mon.noname-a 2024-04-26T00:29:16.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa181fc6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:29:16.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa181fc6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:29:16.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa181fc6640 1 -- 172.21.15.37:0/585793289 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa17c0720a0 con 0x7fa17c13a060 2024-04-26T00:29:16.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa1817c5640 1 --2- 172.21.15.37:0/585793289 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa17c133b20 0x7fa17c070ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:29:16.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa181fc6640 1 -- 172.21.15.37:0/585793289 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa17c13a060 msgr2=0x7fa17c0712e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:16.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa181fc6640 1 --2- 172.21.15.37:0/585793289 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa17c13a060 0x7fa17c0712e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa17c072ed0 tx=0x7fa174019370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:16.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa171ffb640 1 -- 172.21.15.37:0/585793289 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1740b0810 con 0x7fa17c13a060 2024-04-26T00:29:16.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa171ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:16.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa171ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:29:16.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa171ffb640 10 monclient: dump: 2024-04-26T00:29:16.776 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:16.776 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:16.777 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:16.777 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:16.777 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:16.777 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:16.777 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:16.777 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:16.777 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:16.777 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:16.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa171ffb640 10 monclient: _finish_auth 0 2024-04-26T00:29:16.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa171ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:29:16.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa171ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:29:16.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa171ffb640 1 -- 172.21.15.37:0/585793289 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1740bf070 con 0x7fa17c13a060 2024-04-26T00:29:16.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa183a29640 5 monclient: authenticate success, global_id 94323 2024-04-26T00:29:16.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa183a29640 10 monclient: _renew_subs 2024-04-26T00:29:16.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa183a29640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:29:16.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa183a29640 1 -- 172.21.15.37:0/585793289 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa17c0a0000 con 0x7fa17c13a060 2024-04-26T00:29:16.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa171ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:29:16.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa171ffb640 1 -- 172.21.15.37:0/585793289 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1740b0bb0 con 0x7fa17c13a060 2024-04-26T00:29:16.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa171ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:16.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa171ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:29:16.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa171ffb640 10 monclient: dump: 2024-04-26T00:29:16.778 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:16.778 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:16.778 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:16.778 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:16.778 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:16.778 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:16.778 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:16.779 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:16.779 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:16.779 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:16.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa183a29640 10 monclient: _renew_subs 2024-04-26T00:29:16.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa183a29640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:29:16.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa183a29640 1 -- 172.21.15.37:0/585793289 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa17c132a00 con 0x7fa17c13a060 2024-04-26T00:29:16.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa1737fe640 4 set_mon_vals no callback set 2024-04-26T00:29:16.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa183a29640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:29:16.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa183a29640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:29:16.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa183a29640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:29:16.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.778+0000 7fa183a29640 1 -- 172.21.15.37:0/585793289 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa17c066f40 con 0x7fa17c13a060 2024-04-26T00:29:16.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.782+0000 7fa171ffb640 1 -- 172.21.15.37:0/585793289 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa1740c4460 con 0x7fa17c13a060 2024-04-26T00:29:16.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.782+0000 7fa171ffb640 1 --2- 172.21.15.37:0/585793289 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa160035ed0 0x7fa160038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:16.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.782+0000 7fa171ffb640 1 -- 172.21.15.37:0/585793289 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(368..368 src has 177..368) ==== 3959+0+0 (secure 0 0 0) 0x7fa1740e3db0 con 0x7fa17c13a060 2024-04-26T00:29:16.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.782+0000 7fa1817c5640 1 --2- 172.21.15.37:0/585793289 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa160035ed0 0x7fa160038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:16.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.782+0000 7fa1817c5640 10 monclient: get_auth_request con 0x7fa160035ed0 auth_method 0 2024-04-26T00:29:16.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.782+0000 7fa1817c5640 1 --2- 172.21.15.37:0/585793289 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa160035ed0 0x7fa160038390 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa178007d60 tx=0x7fa178007cb0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:16.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.786+0000 7fa171ffb640 1 -- 172.21.15.37:0/585793289 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa17c0a0000 con 0x7fa17c13a060 2024-04-26T00:29:16.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.786+0000 7fa171ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:29:16.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:16.786+0000 7fa171ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:29:17.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.014+0000 7fa183a29640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:29:17.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.014+0000 7fa183a29640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:29:17.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.014+0000 7fa183a29640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:29:17.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.014+0000 7fa183a29640 1 -- 172.21.15.37:0/585793289 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa17c134330 con 0x7fa17c13a060 2024-04-26T00:29:17.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.014+0000 7fa171ffb640 1 -- 172.21.15.37:0/585793289 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fa1740b7070 con 0x7fa17c13a060 2024-04-26T00:29:17.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.014+0000 7fa171ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:29:17.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.014+0000 7fa171ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:29:17.004 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:29:17.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.018+0000 7fa183a29640 1 -- 172.21.15.37:0/585793289 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa160035ed0 msgr2=0x7fa160038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:17.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.018+0000 7fa183a29640 1 --2- 172.21.15.37:0/585793289 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa160035ed0 0x7fa160038390 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa178007d60 tx=0x7fa178007cb0 comp rx=0 tx=0).stop 2024-04-26T00:29:17.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.018+0000 7fa183a29640 10 monclient: shutdown 2024-04-26T00:29:17.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.018+0000 7fa183a29640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:29:17.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.018+0000 7fa183a29640 1 -- 172.21.15.37:0/585793289 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa17c13a060 msgr2=0x7fa17c0712e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:17.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.018+0000 7fa183a29640 1 --2- 172.21.15.37:0/585793289 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa17c13a060 0x7fa17c0712e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa17c072ed0 tx=0x7fa174019370 comp rx=0 tx=0).stop 2024-04-26T00:29:17.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.018+0000 7fa183a29640 1 -- 172.21.15.37:0/585793289 shutdown_connections 2024-04-26T00:29:17.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.018+0000 7fa183a29640 1 --2- 172.21.15.37:0/585793289 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa160035ed0 0x7fa160038390 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:17.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.018+0000 7fa183a29640 1 --2- 172.21.15.37:0/585793289 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa17c13a060 0x7fa17c0712e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:17.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.018+0000 7fa183a29640 1 --2- 172.21.15.37:0/585793289 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa17c133f00 0x7fa17c11b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:17.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.018+0000 7fa183a29640 1 --2- 172.21.15.37:0/585793289 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa17c133b20 0x7fa17c070ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:17.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.018+0000 7fa183a29640 1 -- 172.21.15.37:0/585793289 >> 172.21.15.37:0/585793289 conn(0x7fa17c0a1ad0 msgr2=0x7fa17c0a0260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:29:17.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.018+0000 7fa183a29640 1 -- 172.21.15.37:0/585793289 shutdown_connections 2024-04-26T00:29:17.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.018+0000 7fa183a29640 1 -- 172.21.15.37:0/585793289 wait complete. 2024-04-26T00:29:17.023 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:29:17.023 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:29:17.024 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:29:17.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa958d8640 10 monclient: get_monmap_and_config 2024-04-26T00:29:17.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa958d8640 10 monclient: build_initial_monmap 2024-04-26T00:29:17.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa958d8640 10 monclient: monmap: 2024-04-26T00:29:17.150 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:29:17.150 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:17.150 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:29:17.164196+0000 2024-04-26T00:29:17.150 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:29:17.164196+0000 2024-04-26T00:29:17.150 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:29:17.150 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:17.150 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:29:17.150 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:29:17.150 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:29:17.151 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:29:17.151 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:29:17.151 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:17.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa958d8640 1 Processor -- start 2024-04-26T00:29:17.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa958d8640 1 -- start start 2024-04-26T00:29:17.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa958d8640 10 monclient: init 2024-04-26T00:29:17.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa958d8640 10 monclient: _reopen_session rank -1 2024-04-26T00:29:17.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa958d8640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-04-26T00:29:17.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa958d8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffa900a3d90 addr v1:172.21.15.37:6789/0 2024-04-26T00:29:17.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa958d8640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ffa900a4170 0x7ffa900a45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:17.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa958d8640 10 monclient(hunting): picked mon.noname-a con 0x7ffa900a4170 addr v2:172.21.15.37:3300/0 2024-04-26T00:29:17.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa958d8640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ffa9013c3e0 0x7ffa9013e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:17.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa958d8640 10 monclient(hunting): picked mon.noname-c con 0x7ffa9013c3e0 addr v2:172.21.15.45:3300/0 2024-04-26T00:29:17.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa958d8640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa900a5150 con 0x7ffa900a3d90 2024-04-26T00:29:17.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa958d8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:17.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa958d8640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7ffa9009c650 con 0x7ffa900a4170 2024-04-26T00:29:17.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa958d8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:17.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa958d8640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7ffa9009b280 con 0x7ffa9013c3e0 2024-04-26T00:29:17.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa958d8640 10 monclient(hunting): _renew_subs 2024-04-26T00:29:17.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa958d8640 10 monclient(hunting): authenticate will time out at 1538.469817s 2024-04-26T00:29:17.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa8e7fc640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ffa900a4170 0x7ffa900a45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:17.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa8effd640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7ffa900a3d90 0x7ffa9009bf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:46786/0 (socket says 172.21.15.37:46786) 2024-04-26T00:29:17.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa8effd640 1 -- 172.21.15.37:0/1729099906 learned_addr learned my addr 172.21.15.37:0/1729099906 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:29:17.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa8e7fc640 10 monclient(hunting): get_auth_request con 0x7ffa900a4170 auth_method 0 2024-04-26T00:29:17.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa8e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:17.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa8e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:17.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:17.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa8e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:17.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa8e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:17.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa8e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:17.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa8f7fe640 1 --2- 172.21.15.37:0/1729099906 >> v2:172.21.15.45:3300/0 conn(0x7ffa9013c3e0 0x7ffa9013e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:17.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa8f7fe640 10 monclient(hunting): get_auth_request con 0x7ffa9013c3e0 auth_method 0 2024-04-26T00:29:17.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:17.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa8f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:17.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:17.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa8e7fc640 10 monclient(hunting): handle_auth_done global_id 94326 payload 274 2024-04-26T00:29:17.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa8e7fc640 1 -- 172.21.15.37:0/1729099906 >> v2:172.21.15.45:3300/0 conn(0x7ffa9013c3e0 msgr2=0x7ffa9013e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:29:17.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa8e7fc640 1 --2- 172.21.15.37:0/1729099906 >> v2:172.21.15.45:3300/0 conn(0x7ffa9013c3e0 0x7ffa9013e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:17.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa8e7fc640 1 -- 172.21.15.37:0/1729099906 >> v1:172.21.15.37:6789/0 conn(0x7ffa900a3d90 legacy=0x7ffa9009bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:29:17.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa8e7fc640 10 monclient: _finish_hunting 0 2024-04-26T00:29:17.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa8e7fc640 1 monclient: found mon.noname-a 2024-04-26T00:29:17.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa8e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:29:17.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa8e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:29:17.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.162+0000 7ffa8e7fc640 1 -- 172.21.15.37:0/1729099906 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa9009b560 con 0x7ffa900a4170 2024-04-26T00:29:17.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.166+0000 7ffa8e7fc640 1 -- 172.21.15.37:0/1729099906 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa900a4170 msgr2=0x7ffa900a45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:17.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.166+0000 7ffa8e7fc640 1 --2- 172.21.15.37:0/1729099906 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa900a4170 0x7ffa900a45e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ffa900a6e00 tx=0x7ffa8401c300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1593b710d360f5b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:17.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.166+0000 7ffa8dffb640 1 -- 172.21.15.37:0/1729099906 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa840b0070 con 0x7ffa900a4170 2024-04-26T00:29:17.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.166+0000 7ffa8dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:17.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.166+0000 7ffa8dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:29:17.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.166+0000 7ffa8dffb640 10 monclient: dump: 2024-04-26T00:29:17.158 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:17.158 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:17.158 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.158 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.158 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:17.158 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:17.158 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:17.158 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:17.158 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:17.158 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:17.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.166+0000 7ffa8dffb640 10 monclient: _finish_auth 0 2024-04-26T00:29:17.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.166+0000 7ffa8dffb640 10 monclient: _check_auth_tickets 2024-04-26T00:29:17.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.166+0000 7ffa8dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:29:17.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.166+0000 7ffa8dffb640 1 -- 172.21.15.37:0/1729099906 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa840ab600 con 0x7ffa900a4170 2024-04-26T00:29:17.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.166+0000 7ffa8dffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:29:17.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.166+0000 7ffa8dffb640 1 -- 172.21.15.37:0/1729099906 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa840ab9d0 con 0x7ffa900a4170 2024-04-26T00:29:17.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.166+0000 7ffa958d8640 5 monclient: authenticate success, global_id 94326 2024-04-26T00:29:17.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.166+0000 7ffa958d8640 10 monclient: get_monmap_and_config success 2024-04-26T00:29:17.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.166+0000 7ffa958d8640 4 set_mon_vals no callback set 2024-04-26T00:29:17.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.166+0000 7ffa8dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:17.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.166+0000 7ffa8dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:29:17.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.166+0000 7ffa8dffb640 10 monclient: dump: 2024-04-26T00:29:17.159 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:17.159 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:17.159 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.159 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.159 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:17.160 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:17.160 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:17.160 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:17.160 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:17.160 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:17.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.166+0000 7ffa958d8640 10 monclient: shutdown 2024-04-26T00:29:17.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.166+0000 7ffa958d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:29:17.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.166+0000 7ffa958d8640 1 -- 172.21.15.37:0/1729099906 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa900a4170 msgr2=0x7ffa900a45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:17.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.166+0000 7ffa958d8640 1 --2- 172.21.15.37:0/1729099906 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa900a4170 0x7ffa900a45e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ffa900a6e00 tx=0x7ffa8401c300 comp rx=0 tx=0).stop 2024-04-26T00:29:17.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.166+0000 7ffa958d8640 1 -- 172.21.15.37:0/1729099906 shutdown_connections 2024-04-26T00:29:17.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.166+0000 7ffa958d8640 1 --2- 172.21.15.37:0/1729099906 >> v2:172.21.15.45:3300/0 conn(0x7ffa9013c3e0 0x7ffa9013e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:17.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.166+0000 7ffa958d8640 1 --2- 172.21.15.37:0/1729099906 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa900a4170 0x7ffa900a45e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:17.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.166+0000 7ffa958d8640 1 -- 172.21.15.37:0/1729099906 >> 172.21.15.37:0/1729099906 conn(0x7ffa900aa0a0 msgr2=0x7ffa900a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:29:17.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.166+0000 7ffa958d8640 1 -- 172.21.15.37:0/1729099906 shutdown_connections 2024-04-26T00:29:17.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.166+0000 7ffa958d8640 1 -- 172.21.15.37:0/1729099906 wait complete. 2024-04-26T00:29:17.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa958d8640 10 monclient: build_initial_monmap 2024-04-26T00:29:17.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa958d8640 10 monclient: monmap: 2024-04-26T00:29:17.161 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:29:17.161 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:29:17.161 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:29:17.161 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:29:17.161 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:29:17.161 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:29:17.161 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:29:17.161 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:29:17.161 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:29:17.161 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:17.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa958d8640 1 Processor -- start 2024-04-26T00:29:17.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa958d8640 1 -- start start 2024-04-26T00:29:17.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa958d8640 10 monclient: init 2024-04-26T00:29:17.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa958d8640 10 monclient: _reopen_session rank -1 2024-04-26T00:29:17.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa958d8640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:29:17.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa958d8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffa900a3d90 0x7ffa90136e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:17.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa958d8640 10 monclient(hunting): picked mon.noname-c con 0x7ffa900a3d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:29:17.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa958d8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa900a4170 0x7ffa90137340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:17.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa958d8640 10 monclient(hunting): picked mon.noname-b con 0x7ffa900a4170 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:29:17.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa958d8640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa9013c3e0 0x7ffa90139f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:17.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa958d8640 10 monclient(hunting): picked mon.noname-a con 0x7ffa9013c3e0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:29:17.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa958d8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:17.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa958d8640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ffa901418c0 con 0x7ffa9013c3e0 2024-04-26T00:29:17.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa958d8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:17.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa958d8640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ffa90141740 con 0x7ffa900a4170 2024-04-26T00:29:17.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa958d8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:17.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa958d8640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ffa90141a40 con 0x7ffa900a3d90 2024-04-26T00:29:17.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa958d8640 10 monclient(hunting): _renew_subs 2024-04-26T00:29:17.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa958d8640 10 monclient(hunting): authenticate will time out at 1538.477371s 2024-04-26T00:29:17.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa8effd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffa900a3d90 0x7ffa90136e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:17.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa8f7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa9013c3e0 0x7ffa90139f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:17.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa8effd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffa900a3d90 0x7ffa90136e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:41310/0 (socket says 172.21.15.37:41310) 2024-04-26T00:29:17.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa8f7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa9013c3e0 0x7ffa90139f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33826/0 (socket says 172.21.15.37:33826) 2024-04-26T00:29:17.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa8effd640 1 -- 172.21.15.37:0/148865763 learned_addr learned my addr 172.21.15.37:0/148865763 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:29:17.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa8effd640 10 monclient(hunting): get_auth_request con 0x7ffa900a3d90 auth_method 0 2024-04-26T00:29:17.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa8effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:17.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa8effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:17.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa8effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:17.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa8f7fe640 10 monclient(hunting): get_auth_request con 0x7ffa9013c3e0 auth_method 0 2024-04-26T00:29:17.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:17.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa8f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:17.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:17.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa8effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:17.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa8effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:17.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa8effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:17.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa8f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:17.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa8f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:17.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa8f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:17.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa8e7fc640 1 --2- 172.21.15.37:0/148865763 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa900a4170 0x7ffa90137340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:17.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa8e7fc640 10 monclient(hunting): get_auth_request con 0x7ffa900a4170 auth_method 0 2024-04-26T00:29:17.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa8e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:17.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa8e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:17.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:17.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa8f7fe640 10 monclient(hunting): handle_auth_done global_id 94329 payload 824 2024-04-26T00:29:17.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa8f7fe640 1 -- 172.21.15.37:0/148865763 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffa900a3d90 msgr2=0x7ffa90136e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:17.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa8f7fe640 1 --2- 172.21.15.37:0/148865763 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffa900a3d90 0x7ffa90136e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:17.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa8f7fe640 1 -- 172.21.15.37:0/148865763 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa900a4170 msgr2=0x7ffa90137340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:17.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa8f7fe640 1 --2- 172.21.15.37:0/148865763 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa900a4170 0x7ffa90137340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:17.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa8f7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:29:17.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa8f7fe640 1 monclient: found mon.noname-a 2024-04-26T00:29:17.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa8f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:29:17.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa8f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:29:17.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa8f7fe640 1 -- 172.21.15.37:0/148865763 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa901218e0 con 0x7ffa9013c3e0 2024-04-26T00:29:17.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.170+0000 7ffa8effd640 1 --2- 172.21.15.37:0/148865763 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffa900a3d90 0x7ffa90136e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:29:17.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa8f7fe640 1 -- 172.21.15.37:0/148865763 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa9013c3e0 msgr2=0x7ffa90139f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:17.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa8f7fe640 1 --2- 172.21.15.37:0/148865763 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa9013c3e0 0x7ffa90139f20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ffa88007d90 tx=0x7ffa8800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:17.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa77fff640 1 -- 172.21.15.37:0/148865763 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa880039a0 con 0x7ffa9013c3e0 2024-04-26T00:29:17.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa77fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:17.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa77fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:29:17.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa77fff640 10 monclient: dump: 2024-04-26T00:29:17.170 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:17.170 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:17.170 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.170 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.170 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:17.170 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:17.170 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:17.170 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:17.170 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:17.170 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:17.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa77fff640 10 monclient: _finish_auth 0 2024-04-26T00:29:17.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa77fff640 10 monclient: _check_auth_tickets 2024-04-26T00:29:17.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa77fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:29:17.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa77fff640 1 -- 172.21.15.37:0/148865763 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa88003b40 con 0x7ffa9013c3e0 2024-04-26T00:29:17.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa77fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:29:17.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa77fff640 1 -- 172.21.15.37:0/148865763 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa88004020 con 0x7ffa9013c3e0 2024-04-26T00:29:17.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa77fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:17.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa77fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:29:17.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa77fff640 10 monclient: dump: 2024-04-26T00:29:17.171 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:17.171 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:17.171 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.171 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.171 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:17.171 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:17.172 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:17.172 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:17.172 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:17.172 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:17.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa958d8640 5 monclient: authenticate success, global_id 94329 2024-04-26T00:29:17.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa958d8640 10 monclient: _renew_subs 2024-04-26T00:29:17.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa958d8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:29:17.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa958d8640 1 -- 172.21.15.37:0/148865763 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa9009ada0 con 0x7ffa9013c3e0 2024-04-26T00:29:17.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa8d7fa640 4 set_mon_vals no callback set 2024-04-26T00:29:17.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa958d8640 10 monclient: _renew_subs 2024-04-26T00:29:17.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa958d8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:29:17.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa958d8640 1 -- 172.21.15.37:0/148865763 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa90121600 con 0x7ffa9013c3e0 2024-04-26T00:29:17.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa958d8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:29:17.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa958d8640 1 --2- 172.21.15.37:0/148865763 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa9013a460 0x7ffa9013a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:17.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa958d8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:17.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa958d8640 1 -- 172.21.15.37:0/148865763 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ffa90121120 con 0x7ffa9013a460 2024-04-26T00:29:17.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa8effd640 1 --2- 172.21.15.37:0/148865763 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa9013a460 0x7ffa9013a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:17.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa8effd640 10 monclient: get_auth_request con 0x7ffa9013a460 auth_method 0 2024-04-26T00:29:17.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa8effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:17.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa8effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:17.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa8effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:17.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa8effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:17.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa8effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:17.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa77fff640 1 -- 172.21.15.37:0/148865763 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ffa8801b020 con 0x7ffa9013c3e0 2024-04-26T00:29:17.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa8effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:17.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa8effd640 10 monclient(hunting): handle_auth_done global_id 94332 payload 824 2024-04-26T00:29:17.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa8effd640 1 -- 172.21.15.37:0/148865763 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ffa9004f510 con 0x7ffa9013a460 2024-04-26T00:29:17.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa77fff640 1 --2- 172.21.15.37:0/148865763 >> v2:172.21.15.45:6812/2379918623 conn(0x7ffa7c035ed0 0x7ffa7c038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:17.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa77fff640 1 -- 172.21.15.37:0/148865763 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(368..368 src has 177..368) ==== 3959+0+0 (secure 0 0 0) 0x7ffa88048c50 con 0x7ffa9013c3e0 2024-04-26T00:29:17.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa8e7fc640 1 --2- 172.21.15.37:0/148865763 >> v2:172.21.15.45:6812/2379918623 conn(0x7ffa7c035ed0 0x7ffa7c038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:17.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.174+0000 7ffa8e7fc640 10 monclient: get_auth_request con 0x7ffa7c035ed0 auth_method 0 2024-04-26T00:29:17.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.178+0000 7ffa8effd640 1 -- 172.21.15.37:0/148865763 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa9013a460 msgr2=0x7ffa9013a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:17.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.178+0000 7ffa8effd640 1 --2- 172.21.15.37:0/148865763 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa9013a460 0x7ffa9013a840 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ffa800072a0 tx=0x7ffa80007760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:17.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.178+0000 7ffa77fff640 1 -- 172.21.15.37:0/148865763 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa8000fde0 con 0x7ffa9013a460 2024-04-26T00:29:17.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.178+0000 7ffa77fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:17.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.178+0000 7ffa77fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:29:17.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.178+0000 7ffa77fff640 10 monclient: dump: 2024-04-26T00:29:17.174 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:17.174 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:17.174 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.174 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.174 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:17.174 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:17.174 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:17.174 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:17.174 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:17.174 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:17.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.178+0000 7ffa77fff640 1 -- 172.21.15.37:0/148865763 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ffa8000fde0 con 0x7ffa9013a460 2024-04-26T00:29:17.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.178+0000 7ffa77fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:29:17.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.178+0000 7ffa77fff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:29:17.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.178+0000 7ffa77fff640 1 -- 172.21.15.37:0/148865763 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa9013a460 msgr2=0x7ffa9013a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:17.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.178+0000 7ffa77fff640 1 --2- 172.21.15.37:0/148865763 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa9013a460 0x7ffa9013a840 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ffa800072a0 tx=0x7ffa80007760 comp rx=0 tx=0).stop 2024-04-26T00:29:17.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.178+0000 7ffa77fff640 1 -- 172.21.15.37:0/148865763 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa9013a460 msgr2=0x7ffa9013a840 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:29:17.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.178+0000 7ffa77fff640 1 --2- 172.21.15.37:0/148865763 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa9013a460 0x7ffa9013a840 secure :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ffa800072a0 tx=0x7ffa80007760 comp rx=0 tx=0).stop 2024-04-26T00:29:17.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.178+0000 7ffa8e7fc640 1 --2- 172.21.15.37:0/148865763 >> v2:172.21.15.45:6812/2379918623 conn(0x7ffa7c035ed0 0x7ffa7c038390 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ffa900a57e0 tx=0x7ffa84008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:17.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.182+0000 7ffa958d8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:29:17.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.182+0000 7ffa958d8640 1 --2- 172.21.15.37:0/148865763 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa9013ad80 0x7ffa9009e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:17.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.182+0000 7ffa958d8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:17.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.182+0000 7ffa958d8640 1 -- 172.21.15.37:0/148865763 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ffa9011fed0 con 0x7ffa9013ad80 2024-04-26T00:29:17.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.182+0000 7ffa8effd640 1 --2- 172.21.15.37:0/148865763 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa9013ad80 0x7ffa9009e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:17.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.182+0000 7ffa8effd640 10 monclient: get_auth_request con 0x7ffa9013ad80 auth_method 0 2024-04-26T00:29:17.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.182+0000 7ffa8effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:17.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.182+0000 7ffa8effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:17.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.182+0000 7ffa8effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:17.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.182+0000 7ffa8effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:17.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.182+0000 7ffa8effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:17.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.182+0000 7ffa8effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:17.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.182+0000 7ffa8effd640 10 monclient(hunting): handle_auth_done global_id 94335 payload 824 2024-04-26T00:29:17.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.182+0000 7ffa8effd640 1 -- 172.21.15.37:0/148865763 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ffa90075dc0 con 0x7ffa9013ad80 2024-04-26T00:29:17.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.186+0000 7ffa8effd640 1 -- 172.21.15.37:0/148865763 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa9013ad80 msgr2=0x7ffa9009e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:17.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.186+0000 7ffa8effd640 1 --2- 172.21.15.37:0/148865763 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa9013ad80 0x7ffa9009e080 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ffa80017000 tx=0x7ffa80004950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:17.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.186+0000 7ffa77fff640 1 -- 172.21.15.37:0/148865763 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa8001ee40 con 0x7ffa9013ad80 2024-04-26T00:29:17.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.186+0000 7ffa77fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:17.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.186+0000 7ffa77fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:29:17.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.186+0000 7ffa77fff640 10 monclient: dump: 2024-04-26T00:29:17.178 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:17.178 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:17.178 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.178 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.178 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:17.178 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:17.178 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:17.178 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:17.178 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:17.178 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:17.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.186+0000 7ffa77fff640 1 -- 172.21.15.37:0/148865763 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4549 (secure 0 0 0) 0x7ffa80006240 con 0x7ffa9013ad80 2024-04-26T00:29:17.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.186+0000 7ffa77fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:29:17.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.186+0000 7ffa77fff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:29:17.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.186+0000 7ffa77fff640 1 -- 172.21.15.37:0/148865763 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa9013ad80 msgr2=0x7ffa9009e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:17.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.186+0000 7ffa77fff640 1 --2- 172.21.15.37:0/148865763 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa9013ad80 0x7ffa9009e080 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ffa80017000 tx=0x7ffa80004950 comp rx=0 tx=0).stop 2024-04-26T00:29:17.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.186+0000 7ffa77fff640 1 -- 172.21.15.37:0/148865763 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa9013ad80 msgr2=0x7ffa9009e080 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:29:17.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.186+0000 7ffa77fff640 1 --2- 172.21.15.37:0/148865763 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa9013ad80 0x7ffa9009e080 secure :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ffa80017000 tx=0x7ffa80004950 comp rx=0 tx=0).stop 2024-04-26T00:29:17.179 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:29:17.179 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:29:17.179 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:29:17.179 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:29:17.179 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 70, 2024-04-26T00:29:17.180 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 117946, 2024-04-26T00:29:17.180 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:29:17.180 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:29:17.180 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:29:17.180 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:29:17.180 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:17.180 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 2, 2024-04-26T00:29:17.180 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:29:17.180 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:29:17.180 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:29:17.180 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:29:17.180 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:29:17.180 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:29:17.180 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:29:17.180 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:29:17.181 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:29:17.181 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:29:17.181 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:29:17.181 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:29:17.181 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:29:17.181 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:29:17.181 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:17.181 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:29:17.181 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:29:17.181 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:29:17.181 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:29:17.181 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:29:17.181 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:29:17.181 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:29:17.182 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:29:17.182 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:29:17.182 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:29:17.182 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:29:17.182 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:29:17.182 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:29:17.182 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:17.182 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:17.182 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:29:17.182 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:29:17.182 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:29:17.182 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:29:17.182 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:29:17.182 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:29:17.183 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:29:17.183 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:29:17.183 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:29:17.183 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:29:17.183 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:29:17.183 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:29:17.183 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:29:17.183 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:29:17.183 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:29:17.183 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:29:17.183 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:29:17.183 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:29:17.183 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:29:17.183 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:29:17.183 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:29:17.183 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:29:17.184 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:29:17.184 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:29:17.184 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:29:17.184 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:29:17.184 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:29:17.184 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:29:17.184 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:17.184 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:29:17.184 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:17.184 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:29:17.184 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:17.184 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:29:17.184 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:29:17.184 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:29:17.184 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:29:17.184 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:17.185 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:29:17.185 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:29:17.185 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:17.185 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:17.185 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:17.185 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:29:17.185 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:29:17.185 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:17.185 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:17.185 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:17.185 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:17.185 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:29:17.185 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:29:17.185 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:29:17.185 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:29:17.185 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:29:17.186 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:17.186 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:17.186 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:29:17.186 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:29:17.186 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:29:17.186 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:29:17.186 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:17.186 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:29:17.186 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:29:17.186 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:17.186 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:17.186 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:17.186 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:29:17.186 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:29:17.186 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:17.186 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:17.187 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:17.187 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:17.187 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:29:17.187 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:29:17.187 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:29:17.187 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:29:17.187 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:29:17.187 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:17.187 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:17.187 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:29:17.187 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:29:17.187 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:29:17.187 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:29:17.187 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:17.187 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:29:17.188 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:29:17.188 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:17.188 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:17.188 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:17.188 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:29:17.188 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:29:17.188 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:17.188 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:17.188 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:17.188 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:17.188 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:29:17.188 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:29:17.188 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:29:17.188 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:29:17.188 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:29:17.188 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:17.188 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:17.189 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:17.189 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:29:17.189 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:29:17.189 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:17.189 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:29:17.189 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:29:17.189 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:29:17.189 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:17.189 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:17.189 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:29:17.189 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:17.189 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:29:17.189 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:29:17.189 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:29:17.189 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:17.189 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:17.189 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:17.190 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:29:17.190 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:29:17.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.186+0000 7ffa958d8640 1 -- 172.21.15.37:0/148865763 >> v2:172.21.15.45:6812/2379918623 conn(0x7ffa7c035ed0 msgr2=0x7ffa7c038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:17.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.186+0000 7ffa958d8640 1 --2- 172.21.15.37:0/148865763 >> v2:172.21.15.45:6812/2379918623 conn(0x7ffa7c035ed0 0x7ffa7c038390 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ffa900a57e0 tx=0x7ffa84008040 comp rx=0 tx=0).stop 2024-04-26T00:29:17.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.186+0000 7ffa958d8640 10 monclient: shutdown 2024-04-26T00:29:17.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.186+0000 7ffa958d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:29:17.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.186+0000 7ffa958d8640 1 -- 172.21.15.37:0/148865763 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa9013c3e0 msgr2=0x7ffa90139f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:17.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.186+0000 7ffa8f7fe640 1 -- 172.21.15.37:0/148865763 reap_dead start 2024-04-26T00:29:17.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.186+0000 7ffa958d8640 1 --2- 172.21.15.37:0/148865763 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa9013c3e0 0x7ffa90139f20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ffa88007d90 tx=0x7ffa8800e4b0 comp rx=0 tx=0).stop 2024-04-26T00:29:17.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.186+0000 7ffa958d8640 1 -- 172.21.15.37:0/148865763 shutdown_connections 2024-04-26T00:29:17.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.186+0000 7ffa958d8640 1 --2- 172.21.15.37:0/148865763 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa9013c3e0 0x7ffa90139f20 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:17.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.186+0000 7ffa958d8640 1 -- 172.21.15.37:0/148865763 >> 172.21.15.37:0/148865763 conn(0x7ffa900aa0a0 msgr2=0x7ffa900a8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:29:17.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.186+0000 7ffa958d8640 1 -- 172.21.15.37:0/148865763 shutdown_connections 2024-04-26T00:29:17.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.186+0000 7ffa958d8640 1 -- 172.21.15.37:0/148865763 wait complete. 2024-04-26T00:29:17.192 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:29:17.192 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:29:17.192 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:29:17.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c6e39b640 10 monclient: get_monmap_and_config 2024-04-26T00:29:17.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c6e39b640 10 monclient: build_initial_monmap 2024-04-26T00:29:17.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c6e39b640 10 monclient: monmap: 2024-04-26T00:29:17.307 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:29:17.307 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:17.307 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:29:17.321033+0000 2024-04-26T00:29:17.307 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:29:17.321033+0000 2024-04-26T00:29:17.307 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:29:17.307 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:17.307 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:29:17.307 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:29:17.307 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:29:17.307 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:29:17.307 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:29:17.307 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:17.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c6e39b640 1 Processor -- start 2024-04-26T00:29:17.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c6e39b640 1 -- start start 2024-04-26T00:29:17.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c6e39b640 10 monclient: init 2024-04-26T00:29:17.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c6e39b640 10 monclient: _reopen_session rank -1 2024-04-26T00:29:17.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c6e39b640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-04-26T00:29:17.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c6e39b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7c68136f20 addr v1:172.21.15.45:6789/0 2024-04-26T00:29:17.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c6e39b640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f7c68137a10 0x7f7c6809bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:17.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c6e39b640 10 monclient(hunting): picked mon.noname-a con 0x7f7c68137a10 addr v2:172.21.15.37:3300/0 2024-04-26T00:29:17.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c6e39b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c6813c450 0x7f7c6813e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:17.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c6e39b640 10 monclient(hunting): picked mon.noname-b con 0x7f7c6813c450 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:29:17.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c6e39b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c680a50f0 con 0x7f7c68136f20 2024-04-26T00:29:17.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c6e39b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:17.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c6e39b640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f7c6809b330 con 0x7f7c68137a10 2024-04-26T00:29:17.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c6e39b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:17.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c6e39b640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f7c6809b610 con 0x7f7c6813c450 2024-04-26T00:29:17.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c6e39b640 10 monclient(hunting): _renew_subs 2024-04-26T00:29:17.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c6e39b640 10 monclient(hunting): authenticate will time out at 1538.627384s 2024-04-26T00:29:17.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c677fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f7c68137a10 0x7f7c6809bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:17.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c6c938640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c6813c450 0x7f7c6813e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:17.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c6c938640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c6813c450 0x7f7c6813e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:41322/0 (socket says 172.21.15.37:41322) 2024-04-26T00:29:17.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c677fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f7c68137a10 0x7f7c6809bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33850/0 (socket says 172.21.15.37:33850) 2024-04-26T00:29:17.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c6c938640 1 -- 172.21.15.37:0/2054717373 learned_addr learned my addr 172.21.15.37:0/2054717373 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:29:17.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c6c938640 10 monclient(hunting): get_auth_request con 0x7f7c6813c450 auth_method 0 2024-04-26T00:29:17.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c6c938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:17.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c6c938640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:17.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c6c938640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:17.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c677fe640 10 monclient(hunting): get_auth_request con 0x7f7c68137a10 auth_method 0 2024-04-26T00:29:17.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:17.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c677fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:17.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c677fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:17.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c6c938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:17.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c6c938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:17.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c6c938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:17.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:17.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:17.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:17.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c6c938640 10 monclient(hunting): handle_auth_done global_id 84392 payload 274 2024-04-26T00:29:17.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c6c938640 1 -- 172.21.15.37:0/2054717373 >> v2:172.21.15.37:3300/0 conn(0x7f7c68137a10 msgr2=0x7f7c6809bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:29:17.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c6c938640 1 --2- 172.21.15.37:0/2054717373 >> v2:172.21.15.37:3300/0 conn(0x7f7c68137a10 0x7f7c6809bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:17.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c6c938640 1 -- 172.21.15.37:0/2054717373 >> v1:172.21.15.45:6789/0 conn(0x7f7c68136f20 legacy=0x7f7c68137300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:29:17.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c6c938640 10 monclient: _finish_hunting 0 2024-04-26T00:29:17.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c6c938640 1 monclient: found mon.noname-b 2024-04-26T00:29:17.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c6c938640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:29:17.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c6c938640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:29:17.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c6c938640 1 -- 172.21.15.37:0/2054717373 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c6809b8f0 con 0x7f7c6813c450 2024-04-26T00:29:17.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.318+0000 7f7c677fe640 1 --2- 172.21.15.37:0/2054717373 >> v2:172.21.15.37:3300/0 conn(0x7f7c68137a10 0x7f7c6809bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:29:17.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6c938640 1 -- 172.21.15.37:0/2054717373 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c6813c450 msgr2=0x7f7c6813e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:17.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6c938640 1 --2- 172.21.15.37:0/2054717373 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c6813c450 0x7f7c6813e840 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7c60009870 tx=0x7f7c6001c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b7f7e71f08fa348 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:17.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c66ffd640 1 -- 172.21.15.37:0/2054717373 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c600b0070 con 0x7f7c6813c450 2024-04-26T00:29:17.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:17.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c66ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:29:17.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c66ffd640 10 monclient: dump: 2024-04-26T00:29:17.314 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:17.315 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:17.315 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.315 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.315 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:17.315 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:17.315 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:17.315 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:17.315 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:17.315 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:17.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c66ffd640 10 monclient: _finish_auth 0 2024-04-26T00:29:17.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c66ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:29:17.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:29:17.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c66ffd640 1 -- 172.21.15.37:0/2054717373 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c600ab600 con 0x7f7c6813c450 2024-04-26T00:29:17.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c66ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:29:17.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c66ffd640 1 -- 172.21.15.37:0/2054717373 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c600ab9d0 con 0x7f7c6813c450 2024-04-26T00:29:17.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6e39b640 5 monclient: authenticate success, global_id 84392 2024-04-26T00:29:17.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6e39b640 10 monclient: get_monmap_and_config success 2024-04-26T00:29:17.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6e39b640 4 set_mon_vals no callback set 2024-04-26T00:29:17.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:17.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c66ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:29:17.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c66ffd640 10 monclient: dump: 2024-04-26T00:29:17.316 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:17.316 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:17.316 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.316 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.316 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:17.316 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:17.317 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:17.317 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:17.317 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:17.317 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:17.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6e39b640 10 monclient: shutdown 2024-04-26T00:29:17.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6e39b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:29:17.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6e39b640 1 -- 172.21.15.37:0/2054717373 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c6813c450 msgr2=0x7f7c6813e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:17.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6e39b640 1 --2- 172.21.15.37:0/2054717373 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c6813c450 0x7f7c6813e840 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7c60009870 tx=0x7f7c6001c470 comp rx=0 tx=0).stop 2024-04-26T00:29:17.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6e39b640 1 -- 172.21.15.37:0/2054717373 shutdown_connections 2024-04-26T00:29:17.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6e39b640 1 --2- 172.21.15.37:0/2054717373 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c6813c450 0x7f7c6813e840 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:17.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6e39b640 1 --2- 172.21.15.37:0/2054717373 >> v2:172.21.15.37:3300/0 conn(0x7f7c68137a10 0x7f7c6809bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:17.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6e39b640 1 -- 172.21.15.37:0/2054717373 >> 172.21.15.37:0/2054717373 conn(0x7f7c680a61b0 msgr2=0x7f7c680a65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:29:17.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6e39b640 1 -- 172.21.15.37:0/2054717373 shutdown_connections 2024-04-26T00:29:17.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6e39b640 1 -- 172.21.15.37:0/2054717373 wait complete. 2024-04-26T00:29:17.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6e39b640 10 monclient: build_initial_monmap 2024-04-26T00:29:17.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6e39b640 10 monclient: monmap: 2024-04-26T00:29:17.322 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:29:17.322 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:29:17.323 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:29:17.323 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:29:17.323 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:29:17.323 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:29:17.323 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:29:17.323 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:29:17.323 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:29:17.323 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:17.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6e39b640 1 Processor -- start 2024-04-26T00:29:17.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6e39b640 1 -- start start 2024-04-26T00:29:17.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6e39b640 10 monclient: init 2024-04-26T00:29:17.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6e39b640 10 monclient: _reopen_session rank -1 2024-04-26T00:29:17.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6e39b640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:29:17.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6e39b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c68136f20 0x7f7c681cec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:17.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6e39b640 10 monclient(hunting): picked mon.noname-c con 0x7f7c68136f20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:29:17.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6e39b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c68137a10 0x7f7c68134f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:17.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6e39b640 10 monclient(hunting): picked mon.noname-b con 0x7f7c68137a10 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:29:17.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6e39b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c6813c450 0x7f7c68137ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:17.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6e39b640 10 monclient(hunting): picked mon.noname-a con 0x7f7c6813c450 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:29:17.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6e39b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:17.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6e39b640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f7c68141930 con 0x7f7c6813c450 2024-04-26T00:29:17.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6e39b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:17.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6e39b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7c681417b0 con 0x7f7c68137a10 2024-04-26T00:29:17.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6e39b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:17.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6e39b640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f7c68141ab0 con 0x7f7c68136f20 2024-04-26T00:29:17.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6e39b640 10 monclient(hunting): _renew_subs 2024-04-26T00:29:17.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6e39b640 10 monclient(hunting): authenticate will time out at 1538.634477s 2024-04-26T00:29:17.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6c938640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c6813c450 0x7f7c68137ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:17.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c67fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c68136f20 0x7f7c681cec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:17.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6c938640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c6813c450 0x7f7c68137ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33852/0 (socket says 172.21.15.37:33852) 2024-04-26T00:29:17.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6c938640 1 -- 172.21.15.37:0/191864880 learned_addr learned my addr 172.21.15.37:0/191864880 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:29:17.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6c938640 10 monclient(hunting): get_auth_request con 0x7f7c6813c450 auth_method 0 2024-04-26T00:29:17.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6c938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:17.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6c938640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:17.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6c938640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:17.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c67fff640 10 monclient(hunting): get_auth_request con 0x7f7c68136f20 auth_method 0 2024-04-26T00:29:17.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:17.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c67fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:17.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c67fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:17.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6c938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:17.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6c938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:17.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6c938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:17.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:17.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:17.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c67fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:17.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c677fe640 1 --2- 172.21.15.37:0/191864880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c68137a10 0x7f7c68134f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:17.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6c938640 10 monclient(hunting): handle_auth_done global_id 94341 payload 824 2024-04-26T00:29:17.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6c938640 1 -- 172.21.15.37:0/191864880 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c68136f20 msgr2=0x7f7c681cec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:17.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6c938640 1 --2- 172.21.15.37:0/191864880 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c68136f20 0x7f7c681cec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:17.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6c938640 1 -- 172.21.15.37:0/191864880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c68137a10 msgr2=0x7f7c68134f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:17.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6c938640 1 --2- 172.21.15.37:0/191864880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c68137a10 0x7f7c68134f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:17.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6c938640 10 monclient: _finish_hunting 0 2024-04-26T00:29:17.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6c938640 1 monclient: found mon.noname-a 2024-04-26T00:29:17.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6c938640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:29:17.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6c938640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:29:17.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c6c938640 1 -- 172.21.15.37:0/191864880 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c6813bce0 con 0x7f7c6813c450 2024-04-26T00:29:17.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c677fe640 10 monclient: get_auth_request con 0x7f7c68137a10 auth_method 0 2024-04-26T00:29:17.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c67fff640 1 --2- 172.21.15.37:0/191864880 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c68136f20 0x7f7c681cec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:29:17.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.326+0000 7f7c677fe640 1 --2- 172.21.15.37:0/191864880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c68137a10 0x7f7c68134f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:29:17.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c6c938640 1 -- 172.21.15.37:0/191864880 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c6813c450 msgr2=0x7f7c68137ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:17.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c6c938640 1 --2- 172.21.15.37:0/191864880 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c6813c450 0x7f7c68137ec0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7c6001cd10 tx=0x7f7c60019210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:17.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c64ff9640 1 -- 172.21.15.37:0/191864880 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c600bb020 con 0x7f7c6813c450 2024-04-26T00:29:17.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c64ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:17.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c64ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:29:17.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c64ff9640 10 monclient: dump: 2024-04-26T00:29:17.327 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:17.327 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:17.327 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.328 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.328 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:17.328 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:17.328 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:17.328 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:17.328 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:17.328 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:17.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c64ff9640 10 monclient: _finish_auth 0 2024-04-26T00:29:17.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c64ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:29:17.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c64ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:29:17.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c64ff9640 1 -- 172.21.15.37:0/191864880 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c600b1040 con 0x7f7c6813c450 2024-04-26T00:29:17.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c64ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:29:17.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c64ff9640 1 -- 172.21.15.37:0/191864880 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c600b0070 con 0x7f7c6813c450 2024-04-26T00:29:17.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c64ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:17.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c64ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:29:17.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c64ff9640 10 monclient: dump: 2024-04-26T00:29:17.329 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:17.329 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:17.329 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.329 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.329 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:17.329 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:17.329 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:17.329 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:17.329 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:17.329 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:17.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c6e39b640 5 monclient: authenticate success, global_id 94341 2024-04-26T00:29:17.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c6e39b640 10 monclient: _renew_subs 2024-04-26T00:29:17.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c6e39b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:29:17.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c6e39b640 1 -- 172.21.15.37:0/191864880 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c6813ba00 con 0x7f7c6813c450 2024-04-26T00:29:17.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c667fc640 4 set_mon_vals no callback set 2024-04-26T00:29:17.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c6e39b640 10 monclient: _renew_subs 2024-04-26T00:29:17.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c6e39b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:29:17.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c6e39b640 1 -- 172.21.15.37:0/191864880 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c68135650 con 0x7f7c6813c450 2024-04-26T00:29:17.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c6e39b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:29:17.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c6e39b640 1 --2- 172.21.15.37:0/191864880 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c68138bc0 0x7f7c6802c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:17.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c6e39b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:17.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c6e39b640 1 -- 172.21.15.37:0/191864880 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f7c68136050 con 0x7f7c68138bc0 2024-04-26T00:29:17.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c67fff640 1 --2- 172.21.15.37:0/191864880 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c68138bc0 0x7f7c6802c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:17.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c64ff9640 1 -- 172.21.15.37:0/191864880 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f7c600be020 con 0x7f7c6813c450 2024-04-26T00:29:17.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c67fff640 10 monclient: get_auth_request con 0x7f7c68138bc0 auth_method 0 2024-04-26T00:29:17.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:17.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c67fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:17.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c67fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:17.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:17.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:17.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c67fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:17.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c64ff9640 1 --2- 172.21.15.37:0/191864880 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7c54035e80 0x7f7c54038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:17.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c67fff640 10 monclient(hunting): handle_auth_done global_id 84398 payload 824 2024-04-26T00:29:17.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c64ff9640 1 -- 172.21.15.37:0/191864880 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(368..368 src has 177..368) ==== 3959+0+0 (secure 0 0 0) 0x7f7c600e2f40 con 0x7f7c6813c450 2024-04-26T00:29:17.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c67fff640 1 -- 172.21.15.37:0/191864880 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7c6804f510 con 0x7f7c68138bc0 2024-04-26T00:29:17.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c677fe640 1 --2- 172.21.15.37:0/191864880 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7c54035e80 0x7f7c54038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:17.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.330+0000 7f7c677fe640 10 monclient: get_auth_request con 0x7f7c54035e80 auth_method 0 2024-04-26T00:29:17.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.334+0000 7f7c67fff640 1 -- 172.21.15.37:0/191864880 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c68138bc0 msgr2=0x7f7c6802c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:17.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.334+0000 7f7c67fff640 1 --2- 172.21.15.37:0/191864880 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c68138bc0 0x7f7c6802c000 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7c580048e0 tx=0x7f7c5800d520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:17.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.334+0000 7f7c64ff9640 1 -- 172.21.15.37:0/191864880 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c58004970 con 0x7f7c68138bc0 2024-04-26T00:29:17.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.334+0000 7f7c64ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:17.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.334+0000 7f7c64ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:29:17.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.334+0000 7f7c64ff9640 10 monclient: dump: 2024-04-26T00:29:17.331 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:17.332 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:17.332 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.332 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.332 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:17.332 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:17.332 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:17.332 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:17.332 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:17.332 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:17.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.334+0000 7f7c64ff9640 1 -- 172.21.15.37:0/191864880 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7c58004970 con 0x7f7c68138bc0 2024-04-26T00:29:17.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.334+0000 7f7c64ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:29:17.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.334+0000 7f7c64ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:29:17.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.334+0000 7f7c64ff9640 1 -- 172.21.15.37:0/191864880 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c68138bc0 msgr2=0x7f7c6802c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:17.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.334+0000 7f7c64ff9640 1 --2- 172.21.15.37:0/191864880 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c68138bc0 0x7f7c6802c000 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7c580048e0 tx=0x7f7c5800d520 comp rx=0 tx=0).stop 2024-04-26T00:29:17.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.334+0000 7f7c64ff9640 1 -- 172.21.15.37:0/191864880 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c68138bc0 msgr2=0x7f7c6802c000 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:29:17.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.334+0000 7f7c64ff9640 1 --2- 172.21.15.37:0/191864880 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c68138bc0 0x7f7c6802c000 secure :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7c580048e0 tx=0x7f7c5800d520 comp rx=0 tx=0).stop 2024-04-26T00:29:17.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.334+0000 7f7c677fe640 1 --2- 172.21.15.37:0/191864880 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7c54035e80 0x7f7c54038340 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7c680a03f0 tx=0x7f7c5c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:17.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.338+0000 7f7c6e39b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:29:17.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.338+0000 7f7c6e39b640 1 --2- 172.21.15.37:0/191864880 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c68138fa0 0x7f7c6802c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:17.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.338+0000 7f7c6e39b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:17.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.338+0000 7f7c6e39b640 1 -- 172.21.15.37:0/191864880 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f7c680a3000 con 0x7f7c68138fa0 2024-04-26T00:29:17.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.338+0000 7f7c67fff640 1 --2- 172.21.15.37:0/191864880 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c68138fa0 0x7f7c6802c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:17.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.338+0000 7f7c67fff640 10 monclient: get_auth_request con 0x7f7c68138fa0 auth_method 0 2024-04-26T00:29:17.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.338+0000 7f7c67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:17.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.338+0000 7f7c67fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:17.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.338+0000 7f7c67fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:17.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.342+0000 7f7c67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:17.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.342+0000 7f7c67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:17.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.342+0000 7f7c67fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:17.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.342+0000 7f7c67fff640 10 monclient(hunting): handle_auth_done global_id 84401 payload 824 2024-04-26T00:29:17.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.342+0000 7f7c67fff640 1 -- 172.21.15.37:0/191864880 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7c68075f10 con 0x7f7c68138fa0 2024-04-26T00:29:17.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.342+0000 7f7c67fff640 1 -- 172.21.15.37:0/191864880 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c68138fa0 msgr2=0x7f7c6802c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:17.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.342+0000 7f7c67fff640 1 --2- 172.21.15.37:0/191864880 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c68138fa0 0x7f7c6802c540 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7c58018000 tx=0x7f7c58003af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:17.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.342+0000 7f7c64ff9640 1 -- 172.21.15.37:0/191864880 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c5801ee60 con 0x7f7c68138fa0 2024-04-26T00:29:17.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.342+0000 7f7c64ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:17.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.342+0000 7f7c64ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:29:17.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.342+0000 7f7c64ff9640 10 monclient: dump: 2024-04-26T00:29:17.337 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:17.337 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:17.337 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.337 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.337 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:17.337 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:17.337 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:17.337 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:17.337 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:17.337 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:17.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.342+0000 7f7c64ff9640 1 -- 172.21.15.37:0/191864880 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f7c58006b60 con 0x7f7c68138fa0 2024-04-26T00:29:17.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.342+0000 7f7c64ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:29:17.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.342+0000 7f7c64ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:29:17.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.342+0000 7f7c64ff9640 1 -- 172.21.15.37:0/191864880 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c68138fa0 msgr2=0x7f7c6802c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:17.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.342+0000 7f7c64ff9640 1 --2- 172.21.15.37:0/191864880 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c68138fa0 0x7f7c6802c540 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7c58018000 tx=0x7f7c58003af0 comp rx=0 tx=0).stop 2024-04-26T00:29:17.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.342+0000 7f7c64ff9640 1 -- 172.21.15.37:0/191864880 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c68138fa0 msgr2=0x7f7c6802c540 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:29:17.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.342+0000 7f7c64ff9640 1 --2- 172.21.15.37:0/191864880 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c68138fa0 0x7f7c6802c540 secure :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7c58018000 tx=0x7f7c58003af0 comp rx=0 tx=0).stop 2024-04-26T00:29:17.338 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:29:17.338 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:29:17.338 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:29:17.338 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:29:17.338 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 70, 2024-04-26T00:29:17.338 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 162277, 2024-04-26T00:29:17.338 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:29:17.338 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:29:17.339 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:29:17.339 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:29:17.339 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:17.339 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 2, 2024-04-26T00:29:17.339 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:29:17.339 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:29:17.339 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:29:17.339 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:29:17.340 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:29:17.340 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:29:17.340 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:29:17.340 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:29:17.340 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:29:17.340 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:29:17.340 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:29:17.341 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:29:17.341 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:29:17.341 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:29:17.341 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:17.341 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:29:17.341 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:29:17.341 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:29:17.342 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:29:17.342 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:29:17.342 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:29:17.342 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:29:17.342 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:29:17.342 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:29:17.343 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:29:17.343 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:29:17.343 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:29:17.343 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:29:17.343 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:17.344 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:17.344 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:29:17.344 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:29:17.344 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:29:17.344 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:29:17.344 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:29:17.344 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:29:17.345 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:29:17.345 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:29:17.345 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:29:17.345 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:29:17.345 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:29:17.345 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:29:17.345 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:29:17.345 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:29:17.345 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:29:17.346 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:29:17.346 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:29:17.346 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:29:17.346 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:29:17.346 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:29:17.346 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:29:17.347 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:29:17.347 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:29:17.347 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:29:17.347 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:29:17.347 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:29:17.347 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:29:17.348 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:29:17.348 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:17.348 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:29:17.348 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:17.348 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:29:17.348 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:17.349 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:29:17.349 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:29:17.349 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:29:17.349 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:29:17.349 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:17.349 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:29:17.349 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:29:17.349 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:17.349 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:17.349 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:17.350 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:29:17.350 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:29:17.350 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:17.350 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:17.350 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:17.350 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:17.350 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:29:17.350 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:29:17.350 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:29:17.350 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:29:17.350 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:29:17.350 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:17.351 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:17.351 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:29:17.351 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:29:17.351 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:29:17.351 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:29:17.351 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:17.351 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:29:17.351 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:29:17.351 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:17.351 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:17.351 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:17.351 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:29:17.351 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:29:17.351 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:17.351 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:17.352 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:17.352 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:17.352 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:29:17.352 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:29:17.352 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:29:17.352 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:29:17.352 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:29:17.352 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:17.352 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:17.352 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:29:17.352 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:29:17.352 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:29:17.352 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:29:17.352 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:17.352 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:29:17.353 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:29:17.353 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:17.353 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:17.353 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:17.353 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:29:17.353 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:29:17.353 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:17.353 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:17.353 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:17.353 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:17.353 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:29:17.353 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:29:17.353 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:29:17.353 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:29:17.353 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:29:17.354 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:17.354 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:17.354 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:17.354 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:29:17.354 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:29:17.354 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:17.354 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:29:17.354 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:29:17.354 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:29:17.354 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:17.354 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:17.354 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:29:17.354 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:17.355 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:29:17.355 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:29:17.355 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:29:17.355 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:17.355 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:17.355 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:29:17.355 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:17.355 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:29:17.355 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:29:17.355 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:29:17.355 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:17.355 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:17.355 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:29:17.355 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:17.355 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:29:17.356 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:29:17.356 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:29:17.356 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:17.356 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:17.356 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:17.356 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:29:17.356 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:29:17.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.342+0000 7f7c6e39b640 1 -- 172.21.15.37:0/191864880 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7c54035e80 msgr2=0x7f7c54038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:17.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.342+0000 7f7c6e39b640 1 --2- 172.21.15.37:0/191864880 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7c54035e80 0x7f7c54038340 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7c680a03f0 tx=0x7f7c5c008040 comp rx=0 tx=0).stop 2024-04-26T00:29:17.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.342+0000 7f7c6e39b640 10 monclient: shutdown 2024-04-26T00:29:17.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.342+0000 7f7c6e39b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:29:17.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.342+0000 7f7c6e39b640 1 -- 172.21.15.37:0/191864880 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c6813c450 msgr2=0x7f7c68137ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:17.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.342+0000 7f7c6e39b640 1 --2- 172.21.15.37:0/191864880 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c6813c450 0x7f7c68137ec0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7c6001cd10 tx=0x7f7c60019210 comp rx=0 tx=0).stop 2024-04-26T00:29:17.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.342+0000 7f7c6c938640 1 -- 172.21.15.37:0/191864880 reap_dead start 2024-04-26T00:29:17.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.342+0000 7f7c6c938640 1 -- 172.21.15.37:0/191864880 reap_dead start 2024-04-26T00:29:17.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.342+0000 7f7c6e39b640 1 -- 172.21.15.37:0/191864880 shutdown_connections 2024-04-26T00:29:17.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.342+0000 7f7c6e39b640 1 -- 172.21.15.37:0/191864880 >> 172.21.15.37:0/191864880 conn(0x7f7c680a61b0 msgr2=0x7f7c680a7ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:29:17.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.342+0000 7f7c6e39b640 1 -- 172.21.15.37:0/191864880 shutdown_connections 2024-04-26T00:29:17.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.342+0000 7f7c6e39b640 1 -- 172.21.15.37:0/191864880 wait complete. 2024-04-26T00:29:17.359 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:29:17.359 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:29:17.359 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:29:17.423 INFO:tasks.rados.rados.0.smithi045.stdout:1604: done (2 left) 2024-04-26T00:29:17.423 INFO:tasks.rados.rados.0.smithi045.stdout:1607: done (1 left) 2024-04-26T00:29:17.423 INFO:tasks.rados.rados.0.smithi045.stdout:1608: done (0 left) 2024-04-26T00:29:17.423 INFO:tasks.rados.rados.0.smithi045.stdout:1609: read oid 39 snap -1 2024-04-26T00:29:17.423 INFO:tasks.rados.rados.0.smithi045.stdout:1609: expect (ObjNum 586 snap 182 seq_num 586) 2024-04-26T00:29:17.423 INFO:tasks.rados.rados.0.smithi045.stdout:1610: read oid 10 snap 172 2024-04-26T00:29:17.423 INFO:tasks.rados.rados.0.smithi045.stdout:1610: expect deleted 2024-04-26T00:29:17.423 INFO:tasks.rados.rados.0.smithi045.stdout:1611: snap_remove snap 181 2024-04-26T00:29:17.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f8213d54640 10 monclient: get_monmap_and_config 2024-04-26T00:29:17.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f8213d54640 10 monclient: build_initial_monmap 2024-04-26T00:29:17.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f8213d54640 10 monclient: monmap: 2024-04-26T00:29:17.483 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:29:17.483 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:17.483 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:29:17.496933+0000 2024-04-26T00:29:17.483 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:29:17.496933+0000 2024-04-26T00:29:17.483 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:29:17.483 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:17.483 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:29:17.483 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:29:17.483 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:29:17.483 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:29:17.483 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:29:17.483 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:17.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f8213d54640 1 Processor -- start 2024-04-26T00:29:17.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f8213d54640 1 -- start start 2024-04-26T00:29:17.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f8213d54640 10 monclient: init 2024-04-26T00:29:17.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f8213d54640 10 monclient: _reopen_session rank -1 2024-04-26T00:29:17.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f8213d54640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-04-26T00:29:17.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f8213d54640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f820c0a0300 0x7f820c09fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:17.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f8213d54640 10 monclient(hunting): picked mon.noname-b con 0x7f820c0a0300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:29:17.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f8213d54640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f820c0a06e0 0x7f820c134ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:17.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f8213d54640 10 monclient(hunting): picked mon.noname-a con 0x7f820c0a06e0 addr v2:172.21.15.37:3300/0 2024-04-26T00:29:17.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f8213d54640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f820c1350e0 addr v1:172.21.15.37:6789/0 2024-04-26T00:29:17.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f8213d54640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f820c09c190 con 0x7f820c1350e0 2024-04-26T00:29:17.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f8213d54640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:17.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f8213d54640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f820c0a13f0 con 0x7f820c0a06e0 2024-04-26T00:29:17.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f8213d54640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:17.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f8213d54640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f820c0a1110 con 0x7f820c0a0300 2024-04-26T00:29:17.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f8213d54640 10 monclient(hunting): _renew_subs 2024-04-26T00:29:17.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f8213d54640 10 monclient(hunting): authenticate will time out at 1538.802756s 2024-04-26T00:29:17.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f8211af0640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f820c0a0300 0x7f820c09fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:17.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f82112ef640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f820c0a06e0 0x7f820c134ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:17.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f8211af0640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f820c0a0300 0x7f820c09fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:41374/0 (socket says 172.21.15.37:41374) 2024-04-26T00:29:17.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f8211af0640 1 -- 172.21.15.37:0/2148196915 learned_addr learned my addr 172.21.15.37:0/2148196915 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:29:17.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f8211af0640 10 monclient(hunting): get_auth_request con 0x7f820c0a0300 auth_method 0 2024-04-26T00:29:17.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f8211af0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:17.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f8211af0640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:17.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f8211af0640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:17.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f82112ef640 10 monclient(hunting): get_auth_request con 0x7f820c0a06e0 auth_method 0 2024-04-26T00:29:17.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f82112ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:17.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f82112ef640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:17.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f82112ef640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:17.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f8211af0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:17.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f8211af0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:17.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f8211af0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:17.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f82112ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:17.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f82112ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:17.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f82112ef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:17.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f8211af0640 10 monclient(hunting): handle_auth_done global_id 84404 payload 274 2024-04-26T00:29:17.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f8211af0640 1 -- 172.21.15.37:0/2148196915 >> v2:172.21.15.37:3300/0 conn(0x7f820c0a06e0 msgr2=0x7f820c134ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:29:17.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f8211af0640 1 --2- 172.21.15.37:0/2148196915 >> v2:172.21.15.37:3300/0 conn(0x7f820c0a06e0 0x7f820c134ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:17.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f8211af0640 1 -- 172.21.15.37:0/2148196915 >> v1:172.21.15.37:6789/0 conn(0x7f820c1350e0 legacy=0x7f820c13bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:29:17.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f8211af0640 10 monclient: _finish_hunting 0 2024-04-26T00:29:17.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f8211af0640 1 monclient: found mon.noname-b 2024-04-26T00:29:17.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f8211af0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:29:17.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f8211af0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:29:17.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f8211af0640 1 -- 172.21.15.37:0/2148196915 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f820c0a16d0 con 0x7f820c0a0300 2024-04-26T00:29:17.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.494+0000 7f82112ef640 1 --2- 172.21.15.37:0/2148196915 >> v2:172.21.15.37:3300/0 conn(0x7f820c0a06e0 0x7f820c134ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:29:17.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.498+0000 7f8211af0640 1 -- 172.21.15.37:0/2148196915 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f820c0a0300 msgr2=0x7f820c09fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:17.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.498+0000 7f8211af0640 1 --2- 172.21.15.37:0/2148196915 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f820c0a0300 0x7f820c09fc50 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8208009870 tx=0x7f820801c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6695c41e4aa7bc24 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:17.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.498+0000 7f8210aee640 1 -- 172.21.15.37:0/2148196915 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82080b0070 con 0x7f820c0a0300 2024-04-26T00:29:17.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.498+0000 7f8210aee640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:17.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.498+0000 7f8210aee640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:29:17.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.498+0000 7f8210aee640 10 monclient: dump: 2024-04-26T00:29:17.491 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:17.491 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:17.491 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.491 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.491 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:17.491 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:17.491 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:17.491 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:17.491 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:17.491 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:17.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.498+0000 7f8210aee640 10 monclient: _finish_auth 0 2024-04-26T00:29:17.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.498+0000 7f8210aee640 10 monclient: _check_auth_tickets 2024-04-26T00:29:17.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.498+0000 7f8210aee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:29:17.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.498+0000 7f8210aee640 1 -- 172.21.15.37:0/2148196915 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82080ab600 con 0x7f820c0a0300 2024-04-26T00:29:17.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.498+0000 7f8210aee640 10 monclient: handle_config config(0 keys) 2024-04-26T00:29:17.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.498+0000 7f8210aee640 1 -- 172.21.15.37:0/2148196915 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82080ab9d0 con 0x7f820c0a0300 2024-04-26T00:29:17.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.498+0000 7f8213d54640 5 monclient: authenticate success, global_id 84404 2024-04-26T00:29:17.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.498+0000 7f8213d54640 10 monclient: get_monmap_and_config success 2024-04-26T00:29:17.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.498+0000 7f8213d54640 4 set_mon_vals no callback set 2024-04-26T00:29:17.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.498+0000 7f8210aee640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:17.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.498+0000 7f8210aee640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:29:17.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.498+0000 7f8210aee640 10 monclient: dump: 2024-04-26T00:29:17.492 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:17.492 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:17.492 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.492 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.492 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:17.493 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:17.493 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:17.493 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:17.493 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:17.493 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:17.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8213d54640 10 monclient: shutdown 2024-04-26T00:29:17.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8213d54640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:29:17.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8213d54640 1 -- 172.21.15.37:0/2148196915 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f820c0a0300 msgr2=0x7f820c09fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:17.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8213d54640 1 --2- 172.21.15.37:0/2148196915 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f820c0a0300 0x7f820c09fc50 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8208009870 tx=0x7f820801c410 comp rx=0 tx=0).stop 2024-04-26T00:29:17.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8213d54640 1 -- 172.21.15.37:0/2148196915 shutdown_connections 2024-04-26T00:29:17.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8213d54640 1 --2- 172.21.15.37:0/2148196915 >> v2:172.21.15.37:3300/0 conn(0x7f820c0a06e0 0x7f820c134ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:17.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8213d54640 1 --2- 172.21.15.37:0/2148196915 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f820c0a0300 0x7f820c09fc50 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:17.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8213d54640 1 -- 172.21.15.37:0/2148196915 >> 172.21.15.37:0/2148196915 conn(0x7f820c0a1d50 msgr2=0x7f820c0a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:29:17.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8213d54640 1 -- 172.21.15.37:0/2148196915 shutdown_connections 2024-04-26T00:29:17.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8213d54640 1 -- 172.21.15.37:0/2148196915 wait complete. 2024-04-26T00:29:17.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8213d54640 10 monclient: build_initial_monmap 2024-04-26T00:29:17.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8213d54640 10 monclient: monmap: 2024-04-26T00:29:17.494 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:29:17.494 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:29:17.494 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:29:17.494 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:29:17.494 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:29:17.494 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:29:17.494 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:29:17.494 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:29:17.494 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:29:17.494 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:17.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8213d54640 1 Processor -- start 2024-04-26T00:29:17.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8213d54640 1 -- start start 2024-04-26T00:29:17.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8213d54640 10 monclient: init 2024-04-26T00:29:17.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8213d54640 10 monclient: _reopen_session rank -1 2024-04-26T00:29:17.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8213d54640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:29:17.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8213d54640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f820c0a0300 0x7f820c1cd490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:17.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8213d54640 10 monclient(hunting): picked mon.noname-a con 0x7f820c0a0300 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:29:17.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8213d54640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f820c0a06e0 0x7f820c13bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:17.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8213d54640 10 monclient(hunting): picked mon.noname-c con 0x7f820c0a06e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:29:17.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8213d54640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f820c1350e0 0x7f820c13c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:17.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8213d54640 10 monclient(hunting): picked mon.noname-b con 0x7f820c1350e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:29:17.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8213d54640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:17.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8213d54640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f820c0a0c40 con 0x7f820c0a0300 2024-04-26T00:29:17.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8213d54640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:17.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8213d54640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f820c0a0ac0 con 0x7f820c1350e0 2024-04-26T00:29:17.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8213d54640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:17.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8213d54640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f820c13fe10 con 0x7f820c0a06e0 2024-04-26T00:29:17.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8213d54640 10 monclient(hunting): _renew_subs 2024-04-26T00:29:17.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8213d54640 10 monclient(hunting): authenticate will time out at 1538.810522s 2024-04-26T00:29:17.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8211af0640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f820c0a0300 0x7f820c1cd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:17.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8211af0640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f820c0a0300 0x7f820c1cd490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33868/0 (socket says 172.21.15.37:33868) 2024-04-26T00:29:17.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8211af0640 1 -- 172.21.15.37:0/1061333949 learned_addr learned my addr 172.21.15.37:0/1061333949 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:29:17.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8211af0640 10 monclient(hunting): get_auth_request con 0x7f820c0a0300 auth_method 0 2024-04-26T00:29:17.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8211af0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:17.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8211af0640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:17.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8211af0640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:17.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f82112ef640 1 --2- 172.21.15.37:0/1061333949 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f820c0a06e0 0x7f820c13bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:17.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f82112ef640 10 monclient(hunting): get_auth_request con 0x7f820c0a06e0 auth_method 0 2024-04-26T00:29:17.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f82112ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:17.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f82112ef640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:17.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f82112ef640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:17.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8211af0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:17.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8211af0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:17.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8211af0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:17.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f82112ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:17.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f82112ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:17.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f82112ef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:17.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8211af0640 10 monclient(hunting): handle_auth_done global_id 94347 payload 824 2024-04-26T00:29:17.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8211af0640 1 -- 172.21.15.37:0/1061333949 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f820c0a06e0 msgr2=0x7f820c13bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:17.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8211af0640 1 --2- 172.21.15.37:0/1061333949 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f820c0a06e0 0x7f820c13bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:17.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8211af0640 1 -- 172.21.15.37:0/1061333949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f820c1350e0 msgr2=0x7f820c13c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:17.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8211af0640 1 --2- 172.21.15.37:0/1061333949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f820c1350e0 0x7f820c13c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:17.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8211af0640 10 monclient: _finish_hunting 0 2024-04-26T00:29:17.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8211af0640 1 monclient: found mon.noname-a 2024-04-26T00:29:17.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8211af0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:29:17.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8211af0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:29:17.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f8211af0640 1 -- 172.21.15.37:0/1061333949 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f820c09f550 con 0x7f820c0a0300 2024-04-26T00:29:17.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.502+0000 7f82112ef640 1 --2- 172.21.15.37:0/1061333949 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f820c0a06e0 0x7f820c13bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:29:17.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f8211af0640 1 -- 172.21.15.37:0/1061333949 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f820c0a0300 msgr2=0x7f820c1cd490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:17.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f8211af0640 1 --2- 172.21.15.37:0/1061333949 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f820c0a0300 0x7f820c1cd490 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f82080ae040 tx=0x7f8208019210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:17.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f82027fc640 1 -- 172.21.15.37:0/1061333949 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82080b0040 con 0x7f820c0a0300 2024-04-26T00:29:17.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f82027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:17.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f82027fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:29:17.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f82027fc640 10 monclient: dump: 2024-04-26T00:29:17.502 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:17.502 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:17.502 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.502 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.502 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:17.502 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:17.503 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:17.503 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:17.503 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:17.503 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:17.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f82027fc640 10 monclient: _finish_auth 0 2024-04-26T00:29:17.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f82027fc640 10 monclient: _check_auth_tickets 2024-04-26T00:29:17.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f82027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:29:17.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f82027fc640 1 -- 172.21.15.37:0/1061333949 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82080aa5b0 con 0x7f820c0a0300 2024-04-26T00:29:17.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f8213d54640 5 monclient: authenticate success, global_id 94347 2024-04-26T00:29:17.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f8213d54640 10 monclient: _renew_subs 2024-04-26T00:29:17.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f8213d54640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:29:17.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f8213d54640 1 -- 172.21.15.37:0/1061333949 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f820c138070 con 0x7f820c0a0300 2024-04-26T00:29:17.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f82027fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:29:17.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f82027fc640 1 -- 172.21.15.37:0/1061333949 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82080aa9a0 con 0x7f820c0a0300 2024-04-26T00:29:17.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f82027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:17.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f82027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:29:17.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f82027fc640 10 monclient: dump: 2024-04-26T00:29:17.504 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:17.504 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:17.504 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.504 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.504 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:17.504 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:17.504 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:17.504 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:17.504 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:17.504 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:17.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f8203fff640 4 set_mon_vals no callback set 2024-04-26T00:29:17.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f8213d54640 10 monclient: _renew_subs 2024-04-26T00:29:17.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f8213d54640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:29:17.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f8213d54640 1 -- 172.21.15.37:0/1061333949 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f820c06e660 con 0x7f820c0a0300 2024-04-26T00:29:17.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f8213d54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:29:17.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f8213d54640 1 --2- 172.21.15.37:0/1061333949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f820c135c60 0x7f820c136040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:17.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f8213d54640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:17.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f8213d54640 1 -- 172.21.15.37:0/1061333949 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f820c1392f0 con 0x7f820c135c60 2024-04-26T00:29:17.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f82027fc640 1 -- 172.21.15.37:0/1061333949 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f82080aab40 con 0x7f820c0a0300 2024-04-26T00:29:17.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f82027fc640 1 --2- 172.21.15.37:0/1061333949 >> v2:172.21.15.45:6812/2379918623 conn(0x7f81f0035e80 0x7f81f0038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:17.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f82112ef640 1 --2- 172.21.15.37:0/1061333949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f820c135c60 0x7f820c136040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:17.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f82027fc640 1 -- 172.21.15.37:0/1061333949 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(369..369 src has 177..369) ==== 3931+0+0 (secure 0 0 0) 0x7f82080e26f0 con 0x7f820c0a0300 2024-04-26T00:29:17.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f82112ef640 10 monclient: get_auth_request con 0x7f820c135c60 auth_method 0 2024-04-26T00:29:17.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f82112ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:17.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f82112ef640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:17.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f82112ef640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:17.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f82112ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:17.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f82112ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:17.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f82112ef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:17.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f82122f1640 1 --2- 172.21.15.37:0/1061333949 >> v2:172.21.15.45:6812/2379918623 conn(0x7f81f0035e80 0x7f81f0038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:17.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f82122f1640 10 monclient: get_auth_request con 0x7f81f0035e80 auth_method 0 2024-04-26T00:29:17.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f82112ef640 10 monclient(hunting): handle_auth_done global_id 114109 payload 824 2024-04-26T00:29:17.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.506+0000 7f82112ef640 1 -- 172.21.15.37:0/1061333949 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f820c04f510 con 0x7f820c135c60 2024-04-26T00:29:17.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.510+0000 7f82112ef640 1 -- 172.21.15.37:0/1061333949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f820c135c60 msgr2=0x7f820c136040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:17.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.510+0000 7f82112ef640 1 --2- 172.21.15.37:0/1061333949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f820c135c60 0x7f820c136040 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f820c1373b0 tx=0x7f81fc00f610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:17.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.510+0000 7f82122f1640 1 --2- 172.21.15.37:0/1061333949 >> v2:172.21.15.45:6812/2379918623 conn(0x7f81f0035e80 0x7f81f0038340 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f820c1ce310 tx=0x7f8204007d40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:17.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.510+0000 7f82027fc640 1 -- 172.21.15.37:0/1061333949 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81fc019070 con 0x7f820c135c60 2024-04-26T00:29:17.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.510+0000 7f82027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:17.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.510+0000 7f82027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:29:17.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.510+0000 7f82027fc640 10 monclient: dump: 2024-04-26T00:29:17.508 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:17.508 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:17.508 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.508 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.508 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:17.508 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:17.508 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:17.509 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:17.509 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:17.509 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:17.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.510+0000 7f82027fc640 1 -- 172.21.15.37:0/1061333949 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f81fc00fe40 con 0x7f820c135c60 2024-04-26T00:29:17.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.510+0000 7f82027fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:29:17.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.510+0000 7f82027fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:29:17.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.510+0000 7f82027fc640 1 -- 172.21.15.37:0/1061333949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f820c135c60 msgr2=0x7f820c136040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:17.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.510+0000 7f82027fc640 1 --2- 172.21.15.37:0/1061333949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f820c135c60 0x7f820c136040 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f820c1373b0 tx=0x7f81fc00f610 comp rx=0 tx=0).stop 2024-04-26T00:29:17.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.510+0000 7f82027fc640 1 -- 172.21.15.37:0/1061333949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f820c135c60 msgr2=0x7f820c136040 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:29:17.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.510+0000 7f82027fc640 1 --2- 172.21.15.37:0/1061333949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f820c135c60 0x7f820c136040 secure :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f820c1373b0 tx=0x7f81fc00f610 comp rx=0 tx=0).stop 2024-04-26T00:29:17.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.518+0000 7f8213d54640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:29:17.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.518+0000 7f8213d54640 1 --2- 172.21.15.37:0/1061333949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f820c136580 0x7f820c070640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:17.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.518+0000 7f8213d54640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:17.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.518+0000 7f8213d54640 1 -- 172.21.15.37:0/1061333949 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f820c1398a0 con 0x7f820c136580 2024-04-26T00:29:17.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.518+0000 7f82112ef640 1 --2- 172.21.15.37:0/1061333949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f820c136580 0x7f820c070640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:17.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.518+0000 7f82112ef640 10 monclient: get_auth_request con 0x7f820c136580 auth_method 0 2024-04-26T00:29:17.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.518+0000 7f82112ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:17.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.518+0000 7f82112ef640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:17.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.518+0000 7f82112ef640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:17.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.518+0000 7f82112ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:17.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.518+0000 7f82112ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:17.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.518+0000 7f82112ef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:17.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.518+0000 7f82112ef640 10 monclient(hunting): handle_auth_done global_id 114112 payload 824 2024-04-26T00:29:17.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.518+0000 7f82112ef640 1 -- 172.21.15.37:0/1061333949 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f820c09eb20 con 0x7f820c136580 2024-04-26T00:29:17.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.522+0000 7f82112ef640 1 -- 172.21.15.37:0/1061333949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f820c136580 msgr2=0x7f820c070640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:17.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.522+0000 7f82112ef640 1 --2- 172.21.15.37:0/1061333949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f820c136580 0x7f820c070640 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f81fc01b000 tx=0x7f81fc0046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:17.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.522+0000 7f82027fc640 1 -- 172.21.15.37:0/1061333949 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81fc019040 con 0x7f820c136580 2024-04-26T00:29:17.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.522+0000 7f82027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:17.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.522+0000 7f82027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:29:17.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.522+0000 7f82027fc640 10 monclient: dump: 2024-04-26T00:29:17.512 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:17.512 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:17.512 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.512 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.512 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:17.512 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:17.512 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:17.512 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:17.512 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:17.512 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:17.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.522+0000 7f82027fc640 1 -- 172.21.15.37:0/1061333949 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4715 (secure 0 0 0) 0x7f81fc01e870 con 0x7f820c136580 2024-04-26T00:29:17.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.522+0000 7f82027fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:29:17.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.522+0000 7f82027fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:29:17.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.522+0000 7f82027fc640 1 -- 172.21.15.37:0/1061333949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f820c136580 msgr2=0x7f820c070640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:17.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.522+0000 7f82027fc640 1 --2- 172.21.15.37:0/1061333949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f820c136580 0x7f820c070640 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f81fc01b000 tx=0x7f81fc0046d0 comp rx=0 tx=0).stop 2024-04-26T00:29:17.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.522+0000 7f82027fc640 1 -- 172.21.15.37:0/1061333949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f820c136580 msgr2=0x7f820c070640 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:29:17.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.522+0000 7f82027fc640 1 --2- 172.21.15.37:0/1061333949 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f820c136580 0x7f820c070640 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:17.513 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:29:17.513 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:29:17.513 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:29:17.513 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:29:17.513 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 70, 2024-04-26T00:29:17.513 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 7607, 2024-04-26T00:29:17.513 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:29:17.514 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:29:17.514 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:29:17.514 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:29:17.514 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:17.514 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 2, 2024-04-26T00:29:17.514 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:29:17.514 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:29:17.514 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:29:17.514 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:29:17.514 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:29:17.514 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:29:17.514 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:29:17.514 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:29:17.515 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:29:17.515 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:29:17.515 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:29:17.515 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:29:17.515 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:29:17.515 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:29:17.515 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:17.515 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:29:17.515 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:29:17.515 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:29:17.515 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:29:17.515 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:29:17.515 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:29:17.515 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:29:17.516 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:29:17.516 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:29:17.516 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:29:17.516 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:29:17.516 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:29:17.516 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:29:17.516 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:17.516 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:17.516 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:29:17.516 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:29:17.516 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:29:17.516 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:29:17.516 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:29:17.516 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:29:17.516 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:29:17.517 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:29:17.517 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:29:17.517 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:29:17.517 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:29:17.517 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:29:17.517 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:29:17.517 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:29:17.517 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:29:17.517 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:29:17.517 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:29:17.517 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:29:17.517 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:29:17.517 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:29:17.517 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:29:17.518 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:29:17.518 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:29:17.518 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:29:17.518 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:29:17.518 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:29:17.518 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:29:17.518 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:29:17.518 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:17.518 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:29:17.518 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:17.518 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:29:17.518 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:17.518 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:29:17.518 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:29:17.519 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:29:17.519 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:29:17.519 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:17.519 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:29:17.519 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:29:17.519 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:17.519 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:17.519 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:17.519 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:29:17.519 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:29:17.519 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:17.519 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:17.519 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:17.519 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:17.520 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:29:17.520 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:29:17.520 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:29:17.520 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:29:17.520 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:29:17.520 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:17.520 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:17.520 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:29:17.520 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:29:17.520 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:29:17.520 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:29:17.520 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:17.520 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:29:17.520 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:29:17.520 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:17.521 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:17.521 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:17.521 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:29:17.521 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:29:17.521 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:17.521 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:17.521 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:17.521 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:17.521 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:29:17.521 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:29:17.521 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:29:17.521 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:29:17.521 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:29:17.521 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:17.522 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:17.522 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:29:17.522 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:29:17.522 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:29:17.522 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:29:17.522 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:17.522 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:29:17.522 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:29:17.522 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:17.522 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:17.522 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:17.522 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:29:17.522 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:29:17.522 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:17.523 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:17.523 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:17.523 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:17.523 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:29:17.523 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:29:17.523 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:29:17.523 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:29:17.523 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:29:17.523 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:17.523 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:17.523 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:17.523 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:29:17.523 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:29:17.523 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:17.524 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:29:17.524 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:29:17.524 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:29:17.524 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:17.524 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:17.524 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:29:17.524 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:17.524 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:29:17.524 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:29:17.524 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:29:17.524 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:17.524 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:17.524 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:29:17.525 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:17.525 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:29:17.525 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:29:17.525 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:29:17.525 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:17.525 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:17.525 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:17.525 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:29:17.525 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:29:17.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.522+0000 7f8213d54640 1 -- 172.21.15.37:0/1061333949 >> v2:172.21.15.45:6812/2379918623 conn(0x7f81f0035e80 msgr2=0x7f81f0038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:17.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.522+0000 7f8213d54640 1 --2- 172.21.15.37:0/1061333949 >> v2:172.21.15.45:6812/2379918623 conn(0x7f81f0035e80 0x7f81f0038340 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f820c1ce310 tx=0x7f8204007d40 comp rx=0 tx=0).stop 2024-04-26T00:29:17.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.522+0000 7f8213d54640 10 monclient: shutdown 2024-04-26T00:29:17.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.522+0000 7f8213d54640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:29:17.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.522+0000 7f8213d54640 1 -- 172.21.15.37:0/1061333949 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f820c0a0300 msgr2=0x7f820c1cd490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:17.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.522+0000 7f8213d54640 1 --2- 172.21.15.37:0/1061333949 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f820c0a0300 0x7f820c1cd490 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f82080ae040 tx=0x7f8208019210 comp rx=0 tx=0).stop 2024-04-26T00:29:17.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.522+0000 7f82122f1640 1 -- 172.21.15.37:0/1061333949 reap_dead start 2024-04-26T00:29:17.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.522+0000 7f82122f1640 1 -- 172.21.15.37:0/1061333949 reap_dead start 2024-04-26T00:29:17.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.522+0000 7f8213d54640 1 -- 172.21.15.37:0/1061333949 shutdown_connections 2024-04-26T00:29:17.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.522+0000 7f8213d54640 1 -- 172.21.15.37:0/1061333949 >> 172.21.15.37:0/1061333949 conn(0x7f820c0a1d50 msgr2=0x7f820c130b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:29:17.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.522+0000 7f8213d54640 1 -- 172.21.15.37:0/1061333949 shutdown_connections 2024-04-26T00:29:17.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.522+0000 7f8213d54640 1 -- 172.21.15.37:0/1061333949 wait complete. 2024-04-26T00:29:17.528 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:29:17.528 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-26T00:29:17.528 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-26T00:29:17.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.658+0000 7f5b44247640 10 monclient: get_monmap_and_config 2024-04-26T00:29:17.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.658+0000 7f5b44247640 10 monclient: build_initial_monmap 2024-04-26T00:29:17.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.658+0000 7f5b44247640 10 monclient: monmap: 2024-04-26T00:29:17.649 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:29:17.649 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:17.649 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:29:17.663432+0000 2024-04-26T00:29:17.649 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:29:17.663432+0000 2024-04-26T00:29:17.649 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:29:17.649 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:17.650 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:29:17.650 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:29:17.650 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:29:17.650 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:29:17.650 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:29:17.650 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:17.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.662+0000 7f5b44247640 1 Processor -- start 2024-04-26T00:29:17.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.662+0000 7f5b44247640 1 -- start start 2024-04-26T00:29:17.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.662+0000 7f5b44247640 10 monclient: init 2024-04-26T00:29:17.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.662+0000 7f5b44247640 10 monclient: _reopen_session rank -1 2024-04-26T00:29:17.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.662+0000 7f5b44247640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-04-26T00:29:17.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.662+0000 7f5b44247640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5b3c136e10 0x7f5b3c09c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:17.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.662+0000 7f5b44247640 10 monclient(hunting): picked mon.noname-c con 0x7f5b3c136e10 addr v2:172.21.15.45:3300/0 2024-04-26T00:29:17.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.662+0000 7f5b44247640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f5b3c1371f0 0x7f5b3c137600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:17.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.662+0000 7f5b44247640 10 monclient(hunting): picked mon.noname-a con 0x7f5b3c1371f0 addr v2:172.21.15.37:3300/0 2024-04-26T00:29:17.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.662+0000 7f5b44247640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5b3c135d90 addr v1:172.21.15.45:6789/0 2024-04-26T00:29:17.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.662+0000 7f5b44247640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b3c09b680 con 0x7f5b3c135d90 2024-04-26T00:29:17.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.662+0000 7f5b44247640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:17.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.662+0000 7f5b44247640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f5b3c09b8f0 con 0x7f5b3c1371f0 2024-04-26T00:29:17.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.662+0000 7f5b44247640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:17.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.662+0000 7f5b44247640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f5b3c09bbd0 con 0x7f5b3c136e10 2024-04-26T00:29:17.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.662+0000 7f5b44247640 10 monclient(hunting): _renew_subs 2024-04-26T00:29:17.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.662+0000 7f5b44247640 10 monclient(hunting): authenticate will time out at 1538.969092s 2024-04-26T00:29:17.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.662+0000 7f5b417e2640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f5b3c1371f0 0x7f5b3c137600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:17.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.662+0000 7f5b417e2640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f5b3c1371f0 0x7f5b3c137600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33880/0 (socket says 172.21.15.37:33880) 2024-04-26T00:29:17.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.662+0000 7f5b417e2640 1 -- 172.21.15.37:0/2446482367 learned_addr learned my addr 172.21.15.37:0/2446482367 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:29:17.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.662+0000 7f5b417e2640 10 monclient(hunting): get_auth_request con 0x7f5b3c1371f0 auth_method 0 2024-04-26T00:29:17.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.662+0000 7f5b417e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:17.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.662+0000 7f5b417e2640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:17.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.662+0000 7f5b417e2640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:17.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.662+0000 7f5b417e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:17.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.662+0000 7f5b417e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:17.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.662+0000 7f5b417e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:17.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.662+0000 7f5b41fe3640 1 --2- 172.21.15.37:0/2446482367 >> v2:172.21.15.45:3300/0 conn(0x7f5b3c136e10 0x7f5b3c09c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:17.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.662+0000 7f5b417e2640 10 monclient(hunting): handle_auth_done global_id 94350 payload 274 2024-04-26T00:29:17.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.662+0000 7f5b417e2640 1 -- 172.21.15.37:0/2446482367 >> v2:172.21.15.45:3300/0 conn(0x7f5b3c136e10 msgr2=0x7f5b3c09c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:29:17.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.662+0000 7f5b417e2640 1 --2- 172.21.15.37:0/2446482367 >> v2:172.21.15.45:3300/0 conn(0x7f5b3c136e10 0x7f5b3c09c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:17.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.662+0000 7f5b417e2640 1 -- 172.21.15.37:0/2446482367 >> v1:172.21.15.45:6789/0 conn(0x7f5b3c135d90 legacy=0x7f5b3c136220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:29:17.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.662+0000 7f5b417e2640 10 monclient: _finish_hunting 0 2024-04-26T00:29:17.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.662+0000 7f5b417e2640 1 monclient: found mon.noname-a 2024-04-26T00:29:17.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.662+0000 7f5b417e2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:29:17.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.662+0000 7f5b417e2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:29:17.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.662+0000 7f5b417e2640 1 -- 172.21.15.37:0/2446482367 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b3c137b40 con 0x7f5b3c1371f0 2024-04-26T00:29:17.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.662+0000 7f5b41fe3640 10 monclient: get_auth_request con 0x7f5b3c136e10 auth_method 0 2024-04-26T00:29:17.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.662+0000 7f5b41fe3640 1 --2- 172.21.15.37:0/2446482367 >> v2:172.21.15.45:3300/0 conn(0x7f5b3c136e10 0x7f5b3c09c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:29:17.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.666+0000 7f5b417e2640 1 -- 172.21.15.37:0/2446482367 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b3c1371f0 msgr2=0x7f5b3c137600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:17.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.666+0000 7f5b417e2640 1 --2- 172.21.15.37:0/2446482367 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b3c1371f0 0x7f5b3c137600 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c0a5b00 tx=0x7f5b2c01c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7e31e8f404dc254 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:17.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.666+0000 7f5b40fe1640 1 -- 172.21.15.37:0/2446482367 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b2c0b1070 con 0x7f5b3c1371f0 2024-04-26T00:29:17.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.666+0000 7f5b40fe1640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:17.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.666+0000 7f5b40fe1640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:29:17.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.666+0000 7f5b40fe1640 10 monclient: dump: 2024-04-26T00:29:17.657 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:17.657 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:17.657 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.657 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.657 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:17.657 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:17.657 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:17.657 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:17.657 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:17.657 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:17.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.666+0000 7f5b40fe1640 10 monclient: _finish_auth 0 2024-04-26T00:29:17.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.666+0000 7f5b40fe1640 10 monclient: _check_auth_tickets 2024-04-26T00:29:17.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.666+0000 7f5b40fe1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:29:17.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.666+0000 7f5b40fe1640 1 -- 172.21.15.37:0/2446482367 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b2c0ac620 con 0x7f5b3c1371f0 2024-04-26T00:29:17.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.666+0000 7f5b44247640 5 monclient: authenticate success, global_id 94350 2024-04-26T00:29:17.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.666+0000 7f5b44247640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:29:17.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.666+0000 7f5b40fe1640 10 monclient: handle_config config(0 keys) 2024-04-26T00:29:17.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.666+0000 7f5b44247640 10 monclient: get_monmap_and_config success 2024-04-26T00:29:17.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.666+0000 7f5b44247640 4 set_mon_vals no callback set 2024-04-26T00:29:17.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.666+0000 7f5b40fe1640 1 -- 172.21.15.37:0/2446482367 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b2c0ac9f0 con 0x7f5b3c1371f0 2024-04-26T00:29:17.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.666+0000 7f5b40fe1640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:17.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.666+0000 7f5b40fe1640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:29:17.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.666+0000 7f5b40fe1640 10 monclient: dump: 2024-04-26T00:29:17.658 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:17.658 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:17.659 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.659 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.659 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:17.659 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:17.659 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:17.659 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:17.659 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:17.659 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:17.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.666+0000 7f5b44247640 10 monclient: shutdown 2024-04-26T00:29:17.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.666+0000 7f5b44247640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:29:17.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.666+0000 7f5b44247640 1 -- 172.21.15.37:0/2446482367 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b3c1371f0 msgr2=0x7f5b3c137600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:17.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.666+0000 7f5b44247640 1 --2- 172.21.15.37:0/2446482367 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b3c1371f0 0x7f5b3c137600 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c0a5b00 tx=0x7f5b2c01c3a0 comp rx=0 tx=0).stop 2024-04-26T00:29:17.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.666+0000 7f5b44247640 1 -- 172.21.15.37:0/2446482367 shutdown_connections 2024-04-26T00:29:17.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.666+0000 7f5b44247640 1 --2- 172.21.15.37:0/2446482367 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b3c1371f0 0x7f5b3c137600 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:17.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.666+0000 7f5b44247640 1 --2- 172.21.15.37:0/2446482367 >> v2:172.21.15.45:3300/0 conn(0x7f5b3c136e10 0x7f5b3c09c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:17.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.666+0000 7f5b44247640 1 -- 172.21.15.37:0/2446482367 >> 172.21.15.37:0/2446482367 conn(0x7f5b3c0a64e0 msgr2=0x7f5b3c0a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:29:17.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.666+0000 7f5b44247640 1 -- 172.21.15.37:0/2446482367 shutdown_connections 2024-04-26T00:29:17.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.666+0000 7f5b44247640 1 -- 172.21.15.37:0/2446482367 wait complete. 2024-04-26T00:29:17.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.666+0000 7f5b44247640 10 monclient: build_initial_monmap 2024-04-26T00:29:17.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.666+0000 7f5b44247640 10 monclient: monmap: 2024-04-26T00:29:17.660 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:29:17.660 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:29:17.660 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:29:17.660 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:29:17.660 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:29:17.660 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:29:17.660 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:29:17.660 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:29:17.660 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:29:17.661 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:17.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.666+0000 7f5b44247640 1 Processor -- start 2024-04-26T00:29:17.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.666+0000 7f5b44247640 1 -- start start 2024-04-26T00:29:17.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.666+0000 7f5b44247640 10 monclient: init 2024-04-26T00:29:17.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b44247640 10 monclient: _reopen_session rank -1 2024-04-26T00:29:17.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b44247640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:29:17.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b44247640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b3c135d90 0x7f5b3c1cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:17.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b44247640 10 monclient(hunting): picked mon.noname-a con 0x7f5b3c135d90 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:29:17.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b44247640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b3c136e10 0x7f5b3c134f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:17.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b44247640 10 monclient(hunting): picked mon.noname-b con 0x7f5b3c136e10 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:29:17.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b44247640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b3c1371f0 0x7f5b3c13adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:17.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b44247640 10 monclient(hunting): picked mon.noname-c con 0x7f5b3c1371f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:29:17.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b44247640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:17.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b44247640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f5b3c141f10 con 0x7f5b3c135d90 2024-04-26T00:29:17.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b44247640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:17.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b44247640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5b3c141d90 con 0x7f5b3c136e10 2024-04-26T00:29:17.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b44247640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:17.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b44247640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5b3c142090 con 0x7f5b3c1371f0 2024-04-26T00:29:17.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b44247640 10 monclient(hunting): _renew_subs 2024-04-26T00:29:17.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b44247640 10 monclient(hunting): authenticate will time out at 1538.976340s 2024-04-26T00:29:17.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b41fe3640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b3c135d90 0x7f5b3c1cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:17.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b427e4640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b3c1371f0 0x7f5b3c13adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:17.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b41fe3640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b3c135d90 0x7f5b3c1cf4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33890/0 (socket says 172.21.15.37:33890) 2024-04-26T00:29:17.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b41fe3640 1 -- 172.21.15.37:0/3447676132 learned_addr learned my addr 172.21.15.37:0/3447676132 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:29:17.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b41fe3640 10 monclient(hunting): get_auth_request con 0x7f5b3c135d90 auth_method 0 2024-04-26T00:29:17.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b41fe3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:17.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b41fe3640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:17.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b41fe3640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:17.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b427e4640 10 monclient(hunting): get_auth_request con 0x7f5b3c1371f0 auth_method 0 2024-04-26T00:29:17.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b427e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:17.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b427e4640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:17.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b427e4640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:17.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b41fe3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:17.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b41fe3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:17.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b41fe3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:17.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b427e4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:17.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b427e4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:17.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b427e4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:17.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b417e2640 1 --2- 172.21.15.37:0/3447676132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b3c136e10 0x7f5b3c134f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:17.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b417e2640 10 monclient(hunting): get_auth_request con 0x7f5b3c136e10 auth_method 0 2024-04-26T00:29:17.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b417e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:17.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b417e2640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:17.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b417e2640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:17.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b41fe3640 10 monclient(hunting): handle_auth_done global_id 94353 payload 824 2024-04-26T00:29:17.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b41fe3640 1 -- 172.21.15.37:0/3447676132 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b3c1371f0 msgr2=0x7f5b3c13adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:17.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b41fe3640 1 --2- 172.21.15.37:0/3447676132 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b3c1371f0 0x7f5b3c13adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:17.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b41fe3640 1 -- 172.21.15.37:0/3447676132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b3c136e10 msgr2=0x7f5b3c134f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:17.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b41fe3640 1 --2- 172.21.15.37:0/3447676132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b3c136e10 0x7f5b3c134f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:17.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b41fe3640 10 monclient: _finish_hunting 0 2024-04-26T00:29:17.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b41fe3640 1 monclient: found mon.noname-a 2024-04-26T00:29:17.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b41fe3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:29:17.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b41fe3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:29:17.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b41fe3640 1 -- 172.21.15.37:0/3447676132 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b3c1d03c0 con 0x7f5b3c135d90 2024-04-26T00:29:17.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b427e4640 1 --2- 172.21.15.37:0/3447676132 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b3c1371f0 0x7f5b3c13adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:29:17.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b41fe3640 1 -- 172.21.15.37:0/3447676132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b3c135d90 msgr2=0x7f5b3c1cf4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:17.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b41fe3640 1 --2- 172.21.15.37:0/3447676132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b3c135d90 0x7f5b3c1cf4d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5b3800c7f0 tx=0x7f5b3800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:17.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b327fc640 1 -- 172.21.15.37:0/3447676132 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b38007c40 con 0x7f5b3c135d90 2024-04-26T00:29:17.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:17.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b327fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:29:17.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b327fc640 10 monclient: dump: 2024-04-26T00:29:17.669 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:17.669 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:17.669 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.669 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.670 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:17.670 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:17.670 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:17.670 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:17.670 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:17.670 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:17.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b327fc640 10 monclient: _finish_auth 0 2024-04-26T00:29:17.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b327fc640 10 monclient: _check_auth_tickets 2024-04-26T00:29:17.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:29:17.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b327fc640 1 -- 172.21.15.37:0/3447676132 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b38007de0 con 0x7f5b3c135d90 2024-04-26T00:29:17.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b327fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:29:17.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b327fc640 1 -- 172.21.15.37:0/3447676132 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b38011630 con 0x7f5b3c135d90 2024-04-26T00:29:17.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:17.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b327fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:29:17.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b327fc640 10 monclient: dump: 2024-04-26T00:29:17.671 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:17.671 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:17.671 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.671 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:17.671 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:17.671 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:17.671 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:17.671 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:17.671 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:17.671 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:17.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b337fe640 4 set_mon_vals no callback set 2024-04-26T00:29:17.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b44247640 5 monclient: authenticate success, global_id 94353 2024-04-26T00:29:17.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b44247640 10 monclient: _renew_subs 2024-04-26T00:29:17.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b44247640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:29:17.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b44247640 1 -- 172.21.15.37:0/3447676132 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b3c09aca0 con 0x7f5b3c135d90 2024-04-26T00:29:17.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b44247640 10 monclient: _renew_subs 2024-04-26T00:29:17.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b44247640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:29:17.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.670+0000 7f5b44247640 1 -- 172.21.15.37:0/3447676132 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b3c13e9c0 con 0x7f5b3c135d90 2024-04-26T00:29:17.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.674+0000 7f5b44247640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:29:17.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.674+0000 7f5b44247640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:29:17.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.674+0000 7f5b44247640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:29:17.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.674+0000 7f5b44247640 1 -- 172.21.15.37:0/3447676132 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b3c13d300 con 0x7f5b3c135d90 2024-04-26T00:29:17.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.674+0000 7f5b327fc640 1 -- 172.21.15.37:0/3447676132 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f5b380117d0 con 0x7f5b3c135d90 2024-04-26T00:29:17.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.674+0000 7f5b327fc640 1 --2- 172.21.15.37:0/3447676132 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5b20035e80 0x7f5b20038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:17.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.674+0000 7f5b327fc640 1 -- 172.21.15.37:0/3447676132 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(369..369 src has 177..369) ==== 3931+0+0 (secure 0 0 0) 0x7f5b38048d40 con 0x7f5b3c135d90 2024-04-26T00:29:17.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.674+0000 7f5b417e2640 1 --2- 172.21.15.37:0/3447676132 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5b20035e80 0x7f5b20038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:17.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.674+0000 7f5b417e2640 10 monclient: get_auth_request con 0x7f5b20035e80 auth_method 0 2024-04-26T00:29:17.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.678+0000 7f5b417e2640 1 --2- 172.21.15.37:0/3447676132 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5b20035e80 0x7f5b20038340 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c0ac000 tx=0x7f5b2c0a7000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:17.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.678+0000 7f5b327fc640 1 -- 172.21.15.37:0/3447676132 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5b38015020 con 0x7f5b3c135d90 2024-04-26T00:29:17.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.678+0000 7f5b327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:29:17.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.678+0000 7f5b327fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:29:17.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.906+0000 7f5b44247640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-26T00:29:17.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.906+0000 7f5b44247640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-26T00:29:17.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.906+0000 7f5b44247640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:29:17.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.906+0000 7f5b44247640 1 -- 172.21.15.37:0/3447676132 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f5b3c075910 con 0x7f5b3c135d90 2024-04-26T00:29:17.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.906+0000 7f5b327fc640 1 -- 172.21.15.37:0/3447676132 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f5b38011da0 con 0x7f5b3c135d90 2024-04-26T00:29:17.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.906+0000 7f5b327fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-26T00:29:17.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.906+0000 7f5b327fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:29:17.897 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.906+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.897 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.906+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=12,kv=10,logm=57} crc {auth=1090356750,config=692619927,health=1688807597,kv=3650273984,logm=44653780}) 2024-04-26T00:29:17.897 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.910+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=18,config=2,health=12,kv=9,logm=59} crc {auth=2931093999,config=692619927,health=3334251417,kv=2491615610,logm=117979421}) 2024-04-26T00:29:17.897 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.910+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.897 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.910+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=12,kv=10,logm=57} crc {auth=1090356750,config=692619927,health=1688807597,kv=3650273984,logm=44653780}) 2024-04-26T00:29:17.898 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.910+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=18,config=2,health=12,kv=10,logm=58} crc {auth=1977575287,config=692619927,health=185112578,kv=1034996044,logm=336680523}) 2024-04-26T00:29:17.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.910+0000 7f5b44247640 1 -- 172.21.15.37:0/3447676132 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5b20035e80 msgr2=0x7f5b20038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:17.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.910+0000 7f5b44247640 1 --2- 172.21.15.37:0/3447676132 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5b20035e80 0x7f5b20038340 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c0ac000 tx=0x7f5b2c0a7000 comp rx=0 tx=0).stop 2024-04-26T00:29:17.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.910+0000 7f5b44247640 10 monclient: shutdown 2024-04-26T00:29:17.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.910+0000 7f5b44247640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:29:17.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.910+0000 7f5b44247640 1 -- 172.21.15.37:0/3447676132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b3c135d90 msgr2=0x7f5b3c1cf4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:17.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.910+0000 7f5b44247640 1 --2- 172.21.15.37:0/3447676132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b3c135d90 0x7f5b3c1cf4d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5b3800c7f0 tx=0x7f5b3800ccb0 comp rx=0 tx=0).stop 2024-04-26T00:29:17.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.910+0000 7f5b44247640 1 -- 172.21.15.37:0/3447676132 shutdown_connections 2024-04-26T00:29:17.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.910+0000 7f5b44247640 1 --2- 172.21.15.37:0/3447676132 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5b20035e80 0x7f5b20038340 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:17.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.910+0000 7f5b44247640 1 --2- 172.21.15.37:0/3447676132 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b3c1371f0 0x7f5b3c13adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:17.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.910+0000 7f5b44247640 1 --2- 172.21.15.37:0/3447676132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b3c136e10 0x7f5b3c134f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:17.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.910+0000 7f5b44247640 1 --2- 172.21.15.37:0/3447676132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b3c135d90 0x7f5b3c1cf4d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:17.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.910+0000 7f5b44247640 1 -- 172.21.15.37:0/3447676132 >> 172.21.15.37:0/3447676132 conn(0x7f5b3c0a64e0 msgr2=0x7f5b3c0a4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:29:17.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.910+0000 7f5b44247640 1 -- 172.21.15.37:0/3447676132 shutdown_connections 2024-04-26T00:29:17.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:17.910+0000 7f5b44247640 1 -- 172.21.15.37:0/3447676132 wait complete. 2024-04-26T00:29:17.901 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.910+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.901 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.910+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1240976655}) 2024-04-26T00:29:17.901 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.910+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4224910833}) 2024-04-26T00:29:17.901 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.910+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.901 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.910+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1240976655}) 2024-04-26T00:29:17.901 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.910+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1473388101}) 2024-04-26T00:29:17.903 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.914+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.903 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.914+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=612958882}) 2024-04-26T00:29:17.903 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.914+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2376441764}) 2024-04-26T00:29:17.903 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.914+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.904 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.914+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=612958882}) 2024-04-26T00:29:17.904 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.914+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3962833795}) 2024-04-26T00:29:17.905 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.914+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.905 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.914+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2672481148}) 2024-04-26T00:29:17.905 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.914+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1914792722}) 2024-04-26T00:29:17.905 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.914+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.905 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.914+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2672481148}) 2024-04-26T00:29:17.906 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.918+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1306192701}) 2024-04-26T00:29:17.907 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.918+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.907 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.918+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=899402526}) 2024-04-26T00:29:17.908 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.918+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=658468182}) 2024-04-26T00:29:17.908 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.918+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.908 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.918+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=899402526}) 2024-04-26T00:29:17.908 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.918+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2036136149}) 2024-04-26T00:29:17.910 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.922+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.910 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.922+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2273319348}) 2024-04-26T00:29:17.910 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.922+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2045286350}) 2024-04-26T00:29:17.910 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.922+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.910 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.922+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2273319348}) 2024-04-26T00:29:17.910 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.922+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4270382173}) 2024-04-26T00:29:17.912 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.922+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.912 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.922+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2532095615}) 2024-04-26T00:29:17.912 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.922+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3243723983}) 2024-04-26T00:29:17.912 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.922+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.912 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.922+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2532095615}) 2024-04-26T00:29:17.912 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.922+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2016459252}) 2024-04-26T00:29:17.914 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.926+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.914 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.926+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4033059260}) 2024-04-26T00:29:17.914 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.926+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4142746913}) 2024-04-26T00:29:17.914 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.926+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.914 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.926+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4033059260}) 2024-04-26T00:29:17.915 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.926+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3483390990}) 2024-04-26T00:29:17.916 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.926+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.916 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.926+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3864152227}) 2024-04-26T00:29:17.916 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.926+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1959062822}) 2024-04-26T00:29:17.916 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.926+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.917 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.926+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3864152227}) 2024-04-26T00:29:17.917 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.926+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=608906178}) 2024-04-26T00:29:17.917 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-26T00:29:17.919 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.930+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.919 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.930+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3900503855}) 2024-04-26T00:29:17.919 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.930+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1949001002}) 2024-04-26T00:29:17.919 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.930+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.919 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.930+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3900503855}) 2024-04-26T00:29:17.919 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.930+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4218842305}) 2024-04-26T00:29:17.921 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.930+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.921 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.930+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1268958197}) 2024-04-26T00:29:17.921 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.930+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2812168503}) 2024-04-26T00:29:17.922 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.934+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.922 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.934+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1268958197}) 2024-04-26T00:29:17.922 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.934+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1443368684}) 2024-04-26T00:29:17.924 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.934+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.924 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.934+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3592840455}) 2024-04-26T00:29:17.924 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.934+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=135448978}) 2024-04-26T00:29:17.924 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.934+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.924 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.934+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3592840455}) 2024-04-26T00:29:17.924 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.934+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4184822374}) 2024-04-26T00:29:17.926 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.938+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.926 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.938+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=780847277}) 2024-04-26T00:29:17.926 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.938+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3095752283}) 2024-04-26T00:29:17.926 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.938+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.926 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.938+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=780847277}) 2024-04-26T00:29:17.926 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.938+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=840735849}) 2024-04-26T00:29:17.928 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.938+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.928 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.938+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4273345092}) 2024-04-26T00:29:17.928 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.938+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2391320233}) 2024-04-26T00:29:17.928 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.938+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.928 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.938+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4273345092}) 2024-04-26T00:29:17.928 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.938+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1748024852}) 2024-04-26T00:29:17.930 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.942+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.930 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.942+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1957150500}) 2024-04-26T00:29:17.930 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.942+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3397628789}) 2024-04-26T00:29:17.931 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.942+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.931 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.942+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1957150500}) 2024-04-26T00:29:17.931 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.942+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1763193144}) 2024-04-26T00:29:17.932 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.942+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.932 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.942+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1940548741}) 2024-04-26T00:29:17.932 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.942+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1963171534}) 2024-04-26T00:29:17.933 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.942+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.933 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.942+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1940548741}) 2024-04-26T00:29:17.933 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.942+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3573548652}) 2024-04-26T00:29:17.935 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.946+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.935 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.946+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3029020189}) 2024-04-26T00:29:17.935 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.946+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=630668748}) 2024-04-26T00:29:17.935 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.946+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.935 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.946+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3029020189}) 2024-04-26T00:29:17.935 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.946+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2965380165}) 2024-04-26T00:29:17.937 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.946+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.937 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.946+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2616066657}) 2024-04-26T00:29:17.937 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.946+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2134649947}) 2024-04-26T00:29:17.937 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.946+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.937 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.946+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2616066657}) 2024-04-26T00:29:17.937 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.946+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3131445170}) 2024-04-26T00:29:17.939 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.950+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.939 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.950+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4053388266}) 2024-04-26T00:29:17.939 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.950+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1623596218}) 2024-04-26T00:29:17.939 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.950+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.939 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.950+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4053388266}) 2024-04-26T00:29:17.939 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.950+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1212904650}) 2024-04-26T00:29:17.941 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.950+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.941 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.950+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1947667963}) 2024-04-26T00:29:17.941 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.950+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1579786151}) 2024-04-26T00:29:17.941 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.950+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.941 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.950+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1947667963}) 2024-04-26T00:29:17.941 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.950+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1122888373}) 2024-04-26T00:29:17.943 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.954+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.943 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.954+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=496491645}) 2024-04-26T00:29:17.943 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.954+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=679777840}) 2024-04-26T00:29:17.943 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.954+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.943 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.954+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=496491645}) 2024-04-26T00:29:17.944 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.954+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4045693156}) 2024-04-26T00:29:17.945 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.954+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.945 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.954+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=139267802}) 2024-04-26T00:29:17.946 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.958+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3970096153}) 2024-04-26T00:29:17.946 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.958+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.946 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.958+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=139267802}) 2024-04-26T00:29:17.946 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.958+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=603098449}) 2024-04-26T00:29:17.947 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.958+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.948 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.958+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3351511570}) 2024-04-26T00:29:17.948 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.958+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4272564413}) 2024-04-26T00:29:17.948 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.958+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.948 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.958+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3351511570}) 2024-04-26T00:29:17.948 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.958+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2692481083}) 2024-04-26T00:29:17.950 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.962+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.950 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.962+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4180782029}) 2024-04-26T00:29:17.950 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.962+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1486063756}) 2024-04-26T00:29:17.950 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.962+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.950 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.962+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4180782029}) 2024-04-26T00:29:17.950 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.962+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3309541088}) 2024-04-26T00:29:17.952 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.962+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.952 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.962+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2471761795}) 2024-04-26T00:29:17.952 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.962+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2907758808}) 2024-04-26T00:29:17.952 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.962+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.952 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.962+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2471761795}) 2024-04-26T00:29:17.952 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.962+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1664034997}) 2024-04-26T00:29:17.954 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.966+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.954 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.966+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=751833363}) 2024-04-26T00:29:17.954 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.966+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=901241524}) 2024-04-26T00:29:17.954 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.966+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.955 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.966+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=751833363}) 2024-04-26T00:29:17.955 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.966+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1821318088}) 2024-04-26T00:29:17.957 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.966+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.957 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.966+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=65,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=10,monmap=1} crc {logm=3001578375,mdsmap=3764978221,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=2603055307,mon_config_key=1078409208,monmap=3249110452}) 2024-04-26T00:29:17.957 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.966+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=64,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=9,monmap=1} crc {logm=2248691641,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=3373513014,mon_config_key=206417726,monmap=3249110452}) 2024-04-26T00:29:17.957 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.966+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.957 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.966+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=65,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=10,monmap=1} crc {logm=3001578375,mdsmap=3764978221,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=2603055307,mon_config_key=1078409208,monmap=3249110452}) 2024-04-26T00:29:17.957 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.966+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=65,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=10} crc {logm=3619200177,mdsmap=2840952586,mgr=3887433460,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=3373513014,mon_config_key=742358462}) 2024-04-26T00:29:17.959 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.970+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.959 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.970+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=63,osdmap=28} crc {monmap=3140514009,osd_metadata=3306121363,osd_pg_creating=3315247672,osd_snap=1846437429,osdmap=3228138591}) 2024-04-26T00:29:17.959 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.970+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=63,osdmap=28} crc {monmap=3140514009,osd_metadata=3306121363,osd_pg_creating=3315247672,osd_snap=3350176526,osdmap=3169070916}) 2024-04-26T00:29:17.959 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.970+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.959 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.970+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=63,osdmap=28} crc {monmap=3140514009,osd_metadata=3306121363,osd_pg_creating=3315247672,osd_snap=1846437429,osdmap=3228138591}) 2024-04-26T00:29:17.959 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.970+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=61,osdmap=30} crc {monmap=3140514009,osd_metadata=2845665730,osd_pg_creating=3315247672,osd_snap=2622400828,osdmap=4255751704}) 2024-04-26T00:29:17.961 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.970+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.961 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.970+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=243422330}) 2024-04-26T00:29:17.961 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.970+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1233221514}) 2024-04-26T00:29:17.961 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.970+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.962 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.970+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=243422330}) 2024-04-26T00:29:17.962 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.970+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=321780000}) 2024-04-26T00:29:17.964 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.974+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.964 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.974+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1255759537}) 2024-04-26T00:29:17.964 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.974+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2342604073}) 2024-04-26T00:29:17.964 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.974+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.964 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.974+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1255759537}) 2024-04-26T00:29:17.964 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.974+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3231686974}) 2024-04-26T00:29:17.974 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.986+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.974 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.986+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=67} crc {osdmap=1254027969}) 2024-04-26T00:29:17.975 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.986+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=67} crc {osdmap=1996064483}) 2024-04-26T00:29:17.975 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.986+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:29:17.975 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.986+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=67} crc {osdmap=1254027969}) 2024-04-26T00:29:17.975 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:29:17.986+0000 7f1334c37640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=64} crc {osdmap=2948154673}) 2024-04-26T00:29:18.445 INFO:tasks.rados.rados.0.smithi045.stdout:1611: done (2 left) 2024-04-26T00:29:18.445 INFO:tasks.rados.rados.0.smithi045.stdout:1612: snap_create 2024-04-26T00:29:18.447 INFO:tasks.rados.rados.0.smithi045.stdout:1609: expect (ObjNum 586 snap 182 seq_num 586) 2024-04-26T00:29:18.919 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:29:18.919 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:29:19.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f213097e640 10 monclient: get_monmap_and_config 2024-04-26T00:29:19.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f213097e640 10 monclient: build_initial_monmap 2024-04-26T00:29:19.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f213097e640 10 monclient: monmap: 2024-04-26T00:29:19.051 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:29:19.051 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:19.051 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:29:19.065349+0000 2024-04-26T00:29:19.051 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:29:19.065349+0000 2024-04-26T00:29:19.051 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:29:19.051 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:19.051 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:29:19.051 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:29:19.051 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:29:19.052 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:29:19.052 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:29:19.052 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:19.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f213097e640 1 Processor -- start 2024-04-26T00:29:19.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f213097e640 1 -- start start 2024-04-26T00:29:19.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f213097e640 10 monclient: init 2024-04-26T00:29:19.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f213097e640 10 monclient: _reopen_session rank -1 2024-04-26T00:29:19.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f213097e640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-04-26T00:29:19.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f213097e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f212c0a0260 0x7f212c0a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:19.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f213097e640 10 monclient(hunting): picked mon.noname-b con 0x7f212c0a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:29:19.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f213097e640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f212c134b30 0x7f212c134f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:19.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f213097e640 10 monclient(hunting): picked mon.noname-a con 0x7f212c134b30 addr v2:172.21.15.37:3300/0 2024-04-26T00:29:19.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f213097e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f212c135490 0x7f212c13be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:19.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f213097e640 10 monclient(hunting): picked mon.noname-c con 0x7f212c135490 addr v2:172.21.15.45:3300/0 2024-04-26T00:29:19.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f213097e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:19.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f213097e640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f212c09ec10 con 0x7f212c134b30 2024-04-26T00:29:19.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f213097e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:19.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f213097e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f212c09f0f0 con 0x7f212c135490 2024-04-26T00:29:19.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f213097e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:19.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f213097e640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f212c0a1130 con 0x7f212c0a0260 2024-04-26T00:29:19.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f213097e640 10 monclient(hunting): _renew_subs 2024-04-26T00:29:19.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f213097e640 10 monclient(hunting): authenticate will time out at 1540.371235s 2024-04-26T00:29:19.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f2129d9b640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f212c134b30 0x7f212c134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:19.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f2129d9b640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f212c134b30 0x7f212c134f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33900/0 (socket says 172.21.15.37:33900) 2024-04-26T00:29:19.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f2129d9b640 1 -- 172.21.15.37:0/208345702 learned_addr learned my addr 172.21.15.37:0/208345702 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:29:19.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f2129d9b640 10 monclient(hunting): get_auth_request con 0x7f212c134b30 auth_method 0 2024-04-26T00:29:19.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f2129d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:19.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f2129d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:19.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f2129d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:19.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f212a59c640 1 --2- 172.21.15.37:0/208345702 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f212c0a0260 0x7f212c0a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:19.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f212a59c640 10 monclient(hunting): get_auth_request con 0x7f212c0a0260 auth_method 0 2024-04-26T00:29:19.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f212a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:19.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f212a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:19.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f212a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:19.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f2129d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:19.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f2129d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:19.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f2129d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:19.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f212a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:19.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f212a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:19.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f212a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:19.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f212ad9d640 1 --2- 172.21.15.37:0/208345702 >> v2:172.21.15.45:3300/0 conn(0x7f212c135490 0x7f212c13be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:19.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f212ad9d640 10 monclient(hunting): get_auth_request con 0x7f212c135490 auth_method 0 2024-04-26T00:29:19.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f212ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:19.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f212ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:19.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f212ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:19.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f212a59c640 10 monclient(hunting): handle_auth_done global_id 84413 payload 274 2024-04-26T00:29:19.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f212a59c640 1 -- 172.21.15.37:0/208345702 >> v2:172.21.15.45:3300/0 conn(0x7f212c135490 msgr2=0x7f212c13be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:29:19.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f212a59c640 1 --2- 172.21.15.37:0/208345702 >> v2:172.21.15.45:3300/0 conn(0x7f212c135490 0x7f212c13be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:19.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f212a59c640 1 -- 172.21.15.37:0/208345702 >> v2:172.21.15.37:3300/0 conn(0x7f212c134b30 msgr2=0x7f212c134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:29:19.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f212a59c640 1 --2- 172.21.15.37:0/208345702 >> v2:172.21.15.37:3300/0 conn(0x7f212c134b30 0x7f212c134f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:19.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f212a59c640 10 monclient: _finish_hunting 0 2024-04-26T00:29:19.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f212a59c640 1 monclient: found mon.noname-b 2024-04-26T00:29:19.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f212a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:29:19.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f212a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:29:19.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f212a59c640 1 -- 172.21.15.37:0/208345702 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f212c06cff0 con 0x7f212c0a0260 2024-04-26T00:29:19.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.062+0000 7f2129d9b640 1 --2- 172.21.15.37:0/208345702 >> v2:172.21.15.37:3300/0 conn(0x7f212c134b30 0x7f212c134f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:29:19.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.066+0000 7f212a59c640 1 -- 172.21.15.37:0/208345702 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f212c0a0260 msgr2=0x7f212c0a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:19.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.066+0000 7f212a59c640 1 --2- 172.21.15.37:0/208345702 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f212c0a0260 0x7f212c0a0640 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f212c135bd0 tx=0x7f211c01c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=754f024e570696c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:19.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212959a640 1 -- 172.21.15.37:0/208345702 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f211c0ac070 con 0x7f212c0a0260 2024-04-26T00:29:19.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:19.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:29:19.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212959a640 10 monclient: dump: 2024-04-26T00:29:19.062 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:19.062 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:19.062 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.062 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.062 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:19.062 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:19.063 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:19.063 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:19.063 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:19.063 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:19.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212959a640 10 monclient: _finish_auth 0 2024-04-26T00:29:19.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212959a640 10 monclient: _check_auth_tickets 2024-04-26T00:29:19.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:29:19.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212959a640 1 -- 172.21.15.37:0/208345702 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f211c0a6680 con 0x7f212c0a0260 2024-04-26T00:29:19.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212959a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:29:19.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212959a640 1 -- 172.21.15.37:0/208345702 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f211c0a6a50 con 0x7f212c0a0260 2024-04-26T00:29:19.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:19.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:29:19.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212959a640 10 monclient: dump: 2024-04-26T00:29:19.063 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:19.064 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:19.064 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.064 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.064 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:19.064 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:19.064 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:19.064 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:19.064 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:19.064 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:19.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f213097e640 5 monclient: authenticate success, global_id 84413 2024-04-26T00:29:19.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f213097e640 10 monclient: get_monmap_and_config success 2024-04-26T00:29:19.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f213097e640 4 set_mon_vals no callback set 2024-04-26T00:29:19.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f213097e640 10 monclient: shutdown 2024-04-26T00:29:19.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f213097e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:29:19.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f213097e640 1 -- 172.21.15.37:0/208345702 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f212c0a0260 msgr2=0x7f212c0a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:19.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f213097e640 1 --2- 172.21.15.37:0/208345702 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f212c0a0260 0x7f212c0a0640 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f212c135bd0 tx=0x7f211c01c7f0 comp rx=0 tx=0).stop 2024-04-26T00:29:19.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f213097e640 1 -- 172.21.15.37:0/208345702 shutdown_connections 2024-04-26T00:29:19.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f213097e640 1 --2- 172.21.15.37:0/208345702 >> v2:172.21.15.45:3300/0 conn(0x7f212c135490 0x7f212c13be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:19.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f213097e640 1 --2- 172.21.15.37:0/208345702 >> v2:172.21.15.37:3300/0 conn(0x7f212c134b30 0x7f212c134f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:19.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f213097e640 1 --2- 172.21.15.37:0/208345702 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f212c0a0260 0x7f212c0a0640 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:19.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f213097e640 1 -- 172.21.15.37:0/208345702 >> 172.21.15.37:0/208345702 conn(0x7f212c0a1ad0 msgr2=0x7f212c0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:29:19.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f213097e640 1 -- 172.21.15.37:0/208345702 shutdown_connections 2024-04-26T00:29:19.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f213097e640 1 -- 172.21.15.37:0/208345702 wait complete. 2024-04-26T00:29:19.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f213097e640 10 monclient: build_initial_monmap 2024-04-26T00:29:19.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f213097e640 10 monclient: monmap: 2024-04-26T00:29:19.065 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:29:19.065 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:29:19.065 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:29:19.065 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:29:19.066 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:29:19.066 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:29:19.066 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:29:19.066 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:29:19.066 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:29:19.066 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:19.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f213097e640 1 Processor -- start 2024-04-26T00:29:19.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f213097e640 1 -- start start 2024-04-26T00:29:19.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f213097e640 10 monclient: init 2024-04-26T00:29:19.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f213097e640 10 monclient: _reopen_session rank -1 2024-04-26T00:29:19.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f213097e640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:29:19.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f213097e640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f212c0a0260 0x7f212c11b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:19.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f213097e640 10 monclient(hunting): picked mon.noname-a con 0x7f212c0a0260 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:29:19.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f213097e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f212c134b30 0x7f212c131700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:19.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f213097e640 10 monclient(hunting): picked mon.noname-b con 0x7f212c134b30 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:29:19.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f213097e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f212c135490 0x7f212c139760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:19.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f213097e640 10 monclient(hunting): picked mon.noname-c con 0x7f212c135490 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:29:19.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f213097e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:19.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f213097e640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f212c09f430 con 0x7f212c0a0260 2024-04-26T00:29:19.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f213097e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:19.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f213097e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f212c09f2b0 con 0x7f212c134b30 2024-04-26T00:29:19.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f213097e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:19.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f213097e640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f212c09fe90 con 0x7f212c135490 2024-04-26T00:29:19.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f213097e640 10 monclient(hunting): _renew_subs 2024-04-26T00:29:19.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f213097e640 10 monclient(hunting): authenticate will time out at 1540.378687s 2024-04-26T00:29:19.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212ad9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f212c135490 0x7f212c139760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:19.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212a59c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f212c0a0260 0x7f212c11b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:19.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212a59c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f212c0a0260 0x7f212c11b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33916/0 (socket says 172.21.15.37:33916) 2024-04-26T00:29:19.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212a59c640 1 -- 172.21.15.37:0/2095947994 learned_addr learned my addr 172.21.15.37:0/2095947994 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:29:19.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212a59c640 10 monclient(hunting): get_auth_request con 0x7f212c0a0260 auth_method 0 2024-04-26T00:29:19.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212ad9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f212c135490 0x7f212c139760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:41412/0 (socket says 172.21.15.37:41412) 2024-04-26T00:29:19.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:19.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:19.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:19.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212ad9d640 10 monclient(hunting): get_auth_request con 0x7f212c135490 auth_method 0 2024-04-26T00:29:19.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:19.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:19.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:19.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f2129d9b640 1 --2- 172.21.15.37:0/2095947994 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f212c134b30 0x7f212c131700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:19.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f2129d9b640 10 monclient(hunting): get_auth_request con 0x7f212c134b30 auth_method 0 2024-04-26T00:29:19.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f2129d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:19.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f2129d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:19.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f2129d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:19.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:19.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:19.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:19.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:19.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:19.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:19.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f2129d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:19.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f2129d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:19.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f2129d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:19.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212a59c640 10 monclient(hunting): handle_auth_done global_id 94359 payload 824 2024-04-26T00:29:19.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212a59c640 1 -- 172.21.15.37:0/2095947994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f212c135490 msgr2=0x7f212c139760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:19.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212a59c640 1 --2- 172.21.15.37:0/2095947994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f212c135490 0x7f212c139760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:19.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212a59c640 1 -- 172.21.15.37:0/2095947994 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f212c134b30 msgr2=0x7f212c131700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:19.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212a59c640 1 --2- 172.21.15.37:0/2095947994 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f212c134b30 0x7f212c131700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:19.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212a59c640 10 monclient: _finish_hunting 0 2024-04-26T00:29:19.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212a59c640 1 monclient: found mon.noname-a 2024-04-26T00:29:19.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:29:19.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:29:19.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212a59c640 1 -- 172.21.15.37:0/2095947994 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f212c11cdc0 con 0x7f212c0a0260 2024-04-26T00:29:19.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.070+0000 7f212ad9d640 1 --2- 172.21.15.37:0/2095947994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f212c135490 0x7f212c139760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:29:19.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.074+0000 7f212a59c640 1 -- 172.21.15.37:0/2095947994 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f212c0a0260 msgr2=0x7f212c11b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:19.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.074+0000 7f212a59c640 1 --2- 172.21.15.37:0/2095947994 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f212c0a0260 0x7f212c11b910 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f211c01d090 tx=0x7f211c019590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:19.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.074+0000 7f2122ffd640 1 -- 172.21.15.37:0/2095947994 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f211c0c0070 con 0x7f212c0a0260 2024-04-26T00:29:19.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.074+0000 7f2122ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:19.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.074+0000 7f2122ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:29:19.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.074+0000 7f2122ffd640 10 monclient: dump: 2024-04-26T00:29:19.072 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:19.072 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:19.073 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.073 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.073 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:19.073 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:19.073 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:19.073 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:19.073 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:19.073 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:19.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.074+0000 7f2122ffd640 10 monclient: _finish_auth 0 2024-04-26T00:29:19.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.074+0000 7f2122ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:29:19.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.074+0000 7f2122ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:29:19.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.074+0000 7f2122ffd640 1 -- 172.21.15.37:0/2095947994 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f211c0a6480 con 0x7f212c0a0260 2024-04-26T00:29:19.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.074+0000 7f2122ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:29:19.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.074+0000 7f2122ffd640 1 -- 172.21.15.37:0/2095947994 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f211c0ac050 con 0x7f212c0a0260 2024-04-26T00:29:19.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.074+0000 7f2122ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:19.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.074+0000 7f2122ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:29:19.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.074+0000 7f2122ffd640 10 monclient: dump: 2024-04-26T00:29:19.074 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:19.074 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:19.074 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.074 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.074 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:19.074 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:19.074 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:19.074 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:19.074 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:19.074 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:19.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.074+0000 7f213097e640 5 monclient: authenticate success, global_id 94359 2024-04-26T00:29:19.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.074+0000 7f213097e640 10 monclient: _renew_subs 2024-04-26T00:29:19.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.074+0000 7f2123fff640 4 set_mon_vals no callback set 2024-04-26T00:29:19.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.074+0000 7f213097e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:29:19.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.074+0000 7f213097e640 1 -- 172.21.15.37:0/2095947994 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f212c11c340 con 0x7f212c0a0260 2024-04-26T00:29:19.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.074+0000 7f213097e640 10 monclient: _renew_subs 2024-04-26T00:29:19.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.074+0000 7f213097e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:29:19.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.074+0000 7f213097e640 1 -- 172.21.15.37:0/2095947994 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f212c11cae0 con 0x7f212c0a0260 2024-04-26T00:29:19.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.074+0000 7f213097e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:29:19.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.074+0000 7f213097e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:29:19.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.074+0000 7f213097e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:29:19.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.074+0000 7f213097e640 1 -- 172.21.15.37:0/2095947994 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f212c066f40 con 0x7f212c0a0260 2024-04-26T00:29:19.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.074+0000 7f2122ffd640 1 -- 172.21.15.37:0/2095947994 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f211c0c3070 con 0x7f212c0a0260 2024-04-26T00:29:19.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.074+0000 7f2122ffd640 1 --2- 172.21.15.37:0/2095947994 >> v2:172.21.15.45:6812/2379918623 conn(0x7f210c035e80 0x7f210c038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:19.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.074+0000 7f2122ffd640 1 -- 172.21.15.37:0/2095947994 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(370..370 src has 177..370) ==== 3975+0+0 (secure 0 0 0) 0x7f211c0e33e0 con 0x7f212c0a0260 2024-04-26T00:29:19.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.074+0000 7f2129d9b640 1 --2- 172.21.15.37:0/2095947994 >> v2:172.21.15.45:6812/2379918623 conn(0x7f210c035e80 0x7f210c038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:19.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.074+0000 7f2129d9b640 10 monclient: get_auth_request con 0x7f210c035e80 auth_method 0 2024-04-26T00:29:19.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.078+0000 7f2129d9b640 1 --2- 172.21.15.37:0/2095947994 >> v2:172.21.15.45:6812/2379918623 conn(0x7f210c035e80 0x7f210c038340 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f212c13afb0 tx=0x7f2114009240 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:19.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.082+0000 7f2122ffd640 1 -- 172.21.15.37:0/2095947994 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f211c016500 con 0x7f212c0a0260 2024-04-26T00:29:19.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.082+0000 7f2122ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:29:19.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.082+0000 7f2122ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:29:19.279 INFO:tasks.rados.rados.0.smithi045.stdout:1609: done (2 left) 2024-04-26T00:29:19.279 INFO:tasks.rados.rados.0.smithi045.stdout:1610: done (1 left) 2024-04-26T00:29:19.279 INFO:tasks.rados.rados.0.smithi045.stdout:1612: done (0 left) 2024-04-26T00:29:19.279 INFO:tasks.rados.rados.0.smithi045.stdout:1613: write (excl) oid 36 current snap is 186 2024-04-26T00:29:19.279 INFO:tasks.rados.rados.0.smithi045.stdout:1613: seq_num 600 ranges {484200=624934,1860843=223357} 2024-04-26T00:29:19.296 INFO:tasks.rados.rados.0.smithi045.stdout:1613: writing smithi04526579-36 from 484200 to 1109134 tid 1 2024-04-26T00:29:19.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.310+0000 7f213097e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:29:19.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.310+0000 7f213097e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:29:19.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.310+0000 7f213097e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:29:19.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.310+0000 7f213097e640 1 -- 172.21.15.37:0/2095947994 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f212c11b690 con 0x7f212c0a0260 2024-04-26T00:29:19.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.310+0000 7f2122ffd640 1 -- 172.21.15.37:0/2095947994 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f211c0a7ae0 con 0x7f212c0a0260 2024-04-26T00:29:19.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.310+0000 7f2122ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:29:19.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.310+0000 7f2122ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:29:19.300 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:29:19.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.314+0000 7f213097e640 1 -- 172.21.15.37:0/2095947994 >> v2:172.21.15.45:6812/2379918623 conn(0x7f210c035e80 msgr2=0x7f210c038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:19.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.314+0000 7f213097e640 1 --2- 172.21.15.37:0/2095947994 >> v2:172.21.15.45:6812/2379918623 conn(0x7f210c035e80 0x7f210c038340 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f212c13afb0 tx=0x7f2114009240 comp rx=0 tx=0).stop 2024-04-26T00:29:19.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.314+0000 7f213097e640 10 monclient: shutdown 2024-04-26T00:29:19.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.314+0000 7f213097e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:29:19.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.314+0000 7f213097e640 1 -- 172.21.15.37:0/2095947994 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f212c0a0260 msgr2=0x7f212c11b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:19.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.314+0000 7f213097e640 1 --2- 172.21.15.37:0/2095947994 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f212c0a0260 0x7f212c11b910 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f211c01d090 tx=0x7f211c019590 comp rx=0 tx=0).stop 2024-04-26T00:29:19.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.314+0000 7f213097e640 1 -- 172.21.15.37:0/2095947994 shutdown_connections 2024-04-26T00:29:19.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.314+0000 7f213097e640 1 --2- 172.21.15.37:0/2095947994 >> v2:172.21.15.45:6812/2379918623 conn(0x7f210c035e80 0x7f210c038340 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:19.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.314+0000 7f213097e640 1 --2- 172.21.15.37:0/2095947994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f212c135490 0x7f212c139760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:19.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.314+0000 7f213097e640 1 --2- 172.21.15.37:0/2095947994 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f212c134b30 0x7f212c131700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:19.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.314+0000 7f213097e640 1 --2- 172.21.15.37:0/2095947994 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f212c0a0260 0x7f212c11b910 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:19.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.314+0000 7f213097e640 1 -- 172.21.15.37:0/2095947994 >> 172.21.15.37:0/2095947994 conn(0x7f212c0a1ad0 msgr2=0x7f212c0a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:29:19.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.314+0000 7f213097e640 1 -- 172.21.15.37:0/2095947994 shutdown_connections 2024-04-26T00:29:19.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.314+0000 7f213097e640 1 -- 172.21.15.37:0/2095947994 wait complete. 2024-04-26T00:29:19.313 INFO:tasks.rados.rados.0.smithi045.stdout:1613: writing smithi04526579-36 from 1860843 to 2084200 tid 2 2024-04-26T00:29:19.313 INFO:tasks.rados.rados.0.smithi045.stdout:1614: read oid 4 snap -1 2024-04-26T00:29:19.313 INFO:tasks.rados.rados.0.smithi045.stdout:1614: expect (ObjNum 474 snap 147 seq_num 474) 2024-04-26T00:29:19.314 INFO:tasks.rados.rados.0.smithi045.stdout:1615: write (excl) oid 37 current snap is 186 2024-04-26T00:29:19.314 INFO:tasks.rados.rados.0.smithi045.stdout:1615: seq_num 601 ranges {501007=416536,1635678=465329} 2024-04-26T00:29:19.324 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:29:19.324 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-26T00:29:19.324 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:29:19.324 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:29:19.327 INFO:tasks.rados.rados.0.smithi045.stdout:1615: writing smithi04526579-37 from 501007 to 917543 tid 1 2024-04-26T00:29:19.343 INFO:tasks.rados.rados.0.smithi045.stdout:1615: writing smithi04526579-37 from 1635678 to 2101007 tid 2 2024-04-26T00:29:19.343 INFO:tasks.rados.rados.0.smithi045.stdout:1616: copy_from oid 38 from oid 14 current snap is 186 2024-04-26T00:29:19.343 INFO:tasks.rados.rados.0.smithi045.stdout:1613: finishing write tid 1 to smithi04526579-36 2024-04-26T00:29:19.343 INFO:tasks.rados.rados.0.smithi045.stdout:1613: finishing write tid 2 to smithi04526579-36 2024-04-26T00:29:19.344 INFO:tasks.rados.rados.0.smithi045.stdout:1613: finishing write tid 3 to smithi04526579-36 2024-04-26T00:29:19.344 INFO:tasks.rados.rados.0.smithi045.stdout:1613: finishing write tid 4 to smithi04526579-36 2024-04-26T00:29:19.344 INFO:tasks.rados.rados.0.smithi045.stdout:1613: oid 36 updating version 0 to 770 2024-04-26T00:29:19.344 INFO:tasks.rados.rados.0.smithi045.stdout:1613: oid 36 updating version 770 to 772 2024-04-26T00:29:19.344 INFO:tasks.rados.rados.0.smithi045.stdout:1613: oid 36 version 772 is already newer than 771 2024-04-26T00:29:19.344 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 772 (ObjNum 600 snap 186 seq_num 600) dirty exists 2024-04-26T00:29:19.344 INFO:tasks.rados.rados.0.smithi045.stdout:1613: left oid 36 (ObjNum 600 snap 186 seq_num 600) 2024-04-26T00:29:19.344 INFO:tasks.rados.rados.0.smithi045.stdout:1615: finishing write tid 1 to smithi04526579-37 2024-04-26T00:29:19.344 INFO:tasks.rados.rados.0.smithi045.stdout:1613: done (3 left) 2024-04-26T00:29:19.344 INFO:tasks.rados.rados.0.smithi045.stdout:1617: copy_from oid 41 from oid 5 current snap is 186 2024-04-26T00:29:19.344 INFO:tasks.rados.rados.0.smithi045.stdout:1618: read oid 48 snap -1 2024-04-26T00:29:19.344 INFO:tasks.rados.rados.0.smithi045.stdout:1618: expect deleted 2024-04-26T00:29:19.344 INFO:tasks.rados.rados.0.smithi045.stdout:1619: rollback oid 24 current snap is 186 2024-04-26T00:29:19.345 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 24 to 180 2024-04-26T00:29:19.345 INFO:tasks.rados.rados.0.smithi045.stdout:1620: copy_from oid 18 from oid 8 current snap is 186 2024-04-26T00:29:19.345 INFO:tasks.rados.rados.0.smithi045.stdout:1621: copy_from oid 49 from oid 32 current snap is 186 2024-04-26T00:29:19.345 INFO:tasks.rados.rados.0.smithi045.stdout:1622: copy_from oid 16 from oid 40 current snap is 186 2024-04-26T00:29:19.345 INFO:tasks.rados.rados.0.smithi045.stdout:1623: write oid 9 current snap is 186 2024-04-26T00:29:19.345 INFO:tasks.rados.rados.0.smithi045.stdout:1623: seq_num 602 ranges {517814=491785,1577807=540007} 2024-04-26T00:29:19.353 INFO:tasks.rados.rados.0.smithi045.stdout:1623: writing smithi04526579-9 from 517814 to 1009599 tid 1 2024-04-26T00:29:19.368 INFO:tasks.rados.rados.0.smithi045.stdout:1623: writing smithi04526579-9 from 1577807 to 2117814 tid 2 2024-04-26T00:29:19.369 INFO:tasks.rados.rados.0.smithi045.stdout:1624: write oid 3 current snap is 186 2024-04-26T00:29:19.369 INFO:tasks.rados.rados.0.smithi045.stdout:1624: seq_num 603 ranges {534621=567034,1636289=498332} 2024-04-26T00:29:19.377 INFO:tasks.rados.rados.0.smithi045.stdout:1624: writing smithi04526579-3 from 534621 to 1101655 tid 1 2024-04-26T00:29:19.388 INFO:tasks.rados.rados.0.smithi045.stdout:1624: writing smithi04526579-3 from 1636289 to 2134621 tid 2 2024-04-26T00:29:19.388 INFO:tasks.rados.rados.0.smithi045.stdout:1625: read oid 23 snap -1 2024-04-26T00:29:19.388 INFO:tasks.rados.rados.0.smithi045.stdout:1625: expect (ObjNum 558 snap 173 seq_num 558) 2024-04-26T00:29:19.388 INFO:tasks.rados.rados.0.smithi045.stdout:1626: write (excl) oid 25 current snap is 186 2024-04-26T00:29:19.388 INFO:tasks.rados.rados.0.smithi045.stdout:1617: finishing copy_from to smithi04526579-41 2024-04-26T00:29:19.389 INFO:tasks.rados.rados.0.smithi045.stdout:1617: got expected ENOENT (src dne) 2024-04-26T00:29:19.389 INFO:tasks.rados.rados.0.smithi045.stdout:1617: finishing copy_from racing read to smithi04526579-41 2024-04-26T00:29:19.389 INFO:tasks.rados.rados.0.smithi045.stdout:1622: finishing copy_from to smithi04526579-16 2024-04-26T00:29:19.389 INFO:tasks.rados.rados.0.smithi045.stdout:1622: got expected ENOENT (src dne) 2024-04-26T00:29:19.389 INFO:tasks.rados.rados.0.smithi045.stdout:1622: finishing copy_from racing read to smithi04526579-16 2024-04-26T00:29:19.389 INFO:tasks.rados.rados.0.smithi045.stdout:1615: finishing write tid 2 to smithi04526579-37 2024-04-26T00:29:19.389 INFO:tasks.rados.rados.0.smithi045.stdout:1621: finishing copy_from racing read to smithi04526579-49 2024-04-26T00:29:19.389 INFO:tasks.rados.rados.0.smithi045.stdout:1614: expect (ObjNum 474 snap 147 seq_num 474) 2024-04-26T00:29:19.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.466+0000 7f38c520a640 10 monclient: get_monmap_and_config 2024-04-26T00:29:19.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.466+0000 7f38c520a640 10 monclient: build_initial_monmap 2024-04-26T00:29:19.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.466+0000 7f38c520a640 10 monclient: monmap: 2024-04-26T00:29:19.454 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:29:19.454 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:19.454 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:29:19.468565+0000 2024-04-26T00:29:19.455 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:29:19.468565+0000 2024-04-26T00:29:19.455 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:29:19.455 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:19.455 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:29:19.455 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:29:19.455 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:29:19.455 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:29:19.455 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:29:19.455 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:19.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.466+0000 7f38c520a640 1 Processor -- start 2024-04-26T00:29:19.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.466+0000 7f38c520a640 1 -- start start 2024-04-26T00:29:19.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.466+0000 7f38c520a640 10 monclient: init 2024-04-26T00:29:19.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.466+0000 7f38c520a640 10 monclient: _reopen_session rank -1 2024-04-26T00:29:19.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.466+0000 7f38c520a640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-04-26T00:29:19.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.466+0000 7f38c520a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f38c00a0300 addr v1:172.21.15.37:6789/0 2024-04-26T00:29:19.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.466+0000 7f38c520a640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f38c009fc50 0x7f38c0139ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:19.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.466+0000 7f38c520a640 10 monclient(hunting): picked mon.noname-a con 0x7f38c009fc50 addr v2:172.21.15.37:3300/0 2024-04-26T00:29:19.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.466+0000 7f38c520a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f38c013a400 addr v1:172.21.15.45:6789/0 2024-04-26T00:29:19.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.466+0000 7f38c520a640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38c009c190 con 0x7f38c00a0300 2024-04-26T00:29:19.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.466+0000 7f38c520a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38c00a1110 con 0x7f38c013a400 2024-04-26T00:29:19.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.466+0000 7f38c520a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:19.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.466+0000 7f38c520a640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f38c00a16d0 con 0x7f38c009fc50 2024-04-26T00:29:19.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.466+0000 7f38c520a640 10 monclient(hunting): _renew_subs 2024-04-26T00:29:19.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.466+0000 7f38c520a640 10 monclient(hunting): authenticate will time out at 1540.774581s 2024-04-26T00:29:19.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.466+0000 7f38b7fff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f38c009fc50 0x7f38c0139ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:19.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.466+0000 7f38b7fff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f38c009fc50 0x7f38c0139ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33926/0 (socket says 172.21.15.37:33926) 2024-04-26T00:29:19.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.466+0000 7f38bed9d640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f38c00a0300 0x7f38c00a06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:46804/0 (socket says 172.21.15.37:46804) 2024-04-26T00:29:19.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.466+0000 7f38b7fff640 1 -- 172.21.15.37:0/379515043 learned_addr learned my addr 172.21.15.37:0/379515043 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:29:19.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.466+0000 7f38b7fff640 10 monclient(hunting): get_auth_request con 0x7f38c009fc50 auth_method 0 2024-04-26T00:29:19.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.466+0000 7f38b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:19.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.466+0000 7f38b7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:19.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.466+0000 7f38b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:19.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.466+0000 7f38b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:19.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.466+0000 7f38b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:19.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.466+0000 7f38b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:19.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.466+0000 7f38b7fff640 10 monclient(hunting): handle_auth_done global_id 94362 payload 274 2024-04-26T00:29:19.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.466+0000 7f38b7fff640 1 -- 172.21.15.37:0/379515043 >> v1:172.21.15.45:6789/0 conn(0x7f38c013a400 legacy=0x7f38c013c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:29:19.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.466+0000 7f38b7fff640 1 -- 172.21.15.37:0/379515043 >> v1:172.21.15.37:6789/0 conn(0x7f38c00a0300 legacy=0x7f38c00a06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:29:19.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.466+0000 7f38b7fff640 10 monclient: _finish_hunting 0 2024-04-26T00:29:19.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.466+0000 7f38b7fff640 1 monclient: found mon.noname-a 2024-04-26T00:29:19.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.466+0000 7f38b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:29:19.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.466+0000 7f38b7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:29:19.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.466+0000 7f38b7fff640 1 -- 172.21.15.37:0/379515043 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38c009f2a0 con 0x7f38c009fc50 2024-04-26T00:29:19.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.470+0000 7f38b7fff640 1 -- 172.21.15.37:0/379515043 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f38c009fc50 msgr2=0x7f38c0139ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:19.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.470+0000 7f38b7fff640 1 --2- 172.21.15.37:0/379515043 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f38c009fc50 0x7f38c0139ec0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f38c009f4d0 tx=0x7f38b001c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ee4cddea37e17cf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:19.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.470+0000 7f38be59c640 1 -- 172.21.15.37:0/379515043 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b00b0070 con 0x7f38c009fc50 2024-04-26T00:29:19.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.470+0000 7f38be59c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:19.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.470+0000 7f38be59c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:29:19.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.470+0000 7f38be59c640 10 monclient: dump: 2024-04-26T00:29:19.463 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:19.463 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:19.463 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.463 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.463 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:19.463 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:19.464 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:19.464 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:19.464 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:19.464 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:19.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.470+0000 7f38be59c640 10 monclient: _finish_auth 0 2024-04-26T00:29:19.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.470+0000 7f38be59c640 10 monclient: _check_auth_tickets 2024-04-26T00:29:19.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.470+0000 7f38be59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:29:19.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.470+0000 7f38be59c640 1 -- 172.21.15.37:0/379515043 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38b00ab600 con 0x7f38c009fc50 2024-04-26T00:29:19.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.470+0000 7f38c520a640 5 monclient: authenticate success, global_id 94362 2024-04-26T00:29:19.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.470+0000 7f38c520a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:29:19.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.470+0000 7f38be59c640 10 monclient: handle_config config(0 keys) 2024-04-26T00:29:19.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.470+0000 7f38c520a640 10 monclient: get_monmap_and_config success 2024-04-26T00:29:19.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.470+0000 7f38c520a640 4 set_mon_vals no callback set 2024-04-26T00:29:19.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.470+0000 7f38be59c640 1 -- 172.21.15.37:0/379515043 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b00ab9d0 con 0x7f38c009fc50 2024-04-26T00:29:19.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.470+0000 7f38be59c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:19.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.470+0000 7f38be59c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:29:19.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.470+0000 7f38be59c640 10 monclient: dump: 2024-04-26T00:29:19.465 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:19.465 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:19.465 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.465 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.465 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:19.466 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:19.466 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:19.466 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:19.466 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:19.466 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:19.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38c520a640 10 monclient: shutdown 2024-04-26T00:29:19.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38c520a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:29:19.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38c520a640 1 -- 172.21.15.37:0/379515043 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f38c009fc50 msgr2=0x7f38c0139ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:19.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38c520a640 1 --2- 172.21.15.37:0/379515043 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f38c009fc50 0x7f38c0139ec0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f38c009f4d0 tx=0x7f38b001c380 comp rx=0 tx=0).stop 2024-04-26T00:29:19.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38c520a640 1 -- 172.21.15.37:0/379515043 shutdown_connections 2024-04-26T00:29:19.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38c520a640 1 --2- 172.21.15.37:0/379515043 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f38c009fc50 0x7f38c0139ec0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:19.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38c520a640 1 -- 172.21.15.37:0/379515043 >> 172.21.15.37:0/379515043 conn(0x7f38c00a1d50 msgr2=0x7f38c00a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:29:19.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38c520a640 1 -- 172.21.15.37:0/379515043 shutdown_connections 2024-04-26T00:29:19.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38c520a640 1 -- 172.21.15.37:0/379515043 wait complete. 2024-04-26T00:29:19.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38c520a640 10 monclient: build_initial_monmap 2024-04-26T00:29:19.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38c520a640 10 monclient: monmap: 2024-04-26T00:29:19.467 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:29:19.467 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:29:19.467 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:29:19.467 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:29:19.467 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:29:19.467 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:29:19.467 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:29:19.467 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:29:19.467 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:29:19.467 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:19.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38c520a640 1 Processor -- start 2024-04-26T00:29:19.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38c520a640 1 -- start start 2024-04-26T00:29:19.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38c520a640 10 monclient: init 2024-04-26T00:29:19.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38c520a640 10 monclient: _reopen_session rank -1 2024-04-26T00:29:19.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38c520a640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:29:19.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38c520a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f38c009fc50 0x7f38c011b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:19.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38c520a640 10 monclient(hunting): picked mon.noname-a con 0x7f38c009fc50 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:29:19.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38c520a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f38c00a0300 0x7f38c0138ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:19.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38c520a640 10 monclient(hunting): picked mon.noname-c con 0x7f38c00a0300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:29:19.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38c520a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38c013a400 0x7f38c0134cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:19.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38c520a640 10 monclient(hunting): picked mon.noname-b con 0x7f38c013a400 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:29:19.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38c520a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:19.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38c520a640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f38c0140ac0 con 0x7f38c009fc50 2024-04-26T00:29:19.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38c520a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:19.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38c520a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f38c0140940 con 0x7f38c013a400 2024-04-26T00:29:19.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38c520a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:19.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38c520a640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f38c0140c40 con 0x7f38c00a0300 2024-04-26T00:29:19.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38c520a640 10 monclient(hunting): _renew_subs 2024-04-26T00:29:19.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38c520a640 10 monclient(hunting): authenticate will time out at 1540.782016s 2024-04-26T00:29:19.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38bed9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f38c009fc50 0x7f38c011b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:19.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38bed9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f38c009fc50 0x7f38c011b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33934/0 (socket says 172.21.15.37:33934) 2024-04-26T00:29:19.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38bed9d640 1 -- 172.21.15.37:0/69188380 learned_addr learned my addr 172.21.15.37:0/69188380 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:29:19.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38bed9d640 10 monclient(hunting): get_auth_request con 0x7f38c009fc50 auth_method 0 2024-04-26T00:29:19.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38bed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:19.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38bed9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:19.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38bed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:19.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38b7fff640 1 --2- 172.21.15.37:0/69188380 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f38c00a0300 0x7f38c0138ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:19.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38b7fff640 10 monclient(hunting): get_auth_request con 0x7f38c00a0300 auth_method 0 2024-04-26T00:29:19.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:19.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38b7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:19.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:19.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38bed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:19.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38bed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:19.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38bed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:19.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:19.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:19.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:19.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38bf59e640 1 --2- 172.21.15.37:0/69188380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38c013a400 0x7f38c0134cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:19.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38bf59e640 10 monclient(hunting): get_auth_request con 0x7f38c013a400 auth_method 0 2024-04-26T00:29:19.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38bf59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:19.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38bf59e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:19.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38bf59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:19.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38bed9d640 10 monclient(hunting): handle_auth_done global_id 94365 payload 824 2024-04-26T00:29:19.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38bed9d640 1 -- 172.21.15.37:0/69188380 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f38c00a0300 msgr2=0x7f38c0138ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:19.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38bed9d640 1 --2- 172.21.15.37:0/69188380 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f38c00a0300 0x7f38c0138ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:19.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38bed9d640 1 -- 172.21.15.37:0/69188380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38c013a400 msgr2=0x7f38c0134cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:19.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38bed9d640 1 --2- 172.21.15.37:0/69188380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38c013a400 0x7f38c0134cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:19.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38bed9d640 10 monclient: _finish_hunting 0 2024-04-26T00:29:19.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38bed9d640 1 monclient: found mon.noname-a 2024-04-26T00:29:19.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38bed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:29:19.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38bed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:29:19.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.474+0000 7f38bed9d640 1 -- 172.21.15.37:0/69188380 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38c009f550 con 0x7f38c009fc50 2024-04-26T00:29:19.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38bed9d640 1 -- 172.21.15.37:0/69188380 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f38c009fc50 msgr2=0x7f38c011b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:19.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38bed9d640 1 --2- 172.21.15.37:0/69188380 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f38c009fc50 0x7f38c011b8a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f38a800c3e0 tx=0x7f38a800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:19.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38b77fe640 1 -- 172.21.15.37:0/69188380 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a8017070 con 0x7f38c009fc50 2024-04-26T00:29:19.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:19.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38b77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:29:19.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38b77fe640 10 monclient: dump: 2024-04-26T00:29:19.477 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:19.477 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:19.477 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.477 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.477 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:19.477 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:19.477 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:19.477 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:19.477 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:19.477 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:19.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38b77fe640 10 monclient: _finish_auth 0 2024-04-26T00:29:19.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38b77fe640 10 monclient: _check_auth_tickets 2024-04-26T00:29:19.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:29:19.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38b77fe640 1 -- 172.21.15.37:0/69188380 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38a8005820 con 0x7f38c009fc50 2024-04-26T00:29:19.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38c520a640 5 monclient: authenticate success, global_id 94365 2024-04-26T00:29:19.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38c520a640 10 monclient: _renew_subs 2024-04-26T00:29:19.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38c520a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:29:19.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38c520a640 1 -- 172.21.15.37:0/69188380 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38c011d8d0 con 0x7f38c009fc50 2024-04-26T00:29:19.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38b77fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:29:19.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38b77fe640 1 -- 172.21.15.37:0/69188380 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a8005c10 con 0x7f38c009fc50 2024-04-26T00:29:19.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:19.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38b77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:29:19.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38b77fe640 10 monclient: dump: 2024-04-26T00:29:19.478 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:19.478 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:19.478 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.478 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.479 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:19.479 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:19.479 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:19.479 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:19.479 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:19.479 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:19.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38c520a640 10 monclient: _renew_subs 2024-04-26T00:29:19.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38c520a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:29:19.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38bdd9b640 4 set_mon_vals no callback set 2024-04-26T00:29:19.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38c520a640 1 -- 172.21.15.37:0/69188380 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38c0135210 con 0x7f38c009fc50 2024-04-26T00:29:19.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38b77fe640 1 -- 172.21.15.37:0/69188380 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f38a801a070 con 0x7f38c009fc50 2024-04-26T00:29:19.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38c520a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:29:19.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38c520a640 1 --2- 172.21.15.37:0/69188380 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f38c01cd720 0x7f38c002c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:19.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38c520a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:19.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38c520a640 1 -- 172.21.15.37:0/69188380 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f38c013a8e0 con 0x7f38c01cd720 2024-04-26T00:29:19.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38b7fff640 1 --2- 172.21.15.37:0/69188380 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f38c01cd720 0x7f38c002c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:19.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38b77fe640 1 --2- 172.21.15.37:0/69188380 >> v2:172.21.15.45:6812/2379918623 conn(0x7f38a0035f00 0x7f38a00383c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:19.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38b7fff640 10 monclient: get_auth_request con 0x7f38c01cd720 auth_method 0 2024-04-26T00:29:19.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:19.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38b7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:19.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:19.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38b77fe640 1 -- 172.21.15.37:0/69188380 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(371..371 src has 177..371) ==== 3959+0+0 (secure 0 0 0) 0x7f38a8047f50 con 0x7f38c009fc50 2024-04-26T00:29:19.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:19.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:19.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:19.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38b7fff640 10 monclient(hunting): handle_auth_done global_id 94368 payload 824 2024-04-26T00:29:19.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38b7fff640 1 -- 172.21.15.37:0/69188380 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f38c011ca30 con 0x7f38c01cd720 2024-04-26T00:29:19.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38bf59e640 1 --2- 172.21.15.37:0/69188380 >> v2:172.21.15.45:6812/2379918623 conn(0x7f38a0035f00 0x7f38a00383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:19.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.478+0000 7f38bf59e640 10 monclient: get_auth_request con 0x7f38a0035f00 auth_method 0 2024-04-26T00:29:19.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.482+0000 7f38b7fff640 1 -- 172.21.15.37:0/69188380 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f38c01cd720 msgr2=0x7f38c002c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:19.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.482+0000 7f38b7fff640 1 --2- 172.21.15.37:0/69188380 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f38c01cd720 0x7f38c002c000 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f38c00764d0 tx=0x7f38b0019120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:19.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.482+0000 7f38b77fe640 1 -- 172.21.15.37:0/69188380 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b00ba070 con 0x7f38c01cd720 2024-04-26T00:29:19.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.482+0000 7f38b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:19.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.482+0000 7f38b77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:29:19.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.482+0000 7f38b77fe640 10 monclient: dump: 2024-04-26T00:29:19.481 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:19.481 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:19.481 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.481 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.481 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:19.481 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:19.482 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:19.482 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:19.482 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:19.482 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:19.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.482+0000 7f38b77fe640 1 -- 172.21.15.37:0/69188380 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f38b00b1040 con 0x7f38c01cd720 2024-04-26T00:29:19.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.482+0000 7f38b77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:29:19.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.482+0000 7f38b77fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:29:19.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.482+0000 7f38b77fe640 1 -- 172.21.15.37:0/69188380 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f38c01cd720 msgr2=0x7f38c002c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:19.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.482+0000 7f38b77fe640 1 --2- 172.21.15.37:0/69188380 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f38c01cd720 0x7f38c002c000 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f38c00764d0 tx=0x7f38b0019120 comp rx=0 tx=0).stop 2024-04-26T00:29:19.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.482+0000 7f38b77fe640 1 -- 172.21.15.37:0/69188380 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f38c01cd720 msgr2=0x7f38c002c000 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:29:19.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.482+0000 7f38b77fe640 1 --2- 172.21.15.37:0/69188380 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f38c01cd720 0x7f38c002c000 secure :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f38c00764d0 tx=0x7f38b0019120 comp rx=0 tx=0).stop 2024-04-26T00:29:19.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.482+0000 7f38bf59e640 1 --2- 172.21.15.37:0/69188380 >> v2:172.21.15.45:6812/2379918623 conn(0x7f38a0035f00 0x7f38a00383c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f38c00a2dd0 tx=0x7f38b8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:19.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.486+0000 7f38c520a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:29:19.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.486+0000 7f38c520a640 1 --2- 172.21.15.37:0/69188380 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f38c01cdb00 0x7f38c002c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:19.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.486+0000 7f38c520a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:19.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.486+0000 7f38c520a640 1 -- 172.21.15.37:0/69188380 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f38c011df20 con 0x7f38c01cdb00 2024-04-26T00:29:19.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.486+0000 7f38b7fff640 1 --2- 172.21.15.37:0/69188380 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f38c01cdb00 0x7f38c002c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:19.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.486+0000 7f38b7fff640 10 monclient: get_auth_request con 0x7f38c01cdb00 auth_method 0 2024-04-26T00:29:19.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.486+0000 7f38b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:19.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.486+0000 7f38b7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:19.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.486+0000 7f38b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:19.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.486+0000 7f38b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:19.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.486+0000 7f38b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:19.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.486+0000 7f38b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:19.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.486+0000 7f38b7fff640 10 monclient(hunting): handle_auth_done global_id 94371 payload 824 2024-04-26T00:29:19.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.486+0000 7f38b7fff640 1 -- 172.21.15.37:0/69188380 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f38c004f510 con 0x7f38c01cdb00 2024-04-26T00:29:19.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.490+0000 7f38b7fff640 1 -- 172.21.15.37:0/69188380 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f38c01cdb00 msgr2=0x7f38c002c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:19.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.490+0000 7f38b7fff640 1 --2- 172.21.15.37:0/69188380 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f38c01cdb00 0x7f38c002c540 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f38b0019fb0 tx=0x7f38b00a8000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:19.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.490+0000 7f38b77fe640 1 -- 172.21.15.37:0/69188380 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b00ba070 con 0x7f38c01cdb00 2024-04-26T00:29:19.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.490+0000 7f38b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:19.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.490+0000 7f38b77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:29:19.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.490+0000 7f38b77fe640 10 monclient: dump: 2024-04-26T00:29:19.485 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:19.485 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:19.485 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.485 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.486 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:19.486 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:19.486 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:19.486 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:19.486 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:19.486 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:19.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.490+0000 7f38b77fe640 1 -- 172.21.15.37:0/69188380 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4549 (secure 0 0 0) 0x7f38b0003e10 con 0x7f38c01cdb00 2024-04-26T00:29:19.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.490+0000 7f38b77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:29:19.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.490+0000 7f38b77fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:29:19.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.490+0000 7f38b77fe640 1 -- 172.21.15.37:0/69188380 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f38c01cdb00 msgr2=0x7f38c002c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:19.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.490+0000 7f38b77fe640 1 --2- 172.21.15.37:0/69188380 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f38c01cdb00 0x7f38c002c540 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f38b0019fb0 tx=0x7f38b00a8000 comp rx=0 tx=0).stop 2024-04-26T00:29:19.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.490+0000 7f38b77fe640 1 -- 172.21.15.37:0/69188380 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f38c01cdb00 msgr2=0x7f38c002c540 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:29:19.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.490+0000 7f38b77fe640 1 --2- 172.21.15.37:0/69188380 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f38c01cdb00 0x7f38c002c540 secure :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f38b0019fb0 tx=0x7f38b00a8000 comp rx=0 tx=0).stop 2024-04-26T00:29:19.487 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:29:19.487 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:29:19.487 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:29:19.487 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:29:19.487 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 70, 2024-04-26T00:29:19.487 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 120250, 2024-04-26T00:29:19.487 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:29:19.487 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:29:19.487 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:29:19.487 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:29:19.487 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:19.487 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 4, 2024-04-26T00:29:19.488 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:29:19.488 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:29:19.488 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:29:19.488 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:29:19.488 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:29:19.488 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:29:19.488 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:29:19.488 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:29:19.488 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:29:19.488 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:29:19.488 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:29:19.488 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:29:19.488 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:29:19.488 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:29:19.489 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:19.489 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:29:19.489 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:29:19.489 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:29:19.489 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:29:19.489 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:29:19.489 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:29:19.489 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:29:19.489 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:29:19.489 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:29:19.489 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:29:19.490 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:29:19.490 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:29:19.490 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:29:19.490 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:19.490 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:19.490 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:29:19.490 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:29:19.490 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:29:19.490 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:29:19.490 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:29:19.490 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:29:19.490 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:29:19.491 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:29:19.491 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:29:19.491 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:29:19.491 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:29:19.491 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:29:19.491 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:29:19.491 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:29:19.491 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:29:19.491 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:29:19.491 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:29:19.491 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:29:19.491 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:29:19.491 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:29:19.491 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:29:19.492 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:29:19.492 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:29:19.492 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:29:19.492 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:29:19.492 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:29:19.492 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:29:19.492 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:29:19.492 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:19.492 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:29:19.492 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:19.492 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:29:19.492 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:19.492 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:29:19.493 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:29:19.493 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:29:19.493 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:29:19.493 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:19.493 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:29:19.493 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:29:19.493 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:19.493 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:19.493 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:19.493 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:29:19.493 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:29:19.493 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:19.493 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:19.494 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:19.494 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:19.494 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:29:19.494 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:29:19.494 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:29:19.494 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:29:19.494 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:29:19.494 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:19.494 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:19.494 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:29:19.494 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:29:19.494 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:29:19.494 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:29:19.495 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:19.495 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:29:19.495 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:29:19.495 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:19.495 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:19.495 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:19.495 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:29:19.495 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:29:19.495 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:19.495 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:19.495 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:19.495 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:19.495 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:29:19.495 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:29:19.495 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:29:19.496 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:29:19.496 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:29:19.496 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:19.496 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:19.496 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:29:19.496 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:29:19.496 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:29:19.496 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:29:19.496 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:19.496 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:29:19.496 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:29:19.496 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:19.496 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:19.496 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:19.496 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:29:19.497 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:29:19.497 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:19.497 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:19.497 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:19.497 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:19.497 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:29:19.497 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:29:19.497 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:29:19.497 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:29:19.497 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:29:19.497 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:19.497 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:19.497 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:19.497 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:29:19.497 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:29:19.498 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:19.498 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:29:19.498 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:29:19.498 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:29:19.498 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:19.498 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:19.498 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:29:19.498 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:19.498 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:29:19.498 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:29:19.498 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:29:19.498 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:19.498 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:19.499 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:19.499 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:29:19.499 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:29:19.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.490+0000 7f38c520a640 1 -- 172.21.15.37:0/69188380 >> v2:172.21.15.45:6812/2379918623 conn(0x7f38a0035f00 msgr2=0x7f38a00383c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:19.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.490+0000 7f38c520a640 1 --2- 172.21.15.37:0/69188380 >> v2:172.21.15.45:6812/2379918623 conn(0x7f38a0035f00 0x7f38a00383c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f38c00a2dd0 tx=0x7f38b8009040 comp rx=0 tx=0).stop 2024-04-26T00:29:19.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.490+0000 7f38c520a640 10 monclient: shutdown 2024-04-26T00:29:19.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.490+0000 7f38c520a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:29:19.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.490+0000 7f38c520a640 1 -- 172.21.15.37:0/69188380 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f38c009fc50 msgr2=0x7f38c011b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:19.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.490+0000 7f38c520a640 1 --2- 172.21.15.37:0/69188380 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f38c009fc50 0x7f38c011b8a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f38a800c3e0 tx=0x7f38a800c8a0 comp rx=0 tx=0).stop 2024-04-26T00:29:19.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.490+0000 7f38bf59e640 1 -- 172.21.15.37:0/69188380 reap_dead start 2024-04-26T00:29:19.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.490+0000 7f38bf59e640 1 -- 172.21.15.37:0/69188380 reap_dead start 2024-04-26T00:29:19.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.490+0000 7f38c520a640 1 -- 172.21.15.37:0/69188380 shutdown_connections 2024-04-26T00:29:19.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.490+0000 7f38c520a640 1 -- 172.21.15.37:0/69188380 >> 172.21.15.37:0/69188380 conn(0x7f38c00a1d50 msgr2=0x7f38c013c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:29:19.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.490+0000 7f38c520a640 1 -- 172.21.15.37:0/69188380 shutdown_connections 2024-04-26T00:29:19.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.490+0000 7f38c520a640 1 -- 172.21.15.37:0/69188380 wait complete. 2024-04-26T00:29:19.501 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:29:19.501 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:29:19.501 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:29:19.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e929c640 10 monclient: get_monmap_and_config 2024-04-26T00:29:19.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e929c640 10 monclient: build_initial_monmap 2024-04-26T00:29:19.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e929c640 10 monclient: monmap: 2024-04-26T00:29:19.608 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:29:19.608 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:19.608 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:29:19.621373+0000 2024-04-26T00:29:19.608 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:29:19.621373+0000 2024-04-26T00:29:19.608 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:29:19.608 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:19.608 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:29:19.608 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:29:19.608 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:29:19.608 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:29:19.608 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:29:19.609 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:19.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e929c640 1 Processor -- start 2024-04-26T00:29:19.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e929c640 1 -- start start 2024-04-26T00:29:19.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e929c640 10 monclient: init 2024-04-26T00:29:19.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e929c640 10 monclient: _reopen_session rank -1 2024-04-26T00:29:19.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e929c640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-04-26T00:29:19.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e929c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8e40a0300 0x7fc8e409fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:19.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e929c640 10 monclient(hunting): picked mon.noname-b con 0x7fc8e40a0300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:29:19.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e929c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc8e40a06e0 0x7fc8e4134ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:19.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e929c640 10 monclient(hunting): picked mon.noname-a con 0x7fc8e40a06e0 addr v2:172.21.15.37:3300/0 2024-04-26T00:29:19.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e929c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc8e41350e0 addr v1:172.21.15.45:6789/0 2024-04-26T00:29:19.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e929c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8e409c190 con 0x7fc8e41350e0 2024-04-26T00:29:19.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e929c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:19.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e929c640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fc8e40a13f0 con 0x7fc8e40a06e0 2024-04-26T00:29:19.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e929c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:19.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e929c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc8e40a1110 con 0x7fc8e40a0300 2024-04-26T00:29:19.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e929c640 10 monclient(hunting): _renew_subs 2024-04-26T00:29:19.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e929c640 10 monclient(hunting): authenticate will time out at 1540.926735s 2024-04-26T00:29:19.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e2ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8e40a0300 0x7fc8e409fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:19.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e27fc640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc8e40a06e0 0x7fc8e4134ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:19.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e27fc640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc8e40a06e0 0x7fc8e4134ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33968/0 (socket says 172.21.15.37:33968) 2024-04-26T00:29:19.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e27fc640 1 -- 172.21.15.37:0/1243018022 learned_addr learned my addr 172.21.15.37:0/1243018022 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:29:19.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e2ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8e40a0300 0x7fc8e409fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:41422/0 (socket says 172.21.15.37:41422) 2024-04-26T00:29:19.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e27fc640 10 monclient(hunting): get_auth_request con 0x7fc8e40a06e0 auth_method 0 2024-04-26T00:29:19.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:19.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e27fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:19.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:19.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e2ffd640 10 monclient(hunting): get_auth_request con 0x7fc8e40a0300 auth_method 0 2024-04-26T00:29:19.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:19.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:19.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:19.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:19.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:19.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:19.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:19.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:19.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:19.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e2ffd640 10 monclient(hunting): handle_auth_done global_id 84422 payload 274 2024-04-26T00:29:19.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e2ffd640 1 -- 172.21.15.37:0/1243018022 >> v2:172.21.15.37:3300/0 conn(0x7fc8e40a06e0 msgr2=0x7fc8e4134ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:29:19.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e2ffd640 1 --2- 172.21.15.37:0/1243018022 >> v2:172.21.15.37:3300/0 conn(0x7fc8e40a06e0 0x7fc8e4134ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:19.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e2ffd640 1 -- 172.21.15.37:0/1243018022 >> v1:172.21.15.45:6789/0 conn(0x7fc8e41350e0 legacy=0x7fc8e413bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:29:19.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e37fe640 1 -- 172.21.15.37:0/1243018022 >> v1:172.21.15.45:6789/0 conn(0x7fc8e41350e0 legacy=0x7fc8e413bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:29:19.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e37fe640 1 -- 172.21.15.37:0/1243018022 >> v1:172.21.15.45:6789/0 conn(0x7fc8e41350e0 legacy=0x7fc8e413bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:29:19.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e37fe640 1 --1- 172.21.15.37:0/1243018022 >> v1:172.21.15.45:6789/0 conn(0x7fc8e41350e0 0x7fc8e413bed0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-04-26T00:29:19.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e2ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:29:19.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e2ffd640 1 monclient: found mon.noname-b 2024-04-26T00:29:19.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:29:19.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:29:19.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.618+0000 7fc8e2ffd640 1 -- 172.21.15.37:0/1243018022 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8e40a16d0 con 0x7fc8e40a0300 2024-04-26T00:29:19.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.622+0000 7fc8e2ffd640 1 -- 172.21.15.37:0/1243018022 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8e40a0300 msgr2=0x7fc8e409fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:19.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.622+0000 7fc8e2ffd640 1 --2- 172.21.15.37:0/1243018022 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8e40a0300 0x7fc8e409fc50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc8d40097c0 tx=0x7fc8d401c350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5d9ed72d583e817 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:19.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.622+0000 7fc8e1ffb640 1 -- 172.21.15.37:0/1243018022 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d40b0070 con 0x7fc8e40a0300 2024-04-26T00:29:19.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.622+0000 7fc8e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:19.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.622+0000 7fc8e1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:29:19.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.622+0000 7fc8e1ffb640 10 monclient: dump: 2024-04-26T00:29:19.627 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:19.627 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:19.627 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.627 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.627 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:19.628 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:19.628 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:19.628 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:19.628 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:19.628 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:19.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.622+0000 7fc8e1ffb640 10 monclient: _finish_auth 0 2024-04-26T00:29:19.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.622+0000 7fc8e1ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:29:19.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.622+0000 7fc8e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:29:19.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.622+0000 7fc8e1ffb640 1 -- 172.21.15.37:0/1243018022 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8d40ab600 con 0x7fc8e40a0300 2024-04-26T00:29:19.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.622+0000 7fc8e929c640 5 monclient: authenticate success, global_id 84422 2024-04-26T00:29:19.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.622+0000 7fc8e929c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:29:19.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.622+0000 7fc8e1ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:29:19.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.622+0000 7fc8e929c640 10 monclient: get_monmap_and_config success 2024-04-26T00:29:19.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.622+0000 7fc8e1ffb640 1 -- 172.21.15.37:0/1243018022 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d40ab9d0 con 0x7fc8e40a0300 2024-04-26T00:29:19.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.622+0000 7fc8e929c640 4 set_mon_vals no callback set 2024-04-26T00:29:19.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.622+0000 7fc8e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:19.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.622+0000 7fc8e1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:29:19.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.622+0000 7fc8e1ffb640 10 monclient: dump: 2024-04-26T00:29:19.629 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:19.629 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:19.629 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.629 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.629 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:19.629 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:19.629 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:19.630 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:19.630 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:19.630 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:19.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e929c640 10 monclient: shutdown 2024-04-26T00:29:19.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e929c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:29:19.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e929c640 1 -- 172.21.15.37:0/1243018022 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8e40a0300 msgr2=0x7fc8e409fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:19.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e929c640 1 --2- 172.21.15.37:0/1243018022 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8e40a0300 0x7fc8e409fc50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc8d40097c0 tx=0x7fc8d401c350 comp rx=0 tx=0).stop 2024-04-26T00:29:19.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e929c640 1 -- 172.21.15.37:0/1243018022 shutdown_connections 2024-04-26T00:29:19.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e929c640 1 --2- 172.21.15.37:0/1243018022 >> v2:172.21.15.37:3300/0 conn(0x7fc8e40a06e0 0x7fc8e4134ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:19.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e929c640 1 --2- 172.21.15.37:0/1243018022 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8e40a0300 0x7fc8e409fc50 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:19.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e929c640 1 -- 172.21.15.37:0/1243018022 >> 172.21.15.37:0/1243018022 conn(0x7fc8e40a1d50 msgr2=0x7fc8e40a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:29:19.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e929c640 1 -- 172.21.15.37:0/1243018022 shutdown_connections 2024-04-26T00:29:19.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e929c640 1 -- 172.21.15.37:0/1243018022 wait complete. 2024-04-26T00:29:19.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e929c640 10 monclient: build_initial_monmap 2024-04-26T00:29:19.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e929c640 10 monclient: monmap: 2024-04-26T00:29:19.631 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:29:19.631 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:29:19.631 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:29:19.632 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:29:19.632 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:29:19.632 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:29:19.632 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:29:19.632 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:29:19.632 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:29:19.632 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:19.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e929c640 1 Processor -- start 2024-04-26T00:29:19.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e929c640 1 -- start start 2024-04-26T00:29:19.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e929c640 10 monclient: init 2024-04-26T00:29:19.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e929c640 10 monclient: _reopen_session rank -1 2024-04-26T00:29:19.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e929c640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:29:19.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e929c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8e40a0300 0x7fc8e4070ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:19.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e929c640 10 monclient(hunting): picked mon.noname-c con 0x7fc8e40a0300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:29:19.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e929c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc8e40a06e0 0x7fc8e4131b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:19.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e929c640 10 monclient(hunting): picked mon.noname-a con 0x7fc8e40a06e0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:29:19.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e929c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8e41350e0 0x7fc8e41320c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:19.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e929c640 10 monclient(hunting): picked mon.noname-b con 0x7fc8e41350e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:29:19.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e929c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:19.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e929c640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e40a0c40 con 0x7fc8e40a06e0 2024-04-26T00:29:19.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e929c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:19.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e929c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e40a0ac0 con 0x7fc8e41350e0 2024-04-26T00:29:19.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e929c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:19.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e929c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc8e413fe10 con 0x7fc8e40a0300 2024-04-26T00:29:19.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e929c640 10 monclient(hunting): _renew_subs 2024-04-26T00:29:19.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e929c640 10 monclient(hunting): authenticate will time out at 1540.933615s 2024-04-26T00:29:19.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e2ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8e40a0300 0x7fc8e4070ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:19.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e2ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8e40a0300 0x7fc8e4070ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:41426/0 (socket says 172.21.15.37:41426) 2024-04-26T00:29:19.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e2ffd640 1 -- 172.21.15.37:0/444733128 learned_addr learned my addr 172.21.15.37:0/444733128 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:29:19.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e2ffd640 10 monclient(hunting): get_auth_request con 0x7fc8e40a0300 auth_method 0 2024-04-26T00:29:19.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:19.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:19.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:19.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e27fc640 1 --2- 172.21.15.37:0/444733128 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc8e40a06e0 0x7fc8e4131b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:19.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e27fc640 10 monclient(hunting): get_auth_request con 0x7fc8e40a06e0 auth_method 0 2024-04-26T00:29:19.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:19.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e27fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:19.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:19.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:19.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:19.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:19.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:19.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:19.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e37fe640 1 --2- 172.21.15.37:0/444733128 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8e41350e0 0x7fc8e41320c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:19.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:19.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e37fe640 10 monclient(hunting): get_auth_request con 0x7fc8e41350e0 auth_method 0 2024-04-26T00:29:19.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:19.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:19.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:19.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e2ffd640 10 monclient(hunting): handle_auth_done global_id 84425 payload 824 2024-04-26T00:29:19.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e2ffd640 1 -- 172.21.15.37:0/444733128 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8e41350e0 msgr2=0x7fc8e41320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:19.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e2ffd640 1 --2- 172.21.15.37:0/444733128 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc8e41350e0 0x7fc8e41320c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:19.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e2ffd640 1 -- 172.21.15.37:0/444733128 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc8e40a06e0 msgr2=0x7fc8e4131b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:19.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e2ffd640 1 --2- 172.21.15.37:0/444733128 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc8e40a06e0 0x7fc8e4131b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:19.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e2ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:29:19.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e2ffd640 1 monclient: found mon.noname-c 2024-04-26T00:29:19.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:29:19.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e2ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:29:19.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.626+0000 7fc8e2ffd640 1 -- 172.21.15.37:0/444733128 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8e411dc90 con 0x7fc8e40a0300 2024-04-26T00:29:19.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8e2ffd640 1 -- 172.21.15.37:0/444733128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8e40a0300 msgr2=0x7fc8e4070ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:19.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8e2ffd640 1 --2- 172.21.15.37:0/444733128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8e40a0300 0x7fc8e4070ca0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc8d401aa60 tx=0x7fc8d4019150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:19.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8db7fe640 1 -- 172.21.15.37:0/444733128 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d40b9020 con 0x7fc8e40a0300 2024-04-26T00:29:19.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:19.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8db7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:29:19.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8db7fe640 10 monclient: dump: 2024-04-26T00:29:19.637 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:19.637 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:19.637 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.637 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.637 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:19.637 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:19.637 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:19.637 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:19.637 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:19.637 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:19.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8db7fe640 10 monclient: _finish_auth 0 2024-04-26T00:29:19.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8db7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:29:19.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:29:19.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8db7fe640 1 -- 172.21.15.37:0/444733128 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8d40b5690 con 0x7fc8e40a0300 2024-04-26T00:29:19.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8db7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:29:19.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8db7fe640 1 -- 172.21.15.37:0/444733128 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d40b0040 con 0x7fc8e40a0300 2024-04-26T00:29:19.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:19.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8db7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:29:19.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8db7fe640 10 monclient: dump: 2024-04-26T00:29:19.638 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:19.638 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:19.638 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.638 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.638 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:19.638 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:19.638 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:19.638 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:19.639 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:19.639 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:19.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8e17fa640 4 set_mon_vals no callback set 2024-04-26T00:29:19.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8e929c640 5 monclient: authenticate success, global_id 84425 2024-04-26T00:29:19.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8e929c640 10 monclient: _renew_subs 2024-04-26T00:29:19.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8e929c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:29:19.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8e929c640 1 -- 172.21.15.37:0/444733128 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8e411bd80 con 0x7fc8e40a0300 2024-04-26T00:29:19.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8e929c640 10 monclient: _renew_subs 2024-04-26T00:29:19.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8e929c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:29:19.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8e929c640 1 -- 172.21.15.37:0/444733128 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8e411d190 con 0x7fc8e40a0300 2024-04-26T00:29:19.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8e929c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:29:19.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8e929c640 1 --2- 172.21.15.37:0/444733128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8e41cf150 0x7fc8e41cf530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:19.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8e929c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:19.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8e929c640 1 -- 172.21.15.37:0/444733128 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc8e4071930 con 0x7fc8e41cf150 2024-04-26T00:29:19.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8db7fe640 1 -- 172.21.15.37:0/444733128 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc8d40bc020 con 0x7fc8e40a0300 2024-04-26T00:29:19.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8e27fc640 1 --2- 172.21.15.37:0/444733128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8e41cf150 0x7fc8e41cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:19.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8e27fc640 10 monclient: get_auth_request con 0x7fc8e41cf150 auth_method 0 2024-04-26T00:29:19.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8e27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:19.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8e27fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:19.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:19.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8e27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:19.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8e27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:19.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8e27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:19.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8db7fe640 1 --2- 172.21.15.37:0/444733128 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc8cc035ed0 0x7fc8cc038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:19.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8db7fe640 1 -- 172.21.15.37:0/444733128 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(371..371 src has 177..371) ==== 3959+0+0 (secure 0 0 0) 0x7fc8d40e29a0 con 0x7fc8e40a0300 2024-04-26T00:29:19.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8e27fc640 10 monclient(hunting): handle_auth_done global_id 84428 payload 824 2024-04-26T00:29:19.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8e27fc640 1 -- 172.21.15.37:0/444733128 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc8e404f510 con 0x7fc8e41cf150 2024-04-26T00:29:19.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8e37fe640 1 --2- 172.21.15.37:0/444733128 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc8cc035ed0 0x7fc8cc038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:19.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.630+0000 7fc8e37fe640 10 monclient: get_auth_request con 0x7fc8cc035ed0 auth_method 0 2024-04-26T00:29:19.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.634+0000 7fc8e27fc640 1 -- 172.21.15.37:0/444733128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8e41cf150 msgr2=0x7fc8e41cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:19.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.634+0000 7fc8e27fc640 1 --2- 172.21.15.37:0/444733128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8e41cf150 0x7fc8e41cf530 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4071530 tx=0x7fc8d0011d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:19.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.634+0000 7fc8db7fe640 1 -- 172.21.15.37:0/444733128 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d000fe20 con 0x7fc8e41cf150 2024-04-26T00:29:19.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.634+0000 7fc8db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:19.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.634+0000 7fc8db7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:29:19.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.634+0000 7fc8db7fe640 10 monclient: dump: 2024-04-26T00:29:19.641 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:19.641 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:19.641 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.641 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.641 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:19.642 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:19.642 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:19.642 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:19.642 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:19.642 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:19.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.634+0000 7fc8db7fe640 1 -- 172.21.15.37:0/444733128 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc8d000fe20 con 0x7fc8e41cf150 2024-04-26T00:29:19.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.634+0000 7fc8db7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:29:19.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.634+0000 7fc8db7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:29:19.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.634+0000 7fc8db7fe640 1 -- 172.21.15.37:0/444733128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8e41cf150 msgr2=0x7fc8e41cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:19.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.634+0000 7fc8db7fe640 1 --2- 172.21.15.37:0/444733128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8e41cf150 0x7fc8e41cf530 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4071530 tx=0x7fc8d0011d60 comp rx=0 tx=0).stop 2024-04-26T00:29:19.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.634+0000 7fc8db7fe640 1 -- 172.21.15.37:0/444733128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8e41cf150 msgr2=0x7fc8e41cf530 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:29:19.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.634+0000 7fc8db7fe640 1 --2- 172.21.15.37:0/444733128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8e41cf150 0x7fc8e41cf530 secure :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4071530 tx=0x7fc8d0011d60 comp rx=0 tx=0).stop 2024-04-26T00:29:19.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.634+0000 7fc8e37fe640 1 --2- 172.21.15.37:0/444733128 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc8cc035ed0 0x7fc8cc038390 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc003a70 tx=0x7fc8dc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:19.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.638+0000 7fc8e929c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:29:19.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.638+0000 7fc8e929c640 1 --2- 172.21.15.37:0/444733128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8e41cfa70 0x7fc8e4133180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:19.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.638+0000 7fc8e929c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:19.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.638+0000 7fc8e929c640 1 -- 172.21.15.37:0/444733128 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc8e409eb20 con 0x7fc8e41cfa70 2024-04-26T00:29:19.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.638+0000 7fc8e27fc640 1 --2- 172.21.15.37:0/444733128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8e41cfa70 0x7fc8e4133180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:19.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.638+0000 7fc8e27fc640 10 monclient: get_auth_request con 0x7fc8e41cfa70 auth_method 0 2024-04-26T00:29:19.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.638+0000 7fc8e27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:19.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.638+0000 7fc8e27fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:19.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.638+0000 7fc8e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:19.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.638+0000 7fc8e27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:19.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.638+0000 7fc8e27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:19.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.638+0000 7fc8e27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:19.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.638+0000 7fc8e27fc640 10 monclient(hunting): handle_auth_done global_id 84431 payload 824 2024-04-26T00:29:19.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.638+0000 7fc8e27fc640 1 -- 172.21.15.37:0/444733128 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc8e411de60 con 0x7fc8e41cfa70 2024-04-26T00:29:19.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.642+0000 7fc8e27fc640 1 -- 172.21.15.37:0/444733128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8e41cfa70 msgr2=0x7fc8e4133180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:19.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.642+0000 7fc8e27fc640 1 --2- 172.21.15.37:0/444733128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8e41cfa70 0x7fc8e4133180 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4071530 tx=0x7fc8d00046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:19.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.642+0000 7fc8db7fe640 1 -- 172.21.15.37:0/444733128 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d0014640 con 0x7fc8e41cfa70 2024-04-26T00:29:19.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.642+0000 7fc8db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:19.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.642+0000 7fc8db7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:29:19.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.642+0000 7fc8db7fe640 10 monclient: dump: 2024-04-26T00:29:19.646 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:19.646 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:19.646 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.647 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.647 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:19.647 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:19.647 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:19.647 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:19.647 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:19.647 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:19.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.642+0000 7fc8db7fe640 1 -- 172.21.15.37:0/444733128 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7fc8d00273a0 con 0x7fc8e41cfa70 2024-04-26T00:29:19.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.642+0000 7fc8db7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:29:19.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.642+0000 7fc8db7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:29:19.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.642+0000 7fc8db7fe640 1 -- 172.21.15.37:0/444733128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8e41cfa70 msgr2=0x7fc8e4133180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:19.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.642+0000 7fc8db7fe640 1 --2- 172.21.15.37:0/444733128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8e41cfa70 0x7fc8e4133180 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4071530 tx=0x7fc8d00046d0 comp rx=0 tx=0).stop 2024-04-26T00:29:19.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.642+0000 7fc8db7fe640 1 -- 172.21.15.37:0/444733128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8e41cfa70 msgr2=0x7fc8e4133180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:29:19.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.642+0000 7fc8db7fe640 1 --2- 172.21.15.37:0/444733128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8e41cfa70 0x7fc8e4133180 secure :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4071530 tx=0x7fc8d00046d0 comp rx=0 tx=0).stop 2024-04-26T00:29:19.648 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:29:19.648 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:29:19.648 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:29:19.648 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:29:19.648 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 70, 2024-04-26T00:29:19.648 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 164577, 2024-04-26T00:29:19.648 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:29:19.648 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:29:19.648 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:29:19.648 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:29:19.649 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:19.649 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 4, 2024-04-26T00:29:19.649 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:29:19.649 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:29:19.649 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:29:19.649 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:29:19.649 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:29:19.649 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:29:19.649 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:29:19.649 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:29:19.649 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:29:19.649 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:29:19.649 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:29:19.649 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:29:19.649 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:29:19.650 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:29:19.650 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:19.650 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:29:19.650 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:29:19.650 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:29:19.650 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:29:19.650 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:29:19.650 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:29:19.650 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:29:19.650 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:29:19.650 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:29:19.650 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:29:19.650 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:29:19.651 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:29:19.651 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:29:19.651 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:19.651 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:19.651 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:29:19.651 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:29:19.651 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:29:19.651 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:29:19.651 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:29:19.651 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:29:19.651 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:29:19.651 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:29:19.651 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:29:19.651 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:29:19.652 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:29:19.652 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:29:19.652 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:29:19.652 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:29:19.652 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:29:19.652 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:29:19.652 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:29:19.652 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:29:19.652 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:29:19.652 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:29:19.652 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:29:19.652 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:29:19.652 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:29:19.653 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:29:19.653 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:29:19.653 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:29:19.653 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:29:19.653 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:29:19.653 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:19.653 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:29:19.653 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:19.653 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:29:19.653 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:19.653 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:29:19.654 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:29:19.654 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:29:19.654 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:29:19.654 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:19.654 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:29:19.654 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:29:19.654 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:19.654 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:19.654 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:19.654 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:29:19.654 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:29:19.654 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:19.654 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:19.655 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:19.655 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:19.655 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:29:19.655 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:29:19.655 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:29:19.655 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:29:19.655 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:29:19.655 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:19.655 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:19.655 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:29:19.655 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:29:19.655 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:29:19.655 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:29:19.655 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:19.655 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:29:19.656 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:29:19.656 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:19.656 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:19.656 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:19.656 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:29:19.656 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:29:19.656 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:19.656 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:19.656 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:19.656 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:19.656 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:29:19.656 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:29:19.656 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:29:19.656 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:29:19.657 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:29:19.657 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:19.657 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:19.657 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:29:19.657 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:29:19.657 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:29:19.657 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:29:19.657 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:19.657 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:29:19.657 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:29:19.657 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:19.657 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:19.657 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:19.657 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:29:19.658 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:29:19.658 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:19.658 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:19.658 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:19.658 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:19.658 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:29:19.658 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:29:19.658 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:29:19.658 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:29:19.658 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:29:19.658 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:19.658 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:19.658 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:19.658 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:29:19.659 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:29:19.659 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:19.659 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:29:19.659 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:29:19.659 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:29:19.659 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:19.659 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:19.659 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:29:19.659 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:19.659 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:29:19.659 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:29:19.659 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:29:19.659 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:19.659 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:19.660 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:29:19.660 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:19.660 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:29:19.660 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:29:19.660 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:29:19.660 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:19.660 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:19.660 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:29:19.660 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:19.660 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:29:19.660 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:29:19.660 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:29:19.660 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:19.660 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:19.661 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:19.661 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:29:19.661 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:29:19.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.642+0000 7fc8e929c640 1 -- 172.21.15.37:0/444733128 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc8cc035ed0 msgr2=0x7fc8cc038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:19.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.642+0000 7fc8e929c640 1 --2- 172.21.15.37:0/444733128 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc8cc035ed0 0x7fc8cc038390 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc003a70 tx=0x7fc8dc009040 comp rx=0 tx=0).stop 2024-04-26T00:29:19.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.642+0000 7fc8e929c640 10 monclient: shutdown 2024-04-26T00:29:19.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.642+0000 7fc8e929c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:29:19.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.642+0000 7fc8e929c640 1 -- 172.21.15.37:0/444733128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8e40a0300 msgr2=0x7fc8e4070ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:19.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.642+0000 7fc8e929c640 1 --2- 172.21.15.37:0/444733128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8e40a0300 0x7fc8e4070ca0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc8d401aa60 tx=0x7fc8d4019150 comp rx=0 tx=0).stop 2024-04-26T00:29:19.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.642+0000 7fc8e37fe640 1 -- 172.21.15.37:0/444733128 reap_dead start 2024-04-26T00:29:19.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.642+0000 7fc8e37fe640 1 -- 172.21.15.37:0/444733128 reap_dead start 2024-04-26T00:29:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.642+0000 7fc8e929c640 1 -- 172.21.15.37:0/444733128 shutdown_connections 2024-04-26T00:29:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.642+0000 7fc8e929c640 1 -- 172.21.15.37:0/444733128 >> 172.21.15.37:0/444733128 conn(0x7fc8e40a1d50 msgr2=0x7fc8e40a3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:29:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.642+0000 7fc8e929c640 1 -- 172.21.15.37:0/444733128 shutdown_connections 2024-04-26T00:29:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.642+0000 7fc8e929c640 1 -- 172.21.15.37:0/444733128 wait complete. 2024-04-26T00:29:19.664 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:29:19.664 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:29:19.664 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:29:19.698 INFO:tasks.rados.rados.0.smithi045.stdout:1615: finishing write tid 3 to smithi04526579-37 2024-04-26T00:29:19.698 INFO:tasks.rados.rados.0.smithi045.stdout:1615: finishing write tid 4 to smithi04526579-37 2024-04-26T00:29:19.698 INFO:tasks.rados.rados.0.smithi045.stdout:1615: oid 37 updating version 0 to 1293 2024-04-26T00:29:19.698 INFO:tasks.rados.rados.0.smithi045.stdout:1615: oid 37 updating version 1293 to 1295 2024-04-26T00:29:19.699 INFO:tasks.rados.rados.0.smithi045.stdout:1615: oid 37 version 1295 is already newer than 1294 2024-04-26T00:29:19.699 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 1295 (ObjNum 601 snap 186 seq_num 601) dirty exists 2024-04-26T00:29:19.699 INFO:tasks.rados.rados.0.smithi045.stdout:1615: left oid 37 (ObjNum 601 snap 186 seq_num 601) 2024-04-26T00:29:19.699 INFO:tasks.rados.rados.0.smithi045.stdout:1619: finishing rollback tid 1 to smithi04526579-24 2024-04-26T00:29:19.699 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 1203 (ObjNum 129 snap 0 seq_num 2242211168) dirty dne 2024-04-26T00:29:19.699 INFO:tasks.rados.rados.0.smithi045.stdout:1616: finishing copy_from racing read to smithi04526579-38 2024-04-26T00:29:19.699 INFO:tasks.rados.rados.0.smithi045.stdout:1621: finishing copy_from to smithi04526579-49 2024-04-26T00:29:19.699 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 971 (ObjNum 591 snap 183 seq_num 591) dirty exists 2024-04-26T00:29:19.699 INFO:tasks.rados.rados.0.smithi045.stdout:1623: finishing write tid 1 to smithi04526579-9 2024-04-26T00:29:19.699 INFO:tasks.rados.rados.0.smithi045.stdout:1616: finishing copy_from to smithi04526579-38 2024-04-26T00:29:19.699 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 973 (ObjNum 534 snap 165 seq_num 534) dirty exists 2024-04-26T00:29:19.699 INFO:tasks.rados.rados.0.smithi045.stdout:1623: finishing write tid 2 to smithi04526579-9 2024-04-26T00:29:19.699 INFO:tasks.rados.rados.0.smithi045.stdout:1623: finishing write tid 3 to smithi04526579-9 2024-04-26T00:29:19.699 INFO:tasks.rados.rados.0.smithi045.stdout:1623: finishing write tid 4 to smithi04526579-9 2024-04-26T00:29:19.700 INFO:tasks.rados.rados.0.smithi045.stdout:1623: oid 9 updating version 0 to 885 2024-04-26T00:29:19.700 INFO:tasks.rados.rados.0.smithi045.stdout:1623: oid 9 updating version 885 to 886 2024-04-26T00:29:19.700 INFO:tasks.rados.rados.0.smithi045.stdout:1623: oid 9 version 886 is already newer than 884 2024-04-26T00:29:19.700 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 886 (ObjNum 602 snap 186 seq_num 602) dirty exists 2024-04-26T00:29:19.700 INFO:tasks.rados.rados.0.smithi045.stdout:1623: left oid 9 (ObjNum 602 snap 186 seq_num 602) 2024-04-26T00:29:19.700 INFO:tasks.rados.rados.0.smithi045.stdout:1620: finishing copy_from racing read to smithi04526579-18 2024-04-26T00:29:19.700 INFO:tasks.rados.rados.0.smithi045.stdout:1620: finishing copy_from to smithi04526579-18 2024-04-26T00:29:19.700 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 869 (ObjNum 551 snap 168 seq_num 551) dirty exists 2024-04-26T00:29:19.700 INFO:tasks.rados.rados.0.smithi045.stdout:1624: finishing write tid 1 to smithi04526579-3 2024-04-26T00:29:19.700 INFO:tasks.rados.rados.0.smithi045.stdout:1624: finishing write tid 2 to smithi04526579-3 2024-04-26T00:29:19.700 INFO:tasks.rados.rados.0.smithi045.stdout:1624: finishing write tid 3 to smithi04526579-3 2024-04-26T00:29:19.700 INFO:tasks.rados.rados.0.smithi045.stdout:1626: seq_num 604 ranges {551428=642283,1694771=456657} 2024-04-26T00:29:19.712 INFO:tasks.rados.rados.0.smithi045.stdout:1626: writing smithi04526579-25 from 551428 to 1193711 tid 1 2024-04-26T00:29:19.723 INFO:tasks.rados.rados.0.smithi045.stdout:1626: writing smithi04526579-25 from 1694771 to 2151428 tid 2 2024-04-26T00:29:19.723 INFO:tasks.rados.rados.0.smithi045.stdout:1614: done (12 left) 2024-04-26T00:29:19.723 INFO:tasks.rados.rados.0.smithi045.stdout:1615: done (11 left) 2024-04-26T00:29:19.723 INFO:tasks.rados.rados.0.smithi045.stdout:1616: done (10 left) 2024-04-26T00:29:19.724 INFO:tasks.rados.rados.0.smithi045.stdout:1617: done (9 left) 2024-04-26T00:29:19.724 INFO:tasks.rados.rados.0.smithi045.stdout:1618: done (8 left) 2024-04-26T00:29:19.724 INFO:tasks.rados.rados.0.smithi045.stdout:1619: done (7 left) 2024-04-26T00:29:19.724 INFO:tasks.rados.rados.0.smithi045.stdout:1620: done (6 left) 2024-04-26T00:29:19.724 INFO:tasks.rados.rados.0.smithi045.stdout:1621: done (5 left) 2024-04-26T00:29:19.724 INFO:tasks.rados.rados.0.smithi045.stdout:1622: done (4 left) 2024-04-26T00:29:19.724 INFO:tasks.rados.rados.0.smithi045.stdout:1623: done (3 left) 2024-04-26T00:29:19.724 INFO:tasks.rados.rados.0.smithi045.stdout:1627: write oid 26 current snap is 186 2024-04-26T00:29:19.724 INFO:tasks.rados.rados.0.smithi045.stdout:1624: finishing write tid 4 to smithi04526579-3 2024-04-26T00:29:19.724 INFO:tasks.rados.rados.0.smithi045.stdout:1624: oid 3 updating version 0 to 972 2024-04-26T00:29:19.724 INFO:tasks.rados.rados.0.smithi045.stdout:1624: oid 3 updating version 972 to 974 2024-04-26T00:29:19.724 INFO:tasks.rados.rados.0.smithi045.stdout:1624: oid 3 version 974 is already newer than 973 2024-04-26T00:29:19.724 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 974 (ObjNum 603 snap 186 seq_num 603) dirty exists 2024-04-26T00:29:19.724 INFO:tasks.rados.rados.0.smithi045.stdout:1624: left oid 3 (ObjNum 603 snap 186 seq_num 603) 2024-04-26T00:29:19.725 INFO:tasks.rados.rados.0.smithi045.stdout:1625: expect (ObjNum 558 snap 173 seq_num 558) 2024-04-26T00:29:19.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef9d87640 10 monclient: get_monmap_and_config 2024-04-26T00:29:19.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef9d87640 10 monclient: build_initial_monmap 2024-04-26T00:29:19.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef9d87640 10 monclient: monmap: 2024-04-26T00:29:19.779 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:29:19.779 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:19.779 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:29:19.792988+0000 2024-04-26T00:29:19.779 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:29:19.792988+0000 2024-04-26T00:29:19.779 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:29:19.779 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:19.779 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:29:19.779 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:29:19.779 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:29:19.779 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:29:19.780 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:29:19.780 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:19.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef9d87640 1 Processor -- start 2024-04-26T00:29:19.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef9d87640 1 -- start start 2024-04-26T00:29:19.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef9d87640 10 monclient: init 2024-04-26T00:29:19.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef9d87640 10 monclient: _reopen_session rank -1 2024-04-26T00:29:19.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef9d87640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-04-26T00:29:19.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef9d87640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fcef4130eb0 0x7fcef409ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:19.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef9d87640 10 monclient(hunting): picked mon.noname-a con 0x7fcef4130eb0 addr v2:172.21.15.37:3300/0 2024-04-26T00:29:19.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef9d87640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fcef4131290 0x7fcef4133b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:19.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef9d87640 10 monclient(hunting): picked mon.noname-c con 0x7fcef4131290 addr v2:172.21.15.45:3300/0 2024-04-26T00:29:19.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef9d87640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcef41340d0 0x7fcef4134580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:19.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef9d87640 10 monclient(hunting): picked mon.noname-b con 0x7fcef41340d0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:29:19.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef9d87640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:19.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef9d87640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fcef409df60 con 0x7fcef4130eb0 2024-04-26T00:29:19.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef9d87640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:19.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef9d87640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fcef409c190 con 0x7fcef4131290 2024-04-26T00:29:19.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef9d87640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:19.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef9d87640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fcef40a11c0 con 0x7fcef41340d0 2024-04-26T00:29:19.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef9d87640 10 monclient(hunting): _renew_subs 2024-04-26T00:29:19.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef9d87640 10 monclient(hunting): authenticate will time out at 1541.098653s 2024-04-26T00:29:19.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef3fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcef41340d0 0x7fcef4134580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:19.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef3fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcef41340d0 0x7fcef4134580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:55036/0 (socket says 172.21.15.37:55036) 2024-04-26T00:29:19.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef37fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fcef4130eb0 0x7fcef409ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:19.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef3fff640 1 -- 172.21.15.37:0/316569000 learned_addr learned my addr 172.21.15.37:0/316569000 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:29:19.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef3fff640 10 monclient(hunting): get_auth_request con 0x7fcef41340d0 auth_method 0 2024-04-26T00:29:19.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:19.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:19.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:19.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef37fe640 10 monclient(hunting): get_auth_request con 0x7fcef4130eb0 auth_method 0 2024-04-26T00:29:19.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:19.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:19.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:19.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:19.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:19.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef2ffd640 1 --2- 172.21.15.37:0/316569000 >> v2:172.21.15.45:3300/0 conn(0x7fcef4131290 0x7fcef4133b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:19.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:19.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:19.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:19.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:19.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef2ffd640 10 monclient(hunting): get_auth_request con 0x7fcef4131290 auth_method 0 2024-04-26T00:29:19.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:19.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:19.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:19.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef37fe640 10 monclient(hunting): handle_auth_done global_id 94380 payload 274 2024-04-26T00:29:19.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef37fe640 1 -- 172.21.15.37:0/316569000 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcef41340d0 msgr2=0x7fcef4134580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:29:19.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef37fe640 1 --2- 172.21.15.37:0/316569000 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcef41340d0 0x7fcef4134580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:19.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef37fe640 1 -- 172.21.15.37:0/316569000 >> v2:172.21.15.45:3300/0 conn(0x7fcef4131290 msgr2=0x7fcef4133b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:29:19.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef37fe640 1 --2- 172.21.15.37:0/316569000 >> v2:172.21.15.45:3300/0 conn(0x7fcef4131290 0x7fcef4133b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:19.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef37fe640 10 monclient: _finish_hunting 0 2024-04-26T00:29:19.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef37fe640 1 monclient: found mon.noname-a 2024-04-26T00:29:19.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:29:19.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:29:19.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef37fe640 1 -- 172.21.15.37:0/316569000 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcef409dc80 con 0x7fcef4130eb0 2024-04-26T00:29:19.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.790+0000 7fcef3fff640 1 --2- 172.21.15.37:0/316569000 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcef41340d0 0x7fcef4134580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:29:19.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.794+0000 7fcef37fe640 1 -- 172.21.15.37:0/316569000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcef4130eb0 msgr2=0x7fcef409ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:19.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.794+0000 7fcef37fe640 1 --2- 172.21.15.37:0/316569000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcef4130eb0 0x7fcef409ef20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fcef406cd00 tx=0x7fceec01c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a49a26e4769e93f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:19.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.794+0000 7fcef27fc640 1 -- 172.21.15.37:0/316569000 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceec0ac070 con 0x7fcef4130eb0 2024-04-26T00:29:19.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.794+0000 7fcef27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:19.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.794+0000 7fcef27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:29:19.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.794+0000 7fcef27fc640 10 monclient: dump: 2024-04-26T00:29:19.793 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:19.793 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:19.793 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.793 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.794 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:19.794 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:19.794 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:19.794 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:19.794 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:19.794 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:19.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.794+0000 7fcef27fc640 10 monclient: _finish_auth 0 2024-04-26T00:29:19.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.794+0000 7fcef27fc640 10 monclient: _check_auth_tickets 2024-04-26T00:29:19.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.794+0000 7fcef27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:29:19.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.794+0000 7fcef27fc640 1 -- 172.21.15.37:0/316569000 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fceec0a6690 con 0x7fcef4130eb0 2024-04-26T00:29:19.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.794+0000 7fcef9d87640 5 monclient: authenticate success, global_id 94380 2024-04-26T00:29:19.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.794+0000 7fcef9d87640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:29:19.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.794+0000 7fcef27fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:29:19.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.794+0000 7fcef9d87640 10 monclient: get_monmap_and_config success 2024-04-26T00:29:19.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.794+0000 7fcef27fc640 1 -- 172.21.15.37:0/316569000 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceec0a6a60 con 0x7fcef4130eb0 2024-04-26T00:29:19.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.794+0000 7fcef27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:19.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.794+0000 7fcef9d87640 4 set_mon_vals no callback set 2024-04-26T00:29:19.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.794+0000 7fcef27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:29:19.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.794+0000 7fcef27fc640 10 monclient: dump: 2024-04-26T00:29:19.795 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:19.795 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:19.795 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.795 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.795 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:19.795 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:19.795 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:19.795 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:19.795 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:19.796 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:19.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef9d87640 10 monclient: shutdown 2024-04-26T00:29:19.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef9d87640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:29:19.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef9d87640 1 -- 172.21.15.37:0/316569000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcef4130eb0 msgr2=0x7fcef409ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:19.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef9d87640 1 --2- 172.21.15.37:0/316569000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcef4130eb0 0x7fcef409ef20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fcef406cd00 tx=0x7fceec01c7f0 comp rx=0 tx=0).stop 2024-04-26T00:29:19.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef9d87640 1 -- 172.21.15.37:0/316569000 shutdown_connections 2024-04-26T00:29:19.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef9d87640 1 --2- 172.21.15.37:0/316569000 >> v2:172.21.15.45:3300/0 conn(0x7fcef4131290 0x7fcef4133b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:19.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef9d87640 1 --2- 172.21.15.37:0/316569000 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcef41340d0 0x7fcef4134580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:19.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef9d87640 1 --2- 172.21.15.37:0/316569000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcef4130eb0 0x7fcef409ef20 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:19.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef9d87640 1 -- 172.21.15.37:0/316569000 >> 172.21.15.37:0/316569000 conn(0x7fcef40a1d50 msgr2=0x7fcef40a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:29:19.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef9d87640 1 -- 172.21.15.37:0/316569000 shutdown_connections 2024-04-26T00:29:19.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef9d87640 1 -- 172.21.15.37:0/316569000 wait complete. 2024-04-26T00:29:19.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef9d87640 10 monclient: build_initial_monmap 2024-04-26T00:29:19.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef9d87640 10 monclient: monmap: 2024-04-26T00:29:19.796 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:29:19.797 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:29:19.797 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:29:19.797 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:29:19.797 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:29:19.797 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:29:19.797 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:29:19.797 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:29:19.797 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:29:19.797 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:19.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef9d87640 1 Processor -- start 2024-04-26T00:29:19.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef9d87640 1 -- start start 2024-04-26T00:29:19.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef9d87640 10 monclient: init 2024-04-26T00:29:19.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef9d87640 10 monclient: _reopen_session rank -1 2024-04-26T00:29:19.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef9d87640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:29:19.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef9d87640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcef4130eb0 0x7fcef41cef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:19.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef9d87640 10 monclient(hunting): picked mon.noname-a con 0x7fcef4130eb0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:29:19.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef9d87640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcef4131290 0x7fcef412dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:19.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef9d87640 10 monclient(hunting): picked mon.noname-b con 0x7fcef4131290 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:29:19.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef9d87640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcef41340d0 0x7fcef4139090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:19.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef9d87640 10 monclient(hunting): picked mon.noname-c con 0x7fcef41340d0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:29:19.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef9d87640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:19.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef9d87640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fcef40a05e0 con 0x7fcef4130eb0 2024-04-26T00:29:19.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef9d87640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:19.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef9d87640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcef40a0460 con 0x7fcef4131290 2024-04-26T00:29:19.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef9d87640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:19.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef9d87640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fcef40a0760 con 0x7fcef41340d0 2024-04-26T00:29:19.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef9d87640 10 monclient(hunting): _renew_subs 2024-04-26T00:29:19.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef9d87640 10 monclient(hunting): authenticate will time out at 1541.105818s 2024-04-26T00:29:19.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef37fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcef4130eb0 0x7fcef41cef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:19.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef3fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcef41340d0 0x7fcef4139090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:19.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef37fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcef4130eb0 0x7fcef41cef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:47356/0 (socket says 172.21.15.37:47356) 2024-04-26T00:29:19.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef37fe640 1 -- 172.21.15.37:0/3143771891 learned_addr learned my addr 172.21.15.37:0/3143771891 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:29:19.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef3fff640 10 monclient(hunting): get_auth_request con 0x7fcef41340d0 auth_method 0 2024-04-26T00:29:19.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:19.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:19.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:19.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef37fe640 10 monclient(hunting): get_auth_request con 0x7fcef4130eb0 auth_method 0 2024-04-26T00:29:19.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:19.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:19.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:19.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:19.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:19.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:19.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:19.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:19.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:19.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef2ffd640 1 --2- 172.21.15.37:0/3143771891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcef4131290 0x7fcef412dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:19.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef2ffd640 10 monclient(hunting): get_auth_request con 0x7fcef4131290 auth_method 0 2024-04-26T00:29:19.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:19.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:19.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:19.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef3fff640 10 monclient(hunting): handle_auth_done global_id 84437 payload 824 2024-04-26T00:29:19.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef3fff640 1 -- 172.21.15.37:0/3143771891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcef4131290 msgr2=0x7fcef412dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:19.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef3fff640 1 --2- 172.21.15.37:0/3143771891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcef4131290 0x7fcef412dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:19.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef3fff640 1 -- 172.21.15.37:0/3143771891 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcef4130eb0 msgr2=0x7fcef41cef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:19.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef3fff640 1 --2- 172.21.15.37:0/3143771891 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcef4130eb0 0x7fcef41cef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:19.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef3fff640 10 monclient: _finish_hunting 0 2024-04-26T00:29:19.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef3fff640 1 monclient: found mon.noname-c 2024-04-26T00:29:19.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:29:19.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:29:19.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.798+0000 7fcef3fff640 1 -- 172.21.15.37:0/3143771891 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcef413a7e0 con 0x7fcef41340d0 2024-04-26T00:29:19.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fcef3fff640 1 -- 172.21.15.37:0/3143771891 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcef41340d0 msgr2=0x7fcef4139090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:19.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fcef3fff640 1 --2- 172.21.15.37:0/3143771891 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcef41340d0 0x7fcef4139090 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fcee400e8b0 tx=0x7fcee400ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:19.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fceebfff640 1 -- 172.21.15.37:0/3143771891 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee4017070 con 0x7fcef41340d0 2024-04-26T00:29:19.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fceebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:19.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fceebfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:29:19.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fceebfff640 10 monclient: dump: 2024-04-26T00:29:19.801 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:19.801 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:19.801 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.801 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.801 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:19.802 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:19.802 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:19.802 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:19.802 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:19.802 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:19.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fceebfff640 10 monclient: _finish_auth 0 2024-04-26T00:29:19.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fceebfff640 10 monclient: _check_auth_tickets 2024-04-26T00:29:19.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fceebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:29:19.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fceebfff640 1 -- 172.21.15.37:0/3143771891 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcee400cb50 con 0x7fcef41340d0 2024-04-26T00:29:19.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fcef9d87640 5 monclient: authenticate success, global_id 84437 2024-04-26T00:29:19.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fcef9d87640 10 monclient: _renew_subs 2024-04-26T00:29:19.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fcef9d87640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:29:19.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fcef9d87640 1 -- 172.21.15.37:0/3143771891 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcef413c410 con 0x7fcef41340d0 2024-04-26T00:29:19.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fceebfff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:29:19.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fceebfff640 1 -- 172.21.15.37:0/3143771891 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee4004020 con 0x7fcef41340d0 2024-04-26T00:29:19.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fceebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:19.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fceebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:29:19.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fceebfff640 10 monclient: dump: 2024-04-26T00:29:19.803 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:19.803 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:19.803 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.803 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.803 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:19.803 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:19.803 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:19.803 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:19.803 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:19.804 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:19.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fcef1ffb640 4 set_mon_vals no callback set 2024-04-26T00:29:19.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fcef9d87640 10 monclient: _renew_subs 2024-04-26T00:29:19.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fcef9d87640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:29:19.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fcef9d87640 1 -- 172.21.15.37:0/3143771891 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcef412e000 con 0x7fcef41340d0 2024-04-26T00:29:19.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fcef9d87640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:29:19.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fcef9d87640 1 --2- 172.21.15.37:0/3143771891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcef413b520 0x7fcef402c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:19.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fcef9d87640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:19.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fcef9d87640 1 -- 172.21.15.37:0/3143771891 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcef404f510 con 0x7fcef413b520 2024-04-26T00:29:19.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fceebfff640 1 -- 172.21.15.37:0/3143771891 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fcee40041c0 con 0x7fcef41340d0 2024-04-26T00:29:19.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fcef37fe640 1 --2- 172.21.15.37:0/3143771891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcef413b520 0x7fcef402c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:19.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fcef37fe640 10 monclient: get_auth_request con 0x7fcef413b520 auth_method 0 2024-04-26T00:29:19.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fcef37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:19.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fcef37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:19.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fcef37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:19.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fceebfff640 1 --2- 172.21.15.37:0/3143771891 >> v2:172.21.15.45:6812/2379918623 conn(0x7fced4035f00 0x7fced40383c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:19.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fcef37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:19.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fcef37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:19.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fcef37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:19.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fceebfff640 1 -- 172.21.15.37:0/3143771891 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(371..371 src has 177..371) ==== 3959+0+0 (secure 0 0 0) 0x7fcee4048850 con 0x7fcef41340d0 2024-04-26T00:29:19.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fcef2ffd640 1 --2- 172.21.15.37:0/3143771891 >> v2:172.21.15.45:6812/2379918623 conn(0x7fced4035f00 0x7fced40383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:19.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fcef2ffd640 10 monclient: get_auth_request con 0x7fced4035f00 auth_method 0 2024-04-26T00:29:19.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fcef37fe640 10 monclient(hunting): handle_auth_done global_id 114136 payload 824 2024-04-26T00:29:19.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.802+0000 7fcef37fe640 1 -- 172.21.15.37:0/3143771891 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcef40a0f60 con 0x7fcef413b520 2024-04-26T00:29:19.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.806+0000 7fcef37fe640 1 -- 172.21.15.37:0/3143771891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcef413b520 msgr2=0x7fcef402c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:19.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.806+0000 7fcef37fe640 1 --2- 172.21.15.37:0/3143771891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcef413b520 0x7fcef402c000 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fceec000f90 tx=0x7fceec019590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:19.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.806+0000 7fcef2ffd640 1 --2- 172.21.15.37:0/3143771891 >> v2:172.21.15.45:6812/2379918623 conn(0x7fced4035f00 0x7fced40383c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fcef413b4b0 tx=0x7fcedc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:19.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.806+0000 7fceebfff640 1 -- 172.21.15.37:0/3143771891 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceec0ac040 con 0x7fcef413b520 2024-04-26T00:29:19.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.806+0000 7fceebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:19.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.806+0000 7fceebfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:29:19.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.806+0000 7fceebfff640 10 monclient: dump: 2024-04-26T00:29:19.808 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:19.808 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:19.809 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.809 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.809 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:19.809 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:19.809 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:19.809 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:19.809 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:19.809 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:19.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.806+0000 7fceebfff640 1 -- 172.21.15.37:0/3143771891 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fceec0bc460 con 0x7fcef413b520 2024-04-26T00:29:19.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.806+0000 7fceebfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:29:19.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.806+0000 7fceebfff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:29:19.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.806+0000 7fceebfff640 1 -- 172.21.15.37:0/3143771891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcef413b520 msgr2=0x7fcef402c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:19.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.806+0000 7fceebfff640 1 --2- 172.21.15.37:0/3143771891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcef413b520 0x7fcef402c000 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fceec000f90 tx=0x7fceec019590 comp rx=0 tx=0).stop 2024-04-26T00:29:19.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.806+0000 7fceebfff640 1 -- 172.21.15.37:0/3143771891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcef413b520 msgr2=0x7fcef402c000 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:29:19.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.806+0000 7fceebfff640 1 --2- 172.21.15.37:0/3143771891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcef413b520 0x7fcef402c000 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fceec000f90 tx=0x7fceec019590 comp rx=0 tx=0).stop 2024-04-26T00:29:19.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.810+0000 7fcef9d87640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:29:19.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.810+0000 7fcef9d87640 1 --2- 172.21.15.37:0/3143771891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcef412e430 0x7fcef402c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:19.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.810+0000 7fcef9d87640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:19.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.810+0000 7fcef9d87640 1 -- 172.21.15.37:0/3143771891 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcef406f350 con 0x7fcef412e430 2024-04-26T00:29:19.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.814+0000 7fcef37fe640 1 --2- 172.21.15.37:0/3143771891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcef412e430 0x7fcef402c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:19.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.814+0000 7fcef37fe640 10 monclient: get_auth_request con 0x7fcef412e430 auth_method 0 2024-04-26T00:29:19.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.814+0000 7fcef37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:19.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.814+0000 7fcef37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:19.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.814+0000 7fcef37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:19.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.814+0000 7fcef37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:19.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.814+0000 7fcef37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:19.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.814+0000 7fcef37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:19.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.814+0000 7fcef37fe640 10 monclient(hunting): handle_auth_done global_id 114139 payload 824 2024-04-26T00:29:19.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.814+0000 7fcef37fe640 1 -- 172.21.15.37:0/3143771891 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcef4139d90 con 0x7fcef412e430 2024-04-26T00:29:19.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.814+0000 7fcef37fe640 1 -- 172.21.15.37:0/3143771891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcef412e430 msgr2=0x7fcef402c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:19.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.814+0000 7fcef37fe640 1 --2- 172.21.15.37:0/3143771891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcef412e430 0x7fcef402c540 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcef402d740 tx=0x7fceec0a9000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:19.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.814+0000 7fceebfff640 1 -- 172.21.15.37:0/3143771891 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceec0c9030 con 0x7fcef412e430 2024-04-26T00:29:19.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.814+0000 7fceebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:19.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.814+0000 7fceebfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:29:19.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.814+0000 7fceebfff640 10 monclient: dump: 2024-04-26T00:29:19.811 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:19.811 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:19.811 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.811 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:19.811 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:19.811 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:19.811 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:19.812 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:19.812 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:19.812 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:19.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.814+0000 7fceebfff640 1 -- 172.21.15.37:0/3143771891 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4715 (secure 0 0 0) 0x7fceec0c5770 con 0x7fcef412e430 2024-04-26T00:29:19.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.814+0000 7fceebfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:29:19.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.814+0000 7fceebfff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:29:19.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.814+0000 7fceebfff640 1 -- 172.21.15.37:0/3143771891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcef412e430 msgr2=0x7fcef402c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:19.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.814+0000 7fceebfff640 1 --2- 172.21.15.37:0/3143771891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcef412e430 0x7fcef402c540 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcef402d740 tx=0x7fceec0a9000 comp rx=0 tx=0).stop 2024-04-26T00:29:19.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.814+0000 7fceebfff640 1 -- 172.21.15.37:0/3143771891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcef412e430 msgr2=0x7fcef402c540 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:29:19.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.814+0000 7fceebfff640 1 --2- 172.21.15.37:0/3143771891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcef412e430 0x7fcef402c540 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcef402d740 tx=0x7fceec0a9000 comp rx=0 tx=0).stop 2024-04-26T00:29:19.813 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:29:19.813 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:29:19.813 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:29:19.813 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:29:19.813 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 70, 2024-04-26T00:29:19.813 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 9903, 2024-04-26T00:29:19.813 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:29:19.813 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:29:19.813 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:29:19.813 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:29:19.813 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:19.813 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 5, 2024-04-26T00:29:19.813 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:29:19.813 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:29:19.814 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:29:19.814 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:29:19.814 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:29:19.814 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:29:19.814 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:29:19.814 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:29:19.814 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:29:19.814 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:29:19.814 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:29:19.814 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:29:19.814 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:29:19.814 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:29:19.814 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:19.814 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:29:19.815 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:29:19.815 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:29:19.815 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:29:19.815 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:29:19.815 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:29:19.815 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:29:19.815 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:29:19.815 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:29:19.815 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:29:19.815 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:29:19.815 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:29:19.815 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:29:19.815 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:19.815 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:19.815 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:29:19.816 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:29:19.816 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:29:19.816 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:29:19.816 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:29:19.816 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:29:19.816 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:29:19.816 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:29:19.816 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:29:19.816 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:29:19.816 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:29:19.816 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:29:19.816 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:29:19.816 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:29:19.816 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:29:19.817 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:29:19.817 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:29:19.817 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:29:19.817 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:29:19.817 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:29:19.817 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:29:19.817 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:29:19.817 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:29:19.817 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:29:19.817 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:29:19.817 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:29:19.817 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:29:19.817 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:29:19.817 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:19.818 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:29:19.818 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:19.818 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:29:19.818 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:19.818 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:29:19.818 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:29:19.818 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:29:19.818 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:29:19.818 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:19.818 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:29:19.818 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:29:19.818 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:19.818 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:19.818 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:19.819 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:29:19.819 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:29:19.819 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:19.819 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:19.819 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:19.819 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:19.819 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:29:19.819 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:29:19.819 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:29:19.819 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:29:19.819 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:29:19.819 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:19.819 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:19.819 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:29:19.820 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:29:19.820 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:29:19.820 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:29:19.820 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:19.820 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:29:19.820 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:29:19.820 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:19.820 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:19.820 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:19.820 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:29:19.820 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:29:19.820 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:19.820 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:19.820 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:19.821 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:19.821 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:29:19.821 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:29:19.821 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:29:19.821 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:29:19.821 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:29:19.821 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:19.821 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:19.821 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:29:19.821 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:29:19.821 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:29:19.821 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:29:19.821 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:19.821 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:29:19.822 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:29:19.822 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:19.822 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:19.822 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:19.822 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:29:19.822 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:29:19.822 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:19.822 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:19.822 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:19.822 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:19.822 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:29:19.822 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:29:19.822 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:29:19.822 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:29:19.822 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:29:19.823 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:19.823 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:19.823 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:19.823 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:29:19.823 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:29:19.823 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:19.823 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:29:19.823 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:29:19.823 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:29:19.823 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:19.823 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:19.823 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:29:19.823 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:19.823 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:29:19.824 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:29:19.824 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:29:19.824 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:19.824 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:19.824 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:29:19.824 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:19.824 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:29:19.824 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:29:19.824 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:29:19.824 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:19.824 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:19.824 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:19.824 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:29:19.824 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:29:19.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.818+0000 7fcef9d87640 1 -- 172.21.15.37:0/3143771891 >> v2:172.21.15.45:6812/2379918623 conn(0x7fced4035f00 msgr2=0x7fced40383c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:19.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.818+0000 7fcef9d87640 1 --2- 172.21.15.37:0/3143771891 >> v2:172.21.15.45:6812/2379918623 conn(0x7fced4035f00 0x7fced40383c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fcef413b4b0 tx=0x7fcedc008040 comp rx=0 tx=0).stop 2024-04-26T00:29:19.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.818+0000 7fcef9d87640 10 monclient: shutdown 2024-04-26T00:29:19.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.818+0000 7fcef9d87640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:29:19.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.818+0000 7fcef9d87640 1 -- 172.21.15.37:0/3143771891 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcef41340d0 msgr2=0x7fcef4139090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:19.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.818+0000 7fcef3fff640 1 -- 172.21.15.37:0/3143771891 reap_dead start 2024-04-26T00:29:19.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.818+0000 7fcef9d87640 1 --2- 172.21.15.37:0/3143771891 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcef41340d0 0x7fcef4139090 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fcee400e8b0 tx=0x7fcee400ed70 comp rx=0 tx=0).stop 2024-04-26T00:29:19.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.818+0000 7fcef9d87640 1 -- 172.21.15.37:0/3143771891 shutdown_connections 2024-04-26T00:29:19.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.818+0000 7fcef9d87640 1 --2- 172.21.15.37:0/3143771891 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcef41340d0 0x7fcef4139090 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:19.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.818+0000 7fcef9d87640 1 -- 172.21.15.37:0/3143771891 >> 172.21.15.37:0/3143771891 conn(0x7fcef40a1d50 msgr2=0x7fcef40a3050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:29:19.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.818+0000 7fcef9d87640 1 -- 172.21.15.37:0/3143771891 shutdown_connections 2024-04-26T00:29:19.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:19.818+0000 7fcef9d87640 1 -- 172.21.15.37:0/3143771891 wait complete. 2024-04-26T00:29:19.827 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:29:19.827 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-04-26T00:29:19.827 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2024-04-26T00:29:19.827 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-04-26T00:29:19.827 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-04-26T00:29:19.828 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-26T00:29:19.828 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T00:29:19.840 INFO:tasks.ceph.mon.a:Stopped 2024-04-26T00:29:19.840 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-04-26T00:29:19.841 INFO:tasks.ceph.mon.b:Sent signal 19 2024-04-26T00:29:19.841 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2024-04-26T00:29:19.884 INFO:tasks.rados.rados.0.smithi045.stdout:1626: finishing write tid 1 to smithi04526579-25 2024-04-26T00:29:19.884 INFO:tasks.rados.rados.0.smithi045.stdout:1626: finishing write tid 2 to smithi04526579-25 2024-04-26T00:29:19.884 INFO:tasks.rados.rados.0.smithi045.stdout:1626: finishing write tid 3 to smithi04526579-25 2024-04-26T00:29:19.884 INFO:tasks.rados.rados.0.smithi045.stdout:1626: finishing write tid 4 to smithi04526579-25 2024-04-26T00:29:19.884 INFO:tasks.rados.rados.0.smithi045.stdout:1626: oid 25 updating version 0 to 1302 2024-04-26T00:29:19.884 INFO:tasks.rados.rados.0.smithi045.stdout:1626: oid 25 version 1302 is already newer than 1301 2024-04-26T00:29:19.884 INFO:tasks.rados.rados.0.smithi045.stdout:1626: oid 25 version 1302 is already newer than 1300 2024-04-26T00:29:19.885 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 1302 (ObjNum 604 snap 186 seq_num 604) dirty exists 2024-04-26T00:29:19.885 INFO:tasks.rados.rados.0.smithi045.stdout:1626: left oid 25 (ObjNum 604 snap 186 seq_num 604) 2024-04-26T00:29:19.885 INFO:tasks.rados.rados.0.smithi045.stdout:1627: seq_num 605 ranges {568235=717532,1753253=414982} 2024-04-26T00:29:19.903 INFO:tasks.rados.rados.0.smithi045.stdout:1627: writing smithi04526579-26 from 568235 to 1285767 tid 1 2024-04-26T00:29:19.913 INFO:tasks.rados.rados.0.smithi045.stdout:1627: writing smithi04526579-26 from 1753253 to 2168235 tid 2 2024-04-26T00:29:19.913 INFO:tasks.rados.rados.0.smithi045.stdout:1624: done (3 left) 2024-04-26T00:29:19.913 INFO:tasks.rados.rados.0.smithi045.stdout:1625: done (2 left) 2024-04-26T00:29:19.913 INFO:tasks.rados.rados.0.smithi045.stdout:1626: done (1 left) 2024-04-26T00:29:19.913 INFO:tasks.rados.rados.0.smithi045.stdout:1628: read oid 37 snap -1 2024-04-26T00:29:19.913 INFO:tasks.rados.rados.0.smithi045.stdout:1628: expect (ObjNum 601 snap 186 seq_num 601) 2024-04-26T00:29:19.914 INFO:tasks.rados.rados.0.smithi045.stdout:1629: write oid 39 current snap is 186 2024-04-26T00:29:19.914 INFO:tasks.rados.rados.0.smithi045.stdout:1627: finishing write tid 1 to smithi04526579-26 2024-04-26T00:29:19.914 INFO:tasks.rados.rados.0.smithi045.stdout:1629: seq_num 606 ranges {585042=792781,2095382=89660} 2024-04-26T00:29:19.930 INFO:tasks.rados.rados.0.smithi045.stdout:1629: writing smithi04526579-39 from 585042 to 1377823 tid 1 2024-04-26T00:29:19.942 INFO:tasks.rados.rados.0.smithi045.stdout:1629: writing smithi04526579-39 from 2095382 to 2185042 tid 2 2024-04-26T00:29:19.942 INFO:tasks.rados.rados.0.smithi045.stdout:1630: delete oid 16 current snap is 186 2024-04-26T00:29:19.942 INFO:tasks.rados.rados.0.smithi045.stdout:1627: finishing write tid 2 to smithi04526579-26 2024-04-26T00:29:19.942 INFO:tasks.rados.rados.0.smithi045.stdout:1627: finishing write tid 3 to smithi04526579-26 2024-04-26T00:29:19.943 INFO:tasks.rados.rados.0.smithi045.stdout:1627: finishing write tid 4 to smithi04526579-26 2024-04-26T00:29:19.943 INFO:tasks.rados.rados.0.smithi045.stdout:1627: oid 26 updating version 0 to 1065 2024-04-26T00:29:19.943 INFO:tasks.rados.rados.0.smithi045.stdout:1627: oid 26 updating version 1065 to 1066 2024-04-26T00:29:19.943 INFO:tasks.rados.rados.0.smithi045.stdout:1627: oid 26 updating version 1066 to 1067 2024-04-26T00:29:19.943 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 1067 (ObjNum 605 snap 186 seq_num 605) dirty exists 2024-04-26T00:29:19.943 INFO:tasks.rados.rados.0.smithi045.stdout:1627: left oid 26 (ObjNum 605 snap 186 seq_num 605) 2024-04-26T00:29:19.943 INFO:tasks.rados.rados.0.smithi045.stdout:1628: expect (ObjNum 601 snap 186 seq_num 601) 2024-04-26T00:29:20.037 INFO:tasks.rados.rados.0.smithi045.stdout:1629: finishing write tid 1 to smithi04526579-39 2024-04-26T00:29:20.037 INFO:tasks.rados.rados.0.smithi045.stdout:1629: finishing write tid 2 to smithi04526579-39 2024-04-26T00:29:20.037 INFO:tasks.rados.rados.0.smithi045.stdout:1629: finishing write tid 3 to smithi04526579-39 2024-04-26T00:29:20.037 INFO:tasks.rados.rados.0.smithi045.stdout:1629: finishing write tid 4 to smithi04526579-39 2024-04-26T00:29:20.037 INFO:tasks.rados.rados.0.smithi045.stdout:1629: oid 39 updating version 0 to 1047 2024-04-26T00:29:20.037 INFO:tasks.rados.rados.0.smithi045.stdout:1629: oid 39 version 1047 is already newer than 1046 2024-04-26T00:29:20.037 INFO:tasks.rados.rados.0.smithi045.stdout:1629: oid 39 updating version 1047 to 1048 2024-04-26T00:29:20.037 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 1048 (ObjNum 606 snap 186 seq_num 606) dirty exists 2024-04-26T00:29:20.037 INFO:tasks.rados.rados.0.smithi045.stdout:1629: left oid 39 (ObjNum 606 snap 186 seq_num 606) 2024-04-26T00:29:20.037 INFO:tasks.rados.rados.0.smithi045.stdout:1627: done (3 left) 2024-04-26T00:29:20.038 INFO:tasks.rados.rados.0.smithi045.stdout:1628: done (2 left) 2024-04-26T00:29:20.038 INFO:tasks.rados.rados.0.smithi045.stdout:1629: done (1 left) 2024-04-26T00:29:20.038 INFO:tasks.rados.rados.0.smithi045.stdout:1630: done (0 left) 2024-04-26T00:29:20.038 INFO:tasks.rados.rados.0.smithi045.stdout:1631: delete oid 1 current snap is 186 2024-04-26T00:29:20.043 INFO:tasks.rados.rados.0.smithi045.stdout:1631: done (0 left) 2024-04-26T00:29:20.043 INFO:tasks.rados.rados.0.smithi045.stdout:1632: copy_from oid 45 from oid 3 current snap is 186 2024-04-26T00:29:20.043 INFO:tasks.rados.rados.0.smithi045.stdout:1633: delete oid 39 current snap is 186 2024-04-26T00:29:20.047 INFO:tasks.rados.rados.0.smithi045.stdout:1633: done (1 left) 2024-04-26T00:29:20.047 INFO:tasks.rados.rados.0.smithi045.stdout:1634: read oid 49 snap -1 2024-04-26T00:29:20.047 INFO:tasks.rados.rados.0.smithi045.stdout:1634: expect (ObjNum 591 snap 183 seq_num 591) 2024-04-26T00:29:20.047 INFO:tasks.rados.rados.0.smithi045.stdout:1635: snap_remove snap 146 2024-04-26T00:29:39.842 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-04-26T00:29:39.843 INFO:tasks.ceph.mon.b:Sent signal 18 2024-04-26T00:29:39.843 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-26T00:29:39.844 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:29:39.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.986+0000 7f3272380640 10 monclient: get_monmap_and_config 2024-04-26T00:29:39.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.986+0000 7f3272380640 10 monclient: build_initial_monmap 2024-04-26T00:29:39.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.986+0000 7f3272380640 10 monclient: monmap: 2024-04-26T00:29:39.977 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:29:39.977 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:39.977 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:29:39.990308+0000 2024-04-26T00:29:39.977 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:29:39.990308+0000 2024-04-26T00:29:39.978 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:29:39.978 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:39.978 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:29:39.978 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:29:39.978 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:29:39.978 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:29:39.978 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:29:39.978 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:39.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.986+0000 7f3272380640 1 Processor -- start 2024-04-26T00:29:39.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.986+0000 7f3272380640 1 -- start start 2024-04-26T00:29:39.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.986+0000 7f3272380640 10 monclient: init 2024-04-26T00:29:39.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.990+0000 7f3272380640 10 monclient: _reopen_session rank -1 2024-04-26T00:29:39.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.990+0000 7f3272380640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-04-26T00:29:39.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.990+0000 7f3272380640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f326c133b20 0x7f326c09e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:39.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.990+0000 7f3272380640 10 monclient(hunting): picked mon.noname-c con 0x7f326c133b20 addr v2:172.21.15.45:3300/0 2024-04-26T00:29:39.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.990+0000 7f3272380640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f326c133f00 addr v1:172.21.15.45:6789/0 2024-04-26T00:29:39.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.990+0000 7f3272380640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 0x7f326c13b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:39.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.990+0000 7f3272380640 10 monclient(hunting): picked mon.noname-b con 0x7f326c139050 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:29:39.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.990+0000 7f3272380640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f326c09fe20 con 0x7f326c133f00 2024-04-26T00:29:39.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.990+0000 7f3272380640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:39.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.990+0000 7f3272380640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f326c072d00 con 0x7f326c133b20 2024-04-26T00:29:39.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.990+0000 7f3272380640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:39.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.990+0000 7f3272380640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f326c09ba60 con 0x7f326c139050 2024-04-26T00:29:39.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.990+0000 7f3272380640 10 monclient(hunting): _renew_subs 2024-04-26T00:29:39.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.990+0000 7f3272380640 10 monclient(hunting): authenticate will time out at 1561.296482s 2024-04-26T00:29:39.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.990+0000 7f327091d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 0x7f326c13b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:39.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.990+0000 7f327091d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 0x7f326c13b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:38726/0 (socket says 172.21.15.37:38726) 2024-04-26T00:29:39.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.990+0000 7f327091d640 1 -- 172.21.15.37:0/2483517554 learned_addr learned my addr 172.21.15.37:0/2483517554 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:29:39.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.990+0000 7f327091d640 10 monclient(hunting): get_auth_request con 0x7f326c139050 auth_method 0 2024-04-26T00:29:39.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.990+0000 7f327091d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:39.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.990+0000 7f327091d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:39.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.990+0000 7f327091d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:39.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.990+0000 7f327091d640 1 -- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 msgr2=0x7f326c13b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:29:39.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.990+0000 7f327091d640 1 -- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 msgr2=0x7f326c13b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:29:39.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.990+0000 7f327091d640 1 --2- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 0x7f326c13b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:29:39.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.990+0000 7f326bfff640 1 --2- 172.21.15.37:0/2483517554 >> v2:172.21.15.45:3300/0 conn(0x7f326c133b20 0x7f326c09e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:39.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.990+0000 7f327091d640 1 --2- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 0x7f326c13b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:29:39.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.990+0000 7f326bfff640 10 monclient(hunting): get_auth_request con 0x7f326c133b20 auth_method 0 2024-04-26T00:29:39.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.990+0000 7f326bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:39.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.990+0000 7f326bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:39.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.990+0000 7f326bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:39.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.990+0000 7f326bfff640 1 -- 172.21.15.37:0/2483517554 >> v2:172.21.15.45:3300/0 conn(0x7f326c133b20 msgr2=0x7f326c09e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:29:39.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.990+0000 7f326bfff640 1 -- 172.21.15.37:0/2483517554 >> v2:172.21.15.45:3300/0 conn(0x7f326c133b20 msgr2=0x7f326c09e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:29:39.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.990+0000 7f326bfff640 1 --2- 172.21.15.37:0/2483517554 >> v2:172.21.15.45:3300/0 conn(0x7f326c133b20 0x7f326c09e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:29:39.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.990+0000 7f326bfff640 1 --2- 172.21.15.37:0/2483517554 >> v2:172.21.15.45:3300/0 conn(0x7f326c133b20 0x7f326c09e9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:29:39.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:39.990+0000 7f326b7fe640 1 -- 172.21.15.37:0/2483517554 >> v1:172.21.15.45:6789/0 conn(0x7f326c133f00 legacy=0x7f326c134350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:40.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:40.194+0000 7f327091d640 1 --2- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 0x7f326c13b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:40.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:40.194+0000 7f327091d640 10 monclient(hunting): get_auth_request con 0x7f326c139050 auth_method 0 2024-04-26T00:29:40.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:40.194+0000 7f327091d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:40.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:40.194+0000 7f327091d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:40.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:40.194+0000 7f327091d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:29:40.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:40.194+0000 7f326bfff640 1 --2- 172.21.15.37:0/2483517554 >> v2:172.21.15.45:3300/0 conn(0x7f326c133b20 0x7f326c09e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:40.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:40.194+0000 7f326bfff640 10 monclient(hunting): get_auth_request con 0x7f326c133b20 auth_method 0 2024-04-26T00:29:40.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:40.194+0000 7f326bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:40.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:40.194+0000 7f326bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:40.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:40.194+0000 7f326bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:29:40.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:40.194+0000 7f327091d640 1 -- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 msgr2=0x7f326c13b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:29:40.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:40.194+0000 7f327091d640 1 -- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 msgr2=0x7f326c13b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:29:40.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:40.194+0000 7f327091d640 1 --2- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 0x7f326c13b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:29:40.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:40.194+0000 7f326bfff640 1 -- 172.21.15.37:0/2483517554 >> v2:172.21.15.45:3300/0 conn(0x7f326c133b20 msgr2=0x7f326c09e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:29:40.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:40.194+0000 7f326bfff640 1 -- 172.21.15.37:0/2483517554 >> v2:172.21.15.45:3300/0 conn(0x7f326c133b20 msgr2=0x7f326c09e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:29:40.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:40.194+0000 7f326bfff640 1 --2- 172.21.15.37:0/2483517554 >> v2:172.21.15.45:3300/0 conn(0x7f326c133b20 0x7f326c09e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:29:40.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:40.194+0000 7f326bfff640 1 --2- 172.21.15.37:0/2483517554 >> v2:172.21.15.45:3300/0 conn(0x7f326c133b20 0x7f326c09e9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:29:40.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:40.194+0000 7f327091d640 1 --2- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 0x7f326c13b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:29:40.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:40.598+0000 7f327091d640 1 --2- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 0x7f326c13b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:40.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:40.598+0000 7f327091d640 10 monclient(hunting): get_auth_request con 0x7f326c139050 auth_method 0 2024-04-26T00:29:40.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:40.598+0000 7f327091d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:40.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:40.598+0000 7f327091d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:40.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:40.598+0000 7f327091d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:29:40.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:40.598+0000 7f326bfff640 1 --2- 172.21.15.37:0/2483517554 >> v2:172.21.15.45:3300/0 conn(0x7f326c133b20 0x7f326c09e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:40.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:40.598+0000 7f326bfff640 10 monclient(hunting): get_auth_request con 0x7f326c133b20 auth_method 0 2024-04-26T00:29:40.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:40.598+0000 7f326bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:40.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:40.598+0000 7f326bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:40.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:40.598+0000 7f326bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:29:40.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:40.598+0000 7f327091d640 1 -- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 msgr2=0x7f326c13b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:29:40.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:40.598+0000 7f327091d640 1 -- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 msgr2=0x7f326c13b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:29:40.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:40.598+0000 7f327091d640 1 --2- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 0x7f326c13b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:29:40.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:40.598+0000 7f326bfff640 1 -- 172.21.15.37:0/2483517554 >> v2:172.21.15.45:3300/0 conn(0x7f326c133b20 msgr2=0x7f326c09e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:29:40.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:40.598+0000 7f326bfff640 1 -- 172.21.15.37:0/2483517554 >> v2:172.21.15.45:3300/0 conn(0x7f326c133b20 msgr2=0x7f326c09e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:29:40.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:40.598+0000 7f326bfff640 1 --2- 172.21.15.37:0/2483517554 >> v2:172.21.15.45:3300/0 conn(0x7f326c133b20 0x7f326c09e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:29:40.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:40.598+0000 7f327091d640 1 --2- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 0x7f326c13b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:29:40.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:40.598+0000 7f326bfff640 1 --2- 172.21.15.37:0/2483517554 >> v2:172.21.15.45:3300/0 conn(0x7f326c133b20 0x7f326c09e9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:29:41.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:41.402+0000 7f327091d640 1 --2- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 0x7f326c13b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:41.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:41.402+0000 7f327091d640 10 monclient(hunting): get_auth_request con 0x7f326c139050 auth_method 0 2024-04-26T00:29:41.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:41.402+0000 7f327091d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:41.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:41.402+0000 7f327091d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:41.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:41.402+0000 7f327091d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:29:41.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:41.402+0000 7f326bfff640 1 --2- 172.21.15.37:0/2483517554 >> v2:172.21.15.45:3300/0 conn(0x7f326c133b20 0x7f326c09e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:41.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:41.402+0000 7f326bfff640 10 monclient(hunting): get_auth_request con 0x7f326c133b20 auth_method 0 2024-04-26T00:29:41.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:41.402+0000 7f326bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:41.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:41.402+0000 7f326bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:41.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:41.402+0000 7f326bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:29:41.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:41.402+0000 7f327091d640 1 -- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 msgr2=0x7f326c13b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:29:41.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:41.402+0000 7f327091d640 1 -- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 msgr2=0x7f326c13b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:29:41.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:41.402+0000 7f327091d640 1 --2- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 0x7f326c13b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:29:41.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:41.402+0000 7f326bfff640 1 -- 172.21.15.37:0/2483517554 >> v2:172.21.15.45:3300/0 conn(0x7f326c133b20 msgr2=0x7f326c09e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:29:41.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:41.402+0000 7f326bfff640 1 -- 172.21.15.37:0/2483517554 >> v2:172.21.15.45:3300/0 conn(0x7f326c133b20 msgr2=0x7f326c09e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:29:41.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:41.402+0000 7f326bfff640 1 --2- 172.21.15.37:0/2483517554 >> v2:172.21.15.45:3300/0 conn(0x7f326c133b20 0x7f326c09e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:29:41.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:41.402+0000 7f327091d640 1 --2- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 0x7f326c13b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:29:41.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:41.402+0000 7f326bfff640 1 --2- 172.21.15.37:0/2483517554 >> v2:172.21.15.45:3300/0 conn(0x7f326c133b20 0x7f326c09e9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:29:42.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:42.990+0000 7f3269ffb640 10 monclient(hunting): tick 2024-04-26T00:29:42.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:42.990+0000 7f3269ffb640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:29:42.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:42.990+0000 7f3269ffb640 1 monclient(hunting): continuing hunt 2024-04-26T00:29:42.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:42.990+0000 7f3269ffb640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:29:42.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:42.990+0000 7f3269ffb640 1 -- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 msgr2=0x7f326c13b440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:29:42.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:42.990+0000 7f3269ffb640 1 --2- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 0x7f326c13b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:42.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:42.990+0000 7f3269ffb640 1 -- 172.21.15.37:0/2483517554 >> v2:172.21.15.45:3300/0 conn(0x7f326c133b20 msgr2=0x7f326c09e9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:29:42.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:42.990+0000 7f3269ffb640 1 --2- 172.21.15.37:0/2483517554 >> v2:172.21.15.45:3300/0 conn(0x7f326c133b20 0x7f326c09e9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:42.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:42.990+0000 7f3269ffb640 1 -- 172.21.15.37:0/2483517554 >> v1:172.21.15.45:6789/0 conn(0x7f326c133f00 legacy=0x7f326c134350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:42.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:42.990+0000 7f3269ffb640 10 monclient: _add_conns ranks=[3,0] 2024-04-26T00:29:42.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:42.990+0000 7f3269ffb640 1 --2- 172.21.15.37:0/2483517554 >> v2:172.21.15.37:3300/0 conn(0x7f3258003770 0x7f3258005bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:42.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:42.990+0000 7f3269ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f3258003770 addr v2:172.21.15.37:3300/0 2024-04-26T00:29:42.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:42.990+0000 7f3269ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3258006200 addr v1:172.21.15.37:6789/0 2024-04-26T00:29:42.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:42.990+0000 7f3269ffb640 1 -- 172.21.15.37:0/2483517554 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f326c09dd10 con 0x7f3258006200 2024-04-26T00:29:42.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:42.990+0000 7f3269ffb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:42.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:42.990+0000 7f3269ffb640 1 -- 172.21.15.37:0/2483517554 --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f326c072d00 con 0x7f3258003770 2024-04-26T00:29:42.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:42.990+0000 7f3269ffb640 10 monclient(hunting): _renew_subs 2024-04-26T00:29:42.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:42.990+0000 7f326bfff640 1 -- 172.21.15.37:0/2483517554 >> v2:172.21.15.37:3300/0 conn(0x7f3258003770 msgr2=0x7f3258005bb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:29:42.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:42.990+0000 7f326bfff640 1 --2- 172.21.15.37:0/2483517554 >> v2:172.21.15.37:3300/0 conn(0x7f3258003770 0x7f3258005bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:29:42.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:42.990+0000 7f326b7fe640 1 -- 172.21.15.37:0/2483517554 >> v1:172.21.15.37:6789/0 conn(0x7f3258006200 legacy=0x7f32580086a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:29:43.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:43.194+0000 7f326bfff640 1 -- 172.21.15.37:0/2483517554 >> v2:172.21.15.37:3300/0 conn(0x7f3258003770 msgr2=0x7f3258005bb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:29:43.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:43.194+0000 7f326b7fe640 1 -- 172.21.15.37:0/2483517554 >> v1:172.21.15.37:6789/0 conn(0x7f3258006200 legacy=0x7f32580086a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:29:43.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:43.194+0000 7f326bfff640 1 --2- 172.21.15.37:0/2483517554 >> v2:172.21.15.37:3300/0 conn(0x7f3258003770 0x7f3258005bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:29:43.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:43.598+0000 7f326bfff640 1 -- 172.21.15.37:0/2483517554 >> v2:172.21.15.37:3300/0 conn(0x7f3258003770 msgr2=0x7f3258005bb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:29:43.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:43.598+0000 7f326b7fe640 1 -- 172.21.15.37:0/2483517554 >> v1:172.21.15.37:6789/0 conn(0x7f3258006200 legacy=0x7f32580086a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:29:43.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:43.598+0000 7f326bfff640 1 --2- 172.21.15.37:0/2483517554 >> v2:172.21.15.37:3300/0 conn(0x7f3258003770 0x7f3258005bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:29:44.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:44.402+0000 7f326bfff640 1 -- 172.21.15.37:0/2483517554 >> v2:172.21.15.37:3300/0 conn(0x7f3258003770 msgr2=0x7f3258005bb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:29:44.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:44.402+0000 7f326b7fe640 1 -- 172.21.15.37:0/2483517554 >> v1:172.21.15.37:6789/0 conn(0x7f3258006200 legacy=0x7f32580086a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:29:44.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:44.402+0000 7f326bfff640 1 --2- 172.21.15.37:0/2483517554 >> v2:172.21.15.37:3300/0 conn(0x7f3258003770 0x7f3258005bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:29:45.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:45.990+0000 7f3269ffb640 10 monclient(hunting): tick 2024-04-26T00:29:45.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:45.990+0000 7f3269ffb640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:29:45.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:45.990+0000 7f3269ffb640 1 monclient(hunting): continuing hunt 2024-04-26T00:29:45.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:45.990+0000 7f3269ffb640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:29:45.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:45.990+0000 7f3269ffb640 1 -- 172.21.15.37:0/2483517554 >> v2:172.21.15.37:3300/0 conn(0x7f3258003770 msgr2=0x7f3258005bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:29:45.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:45.990+0000 7f3269ffb640 1 --2- 172.21.15.37:0/2483517554 >> v2:172.21.15.37:3300/0 conn(0x7f3258003770 0x7f3258005bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:45.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:45.990+0000 7f3269ffb640 1 -- 172.21.15.37:0/2483517554 >> v1:172.21.15.37:6789/0 conn(0x7f3258006200 legacy=0x7f32580086a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:29:45.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:45.990+0000 7f3269ffb640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-04-26T00:29:45.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:45.990+0000 7f327091d640 1 -- 172.21.15.37:0/2483517554 reap_dead start 2024-04-26T00:29:45.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:45.990+0000 7f3269ffb640 1 --2- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 0x7f3258009f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:45.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:45.990+0000 7f3269ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f326c139050 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:29:45.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:45.990+0000 7f3269ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3258006200 addr v1:172.21.15.37:6789/0 2024-04-26T00:29:45.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:45.990+0000 7f3269ffb640 1 --2- 172.21.15.37:0/2483517554 >> v2:172.21.15.37:3300/0 conn(0x7f325800a490 0x7f3258005bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:45.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:45.990+0000 7f3269ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f325800a490 addr v2:172.21.15.37:3300/0 2024-04-26T00:29:45.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:45.990+0000 7f3269ffb640 1 -- 172.21.15.37:0/2483517554 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f325800aff0 con 0x7f3258006200 2024-04-26T00:29:45.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:45.990+0000 7f3269ffb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:45.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:45.990+0000 7f3269ffb640 1 -- 172.21.15.37:0/2483517554 --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f326c072d00 con 0x7f325800a490 2024-04-26T00:29:45.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:45.990+0000 7f3269ffb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:45.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:45.990+0000 7f3269ffb640 1 -- 172.21.15.37:0/2483517554 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f326c09ba60 con 0x7f326c139050 2024-04-26T00:29:45.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:45.990+0000 7f3269ffb640 10 monclient(hunting): _renew_subs 2024-04-26T00:29:45.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:45.990+0000 7f326b7fe640 1 -- 172.21.15.37:0/2483517554 >> v1:172.21.15.37:6789/0 conn(0x7f3258006200 legacy=0x7f32580086a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:29:45.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:45.990+0000 7f327091d640 1 -- 172.21.15.37:0/2483517554 >> v2:172.21.15.37:3300/0 conn(0x7f325800a490 msgr2=0x7f3258005bb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:29:45.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:45.990+0000 7f327091d640 1 --2- 172.21.15.37:0/2483517554 >> v2:172.21.15.37:3300/0 conn(0x7f325800a490 0x7f3258005bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:29:45.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:45.990+0000 7f326bfff640 1 --2- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 0x7f3258009f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:45.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:45.990+0000 7f326bfff640 10 monclient(hunting): get_auth_request con 0x7f326c139050 auth_method 0 2024-04-26T00:29:45.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:45.990+0000 7f326bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:45.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:45.990+0000 7f326bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:45.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:45.990+0000 7f326bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:45.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:45.990+0000 7f326bfff640 1 -- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 msgr2=0x7f3258009f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:29:45.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:45.990+0000 7f326bfff640 1 -- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 msgr2=0x7f3258009f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:29:45.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:45.990+0000 7f326bfff640 1 --2- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 0x7f3258009f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:29:45.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:45.990+0000 7f326bfff640 1 --2- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 0x7f3258009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:29:46.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:46.194+0000 7f327091d640 1 -- 172.21.15.37:0/2483517554 >> v2:172.21.15.37:3300/0 conn(0x7f325800a490 msgr2=0x7f3258005bb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:29:46.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:46.194+0000 7f326b7fe640 1 -- 172.21.15.37:0/2483517554 >> v1:172.21.15.37:6789/0 conn(0x7f3258006200 legacy=0x7f32580086a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:29:46.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:46.194+0000 7f327091d640 1 --2- 172.21.15.37:0/2483517554 >> v2:172.21.15.37:3300/0 conn(0x7f325800a490 0x7f3258005bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:29:46.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:46.194+0000 7f326bfff640 1 --2- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 0x7f3258009f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:46.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:46.194+0000 7f326bfff640 10 monclient(hunting): get_auth_request con 0x7f326c139050 auth_method 0 2024-04-26T00:29:46.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:46.194+0000 7f326bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:46.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:46.194+0000 7f326bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:46.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:46.194+0000 7f326bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:29:46.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:46.194+0000 7f326bfff640 1 -- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 msgr2=0x7f3258009f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:29:46.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:46.194+0000 7f326bfff640 1 -- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 msgr2=0x7f3258009f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:29:46.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:46.194+0000 7f326bfff640 1 --2- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 0x7f3258009f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:29:46.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:46.194+0000 7f326bfff640 1 --2- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 0x7f3258009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:29:46.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:46.598+0000 7f327091d640 1 -- 172.21.15.37:0/2483517554 >> v2:172.21.15.37:3300/0 conn(0x7f325800a490 msgr2=0x7f3258005bb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:29:46.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:46.598+0000 7f326b7fe640 1 -- 172.21.15.37:0/2483517554 >> v1:172.21.15.37:6789/0 conn(0x7f3258006200 legacy=0x7f32580086a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:29:46.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:46.598+0000 7f327091d640 1 --2- 172.21.15.37:0/2483517554 >> v2:172.21.15.37:3300/0 conn(0x7f325800a490 0x7f3258005bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:29:46.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:46.598+0000 7f326bfff640 1 --2- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 0x7f3258009f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:46.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:46.598+0000 7f326bfff640 10 monclient(hunting): get_auth_request con 0x7f326c139050 auth_method 0 2024-04-26T00:29:46.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:46.598+0000 7f326bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:46.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:46.598+0000 7f326bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:46.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:46.598+0000 7f326bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:29:46.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:46.598+0000 7f326bfff640 1 -- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 msgr2=0x7f3258009f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:29:46.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:46.598+0000 7f326bfff640 1 -- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 msgr2=0x7f3258009f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:29:46.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:46.598+0000 7f326bfff640 1 --2- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 0x7f3258009f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:29:46.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:46.598+0000 7f326bfff640 1 --2- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 0x7f3258009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:29:47.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:47.402+0000 7f327091d640 1 -- 172.21.15.37:0/2483517554 >> v2:172.21.15.37:3300/0 conn(0x7f325800a490 msgr2=0x7f3258005bb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:29:47.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:47.402+0000 7f326b7fe640 1 -- 172.21.15.37:0/2483517554 >> v1:172.21.15.37:6789/0 conn(0x7f3258006200 legacy=0x7f32580086a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:29:47.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:47.402+0000 7f327091d640 1 --2- 172.21.15.37:0/2483517554 >> v2:172.21.15.37:3300/0 conn(0x7f325800a490 0x7f3258005bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:29:47.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:47.402+0000 7f326bfff640 1 --2- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 0x7f3258009f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:47.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:47.402+0000 7f326bfff640 10 monclient(hunting): get_auth_request con 0x7f326c139050 auth_method 0 2024-04-26T00:29:47.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:47.402+0000 7f326bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:47.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:47.402+0000 7f326bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:47.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:47.402+0000 7f326bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:29:47.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:47.402+0000 7f326bfff640 1 -- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 msgr2=0x7f3258009f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:29:47.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:47.402+0000 7f326bfff640 1 -- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 msgr2=0x7f3258009f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:29:47.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:47.402+0000 7f326bfff640 1 --2- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 0x7f3258009f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:29:47.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:47.402+0000 7f326bfff640 1 --2- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 0x7f3258009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:29:48.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:48.990+0000 7f3269ffb640 10 monclient(hunting): tick 2024-04-26T00:29:48.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:48.990+0000 7f3269ffb640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:29:48.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:48.990+0000 7f3269ffb640 1 monclient(hunting): continuing hunt 2024-04-26T00:29:48.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:48.990+0000 7f3269ffb640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:29:48.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:48.990+0000 7f3269ffb640 1 -- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 msgr2=0x7f3258009f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:29:48.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:48.990+0000 7f3269ffb640 1 --2- 172.21.15.37:0/2483517554 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c139050 0x7f3258009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:48.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:48.990+0000 7f3269ffb640 1 -- 172.21.15.37:0/2483517554 >> v2:172.21.15.37:3300/0 conn(0x7f325800a490 msgr2=0x7f3258005bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:29:48.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:48.990+0000 7f3269ffb640 1 --2- 172.21.15.37:0/2483517554 >> v2:172.21.15.37:3300/0 conn(0x7f325800a490 0x7f3258005bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:48.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:48.990+0000 7f3269ffb640 1 -- 172.21.15.37:0/2483517554 >> v1:172.21.15.37:6789/0 conn(0x7f3258006200 legacy=0x7f32580086a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:29:48.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:48.990+0000 7f3269ffb640 10 monclient: _add_conns ranks=[1,4] 2024-04-26T00:29:48.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:48.990+0000 7f3269ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f325800d430 addr v1:172.21.15.45:6789/0 2024-04-26T00:29:48.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:48.990+0000 7f3269ffb640 1 --2- 172.21.15.37:0/2483517554 >> v2:172.21.15.45:3300/0 conn(0x7f3258010f40 0x7f3258013330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:48.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:48.990+0000 7f3269ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f3258010f40 addr v2:172.21.15.45:3300/0 2024-04-26T00:29:48.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:48.990+0000 7f3269ffb640 1 -- 172.21.15.37:0/2483517554 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f326c09dd10 con 0x7f325800d430 2024-04-26T00:29:48.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:48.990+0000 7f3269ffb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:48.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:48.990+0000 7f3269ffb640 1 -- 172.21.15.37:0/2483517554 --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f326c072d00 con 0x7f3258010f40 2024-04-26T00:29:48.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:48.990+0000 7f3269ffb640 10 monclient(hunting): _renew_subs 2024-04-26T00:29:48.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:48.990+0000 7f326b7fe640 1 --2- 172.21.15.37:0/2483517554 >> v2:172.21.15.45:3300/0 conn(0x7f3258010f40 0x7f3258013330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:48.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:48.990+0000 7f326b7fe640 10 monclient(hunting): get_auth_request con 0x7f3258010f40 auth_method 0 2024-04-26T00:29:48.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:48.990+0000 7f326b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:48.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:48.990+0000 7f326b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:48.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:48.990+0000 7f326b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:48.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:48.990+0000 7f326b7fe640 1 -- 172.21.15.37:0/2483517554 >> v2:172.21.15.45:3300/0 conn(0x7f3258010f40 msgr2=0x7f3258013330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:29:48.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:48.990+0000 7f326b7fe640 1 -- 172.21.15.37:0/2483517554 >> v2:172.21.15.45:3300/0 conn(0x7f3258010f40 msgr2=0x7f3258013330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:29:48.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:48.990+0000 7f326b7fe640 1 --2- 172.21.15.37:0/2483517554 >> v2:172.21.15.45:3300/0 conn(0x7f3258010f40 0x7f3258013330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:29:48.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:48.990+0000 7f326b7fe640 1 --2- 172.21.15.37:0/2483517554 >> v2:172.21.15.45:3300/0 conn(0x7f3258010f40 0x7f3258013330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:29:48.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:48.994+0000 7f326bfff640 1 -- 172.21.15.37:0/2483517554 >> v1:172.21.15.45:6789/0 conn(0x7f325800d430 legacy=0x7f325800f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:49.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:49.194+0000 7f326b7fe640 1 --2- 172.21.15.37:0/2483517554 >> v2:172.21.15.45:3300/0 conn(0x7f3258010f40 0x7f3258013330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:49.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:49.194+0000 7f326b7fe640 10 monclient(hunting): get_auth_request con 0x7f3258010f40 auth_method 0 2024-04-26T00:29:49.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:49.194+0000 7f326b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:49.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:49.194+0000 7f326b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:49.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:49.194+0000 7f326b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:29:49.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:49.194+0000 7f326b7fe640 1 -- 172.21.15.37:0/2483517554 >> v2:172.21.15.45:3300/0 conn(0x7f3258010f40 msgr2=0x7f3258013330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:29:49.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:49.194+0000 7f326b7fe640 1 -- 172.21.15.37:0/2483517554 >> v2:172.21.15.45:3300/0 conn(0x7f3258010f40 msgr2=0x7f3258013330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:29:49.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:49.194+0000 7f326b7fe640 1 --2- 172.21.15.37:0/2483517554 >> v2:172.21.15.45:3300/0 conn(0x7f3258010f40 0x7f3258013330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:29:49.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:49.194+0000 7f326b7fe640 1 --2- 172.21.15.37:0/2483517554 >> v2:172.21.15.45:3300/0 conn(0x7f3258010f40 0x7f3258013330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:29:49.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:49.598+0000 7f326b7fe640 1 --2- 172.21.15.37:0/2483517554 >> v2:172.21.15.45:3300/0 conn(0x7f3258010f40 0x7f3258013330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:49.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:49.598+0000 7f326b7fe640 10 monclient(hunting): get_auth_request con 0x7f3258010f40 auth_method 0 2024-04-26T00:29:49.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:49.598+0000 7f326b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:49.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:49.598+0000 7f326b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:49.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:49.598+0000 7f326b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:29:49.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:49.598+0000 7f326b7fe640 1 -- 172.21.15.37:0/2483517554 >> v2:172.21.15.45:3300/0 conn(0x7f3258010f40 msgr2=0x7f3258013330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:29:49.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:49.598+0000 7f326b7fe640 1 -- 172.21.15.37:0/2483517554 >> v2:172.21.15.45:3300/0 conn(0x7f3258010f40 msgr2=0x7f3258013330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:29:49.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:49.598+0000 7f326b7fe640 1 --2- 172.21.15.37:0/2483517554 >> v2:172.21.15.45:3300/0 conn(0x7f3258010f40 0x7f3258013330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:29:49.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:49.598+0000 7f326b7fe640 1 --2- 172.21.15.37:0/2483517554 >> v2:172.21.15.45:3300/0 conn(0x7f3258010f40 0x7f3258013330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:29:49.846 INFO:tasks.ceph.mon.b.smithi045.stderr:2024-04-26T00:29:49.857+0000 7f1e443a1640 -1 mon.b@1(electing) e1 get_health_metrics reporting 2 slow ops, oldest is log(1 entries from seq 4 at 2024-04-26T00:29:19.819507+0000) 2024-04-26T00:29:49.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:49.858+0000 7f326bfff640 1 -- 172.21.15.37:0/2483517554 >> v1:172.21.15.45:6789/0 conn(0x7f325800d430 legacy=0x7f325800f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-26T00:29:49.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:49.858+0000 7f326bfff640 1 -- 172.21.15.37:0/2483517554 >> v1:172.21.15.45:6789/0 conn(0x7f325800d430 legacy=0x7f325800f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:29:49.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:49.858+0000 7f326bfff640 1 --1- 172.21.15.37:0/2483517554 >> v1:172.21.15.45:6789/0 conn(0x7f325800d430 0x7f325800f820 :-1 s=OPENED pgs=40 cs=1 l=1).handle_message read tag failed 2024-04-26T00:29:49.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:49.858+0000 7f326bfff640 1 --1- 172.21.15.37:0/2483517554 >> v1:172.21.15.45:6789/0 conn(0x7f325800d430 0x7f325800f820 :-1 s=OPENED pgs=40 cs=1 l=1).fault on lossy channel, failing 2024-04-26T00:29:49.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:49.858+0000 7f326affd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.45:6789/0 2024-04-26T00:29:49.981 INFO:tasks.rados.rados.0.smithi045.stdout:1635: done (2 left) 2024-04-26T00:29:49.982 INFO:tasks.rados.rados.0.smithi045.stdout:1636: delete oid 9 current snap is 186 2024-04-26T00:29:49.982 INFO:tasks.rados.rados.0.smithi045.stdout:1632: finishing copy_from racing read to smithi04526579-45 2024-04-26T00:29:49.982 INFO:tasks.rados.rados.0.smithi045.stdout:1632: finishing copy_from to smithi04526579-45 2024-04-26T00:29:49.982 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 974 (ObjNum 603 snap 186 seq_num 603) dirty exists 2024-04-26T00:29:49.982 INFO:tasks.rados.rados.0.smithi045.stdout:1634: expect (ObjNum 591 snap 183 seq_num 591) 2024-04-26T00:29:50.303 INFO:tasks.rados.rados.0.smithi045.stdout:1632: done (2 left) 2024-04-26T00:29:50.304 INFO:tasks.rados.rados.0.smithi045.stdout:1634: done (1 left) 2024-04-26T00:29:50.304 INFO:tasks.rados.rados.0.smithi045.stdout:1636: done (0 left) 2024-04-26T00:29:50.304 INFO:tasks.rados.rados.0.smithi045.stdout:1637: read oid 44 snap -1 2024-04-26T00:29:50.304 INFO:tasks.rados.rados.0.smithi045.stdout:1637: expect (ObjNum 546 snap 168 seq_num 546) 2024-04-26T00:29:50.304 INFO:tasks.rados.rados.0.smithi045.stdout:1638: write oid 7 current snap is 186 2024-04-26T00:29:50.304 INFO:tasks.rados.rados.0.smithi045.stdout:1638: seq_num 611 ranges {669077=485379,1587792=681285} 2024-04-26T00:29:50.315 INFO:tasks.rados.rados.0.smithi045.stdout:1638: writing smithi04526579-7 from 669077 to 1154456 tid 1 2024-04-26T00:29:50.334 INFO:tasks.rados.rados.0.smithi045.stdout:1638: writing smithi04526579-7 from 1587792 to 2269077 tid 2 2024-04-26T00:29:50.334 INFO:tasks.rados.rados.0.smithi045.stdout:1639: delete oid 8 current snap is 186 2024-04-26T00:29:50.334 INFO:tasks.rados.rados.0.smithi045.stdout:1637: expect (ObjNum 546 snap 168 seq_num 546) 2024-04-26T00:29:50.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.402+0000 7f326b7fe640 1 --2- 172.21.15.37:0/2483517554 >> v2:172.21.15.45:3300/0 conn(0x7f3258010f40 0x7f3258013330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:50.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.402+0000 7f326b7fe640 10 monclient(hunting): get_auth_request con 0x7f3258010f40 auth_method 0 2024-04-26T00:29:50.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.402+0000 7f326b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:50.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.402+0000 7f326b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:50.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.402+0000 7f326b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:29:50.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.402+0000 7f326b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:50.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.402+0000 7f326b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:50.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.402+0000 7f326b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:50.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.402+0000 7f326b7fe640 10 monclient(hunting): handle_auth_done global_id 114142 payload 274 2024-04-26T00:29:50.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.402+0000 7f326b7fe640 1 -- 172.21.15.37:0/2483517554 >> v1:172.21.15.45:6789/0 conn(0x7f325800d430 legacy=0x7f325800f820 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:29:50.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.402+0000 7f326b7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:29:50.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.402+0000 7f326b7fe640 1 monclient: found mon.noname-c 2024-04-26T00:29:50.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.402+0000 7f326b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:29:50.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.402+0000 7f326b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:29:50.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.402+0000 7f326b7fe640 1 -- 172.21.15.37:0/2483517554 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f325800aff0 con 0x7f3258010f40 2024-04-26T00:29:50.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.410+0000 7f326b7fe640 1 -- 172.21.15.37:0/2483517554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3258010f40 msgr2=0x7f3258013330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:50.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.410+0000 7f326b7fe640 1 --2- 172.21.15.37:0/2483517554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3258010f40 0x7f3258013330 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f325800ac90 tx=0x7f326001ed50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f068624fe87f23dc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:50.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.410+0000 7f326affd640 1 -- 172.21.15.37:0/2483517554 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32600a7910 con 0x7f3258010f40 2024-04-26T00:29:50.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.410+0000 7f326affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:50.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.410+0000 7f326affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:29:50.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.410+0000 7f326affd640 10 monclient: dump: 2024-04-26T00:29:50.400 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:50.400 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:50.401 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:50.401 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:50.401 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:50.401 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:50.401 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:50.401 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:50.401 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:50.401 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:50.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.410+0000 7f326affd640 10 monclient: _finish_auth 0 2024-04-26T00:29:50.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.410+0000 7f326affd640 10 monclient: _check_auth_tickets 2024-04-26T00:29:50.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.410+0000 7f326affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:29:50.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.410+0000 7f326affd640 1 -- 172.21.15.37:0/2483517554 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32600a7ab0 con 0x7f3258010f40 2024-04-26T00:29:50.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.410+0000 7f326affd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:29:50.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.410+0000 7f326affd640 1 -- 172.21.15.37:0/2483517554 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32600a7e60 con 0x7f3258010f40 2024-04-26T00:29:50.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.410+0000 7f326affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:50.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.410+0000 7f326affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:29:50.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.410+0000 7f326affd640 10 monclient: dump: 2024-04-26T00:29:50.402 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:50.402 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:50.402 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:50.402 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:50.402 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:50.402 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:50.402 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:50.402 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:50.402 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:50.402 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:50.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.410+0000 7f3272380640 5 monclient: authenticate success, global_id 114142 2024-04-26T00:29:50.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.410+0000 7f3272380640 10 monclient: get_monmap_and_config success 2024-04-26T00:29:50.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.410+0000 7f3272380640 4 set_mon_vals no callback set 2024-04-26T00:29:50.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.410+0000 7f3272380640 10 monclient: shutdown 2024-04-26T00:29:50.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.410+0000 7f3272380640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:29:50.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.410+0000 7f3272380640 1 -- 172.21.15.37:0/2483517554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3258010f40 msgr2=0x7f3258013330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:50.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.410+0000 7f3272380640 1 --2- 172.21.15.37:0/2483517554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3258010f40 0x7f3258013330 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f325800ac90 tx=0x7f326001ed50 comp rx=0 tx=0).stop 2024-04-26T00:29:50.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.410+0000 7f327091d640 1 -- 172.21.15.37:0/2483517554 reap_dead start 2024-04-26T00:29:50.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.410+0000 7f3272380640 1 -- 172.21.15.37:0/2483517554 shutdown_connections 2024-04-26T00:29:50.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.410+0000 7f3272380640 1 --2- 172.21.15.37:0/2483517554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3258010f40 0x7f3258013330 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:50.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.410+0000 7f3272380640 1 -- 172.21.15.37:0/2483517554 >> 172.21.15.37:0/2483517554 conn(0x7f326c0a1ad0 msgr2=0x7f326c0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:29:50.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.410+0000 7f3272380640 1 -- 172.21.15.37:0/2483517554 shutdown_connections 2024-04-26T00:29:50.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.410+0000 7f3272380640 1 -- 172.21.15.37:0/2483517554 wait complete. 2024-04-26T00:29:50.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f3272380640 10 monclient: build_initial_monmap 2024-04-26T00:29:50.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f3272380640 10 monclient: monmap: 2024-04-26T00:29:50.405 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:29:50.405 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:29:50.405 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:29:50.405 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:29:50.405 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:29:50.405 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:29:50.405 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:29:50.405 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:29:50.405 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:29:50.406 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:50.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f3272380640 1 Processor -- start 2024-04-26T00:29:50.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f3272380640 1 -- start start 2024-04-26T00:29:50.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f3272380640 10 monclient: init 2024-04-26T00:29:50.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f3272380640 10 monclient: _reopen_session rank -1 2024-04-26T00:29:50.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f3272380640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:29:50.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f3272380640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3258010f40 0x7f326c1cb470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:50.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f3272380640 10 monclient(hunting): picked mon.noname-b con 0x7f3258010f40 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:29:50.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f3272380640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f326c1cb9b0 0x7f326c0702d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:50.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f3272380640 10 monclient(hunting): picked mon.noname-a con 0x7f326c1cb9b0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:29:50.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f3272380640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c1cbe10 0x7f326c070810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:50.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f3272380640 10 monclient(hunting): picked mon.noname-c con 0x7f326c1cbe10 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:29:50.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f3272380640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:50.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f3272380640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f326c1308f0 con 0x7f326c1cb9b0 2024-04-26T00:29:50.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f3272380640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:50.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f3272380640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f326c130770 con 0x7f3258010f40 2024-04-26T00:29:50.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f3272380640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:50.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f3272380640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f326c130a70 con 0x7f326c1cbe10 2024-04-26T00:29:50.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f3272380640 10 monclient(hunting): _renew_subs 2024-04-26T00:29:50.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f3272380640 10 monclient(hunting): authenticate will time out at 1571.723230s 2024-04-26T00:29:50.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f326b7fe640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f326c1cb9b0 msgr2=0x7f326c0702d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:29:50.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f326b7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f326c1cb9b0 0x7f326c0702d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:29:50.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f327091d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c1cbe10 0x7f326c070810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:50.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f327091d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c1cbe10 0x7f326c070810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:59694/0 (socket says 172.21.15.37:59694) 2024-04-26T00:29:50.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f327091d640 1 -- 172.21.15.37:0/4270735792 learned_addr learned my addr 172.21.15.37:0/4270735792 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:29:50.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f327091d640 10 monclient(hunting): get_auth_request con 0x7f326c1cbe10 auth_method 0 2024-04-26T00:29:50.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f327091d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:50.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f327091d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:50.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f327091d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:50.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f326bfff640 1 --2- 172.21.15.37:0/4270735792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3258010f40 0x7f326c1cb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:50.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f326bfff640 10 monclient(hunting): get_auth_request con 0x7f3258010f40 auth_method 0 2024-04-26T00:29:50.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f326bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:50.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f326bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:50.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f326bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:50.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f327091d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:50.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f327091d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:50.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f327091d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:50.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f326bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:50.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f326bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:50.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.414+0000 7f326bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:50.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.418+0000 7f327091d640 10 monclient(hunting): handle_auth_done global_id 84440 payload 824 2024-04-26T00:29:50.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.418+0000 7f327091d640 1 -- 172.21.15.37:0/4270735792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3258010f40 msgr2=0x7f326c1cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:50.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.418+0000 7f327091d640 1 --2- 172.21.15.37:0/4270735792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3258010f40 0x7f326c1cb470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:50.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.418+0000 7f327091d640 1 -- 172.21.15.37:0/4270735792 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f326c1cb9b0 msgr2=0x7f326c0702d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:29:50.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.418+0000 7f327091d640 1 --2- 172.21.15.37:0/4270735792 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f326c1cb9b0 0x7f326c0702d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:50.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.418+0000 7f327091d640 10 monclient: _finish_hunting 0 2024-04-26T00:29:50.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.418+0000 7f327091d640 1 monclient: found mon.noname-c 2024-04-26T00:29:50.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.418+0000 7f327091d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:29:50.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.418+0000 7f327091d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:29:50.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.418+0000 7f327091d640 1 -- 172.21.15.37:0/4270735792 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f326c11ca50 con 0x7f326c1cbe10 2024-04-26T00:29:50.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.418+0000 7f327091d640 1 -- 172.21.15.37:0/4270735792 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c1cbe10 msgr2=0x7f326c070810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:50.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.418+0000 7f327091d640 1 --2- 172.21.15.37:0/4270735792 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c1cbe10 0x7f326c070810 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3258006170 tx=0x7f3264011a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:50.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.418+0000 7f3268ff9640 1 -- 172.21.15.37:0/4270735792 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326401a030 con 0x7f326c1cbe10 2024-04-26T00:29:50.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.418+0000 7f3268ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:50.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.418+0000 7f3268ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:29:50.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.418+0000 7f3268ff9640 10 monclient: dump: 2024-04-26T00:29:50.413 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:50.413 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:50.413 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:50.413 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:50.413 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:50.413 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:50.414 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:50.414 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:50.414 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:50.414 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:50.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.418+0000 7f3268ff9640 10 monclient: _finish_auth 0 2024-04-26T00:29:50.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.418+0000 7f3268ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:29:50.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.418+0000 7f3268ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:29:50.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.418+0000 7f3272380640 5 monclient: authenticate success, global_id 84440 2024-04-26T00:29:50.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.418+0000 7f3268ff9640 1 -- 172.21.15.37:0/4270735792 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3264015960 con 0x7f326c1cbe10 2024-04-26T00:29:50.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.418+0000 7f3268ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:29:50.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.418+0000 7f3268ff9640 1 -- 172.21.15.37:0/4270735792 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326400f040 con 0x7f326c1cbe10 2024-04-26T00:29:50.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.418+0000 7f3272380640 10 monclient: _renew_subs 2024-04-26T00:29:50.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.418+0000 7f3272380640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:29:50.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.418+0000 7f3272380640 1 -- 172.21.15.37:0/4270735792 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f326c0a0000 con 0x7f326c1cbe10 2024-04-26T00:29:50.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.418+0000 7f3268ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:50.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.418+0000 7f3268ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:29:50.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.418+0000 7f3268ff9640 10 monclient: dump: 2024-04-26T00:29:50.415 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:50.415 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:50.415 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:50.415 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:50.415 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:50.415 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:50.415 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:50.415 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:50.415 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:50.415 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:50.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.418+0000 7f326a7fc640 4 set_mon_vals no callback set 2024-04-26T00:29:50.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.418+0000 7f3272380640 10 monclient: _renew_subs 2024-04-26T00:29:50.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.418+0000 7f3272380640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:29:50.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.418+0000 7f3272380640 1 -- 172.21.15.37:0/4270735792 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f326c11c770 con 0x7f326c1cbe10 2024-04-26T00:29:50.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.422+0000 7f3272380640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:29:50.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.422+0000 7f3272380640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:29:50.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.422+0000 7f3272380640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:29:50.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.422+0000 7f3268ff9640 1 -- 172.21.15.37:0/4270735792 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f326401d030 con 0x7f326c1cbe10 2024-04-26T00:29:50.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.422+0000 7f3272380640 1 -- 172.21.15.37:0/4270735792 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f326c11c1b0 con 0x7f326c1cbe10 2024-04-26T00:29:50.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.422+0000 7f3268ff9640 1 --2- 172.21.15.37:0/4270735792 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3258037a00 0x7f3258039ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:50.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.422+0000 7f3268ff9640 1 -- 172.21.15.37:0/4270735792 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(373..373 src has 177..373) ==== 3931+0+0 (secure 0 0 0) 0x7f3264048b80 con 0x7f326c1cbe10 2024-04-26T00:29:50.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.422+0000 7f326bfff640 1 --2- 172.21.15.37:0/4270735792 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3258037a00 0x7f3258039ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:50.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.422+0000 7f326bfff640 10 monclient: get_auth_request con 0x7f3258037a00 auth_method 0 2024-04-26T00:29:50.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.426+0000 7f326bfff640 1 --2- 172.21.15.37:0/4270735792 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3258037a00 0x7f3258039ec0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f325c003fd0 tx=0x7f325c01b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:50.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.426+0000 7f3268ff9640 1 -- 172.21.15.37:0/4270735792 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f326c11c1b0 con 0x7f326c1cbe10 2024-04-26T00:29:50.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.426+0000 7f3268ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:29:50.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.426+0000 7f3268ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:29:50.473 INFO:tasks.rados.rados.0.smithi045.stdout:1638: finishing write tid 1 to smithi04526579-7 2024-04-26T00:29:50.473 INFO:tasks.rados.rados.0.smithi045.stdout:1637: done (2 left) 2024-04-26T00:29:50.474 INFO:tasks.rados.rados.0.smithi045.stdout:1639: done (1 left) 2024-04-26T00:29:50.474 INFO:tasks.rados.rados.0.smithi045.stdout:1640: delete oid 3 current snap is 186 2024-04-26T00:29:50.474 INFO:tasks.rados.rados.0.smithi045.stdout:1638: finishing write tid 2 to smithi04526579-7 2024-04-26T00:29:50.474 INFO:tasks.rados.rados.0.smithi045.stdout:1638: finishing write tid 3 to smithi04526579-7 2024-04-26T00:29:50.474 INFO:tasks.rados.rados.0.smithi045.stdout:1638: finishing write tid 4 to smithi04526579-7 2024-04-26T00:29:50.474 INFO:tasks.rados.rados.0.smithi045.stdout:1638: oid 7 updating version 0 to 1310 2024-04-26T00:29:50.474 INFO:tasks.rados.rados.0.smithi045.stdout:1638: oid 7 updating version 1310 to 1311 2024-04-26T00:29:50.474 INFO:tasks.rados.rados.0.smithi045.stdout:1638: oid 7 updating version 1311 to 1312 2024-04-26T00:29:50.475 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 1312 (ObjNum 611 snap 186 seq_num 611) dirty exists 2024-04-26T00:29:50.475 INFO:tasks.rados.rados.0.smithi045.stdout:1638: left oid 7 (ObjNum 611 snap 186 seq_num 611) 2024-04-26T00:29:50.481 INFO:tasks.rados.rados.0.smithi045.stdout:1638: done (1 left) 2024-04-26T00:29:50.481 INFO:tasks.rados.rados.0.smithi045.stdout:1640: done (0 left) 2024-04-26T00:29:50.481 INFO:tasks.rados.rados.0.smithi045.stdout:1641: write oid 15 current snap is 186 2024-04-26T00:29:50.481 INFO:tasks.rados.rados.0.smithi045.stdout:1641: seq_num 614 ranges {719498=711126,2046885=272613} 2024-04-26T00:29:50.499 INFO:tasks.rados.rados.0.smithi045.stdout:1641: writing smithi04526579-15 from 719498 to 1430624 tid 1 2024-04-26T00:29:50.512 INFO:tasks.rados.rados.0.smithi045.stdout:1641: writing smithi04526579-15 from 2046885 to 2319498 tid 2 2024-04-26T00:29:50.513 INFO:tasks.rados.rados.0.smithi045.stdout:1642: snap_remove snap 172 2024-04-26T00:29:50.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.650+0000 7f3272380640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:29:50.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.650+0000 7f3272380640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:29:50.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.650+0000 7f3272380640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:29:50.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.650+0000 7f3272380640 1 -- 172.21.15.37:0/4270735792 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f326c066f40 con 0x7f326c1cbe10 2024-04-26T00:29:50.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.650+0000 7f3268ff9640 1 -- 172.21.15.37:0/4270735792 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7f32640478d0 con 0x7f326c1cbe10 2024-04-26T00:29:50.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.650+0000 7f3268ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:29:50.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.650+0000 7f3268ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:29:50.642 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":76,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:29:50.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.654+0000 7f3272380640 1 -- 172.21.15.37:0/4270735792 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3258037a00 msgr2=0x7f3258039ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:50.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.654+0000 7f3272380640 1 --2- 172.21.15.37:0/4270735792 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3258037a00 0x7f3258039ec0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f325c003fd0 tx=0x7f325c01b040 comp rx=0 tx=0).stop 2024-04-26T00:29:50.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.654+0000 7f3272380640 10 monclient: shutdown 2024-04-26T00:29:50.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.654+0000 7f3272380640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:29:50.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.654+0000 7f3272380640 1 -- 172.21.15.37:0/4270735792 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c1cbe10 msgr2=0x7f326c070810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:50.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.654+0000 7f3272380640 1 --2- 172.21.15.37:0/4270735792 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c1cbe10 0x7f326c070810 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3258006170 tx=0x7f3264011a20 comp rx=0 tx=0).stop 2024-04-26T00:29:50.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.654+0000 7f3272380640 1 -- 172.21.15.37:0/4270735792 shutdown_connections 2024-04-26T00:29:50.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.654+0000 7f3272380640 1 --2- 172.21.15.37:0/4270735792 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3258037a00 0x7f3258039ec0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:50.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.654+0000 7f3272380640 1 --2- 172.21.15.37:0/4270735792 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f326c1cbe10 0x7f326c070810 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:50.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.654+0000 7f3272380640 1 --2- 172.21.15.37:0/4270735792 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f326c1cb9b0 0x7f326c0702d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:50.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.654+0000 7f3272380640 1 --2- 172.21.15.37:0/4270735792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3258010f40 0x7f326c1cb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:50.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.654+0000 7f3272380640 1 -- 172.21.15.37:0/4270735792 >> 172.21.15.37:0/4270735792 conn(0x7f326c0a1ad0 msgr2=0x7f326c12fc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:29:50.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.654+0000 7f3272380640 1 -- 172.21.15.37:0/4270735792 shutdown_connections 2024-04-26T00:29:50.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.654+0000 7f3272380640 1 -- 172.21.15.37:0/4270735792 wait complete. 2024-04-26T00:29:50.661 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-26T00:29:50.661 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-04-26T00:29:50.661 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:29:50.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.798+0000 7f421a749640 10 monclient: get_monmap_and_config 2024-04-26T00:29:50.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.798+0000 7f421a749640 10 monclient: build_initial_monmap 2024-04-26T00:29:50.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.798+0000 7f421a749640 10 monclient: monmap: 2024-04-26T00:29:50.788 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:29:50.788 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:50.788 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:29:50.800934+0000 2024-04-26T00:29:50.788 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:29:50.800934+0000 2024-04-26T00:29:50.788 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:29:50.788 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:50.788 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:29:50.788 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:29:50.788 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:29:50.788 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:29:50.789 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:29:50.789 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:50.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.798+0000 7f421a749640 1 Processor -- start 2024-04-26T00:29:50.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.798+0000 7f421a749640 1 -- start start 2024-04-26T00:29:50.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.798+0000 7f421a749640 10 monclient: init 2024-04-26T00:29:50.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.798+0000 7f421a749640 10 monclient: _reopen_session rank -1 2024-04-26T00:29:50.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.798+0000 7f421a749640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-04-26T00:29:50.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.798+0000 7f421a749640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4214130eb0 0x7f421409ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:50.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.798+0000 7f421a749640 10 monclient(hunting): picked mon.noname-c con 0x7f4214130eb0 addr v2:172.21.15.45:3300/0 2024-04-26T00:29:50.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.798+0000 7f421a749640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4214131290 addr v1:172.21.15.45:6789/0 2024-04-26T00:29:50.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.798+0000 7f421a749640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f42141342a0 addr v1:172.21.15.37:6789/0 2024-04-26T00:29:50.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.798+0000 7f421a749640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f421409c190 con 0x7f42141342a0 2024-04-26T00:29:50.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.798+0000 7f421a749640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42140a11c0 con 0x7f4214131290 2024-04-26T00:29:50.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.798+0000 7f421a749640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:50.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.798+0000 7f421a749640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f421409df60 con 0x7f4214130eb0 2024-04-26T00:29:50.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.798+0000 7f421a749640 10 monclient(hunting): _renew_subs 2024-04-26T00:29:50.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.798+0000 7f421a749640 10 monclient(hunting): authenticate will time out at 1572.106728s 2024-04-26T00:29:50.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.798+0000 7f4218ce6640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f42141342a0 legacy=0x7f421413bed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:29:50.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.798+0000 7f4213fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4214130eb0 0x7f421409ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:50.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.798+0000 7f4213fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4214130eb0 0x7f421409ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.37:57822/0 (socket says 172.21.15.37:57822) 2024-04-26T00:29:50.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.798+0000 7f4213fff640 1 -- 172.21.15.37:0/2192025404 learned_addr learned my addr 172.21.15.37:0/2192025404 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:29:50.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.798+0000 7f4213fff640 10 monclient(hunting): get_auth_request con 0x7f4214130eb0 auth_method 0 2024-04-26T00:29:50.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.798+0000 7f4213fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:50.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.798+0000 7f4213fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:50.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.798+0000 7f4213fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:50.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.798+0000 7f4213fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:50.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.798+0000 7f4213fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:50.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.798+0000 7f4213fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:50.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.798+0000 7f4213fff640 10 monclient(hunting): handle_auth_done global_id 114148 payload 274 2024-04-26T00:29:50.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.798+0000 7f4213fff640 1 -- 172.21.15.37:0/2192025404 >> v1:172.21.15.45:6789/0 conn(0x7f4214131290 legacy=0x7f4214133b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:29:50.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.798+0000 7f4213fff640 1 -- 172.21.15.37:0/2192025404 >> v1:172.21.15.37:6789/0 conn(0x7f42141342a0 legacy=0x7f421413bed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:29:50.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.798+0000 7f4213fff640 10 monclient: _finish_hunting 0 2024-04-26T00:29:50.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.798+0000 7f4213fff640 1 monclient: found mon.noname-c 2024-04-26T00:29:50.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.798+0000 7f4213fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:29:50.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.798+0000 7f4213fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:29:50.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.798+0000 7f4213fff640 1 -- 172.21.15.37:0/2192025404 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f421409dc80 con 0x7f4214130eb0 2024-04-26T00:29:50.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f4213fff640 1 -- 172.21.15.37:0/2192025404 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4214130eb0 msgr2=0x7f421409ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:50.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f4213fff640 1 --2- 172.21.15.37:0/2192025404 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4214130eb0 0x7f421409ef20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4204009870 tx=0x7f420401c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d3755eac2f883bd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:50.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f4212ffd640 1 -- 172.21.15.37:0/2192025404 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42040b0070 con 0x7f4214130eb0 2024-04-26T00:29:50.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f4212ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:50.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f4212ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:29:50.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f4212ffd640 10 monclient: dump: 2024-04-26T00:29:50.800 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:50.800 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:50.800 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:50.800 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:50.800 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:50.800 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:50.800 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:50.800 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:50.800 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:50.800 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:50.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f4212ffd640 10 monclient: _finish_auth 0 2024-04-26T00:29:50.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f4212ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:29:50.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f4212ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:29:50.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f4212ffd640 1 -- 172.21.15.37:0/2192025404 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42040ab600 con 0x7f4214130eb0 2024-04-26T00:29:50.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f421a749640 5 monclient: authenticate success, global_id 114148 2024-04-26T00:29:50.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f421a749640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:29:50.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f4212ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:29:50.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f421a749640 10 monclient: get_monmap_and_config success 2024-04-26T00:29:50.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f421a749640 4 set_mon_vals no callback set 2024-04-26T00:29:50.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f4212ffd640 1 -- 172.21.15.37:0/2192025404 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42040ab9d0 con 0x7f4214130eb0 2024-04-26T00:29:50.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f4212ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:50.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f4212ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:29:50.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f4212ffd640 10 monclient: dump: 2024-04-26T00:29:50.802 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:50.802 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:50.802 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:50.802 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:50.802 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:50.802 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:50.802 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:50.802 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:50.802 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:50.802 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:50.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f421a749640 10 monclient: shutdown 2024-04-26T00:29:50.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f421a749640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:29:50.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f421a749640 1 -- 172.21.15.37:0/2192025404 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4214130eb0 msgr2=0x7f421409ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:50.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f421a749640 1 --2- 172.21.15.37:0/2192025404 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4214130eb0 0x7f421409ef20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4204009870 tx=0x7f420401c3a0 comp rx=0 tx=0).stop 2024-04-26T00:29:50.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f421a749640 1 -- 172.21.15.37:0/2192025404 shutdown_connections 2024-04-26T00:29:50.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f421a749640 1 --2- 172.21.15.37:0/2192025404 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4214130eb0 0x7f421409ef20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:50.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f421a749640 1 -- 172.21.15.37:0/2192025404 >> 172.21.15.37:0/2192025404 conn(0x7f42140a1d50 msgr2=0x7f42140a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:29:50.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f421a749640 1 -- 172.21.15.37:0/2192025404 shutdown_connections 2024-04-26T00:29:50.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f421a749640 1 -- 172.21.15.37:0/2192025404 wait complete. 2024-04-26T00:29:50.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f421a749640 10 monclient: build_initial_monmap 2024-04-26T00:29:50.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f421a749640 10 monclient: monmap: 2024-04-26T00:29:50.803 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:29:50.803 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:29:50.803 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:29:50.803 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:29:50.803 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:29:50.803 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:29:50.803 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:29:50.804 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:29:50.804 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:29:50.804 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:50.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f421a749640 1 Processor -- start 2024-04-26T00:29:50.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f421a749640 1 -- start start 2024-04-26T00:29:50.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f421a749640 10 monclient: init 2024-04-26T00:29:50.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f421a749640 10 monclient: _reopen_session rank -1 2024-04-26T00:29:50.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f421a749640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:29:50.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f421a749640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4214130eb0 0x7f42141d3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:50.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f421a749640 10 monclient(hunting): picked mon.noname-a con 0x7f4214130eb0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:29:50.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f421a749640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4214131290 0x7f42141d43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:50.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f421a749640 10 monclient(hunting): picked mon.noname-b con 0x7f4214131290 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:29:50.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f421a749640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f42141342a0 0x7f421406dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:50.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f421a749640 10 monclient(hunting): picked mon.noname-c con 0x7f42141342a0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:29:50.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f421a749640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:50.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f421a749640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f421409fdd0 con 0x7f4214130eb0 2024-04-26T00:29:50.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f421a749640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:50.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f421a749640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f421409fc50 con 0x7f4214131290 2024-04-26T00:29:50.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f421a749640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:50.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f421a749640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f421409ff50 con 0x7f42141342a0 2024-04-26T00:29:50.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f421a749640 10 monclient(hunting): _renew_subs 2024-04-26T00:29:50.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f421a749640 10 monclient(hunting): authenticate will time out at 1572.114952s 2024-04-26T00:29:50.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f4213fff640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4214130eb0 msgr2=0x7f42141d3e70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:29:50.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f4213fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4214130eb0 0x7f42141d3e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:29:50.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f4218ce6640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f42141342a0 0x7f421406dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:50.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f4218ce6640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f42141342a0 0x7f421406dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:59706/0 (socket says 172.21.15.37:59706) 2024-04-26T00:29:50.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f4218ce6640 1 -- 172.21.15.37:0/151705487 learned_addr learned my addr 172.21.15.37:0/151705487 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:29:50.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f4218ce6640 10 monclient(hunting): get_auth_request con 0x7f42141342a0 auth_method 0 2024-04-26T00:29:50.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f4218ce6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:50.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f4218ce6640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:50.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f4218ce6640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:50.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f4218ce6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:50.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f4218ce6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:50.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f4218ce6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:50.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f42137fe640 1 --2- 172.21.15.37:0/151705487 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4214131290 0x7f42141d43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:50.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f42137fe640 10 monclient(hunting): get_auth_request con 0x7f4214131290 auth_method 0 2024-04-26T00:29:50.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f42137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:50.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f42137fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:50.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f42137fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:50.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f4218ce6640 10 monclient(hunting): handle_auth_done global_id 84443 payload 824 2024-04-26T00:29:50.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f4218ce6640 1 -- 172.21.15.37:0/151705487 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4214131290 msgr2=0x7f42141d43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:50.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f4218ce6640 1 --2- 172.21.15.37:0/151705487 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4214131290 0x7f42141d43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:50.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f4218ce6640 1 -- 172.21.15.37:0/151705487 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4214130eb0 msgr2=0x7f42141d3e70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:29:50.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f4218ce6640 1 --2- 172.21.15.37:0/151705487 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4214130eb0 0x7f42141d3e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:50.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f4218ce6640 10 monclient: _finish_hunting 0 2024-04-26T00:29:50.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f4218ce6640 1 monclient: found mon.noname-c 2024-04-26T00:29:50.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f4218ce6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:29:50.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f4218ce6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:29:50.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.806+0000 7f4218ce6640 1 -- 172.21.15.37:0/151705487 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42140a1470 con 0x7f42141342a0 2024-04-26T00:29:50.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f4218ce6640 1 -- 172.21.15.37:0/151705487 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f42141342a0 msgr2=0x7f421406dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:50.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f4218ce6640 1 --2- 172.21.15.37:0/151705487 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f42141342a0 0x7f421406dc70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f420c00be90 tx=0x7f420c00bf80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:50.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f4210ff9640 1 -- 172.21.15.37:0/151705487 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f420c016630 con 0x7f42141342a0 2024-04-26T00:29:50.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f4210ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:50.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f4210ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:29:50.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f4210ff9640 10 monclient: dump: 2024-04-26T00:29:50.811 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:50.812 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:50.812 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:50.812 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:50.812 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:50.812 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:50.812 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:50.812 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:50.812 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:50.812 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:50.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f4210ff9640 10 monclient: _finish_auth 0 2024-04-26T00:29:50.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f4210ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:29:50.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f4210ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:29:50.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f421a749640 5 monclient: authenticate success, global_id 84443 2024-04-26T00:29:50.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f421a749640 10 monclient: _renew_subs 2024-04-26T00:29:50.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f4210ff9640 1 -- 172.21.15.37:0/151705487 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f420c010030 con 0x7f42141342a0 2024-04-26T00:29:50.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f421a749640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:29:50.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f421a749640 1 -- 172.21.15.37:0/151705487 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f421406fd30 con 0x7f42141342a0 2024-04-26T00:29:50.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f4210ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:29:50.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f4210ff9640 1 -- 172.21.15.37:0/151705487 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f420c016a10 con 0x7f42141342a0 2024-04-26T00:29:50.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f4210ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:50.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f4210ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:29:50.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f4210ff9640 10 monclient: dump: 2024-04-26T00:29:50.813 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:50.813 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:50.813 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:50.813 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:50.813 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:50.814 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:50.814 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:50.814 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:50.814 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:50.814 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:50.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f421a749640 10 monclient: _renew_subs 2024-04-26T00:29:50.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f42127fc640 4 set_mon_vals no callback set 2024-04-26T00:29:50.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f421a749640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:29:50.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f421a749640 1 -- 172.21.15.37:0/151705487 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f42141339e0 con 0x7f42141342a0 2024-04-26T00:29:50.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f421a749640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:29:50.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f421a749640 1 --2- 172.21.15.37:0/151705487 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4214071520 0x7f421406e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:50.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f421a749640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:50.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f421a749640 1 -- 172.21.15.37:0/151705487 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f421404f510 con 0x7f4214071520 2024-04-26T00:29:50.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f4213fff640 1 --2- 172.21.15.37:0/151705487 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4214071520 0x7f421406e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:50.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f4213fff640 10 monclient: get_auth_request con 0x7f4214071520 auth_method 0 2024-04-26T00:29:50.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f4213fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:50.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f4213fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:50.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f4213fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:50.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f4210ff9640 1 -- 172.21.15.37:0/151705487 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f420c0153e0 con 0x7f42141342a0 2024-04-26T00:29:50.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f4213fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:50.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f4213fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:50.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f4213fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:50.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f4213fff640 10 monclient(hunting): handle_auth_done global_id 84446 payload 824 2024-04-26T00:29:50.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f4213fff640 1 -- 172.21.15.37:0/151705487 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f42140a0f60 con 0x7f4214071520 2024-04-26T00:29:50.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f4210ff9640 1 --2- 172.21.15.37:0/151705487 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4200035e30 0x7f42000382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:50.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f4210ff9640 1 -- 172.21.15.37:0/151705487 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(373..373 src has 177..373) ==== 3931+0+0 (secure 0 0 0) 0x7f420c049410 con 0x7f42141342a0 2024-04-26T00:29:50.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f42137fe640 1 --2- 172.21.15.37:0/151705487 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4200035e30 0x7f42000382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:50.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.810+0000 7f42137fe640 10 monclient: get_auth_request con 0x7f4200035e30 auth_method 0 2024-04-26T00:29:50.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.814+0000 7f4213fff640 1 -- 172.21.15.37:0/151705487 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4214071520 msgr2=0x7f421406e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:50.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.814+0000 7f4213fff640 1 --2- 172.21.15.37:0/151705487 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4214071520 0x7f421406e1b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4214073220 tx=0x7f420401abe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:50.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.814+0000 7f4210ff9640 1 -- 172.21.15.37:0/151705487 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42040bb020 con 0x7f4214071520 2024-04-26T00:29:50.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.814+0000 7f4210ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:50.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.814+0000 7f4210ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:29:50.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.814+0000 7f4210ff9640 10 monclient: dump: 2024-04-26T00:29:50.817 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:50.817 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:50.817 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:50.817 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:50.817 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:50.817 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:50.817 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:50.817 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:50.817 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:50.817 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:50.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.814+0000 7f4210ff9640 1 -- 172.21.15.37:0/151705487 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f42040ab9b0 con 0x7f4214071520 2024-04-26T00:29:50.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.814+0000 7f4210ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:29:50.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.814+0000 7f4210ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:29:50.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.814+0000 7f4210ff9640 1 -- 172.21.15.37:0/151705487 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4214071520 msgr2=0x7f421406e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:50.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.814+0000 7f4210ff9640 1 --2- 172.21.15.37:0/151705487 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4214071520 0x7f421406e1b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4214073220 tx=0x7f420401abe0 comp rx=0 tx=0).stop 2024-04-26T00:29:50.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.814+0000 7f4210ff9640 1 -- 172.21.15.37:0/151705487 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4214071520 msgr2=0x7f421406e1b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:29:50.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.814+0000 7f4210ff9640 1 --2- 172.21.15.37:0/151705487 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4214071520 0x7f421406e1b0 secure :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4214073220 tx=0x7f420401abe0 comp rx=0 tx=0).stop 2024-04-26T00:29:50.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.814+0000 7f42137fe640 1 --2- 172.21.15.37:0/151705487 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4200035e30 0x7f42000382f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f421406f490 tx=0x7f4208009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:50.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.822+0000 7f421a749640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:29:50.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.822+0000 7f421a749640 1 --2- 172.21.15.37:0/151705487 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f421406e6f0 0x7f421406eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:50.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.822+0000 7f421a749640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:50.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.822+0000 7f421a749640 1 -- 172.21.15.37:0/151705487 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f421406f040 con 0x7f421406e6f0 2024-04-26T00:29:50.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.822+0000 7f4213fff640 1 --2- 172.21.15.37:0/151705487 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f421406e6f0 0x7f421406eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:50.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.822+0000 7f4213fff640 10 monclient: get_auth_request con 0x7f421406e6f0 auth_method 0 2024-04-26T00:29:50.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.822+0000 7f4213fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:50.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.822+0000 7f4213fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:50.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.822+0000 7f4213fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:50.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.822+0000 7f4213fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:50.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.822+0000 7f4213fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:50.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.822+0000 7f4213fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:50.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.822+0000 7f4213fff640 10 monclient(hunting): handle_auth_done global_id 84449 payload 824 2024-04-26T00:29:50.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.822+0000 7f4213fff640 1 -- 172.21.15.37:0/151705487 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4214071900 con 0x7f421406e6f0 2024-04-26T00:29:50.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.822+0000 7f4213fff640 1 -- 172.21.15.37:0/151705487 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f421406e6f0 msgr2=0x7f421406eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:50.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.822+0000 7f4213fff640 1 --2- 172.21.15.37:0/151705487 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f421406e6f0 0x7f421406eb00 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4204019fd0 tx=0x7f4204009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:50.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.822+0000 7f4210ff9640 1 -- 172.21.15.37:0/151705487 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42040bb030 con 0x7f421406e6f0 2024-04-26T00:29:50.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.822+0000 7f4210ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:50.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.822+0000 7f4210ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:29:50.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.822+0000 7f4210ff9640 10 monclient: dump: 2024-04-26T00:29:50.821 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:50.821 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:50.821 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:50.822 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:50.822 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:50.822 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:50.822 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:50.822 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:50.822 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:50.822 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:50.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.822+0000 7f4210ff9640 1 -- 172.21.15.37:0/151705487 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4708 (secure 0 0 0) 0x7f42040b68f0 con 0x7f421406e6f0 2024-04-26T00:29:50.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.822+0000 7f4210ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:29:50.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.822+0000 7f4210ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:29:50.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.822+0000 7f4210ff9640 1 -- 172.21.15.37:0/151705487 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f421406e6f0 msgr2=0x7f421406eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:50.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.822+0000 7f4210ff9640 1 --2- 172.21.15.37:0/151705487 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f421406e6f0 0x7f421406eb00 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4204019fd0 tx=0x7f4204009240 comp rx=0 tx=0).stop 2024-04-26T00:29:50.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.822+0000 7f4210ff9640 1 -- 172.21.15.37:0/151705487 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f421406e6f0 msgr2=0x7f421406eb00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:29:50.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.822+0000 7f4210ff9640 1 --2- 172.21.15.37:0/151705487 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f421406e6f0 0x7f421406eb00 secure :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4204019fd0 tx=0x7f4204009240 comp rx=0 tx=0).stop 2024-04-26T00:29:50.823 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:29:50.823 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:29:50.823 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:29:50.824 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:29:50.824 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 76, 2024-04-26T00:29:50.824 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 195757, 2024-04-26T00:29:50.824 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:29:50.824 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:29:50.824 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:29:50.825 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:50.825 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:29:50.825 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:29:50.825 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:29:50.825 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:29:50.825 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:29:50.826 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:29:50.826 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:29:50.826 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:29:50.826 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:29:50.826 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:29:50.826 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:29:50.827 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:29:50.827 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:29:50.827 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:29:50.827 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:29:50.827 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:50.827 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:29:50.827 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:29:50.827 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:29:50.827 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:29:50.828 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:29:50.828 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:29:50.828 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:29:50.828 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:29:50.828 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:29:50.828 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:29:50.828 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:29:50.828 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:29:50.829 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:29:50.829 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:50.829 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:50.829 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:29:50.829 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:29:50.829 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:29:50.829 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:29:50.829 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:29:50.829 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:29:50.830 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:29:50.830 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:29:50.830 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:29:50.830 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:29:50.830 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:29:50.830 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:29:50.830 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:29:50.831 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:29:50.831 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:29:50.831 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:29:50.831 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:29:50.831 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:29:50.831 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:29:50.831 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:29:50.832 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:29:50.832 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:29:50.832 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:29:50.832 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:29:50.832 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:29:50.832 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:29:50.832 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:29:50.832 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:29:50.832 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:50.832 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:29:50.832 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:50.832 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:29:50.832 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:50.833 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:29:50.833 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:29:50.833 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:29:50.833 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:29:50.833 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:50.833 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:29:50.833 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:29:50.833 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:50.833 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:50.833 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:50.833 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:29:50.833 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:29:50.833 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:50.833 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:50.834 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:50.834 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:50.834 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:29:50.834 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:29:50.834 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:29:50.834 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:29:50.834 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:29:50.835 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:50.835 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:50.835 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:29:50.835 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:29:50.835 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:29:50.835 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:29:50.835 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:50.835 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:29:50.836 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:29:50.836 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:50.836 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:50.836 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:50.836 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:29:50.836 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:29:50.836 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:50.836 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:50.837 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:50.837 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:50.837 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:29:50.837 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:29:50.837 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:29:50.837 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:29:50.837 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:29:50.837 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:50.838 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:50.838 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:29:50.838 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:29:50.838 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:29:50.838 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:29:50.838 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:50.838 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:29:50.838 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:29:50.839 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:50.839 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:50.839 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:50.839 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:29:50.839 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:29:50.839 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:50.839 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:50.840 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:50.840 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:50.840 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:29:50.840 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:29:50.840 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:29:50.840 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:29:50.840 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:29:50.840 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:50.840 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:50.841 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:50.841 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:29:50.841 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:29:50.841 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:50.841 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:29:50.841 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:29:50.841 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:29:50.841 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:50.841 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:50.842 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:29:50.842 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:50.842 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:29:50.842 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:29:50.842 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:29:50.842 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:50.842 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:50.843 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:29:50.843 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:50.843 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:29:50.843 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:29:50.843 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:29:50.843 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:50.843 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:50.843 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:50.844 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:29:50.844 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:29:50.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.822+0000 7f421a749640 1 -- 172.21.15.37:0/151705487 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4200035e30 msgr2=0x7f42000382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:50.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.822+0000 7f421a749640 1 --2- 172.21.15.37:0/151705487 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4200035e30 0x7f42000382f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f421406f490 tx=0x7f4208009040 comp rx=0 tx=0).stop 2024-04-26T00:29:50.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.822+0000 7f421a749640 10 monclient: shutdown 2024-04-26T00:29:50.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.822+0000 7f421a749640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:29:50.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.822+0000 7f421a749640 1 -- 172.21.15.37:0/151705487 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f42141342a0 msgr2=0x7f421406dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:50.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.822+0000 7f421a749640 1 --2- 172.21.15.37:0/151705487 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f42141342a0 0x7f421406dc70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f420c00be90 tx=0x7f420c00bf80 comp rx=0 tx=0).stop 2024-04-26T00:29:50.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.822+0000 7f4218ce6640 1 -- 172.21.15.37:0/151705487 reap_dead start 2024-04-26T00:29:50.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.822+0000 7f4218ce6640 1 -- 172.21.15.37:0/151705487 reap_dead start 2024-04-26T00:29:50.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.826+0000 7f421a749640 1 -- 172.21.15.37:0/151705487 shutdown_connections 2024-04-26T00:29:50.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.826+0000 7f421a749640 1 -- 172.21.15.37:0/151705487 >> 172.21.15.37:0/151705487 conn(0x7f42140a1d50 msgr2=0x7f4214133010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:29:50.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.826+0000 7f421a749640 1 -- 172.21.15.37:0/151705487 shutdown_connections 2024-04-26T00:29:50.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.826+0000 7f421a749640 1 -- 172.21.15.37:0/151705487 wait complete. 2024-04-26T00:29:50.848 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:29:50.848 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-04-26T00:29:50.848 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:29:50.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bee72640 10 monclient: get_monmap_and_config 2024-04-26T00:29:50.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bee72640 10 monclient: build_initial_monmap 2024-04-26T00:29:50.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bee72640 10 monclient: monmap: 2024-04-26T00:29:50.972 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:29:50.972 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:50.972 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:29:50.985114+0000 2024-04-26T00:29:50.973 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:29:50.985114+0000 2024-04-26T00:29:50.973 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:29:50.973 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:50.973 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:29:50.973 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:29:50.973 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:29:50.973 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:29:50.973 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:29:50.973 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:50.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bee72640 1 Processor -- start 2024-04-26T00:29:50.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bee72640 1 -- start start 2024-04-26T00:29:50.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bee72640 10 monclient: init 2024-04-26T00:29:50.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bee72640 10 monclient: _reopen_session rank -1 2024-04-26T00:29:50.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bee72640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-04-26T00:29:50.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bee72640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb6b80726b0 0x7fb6b8072a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:50.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bee72640 10 monclient(hunting): picked mon.noname-a con 0x7fb6b80726b0 addr v2:172.21.15.37:3300/0 2024-04-26T00:29:50.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bee72640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb6b80a0110 addr v1:172.21.15.37:6789/0 2024-04-26T00:29:50.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bee72640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb6b81374e0 0x7fb6b813f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:50.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bee72640 10 monclient(hunting): picked mon.noname-c con 0x7fb6b81374e0 addr v2:172.21.15.45:3300/0 2024-04-26T00:29:50.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bee72640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6b80a1470 con 0x7fb6b80a0110 2024-04-26T00:29:50.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bee72640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:50.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bee72640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fb6b80a1bd0 con 0x7fb6b80726b0 2024-04-26T00:29:50.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bee72640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:50.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bee72640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fb6b80a0eb0 con 0x7fb6b81374e0 2024-04-26T00:29:50.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bee72640 10 monclient(hunting): _renew_subs 2024-04-26T00:29:50.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bee72640 10 monclient(hunting): authenticate will time out at 1572.290667s 2024-04-26T00:29:50.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6b7fff640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7fb6b80a0110 legacy=0x7fb6b8136dd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:29:50.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bcc0e640 1 -- >> v2:172.21.15.37:3300/0 conn(0x7fb6b80726b0 msgr2=0x7fb6b8072a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:29:50.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bcc0e640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb6b80726b0 0x7fb6b8072a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:29:50.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bd40f640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb6b81374e0 0x7fb6b813f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:50.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bd40f640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb6b81374e0 0x7fb6b813f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.37:57852/0 (socket says 172.21.15.37:57852) 2024-04-26T00:29:50.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bd40f640 1 -- 172.21.15.37:0/1691289337 learned_addr learned my addr 172.21.15.37:0/1691289337 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:29:50.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bd40f640 10 monclient(hunting): get_auth_request con 0x7fb6b81374e0 auth_method 0 2024-04-26T00:29:50.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bd40f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:50.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bd40f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:50.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bd40f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:50.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bd40f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:50.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bd40f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:50.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bd40f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:50.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bd40f640 10 monclient(hunting): handle_auth_done global_id 114154 payload 274 2024-04-26T00:29:50.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bd40f640 1 -- 172.21.15.37:0/1691289337 >> v2:172.21.15.37:3300/0 conn(0x7fb6b80726b0 msgr2=0x7fb6b8072a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:29:50.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bd40f640 1 --2- 172.21.15.37:0/1691289337 >> v2:172.21.15.37:3300/0 conn(0x7fb6b80726b0 0x7fb6b8072a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:50.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bd40f640 1 -- 172.21.15.37:0/1691289337 >> v1:172.21.15.37:6789/0 conn(0x7fb6b80a0110 legacy=0x7fb6b8136dd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:29:50.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bd40f640 10 monclient: _finish_hunting 0 2024-04-26T00:29:50.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bd40f640 1 monclient: found mon.noname-c 2024-04-26T00:29:50.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bd40f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:29:50.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bd40f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:29:50.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.982+0000 7fb6bd40f640 1 -- 172.21.15.37:0/1691289337 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6b80a0bd0 con 0x7fb6b81374e0 2024-04-26T00:29:50.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.990+0000 7fb6bd40f640 1 -- 172.21.15.37:0/1691289337 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6b81374e0 msgr2=0x7fb6b813f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:50.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.990+0000 7fb6bd40f640 1 --2- 172.21.15.37:0/1691289337 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6b81374e0 0x7fb6b813f110 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb6b80a2da0 tx=0x7fb6a801c1b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b15a72c48d030ed server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:50.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.990+0000 7fb6b77fe640 1 -- 172.21.15.37:0/1691289337 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a80b0070 con 0x7fb6b81374e0 2024-04-26T00:29:50.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.990+0000 7fb6b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:50.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.990+0000 7fb6b77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:29:50.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.990+0000 7fb6b77fe640 10 monclient: dump: 2024-04-26T00:29:50.980 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:50.980 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:50.980 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:50.980 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:50.980 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:50.980 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:50.980 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:50.980 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:50.981 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:50.981 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:50.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.990+0000 7fb6b77fe640 10 monclient: _finish_auth 0 2024-04-26T00:29:50.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.990+0000 7fb6b77fe640 10 monclient: _check_auth_tickets 2024-04-26T00:29:50.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.990+0000 7fb6b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:29:50.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.990+0000 7fb6bee72640 5 monclient: authenticate success, global_id 114154 2024-04-26T00:29:50.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.990+0000 7fb6bee72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:29:50.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.994+0000 7fb6b77fe640 1 -- 172.21.15.37:0/1691289337 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6a80ab600 con 0x7fb6b81374e0 2024-04-26T00:29:50.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.994+0000 7fb6b77fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:29:50.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.994+0000 7fb6b77fe640 1 -- 172.21.15.37:0/1691289337 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a80a4e60 con 0x7fb6b81374e0 2024-04-26T00:29:50.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.994+0000 7fb6b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:50.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.994+0000 7fb6b77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:29:50.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.994+0000 7fb6b77fe640 10 monclient: dump: 2024-04-26T00:29:50.986 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:50.986 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:50.986 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:50.986 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:50.986 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:50.986 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:50.986 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:50.986 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:50.986 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:50.986 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:50.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.994+0000 7fb6bee72640 10 monclient: get_monmap_and_config success 2024-04-26T00:29:50.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.994+0000 7fb6bee72640 4 set_mon_vals no callback set 2024-04-26T00:29:50.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.994+0000 7fb6bee72640 10 monclient: shutdown 2024-04-26T00:29:50.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.994+0000 7fb6bee72640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:29:50.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.994+0000 7fb6bee72640 1 -- 172.21.15.37:0/1691289337 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6b81374e0 msgr2=0x7fb6b813f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:50.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.994+0000 7fb6bee72640 1 --2- 172.21.15.37:0/1691289337 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6b81374e0 0x7fb6b813f110 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb6b80a2da0 tx=0x7fb6a801c1b0 comp rx=0 tx=0).stop 2024-04-26T00:29:50.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.994+0000 7fb6bee72640 1 -- 172.21.15.37:0/1691289337 shutdown_connections 2024-04-26T00:29:50.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.994+0000 7fb6bee72640 1 --2- 172.21.15.37:0/1691289337 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6b81374e0 0x7fb6b813f110 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:50.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.994+0000 7fb6bee72640 1 --2- 172.21.15.37:0/1691289337 >> v2:172.21.15.37:3300/0 conn(0x7fb6b80726b0 0x7fb6b8072a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:50.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.994+0000 7fb6bee72640 1 -- 172.21.15.37:0/1691289337 >> 172.21.15.37:0/1691289337 conn(0x7fb6b80a1d50 msgr2=0x7fb6b80a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:29:50.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.994+0000 7fb6bee72640 1 -- 172.21.15.37:0/1691289337 shutdown_connections 2024-04-26T00:29:50.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.994+0000 7fb6bee72640 1 -- 172.21.15.37:0/1691289337 wait complete. 2024-04-26T00:29:50.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6bee72640 10 monclient: build_initial_monmap 2024-04-26T00:29:50.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6bee72640 10 monclient: monmap: 2024-04-26T00:29:50.993 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:29:50.993 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:29:50.993 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:29:50.993 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:29:50.993 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:29:50.994 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:29:50.994 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:29:50.994 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:29:50.994 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:29:50.994 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:50.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6bee72640 1 Processor -- start 2024-04-26T00:29:50.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6bee72640 1 -- start start 2024-04-26T00:29:50.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6bee72640 10 monclient: init 2024-04-26T00:29:50.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6bee72640 10 monclient: _reopen_session rank -1 2024-04-26T00:29:50.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6bee72640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:29:50.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6bee72640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb6b80726b0 0x7fb6b8071980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:50.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6bee72640 10 monclient(hunting): picked mon.noname-a con 0x7fb6b80726b0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:29:50.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6bee72640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb6b80a0110 0x7fb6b813bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:50.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6bee72640 10 monclient(hunting): picked mon.noname-c con 0x7fb6b80a0110 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:29:50.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6bee72640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6b81374e0 0x7fb6b813c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:50.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6bee72640 10 monclient(hunting): picked mon.noname-b con 0x7fb6b81374e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:29:50.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6bee72640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:50.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6bee72640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb6b809bdd0 con 0x7fb6b80726b0 2024-04-26T00:29:50.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6bee72640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:50.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6bee72640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb6b809bc50 con 0x7fb6b81374e0 2024-04-26T00:29:50.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6bee72640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:50.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6bee72640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb6b8141e70 con 0x7fb6b80a0110 2024-04-26T00:29:50.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6bee72640 10 monclient(hunting): _renew_subs 2024-04-26T00:29:50.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6bcc0e640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb6b80726b0 msgr2=0x7fb6b8071980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:29:50.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6bee72640 10 monclient(hunting): authenticate will time out at 1572.305420s 2024-04-26T00:29:50.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6bcc0e640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb6b80726b0 0x7fb6b8071980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:29:50.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6b7fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb6b80a0110 0x7fb6b813bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:50.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6b7fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb6b80a0110 0x7fb6b813bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:59742/0 (socket says 172.21.15.37:59742) 2024-04-26T00:29:50.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6b7fff640 1 -- 172.21.15.37:0/2452626397 learned_addr learned my addr 172.21.15.37:0/2452626397 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:29:50.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6b7fff640 10 monclient(hunting): get_auth_request con 0x7fb6b80a0110 auth_method 0 2024-04-26T00:29:50.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:50.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6b7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:50.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:50.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:50.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:50.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:50.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6bd40f640 1 --2- 172.21.15.37:0/2452626397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6b81374e0 0x7fb6b813c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:50.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6bd40f640 10 monclient(hunting): get_auth_request con 0x7fb6b81374e0 auth_method 0 2024-04-26T00:29:50.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6bd40f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:50.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6bd40f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:50.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6bd40f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:50.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6b7fff640 10 monclient(hunting): handle_auth_done global_id 84452 payload 824 2024-04-26T00:29:50.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6b7fff640 1 -- 172.21.15.37:0/2452626397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6b81374e0 msgr2=0x7fb6b813c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:50.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6b7fff640 1 --2- 172.21.15.37:0/2452626397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6b81374e0 0x7fb6b813c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:50.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6b7fff640 1 -- 172.21.15.37:0/2452626397 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb6b80726b0 msgr2=0x7fb6b8071980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:29:50.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6b7fff640 1 --2- 172.21.15.37:0/2452626397 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb6b80726b0 0x7fb6b8071980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:29:50.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6b7fff640 10 monclient: _finish_hunting 0 2024-04-26T00:29:50.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6b7fff640 1 monclient: found mon.noname-c 2024-04-26T00:29:50.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:29:50.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6b7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:29:50.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:50.998+0000 7fb6b7fff640 1 -- 172.21.15.37:0/2452626397 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6b813e590 con 0x7fb6b80a0110 2024-04-26T00:29:50.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6b7fff640 1 -- 172.21.15.37:0/2452626397 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb6b80a0110 msgr2=0x7fb6b813bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:50.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6b7fff640 1 --2- 172.21.15.37:0/2452626397 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb6b80a0110 0x7fb6b813bd90 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb6a000b550 tx=0x7fb6a000ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:50.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6b57fa640 1 -- 172.21.15.37:0/2452626397 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a00050f0 con 0x7fb6b80a0110 2024-04-26T00:29:50.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6b57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:50.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6b57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:29:50.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6b57fa640 10 monclient: dump: 2024-04-26T00:29:50.998 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:50.998 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:50.998 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:50.998 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:50.998 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:50.998 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:50.999 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:50.999 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:50.999 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:50.999 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:50.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6b57fa640 10 monclient: _finish_auth 0 2024-04-26T00:29:50.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6b57fa640 10 monclient: _check_auth_tickets 2024-04-26T00:29:50.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6b57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:29:50.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6b57fa640 1 -- 172.21.15.37:0/2452626397 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6a0015070 con 0x7fb6b80a0110 2024-04-26T00:29:50.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6b57fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:29:50.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6b57fa640 1 -- 172.21.15.37:0/2452626397 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a00054d0 con 0x7fb6b80a0110 2024-04-26T00:29:50.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6b57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:50.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6b57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:29:50.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6b57fa640 10 monclient: dump: 2024-04-26T00:29:50.999 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:50.999 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:51.000 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:51.000 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:51.000 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:51.000 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:51.000 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:51.000 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:51.000 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:51.000 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:51.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6bee72640 5 monclient: authenticate success, global_id 84452 2024-04-26T00:29:51.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6bee72640 10 monclient: _renew_subs 2024-04-26T00:29:51.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6b6ffd640 4 set_mon_vals no callback set 2024-04-26T00:29:51.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6bee72640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:29:51.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6bee72640 1 -- 172.21.15.37:0/2452626397 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6b806e4e0 con 0x7fb6b80a0110 2024-04-26T00:29:51.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6bee72640 10 monclient: _renew_subs 2024-04-26T00:29:51.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6bee72640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:29:51.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6bee72640 1 -- 172.21.15.37:0/2452626397 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6b806e200 con 0x7fb6b80a0110 2024-04-26T00:29:51.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6bee72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:29:51.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6bee72640 1 --2- 172.21.15.37:0/2452626397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6b813c810 0x7fb6b8137ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:51.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6bee72640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:51.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6bee72640 1 -- 172.21.15.37:0/2452626397 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb6b813cbf0 con 0x7fb6b813c810 2024-04-26T00:29:51.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6b57fa640 1 -- 172.21.15.37:0/2452626397 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb6a00056b0 con 0x7fb6b80a0110 2024-04-26T00:29:51.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6b57fa640 1 --2- 172.21.15.37:0/2452626397 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb698035e80 0x7fb698038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:51.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6b57fa640 1 -- 172.21.15.37:0/2452626397 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(373..373 src has 177..373) ==== 3931+0+0 (secure 0 0 0) 0x7fb6a0048980 con 0x7fb6b80a0110 2024-04-26T00:29:51.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6bcc0e640 1 --2- 172.21.15.37:0/2452626397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6b813c810 0x7fb6b8137ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:51.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6bcc0e640 10 monclient: get_auth_request con 0x7fb6b813c810 auth_method 0 2024-04-26T00:29:51.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6bcc0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:51.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6bcc0e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:51.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6bcc0e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:51.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6bcc0e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:51.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6bcc0e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:51.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6bcc0e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:51.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6bd40f640 1 --2- 172.21.15.37:0/2452626397 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb698035e80 0x7fb698038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:51.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6bd40f640 10 monclient: get_auth_request con 0x7fb698035e80 auth_method 0 2024-04-26T00:29:51.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6bcc0e640 10 monclient(hunting): handle_auth_done global_id 114160 payload 824 2024-04-26T00:29:51.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.002+0000 7fb6bcc0e640 1 -- 172.21.15.37:0/2452626397 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb6b804f510 con 0x7fb6b813c810 2024-04-26T00:29:51.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.006+0000 7fb6bcc0e640 1 -- 172.21.15.37:0/2452626397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6b813c810 msgr2=0x7fb6b8137ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:51.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.006+0000 7fb6bcc0e640 1 --2- 172.21.15.37:0/2452626397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6b813c810 0x7fb6b8137ec0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb6b000bd00 tx=0x7fb6b0009f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:51.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.006+0000 7fb6bd40f640 1 --2- 172.21.15.37:0/2452626397 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb698035e80 0x7fb698038340 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb6b80a2da0 tx=0x7fb6a80a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:51.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.006+0000 7fb6b57fa640 1 -- 172.21.15.37:0/2452626397 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b0015670 con 0x7fb6b813c810 2024-04-26T00:29:51.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.006+0000 7fb6b57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:51.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.006+0000 7fb6b57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:29:51.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.006+0000 7fb6b57fa640 10 monclient: dump: 2024-04-26T00:29:51.004 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:51.004 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:51.004 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:51.004 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:51.004 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:51.004 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:51.004 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:51.004 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:51.004 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:51.005 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:51.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.006+0000 7fb6b57fa640 1 -- 172.21.15.37:0/2452626397 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb6b0010070 con 0x7fb6b813c810 2024-04-26T00:29:51.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.006+0000 7fb6b57fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:29:51.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.006+0000 7fb6b57fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:29:51.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.006+0000 7fb6b57fa640 1 -- 172.21.15.37:0/2452626397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6b813c810 msgr2=0x7fb6b8137ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:51.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.006+0000 7fb6b57fa640 1 --2- 172.21.15.37:0/2452626397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6b813c810 0x7fb6b8137ec0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb6b000bd00 tx=0x7fb6b0009f70 comp rx=0 tx=0).stop 2024-04-26T00:29:51.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.006+0000 7fb6b57fa640 1 -- 172.21.15.37:0/2452626397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6b813c810 msgr2=0x7fb6b8137ec0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:29:51.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.006+0000 7fb6b57fa640 1 --2- 172.21.15.37:0/2452626397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6b813c810 0x7fb6b8137ec0 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb6b000bd00 tx=0x7fb6b0009f70 comp rx=0 tx=0).stop 2024-04-26T00:29:51.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.014+0000 7fb6bee72640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:29:51.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.014+0000 7fb6bee72640 1 --2- 172.21.15.37:0/2452626397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6b8138400 0x7fb6b81387e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:29:51.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.014+0000 7fb6bee72640 10 monclient(hunting): start opening mon connection 2024-04-26T00:29:51.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.014+0000 7fb6bee72640 1 -- 172.21.15.37:0/2452626397 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb6b8073280 con 0x7fb6b8138400 2024-04-26T00:29:51.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.014+0000 7fb6bcc0e640 1 --2- 172.21.15.37:0/2452626397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6b8138400 0x7fb6b81387e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:29:51.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.014+0000 7fb6bcc0e640 10 monclient: get_auth_request con 0x7fb6b8138400 auth_method 0 2024-04-26T00:29:51.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.014+0000 7fb6bcc0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:29:51.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.014+0000 7fb6bcc0e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:29:51.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.014+0000 7fb6bcc0e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:29:51.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.014+0000 7fb6bcc0e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:29:51.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.014+0000 7fb6bcc0e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:29:51.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.014+0000 7fb6bcc0e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:29:51.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.014+0000 7fb6bcc0e640 10 monclient(hunting): handle_auth_done global_id 114163 payload 824 2024-04-26T00:29:51.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.014+0000 7fb6bcc0e640 1 -- 172.21.15.37:0/2452626397 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb6b80008e0 con 0x7fb6b8138400 2024-04-26T00:29:51.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.014+0000 7fb6bcc0e640 1 -- 172.21.15.37:0/2452626397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6b8138400 msgr2=0x7fb6b81387e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:29:51.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.014+0000 7fb6bcc0e640 1 --2- 172.21.15.37:0/2452626397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6b8138400 0x7fb6b81387e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb6b0014000 tx=0x7fb6b0005d20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:29:51.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.018+0000 7fb6b57fa640 1 -- 172.21.15.37:0/2452626397 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b001a780 con 0x7fb6b8138400 2024-04-26T00:29:51.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.018+0000 7fb6b57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:29:51.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.018+0000 7fb6b57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:29:51.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.018+0000 7fb6b57fa640 10 monclient: dump: 2024-04-26T00:29:51.007 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:29:51.007 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:29:51.008 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:51.008 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:29:51.008 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:29:51.008 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:29:51.008 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:29:51.008 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:29:51.008 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:29:51.008 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:29:51.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.018+0000 7fb6b57fa640 1 -- 172.21.15.37:0/2452626397 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4875 (secure 0 0 0) 0x7fb6b0022440 con 0x7fb6b8138400 2024-04-26T00:29:51.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.018+0000 7fb6b57fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:29:51.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.018+0000 7fb6b57fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:29:51.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.018+0000 7fb6b57fa640 1 -- 172.21.15.37:0/2452626397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6b8138400 msgr2=0x7fb6b81387e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:51.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.018+0000 7fb6b57fa640 1 --2- 172.21.15.37:0/2452626397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6b8138400 0x7fb6b81387e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb6b0014000 tx=0x7fb6b0005d20 comp rx=0 tx=0).stop 2024-04-26T00:29:51.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.018+0000 7fb6b57fa640 1 -- 172.21.15.37:0/2452626397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6b8138400 msgr2=0x7fb6b81387e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:29:51.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.018+0000 7fb6b57fa640 1 --2- 172.21.15.37:0/2452626397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb6b8138400 0x7fb6b81387e0 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb6b0014000 tx=0x7fb6b0005d20 comp rx=0 tx=0).stop 2024-04-26T00:29:51.009 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:29:51.009 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:29:51.009 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:29:51.009 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:29:51.009 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 76, 2024-04-26T00:29:51.009 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 41103, 2024-04-26T00:29:51.009 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:29:51.009 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:29:51.009 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:29:51.009 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:51.009 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 1, 2024-04-26T00:29:51.010 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:29:51.010 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:29:51.010 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:29:51.010 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:29:51.010 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:29:51.010 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:29:51.010 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:29:51.010 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:29:51.010 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:29:51.010 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:29:51.010 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:29:51.010 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:29:51.010 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:29:51.010 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:29:51.011 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:51.011 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:29:51.011 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:29:51.011 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:29:51.011 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:29:51.011 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:29:51.011 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:29:51.011 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:29:51.011 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:29:51.011 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:29:51.011 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:29:51.012 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:29:51.012 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:29:51.012 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:29:51.012 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:51.012 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:51.012 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:29:51.012 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:29:51.012 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:29:51.013 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:29:51.013 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:29:51.013 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:29:51.013 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:29:51.013 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:29:51.013 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:29:51.013 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:29:51.013 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:29:51.013 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:29:51.013 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:29:51.013 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:29:51.013 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:29:51.013 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:29:51.013 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:29:51.014 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:29:51.014 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:29:51.014 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:29:51.014 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:29:51.014 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:29:51.014 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:29:51.014 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:29:51.014 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:29:51.014 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:29:51.014 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:29:51.014 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:29:51.014 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:51.014 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:29:51.015 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:51.015 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:29:51.015 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:51.015 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:29:51.015 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:29:51.015 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:29:51.015 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:29:51.015 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:51.015 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:29:51.015 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:29:51.015 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:51.015 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:51.015 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:51.015 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:29:51.015 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:29:51.016 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:51.016 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:51.016 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:51.016 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:51.016 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:29:51.016 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:29:51.016 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:29:51.016 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:29:51.016 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:29:51.016 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:51.016 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:51.016 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:29:51.016 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:29:51.016 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:29:51.017 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:29:51.017 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:51.017 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:29:51.017 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:29:51.017 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:51.017 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:51.017 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:51.017 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:29:51.017 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:29:51.017 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:51.017 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:51.017 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:51.017 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:51.017 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:29:51.018 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:29:51.018 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:29:51.018 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:29:51.018 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:29:51.018 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:51.018 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:51.018 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:29:51.018 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:29:51.018 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:29:51.018 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:29:51.018 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:51.018 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:29:51.018 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:29:51.018 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:51.019 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:51.019 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:51.019 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:29:51.019 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:29:51.019 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:29:51.019 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:51.019 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:51.019 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:51.019 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:29:51.019 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:29:51.019 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:29:51.019 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:29:51.019 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:29:51.019 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:51.020 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:51.020 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:51.020 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:29:51.020 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:29:51.020 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:51.020 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:29:51.020 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:29:51.020 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:29:51.020 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:51.020 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:51.020 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:29:51.020 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:51.020 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:29:51.020 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:29:51.020 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:29:51.021 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:51.021 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:51.021 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:29:51.021 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:51.021 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:29:51.021 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:29:51.021 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:29:51.021 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:51.021 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:29:51.021 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:29:51.021 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:29:51.021 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:29:51.021 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:29:51.021 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:29:51.021 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:29:51.022 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:29:51.022 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:29:51.022 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:29:51.022 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:29:51.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.018+0000 7fb6bee72640 1 -- 172.21.15.37:0/2452626397 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb698035e80 msgr2=0x7fb698038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:51.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.018+0000 7fb6bee72640 1 --2- 172.21.15.37:0/2452626397 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb698035e80 0x7fb698038340 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb6b80a2da0 tx=0x7fb6a80a6000 comp rx=0 tx=0).stop 2024-04-26T00:29:51.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.018+0000 7fb6bee72640 10 monclient: shutdown 2024-04-26T00:29:51.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.018+0000 7fb6bee72640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:29:51.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.018+0000 7fb6bee72640 1 -- 172.21.15.37:0/2452626397 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb6b80a0110 msgr2=0x7fb6b813bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:29:51.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.018+0000 7fb6bee72640 1 --2- 172.21.15.37:0/2452626397 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb6b80a0110 0x7fb6b813bd90 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb6a000b550 tx=0x7fb6a000ba10 comp rx=0 tx=0).stop 2024-04-26T00:29:51.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.018+0000 7fb6bd40f640 1 -- 172.21.15.37:0/2452626397 reap_dead start 2024-04-26T00:29:51.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.018+0000 7fb6bd40f640 1 -- 172.21.15.37:0/2452626397 reap_dead start 2024-04-26T00:29:51.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.018+0000 7fb6bee72640 1 -- 172.21.15.37:0/2452626397 shutdown_connections 2024-04-26T00:29:51.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.018+0000 7fb6bee72640 1 -- 172.21.15.37:0/2452626397 >> 172.21.15.37:0/2452626397 conn(0x7fb6b80a1d50 msgr2=0x7fb6b809d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:29:51.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.018+0000 7fb6bee72640 1 -- 172.21.15.37:0/2452626397 shutdown_connections 2024-04-26T00:29:51.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:29:51.018+0000 7fb6bee72640 1 -- 172.21.15.37:0/2452626397 wait complete. 2024-04-26T00:29:51.025 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:29:51.025 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-04-26T00:29:51.112 INFO:tasks.rados.rados.0.smithi045.stdout:1642: done (1 left) 2024-04-26T00:29:51.112 INFO:tasks.rados.rados.0.smithi045.stdout:1643: read oid 1 snap -1 2024-04-26T00:29:51.112 INFO:tasks.rados.rados.0.smithi045.stdout:1643: expect deleted 2024-04-26T00:29:51.112 INFO:tasks.rados.rados.0.smithi045.stdout:1644: snap_create 2024-04-26T00:29:51.112 INFO:tasks.rados.rados.0.smithi045.stdout:1641: finishing write tid 1 to smithi04526579-15 2024-04-26T00:29:51.112 INFO:tasks.rados.rados.0.smithi045.stdout:1641: finishing write tid 2 to smithi04526579-15 2024-04-26T00:29:51.112 INFO:tasks.rados.rados.0.smithi045.stdout:1641: finishing write tid 3 to smithi04526579-15 2024-04-26T00:29:51.112 INFO:tasks.rados.rados.0.smithi045.stdout:1641: finishing write tid 4 to smithi04526579-15 2024-04-26T00:29:51.112 INFO:tasks.rados.rados.0.smithi045.stdout:1641: oid 15 updating version 0 to 1070 2024-04-26T00:29:51.112 INFO:tasks.rados.rados.0.smithi045.stdout:1641: oid 15 version 1070 is already newer than 1069 2024-04-26T00:29:51.112 INFO:tasks.rados.rados.0.smithi045.stdout:1641: oid 15 updating version 1070 to 1071 2024-04-26T00:29:51.112 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 1071 (ObjNum 614 snap 186 seq_num 614) dirty exists 2024-04-26T00:29:51.113 INFO:tasks.rados.rados.0.smithi045.stdout:1641: left oid 15 (ObjNum 614 snap 186 seq_num 614) 2024-04-26T00:29:52.108 INFO:tasks.rados.rados.0.smithi045.stdout:1641: done (2 left) 2024-04-26T00:29:52.108 INFO:tasks.rados.rados.0.smithi045.stdout:1643: done (1 left) 2024-04-26T00:29:52.109 INFO:tasks.rados.rados.0.smithi045.stdout:1644: done (0 left) 2024-04-26T00:29:52.109 INFO:tasks.rados.rados.0.smithi045.stdout:1645: snap_create 2024-04-26T00:29:53.111 INFO:tasks.rados.rados.0.smithi045.stdout:1645: done (0 left) 2024-04-26T00:29:53.111 INFO:tasks.rados.rados.0.smithi045.stdout:1646: rollback oid 22 current snap is 188 2024-04-26T00:29:53.111 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 22 to 187 2024-04-26T00:29:53.111 INFO:tasks.rados.rados.0.smithi045.stdout:1647: write oid 11 current snap is 188 2024-04-26T00:29:53.111 INFO:tasks.rados.rados.0.smithi045.stdout:1647: seq_num 615 ranges {736305=786375,2105367=230938} 2024-04-26T00:29:53.137 INFO:tasks.rados.rados.0.smithi045.stdout:1647: writing smithi04526579-11 from 736305 to 1522680 tid 1 2024-04-26T00:29:53.151 INFO:tasks.rados.rados.0.smithi045.stdout:1647: writing smithi04526579-11 from 2105367 to 2336305 tid 2 2024-04-26T00:29:53.151 INFO:tasks.rados.rados.0.smithi045.stdout:1648: snap_create 2024-04-26T00:29:53.151 INFO:tasks.rados.rados.0.smithi045.stdout:1646: finishing rollback tid 0 to smithi04526579-22 2024-04-26T00:29:53.151 INFO:tasks.rados.rados.0.smithi045.stdout:1646: finishing rollback tid 1 to smithi04526579-22 2024-04-26T00:29:53.151 INFO:tasks.rados.rados.0.smithi045.stdout:1646: finishing rollback tid 2 to smithi04526579-22 2024-04-26T00:29:53.151 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 775 (ObjNum 587 snap 182 seq_num 587) dirty exists 2024-04-26T00:29:53.151 INFO:tasks.rados.rados.0.smithi045.stdout:1647: finishing write tid 1 to smithi04526579-11 2024-04-26T00:29:53.157 INFO:tasks.rados.rados.0.smithi045.stdout:1647: finishing write tid 2 to smithi04526579-11 2024-04-26T00:29:53.157 INFO:tasks.rados.rados.0.smithi045.stdout:1647: finishing write tid 3 to smithi04526579-11 2024-04-26T00:29:53.157 INFO:tasks.rados.rados.0.smithi045.stdout:1647: finishing write tid 4 to smithi04526579-11 2024-04-26T00:29:53.157 INFO:tasks.rados.rados.0.smithi045.stdout:1647: oid 11 updating version 0 to 975 2024-04-26T00:29:53.157 INFO:tasks.rados.rados.0.smithi045.stdout:1647: oid 11 updating version 975 to 976 2024-04-26T00:29:53.157 INFO:tasks.rados.rados.0.smithi045.stdout:1647: oid 11 updating version 976 to 977 2024-04-26T00:29:53.158 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 977 (ObjNum 615 snap 188 seq_num 615) dirty exists 2024-04-26T00:29:53.158 INFO:tasks.rados.rados.0.smithi045.stdout:1647: left oid 11 (ObjNum 615 snap 188 seq_num 615) 2024-04-26T00:29:54.127 INFO:tasks.rados.rados.0.smithi045.stdout:1646: done (2 left) 2024-04-26T00:29:54.128 INFO:tasks.rados.rados.0.smithi045.stdout:1647: done (1 left) 2024-04-26T00:29:54.128 INFO:tasks.rados.rados.0.smithi045.stdout:1648: done (0 left) 2024-04-26T00:29:54.128 INFO:tasks.rados.rados.0.smithi045.stdout:1649: write oid 45 current snap is 189 2024-04-26T00:29:54.128 INFO:tasks.rados.rados.0.smithi045.stdout:1649: seq_num 616 ranges {753112=577977,1880202=472910} 2024-04-26T00:29:54.149 INFO:tasks.rados.rados.0.smithi045.stdout:1649: writing smithi04526579-45 from 753112 to 1331089 tid 1 2024-04-26T00:29:54.167 INFO:tasks.rados.rados.0.smithi045.stdout:1649: writing smithi04526579-45 from 1880202 to 2353112 tid 2 2024-04-26T00:29:54.167 INFO:tasks.rados.rados.0.smithi045.stdout:1650: read oid 6 snap -1 2024-04-26T00:29:54.167 INFO:tasks.rados.rados.0.smithi045.stdout:1650: expect (ObjNum 574 snap 176 seq_num 574) 2024-04-26T00:29:54.167 INFO:tasks.rados.rados.0.smithi045.stdout:1651: snap_remove snap 143 2024-04-26T00:29:55.095 INFO:tasks.rados.rados.0.smithi045.stdout:1651: done (2 left) 2024-04-26T00:29:55.095 INFO:tasks.rados.rados.0.smithi045.stdout:1652: write (excl) oid 17 current snap is 189 2024-04-26T00:29:55.096 INFO:tasks.rados.rados.0.smithi045.stdout:1652: seq_num 617 ranges {769919=653226,1938684=431235} 2024-04-26T00:29:55.123 INFO:tasks.rados.rados.0.smithi045.stdout:1652: writing smithi04526579-17 from 769919 to 1423145 tid 1 2024-04-26T00:29:55.141 INFO:tasks.rados.rados.0.smithi045.stdout:1652: writing smithi04526579-17 from 1938684 to 2369919 tid 2 2024-04-26T00:29:55.141 INFO:tasks.rados.rados.0.smithi045.stdout:1649: finishing write tid 1 to smithi04526579-45 2024-04-26T00:29:55.141 INFO:tasks.rados.rados.0.smithi045.stdout:1653: read oid 30 snap 160 2024-04-26T00:29:55.141 INFO:tasks.rados.rados.0.smithi045.stdout:1653: expect (ObjNum 447 snap 139 seq_num 447) 2024-04-26T00:29:55.141 INFO:tasks.rados.rados.0.smithi045.stdout:1654: read oid 12 snap -1 2024-04-26T00:29:55.141 INFO:tasks.rados.rados.0.smithi045.stdout:1654: expect (ObjNum 401 snap 122 seq_num 401) 2024-04-26T00:29:55.141 INFO:tasks.rados.rados.0.smithi045.stdout:1655: delete oid 44 current snap is 189 2024-04-26T00:29:55.141 INFO:tasks.rados.rados.0.smithi045.stdout:1649: finishing write tid 2 to smithi04526579-45 2024-04-26T00:29:55.141 INFO:tasks.rados.rados.0.smithi045.stdout:1649: finishing write tid 3 to smithi04526579-45 2024-04-26T00:29:55.141 INFO:tasks.rados.rados.0.smithi045.stdout:1649: finishing write tid 4 to smithi04526579-45 2024-04-26T00:29:55.141 INFO:tasks.rados.rados.0.smithi045.stdout:1649: oid 45 updating version 0 to 979 2024-04-26T00:29:55.142 INFO:tasks.rados.rados.0.smithi045.stdout:1649: oid 45 version 979 is already newer than 978 2024-04-26T00:29:55.142 INFO:tasks.rados.rados.0.smithi045.stdout:1649: oid 45 updating version 979 to 980 2024-04-26T00:29:55.142 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 980 (ObjNum 616 snap 189 seq_num 616) dirty exists 2024-04-26T00:29:55.142 INFO:tasks.rados.rados.0.smithi045.stdout:1649: left oid 45 (ObjNum 616 snap 189 seq_num 616) 2024-04-26T00:29:55.142 INFO:tasks.rados.rados.0.smithi045.stdout:1650: expect (ObjNum 574 snap 176 seq_num 574) 2024-04-26T00:29:55.397 INFO:tasks.rados.rados.0.smithi045.stdout:1654: expect (ObjNum 401 snap 122 seq_num 401) 2024-04-26T00:29:55.487 INFO:tasks.rados.rados.0.smithi045.stdout:1652: finishing write tid 1 to smithi04526579-17 2024-04-26T00:29:55.487 INFO:tasks.rados.rados.0.smithi045.stdout:1652: finishing write tid 2 to smithi04526579-17 2024-04-26T00:29:55.488 INFO:tasks.rados.rados.0.smithi045.stdout:1652: finishing write tid 3 to smithi04526579-17 2024-04-26T00:29:55.488 INFO:tasks.rados.rados.0.smithi045.stdout:1652: finishing write tid 4 to smithi04526579-17 2024-04-26T00:29:55.488 INFO:tasks.rados.rados.0.smithi045.stdout:1652: oid 17 updating version 0 to 886 2024-04-26T00:29:55.488 INFO:tasks.rados.rados.0.smithi045.stdout:1652: oid 17 version 886 is already newer than 885 2024-04-26T00:29:55.488 INFO:tasks.rados.rados.0.smithi045.stdout:1652: oid 17 updating version 886 to 887 2024-04-26T00:29:55.488 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 887 (ObjNum 617 snap 189 seq_num 617) dirty exists 2024-04-26T00:29:55.488 INFO:tasks.rados.rados.0.smithi045.stdout:1652: left oid 17 (ObjNum 617 snap 189 seq_num 617) 2024-04-26T00:29:55.488 INFO:tasks.rados.rados.0.smithi045.stdout:1653: expect (ObjNum 447 snap 139 seq_num 447) 2024-04-26T00:29:55.637 INFO:tasks.rados.rados.0.smithi045.stdout:1649: done (5 left) 2024-04-26T00:29:55.638 INFO:tasks.rados.rados.0.smithi045.stdout:1650: done (4 left) 2024-04-26T00:29:55.638 INFO:tasks.rados.rados.0.smithi045.stdout:1652: done (3 left) 2024-04-26T00:29:55.638 INFO:tasks.rados.rados.0.smithi045.stdout:1653: done (2 left) 2024-04-26T00:29:55.638 INFO:tasks.rados.rados.0.smithi045.stdout:1654: done (1 left) 2024-04-26T00:29:55.638 INFO:tasks.rados.rados.0.smithi045.stdout:1655: done (0 left) 2024-04-26T00:29:55.638 INFO:tasks.rados.rados.0.smithi045.stdout:1656: snap_remove snap 183 2024-04-26T00:29:56.138 INFO:tasks.rados.rados.0.smithi045.stdout:1656: done (0 left) 2024-04-26T00:29:56.138 INFO:tasks.rados.rados.0.smithi045.stdout:1657: snap_create 2024-04-26T00:29:57.140 INFO:tasks.rados.rados.0.smithi045.stdout:1657: done (0 left) 2024-04-26T00:29:57.141 INFO:tasks.rados.rados.0.smithi045.stdout:1658: write (excl) oid 41 current snap is 190 2024-04-26T00:29:57.141 INFO:tasks.rados.rados.0.smithi045.stdout:1658: seq_num 619 ranges {403533=403724,1539295=652201} 2024-04-26T00:29:57.153 INFO:tasks.rados.rados.0.smithi045.stdout:1658: writing smithi04526579-41 from 403533 to 807257 tid 1 2024-04-26T00:29:57.177 INFO:tasks.rados.rados.0.smithi045.stdout:1658: writing smithi04526579-41 from 1539295 to 2191496 tid 2 2024-04-26T00:29:57.177 INFO:tasks.rados.rados.0.smithi045.stdout:1659: rollback oid 2 current snap is 190 2024-04-26T00:29:57.178 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 2 to 171 2024-04-26T00:29:57.178 INFO:tasks.rados.rados.0.smithi045.stdout:1658: finishing write tid 1 to smithi04526579-41 2024-04-26T00:29:57.178 INFO:tasks.rados.rados.0.smithi045.stdout:1660: copy_from oid 19 from oid 5 current snap is 190 2024-04-26T00:29:57.178 INFO:tasks.rados.rados.0.smithi045.stdout:1661: delete oid 37 current snap is 190 2024-04-26T00:29:57.188 INFO:tasks.rados.rados.0.smithi045.stdout:1660: finishing copy_from to smithi04526579-19 2024-04-26T00:29:57.188 INFO:tasks.rados.rados.0.smithi045.stdout:1660: got expected ENOENT (src dne) 2024-04-26T00:29:57.188 INFO:tasks.rados.rados.0.smithi045.stdout:1658: finishing write tid 2 to smithi04526579-41 2024-04-26T00:29:57.188 INFO:tasks.rados.rados.0.smithi045.stdout:1660: finishing copy_from racing read to smithi04526579-19 2024-04-26T00:29:57.189 INFO:tasks.rados.rados.0.smithi045.stdout:1658: finishing write tid 3 to smithi04526579-41 2024-04-26T00:29:57.189 INFO:tasks.rados.rados.0.smithi045.stdout:1658: finishing write tid 4 to smithi04526579-41 2024-04-26T00:29:57.189 INFO:tasks.rados.rados.0.smithi045.stdout:1658: oid 41 updating version 0 to 1074 2024-04-26T00:29:57.190 INFO:tasks.rados.rados.0.smithi045.stdout:1658: oid 41 version 1074 is already newer than 1073 2024-04-26T00:29:57.190 INFO:tasks.rados.rados.0.smithi045.stdout:1658: oid 41 version 1074 is already newer than 1072 2024-04-26T00:29:57.190 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 1074 (ObjNum 619 snap 190 seq_num 619) dirty exists 2024-04-26T00:29:57.190 INFO:tasks.rados.rados.0.smithi045.stdout:1658: left oid 41 (ObjNum 619 snap 190 seq_num 619) 2024-04-26T00:29:57.191 INFO:tasks.rados.rados.0.smithi045.stdout:1658: done (3 left) 2024-04-26T00:29:57.191 INFO:tasks.rados.rados.0.smithi045.stdout:1660: done (2 left) 2024-04-26T00:29:57.191 INFO:tasks.rados.rados.0.smithi045.stdout:1661: done (1 left) 2024-04-26T00:29:57.191 INFO:tasks.rados.rados.0.smithi045.stdout:1662: rollback oid 47 current snap is 190 2024-04-26T00:29:57.191 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 47 to 170 2024-04-26T00:29:57.191 INFO:tasks.rados.rados.0.smithi045.stdout:1663: snap_create 2024-04-26T00:29:57.195 INFO:tasks.rados.rados.0.smithi045.stdout:1659: finishing rollback tid 0 to smithi04526579-2 2024-04-26T00:29:57.195 INFO:tasks.rados.rados.0.smithi045.stdout:1659: finishing rollback tid 1 to smithi04526579-2 2024-04-26T00:29:57.197 INFO:tasks.rados.rados.0.smithi045.stdout:1659: finishing rollback tid 2 to smithi04526579-2 2024-04-26T00:29:57.197 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 911 (ObjNum 544 snap 168 seq_num 544) dirty exists 2024-04-26T00:29:57.198 INFO:tasks.rados.rados.0.smithi045.stdout:1662: finishing rollback tid 1 to smithi04526579-47 2024-04-26T00:29:57.199 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 831 (ObjNum 129 snap 0 seq_num 2241767080) dirty dne 2024-04-26T00:29:58.195 INFO:tasks.rados.rados.0.smithi045.stdout:1659: done (2 left) 2024-04-26T00:29:58.195 INFO:tasks.rados.rados.0.smithi045.stdout:1662: done (1 left) 2024-04-26T00:29:58.195 INFO:tasks.rados.rados.0.smithi045.stdout:1663: done (0 left) 2024-04-26T00:29:58.195 INFO:tasks.rados.rados.0.smithi045.stdout:1664: read oid 3 snap -1 2024-04-26T00:29:58.195 INFO:tasks.rados.rados.0.smithi045.stdout:1664: expect deleted 2024-04-26T00:29:58.195 INFO:tasks.rados.rados.0.smithi045.stdout:1665: read oid 29 snap -1 2024-04-26T00:29:58.195 INFO:tasks.rados.rados.0.smithi045.stdout:1665: expect deleted 2024-04-26T00:29:58.195 INFO:tasks.rados.rados.0.smithi045.stdout:1666: read oid 39 snap -1 2024-04-26T00:29:58.195 INFO:tasks.rados.rados.0.smithi045.stdout:1666: expect deleted 2024-04-26T00:29:58.195 INFO:tasks.rados.rados.0.smithi045.stdout:1667: copy_from oid 47 from oid 35 current snap is 191 2024-04-26T00:29:58.196 INFO:tasks.rados.rados.0.smithi045.stdout:1668: rollback oid 12 current snap is 191 2024-04-26T00:29:58.196 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 12 to 140 2024-04-26T00:29:58.196 INFO:tasks.rados.rados.0.smithi045.stdout:1669: read oid 11 snap -1 2024-04-26T00:29:58.196 INFO:tasks.rados.rados.0.smithi045.stdout:1669: expect (ObjNum 615 snap 188 seq_num 615) 2024-04-26T00:29:58.196 INFO:tasks.rados.rados.0.smithi045.stdout:1670: rollback oid 50 current snap is 191 2024-04-26T00:29:58.196 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 50 to 154 2024-04-26T00:29:58.196 INFO:tasks.rados.rados.0.smithi045.stdout:1671: copy_from oid 38 from oid 28 current snap is 191 2024-04-26T00:29:58.196 INFO:tasks.rados.rados.0.smithi045.stdout:1672: delete oid 5 current snap is 191 2024-04-26T00:29:58.203 INFO:tasks.rados.rados.0.smithi045.stdout:1668: finishing rollback tid 0 to smithi04526579-12 2024-04-26T00:29:58.204 INFO:tasks.rados.rados.0.smithi045.stdout:1668: finishing rollback tid 1 to smithi04526579-12 2024-04-26T00:29:58.204 INFO:tasks.rados.rados.0.smithi045.stdout:1668: finishing rollback tid 2 to smithi04526579-12 2024-04-26T00:29:58.204 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 619 (ObjNum 401 snap 122 seq_num 401) dirty exists 2024-04-26T00:29:58.205 INFO:tasks.rados.rados.0.smithi045.stdout:1670: finishing rollback tid 1 to smithi04526579-50 2024-04-26T00:29:58.205 INFO:tasks.rados.rados.0.smithi045.stdout:1670: finishing rollback tid 2 to smithi04526579-50 2024-04-26T00:29:58.205 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 889 (ObjNum 470 snap 147 seq_num 470) dirty exists 2024-04-26T00:29:58.208 INFO:tasks.rados.rados.0.smithi045.stdout:1669: expect (ObjNum 615 snap 188 seq_num 615) 2024-04-26T00:29:58.317 INFO:tasks.rados.rados.0.smithi045.stdout:1667: finishing copy_from racing read to smithi04526579-47 2024-04-26T00:29:58.317 INFO:tasks.rados.rados.0.smithi045.stdout:1667: finishing copy_from to smithi04526579-47 2024-04-26T00:29:58.317 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 977 (ObjNum 553 snap 168 seq_num 553) dirty exists 2024-04-26T00:29:58.317 INFO:tasks.rados.rados.0.smithi045.stdout:1671: finishing copy_from racing read to smithi04526579-38 2024-04-26T00:29:58.317 INFO:tasks.rados.rados.0.smithi045.stdout:1671: finishing copy_from to smithi04526579-38 2024-04-26T00:29:58.317 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 1059 (ObjNum 582 snap 180 seq_num 582) dirty exists 2024-04-26T00:29:58.318 INFO:tasks.rados.rados.0.smithi045.stdout:1664: done (8 left) 2024-04-26T00:29:58.318 INFO:tasks.rados.rados.0.smithi045.stdout:1665: done (7 left) 2024-04-26T00:29:58.318 INFO:tasks.rados.rados.0.smithi045.stdout:1666: done (6 left) 2024-04-26T00:29:58.318 INFO:tasks.rados.rados.0.smithi045.stdout:1667: done (5 left) 2024-04-26T00:29:58.318 INFO:tasks.rados.rados.0.smithi045.stdout:1668: done (4 left) 2024-04-26T00:29:58.318 INFO:tasks.rados.rados.0.smithi045.stdout:1669: done (3 left) 2024-04-26T00:29:58.318 INFO:tasks.rados.rados.0.smithi045.stdout:1670: done (2 left) 2024-04-26T00:29:58.318 INFO:tasks.rados.rados.0.smithi045.stdout:1671: done (1 left) 2024-04-26T00:29:58.318 INFO:tasks.rados.rados.0.smithi045.stdout:1672: done (0 left) 2024-04-26T00:29:58.318 INFO:tasks.rados.rados.0.smithi045.stdout:1673: snap_remove snap 188 2024-04-26T00:29:59.200 INFO:tasks.rados.rados.0.smithi045.stdout:1673: done (0 left) 2024-04-26T00:29:59.200 INFO:tasks.rados.rados.0.smithi045.stdout:1674: write oid 40 current snap is 191 2024-04-26T00:29:59.200 INFO:tasks.rados.rados.0.smithi045.stdout:1674: seq_num 622 ranges {453954=629471,1598388=515881} 2024-04-26T00:29:59.219 INFO:tasks.rados.rados.0.smithi045.stdout:1674: writing smithi04526579-40 from 453954 to 1083425 tid 1 2024-04-26T00:29:59.235 INFO:tasks.rados.rados.0.smithi045.stdout:1674: writing smithi04526579-40 from 1598388 to 2114269 tid 2 2024-04-26T00:29:59.235 INFO:tasks.rados.rados.0.smithi045.stdout:1675: delete oid 32 current snap is 191 2024-04-26T00:29:59.235 INFO:tasks.rados.rados.0.smithi045.stdout:1674: finishing write tid 1 to smithi04526579-40 2024-04-26T00:29:59.242 INFO:tasks.rados.rados.0.smithi045.stdout:1674: finishing write tid 2 to smithi04526579-40 2024-04-26T00:29:59.242 INFO:tasks.rados.rados.0.smithi045.stdout:1674: finishing write tid 3 to smithi04526579-40 2024-04-26T00:29:59.243 INFO:tasks.rados.rados.0.smithi045.stdout:1674: finishing write tid 4 to smithi04526579-40 2024-04-26T00:29:59.243 INFO:tasks.rados.rados.0.smithi045.stdout:1674: oid 40 updating version 0 to 621 2024-04-26T00:29:59.243 INFO:tasks.rados.rados.0.smithi045.stdout:1674: oid 40 updating version 621 to 622 2024-04-26T00:29:59.243 INFO:tasks.rados.rados.0.smithi045.stdout:1674: oid 40 version 622 is already newer than 620 2024-04-26T00:29:59.243 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 622 (ObjNum 622 snap 191 seq_num 622) dirty exists 2024-04-26T00:29:59.243 INFO:tasks.rados.rados.0.smithi045.stdout:1674: left oid 40 (ObjNum 622 snap 191 seq_num 622) 2024-04-26T00:29:59.248 INFO:tasks.rados.rados.0.smithi045.stdout:1674: done (1 left) 2024-04-26T00:29:59.248 INFO:tasks.rados.rados.0.smithi045.stdout:1675: done (0 left) 2024-04-26T00:29:59.248 INFO:tasks.rados.rados.0.smithi045.stdout:1676: copy_from oid 17 from oid 30 current snap is 191 2024-04-26T00:29:59.248 INFO:tasks.rados.rados.0.smithi045.stdout:1677: snap_create 2024-04-26T00:29:59.268 INFO:tasks.rados.rados.0.smithi045.stdout:1676: finishing copy_from racing read to smithi04526579-17 2024-04-26T00:29:59.274 INFO:tasks.rados.rados.0.smithi045.stdout:1676: finishing copy_from to smithi04526579-17 2024-04-26T00:29:59.274 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 1046 (ObjNum 595 snap 184 seq_num 595) dirty exists 2024-04-26T00:30:00.097 INFO:tasks.rados.rados.0.smithi045.stdout:1676: done (1 left) 2024-04-26T00:30:00.098 INFO:tasks.rados.rados.0.smithi045.stdout:1677: done (0 left) 2024-04-26T00:30:00.098 INFO:tasks.rados.rados.0.smithi045.stdout:1678: snap_remove snap 185 2024-04-26T00:30:01.102 INFO:tasks.rados.rados.0.smithi045.stdout:1678: done (0 left) 2024-04-26T00:30:01.102 INFO:tasks.rados.rados.0.smithi045.stdout:1679: rollback oid 44 current snap is 192 2024-04-26T00:30:01.102 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 44 to 190 2024-04-26T00:30:01.102 INFO:tasks.rados.rados.0.smithi045.stdout:1680: copy_from oid 23 from oid 17 current snap is 192 2024-04-26T00:30:01.102 INFO:tasks.rados.rados.0.smithi045.stdout:1681: read oid 2 snap -1 2024-04-26T00:30:01.102 INFO:tasks.rados.rados.0.smithi045.stdout:1681: expect (ObjNum 544 snap 168 seq_num 544) 2024-04-26T00:30:01.103 INFO:tasks.rados.rados.0.smithi045.stdout:1682: delete oid 9 current snap is 192 2024-04-26T00:30:01.116 INFO:tasks.rados.rados.0.smithi045.stdout:1679: finishing rollback tid 1 to smithi04526579-44 2024-04-26T00:30:01.116 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 1050 (ObjNum 49 snap 0 seq_num 2241747392) dirty dne 2024-04-26T00:30:01.130 INFO:tasks.rados.rados.0.smithi045.stdout:1680: finishing copy_from racing read to smithi04526579-23 2024-04-26T00:30:01.131 INFO:tasks.rados.rados.0.smithi045.stdout:1681: expect (ObjNum 544 snap 168 seq_num 544) 2024-04-26T00:30:01.272 INFO:tasks.rados.rados.0.smithi045.stdout:1680: finishing copy_from to smithi04526579-23 2024-04-26T00:30:01.273 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 1046 (ObjNum 595 snap 184 seq_num 595) dirty exists 2024-04-26T00:30:01.273 INFO:tasks.rados.rados.0.smithi045.stdout:1679: done (3 left) 2024-04-26T00:30:01.273 INFO:tasks.rados.rados.0.smithi045.stdout:1680: done (2 left) 2024-04-26T00:30:01.273 INFO:tasks.rados.rados.0.smithi045.stdout:1681: done (1 left) 2024-04-26T00:30:01.273 INFO:tasks.rados.rados.0.smithi045.stdout:1682: done (0 left) 2024-04-26T00:30:01.273 INFO:tasks.rados.rados.0.smithi045.stdout:1683: rollback oid 24 current snap is 192 2024-04-26T00:30:01.273 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 24 to 175 2024-04-26T00:30:01.273 INFO:tasks.rados.rados.0.smithi045.stdout:1684: read oid 16 snap -1 2024-04-26T00:30:01.273 INFO:tasks.rados.rados.0.smithi045.stdout:1684: expect deleted 2024-04-26T00:30:01.273 INFO:tasks.rados.rados.0.smithi045.stdout:1685: rollback oid 22 current snap is 192 2024-04-26T00:30:01.273 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 22 to 169 2024-04-26T00:30:01.273 INFO:tasks.rados.rados.0.smithi045.stdout:1686: delete oid 41 current snap is 192 2024-04-26T00:30:01.277 INFO:tasks.rados.rados.0.smithi045.stdout:1684: done (3 left) 2024-04-26T00:30:01.277 INFO:tasks.rados.rados.0.smithi045.stdout:1686: done (2 left) 2024-04-26T00:30:01.277 INFO:tasks.rados.rados.0.smithi045.stdout:1687: write oid 6 current snap is 192 2024-04-26T00:30:01.277 INFO:tasks.rados.rados.0.smithi045.stdout:1687: seq_num 626 ranges {521182=646820,1948669=523219} 2024-04-26T00:30:01.283 INFO:tasks.rados.rados.0.smithi045.stdout:1687: writing smithi04526579-6 from 521182 to 1168002 tid 1 2024-04-26T00:30:01.292 INFO:tasks.rados.rados.0.smithi045.stdout:1687: writing smithi04526579-6 from 1948669 to 2471888 tid 2 2024-04-26T00:30:01.292 INFO:tasks.rados.rados.0.smithi045.stdout:1688: read oid 10 snap 191 2024-04-26T00:30:01.293 INFO:tasks.rados.rados.0.smithi045.stdout:1688: expect (ObjNum 573 snap 176 seq_num 573) 2024-04-26T00:30:01.293 INFO:tasks.rados.rados.0.smithi045.stdout:1689: write oid 5 current snap is 192 2024-04-26T00:30:01.293 INFO:tasks.rados.rados.0.smithi045.stdout:1683: finishing rollback tid 1 to smithi04526579-24 2024-04-26T00:30:01.293 INFO:tasks.rados.rados.0.smithi045.stdout:1683: finishing rollback tid 2 to smithi04526579-24 2024-04-26T00:30:01.293 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 1353 (ObjNum 534 snap 165 seq_num 534) dirty exists 2024-04-26T00:30:01.293 INFO:tasks.rados.rados.0.smithi045.stdout:1685: finishing rollback tid 0 to smithi04526579-22 2024-04-26T00:30:01.293 INFO:tasks.rados.rados.0.smithi045.stdout:1685: finishing rollback tid 1 to smithi04526579-22 2024-04-26T00:30:01.293 INFO:tasks.rados.rados.0.smithi045.stdout:1689: seq_num 627 ranges {537989=722069,1890798=647191} 2024-04-26T00:30:01.301 INFO:tasks.rados.rados.0.smithi045.stdout:1689: writing smithi04526579-5 from 537989 to 1260058 tid 1 2024-04-26T00:30:01.309 INFO:tasks.rados.rados.0.smithi045.stdout:1689: writing smithi04526579-5 from 1890798 to 2537989 tid 2 2024-04-26T00:30:01.309 INFO:tasks.rados.rados.0.smithi045.stdout:1683: done (4 left) 2024-04-26T00:30:01.309 INFO:tasks.rados.rados.0.smithi045.stdout:1690: copy_from oid 14 from oid 34 current snap is 192 2024-04-26T00:30:01.309 INFO:tasks.rados.rados.0.smithi045.stdout:1685: finishing rollback tid 2 to smithi04526579-22 2024-04-26T00:30:01.309 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 778 (ObjNum 330 snap 96 seq_num 330) dirty exists 2024-04-26T00:30:01.309 INFO:tasks.rados.rados.0.smithi045.stdout:1687: finishing write tid 1 to smithi04526579-6 2024-04-26T00:30:01.309 INFO:tasks.rados.rados.0.smithi045.stdout:1687: finishing write tid 2 to smithi04526579-6 2024-04-26T00:30:01.309 INFO:tasks.rados.rados.0.smithi045.stdout:1688: expect (ObjNum 573 snap 176 seq_num 573) 2024-04-26T00:30:01.434 INFO:tasks.rados.rados.0.smithi045.stdout:1687: finishing write tid 3 to smithi04526579-6 2024-04-26T00:30:01.434 INFO:tasks.rados.rados.0.smithi045.stdout:1687: finishing write tid 4 to smithi04526579-6 2024-04-26T00:30:01.434 INFO:tasks.rados.rados.0.smithi045.stdout:1687: oid 6 updating version 0 to 1048 2024-04-26T00:30:01.434 INFO:tasks.rados.rados.0.smithi045.stdout:1687: oid 6 updating version 1048 to 1049 2024-04-26T00:30:01.434 INFO:tasks.rados.rados.0.smithi045.stdout:1687: oid 6 version 1049 is already newer than 1047 2024-04-26T00:30:01.434 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 1049 (ObjNum 626 snap 192 seq_num 626) dirty exists 2024-04-26T00:30:01.434 INFO:tasks.rados.rados.0.smithi045.stdout:1687: left oid 6 (ObjNum 626 snap 192 seq_num 626) 2024-04-26T00:30:01.434 INFO:tasks.rados.rados.0.smithi045.stdout:1689: finishing write tid 1 to smithi04526579-5 2024-04-26T00:30:01.434 INFO:tasks.rados.rados.0.smithi045.stdout:1689: finishing write tid 2 to smithi04526579-5 2024-04-26T00:30:01.435 INFO:tasks.rados.rados.0.smithi045.stdout:1689: finishing write tid 3 to smithi04526579-5 2024-04-26T00:30:01.435 INFO:tasks.rados.rados.0.smithi045.stdout:1689: finishing write tid 4 to smithi04526579-5 2024-04-26T00:30:01.435 INFO:tasks.rados.rados.0.smithi045.stdout:1689: oid 5 updating version 0 to 1354 2024-04-26T00:30:01.435 INFO:tasks.rados.rados.0.smithi045.stdout:1689: oid 5 updating version 1354 to 1356 2024-04-26T00:30:01.435 INFO:tasks.rados.rados.0.smithi045.stdout:1689: oid 5 version 1356 is already newer than 1355 2024-04-26T00:30:01.435 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 1356 (ObjNum 627 snap 192 seq_num 627) dirty exists 2024-04-26T00:30:01.435 INFO:tasks.rados.rados.0.smithi045.stdout:1689: left oid 5 (ObjNum 627 snap 192 seq_num 627) 2024-04-26T00:30:01.435 INFO:tasks.rados.rados.0.smithi045.stdout:1690: finishing copy_from racing read to smithi04526579-14 2024-04-26T00:30:01.435 INFO:tasks.rados.rados.0.smithi045.stdout:1690: finishing copy_from to smithi04526579-14 2024-04-26T00:30:01.435 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 737 (ObjNum 463 snap 145 seq_num 463) dirty exists 2024-04-26T00:30:01.435 INFO:tasks.rados.rados.0.smithi045.stdout:1685: done (4 left) 2024-04-26T00:30:01.435 INFO:tasks.rados.rados.0.smithi045.stdout:1687: done (3 left) 2024-04-26T00:30:01.435 INFO:tasks.rados.rados.0.smithi045.stdout:1688: done (2 left) 2024-04-26T00:30:01.435 INFO:tasks.rados.rados.0.smithi045.stdout:1689: done (1 left) 2024-04-26T00:30:01.436 INFO:tasks.rados.rados.0.smithi045.stdout:1690: done (0 left) 2024-04-26T00:30:01.436 INFO:tasks.rados.rados.0.smithi045.stdout:1691: snap_remove snap 162 2024-04-26T00:30:02.219 INFO:tasks.rados.rados.0.smithi045.stdout:1691: done (0 left) 2024-04-26T00:30:02.220 INFO:tasks.rados.rados.0.smithi045.stdout:1692: read oid 25 snap -1 2024-04-26T00:30:02.220 INFO:tasks.rados.rados.0.smithi045.stdout:1692: expect (ObjNum 604 snap 186 seq_num 604) 2024-04-26T00:30:02.220 INFO:tasks.rados.rados.0.smithi045.stdout:1693: snap_create 2024-04-26T00:30:02.242 INFO:tasks.rados.rados.0.smithi045.stdout:1692: expect (ObjNum 604 snap 186 seq_num 604) 2024-04-26T00:30:03.223 INFO:tasks.rados.rados.0.smithi045.stdout:1692: done (1 left) 2024-04-26T00:30:03.223 INFO:tasks.rados.rados.0.smithi045.stdout:1693: done (0 left) 2024-04-26T00:30:03.223 INFO:tasks.rados.rados.0.smithi045.stdout:1694: copy_from oid 42 from oid 31 current snap is 193 2024-04-26T00:30:03.223 INFO:tasks.rados.rados.0.smithi045.stdout:1695: read oid 24 snap -1 2024-04-26T00:30:03.223 INFO:tasks.rados.rados.0.smithi045.stdout:1695: expect (ObjNum 534 snap 165 seq_num 534) 2024-04-26T00:30:03.223 INFO:tasks.rados.rados.0.smithi045.stdout:1696: snap_create 2024-04-26T00:30:03.228 INFO:tasks.rados.rados.0.smithi045.stdout:1694: finishing copy_from to smithi04526579-42 2024-04-26T00:30:03.229 INFO:tasks.rados.rados.0.smithi045.stdout:1694: got expected ENOENT (src dne) 2024-04-26T00:30:03.229 INFO:tasks.rados.rados.0.smithi045.stdout:1694: finishing copy_from racing read to smithi04526579-42 2024-04-26T00:30:03.243 INFO:tasks.rados.rados.0.smithi045.stdout:1695: expect (ObjNum 534 snap 165 seq_num 534) 2024-04-26T00:30:04.232 INFO:tasks.rados.rados.0.smithi045.stdout:1694: done (2 left) 2024-04-26T00:30:04.232 INFO:tasks.rados.rados.0.smithi045.stdout:1695: done (1 left) 2024-04-26T00:30:04.233 INFO:tasks.rados.rados.0.smithi045.stdout:1696: done (0 left) 2024-04-26T00:30:04.233 INFO:tasks.rados.rados.0.smithi045.stdout:1697: rollback oid 40 current snap is 194 2024-04-26T00:30:04.233 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 40 to 178 2024-04-26T00:30:04.233 INFO:tasks.rados.rados.0.smithi045.stdout:1698: delete oid 6 current snap is 194 2024-04-26T00:30:04.242 INFO:tasks.rados.rados.0.smithi045.stdout:1698: done (1 left) 2024-04-26T00:30:04.242 INFO:tasks.rados.rados.0.smithi045.stdout:1699: write (excl) oid 5 current snap is 194 2024-04-26T00:30:04.242 INFO:tasks.rados.rados.0.smithi045.stdout:1699: seq_num 629 ranges {571603=472567,1607762=670546} 2024-04-26T00:30:04.259 INFO:tasks.rados.rados.0.smithi045.stdout:1699: writing smithi04526579-5 from 571603 to 1044170 tid 1 2024-04-26T00:30:04.279 INFO:tasks.rados.rados.0.smithi045.stdout:1699: writing smithi04526579-5 from 1607762 to 2278308 tid 2 2024-04-26T00:30:04.279 INFO:tasks.rados.rados.0.smithi045.stdout:1700: snap_remove snap 186 2024-04-26T00:30:05.100 INFO:tasks.rados.rados.0.smithi045.stdout:1700: done (2 left) 2024-04-26T00:30:05.100 INFO:tasks.rados.rados.0.smithi045.stdout:1701: copy_from oid 15 from oid 13 current snap is 194 2024-04-26T00:30:05.100 INFO:tasks.rados.rados.0.smithi045.stdout:1702: snap_create 2024-04-26T00:30:05.101 INFO:tasks.rados.rados.0.smithi045.stdout:1697: finishing rollback tid 0 to smithi04526579-40 2024-04-26T00:30:05.101 INFO:tasks.rados.rados.0.smithi045.stdout:1697: finishing rollback tid 1 to smithi04526579-40 2024-04-26T00:30:05.101 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 1048 (ObjNum 33 snap 0 seq_num 2241633136) dirty dne 2024-04-26T00:30:05.101 INFO:tasks.rados.rados.0.smithi045.stdout:1699: finishing write tid 1 to smithi04526579-5 2024-04-26T00:30:05.102 INFO:tasks.rados.rados.0.smithi045.stdout:1699: finishing write tid 2 to smithi04526579-5 2024-04-26T00:30:05.102 INFO:tasks.rados.rados.0.smithi045.stdout:1699: finishing write tid 3 to smithi04526579-5 2024-04-26T00:30:05.102 INFO:tasks.rados.rados.0.smithi045.stdout:1699: finishing write tid 4 to smithi04526579-5 2024-04-26T00:30:05.102 INFO:tasks.rados.rados.0.smithi045.stdout:1699: oid 5 updating version 0 to 1371 2024-04-26T00:30:05.102 INFO:tasks.rados.rados.0.smithi045.stdout:1699: oid 5 updating version 1371 to 1372 2024-04-26T00:30:05.102 INFO:tasks.rados.rados.0.smithi045.stdout:1699: oid 5 version 1372 is already newer than 1370 2024-04-26T00:30:05.102 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 1372 (ObjNum 629 snap 194 seq_num 629) dirty exists 2024-04-26T00:30:05.102 INFO:tasks.rados.rados.0.smithi045.stdout:1699: left oid 5 (ObjNum 629 snap 194 seq_num 629) 2024-04-26T00:30:05.119 INFO:tasks.rados.rados.0.smithi045.stdout:1701: finishing copy_from to smithi04526579-15 2024-04-26T00:30:05.119 INFO:tasks.rados.rados.0.smithi045.stdout:1701: got expected ENOENT (src dne) 2024-04-26T00:30:05.120 INFO:tasks.rados.rados.0.smithi045.stdout:1701: finishing copy_from racing read to smithi04526579-15 2024-04-26T00:30:06.109 INFO:tasks.rados.rados.0.smithi045.stdout:1697: done (3 left) 2024-04-26T00:30:06.109 INFO:tasks.rados.rados.0.smithi045.stdout:1699: done (2 left) 2024-04-26T00:30:06.110 INFO:tasks.rados.rados.0.smithi045.stdout:1701: done (1 left) 2024-04-26T00:30:06.110 INFO:tasks.rados.rados.0.smithi045.stdout:1702: done (0 left) 2024-04-26T00:30:06.110 INFO:tasks.rados.rados.0.smithi045.stdout:1703: rollback oid 10 current snap is 195 2024-04-26T00:30:06.110 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 10 to 189 2024-04-26T00:30:06.110 INFO:tasks.rados.rados.0.smithi045.stdout:1704: rollback oid 1 current snap is 195 2024-04-26T00:30:06.110 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 1 to 175 2024-04-26T00:30:06.110 INFO:tasks.rados.rados.0.smithi045.stdout:1705: write oid 13 current snap is 195 2024-04-26T00:30:06.110 INFO:tasks.rados.rados.0.smithi045.stdout:1705: seq_num 630 ranges {588410=547816,1666244=414204,2484603=103807} 2024-04-26T00:30:06.128 INFO:tasks.rados.rados.0.smithi045.stdout:1705: writing smithi04526579-13 from 588410 to 1136226 tid 1 2024-04-26T00:30:06.145 INFO:tasks.rados.rados.0.smithi045.stdout:1705: writing smithi04526579-13 from 1666244 to 2080448 tid 2 2024-04-26T00:30:06.154 INFO:tasks.rados.rados.0.smithi045.stdout:1705: writing smithi04526579-13 from 2484603 to 2588410 tid 3 2024-04-26T00:30:06.154 INFO:tasks.rados.rados.0.smithi045.stdout:1706: snap_create 2024-04-26T00:30:06.154 INFO:tasks.rados.rados.0.smithi045.stdout:1704: finishing rollback tid 1 to smithi04526579-1 2024-04-26T00:30:06.154 INFO:tasks.rados.rados.0.smithi045.stdout:1704: finishing rollback tid 2 to smithi04526579-1 2024-04-26T00:30:06.154 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 1077 (ObjNum 561 snap 175 seq_num 561) dirty exists 2024-04-26T00:30:06.154 INFO:tasks.rados.rados.0.smithi045.stdout:1703: finishing rollback tid 0 to smithi04526579-10 2024-04-26T00:30:06.154 INFO:tasks.rados.rados.0.smithi045.stdout:1703: finishing rollback tid 1 to smithi04526579-10 2024-04-26T00:30:06.154 INFO:tasks.rados.rados.0.smithi045.stdout:1703: finishing rollback tid 2 to smithi04526579-10 2024-04-26T00:30:06.154 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 1049 (ObjNum 573 snap 176 seq_num 573) dirty exists 2024-04-26T00:30:06.154 INFO:tasks.rados.rados.0.smithi045.stdout:1705: finishing write tid 1 to smithi04526579-13 2024-04-26T00:30:06.155 INFO:tasks.rados.rados.0.smithi045.stdout:1705: finishing write tid 2 to smithi04526579-13 2024-04-26T00:30:06.158 INFO:tasks.rados.rados.0.smithi045.stdout:1705: finishing write tid 3 to smithi04526579-13 2024-04-26T00:30:06.159 INFO:tasks.rados.rados.0.smithi045.stdout:1705: finishing write tid 4 to smithi04526579-13 2024-04-26T00:30:06.159 INFO:tasks.rados.rados.0.smithi045.stdout:1705: finishing write tid 5 to smithi04526579-13 2024-04-26T00:30:06.160 INFO:tasks.rados.rados.0.smithi045.stdout:1705: oid 13 updating version 0 to 872 2024-04-26T00:30:06.160 INFO:tasks.rados.rados.0.smithi045.stdout:1705: oid 13 updating version 872 to 874 2024-04-26T00:30:06.160 INFO:tasks.rados.rados.0.smithi045.stdout:1705: oid 13 version 874 is already newer than 873 2024-04-26T00:30:06.160 INFO:tasks.rados.rados.0.smithi045.stdout:1705: oid 13 version 874 is already newer than 871 2024-04-26T00:30:06.160 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 874 (ObjNum 630 snap 195 seq_num 630) dirty exists 2024-04-26T00:30:06.160 INFO:tasks.rados.rados.0.smithi045.stdout:1705: left oid 13 (ObjNum 630 snap 195 seq_num 630) 2024-04-26T00:30:07.118 INFO:tasks.rados.rados.0.smithi045.stdout:1703: done (3 left) 2024-04-26T00:30:07.119 INFO:tasks.rados.rados.0.smithi045.stdout:1704: done (2 left) 2024-04-26T00:30:07.119 INFO:tasks.rados.rados.0.smithi045.stdout:1705: done (1 left) 2024-04-26T00:30:07.119 INFO:tasks.rados.rados.0.smithi045.stdout:1706: done (0 left) 2024-04-26T00:30:07.119 INFO:tasks.rados.rados.0.smithi045.stdout:1707: read oid 26 snap -1 2024-04-26T00:30:07.119 INFO:tasks.rados.rados.0.smithi045.stdout:1707: expect (ObjNum 605 snap 186 seq_num 605) 2024-04-26T00:30:07.119 INFO:tasks.rados.rados.0.smithi045.stdout:1708: read oid 33 snap 182 2024-04-26T00:30:07.119 INFO:tasks.rados.rados.0.smithi045.stdout:1708: expect (ObjNum 557 snap 173 seq_num 557) 2024-04-26T00:30:07.119 INFO:tasks.rados.rados.0.smithi045.stdout:1709: write (excl) oid 13 current snap is 196 2024-04-26T00:30:07.119 INFO:tasks.rados.rados.0.smithi045.stdout:1709: seq_num 631 ranges {605217=739418,2124726=480491} 2024-04-26T00:30:07.135 INFO:tasks.rados.rados.0.smithi045.stdout:1709: writing smithi04526579-13 from 605217 to 1344635 tid 1 2024-04-26T00:30:07.156 INFO:tasks.rados.rados.0.smithi045.stdout:1709: writing smithi04526579-13 from 2124726 to 2605217 tid 2 2024-04-26T00:30:07.157 INFO:tasks.rados.rados.0.smithi045.stdout:1710: rollback oid 29 current snap is 196 2024-04-26T00:30:07.157 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 29 to 191 2024-04-26T00:30:07.157 INFO:tasks.rados.rados.0.smithi045.stdout:1707: expect (ObjNum 605 snap 186 seq_num 605) 2024-04-26T00:30:07.282 INFO:tasks.rados.rados.0.smithi045.stdout:1708: expect (ObjNum 557 snap 173 seq_num 557) 2024-04-26T00:30:07.445 INFO:tasks.rados.rados.0.smithi045.stdout:1709: finishing write tid 1 to smithi04526579-13 2024-04-26T00:30:07.445 INFO:tasks.rados.rados.0.smithi045.stdout:1709: finishing write tid 2 to smithi04526579-13 2024-04-26T00:30:07.445 INFO:tasks.rados.rados.0.smithi045.stdout:1709: finishing write tid 3 to smithi04526579-13 2024-04-26T00:30:07.445 INFO:tasks.rados.rados.0.smithi045.stdout:1709: finishing write tid 4 to smithi04526579-13 2024-04-26T00:30:07.445 INFO:tasks.rados.rados.0.smithi045.stdout:1709: oid 13 updating version 0 to 875 2024-04-26T00:30:07.445 INFO:tasks.rados.rados.0.smithi045.stdout:1709: oid 13 updating version 875 to 876 2024-04-26T00:30:07.445 INFO:tasks.rados.rados.0.smithi045.stdout:1709: oid 13 updating version 876 to 877 2024-04-26T00:30:07.445 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 877 (ObjNum 631 snap 196 seq_num 631) dirty exists 2024-04-26T00:30:07.445 INFO:tasks.rados.rados.0.smithi045.stdout:1709: left oid 13 (ObjNum 631 snap 196 seq_num 631) 2024-04-26T00:30:07.445 INFO:tasks.rados.rados.0.smithi045.stdout:1710: finishing rollback tid 1 to smithi04526579-29 2024-04-26T00:30:07.446 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 489 (ObjNum 81 snap 0 seq_num 1) dirty dne 2024-04-26T00:30:07.446 INFO:tasks.rados.rados.0.smithi045.stdout:1707: done (3 left) 2024-04-26T00:30:07.446 INFO:tasks.rados.rados.0.smithi045.stdout:1708: done (2 left) 2024-04-26T00:30:07.446 INFO:tasks.rados.rados.0.smithi045.stdout:1709: done (1 left) 2024-04-26T00:30:07.446 INFO:tasks.rados.rados.0.smithi045.stdout:1710: done (0 left) 2024-04-26T00:30:07.446 INFO:tasks.rados.rados.0.smithi045.stdout:1711: read oid 12 snap -1 2024-04-26T00:30:07.446 INFO:tasks.rados.rados.0.smithi045.stdout:1711: expect (ObjNum 401 snap 122 seq_num 401) 2024-04-26T00:30:07.446 INFO:tasks.rados.rados.0.smithi045.stdout:1712: delete oid 15 current snap is 196 2024-04-26T00:30:07.449 INFO:tasks.rados.rados.0.smithi045.stdout:1712: done (1 left) 2024-04-26T00:30:07.449 INFO:tasks.rados.rados.0.smithi045.stdout:1713: copy_from oid 24 from oid 42 current snap is 196 2024-04-26T00:30:07.449 INFO:tasks.rados.rados.0.smithi045.stdout:1714: copy_from oid 27 from oid 28 current snap is 196 2024-04-26T00:30:07.450 INFO:tasks.rados.rados.0.smithi045.stdout:1715: copy_from oid 2 from oid 35 current snap is 196 2024-04-26T00:30:07.450 INFO:tasks.rados.rados.0.smithi045.stdout:1716: read oid 20 snap -1 2024-04-26T00:30:07.450 INFO:tasks.rados.rados.0.smithi045.stdout:1716: expect (ObjNum 394 snap 120 seq_num 394) 2024-04-26T00:30:07.450 INFO:tasks.rados.rados.0.smithi045.stdout:1717: rollback oid 26 current snap is 196 2024-04-26T00:30:07.450 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 26 to 179 2024-04-26T00:30:07.450 INFO:tasks.rados.rados.0.smithi045.stdout:1718: copy_from oid 21 from oid 17 current snap is 196 2024-04-26T00:30:07.451 INFO:tasks.rados.rados.0.smithi045.stdout:1719: snap_create 2024-04-26T00:30:07.457 INFO:tasks.rados.rados.0.smithi045.stdout:1711: expect (ObjNum 401 snap 122 seq_num 401) 2024-04-26T00:30:07.607 INFO:tasks.rados.rados.0.smithi045.stdout:1717: finishing rollback tid 0 to smithi04526579-26 2024-04-26T00:30:07.607 INFO:tasks.rados.rados.0.smithi045.stdout:1717: finishing rollback tid 1 to smithi04526579-26 2024-04-26T00:30:07.607 INFO:tasks.rados.rados.0.smithi045.stdout:1717: finishing rollback tid 2 to smithi04526579-26 2024-04-26T00:30:07.607 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 1081 (ObjNum 459 snap 143 seq_num 459) dirty exists 2024-04-26T00:30:07.607 INFO:tasks.rados.rados.0.smithi045.stdout:1718: finishing copy_from racing read to smithi04526579-21 2024-04-26T00:30:07.607 INFO:tasks.rados.rados.0.smithi045.stdout:1716: expect (ObjNum 394 snap 120 seq_num 394) 2024-04-26T00:30:07.719 INFO:tasks.rados.rados.0.smithi045.stdout:1718: finishing copy_from to smithi04526579-21 2024-04-26T00:30:07.719 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 1046 (ObjNum 595 snap 184 seq_num 595) dirty exists 2024-04-26T00:30:07.719 INFO:tasks.rados.rados.0.smithi045.stdout:1713: finishing copy_from racing read to smithi04526579-24 2024-04-26T00:30:07.719 INFO:tasks.rados.rados.0.smithi045.stdout:1713: finishing copy_from to smithi04526579-24 2024-04-26T00:30:07.720 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 987 (ObjNum 599 snap 184 seq_num 599) dirty exists 2024-04-26T00:30:07.720 INFO:tasks.rados.rados.0.smithi045.stdout:1714: finishing copy_from racing read to smithi04526579-27 2024-04-26T00:30:07.720 INFO:tasks.rados.rados.0.smithi045.stdout:1715: finishing copy_from racing read to smithi04526579-2 2024-04-26T00:30:07.720 INFO:tasks.rados.rados.0.smithi045.stdout:1714: finishing copy_from to smithi04526579-27 2024-04-26T00:30:07.720 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 1059 (ObjNum 582 snap 180 seq_num 582) dirty exists 2024-04-26T00:30:07.720 INFO:tasks.rados.rados.0.smithi045.stdout:1715: finishing copy_from to smithi04526579-2 2024-04-26T00:30:07.721 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 977 (ObjNum 553 snap 168 seq_num 553) dirty exists 2024-04-26T00:30:08.127 INFO:tasks.rados.rados.0.smithi045.stdout:1711: done (7 left) 2024-04-26T00:30:08.127 INFO:tasks.rados.rados.0.smithi045.stdout:1713: done (6 left) 2024-04-26T00:30:08.127 INFO:tasks.rados.rados.0.smithi045.stdout:1714: done (5 left) 2024-04-26T00:30:08.127 INFO:tasks.rados.rados.0.smithi045.stdout:1715: done (4 left) 2024-04-26T00:30:08.127 INFO:tasks.rados.rados.0.smithi045.stdout:1716: done (3 left) 2024-04-26T00:30:08.127 INFO:tasks.rados.rados.0.smithi045.stdout:1717: done (2 left) 2024-04-26T00:30:08.127 INFO:tasks.rados.rados.0.smithi045.stdout:1718: done (1 left) 2024-04-26T00:30:08.127 INFO:tasks.rados.rados.0.smithi045.stdout:1719: done (0 left) 2024-04-26T00:30:08.127 INFO:tasks.rados.rados.0.smithi045.stdout:1720: write (excl) oid 33 current snap is 197 2024-04-26T00:30:08.127 INFO:tasks.rados.rados.0.smithi045.stdout:1720: seq_num 633 ranges {638831=489916,1841690=677884} 2024-04-26T00:30:08.145 INFO:tasks.rados.rados.0.smithi045.stdout:1720: writing smithi04526579-33 from 638831 to 1128747 tid 1 2024-04-26T00:30:08.168 INFO:tasks.rados.rados.0.smithi045.stdout:1720: writing smithi04526579-33 from 1841690 to 2519574 tid 2 2024-04-26T00:30:08.168 INFO:tasks.rados.rados.0.smithi045.stdout:1721: snap_remove snap 189 2024-04-26T00:30:09.133 INFO:tasks.rados.rados.0.smithi045.stdout:1721: done (1 left) 2024-04-26T00:30:09.133 INFO:tasks.rados.rados.0.smithi045.stdout:1722: delete oid 7 current snap is 197 2024-04-26T00:30:09.134 INFO:tasks.rados.rados.0.smithi045.stdout:1720: finishing write tid 1 to smithi04526579-33 2024-04-26T00:30:09.134 INFO:tasks.rados.rados.0.smithi045.stdout:1720: finishing write tid 2 to smithi04526579-33 2024-04-26T00:30:09.134 INFO:tasks.rados.rados.0.smithi045.stdout:1720: finishing write tid 3 to smithi04526579-33 2024-04-26T00:30:09.134 INFO:tasks.rados.rados.0.smithi045.stdout:1720: finishing write tid 4 to smithi04526579-33 2024-04-26T00:30:09.134 INFO:tasks.rados.rados.0.smithi045.stdout:1720: oid 33 updating version 0 to 1380 2024-04-26T00:30:09.135 INFO:tasks.rados.rados.0.smithi045.stdout:1720: oid 33 version 1380 is already newer than 1379 2024-04-26T00:30:09.135 INFO:tasks.rados.rados.0.smithi045.stdout:1720: oid 33 version 1380 is already newer than 1378 2024-04-26T00:30:09.135 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 1380 (ObjNum 633 snap 197 seq_num 633) dirty exists 2024-04-26T00:30:09.135 INFO:tasks.rados.rados.0.smithi045.stdout:1720: left oid 33 (ObjNum 633 snap 197 seq_num 633) 2024-04-26T00:30:09.155 INFO:tasks.rados.rados.0.smithi045.stdout:1720: done (1 left) 2024-04-26T00:30:09.155 INFO:tasks.rados.rados.0.smithi045.stdout:1722: done (0 left) 2024-04-26T00:30:09.155 INFO:tasks.rados.rados.0.smithi045.stdout:1723: snap_create 2024-04-26T00:30:10.104 INFO:tasks.rados.rados.0.smithi045.stdout:1723: done (0 left) 2024-04-26T00:30:10.104 INFO:tasks.rados.rados.0.smithi045.stdout:1724: write (excl) oid 45 current snap is 198 2024-04-26T00:30:10.104 INFO:tasks.rados.rados.0.smithi045.stdout:1724: seq_num 635 ranges {672445=640414,1842301=681553} 2024-04-26T00:30:10.125 INFO:tasks.rados.rados.0.smithi045.stdout:1724: writing smithi04526579-45 from 672445 to 1312859 tid 1 2024-04-26T00:30:10.145 INFO:tasks.rados.rados.0.smithi045.stdout:1724: writing smithi04526579-45 from 1842301 to 2523854 tid 2 2024-04-26T00:30:10.145 INFO:tasks.rados.rados.0.smithi045.stdout:1725: snap_create 2024-04-26T00:30:10.146 INFO:tasks.rados.rados.0.smithi045.stdout:1724: finishing write tid 1 to smithi04526579-45 2024-04-26T00:30:10.155 INFO:tasks.rados.rados.0.smithi045.stdout:1724: finishing write tid 2 to smithi04526579-45 2024-04-26T00:30:10.155 INFO:tasks.rados.rados.0.smithi045.stdout:1724: finishing write tid 3 to smithi04526579-45 2024-04-26T00:30:10.156 INFO:tasks.rados.rados.0.smithi045.stdout:1724: finishing write tid 4 to smithi04526579-45 2024-04-26T00:30:10.156 INFO:tasks.rados.rados.0.smithi045.stdout:1724: oid 45 updating version 0 to 983 2024-04-26T00:30:10.156 INFO:tasks.rados.rados.0.smithi045.stdout:1724: oid 45 version 983 is already newer than 981 2024-04-26T00:30:10.156 INFO:tasks.rados.rados.0.smithi045.stdout:1724: oid 45 version 983 is already newer than 982 2024-04-26T00:30:10.156 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 983 (ObjNum 635 snap 198 seq_num 635) dirty exists 2024-04-26T00:30:10.156 INFO:tasks.rados.rados.0.smithi045.stdout:1724: left oid 45 (ObjNum 635 snap 198 seq_num 635) 2024-04-26T00:30:11.025 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-04-26T00:30:11.026 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-04-26T00:30:11.026 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-26T00:30:11.026 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-26T00:30:11.029 INFO:tasks.ceph.mon.a:Started 2024-04-26T00:30:11.029 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-04-26T00:30:11.029 INFO:tasks.ceph.mon.b:Sent signal 19 2024-04-26T00:30:11.029 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2024-04-26T00:30:23.414 INFO:tasks.rados.rados.0.smithi045.stdout:1724: done (1 left) 2024-04-26T00:30:23.414 INFO:tasks.rados.rados.0.smithi045.stdout:1725: done (0 left) 2024-04-26T00:30:23.414 INFO:tasks.rados.rados.0.smithi045.stdout:1726: read oid 16 snap -1 2024-04-26T00:30:23.414 INFO:tasks.rados.rados.0.smithi045.stdout:1726: expect deleted 2024-04-26T00:30:23.414 INFO:tasks.rados.rados.0.smithi045.stdout:1727: rollback oid 37 current snap is 199 2024-04-26T00:30:23.414 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 37 to 175 2024-04-26T00:30:23.414 INFO:tasks.rados.rados.0.smithi045.stdout:1728: snap_remove snap 190 2024-04-26T00:30:24.421 INFO:tasks.rados.rados.0.smithi045.stdout:1728: done (2 left) 2024-04-26T00:30:24.422 INFO:tasks.rados.rados.0.smithi045.stdout:1729: write oid 2 current snap is 199 2024-04-26T00:30:24.422 INFO:tasks.rados.rados.0.smithi045.stdout:1729: seq_num 636 ranges {689252=715663,1900783=425211} 2024-04-26T00:30:24.450 INFO:tasks.rados.rados.0.smithi045.stdout:1729: writing smithi04526579-2 from 689252 to 1404915 tid 1 2024-04-26T00:30:24.462 INFO:tasks.rados.rados.0.smithi045.stdout:1729: writing smithi04526579-2 from 1900783 to 2325994 tid 2 2024-04-26T00:30:24.462 INFO:tasks.rados.rados.0.smithi045.stdout:1730: snap_remove snap 191 2024-04-26T00:30:25.426 INFO:tasks.rados.rados.0.smithi045.stdout:1730: done (3 left) 2024-04-26T00:30:25.426 INFO:tasks.rados.rados.0.smithi045.stdout:1731: rollback oid 19 current snap is 199 2024-04-26T00:30:25.426 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 19 to 175 2024-04-26T00:30:25.427 INFO:tasks.rados.rados.0.smithi045.stdout:1727: finishing rollback tid 1 to smithi04526579-37 2024-04-26T00:30:25.427 INFO:tasks.rados.rados.0.smithi045.stdout:1727: finishing rollback tid 2 to smithi04526579-37 2024-04-26T00:30:25.427 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 1388 (ObjNum 556 snap 173 seq_num 556) dirty exists 2024-04-26T00:30:25.427 INFO:tasks.rados.rados.0.smithi045.stdout:1729: finishing write tid 1 to smithi04526579-2 2024-04-26T00:30:25.427 INFO:tasks.rados.rados.0.smithi045.stdout:1729: finishing write tid 2 to smithi04526579-2 2024-04-26T00:30:25.427 INFO:tasks.rados.rados.0.smithi045.stdout:1729: finishing write tid 3 to smithi04526579-2 2024-04-26T00:30:25.427 INFO:tasks.rados.rados.0.smithi045.stdout:1729: finishing write tid 4 to smithi04526579-2 2024-04-26T00:30:25.427 INFO:tasks.rados.rados.0.smithi045.stdout:1729: oid 2 updating version 0 to 980 2024-04-26T00:30:25.427 INFO:tasks.rados.rados.0.smithi045.stdout:1729: oid 2 version 980 is already newer than 979 2024-04-26T00:30:25.427 INFO:tasks.rados.rados.0.smithi045.stdout:1729: oid 2 version 980 is already newer than 978 2024-04-26T00:30:25.427 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 980 (ObjNum 636 snap 199 seq_num 636) dirty exists 2024-04-26T00:30:25.427 INFO:tasks.rados.rados.0.smithi045.stdout:1729: left oid 2 (ObjNum 636 snap 199 seq_num 636) 2024-04-26T00:30:25.427 INFO:tasks.rados.rados.0.smithi045.stdout:1726: done (3 left) 2024-04-26T00:30:25.427 INFO:tasks.rados.rados.0.smithi045.stdout:1727: done (2 left) 2024-04-26T00:30:25.428 INFO:tasks.rados.rados.0.smithi045.stdout:1729: done (1 left) 2024-04-26T00:30:25.428 INFO:tasks.rados.rados.0.smithi045.stdout:1732: rollback oid 39 current snap is 199 2024-04-26T00:30:25.428 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 39 to 193 2024-04-26T00:30:25.428 INFO:tasks.rados.rados.0.smithi045.stdout:1733: write (excl) oid 22 current snap is 199 2024-04-26T00:30:25.428 INFO:tasks.rados.rados.0.smithi045.stdout:1733: seq_num 637 ranges {706059=790912,1959265=685222} 2024-04-26T00:30:25.451 INFO:tasks.rados.rados.0.smithi045.stdout:1733: writing smithi04526579-22 from 706059 to 1496971 tid 1 2024-04-26T00:30:25.474 INFO:tasks.rados.rados.0.smithi045.stdout:1733: writing smithi04526579-22 from 1959265 to 2644487 tid 2 2024-04-26T00:30:25.474 INFO:tasks.rados.rados.0.smithi045.stdout:1734: read oid 24 snap -1 2024-04-26T00:30:25.474 INFO:tasks.rados.rados.0.smithi045.stdout:1734: expect (ObjNum 599 snap 184 seq_num 599) 2024-04-26T00:30:25.474 INFO:tasks.rados.rados.0.smithi045.stdout:1735: snap_create 2024-04-26T00:30:25.475 INFO:tasks.rados.rados.0.smithi045.stdout:1732: finishing rollback tid 1 to smithi04526579-39 2024-04-26T00:30:25.475 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 1049 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-04-26T00:30:25.475 INFO:tasks.rados.rados.0.smithi045.stdout:1731: finishing rollback tid 0 to smithi04526579-19 2024-04-26T00:30:25.475 INFO:tasks.rados.rados.0.smithi045.stdout:1731: finishing rollback tid 1 to smithi04526579-19 2024-04-26T00:30:25.475 INFO:tasks.rados.rados.0.smithi045.stdout:1731: finishing rollback tid 2 to smithi04526579-19 2024-04-26T00:30:25.475 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 1062 (ObjNum 534 snap 165 seq_num 534) dirty exists 2024-04-26T00:30:25.475 INFO:tasks.rados.rados.0.smithi045.stdout:1733: finishing write tid 1 to smithi04526579-22 2024-04-26T00:30:25.483 INFO:tasks.rados.rados.0.smithi045.stdout:1734: expect (ObjNum 599 snap 184 seq_num 599) 2024-04-26T00:30:25.637 INFO:tasks.rados.rados.0.smithi045.stdout:1733: finishing write tid 2 to smithi04526579-22 2024-04-26T00:30:25.637 INFO:tasks.rados.rados.0.smithi045.stdout:1733: finishing write tid 3 to smithi04526579-22 2024-04-26T00:30:25.637 INFO:tasks.rados.rados.0.smithi045.stdout:1733: finishing write tid 4 to smithi04526579-22 2024-04-26T00:30:25.637 INFO:tasks.rados.rados.0.smithi045.stdout:1733: oid 22 updating version 0 to 781 2024-04-26T00:30:25.638 INFO:tasks.rados.rados.0.smithi045.stdout:1733: oid 22 version 781 is already newer than 779 2024-04-26T00:30:25.638 INFO:tasks.rados.rados.0.smithi045.stdout:1733: oid 22 version 781 is already newer than 780 2024-04-26T00:30:25.638 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 781 (ObjNum 637 snap 199 seq_num 637) dirty exists 2024-04-26T00:30:25.638 INFO:tasks.rados.rados.0.smithi045.stdout:1733: left oid 22 (ObjNum 637 snap 199 seq_num 637) 2024-04-26T00:30:26.447 INFO:tasks.rados.rados.0.smithi045.stdout:1731: done (4 left) 2024-04-26T00:30:26.447 INFO:tasks.rados.rados.0.smithi045.stdout:1732: done (3 left) 2024-04-26T00:30:26.447 INFO:tasks.rados.rados.0.smithi045.stdout:1733: done (2 left) 2024-04-26T00:30:26.447 INFO:tasks.rados.rados.0.smithi045.stdout:1734: done (1 left) 2024-04-26T00:30:26.447 INFO:tasks.rados.rados.0.smithi045.stdout:1735: done (0 left) 2024-04-26T00:30:26.447 INFO:tasks.rados.rados.0.smithi045.stdout:1736: copy_from oid 34 from oid 4 current snap is 200 2024-04-26T00:30:26.447 INFO:tasks.rados.rados.0.smithi045.stdout:1737: read oid 47 snap 184 2024-04-26T00:30:26.447 INFO:tasks.rados.rados.0.smithi045.stdout:1737: expect deleted 2024-04-26T00:30:26.447 INFO:tasks.rados.rados.0.smithi045.stdout:1738: read oid 46 snap 193 2024-04-26T00:30:26.447 INFO:tasks.rados.rados.0.smithi045.stdout:1738: expect (ObjNum 329 snap 96 seq_num 329) 2024-04-26T00:30:26.447 INFO:tasks.rados.rados.0.smithi045.stdout:1739: snap_create 2024-04-26T00:30:26.462 INFO:tasks.rados.rados.0.smithi045.stdout:1738: expect (ObjNum 329 snap 96 seq_num 329) 2024-04-26T00:30:26.638 INFO:tasks.rados.rados.0.smithi045.stdout:1736: finishing copy_from racing read to smithi04526579-34 2024-04-26T00:30:26.638 INFO:tasks.rados.rados.0.smithi045.stdout:1736: finishing copy_from to smithi04526579-34 2024-04-26T00:30:26.638 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 700 (ObjNum 474 snap 147 seq_num 474) dirty exists 2024-04-26T00:30:27.453 INFO:tasks.rados.rados.0.smithi045.stdout:1736: done (3 left) 2024-04-26T00:30:27.453 INFO:tasks.rados.rados.0.smithi045.stdout:1737: done (2 left) 2024-04-26T00:30:27.453 INFO:tasks.rados.rados.0.smithi045.stdout:1738: done (1 left) 2024-04-26T00:30:27.453 INFO:tasks.rados.rados.0.smithi045.stdout:1739: done (0 left) 2024-04-26T00:30:27.453 INFO:tasks.rados.rados.0.smithi045.stdout:1740: read oid 31 snap -1 2024-04-26T00:30:27.453 INFO:tasks.rados.rados.0.smithi045.stdout:1740: expect deleted 2024-04-26T00:30:27.453 INFO:tasks.rados.rados.0.smithi045.stdout:1741: write (excl) oid 5 current snap is 201 2024-04-26T00:30:27.453 INFO:tasks.rados.rados.0.smithi045.stdout:1741: seq_num 638 ranges {722866=466161,1617747=428880,2587634=135232} 2024-04-26T00:30:27.460 INFO:tasks.rados.rados.0.smithi045.stdout:1741: writing smithi04526579-5 from 722866 to 1189027 tid 1 2024-04-26T00:30:27.466 INFO:tasks.rados.rados.0.smithi045.stdout:1741: writing smithi04526579-5 from 1617747 to 2046627 tid 2 2024-04-26T00:30:27.474 INFO:tasks.rados.rados.0.smithi045.stdout:1741: writing smithi04526579-5 from 2587634 to 2722866 tid 3 2024-04-26T00:30:27.474 INFO:tasks.rados.rados.0.smithi045.stdout:1742: snap_remove snap 197 2024-04-26T00:30:28.459 INFO:tasks.rados.rados.0.smithi045.stdout:1742: done (2 left) 2024-04-26T00:30:28.459 INFO:tasks.rados.rados.0.smithi045.stdout:1743: write oid 43 current snap is 201 2024-04-26T00:30:28.459 INFO:tasks.rados.rados.0.smithi045.stdout:1743: seq_num 639 ranges {739673=657763,2076229=663444} 2024-04-26T00:30:28.481 INFO:tasks.rados.rados.0.smithi045.stdout:1743: writing smithi04526579-43 from 739673 to 1397436 tid 1 2024-04-26T00:30:28.506 INFO:tasks.rados.rados.0.smithi045.stdout:1743: writing smithi04526579-43 from 2076229 to 2739673 tid 2 2024-04-26T00:30:28.506 INFO:tasks.rados.rados.0.smithi045.stdout:1744: write oid 11 current snap is 201 2024-04-26T00:30:28.506 INFO:tasks.rados.rados.0.smithi045.stdout:1744: seq_num 640 ranges {756480=733012,2134711=432549} 2024-04-26T00:30:28.525 INFO:tasks.rados.rados.0.smithi045.stdout:1744: writing smithi04526579-11 from 756480 to 1489492 tid 1 2024-04-26T00:30:28.537 INFO:tasks.rados.rados.0.smithi045.stdout:1744: writing smithi04526579-11 from 2134711 to 2567260 tid 2 2024-04-26T00:30:28.537 INFO:tasks.rados.rados.0.smithi045.stdout:1741: finishing write tid 1 to smithi04526579-5 2024-04-26T00:30:28.537 INFO:tasks.rados.rados.0.smithi045.stdout:1741: finishing write tid 2 to smithi04526579-5 2024-04-26T00:30:28.537 INFO:tasks.rados.rados.0.smithi045.stdout:1745: read oid 7 snap -1 2024-04-26T00:30:28.537 INFO:tasks.rados.rados.0.smithi045.stdout:1745: expect deleted 2024-04-26T00:30:28.537 INFO:tasks.rados.rados.0.smithi045.stdout:1740: done (4 left) 2024-04-26T00:30:28.538 INFO:tasks.rados.rados.0.smithi045.stdout:1746: write (excl) oid 26 current snap is 201 2024-04-26T00:30:28.538 INFO:tasks.rados.rados.0.smithi045.stdout:1746: seq_num 641 ranges {773287=408261,1793193=576207} 2024-04-26T00:30:28.548 INFO:tasks.rados.rados.0.smithi045.stdout:1746: writing smithi04526579-26 from 773287 to 1181548 tid 1 2024-04-26T00:30:28.551 INFO:tasks.rados.rados.0.smithi045.stdout:1746: writing smithi04526579-26 from 1793193 to 2369400 tid 2 2024-04-26T00:30:28.552 INFO:tasks.rados.rados.0.smithi045.stdout:1747: write (excl) oid 28 current snap is 201 2024-04-26T00:30:28.552 INFO:tasks.rados.rados.0.smithi045.stdout:1741: finishing write tid 3 to smithi04526579-5 2024-04-26T00:30:28.552 INFO:tasks.rados.rados.0.smithi045.stdout:1741: finishing write tid 4 to smithi04526579-5 2024-04-26T00:30:28.552 INFO:tasks.rados.rados.0.smithi045.stdout:1747: seq_num 642 ranges {790094=483510,1851675=436218} 2024-04-26T00:30:28.559 INFO:tasks.rados.rados.0.smithi045.stdout:1747: writing smithi04526579-28 from 790094 to 1273604 tid 1 2024-04-26T00:30:28.566 INFO:tasks.rados.rados.0.smithi045.stdout:1747: writing smithi04526579-28 from 1851675 to 2287893 tid 2 2024-04-26T00:30:28.566 INFO:tasks.rados.rados.0.smithi045.stdout:1748: read oid 50 snap -1 2024-04-26T00:30:28.567 INFO:tasks.rados.rados.0.smithi045.stdout:1748: expect (ObjNum 470 snap 147 seq_num 470) 2024-04-26T00:30:28.567 INFO:tasks.rados.rados.0.smithi045.stdout:1749: rollback oid 39 current snap is 201 2024-04-26T00:30:28.567 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 39 to 160 2024-04-26T00:30:28.567 INFO:tasks.rados.rados.0.smithi045.stdout:1741: finishing write tid 5 to smithi04526579-5 2024-04-26T00:30:28.567 INFO:tasks.rados.rados.0.smithi045.stdout:1741: oid 5 updating version 0 to 1399 2024-04-26T00:30:28.567 INFO:tasks.rados.rados.0.smithi045.stdout:1741: oid 5 updating version 1399 to 1401 2024-04-26T00:30:28.567 INFO:tasks.rados.rados.0.smithi045.stdout:1741: oid 5 version 1401 is already newer than 1398 2024-04-26T00:30:28.567 INFO:tasks.rados.rados.0.smithi045.stdout:1741: oid 5 version 1401 is already newer than 1400 2024-04-26T00:30:28.567 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 1401 (ObjNum 638 snap 201 seq_num 638) dirty exists 2024-04-26T00:30:28.567 INFO:tasks.rados.rados.0.smithi045.stdout:1741: left oid 5 (ObjNum 638 snap 201 seq_num 638) 2024-04-26T00:30:28.567 INFO:tasks.rados.rados.0.smithi045.stdout:1743: finishing write tid 1 to smithi04526579-43 2024-04-26T00:30:28.567 INFO:tasks.rados.rados.0.smithi045.stdout:1741: done (7 left) 2024-04-26T00:30:28.567 INFO:tasks.rados.rados.0.smithi045.stdout:1750: read oid 42 snap -1 2024-04-26T00:30:28.567 INFO:tasks.rados.rados.0.smithi045.stdout:1750: expect (ObjNum 599 snap 184 seq_num 599) 2024-04-26T00:30:28.567 INFO:tasks.rados.rados.0.smithi045.stdout:1751: copy_from oid 2 from oid 18 current snap is 201 2024-04-26T00:30:28.568 INFO:tasks.rados.rados.0.smithi045.stdout:1743: finishing write tid 2 to smithi04526579-43 2024-04-26T00:30:28.568 INFO:tasks.rados.rados.0.smithi045.stdout:1743: finishing write tid 3 to smithi04526579-43 2024-04-26T00:30:28.568 INFO:tasks.rados.rados.0.smithi045.stdout:1743: finishing write tid 4 to smithi04526579-43 2024-04-26T00:30:28.568 INFO:tasks.rados.rados.0.smithi045.stdout:1743: oid 43 updating version 0 to 1051 2024-04-26T00:30:28.568 INFO:tasks.rados.rados.0.smithi045.stdout:1743: oid 43 updating version 1051 to 1052 2024-04-26T00:30:28.568 INFO:tasks.rados.rados.0.smithi045.stdout:1743: oid 43 version 1052 is already newer than 1050 2024-04-26T00:30:28.568 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 1052 (ObjNum 639 snap 201 seq_num 639) dirty exists 2024-04-26T00:30:28.568 INFO:tasks.rados.rados.0.smithi045.stdout:1743: left oid 43 (ObjNum 639 snap 201 seq_num 639) 2024-04-26T00:30:28.568 INFO:tasks.rados.rados.0.smithi045.stdout:1743: done (8 left) 2024-04-26T00:30:28.568 INFO:tasks.rados.rados.0.smithi045.stdout:1752: delete oid 48 current snap is 201 2024-04-26T00:30:28.568 INFO:tasks.rados.rados.0.smithi045.stdout:1744: finishing write tid 1 to smithi04526579-11 2024-04-26T00:30:28.568 INFO:tasks.rados.rados.0.smithi045.stdout:1744: finishing write tid 2 to smithi04526579-11 2024-04-26T00:30:28.568 INFO:tasks.rados.rados.0.smithi045.stdout:1744: finishing write tid 3 to smithi04526579-11 2024-04-26T00:30:28.568 INFO:tasks.rados.rados.0.smithi045.stdout:1744: finishing write tid 4 to smithi04526579-11 2024-04-26T00:30:28.569 INFO:tasks.rados.rados.0.smithi045.stdout:1744: oid 11 updating version 0 to 984 2024-04-26T00:30:28.569 INFO:tasks.rados.rados.0.smithi045.stdout:1744: oid 11 updating version 984 to 985 2024-04-26T00:30:28.569 INFO:tasks.rados.rados.0.smithi045.stdout:1744: oid 11 updating version 985 to 986 2024-04-26T00:30:28.569 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 986 (ObjNum 640 snap 201 seq_num 640) dirty exists 2024-04-26T00:30:28.569 INFO:tasks.rados.rados.0.smithi045.stdout:1744: left oid 11 (ObjNum 640 snap 201 seq_num 640) 2024-04-26T00:30:28.569 INFO:tasks.rados.rados.0.smithi045.stdout:1746: finishing write tid 1 to smithi04526579-26 2024-04-26T00:30:28.569 INFO:tasks.rados.rados.0.smithi045.stdout:1746: finishing write tid 2 to smithi04526579-26 2024-04-26T00:30:28.569 INFO:tasks.rados.rados.0.smithi045.stdout:1746: finishing write tid 3 to smithi04526579-26 2024-04-26T00:30:28.569 INFO:tasks.rados.rados.0.smithi045.stdout:1746: finishing write tid 4 to smithi04526579-26 2024-04-26T00:30:28.569 INFO:tasks.rados.rados.0.smithi045.stdout:1746: oid 26 updating version 0 to 1092 2024-04-26T00:30:28.569 INFO:tasks.rados.rados.0.smithi045.stdout:1746: oid 26 updating version 1092 to 1093 2024-04-26T00:30:28.569 INFO:tasks.rados.rados.0.smithi045.stdout:1746: oid 26 updating version 1093 to 1094 2024-04-26T00:30:28.569 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 1094 (ObjNum 641 snap 201 seq_num 641) dirty exists 2024-04-26T00:30:28.569 INFO:tasks.rados.rados.0.smithi045.stdout:1746: left oid 26 (ObjNum 641 snap 201 seq_num 641) 2024-04-26T00:30:28.570 INFO:tasks.rados.rados.0.smithi045.stdout:1747: finishing write tid 1 to smithi04526579-28 2024-04-26T00:30:28.571 INFO:tasks.rados.rados.0.smithi045.stdout:1744: done (8 left) 2024-04-26T00:30:28.571 INFO:tasks.rados.rados.0.smithi045.stdout:1745: done (7 left) 2024-04-26T00:30:28.571 INFO:tasks.rados.rados.0.smithi045.stdout:1746: done (6 left) 2024-04-26T00:30:28.571 INFO:tasks.rados.rados.0.smithi045.stdout:1752: done (5 left) 2024-04-26T00:30:28.572 INFO:tasks.rados.rados.0.smithi045.stdout:1753: snap_remove snap 171 2024-04-26T00:30:29.474 INFO:tasks.rados.rados.0.smithi045.stdout:1753: done (5 left) 2024-04-26T00:30:29.474 INFO:tasks.rados.rados.0.smithi045.stdout:1754: read oid 10 snap -1 2024-04-26T00:30:29.474 INFO:tasks.rados.rados.0.smithi045.stdout:1754: expect (ObjNum 573 snap 176 seq_num 573) 2024-04-26T00:30:29.474 INFO:tasks.rados.rados.0.smithi045.stdout:1755: snap_create 2024-04-26T00:30:29.474 INFO:tasks.rados.rados.0.smithi045.stdout:1749: finishing rollback tid 1 to smithi04526579-39 2024-04-26T00:30:29.474 INFO:tasks.rados.rados.0.smithi045.stdout:1749: finishing rollback tid 2 to smithi04526579-39 2024-04-26T00:30:29.474 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 1061 (ObjNum 508 snap 157 seq_num 508) dirty exists 2024-04-26T00:30:29.474 INFO:tasks.rados.rados.0.smithi045.stdout:1750: expect (ObjNum 599 snap 184 seq_num 599) 2024-04-26T00:30:29.631 INFO:tasks.rados.rados.0.smithi045.stdout:1748: expect (ObjNum 470 snap 147 seq_num 470) 2024-04-26T00:30:29.858 INFO:tasks.rados.rados.0.smithi045.stdout:1747: finishing write tid 2 to smithi04526579-28 2024-04-26T00:30:29.858 INFO:tasks.rados.rados.0.smithi045.stdout:1747: finishing write tid 3 to smithi04526579-28 2024-04-26T00:30:29.858 INFO:tasks.rados.rados.0.smithi045.stdout:1747: finishing write tid 4 to smithi04526579-28 2024-04-26T00:30:29.858 INFO:tasks.rados.rados.0.smithi045.stdout:1747: oid 28 updating version 0 to 1097 2024-04-26T00:30:29.858 INFO:tasks.rados.rados.0.smithi045.stdout:1747: oid 28 updating version 1097 to 1098 2024-04-26T00:30:29.858 INFO:tasks.rados.rados.0.smithi045.stdout:1747: oid 28 version 1098 is already newer than 1096 2024-04-26T00:30:29.858 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 1098 (ObjNum 642 snap 201 seq_num 642) dirty exists 2024-04-26T00:30:29.858 INFO:tasks.rados.rados.0.smithi045.stdout:1747: left oid 28 (ObjNum 642 snap 201 seq_num 642) 2024-04-26T00:30:29.858 INFO:tasks.rados.rados.0.smithi045.stdout:1751: finishing copy_from racing read to smithi04526579-2 2024-04-26T00:30:29.859 INFO:tasks.rados.rados.0.smithi045.stdout:1751: finishing copy_from to smithi04526579-2 2024-04-26T00:30:29.859 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 869 (ObjNum 551 snap 168 seq_num 551) dirty exists 2024-04-26T00:30:29.859 INFO:tasks.rados.rados.0.smithi045.stdout:1754: expect (ObjNum 573 snap 176 seq_num 573) 2024-04-26T00:30:30.483 INFO:tasks.rados.rados.0.smithi045.stdout:1747: done (6 left) 2024-04-26T00:30:30.483 INFO:tasks.rados.rados.0.smithi045.stdout:1748: done (5 left) 2024-04-26T00:30:30.483 INFO:tasks.rados.rados.0.smithi045.stdout:1749: done (4 left) 2024-04-26T00:30:30.483 INFO:tasks.rados.rados.0.smithi045.stdout:1750: done (3 left) 2024-04-26T00:30:30.483 INFO:tasks.rados.rados.0.smithi045.stdout:1751: done (2 left) 2024-04-26T00:30:30.483 INFO:tasks.rados.rados.0.smithi045.stdout:1754: done (1 left) 2024-04-26T00:30:30.483 INFO:tasks.rados.rados.0.smithi045.stdout:1755: done (0 left) 2024-04-26T00:30:30.483 INFO:tasks.rados.rados.0.smithi045.stdout:1756: snap_remove snap 182 2024-04-26T00:30:31.031 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-04-26T00:30:31.031 INFO:tasks.ceph.mon.b:Sent signal 18 2024-04-26T00:30:31.031 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:30:31.032 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:30:31.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.181+0000 7f3cc9bd4640 10 monclient: get_monmap_and_config 2024-04-26T00:30:31.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.181+0000 7f3cc9bd4640 10 monclient: build_initial_monmap 2024-04-26T00:30:31.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.181+0000 7f3cc9bd4640 10 monclient: monmap: 2024-04-26T00:30:31.174 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:30:31.174 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:31.174 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:30:31.186197+0000 2024-04-26T00:30:31.174 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:30:31.186197+0000 2024-04-26T00:30:31.174 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:30:31.174 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:31.175 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:30:31.175 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:30:31.175 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:30:31.175 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:30:31.175 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:30:31.175 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:31.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.181+0000 7f3cc9bd4640 1 Processor -- start 2024-04-26T00:30:31.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.181+0000 7f3cc9bd4640 1 -- start start 2024-04-26T00:30:31.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.181+0000 7f3cc9bd4640 10 monclient: init 2024-04-26T00:30:31.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.185+0000 7f3cc9bd4640 10 monclient: _reopen_session rank -1 2024-04-26T00:30:31.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.185+0000 7f3cc9bd4640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-04-26T00:30:31.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.185+0000 7f3cc9bd4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3cc40a0f50 addr v1:172.21.15.37:6789/0 2024-04-26T00:30:31.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.185+0000 7f3cc9bd4640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f3cc40a1330 0x7f3cc409f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:31.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.185+0000 7f3cc9bd4640 10 monclient(hunting): picked mon.noname-a con 0x7f3cc40a1330 addr v2:172.21.15.37:3300/0 2024-04-26T00:30:31.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.185+0000 7f3cc9bd4640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3cc406c9f0 0x7f3cc413ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:31.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.185+0000 7f3cc9bd4640 10 monclient(hunting): picked mon.noname-c con 0x7f3cc406c9f0 addr v2:172.21.15.45:3300/0 2024-04-26T00:30:31.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.185+0000 7f3cc9bd4640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cc40a36f0 con 0x7f3cc40a0f50 2024-04-26T00:30:31.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.185+0000 7f3cc9bd4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:31.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.185+0000 7f3cc9bd4640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f3cc409fd90 con 0x7f3cc40a1330 2024-04-26T00:30:31.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.185+0000 7f3cc9bd4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:31.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.185+0000 7f3cc9bd4640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f3cc409df60 con 0x7f3cc406c9f0 2024-04-26T00:30:31.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.185+0000 7f3cc9bd4640 10 monclient(hunting): _renew_subs 2024-04-26T00:30:31.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.185+0000 7f3cc9bd4640 10 monclient(hunting): authenticate will time out at 1612.492492s 2024-04-26T00:30:31.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.185+0000 7f3cbaffd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f3cc40a1330 0x7f3cc409f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:31.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.185+0000 7f3cc37fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f3cc40a0f50 0x7f3cc406c2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:34754/0 (socket says 172.21.15.37:34754) 2024-04-26T00:30:31.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.185+0000 7f3cc37fe640 1 -- 172.21.15.37:0/1533948235 learned_addr learned my addr 172.21.15.37:0/1533948235 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:30:31.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.185+0000 7f3cbaffd640 10 monclient(hunting): get_auth_request con 0x7f3cc40a1330 auth_method 0 2024-04-26T00:30:31.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.185+0000 7f3cbaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:31.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.185+0000 7f3cbaffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:31.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.185+0000 7f3cbaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:31.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.185+0000 7f3cbaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:31.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.185+0000 7f3cbaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:31.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.185+0000 7f3cbaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:31.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.185+0000 7f3cc3fff640 1 --2- 172.21.15.37:0/1533948235 >> v2:172.21.15.45:3300/0 conn(0x7f3cc406c9f0 0x7f3cc413ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:31.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.185+0000 7f3cc3fff640 10 monclient(hunting): get_auth_request con 0x7f3cc406c9f0 auth_method 0 2024-04-26T00:30:31.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.185+0000 7f3cc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:31.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.185+0000 7f3cc3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:31.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.185+0000 7f3cc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:31.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.185+0000 7f3cbaffd640 10 monclient(hunting): handle_auth_done global_id 124098 payload 274 2024-04-26T00:30:31.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.185+0000 7f3cbaffd640 1 -- 172.21.15.37:0/1533948235 >> v2:172.21.15.45:3300/0 conn(0x7f3cc406c9f0 msgr2=0x7f3cc413ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:30:31.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.185+0000 7f3cbaffd640 1 --2- 172.21.15.37:0/1533948235 >> v2:172.21.15.45:3300/0 conn(0x7f3cc406c9f0 0x7f3cc413ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:31.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.185+0000 7f3cbaffd640 1 -- 172.21.15.37:0/1533948235 >> v1:172.21.15.37:6789/0 conn(0x7f3cc40a0f50 legacy=0x7f3cc406c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:30:31.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.185+0000 7f3cbaffd640 10 monclient: _finish_hunting 0 2024-04-26T00:30:31.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.185+0000 7f3cbaffd640 1 monclient: found mon.noname-a 2024-04-26T00:30:31.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.185+0000 7f3cbaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:30:31.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.185+0000 7f3cbaffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:30:31.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.185+0000 7f3cbaffd640 1 -- 172.21.15.37:0/1533948235 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cc40a1740 con 0x7f3cc40a1330 2024-04-26T00:30:31.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cbaffd640 1 -- 172.21.15.37:0/1533948235 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3cc40a1330 msgr2=0x7f3cc409f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:31.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cbaffd640 1 --2- 172.21.15.37:0/1533948235 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3cc40a1330 0x7f3cc409f850 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3cc409e930 tx=0x7f3cac01c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc555130393869f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:31.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc2ffd640 1 -- 172.21.15.37:0/1533948235 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cac0b0070 con 0x7f3cc40a1330 2024-04-26T00:30:31.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:31.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:30:31.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc2ffd640 10 monclient: dump: 2024-04-26T00:30:31.182 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:31.182 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:31.182 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.182 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.182 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:31.183 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:31.183 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:31.183 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:31.183 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:31.183 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:31.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc2ffd640 10 monclient: _finish_auth 0 2024-04-26T00:30:31.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc2ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:30:31.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:30:31.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc9bd4640 5 monclient: authenticate success, global_id 124098 2024-04-26T00:30:31.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc9bd4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:30:31.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc2ffd640 1 -- 172.21.15.37:0/1533948235 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cac0ab600 con 0x7f3cc40a1330 2024-04-26T00:30:31.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc2ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:30:31.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc9bd4640 10 monclient: get_monmap_and_config success 2024-04-26T00:30:31.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc9bd4640 4 set_mon_vals no callback set 2024-04-26T00:30:31.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc2ffd640 1 -- 172.21.15.37:0/1533948235 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cac0ab9d0 con 0x7f3cc40a1330 2024-04-26T00:30:31.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:31.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:30:31.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc2ffd640 10 monclient: dump: 2024-04-26T00:30:31.184 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:31.184 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:31.184 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.184 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.184 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:31.184 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:31.184 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:31.184 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:31.184 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:31.184 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:31.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc9bd4640 10 monclient: shutdown 2024-04-26T00:30:31.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc9bd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:30:31.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc9bd4640 1 -- 172.21.15.37:0/1533948235 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3cc40a1330 msgr2=0x7f3cc409f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:31.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc9bd4640 1 --2- 172.21.15.37:0/1533948235 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3cc40a1330 0x7f3cc409f850 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3cc409e930 tx=0x7f3cac01c2d0 comp rx=0 tx=0).stop 2024-04-26T00:30:31.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc9bd4640 1 -- 172.21.15.37:0/1533948235 shutdown_connections 2024-04-26T00:30:31.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc9bd4640 1 --2- 172.21.15.37:0/1533948235 >> v2:172.21.15.45:3300/0 conn(0x7f3cc406c9f0 0x7f3cc413ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:31.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc9bd4640 1 --2- 172.21.15.37:0/1533948235 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3cc40a1330 0x7f3cc409f850 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:31.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc9bd4640 1 -- 172.21.15.37:0/1533948235 >> 172.21.15.37:0/1533948235 conn(0x7f3cc40a4170 msgr2=0x7f3cc40a4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:30:31.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc9bd4640 1 -- 172.21.15.37:0/1533948235 shutdown_connections 2024-04-26T00:30:31.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc9bd4640 1 -- 172.21.15.37:0/1533948235 wait complete. 2024-04-26T00:30:31.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc9bd4640 10 monclient: build_initial_monmap 2024-04-26T00:30:31.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc9bd4640 10 monclient: monmap: 2024-04-26T00:30:31.189 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:30:31.189 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:30:31.189 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:30:31.189 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:30:31.189 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:30:31.190 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:30:31.190 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:30:31.190 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:30:31.190 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:30:31.190 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:31.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc9bd4640 1 Processor -- start 2024-04-26T00:30:31.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc9bd4640 1 -- start start 2024-04-26T00:30:31.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc9bd4640 10 monclient: init 2024-04-26T00:30:31.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc9bd4640 10 monclient: _reopen_session rank -1 2024-04-26T00:30:31.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc9bd4640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:30:31.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc9bd4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3cc406c9f0 0x7f3cc41cf540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:31.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc9bd4640 10 monclient(hunting): picked mon.noname-b con 0x7f3cc406c9f0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:30:31.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc9bd4640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3cc40a0f50 0x7f3cc4138220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:31.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc9bd4640 10 monclient(hunting): picked mon.noname-c con 0x7f3cc40a0f50 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:30:31.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc9bd4640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3cc40a1330 0x7f3cc41cc040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:31.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc9bd4640 10 monclient(hunting): picked mon.noname-a con 0x7f3cc40a1330 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:30:31.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc9bd4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:31.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc9bd4640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc4141d50 con 0x7f3cc40a1330 2024-04-26T00:30:31.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc9bd4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:31.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc9bd4640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc4141bd0 con 0x7f3cc406c9f0 2024-04-26T00:30:31.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc9bd4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:31.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc9bd4640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f3cc4141ed0 con 0x7f3cc40a0f50 2024-04-26T00:30:31.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc9bd4640 10 monclient(hunting): _renew_subs 2024-04-26T00:30:31.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.189+0000 7f3cc9bd4640 10 monclient(hunting): authenticate will time out at 1612.499966s 2024-04-26T00:30:31.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cbaffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3cc40a0f50 0x7f3cc4138220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:31.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cc3fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3cc40a1330 0x7f3cc41cc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:31.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cbaffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3cc40a0f50 0x7f3cc4138220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:56390/0 (socket says 172.21.15.37:56390) 2024-04-26T00:30:31.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cbaffd640 1 -- 172.21.15.37:0/1660520900 learned_addr learned my addr 172.21.15.37:0/1660520900 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:30:31.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cbaffd640 10 monclient(hunting): get_auth_request con 0x7f3cc40a0f50 auth_method 0 2024-04-26T00:30:31.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cbaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:31.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cbaffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:31.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cbaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:31.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cc3fff640 10 monclient(hunting): get_auth_request con 0x7f3cc40a1330 auth_method 0 2024-04-26T00:30:31.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:31.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cc3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:31.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:31.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cbaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:31.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cbaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:31.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cbaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:31.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cc3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:31.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cc3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:31.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cc3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:31.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cc37fe640 1 --2- 172.21.15.37:0/1660520900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3cc406c9f0 0x7f3cc41cf540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:31.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cc37fe640 10 monclient(hunting): get_auth_request con 0x7f3cc406c9f0 auth_method 0 2024-04-26T00:30:31.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cc37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:31.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cc37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:31.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:31.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cbaffd640 10 monclient(hunting): handle_auth_done global_id 84455 payload 824 2024-04-26T00:30:31.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cbaffd640 1 -- 172.21.15.37:0/1660520900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3cc406c9f0 msgr2=0x7f3cc41cf540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:31.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cbaffd640 1 --2- 172.21.15.37:0/1660520900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3cc406c9f0 0x7f3cc41cf540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:31.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cbaffd640 1 -- 172.21.15.37:0/1660520900 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3cc40a1330 msgr2=0x7f3cc41cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:31.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cbaffd640 1 --2- 172.21.15.37:0/1660520900 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3cc40a1330 0x7f3cc41cc040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:31.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cbaffd640 10 monclient: _finish_hunting 0 2024-04-26T00:30:31.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cbaffd640 1 monclient: found mon.noname-c 2024-04-26T00:30:31.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cbaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:30:31.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cbaffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:30:31.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cbaffd640 1 -- 172.21.15.37:0/1660520900 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cc406c550 con 0x7f3cc40a0f50 2024-04-26T00:30:31.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cbaffd640 1 -- 172.21.15.37:0/1660520900 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3cc40a0f50 msgr2=0x7f3cc4138220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:31.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cbaffd640 1 --2- 172.21.15.37:0/1660520900 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3cc40a0f50 0x7f3cc4138220 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3cac01a980 tx=0x7f3cac019070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:31.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cc0ff9640 1 -- 172.21.15.37:0/1660520900 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cac0b0040 con 0x7f3cc40a0f50 2024-04-26T00:30:31.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cc0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:31.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cc0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:30:31.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cc0ff9640 10 monclient: dump: 2024-04-26T00:30:31.194 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:31.194 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:31.194 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.195 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.195 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:31.195 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:31.195 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:31.195 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:31.195 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:31.195 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:31.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cc0ff9640 10 monclient: _finish_auth 0 2024-04-26T00:30:31.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cc0ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:30:31.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cc0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:30:31.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cc0ff9640 1 -- 172.21.15.37:0/1660520900 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cac0b9070 con 0x7f3cc40a0f50 2024-04-26T00:30:31.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cc9bd4640 5 monclient: authenticate success, global_id 84455 2024-04-26T00:30:31.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cc9bd4640 10 monclient: _renew_subs 2024-04-26T00:30:31.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cc9bd4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:30:31.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cc9bd4640 1 -- 172.21.15.37:0/1660520900 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cc409d5c0 con 0x7f3cc40a0f50 2024-04-26T00:30:31.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cc0ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:30:31.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cc0ff9640 1 -- 172.21.15.37:0/1660520900 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cac0aa7d0 con 0x7f3cc40a0f50 2024-04-26T00:30:31.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cc0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:31.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cc0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:30:31.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cc0ff9640 10 monclient: dump: 2024-04-26T00:30:31.196 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:31.196 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:31.196 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.196 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.196 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:31.196 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:31.196 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:31.196 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:31.196 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:31.196 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:31.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cc9bd4640 10 monclient: _renew_subs 2024-04-26T00:30:31.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cc9bd4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:30:31.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cc9bd4640 1 -- 172.21.15.37:0/1660520900 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cc4071a00 con 0x7f3cc40a0f50 2024-04-26T00:30:31.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cc27fc640 4 set_mon_vals no callback set 2024-04-26T00:30:31.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cc9bd4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:30:31.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cc9bd4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:30:31.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cc9bd4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:30:31.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cc0ff9640 1 -- 172.21.15.37:0/1660520900 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f3cac0aa970 con 0x7f3cc40a0f50 2024-04-26T00:30:31.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.193+0000 7f3cc9bd4640 1 -- 172.21.15.37:0/1660520900 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cc4122620 con 0x7f3cc40a0f50 2024-04-26T00:30:31.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.197+0000 7f3cc0ff9640 1 --2- 172.21.15.37:0/1660520900 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3ca4035ed0 0x7f3ca4038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:31.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.197+0000 7f3cc0ff9640 1 -- 172.21.15.37:0/1660520900 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(401..401 src has 370..401) ==== 3931+0+0 (secure 0 0 0) 0x7f3cac0e2720 con 0x7f3cc40a0f50 2024-04-26T00:30:31.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.197+0000 7f3cc37fe640 1 --2- 172.21.15.37:0/1660520900 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3ca4035ed0 0x7f3ca4038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:31.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.197+0000 7f3cc37fe640 10 monclient: get_auth_request con 0x7f3ca4035ed0 auth_method 0 2024-04-26T00:30:31.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.197+0000 7f3cc37fe640 1 --2- 172.21.15.37:0/1660520900 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3ca4035ed0 0x7f3ca4038390 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4072280 tx=0x7f3cb4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:31.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.201+0000 7f3cc0ff9640 1 -- 172.21.15.37:0/1660520900 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f3cac016010 con 0x7f3cc40a0f50 2024-04-26T00:30:31.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.201+0000 7f3cc0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:30:31.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.201+0000 7f3cc0ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:30:31.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.425+0000 7f3cc9bd4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:30:31.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.425+0000 7f3cc9bd4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:30:31.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.425+0000 7f3cc9bd4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:30:31.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.425+0000 7f3cc9bd4640 1 -- 172.21.15.37:0/1660520900 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3cc4066f40 con 0x7f3cc40a0f50 2024-04-26T00:30:31.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.425+0000 7f3cc0ff9640 1 -- 172.21.15.37:0/1660520900 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f3cac0e1470 con 0x7f3cc40a0f50 2024-04-26T00:30:31.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.425+0000 7f3cc0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:30:31.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.425+0000 7f3cc0ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:30:31.419 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:30:31.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.429+0000 7f3cc9bd4640 1 -- 172.21.15.37:0/1660520900 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3ca4035ed0 msgr2=0x7f3ca4038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:31.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.429+0000 7f3cc9bd4640 1 --2- 172.21.15.37:0/1660520900 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3ca4035ed0 0x7f3ca4038390 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4072280 tx=0x7f3cb4009040 comp rx=0 tx=0).stop 2024-04-26T00:30:31.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.429+0000 7f3cc9bd4640 10 monclient: shutdown 2024-04-26T00:30:31.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.429+0000 7f3cc9bd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:30:31.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.429+0000 7f3cc9bd4640 1 -- 172.21.15.37:0/1660520900 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3cc40a0f50 msgr2=0x7f3cc4138220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:31.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.429+0000 7f3cc9bd4640 1 --2- 172.21.15.37:0/1660520900 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3cc40a0f50 0x7f3cc4138220 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3cac01a980 tx=0x7f3cac019070 comp rx=0 tx=0).stop 2024-04-26T00:30:31.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.429+0000 7f3cc9bd4640 1 -- 172.21.15.37:0/1660520900 shutdown_connections 2024-04-26T00:30:31.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.429+0000 7f3cc9bd4640 1 --2- 172.21.15.37:0/1660520900 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3ca4035ed0 0x7f3ca4038390 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:31.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.429+0000 7f3cc9bd4640 1 --2- 172.21.15.37:0/1660520900 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3cc40a1330 0x7f3cc41cc040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:31.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.429+0000 7f3cc9bd4640 1 --2- 172.21.15.37:0/1660520900 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3cc40a0f50 0x7f3cc4138220 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:31.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.429+0000 7f3cc9bd4640 1 --2- 172.21.15.37:0/1660520900 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3cc406c9f0 0x7f3cc41cf540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:31.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.429+0000 7f3cc9bd4640 1 -- 172.21.15.37:0/1660520900 >> 172.21.15.37:0/1660520900 conn(0x7f3cc40a4170 msgr2=0x7f3cc40a5fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:30:31.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.429+0000 7f3cc9bd4640 1 -- 172.21.15.37:0/1660520900 shutdown_connections 2024-04-26T00:30:31.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.429+0000 7f3cc9bd4640 1 -- 172.21.15.37:0/1660520900 wait complete. 2024-04-26T00:30:31.437 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:30:31.437 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:30:31.437 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:30:31.487 INFO:tasks.rados.rados.0.smithi045.stdout:1756: done (0 left) 2024-04-26T00:30:31.487 INFO:tasks.rados.rados.0.smithi045.stdout:1757: copy_from oid 32 from oid 43 current snap is 202 2024-04-26T00:30:31.487 INFO:tasks.rados.rados.0.smithi045.stdout:1758: write (excl) oid 31 current snap is 202 2024-04-26T00:30:31.487 INFO:tasks.rados.rados.0.smithi045.stdout:1758: seq_num 644 ranges {423708=634008,1852286=439887,2752172=71536} 2024-04-26T00:30:31.498 INFO:tasks.rados.rados.0.smithi045.stdout:1758: writing smithi04526579-31 from 423708 to 1057716 tid 1 2024-04-26T00:30:31.509 INFO:tasks.rados.rados.0.smithi045.stdout:1758: writing smithi04526579-31 from 1852286 to 2292173 tid 2 2024-04-26T00:30:31.514 INFO:tasks.rados.rados.0.smithi045.stdout:1758: writing smithi04526579-31 from 2752172 to 2823708 tid 3 2024-04-26T00:30:31.514 INFO:tasks.rados.rados.0.smithi045.stdout:1759: write (excl) oid 1 current snap is 202 2024-04-26T00:30:31.514 INFO:tasks.rados.rados.0.smithi045.stdout:1759: seq_num 645 ranges {440515=709257,1910768=583545} 2024-04-26T00:30:31.523 INFO:tasks.rados.rados.0.smithi045.stdout:1759: writing smithi04526579-1 from 440515 to 1149772 tid 1 2024-04-26T00:30:31.535 INFO:tasks.rados.rados.0.smithi045.stdout:1759: writing smithi04526579-1 from 1910768 to 2494313 tid 2 2024-04-26T00:30:31.535 INFO:tasks.rados.rados.0.smithi045.stdout:1760: rollback oid 18 current snap is 202 2024-04-26T00:30:31.535 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 18 to 195 2024-04-26T00:30:31.535 INFO:tasks.rados.rados.0.smithi045.stdout:1757: finishing copy_from racing read to smithi04526579-32 2024-04-26T00:30:31.536 INFO:tasks.rados.rados.0.smithi045.stdout:1758: finishing write tid 1 to smithi04526579-31 2024-04-26T00:30:31.536 INFO:tasks.rados.rados.0.smithi045.stdout:1758: finishing write tid 2 to smithi04526579-31 2024-04-26T00:30:31.536 INFO:tasks.rados.rados.0.smithi045.stdout:1758: finishing write tid 3 to smithi04526579-31 2024-04-26T00:30:31.536 INFO:tasks.rados.rados.0.smithi045.stdout:1757: finishing copy_from to smithi04526579-32 2024-04-26T00:30:31.536 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 1052 (ObjNum 639 snap 201 seq_num 639) dirty exists 2024-04-26T00:30:31.536 INFO:tasks.rados.rados.0.smithi045.stdout:1758: finishing write tid 4 to smithi04526579-31 2024-04-26T00:30:31.536 INFO:tasks.rados.rados.0.smithi045.stdout:1758: finishing write tid 5 to smithi04526579-31 2024-04-26T00:30:31.536 INFO:tasks.rados.rados.0.smithi045.stdout:1758: oid 31 updating version 0 to 983 2024-04-26T00:30:31.536 INFO:tasks.rados.rados.0.smithi045.stdout:1758: oid 31 version 983 is already newer than 981 2024-04-26T00:30:31.536 INFO:tasks.rados.rados.0.smithi045.stdout:1758: oid 31 version 983 is already newer than 982 2024-04-26T00:30:31.536 INFO:tasks.rados.rados.0.smithi045.stdout:1758: oid 31 updating version 983 to 984 2024-04-26T00:30:31.536 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 984 (ObjNum 644 snap 202 seq_num 644) dirty exists 2024-04-26T00:30:31.536 INFO:tasks.rados.rados.0.smithi045.stdout:1758: left oid 31 (ObjNum 644 snap 202 seq_num 644) 2024-04-26T00:30:31.536 INFO:tasks.rados.rados.0.smithi045.stdout:1757: done (3 left) 2024-04-26T00:30:31.536 INFO:tasks.rados.rados.0.smithi045.stdout:1758: done (2 left) 2024-04-26T00:30:31.537 INFO:tasks.rados.rados.0.smithi045.stdout:1761: rollback oid 10 current snap is 202 2024-04-26T00:30:31.537 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 10 to 192 2024-04-26T00:30:31.537 INFO:tasks.rados.rados.0.smithi045.stdout:1762: snap_remove snap 164 2024-04-26T00:30:31.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe229325640 10 monclient: get_monmap_and_config 2024-04-26T00:30:31.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe229325640 10 monclient: build_initial_monmap 2024-04-26T00:30:31.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe229325640 10 monclient: monmap: 2024-04-26T00:30:31.573 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:30:31.573 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:31.573 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:30:31.585421+0000 2024-04-26T00:30:31.573 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:30:31.585421+0000 2024-04-26T00:30:31.573 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:30:31.573 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:31.574 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:30:31.574 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:30:31.574 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:30:31.574 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:30:31.574 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:30:31.574 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:31.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe229325640 1 Processor -- start 2024-04-26T00:30:31.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe229325640 1 -- start start 2024-04-26T00:30:31.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe229325640 10 monclient: init 2024-04-26T00:30:31.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe229325640 10 monclient: _reopen_session rank -1 2024-04-26T00:30:31.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe229325640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-04-26T00:30:31.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe229325640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe224136f20 0x7fe22409bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:31.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe229325640 10 monclient(hunting): picked mon.noname-b con 0x7fe224136f20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:30:31.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe229325640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe224137300 0x7fe224137780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:31.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe229325640 10 monclient(hunting): picked mon.noname-c con 0x7fe224137300 addr v2:172.21.15.45:3300/0 2024-04-26T00:30:31.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe229325640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe224133c70 addr v1:172.21.15.45:6789/0 2024-04-26T00:30:31.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe229325640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2240a50f0 con 0x7fe224133c70 2024-04-26T00:30:31.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe229325640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:31.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe229325640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fe22409b330 con 0x7fe224137300 2024-04-26T00:30:31.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe229325640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:31.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe229325640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe22409b610 con 0x7fe224136f20 2024-04-26T00:30:31.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe229325640 10 monclient(hunting): _renew_subs 2024-04-26T00:30:31.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe229325640 10 monclient(hunting): authenticate will time out at 1612.890913s 2024-04-26T00:30:31.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe222ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe224136f20 0x7fe22409bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:31.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe222ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe224136f20 0x7fe22409bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:56394/0 (socket says 172.21.15.37:56394) 2024-04-26T00:30:31.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe222ffd640 1 -- 172.21.15.37:0/1376850175 learned_addr learned my addr 172.21.15.37:0/1376850175 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:30:31.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe222ffd640 10 monclient(hunting): get_auth_request con 0x7fe224136f20 auth_method 0 2024-04-26T00:30:31.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe222ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:31.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe222ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:31.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe222ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:31.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe222ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:31.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe222ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:31.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe222ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:31.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe2227fc640 1 --2- 172.21.15.37:0/1376850175 >> v2:172.21.15.45:3300/0 conn(0x7fe224137300 0x7fe224137780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:31.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe2227fc640 10 monclient(hunting): get_auth_request con 0x7fe224137300 auth_method 0 2024-04-26T00:30:31.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe2227fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:31.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe2227fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:31.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe2227fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:31.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe222ffd640 10 monclient(hunting): handle_auth_done global_id 84458 payload 274 2024-04-26T00:30:31.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe222ffd640 1 -- 172.21.15.37:0/1376850175 >> v2:172.21.15.45:3300/0 conn(0x7fe224137300 msgr2=0x7fe224137780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:30:31.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe222ffd640 1 --2- 172.21.15.37:0/1376850175 >> v2:172.21.15.45:3300/0 conn(0x7fe224137300 0x7fe224137780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:31.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe222ffd640 1 -- 172.21.15.37:0/1376850175 >> v1:172.21.15.45:6789/0 conn(0x7fe224133c70 legacy=0x7fe2241340e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:30:31.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe222ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:30:31.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe222ffd640 1 monclient: found mon.noname-b 2024-04-26T00:30:31.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe222ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:30:31.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe222ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:30:31.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.581+0000 7fe222ffd640 1 -- 172.21.15.37:0/1376850175 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe22409b8f0 con 0x7fe224136f20 2024-04-26T00:30:31.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.585+0000 7fe222ffd640 1 -- 172.21.15.37:0/1376850175 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe224136f20 msgr2=0x7fe22409bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:31.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.585+0000 7fe222ffd640 1 --2- 172.21.15.37:0/1376850175 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe224136f20 0x7fe22409bff0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe214009e50 tx=0x7fe21401c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62ca30e605798210 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:31.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.585+0000 7fe221ffb640 1 -- 172.21.15.37:0/1376850175 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2140a56d0 con 0x7fe224136f20 2024-04-26T00:30:31.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.585+0000 7fe221ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:31.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.585+0000 7fe221ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:30:31.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.585+0000 7fe221ffb640 10 monclient: dump: 2024-04-26T00:30:31.587 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:31.587 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:31.587 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.587 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.587 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:31.587 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:31.587 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:31.587 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:31.587 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:31.588 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:31.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.585+0000 7fe221ffb640 10 monclient: _finish_auth 0 2024-04-26T00:30:31.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.585+0000 7fe221ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:30:31.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.585+0000 7fe221ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:30:31.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.585+0000 7fe221ffb640 1 -- 172.21.15.37:0/1376850175 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2140a5870 con 0x7fe224136f20 2024-04-26T00:30:31.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.585+0000 7fe221ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:30:31.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.585+0000 7fe221ffb640 1 -- 172.21.15.37:0/1376850175 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2140a5c40 con 0x7fe224136f20 2024-04-26T00:30:31.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.585+0000 7fe229325640 5 monclient: authenticate success, global_id 84458 2024-04-26T00:30:31.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.585+0000 7fe229325640 10 monclient: get_monmap_and_config success 2024-04-26T00:30:31.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.585+0000 7fe229325640 4 set_mon_vals no callback set 2024-04-26T00:30:31.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.585+0000 7fe221ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:31.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.585+0000 7fe221ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:30:31.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.585+0000 7fe221ffb640 10 monclient: dump: 2024-04-26T00:30:31.589 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:31.589 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:31.589 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.589 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.589 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:31.589 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:31.589 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:31.589 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:31.589 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:31.589 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:31.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe229325640 10 monclient: shutdown 2024-04-26T00:30:31.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe229325640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:30:31.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe229325640 1 -- 172.21.15.37:0/1376850175 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe224136f20 msgr2=0x7fe22409bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:31.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe229325640 1 --2- 172.21.15.37:0/1376850175 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe224136f20 0x7fe22409bff0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe214009e50 tx=0x7fe21401c940 comp rx=0 tx=0).stop 2024-04-26T00:30:31.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe229325640 1 -- 172.21.15.37:0/1376850175 shutdown_connections 2024-04-26T00:30:31.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe229325640 1 --2- 172.21.15.37:0/1376850175 >> v2:172.21.15.45:3300/0 conn(0x7fe224137300 0x7fe224137780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:31.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe229325640 1 --2- 172.21.15.37:0/1376850175 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe224136f20 0x7fe22409bff0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:31.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe229325640 1 -- 172.21.15.37:0/1376850175 >> 172.21.15.37:0/1376850175 conn(0x7fe2240a61b0 msgr2=0x7fe2240a65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:30:31.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe229325640 1 -- 172.21.15.37:0/1376850175 shutdown_connections 2024-04-26T00:30:31.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe229325640 1 -- 172.21.15.37:0/1376850175 wait complete. 2024-04-26T00:30:31.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe229325640 10 monclient: build_initial_monmap 2024-04-26T00:30:31.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe229325640 10 monclient: monmap: 2024-04-26T00:30:31.590 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:30:31.590 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:30:31.590 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:30:31.590 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:30:31.590 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:30:31.591 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:30:31.591 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:30:31.591 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:30:31.591 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:30:31.591 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:31.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe229325640 1 Processor -- start 2024-04-26T00:30:31.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe229325640 1 -- start start 2024-04-26T00:30:31.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe229325640 10 monclient: init 2024-04-26T00:30:31.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe229325640 10 monclient: _reopen_session rank -1 2024-04-26T00:30:31.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe229325640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:30:31.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe229325640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe224133c70 0x7fe2241cf4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:31.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe229325640 10 monclient(hunting): picked mon.noname-b con 0x7fe224133c70 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:30:31.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe229325640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe224136f20 0x7fe22413e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:31.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe229325640 10 monclient(hunting): picked mon.noname-c con 0x7fe224136f20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:30:31.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe229325640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe224137300 0x7fe22413e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:31.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe229325640 10 monclient(hunting): picked mon.noname-a con 0x7fe224137300 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:30:31.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe229325640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:31.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe229325640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe224141e10 con 0x7fe224137300 2024-04-26T00:30:31.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe229325640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:31.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe229325640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe224141c90 con 0x7fe224133c70 2024-04-26T00:30:31.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe229325640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:31.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe229325640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe224141f90 con 0x7fe224136f20 2024-04-26T00:30:31.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe229325640 10 monclient(hunting): _renew_subs 2024-04-26T00:30:31.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe229325640 10 monclient(hunting): authenticate will time out at 1612.898102s 2024-04-26T00:30:31.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe2237fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe224137300 0x7fe22413e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:31.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe2237fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe224137300 0x7fe22413e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:43208/0 (socket says 172.21.15.37:43208) 2024-04-26T00:30:31.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe2237fe640 1 -- 172.21.15.37:0/3353362186 learned_addr learned my addr 172.21.15.37:0/3353362186 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:30:31.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe2237fe640 10 monclient(hunting): get_auth_request con 0x7fe224137300 auth_method 0 2024-04-26T00:30:31.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe2237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:31.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe2237fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:31.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe2237fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:31.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe2227fc640 1 --2- 172.21.15.37:0/3353362186 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe224136f20 0x7fe22413e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:31.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe2237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:31.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe2237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:31.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe2237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:31.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe222ffd640 1 --2- 172.21.15.37:0/3353362186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe224133c70 0x7fe2241cf4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:31.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe2227fc640 10 monclient(hunting): get_auth_request con 0x7fe224136f20 auth_method 0 2024-04-26T00:30:31.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe2227fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:31.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe2227fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:31.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe2227fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:31.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe222ffd640 10 monclient(hunting): get_auth_request con 0x7fe224133c70 auth_method 0 2024-04-26T00:30:31.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe222ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:31.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe222ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:31.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe222ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:31.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe2227fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:31.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe2227fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:31.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe2227fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:31.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe2237fe640 10 monclient(hunting): handle_auth_done global_id 124104 payload 824 2024-04-26T00:30:31.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe2237fe640 1 -- 172.21.15.37:0/3353362186 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe224136f20 msgr2=0x7fe22413e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:31.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe2237fe640 1 --2- 172.21.15.37:0/3353362186 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe224136f20 0x7fe22413e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:31.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe2237fe640 1 -- 172.21.15.37:0/3353362186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe224133c70 msgr2=0x7fe2241cf4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:31.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe2237fe640 1 --2- 172.21.15.37:0/3353362186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe224133c70 0x7fe2241cf4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:31.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe2237fe640 10 monclient: _finish_hunting 0 2024-04-26T00:30:31.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe2237fe640 1 monclient: found mon.noname-a 2024-04-26T00:30:31.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe2237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:30:31.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe2237fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:30:31.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe2237fe640 1 -- 172.21.15.37:0/3353362186 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe22413cc10 con 0x7fe224137300 2024-04-26T00:30:31.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.589+0000 7fe222ffd640 1 --2- 172.21.15.37:0/3353362186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe224133c70 0x7fe2241cf4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:30:31.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe2237fe640 1 -- 172.21.15.37:0/3353362186 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe224137300 msgr2=0x7fe22413e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:31.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe2237fe640 1 --2- 172.21.15.37:0/3353362186 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe224137300 0x7fe22413e8b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe21c00c3c0 tx=0x7fe21c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:31.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe20b7fe640 1 -- 172.21.15.37:0/3353362186 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe21c0119a0 con 0x7fe224137300 2024-04-26T00:30:31.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe20b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:31.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe20b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:30:31.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe20b7fe640 10 monclient: dump: 2024-04-26T00:30:31.595 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:31.595 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:31.595 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.595 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.595 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:31.595 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:31.595 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:31.595 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:31.595 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:31.595 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:31.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe20b7fe640 10 monclient: _finish_auth 0 2024-04-26T00:30:31.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe20b7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:30:31.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe20b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:30:31.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe229325640 5 monclient: authenticate success, global_id 124104 2024-04-26T00:30:31.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe20b7fe640 1 -- 172.21.15.37:0/3353362186 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe21c017070 con 0x7fe224137300 2024-04-26T00:30:31.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe229325640 10 monclient: _renew_subs 2024-04-26T00:30:31.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe229325640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:30:31.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe229325640 1 -- 172.21.15.37:0/3353362186 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe22413b200 con 0x7fe224137300 2024-04-26T00:30:31.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe20b7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:30:31.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe20b7fe640 1 -- 172.21.15.37:0/3353362186 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe21c011d80 con 0x7fe224137300 2024-04-26T00:30:31.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe20b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:31.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe20b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:30:31.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe20b7fe640 10 monclient: dump: 2024-04-26T00:30:31.596 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:31.596 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:31.597 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.597 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.597 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:31.597 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:31.597 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:31.597 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:31.597 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:31.597 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:31.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe229325640 10 monclient: _renew_subs 2024-04-26T00:30:31.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe229325640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:30:31.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe229325640 1 -- 172.21.15.37:0/3353362186 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe224135ab0 con 0x7fe224137300 2024-04-26T00:30:31.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe2217fa640 4 set_mon_vals no callback set 2024-04-26T00:30:31.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe229325640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:30:31.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe229325640 1 --2- 172.21.15.37:0/3353362186 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe224138d20 0x7fe224139100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:31.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe229325640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:31.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe229325640 1 -- 172.21.15.37:0/3353362186 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe22413c400 con 0x7fe224138d20 2024-04-26T00:30:31.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe222ffd640 1 --2- 172.21.15.37:0/3353362186 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe224138d20 0x7fe224139100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:31.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe222ffd640 10 monclient: get_auth_request con 0x7fe224138d20 auth_method 0 2024-04-26T00:30:31.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe222ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:31.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe222ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:31.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe222ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:31.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe20b7fe640 1 -- 172.21.15.37:0/3353362186 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe21c01c460 con 0x7fe224137300 2024-04-26T00:30:31.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe222ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:31.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe222ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:31.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe222ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:31.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe222ffd640 10 monclient(hunting): handle_auth_done global_id 124107 payload 824 2024-04-26T00:30:31.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe20b7fe640 1 --2- 172.21.15.37:0/3353362186 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe210035f30 0x7fe2100383f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:31.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe222ffd640 1 -- 172.21.15.37:0/3353362186 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe22404f510 con 0x7fe224138d20 2024-04-26T00:30:31.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe20b7fe640 1 -- 172.21.15.37:0/3353362186 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(402..402 src has 370..402) ==== 3975+0+0 (secure 0 0 0) 0x7fe21c048810 con 0x7fe224137300 2024-04-26T00:30:31.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe2227fc640 1 --2- 172.21.15.37:0/3353362186 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe210035f30 0x7fe2100383f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:31.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.593+0000 7fe2227fc640 10 monclient: get_auth_request con 0x7fe210035f30 auth_method 0 2024-04-26T00:30:31.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.597+0000 7fe222ffd640 1 -- 172.21.15.37:0/3353362186 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe224138d20 msgr2=0x7fe224139100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:31.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.597+0000 7fe222ffd640 1 --2- 172.21.15.37:0/3353362186 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe224138d20 0x7fe224139100 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe22409bbd0 tx=0x7fe2140196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:31.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.597+0000 7fe20b7fe640 1 -- 172.21.15.37:0/3353362186 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2140b6830 con 0x7fe224138d20 2024-04-26T00:30:31.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.597+0000 7fe20b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:31.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.597+0000 7fe20b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:30:31.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.597+0000 7fe20b7fe640 10 monclient: dump: 2024-04-26T00:30:31.601 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:31.602 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:31.602 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.602 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.602 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:31.602 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:31.602 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:31.602 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:31.602 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:31.602 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:31.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.597+0000 7fe20b7fe640 1 -- 172.21.15.37:0/3353362186 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe2140b6830 con 0x7fe224138d20 2024-04-26T00:30:31.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.597+0000 7fe20b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:30:31.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.597+0000 7fe20b7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:30:31.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.597+0000 7fe20b7fe640 1 -- 172.21.15.37:0/3353362186 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe224138d20 msgr2=0x7fe224139100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:31.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.597+0000 7fe20b7fe640 1 --2- 172.21.15.37:0/3353362186 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe224138d20 0x7fe224139100 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe22409bbd0 tx=0x7fe2140196e0 comp rx=0 tx=0).stop 2024-04-26T00:30:31.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.597+0000 7fe20b7fe640 1 -- 172.21.15.37:0/3353362186 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe224138d20 msgr2=0x7fe224139100 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:30:31.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.597+0000 7fe20b7fe640 1 --2- 172.21.15.37:0/3353362186 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe224138d20 0x7fe224139100 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe22409bbd0 tx=0x7fe2140196e0 comp rx=0 tx=0).stop 2024-04-26T00:30:31.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.597+0000 7fe2227fc640 1 --2- 172.21.15.37:0/3353362186 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe210035f30 0x7fe2100383f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe22413ce20 tx=0x7fe218008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:31.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.601+0000 7fe229325640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:30:31.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.601+0000 7fe229325640 1 --2- 172.21.15.37:0/3353362186 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe224139640 0x7fe22409d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:31.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.601+0000 7fe229325640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:31.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.601+0000 7fe229325640 1 -- 172.21.15.37:0/3353362186 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe22413a920 con 0x7fe224139640 2024-04-26T00:30:31.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.601+0000 7fe222ffd640 1 --2- 172.21.15.37:0/3353362186 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe224139640 0x7fe22409d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:31.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.601+0000 7fe222ffd640 10 monclient: get_auth_request con 0x7fe224139640 auth_method 0 2024-04-26T00:30:31.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.601+0000 7fe222ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:31.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.601+0000 7fe222ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:31.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.601+0000 7fe222ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:31.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.601+0000 7fe222ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:31.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.601+0000 7fe222ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:31.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.601+0000 7fe222ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:31.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.601+0000 7fe222ffd640 10 monclient(hunting): handle_auth_done global_id 124110 payload 824 2024-04-26T00:30:31.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.601+0000 7fe222ffd640 1 -- 172.21.15.37:0/3353362186 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe224075f10 con 0x7fe224139640 2024-04-26T00:30:31.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.605+0000 7fe222ffd640 1 -- 172.21.15.37:0/3353362186 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe224139640 msgr2=0x7fe22409d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:31.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.605+0000 7fe222ffd640 1 --2- 172.21.15.37:0/3353362186 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe224139640 0x7fe22409d2f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe214009e50 tx=0x7fe2140a7000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:31.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.605+0000 7fe20b7fe640 1 -- 172.21.15.37:0/3353362186 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2140c3c60 con 0x7fe224139640 2024-04-26T00:30:31.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.605+0000 7fe20b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:31.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.605+0000 7fe20b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:30:31.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.605+0000 7fe20b7fe640 10 monclient: dump: 2024-04-26T00:30:31.604 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:31.604 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:31.604 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.604 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.604 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:31.605 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:31.605 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:31.605 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:31.605 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:31.605 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:31.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.605+0000 7fe20b7fe640 1 -- 172.21.15.37:0/3353362186 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7fe2140c7070 con 0x7fe224139640 2024-04-26T00:30:31.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.605+0000 7fe20b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:30:31.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.605+0000 7fe20b7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:30:31.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.605+0000 7fe20b7fe640 1 -- 172.21.15.37:0/3353362186 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe224139640 msgr2=0x7fe22409d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:31.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.605+0000 7fe20b7fe640 1 --2- 172.21.15.37:0/3353362186 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe224139640 0x7fe22409d2f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe214009e50 tx=0x7fe2140a7000 comp rx=0 tx=0).stop 2024-04-26T00:30:31.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.605+0000 7fe20b7fe640 1 -- 172.21.15.37:0/3353362186 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe224139640 msgr2=0x7fe22409d2f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:30:31.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.605+0000 7fe20b7fe640 1 --2- 172.21.15.37:0/3353362186 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe224139640 0x7fe22409d2f0 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe214009e50 tx=0x7fe2140a7000 comp rx=0 tx=0).stop 2024-04-26T00:30:31.606 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:30:31.606 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:30:31.606 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:30:31.606 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:30:31.606 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 80, 2024-04-26T00:30:31.606 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 20483, 2024-04-26T00:30:31.606 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:30:31.606 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:30:31.606 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:30:31.606 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:30:31.606 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:30:31.607 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:30:31.607 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:30:31.607 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:30:31.607 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:30:31.607 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:30:31.607 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:30:31.607 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:30:31.607 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:30:31.607 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:30:31.607 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:30:31.607 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:30:31.607 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:30:31.607 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:30:31.607 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:30:31.607 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:30:31.608 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:30:31.608 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:30:31.608 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:30:31.608 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:30:31.608 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:30:31.608 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:30:31.608 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:30:31.608 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:30:31.608 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:30:31.608 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:30:31.608 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:30:31.608 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:30:31.608 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:30:31.608 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:30:31.609 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:31.609 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:31.609 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:30:31.609 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:30:31.609 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:30:31.609 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:30:31.609 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:30:31.609 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:30:31.609 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:30:31.609 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:30:31.609 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:30:31.609 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:30:31.609 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:30:31.609 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:30:31.609 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:30:31.610 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:30:31.610 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:30:31.610 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:30:31.610 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:30:31.610 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:30:31.610 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:30:31.610 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:30:31.610 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:30:31.610 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:30:31.610 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:30:31.610 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:30:31.610 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:30:31.610 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:30:31.611 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:30:31.611 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:30:31.611 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:30:31.611 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:30:31.611 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:31.611 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:30:31.611 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:31.611 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:30:31.611 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:30:31.611 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:30:31.611 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:30:31.611 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:31.611 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:30:31.611 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:30:31.612 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:31.612 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:31.612 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:31.612 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:30:31.612 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:30:31.612 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:31.612 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:31.612 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:31.612 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:31.612 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:30:31.612 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:30:31.612 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:30:31.612 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:30:31.612 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:30:31.613 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:31.613 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:31.613 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:30:31.613 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:30:31.613 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:30:31.613 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:30:31.613 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:31.613 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:30:31.613 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:30:31.613 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:31.613 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:31.613 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:31.613 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:30:31.613 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:30:31.613 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:31.614 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:31.614 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:31.614 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:31.614 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:30:31.614 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:30:31.614 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:30:31.614 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:30:31.614 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:30:31.614 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:31.614 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:31.614 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:30:31.614 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:30:31.614 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:30:31.614 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:30:31.615 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:31.615 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:30:31.615 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:30:31.615 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:31.615 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:31.615 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:31.615 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:30:31.615 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:30:31.615 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:31.615 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:31.615 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:31.615 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:31.615 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:30:31.615 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:30:31.615 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:30:31.616 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:30:31.616 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:30:31.616 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:31.616 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:31.616 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:31.616 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:30:31.616 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:30:31.616 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:31.616 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:30:31.616 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:30:31.616 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:30:31.616 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:31.616 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:30:31.616 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:30:31.616 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:31.617 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:30:31.617 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:30:31.617 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:30:31.617 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:31.617 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:31.617 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:31.617 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:30:31.617 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:30:31.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.605+0000 7fe229325640 1 -- 172.21.15.37:0/3353362186 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe210035f30 msgr2=0x7fe2100383f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:31.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.605+0000 7fe229325640 1 --2- 172.21.15.37:0/3353362186 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe210035f30 0x7fe2100383f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe22413ce20 tx=0x7fe218008040 comp rx=0 tx=0).stop 2024-04-26T00:30:31.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.605+0000 7fe229325640 10 monclient: shutdown 2024-04-26T00:30:31.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.605+0000 7fe229325640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:30:31.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.605+0000 7fe229325640 1 -- 172.21.15.37:0/3353362186 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe224137300 msgr2=0x7fe22413e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:31.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.605+0000 7fe229325640 1 --2- 172.21.15.37:0/3353362186 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe224137300 0x7fe22413e8b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe21c00c3c0 tx=0x7fe21c00c880 comp rx=0 tx=0).stop 2024-04-26T00:30:31.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.605+0000 7fe2237fe640 1 -- 172.21.15.37:0/3353362186 reap_dead start 2024-04-26T00:30:31.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.605+0000 7fe2237fe640 1 -- 172.21.15.37:0/3353362186 reap_dead start 2024-04-26T00:30:31.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.605+0000 7fe229325640 1 -- 172.21.15.37:0/3353362186 shutdown_connections 2024-04-26T00:30:31.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.605+0000 7fe229325640 1 -- 172.21.15.37:0/3353362186 >> 172.21.15.37:0/3353362186 conn(0x7fe2240a61b0 msgr2=0x7fe2240a7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:30:31.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.605+0000 7fe229325640 1 -- 172.21.15.37:0/3353362186 shutdown_connections 2024-04-26T00:30:31.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.605+0000 7fe229325640 1 -- 172.21.15.37:0/3353362186 wait complete. 2024-04-26T00:30:31.620 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:30:31.620 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:30:31.620 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:30:31.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.733+0000 7fc10c1ae640 10 monclient: get_monmap_and_config 2024-04-26T00:30:31.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.733+0000 7fc10c1ae640 10 monclient: build_initial_monmap 2024-04-26T00:30:31.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.733+0000 7fc10c1ae640 10 monclient: monmap: 2024-04-26T00:30:31.726 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:30:31.726 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:31.726 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:30:31.738012+0000 2024-04-26T00:30:31.726 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:30:31.738012+0000 2024-04-26T00:30:31.726 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:30:31.726 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:31.726 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:30:31.726 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:30:31.726 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:30:31.726 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:30:31.726 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:30:31.727 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:31.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.733+0000 7fc10c1ae640 1 Processor -- start 2024-04-26T00:30:31.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.733+0000 7fc10c1ae640 1 -- start start 2024-04-26T00:30:31.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.733+0000 7fc10c1ae640 10 monclient: init 2024-04-26T00:30:31.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc10c1ae640 10 monclient: _reopen_session rank -1 2024-04-26T00:30:31.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc10c1ae640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-04-26T00:30:31.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc10c1ae640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc104072180 addr v1:172.21.15.45:6789/0 2024-04-26T00:30:31.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc10c1ae640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc104071670 0x7fc1041384c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:31.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc10c1ae640 10 monclient(hunting): picked mon.noname-b con 0x7fc104071670 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:30:31.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc10c1ae640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc104071a80 0x7fc10413aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:31.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc10c1ae640 10 monclient(hunting): picked mon.noname-a con 0x7fc104071a80 addr v2:172.21.15.37:3300/0 2024-04-26T00:30:31.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc10c1ae640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc104070580 con 0x7fc104072180 2024-04-26T00:30:31.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc10c1ae640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:31.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc10c1ae640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fc1040702a0 con 0x7fc104071a80 2024-04-26T00:30:31.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc10c1ae640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:31.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc10c1ae640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc104070860 con 0x7fc104071670 2024-04-26T00:30:31.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc10c1ae640 10 monclient(hunting): _renew_subs 2024-04-26T00:30:31.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc10c1ae640 10 monclient(hunting): authenticate will time out at 1613.044453s 2024-04-26T00:30:31.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc109749640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc104071670 0x7fc1041384c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:31.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc10a74b640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc104071a80 0x7fc10413aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:31.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc109749640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc104071670 0x7fc1041384c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:56410/0 (socket says 172.21.15.37:56410) 2024-04-26T00:30:31.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc10a74b640 10 monclient(hunting): get_auth_request con 0x7fc104071a80 auth_method 0 2024-04-26T00:30:31.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc109749640 1 -- 172.21.15.37:0/2323628960 learned_addr learned my addr 172.21.15.37:0/2323628960 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:30:31.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc10a74b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:31.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc10a74b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:31.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc10a74b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:31.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc109749640 10 monclient(hunting): get_auth_request con 0x7fc104071670 auth_method 0 2024-04-26T00:30:31.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc109749640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:31.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc109749640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:31.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc109749640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:31.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc10a74b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:31.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc10a74b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:31.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc10a74b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:31.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc109749640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:31.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc109749640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:31.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc109749640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:31.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc10a74b640 10 monclient(hunting): handle_auth_done global_id 124113 payload 274 2024-04-26T00:30:31.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc10a74b640 1 -- 172.21.15.37:0/2323628960 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc104071670 msgr2=0x7fc1041384c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:30:31.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc10a74b640 1 --2- 172.21.15.37:0/2323628960 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc104071670 0x7fc1041384c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:31.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc10a74b640 1 -- 172.21.15.37:0/2323628960 >> v1:172.21.15.45:6789/0 conn(0x7fc104072180 legacy=0x7fc104072560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:30:31.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc10a74b640 10 monclient: _finish_hunting 0 2024-04-26T00:30:31.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc10a74b640 1 monclient: found mon.noname-a 2024-04-26T00:30:31.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc10a74b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:30:31.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc10a74b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:30:31.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.737+0000 7fc10a74b640 1 -- 172.21.15.37:0/2323628960 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1040a3240 con 0x7fc104071a80 2024-04-26T00:30:31.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10a74b640 1 -- 172.21.15.37:0/2323628960 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc104071a80 msgr2=0x7fc10413aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:31.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10a74b640 1 --2- 172.21.15.37:0/2323628960 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc104071a80 0x7fc10413aa10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc009d90 tx=0x7fc0fc01c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a97939728572eda0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:31.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc108f48640 1 -- 172.21.15.37:0/2323628960 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc0b2070 con 0x7fc104071a80 2024-04-26T00:30:31.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc108f48640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:31.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc108f48640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:30:31.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc108f48640 10 monclient: dump: 2024-04-26T00:30:31.734 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:31.734 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:31.734 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.734 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.734 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:31.734 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:31.734 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:31.735 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:31.735 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:31.735 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:31.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc108f48640 10 monclient: _finish_auth 0 2024-04-26T00:30:31.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc108f48640 10 monclient: _check_auth_tickets 2024-04-26T00:30:31.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc108f48640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:30:31.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc108f48640 1 -- 172.21.15.37:0/2323628960 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0fc0a5a30 con 0x7fc104071a80 2024-04-26T00:30:31.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc108f48640 10 monclient: handle_config config(0 keys) 2024-04-26T00:30:31.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc108f48640 1 -- 172.21.15.37:0/2323628960 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc0a5e00 con 0x7fc104071a80 2024-04-26T00:30:31.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10c1ae640 5 monclient: authenticate success, global_id 124113 2024-04-26T00:30:31.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10c1ae640 10 monclient: get_monmap_and_config success 2024-04-26T00:30:31.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10c1ae640 4 set_mon_vals no callback set 2024-04-26T00:30:31.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc108f48640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:31.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc108f48640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:30:31.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc108f48640 10 monclient: dump: 2024-04-26T00:30:31.736 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:31.736 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:31.736 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.736 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.736 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:31.736 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:31.736 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:31.736 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:31.736 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:31.736 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:31.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10c1ae640 10 monclient: shutdown 2024-04-26T00:30:31.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10c1ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:30:31.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10c1ae640 1 -- 172.21.15.37:0/2323628960 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc104071a80 msgr2=0x7fc10413aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:31.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10c1ae640 1 --2- 172.21.15.37:0/2323628960 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc104071a80 0x7fc10413aa10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc009d90 tx=0x7fc0fc01c910 comp rx=0 tx=0).stop 2024-04-26T00:30:31.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10c1ae640 1 -- 172.21.15.37:0/2323628960 shutdown_connections 2024-04-26T00:30:31.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10c1ae640 1 --2- 172.21.15.37:0/2323628960 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc104071a80 0x7fc10413aa10 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:31.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10c1ae640 1 --2- 172.21.15.37:0/2323628960 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc104071670 0x7fc1041384c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:31.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10c1ae640 1 -- 172.21.15.37:0/2323628960 >> 172.21.15.37:0/2323628960 conn(0x7fc1040a24c0 msgr2=0x7fc1040a28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:30:31.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10c1ae640 1 -- 172.21.15.37:0/2323628960 shutdown_connections 2024-04-26T00:30:31.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10c1ae640 1 -- 172.21.15.37:0/2323628960 wait complete. 2024-04-26T00:30:31.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10c1ae640 10 monclient: build_initial_monmap 2024-04-26T00:30:31.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10c1ae640 10 monclient: monmap: 2024-04-26T00:30:31.737 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:30:31.737 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:30:31.737 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:30:31.737 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:30:31.737 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:30:31.737 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:30:31.737 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:30:31.738 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:30:31.738 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:30:31.738 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:31.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10c1ae640 1 Processor -- start 2024-04-26T00:30:31.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10c1ae640 1 -- start start 2024-04-26T00:30:31.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10c1ae640 10 monclient: init 2024-04-26T00:30:31.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10c1ae640 10 monclient: _reopen_session rank -1 2024-04-26T00:30:31.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10c1ae640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:30:31.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10c1ae640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc104071670 0x7fc1041cd350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:31.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10c1ae640 10 monclient(hunting): picked mon.noname-b con 0x7fc104071670 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:30:31.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10c1ae640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc104071a80 0x7fc10409ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:31.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10c1ae640 10 monclient(hunting): picked mon.noname-a con 0x7fc104071a80 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:30:31.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10c1ae640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc104072180 0x7fc10409ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:31.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10c1ae640 10 monclient(hunting): picked mon.noname-c con 0x7fc104072180 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:30:31.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10c1ae640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:31.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10c1ae640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fc10413bb30 con 0x7fc104071a80 2024-04-26T00:30:31.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10c1ae640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:31.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10c1ae640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc10413b9b0 con 0x7fc104071670 2024-04-26T00:30:31.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10c1ae640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:31.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10c1ae640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc10413bcb0 con 0x7fc104072180 2024-04-26T00:30:31.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10c1ae640 10 monclient(hunting): _renew_subs 2024-04-26T00:30:31.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10c1ae640 10 monclient(hunting): authenticate will time out at 1613.051147s 2024-04-26T00:30:31.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc109749640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc104071a80 0x7fc10409ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:31.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10a74b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc104072180 0x7fc10409ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:31.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10a74b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc104072180 0x7fc10409ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:56424/0 (socket says 172.21.15.37:56424) 2024-04-26T00:30:31.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10a74b640 1 -- 172.21.15.37:0/1536941194 learned_addr learned my addr 172.21.15.37:0/1536941194 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:30:31.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10a74b640 10 monclient(hunting): get_auth_request con 0x7fc104072180 auth_method 0 2024-04-26T00:30:31.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10a74b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:31.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10a74b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:31.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10a74b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:31.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc109749640 10 monclient(hunting): get_auth_request con 0x7fc104071a80 auth_method 0 2024-04-26T00:30:31.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc109749640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:31.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc109749640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:31.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc109749640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:31.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10a74b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:31.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10a74b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:31.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10a74b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:31.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc109749640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:31.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc109749640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:31.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc109749640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:31.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10a74b640 10 monclient(hunting): handle_auth_done global_id 84467 payload 824 2024-04-26T00:30:31.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10a74b640 1 -- 172.21.15.37:0/1536941194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc104071670 msgr2=0x7fc1041cd350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:31.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc109f4a640 1 --2- 172.21.15.37:0/1536941194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc104071670 0x7fc1041cd350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:31.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10a74b640 1 --2- 172.21.15.37:0/1536941194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc104071670 0x7fc1041cd350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:31.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10a74b640 1 -- 172.21.15.37:0/1536941194 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc104071a80 msgr2=0x7fc10409ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:31.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10a74b640 1 --2- 172.21.15.37:0/1536941194 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc104071a80 0x7fc10409ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:31.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10a74b640 10 monclient: _finish_hunting 0 2024-04-26T00:30:31.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10a74b640 1 monclient: found mon.noname-c 2024-04-26T00:30:31.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10a74b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:30:31.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10a74b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:30:31.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc10a74b640 1 -- 172.21.15.37:0/1536941194 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc104138b40 con 0x7fc104072180 2024-04-26T00:30:31.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.741+0000 7fc109749640 1 --2- 172.21.15.37:0/1536941194 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc104071a80 0x7fc10409ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:30:31.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc10a74b640 1 -- 172.21.15.37:0/1536941194 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc104072180 msgr2=0x7fc10409ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:31.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc10a74b640 1 --2- 172.21.15.37:0/1536941194 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc104072180 0x7fc10409ef40 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc009d90 tx=0x7fc0fc0196b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:31.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc0fa7fc640 1 -- 172.21.15.37:0/1536941194 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc0b2050 con 0x7fc104072180 2024-04-26T00:30:31.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc0fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:31.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc0fa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:30:31.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc0fa7fc640 10 monclient: dump: 2024-04-26T00:30:31.746 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:31.746 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:31.747 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.747 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.747 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:31.747 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:31.747 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:31.747 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:31.747 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:31.747 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:31.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc0fa7fc640 10 monclient: _finish_auth 0 2024-04-26T00:30:31.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc0fa7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:30:31.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc0fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:30:31.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc0fa7fc640 1 -- 172.21.15.37:0/1536941194 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0fc0be070 con 0x7fc104072180 2024-04-26T00:30:31.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc10c1ae640 5 monclient: authenticate success, global_id 84467 2024-04-26T00:30:31.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc10c1ae640 10 monclient: _renew_subs 2024-04-26T00:30:31.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc10c1ae640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:30:31.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc10c1ae640 1 -- 172.21.15.37:0/1536941194 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc10409b4b0 con 0x7fc104072180 2024-04-26T00:30:31.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc0fa7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:30:31.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc0fa7fc640 1 -- 172.21.15.37:0/1536941194 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc0b9450 con 0x7fc104072180 2024-04-26T00:30:31.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc0fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:31.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc0fa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:30:31.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc0fa7fc640 10 monclient: dump: 2024-04-26T00:30:31.748 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:31.748 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:31.748 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.748 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.748 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:31.748 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:31.748 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:31.748 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:31.749 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:31.749 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:31.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc0fbfff640 4 set_mon_vals no callback set 2024-04-26T00:30:31.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc10c1ae640 10 monclient: _renew_subs 2024-04-26T00:30:31.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc10c1ae640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:30:31.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc10c1ae640 1 -- 172.21.15.37:0/1536941194 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc10409b1d0 con 0x7fc104072180 2024-04-26T00:30:31.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc10c1ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:30:31.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc10c1ae640 1 --2- 172.21.15.37:0/1536941194 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc104136f30 0x7fc104137310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:31.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc10c1ae640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:31.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc10c1ae640 1 -- 172.21.15.37:0/1536941194 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc1041cd890 con 0x7fc104136f30 2024-04-26T00:30:31.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc109f4a640 1 --2- 172.21.15.37:0/1536941194 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc104136f30 0x7fc104137310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:31.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc109f4a640 10 monclient: get_auth_request con 0x7fc104136f30 auth_method 0 2024-04-26T00:30:31.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc109f4a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:31.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc109f4a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:31.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc109f4a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:31.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc0fa7fc640 1 -- 172.21.15.37:0/1536941194 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc0fc0b95f0 con 0x7fc104072180 2024-04-26T00:30:31.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc109f4a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:31.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc109f4a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:31.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc109f4a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:31.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc109f4a640 10 monclient(hunting): handle_auth_done global_id 84470 payload 824 2024-04-26T00:30:31.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc109f4a640 1 -- 172.21.15.37:0/1536941194 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc10404f510 con 0x7fc104136f30 2024-04-26T00:30:31.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc0fa7fc640 1 --2- 172.21.15.37:0/1536941194 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc0e8035e80 0x7fc0e8038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:31.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc0fa7fc640 1 -- 172.21.15.37:0/1536941194 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(402..402 src has 370..402) ==== 3975+0+0 (secure 0 0 0) 0x7fc0fc0e37d0 con 0x7fc104072180 2024-04-26T00:30:31.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc109749640 1 --2- 172.21.15.37:0/1536941194 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc0e8035e80 0x7fc0e8038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:31.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.745+0000 7fc109749640 10 monclient: get_auth_request con 0x7fc0e8035e80 auth_method 0 2024-04-26T00:30:31.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.749+0000 7fc109f4a640 1 -- 172.21.15.37:0/1536941194 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc104136f30 msgr2=0x7fc104137310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:31.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.749+0000 7fc109f4a640 1 --2- 172.21.15.37:0/1536941194 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc104136f30 0x7fc104137310 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc10000d380 tx=0x7fc10000d840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:31.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.749+0000 7fc0fa7fc640 1 -- 172.21.15.37:0/1536941194 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc100016020 con 0x7fc104136f30 2024-04-26T00:30:31.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.749+0000 7fc0fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:31.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.749+0000 7fc0fa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:30:31.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.749+0000 7fc0fa7fc640 10 monclient: dump: 2024-04-26T00:30:31.751 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:31.751 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:31.751 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.751 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.751 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:31.751 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:31.751 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:31.751 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:31.751 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:31.751 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:31.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.749+0000 7fc0fa7fc640 1 -- 172.21.15.37:0/1536941194 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc100012a10 con 0x7fc104136f30 2024-04-26T00:30:31.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.749+0000 7fc0fa7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:30:31.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.749+0000 7fc0fa7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:30:31.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.749+0000 7fc0fa7fc640 1 -- 172.21.15.37:0/1536941194 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc104136f30 msgr2=0x7fc104137310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:31.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.749+0000 7fc0fa7fc640 1 --2- 172.21.15.37:0/1536941194 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc104136f30 0x7fc104137310 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc10000d380 tx=0x7fc10000d840 comp rx=0 tx=0).stop 2024-04-26T00:30:31.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.749+0000 7fc0fa7fc640 1 -- 172.21.15.37:0/1536941194 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc104136f30 msgr2=0x7fc104137310 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:30:31.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.749+0000 7fc0fa7fc640 1 --2- 172.21.15.37:0/1536941194 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc104136f30 0x7fc104137310 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:31.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.749+0000 7fc109749640 1 --2- 172.21.15.37:0/1536941194 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc0e8035e80 0x7fc0e8038340 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fc10406f820 tx=0x7fc0f4009240 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:31.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.753+0000 7fc10c1ae640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:30:31.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.753+0000 7fc10c1ae640 1 --2- 172.21.15.37:0/1536941194 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc104137850 0x7fc104132a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:31.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.753+0000 7fc10c1ae640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:31.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.753+0000 7fc10c1ae640 1 -- 172.21.15.37:0/1536941194 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc10409f480 con 0x7fc104137850 2024-04-26T00:30:31.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.753+0000 7fc109f4a640 1 --2- 172.21.15.37:0/1536941194 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc104137850 0x7fc104132a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:31.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.753+0000 7fc109f4a640 10 monclient: get_auth_request con 0x7fc104137850 auth_method 0 2024-04-26T00:30:31.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.753+0000 7fc109f4a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:31.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.753+0000 7fc109f4a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:31.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.753+0000 7fc109f4a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:31.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.757+0000 7fc109f4a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:31.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.757+0000 7fc109f4a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:31.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.757+0000 7fc109f4a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:31.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.757+0000 7fc109f4a640 10 monclient(hunting): handle_auth_done global_id 84473 payload 824 2024-04-26T00:30:31.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.757+0000 7fc109f4a640 1 -- 172.21.15.37:0/1536941194 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc104137c60 con 0x7fc104137850 2024-04-26T00:30:31.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.757+0000 7fc109f4a640 1 -- 172.21.15.37:0/1536941194 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc104137850 msgr2=0x7fc104132a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:31.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.757+0000 7fc109f4a640 1 --2- 172.21.15.37:0/1536941194 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc104137850 0x7fc104132a60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc10000d380 tx=0x7fc100003770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:31.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.757+0000 7fc0fa7fc640 1 -- 172.21.15.37:0/1536941194 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc100016040 con 0x7fc104137850 2024-04-26T00:30:31.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.757+0000 7fc0fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:31.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.757+0000 7fc0fa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:30:31.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.757+0000 7fc0fa7fc640 10 monclient: dump: 2024-04-26T00:30:31.755 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:31.755 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:31.755 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.755 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.755 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:31.755 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:31.755 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:31.755 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:31.756 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:31.756 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:31.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.757+0000 7fc0fa7fc640 1 -- 172.21.15.37:0/1536941194 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4719 (secure 0 0 0) 0x7fc10001a8c0 con 0x7fc104137850 2024-04-26T00:30:31.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.757+0000 7fc0fa7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:30:31.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.757+0000 7fc0fa7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:30:31.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.757+0000 7fc0fa7fc640 1 -- 172.21.15.37:0/1536941194 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc104137850 msgr2=0x7fc104132a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:31.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.757+0000 7fc0fa7fc640 1 --2- 172.21.15.37:0/1536941194 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc104137850 0x7fc104132a60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc10000d380 tx=0x7fc100003770 comp rx=0 tx=0).stop 2024-04-26T00:30:31.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.757+0000 7fc0fa7fc640 1 -- 172.21.15.37:0/1536941194 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc104137850 msgr2=0x7fc104132a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:30:31.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.757+0000 7fc0fa7fc640 1 --2- 172.21.15.37:0/1536941194 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc104137850 0x7fc104132a60 secure :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc10000d380 tx=0x7fc100003770 comp rx=0 tx=0).stop 2024-04-26T00:30:31.756 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:30:31.756 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:30:31.757 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:30:31.757 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:30:31.757 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 80, 2024-04-26T00:30:31.757 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 236692, 2024-04-26T00:30:31.757 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:30:31.757 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:30:31.757 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:30:31.757 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:30:31.757 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:30:31.757 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:30:31.757 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:30:31.757 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:30:31.757 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:30:31.758 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:30:31.758 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:30:31.758 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:30:31.758 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:30:31.758 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:30:31.758 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:30:31.758 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:30:31.758 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:30:31.758 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:30:31.758 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:30:31.758 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:30:31.758 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:30:31.758 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:30:31.759 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:30:31.759 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:30:31.759 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:30:31.759 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:30:31.759 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:30:31.759 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:30:31.759 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:30:31.759 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:30:31.759 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:30:31.759 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:30:31.759 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:30:31.759 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:30:31.759 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:31.759 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:31.759 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:30:31.760 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:30:31.760 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:30:31.760 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:30:31.760 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:30:31.760 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:30:31.760 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:30:31.760 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:30:31.760 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:30:31.760 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:30:31.760 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:30:31.760 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:30:31.760 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:30:31.760 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:30:31.761 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:30:31.761 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:30:31.761 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:30:31.761 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:30:31.761 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:30:31.761 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:30:31.761 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:30:31.761 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:30:31.761 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:30:31.761 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:30:31.761 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:30:31.761 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:30:31.761 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:30:31.761 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:30:31.762 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:30:31.762 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:30:31.762 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:31.762 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:30:31.762 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:31.762 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:30:31.762 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:30:31.762 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:30:31.762 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:30:31.762 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:31.762 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:30:31.762 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:30:31.762 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:31.762 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:31.763 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:31.763 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:30:31.763 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:30:31.763 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:31.763 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:31.763 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:31.763 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:31.763 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:30:31.763 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:30:31.763 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:30:31.763 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:30:31.763 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:30:31.763 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:31.763 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:31.763 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:30:31.764 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:30:31.764 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:30:31.764 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:30:31.764 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:31.764 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:30:31.764 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:30:31.764 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:31.764 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:31.764 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:31.764 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:30:31.764 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:30:31.764 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:31.764 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:31.764 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:31.764 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:31.765 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:30:31.765 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:30:31.765 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:30:31.765 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:30:31.765 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:30:31.765 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:31.765 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:31.765 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:30:31.765 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:30:31.765 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:30:31.765 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:30:31.765 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:31.765 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:30:31.765 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:30:31.765 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:31.766 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:31.766 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:31.766 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:30:31.766 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:30:31.766 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:31.766 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:31.766 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:31.766 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:31.766 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:30:31.766 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:30:31.766 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:30:31.766 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:30:31.766 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:30:31.766 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:31.766 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:31.767 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:31.767 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:30:31.767 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:30:31.767 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:31.767 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:30:31.767 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:30:31.767 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:30:31.767 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:31.767 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:30:31.767 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:30:31.767 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:31.767 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:30:31.767 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:30:31.767 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:30:31.767 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:31.768 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:30:31.768 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:30:31.768 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:31.768 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:30:31.768 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:30:31.768 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:30:31.768 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:31.768 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:31.768 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:31.768 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:30:31.768 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:30:31.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.757+0000 7fc10c1ae640 1 -- 172.21.15.37:0/1536941194 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc0e8035e80 msgr2=0x7fc0e8038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:31.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.757+0000 7fc10c1ae640 1 --2- 172.21.15.37:0/1536941194 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc0e8035e80 0x7fc0e8038340 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fc10406f820 tx=0x7fc0f4009240 comp rx=0 tx=0).stop 2024-04-26T00:30:31.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.757+0000 7fc10c1ae640 10 monclient: shutdown 2024-04-26T00:30:31.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.757+0000 7fc10c1ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:30:31.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.757+0000 7fc10c1ae640 1 -- 172.21.15.37:0/1536941194 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc104072180 msgr2=0x7fc10409ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:31.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.757+0000 7fc10c1ae640 1 --2- 172.21.15.37:0/1536941194 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc104072180 0x7fc10409ef40 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc009d90 tx=0x7fc0fc0196b0 comp rx=0 tx=0).stop 2024-04-26T00:30:31.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.757+0000 7fc10a74b640 1 -- 172.21.15.37:0/1536941194 reap_dead start 2024-04-26T00:30:31.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.757+0000 7fc10c1ae640 1 -- 172.21.15.37:0/1536941194 shutdown_connections 2024-04-26T00:30:31.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.757+0000 7fc10c1ae640 1 --2- 172.21.15.37:0/1536941194 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc104072180 0x7fc10409ef40 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:31.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.757+0000 7fc10c1ae640 1 -- 172.21.15.37:0/1536941194 >> 172.21.15.37:0/1536941194 conn(0x7fc1040a24c0 msgr2=0x7fc1040a1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:30:31.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.757+0000 7fc10c1ae640 1 -- 172.21.15.37:0/1536941194 shutdown_connections 2024-04-26T00:30:31.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.757+0000 7fc10c1ae640 1 -- 172.21.15.37:0/1536941194 wait complete. 2024-04-26T00:30:31.770 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:30:31.771 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:30:31.771 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:30:31.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0dd91d640 10 monclient: get_monmap_and_config 2024-04-26T00:30:31.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0dd91d640 10 monclient: build_initial_monmap 2024-04-26T00:30:31.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0dd91d640 10 monclient: monmap: 2024-04-26T00:30:31.880 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:30:31.880 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:31.880 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:30:31.892491+0000 2024-04-26T00:30:31.880 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:30:31.892491+0000 2024-04-26T00:30:31.880 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:30:31.880 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:31.881 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:30:31.881 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:30:31.881 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:30:31.881 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:30:31.881 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:30:31.881 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:31.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0dd91d640 1 Processor -- start 2024-04-26T00:30:31.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0dd91d640 1 -- start start 2024-04-26T00:30:31.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0dd91d640 10 monclient: init 2024-04-26T00:30:31.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0dd91d640 10 monclient: _reopen_session rank -1 2024-04-26T00:30:31.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0dd91d640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-04-26T00:30:31.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0dd91d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb0d8130eb0 addr v1:172.21.15.37:6789/0 2024-04-26T00:30:31.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0dd91d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb0d809ef20 0x7fb0d8138eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:31.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0dd91d640 10 monclient(hunting): picked mon.noname-c con 0x7fb0d809ef20 addr v2:172.21.15.45:3300/0 2024-04-26T00:30:31.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0dd91d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb0d81393f0 0x7fb0d813b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:31.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0dd91d640 10 monclient(hunting): picked mon.noname-b con 0x7fb0d81393f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:30:31.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0dd91d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0d809c190 con 0x7fb0d8130eb0 2024-04-26T00:30:31.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0dd91d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:31.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0dd91d640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fb0d809df60 con 0x7fb0d809ef20 2024-04-26T00:30:31.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0dd91d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:31.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0dd91d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb0d80a11c0 con 0x7fb0d81393f0 2024-04-26T00:30:31.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0dd91d640 10 monclient(hunting): _renew_subs 2024-04-26T00:30:31.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0dd91d640 10 monclient(hunting): authenticate will time out at 1613.198346s 2024-04-26T00:30:31.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0d6ffd640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fb0d8130eb0 0x7fb0d8131290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:34756/0 (socket says 172.21.15.37:34756) 2024-04-26T00:30:31.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0d77fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb0d81393f0 0x7fb0d813b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:31.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0d6ffd640 1 -- 172.21.15.37:0/2729811911 learned_addr learned my addr 172.21.15.37:0/2729811911 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:30:31.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0d77fe640 10 monclient(hunting): get_auth_request con 0x7fb0d81393f0 auth_method 0 2024-04-26T00:30:31.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:31.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0d77fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:31.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:31.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:31.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:31.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:31.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0d67fc640 1 --2- 172.21.15.37:0/2729811911 >> v2:172.21.15.45:3300/0 conn(0x7fb0d809ef20 0x7fb0d8138eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:31.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0d67fc640 10 monclient(hunting): get_auth_request con 0x7fb0d809ef20 auth_method 0 2024-04-26T00:30:31.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0d67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:31.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0d67fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:31.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:31.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0d77fe640 10 monclient(hunting): handle_auth_done global_id 84476 payload 274 2024-04-26T00:30:31.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0d77fe640 1 -- 172.21.15.37:0/2729811911 >> v2:172.21.15.45:3300/0 conn(0x7fb0d809ef20 msgr2=0x7fb0d8138eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:30:31.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0d77fe640 1 --2- 172.21.15.37:0/2729811911 >> v2:172.21.15.45:3300/0 conn(0x7fb0d809ef20 0x7fb0d8138eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:31.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0d77fe640 1 -- 172.21.15.37:0/2729811911 >> v1:172.21.15.37:6789/0 conn(0x7fb0d8130eb0 legacy=0x7fb0d8131290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:30:31.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0d77fe640 10 monclient: _finish_hunting 0 2024-04-26T00:30:31.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0d77fe640 1 monclient: found mon.noname-b 2024-04-26T00:30:31.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:30:31.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0d77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:30:31.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.889+0000 7fb0d77fe640 1 -- 172.21.15.37:0/2729811911 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0d809dc80 con 0x7fb0d81393f0 2024-04-26T00:30:31.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.893+0000 7fb0d77fe640 1 -- 172.21.15.37:0/2729811911 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb0d81393f0 msgr2=0x7fb0d813b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:31.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.893+0000 7fb0d77fe640 1 --2- 172.21.15.37:0/2729811911 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb0d81393f0 0x7fb0d813b7e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb0d806cd00 tx=0x7fb0d001c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7dfdb29d3dbe039 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:31.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.893+0000 7fb0d5ffb640 1 -- 172.21.15.37:0/2729811911 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d00b0070 con 0x7fb0d81393f0 2024-04-26T00:30:31.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.893+0000 7fb0d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:31.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.893+0000 7fb0d5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:30:31.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.893+0000 7fb0d5ffb640 10 monclient: dump: 2024-04-26T00:30:31.887 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:31.887 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:31.888 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.888 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.888 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:31.888 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:31.888 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:31.888 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:31.888 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:31.888 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:31.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.893+0000 7fb0d5ffb640 10 monclient: _finish_auth 0 2024-04-26T00:30:31.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.893+0000 7fb0d5ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:30:31.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.893+0000 7fb0d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:30:31.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.893+0000 7fb0d5ffb640 1 -- 172.21.15.37:0/2729811911 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0d00ab600 con 0x7fb0d81393f0 2024-04-26T00:30:31.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.893+0000 7fb0d5ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:30:31.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.893+0000 7fb0d5ffb640 1 -- 172.21.15.37:0/2729811911 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d00ab9d0 con 0x7fb0d81393f0 2024-04-26T00:30:31.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.893+0000 7fb0dd91d640 5 monclient: authenticate success, global_id 84476 2024-04-26T00:30:31.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.893+0000 7fb0dd91d640 10 monclient: get_monmap_and_config success 2024-04-26T00:30:31.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.893+0000 7fb0dd91d640 4 set_mon_vals no callback set 2024-04-26T00:30:31.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.893+0000 7fb0d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:31.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.893+0000 7fb0d5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:30:31.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.893+0000 7fb0d5ffb640 10 monclient: dump: 2024-04-26T00:30:31.889 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:31.889 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:31.889 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.889 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.889 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:31.889 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:31.889 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:31.889 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:31.889 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:31.889 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:31.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0dd91d640 10 monclient: shutdown 2024-04-26T00:30:31.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0dd91d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:30:31.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0dd91d640 1 -- 172.21.15.37:0/2729811911 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb0d81393f0 msgr2=0x7fb0d813b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:31.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0dd91d640 1 --2- 172.21.15.37:0/2729811911 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb0d81393f0 0x7fb0d813b7e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb0d806cd00 tx=0x7fb0d001c2d0 comp rx=0 tx=0).stop 2024-04-26T00:30:31.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0dd91d640 1 -- 172.21.15.37:0/2729811911 shutdown_connections 2024-04-26T00:30:31.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0dd91d640 1 --2- 172.21.15.37:0/2729811911 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb0d81393f0 0x7fb0d813b7e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:31.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0dd91d640 1 --2- 172.21.15.37:0/2729811911 >> v2:172.21.15.45:3300/0 conn(0x7fb0d809ef20 0x7fb0d8138eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:31.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0dd91d640 1 -- 172.21.15.37:0/2729811911 >> 172.21.15.37:0/2729811911 conn(0x7fb0d80a1d50 msgr2=0x7fb0d80a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:30:31.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0dd91d640 1 -- 172.21.15.37:0/2729811911 shutdown_connections 2024-04-26T00:30:31.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0dd91d640 1 -- 172.21.15.37:0/2729811911 wait complete. 2024-04-26T00:30:31.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0dd91d640 10 monclient: build_initial_monmap 2024-04-26T00:30:31.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0dd91d640 10 monclient: monmap: 2024-04-26T00:30:31.896 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:30:31.896 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:30:31.896 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:30:31.896 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:30:31.896 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:30:31.896 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:30:31.896 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:30:31.896 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:30:31.896 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:30:31.896 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:31.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0dd91d640 1 Processor -- start 2024-04-26T00:30:31.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0dd91d640 1 -- start start 2024-04-26T00:30:31.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0dd91d640 10 monclient: init 2024-04-26T00:30:31.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0dd91d640 10 monclient: _reopen_session rank -1 2024-04-26T00:30:31.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0dd91d640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:30:31.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0dd91d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0d809ef20 0x7fb0d8071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:31.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0dd91d640 10 monclient(hunting): picked mon.noname-a con 0x7fb0d809ef20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:30:31.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0dd91d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb0d8130eb0 0x7fb0d812ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:31.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0dd91d640 10 monclient(hunting): picked mon.noname-c con 0x7fb0d8130eb0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:30:31.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0dd91d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0d81393f0 0x7fb0d812f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:31.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0dd91d640 10 monclient(hunting): picked mon.noname-b con 0x7fb0d81393f0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:30:31.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0dd91d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:31.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0dd91d640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d80a05e0 con 0x7fb0d809ef20 2024-04-26T00:30:31.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0dd91d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:31.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0dd91d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d80a0460 con 0x7fb0d81393f0 2024-04-26T00:30:31.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0dd91d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:31.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0dd91d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb0d80a0760 con 0x7fb0d8130eb0 2024-04-26T00:30:31.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0dd91d640 10 monclient(hunting): _renew_subs 2024-04-26T00:30:31.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0dd91d640 10 monclient(hunting): authenticate will time out at 1613.206159s 2024-04-26T00:30:31.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0d6ffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0d809ef20 0x7fb0d8071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:31.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0d6ffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0d809ef20 0x7fb0d8071390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:43240/0 (socket says 172.21.15.37:43240) 2024-04-26T00:30:31.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0d67fc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb0d8130eb0 0x7fb0d812ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:31.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0d6ffd640 1 -- 172.21.15.37:0/544939264 learned_addr learned my addr 172.21.15.37:0/544939264 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:30:31.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0d6ffd640 10 monclient(hunting): get_auth_request con 0x7fb0d809ef20 auth_method 0 2024-04-26T00:30:31.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:31.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0d6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:31.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:31.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0d67fc640 10 monclient(hunting): get_auth_request con 0x7fb0d8130eb0 auth_method 0 2024-04-26T00:30:31.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0d67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:31.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0d67fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:31.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:31.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0d67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:31.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0d67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:31.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0d67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:31.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:31.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:31.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:31.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0d77fe640 1 --2- 172.21.15.37:0/544939264 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0d81393f0 0x7fb0d812f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:31.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0d67fc640 10 monclient(hunting): handle_auth_done global_id 84479 payload 824 2024-04-26T00:30:31.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0d67fc640 1 -- 172.21.15.37:0/544939264 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0d81393f0 msgr2=0x7fb0d812f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:31.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0d67fc640 1 --2- 172.21.15.37:0/544939264 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0d81393f0 0x7fb0d812f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:31.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0d67fc640 1 -- 172.21.15.37:0/544939264 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0d809ef20 msgr2=0x7fb0d8071390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:31.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0d67fc640 1 --2- 172.21.15.37:0/544939264 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0d809ef20 0x7fb0d8071390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:31.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0d67fc640 10 monclient: _finish_hunting 0 2024-04-26T00:30:31.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0d67fc640 1 monclient: found mon.noname-c 2024-04-26T00:30:31.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:30:31.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0d67fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:30:31.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0d67fc640 1 -- 172.21.15.37:0/544939264 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0d81386a0 con 0x7fb0d8130eb0 2024-04-26T00:30:31.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0d77fe640 10 monclient: get_auth_request con 0x7fb0d81393f0 auth_method 0 2024-04-26T00:30:31.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0d77fe640 1 --2- 172.21.15.37:0/544939264 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0d81393f0 0x7fb0d812f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:30:31.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.897+0000 7fb0d6ffd640 1 --2- 172.21.15.37:0/544939264 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0d809ef20 0x7fb0d8071390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:30:31.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0d67fc640 1 -- 172.21.15.37:0/544939264 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb0d8130eb0 msgr2=0x7fb0d812ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:31.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0d67fc640 1 --2- 172.21.15.37:0/544939264 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb0d8130eb0 0x7fb0d812ec60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc00d5f0 tx=0x7fb0cc00dab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:31.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0bffff640 1 -- 172.21.15.37:0/544939264 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc018020 con 0x7fb0d8130eb0 2024-04-26T00:30:31.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:31.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0bffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:30:31.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0bffff640 10 monclient: dump: 2024-04-26T00:30:31.901 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:31.901 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:31.901 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.901 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.901 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:31.901 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:31.901 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:31.901 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:31.901 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:31.901 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:31.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0bffff640 10 monclient: _finish_auth 0 2024-04-26T00:30:31.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0bffff640 10 monclient: _check_auth_tickets 2024-04-26T00:30:31.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:30:31.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0bffff640 1 -- 172.21.15.37:0/544939264 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0cc014830 con 0x7fb0d8130eb0 2024-04-26T00:30:31.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0dd91d640 5 monclient: authenticate success, global_id 84479 2024-04-26T00:30:31.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0dd91d640 10 monclient: _renew_subs 2024-04-26T00:30:31.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0dd91d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:30:31.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0dd91d640 1 -- 172.21.15.37:0/544939264 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0d8070110 con 0x7fb0d8130eb0 2024-04-26T00:30:31.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0bffff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:30:31.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0bffff640 1 -- 172.21.15.37:0/544939264 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc014c00 con 0x7fb0d8130eb0 2024-04-26T00:30:31.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:31.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0bffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:30:31.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0bffff640 10 monclient: dump: 2024-04-26T00:30:31.902 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:31.902 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:31.902 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.902 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.902 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:31.902 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:31.903 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:31.903 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:31.903 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:31.903 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:31.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0dd91d640 10 monclient: _renew_subs 2024-04-26T00:30:31.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0dd91d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:30:31.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0dd91d640 1 -- 172.21.15.37:0/544939264 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0d812f6e0 con 0x7fb0d8130eb0 2024-04-26T00:30:31.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0d57fa640 4 set_mon_vals no callback set 2024-04-26T00:30:31.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0dd91d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:30:31.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0dd91d640 1 --2- 172.21.15.37:0/544939264 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0d81397d0 0x7fb0d8139bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:31.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0dd91d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:31.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0dd91d640 1 -- 172.21.15.37:0/544939264 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d806edc0 con 0x7fb0d81397d0 2024-04-26T00:30:31.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0bffff640 1 -- 172.21.15.37:0/544939264 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb0cc01b020 con 0x7fb0d8130eb0 2024-04-26T00:30:31.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0bffff640 1 --2- 172.21.15.37:0/544939264 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb0c4035ed0 0x7fb0c4038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:31.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0bffff640 1 -- 172.21.15.37:0/544939264 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(402..402 src has 370..402) ==== 3975+0+0 (secure 0 0 0) 0x7fb0cc048fa0 con 0x7fb0d8130eb0 2024-04-26T00:30:31.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0d6ffd640 1 --2- 172.21.15.37:0/544939264 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0d81397d0 0x7fb0d8139bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:31.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0d6ffd640 10 monclient: get_auth_request con 0x7fb0d81397d0 auth_method 0 2024-04-26T00:30:31.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:31.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0d6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:31.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:31.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:31.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:31.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0d77fe640 1 --2- 172.21.15.37:0/544939264 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb0c4035ed0 0x7fb0c4038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:31.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:31.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0d77fe640 10 monclient: get_auth_request con 0x7fb0c4035ed0 auth_method 0 2024-04-26T00:30:31.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0d6ffd640 10 monclient(hunting): handle_auth_done global_id 114181 payload 824 2024-04-26T00:30:31.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.901+0000 7fb0d6ffd640 1 -- 172.21.15.37:0/544939264 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb0d804f510 con 0x7fb0d81397d0 2024-04-26T00:30:31.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.905+0000 7fb0d77fe640 1 --2- 172.21.15.37:0/544939264 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb0c4035ed0 0x7fb0c4038390 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb0d00ab000 tx=0x7fb0d00a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:31.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.905+0000 7fb0d6ffd640 1 -- 172.21.15.37:0/544939264 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0d81397d0 msgr2=0x7fb0d8139bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:31.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.905+0000 7fb0d6ffd640 1 --2- 172.21.15.37:0/544939264 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0d81397d0 0x7fb0d8139bb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb0c800d3d0 tx=0x7fb0c800d890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:31.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.905+0000 7fb0bffff640 1 -- 172.21.15.37:0/544939264 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c80075b0 con 0x7fb0d81397d0 2024-04-26T00:30:31.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.905+0000 7fb0bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:31.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.905+0000 7fb0bffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:30:31.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.905+0000 7fb0bffff640 10 monclient: dump: 2024-04-26T00:30:31.907 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:31.907 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:31.907 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.907 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.907 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:31.907 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:31.907 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:31.907 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:31.907 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:31.907 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:31.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.905+0000 7fb0bffff640 1 -- 172.21.15.37:0/544939264 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb0c8018070 con 0x7fb0d81397d0 2024-04-26T00:30:31.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.905+0000 7fb0bffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:30:31.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.905+0000 7fb0bffff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:30:31.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.905+0000 7fb0bffff640 1 -- 172.21.15.37:0/544939264 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0d81397d0 msgr2=0x7fb0d8139bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:31.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.905+0000 7fb0bffff640 1 --2- 172.21.15.37:0/544939264 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0d81397d0 0x7fb0d8139bb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb0c800d3d0 tx=0x7fb0c800d890 comp rx=0 tx=0).stop 2024-04-26T00:30:31.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.905+0000 7fb0bffff640 1 -- 172.21.15.37:0/544939264 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0d81397d0 msgr2=0x7fb0d8139bb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:30:31.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.905+0000 7fb0bffff640 1 --2- 172.21.15.37:0/544939264 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0d81397d0 0x7fb0d8139bb0 secure :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb0c800d3d0 tx=0x7fb0c800d890 comp rx=0 tx=0).stop 2024-04-26T00:30:31.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.913+0000 7fb0dd91d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:30:31.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.913+0000 7fb0dd91d640 1 --2- 172.21.15.37:0/544939264 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0d813a0f0 0x7fb0d806dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:31.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.913+0000 7fb0dd91d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:31.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.913+0000 7fb0dd91d640 1 -- 172.21.15.37:0/544939264 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d80a0f60 con 0x7fb0d813a0f0 2024-04-26T00:30:31.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.913+0000 7fb0d6ffd640 1 --2- 172.21.15.37:0/544939264 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0d813a0f0 0x7fb0d806dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:31.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.913+0000 7fb0d6ffd640 10 monclient: get_auth_request con 0x7fb0d813a0f0 auth_method 0 2024-04-26T00:30:31.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.913+0000 7fb0d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:31.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.913+0000 7fb0d6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:31.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.913+0000 7fb0d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:31.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.913+0000 7fb0d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:31.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.913+0000 7fb0d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:31.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.913+0000 7fb0d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:31.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.913+0000 7fb0d6ffd640 10 monclient(hunting): handle_auth_done global_id 114184 payload 824 2024-04-26T00:30:31.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.913+0000 7fb0d6ffd640 1 -- 172.21.15.37:0/544939264 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb0d80704a0 con 0x7fb0d813a0f0 2024-04-26T00:30:31.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.917+0000 7fb0d6ffd640 1 -- 172.21.15.37:0/544939264 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0d813a0f0 msgr2=0x7fb0d806dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:31.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.917+0000 7fb0d6ffd640 1 --2- 172.21.15.37:0/544939264 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0d813a0f0 0x7fb0d806dc40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb0c800d3d0 tx=0x7fb0c8004cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:31.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.917+0000 7fb0bffff640 1 -- 172.21.15.37:0/544939264 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c801de20 con 0x7fb0d813a0f0 2024-04-26T00:30:31.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.917+0000 7fb0bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:31.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.917+0000 7fb0bffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:30:31.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.917+0000 7fb0bffff640 10 monclient: dump: 2024-04-26T00:30:31.910 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:31.910 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:31.910 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.910 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:31.910 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:31.910 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:31.910 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:31.911 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:31.911 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:31.911 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:31.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.917+0000 7fb0bffff640 1 -- 172.21.15.37:0/544939264 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4716 (secure 0 0 0) 0x7fb0c801de20 con 0x7fb0d813a0f0 2024-04-26T00:30:31.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.917+0000 7fb0bffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:30:31.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.917+0000 7fb0bffff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:30:31.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.917+0000 7fb0bffff640 1 -- 172.21.15.37:0/544939264 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0d813a0f0 msgr2=0x7fb0d806dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:31.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.917+0000 7fb0bffff640 1 --2- 172.21.15.37:0/544939264 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0d813a0f0 0x7fb0d806dc40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb0c800d3d0 tx=0x7fb0c8004cd0 comp rx=0 tx=0).stop 2024-04-26T00:30:31.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.917+0000 7fb0bffff640 1 -- 172.21.15.37:0/544939264 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0d813a0f0 msgr2=0x7fb0d806dc40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:30:31.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.917+0000 7fb0bffff640 1 --2- 172.21.15.37:0/544939264 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0d813a0f0 0x7fb0d806dc40 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb0c800d3d0 tx=0x7fb0c8004cd0 comp rx=0 tx=0).stop 2024-04-26T00:30:31.911 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:30:31.912 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:30:31.912 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:30:31.912 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:30:31.912 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 80, 2024-04-26T00:30:31.912 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 82003, 2024-04-26T00:30:31.912 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:30:31.912 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:30:31.912 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:30:31.912 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:30:31.912 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:30:31.912 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:30:31.912 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:30:31.912 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:30:31.912 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:30:31.913 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:30:31.913 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:30:31.913 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:30:31.913 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:30:31.913 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:30:31.913 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:30:31.913 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:30:31.913 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:30:31.913 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:30:31.913 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:30:31.913 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:30:31.913 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:30:31.913 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:30:31.913 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:30:31.914 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:30:31.914 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:30:31.914 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:30:31.914 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:30:31.914 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:30:31.914 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:30:31.914 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:30:31.914 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:30:31.914 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:30:31.914 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:30:31.914 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:30:31.914 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:31.914 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:31.914 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:30:31.915 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:30:31.915 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:30:31.915 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:30:31.915 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:30:31.915 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:30:31.915 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:30:31.915 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:30:31.915 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:30:31.915 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:30:31.915 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:30:31.915 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:30:31.915 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:30:31.915 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:30:31.915 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:30:31.915 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:30:31.915 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:30:31.915 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:30:31.916 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:30:31.916 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:30:31.916 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:30:31.916 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:30:31.916 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:30:31.916 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:30:31.916 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:30:31.916 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:30:31.916 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:30:31.916 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:30:31.916 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:30:31.916 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:30:31.916 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:31.916 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:30:31.916 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:31.916 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:30:31.916 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:30:31.917 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:30:31.917 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:30:31.917 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:31.917 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:30:31.917 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:30:31.917 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:31.917 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:31.917 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:31.917 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:30:31.917 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:30:31.917 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:31.917 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:31.917 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:31.917 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:31.917 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:30:31.918 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:30:31.918 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:30:31.918 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:30:31.918 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:30:31.918 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:31.918 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:31.918 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:30:31.918 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:30:31.918 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:30:31.918 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:30:31.918 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:31.918 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:30:31.918 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:30:31.918 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:31.919 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:31.919 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:31.919 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:30:31.919 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:30:31.919 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:31.919 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:31.919 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:31.919 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:31.919 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:30:31.919 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:30:31.919 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:30:31.919 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:30:31.919 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:30:31.919 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:31.919 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:31.920 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:30:31.920 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:30:31.920 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:30:31.920 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:30:31.920 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:31.920 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:30:31.920 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:30:31.920 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:31.920 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:31.920 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:31.920 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:30:31.920 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:30:31.920 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:31.920 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:31.920 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:31.921 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:31.921 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:30:31.921 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:30:31.921 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:30:31.921 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:30:31.921 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:30:31.921 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:31.921 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:31.921 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:31.921 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:30:31.921 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:30:31.921 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:31.921 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:30:31.921 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:30:31.921 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:30:31.922 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:31.922 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:30:31.922 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:30:31.922 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:31.922 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:30:31.922 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:30:31.922 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:30:31.922 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:31.922 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:30:31.922 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:30:31.922 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:31.922 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:30:31.922 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:30:31.922 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:30:31.923 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:31.923 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:31.923 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:31.923 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:30:31.923 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:30:31.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.917+0000 7fb0dd91d640 1 -- 172.21.15.37:0/544939264 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb0c4035ed0 msgr2=0x7fb0c4038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:31.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.917+0000 7fb0dd91d640 1 --2- 172.21.15.37:0/544939264 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb0c4035ed0 0x7fb0c4038390 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb0d00ab000 tx=0x7fb0d00a6000 comp rx=0 tx=0).stop 2024-04-26T00:30:31.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.917+0000 7fb0dd91d640 10 monclient: shutdown 2024-04-26T00:30:31.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.917+0000 7fb0dd91d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:30:31.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.917+0000 7fb0dd91d640 1 -- 172.21.15.37:0/544939264 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb0d8130eb0 msgr2=0x7fb0d812ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:31.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.917+0000 7fb0d77fe640 1 -- 172.21.15.37:0/544939264 reap_dead start 2024-04-26T00:30:31.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.917+0000 7fb0dd91d640 1 --2- 172.21.15.37:0/544939264 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb0d8130eb0 0x7fb0d812ec60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc00d5f0 tx=0x7fb0cc00dab0 comp rx=0 tx=0).stop 2024-04-26T00:30:31.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.917+0000 7fb0dd91d640 1 -- 172.21.15.37:0/544939264 shutdown_connections 2024-04-26T00:30:31.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.917+0000 7fb0dd91d640 1 --2- 172.21.15.37:0/544939264 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb0d8130eb0 0x7fb0d812ec60 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:31.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.917+0000 7fb0dd91d640 1 -- 172.21.15.37:0/544939264 >> 172.21.15.37:0/544939264 conn(0x7fb0d80a1d50 msgr2=0x7fb0d81339e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:30:31.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.917+0000 7fb0dd91d640 1 -- 172.21.15.37:0/544939264 shutdown_connections 2024-04-26T00:30:31.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:31.917+0000 7fb0dd91d640 1 -- 172.21.15.37:0/544939264 wait complete. 2024-04-26T00:30:31.925 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:30:31.926 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-26T00:30:31.926 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-26T00:30:32.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f800be5e640 10 monclient: get_monmap_and_config 2024-04-26T00:30:32.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f800be5e640 10 monclient: build_initial_monmap 2024-04-26T00:30:32.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f800be5e640 10 monclient: monmap: 2024-04-26T00:30:32.037 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:30:32.037 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:32.037 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:30:32.049219+0000 2024-04-26T00:30:32.037 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:30:32.049219+0000 2024-04-26T00:30:32.037 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:30:32.037 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:32.037 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:30:32.037 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:30:32.037 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:30:32.037 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:30:32.038 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:30:32.038 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:32.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f800be5e640 1 Processor -- start 2024-04-26T00:30:32.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f800be5e640 1 -- start start 2024-04-26T00:30:32.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f800be5e640 10 monclient: init 2024-04-26T00:30:32.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f800be5e640 10 monclient: _reopen_session rank -1 2024-04-26T00:30:32.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f800be5e640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-04-26T00:30:32.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f800be5e640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f80040a0260 0x7f80040a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:32.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f800be5e640 10 monclient(hunting): picked mon.noname-a con 0x7f80040a0260 addr v2:172.21.15.37:3300/0 2024-04-26T00:30:32.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f800be5e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8004134b30 0x7f8004134f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:32.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f800be5e640 10 monclient(hunting): picked mon.noname-c con 0x7f8004134b30 addr v2:172.21.15.45:3300/0 2024-04-26T00:30:32.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f800be5e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8004135490 addr v1:172.21.15.45:6789/0 2024-04-26T00:30:32.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f800be5e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80040a1130 con 0x7f8004135490 2024-04-26T00:30:32.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f800be5e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:32.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f800be5e640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f800409ec10 con 0x7f80040a0260 2024-04-26T00:30:32.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f800be5e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:32.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f800be5e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f800409f0f0 con 0x7f8004134b30 2024-04-26T00:30:32.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f800be5e640 10 monclient(hunting): _renew_subs 2024-04-26T00:30:32.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f800be5e640 10 monclient(hunting): authenticate will time out at 1613.354636s 2024-04-26T00:30:32.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f8009bfa640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f80040a0260 0x7f80040a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:32.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f8009bfa640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f80040a0260 0x7f80040a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:43248/0 (socket says 172.21.15.37:43248) 2024-04-26T00:30:32.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f8009bfa640 1 -- 172.21.15.37:0/930452402 learned_addr learned my addr 172.21.15.37:0/930452402 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:30:32.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f8009bfa640 10 monclient(hunting): get_auth_request con 0x7f80040a0260 auth_method 0 2024-04-26T00:30:32.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f8009bfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:32.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f8009bfa640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:32.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f8009bfa640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:32.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f8009bfa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:32.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f8009bfa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:32.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f8009bfa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:32.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f80093f9640 1 --2- 172.21.15.37:0/930452402 >> v2:172.21.15.45:3300/0 conn(0x7f8004134b30 0x7f8004134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:32.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f80093f9640 10 monclient(hunting): get_auth_request con 0x7f8004134b30 auth_method 0 2024-04-26T00:30:32.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f80093f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:32.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f80093f9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:32.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f80093f9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:32.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f8009bfa640 10 monclient(hunting): handle_auth_done global_id 124122 payload 274 2024-04-26T00:30:32.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f8009bfa640 1 -- 172.21.15.37:0/930452402 >> v2:172.21.15.45:3300/0 conn(0x7f8004134b30 msgr2=0x7f8004134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:30:32.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f8009bfa640 1 --2- 172.21.15.37:0/930452402 >> v2:172.21.15.45:3300/0 conn(0x7f8004134b30 0x7f8004134f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:32.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f8009bfa640 1 -- 172.21.15.37:0/930452402 >> v1:172.21.15.45:6789/0 conn(0x7f8004135490 legacy=0x7f800413be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:30:32.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f8009bfa640 10 monclient: _finish_hunting 0 2024-04-26T00:30:32.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f8009bfa640 1 monclient: found mon.noname-a 2024-04-26T00:30:32.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f8009bfa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:30:32.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f8009bfa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:30:32.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.045+0000 7f8009bfa640 1 -- 172.21.15.37:0/930452402 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f800406cff0 con 0x7f80040a0260 2024-04-26T00:30:32.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.049+0000 7f8009bfa640 1 -- 172.21.15.37:0/930452402 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80040a0260 msgr2=0x7f80040a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:32.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.049+0000 7f8009bfa640 1 --2- 172.21.15.37:0/930452402 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80040a0260 0x7f80040a0640 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f800409f690 tx=0x7f800001c1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=856c3d8074c4dd0c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:32.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.049+0000 7f8008bf8640 1 -- 172.21.15.37:0/930452402 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80000b0070 con 0x7f80040a0260 2024-04-26T00:30:32.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.049+0000 7f8008bf8640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:32.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.049+0000 7f8008bf8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:30:32.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.049+0000 7f8008bf8640 10 monclient: dump: 2024-04-26T00:30:32.047 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:32.047 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:32.047 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:32.047 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:32.047 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:32.047 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:32.047 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:32.047 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:32.047 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:32.047 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:32.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.049+0000 7f8008bf8640 10 monclient: _finish_auth 0 2024-04-26T00:30:32.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.049+0000 7f8008bf8640 10 monclient: _check_auth_tickets 2024-04-26T00:30:32.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.049+0000 7f8008bf8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:30:32.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.049+0000 7f800be5e640 5 monclient: authenticate success, global_id 124122 2024-04-26T00:30:32.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.049+0000 7f800be5e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:30:32.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.049+0000 7f8008bf8640 1 -- 172.21.15.37:0/930452402 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80000ab600 con 0x7f80040a0260 2024-04-26T00:30:32.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.049+0000 7f8008bf8640 10 monclient: handle_config config(0 keys) 2024-04-26T00:30:32.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.049+0000 7f8008bf8640 1 -- 172.21.15.37:0/930452402 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80000ab9d0 con 0x7f80040a0260 2024-04-26T00:30:32.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.049+0000 7f8008bf8640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:32.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.049+0000 7f8008bf8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:30:32.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.049+0000 7f8008bf8640 10 monclient: dump: 2024-04-26T00:30:32.048 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:32.048 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:32.048 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:32.049 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:32.049 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:32.049 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:32.049 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:32.049 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:32.049 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:32.049 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:32.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.049+0000 7f800be5e640 10 monclient: get_monmap_and_config success 2024-04-26T00:30:32.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.049+0000 7f800be5e640 4 set_mon_vals no callback set 2024-04-26T00:30:32.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f800be5e640 10 monclient: shutdown 2024-04-26T00:30:32.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f800be5e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:30:32.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f800be5e640 1 -- 172.21.15.37:0/930452402 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80040a0260 msgr2=0x7f80040a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:32.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f800be5e640 1 --2- 172.21.15.37:0/930452402 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80040a0260 0x7f80040a0640 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f800409f690 tx=0x7f800001c1c0 comp rx=0 tx=0).stop 2024-04-26T00:30:32.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f800be5e640 1 -- 172.21.15.37:0/930452402 shutdown_connections 2024-04-26T00:30:32.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f800be5e640 1 --2- 172.21.15.37:0/930452402 >> v2:172.21.15.45:3300/0 conn(0x7f8004134b30 0x7f8004134f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:32.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f800be5e640 1 --2- 172.21.15.37:0/930452402 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80040a0260 0x7f80040a0640 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:32.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f800be5e640 1 -- 172.21.15.37:0/930452402 >> 172.21.15.37:0/930452402 conn(0x7f80040a1ad0 msgr2=0x7f80040a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:30:32.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f800be5e640 1 -- 172.21.15.37:0/930452402 shutdown_connections 2024-04-26T00:30:32.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f800be5e640 1 -- 172.21.15.37:0/930452402 wait complete. 2024-04-26T00:30:32.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f800be5e640 10 monclient: build_initial_monmap 2024-04-26T00:30:32.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f800be5e640 10 monclient: monmap: 2024-04-26T00:30:32.050 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:30:32.050 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:30:32.050 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:30:32.050 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:30:32.050 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:30:32.050 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:30:32.050 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:30:32.050 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:30:32.050 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:30:32.051 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:32.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f800be5e640 1 Processor -- start 2024-04-26T00:30:32.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f800be5e640 1 -- start start 2024-04-26T00:30:32.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f800be5e640 10 monclient: init 2024-04-26T00:30:32.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f800be5e640 10 monclient: _reopen_session rank -1 2024-04-26T00:30:32.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f800be5e640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:30:32.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f800be5e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80040a0260 0x7f80041ccba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:32.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f800be5e640 10 monclient(hunting): picked mon.noname-b con 0x7f80040a0260 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:30:32.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f800be5e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8004134b30 0x7f80041318a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:32.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f800be5e640 10 monclient(hunting): picked mon.noname-c con 0x7f8004134b30 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:30:32.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f800be5e640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8004135490 0x7f80041397d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:32.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f800be5e640 10 monclient(hunting): picked mon.noname-a con 0x7f8004135490 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:30:32.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f800be5e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:32.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f800be5e640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f800409f430 con 0x7f8004135490 2024-04-26T00:30:32.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f800be5e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:32.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f800be5e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f800409f2b0 con 0x7f80040a0260 2024-04-26T00:30:32.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f800be5e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:32.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f800be5e640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f800409fe90 con 0x7f8004134b30 2024-04-26T00:30:32.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f800be5e640 10 monclient(hunting): _renew_subs 2024-04-26T00:30:32.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f800be5e640 10 monclient(hunting): authenticate will time out at 1613.361677s 2024-04-26T00:30:32.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f80093f9640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8004134b30 0x7f80041318a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:32.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f800a3fb640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8004135490 0x7f80041397d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:32.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f80093f9640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8004134b30 0x7f80041318a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:56460/0 (socket says 172.21.15.37:56460) 2024-04-26T00:30:32.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f80093f9640 1 -- 172.21.15.37:0/429292034 learned_addr learned my addr 172.21.15.37:0/429292034 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:30:32.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f80093f9640 10 monclient(hunting): get_auth_request con 0x7f8004134b30 auth_method 0 2024-04-26T00:30:32.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f80093f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:32.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f80093f9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:32.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f80093f9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:32.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f800a3fb640 10 monclient(hunting): get_auth_request con 0x7f8004135490 auth_method 0 2024-04-26T00:30:32.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f800a3fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:32.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f800a3fb640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:32.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f800a3fb640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:32.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f80093f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:32.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f80093f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:32.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f80093f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:32.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f800a3fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:32.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f800a3fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:32.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f800a3fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:32.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f8009bfa640 1 --2- 172.21.15.37:0/429292034 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80040a0260 0x7f80041ccba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:32.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f8009bfa640 10 monclient(hunting): get_auth_request con 0x7f80040a0260 auth_method 0 2024-04-26T00:30:32.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f8009bfa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:32.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f8009bfa640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:32.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f8009bfa640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:32.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f80093f9640 10 monclient(hunting): handle_auth_done global_id 84482 payload 824 2024-04-26T00:30:32.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f80093f9640 1 -- 172.21.15.37:0/429292034 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80040a0260 msgr2=0x7f80041ccba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:32.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f80093f9640 1 --2- 172.21.15.37:0/429292034 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80040a0260 0x7f80041ccba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:32.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f80093f9640 1 -- 172.21.15.37:0/429292034 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8004135490 msgr2=0x7f80041397d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:32.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f80093f9640 1 --2- 172.21.15.37:0/429292034 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8004135490 0x7f80041397d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:32.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f80093f9640 10 monclient: _finish_hunting 0 2024-04-26T00:30:32.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f80093f9640 1 monclient: found mon.noname-c 2024-04-26T00:30:32.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f80093f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:30:32.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f80093f9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:30:32.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f80093f9640 1 -- 172.21.15.37:0/429292034 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f800411bf50 con 0x7f8004134b30 2024-04-26T00:30:32.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f800a3fb640 1 --2- 172.21.15.37:0/429292034 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8004135490 0x7f80041397d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:30:32.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.053+0000 7f8009bfa640 1 --2- 172.21.15.37:0/429292034 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80040a0260 0x7f80041ccba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:30:32.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.057+0000 7f80093f9640 1 -- 172.21.15.37:0/429292034 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8004134b30 msgr2=0x7f80041318a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:32.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.057+0000 7f80093f9640 1 --2- 172.21.15.37:0/429292034 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8004134b30 0x7f80041318a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7ff400d830 tx=0x7f7ff400dcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:32.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.057+0000 7f7ffa7fc640 1 -- 172.21.15.37:0/429292034 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ff400b8d0 con 0x7f8004134b30 2024-04-26T00:30:32.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.057+0000 7f7ffa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:32.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.057+0000 7f7ffa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:30:32.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.057+0000 7f7ffa7fc640 10 monclient: dump: 2024-04-26T00:30:32.057 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:32.057 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:32.057 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:32.057 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:32.057 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:32.057 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:32.057 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:32.057 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:32.057 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:32.057 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:32.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.057+0000 7f7ffa7fc640 10 monclient: _finish_auth 0 2024-04-26T00:30:32.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.057+0000 7f7ffa7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:30:32.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.057+0000 7f7ffa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:30:32.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.057+0000 7f7ffa7fc640 1 -- 172.21.15.37:0/429292034 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ff4019070 con 0x7f8004134b30 2024-04-26T00:30:32.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.057+0000 7f800be5e640 5 monclient: authenticate success, global_id 84482 2024-04-26T00:30:32.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.057+0000 7f800be5e640 10 monclient: _renew_subs 2024-04-26T00:30:32.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.057+0000 7f800be5e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:30:32.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.057+0000 7f800be5e640 1 -- 172.21.15.37:0/429292034 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f800411c230 con 0x7f8004134b30 2024-04-26T00:30:32.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.057+0000 7f7ffa7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:30:32.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.057+0000 7f7ffa7fc640 1 -- 172.21.15.37:0/429292034 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ff400bc70 con 0x7f8004134b30 2024-04-26T00:30:32.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.057+0000 7f7ffa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:32.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.057+0000 7f7ffa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:30:32.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.057+0000 7f7ffa7fc640 10 monclient: dump: 2024-04-26T00:30:32.058 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:32.058 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:32.058 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:32.058 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:32.058 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:32.059 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:32.059 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:32.059 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:32.059 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:32.059 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:32.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.057+0000 7f7ffbfff640 4 set_mon_vals no callback set 2024-04-26T00:30:32.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.057+0000 7f800be5e640 10 monclient: _renew_subs 2024-04-26T00:30:32.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.057+0000 7f800be5e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:30:32.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.057+0000 7f800be5e640 1 -- 172.21.15.37:0/429292034 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8004132d40 con 0x7f8004134b30 2024-04-26T00:30:32.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.057+0000 7f7ffa7fc640 1 -- 172.21.15.37:0/429292034 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f7ff401e460 con 0x7f8004134b30 2024-04-26T00:30:32.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.057+0000 7f800be5e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:30:32.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.057+0000 7f800be5e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:30:32.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.057+0000 7f800be5e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:30:32.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.057+0000 7f800be5e640 1 -- 172.21.15.37:0/429292034 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8004066f40 con 0x7f8004134b30 2024-04-26T00:30:32.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.057+0000 7f7ffa7fc640 1 --2- 172.21.15.37:0/429292034 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7fe8035ed0 0x7f7fe8038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:32.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.057+0000 7f7ffa7fc640 1 -- 172.21.15.37:0/429292034 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(402..402 src has 370..402) ==== 3975+0+0 (secure 0 0 0) 0x7f7ff4048f10 con 0x7f8004134b30 2024-04-26T00:30:32.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.057+0000 7f8009bfa640 1 --2- 172.21.15.37:0/429292034 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7fe8035ed0 0x7f7fe8038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:32.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.057+0000 7f8009bfa640 10 monclient: get_auth_request con 0x7f7fe8035ed0 auth_method 0 2024-04-26T00:30:32.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.061+0000 7f8009bfa640 1 --2- 172.21.15.37:0/429292034 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7fe8035ed0 0x7f7fe8038390 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f80040a1a60 tx=0x7f8000008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:32.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.061+0000 7f7ffa7fc640 1 -- 172.21.15.37:0/429292034 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7ff400be10 con 0x7f8004134b30 2024-04-26T00:30:32.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.061+0000 7f7ffa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:30:32.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.061+0000 7f7ffa7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:30:32.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.273+0000 7f800be5e640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-26T00:30:32.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.277+0000 7f800be5e640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-26T00:30:32.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.277+0000 7f800be5e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:30:32.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.277+0000 7f800be5e640 1 -- 172.21.15.37:0/429292034 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f80041cda20 con 0x7f8004134b30 2024-04-26T00:30:32.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.277+0000 7f7ffa7fc640 1 -- 172.21.15.37:0/429292034 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f7ff4047c30 con 0x7f8004134b30 2024-04-26T00:30:32.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.277+0000 7f7ffa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-26T00:30:32.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.277+0000 7f7ffa7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:30:32.270 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.277+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.270 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.277+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=20,config=2,health=10,kv=9,logm=59} crc {auth=2445562336,config=692619927,health=366809976,kv=377031510,logm=1380601258}) 2024-04-26T00:30:32.270 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.277+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=2,health=12,kv=9,logm=57} crc {auth=1025518602,config=692619927,health=2974672015,kv=966672740,logm=413183594}) 2024-04-26T00:30:32.270 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.277+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.270 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.277+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=20,config=2,health=10,kv=9,logm=59} crc {auth=2445562336,config=692619927,health=366809976,kv=377031510,logm=1380601258}) 2024-04-26T00:30:32.270 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.277+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=20,config=2,health=11,kv=10,logm=57} crc {auth=3647294527,config=692619927,health=271994670,kv=1034996044,logm=3505607171}) 2024-04-26T00:30:32.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.277+0000 7f800be5e640 1 -- 172.21.15.37:0/429292034 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7fe8035ed0 msgr2=0x7f7fe8038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:32.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.277+0000 7f800be5e640 1 --2- 172.21.15.37:0/429292034 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7fe8035ed0 0x7f7fe8038390 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f80040a1a60 tx=0x7f8000008040 comp rx=0 tx=0).stop 2024-04-26T00:30:32.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.277+0000 7f800be5e640 10 monclient: shutdown 2024-04-26T00:30:32.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.277+0000 7f800be5e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:30:32.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.277+0000 7f800be5e640 1 -- 172.21.15.37:0/429292034 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8004134b30 msgr2=0x7f80041318a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:32.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.277+0000 7f800be5e640 1 --2- 172.21.15.37:0/429292034 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8004134b30 0x7f80041318a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7ff400d830 tx=0x7f7ff400dcf0 comp rx=0 tx=0).stop 2024-04-26T00:30:32.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.277+0000 7f800be5e640 1 -- 172.21.15.37:0/429292034 shutdown_connections 2024-04-26T00:30:32.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.277+0000 7f800be5e640 1 --2- 172.21.15.37:0/429292034 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7fe8035ed0 0x7f7fe8038390 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:32.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.277+0000 7f800be5e640 1 --2- 172.21.15.37:0/429292034 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8004135490 0x7f80041397d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:32.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.277+0000 7f800be5e640 1 --2- 172.21.15.37:0/429292034 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8004134b30 0x7f80041318a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:32.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.277+0000 7f800be5e640 1 --2- 172.21.15.37:0/429292034 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80040a0260 0x7f80041ccba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:32.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.277+0000 7f800be5e640 1 -- 172.21.15.37:0/429292034 >> 172.21.15.37:0/429292034 conn(0x7f80040a1ad0 msgr2=0x7f80040a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:30:32.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.277+0000 7f800be5e640 1 -- 172.21.15.37:0/429292034 shutdown_connections 2024-04-26T00:30:32.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:32.281+0000 7f800be5e640 1 -- 172.21.15.37:0/429292034 wait complete. 2024-04-26T00:30:32.273 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.281+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.273 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.281+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1675321286}) 2024-04-26T00:30:32.273 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.281+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1829534184}) 2024-04-26T00:30:32.273 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.281+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.273 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.281+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1675321286}) 2024-04-26T00:30:32.273 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.281+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1490469496}) 2024-04-26T00:30:32.275 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.281+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.275 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.281+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2565772529}) 2024-04-26T00:30:32.275 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.281+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3285094028}) 2024-04-26T00:30:32.275 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.281+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.275 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.281+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2565772529}) 2024-04-26T00:30:32.275 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.281+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1734365350}) 2024-04-26T00:30:32.277 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.285+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.277 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.285+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2457101128}) 2024-04-26T00:30:32.277 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.285+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3387650650}) 2024-04-26T00:30:32.277 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.285+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.278 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.285+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2457101128}) 2024-04-26T00:30:32.278 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.285+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1592485199}) 2024-04-26T00:30:32.279 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.285+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.279 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.285+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1915474159}) 2024-04-26T00:30:32.279 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.285+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1495217965}) 2024-04-26T00:30:32.279 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.285+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.279 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.285+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1915474159}) 2024-04-26T00:30:32.280 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.285+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2408449484}) 2024-04-26T00:30:32.281 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.289+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.281 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.289+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2979668943}) 2024-04-26T00:30:32.281 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.289+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=737078957}) 2024-04-26T00:30:32.281 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.289+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.281 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.289+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2979668943}) 2024-04-26T00:30:32.282 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.289+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1253528330}) 2024-04-26T00:30:32.283 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.289+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.283 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.289+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=757030975}) 2024-04-26T00:30:32.283 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.289+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1377744043}) 2024-04-26T00:30:32.283 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.289+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.284 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.289+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=757030975}) 2024-04-26T00:30:32.284 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.289+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1478411936}) 2024-04-26T00:30:32.285 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.293+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.285 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.293+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2122666101}) 2024-04-26T00:30:32.285 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.293+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2502940862}) 2024-04-26T00:30:32.286 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.293+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.286 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.293+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2122666101}) 2024-04-26T00:30:32.286 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.293+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3749247803}) 2024-04-26T00:30:32.287 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.293+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.287 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.293+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3093615592}) 2024-04-26T00:30:32.287 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.293+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2558815262}) 2024-04-26T00:30:32.287 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.293+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.287 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.293+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3093615592}) 2024-04-26T00:30:32.287 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.293+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=541317048}) 2024-04-26T00:30:32.289 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.297+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.289 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.297+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2538447695}) 2024-04-26T00:30:32.289 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.297+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2821902731}) 2024-04-26T00:30:32.289 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.297+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.289 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.297+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2538447695}) 2024-04-26T00:30:32.289 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.297+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3462730550}) 2024-04-26T00:30:32.290 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-26T00:30:32.291 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.297+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.291 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.297+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2916763550}) 2024-04-26T00:30:32.291 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.297+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1443816934}) 2024-04-26T00:30:32.291 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.297+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.291 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.297+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2916763550}) 2024-04-26T00:30:32.291 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.297+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4080928847}) 2024-04-26T00:30:32.293 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.301+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.293 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.301+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4003367787}) 2024-04-26T00:30:32.293 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.301+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3232757556}) 2024-04-26T00:30:32.293 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.301+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.293 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.301+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4003367787}) 2024-04-26T00:30:32.293 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.301+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2587351938}) 2024-04-26T00:30:32.295 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.301+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.295 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.301+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4056166480}) 2024-04-26T00:30:32.295 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.301+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2935923443}) 2024-04-26T00:30:32.295 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.301+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.295 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.301+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4056166480}) 2024-04-26T00:30:32.295 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.301+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4290542894}) 2024-04-26T00:30:32.297 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.305+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.297 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.305+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3894660029}) 2024-04-26T00:30:32.297 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.305+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1799501802}) 2024-04-26T00:30:32.297 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.305+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.297 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.305+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3894660029}) 2024-04-26T00:30:32.297 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.305+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1250878851}) 2024-04-26T00:30:32.299 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.305+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.299 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.305+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3565091382}) 2024-04-26T00:30:32.299 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.305+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4289974781}) 2024-04-26T00:30:32.299 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.305+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.299 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.305+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3565091382}) 2024-04-26T00:30:32.299 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.305+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1953891557}) 2024-04-26T00:30:32.301 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.309+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.301 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.309+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=570973519}) 2024-04-26T00:30:32.301 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.309+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=582423133}) 2024-04-26T00:30:32.301 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.309+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.302 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.309+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=570973519}) 2024-04-26T00:30:32.302 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.309+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3861235696}) 2024-04-26T00:30:32.303 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.313+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.303 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.313+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=75014612}) 2024-04-26T00:30:32.303 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.313+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1595184914}) 2024-04-26T00:30:32.304 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.313+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.304 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.313+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=75014612}) 2024-04-26T00:30:32.304 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.313+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1074753536}) 2024-04-26T00:30:32.305 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.313+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.305 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.313+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1729596065}) 2024-04-26T00:30:32.305 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.313+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2339324634}) 2024-04-26T00:30:32.305 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.313+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.305 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.313+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1729596065}) 2024-04-26T00:30:32.306 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.313+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3539926785}) 2024-04-26T00:30:32.307 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.313+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.307 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.313+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2256574455}) 2024-04-26T00:30:32.308 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.317+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=802971084}) 2024-04-26T00:30:32.308 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.317+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.308 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.317+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2256574455}) 2024-04-26T00:30:32.308 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.317+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3585108438}) 2024-04-26T00:30:32.309 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.317+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.309 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.317+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3100301026}) 2024-04-26T00:30:32.309 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.317+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2175389567}) 2024-04-26T00:30:32.309 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.317+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.310 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.317+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3100301026}) 2024-04-26T00:30:32.310 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.317+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=297824486}) 2024-04-26T00:30:32.311 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.317+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.311 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.317+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=448510263}) 2024-04-26T00:30:32.311 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.317+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3453288512}) 2024-04-26T00:30:32.312 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.317+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.312 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.317+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=448510263}) 2024-04-26T00:30:32.312 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.317+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=792881581}) 2024-04-26T00:30:32.314 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.321+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.314 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.321+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=182173863}) 2024-04-26T00:30:32.314 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.321+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4270845551}) 2024-04-26T00:30:32.314 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.321+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.314 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.321+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=182173863}) 2024-04-26T00:30:32.314 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.321+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1689514848}) 2024-04-26T00:30:32.316 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.325+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.316 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.325+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3321123856}) 2024-04-26T00:30:32.316 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.325+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=955180401}) 2024-04-26T00:30:32.316 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.325+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.316 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.325+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3321123856}) 2024-04-26T00:30:32.316 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.325+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1277806334}) 2024-04-26T00:30:32.318 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.325+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.318 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.325+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3436121564}) 2024-04-26T00:30:32.318 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.325+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3853267856}) 2024-04-26T00:30:32.318 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.325+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.318 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.325+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3436121564}) 2024-04-26T00:30:32.318 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.325+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2818349867}) 2024-04-26T00:30:32.320 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.329+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.320 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.329+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1748956513}) 2024-04-26T00:30:32.321 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.329+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1816091738}) 2024-04-26T00:30:32.321 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.329+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.321 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.329+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1748956513}) 2024-04-26T00:30:32.321 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.329+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2797431024}) 2024-04-26T00:30:32.322 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.329+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.322 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.329+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2261550571}) 2024-04-26T00:30:32.323 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.329+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=449266871}) 2024-04-26T00:30:32.323 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.329+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.323 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.329+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2261550571}) 2024-04-26T00:30:32.323 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.329+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=749866652}) 2024-04-26T00:30:32.359 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.365+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.359 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.365+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2168600781}) 2024-04-26T00:30:32.359 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.365+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3706513661}) 2024-04-26T00:30:32.359 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.365+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.359 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.365+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2168600781}) 2024-04-26T00:30:32.359 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.365+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=660105943}) 2024-04-26T00:30:32.407 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.413+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.407 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.413+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3915076733}) 2024-04-26T00:30:32.407 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.413+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=856359234}) 2024-04-26T00:30:32.407 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.413+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.407 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.413+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3915076733}) 2024-04-26T00:30:32.407 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.413+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=106633060}) 2024-04-26T00:30:32.409 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.417+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.409 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.417+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=847356671}) 2024-04-26T00:30:32.409 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.417+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3724421068}) 2024-04-26T00:30:32.409 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.417+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.409 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.417+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=847356671}) 2024-04-26T00:30:32.409 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.417+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2179710837}) 2024-04-26T00:30:32.411 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.417+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.411 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.417+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3070627442}) 2024-04-26T00:30:32.411 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.417+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1895041631}) 2024-04-26T00:30:32.411 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.417+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.411 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.417+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3070627442}) 2024-04-26T00:30:32.411 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.417+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1388630196}) 2024-04-26T00:30:32.414 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.421+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.414 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.421+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=28,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=27} crc {logm=1838840501,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=2646532840,mon_config_key=4061063117,monmap=3876866194,osd_metadata=1025741062,osd_pg_creating=1165943392,osd_snap=3694984898}) 2024-04-26T00:30:32.414 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.421+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=28,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=27} crc {logm=3799061086,mdsmap=2840952586,mgr=1137150072,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=2646532840,mon_config_key=4061063117,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=1165943392,osd_snap=4083665013}) 2024-04-26T00:30:32.414 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.421+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.414 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.421+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=28,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=27} crc {logm=1838840501,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=2646532840,mon_config_key=4061063117,monmap=3876866194,osd_metadata=1025741062,osd_pg_creating=1165943392,osd_snap=3694984898}) 2024-04-26T00:30:32.414 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.421+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=28,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=28} crc {logm=1622456748,mdsmap=2840952586,mgr=4237326694,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=2776070140,mon_config_key=752439442,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=1165943392,osd_snap=1923760120}) 2024-04-26T00:30:32.417 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.425+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.417 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.425+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=43,osdmap=57} crc {osd_snap=3984506122,osdmap=699469681}) 2024-04-26T00:30:32.417 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.425+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=43,osdmap=57} crc {osd_snap=1158793406,osdmap=1716752092}) 2024-04-26T00:30:32.417 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.425+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:30:32.417 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.425+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=43,osdmap=57} crc {osd_snap=3984506122,osdmap=699469681}) 2024-04-26T00:30:32.417 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:30:32.425+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=44,osdmap=56} crc {osd_snap=4238288766,osdmap=4175940265}) 2024-04-26T00:30:32.495 INFO:tasks.rados.rados.0.smithi045.stdout:1762: done (3 left) 2024-04-26T00:30:32.659 INFO:tasks.rados.rados.0.smithi045.stdout:1763: write (excl) oid 43 current snap is 202 2024-04-26T00:30:32.659 INFO:tasks.rados.rados.0.smithi045.stdout:1760: finishing rollback tid 0 to smithi04526579-18 2024-04-26T00:30:32.659 INFO:tasks.rados.rados.0.smithi045.stdout:1760: finishing rollback tid 1 to smithi04526579-18 2024-04-26T00:30:32.659 INFO:tasks.rados.rados.0.smithi045.stdout:1760: finishing rollback tid 2 to smithi04526579-18 2024-04-26T00:30:32.659 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 1429 (ObjNum 551 snap 168 seq_num 551) dirty exists 2024-04-26T00:30:32.659 INFO:tasks.rados.rados.0.smithi045.stdout:1759: finishing write tid 1 to smithi04526579-1 2024-04-26T00:30:32.659 INFO:tasks.rados.rados.0.smithi045.stdout:1759: finishing write tid 2 to smithi04526579-1 2024-04-26T00:30:32.659 INFO:tasks.rados.rados.0.smithi045.stdout:1761: finishing rollback tid 0 to smithi04526579-10 2024-04-26T00:30:32.659 INFO:tasks.rados.rados.0.smithi045.stdout:1761: finishing rollback tid 1 to smithi04526579-10 2024-04-26T00:30:32.659 INFO:tasks.rados.rados.0.smithi045.stdout:1759: finishing write tid 3 to smithi04526579-1 2024-04-26T00:30:32.659 INFO:tasks.rados.rados.0.smithi045.stdout:1761: finishing rollback tid 2 to smithi04526579-10 2024-04-26T00:30:32.659 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 1055 (ObjNum 573 snap 176 seq_num 573) dirty exists 2024-04-26T00:30:32.660 INFO:tasks.rados.rados.0.smithi045.stdout:1759: finishing write tid 4 to smithi04526579-1 2024-04-26T00:30:32.660 INFO:tasks.rados.rados.0.smithi045.stdout:1759: oid 1 updating version 0 to 1120 2024-04-26T00:30:32.660 INFO:tasks.rados.rados.0.smithi045.stdout:1759: oid 1 version 1120 is already newer than 1119 2024-04-26T00:30:32.660 INFO:tasks.rados.rados.0.smithi045.stdout:1759: oid 1 version 1120 is already newer than 1118 2024-04-26T00:30:32.660 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 1120 (ObjNum 645 snap 202 seq_num 645) dirty exists 2024-04-26T00:30:32.660 INFO:tasks.rados.rados.0.smithi045.stdout:1759: left oid 1 (ObjNum 645 snap 202 seq_num 645) 2024-04-26T00:30:32.660 INFO:tasks.rados.rados.0.smithi045.stdout:1763: seq_num 646 ranges {457322=784506,1969250=443556} 2024-04-26T00:30:32.662 INFO:tasks.rados.rados.0.smithi045.stdout:1763: writing smithi04526579-43 from 457322 to 1241828 tid 1 2024-04-26T00:30:32.662 INFO:tasks.rados.rados.0.smithi045.stdout:1763: writing smithi04526579-43 from 1969250 to 2412806 tid 2 2024-04-26T00:30:32.662 INFO:tasks.rados.rados.0.smithi045.stdout:1759: done (3 left) 2024-04-26T00:30:32.662 INFO:tasks.rados.rados.0.smithi045.stdout:1760: done (2 left) 2024-04-26T00:30:32.662 INFO:tasks.rados.rados.0.smithi045.stdout:1761: done (1 left) 2024-04-26T00:30:32.662 INFO:tasks.rados.rados.0.smithi045.stdout:1764: read oid 39 snap -1 2024-04-26T00:30:32.662 INFO:tasks.rados.rados.0.smithi045.stdout:1764: expect (ObjNum 508 snap 157 seq_num 508) 2024-04-26T00:30:32.663 INFO:tasks.rados.rados.0.smithi045.stdout:1765: read oid 3 snap -1 2024-04-26T00:30:32.663 INFO:tasks.rados.rados.0.smithi045.stdout:1765: expect deleted 2024-04-26T00:30:32.663 INFO:tasks.rados.rados.0.smithi045.stdout:1766: delete oid 11 current snap is 202 2024-04-26T00:30:32.663 INFO:tasks.rados.rados.0.smithi045.stdout:1765: done (3 left) 2024-04-26T00:30:32.663 INFO:tasks.rados.rados.0.smithi045.stdout:1766: done (2 left) 2024-04-26T00:30:32.663 INFO:tasks.rados.rados.0.smithi045.stdout:1767: write (excl) oid 46 current snap is 202 2024-04-26T00:30:32.663 INFO:tasks.rados.rados.0.smithi045.stdout:1767: seq_num 648 ranges {490936=651357,1686214=447225,2861864=29072} 2024-04-26T00:30:32.663 INFO:tasks.rados.rados.0.smithi045.stdout:1767: writing smithi04526579-46 from 490936 to 1142293 tid 1 2024-04-26T00:30:32.663 INFO:tasks.rados.rados.0.smithi045.stdout:1767: writing smithi04526579-46 from 1686214 to 2133439 tid 2 2024-04-26T00:30:32.663 INFO:tasks.rados.rados.0.smithi045.stdout:1767: writing smithi04526579-46 from 2861864 to 2890936 tid 3 2024-04-26T00:30:32.663 INFO:tasks.rados.rados.0.smithi045.stdout:1768: read oid 31 snap -1 2024-04-26T00:30:32.663 INFO:tasks.rados.rados.0.smithi045.stdout:1768: expect (ObjNum 644 snap 202 seq_num 644) 2024-04-26T00:30:32.663 INFO:tasks.rados.rados.0.smithi045.stdout:1769: write oid 33 current snap is 202 2024-04-26T00:30:32.663 INFO:tasks.rados.rados.0.smithi045.stdout:1764: expect (ObjNum 508 snap 157 seq_num 508) 2024-04-26T00:30:32.686 INFO:tasks.rados.rados.0.smithi045.stdout:1763: finishing write tid 1 to smithi04526579-43 2024-04-26T00:30:32.686 INFO:tasks.rados.rados.0.smithi045.stdout:1763: finishing write tid 2 to smithi04526579-43 2024-04-26T00:30:32.686 INFO:tasks.rados.rados.0.smithi045.stdout:1763: finishing write tid 3 to smithi04526579-43 2024-04-26T00:30:32.686 INFO:tasks.rados.rados.0.smithi045.stdout:1769: seq_num 649 ranges {507743=726606,1744696=590883,2772934=134809} 2024-04-26T00:30:32.692 INFO:tasks.rados.rados.0.smithi045.stdout:1769: writing smithi04526579-33 from 507743 to 1234349 tid 1 2024-04-26T00:30:32.699 INFO:tasks.rados.rados.0.smithi045.stdout:1769: writing smithi04526579-33 from 1744696 to 2335579 tid 2 2024-04-26T00:30:32.702 INFO:tasks.rados.rados.0.smithi045.stdout:1769: writing smithi04526579-33 from 2772934 to 2907743 tid 3 2024-04-26T00:30:32.702 INFO:tasks.rados.rados.0.smithi045.stdout:1764: done (4 left) 2024-04-26T00:30:32.702 INFO:tasks.rados.rados.0.smithi045.stdout:1770: rollback oid 20 current snap is 202 2024-04-26T00:30:32.702 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 20 to 169 2024-04-26T00:30:32.702 INFO:tasks.rados.rados.0.smithi045.stdout:1763: finishing write tid 4 to smithi04526579-43 2024-04-26T00:30:32.703 INFO:tasks.rados.rados.0.smithi045.stdout:1763: oid 43 updating version 0 to 1058 2024-04-26T00:30:32.703 INFO:tasks.rados.rados.0.smithi045.stdout:1763: oid 43 version 1058 is already newer than 1057 2024-04-26T00:30:32.703 INFO:tasks.rados.rados.0.smithi045.stdout:1763: oid 43 version 1058 is already newer than 1056 2024-04-26T00:30:32.703 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 1058 (ObjNum 646 snap 202 seq_num 646) dirty exists 2024-04-26T00:30:32.703 INFO:tasks.rados.rados.0.smithi045.stdout:1763: left oid 43 (ObjNum 646 snap 202 seq_num 646) 2024-04-26T00:30:32.703 INFO:tasks.rados.rados.0.smithi045.stdout:1767: finishing write tid 1 to smithi04526579-46 2024-04-26T00:30:32.703 INFO:tasks.rados.rados.0.smithi045.stdout:1767: finishing write tid 2 to smithi04526579-46 2024-04-26T00:30:32.703 INFO:tasks.rados.rados.0.smithi045.stdout:1767: finishing write tid 3 to smithi04526579-46 2024-04-26T00:30:32.703 INFO:tasks.rados.rados.0.smithi045.stdout:1767: finishing write tid 4 to smithi04526579-46 2024-04-26T00:30:32.703 INFO:tasks.rados.rados.0.smithi045.stdout:1767: finishing write tid 5 to smithi04526579-46 2024-04-26T00:30:32.703 INFO:tasks.rados.rados.0.smithi045.stdout:1767: oid 46 updating version 0 to 784 2024-04-26T00:30:32.703 INFO:tasks.rados.rados.0.smithi045.stdout:1767: oid 46 version 784 is already newer than 782 2024-04-26T00:30:32.703 INFO:tasks.rados.rados.0.smithi045.stdout:1767: oid 46 updating version 784 to 785 2024-04-26T00:30:32.703 INFO:tasks.rados.rados.0.smithi045.stdout:1767: oid 46 version 785 is already newer than 783 2024-04-26T00:30:32.703 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 785 (ObjNum 648 snap 202 seq_num 648) dirty exists 2024-04-26T00:30:32.704 INFO:tasks.rados.rados.0.smithi045.stdout:1767: left oid 46 (ObjNum 648 snap 202 seq_num 648) 2024-04-26T00:30:32.704 INFO:tasks.rados.rados.0.smithi045.stdout:1763: done (4 left) 2024-04-26T00:30:32.704 INFO:tasks.rados.rados.0.smithi045.stdout:1767: done (3 left) 2024-04-26T00:30:32.704 INFO:tasks.rados.rados.0.smithi045.stdout:1771: delete oid 42 current snap is 202 2024-04-26T00:30:32.704 INFO:tasks.rados.rados.0.smithi045.stdout:1768: expect (ObjNum 644 snap 202 seq_num 644) 2024-04-26T00:30:32.796 INFO:tasks.rados.rados.0.smithi045.stdout:1769: finishing write tid 1 to smithi04526579-33 2024-04-26T00:30:32.796 INFO:tasks.rados.rados.0.smithi045.stdout:1769: finishing write tid 2 to smithi04526579-33 2024-04-26T00:30:32.796 INFO:tasks.rados.rados.0.smithi045.stdout:1770: finishing rollback tid 0 to smithi04526579-20 2024-04-26T00:30:32.796 INFO:tasks.rados.rados.0.smithi045.stdout:1769: finishing write tid 3 to smithi04526579-33 2024-04-26T00:30:32.796 INFO:tasks.rados.rados.0.smithi045.stdout:1768: done (3 left) 2024-04-26T00:30:32.796 INFO:tasks.rados.rados.0.smithi045.stdout:1771: done (2 left) 2024-04-26T00:30:32.796 INFO:tasks.rados.rados.0.smithi045.stdout:1772: write oid 37 current snap is 202 2024-04-26T00:30:32.796 INFO:tasks.rados.rados.0.smithi045.stdout:1770: finishing rollback tid 1 to smithi04526579-20 2024-04-26T00:30:32.796 INFO:tasks.rados.rados.0.smithi045.stdout:1769: finishing write tid 4 to smithi04526579-33 2024-04-26T00:30:32.796 INFO:tasks.rados.rados.0.smithi045.stdout:1770: finishing rollback tid 2 to smithi04526579-20 2024-04-26T00:30:32.797 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 1064 (ObjNum 542 snap 167 seq_num 542) dirty exists 2024-04-26T00:30:32.797 INFO:tasks.rados.rados.0.smithi045.stdout:1769: finishing write tid 5 to smithi04526579-33 2024-04-26T00:30:32.797 INFO:tasks.rados.rados.0.smithi045.stdout:1769: oid 33 updating version 0 to 1443 2024-04-26T00:30:32.797 INFO:tasks.rados.rados.0.smithi045.stdout:1769: oid 33 updating version 1443 to 1444 2024-04-26T00:30:32.797 INFO:tasks.rados.rados.0.smithi045.stdout:1769: oid 33 updating version 1444 to 1446 2024-04-26T00:30:32.797 INFO:tasks.rados.rados.0.smithi045.stdout:1769: oid 33 version 1446 is already newer than 1445 2024-04-26T00:30:32.798 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 1446 (ObjNum 649 snap 202 seq_num 649) dirty exists 2024-04-26T00:30:32.798 INFO:tasks.rados.rados.0.smithi045.stdout:1769: left oid 33 (ObjNum 649 snap 202 seq_num 649) 2024-04-26T00:30:32.798 INFO:tasks.rados.rados.0.smithi045.stdout:1772: seq_num 651 ranges {541357=477104,1745307=594552} 2024-04-26T00:30:32.801 INFO:tasks.rados.rados.0.smithi045.stdout:1772: writing smithi04526579-37 from 541357 to 1018461 tid 1 2024-04-26T00:30:32.809 INFO:tasks.rados.rados.0.smithi045.stdout:1772: writing smithi04526579-37 from 1745307 to 2339859 tid 2 2024-04-26T00:30:32.809 INFO:tasks.rados.rados.0.smithi045.stdout:1769: done (2 left) 2024-04-26T00:30:32.809 INFO:tasks.rados.rados.0.smithi045.stdout:1770: done (1 left) 2024-04-26T00:30:32.809 INFO:tasks.rados.rados.0.smithi045.stdout:1773: write oid 46 current snap is 202 2024-04-26T00:30:32.809 INFO:tasks.rados.rados.0.smithi045.stdout:1773: seq_num 652 ranges {558164=552353,1803789=454563,2855203=102961} 2024-04-26T00:30:32.815 INFO:tasks.rados.rados.0.smithi045.stdout:1773: writing smithi04526579-46 from 558164 to 1110517 tid 1 2024-04-26T00:30:32.824 INFO:tasks.rados.rados.0.smithi045.stdout:1773: writing smithi04526579-46 from 1803789 to 2258352 tid 2 2024-04-26T00:30:32.829 INFO:tasks.rados.rados.0.smithi045.stdout:1773: writing smithi04526579-46 from 2855203 to 2958164 tid 3 2024-04-26T00:30:32.829 INFO:tasks.rados.rados.0.smithi045.stdout:1774: read oid 18 snap -1 2024-04-26T00:30:32.830 INFO:tasks.rados.rados.0.smithi045.stdout:1774: expect (ObjNum 551 snap 168 seq_num 551) 2024-04-26T00:30:32.830 INFO:tasks.rados.rados.0.smithi045.stdout:1775: read oid 27 snap 160 2024-04-26T00:30:32.830 INFO:tasks.rados.rados.0.smithi045.stdout:1775: expect (ObjNum 482 snap 150 seq_num 482) 2024-04-26T00:30:32.830 INFO:tasks.rados.rados.0.smithi045.stdout:1776: write (excl) oid 2 current snap is 202 2024-04-26T00:30:32.830 INFO:tasks.rados.rados.0.smithi045.stdout:1776: seq_num 653 ranges {574971=627602,1862271=598221} 2024-04-26T00:30:32.840 INFO:tasks.rados.rados.0.smithi045.stdout:1776: writing smithi04526579-2 from 574971 to 1202573 tid 1 2024-04-26T00:30:32.853 INFO:tasks.rados.rados.0.smithi045.stdout:1776: writing smithi04526579-2 from 1862271 to 2460492 tid 2 2024-04-26T00:30:32.853 INFO:tasks.rados.rados.0.smithi045.stdout:1777: write oid 50 current snap is 202 2024-04-26T00:30:32.853 INFO:tasks.rados.rados.0.smithi045.stdout:1777: seq_num 654 ranges {591778=419204,1637106=741879,2910049=81729} 2024-04-26T00:30:32.860 INFO:tasks.rados.rados.0.smithi045.stdout:1777: writing smithi04526579-50 from 591778 to 1010982 tid 1 2024-04-26T00:30:32.870 INFO:tasks.rados.rados.0.smithi045.stdout:1777: writing smithi04526579-50 from 1637106 to 2378985 tid 2 2024-04-26T00:30:32.873 INFO:tasks.rados.rados.0.smithi045.stdout:1777: writing smithi04526579-50 from 2910049 to 2991778 tid 3 2024-04-26T00:30:32.873 INFO:tasks.rados.rados.0.smithi045.stdout:1778: write (excl) oid 28 current snap is 202 2024-04-26T00:30:32.873 INFO:tasks.rados.rados.0.smithi045.stdout:1778: seq_num 655 ranges {608585=494453,1579235=601890,2821119=187466} 2024-04-26T00:30:32.879 INFO:tasks.rados.rados.0.smithi045.stdout:1778: writing smithi04526579-28 from 608585 to 1103038 tid 1 2024-04-26T00:30:32.886 INFO:tasks.rados.rados.0.smithi045.stdout:1778: writing smithi04526579-28 from 1579235 to 2181125 tid 2 2024-04-26T00:30:32.891 INFO:tasks.rados.rados.0.smithi045.stdout:1778: writing smithi04526579-28 from 2821119 to 3008585 tid 3 2024-04-26T00:30:32.891 INFO:tasks.rados.rados.0.smithi045.stdout:1779: read oid 49 snap 179 2024-04-26T00:30:32.891 INFO:tasks.rados.rados.0.smithi045.stdout:1779: expect (ObjNum 543 snap 167 seq_num 543) 2024-04-26T00:30:32.892 INFO:tasks.rados.rados.0.smithi045.stdout:1780: copy_from oid 15 from oid 29 current snap is 202 2024-04-26T00:30:32.892 INFO:tasks.rados.rados.0.smithi045.stdout:1772: finishing write tid 1 to smithi04526579-37 2024-04-26T00:30:32.892 INFO:tasks.rados.rados.0.smithi045.stdout:1772: finishing write tid 2 to smithi04526579-37 2024-04-26T00:30:32.892 INFO:tasks.rados.rados.0.smithi045.stdout:1772: finishing write tid 3 to smithi04526579-37 2024-04-26T00:30:32.892 INFO:tasks.rados.rados.0.smithi045.stdout:1772: finishing write tid 4 to smithi04526579-37 2024-04-26T00:30:32.892 INFO:tasks.rados.rados.0.smithi045.stdout:1772: oid 37 updating version 0 to 1449 2024-04-26T00:30:32.892 INFO:tasks.rados.rados.0.smithi045.stdout:1772: oid 37 updating version 1449 to 1450 2024-04-26T00:30:32.892 INFO:tasks.rados.rados.0.smithi045.stdout:1772: oid 37 version 1450 is already newer than 1448 2024-04-26T00:30:32.892 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 1450 (ObjNum 651 snap 202 seq_num 651) dirty exists 2024-04-26T00:30:32.892 INFO:tasks.rados.rados.0.smithi045.stdout:1772: left oid 37 (ObjNum 651 snap 202 seq_num 651) 2024-04-26T00:30:32.892 INFO:tasks.rados.rados.0.smithi045.stdout:1773: finishing write tid 1 to smithi04526579-46 2024-04-26T00:30:32.892 INFO:tasks.rados.rados.0.smithi045.stdout:1773: finishing write tid 2 to smithi04526579-46 2024-04-26T00:30:32.892 INFO:tasks.rados.rados.0.smithi045.stdout:1773: finishing write tid 3 to smithi04526579-46 2024-04-26T00:30:32.892 INFO:tasks.rados.rados.0.smithi045.stdout:1773: finishing write tid 4 to smithi04526579-46 2024-04-26T00:30:32.892 INFO:tasks.rados.rados.0.smithi045.stdout:1772: done (8 left) 2024-04-26T00:30:32.893 INFO:tasks.rados.rados.0.smithi045.stdout:1781: copy_from oid 35 from oid 12 current snap is 202 2024-04-26T00:30:32.893 INFO:tasks.rados.rados.0.smithi045.stdout:1773: finishing write tid 5 to smithi04526579-46 2024-04-26T00:30:32.893 INFO:tasks.rados.rados.0.smithi045.stdout:1773: oid 46 updating version 0 to 788 2024-04-26T00:30:32.893 INFO:tasks.rados.rados.0.smithi045.stdout:1773: oid 46 version 788 is already newer than 786 2024-04-26T00:30:32.893 INFO:tasks.rados.rados.0.smithi045.stdout:1773: oid 46 version 788 is already newer than 787 2024-04-26T00:30:32.893 INFO:tasks.rados.rados.0.smithi045.stdout:1773: oid 46 updating version 788 to 789 2024-04-26T00:30:32.893 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 789 (ObjNum 652 snap 202 seq_num 652) dirty exists 2024-04-26T00:30:32.893 INFO:tasks.rados.rados.0.smithi045.stdout:1773: left oid 46 (ObjNum 652 snap 202 seq_num 652) 2024-04-26T00:30:32.893 INFO:tasks.rados.rados.0.smithi045.stdout:1775: expect (ObjNum 482 snap 150 seq_num 482) 2024-04-26T00:30:33.044 INFO:tasks.rados.rados.0.smithi045.stdout:1776: finishing write tid 1 to smithi04526579-2 2024-04-26T00:30:33.045 INFO:tasks.rados.rados.0.smithi045.stdout:1774: expect (ObjNum 551 snap 168 seq_num 551) 2024-04-26T00:30:33.184 INFO:tasks.rados.rados.0.smithi045.stdout:1776: finishing write tid 2 to smithi04526579-2 2024-04-26T00:30:33.184 INFO:tasks.rados.rados.0.smithi045.stdout:1776: finishing write tid 3 to smithi04526579-2 2024-04-26T00:30:33.184 INFO:tasks.rados.rados.0.smithi045.stdout:1776: finishing write tid 4 to smithi04526579-2 2024-04-26T00:30:33.184 INFO:tasks.rados.rados.0.smithi045.stdout:1776: oid 2 updating version 0 to 986 2024-04-26T00:30:33.184 INFO:tasks.rados.rados.0.smithi045.stdout:1776: oid 2 updating version 986 to 987 2024-04-26T00:30:33.185 INFO:tasks.rados.rados.0.smithi045.stdout:1776: oid 2 version 987 is already newer than 985 2024-04-26T00:30:33.185 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 987 (ObjNum 653 snap 202 seq_num 653) dirty exists 2024-04-26T00:30:33.185 INFO:tasks.rados.rados.0.smithi045.stdout:1776: left oid 2 (ObjNum 653 snap 202 seq_num 653) 2024-04-26T00:30:33.185 INFO:tasks.rados.rados.0.smithi045.stdout:1777: finishing write tid 1 to smithi04526579-50 2024-04-26T00:30:33.185 INFO:tasks.rados.rados.0.smithi045.stdout:1777: finishing write tid 2 to smithi04526579-50 2024-04-26T00:30:33.185 INFO:tasks.rados.rados.0.smithi045.stdout:1773: done (8 left) 2024-04-26T00:30:33.185 INFO:tasks.rados.rados.0.smithi045.stdout:1774: done (7 left) 2024-04-26T00:30:33.185 INFO:tasks.rados.rados.0.smithi045.stdout:1775: done (6 left) 2024-04-26T00:30:33.185 INFO:tasks.rados.rados.0.smithi045.stdout:1776: done (5 left) 2024-04-26T00:30:33.185 INFO:tasks.rados.rados.0.smithi045.stdout:1782: snap_remove snap 161 2024-04-26T00:30:33.291 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:30:33.291 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:30:33.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a24c6e640 10 monclient: get_monmap_and_config 2024-04-26T00:30:33.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a24c6e640 10 monclient: build_initial_monmap 2024-04-26T00:30:33.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a24c6e640 10 monclient: monmap: 2024-04-26T00:30:33.425 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:30:33.425 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:33.425 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:30:33.437028+0000 2024-04-26T00:30:33.425 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:30:33.437028+0000 2024-04-26T00:30:33.425 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:30:33.425 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:33.425 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:30:33.425 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:30:33.425 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:30:33.425 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:30:33.425 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:30:33.425 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:33.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a24c6e640 1 Processor -- start 2024-04-26T00:30:33.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a24c6e640 1 -- start start 2024-04-26T00:30:33.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a24c6e640 10 monclient: init 2024-04-26T00:30:33.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a24c6e640 10 monclient: _reopen_session rank -1 2024-04-26T00:30:33.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a24c6e640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-04-26T00:30:33.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a24c6e640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f8a200a0260 0x7f8a200a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:33.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a24c6e640 10 monclient(hunting): picked mon.noname-a con 0x7f8a200a0260 addr v2:172.21.15.37:3300/0 2024-04-26T00:30:33.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a24c6e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a20134b30 0x7f8a20134f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:33.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a24c6e640 10 monclient(hunting): picked mon.noname-b con 0x7f8a20134b30 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:30:33.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a24c6e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8a20135490 0x7f8a2013be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:33.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a24c6e640 10 monclient(hunting): picked mon.noname-c con 0x7f8a20135490 addr v2:172.21.15.45:3300/0 2024-04-26T00:30:33.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a24c6e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:33.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a24c6e640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f8a2009ec10 con 0x7f8a200a0260 2024-04-26T00:30:33.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a24c6e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:33.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a24c6e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f8a2009f0f0 con 0x7f8a20135490 2024-04-26T00:30:33.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a24c6e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:33.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a24c6e640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8a200a1130 con 0x7f8a20134b30 2024-04-26T00:30:33.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a24c6e640 10 monclient(hunting): _renew_subs 2024-04-26T00:30:33.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a24c6e640 10 monclient(hunting): authenticate will time out at 1614.742568s 2024-04-26T00:30:33.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a1e59c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f8a200a0260 0x7f8a200a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:33.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a1dd9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a20134b30 0x7f8a20134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:33.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a1e59c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f8a200a0260 0x7f8a200a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:43274/0 (socket says 172.21.15.37:43274) 2024-04-26T00:30:33.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a1e59c640 1 -- 172.21.15.37:0/13462393 learned_addr learned my addr 172.21.15.37:0/13462393 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:30:33.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a1e59c640 10 monclient(hunting): get_auth_request con 0x7f8a200a0260 auth_method 0 2024-04-26T00:30:33.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a1dd9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a20134b30 0x7f8a20134f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:56464/0 (socket says 172.21.15.37:56464) 2024-04-26T00:30:33.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a1e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:33.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a1e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:33.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a1e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:33.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a1dd9b640 10 monclient(hunting): get_auth_request con 0x7f8a20134b30 auth_method 0 2024-04-26T00:30:33.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a1dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:33.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a1dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:33.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a1dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:33.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a1e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:33.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a1e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:33.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a1e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:33.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a1dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:33.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a1dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:33.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a1dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:33.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a1ed9d640 1 --2- 172.21.15.37:0/13462393 >> v2:172.21.15.45:3300/0 conn(0x7f8a20135490 0x7f8a2013be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:33.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a1e59c640 10 monclient(hunting): handle_auth_done global_id 124128 payload 274 2024-04-26T00:30:33.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a1e59c640 1 -- 172.21.15.37:0/13462393 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a20134b30 msgr2=0x7f8a20134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:30:33.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a1e59c640 1 --2- 172.21.15.37:0/13462393 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a20134b30 0x7f8a20134f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:33.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a1e59c640 1 -- 172.21.15.37:0/13462393 >> v2:172.21.15.45:3300/0 conn(0x7f8a20135490 msgr2=0x7f8a2013be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:30:33.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a1e59c640 1 --2- 172.21.15.37:0/13462393 >> v2:172.21.15.45:3300/0 conn(0x7f8a20135490 0x7f8a2013be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:33.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a1e59c640 10 monclient: _finish_hunting 0 2024-04-26T00:30:33.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a1e59c640 1 monclient: found mon.noname-a 2024-04-26T00:30:33.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a1e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:30:33.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a1e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:30:33.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a1e59c640 1 -- 172.21.15.37:0/13462393 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a2006cff0 con 0x7f8a200a0260 2024-04-26T00:30:33.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a1ed9d640 10 monclient: get_auth_request con 0x7f8a20135490 auth_method 0 2024-04-26T00:30:33.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a1ed9d640 1 --2- 172.21.15.37:0/13462393 >> v2:172.21.15.45:3300/0 conn(0x7f8a20135490 0x7f8a2013be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:30:33.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.433+0000 7f8a1dd9b640 1 --2- 172.21.15.37:0/13462393 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a20134b30 0x7f8a20134f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:30:33.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1e59c640 1 -- 172.21.15.37:0/13462393 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a200a0260 msgr2=0x7f8a200a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:33.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1e59c640 1 --2- 172.21.15.37:0/13462393 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a200a0260 0x7f8a200a0640 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8a20135bd0 tx=0x7f8a1401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=613b0b71d1ee0f49 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:33.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1d59a640 1 -- 172.21.15.37:0/13462393 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a140b0070 con 0x7f8a200a0260 2024-04-26T00:30:33.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:33.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:30:33.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1d59a640 10 monclient: dump: 2024-04-26T00:30:33.436 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:33.436 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:33.436 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:33.436 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:33.436 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:33.436 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:33.436 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:33.436 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:33.437 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:33.437 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:33.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1d59a640 10 monclient: _finish_auth 0 2024-04-26T00:30:33.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1d59a640 10 monclient: _check_auth_tickets 2024-04-26T00:30:33.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:30:33.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1d59a640 1 -- 172.21.15.37:0/13462393 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a140ab600 con 0x7f8a200a0260 2024-04-26T00:30:33.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a24c6e640 5 monclient: authenticate success, global_id 124128 2024-04-26T00:30:33.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a24c6e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:30:33.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1d59a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:30:33.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a24c6e640 10 monclient: get_monmap_and_config success 2024-04-26T00:30:33.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1d59a640 1 -- 172.21.15.37:0/13462393 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a140ab9d0 con 0x7f8a200a0260 2024-04-26T00:30:33.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:33.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a24c6e640 4 set_mon_vals no callback set 2024-04-26T00:30:33.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:30:33.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1d59a640 10 monclient: dump: 2024-04-26T00:30:33.438 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:33.438 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:33.438 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:33.438 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:33.438 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:33.438 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:33.438 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:33.438 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:33.438 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:33.438 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:33.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a24c6e640 10 monclient: shutdown 2024-04-26T00:30:33.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a24c6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:30:33.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a24c6e640 1 -- 172.21.15.37:0/13462393 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a200a0260 msgr2=0x7f8a200a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:33.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a24c6e640 1 --2- 172.21.15.37:0/13462393 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a200a0260 0x7f8a200a0640 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8a20135bd0 tx=0x7f8a1401c3e0 comp rx=0 tx=0).stop 2024-04-26T00:30:33.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a24c6e640 1 -- 172.21.15.37:0/13462393 shutdown_connections 2024-04-26T00:30:33.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a24c6e640 1 --2- 172.21.15.37:0/13462393 >> v2:172.21.15.45:3300/0 conn(0x7f8a20135490 0x7f8a2013be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:33.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a24c6e640 1 --2- 172.21.15.37:0/13462393 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a20134b30 0x7f8a20134f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:33.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a24c6e640 1 --2- 172.21.15.37:0/13462393 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a200a0260 0x7f8a200a0640 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:33.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a24c6e640 1 -- 172.21.15.37:0/13462393 >> 172.21.15.37:0/13462393 conn(0x7f8a200a1ad0 msgr2=0x7f8a200a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:30:33.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a24c6e640 1 -- 172.21.15.37:0/13462393 shutdown_connections 2024-04-26T00:30:33.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a24c6e640 1 -- 172.21.15.37:0/13462393 wait complete. 2024-04-26T00:30:33.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a24c6e640 10 monclient: build_initial_monmap 2024-04-26T00:30:33.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a24c6e640 10 monclient: monmap: 2024-04-26T00:30:33.439 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:30:33.439 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:30:33.439 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:30:33.439 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:30:33.439 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:30:33.440 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:30:33.440 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:30:33.440 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:30:33.440 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:30:33.440 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:33.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a24c6e640 1 Processor -- start 2024-04-26T00:30:33.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a24c6e640 1 -- start start 2024-04-26T00:30:33.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a24c6e640 10 monclient: init 2024-04-26T00:30:33.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a24c6e640 10 monclient: _reopen_session rank -1 2024-04-26T00:30:33.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a24c6e640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:30:33.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a24c6e640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a200a0260 0x7f8a2011b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:33.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a24c6e640 10 monclient(hunting): picked mon.noname-a con 0x7f8a200a0260 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:30:33.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a24c6e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a20134b30 0x7f8a20131770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:33.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a24c6e640 10 monclient(hunting): picked mon.noname-c con 0x7f8a20134b30 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:30:33.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a24c6e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a20135490 0x7f8a20139960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:33.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a24c6e640 10 monclient(hunting): picked mon.noname-b con 0x7f8a20135490 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:30:33.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a24c6e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:33.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a24c6e640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f8a2009f430 con 0x7f8a200a0260 2024-04-26T00:30:33.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a24c6e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:33.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a24c6e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8a2009f2b0 con 0x7f8a20135490 2024-04-26T00:30:33.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a24c6e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:33.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a24c6e640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8a2009fe90 con 0x7f8a20134b30 2024-04-26T00:30:33.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a24c6e640 10 monclient(hunting): _renew_subs 2024-04-26T00:30:33.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a24c6e640 10 monclient(hunting): authenticate will time out at 1614.750613s 2024-04-26T00:30:33.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1dd9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a20134b30 0x7f8a20131770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:33.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1e59c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a200a0260 0x7f8a2011b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:33.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1dd9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a20134b30 0x7f8a20131770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:56474/0 (socket says 172.21.15.37:56474) 2024-04-26T00:30:33.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1dd9b640 1 -- 172.21.15.37:0/620774207 learned_addr learned my addr 172.21.15.37:0/620774207 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:30:33.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1dd9b640 10 monclient(hunting): get_auth_request con 0x7f8a20134b30 auth_method 0 2024-04-26T00:30:33.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:33.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:33.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:33.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1e59c640 10 monclient(hunting): get_auth_request con 0x7f8a200a0260 auth_method 0 2024-04-26T00:30:33.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:33.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:33.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:33.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:33.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:33.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:33.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:33.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:33.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:33.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1ed9d640 1 --2- 172.21.15.37:0/620774207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a20135490 0x7f8a20139960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:33.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1dd9b640 10 monclient(hunting): handle_auth_done global_id 84488 payload 824 2024-04-26T00:30:33.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1dd9b640 1 -- 172.21.15.37:0/620774207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a20135490 msgr2=0x7f8a20139960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:33.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1dd9b640 1 --2- 172.21.15.37:0/620774207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a20135490 0x7f8a20139960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:33.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1dd9b640 1 -- 172.21.15.37:0/620774207 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a200a0260 msgr2=0x7f8a2011b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:33.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1dd9b640 1 --2- 172.21.15.37:0/620774207 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a200a0260 0x7f8a2011b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:33.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1dd9b640 10 monclient: _finish_hunting 0 2024-04-26T00:30:33.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1dd9b640 1 monclient: found mon.noname-c 2024-04-26T00:30:33.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:30:33.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:30:33.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.441+0000 7f8a1dd9b640 1 -- 172.21.15.37:0/620774207 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a2011ce30 con 0x7f8a20134b30 2024-04-26T00:30:33.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.445+0000 7f8a1dd9b640 1 -- 172.21.15.37:0/620774207 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a20134b30 msgr2=0x7f8a20131770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:33.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.445+0000 7f8a1dd9b640 1 --2- 172.21.15.37:0/620774207 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a20134b30 0x7f8a20131770 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8a1000ebe0 tx=0x7f8a1000c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:33.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.445+0000 7f8a0effd640 1 -- 172.21.15.37:0/620774207 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a10017070 con 0x7f8a20134b30 2024-04-26T00:30:33.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.445+0000 7f8a0effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:33.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.445+0000 7f8a0effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:30:33.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.445+0000 7f8a0effd640 10 monclient: dump: 2024-04-26T00:30:33.446 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:33.446 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:33.446 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:33.446 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:33.446 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:33.446 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:33.446 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:33.446 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:33.446 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:33.446 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:33.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.445+0000 7f8a0effd640 10 monclient: _finish_auth 0 2024-04-26T00:30:33.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.445+0000 7f8a0effd640 10 monclient: _check_auth_tickets 2024-04-26T00:30:33.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.445+0000 7f8a0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:30:33.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.445+0000 7f8a0effd640 1 -- 172.21.15.37:0/620774207 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a1000ce40 con 0x7f8a20134b30 2024-04-26T00:30:33.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.445+0000 7f8a0effd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:30:33.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.445+0000 7f8a0effd640 1 -- 172.21.15.37:0/620774207 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a10013690 con 0x7f8a20134b30 2024-04-26T00:30:33.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.445+0000 7f8a24c6e640 5 monclient: authenticate success, global_id 84488 2024-04-26T00:30:33.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.445+0000 7f8a24c6e640 10 monclient: _renew_subs 2024-04-26T00:30:33.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.445+0000 7f8a24c6e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:30:33.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.445+0000 7f8a24c6e640 1 -- 172.21.15.37:0/620774207 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a2011c3b0 con 0x7f8a20134b30 2024-04-26T00:30:33.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.445+0000 7f8a0effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:33.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.445+0000 7f8a0effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:30:33.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.445+0000 7f8a1cd99640 4 set_mon_vals no callback set 2024-04-26T00:30:33.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.445+0000 7f8a0effd640 10 monclient: dump: 2024-04-26T00:30:33.447 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:33.448 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:33.448 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:33.448 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:33.448 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:33.448 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:33.448 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:33.448 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:33.448 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:33.448 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:33.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.445+0000 7f8a24c6e640 10 monclient: _renew_subs 2024-04-26T00:30:33.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.445+0000 7f8a24c6e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:30:33.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.445+0000 7f8a24c6e640 1 -- 172.21.15.37:0/620774207 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a2011cb50 con 0x7f8a20134b30 2024-04-26T00:30:33.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.445+0000 7f8a24c6e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:30:33.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.445+0000 7f8a24c6e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:30:33.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.445+0000 7f8a0effd640 1 -- 172.21.15.37:0/620774207 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f8a1001a070 con 0x7f8a20134b30 2024-04-26T00:30:33.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.445+0000 7f8a24c6e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:30:33.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.445+0000 7f8a24c6e640 1 -- 172.21.15.37:0/620774207 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a20066f40 con 0x7f8a20134b30 2024-04-26T00:30:33.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.445+0000 7f8a0effd640 1 --2- 172.21.15.37:0/620774207 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8a00035ed0 0x7f8a00038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:33.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.445+0000 7f8a0effd640 1 -- 172.21.15.37:0/620774207 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(403..403 src has 370..403) ==== 3991+0+0 (secure 0 0 0) 0x7f8a10048ac0 con 0x7f8a20134b30 2024-04-26T00:30:33.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.445+0000 7f8a1e59c640 1 --2- 172.21.15.37:0/620774207 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8a00035ed0 0x7f8a00038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:33.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.445+0000 7f8a1e59c640 10 monclient: get_auth_request con 0x7f8a00035ed0 auth_method 0 2024-04-26T00:30:33.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.449+0000 7f8a1e59c640 1 --2- 172.21.15.37:0/620774207 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8a00035ed0 0x7f8a00038390 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8a200a2ad0 tx=0x7f8a14008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:33.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.449+0000 7f8a0effd640 1 -- 172.21.15.37:0/620774207 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f8a10013830 con 0x7f8a20134b30 2024-04-26T00:30:33.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.449+0000 7f8a0effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:30:33.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.449+0000 7f8a0effd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:30:33.514 INFO:tasks.rados.rados.0.smithi045.stdout:1782: done (5 left) 2024-04-26T00:30:33.515 INFO:tasks.rados.rados.0.smithi045.stdout:1783: copy_from oid 13 from oid 34 current snap is 202 2024-04-26T00:30:33.515 INFO:tasks.rados.rados.0.smithi045.stdout:1777: finishing write tid 3 to smithi04526579-50 2024-04-26T00:30:33.515 INFO:tasks.rados.rados.0.smithi045.stdout:1777: finishing write tid 4 to smithi04526579-50 2024-04-26T00:30:33.515 INFO:tasks.rados.rados.0.smithi045.stdout:1777: finishing write tid 5 to smithi04526579-50 2024-04-26T00:30:33.515 INFO:tasks.rados.rados.0.smithi045.stdout:1777: oid 50 updating version 0 to 1060 2024-04-26T00:30:33.515 INFO:tasks.rados.rados.0.smithi045.stdout:1777: oid 50 version 1060 is already newer than 1059 2024-04-26T00:30:33.515 INFO:tasks.rados.rados.0.smithi045.stdout:1777: oid 50 updating version 1060 to 1062 2024-04-26T00:30:33.515 INFO:tasks.rados.rados.0.smithi045.stdout:1777: oid 50 version 1062 is already newer than 1061 2024-04-26T00:30:33.515 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 1062 (ObjNum 654 snap 202 seq_num 654) dirty exists 2024-04-26T00:30:33.515 INFO:tasks.rados.rados.0.smithi045.stdout:1777: left oid 50 (ObjNum 654 snap 202 seq_num 654) 2024-04-26T00:30:33.515 INFO:tasks.rados.rados.0.smithi045.stdout:1778: finishing write tid 1 to smithi04526579-28 2024-04-26T00:30:33.515 INFO:tasks.rados.rados.0.smithi045.stdout:1778: finishing write tid 2 to smithi04526579-28 2024-04-26T00:30:33.516 INFO:tasks.rados.rados.0.smithi045.stdout:1778: finishing write tid 3 to smithi04526579-28 2024-04-26T00:30:33.516 INFO:tasks.rados.rados.0.smithi045.stdout:1778: finishing write tid 4 to smithi04526579-28 2024-04-26T00:30:33.516 INFO:tasks.rados.rados.0.smithi045.stdout:1778: finishing write tid 5 to smithi04526579-28 2024-04-26T00:30:33.516 INFO:tasks.rados.rados.0.smithi045.stdout:1778: oid 28 updating version 0 to 1128 2024-04-26T00:30:33.516 INFO:tasks.rados.rados.0.smithi045.stdout:1778: oid 28 version 1128 is already newer than 1126 2024-04-26T00:30:33.516 INFO:tasks.rados.rados.0.smithi045.stdout:1778: oid 28 updating version 1128 to 1129 2024-04-26T00:30:33.516 INFO:tasks.rados.rados.0.smithi045.stdout:1778: oid 28 version 1129 is already newer than 1127 2024-04-26T00:30:33.516 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 1129 (ObjNum 655 snap 202 seq_num 655) dirty exists 2024-04-26T00:30:33.516 INFO:tasks.rados.rados.0.smithi045.stdout:1778: left oid 28 (ObjNum 655 snap 202 seq_num 655) 2024-04-26T00:30:33.516 INFO:tasks.rados.rados.0.smithi045.stdout:1780: finishing copy_from to smithi04526579-15 2024-04-26T00:30:33.516 INFO:tasks.rados.rados.0.smithi045.stdout:1780: got expected ENOENT (src dne) 2024-04-26T00:30:33.516 INFO:tasks.rados.rados.0.smithi045.stdout:1780: finishing copy_from racing read to smithi04526579-15 2024-04-26T00:30:33.516 INFO:tasks.rados.rados.0.smithi045.stdout:1779: expect (ObjNum 543 snap 167 seq_num 543) 2024-04-26T00:30:33.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.697+0000 7f8a24c6e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:30:33.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.697+0000 7f8a24c6e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:30:33.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.697+0000 7f8a24c6e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:30:33.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.697+0000 7f8a24c6e640 1 -- 172.21.15.37:0/620774207 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8a2006b9a0 con 0x7f8a20134b30 2024-04-26T00:30:33.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.697+0000 7f8a0effd640 1 -- 172.21.15.37:0/620774207 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f8a100477d0 con 0x7f8a20134b30 2024-04-26T00:30:33.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.697+0000 7f8a0effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:30:33.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.697+0000 7f8a0effd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:30:33.691 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:30:33.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.701+0000 7f8a24c6e640 1 -- 172.21.15.37:0/620774207 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8a00035ed0 msgr2=0x7f8a00038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:33.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.701+0000 7f8a24c6e640 1 --2- 172.21.15.37:0/620774207 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8a00035ed0 0x7f8a00038390 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8a200a2ad0 tx=0x7f8a14008040 comp rx=0 tx=0).stop 2024-04-26T00:30:33.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.701+0000 7f8a24c6e640 10 monclient: shutdown 2024-04-26T00:30:33.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.701+0000 7f8a24c6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:30:33.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.701+0000 7f8a24c6e640 1 -- 172.21.15.37:0/620774207 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a20134b30 msgr2=0x7f8a20131770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:33.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.701+0000 7f8a24c6e640 1 --2- 172.21.15.37:0/620774207 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a20134b30 0x7f8a20131770 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8a1000ebe0 tx=0x7f8a1000c520 comp rx=0 tx=0).stop 2024-04-26T00:30:33.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.701+0000 7f8a24c6e640 1 -- 172.21.15.37:0/620774207 shutdown_connections 2024-04-26T00:30:33.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.701+0000 7f8a24c6e640 1 --2- 172.21.15.37:0/620774207 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8a00035ed0 0x7f8a00038390 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:33.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.701+0000 7f8a24c6e640 1 --2- 172.21.15.37:0/620774207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a20135490 0x7f8a20139960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:33.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.701+0000 7f8a24c6e640 1 --2- 172.21.15.37:0/620774207 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a20134b30 0x7f8a20131770 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:33.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.701+0000 7f8a24c6e640 1 --2- 172.21.15.37:0/620774207 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a200a0260 0x7f8a2011b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:33.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.701+0000 7f8a24c6e640 1 -- 172.21.15.37:0/620774207 >> 172.21.15.37:0/620774207 conn(0x7f8a200a1ad0 msgr2=0x7f8a200a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:30:33.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.701+0000 7f8a24c6e640 1 -- 172.21.15.37:0/620774207 shutdown_connections 2024-04-26T00:30:33.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.701+0000 7f8a24c6e640 1 -- 172.21.15.37:0/620774207 wait complete. 2024-04-26T00:30:33.710 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:30:33.710 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-26T00:30:33.710 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:30:33.710 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:30:33.728 INFO:tasks.rados.rados.0.smithi045.stdout:1781: finishing copy_from racing read to smithi04526579-35 2024-04-26T00:30:33.728 INFO:tasks.rados.rados.0.smithi045.stdout:1781: finishing copy_from to smithi04526579-35 2024-04-26T00:30:33.728 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 619 (ObjNum 401 snap 122 seq_num 401) dirty exists 2024-04-26T00:30:33.728 INFO:tasks.rados.rados.0.smithi045.stdout:1777: done (5 left) 2024-04-26T00:30:33.728 INFO:tasks.rados.rados.0.smithi045.stdout:1778: done (4 left) 2024-04-26T00:30:33.729 INFO:tasks.rados.rados.0.smithi045.stdout:1779: done (3 left) 2024-04-26T00:30:33.729 INFO:tasks.rados.rados.0.smithi045.stdout:1780: done (2 left) 2024-04-26T00:30:33.729 INFO:tasks.rados.rados.0.smithi045.stdout:1781: done (1 left) 2024-04-26T00:30:33.729 INFO:tasks.rados.rados.0.smithi045.stdout:1784: snap_create 2024-04-26T00:30:33.729 INFO:tasks.rados.rados.0.smithi045.stdout:1783: finishing copy_from racing read to smithi04526579-13 2024-04-26T00:30:33.729 INFO:tasks.rados.rados.0.smithi045.stdout:1783: finishing copy_from to smithi04526579-13 2024-04-26T00:30:33.729 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 700 (ObjNum 474 snap 147 seq_num 474) dirty exists 2024-04-26T00:30:33.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.837+0000 7f7a17d80640 10 monclient: get_monmap_and_config 2024-04-26T00:30:33.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.837+0000 7f7a17d80640 10 monclient: build_initial_monmap 2024-04-26T00:30:33.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.837+0000 7f7a17d80640 10 monclient: monmap: 2024-04-26T00:30:33.828 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:30:33.828 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:33.829 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:30:33.840612+0000 2024-04-26T00:30:33.829 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:30:33.840612+0000 2024-04-26T00:30:33.829 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:30:33.829 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:33.829 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:30:33.829 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:30:33.829 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:30:33.829 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:30:33.829 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:30:33.829 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:33.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.837+0000 7f7a17d80640 1 Processor -- start 2024-04-26T00:30:33.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.837+0000 7f7a17d80640 1 -- start start 2024-04-26T00:30:33.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.837+0000 7f7a17d80640 10 monclient: init 2024-04-26T00:30:33.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.837+0000 7f7a17d80640 10 monclient: _reopen_session rank -1 2024-04-26T00:30:33.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.837+0000 7f7a17d80640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-04-26T00:30:33.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.837+0000 7f7a17d80640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7a100a0300 addr v1:172.21.15.37:6789/0 2024-04-26T00:30:33.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.837+0000 7f7a17d80640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7a1009fc50 0x7f7a10139ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:33.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.837+0000 7f7a17d80640 10 monclient(hunting): picked mon.noname-b con 0x7f7a1009fc50 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:30:33.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.837+0000 7f7a17d80640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7a1013a400 addr v1:172.21.15.45:6789/0 2024-04-26T00:30:33.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.837+0000 7f7a17d80640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a1009c190 con 0x7f7a100a0300 2024-04-26T00:30:33.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.837+0000 7f7a17d80640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a100a1110 con 0x7f7a1013a400 2024-04-26T00:30:33.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.837+0000 7f7a17d80640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:33.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.837+0000 7f7a17d80640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f7a100a16d0 con 0x7f7a1009fc50 2024-04-26T00:30:33.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.837+0000 7f7a17d80640 10 monclient(hunting): _renew_subs 2024-04-26T00:30:33.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.837+0000 7f7a17d80640 10 monclient(hunting): authenticate will time out at 1615.146578s 2024-04-26T00:30:33.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.837+0000 7f7a1531b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7a1009fc50 0x7f7a10139ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:33.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.837+0000 7f7a15b1c640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f7a100a0300 0x7f7a100a06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:34764/0 (socket says 172.21.15.37:34764) 2024-04-26T00:30:33.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.837+0000 7f7a15b1c640 1 -- 172.21.15.37:0/2073460594 learned_addr learned my addr 172.21.15.37:0/2073460594 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:30:33.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.837+0000 7f7a1531b640 10 monclient(hunting): get_auth_request con 0x7f7a1009fc50 auth_method 0 2024-04-26T00:30:33.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.837+0000 7f7a1531b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:33.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.837+0000 7f7a1531b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:33.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.837+0000 7f7a1531b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:33.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.837+0000 7f7a1531b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:33.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.837+0000 7f7a1531b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:33.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.837+0000 7f7a1531b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:33.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.837+0000 7f7a1531b640 10 monclient(hunting): handle_auth_done global_id 84491 payload 274 2024-04-26T00:30:33.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.837+0000 7f7a1531b640 1 -- 172.21.15.37:0/2073460594 >> v1:172.21.15.45:6789/0 conn(0x7f7a1013a400 legacy=0x7f7a1013c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:30:33.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.837+0000 7f7a1531b640 1 -- 172.21.15.37:0/2073460594 >> v1:172.21.15.37:6789/0 conn(0x7f7a100a0300 legacy=0x7f7a100a06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:30:33.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.837+0000 7f7a1531b640 10 monclient: _finish_hunting 0 2024-04-26T00:30:33.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.837+0000 7f7a1531b640 1 monclient: found mon.noname-b 2024-04-26T00:30:33.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.837+0000 7f7a1531b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:30:33.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.837+0000 7f7a1531b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:30:33.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.837+0000 7f7a1531b640 1 -- 172.21.15.37:0/2073460594 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a1009f2a0 con 0x7f7a1009fc50 2024-04-26T00:30:33.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.841+0000 7f7a1531b640 1 -- 172.21.15.37:0/2073460594 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7a1009fc50 msgr2=0x7f7a10139ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:33.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.841+0000 7f7a1531b640 1 --2- 172.21.15.37:0/2073460594 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7a1009fc50 0x7f7a10139ec0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c009820 tx=0x7f7a0c01c300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c7c1b835df5f609 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:33.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.841+0000 7f7a14b1a640 1 -- 172.21.15.37:0/2073460594 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c0b0070 con 0x7f7a1009fc50 2024-04-26T00:30:33.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.841+0000 7f7a14b1a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:33.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.841+0000 7f7a14b1a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:30:33.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.841+0000 7f7a14b1a640 10 monclient: dump: 2024-04-26T00:30:33.836 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:33.836 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:33.836 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:33.836 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:33.836 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:33.836 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:33.836 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:33.836 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:33.836 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:33.836 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:33.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.841+0000 7f7a14b1a640 10 monclient: _finish_auth 0 2024-04-26T00:30:33.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.841+0000 7f7a14b1a640 10 monclient: _check_auth_tickets 2024-04-26T00:30:33.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.841+0000 7f7a14b1a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:30:33.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.841+0000 7f7a14b1a640 1 -- 172.21.15.37:0/2073460594 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a0c0ab600 con 0x7f7a1009fc50 2024-04-26T00:30:33.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.841+0000 7f7a14b1a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:30:33.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.841+0000 7f7a14b1a640 1 -- 172.21.15.37:0/2073460594 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c0a4e60 con 0x7f7a1009fc50 2024-04-26T00:30:33.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.841+0000 7f7a17d80640 5 monclient: authenticate success, global_id 84491 2024-04-26T00:30:33.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.841+0000 7f7a17d80640 10 monclient: get_monmap_and_config success 2024-04-26T00:30:33.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.841+0000 7f7a17d80640 4 set_mon_vals no callback set 2024-04-26T00:30:33.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.841+0000 7f7a14b1a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:33.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.841+0000 7f7a14b1a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:30:33.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.841+0000 7f7a14b1a640 10 monclient: dump: 2024-04-26T00:30:33.837 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:33.837 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:33.838 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:33.838 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:33.838 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:33.838 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:33.838 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:33.838 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:33.838 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:33.838 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:33.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a17d80640 10 monclient: shutdown 2024-04-26T00:30:33.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a17d80640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:30:33.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a17d80640 1 -- 172.21.15.37:0/2073460594 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7a1009fc50 msgr2=0x7f7a10139ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:33.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a17d80640 1 --2- 172.21.15.37:0/2073460594 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7a1009fc50 0x7f7a10139ec0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c009820 tx=0x7f7a0c01c300 comp rx=0 tx=0).stop 2024-04-26T00:30:33.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a17d80640 1 -- 172.21.15.37:0/2073460594 shutdown_connections 2024-04-26T00:30:33.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a17d80640 1 --2- 172.21.15.37:0/2073460594 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7a1009fc50 0x7f7a10139ec0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:33.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a17d80640 1 -- 172.21.15.37:0/2073460594 >> 172.21.15.37:0/2073460594 conn(0x7f7a100a1d50 msgr2=0x7f7a100a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:30:33.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a17d80640 1 -- 172.21.15.37:0/2073460594 shutdown_connections 2024-04-26T00:30:33.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a17d80640 1 -- 172.21.15.37:0/2073460594 wait complete. 2024-04-26T00:30:33.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a17d80640 10 monclient: build_initial_monmap 2024-04-26T00:30:33.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a17d80640 10 monclient: monmap: 2024-04-26T00:30:33.844 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:30:33.844 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:30:33.844 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:30:33.844 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:30:33.844 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:30:33.844 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:30:33.845 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:30:33.845 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:30:33.845 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:30:33.845 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:33.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a17d80640 1 Processor -- start 2024-04-26T00:30:33.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a17d80640 1 -- start start 2024-04-26T00:30:33.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a17d80640 10 monclient: init 2024-04-26T00:30:33.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a17d80640 10 monclient: _reopen_session rank -1 2024-04-26T00:30:33.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a17d80640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:30:33.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a17d80640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a1009fc50 0x7f7a10070f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:33.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a17d80640 10 monclient(hunting): picked mon.noname-b con 0x7f7a1009fc50 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:30:33.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a17d80640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7a100a0300 0x7f7a10134fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:33.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a17d80640 10 monclient(hunting): picked mon.noname-a con 0x7f7a100a0300 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:30:33.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a17d80640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7a1013a400 0x7f7a10135520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:33.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a17d80640 10 monclient(hunting): picked mon.noname-c con 0x7f7a1013a400 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:30:33.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a17d80640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:33.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a17d80640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f7a10140b00 con 0x7f7a100a0300 2024-04-26T00:30:33.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a17d80640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:33.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a17d80640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7a10140980 con 0x7f7a1009fc50 2024-04-26T00:30:33.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a17d80640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:33.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a17d80640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f7a10140c80 con 0x7f7a1013a400 2024-04-26T00:30:33.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a17d80640 10 monclient(hunting): _renew_subs 2024-04-26T00:30:33.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a17d80640 10 monclient(hunting): authenticate will time out at 1615.153665s 2024-04-26T00:30:33.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a1631d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7a1013a400 0x7f7a10135520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:33.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a1531b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7a100a0300 0x7f7a10134fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:33.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a1531b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7a100a0300 0x7f7a10134fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:43294/0 (socket says 172.21.15.37:43294) 2024-04-26T00:30:33.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a1631d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7a1013a400 0x7f7a10135520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:56490/0 (socket says 172.21.15.37:56490) 2024-04-26T00:30:33.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a1531b640 1 -- 172.21.15.37:0/192487750 learned_addr learned my addr 172.21.15.37:0/192487750 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:30:33.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a1531b640 10 monclient(hunting): get_auth_request con 0x7f7a100a0300 auth_method 0 2024-04-26T00:30:33.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a1531b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:33.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a1531b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:33.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a1531b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:33.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a1631d640 10 monclient(hunting): get_auth_request con 0x7f7a1013a400 auth_method 0 2024-04-26T00:30:33.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a1631d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:33.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a1631d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:33.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a1631d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:33.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a1531b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:33.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a1531b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:33.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a1531b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:33.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a1631d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:33.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a1631d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:33.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a1631d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:33.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a15b1c640 1 --2- 172.21.15.37:0/192487750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a1009fc50 0x7f7a10070f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:33.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a15b1c640 10 monclient(hunting): get_auth_request con 0x7f7a1009fc50 auth_method 0 2024-04-26T00:30:33.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a15b1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:33.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a15b1c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:33.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a15b1c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:33.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a1531b640 10 monclient(hunting): handle_auth_done global_id 124134 payload 824 2024-04-26T00:30:33.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a1531b640 1 -- 172.21.15.37:0/192487750 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7a1013a400 msgr2=0x7f7a10135520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:33.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a1531b640 1 --2- 172.21.15.37:0/192487750 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7a1013a400 0x7f7a10135520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:33.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a1531b640 1 -- 172.21.15.37:0/192487750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a1009fc50 msgr2=0x7f7a10070f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:33.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a1531b640 1 --2- 172.21.15.37:0/192487750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a1009fc50 0x7f7a10070f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:33.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a1531b640 10 monclient: _finish_hunting 0 2024-04-26T00:30:33.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a1531b640 1 monclient: found mon.noname-a 2024-04-26T00:30:33.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a1531b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:30:33.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a1531b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:30:33.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.845+0000 7f7a1531b640 1 -- 172.21.15.37:0/192487750 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a10120100 con 0x7f7a100a0300 2024-04-26T00:30:33.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a1531b640 1 -- 172.21.15.37:0/192487750 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7a100a0300 msgr2=0x7f7a10134fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:33.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a1531b640 1 --2- 172.21.15.37:0/192487750 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7a100a0300 0x7f7a10134fe0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c0ae040 tx=0x7f7a0c01ab40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:33.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a027fc640 1 -- 172.21.15.37:0/192487750 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c0b0050 con 0x7f7a100a0300 2024-04-26T00:30:33.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:33.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a027fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:30:33.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a027fc640 10 monclient: dump: 2024-04-26T00:30:33.849 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:33.849 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:33.849 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:33.849 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:33.849 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:33.849 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:33.849 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:33.849 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:33.850 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:33.850 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:33.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a027fc640 10 monclient: _finish_auth 0 2024-04-26T00:30:33.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a027fc640 10 monclient: _check_auth_tickets 2024-04-26T00:30:33.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:30:33.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a17d80640 5 monclient: authenticate success, global_id 124134 2024-04-26T00:30:33.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a17d80640 10 monclient: _renew_subs 2024-04-26T00:30:33.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a027fc640 1 -- 172.21.15.37:0/192487750 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a0c0bc070 con 0x7f7a100a0300 2024-04-26T00:30:33.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a17d80640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:30:33.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a17d80640 1 -- 172.21.15.37:0/192487750 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a101206c0 con 0x7f7a100a0300 2024-04-26T00:30:33.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a027fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:30:33.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a027fc640 1 -- 172.21.15.37:0/192487750 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c0b5a40 con 0x7f7a100a0300 2024-04-26T00:30:33.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:33.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:30:33.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a027fc640 10 monclient: dump: 2024-04-26T00:30:33.851 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:33.851 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:33.851 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:33.851 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:33.851 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:33.851 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:33.851 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:33.851 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:33.851 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:33.851 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:33.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a03fff640 4 set_mon_vals no callback set 2024-04-26T00:30:33.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a17d80640 10 monclient: _renew_subs 2024-04-26T00:30:33.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a17d80640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:30:33.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a17d80640 1 -- 172.21.15.37:0/192487750 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a101cf8b0 con 0x7f7a100a0300 2024-04-26T00:30:33.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a17d80640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:30:33.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a17d80640 1 --2- 172.21.15.37:0/192487750 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7a1009cf10 0x7f7a1002c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:33.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a17d80640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:33.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a17d80640 1 -- 172.21.15.37:0/192487750 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f7a1004f510 con 0x7f7a1009cf10 2024-04-26T00:30:33.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a15b1c640 1 --2- 172.21.15.37:0/192487750 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7a1009cf10 0x7f7a1002c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:33.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a15b1c640 10 monclient: get_auth_request con 0x7f7a1009cf10 auth_method 0 2024-04-26T00:30:33.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a15b1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:33.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a15b1c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:33.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a15b1c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:33.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a15b1c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:33.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a15b1c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:33.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a15b1c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:33.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a15b1c640 10 monclient(hunting): handle_auth_done global_id 124137 payload 824 2024-04-26T00:30:33.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a15b1c640 1 -- 172.21.15.37:0/192487750 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7a101393b0 con 0x7f7a1009cf10 2024-04-26T00:30:33.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a027fc640 1 -- 172.21.15.37:0/192487750 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f7a0c0b5c60 con 0x7f7a100a0300 2024-04-26T00:30:33.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a027fc640 1 --2- 172.21.15.37:0/192487750 >> v2:172.21.15.45:6812/2379918623 conn(0x7f79f4035eb0 0x7f79f4038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:33.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a027fc640 1 -- 172.21.15.37:0/192487750 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(404..404 src has 370..404) ==== 4007+0+0 (secure 0 0 0) 0x7f7a0c0e3480 con 0x7f7a100a0300 2024-04-26T00:30:33.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a1631d640 1 --2- 172.21.15.37:0/192487750 >> v2:172.21.15.45:6812/2379918623 conn(0x7f79f4035eb0 0x7f79f4038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:33.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.849+0000 7f7a1631d640 10 monclient: get_auth_request con 0x7f79f4035eb0 auth_method 0 2024-04-26T00:30:33.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.853+0000 7f7a15b1c640 1 -- 172.21.15.37:0/192487750 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7a1009cf10 msgr2=0x7f7a1002c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:33.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.853+0000 7f7a15b1c640 1 --2- 172.21.15.37:0/192487750 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7a1009cf10 0x7f7a1002c000 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7a0400e500 tx=0x7f7a0400e9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:33.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.853+0000 7f7a027fc640 1 -- 172.21.15.37:0/192487750 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a04004e40 con 0x7f7a1009cf10 2024-04-26T00:30:33.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.853+0000 7f7a027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:33.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.853+0000 7f7a027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:30:33.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.853+0000 7f7a027fc640 10 monclient: dump: 2024-04-26T00:30:33.853 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:33.854 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:33.854 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:33.854 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:33.854 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:33.854 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:33.854 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:33.854 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:33.854 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:33.854 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:33.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.853+0000 7f7a027fc640 1 -- 172.21.15.37:0/192487750 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7a04018070 con 0x7f7a1009cf10 2024-04-26T00:30:33.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.853+0000 7f7a027fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:30:33.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.853+0000 7f7a027fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:30:33.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.853+0000 7f7a027fc640 1 -- 172.21.15.37:0/192487750 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7a1009cf10 msgr2=0x7f7a1002c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:33.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.853+0000 7f7a027fc640 1 --2- 172.21.15.37:0/192487750 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7a1009cf10 0x7f7a1002c000 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7a0400e500 tx=0x7f7a0400e9c0 comp rx=0 tx=0).stop 2024-04-26T00:30:33.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.853+0000 7f7a027fc640 1 -- 172.21.15.37:0/192487750 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7a1009cf10 msgr2=0x7f7a1002c000 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:30:33.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.853+0000 7f7a027fc640 1 --2- 172.21.15.37:0/192487750 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7a1009cf10 0x7f7a1002c000 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7a0400e500 tx=0x7f7a0400e9c0 comp rx=0 tx=0).stop 2024-04-26T00:30:33.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.853+0000 7f7a1631d640 1 --2- 172.21.15.37:0/192487750 >> v2:172.21.15.45:6812/2379918623 conn(0x7f79f4035eb0 0x7f79f4038370 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7a100a3120 tx=0x7f7a08009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:33.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.857+0000 7f7a17d80640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:30:33.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.857+0000 7f7a17d80640 1 --2- 172.21.15.37:0/192487750 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7a1009d2f0 0x7f7a1002c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:33.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.857+0000 7f7a17d80640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:33.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.857+0000 7f7a17d80640 1 -- 172.21.15.37:0/192487750 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f7a1009eb20 con 0x7f7a1009d2f0 2024-04-26T00:30:33.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.857+0000 7f7a15b1c640 1 --2- 172.21.15.37:0/192487750 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7a1009d2f0 0x7f7a1002c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:33.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.857+0000 7f7a15b1c640 10 monclient: get_auth_request con 0x7f7a1009d2f0 auth_method 0 2024-04-26T00:30:33.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.857+0000 7f7a15b1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:33.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.857+0000 7f7a15b1c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:33.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.857+0000 7f7a15b1c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:33.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.857+0000 7f7a15b1c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:33.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.857+0000 7f7a15b1c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:33.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.857+0000 7f7a15b1c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:33.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.857+0000 7f7a15b1c640 10 monclient(hunting): handle_auth_done global_id 124140 payload 824 2024-04-26T00:30:33.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.857+0000 7f7a15b1c640 1 -- 172.21.15.37:0/192487750 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7a1006ba50 con 0x7f7a1009d2f0 2024-04-26T00:30:33.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.861+0000 7f7a15b1c640 1 -- 172.21.15.37:0/192487750 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7a1009d2f0 msgr2=0x7f7a1002c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:33.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.861+0000 7f7a15b1c640 1 --2- 172.21.15.37:0/192487750 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7a1009d2f0 0x7f7a1002c540 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7a0400e4d0 tx=0x7f7a04004950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:33.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.861+0000 7f7a027fc640 1 -- 172.21.15.37:0/192487750 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a040064e0 con 0x7f7a1009d2f0 2024-04-26T00:30:33.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.861+0000 7f7a027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:33.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.861+0000 7f7a027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:30:33.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.861+0000 7f7a027fc640 10 monclient: dump: 2024-04-26T00:30:33.858 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:33.858 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:33.858 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:33.858 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:33.858 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:33.858 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:33.859 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:33.859 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:33.859 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:33.859 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:33.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.861+0000 7f7a027fc640 1 -- 172.21.15.37:0/192487750 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7f7a040064e0 con 0x7f7a1009d2f0 2024-04-26T00:30:33.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.861+0000 7f7a027fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:30:33.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.861+0000 7f7a027fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:30:33.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.861+0000 7f7a027fc640 1 -- 172.21.15.37:0/192487750 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7a1009d2f0 msgr2=0x7f7a1002c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:33.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.861+0000 7f7a027fc640 1 --2- 172.21.15.37:0/192487750 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7a1009d2f0 0x7f7a1002c540 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7a0400e4d0 tx=0x7f7a04004950 comp rx=0 tx=0).stop 2024-04-26T00:30:33.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.861+0000 7f7a027fc640 1 -- 172.21.15.37:0/192487750 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7a1009d2f0 msgr2=0x7f7a1002c540 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:30:33.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.861+0000 7f7a027fc640 1 --2- 172.21.15.37:0/192487750 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7a1009d2f0 0x7f7a1002c540 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7a0400e4d0 tx=0x7f7a04004950 comp rx=0 tx=0).stop 2024-04-26T00:30:33.860 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:30:33.860 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:30:33.860 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:30:33.860 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:30:33.860 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 80, 2024-04-26T00:30:33.860 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 22739, 2024-04-26T00:30:33.860 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:30:33.860 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:30:33.860 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:30:33.860 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:30:33.860 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:30:33.860 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 2, 2024-04-26T00:30:33.860 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:30:33.861 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:30:33.861 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:30:33.861 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:30:33.861 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:30:33.861 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:30:33.861 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:30:33.861 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:30:33.861 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:30:33.861 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:30:33.861 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:30:33.861 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:30:33.861 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:30:33.861 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:30:33.861 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:30:33.862 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:30:33.862 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:30:33.862 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:30:33.862 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:30:33.862 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:30:33.862 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:30:33.862 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:30:33.862 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:30:33.862 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:30:33.862 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:30:33.862 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:30:33.862 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:30:33.862 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:30:33.863 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:33.863 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:33.863 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:30:33.863 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:30:33.863 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:30:33.863 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:30:33.863 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:30:33.863 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:30:33.863 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:30:33.863 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:30:33.863 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:30:33.864 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:30:33.864 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:30:33.864 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:30:33.864 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:30:33.864 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:30:33.864 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:30:33.864 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:30:33.864 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:30:33.864 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:30:33.864 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:30:33.864 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:30:33.864 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:30:33.864 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:30:33.864 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:30:33.865 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:30:33.865 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:30:33.865 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:30:33.865 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:30:33.865 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:30:33.865 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:30:33.865 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:30:33.865 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:33.865 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:30:33.865 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:33.865 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:30:33.865 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:30:33.865 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:30:33.865 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:30:33.865 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:33.865 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:30:33.866 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:30:33.866 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:33.866 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:33.866 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:33.866 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:30:33.866 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:30:33.866 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:33.866 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:33.866 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:33.866 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:33.866 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:30:33.866 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:30:33.866 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:30:33.866 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:30:33.866 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:30:33.867 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:33.867 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:33.867 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:30:33.867 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:30:33.867 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:30:33.867 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:30:33.867 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:33.867 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:30:33.867 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:30:33.867 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:33.867 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:33.867 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:33.867 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:30:33.867 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:30:33.867 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:33.868 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:33.868 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:33.868 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:33.868 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:30:33.868 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:30:33.868 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:30:33.868 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:30:33.868 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:30:33.868 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:33.868 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:33.868 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:30:33.868 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:30:33.868 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:30:33.868 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:30:33.868 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:33.868 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:30:33.869 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:30:33.869 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:33.869 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:33.869 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:33.869 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:30:33.869 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:30:33.869 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:33.869 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:33.869 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:33.869 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:33.869 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:30:33.869 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:30:33.869 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:30:33.869 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:30:33.869 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:30:33.870 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:33.870 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:33.870 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:33.870 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:30:33.870 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:30:33.870 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:33.870 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:30:33.870 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:30:33.870 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:30:33.870 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:33.870 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:30:33.870 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:30:33.870 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:33.870 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:30:33.870 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:30:33.871 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:30:33.871 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:33.871 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:33.871 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:33.871 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:30:33.871 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:30:33.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.861+0000 7f7a17d80640 1 -- 172.21.15.37:0/192487750 >> v2:172.21.15.45:6812/2379918623 conn(0x7f79f4035eb0 msgr2=0x7f79f4038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:33.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.861+0000 7f7a17d80640 1 --2- 172.21.15.37:0/192487750 >> v2:172.21.15.45:6812/2379918623 conn(0x7f79f4035eb0 0x7f79f4038370 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7a100a3120 tx=0x7f7a08009040 comp rx=0 tx=0).stop 2024-04-26T00:30:33.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.861+0000 7f7a17d80640 10 monclient: shutdown 2024-04-26T00:30:33.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.861+0000 7f7a17d80640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:30:33.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.861+0000 7f7a17d80640 1 -- 172.21.15.37:0/192487750 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7a100a0300 msgr2=0x7f7a10134fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:33.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.861+0000 7f7a17d80640 1 --2- 172.21.15.37:0/192487750 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7a100a0300 0x7f7a10134fe0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c0ae040 tx=0x7f7a0c01ab40 comp rx=0 tx=0).stop 2024-04-26T00:30:33.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.861+0000 7f7a1631d640 1 -- 172.21.15.37:0/192487750 reap_dead start 2024-04-26T00:30:33.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.861+0000 7f7a1631d640 1 -- 172.21.15.37:0/192487750 reap_dead start 2024-04-26T00:30:33.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.861+0000 7f7a17d80640 1 -- 172.21.15.37:0/192487750 shutdown_connections 2024-04-26T00:30:33.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.861+0000 7f7a17d80640 1 -- 172.21.15.37:0/192487750 >> 172.21.15.37:0/192487750 conn(0x7f7a100a1d50 msgr2=0x7f7a1013c430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:30:33.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.861+0000 7f7a17d80640 1 -- 172.21.15.37:0/192487750 shutdown_connections 2024-04-26T00:30:33.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:33.861+0000 7f7a17d80640 1 -- 172.21.15.37:0/192487750 wait complete. 2024-04-26T00:30:33.873 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:30:33.873 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:30:33.873 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:30:33.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.001+0000 7f6d3c32c640 10 monclient: get_monmap_and_config 2024-04-26T00:30:33.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.001+0000 7f6d3c32c640 10 monclient: build_initial_monmap 2024-04-26T00:30:33.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.001+0000 7f6d3c32c640 10 monclient: monmap: 2024-04-26T00:30:33.992 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:30:33.992 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:33.992 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:30:34.004003+0000 2024-04-26T00:30:33.992 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:30:34.004003+0000 2024-04-26T00:30:33.992 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:30:33.992 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:33.992 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:30:33.992 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:30:33.992 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:30:33.992 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:30:33.993 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:30:33.993 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:33.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.001+0000 7f6d3c32c640 1 Processor -- start 2024-04-26T00:30:33.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.001+0000 7f6d3c32c640 1 -- start start 2024-04-26T00:30:33.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.001+0000 7f6d3c32c640 10 monclient: init 2024-04-26T00:30:33.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.001+0000 7f6d3c32c640 10 monclient: _reopen_session rank -1 2024-04-26T00:30:33.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.001+0000 7f6d3c32c640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-04-26T00:30:33.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.001+0000 7f6d3c32c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d340a3d90 0x7f6d3409bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:33.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.001+0000 7f6d3c32c640 10 monclient(hunting): picked mon.noname-b con 0x7f6d340a3d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:30:33.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.001+0000 7f6d3c32c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6d340a4170 addr v1:172.21.15.37:6789/0 2024-04-26T00:30:33.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.001+0000 7f6d3c32c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6d3413c3e0 addr v1:172.21.15.45:6789/0 2024-04-26T00:30:33.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.001+0000 7f6d3c32c640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d340a5150 con 0x7f6d340a4170 2024-04-26T00:30:33.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.001+0000 7f6d3c32c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d3409b280 con 0x7f6d3413c3e0 2024-04-26T00:30:33.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.001+0000 7f6d3c32c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:33.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.001+0000 7f6d3c32c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f6d3409b560 con 0x7f6d340a3d90 2024-04-26T00:30:33.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.001+0000 7f6d3c32c640 10 monclient(hunting): _renew_subs 2024-04-26T00:30:33.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.001+0000 7f6d3c32c640 10 monclient(hunting): authenticate will time out at 1615.309684s 2024-04-26T00:30:33.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.001+0000 7f6d3a0c8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d340a3d90 0x7f6d3409bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:33.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.001+0000 7f6d3a0c8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d340a3d90 0x7f6d3409bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:56494/0 (socket says 172.21.15.37:56494) 2024-04-26T00:30:33.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.001+0000 7f6d3a0c8640 1 -- 172.21.15.37:0/2038708181 learned_addr learned my addr 172.21.15.37:0/2038708181 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:30:33.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.001+0000 7f6d3a0c8640 10 monclient(hunting): get_auth_request con 0x7f6d340a3d90 auth_method 0 2024-04-26T00:30:33.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.001+0000 7f6d3a0c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:33.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.001+0000 7f6d3a0c8640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:33.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.001+0000 7f6d3a0c8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:33.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.001+0000 7f6d3a0c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:33.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.001+0000 7f6d3a0c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:33.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.001+0000 7f6d3a0c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:33.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.001+0000 7f6d3a0c8640 10 monclient(hunting): handle_auth_done global_id 84497 payload 274 2024-04-26T00:30:33.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.001+0000 7f6d3a0c8640 1 -- 172.21.15.37:0/2038708181 >> v1:172.21.15.45:6789/0 conn(0x7f6d3413c3e0 legacy=0x7f6d3413e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:30:33.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.001+0000 7f6d3a0c8640 1 -- 172.21.15.37:0/2038708181 >> v1:172.21.15.37:6789/0 conn(0x7f6d340a4170 legacy=0x7f6d340a45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:30:33.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.001+0000 7f6d3a0c8640 10 monclient: _finish_hunting 0 2024-04-26T00:30:33.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.001+0000 7f6d3a0c8640 1 monclient: found mon.noname-b 2024-04-26T00:30:33.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.001+0000 7f6d3a0c8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:30:33.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.001+0000 7f6d3a0c8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:30:33.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.001+0000 7f6d3a0c8640 1 -- 172.21.15.37:0/2038708181 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d3406dc60 con 0x7f6d340a3d90 2024-04-26T00:30:33.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.005+0000 7f6d3a0c8640 1 -- 172.21.15.37:0/2038708181 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d340a3d90 msgr2=0x7f6d3409bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:33.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.005+0000 7f6d3a0c8640 1 --2- 172.21.15.37:0/2038708181 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d340a3d90 0x7f6d3409bf40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c0098a0 tx=0x7f6d2c01c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=981c4672703c62d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:33.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.005+0000 7f6d390c6640 1 -- 172.21.15.37:0/2038708181 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d2c0b0070 con 0x7f6d340a3d90 2024-04-26T00:30:33.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.005+0000 7f6d390c6640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:33.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.005+0000 7f6d390c6640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:30:33.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.005+0000 7f6d390c6640 10 monclient: dump: 2024-04-26T00:30:34.000 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:34.000 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:34.000 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:34.000 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:34.000 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:34.000 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:34.000 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:34.000 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:34.000 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:34.000 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:34.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.005+0000 7f6d390c6640 10 monclient: _finish_auth 0 2024-04-26T00:30:34.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.005+0000 7f6d390c6640 10 monclient: _check_auth_tickets 2024-04-26T00:30:34.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.005+0000 7f6d390c6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:30:34.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.005+0000 7f6d3c32c640 5 monclient: authenticate success, global_id 84497 2024-04-26T00:30:34.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.005+0000 7f6d390c6640 1 -- 172.21.15.37:0/2038708181 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d2c0ab600 con 0x7f6d340a3d90 2024-04-26T00:30:34.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.005+0000 7f6d3c32c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:30:34.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.005+0000 7f6d390c6640 10 monclient: handle_config config(0 keys) 2024-04-26T00:30:34.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.005+0000 7f6d3c32c640 10 monclient: get_monmap_and_config success 2024-04-26T00:30:34.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.005+0000 7f6d3c32c640 4 set_mon_vals no callback set 2024-04-26T00:30:34.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.005+0000 7f6d390c6640 1 -- 172.21.15.37:0/2038708181 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d2c0ab9d0 con 0x7f6d340a3d90 2024-04-26T00:30:34.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.005+0000 7f6d390c6640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:34.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.005+0000 7f6d390c6640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:30:34.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.005+0000 7f6d390c6640 10 monclient: dump: 2024-04-26T00:30:34.001 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:34.001 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:34.001 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:34.001 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:34.001 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:34.001 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:34.002 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:34.002 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:34.002 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:34.002 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:34.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.005+0000 7f6d3c32c640 10 monclient: shutdown 2024-04-26T00:30:34.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.005+0000 7f6d3c32c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:30:34.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.005+0000 7f6d3c32c640 1 -- 172.21.15.37:0/2038708181 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d340a3d90 msgr2=0x7f6d3409bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:34.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.005+0000 7f6d3c32c640 1 --2- 172.21.15.37:0/2038708181 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d340a3d90 0x7f6d3409bf40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c0098a0 tx=0x7f6d2c01c3c0 comp rx=0 tx=0).stop 2024-04-26T00:30:34.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.005+0000 7f6d3c32c640 1 -- 172.21.15.37:0/2038708181 shutdown_connections 2024-04-26T00:30:34.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.005+0000 7f6d3c32c640 1 --2- 172.21.15.37:0/2038708181 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d340a3d90 0x7f6d3409bf40 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:34.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.005+0000 7f6d3c32c640 1 -- 172.21.15.37:0/2038708181 >> 172.21.15.37:0/2038708181 conn(0x7f6d340aa0a0 msgr2=0x7f6d340a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:30:34.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.005+0000 7f6d3c32c640 1 -- 172.21.15.37:0/2038708181 shutdown_connections 2024-04-26T00:30:34.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.005+0000 7f6d3c32c640 1 -- 172.21.15.37:0/2038708181 wait complete. 2024-04-26T00:30:34.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3c32c640 10 monclient: build_initial_monmap 2024-04-26T00:30:34.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3c32c640 10 monclient: monmap: 2024-04-26T00:30:34.003 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:30:34.003 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:30:34.003 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:30:34.003 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:30:34.003 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:30:34.003 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:30:34.003 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:30:34.003 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:30:34.003 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:30:34.003 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:34.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3c32c640 1 Processor -- start 2024-04-26T00:30:34.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3c32c640 1 -- start start 2024-04-26T00:30:34.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3c32c640 10 monclient: init 2024-04-26T00:30:34.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3c32c640 10 monclient: _reopen_session rank -1 2024-04-26T00:30:34.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3c32c640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:30:34.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3c32c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6d340a3d90 0x7f6d34139750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:34.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3c32c640 10 monclient(hunting): picked mon.noname-a con 0x7f6d340a3d90 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:30:34.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3c32c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d340a4170 0x7f6d3413bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:34.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3c32c640 10 monclient(hunting): picked mon.noname-c con 0x7f6d340a4170 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:30:34.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3c32c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d3413c3e0 0x7f6d34139f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:34.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3c32c640 10 monclient(hunting): picked mon.noname-b con 0x7f6d3413c3e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:30:34.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3c32c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:34.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3c32c640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f6d34142920 con 0x7f6d340a3d90 2024-04-26T00:30:34.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3c32c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:34.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3c32c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6d3409c690 con 0x7f6d3413c3e0 2024-04-26T00:30:34.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3c32c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:34.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3c32c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f6d34142aa0 con 0x7f6d340a4170 2024-04-26T00:30:34.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3c32c640 10 monclient(hunting): _renew_subs 2024-04-26T00:30:34.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3c32c640 10 monclient(hunting): authenticate will time out at 1615.317136s 2024-04-26T00:30:34.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3a0c8640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6d340a3d90 0x7f6d34139750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:34.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d398c7640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d340a4170 0x7f6d3413bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:34.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3a0c8640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6d340a3d90 0x7f6d34139750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:43328/0 (socket says 172.21.15.37:43328) 2024-04-26T00:30:34.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3a0c8640 1 -- 172.21.15.37:0/1501819396 learned_addr learned my addr 172.21.15.37:0/1501819396 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:30:34.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3a0c8640 10 monclient(hunting): get_auth_request con 0x7f6d340a3d90 auth_method 0 2024-04-26T00:30:34.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3a0c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:34.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3a0c8640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:34.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3a0c8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:34.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d398c7640 10 monclient(hunting): get_auth_request con 0x7f6d340a4170 auth_method 0 2024-04-26T00:30:34.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d398c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:34.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d398c7640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:34.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d398c7640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:34.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3a0c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:34.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3a0c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:34.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3a0c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:34.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d398c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:34.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d398c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:34.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d398c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:34.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3a8c9640 1 --2- 172.21.15.37:0/1501819396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d3413c3e0 0x7f6d34139f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:34.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3a8c9640 10 monclient(hunting): get_auth_request con 0x7f6d3413c3e0 auth_method 0 2024-04-26T00:30:34.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3a8c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:34.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3a8c9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:34.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3a8c9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:34.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3a0c8640 10 monclient(hunting): handle_auth_done global_id 124143 payload 824 2024-04-26T00:30:34.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3a0c8640 1 -- 172.21.15.37:0/1501819396 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d340a4170 msgr2=0x7f6d3413bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:34.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3a0c8640 1 --2- 172.21.15.37:0/1501819396 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d340a4170 0x7f6d3413bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:34.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3a0c8640 1 -- 172.21.15.37:0/1501819396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d3413c3e0 msgr2=0x7f6d34139f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:34.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3a0c8640 1 --2- 172.21.15.37:0/1501819396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d3413c3e0 0x7f6d34139f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:34.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3a0c8640 10 monclient: _finish_hunting 0 2024-04-26T00:30:34.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3a0c8640 1 monclient: found mon.noname-a 2024-04-26T00:30:34.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3a0c8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:30:34.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3a0c8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:30:34.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.009+0000 7f6d3a0c8640 1 -- 172.21.15.37:0/1501819396 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d34120140 con 0x7f6d340a3d90 2024-04-26T00:30:34.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d3a0c8640 1 -- 172.21.15.37:0/1501819396 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6d340a3d90 msgr2=0x7f6d34139750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:34.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d3a0c8640 1 --2- 172.21.15.37:0/1501819396 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6d340a3d90 0x7f6d34139750 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c01aa70 tx=0x7f6d2c01ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:34.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d32ffd640 1 -- 172.21.15.37:0/1501819396 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d2c0ba020 con 0x7f6d340a3d90 2024-04-26T00:30:34.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:34.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d32ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:30:34.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d32ffd640 10 monclient: dump: 2024-04-26T00:30:34.012 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:34.012 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:34.012 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:34.012 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:34.012 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:34.012 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:34.013 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:34.013 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:34.013 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:34.013 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:34.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d32ffd640 10 monclient: _finish_auth 0 2024-04-26T00:30:34.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d32ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:30:34.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:30:34.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d32ffd640 1 -- 172.21.15.37:0/1501819396 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d2c0abe30 con 0x7f6d340a3d90 2024-04-26T00:30:34.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d32ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:30:34.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d32ffd640 1 -- 172.21.15.37:0/1501819396 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d2c0b0040 con 0x7f6d340a3d90 2024-04-26T00:30:34.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d3c32c640 5 monclient: authenticate success, global_id 124143 2024-04-26T00:30:34.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:34.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d32ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:30:34.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d32ffd640 10 monclient: dump: 2024-04-26T00:30:34.013 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:34.014 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:34.014 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:34.014 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:34.014 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:34.014 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:34.014 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:34.014 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:34.014 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:34.014 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:34.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d3c32c640 10 monclient: _renew_subs 2024-04-26T00:30:34.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d3c32c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:30:34.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d3c32c640 1 -- 172.21.15.37:0/1501819396 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d340a08b0 con 0x7f6d340a3d90 2024-04-26T00:30:34.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d388c5640 4 set_mon_vals no callback set 2024-04-26T00:30:34.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d3c32c640 10 monclient: _renew_subs 2024-04-26T00:30:34.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d3c32c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:30:34.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d3c32c640 1 -- 172.21.15.37:0/1501819396 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d341209e0 con 0x7f6d340a3d90 2024-04-26T00:30:34.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d3c32c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:30:34.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d3c32c640 1 --2- 172.21.15.37:0/1501819396 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d3413a450 0x7f6d341ceed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:34.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d3c32c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:34.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d3c32c640 1 -- 172.21.15.37:0/1501819396 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f6d3404f510 con 0x7f6d3413a450 2024-04-26T00:30:34.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d32ffd640 1 -- 172.21.15.37:0/1501819396 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f6d2c0bd020 con 0x7f6d340a3d90 2024-04-26T00:30:34.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d398c7640 1 --2- 172.21.15.37:0/1501819396 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d3413a450 0x7f6d341ceed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:34.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d398c7640 10 monclient: get_auth_request con 0x7f6d3413a450 auth_method 0 2024-04-26T00:30:34.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d398c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:34.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d398c7640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:34.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d398c7640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:34.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d398c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:34.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d398c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:34.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d32ffd640 1 --2- 172.21.15.37:0/1501819396 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6d20035ed0 0x7f6d20038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:34.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d398c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:34.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d32ffd640 1 -- 172.21.15.37:0/1501819396 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(404..404 src has 370..404) ==== 4007+0+0 (secure 0 0 0) 0x7f6d2c0e2dd0 con 0x7f6d340a3d90 2024-04-26T00:30:34.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d398c7640 10 monclient(hunting): handle_auth_done global_id 84503 payload 824 2024-04-26T00:30:34.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d398c7640 1 -- 172.21.15.37:0/1501819396 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6d340008e0 con 0x7f6d3413a450 2024-04-26T00:30:34.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d3a8c9640 1 --2- 172.21.15.37:0/1501819396 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6d20035ed0 0x7f6d20038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:34.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d3a8c9640 10 monclient: get_auth_request con 0x7f6d20035ed0 auth_method 0 2024-04-26T00:30:34.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d398c7640 1 -- 172.21.15.37:0/1501819396 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d3413a450 msgr2=0x7f6d341ceed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:34.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d398c7640 1 --2- 172.21.15.37:0/1501819396 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d3413a450 0x7f6d341ceed0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6d3413aa90 tx=0x7f6d2800ff70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:34.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d32ffd640 1 -- 172.21.15.37:0/1501819396 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d28018030 con 0x7f6d3413a450 2024-04-26T00:30:34.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:34.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d32ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:30:34.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.013+0000 7f6d32ffd640 10 monclient: dump: 2024-04-26T00:30:34.017 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:34.017 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:34.017 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:34.017 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:34.017 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:34.017 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:34.018 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:34.018 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:34.018 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:34.018 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:34.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.017+0000 7f6d32ffd640 1 -- 172.21.15.37:0/1501819396 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6d2800dd60 con 0x7f6d3413a450 2024-04-26T00:30:34.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.017+0000 7f6d32ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:30:34.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.017+0000 7f6d32ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:30:34.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.017+0000 7f6d32ffd640 1 -- 172.21.15.37:0/1501819396 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d3413a450 msgr2=0x7f6d341ceed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:34.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.017+0000 7f6d32ffd640 1 --2- 172.21.15.37:0/1501819396 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d3413a450 0x7f6d341ceed0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6d3413aa90 tx=0x7f6d2800ff70 comp rx=0 tx=0).stop 2024-04-26T00:30:34.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.017+0000 7f6d32ffd640 1 -- 172.21.15.37:0/1501819396 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d3413a450 msgr2=0x7f6d341ceed0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:30:34.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.017+0000 7f6d32ffd640 1 --2- 172.21.15.37:0/1501819396 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d3413a450 0x7f6d341ceed0 secure :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6d3413aa90 tx=0x7f6d2800ff70 comp rx=0 tx=0).stop 2024-04-26T00:30:34.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.017+0000 7f6d3a8c9640 1 --2- 172.21.15.37:0/1501819396 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6d20035ed0 0x7f6d20038390 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6d3413b620 tx=0x7f6d24009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:34.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.021+0000 7f6d3c32c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:30:34.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.021+0000 7f6d3c32c640 1 --2- 172.21.15.37:0/1501819396 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d341cf410 0x7f6d341cf820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:34.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.021+0000 7f6d3c32c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:34.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.021+0000 7f6d3c32c640 1 -- 172.21.15.37:0/1501819396 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f6d341cfd60 con 0x7f6d341cf410 2024-04-26T00:30:34.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.021+0000 7f6d398c7640 1 --2- 172.21.15.37:0/1501819396 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d341cf410 0x7f6d341cf820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:34.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.021+0000 7f6d398c7640 10 monclient: get_auth_request con 0x7f6d341cf410 auth_method 0 2024-04-26T00:30:34.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.021+0000 7f6d398c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:34.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.021+0000 7f6d398c7640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:34.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.021+0000 7f6d398c7640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:34.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.021+0000 7f6d398c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:34.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.021+0000 7f6d398c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:34.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.021+0000 7f6d398c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:34.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.021+0000 7f6d398c7640 10 monclient(hunting): handle_auth_done global_id 84506 payload 824 2024-04-26T00:30:34.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.021+0000 7f6d398c7640 1 -- 172.21.15.37:0/1501819396 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6d341205b0 con 0x7f6d341cf410 2024-04-26T00:30:34.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.025+0000 7f6d398c7640 1 -- 172.21.15.37:0/1501819396 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d341cf410 msgr2=0x7f6d341cf820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:34.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.025+0000 7f6d398c7640 1 --2- 172.21.15.37:0/1501819396 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d341cf410 0x7f6d341cf820 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6d28018000 tx=0x7f6d28003a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:34.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.025+0000 7f6d32ffd640 1 -- 172.21.15.37:0/1501819396 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d2801de00 con 0x7f6d341cf410 2024-04-26T00:30:34.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.025+0000 7f6d32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:34.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.025+0000 7f6d32ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:30:34.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.025+0000 7f6d32ffd640 10 monclient: dump: 2024-04-26T00:30:34.022 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:34.022 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:34.022 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:34.022 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:34.022 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:34.022 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:34.022 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:34.022 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:34.022 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:34.022 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:34.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.025+0000 7f6d32ffd640 1 -- 172.21.15.37:0/1501819396 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4719 (secure 0 0 0) 0x7f6d2801de00 con 0x7f6d341cf410 2024-04-26T00:30:34.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.025+0000 7f6d32ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:30:34.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.025+0000 7f6d32ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:30:34.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.025+0000 7f6d32ffd640 1 -- 172.21.15.37:0/1501819396 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d341cf410 msgr2=0x7f6d341cf820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:34.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.025+0000 7f6d32ffd640 1 --2- 172.21.15.37:0/1501819396 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d341cf410 0x7f6d341cf820 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6d28018000 tx=0x7f6d28003a80 comp rx=0 tx=0).stop 2024-04-26T00:30:34.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.025+0000 7f6d32ffd640 1 -- 172.21.15.37:0/1501819396 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d341cf410 msgr2=0x7f6d341cf820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:30:34.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.025+0000 7f6d32ffd640 1 --2- 172.21.15.37:0/1501819396 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d341cf410 0x7f6d341cf820 secure :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6d28018000 tx=0x7f6d28003a80 comp rx=0 tx=0).stop 2024-04-26T00:30:34.023 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:30:34.023 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:30:34.023 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:30:34.023 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:30:34.023 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 80, 2024-04-26T00:30:34.023 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 238960, 2024-04-26T00:30:34.023 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:30:34.023 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:30:34.024 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:30:34.024 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:30:34.024 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:30:34.024 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 2, 2024-04-26T00:30:34.024 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:30:34.024 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:30:34.024 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:30:34.024 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:30:34.024 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:30:34.024 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:30:34.024 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:30:34.024 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:30:34.024 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:30:34.024 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:30:34.025 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:30:34.025 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:30:34.025 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:30:34.025 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:30:34.025 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:30:34.025 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:30:34.025 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:30:34.025 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:30:34.025 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:30:34.025 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:30:34.025 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:30:34.025 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:30:34.025 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:30:34.025 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:30:34.025 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:30:34.026 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:30:34.026 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:30:34.026 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:30:34.026 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:34.026 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:34.026 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:30:34.026 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:30:34.026 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:30:34.026 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:30:34.026 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:30:34.026 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:30:34.026 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:30:34.026 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:30:34.026 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:30:34.027 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:30:34.027 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:30:34.027 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:30:34.027 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:30:34.027 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:30:34.027 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:30:34.027 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:30:34.027 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:30:34.027 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:30:34.027 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:30:34.027 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:30:34.027 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:30:34.027 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:30:34.027 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:30:34.028 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:30:34.028 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:30:34.028 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:30:34.028 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:30:34.028 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:30:34.028 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:30:34.028 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:30:34.028 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:34.028 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:30:34.028 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:34.028 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:30:34.028 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:30:34.028 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:30:34.028 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:30:34.029 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:34.029 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:30:34.029 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:30:34.029 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:34.029 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:34.029 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:34.029 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:30:34.029 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:30:34.029 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:34.029 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:34.029 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:34.029 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:34.029 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:30:34.029 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:30:34.029 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:30:34.030 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:30:34.030 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:30:34.030 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:34.030 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:34.030 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:30:34.030 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:30:34.030 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:30:34.030 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:30:34.030 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:34.030 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:30:34.030 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:30:34.030 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:34.030 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:34.031 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:34.031 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:30:34.031 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:30:34.031 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:34.031 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:34.031 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:34.031 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:34.031 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:30:34.031 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:30:34.031 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:30:34.031 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:30:34.031 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:30:34.031 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:34.031 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:34.032 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:30:34.032 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:30:34.032 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:30:34.032 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:30:34.032 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:34.032 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:30:34.032 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:30:34.032 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:34.032 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:34.032 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:34.032 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:30:34.032 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:30:34.032 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:34.032 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:34.033 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:34.033 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:34.033 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:30:34.033 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:30:34.033 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:30:34.033 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:30:34.033 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:30:34.033 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:34.033 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:34.033 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:34.033 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:30:34.033 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:30:34.033 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:34.033 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:30:34.033 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:30:34.034 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:30:34.034 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:34.034 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:30:34.034 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:30:34.034 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:34.034 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:30:34.034 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:30:34.034 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:30:34.034 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:34.034 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:30:34.034 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:30:34.034 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:34.034 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:30:34.034 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:30:34.035 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:30:34.035 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:34.035 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:34.035 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:34.035 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:30:34.035 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:30:34.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.025+0000 7f6d3c32c640 1 -- 172.21.15.37:0/1501819396 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6d20035ed0 msgr2=0x7f6d20038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:34.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.025+0000 7f6d3c32c640 1 --2- 172.21.15.37:0/1501819396 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6d20035ed0 0x7f6d20038390 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6d3413b620 tx=0x7f6d24009040 comp rx=0 tx=0).stop 2024-04-26T00:30:34.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.025+0000 7f6d3c32c640 10 monclient: shutdown 2024-04-26T00:30:34.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.025+0000 7f6d3c32c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:30:34.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.025+0000 7f6d3c32c640 1 -- 172.21.15.37:0/1501819396 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6d340a3d90 msgr2=0x7f6d34139750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:34.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.025+0000 7f6d3c32c640 1 --2- 172.21.15.37:0/1501819396 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6d340a3d90 0x7f6d34139750 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c01aa70 tx=0x7f6d2c01ac00 comp rx=0 tx=0).stop 2024-04-26T00:30:34.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.025+0000 7f6d3a8c9640 1 -- 172.21.15.37:0/1501819396 reap_dead start 2024-04-26T00:30:34.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.025+0000 7f6d3c32c640 1 -- 172.21.15.37:0/1501819396 shutdown_connections 2024-04-26T00:30:34.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.025+0000 7f6d3c32c640 1 --2- 172.21.15.37:0/1501819396 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6d340a3d90 0x7f6d34139750 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:34.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.025+0000 7f6d3c32c640 1 -- 172.21.15.37:0/1501819396 >> 172.21.15.37:0/1501819396 conn(0x7f6d340aa0a0 msgr2=0x7f6d340a2480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:30:34.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.025+0000 7f6d3c32c640 1 -- 172.21.15.37:0/1501819396 shutdown_connections 2024-04-26T00:30:34.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.025+0000 7f6d3c32c640 1 -- 172.21.15.37:0/1501819396 wait complete. 2024-04-26T00:30:34.037 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:30:34.038 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:30:34.038 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:30:34.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.169+0000 7f97f8317640 10 monclient: get_monmap_and_config 2024-04-26T00:30:34.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.169+0000 7f97f8317640 10 monclient: build_initial_monmap 2024-04-26T00:30:34.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.169+0000 7f97f8317640 10 monclient: monmap: 2024-04-26T00:30:34.160 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:30:34.161 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:34.161 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:30:34.172715+0000 2024-04-26T00:30:34.161 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:30:34.172715+0000 2024-04-26T00:30:34.161 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:30:34.161 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:34.161 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:30:34.161 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:30:34.161 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:30:34.161 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:30:34.161 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:30:34.161 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:34.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.169+0000 7f97f8317640 1 Processor -- start 2024-04-26T00:30:34.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.169+0000 7f97f8317640 1 -- start start 2024-04-26T00:30:34.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.169+0000 7f97f8317640 10 monclient: init 2024-04-26T00:30:34.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.169+0000 7f97f8317640 10 monclient: _reopen_session rank -1 2024-04-26T00:30:34.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.169+0000 7f97f8317640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-04-26T00:30:34.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.169+0000 7f97f8317640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97f00a0300 0x7f97f009fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:34.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.169+0000 7f97f8317640 10 monclient(hunting): picked mon.noname-b con 0x7f97f00a0300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:30:34.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.169+0000 7f97f8317640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f97f00a06e0 addr v1:172.21.15.37:6789/0 2024-04-26T00:30:34.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.169+0000 7f97f8317640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f97f01352b0 addr v1:172.21.15.45:6789/0 2024-04-26T00:30:34.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.169+0000 7f97f8317640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97f009c190 con 0x7f97f00a06e0 2024-04-26T00:30:34.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.169+0000 7f97f8317640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97f00a1110 con 0x7f97f01352b0 2024-04-26T00:30:34.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.169+0000 7f97f8317640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:34.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.169+0000 7f97f8317640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f97f00a13f0 con 0x7f97f00a0300 2024-04-26T00:30:34.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.169+0000 7f97f8317640 10 monclient(hunting): _renew_subs 2024-04-26T00:30:34.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.169+0000 7f97f8317640 10 monclient(hunting): authenticate will time out at 1615.478406s 2024-04-26T00:30:34.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.169+0000 7f97f58b2640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f97f00a06e0 0x7f97f0134ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:34776/0 (socket says 172.21.15.37:34776) 2024-04-26T00:30:34.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.169+0000 7f97f58b2640 1 -- 172.21.15.37:0/625704632 learned_addr learned my addr 172.21.15.37:0/625704632 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:30:34.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.169+0000 7f97f60b3640 1 --2- 172.21.15.37:0/625704632 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97f00a0300 0x7f97f009fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:34.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.169+0000 7f97f60b3640 10 monclient(hunting): get_auth_request con 0x7f97f00a0300 auth_method 0 2024-04-26T00:30:34.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.169+0000 7f97f60b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:34.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.169+0000 7f97f60b3640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:34.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.169+0000 7f97f60b3640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:34.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.169+0000 7f97f60b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:34.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.169+0000 7f97f60b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:34.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.169+0000 7f97f60b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:34.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.169+0000 7f97f60b3640 10 monclient(hunting): handle_auth_done global_id 84509 payload 274 2024-04-26T00:30:34.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.169+0000 7f97f60b3640 1 -- 172.21.15.37:0/625704632 >> v1:172.21.15.45:6789/0 conn(0x7f97f01352b0 legacy=0x7f97f013cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:30:34.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.169+0000 7f97f60b3640 1 -- 172.21.15.37:0/625704632 >> v1:172.21.15.37:6789/0 conn(0x7f97f00a06e0 legacy=0x7f97f0134ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:30:34.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.169+0000 7f97f60b3640 10 monclient: _finish_hunting 0 2024-04-26T00:30:34.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.169+0000 7f97f60b3640 1 monclient: found mon.noname-b 2024-04-26T00:30:34.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.169+0000 7f97f60b3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:30:34.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.169+0000 7f97f60b3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:30:34.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.169+0000 7f97f60b3640 1 -- 172.21.15.37:0/625704632 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97f00a16d0 con 0x7f97f00a0300 2024-04-26T00:30:34.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.173+0000 7f97f60b3640 1 -- 172.21.15.37:0/625704632 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97f00a0300 msgr2=0x7f97f009fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:34.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.173+0000 7f97f60b3640 1 --2- 172.21.15.37:0/625704632 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97f00a0300 0x7f97f009fc50 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f97e8009dc0 tx=0x7f97e801c7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9cca899cf730ad3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:34.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.173+0000 7f97f50b1640 1 -- 172.21.15.37:0/625704632 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e80b2070 con 0x7f97f00a0300 2024-04-26T00:30:34.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.173+0000 7f97f50b1640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:34.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.173+0000 7f97f50b1640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:30:34.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.173+0000 7f97f50b1640 10 monclient: dump: 2024-04-26T00:30:34.167 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:34.168 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.173+0000 7f97f50b1640 10 monclient: _finish_auth 0 2024-04-26T00:30:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.173+0000 7f97f50b1640 10 monclient: _check_auth_tickets 2024-04-26T00:30:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.173+0000 7f97f50b1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:30:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.173+0000 7f97f50b1640 1 -- 172.21.15.37:0/625704632 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97e80a5910 con 0x7f97f00a0300 2024-04-26T00:30:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.173+0000 7f97f50b1640 10 monclient: handle_config config(0 keys) 2024-04-26T00:30:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.173+0000 7f97f50b1640 1 -- 172.21.15.37:0/625704632 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e80a5ce0 con 0x7f97f00a0300 2024-04-26T00:30:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.173+0000 7f97f50b1640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:34.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.173+0000 7f97f50b1640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:30:34.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.173+0000 7f97f50b1640 10 monclient: dump: 2024-04-26T00:30:34.169 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:34.169 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:34.169 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:34.169 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:34.169 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:34.169 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:34.169 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:34.169 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:34.169 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:34.169 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:34.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.173+0000 7f97f8317640 5 monclient: authenticate success, global_id 84509 2024-04-26T00:30:34.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.173+0000 7f97f8317640 10 monclient: get_monmap_and_config success 2024-04-26T00:30:34.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.173+0000 7f97f8317640 4 set_mon_vals no callback set 2024-04-26T00:30:34.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f8317640 10 monclient: shutdown 2024-04-26T00:30:34.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f8317640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:30:34.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f8317640 1 -- 172.21.15.37:0/625704632 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97f00a0300 msgr2=0x7f97f009fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:34.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f8317640 1 --2- 172.21.15.37:0/625704632 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97f00a0300 0x7f97f009fc50 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f97e8009dc0 tx=0x7f97e801c7d0 comp rx=0 tx=0).stop 2024-04-26T00:30:34.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f8317640 1 -- 172.21.15.37:0/625704632 shutdown_connections 2024-04-26T00:30:34.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f8317640 1 --2- 172.21.15.37:0/625704632 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97f00a0300 0x7f97f009fc50 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:34.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f8317640 1 -- 172.21.15.37:0/625704632 >> 172.21.15.37:0/625704632 conn(0x7f97f00a1d50 msgr2=0x7f97f00a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:30:34.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f8317640 1 -- 172.21.15.37:0/625704632 shutdown_connections 2024-04-26T00:30:34.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f8317640 1 -- 172.21.15.37:0/625704632 wait complete. 2024-04-26T00:30:34.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f8317640 10 monclient: build_initial_monmap 2024-04-26T00:30:34.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f8317640 10 monclient: monmap: 2024-04-26T00:30:34.175 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:30:34.175 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:30:34.175 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:30:34.175 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:30:34.176 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:30:34.176 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:30:34.176 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:30:34.176 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:30:34.176 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:30:34.176 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:34.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f8317640 1 Processor -- start 2024-04-26T00:30:34.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f8317640 1 -- start start 2024-04-26T00:30:34.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f8317640 10 monclient: init 2024-04-26T00:30:34.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f8317640 10 monclient: _reopen_session rank -1 2024-04-26T00:30:34.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f8317640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:30:34.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f8317640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97f00a0300 0x7f97f012ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:34.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f8317640 10 monclient(hunting): picked mon.noname-c con 0x7f97f00a0300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:30:34.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f8317640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f97f00a06e0 0x7f97f012d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:34.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f8317640 10 monclient(hunting): picked mon.noname-a con 0x7f97f00a06e0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:30:34.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f8317640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97f01352b0 0x7f97f0135c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:34.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f8317640 10 monclient(hunting): picked mon.noname-b con 0x7f97f01352b0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:30:34.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f8317640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:34.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f8317640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f97f0140fa0 con 0x7f97f00a06e0 2024-04-26T00:30:34.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f8317640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:34.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f8317640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f97f0140e20 con 0x7f97f01352b0 2024-04-26T00:30:34.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f8317640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:34.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f8317640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f97f0141120 con 0x7f97f00a0300 2024-04-26T00:30:34.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f8317640 10 monclient(hunting): _renew_subs 2024-04-26T00:30:34.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f8317640 10 monclient(hunting): authenticate will time out at 1615.486024s 2024-04-26T00:30:34.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f60b3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97f00a0300 0x7f97f012ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:34.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f60b3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97f00a0300 0x7f97f012ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:56552/0 (socket says 172.21.15.37:56552) 2024-04-26T00:30:34.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f60b3640 1 -- 172.21.15.37:0/1649045503 learned_addr learned my addr 172.21.15.37:0/1649045503 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:30:34.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f60b3640 10 monclient(hunting): get_auth_request con 0x7f97f00a0300 auth_method 0 2024-04-26T00:30:34.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f60b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:34.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f60b3640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:34.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f60b3640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:34.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f58b2640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f97f00a06e0 0x7f97f012d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:34.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f58b2640 10 monclient(hunting): get_auth_request con 0x7f97f00a06e0 auth_method 0 2024-04-26T00:30:34.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f58b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:34.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f58b2640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:34.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f58b2640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:34.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f60b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:34.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f60b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:34.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f60b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:34.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f58b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:34.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f58b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:34.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f58b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:34.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f68b4640 1 --2- 172.21.15.37:0/1649045503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97f01352b0 0x7f97f0135c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:34.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f60b3640 10 monclient(hunting): handle_auth_done global_id 84512 payload 824 2024-04-26T00:30:34.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f60b3640 1 -- 172.21.15.37:0/1649045503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97f01352b0 msgr2=0x7f97f0135c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:34.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f60b3640 1 --2- 172.21.15.37:0/1649045503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97f01352b0 0x7f97f0135c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:34.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f60b3640 1 -- 172.21.15.37:0/1649045503 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f97f00a06e0 msgr2=0x7f97f012d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:34.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f60b3640 1 --2- 172.21.15.37:0/1649045503 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f97f00a06e0 0x7f97f012d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:34.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f60b3640 10 monclient: _finish_hunting 0 2024-04-26T00:30:34.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f60b3640 1 monclient: found mon.noname-c 2024-04-26T00:30:34.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f60b3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:30:34.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f60b3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:30:34.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f60b3640 1 -- 172.21.15.37:0/1649045503 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97f01349e0 con 0x7f97f00a0300 2024-04-26T00:30:34.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f58b2640 1 --2- 172.21.15.37:0/1649045503 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f97f00a06e0 0x7f97f012d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:30:34.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f68b4640 10 monclient: get_auth_request con 0x7f97f01352b0 auth_method 0 2024-04-26T00:30:34.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.177+0000 7f97f68b4640 1 --2- 172.21.15.37:0/1649045503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97f01352b0 0x7f97f0135c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:30:34.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97f60b3640 1 -- 172.21.15.37:0/1649045503 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97f00a0300 msgr2=0x7f97f012ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:34.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97f60b3640 1 --2- 172.21.15.37:0/1649045503 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97f00a0300 0x7f97f012ce00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f97e801d070 tx=0x7f97e8019570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:34.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97eeffd640 1 -- 172.21.15.37:0/1649045503 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e80bd020 con 0x7f97f00a0300 2024-04-26T00:30:34.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:34.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97eeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:30:34.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97eeffd640 10 monclient: dump: 2024-04-26T00:30:34.180 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:34.180 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:34.180 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:34.180 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:34.180 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:34.180 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:34.180 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:34.180 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:34.180 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:34.180 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:34.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97eeffd640 10 monclient: _finish_auth 0 2024-04-26T00:30:34.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97eeffd640 10 monclient: _check_auth_tickets 2024-04-26T00:30:34.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:30:34.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97eeffd640 1 -- 172.21.15.37:0/1649045503 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97e80b8880 con 0x7f97f00a0300 2024-04-26T00:30:34.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97f8317640 5 monclient: authenticate success, global_id 84512 2024-04-26T00:30:34.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97f8317640 10 monclient: _renew_subs 2024-04-26T00:30:34.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97f8317640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:30:34.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97f8317640 1 -- 172.21.15.37:0/1649045503 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97f0137e50 con 0x7f97f00a0300 2024-04-26T00:30:34.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97eeffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:30:34.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97eeffd640 1 -- 172.21.15.37:0/1649045503 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e80b2040 con 0x7f97f00a0300 2024-04-26T00:30:34.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:34.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97eeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:30:34.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97eeffd640 10 monclient: dump: 2024-04-26T00:30:34.181 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:34.181 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:34.181 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:34.182 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:34.182 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:34.182 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:34.182 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:34.182 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:34.182 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:34.182 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:34.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97f48b0640 4 set_mon_vals no callback set 2024-04-26T00:30:34.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97f8317640 10 monclient: _renew_subs 2024-04-26T00:30:34.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97f8317640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:30:34.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97f8317640 1 -- 172.21.15.37:0/1649045503 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f97f0133fb0 con 0x7f97f00a0300 2024-04-26T00:30:34.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97f8317640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:30:34.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97f8317640 1 --2- 172.21.15.37:0/1649045503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97f01390f0 0x7f97f01361a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:34.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97f8317640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:34.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97f8317640 1 -- 172.21.15.37:0/1649045503 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f97f006e560 con 0x7f97f01390f0 2024-04-26T00:30:34.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97eeffd640 1 -- 172.21.15.37:0/1649045503 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f97e80c0020 con 0x7f97f00a0300 2024-04-26T00:30:34.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97f58b2640 1 --2- 172.21.15.37:0/1649045503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97f01390f0 0x7f97f01361a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:34.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97f58b2640 10 monclient: get_auth_request con 0x7f97f01390f0 auth_method 0 2024-04-26T00:30:34.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97f58b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:34.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97f58b2640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:34.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97f58b2640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:34.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97eeffd640 1 --2- 172.21.15.37:0/1649045503 >> v2:172.21.15.45:6812/2379918623 conn(0x7f97dc035f00 0x7f97dc0383c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:34.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97eeffd640 1 -- 172.21.15.37:0/1649045503 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(404..404 src has 370..404) ==== 4007+0+0 (secure 0 0 0) 0x7f97e80e3aa0 con 0x7f97f00a0300 2024-04-26T00:30:34.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97f58b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:34.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97f58b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:34.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97f58b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:34.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97f68b4640 1 --2- 172.21.15.37:0/1649045503 >> v2:172.21.15.45:6812/2379918623 conn(0x7f97dc035f00 0x7f97dc0383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:34.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97f68b4640 10 monclient: get_auth_request con 0x7f97dc035f00 auth_method 0 2024-04-26T00:30:34.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97f58b2640 10 monclient(hunting): handle_auth_done global_id 114199 payload 824 2024-04-26T00:30:34.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.181+0000 7f97f58b2640 1 -- 172.21.15.37:0/1649045503 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f97f004f510 con 0x7f97f01390f0 2024-04-26T00:30:34.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.185+0000 7f97f58b2640 1 -- 172.21.15.37:0/1649045503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97f01390f0 msgr2=0x7f97f01361a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:34.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.185+0000 7f97f58b2640 1 --2- 172.21.15.37:0/1649045503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97f01390f0 0x7f97f01361a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f97e4007270 tx=0x7f97e4007730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:34.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.185+0000 7f97f68b4640 1 --2- 172.21.15.37:0/1649045503 >> v2:172.21.15.45:6812/2379918623 conn(0x7f97dc035f00 0x7f97dc0383c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f97e0003dc0 tx=0x7f97e0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:34.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.185+0000 7f97eeffd640 1 -- 172.21.15.37:0/1649045503 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e400fce0 con 0x7f97f01390f0 2024-04-26T00:30:34.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.185+0000 7f97eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:34.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.185+0000 7f97eeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:30:34.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.185+0000 7f97eeffd640 10 monclient: dump: 2024-04-26T00:30:34.186 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:34.186 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:34.186 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:34.187 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:34.187 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:34.187 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:34.187 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:34.187 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:34.187 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:34.187 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:34.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.185+0000 7f97eeffd640 1 -- 172.21.15.37:0/1649045503 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f97e400fce0 con 0x7f97f01390f0 2024-04-26T00:30:34.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.185+0000 7f97eeffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:30:34.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.185+0000 7f97eeffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:30:34.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.185+0000 7f97eeffd640 1 -- 172.21.15.37:0/1649045503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97f01390f0 msgr2=0x7f97f01361a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:34.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.185+0000 7f97eeffd640 1 --2- 172.21.15.37:0/1649045503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97f01390f0 0x7f97f01361a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f97e4007270 tx=0x7f97e4007730 comp rx=0 tx=0).stop 2024-04-26T00:30:34.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.185+0000 7f97eeffd640 1 -- 172.21.15.37:0/1649045503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97f01390f0 msgr2=0x7f97f01361a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:30:34.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.185+0000 7f97eeffd640 1 --2- 172.21.15.37:0/1649045503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97f01390f0 0x7f97f01361a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:34.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.193+0000 7f97f8317640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:30:34.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.193+0000 7f97f8317640 1 --2- 172.21.15.37:0/1649045503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97f01366e0 0x7f97f006fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:34.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.193+0000 7f97f8317640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:34.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.193+0000 7f97f8317640 1 -- 172.21.15.37:0/1649045503 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f97f0136af0 con 0x7f97f01366e0 2024-04-26T00:30:34.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.193+0000 7f97f58b2640 1 --2- 172.21.15.37:0/1649045503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97f01366e0 0x7f97f006fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:34.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.193+0000 7f97f58b2640 10 monclient: get_auth_request con 0x7f97f01366e0 auth_method 0 2024-04-26T00:30:34.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.193+0000 7f97f58b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:34.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.193+0000 7f97f58b2640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:34.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.193+0000 7f97f58b2640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:34.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.193+0000 7f97f58b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:34.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.193+0000 7f97f58b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:34.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.193+0000 7f97f58b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:34.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.193+0000 7f97f58b2640 10 monclient(hunting): handle_auth_done global_id 114202 payload 824 2024-04-26T00:30:34.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.193+0000 7f97f58b2640 1 -- 172.21.15.37:0/1649045503 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f97f009eb60 con 0x7f97f01366e0 2024-04-26T00:30:34.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.197+0000 7f97f58b2640 1 -- 172.21.15.37:0/1649045503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97f01366e0 msgr2=0x7f97f006fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:34.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.197+0000 7f97f58b2640 1 --2- 172.21.15.37:0/1649045503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97f01366e0 0x7f97f006fff0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f97e4017000 tx=0x7f97e4004950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:34.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.197+0000 7f97eeffd640 1 -- 172.21.15.37:0/1649045503 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e4006450 con 0x7f97f01366e0 2024-04-26T00:30:34.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.197+0000 7f97eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:34.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.197+0000 7f97eeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:30:34.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.197+0000 7f97eeffd640 10 monclient: dump: 2024-04-26T00:30:34.190 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:34.190 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:34.190 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:34.190 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:34.190 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:34.190 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:34.190 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:34.190 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:34.190 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:34.190 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:34.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.197+0000 7f97eeffd640 1 -- 172.21.15.37:0/1649045503 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4716 (secure 0 0 0) 0x7f97e4006450 con 0x7f97f01366e0 2024-04-26T00:30:34.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.197+0000 7f97eeffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:30:34.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.197+0000 7f97eeffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:30:34.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.197+0000 7f97eeffd640 1 -- 172.21.15.37:0/1649045503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97f01366e0 msgr2=0x7f97f006fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:34.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.197+0000 7f97eeffd640 1 --2- 172.21.15.37:0/1649045503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97f01366e0 0x7f97f006fff0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f97e4017000 tx=0x7f97e4004950 comp rx=0 tx=0).stop 2024-04-26T00:30:34.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.197+0000 7f97eeffd640 1 -- 172.21.15.37:0/1649045503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97f01366e0 msgr2=0x7f97f006fff0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:30:34.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.197+0000 7f97eeffd640 1 --2- 172.21.15.37:0/1649045503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f97f01366e0 0x7f97f006fff0 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f97e4017000 tx=0x7f97e4004950 comp rx=0 tx=0).stop 2024-04-26T00:30:34.191 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:30:34.191 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:30:34.191 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:30:34.191 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:30:34.191 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 80, 2024-04-26T00:30:34.191 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 84283, 2024-04-26T00:30:34.191 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:30:34.192 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:30:34.192 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:30:34.192 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:30:34.192 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:30:34.192 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 3, 2024-04-26T00:30:34.192 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:30:34.192 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:30:34.192 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:30:34.192 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:30:34.192 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:30:34.192 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:30:34.192 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:30:34.192 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:30:34.192 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:30:34.192 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:30:34.193 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:30:34.193 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:30:34.193 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:30:34.193 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:30:34.193 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:30:34.193 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:30:34.193 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:30:34.193 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:30:34.193 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:30:34.193 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:30:34.193 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:30:34.193 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:30:34.193 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:30:34.193 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:30:34.194 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:30:34.194 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:30:34.194 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:30:34.194 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:30:34.194 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:34.194 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:34.194 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:30:34.194 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:30:34.194 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:30:34.194 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:30:34.194 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:30:34.194 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:30:34.194 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:30:34.194 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:30:34.194 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:30:34.195 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:30:34.195 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:30:34.195 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:30:34.195 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:30:34.195 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:30:34.195 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:30:34.195 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:30:34.195 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:30:34.195 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:30:34.195 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:30:34.195 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:30:34.195 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:30:34.195 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:30:34.195 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:30:34.196 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:30:34.196 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:30:34.196 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:30:34.196 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:30:34.196 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:30:34.196 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:30:34.196 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:30:34.196 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:34.196 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:30:34.196 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:34.196 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:30:34.196 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:30:34.196 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:30:34.196 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:30:34.196 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:34.196 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:30:34.197 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:30:34.197 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:34.197 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:34.197 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:34.197 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:30:34.197 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:30:34.197 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:34.197 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:34.197 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:34.197 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:34.197 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:30:34.197 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:30:34.197 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:30:34.197 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:30:34.198 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:30:34.198 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:34.198 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:34.198 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:30:34.198 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:30:34.198 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:30:34.198 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:30:34.198 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:34.198 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:30:34.198 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:30:34.198 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:34.198 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:34.198 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:34.198 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:30:34.199 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:30:34.199 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:34.199 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:34.199 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:34.199 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:34.199 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:30:34.199 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:30:34.199 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:30:34.199 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:30:34.199 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:30:34.199 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:34.199 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:34.199 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:30:34.199 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:30:34.199 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:30:34.200 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:30:34.200 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:34.200 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:30:34.200 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:30:34.200 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:34.200 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:34.200 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:34.200 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:30:34.200 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:30:34.200 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:34.200 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:34.200 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:34.200 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:34.200 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:30:34.200 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:30:34.201 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:30:34.201 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:30:34.201 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:30:34.201 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:34.201 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:34.201 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:34.201 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:30:34.201 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:30:34.201 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:34.201 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:30:34.201 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:30:34.201 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:30:34.201 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:34.201 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:30:34.201 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:30:34.202 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:34.202 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:30:34.202 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:30:34.202 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:30:34.202 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:34.202 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:30:34.202 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:30:34.202 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:34.202 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:30:34.202 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:30:34.202 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:30:34.202 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:34.202 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:34.202 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:34.203 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:30:34.203 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:30:34.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.197+0000 7f97f8317640 1 -- 172.21.15.37:0/1649045503 >> v2:172.21.15.45:6812/2379918623 conn(0x7f97dc035f00 msgr2=0x7f97dc0383c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:34.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.197+0000 7f97f8317640 1 --2- 172.21.15.37:0/1649045503 >> v2:172.21.15.45:6812/2379918623 conn(0x7f97dc035f00 0x7f97dc0383c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f97e0003dc0 tx=0x7f97e0009040 comp rx=0 tx=0).stop 2024-04-26T00:30:34.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.197+0000 7f97f8317640 10 monclient: shutdown 2024-04-26T00:30:34.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.197+0000 7f97f8317640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:30:34.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.197+0000 7f97f8317640 1 -- 172.21.15.37:0/1649045503 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97f00a0300 msgr2=0x7f97f012ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:34.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.197+0000 7f97f8317640 1 --2- 172.21.15.37:0/1649045503 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97f00a0300 0x7f97f012ce00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f97e801d070 tx=0x7f97e8019570 comp rx=0 tx=0).stop 2024-04-26T00:30:34.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.197+0000 7f97f68b4640 1 -- 172.21.15.37:0/1649045503 reap_dead start 2024-04-26T00:30:34.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.197+0000 7f97f8317640 1 -- 172.21.15.37:0/1649045503 shutdown_connections 2024-04-26T00:30:34.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.197+0000 7f97f8317640 1 --2- 172.21.15.37:0/1649045503 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f97f00a0300 0x7f97f012ce00 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:34.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.197+0000 7f97f8317640 1 -- 172.21.15.37:0/1649045503 >> 172.21.15.37:0/1649045503 conn(0x7f97f00a1d50 msgr2=0x7f97f00a3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:30:34.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.197+0000 7f97f8317640 1 -- 172.21.15.37:0/1649045503 shutdown_connections 2024-04-26T00:30:34.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.197+0000 7f97f8317640 1 -- 172.21.15.37:0/1649045503 wait complete. 2024-04-26T00:30:34.206 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:30:34.207 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-04-26T00:30:34.207 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-26T00:30:34.207 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-04-26T00:30:34.207 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-04-26T00:30:34.207 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-26T00:30:34.207 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T00:30:34.232 INFO:tasks.ceph.mon.b:Stopped 2024-04-26T00:30:34.232 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-26T00:30:34.232 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:30:34.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.365+0000 7f049d2b3640 10 monclient: get_monmap_and_config 2024-04-26T00:30:34.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.365+0000 7f049d2b3640 10 monclient: build_initial_monmap 2024-04-26T00:30:34.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.365+0000 7f049d2b3640 10 monclient: monmap: 2024-04-26T00:30:34.357 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:30:34.357 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:34.357 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:30:34.368968+0000 2024-04-26T00:30:34.357 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:30:34.368968+0000 2024-04-26T00:30:34.357 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:30:34.357 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:34.357 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:30:34.357 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:30:34.357 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:30:34.357 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:30:34.357 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:30:34.357 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:34.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.365+0000 7f049d2b3640 1 Processor -- start 2024-04-26T00:30:34.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.365+0000 7f049d2b3640 1 -- start start 2024-04-26T00:30:34.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.365+0000 7f049d2b3640 10 monclient: init 2024-04-26T00:30:34.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.365+0000 7f049d2b3640 10 monclient: _reopen_session rank -1 2024-04-26T00:30:34.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.365+0000 7f049d2b3640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-04-26T00:30:34.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.365+0000 7f049d2b3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f04980a0260 addr v1:172.21.15.37:6789/0 2024-04-26T00:30:34.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.365+0000 7f049d2b3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0498137e40 addr v1:172.21.15.45:6789/0 2024-04-26T00:30:34.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.365+0000 7f049d2b3640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f049813b950 0x7f049813dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:34.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.365+0000 7f049d2b3640 10 monclient(hunting): picked mon.noname-a con 0x7f049813b950 addr v2:172.21.15.37:3300/0 2024-04-26T00:30:34.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.365+0000 7f049d2b3640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04980a1130 con 0x7f04980a0260 2024-04-26T00:30:34.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.365+0000 7f049d2b3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f049806cff0 con 0x7f0498137e40 2024-04-26T00:30:34.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.365+0000 7f049d2b3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:34.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.365+0000 7f049d2b3640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f04980a0d50 con 0x7f049813b950 2024-04-26T00:30:34.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.365+0000 7f049d2b3640 10 monclient(hunting): _renew_subs 2024-04-26T00:30:34.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.365+0000 7f049d2b3640 10 monclient(hunting): authenticate will time out at 1615.674839s 2024-04-26T00:30:34.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.365+0000 7f04977fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f049813b950 0x7f049813dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:34.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.365+0000 7f04977fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f049813b950 0x7f049813dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:43354/0 (socket says 172.21.15.37:43354) 2024-04-26T00:30:34.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.365+0000 7f04977fe640 1 -- 172.21.15.37:0/1586120913 learned_addr learned my addr 172.21.15.37:0/1586120913 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:30:34.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.365+0000 7f04977fe640 10 monclient(hunting): get_auth_request con 0x7f049813b950 auth_method 0 2024-04-26T00:30:34.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.365+0000 7f04977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:34.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.365+0000 7f04977fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:34.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.365+0000 7f04977fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:34.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.365+0000 7f0496ffd640 1 --1- 172.21.15.37:0/1586120913 >> v1:172.21.15.37:6789/0 conn(0x7f04980a0260 0x7f04980a0640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:34788/0 (socket says 172.21.15.37:34788) 2024-04-26T00:30:34.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.365+0000 7f04967fc640 1 -- 172.21.15.37:0/1586120913 >> v1:172.21.15.45:6789/0 conn(0x7f0498137e40 legacy=0x7f049813a230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:30:34.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.365+0000 7f04977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:34.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.365+0000 7f04977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:34.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.365+0000 7f04977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:34.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.365+0000 7f04977fe640 10 monclient(hunting): handle_auth_done global_id 124149 payload 274 2024-04-26T00:30:34.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.365+0000 7f04977fe640 1 -- 172.21.15.37:0/1586120913 >> v1:172.21.15.45:6789/0 conn(0x7f0498137e40 legacy=0x7f049813a230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:30:34.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.365+0000 7f04977fe640 1 -- 172.21.15.37:0/1586120913 >> v1:172.21.15.37:6789/0 conn(0x7f04980a0260 legacy=0x7f04980a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:30:34.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.365+0000 7f04977fe640 10 monclient: _finish_hunting 0 2024-04-26T00:30:34.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.365+0000 7f04977fe640 1 monclient: found mon.noname-a 2024-04-26T00:30:34.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.365+0000 7f04977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:30:34.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.365+0000 7f04977fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:30:34.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.365+0000 7f04977fe640 1 -- 172.21.15.37:0/1586120913 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f049809bc50 con 0x7f049813b950 2024-04-26T00:30:34.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.369+0000 7f04977fe640 1 -- 172.21.15.37:0/1586120913 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f049813b950 msgr2=0x7f049813dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:34.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.369+0000 7f04977fe640 1 --2- 172.21.15.37:0/1586120913 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f049813b950 0x7f049813dd40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0488009870 tx=0x7f048801c1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa99f0e2b6c79dcf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:34.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f0495ffb640 1 -- 172.21.15.37:0/1586120913 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04880b0070 con 0x7f049813b950 2024-04-26T00:30:34.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f0495ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:34.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f0495ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:30:34.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f0495ffb640 10 monclient: dump: 2024-04-26T00:30:34.365 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:34.365 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:34.365 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:34.365 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:34.365 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:34.365 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:34.365 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:34.365 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:34.365 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:34.365 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:34.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f0495ffb640 10 monclient: _finish_auth 0 2024-04-26T00:30:34.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f0495ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:30:34.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f0495ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:30:34.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f049d2b3640 5 monclient: authenticate success, global_id 124149 2024-04-26T00:30:34.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f049d2b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:30:34.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f0495ffb640 1 -- 172.21.15.37:0/1586120913 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04880ab600 con 0x7f049813b950 2024-04-26T00:30:34.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f0495ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:30:34.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f049d2b3640 10 monclient: get_monmap_and_config success 2024-04-26T00:30:34.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f049d2b3640 4 set_mon_vals no callback set 2024-04-26T00:30:34.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f0495ffb640 1 -- 172.21.15.37:0/1586120913 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04880ab9d0 con 0x7f049813b950 2024-04-26T00:30:34.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f0495ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:34.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f0495ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:30:34.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f0495ffb640 10 monclient: dump: 2024-04-26T00:30:34.366 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:34.366 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:34.366 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:34.366 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:34.367 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:34.367 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:34.367 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:34.367 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:34.367 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:34.367 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:34.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f049d2b3640 10 monclient: shutdown 2024-04-26T00:30:34.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f049d2b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:30:34.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f049d2b3640 1 -- 172.21.15.37:0/1586120913 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f049813b950 msgr2=0x7f049813dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:34.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f049d2b3640 1 --2- 172.21.15.37:0/1586120913 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f049813b950 0x7f049813dd40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0488009870 tx=0x7f048801c1c0 comp rx=0 tx=0).stop 2024-04-26T00:30:34.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f049d2b3640 1 -- 172.21.15.37:0/1586120913 shutdown_connections 2024-04-26T00:30:34.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f049d2b3640 1 --2- 172.21.15.37:0/1586120913 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f049813b950 0x7f049813dd40 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:34.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f049d2b3640 1 -- 172.21.15.37:0/1586120913 >> 172.21.15.37:0/1586120913 conn(0x7f04980a1ad0 msgr2=0x7f04980a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:30:34.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f049d2b3640 1 -- 172.21.15.37:0/1586120913 shutdown_connections 2024-04-26T00:30:34.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f049d2b3640 1 -- 172.21.15.37:0/1586120913 wait complete. 2024-04-26T00:30:34.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f049d2b3640 10 monclient: build_initial_monmap 2024-04-26T00:30:34.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f049d2b3640 10 monclient: monmap: 2024-04-26T00:30:34.372 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:30:34.372 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:30:34.372 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:30:34.372 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:30:34.372 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:30:34.372 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:30:34.372 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:30:34.372 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:30:34.372 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:30:34.372 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:34.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f049d2b3640 1 Processor -- start 2024-04-26T00:30:34.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f049d2b3640 1 -- start start 2024-04-26T00:30:34.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f049d2b3640 10 monclient: init 2024-04-26T00:30:34.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f049d2b3640 10 monclient: _reopen_session rank -1 2024-04-26T00:30:34.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f049d2b3640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:30:34.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f049d2b3640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f04980a0260 0x7f04981ccc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:34.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f049d2b3640 10 monclient(hunting): picked mon.noname-a con 0x7f04980a0260 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:30:34.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f049d2b3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0498137e40 0x7f0498136940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:34.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f049d2b3640 10 monclient(hunting): picked mon.noname-b con 0x7f0498137e40 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:30:34.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f049d2b3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f049813b950 0x7f049811c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:34.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f049d2b3640 10 monclient(hunting): picked mon.noname-c con 0x7f049813b950 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:30:34.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f049d2b3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:34.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f049d2b3640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f049809f0f0 con 0x7f04980a0260 2024-04-26T00:30:34.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f049d2b3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:34.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f049d2b3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f049809f4c0 con 0x7f0498137e40 2024-04-26T00:30:34.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f049d2b3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:34.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f049d2b3640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f04980a00a0 con 0x7f049813b950 2024-04-26T00:30:34.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f049d2b3640 10 monclient(hunting): _renew_subs 2024-04-26T00:30:34.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f049d2b3640 10 monclient(hunting): authenticate will time out at 1615.682636s 2024-04-26T00:30:34.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f04977fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f049813b950 0x7f049811c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:34.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f04967fc640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0498137e40 msgr2=0x7f0498136940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:30:34.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f04977fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f049813b950 0x7f049811c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:56556/0 (socket says 172.21.15.37:56556) 2024-04-26T00:30:34.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f04967fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0498137e40 0x7f0498136940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:30:34.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f04977fe640 1 -- 172.21.15.37:0/2091677712 learned_addr learned my addr 172.21.15.37:0/2091677712 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:30:34.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f04977fe640 10 monclient(hunting): get_auth_request con 0x7f049813b950 auth_method 0 2024-04-26T00:30:34.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f04977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:34.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f04977fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:34.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f04977fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:34.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f0496ffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f04980a0260 0x7f04981ccc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:34.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f0496ffd640 10 monclient(hunting): get_auth_request con 0x7f04980a0260 auth_method 0 2024-04-26T00:30:34.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f0496ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:34.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f0496ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:34.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f0496ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:34.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f04977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:34.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f04977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:34.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f04977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:34.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f0496ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:34.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f0496ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:34.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f0496ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:34.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f0496ffd640 10 monclient(hunting): handle_auth_done global_id 124152 payload 824 2024-04-26T00:30:34.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f0496ffd640 1 -- 172.21.15.37:0/2091677712 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f049813b950 msgr2=0x7f049811c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:34.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f0496ffd640 1 --2- 172.21.15.37:0/2091677712 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f049813b950 0x7f049811c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:34.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f0496ffd640 1 -- 172.21.15.37:0/2091677712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0498137e40 msgr2=0x7f0498136940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:30:34.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f0496ffd640 1 --2- 172.21.15.37:0/2091677712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0498137e40 0x7f0498136940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:34.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f0496ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:30:34.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f0496ffd640 1 monclient: found mon.noname-a 2024-04-26T00:30:34.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f0496ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:30:34.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f0496ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:30:34.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f0496ffd640 1 -- 172.21.15.37:0/2091677712 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04981202c0 con 0x7f04980a0260 2024-04-26T00:30:34.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.373+0000 7f04977fe640 1 --2- 172.21.15.37:0/2091677712 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f049813b950 0x7f049811c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:30:34.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.377+0000 7f0496ffd640 1 -- 172.21.15.37:0/2091677712 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f04980a0260 msgr2=0x7f04981ccc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:34.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.377+0000 7f0496ffd640 1 --2- 172.21.15.37:0/2091677712 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f04980a0260 0x7f04981ccc40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f049000c7b0 tx=0x7f049000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:34.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.377+0000 7f048f7fe640 1 -- 172.21.15.37:0/2091677712 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0490017070 con 0x7f04980a0260 2024-04-26T00:30:34.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.377+0000 7f048f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:34.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.377+0000 7f048f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:30:34.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.377+0000 7f048f7fe640 10 monclient: dump: 2024-04-26T00:30:34.376 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:34.376 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:34.376 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:34.376 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:34.376 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:34.377 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:34.377 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:34.377 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:34.377 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:34.377 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:34.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.377+0000 7f048f7fe640 10 monclient: _finish_auth 0 2024-04-26T00:30:34.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.377+0000 7f048f7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:30:34.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.377+0000 7f048f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:30:34.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.377+0000 7f048f7fe640 1 -- 172.21.15.37:0/2091677712 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f049000a800 con 0x7f04980a0260 2024-04-26T00:30:34.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.377+0000 7f049d2b3640 5 monclient: authenticate success, global_id 124152 2024-04-26T00:30:34.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.377+0000 7f049d2b3640 10 monclient: _renew_subs 2024-04-26T00:30:34.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.377+0000 7f049d2b3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:30:34.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.377+0000 7f049d2b3640 1 -- 172.21.15.37:0/2091677712 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f049813be50 con 0x7f04980a0260 2024-04-26T00:30:34.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.377+0000 7f048f7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:30:34.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.377+0000 7f048f7fe640 1 -- 172.21.15.37:0/2091677712 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049000ce60 con 0x7f04980a0260 2024-04-26T00:30:34.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.377+0000 7f048f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:34.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.377+0000 7f048f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:30:34.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.377+0000 7f048f7fe640 10 monclient: dump: 2024-04-26T00:30:34.378 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:34.378 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:34.378 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:34.378 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:34.378 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:34.378 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:34.378 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:34.378 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:34.378 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:34.378 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:34.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.377+0000 7f04957fa640 4 set_mon_vals no callback set 2024-04-26T00:30:34.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.377+0000 7f049d2b3640 10 monclient: _renew_subs 2024-04-26T00:30:34.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.377+0000 7f049d2b3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:30:34.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.377+0000 7f049d2b3640 1 -- 172.21.15.37:0/2091677712 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f049811ffe0 con 0x7f04980a0260 2024-04-26T00:30:34.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.377+0000 7f048f7fe640 1 -- 172.21.15.37:0/2091677712 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f049001a070 con 0x7f04980a0260 2024-04-26T00:30:34.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.377+0000 7f049d2b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:30:34.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.377+0000 7f049d2b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:30:34.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.377+0000 7f049d2b3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:30:34.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.377+0000 7f049d2b3640 1 -- 172.21.15.37:0/2091677712 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0498066f40 con 0x7f04980a0260 2024-04-26T00:30:34.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.377+0000 7f048f7fe640 1 --2- 172.21.15.37:0/2091677712 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0478035e80 0x7f0478038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:34.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.377+0000 7f048f7fe640 1 -- 172.21.15.37:0/2091677712 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(404..404 src has 370..404) ==== 4007+0+0 (secure 0 0 0) 0x7f0490048220 con 0x7f04980a0260 2024-04-26T00:30:34.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.377+0000 7f04967fc640 1 --2- 172.21.15.37:0/2091677712 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0478035e80 0x7f0478038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:34.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.377+0000 7f04967fc640 10 monclient: get_auth_request con 0x7f0478035e80 auth_method 0 2024-04-26T00:30:34.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.381+0000 7f04967fc640 1 --2- 172.21.15.37:0/2091677712 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0478035e80 0x7f0478038340 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f0480006fd0 tx=0x7f0480006d40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:34.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.385+0000 7f048f7fe640 1 -- 172.21.15.37:0/2091677712 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f049001a360 con 0x7f04980a0260 2024-04-26T00:30:34.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.385+0000 7f048f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:30:34.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.385+0000 7f048f7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:30:34.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.605+0000 7f049d2b3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:30:34.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.605+0000 7f049d2b3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:30:34.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.605+0000 7f049d2b3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:30:34.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.605+0000 7f049d2b3640 1 -- 172.21.15.37:0/2091677712 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f049806b9a0 con 0x7f04980a0260 2024-04-26T00:30:34.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.605+0000 7f048f7fe640 1 -- 172.21.15.37:0/2091677712 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f0490046f20 con 0x7f04980a0260 2024-04-26T00:30:34.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.605+0000 7f048f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:30:34.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.605+0000 7f048f7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:30:34.599 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:30:34.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.609+0000 7f049d2b3640 1 -- 172.21.15.37:0/2091677712 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0478035e80 msgr2=0x7f0478038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:34.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.609+0000 7f049d2b3640 1 --2- 172.21.15.37:0/2091677712 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0478035e80 0x7f0478038340 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f0480006fd0 tx=0x7f0480006d40 comp rx=0 tx=0).stop 2024-04-26T00:30:34.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.609+0000 7f049d2b3640 10 monclient: shutdown 2024-04-26T00:30:34.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.609+0000 7f049d2b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:30:34.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.609+0000 7f049d2b3640 1 -- 172.21.15.37:0/2091677712 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f04980a0260 msgr2=0x7f04981ccc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:34.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.609+0000 7f049d2b3640 1 --2- 172.21.15.37:0/2091677712 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f04980a0260 0x7f04981ccc40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f049000c7b0 tx=0x7f049000cc70 comp rx=0 tx=0).stop 2024-04-26T00:30:34.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.609+0000 7f049d2b3640 1 -- 172.21.15.37:0/2091677712 shutdown_connections 2024-04-26T00:30:34.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.609+0000 7f049d2b3640 1 --2- 172.21.15.37:0/2091677712 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0478035e80 0x7f0478038340 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:34.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.609+0000 7f049d2b3640 1 --2- 172.21.15.37:0/2091677712 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f049813b950 0x7f049811c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:34.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.609+0000 7f049d2b3640 1 --2- 172.21.15.37:0/2091677712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0498137e40 0x7f0498136940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:34.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.609+0000 7f049d2b3640 1 --2- 172.21.15.37:0/2091677712 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f04980a0260 0x7f04981ccc40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:34.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.609+0000 7f049d2b3640 1 -- 172.21.15.37:0/2091677712 >> 172.21.15.37:0/2091677712 conn(0x7f04980a1ad0 msgr2=0x7f04980a3d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:30:34.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.609+0000 7f049d2b3640 1 -- 172.21.15.37:0/2091677712 shutdown_connections 2024-04-26T00:30:34.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:34.609+0000 7f049d2b3640 1 -- 172.21.15.37:0/2091677712 wait complete. 2024-04-26T00:30:37.619 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:30:37.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.757+0000 7f81dfb8a640 10 monclient: get_monmap_and_config 2024-04-26T00:30:37.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.757+0000 7f81dfb8a640 10 monclient: build_initial_monmap 2024-04-26T00:30:37.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.757+0000 7f81dfb8a640 10 monclient: monmap: 2024-04-26T00:30:37.749 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:30:37.749 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:37.749 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:30:37.761458+0000 2024-04-26T00:30:37.749 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:30:37.761458+0000 2024-04-26T00:30:37.750 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:30:37.750 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:37.750 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:30:37.750 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:30:37.750 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:30:37.750 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:30:37.750 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:30:37.750 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:37.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.757+0000 7f81dfb8a640 1 Processor -- start 2024-04-26T00:30:37.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.757+0000 7f81dfb8a640 1 -- start start 2024-04-26T00:30:37.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.757+0000 7f81dfb8a640 10 monclient: init 2024-04-26T00:30:37.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.757+0000 7f81dfb8a640 10 monclient: _reopen_session rank -1 2024-04-26T00:30:37.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.757+0000 7f81dfb8a640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-04-26T00:30:37.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.757+0000 7f81dfb8a640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f81d8136e10 0x7f81d809c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:37.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.757+0000 7f81dfb8a640 10 monclient(hunting): picked mon.noname-a con 0x7f81d8136e10 addr v2:172.21.15.37:3300/0 2024-04-26T00:30:37.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.757+0000 7f81dfb8a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81d81371f0 0x7f81d81375d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:37.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.757+0000 7f81dfb8a640 10 monclient(hunting): picked mon.noname-b con 0x7f81d81371f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:30:37.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.757+0000 7f81dfb8a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f81d8135d90 addr v1:172.21.15.37:6789/0 2024-04-26T00:30:37.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.757+0000 7f81dfb8a640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81d809b680 con 0x7f81d8135d90 2024-04-26T00:30:37.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.757+0000 7f81dfb8a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:37.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.757+0000 7f81dfb8a640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f81d809b8f0 con 0x7f81d8136e10 2024-04-26T00:30:37.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.757+0000 7f81dfb8a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:37.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.757+0000 7f81dfb8a640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f81d809bbd0 con 0x7f81d81371f0 2024-04-26T00:30:37.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.757+0000 7f81dfb8a640 10 monclient(hunting): _renew_subs 2024-04-26T00:30:37.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.757+0000 7f81dfb8a640 10 monclient(hunting): authenticate will time out at 1619.067469s 2024-04-26T00:30:37.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.757+0000 7f81de127640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f81d8135d90 0x7f81d8136240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:34798/0 (socket says 172.21.15.37:34798) 2024-04-26T00:30:37.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.757+0000 7f81dd926640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f81d8136e10 0x7f81d809c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:37.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.757+0000 7f81de127640 1 -- 172.21.15.37:0/1528111753 learned_addr learned my addr 172.21.15.37:0/1528111753 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:30:37.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.757+0000 7f81dd125640 1 --2- 172.21.15.37:0/1528111753 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81d81371f0 0x7f81d81375d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:37.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.757+0000 7f81dd125640 10 monclient(hunting): get_auth_request con 0x7f81d81371f0 auth_method 0 2024-04-26T00:30:37.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.757+0000 7f81dd125640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:37.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.757+0000 7f81dd125640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:37.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.757+0000 7f81dd125640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:37.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.757+0000 7f81dd926640 10 monclient(hunting): get_auth_request con 0x7f81d8136e10 auth_method 0 2024-04-26T00:30:37.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.757+0000 7f81dd926640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:37.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.757+0000 7f81dd926640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:37.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.757+0000 7f81dd926640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:37.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.757+0000 7f81dd125640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:37.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.757+0000 7f81dd125640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:37.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.757+0000 7f81dd125640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:37.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.757+0000 7f81dd926640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:37.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.757+0000 7f81dd926640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:37.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.757+0000 7f81dd926640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:37.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.761+0000 7f81dd926640 10 monclient(hunting): handle_auth_done global_id 124155 payload 274 2024-04-26T00:30:37.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.761+0000 7f81dd926640 1 -- 172.21.15.37:0/1528111753 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81d81371f0 msgr2=0x7f81d81375d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:30:37.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.761+0000 7f81dd926640 1 --2- 172.21.15.37:0/1528111753 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81d81371f0 0x7f81d81375d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:37.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.761+0000 7f81dd926640 1 -- 172.21.15.37:0/1528111753 >> v1:172.21.15.37:6789/0 conn(0x7f81d8135d90 legacy=0x7f81d8136240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:30:37.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.761+0000 7f81dd926640 10 monclient: _finish_hunting 0 2024-04-26T00:30:37.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.761+0000 7f81dd926640 1 monclient: found mon.noname-a 2024-04-26T00:30:37.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.761+0000 7f81dd926640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:30:37.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.761+0000 7f81dd926640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:30:37.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.761+0000 7f81dd926640 1 -- 172.21.15.37:0/1528111753 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81d8137b10 con 0x7f81d8136e10 2024-04-26T00:30:37.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.761+0000 7f81dd125640 1 --2- 172.21.15.37:0/1528111753 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81d81371f0 0x7f81d81375d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:30:37.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dd926640 1 -- 172.21.15.37:0/1528111753 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81d8136e10 msgr2=0x7f81d809c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:37.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dd926640 1 --2- 172.21.15.37:0/1528111753 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81d8136e10 0x7f81d809c5b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f81cc009d90 tx=0x7f81cc01c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84e1153f96f1f6b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:37.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dc924640 1 -- 172.21.15.37:0/1528111753 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81cc0b2070 con 0x7f81d8136e10 2024-04-26T00:30:37.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dc924640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:37.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dc924640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:30:37.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dc924640 10 monclient: dump: 2024-04-26T00:30:37.757 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:37.758 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:37.758 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:37.758 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:37.758 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:37.758 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:37.758 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:37.758 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:37.758 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:37.758 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:37.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dc924640 10 monclient: _finish_auth 0 2024-04-26T00:30:37.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dc924640 10 monclient: _check_auth_tickets 2024-04-26T00:30:37.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dc924640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:30:37.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dc924640 1 -- 172.21.15.37:0/1528111753 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81cc0a5890 con 0x7f81d8136e10 2024-04-26T00:30:37.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dfb8a640 5 monclient: authenticate success, global_id 124155 2024-04-26T00:30:37.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dfb8a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:30:37.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dc924640 10 monclient: handle_config config(0 keys) 2024-04-26T00:30:37.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dc924640 1 -- 172.21.15.37:0/1528111753 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81cc0a5c60 con 0x7f81d8136e10 2024-04-26T00:30:37.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dc924640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:37.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dc924640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:30:37.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dc924640 10 monclient: dump: 2024-04-26T00:30:37.759 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:37.759 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:37.759 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:37.759 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:37.759 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:37.759 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:37.759 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:37.759 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:37.759 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:37.760 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:37.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dfb8a640 10 monclient: get_monmap_and_config success 2024-04-26T00:30:37.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dfb8a640 4 set_mon_vals no callback set 2024-04-26T00:30:37.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dfb8a640 10 monclient: shutdown 2024-04-26T00:30:37.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dfb8a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:30:37.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dfb8a640 1 -- 172.21.15.37:0/1528111753 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81d8136e10 msgr2=0x7f81d809c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:37.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dfb8a640 1 --2- 172.21.15.37:0/1528111753 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81d8136e10 0x7f81d809c5b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f81cc009d90 tx=0x7f81cc01c7f0 comp rx=0 tx=0).stop 2024-04-26T00:30:37.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dfb8a640 1 -- 172.21.15.37:0/1528111753 shutdown_connections 2024-04-26T00:30:37.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dfb8a640 1 --2- 172.21.15.37:0/1528111753 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81d81371f0 0x7f81d81375d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:37.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dfb8a640 1 --2- 172.21.15.37:0/1528111753 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81d8136e10 0x7f81d809c5b0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:37.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dfb8a640 1 -- 172.21.15.37:0/1528111753 >> 172.21.15.37:0/1528111753 conn(0x7f81d80a64e0 msgr2=0x7f81d80a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:30:37.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dfb8a640 1 -- 172.21.15.37:0/1528111753 shutdown_connections 2024-04-26T00:30:37.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dfb8a640 1 -- 172.21.15.37:0/1528111753 wait complete. 2024-04-26T00:30:37.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dfb8a640 10 monclient: build_initial_monmap 2024-04-26T00:30:37.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dfb8a640 10 monclient: monmap: 2024-04-26T00:30:37.765 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:30:37.765 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:30:37.765 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:30:37.765 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:30:37.765 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:30:37.765 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:30:37.765 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:30:37.765 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:30:37.765 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:30:37.765 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:37.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dfb8a640 1 Processor -- start 2024-04-26T00:30:37.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dfb8a640 1 -- start start 2024-04-26T00:30:37.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dfb8a640 10 monclient: init 2024-04-26T00:30:37.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dfb8a640 10 monclient: _reopen_session rank -1 2024-04-26T00:30:37.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dfb8a640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:30:37.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dfb8a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81d8135d90 0x7f81d81cf5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:37.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dfb8a640 10 monclient(hunting): picked mon.noname-c con 0x7f81d8135d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:30:37.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dfb8a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81d8136e10 0x7f81d81cfd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:37.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dfb8a640 10 monclient(hunting): picked mon.noname-a con 0x7f81d8136e10 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:30:37.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dfb8a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81d81371f0 0x7f81d813ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:37.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dfb8a640 10 monclient(hunting): picked mon.noname-b con 0x7f81d81371f0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:30:37.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dfb8a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:37.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dfb8a640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f81d8141f10 con 0x7f81d8136e10 2024-04-26T00:30:37.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dfb8a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:37.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dfb8a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f81d8141d90 con 0x7f81d81371f0 2024-04-26T00:30:37.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dfb8a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:37.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dfb8a640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f81d8142090 con 0x7f81d8135d90 2024-04-26T00:30:37.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dfb8a640 10 monclient(hunting): _renew_subs 2024-04-26T00:30:37.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dfb8a640 10 monclient(hunting): authenticate will time out at 1619.075155s 2024-04-26T00:30:37.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dd926640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81d8135d90 0x7f81d81cf5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:37.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dd926640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81d8135d90 0x7f81d81cf5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:56576/0 (socket says 172.21.15.37:56576) 2024-04-26T00:30:37.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dd125640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81d8136e10 0x7f81d81cfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:37.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dd926640 1 -- 172.21.15.37:0/1843918340 learned_addr learned my addr 172.21.15.37:0/1843918340 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:30:37.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dd926640 10 monclient(hunting): get_auth_request con 0x7f81d8135d90 auth_method 0 2024-04-26T00:30:37.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dd926640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:37.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dd926640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:37.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dd926640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:37.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dd125640 10 monclient(hunting): get_auth_request con 0x7f81d8136e10 auth_method 0 2024-04-26T00:30:37.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dd125640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:37.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dd125640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:37.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dd125640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:37.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81de127640 1 -- 172.21.15.37:0/1843918340 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81d81371f0 msgr2=0x7f81d813ac00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:30:37.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81de127640 1 --2- 172.21.15.37:0/1843918340 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81d81371f0 0x7f81d813ac00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:30:37.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dd926640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:37.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dd926640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:37.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dd926640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:37.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dd125640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:37.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dd125640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:37.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dd125640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:37.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dd926640 10 monclient(hunting): handle_auth_done global_id 84521 payload 824 2024-04-26T00:30:37.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dd926640 1 -- 172.21.15.37:0/1843918340 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81d81371f0 msgr2=0x7f81d813ac00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:30:37.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dd926640 1 --2- 172.21.15.37:0/1843918340 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81d81371f0 0x7f81d813ac00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:37.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dd926640 1 -- 172.21.15.37:0/1843918340 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81d8136e10 msgr2=0x7f81d81cfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:37.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dd926640 1 --2- 172.21.15.37:0/1843918340 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81d8136e10 0x7f81d81cfd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:37.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dd926640 10 monclient: _finish_hunting 0 2024-04-26T00:30:37.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dd926640 1 monclient: found mon.noname-c 2024-04-26T00:30:37.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dd926640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:30:37.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dd926640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:30:37.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.765+0000 7f81dd926640 1 -- 172.21.15.37:0/1843918340 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81d813dfd0 con 0x7f81d8135d90 2024-04-26T00:30:37.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.769+0000 7f81dd926640 1 -- 172.21.15.37:0/1843918340 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81d8135d90 msgr2=0x7f81d81cf5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:37.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.769+0000 7f81dd926640 1 --2- 172.21.15.37:0/1843918340 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81d8135d90 0x7f81d81cf5f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f81cc0b0040 tx=0x7f81cc019590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:37.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.769+0000 7f81d67fc640 1 -- 172.21.15.37:0/1843918340 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81cc0bd020 con 0x7f81d8135d90 2024-04-26T00:30:37.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.769+0000 7f81d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:37.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.769+0000 7f81d67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:30:37.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.769+0000 7f81d67fc640 10 monclient: dump: 2024-04-26T00:30:37.769 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:37.769 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:37.769 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:37.769 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:37.769 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:37.769 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:37.769 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:37.769 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:37.770 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:37.770 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:37.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.769+0000 7f81d67fc640 10 monclient: _finish_auth 0 2024-04-26T00:30:37.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.769+0000 7f81d67fc640 10 monclient: _check_auth_tickets 2024-04-26T00:30:37.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.769+0000 7f81d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:30:37.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.769+0000 7f81dfb8a640 5 monclient: authenticate success, global_id 84521 2024-04-26T00:30:37.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.769+0000 7f81d67fc640 1 -- 172.21.15.37:0/1843918340 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81cc0b8880 con 0x7f81d8135d90 2024-04-26T00:30:37.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.769+0000 7f81d67fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:30:37.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.769+0000 7f81d67fc640 1 -- 172.21.15.37:0/1843918340 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81cc0b2040 con 0x7f81d8135d90 2024-04-26T00:30:37.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.769+0000 7f81d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:37.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.769+0000 7f81d67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:30:37.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.769+0000 7f81d67fc640 10 monclient: dump: 2024-04-26T00:30:37.770 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:37.770 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:37.770 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:37.771 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:37.771 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:37.771 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:37.771 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:37.771 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:37.771 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:37.771 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:37.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.769+0000 7f81d7fff640 4 set_mon_vals no callback set 2024-04-26T00:30:37.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.769+0000 7f81dfb8a640 10 monclient: _renew_subs 2024-04-26T00:30:37.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.769+0000 7f81dfb8a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:30:37.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.769+0000 7f81dfb8a640 1 -- 172.21.15.37:0/1843918340 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81d813e830 con 0x7f81d8135d90 2024-04-26T00:30:37.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.769+0000 7f81dfb8a640 10 monclient: _renew_subs 2024-04-26T00:30:37.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.769+0000 7f81dfb8a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:30:37.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.769+0000 7f81dfb8a640 1 -- 172.21.15.37:0/1843918340 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f81d809aca0 con 0x7f81d8135d90 2024-04-26T00:30:37.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.769+0000 7f81dfb8a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:30:37.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.769+0000 7f81dfb8a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:30:37.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.769+0000 7f81dfb8a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:30:37.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.769+0000 7f81dfb8a640 1 -- 172.21.15.37:0/1843918340 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81d8075910 con 0x7f81d8135d90 2024-04-26T00:30:37.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.769+0000 7f81d67fc640 1 -- 172.21.15.37:0/1843918340 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f81cc0c0020 con 0x7f81d8135d90 2024-04-26T00:30:37.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.769+0000 7f81d67fc640 1 --2- 172.21.15.37:0/1843918340 >> v2:172.21.15.45:6812/2379918623 conn(0x7f81bc035e80 0x7f81bc038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:37.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.769+0000 7f81d67fc640 1 -- 172.21.15.37:0/1843918340 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(404..404 src has 370..404) ==== 4007+0+0 (secure 0 0 0) 0x7f81cc0e39e0 con 0x7f81d8135d90 2024-04-26T00:30:37.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.773+0000 7f81dd125640 1 --2- 172.21.15.37:0/1843918340 >> v2:172.21.15.45:6812/2379918623 conn(0x7f81bc035e80 0x7f81bc038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:37.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.773+0000 7f81dd125640 10 monclient: get_auth_request con 0x7f81bc035e80 auth_method 0 2024-04-26T00:30:37.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.773+0000 7f81dd125640 1 --2- 172.21.15.37:0/1843918340 >> v2:172.21.15.45:6812/2379918623 conn(0x7f81bc035e80 0x7f81bc038340 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f81c4009580 tx=0x7f81c4009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:37.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.777+0000 7f81d67fc640 1 -- 172.21.15.37:0/1843918340 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f81cc0c7030 con 0x7f81d8135d90 2024-04-26T00:30:37.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.777+0000 7f81d67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:30:37.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.777+0000 7f81d67fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:30:37.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.993+0000 7f81dfb8a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:30:37.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.993+0000 7f81dfb8a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:30:37.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.993+0000 7f81dfb8a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:30:37.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.993+0000 7f81dfb8a640 1 -- 172.21.15.37:0/1843918340 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f81d8066f40 con 0x7f81d8135d90 2024-04-26T00:30:37.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.993+0000 7f81d67fc640 1 -- 172.21.15.37:0/1843918340 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f81cc016500 con 0x7f81d8135d90 2024-04-26T00:30:37.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.993+0000 7f81d67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:30:37.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.993+0000 7f81d67fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:30:37.986 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:30:37.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.993+0000 7f81dfb8a640 1 -- 172.21.15.37:0/1843918340 >> v2:172.21.15.45:6812/2379918623 conn(0x7f81bc035e80 msgr2=0x7f81bc038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:37.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.993+0000 7f81dfb8a640 1 --2- 172.21.15.37:0/1843918340 >> v2:172.21.15.45:6812/2379918623 conn(0x7f81bc035e80 0x7f81bc038340 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f81c4009580 tx=0x7f81c4009190 comp rx=0 tx=0).stop 2024-04-26T00:30:37.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.993+0000 7f81dfb8a640 10 monclient: shutdown 2024-04-26T00:30:37.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.993+0000 7f81dfb8a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:30:37.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.993+0000 7f81dfb8a640 1 -- 172.21.15.37:0/1843918340 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81d8135d90 msgr2=0x7f81d81cf5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:37.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.993+0000 7f81dfb8a640 1 --2- 172.21.15.37:0/1843918340 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81d8135d90 0x7f81d81cf5f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f81cc0b0040 tx=0x7f81cc019590 comp rx=0 tx=0).stop 2024-04-26T00:30:37.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.997+0000 7f81dfb8a640 1 -- 172.21.15.37:0/1843918340 shutdown_connections 2024-04-26T00:30:37.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.997+0000 7f81dfb8a640 1 --2- 172.21.15.37:0/1843918340 >> v2:172.21.15.45:6812/2379918623 conn(0x7f81bc035e80 0x7f81bc038340 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:37.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.997+0000 7f81dfb8a640 1 --2- 172.21.15.37:0/1843918340 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f81d81371f0 0x7f81d813ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:37.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.997+0000 7f81dfb8a640 1 --2- 172.21.15.37:0/1843918340 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81d8136e10 0x7f81d81cfd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:37.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.997+0000 7f81dfb8a640 1 --2- 172.21.15.37:0/1843918340 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81d8135d90 0x7f81d81cf5f0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:37.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.997+0000 7f81dfb8a640 1 -- 172.21.15.37:0/1843918340 >> 172.21.15.37:0/1843918340 conn(0x7f81d80a64e0 msgr2=0x7f81d80a4bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:30:37.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.997+0000 7f81dfb8a640 1 -- 172.21.15.37:0/1843918340 shutdown_connections 2024-04-26T00:30:37.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:37.997+0000 7f81dfb8a640 1 -- 172.21.15.37:0/1843918340 wait complete. 2024-04-26T00:30:41.003 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:30:41.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.141+0000 7f68d4a65640 10 monclient: get_monmap_and_config 2024-04-26T00:30:41.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.141+0000 7f68d4a65640 10 monclient: build_initial_monmap 2024-04-26T00:30:41.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.141+0000 7f68d4a65640 10 monclient: monmap: 2024-04-26T00:30:41.134 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:30:41.134 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:41.134 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:30:41.145779+0000 2024-04-26T00:30:41.134 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:30:41.145779+0000 2024-04-26T00:30:41.134 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:30:41.134 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:41.134 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:30:41.134 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:30:41.134 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:30:41.134 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:30:41.134 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:30:41.135 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:41.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.141+0000 7f68d4a65640 1 Processor -- start 2024-04-26T00:30:41.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.141+0000 7f68d4a65640 1 -- start start 2024-04-26T00:30:41.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.141+0000 7f68d4a65640 10 monclient: init 2024-04-26T00:30:41.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.141+0000 7f68d4a65640 10 monclient: _reopen_session rank -1 2024-04-26T00:30:41.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.141+0000 7f68d4a65640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-04-26T00:30:41.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.141+0000 7f68d4a65640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f68d0133b20 0x7f68d009e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:41.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.141+0000 7f68d4a65640 10 monclient(hunting): picked mon.noname-a con 0x7f68d0133b20 addr v2:172.21.15.37:3300/0 2024-04-26T00:30:41.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.141+0000 7f68d4a65640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f68d0133f00 addr v1:172.21.15.37:6789/0 2024-04-26T00:30:41.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.141+0000 7f68d4a65640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68d0139050 0x7f68d013b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:41.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.141+0000 7f68d4a65640 10 monclient(hunting): picked mon.noname-b con 0x7f68d0139050 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:30:41.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.141+0000 7f68d4a65640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68d009fe20 con 0x7f68d0133f00 2024-04-26T00:30:41.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.141+0000 7f68d4a65640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:41.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.141+0000 7f68d4a65640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f68d0072d00 con 0x7f68d0133b20 2024-04-26T00:30:41.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.141+0000 7f68d4a65640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:41.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.141+0000 7f68d4a65640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f68d009ba60 con 0x7f68d0139050 2024-04-26T00:30:41.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.141+0000 7f68d4a65640 10 monclient(hunting): _renew_subs 2024-04-26T00:30:41.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.141+0000 7f68d4a65640 10 monclient(hunting): authenticate will time out at 1622.451472s 2024-04-26T00:30:41.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.141+0000 7f68ce59c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f68d0133b20 0x7f68d009e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:41.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.141+0000 7f68cdd9b640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f68d0133f00 0x7f68d0134350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:59566/0 (socket says 172.21.15.37:59566) 2024-04-26T00:30:41.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.141+0000 7f68ce59c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f68d0133b20 0x7f68d009e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:41044/0 (socket says 172.21.15.37:41044) 2024-04-26T00:30:41.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.141+0000 7f68cdd9b640 1 -- 172.21.15.37:0/3495225693 learned_addr learned my addr 172.21.15.37:0/3495225693 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:30:41.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.141+0000 7f68ce59c640 10 monclient(hunting): get_auth_request con 0x7f68d0133b20 auth_method 0 2024-04-26T00:30:41.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.141+0000 7f68ce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:41.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.141+0000 7f68ce59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:41.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.141+0000 7f68ce59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:41.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.141+0000 7f68ced9d640 1 --2- 172.21.15.37:0/3495225693 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68d0139050 0x7f68d013b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:41.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.141+0000 7f68ced9d640 10 monclient(hunting): get_auth_request con 0x7f68d0139050 auth_method 0 2024-04-26T00:30:41.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.141+0000 7f68ced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:41.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.141+0000 7f68ced9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:41.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.141+0000 7f68ced9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:41.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.141+0000 7f68ce59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:41.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.141+0000 7f68ce59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:41.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.141+0000 7f68ce59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:41.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.141+0000 7f68ced9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:41.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.141+0000 7f68ced9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:41.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.141+0000 7f68ced9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:41.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.145+0000 7f68ced9d640 10 monclient(hunting): handle_auth_done global_id 84524 payload 274 2024-04-26T00:30:41.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.145+0000 7f68ced9d640 1 -- 172.21.15.37:0/3495225693 >> v2:172.21.15.37:3300/0 conn(0x7f68d0133b20 msgr2=0x7f68d009e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:30:41.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.145+0000 7f68ced9d640 1 --2- 172.21.15.37:0/3495225693 >> v2:172.21.15.37:3300/0 conn(0x7f68d0133b20 0x7f68d009e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:41.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.145+0000 7f68ced9d640 1 -- 172.21.15.37:0/3495225693 >> v1:172.21.15.37:6789/0 conn(0x7f68d0133f00 legacy=0x7f68d0134350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:30:41.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.145+0000 7f68ced9d640 10 monclient: _finish_hunting 0 2024-04-26T00:30:41.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.145+0000 7f68ced9d640 1 monclient: found mon.noname-b 2024-04-26T00:30:41.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.145+0000 7f68ced9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:30:41.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.145+0000 7f68ced9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:30:41.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.145+0000 7f68ced9d640 1 -- 172.21.15.37:0/3495225693 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68d009dd10 con 0x7f68d0139050 2024-04-26T00:30:41.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.145+0000 7f68ce59c640 1 --2- 172.21.15.37:0/3495225693 >> v2:172.21.15.37:3300/0 conn(0x7f68d0133b20 0x7f68d009e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:30:41.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68ced9d640 1 -- 172.21.15.37:0/3495225693 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68d0139050 msgr2=0x7f68d013b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:41.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68ced9d640 1 --2- 172.21.15.37:0/3495225693 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68d0139050 0x7f68d013b440 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f68c8009d90 tx=0x7f68c801c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=692a16f1efea2c12 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:41.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68cd59a640 1 -- 172.21.15.37:0/3495225693 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c80ad070 con 0x7f68d0139050 2024-04-26T00:30:41.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68cd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:41.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68cd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:30:41.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68cd59a640 10 monclient: dump: 2024-04-26T00:30:41.145 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:41.145 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:41.145 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:41.145 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:41.145 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:41.145 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:41.145 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:41.145 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:41.146 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:41.146 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:41.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68cd59a640 10 monclient: _finish_auth 0 2024-04-26T00:30:41.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68cd59a640 10 monclient: _check_auth_tickets 2024-04-26T00:30:41.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68cd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:30:41.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68cd59a640 1 -- 172.21.15.37:0/3495225693 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68c80a9630 con 0x7f68d0139050 2024-04-26T00:30:41.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68cd59a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:30:41.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68cd59a640 1 -- 172.21.15.37:0/3495225693 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c80a9a00 con 0x7f68d0139050 2024-04-26T00:30:41.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68cd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:41.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68cd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:30:41.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68cd59a640 10 monclient: dump: 2024-04-26T00:30:41.146 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:41.146 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:41.147 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:41.147 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:41.147 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:41.147 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:41.147 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:41.147 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:41.147 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:41.147 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:41.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68d4a65640 5 monclient: authenticate success, global_id 84524 2024-04-26T00:30:41.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68d4a65640 10 monclient: get_monmap_and_config success 2024-04-26T00:30:41.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68d4a65640 4 set_mon_vals no callback set 2024-04-26T00:30:41.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68d4a65640 10 monclient: shutdown 2024-04-26T00:30:41.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68d4a65640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:30:41.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68d4a65640 1 -- 172.21.15.37:0/3495225693 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68d0139050 msgr2=0x7f68d013b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:41.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68d4a65640 1 --2- 172.21.15.37:0/3495225693 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68d0139050 0x7f68d013b440 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f68c8009d90 tx=0x7f68c801c6a0 comp rx=0 tx=0).stop 2024-04-26T00:30:41.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68d4a65640 1 -- 172.21.15.37:0/3495225693 shutdown_connections 2024-04-26T00:30:41.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68d4a65640 1 --2- 172.21.15.37:0/3495225693 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68d0139050 0x7f68d013b440 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:41.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68d4a65640 1 --2- 172.21.15.37:0/3495225693 >> v2:172.21.15.37:3300/0 conn(0x7f68d0133b20 0x7f68d009e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:41.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68d4a65640 1 -- 172.21.15.37:0/3495225693 >> 172.21.15.37:0/3495225693 conn(0x7f68d00a1ad0 msgr2=0x7f68d00a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:30:41.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68d4a65640 1 -- 172.21.15.37:0/3495225693 shutdown_connections 2024-04-26T00:30:41.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68d4a65640 1 -- 172.21.15.37:0/3495225693 wait complete. 2024-04-26T00:30:41.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68d4a65640 10 monclient: build_initial_monmap 2024-04-26T00:30:41.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68d4a65640 10 monclient: monmap: 2024-04-26T00:30:41.148 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:30:41.148 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:30:41.148 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:30:41.148 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:30:41.148 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:30:41.148 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:30:41.149 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:30:41.149 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:30:41.149 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:30:41.149 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:41.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68d4a65640 1 Processor -- start 2024-04-26T00:30:41.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68d4a65640 1 -- start start 2024-04-26T00:30:41.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68d4a65640 10 monclient: init 2024-04-26T00:30:41.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68d4a65640 10 monclient: _reopen_session rank -1 2024-04-26T00:30:41.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68d4a65640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:30:41.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68d4a65640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68d0133b20 0x7f68d0072700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:41.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68d4a65640 10 monclient(hunting): picked mon.noname-c con 0x7f68d0133b20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:30:41.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68d4a65640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68d0133f00 0x7f68d0132c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:41.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68d4a65640 10 monclient(hunting): picked mon.noname-b con 0x7f68d0133f00 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:30:41.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68d4a65640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f68d0139050 0x7f68d01ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:41.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68d4a65640 10 monclient(hunting): picked mon.noname-a con 0x7f68d0139050 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:30:41.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68d4a65640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:41.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68d4a65640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f68d013e320 con 0x7f68d0139050 2024-04-26T00:30:41.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68d4a65640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:41.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68d4a65640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f68d013e1a0 con 0x7f68d0133f00 2024-04-26T00:30:41.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68d4a65640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:41.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68d4a65640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f68d013e4a0 con 0x7f68d0133b20 2024-04-26T00:30:41.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68d4a65640 10 monclient(hunting): _renew_subs 2024-04-26T00:30:41.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68d4a65640 10 monclient(hunting): authenticate will time out at 1622.458706s 2024-04-26T00:30:41.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68ced9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f68d0139050 0x7f68d01ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:41.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68ced9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f68d0139050 0x7f68d01ccae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:41060/0 (socket says 172.21.15.37:41060) 2024-04-26T00:30:41.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68ced9d640 1 -- 172.21.15.37:0/2942034055 learned_addr learned my addr 172.21.15.37:0/2942034055 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:30:41.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68ced9d640 10 monclient(hunting): get_auth_request con 0x7f68d0139050 auth_method 0 2024-04-26T00:30:41.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68ced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:41.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68ced9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:41.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68ced9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:41.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68cdd9b640 1 -- 172.21.15.37:0/2942034055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68d0133f00 msgr2=0x7f68d0132c80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:30:41.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68cdd9b640 1 --2- 172.21.15.37:0/2942034055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68d0133f00 0x7f68d0132c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:30:41.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68ce59c640 1 --2- 172.21.15.37:0/2942034055 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68d0133b20 0x7f68d0072700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:41.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68ced9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:41.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68ced9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:41.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68ced9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:41.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68ce59c640 10 monclient(hunting): get_auth_request con 0x7f68d0133b20 auth_method 0 2024-04-26T00:30:41.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68ce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:41.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68ce59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:41.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68ce59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:41.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68ce59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:41.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68ce59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:41.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68ce59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:41.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68ced9d640 10 monclient(hunting): handle_auth_done global_id 124164 payload 824 2024-04-26T00:30:41.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68ced9d640 1 -- 172.21.15.37:0/2942034055 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68d0133b20 msgr2=0x7f68d0072700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:41.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68ced9d640 1 --2- 172.21.15.37:0/2942034055 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68d0133b20 0x7f68d0072700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:41.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68ced9d640 1 -- 172.21.15.37:0/2942034055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68d0133f00 msgr2=0x7f68d0132c80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:30:41.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68ced9d640 1 --2- 172.21.15.37:0/2942034055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68d0133f00 0x7f68d0132c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:41.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68ced9d640 10 monclient: _finish_hunting 0 2024-04-26T00:30:41.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68ced9d640 1 monclient: found mon.noname-a 2024-04-26T00:30:41.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68ced9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:30:41.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68ced9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:30:41.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.149+0000 7f68ced9d640 1 -- 172.21.15.37:0/2942034055 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68d011c320 con 0x7f68d0139050 2024-04-26T00:30:41.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.153+0000 7f68ced9d640 1 -- 172.21.15.37:0/2942034055 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f68d0139050 msgr2=0x7f68d01ccae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:41.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.153+0000 7f68ced9d640 1 --2- 172.21.15.37:0/2942034055 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f68d0139050 0x7f68d01ccae0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f68c801ad50 tx=0x7f68c8019440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:41.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.153+0000 7f68c6ffd640 1 -- 172.21.15.37:0/2942034055 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c80c0070 con 0x7f68d0139050 2024-04-26T00:30:41.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.153+0000 7f68c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:41.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.153+0000 7f68c6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:30:41.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.153+0000 7f68c6ffd640 10 monclient: dump: 2024-04-26T00:30:41.154 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:41.154 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:41.154 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:41.154 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:41.154 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:41.154 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:41.155 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:41.155 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:41.155 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:41.155 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:41.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.153+0000 7f68c6ffd640 10 monclient: _finish_auth 0 2024-04-26T00:30:41.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.153+0000 7f68c6ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:30:41.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.153+0000 7f68c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:30:41.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.153+0000 7f68c6ffd640 1 -- 172.21.15.37:0/2942034055 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68c80a9430 con 0x7f68d0139050 2024-04-26T00:30:41.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.153+0000 7f68c6ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:30:41.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.153+0000 7f68c6ffd640 1 -- 172.21.15.37:0/2942034055 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c80ad040 con 0x7f68d0139050 2024-04-26T00:30:41.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.153+0000 7f68c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:41.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.153+0000 7f68c6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:30:41.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.153+0000 7f68c6ffd640 10 monclient: dump: 2024-04-26T00:30:41.156 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:41.156 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:41.156 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:41.156 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:41.156 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:41.156 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:41.156 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:41.156 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:41.156 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:41.156 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:41.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.153+0000 7f68d4a65640 5 monclient: authenticate success, global_id 124164 2024-04-26T00:30:41.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.153+0000 7f68d4a65640 10 monclient: _renew_subs 2024-04-26T00:30:41.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.153+0000 7f68d4a65640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:30:41.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.153+0000 7f68ccd99640 4 set_mon_vals no callback set 2024-04-26T00:30:41.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.153+0000 7f68d4a65640 1 -- 172.21.15.37:0/2942034055 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68d011c600 con 0x7f68d0139050 2024-04-26T00:30:41.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.153+0000 7f68d4a65640 10 monclient: _renew_subs 2024-04-26T00:30:41.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.153+0000 7f68d4a65640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:30:41.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.153+0000 7f68d4a65640 1 -- 172.21.15.37:0/2942034055 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68d013b6e0 con 0x7f68d0139050 2024-04-26T00:30:41.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.153+0000 7f68d4a65640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:30:41.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.153+0000 7f68d4a65640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:30:41.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.153+0000 7f68d4a65640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:30:41.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.153+0000 7f68d4a65640 1 -- 172.21.15.37:0/2942034055 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68d0066f40 con 0x7f68d0139050 2024-04-26T00:30:41.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.153+0000 7f68c6ffd640 1 -- 172.21.15.37:0/2942034055 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f68c80b0070 con 0x7f68d0139050 2024-04-26T00:30:41.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.153+0000 7f68c6ffd640 1 --2- 172.21.15.37:0/2942034055 >> v2:172.21.15.45:6812/2379918623 conn(0x7f68b0035e30 0x7f68b00382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:41.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.153+0000 7f68c6ffd640 1 -- 172.21.15.37:0/2942034055 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(404..404 src has 370..404) ==== 4007+0+0 (secure 0 0 0) 0x7f68c80e2da0 con 0x7f68d0139050 2024-04-26T00:30:41.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.153+0000 7f68ce59c640 1 --2- 172.21.15.37:0/2942034055 >> v2:172.21.15.45:6812/2379918623 conn(0x7f68b0035e30 0x7f68b00382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:41.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.153+0000 7f68ce59c640 10 monclient: get_auth_request con 0x7f68b0035e30 auth_method 0 2024-04-26T00:30:41.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.157+0000 7f68ce59c640 1 --2- 172.21.15.37:0/2942034055 >> v2:172.21.15.45:6812/2379918623 conn(0x7f68b0035e30 0x7f68b00382f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f68c0009580 tx=0x7f68c0009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:41.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.157+0000 7f68c6ffd640 1 -- 172.21.15.37:0/2942034055 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f68d009fe20 con 0x7f68d0139050 2024-04-26T00:30:41.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.157+0000 7f68c6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:30:41.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.157+0000 7f68c6ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:30:41.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.381+0000 7f68d4a65640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:30:41.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.381+0000 7f68d4a65640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:30:41.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.381+0000 7f68d4a65640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:30:41.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.381+0000 7f68d4a65640 1 -- 172.21.15.37:0/2942034055 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f68d006b9a0 con 0x7f68d0139050 2024-04-26T00:30:41.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.381+0000 7f68c6ffd640 1 -- 172.21.15.37:0/2942034055 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f68c80b2af0 con 0x7f68d0139050 2024-04-26T00:30:41.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.381+0000 7f68c6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:30:41.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.381+0000 7f68c6ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:30:41.374 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:30:41.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.385+0000 7f68d4a65640 1 -- 172.21.15.37:0/2942034055 >> v2:172.21.15.45:6812/2379918623 conn(0x7f68b0035e30 msgr2=0x7f68b00382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:41.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.385+0000 7f68d4a65640 1 --2- 172.21.15.37:0/2942034055 >> v2:172.21.15.45:6812/2379918623 conn(0x7f68b0035e30 0x7f68b00382f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f68c0009580 tx=0x7f68c0009190 comp rx=0 tx=0).stop 2024-04-26T00:30:41.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.385+0000 7f68d4a65640 10 monclient: shutdown 2024-04-26T00:30:41.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.385+0000 7f68d4a65640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:30:41.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.385+0000 7f68d4a65640 1 -- 172.21.15.37:0/2942034055 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f68d0139050 msgr2=0x7f68d01ccae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:41.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.385+0000 7f68d4a65640 1 --2- 172.21.15.37:0/2942034055 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f68d0139050 0x7f68d01ccae0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f68c801ad50 tx=0x7f68c8019440 comp rx=0 tx=0).stop 2024-04-26T00:30:41.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.385+0000 7f68d4a65640 1 -- 172.21.15.37:0/2942034055 shutdown_connections 2024-04-26T00:30:41.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.385+0000 7f68d4a65640 1 --2- 172.21.15.37:0/2942034055 >> v2:172.21.15.45:6812/2379918623 conn(0x7f68b0035e30 0x7f68b00382f0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:41.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.385+0000 7f68d4a65640 1 --2- 172.21.15.37:0/2942034055 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f68d0139050 0x7f68d01ccae0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:41.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.385+0000 7f68d4a65640 1 --2- 172.21.15.37:0/2942034055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68d0133f00 0x7f68d0132c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:41.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.385+0000 7f68d4a65640 1 --2- 172.21.15.37:0/2942034055 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68d0133b20 0x7f68d0072700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:41.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.385+0000 7f68d4a65640 1 -- 172.21.15.37:0/2942034055 >> 172.21.15.37:0/2942034055 conn(0x7f68d00a1ad0 msgr2=0x7f68d009c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:30:41.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.385+0000 7f68d4a65640 1 -- 172.21.15.37:0/2942034055 shutdown_connections 2024-04-26T00:30:41.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:41.385+0000 7f68d4a65640 1 -- 172.21.15.37:0/2942034055 wait complete. 2024-04-26T00:30:44.394 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:30:44.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.533+0000 7f3de6b12640 10 monclient: get_monmap_and_config 2024-04-26T00:30:44.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.533+0000 7f3de6b12640 10 monclient: build_initial_monmap 2024-04-26T00:30:44.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.533+0000 7f3de6b12640 10 monclient: monmap: 2024-04-26T00:30:44.527 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:30:44.528 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:44.528 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:30:44.539530+0000 2024-04-26T00:30:44.528 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:30:44.539530+0000 2024-04-26T00:30:44.528 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:30:44.528 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:44.528 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:30:44.528 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:30:44.528 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:30:44.528 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:30:44.528 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:30:44.528 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:44.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.537+0000 7f3de6b12640 1 Processor -- start 2024-04-26T00:30:44.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.537+0000 7f3de6b12640 1 -- start start 2024-04-26T00:30:44.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.537+0000 7f3de6b12640 10 monclient: init 2024-04-26T00:30:44.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.537+0000 7f3de6b12640 10 monclient: _reopen_session rank -1 2024-04-26T00:30:44.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.537+0000 7f3de6b12640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-04-26T00:30:44.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.537+0000 7f3de6b12640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3de00a0260 addr v1:172.21.15.37:6789/0 2024-04-26T00:30:44.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.537+0000 7f3de6b12640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3de0137e40 0x7f3de013a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:44.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.537+0000 7f3de6b12640 10 monclient(hunting): picked mon.noname-c con 0x7f3de0137e40 addr v2:172.21.15.45:3300/0 2024-04-26T00:30:44.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.537+0000 7f3de6b12640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3de013a770 0x7f3de013cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:44.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.537+0000 7f3de6b12640 10 monclient(hunting): picked mon.noname-b con 0x7f3de013a770 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:30:44.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.537+0000 7f3de6b12640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3de00a1130 con 0x7f3de00a0260 2024-04-26T00:30:44.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.537+0000 7f3de6b12640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:44.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.537+0000 7f3de6b12640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f3de00a0d50 con 0x7f3de0137e40 2024-04-26T00:30:44.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.537+0000 7f3de6b12640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:44.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.537+0000 7f3de6b12640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f3de009ec10 con 0x7f3de013a770 2024-04-26T00:30:44.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.537+0000 7f3de6b12640 10 monclient(hunting): _renew_subs 2024-04-26T00:30:44.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.537+0000 7f3de6b12640 10 monclient(hunting): authenticate will time out at 1625.845184s 2024-04-26T00:30:44.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.537+0000 7f3de50af640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3de013a770 0x7f3de013cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:44.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.537+0000 7f3de48ae640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f3de00a0260 0x7f3de00a0640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:59570/0 (socket says 172.21.15.37:59570) 2024-04-26T00:30:44.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.537+0000 7f3de48ae640 1 -- 172.21.15.37:0/2140784193 learned_addr learned my addr 172.21.15.37:0/2140784193 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:30:44.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.537+0000 7f3ddffff640 1 -- >> v2:172.21.15.45:3300/0 conn(0x7f3de0137e40 msgr2=0x7f3de013a230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:30:44.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.537+0000 7f3ddffff640 1 --2- 172.21.15.37:0/2140784193 >> v2:172.21.15.45:3300/0 conn(0x7f3de0137e40 0x7f3de013a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:30:44.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.537+0000 7f3de50af640 10 monclient(hunting): get_auth_request con 0x7f3de013a770 auth_method 0 2024-04-26T00:30:44.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.537+0000 7f3de50af640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:44.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.537+0000 7f3de50af640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:44.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.537+0000 7f3de50af640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:44.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.537+0000 7f3de50af640 1 -- 172.21.15.37:0/2140784193 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3de013a770 msgr2=0x7f3de013cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:30:44.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.537+0000 7f3de50af640 1 -- 172.21.15.37:0/2140784193 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3de013a770 msgr2=0x7f3de013cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:30:44.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.537+0000 7f3de50af640 1 --2- 172.21.15.37:0/2140784193 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3de013a770 0x7f3de013cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:30:44.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.537+0000 7f3de50af640 1 --2- 172.21.15.37:0/2140784193 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3de013a770 0x7f3de013cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:30:44.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.537+0000 7f3de48ae640 1 -- 172.21.15.37:0/2140784193 >> v1:172.21.15.37:6789/0 conn(0x7f3de00a0260 legacy=0x7f3de00a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:44.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.741+0000 7f3ddffff640 1 -- 172.21.15.37:0/2140784193 >> v2:172.21.15.45:3300/0 conn(0x7f3de0137e40 msgr2=0x7f3de013a230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:30:44.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.741+0000 7f3de50af640 1 --2- 172.21.15.37:0/2140784193 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3de013a770 0x7f3de013cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:44.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.741+0000 7f3ddffff640 1 --2- 172.21.15.37:0/2140784193 >> v2:172.21.15.45:3300/0 conn(0x7f3de0137e40 0x7f3de013a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:30:44.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.741+0000 7f3de50af640 10 monclient(hunting): get_auth_request con 0x7f3de013a770 auth_method 0 2024-04-26T00:30:44.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.741+0000 7f3de50af640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:44.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.741+0000 7f3de50af640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:44.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.741+0000 7f3de50af640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:30:44.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.741+0000 7f3de50af640 1 -- 172.21.15.37:0/2140784193 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3de013a770 msgr2=0x7f3de013cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:30:44.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.741+0000 7f3de50af640 1 -- 172.21.15.37:0/2140784193 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3de013a770 msgr2=0x7f3de013cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:30:44.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.741+0000 7f3de50af640 1 --2- 172.21.15.37:0/2140784193 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3de013a770 0x7f3de013cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:30:44.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:44.741+0000 7f3de50af640 1 --2- 172.21.15.37:0/2140784193 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3de013a770 0x7f3de013cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:30:45.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:45.145+0000 7f3ddffff640 1 -- 172.21.15.37:0/2140784193 >> v2:172.21.15.45:3300/0 conn(0x7f3de0137e40 msgr2=0x7f3de013a230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:30:45.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:45.145+0000 7f3de50af640 1 --2- 172.21.15.37:0/2140784193 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3de013a770 0x7f3de013cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:45.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:45.145+0000 7f3ddffff640 1 --2- 172.21.15.37:0/2140784193 >> v2:172.21.15.45:3300/0 conn(0x7f3de0137e40 0x7f3de013a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:30:45.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:45.145+0000 7f3de50af640 10 monclient(hunting): get_auth_request con 0x7f3de013a770 auth_method 0 2024-04-26T00:30:45.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:45.145+0000 7f3de50af640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:45.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:45.145+0000 7f3de50af640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:45.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:45.145+0000 7f3de50af640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:30:45.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:45.145+0000 7f3de50af640 1 -- 172.21.15.37:0/2140784193 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3de013a770 msgr2=0x7f3de013cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:30:45.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:45.145+0000 7f3de50af640 1 -- 172.21.15.37:0/2140784193 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3de013a770 msgr2=0x7f3de013cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:30:45.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:45.145+0000 7f3de50af640 1 --2- 172.21.15.37:0/2140784193 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3de013a770 0x7f3de013cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:30:45.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:45.145+0000 7f3de50af640 1 --2- 172.21.15.37:0/2140784193 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3de013a770 0x7f3de013cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:30:45.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:45.949+0000 7f3ddffff640 1 -- 172.21.15.37:0/2140784193 >> v2:172.21.15.45:3300/0 conn(0x7f3de0137e40 msgr2=0x7f3de013a230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:30:45.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:45.949+0000 7f3ddffff640 1 --2- 172.21.15.37:0/2140784193 >> v2:172.21.15.45:3300/0 conn(0x7f3de0137e40 0x7f3de013a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:30:45.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:45.949+0000 7f3de50af640 1 --2- 172.21.15.37:0/2140784193 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3de013a770 0x7f3de013cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:45.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:45.949+0000 7f3de50af640 10 monclient(hunting): get_auth_request con 0x7f3de013a770 auth_method 0 2024-04-26T00:30:45.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:45.949+0000 7f3de50af640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:45.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:45.949+0000 7f3de50af640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:45.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:45.949+0000 7f3de50af640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:30:45.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:45.949+0000 7f3de50af640 1 -- 172.21.15.37:0/2140784193 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3de013a770 msgr2=0x7f3de013cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:30:45.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:45.949+0000 7f3de50af640 1 -- 172.21.15.37:0/2140784193 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3de013a770 msgr2=0x7f3de013cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:30:45.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:45.949+0000 7f3de50af640 1 --2- 172.21.15.37:0/2140784193 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3de013a770 0x7f3de013cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:30:45.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:45.949+0000 7f3de50af640 1 --2- 172.21.15.37:0/2140784193 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3de013a770 0x7f3de013cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:30:47.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:47.537+0000 7f3dde7fc640 10 monclient(hunting): tick 2024-04-26T00:30:47.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:47.537+0000 7f3dde7fc640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:30:47.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:47.537+0000 7f3dde7fc640 1 monclient(hunting): continuing hunt 2024-04-26T00:30:47.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:47.537+0000 7f3dde7fc640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:30:47.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:47.537+0000 7f3dde7fc640 1 -- 172.21.15.37:0/2140784193 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3de013a770 msgr2=0x7f3de013cb60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:30:47.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:47.537+0000 7f3dde7fc640 1 --2- 172.21.15.37:0/2140784193 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3de013a770 0x7f3de013cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:47.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:47.537+0000 7f3dde7fc640 1 -- 172.21.15.37:0/2140784193 >> v2:172.21.15.45:3300/0 conn(0x7f3de0137e40 msgr2=0x7f3de013a230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:30:47.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:47.537+0000 7f3dde7fc640 1 --2- 172.21.15.37:0/2140784193 >> v2:172.21.15.45:3300/0 conn(0x7f3de0137e40 0x7f3de013a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:47.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:47.537+0000 7f3dde7fc640 1 -- 172.21.15.37:0/2140784193 >> v1:172.21.15.37:6789/0 conn(0x7f3de00a0260 legacy=0x7f3de00a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:47.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:47.537+0000 7f3dde7fc640 10 monclient: _add_conns ranks=[3,1] 2024-04-26T00:30:47.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:47.537+0000 7f3dde7fc640 1 --2- 172.21.15.37:0/2140784193 >> v2:172.21.15.37:3300/0 conn(0x7f3dcc003750 0x7f3dcc005bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:47.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:47.537+0000 7f3dde7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f3dcc003750 addr v2:172.21.15.37:3300/0 2024-04-26T00:30:47.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:47.537+0000 7f3dde7fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3dcc0062c0 addr v1:172.21.15.45:6789/0 2024-04-26T00:30:47.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:47.537+0000 7f3dde7fc640 1 -- 172.21.15.37:0/2140784193 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3de006cff0 con 0x7f3dcc0062c0 2024-04-26T00:30:47.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:47.537+0000 7f3dde7fc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:47.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:47.537+0000 7f3dde7fc640 1 -- 172.21.15.37:0/2140784193 --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f3de00a0d50 con 0x7f3dcc003750 2024-04-26T00:30:47.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:47.537+0000 7f3dde7fc640 10 monclient(hunting): _renew_subs 2024-04-26T00:30:47.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:47.537+0000 7f3ddffff640 1 -- 172.21.15.37:0/2140784193 >> v1:172.21.15.45:6789/0 conn(0x7f3dcc0062c0 legacy=0x7f3dcc008760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:30:47.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:47.537+0000 7f3de48ae640 1 --2- 172.21.15.37:0/2140784193 >> v2:172.21.15.37:3300/0 conn(0x7f3dcc003750 0x7f3dcc005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:47.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:47.537+0000 7f3de48ae640 10 monclient(hunting): get_auth_request con 0x7f3dcc003750 auth_method 0 2024-04-26T00:30:47.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:47.537+0000 7f3de48ae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:47.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:47.537+0000 7f3de48ae640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:47.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:47.537+0000 7f3de48ae640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:47.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:47.537+0000 7f3de48ae640 1 -- 172.21.15.37:0/2140784193 >> v2:172.21.15.37:3300/0 conn(0x7f3dcc003750 msgr2=0x7f3dcc005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:30:47.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:47.537+0000 7f3de48ae640 1 -- 172.21.15.37:0/2140784193 >> v2:172.21.15.37:3300/0 conn(0x7f3dcc003750 msgr2=0x7f3dcc005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:30:47.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:47.537+0000 7f3de48ae640 1 --2- 172.21.15.37:0/2140784193 >> v2:172.21.15.37:3300/0 conn(0x7f3dcc003750 0x7f3dcc005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:30:47.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:47.537+0000 7f3de48ae640 1 --2- 172.21.15.37:0/2140784193 >> v2:172.21.15.37:3300/0 conn(0x7f3dcc003750 0x7f3dcc005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:30:47.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:47.741+0000 7f3ddffff640 1 -- 172.21.15.37:0/2140784193 >> v1:172.21.15.45:6789/0 conn(0x7f3dcc0062c0 legacy=0x7f3dcc008760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:30:47.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:47.741+0000 7f3de48ae640 1 --2- 172.21.15.37:0/2140784193 >> v2:172.21.15.37:3300/0 conn(0x7f3dcc003750 0x7f3dcc005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:47.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:47.741+0000 7f3de48ae640 10 monclient(hunting): get_auth_request con 0x7f3dcc003750 auth_method 0 2024-04-26T00:30:47.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:47.741+0000 7f3de48ae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:47.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:47.741+0000 7f3de48ae640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:47.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:47.741+0000 7f3de48ae640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:30:47.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:47.741+0000 7f3de48ae640 1 -- 172.21.15.37:0/2140784193 >> v2:172.21.15.37:3300/0 conn(0x7f3dcc003750 msgr2=0x7f3dcc005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:30:47.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:47.741+0000 7f3de48ae640 1 -- 172.21.15.37:0/2140784193 >> v2:172.21.15.37:3300/0 conn(0x7f3dcc003750 msgr2=0x7f3dcc005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:30:47.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:47.741+0000 7f3de48ae640 1 --2- 172.21.15.37:0/2140784193 >> v2:172.21.15.37:3300/0 conn(0x7f3dcc003750 0x7f3dcc005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:30:47.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:47.741+0000 7f3de48ae640 1 --2- 172.21.15.37:0/2140784193 >> v2:172.21.15.37:3300/0 conn(0x7f3dcc003750 0x7f3dcc005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:30:48.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.145+0000 7f3ddffff640 1 -- 172.21.15.37:0/2140784193 >> v1:172.21.15.45:6789/0 conn(0x7f3dcc0062c0 legacy=0x7f3dcc008760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:30:48.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.145+0000 7f3de48ae640 1 --2- 172.21.15.37:0/2140784193 >> v2:172.21.15.37:3300/0 conn(0x7f3dcc003750 0x7f3dcc005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:48.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.145+0000 7f3de48ae640 10 monclient(hunting): get_auth_request con 0x7f3dcc003750 auth_method 0 2024-04-26T00:30:48.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.145+0000 7f3de48ae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:48.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.145+0000 7f3de48ae640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:48.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.145+0000 7f3de48ae640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:30:48.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.145+0000 7f3de48ae640 1 -- 172.21.15.37:0/2140784193 >> v2:172.21.15.37:3300/0 conn(0x7f3dcc003750 msgr2=0x7f3dcc005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:30:48.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.145+0000 7f3de48ae640 1 -- 172.21.15.37:0/2140784193 >> v2:172.21.15.37:3300/0 conn(0x7f3dcc003750 msgr2=0x7f3dcc005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:30:48.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.145+0000 7f3de48ae640 1 --2- 172.21.15.37:0/2140784193 >> v2:172.21.15.37:3300/0 conn(0x7f3dcc003750 0x7f3dcc005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:30:48.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.145+0000 7f3de48ae640 1 --2- 172.21.15.37:0/2140784193 >> v2:172.21.15.37:3300/0 conn(0x7f3dcc003750 0x7f3dcc005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:30:48.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.949+0000 7f3ddffff640 1 -- 172.21.15.37:0/2140784193 >> v1:172.21.15.45:6789/0 conn(0x7f3dcc0062c0 legacy=0x7f3dcc008760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:30:48.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.949+0000 7f3de48ae640 1 --2- 172.21.15.37:0/2140784193 >> v2:172.21.15.37:3300/0 conn(0x7f3dcc003750 0x7f3dcc005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:48.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.949+0000 7f3de48ae640 10 monclient(hunting): get_auth_request con 0x7f3dcc003750 auth_method 0 2024-04-26T00:30:48.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.949+0000 7f3de48ae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:48.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.949+0000 7f3de48ae640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:48.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.949+0000 7f3de48ae640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:30:48.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.949+0000 7f3de48ae640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:48.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.949+0000 7f3de48ae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:48.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.949+0000 7f3de48ae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:48.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.949+0000 7f3de48ae640 10 monclient(hunting): handle_auth_done global_id 124167 payload 274 2024-04-26T00:30:48.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.949+0000 7f3de48ae640 1 -- 172.21.15.37:0/2140784193 >> v1:172.21.15.45:6789/0 conn(0x7f3dcc0062c0 legacy=0x7f3dcc008760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:30:48.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.949+0000 7f3de48ae640 10 monclient: _finish_hunting 0 2024-04-26T00:30:48.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.949+0000 7f3de48ae640 1 monclient: found mon.noname-a 2024-04-26T00:30:48.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.949+0000 7f3de48ae640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:30:48.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.949+0000 7f3de48ae640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:30:48.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.949+0000 7f3de48ae640 1 -- 172.21.15.37:0/2140784193 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dcc00b050 con 0x7f3dcc003750 2024-04-26T00:30:48.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.957+0000 7f3de48ae640 1 -- 172.21.15.37:0/2140784193 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3dcc003750 msgr2=0x7f3dcc005bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:48.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.957+0000 7f3de48ae640 1 --2- 172.21.15.37:0/2140784193 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3dcc003750 0x7f3dcc005bd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3dd0007bd0 tx=0x7f3dd001ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=287aa7b2f745bf90 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:48.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.957+0000 7f3ddf7fe640 1 -- 172.21.15.37:0/2140784193 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd0004c50 con 0x7f3dcc003750 2024-04-26T00:30:48.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.957+0000 7f3ddf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:48.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.957+0000 7f3ddf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:30:48.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.957+0000 7f3ddf7fe640 10 monclient: dump: 2024-04-26T00:30:48.950 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:48.950 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:48.950 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:48.950 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:48.950 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:48.950 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:48.950 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:48.950 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:48.950 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:48.951 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:48.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.957+0000 7f3ddf7fe640 10 monclient: _finish_auth 0 2024-04-26T00:30:48.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.957+0000 7f3ddf7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:30:48.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.957+0000 7f3ddf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:30:48.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.957+0000 7f3ddf7fe640 1 -- 172.21.15.37:0/2140784193 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dd00a9070 con 0x7f3dcc003750 2024-04-26T00:30:48.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.957+0000 7f3ddf7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:30:48.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.957+0000 7f3ddf7fe640 1 -- 172.21.15.37:0/2140784193 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd0005030 con 0x7f3dcc003750 2024-04-26T00:30:48.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.957+0000 7f3ddf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:48.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.957+0000 7f3ddf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:30:48.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.957+0000 7f3ddf7fe640 10 monclient: dump: 2024-04-26T00:30:48.951 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:48.951 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:48.952 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:48.952 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:48.952 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:48.952 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:48.952 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:48.952 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:48.952 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:48.952 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:48.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.957+0000 7f3de6b12640 5 monclient: authenticate success, global_id 124167 2024-04-26T00:30:48.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.957+0000 7f3de6b12640 10 monclient: get_monmap_and_config success 2024-04-26T00:30:48.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.957+0000 7f3de6b12640 4 set_mon_vals no callback set 2024-04-26T00:30:48.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.957+0000 7f3de6b12640 10 monclient: shutdown 2024-04-26T00:30:48.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.957+0000 7f3de6b12640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:30:48.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.957+0000 7f3de6b12640 1 -- 172.21.15.37:0/2140784193 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3dcc003750 msgr2=0x7f3dcc005bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:48.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.957+0000 7f3de6b12640 1 --2- 172.21.15.37:0/2140784193 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3dcc003750 0x7f3dcc005bd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3dd0007bd0 tx=0x7f3dd001ca90 comp rx=0 tx=0).stop 2024-04-26T00:30:48.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.957+0000 7f3de50af640 1 -- 172.21.15.37:0/2140784193 reap_dead start 2024-04-26T00:30:48.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.957+0000 7f3de6b12640 1 -- 172.21.15.37:0/2140784193 shutdown_connections 2024-04-26T00:30:48.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.957+0000 7f3de6b12640 1 --2- 172.21.15.37:0/2140784193 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3dcc003750 0x7f3dcc005bd0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:48.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.957+0000 7f3de6b12640 1 -- 172.21.15.37:0/2140784193 >> 172.21.15.37:0/2140784193 conn(0x7f3de00a1ad0 msgr2=0x7f3de00a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:30:48.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.957+0000 7f3de6b12640 1 -- 172.21.15.37:0/2140784193 shutdown_connections 2024-04-26T00:30:48.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.957+0000 7f3de6b12640 1 -- 172.21.15.37:0/2140784193 wait complete. 2024-04-26T00:30:48.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3de6b12640 10 monclient: build_initial_monmap 2024-04-26T00:30:48.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3de6b12640 10 monclient: monmap: 2024-04-26T00:30:48.957 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:30:48.957 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:30:48.957 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:30:48.957 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:30:48.957 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:30:48.957 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:30:48.957 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:30:48.957 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:30:48.957 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:30:48.958 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:48.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3de6b12640 1 Processor -- start 2024-04-26T00:30:48.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3de6b12640 1 -- start start 2024-04-26T00:30:48.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3de6b12640 10 monclient: init 2024-04-26T00:30:48.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3de6b12640 10 monclient: _reopen_session rank -1 2024-04-26T00:30:48.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3de6b12640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:30:48.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3de6b12640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3dcc003750 0x7f3de0136930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:48.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3de6b12640 10 monclient(hunting): picked mon.noname-c con 0x7f3dcc003750 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:30:48.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3de6b12640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3de011d350 0x7f3de011d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:48.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3de6b12640 10 monclient(hunting): picked mon.noname-a con 0x7f3de011d350 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:30:48.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3de6b12640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3de011dd10 0x7f3de011e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:48.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3de6b12640 10 monclient(hunting): picked mon.noname-b con 0x7f3de011dd10 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:30:48.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3de6b12640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:48.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3de6b12640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f3de009f0d0 con 0x7f3de011d350 2024-04-26T00:30:48.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3de6b12640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:48.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3de6b12640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3de009f4c0 con 0x7f3de011dd10 2024-04-26T00:30:48.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3de6b12640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:48.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3de6b12640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f3de013f8c0 con 0x7f3dcc003750 2024-04-26T00:30:48.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3de6b12640 10 monclient(hunting): _renew_subs 2024-04-26T00:30:48.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3de6b12640 10 monclient(hunting): authenticate will time out at 1630.270197s 2024-04-26T00:30:48.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3de50af640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3de011dd10 msgr2=0x7f3de011e1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:30:48.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3de48ae640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3dcc003750 0x7f3de0136930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:48.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3de50af640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3de011dd10 0x7f3de011e1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:30:48.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3ddd7fa640 1 -- DispatchQueue::entry inject delay of 0.002000 2024-04-26T00:30:48.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3de48ae640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3dcc003750 0x7f3de0136930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:48632/0 (socket says 172.21.15.37:48632) 2024-04-26T00:30:48.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3ddffff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3de011d350 0x7f3de011d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:48.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3de48ae640 1 -- 172.21.15.37:0/838782529 learned_addr learned my addr 172.21.15.37:0/838782529 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:30:48.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3de48ae640 10 monclient(hunting): get_auth_request con 0x7f3dcc003750 auth_method 0 2024-04-26T00:30:48.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3de48ae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:48.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3de48ae640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:48.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3de48ae640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:48.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3ddffff640 10 monclient(hunting): get_auth_request con 0x7f3de011d350 auth_method 0 2024-04-26T00:30:48.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3ddffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:48.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3ddffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:48.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3ddffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:48.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3ddffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:48.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3ddffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:48.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3ddffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:48.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3de48ae640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:48.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3de48ae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:48.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3de48ae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:48.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3ddffff640 10 monclient(hunting): handle_auth_done global_id 124170 payload 824 2024-04-26T00:30:48.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3ddffff640 1 -- 172.21.15.37:0/838782529 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3dcc003750 msgr2=0x7f3de0136930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:48.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3ddffff640 1 --2- 172.21.15.37:0/838782529 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3dcc003750 0x7f3de0136930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:48.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3ddffff640 1 -- 172.21.15.37:0/838782529 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3de011dd10 msgr2=0x7f3de011e1a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:30:48.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3ddffff640 1 --2- 172.21.15.37:0/838782529 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3de011dd10 0x7f3de011e1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:48.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3ddffff640 10 monclient: _finish_hunting 0 2024-04-26T00:30:48.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3ddffff640 1 monclient: found mon.noname-a 2024-04-26T00:30:48.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3ddffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:30:48.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3ddffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:30:48.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3ddffff640 1 -- 172.21.15.37:0/838782529 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3de01cd6f0 con 0x7f3de011d350 2024-04-26T00:30:48.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.961+0000 7f3de48ae640 1 --2- 172.21.15.37:0/838782529 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3dcc003750 0x7f3de0136930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:30:48.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.965+0000 7f3ddffff640 1 -- 172.21.15.37:0/838782529 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3de011d350 msgr2=0x7f3de011d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:48.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.965+0000 7f3ddffff640 1 --2- 172.21.15.37:0/838782529 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3de011d350 0x7f3de011d7d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3dd400b850 tx=0x7f3dd400bc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:48.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.965+0000 7f3ddd7fa640 1 -- 172.21.15.37:0/838782529 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd400cce0 con 0x7f3de011d350 2024-04-26T00:30:48.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.965+0000 7f3ddd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:48.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.965+0000 7f3ddd7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:30:48.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.965+0000 7f3ddd7fa640 10 monclient: dump: 2024-04-26T00:30:48.965 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:48.965 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:48.965 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:48.965 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:48.965 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:48.965 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:48.965 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:48.965 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:48.965 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:48.965 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:48.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.965+0000 7f3ddd7fa640 10 monclient: _finish_auth 0 2024-04-26T00:30:48.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.965+0000 7f3ddd7fa640 10 monclient: _check_auth_tickets 2024-04-26T00:30:48.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.965+0000 7f3ddd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:30:48.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.965+0000 7f3ddd7fa640 1 -- 172.21.15.37:0/838782529 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dd4010030 con 0x7f3de011d350 2024-04-26T00:30:48.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.965+0000 7f3ddd7fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:30:48.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.965+0000 7f3ddd7fa640 1 -- 172.21.15.37:0/838782529 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd4016670 con 0x7f3de011d350 2024-04-26T00:30:48.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.965+0000 7f3ddd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:48.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.965+0000 7f3ddd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:30:48.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.965+0000 7f3ddd7fa640 10 monclient: dump: 2024-04-26T00:30:48.966 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:48.966 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:48.966 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:48.967 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:48.967 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:48.967 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:48.967 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:48.967 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:48.967 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:48.967 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:48.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.965+0000 7f3de6b12640 5 monclient: authenticate success, global_id 124170 2024-04-26T00:30:48.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.965+0000 7f3ddeffd640 4 set_mon_vals no callback set 2024-04-26T00:30:48.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.965+0000 7f3de6b12640 10 monclient: _renew_subs 2024-04-26T00:30:48.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.965+0000 7f3de6b12640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:30:48.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.965+0000 7f3de6b12640 1 -- 172.21.15.37:0/838782529 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3de00729d0 con 0x7f3de011d350 2024-04-26T00:30:48.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.965+0000 7f3de6b12640 10 monclient: _renew_subs 2024-04-26T00:30:48.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.965+0000 7f3de6b12640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:30:48.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.965+0000 7f3de6b12640 1 -- 172.21.15.37:0/838782529 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3de01212e0 con 0x7f3de011d350 2024-04-26T00:30:48.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.965+0000 7f3ddd7fa640 1 -- 172.21.15.37:0/838782529 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f3dd40153e0 con 0x7f3de011d350 2024-04-26T00:30:48.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.965+0000 7f3de6b12640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:30:48.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.965+0000 7f3de6b12640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:30:48.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.965+0000 7f3de6b12640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:30:48.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.965+0000 7f3ddd7fa640 1 --2- 172.21.15.37:0/838782529 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3dcc036da0 0x7f3dcc039260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:48.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.965+0000 7f3de6b12640 1 -- 172.21.15.37:0/838782529 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3de0066f40 con 0x7f3de011d350 2024-04-26T00:30:48.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.965+0000 7f3ddd7fa640 1 -- 172.21.15.37:0/838782529 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(405..405 src has 370..405) ==== 3979+0+0 (secure 0 0 0) 0x7f3dd40484b0 con 0x7f3de011d350 2024-04-26T00:30:48.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.965+0000 7f3de48ae640 1 --2- 172.21.15.37:0/838782529 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3dcc036da0 0x7f3dcc039260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:48.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.965+0000 7f3de48ae640 10 monclient: get_auth_request con 0x7f3dcc036da0 auth_method 0 2024-04-26T00:30:48.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.969+0000 7f3de48ae640 1 --2- 172.21.15.37:0/838782529 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3dcc036da0 0x7f3dcc039260 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f3dd001b140 tx=0x7f3dd00b2040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:48.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.973+0000 7f3ddd7fa640 1 -- 172.21.15.37:0/838782529 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f3dd401b030 con 0x7f3de011d350 2024-04-26T00:30:48.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.973+0000 7f3ddd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:30:48.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:48.973+0000 7f3ddd7fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:30:49.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.201+0000 7f3de6b12640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:30:49.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.201+0000 7f3de6b12640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:30:49.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.201+0000 7f3de6b12640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:30:49.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.205+0000 7f3de6b12640 1 -- 172.21.15.37:0/838782529 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3de009b450 con 0x7f3de011d350 2024-04-26T00:30:49.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.205+0000 7f3ddd7fa640 1 -- 172.21.15.37:0/838782529 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7f3de01cd6f0 con 0x7f3de011d350 2024-04-26T00:30:49.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.205+0000 7f3ddd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:30:49.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.205+0000 7f3ddd7fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:30:49.197 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":84,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:30:49.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.205+0000 7f3de6b12640 1 -- 172.21.15.37:0/838782529 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3dcc036da0 msgr2=0x7f3dcc039260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:49.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.205+0000 7f3de6b12640 1 --2- 172.21.15.37:0/838782529 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3dcc036da0 0x7f3dcc039260 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f3dd001b140 tx=0x7f3dd00b2040 comp rx=0 tx=0).stop 2024-04-26T00:30:49.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.205+0000 7f3de6b12640 10 monclient: shutdown 2024-04-26T00:30:49.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.205+0000 7f3de6b12640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:30:49.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.205+0000 7f3de6b12640 1 -- 172.21.15.37:0/838782529 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3de011d350 msgr2=0x7f3de011d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:49.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.205+0000 7f3de6b12640 1 --2- 172.21.15.37:0/838782529 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3de011d350 0x7f3de011d7d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3dd400b850 tx=0x7f3dd400bc50 comp rx=0 tx=0).stop 2024-04-26T00:30:49.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.205+0000 7f3de6b12640 1 -- 172.21.15.37:0/838782529 shutdown_connections 2024-04-26T00:30:49.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.205+0000 7f3de6b12640 1 --2- 172.21.15.37:0/838782529 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3dcc036da0 0x7f3dcc039260 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:49.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.205+0000 7f3de6b12640 1 --2- 172.21.15.37:0/838782529 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3de011dd10 0x7f3de011e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:49.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.205+0000 7f3de6b12640 1 --2- 172.21.15.37:0/838782529 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3de011d350 0x7f3de011d7d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:49.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.205+0000 7f3de6b12640 1 --2- 172.21.15.37:0/838782529 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3dcc003750 0x7f3de0136930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:49.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.205+0000 7f3de6b12640 1 -- 172.21.15.37:0/838782529 >> 172.21.15.37:0/838782529 conn(0x7f3de00a1ad0 msgr2=0x7f3de00a3d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:30:49.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.205+0000 7f3de6b12640 1 -- 172.21.15.37:0/838782529 shutdown_connections 2024-04-26T00:30:49.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.205+0000 7f3de6b12640 1 -- 172.21.15.37:0/838782529 wait complete. 2024-04-26T00:30:49.216 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-26T00:30:49.216 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-04-26T00:30:49.217 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:30:49.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0f6a82640 10 monclient: get_monmap_and_config 2024-04-26T00:30:49.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0f6a82640 10 monclient: build_initial_monmap 2024-04-26T00:30:49.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0f6a82640 10 monclient: monmap: 2024-04-26T00:30:49.337 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:30:49.337 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:49.337 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:30:49.348613+0000 2024-04-26T00:30:49.337 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:30:49.348613+0000 2024-04-26T00:30:49.337 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:30:49.337 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:49.337 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:30:49.337 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:30:49.337 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:30:49.338 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:30:49.338 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:30:49.338 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:49.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0f6a82640 1 Processor -- start 2024-04-26T00:30:49.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0f6a82640 1 -- start start 2024-04-26T00:30:49.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0f6a82640 10 monclient: init 2024-04-26T00:30:49.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0f6a82640 10 monclient: _reopen_session rank -1 2024-04-26T00:30:49.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0f6a82640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-04-26T00:30:49.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0f6a82640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd0f00a0300 0x7fd0f009fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:49.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0f6a82640 10 monclient(hunting): picked mon.noname-c con 0x7fd0f00a0300 addr v2:172.21.15.45:3300/0 2024-04-26T00:30:49.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0f6a82640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0f00a06e0 0x7fd0f0134ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:49.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0f6a82640 10 monclient(hunting): picked mon.noname-b con 0x7fd0f00a06e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:30:49.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0f6a82640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd0f01350e0 addr v1:172.21.15.37:6789/0 2024-04-26T00:30:49.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0f6a82640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0f009c190 con 0x7fd0f01350e0 2024-04-26T00:30:49.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0f6a82640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:49.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0f6a82640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fd0f00a13f0 con 0x7fd0f00a0300 2024-04-26T00:30:49.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0f6a82640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:49.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0f6a82640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd0f00a1110 con 0x7fd0f00a06e0 2024-04-26T00:30:49.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0f6a82640 10 monclient(hunting): _renew_subs 2024-04-26T00:30:49.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0f6a82640 10 monclient(hunting): authenticate will time out at 1630.654318s 2024-04-26T00:30:49.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0f501f640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fd0f01350e0 0x7fd0f013bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:59576/0 (socket says 172.21.15.37:59576) 2024-04-26T00:30:49.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0f501f640 1 -- 172.21.15.37:0/4048372930 learned_addr learned my addr 172.21.15.37:0/4048372930 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:30:49.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0f481e640 1 -- 172.21.15.37:0/4048372930 >> v2:172.21.15.45:3300/0 conn(0x7fd0f00a0300 msgr2=0x7fd0f009fc50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:30:49.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0effff640 1 --2- 172.21.15.37:0/4048372930 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0f00a06e0 0x7fd0f0134ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:49.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0f481e640 1 --2- 172.21.15.37:0/4048372930 >> v2:172.21.15.45:3300/0 conn(0x7fd0f00a0300 0x7fd0f009fc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:30:49.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0effff640 10 monclient(hunting): get_auth_request con 0x7fd0f00a06e0 auth_method 0 2024-04-26T00:30:49.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:49.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0effff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:49.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0effff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:49.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:49.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:49.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:49.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0effff640 10 monclient(hunting): handle_auth_done global_id 84533 payload 274 2024-04-26T00:30:49.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0effff640 1 -- 172.21.15.37:0/4048372930 >> v2:172.21.15.45:3300/0 conn(0x7fd0f00a0300 msgr2=0x7fd0f009fc50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:30:49.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0effff640 1 --2- 172.21.15.37:0/4048372930 >> v2:172.21.15.45:3300/0 conn(0x7fd0f00a0300 0x7fd0f009fc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:49.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0effff640 1 -- 172.21.15.37:0/4048372930 >> v1:172.21.15.37:6789/0 conn(0x7fd0f01350e0 legacy=0x7fd0f013bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:30:49.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0effff640 10 monclient: _finish_hunting 0 2024-04-26T00:30:49.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0effff640 1 monclient: found mon.noname-b 2024-04-26T00:30:49.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:30:49.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0effff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:30:49.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.345+0000 7fd0effff640 1 -- 172.21.15.37:0/4048372930 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0f00a16d0 con 0x7fd0f00a06e0 2024-04-26T00:30:49.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.349+0000 7fd0effff640 1 -- 172.21.15.37:0/4048372930 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0f00a06e0 msgr2=0x7fd0f0134ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:49.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.349+0000 7fd0effff640 1 --2- 172.21.15.37:0/4048372930 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0f00a06e0 0x7fd0f0134ba0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd0e40097c0 tx=0x7fd0e401c370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fd2503f91dcb744 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:49.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.349+0000 7fd0ef7fe640 1 -- 172.21.15.37:0/4048372930 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e40b0070 con 0x7fd0f00a06e0 2024-04-26T00:30:49.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.349+0000 7fd0ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:49.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.349+0000 7fd0ef7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:30:49.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.349+0000 7fd0ef7fe640 10 monclient: dump: 2024-04-26T00:30:49.345 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:49.345 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:49.345 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:49.345 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:49.345 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:49.345 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:49.345 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:49.345 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:49.345 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:49.345 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:49.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.349+0000 7fd0ef7fe640 10 monclient: _finish_auth 0 2024-04-26T00:30:49.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.349+0000 7fd0ef7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:30:49.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.349+0000 7fd0ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:30:49.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.349+0000 7fd0ef7fe640 1 -- 172.21.15.37:0/4048372930 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0e40ab600 con 0x7fd0f00a06e0 2024-04-26T00:30:49.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.349+0000 7fd0f6a82640 5 monclient: authenticate success, global_id 84533 2024-04-26T00:30:49.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.349+0000 7fd0f6a82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:30:49.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.349+0000 7fd0ef7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:30:49.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.349+0000 7fd0f6a82640 10 monclient: get_monmap_and_config success 2024-04-26T00:30:49.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.349+0000 7fd0ef7fe640 1 -- 172.21.15.37:0/4048372930 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e40ab9d0 con 0x7fd0f00a06e0 2024-04-26T00:30:49.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.349+0000 7fd0f6a82640 4 set_mon_vals no callback set 2024-04-26T00:30:49.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.349+0000 7fd0ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:49.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.349+0000 7fd0ef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:30:49.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.349+0000 7fd0ef7fe640 10 monclient: dump: 2024-04-26T00:30:49.346 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:49.346 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:49.347 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:49.347 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:49.347 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:49.347 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:49.347 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:49.347 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:49.347 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:49.347 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:49.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.349+0000 7fd0f6a82640 10 monclient: shutdown 2024-04-26T00:30:49.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.349+0000 7fd0f6a82640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:30:49.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.349+0000 7fd0f6a82640 1 -- 172.21.15.37:0/4048372930 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0f00a06e0 msgr2=0x7fd0f0134ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:49.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.349+0000 7fd0f6a82640 1 --2- 172.21.15.37:0/4048372930 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0f00a06e0 0x7fd0f0134ba0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd0e40097c0 tx=0x7fd0e401c370 comp rx=0 tx=0).stop 2024-04-26T00:30:49.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f6a82640 1 -- 172.21.15.37:0/4048372930 shutdown_connections 2024-04-26T00:30:49.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f6a82640 1 --2- 172.21.15.37:0/4048372930 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0f00a06e0 0x7fd0f0134ba0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:49.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f6a82640 1 --2- 172.21.15.37:0/4048372930 >> v2:172.21.15.45:3300/0 conn(0x7fd0f00a0300 0x7fd0f009fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:49.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f6a82640 1 -- 172.21.15.37:0/4048372930 >> 172.21.15.37:0/4048372930 conn(0x7fd0f00a1d50 msgr2=0x7fd0f00a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:30:49.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f6a82640 1 -- 172.21.15.37:0/4048372930 shutdown_connections 2024-04-26T00:30:49.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f6a82640 1 -- 172.21.15.37:0/4048372930 wait complete. 2024-04-26T00:30:49.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f6a82640 10 monclient: build_initial_monmap 2024-04-26T00:30:49.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f6a82640 10 monclient: monmap: 2024-04-26T00:30:49.348 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:30:49.348 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:30:49.348 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:30:49.348 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:30:49.348 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:30:49.348 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:30:49.348 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:30:49.348 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:30:49.348 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:30:49.348 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:49.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f6a82640 1 Processor -- start 2024-04-26T00:30:49.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f6a82640 1 -- start start 2024-04-26T00:30:49.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f6a82640 10 monclient: init 2024-04-26T00:30:49.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f6a82640 10 monclient: _reopen_session rank -1 2024-04-26T00:30:49.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f6a82640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:30:49.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f6a82640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0f00a0300 0x7fd0f0131180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:49.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f6a82640 10 monclient(hunting): picked mon.noname-c con 0x7fd0f00a0300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:30:49.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f6a82640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0f00a06e0 0x7fd0f009c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:49.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f6a82640 10 monclient(hunting): picked mon.noname-b con 0x7fd0f00a06e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:30:49.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f6a82640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd0f01350e0 0x7fd0f009cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:49.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f6a82640 10 monclient(hunting): picked mon.noname-a con 0x7fd0f01350e0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:30:49.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f6a82640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:49.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f6a82640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f00a0c40 con 0x7fd0f01350e0 2024-04-26T00:30:49.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f6a82640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:49.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f6a82640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f00a0ac0 con 0x7fd0f00a06e0 2024-04-26T00:30:49.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f6a82640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:49.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f6a82640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd0f013fe10 con 0x7fd0f00a0300 2024-04-26T00:30:49.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f6a82640 10 monclient(hunting): _renew_subs 2024-04-26T00:30:49.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f6a82640 10 monclient(hunting): authenticate will time out at 1630.661657s 2024-04-26T00:30:49.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f481e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0f00a0300 0x7fd0f0131180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:49.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f481e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0f00a0300 0x7fd0f0131180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:48650/0 (socket says 172.21.15.37:48650) 2024-04-26T00:30:49.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f481e640 1 -- 172.21.15.37:0/1775200508 learned_addr learned my addr 172.21.15.37:0/1775200508 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:30:49.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f481e640 10 monclient(hunting): get_auth_request con 0x7fd0f00a0300 auth_method 0 2024-04-26T00:30:49.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f481e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:49.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f481e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:49.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f481e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:49.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0effff640 1 -- 172.21.15.37:0/1775200508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0f00a06e0 msgr2=0x7fd0f009c750 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:30:49.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f501f640 1 --2- 172.21.15.37:0/1775200508 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd0f01350e0 0x7fd0f009cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:49.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0effff640 1 --2- 172.21.15.37:0/1775200508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0f00a06e0 0x7fd0f009c750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:30:49.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f501f640 10 monclient(hunting): get_auth_request con 0x7fd0f01350e0 auth_method 0 2024-04-26T00:30:49.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f501f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:49.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f501f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:49.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f501f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:49.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f481e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:49.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f481e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:49.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f481e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:49.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f501f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:49.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f501f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:49.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f501f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:49.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f481e640 10 monclient(hunting): handle_auth_done global_id 84536 payload 824 2024-04-26T00:30:49.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f481e640 1 -- 172.21.15.37:0/1775200508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0f00a06e0 msgr2=0x7fd0f009c750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:30:49.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f481e640 1 --2- 172.21.15.37:0/1775200508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0f00a06e0 0x7fd0f009c750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:49.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f481e640 1 -- 172.21.15.37:0/1775200508 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd0f01350e0 msgr2=0x7fd0f009cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:49.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f481e640 1 --2- 172.21.15.37:0/1775200508 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd0f01350e0 0x7fd0f009cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:49.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f481e640 10 monclient: _finish_hunting 0 2024-04-26T00:30:49.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f481e640 1 monclient: found mon.noname-c 2024-04-26T00:30:49.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f481e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:30:49.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f481e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:30:49.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.353+0000 7fd0f481e640 1 -- 172.21.15.37:0/1775200508 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0f006e4d0 con 0x7fd0f00a0300 2024-04-26T00:30:49.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0f481e640 1 -- 172.21.15.37:0/1775200508 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0f00a0300 msgr2=0x7fd0f0131180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:49.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0f481e640 1 --2- 172.21.15.37:0/1775200508 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0f00a0300 0x7fd0f0131180 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd0e800b560 tx=0x7fd0e800ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:49.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0ed7fa640 1 -- 172.21.15.37:0/1775200508 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e8013020 con 0x7fd0f00a0300 2024-04-26T00:30:49.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0ed7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:49.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0ed7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:30:49.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0ed7fa640 10 monclient: dump: 2024-04-26T00:30:49.358 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:49.358 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:49.358 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:49.358 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:49.358 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:49.358 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:49.358 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:49.358 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:49.359 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:49.359 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:49.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0ed7fa640 10 monclient: _finish_auth 0 2024-04-26T00:30:49.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0ed7fa640 10 monclient: _check_auth_tickets 2024-04-26T00:30:49.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:30:49.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0ed7fa640 1 -- 172.21.15.37:0/1775200508 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0e8005070 con 0x7fd0f00a0300 2024-04-26T00:30:49.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0ed7fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:30:49.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0ed7fa640 1 -- 172.21.15.37:0/1775200508 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e8005460 con 0x7fd0f00a0300 2024-04-26T00:30:49.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0f6a82640 5 monclient: authenticate success, global_id 84536 2024-04-26T00:30:49.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0ed7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:49.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0ed7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:30:49.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0ed7fa640 10 monclient: dump: 2024-04-26T00:30:49.359 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:49.359 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:49.360 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:49.360 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:49.360 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:49.360 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:49.360 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:49.360 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:49.360 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:49.360 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:49.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0f6a82640 10 monclient: _renew_subs 2024-04-26T00:30:49.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0f6a82640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:30:49.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0f6a82640 1 -- 172.21.15.37:0/1775200508 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0f013a7b0 con 0x7fd0f00a0300 2024-04-26T00:30:49.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0ee7fc640 4 set_mon_vals no callback set 2024-04-26T00:30:49.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0f6a82640 10 monclient: _renew_subs 2024-04-26T00:30:49.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0f6a82640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:30:49.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0f6a82640 1 -- 172.21.15.37:0/1775200508 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0f01316c0 con 0x7fd0f00a0300 2024-04-26T00:30:49.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0f6a82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:30:49.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0f6a82640 1 --2- 172.21.15.37:0/1775200508 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd0f009d1d0 0x7fd0f006fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:49.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0f6a82640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:49.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0f6a82640 1 -- 172.21.15.37:0/1775200508 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f0139f10 con 0x7fd0f009d1d0 2024-04-26T00:30:49.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0effff640 1 --2- 172.21.15.37:0/1775200508 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd0f009d1d0 0x7fd0f006fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:49.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0effff640 10 monclient: get_auth_request con 0x7fd0f009d1d0 auth_method 0 2024-04-26T00:30:49.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:49.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0effff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:49.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0effff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:49.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0ed7fa640 1 -- 172.21.15.37:0/1775200508 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd0e8016020 con 0x7fd0f00a0300 2024-04-26T00:30:49.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:49.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:49.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:49.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0effff640 10 monclient(hunting): handle_auth_done global_id 124176 payload 824 2024-04-26T00:30:49.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0ed7fa640 1 --2- 172.21.15.37:0/1775200508 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd0dc035e60 0x7fd0dc038320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:49.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0effff640 1 -- 172.21.15.37:0/1775200508 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd0f009d5b0 con 0x7fd0f009d1d0 2024-04-26T00:30:49.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0ed7fa640 1 -- 172.21.15.37:0/1775200508 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(405..405 src has 370..405) ==== 3979+0+0 (secure 0 0 0) 0x7fd0e80488c0 con 0x7fd0f00a0300 2024-04-26T00:30:49.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0f501f640 1 --2- 172.21.15.37:0/1775200508 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd0dc035e60 0x7fd0dc038320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:49.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.357+0000 7fd0f501f640 10 monclient: get_auth_request con 0x7fd0dc035e60 auth_method 0 2024-04-26T00:30:49.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.361+0000 7fd0effff640 1 -- 172.21.15.37:0/1775200508 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd0f009d1d0 msgr2=0x7fd0f006fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:49.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.361+0000 7fd0effff640 1 --2- 172.21.15.37:0/1775200508 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd0f009d1d0 0x7fd0f006fdd0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd0e40ad000 tx=0x7fd0e401ac10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:49.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.361+0000 7fd0ed7fa640 1 -- 172.21.15.37:0/1775200508 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e40bb020 con 0x7fd0f009d1d0 2024-04-26T00:30:49.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.361+0000 7fd0ed7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:49.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.361+0000 7fd0ed7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:30:49.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.361+0000 7fd0ed7fa640 10 monclient: dump: 2024-04-26T00:30:49.363 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:49.363 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:49.363 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:49.363 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:49.363 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:49.363 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:49.363 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:49.363 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:49.363 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:49.363 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:49.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.361+0000 7fd0ed7fa640 1 -- 172.21.15.37:0/1775200508 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd0e40ab9e0 con 0x7fd0f009d1d0 2024-04-26T00:30:49.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.361+0000 7fd0ed7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:30:49.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.361+0000 7fd0ed7fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:30:49.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.361+0000 7fd0ed7fa640 1 -- 172.21.15.37:0/1775200508 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd0f009d1d0 msgr2=0x7fd0f006fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:49.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.361+0000 7fd0ed7fa640 1 --2- 172.21.15.37:0/1775200508 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd0f009d1d0 0x7fd0f006fdd0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd0e40ad000 tx=0x7fd0e401ac10 comp rx=0 tx=0).stop 2024-04-26T00:30:49.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.361+0000 7fd0ed7fa640 1 -- 172.21.15.37:0/1775200508 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd0f009d1d0 msgr2=0x7fd0f006fdd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:30:49.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.361+0000 7fd0ed7fa640 1 --2- 172.21.15.37:0/1775200508 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd0f009d1d0 0x7fd0f006fdd0 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd0e40ad000 tx=0x7fd0e401ac10 comp rx=0 tx=0).stop 2024-04-26T00:30:49.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.361+0000 7fd0f501f640 1 --2- 172.21.15.37:0/1775200508 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd0dc035e60 0x7fd0dc038320 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd0e00055b0 tx=0x7fd0e0005540 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:49.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.365+0000 7fd0f6a82640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:30:49.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.365+0000 7fd0f6a82640 1 --2- 172.21.15.37:0/1775200508 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd0f0070310 0x7fd0f0070720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:49.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.365+0000 7fd0f6a82640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:49.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.365+0000 7fd0f6a82640 1 -- 172.21.15.37:0/1775200508 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f013a410 con 0x7fd0f0070310 2024-04-26T00:30:49.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.365+0000 7fd0effff640 1 --2- 172.21.15.37:0/1775200508 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd0f0070310 0x7fd0f0070720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:49.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.365+0000 7fd0effff640 10 monclient: get_auth_request con 0x7fd0f0070310 auth_method 0 2024-04-26T00:30:49.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.365+0000 7fd0effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:49.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.365+0000 7fd0effff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:49.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.365+0000 7fd0effff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:49.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.365+0000 7fd0effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:49.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.365+0000 7fd0effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:49.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.365+0000 7fd0effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:49.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.365+0000 7fd0effff640 10 monclient(hunting): handle_auth_done global_id 124179 payload 824 2024-04-26T00:30:49.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.365+0000 7fd0effff640 1 -- 172.21.15.37:0/1775200508 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd0f009e390 con 0x7fd0f0070310 2024-04-26T00:30:49.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.369+0000 7fd0effff640 1 -- 172.21.15.37:0/1775200508 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd0f0070310 msgr2=0x7fd0f0070720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:49.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.369+0000 7fd0effff640 1 --2- 172.21.15.37:0/1775200508 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd0f0070310 0x7fd0f0070720 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd0e401cc10 tx=0x7fd0e4019170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:49.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.369+0000 7fd0ed7fa640 1 -- 172.21.15.37:0/1775200508 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e40bb070 con 0x7fd0f0070310 2024-04-26T00:30:49.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.369+0000 7fd0ed7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:49.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.369+0000 7fd0ed7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:30:49.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.369+0000 7fd0ed7fa640 10 monclient: dump: 2024-04-26T00:30:49.367 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:49.367 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:49.367 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:49.367 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:49.367 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:49.367 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:49.367 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:49.367 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:49.367 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:49.367 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:49.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.369+0000 7fd0ed7fa640 1 -- 172.21.15.37:0/1775200508 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4535 (secure 0 0 0) 0x7fd0e40c2020 con 0x7fd0f0070310 2024-04-26T00:30:49.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.369+0000 7fd0ed7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:30:49.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.369+0000 7fd0ed7fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:30:49.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.369+0000 7fd0ed7fa640 1 -- 172.21.15.37:0/1775200508 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd0f0070310 msgr2=0x7fd0f0070720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:49.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.369+0000 7fd0ed7fa640 1 --2- 172.21.15.37:0/1775200508 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd0f0070310 0x7fd0f0070720 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd0e401cc10 tx=0x7fd0e4019170 comp rx=0 tx=0).stop 2024-04-26T00:30:49.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.369+0000 7fd0ed7fa640 1 -- 172.21.15.37:0/1775200508 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd0f0070310 msgr2=0x7fd0f0070720 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:30:49.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.369+0000 7fd0ed7fa640 1 --2- 172.21.15.37:0/1775200508 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd0f0070310 0x7fd0f0070720 secure :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd0e401cc10 tx=0x7fd0e4019170 comp rx=0 tx=0).stop 2024-04-26T00:30:49.368 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:30:49.368 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:30:49.368 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:30:49.368 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:30:49.368 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 84, 2024-04-26T00:30:49.369 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 38247, 2024-04-26T00:30:49.369 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:30:49.369 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:30:49.369 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:30:49.369 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:30:49.369 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:30:49.369 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:30:49.369 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:30:49.369 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:30:49.369 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:30:49.369 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:30:49.369 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:30:49.369 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:30:49.369 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:30:49.370 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:30:49.370 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:30:49.370 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:30:49.370 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:30:49.370 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:30:49.370 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:30:49.370 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:30:49.370 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:30:49.370 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:30:49.370 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:30:49.370 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:30:49.370 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:30:49.370 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:30:49.370 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:30:49.371 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:30:49.371 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:30:49.371 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:30:49.371 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:30:49.371 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:30:49.371 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:30:49.371 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:49.371 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:49.371 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:30:49.371 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:30:49.371 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:30:49.371 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:30:49.371 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:30:49.371 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:30:49.372 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:30:49.372 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:30:49.372 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:30:49.372 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:30:49.372 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:30:49.372 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:30:49.372 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:30:49.372 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:30:49.372 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:30:49.372 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:30:49.372 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:30:49.372 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:30:49.372 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:30:49.372 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:30:49.372 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:30:49.373 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:30:49.373 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:30:49.373 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:30:49.373 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:30:49.373 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:30:49.373 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:30:49.373 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:30:49.373 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:30:49.373 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:30:49.373 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:49.373 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:30:49.373 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:49.373 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:30:49.373 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:30:49.373 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:30:49.373 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:30:49.374 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:49.374 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:30:49.374 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:30:49.374 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:49.374 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:49.374 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:49.374 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:30:49.374 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:30:49.374 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:49.374 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:49.374 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:49.374 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:49.374 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:30:49.374 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:30:49.375 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:30:49.375 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:30:49.375 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:30:49.375 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:49.375 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:49.375 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:30:49.375 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:30:49.375 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:30:49.375 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:30:49.375 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:49.375 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:30:49.375 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:30:49.375 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:49.375 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:49.376 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:49.376 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:30:49.376 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:30:49.376 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:49.376 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:49.376 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:49.376 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:49.376 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:30:49.376 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:30:49.376 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:30:49.376 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:30:49.376 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:30:49.377 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:49.377 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:49.377 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:30:49.378 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:30:49.378 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:30:49.378 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:30:49.378 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:49.378 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:30:49.378 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:30:49.378 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:49.378 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:49.378 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:49.378 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:30:49.378 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:30:49.378 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:49.378 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:49.379 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:49.379 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:49.379 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:30:49.379 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:30:49.379 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:30:49.379 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:30:49.379 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:30:49.379 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:49.379 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:49.379 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:49.379 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:30:49.379 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:30:49.379 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:49.379 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:30:49.380 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:30:49.380 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:30:49.380 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:49.380 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:30:49.380 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:30:49.380 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:49.380 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:30:49.380 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:30:49.380 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:30:49.380 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:49.380 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:49.380 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:49.380 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:30:49.381 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:30:49.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.369+0000 7fd0f6a82640 1 -- 172.21.15.37:0/1775200508 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd0dc035e60 msgr2=0x7fd0dc038320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:49.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.369+0000 7fd0f6a82640 1 --2- 172.21.15.37:0/1775200508 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd0dc035e60 0x7fd0dc038320 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd0e00055b0 tx=0x7fd0e0005540 comp rx=0 tx=0).stop 2024-04-26T00:30:49.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.369+0000 7fd0f6a82640 10 monclient: shutdown 2024-04-26T00:30:49.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.369+0000 7fd0f6a82640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:30:49.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.369+0000 7fd0f6a82640 1 -- 172.21.15.37:0/1775200508 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0f00a0300 msgr2=0x7fd0f0131180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:49.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.369+0000 7fd0f6a82640 1 --2- 172.21.15.37:0/1775200508 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0f00a0300 0x7fd0f0131180 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd0e800b560 tx=0x7fd0e800ba20 comp rx=0 tx=0).stop 2024-04-26T00:30:49.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.369+0000 7fd0f501f640 1 -- 172.21.15.37:0/1775200508 reap_dead start 2024-04-26T00:30:49.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.369+0000 7fd0f6a82640 1 -- 172.21.15.37:0/1775200508 shutdown_connections 2024-04-26T00:30:49.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.369+0000 7fd0f6a82640 1 --2- 172.21.15.37:0/1775200508 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0f00a0300 0x7fd0f0131180 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:49.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.369+0000 7fd0f6a82640 1 -- 172.21.15.37:0/1775200508 >> 172.21.15.37:0/1775200508 conn(0x7fd0f00a1d50 msgr2=0x7fd0f00a3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:30:49.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.369+0000 7fd0f6a82640 1 -- 172.21.15.37:0/1775200508 shutdown_connections 2024-04-26T00:30:49.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.369+0000 7fd0f6a82640 1 -- 172.21.15.37:0/1775200508 wait complete. 2024-04-26T00:30:49.383 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:30:49.384 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-04-26T00:30:49.384 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:30:49.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.501+0000 7fc838779640 10 monclient: get_monmap_and_config 2024-04-26T00:30:49.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.501+0000 7fc838779640 10 monclient: build_initial_monmap 2024-04-26T00:30:49.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.501+0000 7fc838779640 10 monclient: monmap: 2024-04-26T00:30:49.496 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:30:49.496 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:49.496 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:30:49.507727+0000 2024-04-26T00:30:49.496 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:30:49.507727+0000 2024-04-26T00:30:49.496 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:30:49.496 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:49.496 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:30:49.496 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:30:49.496 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:30:49.497 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:30:49.497 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:30:49.497 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:49.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.505+0000 7fc838779640 1 Processor -- start 2024-04-26T00:30:49.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.505+0000 7fc838779640 1 -- start start 2024-04-26T00:30:49.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.505+0000 7fc838779640 10 monclient: init 2024-04-26T00:30:49.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.505+0000 7fc838779640 10 monclient: _reopen_session rank -1 2024-04-26T00:30:49.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.505+0000 7fc838779640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-04-26T00:30:49.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.505+0000 7fc838779640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc830136f20 addr v1:172.21.15.37:6789/0 2024-04-26T00:30:49.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.505+0000 7fc838779640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc830137a10 0x7fc83009bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:49.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.505+0000 7fc838779640 10 monclient(hunting): picked mon.noname-c con 0x7fc830137a10 addr v2:172.21.15.45:3300/0 2024-04-26T00:30:49.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.505+0000 7fc838779640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc83013c450 0x7fc83013e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:49.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.505+0000 7fc838779640 10 monclient(hunting): picked mon.noname-a con 0x7fc83013c450 addr v2:172.21.15.37:3300/0 2024-04-26T00:30:49.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.505+0000 7fc838779640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8300a50f0 con 0x7fc830136f20 2024-04-26T00:30:49.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.505+0000 7fc838779640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:49.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.505+0000 7fc838779640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fc83009b330 con 0x7fc83013c450 2024-04-26T00:30:49.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.505+0000 7fc838779640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:49.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.505+0000 7fc838779640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc83009b610 con 0x7fc830137a10 2024-04-26T00:30:49.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.505+0000 7fc838779640 10 monclient(hunting): _renew_subs 2024-04-26T00:30:49.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.505+0000 7fc838779640 10 monclient(hunting): authenticate will time out at 1630.813352s 2024-04-26T00:30:49.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.505+0000 7fc836d16640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc83013c450 0x7fc83013e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:49.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.505+0000 7fc836515640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fc830136f20 0x7fc830137300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:59580/0 (socket says 172.21.15.37:59580) 2024-04-26T00:30:49.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.505+0000 7fc836515640 1 -- 172.21.15.37:0/3757754825 learned_addr learned my addr 172.21.15.37:0/3757754825 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:30:49.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.505+0000 7fc835d14640 1 -- 172.21.15.37:0/3757754825 >> v2:172.21.15.45:3300/0 conn(0x7fc830137a10 msgr2=0x7fc83009bff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:30:49.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.505+0000 7fc836d16640 10 monclient(hunting): get_auth_request con 0x7fc83013c450 auth_method 0 2024-04-26T00:30:49.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.505+0000 7fc836d16640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:49.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.505+0000 7fc836d16640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:49.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.505+0000 7fc836d16640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:49.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.505+0000 7fc835d14640 1 --2- 172.21.15.37:0/3757754825 >> v2:172.21.15.45:3300/0 conn(0x7fc830137a10 0x7fc83009bff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:30:49.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.505+0000 7fc836d16640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:49.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.505+0000 7fc836d16640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:49.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.505+0000 7fc836d16640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:49.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.505+0000 7fc836d16640 10 monclient(hunting): handle_auth_done global_id 124182 payload 274 2024-04-26T00:30:49.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.505+0000 7fc836d16640 1 -- 172.21.15.37:0/3757754825 >> v2:172.21.15.45:3300/0 conn(0x7fc830137a10 msgr2=0x7fc83009bff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:30:49.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.505+0000 7fc836d16640 1 --2- 172.21.15.37:0/3757754825 >> v2:172.21.15.45:3300/0 conn(0x7fc830137a10 0x7fc83009bff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:49.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.505+0000 7fc836d16640 1 -- 172.21.15.37:0/3757754825 >> v1:172.21.15.37:6789/0 conn(0x7fc830136f20 legacy=0x7fc830137300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:30:49.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.505+0000 7fc836d16640 10 monclient: _finish_hunting 0 2024-04-26T00:30:49.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.505+0000 7fc836d16640 1 monclient: found mon.noname-a 2024-04-26T00:30:49.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.505+0000 7fc836d16640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:30:49.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.505+0000 7fc836d16640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:30:49.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.505+0000 7fc836d16640 1 -- 172.21.15.37:0/3757754825 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc83009b8f0 con 0x7fc83013c450 2024-04-26T00:30:49.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.509+0000 7fc836d16640 1 -- 172.21.15.37:0/3757754825 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc83013c450 msgr2=0x7fc83013e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:49.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.509+0000 7fc836d16640 1 --2- 172.21.15.37:0/3757754825 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc83013c450 0x7fc83013e840 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc830137ec0 tx=0x7fc82801c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d9c788220e80a70 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:49.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.509+0000 7fc835513640 1 -- 172.21.15.37:0/3757754825 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8280b2070 con 0x7fc83013c450 2024-04-26T00:30:49.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.509+0000 7fc835513640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:49.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.509+0000 7fc835513640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:30:49.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.509+0000 7fc835513640 10 monclient: dump: 2024-04-26T00:30:49.506 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:49.506 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:49.506 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:49.506 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:49.506 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:49.506 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:49.506 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:49.506 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:49.506 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:49.506 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:49.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.509+0000 7fc835513640 10 monclient: _finish_auth 0 2024-04-26T00:30:49.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.509+0000 7fc835513640 10 monclient: _check_auth_tickets 2024-04-26T00:30:49.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.509+0000 7fc835513640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:30:49.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.509+0000 7fc835513640 1 -- 172.21.15.37:0/3757754825 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8280a5a20 con 0x7fc83013c450 2024-04-26T00:30:49.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.509+0000 7fc835513640 10 monclient: handle_config config(0 keys) 2024-04-26T00:30:49.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.509+0000 7fc835513640 1 -- 172.21.15.37:0/3757754825 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8280a5df0 con 0x7fc83013c450 2024-04-26T00:30:49.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.509+0000 7fc835513640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:49.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.509+0000 7fc835513640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:30:49.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.509+0000 7fc835513640 10 monclient: dump: 2024-04-26T00:30:49.507 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:49.507 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:49.507 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:49.507 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:49.507 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:49.507 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:49.507 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:49.507 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:49.507 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:49.508 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:49.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.509+0000 7fc838779640 5 monclient: authenticate success, global_id 124182 2024-04-26T00:30:49.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.509+0000 7fc838779640 10 monclient: get_monmap_and_config success 2024-04-26T00:30:49.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.509+0000 7fc838779640 4 set_mon_vals no callback set 2024-04-26T00:30:49.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.509+0000 7fc838779640 10 monclient: shutdown 2024-04-26T00:30:49.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.509+0000 7fc838779640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:30:49.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.509+0000 7fc838779640 1 -- 172.21.15.37:0/3757754825 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc83013c450 msgr2=0x7fc83013e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:49.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.509+0000 7fc838779640 1 --2- 172.21.15.37:0/3757754825 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc83013c450 0x7fc83013e840 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc830137ec0 tx=0x7fc82801c860 comp rx=0 tx=0).stop 2024-04-26T00:30:49.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.509+0000 7fc838779640 1 -- 172.21.15.37:0/3757754825 shutdown_connections 2024-04-26T00:30:49.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.509+0000 7fc838779640 1 --2- 172.21.15.37:0/3757754825 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc83013c450 0x7fc83013e840 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:49.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.509+0000 7fc838779640 1 --2- 172.21.15.37:0/3757754825 >> v2:172.21.15.45:3300/0 conn(0x7fc830137a10 0x7fc83009bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:49.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.509+0000 7fc838779640 1 -- 172.21.15.37:0/3757754825 >> 172.21.15.37:0/3757754825 conn(0x7fc8300a61b0 msgr2=0x7fc8300a65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:30:49.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.509+0000 7fc838779640 1 -- 172.21.15.37:0/3757754825 shutdown_connections 2024-04-26T00:30:49.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.509+0000 7fc838779640 1 -- 172.21.15.37:0/3757754825 wait complete. 2024-04-26T00:30:49.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.509+0000 7fc838779640 10 monclient: build_initial_monmap 2024-04-26T00:30:49.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.509+0000 7fc838779640 10 monclient: monmap: 2024-04-26T00:30:49.510 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:30:49.510 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:30:49.510 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:30:49.510 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:30:49.510 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:30:49.510 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:30:49.511 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:30:49.511 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:30:49.511 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:30:49.511 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:49.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.509+0000 7fc838779640 1 Processor -- start 2024-04-26T00:30:49.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc838779640 1 -- start start 2024-04-26T00:30:49.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc838779640 10 monclient: init 2024-04-26T00:30:49.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc838779640 10 monclient: _reopen_session rank -1 2024-04-26T00:30:49.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc838779640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:30:49.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc838779640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc830136f20 0x7fc8301ced60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:49.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc838779640 10 monclient(hunting): picked mon.noname-a con 0x7fc830136f20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:30:49.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc838779640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc830137a10 0x7fc8301cf530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:49.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc838779640 10 monclient(hunting): picked mon.noname-b con 0x7fc830137a10 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:30:49.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc838779640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc83013c450 0x7fc830137f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:49.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc838779640 10 monclient(hunting): picked mon.noname-c con 0x7fc83013c450 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:30:49.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc838779640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:49.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc838779640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fc830141930 con 0x7fc830136f20 2024-04-26T00:30:49.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc838779640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:49.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc838779640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc8301417b0 con 0x7fc830137a10 2024-04-26T00:30:49.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc838779640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:49.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc838779640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc830141ab0 con 0x7fc83013c450 2024-04-26T00:30:49.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc838779640 10 monclient(hunting): _renew_subs 2024-04-26T00:30:49.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc838779640 10 monclient(hunting): authenticate will time out at 1630.820524s 2024-04-26T00:30:49.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc836515640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc830136f20 0x7fc8301ced60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:49.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc836d16640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc83013c450 0x7fc830137f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:49.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc836515640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc830136f20 0x7fc8301ced60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:41154/0 (socket says 172.21.15.37:41154) 2024-04-26T00:30:49.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc836515640 1 -- 172.21.15.37:0/3785702412 learned_addr learned my addr 172.21.15.37:0/3785702412 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:30:49.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc836515640 10 monclient(hunting): get_auth_request con 0x7fc830136f20 auth_method 0 2024-04-26T00:30:49.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc835d14640 1 -- 172.21.15.37:0/3785702412 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc830137a10 msgr2=0x7fc8301cf530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:30:49.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc836515640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:49.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc836515640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:49.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc836515640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:49.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc836d16640 10 monclient(hunting): get_auth_request con 0x7fc83013c450 auth_method 0 2024-04-26T00:30:49.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc836d16640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:49.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc836d16640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:49.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc836d16640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:49.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc835d14640 1 --2- 172.21.15.37:0/3785702412 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc830137a10 0x7fc8301cf530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:30:49.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc836515640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:49.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc836515640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:49.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc836515640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:49.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc836d16640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:49.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc836d16640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:49.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc836d16640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:49.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc836515640 10 monclient(hunting): handle_auth_done global_id 124185 payload 824 2024-04-26T00:30:49.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc836515640 1 -- 172.21.15.37:0/3785702412 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc83013c450 msgr2=0x7fc830137f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:49.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc836515640 1 --2- 172.21.15.37:0/3785702412 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc83013c450 0x7fc830137f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:49.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc836515640 1 -- 172.21.15.37:0/3785702412 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc830137a10 msgr2=0x7fc8301cf530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:30:49.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc836515640 1 --2- 172.21.15.37:0/3785702412 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc830137a10 0x7fc8301cf530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:30:49.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc836515640 10 monclient: _finish_hunting 0 2024-04-26T00:30:49.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc836515640 1 monclient: found mon.noname-a 2024-04-26T00:30:49.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc836515640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:30:49.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc836515640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:30:49.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc836515640 1 -- 172.21.15.37:0/3785702412 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc830135f30 con 0x7fc830136f20 2024-04-26T00:30:49.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc836d16640 1 --2- 172.21.15.37:0/3785702412 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc83013c450 0x7fc830137f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:30:49.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc836515640 1 -- 172.21.15.37:0/3785702412 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc830136f20 msgr2=0x7fc8301ced60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:49.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc836515640 1 --2- 172.21.15.37:0/3785702412 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc830136f20 0x7fc8301ced60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc82c00c470 tx=0x7fc82c00c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:49.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc826ffd640 1 -- 172.21.15.37:0/3785702412 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc82c007070 con 0x7fc830136f20 2024-04-26T00:30:49.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc826ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:49.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc826ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:30:49.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc826ffd640 10 monclient: dump: 2024-04-26T00:30:49.519 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:49.519 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:49.519 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:49.519 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:49.519 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:49.519 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:49.519 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:49.519 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:49.520 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:49.520 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:49.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc826ffd640 10 monclient: _finish_auth 0 2024-04-26T00:30:49.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc826ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:30:49.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc826ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:30:49.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc826ffd640 1 -- 172.21.15.37:0/3785702412 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc82c007210 con 0x7fc830136f20 2024-04-26T00:30:49.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc826ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:30:49.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc826ffd640 1 -- 172.21.15.37:0/3785702412 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc82c007600 con 0x7fc830136f20 2024-04-26T00:30:49.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc826ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:49.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc826ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:30:49.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc826ffd640 10 monclient: dump: 2024-04-26T00:30:49.521 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:49.521 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:49.521 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:49.521 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:49.521 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:49.521 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:49.521 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:49.521 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:49.521 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:49.521 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:49.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc838779640 5 monclient: authenticate success, global_id 124185 2024-04-26T00:30:49.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc838779640 10 monclient: _renew_subs 2024-04-26T00:30:49.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc834d12640 4 set_mon_vals no callback set 2024-04-26T00:30:49.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc838779640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:30:49.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc838779640 1 -- 172.21.15.37:0/3785702412 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc83013bc90 con 0x7fc830136f20 2024-04-26T00:30:49.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc838779640 10 monclient: _renew_subs 2024-04-26T00:30:49.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc838779640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:30:49.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.513+0000 7fc838779640 1 -- 172.21.15.37:0/3785702412 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8301354e0 con 0x7fc830136f20 2024-04-26T00:30:49.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.517+0000 7fc838779640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:30:49.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.517+0000 7fc838779640 1 --2- 172.21.15.37:0/3785702412 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8301cfa70 0x7fc830138c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:49.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.517+0000 7fc838779640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:49.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.517+0000 7fc838779640 1 -- 172.21.15.37:0/3785702412 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc83013a330 con 0x7fc8301cfa70 2024-04-26T00:30:49.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.517+0000 7fc835d14640 1 --2- 172.21.15.37:0/3785702412 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8301cfa70 0x7fc830138c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:49.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.517+0000 7fc835d14640 10 monclient: get_auth_request con 0x7fc8301cfa70 auth_method 0 2024-04-26T00:30:49.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.517+0000 7fc835d14640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:49.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.517+0000 7fc835d14640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:49.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.517+0000 7fc835d14640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:49.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.517+0000 7fc826ffd640 1 -- 172.21.15.37:0/3785702412 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc82c01b400 con 0x7fc830136f20 2024-04-26T00:30:49.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.517+0000 7fc835d14640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:49.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.517+0000 7fc835d14640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:49.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.517+0000 7fc835d14640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:49.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.517+0000 7fc826ffd640 1 --2- 172.21.15.37:0/3785702412 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc814035e30 0x7fc8140382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:49.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.517+0000 7fc835d14640 10 monclient(hunting): handle_auth_done global_id 84542 payload 824 2024-04-26T00:30:49.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.517+0000 7fc826ffd640 1 -- 172.21.15.37:0/3785702412 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(405..405 src has 370..405) ==== 3979+0+0 (secure 0 0 0) 0x7fc82c048820 con 0x7fc830136f20 2024-04-26T00:30:49.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.517+0000 7fc835d14640 1 -- 172.21.15.37:0/3785702412 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc83004f510 con 0x7fc8301cfa70 2024-04-26T00:30:49.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.517+0000 7fc836d16640 1 --2- 172.21.15.37:0/3785702412 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc814035e30 0x7fc8140382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:49.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.517+0000 7fc836d16640 10 monclient: get_auth_request con 0x7fc814035e30 auth_method 0 2024-04-26T00:30:49.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.517+0000 7fc835d14640 1 -- 172.21.15.37:0/3785702412 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8301cfa70 msgr2=0x7fc830138c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:49.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.517+0000 7fc835d14640 1 --2- 172.21.15.37:0/3785702412 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8301cfa70 0x7fc830138c30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc8301395a0 tx=0x7fc82000b8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:49.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.517+0000 7fc826ffd640 1 -- 172.21.15.37:0/3785702412 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc820005270 con 0x7fc8301cfa70 2024-04-26T00:30:49.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.517+0000 7fc826ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:49.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.517+0000 7fc826ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:30:49.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.517+0000 7fc826ffd640 10 monclient: dump: 2024-04-26T00:30:49.524 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:49.524 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:49.524 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:49.524 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:49.524 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:49.525 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:49.525 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:49.525 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:49.525 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:49.525 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:49.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.517+0000 7fc826ffd640 1 -- 172.21.15.37:0/3785702412 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc820015070 con 0x7fc8301cfa70 2024-04-26T00:30:49.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.517+0000 7fc826ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:30:49.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.517+0000 7fc826ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:30:49.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.517+0000 7fc826ffd640 1 -- 172.21.15.37:0/3785702412 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8301cfa70 msgr2=0x7fc830138c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:49.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.517+0000 7fc826ffd640 1 --2- 172.21.15.37:0/3785702412 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8301cfa70 0x7fc830138c30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc8301395a0 tx=0x7fc82000b8e0 comp rx=0 tx=0).stop 2024-04-26T00:30:49.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.517+0000 7fc826ffd640 1 -- 172.21.15.37:0/3785702412 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8301cfa70 msgr2=0x7fc830138c30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:30:49.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.517+0000 7fc826ffd640 1 --2- 172.21.15.37:0/3785702412 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8301cfa70 0x7fc830138c30 secure :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc8301395a0 tx=0x7fc82000b8e0 comp rx=0 tx=0).stop 2024-04-26T00:30:49.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.521+0000 7fc836d16640 1 --2- 172.21.15.37:0/3785702412 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc814035e30 0x7fc8140382f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fc830068940 tx=0x7fc8280a7070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:49.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.525+0000 7fc838779640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:30:49.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.525+0000 7fc838779640 1 --2- 172.21.15.37:0/3785702412 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc83002c000 0x7fc83002c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:30:49.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.525+0000 7fc838779640 10 monclient(hunting): start opening mon connection 2024-04-26T00:30:49.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.525+0000 7fc838779640 1 -- 172.21.15.37:0/3785702412 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc83009ae30 con 0x7fc83002c000 2024-04-26T00:30:49.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.525+0000 7fc835d14640 1 --2- 172.21.15.37:0/3785702412 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc83002c000 0x7fc83002c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:30:49.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.525+0000 7fc835d14640 10 monclient: get_auth_request con 0x7fc83002c000 auth_method 0 2024-04-26T00:30:49.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.525+0000 7fc835d14640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:30:49.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.525+0000 7fc835d14640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:30:49.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.525+0000 7fc835d14640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:30:49.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.525+0000 7fc835d14640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:30:49.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.525+0000 7fc835d14640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:30:49.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.525+0000 7fc835d14640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:30:49.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.525+0000 7fc835d14640 10 monclient(hunting): handle_auth_done global_id 84545 payload 824 2024-04-26T00:30:49.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.525+0000 7fc835d14640 1 -- 172.21.15.37:0/3785702412 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc83009fa40 con 0x7fc83002c000 2024-04-26T00:30:49.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.529+0000 7fc835d14640 1 -- 172.21.15.37:0/3785702412 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc83002c000 msgr2=0x7fc83002c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:30:49.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.529+0000 7fc835d14640 1 --2- 172.21.15.37:0/3785702412 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc83002c000 0x7fc83002c3e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc8301395a0 tx=0x7fc820005a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:30:49.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.529+0000 7fc826ffd640 1 -- 172.21.15.37:0/3785702412 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc82001ae60 con 0x7fc83002c000 2024-04-26T00:30:49.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.529+0000 7fc826ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:30:49.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.529+0000 7fc826ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:30:49.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.529+0000 7fc826ffd640 10 monclient: dump: 2024-04-26T00:30:49.528 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:30:49.528 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:30:49.528 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:49.528 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:30:49.528 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:30:49.528 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:30:49.528 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:30:49.529 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:30:49.529 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:30:49.529 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:30:49.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.529+0000 7fc826ffd640 1 -- 172.21.15.37:0/3785702412 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4878 (secure 0 0 0) 0x7fc820010400 con 0x7fc83002c000 2024-04-26T00:30:49.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.529+0000 7fc826ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:30:49.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.529+0000 7fc826ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:30:49.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.529+0000 7fc826ffd640 1 -- 172.21.15.37:0/3785702412 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc83002c000 msgr2=0x7fc83002c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:49.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.529+0000 7fc826ffd640 1 --2- 172.21.15.37:0/3785702412 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc83002c000 0x7fc83002c3e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc8301395a0 tx=0x7fc820005a30 comp rx=0 tx=0).stop 2024-04-26T00:30:49.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.529+0000 7fc826ffd640 1 -- 172.21.15.37:0/3785702412 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc83002c000 msgr2=0x7fc83002c3e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:30:49.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.529+0000 7fc826ffd640 1 --2- 172.21.15.37:0/3785702412 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc83002c000 0x7fc83002c3e0 secure :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc8301395a0 tx=0x7fc820005a30 comp rx=0 tx=0).stop 2024-04-26T00:30:49.529 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:30:49.530 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:30:49.530 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:30:49.530 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:30:49.530 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 84, 2024-04-26T00:30:49.530 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 254464, 2024-04-26T00:30:49.530 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:30:49.530 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:30:49.530 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:30:49.530 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:30:49.530 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:30:49.530 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:30:49.531 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:30:49.531 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:30:49.531 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:30:49.531 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:30:49.531 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:30:49.531 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:30:49.531 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:30:49.531 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:30:49.531 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:30:49.531 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:30:49.531 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:30:49.531 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:30:49.531 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:30:49.531 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:30:49.531 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:30:49.532 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:30:49.532 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:30:49.532 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:30:49.532 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:30:49.532 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:30:49.532 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:30:49.532 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:30:49.532 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:30:49.532 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:30:49.532 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:30:49.532 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:30:49.532 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:30:49.532 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:49.532 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:49.533 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:30:49.533 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:30:49.533 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:30:49.533 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:30:49.533 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:30:49.533 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:30:49.533 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:30:49.533 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:30:49.533 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:30:49.533 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:30:49.533 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:30:49.533 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:30:49.533 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:30:49.533 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:30:49.534 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:30:49.534 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:30:49.534 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:30:49.534 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:30:49.534 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:30:49.534 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:30:49.534 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:30:49.534 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:30:49.534 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:30:49.534 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:30:49.534 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:30:49.534 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:30:49.534 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:30:49.534 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:30:49.535 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:30:49.535 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:30:49.535 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:49.535 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:30:49.535 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:49.535 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:30:49.535 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:30:49.535 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:30:49.535 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:30:49.535 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:49.535 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:30:49.535 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:30:49.535 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:49.535 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:49.535 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:49.536 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:30:49.536 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:30:49.536 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:49.536 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:49.536 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:49.536 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:49.536 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:30:49.536 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:30:49.536 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:30:49.536 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:30:49.536 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:30:49.536 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:49.536 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:49.536 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:30:49.537 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:30:49.537 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:30:49.537 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:30:49.537 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:49.537 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:30:49.537 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:30:49.537 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:49.537 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:49.537 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:49.537 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:30:49.537 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:30:49.537 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:49.537 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:49.537 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:49.537 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:49.538 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:30:49.538 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:30:49.538 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:30:49.538 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:30:49.538 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:30:49.538 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:49.538 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:49.538 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:30:49.538 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:30:49.538 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:30:49.538 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:30:49.538 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:49.538 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:30:49.538 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:30:49.539 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:49.539 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:49.539 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:49.539 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:30:49.539 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:30:49.539 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:30:49.539 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:49.539 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:49.539 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:49.539 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:30:49.539 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:30:49.539 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:30:49.539 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:30:49.539 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:30:49.540 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:49.540 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:49.540 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:49.540 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:30:49.540 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:30:49.540 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:49.540 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:30:49.540 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:30:49.540 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:30:49.540 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:49.540 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:30:49.540 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:30:49.540 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:49.541 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:30:49.541 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:30:49.541 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:30:49.541 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:49.541 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:30:49.541 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:30:49.541 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:49.541 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:30:49.541 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:30:49.541 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:30:49.541 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:49.541 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:30:49.542 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:30:49.542 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:30:49.542 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:30:49.542 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:30:49.542 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:30:49.542 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:30:49.542 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:30:49.542 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:30:49.542 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:30:49.542 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:30:49.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.529+0000 7fc838779640 1 -- 172.21.15.37:0/3785702412 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc814035e30 msgr2=0x7fc8140382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:49.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.529+0000 7fc838779640 1 --2- 172.21.15.37:0/3785702412 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc814035e30 0x7fc8140382f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fc830068940 tx=0x7fc8280a7070 comp rx=0 tx=0).stop 2024-04-26T00:30:49.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.529+0000 7fc838779640 10 monclient: shutdown 2024-04-26T00:30:49.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.529+0000 7fc838779640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:30:49.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.529+0000 7fc838779640 1 -- 172.21.15.37:0/3785702412 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc830136f20 msgr2=0x7fc8301ced60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:30:49.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.529+0000 7fc838779640 1 --2- 172.21.15.37:0/3785702412 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc830136f20 0x7fc8301ced60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc82c00c470 tx=0x7fc82c00c930 comp rx=0 tx=0).stop 2024-04-26T00:30:49.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.529+0000 7fc836d16640 1 -- 172.21.15.37:0/3785702412 reap_dead start 2024-04-26T00:30:49.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.529+0000 7fc836d16640 1 -- 172.21.15.37:0/3785702412 reap_dead start 2024-04-26T00:30:49.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.529+0000 7fc838779640 1 -- 172.21.15.37:0/3785702412 shutdown_connections 2024-04-26T00:30:49.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.529+0000 7fc838779640 1 -- 172.21.15.37:0/3785702412 >> 172.21.15.37:0/3785702412 conn(0x7fc8300a61b0 msgr2=0x7fc8300a80d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:30:49.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.529+0000 7fc838779640 1 -- 172.21.15.37:0/3785702412 shutdown_connections 2024-04-26T00:30:49.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:30:49.529+0000 7fc838779640 1 -- 172.21.15.37:0/3785702412 wait complete. 2024-04-26T00:30:49.545 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:30:49.545 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-04-26T00:30:49.575 INFO:tasks.rados.rados.0.smithi045.stdout:1783: done (1 left) 2024-04-26T00:30:49.575 INFO:tasks.rados.rados.0.smithi045.stdout:1784: done (0 left) 2024-04-26T00:30:49.575 INFO:tasks.rados.rados.0.smithi045.stdout:1785: read oid 45 snap -1 2024-04-26T00:30:49.576 INFO:tasks.rados.rados.0.smithi045.stdout:1785: expect (ObjNum 635 snap 198 seq_num 635) 2024-04-26T00:30:49.576 INFO:tasks.rados.rados.0.smithi045.stdout:1786: write oid 47 current snap is 203 2024-04-26T00:30:49.576 INFO:tasks.rados.rados.0.smithi045.stdout:1786: seq_num 656 ranges {625392=569702,1637717=745548,2848542=176850} 2024-04-26T00:30:49.591 INFO:tasks.rados.rados.0.smithi045.stdout:1786: writing smithi04526579-47 from 625392 to 1195094 tid 1 2024-04-26T00:30:49.606 INFO:tasks.rados.rados.0.smithi045.stdout:1786: writing smithi04526579-47 from 1637717 to 2383265 tid 2 2024-04-26T00:30:49.615 INFO:tasks.rados.rados.0.smithi045.stdout:1786: writing smithi04526579-47 from 2848542 to 3025392 tid 3 2024-04-26T00:30:49.615 INFO:tasks.rados.rados.0.smithi045.stdout:1787: snap_create 2024-04-26T00:30:49.615 INFO:tasks.rados.rados.0.smithi045.stdout:1785: expect (ObjNum 635 snap 198 seq_num 635) 2024-04-26T00:30:49.798 INFO:tasks.rados.rados.0.smithi045.stdout:1786: finishing write tid 1 to smithi04526579-47 2024-04-26T00:30:49.799 INFO:tasks.rados.rados.0.smithi045.stdout:1786: finishing write tid 2 to smithi04526579-47 2024-04-26T00:30:49.799 INFO:tasks.rados.rados.0.smithi045.stdout:1786: finishing write tid 3 to smithi04526579-47 2024-04-26T00:30:49.799 INFO:tasks.rados.rados.0.smithi045.stdout:1786: finishing write tid 4 to smithi04526579-47 2024-04-26T00:30:49.799 INFO:tasks.rados.rados.0.smithi045.stdout:1786: finishing write tid 5 to smithi04526579-47 2024-04-26T00:30:49.799 INFO:tasks.rados.rados.0.smithi045.stdout:1786: oid 47 updating version 0 to 978 2024-04-26T00:30:49.799 INFO:tasks.rados.rados.0.smithi045.stdout:1786: oid 47 updating version 978 to 980 2024-04-26T00:30:49.799 INFO:tasks.rados.rados.0.smithi045.stdout:1786: oid 47 updating version 980 to 981 2024-04-26T00:30:49.799 INFO:tasks.rados.rados.0.smithi045.stdout:1786: oid 47 version 981 is already newer than 979 2024-04-26T00:30:49.799 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 981 (ObjNum 656 snap 203 seq_num 656) dirty exists 2024-04-26T00:30:49.799 INFO:tasks.rados.rados.0.smithi045.stdout:1786: left oid 47 (ObjNum 656 snap 203 seq_num 656) 2024-04-26T00:30:50.115 INFO:tasks.rados.rados.0.smithi045.stdout:1785: done (2 left) 2024-04-26T00:30:50.115 INFO:tasks.rados.rados.0.smithi045.stdout:1786: done (1 left) 2024-04-26T00:30:50.115 INFO:tasks.rados.rados.0.smithi045.stdout:1787: done (0 left) 2024-04-26T00:30:50.116 INFO:tasks.rados.rados.0.smithi045.stdout:1788: write (excl) oid 49 current snap is 204 2024-04-26T00:30:50.116 INFO:tasks.rados.rados.0.smithi045.stdout:1788: seq_num 657 ranges {642199=644951,1696199=605559,2992318=49881} 2024-04-26T00:30:50.132 INFO:tasks.rados.rados.0.smithi045.stdout:1788: writing smithi04526579-49 from 642199 to 1287150 tid 1 2024-04-26T00:30:50.146 INFO:tasks.rados.rados.0.smithi045.stdout:1788: writing smithi04526579-49 from 1696199 to 2301758 tid 2 2024-04-26T00:30:50.157 INFO:tasks.rados.rados.0.smithi045.stdout:1788: writing smithi04526579-49 from 2992318 to 3042199 tid 3 2024-04-26T00:30:50.157 INFO:tasks.rados.rados.0.smithi045.stdout:1789: delete oid 3 current snap is 204 2024-04-26T00:30:50.157 INFO:tasks.rados.rados.0.smithi045.stdout:1788: finishing write tid 1 to smithi04526579-49 2024-04-26T00:30:50.157 INFO:tasks.rados.rados.0.smithi045.stdout:1788: finishing write tid 2 to smithi04526579-49 2024-04-26T00:30:50.160 INFO:tasks.rados.rados.0.smithi045.stdout:1788: finishing write tid 3 to smithi04526579-49 2024-04-26T00:30:50.162 INFO:tasks.rados.rados.0.smithi045.stdout:1788: finishing write tid 4 to smithi04526579-49 2024-04-26T00:30:50.162 INFO:tasks.rados.rados.0.smithi045.stdout:1788: finishing write tid 5 to smithi04526579-49 2024-04-26T00:30:50.162 INFO:tasks.rados.rados.0.smithi045.stdout:1788: oid 49 updating version 0 to 985 2024-04-26T00:30:50.162 INFO:tasks.rados.rados.0.smithi045.stdout:1788: oid 49 version 985 is already newer than 984 2024-04-26T00:30:50.163 INFO:tasks.rados.rados.0.smithi045.stdout:1788: oid 49 version 985 is already newer than 983 2024-04-26T00:30:50.163 INFO:tasks.rados.rados.0.smithi045.stdout:1788: oid 49 updating version 985 to 986 2024-04-26T00:30:50.163 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 986 (ObjNum 657 snap 204 seq_num 657) dirty exists 2024-04-26T00:30:50.163 INFO:tasks.rados.rados.0.smithi045.stdout:1788: left oid 49 (ObjNum 657 snap 204 seq_num 657) 2024-04-26T00:30:50.163 INFO:tasks.rados.rados.0.smithi045.stdout:1788: done (1 left) 2024-04-26T00:30:50.163 INFO:tasks.rados.rados.0.smithi045.stdout:1789: done (0 left) 2024-04-26T00:30:50.163 INFO:tasks.rados.rados.0.smithi045.stdout:1790: rollback oid 7 current snap is 204 2024-04-26T00:30:50.163 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 7 to 199 2024-04-26T00:30:50.163 INFO:tasks.rados.rados.0.smithi045.stdout:1791: copy_from oid 39 from oid 47 current snap is 204 2024-04-26T00:30:50.163 INFO:tasks.rados.rados.0.smithi045.stdout:1792: read oid 41 snap -1 2024-04-26T00:30:50.163 INFO:tasks.rados.rados.0.smithi045.stdout:1792: expect deleted 2024-04-26T00:30:50.163 INFO:tasks.rados.rados.0.smithi045.stdout:1793: rollback oid 2 current snap is 204 2024-04-26T00:30:50.163 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 2 to 169 2024-04-26T00:30:50.163 INFO:tasks.rados.rados.0.smithi045.stdout:1794: snap_remove snap 201 2024-04-26T00:30:51.125 INFO:tasks.rados.rados.0.smithi045.stdout:1794: done (4 left) 2024-04-26T00:30:51.125 INFO:tasks.rados.rados.0.smithi045.stdout:1795: snap_create 2024-04-26T00:30:51.125 INFO:tasks.rados.rados.0.smithi045.stdout:1790: finishing rollback tid 1 to smithi04526579-7 2024-04-26T00:30:51.125 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 1382 (ObjNum -2052811888 snap 21888 seq_num 304) dirty dne 2024-04-26T00:30:51.125 INFO:tasks.rados.rados.0.smithi045.stdout:1793: finishing rollback tid 0 to smithi04526579-2 2024-04-26T00:30:51.125 INFO:tasks.rados.rados.0.smithi045.stdout:1793: finishing rollback tid 1 to smithi04526579-2 2024-04-26T00:30:51.125 INFO:tasks.rados.rados.0.smithi045.stdout:1793: finishing rollback tid 2 to smithi04526579-2 2024-04-26T00:30:51.125 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 990 (ObjNum 544 snap 168 seq_num 544) dirty exists 2024-04-26T00:30:51.125 INFO:tasks.rados.rados.0.smithi045.stdout:1791: finishing copy_from racing read to smithi04526579-39 2024-04-26T00:30:51.125 INFO:tasks.rados.rados.0.smithi045.stdout:1791: finishing copy_from to smithi04526579-39 2024-04-26T00:30:51.126 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 981 (ObjNum 656 snap 203 seq_num 656) dirty exists 2024-04-26T00:30:52.133 INFO:tasks.rados.rados.0.smithi045.stdout:1790: done (4 left) 2024-04-26T00:30:52.133 INFO:tasks.rados.rados.0.smithi045.stdout:1791: done (3 left) 2024-04-26T00:30:52.133 INFO:tasks.rados.rados.0.smithi045.stdout:1792: done (2 left) 2024-04-26T00:30:52.133 INFO:tasks.rados.rados.0.smithi045.stdout:1793: done (1 left) 2024-04-26T00:30:52.133 INFO:tasks.rados.rados.0.smithi045.stdout:1795: done (0 left) 2024-04-26T00:30:52.133 INFO:tasks.rados.rados.0.smithi045.stdout:1796: write (excl) oid 24 current snap is 205 2024-04-26T00:30:52.133 INFO:tasks.rados.rados.0.smithi045.stdout:1796: seq_num 659 ranges {675813=795449,2096810=609228} 2024-04-26T00:30:52.156 INFO:tasks.rados.rados.0.smithi045.stdout:1796: writing smithi04526579-24 from 675813 to 1471262 tid 1 2024-04-26T00:30:52.177 INFO:tasks.rados.rados.0.smithi045.stdout:1796: writing smithi04526579-24 from 2096810 to 2706038 tid 2 2024-04-26T00:30:52.177 INFO:tasks.rados.rados.0.smithi045.stdout:1797: copy_from oid 16 from oid 1 current snap is 205 2024-04-26T00:30:52.177 INFO:tasks.rados.rados.0.smithi045.stdout:1796: finishing write tid 1 to smithi04526579-24 2024-04-26T00:30:52.177 INFO:tasks.rados.rados.0.smithi045.stdout:1798: read oid 11 snap -1 2024-04-26T00:30:52.177 INFO:tasks.rados.rados.0.smithi045.stdout:1798: expect deleted 2024-04-26T00:30:52.177 INFO:tasks.rados.rados.0.smithi045.stdout:1799: read oid 46 snap 175 2024-04-26T00:30:52.177 INFO:tasks.rados.rados.0.smithi045.stdout:1799: expect (ObjNum 329 snap 96 seq_num 329) 2024-04-26T00:30:52.178 INFO:tasks.rados.rados.0.smithi045.stdout:1800: snap_remove snap 178 2024-04-26T00:30:53.136 INFO:tasks.rados.rados.0.smithi045.stdout:1800: done (4 left) 2024-04-26T00:30:53.137 INFO:tasks.rados.rados.0.smithi045.stdout:1801: read oid 50 snap -1 2024-04-26T00:30:53.137 INFO:tasks.rados.rados.0.smithi045.stdout:1801: expect (ObjNum 654 snap 202 seq_num 654) 2024-04-26T00:30:53.137 INFO:tasks.rados.rados.0.smithi045.stdout:1802: snap_remove snap 198 2024-04-26T00:30:54.150 INFO:tasks.rados.rados.0.smithi045.stdout:1802: done (5 left) 2024-04-26T00:30:54.151 INFO:tasks.rados.rados.0.smithi045.stdout:1803: delete oid 49 current snap is 205 2024-04-26T00:30:54.151 INFO:tasks.rados.rados.0.smithi045.stdout:1796: finishing write tid 2 to smithi04526579-24 2024-04-26T00:30:54.151 INFO:tasks.rados.rados.0.smithi045.stdout:1796: finishing write tid 3 to smithi04526579-24 2024-04-26T00:30:54.151 INFO:tasks.rados.rados.0.smithi045.stdout:1796: finishing write tid 4 to smithi04526579-24 2024-04-26T00:30:54.151 INFO:tasks.rados.rados.0.smithi045.stdout:1796: oid 24 updating version 0 to 1471 2024-04-26T00:30:54.151 INFO:tasks.rados.rados.0.smithi045.stdout:1796: oid 24 updating version 1471 to 1472 2024-04-26T00:30:54.151 INFO:tasks.rados.rados.0.smithi045.stdout:1796: oid 24 updating version 1472 to 1473 2024-04-26T00:30:54.151 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 1473 (ObjNum 659 snap 205 seq_num 659) dirty exists 2024-04-26T00:30:54.151 INFO:tasks.rados.rados.0.smithi045.stdout:1796: left oid 24 (ObjNum 659 snap 205 seq_num 659) 2024-04-26T00:30:54.151 INFO:tasks.rados.rados.0.smithi045.stdout:1799: expect (ObjNum 329 snap 96 seq_num 329) 2024-04-26T00:30:54.321 INFO:tasks.rados.rados.0.smithi045.stdout:1797: finishing copy_from racing read to smithi04526579-16 2024-04-26T00:30:54.321 INFO:tasks.rados.rados.0.smithi045.stdout:1797: finishing copy_from to smithi04526579-16 2024-04-26T00:30:54.321 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 1120 (ObjNum 645 snap 202 seq_num 645) dirty exists 2024-04-26T00:30:54.321 INFO:tasks.rados.rados.0.smithi045.stdout:1801: expect (ObjNum 654 snap 202 seq_num 654) 2024-04-26T00:30:54.545 INFO:tasks.rados.rados.0.smithi045.stdout:1796: done (5 left) 2024-04-26T00:30:54.545 INFO:tasks.rados.rados.0.smithi045.stdout:1797: done (4 left) 2024-04-26T00:30:54.545 INFO:tasks.rados.rados.0.smithi045.stdout:1798: done (3 left) 2024-04-26T00:30:54.545 INFO:tasks.rados.rados.0.smithi045.stdout:1799: done (2 left) 2024-04-26T00:30:54.545 INFO:tasks.rados.rados.0.smithi045.stdout:1801: done (1 left) 2024-04-26T00:30:54.545 INFO:tasks.rados.rados.0.smithi045.stdout:1803: done (0 left) 2024-04-26T00:30:54.545 INFO:tasks.rados.rados.0.smithi045.stdout:1804: write oid 1 current snap is 205 2024-04-26T00:30:54.546 INFO:tasks.rados.rados.0.smithi045.stdout:1804: seq_num 661 ranges {709427=545947,1813774=612897,2985657=123770} 2024-04-26T00:30:54.552 INFO:tasks.rados.rados.0.smithi045.stdout:1804: writing smithi04526579-1 from 709427 to 1255374 tid 1 2024-04-26T00:30:54.558 INFO:tasks.rados.rados.0.smithi045.stdout:1804: writing smithi04526579-1 from 1813774 to 2426671 tid 2 2024-04-26T00:30:54.562 INFO:tasks.rados.rados.0.smithi045.stdout:1804: writing smithi04526579-1 from 2985657 to 3109427 tid 3 2024-04-26T00:30:54.562 INFO:tasks.rados.rados.0.smithi045.stdout:1805: delete oid 21 current snap is 205 2024-04-26T00:30:54.568 INFO:tasks.rados.rados.0.smithi045.stdout:1805: done (1 left) 2024-04-26T00:30:54.568 INFO:tasks.rados.rados.0.smithi045.stdout:1806: snap_create 2024-04-26T00:30:54.571 INFO:tasks.rados.rados.0.smithi045.stdout:1804: finishing write tid 1 to smithi04526579-1 2024-04-26T00:30:54.571 INFO:tasks.rados.rados.0.smithi045.stdout:1804: finishing write tid 2 to smithi04526579-1 2024-04-26T00:30:54.573 INFO:tasks.rados.rados.0.smithi045.stdout:1804: finishing write tid 3 to smithi04526579-1 2024-04-26T00:30:54.573 INFO:tasks.rados.rados.0.smithi045.stdout:1804: finishing write tid 4 to smithi04526579-1 2024-04-26T00:30:54.574 INFO:tasks.rados.rados.0.smithi045.stdout:1804: finishing write tid 5 to smithi04526579-1 2024-04-26T00:30:54.574 INFO:tasks.rados.rados.0.smithi045.stdout:1804: oid 1 updating version 0 to 1155 2024-04-26T00:30:54.574 INFO:tasks.rados.rados.0.smithi045.stdout:1804: oid 1 updating version 1155 to 1158 2024-04-26T00:30:54.574 INFO:tasks.rados.rados.0.smithi045.stdout:1804: oid 1 version 1158 is already newer than 1156 2024-04-26T00:30:54.574 INFO:tasks.rados.rados.0.smithi045.stdout:1804: oid 1 version 1158 is already newer than 1157 2024-04-26T00:30:54.574 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 1158 (ObjNum 661 snap 205 seq_num 661) dirty exists 2024-04-26T00:30:54.574 INFO:tasks.rados.rados.0.smithi045.stdout:1804: left oid 1 (ObjNum 661 snap 205 seq_num 661) 2024-04-26T00:30:55.116 INFO:tasks.rados.rados.0.smithi045.stdout:1804: done (1 left) 2024-04-26T00:30:55.116 INFO:tasks.rados.rados.0.smithi045.stdout:1806: done (0 left) 2024-04-26T00:30:55.117 INFO:tasks.rados.rados.0.smithi045.stdout:1807: write (excl) oid 5 current snap is 206 2024-04-26T00:30:55.117 INFO:tasks.rados.rados.0.smithi045.stdout:1807: seq_num 663 ranges {743041=412798,1930738=616566,3040503=102538} 2024-04-26T00:30:55.132 INFO:tasks.rados.rados.0.smithi045.stdout:1807: writing smithi04526579-5 from 743041 to 1155839 tid 1 2024-04-26T00:30:55.152 INFO:tasks.rados.rados.0.smithi045.stdout:1807: writing smithi04526579-5 from 1930738 to 2547304 tid 2 2024-04-26T00:30:55.161 INFO:tasks.rados.rados.0.smithi045.stdout:1807: writing smithi04526579-5 from 3040503 to 3143041 tid 3 2024-04-26T00:30:55.161 INFO:tasks.rados.rados.0.smithi045.stdout:1808: read oid 41 snap -1 2024-04-26T00:30:55.161 INFO:tasks.rados.rados.0.smithi045.stdout:1808: expect deleted 2024-04-26T00:30:55.161 INFO:tasks.rados.rados.0.smithi045.stdout:1809: read oid 35 snap -1 2024-04-26T00:30:55.161 INFO:tasks.rados.rados.0.smithi045.stdout:1809: expect (ObjNum 401 snap 122 seq_num 401) 2024-04-26T00:30:55.161 INFO:tasks.rados.rados.0.smithi045.stdout:1810: copy_from oid 45 from oid 49 current snap is 206 2024-04-26T00:30:55.162 INFO:tasks.rados.rados.0.smithi045.stdout:1807: finishing write tid 1 to smithi04526579-5 2024-04-26T00:30:55.162 INFO:tasks.rados.rados.0.smithi045.stdout:1811: snap_create 2024-04-26T00:30:55.167 INFO:tasks.rados.rados.0.smithi045.stdout:1807: finishing write tid 2 to smithi04526579-5 2024-04-26T00:30:55.168 INFO:tasks.rados.rados.0.smithi045.stdout:1807: finishing write tid 3 to smithi04526579-5 2024-04-26T00:30:55.168 INFO:tasks.rados.rados.0.smithi045.stdout:1810: finishing copy_from to smithi04526579-45 2024-04-26T00:30:55.168 INFO:tasks.rados.rados.0.smithi045.stdout:1810: got expected ENOENT (src dne) 2024-04-26T00:30:55.169 INFO:tasks.rados.rados.0.smithi045.stdout:1807: finishing write tid 4 to smithi04526579-5 2024-04-26T00:30:55.169 INFO:tasks.rados.rados.0.smithi045.stdout:1810: finishing copy_from racing read to smithi04526579-45 2024-04-26T00:30:55.170 INFO:tasks.rados.rados.0.smithi045.stdout:1807: finishing write tid 5 to smithi04526579-5 2024-04-26T00:30:55.170 INFO:tasks.rados.rados.0.smithi045.stdout:1807: oid 5 updating version 0 to 1491 2024-04-26T00:30:55.170 INFO:tasks.rados.rados.0.smithi045.stdout:1807: oid 5 updating version 1491 to 1492 2024-04-26T00:30:55.170 INFO:tasks.rados.rados.0.smithi045.stdout:1807: oid 5 updating version 1492 to 1493 2024-04-26T00:30:55.170 INFO:tasks.rados.rados.0.smithi045.stdout:1807: oid 5 updating version 1493 to 1494 2024-04-26T00:30:55.170 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 1494 (ObjNum 663 snap 206 seq_num 663) dirty exists 2024-04-26T00:30:55.170 INFO:tasks.rados.rados.0.smithi045.stdout:1807: left oid 5 (ObjNum 663 snap 206 seq_num 663) 2024-04-26T00:30:55.181 INFO:tasks.rados.rados.0.smithi045.stdout:1809: expect (ObjNum 401 snap 122 seq_num 401) 2024-04-26T00:30:56.242 INFO:tasks.rados.rados.0.smithi045.stdout:1807: done (4 left) 2024-04-26T00:30:56.243 INFO:tasks.rados.rados.0.smithi045.stdout:1808: done (3 left) 2024-04-26T00:30:56.243 INFO:tasks.rados.rados.0.smithi045.stdout:1809: done (2 left) 2024-04-26T00:30:56.243 INFO:tasks.rados.rados.0.smithi045.stdout:1810: done (1 left) 2024-04-26T00:30:56.243 INFO:tasks.rados.rados.0.smithi045.stdout:1811: done (0 left) 2024-04-26T00:30:56.243 INFO:tasks.rados.rados.0.smithi045.stdout:1812: rollback oid 16 current snap is 207 2024-04-26T00:30:56.243 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 16 to 180 2024-04-26T00:30:56.243 INFO:tasks.rados.rados.0.smithi045.stdout:1813: rollback oid 32 current snap is 207 2024-04-26T00:30:56.243 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 32 to 206 2024-04-26T00:30:56.243 INFO:tasks.rados.rados.0.smithi045.stdout:1814: snap_create 2024-04-26T00:30:56.260 INFO:tasks.rados.rados.0.smithi045.stdout:1812: finishing rollback tid 0 to smithi04526579-16 2024-04-26T00:30:56.261 INFO:tasks.rados.rados.0.smithi045.stdout:1813: finishing rollback tid 0 to smithi04526579-32 2024-04-26T00:30:56.261 INFO:tasks.rados.rados.0.smithi045.stdout:1812: finishing rollback tid 1 to smithi04526579-16 2024-04-26T00:30:56.261 INFO:tasks.rados.rados.0.smithi045.stdout:1813: finishing rollback tid 1 to smithi04526579-32 2024-04-26T00:30:56.262 INFO:tasks.rados.rados.0.smithi045.stdout:1812: finishing rollback tid 2 to smithi04526579-16 2024-04-26T00:30:56.262 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 1123 (ObjNum 515 snap 157 seq_num 515) dirty exists 2024-04-26T00:30:56.262 INFO:tasks.rados.rados.0.smithi045.stdout:1813: finishing rollback tid 2 to smithi04526579-32 2024-04-26T00:30:56.262 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 1126 (ObjNum 639 snap 201 seq_num 639) dirty exists 2024-04-26T00:30:57.243 INFO:tasks.rados.rados.0.smithi045.stdout:1812: done (2 left) 2024-04-26T00:30:57.243 INFO:tasks.rados.rados.0.smithi045.stdout:1813: done (1 left) 2024-04-26T00:30:57.244 INFO:tasks.rados.rados.0.smithi045.stdout:1814: done (0 left) 2024-04-26T00:30:57.244 INFO:tasks.rados.rados.0.smithi045.stdout:1815: snap_create 2024-04-26T00:30:58.252 INFO:tasks.rados.rados.0.smithi045.stdout:1815: done (0 left) 2024-04-26T00:30:58.252 INFO:tasks.rados.rados.0.smithi045.stdout:1816: rollback oid 4 current snap is 209 2024-04-26T00:30:58.252 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 4 to 140 2024-04-26T00:30:58.252 INFO:tasks.rados.rados.0.smithi045.stdout:1817: read oid 27 snap -1 2024-04-26T00:30:58.253 INFO:tasks.rados.rados.0.smithi045.stdout:1817: expect (ObjNum 582 snap 180 seq_num 582) 2024-04-26T00:30:58.253 INFO:tasks.rados.rados.0.smithi045.stdout:1818: rollback oid 34 current snap is 209 2024-04-26T00:30:58.253 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 34 to 160 2024-04-26T00:30:58.253 INFO:tasks.rados.rados.0.smithi045.stdout:1819: delete oid 28 current snap is 209 2024-04-26T00:30:58.267 INFO:tasks.rados.rados.0.smithi045.stdout:1819: done (3 left) 2024-04-26T00:30:58.267 INFO:tasks.rados.rados.0.smithi045.stdout:1820: write (excl) oid 47 current snap is 209 2024-04-26T00:30:58.267 INFO:tasks.rados.rados.0.smithi045.stdout:1820: seq_num 665 ranges {776655=563296,2047702=620235,3095349=81306} 2024-04-26T00:30:58.290 INFO:tasks.rados.rados.0.smithi045.stdout:1820: writing smithi04526579-47 from 776655 to 1339951 tid 1 2024-04-26T00:30:58.312 INFO:tasks.rados.rados.0.smithi045.stdout:1820: writing smithi04526579-47 from 2047702 to 2667937 tid 2 2024-04-26T00:30:58.319 INFO:tasks.rados.rados.0.smithi045.stdout:1820: writing smithi04526579-47 from 3095349 to 3176655 tid 3 2024-04-26T00:30:58.320 INFO:tasks.rados.rados.0.smithi045.stdout:1821: read oid 10 snap -1 2024-04-26T00:30:58.320 INFO:tasks.rados.rados.0.smithi045.stdout:1821: expect (ObjNum 573 snap 176 seq_num 573) 2024-04-26T00:30:58.320 INFO:tasks.rados.rados.0.smithi045.stdout:1822: write oid 1 current snap is 209 2024-04-26T00:30:58.320 INFO:tasks.rados.rados.0.smithi045.stdout:1822: seq_num 666 ranges {793462=638545,2106184=763893} 2024-04-26T00:30:58.336 INFO:tasks.rados.rados.0.smithi045.stdout:1822: writing smithi04526579-1 from 793462 to 1432007 tid 1 2024-04-26T00:30:58.349 INFO:tasks.rados.rados.0.smithi045.stdout:1822: writing smithi04526579-1 from 2106184 to 2870077 tid 2 2024-04-26T00:30:58.349 INFO:tasks.rados.rados.0.smithi045.stdout:1823: write (excl) oid 23 current snap is 209 2024-04-26T00:30:58.350 INFO:tasks.rados.rados.0.smithi045.stdout:1823: seq_num 667 ranges {410269=713794,1648313=507551,2917489=292780} 2024-04-26T00:30:58.359 INFO:tasks.rados.rados.0.smithi045.stdout:1823: writing smithi04526579-23 from 410269 to 1124063 tid 1 2024-04-26T00:30:58.367 INFO:tasks.rados.rados.0.smithi045.stdout:1823: writing smithi04526579-23 from 1648313 to 2155864 tid 2 2024-04-26T00:30:58.374 INFO:tasks.rados.rados.0.smithi045.stdout:1823: writing smithi04526579-23 from 2917489 to 3210269 tid 3 2024-04-26T00:30:58.374 INFO:tasks.rados.rados.0.smithi045.stdout:1824: read oid 24 snap -1 2024-04-26T00:30:58.374 INFO:tasks.rados.rados.0.smithi045.stdout:1824: expect (ObjNum 659 snap 205 seq_num 659) 2024-04-26T00:30:58.374 INFO:tasks.rados.rados.0.smithi045.stdout:1825: snap_remove snap 193 2024-04-26T00:30:59.260 INFO:tasks.rados.rados.0.smithi045.stdout:1825: done (8 left) 2024-04-26T00:30:59.261 INFO:tasks.rados.rados.0.smithi045.stdout:1826: rollback oid 28 current snap is 209 2024-04-26T00:30:59.261 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 28 to 203 2024-04-26T00:30:59.261 INFO:tasks.rados.rados.0.smithi045.stdout:1818: finishing rollback tid 0 to smithi04526579-34 2024-04-26T00:30:59.261 INFO:tasks.rados.rados.0.smithi045.stdout:1818: finishing rollback tid 1 to smithi04526579-34 2024-04-26T00:30:59.261 INFO:tasks.rados.rados.0.smithi045.stdout:1818: finishing rollback tid 2 to smithi04526579-34 2024-04-26T00:30:59.261 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 1011 (ObjNum 463 snap 145 seq_num 463) dirty exists 2024-04-26T00:30:59.261 INFO:tasks.rados.rados.0.smithi045.stdout:1816: finishing rollback tid 0 to smithi04526579-4 2024-04-26T00:30:59.261 INFO:tasks.rados.rados.0.smithi045.stdout:1816: finishing rollback tid 1 to smithi04526579-4 2024-04-26T00:30:59.261 INFO:tasks.rados.rados.0.smithi045.stdout:1816: finishing rollback tid 2 to smithi04526579-4 2024-04-26T00:30:59.261 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 993 (ObjNum 406 snap 125 seq_num 406) dirty exists 2024-04-26T00:30:59.261 INFO:tasks.rados.rados.0.smithi045.stdout:1817: expect (ObjNum 582 snap 180 seq_num 582) 2024-04-26T00:30:59.496 INFO:tasks.rados.rados.0.smithi045.stdout:1820: finishing write tid 1 to smithi04526579-47 2024-04-26T00:30:59.496 INFO:tasks.rados.rados.0.smithi045.stdout:1820: finishing write tid 2 to smithi04526579-47 2024-04-26T00:30:59.496 INFO:tasks.rados.rados.0.smithi045.stdout:1820: finishing write tid 3 to smithi04526579-47 2024-04-26T00:30:59.496 INFO:tasks.rados.rados.0.smithi045.stdout:1820: finishing write tid 4 to smithi04526579-47 2024-04-26T00:30:59.497 INFO:tasks.rados.rados.0.smithi045.stdout:1820: finishing write tid 5 to smithi04526579-47 2024-04-26T00:30:59.497 INFO:tasks.rados.rados.0.smithi045.stdout:1820: oid 47 updating version 0 to 1014 2024-04-26T00:30:59.497 INFO:tasks.rados.rados.0.smithi045.stdout:1820: oid 47 updating version 1014 to 1015 2024-04-26T00:30:59.497 INFO:tasks.rados.rados.0.smithi045.stdout:1820: oid 47 version 1015 is already newer than 1013 2024-04-26T00:30:59.497 INFO:tasks.rados.rados.0.smithi045.stdout:1820: oid 47 updating version 1015 to 1016 2024-04-26T00:30:59.497 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 1016 (ObjNum 665 snap 209 seq_num 665) dirty exists 2024-04-26T00:30:59.497 INFO:tasks.rados.rados.0.smithi045.stdout:1820: left oid 47 (ObjNum 665 snap 209 seq_num 665) 2024-04-26T00:30:59.497 INFO:tasks.rados.rados.0.smithi045.stdout:1821: expect (ObjNum 573 snap 176 seq_num 573) 2024-04-26T00:30:59.617 INFO:tasks.rados.rados.0.smithi045.stdout:1822: finishing write tid 1 to smithi04526579-1 2024-04-26T00:30:59.617 INFO:tasks.rados.rados.0.smithi045.stdout:1822: finishing write tid 2 to smithi04526579-1 2024-04-26T00:30:59.617 INFO:tasks.rados.rados.0.smithi045.stdout:1822: finishing write tid 3 to smithi04526579-1 2024-04-26T00:30:59.617 INFO:tasks.rados.rados.0.smithi045.stdout:1822: finishing write tid 4 to smithi04526579-1 2024-04-26T00:30:59.617 INFO:tasks.rados.rados.0.smithi045.stdout:1822: oid 1 updating version 0 to 1163 2024-04-26T00:30:59.617 INFO:tasks.rados.rados.0.smithi045.stdout:1822: oid 1 version 1163 is already newer than 1162 2024-04-26T00:30:59.618 INFO:tasks.rados.rados.0.smithi045.stdout:1822: oid 1 updating version 1163 to 1164 2024-04-26T00:30:59.618 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 1164 (ObjNum 666 snap 209 seq_num 666) dirty exists 2024-04-26T00:30:59.618 INFO:tasks.rados.rados.0.smithi045.stdout:1822: left oid 1 (ObjNum 666 snap 209 seq_num 666) 2024-04-26T00:30:59.618 INFO:tasks.rados.rados.0.smithi045.stdout:1823: finishing write tid 1 to smithi04526579-23 2024-04-26T00:30:59.618 INFO:tasks.rados.rados.0.smithi045.stdout:1823: finishing write tid 2 to smithi04526579-23 2024-04-26T00:30:59.618 INFO:tasks.rados.rados.0.smithi045.stdout:1823: finishing write tid 3 to smithi04526579-23 2024-04-26T00:30:59.618 INFO:tasks.rados.rados.0.smithi045.stdout:1823: finishing write tid 4 to smithi04526579-23 2024-04-26T00:30:59.618 INFO:tasks.rados.rados.0.smithi045.stdout:1823: finishing write tid 5 to smithi04526579-23 2024-04-26T00:30:59.618 INFO:tasks.rados.rados.0.smithi045.stdout:1823: oid 23 updating version 0 to 1050 2024-04-26T00:30:59.618 INFO:tasks.rados.rados.0.smithi045.stdout:1823: oid 23 updating version 1050 to 1052 2024-04-26T00:30:59.618 INFO:tasks.rados.rados.0.smithi045.stdout:1823: oid 23 version 1052 is already newer than 1049 2024-04-26T00:30:59.618 INFO:tasks.rados.rados.0.smithi045.stdout:1823: oid 23 version 1052 is already newer than 1051 2024-04-26T00:30:59.618 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 1052 (ObjNum 667 snap 209 seq_num 667) dirty exists 2024-04-26T00:30:59.619 INFO:tasks.rados.rados.0.smithi045.stdout:1823: left oid 23 (ObjNum 667 snap 209 seq_num 667) 2024-04-26T00:30:59.619 INFO:tasks.rados.rados.0.smithi045.stdout:1824: expect (ObjNum 659 snap 205 seq_num 659) 2024-04-26T00:30:59.785 INFO:tasks.rados.rados.0.smithi045.stdout:1826: finishing rollback tid 1 to smithi04526579-28 2024-04-26T00:30:59.785 INFO:tasks.rados.rados.0.smithi045.stdout:1826: finishing rollback tid 2 to smithi04526579-28 2024-04-26T00:30:59.785 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 1166 (ObjNum 655 snap 202 seq_num 655) dirty exists 2024-04-26T00:30:59.785 INFO:tasks.rados.rados.0.smithi045.stdout:1816: done (8 left) 2024-04-26T00:30:59.785 INFO:tasks.rados.rados.0.smithi045.stdout:1817: done (7 left) 2024-04-26T00:30:59.785 INFO:tasks.rados.rados.0.smithi045.stdout:1818: done (6 left) 2024-04-26T00:30:59.785 INFO:tasks.rados.rados.0.smithi045.stdout:1820: done (5 left) 2024-04-26T00:30:59.785 INFO:tasks.rados.rados.0.smithi045.stdout:1821: done (4 left) 2024-04-26T00:30:59.785 INFO:tasks.rados.rados.0.smithi045.stdout:1822: done (3 left) 2024-04-26T00:30:59.785 INFO:tasks.rados.rados.0.smithi045.stdout:1823: done (2 left) 2024-04-26T00:30:59.785 INFO:tasks.rados.rados.0.smithi045.stdout:1824: done (1 left) 2024-04-26T00:30:59.785 INFO:tasks.rados.rados.0.smithi045.stdout:1826: done (0 left) 2024-04-26T00:30:59.785 INFO:tasks.rados.rados.0.smithi045.stdout:1827: snap_create 2024-04-26T00:31:00.119 INFO:tasks.rados.rados.0.smithi045.stdout:1827: done (0 left) 2024-04-26T00:31:00.120 INFO:tasks.rados.rados.0.smithi045.stdout:1828: write (excl) oid 13 current snap is 210 2024-04-26T00:31:00.120 INFO:tasks.rados.rados.0.smithi045.stdout:1828: seq_num 668 ranges {427076=789043,1706795=767562,2944912=282164} 2024-04-26T00:31:00.126 INFO:tasks.rados.rados.0.smithi045.stdout:1828: writing smithi04526579-13 from 427076 to 1216119 tid 1 2024-04-26T00:31:00.133 INFO:tasks.rados.rados.0.smithi045.stdout:1828: writing smithi04526579-13 from 1706795 to 2474357 tid 2 2024-04-26T00:31:00.138 INFO:tasks.rados.rados.0.smithi045.stdout:1828: writing smithi04526579-13 from 2944912 to 3227076 tid 3 2024-04-26T00:31:00.138 INFO:tasks.rados.rados.0.smithi045.stdout:1829: read oid 7 snap -1 2024-04-26T00:31:00.138 INFO:tasks.rados.rados.0.smithi045.stdout:1829: expect deleted 2024-04-26T00:31:00.138 INFO:tasks.rados.rados.0.smithi045.stdout:1830: write (excl) oid 37 current snap is 210 2024-04-26T00:31:00.138 INFO:tasks.rados.rados.0.smithi045.stdout:1830: seq_num 669 ranges {443883=464292,1365277=511220,2572335=671548} 2024-04-26T00:31:00.143 INFO:tasks.rados.rados.0.smithi045.stdout:1830: writing smithi04526579-37 from 443883 to 908175 tid 1 2024-04-26T00:31:00.149 INFO:tasks.rados.rados.0.smithi045.stdout:1830: writing smithi04526579-37 from 1365277 to 1876497 tid 2 2024-04-26T00:31:00.158 INFO:tasks.rados.rados.0.smithi045.stdout:1830: writing smithi04526579-37 from 2572335 to 3243883 tid 3 2024-04-26T00:31:00.158 INFO:tasks.rados.rados.0.smithi045.stdout:1831: copy_from oid 26 from oid 9 current snap is 210 2024-04-26T00:31:00.158 INFO:tasks.rados.rados.0.smithi045.stdout:1828: finishing write tid 1 to smithi04526579-13 2024-04-26T00:31:00.158 INFO:tasks.rados.rados.0.smithi045.stdout:1828: finishing write tid 2 to smithi04526579-13 2024-04-26T00:31:00.158 INFO:tasks.rados.rados.0.smithi045.stdout:1828: finishing write tid 3 to smithi04526579-13 2024-04-26T00:31:00.159 INFO:tasks.rados.rados.0.smithi045.stdout:1828: finishing write tid 4 to smithi04526579-13 2024-04-26T00:31:00.159 INFO:tasks.rados.rados.0.smithi045.stdout:1828: finishing write tid 5 to smithi04526579-13 2024-04-26T00:31:00.159 INFO:tasks.rados.rados.0.smithi045.stdout:1828: oid 13 updating version 0 to 878 2024-04-26T00:31:00.159 INFO:tasks.rados.rados.0.smithi045.stdout:1828: oid 13 updating version 878 to 881 2024-04-26T00:31:00.159 INFO:tasks.rados.rados.0.smithi045.stdout:1828: oid 13 version 881 is already newer than 880 2024-04-26T00:31:00.159 INFO:tasks.rados.rados.0.smithi045.stdout:1828: oid 13 version 881 is already newer than 879 2024-04-26T00:31:00.159 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 881 (ObjNum 668 snap 210 seq_num 668) dirty exists 2024-04-26T00:31:00.159 INFO:tasks.rados.rados.0.smithi045.stdout:1828: left oid 13 (ObjNum 668 snap 210 seq_num 668) 2024-04-26T00:31:00.159 INFO:tasks.rados.rados.0.smithi045.stdout:1828: done (3 left) 2024-04-26T00:31:00.159 INFO:tasks.rados.rados.0.smithi045.stdout:1829: done (2 left) 2024-04-26T00:31:00.159 INFO:tasks.rados.rados.0.smithi045.stdout:1832: delete oid 47 current snap is 210 2024-04-26T00:31:00.160 INFO:tasks.rados.rados.0.smithi045.stdout:1831: finishing copy_from to smithi04526579-26 2024-04-26T00:31:00.160 INFO:tasks.rados.rados.0.smithi045.stdout:1831: got expected ENOENT (src dne) 2024-04-26T00:31:00.160 INFO:tasks.rados.rados.0.smithi045.stdout:1831: finishing copy_from racing read to smithi04526579-26 2024-04-26T00:31:00.167 INFO:tasks.rados.rados.0.smithi045.stdout:1830: finishing write tid 1 to smithi04526579-37 2024-04-26T00:31:00.168 INFO:tasks.rados.rados.0.smithi045.stdout:1830: finishing write tid 2 to smithi04526579-37 2024-04-26T00:31:00.169 INFO:tasks.rados.rados.0.smithi045.stdout:1830: finishing write tid 3 to smithi04526579-37 2024-04-26T00:31:00.169 INFO:tasks.rados.rados.0.smithi045.stdout:1830: finishing write tid 4 to smithi04526579-37 2024-04-26T00:31:00.170 INFO:tasks.rados.rados.0.smithi045.stdout:1830: finishing write tid 5 to smithi04526579-37 2024-04-26T00:31:00.170 INFO:tasks.rados.rados.0.smithi045.stdout:1830: oid 37 updating version 0 to 1506 2024-04-26T00:31:00.170 INFO:tasks.rados.rados.0.smithi045.stdout:1830: oid 37 updating version 1506 to 1508 2024-04-26T00:31:00.170 INFO:tasks.rados.rados.0.smithi045.stdout:1830: oid 37 updating version 1508 to 1509 2024-04-26T00:31:00.170 INFO:tasks.rados.rados.0.smithi045.stdout:1830: oid 37 version 1509 is already newer than 1507 2024-04-26T00:31:00.170 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 1509 (ObjNum 669 snap 210 seq_num 669) dirty exists 2024-04-26T00:31:00.170 INFO:tasks.rados.rados.0.smithi045.stdout:1830: left oid 37 (ObjNum 669 snap 210 seq_num 669) 2024-04-26T00:31:00.173 INFO:tasks.rados.rados.0.smithi045.stdout:1830: done (2 left) 2024-04-26T00:31:00.173 INFO:tasks.rados.rados.0.smithi045.stdout:1831: done (1 left) 2024-04-26T00:31:00.173 INFO:tasks.rados.rados.0.smithi045.stdout:1832: done (0 left) 2024-04-26T00:31:00.173 INFO:tasks.rados.rados.0.smithi045.stdout:1833: read oid 28 snap -1 2024-04-26T00:31:00.173 INFO:tasks.rados.rados.0.smithi045.stdout:1833: expect (ObjNum 655 snap 202 seq_num 655) 2024-04-26T00:31:00.173 INFO:tasks.rados.rados.0.smithi045.stdout:1834: write oid 44 current snap is 210 2024-04-26T00:31:00.173 INFO:tasks.rados.rados.0.smithi045.stdout:1834: seq_num 671 ranges {477497=731143,1998594=514889,3143534=133963} 2024-04-26T00:31:00.183 INFO:tasks.rados.rados.0.smithi045.stdout:1834: writing smithi04526579-44 from 477497 to 1208640 tid 1 2024-04-26T00:31:00.193 INFO:tasks.rados.rados.0.smithi045.stdout:1834: writing smithi04526579-44 from 1998594 to 2513483 tid 2 2024-04-26T00:31:00.199 INFO:tasks.rados.rados.0.smithi045.stdout:1834: writing smithi04526579-44 from 3143534 to 3277497 tid 3 2024-04-26T00:31:00.199 INFO:tasks.rados.rados.0.smithi045.stdout:1835: copy_from oid 24 from oid 27 current snap is 210 2024-04-26T00:31:00.199 INFO:tasks.rados.rados.0.smithi045.stdout:1833: expect (ObjNum 655 snap 202 seq_num 655) 2024-04-26T00:31:00.460 INFO:tasks.rados.rados.0.smithi045.stdout:1834: finishing write tid 1 to smithi04526579-44 2024-04-26T00:31:00.461 INFO:tasks.rados.rados.0.smithi045.stdout:1834: finishing write tid 2 to smithi04526579-44 2024-04-26T00:31:00.461 INFO:tasks.rados.rados.0.smithi045.stdout:1834: finishing write tid 3 to smithi04526579-44 2024-04-26T00:31:00.461 INFO:tasks.rados.rados.0.smithi045.stdout:1834: finishing write tid 4 to smithi04526579-44 2024-04-26T00:31:00.461 INFO:tasks.rados.rados.0.smithi045.stdout:1834: finishing write tid 5 to smithi04526579-44 2024-04-26T00:31:00.461 INFO:tasks.rados.rados.0.smithi045.stdout:1834: oid 44 updating version 0 to 1068 2024-04-26T00:31:00.461 INFO:tasks.rados.rados.0.smithi045.stdout:1834: oid 44 version 1068 is already newer than 1066 2024-04-26T00:31:00.461 INFO:tasks.rados.rados.0.smithi045.stdout:1834: oid 44 version 1068 is already newer than 1067 2024-04-26T00:31:00.461 INFO:tasks.rados.rados.0.smithi045.stdout:1834: oid 44 version 1068 is already newer than 1065 2024-04-26T00:31:00.461 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 1068 (ObjNum 671 snap 210 seq_num 671) dirty exists 2024-04-26T00:31:00.461 INFO:tasks.rados.rados.0.smithi045.stdout:1834: left oid 44 (ObjNum 671 snap 210 seq_num 671) 2024-04-26T00:31:00.461 INFO:tasks.rados.rados.0.smithi045.stdout:1835: finishing copy_from racing read to smithi04526579-24 2024-04-26T00:31:00.461 INFO:tasks.rados.rados.0.smithi045.stdout:1835: finishing copy_from to smithi04526579-24 2024-04-26T00:31:00.461 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 1059 (ObjNum 582 snap 180 seq_num 582) dirty exists 2024-04-26T00:31:00.462 INFO:tasks.rados.rados.0.smithi045.stdout:1833: done (2 left) 2024-04-26T00:31:00.462 INFO:tasks.rados.rados.0.smithi045.stdout:1834: done (1 left) 2024-04-26T00:31:00.462 INFO:tasks.rados.rados.0.smithi045.stdout:1835: done (0 left) 2024-04-26T00:31:00.462 INFO:tasks.rados.rados.0.smithi045.stdout:1836: snap_remove snap 209 2024-04-26T00:31:01.271 INFO:tasks.rados.rados.0.smithi045.stdout:1836: done (0 left) 2024-04-26T00:31:01.271 INFO:tasks.rados.rados.0.smithi045.stdout:1837: snap_remove snap 179 2024-04-26T00:31:02.286 INFO:tasks.rados.rados.0.smithi045.stdout:1837: done (0 left) 2024-04-26T00:31:02.286 INFO:tasks.rados.rados.0.smithi045.stdout:1838: read oid 36 snap -1 2024-04-26T00:31:02.286 INFO:tasks.rados.rados.0.smithi045.stdout:1838: expect (ObjNum 600 snap 186 seq_num 600) 2024-04-26T00:31:02.286 INFO:tasks.rados.rados.0.smithi045.stdout:1839: rollback oid 44 current snap is 210 2024-04-26T00:31:02.286 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 44 to 153 2024-04-26T00:31:02.286 INFO:tasks.rados.rados.0.smithi045.stdout:1840: read oid 38 snap 206 2024-04-26T00:31:02.287 INFO:tasks.rados.rados.0.smithi045.stdout:1840: expect (ObjNum 582 snap 180 seq_num 582) 2024-04-26T00:31:02.287 INFO:tasks.rados.rados.0.smithi045.stdout:1841: write (excl) oid 23 current snap is 210 2024-04-26T00:31:02.287 INFO:tasks.rados.rados.0.smithi045.stdout:1841: seq_num 672 ranges {494304=406392,1540723=774900,2770957=523347} 2024-04-26T00:31:02.300 INFO:tasks.rados.rados.0.smithi045.stdout:1841: writing smithi04526579-23 from 494304 to 900696 tid 1 2024-04-26T00:31:02.322 INFO:tasks.rados.rados.0.smithi045.stdout:1841: writing smithi04526579-23 from 1540723 to 2315623 tid 2 2024-04-26T00:31:02.337 INFO:tasks.rados.rados.0.smithi045.stdout:1841: writing smithi04526579-23 from 2770957 to 3294304 tid 3 2024-04-26T00:31:02.338 INFO:tasks.rados.rados.0.smithi045.stdout:1842: copy_from oid 26 from oid 41 current snap is 210 2024-04-26T00:31:02.338 INFO:tasks.rados.rados.0.smithi045.stdout:1838: expect (ObjNum 600 snap 186 seq_num 600) 2024-04-26T00:31:02.478 INFO:tasks.rados.rados.0.smithi045.stdout:1841: finishing write tid 1 to smithi04526579-23 2024-04-26T00:31:02.478 INFO:tasks.rados.rados.0.smithi045.stdout:1840: expect (ObjNum 582 snap 180 seq_num 582) 2024-04-26T00:31:02.652 INFO:tasks.rados.rados.0.smithi045.stdout:1839: finishing rollback tid 0 to smithi04526579-44 2024-04-26T00:31:02.652 INFO:tasks.rados.rados.0.smithi045.stdout:1839: finishing rollback tid 1 to smithi04526579-44 2024-04-26T00:31:02.652 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 1070 (ObjNum 209 snap 0 seq_num 2242217520) dirty dne 2024-04-26T00:31:02.652 INFO:tasks.rados.rados.0.smithi045.stdout:1842: finishing copy_from to smithi04526579-26 2024-04-26T00:31:02.652 INFO:tasks.rados.rados.0.smithi045.stdout:1842: got expected ENOENT (src dne) 2024-04-26T00:31:02.652 INFO:tasks.rados.rados.0.smithi045.stdout:1842: finishing copy_from racing read to smithi04526579-26 2024-04-26T00:31:02.652 INFO:tasks.rados.rados.0.smithi045.stdout:1841: finishing write tid 2 to smithi04526579-23 2024-04-26T00:31:02.653 INFO:tasks.rados.rados.0.smithi045.stdout:1841: finishing write tid 3 to smithi04526579-23 2024-04-26T00:31:02.653 INFO:tasks.rados.rados.0.smithi045.stdout:1841: finishing write tid 4 to smithi04526579-23 2024-04-26T00:31:02.653 INFO:tasks.rados.rados.0.smithi045.stdout:1841: finishing write tid 5 to smithi04526579-23 2024-04-26T00:31:02.653 INFO:tasks.rados.rados.0.smithi045.stdout:1841: oid 23 updating version 0 to 1056 2024-04-26T00:31:02.653 INFO:tasks.rados.rados.0.smithi045.stdout:1841: oid 23 version 1056 is already newer than 1053 2024-04-26T00:31:02.653 INFO:tasks.rados.rados.0.smithi045.stdout:1841: oid 23 version 1056 is already newer than 1055 2024-04-26T00:31:02.653 INFO:tasks.rados.rados.0.smithi045.stdout:1841: oid 23 version 1056 is already newer than 1054 2024-04-26T00:31:02.653 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 1056 (ObjNum 672 snap 210 seq_num 672) dirty exists 2024-04-26T00:31:02.653 INFO:tasks.rados.rados.0.smithi045.stdout:1841: left oid 23 (ObjNum 672 snap 210 seq_num 672) 2024-04-26T00:31:02.653 INFO:tasks.rados.rados.0.smithi045.stdout:1838: done (4 left) 2024-04-26T00:31:02.653 INFO:tasks.rados.rados.0.smithi045.stdout:1839: done (3 left) 2024-04-26T00:31:02.653 INFO:tasks.rados.rados.0.smithi045.stdout:1840: done (2 left) 2024-04-26T00:31:02.653 INFO:tasks.rados.rados.0.smithi045.stdout:1841: done (1 left) 2024-04-26T00:31:02.653 INFO:tasks.rados.rados.0.smithi045.stdout:1842: done (0 left) 2024-04-26T00:31:02.654 INFO:tasks.rados.rados.0.smithi045.stdout:1843: snap_create 2024-04-26T00:31:03.289 INFO:tasks.rados.rados.0.smithi045.stdout:1843: done (0 left) 2024-04-26T00:31:03.289 INFO:tasks.rados.rados.0.smithi045.stdout:1844: read oid 27 snap -1 2024-04-26T00:31:03.289 INFO:tasks.rados.rados.0.smithi045.stdout:1844: expect (ObjNum 582 snap 180 seq_num 582) 2024-04-26T00:31:03.289 INFO:tasks.rados.rados.0.smithi045.stdout:1845: delete oid 30 current snap is 211 2024-04-26T00:31:03.299 INFO:tasks.rados.rados.0.smithi045.stdout:1845: done (1 left) 2024-04-26T00:31:03.299 INFO:tasks.rados.rados.0.smithi045.stdout:1846: read oid 21 snap -1 2024-04-26T00:31:03.299 INFO:tasks.rados.rados.0.smithi045.stdout:1846: expect deleted 2024-04-26T00:31:03.299 INFO:tasks.rados.rados.0.smithi045.stdout:1847: write (excl) oid 11 current snap is 211 2024-04-26T00:31:03.299 INFO:tasks.rados.rados.0.smithi045.stdout:1847: seq_num 674 ranges {527918=556890,1657687=778569,3225803=102115} 2024-04-26T00:31:03.317 INFO:tasks.rados.rados.0.smithi045.stdout:1847: writing smithi04526579-11 from 527918 to 1084808 tid 1 2024-04-26T00:31:03.340 INFO:tasks.rados.rados.0.smithi045.stdout:1847: writing smithi04526579-11 from 1657687 to 2436256 tid 2 2024-04-26T00:31:03.350 INFO:tasks.rados.rados.0.smithi045.stdout:1847: writing smithi04526579-11 from 3225803 to 3327918 tid 3 2024-04-26T00:31:03.350 INFO:tasks.rados.rados.0.smithi045.stdout:1848: delete oid 32 current snap is 211 2024-04-26T00:31:03.350 INFO:tasks.rados.rados.0.smithi045.stdout:1844: expect (ObjNum 582 snap 180 seq_num 582) 2024-04-26T00:31:03.528 INFO:tasks.rados.rados.0.smithi045.stdout:1847: finishing write tid 1 to smithi04526579-11 2024-04-26T00:31:03.528 INFO:tasks.rados.rados.0.smithi045.stdout:1847: finishing write tid 2 to smithi04526579-11 2024-04-26T00:31:03.528 INFO:tasks.rados.rados.0.smithi045.stdout:1847: finishing write tid 3 to smithi04526579-11 2024-04-26T00:31:03.528 INFO:tasks.rados.rados.0.smithi045.stdout:1847: finishing write tid 4 to smithi04526579-11 2024-04-26T00:31:03.528 INFO:tasks.rados.rados.0.smithi045.stdout:1847: finishing write tid 5 to smithi04526579-11 2024-04-26T00:31:03.528 INFO:tasks.rados.rados.0.smithi045.stdout:1847: oid 11 updating version 0 to 990 2024-04-26T00:31:03.528 INFO:tasks.rados.rados.0.smithi045.stdout:1847: oid 11 version 990 is already newer than 989 2024-04-26T00:31:03.528 INFO:tasks.rados.rados.0.smithi045.stdout:1847: oid 11 version 990 is already newer than 988 2024-04-26T00:31:03.528 INFO:tasks.rados.rados.0.smithi045.stdout:1847: oid 11 updating version 990 to 991 2024-04-26T00:31:03.529 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 991 (ObjNum 674 snap 211 seq_num 674) dirty exists 2024-04-26T00:31:03.529 INFO:tasks.rados.rados.0.smithi045.stdout:1847: left oid 11 (ObjNum 674 snap 211 seq_num 674) 2024-04-26T00:31:03.529 INFO:tasks.rados.rados.0.smithi045.stdout:1844: done (3 left) 2024-04-26T00:31:03.529 INFO:tasks.rados.rados.0.smithi045.stdout:1846: done (2 left) 2024-04-26T00:31:03.529 INFO:tasks.rados.rados.0.smithi045.stdout:1847: done (1 left) 2024-04-26T00:31:03.529 INFO:tasks.rados.rados.0.smithi045.stdout:1848: done (0 left) 2024-04-26T00:31:03.529 INFO:tasks.rados.rados.0.smithi045.stdout:1849: rollback oid 42 current snap is 211 2024-04-26T00:31:03.529 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 42 to 204 2024-04-26T00:31:03.529 INFO:tasks.rados.rados.0.smithi045.stdout:1850: write (excl) oid 39 current snap is 211 2024-04-26T00:31:03.529 INFO:tasks.rados.rados.0.smithi045.stdout:1850: seq_num 676 ranges {561532=707388,2058298=782238} 2024-04-26T00:31:03.547 INFO:tasks.rados.rados.0.smithi045.stdout:1850: writing smithi04526579-39 from 561532 to 1268920 tid 1 2024-04-26T00:31:03.572 INFO:tasks.rados.rados.0.smithi045.stdout:1850: writing smithi04526579-39 from 2058298 to 2840536 tid 2 2024-04-26T00:31:03.572 INFO:tasks.rados.rados.0.smithi045.stdout:1851: snap_create 2024-04-26T00:31:03.572 INFO:tasks.rados.rados.0.smithi045.stdout:1849: finishing rollback tid 1 to smithi04526579-42 2024-04-26T00:31:03.573 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 1090 (ObjNum 129 snap 0 seq_num 2241798816) dirty dne 2024-04-26T00:31:03.573 INFO:tasks.rados.rados.0.smithi045.stdout:1850: finishing write tid 1 to smithi04526579-39 2024-04-26T00:31:03.580 INFO:tasks.rados.rados.0.smithi045.stdout:1850: finishing write tid 2 to smithi04526579-39 2024-04-26T00:31:03.581 INFO:tasks.rados.rados.0.smithi045.stdout:1850: finishing write tid 3 to smithi04526579-39 2024-04-26T00:31:03.582 INFO:tasks.rados.rados.0.smithi045.stdout:1850: finishing write tid 4 to smithi04526579-39 2024-04-26T00:31:03.582 INFO:tasks.rados.rados.0.smithi045.stdout:1850: oid 39 updating version 0 to 1072 2024-04-26T00:31:03.582 INFO:tasks.rados.rados.0.smithi045.stdout:1850: oid 39 version 1072 is already newer than 1071 2024-04-26T00:31:03.582 INFO:tasks.rados.rados.0.smithi045.stdout:1850: oid 39 updating version 1072 to 1073 2024-04-26T00:31:03.582 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 1073 (ObjNum 676 snap 211 seq_num 676) dirty exists 2024-04-26T00:31:03.582 INFO:tasks.rados.rados.0.smithi045.stdout:1850: left oid 39 (ObjNum 676 snap 211 seq_num 676) 2024-04-26T00:31:04.296 INFO:tasks.rados.rados.0.smithi045.stdout:1849: done (2 left) 2024-04-26T00:31:04.297 INFO:tasks.rados.rados.0.smithi045.stdout:1850: done (1 left) 2024-04-26T00:31:04.297 INFO:tasks.rados.rados.0.smithi045.stdout:1851: done (0 left) 2024-04-26T00:31:04.297 INFO:tasks.rados.rados.0.smithi045.stdout:1852: write oid 49 current snap is 212 2024-04-26T00:31:04.297 INFO:tasks.rados.rados.0.smithi045.stdout:1852: seq_num 677 ranges {578339=498990,1833133=525896,2791719=586620} 2024-04-26T00:31:04.312 INFO:tasks.rados.rados.0.smithi045.stdout:1852: writing smithi04526579-49 from 578339 to 1077329 tid 1 2024-04-26T00:31:04.332 INFO:tasks.rados.rados.0.smithi045.stdout:1852: writing smithi04526579-49 from 1833133 to 2359029 tid 2 2024-04-26T00:31:04.347 INFO:tasks.rados.rados.0.smithi045.stdout:1852: writing smithi04526579-49 from 2791719 to 3378339 tid 3 2024-04-26T00:31:04.348 INFO:tasks.rados.rados.0.smithi045.stdout:1853: read oid 27 snap 205 2024-04-26T00:31:04.348 INFO:tasks.rados.rados.0.smithi045.stdout:1853: expect (ObjNum 582 snap 180 seq_num 582) 2024-04-26T00:31:04.348 INFO:tasks.rados.rados.0.smithi045.stdout:1854: write oid 38 current snap is 212 2024-04-26T00:31:04.348 INFO:tasks.rados.rados.0.smithi045.stdout:1854: seq_num 678 ranges {595146=574239,1891615=669554,3219142=176004} 2024-04-26T00:31:04.362 INFO:tasks.rados.rados.0.smithi045.stdout:1854: writing smithi04526579-38 from 595146 to 1169385 tid 1 2024-04-26T00:31:04.376 INFO:tasks.rados.rados.0.smithi045.stdout:1854: writing smithi04526579-38 from 1891615 to 2561169 tid 2 2024-04-26T00:31:04.383 INFO:tasks.rados.rados.0.smithi045.stdout:1854: writing smithi04526579-38 from 3219142 to 3395146 tid 3 2024-04-26T00:31:04.383 INFO:tasks.rados.rados.0.smithi045.stdout:1855: snap_remove snap 154 2024-04-26T00:31:05.123 INFO:tasks.rados.rados.0.smithi045.stdout:1852: finishing write tid 1 to smithi04526579-49 2024-04-26T00:31:05.123 INFO:tasks.rados.rados.0.smithi045.stdout:1852: finishing write tid 2 to smithi04526579-49 2024-04-26T00:31:05.123 INFO:tasks.rados.rados.0.smithi045.stdout:1852: finishing write tid 3 to smithi04526579-49 2024-04-26T00:31:05.123 INFO:tasks.rados.rados.0.smithi045.stdout:1852: finishing write tid 4 to smithi04526579-49 2024-04-26T00:31:05.123 INFO:tasks.rados.rados.0.smithi045.stdout:1852: finishing write tid 5 to smithi04526579-49 2024-04-26T00:31:05.124 INFO:tasks.rados.rados.0.smithi045.stdout:1852: oid 49 updating version 0 to 1037 2024-04-26T00:31:05.124 INFO:tasks.rados.rados.0.smithi045.stdout:1852: oid 49 version 1037 is already newer than 1036 2024-04-26T00:31:05.124 INFO:tasks.rados.rados.0.smithi045.stdout:1852: oid 49 version 1037 is already newer than 1035 2024-04-26T00:31:05.124 INFO:tasks.rados.rados.0.smithi045.stdout:1852: oid 49 updating version 1037 to 1038 2024-04-26T00:31:05.124 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 1038 (ObjNum 677 snap 212 seq_num 677) dirty exists 2024-04-26T00:31:05.124 INFO:tasks.rados.rados.0.smithi045.stdout:1852: left oid 49 (ObjNum 677 snap 212 seq_num 677) 2024-04-26T00:31:05.124 INFO:tasks.rados.rados.0.smithi045.stdout:1853: expect (ObjNum 582 snap 180 seq_num 582) 2024-04-26T00:31:05.324 INFO:tasks.rados.rados.0.smithi045.stdout:1854: finishing write tid 1 to smithi04526579-38 2024-04-26T00:31:05.324 INFO:tasks.rados.rados.0.smithi045.stdout:1854: finishing write tid 2 to smithi04526579-38 2024-04-26T00:31:05.324 INFO:tasks.rados.rados.0.smithi045.stdout:1854: finishing write tid 3 to smithi04526579-38 2024-04-26T00:31:05.324 INFO:tasks.rados.rados.0.smithi045.stdout:1854: finishing write tid 4 to smithi04526579-38 2024-04-26T00:31:05.324 INFO:tasks.rados.rados.0.smithi045.stdout:1854: finishing write tid 5 to smithi04526579-38 2024-04-26T00:31:05.324 INFO:tasks.rados.rados.0.smithi045.stdout:1854: oid 38 updating version 0 to 1076 2024-04-26T00:31:05.324 INFO:tasks.rados.rados.0.smithi045.stdout:1854: oid 38 version 1076 is already newer than 1075 2024-04-26T00:31:05.324 INFO:tasks.rados.rados.0.smithi045.stdout:1854: oid 38 updating version 1076 to 1077 2024-04-26T00:31:05.324 INFO:tasks.rados.rados.0.smithi045.stdout:1854: oid 38 version 1077 is already newer than 1074 2024-04-26T00:31:05.324 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 1077 (ObjNum 678 snap 212 seq_num 678) dirty exists 2024-04-26T00:31:05.325 INFO:tasks.rados.rados.0.smithi045.stdout:1854: left oid 38 (ObjNum 678 snap 212 seq_num 678) 2024-04-26T00:31:05.325 INFO:tasks.rados.rados.0.smithi045.stdout:1852: done (3 left) 2024-04-26T00:31:05.325 INFO:tasks.rados.rados.0.smithi045.stdout:1853: done (2 left) 2024-04-26T00:31:05.325 INFO:tasks.rados.rados.0.smithi045.stdout:1854: done (1 left) 2024-04-26T00:31:05.325 INFO:tasks.rados.rados.0.smithi045.stdout:1855: done (0 left) 2024-04-26T00:31:05.325 INFO:tasks.rados.rados.0.smithi045.stdout:1856: snap_remove snap 176 2024-04-26T00:31:06.307 INFO:tasks.rados.rados.0.smithi045.stdout:1856: done (0 left) 2024-04-26T00:31:06.308 INFO:tasks.rados.rados.0.smithi045.stdout:1857: write (excl) oid 48 current snap is 212 2024-04-26T00:31:06.308 INFO:tasks.rados.rados.0.smithi045.stdout:1857: seq_num 679 ranges {611953=649488,1950097=529565,3246565=165388} 2024-04-26T00:31:06.327 INFO:tasks.rados.rados.0.smithi045.stdout:1857: writing smithi04526579-48 from 611953 to 1261441 tid 1 2024-04-26T00:31:06.345 INFO:tasks.rados.rados.0.smithi045.stdout:1857: writing smithi04526579-48 from 1950097 to 2479662 tid 2 2024-04-26T00:31:06.359 INFO:tasks.rados.rados.0.smithi045.stdout:1857: writing smithi04526579-48 from 3246565 to 3411953 tid 3 2024-04-26T00:31:06.359 INFO:tasks.rados.rados.0.smithi045.stdout:1858: snap_remove snap 169 2024-04-26T00:31:07.308 INFO:tasks.rados.rados.0.smithi045.stdout:1858: done (1 left) 2024-04-26T00:31:07.309 INFO:tasks.rados.rados.0.smithi045.stdout:1859: snap_create 2024-04-26T00:31:07.309 INFO:tasks.rados.rados.0.smithi045.stdout:1857: finishing write tid 1 to smithi04526579-48 2024-04-26T00:31:07.309 INFO:tasks.rados.rados.0.smithi045.stdout:1857: finishing write tid 2 to smithi04526579-48 2024-04-26T00:31:07.309 INFO:tasks.rados.rados.0.smithi045.stdout:1857: finishing write tid 3 to smithi04526579-48 2024-04-26T00:31:07.309 INFO:tasks.rados.rados.0.smithi045.stdout:1857: finishing write tid 4 to smithi04526579-48 2024-04-26T00:31:07.309 INFO:tasks.rados.rados.0.smithi045.stdout:1857: finishing write tid 5 to smithi04526579-48 2024-04-26T00:31:07.309 INFO:tasks.rados.rados.0.smithi045.stdout:1857: oid 48 updating version 0 to 1056 2024-04-26T00:31:07.309 INFO:tasks.rados.rados.0.smithi045.stdout:1857: oid 48 version 1056 is already newer than 1054 2024-04-26T00:31:07.309 INFO:tasks.rados.rados.0.smithi045.stdout:1857: oid 48 version 1056 is already newer than 1053 2024-04-26T00:31:07.309 INFO:tasks.rados.rados.0.smithi045.stdout:1857: oid 48 version 1056 is already newer than 1055 2024-04-26T00:31:07.309 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 1056 (ObjNum 679 snap 212 seq_num 679) dirty exists 2024-04-26T00:31:07.309 INFO:tasks.rados.rados.0.smithi045.stdout:1857: left oid 48 (ObjNum 679 snap 212 seq_num 679) 2024-04-26T00:31:08.311 INFO:tasks.rados.rados.0.smithi045.stdout:1857: done (1 left) 2024-04-26T00:31:08.311 INFO:tasks.rados.rados.0.smithi045.stdout:1859: done (0 left) 2024-04-26T00:31:08.311 INFO:tasks.rados.rados.0.smithi045.stdout:1860: read oid 47 snap -1 2024-04-26T00:31:08.311 INFO:tasks.rados.rados.0.smithi045.stdout:1860: expect deleted 2024-04-26T00:31:08.311 INFO:tasks.rados.rados.0.smithi045.stdout:1861: write (excl) oid 1 current snap is 213 2024-04-26T00:31:08.311 INFO:tasks.rados.rados.0.smithi045.stdout:1861: seq_num 680 ranges {628760=724737,1892226=673223,3157635=271125} 2024-04-26T00:31:08.327 INFO:tasks.rados.rados.0.smithi045.stdout:1861: writing smithi04526579-1 from 628760 to 1353497 tid 1 2024-04-26T00:31:08.343 INFO:tasks.rados.rados.0.smithi045.stdout:1861: writing smithi04526579-1 from 1892226 to 2565449 tid 2 2024-04-26T00:31:08.354 INFO:tasks.rados.rados.0.smithi045.stdout:1861: writing smithi04526579-1 from 3157635 to 3428760 tid 3 2024-04-26T00:31:08.354 INFO:tasks.rados.rados.0.smithi045.stdout:1862: snap_remove snap 180 2024-04-26T00:31:09.324 INFO:tasks.rados.rados.0.smithi045.stdout:1862: done (2 left) 2024-04-26T00:31:09.325 INFO:tasks.rados.rados.0.smithi045.stdout:1863: read oid 20 snap -1 2024-04-26T00:31:09.325 INFO:tasks.rados.rados.0.smithi045.stdout:1863: expect (ObjNum 542 snap 167 seq_num 542) 2024-04-26T00:31:09.325 INFO:tasks.rados.rados.0.smithi045.stdout:1864: write oid 23 current snap is 213 2024-04-26T00:31:09.325 INFO:tasks.rados.rados.0.smithi045.stdout:1864: seq_num 681 ranges {645567=799986,1950708=533234,3185058=260509} 2024-04-26T00:31:09.336 INFO:tasks.rados.rados.0.smithi045.stdout:1864: writing smithi04526579-23 from 645567 to 1445553 tid 1 2024-04-26T00:31:09.350 INFO:tasks.rados.rados.0.smithi045.stdout:1864: writing smithi04526579-23 from 1950708 to 2483942 tid 2 2024-04-26T00:31:09.363 INFO:tasks.rados.rados.0.smithi045.stdout:1864: writing smithi04526579-23 from 3185058 to 3445567 tid 3 2024-04-26T00:31:09.363 INFO:tasks.rados.rados.0.smithi045.stdout:1865: copy_from oid 36 from oid 18 current snap is 213 2024-04-26T00:31:09.363 INFO:tasks.rados.rados.0.smithi045.stdout:1861: finishing write tid 1 to smithi04526579-1 2024-04-26T00:31:09.363 INFO:tasks.rados.rados.0.smithi045.stdout:1861: finishing write tid 2 to smithi04526579-1 2024-04-26T00:31:09.363 INFO:tasks.rados.rados.0.smithi045.stdout:1861: finishing write tid 3 to smithi04526579-1 2024-04-26T00:31:09.363 INFO:tasks.rados.rados.0.smithi045.stdout:1861: finishing write tid 4 to smithi04526579-1 2024-04-26T00:31:09.363 INFO:tasks.rados.rados.0.smithi045.stdout:1861: finishing write tid 5 to smithi04526579-1 2024-04-26T00:31:09.364 INFO:tasks.rados.rados.0.smithi045.stdout:1861: oid 1 updating version 0 to 1209 2024-04-26T00:31:09.364 INFO:tasks.rados.rados.0.smithi045.stdout:1861: oid 1 version 1209 is already newer than 1208 2024-04-26T00:31:09.364 INFO:tasks.rados.rados.0.smithi045.stdout:1861: oid 1 version 1209 is already newer than 1206 2024-04-26T00:31:09.364 INFO:tasks.rados.rados.0.smithi045.stdout:1861: oid 1 version 1209 is already newer than 1207 2024-04-26T00:31:09.364 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 1209 (ObjNum 680 snap 213 seq_num 680) dirty exists 2024-04-26T00:31:09.364 INFO:tasks.rados.rados.0.smithi045.stdout:1861: left oid 1 (ObjNum 680 snap 213 seq_num 680) 2024-04-26T00:31:09.364 INFO:tasks.rados.rados.0.smithi045.stdout:1863: expect (ObjNum 542 snap 167 seq_num 542) 2024-04-26T00:31:09.546 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-04-26T00:31:09.546 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-04-26T00:31:09.546 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-26T00:31:09.547 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-26T00:31:09.550 INFO:tasks.ceph.mon.b:Started 2024-04-26T00:31:09.550 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:31:09.550 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:31:09.559 INFO:tasks.rados.rados.0.smithi045.stdout:1864: finishing write tid 1 to smithi04526579-23 2024-04-26T00:31:09.559 INFO:tasks.rados.rados.0.smithi045.stdout:1864: finishing write tid 2 to smithi04526579-23 2024-04-26T00:31:09.559 INFO:tasks.rados.rados.0.smithi045.stdout:1864: finishing write tid 3 to smithi04526579-23 2024-04-26T00:31:09.559 INFO:tasks.rados.rados.0.smithi045.stdout:1864: finishing write tid 4 to smithi04526579-23 2024-04-26T00:31:09.559 INFO:tasks.rados.rados.0.smithi045.stdout:1864: finishing write tid 5 to smithi04526579-23 2024-04-26T00:31:09.559 INFO:tasks.rados.rados.0.smithi045.stdout:1864: oid 23 updating version 0 to 1058 2024-04-26T00:31:09.560 INFO:tasks.rados.rados.0.smithi045.stdout:1864: oid 23 version 1058 is already newer than 1057 2024-04-26T00:31:09.560 INFO:tasks.rados.rados.0.smithi045.stdout:1864: oid 23 updating version 1058 to 1060 2024-04-26T00:31:09.560 INFO:tasks.rados.rados.0.smithi045.stdout:1864: oid 23 version 1060 is already newer than 1059 2024-04-26T00:31:09.560 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 1060 (ObjNum 681 snap 213 seq_num 681) dirty exists 2024-04-26T00:31:09.560 INFO:tasks.rados.rados.0.smithi045.stdout:1864: left oid 23 (ObjNum 681 snap 213 seq_num 681) 2024-04-26T00:31:09.560 INFO:tasks.rados.rados.0.smithi045.stdout:1865: finishing copy_from racing read to smithi04526579-36 2024-04-26T00:31:09.560 INFO:tasks.rados.rados.0.smithi045.stdout:1865: finishing copy_from to smithi04526579-36 2024-04-26T00:31:09.560 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 1429 (ObjNum 551 snap 168 seq_num 551) dirty exists 2024-04-26T00:31:09.560 INFO:tasks.rados.rados.0.smithi045.stdout:1860: done (4 left) 2024-04-26T00:31:09.560 INFO:tasks.rados.rados.0.smithi045.stdout:1861: done (3 left) 2024-04-26T00:31:09.560 INFO:tasks.rados.rados.0.smithi045.stdout:1863: done (2 left) 2024-04-26T00:31:09.560 INFO:tasks.rados.rados.0.smithi045.stdout:1864: done (1 left) 2024-04-26T00:31:09.560 INFO:tasks.rados.rados.0.smithi045.stdout:1865: done (0 left) 2024-04-26T00:31:09.560 INFO:tasks.rados.rados.0.smithi045.stdout:1866: snap_remove snap 207 2024-04-26T00:31:09.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.689+0000 7fa89753f640 10 monclient: get_monmap_and_config 2024-04-26T00:31:09.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.689+0000 7fa89753f640 10 monclient: build_initial_monmap 2024-04-26T00:31:09.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.689+0000 7fa89753f640 10 monclient: monmap: 2024-04-26T00:31:09.682 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:31:09.682 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:09.683 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:31:09.693660+0000 2024-04-26T00:31:09.683 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:31:09.693660+0000 2024-04-26T00:31:09.683 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:31:09.683 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:09.683 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:31:09.683 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:31:09.683 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:31:09.683 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:31:09.683 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:31:09.683 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:09.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.689+0000 7fa89753f640 1 Processor -- start 2024-04-26T00:31:09.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.689+0000 7fa89753f640 1 -- start start 2024-04-26T00:31:09.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.689+0000 7fa89753f640 10 monclient: init 2024-04-26T00:31:09.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.689+0000 7fa89753f640 10 monclient: _reopen_session rank -1 2024-04-26T00:31:09.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.689+0000 7fa89753f640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-04-26T00:31:09.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.689+0000 7fa89753f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa890136e10 addr v1:172.21.15.37:6789/0 2024-04-26T00:31:09.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.693+0000 7fa89753f640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa890137900 0x7fa89009c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:09.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.693+0000 7fa89753f640 10 monclient(hunting): picked mon.noname-a con 0x7fa890137900 addr v2:172.21.15.37:3300/0 2024-04-26T00:31:09.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.693+0000 7fa89753f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa89013c340 0x7fa89013e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:09.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.693+0000 7fa89753f640 10 monclient(hunting): picked mon.noname-b con 0x7fa89013c340 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:31:09.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.693+0000 7fa89753f640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa89009b680 con 0x7fa890136e10 2024-04-26T00:31:09.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.693+0000 7fa89753f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:09.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.693+0000 7fa89753f640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fa89009b8f0 con 0x7fa890137900 2024-04-26T00:31:09.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.693+0000 7fa89753f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:09.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.693+0000 7fa89753f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa89009bbd0 con 0x7fa89013c340 2024-04-26T00:31:09.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.693+0000 7fa89753f640 10 monclient(hunting): _renew_subs 2024-04-26T00:31:09.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.693+0000 7fa89753f640 10 monclient(hunting): authenticate will time out at 1651.000080s 2024-04-26T00:31:09.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.693+0000 7fa8952db640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fa890136e10 0x7fa8901371f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:43532/0 (socket says 172.21.15.37:43532) 2024-04-26T00:31:09.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.693+0000 7fa8952db640 1 -- 172.21.15.37:0/3230393650 learned_addr learned my addr 172.21.15.37:0/3230393650 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:31:09.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.693+0000 7fa894ada640 1 --2- 172.21.15.37:0/3230393650 >> v2:172.21.15.37:3300/0 conn(0x7fa890137900 0x7fa89009c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:09.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.693+0000 7fa895adc640 1 --2- 172.21.15.37:0/3230393650 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa89013c340 0x7fa89013e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:09.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.693+0000 7fa894ada640 10 monclient(hunting): get_auth_request con 0x7fa890137900 auth_method 0 2024-04-26T00:31:09.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.693+0000 7fa894ada640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:09.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.693+0000 7fa894ada640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:09.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.693+0000 7fa894ada640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:09.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.693+0000 7fa895adc640 10 monclient(hunting): get_auth_request con 0x7fa89013c340 auth_method 0 2024-04-26T00:31:09.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.693+0000 7fa895adc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:09.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.693+0000 7fa895adc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:09.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.693+0000 7fa895adc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:09.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.693+0000 7fa894ada640 1 -- 172.21.15.37:0/3230393650 >> v2:172.21.15.37:3300/0 conn(0x7fa890137900 msgr2=0x7fa89009c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:31:09.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.693+0000 7fa894ada640 1 -- 172.21.15.37:0/3230393650 >> v2:172.21.15.37:3300/0 conn(0x7fa890137900 msgr2=0x7fa89009c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:31:09.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.693+0000 7fa895adc640 1 -- 172.21.15.37:0/3230393650 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa89013c340 msgr2=0x7fa89013e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:31:09.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.693+0000 7fa894ada640 1 --2- 172.21.15.37:0/3230393650 >> v2:172.21.15.37:3300/0 conn(0x7fa890137900 0x7fa89009c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:31:09.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.693+0000 7fa895adc640 1 -- 172.21.15.37:0/3230393650 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa89013c340 msgr2=0x7fa89013e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:31:09.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.693+0000 7fa895adc640 1 --2- 172.21.15.37:0/3230393650 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa89013c340 0x7fa89013e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:31:09.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.693+0000 7fa894ada640 1 --2- 172.21.15.37:0/3230393650 >> v2:172.21.15.37:3300/0 conn(0x7fa890137900 0x7fa89009c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:31:09.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.693+0000 7fa895adc640 1 --2- 172.21.15.37:0/3230393650 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa89013c340 0x7fa89013e730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:31:09.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.693+0000 7fa8952db640 1 -- 172.21.15.37:0/3230393650 >> v1:172.21.15.37:6789/0 conn(0x7fa890136e10 legacy=0x7fa8901371f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:09.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 1 -- 172.21.15.37:0/3230393650 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1510409794 0 0) 0x7fa89009b680 con 0x7fa890136e10 2024-04-26T00:31:09.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 10 monclient(hunting): _negotiate 2024-04-26T00:31:09.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:09.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:09.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 10 monclient(hunting): my global_id is 124188 2024-04-26T00:31:09.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 1 -- 172.21.15.37:0/3230393650 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa87c003530 con 0x7fa890136e10 2024-04-26T00:31:09.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 1 -- 172.21.15.37:0/3230393650 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 380916377 0 0) 0x7fa87c003530 con 0x7fa890136e10 2024-04-26T00:31:09.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 1 -- 172.21.15.37:0/3230393650 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa89013c340 msgr2=0x7fa89013e730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:31:09.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 1 --2- 172.21.15.37:0/3230393650 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa89013c340 0x7fa89013e730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:09.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 1 -- 172.21.15.37:0/3230393650 >> v2:172.21.15.37:3300/0 conn(0x7fa890137900 msgr2=0x7fa89009c5b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:31:09.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 1 --2- 172.21.15.37:0/3230393650 >> v2:172.21.15.37:3300/0 conn(0x7fa890137900 0x7fa89009c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:09.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 10 monclient: _finish_hunting 0 2024-04-26T00:31:09.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 1 monclient: found mon.noname-a-legacy 2024-04-26T00:31:09.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:31:09.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.37:6789/0 2024-04-26T00:31:09.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 1 -- 172.21.15.37:0/3230393650 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8900a5d90 con 0x7fa890136e10 2024-04-26T00:31:09.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 10 monclient: _finish_auth 0 2024-04-26T00:31:09.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 10 monclient: _check_auth_tickets 2024-04-26T00:31:09.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:31:09.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 1 -- 172.21.15.37:0/3230393650 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 866326386 0 0) 0x7fa88c001930 con 0x7fa890136e10 2024-04-26T00:31:09.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:09.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-26T00:31:09.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 10 monclient: dump: 2024-04-26T00:31:09.833 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:09.833 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:09.833 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:09.833 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:09.833 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:09.833 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:09.833 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:09.833 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:09.833 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:09.833 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:09.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] but i'm connected to v1:172.21.15.37:6789/0, reconnecting 2024-04-26T00:31:09.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 10 monclient: _reopen_session rank -1 2024-04-26T00:31:09.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 1 -- 172.21.15.37:0/3230393650 >> v1:172.21.15.37:6789/0 conn(0x7fa890136e10 legacy=0x7fa8901371f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:09.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:31:09.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 1 --2- 172.21.15.37:0/3230393650 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa89013c340 0x7fa87c003a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:09.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 10 monclient(hunting): picked mon.c con 0x7fa89013c340 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:31:09.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 1 --2- 172.21.15.37:0/3230393650 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa87c0040c0 0x7fa87c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:09.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 10 monclient(hunting): picked mon.a con 0x7fa87c0040c0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:31:09.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 1 --2- 172.21.15.37:0/3230393650 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa87c004d50 0x7fa87c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:09.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 10 monclient(hunting): picked mon.b con 0x7fa87c004d50 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:31:09.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:09.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 1 -- 172.21.15.37:0/3230393650 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa89009b8f0 con 0x7fa87c0040c0 2024-04-26T00:31:09.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:09.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 1 -- 172.21.15.37:0/3230393650 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa89009bbd0 con 0x7fa87c004d50 2024-04-26T00:31:09.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:09.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 1 -- 172.21.15.37:0/3230393650 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa87c00bce0 con 0x7fa89013c340 2024-04-26T00:31:09.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 10 monclient(hunting): _renew_subs 2024-04-26T00:31:09.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa887fff640 10 monclient(hunting): _finish_auth 0 2024-04-26T00:31:09.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa8952db640 1 --2- 172.21.15.37:0/3230393650 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa89013c340 0x7fa87c003a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:09.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa894ada640 1 --2- 172.21.15.37:0/3230393650 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa87c0040c0 0x7fa87c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:09.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa894ada640 10 monclient(hunting): get_auth_request con 0x7fa87c0040c0 auth_method 0 2024-04-26T00:31:09.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa894ada640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:09.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa894ada640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:09.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa894ada640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:31:09.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa8952db640 10 monclient(hunting): get_auth_request con 0x7fa89013c340 auth_method 0 2024-04-26T00:31:09.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa8952db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:09.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa8952db640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:09.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa8952db640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:31:09.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa894ada640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:09.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa894ada640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:09.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa894ada640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:31:09.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa8952db640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:09.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa8952db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:09.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa895adc640 1 --2- 172.21.15.37:0/3230393650 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa87c004d50 0x7fa87c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:09.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa8952db640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:31:09.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa895adc640 10 monclient(hunting): get_auth_request con 0x7fa87c004d50 auth_method 0 2024-04-26T00:31:09.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa895adc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:09.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa895adc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:09.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa895adc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:31:09.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa895adc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:09.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa895adc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:09.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa895adc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:31:09.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa8952db640 10 monclient(hunting): handle_auth_done global_id 124188 payload 293 2024-04-26T00:31:09.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa8952db640 1 -- 172.21.15.37:0/3230393650 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa87c004d50 msgr2=0x7fa87c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:31:09.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa8952db640 1 --2- 172.21.15.37:0/3230393650 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa87c004d50 0x7fa87c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:09.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa8952db640 1 -- 172.21.15.37:0/3230393650 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa87c0040c0 msgr2=0x7fa87c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:31:09.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa8952db640 1 --2- 172.21.15.37:0/3230393650 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa87c0040c0 0x7fa87c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:09.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa8952db640 10 monclient: _finish_hunting 0 2024-04-26T00:31:09.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa8952db640 1 monclient: found mon.c 2024-04-26T00:31:09.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa8952db640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:31:09.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa8952db640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:09.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa8952db640 1 -- 172.21.15.37:0/3230393650 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa87c003530 con 0x7fa89013c340 2024-04-26T00:31:09.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa8952db640 10 monclient: _finish_auth 0 2024-04-26T00:31:09.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa8952db640 10 monclient: _check_auth_tickets 2024-04-26T00:31:09.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa8952db640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:31:09.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa89753f640 5 monclient: authenticate success, global_id 124188 2024-04-26T00:31:09.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa89753f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:31:09.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.837+0000 7fa894ada640 1 --2- 172.21.15.37:0/3230393650 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa87c0040c0 0x7fa87c004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:31:09.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.841+0000 7fa8952db640 1 -- 172.21.15.37:0/3230393650 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa89013c340 msgr2=0x7fa87c003a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:09.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.841+0000 7fa8952db640 1 --2- 172.21.15.37:0/3230393650 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa89013c340 0x7fa87c003a40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa88c001ec0 tx=0x7fa88c01cd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36d30a4bd7469981 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:09.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.845+0000 7fa887fff640 1 -- 172.21.15.37:0/3230393650 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa88c00c0e0 con 0x7fa89013c340 2024-04-26T00:31:09.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.845+0000 7fa887fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:09.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.845+0000 7fa887fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:31:09.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.845+0000 7fa887fff640 10 monclient: dump: 2024-04-26T00:31:09.839 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:09.840 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:09.840 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:09.840 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:09.840 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:09.840 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:09.840 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:09.840 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:09.840 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:09.840 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:09.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.845+0000 7fa887fff640 1 -- 172.21.15.37:0/3230393650 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa88c0a9020 con 0x7fa89013c340 2024-04-26T00:31:09.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.845+0000 7fa887fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:31:09.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.845+0000 7fa89753f640 10 monclient: get_monmap_and_config success 2024-04-26T00:31:09.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.845+0000 7fa887fff640 1 -- 172.21.15.37:0/3230393650 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa88c005b20 con 0x7fa89013c340 2024-04-26T00:31:09.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.845+0000 7fa887fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:09.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.845+0000 7fa89753f640 4 set_mon_vals no callback set 2024-04-26T00:31:09.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.845+0000 7fa887fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:31:09.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.845+0000 7fa887fff640 10 monclient: dump: 2024-04-26T00:31:09.841 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:09.841 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:09.841 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:09.841 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:09.841 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:09.841 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:09.841 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:09.841 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:09.841 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:09.841 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:09.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.845+0000 7fa89753f640 10 monclient: shutdown 2024-04-26T00:31:09.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.845+0000 7fa89753f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:31:09.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.845+0000 7fa89753f640 1 -- 172.21.15.37:0/3230393650 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa89013c340 msgr2=0x7fa87c003a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:09.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.845+0000 7fa89753f640 1 --2- 172.21.15.37:0/3230393650 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa89013c340 0x7fa87c003a40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa88c001ec0 tx=0x7fa88c01cd50 comp rx=0 tx=0).stop 2024-04-26T00:31:09.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.845+0000 7fa895adc640 1 -- 172.21.15.37:0/3230393650 reap_dead start 2024-04-26T00:31:09.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.845+0000 7fa89753f640 1 -- 172.21.15.37:0/3230393650 shutdown_connections 2024-04-26T00:31:09.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.845+0000 7fa89753f640 1 -- 172.21.15.37:0/3230393650 >> 172.21.15.37:0/3230393650 conn(0x7fa8900a64e0 msgr2=0x7fa8900a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:31:09.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.845+0000 7fa89753f640 1 -- 172.21.15.37:0/3230393650 shutdown_connections 2024-04-26T00:31:09.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.845+0000 7fa89753f640 1 -- 172.21.15.37:0/3230393650 wait complete. 2024-04-26T00:31:09.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.845+0000 7fa89753f640 10 monclient: build_initial_monmap 2024-04-26T00:31:09.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.845+0000 7fa89753f640 10 monclient: monmap: 2024-04-26T00:31:09.842 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:31:09.842 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:31:09.842 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:31:09.842 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:31:09.842 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:31:09.842 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:31:09.842 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:31:09.843 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:31:09.843 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:31:09.843 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:09.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.845+0000 7fa89753f640 1 Processor -- start 2024-04-26T00:31:09.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa89753f640 1 -- start start 2024-04-26T00:31:09.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa89753f640 10 monclient: init 2024-04-26T00:31:09.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa89753f640 10 monclient: _reopen_session rank -1 2024-04-26T00:31:09.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa89753f640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:31:09.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa89753f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8901cb7f0 0x7fa8901344f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:09.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa89753f640 10 monclient(hunting): picked mon.noname-b con 0x7fa8901cb7f0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:31:09.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa89753f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8901ced50 0x7fa8901cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:09.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa89753f640 10 monclient(hunting): picked mon.noname-a con 0x7fa8901ced50 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:31:09.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa89753f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa8901cf6f0 0x7fa8901d60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:09.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa89753f640 10 monclient(hunting): picked mon.noname-c con 0x7fa8901cf6f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:31:09.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa89753f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:09.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa89753f640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa89013f2d0 con 0x7fa8901ced50 2024-04-26T00:31:09.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa89753f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:09.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa89753f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa89013f150 con 0x7fa8901cb7f0 2024-04-26T00:31:09.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa89753f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:09.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa89753f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa89013f450 con 0x7fa8901cf6f0 2024-04-26T00:31:09.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa89753f640 10 monclient(hunting): _renew_subs 2024-04-26T00:31:09.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa89753f640 10 monclient(hunting): authenticate will time out at 1651.157313s 2024-04-26T00:31:09.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa894ada640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8901ced50 0x7fa8901cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:09.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa894ada640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8901ced50 0x7fa8901cf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:55986/0 (socket says 172.21.15.37:55986) 2024-04-26T00:31:09.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa894ada640 1 -- 172.21.15.37:0/582197924 learned_addr learned my addr 172.21.15.37:0/582197924 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:31:09.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa894ada640 10 monclient(hunting): get_auth_request con 0x7fa8901ced50 auth_method 0 2024-04-26T00:31:09.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa894ada640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:09.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa894ada640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:09.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa894ada640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:09.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa894ada640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:09.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa894ada640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:09.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa894ada640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:09.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa895adc640 1 --2- 172.21.15.37:0/582197924 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa8901cf6f0 0x7fa8901d60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:09.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa895adc640 10 monclient(hunting): get_auth_request con 0x7fa8901cf6f0 auth_method 0 2024-04-26T00:31:09.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa895adc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:09.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa895adc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:09.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa895adc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:09.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa8952db640 1 --2- 172.21.15.37:0/582197924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8901cb7f0 0x7fa8901344f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:09.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa8952db640 10 monclient(hunting): get_auth_request con 0x7fa8901cb7f0 auth_method 0 2024-04-26T00:31:09.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa8952db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:09.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa8952db640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:09.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa8952db640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:09.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa894ada640 10 monclient(hunting): handle_auth_done global_id 124191 payload 824 2024-04-26T00:31:09.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa894ada640 1 -- 172.21.15.37:0/582197924 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa8901cf6f0 msgr2=0x7fa8901d60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:09.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa894ada640 1 --2- 172.21.15.37:0/582197924 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa8901cf6f0 0x7fa8901d60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:09.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa894ada640 1 -- 172.21.15.37:0/582197924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8901cb7f0 msgr2=0x7fa8901344f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:09.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa894ada640 1 --2- 172.21.15.37:0/582197924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8901cb7f0 0x7fa8901344f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:09.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa894ada640 10 monclient: _finish_hunting 0 2024-04-26T00:31:09.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa894ada640 1 monclient: found mon.noname-a 2024-04-26T00:31:09.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa894ada640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:31:09.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa894ada640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:31:09.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.849+0000 7fa894ada640 1 -- 172.21.15.37:0/582197924 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa89009aca0 con 0x7fa8901ced50 2024-04-26T00:31:09.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.853+0000 7fa894ada640 1 -- 172.21.15.37:0/582197924 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8901ced50 msgr2=0x7fa8901cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:09.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.853+0000 7fa894ada640 1 --2- 172.21.15.37:0/582197924 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8901ced50 0x7fa8901cf1b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa880011600 tx=0x7fa880011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:09.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.853+0000 7fa885ffb640 1 -- 172.21.15.37:0/582197924 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa880019070 con 0x7fa8901ced50 2024-04-26T00:31:09.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.853+0000 7fa885ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:09.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.853+0000 7fa885ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:31:09.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.853+0000 7fa885ffb640 10 monclient: dump: 2024-04-26T00:31:09.850 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:09.850 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:09.850 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:09.850 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:09.850 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:09.851 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:09.851 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:09.851 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:09.851 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:09.851 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:09.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.853+0000 7fa885ffb640 10 monclient: _finish_auth 0 2024-04-26T00:31:09.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.853+0000 7fa885ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:31:09.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.853+0000 7fa885ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:31:09.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.853+0000 7fa885ffb640 1 -- 172.21.15.37:0/582197924 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa88000fbf0 con 0x7fa8901ced50 2024-04-26T00:31:09.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.853+0000 7fa885ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:31:09.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.853+0000 7fa89753f640 5 monclient: authenticate success, global_id 124191 2024-04-26T00:31:09.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.853+0000 7fa885ffb640 1 -- 172.21.15.37:0/582197924 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa880015480 con 0x7fa8901ced50 2024-04-26T00:31:09.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.853+0000 7fa89753f640 10 monclient: _renew_subs 2024-04-26T00:31:09.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.853+0000 7fa89753f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:31:09.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.853+0000 7fa89753f640 1 -- 172.21.15.37:0/582197924 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa890120ea0 con 0x7fa8901ced50 2024-04-26T00:31:09.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.853+0000 7fa8877fe640 4 set_mon_vals no callback set 2024-04-26T00:31:09.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.853+0000 7fa885ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:09.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.853+0000 7fa885ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:31:09.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.853+0000 7fa885ffb640 10 monclient: dump: 2024-04-26T00:31:09.853 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:09.853 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:09.853 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:09.853 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:09.853 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:09.853 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:09.853 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:09.853 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:09.853 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:09.853 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:09.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.853+0000 7fa89753f640 10 monclient: _renew_subs 2024-04-26T00:31:09.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.853+0000 7fa89753f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:31:09.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.853+0000 7fa89753f640 1 -- 172.21.15.37:0/582197924 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa89011fc30 con 0x7fa8901ced50 2024-04-26T00:31:09.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.853+0000 7fa885ffb640 1 -- 172.21.15.37:0/582197924 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa88001c070 con 0x7fa8901ced50 2024-04-26T00:31:09.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.853+0000 7fa89753f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:09.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.853+0000 7fa89753f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:09.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.853+0000 7fa89753f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:31:09.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.853+0000 7fa89753f640 1 -- 172.21.15.37:0/582197924 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa890075910 con 0x7fa8901ced50 2024-04-26T00:31:09.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.853+0000 7fa885ffb640 1 --2- 172.21.15.37:0/582197924 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa874035e80 0x7fa874038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:09.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.853+0000 7fa885ffb640 1 -- 172.21.15.37:0/582197924 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(426..426 src has 370..426) ==== 3991+0+0 (secure 0 0 0) 0x7fa880048e90 con 0x7fa8901ced50 2024-04-26T00:31:09.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.853+0000 7fa8952db640 1 --2- 172.21.15.37:0/582197924 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa874035e80 0x7fa874038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:09.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.853+0000 7fa8952db640 10 monclient: get_auth_request con 0x7fa874035e80 auth_method 0 2024-04-26T00:31:09.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.857+0000 7fa8952db640 1 --2- 172.21.15.37:0/582197924 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa874035e80 0x7fa874038340 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa88c009000 tx=0x7fa88c001d20 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:09.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.857+0000 7fa885ffb640 1 -- 172.21.15.37:0/582197924 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa89009aca0 con 0x7fa8901ced50 2024-04-26T00:31:09.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.857+0000 7fa885ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:09.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:09.857+0000 7fa885ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:31:10.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.109+0000 7fa89753f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:31:10.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.109+0000 7fa89753f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:31:10.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.109+0000 7fa89753f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:31:10.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.109+0000 7fa89753f640 1 -- 172.21.15.37:0/582197924 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa8900a89f0 con 0x7fa8901ced50 2024-04-26T00:31:10.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.113+0000 7fa885ffb640 1 -- 172.21.15.37:0/582197924 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fa880047ba0 con 0x7fa8901ced50 2024-04-26T00:31:10.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.113+0000 7fa885ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:31:10.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.113+0000 7fa885ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:31:10.105 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":86,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:31:10.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.113+0000 7fa89753f640 1 -- 172.21.15.37:0/582197924 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa874035e80 msgr2=0x7fa874038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:10.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.113+0000 7fa89753f640 1 --2- 172.21.15.37:0/582197924 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa874035e80 0x7fa874038340 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa88c009000 tx=0x7fa88c001d20 comp rx=0 tx=0).stop 2024-04-26T00:31:10.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.113+0000 7fa89753f640 10 monclient: shutdown 2024-04-26T00:31:10.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.113+0000 7fa89753f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:31:10.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.113+0000 7fa89753f640 1 -- 172.21.15.37:0/582197924 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8901ced50 msgr2=0x7fa8901cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:10.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.113+0000 7fa89753f640 1 --2- 172.21.15.37:0/582197924 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8901ced50 0x7fa8901cf1b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa880011600 tx=0x7fa880011ac0 comp rx=0 tx=0).stop 2024-04-26T00:31:10.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.113+0000 7fa89753f640 1 -- 172.21.15.37:0/582197924 shutdown_connections 2024-04-26T00:31:10.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.113+0000 7fa89753f640 1 --2- 172.21.15.37:0/582197924 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa874035e80 0x7fa874038340 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:10.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.113+0000 7fa89753f640 1 --2- 172.21.15.37:0/582197924 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa8901cf6f0 0x7fa8901d60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:10.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.113+0000 7fa89753f640 1 --2- 172.21.15.37:0/582197924 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8901ced50 0x7fa8901cf1b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:10.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.113+0000 7fa89753f640 1 --2- 172.21.15.37:0/582197924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8901cb7f0 0x7fa8901344f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:10.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.113+0000 7fa89753f640 1 -- 172.21.15.37:0/582197924 >> 172.21.15.37:0/582197924 conn(0x7fa8900a64e0 msgr2=0x7fa890136150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:31:10.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.113+0000 7fa89753f640 1 -- 172.21.15.37:0/582197924 shutdown_connections 2024-04-26T00:31:10.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.113+0000 7fa89753f640 1 -- 172.21.15.37:0/582197924 wait complete. 2024-04-26T00:31:10.120 INFO:tasks.rados.rados.0.smithi045.stdout:1866: done (0 left) 2024-04-26T00:31:10.120 INFO:tasks.rados.rados.0.smithi045.stdout:1867: write oid 34 current snap is 213 2024-04-26T00:31:10.120 INFO:tasks.rados.rados.0.smithi045.stdout:1867: seq_num 682 ranges {662374=475235,1609190=676892,2812481=624101} 2024-04-26T00:31:10.127 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:31:10.127 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:31:10.127 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:31:10.139 INFO:tasks.rados.rados.0.smithi045.stdout:1867: writing smithi04526579-34 from 662374 to 1137609 tid 1 2024-04-26T00:31:10.156 INFO:tasks.rados.rados.0.smithi045.stdout:1867: writing smithi04526579-34 from 1609190 to 2286082 tid 2 2024-04-26T00:31:10.174 INFO:tasks.rados.rados.0.smithi045.stdout:1867: writing smithi04526579-34 from 2812481 to 3436582 tid 3 2024-04-26T00:31:10.174 INFO:tasks.rados.rados.0.smithi045.stdout:1868: write oid 47 current snap is 213 2024-04-26T00:31:10.174 INFO:tasks.rados.rados.0.smithi045.stdout:1867: finishing write tid 1 to smithi04526579-34 2024-04-26T00:31:10.174 INFO:tasks.rados.rados.0.smithi045.stdout:1867: finishing write tid 2 to smithi04526579-34 2024-04-26T00:31:10.174 INFO:tasks.rados.rados.0.smithi045.stdout:1868: seq_num 683 ranges {679181=550484,1667672=536903,2839904=435373} 2024-04-26T00:31:10.190 INFO:tasks.rados.rados.0.smithi045.stdout:1868: writing smithi04526579-47 from 679181 to 1229665 tid 1 2024-04-26T00:31:10.203 INFO:tasks.rados.rados.0.smithi045.stdout:1868: writing smithi04526579-47 from 1667672 to 2204575 tid 2 2024-04-26T00:31:10.217 INFO:tasks.rados.rados.0.smithi045.stdout:1868: writing smithi04526579-47 from 2839904 to 3275277 tid 3 2024-04-26T00:31:10.217 INFO:tasks.rados.rados.0.smithi045.stdout:1869: read oid 4 snap -1 2024-04-26T00:31:10.217 INFO:tasks.rados.rados.0.smithi045.stdout:1869: expect (ObjNum 406 snap 125 seq_num 406) 2024-04-26T00:31:10.217 INFO:tasks.rados.rados.0.smithi045.stdout:1870: write (excl) oid 48 current snap is 213 2024-04-26T00:31:10.217 INFO:tasks.rados.rados.0.smithi045.stdout:1870: seq_num 684 ranges {695988=625733,2009801=680561,3150974=345014} 2024-04-26T00:31:10.229 INFO:tasks.rados.rados.0.smithi045.stdout:1870: writing smithi04526579-48 from 695988 to 1321721 tid 1 2024-04-26T00:31:10.240 INFO:tasks.rados.rados.0.smithi045.stdout:1870: writing smithi04526579-48 from 2009801 to 2690362 tid 2 2024-04-26T00:31:10.247 INFO:tasks.rados.rados.0.smithi045.stdout:1870: writing smithi04526579-48 from 3150974 to 3495988 tid 3 2024-04-26T00:31:10.247 INFO:tasks.rados.rados.0.smithi045.stdout:1871: read oid 43 snap -1 2024-04-26T00:31:10.248 INFO:tasks.rados.rados.0.smithi045.stdout:1871: expect (ObjNum 646 snap 202 seq_num 646) 2024-04-26T00:31:10.248 INFO:tasks.rados.rados.0.smithi045.stdout:1872: rollback oid 3 current snap is 213 2024-04-26T00:31:10.248 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 3 to 211 2024-04-26T00:31:10.248 INFO:tasks.rados.rados.0.smithi045.stdout:1867: finishing write tid 3 to smithi04526579-34 2024-04-26T00:31:10.248 INFO:tasks.rados.rados.0.smithi045.stdout:1867: finishing write tid 4 to smithi04526579-34 2024-04-26T00:31:10.248 INFO:tasks.rados.rados.0.smithi045.stdout:1867: finishing write tid 5 to smithi04526579-34 2024-04-26T00:31:10.248 INFO:tasks.rados.rados.0.smithi045.stdout:1867: oid 34 updating version 0 to 1080 2024-04-26T00:31:10.248 INFO:tasks.rados.rados.0.smithi045.stdout:1867: oid 34 version 1080 is already newer than 1078 2024-04-26T00:31:10.248 INFO:tasks.rados.rados.0.smithi045.stdout:1867: oid 34 version 1080 is already newer than 1079 2024-04-26T00:31:10.248 INFO:tasks.rados.rados.0.smithi045.stdout:1867: oid 34 updating version 1080 to 1081 2024-04-26T00:31:10.248 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 1081 (ObjNum 682 snap 213 seq_num 682) dirty exists 2024-04-26T00:31:10.248 INFO:tasks.rados.rados.0.smithi045.stdout:1867: left oid 34 (ObjNum 682 snap 213 seq_num 682) 2024-04-26T00:31:10.248 INFO:tasks.rados.rados.0.smithi045.stdout:1868: finishing write tid 1 to smithi04526579-47 2024-04-26T00:31:10.249 INFO:tasks.rados.rados.0.smithi045.stdout:1868: finishing write tid 2 to smithi04526579-47 2024-04-26T00:31:10.249 INFO:tasks.rados.rados.0.smithi045.stdout:1868: finishing write tid 3 to smithi04526579-47 2024-04-26T00:31:10.249 INFO:tasks.rados.rados.0.smithi045.stdout:1868: finishing write tid 4 to smithi04526579-47 2024-04-26T00:31:10.249 INFO:tasks.rados.rados.0.smithi045.stdout:1868: finishing write tid 5 to smithi04526579-47 2024-04-26T00:31:10.249 INFO:tasks.rados.rados.0.smithi045.stdout:1868: oid 47 updating version 0 to 1083 2024-04-26T00:31:10.249 INFO:tasks.rados.rados.0.smithi045.stdout:1868: oid 47 updating version 1083 to 1084 2024-04-26T00:31:10.249 INFO:tasks.rados.rados.0.smithi045.stdout:1868: oid 47 updating version 1084 to 1085 2024-04-26T00:31:10.249 INFO:tasks.rados.rados.0.smithi045.stdout:1868: oid 47 version 1085 is already newer than 1082 2024-04-26T00:31:10.249 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 1085 (ObjNum 683 snap 213 seq_num 683) dirty exists 2024-04-26T00:31:10.249 INFO:tasks.rados.rados.0.smithi045.stdout:1868: left oid 47 (ObjNum 683 snap 213 seq_num 683) 2024-04-26T00:31:10.249 INFO:tasks.rados.rados.0.smithi045.stdout:1869: expect (ObjNum 406 snap 125 seq_num 406) 2024-04-26T00:31:10.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.257+0000 7f9300e30640 10 monclient: get_monmap_and_config 2024-04-26T00:31:10.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.257+0000 7f9300e30640 10 monclient: build_initial_monmap 2024-04-26T00:31:10.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.257+0000 7f9300e30640 10 monclient: monmap: 2024-04-26T00:31:10.250 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:31:10.250 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:10.250 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:31:10.261489+0000 2024-04-26T00:31:10.250 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:31:10.261489+0000 2024-04-26T00:31:10.250 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:31:10.251 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:10.251 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:31:10.251 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:31:10.251 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:31:10.251 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:31:10.251 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:31:10.251 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:10.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.257+0000 7f9300e30640 1 Processor -- start 2024-04-26T00:31:10.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.257+0000 7f9300e30640 1 -- start start 2024-04-26T00:31:10.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.257+0000 7f9300e30640 10 monclient: init 2024-04-26T00:31:10.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f9300e30640 10 monclient: _reopen_session rank -1 2024-04-26T00:31:10.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f9300e30640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-04-26T00:31:10.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f9300e30640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f92fc09c780 0x7f92fc1350b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:10.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f9300e30640 10 monclient(hunting): picked mon.noname-b con 0x7f92fc09c780 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:31:10.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f9300e30640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f92fc09cb60 0x7f92fc0a1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:10.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f9300e30640 10 monclient(hunting): picked mon.noname-a con 0x7f92fc09cb60 addr v2:172.21.15.37:3300/0 2024-04-26T00:31:10.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f9300e30640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f92fc0a20a0 addr v1:172.21.15.37:6789/0 2024-04-26T00:31:10.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f9300e30640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92fc06b560 con 0x7f92fc0a20a0 2024-04-26T00:31:10.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f9300e30640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:10.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f9300e30640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f92fc135b50 con 0x7f92fc09cb60 2024-04-26T00:31:10.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f9300e30640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:10.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f9300e30640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f92fc1355f0 con 0x7f92fc09c780 2024-04-26T00:31:10.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f9300e30640 10 monclient(hunting): _renew_subs 2024-04-26T00:31:10.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f9300e30640 10 monclient(hunting): authenticate will time out at 1651.568346s 2024-04-26T00:31:10.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f92fa59c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f92fc09c780 0x7f92fc1350b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:10.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f92fa59c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f92fc09c780 0x7f92fc1350b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:33606/0 (socket says 172.21.15.37:33606) 2024-04-26T00:31:10.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f92fa59c640 1 -- 172.21.15.37:0/2691492460 learned_addr learned my addr 172.21.15.37:0/2691492460 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:31:10.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f92fa59c640 10 monclient(hunting): get_auth_request con 0x7f92fc09c780 auth_method 0 2024-04-26T00:31:10.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f92fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:10.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f92fa59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:10.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f92fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:10.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f92f9d9b640 1 --2- 172.21.15.37:0/2691492460 >> v2:172.21.15.37:3300/0 conn(0x7f92fc09cb60 0x7f92fc0a1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:10.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f92f9d9b640 10 monclient(hunting): get_auth_request con 0x7f92fc09cb60 auth_method 0 2024-04-26T00:31:10.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f92f9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:10.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f92f9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:10.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f92f9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:10.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f92fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:10.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f92fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:10.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f92fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:10.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f92f9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:10.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f92f9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:10.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f92f9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:10.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f92fa59c640 10 monclient(hunting): handle_auth_done global_id 84551 payload 274 2024-04-26T00:31:10.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f92fa59c640 1 -- 172.21.15.37:0/2691492460 >> v2:172.21.15.37:3300/0 conn(0x7f92fc09cb60 msgr2=0x7f92fc0a1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:31:10.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f92fa59c640 1 --2- 172.21.15.37:0/2691492460 >> v2:172.21.15.37:3300/0 conn(0x7f92fc09cb60 0x7f92fc0a1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:10.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f92fa59c640 1 -- 172.21.15.37:0/2691492460 >> v1:172.21.15.37:6789/0 conn(0x7f92fc0a20a0 legacy=0x7f92fc13bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:31:10.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f92fa59c640 10 monclient: _finish_hunting 0 2024-04-26T00:31:10.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f92fa59c640 1 monclient: found mon.noname-b 2024-04-26T00:31:10.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f92fa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:31:10.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f92fa59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:31:10.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.261+0000 7f92fa59c640 1 -- 172.21.15.37:0/2691492460 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92fc06b1d0 con 0x7f92fc09c780 2024-04-26T00:31:10.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f92fa59c640 1 -- 172.21.15.37:0/2691492460 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f92fc09c780 msgr2=0x7f92fc1350b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:10.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f92fa59c640 1 --2- 172.21.15.37:0/2691492460 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f92fc09c780 0x7f92fc1350b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f92ec009870 tx=0x7f92ec01c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3123b4f9f9eef3f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:10.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f92f959a640 1 -- 172.21.15.37:0/2691492460 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92ec0b0070 con 0x7f92fc09c780 2024-04-26T00:31:10.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f92f959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:10.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f92f959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:31:10.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f92f959a640 10 monclient: dump: 2024-04-26T00:31:10.262 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:10.262 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:10.262 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:10.262 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:10.262 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:10.262 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:10.262 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:10.263 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:10.263 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:10.263 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:10.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f92f959a640 10 monclient: _finish_auth 0 2024-04-26T00:31:10.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f92f959a640 10 monclient: _check_auth_tickets 2024-04-26T00:31:10.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f92f959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:31:10.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f9300e30640 5 monclient: authenticate success, global_id 84551 2024-04-26T00:31:10.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f9300e30640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:31:10.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f92f959a640 1 -- 172.21.15.37:0/2691492460 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92ec0ab600 con 0x7f92fc09c780 2024-04-26T00:31:10.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f92f959a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:31:10.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f9300e30640 10 monclient: get_monmap_and_config success 2024-04-26T00:31:10.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f9300e30640 4 set_mon_vals no callback set 2024-04-26T00:31:10.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f92f959a640 1 -- 172.21.15.37:0/2691492460 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92ec0ab9d0 con 0x7f92fc09c780 2024-04-26T00:31:10.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f92f959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:10.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f92f959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:31:10.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f92f959a640 10 monclient: dump: 2024-04-26T00:31:10.264 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:10.264 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:10.264 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:10.264 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:10.264 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:10.264 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:10.264 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:10.264 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:10.264 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:10.264 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:10.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f9300e30640 10 monclient: shutdown 2024-04-26T00:31:10.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f9300e30640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:31:10.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f9300e30640 1 -- 172.21.15.37:0/2691492460 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f92fc09c780 msgr2=0x7f92fc1350b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:10.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f9300e30640 1 --2- 172.21.15.37:0/2691492460 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f92fc09c780 0x7f92fc1350b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f92ec009870 tx=0x7f92ec01c440 comp rx=0 tx=0).stop 2024-04-26T00:31:10.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f9300e30640 1 -- 172.21.15.37:0/2691492460 shutdown_connections 2024-04-26T00:31:10.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f9300e30640 1 --2- 172.21.15.37:0/2691492460 >> v2:172.21.15.37:3300/0 conn(0x7f92fc09cb60 0x7f92fc0a1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:10.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f9300e30640 1 --2- 172.21.15.37:0/2691492460 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f92fc09c780 0x7f92fc1350b0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:10.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f9300e30640 1 -- 172.21.15.37:0/2691492460 >> 172.21.15.37:0/2691492460 conn(0x7f92fc0a2e20 msgr2=0x7f92fc0a3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:31:10.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f9300e30640 1 -- 172.21.15.37:0/2691492460 shutdown_connections 2024-04-26T00:31:10.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f9300e30640 1 -- 172.21.15.37:0/2691492460 wait complete. 2024-04-26T00:31:10.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f9300e30640 10 monclient: build_initial_monmap 2024-04-26T00:31:10.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f9300e30640 10 monclient: monmap: 2024-04-26T00:31:10.265 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:31:10.265 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:31:10.265 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:31:10.265 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:31:10.266 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:31:10.266 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:31:10.266 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:31:10.266 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:31:10.266 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:31:10.266 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:10.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f9300e30640 1 Processor -- start 2024-04-26T00:31:10.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f9300e30640 1 -- start start 2024-04-26T00:31:10.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f9300e30640 10 monclient: init 2024-04-26T00:31:10.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f9300e30640 10 monclient: _reopen_session rank -1 2024-04-26T00:31:10.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f9300e30640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:31:10.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f9300e30640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f92fc09c780 0x7f92fc072310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:10.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f9300e30640 10 monclient(hunting): picked mon.noname-a con 0x7f92fc09c780 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:31:10.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f9300e30640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f92fc09cb60 0x7f92fc13c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:10.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f9300e30640 10 monclient(hunting): picked mon.noname-c con 0x7f92fc09cb60 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:31:10.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f9300e30640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92fc0a20a0 0x7f92fc09d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:10.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f9300e30640 10 monclient(hunting): picked mon.noname-b con 0x7f92fc0a20a0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:31:10.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f9300e30640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:10.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f9300e30640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f92fc09d0c0 con 0x7f92fc09c780 2024-04-26T00:31:10.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f9300e30640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:10.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f9300e30640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f92fc09cf40 con 0x7f92fc0a20a0 2024-04-26T00:31:10.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f9300e30640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:10.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f9300e30640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f92fc13fe60 con 0x7f92fc09cb60 2024-04-26T00:31:10.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f9300e30640 10 monclient(hunting): _renew_subs 2024-04-26T00:31:10.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f9300e30640 10 monclient(hunting): authenticate will time out at 1651.575549s 2024-04-26T00:31:10.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f92f9d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f92fc09cb60 0x7f92fc13c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:10.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f92fa59c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f92fc09c780 0x7f92fc072310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:10.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f92fa59c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f92fc09c780 0x7f92fc072310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:55996/0 (socket says 172.21.15.37:55996) 2024-04-26T00:31:10.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f92fa59c640 1 -- 172.21.15.37:0/1959776260 learned_addr learned my addr 172.21.15.37:0/1959776260 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:31:10.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f92f9d9b640 10 monclient(hunting): get_auth_request con 0x7f92fc09cb60 auth_method 0 2024-04-26T00:31:10.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f92f9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:10.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f92f9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:10.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f92f9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:10.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f92fa59c640 10 monclient(hunting): get_auth_request con 0x7f92fc09c780 auth_method 0 2024-04-26T00:31:10.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f92fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:10.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f92fa59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:10.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f92fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:10.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f92fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:10.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f92fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:10.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f92fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:10.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f92f9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:10.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f92f9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:10.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f92f9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:10.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f92fad9d640 1 --2- 172.21.15.37:0/1959776260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92fc0a20a0 0x7f92fc09d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:10.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f92fad9d640 10 monclient(hunting): get_auth_request con 0x7f92fc0a20a0 auth_method 0 2024-04-26T00:31:10.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f92fad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:10.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f92fad9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:10.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f92fad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:10.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.265+0000 7f92fa59c640 10 monclient(hunting): handle_auth_done global_id 124197 payload 824 2024-04-26T00:31:10.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f92fa59c640 1 -- 172.21.15.37:0/1959776260 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f92fc09cb60 msgr2=0x7f92fc13c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:10.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f92fa59c640 1 --2- 172.21.15.37:0/1959776260 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f92fc09cb60 0x7f92fc13c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:10.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f92fa59c640 1 -- 172.21.15.37:0/1959776260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92fc0a20a0 msgr2=0x7f92fc09d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:10.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f92fa59c640 1 --2- 172.21.15.37:0/1959776260 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92fc0a20a0 0x7f92fc09d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:10.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f92fa59c640 10 monclient: _finish_hunting 0 2024-04-26T00:31:10.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f92fa59c640 1 monclient: found mon.noname-a 2024-04-26T00:31:10.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f92fa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:31:10.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f92fa59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:31:10.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f92fa59c640 1 -- 172.21.15.37:0/1959776260 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92fc072ee0 con 0x7f92fc09c780 2024-04-26T00:31:10.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f92fa59c640 1 -- 172.21.15.37:0/1959776260 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f92fc09c780 msgr2=0x7f92fc072310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:10.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f92fa59c640 1 --2- 172.21.15.37:0/1959776260 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f92fc09c780 0x7f92fc072310 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f92ec0ae040 tx=0x7f92ec01ac80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:10.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f92eaffd640 1 -- 172.21.15.37:0/1959776260 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92ec0ba020 con 0x7f92fc09c780 2024-04-26T00:31:10.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f92eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:10.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f92eaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:31:10.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f92eaffd640 10 monclient: dump: 2024-04-26T00:31:10.274 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:10.274 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:10.274 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:10.275 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:10.275 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:10.275 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:10.275 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:10.275 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:10.275 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:10.275 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:10.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f92eaffd640 10 monclient: _finish_auth 0 2024-04-26T00:31:10.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f92eaffd640 10 monclient: _check_auth_tickets 2024-04-26T00:31:10.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f92eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:31:10.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f9300e30640 5 monclient: authenticate success, global_id 124197 2024-04-26T00:31:10.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f92eaffd640 1 -- 172.21.15.37:0/1959776260 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92ec0b6630 con 0x7f92fc09c780 2024-04-26T00:31:10.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f9300e30640 10 monclient: _renew_subs 2024-04-26T00:31:10.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f9300e30640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:31:10.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f9300e30640 1 -- 172.21.15.37:0/1959776260 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92fc0a0360 con 0x7f92fc09c780 2024-04-26T00:31:10.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f92eaffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:31:10.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f92eaffd640 1 -- 172.21.15.37:0/1959776260 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92ec0b0040 con 0x7f92fc09c780 2024-04-26T00:31:10.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f92eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:10.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f92eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:31:10.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f92eaffd640 10 monclient: dump: 2024-04-26T00:31:10.276 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:10.276 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:10.276 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:10.276 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:10.276 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:10.276 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:10.276 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:10.277 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:10.277 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:10.277 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:10.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f92f8d99640 4 set_mon_vals no callback set 2024-04-26T00:31:10.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f9300e30640 10 monclient: _renew_subs 2024-04-26T00:31:10.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f9300e30640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:31:10.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f9300e30640 1 -- 172.21.15.37:0/1959776260 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92fc09a910 con 0x7f92fc09c780 2024-04-26T00:31:10.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f9300e30640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:10.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f9300e30640 1 --2- 172.21.15.37:0/1959776260 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f92fc13c6c0 0x7f92fc09db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:10.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f9300e30640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:10.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f9300e30640 1 -- 172.21.15.37:0/1959776260 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f92fc04f510 con 0x7f92fc13c6c0 2024-04-26T00:31:10.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f92eaffd640 1 -- 172.21.15.37:0/1959776260 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f92ec0bd020 con 0x7f92fc09c780 2024-04-26T00:31:10.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f92f9d9b640 1 --2- 172.21.15.37:0/1959776260 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f92fc13c6c0 0x7f92fc09db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:10.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f92f9d9b640 10 monclient: get_auth_request con 0x7f92fc13c6c0 auth_method 0 2024-04-26T00:31:10.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f92f9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:10.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f92f9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:10.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f92f9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:10.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f92f9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:10.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f92f9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:10.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f92f9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:10.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f92eaffd640 1 --2- 172.21.15.37:0/1959776260 >> v2:172.21.15.45:6812/2379918623 conn(0x7f92dc035e80 0x7f92dc038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:10.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f92eaffd640 1 -- 172.21.15.37:0/1959776260 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(427..427 src has 370..427) ==== 4035+0+0 (secure 0 0 0) 0x7f92ec0e2c50 con 0x7f92fc09c780 2024-04-26T00:31:10.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f92f9d9b640 10 monclient(hunting): handle_auth_done global_id 124200 payload 824 2024-04-26T00:31:10.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.269+0000 7f92f9d9b640 1 -- 172.21.15.37:0/1959776260 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f92fc1cffe0 con 0x7f92fc13c6c0 2024-04-26T00:31:10.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.273+0000 7f92fad9d640 1 --2- 172.21.15.37:0/1959776260 >> v2:172.21.15.45:6812/2379918623 conn(0x7f92dc035e80 0x7f92dc038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:10.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.273+0000 7f92fad9d640 10 monclient: get_auth_request con 0x7f92dc035e80 auth_method 0 2024-04-26T00:31:10.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.273+0000 7f92f9d9b640 1 -- 172.21.15.37:0/1959776260 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f92fc13c6c0 msgr2=0x7f92fc09db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:10.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.273+0000 7f92f9d9b640 1 --2- 172.21.15.37:0/1959776260 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f92fc13c6c0 0x7f92fc09db90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f92fc072a00 tx=0x7f92f0011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:10.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.273+0000 7f92eaffd640 1 -- 172.21.15.37:0/1959776260 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f0019070 con 0x7f92fc13c6c0 2024-04-26T00:31:10.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.273+0000 7f92eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:10.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.273+0000 7f92eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:31:10.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.273+0000 7f92eaffd640 10 monclient: dump: 2024-04-26T00:31:10.279 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:10.279 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:10.279 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:10.279 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:10.279 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:10.279 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:10.279 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:10.279 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:10.279 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:10.279 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:10.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.273+0000 7f92eaffd640 1 -- 172.21.15.37:0/1959776260 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f92f000f860 con 0x7f92fc13c6c0 2024-04-26T00:31:10.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.273+0000 7f92eaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:10.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.273+0000 7f92eaffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:31:10.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.273+0000 7f92eaffd640 1 -- 172.21.15.37:0/1959776260 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f92fc13c6c0 msgr2=0x7f92fc09db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:10.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.273+0000 7f92eaffd640 1 --2- 172.21.15.37:0/1959776260 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f92fc13c6c0 0x7f92fc09db90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f92fc072a00 tx=0x7f92f0011d60 comp rx=0 tx=0).stop 2024-04-26T00:31:10.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.273+0000 7f92eaffd640 1 -- 172.21.15.37:0/1959776260 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f92fc13c6c0 msgr2=0x7f92fc09db90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:31:10.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.273+0000 7f92eaffd640 1 --2- 172.21.15.37:0/1959776260 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f92fc13c6c0 0x7f92fc09db90 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f92fc072a00 tx=0x7f92f0011d60 comp rx=0 tx=0).stop 2024-04-26T00:31:10.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.273+0000 7f92fad9d640 1 --2- 172.21.15.37:0/1959776260 >> v2:172.21.15.45:6812/2379918623 conn(0x7f92dc035e80 0x7f92dc038340 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f92f4005550 tx=0x7f92f4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:10.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.281+0000 7f9300e30640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:31:10.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.281+0000 7f9300e30640 1 --2- 172.21.15.37:0/1959776260 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f92fc13a8d0 0x7f92fc09e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:10.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.281+0000 7f9300e30640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:10.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.281+0000 7f9300e30640 1 -- 172.21.15.37:0/1959776260 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f92fc06ca10 con 0x7f92fc13a8d0 2024-04-26T00:31:10.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.281+0000 7f92f9d9b640 1 --2- 172.21.15.37:0/1959776260 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f92fc13a8d0 0x7f92fc09e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:10.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.281+0000 7f92f9d9b640 10 monclient: get_auth_request con 0x7f92fc13a8d0 auth_method 0 2024-04-26T00:31:10.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.281+0000 7f92f9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:10.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.281+0000 7f92f9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:10.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.281+0000 7f92f9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:10.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.281+0000 7f92f9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:10.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.281+0000 7f92f9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:10.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.281+0000 7f92f9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:10.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.281+0000 7f92f9d9b640 10 monclient(hunting): handle_auth_done global_id 124203 payload 824 2024-04-26T00:31:10.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.281+0000 7f92f9d9b640 1 -- 172.21.15.37:0/1959776260 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f92fc1cf5f0 con 0x7f92fc13a8d0 2024-04-26T00:31:10.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.281+0000 7f92f9d9b640 1 -- 172.21.15.37:0/1959776260 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f92fc13a8d0 msgr2=0x7f92fc09e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:10.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.281+0000 7f92f9d9b640 1 --2- 172.21.15.37:0/1959776260 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f92fc13a8d0 0x7f92fc09e0d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f92fc072a00 tx=0x7f92f00046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:10.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.281+0000 7f92eaffd640 1 -- 172.21.15.37:0/1959776260 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f0019040 con 0x7f92fc13a8d0 2024-04-26T00:31:10.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.281+0000 7f92eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:10.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.281+0000 7f92eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:31:10.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.281+0000 7f92eaffd640 10 monclient: dump: 2024-04-26T00:31:10.282 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:10.282 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:10.282 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:10.282 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:10.283 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:10.283 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:10.283 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:10.283 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:10.283 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:10.283 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:10.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.281+0000 7f92eaffd640 1 -- 172.21.15.37:0/1959776260 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7f92f0003e10 con 0x7f92fc13a8d0 2024-04-26T00:31:10.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.281+0000 7f92eaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:31:10.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.281+0000 7f92eaffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:31:10.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.285+0000 7f92eaffd640 1 -- 172.21.15.37:0/1959776260 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f92fc13a8d0 msgr2=0x7f92fc09e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:10.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.285+0000 7f92eaffd640 1 --2- 172.21.15.37:0/1959776260 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f92fc13a8d0 0x7f92fc09e0d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f92fc072a00 tx=0x7f92f00046d0 comp rx=0 tx=0).stop 2024-04-26T00:31:10.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.285+0000 7f92eaffd640 1 -- 172.21.15.37:0/1959776260 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f92fc13a8d0 msgr2=0x7f92fc09e0d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:31:10.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.285+0000 7f92eaffd640 1 --2- 172.21.15.37:0/1959776260 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f92fc13a8d0 0x7f92fc09e0d0 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f92fc072a00 tx=0x7f92f00046d0 comp rx=0 tx=0).stop 2024-04-26T00:31:10.284 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:31:10.284 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:31:10.284 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:31:10.284 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:31:10.284 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 86, 2024-04-26T00:31:10.284 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 59159, 2024-04-26T00:31:10.284 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:31:10.284 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:31:10.284 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:31:10.284 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:31:10.284 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:31:10.285 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:31:10.285 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:31:10.285 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:31:10.285 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:31:10.285 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:31:10.285 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:31:10.285 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:31:10.285 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:31:10.285 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:31:10.285 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:31:10.285 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:31:10.285 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:31:10.285 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:31:10.285 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:31:10.285 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:31:10.286 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:31:10.286 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:31:10.286 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:31:10.286 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:31:10.286 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:31:10.286 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:31:10.286 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:31:10.286 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:31:10.286 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:31:10.286 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:31:10.286 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:31:10.286 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:31:10.286 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:31:10.286 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:31:10.287 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:10.287 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:10.287 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:31:10.287 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:31:10.287 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:31:10.287 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:31:10.287 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:31:10.287 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:31:10.287 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:31:10.288 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:31:10.288 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:31:10.288 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:31:10.288 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:31:10.288 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:31:10.288 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:31:10.288 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:31:10.288 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:31:10.288 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:31:10.288 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:31:10.288 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:31:10.288 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:31:10.288 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:31:10.289 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:31:10.289 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:31:10.289 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:31:10.289 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:31:10.289 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:31:10.289 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:31:10.289 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:31:10.289 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:31:10.289 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:31:10.289 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:31:10.289 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:10.289 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:31:10.289 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:10.289 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:31:10.290 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:31:10.290 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:31:10.290 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:31:10.290 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:10.290 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:31:10.290 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:31:10.290 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:10.290 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:10.290 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:10.290 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:31:10.290 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:31:10.290 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:10.290 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:10.290 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:10.291 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:10.291 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:31:10.291 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:31:10.291 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:31:10.291 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:31:10.291 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:31:10.291 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:10.291 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:10.291 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:31:10.291 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:31:10.291 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:31:10.291 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:31:10.292 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:10.292 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:31:10.292 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:31:10.292 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:10.292 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:10.292 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:10.292 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:31:10.292 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:31:10.292 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:10.292 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:10.292 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:10.292 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:10.292 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:31:10.292 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:31:10.293 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:31:10.293 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:31:10.293 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:31:10.293 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:10.293 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:10.293 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:31:10.293 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:31:10.293 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:31:10.293 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:31:10.293 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:10.293 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:31:10.293 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:31:10.293 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:10.293 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:10.294 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:10.294 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:31:10.294 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:31:10.294 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:10.294 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:10.294 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:10.294 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:10.294 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:31:10.294 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:31:10.294 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:31:10.294 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:31:10.294 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:31:10.294 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:10.295 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:10.295 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:10.295 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:31:10.295 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:31:10.295 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:10.295 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:31:10.295 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:31:10.295 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:31:10.295 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:10.295 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:31:10.295 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:31:10.295 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:10.295 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:31:10.295 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:31:10.296 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:31:10.296 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:10.296 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:10.296 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:10.296 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:31:10.296 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:31:10.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.285+0000 7f9300e30640 1 -- 172.21.15.37:0/1959776260 >> v2:172.21.15.45:6812/2379918623 conn(0x7f92dc035e80 msgr2=0x7f92dc038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:10.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.285+0000 7f9300e30640 1 --2- 172.21.15.37:0/1959776260 >> v2:172.21.15.45:6812/2379918623 conn(0x7f92dc035e80 0x7f92dc038340 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f92f4005550 tx=0x7f92f4009040 comp rx=0 tx=0).stop 2024-04-26T00:31:10.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.285+0000 7f9300e30640 10 monclient: shutdown 2024-04-26T00:31:10.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.285+0000 7f9300e30640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:31:10.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.285+0000 7f9300e30640 1 -- 172.21.15.37:0/1959776260 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f92fc09c780 msgr2=0x7f92fc072310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:10.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.285+0000 7f9300e30640 1 --2- 172.21.15.37:0/1959776260 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f92fc09c780 0x7f92fc072310 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f92ec0ae040 tx=0x7f92ec01ac80 comp rx=0 tx=0).stop 2024-04-26T00:31:10.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.285+0000 7f92fad9d640 1 -- 172.21.15.37:0/1959776260 reap_dead start 2024-04-26T00:31:10.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.285+0000 7f92fad9d640 1 -- 172.21.15.37:0/1959776260 reap_dead start 2024-04-26T00:31:10.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.285+0000 7f9300e30640 1 -- 172.21.15.37:0/1959776260 shutdown_connections 2024-04-26T00:31:10.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.285+0000 7f9300e30640 1 -- 172.21.15.37:0/1959776260 >> 172.21.15.37:0/1959776260 conn(0x7f92fc0a2e20 msgr2=0x7f92fc09ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:31:10.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.285+0000 7f9300e30640 1 -- 172.21.15.37:0/1959776260 shutdown_connections 2024-04-26T00:31:10.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.285+0000 7f9300e30640 1 -- 172.21.15.37:0/1959776260 wait complete. 2024-04-26T00:31:10.298 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:31:10.299 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:31:10.299 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:31:10.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.421+0000 7f649faaa640 10 monclient: get_monmap_and_config 2024-04-26T00:31:10.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.421+0000 7f649faaa640 10 monclient: build_initial_monmap 2024-04-26T00:31:10.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.421+0000 7f649faaa640 10 monclient: monmap: 2024-04-26T00:31:10.415 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:31:10.415 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:10.415 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:31:10.425907+0000 2024-04-26T00:31:10.415 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:31:10.425907+0000 2024-04-26T00:31:10.415 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:31:10.415 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:10.415 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:31:10.415 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:31:10.415 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:31:10.415 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:31:10.415 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:31:10.415 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:10.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.421+0000 7f649faaa640 1 Processor -- start 2024-04-26T00:31:10.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.421+0000 7f649faaa640 1 -- start start 2024-04-26T00:31:10.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.421+0000 7f649faaa640 10 monclient: init 2024-04-26T00:31:10.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.421+0000 7f649faaa640 10 monclient: _reopen_session rank -1 2024-04-26T00:31:10.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.421+0000 7f649faaa640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-04-26T00:31:10.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.421+0000 7f649faaa640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6498136f20 0x7f649809bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:10.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.421+0000 7f649faaa640 10 monclient(hunting): picked mon.noname-a con 0x7f6498136f20 addr v2:172.21.15.37:3300/0 2024-04-26T00:31:10.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.421+0000 7f649faaa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6498137300 addr v1:172.21.15.45:6789/0 2024-04-26T00:31:10.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.421+0000 7f649faaa640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f649813c450 0x7f649813e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:10.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.421+0000 7f649faaa640 10 monclient(hunting): picked mon.noname-b con 0x7f649813c450 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:31:10.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.421+0000 7f649faaa640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64980a50f0 con 0x7f6498137300 2024-04-26T00:31:10.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.421+0000 7f649faaa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:10.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.421+0000 7f649faaa640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f649809b330 con 0x7f6498136f20 2024-04-26T00:31:10.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.421+0000 7f649faaa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:10.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.421+0000 7f649faaa640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f649809b610 con 0x7f649813c450 2024-04-26T00:31:10.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.421+0000 7f649faaa640 10 monclient(hunting): _renew_subs 2024-04-26T00:31:10.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.421+0000 7f649faaa640 10 monclient(hunting): authenticate will time out at 1651.731783s 2024-04-26T00:31:10.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.421+0000 7f649e047640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f649813c450 0x7f649813e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:10.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.421+0000 7f649d846640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6498136f20 0x7f649809bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:10.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.421+0000 7f649e047640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f649813c450 0x7f649813e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:33616/0 (socket says 172.21.15.37:33616) 2024-04-26T00:31:10.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.425+0000 7f649e047640 1 -- 172.21.15.37:0/721579500 learned_addr learned my addr 172.21.15.37:0/721579500 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:31:10.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.425+0000 7f649e047640 10 monclient(hunting): get_auth_request con 0x7f649813c450 auth_method 0 2024-04-26T00:31:10.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.425+0000 7f649d846640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6498136f20 0x7f649809bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:56026/0 (socket says 172.21.15.37:56026) 2024-04-26T00:31:10.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.425+0000 7f649e047640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:10.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.425+0000 7f649e047640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:10.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.425+0000 7f649e047640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:10.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.425+0000 7f649d846640 10 monclient(hunting): get_auth_request con 0x7f6498136f20 auth_method 0 2024-04-26T00:31:10.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.425+0000 7f649d846640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:10.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.425+0000 7f649d846640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:10.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.425+0000 7f649d846640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:10.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.425+0000 7f649e047640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:10.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.425+0000 7f649e047640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:10.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.425+0000 7f649e047640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:10.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.425+0000 7f649d846640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:10.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.425+0000 7f649d846640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:10.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.425+0000 7f649d846640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:10.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.425+0000 7f649e047640 10 monclient(hunting): handle_auth_done global_id 84557 payload 274 2024-04-26T00:31:10.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.425+0000 7f649e047640 1 -- 172.21.15.37:0/721579500 >> v2:172.21.15.37:3300/0 conn(0x7f6498136f20 msgr2=0x7f649809bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:31:10.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.425+0000 7f649e047640 1 --2- 172.21.15.37:0/721579500 >> v2:172.21.15.37:3300/0 conn(0x7f6498136f20 0x7f649809bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:10.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.425+0000 7f649e047640 1 -- 172.21.15.37:0/721579500 >> v1:172.21.15.45:6789/0 conn(0x7f6498137300 legacy=0x7f64981377b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:31:10.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.425+0000 7f649e047640 10 monclient: _finish_hunting 0 2024-04-26T00:31:10.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.425+0000 7f649e047640 1 monclient: found mon.noname-b 2024-04-26T00:31:10.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.425+0000 7f649e047640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:31:10.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.425+0000 7f649e047640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:31:10.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.425+0000 7f649e047640 1 -- 172.21.15.37:0/721579500 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f649809b8f0 con 0x7f649813c450 2024-04-26T00:31:10.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649e047640 1 -- 172.21.15.37:0/721579500 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f649813c450 msgr2=0x7f649813e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:10.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649e047640 1 --2- 172.21.15.37:0/721579500 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f649813c450 0x7f649813e840 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6490009e50 tx=0x7f649001c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c71e86c62df1078 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:10.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649c844640 1 -- 172.21.15.37:0/721579500 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64900a55c0 con 0x7f649813c450 2024-04-26T00:31:10.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649c844640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:10.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649c844640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:31:10.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649c844640 10 monclient: dump: 2024-04-26T00:31:10.427 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:10.427 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:10.427 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:10.427 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:10.427 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:10.427 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:10.427 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:10.427 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:10.427 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:10.428 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:10.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649c844640 10 monclient: _finish_auth 0 2024-04-26T00:31:10.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649c844640 10 monclient: _check_auth_tickets 2024-04-26T00:31:10.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649c844640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:31:10.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649faaa640 5 monclient: authenticate success, global_id 84557 2024-04-26T00:31:10.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649c844640 1 -- 172.21.15.37:0/721579500 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64900a5760 con 0x7f649813c450 2024-04-26T00:31:10.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649faaa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:31:10.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649c844640 10 monclient: handle_config config(0 keys) 2024-04-26T00:31:10.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649faaa640 10 monclient: get_monmap_and_config success 2024-04-26T00:31:10.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649faaa640 4 set_mon_vals no callback set 2024-04-26T00:31:10.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649c844640 1 -- 172.21.15.37:0/721579500 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64900a5b30 con 0x7f649813c450 2024-04-26T00:31:10.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649c844640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:10.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649c844640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:31:10.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649c844640 10 monclient: dump: 2024-04-26T00:31:10.429 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:10.429 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:10.429 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:10.429 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:10.429 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:10.429 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:10.429 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:10.429 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:10.429 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:10.429 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:10.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649faaa640 10 monclient: shutdown 2024-04-26T00:31:10.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649faaa640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:31:10.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649faaa640 1 -- 172.21.15.37:0/721579500 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f649813c450 msgr2=0x7f649813e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:10.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649faaa640 1 --2- 172.21.15.37:0/721579500 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f649813c450 0x7f649813e840 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6490009e50 tx=0x7f649001c940 comp rx=0 tx=0).stop 2024-04-26T00:31:10.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649faaa640 1 -- 172.21.15.37:0/721579500 shutdown_connections 2024-04-26T00:31:10.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649faaa640 1 --2- 172.21.15.37:0/721579500 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f649813c450 0x7f649813e840 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:10.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649faaa640 1 --2- 172.21.15.37:0/721579500 >> v2:172.21.15.37:3300/0 conn(0x7f6498136f20 0x7f649809bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:10.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649faaa640 1 -- 172.21.15.37:0/721579500 >> 172.21.15.37:0/721579500 conn(0x7f64980a61b0 msgr2=0x7f64980a65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:31:10.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649faaa640 1 -- 172.21.15.37:0/721579500 shutdown_connections 2024-04-26T00:31:10.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649faaa640 1 -- 172.21.15.37:0/721579500 wait complete. 2024-04-26T00:31:10.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649faaa640 10 monclient: build_initial_monmap 2024-04-26T00:31:10.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649faaa640 10 monclient: monmap: 2024-04-26T00:31:10.430 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:31:10.431 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:31:10.431 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:31:10.431 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:31:10.431 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:31:10.431 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:31:10.431 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:31:10.431 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:31:10.431 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:31:10.431 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:10.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649faaa640 1 Processor -- start 2024-04-26T00:31:10.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649faaa640 1 -- start start 2024-04-26T00:31:10.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649faaa640 10 monclient: init 2024-04-26T00:31:10.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649faaa640 10 monclient: _reopen_session rank -1 2024-04-26T00:31:10.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649faaa640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:31:10.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649faaa640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6498136f20 0x7f64981cebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:10.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649faaa640 10 monclient(hunting): picked mon.noname-a con 0x7f6498136f20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:31:10.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649faaa640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6498137300 0x7f6498134da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:10.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649faaa640 10 monclient(hunting): picked mon.noname-b con 0x7f6498137300 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:31:10.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649faaa640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f649813c450 0x7f6498137ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:10.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649faaa640 10 monclient(hunting): picked mon.noname-c con 0x7f649813c450 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:31:10.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649faaa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:10.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649faaa640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f6498141930 con 0x7f6498136f20 2024-04-26T00:31:10.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649faaa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:10.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649faaa640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f64981417b0 con 0x7f6498137300 2024-04-26T00:31:10.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649faaa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:10.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649faaa640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f6498141ab0 con 0x7f649813c450 2024-04-26T00:31:10.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649faaa640 10 monclient(hunting): _renew_subs 2024-04-26T00:31:10.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649faaa640 10 monclient(hunting): authenticate will time out at 1651.738760s 2024-04-26T00:31:10.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649e047640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f649813c450 0x7f6498137ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:10.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649d846640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6498136f20 0x7f64981cebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:10.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649e047640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f649813c450 0x7f6498137ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:33622/0 (socket says 172.21.15.37:33622) 2024-04-26T00:31:10.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649e047640 1 -- 172.21.15.37:0/2786561311 learned_addr learned my addr 172.21.15.37:0/2786561311 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:31:10.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649e047640 10 monclient(hunting): get_auth_request con 0x7f649813c450 auth_method 0 2024-04-26T00:31:10.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649e047640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:10.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649e047640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:10.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649e047640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:10.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649d846640 10 monclient(hunting): get_auth_request con 0x7f6498136f20 auth_method 0 2024-04-26T00:31:10.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649d846640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:10.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649d846640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:10.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649d846640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:10.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649e047640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:10.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649e047640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:10.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649e047640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:10.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649d846640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:10.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649d846640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:10.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649d045640 1 --2- 172.21.15.37:0/2786561311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6498137300 0x7f6498134da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:10.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649d846640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:10.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649d045640 10 monclient(hunting): get_auth_request con 0x7f6498137300 auth_method 0 2024-04-26T00:31:10.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649d045640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:10.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649d045640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:10.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649d045640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:10.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649e047640 10 monclient(hunting): handle_auth_done global_id 84560 payload 824 2024-04-26T00:31:10.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649e047640 1 -- 172.21.15.37:0/2786561311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6498137300 msgr2=0x7f6498134da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:10.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649e047640 1 --2- 172.21.15.37:0/2786561311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6498137300 0x7f6498134da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:10.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649e047640 1 -- 172.21.15.37:0/2786561311 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6498136f20 msgr2=0x7f64981cebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:10.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649e047640 1 --2- 172.21.15.37:0/2786561311 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6498136f20 0x7f64981cebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:10.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649e047640 10 monclient: _finish_hunting 0 2024-04-26T00:31:10.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649e047640 1 monclient: found mon.noname-c 2024-04-26T00:31:10.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649e047640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:31:10.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649e047640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:31:10.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.429+0000 7f649e047640 1 -- 172.21.15.37:0/2786561311 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f649813b8e0 con 0x7f649813c450 2024-04-26T00:31:10.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.433+0000 7f649e047640 1 -- 172.21.15.37:0/2786561311 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f649813c450 msgr2=0x7f6498137ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:10.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.433+0000 7f649e047640 1 --2- 172.21.15.37:0/2786561311 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f649813c450 0x7f6498137ef0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f64900af040 tx=0x7f64900196e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:10.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.433+0000 7f648e7fc640 1 -- 172.21.15.37:0/2786561311 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64900bc070 con 0x7f649813c450 2024-04-26T00:31:10.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.433+0000 7f648e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:10.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.433+0000 7f648e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:31:10.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.433+0000 7f648e7fc640 10 monclient: dump: 2024-04-26T00:31:10.436 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:10.436 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:10.436 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:10.436 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:10.436 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:10.436 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:10.436 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:10.436 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:10.436 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:10.436 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:10.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.433+0000 7f648e7fc640 10 monclient: _finish_auth 0 2024-04-26T00:31:10.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.433+0000 7f648e7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:31:10.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.433+0000 7f648e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:31:10.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.433+0000 7f648e7fc640 1 -- 172.21.15.37:0/2786561311 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64900b8850 con 0x7f649813c450 2024-04-26T00:31:10.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.433+0000 7f648e7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:31:10.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.433+0000 7f648e7fc640 1 -- 172.21.15.37:0/2786561311 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64900a55c0 con 0x7f649813c450 2024-04-26T00:31:10.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.433+0000 7f648e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:10.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.433+0000 7f648e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:31:10.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.433+0000 7f648e7fc640 10 monclient: dump: 2024-04-26T00:31:10.437 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:10.437 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:10.437 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:10.437 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:10.437 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:10.437 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:10.437 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:10.438 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:10.438 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:10.438 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:10.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.433+0000 7f648ffff640 4 set_mon_vals no callback set 2024-04-26T00:31:10.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.433+0000 7f649faaa640 5 monclient: authenticate success, global_id 84560 2024-04-26T00:31:10.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.433+0000 7f649faaa640 10 monclient: _renew_subs 2024-04-26T00:31:10.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.433+0000 7f649faaa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:10.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.433+0000 7f649faaa640 1 -- 172.21.15.37:0/2786561311 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f649813bbc0 con 0x7f649813c450 2024-04-26T00:31:10.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.433+0000 7f649faaa640 10 monclient: _renew_subs 2024-04-26T00:31:10.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.433+0000 7f649faaa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:10.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.433+0000 7f649faaa640 1 -- 172.21.15.37:0/2786561311 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f64981391a0 con 0x7f649813c450 2024-04-26T00:31:10.439 INFO:tasks.rados.rados.0.smithi045.stdout:1870: finishing write tid 1 to smithi04526579-48 2024-04-26T00:31:10.439 INFO:tasks.rados.rados.0.smithi045.stdout:1870: finishing write tid 2 to smithi04526579-48 2024-04-26T00:31:10.439 INFO:tasks.rados.rados.0.smithi045.stdout:1870: finishing write tid 3 to smithi04526579-48 2024-04-26T00:31:10.439 INFO:tasks.rados.rados.0.smithi045.stdout:1870: finishing write tid 4 to smithi04526579-48 2024-04-26T00:31:10.440 INFO:tasks.rados.rados.0.smithi045.stdout:1870: finishing write tid 5 to smithi04526579-48 2024-04-26T00:31:10.440 INFO:tasks.rados.rados.0.smithi045.stdout:1870: oid 48 updating version 0 to 1087 2024-04-26T00:31:10.440 INFO:tasks.rados.rados.0.smithi045.stdout:1870: oid 48 updating version 1087 to 1090 2024-04-26T00:31:10.440 INFO:tasks.rados.rados.0.smithi045.stdout:1870: oid 48 version 1090 is already newer than 1088 2024-04-26T00:31:10.440 INFO:tasks.rados.rados.0.smithi045.stdout:1870: oid 48 version 1090 is already newer than 1089 2024-04-26T00:31:10.440 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 1090 (ObjNum 684 snap 213 seq_num 684) dirty exists 2024-04-26T00:31:10.440 INFO:tasks.rados.rados.0.smithi045.stdout:1870: left oid 48 (ObjNum 684 snap 213 seq_num 684) 2024-04-26T00:31:10.440 INFO:tasks.rados.rados.0.smithi045.stdout:1872: finishing rollback tid 1 to smithi04526579-3 2024-04-26T00:31:10.440 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 975 (ObjNum 129 snap 0 seq_num 2240331104) dirty dne 2024-04-26T00:31:10.440 INFO:tasks.rados.rados.0.smithi045.stdout:1871: expect (ObjNum 646 snap 202 seq_num 646) 2024-04-26T00:31:10.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.433+0000 7f649faaa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:10.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.433+0000 7f649faaa640 1 --2- 172.21.15.37:0/2786561311 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f649813b320 0x7f649802c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:10.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.433+0000 7f649faaa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:10.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.433+0000 7f649faaa640 1 -- 172.21.15.37:0/2786561311 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f649804f510 con 0x7f649813b320 2024-04-26T00:31:10.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.433+0000 7f649d846640 1 --2- 172.21.15.37:0/2786561311 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f649813b320 0x7f649802c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:10.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.433+0000 7f649d846640 10 monclient: get_auth_request con 0x7f649813b320 auth_method 0 2024-04-26T00:31:10.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.433+0000 7f649d846640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:10.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.433+0000 7f649d846640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:10.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.433+0000 7f649d846640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:10.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.433+0000 7f649d846640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:10.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.433+0000 7f649d846640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:10.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.433+0000 7f649e047640 1 --2- 172.21.15.37:0/2786561311 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f649813c450 0x7f6498137ef0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f64900af040 tx=0x7f64900196e0 comp rx=0 tx=0).queue_received will delay after 1.71409e+09 on 0x7f64900e3760 osd_map(427..427 src has 370..427) 2024-04-26T00:31:10.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.433+0000 7f649d846640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:10.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.433+0000 7f648e7fc640 1 -- 172.21.15.37:0/2786561311 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f64900b8c40 con 0x7f649813c450 2024-04-26T00:31:10.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.433+0000 7f649d846640 10 monclient(hunting): handle_auth_done global_id 84563 payload 824 2024-04-26T00:31:10.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.433+0000 7f648e7fc640 1 --2- 172.21.15.37:0/2786561311 >> v2:172.21.15.45:6812/2379918623 conn(0x7f647c035ed0 0x7f647c038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:10.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.433+0000 7f649d846640 1 -- 172.21.15.37:0/2786561311 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f64981398d0 con 0x7f649813b320 2024-04-26T00:31:10.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.437+0000 7f649d045640 1 --2- 172.21.15.37:0/2786561311 >> v2:172.21.15.45:6812/2379918623 conn(0x7f647c035ed0 0x7f647c038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:10.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.437+0000 7f649d045640 10 monclient: get_auth_request con 0x7f647c035ed0 auth_method 0 2024-04-26T00:31:10.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.437+0000 7f649d846640 1 -- 172.21.15.37:0/2786561311 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f649813b320 msgr2=0x7f649802c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:10.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.437+0000 7f649d846640 1 --2- 172.21.15.37:0/2786561311 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f649813b320 0x7f649802c000 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6494011900 tx=0x7f6494011dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:10.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.437+0000 7f648e7fc640 1 -- 172.21.15.37:0/2786561311 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6494019070 con 0x7f649813b320 2024-04-26T00:31:10.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.437+0000 7f648e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:10.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.437+0000 7f648e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:31:10.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.437+0000 7f648e7fc640 10 monclient: dump: 2024-04-26T00:31:10.445 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:10.445 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:10.445 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:10.445 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:10.446 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:10.446 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:10.446 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:10.446 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:10.446 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:10.446 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:10.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.437+0000 7f648e7fc640 1 -- 172.21.15.37:0/2786561311 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f649400f890 con 0x7f649813b320 2024-04-26T00:31:10.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.437+0000 7f648e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:10.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.437+0000 7f648e7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:31:10.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.437+0000 7f648e7fc640 1 -- 172.21.15.37:0/2786561311 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f649813b320 msgr2=0x7f649802c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:10.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.437+0000 7f648e7fc640 1 --2- 172.21.15.37:0/2786561311 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f649813b320 0x7f649802c000 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6494011900 tx=0x7f6494011dc0 comp rx=0 tx=0).stop 2024-04-26T00:31:10.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.437+0000 7f648e7fc640 1 -- 172.21.15.37:0/2786561311 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f649813b320 msgr2=0x7f649802c000 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:31:10.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.437+0000 7f648e7fc640 1 --2- 172.21.15.37:0/2786561311 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f649813b320 0x7f649802c000 secure :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6494011900 tx=0x7f6494011dc0 comp rx=0 tx=0).stop 2024-04-26T00:31:10.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.437+0000 7f649d045640 1 --2- 172.21.15.37:0/2786561311 >> v2:172.21.15.45:6812/2379918623 conn(0x7f647c035ed0 0x7f647c038390 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f649813bdd0 tx=0x7f6488009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:10.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.445+0000 7f649faaa640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:31:10.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.445+0000 7f649faaa640 1 --2- 172.21.15.37:0/2786561311 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6498138bf0 0x7f649802c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:10.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.445+0000 7f649faaa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:10.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.445+0000 7f649faaa640 1 -- 172.21.15.37:0/2786561311 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f649809fa40 con 0x7f6498138bf0 2024-04-26T00:31:10.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.445+0000 7f649d846640 1 --2- 172.21.15.37:0/2786561311 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6498138bf0 0x7f649802c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:10.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.445+0000 7f649d846640 10 monclient: get_auth_request con 0x7f6498138bf0 auth_method 0 2024-04-26T00:31:10.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.445+0000 7f649d846640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:10.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.445+0000 7f649d846640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:10.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.445+0000 7f649d846640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:10.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.445+0000 7f649d846640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:10.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.445+0000 7f649d846640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:10.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.445+0000 7f649d846640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:10.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.445+0000 7f649d846640 10 monclient(hunting): handle_auth_done global_id 84566 payload 824 2024-04-26T00:31:10.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.445+0000 7f649d846640 1 -- 172.21.15.37:0/2786561311 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6498075f10 con 0x7f6498138bf0 2024-04-26T00:31:10.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.445+0000 7f649d846640 1 -- 172.21.15.37:0/2786561311 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6498138bf0 msgr2=0x7f649802c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:10.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.445+0000 7f649d846640 1 --2- 172.21.15.37:0/2786561311 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6498138bf0 0x7f649802c540 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6494011900 tx=0x7f64940046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:10.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.445+0000 7f648e7fc640 1 -- 172.21.15.37:0/2786561311 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6494019040 con 0x7f6498138bf0 2024-04-26T00:31:10.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.445+0000 7f648e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:10.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.445+0000 7f648e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:31:10.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.445+0000 7f648e7fc640 10 monclient: dump: 2024-04-26T00:31:10.448 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:10.448 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:10.448 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:10.448 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:10.448 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:10.448 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:10.449 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:10.449 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:10.449 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:10.449 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:10.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.445+0000 7f648e7fc640 1 -- 172.21.15.37:0/2786561311 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4889 (secure 0 0 0) 0x7f649401e450 con 0x7f6498138bf0 2024-04-26T00:31:10.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.445+0000 7f648e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:31:10.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.445+0000 7f648e7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:31:10.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.445+0000 7f648e7fc640 1 -- 172.21.15.37:0/2786561311 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6498138bf0 msgr2=0x7f649802c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:10.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.445+0000 7f648e7fc640 1 --2- 172.21.15.37:0/2786561311 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6498138bf0 0x7f649802c540 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6494011900 tx=0x7f64940046d0 comp rx=0 tx=0).stop 2024-04-26T00:31:10.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.445+0000 7f648e7fc640 1 -- 172.21.15.37:0/2786561311 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6498138bf0 msgr2=0x7f649802c540 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:31:10.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.445+0000 7f648e7fc640 1 --2- 172.21.15.37:0/2786561311 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6498138bf0 0x7f649802c540 secure :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6494011900 tx=0x7f64940046d0 comp rx=0 tx=0).stop 2024-04-26T00:31:10.449 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:31:10.450 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:31:10.450 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:31:10.450 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:31:10.450 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 86, 2024-04-26T00:31:10.450 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 275380, 2024-04-26T00:31:10.450 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:31:10.450 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:31:10.450 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:31:10.450 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:31:10.450 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:31:10.450 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:31:10.450 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:31:10.451 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:31:10.451 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:31:10.451 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:31:10.451 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:31:10.451 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:31:10.451 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:31:10.451 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:31:10.451 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:31:10.451 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:31:10.451 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:31:10.451 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:31:10.451 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:31:10.451 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:31:10.451 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:31:10.452 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:31:10.452 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:31:10.452 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:31:10.452 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:31:10.452 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:31:10.452 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:31:10.452 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:31:10.452 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:31:10.452 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:31:10.452 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:31:10.452 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:31:10.452 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:31:10.452 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:31:10.452 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:10.453 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:10.453 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:31:10.453 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:31:10.453 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:31:10.453 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:31:10.453 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:31:10.453 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:31:10.453 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:31:10.453 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:31:10.453 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:31:10.453 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:31:10.453 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:31:10.453 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:31:10.453 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:31:10.454 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:31:10.454 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:31:10.454 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:31:10.454 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:31:10.454 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:31:10.454 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:31:10.454 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:31:10.454 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:31:10.454 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:31:10.454 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:31:10.454 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:31:10.454 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:31:10.454 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:31:10.455 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:31:10.455 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:31:10.455 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:31:10.455 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:31:10.455 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:10.455 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:31:10.455 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:10.455 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:31:10.455 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:31:10.455 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:31:10.455 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:31:10.455 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:10.455 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:31:10.455 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:31:10.456 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:10.456 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:10.456 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:10.456 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:31:10.456 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:31:10.456 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:10.456 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:10.456 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:10.456 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:10.456 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:31:10.456 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:31:10.456 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:31:10.456 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:31:10.457 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:31:10.457 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:10.457 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:10.457 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:31:10.457 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:31:10.457 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:31:10.457 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:31:10.457 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:10.457 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:31:10.457 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:31:10.457 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:10.457 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:10.457 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:10.458 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:31:10.458 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:31:10.458 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:10.458 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:10.458 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:10.458 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:10.458 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:31:10.458 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:31:10.458 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:31:10.458 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:31:10.458 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:31:10.458 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:10.458 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:10.459 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:31:10.459 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:31:10.459 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:31:10.459 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:31:10.459 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:10.459 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:31:10.459 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:31:10.459 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:10.459 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:10.459 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:10.459 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:31:10.459 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:31:10.459 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:10.460 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:10.460 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:10.460 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:10.460 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:31:10.460 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:31:10.460 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:31:10.460 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:31:10.460 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:31:10.460 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:10.460 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:10.460 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:10.460 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:31:10.460 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:31:10.460 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:10.461 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:31:10.461 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:31:10.461 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:31:10.461 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:10.461 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:31:10.461 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:31:10.461 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:10.461 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:31:10.461 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:31:10.461 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:31:10.461 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:10.461 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:31:10.461 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:31:10.461 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:10.462 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:31:10.462 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:31:10.462 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 7 2024-04-26T00:31:10.462 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:10.462 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:31:10.462 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:31:10.462 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:10.462 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:31:10.462 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:31:10.462 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:31:10.462 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:10.462 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:10.462 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:10.462 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:31:10.463 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:31:10.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.449+0000 7f649faaa640 1 -- 172.21.15.37:0/2786561311 >> v2:172.21.15.45:6812/2379918623 conn(0x7f647c035ed0 msgr2=0x7f647c038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:10.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.449+0000 7f649faaa640 1 --2- 172.21.15.37:0/2786561311 >> v2:172.21.15.45:6812/2379918623 conn(0x7f647c035ed0 0x7f647c038390 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f649813bdd0 tx=0x7f6488009040 comp rx=0 tx=0).stop 2024-04-26T00:31:10.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.449+0000 7f649faaa640 10 monclient: shutdown 2024-04-26T00:31:10.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.449+0000 7f649faaa640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:31:10.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.449+0000 7f649faaa640 1 -- 172.21.15.37:0/2786561311 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f649813c450 msgr2=0x7f6498137ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:10.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.449+0000 7f649faaa640 1 --2- 172.21.15.37:0/2786561311 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f649813c450 0x7f6498137ef0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f64900af040 tx=0x7f64900196e0 comp rx=0 tx=0).stop 2024-04-26T00:31:10.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.449+0000 7f649e047640 1 -- 172.21.15.37:0/2786561311 reap_dead start 2024-04-26T00:31:10.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.449+0000 7f649e047640 1 -- 172.21.15.37:0/2786561311 reap_dead start 2024-04-26T00:31:10.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.449+0000 7f648e7fc640 1 -- 172.21.15.37:0/2786561311 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(427..427 src has 370..427) ==== 4035+0+0 (unknown 0 0 0) 0x7f64900e3760 con 0x7f649813c450 2024-04-26T00:31:10.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.449+0000 7f649faaa640 1 -- 172.21.15.37:0/2786561311 shutdown_connections 2024-04-26T00:31:10.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.449+0000 7f649faaa640 1 -- 172.21.15.37:0/2786561311 >> 172.21.15.37:0/2786561311 conn(0x7f64980a61b0 msgr2=0x7f64980a8260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:31:10.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.449+0000 7f649faaa640 1 -- 172.21.15.37:0/2786561311 shutdown_connections 2024-04-26T00:31:10.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.449+0000 7f649faaa640 1 -- 172.21.15.37:0/2786561311 wait complete. 2024-04-26T00:31:10.465 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:31:10.466 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:31:10.466 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:31:10.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8c1d51640 10 monclient: get_monmap_and_config 2024-04-26T00:31:10.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8c1d51640 10 monclient: build_initial_monmap 2024-04-26T00:31:10.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8c1d51640 10 monclient: monmap: 2024-04-26T00:31:10.581 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:31:10.581 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:10.581 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:31:10.592549+0000 2024-04-26T00:31:10.581 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:31:10.592549+0000 2024-04-26T00:31:10.581 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:31:10.581 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:10.582 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:31:10.582 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:31:10.582 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:31:10.582 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:31:10.582 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:31:10.582 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:10.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8c1d51640 1 Processor -- start 2024-04-26T00:31:10.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8c1d51640 1 -- start start 2024-04-26T00:31:10.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8c1d51640 10 monclient: init 2024-04-26T00:31:10.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8c1d51640 10 monclient: _reopen_session rank -1 2024-04-26T00:31:10.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8c1d51640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-04-26T00:31:10.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8c1d51640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff8bc136f20 addr v1:172.21.15.37:6789/0 2024-04-26T00:31:10.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8c1d51640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff8bc137a10 0x7ff8bc09bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:10.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8c1d51640 10 monclient(hunting): picked mon.noname-b con 0x7ff8bc137a10 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:31:10.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8c1d51640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff8bc13c450 0x7ff8bc13e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:10.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8c1d51640 10 monclient(hunting): picked mon.noname-c con 0x7ff8bc13c450 addr v2:172.21.15.45:3300/0 2024-04-26T00:31:10.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8c1d51640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8bc0a50f0 con 0x7ff8bc136f20 2024-04-26T00:31:10.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8c1d51640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:10.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8c1d51640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7ff8bc09b330 con 0x7ff8bc13c450 2024-04-26T00:31:10.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8c1d51640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:10.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8c1d51640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff8bc09b610 con 0x7ff8bc137a10 2024-04-26T00:31:10.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8c1d51640 10 monclient(hunting): _renew_subs 2024-04-26T00:31:10.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8c1d51640 10 monclient(hunting): authenticate will time out at 1651.898866s 2024-04-26T00:31:10.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8baffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff8bc137a10 0x7ff8bc09bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:10.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8bb7fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7ff8bc136f20 0x7ff8bc137300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:42702/0 (socket says 172.21.15.37:42702) 2024-04-26T00:31:10.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8bb7fe640 1 -- 172.21.15.37:0/499022952 learned_addr learned my addr 172.21.15.37:0/499022952 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:31:10.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8baffd640 10 monclient(hunting): get_auth_request con 0x7ff8bc137a10 auth_method 0 2024-04-26T00:31:10.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8baffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:10.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8baffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:10.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8baffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:10.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8baffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:10.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8baffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:10.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8bbfff640 1 --2- 172.21.15.37:0/499022952 >> v2:172.21.15.45:3300/0 conn(0x7ff8bc13c450 0x7ff8bc13e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:10.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8baffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:10.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8bbfff640 10 monclient(hunting): get_auth_request con 0x7ff8bc13c450 auth_method 0 2024-04-26T00:31:10.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:10.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8bbfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:10.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:10.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8baffd640 10 monclient(hunting): handle_auth_done global_id 84569 payload 274 2024-04-26T00:31:10.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8baffd640 1 -- 172.21.15.37:0/499022952 >> v2:172.21.15.45:3300/0 conn(0x7ff8bc13c450 msgr2=0x7ff8bc13e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:31:10.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8baffd640 1 --2- 172.21.15.37:0/499022952 >> v2:172.21.15.45:3300/0 conn(0x7ff8bc13c450 0x7ff8bc13e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:10.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8baffd640 1 -- 172.21.15.37:0/499022952 >> v1:172.21.15.37:6789/0 conn(0x7ff8bc136f20 legacy=0x7ff8bc137300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:31:10.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8baffd640 10 monclient: _finish_hunting 0 2024-04-26T00:31:10.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8baffd640 1 monclient: found mon.noname-b 2024-04-26T00:31:10.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:31:10.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8baffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:31:10.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.589+0000 7ff8baffd640 1 -- 172.21.15.37:0/499022952 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8bc09b8f0 con 0x7ff8bc137a10 2024-04-26T00:31:10.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.593+0000 7ff8baffd640 1 -- 172.21.15.37:0/499022952 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff8bc137a10 msgr2=0x7ff8bc09bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:10.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.593+0000 7ff8baffd640 1 --2- 172.21.15.37:0/499022952 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff8bc137a10 0x7ff8bc09bff0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc137e90 tx=0x7ff8b001c390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad5a752ce61ea504 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:10.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.593+0000 7ff8ba7fc640 1 -- 172.21.15.37:0/499022952 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b00b0070 con 0x7ff8bc137a10 2024-04-26T00:31:10.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.593+0000 7ff8ba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:10.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.593+0000 7ff8ba7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:31:10.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.593+0000 7ff8ba7fc640 10 monclient: dump: 2024-04-26T00:31:10.589 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:10.589 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:10.589 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:10.589 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:10.589 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:10.589 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:10.589 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:10.590 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:10.590 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:10.590 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:10.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.593+0000 7ff8ba7fc640 10 monclient: _finish_auth 0 2024-04-26T00:31:10.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.593+0000 7ff8ba7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:31:10.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.593+0000 7ff8ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:31:10.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.593+0000 7ff8ba7fc640 1 -- 172.21.15.37:0/499022952 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8b00ab600 con 0x7ff8bc137a10 2024-04-26T00:31:10.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.593+0000 7ff8c1d51640 5 monclient: authenticate success, global_id 84569 2024-04-26T00:31:10.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.593+0000 7ff8c1d51640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:31:10.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.593+0000 7ff8ba7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:31:10.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.593+0000 7ff8c1d51640 10 monclient: get_monmap_and_config success 2024-04-26T00:31:10.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.593+0000 7ff8c1d51640 4 set_mon_vals no callback set 2024-04-26T00:31:10.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.593+0000 7ff8ba7fc640 1 -- 172.21.15.37:0/499022952 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b00ab9d0 con 0x7ff8bc137a10 2024-04-26T00:31:10.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.593+0000 7ff8ba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:10.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.593+0000 7ff8ba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:31:10.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.593+0000 7ff8ba7fc640 10 monclient: dump: 2024-04-26T00:31:10.591 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:10.591 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:10.591 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:10.591 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:10.591 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:10.591 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:10.591 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:10.591 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:10.591 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:10.591 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:10.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8c1d51640 10 monclient: shutdown 2024-04-26T00:31:10.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8c1d51640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:31:10.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8c1d51640 1 -- 172.21.15.37:0/499022952 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff8bc137a10 msgr2=0x7ff8bc09bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:10.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8c1d51640 1 --2- 172.21.15.37:0/499022952 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff8bc137a10 0x7ff8bc09bff0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc137e90 tx=0x7ff8b001c390 comp rx=0 tx=0).stop 2024-04-26T00:31:10.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8c1d51640 1 -- 172.21.15.37:0/499022952 shutdown_connections 2024-04-26T00:31:10.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8c1d51640 1 --2- 172.21.15.37:0/499022952 >> v2:172.21.15.45:3300/0 conn(0x7ff8bc13c450 0x7ff8bc13e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:10.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8c1d51640 1 --2- 172.21.15.37:0/499022952 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff8bc137a10 0x7ff8bc09bff0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:10.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8c1d51640 1 -- 172.21.15.37:0/499022952 >> 172.21.15.37:0/499022952 conn(0x7ff8bc0a61b0 msgr2=0x7ff8bc0a65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:31:10.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8c1d51640 1 -- 172.21.15.37:0/499022952 shutdown_connections 2024-04-26T00:31:10.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8c1d51640 1 -- 172.21.15.37:0/499022952 wait complete. 2024-04-26T00:31:10.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8c1d51640 10 monclient: build_initial_monmap 2024-04-26T00:31:10.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8c1d51640 10 monclient: monmap: 2024-04-26T00:31:10.597 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:31:10.597 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:31:10.597 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:31:10.597 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:31:10.597 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:31:10.597 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:31:10.597 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:31:10.598 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:31:10.598 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:31:10.598 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:10.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8c1d51640 1 Processor -- start 2024-04-26T00:31:10.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8c1d51640 1 -- start start 2024-04-26T00:31:10.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8c1d51640 10 monclient: init 2024-04-26T00:31:10.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8c1d51640 10 monclient: _reopen_session rank -1 2024-04-26T00:31:10.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8c1d51640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:31:10.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8c1d51640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff8bc136f20 0x7ff8bc1ceda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:10.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8c1d51640 10 monclient(hunting): picked mon.noname-a con 0x7ff8bc136f20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:31:10.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8c1d51640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc137a10 0x7ff8bc134f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:10.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8c1d51640 10 monclient(hunting): picked mon.noname-b con 0x7ff8bc137a10 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:31:10.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8c1d51640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff8bc13c450 0x7ff8bc137ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:10.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8c1d51640 10 monclient(hunting): picked mon.noname-c con 0x7ff8bc13c450 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:31:10.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8c1d51640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:10.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8c1d51640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff8bc141930 con 0x7ff8bc136f20 2024-04-26T00:31:10.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8c1d51640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:10.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8c1d51640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff8bc1417b0 con 0x7ff8bc137a10 2024-04-26T00:31:10.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8c1d51640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:10.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8c1d51640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff8bc141ab0 con 0x7ff8bc13c450 2024-04-26T00:31:10.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8c1d51640 10 monclient(hunting): _renew_subs 2024-04-26T00:31:10.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8c1d51640 10 monclient(hunting): authenticate will time out at 1651.906066s 2024-04-26T00:31:10.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8bb7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff8bc136f20 0x7ff8bc1ceda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:10.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8bb7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff8bc136f20 0x7ff8bc1ceda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:56044/0 (socket says 172.21.15.37:56044) 2024-04-26T00:31:10.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8bbfff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff8bc13c450 0x7ff8bc137ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:10.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8bb7fe640 1 -- 172.21.15.37:0/3448445169 learned_addr learned my addr 172.21.15.37:0/3448445169 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:31:10.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8bb7fe640 10 monclient(hunting): get_auth_request con 0x7ff8bc136f20 auth_method 0 2024-04-26T00:31:10.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:10.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8bb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:10.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:10.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8bbfff640 10 monclient(hunting): get_auth_request con 0x7ff8bc13c450 auth_method 0 2024-04-26T00:31:10.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:10.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8bbfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:10.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:10.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8bb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:10.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8baffd640 1 --2- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc137a10 0x7ff8bc134f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:10.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8bb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:10.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8bb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:10.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:10.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:10.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:10.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8baffd640 10 monclient(hunting): get_auth_request con 0x7ff8bc137a10 auth_method 0 2024-04-26T00:31:10.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8baffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:10.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8baffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:10.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8baffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:10.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8baffd640 1 -- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc137a10 msgr2=0x7ff8bc134f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:31:10.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8baffd640 1 -- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc137a10 msgr2=0x7ff8bc134f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:31:10.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8baffd640 1 --2- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc137a10 0x7ff8bc134f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:31:10.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8baffd640 1 --2- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc137a10 0x7ff8bc134f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:31:10.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8bb7fe640 10 monclient(hunting): handle_auth_done global_id 124212 payload 824 2024-04-26T00:31:10.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8bb7fe640 1 -- 172.21.15.37:0/3448445169 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff8bc13c450 msgr2=0x7ff8bc137ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:10.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8bb7fe640 1 --2- 172.21.15.37:0/3448445169 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff8bc13c450 0x7ff8bc137ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:10.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8bb7fe640 1 -- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc137a10 msgr2=0x7ff8bc134f60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:31:10.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8bb7fe640 1 --2- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc137a10 0x7ff8bc134f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:10.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8bb7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:31:10.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8bb7fe640 1 monclient: found mon.noname-a 2024-04-26T00:31:10.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:31:10.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8bb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:31:10.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8bb7fe640 1 -- 172.21.15.37:0/3448445169 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8bc139a20 con 0x7ff8bc136f20 2024-04-26T00:31:10.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.597+0000 7ff8bbfff640 1 --2- 172.21.15.37:0/3448445169 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff8bc13c450 0x7ff8bc137ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:31:10.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8bb7fe640 1 -- 172.21.15.37:0/3448445169 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff8bc136f20 msgr2=0x7ff8bc1ceda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:10.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8bb7fe640 1 --2- 172.21.15.37:0/3448445169 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff8bc136f20 0x7ff8bc1ceda0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac00cc10 tx=0x7ff8ac00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:10.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8a3fff640 1 -- 172.21.15.37:0/3448445169 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac012400 con 0x7ff8bc136f20 2024-04-26T00:31:10.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:10.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8a3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:31:10.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8a3fff640 10 monclient: dump: 2024-04-26T00:31:10.602 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:10.602 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:10.602 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:10.602 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:10.603 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:10.603 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:10.603 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:10.603 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:10.603 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:10.603 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:10.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8a3fff640 10 monclient: _finish_auth 0 2024-04-26T00:31:10.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8a3fff640 10 monclient: _check_auth_tickets 2024-04-26T00:31:10.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:31:10.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8a3fff640 1 -- 172.21.15.37:0/3448445169 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8ac0125e0 con 0x7ff8bc136f20 2024-04-26T00:31:10.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8c1d51640 5 monclient: authenticate success, global_id 124212 2024-04-26T00:31:10.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8c1d51640 10 monclient: _renew_subs 2024-04-26T00:31:10.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8c1d51640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:31:10.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8c1d51640 1 -- 172.21.15.37:0/3448445169 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8bc09ae30 con 0x7ff8bc136f20 2024-04-26T00:31:10.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8a3fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:31:10.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8a3fff640 1 -- 172.21.15.37:0/3448445169 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac0129d0 con 0x7ff8bc136f20 2024-04-26T00:31:10.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:10.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8a3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:31:10.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8a3fff640 10 monclient: dump: 2024-04-26T00:31:10.604 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:10.604 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:10.604 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:10.604 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:10.604 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:10.604 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:10.604 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:10.604 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:10.604 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:10.605 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:10.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8b97fa640 4 set_mon_vals no callback set 2024-04-26T00:31:10.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8c1d51640 10 monclient: _renew_subs 2024-04-26T00:31:10.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8c1d51640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:31:10.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8c1d51640 1 -- 172.21.15.37:0/3448445169 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8bc13bd10 con 0x7ff8bc136f20 2024-04-26T00:31:10.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8c1d51640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:10.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8c1d51640 1 --2- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc138430 0x7ff8bc138810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:10.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8c1d51640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:10.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8c1d51640 1 -- 172.21.15.37:0/3448445169 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff8bc13bb90 con 0x7ff8bc138430 2024-04-26T00:31:10.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8a3fff640 1 -- 172.21.15.37:0/3448445169 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff8ac012bf0 con 0x7ff8bc136f20 2024-04-26T00:31:10.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8a3fff640 1 --2- 172.21.15.37:0/3448445169 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff8a8035f00 0x7ff8a80383c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:10.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8a3fff640 1 -- 172.21.15.37:0/3448445169 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(427..427 src has 370..427) ==== 4035+0+0 (secure 0 0 0) 0x7ff8ac0491d0 con 0x7ff8bc136f20 2024-04-26T00:31:10.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8baffd640 1 --2- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc138430 0x7ff8bc138810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:10.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8baffd640 10 monclient: get_auth_request con 0x7ff8bc138430 auth_method 0 2024-04-26T00:31:10.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8baffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:10.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8baffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:10.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8baffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:10.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8baffd640 1 -- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc138430 msgr2=0x7ff8bc138810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:31:10.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8baffd640 1 -- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc138430 msgr2=0x7ff8bc138810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:31:10.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8baffd640 1 --2- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc138430 0x7ff8bc138810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:31:10.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8baffd640 1 --2- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc138430 0x7ff8bc138810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:31:10.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8bbfff640 1 --2- 172.21.15.37:0/3448445169 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff8a8035f00 0x7ff8a80383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:10.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.601+0000 7ff8bbfff640 10 monclient: get_auth_request con 0x7ff8a8035f00 auth_method 0 2024-04-26T00:31:10.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.605+0000 7ff8bbfff640 1 --2- 172.21.15.37:0/3448445169 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff8a8035f00 0x7ff8a80383c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff8b4006aa0 tx=0x7ff8b4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:10.607 INFO:tasks.rados.rados.0.smithi045.stdout:1867: done (5 left) 2024-04-26T00:31:10.607 INFO:tasks.rados.rados.0.smithi045.stdout:1868: done (4 left) 2024-04-26T00:31:10.607 INFO:tasks.rados.rados.0.smithi045.stdout:1869: done (3 left) 2024-04-26T00:31:10.607 INFO:tasks.rados.rados.0.smithi045.stdout:1870: done (2 left) 2024-04-26T00:31:10.607 INFO:tasks.rados.rados.0.smithi045.stdout:1871: done (1 left) 2024-04-26T00:31:10.607 INFO:tasks.rados.rados.0.smithi045.stdout:1872: done (0 left) 2024-04-26T00:31:10.607 INFO:tasks.rados.rados.0.smithi045.stdout:1873: write (excl) oid 15 current snap is 213 2024-04-26T00:31:10.608 INFO:tasks.rados.rados.0.smithi045.stdout:1873: seq_num 685 ranges {712795=417335,1784636=540572,2894750=457917} 2024-04-26T00:31:10.614 INFO:tasks.rados.rados.0.smithi045.stdout:1873: writing smithi04526579-15 from 712795 to 1130130 tid 1 2024-04-26T00:31:10.627 INFO:tasks.rados.rados.0.smithi045.stdout:1873: writing smithi04526579-15 from 1784636 to 2325208 tid 2 2024-04-26T00:31:10.638 INFO:tasks.rados.rados.0.smithi045.stdout:1873: writing smithi04526579-15 from 2894750 to 3352667 tid 3 2024-04-26T00:31:10.638 INFO:tasks.rados.rados.0.smithi045.stdout:1874: delete oid 5 current snap is 213 2024-04-26T00:31:10.638 INFO:tasks.rados.rados.0.smithi045.stdout:1873: finishing write tid 1 to smithi04526579-15 2024-04-26T00:31:10.638 INFO:tasks.rados.rados.0.smithi045.stdout:1873: finishing write tid 2 to smithi04526579-15 2024-04-26T00:31:10.644 INFO:tasks.rados.rados.0.smithi045.stdout:1873: finishing write tid 3 to smithi04526579-15 2024-04-26T00:31:10.644 INFO:tasks.rados.rados.0.smithi045.stdout:1873: finishing write tid 4 to smithi04526579-15 2024-04-26T00:31:10.645 INFO:tasks.rados.rados.0.smithi045.stdout:1873: finishing write tid 5 to smithi04526579-15 2024-04-26T00:31:10.645 INFO:tasks.rados.rados.0.smithi045.stdout:1873: oid 15 updating version 0 to 1223 2024-04-26T00:31:10.645 INFO:tasks.rados.rados.0.smithi045.stdout:1873: oid 15 version 1223 is already newer than 1222 2024-04-26T00:31:10.645 INFO:tasks.rados.rados.0.smithi045.stdout:1873: oid 15 updating version 1223 to 1225 2024-04-26T00:31:10.645 INFO:tasks.rados.rados.0.smithi045.stdout:1873: oid 15 version 1225 is already newer than 1224 2024-04-26T00:31:10.645 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 1225 (ObjNum 685 snap 213 seq_num 685) dirty exists 2024-04-26T00:31:10.645 INFO:tasks.rados.rados.0.smithi045.stdout:1873: left oid 15 (ObjNum 685 snap 213 seq_num 685) 2024-04-26T00:31:10.647 INFO:tasks.rados.rados.0.smithi045.stdout:1873: done (1 left) 2024-04-26T00:31:10.647 INFO:tasks.rados.rados.0.smithi045.stdout:1874: done (0 left) 2024-04-26T00:31:10.647 INFO:tasks.rados.rados.0.smithi045.stdout:1875: read oid 39 snap -1 2024-04-26T00:31:10.647 INFO:tasks.rados.rados.0.smithi045.stdout:1875: expect (ObjNum 676 snap 211 seq_num 676) 2024-04-26T00:31:10.647 INFO:tasks.rados.rados.0.smithi045.stdout:1876: write (excl) oid 36 current snap is 213 2024-04-26T00:31:10.647 INFO:tasks.rados.rados.0.smithi045.stdout:1876: seq_num 687 ranges {746409=567833,1901600=544241,3065949=480460} 2024-04-26T00:31:10.660 INFO:tasks.rados.rados.0.smithi045.stdout:1876: writing smithi04526579-36 from 746409 to 1314242 tid 1 2024-04-26T00:31:10.674 INFO:tasks.rados.rados.0.smithi045.stdout:1876: writing smithi04526579-36 from 1901600 to 2445841 tid 2 2024-04-26T00:31:10.685 INFO:tasks.rados.rados.0.smithi045.stdout:1876: writing smithi04526579-36 from 3065949 to 3546409 tid 3 2024-04-26T00:31:10.685 INFO:tasks.rados.rados.0.smithi045.stdout:1877: snap_create 2024-04-26T00:31:10.685 INFO:tasks.rados.rados.0.smithi045.stdout:1875: expect (ObjNum 676 snap 211 seq_num 676) 2024-04-26T00:31:10.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.805+0000 7ff8baffd640 1 --2- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc138430 0x7ff8bc138810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:10.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.805+0000 7ff8baffd640 10 monclient: get_auth_request con 0x7ff8bc138430 auth_method 0 2024-04-26T00:31:10.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.805+0000 7ff8baffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:10.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.805+0000 7ff8baffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:10.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.805+0000 7ff8baffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:31:10.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.805+0000 7ff8baffd640 1 -- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc138430 msgr2=0x7ff8bc138810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:31:10.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.805+0000 7ff8baffd640 1 -- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc138430 msgr2=0x7ff8bc138810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:31:10.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.805+0000 7ff8baffd640 1 --2- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc138430 0x7ff8bc138810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:31:10.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:10.805+0000 7ff8baffd640 1 --2- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc138430 0x7ff8bc138810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:31:10.923 INFO:tasks.rados.rados.0.smithi045.stdout:1876: finishing write tid 1 to smithi04526579-36 2024-04-26T00:31:10.924 INFO:tasks.rados.rados.0.smithi045.stdout:1876: finishing write tid 2 to smithi04526579-36 2024-04-26T00:31:10.924 INFO:tasks.rados.rados.0.smithi045.stdout:1876: finishing write tid 3 to smithi04526579-36 2024-04-26T00:31:10.924 INFO:tasks.rados.rados.0.smithi045.stdout:1876: finishing write tid 4 to smithi04526579-36 2024-04-26T00:31:10.924 INFO:tasks.rados.rados.0.smithi045.stdout:1876: finishing write tid 5 to smithi04526579-36 2024-04-26T00:31:10.924 INFO:tasks.rados.rados.0.smithi045.stdout:1876: oid 36 updating version 0 to 1431 2024-04-26T00:31:10.924 INFO:tasks.rados.rados.0.smithi045.stdout:1876: oid 36 version 1431 is already newer than 1430 2024-04-26T00:31:10.924 INFO:tasks.rados.rados.0.smithi045.stdout:1876: oid 36 updating version 1431 to 1432 2024-04-26T00:31:10.924 INFO:tasks.rados.rados.0.smithi045.stdout:1876: oid 36 updating version 1432 to 1433 2024-04-26T00:31:10.924 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 1433 (ObjNum 687 snap 213 seq_num 687) dirty exists 2024-04-26T00:31:10.924 INFO:tasks.rados.rados.0.smithi045.stdout:1876: left oid 36 (ObjNum 687 snap 213 seq_num 687) 2024-04-26T00:31:11.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:11.209+0000 7ff8baffd640 1 --2- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc138430 0x7ff8bc138810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:11.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:11.209+0000 7ff8baffd640 10 monclient: get_auth_request con 0x7ff8bc138430 auth_method 0 2024-04-26T00:31:11.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:11.209+0000 7ff8baffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:11.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:11.209+0000 7ff8baffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:11.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:11.209+0000 7ff8baffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:31:11.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:11.209+0000 7ff8baffd640 1 -- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc138430 msgr2=0x7ff8bc138810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:31:11.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:11.209+0000 7ff8baffd640 1 -- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc138430 msgr2=0x7ff8bc138810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:31:11.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:11.209+0000 7ff8baffd640 1 --2- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc138430 0x7ff8bc138810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:31:11.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:11.209+0000 7ff8baffd640 1 --2- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc138430 0x7ff8bc138810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:31:11.337 INFO:tasks.rados.rados.0.smithi045.stdout:1875: done (2 left) 2024-04-26T00:31:11.337 INFO:tasks.rados.rados.0.smithi045.stdout:1876: done (1 left) 2024-04-26T00:31:11.337 INFO:tasks.rados.rados.0.smithi045.stdout:1877: done (0 left) 2024-04-26T00:31:11.337 INFO:tasks.rados.rados.0.smithi045.stdout:1878: rollback oid 21 current snap is 214 2024-04-26T00:31:11.337 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 21 to 153 2024-04-26T00:31:11.338 INFO:tasks.rados.rados.0.smithi045.stdout:1879: read oid 45 snap -1 2024-04-26T00:31:11.338 INFO:tasks.rados.rados.0.smithi045.stdout:1879: expect (ObjNum 635 snap 198 seq_num 635) 2024-04-26T00:31:11.338 INFO:tasks.rados.rados.0.smithi045.stdout:1880: copy_from oid 35 from oid 43 current snap is 214 2024-04-26T00:31:11.338 INFO:tasks.rados.rados.0.smithi045.stdout:1881: delete oid 7 current snap is 214 2024-04-26T00:31:11.346 INFO:tasks.rados.rados.0.smithi045.stdout:1878: finishing rollback tid 1 to smithi04526579-21 2024-04-26T00:31:11.348 INFO:tasks.rados.rados.0.smithi045.stdout:1881: done (3 left) 2024-04-26T00:31:11.348 INFO:tasks.rados.rados.0.smithi045.stdout:1882: snap_create 2024-04-26T00:31:11.349 INFO:tasks.rados.rados.0.smithi045.stdout:1878: finishing rollback tid 2 to smithi04526579-21 2024-04-26T00:31:11.349 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 1049 (ObjNum 254 snap 64 seq_num 254) dirty exists 2024-04-26T00:31:11.349 INFO:tasks.rados.rados.0.smithi045.stdout:1879: expect (ObjNum 635 snap 198 seq_num 635) 2024-04-26T00:31:11.530 INFO:tasks.rados.rados.0.smithi045.stdout:1880: finishing copy_from racing read to smithi04526579-35 2024-04-26T00:31:11.530 INFO:tasks.rados.rados.0.smithi045.stdout:1880: finishing copy_from to smithi04526579-35 2024-04-26T00:31:11.530 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 1058 (ObjNum 646 snap 202 seq_num 646) dirty exists 2024-04-26T00:31:12.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:12.013+0000 7ff8baffd640 1 --2- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc138430 0x7ff8bc138810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:12.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:12.013+0000 7ff8baffd640 10 monclient: get_auth_request con 0x7ff8bc138430 auth_method 0 2024-04-26T00:31:12.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:12.013+0000 7ff8baffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:12.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:12.013+0000 7ff8baffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:12.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:12.013+0000 7ff8baffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:31:12.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:12.013+0000 7ff8baffd640 1 -- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc138430 msgr2=0x7ff8bc138810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:31:12.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:12.013+0000 7ff8baffd640 1 -- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc138430 msgr2=0x7ff8bc138810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:31:12.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:12.013+0000 7ff8baffd640 1 --2- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc138430 0x7ff8bc138810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:31:12.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:12.013+0000 7ff8baffd640 1 --2- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc138430 0x7ff8bc138810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:31:12.345 INFO:tasks.rados.rados.0.smithi045.stdout:1878: done (3 left) 2024-04-26T00:31:12.345 INFO:tasks.rados.rados.0.smithi045.stdout:1879: done (2 left) 2024-04-26T00:31:12.345 INFO:tasks.rados.rados.0.smithi045.stdout:1880: done (1 left) 2024-04-26T00:31:12.345 INFO:tasks.rados.rados.0.smithi045.stdout:1882: done (0 left) 2024-04-26T00:31:12.345 INFO:tasks.rados.rados.0.smithi045.stdout:1883: rollback oid 25 current snap is 215 2024-04-26T00:31:12.345 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 25 to 195 2024-04-26T00:31:12.346 INFO:tasks.rados.rados.0.smithi045.stdout:1884: delete oid 28 current snap is 215 2024-04-26T00:31:12.359 INFO:tasks.rados.rados.0.smithi045.stdout:1884: done (1 left) 2024-04-26T00:31:12.359 INFO:tasks.rados.rados.0.smithi045.stdout:1885: write oid 46 current snap is 215 2024-04-26T00:31:12.359 INFO:tasks.rados.rados.0.smithi045.stdout:1885: seq_num 690 ranges {796830=793580,2360693=691568} 2024-04-26T00:31:12.385 INFO:tasks.rados.rados.0.smithi045.stdout:1885: writing smithi04526579-46 from 796830 to 1590410 tid 1 2024-04-26T00:31:12.408 INFO:tasks.rados.rados.0.smithi045.stdout:1885: writing smithi04526579-46 from 2360693 to 3052261 tid 2 2024-04-26T00:31:12.408 INFO:tasks.rados.rados.0.smithi045.stdout:1886: snap_create 2024-04-26T00:31:12.408 INFO:tasks.rados.rados.0.smithi045.stdout:1883: finishing rollback tid 0 to smithi04526579-25 2024-04-26T00:31:12.408 INFO:tasks.rados.rados.0.smithi045.stdout:1883: finishing rollback tid 1 to smithi04526579-25 2024-04-26T00:31:12.408 INFO:tasks.rados.rados.0.smithi045.stdout:1883: finishing rollback tid 2 to smithi04526579-25 2024-04-26T00:31:12.408 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 1574 (ObjNum 604 snap 186 seq_num 604) dirty exists 2024-04-26T00:31:12.408 INFO:tasks.rados.rados.0.smithi045.stdout:1885: finishing write tid 1 to smithi04526579-46 2024-04-26T00:31:12.414 INFO:tasks.rados.rados.0.smithi045.stdout:1885: finishing write tid 2 to smithi04526579-46 2024-04-26T00:31:12.415 INFO:tasks.rados.rados.0.smithi045.stdout:1885: finishing write tid 3 to smithi04526579-46 2024-04-26T00:31:12.415 INFO:tasks.rados.rados.0.smithi045.stdout:1885: finishing write tid 4 to smithi04526579-46 2024-04-26T00:31:12.415 INFO:tasks.rados.rados.0.smithi045.stdout:1885: oid 46 updating version 0 to 1435 2024-04-26T00:31:12.415 INFO:tasks.rados.rados.0.smithi045.stdout:1885: oid 46 version 1435 is already newer than 1434 2024-04-26T00:31:12.415 INFO:tasks.rados.rados.0.smithi045.stdout:1885: oid 46 updating version 1435 to 1436 2024-04-26T00:31:12.415 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 1436 (ObjNum 690 snap 215 seq_num 690) dirty exists 2024-04-26T00:31:12.415 INFO:tasks.rados.rados.0.smithi045.stdout:1885: left oid 46 (ObjNum 690 snap 215 seq_num 690) 2024-04-26T00:31:13.353 INFO:tasks.rados.rados.0.smithi045.stdout:1883: done (2 left) 2024-04-26T00:31:13.353 INFO:tasks.rados.rados.0.smithi045.stdout:1885: done (1 left) 2024-04-26T00:31:13.353 INFO:tasks.rados.rados.0.smithi045.stdout:1886: done (0 left) 2024-04-26T00:31:13.354 INFO:tasks.rados.rados.0.smithi045.stdout:1887: write (excl) oid 19 current snap is 216 2024-04-26T00:31:13.354 INFO:tasks.rados.rados.0.smithi045.stdout:1887: seq_num 691 ranges {413637=468829,1619175=435226,2542935=758255} 2024-04-26T00:31:13.365 INFO:tasks.rados.rados.0.smithi045.stdout:1887: writing smithi04526579-19 from 413637 to 882466 tid 1 2024-04-26T00:31:13.382 INFO:tasks.rados.rados.0.smithi045.stdout:1887: writing smithi04526579-19 from 1619175 to 2054401 tid 2 2024-04-26T00:31:13.400 INFO:tasks.rados.rados.0.smithi045.stdout:1887: writing smithi04526579-19 from 2542935 to 3301190 tid 3 2024-04-26T00:31:13.400 INFO:tasks.rados.rados.0.smithi045.stdout:1888: snap_create 2024-04-26T00:31:13.400 INFO:tasks.rados.rados.0.smithi045.stdout:1887: finishing write tid 1 to smithi04526579-19 2024-04-26T00:31:13.400 INFO:tasks.rados.rados.0.smithi045.stdout:1887: finishing write tid 2 to smithi04526579-19 2024-04-26T00:31:13.410 INFO:tasks.rados.rados.0.smithi045.stdout:1887: finishing write tid 3 to smithi04526579-19 2024-04-26T00:31:13.411 INFO:tasks.rados.rados.0.smithi045.stdout:1887: finishing write tid 4 to smithi04526579-19 2024-04-26T00:31:13.411 INFO:tasks.rados.rados.0.smithi045.stdout:1887: finishing write tid 5 to smithi04526579-19 2024-04-26T00:31:13.411 INFO:tasks.rados.rados.0.smithi045.stdout:1887: oid 19 updating version 0 to 1192 2024-04-26T00:31:13.411 INFO:tasks.rados.rados.0.smithi045.stdout:1887: oid 19 updating version 1192 to 1193 2024-04-26T00:31:13.411 INFO:tasks.rados.rados.0.smithi045.stdout:1887: oid 19 updating version 1193 to 1194 2024-04-26T00:31:13.412 INFO:tasks.rados.rados.0.smithi045.stdout:1887: oid 19 version 1194 is already newer than 1191 2024-04-26T00:31:13.412 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 1194 (ObjNum 691 snap 216 seq_num 691) dirty exists 2024-04-26T00:31:13.412 INFO:tasks.rados.rados.0.smithi045.stdout:1887: left oid 19 (ObjNum 691 snap 216 seq_num 691) 2024-04-26T00:31:13.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:13.601+0000 7ff8a2ffd640 10 monclient: tick 2024-04-26T00:31:13.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:13.601+0000 7ff8a2ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:31:13.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:13.601+0000 7ff8a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:31:13.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:13.601+0000 7ff8a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:31:13.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:13.617+0000 7ff8baffd640 1 --2- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc138430 0x7ff8bc138810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:13.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:13.617+0000 7ff8baffd640 10 monclient: get_auth_request con 0x7ff8bc138430 auth_method 0 2024-04-26T00:31:13.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:13.617+0000 7ff8baffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:13.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:13.617+0000 7ff8baffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:13.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:13.617+0000 7ff8baffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:31:13.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:13.617+0000 7ff8baffd640 1 -- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc138430 msgr2=0x7ff8bc138810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:31:13.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:13.617+0000 7ff8baffd640 1 -- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc138430 msgr2=0x7ff8bc138810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:31:13.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:13.617+0000 7ff8baffd640 1 --2- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc138430 0x7ff8bc138810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:31:13.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:13.617+0000 7ff8baffd640 1 --2- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc138430 0x7ff8bc138810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-26T00:31:14.356 INFO:tasks.rados.rados.0.smithi045.stdout:1887: done (1 left) 2024-04-26T00:31:14.356 INFO:tasks.rados.rados.0.smithi045.stdout:1888: done (0 left) 2024-04-26T00:31:14.357 INFO:tasks.rados.rados.0.smithi045.stdout:1889: read oid 28 snap 203 2024-04-26T00:31:14.357 INFO:tasks.rados.rados.0.smithi045.stdout:1889: expect (ObjNum 655 snap 202 seq_num 655) 2024-04-26T00:31:14.357 INFO:tasks.rados.rados.0.smithi045.stdout:1890: copy_from oid 45 from oid 46 current snap is 217 2024-04-26T00:31:14.357 INFO:tasks.rados.rados.0.smithi045.stdout:1891: read oid 12 snap -1 2024-04-26T00:31:14.357 INFO:tasks.rados.rados.0.smithi045.stdout:1891: expect (ObjNum 401 snap 122 seq_num 401) 2024-04-26T00:31:14.357 INFO:tasks.rados.rados.0.smithi045.stdout:1892: snap_remove snap 184 2024-04-26T00:31:14.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.597+0000 7ff8a2ffd640 10 monclient: tick 2024-04-26T00:31:14.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.597+0000 7ff8a2ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:31:14.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.597+0000 7ff8a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:31:14.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.597+0000 7ff8a2ffd640 5 monclient: _check_tell_commands timeout tell command 1 2024-04-26T00:31:14.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.597+0000 7ff8a2ffd640 1 -- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc138430 msgr2=0x7ff8bc138810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:31:14.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.597+0000 7ff8a2ffd640 1 --2- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc138430 0x7ff8bc138810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:14.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.597+0000 7ff8a2ffd640 1 --2- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff894003550 0x7ff8940059e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:14.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.597+0000 7ff8a2ffd640 1 -- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc138430 msgr2=0x7ff8bc138810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:31:14.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.597+0000 7ff8a2ffd640 1 --2- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc138430 0x7ff8bc138810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:14.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.597+0000 7ff8a2ffd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:14.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.597+0000 7ff8a2ffd640 1 -- 172.21.15.37:0/3448445169 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff8bc13bb90 con 0x7ff894003550 2024-04-26T00:31:14.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.597+0000 7ff8baffd640 1 --2- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff894003550 0x7ff8940059e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:14.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.597+0000 7ff8baffd640 10 monclient: get_auth_request con 0x7ff894003550 auth_method 0 2024-04-26T00:31:14.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.597+0000 7ff8baffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:14.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.597+0000 7ff8baffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:14.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.597+0000 7ff8baffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:14.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.597+0000 7ff8baffd640 1 -- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff894003550 msgr2=0x7ff8940059e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:31:14.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.597+0000 7ff8baffd640 1 -- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff894003550 msgr2=0x7ff8940059e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:31:14.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.597+0000 7ff8baffd640 1 --2- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff894003550 0x7ff8940059e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:31:14.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.597+0000 7ff8baffd640 1 --2- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff894003550 0x7ff8940059e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:31:14.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.801+0000 7ff8baffd640 1 --2- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff894003550 0x7ff8940059e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:14.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.801+0000 7ff8baffd640 10 monclient: get_auth_request con 0x7ff894003550 auth_method 0 2024-04-26T00:31:14.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.801+0000 7ff8baffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:14.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.801+0000 7ff8baffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:14.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.801+0000 7ff8baffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:31:14.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.801+0000 7ff8baffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:14.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.801+0000 7ff8baffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:14.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.801+0000 7ff8baffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:14.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.801+0000 7ff8baffd640 10 monclient(hunting): handle_auth_done global_id 134098 payload 824 2024-04-26T00:31:14.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.801+0000 7ff8baffd640 1 -- 172.21.15.37:0/3448445169 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff8940060c0 con 0x7ff894003550 2024-04-26T00:31:14.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.801+0000 7ff8baffd640 1 -- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff894003550 msgr2=0x7ff8940059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:14.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.801+0000 7ff8baffd640 1 --2- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff894003550 0x7ff8940059e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff8b00ad000 tx=0x7ff8b0019130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:14.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.805+0000 7ff8a3fff640 1 -- 172.21.15.37:0/3448445169 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b00b0040 con 0x7ff894003550 2024-04-26T00:31:14.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.805+0000 7ff8a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:14.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.805+0000 7ff8a3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:31:14.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.805+0000 7ff8a3fff640 10 monclient: dump: 2024-04-26T00:31:14.797 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:14.797 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:14.797 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:14.797 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:14.797 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:14.797 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:14.797 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:14.797 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:14.797 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:14.798 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:14.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.805+0000 7ff8a3fff640 1 -- 172.21.15.37:0/3448445169 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff8b00b4610 con 0x7ff894003550 2024-04-26T00:31:14.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.805+0000 7ff8a3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:14.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.805+0000 7ff8a3fff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:31:14.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.805+0000 7ff8a3fff640 1 -- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff894003550 msgr2=0x7ff8940059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:14.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.805+0000 7ff8a3fff640 1 --2- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff894003550 0x7ff8940059e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff8b00ad000 tx=0x7ff8b0019130 comp rx=0 tx=0).stop 2024-04-26T00:31:14.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.805+0000 7ff8a3fff640 1 -- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff894003550 msgr2=0x7ff8940059e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:31:14.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.805+0000 7ff8a3fff640 1 --2- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff894003550 0x7ff8940059e0 secure :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff8b00ad000 tx=0x7ff8b0019130 comp rx=0 tx=0).stop 2024-04-26T00:31:14.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.813+0000 7ff8c1d51640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:31:14.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.813+0000 7ff8c1d51640 1 --2- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc138d50 0x7ff8bc0a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:14.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.813+0000 7ff8c1d51640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:14.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.813+0000 7ff8c1d51640 1 -- 172.21.15.37:0/3448445169 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff8bc13b470 con 0x7ff8bc138d50 2024-04-26T00:31:14.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.813+0000 7ff8baffd640 1 --2- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc138d50 0x7ff8bc0a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:14.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.813+0000 7ff8baffd640 10 monclient: get_auth_request con 0x7ff8bc138d50 auth_method 0 2024-04-26T00:31:14.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.813+0000 7ff8baffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:14.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.813+0000 7ff8baffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:14.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.813+0000 7ff8baffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:14.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.813+0000 7ff8baffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:14.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.813+0000 7ff8baffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:14.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.813+0000 7ff8baffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:14.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.813+0000 7ff8baffd640 10 monclient(hunting): handle_auth_done global_id 134101 payload 824 2024-04-26T00:31:14.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.813+0000 7ff8baffd640 1 -- 172.21.15.37:0/3448445169 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff8bc075f10 con 0x7ff8bc138d50 2024-04-26T00:31:14.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.817+0000 7ff8baffd640 1 -- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc138d50 msgr2=0x7ff8bc0a18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:14.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.817+0000 7ff8baffd640 1 --2- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc138d50 0x7ff8bc0a18f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff8b001c3d0 tx=0x7ff8b00a8000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:14.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.817+0000 7ff8a3fff640 1 -- 172.21.15.37:0/3448445169 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b00b0050 con 0x7ff8bc138d50 2024-04-26T00:31:14.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.817+0000 7ff8a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:14.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.817+0000 7ff8a3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:31:14.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.817+0000 7ff8a3fff640 10 monclient: dump: 2024-04-26T00:31:14.810 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:14.810 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:14.810 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:14.810 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:14.810 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:14.810 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:14.810 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:14.810 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:14.811 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:14.811 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:14.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.817+0000 7ff8a3fff640 1 -- 172.21.15.37:0/3448445169 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7ff8b00c2070 con 0x7ff8bc138d50 2024-04-26T00:31:14.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.817+0000 7ff8a3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:31:14.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.817+0000 7ff8a3fff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:31:14.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.817+0000 7ff8a3fff640 1 -- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc138d50 msgr2=0x7ff8bc0a18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:14.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.817+0000 7ff8a3fff640 1 --2- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc138d50 0x7ff8bc0a18f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff8b001c3d0 tx=0x7ff8b00a8000 comp rx=0 tx=0).stop 2024-04-26T00:31:14.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.817+0000 7ff8a3fff640 1 -- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc138d50 msgr2=0x7ff8bc0a18f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:31:14.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.817+0000 7ff8a3fff640 1 --2- 172.21.15.37:0/3448445169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8bc138d50 0x7ff8bc0a18f0 secure :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff8b001c3d0 tx=0x7ff8b00a8000 comp rx=0 tx=0).stop 2024-04-26T00:31:14.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.817+0000 7ff8bbfff640 1 -- 172.21.15.37:0/3448445169 reap_dead start 2024-04-26T00:31:14.812 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:31:14.812 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:31:14.812 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:31:14.812 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:31:14.812 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 86, 2024-04-26T00:31:14.813 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 5175, 2024-04-26T00:31:14.813 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:31:14.813 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:31:14.813 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:31:14.813 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:31:14.813 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:31:14.813 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 5, 2024-04-26T00:31:14.813 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:31:14.813 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:31:14.813 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:31:14.813 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:31:14.813 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:31:14.813 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:31:14.813 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:31:14.813 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:31:14.814 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:31:14.814 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:31:14.814 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:31:14.814 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:31:14.814 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:31:14.814 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:31:14.814 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:31:14.814 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:31:14.814 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:31:14.814 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:31:14.814 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:31:14.814 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:31:14.815 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:31:14.815 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:31:14.815 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:31:14.815 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:31:14.815 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:31:14.815 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:31:14.815 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:31:14.815 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:31:14.815 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:14.815 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:14.815 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:31:14.815 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:31:14.815 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:31:14.815 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:31:14.815 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:31:14.816 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:31:14.816 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:31:14.816 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:31:14.816 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:31:14.816 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:31:14.816 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:31:14.816 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:31:14.816 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:31:14.816 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:31:14.816 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:31:14.816 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:31:14.816 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:31:14.817 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:31:14.817 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:31:14.817 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:31:14.817 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:31:14.817 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:31:14.817 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:31:14.817 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:31:14.817 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:31:14.817 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:31:14.817 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:31:14.817 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:31:14.817 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:31:14.817 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:31:14.817 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:14.817 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:31:14.818 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:14.818 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:31:14.818 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:31:14.818 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:31:14.818 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:31:14.818 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:14.818 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:31:14.818 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:31:14.818 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:14.818 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:14.818 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:14.818 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:31:14.818 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:31:14.818 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:14.819 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:14.819 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:14.819 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:14.819 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:31:14.819 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:31:14.819 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:31:14.819 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:31:14.819 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:31:14.819 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:14.819 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:14.819 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:31:14.819 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:31:14.819 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:31:14.819 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:31:14.819 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:14.820 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:31:14.820 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:31:14.820 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:14.820 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:14.820 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:14.820 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:31:14.820 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:31:14.820 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:14.820 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:14.820 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:14.820 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:14.820 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:31:14.820 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:31:14.820 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:31:14.821 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:31:14.821 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:31:14.821 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:14.821 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:14.821 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:31:14.821 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:31:14.821 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:31:14.821 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:31:14.821 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:14.821 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:31:14.821 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:31:14.821 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:14.821 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:14.821 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:14.822 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:31:14.822 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:31:14.822 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:14.822 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:14.822 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:14.822 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:14.822 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:31:14.822 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:31:14.822 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:31:14.822 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:31:14.822 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:31:14.822 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:14.822 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:14.822 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:14.823 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:31:14.823 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:31:14.823 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:14.823 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:31:14.823 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:31:14.823 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:31:14.823 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:14.823 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:31:14.823 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:31:14.823 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:14.823 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:31:14.823 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:31:14.823 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:31:14.823 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:14.824 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:14.824 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:14.824 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:31:14.824 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:31:14.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.817+0000 7ff8c1d51640 1 -- 172.21.15.37:0/3448445169 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff8a8035f00 msgr2=0x7ff8a80383c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:14.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.817+0000 7ff8c1d51640 1 --2- 172.21.15.37:0/3448445169 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff8a8035f00 0x7ff8a80383c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff8b4006aa0 tx=0x7ff8b4008040 comp rx=0 tx=0).stop 2024-04-26T00:31:14.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.817+0000 7ff8c1d51640 10 monclient: shutdown 2024-04-26T00:31:14.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.817+0000 7ff8c1d51640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:31:14.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.817+0000 7ff8c1d51640 1 -- 172.21.15.37:0/3448445169 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff8bc136f20 msgr2=0x7ff8bc1ceda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:14.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.817+0000 7ff8c1d51640 1 --2- 172.21.15.37:0/3448445169 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff8bc136f20 0x7ff8bc1ceda0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac00cc10 tx=0x7ff8ac00a430 comp rx=0 tx=0).stop 2024-04-26T00:31:14.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.817+0000 7ff8c1d51640 1 -- 172.21.15.37:0/3448445169 shutdown_connections 2024-04-26T00:31:14.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.817+0000 7ff8c1d51640 1 --2- 172.21.15.37:0/3448445169 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff8a8035f00 0x7ff8a80383c0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:14.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.817+0000 7ff8c1d51640 1 --2- 172.21.15.37:0/3448445169 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff8bc136f20 0x7ff8bc1ceda0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:14.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.817+0000 7ff8c1d51640 1 -- 172.21.15.37:0/3448445169 >> 172.21.15.37:0/3448445169 conn(0x7ff8bc0a61b0 msgr2=0x7ff8bc0a7a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:31:14.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.817+0000 7ff8c1d51640 1 -- 172.21.15.37:0/3448445169 shutdown_connections 2024-04-26T00:31:14.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.817+0000 7ff8c1d51640 1 -- 172.21.15.37:0/3448445169 wait complete. 2024-04-26T00:31:14.840 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:31:14.840 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-26T00:31:14.840 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-26T00:31:14.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.977+0000 7f0378819640 10 monclient: get_monmap_and_config 2024-04-26T00:31:14.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.977+0000 7f0378819640 10 monclient: build_initial_monmap 2024-04-26T00:31:14.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.977+0000 7f0378819640 10 monclient: monmap: 2024-04-26T00:31:14.972 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:31:14.972 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:14.972 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:31:14.983020+0000 2024-04-26T00:31:14.972 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:31:14.983020+0000 2024-04-26T00:31:14.972 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:31:14.972 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:14.972 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:31:14.972 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:31:14.973 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:31:14.973 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:31:14.973 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:31:14.973 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:14.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.977+0000 7f0378819640 1 Processor -- start 2024-04-26T00:31:14.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.981+0000 7f0378819640 1 -- start start 2024-04-26T00:31:14.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.981+0000 7f0378819640 10 monclient: init 2024-04-26T00:31:14.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.981+0000 7f0378819640 10 monclient: _reopen_session rank -1 2024-04-26T00:31:14.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.981+0000 7f0378819640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-04-26T00:31:14.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.981+0000 7f0378819640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0374136e10 0x7f037409c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:14.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.981+0000 7f0378819640 10 monclient(hunting): picked mon.noname-c con 0x7f0374136e10 addr v2:172.21.15.45:3300/0 2024-04-26T00:31:14.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.981+0000 7f0378819640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f03741371f0 0x7f03741375d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:14.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.981+0000 7f0378819640 10 monclient(hunting): picked mon.noname-b con 0x7f03741371f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:31:14.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.981+0000 7f0378819640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0374135d90 addr v1:172.21.15.45:6789/0 2024-04-26T00:31:14.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.981+0000 7f0378819640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f037409b680 con 0x7f0374135d90 2024-04-26T00:31:14.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.981+0000 7f0378819640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:14.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.981+0000 7f0378819640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f037409b8f0 con 0x7f0374136e10 2024-04-26T00:31:14.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.981+0000 7f0378819640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:14.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.981+0000 7f0378819640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f037409bbd0 con 0x7f03741371f0 2024-04-26T00:31:14.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.981+0000 7f0378819640 10 monclient(hunting): _renew_subs 2024-04-26T00:31:14.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.981+0000 7f0378819640 10 monclient(hunting): authenticate will time out at 1656.288693s 2024-04-26T00:31:14.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.981+0000 7f0371d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f03741371f0 0x7f03741375d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:14.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.981+0000 7f0371d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f03741371f0 0x7f03741375d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:33674/0 (socket says 172.21.15.37:33674) 2024-04-26T00:31:14.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.981+0000 7f0371d9b640 1 -- 172.21.15.37:0/2743887124 learned_addr learned my addr 172.21.15.37:0/2743887124 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:31:14.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.981+0000 7f0371d9b640 10 monclient(hunting): get_auth_request con 0x7f03741371f0 auth_method 0 2024-04-26T00:31:14.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.981+0000 7f0371d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:14.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.981+0000 7f0371d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:14.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.981+0000 7f0371d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:14.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.981+0000 7f0371d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:14.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.981+0000 7f0371d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:14.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.981+0000 7f0371d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:14.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.981+0000 7f037259c640 1 --2- 172.21.15.37:0/2743887124 >> v2:172.21.15.45:3300/0 conn(0x7f0374136e10 0x7f037409c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:14.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.981+0000 7f037259c640 10 monclient(hunting): get_auth_request con 0x7f0374136e10 auth_method 0 2024-04-26T00:31:14.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.981+0000 7f037259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:14.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.981+0000 7f037259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:14.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.981+0000 7f037259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:14.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.981+0000 7f0371d9b640 10 monclient(hunting): handle_auth_done global_id 84575 payload 274 2024-04-26T00:31:14.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.981+0000 7f0371d9b640 1 -- 172.21.15.37:0/2743887124 >> v2:172.21.15.45:3300/0 conn(0x7f0374136e10 msgr2=0x7f037409c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:31:14.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.981+0000 7f0371d9b640 1 --2- 172.21.15.37:0/2743887124 >> v2:172.21.15.45:3300/0 conn(0x7f0374136e10 0x7f037409c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:14.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.981+0000 7f0371d9b640 1 -- 172.21.15.37:0/2743887124 >> v1:172.21.15.45:6789/0 conn(0x7f0374135d90 legacy=0x7f0374136240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:31:14.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.981+0000 7f0371d9b640 10 monclient: _finish_hunting 0 2024-04-26T00:31:14.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.981+0000 7f0371d9b640 1 monclient: found mon.noname-b 2024-04-26T00:31:14.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.981+0000 7f0371d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:31:14.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.981+0000 7f0371d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:31:14.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.981+0000 7f0371d9b640 1 -- 172.21.15.37:0/2743887124 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0374137b10 con 0x7f03741371f0 2024-04-26T00:31:14.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.985+0000 7f0371d9b640 1 -- 172.21.15.37:0/2743887124 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f03741371f0 msgr2=0x7f03741375d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:14.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.985+0000 7f0371d9b640 1 --2- 172.21.15.37:0/2743887124 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f03741371f0 0x7f03741375d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0368009870 tx=0x7f036801c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9626bf653b26a967 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:14.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.985+0000 7f037159a640 1 -- 172.21.15.37:0/2743887124 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03680b1070 con 0x7f03741371f0 2024-04-26T00:31:14.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.985+0000 7f037159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:14.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.985+0000 7f037159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:31:14.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.985+0000 7f037159a640 10 monclient: dump: 2024-04-26T00:31:14.983 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:14.983 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:14.983 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:14.983 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:14.983 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:14.983 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:14.983 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:14.983 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:14.984 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:14.984 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:14.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.985+0000 7f037159a640 10 monclient: _finish_auth 0 2024-04-26T00:31:14.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.985+0000 7f037159a640 10 monclient: _check_auth_tickets 2024-04-26T00:31:14.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.985+0000 7f037159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:31:14.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.985+0000 7f037159a640 1 -- 172.21.15.37:0/2743887124 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03680ac620 con 0x7f03741371f0 2024-04-26T00:31:14.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.985+0000 7f037159a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:31:14.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.985+0000 7f037159a640 1 -- 172.21.15.37:0/2743887124 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03680ac9f0 con 0x7f03741371f0 2024-04-26T00:31:14.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.985+0000 7f037159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:14.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.985+0000 7f037159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:31:14.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.985+0000 7f037159a640 10 monclient: dump: 2024-04-26T00:31:14.984 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:14.984 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:14.984 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:14.985 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:14.985 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:14.985 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:14.985 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:14.985 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:14.985 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:14.985 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:14.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.985+0000 7f0378819640 5 monclient: authenticate success, global_id 84575 2024-04-26T00:31:14.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.985+0000 7f0378819640 10 monclient: get_monmap_and_config success 2024-04-26T00:31:14.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.985+0000 7f0378819640 4 set_mon_vals no callback set 2024-04-26T00:31:14.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.985+0000 7f0378819640 10 monclient: shutdown 2024-04-26T00:31:14.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.985+0000 7f0378819640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:31:14.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.985+0000 7f0378819640 1 -- 172.21.15.37:0/2743887124 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f03741371f0 msgr2=0x7f03741375d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:14.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.985+0000 7f0378819640 1 --2- 172.21.15.37:0/2743887124 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f03741371f0 0x7f03741375d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0368009870 tx=0x7f036801c3e0 comp rx=0 tx=0).stop 2024-04-26T00:31:14.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.985+0000 7f0378819640 1 -- 172.21.15.37:0/2743887124 shutdown_connections 2024-04-26T00:31:14.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.985+0000 7f0378819640 1 --2- 172.21.15.37:0/2743887124 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f03741371f0 0x7f03741375d0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:14.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.985+0000 7f0378819640 1 --2- 172.21.15.37:0/2743887124 >> v2:172.21.15.45:3300/0 conn(0x7f0374136e10 0x7f037409c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:14.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.985+0000 7f0378819640 1 -- 172.21.15.37:0/2743887124 >> 172.21.15.37:0/2743887124 conn(0x7f03740a64e0 msgr2=0x7f03740a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:31:14.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.985+0000 7f0378819640 1 -- 172.21.15.37:0/2743887124 shutdown_connections 2024-04-26T00:31:14.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.985+0000 7f0378819640 1 -- 172.21.15.37:0/2743887124 wait complete. 2024-04-26T00:31:14.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.985+0000 7f0378819640 10 monclient: build_initial_monmap 2024-04-26T00:31:14.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.985+0000 7f0378819640 10 monclient: monmap: 2024-04-26T00:31:14.986 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:31:14.986 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:31:14.986 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:31:14.986 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:31:14.986 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:31:14.986 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:31:14.987 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:31:14.987 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:31:14.987 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:31:14.987 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:14.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.985+0000 7f0378819640 1 Processor -- start 2024-04-26T00:31:14.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.985+0000 7f0378819640 1 -- start start 2024-04-26T00:31:14.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.985+0000 7f0378819640 10 monclient: init 2024-04-26T00:31:14.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0378819640 10 monclient: _reopen_session rank -1 2024-04-26T00:31:14.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0378819640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:31:14.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0378819640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0374135d90 0x7f0374071400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:14.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0378819640 10 monclient(hunting): picked mon.noname-b con 0x7f0374135d90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:31:14.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0378819640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0374136e10 0x7f037406e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:14.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0378819640 10 monclient(hunting): picked mon.noname-c con 0x7f0374136e10 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:31:14.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0378819640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f03741371f0 0x7f037406ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:14.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0378819640 10 monclient(hunting): picked mon.noname-a con 0x7f03741371f0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:31:14.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0378819640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:14.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0378819640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f0374141f10 con 0x7f03741371f0 2024-04-26T00:31:14.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0378819640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:14.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0378819640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0374141d90 con 0x7f0374135d90 2024-04-26T00:31:14.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0378819640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:14.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0378819640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0374142090 con 0x7f0374136e10 2024-04-26T00:31:14.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0378819640 10 monclient(hunting): _renew_subs 2024-04-26T00:31:14.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0378819640 10 monclient(hunting): authenticate will time out at 1656.296323s 2024-04-26T00:31:14.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0371d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0374136e10 0x7f037406e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:14.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0371d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0374136e10 0x7f037406e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:33682/0 (socket says 172.21.15.37:33682) 2024-04-26T00:31:14.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0371d9b640 1 -- 172.21.15.37:0/203863946 learned_addr learned my addr 172.21.15.37:0/203863946 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:31:14.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0371d9b640 10 monclient(hunting): get_auth_request con 0x7f0374136e10 auth_method 0 2024-04-26T00:31:14.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0371d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:14.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0371d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:14.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0371d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:14.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0372d9d640 1 --2- 172.21.15.37:0/203863946 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f03741371f0 0x7f037406ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:14.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f037259c640 1 --2- 172.21.15.37:0/203863946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0374135d90 0x7f0374071400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:14.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0371d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:14.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0371d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:14.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0371d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:14.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0372d9d640 10 monclient(hunting): get_auth_request con 0x7f03741371f0 auth_method 0 2024-04-26T00:31:14.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0372d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:14.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0372d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:14.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0372d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:14.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f037259c640 10 monclient(hunting): get_auth_request con 0x7f0374135d90 auth_method 0 2024-04-26T00:31:14.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f037259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:14.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f037259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:14.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f037259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:14.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0372d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:14.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0372d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:14.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0372d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:14.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f037259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:14.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f037259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:14.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f037259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:14.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0371d9b640 10 monclient(hunting): handle_auth_done global_id 84578 payload 824 2024-04-26T00:31:14.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0371d9b640 1 -- 172.21.15.37:0/203863946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0374135d90 msgr2=0x7f0374071400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:14.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0371d9b640 1 --2- 172.21.15.37:0/203863946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0374135d90 0x7f0374071400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:14.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0371d9b640 1 -- 172.21.15.37:0/203863946 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f03741371f0 msgr2=0x7f037406ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:14.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0371d9b640 1 --2- 172.21.15.37:0/203863946 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f03741371f0 0x7f037406ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:14.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0371d9b640 10 monclient: _finish_hunting 0 2024-04-26T00:31:14.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0371d9b640 1 monclient: found mon.noname-c 2024-04-26T00:31:14.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0371d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:31:14.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0371d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:31:14.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0371d9b640 1 -- 172.21.15.37:0/203863946 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f037409aca0 con 0x7f0374136e10 2024-04-26T00:31:14.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0371d9b640 1 -- 172.21.15.37:0/203863946 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0374136e10 msgr2=0x7f037406e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:14.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0371d9b640 1 --2- 172.21.15.37:0/203863946 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0374136e10 0x7f037406e520 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f036801a010 tx=0x7f036801ac20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:14.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0362ffd640 1 -- 172.21.15.37:0/203863946 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03680bb020 con 0x7f0374136e10 2024-04-26T00:31:14.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0362ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:14.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0362ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:31:14.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0362ffd640 10 monclient: dump: 2024-04-26T00:31:14.993 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:14.993 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:14.994 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:14.994 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:14.994 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:14.994 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:14.994 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:14.994 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:14.994 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:14.994 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:14.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0362ffd640 10 monclient: _finish_auth 0 2024-04-26T00:31:14.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0362ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:31:14.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0362ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:31:14.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0362ffd640 1 -- 172.21.15.37:0/203863946 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03680b7420 con 0x7f0374136e10 2024-04-26T00:31:14.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0378819640 5 monclient: authenticate success, global_id 84578 2024-04-26T00:31:14.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0378819640 10 monclient: _renew_subs 2024-04-26T00:31:14.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0378819640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:14.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0378819640 1 -- 172.21.15.37:0/203863946 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03740a7df0 con 0x7f0374136e10 2024-04-26T00:31:14.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0362ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:31:14.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0362ffd640 1 -- 172.21.15.37:0/203863946 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03680b1040 con 0x7f0374136e10 2024-04-26T00:31:14.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0362ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:14.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0362ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:31:14.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0362ffd640 10 monclient: dump: 2024-04-26T00:31:14.995 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:14.995 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:14.995 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:14.995 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:14.995 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:14.995 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:14.996 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:14.996 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:14.996 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:14.996 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:14.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0370d99640 4 set_mon_vals no callback set 2024-04-26T00:31:14.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0378819640 10 monclient: _renew_subs 2024-04-26T00:31:14.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0378819640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:14.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.989+0000 7f0378819640 1 -- 172.21.15.37:0/203863946 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f03740724b0 con 0x7f0374136e10 2024-04-26T00:31:14.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.993+0000 7f0362ffd640 1 -- 172.21.15.37:0/203863946 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f03680be020 con 0x7f0374136e10 2024-04-26T00:31:14.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.993+0000 7f0378819640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:14.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.993+0000 7f0378819640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:14.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.993+0000 7f0378819640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:14.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.993+0000 7f0378819640 1 -- 172.21.15.37:0/203863946 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0374075910 con 0x7f0374136e10 2024-04-26T00:31:14.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.993+0000 7f0362ffd640 1 --2- 172.21.15.37:0/203863946 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0354035eb0 0x7f0354038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:14.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.993+0000 7f0362ffd640 1 -- 172.21.15.37:0/203863946 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(431..431 src has 370..431) ==== 3963+0+0 (secure 0 0 0) 0x7f03680e29b0 con 0x7f0374136e10 2024-04-26T00:31:14.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.993+0000 7f037259c640 1 --2- 172.21.15.37:0/203863946 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0354035eb0 0x7f0354038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:14.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.993+0000 7f037259c640 10 monclient: get_auth_request con 0x7f0354035eb0 auth_method 0 2024-04-26T00:31:14.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.997+0000 7f037259c640 1 --2- 172.21.15.37:0/203863946 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0354035eb0 0x7f0354038370 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f0364006aa0 tx=0x7f0364008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:14.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.997+0000 7f0362ffd640 1 -- 172.21.15.37:0/203863946 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f03680c5070 con 0x7f0374136e10 2024-04-26T00:31:14.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.997+0000 7f0362ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:14.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:14.997+0000 7f0362ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:31:15.125 INFO:tasks.rados.rados.0.smithi045.stdout:1892: done (3 left) 2024-04-26T00:31:15.125 INFO:tasks.rados.rados.0.smithi045.stdout:1893: copy_from oid 32 from oid 15 current snap is 217 2024-04-26T00:31:15.125 INFO:tasks.rados.rados.0.smithi045.stdout:1894: delete oid 48 current snap is 217 2024-04-26T00:31:15.125 INFO:tasks.rados.rados.0.smithi045.stdout:1891: expect (ObjNum 401 snap 122 seq_num 401) 2024-04-26T00:31:15.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:15.245+0000 7f0378819640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-26T00:31:15.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:15.245+0000 7f0378819640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-26T00:31:15.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:15.245+0000 7f0378819640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:15.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:15.245+0000 7f0378819640 1 -- 172.21.15.37:0/203863946 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f037413e4b0 con 0x7f0374136e10 2024-04-26T00:31:15.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:15.249+0000 7f0362ffd640 1 -- 172.21.15.37:0/203863946 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f03680160f0 con 0x7f0374136e10 2024-04-26T00:31:15.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:15.249+0000 7f0362ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-26T00:31:15.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:15.249+0000 7f0362ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:31:15.243 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.249+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.243 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.249+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=18,config=2,health=12,kv=10,logm=58} crc {auth=1681851022,config=692619927,health=3152313405,kv=1034996044,logm=2170990875}) 2024-04-26T00:31:15.243 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.249+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=2,health=12,kv=9,logm=57} crc {auth=2661709515,config=692619927,health=3152313405,kv=3680767304,logm=3485846958}) 2024-04-26T00:31:15.243 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.249+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.243 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.249+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=18,config=2,health=12,kv=10,logm=58} crc {auth=1681851022,config=692619927,health=3152313405,kv=1034996044,logm=2170990875}) 2024-04-26T00:31:15.243 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.249+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=19,config=2,health=10,kv=9,logm=60} crc {auth=1687954244,config=692619927,health=3842293809,kv=3748211275,logm=536310316}) 2024-04-26T00:31:15.245 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.253+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.245 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.253+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=677985438}) 2024-04-26T00:31:15.245 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.253+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3953354768}) 2024-04-26T00:31:15.245 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.253+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.245 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.253+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=677985438}) 2024-04-26T00:31:15.245 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.253+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1769866360}) 2024-04-26T00:31:15.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:15.253+0000 7f0378819640 1 -- 172.21.15.37:0/203863946 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0354035eb0 msgr2=0x7f0354038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:15.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:15.253+0000 7f0378819640 1 --2- 172.21.15.37:0/203863946 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0354035eb0 0x7f0354038370 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f0364006aa0 tx=0x7f0364008040 comp rx=0 tx=0).stop 2024-04-26T00:31:15.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:15.253+0000 7f0378819640 10 monclient: shutdown 2024-04-26T00:31:15.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:15.253+0000 7f0378819640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:31:15.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:15.253+0000 7f0378819640 1 -- 172.21.15.37:0/203863946 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0374136e10 msgr2=0x7f037406e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:15.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:15.253+0000 7f0378819640 1 --2- 172.21.15.37:0/203863946 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0374136e10 0x7f037406e520 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f036801a010 tx=0x7f036801ac20 comp rx=0 tx=0).stop 2024-04-26T00:31:15.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:15.253+0000 7f0378819640 1 -- 172.21.15.37:0/203863946 shutdown_connections 2024-04-26T00:31:15.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:15.253+0000 7f0378819640 1 --2- 172.21.15.37:0/203863946 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0354035eb0 0x7f0354038370 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:15.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:15.253+0000 7f0378819640 1 --2- 172.21.15.37:0/203863946 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f03741371f0 0x7f037406ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:15.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:15.253+0000 7f0378819640 1 --2- 172.21.15.37:0/203863946 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0374136e10 0x7f037406e520 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:15.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:15.253+0000 7f0378819640 1 --2- 172.21.15.37:0/203863946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0374135d90 0x7f0374071400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:15.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:15.253+0000 7f0378819640 1 -- 172.21.15.37:0/203863946 >> 172.21.15.37:0/203863946 conn(0x7f03740a64e0 msgr2=0x7f03740a4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:31:15.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:15.253+0000 7f0378819640 1 -- 172.21.15.37:0/203863946 shutdown_connections 2024-04-26T00:31:15.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:15.253+0000 7f0378819640 1 -- 172.21.15.37:0/203863946 wait complete. 2024-04-26T00:31:15.252 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.253+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.252 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.253+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1040569106}) 2024-04-26T00:31:15.252 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.253+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3115634971}) 2024-04-26T00:31:15.252 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.253+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.252 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.253+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1040569106}) 2024-04-26T00:31:15.252 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.253+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2223208309}) 2024-04-26T00:31:15.252 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.257+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.252 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.257+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1360243652}) 2024-04-26T00:31:15.252 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.257+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2399763848}) 2024-04-26T00:31:15.252 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.257+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.252 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.257+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1360243652}) 2024-04-26T00:31:15.252 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.257+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=546557422}) 2024-04-26T00:31:15.252 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.257+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.252 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.257+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3737860983}) 2024-04-26T00:31:15.253 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.257+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=113719182}) 2024-04-26T00:31:15.253 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.257+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.253 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.257+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3737860983}) 2024-04-26T00:31:15.253 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.257+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1661903620}) 2024-04-26T00:31:15.253 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.261+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.253 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.261+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3460083061}) 2024-04-26T00:31:15.254 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.261+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1283172946}) 2024-04-26T00:31:15.254 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.261+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.254 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.261+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3460083061}) 2024-04-26T00:31:15.254 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.261+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3585453764}) 2024-04-26T00:31:15.255 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.261+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.255 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.261+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2091618538}) 2024-04-26T00:31:15.255 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.261+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3970926426}) 2024-04-26T00:31:15.255 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.261+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.255 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.261+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2091618538}) 2024-04-26T00:31:15.255 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.261+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4162117881}) 2024-04-26T00:31:15.265 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-26T00:31:15.267 INFO:tasks.rados.rados.0.smithi045.stdout:1889: expect (ObjNum 655 snap 202 seq_num 655) 2024-04-26T00:31:15.360 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.365+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.360 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.365+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2664854908}) 2024-04-26T00:31:15.360 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.365+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2686396247}) 2024-04-26T00:31:15.360 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.365+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.360 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.365+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2664854908}) 2024-04-26T00:31:15.360 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.365+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2108755490}) 2024-04-26T00:31:15.368 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.373+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.368 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.373+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=168368797}) 2024-04-26T00:31:15.368 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.373+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=102814265}) 2024-04-26T00:31:15.368 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.373+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.368 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.373+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=168368797}) 2024-04-26T00:31:15.368 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.373+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2511648913}) 2024-04-26T00:31:15.375 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.381+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.375 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.381+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3538989957}) 2024-04-26T00:31:15.375 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.381+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3451878019}) 2024-04-26T00:31:15.375 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.381+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.375 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.381+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3538989957}) 2024-04-26T00:31:15.375 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.381+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3228460570}) 2024-04-26T00:31:15.492 INFO:tasks.rados.rados.0.smithi045.stdout:1890: finishing copy_from racing read to smithi04526579-45 2024-04-26T00:31:15.492 INFO:tasks.rados.rados.0.smithi045.stdout:1890: finishing copy_from to smithi04526579-45 2024-04-26T00:31:15.492 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 1436 (ObjNum 690 snap 215 seq_num 690) dirty exists 2024-04-26T00:31:15.492 INFO:tasks.rados.rados.0.smithi045.stdout:1889: done (4 left) 2024-04-26T00:31:15.492 INFO:tasks.rados.rados.0.smithi045.stdout:1890: done (3 left) 2024-04-26T00:31:15.492 INFO:tasks.rados.rados.0.smithi045.stdout:1891: done (2 left) 2024-04-26T00:31:15.492 INFO:tasks.rados.rados.0.smithi045.stdout:1894: done (1 left) 2024-04-26T00:31:15.492 INFO:tasks.rados.rados.0.smithi045.stdout:1895: snap_remove snap 213 2024-04-26T00:31:15.955 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.961+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.955 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.961+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3312661447}) 2024-04-26T00:31:15.955 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.961+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2022629009}) 2024-04-26T00:31:15.955 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.961+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.955 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.961+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3312661447}) 2024-04-26T00:31:15.955 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.961+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=563951111}) 2024-04-26T00:31:15.957 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.965+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.957 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.965+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3578997122}) 2024-04-26T00:31:15.957 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.965+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3610541831}) 2024-04-26T00:31:15.957 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.965+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.957 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.965+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3578997122}) 2024-04-26T00:31:15.957 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.965+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1972313402}) 2024-04-26T00:31:15.959 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.965+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.960 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.965+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=712902684}) 2024-04-26T00:31:15.960 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.965+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1292364266}) 2024-04-26T00:31:15.960 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.965+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.960 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.965+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=712902684}) 2024-04-26T00:31:15.960 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.965+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2816451301}) 2024-04-26T00:31:15.962 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.969+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.962 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.969+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1435765957}) 2024-04-26T00:31:15.962 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.969+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1160941354}) 2024-04-26T00:31:15.962 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.969+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.962 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.969+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1435765957}) 2024-04-26T00:31:15.963 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.969+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=379524719}) 2024-04-26T00:31:15.965 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.973+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.965 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.973+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2827583421}) 2024-04-26T00:31:15.965 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.973+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1235288244}) 2024-04-26T00:31:15.965 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.973+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.965 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.973+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2827583421}) 2024-04-26T00:31:15.965 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.973+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2187082852}) 2024-04-26T00:31:15.967 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.973+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.967 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.973+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1232617059}) 2024-04-26T00:31:15.967 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.973+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3186856254}) 2024-04-26T00:31:15.967 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.973+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.967 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.973+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1232617059}) 2024-04-26T00:31:15.968 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.973+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2125287131}) 2024-04-26T00:31:15.970 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.977+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.970 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.977+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=117956888}) 2024-04-26T00:31:15.970 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.977+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2678787154}) 2024-04-26T00:31:15.970 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.977+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.970 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.977+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=117956888}) 2024-04-26T00:31:15.970 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.977+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=453953814}) 2024-04-26T00:31:15.972 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.977+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.972 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.977+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4173206351}) 2024-04-26T00:31:15.972 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.977+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3747717269}) 2024-04-26T00:31:15.972 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.977+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.972 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.977+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4173206351}) 2024-04-26T00:31:15.972 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.977+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2653749691}) 2024-04-26T00:31:15.974 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.981+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.975 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.981+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2566404257}) 2024-04-26T00:31:15.975 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.981+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2573266533}) 2024-04-26T00:31:15.975 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.981+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.975 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.981+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2566404257}) 2024-04-26T00:31:15.975 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.981+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1057855770}) 2024-04-26T00:31:15.977 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.985+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.977 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.985+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1202483234}) 2024-04-26T00:31:15.977 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.985+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2823245639}) 2024-04-26T00:31:15.977 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.985+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.977 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.985+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1202483234}) 2024-04-26T00:31:15.977 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.985+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2444138557}) 2024-04-26T00:31:15.979 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.985+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.979 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.985+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2958692414}) 2024-04-26T00:31:15.979 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.985+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3139286323}) 2024-04-26T00:31:15.979 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.985+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.979 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.985+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2958692414}) 2024-04-26T00:31:15.979 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.985+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2950498012}) 2024-04-26T00:31:15.982 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.989+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.982 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.989+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2685537070}) 2024-04-26T00:31:15.982 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.989+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3559549256}) 2024-04-26T00:31:15.982 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.989+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.982 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.989+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2685537070}) 2024-04-26T00:31:15.982 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.989+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3985245001}) 2024-04-26T00:31:15.984 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.989+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.984 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.989+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=604416988}) 2024-04-26T00:31:15.984 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.989+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2316607975}) 2024-04-26T00:31:15.984 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.989+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.984 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.989+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=604416988}) 2024-04-26T00:31:15.984 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.989+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1152741284}) 2024-04-26T00:31:15.986 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.993+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.986 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.993+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1926027195}) 2024-04-26T00:31:15.986 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.993+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=546823430}) 2024-04-26T00:31:15.987 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.993+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.987 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.993+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1926027195}) 2024-04-26T00:31:15.987 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.993+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3996430681}) 2024-04-26T00:31:15.989 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.997+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.989 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.997+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3528483523}) 2024-04-26T00:31:15.989 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.997+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3826589759}) 2024-04-26T00:31:15.989 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.997+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.989 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.997+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3528483523}) 2024-04-26T00:31:15.989 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.997+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=814675303}) 2024-04-26T00:31:15.991 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.997+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.991 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.997+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2988803808}) 2024-04-26T00:31:15.991 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.997+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1434255995}) 2024-04-26T00:31:15.991 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.997+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.991 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.997+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2988803808}) 2024-04-26T00:31:15.991 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:15.997+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=311989487}) 2024-04-26T00:31:15.994 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.001+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.994 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.001+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1383750590}) 2024-04-26T00:31:15.994 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.001+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4166608441}) 2024-04-26T00:31:15.994 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.001+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.994 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.001+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1383750590}) 2024-04-26T00:31:15.994 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.001+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4183386269}) 2024-04-26T00:31:15.996 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.001+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.996 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.001+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1779214090}) 2024-04-26T00:31:15.996 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.001+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=499872999}) 2024-04-26T00:31:15.996 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.001+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.996 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.001+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1779214090}) 2024-04-26T00:31:15.996 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.005+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=190209429}) 2024-04-26T00:31:15.998 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.005+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.999 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.005+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2192014943}) 2024-04-26T00:31:15.999 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.005+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1237537801}) 2024-04-26T00:31:15.999 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.005+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:15.999 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.005+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2192014943}) 2024-04-26T00:31:15.999 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.005+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=172249066}) 2024-04-26T00:31:16.001 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.009+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:16.001 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.009+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2499039424}) 2024-04-26T00:31:16.001 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.009+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2301460673}) 2024-04-26T00:31:16.001 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.009+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:16.001 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.009+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2499039424}) 2024-04-26T00:31:16.001 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.009+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2703647765}) 2024-04-26T00:31:16.003 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.009+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:16.003 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.009+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=610465569}) 2024-04-26T00:31:16.004 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.009+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2574745055}) 2024-04-26T00:31:16.004 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.009+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:16.004 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.009+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=610465569}) 2024-04-26T00:31:16.004 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.009+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=991554618}) 2024-04-26T00:31:16.006 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.013+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:16.006 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.013+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3470851541}) 2024-04-26T00:31:16.006 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.013+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=870512368}) 2024-04-26T00:31:16.006 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.013+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:16.006 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.013+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3470851541}) 2024-04-26T00:31:16.006 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.013+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3157908429}) 2024-04-26T00:31:16.008 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.017+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:16.009 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.017+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=374885803}) 2024-04-26T00:31:16.009 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.017+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3124531919}) 2024-04-26T00:31:16.009 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.017+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:16.009 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.017+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=374885803}) 2024-04-26T00:31:16.009 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.017+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3833213570}) 2024-04-26T00:31:16.011 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.017+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:16.011 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.017+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=47,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=8} crc {logm=273746289,mdsmap=2840952586,mgr=3887433460,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=1783619864,mon_config_key=4061063117,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=1457668320,osd_snap=1121735418}) 2024-04-26T00:31:16.011 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.017+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=46,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=9} crc {logm=2730451102,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=1783619864,mon_config_key=4061063117,monmap=3876866194,osd_metadata=124267790,osd_pg_creating=1457668320,osd_snap=643333606}) 2024-04-26T00:31:16.012 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.017+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:16.012 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.017+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=47,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=8} crc {logm=273746289,mdsmap=2840952586,mgr=3887433460,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=1783619864,mon_config_key=4061063117,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=1457668320,osd_snap=1121735418}) 2024-04-26T00:31:16.012 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.017+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=48,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=11,mon_config_key=10,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=10} crc {logm=451499978,mdsmap=2840952586,mgr=1189067752,mgr_metadata=3686404438,mgrstat=2670761982,mon_config_key=3412062517,monmap=3876866194,osd_metadata=1980645665,osd_pg_creating=1457668320,osd_snap=3793744976}) 2024-04-26T00:31:16.014 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.021+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:16.014 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.021+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=65,osdmap=35} crc {osd_snap=3529719472,osdmap=3258558434}) 2024-04-26T00:31:16.014 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.021+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=69,osdmap=31} crc {osd_snap=3205459240,osdmap=927981219}) 2024-04-26T00:31:16.014 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.021+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:16.014 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.021+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=65,osdmap=35} crc {osd_snap=3529719472,osdmap=3258558434}) 2024-04-26T00:31:16.014 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.021+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=65,osdmap=35} crc {osd_snap=3478177771,osdmap=2476732423}) 2024-04-26T00:31:16.023 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.029+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:16.023 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.029+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=72} crc {osdmap=680193312}) 2024-04-26T00:31:16.023 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.029+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=73} crc {osdmap=2907263712}) 2024-04-26T00:31:16.023 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.029+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:31:16.023 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.029+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=72} crc {osdmap=680193312}) 2024-04-26T00:31:16.023 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:31:16.029+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=74} crc {osdmap=3958921685}) 2024-04-26T00:31:16.266 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:31:16.267 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:31:16.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.409+0000 7f006d429640 10 monclient: get_monmap_and_config 2024-04-26T00:31:16.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.409+0000 7f006d429640 10 monclient: build_initial_monmap 2024-04-26T00:31:16.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.409+0000 7f006d429640 10 monclient: monmap: 2024-04-26T00:31:16.404 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:31:16.404 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:16.404 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:31:16.414876+0000 2024-04-26T00:31:16.404 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:31:16.414876+0000 2024-04-26T00:31:16.404 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:31:16.405 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:16.405 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:31:16.405 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:31:16.405 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:31:16.405 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:31:16.405 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:31:16.405 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:16.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.413+0000 7f006d429640 1 Processor -- start 2024-04-26T00:31:16.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.413+0000 7f006d429640 1 -- start start 2024-04-26T00:31:16.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.413+0000 7f006d429640 10 monclient: init 2024-04-26T00:31:16.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.413+0000 7f006d429640 10 monclient: _reopen_session rank -1 2024-04-26T00:31:16.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.413+0000 7f006d429640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-04-26T00:31:16.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.413+0000 7f006d429640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0068136e10 addr v1:172.21.15.37:6789/0 2024-04-26T00:31:16.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.413+0000 7f006d429640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0068137900 addr v1:172.21.15.45:6789/0 2024-04-26T00:31:16.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.413+0000 7f006d429640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f006809c5b0 0x7f006813fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:16.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.413+0000 7f006d429640 10 monclient(hunting): picked mon.noname-c con 0x7f006809c5b0 addr v2:172.21.15.45:3300/0 2024-04-26T00:31:16.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.413+0000 7f006d429640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f006809b680 con 0x7f0068136e10 2024-04-26T00:31:16.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.413+0000 7f006d429640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f006809bbd0 con 0x7f0068137900 2024-04-26T00:31:16.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.413+0000 7f006d429640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:16.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.413+0000 7f006d429640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f006809b8f0 con 0x7f006809c5b0 2024-04-26T00:31:16.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.413+0000 7f006d429640 10 monclient(hunting): _renew_subs 2024-04-26T00:31:16.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.413+0000 7f006d429640 10 monclient(hunting): authenticate will time out at 1657.720915s 2024-04-26T00:31:16.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.413+0000 7f0066ffd640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f0068136e10 0x7f00681371f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:42716/0 (socket says 172.21.15.37:42716) 2024-04-26T00:31:16.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.413+0000 7f0066ffd640 1 -- 172.21.15.37:0/650446454 learned_addr learned my addr 172.21.15.37:0/650446454 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:31:16.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.413+0000 7f00677fe640 1 --2- 172.21.15.37:0/650446454 >> v2:172.21.15.45:3300/0 conn(0x7f006809c5b0 0x7f006813fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:16.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.413+0000 7f00677fe640 10 monclient(hunting): get_auth_request con 0x7f006809c5b0 auth_method 0 2024-04-26T00:31:16.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.413+0000 7f00677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:16.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.413+0000 7f00677fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:16.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.413+0000 7f00677fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:16.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.413+0000 7f00677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:16.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.413+0000 7f00677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:16.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.413+0000 7f00677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:16.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.413+0000 7f0066ffd640 1 -- 172.21.15.37:0/650446454 >> v1:172.21.15.37:6789/0 conn(0x7f0068136e10 legacy=0x7f00681371f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:16.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.413+0000 7f00667fc640 1 -- 172.21.15.37:0/650446454 >> v1:172.21.15.45:6789/0 conn(0x7f0068137900 legacy=0x7f006813c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:16.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.417+0000 7f00677fe640 10 monclient(hunting): handle_auth_done global_id 134110 payload 274 2024-04-26T00:31:16.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.417+0000 7f00677fe640 1 -- 172.21.15.37:0/650446454 >> v1:172.21.15.45:6789/0 conn(0x7f0068137900 legacy=0x7f006813c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:16.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.417+0000 7f00677fe640 1 -- 172.21.15.37:0/650446454 >> v1:172.21.15.37:6789/0 conn(0x7f0068136e10 legacy=0x7f00681371f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:16.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.417+0000 7f00677fe640 10 monclient: _finish_hunting 0 2024-04-26T00:31:16.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.417+0000 7f00677fe640 1 monclient: found mon.noname-c 2024-04-26T00:31:16.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.417+0000 7f00677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:31:16.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.417+0000 7f00677fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:31:16.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.417+0000 7f00677fe640 1 -- 172.21.15.37:0/650446454 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00680a5d90 con 0x7f006809c5b0 2024-04-26T00:31:16.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f00677fe640 1 -- 172.21.15.37:0/650446454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f006809c5b0 msgr2=0x7f006813fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:16.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f00677fe640 1 --2- 172.21.15.37:0/650446454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f006809c5b0 0x7f006813fa70 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0060009e90 tx=0x7f006001c930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2cd03703805762c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:16.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f0065ffb640 1 -- 172.21.15.37:0/650446454 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00600b2070 con 0x7f006809c5b0 2024-04-26T00:31:16.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f0065ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:16.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f0065ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:31:16.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f0065ffb640 10 monclient: dump: 2024-04-26T00:31:16.415 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:16.415 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:16.415 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:16.415 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:16.415 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:16.415 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:16.415 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:16.415 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:16.415 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:16.415 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:16.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f0065ffb640 10 monclient: _finish_auth 0 2024-04-26T00:31:16.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f0065ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:31:16.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f0065ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:31:16.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f0065ffb640 1 -- 172.21.15.37:0/650446454 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00600a59e0 con 0x7f006809c5b0 2024-04-26T00:31:16.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f0065ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:31:16.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f0065ffb640 1 -- 172.21.15.37:0/650446454 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00600a5db0 con 0x7f006809c5b0 2024-04-26T00:31:16.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f006d429640 5 monclient: authenticate success, global_id 134110 2024-04-26T00:31:16.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f006d429640 10 monclient: get_monmap_and_config success 2024-04-26T00:31:16.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f006d429640 4 set_mon_vals no callback set 2024-04-26T00:31:16.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f0065ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:16.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f0065ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:31:16.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f0065ffb640 10 monclient: dump: 2024-04-26T00:31:16.416 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:16.417 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:16.417 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:16.417 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:16.417 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:16.417 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:16.417 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:16.417 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:16.417 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:16.417 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:16.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f006d429640 10 monclient: shutdown 2024-04-26T00:31:16.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f006d429640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:31:16.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f006d429640 1 -- 172.21.15.37:0/650446454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f006809c5b0 msgr2=0x7f006813fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:16.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f006d429640 1 --2- 172.21.15.37:0/650446454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f006809c5b0 0x7f006813fa70 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0060009e90 tx=0x7f006001c930 comp rx=0 tx=0).stop 2024-04-26T00:31:16.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f006d429640 1 -- 172.21.15.37:0/650446454 shutdown_connections 2024-04-26T00:31:16.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f006d429640 1 --2- 172.21.15.37:0/650446454 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f006809c5b0 0x7f006813fa70 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:16.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f006d429640 1 -- 172.21.15.37:0/650446454 >> 172.21.15.37:0/650446454 conn(0x7f00680a64e0 msgr2=0x7f00680a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:31:16.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f006d429640 1 -- 172.21.15.37:0/650446454 shutdown_connections 2024-04-26T00:31:16.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f006d429640 1 -- 172.21.15.37:0/650446454 wait complete. 2024-04-26T00:31:16.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f006d429640 10 monclient: build_initial_monmap 2024-04-26T00:31:16.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f006d429640 10 monclient: monmap: 2024-04-26T00:31:16.418 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:31:16.418 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:31:16.418 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:31:16.418 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:31:16.418 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:31:16.418 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:31:16.419 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:31:16.419 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:31:16.419 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:31:16.419 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:16.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f006d429640 1 Processor -- start 2024-04-26T00:31:16.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f006d429640 1 -- start start 2024-04-26T00:31:16.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f006d429640 10 monclient: init 2024-04-26T00:31:16.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f006d429640 10 monclient: _reopen_session rank -1 2024-04-26T00:31:16.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f006d429640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:31:16.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f006d429640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f006809c5b0 0x7f00681cf230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:16.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f006d429640 10 monclient(hunting): picked mon.noname-a con 0x7f006809c5b0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:31:16.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f006d429640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0068136e10 0x7f00681cf770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:16.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f006d429640 10 monclient(hunting): picked mon.noname-c con 0x7f0068136e10 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:31:16.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f006d429640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0068137900 0x7f006813b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:16.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f006d429640 10 monclient(hunting): picked mon.noname-b con 0x7f0068137900 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:31:16.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f006d429640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:16.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f006d429640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f0068142b60 con 0x7f006809c5b0 2024-04-26T00:31:16.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f006d429640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:16.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f006d429640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f00681429e0 con 0x7f0068137900 2024-04-26T00:31:16.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f006d429640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:16.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f006d429640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0068142ce0 con 0x7f0068136e10 2024-04-26T00:31:16.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f006d429640 10 monclient(hunting): _renew_subs 2024-04-26T00:31:16.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f006d429640 10 monclient(hunting): authenticate will time out at 1657.731311s 2024-04-26T00:31:16.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f00667fc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0068136e10 0x7f00681cf770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:16.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f00667fc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0068136e10 0x7f00681cf770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:33692/0 (socket says 172.21.15.37:33692) 2024-04-26T00:31:16.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f00667fc640 1 -- 172.21.15.37:0/4105247695 learned_addr learned my addr 172.21.15.37:0/4105247695 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:31:16.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f0066ffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f006809c5b0 0x7f00681cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:16.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f00667fc640 10 monclient(hunting): get_auth_request con 0x7f0068136e10 auth_method 0 2024-04-26T00:31:16.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f00667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:16.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f00667fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:16.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f00667fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:16.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f0066ffd640 10 monclient(hunting): get_auth_request con 0x7f006809c5b0 auth_method 0 2024-04-26T00:31:16.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f0066ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:16.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f0066ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:16.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f0066ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:16.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f00667fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:16.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f00667fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:16.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f0066ffd640 1 -- 172.21.15.37:0/4105247695 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f006809c5b0 msgr2=0x7f00681cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:31:16.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f0066ffd640 1 -- 172.21.15.37:0/4105247695 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f006809c5b0 msgr2=0x7f00681cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:31:16.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f0066ffd640 1 --2- 172.21.15.37:0/4105247695 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f006809c5b0 0x7f00681cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:31:16.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f00667fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:16.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f0066ffd640 1 --2- 172.21.15.37:0/4105247695 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f006809c5b0 0x7f00681cf230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:31:16.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f00677fe640 1 --2- 172.21.15.37:0/4105247695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0068137900 0x7f006813b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:16.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f00677fe640 10 monclient(hunting): get_auth_request con 0x7f0068137900 auth_method 0 2024-04-26T00:31:16.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f00677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:16.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f00677fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:16.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f00677fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:16.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f00677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:16.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f00677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:16.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f00677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:16.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f00667fc640 10 monclient(hunting): handle_auth_done global_id 84581 payload 824 2024-04-26T00:31:16.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f00667fc640 1 -- 172.21.15.37:0/4105247695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0068137900 msgr2=0x7f006813b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:16.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f00667fc640 1 --2- 172.21.15.37:0/4105247695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0068137900 0x7f006813b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:16.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f00667fc640 1 -- 172.21.15.37:0/4105247695 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f006809c5b0 msgr2=0x7f00681cf230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:31:16.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f00667fc640 1 --2- 172.21.15.37:0/4105247695 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f006809c5b0 0x7f00681cf230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:16.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.421+0000 7f00667fc640 10 monclient: _finish_hunting 0 2024-04-26T00:31:16.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.425+0000 7f00667fc640 1 monclient: found mon.noname-c 2024-04-26T00:31:16.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.425+0000 7f00667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:31:16.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.425+0000 7f00667fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:31:16.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.425+0000 7f00667fc640 1 -- 172.21.15.37:0/4105247695 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f006813f730 con 0x7f0068136e10 2024-04-26T00:31:16.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.425+0000 7f00667fc640 1 -- 172.21.15.37:0/4105247695 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0068136e10 msgr2=0x7f00681cf770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:16.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.425+0000 7f00667fc640 1 --2- 172.21.15.37:0/4105247695 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0068136e10 0x7f00681cf770 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f005c00c430 tx=0x7f005c00c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:16.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.425+0000 7f004f7fe640 1 -- 172.21.15.37:0/4105247695 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f005c017070 con 0x7f0068136e10 2024-04-26T00:31:16.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.425+0000 7f004f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:16.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.425+0000 7f004f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:31:16.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.425+0000 7f004f7fe640 10 monclient: dump: 2024-04-26T00:31:16.429 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:16.429 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:16.429 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:16.429 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:16.429 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:16.429 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:16.429 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:16.429 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:16.429 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:16.429 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:16.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.425+0000 7f004f7fe640 10 monclient: _finish_auth 0 2024-04-26T00:31:16.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.425+0000 7f004f7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:31:16.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.425+0000 7f004f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:31:16.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.425+0000 7f004f7fe640 1 -- 172.21.15.37:0/4105247695 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f005c0057d0 con 0x7f0068136e10 2024-04-26T00:31:16.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.425+0000 7f006d429640 5 monclient: authenticate success, global_id 84581 2024-04-26T00:31:16.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.425+0000 7f006d429640 10 monclient: _renew_subs 2024-04-26T00:31:16.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.425+0000 7f006d429640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:16.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.425+0000 7f006d429640 1 -- 172.21.15.37:0/4105247695 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00680a30c0 con 0x7f0068136e10 2024-04-26T00:31:16.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.425+0000 7f004f7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:31:16.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.425+0000 7f004f7fe640 1 -- 172.21.15.37:0/4105247695 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f005c005bc0 con 0x7f0068136e10 2024-04-26T00:31:16.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.425+0000 7f004f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:16.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.425+0000 7f004f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:31:16.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.425+0000 7f004f7fe640 10 monclient: dump: 2024-04-26T00:31:16.430 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:16.430 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:16.431 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:16.431 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:16.431 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:16.431 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:16.431 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:16.431 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:16.431 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:16.431 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:16.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.425+0000 7f006d429640 10 monclient: _renew_subs 2024-04-26T00:31:16.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.425+0000 7f00657fa640 4 set_mon_vals no callback set 2024-04-26T00:31:16.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.425+0000 7f006d429640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:16.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.425+0000 7f006d429640 1 -- 172.21.15.37:0/4105247695 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f006813dd20 con 0x7f0068136e10 2024-04-26T00:31:16.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.425+0000 7f006d429640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:16.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.425+0000 7f006d429640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:16.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.425+0000 7f006d429640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:16.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.425+0000 7f006d429640 1 -- 172.21.15.37:0/4105247695 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0068075910 con 0x7f0068136e10 2024-04-26T00:31:16.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.425+0000 7f00667fc640 1 --2- 172.21.15.37:0/4105247695 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0068136e10 0x7f00681cf770 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f005c00c430 tx=0x7f005c00c8f0 comp rx=0 tx=0).queue_received will delay after 1.71409e+09 on 0x7f005c01a070 mgrmap(e 5) 2024-04-26T00:31:16.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.425+0000 7f004f7fe640 1 -- 172.21.15.37:0/4105247695 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f005c01a070 con 0x7f0068136e10 2024-04-26T00:31:16.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.429+0000 7f004f7fe640 1 --2- 172.21.15.37:0/4105247695 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0054035e80 0x7f0054038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:16.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.429+0000 7f0066ffd640 1 --2- 172.21.15.37:0/4105247695 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0054035e80 0x7f0054038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:16.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.429+0000 7f0066ffd640 10 monclient: get_auth_request con 0x7f0054035e80 auth_method 0 2024-04-26T00:31:16.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.429+0000 7f0066ffd640 1 --2- 172.21.15.37:0/4105247695 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0054035e80 0x7f0054038340 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f0058005890 tx=0x7f0058009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:16.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:16.433+0000 7f004f7fe640 1 -- 172.21.15.37:0/4105247695 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(432..432 src has 370..432) ==== 4019+0+0 (secure 0 0 0) 0x7f005c047f80 con 0x7f0068136e10 2024-04-26T00:31:16.563 INFO:tasks.rados.rados.0.smithi045.stdout:1895: done (1 left) 2024-04-26T00:31:16.564 INFO:tasks.rados.rados.0.smithi045.stdout:1896: write oid 34 current snap is 217 2024-04-26T00:31:16.564 INFO:tasks.rados.rados.0.smithi045.stdout:1893: finishing copy_from racing read to smithi04526579-32 2024-04-26T00:31:16.564 INFO:tasks.rados.rados.0.smithi045.stdout:1893: finishing copy_from to smithi04526579-32 2024-04-26T00:31:16.564 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 1225 (ObjNum 685 snap 213 seq_num 685) dirty exists 2024-04-26T00:31:16.564 INFO:tasks.rados.rados.0.smithi045.stdout:1896: seq_num 693 ranges {447251=735680,1736139=438895,2597781=780799} 2024-04-26T00:31:16.570 INFO:tasks.rados.rados.0.smithi045.stdout:1896: writing smithi04526579-34 from 447251 to 1182931 tid 1 2024-04-26T00:31:16.575 INFO:tasks.rados.rados.0.smithi045.stdout:1896: writing smithi04526579-34 from 1736139 to 2175034 tid 2 2024-04-26T00:31:16.589 INFO:tasks.rados.rados.0.smithi045.stdout:1896: writing smithi04526579-34 from 2597781 to 3378580 tid 3 2024-04-26T00:31:16.589 INFO:tasks.rados.rados.0.smithi045.stdout:1893: done (1 left) 2024-04-26T00:31:16.589 INFO:tasks.rados.rados.0.smithi045.stdout:1897: write (excl) oid 38 current snap is 217 2024-04-26T00:31:16.590 INFO:tasks.rados.rados.0.smithi045.stdout:1897: seq_num 694 ranges {464058=410929,1394621=698906,2625204=592071,3656107=7951} 2024-04-26T00:31:16.601 INFO:tasks.rados.rados.0.smithi045.stdout:1897: writing smithi04526579-38 from 464058 to 874987 tid 1 2024-04-26T00:31:16.614 INFO:tasks.rados.rados.0.smithi045.stdout:1897: writing smithi04526579-38 from 1394621 to 2093527 tid 2 2024-04-26T00:31:16.624 INFO:tasks.rados.rados.0.smithi045.stdout:1897: writing smithi04526579-38 from 2625204 to 3217275 tid 3 2024-04-26T00:31:16.628 INFO:tasks.rados.rados.0.smithi045.stdout:1897: writing smithi04526579-38 from 3656107 to 3664058 tid 4 2024-04-26T00:31:16.628 INFO:tasks.rados.rados.0.smithi045.stdout:1898: read oid 24 snap -1 2024-04-26T00:31:16.628 INFO:tasks.rados.rados.0.smithi045.stdout:1898: expect (ObjNum 582 snap 180 seq_num 582) 2024-04-26T00:31:16.628 INFO:tasks.rados.rados.0.smithi045.stdout:1899: snap_create 2024-04-26T00:31:16.628 INFO:tasks.rados.rados.0.smithi045.stdout:1896: finishing write tid 1 to smithi04526579-34 2024-04-26T00:31:16.628 INFO:tasks.rados.rados.0.smithi045.stdout:1896: finishing write tid 2 to smithi04526579-34 2024-04-26T00:31:16.628 INFO:tasks.rados.rados.0.smithi045.stdout:1896: finishing write tid 3 to smithi04526579-34 2024-04-26T00:31:16.628 INFO:tasks.rados.rados.0.smithi045.stdout:1896: finishing write tid 4 to smithi04526579-34 2024-04-26T00:31:16.628 INFO:tasks.rados.rados.0.smithi045.stdout:1896: finishing write tid 5 to smithi04526579-34 2024-04-26T00:31:16.628 INFO:tasks.rados.rados.0.smithi045.stdout:1896: oid 34 updating version 0 to 1099 2024-04-26T00:31:16.628 INFO:tasks.rados.rados.0.smithi045.stdout:1896: oid 34 updating version 1099 to 1104 2024-04-26T00:31:16.629 INFO:tasks.rados.rados.0.smithi045.stdout:1896: oid 34 version 1104 is already newer than 1103 2024-04-26T00:31:16.629 INFO:tasks.rados.rados.0.smithi045.stdout:1896: oid 34 version 1104 is already newer than 1100 2024-04-26T00:31:16.629 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 1104 (ObjNum 693 snap 217 seq_num 693) dirty exists 2024-04-26T00:31:16.629 INFO:tasks.rados.rados.0.smithi045.stdout:1896: left oid 34 (ObjNum 693 snap 217 seq_num 693) 2024-04-26T00:31:16.629 INFO:tasks.rados.rados.0.smithi045.stdout:1897: finishing write tid 1 to smithi04526579-38 2024-04-26T00:31:16.629 INFO:tasks.rados.rados.0.smithi045.stdout:1897: finishing write tid 2 to smithi04526579-38 2024-04-26T00:31:16.639 INFO:tasks.rados.rados.0.smithi045.stdout:1897: finishing write tid 3 to smithi04526579-38 2024-04-26T00:31:16.639 INFO:tasks.rados.rados.0.smithi045.stdout:1897: finishing write tid 4 to smithi04526579-38 2024-04-26T00:31:16.640 INFO:tasks.rados.rados.0.smithi045.stdout:1897: finishing write tid 5 to smithi04526579-38 2024-04-26T00:31:16.640 INFO:tasks.rados.rados.0.smithi045.stdout:1897: finishing write tid 6 to smithi04526579-38 2024-04-26T00:31:16.640 INFO:tasks.rados.rados.0.smithi045.stdout:1897: oid 38 updating version 0 to 1078 2024-04-26T00:31:16.640 INFO:tasks.rados.rados.0.smithi045.stdout:1897: oid 38 updating version 1078 to 1082 2024-04-26T00:31:16.640 INFO:tasks.rados.rados.0.smithi045.stdout:1897: oid 38 version 1082 is already newer than 1080 2024-04-26T00:31:16.640 INFO:tasks.rados.rados.0.smithi045.stdout:1897: oid 38 version 1082 is already newer than 1081 2024-04-26T00:31:16.640 INFO:tasks.rados.rados.0.smithi045.stdout:1897: oid 38 version 1082 is already newer than 1079 2024-04-26T00:31:16.640 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 1082 (ObjNum 694 snap 217 seq_num 694) dirty exists 2024-04-26T00:31:16.640 INFO:tasks.rados.rados.0.smithi045.stdout:1897: left oid 38 (ObjNum 694 snap 217 seq_num 694) 2024-04-26T00:31:16.650 INFO:tasks.rados.rados.0.smithi045.stdout:1898: expect (ObjNum 582 snap 180 seq_num 582) 2024-04-26T00:31:17.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.253+0000 7f004f7fe640 1 -- 172.21.15.37:0/4105247695 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f005c01b3b0 con 0x7f0068136e10 2024-04-26T00:31:17.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.253+0000 7f004f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:17.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.253+0000 7f004f7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:31:17.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.484+0000 7f006d429640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:31:17.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.484+0000 7f006d429640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:31:17.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.484+0000 7f006d429640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:17.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.484+0000 7f006d429640 1 -- 172.21.15.37:0/4105247695 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0068075ee0 con 0x7f0068136e10 2024-04-26T00:31:17.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.484+0000 7f004f7fe640 1 -- 172.21.15.37:0/4105247695 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f005c01ad50 con 0x7f0068136e10 2024-04-26T00:31:17.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.484+0000 7f004f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:31:17.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.484+0000 7f004f7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:31:17.481 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":86,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:31:17.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.488+0000 7f006d429640 1 -- 172.21.15.37:0/4105247695 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0054035e80 msgr2=0x7f0054038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:17.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.488+0000 7f006d429640 1 --2- 172.21.15.37:0/4105247695 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0054035e80 0x7f0054038340 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f0058005890 tx=0x7f0058009040 comp rx=0 tx=0).stop 2024-04-26T00:31:17.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.488+0000 7f006d429640 10 monclient: shutdown 2024-04-26T00:31:17.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.488+0000 7f006d429640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:31:17.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.488+0000 7f006d429640 1 -- 172.21.15.37:0/4105247695 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0068136e10 msgr2=0x7f00681cf770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:17.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.488+0000 7f006d429640 1 --2- 172.21.15.37:0/4105247695 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0068136e10 0x7f00681cf770 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f005c00c430 tx=0x7f005c00c8f0 comp rx=0 tx=0).stop 2024-04-26T00:31:17.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.488+0000 7f006d429640 1 -- 172.21.15.37:0/4105247695 shutdown_connections 2024-04-26T00:31:17.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.488+0000 7f006d429640 1 --2- 172.21.15.37:0/4105247695 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0054035e80 0x7f0054038340 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:17.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.488+0000 7f006d429640 1 --2- 172.21.15.37:0/4105247695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0068137900 0x7f006813b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:17.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.488+0000 7f006d429640 1 --2- 172.21.15.37:0/4105247695 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0068136e10 0x7f00681cf770 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:17.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.488+0000 7f006d429640 1 --2- 172.21.15.37:0/4105247695 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f006809c5b0 0x7f00681cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:17.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.488+0000 7f006d429640 1 -- 172.21.15.37:0/4105247695 >> 172.21.15.37:0/4105247695 conn(0x7f00680a64e0 msgr2=0x7f00680a1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:31:17.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.488+0000 7f006d429640 1 -- 172.21.15.37:0/4105247695 shutdown_connections 2024-04-26T00:31:17.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.488+0000 7f006d429640 1 -- 172.21.15.37:0/4105247695 wait complete. 2024-04-26T00:31:17.502 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:31:17.503 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-26T00:31:17.503 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:31:17.503 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:31:17.571 INFO:tasks.rados.rados.0.smithi045.stdout:1896: done (3 left) 2024-04-26T00:31:17.571 INFO:tasks.rados.rados.0.smithi045.stdout:1897: done (2 left) 2024-04-26T00:31:17.571 INFO:tasks.rados.rados.0.smithi045.stdout:1898: done (1 left) 2024-04-26T00:31:17.571 INFO:tasks.rados.rados.0.smithi045.stdout:1899: done (0 left) 2024-04-26T00:31:17.571 INFO:tasks.rados.rados.0.smithi045.stdout:1900: snap_create 2024-04-26T00:31:17.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5cdd39640 10 monclient: get_monmap_and_config 2024-04-26T00:31:17.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5cdd39640 10 monclient: build_initial_monmap 2024-04-26T00:31:17.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5cdd39640 10 monclient: monmap: 2024-04-26T00:31:17.625 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:31:17.625 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:17.625 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:31:17.636044+0000 2024-04-26T00:31:17.625 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:31:17.636044+0000 2024-04-26T00:31:17.625 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:31:17.625 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:17.625 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:31:17.625 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:31:17.625 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:31:17.625 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:31:17.625 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:31:17.625 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:17.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5cdd39640 1 Processor -- start 2024-04-26T00:31:17.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5cdd39640 1 -- start start 2024-04-26T00:31:17.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5cdd39640 10 monclient: init 2024-04-26T00:31:17.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5cdd39640 10 monclient: _reopen_session rank -1 2024-04-26T00:31:17.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5cdd39640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-04-26T00:31:17.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5cdd39640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5c80a3d90 0x7fb5c809bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:17.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5cdd39640 10 monclient(hunting): picked mon.noname-b con 0x7fb5c80a3d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:31:17.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5cdd39640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb5c80a4170 0x7fb5c80a45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:17.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5cdd39640 10 monclient(hunting): picked mon.noname-a con 0x7fb5c80a4170 addr v2:172.21.15.37:3300/0 2024-04-26T00:31:17.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5cdd39640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb5c8133b20 addr v1:172.21.15.37:6789/0 2024-04-26T00:31:17.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5cdd39640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5c80a5150 con 0x7fb5c8133b20 2024-04-26T00:31:17.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5cdd39640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:17.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5cdd39640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fb5c809b280 con 0x7fb5c80a4170 2024-04-26T00:31:17.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5cdd39640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:17.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5cdd39640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb5c809b560 con 0x7fb5c80a3d90 2024-04-26T00:31:17.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5cdd39640 10 monclient(hunting): _renew_subs 2024-04-26T00:31:17.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5cdd39640 10 monclient(hunting): authenticate will time out at 1658.941691s 2024-04-26T00:31:17.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5c7fff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fb5c8133b20 0x7fb5c8133f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:42718/0 (socket says 172.21.15.37:42718) 2024-04-26T00:31:17.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5c6ffd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb5c80a4170 0x7fb5c80a45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:17.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5c77fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5c80a3d90 0x7fb5c809bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:17.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5c7fff640 1 -- 172.21.15.37:0/4197040466 learned_addr learned my addr 172.21.15.37:0/4197040466 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:31:17.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5c6ffd640 10 monclient(hunting): get_auth_request con 0x7fb5c80a4170 auth_method 0 2024-04-26T00:31:17.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:17.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5c6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:17.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:17.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5c77fe640 10 monclient(hunting): get_auth_request con 0x7fb5c80a3d90 auth_method 0 2024-04-26T00:31:17.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:17.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5c77fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:17.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:17.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5c6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:17.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5c6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:17.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5c6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:17.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:17.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:17.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:17.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5c6ffd640 1 -- 172.21.15.37:0/4197040466 >> v2:172.21.15.37:3300/0 conn(0x7fb5c80a4170 msgr2=0x7fb5c80a45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:31:17.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5c6ffd640 1 -- 172.21.15.37:0/4197040466 >> v2:172.21.15.37:3300/0 conn(0x7fb5c80a4170 msgr2=0x7fb5c80a45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:31:17.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5c6ffd640 1 --2- 172.21.15.37:0/4197040466 >> v2:172.21.15.37:3300/0 conn(0x7fb5c80a4170 0x7fb5c80a45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:31:17.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5c6ffd640 1 --2- 172.21.15.37:0/4197040466 >> v2:172.21.15.37:3300/0 conn(0x7fb5c80a4170 0x7fb5c80a45b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:31:17.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5c77fe640 10 monclient(hunting): handle_auth_done global_id 84584 payload 274 2024-04-26T00:31:17.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5c77fe640 1 -- 172.21.15.37:0/4197040466 >> v2:172.21.15.37:3300/0 conn(0x7fb5c80a4170 msgr2=0x7fb5c80a45b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:31:17.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5c77fe640 1 --2- 172.21.15.37:0/4197040466 >> v2:172.21.15.37:3300/0 conn(0x7fb5c80a4170 0x7fb5c80a45b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:17.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5c77fe640 1 -- 172.21.15.37:0/4197040466 >> v1:172.21.15.37:6789/0 conn(0x7fb5c8133b20 legacy=0x7fb5c8133f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:31:17.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5c77fe640 10 monclient: _finish_hunting 0 2024-04-26T00:31:17.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5c77fe640 1 monclient: found mon.noname-b 2024-04-26T00:31:17.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:31:17.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5c77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:31:17.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.632+0000 7fb5c77fe640 1 -- 172.21.15.37:0/4197040466 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5c80a4af0 con 0x7fb5c80a3d90 2024-04-26T00:31:17.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.636+0000 7fb5c77fe640 1 -- 172.21.15.37:0/4197040466 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5c80a3d90 msgr2=0x7fb5c809bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:17.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.636+0000 7fb5c77fe640 1 --2- 172.21.15.37:0/4197040466 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5c80a3d90 0x7fb5c809bf40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb5b80097c0 tx=0x7fb5b801c320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c13f83ad5cc0edb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:17.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.636+0000 7fb5c67fc640 1 -- 172.21.15.37:0/4197040466 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b80b0070 con 0x7fb5c80a3d90 2024-04-26T00:31:17.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.636+0000 7fb5c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:17.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.636+0000 7fb5c67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:31:17.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.636+0000 7fb5c67fc640 10 monclient: dump: 2024-04-26T00:31:17.635 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:17.635 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:17.635 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:17.635 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:17.635 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:17.635 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:17.635 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:17.635 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:17.636 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:17.636 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:17.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.636+0000 7fb5c67fc640 10 monclient: _finish_auth 0 2024-04-26T00:31:17.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.636+0000 7fb5c67fc640 10 monclient: _check_auth_tickets 2024-04-26T00:31:17.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.636+0000 7fb5c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:31:17.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.636+0000 7fb5c67fc640 1 -- 172.21.15.37:0/4197040466 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5b80ab600 con 0x7fb5c80a3d90 2024-04-26T00:31:17.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.636+0000 7fb5cdd39640 5 monclient: authenticate success, global_id 84584 2024-04-26T00:31:17.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.636+0000 7fb5cdd39640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:31:17.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.636+0000 7fb5c67fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:31:17.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.636+0000 7fb5cdd39640 10 monclient: get_monmap_and_config success 2024-04-26T00:31:17.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.636+0000 7fb5cdd39640 4 set_mon_vals no callback set 2024-04-26T00:31:17.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.636+0000 7fb5c67fc640 1 -- 172.21.15.37:0/4197040466 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b80ab9d0 con 0x7fb5c80a3d90 2024-04-26T00:31:17.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.636+0000 7fb5c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:17.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.636+0000 7fb5c67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:31:17.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.636+0000 7fb5c67fc640 10 monclient: dump: 2024-04-26T00:31:17.637 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:17.637 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:17.637 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:17.637 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:17.637 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:17.637 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:17.637 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:17.637 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:17.637 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:17.638 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:17.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.636+0000 7fb5cdd39640 10 monclient: shutdown 2024-04-26T00:31:17.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.636+0000 7fb5cdd39640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:31:17.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.636+0000 7fb5cdd39640 1 -- 172.21.15.37:0/4197040466 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5c80a3d90 msgr2=0x7fb5c809bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:17.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.636+0000 7fb5cdd39640 1 --2- 172.21.15.37:0/4197040466 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5c80a3d90 0x7fb5c809bf40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb5b80097c0 tx=0x7fb5b801c320 comp rx=0 tx=0).stop 2024-04-26T00:31:17.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.636+0000 7fb5cdd39640 1 -- 172.21.15.37:0/4197040466 shutdown_connections 2024-04-26T00:31:17.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.636+0000 7fb5cdd39640 1 --2- 172.21.15.37:0/4197040466 >> v2:172.21.15.37:3300/0 conn(0x7fb5c80a4170 0x7fb5c80a45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:17.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.636+0000 7fb5cdd39640 1 --2- 172.21.15.37:0/4197040466 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5c80a3d90 0x7fb5c809bf40 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:17.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.636+0000 7fb5cdd39640 1 -- 172.21.15.37:0/4197040466 >> 172.21.15.37:0/4197040466 conn(0x7fb5c80aa0a0 msgr2=0x7fb5c80a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:31:17.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.636+0000 7fb5cdd39640 1 -- 172.21.15.37:0/4197040466 shutdown_connections 2024-04-26T00:31:17.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.636+0000 7fb5cdd39640 1 -- 172.21.15.37:0/4197040466 wait complete. 2024-04-26T00:31:17.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5cdd39640 10 monclient: build_initial_monmap 2024-04-26T00:31:17.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5cdd39640 10 monclient: monmap: 2024-04-26T00:31:17.639 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:31:17.639 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:31:17.639 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:31:17.639 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:31:17.639 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:31:17.639 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:31:17.639 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:31:17.639 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:31:17.639 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:31:17.639 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:17.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5cdd39640 1 Processor -- start 2024-04-26T00:31:17.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5cdd39640 1 -- start start 2024-04-26T00:31:17.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5cdd39640 10 monclient: init 2024-04-26T00:31:17.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5cdd39640 10 monclient: _reopen_session rank -1 2024-04-26T00:31:17.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5cdd39640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:31:17.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5cdd39640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5c80a3d90 0x7fb5c81cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:17.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5cdd39640 10 monclient(hunting): picked mon.noname-b con 0x7fb5c80a3d90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:31:17.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5cdd39640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5c80a4170 0x7fb5c813e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:17.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5cdd39640 10 monclient(hunting): picked mon.noname-c con 0x7fb5c80a4170 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:31:17.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5cdd39640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5c8133b20 0x7fb5c813e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:17.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5cdd39640 10 monclient(hunting): picked mon.noname-a con 0x7fb5c8133b20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:31:17.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5cdd39640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:17.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5cdd39640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb5c8141e30 con 0x7fb5c8133b20 2024-04-26T00:31:17.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5cdd39640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:17.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5cdd39640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb5c809c690 con 0x7fb5c80a3d90 2024-04-26T00:31:17.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5cdd39640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:17.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5cdd39640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb5c8141fb0 con 0x7fb5c80a4170 2024-04-26T00:31:17.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5cdd39640 10 monclient(hunting): _renew_subs 2024-04-26T00:31:17.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5cdd39640 10 monclient(hunting): authenticate will time out at 1658.949135s 2024-04-26T00:31:17.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5c6ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5c80a4170 0x7fb5c813e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:17.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5c6ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5c80a4170 0x7fb5c813e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:33702/0 (socket says 172.21.15.37:33702) 2024-04-26T00:31:17.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5c6ffd640 1 -- 172.21.15.37:0/2814720644 learned_addr learned my addr 172.21.15.37:0/2814720644 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:31:17.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5c6ffd640 10 monclient(hunting): get_auth_request con 0x7fb5c80a4170 auth_method 0 2024-04-26T00:31:17.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:17.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5c6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:17.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:17.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5c7fff640 1 --2- 172.21.15.37:0/2814720644 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5c8133b20 0x7fb5c813e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:17.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5c6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:17.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5c6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:17.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5c6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:17.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5c7fff640 10 monclient(hunting): get_auth_request con 0x7fb5c8133b20 auth_method 0 2024-04-26T00:31:17.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:17.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5c7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:17.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:17.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:17.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:17.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:17.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5c77fe640 1 --2- 172.21.15.37:0/2814720644 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5c80a3d90 0x7fb5c81cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:17.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5c6ffd640 10 monclient(hunting): handle_auth_done global_id 84587 payload 824 2024-04-26T00:31:17.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5c6ffd640 1 -- 172.21.15.37:0/2814720644 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5c80a3d90 msgr2=0x7fb5c81cf3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:17.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5c6ffd640 1 --2- 172.21.15.37:0/2814720644 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5c80a3d90 0x7fb5c81cf3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:17.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5c6ffd640 1 -- 172.21.15.37:0/2814720644 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5c8133b20 msgr2=0x7fb5c813e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:17.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5c6ffd640 1 --2- 172.21.15.37:0/2814720644 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5c8133b20 0x7fb5c813e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:17.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5c6ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:31:17.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5c6ffd640 1 monclient: found mon.noname-c 2024-04-26T00:31:17.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:31:17.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5c6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:31:17.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5c6ffd640 1 -- 172.21.15.37:0/2814720644 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5c813ca50 con 0x7fb5c80a4170 2024-04-26T00:31:17.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5c77fe640 10 monclient: get_auth_request con 0x7fb5c80a3d90 auth_method 0 2024-04-26T00:31:17.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5c77fe640 1 --2- 172.21.15.37:0/2814720644 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5c80a3d90 0x7fb5c81cf3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:31:17.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5c6ffd640 1 -- 172.21.15.37:0/2814720644 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5c80a4170 msgr2=0x7fb5c813e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:17.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.640+0000 7fb5c6ffd640 1 --2- 172.21.15.37:0/2814720644 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5c80a4170 0x7fb5c813e1b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc00e870 tx=0x7fb5bc00ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:17.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5affff640 1 -- 172.21.15.37:0/2814720644 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5bc017070 con 0x7fb5c80a4170 2024-04-26T00:31:17.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5affff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:17.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5affff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:31:17.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5affff640 10 monclient: dump: 2024-04-26T00:31:17.646 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:17.647 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:17.647 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:17.647 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:17.647 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:17.647 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:17.647 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:17.647 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:17.647 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:17.647 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:17.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5affff640 10 monclient: _finish_auth 0 2024-04-26T00:31:17.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5affff640 10 monclient: _check_auth_tickets 2024-04-26T00:31:17.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:31:17.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5affff640 1 -- 172.21.15.37:0/2814720644 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5bc00cb50 con 0x7fb5c80a4170 2024-04-26T00:31:17.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5cdd39640 5 monclient: authenticate success, global_id 84587 2024-04-26T00:31:17.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5cdd39640 10 monclient: _renew_subs 2024-04-26T00:31:17.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5cdd39640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:17.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5cdd39640 1 -- 172.21.15.37:0/2814720644 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5c809ada0 con 0x7fb5c80a4170 2024-04-26T00:31:17.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5affff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:31:17.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5affff640 1 -- 172.21.15.37:0/2814720644 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5bc013470 con 0x7fb5c80a4170 2024-04-26T00:31:17.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5affff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:17.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5affff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:31:17.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5affff640 10 monclient: dump: 2024-04-26T00:31:17.648 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:17.648 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:17.648 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:17.648 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:17.648 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:17.649 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:17.649 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:17.649 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:17.649 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:17.649 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:17.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5cdd39640 10 monclient: _renew_subs 2024-04-26T00:31:17.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5cdd39640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:17.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5cdd39640 1 -- 172.21.15.37:0/2814720644 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5c813a760 con 0x7fb5c80a4170 2024-04-26T00:31:17.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5c5ffb640 4 set_mon_vals no callback set 2024-04-26T00:31:17.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5cdd39640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:17.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5cdd39640 1 --2- 172.21.15.37:0/2814720644 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5c8138ac0 0x7fb5c8138ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:17.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5cdd39640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:17.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5cdd39640 1 -- 172.21.15.37:0/2814720644 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb5c81cf8f0 con 0x7fb5c8138ac0 2024-04-26T00:31:17.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5c77fe640 1 --2- 172.21.15.37:0/2814720644 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5c8138ac0 0x7fb5c8138ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:17.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5c77fe640 10 monclient: get_auth_request con 0x7fb5c8138ac0 auth_method 0 2024-04-26T00:31:17.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:17.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5c77fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:17.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:17.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5affff640 1 -- 172.21.15.37:0/2814720644 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb5bc01a070 con 0x7fb5c80a4170 2024-04-26T00:31:17.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:17.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:17.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:17.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5c77fe640 10 monclient(hunting): handle_auth_done global_id 124224 payload 824 2024-04-26T00:31:17.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5c77fe640 1 -- 172.21.15.37:0/2814720644 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb5c804f510 con 0x7fb5c8138ac0 2024-04-26T00:31:17.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5affff640 1 --2- 172.21.15.37:0/2814720644 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb5b4035f00 0x7fb5b40383c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:17.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5affff640 1 -- 172.21.15.37:0/2814720644 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(434..434 src has 370..434) ==== 3931+0+0 (secure 0 0 0) 0x7fb5bc048a80 con 0x7fb5c80a4170 2024-04-26T00:31:17.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5c7fff640 1 --2- 172.21.15.37:0/2814720644 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb5b4035f00 0x7fb5b40383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:17.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5c7fff640 10 monclient: get_auth_request con 0x7fb5b4035f00 auth_method 0 2024-04-26T00:31:17.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5c77fe640 1 -- 172.21.15.37:0/2814720644 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5c8138ac0 msgr2=0x7fb5c8138ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:17.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5c77fe640 1 --2- 172.21.15.37:0/2814720644 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5c8138ac0 0x7fb5c8138ea0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb5b80ad000 tx=0x7fb5b801ab60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:17.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5affff640 1 -- 172.21.15.37:0/2814720644 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b80bb020 con 0x7fb5c8138ac0 2024-04-26T00:31:17.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5affff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:17.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5affff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:31:17.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.644+0000 7fb5affff640 10 monclient: dump: 2024-04-26T00:31:17.651 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:17.651 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:17.651 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:17.651 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:17.651 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:17.651 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:17.652 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:17.652 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:17.652 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:17.652 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:17.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.648+0000 7fb5affff640 1 -- 172.21.15.37:0/2814720644 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb5b80b6470 con 0x7fb5c8138ac0 2024-04-26T00:31:17.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.648+0000 7fb5affff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:17.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.648+0000 7fb5affff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:31:17.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.648+0000 7fb5affff640 1 -- 172.21.15.37:0/2814720644 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5c8138ac0 msgr2=0x7fb5c8138ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:17.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.648+0000 7fb5affff640 1 --2- 172.21.15.37:0/2814720644 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5c8138ac0 0x7fb5c8138ea0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb5b80ad000 tx=0x7fb5b801ab60 comp rx=0 tx=0).stop 2024-04-26T00:31:17.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.648+0000 7fb5affff640 1 -- 172.21.15.37:0/2814720644 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5c8138ac0 msgr2=0x7fb5c8138ea0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:31:17.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.648+0000 7fb5affff640 1 --2- 172.21.15.37:0/2814720644 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5c8138ac0 0x7fb5c8138ea0 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb5b80ad000 tx=0x7fb5b801ab60 comp rx=0 tx=0).stop 2024-04-26T00:31:17.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.648+0000 7fb5c7fff640 1 --2- 172.21.15.37:0/2814720644 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb5b4035f00 0x7fb5b40383c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fb5c00055b0 tx=0x7fb5c0005540 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:17.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.652+0000 7fb5cdd39640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:31:17.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.652+0000 7fb5cdd39640 1 --2- 172.21.15.37:0/2814720644 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5c81393e0 0x7fb5c8134b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:17.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.652+0000 7fb5cdd39640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:17.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.652+0000 7fb5cdd39640 1 -- 172.21.15.37:0/2814720644 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb5c813b040 con 0x7fb5c81393e0 2024-04-26T00:31:17.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.652+0000 7fb5c77fe640 1 --2- 172.21.15.37:0/2814720644 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5c81393e0 0x7fb5c8134b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:17.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.652+0000 7fb5c77fe640 10 monclient: get_auth_request con 0x7fb5c81393e0 auth_method 0 2024-04-26T00:31:17.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.652+0000 7fb5c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:17.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.652+0000 7fb5c77fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:17.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.652+0000 7fb5c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:17.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.652+0000 7fb5c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:17.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.652+0000 7fb5c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:17.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.652+0000 7fb5c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:17.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.652+0000 7fb5c77fe640 10 monclient(hunting): handle_auth_done global_id 124227 payload 824 2024-04-26T00:31:17.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.652+0000 7fb5c77fe640 1 -- 172.21.15.37:0/2814720644 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb5c8075dc0 con 0x7fb5c81393e0 2024-04-26T00:31:17.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.656+0000 7fb5c77fe640 1 -- 172.21.15.37:0/2814720644 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5c81393e0 msgr2=0x7fb5c8134b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:17.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.656+0000 7fb5c77fe640 1 --2- 172.21.15.37:0/2814720644 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5c81393e0 0x7fb5c8134b30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb5b8000c10 tx=0x7fb5b80190c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:17.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.656+0000 7fb5affff640 1 -- 172.21.15.37:0/2814720644 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b80bb070 con 0x7fb5c81393e0 2024-04-26T00:31:17.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.656+0000 7fb5affff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:17.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.656+0000 7fb5affff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:31:17.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.656+0000 7fb5affff640 10 monclient: dump: 2024-04-26T00:31:17.656 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:17.656 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:17.656 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:17.656 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:17.656 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:17.656 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:17.656 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:17.656 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:17.656 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:17.656 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:17.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.656+0000 7fb5affff640 1 -- 172.21.15.37:0/2814720644 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7fb5b80c22c0 con 0x7fb5c81393e0 2024-04-26T00:31:17.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.656+0000 7fb5affff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:31:17.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.656+0000 7fb5affff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:31:17.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.656+0000 7fb5affff640 1 -- 172.21.15.37:0/2814720644 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5c81393e0 msgr2=0x7fb5c8134b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:17.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.656+0000 7fb5affff640 1 --2- 172.21.15.37:0/2814720644 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5c81393e0 0x7fb5c8134b30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb5b8000c10 tx=0x7fb5b80190c0 comp rx=0 tx=0).stop 2024-04-26T00:31:17.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.656+0000 7fb5affff640 1 -- 172.21.15.37:0/2814720644 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5c81393e0 msgr2=0x7fb5c8134b30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:31:17.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.656+0000 7fb5affff640 1 --2- 172.21.15.37:0/2814720644 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5c81393e0 0x7fb5c8134b30 secure :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb5b8000c10 tx=0x7fb5b80190c0 comp rx=0 tx=0).stop 2024-04-26T00:31:17.657 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:31:17.657 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:31:17.657 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:31:17.658 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:31:17.658 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 86, 2024-04-26T00:31:17.658 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 66535, 2024-04-26T00:31:17.658 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:31:17.658 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:31:17.658 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:31:17.658 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:31:17.658 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:31:17.658 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 7, 2024-04-26T00:31:17.658 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:31:17.658 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:31:17.658 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:31:17.658 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:31:17.659 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:31:17.659 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:31:17.659 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:31:17.659 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:31:17.659 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:31:17.659 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:31:17.659 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:31:17.659 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:31:17.659 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:31:17.659 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:31:17.659 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:31:17.659 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:31:17.659 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:31:17.659 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:31:17.660 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:31:17.660 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:31:17.660 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:31:17.660 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:31:17.660 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:31:17.660 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:31:17.660 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:31:17.660 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:31:17.660 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:31:17.660 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:31:17.660 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:17.660 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:17.660 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:31:17.660 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:31:17.661 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:31:17.661 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:31:17.661 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:31:17.661 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:31:17.661 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:31:17.661 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:31:17.661 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:31:17.661 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:31:17.661 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:31:17.661 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:31:17.661 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:31:17.661 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:31:17.661 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:31:17.661 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:31:17.662 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:31:17.662 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:31:17.662 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:31:17.662 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:31:17.662 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:31:17.662 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:31:17.662 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:31:17.662 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:31:17.662 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:31:17.662 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:31:17.662 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:31:17.662 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:31:17.662 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:31:17.662 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:31:17.663 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:17.663 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:31:17.663 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:17.663 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:31:17.663 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:31:17.663 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:31:17.663 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:31:17.663 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:17.663 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:31:17.663 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:31:17.663 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:17.663 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:17.663 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:17.664 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:31:17.664 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:31:17.664 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:17.664 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:17.664 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:17.664 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:17.664 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:31:17.664 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:31:17.664 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:31:17.664 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:31:17.664 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:31:17.664 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:17.665 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:17.665 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:31:17.665 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:31:17.665 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:31:17.665 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:31:17.665 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:17.665 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:31:17.665 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:31:17.665 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:17.665 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:17.665 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:17.665 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:31:17.665 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:31:17.666 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:17.666 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:17.666 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:17.666 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:17.666 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:31:17.666 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:31:17.666 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:31:17.666 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:31:17.666 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:31:17.666 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:17.666 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:17.666 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:31:17.666 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:31:17.666 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:31:17.667 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:31:17.667 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:17.667 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:31:17.667 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:31:17.667 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:17.667 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:17.667 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:17.667 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:31:17.667 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:31:17.667 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:17.667 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:17.667 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:17.667 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:17.667 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:31:17.668 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:31:17.668 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:31:17.668 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:31:17.668 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:31:17.668 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:17.668 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:17.668 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:17.668 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:31:17.668 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:31:17.668 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:17.668 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:31:17.668 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:31:17.668 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:31:17.668 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:17.668 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:31:17.669 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:31:17.669 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:17.669 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:31:17.669 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:31:17.669 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:31:17.669 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:17.669 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:17.669 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:17.669 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:31:17.669 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:31:17.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.656+0000 7fb5cdd39640 1 -- 172.21.15.37:0/2814720644 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb5b4035f00 msgr2=0x7fb5b40383c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:17.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.656+0000 7fb5cdd39640 1 --2- 172.21.15.37:0/2814720644 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb5b4035f00 0x7fb5b40383c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fb5c00055b0 tx=0x7fb5c0005540 comp rx=0 tx=0).stop 2024-04-26T00:31:17.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.656+0000 7fb5cdd39640 10 monclient: shutdown 2024-04-26T00:31:17.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.656+0000 7fb5cdd39640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:31:17.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.656+0000 7fb5cdd39640 1 -- 172.21.15.37:0/2814720644 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5c80a4170 msgr2=0x7fb5c813e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:17.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.656+0000 7fb5cdd39640 1 --2- 172.21.15.37:0/2814720644 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5c80a4170 0x7fb5c813e1b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc00e870 tx=0x7fb5bc00ed30 comp rx=0 tx=0).stop 2024-04-26T00:31:17.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.656+0000 7fb5c7fff640 1 -- 172.21.15.37:0/2814720644 reap_dead start 2024-04-26T00:31:17.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.656+0000 7fb5cdd39640 1 -- 172.21.15.37:0/2814720644 shutdown_connections 2024-04-26T00:31:17.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.656+0000 7fb5cdd39640 1 --2- 172.21.15.37:0/2814720644 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5c80a4170 0x7fb5c813e1b0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:17.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.656+0000 7fb5cdd39640 1 -- 172.21.15.37:0/2814720644 >> 172.21.15.37:0/2814720644 conn(0x7fb5c80aa0a0 msgr2=0x7fb5c80a1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:31:17.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.656+0000 7fb5cdd39640 1 -- 172.21.15.37:0/2814720644 shutdown_connections 2024-04-26T00:31:17.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.656+0000 7fb5cdd39640 1 -- 172.21.15.37:0/2814720644 wait complete. 2024-04-26T00:31:17.672 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:31:17.672 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:31:17.672 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:31:17.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.800+0000 7fafa486d640 10 monclient: get_monmap_and_config 2024-04-26T00:31:17.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.800+0000 7fafa486d640 10 monclient: build_initial_monmap 2024-04-26T00:31:17.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.800+0000 7fafa486d640 10 monclient: monmap: 2024-04-26T00:31:17.793 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:31:17.793 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:17.793 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:31:17.804361+0000 2024-04-26T00:31:17.793 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:31:17.804361+0000 2024-04-26T00:31:17.793 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:31:17.794 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:17.794 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:31:17.794 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:31:17.794 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:31:17.794 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:31:17.794 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:31:17.794 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:17.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.800+0000 7fafa486d640 1 Processor -- start 2024-04-26T00:31:17.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.800+0000 7fafa486d640 1 -- start start 2024-04-26T00:31:17.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.800+0000 7fafa486d640 10 monclient: init 2024-04-26T00:31:17.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.800+0000 7fafa486d640 10 monclient: _reopen_session rank -1 2024-04-26T00:31:17.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.800+0000 7fafa486d640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-04-26T00:31:17.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.800+0000 7fafa486d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fafa0130eb0 addr v1:172.21.15.45:6789/0 2024-04-26T00:31:17.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.800+0000 7fafa486d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fafa009ef20 addr v1:172.21.15.37:6789/0 2024-04-26T00:31:17.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.800+0000 7fafa486d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fafa013a5d0 0x7fafa013c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:17.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.800+0000 7fafa486d640 10 monclient(hunting): picked mon.noname-a con 0x7fafa013a5d0 addr v2:172.21.15.37:3300/0 2024-04-26T00:31:17.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.800+0000 7fafa486d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafa009c190 con 0x7fafa009ef20 2024-04-26T00:31:17.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.800+0000 7fafa486d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafa00a11c0 con 0x7fafa0130eb0 2024-04-26T00:31:17.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.800+0000 7fafa486d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:17.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.800+0000 7fafa486d640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fafa009df60 con 0x7fafa013a5d0 2024-04-26T00:31:17.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.800+0000 7fafa486d640 10 monclient(hunting): _renew_subs 2024-04-26T00:31:17.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.800+0000 7fafa486d640 10 monclient(hunting): authenticate will time out at 1659.110010s 2024-04-26T00:31:17.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.800+0000 7faf9ed9d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fafa013a5d0 0x7fafa013c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:17.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.800+0000 7faf9dd9b640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fafa009ef20 0x7fafa0138eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:42734/0 (socket says 172.21.15.37:42734) 2024-04-26T00:31:17.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.800+0000 7faf9ed9d640 10 monclient(hunting): get_auth_request con 0x7fafa013a5d0 auth_method 0 2024-04-26T00:31:17.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.800+0000 7faf9dd9b640 1 -- 172.21.15.37:0/868319423 learned_addr learned my addr 172.21.15.37:0/868319423 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:31:17.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.800+0000 7faf9ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:17.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.800+0000 7faf9ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:17.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.800+0000 7faf9ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:17.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.800+0000 7faf9ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:17.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.800+0000 7faf9ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:17.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.800+0000 7faf9ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:17.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.800+0000 7faf9ed9d640 10 monclient(hunting): handle_auth_done global_id 124230 payload 274 2024-04-26T00:31:17.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.800+0000 7faf9ed9d640 1 -- 172.21.15.37:0/868319423 >> v1:172.21.15.45:6789/0 conn(0x7fafa0130eb0 legacy=0x7fafa0131290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:31:17.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.800+0000 7faf9ed9d640 1 -- 172.21.15.37:0/868319423 >> v1:172.21.15.37:6789/0 conn(0x7fafa009ef20 legacy=0x7fafa0138eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:31:17.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.800+0000 7faf9ed9d640 10 monclient: _finish_hunting 0 2024-04-26T00:31:17.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.800+0000 7faf9ed9d640 1 monclient: found mon.noname-a 2024-04-26T00:31:17.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.800+0000 7faf9ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:31:17.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.800+0000 7faf9ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:31:17.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.800+0000 7faf9ed9d640 1 -- 172.21.15.37:0/868319423 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafa009dc80 con 0x7fafa013a5d0 2024-04-26T00:31:17.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.804+0000 7faf9ed9d640 1 -- 172.21.15.37:0/868319423 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafa013a5d0 msgr2=0x7fafa013c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:17.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.804+0000 7faf9ed9d640 1 --2- 172.21.15.37:0/868319423 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafa013a5d0 0x7fafa013c9c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7faf98009d90 tx=0x7faf9801c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8619d703fcade57 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:17.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.804+0000 7faf9d59a640 1 -- 172.21.15.37:0/868319423 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf980b2070 con 0x7fafa013a5d0 2024-04-26T00:31:17.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.804+0000 7faf9d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:17.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.804+0000 7faf9d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:31:17.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.804+0000 7faf9d59a640 10 monclient: dump: 2024-04-26T00:31:17.800 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:17.800 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:17.800 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:17.800 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:17.800 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:17.800 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:17.800 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:17.800 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:17.800 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:17.801 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:17.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.804+0000 7faf9d59a640 10 monclient: _finish_auth 0 2024-04-26T00:31:17.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.804+0000 7faf9d59a640 10 monclient: _check_auth_tickets 2024-04-26T00:31:17.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.804+0000 7faf9d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:31:17.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.804+0000 7faf9d59a640 1 -- 172.21.15.37:0/868319423 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf980a5940 con 0x7fafa013a5d0 2024-04-26T00:31:17.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.804+0000 7fafa486d640 5 monclient: authenticate success, global_id 124230 2024-04-26T00:31:17.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.804+0000 7fafa486d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:31:17.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.804+0000 7faf9d59a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:31:17.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.804+0000 7fafa486d640 10 monclient: get_monmap_and_config success 2024-04-26T00:31:17.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.804+0000 7faf9d59a640 1 -- 172.21.15.37:0/868319423 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf980a5d10 con 0x7fafa013a5d0 2024-04-26T00:31:17.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.804+0000 7fafa486d640 4 set_mon_vals no callback set 2024-04-26T00:31:17.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.804+0000 7faf9d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:17.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.804+0000 7faf9d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:31:17.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.804+0000 7faf9d59a640 10 monclient: dump: 2024-04-26T00:31:17.801 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:17.801 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:17.802 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:17.802 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:17.802 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:17.802 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:17.802 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:17.802 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:17.802 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:17.802 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:17.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.804+0000 7fafa486d640 10 monclient: shutdown 2024-04-26T00:31:17.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.804+0000 7fafa486d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:31:17.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.804+0000 7fafa486d640 1 -- 172.21.15.37:0/868319423 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafa013a5d0 msgr2=0x7fafa013c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:17.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.804+0000 7fafa486d640 1 --2- 172.21.15.37:0/868319423 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafa013a5d0 0x7fafa013c9c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7faf98009d90 tx=0x7faf9801c780 comp rx=0 tx=0).stop 2024-04-26T00:31:17.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.804+0000 7fafa486d640 1 -- 172.21.15.37:0/868319423 shutdown_connections 2024-04-26T00:31:17.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.804+0000 7fafa486d640 1 --2- 172.21.15.37:0/868319423 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafa013a5d0 0x7fafa013c9c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:17.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.804+0000 7fafa486d640 1 -- 172.21.15.37:0/868319423 >> 172.21.15.37:0/868319423 conn(0x7fafa00a1d50 msgr2=0x7fafa00a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:31:17.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.804+0000 7fafa486d640 1 -- 172.21.15.37:0/868319423 shutdown_connections 2024-04-26T00:31:17.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.804+0000 7fafa486d640 1 -- 172.21.15.37:0/868319423 wait complete. 2024-04-26T00:31:17.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7fafa486d640 10 monclient: build_initial_monmap 2024-04-26T00:31:17.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7fafa486d640 10 monclient: monmap: 2024-04-26T00:31:17.809 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:31:17.809 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:31:17.809 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:31:17.809 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:31:17.809 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:31:17.809 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:31:17.809 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:31:17.810 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:31:17.810 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:31:17.810 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:17.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7fafa486d640 1 Processor -- start 2024-04-26T00:31:17.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7fafa486d640 1 -- start start 2024-04-26T00:31:17.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7fafa486d640 10 monclient: init 2024-04-26T00:31:17.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7fafa486d640 10 monclient: _reopen_session rank -1 2024-04-26T00:31:17.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7fafa486d640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:31:17.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7fafa486d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fafa009ef20 0x7fafa0132a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:17.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7fafa486d640 10 monclient(hunting): picked mon.noname-c con 0x7fafa009ef20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:31:17.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7fafa486d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafa0130eb0 0x7fafa0132fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:17.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7fafa486d640 10 monclient(hunting): picked mon.noname-b con 0x7fafa0130eb0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:31:17.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7fafa486d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafa013a5d0 0x7fafa01334f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:17.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7fafa486d640 10 monclient(hunting): picked mon.noname-a con 0x7fafa013a5d0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:31:17.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7fafa486d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:17.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7fafa486d640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fafa013f8a0 con 0x7fafa013a5d0 2024-04-26T00:31:17.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7fafa486d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:17.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7fafa486d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fafa013f720 con 0x7fafa0130eb0 2024-04-26T00:31:17.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7fafa486d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:17.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7fafa486d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fafa013fa20 con 0x7fafa009ef20 2024-04-26T00:31:17.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7fafa486d640 10 monclient(hunting): _renew_subs 2024-04-26T00:31:17.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7fafa486d640 10 monclient(hunting): authenticate will time out at 1659.117247s 2024-04-26T00:31:17.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7faf9ed9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafa013a5d0 0x7fafa01334f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:17.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7faf9e59c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fafa009ef20 0x7fafa0132a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:17.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7faf9ed9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafa013a5d0 0x7fafa01334f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:56116/0 (socket says 172.21.15.37:56116) 2024-04-26T00:31:17.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7faf9ed9d640 1 -- 172.21.15.37:0/3121728549 learned_addr learned my addr 172.21.15.37:0/3121728549 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:31:17.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7faf9ed9d640 10 monclient(hunting): get_auth_request con 0x7fafa013a5d0 auth_method 0 2024-04-26T00:31:17.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7faf9ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:17.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7faf9ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:17.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7faf9ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:17.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7faf9e59c640 10 monclient(hunting): get_auth_request con 0x7fafa009ef20 auth_method 0 2024-04-26T00:31:17.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7faf9e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:17.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7faf9e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:17.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7faf9e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:17.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7faf9ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:17.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7faf9ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:17.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7faf9ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:17.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7faf9e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:17.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7faf9e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:17.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7faf9e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:17.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7faf9dd9b640 1 --2- 172.21.15.37:0/3121728549 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafa0130eb0 0x7fafa0132fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:17.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7faf9ed9d640 10 monclient(hunting): handle_auth_done global_id 124233 payload 824 2024-04-26T00:31:17.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7faf9ed9d640 1 -- 172.21.15.37:0/3121728549 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fafa009ef20 msgr2=0x7fafa0132a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:17.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7faf9ed9d640 1 --2- 172.21.15.37:0/3121728549 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fafa009ef20 0x7fafa0132a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:17.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7faf9ed9d640 1 -- 172.21.15.37:0/3121728549 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafa0130eb0 msgr2=0x7fafa0132fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:17.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7faf9ed9d640 1 --2- 172.21.15.37:0/3121728549 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafa0130eb0 0x7fafa0132fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:17.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7faf9ed9d640 10 monclient: _finish_hunting 0 2024-04-26T00:31:17.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7faf9ed9d640 1 monclient: found mon.noname-a 2024-04-26T00:31:17.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7faf9ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:31:17.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7faf9ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:31:17.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7faf9ed9d640 1 -- 172.21.15.37:0/3121728549 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafa01345c0 con 0x7fafa013a5d0 2024-04-26T00:31:17.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7faf9dd9b640 10 monclient: get_auth_request con 0x7fafa0130eb0 auth_method 0 2024-04-26T00:31:17.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.808+0000 7faf9dd9b640 1 --2- 172.21.15.37:0/3121728549 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fafa0130eb0 0x7fafa0132fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:31:17.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7faf9ed9d640 1 -- 172.21.15.37:0/3121728549 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafa013a5d0 msgr2=0x7fafa01334f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:17.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7faf9ed9d640 1 --2- 172.21.15.37:0/3121728549 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafa013a5d0 0x7fafa01334f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7faf980b0040 tx=0x7faf98019520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:17.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7faf8effd640 1 -- 172.21.15.37:0/3121728549 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf980bd020 con 0x7fafa013a5d0 2024-04-26T00:31:17.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7faf8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:17.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7faf8effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:31:17.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7faf8effd640 10 monclient: dump: 2024-04-26T00:31:17.814 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:17.814 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:17.814 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:17.814 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:17.814 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:17.815 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:17.815 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:17.815 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:17.815 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:17.815 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:17.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7faf8effd640 10 monclient: _finish_auth 0 2024-04-26T00:31:17.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7faf8effd640 10 monclient: _check_auth_tickets 2024-04-26T00:31:17.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7faf8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:31:17.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7faf8effd640 1 -- 172.21.15.37:0/3121728549 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf980b8880 con 0x7fafa013a5d0 2024-04-26T00:31:17.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7faf8effd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:31:17.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7faf8effd640 1 -- 172.21.15.37:0/3121728549 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf980b2040 con 0x7fafa013a5d0 2024-04-26T00:31:17.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7fafa486d640 5 monclient: authenticate success, global_id 124233 2024-04-26T00:31:17.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7fafa486d640 10 monclient: _renew_subs 2024-04-26T00:31:17.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7fafa486d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:31:17.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7fafa486d640 1 -- 172.21.15.37:0/3121728549 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafa01362f0 con 0x7fafa013a5d0 2024-04-26T00:31:17.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7faf8ffff640 4 set_mon_vals no callback set 2024-04-26T00:31:17.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7faf8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:17.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7faf8effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:31:17.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7faf8effd640 10 monclient: dump: 2024-04-26T00:31:17.816 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:17.816 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:17.816 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:17.816 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:17.816 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:17.816 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:17.816 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:17.816 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:17.816 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:17.816 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:17.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7fafa486d640 10 monclient: _renew_subs 2024-04-26T00:31:17.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7fafa486d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:31:17.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7fafa486d640 1 -- 172.21.15.37:0/3121728549 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafa0135120 con 0x7fafa013a5d0 2024-04-26T00:31:17.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7fafa486d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:17.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7fafa486d640 1 --2- 172.21.15.37:0/3121728549 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fafa0139760 0x7fafa01cf100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:17.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7fafa486d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:17.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7fafa486d640 1 -- 172.21.15.37:0/3121728549 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fafa01352f0 con 0x7fafa0139760 2024-04-26T00:31:17.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7faf9e59c640 1 --2- 172.21.15.37:0/3121728549 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fafa0139760 0x7fafa01cf100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:17.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7faf8effd640 1 -- 172.21.15.37:0/3121728549 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7faf980c0020 con 0x7fafa013a5d0 2024-04-26T00:31:17.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7faf9e59c640 10 monclient: get_auth_request con 0x7fafa0139760 auth_method 0 2024-04-26T00:31:17.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7faf9e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:17.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7faf9e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:17.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7faf9e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:17.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7faf9e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:17.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7faf9e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:17.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7faf9e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:17.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7faf8effd640 1 --2- 172.21.15.37:0/3121728549 >> v2:172.21.15.45:6812/2379918623 conn(0x7faf80035ed0 0x7faf80038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:17.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7faf8effd640 1 -- 172.21.15.37:0/3121728549 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(434..434 src has 370..434) ==== 3931+0+0 (secure 0 0 0) 0x7faf980e3ad0 con 0x7fafa013a5d0 2024-04-26T00:31:17.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7faf9e59c640 10 monclient(hunting): handle_auth_done global_id 84593 payload 824 2024-04-26T00:31:17.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7faf9e59c640 1 -- 172.21.15.37:0/3121728549 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fafa004f510 con 0x7fafa0139760 2024-04-26T00:31:17.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7faf9dd9b640 1 --2- 172.21.15.37:0/3121728549 >> v2:172.21.15.45:6812/2379918623 conn(0x7faf80035ed0 0x7faf80038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:17.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7faf9dd9b640 10 monclient: get_auth_request con 0x7faf80035ed0 auth_method 0 2024-04-26T00:31:17.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7faf9e59c640 1 -- 172.21.15.37:0/3121728549 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fafa0139760 msgr2=0x7fafa01cf100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:17.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7faf9e59c640 1 --2- 172.21.15.37:0/3121728549 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fafa0139760 0x7fafa01cf100 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7faf9000f770 tx=0x7faf9000fc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:17.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7faf8effd640 1 -- 172.21.15.37:0/3121728549 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf9000fe40 con 0x7fafa0139760 2024-04-26T00:31:17.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7faf8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:17.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7faf8effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:31:17.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.812+0000 7faf8effd640 10 monclient: dump: 2024-04-26T00:31:17.818 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:17.819 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:17.819 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:17.819 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:17.819 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:17.819 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:17.819 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:17.819 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:17.819 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:17.819 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:17.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.816+0000 7faf8effd640 1 -- 172.21.15.37:0/3121728549 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7faf90019070 con 0x7fafa0139760 2024-04-26T00:31:17.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.816+0000 7faf8effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:17.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.816+0000 7faf8effd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:31:17.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.816+0000 7faf8effd640 1 -- 172.21.15.37:0/3121728549 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fafa0139760 msgr2=0x7fafa01cf100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:17.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.816+0000 7faf8effd640 1 --2- 172.21.15.37:0/3121728549 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fafa0139760 0x7fafa01cf100 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7faf9000f770 tx=0x7faf9000fc30 comp rx=0 tx=0).stop 2024-04-26T00:31:17.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.816+0000 7faf8effd640 1 -- 172.21.15.37:0/3121728549 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fafa0139760 msgr2=0x7fafa01cf100 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:31:17.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.816+0000 7faf8effd640 1 --2- 172.21.15.37:0/3121728549 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fafa0139760 0x7fafa01cf100 secure :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7faf9000f770 tx=0x7faf9000fc30 comp rx=0 tx=0).stop 2024-04-26T00:31:17.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.816+0000 7faf9dd9b640 1 --2- 172.21.15.37:0/3121728549 >> v2:172.21.15.45:6812/2379918623 conn(0x7faf80035ed0 0x7faf80038390 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7faf94007fd0 tx=0x7faf94009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:17.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.820+0000 7fafa486d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:31:17.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.820+0000 7fafa486d640 1 --2- 172.21.15.37:0/3121728549 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fafa01cf640 0x7fafa01cfa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:17.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.820+0000 7fafa486d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:17.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.820+0000 7fafa486d640 1 -- 172.21.15.37:0/3121728549 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fafa009b660 con 0x7fafa01cf640 2024-04-26T00:31:17.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.820+0000 7faf9e59c640 1 --2- 172.21.15.37:0/3121728549 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fafa01cf640 0x7fafa01cfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:17.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.820+0000 7faf9e59c640 10 monclient: get_auth_request con 0x7fafa01cf640 auth_method 0 2024-04-26T00:31:17.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.820+0000 7faf9e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:17.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.820+0000 7faf9e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:17.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.820+0000 7faf9e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:17.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.820+0000 7faf9e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:17.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.820+0000 7faf9e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:17.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.820+0000 7faf9e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:17.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.820+0000 7faf9e59c640 10 monclient(hunting): handle_auth_done global_id 84596 payload 824 2024-04-26T00:31:17.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.820+0000 7faf9e59c640 1 -- 172.21.15.37:0/3121728549 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fafa00a0f60 con 0x7fafa01cf640 2024-04-26T00:31:17.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.824+0000 7faf9e59c640 1 -- 172.21.15.37:0/3121728549 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fafa01cf640 msgr2=0x7fafa01cfa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:17.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.824+0000 7faf9e59c640 1 --2- 172.21.15.37:0/3121728549 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fafa01cf640 0x7fafa01cfa20 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7faf9000f770 tx=0x7faf90003770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:17.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.824+0000 7faf8effd640 1 -- 172.21.15.37:0/3121728549 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf90006e00 con 0x7fafa01cf640 2024-04-26T00:31:17.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.824+0000 7faf8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:17.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.824+0000 7faf8effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:31:17.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.824+0000 7faf8effd640 10 monclient: dump: 2024-04-26T00:31:17.823 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:17.823 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:17.823 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:17.823 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:17.824 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:17.824 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:17.824 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:17.824 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:17.824 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:17.824 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:17.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.824+0000 7faf8effd640 1 -- 172.21.15.37:0/3121728549 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4889 (secure 0 0 0) 0x7faf90006e00 con 0x7fafa01cf640 2024-04-26T00:31:17.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.824+0000 7faf8effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:31:17.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.824+0000 7faf8effd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:31:17.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.824+0000 7faf8effd640 1 -- 172.21.15.37:0/3121728549 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fafa01cf640 msgr2=0x7fafa01cfa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:17.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.824+0000 7faf8effd640 1 --2- 172.21.15.37:0/3121728549 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fafa01cf640 0x7fafa01cfa20 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7faf9000f770 tx=0x7faf90003770 comp rx=0 tx=0).stop 2024-04-26T00:31:17.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.824+0000 7faf8effd640 1 -- 172.21.15.37:0/3121728549 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fafa01cf640 msgr2=0x7fafa01cfa20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:31:17.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.824+0000 7faf8effd640 1 --2- 172.21.15.37:0/3121728549 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fafa01cf640 0x7fafa01cfa20 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:17.825 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:31:17.825 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:31:17.825 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:31:17.825 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:31:17.825 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 86, 2024-04-26T00:31:17.825 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 282760, 2024-04-26T00:31:17.825 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:31:17.825 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:31:17.825 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:31:17.825 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:31:17.826 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:31:17.826 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 8, 2024-04-26T00:31:17.826 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:31:17.826 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:31:17.826 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:31:17.826 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:31:17.826 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:31:17.826 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:31:17.826 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:31:17.826 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:31:17.827 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:31:17.827 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:31:17.827 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:31:17.827 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:31:17.827 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:31:17.827 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:31:17.827 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:31:17.827 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:31:17.827 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:31:17.827 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:31:17.827 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:31:17.827 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:31:17.827 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:31:17.827 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:31:17.827 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:31:17.828 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:31:17.828 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:31:17.828 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:31:17.828 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:31:17.828 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:31:17.828 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:17.828 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:17.828 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:31:17.828 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:31:17.828 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:31:17.828 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:31:17.828 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:31:17.828 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:31:17.828 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:31:17.828 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:31:17.829 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:31:17.829 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:31:17.829 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:31:17.829 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:31:17.829 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:31:17.829 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:31:17.829 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:31:17.829 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:31:17.829 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:31:17.829 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:31:17.829 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:31:17.829 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:31:17.829 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:31:17.830 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:31:17.830 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:31:17.830 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:31:17.830 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:31:17.830 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:31:17.830 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:31:17.830 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:31:17.830 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:31:17.830 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:31:17.830 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:17.830 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:31:17.830 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:17.831 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:31:17.831 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:31:17.831 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:31:17.831 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:31:17.831 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:17.831 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:31:17.831 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:31:17.831 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:17.831 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:17.831 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:17.832 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:31:17.832 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:31:17.832 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:17.832 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:17.832 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:17.832 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:17.832 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:31:17.832 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:31:17.832 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:31:17.832 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:31:17.832 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:31:17.832 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:17.832 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:17.832 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:31:17.833 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:31:17.833 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:31:17.833 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:31:17.833 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:17.833 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:31:17.833 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:31:17.833 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:17.833 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:17.833 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:17.833 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:31:17.833 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:31:17.833 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:17.833 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:17.833 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:17.833 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:17.834 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:31:17.834 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:31:17.834 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:31:17.834 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:31:17.834 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:31:17.834 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:17.835 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:17.835 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:31:17.835 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:31:17.835 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:31:17.835 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:31:17.835 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:17.835 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:31:17.835 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:31:17.835 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:17.836 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:17.836 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:17.836 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:31:17.836 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:31:17.836 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:17.836 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:17.836 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:17.836 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:17.836 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:31:17.836 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:31:17.836 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:31:17.836 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:31:17.836 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:31:17.836 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:17.837 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:17.837 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:17.837 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:31:17.837 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:31:17.837 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:17.837 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:31:17.837 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:31:17.837 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:31:17.837 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:17.837 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:31:17.837 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:31:17.837 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:17.837 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:31:17.837 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:31:17.838 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:31:17.838 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:17.838 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:31:17.838 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:31:17.838 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:17.838 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:31:17.838 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:31:17.838 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:31:17.838 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:17.838 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:31:17.838 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:31:17.838 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:17.839 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:31:17.839 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:31:17.839 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:31:17.839 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:17.839 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:17.839 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:17.839 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:31:17.839 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:31:17.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.824+0000 7fafa486d640 1 -- 172.21.15.37:0/3121728549 >> v2:172.21.15.45:6812/2379918623 conn(0x7faf80035ed0 msgr2=0x7faf80038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:17.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.824+0000 7fafa486d640 1 --2- 172.21.15.37:0/3121728549 >> v2:172.21.15.45:6812/2379918623 conn(0x7faf80035ed0 0x7faf80038390 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7faf94007fd0 tx=0x7faf94009040 comp rx=0 tx=0).stop 2024-04-26T00:31:17.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.824+0000 7fafa486d640 10 monclient: shutdown 2024-04-26T00:31:17.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.824+0000 7fafa486d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:31:17.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.824+0000 7fafa486d640 1 -- 172.21.15.37:0/3121728549 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafa013a5d0 msgr2=0x7fafa01334f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:17.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.824+0000 7fafa486d640 1 --2- 172.21.15.37:0/3121728549 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafa013a5d0 0x7fafa01334f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7faf980b0040 tx=0x7faf98019520 comp rx=0 tx=0).stop 2024-04-26T00:31:17.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.824+0000 7faf9ed9d640 1 -- 172.21.15.37:0/3121728549 reap_dead start 2024-04-26T00:31:17.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.824+0000 7faf9ed9d640 1 -- 172.21.15.37:0/3121728549 reap_dead start 2024-04-26T00:31:17.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.824+0000 7fafa486d640 1 -- 172.21.15.37:0/3121728549 shutdown_connections 2024-04-26T00:31:17.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.824+0000 7fafa486d640 1 -- 172.21.15.37:0/3121728549 >> 172.21.15.37:0/3121728549 conn(0x7fafa00a1d50 msgr2=0x7fafa012fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:31:17.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.824+0000 7fafa486d640 1 -- 172.21.15.37:0/3121728549 shutdown_connections 2024-04-26T00:31:17.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.824+0000 7fafa486d640 1 -- 172.21.15.37:0/3121728549 wait complete. 2024-04-26T00:31:17.844 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:31:17.844 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:31:17.844 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:31:17.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b2b74640 10 monclient: get_monmap_and_config 2024-04-26T00:31:17.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b2b74640 10 monclient: build_initial_monmap 2024-04-26T00:31:17.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b2b74640 10 monclient: monmap: 2024-04-26T00:31:17.961 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:31:17.961 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:17.961 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:31:17.972588+0000 2024-04-26T00:31:17.961 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:31:17.972588+0000 2024-04-26T00:31:17.961 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:31:17.962 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:17.962 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:31:17.962 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:31:17.962 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:31:17.962 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:31:17.962 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:31:17.962 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:17.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b2b74640 1 Processor -- start 2024-04-26T00:31:17.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b2b74640 1 -- start start 2024-04-26T00:31:17.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b2b74640 10 monclient: init 2024-04-26T00:31:17.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b2b74640 10 monclient: _reopen_session rank -1 2024-04-26T00:31:17.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b2b74640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-04-26T00:31:17.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b2b74640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f53ac0a2fe0 0x7f53ac0a13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:17.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b2b74640 10 monclient(hunting): picked mon.noname-c con 0x7f53ac0a2fe0 addr v2:172.21.15.45:3300/0 2024-04-26T00:31:17.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b2b74640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f53ac071da0 addr v1:172.21.15.37:6789/0 2024-04-26T00:31:17.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b2b74640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53ac1391c0 0x7f53ac13b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:17.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b2b74640 10 monclient(hunting): picked mon.noname-b con 0x7f53ac1391c0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:31:17.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b2b74640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53ac072d70 con 0x7f53ac071da0 2024-04-26T00:31:17.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b2b74640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:17.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b2b74640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f53ac0710e0 con 0x7f53ac0a2fe0 2024-04-26T00:31:17.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b2b74640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:17.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b2b74640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f53ac0a33c0 con 0x7f53ac1391c0 2024-04-26T00:31:17.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b2b74640 10 monclient(hunting): _renew_subs 2024-04-26T00:31:17.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b2b74640 10 monclient(hunting): authenticate will time out at 1659.278967s 2024-04-26T00:31:17.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53abfff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f53ac071da0 0x7f53ac0a1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:42738/0 (socket says 172.21.15.37:42738) 2024-04-26T00:31:17.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53abfff640 1 -- 172.21.15.37:0/1046368461 learned_addr learned my addr 172.21.15.37:0/1046368461 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:31:17.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b1111640 1 --2- 172.21.15.37:0/1046368461 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53ac1391c0 0x7f53ac13b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:17.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b1111640 10 monclient(hunting): get_auth_request con 0x7f53ac1391c0 auth_method 0 2024-04-26T00:31:17.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b1111640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:17.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b1111640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:17.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b1111640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:17.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b0910640 1 --2- 172.21.15.37:0/1046368461 >> v2:172.21.15.45:3300/0 conn(0x7f53ac0a2fe0 0x7f53ac0a13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:17.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b0910640 10 monclient(hunting): get_auth_request con 0x7f53ac0a2fe0 auth_method 0 2024-04-26T00:31:17.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b0910640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:17.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b0910640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:17.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b0910640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:17.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b1111640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:17.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b1111640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:17.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b1111640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:17.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b0910640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:17.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b0910640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:17.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b0910640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:17.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b1111640 10 monclient(hunting): handle_auth_done global_id 84599 payload 274 2024-04-26T00:31:17.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b1111640 1 -- 172.21.15.37:0/1046368461 >> v2:172.21.15.45:3300/0 conn(0x7f53ac0a2fe0 msgr2=0x7f53ac0a13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:31:17.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b1111640 1 --2- 172.21.15.37:0/1046368461 >> v2:172.21.15.45:3300/0 conn(0x7f53ac0a2fe0 0x7f53ac0a13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:17.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b1111640 1 -- 172.21.15.37:0/1046368461 >> v1:172.21.15.37:6789/0 conn(0x7f53ac071da0 legacy=0x7f53ac0a1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:31:17.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b1111640 10 monclient: _finish_hunting 0 2024-04-26T00:31:17.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b1111640 1 monclient: found mon.noname-b 2024-04-26T00:31:17.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b1111640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:31:17.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b1111640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:31:17.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.968+0000 7f53b1111640 1 -- 172.21.15.37:0/1046368461 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53ac070b20 con 0x7f53ac1391c0 2024-04-26T00:31:17.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.972+0000 7f53b1111640 1 -- 172.21.15.37:0/1046368461 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53ac1391c0 msgr2=0x7f53ac13b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:17.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.972+0000 7f53b1111640 1 --2- 172.21.15.37:0/1046368461 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53ac1391c0 0x7f53ac13b5b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f53ac06f600 tx=0x7f53a401c7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=388fdacd768d2538 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:17.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53ab7fe640 1 -- 172.21.15.37:0/1046368461 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a40ac070 con 0x7f53ac1391c0 2024-04-26T00:31:17.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:17.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53ab7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:31:17.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53ab7fe640 10 monclient: dump: 2024-04-26T00:31:17.970 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:17.970 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:17.971 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:17.971 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:17.971 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:17.971 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:17.971 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:17.971 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:17.971 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:17.971 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:17.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53ab7fe640 10 monclient: _finish_auth 0 2024-04-26T00:31:17.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53ab7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:31:17.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:31:17.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b2b74640 5 monclient: authenticate success, global_id 84599 2024-04-26T00:31:17.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53ab7fe640 1 -- 172.21.15.37:0/1046368461 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53a40a6640 con 0x7f53ac1391c0 2024-04-26T00:31:17.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b2b74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:31:17.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53ab7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:31:17.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b2b74640 10 monclient: get_monmap_and_config success 2024-04-26T00:31:17.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b2b74640 4 set_mon_vals no callback set 2024-04-26T00:31:17.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53ab7fe640 1 -- 172.21.15.37:0/1046368461 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a40a6a10 con 0x7f53ac1391c0 2024-04-26T00:31:17.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:17.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53ab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:31:17.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53ab7fe640 10 monclient: dump: 2024-04-26T00:31:17.972 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:17.972 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:17.972 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:17.972 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:17.972 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:17.972 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:17.972 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:17.972 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:17.973 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:17.973 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:17.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b2b74640 10 monclient: shutdown 2024-04-26T00:31:17.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b2b74640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:31:17.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b2b74640 1 -- 172.21.15.37:0/1046368461 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53ac1391c0 msgr2=0x7f53ac13b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:17.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b2b74640 1 --2- 172.21.15.37:0/1046368461 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53ac1391c0 0x7f53ac13b5b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f53ac06f600 tx=0x7f53a401c7b0 comp rx=0 tx=0).stop 2024-04-26T00:31:17.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b2b74640 1 -- 172.21.15.37:0/1046368461 shutdown_connections 2024-04-26T00:31:17.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b2b74640 1 --2- 172.21.15.37:0/1046368461 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53ac1391c0 0x7f53ac13b5b0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:17.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b2b74640 1 --2- 172.21.15.37:0/1046368461 >> v2:172.21.15.45:3300/0 conn(0x7f53ac0a2fe0 0x7f53ac0a13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:17.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b2b74640 1 -- 172.21.15.37:0/1046368461 >> 172.21.15.37:0/1046368461 conn(0x7f53ac0a24c0 msgr2=0x7f53ac0a28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:31:17.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b2b74640 1 -- 172.21.15.37:0/1046368461 shutdown_connections 2024-04-26T00:31:17.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b2b74640 1 -- 172.21.15.37:0/1046368461 wait complete. 2024-04-26T00:31:17.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b2b74640 10 monclient: build_initial_monmap 2024-04-26T00:31:17.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b2b74640 10 monclient: monmap: 2024-04-26T00:31:17.973 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:31:17.974 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:31:17.974 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:31:17.974 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:31:17.974 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:31:17.974 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:31:17.974 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:31:17.974 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:31:17.974 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:31:17.974 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:17.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b2b74640 1 Processor -- start 2024-04-26T00:31:17.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b2b74640 1 -- start start 2024-04-26T00:31:17.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b2b74640 10 monclient: init 2024-04-26T00:31:17.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b2b74640 10 monclient: _reopen_session rank -1 2024-04-26T00:31:17.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b2b74640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:31:17.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b2b74640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53ac071da0 0x7f53ac138350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:17.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b2b74640 10 monclient(hunting): picked mon.noname-a con 0x7f53ac071da0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:31:17.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b2b74640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53ac0a2fe0 0x7f53ac138890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:17.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b2b74640 10 monclient(hunting): picked mon.noname-c con 0x7f53ac0a2fe0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:31:17.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b2b74640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53ac1391c0 0x7f53ac11fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:17.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b2b74640 10 monclient(hunting): picked mon.noname-b con 0x7f53ac1391c0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:31:17.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b2b74640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:17.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b2b74640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f53ac13e6a0 con 0x7f53ac071da0 2024-04-26T00:31:17.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b2b74640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:17.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b2b74640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f53ac13e520 con 0x7f53ac1391c0 2024-04-26T00:31:17.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b2b74640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:17.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b2b74640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f53ac13e820 con 0x7f53ac0a2fe0 2024-04-26T00:31:17.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b2b74640 10 monclient(hunting): _renew_subs 2024-04-26T00:31:17.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b2b74640 10 monclient(hunting): authenticate will time out at 1659.286966s 2024-04-26T00:31:17.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b0910640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53ac071da0 0x7f53ac138350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:17.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53abfff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53ac0a2fe0 0x7f53ac138890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:17.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b0910640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53ac071da0 0x7f53ac138350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:56128/0 (socket says 172.21.15.37:56128) 2024-04-26T00:31:17.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b0910640 1 -- 172.21.15.37:0/661807569 learned_addr learned my addr 172.21.15.37:0/661807569 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:31:17.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b0910640 10 monclient(hunting): get_auth_request con 0x7f53ac071da0 auth_method 0 2024-04-26T00:31:17.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b0910640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:17.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b0910640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:17.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b0910640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:17.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53abfff640 10 monclient(hunting): get_auth_request con 0x7f53ac0a2fe0 auth_method 0 2024-04-26T00:31:17.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:17.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53abfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:17.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53abfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:17.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b0910640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:17.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b0910640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:17.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b0910640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:17.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:17.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:17.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:17.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b1111640 1 --2- 172.21.15.37:0/661807569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53ac1391c0 0x7f53ac11fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:17.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b1111640 10 monclient(hunting): get_auth_request con 0x7f53ac1391c0 auth_method 0 2024-04-26T00:31:17.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b1111640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:17.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b1111640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:17.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b1111640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:17.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53abfff640 10 monclient(hunting): handle_auth_done global_id 84602 payload 824 2024-04-26T00:31:17.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53abfff640 1 -- 172.21.15.37:0/661807569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53ac1391c0 msgr2=0x7f53ac11fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:17.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53abfff640 1 --2- 172.21.15.37:0/661807569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53ac1391c0 0x7f53ac11fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:17.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53abfff640 1 -- 172.21.15.37:0/661807569 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53ac071da0 msgr2=0x7f53ac138350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:17.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53abfff640 1 --2- 172.21.15.37:0/661807569 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53ac071da0 0x7f53ac138350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:17.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53abfff640 10 monclient: _finish_hunting 0 2024-04-26T00:31:17.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53abfff640 1 monclient: found mon.noname-c 2024-04-26T00:31:17.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:31:17.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53abfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:31:17.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53abfff640 1 -- 172.21.15.37:0/661807569 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53ac123d10 con 0x7f53ac0a2fe0 2024-04-26T00:31:17.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.976+0000 7f53b0910640 1 --2- 172.21.15.37:0/661807569 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53ac071da0 0x7f53ac138350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:31:17.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53abfff640 1 -- 172.21.15.37:0/661807569 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53ac0a2fe0 msgr2=0x7f53ac138890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:17.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53abfff640 1 --2- 172.21.15.37:0/661807569 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53ac0a2fe0 0x7f53ac138890 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f539800b490 tx=0x7f539800b950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:17.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53a97fa640 1 -- 172.21.15.37:0/661807569 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5398015020 con 0x7f53ac0a2fe0 2024-04-26T00:31:17.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53a97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:17.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53a97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:31:17.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53a97fa640 10 monclient: dump: 2024-04-26T00:31:17.984 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:17.985 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:17.985 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:17.985 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:17.985 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:17.985 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:17.985 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:17.985 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:17.985 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:17.985 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:17.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53a97fa640 10 monclient: _finish_auth 0 2024-04-26T00:31:17.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53a97fa640 10 monclient: _check_auth_tickets 2024-04-26T00:31:17.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53a97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:31:17.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53a97fa640 1 -- 172.21.15.37:0/661807569 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5398011a60 con 0x7f53ac0a2fe0 2024-04-26T00:31:17.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53a97fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:31:17.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53a97fa640 1 -- 172.21.15.37:0/661807569 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5398011e50 con 0x7f53ac0a2fe0 2024-04-26T00:31:17.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53a97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:17.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53a97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:31:17.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53a97fa640 10 monclient: dump: 2024-04-26T00:31:17.986 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:17.986 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:17.986 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:17.986 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:17.986 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:17.986 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:17.986 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:17.986 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:17.986 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:17.986 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:17.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53b2b74640 5 monclient: authenticate success, global_id 84602 2024-04-26T00:31:17.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53b2b74640 10 monclient: _renew_subs 2024-04-26T00:31:17.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53b2b74640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:17.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53aaffd640 4 set_mon_vals no callback set 2024-04-26T00:31:17.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53b2b74640 1 -- 172.21.15.37:0/661807569 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53ac0701d0 con 0x7f53ac0a2fe0 2024-04-26T00:31:17.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53b2b74640 10 monclient: _renew_subs 2024-04-26T00:31:17.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53b2b74640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:17.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53b2b74640 1 -- 172.21.15.37:0/661807569 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f53ac123470 con 0x7f53ac0a2fe0 2024-04-26T00:31:17.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53b2b74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:17.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53b2b74640 1 --2- 172.21.15.37:0/661807569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53ac1cac30 0x7f53ac1202c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:17.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53b2b74640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:17.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53b2b74640 1 -- 172.21.15.37:0/661807569 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f53ac0688f0 con 0x7f53ac1cac30 2024-04-26T00:31:17.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53a97fa640 1 -- 172.21.15.37:0/661807569 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f5398018020 con 0x7f53ac0a2fe0 2024-04-26T00:31:17.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53b0910640 1 --2- 172.21.15.37:0/661807569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53ac1cac30 0x7f53ac1202c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:17.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53a97fa640 1 --2- 172.21.15.37:0/661807569 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5390036050 0x7f5390038510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:17.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53b0910640 10 monclient: get_auth_request con 0x7f53ac1cac30 auth_method 0 2024-04-26T00:31:17.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53b0910640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:17.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53b0910640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:17.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53b0910640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:17.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53a97fa640 1 -- 172.21.15.37:0/661807569 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(434..434 src has 370..434) ==== 3931+0+0 (secure 0 0 0) 0x7f5398048ab0 con 0x7f53ac0a2fe0 2024-04-26T00:31:17.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53b0910640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:17.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53b0910640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:17.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53b0910640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:17.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53b1111640 1 --2- 172.21.15.37:0/661807569 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5390036050 0x7f5390038510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:17.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53b1111640 10 monclient: get_auth_request con 0x7f5390036050 auth_method 0 2024-04-26T00:31:17.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53b0910640 10 monclient(hunting): handle_auth_done global_id 134122 payload 824 2024-04-26T00:31:17.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.980+0000 7f53b0910640 1 -- 172.21.15.37:0/661807569 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f53ac123670 con 0x7f53ac1cac30 2024-04-26T00:31:17.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.984+0000 7f53b0910640 1 -- 172.21.15.37:0/661807569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53ac1cac30 msgr2=0x7f53ac1202c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:17.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.984+0000 7f53b0910640 1 --2- 172.21.15.37:0/661807569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53ac1cac30 0x7f53ac1202c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f53a0009ec0 tx=0x7f53a000f610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:17.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.984+0000 7f53a97fa640 1 -- 172.21.15.37:0/661807569 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a0004020 con 0x7f53ac1cac30 2024-04-26T00:31:17.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.984+0000 7f53a97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:17.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.984+0000 7f53a97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:31:17.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.984+0000 7f53a97fa640 10 monclient: dump: 2024-04-26T00:31:17.989 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:17.989 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:17.989 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:17.989 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:17.989 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:17.989 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:17.989 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:17.989 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:17.989 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:17.989 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:17.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.984+0000 7f53b1111640 1 --2- 172.21.15.37:0/661807569 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5390036050 0x7f5390038510 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f53ac071570 tx=0x7f53a40a7000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:17.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.984+0000 7f53a97fa640 1 -- 172.21.15.37:0/661807569 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f53a0004020 con 0x7f53ac1cac30 2024-04-26T00:31:17.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.984+0000 7f53a97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:17.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.984+0000 7f53a97fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:31:17.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.984+0000 7f53a97fa640 1 -- 172.21.15.37:0/661807569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53ac1cac30 msgr2=0x7f53ac1202c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:17.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.984+0000 7f53a97fa640 1 --2- 172.21.15.37:0/661807569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53ac1cac30 0x7f53ac1202c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f53a0009ec0 tx=0x7f53a000f610 comp rx=0 tx=0).stop 2024-04-26T00:31:17.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.984+0000 7f53a97fa640 1 -- 172.21.15.37:0/661807569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53ac1cac30 msgr2=0x7f53ac1202c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:31:17.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.984+0000 7f53a97fa640 1 --2- 172.21.15.37:0/661807569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53ac1cac30 0x7f53ac1202c0 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f53a0009ec0 tx=0x7f53a000f610 comp rx=0 tx=0).stop 2024-04-26T00:31:17.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.992+0000 7f53b2b74640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:31:17.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.992+0000 7f53b2b74640 1 --2- 172.21.15.37:0/661807569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53ac120800 0x7f53ac120c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:17.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.992+0000 7f53b2b74640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:17.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.992+0000 7f53b2b74640 1 -- 172.21.15.37:0/661807569 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f53ac121150 con 0x7f53ac120800 2024-04-26T00:31:17.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.992+0000 7f53b0910640 1 --2- 172.21.15.37:0/661807569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53ac120800 0x7f53ac120c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:17.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.992+0000 7f53b0910640 10 monclient: get_auth_request con 0x7f53ac120800 auth_method 0 2024-04-26T00:31:17.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.992+0000 7f53b0910640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:17.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.992+0000 7f53b0910640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:17.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.992+0000 7f53b0910640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:17.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.992+0000 7f53b0910640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:17.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.992+0000 7f53b0910640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:17.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.992+0000 7f53b0910640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:17.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.992+0000 7f53b0910640 10 monclient(hunting): handle_auth_done global_id 134125 payload 824 2024-04-26T00:31:17.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.992+0000 7f53b0910640 1 -- 172.21.15.37:0/661807569 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f53ac04f510 con 0x7f53ac120800 2024-04-26T00:31:17.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.996+0000 7f53b0910640 1 -- 172.21.15.37:0/661807569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53ac120800 msgr2=0x7f53ac120c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:17.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.996+0000 7f53b0910640 1 --2- 172.21.15.37:0/661807569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53ac120800 0x7f53ac120c10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f53a001b000 tx=0x7f53a00046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:17.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.996+0000 7f53a97fa640 1 -- 172.21.15.37:0/661807569 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a0003a30 con 0x7f53ac120800 2024-04-26T00:31:17.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.996+0000 7f53a97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:17.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.996+0000 7f53a97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:31:17.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.996+0000 7f53a97fa640 10 monclient: dump: 2024-04-26T00:31:17.993 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:17.993 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:17.993 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:17.993 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:17.993 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:17.993 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:17.993 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:17.993 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:17.994 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:17.994 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:17.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.996+0000 7f53a97fa640 1 -- 172.21.15.37:0/661807569 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7f53a0003a30 con 0x7f53ac120800 2024-04-26T00:31:17.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.996+0000 7f53a97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:31:17.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.996+0000 7f53a97fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:31:17.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.996+0000 7f53a97fa640 1 -- 172.21.15.37:0/661807569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53ac120800 msgr2=0x7f53ac120c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:17.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.996+0000 7f53a97fa640 1 --2- 172.21.15.37:0/661807569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53ac120800 0x7f53ac120c10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f53a001b000 tx=0x7f53a00046d0 comp rx=0 tx=0).stop 2024-04-26T00:31:17.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.996+0000 7f53a97fa640 1 -- 172.21.15.37:0/661807569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53ac120800 msgr2=0x7f53ac120c10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:31:17.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.996+0000 7f53a97fa640 1 --2- 172.21.15.37:0/661807569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53ac120800 0x7f53ac120c10 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f53a001b000 tx=0x7f53a00046d0 comp rx=0 tx=0).stop 2024-04-26T00:31:17.994 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:31:17.995 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:31:17.995 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:31:17.995 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:31:17.995 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 86, 2024-04-26T00:31:17.995 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 8359, 2024-04-26T00:31:17.995 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:31:17.995 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:31:17.995 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:31:17.995 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:31:17.995 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:31:17.995 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 8, 2024-04-26T00:31:17.995 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:31:17.995 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:31:17.996 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:31:17.996 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:31:17.996 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:31:17.996 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:31:17.996 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:31:17.996 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:31:17.996 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:31:17.996 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:31:17.996 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:31:17.996 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:31:17.996 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:31:17.996 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:31:17.996 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:31:17.996 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:31:17.996 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:31:17.997 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:31:17.997 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:31:17.997 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:31:17.997 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:31:17.997 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:31:17.997 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:31:17.997 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:31:17.997 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:31:17.997 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:31:17.997 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:31:17.997 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:31:17.997 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:17.997 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:17.997 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:31:17.997 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:31:17.998 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:31:17.998 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:31:17.998 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:31:17.998 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:31:17.998 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:31:17.998 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:31:17.998 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:31:17.998 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:31:17.998 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:31:17.998 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:31:17.998 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:31:17.998 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:31:17.998 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:31:17.998 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:31:17.999 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:31:17.999 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:31:17.999 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:31:17.999 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:31:17.999 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:31:17.999 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:31:17.999 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:31:17.999 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:31:17.999 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:31:17.999 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:31:17.999 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:31:17.999 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:31:17.999 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:31:17.999 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:31:18.000 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:18.000 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:31:18.000 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:18.000 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:31:18.000 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:31:18.000 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:31:18.000 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:31:18.000 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:18.000 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:31:18.000 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:31:18.000 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:18.000 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:18.000 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:18.000 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:31:18.000 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:31:18.001 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:18.001 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:18.001 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:18.001 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:18.001 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:31:18.001 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:31:18.001 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:31:18.001 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:31:18.001 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:31:18.001 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:18.001 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:18.001 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:31:18.001 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:31:18.001 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:31:18.001 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:31:18.002 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:18.002 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:31:18.002 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:31:18.002 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:18.002 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:18.002 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:18.002 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:31:18.002 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:31:18.002 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:18.002 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:18.002 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:18.002 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:18.002 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:31:18.003 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:31:18.003 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:31:18.003 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:31:18.003 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:31:18.003 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:18.003 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:18.003 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:31:18.003 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:31:18.003 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:31:18.003 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:31:18.003 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:18.003 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:31:18.003 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:31:18.003 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:18.004 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:18.004 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:18.004 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:31:18.004 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:31:18.004 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:18.004 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:18.004 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:18.004 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:18.004 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:31:18.004 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:31:18.004 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:31:18.004 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:31:18.004 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:31:18.005 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:18.005 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:18.005 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:18.005 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:31:18.005 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:31:18.005 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:18.005 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:31:18.005 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:31:18.005 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:31:18.005 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:18.005 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:31:18.005 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:31:18.005 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:18.005 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:31:18.006 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:31:18.006 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:31:18.006 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:18.006 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:18.006 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:18.006 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:31:18.006 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:31:18.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.996+0000 7f53b2b74640 1 -- 172.21.15.37:0/661807569 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5390036050 msgr2=0x7f5390038510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:18.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.996+0000 7f53b2b74640 1 --2- 172.21.15.37:0/661807569 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5390036050 0x7f5390038510 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f53ac071570 tx=0x7f53a40a7000 comp rx=0 tx=0).stop 2024-04-26T00:31:18.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.996+0000 7f53b2b74640 10 monclient: shutdown 2024-04-26T00:31:18.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.996+0000 7f53b2b74640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:31:18.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.996+0000 7f53b2b74640 1 -- 172.21.15.37:0/661807569 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53ac0a2fe0 msgr2=0x7f53ac138890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:18.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.996+0000 7f53b2b74640 1 --2- 172.21.15.37:0/661807569 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53ac0a2fe0 0x7f53ac138890 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f539800b490 tx=0x7f539800b950 comp rx=0 tx=0).stop 2024-04-26T00:31:18.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.996+0000 7f53b1111640 1 -- 172.21.15.37:0/661807569 reap_dead start 2024-04-26T00:31:18.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.996+0000 7f53b1111640 1 -- 172.21.15.37:0/661807569 reap_dead start 2024-04-26T00:31:18.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.996+0000 7f53b2b74640 1 -- 172.21.15.37:0/661807569 shutdown_connections 2024-04-26T00:31:18.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.996+0000 7f53b2b74640 1 -- 172.21.15.37:0/661807569 >> 172.21.15.37:0/661807569 conn(0x7f53ac0a24c0 msgr2=0x7f53ac09c9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:31:18.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.996+0000 7f53b2b74640 1 -- 172.21.15.37:0/661807569 shutdown_connections 2024-04-26T00:31:18.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:17.996+0000 7f53b2b74640 1 -- 172.21.15.37:0/661807569 wait complete. 2024-04-26T00:31:18.009 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:31:18.009 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-04-26T00:31:18.009 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-26T00:31:18.009 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-04-26T00:31:18.009 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-04-26T00:31:18.009 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-26T00:31:18.009 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T00:31:18.021 INFO:tasks.ceph.mon.a:Stopped 2024-04-26T00:31:18.021 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-26T00:31:18.022 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:31:18.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.192+0000 7ff9e373d640 10 monclient: get_monmap_and_config 2024-04-26T00:31:18.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.192+0000 7ff9e373d640 10 monclient: build_initial_monmap 2024-04-26T00:31:18.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.192+0000 7ff9e373d640 10 monclient: monmap: 2024-04-26T00:31:18.186 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:31:18.186 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:18.187 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:31:18.197192+0000 2024-04-26T00:31:18.187 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:31:18.197192+0000 2024-04-26T00:31:18.187 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:31:18.187 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:18.187 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:31:18.187 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:31:18.187 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:31:18.187 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:31:18.188 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:31:18.188 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:18.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.192+0000 7ff9e373d640 1 Processor -- start 2024-04-26T00:31:18.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.192+0000 7ff9e373d640 1 -- start start 2024-04-26T00:31:18.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.192+0000 7ff9e373d640 10 monclient: init 2024-04-26T00:31:18.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.192+0000 7ff9e373d640 10 monclient: _reopen_session rank -1 2024-04-26T00:31:18.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.192+0000 7ff9e373d640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-04-26T00:31:18.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.192+0000 7ff9e373d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff9dc0a0260 addr v1:172.21.15.45:6789/0 2024-04-26T00:31:18.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.192+0000 7ff9e373d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff9dc137e40 0x7ff9dc13a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:18.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.192+0000 7ff9e373d640 10 monclient(hunting): picked mon.noname-c con 0x7ff9dc137e40 addr v2:172.21.15.45:3300/0 2024-04-26T00:31:18.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.192+0000 7ff9e373d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff9dc13a770 addr v1:172.21.15.37:6789/0 2024-04-26T00:31:18.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.192+0000 7ff9e373d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9dc0a1130 con 0x7ff9dc13a770 2024-04-26T00:31:18.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.192+0000 7ff9e373d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9dc06cff0 con 0x7ff9dc0a0260 2024-04-26T00:31:18.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.192+0000 7ff9e373d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:18.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.192+0000 7ff9e373d640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7ff9dc0a0d50 con 0x7ff9dc137e40 2024-04-26T00:31:18.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.192+0000 7ff9e373d640 10 monclient(hunting): _renew_subs 2024-04-26T00:31:18.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.192+0000 7ff9e373d640 10 monclient(hunting): authenticate will time out at 1659.503103s 2024-04-26T00:31:18.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.192+0000 7ff9e1cda640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7ff9dc13a770 legacy=0x7ff9dc13cb60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:31:18.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.192+0000 7ff9e0cd8640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff9dc137e40 0x7ff9dc13a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:18.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.192+0000 7ff9e0cd8640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff9dc137e40 0x7ff9dc13a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.37:55296/0 (socket says 172.21.15.37:55296) 2024-04-26T00:31:18.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.192+0000 7ff9e0cd8640 1 -- 172.21.15.37:0/1450421128 learned_addr learned my addr 172.21.15.37:0/1450421128 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:31:18.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.192+0000 7ff9e0cd8640 10 monclient(hunting): get_auth_request con 0x7ff9dc137e40 auth_method 0 2024-04-26T00:31:18.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.192+0000 7ff9e0cd8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:18.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.192+0000 7ff9e0cd8640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:18.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.192+0000 7ff9e0cd8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:18.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.192+0000 7ff9e0cd8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:18.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.192+0000 7ff9e0cd8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:18.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.192+0000 7ff9e0cd8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:18.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.196+0000 7ff9e14d9640 1 -- 172.21.15.37:0/1450421128 >> v1:172.21.15.45:6789/0 conn(0x7ff9dc0a0260 legacy=0x7ff9dc0a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:18.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.196+0000 7ff9e0cd8640 10 monclient(hunting): handle_auth_done global_id 134128 payload 274 2024-04-26T00:31:18.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.196+0000 7ff9e0cd8640 1 -- 172.21.15.37:0/1450421128 >> v1:172.21.15.45:6789/0 conn(0x7ff9dc0a0260 legacy=0x7ff9dc0a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:18.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.196+0000 7ff9e0cd8640 1 -- 172.21.15.37:0/1450421128 >> v1:172.21.15.37:6789/0 conn(0x7ff9dc13a770 legacy=0x7ff9dc13cb60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:31:18.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.196+0000 7ff9e0cd8640 10 monclient: _finish_hunting 0 2024-04-26T00:31:18.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.196+0000 7ff9e0cd8640 1 monclient: found mon.noname-c 2024-04-26T00:31:18.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.196+0000 7ff9e0cd8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:31:18.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.196+0000 7ff9e0cd8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:31:18.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.196+0000 7ff9e0cd8640 1 -- 172.21.15.37:0/1450421128 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9dc09bc50 con 0x7ff9dc137e40 2024-04-26T00:31:18.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.200+0000 7ff9e0cd8640 1 -- 172.21.15.37:0/1450421128 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9dc137e40 msgr2=0x7ff9dc13a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:18.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.200+0000 7ff9e0cd8640 1 --2- 172.21.15.37:0/1450421128 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9dc137e40 0x7ff9dc13a230 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc0098f0 tx=0x7ff9cc01c280 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44dd55e62af0d3b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:18.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.200+0000 7ff9d3fff640 1 -- 172.21.15.37:0/1450421128 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc0b0070 con 0x7ff9dc137e40 2024-04-26T00:31:18.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.200+0000 7ff9d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:18.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.200+0000 7ff9d3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:31:18.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.200+0000 7ff9d3fff640 10 monclient: dump: 2024-04-26T00:31:18.197 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:18.197 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:18.197 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:18.197 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:18.198 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:18.198 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:18.198 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:18.198 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:18.198 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:18.198 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:18.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.200+0000 7ff9d3fff640 10 monclient: _finish_auth 0 2024-04-26T00:31:18.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.200+0000 7ff9d3fff640 10 monclient: _check_auth_tickets 2024-04-26T00:31:18.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.200+0000 7ff9d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:31:18.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.200+0000 7ff9d3fff640 1 -- 172.21.15.37:0/1450421128 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9cc0ab600 con 0x7ff9dc137e40 2024-04-26T00:31:18.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.200+0000 7ff9e373d640 5 monclient: authenticate success, global_id 134128 2024-04-26T00:31:18.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.200+0000 7ff9e373d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:31:18.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.200+0000 7ff9d3fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:31:18.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.200+0000 7ff9e373d640 10 monclient: get_monmap_and_config success 2024-04-26T00:31:18.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.200+0000 7ff9e373d640 4 set_mon_vals no callback set 2024-04-26T00:31:18.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.200+0000 7ff9d3fff640 1 -- 172.21.15.37:0/1450421128 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc0ab9d0 con 0x7ff9dc137e40 2024-04-26T00:31:18.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.200+0000 7ff9d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:18.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.200+0000 7ff9d3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:31:18.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.200+0000 7ff9d3fff640 10 monclient: dump: 2024-04-26T00:31:18.199 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:18.199 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:18.199 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:18.199 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:18.199 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:18.199 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:18.199 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:18.199 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:18.199 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:18.199 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:18.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.200+0000 7ff9e373d640 10 monclient: shutdown 2024-04-26T00:31:18.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.200+0000 7ff9e373d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:31:18.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.200+0000 7ff9e373d640 1 -- 172.21.15.37:0/1450421128 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9dc137e40 msgr2=0x7ff9dc13a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:18.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.200+0000 7ff9e373d640 1 --2- 172.21.15.37:0/1450421128 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9dc137e40 0x7ff9dc13a230 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc0098f0 tx=0x7ff9cc01c280 comp rx=0 tx=0).stop 2024-04-26T00:31:18.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.200+0000 7ff9e373d640 1 -- 172.21.15.37:0/1450421128 shutdown_connections 2024-04-26T00:31:18.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.200+0000 7ff9e373d640 1 --2- 172.21.15.37:0/1450421128 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9dc137e40 0x7ff9dc13a230 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:18.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.200+0000 7ff9e373d640 1 -- 172.21.15.37:0/1450421128 >> 172.21.15.37:0/1450421128 conn(0x7ff9dc0a1ad0 msgr2=0x7ff9dc0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:31:18.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e373d640 1 -- 172.21.15.37:0/1450421128 shutdown_connections 2024-04-26T00:31:18.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e373d640 1 -- 172.21.15.37:0/1450421128 wait complete. 2024-04-26T00:31:18.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e373d640 10 monclient: build_initial_monmap 2024-04-26T00:31:18.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e373d640 10 monclient: monmap: 2024-04-26T00:31:18.200 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:31:18.200 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:31:18.200 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:31:18.200 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:31:18.200 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:31:18.201 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:31:18.201 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:31:18.201 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:31:18.201 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:31:18.201 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:18.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e373d640 1 Processor -- start 2024-04-26T00:31:18.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e373d640 1 -- start start 2024-04-26T00:31:18.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e373d640 10 monclient: init 2024-04-26T00:31:18.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e373d640 10 monclient: _reopen_session rank -1 2024-04-26T00:31:18.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e373d640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:31:18.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e373d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9dc0a0260 0x7ff9dc1cf1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:18.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e373d640 10 monclient(hunting): picked mon.noname-b con 0x7ff9dc0a0260 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:31:18.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e373d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9dc137e40 0x7ff9dc134ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:18.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e373d640 10 monclient(hunting): picked mon.noname-a con 0x7ff9dc137e40 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:31:18.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e373d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9dc13a770 0x7ff9dc1cbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:18.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e373d640 10 monclient(hunting): picked mon.noname-c con 0x7ff9dc13a770 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:31:18.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e373d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:18.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e373d640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff9dc09c920 con 0x7ff9dc137e40 2024-04-26T00:31:18.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e373d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:18.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e373d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff9dc09c7a0 con 0x7ff9dc0a0260 2024-04-26T00:31:18.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e373d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:18.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e373d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff9dc09caa0 con 0x7ff9dc13a770 2024-04-26T00:31:18.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e373d640 10 monclient(hunting): _renew_subs 2024-04-26T00:31:18.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e373d640 10 monclient(hunting): authenticate will time out at 1659.513487s 2024-04-26T00:31:18.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e0cd8640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9dc137e40 msgr2=0x7ff9dc134ed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:31:18.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e0cd8640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9dc137e40 0x7ff9dc134ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:31:18.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e1cda640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9dc13a770 0x7ff9dc1cbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:18.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e1cda640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9dc13a770 0x7ff9dc1cbcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:33764/0 (socket says 172.21.15.37:33764) 2024-04-26T00:31:18.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e1cda640 1 -- 172.21.15.37:0/1263499214 learned_addr learned my addr 172.21.15.37:0/1263499214 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:31:18.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e1cda640 10 monclient(hunting): get_auth_request con 0x7ff9dc13a770 auth_method 0 2024-04-26T00:31:18.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e1cda640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:18.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e1cda640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:18.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e1cda640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:18.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e1cda640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:18.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e1cda640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:18.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e1cda640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:18.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e14d9640 1 --2- 172.21.15.37:0/1263499214 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9dc0a0260 0x7ff9dc1cf1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:18.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e14d9640 10 monclient(hunting): get_auth_request con 0x7ff9dc0a0260 auth_method 0 2024-04-26T00:31:18.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e14d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:18.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e14d9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:18.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e14d9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:18.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e14d9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:18.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e14d9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:18.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e14d9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:18.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e1cda640 10 monclient(hunting): handle_auth_done global_id 84605 payload 824 2024-04-26T00:31:18.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e1cda640 1 -- 172.21.15.37:0/1263499214 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9dc0a0260 msgr2=0x7ff9dc1cf1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:18.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e1cda640 1 --2- 172.21.15.37:0/1263499214 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9dc0a0260 0x7ff9dc1cf1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:18.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e1cda640 1 -- 172.21.15.37:0/1263499214 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9dc137e40 msgr2=0x7ff9dc134ed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:31:18.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e1cda640 1 --2- 172.21.15.37:0/1263499214 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9dc137e40 0x7ff9dc134ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:18.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e1cda640 10 monclient: _finish_hunting 0 2024-04-26T00:31:18.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e1cda640 1 monclient: found mon.noname-c 2024-04-26T00:31:18.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e1cda640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:31:18.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e1cda640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:31:18.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.204+0000 7ff9e1cda640 1 -- 172.21.15.37:0/1263499214 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9dc13db60 con 0x7ff9dc13a770 2024-04-26T00:31:18.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.208+0000 7ff9e1cda640 1 -- 172.21.15.37:0/1263499214 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9dc13a770 msgr2=0x7ff9dc1cbcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:18.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.208+0000 7ff9e1cda640 1 --2- 172.21.15.37:0/1263499214 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9dc13a770 0x7ff9dc1cbcc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff9d400be90 tx=0x7ff9d400bf80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:18.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.208+0000 7ff9d1ffb640 1 -- 172.21.15.37:0/1263499214 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d4016630 con 0x7ff9dc13a770 2024-04-26T00:31:18.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.208+0000 7ff9d1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:18.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.208+0000 7ff9d1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:31:18.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.208+0000 7ff9d1ffb640 10 monclient: dump: 2024-04-26T00:31:18.208 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:18.208 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:18.208 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:18.208 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:18.209 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:18.209 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:18.209 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:18.209 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:18.209 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:18.209 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:18.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.208+0000 7ff9d1ffb640 10 monclient: _finish_auth 0 2024-04-26T00:31:18.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.208+0000 7ff9d1ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:31:18.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.208+0000 7ff9d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:31:18.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.208+0000 7ff9e373d640 5 monclient: authenticate success, global_id 84605 2024-04-26T00:31:18.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.208+0000 7ff9e373d640 10 monclient: _renew_subs 2024-04-26T00:31:18.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.208+0000 7ff9e373d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:18.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.208+0000 7ff9e373d640 1 -- 172.21.15.37:0/1263499214 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9dc12d0a0 con 0x7ff9dc13a770 2024-04-26T00:31:18.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.208+0000 7ff9e373d640 10 monclient: _renew_subs 2024-04-26T00:31:18.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.208+0000 7ff9e373d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:18.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.208+0000 7ff9d1ffb640 1 -- 172.21.15.37:0/1263499214 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9d4010030 con 0x7ff9dc13a770 2024-04-26T00:31:18.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.208+0000 7ff9e373d640 1 -- 172.21.15.37:0/1263499214 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9dc1cfc20 con 0x7ff9dc13a770 2024-04-26T00:31:18.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.208+0000 7ff9d1ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:31:18.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.208+0000 7ff9d1ffb640 1 -- 172.21.15.37:0/1263499214 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d4016a10 con 0x7ff9dc13a770 2024-04-26T00:31:18.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.208+0000 7ff9d1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:18.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.208+0000 7ff9d1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:31:18.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.208+0000 7ff9d1ffb640 10 monclient: dump: 2024-04-26T00:31:18.210 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:18.210 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:18.210 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:18.210 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:18.210 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:18.210 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:18.210 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:18.211 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:18.211 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:18.211 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:18.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.208+0000 7ff9d2ffd640 4 set_mon_vals no callback set 2024-04-26T00:31:18.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.208+0000 7ff9e373d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:18.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.208+0000 7ff9e373d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:18.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.208+0000 7ff9e373d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:18.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.208+0000 7ff9e373d640 1 -- 172.21.15.37:0/1263499214 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9dc066f40 con 0x7ff9dc13a770 2024-04-26T00:31:18.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.208+0000 7ff9d1ffb640 1 -- 172.21.15.37:0/1263499214 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff9d40153e0 con 0x7ff9dc13a770 2024-04-26T00:31:18.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.208+0000 7ff9d1ffb640 1 --2- 172.21.15.37:0/1263499214 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff9c0035e30 0x7ff9c00382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:18.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.208+0000 7ff9d1ffb640 1 -- 172.21.15.37:0/1263499214 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(434..434 src has 370..434) ==== 3931+0+0 (secure 0 0 0) 0x7ff9d40494b0 con 0x7ff9dc13a770 2024-04-26T00:31:18.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.208+0000 7ff9e14d9640 1 --2- 172.21.15.37:0/1263499214 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff9c0035e30 0x7ff9c00382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:18.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.208+0000 7ff9e14d9640 10 monclient: get_auth_request con 0x7ff9c0035e30 auth_method 0 2024-04-26T00:31:18.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.212+0000 7ff9e14d9640 1 --2- 172.21.15.37:0/1263499214 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff9c0035e30 0x7ff9c00382f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8007fd0 tx=0x7ff9d8007ce0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:18.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.212+0000 7ff9d1ffb640 1 -- 172.21.15.37:0/1263499214 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff9d4048200 con 0x7ff9dc13a770 2024-04-26T00:31:18.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.212+0000 7ff9d1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:18.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.212+0000 7ff9d1ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:31:18.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.436+0000 7ff9e373d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:31:18.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.436+0000 7ff9e373d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:31:18.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.436+0000 7ff9e373d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:18.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.436+0000 7ff9e373d640 1 -- 172.21.15.37:0/1263499214 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff9dc13e060 con 0x7ff9dc13a770 2024-04-26T00:31:18.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.436+0000 7ff9d1ffb640 1 -- 172.21.15.37:0/1263499214 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7ff9d40156d0 con 0x7ff9dc13a770 2024-04-26T00:31:18.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.436+0000 7ff9d1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:31:18.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.436+0000 7ff9d1ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:31:18.430 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":86,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:31:18.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.436+0000 7ff9e373d640 1 -- 172.21.15.37:0/1263499214 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff9c0035e30 msgr2=0x7ff9c00382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:18.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.436+0000 7ff9e373d640 1 --2- 172.21.15.37:0/1263499214 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff9c0035e30 0x7ff9c00382f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8007fd0 tx=0x7ff9d8007ce0 comp rx=0 tx=0).stop 2024-04-26T00:31:18.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.436+0000 7ff9e373d640 10 monclient: shutdown 2024-04-26T00:31:18.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.436+0000 7ff9e373d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:31:18.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.436+0000 7ff9e373d640 1 -- 172.21.15.37:0/1263499214 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9dc13a770 msgr2=0x7ff9dc1cbcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:18.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.436+0000 7ff9e373d640 1 --2- 172.21.15.37:0/1263499214 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9dc13a770 0x7ff9dc1cbcc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff9d400be90 tx=0x7ff9d400bf80 comp rx=0 tx=0).stop 2024-04-26T00:31:18.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.436+0000 7ff9e373d640 1 -- 172.21.15.37:0/1263499214 shutdown_connections 2024-04-26T00:31:18.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.436+0000 7ff9e373d640 1 --2- 172.21.15.37:0/1263499214 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff9c0035e30 0x7ff9c00382f0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:18.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.436+0000 7ff9e373d640 1 --2- 172.21.15.37:0/1263499214 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9dc13a770 0x7ff9dc1cbcc0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:18.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.436+0000 7ff9e373d640 1 --2- 172.21.15.37:0/1263499214 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9dc137e40 0x7ff9dc134ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:18.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.436+0000 7ff9e373d640 1 --2- 172.21.15.37:0/1263499214 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9dc0a0260 0x7ff9dc1cf1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:18.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.436+0000 7ff9e373d640 1 -- 172.21.15.37:0/1263499214 >> 172.21.15.37:0/1263499214 conn(0x7ff9dc0a1ad0 msgr2=0x7ff9dc0a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:31:18.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.436+0000 7ff9e373d640 1 -- 172.21.15.37:0/1263499214 shutdown_connections 2024-04-26T00:31:18.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:18.436+0000 7ff9e373d640 1 -- 172.21.15.37:0/1263499214 wait complete. 2024-04-26T00:31:21.449 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:31:21.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.588+0000 7ff08d84b640 10 monclient: get_monmap_and_config 2024-04-26T00:31:21.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.588+0000 7ff08d84b640 10 monclient: build_initial_monmap 2024-04-26T00:31:21.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.588+0000 7ff08d84b640 10 monclient: monmap: 2024-04-26T00:31:21.582 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:31:21.582 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:21.582 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:31:21.593349+0000 2024-04-26T00:31:21.582 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:31:21.593349+0000 2024-04-26T00:31:21.583 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:31:21.583 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:21.583 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:31:21.583 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:31:21.583 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:31:21.583 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:31:21.583 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:31:21.583 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:21.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.588+0000 7ff08d84b640 1 Processor -- start 2024-04-26T00:31:21.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.588+0000 7ff08d84b640 1 -- start start 2024-04-26T00:31:21.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.588+0000 7ff08d84b640 10 monclient: init 2024-04-26T00:31:21.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.588+0000 7ff08d84b640 10 monclient: _reopen_session rank -1 2024-04-26T00:31:21.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.588+0000 7ff08d84b640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-04-26T00:31:21.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.588+0000 7ff08d84b640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff0880a0260 0x7ff0880a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:21.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.588+0000 7ff08d84b640 10 monclient(hunting): picked mon.noname-a con 0x7ff0880a0260 addr v2:172.21.15.37:3300/0 2024-04-26T00:31:21.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.588+0000 7ff08d84b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff088134b30 0x7ff088134f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:21.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.588+0000 7ff08d84b640 10 monclient(hunting): picked mon.noname-c con 0x7ff088134b30 addr v2:172.21.15.45:3300/0 2024-04-26T00:31:21.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.588+0000 7ff08d84b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff088135490 addr v1:172.21.15.45:6789/0 2024-04-26T00:31:21.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.588+0000 7ff08d84b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0880a1130 con 0x7ff088135490 2024-04-26T00:31:21.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.588+0000 7ff08d84b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:21.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.588+0000 7ff08d84b640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7ff08809ec10 con 0x7ff0880a0260 2024-04-26T00:31:21.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.588+0000 7ff08d84b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:21.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.588+0000 7ff08d84b640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7ff08809f0f0 con 0x7ff088134b30 2024-04-26T00:31:21.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.588+0000 7ff08d84b640 10 monclient(hunting): _renew_subs 2024-04-26T00:31:21.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.588+0000 7ff08d84b640 10 monclient(hunting): authenticate will time out at 1662.899276s 2024-04-26T00:31:21.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.588+0000 7ff086ffd640 1 -- >> v2:172.21.15.37:3300/0 conn(0x7ff0880a0260 msgr2=0x7ff0880a0640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:31:21.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.588+0000 7ff086ffd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff0880a0260 0x7ff0880a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:31:21.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.588+0000 7ff0867fc640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff088134b30 0x7ff088134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:21.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.588+0000 7ff0867fc640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff088134b30 0x7ff088134f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.37:56230/0 (socket says 172.21.15.37:56230) 2024-04-26T00:31:21.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.588+0000 7ff0867fc640 1 -- 172.21.15.37:0/4135861781 learned_addr learned my addr 172.21.15.37:0/4135861781 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:31:21.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.588+0000 7ff0867fc640 10 monclient(hunting): get_auth_request con 0x7ff088134b30 auth_method 0 2024-04-26T00:31:21.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.588+0000 7ff0867fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:21.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.588+0000 7ff0867fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:21.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.588+0000 7ff0867fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:21.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.588+0000 7ff0867fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:21.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.588+0000 7ff0867fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:21.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.588+0000 7ff0867fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:21.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.592+0000 7ff0877fe640 1 -- 172.21.15.37:0/4135861781 >> v1:172.21.15.45:6789/0 conn(0x7ff088135490 legacy=0x7ff08813be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:21.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.592+0000 7ff0867fc640 10 monclient(hunting): handle_auth_done global_id 134137 payload 274 2024-04-26T00:31:21.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.592+0000 7ff0867fc640 1 -- 172.21.15.37:0/4135861781 >> v2:172.21.15.37:3300/0 conn(0x7ff0880a0260 msgr2=0x7ff0880a0640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:31:21.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.592+0000 7ff0867fc640 1 --2- 172.21.15.37:0/4135861781 >> v2:172.21.15.37:3300/0 conn(0x7ff0880a0260 0x7ff0880a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:21.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.592+0000 7ff0867fc640 1 -- 172.21.15.37:0/4135861781 >> v1:172.21.15.45:6789/0 conn(0x7ff088135490 legacy=0x7ff08813be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:21.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.592+0000 7ff0867fc640 10 monclient: _finish_hunting 0 2024-04-26T00:31:21.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.592+0000 7ff0867fc640 1 monclient: found mon.noname-c 2024-04-26T00:31:21.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.592+0000 7ff0867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:31:21.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.592+0000 7ff0867fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:31:21.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.592+0000 7ff0867fc640 1 -- 172.21.15.37:0/4135861781 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff08806cff0 con 0x7ff088134b30 2024-04-26T00:31:21.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.596+0000 7ff0867fc640 1 -- 172.21.15.37:0/4135861781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff088134b30 msgr2=0x7ff088134f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:21.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.596+0000 7ff0867fc640 1 --2- 172.21.15.37:0/4135861781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff088134b30 0x7ff088134f50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff08809f690 tx=0x7ff07001c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b6deb3e2d285ff3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:21.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff085ffb640 1 -- 172.21.15.37:0/4135861781 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0700b0070 con 0x7ff088134b30 2024-04-26T00:31:21.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff085ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:21.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff085ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:31:21.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff085ffb640 10 monclient: dump: 2024-04-26T00:31:21.598 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:21.598 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:21.598 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:21.598 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:21.598 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:21.598 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:21.598 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:21.598 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:21.598 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:21.598 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:21.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff085ffb640 10 monclient: _finish_auth 0 2024-04-26T00:31:21.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff085ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:31:21.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff085ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:31:21.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff085ffb640 1 -- 172.21.15.37:0/4135861781 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0700ab600 con 0x7ff088134b30 2024-04-26T00:31:21.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff08d84b640 5 monclient: authenticate success, global_id 134137 2024-04-26T00:31:21.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff08d84b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:31:21.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff085ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:31:21.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff08d84b640 10 monclient: get_monmap_and_config success 2024-04-26T00:31:21.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff08d84b640 4 set_mon_vals no callback set 2024-04-26T00:31:21.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff085ffb640 1 -- 172.21.15.37:0/4135861781 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0700ab9d0 con 0x7ff088134b30 2024-04-26T00:31:21.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff085ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:21.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff085ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:31:21.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff085ffb640 10 monclient: dump: 2024-04-26T00:31:21.599 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:21.599 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:21.600 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:21.600 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:21.600 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:21.600 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:21.600 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:21.600 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:21.600 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:21.600 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:21.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff08d84b640 10 monclient: shutdown 2024-04-26T00:31:21.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff08d84b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:31:21.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff08d84b640 1 -- 172.21.15.37:0/4135861781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff088134b30 msgr2=0x7ff088134f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:21.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff08d84b640 1 --2- 172.21.15.37:0/4135861781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff088134b30 0x7ff088134f50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff08809f690 tx=0x7ff07001c480 comp rx=0 tx=0).stop 2024-04-26T00:31:21.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff08d84b640 1 -- 172.21.15.37:0/4135861781 shutdown_connections 2024-04-26T00:31:21.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff08d84b640 1 --2- 172.21.15.37:0/4135861781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff088134b30 0x7ff088134f50 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:21.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff08d84b640 1 --2- 172.21.15.37:0/4135861781 >> v2:172.21.15.37:3300/0 conn(0x7ff0880a0260 0x7ff0880a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:21.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff08d84b640 1 -- 172.21.15.37:0/4135861781 >> 172.21.15.37:0/4135861781 conn(0x7ff0880a1ad0 msgr2=0x7ff0880a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:31:21.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff08d84b640 1 -- 172.21.15.37:0/4135861781 shutdown_connections 2024-04-26T00:31:21.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff08d84b640 1 -- 172.21.15.37:0/4135861781 wait complete. 2024-04-26T00:31:21.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff08d84b640 10 monclient: build_initial_monmap 2024-04-26T00:31:21.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff08d84b640 10 monclient: monmap: 2024-04-26T00:31:21.602 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:31:21.602 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:31:21.602 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:31:21.602 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:31:21.602 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:31:21.602 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:31:21.602 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:31:21.602 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:31:21.602 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:31:21.602 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:21.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff08d84b640 1 Processor -- start 2024-04-26T00:31:21.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff08d84b640 1 -- start start 2024-04-26T00:31:21.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff08d84b640 10 monclient: init 2024-04-26T00:31:21.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff08d84b640 10 monclient: _reopen_session rank -1 2024-04-26T00:31:21.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff08d84b640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:31:21.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff08d84b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0880a0260 0x7ff08813bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:21.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff08d84b640 10 monclient(hunting): picked mon.noname-a con 0x7ff0880a0260 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:31:21.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff08d84b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff088134b30 0x7ff0881cd240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:21.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff08d84b640 10 monclient(hunting): picked mon.noname-c con 0x7ff088134b30 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:31:21.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff08d84b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff088135490 0x7ff088139800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:21.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff08d84b640 10 monclient(hunting): picked mon.noname-b con 0x7ff088135490 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:31:21.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff08d84b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:21.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff08d84b640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff08809f430 con 0x7ff0880a0260 2024-04-26T00:31:21.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff08d84b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:21.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff08d84b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff08809f2b0 con 0x7ff088135490 2024-04-26T00:31:21.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff08d84b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:21.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff08d84b640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff08809fe90 con 0x7ff088134b30 2024-04-26T00:31:21.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff08d84b640 10 monclient(hunting): _renew_subs 2024-04-26T00:31:21.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff08d84b640 10 monclient(hunting): authenticate will time out at 1662.910603s 2024-04-26T00:31:21.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff086ffd640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0880a0260 msgr2=0x7ff08813bf80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:31:21.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff086ffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0880a0260 0x7ff08813bf80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:31:21.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff0867fc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff088134b30 0x7ff0881cd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:21.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff0867fc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff088134b30 0x7ff0881cd240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:53104/0 (socket says 172.21.15.37:53104) 2024-04-26T00:31:21.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff0867fc640 1 -- 172.21.15.37:0/638643819 learned_addr learned my addr 172.21.15.37:0/638643819 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:31:21.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff0867fc640 10 monclient(hunting): get_auth_request con 0x7ff088134b30 auth_method 0 2024-04-26T00:31:21.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff0867fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:21.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff0867fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:21.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff0867fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:21.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff0877fe640 1 --2- 172.21.15.37:0/638643819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff088135490 0x7ff088139800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:21.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff0867fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:21.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff0867fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:21.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff0867fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:21.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff0877fe640 10 monclient(hunting): get_auth_request con 0x7ff088135490 auth_method 0 2024-04-26T00:31:21.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff0877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:21.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff0877fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:21.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff0877fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:21.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff0877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:21.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff0877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:21.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff0877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:21.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff0867fc640 10 monclient(hunting): handle_auth_done global_id 84608 payload 824 2024-04-26T00:31:21.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff0867fc640 1 -- 172.21.15.37:0/638643819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff088135490 msgr2=0x7ff088139800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:21.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff0867fc640 1 --2- 172.21.15.37:0/638643819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff088135490 0x7ff088139800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:21.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff0867fc640 1 -- 172.21.15.37:0/638643819 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0880a0260 msgr2=0x7ff08813bf80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:31:21.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff0867fc640 1 --2- 172.21.15.37:0/638643819 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0880a0260 0x7ff08813bf80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:21.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff0867fc640 10 monclient: _finish_hunting 0 2024-04-26T00:31:21.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff0867fc640 1 monclient: found mon.noname-c 2024-04-26T00:31:21.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff0867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:31:21.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff0867fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:31:21.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.600+0000 7ff0867fc640 1 -- 172.21.15.37:0/638643819 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0881348f0 con 0x7ff088134b30 2024-04-26T00:31:21.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.604+0000 7ff0867fc640 1 -- 172.21.15.37:0/638643819 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff088134b30 msgr2=0x7ff0881cd240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:21.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.604+0000 7ff0867fc640 1 --2- 172.21.15.37:0/638643819 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff088134b30 0x7ff0881cd240 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff0700ae040 tx=0x7ff070019220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:21.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.604+0000 7ff07ffff640 1 -- 172.21.15.37:0/638643819 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0700b0040 con 0x7ff088134b30 2024-04-26T00:31:21.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.604+0000 7ff07ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:21.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.604+0000 7ff07ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:31:21.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.604+0000 7ff07ffff640 10 monclient: dump: 2024-04-26T00:31:21.609 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:21.609 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:21.609 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:21.609 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:21.609 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:21.609 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:21.609 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:21.609 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:21.609 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:21.609 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:21.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.604+0000 7ff07ffff640 10 monclient: _finish_auth 0 2024-04-26T00:31:21.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.604+0000 7ff07ffff640 10 monclient: _check_auth_tickets 2024-04-26T00:31:21.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.604+0000 7ff07ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:31:21.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.604+0000 7ff07ffff640 1 -- 172.21.15.37:0/638643819 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0700b9070 con 0x7ff088134b30 2024-04-26T00:31:21.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.604+0000 7ff08d84b640 5 monclient: authenticate success, global_id 84608 2024-04-26T00:31:21.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.604+0000 7ff08d84b640 10 monclient: _renew_subs 2024-04-26T00:31:21.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.604+0000 7ff08d84b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:21.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.604+0000 7ff08d84b640 1 -- 172.21.15.37:0/638643819 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff08813b420 con 0x7ff088134b30 2024-04-26T00:31:21.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.604+0000 7ff07ffff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:31:21.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.604+0000 7ff07ffff640 1 -- 172.21.15.37:0/638643819 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0700b37d0 con 0x7ff088134b30 2024-04-26T00:31:21.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.604+0000 7ff07ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:21.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.604+0000 7ff07ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:31:21.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.604+0000 7ff07ffff640 10 monclient: dump: 2024-04-26T00:31:21.610 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:21.611 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:21.611 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:21.611 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:21.611 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:21.611 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:21.611 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:21.611 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:21.611 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:21.611 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:21.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.604+0000 7ff084ff9640 4 set_mon_vals no callback set 2024-04-26T00:31:21.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.604+0000 7ff08d84b640 10 monclient: _renew_subs 2024-04-26T00:31:21.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.604+0000 7ff08d84b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:21.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.604+0000 7ff08d84b640 1 -- 172.21.15.37:0/638643819 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff08811b800 con 0x7ff088134b30 2024-04-26T00:31:21.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.604+0000 7ff08d84b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:21.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.604+0000 7ff08d84b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:21.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.604+0000 7ff08d84b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:21.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.604+0000 7ff08d84b640 1 -- 172.21.15.37:0/638643819 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff088066f40 con 0x7ff088134b30 2024-04-26T00:31:21.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.604+0000 7ff07ffff640 1 -- 172.21.15.37:0/638643819 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff0700b3970 con 0x7ff088134b30 2024-04-26T00:31:21.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.604+0000 7ff07ffff640 1 --2- 172.21.15.37:0/638643819 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff068035e30 0x7ff0680382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:21.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.604+0000 7ff07ffff640 1 -- 172.21.15.37:0/638643819 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(434..434 src has 370..434) ==== 3931+0+0 (secure 0 0 0) 0x7ff0700e2930 con 0x7ff088134b30 2024-04-26T00:31:21.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.604+0000 7ff086ffd640 1 --2- 172.21.15.37:0/638643819 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff068035e30 0x7ff0680382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:21.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.604+0000 7ff086ffd640 10 monclient: get_auth_request con 0x7ff068035e30 auth_method 0 2024-04-26T00:31:21.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.608+0000 7ff086ffd640 1 --2- 172.21.15.37:0/638643819 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff068035e30 0x7ff0680382f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff080003d10 tx=0x7ff080009270 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:21.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.612+0000 7ff07ffff640 1 -- 172.21.15.37:0/638643819 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff070016190 con 0x7ff088134b30 2024-04-26T00:31:21.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.612+0000 7ff07ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:21.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.612+0000 7ff07ffff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:31:21.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.828+0000 7ff08d84b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:31:21.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.828+0000 7ff08d84b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:31:21.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.828+0000 7ff08d84b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:21.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.828+0000 7ff08d84b640 1 -- 172.21.15.37:0/638643819 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff08811ba00 con 0x7ff088134b30 2024-04-26T00:31:21.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.828+0000 7ff07ffff640 1 -- 172.21.15.37:0/638643819 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7ff0700a9e50 con 0x7ff088134b30 2024-04-26T00:31:21.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.828+0000 7ff07ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:31:21.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.828+0000 7ff07ffff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:31:21.822 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":86,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:31:21.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.828+0000 7ff08d84b640 1 -- 172.21.15.37:0/638643819 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff068035e30 msgr2=0x7ff0680382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:21.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.828+0000 7ff08d84b640 1 --2- 172.21.15.37:0/638643819 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff068035e30 0x7ff0680382f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff080003d10 tx=0x7ff080009270 comp rx=0 tx=0).stop 2024-04-26T00:31:21.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.828+0000 7ff08d84b640 10 monclient: shutdown 2024-04-26T00:31:21.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.828+0000 7ff08d84b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:31:21.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.828+0000 7ff08d84b640 1 -- 172.21.15.37:0/638643819 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff088134b30 msgr2=0x7ff0881cd240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:21.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.828+0000 7ff08d84b640 1 --2- 172.21.15.37:0/638643819 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff088134b30 0x7ff0881cd240 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff0700ae040 tx=0x7ff070019220 comp rx=0 tx=0).stop 2024-04-26T00:31:21.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.828+0000 7ff08d84b640 1 -- 172.21.15.37:0/638643819 shutdown_connections 2024-04-26T00:31:21.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.828+0000 7ff08d84b640 1 --2- 172.21.15.37:0/638643819 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff068035e30 0x7ff0680382f0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:21.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.828+0000 7ff08d84b640 1 --2- 172.21.15.37:0/638643819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff088135490 0x7ff088139800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:21.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.828+0000 7ff08d84b640 1 --2- 172.21.15.37:0/638643819 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff088134b30 0x7ff0881cd240 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:21.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.828+0000 7ff08d84b640 1 --2- 172.21.15.37:0/638643819 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0880a0260 0x7ff08813bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:21.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.828+0000 7ff08d84b640 1 -- 172.21.15.37:0/638643819 >> 172.21.15.37:0/638643819 conn(0x7ff0880a1ad0 msgr2=0x7ff0880a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:31:21.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.828+0000 7ff08d84b640 1 -- 172.21.15.37:0/638643819 shutdown_connections 2024-04-26T00:31:21.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:21.828+0000 7ff08d84b640 1 -- 172.21.15.37:0/638643819 wait complete. 2024-04-26T00:31:24.840 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:31:24.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.976+0000 7f19e9e1a640 10 monclient: get_monmap_and_config 2024-04-26T00:31:24.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.976+0000 7f19e9e1a640 10 monclient: build_initial_monmap 2024-04-26T00:31:24.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.976+0000 7f19e9e1a640 10 monclient: monmap: 2024-04-26T00:31:24.971 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:31:24.971 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:24.971 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:31:24.981641+0000 2024-04-26T00:31:24.971 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:31:24.981641+0000 2024-04-26T00:31:24.971 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:31:24.971 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:24.971 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:31:24.971 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:31:24.971 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:31:24.971 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:31:24.971 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:31:24.971 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:24.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.976+0000 7f19e9e1a640 1 Processor -- start 2024-04-26T00:31:24.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.976+0000 7f19e9e1a640 1 -- start start 2024-04-26T00:31:24.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.976+0000 7f19e9e1a640 10 monclient: init 2024-04-26T00:31:24.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.976+0000 7f19e9e1a640 10 monclient: _reopen_session rank -1 2024-04-26T00:31:24.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.976+0000 7f19e9e1a640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-04-26T00:31:24.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.976+0000 7f19e9e1a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f19e4136eb0 addr v1:172.21.15.45:6789/0 2024-04-26T00:31:24.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.976+0000 7f19e9e1a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19e41379a0 0x7f19e413c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:24.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.976+0000 7f19e9e1a640 10 monclient(hunting): picked mon.noname-b con 0x7f19e41379a0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:31:24.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.976+0000 7f19e9e1a640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f19e413c990 0x7f19e413ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:24.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.976+0000 7f19e9e1a640 10 monclient(hunting): picked mon.noname-a con 0x7f19e413c990 addr v2:172.21.15.37:3300/0 2024-04-26T00:31:24.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.976+0000 7f19e9e1a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19e409b8c0 con 0x7f19e4136eb0 2024-04-26T00:31:24.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.976+0000 7f19e9e1a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:24.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.976+0000 7f19e9e1a640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f19e409c7d0 con 0x7f19e413c990 2024-04-26T00:31:24.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.976+0000 7f19e9e1a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:24.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.976+0000 7f19e9e1a640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f19e409b300 con 0x7f19e41379a0 2024-04-26T00:31:24.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.976+0000 7f19e9e1a640 10 monclient(hunting): _renew_subs 2024-04-26T00:31:24.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.976+0000 7f19e9e1a640 10 monclient(hunting): authenticate will time out at 1666.287513s 2024-04-26T00:31:24.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.976+0000 7f19e3fff640 1 -- >> v2:172.21.15.37:3300/0 conn(0x7f19e413c990 msgr2=0x7f19e413ed80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:31:24.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.976+0000 7f19e3fff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f19e413c990 0x7f19e413ed80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:31:24.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.976+0000 7f19e2ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19e41379a0 0x7f19e413c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:24.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.976+0000 7f19e2ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19e41379a0 0x7f19e413c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:53116/0 (socket says 172.21.15.37:53116) 2024-04-26T00:31:24.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.976+0000 7f19e2ffd640 1 -- 172.21.15.37:0/335424663 learned_addr learned my addr 172.21.15.37:0/335424663 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:31:24.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.976+0000 7f19e2ffd640 10 monclient(hunting): get_auth_request con 0x7f19e41379a0 auth_method 0 2024-04-26T00:31:24.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.976+0000 7f19e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:24.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.976+0000 7f19e2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:24.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.976+0000 7f19e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:24.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.976+0000 7f19e2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:24.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.976+0000 7f19e2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:24.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.980+0000 7f19e2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:24.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.980+0000 7f19e2ffd640 10 monclient(hunting): handle_auth_done global_id 84611 payload 274 2024-04-26T00:31:24.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.980+0000 7f19e2ffd640 1 -- 172.21.15.37:0/335424663 >> v2:172.21.15.37:3300/0 conn(0x7f19e413c990 msgr2=0x7f19e413ed80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:31:24.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.980+0000 7f19e2ffd640 1 --2- 172.21.15.37:0/335424663 >> v2:172.21.15.37:3300/0 conn(0x7f19e413c990 0x7f19e413ed80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:24.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.980+0000 7f19e2ffd640 1 -- 172.21.15.37:0/335424663 >> v1:172.21.15.45:6789/0 conn(0x7f19e4136eb0 legacy=0x7f19e4137290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:31:24.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.980+0000 7f19e2ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:31:24.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.980+0000 7f19e2ffd640 1 monclient: found mon.noname-b 2024-04-26T00:31:24.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.980+0000 7f19e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:31:24.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.980+0000 7f19e2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:31:24.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.980+0000 7f19e2ffd640 1 -- 172.21.15.37:0/335424663 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19e409b5e0 con 0x7f19e41379a0 2024-04-26T00:31:24.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e2ffd640 1 -- 172.21.15.37:0/335424663 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19e41379a0 msgr2=0x7f19e413c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:24.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e2ffd640 1 --2- 172.21.15.37:0/335424663 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19e41379a0 0x7f19e413c450 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f19d8009870 tx=0x7f19d801c220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=822d2d7fbf59b66d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:24.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e27fc640 1 -- 172.21.15.37:0/335424663 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d80b0070 con 0x7f19e41379a0 2024-04-26T00:31:24.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:24.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:31:24.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e27fc640 10 monclient: dump: 2024-04-26T00:31:24.978 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:24.978 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:24.978 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:24.978 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:24.978 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:24.978 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:24.978 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:24.978 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:24.978 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:24.978 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:24.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e27fc640 10 monclient: _finish_auth 0 2024-04-26T00:31:24.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e27fc640 10 monclient: _check_auth_tickets 2024-04-26T00:31:24.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:31:24.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e27fc640 1 -- 172.21.15.37:0/335424663 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19d80ab600 con 0x7f19e41379a0 2024-04-26T00:31:24.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e9e1a640 5 monclient: authenticate success, global_id 84611 2024-04-26T00:31:24.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e9e1a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:31:24.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e27fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:31:24.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e27fc640 1 -- 172.21.15.37:0/335424663 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d80ab9d0 con 0x7f19e41379a0 2024-04-26T00:31:24.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e9e1a640 10 monclient: get_monmap_and_config success 2024-04-26T00:31:24.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e9e1a640 4 set_mon_vals no callback set 2024-04-26T00:31:24.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:24.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:31:24.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e27fc640 10 monclient: dump: 2024-04-26T00:31:24.979 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:24.979 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:24.979 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:24.980 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:24.980 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:24.980 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:24.980 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:24.980 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:24.980 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:24.980 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:24.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e9e1a640 10 monclient: shutdown 2024-04-26T00:31:24.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e9e1a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:31:24.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e9e1a640 1 -- 172.21.15.37:0/335424663 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19e41379a0 msgr2=0x7f19e413c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:24.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e9e1a640 1 --2- 172.21.15.37:0/335424663 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19e41379a0 0x7f19e413c450 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f19d8009870 tx=0x7f19d801c220 comp rx=0 tx=0).stop 2024-04-26T00:31:24.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e9e1a640 1 -- 172.21.15.37:0/335424663 shutdown_connections 2024-04-26T00:31:24.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e9e1a640 1 --2- 172.21.15.37:0/335424663 >> v2:172.21.15.37:3300/0 conn(0x7f19e413c990 0x7f19e413ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:24.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e9e1a640 1 --2- 172.21.15.37:0/335424663 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19e41379a0 0x7f19e413c450 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:24.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e9e1a640 1 -- 172.21.15.37:0/335424663 >> 172.21.15.37:0/335424663 conn(0x7f19e40a6300 msgr2=0x7f19e40a6710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:31:24.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e9e1a640 1 -- 172.21.15.37:0/335424663 shutdown_connections 2024-04-26T00:31:24.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e9e1a640 1 -- 172.21.15.37:0/335424663 wait complete. 2024-04-26T00:31:24.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e9e1a640 10 monclient: build_initial_monmap 2024-04-26T00:31:24.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e9e1a640 10 monclient: monmap: 2024-04-26T00:31:24.986 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:31:24.986 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:31:24.986 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:31:24.986 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:31:24.986 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:31:24.987 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:31:24.987 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:31:24.987 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:31:24.987 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:31:24.987 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:24.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e9e1a640 1 Processor -- start 2024-04-26T00:31:24.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e9e1a640 1 -- start start 2024-04-26T00:31:24.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e9e1a640 10 monclient: init 2024-04-26T00:31:24.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e9e1a640 10 monclient: _reopen_session rank -1 2024-04-26T00:31:24.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e9e1a640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:31:24.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e9e1a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19e4136eb0 0x7f19e413b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:24.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e9e1a640 10 monclient(hunting): picked mon.noname-a con 0x7f19e4136eb0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:31:24.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e9e1a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19e41379a0 0x7f19e4134050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:24.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e9e1a640 10 monclient(hunting): picked mon.noname-c con 0x7f19e41379a0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:31:24.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e9e1a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19e413c990 0x7f19e4134590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:24.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e9e1a640 10 monclient(hunting): picked mon.noname-b con 0x7f19e413c990 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:31:24.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e9e1a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:24.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e9e1a640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f19e409abf0 con 0x7f19e4136eb0 2024-04-26T00:31:24.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e9e1a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:24.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e9e1a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f19e4141ae0 con 0x7f19e413c990 2024-04-26T00:31:24.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e9e1a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:24.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e9e1a640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f19e409fcc0 con 0x7f19e41379a0 2024-04-26T00:31:24.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e9e1a640 10 monclient(hunting): _renew_subs 2024-04-26T00:31:24.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e9e1a640 10 monclient(hunting): authenticate will time out at 1666.295281s 2024-04-26T00:31:24.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e37fe640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19e4136eb0 msgr2=0x7f19e413b840 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:31:24.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e37fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19e4136eb0 0x7f19e413b840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:31:24.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e2ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19e41379a0 0x7f19e4134050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:24.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e2ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19e41379a0 0x7f19e4134050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:53120/0 (socket says 172.21.15.37:53120) 2024-04-26T00:31:24.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e2ffd640 1 -- 172.21.15.37:0/52921531 learned_addr learned my addr 172.21.15.37:0/52921531 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:31:24.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e2ffd640 10 monclient(hunting): get_auth_request con 0x7f19e41379a0 auth_method 0 2024-04-26T00:31:24.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:24.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:24.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:24.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:24.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:24.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:24.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e3fff640 1 --2- 172.21.15.37:0/52921531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19e413c990 0x7f19e4134590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:24.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e3fff640 10 monclient(hunting): get_auth_request con 0x7f19e413c990 auth_method 0 2024-04-26T00:31:24.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:24.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:24.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:24.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:24.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:24.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:24.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e2ffd640 10 monclient(hunting): handle_auth_done global_id 84614 payload 824 2024-04-26T00:31:24.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e2ffd640 1 -- 172.21.15.37:0/52921531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19e413c990 msgr2=0x7f19e4134590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:24.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e2ffd640 1 --2- 172.21.15.37:0/52921531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19e413c990 0x7f19e4134590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:24.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e2ffd640 1 -- 172.21.15.37:0/52921531 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19e4136eb0 msgr2=0x7f19e413b840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:31:24.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e2ffd640 1 --2- 172.21.15.37:0/52921531 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19e4136eb0 0x7f19e413b840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:24.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e2ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:31:24.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e2ffd640 1 monclient: found mon.noname-c 2024-04-26T00:31:24.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:31:24.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e2ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:31:24.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.984+0000 7f19e2ffd640 1 -- 172.21.15.37:0/52921531 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19e409ae10 con 0x7f19e41379a0 2024-04-26T00:31:24.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.988+0000 7f19e2ffd640 1 -- 172.21.15.37:0/52921531 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19e41379a0 msgr2=0x7f19e4134050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:24.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.988+0000 7f19e2ffd640 1 --2- 172.21.15.37:0/52921531 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19e41379a0 0x7f19e4134050 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f19d801a8d0 tx=0x7f19d8018fc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:24.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.988+0000 7f19cbfff640 1 -- 172.21.15.37:0/52921531 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d80b0040 con 0x7f19e41379a0 2024-04-26T00:31:24.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.988+0000 7f19cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:24.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.988+0000 7f19cbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:31:24.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.988+0000 7f19cbfff640 10 monclient: dump: 2024-04-26T00:31:24.993 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:24.994 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:24.994 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:24.994 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:24.994 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:24.994 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:24.994 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:24.994 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:24.994 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:24.994 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:24.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.988+0000 7f19cbfff640 10 monclient: _finish_auth 0 2024-04-26T00:31:24.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.988+0000 7f19cbfff640 10 monclient: _check_auth_tickets 2024-04-26T00:31:24.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.988+0000 7f19cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:31:24.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.988+0000 7f19e9e1a640 5 monclient: authenticate success, global_id 84614 2024-04-26T00:31:24.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.988+0000 7f19e9e1a640 10 monclient: _renew_subs 2024-04-26T00:31:24.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.988+0000 7f19e9e1a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:24.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.988+0000 7f19cbfff640 1 -- 172.21.15.37:0/52921531 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19d80b9070 con 0x7f19e41379a0 2024-04-26T00:31:24.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.988+0000 7f19e9e1a640 1 -- 172.21.15.37:0/52921531 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19e40a8c50 con 0x7f19e41379a0 2024-04-26T00:31:24.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.988+0000 7f19cbfff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:31:24.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.988+0000 7f19cbfff640 1 -- 172.21.15.37:0/52921531 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d80b37b0 con 0x7f19e41379a0 2024-04-26T00:31:24.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.988+0000 7f19cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:24.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.988+0000 7f19cbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:31:24.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.988+0000 7f19cbfff640 10 monclient: dump: 2024-04-26T00:31:24.995 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:24.995 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:24.995 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:24.995 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:24.995 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:24.996 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:24.996 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:24.996 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:24.996 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:24.996 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:24.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.988+0000 7f19e9e1a640 10 monclient: _renew_subs 2024-04-26T00:31:24.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.988+0000 7f19e9e1a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:24.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.988+0000 7f19e9e1a640 1 -- 172.21.15.37:0/52921531 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f19e413a5a0 con 0x7f19e41379a0 2024-04-26T00:31:24.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.988+0000 7f19e17fa640 4 set_mon_vals no callback set 2024-04-26T00:31:24.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.988+0000 7f19e9e1a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:24.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.988+0000 7f19e9e1a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:24.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.988+0000 7f19e9e1a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:24.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.988+0000 7f19e9e1a640 1 -- 172.21.15.37:0/52921531 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19e4066f40 con 0x7f19e41379a0 2024-04-26T00:31:24.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.988+0000 7f19cbfff640 1 -- 172.21.15.37:0/52921531 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f19d80b3990 con 0x7f19e41379a0 2024-04-26T00:31:24.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.988+0000 7f19cbfff640 1 --2- 172.21.15.37:0/52921531 >> v2:172.21.15.45:6812/2379918623 conn(0x7f19d0035e60 0x7f19d0038320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:24.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.988+0000 7f19cbfff640 1 -- 172.21.15.37:0/52921531 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(434..434 src has 370..434) ==== 3931+0+0 (secure 0 0 0) 0x7f19d80e26d0 con 0x7f19e41379a0 2024-04-26T00:31:24.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.992+0000 7f19e37fe640 1 --2- 172.21.15.37:0/52921531 >> v2:172.21.15.45:6812/2379918623 conn(0x7f19d0035e60 0x7f19d0038320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:24.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.992+0000 7f19e37fe640 10 monclient: get_auth_request con 0x7f19d0035e60 auth_method 0 2024-04-26T00:31:24.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.992+0000 7f19e37fe640 1 --2- 172.21.15.37:0/52921531 >> v2:172.21.15.45:6812/2379918623 conn(0x7f19d0035e60 0x7f19d0038320 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f19e409cf50 tx=0x7f19d4007a20 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:24.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.996+0000 7f19cbfff640 1 -- 172.21.15.37:0/52921531 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f19e409b8c0 con 0x7f19e41379a0 2024-04-26T00:31:24.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.996+0000 7f19cbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:24.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:24.996+0000 7f19cbfff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:31:25.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:25.208+0000 7f19e9e1a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:31:25.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:25.208+0000 7f19e9e1a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:31:25.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:25.208+0000 7f19e9e1a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:25.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:25.208+0000 7f19e9e1a640 1 -- 172.21.15.37:0/52921531 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f19e4075e70 con 0x7f19e41379a0 2024-04-26T00:31:25.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:25.212+0000 7f19cbfff640 1 -- 172.21.15.37:0/52921531 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f19d80aa3a0 con 0x7f19e41379a0 2024-04-26T00:31:25.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:25.212+0000 7f19cbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:31:25.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:25.212+0000 7f19cbfff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:31:25.206 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":86,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:31:25.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:25.212+0000 7f19e9e1a640 1 -- 172.21.15.37:0/52921531 >> v2:172.21.15.45:6812/2379918623 conn(0x7f19d0035e60 msgr2=0x7f19d0038320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:25.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:25.212+0000 7f19e9e1a640 1 --2- 172.21.15.37:0/52921531 >> v2:172.21.15.45:6812/2379918623 conn(0x7f19d0035e60 0x7f19d0038320 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f19e409cf50 tx=0x7f19d4007a20 comp rx=0 tx=0).stop 2024-04-26T00:31:25.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:25.212+0000 7f19e9e1a640 10 monclient: shutdown 2024-04-26T00:31:25.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:25.212+0000 7f19e9e1a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:31:25.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:25.212+0000 7f19e9e1a640 1 -- 172.21.15.37:0/52921531 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19e41379a0 msgr2=0x7f19e4134050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:25.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:25.212+0000 7f19e9e1a640 1 --2- 172.21.15.37:0/52921531 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19e41379a0 0x7f19e4134050 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f19d801a8d0 tx=0x7f19d8018fc0 comp rx=0 tx=0).stop 2024-04-26T00:31:25.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:25.212+0000 7f19e9e1a640 1 -- 172.21.15.37:0/52921531 shutdown_connections 2024-04-26T00:31:25.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:25.212+0000 7f19e9e1a640 1 --2- 172.21.15.37:0/52921531 >> v2:172.21.15.45:6812/2379918623 conn(0x7f19d0035e60 0x7f19d0038320 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:25.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:25.212+0000 7f19e9e1a640 1 --2- 172.21.15.37:0/52921531 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f19e413c990 0x7f19e4134590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:25.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:25.212+0000 7f19e9e1a640 1 --2- 172.21.15.37:0/52921531 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19e41379a0 0x7f19e4134050 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:25.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:25.212+0000 7f19e9e1a640 1 --2- 172.21.15.37:0/52921531 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19e4136eb0 0x7f19e413b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:25.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:25.212+0000 7f19e9e1a640 1 -- 172.21.15.37:0/52921531 >> 172.21.15.37:0/52921531 conn(0x7f19e40a6300 msgr2=0x7f19e4136600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:31:25.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:25.212+0000 7f19e9e1a640 1 -- 172.21.15.37:0/52921531 shutdown_connections 2024-04-26T00:31:25.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:25.212+0000 7f19e9e1a640 1 -- 172.21.15.37:0/52921531 wait complete. 2024-04-26T00:31:28.223 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:31:28.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.364+0000 7f0649b5f640 10 monclient: get_monmap_and_config 2024-04-26T00:31:28.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.364+0000 7f0649b5f640 10 monclient: build_initial_monmap 2024-04-26T00:31:28.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.364+0000 7f0649b5f640 10 monclient: monmap: 2024-04-26T00:31:28.359 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:31:28.359 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:28.359 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:31:28.369073+0000 2024-04-26T00:31:28.359 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:31:28.369073+0000 2024-04-26T00:31:28.359 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:31:28.359 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:28.359 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:31:28.360 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:31:28.360 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:31:28.360 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:31:28.360 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:31:28.360 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:28.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.364+0000 7f0649b5f640 1 Processor -- start 2024-04-26T00:31:28.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.364+0000 7f0649b5f640 1 -- start start 2024-04-26T00:31:28.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.364+0000 7f0649b5f640 10 monclient: init 2024-04-26T00:31:28.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.364+0000 7f0649b5f640 10 monclient: _reopen_session rank -1 2024-04-26T00:31:28.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.364+0000 7f0649b5f640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-04-26T00:31:28.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.364+0000 7f0649b5f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0644136da0 addr v1:172.21.15.37:6789/0 2024-04-26T00:31:28.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.364+0000 7f0649b5f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0644137180 addr v1:172.21.15.45:6789/0 2024-04-26T00:31:28.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.364+0000 7f0649b5f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f064413d2e0 0x7f064413f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:28.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.364+0000 7f0649b5f640 10 monclient(hunting): picked mon.noname-b con 0x7f064413d2e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:31:28.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.364+0000 7f0649b5f640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f064409b4f0 con 0x7f0644136da0 2024-04-26T00:31:28.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.364+0000 7f0649b5f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f064409b7d0 con 0x7f0644137180 2024-04-26T00:31:28.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.364+0000 7f0649b5f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:28.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.364+0000 7f0649b5f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f064409bb40 con 0x7f064413d2e0 2024-04-26T00:31:28.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.364+0000 7f0649b5f640 10 monclient(hunting): _renew_subs 2024-04-26T00:31:28.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.364+0000 7f0649b5f640 10 monclient(hunting): authenticate will time out at 1669.674746s 2024-04-26T00:31:28.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.364+0000 7f06437fe640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f0644136da0 legacy=0x7f064409bed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:31:28.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.364+0000 7f0643fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f064413d2e0 0x7f064413f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:28.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.364+0000 7f0643fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f064413d2e0 0x7f064413f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:53126/0 (socket says 172.21.15.37:53126) 2024-04-26T00:31:28.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.364+0000 7f0643fff640 1 -- 172.21.15.37:0/1691023345 learned_addr learned my addr 172.21.15.37:0/1691023345 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:31:28.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.364+0000 7f0643fff640 10 monclient(hunting): get_auth_request con 0x7f064413d2e0 auth_method 0 2024-04-26T00:31:28.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.364+0000 7f0643fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:28.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.364+0000 7f0643fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:28.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.364+0000 7f0643fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:28.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.364+0000 7f0643fff640 1 -- 172.21.15.37:0/1691023345 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f064413d2e0 msgr2=0x7f064413f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:31:28.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.364+0000 7f0643fff640 1 -- 172.21.15.37:0/1691023345 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f064413d2e0 msgr2=0x7f064413f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:31:28.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.364+0000 7f0643fff640 1 --2- 172.21.15.37:0/1691023345 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f064413d2e0 0x7f064413f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:31:28.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.364+0000 7f0643fff640 1 --2- 172.21.15.37:0/1691023345 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f064413d2e0 0x7f064413f6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:31:28.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.368+0000 7f0642ffd640 1 -- 172.21.15.37:0/1691023345 >> v1:172.21.15.45:6789/0 conn(0x7f0644137180 legacy=0x7f06441375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:28.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.568+0000 7f06437fe640 1 -- 172.21.15.37:0/1691023345 >> v1:172.21.15.37:6789/0 conn(0x7f0644136da0 legacy=0x7f064409bed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:31:28.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.568+0000 7f0643fff640 1 --2- 172.21.15.37:0/1691023345 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f064413d2e0 0x7f064413f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:28.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.568+0000 7f0643fff640 10 monclient(hunting): get_auth_request con 0x7f064413d2e0 auth_method 0 2024-04-26T00:31:28.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.568+0000 7f0643fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:28.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.568+0000 7f0643fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:28.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.568+0000 7f0643fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:31:28.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.568+0000 7f0643fff640 1 -- 172.21.15.37:0/1691023345 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f064413d2e0 msgr2=0x7f064413f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:31:28.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.568+0000 7f0643fff640 1 -- 172.21.15.37:0/1691023345 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f064413d2e0 msgr2=0x7f064413f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:31:28.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.568+0000 7f0643fff640 1 --2- 172.21.15.37:0/1691023345 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f064413d2e0 0x7f064413f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:31:28.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.568+0000 7f0643fff640 1 --2- 172.21.15.37:0/1691023345 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f064413d2e0 0x7f064413f6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:31:28.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.972+0000 7f06437fe640 1 -- 172.21.15.37:0/1691023345 >> v1:172.21.15.37:6789/0 conn(0x7f0644136da0 legacy=0x7f064409bed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:31:28.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.972+0000 7f0643fff640 1 --2- 172.21.15.37:0/1691023345 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f064413d2e0 0x7f064413f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:28.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.972+0000 7f0643fff640 10 monclient(hunting): get_auth_request con 0x7f064413d2e0 auth_method 0 2024-04-26T00:31:28.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.972+0000 7f0643fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:28.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.972+0000 7f0643fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:28.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.972+0000 7f0643fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:31:28.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.972+0000 7f0643fff640 1 -- 172.21.15.37:0/1691023345 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f064413d2e0 msgr2=0x7f064413f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:31:28.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.972+0000 7f0643fff640 1 -- 172.21.15.37:0/1691023345 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f064413d2e0 msgr2=0x7f064413f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:31:28.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.972+0000 7f0643fff640 1 --2- 172.21.15.37:0/1691023345 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f064413d2e0 0x7f064413f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:31:28.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:28.972+0000 7f0643fff640 1 --2- 172.21.15.37:0/1691023345 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f064413d2e0 0x7f064413f6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:31:29.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:29.776+0000 7f06437fe640 1 -- 172.21.15.37:0/1691023345 >> v1:172.21.15.37:6789/0 conn(0x7f0644136da0 legacy=0x7f064409bed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:31:29.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:29.776+0000 7f0643fff640 1 --2- 172.21.15.37:0/1691023345 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f064413d2e0 0x7f064413f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:29.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:29.776+0000 7f0643fff640 10 monclient(hunting): get_auth_request con 0x7f064413d2e0 auth_method 0 2024-04-26T00:31:29.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:29.776+0000 7f0643fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:29.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:29.776+0000 7f0643fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:29.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:29.776+0000 7f0643fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:31:29.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:29.776+0000 7f0643fff640 1 -- 172.21.15.37:0/1691023345 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f064413d2e0 msgr2=0x7f064413f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:31:29.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:29.776+0000 7f0643fff640 1 -- 172.21.15.37:0/1691023345 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f064413d2e0 msgr2=0x7f064413f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:31:29.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:29.776+0000 7f0643fff640 1 --2- 172.21.15.37:0/1691023345 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f064413d2e0 0x7f064413f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:31:29.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:29.776+0000 7f0643fff640 1 --2- 172.21.15.37:0/1691023345 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f064413d2e0 0x7f064413f6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:31:31.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.368+0000 7f06417fa640 10 monclient(hunting): tick 2024-04-26T00:31:31.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.368+0000 7f06417fa640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:31:31.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.368+0000 7f06417fa640 1 monclient(hunting): continuing hunt 2024-04-26T00:31:31.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.368+0000 7f06417fa640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:31:31.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.368+0000 7f06417fa640 1 -- 172.21.15.37:0/1691023345 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f064413d2e0 msgr2=0x7f064413f6d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:31:31.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.368+0000 7f06417fa640 1 --2- 172.21.15.37:0/1691023345 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f064413d2e0 0x7f064413f6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:31.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.368+0000 7f06417fa640 1 -- 172.21.15.37:0/1691023345 >> v1:172.21.15.45:6789/0 conn(0x7f0644137180 legacy=0x7f06441375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:31.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.368+0000 7f06417fa640 1 -- 172.21.15.37:0/1691023345 >> v1:172.21.15.37:6789/0 conn(0x7f0644136da0 legacy=0x7f064409bed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:31:31.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.368+0000 7f06417fa640 10 monclient: _add_conns ranks=[4,3] 2024-04-26T00:31:31.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.368+0000 7f06417fa640 1 --2- 172.21.15.37:0/1691023345 >> v2:172.21.15.45:3300/0 conn(0x7f0624003780 0x7f0624005be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:31.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.368+0000 7f06417fa640 10 monclient(hunting): picked mon.noname-c con 0x7f0624003780 addr v2:172.21.15.45:3300/0 2024-04-26T00:31:31.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.368+0000 7f06417fa640 1 --2- 172.21.15.37:0/1691023345 >> v2:172.21.15.37:3300/0 conn(0x7f06240062d0 0x7f0624008770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:31.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.368+0000 7f06417fa640 10 monclient(hunting): picked mon.noname-a con 0x7f06240062d0 addr v2:172.21.15.37:3300/0 2024-04-26T00:31:31.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.368+0000 7f06417fa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:31.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.368+0000 7f06417fa640 1 -- 172.21.15.37:0/1691023345 --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f064409bb40 con 0x7f06240062d0 2024-04-26T00:31:31.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.368+0000 7f06417fa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:31.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.368+0000 7f06417fa640 1 -- 172.21.15.37:0/1691023345 --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f0624008e30 con 0x7f0624003780 2024-04-26T00:31:31.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.368+0000 7f06417fa640 10 monclient(hunting): _renew_subs 2024-04-26T00:31:31.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.368+0000 7f0642ffd640 1 -- 172.21.15.37:0/1691023345 >> v2:172.21.15.37:3300/0 conn(0x7f06240062d0 msgr2=0x7f0624008770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:31:31.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.368+0000 7f0642ffd640 1 --2- 172.21.15.37:0/1691023345 >> v2:172.21.15.37:3300/0 conn(0x7f06240062d0 0x7f0624008770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:31:31.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.368+0000 7f06437fe640 1 --2- 172.21.15.37:0/1691023345 >> v2:172.21.15.45:3300/0 conn(0x7f0624003780 0x7f0624005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:31.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.368+0000 7f06437fe640 10 monclient(hunting): get_auth_request con 0x7f0624003780 auth_method 0 2024-04-26T00:31:31.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.368+0000 7f06437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:31.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.368+0000 7f06437fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:31.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.368+0000 7f06437fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:31.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.368+0000 7f06437fe640 1 -- 172.21.15.37:0/1691023345 >> v2:172.21.15.45:3300/0 conn(0x7f0624003780 msgr2=0x7f0624005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:31:31.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.368+0000 7f06437fe640 1 -- 172.21.15.37:0/1691023345 >> v2:172.21.15.45:3300/0 conn(0x7f0624003780 msgr2=0x7f0624005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:31:31.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.368+0000 7f06437fe640 1 --2- 172.21.15.37:0/1691023345 >> v2:172.21.15.45:3300/0 conn(0x7f0624003780 0x7f0624005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:31:31.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.368+0000 7f06437fe640 1 --2- 172.21.15.37:0/1691023345 >> v2:172.21.15.45:3300/0 conn(0x7f0624003780 0x7f0624005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:31:31.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.572+0000 7f0642ffd640 1 -- 172.21.15.37:0/1691023345 >> v2:172.21.15.37:3300/0 conn(0x7f06240062d0 msgr2=0x7f0624008770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:31:31.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.572+0000 7f0642ffd640 1 --2- 172.21.15.37:0/1691023345 >> v2:172.21.15.37:3300/0 conn(0x7f06240062d0 0x7f0624008770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:31:31.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.572+0000 7f06437fe640 1 --2- 172.21.15.37:0/1691023345 >> v2:172.21.15.45:3300/0 conn(0x7f0624003780 0x7f0624005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:31.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.572+0000 7f06437fe640 10 monclient(hunting): get_auth_request con 0x7f0624003780 auth_method 0 2024-04-26T00:31:31.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.572+0000 7f06437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:31.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.572+0000 7f06437fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:31.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.572+0000 7f06437fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:31:31.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.572+0000 7f06437fe640 1 -- 172.21.15.37:0/1691023345 >> v2:172.21.15.45:3300/0 conn(0x7f0624003780 msgr2=0x7f0624005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:31:31.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.572+0000 7f06437fe640 1 -- 172.21.15.37:0/1691023345 >> v2:172.21.15.45:3300/0 conn(0x7f0624003780 msgr2=0x7f0624005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:31:31.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.572+0000 7f06437fe640 1 --2- 172.21.15.37:0/1691023345 >> v2:172.21.15.45:3300/0 conn(0x7f0624003780 0x7f0624005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:31:31.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.572+0000 7f06437fe640 1 --2- 172.21.15.37:0/1691023345 >> v2:172.21.15.45:3300/0 conn(0x7f0624003780 0x7f0624005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:31:31.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.976+0000 7f0642ffd640 1 -- 172.21.15.37:0/1691023345 >> v2:172.21.15.37:3300/0 conn(0x7f06240062d0 msgr2=0x7f0624008770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:31:31.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.976+0000 7f0642ffd640 1 --2- 172.21.15.37:0/1691023345 >> v2:172.21.15.37:3300/0 conn(0x7f06240062d0 0x7f0624008770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:31:31.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.976+0000 7f06437fe640 1 --2- 172.21.15.37:0/1691023345 >> v2:172.21.15.45:3300/0 conn(0x7f0624003780 0x7f0624005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:31.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.976+0000 7f06437fe640 10 monclient(hunting): get_auth_request con 0x7f0624003780 auth_method 0 2024-04-26T00:31:31.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.976+0000 7f06437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:31.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.976+0000 7f06437fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:31.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.976+0000 7f06437fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:31:31.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.976+0000 7f06437fe640 1 -- 172.21.15.37:0/1691023345 >> v2:172.21.15.45:3300/0 conn(0x7f0624003780 msgr2=0x7f0624005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:31:31.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.976+0000 7f06437fe640 1 -- 172.21.15.37:0/1691023345 >> v2:172.21.15.45:3300/0 conn(0x7f0624003780 msgr2=0x7f0624005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:31:31.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.976+0000 7f06437fe640 1 --2- 172.21.15.37:0/1691023345 >> v2:172.21.15.45:3300/0 conn(0x7f0624003780 0x7f0624005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:31:31.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:31.976+0000 7f06437fe640 1 --2- 172.21.15.37:0/1691023345 >> v2:172.21.15.45:3300/0 conn(0x7f0624003780 0x7f0624005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:31:32.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.780+0000 7f0642ffd640 1 -- 172.21.15.37:0/1691023345 >> v2:172.21.15.37:3300/0 conn(0x7f06240062d0 msgr2=0x7f0624008770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:31:32.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.780+0000 7f0642ffd640 1 --2- 172.21.15.37:0/1691023345 >> v2:172.21.15.37:3300/0 conn(0x7f06240062d0 0x7f0624008770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:31:32.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.780+0000 7f06427fc640 1 -- 172.21.15.37:0/1691023345 DispatchQueue::entry inject delay of 0.002000 2024-04-26T00:31:32.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.780+0000 7f06437fe640 1 --2- 172.21.15.37:0/1691023345 >> v2:172.21.15.45:3300/0 conn(0x7f0624003780 0x7f0624005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:32.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.780+0000 7f06437fe640 10 monclient(hunting): get_auth_request con 0x7f0624003780 auth_method 0 2024-04-26T00:31:32.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.780+0000 7f06437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:32.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.780+0000 7f06437fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:32.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.780+0000 7f06437fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:31:32.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.780+0000 7f06437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:32.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.780+0000 7f06437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:32.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.780+0000 7f06437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:32.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.780+0000 7f06437fe640 10 monclient(hunting): handle_auth_done global_id 134146 payload 274 2024-04-26T00:31:32.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.780+0000 7f06437fe640 1 -- 172.21.15.37:0/1691023345 >> v2:172.21.15.37:3300/0 conn(0x7f06240062d0 msgr2=0x7f0624008770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:31:32.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.780+0000 7f06437fe640 1 --2- 172.21.15.37:0/1691023345 >> v2:172.21.15.37:3300/0 conn(0x7f06240062d0 0x7f0624008770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:32.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.780+0000 7f06437fe640 10 monclient: _finish_hunting 0 2024-04-26T00:31:32.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.780+0000 7f06437fe640 1 monclient: found mon.noname-c 2024-04-26T00:31:32.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.780+0000 7f06437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:31:32.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.780+0000 7f06437fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:31:32.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.780+0000 7f06437fe640 1 -- 172.21.15.37:0/1691023345 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f064409c5e0 con 0x7f0624003780 2024-04-26T00:31:32.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.784+0000 7f06437fe640 1 -- 172.21.15.37:0/1691023345 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0624003780 msgr2=0x7f0624005be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:32.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.784+0000 7f06437fe640 1 --2- 172.21.15.37:0/1691023345 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0624003780 0x7f0624005be0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0624003710 tx=0x7f063c01d180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da6574500bdbffa2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:32.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.784+0000 7f06427fc640 1 -- 172.21.15.37:0/1691023345 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f063c0b3070 con 0x7f0624003780 2024-04-26T00:31:32.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.784+0000 7f06427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:32.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.784+0000 7f06427fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:31:32.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.784+0000 7f06427fc640 10 monclient: dump: 2024-04-26T00:31:32.781 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:32.781 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:32.781 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:32.781 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:32.781 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:32.781 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:32.781 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:32.781 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:32.782 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:32.782 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:32.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.784+0000 7f06427fc640 10 monclient: _finish_auth 0 2024-04-26T00:31:32.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.784+0000 7f06427fc640 10 monclient: _check_auth_tickets 2024-04-26T00:31:32.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.784+0000 7f06427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:31:32.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.784+0000 7f06427fc640 1 -- 172.21.15.37:0/1691023345 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f063c0ae410 con 0x7f0624003780 2024-04-26T00:31:32.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.784+0000 7f06427fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:31:32.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.784+0000 7f06427fc640 1 -- 172.21.15.37:0/1691023345 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f063c0ae800 con 0x7f0624003780 2024-04-26T00:31:32.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.784+0000 7f06427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:32.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.784+0000 7f06427fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:31:32.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.784+0000 7f06427fc640 10 monclient: dump: 2024-04-26T00:31:32.782 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:32.782 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:32.783 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:32.783 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:32.783 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:32.783 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:32.783 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:32.783 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:32.783 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:32.783 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:32.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.784+0000 7f0649b5f640 5 monclient: authenticate success, global_id 134146 2024-04-26T00:31:32.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.784+0000 7f0649b5f640 10 monclient: get_monmap_and_config success 2024-04-26T00:31:32.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.784+0000 7f0649b5f640 4 set_mon_vals no callback set 2024-04-26T00:31:32.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.788+0000 7f0649b5f640 10 monclient: shutdown 2024-04-26T00:31:32.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.788+0000 7f0649b5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:31:32.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.788+0000 7f0649b5f640 1 -- 172.21.15.37:0/1691023345 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0624003780 msgr2=0x7f0624005be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:32.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.788+0000 7f0649b5f640 1 --2- 172.21.15.37:0/1691023345 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0624003780 0x7f0624005be0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0624003710 tx=0x7f063c01d180 comp rx=0 tx=0).stop 2024-04-26T00:31:32.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.788+0000 7f0643fff640 1 -- 172.21.15.37:0/1691023345 reap_dead start 2024-04-26T00:31:32.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.788+0000 7f0649b5f640 1 -- 172.21.15.37:0/1691023345 shutdown_connections 2024-04-26T00:31:32.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.788+0000 7f0649b5f640 1 --2- 172.21.15.37:0/1691023345 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0624003780 0x7f0624005be0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:32.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.788+0000 7f0649b5f640 1 -- 172.21.15.37:0/1691023345 >> 172.21.15.37:0/1691023345 conn(0x7f06440a5f00 msgr2=0x7f06440a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:31:32.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.788+0000 7f0649b5f640 1 -- 172.21.15.37:0/1691023345 shutdown_connections 2024-04-26T00:31:32.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.788+0000 7f0649b5f640 1 -- 172.21.15.37:0/1691023345 wait complete. 2024-04-26T00:31:32.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.788+0000 7f0649b5f640 10 monclient: build_initial_monmap 2024-04-26T00:31:32.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.788+0000 7f0649b5f640 10 monclient: monmap: 2024-04-26T00:31:32.786 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:31:32.786 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:31:32.786 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:31:32.786 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:31:32.786 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:31:32.786 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:31:32.786 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:31:32.786 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:31:32.786 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:31:32.786 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:32.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.788+0000 7f0649b5f640 1 Processor -- start 2024-04-26T00:31:32.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.788+0000 7f0649b5f640 1 -- start start 2024-04-26T00:31:32.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.788+0000 7f0649b5f640 10 monclient: init 2024-04-26T00:31:32.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f0649b5f640 10 monclient: _reopen_session rank -1 2024-04-26T00:31:32.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f0649b5f640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:31:32.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f0649b5f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0624003780 0x7f06441353e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:32.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f0649b5f640 10 monclient(hunting): picked mon.noname-b con 0x7f0624003780 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:31:32.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f0649b5f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0644135920 0x7f06441387c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:32.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f0649b5f640 10 monclient(hunting): picked mon.noname-c con 0x7f0644135920 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:31:32.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f0649b5f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0644135d60 0x7f0644138d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:32.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f0649b5f640 10 monclient(hunting): picked mon.noname-a con 0x7f0644135d60 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:31:32.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f0649b5f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:32.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f0649b5f640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f0644142de0 con 0x7f0644135d60 2024-04-26T00:31:32.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f0649b5f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:32.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f0649b5f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0644142c60 con 0x7f0624003780 2024-04-26T00:31:32.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f0649b5f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:32.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f0649b5f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0644142f60 con 0x7f0644135920 2024-04-26T00:31:32.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f0643fff640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0644135d60 msgr2=0x7f0644138d00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:31:32.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f0643fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0644135d60 0x7f0644138d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:31:32.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f0649b5f640 10 monclient(hunting): _renew_subs 2024-04-26T00:31:32.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f0649b5f640 10 monclient(hunting): authenticate will time out at 1674.101351s 2024-04-26T00:31:32.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f0642ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0644135920 0x7f06441387c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:32.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f0642ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0644135920 0x7f06441387c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:57676/0 (socket says 172.21.15.37:57676) 2024-04-26T00:31:32.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f0642ffd640 1 -- 172.21.15.37:0/1942218360 learned_addr learned my addr 172.21.15.37:0/1942218360 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:31:32.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f0642ffd640 10 monclient(hunting): get_auth_request con 0x7f0644135920 auth_method 0 2024-04-26T00:31:32.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f0642ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:32.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f0642ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:32.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f0642ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:32.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f0642ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:32.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f0642ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:32.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f0642ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:32.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f06437fe640 1 --2- 172.21.15.37:0/1942218360 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0624003780 0x7f06441353e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:32.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f06437fe640 10 monclient(hunting): get_auth_request con 0x7f0624003780 auth_method 0 2024-04-26T00:31:32.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f06437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:32.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f06437fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:32.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f06437fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:32.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f0642ffd640 10 monclient(hunting): handle_auth_done global_id 84617 payload 824 2024-04-26T00:31:32.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f0642ffd640 1 -- 172.21.15.37:0/1942218360 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0624003780 msgr2=0x7f06441353e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:32.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f0642ffd640 1 --2- 172.21.15.37:0/1942218360 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0624003780 0x7f06441353e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:32.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f0642ffd640 1 -- 172.21.15.37:0/1942218360 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0644135d60 msgr2=0x7f0644138d00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:31:32.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f0642ffd640 1 --2- 172.21.15.37:0/1942218360 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0644135d60 0x7f0644138d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:32.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f0642ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:31:32.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f0642ffd640 1 monclient: found mon.noname-c 2024-04-26T00:31:32.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f0642ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:31:32.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f0642ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:31:32.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f0642ffd640 1 -- 172.21.15.37:0/1942218360 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f064413c0f0 con 0x7f0644135920 2024-04-26T00:31:32.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.792+0000 7f06437fe640 1 --2- 172.21.15.37:0/1942218360 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0624003780 0x7f06441353e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:31:32.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.796+0000 7f0642ffd640 1 -- 172.21.15.37:0/1942218360 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0644135920 msgr2=0x7f06441387c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:32.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.796+0000 7f0642ffd640 1 --2- 172.21.15.37:0/1942218360 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0644135920 0x7f06441387c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f062c00c900 tx=0x7f062c00cdc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:32.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.796+0000 7f063bfff640 1 -- 172.21.15.37:0/1942218360 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f062c017070 con 0x7f0644135920 2024-04-26T00:31:32.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.796+0000 7f063bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:32.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.796+0000 7f063bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:31:32.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.796+0000 7f063bfff640 10 monclient: dump: 2024-04-26T00:31:32.794 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:32.794 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:32.794 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:32.794 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:32.794 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:32.794 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:32.795 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:32.795 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:32.795 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:32.795 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:32.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.796+0000 7f063bfff640 10 monclient: _finish_auth 0 2024-04-26T00:31:32.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.796+0000 7f063bfff640 10 monclient: _check_auth_tickets 2024-04-26T00:31:32.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.796+0000 7f063bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:31:32.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.796+0000 7f0649b5f640 5 monclient: authenticate success, global_id 84617 2024-04-26T00:31:32.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.796+0000 7f0649b5f640 10 monclient: _renew_subs 2024-04-26T00:31:32.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.796+0000 7f0649b5f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:32.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.796+0000 7f0649b5f640 1 -- 172.21.15.37:0/1942218360 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f064413af20 con 0x7f0644135920 2024-04-26T00:31:32.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.796+0000 7f063bfff640 1 -- 172.21.15.37:0/1942218360 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f062c00a900 con 0x7f0644135920 2024-04-26T00:31:32.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.796+0000 7f063bfff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:31:32.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.796+0000 7f063bfff640 1 -- 172.21.15.37:0/1942218360 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f062c00acb0 con 0x7f0644135920 2024-04-26T00:31:32.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.796+0000 7f0649b5f640 10 monclient: _renew_subs 2024-04-26T00:31:32.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.796+0000 7f0649b5f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:32.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.796+0000 7f0641ffb640 4 set_mon_vals no callback set 2024-04-26T00:31:32.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.796+0000 7f0649b5f640 1 -- 172.21.15.37:0/1942218360 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f064409aca0 con 0x7f0644135920 2024-04-26T00:31:32.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.796+0000 7f063bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:32.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.796+0000 7f063bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:31:32.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.796+0000 7f063bfff640 10 monclient: dump: 2024-04-26T00:31:32.797 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:32.797 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:32.797 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:32.797 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:32.797 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:32.797 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:32.797 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:32.797 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:32.797 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:32.797 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:32.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.796+0000 7f0649b5f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:32.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.796+0000 7f0649b5f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:32.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.796+0000 7f0649b5f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:32.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.796+0000 7f0649b5f640 1 -- 172.21.15.37:0/1942218360 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0644066f40 con 0x7f0644135920 2024-04-26T00:31:32.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.796+0000 7f063bfff640 1 -- 172.21.15.37:0/1942218360 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f062c01a070 con 0x7f0644135920 2024-04-26T00:31:32.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.796+0000 7f063bfff640 1 --2- 172.21.15.37:0/1942218360 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0624036cc0 0x7f0624039180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:32.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.796+0000 7f063bfff640 1 -- 172.21.15.37:0/1942218360 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(435..435 src has 370..435) ==== 3931+0+0 (secure 0 0 0) 0x7f062c048190 con 0x7f0644135920 2024-04-26T00:31:32.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.796+0000 7f06437fe640 1 --2- 172.21.15.37:0/1942218360 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0624036cc0 0x7f0624039180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:32.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.796+0000 7f06437fe640 10 monclient: get_auth_request con 0x7f0624036cc0 auth_method 0 2024-04-26T00:31:32.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.800+0000 7f06437fe640 1 --2- 172.21.15.37:0/1942218360 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0624036cc0 0x7f0624039180 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f06240062a0 tx=0x7f063c0a7000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:32.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.800+0000 7f063bfff640 1 -- 172.21.15.37:0/1942218360 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f062c010300 con 0x7f0644135920 2024-04-26T00:31:32.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.800+0000 7f063bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:32.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:32.800+0000 7f063bfff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:31:33.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.032+0000 7f0649b5f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:31:33.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.032+0000 7f0649b5f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:31:33.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.032+0000 7f0649b5f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:33.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.032+0000 7f0649b5f640 1 -- 172.21.15.37:0/1942218360 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f064413aca0 con 0x7f0644135920 2024-04-26T00:31:33.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.032+0000 7f063bfff640 1 -- 172.21.15.37:0/1942218360 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7f062c007070 con 0x7f0644135920 2024-04-26T00:31:33.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.032+0000 7f063bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:31:33.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.032+0000 7f063bfff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:31:33.028 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":90,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:31:33.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.036+0000 7f0649b5f640 1 -- 172.21.15.37:0/1942218360 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0624036cc0 msgr2=0x7f0624039180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:33.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.036+0000 7f0649b5f640 1 --2- 172.21.15.37:0/1942218360 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0624036cc0 0x7f0624039180 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f06240062a0 tx=0x7f063c0a7000 comp rx=0 tx=0).stop 2024-04-26T00:31:33.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.036+0000 7f0649b5f640 10 monclient: shutdown 2024-04-26T00:31:33.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.036+0000 7f0649b5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:31:33.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.036+0000 7f0649b5f640 1 -- 172.21.15.37:0/1942218360 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0644135920 msgr2=0x7f06441387c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:33.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.036+0000 7f0649b5f640 1 --2- 172.21.15.37:0/1942218360 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0644135920 0x7f06441387c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f062c00c900 tx=0x7f062c00cdc0 comp rx=0 tx=0).stop 2024-04-26T00:31:33.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.036+0000 7f0649b5f640 1 -- 172.21.15.37:0/1942218360 shutdown_connections 2024-04-26T00:31:33.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.036+0000 7f0649b5f640 1 --2- 172.21.15.37:0/1942218360 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0624036cc0 0x7f0624039180 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:33.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.036+0000 7f0649b5f640 1 --2- 172.21.15.37:0/1942218360 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0644135d60 0x7f0644138d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:33.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.036+0000 7f0649b5f640 1 --2- 172.21.15.37:0/1942218360 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0644135920 0x7f06441387c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:33.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.036+0000 7f0649b5f640 1 --2- 172.21.15.37:0/1942218360 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0624003780 0x7f06441353e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:33.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.036+0000 7f0649b5f640 1 -- 172.21.15.37:0/1942218360 >> 172.21.15.37:0/1942218360 conn(0x7f06440a5f00 msgr2=0x7f064413e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:31:33.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.036+0000 7f0649b5f640 1 -- 172.21.15.37:0/1942218360 shutdown_connections 2024-04-26T00:31:33.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.036+0000 7f0649b5f640 1 -- 172.21.15.37:0/1942218360 wait complete. 2024-04-26T00:31:33.048 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-26T00:31:33.048 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-04-26T00:31:33.048 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:31:33.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.168+0000 7f969d882640 10 monclient: get_monmap_and_config 2024-04-26T00:31:33.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.168+0000 7f969d882640 10 monclient: build_initial_monmap 2024-04-26T00:31:33.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.168+0000 7f969d882640 10 monclient: monmap: 2024-04-26T00:31:33.164 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:31:33.164 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:33.164 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:31:33.175129+0000 2024-04-26T00:31:33.164 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:31:33.175129+0000 2024-04-26T00:31:33.165 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:31:33.165 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:33.165 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:31:33.165 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:31:33.165 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:31:33.165 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:31:33.165 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:31:33.165 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:33.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.172+0000 7f969d882640 1 Processor -- start 2024-04-26T00:31:33.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.172+0000 7f969d882640 1 -- start start 2024-04-26T00:31:33.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.172+0000 7f969d882640 10 monclient: init 2024-04-26T00:31:33.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.172+0000 7f969d882640 10 monclient: _reopen_session rank -1 2024-04-26T00:31:33.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.172+0000 7f969d882640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-04-26T00:31:33.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.172+0000 7f969d882640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9698136f20 0x7f969809bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:33.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.172+0000 7f969d882640 10 monclient(hunting): picked mon.noname-a con 0x7f9698136f20 addr v2:172.21.15.37:3300/0 2024-04-26T00:31:33.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.172+0000 7f969d882640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9698137300 addr v1:172.21.15.37:6789/0 2024-04-26T00:31:33.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.172+0000 7f969d882640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f969813c450 addr v1:172.21.15.45:6789/0 2024-04-26T00:31:33.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.172+0000 7f969d882640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96980a50f0 con 0x7f9698137300 2024-04-26T00:31:33.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.172+0000 7f969d882640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f969809b330 con 0x7f969813c450 2024-04-26T00:31:33.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.172+0000 7f969d882640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:33.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.172+0000 7f969d882640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f969809b610 con 0x7f9698136f20 2024-04-26T00:31:33.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.172+0000 7f969d882640 10 monclient(hunting): _renew_subs 2024-04-26T00:31:33.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.172+0000 7f969d882640 10 monclient(hunting): authenticate will time out at 1674.480875s 2024-04-26T00:31:33.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.172+0000 7f968e7fc640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f9698137300 legacy=0x7f96981377b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:31:33.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.172+0000 7f9696ffd640 1 -- >> v2:172.21.15.37:3300/0 conn(0x7f9698136f20 msgr2=0x7f969809bff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:31:33.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.172+0000 7f9696ffd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9698136f20 0x7f969809bff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:31:33.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.172+0000 7f96977fe640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f969813c450 0x7f969813e840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.37:41646/0 (socket says 172.21.15.37:41646) 2024-04-26T00:31:33.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.172+0000 7f96977fe640 1 -- 172.21.15.37:0/746859797 learned_addr learned my addr 172.21.15.37:0/746859797 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:31:33.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.172+0000 7f96977fe640 1 -- 172.21.15.37:0/746859797 >> v1:172.21.15.45:6789/0 conn(0x7f969813c450 legacy=0x7f969813e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:33.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.172+0000 7f96967fc640 1 -- 172.21.15.37:0/746859797 <== mon.1 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1180396895 0 0) 0x7f969809b330 con 0x7f969813c450 2024-04-26T00:31:33.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.172+0000 7f96967fc640 10 monclient(hunting): _negotiate 2024-04-26T00:31:33.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.172+0000 7f96967fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:33.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.172+0000 7f96967fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:33.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.172+0000 7f96967fc640 10 monclient(hunting): my global_id is 134152 2024-04-26T00:31:33.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.172+0000 7f96967fc640 1 -- 172.21.15.37:0/746859797 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9684003530 con 0x7f969813c450 2024-04-26T00:31:33.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f96967fc640 1 -- 172.21.15.37:0/746859797 <== mon.1 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2956375824 0 0) 0x7f9684003530 con 0x7f969813c450 2024-04-26T00:31:33.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f96967fc640 1 -- 172.21.15.37:0/746859797 >> v2:172.21.15.37:3300/0 conn(0x7f9698136f20 msgr2=0x7f969809bff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:31:33.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f96967fc640 1 --2- 172.21.15.37:0/746859797 >> v2:172.21.15.37:3300/0 conn(0x7f9698136f20 0x7f969809bff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:33.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f96967fc640 1 -- 172.21.15.37:0/746859797 >> v1:172.21.15.37:6789/0 conn(0x7f9698137300 legacy=0x7f96981377b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:31:33.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f96967fc640 10 monclient: _finish_hunting 0 2024-04-26T00:31:33.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f96967fc640 1 monclient: found mon.noname-c-legacy 2024-04-26T00:31:33.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f96967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:31:33.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f96967fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.45:6789/0 2024-04-26T00:31:33.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f96967fc640 1 -- 172.21.15.37:0/746859797 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f969809b8f0 con 0x7f969813c450 2024-04-26T00:31:33.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f96967fc640 10 monclient: _finish_auth 0 2024-04-26T00:31:33.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f96967fc640 10 monclient: _check_auth_tickets 2024-04-26T00:31:33.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f96967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:31:33.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f96967fc640 1 -- 172.21.15.37:0/746859797 <== mon.1 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 866326386 0 0) 0x7f9690001cb0 con 0x7f969813c450 2024-04-26T00:31:33.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f96967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:33.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f96967fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-04-26T00:31:33.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f96967fc640 10 monclient: dump: 2024-04-26T00:31:33.172 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:33.172 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:33.172 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:33.173 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:33.173 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:33.173 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:33.173 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:33.173 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:33.173 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:33.173 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:33.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f96967fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] but i'm connected to v1:172.21.15.45:6789/0, reconnecting 2024-04-26T00:31:33.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f96967fc640 10 monclient: _reopen_session rank -1 2024-04-26T00:31:33.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f96967fc640 1 -- 172.21.15.37:0/746859797 >> v1:172.21.15.45:6789/0 conn(0x7f969813c450 legacy=0x7f969813e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:33.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f96967fc640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:31:33.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f96967fc640 1 --2- 172.21.15.37:0/746859797 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f96840039c0 0x7f9684003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:33.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f96967fc640 10 monclient(hunting): picked mon.a con 0x7f96840039c0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:31:33.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f96967fc640 1 --2- 172.21.15.37:0/746859797 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9684004810 0x7f9684004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:33.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f96967fc640 10 monclient(hunting): picked mon.c con 0x7f9684004810 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:31:33.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f96967fc640 1 --2- 172.21.15.37:0/746859797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96840053d0 0x7f968400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:33.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f96967fc640 10 monclient(hunting): picked mon.b con 0x7f96840053d0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:31:33.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f96967fc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:33.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f96967fc640 1 -- 172.21.15.37:0/746859797 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f969809b610 con 0x7f96840039c0 2024-04-26T00:31:33.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f96967fc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:33.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f96967fc640 1 -- 172.21.15.37:0/746859797 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f968400c270 con 0x7f96840053d0 2024-04-26T00:31:33.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f96967fc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:33.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f9696ffd640 1 -- 172.21.15.37:0/746859797 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f96840039c0 msgr2=0x7f9684003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:31:33.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f96967fc640 1 -- 172.21.15.37:0/746859797 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f968400c470 con 0x7f9684004810 2024-04-26T00:31:33.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f96967fc640 10 monclient(hunting): _renew_subs 2024-04-26T00:31:33.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f96967fc640 10 monclient(hunting): _finish_auth 0 2024-04-26T00:31:33.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f9696ffd640 1 --2- 172.21.15.37:0/746859797 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f96840039c0 0x7f9684003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:31:33.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f968e7fc640 1 --2- 172.21.15.37:0/746859797 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9684004810 0x7f9684004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:33.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f968e7fc640 10 monclient(hunting): get_auth_request con 0x7f9684004810 auth_method 0 2024-04-26T00:31:33.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f968e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:33.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f968e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:33.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f968e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:31:33.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f968e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:33.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f968e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:33.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f968e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:31:33.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f96977fe640 1 --2- 172.21.15.37:0/746859797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96840053d0 0x7f968400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:33.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f96977fe640 10 monclient(hunting): get_auth_request con 0x7f96840053d0 auth_method 0 2024-04-26T00:31:33.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f96977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:33.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f96977fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:33.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f96977fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:31:33.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f968e7fc640 10 monclient(hunting): handle_auth_done global_id 134152 payload 293 2024-04-26T00:31:33.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f968e7fc640 1 -- 172.21.15.37:0/746859797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96840053d0 msgr2=0x7f968400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:31:33.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f968e7fc640 1 --2- 172.21.15.37:0/746859797 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f96840053d0 0x7f968400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:33.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f968e7fc640 1 -- 172.21.15.37:0/746859797 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f96840039c0 msgr2=0x7f9684003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:31:33.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f968e7fc640 1 --2- 172.21.15.37:0/746859797 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f96840039c0 0x7f9684003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:33.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f968e7fc640 10 monclient: _finish_hunting 0 2024-04-26T00:31:33.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f968e7fc640 1 monclient: found mon.c 2024-04-26T00:31:33.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f968e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:31:33.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f968e7fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:33.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f968e7fc640 1 -- 172.21.15.37:0/746859797 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9684003530 con 0x7f9684004810 2024-04-26T00:31:33.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f968e7fc640 10 monclient: _finish_auth 0 2024-04-26T00:31:33.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f968e7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:31:33.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f968e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:31:33.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f96977fe640 1 -- 172.21.15.37:0/746859797 reap_dead start 2024-04-26T00:31:33.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f969d882640 5 monclient: authenticate success, global_id 134152 2024-04-26T00:31:33.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.176+0000 7f969d882640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:31:33.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f968e7fc640 1 -- 172.21.15.37:0/746859797 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9684004810 msgr2=0x7f9684004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:33.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f968e7fc640 1 --2- 172.21.15.37:0/746859797 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9684004810 0x7f9684004c90 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9680003990 tx=0x7f968001c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48ff0d6f1d9b008f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:33.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f96967fc640 1 -- 172.21.15.37:0/746859797 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9680009520 con 0x7f9684004810 2024-04-26T00:31:33.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f96967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:33.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f96967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:31:33.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f96967fc640 10 monclient: dump: 2024-04-26T00:31:33.184 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:33.184 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:33.184 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:33.184 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:33.184 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:33.184 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:33.184 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:33.185 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:33.185 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:33.185 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:33.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f969d882640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:31:33.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f96967fc640 1 -- 172.21.15.37:0/746859797 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9680003bc0 con 0x7f9684004810 2024-04-26T00:31:33.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f96967fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:31:33.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f969d882640 10 monclient: get_monmap_and_config success 2024-04-26T00:31:33.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f96967fc640 1 -- 172.21.15.37:0/746859797 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9680005070 con 0x7f9684004810 2024-04-26T00:31:33.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f96967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:33.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f969d882640 4 set_mon_vals no callback set 2024-04-26T00:31:33.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f96967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:31:33.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f96967fc640 10 monclient: dump: 2024-04-26T00:31:33.185 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:33.185 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:33.185 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:33.186 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:33.186 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:33.186 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:33.186 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:33.186 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:33.186 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:33.186 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:33.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f969d882640 10 monclient: shutdown 2024-04-26T00:31:33.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f969d882640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:31:33.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f969d882640 1 -- 172.21.15.37:0/746859797 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9684004810 msgr2=0x7f9684004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:33.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f969d882640 1 --2- 172.21.15.37:0/746859797 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9684004810 0x7f9684004c90 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9680003990 tx=0x7f968001c420 comp rx=0 tx=0).stop 2024-04-26T00:31:33.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f969d882640 1 -- 172.21.15.37:0/746859797 shutdown_connections 2024-04-26T00:31:33.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f969d882640 1 --2- 172.21.15.37:0/746859797 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9684004810 0x7f9684004c90 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:33.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f969d882640 1 -- 172.21.15.37:0/746859797 >> 172.21.15.37:0/746859797 conn(0x7f96980a61b0 msgr2=0x7f96980a65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:31:33.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f969d882640 1 -- 172.21.15.37:0/746859797 shutdown_connections 2024-04-26T00:31:33.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f969d882640 1 -- 172.21.15.37:0/746859797 wait complete. 2024-04-26T00:31:33.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f969d882640 10 monclient: build_initial_monmap 2024-04-26T00:31:33.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f969d882640 10 monclient: monmap: 2024-04-26T00:31:33.187 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:31:33.187 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:31:33.187 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:31:33.187 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:31:33.187 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:31:33.187 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:31:33.187 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:31:33.187 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:31:33.187 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:31:33.187 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:33.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f969d882640 1 Processor -- start 2024-04-26T00:31:33.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f969d882640 1 -- start start 2024-04-26T00:31:33.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f969d882640 10 monclient: init 2024-04-26T00:31:33.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f969d882640 10 monclient: _reopen_session rank -1 2024-04-26T00:31:33.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f969d882640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:31:33.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f969d882640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9684004810 0x7f96981cbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:33.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f969d882640 10 monclient(hunting): picked mon.noname-b con 0x7f9684004810 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:31:33.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f969d882640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f96981332c0 0x7f96981cf0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:33.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f969d882640 10 monclient(hunting): picked mon.noname-c con 0x7f96981332c0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:31:33.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f969d882640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9698133740 0x7f96981cf620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:33.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f969d882640 10 monclient(hunting): picked mon.noname-a con 0x7f9698133740 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:31:33.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f969d882640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:33.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f969d882640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9698134600 con 0x7f9698133740 2024-04-26T00:31:33.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f969d882640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:33.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f969d882640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9698134480 con 0x7f9684004810 2024-04-26T00:31:33.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f969d882640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:33.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f969d882640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9698134780 con 0x7f96981332c0 2024-04-26T00:31:33.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f969d882640 10 monclient(hunting): _renew_subs 2024-04-26T00:31:33.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f969d882640 10 monclient(hunting): authenticate will time out at 1674.491477s 2024-04-26T00:31:33.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f96977fe640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9698133740 msgr2=0x7f96981cf620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:31:33.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f96977fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9698133740 0x7f96981cf620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:31:33.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f968e7fc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f96981332c0 0x7f96981cf0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:33.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f968e7fc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f96981332c0 0x7f96981cf0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:57690/0 (socket says 172.21.15.37:57690) 2024-04-26T00:31:33.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f968e7fc640 1 -- 172.21.15.37:0/2119562867 learned_addr learned my addr 172.21.15.37:0/2119562867 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:31:33.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f968e7fc640 10 monclient(hunting): get_auth_request con 0x7f96981332c0 auth_method 0 2024-04-26T00:31:33.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f968e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:33.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f968e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:33.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f968e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:33.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f968e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:33.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f968e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:33.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f968e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:33.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f9696ffd640 1 --2- 172.21.15.37:0/2119562867 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9684004810 0x7f96981cbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:33.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f9696ffd640 10 monclient(hunting): get_auth_request con 0x7f9684004810 auth_method 0 2024-04-26T00:31:33.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f9696ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:33.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f9696ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:33.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f9696ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:33.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f968e7fc640 10 monclient(hunting): handle_auth_done global_id 84620 payload 824 2024-04-26T00:31:33.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f968e7fc640 1 -- 172.21.15.37:0/2119562867 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9684004810 msgr2=0x7f96981cbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:33.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f968e7fc640 1 --2- 172.21.15.37:0/2119562867 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9684004810 0x7f96981cbbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:33.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f968e7fc640 1 -- 172.21.15.37:0/2119562867 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9698133740 msgr2=0x7f96981cf620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:31:33.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f968e7fc640 1 --2- 172.21.15.37:0/2119562867 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9698133740 0x7f96981cf620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:33.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f968e7fc640 10 monclient: _finish_hunting 0 2024-04-26T00:31:33.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f968e7fc640 1 monclient: found mon.noname-c 2024-04-26T00:31:33.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f968e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:31:33.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f968e7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:31:33.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.180+0000 7f968e7fc640 1 -- 172.21.15.37:0/2119562867 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9698120a50 con 0x7f96981332c0 2024-04-26T00:31:33.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.184+0000 7f968e7fc640 1 -- 172.21.15.37:0/2119562867 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f96981332c0 msgr2=0x7f96981cf0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:33.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.184+0000 7f968e7fc640 1 --2- 172.21.15.37:0/2119562867 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f96981332c0 0x7f96981cf0e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f968001c900 tx=0x7f9684005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:33.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.184+0000 7f9694ff9640 1 -- 172.21.15.37:0/2119562867 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96800bb070 con 0x7f96981332c0 2024-04-26T00:31:33.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.184+0000 7f9694ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:33.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.184+0000 7f9694ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:31:33.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.184+0000 7f9694ff9640 10 monclient: dump: 2024-04-26T00:31:33.191 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:33.191 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:33.191 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:33.191 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:33.191 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:33.192 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:33.192 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:33.192 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:33.192 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:33.192 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:33.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.184+0000 7f9694ff9640 10 monclient: _finish_auth 0 2024-04-26T00:31:33.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.184+0000 7f9694ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:31:33.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.184+0000 7f9694ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:31:33.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.184+0000 7f9694ff9640 1 -- 172.21.15.37:0/2119562867 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9680003970 con 0x7f96981332c0 2024-04-26T00:31:33.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.184+0000 7f9694ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:31:33.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.184+0000 7f9694ff9640 1 -- 172.21.15.37:0/2119562867 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96800ac420 con 0x7f96981332c0 2024-04-26T00:31:33.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.184+0000 7f9694ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:33.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.184+0000 7f9694ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:31:33.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.184+0000 7f9694ff9640 10 monclient: dump: 2024-04-26T00:31:33.193 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:33.193 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:33.193 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:33.193 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:33.193 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:33.193 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:33.193 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:33.193 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:33.193 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:33.193 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:33.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.184+0000 7f969d882640 5 monclient: authenticate success, global_id 84620 2024-04-26T00:31:33.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.184+0000 7f969d882640 10 monclient: _renew_subs 2024-04-26T00:31:33.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.184+0000 7f969d882640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:33.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.184+0000 7f9695ffb640 4 set_mon_vals no callback set 2024-04-26T00:31:33.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.184+0000 7f969d882640 1 -- 172.21.15.37:0/2119562867 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f969809ae30 con 0x7f96981332c0 2024-04-26T00:31:33.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.184+0000 7f969d882640 10 monclient: _renew_subs 2024-04-26T00:31:33.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.184+0000 7f969d882640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:33.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.184+0000 7f969d882640 1 -- 172.21.15.37:0/2119562867 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f96981cc7f0 con 0x7f96981332c0 2024-04-26T00:31:33.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.184+0000 7f969d882640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:33.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.184+0000 7f969d882640 1 --2- 172.21.15.37:0/2119562867 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f96981cfb60 0x7f96981355e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:33.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.184+0000 7f969d882640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:33.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.184+0000 7f969d882640 1 -- 172.21.15.37:0/2119562867 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f96981cff40 con 0x7f96981cfb60 2024-04-26T00:31:33.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.184+0000 7f9696ffd640 1 --2- 172.21.15.37:0/2119562867 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f96981cfb60 0x7f96981355e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:33.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.184+0000 7f9696ffd640 10 monclient: get_auth_request con 0x7f96981cfb60 auth_method 0 2024-04-26T00:31:33.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.184+0000 7f9696ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:33.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.184+0000 7f9696ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:33.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.184+0000 7f9696ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:33.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.184+0000 7f9696ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:33.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.184+0000 7f9696ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:33.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.184+0000 7f9696ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:33.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.184+0000 7f9696ffd640 10 monclient(hunting): handle_auth_done global_id 84623 payload 824 2024-04-26T00:31:33.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.184+0000 7f9696ffd640 1 -- 172.21.15.37:0/2119562867 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f969804f510 con 0x7f96981cfb60 2024-04-26T00:31:33.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.184+0000 7f9694ff9640 1 -- 172.21.15.37:0/2119562867 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f96800be070 con 0x7f96981332c0 2024-04-26T00:31:33.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.184+0000 7f9694ff9640 1 --2- 172.21.15.37:0/2119562867 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9678035e80 0x7f9678038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:33.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.184+0000 7f9694ff9640 1 -- 172.21.15.37:0/2119562867 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(435..435 src has 370..435) ==== 3931+0+0 (secure 0 0 0) 0x7f96800e1f60 con 0x7f96981332c0 2024-04-26T00:31:33.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.188+0000 7f96977fe640 1 --2- 172.21.15.37:0/2119562867 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9678035e80 0x7f9678038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:33.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.188+0000 7f96977fe640 10 monclient: get_auth_request con 0x7f9678035e80 auth_method 0 2024-04-26T00:31:33.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.188+0000 7f9696ffd640 1 -- 172.21.15.37:0/2119562867 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f96981cfb60 msgr2=0x7f96981355e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:33.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.188+0000 7f9696ffd640 1 --2- 172.21.15.37:0/2119562867 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f96981cfb60 0x7f96981355e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f96880038c0 tx=0x7f968800db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:33.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.188+0000 7f9694ff9640 1 -- 172.21.15.37:0/2119562867 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9688003b00 con 0x7f96981cfb60 2024-04-26T00:31:33.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.188+0000 7f9694ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:33.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.188+0000 7f9694ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:31:33.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.188+0000 7f9694ff9640 10 monclient: dump: 2024-04-26T00:31:33.196 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:33.196 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:33.196 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:33.196 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:33.196 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:33.196 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:33.196 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:33.196 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:33.196 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:33.196 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:33.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.188+0000 7f9694ff9640 1 -- 172.21.15.37:0/2119562867 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9688017070 con 0x7f96981cfb60 2024-04-26T00:31:33.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.188+0000 7f9694ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:33.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.188+0000 7f9694ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:31:33.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.188+0000 7f9694ff9640 1 -- 172.21.15.37:0/2119562867 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f96981cfb60 msgr2=0x7f96981355e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:33.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.188+0000 7f9694ff9640 1 --2- 172.21.15.37:0/2119562867 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f96981cfb60 0x7f96981355e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f96880038c0 tx=0x7f968800db90 comp rx=0 tx=0).stop 2024-04-26T00:31:33.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.188+0000 7f9694ff9640 1 -- 172.21.15.37:0/2119562867 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f96981cfb60 msgr2=0x7f96981355e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:31:33.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.188+0000 7f9694ff9640 1 --2- 172.21.15.37:0/2119562867 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f96981cfb60 0x7f96981355e0 secure :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f96880038c0 tx=0x7f968800db90 comp rx=0 tx=0).stop 2024-04-26T00:31:33.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.188+0000 7f96977fe640 1 --2- 172.21.15.37:0/2119562867 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9678035e80 0x7f9678038340 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f96980a5520 tx=0x7f96900016b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:33.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.196+0000 7f969d882640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:31:33.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.196+0000 7f969d882640 1 --2- 172.21.15.37:0/2119562867 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9698135b20 0x7f9698135f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:33.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.196+0000 7f969d882640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:33.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.196+0000 7f969d882640 1 -- 172.21.15.37:0/2119562867 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f969809f280 con 0x7f9698135b20 2024-04-26T00:31:33.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.196+0000 7f9696ffd640 1 --2- 172.21.15.37:0/2119562867 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9698135b20 0x7f9698135f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:33.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.196+0000 7f9696ffd640 10 monclient: get_auth_request con 0x7f9698135b20 auth_method 0 2024-04-26T00:31:33.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.196+0000 7f9696ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:33.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.196+0000 7f9696ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:33.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.196+0000 7f9696ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:33.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.196+0000 7f9696ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:33.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.196+0000 7f9696ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:33.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.196+0000 7f9696ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:33.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.196+0000 7f9696ffd640 10 monclient(hunting): handle_auth_done global_id 84626 payload 824 2024-04-26T00:31:33.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.196+0000 7f9696ffd640 1 -- 172.21.15.37:0/2119562867 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f969809fb60 con 0x7f9698135b20 2024-04-26T00:31:33.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.196+0000 7f9696ffd640 1 -- 172.21.15.37:0/2119562867 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9698135b20 msgr2=0x7f9698135f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:33.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.196+0000 7f9696ffd640 1 --2- 172.21.15.37:0/2119562867 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9698135b20 0x7f9698135f00 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f96880038c0 tx=0x7f9688005980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:33.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.196+0000 7f9694ff9640 1 -- 172.21.15.37:0/2119562867 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96880052d0 con 0x7f9698135b20 2024-04-26T00:31:33.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.196+0000 7f9694ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:33.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.196+0000 7f9694ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:31:33.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.196+0000 7f9694ff9640 10 monclient: dump: 2024-04-26T00:31:33.200 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:33.201 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:33.201 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:33.201 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:33.201 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:33.201 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:33.201 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:33.201 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:33.201 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:33.201 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:33.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.196+0000 7f9694ff9640 1 -- 172.21.15.37:0/2119562867 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4878 (secure 0 0 0) 0x7f96880052d0 con 0x7f9698135b20 2024-04-26T00:31:33.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.196+0000 7f9694ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:31:33.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.196+0000 7f9694ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:31:33.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.196+0000 7f9694ff9640 1 -- 172.21.15.37:0/2119562867 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9698135b20 msgr2=0x7f9698135f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:33.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.196+0000 7f9694ff9640 1 --2- 172.21.15.37:0/2119562867 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9698135b20 0x7f9698135f00 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f96880038c0 tx=0x7f9688005980 comp rx=0 tx=0).stop 2024-04-26T00:31:33.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.196+0000 7f9694ff9640 1 -- 172.21.15.37:0/2119562867 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9698135b20 msgr2=0x7f9698135f00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:31:33.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.196+0000 7f9694ff9640 1 --2- 172.21.15.37:0/2119562867 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9698135b20 0x7f9698135f00 secure :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f96880038c0 tx=0x7f9688005980 comp rx=0 tx=0).stop 2024-04-26T00:31:33.202 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:31:33.203 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:31:33.203 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:31:33.203 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:31:33.203 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 90, 2024-04-26T00:31:33.203 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 298132, 2024-04-26T00:31:33.203 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:31:33.203 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:31:33.203 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:31:33.203 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:31:33.203 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:31:33.203 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:31:33.203 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:31:33.203 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:31:33.203 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:31:33.204 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:31:33.204 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:31:33.204 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:31:33.204 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:31:33.204 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:31:33.204 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:31:33.204 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:31:33.204 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:31:33.204 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:31:33.204 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:31:33.204 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:31:33.204 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:31:33.204 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:31:33.204 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:31:33.204 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:31:33.205 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:31:33.205 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:31:33.205 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:31:33.205 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:31:33.205 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:31:33.205 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:31:33.205 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:31:33.205 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:31:33.205 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:31:33.205 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:33.205 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:33.205 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:31:33.205 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:31:33.205 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:31:33.205 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:31:33.205 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:31:33.206 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:31:33.206 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:31:33.206 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:31:33.206 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:31:33.206 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:31:33.206 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:31:33.206 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:31:33.206 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:31:33.206 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:31:33.206 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:31:33.206 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:31:33.206 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:31:33.206 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:31:33.206 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:31:33.207 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:31:33.207 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:31:33.207 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:31:33.207 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:31:33.207 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:31:33.207 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:31:33.207 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:31:33.207 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:31:33.207 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:31:33.207 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:31:33.207 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:31:33.207 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:33.207 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:31:33.208 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:33.208 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:31:33.208 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:31:33.208 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:31:33.208 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:31:33.208 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:33.208 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:31:33.208 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:31:33.208 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:33.208 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:33.208 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:33.208 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:31:33.208 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:31:33.209 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:33.209 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:33.209 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:33.209 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:33.209 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:31:33.209 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:31:33.209 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:31:33.209 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:31:33.209 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:31:33.209 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:33.209 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:33.209 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:31:33.209 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:31:33.210 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:31:33.210 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:31:33.210 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:33.210 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:31:33.210 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:31:33.210 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:33.210 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:33.210 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:33.210 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:31:33.210 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:31:33.210 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:33.210 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:33.210 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:33.210 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:33.211 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:31:33.211 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:31:33.211 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:31:33.211 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:31:33.211 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:31:33.211 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:33.211 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:33.211 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:31:33.211 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:31:33.211 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:31:33.211 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:31:33.211 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:33.211 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:31:33.211 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:31:33.211 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:33.212 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:33.212 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:33.212 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:31:33.212 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:31:33.212 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:33.212 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:33.212 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:33.212 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:33.212 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:31:33.212 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:31:33.212 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:31:33.212 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:31:33.212 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:31:33.212 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:33.213 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:33.213 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:33.213 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:31:33.213 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:31:33.213 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:33.213 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:31:33.213 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:31:33.213 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:31:33.213 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:33.213 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:31:33.213 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:31:33.213 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:33.213 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:31:33.213 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:31:33.213 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:31:33.213 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:33.214 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:31:33.214 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:31:33.214 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:33.214 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:31:33.214 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:31:33.214 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 7 2024-04-26T00:31:33.214 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:33.214 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:31:33.214 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:31:33.214 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:33.214 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:31:33.214 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:31:33.214 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:31:33.215 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:33.215 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:33.215 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:33.215 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:31:33.215 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:31:33.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.196+0000 7f969d882640 1 -- 172.21.15.37:0/2119562867 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9678035e80 msgr2=0x7f9678038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:33.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.196+0000 7f969d882640 1 --2- 172.21.15.37:0/2119562867 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9678035e80 0x7f9678038340 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f96980a5520 tx=0x7f96900016b0 comp rx=0 tx=0).stop 2024-04-26T00:31:33.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.196+0000 7f969d882640 10 monclient: shutdown 2024-04-26T00:31:33.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.196+0000 7f969d882640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:31:33.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.196+0000 7f969d882640 1 -- 172.21.15.37:0/2119562867 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f96981332c0 msgr2=0x7f96981cf0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:33.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.196+0000 7f969d882640 1 --2- 172.21.15.37:0/2119562867 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f96981332c0 0x7f96981cf0e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f968001c900 tx=0x7f9684005840 comp rx=0 tx=0).stop 2024-04-26T00:31:33.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.196+0000 7f96977fe640 1 -- 172.21.15.37:0/2119562867 reap_dead start 2024-04-26T00:31:33.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.196+0000 7f96977fe640 1 -- 172.21.15.37:0/2119562867 reap_dead start 2024-04-26T00:31:33.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.196+0000 7f969d882640 1 -- 172.21.15.37:0/2119562867 shutdown_connections 2024-04-26T00:31:33.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.196+0000 7f969d882640 1 -- 172.21.15.37:0/2119562867 >> 172.21.15.37:0/2119562867 conn(0x7f96980a61b0 msgr2=0x7f969813bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:31:33.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.200+0000 7f969d882640 1 -- 172.21.15.37:0/2119562867 shutdown_connections 2024-04-26T00:31:33.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.200+0000 7f969d882640 1 -- 172.21.15.37:0/2119562867 wait complete. 2024-04-26T00:31:33.217 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:31:33.217 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-04-26T00:31:33.218 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:31:33.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f9664122640 10 monclient: get_monmap_and_config 2024-04-26T00:31:33.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f9664122640 10 monclient: build_initial_monmap 2024-04-26T00:31:33.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f9664122640 10 monclient: monmap: 2024-04-26T00:31:33.322 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:31:33.322 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:33.322 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:31:33.332802+0000 2024-04-26T00:31:33.322 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:31:33.332802+0000 2024-04-26T00:31:33.322 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:31:33.322 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:33.322 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:31:33.322 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:31:33.323 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:31:33.323 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:31:33.323 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:31:33.323 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:33.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f9664122640 1 Processor -- start 2024-04-26T00:31:33.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f9664122640 1 -- start start 2024-04-26T00:31:33.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f9664122640 10 monclient: init 2024-04-26T00:31:33.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f9664122640 10 monclient: _reopen_session rank -1 2024-04-26T00:31:33.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f9664122640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-04-26T00:31:33.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f9664122640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f965c0a0300 0x7f965c09fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:33.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f9664122640 10 monclient(hunting): picked mon.noname-b con 0x7f965c0a0300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:31:33.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f9664122640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f965c0a06e0 0x7f965c134ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:33.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f9664122640 10 monclient(hunting): picked mon.noname-c con 0x7f965c0a06e0 addr v2:172.21.15.45:3300/0 2024-04-26T00:31:33.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f9664122640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f965c1350e0 addr v1:172.21.15.37:6789/0 2024-04-26T00:31:33.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f9664122640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f965c09c190 con 0x7f965c1350e0 2024-04-26T00:31:33.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f9664122640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:33.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f9664122640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f965c0a13f0 con 0x7f965c0a06e0 2024-04-26T00:31:33.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f9664122640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:33.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f9664122640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f965c0a1110 con 0x7f965c0a0300 2024-04-26T00:31:33.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f9664122640 10 monclient(hunting): _renew_subs 2024-04-26T00:31:33.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f9664122640 10 monclient(hunting): authenticate will time out at 1674.638159s 2024-04-26T00:31:33.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f96626bf640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f965c1350e0 legacy=0x7f965c13bed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:31:33.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f9661ebe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f965c0a0300 0x7f965c09fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:33.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f9661ebe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f965c0a0300 0x7f965c09fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:57722/0 (socket says 172.21.15.37:57722) 2024-04-26T00:31:33.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f9661ebe640 1 -- 172.21.15.37:0/2783658988 learned_addr learned my addr 172.21.15.37:0/2783658988 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:31:33.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f9661ebe640 10 monclient(hunting): get_auth_request con 0x7f965c0a0300 auth_method 0 2024-04-26T00:31:33.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f9661ebe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:33.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f9661ebe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:33.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f9661ebe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:33.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f9661ebe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:33.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f9661ebe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:33.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f9661ebe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:33.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f96616bd640 1 --2- 172.21.15.37:0/2783658988 >> v2:172.21.15.45:3300/0 conn(0x7f965c0a06e0 0x7f965c134ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:33.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f96616bd640 10 monclient(hunting): get_auth_request con 0x7f965c0a06e0 auth_method 0 2024-04-26T00:31:33.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f96616bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:33.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f96616bd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:33.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f96616bd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:33.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f9661ebe640 10 monclient(hunting): handle_auth_done global_id 84629 payload 274 2024-04-26T00:31:33.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f9661ebe640 1 -- 172.21.15.37:0/2783658988 >> v2:172.21.15.45:3300/0 conn(0x7f965c0a06e0 msgr2=0x7f965c134ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:31:33.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f9661ebe640 1 --2- 172.21.15.37:0/2783658988 >> v2:172.21.15.45:3300/0 conn(0x7f965c0a06e0 0x7f965c134ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:33.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f9661ebe640 1 -- 172.21.15.37:0/2783658988 >> v1:172.21.15.37:6789/0 conn(0x7f965c1350e0 legacy=0x7f965c13bed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:31:33.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f9661ebe640 10 monclient: _finish_hunting 0 2024-04-26T00:31:33.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f9661ebe640 1 monclient: found mon.noname-b 2024-04-26T00:31:33.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f9661ebe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:31:33.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f9661ebe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:31:33.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f9661ebe640 1 -- 172.21.15.37:0/2783658988 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f965c0a16d0 con 0x7f965c0a0300 2024-04-26T00:31:33.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.328+0000 7f96616bd640 1 --2- 172.21.15.37:0/2783658988 >> v2:172.21.15.45:3300/0 conn(0x7f965c0a06e0 0x7f965c134ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:31:33.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9661ebe640 1 -- 172.21.15.37:0/2783658988 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f965c0a0300 msgr2=0x7f965c09fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:33.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9661ebe640 1 --2- 172.21.15.37:0/2783658988 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f965c0a0300 0x7f965c09fc50 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9654009870 tx=0x7f965401c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12903d112cf1ae9b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:33.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9660ebc640 1 -- 172.21.15.37:0/2783658988 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96540b0070 con 0x7f965c0a0300 2024-04-26T00:31:33.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9660ebc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:33.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9660ebc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:31:33.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9660ebc640 10 monclient: dump: 2024-04-26T00:31:33.332 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:33.332 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:33.332 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:33.332 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:33.332 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:33.332 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:33.332 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:33.332 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:33.332 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:33.332 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:33.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9660ebc640 10 monclient: _finish_auth 0 2024-04-26T00:31:33.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9660ebc640 10 monclient: _check_auth_tickets 2024-04-26T00:31:33.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9660ebc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:31:33.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9660ebc640 1 -- 172.21.15.37:0/2783658988 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96540ab600 con 0x7f965c0a0300 2024-04-26T00:31:33.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9664122640 5 monclient: authenticate success, global_id 84629 2024-04-26T00:31:33.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9664122640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:31:33.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9660ebc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:31:33.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9660ebc640 1 -- 172.21.15.37:0/2783658988 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96540ab9d0 con 0x7f965c0a0300 2024-04-26T00:31:33.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9664122640 10 monclient: get_monmap_and_config success 2024-04-26T00:31:33.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9664122640 4 set_mon_vals no callback set 2024-04-26T00:31:33.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9660ebc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:33.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9660ebc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:31:33.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9660ebc640 10 monclient: dump: 2024-04-26T00:31:33.333 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:33.333 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:33.333 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:33.333 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:33.334 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:33.334 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:33.334 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:33.334 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:33.334 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:33.334 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:33.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9664122640 10 monclient: shutdown 2024-04-26T00:31:33.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9664122640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:31:33.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9664122640 1 -- 172.21.15.37:0/2783658988 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f965c0a0300 msgr2=0x7f965c09fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:33.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9664122640 1 --2- 172.21.15.37:0/2783658988 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f965c0a0300 0x7f965c09fc50 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9654009870 tx=0x7f965401c410 comp rx=0 tx=0).stop 2024-04-26T00:31:33.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9664122640 1 -- 172.21.15.37:0/2783658988 shutdown_connections 2024-04-26T00:31:33.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9664122640 1 --2- 172.21.15.37:0/2783658988 >> v2:172.21.15.45:3300/0 conn(0x7f965c0a06e0 0x7f965c134ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:33.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9664122640 1 --2- 172.21.15.37:0/2783658988 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f965c0a0300 0x7f965c09fc50 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:33.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9664122640 1 -- 172.21.15.37:0/2783658988 >> 172.21.15.37:0/2783658988 conn(0x7f965c0a1d50 msgr2=0x7f965c0a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:31:33.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9664122640 1 -- 172.21.15.37:0/2783658988 shutdown_connections 2024-04-26T00:31:33.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9664122640 1 -- 172.21.15.37:0/2783658988 wait complete. 2024-04-26T00:31:33.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9664122640 10 monclient: build_initial_monmap 2024-04-26T00:31:33.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9664122640 10 monclient: monmap: 2024-04-26T00:31:33.335 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:31:33.335 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:31:33.335 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:31:33.335 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:31:33.335 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:31:33.335 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:31:33.335 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:31:33.335 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:31:33.335 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:31:33.335 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:33.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9664122640 1 Processor -- start 2024-04-26T00:31:33.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9664122640 1 -- start start 2024-04-26T00:31:33.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9664122640 10 monclient: init 2024-04-26T00:31:33.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9664122640 10 monclient: _reopen_session rank -1 2024-04-26T00:31:33.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9664122640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:31:33.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9664122640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f965c0a06e0 0x7f965c1cd420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:33.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9664122640 10 monclient(hunting): picked mon.noname-a con 0x7f965c0a06e0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:31:33.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9664122640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f965c1350e0 0x7f965c13c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:33.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9664122640 10 monclient(hunting): picked mon.noname-b con 0x7f965c1350e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:31:33.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9664122640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f965c13c660 0x7f965c135bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:33.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9664122640 10 monclient(hunting): picked mon.noname-c con 0x7f965c13c660 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:31:33.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9664122640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:33.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9664122640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f965c0a0c40 con 0x7f965c0a06e0 2024-04-26T00:31:33.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9664122640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:33.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9664122640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f965c0a0ac0 con 0x7f965c1350e0 2024-04-26T00:31:33.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9664122640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:33.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9664122640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f965c13fe10 con 0x7f965c13c660 2024-04-26T00:31:33.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9664122640 10 monclient(hunting): _renew_subs 2024-04-26T00:31:33.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9664122640 10 monclient(hunting): authenticate will time out at 1674.647271s 2024-04-26T00:31:33.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9661ebe640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f965c0a06e0 msgr2=0x7f965c1cd420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:31:33.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f9661ebe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f965c0a06e0 0x7f965c1cd420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:31:33.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f96626bf640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f965c13c660 0x7f965c135bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:33.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f96626bf640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f965c13c660 0x7f965c135bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:57728/0 (socket says 172.21.15.37:57728) 2024-04-26T00:31:33.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f96626bf640 1 -- 172.21.15.37:0/1044576521 learned_addr learned my addr 172.21.15.37:0/1044576521 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:31:33.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f96626bf640 10 monclient(hunting): get_auth_request con 0x7f965c13c660 auth_method 0 2024-04-26T00:31:33.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f96626bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:33.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f96626bf640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:33.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f96626bf640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:33.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f96626bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:33.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f96626bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:33.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f96626bf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:33.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f96616bd640 1 --2- 172.21.15.37:0/1044576521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f965c1350e0 0x7f965c13c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:33.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f96616bd640 10 monclient(hunting): get_auth_request con 0x7f965c1350e0 auth_method 0 2024-04-26T00:31:33.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f96616bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:33.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f96616bd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:33.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f96616bd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:33.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f96626bf640 10 monclient(hunting): handle_auth_done global_id 84632 payload 824 2024-04-26T00:31:33.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f96626bf640 1 -- 172.21.15.37:0/1044576521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f965c1350e0 msgr2=0x7f965c13c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:33.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f96626bf640 1 --2- 172.21.15.37:0/1044576521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f965c1350e0 0x7f965c13c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:33.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f96626bf640 1 -- 172.21.15.37:0/1044576521 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f965c0a06e0 msgr2=0x7f965c1cd420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:31:33.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f96626bf640 1 --2- 172.21.15.37:0/1044576521 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f965c0a06e0 0x7f965c1cd420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:33.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f96626bf640 10 monclient: _finish_hunting 0 2024-04-26T00:31:33.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f96626bf640 1 monclient: found mon.noname-c 2024-04-26T00:31:33.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f96626bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:31:33.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f96626bf640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:31:33.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.336+0000 7f96626bf640 1 -- 172.21.15.37:0/1044576521 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f965c1376c0 con 0x7f965c13c660 2024-04-26T00:31:33.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f96626bf640 1 -- 172.21.15.37:0/1044576521 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f965c13c660 msgr2=0x7f965c135bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:33.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f96626bf640 1 --2- 172.21.15.37:0/1044576521 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f965c13c660 0x7f965c135bb0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f965000b480 tx=0x7f965000b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:33.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f965a7fc640 1 -- 172.21.15.37:0/1044576521 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9650005070 con 0x7f965c13c660 2024-04-26T00:31:33.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f965a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:33.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f965a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:31:33.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f965a7fc640 10 monclient: dump: 2024-04-26T00:31:33.344 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:33.344 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:33.344 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:33.344 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:33.344 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:33.344 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:33.345 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:33.345 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:33.345 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:33.345 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:33.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f965a7fc640 10 monclient: _finish_auth 0 2024-04-26T00:31:33.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f965a7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:31:33.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f965a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:31:33.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f9664122640 5 monclient: authenticate success, global_id 84632 2024-04-26T00:31:33.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f965a7fc640 1 -- 172.21.15.37:0/1044576521 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9650015070 con 0x7f965c13c660 2024-04-26T00:31:33.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f9664122640 10 monclient: _renew_subs 2024-04-26T00:31:33.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f9664122640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:33.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f9664122640 1 -- 172.21.15.37:0/1044576521 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f965c09f550 con 0x7f965c13c660 2024-04-26T00:31:33.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f965a7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:31:33.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f965a7fc640 1 -- 172.21.15.37:0/1044576521 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9650005450 con 0x7f965c13c660 2024-04-26T00:31:33.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f965a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:33.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f965a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:31:33.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f965a7fc640 10 monclient: dump: 2024-04-26T00:31:33.346 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:33.346 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:33.346 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:33.346 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:33.346 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:33.346 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:33.346 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:33.346 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:33.346 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:33.347 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:33.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f965bfff640 4 set_mon_vals no callback set 2024-04-26T00:31:33.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f9664122640 10 monclient: _renew_subs 2024-04-26T00:31:33.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f9664122640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:33.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f9664122640 1 -- 172.21.15.37:0/1044576521 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f965c1399b0 con 0x7f965c13c660 2024-04-26T00:31:33.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f9664122640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:33.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f9664122640 1 --2- 172.21.15.37:0/1044576521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f965c1360f0 0x7f965c1364d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:33.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f9664122640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:33.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f9664122640 1 -- 172.21.15.37:0/1044576521 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f965c139630 con 0x7f965c1360f0 2024-04-26T00:31:33.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f965a7fc640 1 -- 172.21.15.37:0/1044576521 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9650005630 con 0x7f965c13c660 2024-04-26T00:31:33.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f9661ebe640 1 --2- 172.21.15.37:0/1044576521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f965c1360f0 0x7f965c1364d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:33.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f9661ebe640 10 monclient: get_auth_request con 0x7f965c1360f0 auth_method 0 2024-04-26T00:31:33.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f9661ebe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:33.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f9661ebe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:33.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f9661ebe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:33.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f965a7fc640 1 --2- 172.21.15.37:0/1044576521 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9648035e80 0x7f9648038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:33.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f9661ebe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:33.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f9661ebe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:33.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f9661ebe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:33.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f965a7fc640 1 -- 172.21.15.37:0/1044576521 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(435..435 src has 370..435) ==== 3931+0+0 (secure 0 0 0) 0x7f96500488a0 con 0x7f965c13c660 2024-04-26T00:31:33.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f96616bd640 1 --2- 172.21.15.37:0/1044576521 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9648035e80 0x7f9648038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:33.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f96616bd640 10 monclient: get_auth_request con 0x7f9648035e80 auth_method 0 2024-04-26T00:31:33.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f9661ebe640 10 monclient(hunting): handle_auth_done global_id 134164 payload 824 2024-04-26T00:31:33.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.340+0000 7f9661ebe640 1 -- 172.21.15.37:0/1044576521 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f965c136a10 con 0x7f965c1360f0 2024-04-26T00:31:33.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.344+0000 7f9661ebe640 1 -- 172.21.15.37:0/1044576521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f965c1360f0 msgr2=0x7f965c1364d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:33.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.344+0000 7f9661ebe640 1 --2- 172.21.15.37:0/1044576521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f965c1360f0 0x7f965c1364d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f96540ad000 tx=0x7f965401acb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:33.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.344+0000 7f96616bd640 1 --2- 172.21.15.37:0/1044576521 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9648035e80 0x7f9648038340 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f965c1372c0 tx=0x7f964c003cf0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:33.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.344+0000 7f965a7fc640 1 -- 172.21.15.37:0/1044576521 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96540ba020 con 0x7f965c1360f0 2024-04-26T00:31:33.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.344+0000 7f965a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:33.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.344+0000 7f965a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:31:33.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.344+0000 7f965a7fc640 10 monclient: dump: 2024-04-26T00:31:33.349 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:33.349 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:33.349 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:33.349 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:33.349 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:33.349 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:33.349 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:33.349 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:33.349 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:33.349 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:33.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.344+0000 7f965a7fc640 1 -- 172.21.15.37:0/1044576521 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f96540aba10 con 0x7f965c1360f0 2024-04-26T00:31:33.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.344+0000 7f965a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:33.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.344+0000 7f965a7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:31:33.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.344+0000 7f965a7fc640 1 -- 172.21.15.37:0/1044576521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f965c1360f0 msgr2=0x7f965c1364d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:33.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.344+0000 7f965a7fc640 1 --2- 172.21.15.37:0/1044576521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f965c1360f0 0x7f965c1364d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f96540ad000 tx=0x7f965401acb0 comp rx=0 tx=0).stop 2024-04-26T00:31:33.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.344+0000 7f965a7fc640 1 -- 172.21.15.37:0/1044576521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f965c1360f0 msgr2=0x7f965c1364d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:31:33.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.344+0000 7f965a7fc640 1 --2- 172.21.15.37:0/1044576521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f965c1360f0 0x7f965c1364d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:33.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.352+0000 7f9664122640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:31:33.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.352+0000 7f9664122640 1 --2- 172.21.15.37:0/1044576521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f965c0700c0 0x7f965c0704d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:33.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.352+0000 7f9664122640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:33.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.352+0000 7f9664122640 1 -- 172.21.15.37:0/1044576521 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f965c139110 con 0x7f965c0700c0 2024-04-26T00:31:33.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.352+0000 7f9661ebe640 1 --2- 172.21.15.37:0/1044576521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f965c0700c0 0x7f965c0704d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:33.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.352+0000 7f9661ebe640 10 monclient: get_auth_request con 0x7f965c0700c0 auth_method 0 2024-04-26T00:31:33.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.352+0000 7f9661ebe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:33.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.352+0000 7f9661ebe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:33.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.352+0000 7f9661ebe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:33.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.352+0000 7f9661ebe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:33.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.352+0000 7f9661ebe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:33.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.352+0000 7f9661ebe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:33.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.352+0000 7f9661ebe640 10 monclient(hunting): handle_auth_done global_id 134167 payload 824 2024-04-26T00:31:33.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.352+0000 7f9661ebe640 1 -- 172.21.15.37:0/1044576521 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f965c04f510 con 0x7f965c0700c0 2024-04-26T00:31:33.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.356+0000 7f9661ebe640 1 -- 172.21.15.37:0/1044576521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f965c0700c0 msgr2=0x7f965c0704d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:33.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.356+0000 7f9661ebe640 1 --2- 172.21.15.37:0/1044576521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f965c0700c0 0x7f965c0704d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9654009870 tx=0x7f9654019210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:33.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.356+0000 7f965a7fc640 1 -- 172.21.15.37:0/1044576521 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96540c6070 con 0x7f965c0700c0 2024-04-26T00:31:33.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.356+0000 7f965a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:33.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.356+0000 7f965a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:31:33.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.356+0000 7f965a7fc640 10 monclient: dump: 2024-04-26T00:31:33.353 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:33.353 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:33.353 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:33.353 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:33.353 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:33.353 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:33.353 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:33.353 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:33.353 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:33.353 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:33.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.356+0000 7f965a7fc640 1 -- 172.21.15.37:0/1044576521 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4535 (secure 0 0 0) 0x7f96540c1c80 con 0x7f965c0700c0 2024-04-26T00:31:33.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.356+0000 7f965a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:31:33.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.356+0000 7f965a7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:31:33.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.356+0000 7f965a7fc640 1 -- 172.21.15.37:0/1044576521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f965c0700c0 msgr2=0x7f965c0704d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:33.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.356+0000 7f965a7fc640 1 --2- 172.21.15.37:0/1044576521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f965c0700c0 0x7f965c0704d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9654009870 tx=0x7f9654019210 comp rx=0 tx=0).stop 2024-04-26T00:31:33.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.356+0000 7f965a7fc640 1 -- 172.21.15.37:0/1044576521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f965c0700c0 msgr2=0x7f965c0704d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:31:33.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.356+0000 7f965a7fc640 1 --2- 172.21.15.37:0/1044576521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f965c0700c0 0x7f965c0704d0 secure :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9654009870 tx=0x7f9654019210 comp rx=0 tx=0).stop 2024-04-26T00:31:33.354 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:31:33.354 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:31:33.354 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:31:33.354 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:31:33.354 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 90, 2024-04-26T00:31:33.354 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 23715, 2024-04-26T00:31:33.354 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:31:33.355 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:31:33.355 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:31:33.355 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:31:33.355 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:31:33.355 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:31:33.355 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:31:33.355 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:31:33.355 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:31:33.355 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:31:33.355 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:31:33.355 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:31:33.355 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:31:33.355 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:31:33.355 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:31:33.355 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:31:33.356 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:31:33.356 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:31:33.356 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:31:33.356 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:31:33.356 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:31:33.356 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:31:33.356 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:31:33.356 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:31:33.356 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:31:33.356 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:31:33.356 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:31:33.356 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:31:33.356 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:31:33.356 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:31:33.357 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:31:33.357 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:31:33.357 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:31:33.357 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:33.357 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:33.357 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:31:33.357 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:31:33.357 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:31:33.357 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:31:33.357 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:31:33.357 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:31:33.357 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:31:33.357 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:31:33.357 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:31:33.357 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:31:33.358 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:31:33.358 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:31:33.358 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:31:33.358 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:31:33.358 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:31:33.358 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:31:33.358 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:31:33.358 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:31:33.358 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:31:33.358 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:31:33.358 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:31:33.358 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:31:33.358 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:31:33.358 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:31:33.358 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:31:33.359 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:31:33.359 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:31:33.359 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:31:33.359 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:31:33.359 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:31:33.359 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:33.359 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:31:33.359 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:33.359 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:31:33.359 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:31:33.359 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:31:33.359 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:31:33.359 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:33.359 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:31:33.360 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:31:33.360 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:33.360 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:33.360 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:33.360 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:31:33.360 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:31:33.360 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:33.360 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:33.360 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:33.360 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:33.360 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:31:33.360 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:31:33.360 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:31:33.360 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:31:33.360 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:31:33.361 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:33.361 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:33.361 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:31:33.361 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:31:33.361 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:31:33.361 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:31:33.361 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:33.361 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:31:33.361 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:31:33.361 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:33.361 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:33.361 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:33.361 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:31:33.361 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:31:33.361 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:33.362 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:33.362 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:33.362 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:33.362 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:31:33.362 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:31:33.362 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:31:33.362 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:31:33.362 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:31:33.362 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:33.362 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:33.362 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:31:33.362 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:31:33.363 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:31:33.363 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:31:33.363 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:33.363 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:31:33.363 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:31:33.363 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:33.363 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:33.363 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:33.363 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:31:33.363 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:31:33.363 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:31:33.363 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:33.363 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:33.363 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:33.364 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:31:33.364 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:31:33.364 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:31:33.364 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:31:33.364 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:31:33.364 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:33.364 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:33.364 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:33.364 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:31:33.364 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:31:33.364 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:33.364 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:31:33.364 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:31:33.364 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:31:33.364 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:33.364 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:31:33.365 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:31:33.365 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:31:33.365 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:31:33.365 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:31:33.365 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:31:33.365 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:31:33.365 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:31:33.365 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:31:33.365 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:31:33.365 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:31:33.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.356+0000 7f9664122640 1 -- 172.21.15.37:0/1044576521 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9648035e80 msgr2=0x7f9648038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:33.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.356+0000 7f9664122640 1 --2- 172.21.15.37:0/1044576521 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9648035e80 0x7f9648038340 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f965c1372c0 tx=0x7f964c003cf0 comp rx=0 tx=0).stop 2024-04-26T00:31:33.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.356+0000 7f9664122640 10 monclient: shutdown 2024-04-26T00:31:33.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.356+0000 7f9664122640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:31:33.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.356+0000 7f9664122640 1 -- 172.21.15.37:0/1044576521 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f965c13c660 msgr2=0x7f965c135bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:33.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.356+0000 7f9664122640 1 --2- 172.21.15.37:0/1044576521 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f965c13c660 0x7f965c135bb0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f965000b480 tx=0x7f965000b940 comp rx=0 tx=0).stop 2024-04-26T00:31:33.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.356+0000 7f96626bf640 1 -- 172.21.15.37:0/1044576521 reap_dead start 2024-04-26T00:31:33.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.356+0000 7f96626bf640 1 -- 172.21.15.37:0/1044576521 reap_dead start 2024-04-26T00:31:33.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.356+0000 7f9664122640 1 -- 172.21.15.37:0/1044576521 shutdown_connections 2024-04-26T00:31:33.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.356+0000 7f9664122640 1 -- 172.21.15.37:0/1044576521 >> 172.21.15.37:0/1044576521 conn(0x7f965c0a1d50 msgr2=0x7f965c0a3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:31:33.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.356+0000 7f9664122640 1 -- 172.21.15.37:0/1044576521 shutdown_connections 2024-04-26T00:31:33.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:33.356+0000 7f9664122640 1 -- 172.21.15.37:0/1044576521 wait complete. 2024-04-26T00:31:33.368 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:31:33.368 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-04-26T00:31:33.614 INFO:tasks.rados.rados.0.smithi045.stdout:1900: done (0 left) 2024-04-26T00:31:33.614 INFO:tasks.rados.rados.0.smithi045.stdout:1901: read oid 24 snap -1 2024-04-26T00:31:33.614 INFO:tasks.rados.rados.0.smithi045.stdout:1901: expect (ObjNum 582 snap 180 seq_num 582) 2024-04-26T00:31:33.614 INFO:tasks.rados.rados.0.smithi045.stdout:1902: delete oid 46 current snap is 219 2024-04-26T00:31:33.627 INFO:tasks.rados.rados.0.smithi045.stdout:1901: expect (ObjNum 582 snap 180 seq_num 582) 2024-04-26T00:31:33.840 INFO:tasks.rados.rados.0.smithi045.stdout:1901: done (1 left) 2024-04-26T00:31:33.840 INFO:tasks.rados.rados.0.smithi045.stdout:1902: done (0 left) 2024-04-26T00:31:33.840 INFO:tasks.rados.rados.0.smithi045.stdout:1903: read oid 27 snap -1 2024-04-26T00:31:33.840 INFO:tasks.rados.rados.0.smithi045.stdout:1903: expect (ObjNum 582 snap 180 seq_num 582) 2024-04-26T00:31:33.840 INFO:tasks.rados.rados.0.smithi045.stdout:1904: rollback oid 9 current snap is 219 2024-04-26T00:31:33.840 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 9 to 199 2024-04-26T00:31:33.840 INFO:tasks.rados.rados.0.smithi045.stdout:1905: write oid 32 current snap is 219 2024-04-26T00:31:33.840 INFO:tasks.rados.rados.0.smithi045.stdout:1905: seq_num 696 ranges {497672=561427,1795232=702575,2963697=730968} 2024-04-26T00:31:33.857 INFO:tasks.rados.rados.0.smithi045.stdout:1905: writing smithi04526579-32 from 497672 to 1059099 tid 1 2024-04-26T00:31:33.880 INFO:tasks.rados.rados.0.smithi045.stdout:1905: writing smithi04526579-32 from 1795232 to 2497807 tid 2 2024-04-26T00:31:33.898 INFO:tasks.rados.rados.0.smithi045.stdout:1905: writing smithi04526579-32 from 2963697 to 3694665 tid 3 2024-04-26T00:31:33.898 INFO:tasks.rados.rados.0.smithi045.stdout:1906: copy_from oid 29 from oid 23 current snap is 219 2024-04-26T00:31:33.899 INFO:tasks.rados.rados.0.smithi045.stdout:1904: finishing rollback tid 1 to smithi04526579-9 2024-04-26T00:31:33.899 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 887 (ObjNum 129 snap 0 seq_num 2241759040) dirty dne 2024-04-26T00:31:33.899 INFO:tasks.rados.rados.0.smithi045.stdout:1903: expect (ObjNum 582 snap 180 seq_num 582) 2024-04-26T00:31:34.079 INFO:tasks.rados.rados.0.smithi045.stdout:1905: finishing write tid 1 to smithi04526579-32 2024-04-26T00:31:34.079 INFO:tasks.rados.rados.0.smithi045.stdout:1905: finishing write tid 2 to smithi04526579-32 2024-04-26T00:31:34.079 INFO:tasks.rados.rados.0.smithi045.stdout:1905: finishing write tid 3 to smithi04526579-32 2024-04-26T00:31:34.080 INFO:tasks.rados.rados.0.smithi045.stdout:1905: finishing write tid 4 to smithi04526579-32 2024-04-26T00:31:34.080 INFO:tasks.rados.rados.0.smithi045.stdout:1905: finishing write tid 5 to smithi04526579-32 2024-04-26T00:31:34.080 INFO:tasks.rados.rados.0.smithi045.stdout:1905: oid 32 updating version 0 to 1226 2024-04-26T00:31:34.080 INFO:tasks.rados.rados.0.smithi045.stdout:1905: oid 32 updating version 1226 to 1228 2024-04-26T00:31:34.080 INFO:tasks.rados.rados.0.smithi045.stdout:1905: oid 32 version 1228 is already newer than 1227 2024-04-26T00:31:34.080 INFO:tasks.rados.rados.0.smithi045.stdout:1905: oid 32 updating version 1228 to 1229 2024-04-26T00:31:34.080 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 1229 (ObjNum 696 snap 219 seq_num 696) dirty exists 2024-04-26T00:31:34.080 INFO:tasks.rados.rados.0.smithi045.stdout:1905: left oid 32 (ObjNum 696 snap 219 seq_num 696) 2024-04-26T00:31:34.080 INFO:tasks.rados.rados.0.smithi045.stdout:1906: finishing copy_from racing read to smithi04526579-29 2024-04-26T00:31:34.080 INFO:tasks.rados.rados.0.smithi045.stdout:1906: finishing copy_from to smithi04526579-29 2024-04-26T00:31:34.080 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 1060 (ObjNum 681 snap 213 seq_num 681) dirty exists 2024-04-26T00:31:34.080 INFO:tasks.rados.rados.0.smithi045.stdout:1903: done (3 left) 2024-04-26T00:31:34.080 INFO:tasks.rados.rados.0.smithi045.stdout:1904: done (2 left) 2024-04-26T00:31:34.080 INFO:tasks.rados.rados.0.smithi045.stdout:1905: done (1 left) 2024-04-26T00:31:34.080 INFO:tasks.rados.rados.0.smithi045.stdout:1906: done (0 left) 2024-04-26T00:31:34.081 INFO:tasks.rados.rados.0.smithi045.stdout:1907: delete oid 48 current snap is 219 2024-04-26T00:31:34.084 INFO:tasks.rados.rados.0.smithi045.stdout:1907: done (0 left) 2024-04-26T00:31:34.084 INFO:tasks.rados.rados.0.smithi045.stdout:1908: write (excl) oid 34 current snap is 219 2024-04-26T00:31:34.084 INFO:tasks.rados.rados.0.smithi045.stdout:1908: seq_num 698 ranges {531286=711925,1912196=706244,3018543=712743} 2024-04-26T00:31:34.105 INFO:tasks.rados.rados.0.smithi045.stdout:1908: writing smithi04526579-34 from 531286 to 1243211 tid 1 2024-04-26T00:31:34.127 INFO:tasks.rados.rados.0.smithi045.stdout:1908: writing smithi04526579-34 from 1912196 to 2618440 tid 2 2024-04-26T00:31:34.143 INFO:tasks.rados.rados.0.smithi045.stdout:1908: writing smithi04526579-34 from 3018543 to 3731286 tid 3 2024-04-26T00:31:34.143 INFO:tasks.rados.rados.0.smithi045.stdout:1909: read oid 10 snap 202 2024-04-26T00:31:34.143 INFO:tasks.rados.rados.0.smithi045.stdout:1909: expect (ObjNum 573 snap 176 seq_num 573) 2024-04-26T00:31:34.144 INFO:tasks.rados.rados.0.smithi045.stdout:1910: snap_remove snap 215 2024-04-26T00:31:34.616 INFO:tasks.rados.rados.0.smithi045.stdout:1910: done (2 left) 2024-04-26T00:31:34.617 INFO:tasks.rados.rados.0.smithi045.stdout:1911: write (excl) oid 48 current snap is 219 2024-04-26T00:31:34.617 INFO:tasks.rados.rados.0.smithi045.stdout:1911: seq_num 699 ranges {548093=787174,1970678=449902,3045966=681137} 2024-04-26T00:31:34.632 INFO:tasks.rados.rados.0.smithi045.stdout:1911: writing smithi04526579-48 from 548093 to 1335267 tid 1 2024-04-26T00:31:34.647 INFO:tasks.rados.rados.0.smithi045.stdout:1911: writing smithi04526579-48 from 1970678 to 2420580 tid 2 2024-04-26T00:31:34.664 INFO:tasks.rados.rados.0.smithi045.stdout:1911: writing smithi04526579-48 from 3045966 to 3727103 tid 3 2024-04-26T00:31:34.664 INFO:tasks.rados.rados.0.smithi045.stdout:1912: read oid 20 snap -1 2024-04-26T00:31:34.664 INFO:tasks.rados.rados.0.smithi045.stdout:1912: expect (ObjNum 542 snap 167 seq_num 542) 2024-04-26T00:31:34.664 INFO:tasks.rados.rados.0.smithi045.stdout:1913: rollback oid 50 current snap is 219 2024-04-26T00:31:34.664 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 50 to 140 2024-04-26T00:31:34.664 INFO:tasks.rados.rados.0.smithi045.stdout:1908: finishing write tid 1 to smithi04526579-34 2024-04-26T00:31:34.664 INFO:tasks.rados.rados.0.smithi045.stdout:1908: finishing write tid 2 to smithi04526579-34 2024-04-26T00:31:34.664 INFO:tasks.rados.rados.0.smithi045.stdout:1908: finishing write tid 3 to smithi04526579-34 2024-04-26T00:31:34.664 INFO:tasks.rados.rados.0.smithi045.stdout:1908: finishing write tid 4 to smithi04526579-34 2024-04-26T00:31:34.664 INFO:tasks.rados.rados.0.smithi045.stdout:1908: finishing write tid 5 to smithi04526579-34 2024-04-26T00:31:34.664 INFO:tasks.rados.rados.0.smithi045.stdout:1908: oid 34 updating version 0 to 1110 2024-04-26T00:31:34.664 INFO:tasks.rados.rados.0.smithi045.stdout:1908: oid 34 version 1110 is already newer than 1109 2024-04-26T00:31:34.665 INFO:tasks.rados.rados.0.smithi045.stdout:1908: oid 34 version 1110 is already newer than 1108 2024-04-26T00:31:34.665 INFO:tasks.rados.rados.0.smithi045.stdout:1908: oid 34 version 1110 is already newer than 1107 2024-04-26T00:31:34.665 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 1110 (ObjNum 698 snap 219 seq_num 698) dirty exists 2024-04-26T00:31:34.665 INFO:tasks.rados.rados.0.smithi045.stdout:1908: left oid 34 (ObjNum 698 snap 219 seq_num 698) 2024-04-26T00:31:34.665 INFO:tasks.rados.rados.0.smithi045.stdout:1909: expect (ObjNum 573 snap 176 seq_num 573) 2024-04-26T00:31:34.822 INFO:tasks.rados.rados.0.smithi045.stdout:1911: finishing write tid 1 to smithi04526579-48 2024-04-26T00:31:34.822 INFO:tasks.rados.rados.0.smithi045.stdout:1911: finishing write tid 2 to smithi04526579-48 2024-04-26T00:31:34.822 INFO:tasks.rados.rados.0.smithi045.stdout:1911: finishing write tid 3 to smithi04526579-48 2024-04-26T00:31:34.823 INFO:tasks.rados.rados.0.smithi045.stdout:1911: finishing write tid 4 to smithi04526579-48 2024-04-26T00:31:34.823 INFO:tasks.rados.rados.0.smithi045.stdout:1911: finishing write tid 5 to smithi04526579-48 2024-04-26T00:31:34.823 INFO:tasks.rados.rados.0.smithi045.stdout:1911: oid 48 updating version 0 to 1116 2024-04-26T00:31:34.823 INFO:tasks.rados.rados.0.smithi045.stdout:1911: oid 48 version 1116 is already newer than 1111 2024-04-26T00:31:34.823 INFO:tasks.rados.rados.0.smithi045.stdout:1911: oid 48 updating version 1116 to 1117 2024-04-26T00:31:34.823 INFO:tasks.rados.rados.0.smithi045.stdout:1911: oid 48 updating version 1117 to 1118 2024-04-26T00:31:34.823 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 1118 (ObjNum 699 snap 219 seq_num 699) dirty exists 2024-04-26T00:31:34.823 INFO:tasks.rados.rados.0.smithi045.stdout:1911: left oid 48 (ObjNum 699 snap 219 seq_num 699) 2024-04-26T00:31:34.823 INFO:tasks.rados.rados.0.smithi045.stdout:1912: expect (ObjNum 542 snap 167 seq_num 542) 2024-04-26T00:31:34.968 INFO:tasks.rados.rados.0.smithi045.stdout:1913: finishing rollback tid 0 to smithi04526579-50 2024-04-26T00:31:34.968 INFO:tasks.rados.rados.0.smithi045.stdout:1913: finishing rollback tid 1 to smithi04526579-50 2024-04-26T00:31:34.968 INFO:tasks.rados.rados.0.smithi045.stdout:1913: finishing rollback tid 2 to smithi04526579-50 2024-04-26T00:31:34.968 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 1065 (ObjNum 136 snap 30 seq_num 136) dirty exists 2024-04-26T00:31:34.968 INFO:tasks.rados.rados.0.smithi045.stdout:1908: done (4 left) 2024-04-26T00:31:34.968 INFO:tasks.rados.rados.0.smithi045.stdout:1909: done (3 left) 2024-04-26T00:31:34.968 INFO:tasks.rados.rados.0.smithi045.stdout:1911: done (2 left) 2024-04-26T00:31:34.968 INFO:tasks.rados.rados.0.smithi045.stdout:1912: done (1 left) 2024-04-26T00:31:34.968 INFO:tasks.rados.rados.0.smithi045.stdout:1913: done (0 left) 2024-04-26T00:31:34.968 INFO:tasks.rados.rados.0.smithi045.stdout:1914: snap_create 2024-04-26T00:31:35.131 INFO:tasks.rados.rados.0.smithi045.stdout:1914: done (0 left) 2024-04-26T00:31:35.132 INFO:tasks.rados.rados.0.smithi045.stdout:1915: snap_remove snap 118 2024-04-26T00:31:36.130 INFO:tasks.rados.rados.0.smithi045.stdout:1915: done (0 left) 2024-04-26T00:31:36.130 INFO:tasks.rados.rados.0.smithi045.stdout:1916: delete oid 35 current snap is 220 2024-04-26T00:31:36.157 INFO:tasks.rados.rados.0.smithi045.stdout:1916: done (0 left) 2024-04-26T00:31:36.158 INFO:tasks.rados.rados.0.smithi045.stdout:1917: snap_remove snap 170 2024-04-26T00:31:37.143 INFO:tasks.rados.rados.0.smithi045.stdout:1917: done (0 left) 2024-04-26T00:31:37.144 INFO:tasks.rados.rados.0.smithi045.stdout:1918: write oid 40 current snap is 220 2024-04-26T00:31:37.144 INFO:tasks.rados.rados.0.smithi045.stdout:1918: seq_num 701 ranges {581707=654025,1687642=453571,2700812=703681} 2024-04-26T00:31:37.163 INFO:tasks.rados.rados.0.smithi045.stdout:1918: writing smithi04526579-40 from 581707 to 1235732 tid 1 2024-04-26T00:31:37.181 INFO:tasks.rados.rados.0.smithi045.stdout:1918: writing smithi04526579-40 from 1687642 to 2141213 tid 2 2024-04-26T00:31:37.202 INFO:tasks.rados.rados.0.smithi045.stdout:1918: writing smithi04526579-40 from 2700812 to 3404493 tid 3 2024-04-26T00:31:37.202 INFO:tasks.rados.rados.0.smithi045.stdout:1919: snap_create 2024-04-26T00:31:37.202 INFO:tasks.rados.rados.0.smithi045.stdout:1918: finishing write tid 1 to smithi04526579-40 2024-04-26T00:31:37.202 INFO:tasks.rados.rados.0.smithi045.stdout:1918: finishing write tid 2 to smithi04526579-40 2024-04-26T00:31:37.209 INFO:tasks.rados.rados.0.smithi045.stdout:1918: finishing write tid 3 to smithi04526579-40 2024-04-26T00:31:37.210 INFO:tasks.rados.rados.0.smithi045.stdout:1918: finishing write tid 4 to smithi04526579-40 2024-04-26T00:31:37.211 INFO:tasks.rados.rados.0.smithi045.stdout:1918: finishing write tid 5 to smithi04526579-40 2024-04-26T00:31:37.211 INFO:tasks.rados.rados.0.smithi045.stdout:1918: oid 40 updating version 0 to 1062 2024-04-26T00:31:37.211 INFO:tasks.rados.rados.0.smithi045.stdout:1918: oid 40 version 1062 is already newer than 1061 2024-04-26T00:31:37.211 INFO:tasks.rados.rados.0.smithi045.stdout:1918: oid 40 updating version 1062 to 1063 2024-04-26T00:31:37.211 INFO:tasks.rados.rados.0.smithi045.stdout:1918: oid 40 updating version 1063 to 1064 2024-04-26T00:31:37.211 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 1064 (ObjNum 701 snap 220 seq_num 701) dirty exists 2024-04-26T00:31:37.211 INFO:tasks.rados.rados.0.smithi045.stdout:1918: left oid 40 (ObjNum 701 snap 220 seq_num 701) 2024-04-26T00:31:38.151 INFO:tasks.rados.rados.0.smithi045.stdout:1918: done (1 left) 2024-04-26T00:31:38.151 INFO:tasks.rados.rados.0.smithi045.stdout:1919: done (0 left) 2024-04-26T00:31:38.151 INFO:tasks.rados.rados.0.smithi045.stdout:1920: delete oid 13 current snap is 221 2024-04-26T00:31:38.165 INFO:tasks.rados.rados.0.smithi045.stdout:1920: done (0 left) 2024-04-26T00:31:38.166 INFO:tasks.rados.rados.0.smithi045.stdout:1921: read oid 6 snap -1 2024-04-26T00:31:38.166 INFO:tasks.rados.rados.0.smithi045.stdout:1921: expect deleted 2024-04-26T00:31:38.166 INFO:tasks.rados.rados.0.smithi045.stdout:1922: read oid 42 snap -1 2024-04-26T00:31:38.166 INFO:tasks.rados.rados.0.smithi045.stdout:1922: expect deleted 2024-04-26T00:31:38.166 INFO:tasks.rados.rados.0.smithi045.stdout:1923: snap_remove snap 187 2024-04-26T00:31:39.158 INFO:tasks.rados.rados.0.smithi045.stdout:1923: done (2 left) 2024-04-26T00:31:39.158 INFO:tasks.rados.rados.0.smithi045.stdout:1924: copy_from oid 32 from oid 10 current snap is 221 2024-04-26T00:31:39.159 INFO:tasks.rados.rados.0.smithi045.stdout:1921: done (2 left) 2024-04-26T00:31:39.159 INFO:tasks.rados.rados.0.smithi045.stdout:1922: done (1 left) 2024-04-26T00:31:39.159 INFO:tasks.rados.rados.0.smithi045.stdout:1925: copy_from oid 6 from oid 30 current snap is 221 2024-04-26T00:31:39.159 INFO:tasks.rados.rados.0.smithi045.stdout:1926: write (excl) oid 39 current snap is 221 2024-04-26T00:31:39.159 INFO:tasks.rados.rados.0.smithi045.stdout:1926: seq_num 703 ranges {615321=404523,1804606=457240,2755658=726225} 2024-04-26T00:31:39.170 INFO:tasks.rados.rados.0.smithi045.stdout:1926: writing smithi04526579-39 from 615321 to 1019844 tid 1 2024-04-26T00:31:39.182 INFO:tasks.rados.rados.0.smithi045.stdout:1926: writing smithi04526579-39 from 1804606 to 2261846 tid 2 2024-04-26T00:31:39.194 INFO:tasks.rados.rados.0.smithi045.stdout:1926: writing smithi04526579-39 from 2755658 to 3481883 tid 3 2024-04-26T00:31:39.194 INFO:tasks.rados.rados.0.smithi045.stdout:1925: finishing copy_from to smithi04526579-6 2024-04-26T00:31:39.194 INFO:tasks.rados.rados.0.smithi045.stdout:1925: got expected ENOENT (src dne) 2024-04-26T00:31:39.194 INFO:tasks.rados.rados.0.smithi045.stdout:1925: finishing copy_from racing read to smithi04526579-6 2024-04-26T00:31:39.194 INFO:tasks.rados.rados.0.smithi045.stdout:1927: read oid 31 snap -1 2024-04-26T00:31:39.194 INFO:tasks.rados.rados.0.smithi045.stdout:1927: expect (ObjNum 644 snap 202 seq_num 644) 2024-04-26T00:31:39.194 INFO:tasks.rados.rados.0.smithi045.stdout:1925: done (3 left) 2024-04-26T00:31:39.194 INFO:tasks.rados.rados.0.smithi045.stdout:1928: read oid 19 snap 220 2024-04-26T00:31:39.194 INFO:tasks.rados.rados.0.smithi045.stdout:1928: expect (ObjNum 691 snap 216 seq_num 691) 2024-04-26T00:31:39.194 INFO:tasks.rados.rados.0.smithi045.stdout:1929: snap_remove snap 216 2024-04-26T00:31:40.131 INFO:tasks.rados.rados.0.smithi045.stdout:1929: done (4 left) 2024-04-26T00:31:40.132 INFO:tasks.rados.rados.0.smithi045.stdout:1930: read oid 15 snap -1 2024-04-26T00:31:40.132 INFO:tasks.rados.rados.0.smithi045.stdout:1930: expect (ObjNum 685 snap 213 seq_num 685) 2024-04-26T00:31:40.132 INFO:tasks.rados.rados.0.smithi045.stdout:1926: finishing write tid 1 to smithi04526579-39 2024-04-26T00:31:40.133 INFO:tasks.rados.rados.0.smithi045.stdout:1927: expect (ObjNum 644 snap 202 seq_num 644) 2024-04-26T00:31:40.280 INFO:tasks.rados.rados.0.smithi045.stdout:1926: finishing write tid 2 to smithi04526579-39 2024-04-26T00:31:40.280 INFO:tasks.rados.rados.0.smithi045.stdout:1924: finishing copy_from racing read to smithi04526579-32 2024-04-26T00:31:40.280 INFO:tasks.rados.rados.0.smithi045.stdout:1926: finishing write tid 3 to smithi04526579-39 2024-04-26T00:31:40.280 INFO:tasks.rados.rados.0.smithi045.stdout:1926: finishing write tid 4 to smithi04526579-39 2024-04-26T00:31:40.280 INFO:tasks.rados.rados.0.smithi045.stdout:1926: finishing write tid 5 to smithi04526579-39 2024-04-26T00:31:40.280 INFO:tasks.rados.rados.0.smithi045.stdout:1926: oid 39 updating version 0 to 1083 2024-04-26T00:31:40.281 INFO:tasks.rados.rados.0.smithi045.stdout:1926: oid 39 updating version 1083 to 1086 2024-04-26T00:31:40.281 INFO:tasks.rados.rados.0.smithi045.stdout:1926: oid 39 version 1086 is already newer than 1084 2024-04-26T00:31:40.281 INFO:tasks.rados.rados.0.smithi045.stdout:1926: oid 39 version 1086 is already newer than 1085 2024-04-26T00:31:40.281 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 1086 (ObjNum 703 snap 221 seq_num 703) dirty exists 2024-04-26T00:31:40.281 INFO:tasks.rados.rados.0.smithi045.stdout:1926: left oid 39 (ObjNum 703 snap 221 seq_num 703) 2024-04-26T00:31:40.281 INFO:tasks.rados.rados.0.smithi045.stdout:1924: finishing copy_from to smithi04526579-32 2024-04-26T00:31:40.281 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 1055 (ObjNum 573 snap 176 seq_num 573) dirty exists 2024-04-26T00:31:40.281 INFO:tasks.rados.rados.0.smithi045.stdout:1928: expect (ObjNum 691 snap 216 seq_num 691) 2024-04-26T00:31:40.512 INFO:tasks.rados.rados.0.smithi045.stdout:1930: expect (ObjNum 685 snap 213 seq_num 685) 2024-04-26T00:31:40.629 INFO:tasks.rados.rados.0.smithi045.stdout:1924: done (4 left) 2024-04-26T00:31:40.629 INFO:tasks.rados.rados.0.smithi045.stdout:1926: done (3 left) 2024-04-26T00:31:40.629 INFO:tasks.rados.rados.0.smithi045.stdout:1927: done (2 left) 2024-04-26T00:31:40.629 INFO:tasks.rados.rados.0.smithi045.stdout:1928: done (1 left) 2024-04-26T00:31:40.629 INFO:tasks.rados.rados.0.smithi045.stdout:1930: done (0 left) 2024-04-26T00:31:40.629 INFO:tasks.rados.rados.0.smithi045.stdout:1931: snap_remove snap 218 2024-04-26T00:31:41.185 INFO:tasks.rados.rados.0.smithi045.stdout:1931: done (0 left) 2024-04-26T00:31:41.186 INFO:tasks.rados.rados.0.smithi045.stdout:1932: rollback oid 21 current snap is 221 2024-04-26T00:31:41.186 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 21 to 202 2024-04-26T00:31:41.186 INFO:tasks.rados.rados.0.smithi045.stdout:1933: copy_from oid 12 from oid 18 current snap is 221 2024-04-26T00:31:41.186 INFO:tasks.rados.rados.0.smithi045.stdout:1934: write oid 30 current snap is 221 2024-04-26T00:31:41.186 INFO:tasks.rados.rados.0.smithi045.stdout:1934: seq_num 704 ranges {632128=479772,1746735=600898,3066728=653850} 2024-04-26T00:31:41.203 INFO:tasks.rados.rados.0.smithi045.stdout:1934: writing smithi04526579-30 from 632128 to 1111900 tid 1 2024-04-26T00:31:41.225 INFO:tasks.rados.rados.0.smithi045.stdout:1934: writing smithi04526579-30 from 1746735 to 2347633 tid 2 2024-04-26T00:31:41.243 INFO:tasks.rados.rados.0.smithi045.stdout:1934: writing smithi04526579-30 from 3066728 to 3720578 tid 3 2024-04-26T00:31:41.243 INFO:tasks.rados.rados.0.smithi045.stdout:1935: snap_remove snap 220 2024-04-26T00:31:42.198 INFO:tasks.rados.rados.0.smithi045.stdout:1935: done (3 left) 2024-04-26T00:31:42.198 INFO:tasks.rados.rados.0.smithi045.stdout:1936: snap_remove snap 204 2024-04-26T00:31:43.199 INFO:tasks.rados.rados.0.smithi045.stdout:1936: done (3 left) 2024-04-26T00:31:43.199 INFO:tasks.rados.rados.0.smithi045.stdout:1937: read oid 20 snap -1 2024-04-26T00:31:43.199 INFO:tasks.rados.rados.0.smithi045.stdout:1937: expect (ObjNum 542 snap 167 seq_num 542) 2024-04-26T00:31:43.199 INFO:tasks.rados.rados.0.smithi045.stdout:1938: snap_remove snap 200 2024-04-26T00:31:44.200 INFO:tasks.rados.rados.0.smithi045.stdout:1938: done (4 left) 2024-04-26T00:31:44.200 INFO:tasks.rados.rados.0.smithi045.stdout:1939: snap_remove snap 194 2024-04-26T00:31:45.133 INFO:tasks.rados.rados.0.smithi045.stdout:1939: done (4 left) 2024-04-26T00:31:45.134 INFO:tasks.rados.rados.0.smithi045.stdout:1940: copy_from oid 9 from oid 37 current snap is 221 2024-04-26T00:31:45.134 INFO:tasks.rados.rados.0.smithi045.stdout:1932: finishing rollback tid 0 to smithi04526579-21 2024-04-26T00:31:45.134 INFO:tasks.rados.rados.0.smithi045.stdout:1932: finishing rollback tid 1 to smithi04526579-21 2024-04-26T00:31:45.134 INFO:tasks.rados.rados.0.smithi045.stdout:1932: finishing rollback tid 2 to smithi04526579-21 2024-04-26T00:31:45.134 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 1052 (ObjNum 595 snap 184 seq_num 595) dirty exists 2024-04-26T00:31:45.134 INFO:tasks.rados.rados.0.smithi045.stdout:1934: finishing write tid 1 to smithi04526579-30 2024-04-26T00:31:45.135 INFO:tasks.rados.rados.0.smithi045.stdout:1933: finishing copy_from racing read to smithi04526579-12 2024-04-26T00:31:45.135 INFO:tasks.rados.rados.0.smithi045.stdout:1933: finishing copy_from to smithi04526579-12 2024-04-26T00:31:45.135 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 1429 (ObjNum 551 snap 168 seq_num 551) dirty exists 2024-04-26T00:31:45.135 INFO:tasks.rados.rados.0.smithi045.stdout:1934: finishing write tid 2 to smithi04526579-30 2024-04-26T00:31:45.135 INFO:tasks.rados.rados.0.smithi045.stdout:1934: finishing write tid 3 to smithi04526579-30 2024-04-26T00:31:45.135 INFO:tasks.rados.rados.0.smithi045.stdout:1934: finishing write tid 4 to smithi04526579-30 2024-04-26T00:31:45.136 INFO:tasks.rados.rados.0.smithi045.stdout:1934: finishing write tid 5 to smithi04526579-30 2024-04-26T00:31:45.136 INFO:tasks.rados.rados.0.smithi045.stdout:1934: oid 30 updating version 0 to 1052 2024-04-26T00:31:45.136 INFO:tasks.rados.rados.0.smithi045.stdout:1934: oid 30 updating version 1052 to 1055 2024-04-26T00:31:45.136 INFO:tasks.rados.rados.0.smithi045.stdout:1934: oid 30 version 1055 is already newer than 1054 2024-04-26T00:31:45.136 INFO:tasks.rados.rados.0.smithi045.stdout:1934: oid 30 version 1055 is already newer than 1053 2024-04-26T00:31:45.136 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 1055 (ObjNum 704 snap 221 seq_num 704) dirty exists 2024-04-26T00:31:45.136 INFO:tasks.rados.rados.0.smithi045.stdout:1934: left oid 30 (ObjNum 704 snap 221 seq_num 704) 2024-04-26T00:31:45.136 INFO:tasks.rados.rados.0.smithi045.stdout:1937: expect (ObjNum 542 snap 167 seq_num 542) 2024-04-26T00:31:45.317 INFO:tasks.rados.rados.0.smithi045.stdout:1940: finishing copy_from racing read to smithi04526579-9 2024-04-26T00:31:45.318 INFO:tasks.rados.rados.0.smithi045.stdout:1940: finishing copy_from to smithi04526579-9 2024-04-26T00:31:45.318 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 1509 (ObjNum 669 snap 210 seq_num 669) dirty exists 2024-04-26T00:31:45.318 INFO:tasks.rados.rados.0.smithi045.stdout:1932: done (4 left) 2024-04-26T00:31:45.318 INFO:tasks.rados.rados.0.smithi045.stdout:1933: done (3 left) 2024-04-26T00:31:45.318 INFO:tasks.rados.rados.0.smithi045.stdout:1934: done (2 left) 2024-04-26T00:31:45.318 INFO:tasks.rados.rados.0.smithi045.stdout:1937: done (1 left) 2024-04-26T00:31:45.318 INFO:tasks.rados.rados.0.smithi045.stdout:1940: done (0 left) 2024-04-26T00:31:45.318 INFO:tasks.rados.rados.0.smithi045.stdout:1941: delete oid 29 current snap is 221 2024-04-26T00:31:45.326 INFO:tasks.rados.rados.0.smithi045.stdout:1941: done (0 left) 2024-04-26T00:31:45.326 INFO:tasks.rados.rados.0.smithi045.stdout:1942: write (excl) oid 12 current snap is 221 2024-04-26T00:31:45.326 INFO:tasks.rados.rados.0.smithi045.stdout:1942: seq_num 706 ranges {665742=630270,1863699=604567,3121574=676394} 2024-04-26T00:31:45.347 INFO:tasks.rados.rados.0.smithi045.stdout:1942: writing smithi04526579-12 from 665742 to 1296012 tid 1 2024-04-26T00:31:45.367 INFO:tasks.rados.rados.0.smithi045.stdout:1942: writing smithi04526579-12 from 1863699 to 2468266 tid 2 2024-04-26T00:31:45.386 INFO:tasks.rados.rados.0.smithi045.stdout:1942: writing smithi04526579-12 from 3121574 to 3797968 tid 3 2024-04-26T00:31:45.386 INFO:tasks.rados.rados.0.smithi045.stdout:1943: snap_remove snap 217 2024-04-26T00:31:46.233 INFO:tasks.rados.rados.0.smithi045.stdout:1943: done (1 left) 2024-04-26T00:31:46.233 INFO:tasks.rados.rados.0.smithi045.stdout:1944: rollback oid 38 current snap is 221 2024-04-26T00:31:46.233 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 38 to 208 2024-04-26T00:31:46.233 INFO:tasks.rados.rados.0.smithi045.stdout:1942: finishing write tid 1 to smithi04526579-12 2024-04-26T00:31:46.234 INFO:tasks.rados.rados.0.smithi045.stdout:1942: finishing write tid 2 to smithi04526579-12 2024-04-26T00:31:46.234 INFO:tasks.rados.rados.0.smithi045.stdout:1942: finishing write tid 3 to smithi04526579-12 2024-04-26T00:31:46.234 INFO:tasks.rados.rados.0.smithi045.stdout:1942: finishing write tid 4 to smithi04526579-12 2024-04-26T00:31:46.234 INFO:tasks.rados.rados.0.smithi045.stdout:1942: finishing write tid 5 to smithi04526579-12 2024-04-26T00:31:46.234 INFO:tasks.rados.rados.0.smithi045.stdout:1942: oid 12 updating version 0 to 1430 2024-04-26T00:31:46.234 INFO:tasks.rados.rados.0.smithi045.stdout:1942: oid 12 updating version 1430 to 1433 2024-04-26T00:31:46.235 INFO:tasks.rados.rados.0.smithi045.stdout:1942: oid 12 version 1433 is already newer than 1432 2024-04-26T00:31:46.235 INFO:tasks.rados.rados.0.smithi045.stdout:1942: oid 12 version 1433 is already newer than 1431 2024-04-26T00:31:46.235 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 1433 (ObjNum 706 snap 221 seq_num 706) dirty exists 2024-04-26T00:31:46.235 INFO:tasks.rados.rados.0.smithi045.stdout:1942: left oid 12 (ObjNum 706 snap 221 seq_num 706) 2024-04-26T00:31:46.235 INFO:tasks.rados.rados.0.smithi045.stdout:1942: done (1 left) 2024-04-26T00:31:46.236 INFO:tasks.rados.rados.0.smithi045.stdout:1945: write oid 1 current snap is 221 2024-04-26T00:31:46.236 INFO:tasks.rados.rados.0.smithi045.stdout:1945: seq_num 707 ranges {682549=705519,1922181=464578,3148997=487666} 2024-04-26T00:31:46.251 INFO:tasks.rados.rados.0.smithi045.stdout:1945: writing smithi04526579-1 from 682549 to 1388068 tid 1 2024-04-26T00:31:46.264 INFO:tasks.rados.rados.0.smithi045.stdout:1945: writing smithi04526579-1 from 1922181 to 2386759 tid 2 2024-04-26T00:31:46.280 INFO:tasks.rados.rados.0.smithi045.stdout:1945: writing smithi04526579-1 from 3148997 to 3636663 tid 3 2024-04-26T00:31:46.280 INFO:tasks.rados.rados.0.smithi045.stdout:1946: write (excl) oid 8 current snap is 221 2024-04-26T00:31:46.280 INFO:tasks.rados.rados.0.smithi045.stdout:1944: finishing rollback tid 0 to smithi04526579-38 2024-04-26T00:31:46.280 INFO:tasks.rados.rados.0.smithi045.stdout:1944: finishing rollback tid 1 to smithi04526579-38 2024-04-26T00:31:46.280 INFO:tasks.rados.rados.0.smithi045.stdout:1944: finishing rollback tid 2 to smithi04526579-38 2024-04-26T00:31:46.280 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 1089 (ObjNum 582 snap 180 seq_num 582) dirty exists 2024-04-26T00:31:46.280 INFO:tasks.rados.rados.0.smithi045.stdout:1945: finishing write tid 1 to smithi04526579-1 2024-04-26T00:31:46.280 INFO:tasks.rados.rados.0.smithi045.stdout:1945: finishing write tid 2 to smithi04526579-1 2024-04-26T00:31:46.280 INFO:tasks.rados.rados.0.smithi045.stdout:1946: seq_num 708 ranges {699356=497121,1980663=608236,3176420=415291} 2024-04-26T00:31:46.295 INFO:tasks.rados.rados.0.smithi045.stdout:1946: writing smithi04526579-8 from 699356 to 1196477 tid 1 2024-04-26T00:31:46.312 INFO:tasks.rados.rados.0.smithi045.stdout:1946: writing smithi04526579-8 from 1980663 to 2588899 tid 2 2024-04-26T00:31:46.323 INFO:tasks.rados.rados.0.smithi045.stdout:1946: writing smithi04526579-8 from 3176420 to 3591711 tid 3 2024-04-26T00:31:46.323 INFO:tasks.rados.rados.0.smithi045.stdout:1944: done (2 left) 2024-04-26T00:31:46.323 INFO:tasks.rados.rados.0.smithi045.stdout:1947: snap_create 2024-04-26T00:31:46.323 INFO:tasks.rados.rados.0.smithi045.stdout:1945: finishing write tid 3 to smithi04526579-1 2024-04-26T00:31:46.323 INFO:tasks.rados.rados.0.smithi045.stdout:1945: finishing write tid 4 to smithi04526579-1 2024-04-26T00:31:46.323 INFO:tasks.rados.rados.0.smithi045.stdout:1945: finishing write tid 5 to smithi04526579-1 2024-04-26T00:31:46.323 INFO:tasks.rados.rados.0.smithi045.stdout:1945: oid 1 updating version 0 to 1277 2024-04-26T00:31:46.323 INFO:tasks.rados.rados.0.smithi045.stdout:1945: oid 1 updating version 1277 to 1279 2024-04-26T00:31:46.324 INFO:tasks.rados.rados.0.smithi045.stdout:1945: oid 1 version 1279 is already newer than 1278 2024-04-26T00:31:46.324 INFO:tasks.rados.rados.0.smithi045.stdout:1945: oid 1 updating version 1279 to 1280 2024-04-26T00:31:46.324 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 1280 (ObjNum 707 snap 221 seq_num 707) dirty exists 2024-04-26T00:31:46.324 INFO:tasks.rados.rados.0.smithi045.stdout:1945: left oid 1 (ObjNum 707 snap 221 seq_num 707) 2024-04-26T00:31:46.324 INFO:tasks.rados.rados.0.smithi045.stdout:1946: finishing write tid 1 to smithi04526579-8 2024-04-26T00:31:46.332 INFO:tasks.rados.rados.0.smithi045.stdout:1946: finishing write tid 2 to smithi04526579-8 2024-04-26T00:31:46.332 INFO:tasks.rados.rados.0.smithi045.stdout:1946: finishing write tid 3 to smithi04526579-8 2024-04-26T00:31:46.332 INFO:tasks.rados.rados.0.smithi045.stdout:1946: finishing write tid 4 to smithi04526579-8 2024-04-26T00:31:46.333 INFO:tasks.rados.rados.0.smithi045.stdout:1946: finishing write tid 5 to smithi04526579-8 2024-04-26T00:31:46.333 INFO:tasks.rados.rados.0.smithi045.stdout:1946: oid 8 updating version 0 to 1063 2024-04-26T00:31:46.333 INFO:tasks.rados.rados.0.smithi045.stdout:1946: oid 8 updating version 1063 to 1065 2024-04-26T00:31:46.333 INFO:tasks.rados.rados.0.smithi045.stdout:1946: oid 8 updating version 1065 to 1066 2024-04-26T00:31:46.333 INFO:tasks.rados.rados.0.smithi045.stdout:1946: oid 8 version 1066 is already newer than 1064 2024-04-26T00:31:46.334 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 1066 (ObjNum 708 snap 221 seq_num 708) dirty exists 2024-04-26T00:31:46.334 INFO:tasks.rados.rados.0.smithi045.stdout:1946: left oid 8 (ObjNum 708 snap 221 seq_num 708) 2024-04-26T00:31:47.234 INFO:tasks.rados.rados.0.smithi045.stdout:1945: done (2 left) 2024-04-26T00:31:47.234 INFO:tasks.rados.rados.0.smithi045.stdout:1946: done (1 left) 2024-04-26T00:31:47.234 INFO:tasks.rados.rados.0.smithi045.stdout:1947: done (0 left) 2024-04-26T00:31:47.234 INFO:tasks.rados.rados.0.smithi045.stdout:1948: write oid 19 current snap is 222 2024-04-26T00:31:47.234 INFO:tasks.rados.rados.0.smithi045.stdout:1948: seq_num 709 ranges {716163=572370,2039145=468247,3203843=626563} 2024-04-26T00:31:47.245 INFO:tasks.rados.rados.0.smithi045.stdout:1948: writing smithi04526579-19 from 716163 to 1288533 tid 1 2024-04-26T00:31:47.256 INFO:tasks.rados.rados.0.smithi045.stdout:1948: writing smithi04526579-19 from 2039145 to 2507392 tid 2 2024-04-26T00:31:47.268 INFO:tasks.rados.rados.0.smithi045.stdout:1948: writing smithi04526579-19 from 3203843 to 3830406 tid 3 2024-04-26T00:31:47.268 INFO:tasks.rados.rados.0.smithi045.stdout:1949: delete oid 30 current snap is 222 2024-04-26T00:31:47.268 INFO:tasks.rados.rados.0.smithi045.stdout:1948: finishing write tid 1 to smithi04526579-19 2024-04-26T00:31:47.268 INFO:tasks.rados.rados.0.smithi045.stdout:1948: finishing write tid 2 to smithi04526579-19 2024-04-26T00:31:47.276 INFO:tasks.rados.rados.0.smithi045.stdout:1949: done (1 left) 2024-04-26T00:31:47.276 INFO:tasks.rados.rados.0.smithi045.stdout:1950: read oid 37 snap -1 2024-04-26T00:31:47.276 INFO:tasks.rados.rados.0.smithi045.stdout:1950: expect (ObjNum 669 snap 210 seq_num 669) 2024-04-26T00:31:47.276 INFO:tasks.rados.rados.0.smithi045.stdout:1951: read oid 4 snap -1 2024-04-26T00:31:47.276 INFO:tasks.rados.rados.0.smithi045.stdout:1951: expect (ObjNum 406 snap 125 seq_num 406) 2024-04-26T00:31:47.276 INFO:tasks.rados.rados.0.smithi045.stdout:1952: write oid 3 current snap is 222 2024-04-26T00:31:47.276 INFO:tasks.rados.rados.0.smithi045.stdout:1952: seq_num 711 ranges {749777=722868,2156109=471916,3258689=649107} 2024-04-26T00:31:47.295 INFO:tasks.rados.rados.0.smithi045.stdout:1952: writing smithi04526579-3 from 749777 to 1472645 tid 1 2024-04-26T00:31:47.306 INFO:tasks.rados.rados.0.smithi045.stdout:1952: writing smithi04526579-3 from 2156109 to 2628025 tid 2 2024-04-26T00:31:47.317 INFO:tasks.rados.rados.0.smithi045.stdout:1952: writing smithi04526579-3 from 3258689 to 3907796 tid 3 2024-04-26T00:31:47.318 INFO:tasks.rados.rados.0.smithi045.stdout:1953: write oid 33 current snap is 222 2024-04-26T00:31:47.318 INFO:tasks.rados.rados.0.smithi045.stdout:1953: seq_num 712 ranges {766584=798117,2098238=615574,3169759=460379} 2024-04-26T00:31:47.329 INFO:tasks.rados.rados.0.smithi045.stdout:1953: writing smithi04526579-33 from 766584 to 1564701 tid 1 2024-04-26T00:31:47.337 INFO:tasks.rados.rados.0.smithi045.stdout:1953: writing smithi04526579-33 from 2098238 to 2713812 tid 2 2024-04-26T00:31:47.343 INFO:tasks.rados.rados.0.smithi045.stdout:1953: writing smithi04526579-33 from 3169759 to 3630138 tid 3 2024-04-26T00:31:47.343 INFO:tasks.rados.rados.0.smithi045.stdout:1954: write (excl) oid 16 current snap is 222 2024-04-26T00:31:47.343 INFO:tasks.rados.rados.0.smithi045.stdout:1948: finishing write tid 3 to smithi04526579-19 2024-04-26T00:31:47.344 INFO:tasks.rados.rados.0.smithi045.stdout:1954: seq_num 713 ranges {783391=473366,1756720=475585,2797182=788004} 2024-04-26T00:31:47.351 INFO:tasks.rados.rados.0.smithi045.stdout:1954: writing smithi04526579-16 from 783391 to 1256757 tid 1 2024-04-26T00:31:47.357 INFO:tasks.rados.rados.0.smithi045.stdout:1954: writing smithi04526579-16 from 1756720 to 2232305 tid 2 2024-04-26T00:31:47.367 INFO:tasks.rados.rados.0.smithi045.stdout:1954: writing smithi04526579-16 from 2797182 to 3585186 tid 3 2024-04-26T00:31:47.367 INFO:tasks.rados.rados.0.smithi045.stdout:1955: copy_from oid 13 from oid 41 current snap is 222 2024-04-26T00:31:47.367 INFO:tasks.rados.rados.0.smithi045.stdout:1948: finishing write tid 4 to smithi04526579-19 2024-04-26T00:31:47.367 INFO:tasks.rados.rados.0.smithi045.stdout:1948: finishing write tid 5 to smithi04526579-19 2024-04-26T00:31:47.367 INFO:tasks.rados.rados.0.smithi045.stdout:1948: oid 19 updating version 0 to 1285 2024-04-26T00:31:47.367 INFO:tasks.rados.rados.0.smithi045.stdout:1948: oid 19 version 1285 is already newer than 1283 2024-04-26T00:31:47.367 INFO:tasks.rados.rados.0.smithi045.stdout:1948: oid 19 version 1285 is already newer than 1282 2024-04-26T00:31:47.367 INFO:tasks.rados.rados.0.smithi045.stdout:1948: oid 19 version 1285 is already newer than 1284 2024-04-26T00:31:47.367 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 1285 (ObjNum 709 snap 222 seq_num 709) dirty exists 2024-04-26T00:31:47.367 INFO:tasks.rados.rados.0.smithi045.stdout:1948: left oid 19 (ObjNum 709 snap 222 seq_num 709) 2024-04-26T00:31:47.367 INFO:tasks.rados.rados.0.smithi045.stdout:1948: done (6 left) 2024-04-26T00:31:47.367 INFO:tasks.rados.rados.0.smithi045.stdout:1956: read oid 49 snap 199 2024-04-26T00:31:47.368 INFO:tasks.rados.rados.0.smithi045.stdout:1956: expect (ObjNum 591 snap 183 seq_num 591) 2024-04-26T00:31:47.368 INFO:tasks.rados.rados.0.smithi045.stdout:1957: delete oid 12 current snap is 222 2024-04-26T00:31:47.368 INFO:tasks.rados.rados.0.smithi045.stdout:1950: expect (ObjNum 669 snap 210 seq_num 669) 2024-04-26T00:31:47.628 INFO:tasks.rados.rados.0.smithi045.stdout:1951: expect (ObjNum 406 snap 125 seq_num 406) 2024-04-26T00:31:47.759 INFO:tasks.rados.rados.0.smithi045.stdout:1952: finishing write tid 1 to smithi04526579-3 2024-04-26T00:31:47.759 INFO:tasks.rados.rados.0.smithi045.stdout:1952: finishing write tid 2 to smithi04526579-3 2024-04-26T00:31:47.759 INFO:tasks.rados.rados.0.smithi045.stdout:1952: finishing write tid 3 to smithi04526579-3 2024-04-26T00:31:47.759 INFO:tasks.rados.rados.0.smithi045.stdout:1952: finishing write tid 4 to smithi04526579-3 2024-04-26T00:31:47.759 INFO:tasks.rados.rados.0.smithi045.stdout:1952: finishing write tid 5 to smithi04526579-3 2024-04-26T00:31:47.759 INFO:tasks.rados.rados.0.smithi045.stdout:1952: oid 3 updating version 0 to 1169 2024-04-26T00:31:47.759 INFO:tasks.rados.rados.0.smithi045.stdout:1952: oid 3 version 1169 is already newer than 1167 2024-04-26T00:31:47.759 INFO:tasks.rados.rados.0.smithi045.stdout:1952: oid 3 updating version 1169 to 1170 2024-04-26T00:31:47.759 INFO:tasks.rados.rados.0.smithi045.stdout:1952: oid 3 version 1170 is already newer than 1168 2024-04-26T00:31:47.759 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 1170 (ObjNum 711 snap 222 seq_num 711) dirty exists 2024-04-26T00:31:47.759 INFO:tasks.rados.rados.0.smithi045.stdout:1952: left oid 3 (ObjNum 711 snap 222 seq_num 711) 2024-04-26T00:31:47.759 INFO:tasks.rados.rados.0.smithi045.stdout:1953: finishing write tid 1 to smithi04526579-33 2024-04-26T00:31:47.760 INFO:tasks.rados.rados.0.smithi045.stdout:1953: finishing write tid 2 to smithi04526579-33 2024-04-26T00:31:47.760 INFO:tasks.rados.rados.0.smithi045.stdout:1953: finishing write tid 3 to smithi04526579-33 2024-04-26T00:31:47.760 INFO:tasks.rados.rados.0.smithi045.stdout:1953: finishing write tid 4 to smithi04526579-33 2024-04-26T00:31:47.760 INFO:tasks.rados.rados.0.smithi045.stdout:1953: finishing write tid 5 to smithi04526579-33 2024-04-26T00:31:47.760 INFO:tasks.rados.rados.0.smithi045.stdout:1953: oid 33 updating version 0 to 1652 2024-04-26T00:31:47.760 INFO:tasks.rados.rados.0.smithi045.stdout:1953: oid 33 version 1652 is already newer than 1651 2024-04-26T00:31:47.760 INFO:tasks.rados.rados.0.smithi045.stdout:1953: oid 33 version 1652 is already newer than 1650 2024-04-26T00:31:47.760 INFO:tasks.rados.rados.0.smithi045.stdout:1953: oid 33 updating version 1652 to 1653 2024-04-26T00:31:47.760 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 1653 (ObjNum 712 snap 222 seq_num 712) dirty exists 2024-04-26T00:31:47.760 INFO:tasks.rados.rados.0.smithi045.stdout:1953: left oid 33 (ObjNum 712 snap 222 seq_num 712) 2024-04-26T00:31:47.760 INFO:tasks.rados.rados.0.smithi045.stdout:1954: finishing write tid 1 to smithi04526579-16 2024-04-26T00:31:47.760 INFO:tasks.rados.rados.0.smithi045.stdout:1954: finishing write tid 2 to smithi04526579-16 2024-04-26T00:31:47.760 INFO:tasks.rados.rados.0.smithi045.stdout:1955: finishing copy_from to smithi04526579-13 2024-04-26T00:31:47.760 INFO:tasks.rados.rados.0.smithi045.stdout:1955: got expected ENOENT (src dne) 2024-04-26T00:31:47.760 INFO:tasks.rados.rados.0.smithi045.stdout:1955: finishing copy_from racing read to smithi04526579-13 2024-04-26T00:31:47.761 INFO:tasks.rados.rados.0.smithi045.stdout:1954: finishing write tid 3 to smithi04526579-16 2024-04-26T00:31:47.761 INFO:tasks.rados.rados.0.smithi045.stdout:1954: finishing write tid 4 to smithi04526579-16 2024-04-26T00:31:47.761 INFO:tasks.rados.rados.0.smithi045.stdout:1954: finishing write tid 5 to smithi04526579-16 2024-04-26T00:31:47.761 INFO:tasks.rados.rados.0.smithi045.stdout:1954: oid 16 updating version 0 to 1127 2024-04-26T00:31:47.761 INFO:tasks.rados.rados.0.smithi045.stdout:1954: oid 16 version 1127 is already newer than 1125 2024-04-26T00:31:47.761 INFO:tasks.rados.rados.0.smithi045.stdout:1954: oid 16 version 1127 is already newer than 1126 2024-04-26T00:31:47.761 INFO:tasks.rados.rados.0.smithi045.stdout:1954: oid 16 version 1127 is already newer than 1124 2024-04-26T00:31:47.761 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 1127 (ObjNum 713 snap 222 seq_num 713) dirty exists 2024-04-26T00:31:47.761 INFO:tasks.rados.rados.0.smithi045.stdout:1954: left oid 16 (ObjNum 713 snap 222 seq_num 713) 2024-04-26T00:31:47.761 INFO:tasks.rados.rados.0.smithi045.stdout:1956: expect (ObjNum 591 snap 183 seq_num 591) 2024-04-26T00:31:48.024 INFO:tasks.rados.rados.0.smithi045.stdout:1950: done (7 left) 2024-04-26T00:31:48.024 INFO:tasks.rados.rados.0.smithi045.stdout:1951: done (6 left) 2024-04-26T00:31:48.024 INFO:tasks.rados.rados.0.smithi045.stdout:1952: done (5 left) 2024-04-26T00:31:48.024 INFO:tasks.rados.rados.0.smithi045.stdout:1953: done (4 left) 2024-04-26T00:31:48.025 INFO:tasks.rados.rados.0.smithi045.stdout:1954: done (3 left) 2024-04-26T00:31:48.025 INFO:tasks.rados.rados.0.smithi045.stdout:1955: done (2 left) 2024-04-26T00:31:48.025 INFO:tasks.rados.rados.0.smithi045.stdout:1956: done (1 left) 2024-04-26T00:31:48.025 INFO:tasks.rados.rados.0.smithi045.stdout:1957: done (0 left) 2024-04-26T00:31:48.025 INFO:tasks.rados.rados.0.smithi045.stdout:1958: delete oid 14 current snap is 222 2024-04-26T00:31:48.034 INFO:tasks.rados.rados.0.smithi045.stdout:1958: done (0 left) 2024-04-26T00:31:48.034 INFO:tasks.rados.rados.0.smithi045.stdout:1959: write (excl) oid 16 current snap is 222 2024-04-26T00:31:48.034 INFO:tasks.rados.rados.0.smithi045.stdout:1959: seq_num 716 ranges {433812=415466,1648519=385293} 2024-04-26T00:31:48.047 INFO:tasks.rados.rados.0.smithi045.stdout:1959: writing smithi04526579-16 from 433812 to 849278 tid 1 2024-04-26T00:31:48.067 INFO:tasks.rados.rados.0.smithi045.stdout:1959: writing smithi04526579-16 from 1648519 to 2033812 tid 2 2024-04-26T00:31:48.067 INFO:tasks.rados.rados.0.smithi045.stdout:1960: write oid 35 current snap is 222 2024-04-26T00:31:48.067 INFO:tasks.rados.rados.0.smithi045.stdout:1960: seq_num 717 ranges {450619=490715,1590648=459971} 2024-04-26T00:31:48.082 INFO:tasks.rados.rados.0.smithi045.stdout:1960: writing smithi04526579-35 from 450619 to 941334 tid 1 2024-04-26T00:31:48.098 INFO:tasks.rados.rados.0.smithi045.stdout:1960: writing smithi04526579-35 from 1590648 to 2050619 tid 2 2024-04-26T00:31:48.098 INFO:tasks.rados.rados.0.smithi045.stdout:1961: read oid 7 snap -1 2024-04-26T00:31:48.098 INFO:tasks.rados.rados.0.smithi045.stdout:1961: expect deleted 2024-04-26T00:31:48.098 INFO:tasks.rados.rados.0.smithi045.stdout:1962: snap_remove snap 196 2024-04-26T00:31:48.242 INFO:tasks.rados.rados.0.smithi045.stdout:1962: done (3 left) 2024-04-26T00:31:48.243 INFO:tasks.rados.rados.0.smithi045.stdout:1963: write oid 50 current snap is 222 2024-04-26T00:31:48.243 INFO:tasks.rados.rados.0.smithi045.stdout:1959: finishing write tid 1 to smithi04526579-16 2024-04-26T00:31:48.243 INFO:tasks.rados.rados.0.smithi045.stdout:1963: seq_num 718 ranges {467426=565964,1649130=418296} 2024-04-26T00:31:48.257 INFO:tasks.rados.rados.0.smithi045.stdout:1963: writing smithi04526579-50 from 467426 to 1033390 tid 1 2024-04-26T00:31:48.272 INFO:tasks.rados.rados.0.smithi045.stdout:1963: writing smithi04526579-50 from 1649130 to 2067426 tid 2 2024-04-26T00:31:48.272 INFO:tasks.rados.rados.0.smithi045.stdout:1964: copy_from oid 36 from oid 39 current snap is 222 2024-04-26T00:31:48.272 INFO:tasks.rados.rados.0.smithi045.stdout:1959: finishing write tid 2 to smithi04526579-16 2024-04-26T00:31:48.272 INFO:tasks.rados.rados.0.smithi045.stdout:1959: finishing write tid 3 to smithi04526579-16 2024-04-26T00:31:48.272 INFO:tasks.rados.rados.0.smithi045.stdout:1959: finishing write tid 4 to smithi04526579-16 2024-04-26T00:31:48.272 INFO:tasks.rados.rados.0.smithi045.stdout:1959: oid 16 updating version 0 to 1129 2024-04-26T00:31:48.272 INFO:tasks.rados.rados.0.smithi045.stdout:1959: oid 16 updating version 1129 to 1130 2024-04-26T00:31:48.272 INFO:tasks.rados.rados.0.smithi045.stdout:1959: oid 16 updating version 1130 to 1131 2024-04-26T00:31:48.272 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 1131 (ObjNum 716 snap 222 seq_num 716) dirty exists 2024-04-26T00:31:48.272 INFO:tasks.rados.rados.0.smithi045.stdout:1959: left oid 16 (ObjNum 716 snap 222 seq_num 716) 2024-04-26T00:31:48.272 INFO:tasks.rados.rados.0.smithi045.stdout:1960: finishing write tid 1 to smithi04526579-35 2024-04-26T00:31:48.273 INFO:tasks.rados.rados.0.smithi045.stdout:1960: finishing write tid 2 to smithi04526579-35 2024-04-26T00:31:48.273 INFO:tasks.rados.rados.0.smithi045.stdout:1960: finishing write tid 3 to smithi04526579-35 2024-04-26T00:31:48.273 INFO:tasks.rados.rados.0.smithi045.stdout:1960: finishing write tid 4 to smithi04526579-35 2024-04-26T00:31:48.273 INFO:tasks.rados.rados.0.smithi045.stdout:1960: oid 35 updating version 0 to 1287 2024-04-26T00:31:48.273 INFO:tasks.rados.rados.0.smithi045.stdout:1960: oid 35 version 1287 is already newer than 1286 2024-04-26T00:31:48.273 INFO:tasks.rados.rados.0.smithi045.stdout:1960: oid 35 updating version 1287 to 1288 2024-04-26T00:31:48.273 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 1288 (ObjNum 717 snap 222 seq_num 717) dirty exists 2024-04-26T00:31:48.273 INFO:tasks.rados.rados.0.smithi045.stdout:1960: left oid 35 (ObjNum 717 snap 222 seq_num 717) 2024-04-26T00:31:48.273 INFO:tasks.rados.rados.0.smithi045.stdout:1959: done (4 left) 2024-04-26T00:31:48.273 INFO:tasks.rados.rados.0.smithi045.stdout:1960: done (3 left) 2024-04-26T00:31:48.273 INFO:tasks.rados.rados.0.smithi045.stdout:1961: done (2 left) 2024-04-26T00:31:48.273 INFO:tasks.rados.rados.0.smithi045.stdout:1965: rollback oid 19 current snap is 222 2024-04-26T00:31:48.273 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 19 to 211 2024-04-26T00:31:48.273 INFO:tasks.rados.rados.0.smithi045.stdout:1966: write oid 2 current snap is 222 2024-04-26T00:31:48.273 INFO:tasks.rados.rados.0.smithi045.stdout:1966: seq_num 719 ranges {484233=641213,1707612=376621} 2024-04-26T00:31:48.285 INFO:tasks.rados.rados.0.smithi045.stdout:1966: writing smithi04526579-2 from 484233 to 1125446 tid 1 2024-04-26T00:31:48.297 INFO:tasks.rados.rados.0.smithi045.stdout:1966: writing smithi04526579-2 from 1707612 to 2084233 tid 2 2024-04-26T00:31:48.298 INFO:tasks.rados.rados.0.smithi045.stdout:1967: rollback oid 23 current snap is 222 2024-04-26T00:31:48.298 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 23 to 206 2024-04-26T00:31:48.298 INFO:tasks.rados.rados.0.smithi045.stdout:1963: finishing write tid 1 to smithi04526579-50 2024-04-26T00:31:48.298 INFO:tasks.rados.rados.0.smithi045.stdout:1963: finishing write tid 2 to smithi04526579-50 2024-04-26T00:31:48.298 INFO:tasks.rados.rados.0.smithi045.stdout:1965: finishing rollback tid 0 to smithi04526579-19 2024-04-26T00:31:48.298 INFO:tasks.rados.rados.0.smithi045.stdout:1963: finishing write tid 3 to smithi04526579-50 2024-04-26T00:31:48.298 INFO:tasks.rados.rados.0.smithi045.stdout:1965: finishing rollback tid 1 to smithi04526579-19 2024-04-26T00:31:48.298 INFO:tasks.rados.rados.0.smithi045.stdout:1963: finishing write tid 4 to smithi04526579-50 2024-04-26T00:31:48.298 INFO:tasks.rados.rados.0.smithi045.stdout:1963: oid 50 updating version 0 to 1066 2024-04-26T00:31:48.298 INFO:tasks.rados.rados.0.smithi045.stdout:1963: oid 50 updating version 1066 to 1067 2024-04-26T00:31:48.298 INFO:tasks.rados.rados.0.smithi045.stdout:1963: oid 50 updating version 1067 to 1068 2024-04-26T00:31:48.298 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 1068 (ObjNum 718 snap 222 seq_num 718) dirty exists 2024-04-26T00:31:48.299 INFO:tasks.rados.rados.0.smithi045.stdout:1963: left oid 50 (ObjNum 718 snap 222 seq_num 718) 2024-04-26T00:31:48.299 INFO:tasks.rados.rados.0.smithi045.stdout:1965: finishing rollback tid 2 to smithi04526579-19 2024-04-26T00:31:48.299 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 1297 (ObjNum 534 snap 165 seq_num 534) dirty exists 2024-04-26T00:31:48.299 INFO:tasks.rados.rados.0.smithi045.stdout:1966: finishing write tid 1 to smithi04526579-2 2024-04-26T00:31:48.299 INFO:tasks.rados.rados.0.smithi045.stdout:1963: done (4 left) 2024-04-26T00:31:48.299 INFO:tasks.rados.rados.0.smithi045.stdout:1965: done (3 left) 2024-04-26T00:31:48.299 INFO:tasks.rados.rados.0.smithi045.stdout:1968: read oid 40 snap 153 2024-04-26T00:31:48.299 INFO:tasks.rados.rados.0.smithi045.stdout:1968: expect deleted 2024-04-26T00:31:48.299 INFO:tasks.rados.rados.0.smithi045.stdout:1969: read oid 17 snap 205 2024-04-26T00:31:48.299 INFO:tasks.rados.rados.0.smithi045.stdout:1969: expect (ObjNum 595 snap 184 seq_num 595) 2024-04-26T00:31:48.299 INFO:tasks.rados.rados.0.smithi045.stdout:1970: rollback oid 13 current snap is 222 2024-04-26T00:31:48.299 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 13 to 214 2024-04-26T00:31:48.299 INFO:tasks.rados.rados.0.smithi045.stdout:1971: delete oid 34 current snap is 222 2024-04-26T00:31:48.306 INFO:tasks.rados.rados.0.smithi045.stdout:1969: expect (ObjNum 595 snap 184 seq_num 595) 2024-04-26T00:31:48.426 INFO:tasks.rados.rados.0.smithi045.stdout:1970: finishing rollback tid 1 to smithi04526579-13 2024-04-26T00:31:48.426 INFO:tasks.rados.rados.0.smithi045.stdout:1966: finishing write tid 2 to smithi04526579-2 2024-04-26T00:31:48.426 INFO:tasks.rados.rados.0.smithi045.stdout:1970: finishing rollback tid 2 to smithi04526579-13 2024-04-26T00:31:48.426 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 1068 (ObjNum 668 snap 210 seq_num 668) dirty exists 2024-04-26T00:31:48.427 INFO:tasks.rados.rados.0.smithi045.stdout:1967: finishing rollback tid 0 to smithi04526579-23 2024-04-26T00:31:48.427 INFO:tasks.rados.rados.0.smithi045.stdout:1966: finishing write tid 3 to smithi04526579-2 2024-04-26T00:31:48.427 INFO:tasks.rados.rados.0.smithi045.stdout:1964: finishing copy_from racing read to smithi04526579-36 2024-04-26T00:31:48.427 INFO:tasks.rados.rados.0.smithi045.stdout:1966: finishing write tid 4 to smithi04526579-2 2024-04-26T00:31:48.427 INFO:tasks.rados.rados.0.smithi045.stdout:1966: oid 2 updating version 0 to 1512 2024-04-26T00:31:48.427 INFO:tasks.rados.rados.0.smithi045.stdout:1966: oid 2 version 1512 is already newer than 1510 2024-04-26T00:31:48.427 INFO:tasks.rados.rados.0.smithi045.stdout:1966: oid 2 version 1512 is already newer than 1511 2024-04-26T00:31:48.428 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 1512 (ObjNum 719 snap 222 seq_num 719) dirty exists 2024-04-26T00:31:48.428 INFO:tasks.rados.rados.0.smithi045.stdout:1966: left oid 2 (ObjNum 719 snap 222 seq_num 719) 2024-04-26T00:31:48.428 INFO:tasks.rados.rados.0.smithi045.stdout:1967: finishing rollback tid 1 to smithi04526579-23 2024-04-26T00:31:48.428 INFO:tasks.rados.rados.0.smithi045.stdout:1967: finishing rollback tid 2 to smithi04526579-23 2024-04-26T00:31:48.428 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 1437 (ObjNum 595 snap 184 seq_num 595) dirty exists 2024-04-26T00:31:48.428 INFO:tasks.rados.rados.0.smithi045.stdout:1964: finishing copy_from to smithi04526579-36 2024-04-26T00:31:48.428 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 1086 (ObjNum 703 snap 221 seq_num 703) dirty exists 2024-04-26T00:31:48.428 INFO:tasks.rados.rados.0.smithi045.stdout:1964: done (6 left) 2024-04-26T00:31:48.428 INFO:tasks.rados.rados.0.smithi045.stdout:1966: done (5 left) 2024-04-26T00:31:48.429 INFO:tasks.rados.rados.0.smithi045.stdout:1967: done (4 left) 2024-04-26T00:31:48.429 INFO:tasks.rados.rados.0.smithi045.stdout:1968: done (3 left) 2024-04-26T00:31:48.429 INFO:tasks.rados.rados.0.smithi045.stdout:1969: done (2 left) 2024-04-26T00:31:48.429 INFO:tasks.rados.rados.0.smithi045.stdout:1970: done (1 left) 2024-04-26T00:31:48.429 INFO:tasks.rados.rados.0.smithi045.stdout:1971: done (0 left) 2024-04-26T00:31:48.429 INFO:tasks.rados.rados.0.smithi045.stdout:1972: write (excl) oid 33 current snap is 222 2024-04-26T00:31:48.429 INFO:tasks.rados.rados.0.smithi045.stdout:1972: seq_num 721 ranges {517847=791711,2108223=9624} 2024-04-26T00:31:48.443 INFO:tasks.rados.rados.0.smithi045.stdout:1972: writing smithi04526579-33 from 517847 to 1309558 tid 1 2024-04-26T00:31:48.453 INFO:tasks.rados.rados.0.smithi045.stdout:1972: writing smithi04526579-33 from 2108223 to 2117847 tid 2 2024-04-26T00:31:48.454 INFO:tasks.rados.rados.0.smithi045.stdout:1973: copy_from oid 2 from oid 13 current snap is 222 2024-04-26T00:31:48.454 INFO:tasks.rados.rados.0.smithi045.stdout:1972: finishing write tid 1 to smithi04526579-33 2024-04-26T00:31:48.454 INFO:tasks.rados.rados.0.smithi045.stdout:1974: read oid 9 snap -1 2024-04-26T00:31:48.454 INFO:tasks.rados.rados.0.smithi045.stdout:1974: expect (ObjNum 669 snap 210 seq_num 669) 2024-04-26T00:31:48.454 INFO:tasks.rados.rados.0.smithi045.stdout:1975: snap_remove snap 221 2024-04-26T00:31:49.251 INFO:tasks.rados.rados.0.smithi045.stdout:1975: done (3 left) 2024-04-26T00:31:49.251 INFO:tasks.rados.rados.0.smithi045.stdout:1976: delete oid 23 current snap is 222 2024-04-26T00:31:49.251 INFO:tasks.rados.rados.0.smithi045.stdout:1972: finishing write tid 2 to smithi04526579-33 2024-04-26T00:31:49.252 INFO:tasks.rados.rados.0.smithi045.stdout:1972: finishing write tid 3 to smithi04526579-33 2024-04-26T00:31:49.252 INFO:tasks.rados.rados.0.smithi045.stdout:1972: finishing write tid 4 to smithi04526579-33 2024-04-26T00:31:49.252 INFO:tasks.rados.rados.0.smithi045.stdout:1972: oid 33 updating version 0 to 1667 2024-04-26T00:31:49.252 INFO:tasks.rados.rados.0.smithi045.stdout:1972: oid 33 updating version 1667 to 1668 2024-04-26T00:31:49.252 INFO:tasks.rados.rados.0.smithi045.stdout:1972: oid 33 version 1668 is already newer than 1666 2024-04-26T00:31:49.252 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 1668 (ObjNum 721 snap 222 seq_num 721) dirty exists 2024-04-26T00:31:49.252 INFO:tasks.rados.rados.0.smithi045.stdout:1972: left oid 33 (ObjNum 721 snap 222 seq_num 721) 2024-04-26T00:31:49.252 INFO:tasks.rados.rados.0.smithi045.stdout:1974: expect (ObjNum 669 snap 210 seq_num 669) 2024-04-26T00:31:49.475 INFO:tasks.rados.rados.0.smithi045.stdout:1973: finishing copy_from racing read to smithi04526579-2 2024-04-26T00:31:49.476 INFO:tasks.rados.rados.0.smithi045.stdout:1973: finishing copy_from to smithi04526579-2 2024-04-26T00:31:49.476 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 1068 (ObjNum 668 snap 210 seq_num 668) dirty exists 2024-04-26T00:31:49.476 INFO:tasks.rados.rados.0.smithi045.stdout:1972: done (3 left) 2024-04-26T00:31:49.476 INFO:tasks.rados.rados.0.smithi045.stdout:1973: done (2 left) 2024-04-26T00:31:49.476 INFO:tasks.rados.rados.0.smithi045.stdout:1974: done (1 left) 2024-04-26T00:31:49.476 INFO:tasks.rados.rados.0.smithi045.stdout:1976: done (0 left) 2024-04-26T00:31:49.476 INFO:tasks.rados.rados.0.smithi045.stdout:1977: copy_from oid 34 from oid 48 current snap is 222 2024-04-26T00:31:49.476 INFO:tasks.rados.rados.0.smithi045.stdout:1978: read oid 38 snap -1 2024-04-26T00:31:49.476 INFO:tasks.rados.rados.0.smithi045.stdout:1978: expect (ObjNum 582 snap 180 seq_num 582) 2024-04-26T00:31:49.476 INFO:tasks.rados.rados.0.smithi045.stdout:1979: write oid 27 current snap is 222 2024-04-26T00:31:49.476 INFO:tasks.rados.rados.0.smithi045.stdout:1979: seq_num 723 ranges {551461=658562,1941540=209921} 2024-04-26T00:31:49.501 INFO:tasks.rados.rados.0.smithi045.stdout:1979: writing smithi04526579-27 from 551461 to 1210023 tid 1 2024-04-26T00:31:49.517 INFO:tasks.rados.rados.0.smithi045.stdout:1979: writing smithi04526579-27 from 1941540 to 2151461 tid 2 2024-04-26T00:31:49.517 INFO:tasks.rados.rados.0.smithi045.stdout:1980: write oid 15 current snap is 222 2024-04-26T00:31:49.517 INFO:tasks.rados.rados.0.smithi045.stdout:1980: seq_num 724 ranges {568268=733811,2000022=168246} 2024-04-26T00:31:49.536 INFO:tasks.rados.rados.0.smithi045.stdout:1980: writing smithi04526579-15 from 568268 to 1302079 tid 1 2024-04-26T00:31:49.546 INFO:tasks.rados.rados.0.smithi045.stdout:1980: writing smithi04526579-15 from 2000022 to 2168268 tid 2 2024-04-26T00:31:49.546 INFO:tasks.rados.rados.0.smithi045.stdout:1981: rollback oid 43 current snap is 222 2024-04-26T00:31:49.546 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 43 to 205 2024-04-26T00:31:49.547 INFO:tasks.rados.rados.0.smithi045.stdout:1978: expect (ObjNum 582 snap 180 seq_num 582) 2024-04-26T00:31:49.780 INFO:tasks.rados.rados.0.smithi045.stdout:1977: finishing copy_from racing read to smithi04526579-34 2024-04-26T00:31:49.780 INFO:tasks.rados.rados.0.smithi045.stdout:1977: finishing copy_from to smithi04526579-34 2024-04-26T00:31:49.780 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 1118 (ObjNum 699 snap 219 seq_num 699) dirty exists 2024-04-26T00:31:49.780 INFO:tasks.rados.rados.0.smithi045.stdout:1979: finishing write tid 1 to smithi04526579-27 2024-04-26T00:31:49.780 INFO:tasks.rados.rados.0.smithi045.stdout:1979: finishing write tid 2 to smithi04526579-27 2024-04-26T00:31:49.780 INFO:tasks.rados.rados.0.smithi045.stdout:1979: finishing write tid 3 to smithi04526579-27 2024-04-26T00:31:49.780 INFO:tasks.rados.rados.0.smithi045.stdout:1979: finishing write tid 4 to smithi04526579-27 2024-04-26T00:31:49.780 INFO:tasks.rados.rados.0.smithi045.stdout:1979: oid 27 updating version 0 to 1303 2024-04-26T00:31:49.780 INFO:tasks.rados.rados.0.smithi045.stdout:1979: oid 27 version 1303 is already newer than 1302 2024-04-26T00:31:49.780 INFO:tasks.rados.rados.0.smithi045.stdout:1979: oid 27 version 1303 is already newer than 1301 2024-04-26T00:31:49.780 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 1303 (ObjNum 723 snap 222 seq_num 723) dirty exists 2024-04-26T00:31:49.781 INFO:tasks.rados.rados.0.smithi045.stdout:1979: left oid 27 (ObjNum 723 snap 222 seq_num 723) 2024-04-26T00:31:49.781 INFO:tasks.rados.rados.0.smithi045.stdout:1980: finishing write tid 1 to smithi04526579-15 2024-04-26T00:31:49.781 INFO:tasks.rados.rados.0.smithi045.stdout:1980: finishing write tid 2 to smithi04526579-15 2024-04-26T00:31:49.781 INFO:tasks.rados.rados.0.smithi045.stdout:1980: finishing write tid 3 to smithi04526579-15 2024-04-26T00:31:49.781 INFO:tasks.rados.rados.0.smithi045.stdout:1980: finishing write tid 4 to smithi04526579-15 2024-04-26T00:31:49.781 INFO:tasks.rados.rados.0.smithi045.stdout:1980: oid 15 updating version 0 to 1289 2024-04-26T00:31:49.781 INFO:tasks.rados.rados.0.smithi045.stdout:1980: oid 15 version 1289 is already newer than 1288 2024-04-26T00:31:49.781 INFO:tasks.rados.rados.0.smithi045.stdout:1980: oid 15 updating version 1289 to 1290 2024-04-26T00:31:49.781 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 1290 (ObjNum 724 snap 222 seq_num 724) dirty exists 2024-04-26T00:31:49.781 INFO:tasks.rados.rados.0.smithi045.stdout:1980: left oid 15 (ObjNum 724 snap 222 seq_num 724) 2024-04-26T00:31:49.781 INFO:tasks.rados.rados.0.smithi045.stdout:1981: finishing rollback tid 0 to smithi04526579-43 2024-04-26T00:31:49.781 INFO:tasks.rados.rados.0.smithi045.stdout:1981: finishing rollback tid 1 to smithi04526579-43 2024-04-26T00:31:49.781 INFO:tasks.rados.rados.0.smithi045.stdout:1981: finishing rollback tid 2 to smithi04526579-43 2024-04-26T00:31:49.781 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 1071 (ObjNum 646 snap 202 seq_num 646) dirty exists 2024-04-26T00:31:49.782 INFO:tasks.rados.rados.0.smithi045.stdout:1977: done (4 left) 2024-04-26T00:31:49.782 INFO:tasks.rados.rados.0.smithi045.stdout:1978: done (3 left) 2024-04-26T00:31:49.782 INFO:tasks.rados.rados.0.smithi045.stdout:1979: done (2 left) 2024-04-26T00:31:49.782 INFO:tasks.rados.rados.0.smithi045.stdout:1980: done (1 left) 2024-04-26T00:31:49.782 INFO:tasks.rados.rados.0.smithi045.stdout:1981: done (0 left) 2024-04-26T00:31:49.782 INFO:tasks.rados.rados.0.smithi045.stdout:1982: snap_remove snap 203 2024-04-26T00:31:50.138 INFO:tasks.rados.rados.0.smithi045.stdout:1982: done (0 left) 2024-04-26T00:31:50.138 INFO:tasks.rados.rados.0.smithi045.stdout:1983: snap_remove snap 214 2024-04-26T00:31:51.147 INFO:tasks.rados.rados.0.smithi045.stdout:1983: done (0 left) 2024-04-26T00:31:51.147 INFO:tasks.rados.rados.0.smithi045.stdout:1984: copy_from oid 26 from oid 7 current snap is 222 2024-04-26T00:31:51.147 INFO:tasks.rados.rados.0.smithi045.stdout:1985: delete oid 50 current snap is 222 2024-04-26T00:31:51.154 INFO:tasks.rados.rados.0.smithi045.stdout:1984: finishing copy_from to smithi04526579-26 2024-04-26T00:31:51.154 INFO:tasks.rados.rados.0.smithi045.stdout:1984: got expected ENOENT (src dne) 2024-04-26T00:31:51.156 INFO:tasks.rados.rados.0.smithi045.stdout:1984: finishing copy_from racing read to smithi04526579-26 2024-04-26T00:31:51.171 INFO:tasks.rados.rados.0.smithi045.stdout:1984: done (1 left) 2024-04-26T00:31:51.171 INFO:tasks.rados.rados.0.smithi045.stdout:1985: done (0 left) 2024-04-26T00:31:51.171 INFO:tasks.rados.rados.0.smithi045.stdout:1986: copy_from oid 23 from oid 26 current snap is 222 2024-04-26T00:31:51.171 INFO:tasks.rados.rados.0.smithi045.stdout:1987: snap_create 2024-04-26T00:31:51.191 INFO:tasks.rados.rados.0.smithi045.stdout:1986: finishing copy_from racing read to smithi04526579-23 2024-04-26T00:31:51.194 INFO:tasks.rados.rados.0.smithi045.stdout:1986: finishing copy_from to smithi04526579-23 2024-04-26T00:31:51.194 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 1094 (ObjNum 641 snap 201 seq_num 641) dirty exists 2024-04-26T00:31:52.154 INFO:tasks.rados.rados.0.smithi045.stdout:1986: done (1 left) 2024-04-26T00:31:52.154 INFO:tasks.rados.rados.0.smithi045.stdout:1987: done (0 left) 2024-04-26T00:31:52.154 INFO:tasks.rados.rados.0.smithi045.stdout:1988: read oid 28 snap -1 2024-04-26T00:31:52.154 INFO:tasks.rados.rados.0.smithi045.stdout:1988: expect deleted 2024-04-26T00:31:52.154 INFO:tasks.rados.rados.0.smithi045.stdout:1989: snap_create 2024-04-26T00:31:53.368 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-04-26T00:31:53.369 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-04-26T00:31:53.369 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-26T00:31:53.369 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-26T00:31:53.372 INFO:tasks.ceph.mon.a:Started 2024-04-26T00:31:53.372 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:31:53.372 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:31:53.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.516+0000 7f9e7481d640 10 monclient: get_monmap_and_config 2024-04-26T00:31:53.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.516+0000 7f9e7481d640 10 monclient: build_initial_monmap 2024-04-26T00:31:53.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.516+0000 7f9e7481d640 10 monclient: monmap: 2024-04-26T00:31:53.510 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:31:53.510 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:53.510 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:31:53.520653+0000 2024-04-26T00:31:53.510 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:31:53.520653+0000 2024-04-26T00:31:53.511 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:31:53.511 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:53.511 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:31:53.511 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:31:53.511 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:31:53.511 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:31:53.511 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:31:53.511 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:53.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.516+0000 7f9e7481d640 1 Processor -- start 2024-04-26T00:31:53.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.516+0000 7f9e7481d640 1 -- start start 2024-04-26T00:31:53.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.516+0000 7f9e7481d640 10 monclient: init 2024-04-26T00:31:53.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.516+0000 7f9e7481d640 10 monclient: _reopen_session rank -1 2024-04-26T00:31:53.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.516+0000 7f9e7481d640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-04-26T00:31:53.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.516+0000 7f9e7481d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9e70136eb0 0x7f9e70137290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:53.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.516+0000 7f9e7481d640 10 monclient(hunting): picked mon.noname-c con 0x7f9e70136eb0 addr v2:172.21.15.45:3300/0 2024-04-26T00:31:53.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.516+0000 7f9e7481d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9e701377d0 addr v1:172.21.15.45:6789/0 2024-04-26T00:31:53.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.516+0000 7f9e7481d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9e701342a0 addr v1:172.21.15.37:6789/0 2024-04-26T00:31:53.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.516+0000 7f9e7481d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e7009b8c0 con 0x7f9e701342a0 2024-04-26T00:31:53.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.516+0000 7f9e7481d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e7009b300 con 0x7f9e701377d0 2024-04-26T00:31:53.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.516+0000 7f9e7481d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:53.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.516+0000 7f9e7481d640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f9e7009c7d0 con 0x7f9e70136eb0 2024-04-26T00:31:53.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.516+0000 7f9e7481d640 10 monclient(hunting): _renew_subs 2024-04-26T00:31:53.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.516+0000 7f9e7481d640 10 monclient(hunting): authenticate will time out at 1694.826306s 2024-04-26T00:31:53.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.516+0000 7f9e6ed9d640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f9e701342a0 0x7f9e7013f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:40684/0 (socket says 172.21.15.37:40684) 2024-04-26T00:31:53.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.516+0000 7f9e6ed9d640 1 -- 172.21.15.37:0/3488320946 learned_addr learned my addr 172.21.15.37:0/3488320946 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:31:53.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.516+0000 7f9e6e59c640 1 --2- 172.21.15.37:0/3488320946 >> v2:172.21.15.45:3300/0 conn(0x7f9e70136eb0 0x7f9e70137290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:53.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.516+0000 7f9e6e59c640 10 monclient(hunting): get_auth_request con 0x7f9e70136eb0 auth_method 0 2024-04-26T00:31:53.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.516+0000 7f9e6e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:53.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.516+0000 7f9e6e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:53.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.516+0000 7f9e6e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:53.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.516+0000 7f9e6e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:53.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.516+0000 7f9e6e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:53.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.516+0000 7f9e6e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:53.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.516+0000 7f9e6e59c640 10 monclient(hunting): handle_auth_done global_id 134170 payload 274 2024-04-26T00:31:53.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.516+0000 7f9e6e59c640 1 -- 172.21.15.37:0/3488320946 >> v1:172.21.15.45:6789/0 conn(0x7f9e701377d0 legacy=0x7f9e70133b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:31:53.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.516+0000 7f9e6e59c640 1 -- 172.21.15.37:0/3488320946 >> v1:172.21.15.37:6789/0 conn(0x7f9e701342a0 legacy=0x7f9e7013f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:31:53.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.516+0000 7f9e6e59c640 10 monclient: _finish_hunting 0 2024-04-26T00:31:53.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.516+0000 7f9e6e59c640 1 monclient: found mon.noname-c 2024-04-26T00:31:53.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.516+0000 7f9e6e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:31:53.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.516+0000 7f9e6e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:31:53.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.516+0000 7f9e6e59c640 1 -- 172.21.15.37:0/3488320946 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e7009b5e0 con 0x7f9e70136eb0 2024-04-26T00:31:53.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.520+0000 7f9e6e59c640 1 -- 172.21.15.37:0/3488320946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e70136eb0 msgr2=0x7f9e70137290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:53.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.520+0000 7f9e6e59c640 1 --2- 172.21.15.37:0/3488320946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e70136eb0 0x7f9e70137290 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9e600097c0 tx=0x7f9e6001c100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=580d3e57babc6e5b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:53.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6d59a640 1 -- 172.21.15.37:0/3488320946 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e600b0070 con 0x7f9e70136eb0 2024-04-26T00:31:53.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:53.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:31:53.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6d59a640 10 monclient: dump: 2024-04-26T00:31:53.518 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:53.519 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:53.519 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:53.519 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:53.519 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:53.519 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:53.519 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:53.519 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:53.519 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:53.519 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:53.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6d59a640 10 monclient: _finish_auth 0 2024-04-26T00:31:53.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6d59a640 10 monclient: _check_auth_tickets 2024-04-26T00:31:53.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:31:53.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e7481d640 5 monclient: authenticate success, global_id 134170 2024-04-26T00:31:53.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e7481d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:31:53.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6d59a640 1 -- 172.21.15.37:0/3488320946 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e600ab600 con 0x7f9e70136eb0 2024-04-26T00:31:53.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6d59a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:31:53.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e7481d640 10 monclient: get_monmap_and_config success 2024-04-26T00:31:53.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e7481d640 4 set_mon_vals no callback set 2024-04-26T00:31:53.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6d59a640 1 -- 172.21.15.37:0/3488320946 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e600ab9d0 con 0x7f9e70136eb0 2024-04-26T00:31:53.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:53.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:31:53.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6d59a640 10 monclient: dump: 2024-04-26T00:31:53.520 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:53.520 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:53.521 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:53.521 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:53.521 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:53.521 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:53.521 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:53.521 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:53.521 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:53.521 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:53.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e7481d640 10 monclient: shutdown 2024-04-26T00:31:53.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e7481d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:31:53.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e7481d640 1 -- 172.21.15.37:0/3488320946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e70136eb0 msgr2=0x7f9e70137290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:53.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e7481d640 1 --2- 172.21.15.37:0/3488320946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e70136eb0 0x7f9e70137290 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9e600097c0 tx=0x7f9e6001c100 comp rx=0 tx=0).stop 2024-04-26T00:31:53.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e7481d640 1 -- 172.21.15.37:0/3488320946 shutdown_connections 2024-04-26T00:31:53.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e7481d640 1 --2- 172.21.15.37:0/3488320946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e70136eb0 0x7f9e70137290 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:53.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e7481d640 1 -- 172.21.15.37:0/3488320946 >> 172.21.15.37:0/3488320946 conn(0x7f9e700a6300 msgr2=0x7f9e700a6710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:31:53.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e7481d640 1 -- 172.21.15.37:0/3488320946 shutdown_connections 2024-04-26T00:31:53.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e7481d640 1 -- 172.21.15.37:0/3488320946 wait complete. 2024-04-26T00:31:53.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e7481d640 10 monclient: build_initial_monmap 2024-04-26T00:31:53.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e7481d640 10 monclient: monmap: 2024-04-26T00:31:53.526 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:31:53.526 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:31:53.526 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:31:53.526 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:31:53.526 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:31:53.526 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:31:53.527 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:31:53.527 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:31:53.527 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:31:53.527 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:53.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e7481d640 1 Processor -- start 2024-04-26T00:31:53.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e7481d640 1 -- start start 2024-04-26T00:31:53.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e7481d640 10 monclient: init 2024-04-26T00:31:53.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e7481d640 10 monclient: _reopen_session rank -1 2024-04-26T00:31:53.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e7481d640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:31:53.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e7481d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e701342a0 0x7f9e70132de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:53.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e7481d640 10 monclient(hunting): picked mon.noname-a con 0x7f9e701342a0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:31:53.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e7481d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e70136eb0 0x7f9e70133320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:53.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e7481d640 10 monclient(hunting): picked mon.noname-c con 0x7f9e70136eb0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:31:53.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e7481d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e701377d0 0x7f9e70138f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:53.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e7481d640 10 monclient(hunting): picked mon.noname-b con 0x7f9e701377d0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:31:53.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e7481d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:53.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e7481d640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9e7009a9e0 con 0x7f9e701342a0 2024-04-26T00:31:53.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e7481d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:53.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e7481d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9e7009ab60 con 0x7f9e701377d0 2024-04-26T00:31:53.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e7481d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:53.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e7481d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9e701437c0 con 0x7f9e70136eb0 2024-04-26T00:31:53.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e7481d640 10 monclient(hunting): _renew_subs 2024-04-26T00:31:53.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e7481d640 10 monclient(hunting): authenticate will time out at 1694.834752s 2024-04-26T00:31:53.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6e59c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e701342a0 0x7f9e70132de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:53.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6dd9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e70136eb0 0x7f9e70133320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:53.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6e59c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e701342a0 0x7f9e70132de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:42964/0 (socket says 172.21.15.37:42964) 2024-04-26T00:31:53.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6e59c640 1 -- 172.21.15.37:0/2266018608 learned_addr learned my addr 172.21.15.37:0/2266018608 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:31:53.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6e59c640 10 monclient(hunting): get_auth_request con 0x7f9e701342a0 auth_method 0 2024-04-26T00:31:53.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:53.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:53.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:53.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6dd9b640 10 monclient(hunting): get_auth_request con 0x7f9e70136eb0 auth_method 0 2024-04-26T00:31:53.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:53.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:53.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:53.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6e59c640 1 -- 172.21.15.37:0/2266018608 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e701342a0 msgr2=0x7f9e70132de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:31:53.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6e59c640 1 -- 172.21.15.37:0/2266018608 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e701342a0 msgr2=0x7f9e70132de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:31:53.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6e59c640 1 --2- 172.21.15.37:0/2266018608 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e701342a0 0x7f9e70132de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:31:53.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:53.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:53.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6e59c640 1 --2- 172.21.15.37:0/2266018608 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e701342a0 0x7f9e70132de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:31:53.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:53.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6ed9d640 1 --2- 172.21.15.37:0/2266018608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e701377d0 0x7f9e70138f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:53.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6ed9d640 10 monclient(hunting): get_auth_request con 0x7f9e701377d0 auth_method 0 2024-04-26T00:31:53.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:53.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:53.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:53.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:53.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:53.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:53.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6dd9b640 10 monclient(hunting): handle_auth_done global_id 84635 payload 824 2024-04-26T00:31:53.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6dd9b640 1 -- 172.21.15.37:0/2266018608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e701377d0 msgr2=0x7f9e70138f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:53.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6dd9b640 1 --2- 172.21.15.37:0/2266018608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e701377d0 0x7f9e70138f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:53.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6dd9b640 1 -- 172.21.15.37:0/2266018608 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e701342a0 msgr2=0x7f9e70132de0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:31:53.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6dd9b640 1 --2- 172.21.15.37:0/2266018608 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e701342a0 0x7f9e70132de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:53.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6dd9b640 10 monclient: _finish_hunting 0 2024-04-26T00:31:53.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6dd9b640 1 monclient: found mon.noname-c 2024-04-26T00:31:53.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:31:53.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:31:53.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.524+0000 7f9e6dd9b640 1 -- 172.21.15.37:0/2266018608 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e7013d880 con 0x7f9e70136eb0 2024-04-26T00:31:53.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.528+0000 7f9e6dd9b640 1 -- 172.21.15.37:0/2266018608 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e70136eb0 msgr2=0x7f9e70133320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:53.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.528+0000 7f9e6dd9b640 1 --2- 172.21.15.37:0/2266018608 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e70136eb0 0x7f9e70133320 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9e6400c410 tx=0x7f9e6400c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:53.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.528+0000 7f9e5effd640 1 -- 172.21.15.37:0/2266018608 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e64017070 con 0x7f9e70136eb0 2024-04-26T00:31:53.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.528+0000 7f9e5effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:53.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.528+0000 7f9e5effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:31:53.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.528+0000 7f9e5effd640 10 monclient: dump: 2024-04-26T00:31:53.533 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:53.533 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:53.533 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:53.533 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:53.533 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:53.533 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:53.533 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:53.534 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:53.534 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:53.534 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:53.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.528+0000 7f9e5effd640 10 monclient: _finish_auth 0 2024-04-26T00:31:53.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.528+0000 7f9e5effd640 10 monclient: _check_auth_tickets 2024-04-26T00:31:53.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.528+0000 7f9e5effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:31:53.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.528+0000 7f9e5effd640 1 -- 172.21.15.37:0/2266018608 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e640069e0 con 0x7f9e70136eb0 2024-04-26T00:31:53.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.528+0000 7f9e5effd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:31:53.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.528+0000 7f9e5effd640 1 -- 172.21.15.37:0/2266018608 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e64006dd0 con 0x7f9e70136eb0 2024-04-26T00:31:53.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.528+0000 7f9e5effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:53.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.528+0000 7f9e5effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:31:53.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.528+0000 7f9e5effd640 10 monclient: dump: 2024-04-26T00:31:53.534 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:53.534 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:53.535 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:53.535 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:53.535 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:53.535 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:53.535 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:53.535 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:53.535 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:53.535 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:53.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.528+0000 7f9e7481d640 5 monclient: authenticate success, global_id 84635 2024-04-26T00:31:53.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.528+0000 7f9e7481d640 10 monclient: _renew_subs 2024-04-26T00:31:53.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.528+0000 7f9e7481d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:53.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.528+0000 7f9e7481d640 1 -- 172.21.15.37:0/2266018608 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e7013cea0 con 0x7f9e70136eb0 2024-04-26T00:31:53.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.528+0000 7f9e6cd99640 4 set_mon_vals no callback set 2024-04-26T00:31:53.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.528+0000 7f9e7481d640 10 monclient: _renew_subs 2024-04-26T00:31:53.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.528+0000 7f9e7481d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:53.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.528+0000 7f9e7481d640 1 -- 172.21.15.37:0/2266018608 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e7013c8e0 con 0x7f9e70136eb0 2024-04-26T00:31:53.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.528+0000 7f9e7481d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:53.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.528+0000 7f9e7481d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:53.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.528+0000 7f9e7481d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:53.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.528+0000 7f9e5effd640 1 -- 172.21.15.37:0/2266018608 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9e6401a070 con 0x7f9e70136eb0 2024-04-26T00:31:53.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.528+0000 7f9e7481d640 1 -- 172.21.15.37:0/2266018608 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e70066f40 con 0x7f9e70136eb0 2024-04-26T00:31:53.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.528+0000 7f9e5effd640 1 --2- 172.21.15.37:0/2266018608 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9e50035e80 0x7f9e50038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:53.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.528+0000 7f9e5effd640 1 -- 172.21.15.37:0/2266018608 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(456..456 src has 370..456) ==== 3963+0+0 (secure 0 0 0) 0x7f9e64047e20 con 0x7f9e70136eb0 2024-04-26T00:31:53.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.528+0000 7f9e6e59c640 1 --2- 172.21.15.37:0/2266018608 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9e50035e80 0x7f9e50038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:53.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.528+0000 7f9e6e59c640 10 monclient: get_auth_request con 0x7f9e50035e80 auth_method 0 2024-04-26T00:31:53.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.532+0000 7f9e6e59c640 1 --2- 172.21.15.37:0/2266018608 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9e50035e80 0x7f9e50038340 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9e6001a7b0 tx=0x7f9e600a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:53.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.532+0000 7f9e5effd640 1 -- 172.21.15.37:0/2266018608 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9e7013d880 con 0x7f9e70136eb0 2024-04-26T00:31:53.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.532+0000 7f9e5effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:53.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.532+0000 7f9e5effd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:31:53.738 INFO:tasks.rados.rados.0.smithi045.stdout:1988: done (1 left) 2024-04-26T00:31:53.738 INFO:tasks.rados.rados.0.smithi045.stdout:1989: done (0 left) 2024-04-26T00:31:53.738 INFO:tasks.rados.rados.0.smithi045.stdout:1990: snap_remove snap 222 2024-04-26T00:31:53.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.756+0000 7f9e7481d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:31:53.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.756+0000 7f9e7481d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:31:53.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.756+0000 7f9e7481d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:53.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.756+0000 7f9e7481d640 1 -- 172.21.15.37:0/2266018608 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9e7013b490 con 0x7f9e70136eb0 2024-04-26T00:31:53.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.756+0000 7f9e5effd640 1 -- 172.21.15.37:0/2266018608 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f9e64046b50 con 0x7f9e70136eb0 2024-04-26T00:31:53.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.756+0000 7f9e5effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:31:53.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.756+0000 7f9e5effd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:31:53.752 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":90,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:31:53.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.760+0000 7f9e7481d640 1 -- 172.21.15.37:0/2266018608 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9e50035e80 msgr2=0x7f9e50038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:53.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.760+0000 7f9e7481d640 1 --2- 172.21.15.37:0/2266018608 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9e50035e80 0x7f9e50038340 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9e6001a7b0 tx=0x7f9e600a6000 comp rx=0 tx=0).stop 2024-04-26T00:31:53.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.760+0000 7f9e7481d640 10 monclient: shutdown 2024-04-26T00:31:53.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.760+0000 7f9e7481d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:31:53.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.760+0000 7f9e7481d640 1 -- 172.21.15.37:0/2266018608 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e70136eb0 msgr2=0x7f9e70133320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:53.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.760+0000 7f9e7481d640 1 --2- 172.21.15.37:0/2266018608 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e70136eb0 0x7f9e70133320 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9e6400c410 tx=0x7f9e6400c8d0 comp rx=0 tx=0).stop 2024-04-26T00:31:53.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.760+0000 7f9e7481d640 1 -- 172.21.15.37:0/2266018608 shutdown_connections 2024-04-26T00:31:53.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.760+0000 7f9e7481d640 1 --2- 172.21.15.37:0/2266018608 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9e50035e80 0x7f9e50038340 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:53.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.760+0000 7f9e7481d640 1 --2- 172.21.15.37:0/2266018608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e701377d0 0x7f9e70138f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:53.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.760+0000 7f9e7481d640 1 --2- 172.21.15.37:0/2266018608 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e70136eb0 0x7f9e70133320 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:53.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.760+0000 7f9e7481d640 1 --2- 172.21.15.37:0/2266018608 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e701342a0 0x7f9e70132de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:53.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.760+0000 7f9e7481d640 1 -- 172.21.15.37:0/2266018608 >> 172.21.15.37:0/2266018608 conn(0x7f9e700a6300 msgr2=0x7f9e7009ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:31:53.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.760+0000 7f9e7481d640 1 -- 172.21.15.37:0/2266018608 shutdown_connections 2024-04-26T00:31:53.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:53.760+0000 7f9e7481d640 1 -- 172.21.15.37:0/2266018608 wait complete. 2024-04-26T00:31:54.170 INFO:tasks.rados.rados.0.smithi045.stdout:1990: done (0 left) 2024-04-26T00:31:54.170 INFO:tasks.rados.rados.0.smithi045.stdout:1991: write (excl) oid 31 current snap is 224 2024-04-26T00:31:54.170 INFO:tasks.rados.rados.0.smithi045.stdout:1991: seq_num 726 ranges {601882=484309,1600633=524904} 2024-04-26T00:31:54.188 INFO:tasks.rados.rados.0.smithi045.stdout:1991: writing smithi04526579-31 from 601882 to 1086191 tid 1 2024-04-26T00:31:54.205 INFO:tasks.rados.rados.0.smithi045.stdout:1991: writing smithi04526579-31 from 1600633 to 2125537 tid 2 2024-04-26T00:31:54.205 INFO:tasks.rados.rados.0.smithi045.stdout:1992: read oid 21 snap -1 2024-04-26T00:31:54.205 INFO:tasks.rados.rados.0.smithi045.stdout:1992: expect (ObjNum 595 snap 184 seq_num 595) 2024-04-26T00:31:54.205 INFO:tasks.rados.rados.0.smithi045.stdout:1993: write (excl) oid 5 current snap is 224 2024-04-26T00:31:54.206 INFO:tasks.rados.rados.0.smithi045.stdout:1991: finishing write tid 1 to smithi04526579-31 2024-04-26T00:31:54.206 INFO:tasks.rados.rados.0.smithi045.stdout:1993: seq_num 727 ranges {618689=559558,1659115=559574} 2024-04-26T00:31:54.217 INFO:tasks.rados.rados.0.smithi045.stdout:1993: writing smithi04526579-5 from 618689 to 1178247 tid 1 2024-04-26T00:31:54.228 INFO:tasks.rados.rados.0.smithi045.stdout:1993: writing smithi04526579-5 from 1659115 to 2218689 tid 2 2024-04-26T00:31:54.228 INFO:tasks.rados.rados.0.smithi045.stdout:1994: delete oid 29 current snap is 224 2024-04-26T00:31:54.228 INFO:tasks.rados.rados.0.smithi045.stdout:1992: expect (ObjNum 595 snap 184 seq_num 595) 2024-04-26T00:31:54.351 INFO:tasks.rados.rados.0.smithi045.stdout:1991: finishing write tid 2 to smithi04526579-31 2024-04-26T00:31:54.351 INFO:tasks.rados.rados.0.smithi045.stdout:1991: finishing write tid 3 to smithi04526579-31 2024-04-26T00:31:54.351 INFO:tasks.rados.rados.0.smithi045.stdout:1991: finishing write tid 4 to smithi04526579-31 2024-04-26T00:31:54.351 INFO:tasks.rados.rados.0.smithi045.stdout:1991: oid 31 updating version 0 to 1513 2024-04-26T00:31:54.352 INFO:tasks.rados.rados.0.smithi045.stdout:1991: oid 31 updating version 1513 to 1515 2024-04-26T00:31:54.352 INFO:tasks.rados.rados.0.smithi045.stdout:1991: oid 31 version 1515 is already newer than 1514 2024-04-26T00:31:54.352 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 1515 (ObjNum 726 snap 224 seq_num 726) dirty exists 2024-04-26T00:31:54.352 INFO:tasks.rados.rados.0.smithi045.stdout:1991: left oid 31 (ObjNum 726 snap 224 seq_num 726) 2024-04-26T00:31:54.352 INFO:tasks.rados.rados.0.smithi045.stdout:1993: finishing write tid 1 to smithi04526579-5 2024-04-26T00:31:54.352 INFO:tasks.rados.rados.0.smithi045.stdout:1993: finishing write tid 2 to smithi04526579-5 2024-04-26T00:31:54.352 INFO:tasks.rados.rados.0.smithi045.stdout:1993: finishing write tid 3 to smithi04526579-5 2024-04-26T00:31:54.352 INFO:tasks.rados.rados.0.smithi045.stdout:1993: finishing write tid 4 to smithi04526579-5 2024-04-26T00:31:54.352 INFO:tasks.rados.rados.0.smithi045.stdout:1993: oid 5 updating version 0 to 1687 2024-04-26T00:31:54.352 INFO:tasks.rados.rados.0.smithi045.stdout:1993: oid 5 version 1687 is already newer than 1685 2024-04-26T00:31:54.352 INFO:tasks.rados.rados.0.smithi045.stdout:1993: oid 5 version 1687 is already newer than 1686 2024-04-26T00:31:54.352 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 1687 (ObjNum 727 snap 224 seq_num 727) dirty exists 2024-04-26T00:31:54.352 INFO:tasks.rados.rados.0.smithi045.stdout:1993: left oid 5 (ObjNum 727 snap 224 seq_num 727) 2024-04-26T00:31:54.353 INFO:tasks.rados.rados.0.smithi045.stdout:1991: done (3 left) 2024-04-26T00:31:54.353 INFO:tasks.rados.rados.0.smithi045.stdout:1992: done (2 left) 2024-04-26T00:31:54.353 INFO:tasks.rados.rados.0.smithi045.stdout:1993: done (1 left) 2024-04-26T00:31:54.353 INFO:tasks.rados.rados.0.smithi045.stdout:1994: done (0 left) 2024-04-26T00:31:54.353 INFO:tasks.rados.rados.0.smithi045.stdout:1995: write oid 12 current snap is 224 2024-04-26T00:31:54.353 INFO:tasks.rados.rados.0.smithi045.stdout:1995: seq_num 729 ranges {652303=710056,2059726=192577} 2024-04-26T00:31:54.370 INFO:tasks.rados.rados.0.smithi045.stdout:1995: writing smithi04526579-12 from 652303 to 1362359 tid 1 2024-04-26T00:31:54.383 INFO:tasks.rados.rados.0.smithi045.stdout:1995: writing smithi04526579-12 from 2059726 to 2252303 tid 2 2024-04-26T00:31:54.383 INFO:tasks.rados.rados.0.smithi045.stdout:1996: write (excl) oid 5 current snap is 224 2024-04-26T00:31:54.383 INFO:tasks.rados.rados.0.smithi045.stdout:1996: seq_num 730 ranges {669110=501658,1834561=434549} 2024-04-26T00:31:54.400 INFO:tasks.rados.rados.0.smithi045.stdout:1996: writing smithi04526579-5 from 669110 to 1170768 tid 1 2024-04-26T00:31:54.415 INFO:tasks.rados.rados.0.smithi045.stdout:1996: writing smithi04526579-5 from 1834561 to 2269110 tid 2 2024-04-26T00:31:54.499 INFO:tasks.rados.rados.0.smithi045.stdout:1997: read oid 23 snap -1 2024-04-26T00:31:54.499 INFO:tasks.rados.rados.0.smithi045.stdout:1997: expect (ObjNum 641 snap 201 seq_num 641) 2024-04-26T00:31:54.499 INFO:tasks.rados.rados.0.smithi045.stdout:1998: copy_from oid 46 from oid 30 current snap is 224 2024-04-26T00:31:54.499 INFO:tasks.rados.rados.0.smithi045.stdout:1995: finishing write tid 1 to smithi04526579-12 2024-04-26T00:31:54.499 INFO:tasks.rados.rados.0.smithi045.stdout:1995: finishing write tid 2 to smithi04526579-12 2024-04-26T00:31:54.499 INFO:tasks.rados.rados.0.smithi045.stdout:1995: finishing write tid 3 to smithi04526579-12 2024-04-26T00:31:54.499 INFO:tasks.rados.rados.0.smithi045.stdout:1995: finishing write tid 4 to smithi04526579-12 2024-04-26T00:31:54.499 INFO:tasks.rados.rados.0.smithi045.stdout:1995: oid 12 updating version 0 to 1439 2024-04-26T00:31:54.499 INFO:tasks.rados.rados.0.smithi045.stdout:1995: oid 12 updating version 1439 to 1441 2024-04-26T00:31:54.500 INFO:tasks.rados.rados.0.smithi045.stdout:1995: oid 12 version 1441 is already newer than 1440 2024-04-26T00:31:54.500 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 1441 (ObjNum 729 snap 224 seq_num 729) dirty exists 2024-04-26T00:31:54.500 INFO:tasks.rados.rados.0.smithi045.stdout:1995: left oid 12 (ObjNum 729 snap 224 seq_num 729) 2024-04-26T00:31:54.500 INFO:tasks.rados.rados.0.smithi045.stdout:1996: finishing write tid 1 to smithi04526579-5 2024-04-26T00:31:54.500 INFO:tasks.rados.rados.0.smithi045.stdout:1995: done (3 left) 2024-04-26T00:31:54.500 INFO:tasks.rados.rados.0.smithi045.stdout:1999: rollback oid 19 current snap is 224 2024-04-26T00:31:54.500 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 19 to 206 2024-04-26T00:31:54.500 INFO:tasks.rados.rados.0.smithi045.stdout:2000: write (excl) oid 26 current snap is 224 2024-04-26T00:31:54.500 INFO:tasks.rados.rados.0.smithi045.stdout:2000: seq_num 731 ranges {685917=576907,1893043=392874} 2024-04-26T00:31:54.501 INFO:tasks.rados.rados.0.smithi045.stdout:2000: writing smithi04526579-26 from 685917 to 1262824 tid 1 2024-04-26T00:31:54.501 INFO:tasks.rados.rados.0.smithi045.stdout:2000: writing smithi04526579-26 from 1893043 to 2285917 tid 2 2024-04-26T00:31:54.501 INFO:tasks.rados.rados.0.smithi045.stdout:2001: rollback oid 39 current snap is 224 2024-04-26T00:31:54.501 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 39 to 205 2024-04-26T00:31:54.502 INFO:tasks.rados.rados.0.smithi045.stdout:1998: finishing copy_from to smithi04526579-46 2024-04-26T00:31:54.502 INFO:tasks.rados.rados.0.smithi045.stdout:1998: got expected ENOENT (src dne) 2024-04-26T00:31:54.502 INFO:tasks.rados.rados.0.smithi045.stdout:1998: finishing copy_from racing read to smithi04526579-46 2024-04-26T00:31:54.502 INFO:tasks.rados.rados.0.smithi045.stdout:1996: finishing write tid 2 to smithi04526579-5 2024-04-26T00:31:54.502 INFO:tasks.rados.rados.0.smithi045.stdout:1997: expect (ObjNum 641 snap 201 seq_num 641) 2024-04-26T00:31:54.578 INFO:tasks.rados.rados.0.smithi045.stdout:1996: finishing write tid 3 to smithi04526579-5 2024-04-26T00:31:54.578 INFO:tasks.rados.rados.0.smithi045.stdout:1996: finishing write tid 4 to smithi04526579-5 2024-04-26T00:31:54.578 INFO:tasks.rados.rados.0.smithi045.stdout:1996: oid 5 updating version 0 to 1690 2024-04-26T00:31:54.578 INFO:tasks.rados.rados.0.smithi045.stdout:1996: oid 5 version 1690 is already newer than 1688 2024-04-26T00:31:54.578 INFO:tasks.rados.rados.0.smithi045.stdout:1996: oid 5 version 1690 is already newer than 1689 2024-04-26T00:31:54.578 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 1690 (ObjNum 730 snap 224 seq_num 730) dirty exists 2024-04-26T00:31:54.578 INFO:tasks.rados.rados.0.smithi045.stdout:1996: left oid 5 (ObjNum 730 snap 224 seq_num 730) 2024-04-26T00:31:54.578 INFO:tasks.rados.rados.0.smithi045.stdout:1999: finishing rollback tid 0 to smithi04526579-19 2024-04-26T00:31:54.578 INFO:tasks.rados.rados.0.smithi045.stdout:1999: finishing rollback tid 1 to smithi04526579-19 2024-04-26T00:31:54.578 INFO:tasks.rados.rados.0.smithi045.stdout:1999: finishing rollback tid 2 to smithi04526579-19 2024-04-26T00:31:54.578 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 1321 (ObjNum 534 snap 165 seq_num 534) dirty exists 2024-04-26T00:31:54.579 INFO:tasks.rados.rados.0.smithi045.stdout:2000: finishing write tid 1 to smithi04526579-26 2024-04-26T00:31:54.579 INFO:tasks.rados.rados.0.smithi045.stdout:2000: finishing write tid 2 to smithi04526579-26 2024-04-26T00:31:54.579 INFO:tasks.rados.rados.0.smithi045.stdout:2000: finishing write tid 3 to smithi04526579-26 2024-04-26T00:31:54.579 INFO:tasks.rados.rados.0.smithi045.stdout:2000: finishing write tid 4 to smithi04526579-26 2024-04-26T00:31:54.579 INFO:tasks.rados.rados.0.smithi045.stdout:2000: oid 26 updating version 0 to 1302 2024-04-26T00:31:54.579 INFO:tasks.rados.rados.0.smithi045.stdout:2000: oid 26 updating version 1302 to 1303 2024-04-26T00:31:54.579 INFO:tasks.rados.rados.0.smithi045.stdout:2000: oid 26 updating version 1303 to 1304 2024-04-26T00:31:54.579 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 1304 (ObjNum 731 snap 224 seq_num 731) dirty exists 2024-04-26T00:31:54.579 INFO:tasks.rados.rados.0.smithi045.stdout:2000: left oid 26 (ObjNum 731 snap 224 seq_num 731) 2024-04-26T00:31:54.579 INFO:tasks.rados.rados.0.smithi045.stdout:2001: finishing rollback tid 0 to smithi04526579-39 2024-04-26T00:31:54.579 INFO:tasks.rados.rados.0.smithi045.stdout:2001: finishing rollback tid 1 to smithi04526579-39 2024-04-26T00:31:54.579 INFO:tasks.rados.rados.0.smithi045.stdout:2001: finishing rollback tid 2 to smithi04526579-39 2024-04-26T00:31:54.579 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 1092 (ObjNum 656 snap 203 seq_num 656) dirty exists 2024-04-26T00:31:54.579 INFO:tasks.rados.rados.0.smithi045.stdout:1996: done (5 left) 2024-04-26T00:31:54.580 INFO:tasks.rados.rados.0.smithi045.stdout:1997: done (4 left) 2024-04-26T00:31:54.580 INFO:tasks.rados.rados.0.smithi045.stdout:1998: done (3 left) 2024-04-26T00:31:54.580 INFO:tasks.rados.rados.0.smithi045.stdout:1999: done (2 left) 2024-04-26T00:31:54.580 INFO:tasks.rados.rados.0.smithi045.stdout:2000: done (1 left) 2024-04-26T00:31:54.580 INFO:tasks.rados.rados.0.smithi045.stdout:2001: done (0 left) 2024-04-26T00:31:54.580 INFO:tasks.rados.rados.0.smithi045.stdout:2002: snap_remove snap 206 2024-04-26T00:31:55.139 INFO:tasks.rados.rados.0.smithi045.stdout:2002: done (0 left) 2024-04-26T00:31:55.139 INFO:tasks.rados.rados.0.smithi045.stdout:2003: snap_create 2024-04-26T00:31:56.140 INFO:tasks.rados.rados.0.smithi045.stdout:2003: done (0 left) 2024-04-26T00:31:56.140 INFO:tasks.rados.rados.0.smithi045.stdout:2004: rollback oid 34 current snap is 225 2024-04-26T00:31:56.141 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 34 to 153 2024-04-26T00:31:56.141 INFO:tasks.rados.rados.0.smithi045.stdout:2005: write oid 22 current snap is 225 2024-04-26T00:31:56.141 INFO:tasks.rados.rados.0.smithi045.stdout:2005: seq_num 732 ranges {702724=652156,1951525=351199} 2024-04-26T00:31:56.159 INFO:tasks.rados.rados.0.smithi045.stdout:2005: writing smithi04526579-22 from 702724 to 1354880 tid 1 2024-04-26T00:31:56.174 INFO:tasks.rados.rados.0.smithi045.stdout:2005: writing smithi04526579-22 from 1951525 to 2302724 tid 2 2024-04-26T00:31:56.174 INFO:tasks.rados.rados.0.smithi045.stdout:2006: copy_from oid 7 from oid 42 current snap is 225 2024-04-26T00:31:56.175 INFO:tasks.rados.rados.0.smithi045.stdout:2004: finishing rollback tid 0 to smithi04526579-34 2024-04-26T00:31:56.175 INFO:tasks.rados.rados.0.smithi045.stdout:2004: finishing rollback tid 1 to smithi04526579-34 2024-04-26T00:31:56.175 INFO:tasks.rados.rados.0.smithi045.stdout:2004: finishing rollback tid 2 to smithi04526579-34 2024-04-26T00:31:56.175 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 1199 (ObjNum 463 snap 145 seq_num 463) dirty exists 2024-04-26T00:31:56.175 INFO:tasks.rados.rados.0.smithi045.stdout:2005: finishing write tid 1 to smithi04526579-22 2024-04-26T00:31:56.175 INFO:tasks.rados.rados.0.smithi045.stdout:2004: done (2 left) 2024-04-26T00:31:56.176 INFO:tasks.rados.rados.0.smithi045.stdout:2007: snap_create 2024-04-26T00:31:56.181 INFO:tasks.rados.rados.0.smithi045.stdout:2005: finishing write tid 2 to smithi04526579-22 2024-04-26T00:31:56.183 INFO:tasks.rados.rados.0.smithi045.stdout:2005: finishing write tid 3 to smithi04526579-22 2024-04-26T00:31:56.184 INFO:tasks.rados.rados.0.smithi045.stdout:2005: finishing write tid 4 to smithi04526579-22 2024-04-26T00:31:56.184 INFO:tasks.rados.rados.0.smithi045.stdout:2005: oid 22 updating version 0 to 1440 2024-04-26T00:31:56.184 INFO:tasks.rados.rados.0.smithi045.stdout:2005: oid 22 version 1440 is already newer than 1439 2024-04-26T00:31:56.185 INFO:tasks.rados.rados.0.smithi045.stdout:2005: oid 22 version 1440 is already newer than 1438 2024-04-26T00:31:56.185 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 1440 (ObjNum 732 snap 225 seq_num 732) dirty exists 2024-04-26T00:31:56.185 INFO:tasks.rados.rados.0.smithi045.stdout:2005: left oid 22 (ObjNum 732 snap 225 seq_num 732) 2024-04-26T00:31:56.186 INFO:tasks.rados.rados.0.smithi045.stdout:2006: finishing copy_from to smithi04526579-7 2024-04-26T00:31:56.186 INFO:tasks.rados.rados.0.smithi045.stdout:2006: got expected ENOENT (src dne) 2024-04-26T00:31:56.186 INFO:tasks.rados.rados.0.smithi045.stdout:2006: finishing copy_from racing read to smithi04526579-7 2024-04-26T00:31:56.771 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:31:56.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.912+0000 7feafe6aa640 10 monclient: get_monmap_and_config 2024-04-26T00:31:56.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.912+0000 7feafe6aa640 10 monclient: build_initial_monmap 2024-04-26T00:31:56.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.912+0000 7feafe6aa640 10 monclient: monmap: 2024-04-26T00:31:56.907 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:31:56.907 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:56.907 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:31:56.917003+0000 2024-04-26T00:31:56.907 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:31:56.917003+0000 2024-04-26T00:31:56.907 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:31:56.907 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:56.907 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:31:56.907 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:31:56.907 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:31:56.907 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:31:56.908 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:31:56.908 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:56.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.912+0000 7feafe6aa640 1 Processor -- start 2024-04-26T00:31:56.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.912+0000 7feafe6aa640 1 -- start start 2024-04-26T00:31:56.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.912+0000 7feafe6aa640 10 monclient: init 2024-04-26T00:31:56.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.912+0000 7feafe6aa640 10 monclient: _reopen_session rank -1 2024-04-26T00:31:56.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.912+0000 7feafe6aa640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-04-26T00:31:56.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.912+0000 7feafe6aa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feaf80a0260 addr v1:172.21.15.37:6789/0 2024-04-26T00:31:56.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.912+0000 7feafe6aa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feaf8137e40 addr v1:172.21.15.45:6789/0 2024-04-26T00:31:56.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.912+0000 7feafe6aa640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7feaf813b950 0x7feaf813dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:56.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.912+0000 7feafe6aa640 10 monclient(hunting): picked mon.noname-c con 0x7feaf813b950 addr v2:172.21.15.45:3300/0 2024-04-26T00:31:56.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.912+0000 7feafe6aa640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feaf80a1130 con 0x7feaf80a0260 2024-04-26T00:31:56.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.912+0000 7feafe6aa640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feaf806cff0 con 0x7feaf8137e40 2024-04-26T00:31:56.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.912+0000 7feafe6aa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:56.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.912+0000 7feafe6aa640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7feaf80a0d50 con 0x7feaf813b950 2024-04-26T00:31:56.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.912+0000 7feafe6aa640 10 monclient(hunting): _renew_subs 2024-04-26T00:31:56.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.912+0000 7feafe6aa640 10 monclient(hunting): authenticate will time out at 1698.222956s 2024-04-26T00:31:56.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.912+0000 7feaf7fff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7feaf80a0260 0x7feaf80a0640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:40690/0 (socket says 172.21.15.37:40690) 2024-04-26T00:31:56.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.912+0000 7feaf7fff640 1 -- 172.21.15.37:0/2793983045 learned_addr learned my addr 172.21.15.37:0/2793983045 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:31:56.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.912+0000 7feafcc47640 1 --2- 172.21.15.37:0/2793983045 >> v2:172.21.15.45:3300/0 conn(0x7feaf813b950 0x7feaf813dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:56.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.912+0000 7feafcc47640 10 monclient(hunting): get_auth_request con 0x7feaf813b950 auth_method 0 2024-04-26T00:31:56.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.912+0000 7feafcc47640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:56.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.912+0000 7feafcc47640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:56.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.912+0000 7feafcc47640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:56.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.912+0000 7feafcc47640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:56.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.912+0000 7feafcc47640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:56.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.912+0000 7feafcc47640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:56.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.912+0000 7feafcc47640 10 monclient(hunting): handle_auth_done global_id 134176 payload 274 2024-04-26T00:31:56.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.912+0000 7feafcc47640 1 -- 172.21.15.37:0/2793983045 >> v1:172.21.15.45:6789/0 conn(0x7feaf8137e40 legacy=0x7feaf813a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:31:56.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.912+0000 7feafcc47640 1 -- 172.21.15.37:0/2793983045 >> v1:172.21.15.37:6789/0 conn(0x7feaf80a0260 legacy=0x7feaf80a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:31:56.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.912+0000 7feafcc47640 10 monclient: _finish_hunting 0 2024-04-26T00:31:56.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.912+0000 7feafcc47640 1 monclient: found mon.noname-c 2024-04-26T00:31:56.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.912+0000 7feafcc47640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:31:56.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.912+0000 7feafcc47640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:31:56.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.912+0000 7feafcc47640 1 -- 172.21.15.37:0/2793983045 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaf809bc50 con 0x7feaf813b950 2024-04-26T00:31:56.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feafcc47640 1 -- 172.21.15.37:0/2793983045 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feaf813b950 msgr2=0x7feaf813dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:56.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feafcc47640 1 --2- 172.21.15.37:0/2793983045 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feaf813b950 0x7feaf813dd40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7feaf00097c0 tx=0x7feaf001c100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f4c8bddb7b5771e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:56.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feaf6ffd640 1 -- 172.21.15.37:0/2793983045 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf00b1070 con 0x7feaf813b950 2024-04-26T00:31:56.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feaf6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:56.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feaf6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:31:56.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feaf6ffd640 10 monclient: dump: 2024-04-26T00:31:56.915 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:56.915 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:56.915 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:56.915 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:56.915 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:56.915 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:56.915 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:56.915 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:56.915 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:56.915 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:56.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feaf6ffd640 10 monclient: _finish_auth 0 2024-04-26T00:31:56.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feaf6ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:31:56.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feaf6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:31:56.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feaf6ffd640 1 -- 172.21.15.37:0/2793983045 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feaf00ac440 con 0x7feaf813b950 2024-04-26T00:31:56.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feaf6ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:31:56.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feaf6ffd640 1 -- 172.21.15.37:0/2793983045 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf00ac810 con 0x7feaf813b950 2024-04-26T00:31:56.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feaf6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:56.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feaf6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:31:56.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feaf6ffd640 10 monclient: dump: 2024-04-26T00:31:56.916 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:56.916 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:56.916 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:56.916 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:56.916 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:56.916 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:56.916 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:56.916 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:56.916 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:56.916 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:56.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feafe6aa640 5 monclient: authenticate success, global_id 134176 2024-04-26T00:31:56.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feafe6aa640 10 monclient: get_monmap_and_config success 2024-04-26T00:31:56.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feafe6aa640 4 set_mon_vals no callback set 2024-04-26T00:31:56.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feafe6aa640 10 monclient: shutdown 2024-04-26T00:31:56.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feafe6aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:31:56.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feafe6aa640 1 -- 172.21.15.37:0/2793983045 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feaf813b950 msgr2=0x7feaf813dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:56.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feafe6aa640 1 --2- 172.21.15.37:0/2793983045 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feaf813b950 0x7feaf813dd40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7feaf00097c0 tx=0x7feaf001c100 comp rx=0 tx=0).stop 2024-04-26T00:31:56.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feafe6aa640 1 -- 172.21.15.37:0/2793983045 shutdown_connections 2024-04-26T00:31:56.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feafe6aa640 1 --2- 172.21.15.37:0/2793983045 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feaf813b950 0x7feaf813dd40 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:56.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feafe6aa640 1 -- 172.21.15.37:0/2793983045 >> 172.21.15.37:0/2793983045 conn(0x7feaf80a1ad0 msgr2=0x7feaf80a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:31:56.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feafe6aa640 1 -- 172.21.15.37:0/2793983045 shutdown_connections 2024-04-26T00:31:56.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feafe6aa640 1 -- 172.21.15.37:0/2793983045 wait complete. 2024-04-26T00:31:56.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feafe6aa640 10 monclient: build_initial_monmap 2024-04-26T00:31:56.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feafe6aa640 10 monclient: monmap: 2024-04-26T00:31:56.920 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:31:56.920 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:31:56.920 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:31:56.920 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:31:56.921 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:31:56.921 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:31:56.921 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:31:56.921 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:31:56.921 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:31:56.921 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:56.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feafe6aa640 1 Processor -- start 2024-04-26T00:31:56.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feafe6aa640 1 -- start start 2024-04-26T00:31:56.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feafe6aa640 10 monclient: init 2024-04-26T00:31:56.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feafe6aa640 10 monclient: _reopen_session rank -1 2024-04-26T00:31:56.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feafe6aa640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:31:56.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feafe6aa640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feaf80a0260 0x7feaf81cccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:56.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feafe6aa640 10 monclient(hunting): picked mon.noname-a con 0x7feaf80a0260 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:31:56.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feafe6aa640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feaf8137e40 0x7feaf8136940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:56.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feafe6aa640 10 monclient(hunting): picked mon.noname-b con 0x7feaf8137e40 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:31:56.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feafe6aa640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feaf813b950 0x7feaf811c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:56.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feafe6aa640 10 monclient(hunting): picked mon.noname-c con 0x7feaf813b950 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:31:56.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feafe6aa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:56.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feafe6aa640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7feaf809f0f0 con 0x7feaf80a0260 2024-04-26T00:31:56.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feafe6aa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:56.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feafe6aa640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7feaf809f4c0 con 0x7feaf8137e40 2024-04-26T00:31:56.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feafe6aa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:31:56.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feafe6aa640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7feaf80a00a0 con 0x7feaf813b950 2024-04-26T00:31:56.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.920+0000 7feafe6aa640 10 monclient(hunting): _renew_subs 2024-04-26T00:31:56.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feafe6aa640 10 monclient(hunting): authenticate will time out at 1698.232015s 2024-04-26T00:31:56.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feaf7fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feaf80a0260 0x7feaf81cccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:56.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feafcc47640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feaf813b950 0x7feaf811c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:56.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feafcc47640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feaf813b950 0x7feaf811c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:50504/0 (socket says 172.21.15.37:50504) 2024-04-26T00:31:56.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feafcc47640 1 -- 172.21.15.37:0/3560246697 learned_addr learned my addr 172.21.15.37:0/3560246697 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:31:56.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feaf7fff640 10 monclient(hunting): get_auth_request con 0x7feaf80a0260 auth_method 0 2024-04-26T00:31:56.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feaf7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:56.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feaf7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:56.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feaf7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:56.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feafcc47640 10 monclient(hunting): get_auth_request con 0x7feaf813b950 auth_method 0 2024-04-26T00:31:56.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feafcc47640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:56.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feafcc47640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:56.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feafcc47640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:56.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feafcc47640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:31:56.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feafcc47640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:31:56.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feafcc47640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:31:56.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feaf7fff640 1 -- 172.21.15.37:0/3560246697 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feaf80a0260 msgr2=0x7feaf81cccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:31:56.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feaf7fff640 1 -- 172.21.15.37:0/3560246697 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feaf80a0260 msgr2=0x7feaf81cccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:31:56.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feaf7fff640 1 --2- 172.21.15.37:0/3560246697 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feaf80a0260 0x7feaf81cccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:31:56.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feaf77fe640 1 --2- 172.21.15.37:0/3560246697 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feaf8137e40 0x7feaf8136940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:56.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feaf77fe640 10 monclient(hunting): get_auth_request con 0x7feaf8137e40 auth_method 0 2024-04-26T00:31:56.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feaf77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:31:56.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feaf7fff640 1 --2- 172.21.15.37:0/3560246697 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feaf80a0260 0x7feaf81cccb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:31:56.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feaf77fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:31:56.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feaf77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:31:56.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feafcc47640 10 monclient(hunting): handle_auth_done global_id 84638 payload 824 2024-04-26T00:31:56.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feafcc47640 1 -- 172.21.15.37:0/3560246697 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feaf8137e40 msgr2=0x7feaf8136940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:56.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feafcc47640 1 --2- 172.21.15.37:0/3560246697 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feaf8137e40 0x7feaf8136940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:56.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feafcc47640 1 -- 172.21.15.37:0/3560246697 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feaf80a0260 msgr2=0x7feaf81cccb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:31:56.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feafcc47640 1 --2- 172.21.15.37:0/3560246697 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feaf80a0260 0x7feaf81cccb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:56.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feafcc47640 10 monclient: _finish_hunting 0 2024-04-26T00:31:56.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feafcc47640 1 monclient: found mon.noname-c 2024-04-26T00:31:56.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feafcc47640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:31:56.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feafcc47640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:31:56.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feafcc47640 1 -- 172.21.15.37:0/3560246697 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaf81202c0 con 0x7feaf813b950 2024-04-26T00:31:56.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feafcc47640 1 -- 172.21.15.37:0/3560246697 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feaf813b950 msgr2=0x7feaf811c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:31:56.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feafcc47640 1 --2- 172.21.15.37:0/3560246697 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feaf813b950 0x7feaf811c450 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7feaf001a7b0 tx=0x7feaf0018ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:56.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feaf4ff9640 1 -- 172.21.15.37:0/3560246697 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf00ba020 con 0x7feaf813b950 2024-04-26T00:31:56.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feaf4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:56.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feaf4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:31:56.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feaf4ff9640 10 monclient: dump: 2024-04-26T00:31:56.928 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:56.928 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:56.928 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:56.928 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:56.928 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:56.928 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:56.928 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:56.928 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:56.928 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:56.928 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:56.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feaf4ff9640 10 monclient: _finish_auth 0 2024-04-26T00:31:56.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feaf4ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:31:56.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feaf4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:31:56.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feafe6aa640 5 monclient: authenticate success, global_id 84638 2024-04-26T00:31:56.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feaf4ff9640 1 -- 172.21.15.37:0/3560246697 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feaf00b6890 con 0x7feaf813b950 2024-04-26T00:31:56.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feafe6aa640 10 monclient: _renew_subs 2024-04-26T00:31:56.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feafe6aa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:56.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feafe6aa640 1 -- 172.21.15.37:0/3560246697 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feaf813bda0 con 0x7feaf813b950 2024-04-26T00:31:56.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feaf4ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:31:56.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feaf4ff9640 1 -- 172.21.15.37:0/3560246697 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf00b1040 con 0x7feaf813b950 2024-04-26T00:31:56.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feaf4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:31:56.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feaf4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:31:56.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feaf4ff9640 10 monclient: dump: 2024-04-26T00:31:56.929 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:31:56.929 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:31:56.929 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:56.930 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:31:56.930 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:31:56.930 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:31:56.930 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:31:56.930 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:31:56.930 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:31:56.930 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:31:56.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feafe6aa640 10 monclient: _renew_subs 2024-04-26T00:31:56.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feafe6aa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:56.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feafe6aa640 1 -- 172.21.15.37:0/3560246697 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7feaf811fa20 con 0x7feaf813b950 2024-04-26T00:31:56.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.924+0000 7feaf67fc640 4 set_mon_vals no callback set 2024-04-26T00:31:56.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.928+0000 7feafe6aa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:56.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.928+0000 7feafe6aa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:56.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.928+0000 7feafe6aa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:56.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.928+0000 7feafe6aa640 1 -- 172.21.15.37:0/3560246697 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feaf8066f40 con 0x7feaf813b950 2024-04-26T00:31:56.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.928+0000 7feaf4ff9640 1 -- 172.21.15.37:0/3560246697 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7feaf00bd020 con 0x7feaf813b950 2024-04-26T00:31:56.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.928+0000 7feaf4ff9640 1 --2- 172.21.15.37:0/3560246697 >> v2:172.21.15.45:6812/2379918623 conn(0x7fead8035e80 0x7fead8038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:31:56.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.928+0000 7feaf4ff9640 1 -- 172.21.15.37:0/3560246697 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(459..459 src has 370..459) ==== 3931+0+0 (secure 0 0 0) 0x7feaf00e2be0 con 0x7feaf813b950 2024-04-26T00:31:56.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.928+0000 7feaf7fff640 1 --2- 172.21.15.37:0/3560246697 >> v2:172.21.15.45:6812/2379918623 conn(0x7fead8035e80 0x7fead8038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:31:56.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.928+0000 7feaf7fff640 10 monclient: get_auth_request con 0x7fead8035e80 auth_method 0 2024-04-26T00:31:56.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.928+0000 7feaf7fff640 1 --2- 172.21.15.37:0/3560246697 >> v2:172.21.15.45:6812/2379918623 conn(0x7fead8035e80 0x7fead8038340 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7feae8007d50 tx=0x7feae8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:31:56.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.932+0000 7feaf4ff9640 1 -- 172.21.15.37:0/3560246697 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7feaf00c4030 con 0x7feaf813b950 2024-04-26T00:31:56.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.932+0000 7feaf4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:31:56.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:56.932+0000 7feaf4ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:31:57.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:57.152+0000 7feafe6aa640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:31:57.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:57.152+0000 7feafe6aa640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:31:57.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:57.152+0000 7feafe6aa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:31:57.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:57.152+0000 7feafe6aa640 1 -- 172.21.15.37:0/3560246697 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7feaf813d110 con 0x7feaf813b950 2024-04-26T00:31:57.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:57.152+0000 7feaf4ff9640 1 -- 172.21.15.37:0/3560246697 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7feaf80a1130 con 0x7feaf813b950 2024-04-26T00:31:57.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:57.152+0000 7feaf4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:31:57.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:57.152+0000 7feaf4ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:31:57.149 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":90,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:31:57.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:57.156+0000 7feafe6aa640 1 -- 172.21.15.37:0/3560246697 >> v2:172.21.15.45:6812/2379918623 conn(0x7fead8035e80 msgr2=0x7fead8038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:57.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:57.156+0000 7feafe6aa640 1 --2- 172.21.15.37:0/3560246697 >> v2:172.21.15.45:6812/2379918623 conn(0x7fead8035e80 0x7fead8038340 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7feae8007d50 tx=0x7feae8009040 comp rx=0 tx=0).stop 2024-04-26T00:31:57.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:57.156+0000 7feafe6aa640 10 monclient: shutdown 2024-04-26T00:31:57.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:57.156+0000 7feafe6aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:31:57.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:57.156+0000 7feafe6aa640 1 -- 172.21.15.37:0/3560246697 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feaf813b950 msgr2=0x7feaf811c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:31:57.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:57.156+0000 7feafe6aa640 1 --2- 172.21.15.37:0/3560246697 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feaf813b950 0x7feaf811c450 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7feaf001a7b0 tx=0x7feaf0018ea0 comp rx=0 tx=0).stop 2024-04-26T00:31:57.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:57.156+0000 7feafe6aa640 1 -- 172.21.15.37:0/3560246697 shutdown_connections 2024-04-26T00:31:57.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:57.156+0000 7feafe6aa640 1 --2- 172.21.15.37:0/3560246697 >> v2:172.21.15.45:6812/2379918623 conn(0x7fead8035e80 0x7fead8038340 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:57.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:57.156+0000 7feafe6aa640 1 --2- 172.21.15.37:0/3560246697 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feaf813b950 0x7feaf811c450 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:57.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:57.156+0000 7feafe6aa640 1 --2- 172.21.15.37:0/3560246697 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feaf8137e40 0x7feaf8136940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:57.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:57.156+0000 7feafe6aa640 1 --2- 172.21.15.37:0/3560246697 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feaf80a0260 0x7feaf81cccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:31:57.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:57.156+0000 7feafe6aa640 1 -- 172.21.15.37:0/3560246697 >> 172.21.15.37:0/3560246697 conn(0x7feaf80a1ad0 msgr2=0x7feaf813c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:31:57.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:57.156+0000 7feafe6aa640 1 -- 172.21.15.37:0/3560246697 shutdown_connections 2024-04-26T00:31:57.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:31:57.156+0000 7feafe6aa640 1 -- 172.21.15.37:0/3560246697 wait complete. 2024-04-26T00:31:57.178 INFO:tasks.rados.rados.0.smithi045.stdout:2005: done (2 left) 2024-04-26T00:31:57.178 INFO:tasks.rados.rados.0.smithi045.stdout:2006: done (1 left) 2024-04-26T00:31:57.178 INFO:tasks.rados.rados.0.smithi045.stdout:2007: done (0 left) 2024-04-26T00:31:57.178 INFO:tasks.rados.rados.0.smithi045.stdout:2008: read oid 41 snap 211 2024-04-26T00:31:57.178 INFO:tasks.rados.rados.0.smithi045.stdout:2008: expect deleted 2024-04-26T00:31:57.179 INFO:tasks.rados.rados.0.smithi045.stdout:2009: rollback oid 9 current snap is 226 2024-04-26T00:31:57.179 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 9 to 212 2024-04-26T00:31:57.179 INFO:tasks.rados.rados.0.smithi045.stdout:2010: snap_create 2024-04-26T00:31:57.188 INFO:tasks.rados.rados.0.smithi045.stdout:2009: finishing rollback tid 0 to smithi04526579-9 2024-04-26T00:31:57.188 INFO:tasks.rados.rados.0.smithi045.stdout:2009: finishing rollback tid 1 to smithi04526579-9 2024-04-26T00:31:57.188 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 1517 (ObjNum 129 snap 0 seq_num 2242313128) dirty dne 2024-04-26T00:31:58.184 INFO:tasks.rados.rados.0.smithi045.stdout:2008: done (2 left) 2024-04-26T00:31:58.185 INFO:tasks.rados.rados.0.smithi045.stdout:2009: done (1 left) 2024-04-26T00:31:58.185 INFO:tasks.rados.rados.0.smithi045.stdout:2010: done (0 left) 2024-04-26T00:31:58.185 INFO:tasks.rados.rados.0.smithi045.stdout:2011: read oid 33 snap -1 2024-04-26T00:31:58.185 INFO:tasks.rados.rados.0.smithi045.stdout:2011: expect (ObjNum 721 snap 222 seq_num 721) 2024-04-26T00:31:58.185 INFO:tasks.rados.rados.0.smithi045.stdout:2012: rollback oid 29 current snap is 227 2024-04-26T00:31:58.185 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 29 to 175 2024-04-26T00:31:58.185 INFO:tasks.rados.rados.0.smithi045.stdout:2013: write oid 28 current snap is 227 2024-04-26T00:31:58.185 INFO:tasks.rados.rados.0.smithi045.stdout:2013: seq_num 733 ranges {719531=727405,1893654=425877} 2024-04-26T00:31:58.210 INFO:tasks.rados.rados.0.smithi045.stdout:2013: writing smithi04526579-28 from 719531 to 1446936 tid 1 2024-04-26T00:31:58.225 INFO:tasks.rados.rados.0.smithi045.stdout:2013: writing smithi04526579-28 from 1893654 to 2319531 tid 2 2024-04-26T00:31:58.225 INFO:tasks.rados.rados.0.smithi045.stdout:2014: write (excl) oid 42 current snap is 227 2024-04-26T00:31:58.225 INFO:tasks.rados.rados.0.smithi045.stdout:2014: seq_num 734 ranges {736338=402654,1552136=539580} 2024-04-26T00:31:58.242 INFO:tasks.rados.rados.0.smithi045.stdout:2014: writing smithi04526579-42 from 736338 to 1138992 tid 1 2024-04-26T00:31:58.254 INFO:tasks.rados.rados.0.smithi045.stdout:2014: writing smithi04526579-42 from 1552136 to 2091716 tid 2 2024-04-26T00:31:58.254 INFO:tasks.rados.rados.0.smithi045.stdout:2015: write oid 24 current snap is 227 2024-04-26T00:31:58.254 INFO:tasks.rados.rados.0.smithi045.stdout:2015: seq_num 735 ranges {753145=477903,2010618=342527} 2024-04-26T00:31:58.266 INFO:tasks.rados.rados.0.smithi045.stdout:2015: writing smithi04526579-24 from 753145 to 1231048 tid 1 2024-04-26T00:31:58.275 INFO:tasks.rados.rados.0.smithi045.stdout:2015: writing smithi04526579-24 from 2010618 to 2353145 tid 2 2024-04-26T00:31:58.276 INFO:tasks.rados.rados.0.smithi045.stdout:2016: rollback oid 16 current snap is 227 2024-04-26T00:31:58.276 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 16 to 175 2024-04-26T00:31:58.276 INFO:tasks.rados.rados.0.smithi045.stdout:2012: finishing rollback tid 1 to smithi04526579-29 2024-04-26T00:31:58.276 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 1062 (ObjNum 129 snap 0 seq_num 1141376017) dirty dne 2024-04-26T00:31:58.276 INFO:tasks.rados.rados.0.smithi045.stdout:2012: done (5 left) 2024-04-26T00:31:58.276 INFO:tasks.rados.rados.0.smithi045.stdout:2017: read oid 2 snap -1 2024-04-26T00:31:58.276 INFO:tasks.rados.rados.0.smithi045.stdout:2017: expect (ObjNum 668 snap 210 seq_num 668) 2024-04-26T00:31:58.276 INFO:tasks.rados.rados.0.smithi045.stdout:2018: delete oid 1 current snap is 227 2024-04-26T00:31:58.276 INFO:tasks.rados.rados.0.smithi045.stdout:2011: expect (ObjNum 721 snap 222 seq_num 721) 2024-04-26T00:31:58.505 INFO:tasks.rados.rados.0.smithi045.stdout:2013: finishing write tid 1 to smithi04526579-28 2024-04-26T00:31:58.505 INFO:tasks.rados.rados.0.smithi045.stdout:2013: finishing write tid 2 to smithi04526579-28 2024-04-26T00:31:58.505 INFO:tasks.rados.rados.0.smithi045.stdout:2013: finishing write tid 3 to smithi04526579-28 2024-04-26T00:31:58.505 INFO:tasks.rados.rados.0.smithi045.stdout:2013: finishing write tid 4 to smithi04526579-28 2024-04-26T00:31:58.505 INFO:tasks.rados.rados.0.smithi045.stdout:2013: oid 28 updating version 0 to 1312 2024-04-26T00:31:58.505 INFO:tasks.rados.rados.0.smithi045.stdout:2013: oid 28 updating version 1312 to 1313 2024-04-26T00:31:58.505 INFO:tasks.rados.rados.0.smithi045.stdout:2013: oid 28 version 1313 is already newer than 1311 2024-04-26T00:31:58.505 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 1313 (ObjNum 733 snap 227 seq_num 733) dirty exists 2024-04-26T00:31:58.506 INFO:tasks.rados.rados.0.smithi045.stdout:2013: left oid 28 (ObjNum 733 snap 227 seq_num 733) 2024-04-26T00:31:58.506 INFO:tasks.rados.rados.0.smithi045.stdout:2014: finishing write tid 1 to smithi04526579-42 2024-04-26T00:31:58.506 INFO:tasks.rados.rados.0.smithi045.stdout:2014: finishing write tid 2 to smithi04526579-42 2024-04-26T00:31:58.506 INFO:tasks.rados.rados.0.smithi045.stdout:2014: finishing write tid 3 to smithi04526579-42 2024-04-26T00:31:58.506 INFO:tasks.rados.rados.0.smithi045.stdout:2014: finishing write tid 4 to smithi04526579-42 2024-04-26T00:31:58.506 INFO:tasks.rados.rados.0.smithi045.stdout:2014: oid 42 updating version 0 to 1331 2024-04-26T00:31:58.506 INFO:tasks.rados.rados.0.smithi045.stdout:2014: oid 42 version 1331 is already newer than 1330 2024-04-26T00:31:58.506 INFO:tasks.rados.rados.0.smithi045.stdout:2014: oid 42 updating version 1331 to 1332 2024-04-26T00:31:58.506 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 1332 (ObjNum 734 snap 227 seq_num 734) dirty exists 2024-04-26T00:31:58.506 INFO:tasks.rados.rados.0.smithi045.stdout:2014: left oid 42 (ObjNum 734 snap 227 seq_num 734) 2024-04-26T00:31:58.506 INFO:tasks.rados.rados.0.smithi045.stdout:2011: done (6 left) 2024-04-26T00:31:58.506 INFO:tasks.rados.rados.0.smithi045.stdout:2013: done (5 left) 2024-04-26T00:31:58.506 INFO:tasks.rados.rados.0.smithi045.stdout:2014: done (4 left) 2024-04-26T00:31:58.506 INFO:tasks.rados.rados.0.smithi045.stdout:2018: done (3 left) 2024-04-26T00:31:58.507 INFO:tasks.rados.rados.0.smithi045.stdout:2019: snap_create 2024-04-26T00:31:58.507 INFO:tasks.rados.rados.0.smithi045.stdout:2016: finishing rollback tid 0 to smithi04526579-16 2024-04-26T00:31:58.507 INFO:tasks.rados.rados.0.smithi045.stdout:2016: finishing rollback tid 1 to smithi04526579-16 2024-04-26T00:31:58.507 INFO:tasks.rados.rados.0.smithi045.stdout:2016: finishing rollback tid 2 to smithi04526579-16 2024-04-26T00:31:58.507 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 1134 (ObjNum 515 snap 157 seq_num 515) dirty exists 2024-04-26T00:31:58.507 INFO:tasks.rados.rados.0.smithi045.stdout:2015: finishing write tid 1 to smithi04526579-24 2024-04-26T00:31:58.507 INFO:tasks.rados.rados.0.smithi045.stdout:2015: finishing write tid 2 to smithi04526579-24 2024-04-26T00:31:58.507 INFO:tasks.rados.rados.0.smithi045.stdout:2015: finishing write tid 3 to smithi04526579-24 2024-04-26T00:31:58.507 INFO:tasks.rados.rados.0.smithi045.stdout:2015: finishing write tid 4 to smithi04526579-24 2024-04-26T00:31:58.507 INFO:tasks.rados.rados.0.smithi045.stdout:2015: oid 24 updating version 0 to 1704 2024-04-26T00:31:58.507 INFO:tasks.rados.rados.0.smithi045.stdout:2015: oid 24 version 1704 is already newer than 1702 2024-04-26T00:31:58.507 INFO:tasks.rados.rados.0.smithi045.stdout:2015: oid 24 version 1704 is already newer than 1703 2024-04-26T00:31:58.507 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 1704 (ObjNum 735 snap 227 seq_num 735) dirty exists 2024-04-26T00:31:58.507 INFO:tasks.rados.rados.0.smithi045.stdout:2015: left oid 24 (ObjNum 735 snap 227 seq_num 735) 2024-04-26T00:31:58.507 INFO:tasks.rados.rados.0.smithi045.stdout:2017: expect (ObjNum 668 snap 210 seq_num 668) 2024-04-26T00:31:59.193 INFO:tasks.rados.rados.0.smithi045.stdout:2015: done (3 left) 2024-04-26T00:31:59.193 INFO:tasks.rados.rados.0.smithi045.stdout:2016: done (2 left) 2024-04-26T00:31:59.193 INFO:tasks.rados.rados.0.smithi045.stdout:2017: done (1 left) 2024-04-26T00:31:59.193 INFO:tasks.rados.rados.0.smithi045.stdout:2019: done (0 left) 2024-04-26T00:31:59.193 INFO:tasks.rados.rados.0.smithi045.stdout:2020: snap_remove snap 202 2024-04-26T00:32:00.141 INFO:tasks.rados.rados.0.smithi045.stdout:2020: done (0 left) 2024-04-26T00:32:00.141 INFO:tasks.rados.rados.0.smithi045.stdout:2021: write (excl) oid 5 current snap is 228 2024-04-26T00:32:00.141 INFO:tasks.rados.rados.0.smithi045.stdout:2021: seq_num 737 ranges {786759=628401,2011229=375530} 2024-04-26T00:32:00.163 INFO:tasks.rados.rados.0.smithi045.stdout:2021: writing smithi04526579-5 from 786759 to 1415160 tid 1 2024-04-26T00:32:00.168 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:32:00.176 INFO:tasks.rados.rados.0.smithi045.stdout:2021: writing smithi04526579-5 from 2011229 to 2386759 tid 2 2024-04-26T00:32:00.176 INFO:tasks.rados.rados.0.smithi045.stdout:2022: delete oid 9 current snap is 228 2024-04-26T00:32:00.187 INFO:tasks.rados.rados.0.smithi045.stdout:2021: finishing write tid 1 to smithi04526579-5 2024-04-26T00:32:00.187 INFO:tasks.rados.rados.0.smithi045.stdout:2022: done (1 left) 2024-04-26T00:32:00.187 INFO:tasks.rados.rados.0.smithi045.stdout:2023: delete oid 31 current snap is 228 2024-04-26T00:32:00.188 INFO:tasks.rados.rados.0.smithi045.stdout:2021: finishing write tid 2 to smithi04526579-5 2024-04-26T00:32:00.193 INFO:tasks.rados.rados.0.smithi045.stdout:2021: finishing write tid 3 to smithi04526579-5 2024-04-26T00:32:00.193 INFO:tasks.rados.rados.0.smithi045.stdout:2021: finishing write tid 4 to smithi04526579-5 2024-04-26T00:32:00.193 INFO:tasks.rados.rados.0.smithi045.stdout:2021: oid 5 updating version 0 to 1711 2024-04-26T00:32:00.193 INFO:tasks.rados.rados.0.smithi045.stdout:2021: oid 5 version 1711 is already newer than 1706 2024-04-26T00:32:00.193 INFO:tasks.rados.rados.0.smithi045.stdout:2021: oid 5 updating version 1711 to 1712 2024-04-26T00:32:00.193 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 1712 (ObjNum 737 snap 228 seq_num 737) dirty exists 2024-04-26T00:32:00.193 INFO:tasks.rados.rados.0.smithi045.stdout:2021: left oid 5 (ObjNum 737 snap 228 seq_num 737) 2024-04-26T00:32:00.194 INFO:tasks.rados.rados.0.smithi045.stdout:2021: done (1 left) 2024-04-26T00:32:00.194 INFO:tasks.rados.rados.0.smithi045.stdout:2023: done (0 left) 2024-04-26T00:32:00.194 INFO:tasks.rados.rados.0.smithi045.stdout:2024: snap_create 2024-04-26T00:32:00.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.304+0000 7fde55a23640 10 monclient: get_monmap_and_config 2024-04-26T00:32:00.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.304+0000 7fde55a23640 10 monclient: build_initial_monmap 2024-04-26T00:32:00.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.304+0000 7fde55a23640 10 monclient: monmap: 2024-04-26T00:32:00.299 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:00.299 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:00.299 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:32:00.309590+0000 2024-04-26T00:32:00.299 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:32:00.309590+0000 2024-04-26T00:32:00.300 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:00.300 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:00.300 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:32:00.300 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:32:00.300 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:32:00.300 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:32:00.300 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:32:00.300 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:00.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.304+0000 7fde55a23640 1 Processor -- start 2024-04-26T00:32:00.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.304+0000 7fde55a23640 1 -- start start 2024-04-26T00:32:00.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.304+0000 7fde55a23640 10 monclient: init 2024-04-26T00:32:00.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.304+0000 7fde55a23640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:00.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.304+0000 7fde55a23640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-04-26T00:32:00.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.304+0000 7fde55a23640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fde50136da0 0x7fde5009bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:00.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.304+0000 7fde55a23640 10 monclient(hunting): picked mon.noname-c con 0x7fde50136da0 addr v2:172.21.15.45:3300/0 2024-04-26T00:32:00.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.304+0000 7fde55a23640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fde5009c410 0x7fde50137180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:00.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.304+0000 7fde55a23640 10 monclient(hunting): picked mon.noname-a con 0x7fde5009c410 addr v2:172.21.15.37:3300/0 2024-04-26T00:32:00.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.304+0000 7fde55a23640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fde501376c0 addr v1:172.21.15.37:6789/0 2024-04-26T00:32:00.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.304+0000 7fde55a23640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde5009b4f0 con 0x7fde501376c0 2024-04-26T00:32:00.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.304+0000 7fde55a23640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:00.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.304+0000 7fde55a23640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fde5009b210 con 0x7fde5009c410 2024-04-26T00:32:00.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.304+0000 7fde55a23640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:00.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.304+0000 7fde55a23640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fde5009b7d0 con 0x7fde50136da0 2024-04-26T00:32:00.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.304+0000 7fde55a23640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:00.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.304+0000 7fde55a23640 10 monclient(hunting): authenticate will time out at 1701.615320s 2024-04-26T00:32:00.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.304+0000 7fde4e7fc640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fde5009c410 0x7fde50137180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:00.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.304+0000 7fde4f7fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fde501376c0 0x7fde50133b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:48624/0 (socket says 172.21.15.37:48624) 2024-04-26T00:32:00.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.304+0000 7fde4f7fe640 1 -- 172.21.15.37:0/2085417668 learned_addr learned my addr 172.21.15.37:0/2085417668 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:32:00.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.304+0000 7fde4e7fc640 10 monclient(hunting): get_auth_request con 0x7fde5009c410 auth_method 0 2024-04-26T00:32:00.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.304+0000 7fde4e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:00.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.304+0000 7fde4e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:00.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.304+0000 7fde4e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:00.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.304+0000 7fde4e7fc640 1 -- 172.21.15.37:0/2085417668 >> v2:172.21.15.37:3300/0 conn(0x7fde5009c410 msgr2=0x7fde50137180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:32:00.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.304+0000 7fde4e7fc640 1 -- 172.21.15.37:0/2085417668 >> v2:172.21.15.37:3300/0 conn(0x7fde5009c410 msgr2=0x7fde50137180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:32:00.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.304+0000 7fde4e7fc640 1 --2- 172.21.15.37:0/2085417668 >> v2:172.21.15.37:3300/0 conn(0x7fde5009c410 0x7fde50137180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:32:00.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.304+0000 7fde4effd640 1 --2- 172.21.15.37:0/2085417668 >> v2:172.21.15.45:3300/0 conn(0x7fde50136da0 0x7fde5009bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:00.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.304+0000 7fde4e7fc640 1 --2- 172.21.15.37:0/2085417668 >> v2:172.21.15.37:3300/0 conn(0x7fde5009c410 0x7fde50137180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:32:00.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.304+0000 7fde4effd640 10 monclient(hunting): get_auth_request con 0x7fde50136da0 auth_method 0 2024-04-26T00:32:00.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.304+0000 7fde4effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:00.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.304+0000 7fde4effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:00.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.304+0000 7fde4effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:00.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.308+0000 7fde4effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:00.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.308+0000 7fde4effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:00.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.308+0000 7fde4effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:00.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.308+0000 7fde4effd640 10 monclient(hunting): handle_auth_done global_id 134182 payload 274 2024-04-26T00:32:00.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.308+0000 7fde4effd640 1 -- 172.21.15.37:0/2085417668 >> v2:172.21.15.37:3300/0 conn(0x7fde5009c410 msgr2=0x7fde50137180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:32:00.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.308+0000 7fde4effd640 1 --2- 172.21.15.37:0/2085417668 >> v2:172.21.15.37:3300/0 conn(0x7fde5009c410 0x7fde50137180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:00.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.308+0000 7fde4effd640 1 -- 172.21.15.37:0/2085417668 >> v1:172.21.15.37:6789/0 conn(0x7fde501376c0 legacy=0x7fde50133b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:00.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.308+0000 7fde4effd640 10 monclient: _finish_hunting 0 2024-04-26T00:32:00.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.308+0000 7fde4effd640 1 monclient: found mon.noname-c 2024-04-26T00:32:00.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.308+0000 7fde4effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:00.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.308+0000 7fde4effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:32:00.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.308+0000 7fde4effd640 1 -- 172.21.15.37:0/2085417668 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde50137b30 con 0x7fde50136da0 2024-04-26T00:32:00.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.312+0000 7fde4effd640 1 -- 172.21.15.37:0/2085417668 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde50136da0 msgr2=0x7fde5009bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:00.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.312+0000 7fde4effd640 1 --2- 172.21.15.37:0/2085417668 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde50136da0 0x7fde5009bed0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fde500a58a0 tx=0x7fde4001c390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=231802e2ee6b6444 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:00.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.312+0000 7fde4dffb640 1 -- 172.21.15.37:0/2085417668 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde400b0070 con 0x7fde50136da0 2024-04-26T00:32:00.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.312+0000 7fde4dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:00.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.312+0000 7fde4dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:32:00.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.312+0000 7fde4dffb640 10 monclient: dump: 2024-04-26T00:32:00.308 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:00.309 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:00.309 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:00.309 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:00.309 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:00.309 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:00.309 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:00.309 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:00.309 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:00.309 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:00.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.312+0000 7fde4dffb640 10 monclient: _finish_auth 0 2024-04-26T00:32:00.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.312+0000 7fde4dffb640 10 monclient: _check_auth_tickets 2024-04-26T00:32:00.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.312+0000 7fde4dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:00.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.312+0000 7fde4dffb640 1 -- 172.21.15.37:0/2085417668 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde400ab600 con 0x7fde50136da0 2024-04-26T00:32:00.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.312+0000 7fde4dffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:00.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.312+0000 7fde4dffb640 1 -- 172.21.15.37:0/2085417668 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde400ab9d0 con 0x7fde50136da0 2024-04-26T00:32:00.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.312+0000 7fde4dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:00.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.312+0000 7fde4dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:32:00.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.312+0000 7fde4dffb640 10 monclient: dump: 2024-04-26T00:32:00.310 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:00.310 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:00.310 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:00.310 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:00.310 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:00.310 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:00.310 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:00.310 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:00.310 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:00.310 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:00.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.312+0000 7fde55a23640 5 monclient: authenticate success, global_id 134182 2024-04-26T00:32:00.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.312+0000 7fde55a23640 10 monclient: get_monmap_and_config success 2024-04-26T00:32:00.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.312+0000 7fde55a23640 4 set_mon_vals no callback set 2024-04-26T00:32:00.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde55a23640 10 monclient: shutdown 2024-04-26T00:32:00.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde55a23640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:00.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde55a23640 1 -- 172.21.15.37:0/2085417668 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde50136da0 msgr2=0x7fde5009bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:00.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde55a23640 1 --2- 172.21.15.37:0/2085417668 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde50136da0 0x7fde5009bed0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fde500a58a0 tx=0x7fde4001c390 comp rx=0 tx=0).stop 2024-04-26T00:32:00.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde55a23640 1 -- 172.21.15.37:0/2085417668 shutdown_connections 2024-04-26T00:32:00.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde55a23640 1 --2- 172.21.15.37:0/2085417668 >> v2:172.21.15.37:3300/0 conn(0x7fde5009c410 0x7fde50137180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:00.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde55a23640 1 --2- 172.21.15.37:0/2085417668 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde50136da0 0x7fde5009bed0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:00.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde55a23640 1 -- 172.21.15.37:0/2085417668 >> 172.21.15.37:0/2085417668 conn(0x7fde500a5f00 msgr2=0x7fde500a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:00.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde55a23640 1 -- 172.21.15.37:0/2085417668 shutdown_connections 2024-04-26T00:32:00.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde55a23640 1 -- 172.21.15.37:0/2085417668 wait complete. 2024-04-26T00:32:00.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde55a23640 10 monclient: build_initial_monmap 2024-04-26T00:32:00.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde55a23640 10 monclient: monmap: 2024-04-26T00:32:00.314 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:00.314 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:32:00.314 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:32:00.314 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:32:00.314 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:00.314 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:32:00.314 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:32:00.314 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:32:00.315 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:32:00.315 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:00.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde55a23640 1 Processor -- start 2024-04-26T00:32:00.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde55a23640 1 -- start start 2024-04-26T00:32:00.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde55a23640 10 monclient: init 2024-04-26T00:32:00.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde55a23640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:00.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde55a23640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:32:00.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde55a23640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde5009c410 0x7fde5009fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:00.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde55a23640 10 monclient(hunting): picked mon.noname-c con 0x7fde5009c410 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:00.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde55a23640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde501376c0 0x7fde5012cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:00.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde55a23640 10 monclient(hunting): picked mon.noname-a con 0x7fde501376c0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:32:00.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde55a23640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde5012d420 0x7fde501cf850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:00.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde55a23640 10 monclient(hunting): picked mon.noname-b con 0x7fde5012d420 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:32:00.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde55a23640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:00.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde55a23640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fde50141ea0 con 0x7fde501376c0 2024-04-26T00:32:00.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde55a23640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:00.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde55a23640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fde50141d20 con 0x7fde5012d420 2024-04-26T00:32:00.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde55a23640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:00.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde55a23640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fde50142020 con 0x7fde5009c410 2024-04-26T00:32:00.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde55a23640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:00.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde55a23640 10 monclient(hunting): authenticate will time out at 1701.626086s 2024-04-26T00:32:00.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde4e7fc640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde501376c0 0x7fde5012cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:00.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde4e7fc640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde501376c0 0x7fde5012cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:34954/0 (socket says 172.21.15.37:34954) 2024-04-26T00:32:00.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde4e7fc640 1 -- 172.21.15.37:0/3449449308 learned_addr learned my addr 172.21.15.37:0/3449449308 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:32:00.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde4e7fc640 10 monclient(hunting): get_auth_request con 0x7fde501376c0 auth_method 0 2024-04-26T00:32:00.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde4e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:00.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde4e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:00.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde4e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:00.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde4effd640 1 --2- 172.21.15.37:0/3449449308 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde5009c410 0x7fde5009fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:00.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde4effd640 10 monclient(hunting): get_auth_request con 0x7fde5009c410 auth_method 0 2024-04-26T00:32:00.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde4effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:00.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde4effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:00.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde4effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:00.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde4e7fc640 1 -- 172.21.15.37:0/3449449308 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde501376c0 msgr2=0x7fde5012cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:32:00.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde4e7fc640 1 -- 172.21.15.37:0/3449449308 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde501376c0 msgr2=0x7fde5012cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:32:00.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde4e7fc640 1 --2- 172.21.15.37:0/3449449308 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde501376c0 0x7fde5012cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:32:00.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde4e7fc640 1 --2- 172.21.15.37:0/3449449308 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde501376c0 0x7fde5012cee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:32:00.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde4effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:00.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde4effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:00.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde4effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:00.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde4f7fe640 1 --2- 172.21.15.37:0/3449449308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde5012d420 0x7fde501cf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:00.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde4f7fe640 10 monclient(hunting): get_auth_request con 0x7fde5012d420 auth_method 0 2024-04-26T00:32:00.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde4f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:00.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde4f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:00.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:00.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde4effd640 10 monclient(hunting): handle_auth_done global_id 84641 payload 824 2024-04-26T00:32:00.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde4effd640 1 -- 172.21.15.37:0/3449449308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde5012d420 msgr2=0x7fde501cf850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:00.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde4effd640 1 --2- 172.21.15.37:0/3449449308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde5012d420 0x7fde501cf850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:00.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde4effd640 1 -- 172.21.15.37:0/3449449308 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde501376c0 msgr2=0x7fde5012cee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:32:00.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde4effd640 1 --2- 172.21.15.37:0/3449449308 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde501376c0 0x7fde5012cee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:00.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde4effd640 10 monclient: _finish_hunting 0 2024-04-26T00:32:00.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde4effd640 1 monclient: found mon.noname-c 2024-04-26T00:32:00.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde4effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:00.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde4effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:32:00.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde4effd640 1 -- 172.21.15.37:0/3449449308 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde500a8d50 con 0x7fde5009c410 2024-04-26T00:32:00.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.316+0000 7fde4f7fe640 1 --2- 172.21.15.37:0/3449449308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde5012d420 0x7fde501cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:32:00.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.320+0000 7fde4effd640 1 -- 172.21.15.37:0/3449449308 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde5009c410 msgr2=0x7fde5009fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:00.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.320+0000 7fde4effd640 1 --2- 172.21.15.37:0/3449449308 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde5009c410 0x7fde5009fe20 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fde400ab000 tx=0x7fde40019130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:00.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.320+0000 7fde37fff640 1 -- 172.21.15.37:0/3449449308 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde400b8020 con 0x7fde5009c410 2024-04-26T00:32:00.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.320+0000 7fde37fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:00.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.320+0000 7fde37fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:32:00.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.320+0000 7fde37fff640 10 monclient: dump: 2024-04-26T00:32:00.321 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:00.322 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:00.322 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:00.322 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:00.322 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:00.322 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:00.322 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:00.322 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:00.322 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:00.322 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:00.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.320+0000 7fde37fff640 10 monclient: _finish_auth 0 2024-04-26T00:32:00.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.320+0000 7fde37fff640 10 monclient: _check_auth_tickets 2024-04-26T00:32:00.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.320+0000 7fde37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:00.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.320+0000 7fde55a23640 5 monclient: authenticate success, global_id 84641 2024-04-26T00:32:00.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.320+0000 7fde37fff640 1 -- 172.21.15.37:0/3449449308 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde400abe30 con 0x7fde5009c410 2024-04-26T00:32:00.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.320+0000 7fde55a23640 10 monclient: _renew_subs 2024-04-26T00:32:00.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.320+0000 7fde55a23640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:32:00.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.320+0000 7fde55a23640 1 -- 172.21.15.37:0/3449449308 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde5009d340 con 0x7fde5009c410 2024-04-26T00:32:00.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.320+0000 7fde37fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:00.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.320+0000 7fde37fff640 1 -- 172.21.15.37:0/3449449308 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde400b0040 con 0x7fde5009c410 2024-04-26T00:32:00.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.320+0000 7fde37fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:00.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.320+0000 7fde37fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:32:00.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.320+0000 7fde37fff640 10 monclient: dump: 2024-04-26T00:32:00.323 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:00.323 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:00.323 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:00.323 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:00.323 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:00.323 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:00.324 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:00.324 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:00.324 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:00.324 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:00.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.320+0000 7fde4d7fa640 4 set_mon_vals no callback set 2024-04-26T00:32:00.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.320+0000 7fde55a23640 10 monclient: _renew_subs 2024-04-26T00:32:00.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.320+0000 7fde55a23640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:32:00.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.320+0000 7fde55a23640 1 -- 172.21.15.37:0/3449449308 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fde5009f6a0 con 0x7fde5009c410 2024-04-26T00:32:00.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.320+0000 7fde55a23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:00.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.320+0000 7fde55a23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:00.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.320+0000 7fde55a23640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:32:00.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.320+0000 7fde55a23640 1 -- 172.21.15.37:0/3449449308 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde50066f40 con 0x7fde5009c410 2024-04-26T00:32:00.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.320+0000 7fde37fff640 1 -- 172.21.15.37:0/3449449308 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fde400bb020 con 0x7fde5009c410 2024-04-26T00:32:00.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.320+0000 7fde37fff640 1 --2- 172.21.15.37:0/3449449308 >> v2:172.21.15.45:6812/2379918623 conn(0x7fde3c035e80 0x7fde3c038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:00.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.320+0000 7fde37fff640 1 -- 172.21.15.37:0/3449449308 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(463..463 src has 370..463) ==== 3987+0+0 (secure 0 0 0) 0x7fde400e28d0 con 0x7fde5009c410 2024-04-26T00:32:00.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.320+0000 7fde4e7fc640 1 --2- 172.21.15.37:0/3449449308 >> v2:172.21.15.45:6812/2379918623 conn(0x7fde3c035e80 0x7fde3c038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:00.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.320+0000 7fde4e7fc640 10 monclient: get_auth_request con 0x7fde3c035e80 auth_method 0 2024-04-26T00:32:00.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.324+0000 7fde4e7fc640 1 --2- 172.21.15.37:0/3449449308 >> v2:172.21.15.45:6812/2379918623 conn(0x7fde3c035e80 0x7fde3c038340 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fde501d0910 tx=0x7fde440091a0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:00.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.324+0000 7fde37fff640 1 -- 172.21.15.37:0/3449449308 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fde400c2030 con 0x7fde5009c410 2024-04-26T00:32:00.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.324+0000 7fde37fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:00.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.324+0000 7fde37fff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:32:00.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.548+0000 7fde55a23640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:32:00.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.548+0000 7fde55a23640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:32:00.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.548+0000 7fde55a23640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:32:00.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.548+0000 7fde55a23640 1 -- 172.21.15.37:0/3449449308 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fde500a59e0 con 0x7fde5009c410 2024-04-26T00:32:00.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.548+0000 7fde37fff640 1 -- 172.21.15.37:0/3449449308 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fde40016010 con 0x7fde5009c410 2024-04-26T00:32:00.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.548+0000 7fde37fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:32:00.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.548+0000 7fde37fff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:32:00.544 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":92,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:32:00.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.552+0000 7fde55a23640 1 -- 172.21.15.37:0/3449449308 >> v2:172.21.15.45:6812/2379918623 conn(0x7fde3c035e80 msgr2=0x7fde3c038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:00.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.552+0000 7fde55a23640 1 --2- 172.21.15.37:0/3449449308 >> v2:172.21.15.45:6812/2379918623 conn(0x7fde3c035e80 0x7fde3c038340 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fde501d0910 tx=0x7fde440091a0 comp rx=0 tx=0).stop 2024-04-26T00:32:00.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.552+0000 7fde55a23640 10 monclient: shutdown 2024-04-26T00:32:00.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.552+0000 7fde55a23640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:00.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.552+0000 7fde55a23640 1 -- 172.21.15.37:0/3449449308 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde5009c410 msgr2=0x7fde5009fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:00.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.552+0000 7fde55a23640 1 --2- 172.21.15.37:0/3449449308 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde5009c410 0x7fde5009fe20 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fde400ab000 tx=0x7fde40019130 comp rx=0 tx=0).stop 2024-04-26T00:32:00.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.552+0000 7fde55a23640 1 -- 172.21.15.37:0/3449449308 shutdown_connections 2024-04-26T00:32:00.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.552+0000 7fde55a23640 1 --2- 172.21.15.37:0/3449449308 >> v2:172.21.15.45:6812/2379918623 conn(0x7fde3c035e80 0x7fde3c038340 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:00.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.552+0000 7fde55a23640 1 --2- 172.21.15.37:0/3449449308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde5012d420 0x7fde501cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:00.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.552+0000 7fde55a23640 1 --2- 172.21.15.37:0/3449449308 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde501376c0 0x7fde5012cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:00.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.552+0000 7fde55a23640 1 --2- 172.21.15.37:0/3449449308 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde5009c410 0x7fde5009fe20 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:00.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.552+0000 7fde55a23640 1 -- 172.21.15.37:0/3449449308 >> 172.21.15.37:0/3449449308 conn(0x7fde500a5f00 msgr2=0x7fde501355e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:00.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.552+0000 7fde55a23640 1 -- 172.21.15.37:0/3449449308 shutdown_connections 2024-04-26T00:32:00.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.552+0000 7fde55a23640 1 -- 172.21.15.37:0/3449449308 wait complete. 2024-04-26T00:32:00.563 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:32:00.564 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:32:00.564 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:32:00.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.696+0000 7fd736a0b640 10 monclient: get_monmap_and_config 2024-04-26T00:32:00.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.696+0000 7fd736a0b640 10 monclient: build_initial_monmap 2024-04-26T00:32:00.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.696+0000 7fd736a0b640 10 monclient: monmap: 2024-04-26T00:32:00.692 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:00.692 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:00.692 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:32:00.702441+0000 2024-04-26T00:32:00.692 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:32:00.702441+0000 2024-04-26T00:32:00.692 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:00.692 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:00.693 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:32:00.693 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:32:00.693 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:32:00.693 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:32:00.693 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:32:00.693 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:00.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.696+0000 7fd736a0b640 1 Processor -- start 2024-04-26T00:32:00.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.696+0000 7fd736a0b640 1 -- start start 2024-04-26T00:32:00.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.696+0000 7fd736a0b640 10 monclient: init 2024-04-26T00:32:00.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.700+0000 7fd736a0b640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:00.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.700+0000 7fd736a0b640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-04-26T00:32:00.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.700+0000 7fd736a0b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd73009e220 0x7fd73009fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:00.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.700+0000 7fd736a0b640 10 monclient(hunting): picked mon.noname-c con 0x7fd73009e220 addr v2:172.21.15.45:3300/0 2024-04-26T00:32:00.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.700+0000 7fd736a0b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7300a14f0 0x7fd7300a0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:00.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.700+0000 7fd736a0b640 10 monclient(hunting): picked mon.noname-b con 0x7fd7300a14f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:00.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.700+0000 7fd736a0b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd73009e600 addr v1:172.21.15.37:6789/0 2024-04-26T00:32:00.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.700+0000 7fd736a0b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7300a12b0 con 0x7fd73009e600 2024-04-26T00:32:00.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.700+0000 7fd736a0b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:00.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.700+0000 7fd736a0b640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fd73009cd10 con 0x7fd73009e220 2024-04-26T00:32:00.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.700+0000 7fd736a0b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:00.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.700+0000 7fd736a0b640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd73009da20 con 0x7fd7300a14f0 2024-04-26T00:32:00.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.700+0000 7fd736a0b640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:00.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.700+0000 7fd736a0b640 10 monclient(hunting): authenticate will time out at 1702.008657s 2024-04-26T00:32:00.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.700+0000 7fd72f7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7300a14f0 0x7fd7300a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:00.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.700+0000 7fd734fa8640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fd73009e600 0x7fd730139a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:48636/0 (socket says 172.21.15.37:48636) 2024-04-26T00:32:00.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.700+0000 7fd734fa8640 1 -- 172.21.15.37:0/141386278 learned_addr learned my addr 172.21.15.37:0/141386278 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:32:00.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.700+0000 7fd72f7fe640 10 monclient(hunting): get_auth_request con 0x7fd7300a14f0 auth_method 0 2024-04-26T00:32:00.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.700+0000 7fd72f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:00.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.700+0000 7fd72f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:00.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.700+0000 7fd72f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:00.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.700+0000 7fd72f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:00.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.700+0000 7fd72f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:00.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.700+0000 7fd72f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:00.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.700+0000 7fd72ffff640 1 --2- 172.21.15.37:0/141386278 >> v2:172.21.15.45:3300/0 conn(0x7fd73009e220 0x7fd73009fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:00.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.700+0000 7fd72ffff640 10 monclient(hunting): get_auth_request con 0x7fd73009e220 auth_method 0 2024-04-26T00:32:00.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.700+0000 7fd72ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:00.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.700+0000 7fd72ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:00.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.700+0000 7fd72ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:00.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.700+0000 7fd72f7fe640 10 monclient(hunting): handle_auth_done global_id 84644 payload 274 2024-04-26T00:32:00.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.700+0000 7fd72f7fe640 1 -- 172.21.15.37:0/141386278 >> v2:172.21.15.45:3300/0 conn(0x7fd73009e220 msgr2=0x7fd73009fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:00.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.700+0000 7fd72f7fe640 1 --2- 172.21.15.37:0/141386278 >> v2:172.21.15.45:3300/0 conn(0x7fd73009e220 0x7fd73009fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:00.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.700+0000 7fd72f7fe640 1 -- 172.21.15.37:0/141386278 >> v1:172.21.15.37:6789/0 conn(0x7fd73009e600 legacy=0x7fd730139a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:00.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.700+0000 7fd72f7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:32:00.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.700+0000 7fd72f7fe640 1 monclient: found mon.noname-b 2024-04-26T00:32:00.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.700+0000 7fd72f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:00.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.700+0000 7fd72f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:32:00.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.700+0000 7fd72f7fe640 1 -- 172.21.15.37:0/141386278 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd73009dd00 con 0x7fd7300a14f0 2024-04-26T00:32:00.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd72f7fe640 1 -- 172.21.15.37:0/141386278 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7300a14f0 msgr2=0x7fd7300a0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:00.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd72f7fe640 1 --2- 172.21.15.37:0/141386278 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7300a14f0 0x7fd7300a0190 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fd7180097c0 tx=0x7fd71801c320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=720d4f85c78c4bf8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:00.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd72effd640 1 -- 172.21.15.37:0/141386278 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7180b1070 con 0x7fd7300a14f0 2024-04-26T00:32:00.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd72effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:00.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd72effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:32:00.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd72effd640 10 monclient: dump: 2024-04-26T00:32:00.702 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:00.702 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:00.702 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:00.702 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:00.702 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:00.702 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:00.702 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:00.702 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:00.702 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:00.702 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:00.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd72effd640 10 monclient: _finish_auth 0 2024-04-26T00:32:00.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd72effd640 10 monclient: _check_auth_tickets 2024-04-26T00:32:00.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd72effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:00.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd736a0b640 5 monclient: authenticate success, global_id 84644 2024-04-26T00:32:00.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd736a0b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:32:00.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd72effd640 1 -- 172.21.15.37:0/141386278 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7180ac440 con 0x7fd7300a14f0 2024-04-26T00:32:00.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd72effd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:00.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd72effd640 1 -- 172.21.15.37:0/141386278 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7180ac810 con 0x7fd7300a14f0 2024-04-26T00:32:00.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd736a0b640 10 monclient: get_monmap_and_config success 2024-04-26T00:32:00.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd736a0b640 4 set_mon_vals no callback set 2024-04-26T00:32:00.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd72effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:00.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd72effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:32:00.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd72effd640 10 monclient: dump: 2024-04-26T00:32:00.709 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:00.709 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:00.709 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:00.709 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:00.709 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:00.710 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:00.710 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:00.710 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:00.710 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:00.710 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:00.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd736a0b640 10 monclient: shutdown 2024-04-26T00:32:00.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd736a0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:00.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd736a0b640 1 -- 172.21.15.37:0/141386278 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7300a14f0 msgr2=0x7fd7300a0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:00.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd736a0b640 1 --2- 172.21.15.37:0/141386278 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7300a14f0 0x7fd7300a0190 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fd7180097c0 tx=0x7fd71801c320 comp rx=0 tx=0).stop 2024-04-26T00:32:00.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd736a0b640 1 -- 172.21.15.37:0/141386278 shutdown_connections 2024-04-26T00:32:00.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd736a0b640 1 --2- 172.21.15.37:0/141386278 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7300a14f0 0x7fd7300a0190 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:00.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd736a0b640 1 --2- 172.21.15.37:0/141386278 >> v2:172.21.15.45:3300/0 conn(0x7fd73009e220 0x7fd73009fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:00.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd736a0b640 1 -- 172.21.15.37:0/141386278 >> 172.21.15.37:0/141386278 conn(0x7fd7300a1d50 msgr2=0x7fd7300a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:00.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd736a0b640 1 -- 172.21.15.37:0/141386278 shutdown_connections 2024-04-26T00:32:00.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd736a0b640 1 -- 172.21.15.37:0/141386278 wait complete. 2024-04-26T00:32:00.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd736a0b640 10 monclient: build_initial_monmap 2024-04-26T00:32:00.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd736a0b640 10 monclient: monmap: 2024-04-26T00:32:00.711 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:00.711 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:32:00.711 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:32:00.711 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:32:00.711 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:00.711 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:32:00.711 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:32:00.711 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:32:00.711 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:32:00.711 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:00.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd736a0b640 1 Processor -- start 2024-04-26T00:32:00.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd736a0b640 1 -- start start 2024-04-26T00:32:00.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd736a0b640 10 monclient: init 2024-04-26T00:32:00.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd736a0b640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:00.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd736a0b640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:32:00.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd736a0b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd73009e220 0x7fd73011cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:00.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd736a0b640 10 monclient(hunting): picked mon.noname-b con 0x7fd73009e220 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:32:00.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd736a0b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd73009e600 0x7fd73011b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:00.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd736a0b640 10 monclient(hunting): picked mon.noname-a con 0x7fd73009e600 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:32:00.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd736a0b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7300a14f0 0x7fd73011c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:00.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd736a0b640 10 monclient(hunting): picked mon.noname-c con 0x7fd7300a14f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:00.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd736a0b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:00.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd736a0b640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd73013f990 con 0x7fd73009e600 2024-04-26T00:32:00.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd736a0b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:00.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd736a0b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd73013f810 con 0x7fd73009e220 2024-04-26T00:32:00.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd736a0b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:00.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd736a0b640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd73013fb10 con 0x7fd7300a14f0 2024-04-26T00:32:00.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd736a0b640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:00.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.704+0000 7fd736a0b640 10 monclient(hunting): authenticate will time out at 1702.015901s 2024-04-26T00:32:00.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd734fa8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7300a14f0 0x7fd73011c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:00.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd72f7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd73009e600 0x7fd73011b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:00.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd734fa8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7300a14f0 0x7fd73011c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:56336/0 (socket says 172.21.15.37:56336) 2024-04-26T00:32:00.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd734fa8640 1 -- 172.21.15.37:0/2588322796 learned_addr learned my addr 172.21.15.37:0/2588322796 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:32:00.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd734fa8640 10 monclient(hunting): get_auth_request con 0x7fd7300a14f0 auth_method 0 2024-04-26T00:32:00.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd734fa8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:00.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd734fa8640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:00.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd734fa8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:00.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd72f7fe640 10 monclient(hunting): get_auth_request con 0x7fd73009e600 auth_method 0 2024-04-26T00:32:00.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd72f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:00.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd72f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:00.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd72f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:00.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd72ffff640 1 --2- 172.21.15.37:0/2588322796 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd73009e220 0x7fd73011cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:00.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd734fa8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:00.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd734fa8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:00.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd734fa8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:00.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd72ffff640 10 monclient(hunting): get_auth_request con 0x7fd73009e220 auth_method 0 2024-04-26T00:32:00.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd72ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:00.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd72ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:00.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd72ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:00.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd72f7fe640 1 -- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd73009e600 msgr2=0x7fd73011b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:32:00.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd72f7fe640 1 -- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd73009e600 msgr2=0x7fd73011b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:32:00.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd72f7fe640 1 --2- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd73009e600 0x7fd73011b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:32:00.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd734fa8640 10 monclient(hunting): handle_auth_done global_id 84647 payload 824 2024-04-26T00:32:00.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd72f7fe640 1 --2- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd73009e600 0x7fd73011b8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:32:00.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd734fa8640 1 -- 172.21.15.37:0/2588322796 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd73009e220 msgr2=0x7fd73011cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:00.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd734fa8640 1 --2- 172.21.15.37:0/2588322796 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd73009e220 0x7fd73011cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:00.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd734fa8640 1 -- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd73009e600 msgr2=0x7fd73011b8a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:32:00.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd734fa8640 1 --2- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd73009e600 0x7fd73011b8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:00.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd734fa8640 10 monclient: _finish_hunting 0 2024-04-26T00:32:00.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd734fa8640 1 monclient: found mon.noname-c 2024-04-26T00:32:00.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd734fa8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:00.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd734fa8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:32:00.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd734fa8640 1 -- 172.21.15.37:0/2588322796 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd73013a980 con 0x7fd7300a14f0 2024-04-26T00:32:00.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd734fa8640 1 -- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7300a14f0 msgr2=0x7fd73011c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:00.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd734fa8640 1 --2- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7300a14f0 0x7fd73011c050 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd72800c3c0 tx=0x7fd72800c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:00.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd72cff9640 1 -- 172.21.15.37:0/2588322796 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd728017070 con 0x7fd7300a14f0 2024-04-26T00:32:00.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd72cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:00.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd72cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:32:00.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd72cff9640 10 monclient: dump: 2024-04-26T00:32:00.716 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:00.716 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:00.716 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:00.716 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:00.716 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:00.716 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:00.716 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:00.716 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:00.716 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:00.717 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:00.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd72cff9640 10 monclient: _finish_auth 0 2024-04-26T00:32:00.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd72cff9640 10 monclient: _check_auth_tickets 2024-04-26T00:32:00.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd72cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:00.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd72cff9640 1 -- 172.21.15.37:0/2588322796 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7280070d0 con 0x7fd7300a14f0 2024-04-26T00:32:00.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd72cff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:00.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd72cff9640 1 -- 172.21.15.37:0/2588322796 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7280074c0 con 0x7fd7300a14f0 2024-04-26T00:32:00.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd72cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:00.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd72cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:32:00.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd72cff9640 10 monclient: dump: 2024-04-26T00:32:00.718 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:00.718 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:00.718 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:00.718 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:00.718 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:00.718 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:00.718 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:00.718 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:00.718 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:00.718 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:00.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd736a0b640 5 monclient: authenticate success, global_id 84647 2024-04-26T00:32:00.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd736a0b640 10 monclient: _renew_subs 2024-04-26T00:32:00.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd72e7fc640 4 set_mon_vals no callback set 2024-04-26T00:32:00.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd736a0b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:32:00.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd736a0b640 1 -- 172.21.15.37:0/2588322796 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd73011de50 con 0x7fd7300a14f0 2024-04-26T00:32:00.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd736a0b640 10 monclient: _renew_subs 2024-04-26T00:32:00.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd736a0b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:32:00.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd736a0b640 1 -- 172.21.15.37:0/2588322796 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7301340c0 con 0x7fd7300a14f0 2024-04-26T00:32:00.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd736a0b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:00.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd72cff9640 1 -- 172.21.15.37:0/2588322796 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd72801a070 con 0x7fd7300a14f0 2024-04-26T00:32:00.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd736a0b640 1 --2- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd73011c590 0x7fd7301ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:00.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd736a0b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:00.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.708+0000 7fd736a0b640 1 -- 172.21.15.37:0/2588322796 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd73011e020 con 0x7fd73011c590 2024-04-26T00:32:00.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.712+0000 7fd72ffff640 1 --2- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd73011c590 0x7fd7301ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:00.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.712+0000 7fd72ffff640 10 monclient: get_auth_request con 0x7fd73011c590 auth_method 0 2024-04-26T00:32:00.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.712+0000 7fd72ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:00.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.712+0000 7fd72ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:00.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.712+0000 7fd72ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:00.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.712+0000 7fd72cff9640 1 --2- 172.21.15.37:0/2588322796 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd710035eb0 0x7fd710038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:00.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.712+0000 7fd72ffff640 1 -- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd73011c590 msgr2=0x7fd7301ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:32:00.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.712+0000 7fd72ffff640 1 -- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd73011c590 msgr2=0x7fd7301ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:32:00.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.712+0000 7fd72ffff640 1 --2- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd73011c590 0x7fd7301ccee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:32:00.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.712+0000 7fd72cff9640 1 -- 172.21.15.37:0/2588322796 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(463..463 src has 370..463) ==== 3987+0+0 (secure 0 0 0) 0x7fd728047ed0 con 0x7fd7300a14f0 2024-04-26T00:32:00.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.712+0000 7fd72ffff640 1 --2- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd73011c590 0x7fd7301ccee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:32:00.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.712+0000 7fd72f7fe640 1 --2- 172.21.15.37:0/2588322796 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd710035eb0 0x7fd710038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:00.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.712+0000 7fd72f7fe640 10 monclient: get_auth_request con 0x7fd710035eb0 auth_method 0 2024-04-26T00:32:00.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.712+0000 7fd72f7fe640 1 --2- 172.21.15.37:0/2588322796 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd710035eb0 0x7fd710038370 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd73006d140 tx=0x7fd7180a6070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:00.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.916+0000 7fd72ffff640 1 --2- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd73011c590 0x7fd7301ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:00.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.916+0000 7fd72ffff640 10 monclient: get_auth_request con 0x7fd73011c590 auth_method 0 2024-04-26T00:32:00.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.916+0000 7fd72ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:00.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.916+0000 7fd72ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:00.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.916+0000 7fd72ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:32:00.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.916+0000 7fd72ffff640 1 -- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd73011c590 msgr2=0x7fd7301ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:32:00.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.916+0000 7fd72ffff640 1 -- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd73011c590 msgr2=0x7fd7301ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:32:00.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.916+0000 7fd72ffff640 1 --2- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd73011c590 0x7fd7301ccee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:32:00.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:00.916+0000 7fd72ffff640 1 --2- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd73011c590 0x7fd7301ccee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:32:01.198 INFO:tasks.rados.rados.0.smithi045.stdout:2024: done (0 left) 2024-04-26T00:32:01.199 INFO:tasks.rados.rados.0.smithi045.stdout:2025: read oid 20 snap -1 2024-04-26T00:32:01.199 INFO:tasks.rados.rados.0.smithi045.stdout:2025: expect (ObjNum 542 snap 167 seq_num 542) 2024-04-26T00:32:01.199 INFO:tasks.rados.rados.0.smithi045.stdout:2026: write oid 33 current snap is 229 2024-04-26T00:32:01.199 INFO:tasks.rados.rados.0.smithi045.stdout:2026: seq_num 740 ranges {437180=570501,1503028=550587} 2024-04-26T00:32:01.215 INFO:tasks.rados.rados.0.smithi045.stdout:2026: writing smithi04526579-33 from 437180 to 1007681 tid 1 2024-04-26T00:32:01.233 INFO:tasks.rados.rados.0.smithi045.stdout:2026: writing smithi04526579-33 from 1503028 to 2053615 tid 2 2024-04-26T00:32:01.233 INFO:tasks.rados.rados.0.smithi045.stdout:2027: read oid 29 snap 211 2024-04-26T00:32:01.233 INFO:tasks.rados.rados.0.smithi045.stdout:2027: expect deleted 2024-04-26T00:32:01.233 INFO:tasks.rados.rados.0.smithi045.stdout:2028: write (excl) oid 7 current snap is 229 2024-04-26T00:32:01.233 INFO:tasks.rados.rados.0.smithi045.stdout:2028: seq_num 741 ranges {453987=645750,1845157=608830} 2024-04-26T00:32:01.249 INFO:tasks.rados.rados.0.smithi045.stdout:2028: writing smithi04526579-7 from 453987 to 1099737 tid 1 2024-04-26T00:32:01.267 INFO:tasks.rados.rados.0.smithi045.stdout:2028: writing smithi04526579-7 from 1845157 to 2453987 tid 2 2024-04-26T00:32:01.267 INFO:tasks.rados.rados.0.smithi045.stdout:2029: read oid 15 snap -1 2024-04-26T00:32:01.267 INFO:tasks.rados.rados.0.smithi045.stdout:2029: expect (ObjNum 724 snap 222 seq_num 724) 2024-04-26T00:32:01.267 INFO:tasks.rados.rados.0.smithi045.stdout:2030: read oid 14 snap -1 2024-04-26T00:32:01.267 INFO:tasks.rados.rados.0.smithi045.stdout:2030: expect deleted 2024-04-26T00:32:01.267 INFO:tasks.rados.rados.0.smithi045.stdout:2031: read oid 16 snap -1 2024-04-26T00:32:01.267 INFO:tasks.rados.rados.0.smithi045.stdout:2031: expect (ObjNum 515 snap 157 seq_num 515) 2024-04-26T00:32:01.267 INFO:tasks.rados.rados.0.smithi045.stdout:2032: rollback oid 1 current snap is 229 2024-04-26T00:32:01.267 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 1 to 208 2024-04-26T00:32:01.267 INFO:tasks.rados.rados.0.smithi045.stdout:2033: snap_create 2024-04-26T00:32:01.267 INFO:tasks.rados.rados.0.smithi045.stdout:2025: expect (ObjNum 542 snap 167 seq_num 542) 2024-04-26T00:32:01.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:01.320+0000 7fd72ffff640 1 --2- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd73011c590 0x7fd7301ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:01.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:01.320+0000 7fd72ffff640 10 monclient: get_auth_request con 0x7fd73011c590 auth_method 0 2024-04-26T00:32:01.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:01.320+0000 7fd72ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:01.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:01.320+0000 7fd72ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:01.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:01.320+0000 7fd72ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:32:01.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:01.320+0000 7fd72ffff640 1 -- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd73011c590 msgr2=0x7fd7301ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:32:01.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:01.320+0000 7fd72ffff640 1 -- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd73011c590 msgr2=0x7fd7301ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:32:01.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:01.320+0000 7fd72ffff640 1 --2- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd73011c590 0x7fd7301ccee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:32:01.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:01.320+0000 7fd72ffff640 1 --2- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd73011c590 0x7fd7301ccee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:32:01.468 INFO:tasks.rados.rados.0.smithi045.stdout:2026: finishing write tid 1 to smithi04526579-33 2024-04-26T00:32:01.469 INFO:tasks.rados.rados.0.smithi045.stdout:2026: finishing write tid 2 to smithi04526579-33 2024-04-26T00:32:01.469 INFO:tasks.rados.rados.0.smithi045.stdout:2026: finishing write tid 3 to smithi04526579-33 2024-04-26T00:32:01.469 INFO:tasks.rados.rados.0.smithi045.stdout:2026: finishing write tid 4 to smithi04526579-33 2024-04-26T00:32:01.469 INFO:tasks.rados.rados.0.smithi045.stdout:2026: oid 33 updating version 0 to 1720 2024-04-26T00:32:01.469 INFO:tasks.rados.rados.0.smithi045.stdout:2026: oid 33 updating version 1720 to 1721 2024-04-26T00:32:01.469 INFO:tasks.rados.rados.0.smithi045.stdout:2026: oid 33 updating version 1721 to 1722 2024-04-26T00:32:01.469 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 1722 (ObjNum 740 snap 229 seq_num 740) dirty exists 2024-04-26T00:32:01.469 INFO:tasks.rados.rados.0.smithi045.stdout:2026: left oid 33 (ObjNum 740 snap 229 seq_num 740) 2024-04-26T00:32:01.469 INFO:tasks.rados.rados.0.smithi045.stdout:2028: finishing write tid 1 to smithi04526579-7 2024-04-26T00:32:01.469 INFO:tasks.rados.rados.0.smithi045.stdout:2028: finishing write tid 2 to smithi04526579-7 2024-04-26T00:32:01.469 INFO:tasks.rados.rados.0.smithi045.stdout:2028: finishing write tid 3 to smithi04526579-7 2024-04-26T00:32:01.469 INFO:tasks.rados.rados.0.smithi045.stdout:2028: finishing write tid 4 to smithi04526579-7 2024-04-26T00:32:01.469 INFO:tasks.rados.rados.0.smithi045.stdout:2028: oid 7 updating version 0 to 1724 2024-04-26T00:32:01.469 INFO:tasks.rados.rados.0.smithi045.stdout:2028: oid 7 version 1724 is already newer than 1723 2024-04-26T00:32:01.470 INFO:tasks.rados.rados.0.smithi045.stdout:2028: oid 7 updating version 1724 to 1725 2024-04-26T00:32:01.470 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 1725 (ObjNum 741 snap 229 seq_num 741) dirty exists 2024-04-26T00:32:01.470 INFO:tasks.rados.rados.0.smithi045.stdout:2028: left oid 7 (ObjNum 741 snap 229 seq_num 741) 2024-04-26T00:32:01.470 INFO:tasks.rados.rados.0.smithi045.stdout:2029: expect (ObjNum 724 snap 222 seq_num 724) 2024-04-26T00:32:01.571 INFO:tasks.rados.rados.0.smithi045.stdout:2032: finishing rollback tid 1 to smithi04526579-1 2024-04-26T00:32:01.571 INFO:tasks.rados.rados.0.smithi045.stdout:2032: finishing rollback tid 2 to smithi04526579-1 2024-04-26T00:32:01.571 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 1323 (ObjNum 661 snap 205 seq_num 661) dirty exists 2024-04-26T00:32:01.571 INFO:tasks.rados.rados.0.smithi045.stdout:2031: expect (ObjNum 515 snap 157 seq_num 515) 2024-04-26T00:32:02.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:02.124+0000 7fd72ffff640 1 --2- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd73011c590 0x7fd7301ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:02.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:02.124+0000 7fd72ffff640 10 monclient: get_auth_request con 0x7fd73011c590 auth_method 0 2024-04-26T00:32:02.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:02.124+0000 7fd72ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:02.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:02.124+0000 7fd72ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:02.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:02.124+0000 7fd72ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:32:02.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:02.124+0000 7fd72ffff640 1 -- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd73011c590 msgr2=0x7fd7301ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:32:02.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:02.124+0000 7fd72ffff640 1 -- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd73011c590 msgr2=0x7fd7301ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:32:02.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:02.124+0000 7fd72ffff640 1 --2- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd73011c590 0x7fd7301ccee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:32:02.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:02.124+0000 7fd72ffff640 1 --2- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd73011c590 0x7fd7301ccee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:32:02.218 INFO:tasks.rados.rados.0.smithi045.stdout:2025: done (8 left) 2024-04-26T00:32:02.218 INFO:tasks.rados.rados.0.smithi045.stdout:2026: done (7 left) 2024-04-26T00:32:02.218 INFO:tasks.rados.rados.0.smithi045.stdout:2027: done (6 left) 2024-04-26T00:32:02.218 INFO:tasks.rados.rados.0.smithi045.stdout:2028: done (5 left) 2024-04-26T00:32:02.218 INFO:tasks.rados.rados.0.smithi045.stdout:2029: done (4 left) 2024-04-26T00:32:02.219 INFO:tasks.rados.rados.0.smithi045.stdout:2030: done (3 left) 2024-04-26T00:32:02.219 INFO:tasks.rados.rados.0.smithi045.stdout:2031: done (2 left) 2024-04-26T00:32:02.219 INFO:tasks.rados.rados.0.smithi045.stdout:2032: done (1 left) 2024-04-26T00:32:02.219 INFO:tasks.rados.rados.0.smithi045.stdout:2033: done (0 left) 2024-04-26T00:32:02.219 INFO:tasks.rados.rados.0.smithi045.stdout:2034: read oid 10 snap 195 2024-04-26T00:32:02.219 INFO:tasks.rados.rados.0.smithi045.stdout:2034: expect (ObjNum 573 snap 176 seq_num 573) 2024-04-26T00:32:02.219 INFO:tasks.rados.rados.0.smithi045.stdout:2035: copy_from oid 4 from oid 47 current snap is 230 2024-04-26T00:32:02.219 INFO:tasks.rados.rados.0.smithi045.stdout:2036: copy_from oid 6 from oid 21 current snap is 230 2024-04-26T00:32:02.219 INFO:tasks.rados.rados.0.smithi045.stdout:2037: rollback oid 46 current snap is 230 2024-04-26T00:32:02.219 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 46 to 208 2024-04-26T00:32:02.219 INFO:tasks.rados.rados.0.smithi045.stdout:2038: delete oid 34 current snap is 230 2024-04-26T00:32:02.227 INFO:tasks.rados.rados.0.smithi045.stdout:2037: finishing rollback tid 1 to smithi04526579-46 2024-04-26T00:32:02.228 INFO:tasks.rados.rados.0.smithi045.stdout:2037: finishing rollback tid 2 to smithi04526579-46 2024-04-26T00:32:02.228 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 1442 (ObjNum 652 snap 202 seq_num 652) dirty exists 2024-04-26T00:32:02.236 INFO:tasks.rados.rados.0.smithi045.stdout:2037: done (4 left) 2024-04-26T00:32:02.236 INFO:tasks.rados.rados.0.smithi045.stdout:2038: done (3 left) 2024-04-26T00:32:02.236 INFO:tasks.rados.rados.0.smithi045.stdout:2039: write (excl) oid 15 current snap is 230 2024-04-26T00:32:02.236 INFO:tasks.rados.rados.0.smithi045.stdout:2039: seq_num 743 ranges {487601=796248,1962121=525480} 2024-04-26T00:32:02.254 INFO:tasks.rados.rados.0.smithi045.stdout:2039: writing smithi04526579-15 from 487601 to 1283849 tid 1 2024-04-26T00:32:02.271 INFO:tasks.rados.rados.0.smithi045.stdout:2039: writing smithi04526579-15 from 1962121 to 2487601 tid 2 2024-04-26T00:32:02.271 INFO:tasks.rados.rados.0.smithi045.stdout:2040: delete oid 23 current snap is 230 2024-04-26T00:32:02.271 INFO:tasks.rados.rados.0.smithi045.stdout:2034: expect (ObjNum 573 snap 176 seq_num 573) 2024-04-26T00:32:02.396 INFO:tasks.rados.rados.0.smithi045.stdout:2036: finishing copy_from racing read to smithi04526579-6 2024-04-26T00:32:02.396 INFO:tasks.rados.rados.0.smithi045.stdout:2036: finishing copy_from to smithi04526579-6 2024-04-26T00:32:02.396 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 1052 (ObjNum 595 snap 184 seq_num 595) dirty exists 2024-04-26T00:32:02.396 INFO:tasks.rados.rados.0.smithi045.stdout:2035: finishing copy_from racing read to smithi04526579-4 2024-04-26T00:32:02.396 INFO:tasks.rados.rados.0.smithi045.stdout:2035: finishing copy_from to smithi04526579-4 2024-04-26T00:32:02.396 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 1085 (ObjNum 683 snap 213 seq_num 683) dirty exists 2024-04-26T00:32:02.396 INFO:tasks.rados.rados.0.smithi045.stdout:2039: finishing write tid 1 to smithi04526579-15 2024-04-26T00:32:02.396 INFO:tasks.rados.rados.0.smithi045.stdout:2039: finishing write tid 2 to smithi04526579-15 2024-04-26T00:32:02.396 INFO:tasks.rados.rados.0.smithi045.stdout:2039: finishing write tid 3 to smithi04526579-15 2024-04-26T00:32:02.396 INFO:tasks.rados.rados.0.smithi045.stdout:2039: finishing write tid 4 to smithi04526579-15 2024-04-26T00:32:02.397 INFO:tasks.rados.rados.0.smithi045.stdout:2039: oid 15 updating version 0 to 1327 2024-04-26T00:32:02.397 INFO:tasks.rados.rados.0.smithi045.stdout:2039: oid 15 version 1327 is already newer than 1325 2024-04-26T00:32:02.397 INFO:tasks.rados.rados.0.smithi045.stdout:2039: oid 15 version 1327 is already newer than 1326 2024-04-26T00:32:02.397 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 1327 (ObjNum 743 snap 230 seq_num 743) dirty exists 2024-04-26T00:32:02.397 INFO:tasks.rados.rados.0.smithi045.stdout:2039: left oid 15 (ObjNum 743 snap 230 seq_num 743) 2024-04-26T00:32:02.397 INFO:tasks.rados.rados.0.smithi045.stdout:2034: done (4 left) 2024-04-26T00:32:02.397 INFO:tasks.rados.rados.0.smithi045.stdout:2035: done (3 left) 2024-04-26T00:32:02.397 INFO:tasks.rados.rados.0.smithi045.stdout:2036: done (2 left) 2024-04-26T00:32:02.397 INFO:tasks.rados.rados.0.smithi045.stdout:2039: done (1 left) 2024-04-26T00:32:02.397 INFO:tasks.rados.rados.0.smithi045.stdout:2040: done (0 left) 2024-04-26T00:32:02.397 INFO:tasks.rados.rados.0.smithi045.stdout:2041: write oid 2 current snap is 230 2024-04-26T00:32:02.397 INFO:tasks.rados.rados.0.smithi045.stdout:2041: seq_num 745 ranges {521215=546746,1562732=701583} 2024-04-26T00:32:02.410 INFO:tasks.rados.rados.0.smithi045.stdout:2041: writing smithi04526579-2 from 521215 to 1067961 tid 1 2024-04-26T00:32:02.427 INFO:tasks.rados.rados.0.smithi045.stdout:2041: writing smithi04526579-2 from 1562732 to 2264315 tid 2 2024-04-26T00:32:02.427 INFO:tasks.rados.rados.0.smithi045.stdout:2042: delete oid 22 current snap is 230 2024-04-26T00:32:02.427 INFO:tasks.rados.rados.0.smithi045.stdout:2041: finishing write tid 1 to smithi04526579-2 2024-04-26T00:32:02.435 INFO:tasks.rados.rados.0.smithi045.stdout:2042: done (1 left) 2024-04-26T00:32:02.435 INFO:tasks.rados.rados.0.smithi045.stdout:2043: snap_remove snap 229 2024-04-26T00:32:03.672 INFO:tasks.rados.rados.0.smithi045.stdout:2043: done (1 left) 2024-04-26T00:32:03.672 INFO:tasks.rados.rados.0.smithi045.stdout:2044: read oid 40 snap 219 2024-04-26T00:32:03.672 INFO:tasks.rados.rados.0.smithi045.stdout:2044: expect deleted 2024-04-26T00:32:03.672 INFO:tasks.rados.rados.0.smithi045.stdout:2045: write (excl) oid 49 current snap is 230 2024-04-26T00:32:03.672 INFO:tasks.rados.rados.0.smithi045.stdout:2045: seq_num 747 ranges {554829=413597,1396049=705252} 2024-04-26T00:32:03.692 INFO:tasks.rados.rados.0.smithi045.stdout:2045: writing smithi04526579-49 from 554829 to 968426 tid 1 2024-04-26T00:32:03.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:03.704+0000 7fd7277fe640 10 monclient: tick 2024-04-26T00:32:03.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:03.704+0000 7fd7277fe640 10 monclient: _check_auth_tickets 2024-04-26T00:32:03.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:03.704+0000 7fd7277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:03.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:03.704+0000 7fd7277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:03.710 INFO:tasks.rados.rados.0.smithi045.stdout:2045: writing smithi04526579-49 from 1396049 to 2101301 tid 2 2024-04-26T00:32:03.710 INFO:tasks.rados.rados.0.smithi045.stdout:2046: write (excl) oid 29 current snap is 230 2024-04-26T00:32:03.711 INFO:tasks.rados.rados.0.smithi045.stdout:2046: seq_num 748 ranges {571636=488846,1854531=565263} 2024-04-26T00:32:03.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:03.728+0000 7fd72ffff640 1 --2- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd73011c590 0x7fd7301ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:03.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:03.728+0000 7fd72ffff640 10 monclient: get_auth_request con 0x7fd73011c590 auth_method 0 2024-04-26T00:32:03.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:03.728+0000 7fd72ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:03.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:03.728+0000 7fd72ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:03.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:03.728+0000 7fd72ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:32:03.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:03.728+0000 7fd72ffff640 1 -- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd73011c590 msgr2=0x7fd7301ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:32:03.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:03.728+0000 7fd72ffff640 1 -- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd73011c590 msgr2=0x7fd7301ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:32:03.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:03.728+0000 7fd72ffff640 1 --2- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd73011c590 0x7fd7301ccee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:32:03.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:03.728+0000 7fd72ffff640 1 --2- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd73011c590 0x7fd7301ccee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-26T00:32:03.726 INFO:tasks.rados.rados.0.smithi045.stdout:2046: writing smithi04526579-29 from 571636 to 1060482 tid 1 2024-04-26T00:32:03.743 INFO:tasks.rados.rados.0.smithi045.stdout:2046: writing smithi04526579-29 from 1854531 to 2419794 tid 2 2024-04-26T00:32:03.744 INFO:tasks.rados.rados.0.smithi045.stdout:2047: snap_create 2024-04-26T00:32:03.744 INFO:tasks.rados.rados.0.smithi045.stdout:2041: finishing write tid 2 to smithi04526579-2 2024-04-26T00:32:03.744 INFO:tasks.rados.rados.0.smithi045.stdout:2041: finishing write tid 3 to smithi04526579-2 2024-04-26T00:32:03.744 INFO:tasks.rados.rados.0.smithi045.stdout:2041: finishing write tid 4 to smithi04526579-2 2024-04-26T00:32:03.744 INFO:tasks.rados.rados.0.smithi045.stdout:2041: oid 2 updating version 0 to 1520 2024-04-26T00:32:03.744 INFO:tasks.rados.rados.0.smithi045.stdout:2041: oid 2 version 1520 is already newer than 1519 2024-04-26T00:32:03.744 INFO:tasks.rados.rados.0.smithi045.stdout:2041: oid 2 updating version 1520 to 1521 2024-04-26T00:32:03.744 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 1521 (ObjNum 745 snap 230 seq_num 745) dirty exists 2024-04-26T00:32:03.744 INFO:tasks.rados.rados.0.smithi045.stdout:2041: left oid 2 (ObjNum 745 snap 230 seq_num 745) 2024-04-26T00:32:03.744 INFO:tasks.rados.rados.0.smithi045.stdout:2045: finishing write tid 1 to smithi04526579-49 2024-04-26T00:32:03.744 INFO:tasks.rados.rados.0.smithi045.stdout:2045: finishing write tid 2 to smithi04526579-49 2024-04-26T00:32:03.744 INFO:tasks.rados.rados.0.smithi045.stdout:2045: finishing write tid 3 to smithi04526579-49 2024-04-26T00:32:03.744 INFO:tasks.rados.rados.0.smithi045.stdout:2045: finishing write tid 4 to smithi04526579-49 2024-04-26T00:32:03.745 INFO:tasks.rados.rados.0.smithi045.stdout:2045: oid 49 updating version 0 to 1211 2024-04-26T00:32:03.745 INFO:tasks.rados.rados.0.smithi045.stdout:2045: oid 49 updating version 1211 to 1213 2024-04-26T00:32:03.745 INFO:tasks.rados.rados.0.smithi045.stdout:2045: oid 49 version 1213 is already newer than 1212 2024-04-26T00:32:03.745 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 1213 (ObjNum 747 snap 230 seq_num 747) dirty exists 2024-04-26T00:32:03.745 INFO:tasks.rados.rados.0.smithi045.stdout:2045: left oid 49 (ObjNum 747 snap 230 seq_num 747) 2024-04-26T00:32:03.745 INFO:tasks.rados.rados.0.smithi045.stdout:2046: finishing write tid 1 to smithi04526579-29 2024-04-26T00:32:03.750 INFO:tasks.rados.rados.0.smithi045.stdout:2046: finishing write tid 2 to smithi04526579-29 2024-04-26T00:32:03.750 INFO:tasks.rados.rados.0.smithi045.stdout:2046: finishing write tid 3 to smithi04526579-29 2024-04-26T00:32:03.751 INFO:tasks.rados.rados.0.smithi045.stdout:2046: finishing write tid 4 to smithi04526579-29 2024-04-26T00:32:03.751 INFO:tasks.rados.rados.0.smithi045.stdout:2046: oid 29 updating version 0 to 1069 2024-04-26T00:32:03.751 INFO:tasks.rados.rados.0.smithi045.stdout:2046: oid 29 updating version 1069 to 1071 2024-04-26T00:32:03.751 INFO:tasks.rados.rados.0.smithi045.stdout:2046: oid 29 version 1071 is already newer than 1070 2024-04-26T00:32:03.751 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 1071 (ObjNum 748 snap 230 seq_num 748) dirty exists 2024-04-26T00:32:03.751 INFO:tasks.rados.rados.0.smithi045.stdout:2046: left oid 29 (ObjNum 748 snap 230 seq_num 748) 2024-04-26T00:32:04.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.708+0000 7fd7277fe640 10 monclient: tick 2024-04-26T00:32:04.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.708+0000 7fd7277fe640 10 monclient: _check_auth_tickets 2024-04-26T00:32:04.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.708+0000 7fd7277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:04.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.708+0000 7fd7277fe640 5 monclient: _check_tell_commands timeout tell command 1 2024-04-26T00:32:04.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.708+0000 7fd7277fe640 1 -- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd73011c590 msgr2=0x7fd7301ccee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:32:04.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.708+0000 7fd7277fe640 1 --2- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd73011c590 0x7fd7301ccee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:04.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.708+0000 7fd7277fe640 1 --2- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd708003550 0x7fd7080059e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:04.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.708+0000 7fd7277fe640 1 -- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd73011c590 msgr2=0x7fd7301ccee0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:32:04.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.708+0000 7fd7277fe640 1 --2- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd73011c590 0x7fd7301ccee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:04.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.708+0000 7fd7277fe640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:04.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.708+0000 7fd7277fe640 1 -- 172.21.15.37:0/2588322796 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd73011e020 con 0x7fd708003550 2024-04-26T00:32:04.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.708+0000 7fd72ffff640 1 --2- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd708003550 0x7fd7080059e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:04.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.708+0000 7fd72ffff640 10 monclient: get_auth_request con 0x7fd708003550 auth_method 0 2024-04-26T00:32:04.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.708+0000 7fd72ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:04.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.708+0000 7fd72ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:04.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.708+0000 7fd72ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:04.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.708+0000 7fd72ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:04.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.708+0000 7fd72ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:04.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.708+0000 7fd72ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:04.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.708+0000 7fd72ffff640 10 monclient(hunting): handle_auth_done global_id 144099 payload 824 2024-04-26T00:32:04.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.708+0000 7fd72ffff640 1 -- 172.21.15.37:0/2588322796 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd7080060c0 con 0x7fd708003550 2024-04-26T00:32:04.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.712+0000 7fd72ffff640 1 -- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd708003550 msgr2=0x7fd7080059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:04.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.712+0000 7fd72ffff640 1 --2- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd708003550 0x7fd7080059e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd720019e70 tx=0x7fd720017720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:04.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.712+0000 7fd72cff9640 1 -- 172.21.15.37:0/2588322796 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd72001a860 con 0x7fd708003550 2024-04-26T00:32:04.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.712+0000 7fd72cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:04.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.712+0000 7fd72cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:04.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.712+0000 7fd72cff9640 10 monclient: dump: 2024-04-26T00:32:04.706 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:04.706 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:04.706 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:04.706 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:04.706 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:04.706 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:04.706 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:04.706 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:04.707 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:04.707 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:04.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.712+0000 7fd72cff9640 1 -- 172.21.15.37:0/2588322796 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd72001a860 con 0x7fd708003550 2024-04-26T00:32:04.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.712+0000 7fd72cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:04.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.712+0000 7fd72cff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:32:04.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.712+0000 7fd72cff9640 1 -- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd708003550 msgr2=0x7fd7080059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:04.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.712+0000 7fd72cff9640 1 --2- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd708003550 0x7fd7080059e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd720019e70 tx=0x7fd720017720 comp rx=0 tx=0).stop 2024-04-26T00:32:04.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.712+0000 7fd72cff9640 1 -- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd708003550 msgr2=0x7fd7080059e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:32:04.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.712+0000 7fd72cff9640 1 --2- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd708003550 0x7fd7080059e0 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd720019e70 tx=0x7fd720017720 comp rx=0 tx=0).stop 2024-04-26T00:32:04.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.720+0000 7fd736a0b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:32:04.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.720+0000 7fd736a0b640 1 --2- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd7301cd420 0x7fd7301cd830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:04.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.720+0000 7fd736a0b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:04.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.720+0000 7fd736a0b640 1 -- 172.21.15.37:0/2588322796 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd7301cdd70 con 0x7fd7301cd420 2024-04-26T00:32:04.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.720+0000 7fd72ffff640 1 --2- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd7301cd420 0x7fd7301cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:04.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.720+0000 7fd72ffff640 10 monclient: get_auth_request con 0x7fd7301cd420 auth_method 0 2024-04-26T00:32:04.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.720+0000 7fd72ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:04.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.720+0000 7fd72ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:04.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.720+0000 7fd72ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:04.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.720+0000 7fd72ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:04.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.720+0000 7fd72ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:04.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.720+0000 7fd72ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:04.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.720+0000 7fd72ffff640 10 monclient(hunting): handle_auth_done global_id 144102 payload 824 2024-04-26T00:32:04.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.720+0000 7fd72ffff640 1 -- 172.21.15.37:0/2588322796 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd73011db70 con 0x7fd7301cd420 2024-04-26T00:32:04.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.724+0000 7fd72ffff640 1 -- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd7301cd420 msgr2=0x7fd7301cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:04.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.724+0000 7fd72ffff640 1 --2- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd7301cd420 0x7fd7301cd830 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd720019e70 tx=0x7fd720004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:04.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.724+0000 7fd72cff9640 1 -- 172.21.15.37:0/2588322796 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd72002a780 con 0x7fd7301cd420 2024-04-26T00:32:04.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.724+0000 7fd72cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:04.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.724+0000 7fd72cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:04.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.724+0000 7fd72cff9640 10 monclient: dump: 2024-04-26T00:32:04.719 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:04.719 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:04.719 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:04.719 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:04.719 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:04.719 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:04.719 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:04.719 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:04.719 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:04.719 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:04.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.724+0000 7fd72cff9640 1 -- 172.21.15.37:0/2588322796 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7fd72002a780 con 0x7fd7301cd420 2024-04-26T00:32:04.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.724+0000 7fd72cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:32:04.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.724+0000 7fd72cff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:32:04.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.724+0000 7fd72cff9640 1 -- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd7301cd420 msgr2=0x7fd7301cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:04.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.724+0000 7fd72cff9640 1 --2- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd7301cd420 0x7fd7301cd830 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd720019e70 tx=0x7fd720004ac0 comp rx=0 tx=0).stop 2024-04-26T00:32:04.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.724+0000 7fd72cff9640 1 -- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd7301cd420 msgr2=0x7fd7301cd830 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:32:04.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.724+0000 7fd72cff9640 1 --2- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd7301cd420 0x7fd7301cd830 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:04.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.724+0000 7fd734fa8640 1 -- 172.21.15.37:0/2588322796 reap_dead start 2024-04-26T00:32:04.720 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:32:04.720 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:32:04.720 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:32:04.720 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:32:04.720 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 92, 2024-04-26T00:32:04.720 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 11267, 2024-04-26T00:32:04.720 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:32:04.720 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:32:04.721 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:32:04.721 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:32:04.721 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:04.721 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 4, 2024-04-26T00:32:04.721 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:32:04.721 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:32:04.721 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:32:04.721 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:04.721 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:04.721 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:04.721 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:04.721 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:04.721 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:04.721 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:04.721 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:04.722 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:04.722 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:04.722 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:04.722 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:04.722 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:32:04.722 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:32:04.722 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:04.722 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:04.722 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:04.722 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:04.722 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:04.722 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:04.722 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:04.722 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:04.723 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:04.723 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:04.723 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:04.723 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:04.723 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:04.723 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:32:04.723 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:32:04.723 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:32:04.723 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:32:04.723 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:32:04.723 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:32:04.723 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:32:04.723 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:32:04.723 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:32:04.723 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:32:04.724 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:32:04.724 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:32:04.724 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:32:04.724 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:32:04.724 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:32:04.724 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:32:04.724 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:32:04.724 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:04.724 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:04.724 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:04.724 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:04.724 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:04.724 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:04.724 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:04.724 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:04.725 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:04.725 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:04.725 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:04.725 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:04.725 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:32:04.725 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:04.725 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:32:04.725 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:04.725 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:32:04.725 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:32:04.725 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:04.725 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:04.725 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:04.725 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:04.725 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:32:04.726 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:04.726 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:04.726 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:04.726 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:04.726 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:32:04.726 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:04.726 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:04.726 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:04.726 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:04.726 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:32:04.726 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:32:04.726 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:04.726 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:04.727 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:04.727 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:04.727 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:04.727 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:32:04.727 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:32:04.727 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:04.727 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:04.727 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:04.727 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:04.727 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:32:04.727 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:04.727 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:04.727 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:04.727 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:04.727 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:32:04.728 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:04.728 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:04.728 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:04.728 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:04.728 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:32:04.728 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:32:04.728 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:04.728 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:04.728 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:04.728 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:04.728 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:04.728 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:32:04.728 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:32:04.728 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:04.728 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:04.729 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:04.729 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:04.729 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:32:04.729 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:04.729 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:04.729 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:04.729 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:04.729 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:32:04.729 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:04.729 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:04.729 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:04.729 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:04.729 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:32:04.729 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:32:04.730 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:04.730 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:04.730 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:04.730 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:04.730 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:04.730 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:04.730 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:32:04.730 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:32:04.730 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:04.730 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:04.730 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:04.730 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:32:04.730 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:04.730 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:04.731 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:32:04.731 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:04.731 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:04.731 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:04.731 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:32:04.731 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:04.731 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:04.731 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:04.731 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:32:04.731 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:32:04.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.724+0000 7fd736a0b640 1 -- 172.21.15.37:0/2588322796 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd710035eb0 msgr2=0x7fd710038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:04.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.724+0000 7fd736a0b640 1 --2- 172.21.15.37:0/2588322796 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd710035eb0 0x7fd710038370 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd73006d140 tx=0x7fd7180a6070 comp rx=0 tx=0).stop 2024-04-26T00:32:04.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.724+0000 7fd736a0b640 10 monclient: shutdown 2024-04-26T00:32:04.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.724+0000 7fd736a0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:04.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.724+0000 7fd736a0b640 1 -- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7300a14f0 msgr2=0x7fd73011c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:04.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.724+0000 7fd736a0b640 1 --2- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7300a14f0 0x7fd73011c050 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd72800c3c0 tx=0x7fd72800c880 comp rx=0 tx=0).stop 2024-04-26T00:32:04.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.724+0000 7fd736a0b640 1 -- 172.21.15.37:0/2588322796 shutdown_connections 2024-04-26T00:32:04.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.724+0000 7fd736a0b640 1 --2- 172.21.15.37:0/2588322796 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd710035eb0 0x7fd710038370 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:04.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.724+0000 7fd736a0b640 1 --2- 172.21.15.37:0/2588322796 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7300a14f0 0x7fd73011c050 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:04.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.724+0000 7fd736a0b640 1 -- 172.21.15.37:0/2588322796 >> 172.21.15.37:0/2588322796 conn(0x7fd7300a1d50 msgr2=0x7fd730136c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:04.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.724+0000 7fd736a0b640 1 -- 172.21.15.37:0/2588322796 shutdown_connections 2024-04-26T00:32:04.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.724+0000 7fd736a0b640 1 -- 172.21.15.37:0/2588322796 wait complete. 2024-04-26T00:32:04.733 INFO:tasks.rados.rados.0.smithi045.stdout:2041: done (4 left) 2024-04-26T00:32:04.733 INFO:tasks.rados.rados.0.smithi045.stdout:2044: done (3 left) 2024-04-26T00:32:04.733 INFO:tasks.rados.rados.0.smithi045.stdout:2045: done (2 left) 2024-04-26T00:32:04.733 INFO:tasks.rados.rados.0.smithi045.stdout:2046: done (1 left) 2024-04-26T00:32:04.733 INFO:tasks.rados.rados.0.smithi045.stdout:2047: done (0 left) 2024-04-26T00:32:04.733 INFO:tasks.rados.rados.0.smithi045.stdout:2048: write oid 46 current snap is 231 2024-04-26T00:32:04.733 INFO:tasks.rados.rados.0.smithi045.stdout:2048: seq_num 749 ranges {588443=564095,1796660=708921} 2024-04-26T00:32:04.740 INFO:tasks.rados.rados.0.smithi045.stdout:2048: writing smithi04526579-46 from 588443 to 1152538 tid 1 2024-04-26T00:32:04.742 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:32:04.742 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:32:04.743 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:32:04.754 INFO:tasks.rados.rados.0.smithi045.stdout:2048: writing smithi04526579-46 from 1796660 to 2505581 tid 2 2024-04-26T00:32:04.755 INFO:tasks.rados.rados.0.smithi045.stdout:2049: read oid 2 snap -1 2024-04-26T00:32:04.755 INFO:tasks.rados.rados.0.smithi045.stdout:2049: expect (ObjNum 745 snap 230 seq_num 745) 2024-04-26T00:32:04.755 INFO:tasks.rados.rados.0.smithi045.stdout:2050: copy_from oid 21 from oid 38 current snap is 231 2024-04-26T00:32:04.755 INFO:tasks.rados.rados.0.smithi045.stdout:2048: finishing write tid 1 to smithi04526579-46 2024-04-26T00:32:04.755 INFO:tasks.rados.rados.0.smithi045.stdout:2051: read oid 44 snap 192 2024-04-26T00:32:04.755 INFO:tasks.rados.rados.0.smithi045.stdout:2051: expect deleted 2024-04-26T00:32:04.755 INFO:tasks.rados.rados.0.smithi045.stdout:2052: snap_create 2024-04-26T00:32:04.764 INFO:tasks.rados.rados.0.smithi045.stdout:2048: finishing write tid 2 to smithi04526579-46 2024-04-26T00:32:04.764 INFO:tasks.rados.rados.0.smithi045.stdout:2048: finishing write tid 3 to smithi04526579-46 2024-04-26T00:32:04.766 INFO:tasks.rados.rados.0.smithi045.stdout:2048: finishing write tid 4 to smithi04526579-46 2024-04-26T00:32:04.766 INFO:tasks.rados.rados.0.smithi045.stdout:2048: oid 46 updating version 0 to 1446 2024-04-26T00:32:04.766 INFO:tasks.rados.rados.0.smithi045.stdout:2048: oid 46 version 1446 is already newer than 1445 2024-04-26T00:32:04.766 INFO:tasks.rados.rados.0.smithi045.stdout:2048: oid 46 version 1446 is already newer than 1444 2024-04-26T00:32:04.766 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 1446 (ObjNum 749 snap 231 seq_num 749) dirty exists 2024-04-26T00:32:04.766 INFO:tasks.rados.rados.0.smithi045.stdout:2048: left oid 46 (ObjNum 749 snap 231 seq_num 749) 2024-04-26T00:32:04.784 INFO:tasks.rados.rados.0.smithi045.stdout:2049: expect (ObjNum 745 snap 230 seq_num 745) 2024-04-26T00:32:04.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.872+0000 7fd7f8e88640 10 monclient: get_monmap_and_config 2024-04-26T00:32:04.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.872+0000 7fd7f8e88640 10 monclient: build_initial_monmap 2024-04-26T00:32:04.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.872+0000 7fd7f8e88640 10 monclient: monmap: 2024-04-26T00:32:04.867 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:04.867 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:04.867 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:32:04.877235+0000 2024-04-26T00:32:04.867 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:32:04.877235+0000 2024-04-26T00:32:04.867 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:04.867 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:04.867 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:32:04.867 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:32:04.868 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:32:04.868 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:32:04.868 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:32:04.868 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:04.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.872+0000 7fd7f8e88640 1 Processor -- start 2024-04-26T00:32:04.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.872+0000 7fd7f8e88640 1 -- start start 2024-04-26T00:32:04.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.872+0000 7fd7f8e88640 10 monclient: init 2024-04-26T00:32:04.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.872+0000 7fd7f8e88640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:04.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.872+0000 7fd7f8e88640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-04-26T00:32:04.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.872+0000 7fd7f8e88640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd7f4130eb0 0x7fd7f409ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:04.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.872+0000 7fd7f8e88640 10 monclient(hunting): picked mon.noname-a con 0x7fd7f4130eb0 addr v2:172.21.15.37:3300/0 2024-04-26T00:32:04.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.872+0000 7fd7f8e88640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd7f4131290 addr v1:172.21.15.37:6789/0 2024-04-26T00:32:04.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.872+0000 7fd7f8e88640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd7f41342a0 addr v1:172.21.15.45:6789/0 2024-04-26T00:32:04.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.872+0000 7fd7f8e88640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7f409c190 con 0x7fd7f4131290 2024-04-26T00:32:04.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.872+0000 7fd7f8e88640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7f40a11c0 con 0x7fd7f41342a0 2024-04-26T00:32:04.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.872+0000 7fd7f8e88640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:04.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.872+0000 7fd7f8e88640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fd7f409df60 con 0x7fd7f4130eb0 2024-04-26T00:32:04.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.872+0000 7fd7f8e88640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:04.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.872+0000 7fd7f8e88640 10 monclient(hunting): authenticate will time out at 1706.183482s 2024-04-26T00:32:04.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.872+0000 7fd7f259c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd7f4130eb0 0x7fd7f409ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:04.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.872+0000 7fd7f259c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd7f4130eb0 0x7fd7f409ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:35054/0 (socket says 172.21.15.37:35054) 2024-04-26T00:32:04.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.872+0000 7fd7f259c640 1 -- 172.21.15.37:0/1456437585 learned_addr learned my addr 172.21.15.37:0/1456437585 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:32:04.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.872+0000 7fd7f1d9b640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fd7f4131290 0x7fd7f4133b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:48642/0 (socket says 172.21.15.37:48642) 2024-04-26T00:32:04.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.872+0000 7fd7f259c640 10 monclient(hunting): get_auth_request con 0x7fd7f4130eb0 auth_method 0 2024-04-26T00:32:04.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.872+0000 7fd7f259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:04.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.872+0000 7fd7f259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:04.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.872+0000 7fd7f259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:04.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.872+0000 7fd7f259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:04.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.872+0000 7fd7f259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:04.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.872+0000 7fd7f259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:04.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.876+0000 7fd7f259c640 10 monclient(hunting): handle_auth_done global_id 144105 payload 274 2024-04-26T00:32:04.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.876+0000 7fd7f259c640 1 -- 172.21.15.37:0/1456437585 >> v1:172.21.15.45:6789/0 conn(0x7fd7f41342a0 legacy=0x7fd7f413bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:04.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.876+0000 7fd7f259c640 1 -- 172.21.15.37:0/1456437585 >> v1:172.21.15.37:6789/0 conn(0x7fd7f4131290 legacy=0x7fd7f4133b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:04.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.876+0000 7fd7f259c640 10 monclient: _finish_hunting 0 2024-04-26T00:32:04.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.876+0000 7fd7f259c640 1 monclient: found mon.noname-a 2024-04-26T00:32:04.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.876+0000 7fd7f259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:04.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.876+0000 7fd7f259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:32:04.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.876+0000 7fd7f259c640 1 -- 172.21.15.37:0/1456437585 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7f409dc80 con 0x7fd7f4130eb0 2024-04-26T00:32:04.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f259c640 1 -- 172.21.15.37:0/1456437585 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd7f4130eb0 msgr2=0x7fd7f409ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:04.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f259c640 1 --2- 172.21.15.37:0/1456437585 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd7f4130eb0 0x7fd7f409ef20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4009870 tx=0x7fd7e401c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d17d545eb06a2630 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:04.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f159a640 1 -- 172.21.15.37:0/1456437585 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e40b0070 con 0x7fd7f4130eb0 2024-04-26T00:32:04.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:04.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:32:04.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f159a640 10 monclient: dump: 2024-04-26T00:32:04.875 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:04.875 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:04.875 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:04.875 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:04.875 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:04.875 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:04.875 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:04.875 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:04.875 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:04.876 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:04.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f159a640 10 monclient: _finish_auth 0 2024-04-26T00:32:04.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f159a640 10 monclient: _check_auth_tickets 2024-04-26T00:32:04.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:04.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f159a640 1 -- 172.21.15.37:0/1456437585 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7e40ab600 con 0x7fd7f4130eb0 2024-04-26T00:32:04.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f159a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:04.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f159a640 1 -- 172.21.15.37:0/1456437585 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e40ab9d0 con 0x7fd7f4130eb0 2024-04-26T00:32:04.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f8e88640 5 monclient: authenticate success, global_id 144105 2024-04-26T00:32:04.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f8e88640 10 monclient: get_monmap_and_config success 2024-04-26T00:32:04.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f8e88640 4 set_mon_vals no callback set 2024-04-26T00:32:04.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:04.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:04.876 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f159a640 10 monclient: dump: 2024-04-26T00:32:04.876 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:04.876 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:04.877 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:04.877 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:04.877 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:04.877 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:04.877 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:04.877 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:04.877 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:04.877 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:04.877 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f8e88640 10 monclient: shutdown 2024-04-26T00:32:04.877 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f8e88640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:04.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f8e88640 1 -- 172.21.15.37:0/1456437585 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd7f4130eb0 msgr2=0x7fd7f409ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:04.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f8e88640 1 --2- 172.21.15.37:0/1456437585 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd7f4130eb0 0x7fd7f409ef20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4009870 tx=0x7fd7e401c3a0 comp rx=0 tx=0).stop 2024-04-26T00:32:04.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f8e88640 1 -- 172.21.15.37:0/1456437585 shutdown_connections 2024-04-26T00:32:04.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f8e88640 1 --2- 172.21.15.37:0/1456437585 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd7f4130eb0 0x7fd7f409ef20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:04.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f8e88640 1 -- 172.21.15.37:0/1456437585 >> 172.21.15.37:0/1456437585 conn(0x7fd7f40a1d50 msgr2=0x7fd7f40a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:04.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f8e88640 1 -- 172.21.15.37:0/1456437585 shutdown_connections 2024-04-26T00:32:04.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f8e88640 1 -- 172.21.15.37:0/1456437585 wait complete. 2024-04-26T00:32:04.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f8e88640 10 monclient: build_initial_monmap 2024-04-26T00:32:04.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f8e88640 10 monclient: monmap: 2024-04-26T00:32:04.884 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:04.884 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:32:04.884 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:32:04.884 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:32:04.884 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:04.884 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:32:04.884 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:32:04.884 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:32:04.884 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:32:04.884 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:04.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f8e88640 1 Processor -- start 2024-04-26T00:32:04.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f8e88640 1 -- start start 2024-04-26T00:32:04.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f8e88640 10 monclient: init 2024-04-26T00:32:04.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f8e88640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:04.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f8e88640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:32:04.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f8e88640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7f4130eb0 0x7fd7f41cd340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:04.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f8e88640 10 monclient(hunting): picked mon.noname-c con 0x7fd7f4130eb0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:04.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f8e88640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7f4131290 0x7fd7f413bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:04.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f8e88640 10 monclient(hunting): picked mon.noname-b con 0x7fd7f4131290 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:32:04.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f8e88640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd7f41342a0 0x7fd7f4136980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:04.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f8e88640 10 monclient(hunting): picked mon.noname-a con 0x7fd7f41342a0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:32:04.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f8e88640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:04.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f8e88640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f413ff90 con 0x7fd7f41342a0 2024-04-26T00:32:04.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f8e88640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:04.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f8e88640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f413fe10 con 0x7fd7f4131290 2024-04-26T00:32:04.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f8e88640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:04.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f8e88640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd7f4140110 con 0x7fd7f4130eb0 2024-04-26T00:32:04.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f8e88640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:04.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f8e88640 10 monclient(hunting): authenticate will time out at 1706.190566s 2024-04-26T00:32:04.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f259c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7f4130eb0 0x7fd7f41cd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:04.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f2d9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd7f41342a0 0x7fd7f4136980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:04.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f259c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7f4130eb0 0x7fd7f41cd340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:56354/0 (socket says 172.21.15.37:56354) 2024-04-26T00:32:04.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f2d9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd7f41342a0 0x7fd7f4136980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:35060/0 (socket says 172.21.15.37:35060) 2024-04-26T00:32:04.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f259c640 1 -- 172.21.15.37:0/294740330 learned_addr learned my addr 172.21.15.37:0/294740330 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:32:04.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f2d9d640 10 monclient(hunting): get_auth_request con 0x7fd7f41342a0 auth_method 0 2024-04-26T00:32:04.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:04.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f2d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:04.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:04.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f259c640 10 monclient(hunting): get_auth_request con 0x7fd7f4130eb0 auth_method 0 2024-04-26T00:32:04.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:04.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:04.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:04.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:04.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:04.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:04.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:04.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:04.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:04.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f1d9b640 1 --2- 172.21.15.37:0/294740330 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7f4131290 0x7fd7f413bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:04.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f1d9b640 10 monclient(hunting): get_auth_request con 0x7fd7f4131290 auth_method 0 2024-04-26T00:32:04.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:04.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f1d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:04.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:04.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f2d9d640 10 monclient(hunting): handle_auth_done global_id 144108 payload 824 2024-04-26T00:32:04.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f2d9d640 1 -- 172.21.15.37:0/294740330 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7f4130eb0 msgr2=0x7fd7f41cd340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:04.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f2d9d640 1 --2- 172.21.15.37:0/294740330 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7f4130eb0 0x7fd7f41cd340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:04.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f2d9d640 1 -- 172.21.15.37:0/294740330 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7f4131290 msgr2=0x7fd7f413bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:04.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f2d9d640 1 --2- 172.21.15.37:0/294740330 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7f4131290 0x7fd7f413bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:04.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f2d9d640 10 monclient: _finish_hunting 0 2024-04-26T00:32:04.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f2d9d640 1 monclient: found mon.noname-a 2024-04-26T00:32:04.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:04.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f2d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:32:04.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f2d9d640 1 -- 172.21.15.37:0/294740330 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7f4139130 con 0x7fd7f41342a0 2024-04-26T00:32:04.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.880+0000 7fd7f259c640 1 --2- 172.21.15.37:0/294740330 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7f4130eb0 0x7fd7f41cd340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:32:04.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7f2d9d640 1 -- 172.21.15.37:0/294740330 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd7f41342a0 msgr2=0x7fd7f4136980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:04.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7f2d9d640 1 --2- 172.21.15.37:0/294740330 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd7f41342a0 0x7fd7f4136980 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec00c780 tx=0x7fd7ec00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:04.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7e2ffd640 1 -- 172.21.15.37:0/294740330 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ec016020 con 0x7fd7f41342a0 2024-04-26T00:32:04.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:04.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7e2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:32:04.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7e2ffd640 10 monclient: dump: 2024-04-26T00:32:04.889 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:04.889 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:04.889 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:04.889 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:04.889 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:04.889 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:04.889 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:04.890 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:04.890 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:04.890 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:04.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7e2ffd640 10 monclient: _finish_auth 0 2024-04-26T00:32:04.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7e2ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:32:04.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:04.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7e2ffd640 1 -- 172.21.15.37:0/294740330 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7ec00ac60 con 0x7fd7f41342a0 2024-04-26T00:32:04.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7e2ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:04.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7e2ffd640 1 -- 172.21.15.37:0/294740330 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ec012660 con 0x7fd7f41342a0 2024-04-26T00:32:04.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:04.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7e2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:04.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7e2ffd640 10 monclient: dump: 2024-04-26T00:32:04.890 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:04.890 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:04.891 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:04.891 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:04.891 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:04.891 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:04.891 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:04.891 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:04.891 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:04.891 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:04.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7f0d99640 4 set_mon_vals no callback set 2024-04-26T00:32:04.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7f8e88640 5 monclient: authenticate success, global_id 144108 2024-04-26T00:32:04.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7f8e88640 10 monclient: _renew_subs 2024-04-26T00:32:04.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7f8e88640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:04.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7f8e88640 1 -- 172.21.15.37:0/294740330 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7f413a300 con 0x7fd7f41342a0 2024-04-26T00:32:04.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7f8e88640 10 monclient: _renew_subs 2024-04-26T00:32:04.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7f8e88640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:04.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7f8e88640 1 -- 172.21.15.37:0/294740330 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7f4138eb0 con 0x7fd7f41342a0 2024-04-26T00:32:04.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7f8e88640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:04.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7f8e88640 1 --2- 172.21.15.37:0/294740330 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7f413c230 0x7fd7f402c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:04.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7f8e88640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:04.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7f8e88640 1 -- 172.21.15.37:0/294740330 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd7f404f510 con 0x7fd7f413c230 2024-04-26T00:32:04.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7f259c640 1 --2- 172.21.15.37:0/294740330 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7f413c230 0x7fd7f402c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:04.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7f259c640 10 monclient: get_auth_request con 0x7fd7f413c230 auth_method 0 2024-04-26T00:32:04.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7f259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:04.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7f259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:04.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7f259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:04.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7e2ffd640 1 -- 172.21.15.37:0/294740330 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd7ec019020 con 0x7fd7f41342a0 2024-04-26T00:32:04.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7f259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:04.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7f259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:04.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7f259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:04.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7f259c640 10 monclient(hunting): handle_auth_done global_id 84653 payload 824 2024-04-26T00:32:04.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7f259c640 1 -- 172.21.15.37:0/294740330 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd7f413cae0 con 0x7fd7f413c230 2024-04-26T00:32:04.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7e2ffd640 1 --2- 172.21.15.37:0/294740330 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd7d4035ed0 0x7fd7d4038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:04.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7e2ffd640 1 -- 172.21.15.37:0/294740330 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(467..467 src has 370..467) ==== 3915+0+0 (secure 0 0 0) 0x7fd7ec048950 con 0x7fd7f41342a0 2024-04-26T00:32:04.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7f1d9b640 1 --2- 172.21.15.37:0/294740330 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd7d4035ed0 0x7fd7d4038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:04.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.884+0000 7fd7f1d9b640 10 monclient: get_auth_request con 0x7fd7d4035ed0 auth_method 0 2024-04-26T00:32:04.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.888+0000 7fd7f259c640 1 -- 172.21.15.37:0/294740330 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7f413c230 msgr2=0x7fd7f402c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:04.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.888+0000 7fd7f259c640 1 --2- 172.21.15.37:0/294740330 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7f413c230 0x7fd7f402c000 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd7e401aa50 tx=0x7fd7e401abe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:04.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.888+0000 7fd7e2ffd640 1 -- 172.21.15.37:0/294740330 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e40b0040 con 0x7fd7f413c230 2024-04-26T00:32:04.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.888+0000 7fd7e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:04.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.888+0000 7fd7e2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:32:04.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.888+0000 7fd7e2ffd640 10 monclient: dump: 2024-04-26T00:32:04.894 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:04.894 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:04.894 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:04.894 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:04.894 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:04.894 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:04.894 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:04.894 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:04.894 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:04.894 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:04.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.888+0000 7fd7e2ffd640 1 -- 172.21.15.37:0/294740330 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd7e40b6880 con 0x7fd7f413c230 2024-04-26T00:32:04.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.888+0000 7fd7e2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:04.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.888+0000 7fd7e2ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:32:04.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.888+0000 7fd7e2ffd640 1 -- 172.21.15.37:0/294740330 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7f413c230 msgr2=0x7fd7f402c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:04.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.888+0000 7fd7e2ffd640 1 --2- 172.21.15.37:0/294740330 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7f413c230 0x7fd7f402c000 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd7e401aa50 tx=0x7fd7e401abe0 comp rx=0 tx=0).stop 2024-04-26T00:32:04.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.888+0000 7fd7e2ffd640 1 -- 172.21.15.37:0/294740330 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7f413c230 msgr2=0x7fd7f402c000 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:32:04.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.888+0000 7fd7e2ffd640 1 --2- 172.21.15.37:0/294740330 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7f413c230 0x7fd7f402c000 secure :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd7e401aa50 tx=0x7fd7e401abe0 comp rx=0 tx=0).stop 2024-04-26T00:32:04.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.888+0000 7fd7f1d9b640 1 --2- 172.21.15.37:0/294740330 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd7d4035ed0 0x7fd7d4038390 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fd7f413ca00 tx=0x7fd7e8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:04.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.896+0000 7fd7f8e88640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:32:04.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.896+0000 7fd7f8e88640 1 --2- 172.21.15.37:0/294740330 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7f402c540 0x7fd7f402c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:04.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.896+0000 7fd7f8e88640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:04.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.896+0000 7fd7f8e88640 1 -- 172.21.15.37:0/294740330 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd7f40a0f60 con 0x7fd7f402c540 2024-04-26T00:32:04.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.896+0000 7fd7f259c640 1 --2- 172.21.15.37:0/294740330 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7f402c540 0x7fd7f402c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:04.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.896+0000 7fd7f259c640 10 monclient: get_auth_request con 0x7fd7f402c540 auth_method 0 2024-04-26T00:32:04.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.896+0000 7fd7f259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:04.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.896+0000 7fd7f259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:04.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.896+0000 7fd7f259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:04.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.896+0000 7fd7f259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:04.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.896+0000 7fd7f259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:04.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.896+0000 7fd7f259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:04.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.896+0000 7fd7f259c640 10 monclient(hunting): handle_auth_done global_id 84656 payload 824 2024-04-26T00:32:04.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.896+0000 7fd7f259c640 1 -- 172.21.15.37:0/294740330 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd7f4071360 con 0x7fd7f402c540 2024-04-26T00:32:04.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.896+0000 7fd7f259c640 1 -- 172.21.15.37:0/294740330 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7f402c540 msgr2=0x7fd7f402c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:04.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.896+0000 7fd7f259c640 1 --2- 172.21.15.37:0/294740330 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7f402c540 0x7fd7f402c950 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4000c10 tx=0x7fd7e4019140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:04.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.896+0000 7fd7e2ffd640 1 -- 172.21.15.37:0/294740330 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e40b0040 con 0x7fd7f402c540 2024-04-26T00:32:04.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.896+0000 7fd7e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:04.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.896+0000 7fd7e2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:32:04.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.896+0000 7fd7e2ffd640 10 monclient: dump: 2024-04-26T00:32:04.898 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:04.898 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:04.898 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:04.898 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:04.898 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:04.898 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:04.898 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:04.899 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:04.899 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:04.899 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:04.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.896+0000 7fd7e2ffd640 1 -- 172.21.15.37:0/294740330 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4889 (secure 0 0 0) 0x7fd7e40b7420 con 0x7fd7f402c540 2024-04-26T00:32:04.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.896+0000 7fd7e2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:32:04.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.896+0000 7fd7e2ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:32:04.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.896+0000 7fd7e2ffd640 1 -- 172.21.15.37:0/294740330 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7f402c540 msgr2=0x7fd7f402c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:04.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.896+0000 7fd7e2ffd640 1 --2- 172.21.15.37:0/294740330 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7f402c540 0x7fd7f402c950 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4000c10 tx=0x7fd7e4019140 comp rx=0 tx=0).stop 2024-04-26T00:32:04.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.896+0000 7fd7e2ffd640 1 -- 172.21.15.37:0/294740330 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7f402c540 msgr2=0x7fd7f402c950 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:32:04.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.896+0000 7fd7e2ffd640 1 --2- 172.21.15.37:0/294740330 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7f402c540 0x7fd7f402c950 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:04.899 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:32:04.900 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:32:04.900 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:32:04.900 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:32:04.900 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 92, 2024-04-26T00:32:04.900 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 329831, 2024-04-26T00:32:04.900 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:32:04.900 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:32:04.900 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:32:04.900 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:32:04.900 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:04.900 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 5, 2024-04-26T00:32:04.900 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:32:04.900 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:32:04.901 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:32:04.901 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:04.901 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:04.901 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:04.901 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:04.901 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:04.901 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:04.901 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:04.901 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:04.901 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:04.901 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:04.901 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:04.901 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:04.902 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:32:04.902 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:32:04.902 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:04.902 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:04.902 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:04.902 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:04.902 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:04.902 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:04.902 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:04.902 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:04.902 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:04.902 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:04.902 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:04.902 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:04.903 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:04.903 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:32:04.903 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:32:04.903 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:32:04.903 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:32:04.903 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:32:04.903 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:32:04.903 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:32:04.903 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:32:04.903 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:32:04.903 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:32:04.903 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:32:04.903 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:32:04.903 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:32:04.904 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:32:04.904 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:32:04.904 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:32:04.904 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:32:04.904 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:04.904 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:04.904 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:04.904 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:04.904 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:04.904 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:04.904 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:04.904 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:04.904 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:04.904 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:04.904 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:04.905 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:04.905 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:32:04.905 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:04.905 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:32:04.905 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:04.905 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:32:04.905 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:32:04.905 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:04.905 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:04.905 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:04.905 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:04.905 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:32:04.905 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:04.905 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:04.906 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:04.906 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:04.906 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:32:04.906 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:04.906 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:04.906 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:04.906 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:04.906 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:32:04.906 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:32:04.906 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:04.906 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:04.906 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:04.906 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:04.907 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:04.907 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:32:04.907 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:32:04.907 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:04.907 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:04.907 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:04.907 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:04.907 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:32:04.907 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:04.907 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:04.907 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:04.907 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:04.907 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:32:04.908 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:04.908 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:04.908 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:04.908 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:04.908 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:32:04.908 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:32:04.908 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:04.908 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:04.908 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:04.908 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:04.908 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:04.908 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:32:04.908 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:32:04.908 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:04.908 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:04.908 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:04.909 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:04.909 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:32:04.909 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:04.909 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:04.909 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:04.909 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:04.909 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:32:04.909 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:04.909 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:04.909 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:04.909 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:04.909 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:32:04.909 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:32:04.909 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:04.910 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:04.910 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:04.910 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:04.910 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:04.910 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:04.910 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:32:04.910 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:32:04.910 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:04.910 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:04.910 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:04.910 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:32:04.910 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:04.910 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:04.910 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:32:04.910 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:04.911 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:04.911 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:04.911 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:32:04.911 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:04.911 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:04.911 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:32:04.911 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:04.911 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:04.911 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:04.911 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:32:04.911 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:04.911 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:04.911 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:32:04.911 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:04.912 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:04.912 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:04.912 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:32:04.912 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:04.912 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:04.912 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:04.912 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:32:04.912 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:32:04.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.896+0000 7fd7f8e88640 1 -- 172.21.15.37:0/294740330 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd7d4035ed0 msgr2=0x7fd7d4038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:04.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.896+0000 7fd7f8e88640 1 --2- 172.21.15.37:0/294740330 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd7d4035ed0 0x7fd7d4038390 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fd7f413ca00 tx=0x7fd7e8009040 comp rx=0 tx=0).stop 2024-04-26T00:32:04.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.896+0000 7fd7f8e88640 10 monclient: shutdown 2024-04-26T00:32:04.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.896+0000 7fd7f8e88640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:04.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.896+0000 7fd7f8e88640 1 -- 172.21.15.37:0/294740330 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd7f41342a0 msgr2=0x7fd7f4136980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:04.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.896+0000 7fd7f8e88640 1 --2- 172.21.15.37:0/294740330 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd7f41342a0 0x7fd7f4136980 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec00c780 tx=0x7fd7ec00cc40 comp rx=0 tx=0).stop 2024-04-26T00:32:04.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.896+0000 7fd7f2d9d640 1 -- 172.21.15.37:0/294740330 reap_dead start 2024-04-26T00:32:04.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.896+0000 7fd7f2d9d640 1 -- 172.21.15.37:0/294740330 reap_dead start 2024-04-26T00:32:04.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.896+0000 7fd7f8e88640 1 -- 172.21.15.37:0/294740330 shutdown_connections 2024-04-26T00:32:04.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.896+0000 7fd7f8e88640 1 -- 172.21.15.37:0/294740330 >> 172.21.15.37:0/294740330 conn(0x7fd7f40a1d50 msgr2=0x7fd7f40a3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:04.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.896+0000 7fd7f8e88640 1 -- 172.21.15.37:0/294740330 shutdown_connections 2024-04-26T00:32:04.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:04.896+0000 7fd7f8e88640 1 -- 172.21.15.37:0/294740330 wait complete. 2024-04-26T00:32:04.914 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:32:04.915 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:32:04.915 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:32:05.018 INFO:tasks.rados.rados.0.smithi045.stdout:2050: finishing copy_from racing read to smithi04526579-21 2024-04-26T00:32:05.018 INFO:tasks.rados.rados.0.smithi045.stdout:2050: finishing copy_from to smithi04526579-21 2024-04-26T00:32:05.018 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 1089 (ObjNum 582 snap 180 seq_num 582) dirty exists 2024-04-26T00:32:05.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f22515640 10 monclient: get_monmap_and_config 2024-04-26T00:32:05.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f22515640 10 monclient: build_initial_monmap 2024-04-26T00:32:05.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f22515640 10 monclient: monmap: 2024-04-26T00:32:05.030 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:05.030 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:05.030 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:32:05.040092+0000 2024-04-26T00:32:05.030 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:32:05.040092+0000 2024-04-26T00:32:05.030 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:05.030 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:05.030 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:32:05.030 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:32:05.030 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:32:05.030 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:32:05.030 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:32:05.030 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:05.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f22515640 1 Processor -- start 2024-04-26T00:32:05.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f22515640 1 -- start start 2024-04-26T00:32:05.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f22515640 10 monclient: init 2024-04-26T00:32:05.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f22515640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:05.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f22515640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-04-26T00:32:05.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f22515640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6f1c133ee0 addr v1:172.21.15.45:6789/0 2024-04-26T00:32:05.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f22515640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6f1c09bf40 0x7f6f1c13c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:05.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f22515640 10 monclient(hunting): picked mon.noname-a con 0x7f6f1c09bf40 addr v2:172.21.15.37:3300/0 2024-04-26T00:32:05.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f22515640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6f1c13c6a0 0x7f6f1c13ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:05.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f22515640 10 monclient(hunting): picked mon.noname-c con 0x7f6f1c13c6a0 addr v2:172.21.15.45:3300/0 2024-04-26T00:32:05.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f22515640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f1c1349d0 con 0x7f6f1c133ee0 2024-04-26T00:32:05.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f22515640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:05.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f22515640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f6f1c09ca40 con 0x7f6f1c09bf40 2024-04-26T00:32:05.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f22515640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:05.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f22515640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f6f1c09c3b0 con 0x7f6f1c13c6a0 2024-04-26T00:32:05.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f22515640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:05.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f22515640 10 monclient(hunting): authenticate will time out at 1706.345757s 2024-04-26T00:32:05.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f1b7fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6f1c09bf40 0x7f6f1c13c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:05.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f1b7fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6f1c09bf40 0x7f6f1c13c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:35066/0 (socket says 172.21.15.37:35066) 2024-04-26T00:32:05.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f1b7fe640 1 -- 172.21.15.37:0/1103114473 learned_addr learned my addr 172.21.15.37:0/1103114473 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:32:05.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f1b7fe640 10 monclient(hunting): get_auth_request con 0x7f6f1c09bf40 auth_method 0 2024-04-26T00:32:05.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f1b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:05.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f1b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:05.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:05.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f1b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:05.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f1b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:05.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f20ab2640 1 --2- 172.21.15.37:0/1103114473 >> v2:172.21.15.45:3300/0 conn(0x7f6f1c13c6a0 0x7f6f1c13ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:05.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f1b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:05.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f20ab2640 10 monclient(hunting): get_auth_request con 0x7f6f1c13c6a0 auth_method 0 2024-04-26T00:32:05.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f20ab2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:05.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f20ab2640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:05.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f20ab2640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:05.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f1b7fe640 10 monclient(hunting): handle_auth_done global_id 144111 payload 274 2024-04-26T00:32:05.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f1b7fe640 1 -- 172.21.15.37:0/1103114473 >> v2:172.21.15.45:3300/0 conn(0x7f6f1c13c6a0 msgr2=0x7f6f1c13ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:05.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f1b7fe640 1 --2- 172.21.15.37:0/1103114473 >> v2:172.21.15.45:3300/0 conn(0x7f6f1c13c6a0 0x7f6f1c13ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:05.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f1b7fe640 1 -- 172.21.15.37:0/1103114473 >> v1:172.21.15.45:6789/0 conn(0x7f6f1c133ee0 legacy=0x7f6f1c1342c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:05.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f1b7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:32:05.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f1b7fe640 1 monclient: found mon.noname-a 2024-04-26T00:32:05.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:05.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f1b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:32:05.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.036+0000 7f6f1b7fe640 1 -- 172.21.15.37:0/1103114473 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f1c09b280 con 0x7f6f1c09bf40 2024-04-26T00:32:05.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.040+0000 7f6f1b7fe640 1 -- 172.21.15.37:0/1103114473 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6f1c09bf40 msgr2=0x7f6f1c13c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:05.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.040+0000 7f6f1b7fe640 1 --2- 172.21.15.37:0/1103114473 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6f1c09bf40 0x7f6f1c13c160 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c0a5ea0 tx=0x7f6f1001c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0c4ea3f5cccf75e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:05.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.040+0000 7f6f1affd640 1 -- 172.21.15.37:0/1103114473 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f100b0070 con 0x7f6f1c09bf40 2024-04-26T00:32:05.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.040+0000 7f6f1affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:05.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.040+0000 7f6f1affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:32:05.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.040+0000 7f6f1affd640 10 monclient: dump: 2024-04-26T00:32:05.038 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:05.038 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:05.038 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:05.038 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:05.038 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:05.038 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:05.038 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:05.038 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:05.038 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:05.039 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:05.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.040+0000 7f6f1affd640 10 monclient: _finish_auth 0 2024-04-26T00:32:05.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.040+0000 7f6f1affd640 10 monclient: _check_auth_tickets 2024-04-26T00:32:05.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.040+0000 7f6f1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:05.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.040+0000 7f6f1affd640 1 -- 172.21.15.37:0/1103114473 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f100ab600 con 0x7f6f1c09bf40 2024-04-26T00:32:05.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.040+0000 7f6f22515640 5 monclient: authenticate success, global_id 144111 2024-04-26T00:32:05.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.040+0000 7f6f22515640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:32:05.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.040+0000 7f6f1affd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:05.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.040+0000 7f6f22515640 10 monclient: get_monmap_and_config success 2024-04-26T00:32:05.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.040+0000 7f6f1affd640 1 -- 172.21.15.37:0/1103114473 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f100ab9d0 con 0x7f6f1c09bf40 2024-04-26T00:32:05.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.040+0000 7f6f22515640 4 set_mon_vals no callback set 2024-04-26T00:32:05.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.040+0000 7f6f1affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:05.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.040+0000 7f6f1affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:05.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.040+0000 7f6f1affd640 10 monclient: dump: 2024-04-26T00:32:05.040 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:05.040 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:05.040 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:05.040 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:05.040 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:05.040 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:05.040 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:05.040 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:05.040 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:05.040 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:05.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.040+0000 7f6f22515640 10 monclient: shutdown 2024-04-26T00:32:05.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.040+0000 7f6f22515640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:05.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.040+0000 7f6f22515640 1 -- 172.21.15.37:0/1103114473 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6f1c09bf40 msgr2=0x7f6f1c13c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:05.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.040+0000 7f6f22515640 1 --2- 172.21.15.37:0/1103114473 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6f1c09bf40 0x7f6f1c13c160 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c0a5ea0 tx=0x7f6f1001c370 comp rx=0 tx=0).stop 2024-04-26T00:32:05.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.040+0000 7f6f22515640 1 -- 172.21.15.37:0/1103114473 shutdown_connections 2024-04-26T00:32:05.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.040+0000 7f6f22515640 1 --2- 172.21.15.37:0/1103114473 >> v2:172.21.15.45:3300/0 conn(0x7f6f1c13c6a0 0x7f6f1c13ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:05.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.040+0000 7f6f22515640 1 --2- 172.21.15.37:0/1103114473 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6f1c09bf40 0x7f6f1c13c160 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:05.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.040+0000 7f6f22515640 1 -- 172.21.15.37:0/1103114473 >> 172.21.15.37:0/1103114473 conn(0x7f6f1c0a5f00 msgr2=0x7f6f1c0a62e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:05.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.040+0000 7f6f22515640 1 -- 172.21.15.37:0/1103114473 shutdown_connections 2024-04-26T00:32:05.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.040+0000 7f6f22515640 1 -- 172.21.15.37:0/1103114473 wait complete. 2024-04-26T00:32:05.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f22515640 10 monclient: build_initial_monmap 2024-04-26T00:32:05.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f22515640 10 monclient: monmap: 2024-04-26T00:32:05.041 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:05.041 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:32:05.041 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:32:05.041 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:32:05.042 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:05.042 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:32:05.042 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:32:05.042 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:32:05.042 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:32:05.042 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:05.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f22515640 1 Processor -- start 2024-04-26T00:32:05.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f22515640 1 -- start start 2024-04-26T00:32:05.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f22515640 10 monclient: init 2024-04-26T00:32:05.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f22515640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:05.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f22515640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:32:05.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f22515640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f1c09bf40 0x7f6f1c1cee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:05.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f22515640 10 monclient(hunting): picked mon.noname-b con 0x7f6f1c09bf40 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:32:05.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f22515640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6f1c133ee0 0x7f6f1c13b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:05.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f22515640 10 monclient(hunting): picked mon.noname-a con 0x7f6f1c133ee0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:32:05.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f22515640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6f1c13c6a0 0x7f6f1c13b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:05.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f22515640 10 monclient(hunting): picked mon.noname-c con 0x7f6f1c13c6a0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:05.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f22515640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:05.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f22515640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f6f1c06dec0 con 0x7f6f1c133ee0 2024-04-26T00:32:05.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f22515640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:05.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f22515640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6f1c06dd40 con 0x7f6f1c09bf40 2024-04-26T00:32:05.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f22515640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:05.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f22515640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f6f1c1417f0 con 0x7f6f1c13c6a0 2024-04-26T00:32:05.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f22515640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:05.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f22515640 10 monclient(hunting): authenticate will time out at 1706.353588s 2024-04-26T00:32:05.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f1b7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6f1c133ee0 0x7f6f1c13b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:05.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f20ab2640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6f1c13c6a0 0x7f6f1c13b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:05.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f1b7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6f1c133ee0 0x7f6f1c13b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:35078/0 (socket says 172.21.15.37:35078) 2024-04-26T00:32:05.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f1b7fe640 1 -- 172.21.15.37:0/500612940 learned_addr learned my addr 172.21.15.37:0/500612940 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:32:05.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f20ab2640 10 monclient(hunting): get_auth_request con 0x7f6f1c13c6a0 auth_method 0 2024-04-26T00:32:05.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f20ab2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:05.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f20ab2640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:05.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f20ab2640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:05.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f1b7fe640 10 monclient(hunting): get_auth_request con 0x7f6f1c133ee0 auth_method 0 2024-04-26T00:32:05.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f1b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:05.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f1b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:05.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:05.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f20ab2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:05.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f20ab2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:05.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f20ab2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:05.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f1b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:05.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f1b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:05.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f1b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:05.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f1bfff640 1 --2- 172.21.15.37:0/500612940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f1c09bf40 0x7f6f1c1cee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:05.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f1bfff640 10 monclient(hunting): get_auth_request con 0x7f6f1c09bf40 auth_method 0 2024-04-26T00:32:05.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:05.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f1bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:05.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:05.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f1b7fe640 10 monclient(hunting): handle_auth_done global_id 144114 payload 824 2024-04-26T00:32:05.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f1b7fe640 1 -- 172.21.15.37:0/500612940 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6f1c13c6a0 msgr2=0x7f6f1c13b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:05.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f1b7fe640 1 --2- 172.21.15.37:0/500612940 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6f1c13c6a0 0x7f6f1c13b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:05.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f1b7fe640 1 -- 172.21.15.37:0/500612940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f1c09bf40 msgr2=0x7f6f1c1cee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:05.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f1b7fe640 1 --2- 172.21.15.37:0/500612940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f1c09bf40 0x7f6f1c1cee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:05.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f1b7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:32:05.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f1b7fe640 1 monclient: found mon.noname-a 2024-04-26T00:32:05.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:05.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f1b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:32:05.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f1b7fe640 1 -- 172.21.15.37:0/500612940 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f1c137960 con 0x7f6f1c133ee0 2024-04-26T00:32:05.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f20ab2640 1 --2- 172.21.15.37:0/500612940 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6f1c13c6a0 0x7f6f1c13b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:32:05.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.044+0000 7f6f1bfff640 1 --2- 172.21.15.37:0/500612940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f1c09bf40 0x7f6f1c1cee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:32:05.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f1b7fe640 1 -- 172.21.15.37:0/500612940 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6f1c133ee0 msgr2=0x7f6f1c13b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:05.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f1b7fe640 1 --2- 172.21.15.37:0/500612940 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6f1c133ee0 0x7f6f1c13b200 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6f100ae040 tx=0x7f6f10019110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:05.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f18ff9640 1 -- 172.21.15.37:0/500612940 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f100b0040 con 0x7f6f1c133ee0 2024-04-26T00:32:05.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f18ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:05.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f18ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:32:05.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f18ff9640 10 monclient: dump: 2024-04-26T00:32:05.051 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:05.051 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:05.051 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:05.051 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:05.052 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:05.052 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:05.052 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:05.052 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:05.052 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:05.052 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:05.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f18ff9640 10 monclient: _finish_auth 0 2024-04-26T00:32:05.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f18ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:32:05.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f18ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:05.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f18ff9640 1 -- 172.21.15.37:0/500612940 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f100b9070 con 0x7f6f1c133ee0 2024-04-26T00:32:05.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f18ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:05.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f18ff9640 1 -- 172.21.15.37:0/500612940 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f100b37f0 con 0x7f6f1c133ee0 2024-04-26T00:32:05.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f18ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:05.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f18ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:05.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f18ff9640 10 monclient: dump: 2024-04-26T00:32:05.053 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:05.053 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:05.053 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:05.053 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:05.053 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:05.053 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:05.053 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:05.053 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:05.053 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:05.053 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:05.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f22515640 5 monclient: authenticate success, global_id 144114 2024-04-26T00:32:05.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f22515640 10 monclient: _renew_subs 2024-04-26T00:32:05.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f22515640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:05.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f22515640 1 -- 172.21.15.37:0/500612940 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f1c138a00 con 0x7f6f1c133ee0 2024-04-26T00:32:05.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f1a7fc640 4 set_mon_vals no callback set 2024-04-26T00:32:05.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f22515640 10 monclient: _renew_subs 2024-04-26T00:32:05.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f22515640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:05.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f22515640 1 -- 172.21.15.37:0/500612940 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f1c138540 con 0x7f6f1c133ee0 2024-04-26T00:32:05.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f22515640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:05.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f22515640 1 --2- 172.21.15.37:0/500612940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f1c139430 0x7f6f1c135d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:05.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f22515640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:05.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f22515640 1 -- 172.21.15.37:0/500612940 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6f1c04f510 con 0x7f6f1c139430 2024-04-26T00:32:05.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f18ff9640 1 -- 172.21.15.37:0/500612940 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f6f100b3990 con 0x7f6f1c133ee0 2024-04-26T00:32:05.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f1bfff640 1 --2- 172.21.15.37:0/500612940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f1c139430 0x7f6f1c135d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:05.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f18ff9640 1 --2- 172.21.15.37:0/500612940 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6f08035e80 0x7f6f08038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:05.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f18ff9640 1 -- 172.21.15.37:0/500612940 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(467..467 src has 370..467) ==== 3915+0+0 (secure 0 0 0) 0x7f6f100e27b0 con 0x7f6f1c133ee0 2024-04-26T00:32:05.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f1bfff640 10 monclient: get_auth_request con 0x7f6f1c139430 auth_method 0 2024-04-26T00:32:05.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:05.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f1bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:05.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:05.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:05.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:05.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:05.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f20ab2640 1 --2- 172.21.15.37:0/500612940 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6f08035e80 0x7f6f08038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:05.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f20ab2640 10 monclient: get_auth_request con 0x7f6f08035e80 auth_method 0 2024-04-26T00:32:05.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f1bfff640 10 monclient(hunting): handle_auth_done global_id 134203 payload 824 2024-04-26T00:32:05.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.048+0000 7f6f1bfff640 1 -- 172.21.15.37:0/500612940 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6f1c0a8990 con 0x7f6f1c139430 2024-04-26T00:32:05.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.052+0000 7f6f1bfff640 1 -- 172.21.15.37:0/500612940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f1c139430 msgr2=0x7f6f1c135d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:05.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.052+0000 7f6f1bfff640 1 --2- 172.21.15.37:0/500612940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f1c139430 0x7f6f1c135d10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c00e500 tx=0x7f6f0c00e9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:05.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.052+0000 7f6f20ab2640 1 --2- 172.21.15.37:0/500612940 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6f08035e80 0x7f6f08038340 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6f140077c0 tx=0x7f6f14009820 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:05.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.052+0000 7f6f18ff9640 1 -- 172.21.15.37:0/500612940 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c013800 con 0x7f6f1c139430 2024-04-26T00:32:05.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.052+0000 7f6f18ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:05.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.052+0000 7f6f18ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:32:05.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.052+0000 7f6f18ff9640 10 monclient: dump: 2024-04-26T00:32:05.059 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:05.059 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:05.059 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:05.059 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:05.059 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:05.059 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:05.059 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:05.059 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:05.060 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:05.060 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:05.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.052+0000 7f6f18ff9640 1 -- 172.21.15.37:0/500612940 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6f0c013800 con 0x7f6f1c139430 2024-04-26T00:32:05.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.052+0000 7f6f18ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:05.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.052+0000 7f6f18ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:32:05.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.052+0000 7f6f18ff9640 1 -- 172.21.15.37:0/500612940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f1c139430 msgr2=0x7f6f1c135d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:05.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.052+0000 7f6f18ff9640 1 --2- 172.21.15.37:0/500612940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f1c139430 0x7f6f1c135d10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c00e500 tx=0x7f6f0c00e9c0 comp rx=0 tx=0).stop 2024-04-26T00:32:05.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.052+0000 7f6f18ff9640 1 -- 172.21.15.37:0/500612940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f1c139430 msgr2=0x7f6f1c135d10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:32:05.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.052+0000 7f6f18ff9640 1 --2- 172.21.15.37:0/500612940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f1c139430 0x7f6f1c135d10 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c00e500 tx=0x7f6f0c00e9c0 comp rx=0 tx=0).stop 2024-04-26T00:32:05.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.060+0000 7f6f22515640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:32:05.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.060+0000 7f6f22515640 1 --2- 172.21.15.37:0/500612940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f1c136250 0x7f6f1c136660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:05.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.060+0000 7f6f22515640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:05.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.060+0000 7f6f22515640 1 -- 172.21.15.37:0/500612940 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6f1c136ba0 con 0x7f6f1c136250 2024-04-26T00:32:05.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.060+0000 7f6f1bfff640 1 --2- 172.21.15.37:0/500612940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f1c136250 0x7f6f1c136660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:05.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.060+0000 7f6f1bfff640 10 monclient: get_auth_request con 0x7f6f1c136250 auth_method 0 2024-04-26T00:32:05.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.060+0000 7f6f1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:05.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.060+0000 7f6f1bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:05.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.060+0000 7f6f1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:05.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.060+0000 7f6f1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:05.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.060+0000 7f6f1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:05.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.060+0000 7f6f1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:05.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.060+0000 7f6f1bfff640 10 monclient(hunting): handle_auth_done global_id 134206 payload 824 2024-04-26T00:32:05.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.060+0000 7f6f1bfff640 1 -- 172.21.15.37:0/500612940 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6f1c1342c0 con 0x7f6f1c136250 2024-04-26T00:32:05.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.064+0000 7f6f1bfff640 1 -- 172.21.15.37:0/500612940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f1c136250 msgr2=0x7f6f1c136660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:05.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.064+0000 7f6f1bfff640 1 --2- 172.21.15.37:0/500612940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f1c136250 0x7f6f1c136660 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c00e500 tx=0x7f6f0c003770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:05.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.064+0000 7f6f18ff9640 1 -- 172.21.15.37:0/500612940 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c0069e0 con 0x7f6f1c136250 2024-04-26T00:32:05.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.064+0000 7f6f18ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:05.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.064+0000 7f6f18ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:32:05.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.064+0000 7f6f18ff9640 10 monclient: dump: 2024-04-26T00:32:05.064 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:05.064 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:05.064 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:05.064 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:05.064 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:05.064 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:05.064 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:05.064 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:05.065 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:05.065 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:05.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.064+0000 7f6f18ff9640 1 -- 172.21.15.37:0/500612940 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4716 (secure 0 0 0) 0x7f6f0c0069e0 con 0x7f6f1c136250 2024-04-26T00:32:05.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.064+0000 7f6f18ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:32:05.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.064+0000 7f6f18ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:32:05.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.064+0000 7f6f18ff9640 1 -- 172.21.15.37:0/500612940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f1c136250 msgr2=0x7f6f1c136660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:05.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.064+0000 7f6f18ff9640 1 --2- 172.21.15.37:0/500612940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f1c136250 0x7f6f1c136660 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c00e500 tx=0x7f6f0c003770 comp rx=0 tx=0).stop 2024-04-26T00:32:05.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.064+0000 7f6f18ff9640 1 -- 172.21.15.37:0/500612940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f1c136250 msgr2=0x7f6f1c136660 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:32:05.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.064+0000 7f6f18ff9640 1 --2- 172.21.15.37:0/500612940 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f1c136250 0x7f6f1c136660 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c00e500 tx=0x7f6f0c003770 comp rx=0 tx=0).stop 2024-04-26T00:32:05.066 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:32:05.066 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:32:05.066 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:32:05.066 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:32:05.066 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 92, 2024-04-26T00:32:05.066 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 55423, 2024-04-26T00:32:05.066 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:32:05.066 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:32:05.066 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:32:05.066 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:32:05.067 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:05.067 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 5, 2024-04-26T00:32:05.067 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:32:05.067 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:32:05.067 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:32:05.067 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:05.067 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:05.067 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:05.067 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:05.067 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:05.067 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:05.067 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:05.067 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:05.068 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:05.068 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:05.068 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:05.068 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:05.068 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:32:05.068 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:32:05.068 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:05.068 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:05.068 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:05.068 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:05.068 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:05.068 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:05.068 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:05.068 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:05.069 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:05.069 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:05.069 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:05.069 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:05.069 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:05.069 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:32:05.069 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:32:05.069 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:32:05.069 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:32:05.069 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:32:05.069 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:32:05.069 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:32:05.069 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:32:05.069 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:32:05.070 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:32:05.070 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:32:05.070 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:32:05.070 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:32:05.070 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:32:05.070 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:32:05.070 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:32:05.070 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:32:05.070 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:05.070 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:05.070 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:05.070 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:05.070 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:05.071 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:05.071 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:05.071 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:05.071 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:05.071 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:05.071 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:05.071 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:05.071 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:32:05.071 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:05.071 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:32:05.071 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:05.071 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:32:05.071 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:32:05.071 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:05.072 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:05.072 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:05.072 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:05.072 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:32:05.072 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:05.072 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:05.072 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:05.072 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:05.072 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:32:05.072 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:05.072 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:05.072 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:05.072 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:05.072 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:32:05.072 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:32:05.073 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:05.073 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:05.073 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:05.073 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:05.073 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:05.073 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:32:05.073 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:32:05.073 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:05.073 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:05.073 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:05.073 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:05.073 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:32:05.073 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:05.073 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:05.073 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:05.074 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:05.074 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:32:05.074 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:05.074 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:05.074 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:05.074 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:05.074 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:32:05.074 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:32:05.074 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:05.074 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:05.074 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:05.074 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:05.074 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:05.075 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:32:05.075 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:32:05.075 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:05.075 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:05.075 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:05.075 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:05.075 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:32:05.075 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:05.075 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:05.075 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:05.075 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:05.076 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:32:05.076 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:05.076 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:05.076 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:05.076 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:05.076 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:32:05.076 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:32:05.076 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:05.076 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:05.076 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:05.076 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:05.076 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:05.077 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:05.077 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:32:05.077 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:32:05.077 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:05.077 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:05.077 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:05.077 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:32:05.077 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:05.077 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:05.077 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:32:05.077 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:05.077 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:05.077 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:05.077 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:32:05.078 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:05.078 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:05.078 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:32:05.078 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:05.078 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:05.078 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:05.078 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:32:05.078 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:05.078 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:05.078 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:05.078 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:32:05.078 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:32:05.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.064+0000 7f6f22515640 1 -- 172.21.15.37:0/500612940 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6f08035e80 msgr2=0x7f6f08038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:05.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.064+0000 7f6f22515640 1 --2- 172.21.15.37:0/500612940 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6f08035e80 0x7f6f08038340 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6f140077c0 tx=0x7f6f14009820 comp rx=0 tx=0).stop 2024-04-26T00:32:05.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.064+0000 7f6f22515640 10 monclient: shutdown 2024-04-26T00:32:05.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.064+0000 7f6f22515640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:05.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.064+0000 7f6f22515640 1 -- 172.21.15.37:0/500612940 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6f1c133ee0 msgr2=0x7f6f1c13b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:05.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.064+0000 7f6f20ab2640 1 -- 172.21.15.37:0/500612940 reap_dead start 2024-04-26T00:32:05.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.064+0000 7f6f22515640 1 --2- 172.21.15.37:0/500612940 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6f1c133ee0 0x7f6f1c13b200 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6f100ae040 tx=0x7f6f10019110 comp rx=0 tx=0).stop 2024-04-26T00:32:05.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.064+0000 7f6f22515640 1 -- 172.21.15.37:0/500612940 shutdown_connections 2024-04-26T00:32:05.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.064+0000 7f6f22515640 1 --2- 172.21.15.37:0/500612940 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6f1c133ee0 0x7f6f1c13b200 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:05.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.064+0000 7f6f22515640 1 -- 172.21.15.37:0/500612940 >> 172.21.15.37:0/500612940 conn(0x7f6f1c0a5f00 msgr2=0x7f6f1c09efd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:05.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.064+0000 7f6f22515640 1 -- 172.21.15.37:0/500612940 shutdown_connections 2024-04-26T00:32:05.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.064+0000 7f6f22515640 1 -- 172.21.15.37:0/500612940 wait complete. 2024-04-26T00:32:05.081 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:32:05.081 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-26T00:32:05.081 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-26T00:32:05.139 INFO:tasks.rados.rados.0.smithi045.stdout:2048: done (4 left) 2024-04-26T00:32:05.139 INFO:tasks.rados.rados.0.smithi045.stdout:2049: done (3 left) 2024-04-26T00:32:05.139 INFO:tasks.rados.rados.0.smithi045.stdout:2050: done (2 left) 2024-04-26T00:32:05.139 INFO:tasks.rados.rados.0.smithi045.stdout:2051: done (1 left) 2024-04-26T00:32:05.139 INFO:tasks.rados.rados.0.smithi045.stdout:2052: done (0 left) 2024-04-26T00:32:05.139 INFO:tasks.rados.rados.0.smithi045.stdout:2053: write (excl) oid 27 current snap is 232 2024-04-26T00:32:05.139 INFO:tasks.rados.rados.0.smithi045.stdout:2053: seq_num 750 ranges {605250=639344,1855142=452579} 2024-04-26T00:32:05.153 INFO:tasks.rados.rados.0.smithi045.stdout:2053: writing smithi04526579-27 from 605250 to 1244594 tid 1 2024-04-26T00:32:05.166 INFO:tasks.rados.rados.0.smithi045.stdout:2053: writing smithi04526579-27 from 1855142 to 2307721 tid 2 2024-04-26T00:32:05.166 INFO:tasks.rados.rados.0.smithi045.stdout:2054: rollback oid 24 current snap is 232 2024-04-26T00:32:05.166 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 24 to 230 2024-04-26T00:32:05.166 INFO:tasks.rados.rados.0.smithi045.stdout:2053: finishing write tid 1 to smithi04526579-27 2024-04-26T00:32:05.166 INFO:tasks.rados.rados.0.smithi045.stdout:2055: delete oid 14 current snap is 232 2024-04-26T00:32:05.173 INFO:tasks.rados.rados.0.smithi045.stdout:2055: done (2 left) 2024-04-26T00:32:05.173 INFO:tasks.rados.rados.0.smithi045.stdout:2056: write oid 44 current snap is 232 2024-04-26T00:32:05.173 INFO:tasks.rados.rados.0.smithi045.stdout:2056: seq_num 752 ranges {638864=789842,1972106=456248} 2024-04-26T00:32:05.189 INFO:tasks.rados.rados.0.smithi045.stdout:2056: writing smithi04526579-44 from 638864 to 1428706 tid 1 2024-04-26T00:32:05.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.204+0000 7f64ea460640 10 monclient: get_monmap_and_config 2024-04-26T00:32:05.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.204+0000 7f64ea460640 10 monclient: build_initial_monmap 2024-04-26T00:32:05.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.204+0000 7f64ea460640 10 monclient: monmap: 2024-04-26T00:32:05.199 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:05.199 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:05.199 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:32:05.209618+0000 2024-04-26T00:32:05.199 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:32:05.209618+0000 2024-04-26T00:32:05.199 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:05.199 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:05.199 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:32:05.199 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:32:05.199 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:32:05.200 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:32:05.200 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:32:05.200 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:05.202 INFO:tasks.rados.rados.0.smithi045.stdout:2056: writing smithi04526579-44 from 1972106 to 2428354 tid 2 2024-04-26T00:32:05.202 INFO:tasks.rados.rados.0.smithi045.stdout:2057: copy_from oid 25 from oid 23 current snap is 232 2024-04-26T00:32:05.202 INFO:tasks.rados.rados.0.smithi045.stdout:2053: finishing write tid 2 to smithi04526579-27 2024-04-26T00:32:05.202 INFO:tasks.rados.rados.0.smithi045.stdout:2058: read oid 16 snap -1 2024-04-26T00:32:05.202 INFO:tasks.rados.rados.0.smithi045.stdout:2058: expect (ObjNum 515 snap 157 seq_num 515) 2024-04-26T00:32:05.202 INFO:tasks.rados.rados.0.smithi045.stdout:2059: read oid 18 snap -1 2024-04-26T00:32:05.202 INFO:tasks.rados.rados.0.smithi045.stdout:2059: expect (ObjNum 551 snap 168 seq_num 551) 2024-04-26T00:32:05.202 INFO:tasks.rados.rados.0.smithi045.stdout:2060: delete oid 14 current snap is 232 2024-04-26T00:32:05.203 INFO:tasks.rados.rados.0.smithi045.stdout:2053: finishing write tid 3 to smithi04526579-27 2024-04-26T00:32:05.203 INFO:tasks.rados.rados.0.smithi045.stdout:2054: finishing rollback tid 0 to smithi04526579-24 2024-04-26T00:32:05.203 INFO:tasks.rados.rados.0.smithi045.stdout:2053: finishing write tid 4 to smithi04526579-27 2024-04-26T00:32:05.203 INFO:tasks.rados.rados.0.smithi045.stdout:2053: oid 27 updating version 0 to 1344 2024-04-26T00:32:05.203 INFO:tasks.rados.rados.0.smithi045.stdout:2053: oid 27 version 1344 is already newer than 1343 2024-04-26T00:32:05.203 INFO:tasks.rados.rados.0.smithi045.stdout:2053: oid 27 version 1344 is already newer than 1342 2024-04-26T00:32:05.203 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 1344 (ObjNum 750 snap 232 seq_num 750) dirty exists 2024-04-26T00:32:05.203 INFO:tasks.rados.rados.0.smithi045.stdout:2053: left oid 27 (ObjNum 750 snap 232 seq_num 750) 2024-04-26T00:32:05.203 INFO:tasks.rados.rados.0.smithi045.stdout:2054: finishing rollback tid 1 to smithi04526579-24 2024-04-26T00:32:05.203 INFO:tasks.rados.rados.0.smithi045.stdout:2054: finishing rollback tid 2 to smithi04526579-24 2024-04-26T00:32:05.203 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 1729 (ObjNum 735 snap 227 seq_num 735) dirty exists 2024-04-26T00:32:05.203 INFO:tasks.rados.rados.0.smithi045.stdout:2056: finishing write tid 1 to smithi04526579-44 2024-04-26T00:32:05.208 INFO:tasks.rados.rados.0.smithi045.stdout:2056: finishing write tid 2 to smithi04526579-44 2024-04-26T00:32:05.209 INFO:tasks.rados.rados.0.smithi045.stdout:2056: finishing write tid 3 to smithi04526579-44 2024-04-26T00:32:05.209 INFO:tasks.rados.rados.0.smithi045.stdout:2056: finishing write tid 4 to smithi04526579-44 2024-04-26T00:32:05.209 INFO:tasks.rados.rados.0.smithi045.stdout:2056: oid 44 updating version 0 to 1093 2024-04-26T00:32:05.209 INFO:tasks.rados.rados.0.smithi045.stdout:2056: oid 44 updating version 1093 to 1094 2024-04-26T00:32:05.209 INFO:tasks.rados.rados.0.smithi045.stdout:2056: oid 44 updating version 1094 to 1095 2024-04-26T00:32:05.209 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 1095 (ObjNum 752 snap 232 seq_num 752) dirty exists 2024-04-26T00:32:05.209 INFO:tasks.rados.rados.0.smithi045.stdout:2056: left oid 44 (ObjNum 752 snap 232 seq_num 752) 2024-04-26T00:32:05.214 INFO:tasks.rados.rados.0.smithi045.stdout:2058: expect (ObjNum 515 snap 157 seq_num 515) 2024-04-26T00:32:05.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.204+0000 7f64ea460640 1 Processor -- start 2024-04-26T00:32:05.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.204+0000 7f64ea460640 1 -- start start 2024-04-26T00:32:05.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.204+0000 7f64ea460640 10 monclient: init 2024-04-26T00:32:05.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.204+0000 7f64ea460640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:05.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.204+0000 7f64ea460640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-04-26T00:32:05.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.204+0000 7f64ea460640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f64e40a2c10 0x7f64e41350f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:05.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.204+0000 7f64ea460640 10 monclient(hunting): picked mon.noname-b con 0x7f64e40a2c10 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:05.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.204+0000 7f64ea460640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f64e409ddd0 0x7f64e4135630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:05.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.204+0000 7f64ea460640 10 monclient(hunting): picked mon.noname-c con 0x7f64e409ddd0 addr v2:172.21.15.45:3300/0 2024-04-26T00:32:05.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.204+0000 7f64ea460640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f64e409e700 addr v1:172.21.15.45:6789/0 2024-04-26T00:32:05.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.204+0000 7f64ea460640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64e40a1670 con 0x7f64e409e700 2024-04-26T00:32:05.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.204+0000 7f64ea460640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:05.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.204+0000 7f64ea460640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f64e40a3640 con 0x7f64e409ddd0 2024-04-26T00:32:05.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.204+0000 7f64ea460640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:05.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.204+0000 7f64ea460640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f64e40a2ff0 con 0x7f64e40a2c10 2024-04-26T00:32:05.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.204+0000 7f64ea460640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:05.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.204+0000 7f64ea460640 10 monclient(hunting): authenticate will time out at 1706.515311s 2024-04-26T00:32:05.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.204+0000 7f64e3fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f64e40a2c10 0x7f64e41350f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:05.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.204+0000 7f64e3fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f64e40a2c10 0x7f64e41350f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:56392/0 (socket says 172.21.15.37:56392) 2024-04-26T00:32:05.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.204+0000 7f64e3fff640 1 -- 172.21.15.37:0/2873064662 learned_addr learned my addr 172.21.15.37:0/2873064662 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:32:05.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.204+0000 7f64e3fff640 10 monclient(hunting): get_auth_request con 0x7f64e40a2c10 auth_method 0 2024-04-26T00:32:05.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.204+0000 7f64e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:05.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.204+0000 7f64e3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:05.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.204+0000 7f64e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:05.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.204+0000 7f64e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:05.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.204+0000 7f64e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:05.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.204+0000 7f64e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:05.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.208+0000 7f64e3fff640 10 monclient(hunting): handle_auth_done global_id 84662 payload 274 2024-04-26T00:32:05.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.208+0000 7f64e3fff640 1 -- 172.21.15.37:0/2873064662 >> v2:172.21.15.45:3300/0 conn(0x7f64e409ddd0 msgr2=0x7f64e4135630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:05.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.208+0000 7f64e3fff640 1 --2- 172.21.15.37:0/2873064662 >> v2:172.21.15.45:3300/0 conn(0x7f64e409ddd0 0x7f64e4135630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:05.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.208+0000 7f64e3fff640 1 -- 172.21.15.37:0/2873064662 >> v1:172.21.15.45:6789/0 conn(0x7f64e409e700 legacy=0x7f64e409eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:05.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.208+0000 7f64e3fff640 10 monclient: _finish_hunting 0 2024-04-26T00:32:05.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.208+0000 7f64e3fff640 1 monclient: found mon.noname-b 2024-04-26T00:32:05.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.208+0000 7f64e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:05.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.208+0000 7f64e3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:32:05.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.208+0000 7f64e3fff640 1 -- 172.21.15.37:0/2873064662 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64e40a3880 con 0x7f64e40a2c10 2024-04-26T00:32:05.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.212+0000 7f64e3fff640 1 -- 172.21.15.37:0/2873064662 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f64e40a2c10 msgr2=0x7f64e41350f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:05.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.212+0000 7f64e3fff640 1 --2- 172.21.15.37:0/2873064662 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f64e40a2c10 0x7f64e41350f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f64d4009870 tx=0x7f64d401c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ac5a7f7e216d2cd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:05.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.212+0000 7f64e2ffd640 1 -- 172.21.15.37:0/2873064662 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64d40b0070 con 0x7f64e40a2c10 2024-04-26T00:32:05.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.212+0000 7f64e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:05.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.212+0000 7f64e2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:32:05.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.212+0000 7f64e2ffd640 10 monclient: dump: 2024-04-26T00:32:05.218 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:05.218 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:05.218 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:05.218 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:05.218 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:05.218 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:05.218 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:05.218 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:05.218 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:05.218 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:05.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.212+0000 7f64e2ffd640 10 monclient: _finish_auth 0 2024-04-26T00:32:05.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.212+0000 7f64e2ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:32:05.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.212+0000 7f64e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:05.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.212+0000 7f64e2ffd640 1 -- 172.21.15.37:0/2873064662 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64d40ab600 con 0x7f64e40a2c10 2024-04-26T00:32:05.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.212+0000 7f64e2ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:05.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.212+0000 7f64e2ffd640 1 -- 172.21.15.37:0/2873064662 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64d40ab9d0 con 0x7f64e40a2c10 2024-04-26T00:32:05.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.212+0000 7f64e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:05.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.212+0000 7f64e2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:32:05.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.212+0000 7f64e2ffd640 10 monclient: dump: 2024-04-26T00:32:05.219 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:05.219 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:05.219 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:05.219 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:05.219 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:05.219 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:05.219 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:05.219 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:05.219 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:05.219 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:05.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.212+0000 7f64ea460640 5 monclient: authenticate success, global_id 84662 2024-04-26T00:32:05.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.212+0000 7f64ea460640 10 monclient: get_monmap_and_config success 2024-04-26T00:32:05.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.212+0000 7f64ea460640 4 set_mon_vals no callback set 2024-04-26T00:32:05.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.212+0000 7f64ea460640 10 monclient: shutdown 2024-04-26T00:32:05.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.212+0000 7f64ea460640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:05.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.212+0000 7f64ea460640 1 -- 172.21.15.37:0/2873064662 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f64e40a2c10 msgr2=0x7f64e41350f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:05.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.212+0000 7f64ea460640 1 --2- 172.21.15.37:0/2873064662 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f64e40a2c10 0x7f64e41350f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f64d4009870 tx=0x7f64d401c3e0 comp rx=0 tx=0).stop 2024-04-26T00:32:05.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.212+0000 7f64ea460640 1 -- 172.21.15.37:0/2873064662 shutdown_connections 2024-04-26T00:32:05.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.212+0000 7f64ea460640 1 --2- 172.21.15.37:0/2873064662 >> v2:172.21.15.45:3300/0 conn(0x7f64e409ddd0 0x7f64e4135630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:05.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.212+0000 7f64ea460640 1 --2- 172.21.15.37:0/2873064662 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f64e40a2c10 0x7f64e41350f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:05.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.212+0000 7f64ea460640 1 -- 172.21.15.37:0/2873064662 >> 172.21.15.37:0/2873064662 conn(0x7f64e40a7c40 msgr2=0x7f64e40a3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:05.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.212+0000 7f64ea460640 1 -- 172.21.15.37:0/2873064662 shutdown_connections 2024-04-26T00:32:05.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.212+0000 7f64ea460640 1 -- 172.21.15.37:0/2873064662 wait complete. 2024-04-26T00:32:05.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.212+0000 7f64ea460640 10 monclient: build_initial_monmap 2024-04-26T00:32:05.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.212+0000 7f64ea460640 10 monclient: monmap: 2024-04-26T00:32:05.220 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:05.221 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:32:05.221 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:32:05.221 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:32:05.221 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:05.221 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:32:05.221 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:32:05.221 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:32:05.221 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:32:05.221 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:05.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.212+0000 7f64ea460640 1 Processor -- start 2024-04-26T00:32:05.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.212+0000 7f64ea460640 1 -- start start 2024-04-26T00:32:05.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.212+0000 7f64ea460640 10 monclient: init 2024-04-26T00:32:05.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64ea460640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:05.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64ea460640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:32:05.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64ea460640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64e409ddd0 0x7f64e41335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:05.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64ea460640 10 monclient(hunting): picked mon.noname-b con 0x7f64e409ddd0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:32:05.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64ea460640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f64e409e700 0x7f64e413c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:05.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64ea460640 10 monclient(hunting): picked mon.noname-c con 0x7f64e409e700 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:05.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64ea460640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64e40a2c10 0x7f64e406c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:05.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64ea460640 10 monclient(hunting): picked mon.noname-a con 0x7f64e40a2c10 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:32:05.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64ea460640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:05.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64ea460640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f64e413fb30 con 0x7f64e40a2c10 2024-04-26T00:32:05.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64ea460640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:05.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64ea460640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f64e413f9b0 con 0x7f64e409ddd0 2024-04-26T00:32:05.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64ea460640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:05.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64ea460640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f64e413fcb0 con 0x7f64e409e700 2024-04-26T00:32:05.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64ea460640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:05.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64ea460640 10 monclient(hunting): authenticate will time out at 1706.524212s 2024-04-26T00:32:05.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e37fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f64e409e700 0x7f64e413c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:05.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e89fd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64e40a2c10 0x7f64e406c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:05.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e37fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f64e409e700 0x7f64e413c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:56400/0 (socket says 172.21.15.37:56400) 2024-04-26T00:32:05.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e37fe640 1 -- 172.21.15.37:0/657163698 learned_addr learned my addr 172.21.15.37:0/657163698 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:32:05.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e37fe640 10 monclient(hunting): get_auth_request con 0x7f64e409e700 auth_method 0 2024-04-26T00:32:05.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:05.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:05.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:05.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e89fd640 10 monclient(hunting): get_auth_request con 0x7f64e40a2c10 auth_method 0 2024-04-26T00:32:05.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e89fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:05.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e89fd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:05.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e89fd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:05.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:05.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:05.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:05.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e89fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:05.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e89fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:05.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e89fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:05.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e89fd640 10 monclient(hunting): handle_auth_done global_id 144117 payload 824 2024-04-26T00:32:05.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e89fd640 1 -- 172.21.15.37:0/657163698 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f64e409e700 msgr2=0x7f64e413c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:05.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e89fd640 1 --2- 172.21.15.37:0/657163698 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f64e409e700 0x7f64e413c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:05.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e89fd640 1 -- 172.21.15.37:0/657163698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64e409ddd0 msgr2=0x7f64e41335a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:05.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e89fd640 1 --2- 172.21.15.37:0/657163698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64e409ddd0 0x7f64e41335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:05.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e89fd640 10 monclient: _finish_hunting 0 2024-04-26T00:32:05.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e89fd640 1 monclient: found mon.noname-a 2024-04-26T00:32:05.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e89fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:05.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e89fd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:32:05.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e89fd640 1 -- 172.21.15.37:0/657163698 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64e4138c60 con 0x7f64e40a2c10 2024-04-26T00:32:05.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e37fe640 1 --2- 172.21.15.37:0/657163698 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f64e409e700 0x7f64e413c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:32:05.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e89fd640 1 -- 172.21.15.37:0/657163698 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64e40a2c10 msgr2=0x7f64e406c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:05.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e89fd640 1 --2- 172.21.15.37:0/657163698 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64e40a2c10 0x7f64e406c310 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f64dc00c7b0 tx=0x7f64dc00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:05.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e0ff9640 1 -- 172.21.15.37:0/657163698 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64dc016020 con 0x7f64e40a2c10 2024-04-26T00:32:05.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:05.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:32:05.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e0ff9640 10 monclient: dump: 2024-04-26T00:32:05.225 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:05.225 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:05.225 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:05.226 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:05.226 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:05.226 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:05.226 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:05.226 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:05.226 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:05.226 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:05.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e0ff9640 10 monclient: _finish_auth 0 2024-04-26T00:32:05.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e0ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:32:05.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:05.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e0ff9640 1 -- 172.21.15.37:0/657163698 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64dc00ac00 con 0x7f64e40a2c10 2024-04-26T00:32:05.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64ea460640 5 monclient: authenticate success, global_id 144117 2024-04-26T00:32:05.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64ea460640 10 monclient: _renew_subs 2024-04-26T00:32:05.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64ea460640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:05.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64ea460640 1 -- 172.21.15.37:0/657163698 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64e413a7f0 con 0x7f64e40a2c10 2024-04-26T00:32:05.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e0ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:05.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e0ff9640 1 -- 172.21.15.37:0/657163698 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64dc00ce60 con 0x7f64e40a2c10 2024-04-26T00:32:05.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:05.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:05.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e0ff9640 10 monclient: dump: 2024-04-26T00:32:05.227 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:05.227 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:05.227 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:05.227 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:05.227 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:05.227 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:05.227 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:05.227 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:05.227 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:05.228 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:05.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64ea460640 10 monclient: _renew_subs 2024-04-26T00:32:05.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64ea460640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:05.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64ea460640 1 -- 172.21.15.37:0/657163698 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64e406dfb0 con 0x7f64e40a2c10 2024-04-26T00:32:05.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.216+0000 7f64e27fc640 4 set_mon_vals no callback set 2024-04-26T00:32:05.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.220+0000 7f64e0ff9640 1 -- 172.21.15.37:0/657163698 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f64dc019020 con 0x7f64e40a2c10 2024-04-26T00:32:05.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.220+0000 7f64ea460640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:05.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.220+0000 7f64ea460640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:05.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.220+0000 7f64ea460640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:05.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.220+0000 7f64ea460640 1 -- 172.21.15.37:0/657163698 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64e4066f40 con 0x7f64e40a2c10 2024-04-26T00:32:05.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.220+0000 7f64e0ff9640 1 --2- 172.21.15.37:0/657163698 >> v2:172.21.15.45:6812/2379918623 conn(0x7f64d0035e80 0x7f64d0038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:05.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.220+0000 7f64e0ff9640 1 -- 172.21.15.37:0/657163698 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(468..468 src has 370..468) ==== 3927+0+0 (secure 0 0 0) 0x7f64dc0486a0 con 0x7f64e40a2c10 2024-04-26T00:32:05.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.220+0000 7f64e3fff640 1 --2- 172.21.15.37:0/657163698 >> v2:172.21.15.45:6812/2379918623 conn(0x7f64d0035e80 0x7f64d0038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:05.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.220+0000 7f64e3fff640 10 monclient: get_auth_request con 0x7f64d0035e80 auth_method 0 2024-04-26T00:32:05.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.224+0000 7f64e3fff640 1 --2- 172.21.15.37:0/657163698 >> v2:172.21.15.45:6812/2379918623 conn(0x7f64d0035e80 0x7f64d0038340 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f64d40ab000 tx=0x7f64d4004740 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:05.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.224+0000 7f64e0ff9640 1 -- 172.21.15.37:0/657163698 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f64dc020070 con 0x7f64e40a2c10 2024-04-26T00:32:05.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.224+0000 7f64e0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:05.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.224+0000 7f64e0ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:32:05.425 INFO:tasks.rados.rados.0.smithi045.stdout:2057: finishing copy_from to smithi04526579-25 2024-04-26T00:32:05.425 INFO:tasks.rados.rados.0.smithi045.stdout:2057: got expected ENOENT (src dne) 2024-04-26T00:32:05.425 INFO:tasks.rados.rados.0.smithi045.stdout:2057: finishing copy_from racing read to smithi04526579-25 2024-04-26T00:32:05.425 INFO:tasks.rados.rados.0.smithi045.stdout:2059: expect (ObjNum 551 snap 168 seq_num 551) 2024-04-26T00:32:05.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.448+0000 7f64ea460640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-26T00:32:05.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.448+0000 7f64ea460640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-26T00:32:05.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.448+0000 7f64ea460640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:05.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.448+0000 7f64ea460640 1 -- 172.21.15.37:0/657163698 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f64e41341a0 con 0x7f64e40a2c10 2024-04-26T00:32:05.446 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.452+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.446 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.452+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=23,config=2,health=12,kv=10,logm=53} crc {auth=2054375214,config=692619927,health=2510573225,kv=1034996044,logm=2981389111}) 2024-04-26T00:32:05.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.452+0000 7f64e0ff9640 1 -- 172.21.15.37:0/657163698 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f64dc0473f0 con 0x7f64e40a2c10 2024-04-26T00:32:05.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.452+0000 7f64e0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-26T00:32:05.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.452+0000 7f64e0ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:32:05.447 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.452+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=22,config=2,health=12,kv=10,logm=54} crc {auth=3856790330,config=692619927,health=2510573225,kv=1034996044,logm=945110091}) 2024-04-26T00:32:05.447 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.452+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.447 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.452+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=23,config=2,health=12,kv=10,logm=53} crc {auth=2054375214,config=692619927,health=2510573225,kv=1034996044,logm=2981389111}) 2024-04-26T00:32:05.447 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.452+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=21,config=2,health=12,kv=10,logm=55} crc {auth=159785907,config=692619927,health=2510573225,kv=1034996044,logm=4029318569}) 2024-04-26T00:32:05.448 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.452+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.448 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.452+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=941722053}) 2024-04-26T00:32:05.448 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.452+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2105256214}) 2024-04-26T00:32:05.448 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.452+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.448 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.452+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=941722053}) 2024-04-26T00:32:05.448 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.452+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3255707277}) 2024-04-26T00:32:05.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.452+0000 7f64ea460640 1 -- 172.21.15.37:0/657163698 >> v2:172.21.15.45:6812/2379918623 conn(0x7f64d0035e80 msgr2=0x7f64d0038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:05.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.452+0000 7f64ea460640 1 --2- 172.21.15.37:0/657163698 >> v2:172.21.15.45:6812/2379918623 conn(0x7f64d0035e80 0x7f64d0038340 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f64d40ab000 tx=0x7f64d4004740 comp rx=0 tx=0).stop 2024-04-26T00:32:05.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.452+0000 7f64ea460640 10 monclient: shutdown 2024-04-26T00:32:05.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.452+0000 7f64ea460640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:05.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.452+0000 7f64ea460640 1 -- 172.21.15.37:0/657163698 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64e40a2c10 msgr2=0x7f64e406c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:05.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.452+0000 7f64ea460640 1 --2- 172.21.15.37:0/657163698 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64e40a2c10 0x7f64e406c310 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f64dc00c7b0 tx=0x7f64dc00cc70 comp rx=0 tx=0).stop 2024-04-26T00:32:05.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.452+0000 7f64ea460640 1 -- 172.21.15.37:0/657163698 shutdown_connections 2024-04-26T00:32:05.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.452+0000 7f64ea460640 1 --2- 172.21.15.37:0/657163698 >> v2:172.21.15.45:6812/2379918623 conn(0x7f64d0035e80 0x7f64d0038340 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:05.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.452+0000 7f64ea460640 1 --2- 172.21.15.37:0/657163698 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64e40a2c10 0x7f64e406c310 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:05.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.452+0000 7f64ea460640 1 --2- 172.21.15.37:0/657163698 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f64e409e700 0x7f64e413c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:05.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.452+0000 7f64ea460640 1 --2- 172.21.15.37:0/657163698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f64e409ddd0 0x7f64e41335a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:05.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.452+0000 7f64ea460640 1 -- 172.21.15.37:0/657163698 >> 172.21.15.37:0/657163698 conn(0x7f64e40a7c40 msgr2=0x7f64e406fcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:05.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.456+0000 7f64ea460640 1 -- 172.21.15.37:0/657163698 shutdown_connections 2024-04-26T00:32:05.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:05.456+0000 7f64ea460640 1 -- 172.21.15.37:0/657163698 wait complete. 2024-04-26T00:32:05.451 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.456+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.451 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.456+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3922331935}) 2024-04-26T00:32:05.451 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.456+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=430064049}) 2024-04-26T00:32:05.451 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.456+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.451 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.456+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3922331935}) 2024-04-26T00:32:05.451 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.456+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2591892363}) 2024-04-26T00:32:05.452 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.456+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.452 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.456+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2798434731}) 2024-04-26T00:32:05.452 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.456+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=785508718}) 2024-04-26T00:32:05.453 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.456+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.453 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.456+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2798434731}) 2024-04-26T00:32:05.453 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.456+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3240634775}) 2024-04-26T00:32:05.454 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.460+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.454 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.460+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2426421943}) 2024-04-26T00:32:05.455 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.460+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1021635578}) 2024-04-26T00:32:05.455 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.460+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.455 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.460+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2426421943}) 2024-04-26T00:32:05.455 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.460+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3735743525}) 2024-04-26T00:32:05.456 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.460+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.456 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.460+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3431592471}) 2024-04-26T00:32:05.456 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.460+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=334255972}) 2024-04-26T00:32:05.456 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.460+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.456 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.460+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3431592471}) 2024-04-26T00:32:05.456 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.460+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4096638067}) 2024-04-26T00:32:05.458 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.464+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.458 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.464+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2561861995}) 2024-04-26T00:32:05.458 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.464+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=621766749}) 2024-04-26T00:32:05.458 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.464+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.458 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.464+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2561861995}) 2024-04-26T00:32:05.458 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.464+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3189634836}) 2024-04-26T00:32:05.459 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.464+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.459 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.464+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4214213213}) 2024-04-26T00:32:05.459 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.464+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1035853317}) 2024-04-26T00:32:05.460 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.464+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.460 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.464+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4214213213}) 2024-04-26T00:32:05.460 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.464+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4164867382}) 2024-04-26T00:32:05.461 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.464+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.461 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.464+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3051352245}) 2024-04-26T00:32:05.461 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.468+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=453558534}) 2024-04-26T00:32:05.461 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.468+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.461 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.468+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3051352245}) 2024-04-26T00:32:05.461 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.468+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1890760691}) 2024-04-26T00:32:05.463 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.468+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.463 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.468+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4255887760}) 2024-04-26T00:32:05.463 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.468+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1812020867}) 2024-04-26T00:32:05.463 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.468+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.463 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.468+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4255887760}) 2024-04-26T00:32:05.463 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.468+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=890933361}) 2024-04-26T00:32:05.464 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.468+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.465 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.468+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3995204915}) 2024-04-26T00:32:05.465 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.468+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3620054300}) 2024-04-26T00:32:05.465 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.468+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.465 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.468+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3995204915}) 2024-04-26T00:32:05.465 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.468+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3937853428}) 2024-04-26T00:32:05.466 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.472+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.466 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.472+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2120517193}) 2024-04-26T00:32:05.467 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.472+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1296174283}) 2024-04-26T00:32:05.467 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.472+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.467 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.472+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2120517193}) 2024-04-26T00:32:05.467 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.472+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2482594480}) 2024-04-26T00:32:05.468 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.472+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.468 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.472+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1866417203}) 2024-04-26T00:32:05.468 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.472+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1117145458}) 2024-04-26T00:32:05.468 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.472+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.469 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.472+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1866417203}) 2024-04-26T00:32:05.469 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.472+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1515440921}) 2024-04-26T00:32:05.470 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-26T00:32:05.471 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.476+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.471 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.476+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3848725118}) 2024-04-26T00:32:05.471 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.476+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4094878756}) 2024-04-26T00:32:05.471 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.476+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.471 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.476+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3848725118}) 2024-04-26T00:32:05.471 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.476+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=614587241}) 2024-04-26T00:32:05.472 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.476+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.472 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.476+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3747020782}) 2024-04-26T00:32:05.472 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.476+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1252098522}) 2024-04-26T00:32:05.472 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.476+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.472 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.476+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3747020782}) 2024-04-26T00:32:05.473 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.476+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=568801381}) 2024-04-26T00:32:05.484 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.488+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.484 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.488+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2241742537}) 2024-04-26T00:32:05.484 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.488+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1051525930}) 2024-04-26T00:32:05.484 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.488+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.484 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.488+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2241742537}) 2024-04-26T00:32:05.484 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.488+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1357921466}) 2024-04-26T00:32:05.566 INFO:tasks.rados.rados.0.smithi045.stdout:2053: done (6 left) 2024-04-26T00:32:05.566 INFO:tasks.rados.rados.0.smithi045.stdout:2054: done (5 left) 2024-04-26T00:32:05.566 INFO:tasks.rados.rados.0.smithi045.stdout:2056: done (4 left) 2024-04-26T00:32:05.566 INFO:tasks.rados.rados.0.smithi045.stdout:2057: done (3 left) 2024-04-26T00:32:05.566 INFO:tasks.rados.rados.0.smithi045.stdout:2058: done (2 left) 2024-04-26T00:32:05.566 INFO:tasks.rados.rados.0.smithi045.stdout:2059: done (1 left) 2024-04-26T00:32:05.567 INFO:tasks.rados.rados.0.smithi045.stdout:2060: done (0 left) 2024-04-26T00:32:05.567 INFO:tasks.rados.rados.0.smithi045.stdout:2061: write (excl) oid 23 current snap is 232 2024-04-26T00:32:05.567 INFO:tasks.rados.rados.0.smithi045.stdout:2061: seq_num 754 ranges {672478=656693,2089070=459917} 2024-04-26T00:32:05.587 INFO:tasks.rados.rados.0.smithi045.stdout:2061: writing smithi04526579-23 from 672478 to 1329171 tid 1 2024-04-26T00:32:05.608 INFO:tasks.rados.rados.0.smithi045.stdout:2061: writing smithi04526579-23 from 2089070 to 2548987 tid 2 2024-04-26T00:32:05.608 INFO:tasks.rados.rados.0.smithi045.stdout:2062: delete oid 21 current snap is 232 2024-04-26T00:32:05.608 INFO:tasks.rados.rados.0.smithi045.stdout:2061: finishing write tid 1 to smithi04526579-23 2024-04-26T00:32:05.615 INFO:tasks.rados.rados.0.smithi045.stdout:2061: finishing write tid 2 to smithi04526579-23 2024-04-26T00:32:05.616 INFO:tasks.rados.rados.0.smithi045.stdout:2061: finishing write tid 3 to smithi04526579-23 2024-04-26T00:32:05.616 INFO:tasks.rados.rados.0.smithi045.stdout:2061: finishing write tid 4 to smithi04526579-23 2024-04-26T00:32:05.616 INFO:tasks.rados.rados.0.smithi045.stdout:2061: oid 23 updating version 0 to 1443 2024-04-26T00:32:05.617 INFO:tasks.rados.rados.0.smithi045.stdout:2061: oid 23 updating version 1443 to 1445 2024-04-26T00:32:05.617 INFO:tasks.rados.rados.0.smithi045.stdout:2061: oid 23 version 1445 is already newer than 1444 2024-04-26T00:32:05.617 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 1445 (ObjNum 754 snap 232 seq_num 754) dirty exists 2024-04-26T00:32:05.617 INFO:tasks.rados.rados.0.smithi045.stdout:2061: left oid 23 (ObjNum 754 snap 232 seq_num 754) 2024-04-26T00:32:05.618 INFO:tasks.rados.rados.0.smithi045.stdout:2061: done (1 left) 2024-04-26T00:32:05.618 INFO:tasks.rados.rados.0.smithi045.stdout:2062: done (0 left) 2024-04-26T00:32:05.618 INFO:tasks.rados.rados.0.smithi045.stdout:2063: write (excl) oid 21 current snap is 232 2024-04-26T00:32:05.618 INFO:tasks.rados.rados.0.smithi045.stdout:2063: seq_num 756 ranges {706092=407191,1806034=463586} 2024-04-26T00:32:05.636 INFO:tasks.rados.rados.0.smithi045.stdout:2063: writing smithi04526579-21 from 706092 to 1113283 tid 1 2024-04-26T00:32:05.655 INFO:tasks.rados.rados.0.smithi045.stdout:2063: writing smithi04526579-21 from 1806034 to 2269620 tid 2 2024-04-26T00:32:05.655 INFO:tasks.rados.rados.0.smithi045.stdout:2064: read oid 25 snap -1 2024-04-26T00:32:05.655 INFO:tasks.rados.rados.0.smithi045.stdout:2064: expect (ObjNum 604 snap 186 seq_num 604) 2024-04-26T00:32:05.655 INFO:tasks.rados.rados.0.smithi045.stdout:2065: rollback oid 15 current snap is 232 2024-04-26T00:32:05.655 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 15 to 228 2024-04-26T00:32:05.655 INFO:tasks.rados.rados.0.smithi045.stdout:2063: finishing write tid 1 to smithi04526579-21 2024-04-26T00:32:05.655 INFO:tasks.rados.rados.0.smithi045.stdout:2066: rollback oid 19 current snap is 232 2024-04-26T00:32:05.655 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 19 to 153 2024-04-26T00:32:05.655 INFO:tasks.rados.rados.0.smithi045.stdout:2067: delete oid 40 current snap is 232 2024-04-26T00:32:05.666 INFO:tasks.rados.rados.0.smithi045.stdout:2067: done (4 left) 2024-04-26T00:32:05.666 INFO:tasks.rados.rados.0.smithi045.stdout:2068: write oid 17 current snap is 232 2024-04-26T00:32:05.666 INFO:tasks.rados.rados.0.smithi045.stdout:2068: seq_num 758 ranges {739706=557689,1806645=467255} 2024-04-26T00:32:05.675 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.680+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.675 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.680+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2570922585}) 2024-04-26T00:32:05.675 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.680+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2753628895}) 2024-04-26T00:32:05.675 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.680+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.675 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.680+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2570922585}) 2024-04-26T00:32:05.675 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.680+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3724055123}) 2024-04-26T00:32:05.683 INFO:tasks.rados.rados.0.smithi045.stdout:2068: writing smithi04526579-17 from 739706 to 1297395 tid 1 2024-04-26T00:32:05.688 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.692+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.689 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.692+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3684994040}) 2024-04-26T00:32:05.689 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.692+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1392714884}) 2024-04-26T00:32:05.689 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.692+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.689 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.692+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3684994040}) 2024-04-26T00:32:05.689 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.692+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2081207611}) 2024-04-26T00:32:05.695 INFO:tasks.rados.rados.0.smithi045.stdout:2068: writing smithi04526579-17 from 1806645 to 2273900 tid 2 2024-04-26T00:32:05.695 INFO:tasks.rados.rados.0.smithi045.stdout:2069: read oid 50 snap -1 2024-04-26T00:32:05.695 INFO:tasks.rados.rados.0.smithi045.stdout:2069: expect deleted 2024-04-26T00:32:05.695 INFO:tasks.rados.rados.0.smithi045.stdout:2070: copy_from oid 12 from oid 49 current snap is 232 2024-04-26T00:32:05.695 INFO:tasks.rados.rados.0.smithi045.stdout:2065: finishing rollback tid 0 to smithi04526579-15 2024-04-26T00:32:05.695 INFO:tasks.rados.rados.0.smithi045.stdout:2071: snap_remove snap 208 2024-04-26T00:32:05.696 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.700+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.696 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.700+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1867598230}) 2024-04-26T00:32:05.696 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.700+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1990645639}) 2024-04-26T00:32:05.696 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.700+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.696 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.700+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1867598230}) 2024-04-26T00:32:05.696 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.700+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1591690828}) 2024-04-26T00:32:05.744 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.748+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.744 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.748+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1434486314}) 2024-04-26T00:32:05.744 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.748+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3188759571}) 2024-04-26T00:32:05.744 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.748+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.744 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.748+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1434486314}) 2024-04-26T00:32:05.744 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.748+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=148919351}) 2024-04-26T00:32:05.747 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.752+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.747 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.752+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2322135383}) 2024-04-26T00:32:05.747 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.752+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4291075612}) 2024-04-26T00:32:05.747 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.752+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.747 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.752+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2322135383}) 2024-04-26T00:32:05.747 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.752+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1087952413}) 2024-04-26T00:32:05.750 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.756+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.750 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.756+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2485430433}) 2024-04-26T00:32:05.750 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.756+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2372748493}) 2024-04-26T00:32:05.750 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.756+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.750 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.756+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2485430433}) 2024-04-26T00:32:05.750 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.756+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2752575223}) 2024-04-26T00:32:05.752 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.756+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.752 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.756+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2793203510}) 2024-04-26T00:32:05.752 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.756+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=81648331}) 2024-04-26T00:32:05.752 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.756+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.752 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.756+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2793203510}) 2024-04-26T00:32:05.752 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.756+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2835500937}) 2024-04-26T00:32:05.755 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.760+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.755 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.760+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2783154662}) 2024-04-26T00:32:05.755 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.760+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=657243736}) 2024-04-26T00:32:05.755 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.760+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.755 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.760+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2783154662}) 2024-04-26T00:32:05.755 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.760+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1596524583}) 2024-04-26T00:32:05.758 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.764+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.758 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.764+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3150013886}) 2024-04-26T00:32:05.758 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.764+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1094845891}) 2024-04-26T00:32:05.758 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.764+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.758 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.764+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3150013886}) 2024-04-26T00:32:05.758 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.764+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2309003811}) 2024-04-26T00:32:05.760 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.764+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.760 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.764+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1715320309}) 2024-04-26T00:32:05.760 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.764+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2437805138}) 2024-04-26T00:32:05.761 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.764+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.761 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.764+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1715320309}) 2024-04-26T00:32:05.761 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.764+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=318990012}) 2024-04-26T00:32:05.763 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.768+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.763 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.768+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2074343327}) 2024-04-26T00:32:05.763 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.768+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=331388181}) 2024-04-26T00:32:05.763 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.768+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.763 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.768+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2074343327}) 2024-04-26T00:32:05.763 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.768+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2295600208}) 2024-04-26T00:32:05.765 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.768+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.765 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.768+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3790641722}) 2024-04-26T00:32:05.766 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.772+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1705691747}) 2024-04-26T00:32:05.766 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.772+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.766 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.772+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3790641722}) 2024-04-26T00:32:05.766 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.772+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1183791401}) 2024-04-26T00:32:05.768 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.772+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.768 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.772+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2543649665}) 2024-04-26T00:32:05.768 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.772+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=799969402}) 2024-04-26T00:32:05.768 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.772+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.768 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.772+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2543649665}) 2024-04-26T00:32:05.768 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.772+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3164741384}) 2024-04-26T00:32:05.771 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.776+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.771 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.776+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1160923184}) 2024-04-26T00:32:05.771 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.776+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3808421358}) 2024-04-26T00:32:05.771 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.776+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.771 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.776+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1160923184}) 2024-04-26T00:32:05.771 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.776+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2594768098}) 2024-04-26T00:32:05.773 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.776+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.773 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.776+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3036297922}) 2024-04-26T00:32:05.773 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.776+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1246822284}) 2024-04-26T00:32:05.773 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.776+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.773 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.780+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3036297922}) 2024-04-26T00:32:05.773 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.780+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3159128651}) 2024-04-26T00:32:05.776 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.780+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.776 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.780+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1159797893}) 2024-04-26T00:32:05.776 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.780+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2961605718}) 2024-04-26T00:32:05.776 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.780+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.776 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.780+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1159797893}) 2024-04-26T00:32:05.776 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.780+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=265824827}) 2024-04-26T00:32:05.778 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.784+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.778 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.784+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2484463884}) 2024-04-26T00:32:05.778 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.784+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4269241310}) 2024-04-26T00:32:05.778 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.784+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.779 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.784+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2484463884}) 2024-04-26T00:32:05.779 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.784+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2049695062}) 2024-04-26T00:32:05.781 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.784+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.781 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.784+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1935243552}) 2024-04-26T00:32:05.781 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.784+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=721898640}) 2024-04-26T00:32:05.781 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.784+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.781 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.788+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1935243552}) 2024-04-26T00:32:05.781 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.788+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1823810340}) 2024-04-26T00:32:05.784 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.788+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.784 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.788+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1308448113}) 2024-04-26T00:32:05.784 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.788+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1619944551}) 2024-04-26T00:32:05.784 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.788+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.784 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.788+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1308448113}) 2024-04-26T00:32:05.784 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.788+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2083150703}) 2024-04-26T00:32:05.786 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.792+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.787 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.792+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1250765549}) 2024-04-26T00:32:05.787 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.792+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2890456258}) 2024-04-26T00:32:05.787 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.792+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.787 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.792+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1250765549}) 2024-04-26T00:32:05.787 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.792+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=72639008}) 2024-04-26T00:32:05.788 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.792+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.788 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.792+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2803899001}) 2024-04-26T00:32:05.788 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.792+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1265523212}) 2024-04-26T00:32:05.789 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.792+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.789 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.792+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2803899001}) 2024-04-26T00:32:05.789 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.792+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=481090146}) 2024-04-26T00:32:05.791 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.796+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.792 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.796+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=68,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=7} crc {logm=1922549366,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=1914769218,mon_config_key=4233738560}) 2024-04-26T00:32:05.792 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.796+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=69,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=7} crc {logm=3825380360,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=3126025748,mon_config_key=3496296432}) 2024-04-26T00:32:05.792 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.796+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.792 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.796+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=68,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=7} crc {logm=1922549366,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=1914769218,mon_config_key=4233738560}) 2024-04-26T00:32:05.792 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.796+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=72,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5} crc {logm=2441669869,mdsmap=1401145748,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=4184528659,mon_config_key=3159346785}) 2024-04-26T00:32:05.794 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.800+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.794 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.800+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=2,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=85,osdmap=3} crc {mon_config_key=2908311886,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=1910179296,osd_snap=152030145,osdmap=2603168530}) 2024-04-26T00:32:05.795 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.800+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=2,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=84,osdmap=4} crc {mon_config_key=2908311886,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=1910179296,osd_snap=826505753,osdmap=831844678}) 2024-04-26T00:32:05.795 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.800+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.795 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.800+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=2,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=85,osdmap=3} crc {mon_config_key=2908311886,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=1910179296,osd_snap=152030145,osdmap=2603168530}) 2024-04-26T00:32:05.795 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.800+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=2,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=84,osdmap=4} crc {mon_config_key=2908311886,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=1910179296,osd_snap=1832274964,osdmap=831844678}) 2024-04-26T00:32:05.798 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.804+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.798 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.804+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=911460299}) 2024-04-26T00:32:05.798 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.804+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1550865796}) 2024-04-26T00:32:05.798 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.804+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.798 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.804+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=911460299}) 2024-04-26T00:32:05.798 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.804+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1483129274}) 2024-04-26T00:32:05.808 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.812+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.808 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.812+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=54} crc {osdmap=333154368}) 2024-04-26T00:32:05.808 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.812+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=53} crc {osdmap=590418996}) 2024-04-26T00:32:05.808 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.812+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:05.808 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.812+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=54} crc {osdmap=333154368}) 2024-04-26T00:32:05.808 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:32:05.812+0000 7fcf8af48640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=54} crc {osdmap=2190343222}) 2024-04-26T00:32:06.471 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:32:06.471 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:32:06.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.608+0000 7f4b6ec0e640 10 monclient: get_monmap_and_config 2024-04-26T00:32:06.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.608+0000 7f4b6ec0e640 10 monclient: build_initial_monmap 2024-04-26T00:32:06.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.608+0000 7f4b6ec0e640 10 monclient: monmap: 2024-04-26T00:32:06.603 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:06.603 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:06.604 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:32:06.613636+0000 2024-04-26T00:32:06.604 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:32:06.613636+0000 2024-04-26T00:32:06.604 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:06.604 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:06.604 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:32:06.604 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:32:06.604 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:32:06.604 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:32:06.604 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:32:06.604 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:06.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.608+0000 7f4b6ec0e640 1 Processor -- start 2024-04-26T00:32:06.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.608+0000 7f4b6ec0e640 1 -- start start 2024-04-26T00:32:06.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.608+0000 7f4b6ec0e640 10 monclient: init 2024-04-26T00:32:06.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.608+0000 7f4b6ec0e640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:06.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.608+0000 7f4b6ec0e640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-04-26T00:32:06.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.608+0000 7f4b6ec0e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4b68133b20 0x7f4b6809e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:06.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.608+0000 7f4b6ec0e640 10 monclient(hunting): picked mon.noname-b con 0x7f4b68133b20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:06.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.608+0000 7f4b6ec0e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4b68133f00 addr v1:172.21.15.37:6789/0 2024-04-26T00:32:06.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.608+0000 7f4b6ec0e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4b68139050 addr v1:172.21.15.45:6789/0 2024-04-26T00:32:06.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.608+0000 7f4b6ec0e640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b6809fe20 con 0x7f4b68133f00 2024-04-26T00:32:06.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.608+0000 7f4b6ec0e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b68072d00 con 0x7f4b68139050 2024-04-26T00:32:06.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.608+0000 7f4b6ec0e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:06.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.608+0000 7f4b6ec0e640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4b6809e620 con 0x7f4b68133b20 2024-04-26T00:32:06.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.608+0000 7f4b6ec0e640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:06.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.608+0000 7f4b6ec0e640 10 monclient(hunting): authenticate will time out at 1707.919337s 2024-04-26T00:32:06.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.608+0000 7f4b6c9aa640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4b68133b20 0x7f4b6809e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:06.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.608+0000 7f4b67fff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f4b68133f00 0x7f4b68134320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:48650/0 (socket says 172.21.15.37:48650) 2024-04-26T00:32:06.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.608+0000 7f4b67fff640 1 -- 172.21.15.37:0/40967821 learned_addr learned my addr 172.21.15.37:0/40967821 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:32:06.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.608+0000 7f4b6c9aa640 10 monclient(hunting): get_auth_request con 0x7f4b68133b20 auth_method 0 2024-04-26T00:32:06.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.608+0000 7f4b6c9aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:06.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.608+0000 7f4b6c9aa640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:06.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.608+0000 7f4b6c9aa640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:06.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.608+0000 7f4b6c9aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:06.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.608+0000 7f4b6c9aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:06.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.608+0000 7f4b6c9aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:06.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.608+0000 7f4b6c9aa640 10 monclient(hunting): handle_auth_done global_id 84668 payload 274 2024-04-26T00:32:06.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.608+0000 7f4b6c9aa640 1 -- 172.21.15.37:0/40967821 >> v1:172.21.15.45:6789/0 conn(0x7f4b68139050 legacy=0x7f4b6813b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:06.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.608+0000 7f4b6c9aa640 1 -- 172.21.15.37:0/40967821 >> v1:172.21.15.37:6789/0 conn(0x7f4b68133f00 legacy=0x7f4b68134320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:06.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.608+0000 7f4b6c9aa640 10 monclient: _finish_hunting 0 2024-04-26T00:32:06.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.608+0000 7f4b6c9aa640 1 monclient: found mon.noname-b 2024-04-26T00:32:06.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.608+0000 7f4b6c9aa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:06.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.608+0000 7f4b6c9aa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:32:06.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.608+0000 7f4b6c9aa640 1 -- 172.21.15.37:0/40967821 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b6809ba60 con 0x7f4b68133b20 2024-04-26T00:32:06.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6c9aa640 1 -- 172.21.15.37:0/40967821 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4b68133b20 msgr2=0x7f4b6809e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:06.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6c9aa640 1 --2- 172.21.15.37:0/40967821 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4b68133b20 0x7f4b6809e9d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4b60009dc0 tx=0x7f4b6001c5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f19a6fd8dde429b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:06.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b677fe640 1 -- 172.21.15.37:0/40967821 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b600aa700 con 0x7f4b68133b20 2024-04-26T00:32:06.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:06.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b677fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:32:06.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b677fe640 10 monclient: dump: 2024-04-26T00:32:06.610 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:06.610 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:06.610 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:06.610 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:06.610 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:06.610 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:06.610 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:06.610 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:06.611 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:06.611 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:06.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b677fe640 10 monclient: _finish_auth 0 2024-04-26T00:32:06.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b677fe640 10 monclient: _check_auth_tickets 2024-04-26T00:32:06.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:06.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b677fe640 1 -- 172.21.15.37:0/40967821 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b600af070 con 0x7f4b68133b20 2024-04-26T00:32:06.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6ec0e640 5 monclient: authenticate success, global_id 84668 2024-04-26T00:32:06.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b677fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:06.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b677fe640 1 -- 172.21.15.37:0/40967821 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b600aaac0 con 0x7f4b68133b20 2024-04-26T00:32:06.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6ec0e640 10 monclient: get_monmap_and_config success 2024-04-26T00:32:06.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6ec0e640 4 set_mon_vals no callback set 2024-04-26T00:32:06.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:06.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:32:06.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b677fe640 10 monclient: dump: 2024-04-26T00:32:06.611 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:06.612 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:06.612 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:06.612 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:06.612 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:06.612 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:06.612 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:06.612 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:06.612 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:06.612 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:06.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6ec0e640 10 monclient: shutdown 2024-04-26T00:32:06.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6ec0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:06.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6ec0e640 1 -- 172.21.15.37:0/40967821 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4b68133b20 msgr2=0x7f4b6809e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:06.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6ec0e640 1 --2- 172.21.15.37:0/40967821 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4b68133b20 0x7f4b6809e9d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4b60009dc0 tx=0x7f4b6001c5f0 comp rx=0 tx=0).stop 2024-04-26T00:32:06.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6ec0e640 1 -- 172.21.15.37:0/40967821 shutdown_connections 2024-04-26T00:32:06.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6ec0e640 1 --2- 172.21.15.37:0/40967821 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4b68133b20 0x7f4b6809e9d0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:06.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6ec0e640 1 -- 172.21.15.37:0/40967821 >> 172.21.15.37:0/40967821 conn(0x7f4b680a1ad0 msgr2=0x7f4b680a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:06.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6ec0e640 1 -- 172.21.15.37:0/40967821 shutdown_connections 2024-04-26T00:32:06.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6ec0e640 1 -- 172.21.15.37:0/40967821 wait complete. 2024-04-26T00:32:06.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6ec0e640 10 monclient: build_initial_monmap 2024-04-26T00:32:06.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6ec0e640 10 monclient: monmap: 2024-04-26T00:32:06.618 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:06.618 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:32:06.618 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:32:06.618 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:32:06.618 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:06.618 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:32:06.619 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:32:06.619 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:32:06.619 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:32:06.619 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:06.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6ec0e640 1 Processor -- start 2024-04-26T00:32:06.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6ec0e640 1 -- start start 2024-04-26T00:32:06.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6ec0e640 10 monclient: init 2024-04-26T00:32:06.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6ec0e640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:06.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6ec0e640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:32:06.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6ec0e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4b68133b20 0x7f4b68130c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:06.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6ec0e640 10 monclient(hunting): picked mon.noname-c con 0x7f4b68133b20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:06.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6ec0e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b68133f00 0x7f4b68131170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:06.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6ec0e640 10 monclient(hunting): picked mon.noname-b con 0x7f4b68133f00 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:32:06.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6ec0e640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4b68139050 0x7f4b681cc070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:06.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6ec0e640 10 monclient(hunting): picked mon.noname-a con 0x7f4b68139050 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:32:06.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6ec0e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:06.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6ec0e640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6813f990 con 0x7f4b68139050 2024-04-26T00:32:06.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6ec0e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:06.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6ec0e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6813f810 con 0x7f4b68133f00 2024-04-26T00:32:06.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6ec0e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:06.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6ec0e640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4b6813fb10 con 0x7f4b68133b20 2024-04-26T00:32:06.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6ec0e640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:06.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6ec0e640 10 monclient(hunting): authenticate will time out at 1707.927023s 2024-04-26T00:32:06.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6d1ab640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4b68139050 0x7f4b681cc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:06.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6c9aa640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4b68133b20 0x7f4b68130c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:06.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6d1ab640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4b68139050 0x7f4b681cc070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:35090/0 (socket says 172.21.15.37:35090) 2024-04-26T00:32:06.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6d1ab640 1 -- 172.21.15.37:0/3787010667 learned_addr learned my addr 172.21.15.37:0/3787010667 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:32:06.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6d1ab640 10 monclient(hunting): get_auth_request con 0x7f4b68139050 auth_method 0 2024-04-26T00:32:06.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6d1ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:06.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6d1ab640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:06.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6d1ab640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:06.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6c9aa640 10 monclient(hunting): get_auth_request con 0x7f4b68133b20 auth_method 0 2024-04-26T00:32:06.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6c9aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:06.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6c9aa640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:06.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6c9aa640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:06.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6d1ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:06.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6d1ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:06.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6d1ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:06.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6c9aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:06.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6c9aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:06.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b67fff640 1 --2- 172.21.15.37:0/3787010667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b68133f00 0x7f4b68131170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:06.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6c9aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:06.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b67fff640 10 monclient(hunting): get_auth_request con 0x7f4b68133f00 auth_method 0 2024-04-26T00:32:06.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:06.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b67fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:06.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b67fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:06.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6d1ab640 10 monclient(hunting): handle_auth_done global_id 144120 payload 824 2024-04-26T00:32:06.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6d1ab640 1 -- 172.21.15.37:0/3787010667 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4b68133b20 msgr2=0x7f4b68130c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:06.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6d1ab640 1 --2- 172.21.15.37:0/3787010667 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4b68133b20 0x7f4b68130c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:06.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6d1ab640 1 -- 172.21.15.37:0/3787010667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b68133f00 msgr2=0x7f4b68131170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:06.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6d1ab640 1 --2- 172.21.15.37:0/3787010667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b68133f00 0x7f4b68131170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:06.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6d1ab640 10 monclient: _finish_hunting 0 2024-04-26T00:32:06.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6d1ab640 1 monclient: found mon.noname-a 2024-04-26T00:32:06.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6d1ab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:06.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6d1ab640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:32:06.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6d1ab640 1 -- 172.21.15.37:0/3787010667 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b6806f9d0 con 0x7f4b68139050 2024-04-26T00:32:06.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.616+0000 7f4b6c9aa640 1 --2- 172.21.15.37:0/3787010667 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4b68133b20 0x7f4b68130c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:32:06.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.620+0000 7f4b6d1ab640 1 -- 172.21.15.37:0/3787010667 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4b68139050 msgr2=0x7f4b681cc070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:06.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.620+0000 7f4b6d1ab640 1 --2- 172.21.15.37:0/3787010667 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4b68139050 0x7f4b681cc070 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4b5800c3c0 tx=0x7f4b5800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:06.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.620+0000 7f4b657fa640 1 -- 172.21.15.37:0/3787010667 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b580119a0 con 0x7f4b68139050 2024-04-26T00:32:06.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.620+0000 7f4b657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:06.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.620+0000 7f4b657fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:32:06.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.620+0000 7f4b657fa640 10 monclient: dump: 2024-04-26T00:32:06.623 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:06.623 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:06.623 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:06.623 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:06.623 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:06.623 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:06.623 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:06.623 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:06.623 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:06.623 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:06.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.620+0000 7f4b657fa640 10 monclient: _finish_auth 0 2024-04-26T00:32:06.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.620+0000 7f4b657fa640 10 monclient: _check_auth_tickets 2024-04-26T00:32:06.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.620+0000 7f4b657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:06.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.620+0000 7f4b6ec0e640 5 monclient: authenticate success, global_id 144120 2024-04-26T00:32:06.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.620+0000 7f4b657fa640 1 -- 172.21.15.37:0/3787010667 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b58017070 con 0x7f4b68139050 2024-04-26T00:32:06.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.620+0000 7f4b657fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:06.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.620+0000 7f4b657fa640 1 -- 172.21.15.37:0/3787010667 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b58011d80 con 0x7f4b68139050 2024-04-26T00:32:06.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.620+0000 7f4b657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:06.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.620+0000 7f4b657fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:06.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.620+0000 7f4b657fa640 10 monclient: dump: 2024-04-26T00:32:06.624 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:06.624 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:06.624 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:06.624 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:06.625 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:06.625 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:06.625 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:06.625 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:06.625 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:06.625 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:06.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.620+0000 7f4b66ffd640 4 set_mon_vals no callback set 2024-04-26T00:32:06.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.620+0000 7f4b6ec0e640 10 monclient: _renew_subs 2024-04-26T00:32:06.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.620+0000 7f4b6ec0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:06.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.620+0000 7f4b6ec0e640 1 -- 172.21.15.37:0/3787010667 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b68071420 con 0x7f4b68139050 2024-04-26T00:32:06.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.620+0000 7f4b6ec0e640 10 monclient: _renew_subs 2024-04-26T00:32:06.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.620+0000 7f4b6ec0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:06.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.620+0000 7f4b6ec0e640 1 -- 172.21.15.37:0/3787010667 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b681316b0 con 0x7f4b68139050 2024-04-26T00:32:06.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.620+0000 7f4b6ec0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:06.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.620+0000 7f4b6ec0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:06.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.620+0000 7f4b6ec0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:06.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.620+0000 7f4b657fa640 1 -- 172.21.15.37:0/3787010667 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f4b5801c460 con 0x7f4b68139050 2024-04-26T00:32:06.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.620+0000 7f4b6ec0e640 1 -- 172.21.15.37:0/3787010667 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b68066f40 con 0x7f4b68139050 2024-04-26T00:32:06.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.620+0000 7f4b657fa640 1 --2- 172.21.15.37:0/3787010667 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4b54035e80 0x7f4b54038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:06.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.620+0000 7f4b657fa640 1 -- 172.21.15.37:0/3787010667 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(468..468 src has 370..468) ==== 3927+0+0 (secure 0 0 0) 0x7f4b58048890 con 0x7f4b68139050 2024-04-26T00:32:06.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.624+0000 7f4b6c9aa640 1 --2- 172.21.15.37:0/3787010667 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4b54035e80 0x7f4b54038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:06.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.624+0000 7f4b6c9aa640 10 monclient: get_auth_request con 0x7f4b54035e80 auth_method 0 2024-04-26T00:32:06.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.624+0000 7f4b6c9aa640 1 --2- 172.21.15.37:0/3787010667 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4b54035e80 0x7f4b54038340 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4b600ab000 tx=0x7f4b600a7000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:06.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.628+0000 7f4b657fa640 1 -- 172.21.15.37:0/3787010667 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f4b580475e0 con 0x7f4b68139050 2024-04-26T00:32:06.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.628+0000 7f4b657fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:06.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.628+0000 7f4b657fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:32:06.689 INFO:tasks.rados.rados.0.smithi045.stdout:2071: done (7 left) 2024-04-26T00:32:06.689 INFO:tasks.rados.rados.0.smithi045.stdout:2072: write (excl) oid 11 current snap is 232 2024-04-26T00:32:06.689 INFO:tasks.rados.rados.0.smithi045.stdout:2072: seq_num 759 ranges {756513=632938,1865127=727266} 2024-04-26T00:32:06.707 INFO:tasks.rados.rados.0.smithi045.stdout:2072: writing smithi04526579-11 from 756513 to 1389451 tid 1 2024-04-26T00:32:06.724 INFO:tasks.rados.rados.0.smithi045.stdout:2072: writing smithi04526579-11 from 1865127 to 2592393 tid 2 2024-04-26T00:32:06.724 INFO:tasks.rados.rados.0.smithi045.stdout:2073: read oid 35 snap -1 2024-04-26T00:32:06.724 INFO:tasks.rados.rados.0.smithi045.stdout:2073: expect (ObjNum 717 snap 222 seq_num 717) 2024-04-26T00:32:06.725 INFO:tasks.rados.rados.0.smithi045.stdout:2074: read oid 20 snap -1 2024-04-26T00:32:06.725 INFO:tasks.rados.rados.0.smithi045.stdout:2074: expect (ObjNum 542 snap 167 seq_num 542) 2024-04-26T00:32:06.725 INFO:tasks.rados.rados.0.smithi045.stdout:2075: snap_create 2024-04-26T00:32:06.725 INFO:tasks.rados.rados.0.smithi045.stdout:2065: finishing rollback tid 1 to smithi04526579-15 2024-04-26T00:32:06.725 INFO:tasks.rados.rados.0.smithi045.stdout:2065: finishing rollback tid 2 to smithi04526579-15 2024-04-26T00:32:06.725 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 1333 (ObjNum 724 snap 222 seq_num 724) dirty exists 2024-04-26T00:32:06.725 INFO:tasks.rados.rados.0.smithi045.stdout:2063: finishing write tid 2 to smithi04526579-21 2024-04-26T00:32:06.725 INFO:tasks.rados.rados.0.smithi045.stdout:2063: finishing write tid 3 to smithi04526579-21 2024-04-26T00:32:06.725 INFO:tasks.rados.rados.0.smithi045.stdout:2064: expect (ObjNum 604 snap 186 seq_num 604) 2024-04-26T00:32:06.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.872+0000 7f4b6ec0e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:32:06.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.872+0000 7f4b6ec0e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:32:06.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.872+0000 7f4b6ec0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:06.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.872+0000 7f4b6ec0e640 1 -- 172.21.15.37:0/3787010667 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4b6806b9a0 con 0x7f4b68139050 2024-04-26T00:32:06.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.872+0000 7f4b657fa640 1 -- 172.21.15.37:0/3787010667 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f4b5801a520 con 0x7f4b68139050 2024-04-26T00:32:06.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.872+0000 7f4b657fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:32:06.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.872+0000 7f4b657fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:32:06.869 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":92,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:32:06.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.876+0000 7f4b6ec0e640 1 -- 172.21.15.37:0/3787010667 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4b54035e80 msgr2=0x7f4b54038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:06.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.876+0000 7f4b6ec0e640 1 --2- 172.21.15.37:0/3787010667 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4b54035e80 0x7f4b54038340 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4b600ab000 tx=0x7f4b600a7000 comp rx=0 tx=0).stop 2024-04-26T00:32:06.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.876+0000 7f4b6ec0e640 10 monclient: shutdown 2024-04-26T00:32:06.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.876+0000 7f4b6ec0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:06.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.876+0000 7f4b6ec0e640 1 -- 172.21.15.37:0/3787010667 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4b68139050 msgr2=0x7f4b681cc070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:06.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.876+0000 7f4b6ec0e640 1 --2- 172.21.15.37:0/3787010667 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4b68139050 0x7f4b681cc070 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4b5800c3c0 tx=0x7f4b5800c880 comp rx=0 tx=0).stop 2024-04-26T00:32:06.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.876+0000 7f4b6ec0e640 1 -- 172.21.15.37:0/3787010667 shutdown_connections 2024-04-26T00:32:06.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.876+0000 7f4b6ec0e640 1 --2- 172.21.15.37:0/3787010667 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4b54035e80 0x7f4b54038340 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:06.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.876+0000 7f4b6ec0e640 1 --2- 172.21.15.37:0/3787010667 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4b68139050 0x7f4b681cc070 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:06.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.876+0000 7f4b6ec0e640 1 --2- 172.21.15.37:0/3787010667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b68133f00 0x7f4b68131170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:06.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.876+0000 7f4b6ec0e640 1 --2- 172.21.15.37:0/3787010667 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4b68133b20 0x7f4b68130c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:06.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.876+0000 7f4b6ec0e640 1 -- 172.21.15.37:0/3787010667 >> 172.21.15.37:0/3787010667 conn(0x7f4b680a1ad0 msgr2=0x7f4b68132060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:06.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.876+0000 7f4b6ec0e640 1 -- 172.21.15.37:0/3787010667 shutdown_connections 2024-04-26T00:32:06.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:06.876+0000 7f4b6ec0e640 1 -- 172.21.15.37:0/3787010667 wait complete. 2024-04-26T00:32:06.875 INFO:tasks.rados.rados.0.smithi045.stdout:2063: finishing write tid 4 to smithi04526579-21 2024-04-26T00:32:06.875 INFO:tasks.rados.rados.0.smithi045.stdout:2063: oid 21 updating version 0 to 1093 2024-04-26T00:32:06.875 INFO:tasks.rados.rados.0.smithi045.stdout:2063: oid 21 version 1093 is already newer than 1091 2024-04-26T00:32:06.876 INFO:tasks.rados.rados.0.smithi045.stdout:2063: oid 21 version 1093 is already newer than 1092 2024-04-26T00:32:06.876 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 1093 (ObjNum 756 snap 232 seq_num 756) dirty exists 2024-04-26T00:32:06.876 INFO:tasks.rados.rados.0.smithi045.stdout:2063: left oid 21 (ObjNum 756 snap 232 seq_num 756) 2024-04-26T00:32:06.876 INFO:tasks.rados.rados.0.smithi045.stdout:2066: finishing rollback tid 0 to smithi04526579-19 2024-04-26T00:32:06.876 INFO:tasks.rados.rados.0.smithi045.stdout:2066: finishing rollback tid 1 to smithi04526579-19 2024-04-26T00:32:06.876 INFO:tasks.rados.rados.0.smithi045.stdout:2066: finishing rollback tid 2 to smithi04526579-19 2024-04-26T00:32:06.876 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 1348 (ObjNum 443 snap 138 seq_num 443) dirty exists 2024-04-26T00:32:06.876 INFO:tasks.rados.rados.0.smithi045.stdout:2068: finishing write tid 1 to smithi04526579-17 2024-04-26T00:32:06.876 INFO:tasks.rados.rados.0.smithi045.stdout:2068: finishing write tid 2 to smithi04526579-17 2024-04-26T00:32:06.876 INFO:tasks.rados.rados.0.smithi045.stdout:2068: finishing write tid 3 to smithi04526579-17 2024-04-26T00:32:06.876 INFO:tasks.rados.rados.0.smithi045.stdout:2068: finishing write tid 4 to smithi04526579-17 2024-04-26T00:32:06.876 INFO:tasks.rados.rados.0.smithi045.stdout:2068: oid 17 updating version 0 to 1074 2024-04-26T00:32:06.877 INFO:tasks.rados.rados.0.smithi045.stdout:2068: oid 17 updating version 1074 to 1075 2024-04-26T00:32:06.877 INFO:tasks.rados.rados.0.smithi045.stdout:2068: oid 17 version 1075 is already newer than 1073 2024-04-26T00:32:06.877 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 1075 (ObjNum 758 snap 232 seq_num 758) dirty exists 2024-04-26T00:32:06.877 INFO:tasks.rados.rados.0.smithi045.stdout:2068: left oid 17 (ObjNum 758 snap 232 seq_num 758) 2024-04-26T00:32:06.877 INFO:tasks.rados.rados.0.smithi045.stdout:2070: finishing copy_from racing read to smithi04526579-12 2024-04-26T00:32:06.877 INFO:tasks.rados.rados.0.smithi045.stdout:2070: finishing copy_from to smithi04526579-12 2024-04-26T00:32:06.877 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 1213 (ObjNum 747 snap 230 seq_num 747) dirty exists 2024-04-26T00:32:06.877 INFO:tasks.rados.rados.0.smithi045.stdout:2072: finishing write tid 1 to smithi04526579-11 2024-04-26T00:32:06.877 INFO:tasks.rados.rados.0.smithi045.stdout:2072: finishing write tid 2 to smithi04526579-11 2024-04-26T00:32:06.877 INFO:tasks.rados.rados.0.smithi045.stdout:2072: finishing write tid 3 to smithi04526579-11 2024-04-26T00:32:06.878 INFO:tasks.rados.rados.0.smithi045.stdout:2072: finishing write tid 4 to smithi04526579-11 2024-04-26T00:32:06.878 INFO:tasks.rados.rados.0.smithi045.stdout:2072: oid 11 updating version 0 to 1437 2024-04-26T00:32:06.878 INFO:tasks.rados.rados.0.smithi045.stdout:2072: oid 11 updating version 1437 to 1438 2024-04-26T00:32:06.878 INFO:tasks.rados.rados.0.smithi045.stdout:2072: oid 11 updating version 1438 to 1439 2024-04-26T00:32:06.878 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 1439 (ObjNum 759 snap 232 seq_num 759) dirty exists 2024-04-26T00:32:06.878 INFO:tasks.rados.rados.0.smithi045.stdout:2072: left oid 11 (ObjNum 759 snap 232 seq_num 759) 2024-04-26T00:32:06.878 INFO:tasks.rados.rados.0.smithi045.stdout:2074: expect (ObjNum 542 snap 167 seq_num 542) 2024-04-26T00:32:06.886 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:32:06.886 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-26T00:32:06.886 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:32:06.887 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:32:07.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.008+0000 7f6d75e0d640 10 monclient: get_monmap_and_config 2024-04-26T00:32:07.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.008+0000 7f6d75e0d640 10 monclient: build_initial_monmap 2024-04-26T00:32:07.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.008+0000 7f6d75e0d640 10 monclient: monmap: 2024-04-26T00:32:07.005 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:07.005 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:07.006 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:32:07.015675+0000 2024-04-26T00:32:07.006 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:32:07.015675+0000 2024-04-26T00:32:07.006 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:07.006 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:07.006 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:32:07.006 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:32:07.006 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:32:07.006 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:32:07.006 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:32:07.006 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:07.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d75e0d640 1 Processor -- start 2024-04-26T00:32:07.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d75e0d640 1 -- start start 2024-04-26T00:32:07.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d75e0d640 10 monclient: init 2024-04-26T00:32:07.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d75e0d640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:07.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d75e0d640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-04-26T00:32:07.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d75e0d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d70136f20 0x7f6d7009bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:07.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d75e0d640 10 monclient(hunting): picked mon.noname-b con 0x7f6d70136f20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:07.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d75e0d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6d70137300 0x7f6d70137780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:07.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d75e0d640 10 monclient(hunting): picked mon.noname-a con 0x7f6d70137300 addr v2:172.21.15.37:3300/0 2024-04-26T00:32:07.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d75e0d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6d70133c70 addr v1:172.21.15.37:6789/0 2024-04-26T00:32:07.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d75e0d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d700a50f0 con 0x7f6d70133c70 2024-04-26T00:32:07.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d75e0d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:07.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d75e0d640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f6d7009b330 con 0x7f6d70137300 2024-04-26T00:32:07.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d75e0d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:07.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d75e0d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f6d7009b610 con 0x7f6d70136f20 2024-04-26T00:32:07.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d75e0d640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:07.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d75e0d640 10 monclient(hunting): authenticate will time out at 1708.321310s 2024-04-26T00:32:07.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d6f7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d70136f20 0x7f6d7009bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:07.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d6effd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6d70137300 0x7f6d70137780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:07.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d6ffff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f6d70133c70 0x7f6d701340e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:48662/0 (socket says 172.21.15.37:48662) 2024-04-26T00:32:07.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d6f7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d70136f20 0x7f6d7009bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:56420/0 (socket says 172.21.15.37:56420) 2024-04-26T00:32:07.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d6ffff640 1 -- 172.21.15.37:0/3705650465 learned_addr learned my addr 172.21.15.37:0/3705650465 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:32:07.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d6effd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6d70137300 0x7f6d70137780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:35092/0 (socket says 172.21.15.37:35092) 2024-04-26T00:32:07.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d6f7fe640 10 monclient(hunting): get_auth_request con 0x7f6d70136f20 auth_method 0 2024-04-26T00:32:07.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:07.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d6f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:07.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:07.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d6effd640 10 monclient(hunting): get_auth_request con 0x7f6d70137300 auth_method 0 2024-04-26T00:32:07.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d6effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:07.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d6effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:07.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d6effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:07.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d6f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:07.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d6f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:07.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d6f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:07.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d6effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:07.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d6effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:07.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d6effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:07.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d6f7fe640 10 monclient(hunting): handle_auth_done global_id 84674 payload 274 2024-04-26T00:32:07.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d6f7fe640 1 -- 172.21.15.37:0/3705650465 >> v2:172.21.15.37:3300/0 conn(0x7f6d70137300 msgr2=0x7f6d70137780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:07.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d6f7fe640 1 --2- 172.21.15.37:0/3705650465 >> v2:172.21.15.37:3300/0 conn(0x7f6d70137300 0x7f6d70137780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:07.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d6f7fe640 1 -- 172.21.15.37:0/3705650465 >> v1:172.21.15.37:6789/0 conn(0x7f6d70133c70 legacy=0x7f6d701340e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:07.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d6f7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:32:07.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d6f7fe640 1 monclient: found mon.noname-b 2024-04-26T00:32:07.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:07.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d6f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:32:07.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d6f7fe640 1 -- 172.21.15.37:0/3705650465 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d7009b8f0 con 0x7f6d70136f20 2024-04-26T00:32:07.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.012+0000 7f6d6effd640 1 --2- 172.21.15.37:0/3705650465 >> v2:172.21.15.37:3300/0 conn(0x7f6d70137300 0x7f6d70137780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:32:07.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.016+0000 7f6d6f7fe640 1 -- 172.21.15.37:0/3705650465 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d70136f20 msgr2=0x7f6d7009bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:07.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.016+0000 7f6d6f7fe640 1 --2- 172.21.15.37:0/3705650465 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d70136f20 0x7f6d7009bff0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f6d600097c0 tx=0x7f6d6001c3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fe63fec8b4523e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:07.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.016+0000 7f6d6e7fc640 1 -- 172.21.15.37:0/3705650465 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d600b0070 con 0x7f6d70136f20 2024-04-26T00:32:07.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.016+0000 7f6d6e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:07.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.016+0000 7f6d6e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:32:07.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.016+0000 7f6d6e7fc640 10 monclient: dump: 2024-04-26T00:32:07.014 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:07.014 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:07.014 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.014 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.014 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:07.014 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:07.014 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:07.014 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:07.014 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:07.014 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:07.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.016+0000 7f6d6e7fc640 10 monclient: _finish_auth 0 2024-04-26T00:32:07.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.016+0000 7f6d6e7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:32:07.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.016+0000 7f6d6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:07.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.016+0000 7f6d6e7fc640 1 -- 172.21.15.37:0/3705650465 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d600ab600 con 0x7f6d70136f20 2024-04-26T00:32:07.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.016+0000 7f6d75e0d640 5 monclient: authenticate success, global_id 84674 2024-04-26T00:32:07.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.016+0000 7f6d75e0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:32:07.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.016+0000 7f6d6e7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:07.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.016+0000 7f6d75e0d640 10 monclient: get_monmap_and_config success 2024-04-26T00:32:07.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.016+0000 7f6d75e0d640 4 set_mon_vals no callback set 2024-04-26T00:32:07.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.016+0000 7f6d6e7fc640 1 -- 172.21.15.37:0/3705650465 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d600ab9d0 con 0x7f6d70136f20 2024-04-26T00:32:07.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.016+0000 7f6d6e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:07.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.016+0000 7f6d6e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:32:07.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.016+0000 7f6d6e7fc640 10 monclient: dump: 2024-04-26T00:32:07.015 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:07.015 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:07.015 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.015 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.016 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:07.016 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:07.016 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:07.016 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:07.016 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:07.016 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:07.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.016+0000 7f6d75e0d640 10 monclient: shutdown 2024-04-26T00:32:07.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.016+0000 7f6d75e0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:07.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.016+0000 7f6d75e0d640 1 -- 172.21.15.37:0/3705650465 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d70136f20 msgr2=0x7f6d7009bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:07.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.016+0000 7f6d75e0d640 1 --2- 172.21.15.37:0/3705650465 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d70136f20 0x7f6d7009bff0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f6d600097c0 tx=0x7f6d6001c3b0 comp rx=0 tx=0).stop 2024-04-26T00:32:07.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.016+0000 7f6d75e0d640 1 -- 172.21.15.37:0/3705650465 shutdown_connections 2024-04-26T00:32:07.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.016+0000 7f6d75e0d640 1 --2- 172.21.15.37:0/3705650465 >> v2:172.21.15.37:3300/0 conn(0x7f6d70137300 0x7f6d70137780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:07.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.016+0000 7f6d75e0d640 1 --2- 172.21.15.37:0/3705650465 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d70136f20 0x7f6d7009bff0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:07.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.016+0000 7f6d75e0d640 1 -- 172.21.15.37:0/3705650465 >> 172.21.15.37:0/3705650465 conn(0x7f6d700a61b0 msgr2=0x7f6d700a65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:07.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.016+0000 7f6d75e0d640 1 -- 172.21.15.37:0/3705650465 shutdown_connections 2024-04-26T00:32:07.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.016+0000 7f6d75e0d640 1 -- 172.21.15.37:0/3705650465 wait complete. 2024-04-26T00:32:07.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.016+0000 7f6d75e0d640 10 monclient: build_initial_monmap 2024-04-26T00:32:07.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.016+0000 7f6d75e0d640 10 monclient: monmap: 2024-04-26T00:32:07.017 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:07.017 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:32:07.017 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:32:07.017 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:32:07.017 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:07.017 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:32:07.017 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:32:07.017 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:32:07.017 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:32:07.017 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:07.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d75e0d640 1 Processor -- start 2024-04-26T00:32:07.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d75e0d640 1 -- start start 2024-04-26T00:32:07.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d75e0d640 10 monclient: init 2024-04-26T00:32:07.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d75e0d640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:07.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d75e0d640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:32:07.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d75e0d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d70133c70 0x7f6d701cf4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:07.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d75e0d640 10 monclient(hunting): picked mon.noname-b con 0x7f6d70133c70 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:32:07.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d75e0d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6d70136f20 0x7f6d7013e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:07.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d75e0d640 10 monclient(hunting): picked mon.noname-a con 0x7f6d70136f20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:32:07.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d75e0d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d70137300 0x7f6d7013ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:07.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d75e0d640 10 monclient(hunting): picked mon.noname-c con 0x7f6d70137300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:07.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d75e0d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:07.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d75e0d640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f6d70141e10 con 0x7f6d70136f20 2024-04-26T00:32:07.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d75e0d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:07.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d75e0d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6d70141c90 con 0x7f6d70133c70 2024-04-26T00:32:07.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d75e0d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:07.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d75e0d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f6d70141f90 con 0x7f6d70137300 2024-04-26T00:32:07.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d75e0d640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:07.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d75e0d640 10 monclient(hunting): authenticate will time out at 1708.328556s 2024-04-26T00:32:07.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d6ffff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d70137300 0x7f6d7013ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:07.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d6effd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6d70136f20 0x7f6d7013e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:07.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d6effd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6d70136f20 0x7f6d7013e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:35106/0 (socket says 172.21.15.37:35106) 2024-04-26T00:32:07.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d6effd640 1 -- 172.21.15.37:0/1129744792 learned_addr learned my addr 172.21.15.37:0/1129744792 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:32:07.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d6ffff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d70137300 0x7f6d7013ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:56422/0 (socket says 172.21.15.37:56422) 2024-04-26T00:32:07.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d6effd640 10 monclient(hunting): get_auth_request con 0x7f6d70136f20 auth_method 0 2024-04-26T00:32:07.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d6effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:07.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d6effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:07.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d6effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:07.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d6ffff640 10 monclient(hunting): get_auth_request con 0x7f6d70137300 auth_method 0 2024-04-26T00:32:07.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:07.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d6ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:07.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:07.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d6effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:07.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d6effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:07.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d6effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:07.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:07.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:07.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d6f7fe640 1 --2- 172.21.15.37:0/1129744792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d70133c70 0x7f6d701cf4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:07.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:07.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d6f7fe640 10 monclient(hunting): get_auth_request con 0x7f6d70133c70 auth_method 0 2024-04-26T00:32:07.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:07.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d6f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:07.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:07.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d6effd640 10 monclient(hunting): handle_auth_done global_id 144126 payload 824 2024-04-26T00:32:07.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d6effd640 1 -- 172.21.15.37:0/1129744792 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d70137300 msgr2=0x7f6d7013ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:07.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d6effd640 1 --2- 172.21.15.37:0/1129744792 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6d70137300 0x7f6d7013ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:07.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d6effd640 1 -- 172.21.15.37:0/1129744792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d70133c70 msgr2=0x7f6d701cf4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:07.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d6effd640 1 --2- 172.21.15.37:0/1129744792 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d70133c70 0x7f6d701cf4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:07.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d6effd640 10 monclient: _finish_hunting 0 2024-04-26T00:32:07.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d6effd640 1 monclient: found mon.noname-a 2024-04-26T00:32:07.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d6effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:07.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d6effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:32:07.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d6effd640 1 -- 172.21.15.37:0/1129744792 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d700a01e0 con 0x7f6d70136f20 2024-04-26T00:32:07.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d6effd640 1 -- 172.21.15.37:0/1129744792 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6d70136f20 msgr2=0x7f6d7013e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:07.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d6effd640 1 --2- 172.21.15.37:0/1129744792 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6d70136f20 0x7f6d7013e540 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6d6400ec10 tx=0x7f6d6400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:07.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d57fff640 1 -- 172.21.15.37:0/1129744792 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d64017070 con 0x7f6d70136f20 2024-04-26T00:32:07.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d57fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:07.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d57fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:32:07.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d57fff640 10 monclient: dump: 2024-04-26T00:32:07.026 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:07.026 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:07.026 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.027 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.027 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:07.027 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:07.027 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:07.027 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:07.027 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:07.027 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:07.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d57fff640 10 monclient: _finish_auth 0 2024-04-26T00:32:07.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d57fff640 10 monclient: _check_auth_tickets 2024-04-26T00:32:07.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:07.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d57fff640 1 -- 172.21.15.37:0/1129744792 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d6400ce40 con 0x7f6d70136f20 2024-04-26T00:32:07.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d57fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:07.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d57fff640 1 -- 172.21.15.37:0/1129744792 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d64013690 con 0x7f6d70136f20 2024-04-26T00:32:07.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d57fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:07.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d57fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:07.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d57fff640 10 monclient: dump: 2024-04-26T00:32:07.028 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:07.028 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:07.028 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.028 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.028 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:07.028 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:07.028 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:07.028 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:07.028 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:07.028 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:07.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.020+0000 7f6d6dffb640 4 set_mon_vals no callback set 2024-04-26T00:32:07.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.024+0000 7f6d75e0d640 5 monclient: authenticate success, global_id 144126 2024-04-26T00:32:07.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.024+0000 7f6d75e0d640 10 monclient: _renew_subs 2024-04-26T00:32:07.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.024+0000 7f6d75e0d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:07.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.024+0000 7f6d75e0d640 1 -- 172.21.15.37:0/1129744792 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d701385d0 con 0x7f6d70136f20 2024-04-26T00:32:07.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.024+0000 7f6d75e0d640 10 monclient: _renew_subs 2024-04-26T00:32:07.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.024+0000 7f6d75e0d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:07.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.024+0000 7f6d75e0d640 1 -- 172.21.15.37:0/1129744792 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d701359d0 con 0x7f6d70136f20 2024-04-26T00:32:07.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.024+0000 7f6d75e0d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:07.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.024+0000 7f6d75e0d640 1 --2- 172.21.15.37:0/1129744792 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6d7009e540 0x7f6d7009e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:07.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.024+0000 7f6d75e0d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:07.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.024+0000 7f6d75e0d640 1 -- 172.21.15.37:0/1129744792 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f6d7004f510 con 0x7f6d7009e540 2024-04-26T00:32:07.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.024+0000 7f6d6f7fe640 1 --2- 172.21.15.37:0/1129744792 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6d7009e540 0x7f6d7009e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:07.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.024+0000 7f6d57fff640 1 -- 172.21.15.37:0/1129744792 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f6d6401a070 con 0x7f6d70136f20 2024-04-26T00:32:07.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.024+0000 7f6d6f7fe640 10 monclient: get_auth_request con 0x7f6d7009e540 auth_method 0 2024-04-26T00:32:07.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.024+0000 7f6d6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:07.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.024+0000 7f6d6f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:07.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.024+0000 7f6d6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:07.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.024+0000 7f6d6f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:07.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.024+0000 7f6d6f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:07.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.024+0000 7f6d6f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:07.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.024+0000 7f6d57fff640 1 --2- 172.21.15.37:0/1129744792 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6d5c035eb0 0x7f6d5c038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:07.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.024+0000 7f6d57fff640 1 -- 172.21.15.37:0/1129744792 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(469..469 src has 370..469) ==== 3943+0+0 (secure 0 0 0) 0x7f6d64048ac0 con 0x7f6d70136f20 2024-04-26T00:32:07.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.024+0000 7f6d6f7fe640 10 monclient(hunting): handle_auth_done global_id 144129 payload 824 2024-04-26T00:32:07.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.024+0000 7f6d6f7fe640 1 -- 172.21.15.37:0/1129744792 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6d701cfbd0 con 0x7f6d7009e540 2024-04-26T00:32:07.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.024+0000 7f6d6ffff640 1 --2- 172.21.15.37:0/1129744792 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6d5c035eb0 0x7f6d5c038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:07.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.024+0000 7f6d6ffff640 10 monclient: get_auth_request con 0x7f6d5c035eb0 auth_method 0 2024-04-26T00:32:07.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.024+0000 7f6d6f7fe640 1 -- 172.21.15.37:0/1129744792 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6d7009e540 msgr2=0x7f6d7009e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:07.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.024+0000 7f6d6f7fe640 1 --2- 172.21.15.37:0/1129744792 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6d7009e540 0x7f6d7009e920 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6d7009bbd0 tx=0x7f6d6001abf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:07.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.024+0000 7f6d57fff640 1 -- 172.21.15.37:0/1129744792 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d600ba070 con 0x7f6d7009e540 2024-04-26T00:32:07.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.024+0000 7f6d57fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:07.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.024+0000 7f6d57fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:07.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.024+0000 7f6d57fff640 10 monclient: dump: 2024-04-26T00:32:07.031 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:07.031 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:07.031 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.031 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.031 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:07.031 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:07.031 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:07.031 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:07.031 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:07.031 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:07.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.028+0000 7f6d57fff640 1 -- 172.21.15.37:0/1129744792 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6d600be400 con 0x7f6d7009e540 2024-04-26T00:32:07.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.028+0000 7f6d57fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:07.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.028+0000 7f6d57fff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:32:07.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.028+0000 7f6d57fff640 1 -- 172.21.15.37:0/1129744792 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6d7009e540 msgr2=0x7f6d7009e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:07.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.028+0000 7f6d57fff640 1 --2- 172.21.15.37:0/1129744792 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6d7009e540 0x7f6d7009e920 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6d7009bbd0 tx=0x7f6d6001abf0 comp rx=0 tx=0).stop 2024-04-26T00:32:07.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.028+0000 7f6d57fff640 1 -- 172.21.15.37:0/1129744792 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6d7009e540 msgr2=0x7f6d7009e920 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:32:07.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.028+0000 7f6d57fff640 1 --2- 172.21.15.37:0/1129744792 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6d7009e540 0x7f6d7009e920 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6d7009bbd0 tx=0x7f6d6001abf0 comp rx=0 tx=0).stop 2024-04-26T00:32:07.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.028+0000 7f6d6ffff640 1 --2- 172.21.15.37:0/1129744792 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6d5c035eb0 0x7f6d5c038370 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f6d680055b0 tx=0x7f6d68005540 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:07.033 INFO:tasks.rados.rados.0.smithi045.stdout:2073: expect (ObjNum 717 snap 222 seq_num 717) 2024-04-26T00:32:07.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.032+0000 7f6d75e0d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:32:07.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.032+0000 7f6d75e0d640 1 --2- 172.21.15.37:0/1129744792 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6d7009ee60 0x7f6d700a08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:07.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.032+0000 7f6d75e0d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:07.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.032+0000 7f6d75e0d640 1 -- 172.21.15.37:0/1129744792 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f6d70075f10 con 0x7f6d7009ee60 2024-04-26T00:32:07.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.032+0000 7f6d6f7fe640 1 --2- 172.21.15.37:0/1129744792 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6d7009ee60 0x7f6d700a08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:07.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.032+0000 7f6d6f7fe640 10 monclient: get_auth_request con 0x7f6d7009ee60 auth_method 0 2024-04-26T00:32:07.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.032+0000 7f6d6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:07.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.032+0000 7f6d6f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:07.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.032+0000 7f6d6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:07.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.032+0000 7f6d6f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:07.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.032+0000 7f6d6f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:07.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.032+0000 7f6d6f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:07.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.032+0000 7f6d6f7fe640 10 monclient(hunting): handle_auth_done global_id 144132 payload 824 2024-04-26T00:32:07.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.032+0000 7f6d6f7fe640 1 -- 172.21.15.37:0/1129744792 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6d7009f270 con 0x7f6d7009ee60 2024-04-26T00:32:07.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.036+0000 7f6d6f7fe640 1 -- 172.21.15.37:0/1129744792 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6d7009ee60 msgr2=0x7f6d700a08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:07.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.036+0000 7f6d6f7fe640 1 --2- 172.21.15.37:0/1129744792 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6d7009ee60 0x7f6d700a08e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6d600097c0 tx=0x7f6d60019150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:07.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.036+0000 7f6d57fff640 1 -- 172.21.15.37:0/1129744792 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d600ba050 con 0x7f6d7009ee60 2024-04-26T00:32:07.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.036+0000 7f6d57fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:07.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.036+0000 7f6d57fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:07.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.036+0000 7f6d57fff640 10 monclient: dump: 2024-04-26T00:32:07.036 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:07.036 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:07.036 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.036 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.036 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:07.036 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:07.036 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:07.036 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:07.036 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:07.036 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:07.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.036+0000 7f6d57fff640 1 -- 172.21.15.37:0/1129744792 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7f6d60003c70 con 0x7f6d7009ee60 2024-04-26T00:32:07.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.036+0000 7f6d57fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:32:07.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.036+0000 7f6d57fff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:32:07.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.036+0000 7f6d57fff640 1 -- 172.21.15.37:0/1129744792 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6d7009ee60 msgr2=0x7f6d700a08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:07.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.036+0000 7f6d57fff640 1 --2- 172.21.15.37:0/1129744792 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6d7009ee60 0x7f6d700a08e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6d600097c0 tx=0x7f6d60019150 comp rx=0 tx=0).stop 2024-04-26T00:32:07.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.036+0000 7f6d57fff640 1 -- 172.21.15.37:0/1129744792 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6d7009ee60 msgr2=0x7f6d700a08e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:32:07.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.036+0000 7f6d57fff640 1 --2- 172.21.15.37:0/1129744792 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6d7009ee60 0x7f6d700a08e0 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6d600097c0 tx=0x7f6d60019150 comp rx=0 tx=0).stop 2024-04-26T00:32:07.037 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:32:07.037 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:32:07.037 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:32:07.037 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:32:07.037 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 92, 2024-04-26T00:32:07.038 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 13579, 2024-04-26T00:32:07.038 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:32:07.038 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:32:07.038 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:32:07.038 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:32:07.038 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:07.038 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 7, 2024-04-26T00:32:07.038 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:32:07.038 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:32:07.038 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:32:07.038 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:07.038 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:07.038 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:07.039 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:07.039 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:07.039 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:07.039 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:07.039 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:07.039 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:07.039 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:07.039 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:07.039 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:07.039 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:32:07.039 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:32:07.039 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:07.039 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:07.039 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:07.039 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:07.039 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:07.040 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:07.040 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:07.040 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:07.040 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:07.040 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:07.040 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:07.040 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:07.040 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:07.040 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:32:07.040 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:32:07.040 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:32:07.040 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:32:07.040 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:32:07.040 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:32:07.040 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:32:07.041 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:32:07.041 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:32:07.041 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:32:07.041 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:32:07.041 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:32:07.041 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:32:07.041 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:32:07.041 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:32:07.041 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:32:07.041 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:32:07.041 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:07.041 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:07.041 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:07.041 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:07.041 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:07.042 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:07.042 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:07.042 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:07.042 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:07.042 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:07.042 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:07.042 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:07.042 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:32:07.042 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:07.042 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:32:07.042 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:07.042 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:32:07.045 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:32:07.045 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:07.045 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:07.045 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:07.045 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:07.046 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:32:07.046 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:07.046 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:07.046 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:07.046 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:07.046 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:32:07.046 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:07.046 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:07.046 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:07.046 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:07.046 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:32:07.046 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:32:07.046 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:07.047 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:07.047 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:07.047 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:07.047 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:07.047 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:32:07.047 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:32:07.047 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:07.047 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:07.047 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:07.047 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:07.047 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:32:07.047 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:07.047 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:07.047 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:07.047 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:07.048 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:32:07.048 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:07.048 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:07.048 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:07.048 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:07.048 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:32:07.048 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:32:07.048 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:07.048 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:07.048 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:07.048 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:07.048 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:07.048 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:32:07.048 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:32:07.049 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:07.049 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:07.049 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:07.049 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:07.049 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:32:07.049 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:07.049 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:07.049 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:07.049 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:07.049 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:32:07.049 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:07.049 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:07.049 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:07.049 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:07.050 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:32:07.050 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:32:07.050 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:07.050 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:07.050 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:07.050 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:07.050 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:07.050 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:07.050 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:32:07.050 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:32:07.050 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:07.050 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:07.050 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:07.050 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:32:07.051 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:07.051 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:07.051 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:32:07.051 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:07.051 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:07.051 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:07.051 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:32:07.051 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:07.051 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:07.051 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:07.051 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:32:07.051 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:32:07.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.036+0000 7f6d75e0d640 1 -- 172.21.15.37:0/1129744792 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6d5c035eb0 msgr2=0x7f6d5c038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:07.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.036+0000 7f6d75e0d640 1 --2- 172.21.15.37:0/1129744792 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6d5c035eb0 0x7f6d5c038370 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f6d680055b0 tx=0x7f6d68005540 comp rx=0 tx=0).stop 2024-04-26T00:32:07.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.036+0000 7f6d75e0d640 10 monclient: shutdown 2024-04-26T00:32:07.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.036+0000 7f6d75e0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:07.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.036+0000 7f6d75e0d640 1 -- 172.21.15.37:0/1129744792 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6d70136f20 msgr2=0x7f6d7013e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:07.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.036+0000 7f6d75e0d640 1 --2- 172.21.15.37:0/1129744792 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6d70136f20 0x7f6d7013e540 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6d6400ec10 tx=0x7f6d6400c520 comp rx=0 tx=0).stop 2024-04-26T00:32:07.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.036+0000 7f6d6ffff640 1 -- 172.21.15.37:0/1129744792 reap_dead start 2024-04-26T00:32:07.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.036+0000 7f6d6ffff640 1 -- 172.21.15.37:0/1129744792 reap_dead start 2024-04-26T00:32:07.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.036+0000 7f6d75e0d640 1 -- 172.21.15.37:0/1129744792 shutdown_connections 2024-04-26T00:32:07.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.036+0000 7f6d75e0d640 1 -- 172.21.15.37:0/1129744792 >> 172.21.15.37:0/1129744792 conn(0x7f6d700a61b0 msgr2=0x7f6d700a74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:07.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.036+0000 7f6d75e0d640 1 -- 172.21.15.37:0/1129744792 shutdown_connections 2024-04-26T00:32:07.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.036+0000 7f6d75e0d640 1 -- 172.21.15.37:0/1129744792 wait complete. 2024-04-26T00:32:07.054 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:32:07.054 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:32:07.054 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:32:07.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6463555640 10 monclient: get_monmap_and_config 2024-04-26T00:32:07.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6463555640 10 monclient: build_initial_monmap 2024-04-26T00:32:07.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6463555640 10 monclient: monmap: 2024-04-26T00:32:07.168 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:07.168 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:07.168 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:32:07.177891+0000 2024-04-26T00:32:07.168 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:32:07.177891+0000 2024-04-26T00:32:07.168 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:07.168 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:07.168 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:32:07.168 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:32:07.168 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:32:07.168 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:32:07.168 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:32:07.168 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:07.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6463555640 1 Processor -- start 2024-04-26T00:32:07.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6463555640 1 -- start start 2024-04-26T00:32:07.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6463555640 10 monclient: init 2024-04-26T00:32:07.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6463555640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:07.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6463555640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-04-26T00:32:07.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6463555640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f645c136f20 0x7f645c09bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:07.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6463555640 10 monclient(hunting): picked mon.noname-c con 0x7f645c136f20 addr v2:172.21.15.45:3300/0 2024-04-26T00:32:07.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6463555640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f645c137300 0x7f645c1377b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:07.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6463555640 10 monclient(hunting): picked mon.noname-a con 0x7f645c137300 addr v2:172.21.15.37:3300/0 2024-04-26T00:32:07.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6463555640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f645c133c70 0x7f645c134120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:07.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6463555640 10 monclient(hunting): picked mon.noname-b con 0x7f645c133c70 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:07.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6463555640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:07.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6463555640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f645c0a50f0 con 0x7f645c137300 2024-04-26T00:32:07.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6463555640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:07.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6463555640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f645c09b330 con 0x7f645c136f20 2024-04-26T00:32:07.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6463555640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:07.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6463555640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f645c09b610 con 0x7f645c133c70 2024-04-26T00:32:07.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6463555640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:07.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6463555640 10 monclient(hunting): authenticate will time out at 1708.483324s 2024-04-26T00:32:07.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6460af0640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f645c137300 0x7f645c1377b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:07.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6461af2640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f645c133c70 0x7f645c134120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:07.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6460af0640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f645c137300 0x7f645c1377b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:35150/0 (socket says 172.21.15.37:35150) 2024-04-26T00:32:07.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6461af2640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f645c133c70 0x7f645c134120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:56438/0 (socket says 172.21.15.37:56438) 2024-04-26T00:32:07.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6460af0640 1 -- 172.21.15.37:0/523921668 learned_addr learned my addr 172.21.15.37:0/523921668 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:32:07.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6460af0640 10 monclient(hunting): get_auth_request con 0x7f645c137300 auth_method 0 2024-04-26T00:32:07.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6460af0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:07.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6460af0640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:07.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6460af0640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:07.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6461af2640 10 monclient(hunting): get_auth_request con 0x7f645c133c70 auth_method 0 2024-04-26T00:32:07.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6461af2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:07.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6461af2640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:07.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6461af2640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:07.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6460af0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:07.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6460af0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:07.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6460af0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:07.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6461af2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:07.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6461af2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:07.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6461af2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:07.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f64612f1640 1 --2- 172.21.15.37:0/523921668 >> v2:172.21.15.45:3300/0 conn(0x7f645c136f20 0x7f645c09bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:07.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f64612f1640 10 monclient(hunting): get_auth_request con 0x7f645c136f20 auth_method 0 2024-04-26T00:32:07.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f64612f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:07.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f64612f1640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:07.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f64612f1640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:07.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6461af2640 10 monclient(hunting): handle_auth_done global_id 84680 payload 274 2024-04-26T00:32:07.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6461af2640 1 -- 172.21.15.37:0/523921668 >> v2:172.21.15.45:3300/0 conn(0x7f645c136f20 msgr2=0x7f645c09bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:07.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6461af2640 1 --2- 172.21.15.37:0/523921668 >> v2:172.21.15.45:3300/0 conn(0x7f645c136f20 0x7f645c09bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:07.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6461af2640 1 -- 172.21.15.37:0/523921668 >> v2:172.21.15.37:3300/0 conn(0x7f645c137300 msgr2=0x7f645c1377b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:07.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6461af2640 1 --2- 172.21.15.37:0/523921668 >> v2:172.21.15.37:3300/0 conn(0x7f645c137300 0x7f645c1377b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:07.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6461af2640 10 monclient: _finish_hunting 0 2024-04-26T00:32:07.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6461af2640 1 monclient: found mon.noname-b 2024-04-26T00:32:07.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6461af2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:07.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6461af2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:32:07.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6461af2640 1 -- 172.21.15.37:0/523921668 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f645c09b8f0 con 0x7f645c133c70 2024-04-26T00:32:07.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.172+0000 7f6460af0640 1 --2- 172.21.15.37:0/523921668 >> v2:172.21.15.37:3300/0 conn(0x7f645c137300 0x7f645c1377b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:32:07.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6461af2640 1 -- 172.21.15.37:0/523921668 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f645c133c70 msgr2=0x7f645c134120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:07.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6461af2640 1 --2- 172.21.15.37:0/523921668 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f645c133c70 0x7f645c134120 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f645c068940 tx=0x7f645401c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c9457db6bd77963 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:07.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6453fff640 1 -- 172.21.15.37:0/523921668 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64540b2070 con 0x7f645c133c70 2024-04-26T00:32:07.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6453fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:07.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6453fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:32:07.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6453fff640 10 monclient: dump: 2024-04-26T00:32:07.182 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:07.182 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:07.182 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.182 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.182 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:07.182 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:07.182 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:07.182 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:07.183 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:07.183 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:07.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6453fff640 10 monclient: _finish_auth 0 2024-04-26T00:32:07.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6453fff640 10 monclient: _check_auth_tickets 2024-04-26T00:32:07.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6453fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:07.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6463555640 5 monclient: authenticate success, global_id 84680 2024-04-26T00:32:07.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6463555640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:32:07.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6453fff640 1 -- 172.21.15.37:0/523921668 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64540a5780 con 0x7f645c133c70 2024-04-26T00:32:07.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6453fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:07.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6453fff640 1 -- 172.21.15.37:0/523921668 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64540a5b50 con 0x7f645c133c70 2024-04-26T00:32:07.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6463555640 10 monclient: get_monmap_and_config success 2024-04-26T00:32:07.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6463555640 4 set_mon_vals no callback set 2024-04-26T00:32:07.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6453fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:07.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6453fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:32:07.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6453fff640 10 monclient: dump: 2024-04-26T00:32:07.184 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:07.184 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:07.184 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.184 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.184 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:07.184 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:07.184 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:07.184 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:07.184 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:07.184 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:07.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6463555640 10 monclient: shutdown 2024-04-26T00:32:07.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6463555640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:07.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6463555640 1 -- 172.21.15.37:0/523921668 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f645c133c70 msgr2=0x7f645c134120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:07.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6463555640 1 --2- 172.21.15.37:0/523921668 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f645c133c70 0x7f645c134120 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f645c068940 tx=0x7f645401c7f0 comp rx=0 tx=0).stop 2024-04-26T00:32:07.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6463555640 1 -- 172.21.15.37:0/523921668 shutdown_connections 2024-04-26T00:32:07.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6463555640 1 --2- 172.21.15.37:0/523921668 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f645c133c70 0x7f645c134120 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:07.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6463555640 1 --2- 172.21.15.37:0/523921668 >> v2:172.21.15.37:3300/0 conn(0x7f645c137300 0x7f645c1377b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:07.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6463555640 1 --2- 172.21.15.37:0/523921668 >> v2:172.21.15.45:3300/0 conn(0x7f645c136f20 0x7f645c09bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:07.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6463555640 1 -- 172.21.15.37:0/523921668 >> 172.21.15.37:0/523921668 conn(0x7f645c0a61b0 msgr2=0x7f645c0a65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:07.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6463555640 1 -- 172.21.15.37:0/523921668 shutdown_connections 2024-04-26T00:32:07.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6463555640 1 -- 172.21.15.37:0/523921668 wait complete. 2024-04-26T00:32:07.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6463555640 10 monclient: build_initial_monmap 2024-04-26T00:32:07.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6463555640 10 monclient: monmap: 2024-04-26T00:32:07.185 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:07.185 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:32:07.186 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:32:07.186 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:32:07.186 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:07.186 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:32:07.186 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:32:07.186 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:32:07.186 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:32:07.186 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:07.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6463555640 1 Processor -- start 2024-04-26T00:32:07.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6463555640 1 -- start start 2024-04-26T00:32:07.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6463555640 10 monclient: init 2024-04-26T00:32:07.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6463555640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:07.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6463555640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:32:07.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6463555640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f645c133c70 0x7f645c132fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:07.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6463555640 10 monclient(hunting): picked mon.noname-c con 0x7f645c133c70 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:07.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6463555640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f645c136f20 0x7f645c1334f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:07.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6463555640 10 monclient(hunting): picked mon.noname-b con 0x7f645c136f20 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:32:07.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6463555640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f645c137300 0x7f645c1cee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:07.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6463555640 10 monclient(hunting): picked mon.noname-a con 0x7f645c137300 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:32:07.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6463555640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:07.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6463555640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f645c09ac60 con 0x7f645c137300 2024-04-26T00:32:07.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6463555640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:07.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6463555640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f645c09aae0 con 0x7f645c136f20 2024-04-26T00:32:07.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6463555640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:07.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6463555640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f645c140c80 con 0x7f645c133c70 2024-04-26T00:32:07.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6463555640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:07.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6463555640 10 monclient(hunting): authenticate will time out at 1708.490845s 2024-04-26T00:32:07.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6461af2640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f645c137300 0x7f645c1cee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:07.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f64612f1640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f645c133c70 0x7f645c132fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:07.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6461af2640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f645c137300 0x7f645c1cee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:35160/0 (socket says 172.21.15.37:35160) 2024-04-26T00:32:07.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f64612f1640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f645c133c70 0x7f645c132fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:56448/0 (socket says 172.21.15.37:56448) 2024-04-26T00:32:07.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6461af2640 1 -- 172.21.15.37:0/2669082915 learned_addr learned my addr 172.21.15.37:0/2669082915 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:32:07.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6461af2640 10 monclient(hunting): get_auth_request con 0x7f645c137300 auth_method 0 2024-04-26T00:32:07.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6461af2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:07.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6461af2640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:07.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6461af2640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:07.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f64612f1640 10 monclient(hunting): get_auth_request con 0x7f645c133c70 auth_method 0 2024-04-26T00:32:07.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f64612f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:07.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f64612f1640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:07.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f64612f1640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:07.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6461af2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:07.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6461af2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:07.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6461af2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:07.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f64612f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:07.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6460af0640 1 --2- 172.21.15.37:0/2669082915 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f645c136f20 0x7f645c1334f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:07.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f64612f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:07.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f64612f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:07.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6460af0640 10 monclient(hunting): get_auth_request con 0x7f645c136f20 auth_method 0 2024-04-26T00:32:07.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6460af0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:07.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6460af0640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:07.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6460af0640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:07.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6461af2640 10 monclient(hunting): handle_auth_done global_id 144138 payload 824 2024-04-26T00:32:07.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6461af2640 1 -- 172.21.15.37:0/2669082915 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f645c133c70 msgr2=0x7f645c132fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:07.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6461af2640 1 --2- 172.21.15.37:0/2669082915 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f645c133c70 0x7f645c132fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:07.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6461af2640 1 -- 172.21.15.37:0/2669082915 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f645c136f20 msgr2=0x7f645c1334f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:07.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6461af2640 1 --2- 172.21.15.37:0/2669082915 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f645c136f20 0x7f645c1334f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:07.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6461af2640 10 monclient: _finish_hunting 0 2024-04-26T00:32:07.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6461af2640 1 monclient: found mon.noname-a 2024-04-26T00:32:07.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6461af2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:07.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6461af2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:32:07.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.180+0000 7f6461af2640 1 -- 172.21.15.37:0/2669082915 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f645c09ae30 con 0x7f645c137300 2024-04-26T00:32:07.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f6461af2640 1 -- 172.21.15.37:0/2669082915 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f645c137300 msgr2=0x7f645c1cee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:07.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f6461af2640 1 --2- 172.21.15.37:0/2669082915 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f645c137300 0x7f645c1cee30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f64540b0040 tx=0x7f6454019590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:07.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f6451ffb640 1 -- 172.21.15.37:0/2669082915 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64540b2050 con 0x7f645c137300 2024-04-26T00:32:07.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f6451ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:07.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f6451ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:32:07.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f6451ffb640 10 monclient: dump: 2024-04-26T00:32:07.190 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:07.190 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:07.190 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.191 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.191 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:07.191 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:07.191 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:07.191 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:07.191 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:07.191 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:07.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f6451ffb640 10 monclient: _finish_auth 0 2024-04-26T00:32:07.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f6451ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:32:07.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f6451ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:07.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f6451ffb640 1 -- 172.21.15.37:0/2669082915 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64540be070 con 0x7f645c137300 2024-04-26T00:32:07.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f6463555640 5 monclient: authenticate success, global_id 144138 2024-04-26T00:32:07.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f6463555640 10 monclient: _renew_subs 2024-04-26T00:32:07.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f6463555640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:07.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f6463555640 1 -- 172.21.15.37:0/2669082915 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f645c136260 con 0x7f645c137300 2024-04-26T00:32:07.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f6451ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:07.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f6451ffb640 1 -- 172.21.15.37:0/2669082915 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6454009450 con 0x7f645c137300 2024-04-26T00:32:07.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f6451ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:07.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f6451ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:07.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f6451ffb640 10 monclient: dump: 2024-04-26T00:32:07.192 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:07.192 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:07.192 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.192 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.192 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:07.192 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:07.192 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:07.192 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:07.192 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:07.192 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:07.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f6463555640 10 monclient: _renew_subs 2024-04-26T00:32:07.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f6463555640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:07.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f6463555640 1 -- 172.21.15.37:0/2669082915 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f645c13ca50 con 0x7f645c137300 2024-04-26T00:32:07.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f64537fe640 4 set_mon_vals no callback set 2024-04-26T00:32:07.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f6463555640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:07.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f6463555640 1 --2- 172.21.15.37:0/2669082915 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f645c1cf370 0x7f645c1cf750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:07.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f6463555640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:07.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f6463555640 1 -- 172.21.15.37:0/2669082915 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f645c13c210 con 0x7f645c1cf370 2024-04-26T00:32:07.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f64612f1640 1 --2- 172.21.15.37:0/2669082915 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f645c1cf370 0x7f645c1cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:07.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f64612f1640 10 monclient: get_auth_request con 0x7f645c1cf370 auth_method 0 2024-04-26T00:32:07.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f64612f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:07.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f64612f1640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:07.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f64612f1640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:07.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f64612f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:07.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f64612f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:07.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f64612f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:07.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f6451ffb640 1 -- 172.21.15.37:0/2669082915 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f64540adac0 con 0x7f645c137300 2024-04-26T00:32:07.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f64612f1640 10 monclient(hunting): handle_auth_done global_id 84686 payload 824 2024-04-26T00:32:07.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f64612f1640 1 -- 172.21.15.37:0/2669082915 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f645c1cfc90 con 0x7f645c1cf370 2024-04-26T00:32:07.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f6451ffb640 1 --2- 172.21.15.37:0/2669082915 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6440035ee0 0x7f64400383a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:07.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f6451ffb640 1 -- 172.21.15.37:0/2669082915 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(469..469 src has 370..469) ==== 3943+0+0 (secure 0 0 0) 0x7f64540e39f0 con 0x7f645c137300 2024-04-26T00:32:07.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f6460af0640 1 --2- 172.21.15.37:0/2669082915 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6440035ee0 0x7f64400383a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:07.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.184+0000 7f6460af0640 10 monclient: get_auth_request con 0x7f6440035ee0 auth_method 0 2024-04-26T00:32:07.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.188+0000 7f64612f1640 1 -- 172.21.15.37:0/2669082915 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f645c1cf370 msgr2=0x7f645c1cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:07.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.188+0000 7f64612f1640 1 --2- 172.21.15.37:0/2669082915 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f645c1cf370 0x7f645c1cf750 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f644c010780 tx=0x7f644c010c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:07.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.188+0000 7f6451ffb640 1 -- 172.21.15.37:0/2669082915 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f644c018020 con 0x7f645c1cf370 2024-04-26T00:32:07.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.188+0000 7f6451ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:07.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.188+0000 7f6451ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:32:07.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.188+0000 7f6451ffb640 10 monclient: dump: 2024-04-26T00:32:07.197 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:07.197 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:07.197 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.197 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.197 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:07.198 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:07.198 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:07.198 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:07.198 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:07.198 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:07.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.188+0000 7f6451ffb640 1 -- 172.21.15.37:0/2669082915 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f644c004450 con 0x7f645c1cf370 2024-04-26T00:32:07.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.188+0000 7f6451ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:07.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.188+0000 7f6451ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:32:07.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.188+0000 7f6451ffb640 1 -- 172.21.15.37:0/2669082915 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f645c1cf370 msgr2=0x7f645c1cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:07.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.188+0000 7f6451ffb640 1 --2- 172.21.15.37:0/2669082915 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f645c1cf370 0x7f645c1cf750 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f644c010780 tx=0x7f644c010c40 comp rx=0 tx=0).stop 2024-04-26T00:32:07.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.188+0000 7f6451ffb640 1 -- 172.21.15.37:0/2669082915 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f645c1cf370 msgr2=0x7f645c1cf750 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:32:07.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.188+0000 7f6451ffb640 1 --2- 172.21.15.37:0/2669082915 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f645c1cf370 0x7f645c1cf750 secure :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f644c010780 tx=0x7f644c010c40 comp rx=0 tx=0).stop 2024-04-26T00:32:07.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.188+0000 7f6460af0640 1 --2- 172.21.15.37:0/2669082915 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6440035ee0 0x7f64400383a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f6458009630 tx=0x7f6458008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:07.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.192+0000 7f6463555640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:32:07.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.196+0000 7f6463555640 1 --2- 172.21.15.37:0/2669082915 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f645c09d320 0x7f645c09d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:07.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.196+0000 7f6463555640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:07.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.196+0000 7f6463555640 1 -- 172.21.15.37:0/2669082915 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f645c13b040 con 0x7f645c09d320 2024-04-26T00:32:07.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.196+0000 7f64612f1640 1 --2- 172.21.15.37:0/2669082915 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f645c09d320 0x7f645c09d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:07.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.196+0000 7f64612f1640 10 monclient: get_auth_request con 0x7f645c09d320 auth_method 0 2024-04-26T00:32:07.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.196+0000 7f64612f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:07.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.196+0000 7f64612f1640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:07.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.196+0000 7f64612f1640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:07.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.196+0000 7f64612f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:07.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.196+0000 7f64612f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:07.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.196+0000 7f64612f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:07.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.196+0000 7f64612f1640 10 monclient(hunting): handle_auth_done global_id 84689 payload 824 2024-04-26T00:32:07.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.196+0000 7f64612f1640 1 -- 172.21.15.37:0/2669082915 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f645c133a30 con 0x7f645c09d320 2024-04-26T00:32:07.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.196+0000 7f64612f1640 1 -- 172.21.15.37:0/2669082915 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f645c09d320 msgr2=0x7f645c09d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:07.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.196+0000 7f64612f1640 1 --2- 172.21.15.37:0/2669082915 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f645c09d320 0x7f645c09d730 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f644c010780 tx=0x7f644c0046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:07.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.196+0000 7f6451ffb640 1 -- 172.21.15.37:0/2669082915 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f644c018030 con 0x7f645c09d320 2024-04-26T00:32:07.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.196+0000 7f6451ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:07.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.196+0000 7f6451ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:32:07.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.196+0000 7f6451ffb640 10 monclient: dump: 2024-04-26T00:32:07.200 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:07.200 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:07.200 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.200 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.200 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:07.200 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:07.200 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:07.200 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:07.201 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:07.201 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:07.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.196+0000 7f6451ffb640 1 -- 172.21.15.37:0/2669082915 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4889 (secure 0 0 0) 0x7f644c00ecb0 con 0x7f645c09d320 2024-04-26T00:32:07.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.196+0000 7f6451ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:32:07.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.196+0000 7f6451ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:32:07.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.196+0000 7f6451ffb640 1 -- 172.21.15.37:0/2669082915 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f645c09d320 msgr2=0x7f645c09d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:07.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.196+0000 7f6451ffb640 1 --2- 172.21.15.37:0/2669082915 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f645c09d320 0x7f645c09d730 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f644c010780 tx=0x7f644c0046d0 comp rx=0 tx=0).stop 2024-04-26T00:32:07.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.196+0000 7f6451ffb640 1 -- 172.21.15.37:0/2669082915 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f645c09d320 msgr2=0x7f645c09d730 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:32:07.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.196+0000 7f6451ffb640 1 --2- 172.21.15.37:0/2669082915 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f645c09d320 0x7f645c09d730 secure :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f644c010780 tx=0x7f644c0046d0 comp rx=0 tx=0).stop 2024-04-26T00:32:07.201 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:32:07.201 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:32:07.202 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:32:07.202 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:32:07.202 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 92, 2024-04-26T00:32:07.202 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 332131, 2024-04-26T00:32:07.202 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:32:07.202 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:32:07.202 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:32:07.202 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:32:07.202 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:07.202 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 7, 2024-04-26T00:32:07.202 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:32:07.202 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:32:07.202 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:32:07.203 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:07.203 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:07.203 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:07.203 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:07.203 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:07.203 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:07.203 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:07.203 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:07.203 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:07.203 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:07.203 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:07.203 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:07.203 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:32:07.203 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:32:07.204 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:07.204 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:07.204 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:07.204 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:07.204 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:07.204 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:07.204 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:07.204 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:07.204 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:07.204 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:07.204 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:07.204 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:07.204 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:07.204 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:32:07.205 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:32:07.205 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:32:07.205 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:32:07.205 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:32:07.205 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:32:07.205 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:32:07.205 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:32:07.205 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:32:07.205 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:32:07.205 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:32:07.205 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:32:07.205 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:32:07.205 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:32:07.205 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:32:07.205 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:32:07.206 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:32:07.206 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:07.206 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:07.206 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:07.206 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:07.206 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:07.206 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:07.206 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:07.206 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:07.206 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:07.206 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:07.206 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:07.206 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:07.207 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:32:07.207 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:07.207 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:32:07.207 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:07.207 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:32:07.207 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:32:07.207 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:07.207 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:07.207 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:07.207 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:07.207 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:32:07.207 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:07.207 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:07.207 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:07.207 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:07.208 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:32:07.208 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:07.208 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:07.208 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:07.208 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:07.208 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:32:07.208 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:32:07.208 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:07.208 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:07.208 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:07.208 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:07.208 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:07.208 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:32:07.208 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:32:07.208 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:07.209 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:07.209 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:07.209 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:07.209 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:32:07.209 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:07.209 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:07.209 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:07.209 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:07.209 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:32:07.209 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:07.209 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:07.209 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:07.209 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:07.209 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:32:07.210 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:32:07.210 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:07.210 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:07.210 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:07.210 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:07.210 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:07.210 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:32:07.210 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:32:07.210 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:07.210 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:07.210 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:07.210 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:07.210 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:32:07.211 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:07.211 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:07.211 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:07.211 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:07.211 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:32:07.211 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:07.211 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:07.211 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:07.211 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:07.211 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:32:07.211 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:32:07.211 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:07.211 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:07.211 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:07.212 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:07.212 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:07.212 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:07.212 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:32:07.212 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:32:07.212 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:07.212 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:07.212 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:07.212 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:32:07.212 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:07.212 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:07.212 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:32:07.212 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:07.212 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:07.213 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:07.213 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:32:07.213 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:07.213 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:07.213 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:32:07.213 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:07.213 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:07.213 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:07.213 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:32:07.213 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:07.213 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:07.213 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:32:07.213 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:07.213 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:07.214 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:07.214 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:32:07.214 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:07.214 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:07.214 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:07.214 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:32:07.214 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:32:07.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.196+0000 7f6463555640 1 -- 172.21.15.37:0/2669082915 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6440035ee0 msgr2=0x7f64400383a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:07.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.196+0000 7f6463555640 1 --2- 172.21.15.37:0/2669082915 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6440035ee0 0x7f64400383a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f6458009630 tx=0x7f6458008040 comp rx=0 tx=0).stop 2024-04-26T00:32:07.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.196+0000 7f6463555640 10 monclient: shutdown 2024-04-26T00:32:07.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.196+0000 7f6463555640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:07.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.196+0000 7f6463555640 1 -- 172.21.15.37:0/2669082915 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f645c137300 msgr2=0x7f645c1cee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:07.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.196+0000 7f6463555640 1 --2- 172.21.15.37:0/2669082915 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f645c137300 0x7f645c1cee30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f64540b0040 tx=0x7f6454019590 comp rx=0 tx=0).stop 2024-04-26T00:32:07.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.196+0000 7f6461af2640 1 -- 172.21.15.37:0/2669082915 reap_dead start 2024-04-26T00:32:07.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.196+0000 7f6461af2640 1 -- 172.21.15.37:0/2669082915 reap_dead start 2024-04-26T00:32:07.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.196+0000 7f6463555640 1 -- 172.21.15.37:0/2669082915 shutdown_connections 2024-04-26T00:32:07.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.196+0000 7f6463555640 1 -- 172.21.15.37:0/2669082915 >> 172.21.15.37:0/2669082915 conn(0x7f645c0a61b0 msgr2=0x7f645c0a28f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:07.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.196+0000 7f6463555640 1 -- 172.21.15.37:0/2669082915 shutdown_connections 2024-04-26T00:32:07.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.196+0000 7f6463555640 1 -- 172.21.15.37:0/2669082915 wait complete. 2024-04-26T00:32:07.217 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:32:07.217 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:32:07.217 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:32:07.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.336+0000 7f91c6086640 10 monclient: get_monmap_and_config 2024-04-26T00:32:07.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.336+0000 7f91c6086640 10 monclient: build_initial_monmap 2024-04-26T00:32:07.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.336+0000 7f91c6086640 10 monclient: monmap: 2024-04-26T00:32:07.332 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:07.332 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:07.332 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:32:07.342123+0000 2024-04-26T00:32:07.332 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:32:07.342123+0000 2024-04-26T00:32:07.332 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:07.332 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:07.332 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:32:07.332 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:32:07.332 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:32:07.333 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:32:07.333 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:32:07.333 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:07.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.336+0000 7f91c6086640 1 Processor -- start 2024-04-26T00:32:07.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.336+0000 7f91c6086640 1 -- start start 2024-04-26T00:32:07.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.336+0000 7f91c6086640 10 monclient: init 2024-04-26T00:32:07.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.336+0000 7f91c6086640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:07.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.336+0000 7f91c6086640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-04-26T00:32:07.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.336+0000 7f91c6086640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f91c00a3d90 0x7f91c009bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:07.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.336+0000 7f91c6086640 10 monclient(hunting): picked mon.noname-a con 0x7f91c00a3d90 addr v2:172.21.15.37:3300/0 2024-04-26T00:32:07.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.336+0000 7f91c6086640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f91c00a4170 addr v1:172.21.15.45:6789/0 2024-04-26T00:32:07.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.336+0000 7f91c6086640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f91c013c3e0 addr v1:172.21.15.37:6789/0 2024-04-26T00:32:07.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.336+0000 7f91c6086640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91c00a5150 con 0x7f91c013c3e0 2024-04-26T00:32:07.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.336+0000 7f91c6086640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91c009b280 con 0x7f91c00a4170 2024-04-26T00:32:07.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.336+0000 7f91c6086640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:07.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.336+0000 7f91c6086640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f91c009b560 con 0x7f91c00a3d90 2024-04-26T00:32:07.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.336+0000 7f91c6086640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:07.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.336+0000 7f91c6086640 10 monclient(hunting): authenticate will time out at 1708.647757s 2024-04-26T00:32:07.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.336+0000 7f91bf7fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f91c00a3d90 0x7f91c009bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:07.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.336+0000 7f91bf7fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f91c00a3d90 0x7f91c009bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:35176/0 (socket says 172.21.15.37:35176) 2024-04-26T00:32:07.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.336+0000 7f91bffff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f91c013c3e0 0x7f91c013e7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:48666/0 (socket says 172.21.15.37:48666) 2024-04-26T00:32:07.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.336+0000 7f91bf7fe640 1 -- 172.21.15.37:0/2453300609 learned_addr learned my addr 172.21.15.37:0/2453300609 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:32:07.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.336+0000 7f91bf7fe640 10 monclient(hunting): get_auth_request con 0x7f91c00a3d90 auth_method 0 2024-04-26T00:32:07.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.336+0000 7f91bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:07.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.336+0000 7f91bf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:07.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.336+0000 7f91bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:07.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.340+0000 7f91bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:07.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.340+0000 7f91bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:07.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.340+0000 7f91bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:07.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.340+0000 7f91bf7fe640 10 monclient(hunting): handle_auth_done global_id 144141 payload 274 2024-04-26T00:32:07.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.340+0000 7f91bf7fe640 1 -- 172.21.15.37:0/2453300609 >> v1:172.21.15.45:6789/0 conn(0x7f91c00a4170 legacy=0x7f91c00a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:07.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.340+0000 7f91bf7fe640 1 -- 172.21.15.37:0/2453300609 >> v1:172.21.15.37:6789/0 conn(0x7f91c013c3e0 legacy=0x7f91c013e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:07.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.340+0000 7f91bf7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:32:07.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.340+0000 7f91bf7fe640 1 monclient: found mon.noname-a 2024-04-26T00:32:07.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.340+0000 7f91bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:07.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.340+0000 7f91bf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:32:07.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.340+0000 7f91bf7fe640 1 -- 172.21.15.37:0/2453300609 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91c006dc60 con 0x7f91c00a3d90 2024-04-26T00:32:07.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91bf7fe640 1 -- 172.21.15.37:0/2453300609 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91c00a3d90 msgr2=0x7f91c009bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:07.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91bf7fe640 1 --2- 172.21.15.37:0/2453300609 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91c00a3d90 0x7f91c009bf40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f91b8009e50 tx=0x7f91b801c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6f1cfe01f487f98 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:07.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91be7fc640 1 -- 172.21.15.37:0/2453300609 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b80a5640 con 0x7f91c00a3d90 2024-04-26T00:32:07.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91be7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:07.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91be7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:32:07.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91be7fc640 10 monclient: dump: 2024-04-26T00:32:07.338 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:07.338 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:07.338 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.338 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.339 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:07.339 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:07.339 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:07.339 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:07.339 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:07.339 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:07.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91be7fc640 10 monclient: _finish_auth 0 2024-04-26T00:32:07.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91be7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:32:07.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:07.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91be7fc640 1 -- 172.21.15.37:0/2453300609 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91b80a57e0 con 0x7f91c00a3d90 2024-04-26T00:32:07.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91be7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:07.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91be7fc640 1 -- 172.21.15.37:0/2453300609 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b80a5bb0 con 0x7f91c00a3d90 2024-04-26T00:32:07.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91c6086640 5 monclient: authenticate success, global_id 144141 2024-04-26T00:32:07.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91c6086640 10 monclient: get_monmap_and_config success 2024-04-26T00:32:07.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91c6086640 4 set_mon_vals no callback set 2024-04-26T00:32:07.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91be7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:07.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91be7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:07.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91be7fc640 10 monclient: dump: 2024-04-26T00:32:07.340 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:07.340 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:07.340 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.340 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.340 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:07.340 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:07.340 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:07.340 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:07.340 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:07.340 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:07.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91c6086640 10 monclient: shutdown 2024-04-26T00:32:07.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91c6086640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:07.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91c6086640 1 -- 172.21.15.37:0/2453300609 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91c00a3d90 msgr2=0x7f91c009bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:07.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91c6086640 1 --2- 172.21.15.37:0/2453300609 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91c00a3d90 0x7f91c009bf40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f91b8009e50 tx=0x7f91b801c870 comp rx=0 tx=0).stop 2024-04-26T00:32:07.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91c6086640 1 -- 172.21.15.37:0/2453300609 shutdown_connections 2024-04-26T00:32:07.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91c6086640 1 --2- 172.21.15.37:0/2453300609 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91c00a3d90 0x7f91c009bf40 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:07.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91c6086640 1 -- 172.21.15.37:0/2453300609 >> 172.21.15.37:0/2453300609 conn(0x7f91c00aa0a0 msgr2=0x7f91c00a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:07.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91c6086640 1 -- 172.21.15.37:0/2453300609 shutdown_connections 2024-04-26T00:32:07.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91c6086640 1 -- 172.21.15.37:0/2453300609 wait complete. 2024-04-26T00:32:07.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91c6086640 10 monclient: build_initial_monmap 2024-04-26T00:32:07.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91c6086640 10 monclient: monmap: 2024-04-26T00:32:07.346 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:07.346 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:32:07.346 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:32:07.346 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:32:07.347 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:07.347 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:32:07.347 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:32:07.347 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:32:07.347 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:32:07.347 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:07.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91c6086640 1 Processor -- start 2024-04-26T00:32:07.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91c6086640 1 -- start start 2024-04-26T00:32:07.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91c6086640 10 monclient: init 2024-04-26T00:32:07.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91c6086640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:07.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91c6086640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:32:07.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91c6086640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91c00a3d90 0x7f91c01cec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:07.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91c6086640 10 monclient(hunting): picked mon.noname-a con 0x7f91c00a3d90 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:32:07.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91c6086640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c00a4170 0x7f91c0136dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:07.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91c6086640 10 monclient(hunting): picked mon.noname-b con 0x7f91c00a4170 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:32:07.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91c6086640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91c013c3e0 0x7f91c0138d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:07.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91c6086640 10 monclient(hunting): picked mon.noname-c con 0x7f91c013c3e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:07.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91c6086640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:07.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91c6086640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f91c0142920 con 0x7f91c00a3d90 2024-04-26T00:32:07.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91c6086640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:07.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91c6086640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f91c009c690 con 0x7f91c00a4170 2024-04-26T00:32:07.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91c6086640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:07.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91c6086640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f91c0142aa0 con 0x7f91c013c3e0 2024-04-26T00:32:07.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91c6086640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:07.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91c6086640 10 monclient(hunting): authenticate will time out at 1708.654815s 2024-04-26T00:32:07.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91bf7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91c00a3d90 0x7f91c01cec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:07.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91bf7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91c00a3d90 0x7f91c01cec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:35188/0 (socket says 172.21.15.37:35188) 2024-04-26T00:32:07.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91bf7fe640 1 -- 172.21.15.37:0/2179762652 learned_addr learned my addr 172.21.15.37:0/2179762652 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:32:07.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91bf7fe640 10 monclient(hunting): get_auth_request con 0x7f91c00a3d90 auth_method 0 2024-04-26T00:32:07.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:07.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91bf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:07.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:07.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91bffff640 1 --2- 172.21.15.37:0/2179762652 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91c013c3e0 0x7f91c0138d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:07.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91bffff640 10 monclient(hunting): get_auth_request con 0x7f91c013c3e0 auth_method 0 2024-04-26T00:32:07.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:07.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91bffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:07.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91bffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:07.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:07.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:07.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:07.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:07.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:07.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:07.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91bf7fe640 10 monclient(hunting): handle_auth_done global_id 144144 payload 824 2024-04-26T00:32:07.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91beffd640 1 --2- 172.21.15.37:0/2179762652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c00a4170 0x7f91c0136dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:07.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91bf7fe640 1 -- 172.21.15.37:0/2179762652 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91c013c3e0 msgr2=0x7f91c0138d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:07.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91bf7fe640 1 --2- 172.21.15.37:0/2179762652 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91c013c3e0 0x7f91c0138d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:07.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91bf7fe640 1 -- 172.21.15.37:0/2179762652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c00a4170 msgr2=0x7f91c0136dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:07.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91bf7fe640 1 --2- 172.21.15.37:0/2179762652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c00a4170 0x7f91c0136dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:07.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91bf7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:32:07.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91bf7fe640 1 monclient: found mon.noname-a 2024-04-26T00:32:07.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:07.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91bf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:32:07.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91bf7fe640 1 -- 172.21.15.37:0/2179762652 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91c013a9e0 con 0x7f91c00a3d90 2024-04-26T00:32:07.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91bffff640 1 --2- 172.21.15.37:0/2179762652 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91c013c3e0 0x7f91c0138d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:32:07.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91beffd640 10 monclient: get_auth_request con 0x7f91c00a4170 auth_method 0 2024-04-26T00:32:07.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.344+0000 7f91beffd640 1 --2- 172.21.15.37:0/2179762652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c00a4170 0x7f91c0136dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:32:07.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91bf7fe640 1 -- 172.21.15.37:0/2179762652 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91c00a3d90 msgr2=0x7f91c01cec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:07.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91bf7fe640 1 --2- 172.21.15.37:0/2179762652 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91c00a3d90 0x7f91c01cec00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f91b801a4a0 tx=0x7f91b8019610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:07.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91a7fff640 1 -- 172.21.15.37:0/2179762652 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b80bc020 con 0x7f91c00a3d90 2024-04-26T00:32:07.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:07.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91a7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:32:07.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91a7fff640 10 monclient: dump: 2024-04-26T00:32:07.351 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:07.351 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:07.351 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.351 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.351 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:07.351 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:07.351 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:07.351 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:07.351 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:07.351 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:07.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91a7fff640 10 monclient: _finish_auth 0 2024-04-26T00:32:07.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91a7fff640 10 monclient: _check_auth_tickets 2024-04-26T00:32:07.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:07.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91a7fff640 1 -- 172.21.15.37:0/2179762652 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91b80b8a90 con 0x7f91c00a3d90 2024-04-26T00:32:07.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91a7fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:07.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91a7fff640 1 -- 172.21.15.37:0/2179762652 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b80a5640 con 0x7f91c00a3d90 2024-04-26T00:32:07.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91c6086640 5 monclient: authenticate success, global_id 144144 2024-04-26T00:32:07.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91c6086640 10 monclient: _renew_subs 2024-04-26T00:32:07.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91c6086640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:07.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91c6086640 1 -- 172.21.15.37:0/2179762652 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91c009ada0 con 0x7f91c00a3d90 2024-04-26T00:32:07.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91bdffb640 4 set_mon_vals no callback set 2024-04-26T00:32:07.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:07.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91a7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:07.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91a7fff640 10 monclient: dump: 2024-04-26T00:32:07.352 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:07.353 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:07.353 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.353 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.353 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:07.353 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:07.353 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:07.353 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:07.353 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:07.353 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:07.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91c6086640 10 monclient: _renew_subs 2024-04-26T00:32:07.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91c6086640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:07.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91c6086640 1 -- 172.21.15.37:0/2179762652 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91c0120740 con 0x7f91c00a3d90 2024-04-26T00:32:07.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91c6086640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:07.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91c6086640 1 --2- 172.21.15.37:0/2179762652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c01392d0 0x7f91c01396b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:07.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91c6086640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:07.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91c6086640 1 -- 172.21.15.37:0/2179762652 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f91c0139bf0 con 0x7f91c01392d0 2024-04-26T00:32:07.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91a7fff640 1 -- 172.21.15.37:0/2179762652 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f91b80bf020 con 0x7f91c00a3d90 2024-04-26T00:32:07.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91a7fff640 1 --2- 172.21.15.37:0/2179762652 >> v2:172.21.15.45:6812/2379918623 conn(0x7f91ac035ed0 0x7f91ac038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:07.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91beffd640 1 --2- 172.21.15.37:0/2179762652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c01392d0 0x7f91c01396b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:07.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91a7fff640 1 -- 172.21.15.37:0/2179762652 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(469..469 src has 370..469) ==== 3943+0+0 (secure 0 0 0) 0x7f91b80e3bc0 con 0x7f91c00a3d90 2024-04-26T00:32:07.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91beffd640 10 monclient: get_auth_request con 0x7f91c01392d0 auth_method 0 2024-04-26T00:32:07.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91beffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:07.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91beffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:07.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91beffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:07.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91beffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:07.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91beffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:07.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91beffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:07.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91bffff640 1 --2- 172.21.15.37:0/2179762652 >> v2:172.21.15.45:6812/2379918623 conn(0x7f91ac035ed0 0x7f91ac038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:07.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91bffff640 10 monclient: get_auth_request con 0x7f91ac035ed0 auth_method 0 2024-04-26T00:32:07.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91beffd640 10 monclient(hunting): handle_auth_done global_id 134221 payload 824 2024-04-26T00:32:07.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.348+0000 7f91beffd640 1 -- 172.21.15.37:0/2179762652 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f91c004f510 con 0x7f91c01392d0 2024-04-26T00:32:07.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.352+0000 7f91bffff640 1 --2- 172.21.15.37:0/2179762652 >> v2:172.21.15.45:6812/2379918623 conn(0x7f91ac035ed0 0x7f91ac038390 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f91b0006ec0 tx=0x7f91b0007df0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:07.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.352+0000 7f91beffd640 1 -- 172.21.15.37:0/2179762652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c01392d0 msgr2=0x7f91c01396b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:07.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.352+0000 7f91beffd640 1 --2- 172.21.15.37:0/2179762652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c01392d0 0x7f91c01396b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f91c013a5e0 tx=0x7f91b400d8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:07.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.352+0000 7f91a7fff640 1 -- 172.21.15.37:0/2179762652 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b4012a10 con 0x7f91c01392d0 2024-04-26T00:32:07.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.352+0000 7f91a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:07.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.352+0000 7f91a7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:32:07.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.352+0000 7f91a7fff640 10 monclient: dump: 2024-04-26T00:32:07.357 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:07.357 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:07.357 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.357 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.357 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:07.357 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:07.357 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:07.357 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:07.357 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:07.357 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:07.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.352+0000 7f91a7fff640 1 -- 172.21.15.37:0/2179762652 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f91b4012a10 con 0x7f91c01392d0 2024-04-26T00:32:07.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.352+0000 7f91a7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:07.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.352+0000 7f91a7fff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:32:07.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.352+0000 7f91a7fff640 1 -- 172.21.15.37:0/2179762652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c01392d0 msgr2=0x7f91c01396b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:07.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.352+0000 7f91a7fff640 1 --2- 172.21.15.37:0/2179762652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c01392d0 0x7f91c01396b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f91c013a5e0 tx=0x7f91b400d8e0 comp rx=0 tx=0).stop 2024-04-26T00:32:07.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.352+0000 7f91a7fff640 1 -- 172.21.15.37:0/2179762652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c01392d0 msgr2=0x7f91c01396b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:32:07.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.352+0000 7f91a7fff640 1 --2- 172.21.15.37:0/2179762652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c01392d0 0x7f91c01396b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:07.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.360+0000 7f91c6086640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:32:07.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.360+0000 7f91c6086640 1 --2- 172.21.15.37:0/2179762652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c009d070 0x7f91c009d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:07.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.360+0000 7f91c6086640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:07.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.360+0000 7f91c6086640 1 -- 172.21.15.37:0/2179762652 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f91c013bc90 con 0x7f91c009d070 2024-04-26T00:32:07.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.360+0000 7f91beffd640 1 --2- 172.21.15.37:0/2179762652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c009d070 0x7f91c009d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:07.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.360+0000 7f91beffd640 10 monclient: get_auth_request con 0x7f91c009d070 auth_method 0 2024-04-26T00:32:07.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.360+0000 7f91beffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:07.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.360+0000 7f91beffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:07.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.360+0000 7f91beffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:07.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.360+0000 7f91beffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:07.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.360+0000 7f91beffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:07.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.360+0000 7f91beffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:07.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.360+0000 7f91beffd640 10 monclient(hunting): handle_auth_done global_id 134224 payload 824 2024-04-26T00:32:07.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.360+0000 7f91beffd640 1 -- 172.21.15.37:0/2179762652 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f91c0120260 con 0x7f91c009d070 2024-04-26T00:32:07.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.364+0000 7f91beffd640 1 -- 172.21.15.37:0/2179762652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c009d070 msgr2=0x7f91c009d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:07.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.364+0000 7f91beffd640 1 --2- 172.21.15.37:0/2179762652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c009d070 0x7f91c009d480 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f91b4017000 tx=0x7f91b4003770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:07.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.364+0000 7f91a7fff640 1 -- 172.21.15.37:0/2179762652 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91b40243f0 con 0x7f91c009d070 2024-04-26T00:32:07.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.364+0000 7f91a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:07.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.364+0000 7f91a7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:32:07.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.364+0000 7f91a7fff640 10 monclient: dump: 2024-04-26T00:32:07.360 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:07.360 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:07.360 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.361 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.361 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:07.361 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:07.361 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:07.361 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:07.361 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:07.361 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:07.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.364+0000 7f91a7fff640 1 -- 172.21.15.37:0/2179762652 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4716 (secure 0 0 0) 0x7f91b40243f0 con 0x7f91c009d070 2024-04-26T00:32:07.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.364+0000 7f91a7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:32:07.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.364+0000 7f91a7fff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:32:07.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.364+0000 7f91a7fff640 1 -- 172.21.15.37:0/2179762652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c009d070 msgr2=0x7f91c009d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:07.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.364+0000 7f91a7fff640 1 --2- 172.21.15.37:0/2179762652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c009d070 0x7f91c009d480 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f91b4017000 tx=0x7f91b4003770 comp rx=0 tx=0).stop 2024-04-26T00:32:07.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.364+0000 7f91a7fff640 1 -- 172.21.15.37:0/2179762652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c009d070 msgr2=0x7f91c009d480 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:32:07.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.364+0000 7f91a7fff640 1 --2- 172.21.15.37:0/2179762652 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91c009d070 0x7f91c009d480 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f91b4017000 tx=0x7f91b4003770 comp rx=0 tx=0).stop 2024-04-26T00:32:07.362 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:32:07.362 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:32:07.362 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:32:07.362 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:32:07.362 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 92, 2024-04-26T00:32:07.362 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 57723, 2024-04-26T00:32:07.362 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:32:07.362 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:32:07.362 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:32:07.362 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:32:07.362 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:07.363 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 7, 2024-04-26T00:32:07.363 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:32:07.363 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:32:07.363 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:32:07.363 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:07.363 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:07.363 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:07.363 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:07.363 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:07.363 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:07.363 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:07.363 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:07.363 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:07.363 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:07.363 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:07.364 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:07.364 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:32:07.364 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:32:07.364 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:07.364 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:07.364 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:07.364 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:07.364 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:07.364 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:07.364 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:07.364 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:07.364 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:07.364 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:07.364 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:07.365 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:07.365 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:07.365 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:32:07.365 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:32:07.365 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:32:07.365 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:32:07.365 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:32:07.365 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:32:07.365 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:32:07.365 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:32:07.365 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:32:07.365 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:32:07.365 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:32:07.365 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:32:07.365 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:32:07.366 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:32:07.366 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:32:07.366 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:32:07.366 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:32:07.366 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:07.366 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:07.366 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:07.366 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:07.366 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:07.366 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:07.366 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:07.366 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:07.366 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:07.366 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:07.367 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:07.367 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:07.367 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:32:07.367 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:07.367 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:32:07.367 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:07.367 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:32:07.367 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:32:07.367 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:07.367 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:07.367 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:07.367 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:07.367 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:32:07.368 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:07.368 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:07.368 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:07.368 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:07.368 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:32:07.368 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:07.368 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:07.368 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:07.368 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:07.368 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:32:07.368 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:32:07.368 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:07.368 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:07.369 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:07.369 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:07.369 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:07.369 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:32:07.369 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:32:07.369 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:07.369 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:07.369 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:07.369 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:07.369 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:32:07.369 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:07.369 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:07.369 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:07.369 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:07.370 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:32:07.370 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:07.370 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:07.370 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:07.370 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:07.370 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:32:07.370 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:32:07.370 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:07.370 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:07.370 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:07.370 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:07.370 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:07.370 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:32:07.371 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:32:07.371 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:07.371 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:07.371 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:07.371 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:07.371 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:32:07.371 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:07.371 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:07.371 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:07.371 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:07.371 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:32:07.371 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:07.371 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:07.371 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:07.372 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:07.372 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:32:07.372 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:32:07.372 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:07.372 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:07.372 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:07.372 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:07.372 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:07.372 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:07.372 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:32:07.372 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:32:07.372 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:07.372 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:07.373 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:07.373 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:32:07.373 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:07.373 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:07.373 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:32:07.373 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:07.373 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:07.373 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:07.373 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:32:07.373 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:07.373 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:07.373 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:32:07.373 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:07.373 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:07.374 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:07.374 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:32:07.374 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:07.374 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:07.374 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:07.374 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:32:07.374 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:32:07.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.364+0000 7f91c6086640 1 -- 172.21.15.37:0/2179762652 >> v2:172.21.15.45:6812/2379918623 conn(0x7f91ac035ed0 msgr2=0x7f91ac038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:07.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.364+0000 7f91c6086640 1 --2- 172.21.15.37:0/2179762652 >> v2:172.21.15.45:6812/2379918623 conn(0x7f91ac035ed0 0x7f91ac038390 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f91b0006ec0 tx=0x7f91b0007df0 comp rx=0 tx=0).stop 2024-04-26T00:32:07.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.364+0000 7f91c6086640 10 monclient: shutdown 2024-04-26T00:32:07.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.364+0000 7f91c6086640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:07.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.364+0000 7f91c6086640 1 -- 172.21.15.37:0/2179762652 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91c00a3d90 msgr2=0x7f91c01cec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:07.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.364+0000 7f91c6086640 1 --2- 172.21.15.37:0/2179762652 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91c00a3d90 0x7f91c01cec00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f91b801a4a0 tx=0x7f91b8019610 comp rx=0 tx=0).stop 2024-04-26T00:32:07.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.364+0000 7f91bffff640 1 -- 172.21.15.37:0/2179762652 reap_dead start 2024-04-26T00:32:07.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.364+0000 7f91bffff640 1 -- 172.21.15.37:0/2179762652 reap_dead start 2024-04-26T00:32:07.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.364+0000 7f91c6086640 1 -- 172.21.15.37:0/2179762652 shutdown_connections 2024-04-26T00:32:07.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.364+0000 7f91c6086640 1 -- 172.21.15.37:0/2179762652 >> 172.21.15.37:0/2179762652 conn(0x7f91c00aa0a0 msgr2=0x7f91c00a2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:07.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.364+0000 7f91c6086640 1 -- 172.21.15.37:0/2179762652 shutdown_connections 2024-04-26T00:32:07.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.364+0000 7f91c6086640 1 -- 172.21.15.37:0/2179762652 wait complete. 2024-04-26T00:32:07.376 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:32:07.377 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-04-26T00:32:07.377 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-26T00:32:07.377 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-04-26T00:32:07.377 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-04-26T00:32:07.377 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-26T00:32:07.377 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T00:32:07.391 INFO:tasks.ceph.mon.c:Stopped 2024-04-26T00:32:07.391 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-26T00:32:07.391 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:32:07.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.552+0000 7f073714f640 10 monclient: get_monmap_and_config 2024-04-26T00:32:07.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.552+0000 7f073714f640 10 monclient: build_initial_monmap 2024-04-26T00:32:07.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.552+0000 7f073714f640 10 monclient: monmap: 2024-04-26T00:32:07.548 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:07.548 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:07.548 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:32:07.557874+0000 2024-04-26T00:32:07.548 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:32:07.557874+0000 2024-04-26T00:32:07.548 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:07.548 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:07.548 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:32:07.548 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:32:07.548 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:32:07.548 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:32:07.548 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:32:07.549 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:07.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.552+0000 7f073714f640 1 Processor -- start 2024-04-26T00:32:07.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.552+0000 7f073714f640 1 -- start start 2024-04-26T00:32:07.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.552+0000 7f073714f640 10 monclient: init 2024-04-26T00:32:07.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.552+0000 7f073714f640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:07.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.552+0000 7f073714f640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-04-26T00:32:07.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.552+0000 7f073714f640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f07300a0260 0x7f07300a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:07.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.552+0000 7f073714f640 10 monclient(hunting): picked mon.noname-a con 0x7f07300a0260 addr v2:172.21.15.37:3300/0 2024-04-26T00:32:07.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.552+0000 7f073714f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0730134b30 addr v1:172.21.15.37:6789/0 2024-04-26T00:32:07.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.552+0000 7f073714f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0730135660 addr v1:172.21.15.45:6789/0 2024-04-26T00:32:07.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.552+0000 7f073714f640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07300a1130 con 0x7f0730134b30 2024-04-26T00:32:07.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.552+0000 7f073714f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f073006cff0 con 0x7f0730135660 2024-04-26T00:32:07.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.552+0000 7f073714f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:07.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.552+0000 7f073714f640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f073009ec10 con 0x7f07300a0260 2024-04-26T00:32:07.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.552+0000 7f073714f640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:07.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.552+0000 7f073714f640 10 monclient(hunting): authenticate will time out at 1708.863545s 2024-04-26T00:32:07.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.552+0000 7f072ffff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f0730134b30 0x7f0730134f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:48680/0 (socket says 172.21.15.37:48680) 2024-04-26T00:32:07.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.552+0000 7f0734eeb640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f07300a0260 0x7f07300a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:07.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.552+0000 7f072ffff640 1 -- 172.21.15.37:0/3165645153 learned_addr learned my addr 172.21.15.37:0/3165645153 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:32:07.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.552+0000 7f0734eeb640 10 monclient(hunting): get_auth_request con 0x7f07300a0260 auth_method 0 2024-04-26T00:32:07.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.552+0000 7f0734eeb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:07.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.552+0000 7f0734eeb640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:07.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.552+0000 7f0734eeb640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:07.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.552+0000 7f0734eeb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:07.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.552+0000 7f0734eeb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:07.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.552+0000 7f0734eeb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:07.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.556+0000 7f0734eeb640 10 monclient(hunting): handle_auth_done global_id 144147 payload 274 2024-04-26T00:32:07.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.556+0000 7f0734eeb640 1 -- 172.21.15.37:0/3165645153 >> v1:172.21.15.45:6789/0 conn(0x7f0730135660 legacy=0x7f073013ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:07.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.556+0000 7f0734eeb640 1 -- 172.21.15.37:0/3165645153 >> v1:172.21.15.37:6789/0 conn(0x7f0730134b30 legacy=0x7f0730134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:07.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.556+0000 7f0734eeb640 10 monclient: _finish_hunting 0 2024-04-26T00:32:07.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.556+0000 7f0734eeb640 1 monclient: found mon.noname-a 2024-04-26T00:32:07.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.556+0000 7f0734eeb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:07.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.556+0000 7f0734eeb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:32:07.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.556+0000 7f0734eeb640 1 -- 172.21.15.37:0/3165645153 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f073009bc50 con 0x7f07300a0260 2024-04-26T00:32:07.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f0734eeb640 1 -- 172.21.15.37:0/3165645153 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07300a0260 msgr2=0x7f07300a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:07.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f0734eeb640 1 --2- 172.21.15.37:0/3165645153 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07300a0260 0x7f07300a0640 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0728009d90 tx=0x7f072801c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97f2459b65a918da server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:07.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f072f7fe640 1 -- 172.21.15.37:0/3165645153 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07280a6040 con 0x7f07300a0260 2024-04-26T00:32:07.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f072f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:07.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f072f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:32:07.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f072f7fe640 10 monclient: dump: 2024-04-26T00:32:07.554 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:07.554 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:07.554 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.554 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.555 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:07.555 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:07.555 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:07.555 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:07.555 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:07.555 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:07.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f072f7fe640 10 monclient: _finish_auth 0 2024-04-26T00:32:07.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f072f7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:32:07.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f072f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:07.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f073714f640 5 monclient: authenticate success, global_id 144147 2024-04-26T00:32:07.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f073714f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:32:07.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f072f7fe640 1 -- 172.21.15.37:0/3165645153 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07280af070 con 0x7f07300a0260 2024-04-26T00:32:07.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f072f7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:07.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f073714f640 10 monclient: get_monmap_and_config success 2024-04-26T00:32:07.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f072f7fe640 1 -- 172.21.15.37:0/3165645153 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07280aa940 con 0x7f07300a0260 2024-04-26T00:32:07.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f073714f640 4 set_mon_vals no callback set 2024-04-26T00:32:07.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f072f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:07.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f072f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:07.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f072f7fe640 10 monclient: dump: 2024-04-26T00:32:07.556 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:07.556 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:07.556 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.556 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.556 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:07.556 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:07.556 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:07.556 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:07.556 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:07.557 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:07.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f073714f640 10 monclient: shutdown 2024-04-26T00:32:07.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f073714f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:07.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f073714f640 1 -- 172.21.15.37:0/3165645153 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07300a0260 msgr2=0x7f07300a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:07.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f073714f640 1 --2- 172.21.15.37:0/3165645153 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07300a0260 0x7f07300a0640 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0728009d90 tx=0x7f072801c5d0 comp rx=0 tx=0).stop 2024-04-26T00:32:07.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f073714f640 1 -- 172.21.15.37:0/3165645153 shutdown_connections 2024-04-26T00:32:07.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f073714f640 1 --2- 172.21.15.37:0/3165645153 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07300a0260 0x7f07300a0640 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:07.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f073714f640 1 -- 172.21.15.37:0/3165645153 >> 172.21.15.37:0/3165645153 conn(0x7f07300a1ad0 msgr2=0x7f07300a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:07.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f073714f640 1 -- 172.21.15.37:0/3165645153 shutdown_connections 2024-04-26T00:32:07.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f073714f640 1 -- 172.21.15.37:0/3165645153 wait complete. 2024-04-26T00:32:07.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f073714f640 10 monclient: build_initial_monmap 2024-04-26T00:32:07.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f073714f640 10 monclient: monmap: 2024-04-26T00:32:07.563 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:07.563 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:32:07.563 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:32:07.563 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:32:07.563 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:07.563 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:32:07.563 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:32:07.563 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:32:07.564 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:32:07.564 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:07.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f073714f640 1 Processor -- start 2024-04-26T00:32:07.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f073714f640 1 -- start start 2024-04-26T00:32:07.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f073714f640 10 monclient: init 2024-04-26T00:32:07.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f073714f640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:07.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f073714f640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:32:07.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f073714f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07300a0260 0x7f0730130f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:07.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f073714f640 10 monclient(hunting): picked mon.noname-a con 0x7f07300a0260 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:32:07.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f073714f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0730134b30 0x7f0730131460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:07.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f073714f640 10 monclient(hunting): picked mon.noname-b con 0x7f0730134b30 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:32:07.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f073714f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0730135660 0x7f0730139510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:07.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f073714f640 10 monclient(hunting): picked mon.noname-c con 0x7f0730135660 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:07.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f073714f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:07.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f073714f640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f073009f4c0 con 0x7f07300a0260 2024-04-26T00:32:07.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f073714f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:07.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f073714f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f073009f0f0 con 0x7f0730134b30 2024-04-26T00:32:07.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f073714f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:07.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f073714f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0730140db0 con 0x7f0730135660 2024-04-26T00:32:07.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f073714f640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:07.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f073714f640 10 monclient(hunting): authenticate will time out at 1708.871318s 2024-04-26T00:32:07.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f07356ec640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0730135660 msgr2=0x7f0730139510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:32:07.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f07356ec640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0730135660 0x7f0730139510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:32:07.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f0734eeb640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07300a0260 0x7f0730130f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:07.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f0734eeb640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07300a0260 0x7f0730130f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:35226/0 (socket says 172.21.15.37:35226) 2024-04-26T00:32:07.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f0734eeb640 1 -- 172.21.15.37:0/497479700 learned_addr learned my addr 172.21.15.37:0/497479700 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:32:07.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f0734eeb640 10 monclient(hunting): get_auth_request con 0x7f07300a0260 auth_method 0 2024-04-26T00:32:07.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f0734eeb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:07.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f0734eeb640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:07.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f0734eeb640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:07.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f0734eeb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:07.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f0734eeb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:07.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f0734eeb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:07.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f0734eeb640 10 monclient(hunting): handle_auth_done global_id 144150 payload 824 2024-04-26T00:32:07.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f072ffff640 1 --2- 172.21.15.37:0/497479700 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0730134b30 0x7f0730131460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:07.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f0734eeb640 1 -- 172.21.15.37:0/497479700 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0730135660 msgr2=0x7f0730139510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:32:07.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f0734eeb640 1 --2- 172.21.15.37:0/497479700 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0730135660 0x7f0730139510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:07.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f0734eeb640 1 -- 172.21.15.37:0/497479700 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0730134b30 msgr2=0x7f0730131460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:07.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f0734eeb640 1 --2- 172.21.15.37:0/497479700 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0730134b30 0x7f0730131460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:07.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f0734eeb640 10 monclient: _finish_hunting 0 2024-04-26T00:32:07.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f0734eeb640 1 monclient: found mon.noname-a 2024-04-26T00:32:07.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f0734eeb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:07.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f0734eeb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:32:07.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.560+0000 7f0734eeb640 1 -- 172.21.15.37:0/497479700 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07301348c0 con 0x7f07300a0260 2024-04-26T00:32:07.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.564+0000 7f0734eeb640 1 -- 172.21.15.37:0/497479700 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07300a0260 msgr2=0x7f0730130f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:07.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.564+0000 7f0734eeb640 1 --2- 172.21.15.37:0/497479700 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07300a0260 0x7f0730130f20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f07280ac040 tx=0x7f0728019370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:07.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.564+0000 7f072d7fa640 1 -- 172.21.15.37:0/497479700 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07280c2070 con 0x7f07300a0260 2024-04-26T00:32:07.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.564+0000 7f072d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:07.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.564+0000 7f072d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:32:07.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.564+0000 7f072d7fa640 10 monclient: dump: 2024-04-26T00:32:07.567 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:07.567 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:07.567 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.567 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.568 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:07.568 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:07.568 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:07.568 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:07.568 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:07.568 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:07.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.564+0000 7f072d7fa640 10 monclient: _finish_auth 0 2024-04-26T00:32:07.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.564+0000 7f072d7fa640 10 monclient: _check_auth_tickets 2024-04-26T00:32:07.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.564+0000 7f072d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:07.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.564+0000 7f072d7fa640 1 -- 172.21.15.37:0/497479700 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07280aab80 con 0x7f07300a0260 2024-04-26T00:32:07.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.564+0000 7f073714f640 5 monclient: authenticate success, global_id 144150 2024-04-26T00:32:07.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.564+0000 7f073714f640 10 monclient: _renew_subs 2024-04-26T00:32:07.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.564+0000 7f073714f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:07.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.564+0000 7f073714f640 1 -- 172.21.15.37:0/497479700 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f073013d450 con 0x7f07300a0260 2024-04-26T00:32:07.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.564+0000 7f072d7fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:07.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.564+0000 7f072d7fa640 1 -- 172.21.15.37:0/497479700 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07280bd420 con 0x7f07300a0260 2024-04-26T00:32:07.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.564+0000 7f072d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:07.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.564+0000 7f072d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:07.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.564+0000 7f072d7fa640 10 monclient: dump: 2024-04-26T00:32:07.569 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:07.569 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:07.569 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.569 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:07.569 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:07.569 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:07.570 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:07.570 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:07.570 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:07.570 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:07.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.564+0000 7f072effd640 4 set_mon_vals no callback set 2024-04-26T00:32:07.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.564+0000 7f073714f640 10 monclient: _renew_subs 2024-04-26T00:32:07.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.564+0000 7f073714f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:07.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.564+0000 7f073714f640 1 -- 172.21.15.37:0/497479700 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f073013ce90 con 0x7f07300a0260 2024-04-26T00:32:07.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.564+0000 7f073714f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:07.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.564+0000 7f073714f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:07.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.564+0000 7f073714f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:07.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.564+0000 7f073714f640 1 -- 172.21.15.37:0/497479700 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0730066f40 con 0x7f07300a0260 2024-04-26T00:32:07.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.564+0000 7f072d7fa640 1 -- 172.21.15.37:0/497479700 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f07280b33f0 con 0x7f07300a0260 2024-04-26T00:32:07.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.564+0000 7f072d7fa640 1 --2- 172.21.15.37:0/497479700 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0710035e60 0x7f0710038320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:07.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.564+0000 7f072d7fa640 1 -- 172.21.15.37:0/497479700 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(469..469 src has 370..469) ==== 3943+0+0 (secure 0 0 0) 0x7f07280e2da0 con 0x7f07300a0260 2024-04-26T00:32:07.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.568+0000 7f072ffff640 1 --2- 172.21.15.37:0/497479700 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0710035e60 0x7f0710038320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:07.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.568+0000 7f072ffff640 10 monclient: get_auth_request con 0x7f0710035e60 auth_method 0 2024-04-26T00:32:07.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.568+0000 7f072ffff640 1 --2- 172.21.15.37:0/497479700 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0710035e60 0x7f0710038320 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f073013ad60 tx=0x7f0718009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:07.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.572+0000 7f072d7fa640 1 -- 172.21.15.37:0/497479700 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f07301348c0 con 0x7f07300a0260 2024-04-26T00:32:07.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.572+0000 7f072d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:07.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.572+0000 7f072d7fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:32:07.593 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-04-26T00:32:07.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.800+0000 7f073714f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:32:07.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.800+0000 7f073714f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:32:07.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.800+0000 7f073714f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:07.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.800+0000 7f073714f640 1 -- 172.21.15.37:0/497479700 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f07300008e0 con 0x7f07300a0260 2024-04-26T00:32:07.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.800+0000 7f072d7fa640 1 -- 172.21.15.37:0/497479700 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f07280b8070 con 0x7f07300a0260 2024-04-26T00:32:07.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.800+0000 7f072d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:32:07.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.800+0000 7f072d7fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:32:07.795 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":92,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:32:07.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.804+0000 7f073714f640 1 -- 172.21.15.37:0/497479700 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0710035e60 msgr2=0x7f0710038320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:07.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.804+0000 7f073714f640 1 --2- 172.21.15.37:0/497479700 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0710035e60 0x7f0710038320 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f073013ad60 tx=0x7f0718009040 comp rx=0 tx=0).stop 2024-04-26T00:32:07.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.804+0000 7f073714f640 10 monclient: shutdown 2024-04-26T00:32:07.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.804+0000 7f073714f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:07.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.804+0000 7f073714f640 1 -- 172.21.15.37:0/497479700 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07300a0260 msgr2=0x7f0730130f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:07.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.804+0000 7f073714f640 1 --2- 172.21.15.37:0/497479700 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07300a0260 0x7f0730130f20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f07280ac040 tx=0x7f0728019370 comp rx=0 tx=0).stop 2024-04-26T00:32:07.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.804+0000 7f073714f640 1 -- 172.21.15.37:0/497479700 shutdown_connections 2024-04-26T00:32:07.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.804+0000 7f073714f640 1 --2- 172.21.15.37:0/497479700 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0710035e60 0x7f0710038320 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:07.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.804+0000 7f073714f640 1 --2- 172.21.15.37:0/497479700 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0730135660 0x7f0730139510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:07.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.804+0000 7f073714f640 1 --2- 172.21.15.37:0/497479700 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0730134b30 0x7f0730131460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:07.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.804+0000 7f073714f640 1 --2- 172.21.15.37:0/497479700 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07300a0260 0x7f0730130f20 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:07.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.804+0000 7f073714f640 1 -- 172.21.15.37:0/497479700 >> 172.21.15.37:0/497479700 conn(0x7f07300a1ad0 msgr2=0x7f07300a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:07.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.804+0000 7f073714f640 1 -- 172.21.15.37:0/497479700 shutdown_connections 2024-04-26T00:32:07.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:07.804+0000 7f073714f640 1 -- 172.21.15.37:0/497479700 wait complete. 2024-04-26T00:32:10.816 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:32:10.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.956+0000 7fd0a0c0c640 10 monclient: get_monmap_and_config 2024-04-26T00:32:10.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.956+0000 7fd0a0c0c640 10 monclient: build_initial_monmap 2024-04-26T00:32:10.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.956+0000 7fd0a0c0c640 10 monclient: monmap: 2024-04-26T00:32:10.951 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:10.951 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:10.951 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:32:10.960969+0000 2024-04-26T00:32:10.951 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:32:10.960969+0000 2024-04-26T00:32:10.951 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:10.951 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:10.951 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:32:10.951 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:32:10.951 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:32:10.951 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:32:10.952 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:32:10.952 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:10.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.956+0000 7fd0a0c0c640 1 Processor -- start 2024-04-26T00:32:10.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.956+0000 7fd0a0c0c640 1 -- start start 2024-04-26T00:32:10.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.956+0000 7fd0a0c0c640 10 monclient: init 2024-04-26T00:32:10.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.956+0000 7fd0a0c0c640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:10.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.956+0000 7fd0a0c0c640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-04-26T00:32:10.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.956+0000 7fd0a0c0c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd09c0a0260 addr v1:172.21.15.37:6789/0 2024-04-26T00:32:10.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.956+0000 7fd0a0c0c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd09c137e40 0x7fd09c13a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:10.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.956+0000 7fd0a0c0c640 10 monclient(hunting): picked mon.noname-c con 0x7fd09c137e40 addr v2:172.21.15.45:3300/0 2024-04-26T00:32:10.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.956+0000 7fd0a0c0c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd09c13a770 addr v1:172.21.15.45:6789/0 2024-04-26T00:32:10.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.956+0000 7fd0a0c0c640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd09c0a1130 con 0x7fd09c0a0260 2024-04-26T00:32:10.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.956+0000 7fd0a0c0c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd09c06cff0 con 0x7fd09c13a770 2024-04-26T00:32:10.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.956+0000 7fd0a0c0c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:10.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.956+0000 7fd0a0c0c640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fd09c0a0d50 con 0x7fd09c137e40 2024-04-26T00:32:10.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.956+0000 7fd0a0c0c640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:10.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.956+0000 7fd0a0c0c640 10 monclient(hunting): authenticate will time out at 1712.266666s 2024-04-26T00:32:10.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.956+0000 7fd09a59c640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fd09c0a0260 0x7fd09c0a0640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:35344/0 (socket says 172.21.15.37:35344) 2024-04-26T00:32:10.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.956+0000 7fd09a59c640 1 -- 172.21.15.37:0/3450590409 learned_addr learned my addr 172.21.15.37:0/3450590409 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:32:10.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.956+0000 7fd099d9b640 1 --2- 172.21.15.37:0/3450590409 >> v2:172.21.15.45:3300/0 conn(0x7fd09c137e40 0x7fd09c13a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:10.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.956+0000 7fd099d9b640 10 monclient(hunting): get_auth_request con 0x7fd09c137e40 auth_method 0 2024-04-26T00:32:10.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.956+0000 7fd099d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:10.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.956+0000 7fd099d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:10.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.956+0000 7fd099d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:10.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.956+0000 7fd099d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:10.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.956+0000 7fd099d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:10.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.956+0000 7fd099d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:10.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.956+0000 7fd099d9b640 10 monclient(hunting): handle_auth_done global_id 134227 payload 274 2024-04-26T00:32:10.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.956+0000 7fd099d9b640 1 -- 172.21.15.37:0/3450590409 >> v1:172.21.15.45:6789/0 conn(0x7fd09c13a770 legacy=0x7fd09c13cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:10.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.956+0000 7fd099d9b640 1 -- 172.21.15.37:0/3450590409 >> v1:172.21.15.37:6789/0 conn(0x7fd09c0a0260 legacy=0x7fd09c0a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:10.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.956+0000 7fd099d9b640 10 monclient: _finish_hunting 0 2024-04-26T00:32:10.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.956+0000 7fd099d9b640 1 monclient: found mon.noname-c 2024-04-26T00:32:10.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.956+0000 7fd099d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:10.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.956+0000 7fd099d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:32:10.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.956+0000 7fd099d9b640 1 -- 172.21.15.37:0/3450590409 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd09c09bc50 con 0x7fd09c137e40 2024-04-26T00:32:10.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.964+0000 7fd099d9b640 1 -- 172.21.15.37:0/3450590409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd09c137e40 msgr2=0x7fd09c13a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:10.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.964+0000 7fd099d9b640 1 --2- 172.21.15.37:0/3450590409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd09c137e40 0x7fd09c13a230 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd0900097c0 tx=0x7fd09001c100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9bdc9ed8f3456fd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:10.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd09959a640 1 -- 172.21.15.37:0/3450590409 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0900b1070 con 0x7fd09c137e40 2024-04-26T00:32:10.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd09959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:10.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd09959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:32:10.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd09959a640 10 monclient: dump: 2024-04-26T00:32:10.962 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:10.962 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:10.962 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:10.962 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:10.963 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:10.963 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:10.963 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:10.963 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:10.963 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:10.963 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:10.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd09959a640 10 monclient: _finish_auth 0 2024-04-26T00:32:10.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd09959a640 10 monclient: _check_auth_tickets 2024-04-26T00:32:10.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd09959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:10.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd09959a640 1 -- 172.21.15.37:0/3450590409 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0900ac440 con 0x7fd09c137e40 2024-04-26T00:32:10.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd0a0c0c640 5 monclient: authenticate success, global_id 134227 2024-04-26T00:32:10.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd0a0c0c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:32:10.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd09959a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:10.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd0a0c0c640 10 monclient: get_monmap_and_config success 2024-04-26T00:32:10.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd0a0c0c640 4 set_mon_vals no callback set 2024-04-26T00:32:10.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd09959a640 1 -- 172.21.15.37:0/3450590409 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0900ac810 con 0x7fd09c137e40 2024-04-26T00:32:10.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd09959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:10.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd09959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:32:10.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd09959a640 10 monclient: dump: 2024-04-26T00:32:10.965 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:10.965 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:10.965 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:10.965 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:10.965 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:10.965 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:10.965 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:10.965 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:10.965 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:10.965 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:10.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd0a0c0c640 10 monclient: shutdown 2024-04-26T00:32:10.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd0a0c0c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:10.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd0a0c0c640 1 -- 172.21.15.37:0/3450590409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd09c137e40 msgr2=0x7fd09c13a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:10.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd0a0c0c640 1 --2- 172.21.15.37:0/3450590409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd09c137e40 0x7fd09c13a230 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd0900097c0 tx=0x7fd09001c100 comp rx=0 tx=0).stop 2024-04-26T00:32:10.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd0a0c0c640 1 -- 172.21.15.37:0/3450590409 shutdown_connections 2024-04-26T00:32:10.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd0a0c0c640 1 --2- 172.21.15.37:0/3450590409 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd09c137e40 0x7fd09c13a230 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:10.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd0a0c0c640 1 -- 172.21.15.37:0/3450590409 >> 172.21.15.37:0/3450590409 conn(0x7fd09c0a1ad0 msgr2=0x7fd09c0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:10.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd0a0c0c640 1 -- 172.21.15.37:0/3450590409 shutdown_connections 2024-04-26T00:32:10.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd0a0c0c640 1 -- 172.21.15.37:0/3450590409 wait complete. 2024-04-26T00:32:10.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd0a0c0c640 10 monclient: build_initial_monmap 2024-04-26T00:32:10.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd0a0c0c640 10 monclient: monmap: 2024-04-26T00:32:10.971 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:10.971 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:32:10.971 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:32:10.971 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:32:10.971 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:10.971 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:32:10.971 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:32:10.971 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:32:10.971 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:32:10.971 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:10.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd0a0c0c640 1 Processor -- start 2024-04-26T00:32:10.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd0a0c0c640 1 -- start start 2024-04-26T00:32:10.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd0a0c0c640 10 monclient: init 2024-04-26T00:32:10.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd0a0c0c640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:10.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd0a0c0c640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:32:10.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd0a0c0c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd09c0a0260 0x7fd09c134b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:10.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd0a0c0c640 10 monclient(hunting): picked mon.noname-b con 0x7fd09c0a0260 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:32:10.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd0a0c0c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd09c137e40 0x7fd09c135070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:10.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd0a0c0c640 10 monclient(hunting): picked mon.noname-a con 0x7fd09c137e40 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:32:10.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd0a0c0c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd09c13a770 0x7fd09c1cefe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:10.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd0a0c0c640 10 monclient(hunting): picked mon.noname-c con 0x7fd09c13a770 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:10.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd0a0c0c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:10.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd0a0c0c640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd09c09f0f0 con 0x7fd09c137e40 2024-04-26T00:32:10.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd0a0c0c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:10.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd0a0c0c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd09c09f4c0 con 0x7fd09c0a0260 2024-04-26T00:32:10.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd0a0c0c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:10.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd0a0c0c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd09c0a00a0 con 0x7fd09c13a770 2024-04-26T00:32:10.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd0a0c0c640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:10.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd0a0c0c640 10 monclient(hunting): authenticate will time out at 1712.279320s 2024-04-26T00:32:10.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd09ad9d640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd09c13a770 msgr2=0x7fd09c1cefe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:32:10.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd09ad9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd09c13a770 0x7fd09c1cefe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:32:10.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd099d9b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd09c137e40 0x7fd09c135070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:10.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd099d9b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd09c137e40 0x7fd09c135070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:46294/0 (socket says 172.21.15.37:46294) 2024-04-26T00:32:10.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd099d9b640 1 -- 172.21.15.37:0/905071269 learned_addr learned my addr 172.21.15.37:0/905071269 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:32:10.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd099d9b640 10 monclient(hunting): get_auth_request con 0x7fd09c137e40 auth_method 0 2024-04-26T00:32:10.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd099d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:10.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd099d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:10.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd099d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:10.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd099d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:10.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd099d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:10.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd099d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:10.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd099d9b640 10 monclient(hunting): handle_auth_done global_id 144153 payload 824 2024-04-26T00:32:10.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd09a59c640 1 --2- 172.21.15.37:0/905071269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd09c0a0260 0x7fd09c134b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:10.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd099d9b640 1 -- 172.21.15.37:0/905071269 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd09c13a770 msgr2=0x7fd09c1cefe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:32:10.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd099d9b640 1 --2- 172.21.15.37:0/905071269 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd09c13a770 0x7fd09c1cefe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:10.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd099d9b640 1 -- 172.21.15.37:0/905071269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd09c0a0260 msgr2=0x7fd09c134b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:10.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd099d9b640 1 --2- 172.21.15.37:0/905071269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd09c0a0260 0x7fd09c134b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:10.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd099d9b640 10 monclient: _finish_hunting 0 2024-04-26T00:32:10.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd099d9b640 1 monclient: found mon.noname-a 2024-04-26T00:32:10.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd099d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:10.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd099d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:32:10.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.968+0000 7fd099d9b640 1 -- 172.21.15.37:0/905071269 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd09c071d40 con 0x7fd09c137e40 2024-04-26T00:32:10.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.972+0000 7fd099d9b640 1 -- 172.21.15.37:0/905071269 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd09c137e40 msgr2=0x7fd09c135070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:10.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.972+0000 7fd099d9b640 1 --2- 172.21.15.37:0/905071269 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd09c137e40 0x7fd09c135070 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd090018d10 tx=0x7fd090018ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:10.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.972+0000 7fd099d9b640 1 --2- 172.21.15.37:0/905071269 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd09c137e40 0x7fd09c135070 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd090018d10 tx=0x7fd090018ea0 comp rx=0 tx=0).queue_received will delay after 1.71409e+09 on 0x7fd0900ba020 mon_map magic: 0 2024-04-26T00:32:10.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.972+0000 7fd08affd640 1 -- 172.21.15.37:0/905071269 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0900ba020 con 0x7fd09c137e40 2024-04-26T00:32:10.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.972+0000 7fd08affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:10.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.972+0000 7fd08affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:32:10.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.972+0000 7fd08affd640 10 monclient: dump: 2024-04-26T00:32:10.975 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:10.975 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:10.975 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:10.975 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:10.975 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:10.975 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:10.975 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:10.975 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:10.975 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:10.975 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:10.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.972+0000 7fd08affd640 10 monclient: _finish_auth 0 2024-04-26T00:32:10.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.972+0000 7fd08affd640 10 monclient: _check_auth_tickets 2024-04-26T00:32:10.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.972+0000 7fd08affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:10.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.972+0000 7fd08affd640 1 -- 172.21.15.37:0/905071269 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0900b6890 con 0x7fd09c137e40 2024-04-26T00:32:10.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.972+0000 7fd0a0c0c640 5 monclient: authenticate success, global_id 144153 2024-04-26T00:32:10.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.972+0000 7fd0a0c0c640 10 monclient: _renew_subs 2024-04-26T00:32:10.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.972+0000 7fd0a0c0c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:10.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.972+0000 7fd0a0c0c640 1 -- 172.21.15.37:0/905071269 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd09c120810 con 0x7fd09c137e40 2024-04-26T00:32:10.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.972+0000 7fd08affd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:10.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.972+0000 7fd098d99640 4 set_mon_vals no callback set 2024-04-26T00:32:10.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.972+0000 7fd0a0c0c640 10 monclient: _renew_subs 2024-04-26T00:32:10.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.972+0000 7fd0a0c0c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:10.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.972+0000 7fd0a0c0c640 1 -- 172.21.15.37:0/905071269 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd09c120250 con 0x7fd09c137e40 2024-04-26T00:32:10.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.972+0000 7fd0a0c0c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:10.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.972+0000 7fd0a0c0c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:10.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.972+0000 7fd0a0c0c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:10.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.972+0000 7fd0a0c0c640 1 -- 172.21.15.37:0/905071269 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd09c066f40 con 0x7fd09c137e40 2024-04-26T00:32:10.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.972+0000 7fd08affd640 1 -- 172.21.15.37:0/905071269 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0900b1040 con 0x7fd09c137e40 2024-04-26T00:32:10.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.972+0000 7fd08affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:10.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.972+0000 7fd08affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:10.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.972+0000 7fd08affd640 10 monclient: dump: 2024-04-26T00:32:10.976 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:10.976 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:10.977 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:10.977 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:10.977 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:10.977 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:10.977 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:10.977 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:10.977 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:10.977 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:10.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.972+0000 7fd08affd640 1 -- 172.21.15.37:0/905071269 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd0900bd020 con 0x7fd09c137e40 2024-04-26T00:32:10.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.972+0000 7fd08affd640 1 --2- 172.21.15.37:0/905071269 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd07c035e60 0x7fd07c038320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:10.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.976+0000 7fd09a59c640 1 --2- 172.21.15.37:0/905071269 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd07c035e60 0x7fd07c038320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:10.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.976+0000 7fd09a59c640 10 monclient: get_auth_request con 0x7fd07c035e60 auth_method 0 2024-04-26T00:32:10.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.976+0000 7fd09a59c640 1 --2- 172.21.15.37:0/905071269 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd07c035e60 0x7fd07c038320 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd08c005060 tx=0x7fd08c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:10.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:10.980+0000 7fd08affd640 1 -- 172.21.15.37:0/905071269 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(469..469 src has 370..469) ==== 3943+0+0 (secure 0 0 0) 0x7fd0900e2c70 con 0x7fd09c137e40 2024-04-26T00:32:11.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:11.596+0000 7fd08affd640 1 -- 172.21.15.37:0/905071269 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fd0900c4030 con 0x7fd09c137e40 2024-04-26T00:32:11.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:11.596+0000 7fd08affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:11.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:11.596+0000 7fd08affd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:32:11.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:11.836+0000 7fd0a0c0c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:32:11.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:11.836+0000 7fd0a0c0c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:32:11.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:11.836+0000 7fd0a0c0c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:11.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:11.836+0000 7fd0a0c0c640 1 -- 172.21.15.37:0/905071269 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd09c120420 con 0x7fd09c137e40 2024-04-26T00:32:11.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:11.836+0000 7fd08affd640 1 -- 172.21.15.37:0/905071269 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fd09c0a1130 con 0x7fd09c137e40 2024-04-26T00:32:11.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:11.836+0000 7fd08affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:32:11.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:11.836+0000 7fd08affd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:32:11.833 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":92,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:32:11.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:11.840+0000 7fd0a0c0c640 1 -- 172.21.15.37:0/905071269 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd07c035e60 msgr2=0x7fd07c038320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:11.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:11.840+0000 7fd0a0c0c640 1 --2- 172.21.15.37:0/905071269 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd07c035e60 0x7fd07c038320 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd08c005060 tx=0x7fd08c009040 comp rx=0 tx=0).stop 2024-04-26T00:32:11.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:11.840+0000 7fd0a0c0c640 10 monclient: shutdown 2024-04-26T00:32:11.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:11.840+0000 7fd0a0c0c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:11.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:11.840+0000 7fd0a0c0c640 1 -- 172.21.15.37:0/905071269 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd09c137e40 msgr2=0x7fd09c135070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:11.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:11.840+0000 7fd0a0c0c640 1 --2- 172.21.15.37:0/905071269 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd09c137e40 0x7fd09c135070 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd090018d10 tx=0x7fd090018ea0 comp rx=0 tx=0).stop 2024-04-26T00:32:11.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:11.840+0000 7fd0a0c0c640 1 -- 172.21.15.37:0/905071269 shutdown_connections 2024-04-26T00:32:11.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:11.840+0000 7fd0a0c0c640 1 --2- 172.21.15.37:0/905071269 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd07c035e60 0x7fd07c038320 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:11.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:11.840+0000 7fd0a0c0c640 1 --2- 172.21.15.37:0/905071269 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd09c13a770 0x7fd09c1cefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:11.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:11.840+0000 7fd0a0c0c640 1 --2- 172.21.15.37:0/905071269 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd09c137e40 0x7fd09c135070 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:11.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:11.840+0000 7fd0a0c0c640 1 --2- 172.21.15.37:0/905071269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd09c0a0260 0x7fd09c134b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:11.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:11.840+0000 7fd0a0c0c640 1 -- 172.21.15.37:0/905071269 >> 172.21.15.37:0/905071269 conn(0x7fd09c0a1ad0 msgr2=0x7fd09c137600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:11.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:11.840+0000 7fd0a0c0c640 1 -- 172.21.15.37:0/905071269 shutdown_connections 2024-04-26T00:32:11.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:11.840+0000 7fd0a0c0c640 1 -- 172.21.15.37:0/905071269 wait complete. 2024-04-26T00:32:13.501 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-04-26T00:32:14.853 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:32:14.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:14.992+0000 7fa99c5c8640 10 monclient: get_monmap_and_config 2024-04-26T00:32:14.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:14.992+0000 7fa99c5c8640 10 monclient: build_initial_monmap 2024-04-26T00:32:14.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:14.992+0000 7fa99c5c8640 10 monclient: monmap: 2024-04-26T00:32:14.987 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:14.987 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:14.987 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:32:14.996935+0000 2024-04-26T00:32:14.987 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:32:14.996935+0000 2024-04-26T00:32:14.987 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:14.987 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:14.987 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:32:14.987 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:32:14.987 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:32:14.987 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:32:14.987 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:32:14.988 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:14.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:14.992+0000 7fa99c5c8640 1 Processor -- start 2024-04-26T00:32:14.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:14.992+0000 7fa99c5c8640 1 -- start start 2024-04-26T00:32:14.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:14.992+0000 7fa99c5c8640 10 monclient: init 2024-04-26T00:32:14.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:14.992+0000 7fa99c5c8640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:14.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:14.992+0000 7fa99c5c8640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-04-26T00:32:14.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:14.992+0000 7fa99c5c8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa994136eb0 0x7fa994137290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:14.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:14.992+0000 7fa99c5c8640 10 monclient(hunting): picked mon.noname-b con 0x7fa994136eb0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:14.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:14.992+0000 7fa99c5c8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa9941377d0 addr v1:172.21.15.37:6789/0 2024-04-26T00:32:14.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:14.992+0000 7fa99c5c8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa9941342a0 addr v1:172.21.15.45:6789/0 2024-04-26T00:32:14.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:14.992+0000 7fa99c5c8640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa99409b8c0 con 0x7fa9941377d0 2024-04-26T00:32:14.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:14.992+0000 7fa99c5c8640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa99409b300 con 0x7fa9941342a0 2024-04-26T00:32:14.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:14.992+0000 7fa99c5c8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:14.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:14.992+0000 7fa99c5c8640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa99409c7d0 con 0x7fa994136eb0 2024-04-26T00:32:14.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:14.992+0000 7fa99c5c8640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:14.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:14.992+0000 7fa99c5c8640 10 monclient(hunting): authenticate will time out at 1716.302680s 2024-04-26T00:32:14.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:14.992+0000 7fa99a364640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa994136eb0 msgr2=0x7fa994137290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:32:14.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:14.992+0000 7fa99a364640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa994136eb0 0x7fa994137290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:32:14.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:14.992+0000 7fa999b63640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fa9941377d0 0x7fa994133b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:35354/0 (socket says 172.21.15.37:35354) 2024-04-26T00:32:14.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:14.992+0000 7fa999b63640 1 -- 172.21.15.37:0/92147425 learned_addr learned my addr 172.21.15.37:0/92147425 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:32:14.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:14.996+0000 7fa999b63640 1 -- 172.21.15.37:0/92147425 >> v1:172.21.15.37:6789/0 conn(0x7fa9941377d0 legacy=0x7fa994133b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:14.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:14.996+0000 7fa99ab65640 1 -- 172.21.15.37:0/92147425 >> v1:172.21.15.45:6789/0 conn(0x7fa9941342a0 legacy=0x7fa99413f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:15.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:15.196+0000 7fa99a364640 1 -- 172.21.15.37:0/92147425 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa994136eb0 msgr2=0x7fa994137290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:32:15.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:15.196+0000 7fa99a364640 1 --2- 172.21.15.37:0/92147425 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa994136eb0 0x7fa994137290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:32:15.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:15.600+0000 7fa99a364640 1 -- 172.21.15.37:0/92147425 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa994136eb0 msgr2=0x7fa994137290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:32:15.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:15.600+0000 7fa99a364640 1 --2- 172.21.15.37:0/92147425 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa994136eb0 0x7fa994137290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:32:16.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:16.404+0000 7fa99a364640 1 -- 172.21.15.37:0/92147425 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa994136eb0 msgr2=0x7fa994137290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:32:16.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:16.404+0000 7fa99a364640 1 --2- 172.21.15.37:0/92147425 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa994136eb0 0x7fa994137290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:32:17.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:17.992+0000 7fa987fff640 10 monclient(hunting): tick 2024-04-26T00:32:17.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:17.992+0000 7fa987fff640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:32:17.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:17.992+0000 7fa987fff640 1 monclient(hunting): continuing hunt 2024-04-26T00:32:17.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:17.992+0000 7fa987fff640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:32:17.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:17.992+0000 7fa987fff640 1 -- 172.21.15.37:0/92147425 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa994136eb0 msgr2=0x7fa994137290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:32:17.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:17.992+0000 7fa987fff640 1 --2- 172.21.15.37:0/92147425 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa994136eb0 0x7fa994137290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:17.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:17.992+0000 7fa987fff640 1 -- 172.21.15.37:0/92147425 >> v1:172.21.15.45:6789/0 conn(0x7fa9941342a0 legacy=0x7fa99413f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:17.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:17.992+0000 7fa987fff640 1 -- 172.21.15.37:0/92147425 >> v1:172.21.15.37:6789/0 conn(0x7fa9941377d0 legacy=0x7fa994133b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:17.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:17.992+0000 7fa987fff640 10 monclient: _add_conns ranks=[3,4] 2024-04-26T00:32:17.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:17.992+0000 7fa987fff640 1 --2- 172.21.15.37:0/92147425 >> v2:172.21.15.37:3300/0 conn(0x7fa978003820 0x7fa978005ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:17.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:17.992+0000 7fa987fff640 10 monclient(hunting): picked mon.noname-a con 0x7fa978003820 addr v2:172.21.15.37:3300/0 2024-04-26T00:32:17.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:17.992+0000 7fa987fff640 1 --2- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978006390 0x7fa978008850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:17.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:17.992+0000 7fa987fff640 10 monclient(hunting): picked mon.noname-c con 0x7fa978006390 addr v2:172.21.15.45:3300/0 2024-04-26T00:32:17.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:17.992+0000 7fa987fff640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:17.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:17.992+0000 7fa987fff640 1 -- 172.21.15.37:0/92147425 --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fa99409c7d0 con 0x7fa978003820 2024-04-26T00:32:17.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:17.992+0000 7fa987fff640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:17.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:17.992+0000 7fa987fff640 1 -- 172.21.15.37:0/92147425 --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fa978008ef0 con 0x7fa978006390 2024-04-26T00:32:17.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:17.992+0000 7fa987fff640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:17.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:17.992+0000 7fa99a364640 1 --2- 172.21.15.37:0/92147425 >> v2:172.21.15.37:3300/0 conn(0x7fa978003820 0x7fa978005ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:17.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:17.992+0000 7fa99a364640 10 monclient(hunting): get_auth_request con 0x7fa978003820 auth_method 0 2024-04-26T00:32:17.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:17.992+0000 7fa99a364640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:17.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:17.992+0000 7fa99a364640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:17.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:17.992+0000 7fa99a364640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:17.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:17.992+0000 7fa999b63640 1 --2- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978006390 0x7fa978008850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:17.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:17.992+0000 7fa999b63640 10 monclient(hunting): get_auth_request con 0x7fa978006390 auth_method 0 2024-04-26T00:32:17.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:17.992+0000 7fa999b63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:17.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:17.992+0000 7fa999b63640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:17.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:17.992+0000 7fa999b63640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:17.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:17.992+0000 7fa99a364640 1 -- 172.21.15.37:0/92147425 >> v2:172.21.15.37:3300/0 conn(0x7fa978003820 msgr2=0x7fa978005ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:32:17.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:17.992+0000 7fa99a364640 1 -- 172.21.15.37:0/92147425 >> v2:172.21.15.37:3300/0 conn(0x7fa978003820 msgr2=0x7fa978005ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:32:17.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:17.992+0000 7fa99a364640 1 --2- 172.21.15.37:0/92147425 >> v2:172.21.15.37:3300/0 conn(0x7fa978003820 0x7fa978005ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:32:17.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:17.992+0000 7fa99a364640 1 --2- 172.21.15.37:0/92147425 >> v2:172.21.15.37:3300/0 conn(0x7fa978003820 0x7fa978005ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:32:17.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:17.992+0000 7fa999b63640 1 -- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978006390 msgr2=0x7fa978008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:32:17.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:17.992+0000 7fa999b63640 1 -- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978006390 msgr2=0x7fa978008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:32:17.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:17.992+0000 7fa999b63640 1 --2- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978006390 0x7fa978008850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:32:17.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:17.992+0000 7fa999b63640 1 --2- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978006390 0x7fa978008850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:32:18.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:18.196+0000 7fa99a364640 1 --2- 172.21.15.37:0/92147425 >> v2:172.21.15.37:3300/0 conn(0x7fa978003820 0x7fa978005ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:18.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:18.196+0000 7fa99a364640 10 monclient(hunting): get_auth_request con 0x7fa978003820 auth_method 0 2024-04-26T00:32:18.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:18.196+0000 7fa99a364640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:18.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:18.196+0000 7fa99a364640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:18.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:18.196+0000 7fa99a364640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:32:18.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:18.196+0000 7fa999b63640 1 --2- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978006390 0x7fa978008850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:18.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:18.196+0000 7fa999b63640 10 monclient(hunting): get_auth_request con 0x7fa978006390 auth_method 0 2024-04-26T00:32:18.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:18.196+0000 7fa999b63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:18.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:18.196+0000 7fa999b63640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:18.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:18.196+0000 7fa999b63640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:32:18.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:18.196+0000 7fa99a364640 1 -- 172.21.15.37:0/92147425 >> v2:172.21.15.37:3300/0 conn(0x7fa978003820 msgr2=0x7fa978005ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:32:18.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:18.196+0000 7fa99a364640 1 -- 172.21.15.37:0/92147425 >> v2:172.21.15.37:3300/0 conn(0x7fa978003820 msgr2=0x7fa978005ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:32:18.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:18.196+0000 7fa99a364640 1 --2- 172.21.15.37:0/92147425 >> v2:172.21.15.37:3300/0 conn(0x7fa978003820 0x7fa978005ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:32:18.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:18.196+0000 7fa99a364640 1 --2- 172.21.15.37:0/92147425 >> v2:172.21.15.37:3300/0 conn(0x7fa978003820 0x7fa978005ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:32:18.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:18.196+0000 7fa999b63640 1 -- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978006390 msgr2=0x7fa978008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:32:18.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:18.196+0000 7fa999b63640 1 -- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978006390 msgr2=0x7fa978008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:32:18.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:18.196+0000 7fa999b63640 1 --2- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978006390 0x7fa978008850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:32:18.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:18.196+0000 7fa999b63640 1 --2- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978006390 0x7fa978008850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:32:18.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:18.600+0000 7fa99a364640 1 --2- 172.21.15.37:0/92147425 >> v2:172.21.15.37:3300/0 conn(0x7fa978003820 0x7fa978005ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:18.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:18.600+0000 7fa99a364640 10 monclient(hunting): get_auth_request con 0x7fa978003820 auth_method 0 2024-04-26T00:32:18.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:18.600+0000 7fa99a364640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:18.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:18.600+0000 7fa99a364640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:18.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:18.600+0000 7fa99a364640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:32:18.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:18.600+0000 7fa999b63640 1 --2- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978006390 0x7fa978008850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:18.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:18.600+0000 7fa999b63640 10 monclient(hunting): get_auth_request con 0x7fa978006390 auth_method 0 2024-04-26T00:32:18.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:18.600+0000 7fa999b63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:18.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:18.600+0000 7fa999b63640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:18.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:18.600+0000 7fa999b63640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:32:18.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:18.600+0000 7fa99a364640 1 -- 172.21.15.37:0/92147425 >> v2:172.21.15.37:3300/0 conn(0x7fa978003820 msgr2=0x7fa978005ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:32:18.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:18.600+0000 7fa99a364640 1 -- 172.21.15.37:0/92147425 >> v2:172.21.15.37:3300/0 conn(0x7fa978003820 msgr2=0x7fa978005ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:32:18.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:18.600+0000 7fa99a364640 1 --2- 172.21.15.37:0/92147425 >> v2:172.21.15.37:3300/0 conn(0x7fa978003820 0x7fa978005ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:32:18.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:18.600+0000 7fa999b63640 1 -- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978006390 msgr2=0x7fa978008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:32:18.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:18.600+0000 7fa999b63640 1 -- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978006390 msgr2=0x7fa978008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:32:18.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:18.600+0000 7fa99a364640 1 --2- 172.21.15.37:0/92147425 >> v2:172.21.15.37:3300/0 conn(0x7fa978003820 0x7fa978005ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:32:18.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:18.600+0000 7fa999b63640 1 --2- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978006390 0x7fa978008850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:32:18.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:18.600+0000 7fa999b63640 1 --2- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978006390 0x7fa978008850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:32:19.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:19.404+0000 7fa99a364640 1 --2- 172.21.15.37:0/92147425 >> v2:172.21.15.37:3300/0 conn(0x7fa978003820 0x7fa978005ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:19.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:19.404+0000 7fa99a364640 10 monclient(hunting): get_auth_request con 0x7fa978003820 auth_method 0 2024-04-26T00:32:19.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:19.404+0000 7fa99a364640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:19.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:19.404+0000 7fa99a364640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:19.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:19.404+0000 7fa99a364640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:32:19.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:19.404+0000 7fa999b63640 1 --2- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978006390 0x7fa978008850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:19.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:19.404+0000 7fa999b63640 10 monclient(hunting): get_auth_request con 0x7fa978006390 auth_method 0 2024-04-26T00:32:19.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:19.404+0000 7fa999b63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:19.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:19.404+0000 7fa999b63640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:19.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:19.404+0000 7fa999b63640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:32:19.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:19.404+0000 7fa99a364640 1 -- 172.21.15.37:0/92147425 >> v2:172.21.15.37:3300/0 conn(0x7fa978003820 msgr2=0x7fa978005ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:32:19.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:19.404+0000 7fa99a364640 1 -- 172.21.15.37:0/92147425 >> v2:172.21.15.37:3300/0 conn(0x7fa978003820 msgr2=0x7fa978005ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:32:19.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:19.404+0000 7fa99a364640 1 --2- 172.21.15.37:0/92147425 >> v2:172.21.15.37:3300/0 conn(0x7fa978003820 0x7fa978005ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:32:19.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:19.404+0000 7fa99a364640 1 --2- 172.21.15.37:0/92147425 >> v2:172.21.15.37:3300/0 conn(0x7fa978003820 0x7fa978005ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:32:19.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:19.404+0000 7fa999b63640 1 -- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978006390 msgr2=0x7fa978008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:32:19.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:19.404+0000 7fa999b63640 1 -- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978006390 msgr2=0x7fa978008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:32:19.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:19.404+0000 7fa999b63640 1 --2- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978006390 0x7fa978008850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:32:19.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:19.404+0000 7fa999b63640 1 --2- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978006390 0x7fa978008850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:32:20.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:20.996+0000 7fa987fff640 10 monclient(hunting): tick 2024-04-26T00:32:20.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:20.996+0000 7fa987fff640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:32:20.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:20.996+0000 7fa987fff640 1 monclient(hunting): continuing hunt 2024-04-26T00:32:20.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:20.996+0000 7fa987fff640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:32:20.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:20.996+0000 7fa987fff640 1 -- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978006390 msgr2=0x7fa978008850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:32:20.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:20.996+0000 7fa987fff640 1 --2- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978006390 0x7fa978008850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:20.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:20.996+0000 7fa987fff640 1 -- 172.21.15.37:0/92147425 >> v2:172.21.15.37:3300/0 conn(0x7fa978003820 msgr2=0x7fa978005ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:32:20.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:20.996+0000 7fa987fff640 1 --2- 172.21.15.37:0/92147425 >> v2:172.21.15.37:3300/0 conn(0x7fa978003820 0x7fa978005ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:20.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:20.996+0000 7fa987fff640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-04-26T00:32:20.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:20.996+0000 7fa99ab65640 1 -- 172.21.15.37:0/92147425 reap_dead start 2024-04-26T00:32:20.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:20.996+0000 7fa987fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa9941377d0 addr v1:172.21.15.37:6789/0 2024-04-26T00:32:20.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:20.996+0000 7fa987fff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa978006820 addr v1:172.21.15.45:6789/0 2024-04-26T00:32:20.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:20.996+0000 7fa987fff640 1 --2- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978005320 0x7fa9780057d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:20.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:20.996+0000 7fa987fff640 10 monclient(hunting): picked mon.noname-c con 0x7fa978005320 addr v2:172.21.15.45:3300/0 2024-04-26T00:32:20.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:20.996+0000 7fa987fff640 1 -- 172.21.15.37:0/92147425 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa99409b5e0 con 0x7fa9941377d0 2024-04-26T00:32:20.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:20.996+0000 7fa987fff640 1 -- 172.21.15.37:0/92147425 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa978005e90 con 0x7fa978006820 2024-04-26T00:32:20.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:20.996+0000 7fa987fff640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:20.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:20.996+0000 7fa987fff640 1 -- 172.21.15.37:0/92147425 --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fa99409c7d0 con 0x7fa978005320 2024-04-26T00:32:20.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:20.996+0000 7fa987fff640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:20.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:20.996+0000 7fa99ab65640 1 --2- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978005320 0x7fa9780057d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:20.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:20.996+0000 7fa99ab65640 10 monclient(hunting): get_auth_request con 0x7fa978005320 auth_method 0 2024-04-26T00:32:20.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:20.996+0000 7fa99ab65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:20.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:20.996+0000 7fa99ab65640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:20.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:20.996+0000 7fa99ab65640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:20.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:20.996+0000 7fa99ab65640 1 -- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978005320 msgr2=0x7fa9780057d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:32:20.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:20.996+0000 7fa99ab65640 1 -- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978005320 msgr2=0x7fa9780057d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:32:20.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:20.996+0000 7fa99ab65640 1 --2- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978005320 0x7fa9780057d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:32:20.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:20.996+0000 7fa99ab65640 1 --2- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978005320 0x7fa9780057d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:32:20.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:20.996+0000 7fa99a364640 1 -- 172.21.15.37:0/92147425 >> v1:172.21.15.37:6789/0 conn(0x7fa9941377d0 legacy=0x7fa97800b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:20.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:20.996+0000 7fa999b63640 1 -- 172.21.15.37:0/92147425 >> v1:172.21.15.45:6789/0 conn(0x7fa978006820 legacy=0x7fa978003c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:21.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.200+0000 7fa99ab65640 1 --2- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978005320 0x7fa9780057d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:21.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.200+0000 7fa99ab65640 10 monclient(hunting): get_auth_request con 0x7fa978005320 auth_method 0 2024-04-26T00:32:21.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.200+0000 7fa99ab65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:21.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.200+0000 7fa99ab65640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:21.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.200+0000 7fa99ab65640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:32:21.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.200+0000 7fa99ab65640 1 -- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978005320 msgr2=0x7fa9780057d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:32:21.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.200+0000 7fa99ab65640 1 -- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978005320 msgr2=0x7fa9780057d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:32:21.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.200+0000 7fa99ab65640 1 --2- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978005320 0x7fa9780057d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:32:21.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.200+0000 7fa99ab65640 1 --2- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978005320 0x7fa9780057d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:32:21.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.604+0000 7fa99ab65640 1 --2- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978005320 0x7fa9780057d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:21.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.604+0000 7fa99ab65640 10 monclient(hunting): get_auth_request con 0x7fa978005320 auth_method 0 2024-04-26T00:32:21.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.604+0000 7fa99ab65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:21.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.604+0000 7fa99ab65640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:21.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.604+0000 7fa99ab65640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:32:21.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.604+0000 7fa99ab65640 1 -- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978005320 msgr2=0x7fa9780057d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:32:21.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.604+0000 7fa99ab65640 1 -- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978005320 msgr2=0x7fa9780057d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:32:21.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.604+0000 7fa99ab65640 1 --2- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978005320 0x7fa9780057d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:32:21.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.604+0000 7fa99ab65640 1 --2- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978005320 0x7fa9780057d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:32:21.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999362640 1 -- 172.21.15.37:0/92147425 <== mon.1 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2608523732 0 0) 0x7fa978005e90 con 0x7fa978006820 2024-04-26T00:32:21.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999362640 10 monclient(hunting): _negotiate 2024-04-26T00:32:21.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999362640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:21.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999362640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:21.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999362640 10 monclient(hunting): my global_id is 134230 2024-04-26T00:32:21.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999362640 1 -- 172.21.15.37:0/92147425 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa980003530 con 0x7fa978006820 2024-04-26T00:32:21.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999362640 1 -- 172.21.15.37:0/92147425 <== mon.1 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2369868820 0 0) 0x7fa980003530 con 0x7fa978006820 2024-04-26T00:32:21.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999362640 1 -- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978005320 msgr2=0x7fa9780057d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:32:21.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999362640 1 --2- 172.21.15.37:0/92147425 >> v2:172.21.15.45:3300/0 conn(0x7fa978005320 0x7fa9780057d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:21.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999362640 1 -- 172.21.15.37:0/92147425 >> v1:172.21.15.37:6789/0 conn(0x7fa9941377d0 legacy=0x7fa97800b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:21.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999362640 10 monclient: _finish_hunting 0 2024-04-26T00:32:21.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999362640 1 monclient: found mon.noname-c-legacy 2024-04-26T00:32:21.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999362640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:21.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999362640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.45:6789/0 2024-04-26T00:32:21.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999362640 1 -- 172.21.15.37:0/92147425 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa978010aa0 con 0x7fa978006820 2024-04-26T00:32:21.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999362640 10 monclient: _finish_auth 0 2024-04-26T00:32:21.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999362640 10 monclient: _check_auth_tickets 2024-04-26T00:32:21.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999362640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:21.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999362640 1 -- 172.21.15.37:0/92147425 <== mon.1 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 866326386 0 0) 0x7fa9880018f0 con 0x7fa978006820 2024-04-26T00:32:21.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999362640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:21.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999362640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-04-26T00:32:21.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999362640 10 monclient: dump: 2024-04-26T00:32:21.770 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:21.770 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:21.770 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:21.770 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:21.770 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:21.770 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:21.770 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:21.770 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:21.770 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:21.770 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:21.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999362640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] but i'm connected to v1:172.21.15.45:6789/0, reconnecting 2024-04-26T00:32:21.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999362640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:21.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999362640 1 -- 172.21.15.37:0/92147425 >> v1:172.21.15.45:6789/0 conn(0x7fa978006820 legacy=0x7fa978003c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:21.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999362640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:32:21.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999362640 1 --2- 172.21.15.37:0/92147425 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa9800039c0 0x7fa980003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:21.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999362640 10 monclient(hunting): picked mon.a con 0x7fa9800039c0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:32:21.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999362640 1 --2- 172.21.15.37:0/92147425 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa980004810 0x7fa980004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:21.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999362640 10 monclient(hunting): picked mon.b con 0x7fa980004810 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:32:21.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999362640 1 --2- 172.21.15.37:0/92147425 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa9800052f0 0x7fa98000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:21.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999362640 10 monclient(hunting): picked mon.c con 0x7fa9800052f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:21.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999362640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:21.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999362640 1 -- 172.21.15.37:0/92147425 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa99409c7d0 con 0x7fa9800039c0 2024-04-26T00:32:21.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999362640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:21.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999362640 1 -- 172.21.15.37:0/92147425 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa98000c1c0 con 0x7fa980004810 2024-04-26T00:32:21.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999362640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:21.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999362640 1 -- 172.21.15.37:0/92147425 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa98000c3c0 con 0x7fa9800052f0 2024-04-26T00:32:21.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999362640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:21.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999362640 10 monclient(hunting): _finish_auth 0 2024-04-26T00:32:21.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa99ab65640 1 -- 172.21.15.37:0/92147425 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa9800052f0 msgr2=0x7fa98000b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:32:21.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa99ab65640 1 --2- 172.21.15.37:0/92147425 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa9800052f0 0x7fa98000b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:32:21.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999b63640 1 --2- 172.21.15.37:0/92147425 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa980004810 0x7fa980004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:21.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999b63640 10 monclient(hunting): get_auth_request con 0x7fa980004810 auth_method 0 2024-04-26T00:32:21.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999b63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:21.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999b63640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:21.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999b63640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:32:21.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999b63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:21.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999b63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:21.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999b63640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:32:21.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999b63640 10 monclient(hunting): handle_auth_done global_id 134230 payload 293 2024-04-26T00:32:21.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999b63640 1 -- 172.21.15.37:0/92147425 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa9800052f0 msgr2=0x7fa98000b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:32:21.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999b63640 1 --2- 172.21.15.37:0/92147425 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa9800052f0 0x7fa98000b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:21.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999b63640 1 -- 172.21.15.37:0/92147425 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa9800039c0 msgr2=0x7fa980003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:21.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999b63640 1 --2- 172.21.15.37:0/92147425 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa9800039c0 0x7fa980003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:21.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999b63640 10 monclient: _finish_hunting 0 2024-04-26T00:32:21.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999b63640 1 monclient: found mon.b 2024-04-26T00:32:21.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999b63640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:21.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999b63640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.45:3300/0 2024-04-26T00:32:21.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa99ab65640 1 -- 172.21.15.37:0/92147425 reap_dead start 2024-04-26T00:32:21.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999b63640 1 -- 172.21.15.37:0/92147425 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa980003530 con 0x7fa980004810 2024-04-26T00:32:21.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999b63640 10 monclient: _finish_auth 0 2024-04-26T00:32:21.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999b63640 10 monclient: _check_auth_tickets 2024-04-26T00:32:21.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.772+0000 7fa999b63640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:21.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.776+0000 7fa99c5c8640 5 monclient: authenticate success, global_id 134230 2024-04-26T00:32:21.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.776+0000 7fa99c5c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:32:21.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.780+0000 7fa999b63640 1 -- 172.21.15.37:0/92147425 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa980004810 msgr2=0x7fa980004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:21.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.780+0000 7fa999b63640 1 --2- 172.21.15.37:0/92147425 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa980004810 0x7fa980004c90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa988001ec0 tx=0x7fa98801d660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c591b654ac73a92 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:21.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.784+0000 7fa999362640 1 -- 172.21.15.37:0/92147425 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9880058c0 con 0x7fa980004810 2024-04-26T00:32:21.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.784+0000 7fa999362640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:21.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.784+0000 7fa999362640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:32:21.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.784+0000 7fa999362640 10 monclient: dump: 2024-04-26T00:32:21.780 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:21.780 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:21.780 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:21.780 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:21.780 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:21.780 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:21.780 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:21.780 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:21.780 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:21.780 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:21.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.784+0000 7fa99c5c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:32:21.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.784+0000 7fa999362640 1 -- 172.21.15.37:0/92147425 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9880af470 con 0x7fa980004810 2024-04-26T00:32:21.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.784+0000 7fa999362640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:21.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.784+0000 7fa99c5c8640 10 monclient: get_monmap_and_config success 2024-04-26T00:32:21.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.784+0000 7fa99c5c8640 4 set_mon_vals no callback set 2024-04-26T00:32:21.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.784+0000 7fa999362640 1 -- 172.21.15.37:0/92147425 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9880af840 con 0x7fa980004810 2024-04-26T00:32:21.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.784+0000 7fa999362640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:21.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.784+0000 7fa999362640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:32:21.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.784+0000 7fa999362640 10 monclient: dump: 2024-04-26T00:32:21.781 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:21.781 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:21.781 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:21.781 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:21.781 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:21.781 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:21.781 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:21.781 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:21.782 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:21.782 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:21.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.784+0000 7fa99c5c8640 10 monclient: shutdown 2024-04-26T00:32:21.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.784+0000 7fa99c5c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:21.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.784+0000 7fa99c5c8640 1 -- 172.21.15.37:0/92147425 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa980004810 msgr2=0x7fa980004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:21.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.784+0000 7fa99c5c8640 1 --2- 172.21.15.37:0/92147425 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa980004810 0x7fa980004c90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa988001ec0 tx=0x7fa98801d660 comp rx=0 tx=0).stop 2024-04-26T00:32:21.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.784+0000 7fa99c5c8640 1 -- 172.21.15.37:0/92147425 shutdown_connections 2024-04-26T00:32:21.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.784+0000 7fa99c5c8640 1 --2- 172.21.15.37:0/92147425 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa980004810 0x7fa980004c90 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:21.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.784+0000 7fa99c5c8640 1 -- 172.21.15.37:0/92147425 >> 172.21.15.37:0/92147425 conn(0x7fa9940a6300 msgr2=0x7fa9940a6710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:21.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.784+0000 7fa99c5c8640 1 -- 172.21.15.37:0/92147425 shutdown_connections 2024-04-26T00:32:21.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.784+0000 7fa99c5c8640 1 -- 172.21.15.37:0/92147425 wait complete. 2024-04-26T00:32:21.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99c5c8640 10 monclient: build_initial_monmap 2024-04-26T00:32:21.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99c5c8640 10 monclient: monmap: 2024-04-26T00:32:21.786 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:21.786 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:32:21.786 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:32:21.786 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:32:21.786 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:21.786 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:32:21.786 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:32:21.786 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:32:21.786 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:32:21.787 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:21.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99c5c8640 1 Processor -- start 2024-04-26T00:32:21.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99c5c8640 1 -- start start 2024-04-26T00:32:21.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99c5c8640 10 monclient: init 2024-04-26T00:32:21.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99c5c8640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:21.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99c5c8640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:32:21.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99c5c8640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa980004810 0x7fa9940a0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:21.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99c5c8640 10 monclient(hunting): picked mon.noname-a con 0x7fa980004810 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:32:21.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99c5c8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa9940a0d40 0x7fa9940a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:21.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99c5c8640 10 monclient(hunting): picked mon.noname-c con 0x7fa9940a0d40 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:21.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99c5c8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9941cfd00 0x7fa9941d0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:21.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99c5c8640 10 monclient(hunting): picked mon.noname-b con 0x7fa9941cfd00 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:32:21.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99c5c8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:21.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99c5c8640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa99409a9e0 con 0x7fa980004810 2024-04-26T00:32:21.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99c5c8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:21.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99c5c8640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa99409ab60 con 0x7fa9941cfd00 2024-04-26T00:32:21.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99c5c8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:21.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99c5c8640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa994133b90 con 0x7fa9940a0d40 2024-04-26T00:32:21.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99c5c8640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:21.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99c5c8640 10 monclient(hunting): authenticate will time out at 1723.097532s 2024-04-26T00:32:21.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa999b63640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa9940a0d40 msgr2=0x7fa9940a1180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:32:21.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa999b63640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa9940a0d40 0x7fa9940a1180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:32:21.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99a364640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa980004810 0x7fa9940a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:21.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99a364640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa980004810 0x7fa9940a0800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:56168/0 (socket says 172.21.15.37:56168) 2024-04-26T00:32:21.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99a364640 1 -- 172.21.15.37:0/2065901709 learned_addr learned my addr 172.21.15.37:0/2065901709 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:32:21.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99a364640 10 monclient(hunting): get_auth_request con 0x7fa980004810 auth_method 0 2024-04-26T00:32:21.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99a364640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:21.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99a364640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:21.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99a364640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:21.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99a364640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:21.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99a364640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:21.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99a364640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:21.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99ab65640 1 --2- 172.21.15.37:0/2065901709 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9941cfd00 0x7fa9941d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:21.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99ab65640 10 monclient(hunting): get_auth_request con 0x7fa9941cfd00 auth_method 0 2024-04-26T00:32:21.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99ab65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:21.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99ab65640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:21.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99ab65640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:21.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99ab65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:21.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99ab65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:21.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99ab65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:21.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99a364640 10 monclient(hunting): handle_auth_done global_id 144156 payload 824 2024-04-26T00:32:21.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99a364640 1 -- 172.21.15.37:0/2065901709 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa9940a0d40 msgr2=0x7fa9940a1180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:32:21.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99a364640 1 --2- 172.21.15.37:0/2065901709 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa9940a0d40 0x7fa9940a1180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:21.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99a364640 1 -- 172.21.15.37:0/2065901709 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9941cfd00 msgr2=0x7fa9941d0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:21.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99a364640 1 --2- 172.21.15.37:0/2065901709 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9941cfd00 0x7fa9941d0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:21.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99a364640 10 monclient: _finish_hunting 0 2024-04-26T00:32:21.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99a364640 1 monclient: found mon.noname-a 2024-04-26T00:32:21.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99a364640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:21.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99a364640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:32:21.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.788+0000 7fa99a364640 1 -- 172.21.15.37:0/2065901709 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa99409ae10 con 0x7fa980004810 2024-04-26T00:32:21.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.792+0000 7fa99a364640 1 -- 172.21.15.37:0/2065901709 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa980004810 msgr2=0x7fa9940a0800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:21.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.792+0000 7fa99a364640 1 --2- 172.21.15.37:0/2065901709 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa980004810 0x7fa9940a0800 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa990000c10 tx=0x7fa990009fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:21.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.792+0000 7fa986ffd640 1 -- 172.21.15.37:0/2065901709 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa99001b020 con 0x7fa980004810 2024-04-26T00:32:21.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.792+0000 7fa986ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:21.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.792+0000 7fa986ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:32:21.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.792+0000 7fa986ffd640 10 monclient: dump: 2024-04-26T00:32:21.792 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:21.792 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:21.792 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:21.792 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:21.792 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:21.792 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:21.793 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:21.793 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:21.793 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:21.793 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:21.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.792+0000 7fa986ffd640 10 monclient: _finish_auth 0 2024-04-26T00:32:21.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.792+0000 7fa986ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:32:21.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.792+0000 7fa986ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:21.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.792+0000 7fa986ffd640 1 -- 172.21.15.37:0/2065901709 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9900173f0 con 0x7fa980004810 2024-04-26T00:32:21.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.792+0000 7fa99c5c8640 5 monclient: authenticate success, global_id 144156 2024-04-26T00:32:21.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.792+0000 7fa99c5c8640 10 monclient: _renew_subs 2024-04-26T00:32:21.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.792+0000 7fa99c5c8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:21.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.792+0000 7fa99c5c8640 1 -- 172.21.15.37:0/2065901709 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa994131a50 con 0x7fa980004810 2024-04-26T00:32:21.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.792+0000 7fa986ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:21.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.792+0000 7fa986ffd640 1 -- 172.21.15.37:0/2065901709 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9900177c0 con 0x7fa980004810 2024-04-26T00:32:21.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.792+0000 7fa986ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:21.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.792+0000 7fa986ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:21.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.792+0000 7fa986ffd640 10 monclient: dump: 2024-04-26T00:32:21.794 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:21.794 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:21.794 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:21.794 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:21.794 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:21.794 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:21.794 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:21.794 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:21.794 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:21.795 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:21.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.792+0000 7fa99c5c8640 10 monclient: _renew_subs 2024-04-26T00:32:21.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.792+0000 7fa998b61640 4 set_mon_vals no callback set 2024-04-26T00:32:21.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.792+0000 7fa99c5c8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:21.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.792+0000 7fa99c5c8640 1 -- 172.21.15.37:0/2065901709 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9940a2360 con 0x7fa980004810 2024-04-26T00:32:21.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.792+0000 7fa986ffd640 1 -- 172.21.15.37:0/2065901709 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa99001e020 con 0x7fa980004810 2024-04-26T00:32:21.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.792+0000 7fa986ffd640 1 --2- 172.21.15.37:0/2065901709 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa978036fc0 0x7fa978039480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:21.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.792+0000 7fa986ffd640 1 -- 172.21.15.37:0/2065901709 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(469..469 src has 370..469) ==== 3943+0+0 (secure 0 0 0) 0x7fa990049700 con 0x7fa980004810 2024-04-26T00:32:21.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.792+0000 7fa99c5c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:21.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.792+0000 7fa99c5c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:21.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.792+0000 7fa99c5c8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:21.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.792+0000 7fa99c5c8640 1 -- 172.21.15.37:0/2065901709 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa994066f40 con 0x7fa980004810 2024-04-26T00:32:21.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.792+0000 7fa999b63640 1 --2- 172.21.15.37:0/2065901709 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa978036fc0 0x7fa978039480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:21.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.792+0000 7fa999b63640 10 monclient: get_auth_request con 0x7fa978036fc0 auth_method 0 2024-04-26T00:32:21.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.796+0000 7fa999b63640 1 --2- 172.21.15.37:0/2065901709 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa978036fc0 0x7fa978039480 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fa9880ae000 tx=0x7fa9880014e0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:21.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.796+0000 7fa986ffd640 1 -- 172.21.15.37:0/2065901709 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa990025020 con 0x7fa980004810 2024-04-26T00:32:21.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.796+0000 7fa986ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:21.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:21.796+0000 7fa986ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:32:21.836 INFO:tasks.rados.rados.0.smithi045.stdout:2063: done (10 left) 2024-04-26T00:32:21.836 INFO:tasks.rados.rados.0.smithi045.stdout:2064: done (9 left) 2024-04-26T00:32:21.836 INFO:tasks.rados.rados.0.smithi045.stdout:2065: done (8 left) 2024-04-26T00:32:21.837 INFO:tasks.rados.rados.0.smithi045.stdout:2066: done (7 left) 2024-04-26T00:32:21.837 INFO:tasks.rados.rados.0.smithi045.stdout:2068: done (6 left) 2024-04-26T00:32:21.837 INFO:tasks.rados.rados.0.smithi045.stdout:2069: done (5 left) 2024-04-26T00:32:21.837 INFO:tasks.rados.rados.0.smithi045.stdout:2070: done (4 left) 2024-04-26T00:32:21.837 INFO:tasks.rados.rados.0.smithi045.stdout:2072: done (3 left) 2024-04-26T00:32:21.837 INFO:tasks.rados.rados.0.smithi045.stdout:2073: done (2 left) 2024-04-26T00:32:21.837 INFO:tasks.rados.rados.0.smithi045.stdout:2074: done (1 left) 2024-04-26T00:32:21.837 INFO:tasks.rados.rados.0.smithi045.stdout:2075: done (0 left) 2024-04-26T00:32:21.837 INFO:tasks.rados.rados.0.smithi045.stdout:2076: read oid 36 snap -1 2024-04-26T00:32:21.837 INFO:tasks.rados.rados.0.smithi045.stdout:2076: expect (ObjNum 703 snap 221 seq_num 703) 2024-04-26T00:32:21.837 INFO:tasks.rados.rados.0.smithi045.stdout:2077: snap_remove snap 211 2024-04-26T00:32:22.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.036+0000 7fa99c5c8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:32:22.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.036+0000 7fa99c5c8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:32:22.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.036+0000 7fa99c5c8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:22.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.036+0000 7fa99c5c8640 1 -- 172.21.15.37:0/2065901709 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa99406d6e0 con 0x7fa980004810 2024-04-26T00:32:22.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.036+0000 7fa986ffd640 1 -- 172.21.15.37:0/2065901709 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1550 (secure 0 0 0) 0x7fa990017b80 con 0x7fa980004810 2024-04-26T00:32:22.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.036+0000 7fa986ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:32:22.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.036+0000 7fa986ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:32:22.032 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":94,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:32:22.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.040+0000 7fa99c5c8640 1 -- 172.21.15.37:0/2065901709 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa978036fc0 msgr2=0x7fa978039480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:22.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.040+0000 7fa99c5c8640 1 --2- 172.21.15.37:0/2065901709 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa978036fc0 0x7fa978039480 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fa9880ae000 tx=0x7fa9880014e0 comp rx=0 tx=0).stop 2024-04-26T00:32:22.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.040+0000 7fa99c5c8640 10 monclient: shutdown 2024-04-26T00:32:22.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.040+0000 7fa99c5c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:22.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.040+0000 7fa99c5c8640 1 -- 172.21.15.37:0/2065901709 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa980004810 msgr2=0x7fa9940a0800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:22.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.040+0000 7fa99c5c8640 1 --2- 172.21.15.37:0/2065901709 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa980004810 0x7fa9940a0800 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa990000c10 tx=0x7fa990009fb0 comp rx=0 tx=0).stop 2024-04-26T00:32:22.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.040+0000 7fa99c5c8640 1 -- 172.21.15.37:0/2065901709 shutdown_connections 2024-04-26T00:32:22.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.040+0000 7fa99c5c8640 1 --2- 172.21.15.37:0/2065901709 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa978036fc0 0x7fa978039480 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:22.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.040+0000 7fa99c5c8640 1 --2- 172.21.15.37:0/2065901709 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9941cfd00 0x7fa9941d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:22.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.040+0000 7fa99c5c8640 1 --2- 172.21.15.37:0/2065901709 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa9940a0d40 0x7fa9940a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:22.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.040+0000 7fa99c5c8640 1 --2- 172.21.15.37:0/2065901709 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa980004810 0x7fa9940a0800 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:22.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.040+0000 7fa99c5c8640 1 -- 172.21.15.37:0/2065901709 >> 172.21.15.37:0/2065901709 conn(0x7fa9940a6300 msgr2=0x7fa994133020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:22.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.040+0000 7fa99c5c8640 1 -- 172.21.15.37:0/2065901709 shutdown_connections 2024-04-26T00:32:22.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.040+0000 7fa99c5c8640 1 -- 172.21.15.37:0/2065901709 wait complete. 2024-04-26T00:32:22.051 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-26T00:32:22.051 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-04-26T00:32:22.052 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:32:22.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f7e123640 10 monclient: get_monmap_and_config 2024-04-26T00:32:22.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f7e123640 10 monclient: build_initial_monmap 2024-04-26T00:32:22.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f7e123640 10 monclient: monmap: 2024-04-26T00:32:22.170 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:22.170 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:22.170 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:32:22.180022+0000 2024-04-26T00:32:22.170 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:32:22.180022+0000 2024-04-26T00:32:22.170 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:22.171 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:22.171 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:32:22.171 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:32:22.171 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:32:22.171 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:32:22.171 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:32:22.171 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:22.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f7e123640 1 Processor -- start 2024-04-26T00:32:22.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f7e123640 1 -- start start 2024-04-26T00:32:22.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f7e123640 10 monclient: init 2024-04-26T00:32:22.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f7e123640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:22.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f7e123640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-04-26T00:32:22.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f7e123640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9f780a3d90 addr v1:172.21.15.37:6789/0 2024-04-26T00:32:22.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f7e123640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9f780a4170 0x7f9f780a45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:22.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f7e123640 10 monclient(hunting): picked mon.noname-c con 0x7f9f780a4170 addr v2:172.21.15.45:3300/0 2024-04-26T00:32:22.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f7e123640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9f7813c3e0 0x7f9f7813e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:22.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f7e123640 10 monclient(hunting): picked mon.noname-a con 0x7f9f7813c3e0 addr v2:172.21.15.37:3300/0 2024-04-26T00:32:22.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f7e123640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f780a5150 con 0x7f9f780a3d90 2024-04-26T00:32:22.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f7e123640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:22.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f7e123640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f9f7809c650 con 0x7f9f7813c3e0 2024-04-26T00:32:22.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f7e123640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:22.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f7e123640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f9f7809b280 con 0x7f9f780a4170 2024-04-26T00:32:22.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f7e123640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:22.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f7e123640 10 monclient(hunting): authenticate will time out at 1723.485711s 2024-04-26T00:32:22.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f777fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f9f780a3d90 0x7f9f7809bf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:47066/0 (socket says 172.21.15.37:47066) 2024-04-26T00:32:22.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f777fe640 1 -- 172.21.15.37:0/1684262837 learned_addr learned my addr 172.21.15.37:0/1684262837 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:32:22.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f77fff640 1 --2- 172.21.15.37:0/1684262837 >> v2:172.21.15.37:3300/0 conn(0x7f9f7813c3e0 0x7f9f7813e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:22.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f77fff640 10 monclient(hunting): get_auth_request con 0x7f9f7813c3e0 auth_method 0 2024-04-26T00:32:22.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:22.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f77fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:22.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f77fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:22.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:22.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:22.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:22.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f76ffd640 1 --2- 172.21.15.37:0/1684262837 >> v2:172.21.15.45:3300/0 conn(0x7f9f780a4170 0x7f9f780a45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:22.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f76ffd640 10 monclient(hunting): get_auth_request con 0x7f9f780a4170 auth_method 0 2024-04-26T00:32:22.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f76ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:22.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f76ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:22.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:22.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f77fff640 10 monclient(hunting): handle_auth_done global_id 144159 payload 274 2024-04-26T00:32:22.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f77fff640 1 -- 172.21.15.37:0/1684262837 >> v2:172.21.15.45:3300/0 conn(0x7f9f780a4170 msgr2=0x7f9f780a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:22.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f77fff640 1 --2- 172.21.15.37:0/1684262837 >> v2:172.21.15.45:3300/0 conn(0x7f9f780a4170 0x7f9f780a45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:22.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f77fff640 1 -- 172.21.15.37:0/1684262837 >> v1:172.21.15.37:6789/0 conn(0x7f9f780a3d90 legacy=0x7f9f7809bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:22.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f77fff640 10 monclient: _finish_hunting 0 2024-04-26T00:32:22.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f77fff640 1 monclient: found mon.noname-a 2024-04-26T00:32:22.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f77fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:22.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f77fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:32:22.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.176+0000 7f9f77fff640 1 -- 172.21.15.37:0/1684262837 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f7809b560 con 0x7f9f7813c3e0 2024-04-26T00:32:22.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.180+0000 7f9f77fff640 1 -- 172.21.15.37:0/1684262837 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f7813c3e0 msgr2=0x7f9f7813e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:22.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.180+0000 7f9f77fff640 1 --2- 172.21.15.37:0/1684262837 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f7813c3e0 0x7f9f7813e7d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9f7809d140 tx=0x7f9f6801c300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee5e30f29a4a043d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:22.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.180+0000 7f9f767fc640 1 -- 172.21.15.37:0/1684262837 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f680b0070 con 0x7f9f7813c3e0 2024-04-26T00:32:22.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.180+0000 7f9f767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:22.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.180+0000 7f9f767fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:32:22.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.180+0000 7f9f767fc640 10 monclient: dump: 2024-04-26T00:32:22.184 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:22.184 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:22.184 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:22.184 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:22.184 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:22.184 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:22.184 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:22.184 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:22.184 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:22.184 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:22.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.180+0000 7f9f767fc640 10 monclient: _finish_auth 0 2024-04-26T00:32:22.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.180+0000 7f9f767fc640 10 monclient: _check_auth_tickets 2024-04-26T00:32:22.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.180+0000 7f9f767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:22.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.180+0000 7f9f767fc640 1 -- 172.21.15.37:0/1684262837 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f680ab600 con 0x7f9f7813c3e0 2024-04-26T00:32:22.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.180+0000 7f9f767fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:22.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.180+0000 7f9f767fc640 1 -- 172.21.15.37:0/1684262837 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f680ab9d0 con 0x7f9f7813c3e0 2024-04-26T00:32:22.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.180+0000 7f9f767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:22.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.180+0000 7f9f767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:22.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.180+0000 7f9f767fc640 10 monclient: dump: 2024-04-26T00:32:22.185 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:22.185 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:22.185 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:22.185 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:22.185 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:22.185 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:22.185 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:22.186 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:22.186 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:22.186 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:22.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.180+0000 7f9f7e123640 5 monclient: authenticate success, global_id 144159 2024-04-26T00:32:22.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.180+0000 7f9f7e123640 10 monclient: get_monmap_and_config success 2024-04-26T00:32:22.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.180+0000 7f9f7e123640 4 set_mon_vals no callback set 2024-04-26T00:32:22.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.180+0000 7f9f7e123640 10 monclient: shutdown 2024-04-26T00:32:22.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.180+0000 7f9f7e123640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:22.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.180+0000 7f9f7e123640 1 -- 172.21.15.37:0/1684262837 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f7813c3e0 msgr2=0x7f9f7813e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:22.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.180+0000 7f9f7e123640 1 --2- 172.21.15.37:0/1684262837 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f7813c3e0 0x7f9f7813e7d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9f7809d140 tx=0x7f9f6801c300 comp rx=0 tx=0).stop 2024-04-26T00:32:22.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.180+0000 7f9f7e123640 1 -- 172.21.15.37:0/1684262837 shutdown_connections 2024-04-26T00:32:22.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.180+0000 7f9f7e123640 1 --2- 172.21.15.37:0/1684262837 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f7813c3e0 0x7f9f7813e7d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:22.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.180+0000 7f9f7e123640 1 --2- 172.21.15.37:0/1684262837 >> v2:172.21.15.45:3300/0 conn(0x7f9f780a4170 0x7f9f780a45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:22.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.180+0000 7f9f7e123640 1 -- 172.21.15.37:0/1684262837 >> 172.21.15.37:0/1684262837 conn(0x7f9f780aa0a0 msgr2=0x7f9f780a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:22.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.180+0000 7f9f7e123640 1 -- 172.21.15.37:0/1684262837 shutdown_connections 2024-04-26T00:32:22.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.180+0000 7f9f7e123640 1 -- 172.21.15.37:0/1684262837 wait complete. 2024-04-26T00:32:22.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f7e123640 10 monclient: build_initial_monmap 2024-04-26T00:32:22.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f7e123640 10 monclient: monmap: 2024-04-26T00:32:22.187 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:22.187 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:32:22.187 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:32:22.187 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:32:22.187 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:22.188 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:32:22.188 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:32:22.188 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:32:22.188 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:32:22.188 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:22.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f7e123640 1 Processor -- start 2024-04-26T00:32:22.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f7e123640 1 -- start start 2024-04-26T00:32:22.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f7e123640 10 monclient: init 2024-04-26T00:32:22.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f7e123640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:22.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f7e123640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:32:22.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f7e123640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9f780a3d90 0x7f9f781cec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:22.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f7e123640 10 monclient(hunting): picked mon.noname-c con 0x7f9f780a3d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:22.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f7e123640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f780a4170 0x7f9f78136ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:22.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f7e123640 10 monclient(hunting): picked mon.noname-a con 0x7f9f780a4170 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:32:22.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f7e123640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f7813c3e0 0x7f9f7813a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:22.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f7e123640 10 monclient(hunting): picked mon.noname-b con 0x7f9f7813c3e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:32:22.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f7e123640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:22.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f7e123640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9f781418c0 con 0x7f9f780a4170 2024-04-26T00:32:22.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f7e123640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:22.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f7e123640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9f78141740 con 0x7f9f7813c3e0 2024-04-26T00:32:22.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f7e123640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:22.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f7e123640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9f78141a40 con 0x7f9f780a3d90 2024-04-26T00:32:22.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f7e123640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:22.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f7e123640 10 monclient(hunting): authenticate will time out at 1723.493078s 2024-04-26T00:32:22.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f777fe640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9f780a3d90 msgr2=0x7f9f781cec70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:32:22.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f777fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9f780a3d90 0x7f9f781cec70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:32:22.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f76ffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f780a4170 0x7f9f78136ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:22.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f76ffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f780a4170 0x7f9f78136ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:56178/0 (socket says 172.21.15.37:56178) 2024-04-26T00:32:22.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f76ffd640 1 -- 172.21.15.37:0/1898446546 learned_addr learned my addr 172.21.15.37:0/1898446546 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:32:22.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f76ffd640 10 monclient(hunting): get_auth_request con 0x7f9f780a4170 auth_method 0 2024-04-26T00:32:22.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f76ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:22.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f76ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:22.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:22.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f76ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:22.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f76ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:22.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f76ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:22.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f76ffd640 10 monclient(hunting): handle_auth_done global_id 144162 payload 824 2024-04-26T00:32:22.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f77fff640 1 --2- 172.21.15.37:0/1898446546 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f7813c3e0 0x7f9f7813a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:22.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f76ffd640 1 -- 172.21.15.37:0/1898446546 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9f780a3d90 msgr2=0x7f9f781cec70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:32:22.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f76ffd640 1 --2- 172.21.15.37:0/1898446546 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9f780a3d90 0x7f9f781cec70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:22.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f76ffd640 1 -- 172.21.15.37:0/1898446546 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f7813c3e0 msgr2=0x7f9f7813a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:22.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f76ffd640 1 --2- 172.21.15.37:0/1898446546 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f7813c3e0 0x7f9f7813a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:22.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f76ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:32:22.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f76ffd640 1 monclient: found mon.noname-a 2024-04-26T00:32:22.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f76ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:22.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f76ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:32:22.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f76ffd640 1 -- 172.21.15.37:0/1898446546 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f78121870 con 0x7f9f780a4170 2024-04-26T00:32:22.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f77fff640 10 monclient: get_auth_request con 0x7f9f7813c3e0 auth_method 0 2024-04-26T00:32:22.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f77fff640 1 --2- 172.21.15.37:0/1898446546 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f7813c3e0 0x7f9f7813a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:32:22.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f76ffd640 1 -- 172.21.15.37:0/1898446546 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f780a4170 msgr2=0x7f9f78136ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:22.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f76ffd640 1 --2- 172.21.15.37:0/1898446546 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f780a4170 0x7f9f78136ee0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9f6000d800 tx=0x7f9f6000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:22.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f6ffff640 1 -- 172.21.15.37:0/1898446546 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f6000b8d0 con 0x7f9f780a4170 2024-04-26T00:32:22.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f6ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:22.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f6ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:32:22.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f6ffff640 10 monclient: dump: 2024-04-26T00:32:22.194 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:22.194 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:22.194 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:22.195 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:22.195 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:22.195 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:22.195 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:22.195 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:22.195 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:22.195 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:22.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f6ffff640 10 monclient: _finish_auth 0 2024-04-26T00:32:22.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f6ffff640 10 monclient: _check_auth_tickets 2024-04-26T00:32:22.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f6ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:22.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f6ffff640 1 -- 172.21.15.37:0/1898446546 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f60019070 con 0x7f9f780a4170 2024-04-26T00:32:22.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f6ffff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:22.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f6ffff640 1 -- 172.21.15.37:0/1898446546 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f6000bcf0 con 0x7f9f780a4170 2024-04-26T00:32:22.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.184+0000 7f9f6ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:22.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f6ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:22.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f6ffff640 10 monclient: dump: 2024-04-26T00:32:22.196 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:22.196 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:22.196 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:22.196 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:22.196 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:22.196 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:22.196 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:22.197 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:22.197 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:22.197 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:22.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f75ffb640 4 set_mon_vals no callback set 2024-04-26T00:32:22.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f7e123640 5 monclient: authenticate success, global_id 144162 2024-04-26T00:32:22.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f7e123640 10 monclient: _renew_subs 2024-04-26T00:32:22.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f7e123640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:22.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f7e123640 1 -- 172.21.15.37:0/1898446546 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f7813bea0 con 0x7f9f780a4170 2024-04-26T00:32:22.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f7e123640 10 monclient: _renew_subs 2024-04-26T00:32:22.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f7e123640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:22.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f7e123640 1 -- 172.21.15.37:0/1898446546 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f78120fd0 con 0x7f9f780a4170 2024-04-26T00:32:22.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f7e123640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:22.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f7e123640 1 --2- 172.21.15.37:0/1898446546 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f7813a580 0x7f9f7813a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:22.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f7e123640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:22.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f7e123640 1 -- 172.21.15.37:0/1898446546 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9f7813cba0 con 0x7f9f7813a580 2024-04-26T00:32:22.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f777fe640 1 --2- 172.21.15.37:0/1898446546 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f7813a580 0x7f9f7813a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:22.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f777fe640 10 monclient: get_auth_request con 0x7f9f7813a580 auth_method 0 2024-04-26T00:32:22.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:22.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f777fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:22.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f777fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:22.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:22.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:22.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f6ffff640 1 -- 172.21.15.37:0/1898446546 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9f6001e460 con 0x7f9f780a4170 2024-04-26T00:32:22.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:22.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f777fe640 10 monclient(hunting): handle_auth_done global_id 144165 payload 824 2024-04-26T00:32:22.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f777fe640 1 -- 172.21.15.37:0/1898446546 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9f7804f510 con 0x7f9f7813a580 2024-04-26T00:32:22.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f6ffff640 1 --2- 172.21.15.37:0/1898446546 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9f58035e60 0x7f9f58038320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:22.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f6ffff640 1 -- 172.21.15.37:0/1898446546 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(470..470 src has 370..470) ==== 3915+0+0 (secure 0 0 0) 0x7f9f60049030 con 0x7f9f780a4170 2024-04-26T00:32:22.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f77fff640 1 --2- 172.21.15.37:0/1898446546 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9f58035e60 0x7f9f58038320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:22.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f77fff640 10 monclient: get_auth_request con 0x7f9f58035e60 auth_method 0 2024-04-26T00:32:22.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f777fe640 1 -- 172.21.15.37:0/1898446546 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f7813a580 msgr2=0x7f9f7813a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:22.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f777fe640 1 --2- 172.21.15.37:0/1898446546 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f7813a580 0x7f9f7813a960 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9f7000c9f0 tx=0x7f9f7000ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:22.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f6ffff640 1 -- 172.21.15.37:0/1898446546 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f70012400 con 0x7f9f7813a580 2024-04-26T00:32:22.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f6ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:22.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f6ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:22.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f6ffff640 10 monclient: dump: 2024-04-26T00:32:22.202 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:22.203 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:22.203 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:22.203 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:22.203 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:22.203 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:22.203 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:22.203 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:22.203 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:22.203 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:22.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f6ffff640 1 -- 172.21.15.37:0/1898446546 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9f7000ae80 con 0x7f9f7813a580 2024-04-26T00:32:22.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f6ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:22.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f6ffff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:32:22.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f6ffff640 1 -- 172.21.15.37:0/1898446546 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f7813a580 msgr2=0x7f9f7813a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:22.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f6ffff640 1 --2- 172.21.15.37:0/1898446546 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f7813a580 0x7f9f7813a960 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9f7000c9f0 tx=0x7f9f7000ceb0 comp rx=0 tx=0).stop 2024-04-26T00:32:22.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f6ffff640 1 -- 172.21.15.37:0/1898446546 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f7813a580 msgr2=0x7f9f7813a960 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:32:22.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.188+0000 7f9f6ffff640 1 --2- 172.21.15.37:0/1898446546 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f7813a580 0x7f9f7813a960 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9f7000c9f0 tx=0x7f9f7000ceb0 comp rx=0 tx=0).stop 2024-04-26T00:32:22.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.192+0000 7f9f77fff640 1 --2- 172.21.15.37:0/1898446546 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9f58035e60 0x7f9f58038320 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9f780a57e0 tx=0x7f9f68008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:22.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.196+0000 7f9f7e123640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:32:22.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.196+0000 7f9f7e123640 1 --2- 172.21.15.37:0/1898446546 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f7813aea0 0x7f9f7809e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:22.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.196+0000 7f9f7e123640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:22.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.196+0000 7f9f7e123640 1 -- 172.21.15.37:0/1898446546 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9f78075dc0 con 0x7f9f7813aea0 2024-04-26T00:32:22.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.196+0000 7f9f777fe640 1 --2- 172.21.15.37:0/1898446546 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f7813aea0 0x7f9f7809e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:22.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.196+0000 7f9f777fe640 10 monclient: get_auth_request con 0x7f9f7813aea0 auth_method 0 2024-04-26T00:32:22.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.196+0000 7f9f777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:22.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.196+0000 7f9f777fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:22.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.196+0000 7f9f777fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:22.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.196+0000 7f9f777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:22.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.196+0000 7f9f777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:22.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.196+0000 7f9f777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:22.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.196+0000 7f9f777fe640 10 monclient(hunting): handle_auth_done global_id 144168 payload 824 2024-04-26T00:32:22.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.196+0000 7f9f777fe640 1 -- 172.21.15.37:0/1898446546 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9f78137420 con 0x7f9f7813aea0 2024-04-26T00:32:22.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.200+0000 7f9f777fe640 1 -- 172.21.15.37:0/1898446546 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f7813aea0 msgr2=0x7f9f7809e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:22.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.200+0000 7f9f777fe640 1 --2- 172.21.15.37:0/1898446546 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f7813aea0 0x7f9f7809e080 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9f7000c9f0 tx=0x7f9f70004cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:22.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.200+0000 7f9f6ffff640 1 -- 172.21.15.37:0/1898446546 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f70012aa0 con 0x7f9f7813aea0 2024-04-26T00:32:22.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.200+0000 7f9f6ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:22.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.200+0000 7f9f6ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:22.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.200+0000 7f9f6ffff640 10 monclient: dump: 2024-04-26T00:32:22.205 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:22.205 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:22.205 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:22.205 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:22.205 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:22.205 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:22.205 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:22.206 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:22.206 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:22.206 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:22.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.200+0000 7f9f6ffff640 1 -- 172.21.15.37:0/1898446546 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4707 (secure 0 0 0) 0x7f9f70012aa0 con 0x7f9f7813aea0 2024-04-26T00:32:22.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.200+0000 7f9f6ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:32:22.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.200+0000 7f9f6ffff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:32:22.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.200+0000 7f9f6ffff640 1 -- 172.21.15.37:0/1898446546 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f7813aea0 msgr2=0x7f9f7809e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:22.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.200+0000 7f9f6ffff640 1 --2- 172.21.15.37:0/1898446546 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f7813aea0 0x7f9f7809e080 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9f7000c9f0 tx=0x7f9f70004cc0 comp rx=0 tx=0).stop 2024-04-26T00:32:22.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.200+0000 7f9f6ffff640 1 -- 172.21.15.37:0/1898446546 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f7813aea0 msgr2=0x7f9f7809e080 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:32:22.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.200+0000 7f9f6ffff640 1 --2- 172.21.15.37:0/1898446546 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f7813aea0 0x7f9f7809e080 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9f7000c9f0 tx=0x7f9f70004cc0 comp rx=0 tx=0).stop 2024-04-26T00:32:22.206 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:32:22.207 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:32:22.207 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:32:22.207 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:32:22.207 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 94, 2024-04-26T00:32:22.207 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 28743, 2024-04-26T00:32:22.207 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:32:22.207 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:32:22.207 INFO:teuthology.orchestra.run.smithi037.stdout: 1 2024-04-26T00:32:22.207 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:22.207 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:32:22.207 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:32:22.207 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:32:22.207 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:32:22.208 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:22.208 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:22.208 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:22.208 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:22.208 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:22.208 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:22.208 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:22.208 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:22.208 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:22.208 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:22.208 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:22.208 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:22.208 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:32:22.208 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:32:22.208 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:22.209 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:22.209 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:22.209 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:22.209 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:22.209 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:22.209 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:22.209 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:22.209 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:22.209 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:22.209 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:22.209 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:22.209 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:22.209 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:32:22.209 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:32:22.210 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:32:22.210 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:32:22.210 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:32:22.210 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:32:22.210 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:32:22.210 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:32:22.210 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:32:22.210 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:32:22.210 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:32:22.210 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:32:22.210 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:32:22.210 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:32:22.210 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:32:22.210 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:32:22.211 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:32:22.211 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:22.211 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:22.211 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:22.211 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:22.211 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:22.211 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:22.211 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:22.211 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:22.211 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:22.211 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:22.211 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:22.211 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:22.211 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:32:22.211 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:22.212 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:32:22.212 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:22.212 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:32:22.212 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:32:22.212 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:22.212 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:22.212 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:22.212 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:22.212 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:32:22.212 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:22.212 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:22.212 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:22.212 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:22.213 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:32:22.213 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:22.213 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:22.213 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:22.213 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:22.213 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:32:22.213 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:32:22.213 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:22.213 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:22.213 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:22.213 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:22.213 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:22.213 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:32:22.213 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:32:22.213 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:22.214 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:22.214 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:22.214 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:22.214 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:32:22.214 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:22.214 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:22.214 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:22.214 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:22.214 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:32:22.214 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:22.214 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:22.214 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:22.214 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:22.214 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:32:22.214 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:32:22.215 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:22.215 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:22.215 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:22.215 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:22.215 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:22.215 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:32:22.215 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:32:22.215 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:22.215 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:22.215 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:22.215 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:22.215 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:32:22.215 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:22.215 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:22.215 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:22.216 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:22.216 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:32:22.216 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:22.216 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:22.216 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:22.216 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:22.216 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:32:22.216 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:32:22.216 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:22.216 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:22.216 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:22.216 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:22.216 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:22.216 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:22.216 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:32:22.217 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:32:22.217 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:22.217 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:22.217 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:22.217 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:32:22.217 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:22.217 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:22.217 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:32:22.217 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:22.217 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:22.217 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:22.217 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:32:22.217 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:22.217 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:22.218 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:32:22.218 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:22.218 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:22.218 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:22.218 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:32:22.218 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:22.218 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:22.218 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:22.218 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:32:22.218 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:32:22.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.200+0000 7f9f7e123640 1 -- 172.21.15.37:0/1898446546 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9f58035e60 msgr2=0x7f9f58038320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:22.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.200+0000 7f9f7e123640 1 --2- 172.21.15.37:0/1898446546 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9f58035e60 0x7f9f58038320 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9f780a57e0 tx=0x7f9f68008040 comp rx=0 tx=0).stop 2024-04-26T00:32:22.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.200+0000 7f9f7e123640 10 monclient: shutdown 2024-04-26T00:32:22.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.200+0000 7f9f7e123640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:22.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.200+0000 7f9f7e123640 1 -- 172.21.15.37:0/1898446546 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f780a4170 msgr2=0x7f9f78136ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:22.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.200+0000 7f9f7e123640 1 --2- 172.21.15.37:0/1898446546 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f780a4170 0x7f9f78136ee0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9f6000d800 tx=0x7f9f6000dcc0 comp rx=0 tx=0).stop 2024-04-26T00:32:22.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.200+0000 7f9f77fff640 1 -- 172.21.15.37:0/1898446546 reap_dead start 2024-04-26T00:32:22.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.200+0000 7f9f77fff640 1 -- 172.21.15.37:0/1898446546 reap_dead start 2024-04-26T00:32:22.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.200+0000 7f9f7e123640 1 -- 172.21.15.37:0/1898446546 shutdown_connections 2024-04-26T00:32:22.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.200+0000 7f9f7e123640 1 -- 172.21.15.37:0/1898446546 >> 172.21.15.37:0/1898446546 conn(0x7f9f780aa0a0 msgr2=0x7f9f780a8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:22.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.200+0000 7f9f7e123640 1 -- 172.21.15.37:0/1898446546 shutdown_connections 2024-04-26T00:32:22.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.200+0000 7f9f7e123640 1 -- 172.21.15.37:0/1898446546 wait complete. 2024-04-26T00:32:22.221 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:32:22.221 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-04-26T00:32:22.221 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:32:22.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.340+0000 7fe485eee640 10 monclient: get_monmap_and_config 2024-04-26T00:32:22.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.340+0000 7fe485eee640 10 monclient: build_initial_monmap 2024-04-26T00:32:22.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.340+0000 7fe485eee640 10 monclient: monmap: 2024-04-26T00:32:22.337 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:22.337 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:22.337 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:32:22.347047+0000 2024-04-26T00:32:22.337 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:32:22.347047+0000 2024-04-26T00:32:22.337 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:22.338 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:22.338 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:32:22.338 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:32:22.338 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:32:22.338 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:32:22.338 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:32:22.338 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:22.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe485eee640 1 Processor -- start 2024-04-26T00:32:22.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe485eee640 1 -- start start 2024-04-26T00:32:22.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe485eee640 10 monclient: init 2024-04-26T00:32:22.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe485eee640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:22.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe485eee640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-04-26T00:32:22.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe485eee640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe4800a3d90 0x7fe48009bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:22.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe485eee640 10 monclient(hunting): picked mon.noname-b con 0x7fe4800a3d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:22.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe485eee640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fe4800a4170 0x7fe4800a45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:22.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe485eee640 10 monclient(hunting): picked mon.noname-a con 0x7fe4800a4170 addr v2:172.21.15.37:3300/0 2024-04-26T00:32:22.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe485eee640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe480133b20 0x7fe480133f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:22.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe485eee640 10 monclient(hunting): picked mon.noname-c con 0x7fe480133b20 addr v2:172.21.15.45:3300/0 2024-04-26T00:32:22.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe485eee640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:22.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe485eee640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fe4800a5150 con 0x7fe4800a4170 2024-04-26T00:32:22.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe485eee640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:22.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe485eee640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fe48009b280 con 0x7fe480133b20 2024-04-26T00:32:22.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe485eee640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:22.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe485eee640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe48009b560 con 0x7fe4800a3d90 2024-04-26T00:32:22.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe485eee640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:22.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe485eee640 10 monclient(hunting): authenticate will time out at 1723.652879s 2024-04-26T00:32:22.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe47f7fe640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe4800a3d90 msgr2=0x7fe48009bf40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:32:22.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe47f7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe4800a3d90 0x7fe48009bf40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:32:22.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe47effd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fe4800a4170 0x7fe4800a45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:22.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe47effd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fe4800a4170 0x7fe4800a45b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:56200/0 (socket says 172.21.15.37:56200) 2024-04-26T00:32:22.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe47effd640 1 -- 172.21.15.37:0/1295999004 learned_addr learned my addr 172.21.15.37:0/1295999004 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:32:22.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe47effd640 10 monclient(hunting): get_auth_request con 0x7fe4800a4170 auth_method 0 2024-04-26T00:32:22.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe47effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:22.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe47effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:22.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe47effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:22.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe47effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:22.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe47effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:22.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe47effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:22.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe47ffff640 1 --2- 172.21.15.37:0/1295999004 >> v2:172.21.15.45:3300/0 conn(0x7fe480133b20 0x7fe480133f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:22.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe47ffff640 10 monclient(hunting): get_auth_request con 0x7fe480133b20 auth_method 0 2024-04-26T00:32:22.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe47ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:22.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe47ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:22.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe47ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:22.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe47effd640 10 monclient(hunting): handle_auth_done global_id 144171 payload 274 2024-04-26T00:32:22.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe47effd640 1 -- 172.21.15.37:0/1295999004 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe4800a3d90 msgr2=0x7fe48009bf40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:32:22.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe47effd640 1 --2- 172.21.15.37:0/1295999004 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe4800a3d90 0x7fe48009bf40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:22.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe47effd640 1 -- 172.21.15.37:0/1295999004 >> v2:172.21.15.45:3300/0 conn(0x7fe480133b20 msgr2=0x7fe480133f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:22.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe47effd640 1 --2- 172.21.15.37:0/1295999004 >> v2:172.21.15.45:3300/0 conn(0x7fe480133b20 0x7fe480133f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:22.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe47effd640 10 monclient: _finish_hunting 0 2024-04-26T00:32:22.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe47effd640 1 monclient: found mon.noname-a 2024-04-26T00:32:22.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe47effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:22.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe47effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:32:22.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.344+0000 7fe47effd640 1 -- 172.21.15.37:0/1295999004 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4800a4af0 con 0x7fe4800a4170 2024-04-26T00:32:22.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.348+0000 7fe47effd640 1 -- 172.21.15.37:0/1295999004 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4800a4170 msgr2=0x7fe4800a45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:22.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.348+0000 7fe47effd640 1 --2- 172.21.15.37:0/1295999004 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4800a4170 0x7fe4800a45b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe480076100 tx=0x7fe47001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc031c8430817df8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:22.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.348+0000 7fe47e7fc640 1 -- 172.21.15.37:0/1295999004 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4700b0070 con 0x7fe4800a4170 2024-04-26T00:32:22.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.348+0000 7fe47e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:22.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.348+0000 7fe47e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:32:22.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.348+0000 7fe47e7fc640 10 monclient: dump: 2024-04-26T00:32:22.348 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:22.348 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:22.348 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:22.349 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:22.349 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:22.349 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:22.349 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:22.349 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:22.349 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:22.349 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:22.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.348+0000 7fe47e7fc640 10 monclient: _finish_auth 0 2024-04-26T00:32:22.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.348+0000 7fe47e7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:32:22.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.348+0000 7fe47e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:22.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.348+0000 7fe485eee640 5 monclient: authenticate success, global_id 144171 2024-04-26T00:32:22.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.348+0000 7fe485eee640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:32:22.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.348+0000 7fe47e7fc640 1 -- 172.21.15.37:0/1295999004 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4700ab600 con 0x7fe4800a4170 2024-04-26T00:32:22.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.348+0000 7fe47e7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:22.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.348+0000 7fe485eee640 10 monclient: get_monmap_and_config success 2024-04-26T00:32:22.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.348+0000 7fe485eee640 4 set_mon_vals no callback set 2024-04-26T00:32:22.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.348+0000 7fe47e7fc640 1 -- 172.21.15.37:0/1295999004 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4700ab9d0 con 0x7fe4800a4170 2024-04-26T00:32:22.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.348+0000 7fe47e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:22.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.348+0000 7fe47e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:22.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.348+0000 7fe47e7fc640 10 monclient: dump: 2024-04-26T00:32:22.350 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:22.350 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:22.351 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:22.351 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:22.351 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:22.351 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:22.351 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:22.351 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:22.351 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:22.351 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:22.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.348+0000 7fe485eee640 10 monclient: shutdown 2024-04-26T00:32:22.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.348+0000 7fe485eee640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:22.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.348+0000 7fe485eee640 1 -- 172.21.15.37:0/1295999004 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4800a4170 msgr2=0x7fe4800a45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:22.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.348+0000 7fe485eee640 1 --2- 172.21.15.37:0/1295999004 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4800a4170 0x7fe4800a45b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe480076100 tx=0x7fe47001c400 comp rx=0 tx=0).stop 2024-04-26T00:32:22.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.348+0000 7fe485eee640 1 -- 172.21.15.37:0/1295999004 shutdown_connections 2024-04-26T00:32:22.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.348+0000 7fe485eee640 1 --2- 172.21.15.37:0/1295999004 >> v2:172.21.15.45:3300/0 conn(0x7fe480133b20 0x7fe480133f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:22.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.348+0000 7fe485eee640 1 --2- 172.21.15.37:0/1295999004 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4800a4170 0x7fe4800a45b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:22.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.348+0000 7fe485eee640 1 --2- 172.21.15.37:0/1295999004 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe4800a3d90 0x7fe48009bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:22.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.348+0000 7fe485eee640 1 -- 172.21.15.37:0/1295999004 >> 172.21.15.37:0/1295999004 conn(0x7fe4800aa0a0 msgr2=0x7fe4800a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:22.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.348+0000 7fe485eee640 1 -- 172.21.15.37:0/1295999004 shutdown_connections 2024-04-26T00:32:22.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.348+0000 7fe485eee640 1 -- 172.21.15.37:0/1295999004 wait complete. 2024-04-26T00:32:22.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.348+0000 7fe485eee640 10 monclient: build_initial_monmap 2024-04-26T00:32:22.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.348+0000 7fe485eee640 10 monclient: monmap: 2024-04-26T00:32:22.352 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:22.352 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:32:22.352 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:32:22.352 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:32:22.352 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:22.352 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:32:22.353 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:32:22.353 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:32:22.353 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:32:22.353 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:22.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.348+0000 7fe485eee640 1 Processor -- start 2024-04-26T00:32:22.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.348+0000 7fe485eee640 1 -- start start 2024-04-26T00:32:22.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.348+0000 7fe485eee640 10 monclient: init 2024-04-26T00:32:22.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe485eee640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:22.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe485eee640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:32:22.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe485eee640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4800a3d90 0x7fe4801cf1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:22.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe485eee640 10 monclient(hunting): picked mon.noname-a con 0x7fe4800a3d90 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:32:22.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe485eee640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe4800a4170 0x7fe4801cf700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:22.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe485eee640 10 monclient(hunting): picked mon.noname-c con 0x7fe4800a4170 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:22.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe485eee640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe480133b20 0x7fe4801cfc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:22.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe485eee640 10 monclient(hunting): picked mon.noname-b con 0x7fe480133b20 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:32:22.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe485eee640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:22.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe485eee640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe480140c10 con 0x7fe4800a3d90 2024-04-26T00:32:22.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe485eee640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:22.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe485eee640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe48009c690 con 0x7fe480133b20 2024-04-26T00:32:22.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe485eee640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:22.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe485eee640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe480140d90 con 0x7fe4800a4170 2024-04-26T00:32:22.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe485eee640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:22.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe485eee640 10 monclient(hunting): authenticate will time out at 1723.660280s 2024-04-26T00:32:22.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe47effd640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe4800a4170 msgr2=0x7fe4801cf700 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:32:22.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe47effd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe4800a4170 0x7fe4801cf700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:32:22.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe47f7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4800a3d90 0x7fe4801cf1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:22.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe47f7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4800a3d90 0x7fe4801cf1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:56216/0 (socket says 172.21.15.37:56216) 2024-04-26T00:32:22.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe47f7fe640 1 -- 172.21.15.37:0/3158200087 learned_addr learned my addr 172.21.15.37:0/3158200087 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:32:22.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe47f7fe640 10 monclient(hunting): get_auth_request con 0x7fe4800a3d90 auth_method 0 2024-04-26T00:32:22.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe47f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:22.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe47f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:22.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe47f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:22.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe47f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:22.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe47f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:22.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe47f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:22.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe47ffff640 1 --2- 172.21.15.37:0/3158200087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe480133b20 0x7fe4801cfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:22.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe47f7fe640 10 monclient(hunting): handle_auth_done global_id 144174 payload 824 2024-04-26T00:32:22.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe47f7fe640 1 -- 172.21.15.37:0/3158200087 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe4800a4170 msgr2=0x7fe4801cf700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:32:22.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe47f7fe640 1 --2- 172.21.15.37:0/3158200087 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe4800a4170 0x7fe4801cf700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:22.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe47f7fe640 1 -- 172.21.15.37:0/3158200087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe480133b20 msgr2=0x7fe4801cfc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:22.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe47f7fe640 1 --2- 172.21.15.37:0/3158200087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe480133b20 0x7fe4801cfc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:22.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe47f7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:32:22.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe47f7fe640 1 monclient: found mon.noname-a 2024-04-26T00:32:22.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe47f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:22.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe47f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:32:22.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe47f7fe640 1 -- 172.21.15.37:0/3158200087 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe48013b600 con 0x7fe4800a3d90 2024-04-26T00:32:22.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe47ffff640 10 monclient: get_auth_request con 0x7fe480133b20 auth_method 0 2024-04-26T00:32:22.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe47ffff640 1 --2- 172.21.15.37:0/3158200087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe480133b20 0x7fe4801cfc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:32:22.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe47f7fe640 1 -- 172.21.15.37:0/3158200087 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4800a3d90 msgr2=0x7fe4801cf1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:22.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe47f7fe640 1 --2- 172.21.15.37:0/3158200087 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4800a3d90 0x7fe4801cf1c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe46800b9e0 tx=0x7fe46800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:22.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe477fff640 1 -- 172.21.15.37:0/3158200087 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe46800cdb0 con 0x7fe4800a3d90 2024-04-26T00:32:22.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe477fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:22.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe477fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:32:22.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe477fff640 10 monclient: dump: 2024-04-26T00:32:22.361 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:22.361 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:22.361 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:22.361 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:22.361 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:22.361 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:22.361 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:22.362 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:22.362 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:22.362 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:22.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe477fff640 10 monclient: _finish_auth 0 2024-04-26T00:32:22.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe477fff640 10 monclient: _check_auth_tickets 2024-04-26T00:32:22.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe477fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:22.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe477fff640 1 -- 172.21.15.37:0/3158200087 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe468010070 con 0x7fe4800a3d90 2024-04-26T00:32:22.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe477fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:22.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe477fff640 1 -- 172.21.15.37:0/3158200087 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4680156b0 con 0x7fe4800a3d90 2024-04-26T00:32:22.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe485eee640 5 monclient: authenticate success, global_id 144174 2024-04-26T00:32:22.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe485eee640 10 monclient: _renew_subs 2024-04-26T00:32:22.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe485eee640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:22.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe485eee640 1 -- 172.21.15.37:0/3158200087 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe48013c7d0 con 0x7fe4800a3d90 2024-04-26T00:32:22.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe47d7fa640 4 set_mon_vals no callback set 2024-04-26T00:32:22.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe477fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:22.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe477fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:22.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe477fff640 10 monclient: dump: 2024-04-26T00:32:22.363 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:22.363 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:22.363 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:22.363 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:22.363 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:22.363 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:22.363 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:22.363 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:22.364 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:22.364 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:22.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe485eee640 10 monclient: _renew_subs 2024-04-26T00:32:22.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe485eee640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:22.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe485eee640 1 -- 172.21.15.37:0/3158200087 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe48009ada0 con 0x7fe4800a3d90 2024-04-26T00:32:22.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe485eee640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:22.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe485eee640 1 --2- 172.21.15.37:0/3158200087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe480132b10 0x7fe480132ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:22.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe485eee640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:22.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.352+0000 7fe485eee640 1 -- 172.21.15.37:0/3158200087 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe48004f510 con 0x7fe480132b10 2024-04-26T00:32:22.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.356+0000 7fe477fff640 1 -- 172.21.15.37:0/3158200087 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe46801e3f0 con 0x7fe4800a3d90 2024-04-26T00:32:22.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.356+0000 7fe477fff640 1 --2- 172.21.15.37:0/3158200087 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe464035e80 0x7fe464038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:22.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.356+0000 7fe47effd640 1 --2- 172.21.15.37:0/3158200087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe480132b10 0x7fe480132ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:22.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.356+0000 7fe477fff640 1 -- 172.21.15.37:0/3158200087 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(470..470 src has 370..470) ==== 3915+0+0 (secure 0 0 0) 0x7fe468048de0 con 0x7fe4800a3d90 2024-04-26T00:32:22.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.356+0000 7fe47effd640 10 monclient: get_auth_request con 0x7fe480132b10 auth_method 0 2024-04-26T00:32:22.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.356+0000 7fe47effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:22.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.356+0000 7fe47effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:22.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.356+0000 7fe47effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:22.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.356+0000 7fe47effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:22.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.356+0000 7fe47effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:22.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.356+0000 7fe47effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:22.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.356+0000 7fe47ffff640 1 --2- 172.21.15.37:0/3158200087 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe464035e80 0x7fe464038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:22.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.356+0000 7fe47ffff640 10 monclient: get_auth_request con 0x7fe464035e80 auth_method 0 2024-04-26T00:32:22.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.356+0000 7fe47effd640 10 monclient(hunting): handle_auth_done global_id 134242 payload 824 2024-04-26T00:32:22.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.356+0000 7fe47effd640 1 -- 172.21.15.37:0/3158200087 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe48013b380 con 0x7fe480132b10 2024-04-26T00:32:22.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.356+0000 7fe47effd640 1 -- 172.21.15.37:0/3158200087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe480132b10 msgr2=0x7fe480132ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:22.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.356+0000 7fe47effd640 1 --2- 172.21.15.37:0/3158200087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe480132b10 0x7fe480132ef0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe4700ad000 tx=0x7fe47001ac40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:22.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.356+0000 7fe47ffff640 1 --2- 172.21.15.37:0/3158200087 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe464035e80 0x7fe464038340 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe48013ab50 tx=0x7fe478008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:22.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.356+0000 7fe477fff640 1 -- 172.21.15.37:0/3158200087 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4700ba070 con 0x7fe480132b10 2024-04-26T00:32:22.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.356+0000 7fe477fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:22.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.356+0000 7fe477fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:32:22.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.356+0000 7fe477fff640 10 monclient: dump: 2024-04-26T00:32:22.366 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:22.366 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:22.367 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:22.367 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:22.367 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:22.367 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:22.367 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:22.367 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:22.367 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:22.367 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:22.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.360+0000 7fe477fff640 1 -- 172.21.15.37:0/3158200087 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe4700ab9b0 con 0x7fe480132b10 2024-04-26T00:32:22.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.360+0000 7fe477fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:22.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.360+0000 7fe477fff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:32:22.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.360+0000 7fe477fff640 1 -- 172.21.15.37:0/3158200087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe480132b10 msgr2=0x7fe480132ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:22.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.360+0000 7fe477fff640 1 --2- 172.21.15.37:0/3158200087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe480132b10 0x7fe480132ef0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe4700ad000 tx=0x7fe47001ac40 comp rx=0 tx=0).stop 2024-04-26T00:32:22.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.360+0000 7fe477fff640 1 -- 172.21.15.37:0/3158200087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe480132b10 msgr2=0x7fe480132ef0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:32:22.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.360+0000 7fe477fff640 1 --2- 172.21.15.37:0/3158200087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe480132b10 0x7fe480132ef0 secure :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe4700ad000 tx=0x7fe47001ac40 comp rx=0 tx=0).stop 2024-04-26T00:32:22.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.364+0000 7fe485eee640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:32:22.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.364+0000 7fe485eee640 1 --2- 172.21.15.37:0/3158200087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe480133430 0x7fe480134b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:22.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.364+0000 7fe485eee640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:22.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.364+0000 7fe485eee640 1 -- 172.21.15.37:0/3158200087 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe480075dc0 con 0x7fe480133430 2024-04-26T00:32:22.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.364+0000 7fe47effd640 1 --2- 172.21.15.37:0/3158200087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe480133430 0x7fe480134b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:22.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.364+0000 7fe47effd640 10 monclient: get_auth_request con 0x7fe480133430 auth_method 0 2024-04-26T00:32:22.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.364+0000 7fe47effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:22.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.364+0000 7fe47effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:22.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.364+0000 7fe47effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:22.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.364+0000 7fe47effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:22.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.364+0000 7fe47effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:22.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.364+0000 7fe47effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:22.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.364+0000 7fe47effd640 10 monclient(hunting): handle_auth_done global_id 134245 payload 824 2024-04-26T00:32:22.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.364+0000 7fe47effd640 1 -- 172.21.15.37:0/3158200087 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe48013c4f0 con 0x7fe480133430 2024-04-26T00:32:22.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.368+0000 7fe47effd640 1 -- 172.21.15.37:0/3158200087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe480133430 msgr2=0x7fe480134b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:22.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.368+0000 7fe47effd640 1 --2- 172.21.15.37:0/3158200087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe480133430 0x7fe480134b30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe47001aab0 tx=0x7fe4700191a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:22.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.368+0000 7fe477fff640 1 -- 172.21.15.37:0/3158200087 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4700ba040 con 0x7fe480133430 2024-04-26T00:32:22.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.368+0000 7fe477fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:22.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.368+0000 7fe477fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:32:22.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.368+0000 7fe477fff640 10 monclient: dump: 2024-04-26T00:32:22.370 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:22.370 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:22.371 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:22.371 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:22.371 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:22.371 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:22.371 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:22.371 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:22.371 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:22.371 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:22.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.368+0000 7fe477fff640 1 -- 172.21.15.37:0/3158200087 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4875 (secure 0 0 0) 0x7fe470003e40 con 0x7fe480133430 2024-04-26T00:32:22.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.368+0000 7fe477fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:32:22.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.368+0000 7fe477fff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:32:22.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.368+0000 7fe477fff640 1 -- 172.21.15.37:0/3158200087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe480133430 msgr2=0x7fe480134b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:22.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.368+0000 7fe477fff640 1 --2- 172.21.15.37:0/3158200087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe480133430 0x7fe480134b30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe47001aab0 tx=0x7fe4700191a0 comp rx=0 tx=0).stop 2024-04-26T00:32:22.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.368+0000 7fe477fff640 1 -- 172.21.15.37:0/3158200087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe480133430 msgr2=0x7fe480134b30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:32:22.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.368+0000 7fe477fff640 1 --2- 172.21.15.37:0/3158200087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe480133430 0x7fe480134b30 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe47001aab0 tx=0x7fe4700191a0 comp rx=0 tx=0).stop 2024-04-26T00:32:22.372 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:32:22.372 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:32:22.372 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:32:22.372 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:32:22.372 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 94, 2024-04-26T00:32:22.372 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 72731, 2024-04-26T00:32:22.372 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:32:22.372 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:32:22.373 INFO:teuthology.orchestra.run.smithi037.stdout: 1 2024-04-26T00:32:22.373 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:22.373 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:32:22.373 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:32:22.373 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:32:22.373 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:32:22.373 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:22.373 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:22.373 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:22.373 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:22.373 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:22.373 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:22.373 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:22.373 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:22.373 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:22.374 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:22.374 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:22.374 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:22.374 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:32:22.374 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:32:22.374 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:22.374 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:22.374 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:22.374 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:22.374 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:22.374 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:22.374 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:22.374 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:22.375 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:22.375 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:22.375 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:22.375 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:22.375 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:22.375 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:32:22.375 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:32:22.375 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:32:22.375 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:32:22.375 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:32:22.375 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:32:22.375 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:32:22.375 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:32:22.376 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:32:22.376 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:32:22.376 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:32:22.376 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:32:22.376 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:32:22.376 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:32:22.376 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:32:22.376 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:32:22.376 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:32:22.376 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:22.376 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:22.377 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:22.377 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:22.377 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:22.377 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:22.377 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:22.377 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:22.377 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:22.377 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:22.377 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:22.377 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:22.377 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:32:22.377 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:22.377 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:32:22.378 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:22.378 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:32:22.378 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:32:22.378 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:22.378 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:22.378 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:22.378 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:22.378 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:32:22.378 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:22.378 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:22.378 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:22.378 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:22.378 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:32:22.379 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:22.379 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:22.379 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:22.379 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:22.379 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:32:22.379 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:32:22.379 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:22.379 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:22.379 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:22.379 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:22.379 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:22.379 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:32:22.379 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:32:22.379 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:22.380 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:22.380 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:22.380 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:22.380 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:32:22.380 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:22.380 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:22.380 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:22.380 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:22.380 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:32:22.380 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:22.380 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:22.380 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:22.380 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:22.380 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:32:22.381 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:32:22.381 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:22.381 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:22.381 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:22.381 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:22.381 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:22.381 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:32:22.381 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:32:22.381 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:22.381 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:22.381 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:22.381 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:22.381 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:32:22.382 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:22.382 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:22.382 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:22.382 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:22.382 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:32:22.382 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:22.382 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:22.382 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:22.382 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:22.382 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:32:22.382 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:32:22.382 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:22.382 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:22.382 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:22.382 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:22.383 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:22.383 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:22.383 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:32:22.383 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:32:22.383 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:22.383 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:22.383 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:22.383 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:32:22.383 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:22.383 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:22.383 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:32:22.383 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:22.383 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:22.383 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:22.384 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:32:22.384 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:22.384 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:22.384 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:32:22.384 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:22.384 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:22.384 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:22.384 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:32:22.384 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:22.384 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:22.384 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:32:22.384 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:22.384 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:22.384 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:22.385 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:32:22.385 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:22.385 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:22.385 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:22.385 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:32:22.385 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:32:22.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.372+0000 7fe485eee640 1 -- 172.21.15.37:0/3158200087 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe464035e80 msgr2=0x7fe464038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:22.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.372+0000 7fe485eee640 1 --2- 172.21.15.37:0/3158200087 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe464035e80 0x7fe464038340 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe48013ab50 tx=0x7fe478008040 comp rx=0 tx=0).stop 2024-04-26T00:32:22.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.372+0000 7fe485eee640 10 monclient: shutdown 2024-04-26T00:32:22.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.372+0000 7fe485eee640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:22.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.372+0000 7fe485eee640 1 -- 172.21.15.37:0/3158200087 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4800a3d90 msgr2=0x7fe4801cf1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:22.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.372+0000 7fe485eee640 1 --2- 172.21.15.37:0/3158200087 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4800a3d90 0x7fe4801cf1c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe46800b9e0 tx=0x7fe46800bea0 comp rx=0 tx=0).stop 2024-04-26T00:32:22.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.372+0000 7fe47ffff640 1 -- 172.21.15.37:0/3158200087 reap_dead start 2024-04-26T00:32:22.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.372+0000 7fe47ffff640 1 -- 172.21.15.37:0/3158200087 reap_dead start 2024-04-26T00:32:22.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.372+0000 7fe485eee640 1 -- 172.21.15.37:0/3158200087 shutdown_connections 2024-04-26T00:32:22.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.372+0000 7fe485eee640 1 -- 172.21.15.37:0/3158200087 >> 172.21.15.37:0/3158200087 conn(0x7fe4800aa0a0 msgr2=0x7fe4800a2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:22.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.372+0000 7fe485eee640 1 -- 172.21.15.37:0/3158200087 shutdown_connections 2024-04-26T00:32:22.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:22.372+0000 7fe485eee640 1 -- 172.21.15.37:0/3158200087 wait complete. 2024-04-26T00:32:22.388 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:32:22.388 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-04-26T00:32:22.848 INFO:tasks.rados.rados.0.smithi045.stdout:2077: done (1 left) 2024-04-26T00:32:22.848 INFO:tasks.rados.rados.0.smithi045.stdout:2078: snap_create 2024-04-26T00:32:22.849 INFO:tasks.rados.rados.0.smithi045.stdout:2076: expect (ObjNum 703 snap 221 seq_num 703) 2024-04-26T00:32:23.472 INFO:tasks.rados.rados.0.smithi045.stdout:2076: done (1 left) 2024-04-26T00:32:23.472 INFO:tasks.rados.rados.0.smithi045.stdout:2078: done (0 left) 2024-04-26T00:32:23.472 INFO:tasks.rados.rados.0.smithi045.stdout:2079: rollback oid 32 current snap is 234 2024-04-26T00:32:23.472 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 32 to 205 2024-04-26T00:32:23.472 INFO:tasks.rados.rados.0.smithi045.stdout:2080: rollback oid 40 current snap is 234 2024-04-26T00:32:23.472 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 40 to 175 2024-04-26T00:32:23.472 INFO:tasks.rados.rados.0.smithi045.stdout:2081: read oid 34 snap 219 2024-04-26T00:32:23.472 INFO:tasks.rados.rados.0.smithi045.stdout:2081: expect (ObjNum 698 snap 219 seq_num 698) 2024-04-26T00:32:23.472 INFO:tasks.rados.rados.0.smithi045.stdout:2082: copy_from oid 38 from oid 35 current snap is 234 2024-04-26T00:32:23.472 INFO:tasks.rados.rados.0.smithi045.stdout:2083: copy_from oid 17 from oid 37 current snap is 234 2024-04-26T00:32:23.473 INFO:tasks.rados.rados.0.smithi045.stdout:2084: delete oid 31 current snap is 234 2024-04-26T00:32:23.491 INFO:tasks.rados.rados.0.smithi045.stdout:2080: finishing rollback tid 1 to smithi04526579-40 2024-04-26T00:32:23.491 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 1446 (ObjNum 801 snap 0 seq_num 1141729710) dirty dne 2024-04-26T00:32:23.492 INFO:tasks.rados.rados.0.smithi045.stdout:2079: finishing rollback tid 0 to smithi04526579-32 2024-04-26T00:32:23.493 INFO:tasks.rados.rados.0.smithi045.stdout:2079: finishing rollback tid 1 to smithi04526579-32 2024-04-26T00:32:23.498 INFO:tasks.rados.rados.0.smithi045.stdout:2080: done (5 left) 2024-04-26T00:32:23.499 INFO:tasks.rados.rados.0.smithi045.stdout:2084: done (4 left) 2024-04-26T00:32:23.499 INFO:tasks.rados.rados.0.smithi045.stdout:2085: read oid 10 snap -1 2024-04-26T00:32:23.499 INFO:tasks.rados.rados.0.smithi045.stdout:2085: expect (ObjNum 573 snap 176 seq_num 573) 2024-04-26T00:32:23.499 INFO:tasks.rados.rados.0.smithi045.stdout:2086: snap_create 2024-04-26T00:32:23.504 INFO:tasks.rados.rados.0.smithi045.stdout:2079: finishing rollback tid 2 to smithi04526579-32 2024-04-26T00:32:23.504 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 1366 (ObjNum 639 snap 201 seq_num 639) dirty exists 2024-04-26T00:32:23.508 INFO:tasks.rados.rados.0.smithi045.stdout:2081: expect (ObjNum 698 snap 219 seq_num 698) 2024-04-26T00:32:23.778 INFO:tasks.rados.rados.0.smithi045.stdout:2083: finishing copy_from racing read to smithi04526579-17 2024-04-26T00:32:23.778 INFO:tasks.rados.rados.0.smithi045.stdout:2082: finishing copy_from racing read to smithi04526579-38 2024-04-26T00:32:23.778 INFO:tasks.rados.rados.0.smithi045.stdout:2082: finishing copy_from to smithi04526579-38 2024-04-26T00:32:23.778 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 1288 (ObjNum 717 snap 222 seq_num 717) dirty exists 2024-04-26T00:32:23.778 INFO:tasks.rados.rados.0.smithi045.stdout:2085: expect (ObjNum 573 snap 176 seq_num 573) 2024-04-26T00:32:23.898 INFO:tasks.rados.rados.0.smithi045.stdout:2083: finishing copy_from to smithi04526579-17 2024-04-26T00:32:23.899 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 1509 (ObjNum 669 snap 210 seq_num 669) dirty exists 2024-04-26T00:32:24.481 INFO:tasks.rados.rados.0.smithi045.stdout:2079: done (5 left) 2024-04-26T00:32:24.481 INFO:tasks.rados.rados.0.smithi045.stdout:2081: done (4 left) 2024-04-26T00:32:24.481 INFO:tasks.rados.rados.0.smithi045.stdout:2082: done (3 left) 2024-04-26T00:32:24.481 INFO:tasks.rados.rados.0.smithi045.stdout:2083: done (2 left) 2024-04-26T00:32:24.481 INFO:tasks.rados.rados.0.smithi045.stdout:2085: done (1 left) 2024-04-26T00:32:24.481 INFO:tasks.rados.rados.0.smithi045.stdout:2086: done (0 left) 2024-04-26T00:32:24.481 INFO:tasks.rados.rados.0.smithi045.stdout:2087: delete oid 30 current snap is 235 2024-04-26T00:32:24.485 INFO:tasks.rados.rados.0.smithi045.stdout:2087: done (0 left) 2024-04-26T00:32:24.485 INFO:tasks.rados.rados.0.smithi045.stdout:2088: snap_create 2024-04-26T00:32:25.482 INFO:tasks.rados.rados.0.smithi045.stdout:2088: done (0 left) 2024-04-26T00:32:25.483 INFO:tasks.rados.rados.0.smithi045.stdout:2089: snap_remove snap 224 2024-04-26T00:32:26.490 INFO:tasks.rados.rados.0.smithi045.stdout:2089: done (0 left) 2024-04-26T00:32:26.490 INFO:tasks.rados.rados.0.smithi045.stdout:2090: read oid 21 snap -1 2024-04-26T00:32:26.491 INFO:tasks.rados.rados.0.smithi045.stdout:2090: expect (ObjNum 756 snap 232 seq_num 756) 2024-04-26T00:32:26.491 INFO:tasks.rados.rados.0.smithi045.stdout:2091: delete oid 3 current snap is 236 2024-04-26T00:32:26.510 INFO:tasks.rados.rados.0.smithi045.stdout:2091: done (1 left) 2024-04-26T00:32:26.511 INFO:tasks.rados.rados.0.smithi045.stdout:2092: snap_remove snap 199 2024-04-26T00:32:27.499 INFO:tasks.rados.rados.0.smithi045.stdout:2092: done (1 left) 2024-04-26T00:32:27.499 INFO:tasks.rados.rados.0.smithi045.stdout:2093: rollback oid 17 current snap is 236 2024-04-26T00:32:27.500 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 17 to 233 2024-04-26T00:32:27.500 INFO:tasks.rados.rados.0.smithi045.stdout:2094: rollback oid 28 current snap is 236 2024-04-26T00:32:27.500 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 28 to 219 2024-04-26T00:32:27.500 INFO:tasks.rados.rados.0.smithi045.stdout:2095: read oid 27 snap -1 2024-04-26T00:32:27.500 INFO:tasks.rados.rados.0.smithi045.stdout:2095: expect (ObjNum 750 snap 232 seq_num 750) 2024-04-26T00:32:27.500 INFO:tasks.rados.rados.0.smithi045.stdout:2096: copy_from oid 33 from oid 47 current snap is 236 2024-04-26T00:32:27.500 INFO:tasks.rados.rados.0.smithi045.stdout:2097: snap_remove snap 212 2024-04-26T00:32:28.475 INFO:tasks.rados.rados.0.smithi045.stdout:2097: done (5 left) 2024-04-26T00:32:28.475 INFO:tasks.rados.rados.0.smithi045.stdout:2098: delete oid 2 current snap is 236 2024-04-26T00:32:28.476 INFO:tasks.rados.rados.0.smithi045.stdout:2090: expect (ObjNum 756 snap 232 seq_num 756) 2024-04-26T00:32:28.606 INFO:tasks.rados.rados.0.smithi045.stdout:2094: finishing rollback tid 0 to smithi04526579-28 2024-04-26T00:32:28.606 INFO:tasks.rados.rados.0.smithi045.stdout:2094: finishing rollback tid 1 to smithi04526579-28 2024-04-26T00:32:28.606 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 1356 (ObjNum 305 snap 0 seq_num 907320832) dirty dne 2024-04-26T00:32:28.606 INFO:tasks.rados.rados.0.smithi045.stdout:2093: finishing rollback tid 0 to smithi04526579-17 2024-04-26T00:32:28.606 INFO:tasks.rados.rados.0.smithi045.stdout:2093: finishing rollback tid 1 to smithi04526579-17 2024-04-26T00:32:28.607 INFO:tasks.rados.rados.0.smithi045.stdout:2093: finishing rollback tid 2 to smithi04526579-17 2024-04-26T00:32:28.607 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 1512 (ObjNum 758 snap 232 seq_num 758) dirty exists 2024-04-26T00:32:28.607 INFO:tasks.rados.rados.0.smithi045.stdout:2095: expect (ObjNum 750 snap 232 seq_num 750) 2024-04-26T00:32:28.790 INFO:tasks.rados.rados.0.smithi045.stdout:2096: finishing copy_from racing read to smithi04526579-33 2024-04-26T00:32:28.790 INFO:tasks.rados.rados.0.smithi045.stdout:2096: finishing copy_from to smithi04526579-33 2024-04-26T00:32:28.790 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 1085 (ObjNum 683 snap 213 seq_num 683) dirty exists 2024-04-26T00:32:28.790 INFO:tasks.rados.rados.0.smithi045.stdout:2090: done (5 left) 2024-04-26T00:32:28.790 INFO:tasks.rados.rados.0.smithi045.stdout:2093: done (4 left) 2024-04-26T00:32:28.790 INFO:tasks.rados.rados.0.smithi045.stdout:2094: done (3 left) 2024-04-26T00:32:28.790 INFO:tasks.rados.rados.0.smithi045.stdout:2095: done (2 left) 2024-04-26T00:32:28.790 INFO:tasks.rados.rados.0.smithi045.stdout:2096: done (1 left) 2024-04-26T00:32:28.790 INFO:tasks.rados.rados.0.smithi045.stdout:2098: done (0 left) 2024-04-26T00:32:28.790 INFO:tasks.rados.rados.0.smithi045.stdout:2099: read oid 40 snap -1 2024-04-26T00:32:28.791 INFO:tasks.rados.rados.0.smithi045.stdout:2099: expect deleted 2024-04-26T00:32:28.791 INFO:tasks.rados.rados.0.smithi045.stdout:2100: rollback oid 10 current snap is 236 2024-04-26T00:32:28.791 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 10 to 234 2024-04-26T00:32:28.791 INFO:tasks.rados.rados.0.smithi045.stdout:2101: snap_create 2024-04-26T00:32:28.800 INFO:tasks.rados.rados.0.smithi045.stdout:2100: finishing rollback tid 0 to smithi04526579-10 2024-04-26T00:32:28.800 INFO:tasks.rados.rados.0.smithi045.stdout:2100: finishing rollback tid 1 to smithi04526579-10 2024-04-26T00:32:28.801 INFO:tasks.rados.rados.0.smithi045.stdout:2100: finishing rollback tid 2 to smithi04526579-10 2024-04-26T00:32:28.801 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 1515 (ObjNum 573 snap 176 seq_num 573) dirty exists 2024-04-26T00:32:29.511 INFO:tasks.rados.rados.0.smithi045.stdout:2099: done (2 left) 2024-04-26T00:32:29.511 INFO:tasks.rados.rados.0.smithi045.stdout:2100: done (1 left) 2024-04-26T00:32:29.511 INFO:tasks.rados.rados.0.smithi045.stdout:2101: done (0 left) 2024-04-26T00:32:29.511 INFO:tasks.rados.rados.0.smithi045.stdout:2102: write oid 43 current snap is 237 2024-04-26T00:32:29.511 INFO:tasks.rados.rados.0.smithi045.stdout:2102: seq_num 764 ranges {440548=725536,1757537=478262,2813990=26558} 2024-04-26T00:32:29.529 INFO:tasks.rados.rados.0.smithi045.stdout:2102: writing smithi04526579-43 from 440548 to 1166084 tid 1 2024-04-26T00:32:29.546 INFO:tasks.rados.rados.0.smithi045.stdout:2102: writing smithi04526579-43 from 1757537 to 2235799 tid 2 2024-04-26T00:32:29.549 INFO:tasks.rados.rados.0.smithi045.stdout:2102: writing smithi04526579-43 from 2813990 to 2840548 tid 3 2024-04-26T00:32:29.550 INFO:tasks.rados.rados.0.smithi045.stdout:2103: write (excl) oid 32 current snap is 237 2024-04-26T00:32:29.550 INFO:tasks.rados.rados.0.smithi045.stdout:2102: finishing write tid 1 to smithi04526579-43 2024-04-26T00:32:29.550 INFO:tasks.rados.rados.0.smithi045.stdout:2103: seq_num 765 ranges {457355=400785,1416019=621920,2725060=132295} 2024-04-26T00:32:29.555 INFO:tasks.rados.rados.0.smithi045.stdout:2103: writing smithi04526579-32 from 457355 to 858140 tid 1 2024-04-26T00:32:29.570 INFO:tasks.rados.rados.0.smithi045.stdout:2103: writing smithi04526579-32 from 1416019 to 2037939 tid 2 2024-04-26T00:32:29.581 INFO:tasks.rados.rados.0.smithi045.stdout:2103: writing smithi04526579-32 from 2725060 to 2857355 tid 3 2024-04-26T00:32:29.581 INFO:tasks.rados.rados.0.smithi045.stdout:2104: copy_from oid 4 from oid 1 current snap is 237 2024-04-26T00:32:29.581 INFO:tasks.rados.rados.0.smithi045.stdout:2102: finishing write tid 2 to smithi04526579-43 2024-04-26T00:32:29.581 INFO:tasks.rados.rados.0.smithi045.stdout:2102: finishing write tid 3 to smithi04526579-43 2024-04-26T00:32:29.581 INFO:tasks.rados.rados.0.smithi045.stdout:2102: finishing write tid 4 to smithi04526579-43 2024-04-26T00:32:29.581 INFO:tasks.rados.rados.0.smithi045.stdout:2102: finishing write tid 5 to smithi04526579-43 2024-04-26T00:32:29.581 INFO:tasks.rados.rados.0.smithi045.stdout:2102: oid 43 updating version 0 to 1517 2024-04-26T00:32:29.581 INFO:tasks.rados.rados.0.smithi045.stdout:2102: oid 43 updating version 1517 to 1518 2024-04-26T00:32:29.581 INFO:tasks.rados.rados.0.smithi045.stdout:2102: oid 43 updating version 1518 to 1519 2024-04-26T00:32:29.581 INFO:tasks.rados.rados.0.smithi045.stdout:2102: oid 43 version 1519 is already newer than 1516 2024-04-26T00:32:29.581 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 1519 (ObjNum 764 snap 237 seq_num 764) dirty exists 2024-04-26T00:32:29.581 INFO:tasks.rados.rados.0.smithi045.stdout:2102: left oid 43 (ObjNum 764 snap 237 seq_num 764) 2024-04-26T00:32:29.581 INFO:tasks.rados.rados.0.smithi045.stdout:2103: finishing write tid 1 to smithi04526579-32 2024-04-26T00:32:29.582 INFO:tasks.rados.rados.0.smithi045.stdout:2103: finishing write tid 2 to smithi04526579-32 2024-04-26T00:32:29.582 INFO:tasks.rados.rados.0.smithi045.stdout:2102: done (2 left) 2024-04-26T00:32:29.582 INFO:tasks.rados.rados.0.smithi045.stdout:2105: write oid 7 current snap is 237 2024-04-26T00:32:29.582 INFO:tasks.rados.rados.0.smithi045.stdout:2105: seq_num 766 ranges {474162=476034,1358148=481931,2352483=465303} 2024-04-26T00:32:29.590 INFO:tasks.rados.rados.0.smithi045.stdout:2105: writing smithi04526579-7 from 474162 to 950196 tid 1 2024-04-26T00:32:29.599 INFO:tasks.rados.rados.0.smithi045.stdout:2105: writing smithi04526579-7 from 1358148 to 1840079 tid 2 2024-04-26T00:32:29.607 INFO:tasks.rados.rados.0.smithi045.stdout:2105: writing smithi04526579-7 from 2352483 to 2817786 tid 3 2024-04-26T00:32:29.607 INFO:tasks.rados.rados.0.smithi045.stdout:2106: copy_from oid 12 from oid 26 current snap is 237 2024-04-26T00:32:29.607 INFO:tasks.rados.rados.0.smithi045.stdout:2103: finishing write tid 3 to smithi04526579-32 2024-04-26T00:32:29.607 INFO:tasks.rados.rados.0.smithi045.stdout:2103: finishing write tid 4 to smithi04526579-32 2024-04-26T00:32:29.607 INFO:tasks.rados.rados.0.smithi045.stdout:2103: finishing write tid 5 to smithi04526579-32 2024-04-26T00:32:29.607 INFO:tasks.rados.rados.0.smithi045.stdout:2103: oid 32 updating version 0 to 1391 2024-04-26T00:32:29.607 INFO:tasks.rados.rados.0.smithi045.stdout:2103: oid 32 version 1391 is already newer than 1390 2024-04-26T00:32:29.607 INFO:tasks.rados.rados.0.smithi045.stdout:2103: oid 32 version 1391 is already newer than 1389 2024-04-26T00:32:29.607 INFO:tasks.rados.rados.0.smithi045.stdout:2103: oid 32 version 1391 is already newer than 1388 2024-04-26T00:32:29.607 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 1391 (ObjNum 765 snap 237 seq_num 765) dirty exists 2024-04-26T00:32:29.607 INFO:tasks.rados.rados.0.smithi045.stdout:2103: left oid 32 (ObjNum 765 snap 237 seq_num 765) 2024-04-26T00:32:29.608 INFO:tasks.rados.rados.0.smithi045.stdout:2103: done (3 left) 2024-04-26T00:32:29.608 INFO:tasks.rados.rados.0.smithi045.stdout:2107: write (excl) oid 9 current snap is 237 2024-04-26T00:32:29.608 INFO:tasks.rados.rados.0.smithi045.stdout:2107: seq_num 767 ranges {490969=551283,1816630=625589} 2024-04-26T00:32:29.615 INFO:tasks.rados.rados.0.smithi045.stdout:2107: writing smithi04526579-9 from 490969 to 1042252 tid 1 2024-04-26T00:32:29.627 INFO:tasks.rados.rados.0.smithi045.stdout:2107: writing smithi04526579-9 from 1816630 to 2442219 tid 2 2024-04-26T00:32:29.627 INFO:tasks.rados.rados.0.smithi045.stdout:2108: rollback oid 2 current snap is 237 2024-04-26T00:32:29.627 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 2 to 235 2024-04-26T00:32:29.627 INFO:tasks.rados.rados.0.smithi045.stdout:2104: finishing copy_from racing read to smithi04526579-4 2024-04-26T00:32:29.627 INFO:tasks.rados.rados.0.smithi045.stdout:2105: finishing write tid 1 to smithi04526579-7 2024-04-26T00:32:29.627 INFO:tasks.rados.rados.0.smithi045.stdout:2106: finishing copy_from racing read to smithi04526579-12 2024-04-26T00:32:29.627 INFO:tasks.rados.rados.0.smithi045.stdout:2105: finishing write tid 2 to smithi04526579-7 2024-04-26T00:32:29.627 INFO:tasks.rados.rados.0.smithi045.stdout:2104: finishing copy_from to smithi04526579-4 2024-04-26T00:32:29.627 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 1323 (ObjNum 661 snap 205 seq_num 661) dirty exists 2024-04-26T00:32:29.628 INFO:tasks.rados.rados.0.smithi045.stdout:2105: finishing write tid 3 to smithi04526579-7 2024-04-26T00:32:29.628 INFO:tasks.rados.rados.0.smithi045.stdout:2107: finishing write tid 1 to smithi04526579-9 2024-04-26T00:32:29.628 INFO:tasks.rados.rados.0.smithi045.stdout:2105: finishing write tid 4 to smithi04526579-7 2024-04-26T00:32:29.628 INFO:tasks.rados.rados.0.smithi045.stdout:2105: finishing write tid 5 to smithi04526579-7 2024-04-26T00:32:29.628 INFO:tasks.rados.rados.0.smithi045.stdout:2105: oid 7 updating version 0 to 1779 2024-04-26T00:32:29.628 INFO:tasks.rados.rados.0.smithi045.stdout:2105: oid 7 version 1779 is already newer than 1777 2024-04-26T00:32:29.628 INFO:tasks.rados.rados.0.smithi045.stdout:2105: oid 7 updating version 1779 to 1780 2024-04-26T00:32:29.628 INFO:tasks.rados.rados.0.smithi045.stdout:2105: oid 7 version 1780 is already newer than 1778 2024-04-26T00:32:29.628 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 1780 (ObjNum 766 snap 237 seq_num 766) dirty exists 2024-04-26T00:32:29.628 INFO:tasks.rados.rados.0.smithi045.stdout:2105: left oid 7 (ObjNum 766 snap 237 seq_num 766) 2024-04-26T00:32:29.628 INFO:tasks.rados.rados.0.smithi045.stdout:2104: done (4 left) 2024-04-26T00:32:29.628 INFO:tasks.rados.rados.0.smithi045.stdout:2105: done (3 left) 2024-04-26T00:32:29.628 INFO:tasks.rados.rados.0.smithi045.stdout:2109: write oid 5 current snap is 237 2024-04-26T00:32:29.628 INFO:tasks.rados.rados.0.smithi045.stdout:2109: seq_num 768 ranges {507776=742885,1991465=485600} 2024-04-26T00:32:29.636 INFO:tasks.rados.rados.0.smithi045.stdout:2109: writing smithi04526579-5 from 507776 to 1250661 tid 1 2024-04-26T00:32:29.646 INFO:tasks.rados.rados.0.smithi045.stdout:2109: writing smithi04526579-5 from 1991465 to 2477065 tid 2 2024-04-26T00:32:29.646 INFO:tasks.rados.rados.0.smithi045.stdout:2110: read oid 10 snap -1 2024-04-26T00:32:29.646 INFO:tasks.rados.rados.0.smithi045.stdout:2110: expect (ObjNum 573 snap 176 seq_num 573) 2024-04-26T00:32:29.646 INFO:tasks.rados.rados.0.smithi045.stdout:2111: snap_create 2024-04-26T00:32:29.646 INFO:tasks.rados.rados.0.smithi045.stdout:2106: finishing copy_from to smithi04526579-12 2024-04-26T00:32:29.646 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 1304 (ObjNum 731 snap 224 seq_num 731) dirty exists 2024-04-26T00:32:29.646 INFO:tasks.rados.rados.0.smithi045.stdout:2107: finishing write tid 2 to smithi04526579-9 2024-04-26T00:32:29.646 INFO:tasks.rados.rados.0.smithi045.stdout:2107: finishing write tid 3 to smithi04526579-9 2024-04-26T00:32:29.647 INFO:tasks.rados.rados.0.smithi045.stdout:2107: finishing write tid 4 to smithi04526579-9 2024-04-26T00:32:29.647 INFO:tasks.rados.rados.0.smithi045.stdout:2107: oid 9 updating version 0 to 1524 2024-04-26T00:32:29.647 INFO:tasks.rados.rados.0.smithi045.stdout:2107: oid 9 version 1524 is already newer than 1523 2024-04-26T00:32:29.647 INFO:tasks.rados.rados.0.smithi045.stdout:2107: oid 9 updating version 1524 to 1525 2024-04-26T00:32:29.647 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 1525 (ObjNum 767 snap 237 seq_num 767) dirty exists 2024-04-26T00:32:29.647 INFO:tasks.rados.rados.0.smithi045.stdout:2107: left oid 9 (ObjNum 767 snap 237 seq_num 767) 2024-04-26T00:32:29.647 INFO:tasks.rados.rados.0.smithi045.stdout:2108: finishing rollback tid 1 to smithi04526579-2 2024-04-26T00:32:29.647 INFO:tasks.rados.rados.0.smithi045.stdout:2108: finishing rollback tid 2 to smithi04526579-2 2024-04-26T00:32:29.647 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 1527 (ObjNum 745 snap 230 seq_num 745) dirty exists 2024-04-26T00:32:29.653 INFO:tasks.rados.rados.0.smithi045.stdout:2109: finishing write tid 1 to smithi04526579-5 2024-04-26T00:32:29.653 INFO:tasks.rados.rados.0.smithi045.stdout:2109: finishing write tid 2 to smithi04526579-5 2024-04-26T00:32:29.655 INFO:tasks.rados.rados.0.smithi045.stdout:2110: expect (ObjNum 573 snap 176 seq_num 573) 2024-04-26T00:32:29.807 INFO:tasks.rados.rados.0.smithi045.stdout:2109: finishing write tid 3 to smithi04526579-5 2024-04-26T00:32:29.808 INFO:tasks.rados.rados.0.smithi045.stdout:2109: finishing write tid 4 to smithi04526579-5 2024-04-26T00:32:29.808 INFO:tasks.rados.rados.0.smithi045.stdout:2109: oid 5 updating version 0 to 1784 2024-04-26T00:32:29.808 INFO:tasks.rados.rados.0.smithi045.stdout:2109: oid 5 version 1784 is already newer than 1782 2024-04-26T00:32:29.808 INFO:tasks.rados.rados.0.smithi045.stdout:2109: oid 5 version 1784 is already newer than 1783 2024-04-26T00:32:29.808 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 1784 (ObjNum 768 snap 237 seq_num 768) dirty exists 2024-04-26T00:32:29.808 INFO:tasks.rados.rados.0.smithi045.stdout:2109: left oid 5 (ObjNum 768 snap 237 seq_num 768) 2024-04-26T00:32:30.521 INFO:tasks.rados.rados.0.smithi045.stdout:2106: done (5 left) 2024-04-26T00:32:30.521 INFO:tasks.rados.rados.0.smithi045.stdout:2107: done (4 left) 2024-04-26T00:32:30.521 INFO:tasks.rados.rados.0.smithi045.stdout:2108: done (3 left) 2024-04-26T00:32:30.521 INFO:tasks.rados.rados.0.smithi045.stdout:2109: done (2 left) 2024-04-26T00:32:30.521 INFO:tasks.rados.rados.0.smithi045.stdout:2110: done (1 left) 2024-04-26T00:32:30.521 INFO:tasks.rados.rados.0.smithi045.stdout:2111: done (0 left) 2024-04-26T00:32:30.521 INFO:tasks.rados.rados.0.smithi045.stdout:2112: write oid 38 current snap is 238 2024-04-26T00:32:30.521 INFO:tasks.rados.rados.0.smithi045.stdout:2112: seq_num 769 ranges {524583=418134,1649947=629258,2834752=89831} 2024-04-26T00:32:30.527 INFO:tasks.rados.rados.0.smithi045.stdout:2112: writing smithi04526579-38 from 524583 to 942717 tid 1 2024-04-26T00:32:30.537 INFO:tasks.rados.rados.0.smithi045.stdout:2112: writing smithi04526579-38 from 1649947 to 2279205 tid 2 2024-04-26T00:32:30.542 INFO:tasks.rados.rados.0.smithi045.stdout:2112: writing smithi04526579-38 from 2834752 to 2924583 tid 3 2024-04-26T00:32:30.542 INFO:tasks.rados.rados.0.smithi045.stdout:2113: write (excl) oid 22 current snap is 238 2024-04-26T00:32:30.543 INFO:tasks.rados.rados.0.smithi045.stdout:2113: seq_num 770 ranges {541390=493383,1592076=489269,2862175=79215} 2024-04-26T00:32:30.550 INFO:tasks.rados.rados.0.smithi045.stdout:2113: writing smithi04526579-22 from 541390 to 1034773 tid 1 2024-04-26T00:32:30.557 INFO:tasks.rados.rados.0.smithi045.stdout:2113: writing smithi04526579-22 from 1592076 to 2081345 tid 2 2024-04-26T00:32:30.564 INFO:tasks.rados.rados.0.smithi045.stdout:2113: writing smithi04526579-22 from 2862175 to 2941390 tid 3 2024-04-26T00:32:30.564 INFO:tasks.rados.rados.0.smithi045.stdout:2114: copy_from oid 49 from oid 15 current snap is 238 2024-04-26T00:32:30.565 INFO:tasks.rados.rados.0.smithi045.stdout:2112: finishing write tid 1 to smithi04526579-38 2024-04-26T00:32:30.565 INFO:tasks.rados.rados.0.smithi045.stdout:2112: finishing write tid 2 to smithi04526579-38 2024-04-26T00:32:30.565 INFO:tasks.rados.rados.0.smithi045.stdout:2112: finishing write tid 3 to smithi04526579-38 2024-04-26T00:32:30.565 INFO:tasks.rados.rados.0.smithi045.stdout:2113: finishing write tid 1 to smithi04526579-22 2024-04-26T00:32:30.565 INFO:tasks.rados.rados.0.smithi045.stdout:2112: finishing write tid 4 to smithi04526579-38 2024-04-26T00:32:30.565 INFO:tasks.rados.rados.0.smithi045.stdout:2112: finishing write tid 5 to smithi04526579-38 2024-04-26T00:32:30.565 INFO:tasks.rados.rados.0.smithi045.stdout:2112: oid 38 updating version 0 to 1290 2024-04-26T00:32:30.565 INFO:tasks.rados.rados.0.smithi045.stdout:2112: oid 38 updating version 1290 to 1292 2024-04-26T00:32:30.565 INFO:tasks.rados.rados.0.smithi045.stdout:2112: oid 38 version 1292 is already newer than 1291 2024-04-26T00:32:30.565 INFO:tasks.rados.rados.0.smithi045.stdout:2112: oid 38 version 1292 is already newer than 1289 2024-04-26T00:32:30.565 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 1292 (ObjNum 769 snap 238 seq_num 769) dirty exists 2024-04-26T00:32:30.565 INFO:tasks.rados.rados.0.smithi045.stdout:2112: left oid 38 (ObjNum 769 snap 238 seq_num 769) 2024-04-26T00:32:30.565 INFO:tasks.rados.rados.0.smithi045.stdout:2113: finishing write tid 2 to smithi04526579-22 2024-04-26T00:32:30.565 INFO:tasks.rados.rados.0.smithi045.stdout:2112: done (2 left) 2024-04-26T00:32:30.566 INFO:tasks.rados.rados.0.smithi045.stdout:2115: write oid 6 current snap is 238 2024-04-26T00:32:30.566 INFO:tasks.rados.rados.0.smithi045.stdout:2115: seq_num 771 ranges {558197=568632,1650558=632927,2773245=184952} 2024-04-26T00:32:30.573 INFO:tasks.rados.rados.0.smithi045.stdout:2115: writing smithi04526579-6 from 558197 to 1126829 tid 1 2024-04-26T00:32:30.581 INFO:tasks.rados.rados.0.smithi045.stdout:2115: writing smithi04526579-6 from 1650558 to 2283485 tid 2 2024-04-26T00:32:30.587 INFO:tasks.rados.rados.0.smithi045.stdout:2115: writing smithi04526579-6 from 2773245 to 2958197 tid 3 2024-04-26T00:32:30.587 INFO:tasks.rados.rados.0.smithi045.stdout:2116: rollback oid 7 current snap is 238 2024-04-26T00:32:30.587 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 7 to 219 2024-04-26T00:32:30.587 INFO:tasks.rados.rados.0.smithi045.stdout:2117: write oid 34 current snap is 238 2024-04-26T00:32:30.587 INFO:tasks.rados.rados.0.smithi045.stdout:2117: seq_num 772 ranges {575004=643881,1709040=492938,2917021=57983} 2024-04-26T00:32:30.595 INFO:tasks.rados.rados.0.smithi045.stdout:2117: writing smithi04526579-34 from 575004 to 1218885 tid 1 2024-04-26T00:32:30.602 INFO:tasks.rados.rados.0.smithi045.stdout:2117: writing smithi04526579-34 from 1709040 to 2201978 tid 2 2024-04-26T00:32:30.608 INFO:tasks.rados.rados.0.smithi045.stdout:2117: writing smithi04526579-34 from 2917021 to 2975004 tid 3 2024-04-26T00:32:30.608 INFO:tasks.rados.rados.0.smithi045.stdout:2118: copy_from oid 10 from oid 16 current snap is 238 2024-04-26T00:32:30.608 INFO:tasks.rados.rados.0.smithi045.stdout:2113: finishing write tid 3 to smithi04526579-22 2024-04-26T00:32:30.608 INFO:tasks.rados.rados.0.smithi045.stdout:2113: finishing write tid 4 to smithi04526579-22 2024-04-26T00:32:30.608 INFO:tasks.rados.rados.0.smithi045.stdout:2119: copy_from oid 41 from oid 20 current snap is 238 2024-04-26T00:32:30.608 INFO:tasks.rados.rados.0.smithi045.stdout:2113: finishing write tid 5 to smithi04526579-22 2024-04-26T00:32:30.608 INFO:tasks.rados.rados.0.smithi045.stdout:2113: oid 22 updating version 0 to 1450 2024-04-26T00:32:30.608 INFO:tasks.rados.rados.0.smithi045.stdout:2113: oid 22 version 1450 is already newer than 1449 2024-04-26T00:32:30.608 INFO:tasks.rados.rados.0.smithi045.stdout:2113: oid 22 version 1450 is already newer than 1447 2024-04-26T00:32:30.608 INFO:tasks.rados.rados.0.smithi045.stdout:2113: oid 22 version 1450 is already newer than 1448 2024-04-26T00:32:30.608 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 1450 (ObjNum 770 snap 238 seq_num 770) dirty exists 2024-04-26T00:32:30.608 INFO:tasks.rados.rados.0.smithi045.stdout:2113: left oid 22 (ObjNum 770 snap 238 seq_num 770) 2024-04-26T00:32:30.609 INFO:tasks.rados.rados.0.smithi045.stdout:2113: done (6 left) 2024-04-26T00:32:30.609 INFO:tasks.rados.rados.0.smithi045.stdout:2120: snap_create 2024-04-26T00:32:30.609 INFO:tasks.rados.rados.0.smithi045.stdout:2115: finishing write tid 1 to smithi04526579-6 2024-04-26T00:32:30.609 INFO:tasks.rados.rados.0.smithi045.stdout:2114: finishing copy_from racing read to smithi04526579-49 2024-04-26T00:32:30.609 INFO:tasks.rados.rados.0.smithi045.stdout:2114: finishing copy_from to smithi04526579-49 2024-04-26T00:32:30.609 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 1333 (ObjNum 724 snap 222 seq_num 724) dirty exists 2024-04-26T00:32:30.609 INFO:tasks.rados.rados.0.smithi045.stdout:2115: finishing write tid 2 to smithi04526579-6 2024-04-26T00:32:30.609 INFO:tasks.rados.rados.0.smithi045.stdout:2115: finishing write tid 3 to smithi04526579-6 2024-04-26T00:32:30.609 INFO:tasks.rados.rados.0.smithi045.stdout:2115: finishing write tid 4 to smithi04526579-6 2024-04-26T00:32:30.609 INFO:tasks.rados.rados.0.smithi045.stdout:2115: finishing write tid 5 to smithi04526579-6 2024-04-26T00:32:30.609 INFO:tasks.rados.rados.0.smithi045.stdout:2115: oid 6 updating version 0 to 1060 2024-04-26T00:32:30.609 INFO:tasks.rados.rados.0.smithi045.stdout:2115: oid 6 version 1060 is already newer than 1058 2024-04-26T00:32:30.609 INFO:tasks.rados.rados.0.smithi045.stdout:2115: oid 6 version 1060 is already newer than 1059 2024-04-26T00:32:30.609 INFO:tasks.rados.rados.0.smithi045.stdout:2115: oid 6 version 1060 is already newer than 1057 2024-04-26T00:32:30.610 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 1060 (ObjNum 771 snap 238 seq_num 771) dirty exists 2024-04-26T00:32:30.610 INFO:tasks.rados.rados.0.smithi045.stdout:2115: left oid 6 (ObjNum 771 snap 238 seq_num 771) 2024-04-26T00:32:30.610 INFO:tasks.rados.rados.0.smithi045.stdout:2116: finishing rollback tid 0 to smithi04526579-7 2024-04-26T00:32:30.610 INFO:tasks.rados.rados.0.smithi045.stdout:2116: finishing rollback tid 1 to smithi04526579-7 2024-04-26T00:32:30.610 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 1787 (ObjNum 33 snap 0 seq_num 3719112503) dirty dne 2024-04-26T00:32:30.610 INFO:tasks.rados.rados.0.smithi045.stdout:2117: finishing write tid 1 to smithi04526579-34 2024-04-26T00:32:30.613 INFO:tasks.rados.rados.0.smithi045.stdout:2117: finishing write tid 2 to smithi04526579-34 2024-04-26T00:32:30.613 INFO:tasks.rados.rados.0.smithi045.stdout:2117: finishing write tid 3 to smithi04526579-34 2024-04-26T00:32:30.614 INFO:tasks.rados.rados.0.smithi045.stdout:2117: finishing write tid 4 to smithi04526579-34 2024-04-26T00:32:30.615 INFO:tasks.rados.rados.0.smithi045.stdout:2117: finishing write tid 5 to smithi04526579-34 2024-04-26T00:32:30.615 INFO:tasks.rados.rados.0.smithi045.stdout:2117: oid 34 updating version 0 to 1337 2024-04-26T00:32:30.615 INFO:tasks.rados.rados.0.smithi045.stdout:2117: oid 34 version 1337 is already newer than 1334 2024-04-26T00:32:30.615 INFO:tasks.rados.rados.0.smithi045.stdout:2117: oid 34 version 1337 is already newer than 1336 2024-04-26T00:32:30.615 INFO:tasks.rados.rados.0.smithi045.stdout:2117: oid 34 version 1337 is already newer than 1335 2024-04-26T00:32:30.615 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 1337 (ObjNum 772 snap 238 seq_num 772) dirty exists 2024-04-26T00:32:30.615 INFO:tasks.rados.rados.0.smithi045.stdout:2117: left oid 34 (ObjNum 772 snap 238 seq_num 772) 2024-04-26T00:32:30.625 INFO:tasks.rados.rados.0.smithi045.stdout:2119: finishing copy_from racing read to smithi04526579-41 2024-04-26T00:32:30.626 INFO:tasks.rados.rados.0.smithi045.stdout:2119: finishing copy_from to smithi04526579-41 2024-04-26T00:32:30.626 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 1064 (ObjNum 542 snap 167 seq_num 542) dirty exists 2024-04-26T00:32:30.630 INFO:tasks.rados.rados.0.smithi045.stdout:2118: finishing copy_from racing read to smithi04526579-10 2024-04-26T00:32:30.638 INFO:tasks.rados.rados.0.smithi045.stdout:2118: finishing copy_from to smithi04526579-10 2024-04-26T00:32:30.638 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 1134 (ObjNum 515 snap 157 seq_num 515) dirty exists 2024-04-26T00:32:31.533 INFO:tasks.rados.rados.0.smithi045.stdout:2114: done (6 left) 2024-04-26T00:32:31.533 INFO:tasks.rados.rados.0.smithi045.stdout:2115: done (5 left) 2024-04-26T00:32:31.533 INFO:tasks.rados.rados.0.smithi045.stdout:2116: done (4 left) 2024-04-26T00:32:31.533 INFO:tasks.rados.rados.0.smithi045.stdout:2117: done (3 left) 2024-04-26T00:32:31.533 INFO:tasks.rados.rados.0.smithi045.stdout:2118: done (2 left) 2024-04-26T00:32:31.533 INFO:tasks.rados.rados.0.smithi045.stdout:2119: done (1 left) 2024-04-26T00:32:31.533 INFO:tasks.rados.rados.0.smithi045.stdout:2120: done (0 left) 2024-04-26T00:32:31.533 INFO:tasks.rados.rados.0.smithi045.stdout:2121: delete oid 18 current snap is 239 2024-04-26T00:32:31.542 INFO:tasks.rados.rados.0.smithi045.stdout:2121: done (0 left) 2024-04-26T00:32:31.542 INFO:tasks.rados.rados.0.smithi045.stdout:2122: delete oid 6 current snap is 239 2024-04-26T00:32:31.549 INFO:tasks.rados.rados.0.smithi045.stdout:2122: done (0 left) 2024-04-26T00:32:31.549 INFO:tasks.rados.rados.0.smithi045.stdout:2123: snap_remove snap 192 2024-04-26T00:32:32.538 INFO:tasks.rados.rados.0.smithi045.stdout:2123: done (0 left) 2024-04-26T00:32:32.538 INFO:tasks.rados.rados.0.smithi045.stdout:2124: copy_from oid 9 from oid 4 current snap is 239 2024-04-26T00:32:32.538 INFO:tasks.rados.rados.0.smithi045.stdout:2125: snap_create 2024-04-26T00:32:32.586 INFO:tasks.rados.rados.0.smithi045.stdout:2124: finishing copy_from racing read to smithi04526579-9 2024-04-26T00:32:32.587 INFO:tasks.rados.rados.0.smithi045.stdout:2124: finishing copy_from to smithi04526579-9 2024-04-26T00:32:32.587 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 1323 (ObjNum 661 snap 205 seq_num 661) dirty exists 2024-04-26T00:32:33.476 INFO:tasks.rados.rados.0.smithi045.stdout:2124: done (1 left) 2024-04-26T00:32:33.476 INFO:tasks.rados.rados.0.smithi045.stdout:2125: done (0 left) 2024-04-26T00:32:33.477 INFO:tasks.rados.rados.0.smithi045.stdout:2126: write (excl) oid 1 current snap is 240 2024-04-26T00:32:33.477 INFO:tasks.rados.rados.0.smithi045.stdout:2126: seq_num 775 ranges {625425=469628,1768133=640265} 2024-04-26T00:32:33.494 INFO:tasks.rados.rados.0.smithi045.stdout:2126: writing smithi04526579-1 from 625425 to 1095053 tid 1 2024-04-26T00:32:33.517 INFO:tasks.rados.rados.0.smithi045.stdout:2126: writing smithi04526579-1 from 1768133 to 2408398 tid 2 2024-04-26T00:32:33.518 INFO:tasks.rados.rados.0.smithi045.stdout:2127: write oid 48 current snap is 240 2024-04-26T00:32:33.518 INFO:tasks.rados.rados.0.smithi045.stdout:2127: seq_num 776 ranges {642232=661230,1942968=783923} 2024-04-26T00:32:33.536 INFO:tasks.rados.rados.0.smithi045.stdout:2127: writing smithi04526579-48 from 642232 to 1303462 tid 1 2024-04-26T00:32:33.553 INFO:tasks.rados.rados.0.smithi045.stdout:2127: writing smithi04526579-48 from 1942968 to 2726891 tid 2 2024-04-26T00:32:33.553 INFO:tasks.rados.rados.0.smithi045.stdout:2128: copy_from oid 29 from oid 43 current snap is 240 2024-04-26T00:32:33.553 INFO:tasks.rados.rados.0.smithi045.stdout:2126: finishing write tid 1 to smithi04526579-1 2024-04-26T00:32:33.553 INFO:tasks.rados.rados.0.smithi045.stdout:2126: finishing write tid 2 to smithi04526579-1 2024-04-26T00:32:33.553 INFO:tasks.rados.rados.0.smithi045.stdout:2126: finishing write tid 3 to smithi04526579-1 2024-04-26T00:32:33.553 INFO:tasks.rados.rados.0.smithi045.stdout:2126: finishing write tid 4 to smithi04526579-1 2024-04-26T00:32:33.554 INFO:tasks.rados.rados.0.smithi045.stdout:2126: oid 1 updating version 0 to 1373 2024-04-26T00:32:33.554 INFO:tasks.rados.rados.0.smithi045.stdout:2126: oid 1 updating version 1373 to 1374 2024-04-26T00:32:33.554 INFO:tasks.rados.rados.0.smithi045.stdout:2126: oid 1 updating version 1374 to 1375 2024-04-26T00:32:33.554 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 1375 (ObjNum 775 snap 240 seq_num 775) dirty exists 2024-04-26T00:32:33.554 INFO:tasks.rados.rados.0.smithi045.stdout:2126: left oid 1 (ObjNum 775 snap 240 seq_num 775) 2024-04-26T00:32:33.554 INFO:tasks.rados.rados.0.smithi045.stdout:2127: finishing write tid 1 to smithi04526579-48 2024-04-26T00:32:33.554 INFO:tasks.rados.rados.0.smithi045.stdout:2126: done (2 left) 2024-04-26T00:32:33.554 INFO:tasks.rados.rados.0.smithi045.stdout:2129: read oid 9 snap -1 2024-04-26T00:32:33.554 INFO:tasks.rados.rados.0.smithi045.stdout:2129: expect (ObjNum 661 snap 205 seq_num 661) 2024-04-26T00:32:33.554 INFO:tasks.rados.rados.0.smithi045.stdout:2130: rollback oid 8 current snap is 240 2024-04-26T00:32:33.554 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 8 to 175 2024-04-26T00:32:33.554 INFO:tasks.rados.rados.0.smithi045.stdout:2131: write (excl) oid 12 current snap is 240 2024-04-26T00:32:33.554 INFO:tasks.rados.rados.0.smithi045.stdout:2131: seq_num 777 ranges {659039=736479,2001450=643934} 2024-04-26T00:32:33.566 INFO:tasks.rados.rados.0.smithi045.stdout:2131: writing smithi04526579-12 from 659039 to 1395518 tid 1 2024-04-26T00:32:33.579 INFO:tasks.rados.rados.0.smithi045.stdout:2131: writing smithi04526579-12 from 2001450 to 2645384 tid 2 2024-04-26T00:32:33.579 INFO:tasks.rados.rados.0.smithi045.stdout:2127: finishing write tid 2 to smithi04526579-48 2024-04-26T00:32:33.579 INFO:tasks.rados.rados.0.smithi045.stdout:2127: finishing write tid 3 to smithi04526579-48 2024-04-26T00:32:33.579 INFO:tasks.rados.rados.0.smithi045.stdout:2127: finishing write tid 4 to smithi04526579-48 2024-04-26T00:32:33.580 INFO:tasks.rados.rados.0.smithi045.stdout:2127: oid 48 updating version 0 to 1340 2024-04-26T00:32:33.580 INFO:tasks.rados.rados.0.smithi045.stdout:2127: oid 48 version 1340 is already newer than 1339 2024-04-26T00:32:33.580 INFO:tasks.rados.rados.0.smithi045.stdout:2127: oid 48 version 1340 is already newer than 1338 2024-04-26T00:32:33.580 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 1340 (ObjNum 776 snap 240 seq_num 776) dirty exists 2024-04-26T00:32:33.580 INFO:tasks.rados.rados.0.smithi045.stdout:2127: left oid 48 (ObjNum 776 snap 240 seq_num 776) 2024-04-26T00:32:33.580 INFO:tasks.rados.rados.0.smithi045.stdout:2127: done (4 left) 2024-04-26T00:32:33.580 INFO:tasks.rados.rados.0.smithi045.stdout:2132: read oid 35 snap -1 2024-04-26T00:32:33.580 INFO:tasks.rados.rados.0.smithi045.stdout:2132: expect (ObjNum 717 snap 222 seq_num 717) 2024-04-26T00:32:33.580 INFO:tasks.rados.rados.0.smithi045.stdout:2133: delete oid 16 current snap is 240 2024-04-26T00:32:33.583 INFO:tasks.rados.rados.0.smithi045.stdout:2129: expect (ObjNum 661 snap 205 seq_num 661) 2024-04-26T00:32:33.870 INFO:tasks.rados.rados.0.smithi045.stdout:2131: finishing write tid 1 to smithi04526579-12 2024-04-26T00:32:33.870 INFO:tasks.rados.rados.0.smithi045.stdout:2131: finishing write tid 2 to smithi04526579-12 2024-04-26T00:32:33.870 INFO:tasks.rados.rados.0.smithi045.stdout:2131: finishing write tid 3 to smithi04526579-12 2024-04-26T00:32:33.870 INFO:tasks.rados.rados.0.smithi045.stdout:2131: finishing write tid 4 to smithi04526579-12 2024-04-26T00:32:33.870 INFO:tasks.rados.rados.0.smithi045.stdout:2131: oid 12 updating version 0 to 1449 2024-04-26T00:32:33.870 INFO:tasks.rados.rados.0.smithi045.stdout:2131: oid 12 version 1449 is already newer than 1447 2024-04-26T00:32:33.870 INFO:tasks.rados.rados.0.smithi045.stdout:2131: oid 12 version 1449 is already newer than 1448 2024-04-26T00:32:33.870 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 1449 (ObjNum 777 snap 240 seq_num 777) dirty exists 2024-04-26T00:32:33.870 INFO:tasks.rados.rados.0.smithi045.stdout:2131: left oid 12 (ObjNum 777 snap 240 seq_num 777) 2024-04-26T00:32:33.870 INFO:tasks.rados.rados.0.smithi045.stdout:2128: finishing copy_from racing read to smithi04526579-29 2024-04-26T00:32:33.870 INFO:tasks.rados.rados.0.smithi045.stdout:2130: finishing rollback tid 0 to smithi04526579-8 2024-04-26T00:32:33.870 INFO:tasks.rados.rados.0.smithi045.stdout:2130: finishing rollback tid 1 to smithi04526579-8 2024-04-26T00:32:33.871 INFO:tasks.rados.rados.0.smithi045.stdout:2130: finishing rollback tid 2 to smithi04526579-8 2024-04-26T00:32:33.871 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 1074 (ObjNum 551 snap 168 seq_num 551) dirty exists 2024-04-26T00:32:33.871 INFO:tasks.rados.rados.0.smithi045.stdout:2128: finishing copy_from to smithi04526579-29 2024-04-26T00:32:33.871 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 1519 (ObjNum 764 snap 237 seq_num 764) dirty exists 2024-04-26T00:32:33.871 INFO:tasks.rados.rados.0.smithi045.stdout:2132: expect (ObjNum 717 snap 222 seq_num 717) 2024-04-26T00:32:33.935 INFO:tasks.rados.rados.0.smithi045.stdout:2128: done (5 left) 2024-04-26T00:32:33.935 INFO:tasks.rados.rados.0.smithi045.stdout:2129: done (4 left) 2024-04-26T00:32:33.935 INFO:tasks.rados.rados.0.smithi045.stdout:2130: done (3 left) 2024-04-26T00:32:33.935 INFO:tasks.rados.rados.0.smithi045.stdout:2131: done (2 left) 2024-04-26T00:32:33.935 INFO:tasks.rados.rados.0.smithi045.stdout:2132: done (1 left) 2024-04-26T00:32:33.935 INFO:tasks.rados.rados.0.smithi045.stdout:2133: done (0 left) 2024-04-26T00:32:33.935 INFO:tasks.rados.rados.0.smithi045.stdout:2134: snap_remove snap 175 2024-04-26T00:32:34.558 INFO:tasks.rados.rados.0.smithi045.stdout:2134: done (0 left) 2024-04-26T00:32:34.558 INFO:tasks.rados.rados.0.smithi045.stdout:2135: write (excl) oid 15 current snap is 240 2024-04-26T00:32:34.558 INFO:tasks.rados.rados.0.smithi045.stdout:2135: seq_num 779 ranges {692653=486977,1602061=647603,2992629=100024} 2024-04-26T00:32:34.581 INFO:tasks.rados.rados.0.smithi045.stdout:2135: writing smithi04526579-15 from 692653 to 1179630 tid 1 2024-04-26T00:32:34.602 INFO:tasks.rados.rados.0.smithi045.stdout:2135: writing smithi04526579-15 from 1602061 to 2249664 tid 2 2024-04-26T00:32:34.614 INFO:tasks.rados.rados.0.smithi045.stdout:2135: writing smithi04526579-15 from 2992629 to 3092653 tid 3 2024-04-26T00:32:34.615 INFO:tasks.rados.rados.0.smithi045.stdout:2136: delete oid 42 current snap is 240 2024-04-26T00:32:34.615 INFO:tasks.rados.rados.0.smithi045.stdout:2135: finishing write tid 1 to smithi04526579-15 2024-04-26T00:32:34.615 INFO:tasks.rados.rados.0.smithi045.stdout:2135: finishing write tid 2 to smithi04526579-15 2024-04-26T00:32:34.619 INFO:tasks.rados.rados.0.smithi045.stdout:2135: finishing write tid 3 to smithi04526579-15 2024-04-26T00:32:34.619 INFO:tasks.rados.rados.0.smithi045.stdout:2135: finishing write tid 4 to smithi04526579-15 2024-04-26T00:32:34.620 INFO:tasks.rados.rados.0.smithi045.stdout:2135: finishing write tid 5 to smithi04526579-15 2024-04-26T00:32:34.620 INFO:tasks.rados.rados.0.smithi045.stdout:2135: oid 15 updating version 0 to 1389 2024-04-26T00:32:34.620 INFO:tasks.rados.rados.0.smithi045.stdout:2135: oid 15 updating version 1389 to 1390 2024-04-26T00:32:34.620 INFO:tasks.rados.rados.0.smithi045.stdout:2135: oid 15 version 1390 is already newer than 1388 2024-04-26T00:32:34.620 INFO:tasks.rados.rados.0.smithi045.stdout:2135: oid 15 version 1390 is already newer than 1381 2024-04-26T00:32:34.620 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 1390 (ObjNum 779 snap 240 seq_num 779) dirty exists 2024-04-26T00:32:34.620 INFO:tasks.rados.rados.0.smithi045.stdout:2135: left oid 15 (ObjNum 779 snap 240 seq_num 779) 2024-04-26T00:32:34.622 INFO:tasks.rados.rados.0.smithi045.stdout:2135: done (1 left) 2024-04-26T00:32:34.622 INFO:tasks.rados.rados.0.smithi045.stdout:2136: done (0 left) 2024-04-26T00:32:34.622 INFO:tasks.rados.rados.0.smithi045.stdout:2137: write oid 4 current snap is 240 2024-04-26T00:32:34.622 INFO:tasks.rados.rados.0.smithi045.stdout:2137: seq_num 781 ranges {726267=637475,2119025=651272} 2024-04-26T00:32:34.639 INFO:tasks.rados.rados.0.smithi045.stdout:2137: writing smithi04526579-4 from 726267 to 1363742 tid 1 2024-04-26T00:32:34.655 INFO:tasks.rados.rados.0.smithi045.stdout:2137: writing smithi04526579-4 from 2119025 to 2770297 tid 2 2024-04-26T00:32:34.655 INFO:tasks.rados.rados.0.smithi045.stdout:2138: copy_from oid 40 from oid 3 current snap is 240 2024-04-26T00:32:34.655 INFO:tasks.rados.rados.0.smithi045.stdout:2137: finishing write tid 1 to smithi04526579-4 2024-04-26T00:32:34.656 INFO:tasks.rados.rados.0.smithi045.stdout:2139: read oid 23 snap 227 2024-04-26T00:32:34.656 INFO:tasks.rados.rados.0.smithi045.stdout:2139: expect (ObjNum 641 snap 201 seq_num 641) 2024-04-26T00:32:34.656 INFO:tasks.rados.rados.0.smithi045.stdout:2140: write oid 47 current snap is 240 2024-04-26T00:32:34.656 INFO:tasks.rados.rados.0.smithi045.stdout:2140: seq_num 782 ranges {743074=712724,2061154=794930} 2024-04-26T00:32:34.668 INFO:tasks.rados.rados.0.smithi045.stdout:2140: writing smithi04526579-47 from 743074 to 1455798 tid 1 2024-04-26T00:32:34.680 INFO:tasks.rados.rados.0.smithi045.stdout:2140: writing smithi04526579-47 from 2061154 to 2856084 tid 2 2024-04-26T00:32:34.680 INFO:tasks.rados.rados.0.smithi045.stdout:2141: rollback oid 32 current snap is 240 2024-04-26T00:32:34.681 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 32 to 233 2024-04-26T00:32:34.681 INFO:tasks.rados.rados.0.smithi045.stdout:2137: finishing write tid 2 to smithi04526579-4 2024-04-26T00:32:34.681 INFO:tasks.rados.rados.0.smithi045.stdout:2138: finishing copy_from to smithi04526579-40 2024-04-26T00:32:34.681 INFO:tasks.rados.rados.0.smithi045.stdout:2138: got expected ENOENT (src dne) 2024-04-26T00:32:34.681 INFO:tasks.rados.rados.0.smithi045.stdout:2138: finishing copy_from racing read to smithi04526579-40 2024-04-26T00:32:34.681 INFO:tasks.rados.rados.0.smithi045.stdout:2137: finishing write tid 3 to smithi04526579-4 2024-04-26T00:32:34.681 INFO:tasks.rados.rados.0.smithi045.stdout:2137: finishing write tid 4 to smithi04526579-4 2024-04-26T00:32:34.681 INFO:tasks.rados.rados.0.smithi045.stdout:2137: oid 4 updating version 0 to 1530 2024-04-26T00:32:34.681 INFO:tasks.rados.rados.0.smithi045.stdout:2137: oid 4 version 1530 is already newer than 1528 2024-04-26T00:32:34.681 INFO:tasks.rados.rados.0.smithi045.stdout:2137: oid 4 version 1530 is already newer than 1529 2024-04-26T00:32:34.681 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 1530 (ObjNum 781 snap 240 seq_num 781) dirty exists 2024-04-26T00:32:34.681 INFO:tasks.rados.rados.0.smithi045.stdout:2137: left oid 4 (ObjNum 781 snap 240 seq_num 781) 2024-04-26T00:32:34.681 INFO:tasks.rados.rados.0.smithi045.stdout:2137: done (4 left) 2024-04-26T00:32:34.681 INFO:tasks.rados.rados.0.smithi045.stdout:2138: done (3 left) 2024-04-26T00:32:34.682 INFO:tasks.rados.rados.0.smithi045.stdout:2142: read oid 40 snap -1 2024-04-26T00:32:34.682 INFO:tasks.rados.rados.0.smithi045.stdout:2142: expect deleted 2024-04-26T00:32:34.682 INFO:tasks.rados.rados.0.smithi045.stdout:2143: read oid 9 snap -1 2024-04-26T00:32:34.682 INFO:tasks.rados.rados.0.smithi045.stdout:2143: expect (ObjNum 661 snap 205 seq_num 661) 2024-04-26T00:32:34.682 INFO:tasks.rados.rados.0.smithi045.stdout:2144: write (excl) oid 29 current snap is 240 2024-04-26T00:32:34.682 INFO:tasks.rados.rados.0.smithi045.stdout:2144: seq_num 783 ranges {759881=787973,2119636=654941} 2024-04-26T00:32:34.694 INFO:tasks.rados.rados.0.smithi045.stdout:2144: writing smithi04526579-29 from 759881 to 1547854 tid 1 2024-04-26T00:32:34.704 INFO:tasks.rados.rados.0.smithi045.stdout:2144: writing smithi04526579-29 from 2119636 to 2774577 tid 2 2024-04-26T00:32:34.704 INFO:tasks.rados.rados.0.smithi045.stdout:2145: snap_create 2024-04-26T00:32:34.704 INFO:tasks.rados.rados.0.smithi045.stdout:2139: expect (ObjNum 641 snap 201 seq_num 641) 2024-04-26T00:32:34.894 INFO:tasks.rados.rados.0.smithi045.stdout:2140: finishing write tid 1 to smithi04526579-47 2024-04-26T00:32:34.894 INFO:tasks.rados.rados.0.smithi045.stdout:2140: finishing write tid 2 to smithi04526579-47 2024-04-26T00:32:34.894 INFO:tasks.rados.rados.0.smithi045.stdout:2140: finishing write tid 3 to smithi04526579-47 2024-04-26T00:32:34.894 INFO:tasks.rados.rados.0.smithi045.stdout:2140: finishing write tid 4 to smithi04526579-47 2024-04-26T00:32:34.894 INFO:tasks.rados.rados.0.smithi045.stdout:2140: oid 47 updating version 0 to 1343 2024-04-26T00:32:34.894 INFO:tasks.rados.rados.0.smithi045.stdout:2140: oid 47 version 1343 is already newer than 1341 2024-04-26T00:32:34.894 INFO:tasks.rados.rados.0.smithi045.stdout:2140: oid 47 version 1343 is already newer than 1342 2024-04-26T00:32:34.894 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 1343 (ObjNum 782 snap 240 seq_num 782) dirty exists 2024-04-26T00:32:34.895 INFO:tasks.rados.rados.0.smithi045.stdout:2140: left oid 47 (ObjNum 782 snap 240 seq_num 782) 2024-04-26T00:32:34.895 INFO:tasks.rados.rados.0.smithi045.stdout:2143: expect (ObjNum 661 snap 205 seq_num 661) 2024-04-26T00:32:35.123 INFO:tasks.rados.rados.0.smithi045.stdout:2141: finishing rollback tid 0 to smithi04526579-32 2024-04-26T00:32:35.123 INFO:tasks.rados.rados.0.smithi045.stdout:2141: finishing rollback tid 1 to smithi04526579-32 2024-04-26T00:32:35.124 INFO:tasks.rados.rados.0.smithi045.stdout:2141: finishing rollback tid 2 to smithi04526579-32 2024-04-26T00:32:35.124 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 1415 (ObjNum 573 snap 176 seq_num 573) dirty exists 2024-04-26T00:32:35.124 INFO:tasks.rados.rados.0.smithi045.stdout:2144: finishing write tid 1 to smithi04526579-29 2024-04-26T00:32:35.124 INFO:tasks.rados.rados.0.smithi045.stdout:2144: finishing write tid 2 to smithi04526579-29 2024-04-26T00:32:35.124 INFO:tasks.rados.rados.0.smithi045.stdout:2144: finishing write tid 3 to smithi04526579-29 2024-04-26T00:32:35.124 INFO:tasks.rados.rados.0.smithi045.stdout:2144: finishing write tid 4 to smithi04526579-29 2024-04-26T00:32:35.124 INFO:tasks.rados.rados.0.smithi045.stdout:2144: oid 29 updating version 0 to 1522 2024-04-26T00:32:35.124 INFO:tasks.rados.rados.0.smithi045.stdout:2144: oid 29 version 1522 is already newer than 1520 2024-04-26T00:32:35.124 INFO:tasks.rados.rados.0.smithi045.stdout:2144: oid 29 version 1522 is already newer than 1521 2024-04-26T00:32:35.124 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 1522 (ObjNum 783 snap 240 seq_num 783) dirty exists 2024-04-26T00:32:35.124 INFO:tasks.rados.rados.0.smithi045.stdout:2144: left oid 29 (ObjNum 783 snap 240 seq_num 783) 2024-04-26T00:32:35.565 INFO:tasks.rados.rados.0.smithi045.stdout:2139: done (6 left) 2024-04-26T00:32:35.565 INFO:tasks.rados.rados.0.smithi045.stdout:2140: done (5 left) 2024-04-26T00:32:35.565 INFO:tasks.rados.rados.0.smithi045.stdout:2141: done (4 left) 2024-04-26T00:32:35.565 INFO:tasks.rados.rados.0.smithi045.stdout:2142: done (3 left) 2024-04-26T00:32:35.565 INFO:tasks.rados.rados.0.smithi045.stdout:2143: done (2 left) 2024-04-26T00:32:35.565 INFO:tasks.rados.rados.0.smithi045.stdout:2144: done (1 left) 2024-04-26T00:32:35.565 INFO:tasks.rados.rados.0.smithi045.stdout:2145: done (0 left) 2024-04-26T00:32:35.566 INFO:tasks.rados.rados.0.smithi045.stdout:2146: write (excl) oid 40 current snap is 241 2024-04-26T00:32:35.566 INFO:tasks.rados.rados.0.smithi045.stdout:2146: seq_num 784 ranges {776688=579575,1894471=798599} 2024-04-26T00:32:35.577 INFO:tasks.rados.rados.0.smithi045.stdout:2146: writing smithi04526579-40 from 776688 to 1356263 tid 1 2024-04-26T00:32:35.588 INFO:tasks.rados.rados.0.smithi045.stdout:2146: writing smithi04526579-40 from 1894471 to 2693070 tid 2 2024-04-26T00:32:35.588 INFO:tasks.rados.rados.0.smithi045.stdout:2147: read oid 19 snap 239 2024-04-26T00:32:35.588 INFO:tasks.rados.rados.0.smithi045.stdout:2147: expect (ObjNum 443 snap 138 seq_num 443) 2024-04-26T00:32:35.588 INFO:tasks.rados.rados.0.smithi045.stdout:2148: snap_remove snap 228 2024-04-26T00:32:36.567 INFO:tasks.rados.rados.0.smithi045.stdout:2148: done (2 left) 2024-04-26T00:32:36.567 INFO:tasks.rados.rados.0.smithi045.stdout:2149: rollback oid 15 current snap is 241 2024-04-26T00:32:36.567 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 15 to 225 2024-04-26T00:32:36.568 INFO:tasks.rados.rados.0.smithi045.stdout:2146: finishing write tid 1 to smithi04526579-40 2024-04-26T00:32:36.568 INFO:tasks.rados.rados.0.smithi045.stdout:2146: finishing write tid 2 to smithi04526579-40 2024-04-26T00:32:36.568 INFO:tasks.rados.rados.0.smithi045.stdout:2146: finishing write tid 3 to smithi04526579-40 2024-04-26T00:32:36.568 INFO:tasks.rados.rados.0.smithi045.stdout:2146: finishing write tid 4 to smithi04526579-40 2024-04-26T00:32:36.568 INFO:tasks.rados.rados.0.smithi045.stdout:2146: oid 40 updating version 0 to 1452 2024-04-26T00:32:36.568 INFO:tasks.rados.rados.0.smithi045.stdout:2146: oid 40 version 1452 is already newer than 1451 2024-04-26T00:32:36.568 INFO:tasks.rados.rados.0.smithi045.stdout:2146: oid 40 version 1452 is already newer than 1450 2024-04-26T00:32:36.568 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 1452 (ObjNum 784 snap 241 seq_num 784) dirty exists 2024-04-26T00:32:36.568 INFO:tasks.rados.rados.0.smithi045.stdout:2146: left oid 40 (ObjNum 784 snap 241 seq_num 784) 2024-04-26T00:32:36.568 INFO:tasks.rados.rados.0.smithi045.stdout:2147: expect (ObjNum 443 snap 138 seq_num 443) 2024-04-26T00:32:36.737 INFO:tasks.rados.rados.0.smithi045.stdout:2149: finishing rollback tid 0 to smithi04526579-15 2024-04-26T00:32:36.737 INFO:tasks.rados.rados.0.smithi045.stdout:2149: finishing rollback tid 1 to smithi04526579-15 2024-04-26T00:32:36.737 INFO:tasks.rados.rados.0.smithi045.stdout:2149: finishing rollback tid 2 to smithi04526579-15 2024-04-26T00:32:36.737 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 1394 (ObjNum 724 snap 222 seq_num 724) dirty exists 2024-04-26T00:32:36.737 INFO:tasks.rados.rados.0.smithi045.stdout:2146: done (2 left) 2024-04-26T00:32:36.737 INFO:tasks.rados.rados.0.smithi045.stdout:2147: done (1 left) 2024-04-26T00:32:36.738 INFO:tasks.rados.rados.0.smithi045.stdout:2149: done (0 left) 2024-04-26T00:32:36.738 INFO:tasks.rados.rados.0.smithi045.stdout:2150: delete oid 33 current snap is 241 2024-04-26T00:32:36.743 INFO:tasks.rados.rados.0.smithi045.stdout:2150: done (0 left) 2024-04-26T00:32:36.743 INFO:tasks.rados.rados.0.smithi045.stdout:2151: snap_remove snap 153 2024-04-26T00:32:37.585 INFO:tasks.rados.rados.0.smithi045.stdout:2151: done (0 left) 2024-04-26T00:32:37.585 INFO:tasks.rados.rados.0.smithi045.stdout:2152: copy_from oid 3 from oid 41 current snap is 241 2024-04-26T00:32:37.595 INFO:tasks.rados.rados.0.smithi045.stdout:2153: write (excl) oid 36 current snap is 241 2024-04-26T00:32:37.595 INFO:tasks.rados.rados.0.smithi045.stdout:2153: seq_num 786 ranges {410302=730073,1895082=402268,2951884=258418} 2024-04-26T00:32:37.607 INFO:tasks.rados.rados.0.smithi045.stdout:2153: writing smithi04526579-36 from 410302 to 1140375 tid 1 2024-04-26T00:32:37.627 INFO:tasks.rados.rados.0.smithi045.stdout:2153: writing smithi04526579-36 from 1895082 to 2297350 tid 2 2024-04-26T00:32:37.643 INFO:tasks.rados.rados.0.smithi045.stdout:2153: writing smithi04526579-36 from 2951884 to 3210302 tid 3 2024-04-26T00:32:37.643 INFO:tasks.rados.rados.0.smithi045.stdout:2154: write oid 8 current snap is 241 2024-04-26T00:32:37.643 INFO:tasks.rados.rados.0.smithi045.stdout:2154: seq_num 787 ranges {427109=405322,1553564=545926,2579307=647802} 2024-04-26T00:32:37.656 INFO:tasks.rados.rados.0.smithi045.stdout:2154: writing smithi04526579-8 from 427109 to 832431 tid 1 2024-04-26T00:32:37.672 INFO:tasks.rados.rados.0.smithi045.stdout:2154: writing smithi04526579-8 from 1553564 to 2099490 tid 2 2024-04-26T00:32:37.684 INFO:tasks.rados.rados.0.smithi045.stdout:2154: writing smithi04526579-8 from 2579307 to 3227109 tid 3 2024-04-26T00:32:37.684 INFO:tasks.rados.rados.0.smithi045.stdout:2155: copy_from oid 32 from oid 38 current snap is 241 2024-04-26T00:32:37.684 INFO:tasks.rados.rados.0.smithi045.stdout:2153: finishing write tid 1 to smithi04526579-36 2024-04-26T00:32:37.684 INFO:tasks.rados.rados.0.smithi045.stdout:2153: finishing write tid 2 to smithi04526579-36 2024-04-26T00:32:37.684 INFO:tasks.rados.rados.0.smithi045.stdout:2152: finishing copy_from racing read to smithi04526579-3 2024-04-26T00:32:37.684 INFO:tasks.rados.rados.0.smithi045.stdout:2152: finishing copy_from to smithi04526579-3 2024-04-26T00:32:37.685 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 1064 (ObjNum 542 snap 167 seq_num 542) dirty exists 2024-04-26T00:32:37.685 INFO:tasks.rados.rados.0.smithi045.stdout:2153: finishing write tid 3 to smithi04526579-36 2024-04-26T00:32:37.685 INFO:tasks.rados.rados.0.smithi045.stdout:2152: done (3 left) 2024-04-26T00:32:37.685 INFO:tasks.rados.rados.0.smithi045.stdout:2156: snap_remove snap 232 2024-04-26T00:32:38.480 INFO:tasks.rados.rados.0.smithi045.stdout:2156: done (3 left) 2024-04-26T00:32:38.480 INFO:tasks.rados.rados.0.smithi045.stdout:2157: read oid 13 snap -1 2024-04-26T00:32:38.480 INFO:tasks.rados.rados.0.smithi045.stdout:2157: expect (ObjNum 668 snap 210 seq_num 668) 2024-04-26T00:32:38.480 INFO:tasks.rados.rados.0.smithi045.stdout:2158: write oid 10 current snap is 241 2024-04-26T00:32:38.480 INFO:tasks.rados.rados.0.smithi045.stdout:2153: finishing write tid 4 to smithi04526579-36 2024-04-26T00:32:38.480 INFO:tasks.rados.rados.0.smithi045.stdout:2153: finishing write tid 5 to smithi04526579-36 2024-04-26T00:32:38.480 INFO:tasks.rados.rados.0.smithi045.stdout:2153: oid 36 updating version 0 to 1454 2024-04-26T00:32:38.480 INFO:tasks.rados.rados.0.smithi045.stdout:2153: oid 36 version 1454 is already newer than 1453 2024-04-26T00:32:38.480 INFO:tasks.rados.rados.0.smithi045.stdout:2153: oid 36 version 1454 is already newer than 1451 2024-04-26T00:32:38.480 INFO:tasks.rados.rados.0.smithi045.stdout:2153: oid 36 version 1454 is already newer than 1452 2024-04-26T00:32:38.480 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 1454 (ObjNum 786 snap 241 seq_num 786) dirty exists 2024-04-26T00:32:38.481 INFO:tasks.rados.rados.0.smithi045.stdout:2153: left oid 36 (ObjNum 786 snap 241 seq_num 786) 2024-04-26T00:32:38.481 INFO:tasks.rados.rados.0.smithi045.stdout:2154: finishing write tid 1 to smithi04526579-8 2024-04-26T00:32:38.481 INFO:tasks.rados.rados.0.smithi045.stdout:2158: seq_num 788 ranges {443916=480571,1612046=405937,2606730=495052} 2024-04-26T00:32:38.488 INFO:tasks.rados.rados.0.smithi045.stdout:2158: writing smithi04526579-10 from 443916 to 924487 tid 1 2024-04-26T00:32:38.501 INFO:tasks.rados.rados.0.smithi045.stdout:2158: writing smithi04526579-10 from 1612046 to 2017983 tid 2 2024-04-26T00:32:38.513 INFO:tasks.rados.rados.0.smithi045.stdout:2158: writing smithi04526579-10 from 2606730 to 3101782 tid 3 2024-04-26T00:32:38.514 INFO:tasks.rados.rados.0.smithi045.stdout:2153: done (4 left) 2024-04-26T00:32:38.514 INFO:tasks.rados.rados.0.smithi045.stdout:2159: write (excl) oid 5 current snap is 241 2024-04-26T00:32:38.514 INFO:tasks.rados.rados.0.smithi045.stdout:2154: finishing write tid 2 to smithi04526579-8 2024-04-26T00:32:38.514 INFO:tasks.rados.rados.0.smithi045.stdout:2154: finishing write tid 3 to smithi04526579-8 2024-04-26T00:32:38.514 INFO:tasks.rados.rados.0.smithi045.stdout:2154: finishing write tid 4 to smithi04526579-8 2024-04-26T00:32:38.514 INFO:tasks.rados.rados.0.smithi045.stdout:2159: seq_num 789 ranges {460723=555820,1670528=549595,2634153=626570} 2024-04-26T00:32:38.523 INFO:tasks.rados.rados.0.smithi045.stdout:2159: writing smithi04526579-5 from 460723 to 1016543 tid 1 2024-04-26T00:32:38.535 INFO:tasks.rados.rados.0.smithi045.stdout:2159: writing smithi04526579-5 from 1670528 to 2220123 tid 2 2024-04-26T00:32:38.545 INFO:tasks.rados.rados.0.smithi045.stdout:2159: writing smithi04526579-5 from 2634153 to 3260723 tid 3 2024-04-26T00:32:38.545 INFO:tasks.rados.rados.0.smithi045.stdout:2160: write (excl) oid 3 current snap is 241 2024-04-26T00:32:38.545 INFO:tasks.rados.rados.0.smithi045.stdout:2154: finishing write tid 5 to smithi04526579-8 2024-04-26T00:32:38.545 INFO:tasks.rados.rados.0.smithi045.stdout:2154: oid 8 updating version 0 to 1525 2024-04-26T00:32:38.545 INFO:tasks.rados.rados.0.smithi045.stdout:2154: oid 8 version 1525 is already newer than 1524 2024-04-26T00:32:38.545 INFO:tasks.rados.rados.0.smithi045.stdout:2154: oid 8 updating version 1525 to 1526 2024-04-26T00:32:38.545 INFO:tasks.rados.rados.0.smithi045.stdout:2154: oid 8 version 1526 is already newer than 1523 2024-04-26T00:32:38.545 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 1526 (ObjNum 787 snap 241 seq_num 787) dirty exists 2024-04-26T00:32:38.545 INFO:tasks.rados.rados.0.smithi045.stdout:2154: left oid 8 (ObjNum 787 snap 241 seq_num 787) 2024-04-26T00:32:38.545 INFO:tasks.rados.rados.0.smithi045.stdout:2155: finishing copy_from racing read to smithi04526579-32 2024-04-26T00:32:38.546 INFO:tasks.rados.rados.0.smithi045.stdout:2155: finishing copy_from to smithi04526579-32 2024-04-26T00:32:38.546 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 1292 (ObjNum 769 snap 238 seq_num 769) dirty exists 2024-04-26T00:32:38.546 INFO:tasks.rados.rados.0.smithi045.stdout:2157: expect (ObjNum 668 snap 210 seq_num 668) 2024-04-26T00:32:38.804 INFO:tasks.rados.rados.0.smithi045.stdout:2158: finishing write tid 1 to smithi04526579-10 2024-04-26T00:32:38.804 INFO:tasks.rados.rados.0.smithi045.stdout:2158: finishing write tid 2 to smithi04526579-10 2024-04-26T00:32:38.804 INFO:tasks.rados.rados.0.smithi045.stdout:2160: seq_num 790 ranges {477530=631069,1612657=409606,2545223=517596} 2024-04-26T00:32:38.810 INFO:tasks.rados.rados.0.smithi045.stdout:2160: writing smithi04526579-3 from 477530 to 1108599 tid 1 2024-04-26T00:32:38.815 INFO:tasks.rados.rados.0.smithi045.stdout:2160: writing smithi04526579-3 from 1612657 to 2022263 tid 2 2024-04-26T00:32:38.822 INFO:tasks.rados.rados.0.smithi045.stdout:2160: writing smithi04526579-3 from 2545223 to 3062819 tid 3 2024-04-26T00:32:38.822 INFO:tasks.rados.rados.0.smithi045.stdout:2154: done (5 left) 2024-04-26T00:32:38.822 INFO:tasks.rados.rados.0.smithi045.stdout:2155: done (4 left) 2024-04-26T00:32:38.822 INFO:tasks.rados.rados.0.smithi045.stdout:2157: done (3 left) 2024-04-26T00:32:38.822 INFO:tasks.rados.rados.0.smithi045.stdout:2161: read oid 17 snap 195 2024-04-26T00:32:38.822 INFO:tasks.rados.rados.0.smithi045.stdout:2161: expect (ObjNum 595 snap 184 seq_num 595) 2024-04-26T00:32:38.823 INFO:tasks.rados.rados.0.smithi045.stdout:2162: read oid 29 snap -1 2024-04-26T00:32:38.823 INFO:tasks.rados.rados.0.smithi045.stdout:2162: expect (ObjNum 783 snap 240 seq_num 783) 2024-04-26T00:32:38.823 INFO:tasks.rados.rados.0.smithi045.stdout:2163: read oid 13 snap -1 2024-04-26T00:32:38.823 INFO:tasks.rados.rados.0.smithi045.stdout:2163: expect (ObjNum 668 snap 210 seq_num 668) 2024-04-26T00:32:38.823 INFO:tasks.rados.rados.0.smithi045.stdout:2164: copy_from oid 22 from oid 45 current snap is 241 2024-04-26T00:32:38.823 INFO:tasks.rados.rados.0.smithi045.stdout:2158: finishing write tid 3 to smithi04526579-10 2024-04-26T00:32:38.823 INFO:tasks.rados.rados.0.smithi045.stdout:2158: finishing write tid 4 to smithi04526579-10 2024-04-26T00:32:38.823 INFO:tasks.rados.rados.0.smithi045.stdout:2158: finishing write tid 5 to smithi04526579-10 2024-04-26T00:32:38.823 INFO:tasks.rados.rados.0.smithi045.stdout:2158: oid 10 updating version 0 to 1522 2024-04-26T00:32:38.823 INFO:tasks.rados.rados.0.smithi045.stdout:2158: oid 10 version 1522 is already newer than 1521 2024-04-26T00:32:38.823 INFO:tasks.rados.rados.0.smithi045.stdout:2158: oid 10 version 1522 is already newer than 1520 2024-04-26T00:32:38.823 INFO:tasks.rados.rados.0.smithi045.stdout:2158: oid 10 updating version 1522 to 1523 2024-04-26T00:32:38.823 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 1523 (ObjNum 788 snap 241 seq_num 788) dirty exists 2024-04-26T00:32:38.823 INFO:tasks.rados.rados.0.smithi045.stdout:2158: left oid 10 (ObjNum 788 snap 241 seq_num 788) 2024-04-26T00:32:38.823 INFO:tasks.rados.rados.0.smithi045.stdout:2159: finishing write tid 1 to smithi04526579-5 2024-04-26T00:32:38.824 INFO:tasks.rados.rados.0.smithi045.stdout:2159: finishing write tid 2 to smithi04526579-5 2024-04-26T00:32:38.824 INFO:tasks.rados.rados.0.smithi045.stdout:2159: finishing write tid 3 to smithi04526579-5 2024-04-26T00:32:38.824 INFO:tasks.rados.rados.0.smithi045.stdout:2159: finishing write tid 4 to smithi04526579-5 2024-04-26T00:32:38.824 INFO:tasks.rados.rados.0.smithi045.stdout:2159: finishing write tid 5 to smithi04526579-5 2024-04-26T00:32:38.824 INFO:tasks.rados.rados.0.smithi045.stdout:2159: oid 5 updating version 0 to 1844 2024-04-26T00:32:38.824 INFO:tasks.rados.rados.0.smithi045.stdout:2159: oid 5 version 1844 is already newer than 1842 2024-04-26T00:32:38.825 INFO:tasks.rados.rados.0.smithi045.stdout:2159: oid 5 version 1844 is already newer than 1841 2024-04-26T00:32:38.826 INFO:tasks.rados.rados.0.smithi045.stdout:2159: oid 5 version 1844 is already newer than 1843 2024-04-26T00:32:38.826 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 1844 (ObjNum 789 snap 241 seq_num 789) dirty exists 2024-04-26T00:32:38.826 INFO:tasks.rados.rados.0.smithi045.stdout:2159: left oid 5 (ObjNum 789 snap 241 seq_num 789) 2024-04-26T00:32:38.826 INFO:tasks.rados.rados.0.smithi045.stdout:2160: finishing write tid 1 to smithi04526579-3 2024-04-26T00:32:38.826 INFO:tasks.rados.rados.0.smithi045.stdout:2158: done (6 left) 2024-04-26T00:32:38.826 INFO:tasks.rados.rados.0.smithi045.stdout:2159: done (5 left) 2024-04-26T00:32:38.826 INFO:tasks.rados.rados.0.smithi045.stdout:2165: snap_remove snap 230 2024-04-26T00:32:39.595 INFO:tasks.rados.rados.0.smithi045.stdout:2165: done (5 left) 2024-04-26T00:32:39.595 INFO:tasks.rados.rados.0.smithi045.stdout:2166: write (excl) oid 11 current snap is 241 2024-04-26T00:32:39.595 INFO:tasks.rados.rados.0.smithi045.stdout:2166: seq_num 791 ranges {494337=422671,1387492=553264,2688999=445221} 2024-04-26T00:32:39.601 INFO:tasks.rados.rados.0.smithi045.stdout:2166: writing smithi04526579-11 from 494337 to 917008 tid 1 2024-04-26T00:32:39.615 INFO:tasks.rados.rados.0.smithi045.stdout:2166: writing smithi04526579-11 from 1387492 to 1940756 tid 2 2024-04-26T00:32:39.630 INFO:tasks.rados.rados.0.smithi045.stdout:2166: writing smithi04526579-11 from 2688999 to 3134220 tid 3 2024-04-26T00:32:39.631 INFO:tasks.rados.rados.0.smithi045.stdout:2167: snap_remove snap 227 2024-04-26T00:32:40.601 INFO:tasks.rados.rados.0.smithi045.stdout:2167: done (6 left) 2024-04-26T00:32:40.602 INFO:tasks.rados.rados.0.smithi045.stdout:2168: read oid 14 snap 226 2024-04-26T00:32:40.602 INFO:tasks.rados.rados.0.smithi045.stdout:2168: expect deleted 2024-04-26T00:32:40.602 INFO:tasks.rados.rados.0.smithi045.stdout:2169: rollback oid 33 current snap is 241 2024-04-26T00:32:40.602 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 33 to 239 2024-04-26T00:32:40.602 INFO:tasks.rados.rados.0.smithi045.stdout:2160: finishing write tid 2 to smithi04526579-3 2024-04-26T00:32:40.602 INFO:tasks.rados.rados.0.smithi045.stdout:2161: expect (ObjNum 595 snap 184 seq_num 595) 2024-04-26T00:32:40.675 INFO:tasks.rados.rados.0.smithi045.stdout:2160: finishing write tid 3 to smithi04526579-3 2024-04-26T00:32:40.675 INFO:tasks.rados.rados.0.smithi045.stdout:2160: finishing write tid 4 to smithi04526579-3 2024-04-26T00:32:40.675 INFO:tasks.rados.rados.0.smithi045.stdout:2160: finishing write tid 5 to smithi04526579-3 2024-04-26T00:32:40.675 INFO:tasks.rados.rados.0.smithi045.stdout:2160: oid 3 updating version 0 to 1345 2024-04-26T00:32:40.676 INFO:tasks.rados.rados.0.smithi045.stdout:2160: oid 3 updating version 1345 to 1347 2024-04-26T00:32:40.676 INFO:tasks.rados.rados.0.smithi045.stdout:2160: oid 3 version 1347 is already newer than 1344 2024-04-26T00:32:40.676 INFO:tasks.rados.rados.0.smithi045.stdout:2160: oid 3 version 1347 is already newer than 1346 2024-04-26T00:32:40.676 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 1347 (ObjNum 790 snap 241 seq_num 790) dirty exists 2024-04-26T00:32:40.676 INFO:tasks.rados.rados.0.smithi045.stdout:2160: left oid 3 (ObjNum 790 snap 241 seq_num 790) 2024-04-26T00:32:40.676 INFO:tasks.rados.rados.0.smithi045.stdout:2162: expect (ObjNum 783 snap 240 seq_num 783) 2024-04-26T00:32:40.892 INFO:tasks.rados.rados.0.smithi045.stdout:2164: finishing copy_from racing read to smithi04526579-22 2024-04-26T00:32:40.892 INFO:tasks.rados.rados.0.smithi045.stdout:2163: expect (ObjNum 668 snap 210 seq_num 668) 2024-04-26T00:32:41.129 INFO:tasks.rados.rados.0.smithi045.stdout:2164: finishing copy_from to smithi04526579-22 2024-04-26T00:32:41.129 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 1436 (ObjNum 690 snap 215 seq_num 690) dirty exists 2024-04-26T00:32:41.129 INFO:tasks.rados.rados.0.smithi045.stdout:2166: finishing write tid 1 to smithi04526579-11 2024-04-26T00:32:41.129 INFO:tasks.rados.rados.0.smithi045.stdout:2166: finishing write tid 2 to smithi04526579-11 2024-04-26T00:32:41.129 INFO:tasks.rados.rados.0.smithi045.stdout:2166: finishing write tid 3 to smithi04526579-11 2024-04-26T00:32:41.130 INFO:tasks.rados.rados.0.smithi045.stdout:2166: finishing write tid 4 to smithi04526579-11 2024-04-26T00:32:41.130 INFO:tasks.rados.rados.0.smithi045.stdout:2166: finishing write tid 5 to smithi04526579-11 2024-04-26T00:32:41.130 INFO:tasks.rados.rados.0.smithi045.stdout:2166: oid 11 updating version 0 to 1442 2024-04-26T00:32:41.130 INFO:tasks.rados.rados.0.smithi045.stdout:2166: oid 11 updating version 1442 to 1443 2024-04-26T00:32:41.130 INFO:tasks.rados.rados.0.smithi045.stdout:2166: oid 11 version 1443 is already newer than 1440 2024-04-26T00:32:41.130 INFO:tasks.rados.rados.0.smithi045.stdout:2166: oid 11 version 1443 is already newer than 1441 2024-04-26T00:32:41.130 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 1443 (ObjNum 791 snap 241 seq_num 791) dirty exists 2024-04-26T00:32:41.130 INFO:tasks.rados.rados.0.smithi045.stdout:2166: left oid 11 (ObjNum 791 snap 241 seq_num 791) 2024-04-26T00:32:41.131 INFO:tasks.rados.rados.0.smithi045.stdout:2169: finishing rollback tid 1 to smithi04526579-33 2024-04-26T00:32:41.131 INFO:tasks.rados.rados.0.smithi045.stdout:2169: finishing rollback tid 2 to smithi04526579-33 2024-04-26T00:32:41.131 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 1856 (ObjNum 683 snap 213 seq_num 683) dirty exists 2024-04-26T00:32:41.131 INFO:tasks.rados.rados.0.smithi045.stdout:2160: done (7 left) 2024-04-26T00:32:41.131 INFO:tasks.rados.rados.0.smithi045.stdout:2161: done (6 left) 2024-04-26T00:32:41.131 INFO:tasks.rados.rados.0.smithi045.stdout:2162: done (5 left) 2024-04-26T00:32:41.132 INFO:tasks.rados.rados.0.smithi045.stdout:2163: done (4 left) 2024-04-26T00:32:41.132 INFO:tasks.rados.rados.0.smithi045.stdout:2164: done (3 left) 2024-04-26T00:32:41.132 INFO:tasks.rados.rados.0.smithi045.stdout:2166: done (2 left) 2024-04-26T00:32:41.132 INFO:tasks.rados.rados.0.smithi045.stdout:2168: done (1 left) 2024-04-26T00:32:41.132 INFO:tasks.rados.rados.0.smithi045.stdout:2169: done (0 left) 2024-04-26T00:32:41.132 INFO:tasks.rados.rados.0.smithi045.stdout:2170: rollback oid 45 current snap is 241 2024-04-26T00:32:41.133 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 45 to 238 2024-04-26T00:32:41.133 INFO:tasks.rados.rados.0.smithi045.stdout:2171: read oid 47 snap 236 2024-04-26T00:32:41.133 INFO:tasks.rados.rados.0.smithi045.stdout:2171: expect (ObjNum 683 snap 213 seq_num 683) 2024-04-26T00:32:41.133 INFO:tasks.rados.rados.0.smithi045.stdout:2172: copy_from oid 41 from oid 25 current snap is 241 2024-04-26T00:32:41.133 INFO:tasks.rados.rados.0.smithi045.stdout:2173: snap_create 2024-04-26T00:32:41.140 INFO:tasks.rados.rados.0.smithi045.stdout:2170: finishing rollback tid 0 to smithi04526579-45 2024-04-26T00:32:41.140 INFO:tasks.rados.rados.0.smithi045.stdout:2170: finishing rollback tid 1 to smithi04526579-45 2024-04-26T00:32:41.141 INFO:tasks.rados.rados.0.smithi045.stdout:2170: finishing rollback tid 2 to smithi04526579-45 2024-04-26T00:32:41.141 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 1446 (ObjNum 690 snap 215 seq_num 690) dirty exists 2024-04-26T00:32:41.145 INFO:tasks.rados.rados.0.smithi045.stdout:2172: finishing copy_from racing read to smithi04526579-41 2024-04-26T00:32:41.146 INFO:tasks.rados.rados.0.smithi045.stdout:2171: expect (ObjNum 683 snap 213 seq_num 683) 2024-04-26T00:32:41.260 INFO:tasks.rados.rados.0.smithi045.stdout:2172: finishing copy_from to smithi04526579-41 2024-04-26T00:32:41.260 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 1574 (ObjNum 604 snap 186 seq_num 604) dirty exists 2024-04-26T00:32:41.608 INFO:tasks.rados.rados.0.smithi045.stdout:2170: done (3 left) 2024-04-26T00:32:41.608 INFO:tasks.rados.rados.0.smithi045.stdout:2171: done (2 left) 2024-04-26T00:32:41.608 INFO:tasks.rados.rados.0.smithi045.stdout:2172: done (1 left) 2024-04-26T00:32:41.608 INFO:tasks.rados.rados.0.smithi045.stdout:2173: done (0 left) 2024-04-26T00:32:41.609 INFO:tasks.rados.rados.0.smithi045.stdout:2174: read oid 26 snap -1 2024-04-26T00:32:41.609 INFO:tasks.rados.rados.0.smithi045.stdout:2174: expect (ObjNum 731 snap 224 seq_num 731) 2024-04-26T00:32:41.609 INFO:tasks.rados.rados.0.smithi045.stdout:2175: delete oid 19 current snap is 242 2024-04-26T00:32:41.621 INFO:tasks.rados.rados.0.smithi045.stdout:2174: expect (ObjNum 731 snap 224 seq_num 731) 2024-04-26T00:32:41.733 INFO:tasks.rados.rados.0.smithi045.stdout:2174: done (1 left) 2024-04-26T00:32:41.733 INFO:tasks.rados.rados.0.smithi045.stdout:2175: done (0 left) 2024-04-26T00:32:41.733 INFO:tasks.rados.rados.0.smithi045.stdout:2176: write oid 33 current snap is 242 2024-04-26T00:32:41.734 INFO:tasks.rados.rados.0.smithi045.stdout:2176: seq_num 793 ranges {527951=573169,1504456=556933,2743845=467765} 2024-04-26T00:32:41.751 INFO:tasks.rados.rados.0.smithi045.stdout:2176: writing smithi04526579-33 from 527951 to 1101120 tid 1 2024-04-26T00:32:41.764 INFO:tasks.rados.rados.0.smithi045.stdout:2176: writing smithi04526579-33 from 1504456 to 2061389 tid 2 2024-04-26T00:32:41.775 INFO:tasks.rados.rados.0.smithi045.stdout:2176: writing smithi04526579-33 from 2743845 to 3211610 tid 3 2024-04-26T00:32:41.775 INFO:tasks.rados.rados.0.smithi045.stdout:2177: write (excl) oid 42 current snap is 242 2024-04-26T00:32:41.775 INFO:tasks.rados.rados.0.smithi045.stdout:2176: finishing write tid 1 to smithi04526579-33 2024-04-26T00:32:41.775 INFO:tasks.rados.rados.0.smithi045.stdout:2176: finishing write tid 2 to smithi04526579-33 2024-04-26T00:32:41.776 INFO:tasks.rados.rados.0.smithi045.stdout:2177: seq_num 794 ranges {544758=648418,1846585=416944,2771268=573490} 2024-04-26T00:32:41.787 INFO:tasks.rados.rados.0.smithi045.stdout:2177: writing smithi04526579-42 from 544758 to 1193176 tid 1 2024-04-26T00:32:41.801 INFO:tasks.rados.rados.0.smithi045.stdout:2177: writing smithi04526579-42 from 1846585 to 2263529 tid 2 2024-04-26T00:32:41.813 INFO:tasks.rados.rados.0.smithi045.stdout:2177: writing smithi04526579-42 from 2771268 to 3344758 tid 3 2024-04-26T00:32:41.813 INFO:tasks.rados.rados.0.smithi045.stdout:2178: write oid 43 current snap is 242 2024-04-26T00:32:41.813 INFO:tasks.rados.rados.0.smithi045.stdout:2178: seq_num 795 ranges {561565=723667,1905067=560602,3082338=279227} 2024-04-26T00:32:41.824 INFO:tasks.rados.rados.0.smithi045.stdout:2178: writing smithi04526579-43 from 561565 to 1285232 tid 1 2024-04-26T00:32:41.835 INFO:tasks.rados.rados.0.smithi045.stdout:2178: writing smithi04526579-43 from 1905067 to 2465669 tid 2 2024-04-26T00:32:41.841 INFO:tasks.rados.rados.0.smithi045.stdout:2178: writing smithi04526579-43 from 3082338 to 3361565 tid 3 2024-04-26T00:32:41.841 INFO:tasks.rados.rados.0.smithi045.stdout:2179: snap_remove snap 241 2024-04-26T00:32:42.389 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-04-26T00:32:42.389 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-04-26T00:32:42.390 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-26T00:32:42.390 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-26T00:32:42.393 INFO:tasks.ceph.mon.c:Started 2024-04-26T00:32:42.393 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:32:42.393 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:32:42.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60de105640 10 monclient: get_monmap_and_config 2024-04-26T00:32:42.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60de105640 10 monclient: build_initial_monmap 2024-04-26T00:32:42.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60de105640 10 monclient: monmap: 2024-04-26T00:32:42.520 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:42.520 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:42.520 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:32:42.529560+0000 2024-04-26T00:32:42.520 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:32:42.529560+0000 2024-04-26T00:32:42.521 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:42.521 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:42.521 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:32:42.521 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:32:42.521 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:32:42.521 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:32:42.521 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:32:42.521 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:42.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60de105640 1 Processor -- start 2024-04-26T00:32:42.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60de105640 1 -- start start 2024-04-26T00:32:42.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60de105640 10 monclient: init 2024-04-26T00:32:42.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60de105640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:42.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60de105640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-04-26T00:32:42.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60de105640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f60d8136e10 0x7f60d809c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:42.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60de105640 10 monclient(hunting): picked mon.noname-c con 0x7f60d8136e10 addr v2:172.21.15.45:3300/0 2024-04-26T00:32:42.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60de105640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60d81371f0 0x7f60d81375d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:42.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60de105640 10 monclient(hunting): picked mon.noname-b con 0x7f60d81371f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:42.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60de105640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f60d8135d90 addr v1:172.21.15.45:6789/0 2024-04-26T00:32:42.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60de105640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60d809b680 con 0x7f60d8135d90 2024-04-26T00:32:42.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60de105640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:42.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60de105640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f60d809b8f0 con 0x7f60d8136e10 2024-04-26T00:32:42.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60de105640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:42.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60de105640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f60d809bbd0 con 0x7f60d81371f0 2024-04-26T00:32:42.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60de105640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:42.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60de105640 10 monclient(hunting): authenticate will time out at 1743.835056s 2024-04-26T00:32:42.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60d6ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60d81371f0 0x7f60d81375d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:42.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60d6ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60d81371f0 0x7f60d81375d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:37404/0 (socket says 172.21.15.37:37404) 2024-04-26T00:32:42.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60d6ffd640 1 -- 172.21.15.37:0/240169349 learned_addr learned my addr 172.21.15.37:0/240169349 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:32:42.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60d6ffd640 10 monclient(hunting): get_auth_request con 0x7f60d81371f0 auth_method 0 2024-04-26T00:32:42.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:42.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60d6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:42.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:42.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60d6ffd640 1 -- 172.21.15.37:0/240169349 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60d81371f0 msgr2=0x7f60d81375d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:32:42.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60d6ffd640 1 -- 172.21.15.37:0/240169349 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60d81371f0 msgr2=0x7f60d81375d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:32:42.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60d6ffd640 1 --2- 172.21.15.37:0/240169349 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60d81371f0 0x7f60d81375d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:32:42.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60d6ffd640 1 --2- 172.21.15.37:0/240169349 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60d81371f0 0x7f60d81375d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:32:42.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60d77fe640 1 --2- 172.21.15.37:0/240169349 >> v2:172.21.15.45:3300/0 conn(0x7f60d8136e10 0x7f60d809c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:42.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60d77fe640 10 monclient(hunting): get_auth_request con 0x7f60d8136e10 auth_method 0 2024-04-26T00:32:42.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:42.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60d77fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:42.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:42.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:42.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:42.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:42.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60d77fe640 10 monclient(hunting): handle_auth_done global_id 134248 payload 274 2024-04-26T00:32:42.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60d77fe640 1 -- 172.21.15.37:0/240169349 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60d81371f0 msgr2=0x7f60d81375d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:32:42.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60d77fe640 1 --2- 172.21.15.37:0/240169349 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60d81371f0 0x7f60d81375d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:42.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60d77fe640 1 -- 172.21.15.37:0/240169349 >> v1:172.21.15.45:6789/0 conn(0x7f60d8135d90 legacy=0x7f60d8136240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:42.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60d77fe640 10 monclient: _finish_hunting 0 2024-04-26T00:32:42.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60d77fe640 1 monclient: found mon.noname-c 2024-04-26T00:32:42.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:42.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60d77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:32:42.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.524+0000 7f60d77fe640 1 -- 172.21.15.37:0/240169349 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60d8137b10 con 0x7f60d8136e10 2024-04-26T00:32:42.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d77fe640 1 -- 172.21.15.37:0/240169349 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60d8136e10 msgr2=0x7f60d809c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:42.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d77fe640 1 --2- 172.21.15.37:0/240169349 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60d8136e10 0x7f60d809c5b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f60c80097c0 tx=0x7f60c801c320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74d829bd63d9c960 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:42.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d67fc640 1 -- 172.21.15.37:0/240169349 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c80b0070 con 0x7f60d8136e10 2024-04-26T00:32:42.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:42.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:32:42.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d67fc640 10 monclient: dump: 2024-04-26T00:32:42.531 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:42.531 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:42.531 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:42.531 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:42.531 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:42.531 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:42.531 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:42.531 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:42.532 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:42.532 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:42.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d67fc640 10 monclient: _finish_auth 0 2024-04-26T00:32:42.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d67fc640 10 monclient: _check_auth_tickets 2024-04-26T00:32:42.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:42.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d67fc640 1 -- 172.21.15.37:0/240169349 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60c80ab600 con 0x7f60d8136e10 2024-04-26T00:32:42.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60de105640 5 monclient: authenticate success, global_id 134248 2024-04-26T00:32:42.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60de105640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:32:42.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d67fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:42.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60de105640 10 monclient: get_monmap_and_config success 2024-04-26T00:32:42.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60de105640 4 set_mon_vals no callback set 2024-04-26T00:32:42.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d67fc640 1 -- 172.21.15.37:0/240169349 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c80ab9d0 con 0x7f60d8136e10 2024-04-26T00:32:42.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:42.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:32:42.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d67fc640 10 monclient: dump: 2024-04-26T00:32:42.533 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:42.533 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:42.533 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:42.533 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:42.533 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:42.533 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:42.533 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:42.533 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:42.533 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:42.533 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:42.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60de105640 10 monclient: shutdown 2024-04-26T00:32:42.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60de105640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:42.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60de105640 1 -- 172.21.15.37:0/240169349 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60d8136e10 msgr2=0x7f60d809c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:42.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60de105640 1 --2- 172.21.15.37:0/240169349 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60d8136e10 0x7f60d809c5b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f60c80097c0 tx=0x7f60c801c320 comp rx=0 tx=0).stop 2024-04-26T00:32:42.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60de105640 1 -- 172.21.15.37:0/240169349 shutdown_connections 2024-04-26T00:32:42.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60de105640 1 --2- 172.21.15.37:0/240169349 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60d81371f0 0x7f60d81375d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:42.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60de105640 1 --2- 172.21.15.37:0/240169349 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60d8136e10 0x7f60d809c5b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:42.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60de105640 1 -- 172.21.15.37:0/240169349 >> 172.21.15.37:0/240169349 conn(0x7f60d80a64e0 msgr2=0x7f60d80a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:42.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60de105640 1 -- 172.21.15.37:0/240169349 shutdown_connections 2024-04-26T00:32:42.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60de105640 1 -- 172.21.15.37:0/240169349 wait complete. 2024-04-26T00:32:42.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60de105640 10 monclient: build_initial_monmap 2024-04-26T00:32:42.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60de105640 10 monclient: monmap: 2024-04-26T00:32:42.535 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:42.535 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:32:42.535 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:32:42.535 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:32:42.535 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:42.535 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:32:42.535 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:32:42.535 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:32:42.535 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:32:42.535 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:42.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60de105640 1 Processor -- start 2024-04-26T00:32:42.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60de105640 1 -- start start 2024-04-26T00:32:42.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60de105640 10 monclient: init 2024-04-26T00:32:42.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60de105640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:42.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60de105640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:32:42.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60de105640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60d8135d90 0x7f60d81cf5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:42.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60de105640 10 monclient(hunting): picked mon.noname-b con 0x7f60d8135d90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:32:42.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60de105640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60d8136e10 0x7f60d81cfd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:42.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60de105640 10 monclient(hunting): picked mon.noname-a con 0x7f60d8136e10 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:32:42.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60de105640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60d81371f0 0x7f60d813ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:42.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60de105640 10 monclient(hunting): picked mon.noname-c con 0x7f60d81371f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:42.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60de105640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:42.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60de105640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f60d8141f10 con 0x7f60d8136e10 2024-04-26T00:32:42.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60de105640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:42.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60de105640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f60d8141d90 con 0x7f60d8135d90 2024-04-26T00:32:42.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60de105640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:42.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60de105640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f60d8142090 con 0x7f60d81371f0 2024-04-26T00:32:42.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60de105640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:42.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60de105640 10 monclient(hunting): authenticate will time out at 1743.843084s 2024-04-26T00:32:42.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d7fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60d81371f0 0x7f60d813ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:42.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d6ffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60d8136e10 0x7f60d81cfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:42.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d7fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60d81371f0 0x7f60d813ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:37410/0 (socket says 172.21.15.37:37410) 2024-04-26T00:32:42.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d7fff640 1 -- 172.21.15.37:0/3406554530 learned_addr learned my addr 172.21.15.37:0/3406554530 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:32:42.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d7fff640 10 monclient(hunting): get_auth_request con 0x7f60d81371f0 auth_method 0 2024-04-26T00:32:42.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:42.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:42.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:42.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d6ffd640 10 monclient(hunting): get_auth_request con 0x7f60d8136e10 auth_method 0 2024-04-26T00:32:42.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:42.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:42.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:42.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:42.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:42.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d7fff640 1 -- 172.21.15.37:0/3406554530 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60d81371f0 msgr2=0x7f60d813ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:32:42.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d7fff640 1 -- 172.21.15.37:0/3406554530 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60d81371f0 msgr2=0x7f60d813ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:32:42.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d7fff640 1 --2- 172.21.15.37:0/3406554530 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60d81371f0 0x7f60d813ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:32:42.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:42.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d7fff640 1 --2- 172.21.15.37:0/3406554530 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60d81371f0 0x7f60d813ac00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:32:42.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d77fe640 1 --2- 172.21.15.37:0/3406554530 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60d8135d90 0x7f60d81cf5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:42.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d77fe640 10 monclient(hunting): get_auth_request con 0x7f60d8135d90 auth_method 0 2024-04-26T00:32:42.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:42.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d77fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:42.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:42.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d6ffd640 10 monclient(hunting): handle_auth_done global_id 144177 payload 824 2024-04-26T00:32:42.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d6ffd640 1 -- 172.21.15.37:0/3406554530 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60d81371f0 msgr2=0x7f60d813ac00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:32:42.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d6ffd640 1 --2- 172.21.15.37:0/3406554530 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60d81371f0 0x7f60d813ac00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:42.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d6ffd640 1 -- 172.21.15.37:0/3406554530 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60d8135d90 msgr2=0x7f60d81cf5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:42.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d6ffd640 1 --2- 172.21.15.37:0/3406554530 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60d8135d90 0x7f60d81cf5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:42.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d6ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:32:42.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d6ffd640 1 monclient: found mon.noname-a 2024-04-26T00:32:42.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:42.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:32:42.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d6ffd640 1 -- 172.21.15.37:0/3406554530 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60d813dfd0 con 0x7f60d8136e10 2024-04-26T00:32:42.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.532+0000 7f60d77fe640 1 --2- 172.21.15.37:0/3406554530 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60d8135d90 0x7f60d81cf5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:32:42.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.536+0000 7f60d6ffd640 1 -- 172.21.15.37:0/3406554530 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60d8136e10 msgr2=0x7f60d81cfd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:42.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.536+0000 7f60d6ffd640 1 --2- 172.21.15.37:0/3406554530 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60d8136e10 0x7f60d81cfd20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f60cc00e810 tx=0x7f60cc00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:42.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.536+0000 7f60bffff640 1 -- 172.21.15.37:0/3406554530 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60cc017070 con 0x7f60d8136e10 2024-04-26T00:32:42.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.536+0000 7f60bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:42.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.536+0000 7f60bffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:32:42.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.536+0000 7f60bffff640 10 monclient: dump: 2024-04-26T00:32:42.542 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:42.542 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:42.542 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:42.543 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:42.543 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:42.543 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:42.543 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:42.543 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:42.543 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:42.543 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:42.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.536+0000 7f60bffff640 10 monclient: _finish_auth 0 2024-04-26T00:32:42.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.536+0000 7f60bffff640 10 monclient: _check_auth_tickets 2024-04-26T00:32:42.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.536+0000 7f60bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:42.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.536+0000 7f60bffff640 1 -- 172.21.15.37:0/3406554530 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60cc004260 con 0x7f60d8136e10 2024-04-26T00:32:42.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.536+0000 7f60de105640 5 monclient: authenticate success, global_id 144177 2024-04-26T00:32:42.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.536+0000 7f60de105640 10 monclient: _renew_subs 2024-04-26T00:32:42.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.536+0000 7f60de105640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:42.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.536+0000 7f60de105640 1 -- 172.21.15.37:0/3406554530 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60d813e830 con 0x7f60d8136e10 2024-04-26T00:32:42.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.536+0000 7f60bffff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:42.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.536+0000 7f60bffff640 1 -- 172.21.15.37:0/3406554530 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60cc00f040 con 0x7f60d8136e10 2024-04-26T00:32:42.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.536+0000 7f60bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:42.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.536+0000 7f60bffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:42.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.536+0000 7f60bffff640 10 monclient: dump: 2024-04-26T00:32:42.544 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:42.544 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:42.544 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:42.544 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:42.544 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:42.544 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:42.544 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:42.544 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:42.544 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:42.545 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:42.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.536+0000 7f60d5ffb640 4 set_mon_vals no callback set 2024-04-26T00:32:42.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.536+0000 7f60de105640 10 monclient: _renew_subs 2024-04-26T00:32:42.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.536+0000 7f60de105640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:42.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.536+0000 7f60de105640 1 -- 172.21.15.37:0/3406554530 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60d80a7d20 con 0x7f60d8136e10 2024-04-26T00:32:42.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.536+0000 7f60de105640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:42.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.536+0000 7f60de105640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:42.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.536+0000 7f60de105640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:42.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.536+0000 7f60bffff640 1 -- 172.21.15.37:0/3406554530 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f60cc00c6d0 con 0x7f60d8136e10 2024-04-26T00:32:42.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.536+0000 7f60de105640 1 -- 172.21.15.37:0/3406554530 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60d8075910 con 0x7f60d8136e10 2024-04-26T00:32:42.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.536+0000 7f60bffff640 1 --2- 172.21.15.37:0/3406554530 >> v2:172.21.15.45:6812/2379918623 conn(0x7f60c4035e80 0x7f60c4038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:42.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.536+0000 7f60bffff640 1 -- 172.21.15.37:0/3406554530 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(490..490 src has 370..490) ==== 3947+0+0 (secure 0 0 0) 0x7f60cc0485b0 con 0x7f60d8136e10 2024-04-26T00:32:42.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.536+0000 7f60d77fe640 1 --2- 172.21.15.37:0/3406554530 >> v2:172.21.15.45:6812/2379918623 conn(0x7f60c4035e80 0x7f60c4038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:42.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.536+0000 7f60d77fe640 10 monclient: get_auth_request con 0x7f60c4035e80 auth_method 0 2024-04-26T00:32:42.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.540+0000 7f60d77fe640 1 --2- 172.21.15.37:0/3406554530 >> v2:172.21.15.45:6812/2379918623 conn(0x7f60c4035e80 0x7f60c4038340 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f60c80ab000 tx=0x7f60c8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:42.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.544+0000 7f60bffff640 1 -- 172.21.15.37:0/3406554530 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f60cc0472f0 con 0x7f60d8136e10 2024-04-26T00:32:42.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.544+0000 7f60bffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:42.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.544+0000 7f60bffff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:32:42.612 INFO:tasks.rados.rados.0.smithi045.stdout:2179: done (3 left) 2024-04-26T00:32:42.613 INFO:tasks.rados.rados.0.smithi045.stdout:2180: write oid 36 current snap is 242 2024-04-26T00:32:42.613 INFO:tasks.rados.rados.0.smithi045.stdout:2180: seq_num 796 ranges {578372=798916,1963549=420613,2826114=417934} 2024-04-26T00:32:42.623 INFO:tasks.rados.rados.0.smithi045.stdout:2180: writing smithi04526579-36 from 578372 to 1377288 tid 1 2024-04-26T00:32:42.631 INFO:tasks.rados.rados.0.smithi045.stdout:2180: writing smithi04526579-36 from 1963549 to 2384162 tid 2 2024-04-26T00:32:42.637 INFO:tasks.rados.rados.0.smithi045.stdout:2180: writing smithi04526579-36 from 2826114 to 3244048 tid 3 2024-04-26T00:32:42.637 INFO:tasks.rados.rados.0.smithi045.stdout:2181: copy_from oid 2 from oid 1 current snap is 242 2024-04-26T00:32:42.638 INFO:tasks.rados.rados.0.smithi045.stdout:2176: finishing write tid 3 to smithi04526579-33 2024-04-26T00:32:42.638 INFO:tasks.rados.rados.0.smithi045.stdout:2176: finishing write tid 4 to smithi04526579-33 2024-04-26T00:32:42.638 INFO:tasks.rados.rados.0.smithi045.stdout:2176: finishing write tid 5 to smithi04526579-33 2024-04-26T00:32:42.638 INFO:tasks.rados.rados.0.smithi045.stdout:2176: oid 33 updating version 0 to 1868 2024-04-26T00:32:42.638 INFO:tasks.rados.rados.0.smithi045.stdout:2176: oid 33 version 1868 is already newer than 1866 2024-04-26T00:32:42.638 INFO:tasks.rados.rados.0.smithi045.stdout:2176: oid 33 version 1868 is already newer than 1867 2024-04-26T00:32:42.638 INFO:tasks.rados.rados.0.smithi045.stdout:2176: oid 33 updating version 1868 to 1869 2024-04-26T00:32:42.638 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 1869 (ObjNum 793 snap 242 seq_num 793) dirty exists 2024-04-26T00:32:42.638 INFO:tasks.rados.rados.0.smithi045.stdout:2176: left oid 33 (ObjNum 793 snap 242 seq_num 793) 2024-04-26T00:32:42.638 INFO:tasks.rados.rados.0.smithi045.stdout:2177: finishing write tid 1 to smithi04526579-42 2024-04-26T00:32:42.638 INFO:tasks.rados.rados.0.smithi045.stdout:2177: finishing write tid 2 to smithi04526579-42 2024-04-26T00:32:42.638 INFO:tasks.rados.rados.0.smithi045.stdout:2176: done (4 left) 2024-04-26T00:32:42.638 INFO:tasks.rados.rados.0.smithi045.stdout:2182: read oid 24 snap -1 2024-04-26T00:32:42.638 INFO:tasks.rados.rados.0.smithi045.stdout:2182: expect (ObjNum 735 snap 227 seq_num 735) 2024-04-26T00:32:42.639 INFO:tasks.rados.rados.0.smithi045.stdout:2183: delete oid 34 current snap is 242 2024-04-26T00:32:42.639 INFO:tasks.rados.rados.0.smithi045.stdout:2177: finishing write tid 3 to smithi04526579-42 2024-04-26T00:32:42.639 INFO:tasks.rados.rados.0.smithi045.stdout:2177: finishing write tid 4 to smithi04526579-42 2024-04-26T00:32:42.639 INFO:tasks.rados.rados.0.smithi045.stdout:2177: finishing write tid 5 to smithi04526579-42 2024-04-26T00:32:42.639 INFO:tasks.rados.rados.0.smithi045.stdout:2177: oid 42 updating version 0 to 1460 2024-04-26T00:32:42.639 INFO:tasks.rados.rados.0.smithi045.stdout:2177: oid 42 version 1460 is already newer than 1458 2024-04-26T00:32:42.639 INFO:tasks.rados.rados.0.smithi045.stdout:2177: oid 42 version 1460 is already newer than 1459 2024-04-26T00:32:42.639 INFO:tasks.rados.rados.0.smithi045.stdout:2177: oid 42 updating version 1460 to 1461 2024-04-26T00:32:42.639 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 1461 (ObjNum 794 snap 242 seq_num 794) dirty exists 2024-04-26T00:32:42.639 INFO:tasks.rados.rados.0.smithi045.stdout:2177: left oid 42 (ObjNum 794 snap 242 seq_num 794) 2024-04-26T00:32:42.639 INFO:tasks.rados.rados.0.smithi045.stdout:2178: finishing write tid 1 to smithi04526579-43 2024-04-26T00:32:42.639 INFO:tasks.rados.rados.0.smithi045.stdout:2178: finishing write tid 2 to smithi04526579-43 2024-04-26T00:32:42.639 INFO:tasks.rados.rados.0.smithi045.stdout:2178: finishing write tid 3 to smithi04526579-43 2024-04-26T00:32:42.639 INFO:tasks.rados.rados.0.smithi045.stdout:2178: finishing write tid 4 to smithi04526579-43 2024-04-26T00:32:42.640 INFO:tasks.rados.rados.0.smithi045.stdout:2178: finishing write tid 5 to smithi04526579-43 2024-04-26T00:32:42.640 INFO:tasks.rados.rados.0.smithi045.stdout:2178: oid 43 updating version 0 to 1524 2024-04-26T00:32:42.640 INFO:tasks.rados.rados.0.smithi045.stdout:2178: oid 43 updating version 1524 to 1527 2024-04-26T00:32:42.640 INFO:tasks.rados.rados.0.smithi045.stdout:2178: oid 43 version 1527 is already newer than 1526 2024-04-26T00:32:42.640 INFO:tasks.rados.rados.0.smithi045.stdout:2178: oid 43 version 1527 is already newer than 1525 2024-04-26T00:32:42.640 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 1527 (ObjNum 795 snap 242 seq_num 795) dirty exists 2024-04-26T00:32:42.640 INFO:tasks.rados.rados.0.smithi045.stdout:2178: left oid 43 (ObjNum 795 snap 242 seq_num 795) 2024-04-26T00:32:42.643 INFO:tasks.rados.rados.0.smithi045.stdout:2180: finishing write tid 1 to smithi04526579-36 2024-04-26T00:32:42.643 INFO:tasks.rados.rados.0.smithi045.stdout:2180: finishing write tid 2 to smithi04526579-36 2024-04-26T00:32:42.644 INFO:tasks.rados.rados.0.smithi045.stdout:2180: finishing write tid 3 to smithi04526579-36 2024-04-26T00:32:42.646 INFO:tasks.rados.rados.0.smithi045.stdout:2177: done (5 left) 2024-04-26T00:32:42.646 INFO:tasks.rados.rados.0.smithi045.stdout:2178: done (4 left) 2024-04-26T00:32:42.646 INFO:tasks.rados.rados.0.smithi045.stdout:2183: done (3 left) 2024-04-26T00:32:42.646 INFO:tasks.rados.rados.0.smithi045.stdout:2184: snap_create 2024-04-26T00:32:42.646 INFO:tasks.rados.rados.0.smithi045.stdout:2180: finishing write tid 4 to smithi04526579-36 2024-04-26T00:32:42.647 INFO:tasks.rados.rados.0.smithi045.stdout:2180: finishing write tid 5 to smithi04526579-36 2024-04-26T00:32:42.647 INFO:tasks.rados.rados.0.smithi045.stdout:2180: oid 36 updating version 0 to 1458 2024-04-26T00:32:42.647 INFO:tasks.rados.rados.0.smithi045.stdout:2180: oid 36 version 1458 is already newer than 1457 2024-04-26T00:32:42.647 INFO:tasks.rados.rados.0.smithi045.stdout:2180: oid 36 version 1458 is already newer than 1455 2024-04-26T00:32:42.647 INFO:tasks.rados.rados.0.smithi045.stdout:2180: oid 36 version 1458 is already newer than 1456 2024-04-26T00:32:42.647 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 1458 (ObjNum 796 snap 242 seq_num 796) dirty exists 2024-04-26T00:32:42.647 INFO:tasks.rados.rados.0.smithi045.stdout:2180: left oid 36 (ObjNum 796 snap 242 seq_num 796) 2024-04-26T00:32:42.652 INFO:tasks.rados.rados.0.smithi045.stdout:2182: expect (ObjNum 735 snap 227 seq_num 735) 2024-04-26T00:32:42.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.780+0000 7f60de105640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:32:42.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.780+0000 7f60de105640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:32:42.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.780+0000 7f60de105640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:42.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.780+0000 7f60de105640 1 -- 172.21.15.37:0/3406554530 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f60d8075ee0 con 0x7f60d8136e10 2024-04-26T00:32:42.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.784+0000 7f60bffff640 1 -- 172.21.15.37:0/3406554530 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f60cc022300 con 0x7f60d8136e10 2024-04-26T00:32:42.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.784+0000 7f60bffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:32:42.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.784+0000 7f60bffff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:32:42.779 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":94,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:32:42.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.784+0000 7f60de105640 1 -- 172.21.15.37:0/3406554530 >> v2:172.21.15.45:6812/2379918623 conn(0x7f60c4035e80 msgr2=0x7f60c4038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:42.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.784+0000 7f60de105640 1 --2- 172.21.15.37:0/3406554530 >> v2:172.21.15.45:6812/2379918623 conn(0x7f60c4035e80 0x7f60c4038340 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f60c80ab000 tx=0x7f60c8008040 comp rx=0 tx=0).stop 2024-04-26T00:32:42.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.784+0000 7f60de105640 10 monclient: shutdown 2024-04-26T00:32:42.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.784+0000 7f60de105640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:42.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.784+0000 7f60de105640 1 -- 172.21.15.37:0/3406554530 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60d8136e10 msgr2=0x7f60d81cfd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:42.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.784+0000 7f60de105640 1 --2- 172.21.15.37:0/3406554530 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60d8136e10 0x7f60d81cfd20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f60cc00e810 tx=0x7f60cc00ecd0 comp rx=0 tx=0).stop 2024-04-26T00:32:42.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.784+0000 7f60de105640 1 -- 172.21.15.37:0/3406554530 shutdown_connections 2024-04-26T00:32:42.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.784+0000 7f60de105640 1 --2- 172.21.15.37:0/3406554530 >> v2:172.21.15.45:6812/2379918623 conn(0x7f60c4035e80 0x7f60c4038340 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:42.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.784+0000 7f60de105640 1 --2- 172.21.15.37:0/3406554530 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60d81371f0 0x7f60d813ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:42.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.784+0000 7f60de105640 1 --2- 172.21.15.37:0/3406554530 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60d8136e10 0x7f60d81cfd20 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:42.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.784+0000 7f60de105640 1 --2- 172.21.15.37:0/3406554530 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60d8135d90 0x7f60d81cf5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:42.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.784+0000 7f60de105640 1 -- 172.21.15.37:0/3406554530 >> 172.21.15.37:0/3406554530 conn(0x7f60d80a64e0 msgr2=0x7f60d80a4bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:42.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.784+0000 7f60de105640 1 -- 172.21.15.37:0/3406554530 shutdown_connections 2024-04-26T00:32:42.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:42.784+0000 7f60de105640 1 -- 172.21.15.37:0/3406554530 wait complete. 2024-04-26T00:32:42.841 INFO:tasks.rados.rados.0.smithi045.stdout:2181: finishing copy_from racing read to smithi04526579-2 2024-04-26T00:32:42.841 INFO:tasks.rados.rados.0.smithi045.stdout:2181: finishing copy_from to smithi04526579-2 2024-04-26T00:32:42.841 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 1375 (ObjNum 775 snap 240 seq_num 775) dirty exists 2024-04-26T00:32:43.481 INFO:tasks.rados.rados.0.smithi045.stdout:2180: done (3 left) 2024-04-26T00:32:43.481 INFO:tasks.rados.rados.0.smithi045.stdout:2181: done (2 left) 2024-04-26T00:32:43.481 INFO:tasks.rados.rados.0.smithi045.stdout:2182: done (1 left) 2024-04-26T00:32:43.481 INFO:tasks.rados.rados.0.smithi045.stdout:2184: done (0 left) 2024-04-26T00:32:43.481 INFO:tasks.rados.rados.0.smithi045.stdout:2185: delete oid 24 current snap is 243 2024-04-26T00:32:43.498 INFO:tasks.rados.rados.0.smithi045.stdout:2185: done (0 left) 2024-04-26T00:32:43.498 INFO:tasks.rados.rados.0.smithi045.stdout:2186: snap_create 2024-04-26T00:32:44.622 INFO:tasks.rados.rados.0.smithi045.stdout:2186: done (0 left) 2024-04-26T00:32:44.623 INFO:tasks.rados.rados.0.smithi045.stdout:2187: snap_create 2024-04-26T00:32:45.493 INFO:tasks.rados.rados.0.smithi045.stdout:2187: done (0 left) 2024-04-26T00:32:45.493 INFO:tasks.rados.rados.0.smithi045.stdout:2188: copy_from oid 35 from oid 20 current snap is 245 2024-04-26T00:32:45.493 INFO:tasks.rados.rados.0.smithi045.stdout:2189: rollback oid 36 current snap is 245 2024-04-26T00:32:45.493 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 36 to 235 2024-04-26T00:32:45.493 INFO:tasks.rados.rados.0.smithi045.stdout:2190: read oid 14 snap 236 2024-04-26T00:32:45.493 INFO:tasks.rados.rados.0.smithi045.stdout:2190: expect deleted 2024-04-26T00:32:45.493 INFO:tasks.rados.rados.0.smithi045.stdout:2191: read oid 49 snap -1 2024-04-26T00:32:45.493 INFO:tasks.rados.rados.0.smithi045.stdout:2191: expect (ObjNum 724 snap 222 seq_num 724) 2024-04-26T00:32:45.494 INFO:tasks.rados.rados.0.smithi045.stdout:2192: read oid 37 snap -1 2024-04-26T00:32:45.494 INFO:tasks.rados.rados.0.smithi045.stdout:2192: expect (ObjNum 669 snap 210 seq_num 669) 2024-04-26T00:32:45.494 INFO:tasks.rados.rados.0.smithi045.stdout:2193: delete oid 2 current snap is 245 2024-04-26T00:32:45.516 INFO:tasks.rados.rados.0.smithi045.stdout:2189: finishing rollback tid 0 to smithi04526579-36 2024-04-26T00:32:45.518 INFO:tasks.rados.rados.0.smithi045.stdout:2192: expect (ObjNum 669 snap 210 seq_num 669) 2024-04-26T00:32:45.760 INFO:tasks.rados.rados.0.smithi045.stdout:2189: finishing rollback tid 1 to smithi04526579-36 2024-04-26T00:32:45.760 INFO:tasks.rados.rados.0.smithi045.stdout:2189: finishing rollback tid 2 to smithi04526579-36 2024-04-26T00:32:45.760 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 1461 (ObjNum 703 snap 221 seq_num 703) dirty exists 2024-04-26T00:32:45.760 INFO:tasks.rados.rados.0.smithi045.stdout:2191: expect (ObjNum 724 snap 222 seq_num 724) 2024-04-26T00:32:45.798 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:32:45.862 INFO:tasks.rados.rados.0.smithi045.stdout:2188: finishing copy_from racing read to smithi04526579-35 2024-04-26T00:32:45.863 INFO:tasks.rados.rados.0.smithi045.stdout:2188: finishing copy_from to smithi04526579-35 2024-04-26T00:32:45.863 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 1064 (ObjNum 542 snap 167 seq_num 542) dirty exists 2024-04-26T00:32:45.863 INFO:tasks.rados.rados.0.smithi045.stdout:2188: done (5 left) 2024-04-26T00:32:45.863 INFO:tasks.rados.rados.0.smithi045.stdout:2189: done (4 left) 2024-04-26T00:32:45.863 INFO:tasks.rados.rados.0.smithi045.stdout:2190: done (3 left) 2024-04-26T00:32:45.863 INFO:tasks.rados.rados.0.smithi045.stdout:2191: done (2 left) 2024-04-26T00:32:45.863 INFO:tasks.rados.rados.0.smithi045.stdout:2192: done (1 left) 2024-04-26T00:32:45.863 INFO:tasks.rados.rados.0.smithi045.stdout:2193: done (0 left) 2024-04-26T00:32:45.863 INFO:tasks.rados.rados.0.smithi045.stdout:2194: snap_create 2024-04-26T00:32:45.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.932+0000 7fcb03919640 10 monclient: get_monmap_and_config 2024-04-26T00:32:45.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.932+0000 7fcb03919640 10 monclient: build_initial_monmap 2024-04-26T00:32:45.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.932+0000 7fcb03919640 10 monclient: monmap: 2024-04-26T00:32:45.928 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:45.928 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:45.928 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:32:45.937800+0000 2024-04-26T00:32:45.928 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:32:45.937800+0000 2024-04-26T00:32:45.929 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:45.929 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:45.929 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:32:45.929 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:32:45.929 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:32:45.929 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:32:45.929 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:32:45.929 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:45.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.932+0000 7fcb03919640 1 Processor -- start 2024-04-26T00:32:45.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.932+0000 7fcb03919640 1 -- start start 2024-04-26T00:32:45.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.932+0000 7fcb03919640 10 monclient: init 2024-04-26T00:32:45.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.932+0000 7fcb03919640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:45.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.932+0000 7fcb03919640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-04-26T00:32:45.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.932+0000 7fcb03919640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcafc133b20 addr v1:172.21.15.45:6789/0 2024-04-26T00:32:45.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.932+0000 7fcb03919640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcafc133f00 0x7fcafc134350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:45.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.932+0000 7fcb03919640 10 monclient(hunting): picked mon.noname-b con 0x7fcafc133f00 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:45.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.932+0000 7fcb03919640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcafc139050 addr v1:172.21.15.37:6789/0 2024-04-26T00:32:45.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.932+0000 7fcb03919640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcafc09fe20 con 0x7fcafc139050 2024-04-26T00:32:45.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.932+0000 7fcb03919640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcafc072d00 con 0x7fcafc133b20 2024-04-26T00:32:45.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.932+0000 7fcb03919640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:45.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.932+0000 7fcb03919640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fcafc09e620 con 0x7fcafc133f00 2024-04-26T00:32:45.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.932+0000 7fcb03919640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:45.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.932+0000 7fcb03919640 10 monclient(hunting): authenticate will time out at 1747.243472s 2024-04-26T00:32:45.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.932+0000 7fcb00eb4640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcafc133f00 0x7fcafc134350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:45.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.932+0000 7fcb00eb4640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcafc133f00 0x7fcafc134350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:37420/0 (socket says 172.21.15.37:37420) 2024-04-26T00:32:45.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.932+0000 7fcb00eb4640 1 -- 172.21.15.37:0/3401559643 learned_addr learned my addr 172.21.15.37:0/3401559643 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:32:45.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.932+0000 7fcb01eb6640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fcafc139050 0x7fcafc13b440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:36078/0 (socket says 172.21.15.37:36078) 2024-04-26T00:32:45.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.932+0000 7fcb00eb4640 10 monclient(hunting): get_auth_request con 0x7fcafc133f00 auth_method 0 2024-04-26T00:32:45.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.932+0000 7fcb00eb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:45.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.932+0000 7fcb00eb4640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:45.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.932+0000 7fcb00eb4640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:45.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.932+0000 7fcb00eb4640 1 -- 172.21.15.37:0/3401559643 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcafc133f00 msgr2=0x7fcafc134350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:32:45.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.932+0000 7fcb00eb4640 1 -- 172.21.15.37:0/3401559643 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcafc133f00 msgr2=0x7fcafc134350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:32:45.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.932+0000 7fcb00eb4640 1 --2- 172.21.15.37:0/3401559643 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcafc133f00 0x7fcafc134350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:32:45.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.932+0000 7fcb00eb4640 1 --2- 172.21.15.37:0/3401559643 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcafc133f00 0x7fcafc134350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:32:45.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcb01eb6640 1 -- 172.21.15.37:0/3401559643 >> v1:172.21.15.37:6789/0 conn(0x7fcafc139050 legacy=0x7fcafc13b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:45.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcafbfff640 1 -- 172.21.15.37:0/3401559643 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2818471905 0 0) 0x7fcaec001460 con 0x7fcafc139050 2024-04-26T00:32:45.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcb016b5640 1 -- 172.21.15.37:0/3401559643 >> v1:172.21.15.45:6789/0 conn(0x7fcafc133b20 legacy=0x7fcafc09e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:45.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcafbfff640 10 monclient(hunting): _negotiate 2024-04-26T00:32:45.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcafbfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:45.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcafbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:45.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcafbfff640 10 monclient(hunting): my global_id is 144180 2024-04-26T00:32:45.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcafbfff640 1 -- 172.21.15.37:0/3401559643 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcae4003530 con 0x7fcafc139050 2024-04-26T00:32:45.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcafbfff640 1 -- 172.21.15.37:0/3401559643 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3992791423 0 0) 0x7fcae4003530 con 0x7fcafc139050 2024-04-26T00:32:45.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcafbfff640 1 -- 172.21.15.37:0/3401559643 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcafc133f00 msgr2=0x7fcafc134350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:32:45.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcafbfff640 1 --2- 172.21.15.37:0/3401559643 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcafc133f00 0x7fcafc134350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:45.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcafbfff640 1 -- 172.21.15.37:0/3401559643 >> v1:172.21.15.45:6789/0 conn(0x7fcafc133b20 legacy=0x7fcafc09e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:45.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcafbfff640 10 monclient: _finish_hunting 0 2024-04-26T00:32:45.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcafbfff640 1 monclient: found mon.noname-a-legacy 2024-04-26T00:32:45.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcafbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:45.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcafbfff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.37:6789/0 2024-04-26T00:32:45.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcafbfff640 1 -- 172.21.15.37:0/3401559643 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcafc09ba60 con 0x7fcafc139050 2024-04-26T00:32:45.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcafbfff640 10 monclient: _finish_auth 0 2024-04-26T00:32:45.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcafbfff640 10 monclient: _check_auth_tickets 2024-04-26T00:32:45.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcafbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:45.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcafbfff640 1 -- 172.21.15.37:0/3401559643 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 866326386 0 0) 0x7fcaec001b00 con 0x7fcafc139050 2024-04-26T00:32:45.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcafbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:45.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcafbfff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-26T00:32:45.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcafbfff640 10 monclient: dump: 2024-04-26T00:32:45.937 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:45.937 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:45.937 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:45.937 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:45.937 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:45.938 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:45.938 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:45.938 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:45.938 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:45.938 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:45.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcafbfff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] but i'm connected to v1:172.21.15.37:6789/0, reconnecting 2024-04-26T00:32:45.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcafbfff640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:45.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcafbfff640 1 -- 172.21.15.37:0/3401559643 >> v1:172.21.15.37:6789/0 conn(0x7fcafc139050 legacy=0x7fcafc13b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:45.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcafbfff640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:32:45.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcafbfff640 1 --2- 172.21.15.37:0/3401559643 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcae4003a70 0x7fcae4003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:45.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcafbfff640 10 monclient(hunting): picked mon.a con 0x7fcae4003a70 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:32:45.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcafbfff640 1 --2- 172.21.15.37:0/3401559643 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcafc133f00 0x7fcae4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:45.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcafbfff640 10 monclient(hunting): picked mon.c con 0x7fcafc133f00 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:45.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcafbfff640 1 --2- 172.21.15.37:0/3401559643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcae4004e70 0x7fcae400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:45.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcafbfff640 10 monclient(hunting): picked mon.b con 0x7fcae4004e70 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:32:45.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcafbfff640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:45.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcafbfff640 1 -- 172.21.15.37:0/3401559643 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fcafc09e620 con 0x7fcae4003a70 2024-04-26T00:32:45.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcafbfff640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:45.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcafbfff640 1 -- 172.21.15.37:0/3401559643 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcae4005660 con 0x7fcae4004e70 2024-04-26T00:32:45.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcafbfff640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:45.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcafbfff640 1 -- 172.21.15.37:0/3401559643 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fcae400bf60 con 0x7fcafc133f00 2024-04-26T00:32:45.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcafbfff640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:45.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcafbfff640 10 monclient(hunting): _finish_auth 0 2024-04-26T00:32:45.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcb016b5640 1 --2- 172.21.15.37:0/3401559643 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcae4003a70 0x7fcae4003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:45.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcb016b5640 10 monclient(hunting): get_auth_request con 0x7fcae4003a70 auth_method 0 2024-04-26T00:32:45.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcb016b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:45.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcb016b5640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:45.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcb016b5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:32:45.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcb00eb4640 1 --2- 172.21.15.37:0/3401559643 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcafc133f00 0x7fcae4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:45.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcb00eb4640 10 monclient(hunting): get_auth_request con 0x7fcafc133f00 auth_method 0 2024-04-26T00:32:45.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcb00eb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:45.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcb00eb4640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:45.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcb00eb4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:32:45.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcb016b5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:45.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcb016b5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:45.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcb016b5640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:32:45.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcb00eb4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:45.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcb00eb4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:45.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcb00eb4640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:32:45.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcb01eb6640 1 --2- 172.21.15.37:0/3401559643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcae4004e70 0x7fcae400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:45.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcb01eb6640 10 monclient(hunting): get_auth_request con 0x7fcae4004e70 auth_method 0 2024-04-26T00:32:45.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcb01eb6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:45.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcb01eb6640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:45.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcb01eb6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:32:45.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcb016b5640 10 monclient(hunting): handle_auth_done global_id 144180 payload 293 2024-04-26T00:32:45.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcb016b5640 1 -- 172.21.15.37:0/3401559643 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcafc133f00 msgr2=0x7fcae4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:45.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcb016b5640 1 --2- 172.21.15.37:0/3401559643 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcafc133f00 0x7fcae4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:45.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcb016b5640 1 -- 172.21.15.37:0/3401559643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcae4004e70 msgr2=0x7fcae400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:45.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcb016b5640 1 --2- 172.21.15.37:0/3401559643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcae4004e70 0x7fcae400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:45.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcb016b5640 10 monclient: _finish_hunting 0 2024-04-26T00:32:45.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcb016b5640 1 monclient: found mon.a 2024-04-26T00:32:45.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcb016b5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:45.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcb016b5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:45.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcb016b5640 1 -- 172.21.15.37:0/3401559643 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcae4003530 con 0x7fcae4003a70 2024-04-26T00:32:45.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcb016b5640 10 monclient: _finish_auth 0 2024-04-26T00:32:45.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcb016b5640 10 monclient: _check_auth_tickets 2024-04-26T00:32:45.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcb016b5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:45.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcb03919640 5 monclient: authenticate success, global_id 144180 2024-04-26T00:32:45.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.936+0000 7fcb03919640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:32:45.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.940+0000 7fcb016b5640 1 -- 172.21.15.37:0/3401559643 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcae4003a70 msgr2=0x7fcae4003e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:45.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.940+0000 7fcb016b5640 1 --2- 172.21.15.37:0/3401559643 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcae4003a70 0x7fcae4003e90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcaf40019b0 tx=0x7fcaf401c590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=804ad8a0e27f0f08 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:45.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.940+0000 7fcafbfff640 1 -- 172.21.15.37:0/3401559643 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf4001680 con 0x7fcae4003a70 2024-04-26T00:32:45.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.940+0000 7fcafbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:45.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.940+0000 7fcafbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:45.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.940+0000 7fcafbfff640 10 monclient: dump: 2024-04-26T00:32:45.947 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:45.947 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:45.948 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:45.948 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:45.948 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:45.948 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:45.948 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:45.948 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:45.948 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:45.948 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:45.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.940+0000 7fcafbfff640 1 -- 172.21.15.37:0/3401559643 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcaf4007330 con 0x7fcae4003a70 2024-04-26T00:32:45.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.940+0000 7fcb03919640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:32:45.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.940+0000 7fcafbfff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:45.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.940+0000 7fcb03919640 10 monclient: get_monmap_and_config success 2024-04-26T00:32:45.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.940+0000 7fcb03919640 4 set_mon_vals no callback set 2024-04-26T00:32:45.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.940+0000 7fcafbfff640 1 -- 172.21.15.37:0/3401559643 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf40057d0 con 0x7fcae4003a70 2024-04-26T00:32:45.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.940+0000 7fcafbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:45.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.940+0000 7fcafbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:45.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.940+0000 7fcafbfff640 10 monclient: dump: 2024-04-26T00:32:45.949 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:45.949 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:45.949 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:45.949 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:45.949 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:45.949 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:45.949 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:45.949 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:45.949 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:45.949 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:45.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.940+0000 7fcb03919640 10 monclient: shutdown 2024-04-26T00:32:45.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.940+0000 7fcb03919640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:45.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.940+0000 7fcb03919640 1 -- 172.21.15.37:0/3401559643 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcae4003a70 msgr2=0x7fcae4003e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:45.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.940+0000 7fcb03919640 1 --2- 172.21.15.37:0/3401559643 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcae4003a70 0x7fcae4003e90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcaf40019b0 tx=0x7fcaf401c590 comp rx=0 tx=0).stop 2024-04-26T00:32:45.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.940+0000 7fcb01eb6640 1 -- 172.21.15.37:0/3401559643 reap_dead start 2024-04-26T00:32:45.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.940+0000 7fcb03919640 1 -- 172.21.15.37:0/3401559643 shutdown_connections 2024-04-26T00:32:45.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.940+0000 7fcb03919640 1 -- 172.21.15.37:0/3401559643 >> 172.21.15.37:0/3401559643 conn(0x7fcafc0a1ad0 msgr2=0x7fcafc0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:45.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.940+0000 7fcb03919640 1 -- 172.21.15.37:0/3401559643 shutdown_connections 2024-04-26T00:32:45.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.940+0000 7fcb03919640 1 -- 172.21.15.37:0/3401559643 wait complete. 2024-04-26T00:32:45.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb03919640 10 monclient: build_initial_monmap 2024-04-26T00:32:45.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb03919640 10 monclient: monmap: 2024-04-26T00:32:45.951 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:45.951 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:32:45.951 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:32:45.951 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:32:45.951 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:45.951 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:32:45.951 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:32:45.951 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:32:45.951 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:32:45.951 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:45.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb03919640 1 Processor -- start 2024-04-26T00:32:45.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb03919640 1 -- start start 2024-04-26T00:32:45.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb03919640 10 monclient: init 2024-04-26T00:32:45.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb03919640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:45.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb03919640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:32:45.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb03919640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcae4003a70 0x7fcafc130dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:45.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb03919640 10 monclient(hunting): picked mon.noname-c con 0x7fcae4003a70 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:45.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb03919640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcafc1cacc0 0x7fcafc131310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:45.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb03919640 10 monclient(hunting): picked mon.noname-a con 0x7fcafc1cacc0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:32:45.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb03919640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcafc11b620 0x7fcafc11bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:45.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb03919640 10 monclient(hunting): picked mon.noname-b con 0x7fcafc11b620 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:32:45.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb03919640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:45.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb03919640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fcafc0a05f0 con 0x7fcafc1cacc0 2024-04-26T00:32:45.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb03919640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:45.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb03919640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcafc0a0470 con 0x7fcafc11b620 2024-04-26T00:32:45.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb03919640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:45.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb03919640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fcafc0a0770 con 0x7fcae4003a70 2024-04-26T00:32:45.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb03919640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:45.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb03919640 10 monclient(hunting): authenticate will time out at 1747.253508s 2024-04-26T00:32:45.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb016b5640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcae4003a70 0x7fcafc130dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:45.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb00eb4640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcafc1cacc0 0x7fcafc131310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:45.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb016b5640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcae4003a70 0x7fcafc130dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:37440/0 (socket says 172.21.15.37:37440) 2024-04-26T00:32:45.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb00eb4640 10 monclient(hunting): get_auth_request con 0x7fcafc1cacc0 auth_method 0 2024-04-26T00:32:45.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb016b5640 1 -- 172.21.15.37:0/1509635178 learned_addr learned my addr 172.21.15.37:0/1509635178 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:32:45.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb00eb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:45.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb00eb4640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:45.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb00eb4640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:45.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb016b5640 10 monclient(hunting): get_auth_request con 0x7fcae4003a70 auth_method 0 2024-04-26T00:32:45.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb016b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:45.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb016b5640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:45.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb016b5640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:45.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb00eb4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:45.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb00eb4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:45.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb00eb4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:45.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb016b5640 1 -- 172.21.15.37:0/1509635178 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcae4003a70 msgr2=0x7fcafc130dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:32:45.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb016b5640 1 -- 172.21.15.37:0/1509635178 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcae4003a70 msgr2=0x7fcafc130dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:32:45.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb016b5640 1 --2- 172.21.15.37:0/1509635178 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcae4003a70 0x7fcafc130dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:32:45.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb01eb6640 1 --2- 172.21.15.37:0/1509635178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcafc11b620 0x7fcafc11bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:45.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb01eb6640 10 monclient(hunting): get_auth_request con 0x7fcafc11b620 auth_method 0 2024-04-26T00:32:45.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb01eb6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:45.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb01eb6640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:45.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb01eb6640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:45.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb016b5640 1 --2- 172.21.15.37:0/1509635178 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcae4003a70 0x7fcafc130dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:32:45.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb00eb4640 10 monclient(hunting): handle_auth_done global_id 144183 payload 824 2024-04-26T00:32:45.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb00eb4640 1 -- 172.21.15.37:0/1509635178 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcae4003a70 msgr2=0x7fcafc130dd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:32:45.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb00eb4640 1 --2- 172.21.15.37:0/1509635178 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcae4003a70 0x7fcafc130dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:45.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb00eb4640 1 -- 172.21.15.37:0/1509635178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcafc11b620 msgr2=0x7fcafc11bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:45.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb00eb4640 1 --2- 172.21.15.37:0/1509635178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcafc11b620 0x7fcafc11bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:45.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb00eb4640 10 monclient: _finish_hunting 0 2024-04-26T00:32:45.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb00eb4640 1 monclient: found mon.noname-a 2024-04-26T00:32:45.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb00eb4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:45.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb00eb4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:32:45.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb00eb4640 1 -- 172.21.15.37:0/1509635178 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcafc11e560 con 0x7fcafc1cacc0 2024-04-26T00:32:45.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.944+0000 7fcb01eb6640 1 --2- 172.21.15.37:0/1509635178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcafc11b620 0x7fcafc11bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:32:45.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.948+0000 7fcb00eb4640 1 -- 172.21.15.37:0/1509635178 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcafc1cacc0 msgr2=0x7fcafc131310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:45.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.948+0000 7fcb00eb4640 1 --2- 172.21.15.37:0/1509635178 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcafc1cacc0 0x7fcafc131310 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcaf0007dd0 tx=0x7fcaf00127a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:45.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.948+0000 7fcaf9ffb640 1 -- 172.21.15.37:0/1509635178 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf0013a60 con 0x7fcafc1cacc0 2024-04-26T00:32:45.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.948+0000 7fcaf9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:45.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.948+0000 7fcaf9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:32:45.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.948+0000 7fcaf9ffb640 10 monclient: dump: 2024-04-26T00:32:45.957 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:45.957 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:45.957 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:45.957 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:45.957 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:45.957 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:45.957 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:45.957 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:45.958 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:45.958 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:45.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.948+0000 7fcaf9ffb640 10 monclient: _finish_auth 0 2024-04-26T00:32:45.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.948+0000 7fcaf9ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:32:45.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.948+0000 7fcaf9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:45.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.948+0000 7fcaf9ffb640 1 -- 172.21.15.37:0/1509635178 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcaf0013c00 con 0x7fcafc1cacc0 2024-04-26T00:32:45.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.948+0000 7fcb03919640 5 monclient: authenticate success, global_id 144183 2024-04-26T00:32:45.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.948+0000 7fcb03919640 10 monclient: _renew_subs 2024-04-26T00:32:45.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.948+0000 7fcb03919640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:45.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.948+0000 7fcb03919640 1 -- 172.21.15.37:0/1509635178 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcafc0a0000 con 0x7fcafc1cacc0 2024-04-26T00:32:45.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.948+0000 7fcaf9ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:45.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.948+0000 7fcaf9ffb640 1 -- 172.21.15.37:0/1509635178 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf0009e60 con 0x7fcafc1cacc0 2024-04-26T00:32:45.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.948+0000 7fcaf9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:45.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.948+0000 7fcaf9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:45.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.948+0000 7fcaf9ffb640 10 monclient: dump: 2024-04-26T00:32:45.959 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:45.959 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:45.959 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:45.959 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:45.959 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:45.959 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:45.959 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:45.959 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:45.959 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:45.959 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:45.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.948+0000 7fcafb7fe640 4 set_mon_vals no callback set 2024-04-26T00:32:45.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.948+0000 7fcb03919640 10 monclient: _renew_subs 2024-04-26T00:32:45.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.948+0000 7fcb03919640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:45.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.948+0000 7fcb03919640 1 -- 172.21.15.37:0/1509635178 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcafc11e280 con 0x7fcafc1cacc0 2024-04-26T00:32:45.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.948+0000 7fcb03919640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:45.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.948+0000 7fcb03919640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:45.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.948+0000 7fcb03919640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:45.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.948+0000 7fcb03919640 1 -- 172.21.15.37:0/1509635178 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcafc066f40 con 0x7fcafc1cacc0 2024-04-26T00:32:45.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.948+0000 7fcaf9ffb640 1 -- 172.21.15.37:0/1509635178 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fcaf00273f0 con 0x7fcafc1cacc0 2024-04-26T00:32:45.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.948+0000 7fcaf9ffb640 1 --2- 172.21.15.37:0/1509635178 >> v2:172.21.15.45:6812/2379918623 conn(0x7fcae8035ed0 0x7fcae8038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:45.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.948+0000 7fcaf9ffb640 1 -- 172.21.15.37:0/1509635178 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(494..494 src has 370..494) ==== 3915+0+0 (secure 0 0 0) 0x7fcaf004bf90 con 0x7fcafc1cacc0 2024-04-26T00:32:45.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.948+0000 7fcb016b5640 1 --2- 172.21.15.37:0/1509635178 >> v2:172.21.15.45:6812/2379918623 conn(0x7fcae8035ed0 0x7fcae8038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:45.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.948+0000 7fcb016b5640 10 monclient: get_auth_request con 0x7fcae8035ed0 auth_method 0 2024-04-26T00:32:45.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.952+0000 7fcb016b5640 1 --2- 172.21.15.37:0/1509635178 >> v2:172.21.15.45:6812/2379918623 conn(0x7fcae8035ed0 0x7fcae8038390 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fcaf401ac90 tx=0x7fcaf4001530 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:45.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.952+0000 7fcaf9ffb640 1 -- 172.21.15.37:0/1509635178 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fcaf0012e50 con 0x7fcafc1cacc0 2024-04-26T00:32:45.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.952+0000 7fcaf9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:45.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:45.952+0000 7fcaf9ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:32:46.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:46.176+0000 7fcb03919640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:32:46.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:46.176+0000 7fcb03919640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:32:46.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:46.176+0000 7fcb03919640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:46.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:46.176+0000 7fcb03919640 1 -- 172.21.15.37:0/1509635178 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fcafc11cb50 con 0x7fcafc1cacc0 2024-04-26T00:32:46.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:46.176+0000 7fcaf9ffb640 1 -- 172.21.15.37:0/1509635178 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7fcaf001d5a0 con 0x7fcafc1cacc0 2024-04-26T00:32:46.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:46.176+0000 7fcaf9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:32:46.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:46.176+0000 7fcaf9ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:32:46.173 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":94,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:32:46.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:46.180+0000 7fcb03919640 1 -- 172.21.15.37:0/1509635178 >> v2:172.21.15.45:6812/2379918623 conn(0x7fcae8035ed0 msgr2=0x7fcae8038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:46.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:46.180+0000 7fcb03919640 1 --2- 172.21.15.37:0/1509635178 >> v2:172.21.15.45:6812/2379918623 conn(0x7fcae8035ed0 0x7fcae8038390 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fcaf401ac90 tx=0x7fcaf4001530 comp rx=0 tx=0).stop 2024-04-26T00:32:46.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:46.180+0000 7fcb03919640 10 monclient: shutdown 2024-04-26T00:32:46.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:46.180+0000 7fcb03919640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:46.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:46.180+0000 7fcb03919640 1 -- 172.21.15.37:0/1509635178 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcafc1cacc0 msgr2=0x7fcafc131310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:46.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:46.180+0000 7fcb03919640 1 --2- 172.21.15.37:0/1509635178 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcafc1cacc0 0x7fcafc131310 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcaf0007dd0 tx=0x7fcaf00127a0 comp rx=0 tx=0).stop 2024-04-26T00:32:46.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:46.180+0000 7fcb03919640 1 -- 172.21.15.37:0/1509635178 shutdown_connections 2024-04-26T00:32:46.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:46.180+0000 7fcb03919640 1 --2- 172.21.15.37:0/1509635178 >> v2:172.21.15.45:6812/2379918623 conn(0x7fcae8035ed0 0x7fcae8038390 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:46.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:46.180+0000 7fcb03919640 1 --2- 172.21.15.37:0/1509635178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcafc11b620 0x7fcafc11bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:46.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:46.180+0000 7fcb03919640 1 --2- 172.21.15.37:0/1509635178 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcafc1cacc0 0x7fcafc131310 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:46.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:46.180+0000 7fcb03919640 1 --2- 172.21.15.37:0/1509635178 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcae4003a70 0x7fcafc130dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:46.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:46.180+0000 7fcb03919640 1 -- 172.21.15.37:0/1509635178 >> 172.21.15.37:0/1509635178 conn(0x7fcafc0a1ad0 msgr2=0x7fcafc132080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:46.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:46.180+0000 7fcb03919640 1 -- 172.21.15.37:0/1509635178 shutdown_connections 2024-04-26T00:32:46.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:46.180+0000 7fcb03919640 1 -- 172.21.15.37:0/1509635178 wait complete. 2024-04-26T00:32:46.712 INFO:tasks.rados.rados.0.smithi045.stdout:2194: done (0 left) 2024-04-26T00:32:46.712 INFO:tasks.rados.rados.0.smithi045.stdout:2195: rollback oid 50 current snap is 246 2024-04-26T00:32:46.712 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 50 to 240 2024-04-26T00:32:46.712 INFO:tasks.rados.rados.0.smithi045.stdout:2196: read oid 2 snap -1 2024-04-26T00:32:46.712 INFO:tasks.rados.rados.0.smithi045.stdout:2196: expect deleted 2024-04-26T00:32:46.712 INFO:tasks.rados.rados.0.smithi045.stdout:2197: copy_from oid 30 from oid 6 current snap is 246 2024-04-26T00:32:46.712 INFO:tasks.rados.rados.0.smithi045.stdout:2198: read oid 24 snap -1 2024-04-26T00:32:46.712 INFO:tasks.rados.rados.0.smithi045.stdout:2198: expect deleted 2024-04-26T00:32:46.712 INFO:tasks.rados.rados.0.smithi045.stdout:2199: read oid 38 snap 140 2024-04-26T00:32:46.712 INFO:tasks.rados.rados.0.smithi045.stdout:2199: expect (ObjNum 391 snap 120 seq_num 391) 2024-04-26T00:32:46.713 INFO:tasks.rados.rados.0.smithi045.stdout:2200: delete oid 33 current snap is 246 2024-04-26T00:32:46.713 INFO:tasks.rados.rados.0.smithi045.stdout:2197: finishing copy_from to smithi04526579-30 2024-04-26T00:32:46.713 INFO:tasks.rados.rados.0.smithi045.stdout:2197: got expected ENOENT (src dne) 2024-04-26T00:32:46.714 INFO:tasks.rados.rados.0.smithi045.stdout:2197: finishing copy_from racing read to smithi04526579-30 2024-04-26T00:32:46.722 INFO:tasks.rados.rados.0.smithi045.stdout:2195: finishing rollback tid 1 to smithi04526579-50 2024-04-26T00:32:46.722 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 1072 (ObjNum 33 snap 0 seq_num 2242173648) dirty dne 2024-04-26T00:32:46.731 INFO:tasks.rados.rados.0.smithi045.stdout:2195: done (5 left) 2024-04-26T00:32:46.731 INFO:tasks.rados.rados.0.smithi045.stdout:2196: done (4 left) 2024-04-26T00:32:46.731 INFO:tasks.rados.rados.0.smithi045.stdout:2197: done (3 left) 2024-04-26T00:32:46.731 INFO:tasks.rados.rados.0.smithi045.stdout:2198: done (2 left) 2024-04-26T00:32:46.731 INFO:tasks.rados.rados.0.smithi045.stdout:2200: done (1 left) 2024-04-26T00:32:46.731 INFO:tasks.rados.rados.0.smithi045.stdout:2201: snap_remove snap 239 2024-04-26T00:32:48.143 INFO:tasks.rados.rados.0.smithi045.stdout:2201: done (1 left) 2024-04-26T00:32:48.144 INFO:tasks.rados.rados.0.smithi045.stdout:2202: read oid 49 snap -1 2024-04-26T00:32:48.144 INFO:tasks.rados.rados.0.smithi045.stdout:2202: expect (ObjNum 724 snap 222 seq_num 724) 2024-04-26T00:32:48.144 INFO:tasks.rados.rados.0.smithi045.stdout:2203: read oid 11 snap 233 2024-04-26T00:32:48.144 INFO:tasks.rados.rados.0.smithi045.stdout:2203: expect (ObjNum 759 snap 232 seq_num 759) 2024-04-26T00:32:48.144 INFO:tasks.rados.rados.0.smithi045.stdout:2204: write oid 16 current snap is 246 2024-04-26T00:32:48.144 INFO:tasks.rados.rados.0.smithi045.stdout:2204: seq_num 801 ranges {662407=491514,1855959=571609,2846876=615531} 2024-04-26T00:32:48.165 INFO:tasks.rados.rados.0.smithi045.stdout:2204: writing smithi04526579-16 from 662407 to 1153921 tid 1 2024-04-26T00:32:48.184 INFO:tasks.rados.rados.0.smithi045.stdout:2204: writing smithi04526579-16 from 1855959 to 2427568 tid 2 2024-04-26T00:32:48.196 INFO:tasks.rados.rados.0.smithi045.stdout:2204: writing smithi04526579-16 from 2846876 to 3462407 tid 3 2024-04-26T00:32:48.196 INFO:tasks.rados.rados.0.smithi045.stdout:2205: write (excl) oid 28 current snap is 246 2024-04-26T00:32:48.197 INFO:tasks.rados.rados.0.smithi045.stdout:2205: seq_num 802 ranges {679214=566763,1798088=715267,3157946=321268} 2024-04-26T00:32:48.208 INFO:tasks.rados.rados.0.smithi045.stdout:2205: writing smithi04526579-28 from 679214 to 1245977 tid 1 2024-04-26T00:32:48.219 INFO:tasks.rados.rados.0.smithi045.stdout:2205: writing smithi04526579-28 from 1798088 to 2513355 tid 2 2024-04-26T00:32:48.227 INFO:tasks.rados.rados.0.smithi045.stdout:2205: writing smithi04526579-28 from 3157946 to 3479214 tid 3 2024-04-26T00:32:48.227 INFO:tasks.rados.rados.0.smithi045.stdout:2206: rollback oid 15 current snap is 246 2024-04-26T00:32:48.227 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 15 to 242 2024-04-26T00:32:48.227 INFO:tasks.rados.rados.0.smithi045.stdout:2199: expect (ObjNum 391 snap 120 seq_num 391) 2024-04-26T00:32:48.369 INFO:tasks.rados.rados.0.smithi045.stdout:2203: expect (ObjNum 759 snap 232 seq_num 759) 2024-04-26T00:32:48.483 INFO:tasks.rados.rados.0.smithi045.stdout:2202: expect (ObjNum 724 snap 222 seq_num 724) 2024-04-26T00:32:48.587 INFO:tasks.rados.rados.0.smithi045.stdout:2204: finishing write tid 1 to smithi04526579-16 2024-04-26T00:32:48.587 INFO:tasks.rados.rados.0.smithi045.stdout:2204: finishing write tid 2 to smithi04526579-16 2024-04-26T00:32:48.587 INFO:tasks.rados.rados.0.smithi045.stdout:2204: finishing write tid 3 to smithi04526579-16 2024-04-26T00:32:48.587 INFO:tasks.rados.rados.0.smithi045.stdout:2204: finishing write tid 4 to smithi04526579-16 2024-04-26T00:32:48.587 INFO:tasks.rados.rados.0.smithi045.stdout:2204: finishing write tid 5 to smithi04526579-16 2024-04-26T00:32:48.587 INFO:tasks.rados.rados.0.smithi045.stdout:2204: oid 16 updating version 0 to 1137 2024-04-26T00:32:48.587 INFO:tasks.rados.rados.0.smithi045.stdout:2204: oid 16 version 1137 is already newer than 1136 2024-04-26T00:32:48.587 INFO:tasks.rados.rados.0.smithi045.stdout:2204: oid 16 updating version 1137 to 1139 2024-04-26T00:32:48.587 INFO:tasks.rados.rados.0.smithi045.stdout:2204: oid 16 version 1139 is already newer than 1138 2024-04-26T00:32:48.588 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 1139 (ObjNum 801 snap 246 seq_num 801) dirty exists 2024-04-26T00:32:48.588 INFO:tasks.rados.rados.0.smithi045.stdout:2204: left oid 16 (ObjNum 801 snap 246 seq_num 801) 2024-04-26T00:32:48.588 INFO:tasks.rados.rados.0.smithi045.stdout:2205: finishing write tid 1 to smithi04526579-28 2024-04-26T00:32:48.588 INFO:tasks.rados.rados.0.smithi045.stdout:2205: finishing write tid 2 to smithi04526579-28 2024-04-26T00:32:48.588 INFO:tasks.rados.rados.0.smithi045.stdout:2205: finishing write tid 3 to smithi04526579-28 2024-04-26T00:32:48.588 INFO:tasks.rados.rados.0.smithi045.stdout:2205: finishing write tid 4 to smithi04526579-28 2024-04-26T00:32:48.588 INFO:tasks.rados.rados.0.smithi045.stdout:2199: done (5 left) 2024-04-26T00:32:48.588 INFO:tasks.rados.rados.0.smithi045.stdout:2202: done (4 left) 2024-04-26T00:32:48.588 INFO:tasks.rados.rados.0.smithi045.stdout:2203: done (3 left) 2024-04-26T00:32:48.588 INFO:tasks.rados.rados.0.smithi045.stdout:2204: done (2 left) 2024-04-26T00:32:48.588 INFO:tasks.rados.rados.0.smithi045.stdout:2207: snap_remove snap 223 2024-04-26T00:32:49.191 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:32:49.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b3643640 10 monclient: get_monmap_and_config 2024-04-26T00:32:49.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b3643640 10 monclient: build_initial_monmap 2024-04-26T00:32:49.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b3643640 10 monclient: monmap: 2024-04-26T00:32:49.323 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:49.323 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:49.323 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:32:49.332025+0000 2024-04-26T00:32:49.323 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:32:49.332025+0000 2024-04-26T00:32:49.323 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:49.324 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:49.324 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:32:49.324 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:32:49.324 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:32:49.324 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:32:49.324 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:32:49.324 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:49.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b3643640 1 Processor -- start 2024-04-26T00:32:49.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b3643640 1 -- start start 2024-04-26T00:32:49.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b3643640 10 monclient: init 2024-04-26T00:32:49.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b3643640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:49.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b3643640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-04-26T00:32:49.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b3643640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f14ac136e10 0x7f14ac09c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:49.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b3643640 10 monclient(hunting): picked mon.noname-b con 0x7f14ac136e10 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:49.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b3643640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f14ac1371f0 0x7f14ac1375d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:49.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b3643640 10 monclient(hunting): picked mon.noname-c con 0x7f14ac1371f0 addr v2:172.21.15.45:3300/0 2024-04-26T00:32:49.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b3643640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f14ac135d90 addr v1:172.21.15.37:6789/0 2024-04-26T00:32:49.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b3643640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14ac09b680 con 0x7f14ac135d90 2024-04-26T00:32:49.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b3643640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:49.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b3643640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f14ac09b8f0 con 0x7f14ac1371f0 2024-04-26T00:32:49.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b3643640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:49.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b3643640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f14ac09bbd0 con 0x7f14ac136e10 2024-04-26T00:32:49.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b3643640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:49.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b3643640 10 monclient(hunting): authenticate will time out at 1750.637529s 2024-04-26T00:32:49.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b1be0640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f14ac135d90 0x7f14ac136240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:36082/0 (socket says 172.21.15.37:36082) 2024-04-26T00:32:49.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b13df640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f14ac136e10 0x7f14ac09c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:49.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b1be0640 1 -- 172.21.15.37:0/1545623220 learned_addr learned my addr 172.21.15.37:0/1545623220 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:32:49.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b13df640 10 monclient(hunting): get_auth_request con 0x7f14ac136e10 auth_method 0 2024-04-26T00:32:49.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b13df640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:49.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b13df640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:49.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b13df640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:49.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b13df640 1 -- 172.21.15.37:0/1545623220 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f14ac136e10 msgr2=0x7f14ac09c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:32:49.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b13df640 1 -- 172.21.15.37:0/1545623220 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f14ac136e10 msgr2=0x7f14ac09c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:32:49.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b13df640 1 --2- 172.21.15.37:0/1545623220 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f14ac136e10 0x7f14ac09c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:32:49.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b13df640 1 --2- 172.21.15.37:0/1545623220 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f14ac136e10 0x7f14ac09c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:32:49.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b0bde640 1 --2- 172.21.15.37:0/1545623220 >> v2:172.21.15.45:3300/0 conn(0x7f14ac1371f0 0x7f14ac1375d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:49.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b0bde640 10 monclient(hunting): get_auth_request con 0x7f14ac1371f0 auth_method 0 2024-04-26T00:32:49.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b0bde640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:49.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b0bde640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:49.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b0bde640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:49.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b0bde640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:49.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b0bde640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:49.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b0bde640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:49.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b0bde640 10 monclient(hunting): handle_auth_done global_id 134260 payload 274 2024-04-26T00:32:49.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b0bde640 1 -- 172.21.15.37:0/1545623220 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f14ac136e10 msgr2=0x7f14ac09c5b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:32:49.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b0bde640 1 --2- 172.21.15.37:0/1545623220 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f14ac136e10 0x7f14ac09c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:49.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b0bde640 1 -- 172.21.15.37:0/1545623220 >> v1:172.21.15.37:6789/0 conn(0x7f14ac135d90 legacy=0x7f14ac136240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:49.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b0bde640 10 monclient: _finish_hunting 0 2024-04-26T00:32:49.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b0bde640 1 monclient: found mon.noname-c 2024-04-26T00:32:49.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b0bde640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:49.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b0bde640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:32:49.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.327+0000 7f14b0bde640 1 -- 172.21.15.37:0/1545623220 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14ac137b10 con 0x7f14ac1371f0 2024-04-26T00:32:49.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.331+0000 7f14b0bde640 1 -- 172.21.15.37:0/1545623220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14ac1371f0 msgr2=0x7f14ac1375d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:49.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.331+0000 7f14b0bde640 1 --2- 172.21.15.37:0/1545623220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14ac1371f0 0x7f14ac1375d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f149c0097c0 tx=0x7f149c01c320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eff8ab4d1d8db38b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:49.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.331+0000 7f14a3fff640 1 -- 172.21.15.37:0/1545623220 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f149c0b0070 con 0x7f14ac1371f0 2024-04-26T00:32:49.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.331+0000 7f14a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:49.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.331+0000 7f14a3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:32:49.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.331+0000 7f14a3fff640 10 monclient: dump: 2024-04-26T00:32:49.337 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:49.337 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:49.337 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:49.337 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:49.337 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:49.337 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:49.337 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:49.337 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:49.337 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:49.337 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:49.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.331+0000 7f14a3fff640 10 monclient: _finish_auth 0 2024-04-26T00:32:49.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.331+0000 7f14a3fff640 10 monclient: _check_auth_tickets 2024-04-26T00:32:49.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.331+0000 7f14a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:49.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.331+0000 7f14a3fff640 1 -- 172.21.15.37:0/1545623220 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f149c0ab600 con 0x7f14ac1371f0 2024-04-26T00:32:49.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.331+0000 7f14a3fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:49.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.331+0000 7f14a3fff640 1 -- 172.21.15.37:0/1545623220 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f149c0ab9d0 con 0x7f14ac1371f0 2024-04-26T00:32:49.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.331+0000 7f14b3643640 5 monclient: authenticate success, global_id 134260 2024-04-26T00:32:49.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.331+0000 7f14b3643640 10 monclient: get_monmap_and_config success 2024-04-26T00:32:49.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.331+0000 7f14b3643640 4 set_mon_vals no callback set 2024-04-26T00:32:49.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.331+0000 7f14a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:49.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.331+0000 7f14a3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:32:49.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.331+0000 7f14a3fff640 10 monclient: dump: 2024-04-26T00:32:49.338 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:49.338 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:49.338 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:49.338 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:49.338 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:49.339 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:49.339 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:49.339 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:49.339 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:49.339 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:49.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b3643640 10 monclient: shutdown 2024-04-26T00:32:49.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b3643640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:49.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b3643640 1 -- 172.21.15.37:0/1545623220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14ac1371f0 msgr2=0x7f14ac1375d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:49.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b3643640 1 --2- 172.21.15.37:0/1545623220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14ac1371f0 0x7f14ac1375d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f149c0097c0 tx=0x7f149c01c320 comp rx=0 tx=0).stop 2024-04-26T00:32:49.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b3643640 1 -- 172.21.15.37:0/1545623220 shutdown_connections 2024-04-26T00:32:49.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b3643640 1 --2- 172.21.15.37:0/1545623220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14ac1371f0 0x7f14ac1375d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:49.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b3643640 1 --2- 172.21.15.37:0/1545623220 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f14ac136e10 0x7f14ac09c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:49.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b3643640 1 -- 172.21.15.37:0/1545623220 >> 172.21.15.37:0/1545623220 conn(0x7f14ac0a64e0 msgr2=0x7f14ac0a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:49.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b3643640 1 -- 172.21.15.37:0/1545623220 shutdown_connections 2024-04-26T00:32:49.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b3643640 1 -- 172.21.15.37:0/1545623220 wait complete. 2024-04-26T00:32:49.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b3643640 10 monclient: build_initial_monmap 2024-04-26T00:32:49.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b3643640 10 monclient: monmap: 2024-04-26T00:32:49.340 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:49.340 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:32:49.340 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:32:49.340 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:32:49.340 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:49.340 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:32:49.340 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:32:49.340 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:32:49.341 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:32:49.341 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:49.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b3643640 1 Processor -- start 2024-04-26T00:32:49.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b3643640 1 -- start start 2024-04-26T00:32:49.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b3643640 10 monclient: init 2024-04-26T00:32:49.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b3643640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:49.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b3643640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:32:49.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b3643640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f14ac135d90 0x7f14ac1cf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:49.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b3643640 10 monclient(hunting): picked mon.noname-a con 0x7f14ac135d90 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:32:49.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b3643640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f14ac136e10 0x7f14ac134e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:49.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b3643640 10 monclient(hunting): picked mon.noname-c con 0x7f14ac136e10 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:49.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b3643640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14ac1371f0 0x7f14ac13aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:49.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b3643640 10 monclient(hunting): picked mon.noname-b con 0x7f14ac1371f0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:32:49.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b3643640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:49.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b3643640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f14ac141f10 con 0x7f14ac135d90 2024-04-26T00:32:49.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b3643640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:49.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b3643640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f14ac141d90 con 0x7f14ac1371f0 2024-04-26T00:32:49.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b3643640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:49.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b3643640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f14ac142090 con 0x7f14ac136e10 2024-04-26T00:32:49.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b3643640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:49.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b3643640 10 monclient(hunting): authenticate will time out at 1750.646116s 2024-04-26T00:32:49.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b13df640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f14ac135d90 0x7f14ac1cf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:49.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b0bde640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f14ac136e10 0x7f14ac134e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:49.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b13df640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f14ac135d90 0x7f14ac1cf3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:34428/0 (socket says 172.21.15.37:34428) 2024-04-26T00:32:49.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b13df640 1 -- 172.21.15.37:0/3401982400 learned_addr learned my addr 172.21.15.37:0/3401982400 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:32:49.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b13df640 10 monclient(hunting): get_auth_request con 0x7f14ac135d90 auth_method 0 2024-04-26T00:32:49.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b13df640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:49.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b13df640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:49.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b13df640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:49.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b0bde640 10 monclient(hunting): get_auth_request con 0x7f14ac136e10 auth_method 0 2024-04-26T00:32:49.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b0bde640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:49.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b0bde640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:49.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b0bde640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:49.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b13df640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:49.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b13df640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:49.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b13df640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:49.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b0bde640 1 -- 172.21.15.37:0/3401982400 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f14ac136e10 msgr2=0x7f14ac134e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:32:49.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b0bde640 1 -- 172.21.15.37:0/3401982400 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f14ac136e10 msgr2=0x7f14ac134e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:32:49.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b0bde640 1 --2- 172.21.15.37:0/3401982400 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f14ac136e10 0x7f14ac134e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:32:49.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b0bde640 1 --2- 172.21.15.37:0/3401982400 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f14ac136e10 0x7f14ac134e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:32:49.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b1be0640 1 --2- 172.21.15.37:0/3401982400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14ac1371f0 0x7f14ac13aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:49.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b1be0640 10 monclient(hunting): get_auth_request con 0x7f14ac1371f0 auth_method 0 2024-04-26T00:32:49.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b1be0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:49.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b1be0640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:49.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b1be0640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:49.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b13df640 10 monclient(hunting): handle_auth_done global_id 144186 payload 824 2024-04-26T00:32:49.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b13df640 1 -- 172.21.15.37:0/3401982400 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f14ac136e10 msgr2=0x7f14ac134e10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:32:49.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b13df640 1 --2- 172.21.15.37:0/3401982400 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f14ac136e10 0x7f14ac134e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:49.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b13df640 1 -- 172.21.15.37:0/3401982400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14ac1371f0 msgr2=0x7f14ac13aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:49.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b13df640 1 --2- 172.21.15.37:0/3401982400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14ac1371f0 0x7f14ac13aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:49.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b13df640 10 monclient: _finish_hunting 0 2024-04-26T00:32:49.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b13df640 1 monclient: found mon.noname-a 2024-04-26T00:32:49.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b13df640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:49.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b13df640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:32:49.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.335+0000 7f14b13df640 1 -- 172.21.15.37:0/3401982400 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14ac1d02f0 con 0x7f14ac135d90 2024-04-26T00:32:49.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.339+0000 7f14b13df640 1 -- 172.21.15.37:0/3401982400 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f14ac135d90 msgr2=0x7f14ac1cf3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:49.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.339+0000 7f14b13df640 1 --2- 172.21.15.37:0/3401982400 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f14ac135d90 0x7f14ac1cf3e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f14a8007ed0 tx=0x7f14a800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:49.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.339+0000 7f14a1ffb640 1 -- 172.21.15.37:0/3401982400 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a8013790 con 0x7f14ac135d90 2024-04-26T00:32:49.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.339+0000 7f14a1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:49.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.339+0000 7f14a1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:32:49.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.339+0000 7f14a1ffb640 10 monclient: dump: 2024-04-26T00:32:49.346 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:49.346 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:49.346 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:49.346 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:49.346 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:49.346 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:49.346 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:49.346 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:49.346 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:49.346 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:49.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.339+0000 7f14a1ffb640 10 monclient: _finish_auth 0 2024-04-26T00:32:49.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.339+0000 7f14a1ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:32:49.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.339+0000 7f14a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:49.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.339+0000 7f14a1ffb640 1 -- 172.21.15.37:0/3401982400 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14a8019070 con 0x7f14ac135d90 2024-04-26T00:32:49.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.339+0000 7f14b3643640 5 monclient: authenticate success, global_id 144186 2024-04-26T00:32:49.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.339+0000 7f14b3643640 10 monclient: _renew_subs 2024-04-26T00:32:49.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.339+0000 7f14b3643640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:49.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.339+0000 7f14b3643640 1 -- 172.21.15.37:0/3401982400 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14ac13e9e0 con 0x7f14ac135d90 2024-04-26T00:32:49.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.339+0000 7f14a1ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:49.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.339+0000 7f14a1ffb640 1 -- 172.21.15.37:0/3401982400 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a8013b90 con 0x7f14ac135d90 2024-04-26T00:32:49.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.339+0000 7f14a1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:49.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.339+0000 7f14a1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:49.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.339+0000 7f14a1ffb640 10 monclient: dump: 2024-04-26T00:32:49.347 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:49.347 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:49.347 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:49.347 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:49.347 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:49.347 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:49.347 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:49.348 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:49.348 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:49.348 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:49.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.339+0000 7f14a37fe640 4 set_mon_vals no callback set 2024-04-26T00:32:49.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.339+0000 7f14b3643640 10 monclient: _renew_subs 2024-04-26T00:32:49.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.339+0000 7f14b3643640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:49.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.339+0000 7f14b3643640 1 -- 172.21.15.37:0/3401982400 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14ac13e140 con 0x7f14ac135d90 2024-04-26T00:32:49.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.339+0000 7f14b3643640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:49.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.339+0000 7f14b3643640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:49.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.339+0000 7f14b3643640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:49.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.339+0000 7f14b3643640 1 -- 172.21.15.37:0/3401982400 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14ac075910 con 0x7f14ac135d90 2024-04-26T00:32:49.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.339+0000 7f14a1ffb640 1 -- 172.21.15.37:0/3401982400 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f14a8014420 con 0x7f14ac135d90 2024-04-26T00:32:49.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.339+0000 7f14a1ffb640 1 --2- 172.21.15.37:0/3401982400 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1490035eb0 0x7f1490038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:49.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.339+0000 7f14a1ffb640 1 -- 172.21.15.37:0/3401982400 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(497..497 src has 370..497) ==== 3943+0+0 (secure 0 0 0) 0x7f14a80492d0 con 0x7f14ac135d90 2024-04-26T00:32:49.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.339+0000 7f14b0bde640 1 --2- 172.21.15.37:0/3401982400 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1490035eb0 0x7f1490038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:49.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.339+0000 7f14b0bde640 10 monclient: get_auth_request con 0x7f1490035eb0 auth_method 0 2024-04-26T00:32:49.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.343+0000 7f14b0bde640 1 --2- 172.21.15.37:0/3401982400 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1490035eb0 0x7f1490038370 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f14ac09d490 tx=0x7f149c0a4f90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:49.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.343+0000 7f14a1ffb640 1 -- 172.21.15.37:0/3401982400 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f14ac1d02f0 con 0x7f14ac135d90 2024-04-26T00:32:49.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.343+0000 7f14a1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:49.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.343+0000 7f14a1ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:32:49.556 INFO:tasks.rados.rados.0.smithi045.stdout:2207: done (2 left) 2024-04-26T00:32:49.557 INFO:tasks.rados.rados.0.smithi045.stdout:2208: delete oid 8 current snap is 246 2024-04-26T00:32:49.557 INFO:tasks.rados.rados.0.smithi045.stdout:2205: finishing write tid 5 to smithi04526579-28 2024-04-26T00:32:49.557 INFO:tasks.rados.rados.0.smithi045.stdout:2205: oid 28 updating version 0 to 1576 2024-04-26T00:32:49.557 INFO:tasks.rados.rados.0.smithi045.stdout:2205: oid 28 updating version 1576 to 1578 2024-04-26T00:32:49.557 INFO:tasks.rados.rados.0.smithi045.stdout:2205: oid 28 version 1578 is already newer than 1577 2024-04-26T00:32:49.557 INFO:tasks.rados.rados.0.smithi045.stdout:2205: oid 28 version 1578 is already newer than 1575 2024-04-26T00:32:49.557 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 1578 (ObjNum 802 snap 246 seq_num 802) dirty exists 2024-04-26T00:32:49.558 INFO:tasks.rados.rados.0.smithi045.stdout:2205: left oid 28 (ObjNum 802 snap 246 seq_num 802) 2024-04-26T00:32:49.558 INFO:tasks.rados.rados.0.smithi045.stdout:2206: finishing rollback tid 0 to smithi04526579-15 2024-04-26T00:32:49.558 INFO:tasks.rados.rados.0.smithi045.stdout:2206: finishing rollback tid 1 to smithi04526579-15 2024-04-26T00:32:49.558 INFO:tasks.rados.rados.0.smithi045.stdout:2206: finishing rollback tid 2 to smithi04526579-15 2024-04-26T00:32:49.558 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 1581 (ObjNum 724 snap 222 seq_num 724) dirty exists 2024-04-26T00:32:49.577 INFO:tasks.rados.rados.0.smithi045.stdout:2205: done (2 left) 2024-04-26T00:32:49.578 INFO:tasks.rados.rados.0.smithi045.stdout:2206: done (1 left) 2024-04-26T00:32:49.578 INFO:tasks.rados.rados.0.smithi045.stdout:2208: done (0 left) 2024-04-26T00:32:49.578 INFO:tasks.rados.rados.0.smithi045.stdout:2209: read oid 36 snap -1 2024-04-26T00:32:49.578 INFO:tasks.rados.rados.0.smithi045.stdout:2209: expect (ObjNum 703 snap 221 seq_num 703) 2024-04-26T00:32:49.578 INFO:tasks.rados.rados.0.smithi045.stdout:2210: rollback oid 21 current snap is 246 2024-04-26T00:32:49.578 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 21 to 240 2024-04-26T00:32:49.578 INFO:tasks.rados.rados.0.smithi045.stdout:2211: snap_remove snap 225 2024-04-26T00:32:49.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.587+0000 7f14b3643640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:32:49.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.587+0000 7f14b3643640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:32:49.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.587+0000 7f14b3643640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:49.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.587+0000 7f14b3643640 1 -- 172.21.15.37:0/3401982400 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f14ac0a7bd0 con 0x7f14ac135d90 2024-04-26T00:32:49.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.591+0000 7f14a1ffb640 1 -- 172.21.15.37:0/3401982400 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f14a80147c0 con 0x7f14ac135d90 2024-04-26T00:32:49.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.591+0000 7f14a1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:32:49.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.591+0000 7f14a1ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:32:49.587 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":96,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:32:49.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.595+0000 7f14b3643640 1 -- 172.21.15.37:0/3401982400 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1490035eb0 msgr2=0x7f1490038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:49.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.595+0000 7f14b3643640 1 --2- 172.21.15.37:0/3401982400 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1490035eb0 0x7f1490038370 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f14ac09d490 tx=0x7f149c0a4f90 comp rx=0 tx=0).stop 2024-04-26T00:32:49.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.595+0000 7f14b3643640 10 monclient: shutdown 2024-04-26T00:32:49.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.595+0000 7f14b3643640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:49.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.595+0000 7f14b3643640 1 -- 172.21.15.37:0/3401982400 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f14ac135d90 msgr2=0x7f14ac1cf3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:49.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.595+0000 7f14b3643640 1 --2- 172.21.15.37:0/3401982400 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f14ac135d90 0x7f14ac1cf3e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f14a8007ed0 tx=0x7f14a800e4b0 comp rx=0 tx=0).stop 2024-04-26T00:32:49.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.595+0000 7f14b3643640 1 -- 172.21.15.37:0/3401982400 shutdown_connections 2024-04-26T00:32:49.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.595+0000 7f14b3643640 1 --2- 172.21.15.37:0/3401982400 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1490035eb0 0x7f1490038370 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:49.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.595+0000 7f14b3643640 1 --2- 172.21.15.37:0/3401982400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14ac1371f0 0x7f14ac13aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:49.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.595+0000 7f14b3643640 1 --2- 172.21.15.37:0/3401982400 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f14ac136e10 0x7f14ac134e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:49.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.595+0000 7f14b3643640 1 --2- 172.21.15.37:0/3401982400 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f14ac135d90 0x7f14ac1cf3e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:49.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.595+0000 7f14b3643640 1 -- 172.21.15.37:0/3401982400 >> 172.21.15.37:0/3401982400 conn(0x7f14ac0a64e0 msgr2=0x7f14ac0a86b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:49.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.595+0000 7f14b3643640 1 -- 172.21.15.37:0/3401982400 shutdown_connections 2024-04-26T00:32:49.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.595+0000 7f14b3643640 1 -- 172.21.15.37:0/3401982400 wait complete. 2024-04-26T00:32:49.608 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:32:49.609 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:32:49.609 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:32:49.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.719+0000 7fcec329b640 10 monclient: get_monmap_and_config 2024-04-26T00:32:49.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.719+0000 7fcec329b640 10 monclient: build_initial_monmap 2024-04-26T00:32:49.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.723+0000 7fcec329b640 10 monclient: monmap: 2024-04-26T00:32:49.719 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:49.719 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:49.719 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:32:49.727913+0000 2024-04-26T00:32:49.719 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:32:49.727913+0000 2024-04-26T00:32:49.719 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:49.719 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:49.719 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:32:49.719 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:32:49.719 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:32:49.719 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:32:49.719 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:32:49.719 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:49.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.723+0000 7fcec329b640 1 Processor -- start 2024-04-26T00:32:49.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.723+0000 7fcec329b640 1 -- start start 2024-04-26T00:32:49.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.723+0000 7fcec329b640 10 monclient: init 2024-04-26T00:32:49.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.723+0000 7fcec329b640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:49.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.723+0000 7fcec329b640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-04-26T00:32:49.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.723+0000 7fcec329b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcebc0a3d90 addr v1:172.21.15.45:6789/0 2024-04-26T00:32:49.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.723+0000 7fcec329b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcebc0a4170 addr v1:172.21.15.37:6789/0 2024-04-26T00:32:49.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.723+0000 7fcec329b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcebc13d3f0 0x7fcebc13f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:49.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.723+0000 7fcec329b640 10 monclient(hunting): picked mon.noname-b con 0x7fcebc13d3f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:49.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.723+0000 7fcec329b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcebc0a5150 con 0x7fcebc0a4170 2024-04-26T00:32:49.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.723+0000 7fcec329b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcebc09b280 con 0x7fcebc0a3d90 2024-04-26T00:32:49.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.723+0000 7fcec329b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:49.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.723+0000 7fcec329b640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fcebc09c650 con 0x7fcebc13d3f0 2024-04-26T00:32:49.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.723+0000 7fcec329b640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:49.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.723+0000 7fcec329b640 10 monclient(hunting): authenticate will time out at 1751.033510s 2024-04-26T00:32:49.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.723+0000 7fcec1838640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcebc13d3f0 0x7fcebc13f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:49.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.723+0000 7fcec0836640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fcebc0a4170 0x7fcebc0a45e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:48918/0 (socket says 172.21.15.37:48918) 2024-04-26T00:32:49.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.723+0000 7fcec0836640 1 -- 172.21.15.37:0/2631839468 learned_addr learned my addr 172.21.15.37:0/2631839468 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:32:49.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.723+0000 7fcec1838640 10 monclient(hunting): get_auth_request con 0x7fcebc13d3f0 auth_method 0 2024-04-26T00:32:49.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.723+0000 7fcec1838640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:49.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.723+0000 7fcec1838640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:49.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.723+0000 7fcec1838640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:49.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.723+0000 7fcec1838640 1 -- 172.21.15.37:0/2631839468 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcebc13d3f0 msgr2=0x7fcebc13f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:32:49.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.723+0000 7fcec1838640 1 -- 172.21.15.37:0/2631839468 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcebc13d3f0 msgr2=0x7fcebc13f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:32:49.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.723+0000 7fcec1838640 1 --2- 172.21.15.37:0/2631839468 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcebc13d3f0 0x7fcebc13f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:32:49.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.723+0000 7fcec1838640 1 --2- 172.21.15.37:0/2631839468 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcebc13d3f0 0x7fcebc13f7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:32:49.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.723+0000 7fcec0836640 1 -- 172.21.15.37:0/2631839468 >> v1:172.21.15.37:6789/0 conn(0x7fcebc0a4170 legacy=0x7fcebc0a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:49.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fceb3fff640 1 -- 172.21.15.37:0/2631839468 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1625152179 0 0) 0x7fcebc0a5150 con 0x7fcebc0a4170 2024-04-26T00:32:49.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fceb3fff640 10 monclient(hunting): _negotiate 2024-04-26T00:32:49.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fceb3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:49.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fceb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:49.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fceb3fff640 10 monclient(hunting): my global_id is 144189 2024-04-26T00:32:49.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fceb3fff640 1 -- 172.21.15.37:0/2631839468 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcea8003530 con 0x7fcebc0a4170 2024-04-26T00:32:49.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fcec1037640 1 -- 172.21.15.37:0/2631839468 >> v1:172.21.15.45:6789/0 conn(0x7fcebc0a3d90 legacy=0x7fcebc09bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:49.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fceb3fff640 1 -- 172.21.15.37:0/2631839468 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1533813224 0 0) 0x7fcea8003530 con 0x7fcebc0a4170 2024-04-26T00:32:49.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fceb3fff640 1 -- 172.21.15.37:0/2631839468 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcebc13d3f0 msgr2=0x7fcebc13f7e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:32:49.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fceb3fff640 1 --2- 172.21.15.37:0/2631839468 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcebc13d3f0 0x7fcebc13f7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:49.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fceb3fff640 1 -- 172.21.15.37:0/2631839468 >> v1:172.21.15.45:6789/0 conn(0x7fcebc0a3d90 legacy=0x7fcebc09bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:49.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fceb3fff640 10 monclient: _finish_hunting 0 2024-04-26T00:32:49.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fceb3fff640 1 monclient: found mon.noname-a-legacy 2024-04-26T00:32:49.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fceb3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:49.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fceb3fff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.37:6789/0 2024-04-26T00:32:49.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fceb3fff640 1 -- 172.21.15.37:0/2631839468 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcebc09b560 con 0x7fcebc0a4170 2024-04-26T00:32:49.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fceb3fff640 10 monclient: _finish_auth 0 2024-04-26T00:32:49.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fceb3fff640 10 monclient: _check_auth_tickets 2024-04-26T00:32:49.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fceb3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:49.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fceb3fff640 1 -- 172.21.15.37:0/2631839468 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 866326386 0 0) 0x7fceac001930 con 0x7fcebc0a4170 2024-04-26T00:32:49.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fceb3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:49.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fceb3fff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-26T00:32:49.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fceb3fff640 10 monclient: dump: 2024-04-26T00:32:49.729 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:49.729 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:49.729 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:49.729 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:49.729 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:49.729 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:49.729 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:49.729 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:49.729 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:49.729 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:49.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fceb3fff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] but i'm connected to v1:172.21.15.37:6789/0, reconnecting 2024-04-26T00:32:49.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fceb3fff640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:49.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fceb3fff640 1 -- 172.21.15.37:0/2631839468 >> v1:172.21.15.37:6789/0 conn(0x7fcebc0a4170 legacy=0x7fcebc0a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:49.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fceb3fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:32:49.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fceb3fff640 1 --2- 172.21.15.37:0/2631839468 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcea8003a70 0x7fcea8003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:49.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fceb3fff640 10 monclient(hunting): picked mon.a con 0x7fcea8003a70 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:32:49.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fceb3fff640 1 --2- 172.21.15.37:0/2631839468 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcebc13d3f0 0x7fcea8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:49.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fceb3fff640 10 monclient(hunting): picked mon.c con 0x7fcebc13d3f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:49.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fceb3fff640 1 --2- 172.21.15.37:0/2631839468 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcea8004e70 0x7fcea800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:49.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fceb3fff640 10 monclient(hunting): picked mon.b con 0x7fcea8004e70 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:32:49.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fceb3fff640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:49.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fceb3fff640 1 -- 172.21.15.37:0/2631839468 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fcebc09c650 con 0x7fcea8003a70 2024-04-26T00:32:49.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fceb3fff640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:49.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fceb3fff640 1 -- 172.21.15.37:0/2631839468 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcea8005630 con 0x7fcea8004e70 2024-04-26T00:32:49.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fceb3fff640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:49.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fceb3fff640 1 -- 172.21.15.37:0/2631839468 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fcea800bf40 con 0x7fcebc13d3f0 2024-04-26T00:32:49.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fceb3fff640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:49.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fceb3fff640 10 monclient(hunting): _finish_auth 0 2024-04-26T00:32:49.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fcec0836640 1 --2- 172.21.15.37:0/2631839468 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcebc13d3f0 0x7fcea8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:49.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fcec1037640 1 --2- 172.21.15.37:0/2631839468 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcea8003a70 0x7fcea8003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:49.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fcec0836640 10 monclient(hunting): get_auth_request con 0x7fcebc13d3f0 auth_method 0 2024-04-26T00:32:49.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fcec0836640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:49.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fcec0836640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:49.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fcec0836640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:32:49.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fcec1037640 10 monclient(hunting): get_auth_request con 0x7fcea8003a70 auth_method 0 2024-04-26T00:32:49.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fcec1037640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:49.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fcec1037640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:49.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fcec1037640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:32:49.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fcec1037640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:49.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fcec1037640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:49.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fcec1037640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:32:49.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fcec0836640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:49.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fcec0836640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:49.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fcec0836640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:32:49.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fcec1838640 1 --2- 172.21.15.37:0/2631839468 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcea8004e70 0x7fcea800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:49.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fcec1838640 10 monclient(hunting): get_auth_request con 0x7fcea8004e70 auth_method 0 2024-04-26T00:32:49.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fcec1838640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:49.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fcec1838640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:49.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fcec1838640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:32:49.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fcec1037640 10 monclient(hunting): handle_auth_done global_id 144189 payload 293 2024-04-26T00:32:49.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fcec1037640 1 -- 172.21.15.37:0/2631839468 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcebc13d3f0 msgr2=0x7fcea8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:49.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fcec1037640 1 --2- 172.21.15.37:0/2631839468 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcebc13d3f0 0x7fcea8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:49.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fcec1037640 1 -- 172.21.15.37:0/2631839468 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcea8004e70 msgr2=0x7fcea800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:49.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fcec1037640 1 --2- 172.21.15.37:0/2631839468 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcea8004e70 0x7fcea800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:49.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fcec1037640 10 monclient: _finish_hunting 0 2024-04-26T00:32:49.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fcec1037640 1 monclient: found mon.a 2024-04-26T00:32:49.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fcec1037640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:49.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fcec1037640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:49.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fcec1037640 1 -- 172.21.15.37:0/2631839468 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcea8003530 con 0x7fcea8003a70 2024-04-26T00:32:49.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fcec1037640 10 monclient: _finish_auth 0 2024-04-26T00:32:49.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fcec1037640 10 monclient: _check_auth_tickets 2024-04-26T00:32:49.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fcec1037640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:49.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fcec329b640 5 monclient: authenticate success, global_id 144189 2024-04-26T00:32:49.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.727+0000 7fcec329b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:32:49.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec1037640 1 -- 172.21.15.37:0/2631839468 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcea8003a70 msgr2=0x7fcea8003e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:49.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec1037640 1 --2- 172.21.15.37:0/2631839468 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcea8003a70 0x7fcea8003e90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fceb8001980 tx=0x7fceb801c5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f04b0ac1dc521e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:49.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fceb3fff640 1 -- 172.21.15.37:0/2631839468 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb8001680 con 0x7fcea8003a70 2024-04-26T00:32:49.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fceb3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:49.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fceb3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:49.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fceb3fff640 10 monclient: dump: 2024-04-26T00:32:49.740 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:49.740 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:49.740 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:49.740 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:49.740 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:49.740 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:49.740 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:49.740 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:49.740 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:49.740 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:49.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fceb3fff640 1 -- 172.21.15.37:0/2631839468 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fceb8007330 con 0x7fcea8003a70 2024-04-26T00:32:49.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fceb3fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:49.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fceb3fff640 1 -- 172.21.15.37:0/2631839468 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb80057d0 con 0x7fcea8003a70 2024-04-26T00:32:49.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec329b640 10 monclient: get_monmap_and_config success 2024-04-26T00:32:49.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec329b640 4 set_mon_vals no callback set 2024-04-26T00:32:49.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fceb3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:49.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fceb3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:49.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fceb3fff640 10 monclient: dump: 2024-04-26T00:32:49.741 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:49.741 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:49.741 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:49.741 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:49.741 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:49.741 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:49.742 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:49.742 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:49.742 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:49.742 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:49.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec329b640 10 monclient: shutdown 2024-04-26T00:32:49.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec329b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:49.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec329b640 1 -- 172.21.15.37:0/2631839468 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcea8003a70 msgr2=0x7fcea8003e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:49.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec329b640 1 --2- 172.21.15.37:0/2631839468 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcea8003a70 0x7fcea8003e90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fceb8001980 tx=0x7fceb801c5b0 comp rx=0 tx=0).stop 2024-04-26T00:32:49.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec1838640 1 -- 172.21.15.37:0/2631839468 reap_dead start 2024-04-26T00:32:49.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec329b640 1 -- 172.21.15.37:0/2631839468 shutdown_connections 2024-04-26T00:32:49.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec329b640 1 -- 172.21.15.37:0/2631839468 >> 172.21.15.37:0/2631839468 conn(0x7fcebc0aa0a0 msgr2=0x7fcebc0a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:49.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec329b640 1 -- 172.21.15.37:0/2631839468 shutdown_connections 2024-04-26T00:32:49.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec329b640 1 -- 172.21.15.37:0/2631839468 wait complete. 2024-04-26T00:32:49.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec329b640 10 monclient: build_initial_monmap 2024-04-26T00:32:49.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec329b640 10 monclient: monmap: 2024-04-26T00:32:49.743 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:49.743 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:32:49.743 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:32:49.743 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:32:49.743 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:49.743 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:32:49.743 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:32:49.743 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:32:49.743 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:32:49.743 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:49.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec329b640 1 Processor -- start 2024-04-26T00:32:49.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec329b640 1 -- start start 2024-04-26T00:32:49.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec329b640 10 monclient: init 2024-04-26T00:32:49.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec329b640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:49.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec329b640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:32:49.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec329b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcea8003a70 0x7fcebc13c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:49.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec329b640 10 monclient(hunting): picked mon.noname-b con 0x7fcea8003a70 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:32:49.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec329b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcebc13c7a0 0x7fcebc1ced30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:49.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec329b640 10 monclient(hunting): picked mon.noname-c con 0x7fcebc13c7a0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:49.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec329b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcebc1cf270 0x7fcebc1cf6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:49.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec329b640 10 monclient(hunting): picked mon.noname-a con 0x7fcebc1cf270 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:32:49.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec329b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:49.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec329b640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fcebc1403c0 con 0x7fcebc1cf270 2024-04-26T00:32:49.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec329b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:49.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec329b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcebc140240 con 0x7fcea8003a70 2024-04-26T00:32:49.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec329b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:49.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec329b640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fcebc140540 con 0x7fcebc13c7a0 2024-04-26T00:32:49.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec329b640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:49.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec329b640 10 monclient(hunting): authenticate will time out at 1751.043282s 2024-04-26T00:32:49.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec0836640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcebc13c7a0 0x7fcebc1ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:49.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec0836640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcebc13c7a0 0x7fcebc1ced30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:47940/0 (socket says 172.21.15.37:47940) 2024-04-26T00:32:49.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec0836640 1 -- 172.21.15.37:0/3972125363 learned_addr learned my addr 172.21.15.37:0/3972125363 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:32:49.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec0836640 10 monclient(hunting): get_auth_request con 0x7fcebc13c7a0 auth_method 0 2024-04-26T00:32:49.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec0836640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:49.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec0836640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:49.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec0836640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:49.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec1838640 1 --2- 172.21.15.37:0/3972125363 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcebc1cf270 0x7fcebc1cf6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:49.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec1838640 10 monclient(hunting): get_auth_request con 0x7fcebc1cf270 auth_method 0 2024-04-26T00:32:49.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec1838640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:49.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec1838640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:49.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec1838640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:49.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec0836640 1 -- 172.21.15.37:0/3972125363 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcebc13c7a0 msgr2=0x7fcebc1ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:32:49.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec0836640 1 -- 172.21.15.37:0/3972125363 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcebc13c7a0 msgr2=0x7fcebc1ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:32:49.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec0836640 1 --2- 172.21.15.37:0/3972125363 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcebc13c7a0 0x7fcebc1ced30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:32:49.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec1838640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:49.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec1838640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:49.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec0836640 1 --2- 172.21.15.37:0/3972125363 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcebc13c7a0 0x7fcebc1ced30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:32:49.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec1838640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:49.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec1037640 1 --2- 172.21.15.37:0/3972125363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcea8003a70 0x7fcebc13c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:49.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec1037640 10 monclient(hunting): get_auth_request con 0x7fcea8003a70 auth_method 0 2024-04-26T00:32:49.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec1037640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:49.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec1037640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:49.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec1037640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:49.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec1838640 10 monclient(hunting): handle_auth_done global_id 144192 payload 824 2024-04-26T00:32:49.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec1838640 1 -- 172.21.15.37:0/3972125363 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcebc13c7a0 msgr2=0x7fcebc1ced30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:32:49.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec1838640 1 --2- 172.21.15.37:0/3972125363 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcebc13c7a0 0x7fcebc1ced30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:49.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec1838640 1 -- 172.21.15.37:0/3972125363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcea8003a70 msgr2=0x7fcebc13c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:49.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec1838640 1 --2- 172.21.15.37:0/3972125363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcea8003a70 0x7fcebc13c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:49.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec1838640 10 monclient: _finish_hunting 0 2024-04-26T00:32:49.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec1838640 1 monclient: found mon.noname-a 2024-04-26T00:32:49.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec1838640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:49.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec1838640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:32:49.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.731+0000 7fcec1838640 1 -- 172.21.15.37:0/3972125363 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcebc0a0e90 con 0x7fcebc1cf270 2024-04-26T00:32:49.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fcec1838640 1 -- 172.21.15.37:0/3972125363 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcebc1cf270 msgr2=0x7fcebc1cf6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:49.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fcec1838640 1 --2- 172.21.15.37:0/3972125363 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcebc1cf270 0x7fcebc1cf6f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcebc075c30 tx=0x7fceb40047f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:49.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fceb1ffb640 1 -- 172.21.15.37:0/3972125363 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb4014410 con 0x7fcebc1cf270 2024-04-26T00:32:49.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fceb1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:49.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fceb1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:32:49.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fceb1ffb640 10 monclient: dump: 2024-04-26T00:32:49.747 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:49.747 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:49.748 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:49.748 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:49.748 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:49.748 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:49.748 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:49.748 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:49.748 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:49.748 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:49.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fceb1ffb640 10 monclient: _finish_auth 0 2024-04-26T00:32:49.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fceb1ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:32:49.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fceb1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:49.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fceb1ffb640 1 -- 172.21.15.37:0/3972125363 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fceb4019070 con 0x7fcebc1cf270 2024-04-26T00:32:49.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fceb1ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:49.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fceb1ffb640 1 -- 172.21.15.37:0/3972125363 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb4014810 con 0x7fcebc1cf270 2024-04-26T00:32:49.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fceb1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:49.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fceb1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:49.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fceb1ffb640 10 monclient: dump: 2024-04-26T00:32:49.749 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:49.749 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:49.749 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:49.749 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:49.749 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:49.749 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:49.749 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:49.749 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:49.749 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:49.750 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:49.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fcec329b640 5 monclient: authenticate success, global_id 144192 2024-04-26T00:32:49.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fceb37fe640 4 set_mon_vals no callback set 2024-04-26T00:32:49.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fcec329b640 10 monclient: _renew_subs 2024-04-26T00:32:49.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fcec329b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:49.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fcec329b640 1 -- 172.21.15.37:0/3972125363 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcebc138a40 con 0x7fcebc1cf270 2024-04-26T00:32:49.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fcec329b640 10 monclient: _renew_subs 2024-04-26T00:32:49.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fcec329b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:49.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fcec329b640 1 -- 172.21.15.37:0/3972125363 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcebc13a490 con 0x7fcebc1cf270 2024-04-26T00:32:49.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fcec329b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:49.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fcec329b640 1 --2- 172.21.15.37:0/3972125363 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcebc13a6d0 0x7fcebc132d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:49.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fcec329b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:49.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fcec329b640 1 -- 172.21.15.37:0/3972125363 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fcebc138300 con 0x7fcebc13a6d0 2024-04-26T00:32:49.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fceb1ffb640 1 -- 172.21.15.37:0/3972125363 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fceb40149b0 con 0x7fcebc1cf270 2024-04-26T00:32:49.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fcec1037640 1 --2- 172.21.15.37:0/3972125363 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcebc13a6d0 0x7fcebc132d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:49.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fcec1037640 10 monclient: get_auth_request con 0x7fcebc13a6d0 auth_method 0 2024-04-26T00:32:49.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fcec1037640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:49.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fcec1037640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:49.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fcec1037640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:49.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fcec1037640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:49.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fcec1037640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:49.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fcec1037640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:49.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fceb1ffb640 1 --2- 172.21.15.37:0/3972125363 >> v2:172.21.15.45:6812/2379918623 conn(0x7fcea0035ed0 0x7fcea0038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:49.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fceb1ffb640 1 -- 172.21.15.37:0/3972125363 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(498..498 src has 370..498) ==== 3943+0+0 (secure 0 0 0) 0x7fceb4048df0 con 0x7fcebc1cf270 2024-04-26T00:32:49.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fcec1037640 10 monclient(hunting): handle_auth_done global_id 144195 payload 824 2024-04-26T00:32:49.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fcec1037640 1 -- 172.21.15.37:0/3972125363 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcebc04f510 con 0x7fcebc13a6d0 2024-04-26T00:32:49.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.735+0000 7fcec0836640 1 --2- 172.21.15.37:0/3972125363 >> v2:172.21.15.45:6812/2379918623 conn(0x7fcea0035ed0 0x7fcea0038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:49.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.739+0000 7fcec0836640 10 monclient: get_auth_request con 0x7fcea0035ed0 auth_method 0 2024-04-26T00:32:49.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.739+0000 7fcec1037640 1 -- 172.21.15.37:0/3972125363 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcebc13a6d0 msgr2=0x7fcebc132d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:49.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.739+0000 7fcec1037640 1 --2- 172.21.15.37:0/3972125363 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcebc13a6d0 0x7fcebc132d20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fceb8009000 tx=0x7fceb80193a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:49.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.739+0000 7fceb1ffb640 1 -- 172.21.15.37:0/3972125363 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb8001680 con 0x7fcebc13a6d0 2024-04-26T00:32:49.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.739+0000 7fceb1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:49.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.739+0000 7fceb1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:49.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.739+0000 7fceb1ffb640 10 monclient: dump: 2024-04-26T00:32:49.752 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:49.752 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:49.752 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:49.752 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:49.752 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:49.752 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:49.752 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:49.752 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:49.752 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:49.752 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:49.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.739+0000 7fceb1ffb640 1 -- 172.21.15.37:0/3972125363 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fceb80057d0 con 0x7fcebc13a6d0 2024-04-26T00:32:49.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.739+0000 7fceb1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:49.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.739+0000 7fceb1ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:32:49.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.739+0000 7fceb1ffb640 1 -- 172.21.15.37:0/3972125363 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcebc13a6d0 msgr2=0x7fcebc132d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:49.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.739+0000 7fceb1ffb640 1 --2- 172.21.15.37:0/3972125363 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcebc13a6d0 0x7fcebc132d20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fceb8009000 tx=0x7fceb80193a0 comp rx=0 tx=0).stop 2024-04-26T00:32:49.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.739+0000 7fceb1ffb640 1 -- 172.21.15.37:0/3972125363 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcebc13a6d0 msgr2=0x7fcebc132d20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:32:49.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.739+0000 7fceb1ffb640 1 --2- 172.21.15.37:0/3972125363 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcebc13a6d0 0x7fcebc132d20 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fceb8009000 tx=0x7fceb80193a0 comp rx=0 tx=0).stop 2024-04-26T00:32:49.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.739+0000 7fcec0836640 1 --2- 172.21.15.37:0/3972125363 >> v2:172.21.15.45:6812/2379918623 conn(0x7fcea0035ed0 0x7fcea0038390 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fceac007fd0 tx=0x7fceac009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:49.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.747+0000 7fcec329b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:32:49.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.747+0000 7fcec329b640 1 --2- 172.21.15.37:0/3972125363 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcebc133260 0x7fcebc133670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:49.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.747+0000 7fcec329b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:49.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.747+0000 7fcec329b640 1 -- 172.21.15.37:0/3972125363 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fcebc1397c0 con 0x7fcebc133260 2024-04-26T00:32:49.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.747+0000 7fcec1037640 1 --2- 172.21.15.37:0/3972125363 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcebc133260 0x7fcebc133670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:49.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.747+0000 7fcec1037640 10 monclient: get_auth_request con 0x7fcebc133260 auth_method 0 2024-04-26T00:32:49.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.747+0000 7fcec1037640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:49.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.747+0000 7fcec1037640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:49.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.747+0000 7fcec1037640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:49.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.747+0000 7fcec1037640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:49.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.747+0000 7fcec1037640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:49.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.747+0000 7fcec1037640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:49.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.747+0000 7fcec1037640 10 monclient(hunting): handle_auth_done global_id 144198 payload 824 2024-04-26T00:32:49.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.747+0000 7fcec1037640 1 -- 172.21.15.37:0/3972125363 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcebc133bb0 con 0x7fcebc133260 2024-04-26T00:32:49.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.747+0000 7fcec1037640 1 -- 172.21.15.37:0/3972125363 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcebc133260 msgr2=0x7fcebc133670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:49.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.747+0000 7fcec1037640 1 --2- 172.21.15.37:0/3972125363 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcebc133260 0x7fcebc133670 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fceb801c5f0 tx=0x7fcea8005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:49.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.747+0000 7fceb1ffb640 1 -- 172.21.15.37:0/3972125363 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb80c0d60 con 0x7fcebc133260 2024-04-26T00:32:49.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.747+0000 7fceb1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:49.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.747+0000 7fceb1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:49.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.747+0000 7fceb1ffb640 10 monclient: dump: 2024-04-26T00:32:49.756 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:49.756 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:49.756 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:49.756 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:49.757 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:49.757 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:49.757 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:49.757 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:49.757 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:49.757 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:49.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.747+0000 7fceb1ffb640 1 -- 172.21.15.37:0/3972125363 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4718 (secure 0 0 0) 0x7fceb80c4070 con 0x7fcebc133260 2024-04-26T00:32:49.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.747+0000 7fceb1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:32:49.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.747+0000 7fceb1ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:32:49.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.747+0000 7fceb1ffb640 1 -- 172.21.15.37:0/3972125363 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcebc133260 msgr2=0x7fcebc133670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:49.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.747+0000 7fceb1ffb640 1 --2- 172.21.15.37:0/3972125363 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcebc133260 0x7fcebc133670 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fceb801c5f0 tx=0x7fcea8005840 comp rx=0 tx=0).stop 2024-04-26T00:32:49.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.747+0000 7fceb1ffb640 1 -- 172.21.15.37:0/3972125363 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcebc133260 msgr2=0x7fcebc133670 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:32:49.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.747+0000 7fceb1ffb640 1 --2- 172.21.15.37:0/3972125363 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcebc133260 0x7fcebc133670 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fceb801c5f0 tx=0x7fcea8005840 comp rx=0 tx=0).stop 2024-04-26T00:32:49.758 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:32:49.758 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:32:49.758 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:32:49.758 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:32:49.758 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 96, 2024-04-26T00:32:49.758 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 56291, 2024-04-26T00:32:49.758 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:32:49.758 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:32:49.758 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:32:49.758 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:32:49.758 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:49.759 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 1, 2024-04-26T00:32:49.759 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:32:49.759 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:32:49.759 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:32:49.759 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:49.759 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:49.759 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:49.759 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:49.759 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:49.759 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:49.759 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:49.759 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:49.759 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:49.759 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:49.759 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:49.759 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:49.760 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:32:49.760 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:32:49.760 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:49.760 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:49.760 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:49.760 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:49.761 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:49.761 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:49.761 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:49.761 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:49.761 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:49.761 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:49.762 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:49.762 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:49.762 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:49.762 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:32:49.762 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:32:49.762 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:32:49.762 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:32:49.762 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:32:49.762 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:32:49.762 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:32:49.762 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:32:49.762 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:32:49.762 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:32:49.762 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:32:49.763 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:32:49.763 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:32:49.763 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:32:49.763 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:32:49.763 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:32:49.763 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:32:49.763 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:49.763 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:49.763 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:49.763 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:49.763 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:49.763 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:49.763 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:49.763 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:49.763 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:49.763 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:49.764 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:49.764 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:49.764 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:32:49.764 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:49.764 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:32:49.764 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:49.764 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:32:49.764 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:32:49.764 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:49.764 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:49.764 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:49.764 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:49.764 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:32:49.764 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:49.764 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:49.765 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:49.765 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:49.765 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:32:49.765 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:49.765 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:49.765 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:49.765 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:49.765 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:32:49.765 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:32:49.765 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:49.765 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:49.765 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:49.765 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:49.765 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:49.765 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:32:49.766 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:32:49.766 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:49.766 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:49.766 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:49.766 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:49.766 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:32:49.766 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:49.766 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:49.766 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:49.766 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:49.766 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:32:49.766 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:49.766 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:49.766 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:49.766 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:49.767 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:32:49.767 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:32:49.767 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:49.767 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:49.767 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:49.767 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:49.767 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:49.767 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:32:49.767 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:32:49.767 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:49.767 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:49.767 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:49.767 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:49.767 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:32:49.767 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:49.768 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:49.768 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:49.768 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:49.768 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:32:49.768 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:49.768 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:49.768 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:49.768 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:49.768 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:32:49.768 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:32:49.768 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:49.768 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:49.768 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:49.768 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:49.768 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:49.769 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:49.769 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:32:49.769 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:32:49.769 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:49.769 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:49.769 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:49.769 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:32:49.769 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:49.769 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:49.769 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:32:49.769 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:49.769 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:49.769 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:49.769 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:32:49.769 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:49.769 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:49.770 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:32:49.770 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:49.770 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:49.770 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:49.770 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:32:49.770 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:49.770 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:49.770 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:49.770 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:32:49.770 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:32:49.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.747+0000 7fcec329b640 1 -- 172.21.15.37:0/3972125363 >> v2:172.21.15.45:6812/2379918623 conn(0x7fcea0035ed0 msgr2=0x7fcea0038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:49.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.747+0000 7fcec329b640 1 --2- 172.21.15.37:0/3972125363 >> v2:172.21.15.45:6812/2379918623 conn(0x7fcea0035ed0 0x7fcea0038390 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fceac007fd0 tx=0x7fceac009040 comp rx=0 tx=0).stop 2024-04-26T00:32:49.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.747+0000 7fcec329b640 10 monclient: shutdown 2024-04-26T00:32:49.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.747+0000 7fcec329b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:49.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.747+0000 7fcec329b640 1 -- 172.21.15.37:0/3972125363 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcebc1cf270 msgr2=0x7fcebc1cf6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:49.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.747+0000 7fcec329b640 1 --2- 172.21.15.37:0/3972125363 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcebc1cf270 0x7fcebc1cf6f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcebc075c30 tx=0x7fceb40047f0 comp rx=0 tx=0).stop 2024-04-26T00:32:49.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.747+0000 7fcec1838640 1 -- 172.21.15.37:0/3972125363 reap_dead start 2024-04-26T00:32:49.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.747+0000 7fcec329b640 1 -- 172.21.15.37:0/3972125363 shutdown_connections 2024-04-26T00:32:49.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.747+0000 7fcec329b640 1 --2- 172.21.15.37:0/3972125363 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcebc1cf270 0x7fcebc1cf6f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:49.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.747+0000 7fcec329b640 1 -- 172.21.15.37:0/3972125363 >> 172.21.15.37:0/3972125363 conn(0x7fcebc0aa0a0 msgr2=0x7fcebc0a2630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:49.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.747+0000 7fcec329b640 1 -- 172.21.15.37:0/3972125363 shutdown_connections 2024-04-26T00:32:49.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.747+0000 7fcec329b640 1 -- 172.21.15.37:0/3972125363 wait complete. 2024-04-26T00:32:49.773 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:32:49.773 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:32:49.774 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:32:49.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.895+0000 7f0f50b98640 10 monclient: get_monmap_and_config 2024-04-26T00:32:49.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.895+0000 7f0f50b98640 10 monclient: build_initial_monmap 2024-04-26T00:32:49.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.895+0000 7f0f50b98640 10 monclient: monmap: 2024-04-26T00:32:49.894 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:49.894 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:49.894 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:32:49.903620+0000 2024-04-26T00:32:49.894 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:32:49.903620+0000 2024-04-26T00:32:49.894 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:49.895 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:49.895 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:32:49.895 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:32:49.895 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:32:49.895 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:32:49.895 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:32:49.895 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:49.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f50b98640 1 Processor -- start 2024-04-26T00:32:49.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f50b98640 1 -- start start 2024-04-26T00:32:49.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f50b98640 10 monclient: init 2024-04-26T00:32:49.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f50b98640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:49.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f50b98640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-04-26T00:32:49.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f50b98640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0f4c0a0300 addr v1:172.21.15.45:6789/0 2024-04-26T00:32:49.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f50b98640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f0f4c09fc50 0x7f0f4c139ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:49.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f50b98640 10 monclient(hunting): picked mon.noname-a con 0x7f0f4c09fc50 addr v2:172.21.15.37:3300/0 2024-04-26T00:32:49.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f50b98640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c13a400 0x7f0f4c13c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:49.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f50b98640 10 monclient(hunting): picked mon.noname-b con 0x7f0f4c13a400 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:49.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f50b98640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f4c09c190 con 0x7f0f4c0a0300 2024-04-26T00:32:49.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f50b98640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:49.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f50b98640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f0f4c0a1110 con 0x7f0f4c09fc50 2024-04-26T00:32:49.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f50b98640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:49.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f50b98640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0f4c0a16d0 con 0x7f0f4c13a400 2024-04-26T00:32:49.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f50b98640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:49.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f50b98640 10 monclient(hunting): authenticate will time out at 1751.209253s 2024-04-26T00:32:49.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f49d9b640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f0f4c09fc50 0x7f0f4c139ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:49.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f49d9b640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f0f4c09fc50 0x7f0f4c139ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44640/0 (socket says 172.21.15.37:44640) 2024-04-26T00:32:49.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f49d9b640 1 -- 172.21.15.37:0/3677251224 learned_addr learned my addr 172.21.15.37:0/3677251224 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:32:49.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f49d9b640 10 monclient(hunting): get_auth_request con 0x7f0f4c09fc50 auth_method 0 2024-04-26T00:32:49.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f49d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:49.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f49d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:49.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f49d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:49.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f4ad9d640 1 --2- 172.21.15.37:0/3677251224 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c13a400 0x7f0f4c13c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:49.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f4ad9d640 10 monclient(hunting): get_auth_request con 0x7f0f4c13a400 auth_method 0 2024-04-26T00:32:49.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f4ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:49.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f4ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:49.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f4ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:49.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f49d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:49.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f49d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:49.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f49d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:49.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f4ad9d640 1 -- 172.21.15.37:0/3677251224 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c13a400 msgr2=0x7f0f4c13c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:32:49.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f4ad9d640 1 -- 172.21.15.37:0/3677251224 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c13a400 msgr2=0x7f0f4c13c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:32:49.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f4ad9d640 1 --2- 172.21.15.37:0/3677251224 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c13a400 0x7f0f4c13c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:32:49.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f4ad9d640 1 --2- 172.21.15.37:0/3677251224 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c13a400 0x7f0f4c13c7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:32:49.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f49d9b640 10 monclient(hunting): handle_auth_done global_id 144201 payload 274 2024-04-26T00:32:49.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f49d9b640 1 -- 172.21.15.37:0/3677251224 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c13a400 msgr2=0x7f0f4c13c7f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:32:49.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f49d9b640 1 --2- 172.21.15.37:0/3677251224 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c13a400 0x7f0f4c13c7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:49.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f49d9b640 1 -- 172.21.15.37:0/3677251224 >> v1:172.21.15.45:6789/0 conn(0x7f0f4c0a0300 legacy=0x7f0f4c0a06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:49.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f49d9b640 10 monclient: _finish_hunting 0 2024-04-26T00:32:49.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f49d9b640 1 monclient: found mon.noname-a 2024-04-26T00:32:49.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f49d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:49.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f49d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:32:49.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.899+0000 7f0f49d9b640 1 -- 172.21.15.37:0/3677251224 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f4c09f2a0 con 0x7f0f4c09fc50 2024-04-26T00:32:49.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.903+0000 7f0f49d9b640 1 -- 172.21.15.37:0/3677251224 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0f4c09fc50 msgr2=0x7f0f4c139ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:49.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.903+0000 7f0f49d9b640 1 --2- 172.21.15.37:0/3677251224 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0f4c09fc50 0x7f0f4c139ec0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c09f4d0 tx=0x7f0f4001c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=128b5fef2e615cc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:49.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.903+0000 7f0f4959a640 1 -- 172.21.15.37:0/3677251224 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f400a5670 con 0x7f0f4c09fc50 2024-04-26T00:32:49.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.903+0000 7f0f4959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:49.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.903+0000 7f0f4959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:32:49.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.903+0000 7f0f4959a640 10 monclient: dump: 2024-04-26T00:32:49.908 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:49.908 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:49.908 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:49.908 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:49.908 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:49.908 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:49.908 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:49.908 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:49.908 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:49.908 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:49.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.903+0000 7f0f4959a640 10 monclient: _finish_auth 0 2024-04-26T00:32:49.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.903+0000 7f0f4959a640 10 monclient: _check_auth_tickets 2024-04-26T00:32:49.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.903+0000 7f0f4959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:49.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.903+0000 7f0f4959a640 1 -- 172.21.15.37:0/3677251224 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f400a5810 con 0x7f0f4c09fc50 2024-04-26T00:32:49.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.903+0000 7f0f4959a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:49.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.903+0000 7f0f4959a640 1 -- 172.21.15.37:0/3677251224 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f400a5be0 con 0x7f0f4c09fc50 2024-04-26T00:32:49.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.903+0000 7f0f50b98640 5 monclient: authenticate success, global_id 144201 2024-04-26T00:32:49.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.903+0000 7f0f50b98640 10 monclient: get_monmap_and_config success 2024-04-26T00:32:49.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.903+0000 7f0f50b98640 4 set_mon_vals no callback set 2024-04-26T00:32:49.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.903+0000 7f0f4959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:49.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.903+0000 7f0f4959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:49.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.903+0000 7f0f4959a640 10 monclient: dump: 2024-04-26T00:32:49.909 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:49.909 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:49.909 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:49.909 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:49.909 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:49.910 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:49.910 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:49.910 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:49.910 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:49.910 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:49.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.903+0000 7f0f50b98640 10 monclient: shutdown 2024-04-26T00:32:49.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.903+0000 7f0f50b98640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:49.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.903+0000 7f0f50b98640 1 -- 172.21.15.37:0/3677251224 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0f4c09fc50 msgr2=0x7f0f4c139ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:49.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.903+0000 7f0f50b98640 1 --2- 172.21.15.37:0/3677251224 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0f4c09fc50 0x7f0f4c139ec0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c09f4d0 tx=0x7f0f4001c7d0 comp rx=0 tx=0).stop 2024-04-26T00:32:49.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.903+0000 7f0f50b98640 1 -- 172.21.15.37:0/3677251224 shutdown_connections 2024-04-26T00:32:49.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.903+0000 7f0f50b98640 1 --2- 172.21.15.37:0/3677251224 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c13a400 0x7f0f4c13c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:49.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.903+0000 7f0f50b98640 1 --2- 172.21.15.37:0/3677251224 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0f4c09fc50 0x7f0f4c139ec0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:49.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.903+0000 7f0f50b98640 1 -- 172.21.15.37:0/3677251224 >> 172.21.15.37:0/3677251224 conn(0x7f0f4c0a1d50 msgr2=0x7f0f4c0a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:49.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.903+0000 7f0f50b98640 1 -- 172.21.15.37:0/3677251224 shutdown_connections 2024-04-26T00:32:49.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.903+0000 7f0f50b98640 1 -- 172.21.15.37:0/3677251224 wait complete. 2024-04-26T00:32:49.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.903+0000 7f0f50b98640 10 monclient: build_initial_monmap 2024-04-26T00:32:49.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.903+0000 7f0f50b98640 10 monclient: monmap: 2024-04-26T00:32:49.911 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:49.911 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:32:49.911 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:32:49.911 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:32:49.911 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:49.911 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:32:49.911 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:32:49.911 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:32:49.911 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:32:49.911 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:49.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.903+0000 7f0f50b98640 1 Processor -- start 2024-04-26T00:32:49.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.903+0000 7f0f50b98640 1 -- start start 2024-04-26T00:32:49.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.903+0000 7f0f50b98640 10 monclient: init 2024-04-26T00:32:49.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f50b98640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:49.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f50b98640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:32:49.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f50b98640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f4c09fc50 0x7f0f4c1ccdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:49.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f50b98640 10 monclient(hunting): picked mon.noname-b con 0x7f0f4c09fc50 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:32:49.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f50b98640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c0a0300 0x7f0f4c131ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:49.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f50b98640 10 monclient(hunting): picked mon.noname-c con 0x7f0f4c0a0300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:49.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f50b98640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0f4c13a400 0x7f0f4c06ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:49.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f50b98640 10 monclient(hunting): picked mon.noname-a con 0x7f0f4c13a400 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:32:49.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f50b98640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:49.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f50b98640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f0f4c13f8e0 con 0x7f0f4c13a400 2024-04-26T00:32:49.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f50b98640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:49.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f50b98640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0f4c13f760 con 0x7f0f4c09fc50 2024-04-26T00:32:49.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f50b98640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:49.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f50b98640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0f4c13fa60 con 0x7f0f4c0a0300 2024-04-26T00:32:49.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f50b98640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:49.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f50b98640 10 monclient(hunting): authenticate will time out at 1751.216304s 2024-04-26T00:32:49.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f49d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c0a0300 0x7f0f4c131ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:49.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f4ad9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0f4c13a400 0x7f0f4c06ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:49.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f49d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c0a0300 0x7f0f4c131ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:47968/0 (socket says 172.21.15.37:47968) 2024-04-26T00:32:49.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f49d9b640 1 -- 172.21.15.37:0/1610460079 learned_addr learned my addr 172.21.15.37:0/1610460079 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:32:49.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f4ad9d640 10 monclient(hunting): get_auth_request con 0x7f0f4c13a400 auth_method 0 2024-04-26T00:32:49.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f4ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:49.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f4ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:49.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f4ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:49.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f49d9b640 10 monclient(hunting): get_auth_request con 0x7f0f4c0a0300 auth_method 0 2024-04-26T00:32:49.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f49d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:49.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f49d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:49.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f49d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:49.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f4ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:49.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f4ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:49.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f4ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:49.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f49d9b640 1 -- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c0a0300 msgr2=0x7f0f4c131ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:32:49.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f49d9b640 1 -- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c0a0300 msgr2=0x7f0f4c131ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:32:49.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f49d9b640 1 --2- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c0a0300 0x7f0f4c131ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:32:49.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f49d9b640 1 --2- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c0a0300 0x7f0f4c131ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:32:49.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f4a59c640 1 --2- 172.21.15.37:0/1610460079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f4c09fc50 0x7f0f4c1ccdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:49.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f4ad9d640 10 monclient(hunting): handle_auth_done global_id 144204 payload 824 2024-04-26T00:32:49.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f4ad9d640 1 -- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c0a0300 msgr2=0x7f0f4c131ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:32:49.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f4ad9d640 1 --2- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c0a0300 0x7f0f4c131ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:49.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f4ad9d640 1 -- 172.21.15.37:0/1610460079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f4c09fc50 msgr2=0x7f0f4c1ccdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:49.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f4ad9d640 1 --2- 172.21.15.37:0/1610460079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f4c09fc50 0x7f0f4c1ccdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:49.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f4ad9d640 10 monclient: _finish_hunting 0 2024-04-26T00:32:49.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f4ad9d640 1 monclient: found mon.noname-a 2024-04-26T00:32:49.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f4ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:49.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f4ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:32:49.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f4ad9d640 1 -- 172.21.15.37:0/1610460079 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f4c0707e0 con 0x7f0f4c13a400 2024-04-26T00:32:49.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f4a59c640 10 monclient: get_auth_request con 0x7f0f4c09fc50 auth_method 0 2024-04-26T00:32:49.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f4a59c640 1 --2- 172.21.15.37:0/1610460079 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f4c09fc50 0x7f0f4c1ccdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:32:49.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f4ad9d640 1 -- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0f4c13a400 msgr2=0x7f0f4c06ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:49.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f4ad9d640 1 --2- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0f4c13a400 0x7f0f4c06ebe0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0f44000ee0 tx=0x7f0f4400e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:49.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f3affd640 1 -- 172.21.15.37:0/1610460079 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f44017030 con 0x7f0f4c13a400 2024-04-26T00:32:49.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:49.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f3affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:32:49.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f3affd640 10 monclient: dump: 2024-04-26T00:32:49.916 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:49.916 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:49.916 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:49.916 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:49.916 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:49.916 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:49.916 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:49.916 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:49.916 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:49.916 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:49.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f3affd640 10 monclient: _finish_auth 0 2024-04-26T00:32:49.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f3affd640 10 monclient: _check_auth_tickets 2024-04-26T00:32:49.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:49.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f3affd640 1 -- 172.21.15.37:0/1610460079 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f44004020 con 0x7f0f4c13a400 2024-04-26T00:32:49.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f3affd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:49.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f3affd640 1 -- 172.21.15.37:0/1610460079 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f4400f040 con 0x7f0f4c13a400 2024-04-26T00:32:49.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:49.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f3affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:49.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f3affd640 10 monclient: dump: 2024-04-26T00:32:49.917 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:49.917 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:49.917 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:49.917 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:49.917 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:49.917 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:49.917 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:49.918 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:49.918 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:49.918 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:49.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f48d99640 4 set_mon_vals no callback set 2024-04-26T00:32:49.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f50b98640 5 monclient: authenticate success, global_id 144204 2024-04-26T00:32:49.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f50b98640 10 monclient: _renew_subs 2024-04-26T00:32:49.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f50b98640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:49.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f50b98640 1 -- 172.21.15.37:0/1610460079 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f4c11c160 con 0x7f0f4c13a400 2024-04-26T00:32:49.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f50b98640 10 monclient: _renew_subs 2024-04-26T00:32:49.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f50b98640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:49.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.907+0000 7f0f50b98640 1 -- 172.21.15.37:0/1610460079 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f4c1cdc50 con 0x7f0f4c13a400 2024-04-26T00:32:49.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.911+0000 7f0f50b98640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:49.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.911+0000 7f0f50b98640 1 --2- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c06f120 0x7f0f4c06f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:49.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.911+0000 7f0f50b98640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:49.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.911+0000 7f0f50b98640 1 -- 172.21.15.37:0/1610460079 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0f4c13ab00 con 0x7f0f4c06f120 2024-04-26T00:32:49.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.911+0000 7f0f3affd640 1 -- 172.21.15.37:0/1610460079 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f0f4401a070 con 0x7f0f4c13a400 2024-04-26T00:32:49.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.911+0000 7f0f4a59c640 1 --2- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c06f120 0x7f0f4c06f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:49.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.911+0000 7f0f4a59c640 10 monclient: get_auth_request con 0x7f0f4c06f120 auth_method 0 2024-04-26T00:32:49.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.911+0000 7f0f4a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:49.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.911+0000 7f0f4a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:49.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.911+0000 7f0f4a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:49.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.911+0000 7f0f4a59c640 1 -- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c06f120 msgr2=0x7f0f4c06f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:32:49.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.911+0000 7f0f4a59c640 1 -- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c06f120 msgr2=0x7f0f4c06f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:32:49.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.911+0000 7f0f4a59c640 1 --2- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c06f120 0x7f0f4c06f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:32:49.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.911+0000 7f0f4a59c640 1 --2- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c06f120 0x7f0f4c06f500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:32:49.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.911+0000 7f0f3affd640 1 --2- 172.21.15.37:0/1610460079 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0f2c035ed0 0x7f0f2c038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:49.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.911+0000 7f0f3affd640 1 -- 172.21.15.37:0/1610460079 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(498..498 src has 370..498) ==== 3943+0+0 (secure 0 0 0) 0x7f0f44048350 con 0x7f0f4c13a400 2024-04-26T00:32:49.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.911+0000 7f0f49d9b640 1 --2- 172.21.15.37:0/1610460079 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0f2c035ed0 0x7f0f2c038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:49.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.911+0000 7f0f49d9b640 10 monclient: get_auth_request con 0x7f0f2c035ed0 auth_method 0 2024-04-26T00:32:49.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:49.911+0000 7f0f49d9b640 1 --2- 172.21.15.37:0/1610460079 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0f2c035ed0 0x7f0f2c038390 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c0703e0 tx=0x7f0f400a7070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:50.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:50.115+0000 7f0f4a59c640 1 --2- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c06f120 0x7f0f4c06f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:50.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:50.115+0000 7f0f4a59c640 10 monclient: get_auth_request con 0x7f0f4c06f120 auth_method 0 2024-04-26T00:32:50.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:50.115+0000 7f0f4a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:50.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:50.115+0000 7f0f4a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:50.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:50.115+0000 7f0f4a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:32:50.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:50.115+0000 7f0f4a59c640 1 -- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c06f120 msgr2=0x7f0f4c06f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:32:50.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:50.115+0000 7f0f4a59c640 1 -- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c06f120 msgr2=0x7f0f4c06f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:32:50.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:50.115+0000 7f0f4a59c640 1 --2- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c06f120 0x7f0f4c06f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:32:50.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:50.115+0000 7f0f4a59c640 1 --2- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c06f120 0x7f0f4c06f500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:32:50.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:50.519+0000 7f0f4a59c640 1 --2- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c06f120 0x7f0f4c06f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:50.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:50.519+0000 7f0f4a59c640 10 monclient: get_auth_request con 0x7f0f4c06f120 auth_method 0 2024-04-26T00:32:50.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:50.519+0000 7f0f4a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:50.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:50.519+0000 7f0f4a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:50.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:50.519+0000 7f0f4a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:32:50.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:50.519+0000 7f0f4a59c640 1 -- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c06f120 msgr2=0x7f0f4c06f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:32:50.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:50.519+0000 7f0f4a59c640 1 -- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c06f120 msgr2=0x7f0f4c06f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:32:50.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:50.519+0000 7f0f4a59c640 1 --2- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c06f120 0x7f0f4c06f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:32:50.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:50.519+0000 7f0f4a59c640 1 --2- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c06f120 0x7f0f4c06f500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:32:50.561 INFO:tasks.rados.rados.0.smithi045.stdout:2211: done (2 left) 2024-04-26T00:32:50.562 INFO:tasks.rados.rados.0.smithi045.stdout:2212: snap_create 2024-04-26T00:32:50.562 INFO:tasks.rados.rados.0.smithi045.stdout:2210: finishing rollback tid 0 to smithi04526579-21 2024-04-26T00:32:50.562 INFO:tasks.rados.rados.0.smithi045.stdout:2210: finishing rollback tid 1 to smithi04526579-21 2024-04-26T00:32:50.562 INFO:tasks.rados.rados.0.smithi045.stdout:2210: finishing rollback tid 2 to smithi04526579-21 2024-04-26T00:32:50.562 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 1096 (ObjNum 756 snap 232 seq_num 756) dirty exists 2024-04-26T00:32:50.562 INFO:tasks.rados.rados.0.smithi045.stdout:2209: expect (ObjNum 703 snap 221 seq_num 703) 2024-04-26T00:32:51.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:51.323+0000 7f0f4a59c640 1 --2- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c06f120 0x7f0f4c06f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:51.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:51.323+0000 7f0f4a59c640 10 monclient: get_auth_request con 0x7f0f4c06f120 auth_method 0 2024-04-26T00:32:51.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:51.323+0000 7f0f4a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:51.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:51.323+0000 7f0f4a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:51.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:51.323+0000 7f0f4a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:32:51.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:51.323+0000 7f0f4a59c640 1 -- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c06f120 msgr2=0x7f0f4c06f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:32:51.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:51.323+0000 7f0f4a59c640 1 -- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c06f120 msgr2=0x7f0f4c06f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:32:51.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:51.323+0000 7f0f4a59c640 1 --2- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c06f120 0x7f0f4c06f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:32:51.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:51.323+0000 7f0f4a59c640 1 --2- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c06f120 0x7f0f4c06f500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:32:51.572 INFO:tasks.rados.rados.0.smithi045.stdout:2209: done (2 left) 2024-04-26T00:32:51.573 INFO:tasks.rados.rados.0.smithi045.stdout:2210: done (1 left) 2024-04-26T00:32:51.573 INFO:tasks.rados.rados.0.smithi045.stdout:2212: done (0 left) 2024-04-26T00:32:51.573 INFO:tasks.rados.rados.0.smithi045.stdout:2213: write oid 49 current snap is 247 2024-04-26T00:32:51.573 INFO:tasks.rados.rados.0.smithi045.stdout:2213: seq_num 804 ranges {712828=717261,1915052=718936,3212792=300036} 2024-04-26T00:32:51.596 INFO:tasks.rados.rados.0.smithi045.stdout:2213: writing smithi04526579-49 from 712828 to 1430089 tid 1 2024-04-26T00:32:51.616 INFO:tasks.rados.rados.0.smithi045.stdout:2213: writing smithi04526579-49 from 1915052 to 2633988 tid 2 2024-04-26T00:32:51.629 INFO:tasks.rados.rados.0.smithi045.stdout:2213: writing smithi04526579-49 from 3212792 to 3512828 tid 3 2024-04-26T00:32:51.629 INFO:tasks.rados.rados.0.smithi045.stdout:2214: snap_remove snap 226 2024-04-26T00:32:52.580 INFO:tasks.rados.rados.0.smithi045.stdout:2214: done (1 left) 2024-04-26T00:32:52.580 INFO:tasks.rados.rados.0.smithi045.stdout:2215: read oid 24 snap -1 2024-04-26T00:32:52.580 INFO:tasks.rados.rados.0.smithi045.stdout:2215: expect deleted 2024-04-26T00:32:52.580 INFO:tasks.rados.rados.0.smithi045.stdout:2213: finishing write tid 1 to smithi04526579-49 2024-04-26T00:32:52.580 INFO:tasks.rados.rados.0.smithi045.stdout:2213: finishing write tid 2 to smithi04526579-49 2024-04-26T00:32:52.580 INFO:tasks.rados.rados.0.smithi045.stdout:2213: finishing write tid 3 to smithi04526579-49 2024-04-26T00:32:52.580 INFO:tasks.rados.rados.0.smithi045.stdout:2213: finishing write tid 4 to smithi04526579-49 2024-04-26T00:32:52.580 INFO:tasks.rados.rados.0.smithi045.stdout:2213: finishing write tid 5 to smithi04526579-49 2024-04-26T00:32:52.580 INFO:tasks.rados.rados.0.smithi045.stdout:2213: oid 49 updating version 0 to 1352 2024-04-26T00:32:52.581 INFO:tasks.rados.rados.0.smithi045.stdout:2213: oid 49 version 1352 is already newer than 1351 2024-04-26T00:32:52.581 INFO:tasks.rados.rados.0.smithi045.stdout:2213: oid 49 version 1352 is already newer than 1349 2024-04-26T00:32:52.581 INFO:tasks.rados.rados.0.smithi045.stdout:2213: oid 49 version 1352 is already newer than 1350 2024-04-26T00:32:52.581 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 1352 (ObjNum 804 snap 247 seq_num 804) dirty exists 2024-04-26T00:32:52.581 INFO:tasks.rados.rados.0.smithi045.stdout:2213: left oid 49 (ObjNum 804 snap 247 seq_num 804) 2024-04-26T00:32:52.581 INFO:tasks.rados.rados.0.smithi045.stdout:2213: done (1 left) 2024-04-26T00:32:52.581 INFO:tasks.rados.rados.0.smithi045.stdout:2216: snap_remove snap 242 2024-04-26T00:32:52.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.907+0000 7f0f39ffb640 10 monclient: tick 2024-04-26T00:32:52.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.907+0000 7f0f39ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:32:52.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.907+0000 7f0f39ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:52.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.907+0000 7f0f39ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:52.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.927+0000 7f0f4a59c640 1 --2- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c06f120 0x7f0f4c06f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:52.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.927+0000 7f0f4a59c640 10 monclient: get_auth_request con 0x7f0f4c06f120 auth_method 0 2024-04-26T00:32:52.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.927+0000 7f0f4a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:52.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.927+0000 7f0f4a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:52.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.927+0000 7f0f4a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:32:52.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.927+0000 7f0f4a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:52.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.927+0000 7f0f4a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:52.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.927+0000 7f0f4a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:52.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.927+0000 7f0f4a59c640 10 monclient(hunting): handle_auth_done global_id 154100 payload 824 2024-04-26T00:32:52.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.927+0000 7f0f4a59c640 1 -- 172.21.15.37:0/1610460079 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0f4c09f550 con 0x7f0f4c06f120 2024-04-26T00:32:52.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.927+0000 7f0f4a59c640 1 -- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c06f120 msgr2=0x7f0f4c06f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:52.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.927+0000 7f0f4a59c640 1 --2- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c06f120 0x7f0f4c06f500 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c0158f0 tx=0x7f0f3c015db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:52.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.931+0000 7f0f3affd640 1 -- 172.21.15.37:0/1610460079 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c013e90 con 0x7f0f4c06f120 2024-04-26T00:32:52.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.931+0000 7f0f3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:52.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.931+0000 7f0f3affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:32:52.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.931+0000 7f0f3affd640 10 monclient: dump: 2024-04-26T00:32:52.926 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:52.926 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:52.927 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:52.927 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:52.927 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:52.927 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:52.927 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:52.927 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:52.927 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:52.927 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:52.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.931+0000 7f0f3affd640 1 -- 172.21.15.37:0/1610460079 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0f3c005490 con 0x7f0f4c06f120 2024-04-26T00:32:52.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.931+0000 7f0f3affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:52.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.931+0000 7f0f3affd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:32:52.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.931+0000 7f0f3affd640 1 -- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c06f120 msgr2=0x7f0f4c06f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:52.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.931+0000 7f0f3affd640 1 --2- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c06f120 0x7f0f4c06f500 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c0158f0 tx=0x7f0f3c015db0 comp rx=0 tx=0).stop 2024-04-26T00:32:52.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.931+0000 7f0f3affd640 1 -- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c06f120 msgr2=0x7f0f4c06f500 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:32:52.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.931+0000 7f0f3affd640 1 --2- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c06f120 0x7f0f4c06f500 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c0158f0 tx=0x7f0f3c015db0 comp rx=0 tx=0).stop 2024-04-26T00:32:52.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.939+0000 7f0f50b98640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:32:52.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.939+0000 7f0f50b98640 1 --2- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c135c90 0x7f0f4c136070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:52.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.939+0000 7f0f50b98640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:52.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.939+0000 7f0f50b98640 1 -- 172.21.15.37:0/1610460079 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0f4c06fa40 con 0x7f0f4c135c90 2024-04-26T00:32:52.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.939+0000 7f0f4a59c640 1 --2- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c135c90 0x7f0f4c136070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:52.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.939+0000 7f0f4a59c640 10 monclient: get_auth_request con 0x7f0f4c135c90 auth_method 0 2024-04-26T00:32:52.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.939+0000 7f0f4a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:52.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.939+0000 7f0f4a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:52.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.939+0000 7f0f4a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:52.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.939+0000 7f0f4a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:52.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.939+0000 7f0f4a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:52.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.939+0000 7f0f4a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:52.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.939+0000 7f0f4a59c640 10 monclient(hunting): handle_auth_done global_id 154103 payload 824 2024-04-26T00:32:52.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.939+0000 7f0f4a59c640 1 -- 172.21.15.37:0/1610460079 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0f4c11bba0 con 0x7f0f4c135c90 2024-04-26T00:32:52.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.943+0000 7f0f4a59c640 1 -- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c135c90 msgr2=0x7f0f4c136070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:52.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.943+0000 7f0f4a59c640 1 --2- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c135c90 0x7f0f4c136070 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c0158f0 tx=0x7f0f3c003770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:52.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.943+0000 7f0f3affd640 1 -- 172.21.15.37:0/1610460079 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c0069f0 con 0x7f0f4c135c90 2024-04-26T00:32:52.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.943+0000 7f0f3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:52.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.943+0000 7f0f3affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:32:52.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.943+0000 7f0f3affd640 10 monclient: dump: 2024-04-26T00:32:52.941 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:52.941 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:52.941 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:52.941 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:52.941 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:52.941 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:52.941 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:52.941 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:52.941 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:52.941 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:52.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.943+0000 7f0f3affd640 1 -- 172.21.15.37:0/1610460079 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7f0f3c0069f0 con 0x7f0f4c135c90 2024-04-26T00:32:52.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.943+0000 7f0f3affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:32:52.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.943+0000 7f0f3affd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:32:52.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.943+0000 7f0f3affd640 1 -- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c135c90 msgr2=0x7f0f4c136070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:52.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.943+0000 7f0f3affd640 1 --2- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c135c90 0x7f0f4c136070 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c0158f0 tx=0x7f0f3c003770 comp rx=0 tx=0).stop 2024-04-26T00:32:52.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.943+0000 7f0f3affd640 1 -- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c135c90 msgr2=0x7f0f4c136070 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:32:52.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.943+0000 7f0f3affd640 1 --2- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f4c135c90 0x7f0f4c136070 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c0158f0 tx=0x7f0f3c003770 comp rx=0 tx=0).stop 2024-04-26T00:32:52.943 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:32:52.943 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:32:52.943 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:32:52.943 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:32:52.943 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 96, 2024-04-26T00:32:52.943 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 10463, 2024-04-26T00:32:52.943 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:32:52.943 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:32:52.943 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:32:52.943 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:32:52.944 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:52.944 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 5, 2024-04-26T00:32:52.944 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:32:52.944 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:32:52.944 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:32:52.944 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:52.944 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:52.944 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:52.944 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:52.944 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:52.944 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:52.944 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:52.944 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:52.944 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:52.944 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:52.945 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:52.945 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:52.945 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:32:52.945 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:32:52.945 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:52.945 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:52.945 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:52.945 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:52.945 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:52.945 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:52.945 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:52.945 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:52.945 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:52.946 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:52.946 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:52.946 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:52.946 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:52.946 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:32:52.946 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:32:52.946 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:32:52.946 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:32:52.946 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:32:52.946 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:32:52.946 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:32:52.946 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:32:52.946 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:32:52.946 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:32:52.946 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:32:52.947 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:32:52.947 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:32:52.947 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:32:52.947 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:32:52.947 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:32:52.947 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:32:52.947 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:52.947 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:52.947 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:52.947 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:52.947 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:52.947 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:52.947 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:52.947 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:52.947 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:52.948 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:52.948 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:52.948 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:52.948 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:32:52.948 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:52.948 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:32:52.948 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:52.948 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:32:52.948 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:32:52.948 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:52.948 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:52.948 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:52.948 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:52.948 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:32:52.948 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:52.949 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:52.949 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:52.949 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:52.949 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:32:52.949 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:52.949 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:52.949 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:52.949 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:52.949 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:32:52.949 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:32:52.949 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:52.949 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:52.949 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:52.949 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:52.950 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:52.950 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:32:52.950 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:32:52.950 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:52.950 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:52.950 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:52.950 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:52.950 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:32:52.950 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:52.950 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:52.950 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:52.950 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:52.950 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:32:52.950 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:52.951 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:52.951 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:52.951 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:52.951 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:32:52.951 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:32:52.951 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:52.951 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:52.951 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:52.951 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:52.951 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:52.951 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:32:52.951 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:32:52.951 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:52.951 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:52.952 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:52.952 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:52.952 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:32:52.952 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:52.952 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:52.952 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:52.952 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:52.952 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:32:52.952 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:52.952 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:52.952 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:52.952 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:52.952 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:32:52.952 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:32:52.952 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:52.953 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:52.953 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:52.953 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:52.953 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:52.953 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:52.953 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:32:52.953 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:32:52.953 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:52.953 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:52.953 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:52.953 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:32:52.953 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:52.953 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:52.953 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:32:52.953 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:52.954 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:52.954 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:52.954 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:32:52.954 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:52.954 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:52.954 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:52.954 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:32:52.954 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:32:52.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.943+0000 7f0f50b98640 1 -- 172.21.15.37:0/1610460079 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0f2c035ed0 msgr2=0x7f0f2c038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:52.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.943+0000 7f0f50b98640 1 --2- 172.21.15.37:0/1610460079 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0f2c035ed0 0x7f0f2c038390 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c0703e0 tx=0x7f0f400a7070 comp rx=0 tx=0).stop 2024-04-26T00:32:52.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.943+0000 7f0f50b98640 10 monclient: shutdown 2024-04-26T00:32:52.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.943+0000 7f0f50b98640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:52.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.943+0000 7f0f50b98640 1 -- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0f4c13a400 msgr2=0x7f0f4c06ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:52.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.943+0000 7f0f50b98640 1 --2- 172.21.15.37:0/1610460079 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0f4c13a400 0x7f0f4c06ebe0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0f44000ee0 tx=0x7f0f4400e990 comp rx=0 tx=0).stop 2024-04-26T00:32:52.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.943+0000 7f0f4ad9d640 1 -- 172.21.15.37:0/1610460079 reap_dead start 2024-04-26T00:32:52.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.943+0000 7f0f4ad9d640 1 -- 172.21.15.37:0/1610460079 reap_dead start 2024-04-26T00:32:52.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.943+0000 7f0f50b98640 1 -- 172.21.15.37:0/1610460079 shutdown_connections 2024-04-26T00:32:52.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.943+0000 7f0f50b98640 1 -- 172.21.15.37:0/1610460079 >> 172.21.15.37:0/1610460079 conn(0x7f0f4c0a1d50 msgr2=0x7f0f4c134c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:52.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.943+0000 7f0f50b98640 1 -- 172.21.15.37:0/1610460079 shutdown_connections 2024-04-26T00:32:52.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:52.943+0000 7f0f50b98640 1 -- 172.21.15.37:0/1610460079 wait complete. 2024-04-26T00:32:52.965 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:32:52.965 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:32:52.965 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:32:53.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.099+0000 7f20cf59e640 10 monclient: get_monmap_and_config 2024-04-26T00:32:53.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.099+0000 7f20cf59e640 10 monclient: build_initial_monmap 2024-04-26T00:32:53.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.099+0000 7f20cf59e640 10 monclient: monmap: 2024-04-26T00:32:53.096 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:53.096 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:53.096 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:32:53.105003+0000 2024-04-26T00:32:53.096 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:32:53.105003+0000 2024-04-26T00:32:53.096 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:53.096 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:53.096 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:32:53.096 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:32:53.096 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:32:53.096 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:32:53.096 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:32:53.096 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:53.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.099+0000 7f20cf59e640 1 Processor -- start 2024-04-26T00:32:53.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.099+0000 7f20cf59e640 1 -- start start 2024-04-26T00:32:53.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.099+0000 7f20cf59e640 10 monclient: init 2024-04-26T00:32:53.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.099+0000 7f20cf59e640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:53.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.099+0000 7f20cf59e640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-04-26T00:32:53.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.099+0000 7f20cf59e640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f20c80a3d90 0x7f20c809bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:53.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.099+0000 7f20cf59e640 10 monclient(hunting): picked mon.noname-a con 0x7f20c80a3d90 addr v2:172.21.15.37:3300/0 2024-04-26T00:32:53.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.099+0000 7f20cf59e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f20c80a4170 0x7f20c80a45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:53.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.099+0000 7f20cf59e640 10 monclient(hunting): picked mon.noname-c con 0x7f20c80a4170 addr v2:172.21.15.45:3300/0 2024-04-26T00:32:53.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.099+0000 7f20cf59e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f20c8133b20 addr v1:172.21.15.37:6789/0 2024-04-26T00:32:53.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.099+0000 7f20cf59e640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20c80a5150 con 0x7f20c8133b20 2024-04-26T00:32:53.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.099+0000 7f20cf59e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:53.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.099+0000 7f20cf59e640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f20c809b280 con 0x7f20c80a3d90 2024-04-26T00:32:53.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.099+0000 7f20cf59e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:53.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.099+0000 7f20cf59e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f20c809b560 con 0x7f20c80a4170 2024-04-26T00:32:53.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.099+0000 7f20cf59e640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:53.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.099+0000 7f20cf59e640 10 monclient(hunting): authenticate will time out at 1754.411512s 2024-04-26T00:32:53.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.099+0000 7f20cd33a640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f20c80a3d90 0x7f20c809bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:53.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.099+0000 7f20cdb3b640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f20c8133b20 0x7f20c8133f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:48928/0 (socket says 172.21.15.37:48928) 2024-04-26T00:32:53.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.099+0000 7f20cdb3b640 1 -- 172.21.15.37:0/2584725335 learned_addr learned my addr 172.21.15.37:0/2584725335 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:32:53.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.099+0000 7f20cd33a640 10 monclient(hunting): get_auth_request con 0x7f20c80a3d90 auth_method 0 2024-04-26T00:32:53.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.099+0000 7f20cd33a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:53.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.099+0000 7f20cd33a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:53.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.099+0000 7f20cd33a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:53.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.099+0000 7f20cd33a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:53.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.099+0000 7f20cd33a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:53.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.099+0000 7f20cd33a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:53.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.099+0000 7f20ccb39640 1 --2- 172.21.15.37:0/2584725335 >> v2:172.21.15.45:3300/0 conn(0x7f20c80a4170 0x7f20c80a45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:53.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.099+0000 7f20ccb39640 10 monclient(hunting): get_auth_request con 0x7f20c80a4170 auth_method 0 2024-04-26T00:32:53.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.099+0000 7f20ccb39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:53.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.099+0000 7f20ccb39640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:53.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.099+0000 7f20ccb39640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:53.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.099+0000 7f20cd33a640 10 monclient(hunting): handle_auth_done global_id 144207 payload 274 2024-04-26T00:32:53.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.099+0000 7f20cd33a640 1 -- 172.21.15.37:0/2584725335 >> v2:172.21.15.45:3300/0 conn(0x7f20c80a4170 msgr2=0x7f20c80a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:53.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.103+0000 7f20cd33a640 1 --2- 172.21.15.37:0/2584725335 >> v2:172.21.15.45:3300/0 conn(0x7f20c80a4170 0x7f20c80a45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:53.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.103+0000 7f20cd33a640 1 -- 172.21.15.37:0/2584725335 >> v1:172.21.15.37:6789/0 conn(0x7f20c8133b20 legacy=0x7f20c8133f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:53.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.103+0000 7f20cd33a640 10 monclient: _finish_hunting 0 2024-04-26T00:32:53.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.103+0000 7f20cd33a640 1 monclient: found mon.noname-a 2024-04-26T00:32:53.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.103+0000 7f20cd33a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:53.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.103+0000 7f20cd33a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:32:53.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.103+0000 7f20cd33a640 1 -- 172.21.15.37:0/2584725335 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20c80a4b20 con 0x7f20c80a3d90 2024-04-26T00:32:53.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cd33a640 1 -- 172.21.15.37:0/2584725335 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20c80a3d90 msgr2=0x7f20c809bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:53.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cd33a640 1 --2- 172.21.15.37:0/2584725335 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20c80a3d90 0x7f20c809bf40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f20c809d140 tx=0x7f20c401c300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=387e48b6f3b76725 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:53.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20bffff640 1 -- 172.21.15.37:0/2584725335 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c40b0070 con 0x7f20c80a3d90 2024-04-26T00:32:53.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:53.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20bffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:32:53.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20bffff640 10 monclient: dump: 2024-04-26T00:32:53.103 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:53.103 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:53.103 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:53.103 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:53.103 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:53.103 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:53.103 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:53.103 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:53.103 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:53.103 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:53.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20bffff640 10 monclient: _finish_auth 0 2024-04-26T00:32:53.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20bffff640 10 monclient: _check_auth_tickets 2024-04-26T00:32:53.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:53.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20bffff640 1 -- 172.21.15.37:0/2584725335 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20c40ab600 con 0x7f20c80a3d90 2024-04-26T00:32:53.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20bffff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:53.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20bffff640 1 -- 172.21.15.37:0/2584725335 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c40ab9d0 con 0x7f20c80a3d90 2024-04-26T00:32:53.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cf59e640 5 monclient: authenticate success, global_id 144207 2024-04-26T00:32:53.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cf59e640 10 monclient: get_monmap_and_config success 2024-04-26T00:32:53.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cf59e640 4 set_mon_vals no callback set 2024-04-26T00:32:53.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:53.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20bffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:53.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20bffff640 10 monclient: dump: 2024-04-26T00:32:53.105 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:53.106 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:53.106 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:53.106 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:53.106 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:53.106 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:53.106 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:53.106 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:53.106 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:53.106 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:53.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cf59e640 10 monclient: shutdown 2024-04-26T00:32:53.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cf59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:53.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cf59e640 1 -- 172.21.15.37:0/2584725335 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20c80a3d90 msgr2=0x7f20c809bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:53.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cf59e640 1 --2- 172.21.15.37:0/2584725335 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20c80a3d90 0x7f20c809bf40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f20c809d140 tx=0x7f20c401c300 comp rx=0 tx=0).stop 2024-04-26T00:32:53.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cf59e640 1 -- 172.21.15.37:0/2584725335 shutdown_connections 2024-04-26T00:32:53.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cf59e640 1 --2- 172.21.15.37:0/2584725335 >> v2:172.21.15.45:3300/0 conn(0x7f20c80a4170 0x7f20c80a45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:53.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cf59e640 1 --2- 172.21.15.37:0/2584725335 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20c80a3d90 0x7f20c809bf40 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:53.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cf59e640 1 -- 172.21.15.37:0/2584725335 >> 172.21.15.37:0/2584725335 conn(0x7f20c80aa0a0 msgr2=0x7f20c80a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:53.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cf59e640 1 -- 172.21.15.37:0/2584725335 shutdown_connections 2024-04-26T00:32:53.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cf59e640 1 -- 172.21.15.37:0/2584725335 wait complete. 2024-04-26T00:32:53.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cf59e640 10 monclient: build_initial_monmap 2024-04-26T00:32:53.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cf59e640 10 monclient: monmap: 2024-04-26T00:32:53.117 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:53.118 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:32:53.118 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:32:53.118 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:32:53.118 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:53.118 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:32:53.118 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:32:53.118 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:32:53.118 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:32:53.118 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:53.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cf59e640 1 Processor -- start 2024-04-26T00:32:53.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cf59e640 1 -- start start 2024-04-26T00:32:53.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cf59e640 10 monclient: init 2024-04-26T00:32:53.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cf59e640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:53.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cf59e640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:32:53.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cf59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20c80a3d90 0x7f20c81cf530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:53.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cf59e640 10 monclient(hunting): picked mon.noname-b con 0x7f20c80a3d90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:32:53.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cf59e640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20c80a4170 0x7f20c813e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:53.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cf59e640 10 monclient(hunting): picked mon.noname-a con 0x7f20c80a4170 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:32:53.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cf59e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f20c8133b20 0x7f20c813e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:53.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cf59e640 10 monclient(hunting): picked mon.noname-c con 0x7f20c8133b20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:53.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cf59e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:53.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cf59e640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f20c8141e30 con 0x7f20c80a4170 2024-04-26T00:32:53.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cf59e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:53.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cf59e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f20c809c690 con 0x7f20c80a3d90 2024-04-26T00:32:53.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cf59e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:53.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cf59e640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f20c8141fb0 con 0x7f20c8133b20 2024-04-26T00:32:53.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cf59e640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:53.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cf59e640 10 monclient(hunting): authenticate will time out at 1754.418685s 2024-04-26T00:32:53.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20ccb39640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20c80a4170 0x7f20c813e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:53.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cdb3b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f20c8133b20 0x7f20c813e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:53.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20ccb39640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20c80a4170 0x7f20c813e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44678/0 (socket says 172.21.15.37:44678) 2024-04-26T00:32:53.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20ccb39640 1 -- 172.21.15.37:0/1867282667 learned_addr learned my addr 172.21.15.37:0/1867282667 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:32:53.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20ccb39640 10 monclient(hunting): get_auth_request con 0x7f20c80a4170 auth_method 0 2024-04-26T00:32:53.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20ccb39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:53.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20ccb39640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:53.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20ccb39640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:53.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cdb3b640 10 monclient(hunting): get_auth_request con 0x7f20c8133b20 auth_method 0 2024-04-26T00:32:53.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cdb3b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:53.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cdb3b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:53.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cdb3b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:53.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20ccb39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:53.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20ccb39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:53.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20ccb39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:53.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cdb3b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:53.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cdb3b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:53.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cdb3b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:53.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cd33a640 1 --2- 172.21.15.37:0/1867282667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20c80a3d90 0x7f20c81cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:53.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cd33a640 10 monclient(hunting): get_auth_request con 0x7f20c80a3d90 auth_method 0 2024-04-26T00:32:53.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cd33a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:53.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cd33a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:53.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cd33a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:53.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cdb3b640 10 monclient(hunting): handle_auth_done global_id 154106 payload 824 2024-04-26T00:32:53.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cdb3b640 1 -- 172.21.15.37:0/1867282667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20c80a3d90 msgr2=0x7f20c81cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:53.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cdb3b640 1 --2- 172.21.15.37:0/1867282667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20c80a3d90 0x7f20c81cf530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:53.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cdb3b640 1 -- 172.21.15.37:0/1867282667 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20c80a4170 msgr2=0x7f20c813e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:53.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cdb3b640 1 --2- 172.21.15.37:0/1867282667 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20c80a4170 0x7f20c813e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:53.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cdb3b640 10 monclient: _finish_hunting 0 2024-04-26T00:32:53.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cdb3b640 1 monclient: found mon.noname-c 2024-04-26T00:32:53.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cdb3b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:53.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cdb3b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:32:53.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20cdb3b640 1 -- 172.21.15.37:0/1867282667 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20c813ccf0 con 0x7f20c8133b20 2024-04-26T00:32:53.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.107+0000 7f20ccb39640 1 --2- 172.21.15.37:0/1867282667 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20c80a4170 0x7f20c813e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:32:53.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.111+0000 7f20cdb3b640 1 -- 172.21.15.37:0/1867282667 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f20c8133b20 msgr2=0x7f20c813e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:53.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.111+0000 7f20cdb3b640 1 --2- 172.21.15.37:0/1867282667 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f20c8133b20 0x7f20c813e760 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f20c000caf0 tx=0x7f20c000cef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:53.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.111+0000 7f20bdffb640 1 -- 172.21.15.37:0/1867282667 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c000ade0 con 0x7f20c8133b20 2024-04-26T00:32:53.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.111+0000 7f20bdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:53.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.111+0000 7f20bdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:32:53.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.111+0000 7f20bdffb640 10 monclient: dump: 2024-04-26T00:32:53.123 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:53.123 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:53.123 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:53.123 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:53.123 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:53.123 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:53.123 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:53.124 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:53.124 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:53.124 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:53.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.111+0000 7f20bdffb640 10 monclient: _finish_auth 0 2024-04-26T00:32:53.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.111+0000 7f20bdffb640 10 monclient: _check_auth_tickets 2024-04-26T00:32:53.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.111+0000 7f20bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:53.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.111+0000 7f20bdffb640 1 -- 172.21.15.37:0/1867282667 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20c0012400 con 0x7f20c8133b20 2024-04-26T00:32:53.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.111+0000 7f20bdffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:53.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.111+0000 7f20bdffb640 1 -- 172.21.15.37:0/1867282667 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c00127f0 con 0x7f20c8133b20 2024-04-26T00:32:53.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.111+0000 7f20bdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:53.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.111+0000 7f20bdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:32:53.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.111+0000 7f20bdffb640 10 monclient: dump: 2024-04-26T00:32:53.125 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:53.125 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:53.125 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:53.125 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:53.125 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:53.125 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:53.125 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:53.125 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:53.125 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:53.125 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:53.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.111+0000 7f20bf7fe640 4 set_mon_vals no callback set 2024-04-26T00:32:53.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.111+0000 7f20cf59e640 5 monclient: authenticate success, global_id 154106 2024-04-26T00:32:53.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.111+0000 7f20cf59e640 10 monclient: _renew_subs 2024-04-26T00:32:53.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.111+0000 7f20cf59e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:32:53.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.111+0000 7f20cf59e640 1 -- 172.21.15.37:0/1867282667 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20c809ada0 con 0x7f20c8133b20 2024-04-26T00:32:53.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.111+0000 7f20cf59e640 10 monclient: _renew_subs 2024-04-26T00:32:53.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.111+0000 7f20cf59e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:32:53.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.111+0000 7f20cf59e640 1 -- 172.21.15.37:0/1867282667 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f20c813ca10 con 0x7f20c8133b20 2024-04-26T00:32:53.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.111+0000 7f20cf59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:53.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.111+0000 7f20cf59e640 1 --2- 172.21.15.37:0/1867282667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20c8138d60 0x7f20c8139140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:53.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.111+0000 7f20cf59e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:53.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.111+0000 7f20cf59e640 1 -- 172.21.15.37:0/1867282667 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f20c804f510 con 0x7f20c8138d60 2024-04-26T00:32:53.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.111+0000 7f20bdffb640 1 -- 172.21.15.37:0/1867282667 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f20c0012990 con 0x7f20c8133b20 2024-04-26T00:32:53.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.111+0000 7f20cd33a640 1 --2- 172.21.15.37:0/1867282667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20c8138d60 0x7f20c8139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:53.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.111+0000 7f20cd33a640 10 monclient: get_auth_request con 0x7f20c8138d60 auth_method 0 2024-04-26T00:32:53.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.111+0000 7f20cd33a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:53.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.111+0000 7f20cd33a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:53.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.111+0000 7f20cd33a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:53.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.111+0000 7f20bdffb640 1 --2- 172.21.15.37:0/1867282667 >> v2:172.21.15.45:6812/2379918623 conn(0x7f20ac035ed0 0x7f20ac038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:53.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.111+0000 7f20bdffb640 1 -- 172.21.15.37:0/1867282667 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(501..501 src has 370..501) ==== 3959+0+0 (secure 0 0 0) 0x7f20c0049040 con 0x7f20c8133b20 2024-04-26T00:32:53.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.111+0000 7f20cd33a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:53.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.111+0000 7f20cd33a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:53.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.111+0000 7f20cd33a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:53.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.111+0000 7f20ccb39640 1 --2- 172.21.15.37:0/1867282667 >> v2:172.21.15.45:6812/2379918623 conn(0x7f20ac035ed0 0x7f20ac038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:53.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.111+0000 7f20ccb39640 10 monclient: get_auth_request con 0x7f20ac035ed0 auth_method 0 2024-04-26T00:32:53.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.115+0000 7f20cd33a640 10 monclient(hunting): handle_auth_done global_id 134278 payload 824 2024-04-26T00:32:53.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.115+0000 7f20cd33a640 1 -- 172.21.15.37:0/1867282667 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f20c813b2e0 con 0x7f20c8138d60 2024-04-26T00:32:53.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.115+0000 7f20cd33a640 1 -- 172.21.15.37:0/1867282667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20c8138d60 msgr2=0x7f20c8139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:53.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.115+0000 7f20cd33a640 1 --2- 172.21.15.37:0/1867282667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20c8138d60 0x7f20c8139140 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f20c809a9e0 tx=0x7f20c401ab40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:53.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.115+0000 7f20ccb39640 1 --2- 172.21.15.37:0/1867282667 >> v2:172.21.15.45:6812/2379918623 conn(0x7f20ac035ed0 0x7f20ac038390 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f20c813a600 tx=0x7f20b8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:53.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.115+0000 7f20bdffb640 1 -- 172.21.15.37:0/1867282667 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c40b0050 con 0x7f20c8138d60 2024-04-26T00:32:53.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.115+0000 7f20bdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:53.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.115+0000 7f20bdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:32:53.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.115+0000 7f20bdffb640 10 monclient: dump: 2024-04-26T00:32:53.128 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:53.128 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:53.128 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:53.128 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:53.128 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:53.128 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:53.128 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:53.128 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:53.128 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:53.129 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:53.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.115+0000 7f20bdffb640 1 -- 172.21.15.37:0/1867282667 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f20c40b1040 con 0x7f20c8138d60 2024-04-26T00:32:53.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.115+0000 7f20bdffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:53.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.115+0000 7f20bdffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:32:53.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.115+0000 7f20bdffb640 1 -- 172.21.15.37:0/1867282667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20c8138d60 msgr2=0x7f20c8139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:53.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.115+0000 7f20bdffb640 1 --2- 172.21.15.37:0/1867282667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20c8138d60 0x7f20c8139140 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f20c809a9e0 tx=0x7f20c401ab40 comp rx=0 tx=0).stop 2024-04-26T00:32:53.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.115+0000 7f20bdffb640 1 -- 172.21.15.37:0/1867282667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20c8138d60 msgr2=0x7f20c8139140 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:32:53.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.115+0000 7f20bdffb640 1 --2- 172.21.15.37:0/1867282667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20c8138d60 0x7f20c8139140 secure :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f20c809a9e0 tx=0x7f20c401ab40 comp rx=0 tx=0).stop 2024-04-26T00:32:53.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.123+0000 7f20cf59e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:32:53.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.123+0000 7f20cf59e640 1 --2- 172.21.15.37:0/1867282667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20c8139680 0x7f20c8134b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:53.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.123+0000 7f20cf59e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:53.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.123+0000 7f20cf59e640 1 -- 172.21.15.37:0/1867282667 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f20c8075dc0 con 0x7f20c8139680 2024-04-26T00:32:53.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.123+0000 7f20cd33a640 1 --2- 172.21.15.37:0/1867282667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20c8139680 0x7f20c8134b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:53.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.123+0000 7f20cd33a640 10 monclient: get_auth_request con 0x7f20c8139680 auth_method 0 2024-04-26T00:32:53.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.123+0000 7f20cd33a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:53.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.123+0000 7f20cd33a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:53.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.123+0000 7f20cd33a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:53.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.123+0000 7f20cd33a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:53.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.123+0000 7f20cd33a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:53.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.123+0000 7f20cd33a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:53.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.123+0000 7f20cd33a640 10 monclient(hunting): handle_auth_done global_id 134281 payload 824 2024-04-26T00:32:53.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.123+0000 7f20cd33a640 1 -- 172.21.15.37:0/1867282667 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f20c8139a90 con 0x7f20c8139680 2024-04-26T00:32:53.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.127+0000 7f20cd33a640 1 -- 172.21.15.37:0/1867282667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20c8139680 msgr2=0x7f20c8134b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:53.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.127+0000 7f20cd33a640 1 --2- 172.21.15.37:0/1867282667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20c8139680 0x7f20c8134b30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f20c401c340 tx=0x7f20c40190a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:53.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.127+0000 7f20bdffb640 1 -- 172.21.15.37:0/1867282667 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c4003820 con 0x7f20c8139680 2024-04-26T00:32:53.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.127+0000 7f20bdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:53.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.127+0000 7f20bdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:32:53.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.127+0000 7f20bdffb640 10 monclient: dump: 2024-04-26T00:32:53.133 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:53.133 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:53.133 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:53.133 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:53.133 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:53.133 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:53.133 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:53.133 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:53.133 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:53.133 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:53.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.127+0000 7f20bdffb640 1 -- 172.21.15.37:0/1867282667 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f20c40c0e40 con 0x7f20c8139680 2024-04-26T00:32:53.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.127+0000 7f20bdffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:32:53.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.127+0000 7f20bdffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:32:53.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.127+0000 7f20bdffb640 1 -- 172.21.15.37:0/1867282667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20c8139680 msgr2=0x7f20c8134b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:53.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.127+0000 7f20bdffb640 1 --2- 172.21.15.37:0/1867282667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20c8139680 0x7f20c8134b30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f20c401c340 tx=0x7f20c40190a0 comp rx=0 tx=0).stop 2024-04-26T00:32:53.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.127+0000 7f20bdffb640 1 -- 172.21.15.37:0/1867282667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20c8139680 msgr2=0x7f20c8134b30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:32:53.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.127+0000 7f20bdffb640 1 --2- 172.21.15.37:0/1867282667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20c8139680 0x7f20c8134b30 secure :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f20c401c340 tx=0x7f20c40190a0 comp rx=0 tx=0).stop 2024-04-26T00:32:53.134 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:32:53.134 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:32:53.134 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:32:53.134 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:32:53.134 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 96, 2024-04-26T00:32:53.135 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 103490, 2024-04-26T00:32:53.135 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:32:53.135 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:32:53.135 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:32:53.135 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:32:53.135 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:53.135 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 5, 2024-04-26T00:32:53.135 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:32:53.135 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:32:53.135 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:32:53.136 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:53.136 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:53.136 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:53.136 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:53.136 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:53.136 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:53.136 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:53.136 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:53.136 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:53.136 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:53.136 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:53.136 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:53.136 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:32:53.136 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:32:53.137 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:53.137 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:53.137 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:53.137 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:53.137 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:53.137 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:53.137 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:53.137 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:53.137 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:53.137 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:53.137 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:53.137 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:53.137 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:53.137 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:32:53.137 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:32:53.138 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:32:53.138 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:32:53.138 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:32:53.138 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:32:53.138 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:32:53.138 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:32:53.138 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:32:53.138 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:32:53.138 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:32:53.138 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:32:53.138 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:32:53.138 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:32:53.138 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:32:53.138 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:32:53.138 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:32:53.139 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:53.139 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:53.139 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:53.139 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:53.139 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:53.139 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:53.139 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:53.139 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:53.139 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:53.139 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:53.139 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:53.139 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:53.139 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:32:53.139 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:53.139 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:32:53.140 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:53.140 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:32:53.140 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:32:53.140 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:53.140 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:53.140 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:53.140 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:53.140 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:32:53.140 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:53.140 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:53.140 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:53.140 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:53.140 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:32:53.140 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:53.140 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:53.141 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:53.141 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:53.141 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:32:53.141 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:32:53.141 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:53.141 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:53.141 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:53.141 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:53.141 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:53.141 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:32:53.141 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:32:53.141 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:53.141 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:53.141 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:53.141 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:53.142 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:32:53.142 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:53.142 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:53.142 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:53.142 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:53.142 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:32:53.142 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:53.142 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:53.143 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:53.143 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:53.143 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:32:53.143 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:32:53.143 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:53.143 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:53.143 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:53.143 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:53.144 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:53.144 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:32:53.144 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:32:53.144 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:53.144 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:53.144 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:53.144 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:53.144 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:32:53.144 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:53.144 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:53.145 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:53.145 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:53.145 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:32:53.145 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:53.145 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:53.145 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:53.145 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:53.145 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:32:53.145 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:32:53.145 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:53.145 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:53.145 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:53.145 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:53.145 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:53.145 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:53.146 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:32:53.146 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:32:53.146 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:53.146 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:53.146 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:53.146 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:32:53.146 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:53.146 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:53.146 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:32:53.146 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:53.146 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:53.146 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:53.146 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:32:53.146 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:53.147 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:53.147 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:32:53.147 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:53.147 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:53.147 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:53.147 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:32:53.147 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:53.147 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:53.147 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:32:53.147 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:53.147 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:53.147 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:53.147 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:32:53.147 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:53.147 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:53.148 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:53.148 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:32:53.148 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:32:53.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.127+0000 7f20cf59e640 1 -- 172.21.15.37:0/1867282667 >> v2:172.21.15.45:6812/2379918623 conn(0x7f20ac035ed0 msgr2=0x7f20ac038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:53.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.127+0000 7f20cf59e640 1 --2- 172.21.15.37:0/1867282667 >> v2:172.21.15.45:6812/2379918623 conn(0x7f20ac035ed0 0x7f20ac038390 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f20c813a600 tx=0x7f20b8008040 comp rx=0 tx=0).stop 2024-04-26T00:32:53.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.127+0000 7f20cf59e640 10 monclient: shutdown 2024-04-26T00:32:53.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.127+0000 7f20cf59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:53.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.127+0000 7f20cf59e640 1 -- 172.21.15.37:0/1867282667 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f20c8133b20 msgr2=0x7f20c813e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:53.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.127+0000 7f20cf59e640 1 --2- 172.21.15.37:0/1867282667 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f20c8133b20 0x7f20c813e760 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f20c000caf0 tx=0x7f20c000cef0 comp rx=0 tx=0).stop 2024-04-26T00:32:53.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.127+0000 7f20cdb3b640 1 -- 172.21.15.37:0/1867282667 reap_dead start 2024-04-26T00:32:53.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.127+0000 7f20cdb3b640 1 -- 172.21.15.37:0/1867282667 reap_dead start 2024-04-26T00:32:53.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.127+0000 7f20cf59e640 1 -- 172.21.15.37:0/1867282667 shutdown_connections 2024-04-26T00:32:53.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.127+0000 7f20cf59e640 1 -- 172.21.15.37:0/1867282667 >> 172.21.15.37:0/1867282667 conn(0x7f20c80aa0a0 msgr2=0x7f20c80a2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:53.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.127+0000 7f20cf59e640 1 -- 172.21.15.37:0/1867282667 shutdown_connections 2024-04-26T00:32:53.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.127+0000 7f20cf59e640 1 -- 172.21.15.37:0/1867282667 wait complete. 2024-04-26T00:32:53.150 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:32:53.150 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-26T00:32:53.150 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-26T00:32:53.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.267+0000 7f178e2f4640 10 monclient: get_monmap_and_config 2024-04-26T00:32:53.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.267+0000 7f178e2f4640 10 monclient: build_initial_monmap 2024-04-26T00:32:53.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.267+0000 7f178e2f4640 10 monclient: monmap: 2024-04-26T00:32:53.265 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:53.265 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:53.265 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:32:53.274658+0000 2024-04-26T00:32:53.265 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:32:53.274658+0000 2024-04-26T00:32:53.266 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:53.266 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:53.266 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:32:53.266 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:32:53.266 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:32:53.266 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:32:53.266 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:32:53.266 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:53.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.267+0000 7f178e2f4640 1 Processor -- start 2024-04-26T00:32:53.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.267+0000 7f178e2f4640 1 -- start start 2024-04-26T00:32:53.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.267+0000 7f178e2f4640 10 monclient: init 2024-04-26T00:32:53.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.271+0000 7f178e2f4640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:53.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.271+0000 7f178e2f4640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-04-26T00:32:53.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.271+0000 7f178e2f4640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f17880a0260 0x7f17880a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:53.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.271+0000 7f178e2f4640 10 monclient(hunting): picked mon.noname-a con 0x7f17880a0260 addr v2:172.21.15.37:3300/0 2024-04-26T00:32:53.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.271+0000 7f178e2f4640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1788134b30 0x7f1788134f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:53.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.271+0000 7f178e2f4640 10 monclient(hunting): picked mon.noname-c con 0x7f1788134b30 addr v2:172.21.15.45:3300/0 2024-04-26T00:32:53.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.271+0000 7f178e2f4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1788135490 addr v1:172.21.15.37:6789/0 2024-04-26T00:32:53.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.271+0000 7f178e2f4640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17880a1130 con 0x7f1788135490 2024-04-26T00:32:53.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.271+0000 7f178e2f4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:53.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.271+0000 7f178e2f4640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f178809ec10 con 0x7f17880a0260 2024-04-26T00:32:53.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.271+0000 7f178e2f4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:53.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.271+0000 7f178e2f4640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f178809f0f0 con 0x7f1788134b30 2024-04-26T00:32:53.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.271+0000 7f178e2f4640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:53.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.271+0000 7f178e2f4640 10 monclient(hunting): authenticate will time out at 1754.580418s 2024-04-26T00:32:53.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.271+0000 7f1787fff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f17880a0260 0x7f17880a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:53.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.271+0000 7f178c891640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f1788135490 0x7f178813be60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:48936/0 (socket says 172.21.15.37:48936) 2024-04-26T00:32:53.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.271+0000 7f178c891640 1 -- 172.21.15.37:0/3918062118 learned_addr learned my addr 172.21.15.37:0/3918062118 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:32:53.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.271+0000 7f1787fff640 10 monclient(hunting): get_auth_request con 0x7f17880a0260 auth_method 0 2024-04-26T00:32:53.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.271+0000 7f1787fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:53.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.271+0000 7f1787fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:53.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.271+0000 7f1787fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:53.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.271+0000 7f1787fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:53.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.271+0000 7f1787fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:53.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.271+0000 7f1787fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:53.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.271+0000 7f17877fe640 1 --2- 172.21.15.37:0/3918062118 >> v2:172.21.15.45:3300/0 conn(0x7f1788134b30 0x7f1788134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:53.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.271+0000 7f17877fe640 10 monclient(hunting): get_auth_request con 0x7f1788134b30 auth_method 0 2024-04-26T00:32:53.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.271+0000 7f17877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:53.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.271+0000 7f17877fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:53.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.271+0000 7f17877fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:53.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.271+0000 7f1787fff640 10 monclient(hunting): handle_auth_done global_id 144213 payload 274 2024-04-26T00:32:53.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.271+0000 7f1787fff640 1 -- 172.21.15.37:0/3918062118 >> v2:172.21.15.45:3300/0 conn(0x7f1788134b30 msgr2=0x7f1788134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:53.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.271+0000 7f1787fff640 1 --2- 172.21.15.37:0/3918062118 >> v2:172.21.15.45:3300/0 conn(0x7f1788134b30 0x7f1788134f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:53.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.271+0000 7f1787fff640 1 -- 172.21.15.37:0/3918062118 >> v1:172.21.15.37:6789/0 conn(0x7f1788135490 legacy=0x7f178813be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:53.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.271+0000 7f1787fff640 10 monclient: _finish_hunting 0 2024-04-26T00:32:53.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.271+0000 7f1787fff640 1 monclient: found mon.noname-a 2024-04-26T00:32:53.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.271+0000 7f1787fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:53.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.271+0000 7f1787fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:32:53.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.271+0000 7f1787fff640 1 -- 172.21.15.37:0/3918062118 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f178806cff0 con 0x7f17880a0260 2024-04-26T00:32:53.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f1787fff640 1 -- 172.21.15.37:0/3918062118 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f17880a0260 msgr2=0x7f17880a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:53.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f1787fff640 1 --2- 172.21.15.37:0/3918062118 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f17880a0260 0x7f17880a0640 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f178809f690 tx=0x7f178001c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=289ac717709f403e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:53.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f1786ffd640 1 -- 172.21.15.37:0/3918062118 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17800a6040 con 0x7f17880a0260 2024-04-26T00:32:53.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f1786ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:53.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f1786ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:32:53.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f1786ffd640 10 monclient: dump: 2024-04-26T00:32:53.273 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:53.273 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:53.273 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:53.273 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:53.273 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:53.273 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:53.273 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:53.274 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:53.274 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:53.274 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:53.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f1786ffd640 10 monclient: _finish_auth 0 2024-04-26T00:32:53.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f1786ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:32:53.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f1786ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:53.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f1786ffd640 1 -- 172.21.15.37:0/3918062118 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17800af070 con 0x7f17880a0260 2024-04-26T00:32:53.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f1786ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:53.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f1786ffd640 1 -- 172.21.15.37:0/3918062118 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17800aa860 con 0x7f17880a0260 2024-04-26T00:32:53.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178e2f4640 5 monclient: authenticate success, global_id 144213 2024-04-26T00:32:53.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178e2f4640 10 monclient: get_monmap_and_config success 2024-04-26T00:32:53.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178e2f4640 4 set_mon_vals no callback set 2024-04-26T00:32:53.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f1786ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:53.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f1786ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:53.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f1786ffd640 10 monclient: dump: 2024-04-26T00:32:53.275 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:53.275 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:53.275 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:53.275 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:53.275 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:53.275 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:53.275 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:53.275 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:53.275 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:53.275 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:53.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178e2f4640 10 monclient: shutdown 2024-04-26T00:32:53.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178e2f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:53.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178e2f4640 1 -- 172.21.15.37:0/3918062118 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f17880a0260 msgr2=0x7f17880a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:53.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178e2f4640 1 --2- 172.21.15.37:0/3918062118 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f17880a0260 0x7f17880a0640 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f178809f690 tx=0x7f178001c5d0 comp rx=0 tx=0).stop 2024-04-26T00:32:53.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178e2f4640 1 -- 172.21.15.37:0/3918062118 shutdown_connections 2024-04-26T00:32:53.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178e2f4640 1 --2- 172.21.15.37:0/3918062118 >> v2:172.21.15.45:3300/0 conn(0x7f1788134b30 0x7f1788134f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:53.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178e2f4640 1 --2- 172.21.15.37:0/3918062118 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f17880a0260 0x7f17880a0640 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:53.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178e2f4640 1 -- 172.21.15.37:0/3918062118 >> 172.21.15.37:0/3918062118 conn(0x7f17880a1ad0 msgr2=0x7f17880a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:53.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178e2f4640 1 -- 172.21.15.37:0/3918062118 shutdown_connections 2024-04-26T00:32:53.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178e2f4640 1 -- 172.21.15.37:0/3918062118 wait complete. 2024-04-26T00:32:53.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178e2f4640 10 monclient: build_initial_monmap 2024-04-26T00:32:53.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178e2f4640 10 monclient: monmap: 2024-04-26T00:32:53.276 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:53.276 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:32:53.276 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:32:53.276 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:32:53.276 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:53.276 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:32:53.276 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:32:53.277 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:32:53.277 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:32:53.277 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:53.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178e2f4640 1 Processor -- start 2024-04-26T00:32:53.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178e2f4640 1 -- start start 2024-04-26T00:32:53.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178e2f4640 10 monclient: init 2024-04-26T00:32:53.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178e2f4640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:53.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178e2f4640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:32:53.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178e2f4640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f17880a0260 0x7f1788131680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:53.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178e2f4640 10 monclient(hunting): picked mon.noname-c con 0x7f17880a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:53.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178e2f4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1788134b30 0x7f1788139610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:53.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178e2f4640 10 monclient(hunting): picked mon.noname-b con 0x7f1788134b30 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:32:53.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178e2f4640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1788135490 0x7f1788139b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:53.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178e2f4640 10 monclient(hunting): picked mon.noname-a con 0x7f1788135490 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:32:53.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178e2f4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:53.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178e2f4640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f178809f430 con 0x7f1788135490 2024-04-26T00:32:53.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178e2f4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:53.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178e2f4640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f178809f2b0 con 0x7f1788134b30 2024-04-26T00:32:53.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178e2f4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:53.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178e2f4640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f178809fe90 con 0x7f17880a0260 2024-04-26T00:32:53.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178e2f4640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:53.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178e2f4640 10 monclient(hunting): authenticate will time out at 1754.587683s 2024-04-26T00:32:53.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178c891640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1788135490 0x7f1788139b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:53.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f1787fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f17880a0260 0x7f1788131680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:53.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f1787fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f17880a0260 0x7f1788131680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:48066/0 (socket says 172.21.15.37:48066) 2024-04-26T00:32:53.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178c891640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1788135490 0x7f1788139b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44690/0 (socket says 172.21.15.37:44690) 2024-04-26T00:32:53.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f1787fff640 1 -- 172.21.15.37:0/1588892556 learned_addr learned my addr 172.21.15.37:0/1588892556 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:32:53.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f1787fff640 10 monclient(hunting): get_auth_request con 0x7f17880a0260 auth_method 0 2024-04-26T00:32:53.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f1787fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:53.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f1787fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:53.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f1787fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:53.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178c891640 10 monclient(hunting): get_auth_request con 0x7f1788135490 auth_method 0 2024-04-26T00:32:53.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178c891640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:53.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178c891640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:53.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178c891640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:53.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178c891640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:53.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178c891640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:53.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f178c891640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:53.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f1787fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:53.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f1787fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:53.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.275+0000 7f1787fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:53.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f17877fe640 1 --2- 172.21.15.37:0/1588892556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1788134b30 0x7f1788139610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:53.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f17877fe640 10 monclient(hunting): get_auth_request con 0x7f1788134b30 auth_method 0 2024-04-26T00:32:53.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f17877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:53.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f17877fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:53.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f17877fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:53.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f178c891640 10 monclient(hunting): handle_auth_done global_id 144216 payload 824 2024-04-26T00:32:53.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f178c891640 1 -- 172.21.15.37:0/1588892556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f17880a0260 msgr2=0x7f1788131680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:53.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f178c891640 1 --2- 172.21.15.37:0/1588892556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f17880a0260 0x7f1788131680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:53.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f178c891640 1 -- 172.21.15.37:0/1588892556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1788134b30 msgr2=0x7f1788139610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:53.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f178c891640 1 --2- 172.21.15.37:0/1588892556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1788134b30 0x7f1788139610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:53.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f178c891640 10 monclient: _finish_hunting 0 2024-04-26T00:32:53.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f178c891640 1 monclient: found mon.noname-a 2024-04-26T00:32:53.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f178c891640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:53.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f178c891640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:32:53.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f178c891640 1 -- 172.21.15.37:0/1588892556 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f178811c010 con 0x7f1788135490 2024-04-26T00:32:53.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f178c891640 1 -- 172.21.15.37:0/1588892556 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1788135490 msgr2=0x7f1788139b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:53.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f178c891640 1 --2- 172.21.15.37:0/1588892556 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1788135490 0x7f1788139b50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f177800c410 tx=0x7f177800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:53.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f1784ff9640 1 -- 172.21.15.37:0/1588892556 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1778017070 con 0x7f1788135490 2024-04-26T00:32:53.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f1784ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:53.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f1784ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:32:53.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f1784ff9640 10 monclient: dump: 2024-04-26T00:32:53.284 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:53.284 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:53.284 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:53.284 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:53.284 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:53.284 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:53.285 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:53.285 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:53.285 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:53.285 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:53.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f1784ff9640 10 monclient: _finish_auth 0 2024-04-26T00:32:53.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f1784ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:32:53.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f1784ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:53.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f1784ff9640 1 -- 172.21.15.37:0/1588892556 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17780070d0 con 0x7f1788135490 2024-04-26T00:32:53.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f178e2f4640 5 monclient: authenticate success, global_id 144216 2024-04-26T00:32:53.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f178e2f4640 10 monclient: _renew_subs 2024-04-26T00:32:53.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f178e2f4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:53.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f178e2f4640 1 -- 172.21.15.37:0/1588892556 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17881cd940 con 0x7f1788135490 2024-04-26T00:32:53.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f1784ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:53.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f1784ff9640 1 -- 172.21.15.37:0/1588892556 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17780074c0 con 0x7f1788135490 2024-04-26T00:32:53.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f1784ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:53.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f1784ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:53.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f1784ff9640 10 monclient: dump: 2024-04-26T00:32:53.286 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:53.286 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:53.286 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:53.286 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:53.286 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:53.286 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:53.286 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:53.286 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:53.286 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:53.286 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:53.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f178e2f4640 10 monclient: _renew_subs 2024-04-26T00:32:53.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f17867fc640 4 set_mon_vals no callback set 2024-04-26T00:32:53.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f178e2f4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:53.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f178e2f4640 1 -- 172.21.15.37:0/1588892556 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f178811bd30 con 0x7f1788135490 2024-04-26T00:32:53.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f178e2f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:53.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f178e2f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:53.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f178e2f4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:53.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f1784ff9640 1 -- 172.21.15.37:0/1588892556 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f177801a070 con 0x7f1788135490 2024-04-26T00:32:53.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f178e2f4640 1 -- 172.21.15.37:0/1588892556 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1788066f40 con 0x7f1788135490 2024-04-26T00:32:53.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f1784ff9640 1 --2- 172.21.15.37:0/1588892556 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1768035ed0 0x7f1768038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:53.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.279+0000 7f1784ff9640 1 -- 172.21.15.37:0/1588892556 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(501..501 src has 370..501) ==== 3959+0+0 (secure 0 0 0) 0x7f1778047e60 con 0x7f1788135490 2024-04-26T00:32:53.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.283+0000 7f1787fff640 1 --2- 172.21.15.37:0/1588892556 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1768035ed0 0x7f1768038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:53.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.283+0000 7f1787fff640 10 monclient: get_auth_request con 0x7f1768035ed0 auth_method 0 2024-04-26T00:32:53.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.283+0000 7f1787fff640 1 --2- 172.21.15.37:0/1588892556 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1768035ed0 0x7f1768038390 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1780000f90 tx=0x7f17800b5040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:53.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.287+0000 7f1784ff9640 1 -- 172.21.15.37:0/1588892556 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f1778010300 con 0x7f1788135490 2024-04-26T00:32:53.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.287+0000 7f1784ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:53.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.287+0000 7f1784ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:32:53.484 INFO:tasks.rados.rados.0.smithi045.stdout:2216: done (1 left) 2024-04-26T00:32:53.484 INFO:tasks.rados.rados.0.smithi045.stdout:2217: snap_create 2024-04-26T00:32:53.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.527+0000 7f178e2f4640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-26T00:32:53.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.527+0000 7f178e2f4640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-26T00:32:53.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.527+0000 7f178e2f4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:53.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.527+0000 7f178e2f4640 1 -- 172.21.15.37:0/1588892556 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f178813bb90 con 0x7f1788135490 2024-04-26T00:32:53.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.531+0000 7f1784ff9640 1 -- 172.21.15.37:0/1588892556 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f177801b3b0 con 0x7f1788135490 2024-04-26T00:32:53.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.531+0000 7f1784ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-26T00:32:53.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.531+0000 7f1784ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:32:53.528 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.531+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.528 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.531+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=12,kv=10,logm=53} crc {auth=1547848594,config=692619927,health=2510573225,kv=1034996044,logm=2813348894}) 2024-04-26T00:32:53.529 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.531+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=22,config=2,health=9,kv=9,logm=58} crc {auth=2408446352,config=692619927,health=2896933315,kv=377031510,logm=2840774920}) 2024-04-26T00:32:53.529 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.531+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.529 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.531+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=12,kv=10,logm=53} crc {auth=1547848594,config=692619927,health=2510573225,kv=1034996044,logm=2813348894}) 2024-04-26T00:32:53.529 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.531+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=23,config=2,health=10,kv=10,logm=55} crc {auth=1742812633,config=692619928,health=3893908990,kv=1034996044,logm=3101884428}) 2024-04-26T00:32:53.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.531+0000 7f178e2f4640 1 -- 172.21.15.37:0/1588892556 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1768035ed0 msgr2=0x7f1768038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:53.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.531+0000 7f178e2f4640 1 --2- 172.21.15.37:0/1588892556 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1768035ed0 0x7f1768038390 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1780000f90 tx=0x7f17800b5040 comp rx=0 tx=0).stop 2024-04-26T00:32:53.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.531+0000 7f178e2f4640 10 monclient: shutdown 2024-04-26T00:32:53.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.531+0000 7f178e2f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:53.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.531+0000 7f178e2f4640 1 -- 172.21.15.37:0/1588892556 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1788135490 msgr2=0x7f1788139b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:53.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.531+0000 7f178e2f4640 1 --2- 172.21.15.37:0/1588892556 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1788135490 0x7f1788139b50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f177800c410 tx=0x7f177800c8d0 comp rx=0 tx=0).stop 2024-04-26T00:32:53.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.531+0000 7f178e2f4640 1 -- 172.21.15.37:0/1588892556 shutdown_connections 2024-04-26T00:32:53.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.531+0000 7f178e2f4640 1 --2- 172.21.15.37:0/1588892556 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1768035ed0 0x7f1768038390 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:53.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.531+0000 7f178e2f4640 1 --2- 172.21.15.37:0/1588892556 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1788135490 0x7f1788139b50 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:53.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.531+0000 7f178e2f4640 1 --2- 172.21.15.37:0/1588892556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1788134b30 0x7f1788139610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:53.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.531+0000 7f178e2f4640 1 --2- 172.21.15.37:0/1588892556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f17880a0260 0x7f1788131680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:53.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.531+0000 7f178e2f4640 1 -- 172.21.15.37:0/1588892556 >> 172.21.15.37:0/1588892556 conn(0x7f17880a1ad0 msgr2=0x7f1788132f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:53.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.531+0000 7f178e2f4640 1 -- 172.21.15.37:0/1588892556 shutdown_connections 2024-04-26T00:32:53.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:53.531+0000 7f178e2f4640 1 -- 172.21.15.37:0/1588892556 wait complete. 2024-04-26T00:32:53.539 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.535+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.539 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.535+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=417865342}) 2024-04-26T00:32:53.540 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.535+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3983764951}) 2024-04-26T00:32:53.540 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.535+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.540 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.535+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=417865342}) 2024-04-26T00:32:53.540 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.535+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3609228251}) 2024-04-26T00:32:53.540 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.535+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.540 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.535+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1638497685}) 2024-04-26T00:32:53.540 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.535+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3555280740}) 2024-04-26T00:32:53.540 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.535+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.540 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.535+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1638497685}) 2024-04-26T00:32:53.540 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.535+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4178251115}) 2024-04-26T00:32:53.540 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.539+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.540 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.539+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3289628478}) 2024-04-26T00:32:53.540 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.539+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3654770125}) 2024-04-26T00:32:53.540 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.539+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.541 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.539+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3289628478}) 2024-04-26T00:32:53.541 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.539+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2055034539}) 2024-04-26T00:32:53.541 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.543+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.541 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.543+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=546335579}) 2024-04-26T00:32:53.541 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.543+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2799529347}) 2024-04-26T00:32:53.541 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.543+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.541 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.543+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=546335579}) 2024-04-26T00:32:53.541 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.543+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1252418587}) 2024-04-26T00:32:53.542 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.543+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.542 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.543+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1694884451}) 2024-04-26T00:32:53.542 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.543+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1990457791}) 2024-04-26T00:32:53.542 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.543+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.542 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.543+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1694884451}) 2024-04-26T00:32:53.542 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.543+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2459873947}) 2024-04-26T00:32:53.543 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.547+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.543 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.547+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4208180183}) 2024-04-26T00:32:53.543 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.547+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1037391424}) 2024-04-26T00:32:53.543 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.547+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.543 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.547+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4208180183}) 2024-04-26T00:32:53.543 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.547+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3289608840}) 2024-04-26T00:32:53.545 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.547+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.545 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.547+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2031770573}) 2024-04-26T00:32:53.545 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.547+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2459151592}) 2024-04-26T00:32:53.545 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.547+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.545 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.547+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2031770573}) 2024-04-26T00:32:53.545 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.547+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=170524392}) 2024-04-26T00:32:53.547 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.551+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.547 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.551+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=696134577}) 2024-04-26T00:32:53.547 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.551+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1405444907}) 2024-04-26T00:32:53.547 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.551+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.548 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-26T00:32:53.548 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.551+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=696134577}) 2024-04-26T00:32:53.548 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.551+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2362210743}) 2024-04-26T00:32:53.549 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.551+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.549 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.551+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2299456962}) 2024-04-26T00:32:53.549 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.551+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2907561833}) 2024-04-26T00:32:53.550 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.551+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.550 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.551+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2299456962}) 2024-04-26T00:32:53.550 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.551+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1247569379}) 2024-04-26T00:32:53.551 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.555+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.552 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.555+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2515363404}) 2024-04-26T00:32:53.552 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.555+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=79016192}) 2024-04-26T00:32:53.552 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.555+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.552 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.555+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2515363404}) 2024-04-26T00:32:53.552 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.555+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=927612615}) 2024-04-26T00:32:53.553 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.555+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.553 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.555+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=230678475}) 2024-04-26T00:32:53.553 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.555+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3700798569}) 2024-04-26T00:32:53.553 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.555+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.553 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.555+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=230678475}) 2024-04-26T00:32:53.554 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.555+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2918016972}) 2024-04-26T00:32:53.555 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.559+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.556 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.559+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3282769340}) 2024-04-26T00:32:53.556 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.559+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4011969795}) 2024-04-26T00:32:53.556 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.559+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.556 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.559+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3282769340}) 2024-04-26T00:32:53.556 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.559+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=183012014}) 2024-04-26T00:32:53.558 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.559+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.558 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.559+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3582201197}) 2024-04-26T00:32:53.558 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.559+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=77175636}) 2024-04-26T00:32:53.558 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.559+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.558 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.559+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3582201197}) 2024-04-26T00:32:53.558 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.559+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1352511021}) 2024-04-26T00:32:53.560 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.563+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.560 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.563+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1327784340}) 2024-04-26T00:32:53.560 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.563+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=578782447}) 2024-04-26T00:32:53.560 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.563+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.560 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.563+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1327784340}) 2024-04-26T00:32:53.560 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.563+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4098043487}) 2024-04-26T00:32:53.569 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.571+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.569 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.571+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=990523889}) 2024-04-26T00:32:53.569 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.571+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=86413100}) 2024-04-26T00:32:53.569 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.571+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.569 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.571+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=990523889}) 2024-04-26T00:32:53.569 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.571+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2793914390}) 2024-04-26T00:32:53.571 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.575+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.571 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.575+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2020837105}) 2024-04-26T00:32:53.571 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.575+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=752859571}) 2024-04-26T00:32:53.571 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.575+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.571 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.575+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2020837105}) 2024-04-26T00:32:53.571 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.575+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2729511337}) 2024-04-26T00:32:53.574 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.575+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.574 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.575+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1667446920}) 2024-04-26T00:32:53.574 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.575+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=510723102}) 2024-04-26T00:32:53.574 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.575+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.574 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.575+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1667446920}) 2024-04-26T00:32:53.574 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.575+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=835833053}) 2024-04-26T00:32:53.578 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.579+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.578 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.579+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4159746015}) 2024-04-26T00:32:53.578 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.583+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3861503098}) 2024-04-26T00:32:53.578 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.583+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.579 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.583+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4159746015}) 2024-04-26T00:32:53.579 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.583+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1448250094}) 2024-04-26T00:32:53.588 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.591+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.588 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.591+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=627743505}) 2024-04-26T00:32:53.588 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.591+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3871000006}) 2024-04-26T00:32:53.588 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.591+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.588 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.591+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=627743505}) 2024-04-26T00:32:53.588 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.591+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3378165888}) 2024-04-26T00:32:53.593 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.595+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.593 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.595+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=667836910}) 2024-04-26T00:32:53.593 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.595+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2993912444}) 2024-04-26T00:32:53.593 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.595+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.593 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.595+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=667836910}) 2024-04-26T00:32:53.593 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.595+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1283208613}) 2024-04-26T00:32:53.595 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.599+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.595 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.599+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3808264724}) 2024-04-26T00:32:53.595 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.599+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=128922900}) 2024-04-26T00:32:53.595 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.599+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.595 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.599+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3808264724}) 2024-04-26T00:32:53.595 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.599+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3902480389}) 2024-04-26T00:32:53.597 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.599+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.597 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.599+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1343391136}) 2024-04-26T00:32:53.597 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.599+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=405950685}) 2024-04-26T00:32:53.597 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.599+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.597 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.599+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1343391136}) 2024-04-26T00:32:53.598 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.599+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1128563317}) 2024-04-26T00:32:53.600 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.603+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.600 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.603+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1069810986}) 2024-04-26T00:32:53.600 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.603+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3744207975}) 2024-04-26T00:32:53.600 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.603+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.600 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.603+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1069810986}) 2024-04-26T00:32:53.600 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.603+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2982807976}) 2024-04-26T00:32:53.602 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.603+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.602 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.603+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=627508510}) 2024-04-26T00:32:53.602 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.603+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1729602936}) 2024-04-26T00:32:53.602 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.603+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.602 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.603+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=627508510}) 2024-04-26T00:32:53.602 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.607+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2042858490}) 2024-04-26T00:32:53.604 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.607+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.604 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.607+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1748713608}) 2024-04-26T00:32:53.604 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.607+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=490403833}) 2024-04-26T00:32:53.605 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.607+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.605 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.607+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1748713608}) 2024-04-26T00:32:53.605 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.607+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=674472665}) 2024-04-26T00:32:53.607 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.611+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.607 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.611+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3562976576}) 2024-04-26T00:32:53.607 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.611+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2856927870}) 2024-04-26T00:32:53.607 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.611+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.607 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.611+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3562976576}) 2024-04-26T00:32:53.607 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.611+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1847130338}) 2024-04-26T00:32:53.609 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.611+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.609 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.611+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1969823226}) 2024-04-26T00:32:53.609 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.611+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3314187228}) 2024-04-26T00:32:53.609 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.611+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.609 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.611+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1969823226}) 2024-04-26T00:32:53.609 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.611+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3264148358}) 2024-04-26T00:32:53.611 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.615+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.611 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.615+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=293590616}) 2024-04-26T00:32:53.612 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.615+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3682175082}) 2024-04-26T00:32:53.612 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.615+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.612 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.615+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=293590616}) 2024-04-26T00:32:53.612 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.615+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1993653845}) 2024-04-26T00:32:53.614 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.615+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.614 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.615+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3971465467}) 2024-04-26T00:32:53.614 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.615+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4282276069}) 2024-04-26T00:32:53.614 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.615+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.614 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.615+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3971465467}) 2024-04-26T00:32:53.614 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.615+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3597963430}) 2024-04-26T00:32:53.616 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.619+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.616 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.619+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3843393859}) 2024-04-26T00:32:53.616 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.619+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1789845987}) 2024-04-26T00:32:53.616 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.619+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.616 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.619+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3843393859}) 2024-04-26T00:32:53.616 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.619+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1350784788}) 2024-04-26T00:32:53.618 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.619+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.618 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.619+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=743652207}) 2024-04-26T00:32:53.618 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.623+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3072236101}) 2024-04-26T00:32:53.619 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.623+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.619 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.623+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=743652207}) 2024-04-26T00:32:53.619 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.623+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2470872621}) 2024-04-26T00:32:53.620 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.623+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.621 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.623+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2375393234}) 2024-04-26T00:32:53.621 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.623+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1956650386}) 2024-04-26T00:32:53.621 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.623+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.621 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.623+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2375393234}) 2024-04-26T00:32:53.621 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.623+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3766376002}) 2024-04-26T00:32:53.623 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.627+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.623 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.627+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=381238555}) 2024-04-26T00:32:53.623 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.627+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1442021578}) 2024-04-26T00:32:53.623 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.627+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.623 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.627+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=381238555}) 2024-04-26T00:32:53.624 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.627+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=20508606}) 2024-04-26T00:32:53.627 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.627+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.627 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.627+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2019809894}) 2024-04-26T00:32:53.627 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.627+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=124043774}) 2024-04-26T00:32:53.627 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.627+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.627 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.627+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2019809894}) 2024-04-26T00:32:53.627 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.627+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2625519532}) 2024-04-26T00:32:53.628 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.631+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.628 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.631+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4053807508}) 2024-04-26T00:32:53.628 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.631+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=283504780}) 2024-04-26T00:32:53.629 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.631+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.629 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.631+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4053807508}) 2024-04-26T00:32:53.629 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.631+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3638056522}) 2024-04-26T00:32:53.631 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.635+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.631 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.635+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2822554669}) 2024-04-26T00:32:53.631 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.635+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1088132868}) 2024-04-26T00:32:53.631 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.635+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.631 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.635+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2822554669}) 2024-04-26T00:32:53.631 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.635+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3775770024}) 2024-04-26T00:32:53.634 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.635+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.634 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.635+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1430214174}) 2024-04-26T00:32:53.634 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.635+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2629121578}) 2024-04-26T00:32:53.634 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.635+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.634 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.635+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1430214174}) 2024-04-26T00:32:53.634 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.635+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2783783196}) 2024-04-26T00:32:53.636 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.639+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.636 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.639+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1411772479}) 2024-04-26T00:32:53.636 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.639+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1027599986}) 2024-04-26T00:32:53.637 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.639+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.637 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.639+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1411772479}) 2024-04-26T00:32:53.637 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.639+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3807004183}) 2024-04-26T00:32:53.639 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.643+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.639 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.643+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1710357004}) 2024-04-26T00:32:53.639 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.643+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=809411540}) 2024-04-26T00:32:53.639 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.643+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.639 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.643+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1710357004}) 2024-04-26T00:32:53.639 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.643+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2819630366}) 2024-04-26T00:32:53.641 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.643+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.641 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.643+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=96,mdsmap=3,mgr=1} crc {logm=2267058078,mdsmap=2840952586,mgr=3189685207}) 2024-04-26T00:32:53.641 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.643+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1051527101}) 2024-04-26T00:32:53.641 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.643+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.641 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.643+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=96,mdsmap=3,mgr=1} crc {logm=2267058078,mdsmap=2840952586,mgr=3189685207}) 2024-04-26T00:32:53.641 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.643+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=98,mdsmap=2} crc {logm=2355920524,mdsmap=1401145748}) 2024-04-26T00:32:53.644 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.647+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.644 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.647+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=6,mgr_metadata=1,mgrstat=9,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=63} crc {mgr=3303164479,mgr_metadata=3686404438,mgrstat=4189538649,mon_config_key=4061063117,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=1896968211,osd_snap=839609858}) 2024-04-26T00:32:53.644 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.647+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=5,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=9,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=67} crc {mgr=2745652164,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=797502606,mon_config_key=689823698,monmap=3876866194,osd_metadata=1980645665,osd_pg_creating=1896968211,osd_snap=3962852383}) 2024-04-26T00:32:53.644 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.647+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.644 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.647+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=6,mgr_metadata=1,mgrstat=9,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=63} crc {mgr=3303164479,mgr_metadata=3686404438,mgrstat=4189538649,mon_config_key=4061063117,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=1896968211,osd_snap=839609858}) 2024-04-26T00:32:53.644 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.647+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=11,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=63} crc {mgr=1214999089,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=4189538649,mon_config_key=4061063117,monmap=4194790699,osd_metadata=3124103611,osd_pg_creating=1896968211,osd_snap=633876916}) 2024-04-26T00:32:53.647 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.651+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.647 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.651+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=25,osdmap=75} crc {osd_snap=1450158218,osdmap=2190098352}) 2024-04-26T00:32:53.647 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.651+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=25,osdmap=75} crc {osd_snap=3178157989,osdmap=2458762876}) 2024-04-26T00:32:53.647 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.651+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.647 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.651+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=25,osdmap=75} crc {osd_snap=1450158218,osdmap=2190098352}) 2024-04-26T00:32:53.647 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.651+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=25,osdmap=75} crc {osd_snap=411648583,osdmap=448623499}) 2024-04-26T00:32:53.651 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.651+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.651 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.651+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3864781227}) 2024-04-26T00:32:53.651 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.655+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2495512658}) 2024-04-26T00:32:53.651 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.655+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.651 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.655+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3864781227}) 2024-04-26T00:32:53.651 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.655+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2299175395}) 2024-04-26T00:32:53.661 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.663+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.661 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.663+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=32} crc {osdmap=153893848}) 2024-04-26T00:32:53.661 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.663+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=31} crc {osdmap=3385870280}) 2024-04-26T00:32:53.661 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.663+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:32:53.661 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.663+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=32} crc {osdmap=153893848}) 2024-04-26T00:32:53.661 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:32:53.663+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=29} crc {osdmap=3654661630}) 2024-04-26T00:32:54.494 INFO:tasks.rados.rados.0.smithi045.stdout:2215: done (1 left) 2024-04-26T00:32:54.494 INFO:tasks.rados.rados.0.smithi045.stdout:2217: done (0 left) 2024-04-26T00:32:54.494 INFO:tasks.rados.rados.0.smithi045.stdout:2218: delete oid 38 current snap is 248 2024-04-26T00:32:54.505 INFO:tasks.rados.rados.0.smithi045.stdout:2218: done (0 left) 2024-04-26T00:32:54.505 INFO:tasks.rados.rados.0.smithi045.stdout:2219: read oid 7 snap -1 2024-04-26T00:32:54.505 INFO:tasks.rados.rados.0.smithi045.stdout:2219: expect deleted 2024-04-26T00:32:54.505 INFO:tasks.rados.rados.0.smithi045.stdout:2220: delete oid 9 current snap is 248 2024-04-26T00:32:54.515 INFO:tasks.rados.rados.0.smithi045.stdout:2219: done (1 left) 2024-04-26T00:32:54.515 INFO:tasks.rados.rados.0.smithi045.stdout:2220: done (0 left) 2024-04-26T00:32:54.515 INFO:tasks.rados.rados.0.smithi045.stdout:2221: read oid 11 snap -1 2024-04-26T00:32:54.515 INFO:tasks.rados.rados.0.smithi045.stdout:2221: expect (ObjNum 791 snap 241 seq_num 791) 2024-04-26T00:32:54.515 INFO:tasks.rados.rados.0.smithi045.stdout:2222: snap_create 2024-04-26T00:32:54.527 INFO:tasks.rados.rados.0.smithi045.stdout:2221: expect (ObjNum 791 snap 241 seq_num 791) 2024-04-26T00:32:54.549 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:32:54.549 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:32:54.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.687+0000 7f66009b5640 10 monclient: get_monmap_and_config 2024-04-26T00:32:54.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.687+0000 7f66009b5640 10 monclient: build_initial_monmap 2024-04-26T00:32:54.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.687+0000 7f66009b5640 10 monclient: monmap: 2024-04-26T00:32:54.684 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:54.684 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:54.684 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:32:54.692896+0000 2024-04-26T00:32:54.684 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:32:54.692896+0000 2024-04-26T00:32:54.684 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:54.684 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:54.684 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:32:54.684 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:32:54.684 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:32:54.684 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:32:54.684 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:32:54.684 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:54.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.687+0000 7f66009b5640 1 Processor -- start 2024-04-26T00:32:54.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.687+0000 7f66009b5640 1 -- start start 2024-04-26T00:32:54.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.687+0000 7f66009b5640 10 monclient: init 2024-04-26T00:32:54.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.687+0000 7f66009b5640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:54.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.687+0000 7f66009b5640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-04-26T00:32:54.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.687+0000 7f66009b5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f65fc0a0260 addr v1:172.21.15.37:6789/0 2024-04-26T00:32:54.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.687+0000 7f66009b5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f65fc137e40 addr v1:172.21.15.45:6789/0 2024-04-26T00:32:54.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.687+0000 7f66009b5640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f65fc13b950 0x7f65fc13dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:54.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.687+0000 7f66009b5640 10 monclient(hunting): picked mon.noname-a con 0x7f65fc13b950 addr v2:172.21.15.37:3300/0 2024-04-26T00:32:54.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.687+0000 7f66009b5640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65fc0a1130 con 0x7f65fc0a0260 2024-04-26T00:32:54.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.687+0000 7f66009b5640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65fc06cff0 con 0x7f65fc137e40 2024-04-26T00:32:54.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.687+0000 7f66009b5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:54.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.687+0000 7f66009b5640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f65fc0a0d50 con 0x7f65fc13b950 2024-04-26T00:32:54.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.687+0000 7f66009b5640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:54.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.687+0000 7f66009b5640 10 monclient(hunting): authenticate will time out at 1755.998577s 2024-04-26T00:32:54.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.687+0000 7f65fa59c640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f65fc0a0260 0x7f65fc0a0640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:48952/0 (socket says 172.21.15.37:48952) 2024-04-26T00:32:54.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.687+0000 7f65fa59c640 1 -- 172.21.15.37:0/4177280368 learned_addr learned my addr 172.21.15.37:0/4177280368 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:32:54.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.687+0000 7f65fad9d640 1 --2- 172.21.15.37:0/4177280368 >> v2:172.21.15.37:3300/0 conn(0x7f65fc13b950 0x7f65fc13dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:54.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.687+0000 7f65fad9d640 10 monclient(hunting): get_auth_request con 0x7f65fc13b950 auth_method 0 2024-04-26T00:32:54.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.687+0000 7f65fad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:54.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.687+0000 7f65fad9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:54.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.687+0000 7f65fad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:54.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.691+0000 7f65fa59c640 1 -- 172.21.15.37:0/4177280368 >> v1:172.21.15.37:6789/0 conn(0x7f65fc0a0260 legacy=0x7f65fc0a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:54.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.691+0000 7f65fad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:54.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.691+0000 7f65fad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:54.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.691+0000 7f65fad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:54.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.691+0000 7f65f9d9b640 1 -- 172.21.15.37:0/4177280368 >> v1:172.21.15.45:6789/0 conn(0x7f65fc137e40 legacy=0x7f65fc13a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:54.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.691+0000 7f65fad9d640 10 monclient(hunting): handle_auth_done global_id 144219 payload 274 2024-04-26T00:32:54.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.691+0000 7f65fad9d640 1 -- 172.21.15.37:0/4177280368 >> v1:172.21.15.45:6789/0 conn(0x7f65fc137e40 legacy=0x7f65fc13a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:54.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.691+0000 7f65fad9d640 1 -- 172.21.15.37:0/4177280368 >> v1:172.21.15.37:6789/0 conn(0x7f65fc0a0260 legacy=0x7f65fc0a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:54.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.691+0000 7f65fad9d640 10 monclient: _finish_hunting 0 2024-04-26T00:32:54.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.691+0000 7f65fad9d640 1 monclient: found mon.noname-a 2024-04-26T00:32:54.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.691+0000 7f65fad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:54.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.691+0000 7f65fad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:32:54.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.691+0000 7f65fad9d640 1 -- 172.21.15.37:0/4177280368 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65fc09bc50 con 0x7f65fc13b950 2024-04-26T00:32:54.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f65fad9d640 1 -- 172.21.15.37:0/4177280368 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65fc13b950 msgr2=0x7f65fc13dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:54.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f65fad9d640 1 --2- 172.21.15.37:0/4177280368 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65fc13b950 0x7f65fc13dd40 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f65f4009e90 tx=0x7f65f401c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=750a93fb3eca3980 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:54.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f65f959a640 1 -- 172.21.15.37:0/4177280368 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f40b2070 con 0x7f65fc13b950 2024-04-26T00:32:54.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f65f959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:54.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f65f959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:32:54.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f65f959a640 10 monclient: dump: 2024-04-26T00:32:54.693 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:54.693 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:54.693 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:54.693 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:54.693 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:54.693 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:54.693 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:54.693 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:54.693 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:54.693 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:54.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f65f959a640 10 monclient: _finish_auth 0 2024-04-26T00:32:54.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f65f959a640 10 monclient: _check_auth_tickets 2024-04-26T00:32:54.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f65f959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:54.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f65f959a640 1 -- 172.21.15.37:0/4177280368 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65f40a58d0 con 0x7f65fc13b950 2024-04-26T00:32:54.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f66009b5640 5 monclient: authenticate success, global_id 144219 2024-04-26T00:32:54.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f66009b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:32:54.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f65f959a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:54.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f66009b5640 10 monclient: get_monmap_and_config success 2024-04-26T00:32:54.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f66009b5640 4 set_mon_vals no callback set 2024-04-26T00:32:54.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f65f959a640 1 -- 172.21.15.37:0/4177280368 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f40a5ca0 con 0x7f65fc13b950 2024-04-26T00:32:54.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f65f959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:54.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f65f959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:54.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f65f959a640 10 monclient: dump: 2024-04-26T00:32:54.694 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:54.694 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:54.695 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:54.695 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:54.695 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:54.695 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:54.695 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:54.695 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:54.695 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:54.695 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:54.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f66009b5640 10 monclient: shutdown 2024-04-26T00:32:54.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f66009b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:54.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f66009b5640 1 -- 172.21.15.37:0/4177280368 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65fc13b950 msgr2=0x7f65fc13dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:54.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f66009b5640 1 --2- 172.21.15.37:0/4177280368 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65fc13b950 0x7f65fc13dd40 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f65f4009e90 tx=0x7f65f401c930 comp rx=0 tx=0).stop 2024-04-26T00:32:54.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f66009b5640 1 -- 172.21.15.37:0/4177280368 shutdown_connections 2024-04-26T00:32:54.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f66009b5640 1 --2- 172.21.15.37:0/4177280368 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65fc13b950 0x7f65fc13dd40 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:54.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f66009b5640 1 -- 172.21.15.37:0/4177280368 >> 172.21.15.37:0/4177280368 conn(0x7f65fc0a1ad0 msgr2=0x7f65fc0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:54.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f66009b5640 1 -- 172.21.15.37:0/4177280368 shutdown_connections 2024-04-26T00:32:54.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f66009b5640 1 -- 172.21.15.37:0/4177280368 wait complete. 2024-04-26T00:32:54.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f66009b5640 10 monclient: build_initial_monmap 2024-04-26T00:32:54.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f66009b5640 10 monclient: monmap: 2024-04-26T00:32:54.700 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:54.700 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:32:54.700 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:32:54.700 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:32:54.700 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:54.700 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:32:54.700 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:32:54.700 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:32:54.700 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:32:54.700 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:54.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f66009b5640 1 Processor -- start 2024-04-26T00:32:54.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f66009b5640 1 -- start start 2024-04-26T00:32:54.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f66009b5640 10 monclient: init 2024-04-26T00:32:54.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f66009b5640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:54.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f66009b5640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:32:54.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f66009b5640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65fc0a0260 0x7f65fc136810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:54.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f66009b5640 10 monclient(hunting): picked mon.noname-a con 0x7f65fc0a0260 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:32:54.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f66009b5640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f65fc137e40 0x7f65fc11c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:54.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f66009b5640 10 monclient(hunting): picked mon.noname-c con 0x7f65fc137e40 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:54.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f66009b5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65fc13b950 0x7f65fc11c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:54.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f66009b5640 10 monclient(hunting): picked mon.noname-b con 0x7f65fc13b950 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:32:54.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f66009b5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:54.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f66009b5640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f65fc09f4c0 con 0x7f65fc0a0260 2024-04-26T00:32:54.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f66009b5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:54.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f66009b5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f65fc09f0d0 con 0x7f65fc13b950 2024-04-26T00:32:54.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f66009b5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:54.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f66009b5640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f65fc0a00a0 con 0x7f65fc137e40 2024-04-26T00:32:54.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f66009b5640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:54.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.695+0000 7f66009b5640 10 monclient(hunting): authenticate will time out at 1756.007923s 2024-04-26T00:32:54.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65fa59c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65fc0a0260 0x7f65fc136810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:54.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65f9d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f65fc137e40 0x7f65fc11c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:54.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65fa59c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65fc0a0260 0x7f65fc136810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44718/0 (socket says 172.21.15.37:44718) 2024-04-26T00:32:54.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65fa59c640 1 -- 172.21.15.37:0/2779623603 learned_addr learned my addr 172.21.15.37:0/2779623603 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:32:54.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65fa59c640 10 monclient(hunting): get_auth_request con 0x7f65fc0a0260 auth_method 0 2024-04-26T00:32:54.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:54.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65fa59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:54.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:54.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65f9d9b640 10 monclient(hunting): get_auth_request con 0x7f65fc137e40 auth_method 0 2024-04-26T00:32:54.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65f9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:54.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65f9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:54.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65f9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:54.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:54.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:54.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65fad9d640 1 --2- 172.21.15.37:0/2779623603 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65fc13b950 0x7f65fc11c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:54.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:54.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65f9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:54.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65f9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:54.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65f9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:54.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65fad9d640 10 monclient(hunting): get_auth_request con 0x7f65fc13b950 auth_method 0 2024-04-26T00:32:54.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65fad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:54.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65fad9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:54.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65fad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:54.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65fad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:54.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65fad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:54.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65fad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:54.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65fa59c640 10 monclient(hunting): handle_auth_done global_id 144225 payload 824 2024-04-26T00:32:54.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65fa59c640 1 -- 172.21.15.37:0/2779623603 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f65fc137e40 msgr2=0x7f65fc11c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:54.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65fa59c640 1 --2- 172.21.15.37:0/2779623603 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f65fc137e40 0x7f65fc11c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:54.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65fa59c640 1 -- 172.21.15.37:0/2779623603 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65fc13b950 msgr2=0x7f65fc11c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:54.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65fa59c640 1 --2- 172.21.15.37:0/2779623603 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65fc13b950 0x7f65fc11c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:54.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65fa59c640 10 monclient: _finish_hunting 0 2024-04-26T00:32:54.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65fa59c640 1 monclient: found mon.noname-a 2024-04-26T00:32:54.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65fa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:54.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65fa59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:32:54.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65fa59c640 1 -- 172.21.15.37:0/2779623603 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65fc11dea0 con 0x7f65fc0a0260 2024-04-26T00:32:54.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65fa59c640 1 -- 172.21.15.37:0/2779623603 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65fc0a0260 msgr2=0x7f65fc136810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:54.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65fa59c640 1 --2- 172.21.15.37:0/2779623603 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65fc0a0260 0x7f65fc136810 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f65ec00c500 tx=0x7f65ec00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:54.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65eaffd640 1 -- 172.21.15.37:0/2779623603 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65ec018070 con 0x7f65fc0a0260 2024-04-26T00:32:54.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:54.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65eaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:32:54.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65eaffd640 10 monclient: dump: 2024-04-26T00:32:54.705 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:54.705 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:54.705 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:54.705 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:54.705 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:54.705 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:54.705 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:54.705 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:54.705 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:54.705 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:54.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65eaffd640 10 monclient: _finish_auth 0 2024-04-26T00:32:54.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65eaffd640 10 monclient: _check_auth_tickets 2024-04-26T00:32:54.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:54.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65eaffd640 1 -- 172.21.15.37:0/2779623603 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65ec0070d0 con 0x7f65fc0a0260 2024-04-26T00:32:54.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65eaffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:54.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65eaffd640 1 -- 172.21.15.37:0/2779623603 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65ec0074c0 con 0x7f65fc0a0260 2024-04-26T00:32:54.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f66009b5640 5 monclient: authenticate success, global_id 144225 2024-04-26T00:32:54.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f66009b5640 10 monclient: _renew_subs 2024-04-26T00:32:54.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f66009b5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:54.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f66009b5640 1 -- 172.21.15.37:0/2779623603 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65fc120190 con 0x7f65fc0a0260 2024-04-26T00:32:54.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:54.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65ebfff640 4 set_mon_vals no callback set 2024-04-26T00:32:54.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:54.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f65eaffd640 10 monclient: dump: 2024-04-26T00:32:54.706 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:54.706 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:54.706 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:54.706 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:54.706 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:54.706 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:54.707 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:54.707 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:54.707 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:54.707 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:54.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f66009b5640 10 monclient: _renew_subs 2024-04-26T00:32:54.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f66009b5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:54.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f66009b5640 1 -- 172.21.15.37:0/2779623603 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65fc11e780 con 0x7f65fc0a0260 2024-04-26T00:32:54.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.699+0000 7f66009b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:54.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.703+0000 7f66009b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:54.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.703+0000 7f66009b5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:54.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.703+0000 7f65eaffd640 1 -- 172.21.15.37:0/2779623603 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f65ec01b070 con 0x7f65fc0a0260 2024-04-26T00:32:54.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.703+0000 7f66009b5640 1 -- 172.21.15.37:0/2779623603 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65fc066f40 con 0x7f65fc0a0260 2024-04-26T00:32:54.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.703+0000 7f65eaffd640 1 --2- 172.21.15.37:0/2779623603 >> v2:172.21.15.45:6812/2379918623 conn(0x7f65dc035e80 0x7f65dc038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:54.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.703+0000 7f65eaffd640 1 -- 172.21.15.37:0/2779623603 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(503..503 src has 370..503) ==== 3915+0+0 (secure 0 0 0) 0x7f65ec047e70 con 0x7f65fc0a0260 2024-04-26T00:32:54.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.703+0000 7f65f9d9b640 1 --2- 172.21.15.37:0/2779623603 >> v2:172.21.15.45:6812/2379918623 conn(0x7f65dc035e80 0x7f65dc038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:54.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.703+0000 7f65f9d9b640 10 monclient: get_auth_request con 0x7f65dc035e80 auth_method 0 2024-04-26T00:32:54.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.703+0000 7f65f9d9b640 1 --2- 172.21.15.37:0/2779623603 >> v2:172.21.15.45:6812/2379918623 conn(0x7f65dc035e80 0x7f65dc038340 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f65fc11daa0 tx=0x7f65f0007ce0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:54.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.707+0000 7f65eaffd640 1 -- 172.21.15.37:0/2779623603 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f65fc0a1130 con 0x7f65fc0a0260 2024-04-26T00:32:54.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.707+0000 7f65eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:54.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.707+0000 7f65eaffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:32:54.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.927+0000 7f66009b5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:32:54.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.927+0000 7f66009b5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:32:54.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.927+0000 7f66009b5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:54.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.927+0000 7f66009b5640 1 -- 172.21.15.37:0/2779623603 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f65fc11f9d0 con 0x7f65fc0a0260 2024-04-26T00:32:54.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.931+0000 7f65eaffd640 1 -- 172.21.15.37:0/2779623603 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f65ec011360 con 0x7f65fc0a0260 2024-04-26T00:32:54.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.931+0000 7f65eaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:32:54.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.931+0000 7f65eaffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:32:54.927 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":96,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:32:54.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.931+0000 7f66009b5640 1 -- 172.21.15.37:0/2779623603 >> v2:172.21.15.45:6812/2379918623 conn(0x7f65dc035e80 msgr2=0x7f65dc038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:54.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.931+0000 7f66009b5640 1 --2- 172.21.15.37:0/2779623603 >> v2:172.21.15.45:6812/2379918623 conn(0x7f65dc035e80 0x7f65dc038340 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f65fc11daa0 tx=0x7f65f0007ce0 comp rx=0 tx=0).stop 2024-04-26T00:32:54.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.931+0000 7f66009b5640 10 monclient: shutdown 2024-04-26T00:32:54.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.931+0000 7f66009b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:54.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.931+0000 7f66009b5640 1 -- 172.21.15.37:0/2779623603 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65fc0a0260 msgr2=0x7f65fc136810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:54.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.931+0000 7f66009b5640 1 --2- 172.21.15.37:0/2779623603 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65fc0a0260 0x7f65fc136810 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f65ec00c500 tx=0x7f65ec00c9c0 comp rx=0 tx=0).stop 2024-04-26T00:32:54.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.931+0000 7f66009b5640 1 -- 172.21.15.37:0/2779623603 shutdown_connections 2024-04-26T00:32:54.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.931+0000 7f66009b5640 1 --2- 172.21.15.37:0/2779623603 >> v2:172.21.15.45:6812/2379918623 conn(0x7f65dc035e80 0x7f65dc038340 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:54.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.931+0000 7f66009b5640 1 --2- 172.21.15.37:0/2779623603 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65fc13b950 0x7f65fc11c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:54.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.931+0000 7f66009b5640 1 --2- 172.21.15.37:0/2779623603 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f65fc137e40 0x7f65fc11c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:54.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.931+0000 7f66009b5640 1 --2- 172.21.15.37:0/2779623603 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65fc0a0260 0x7f65fc136810 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:54.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.931+0000 7f66009b5640 1 -- 172.21.15.37:0/2779623603 >> 172.21.15.37:0/2779623603 conn(0x7f65fc0a1ad0 msgr2=0x7f65fc13c140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:54.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.931+0000 7f66009b5640 1 -- 172.21.15.37:0/2779623603 shutdown_connections 2024-04-26T00:32:54.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:54.931+0000 7f66009b5640 1 -- 172.21.15.37:0/2779623603 wait complete. 2024-04-26T00:32:54.945 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:32:54.946 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-26T00:32:54.946 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:32:54.946 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:32:55.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.075+0000 7f228a8ab640 10 monclient: get_monmap_and_config 2024-04-26T00:32:55.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.075+0000 7f228a8ab640 10 monclient: build_initial_monmap 2024-04-26T00:32:55.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.075+0000 7f228a8ab640 10 monclient: monmap: 2024-04-26T00:32:55.071 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:55.071 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:55.072 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:32:55.080596+0000 2024-04-26T00:32:55.072 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:32:55.080596+0000 2024-04-26T00:32:55.072 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:55.072 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:55.072 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:32:55.072 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:32:55.072 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:32:55.072 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:32:55.072 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:32:55.072 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:55.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.075+0000 7f228a8ab640 1 Processor -- start 2024-04-26T00:32:55.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.075+0000 7f228a8ab640 1 -- start start 2024-04-26T00:32:55.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.075+0000 7f228a8ab640 10 monclient: init 2024-04-26T00:32:55.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.075+0000 7f228a8ab640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:55.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.075+0000 7f228a8ab640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-04-26T00:32:55.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.075+0000 7f228a8ab640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f22840a3d90 0x7f228409bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:55.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.075+0000 7f228a8ab640 10 monclient(hunting): picked mon.noname-c con 0x7f22840a3d90 addr v2:172.21.15.45:3300/0 2024-04-26T00:32:55.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.075+0000 7f228a8ab640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f22840a4170 addr v1:172.21.15.37:6789/0 2024-04-26T00:32:55.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.075+0000 7f228a8ab640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f228413c3e0 addr v1:172.21.15.45:6789/0 2024-04-26T00:32:55.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.075+0000 7f228a8ab640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22840a5150 con 0x7f22840a4170 2024-04-26T00:32:55.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.075+0000 7f228a8ab640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f228409b280 con 0x7f228413c3e0 2024-04-26T00:32:55.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.075+0000 7f228a8ab640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:55.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.075+0000 7f228a8ab640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f228409b560 con 0x7f22840a3d90 2024-04-26T00:32:55.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.075+0000 7f228a8ab640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:55.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.075+0000 7f228a8ab640 10 monclient(hunting): authenticate will time out at 1756.386427s 2024-04-26T00:32:55.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.075+0000 7f22837fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f22840a4170 0x7f22840a45e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:48964/0 (socket says 172.21.15.37:48964) 2024-04-26T00:32:55.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.075+0000 7f22837fe640 1 -- 172.21.15.37:0/2256663506 learned_addr learned my addr 172.21.15.37:0/2256663506 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:32:55.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.075+0000 7f2283fff640 1 --2- 172.21.15.37:0/2256663506 >> v2:172.21.15.45:3300/0 conn(0x7f22840a3d90 0x7f228409bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:55.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.075+0000 7f2283fff640 10 monclient(hunting): get_auth_request con 0x7f22840a3d90 auth_method 0 2024-04-26T00:32:55.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.075+0000 7f2283fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:55.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.075+0000 7f2283fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:55.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.075+0000 7f2283fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:55.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.075+0000 7f2283fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:55.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.075+0000 7f2283fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:55.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.075+0000 7f2283fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:55.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.075+0000 7f2283fff640 10 monclient(hunting): handle_auth_done global_id 134293 payload 274 2024-04-26T00:32:55.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.075+0000 7f2283fff640 1 -- 172.21.15.37:0/2256663506 >> v1:172.21.15.45:6789/0 conn(0x7f228413c3e0 legacy=0x7f228413e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:55.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.075+0000 7f2283fff640 1 -- 172.21.15.37:0/2256663506 >> v1:172.21.15.37:6789/0 conn(0x7f22840a4170 legacy=0x7f22840a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:55.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.075+0000 7f2283fff640 10 monclient: _finish_hunting 0 2024-04-26T00:32:55.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.075+0000 7f2283fff640 1 monclient: found mon.noname-c 2024-04-26T00:32:55.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.075+0000 7f2283fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:55.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.075+0000 7f2283fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:32:55.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.075+0000 7f2283fff640 1 -- 172.21.15.37:0/2256663506 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f228406dc60 con 0x7f22840a3d90 2024-04-26T00:32:55.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f2283fff640 1 -- 172.21.15.37:0/2256663506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22840a3d90 msgr2=0x7f228409bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:55.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f2283fff640 1 --2- 172.21.15.37:0/2256663506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22840a3d90 0x7f228409bf40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f227c009d90 tx=0x7f227c01c7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7d77e814a094da4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:55.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f2282ffd640 1 -- 172.21.15.37:0/2256663506 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f227c0b2070 con 0x7f22840a3d90 2024-04-26T00:32:55.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f2282ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:55.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f2282ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:32:55.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f2282ffd640 10 monclient: dump: 2024-04-26T00:32:55.080 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:55.080 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:55.080 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.080 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.080 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:55.080 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:55.080 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:55.080 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:55.080 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:55.080 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:55.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f2282ffd640 10 monclient: _finish_auth 0 2024-04-26T00:32:55.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f2282ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:32:55.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f2282ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:55.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f2282ffd640 1 -- 172.21.15.37:0/2256663506 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f227c0a5860 con 0x7f22840a3d90 2024-04-26T00:32:55.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f228a8ab640 5 monclient: authenticate success, global_id 134293 2024-04-26T00:32:55.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f228a8ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:32:55.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f2282ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:55.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f228a8ab640 10 monclient: get_monmap_and_config success 2024-04-26T00:32:55.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f228a8ab640 4 set_mon_vals no callback set 2024-04-26T00:32:55.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f2282ffd640 1 -- 172.21.15.37:0/2256663506 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f227c0a5c30 con 0x7f22840a3d90 2024-04-26T00:32:55.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f2282ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:55.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f2282ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:32:55.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f2282ffd640 10 monclient: dump: 2024-04-26T00:32:55.081 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:55.081 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:55.081 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.082 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.082 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:55.082 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:55.082 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:55.082 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:55.082 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:55.082 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:55.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f228a8ab640 10 monclient: shutdown 2024-04-26T00:32:55.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f228a8ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:55.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f228a8ab640 1 -- 172.21.15.37:0/2256663506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22840a3d90 msgr2=0x7f228409bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:55.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f228a8ab640 1 --2- 172.21.15.37:0/2256663506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22840a3d90 0x7f228409bf40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f227c009d90 tx=0x7f227c01c7b0 comp rx=0 tx=0).stop 2024-04-26T00:32:55.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f228a8ab640 1 -- 172.21.15.37:0/2256663506 shutdown_connections 2024-04-26T00:32:55.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f228a8ab640 1 --2- 172.21.15.37:0/2256663506 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22840a3d90 0x7f228409bf40 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:55.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f228a8ab640 1 -- 172.21.15.37:0/2256663506 >> 172.21.15.37:0/2256663506 conn(0x7f22840aa0a0 msgr2=0x7f22840a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:55.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f228a8ab640 1 -- 172.21.15.37:0/2256663506 shutdown_connections 2024-04-26T00:32:55.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f228a8ab640 1 -- 172.21.15.37:0/2256663506 wait complete. 2024-04-26T00:32:55.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f228a8ab640 10 monclient: build_initial_monmap 2024-04-26T00:32:55.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f228a8ab640 10 monclient: monmap: 2024-04-26T00:32:55.089 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:55.089 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:32:55.089 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:32:55.089 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:32:55.089 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:55.089 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:32:55.089 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:32:55.089 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:32:55.089 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:32:55.090 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:55.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f228a8ab640 1 Processor -- start 2024-04-26T00:32:55.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f228a8ab640 1 -- start start 2024-04-26T00:32:55.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f228a8ab640 10 monclient: init 2024-04-26T00:32:55.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f228a8ab640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:55.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f228a8ab640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:32:55.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f228a8ab640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22840a3d90 0x7f22841ceed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:55.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f228a8ab640 10 monclient(hunting): picked mon.noname-b con 0x7f22840a3d90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:32:55.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f228a8ab640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f22840a4170 0x7f2284137110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:55.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f228a8ab640 10 monclient(hunting): picked mon.noname-a con 0x7f22840a4170 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:32:55.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f228a8ab640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f228413c3e0 0x7f2284139200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:55.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f228a8ab640 10 monclient(hunting): picked mon.noname-c con 0x7f228413c3e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:55.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f228a8ab640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:55.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f228a8ab640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f2284142920 con 0x7f22840a4170 2024-04-26T00:32:55.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f228a8ab640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:55.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f228a8ab640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f228409c690 con 0x7f22840a3d90 2024-04-26T00:32:55.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f228a8ab640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:55.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f228a8ab640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2284142aa0 con 0x7f228413c3e0 2024-04-26T00:32:55.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f228a8ab640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:55.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f228a8ab640 10 monclient(hunting): authenticate will time out at 1756.394768s 2024-04-26T00:32:55.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f22837fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f22840a4170 0x7f2284137110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:55.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f2288e48640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f228413c3e0 0x7f2284139200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:55.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f22837fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f22840a4170 0x7f2284137110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44732/0 (socket says 172.21.15.37:44732) 2024-04-26T00:32:55.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f22837fe640 1 -- 172.21.15.37:0/2058210153 learned_addr learned my addr 172.21.15.37:0/2058210153 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:32:55.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f22837fe640 10 monclient(hunting): get_auth_request con 0x7f22840a4170 auth_method 0 2024-04-26T00:32:55.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f22837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:55.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f22837fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:55.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f22837fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:55.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f2288e48640 10 monclient(hunting): get_auth_request con 0x7f228413c3e0 auth_method 0 2024-04-26T00:32:55.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f2288e48640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:55.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f2288e48640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:55.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f2288e48640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:55.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f22837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:55.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f22837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:55.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f22837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:55.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f2288e48640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:55.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f2288e48640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:55.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f2288e48640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:55.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f2283fff640 1 --2- 172.21.15.37:0/2058210153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22840a3d90 0x7f22841ceed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:55.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f2283fff640 10 monclient(hunting): get_auth_request con 0x7f22840a3d90 auth_method 0 2024-04-26T00:32:55.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f2283fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:55.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f2283fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:55.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f2283fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:55.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f22837fe640 1 -- 172.21.15.37:0/2058210153 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f22840a4170 msgr2=0x7f2284137110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:32:55.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f22837fe640 1 -- 172.21.15.37:0/2058210153 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f22840a4170 msgr2=0x7f2284137110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:32:55.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f2288e48640 10 monclient(hunting): handle_auth_done global_id 154115 payload 824 2024-04-26T00:32:55.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f22837fe640 1 --2- 172.21.15.37:0/2058210153 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f22840a4170 0x7f2284137110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:32:55.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f2288e48640 1 -- 172.21.15.37:0/2058210153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22840a3d90 msgr2=0x7f22841ceed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:55.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f2288e48640 1 --2- 172.21.15.37:0/2058210153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f22840a3d90 0x7f22841ceed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:55.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f2288e48640 1 -- 172.21.15.37:0/2058210153 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f22840a4170 msgr2=0x7f2284137110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:55.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f22837fe640 1 --2- 172.21.15.37:0/2058210153 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f22840a4170 0x7f2284137110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:32:55.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f2288e48640 1 --2- 172.21.15.37:0/2058210153 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f22840a4170 0x7f2284137110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:55.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f2288e48640 10 monclient: _finish_hunting 0 2024-04-26T00:32:55.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f2288e48640 1 monclient: found mon.noname-c 2024-04-26T00:32:55.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f2288e48640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:55.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f2288e48640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:32:55.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.083+0000 7f2288e48640 1 -- 172.21.15.37:0/2058210153 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22841cfde0 con 0x7f228413c3e0 2024-04-26T00:32:55.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f2288e48640 1 -- 172.21.15.37:0/2058210153 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f228413c3e0 msgr2=0x7f2284139200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:55.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f2288e48640 1 --2- 172.21.15.37:0/2058210153 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f228413c3e0 0x7f2284139200 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f227400c410 tx=0x7f227400c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:55.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f2280ff9640 1 -- 172.21.15.37:0/2058210153 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2274017070 con 0x7f228413c3e0 2024-04-26T00:32:55.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f2280ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:55.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f2280ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:32:55.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f2280ff9640 10 monclient: dump: 2024-04-26T00:32:55.095 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:55.095 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:55.095 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.095 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.095 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:55.095 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:55.095 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:55.095 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:55.095 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:55.095 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:55.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f2280ff9640 10 monclient: _finish_auth 0 2024-04-26T00:32:55.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f2280ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:32:55.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f2280ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:55.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f228a8ab640 5 monclient: authenticate success, global_id 154115 2024-04-26T00:32:55.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f228a8ab640 10 monclient: _renew_subs 2024-04-26T00:32:55.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f228a8ab640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:32:55.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f228a8ab640 1 -- 172.21.15.37:0/2058210153 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2284120a90 con 0x7f228413c3e0 2024-04-26T00:32:55.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f2280ff9640 1 -- 172.21.15.37:0/2058210153 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2274005230 con 0x7f228413c3e0 2024-04-26T00:32:55.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f2280ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:55.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f2280ff9640 1 -- 172.21.15.37:0/2058210153 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22740117a0 con 0x7f228413c3e0 2024-04-26T00:32:55.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f2280ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:55.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f2280ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:32:55.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f2280ff9640 10 monclient: dump: 2024-04-26T00:32:55.096 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:55.096 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:55.096 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.096 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.096 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:55.097 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:55.097 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:55.097 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:55.097 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:55.097 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:55.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f228a8ab640 10 monclient: _renew_subs 2024-04-26T00:32:55.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f228a8ab640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:32:55.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f228a8ab640 1 -- 172.21.15.37:0/2058210153 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f22841207b0 con 0x7f228413c3e0 2024-04-26T00:32:55.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f22827fc640 4 set_mon_vals no callback set 2024-04-26T00:32:55.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f228a8ab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:55.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f228a8ab640 1 --2- 172.21.15.37:0/2058210153 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2284139740 0x7f2284139b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:55.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f228a8ab640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:55.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f228a8ab640 1 -- 172.21.15.37:0/2058210153 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f228413a060 con 0x7f2284139740 2024-04-26T00:32:55.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f2283fff640 1 --2- 172.21.15.37:0/2058210153 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2284139740 0x7f2284139b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:55.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f2283fff640 10 monclient: get_auth_request con 0x7f2284139740 auth_method 0 2024-04-26T00:32:55.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f2283fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:55.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f2283fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:55.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f2283fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:55.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f2283fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:55.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f2283fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:55.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f2283fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:55.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f2280ff9640 1 -- 172.21.15.37:0/2058210153 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f227401a070 con 0x7f228413c3e0 2024-04-26T00:32:55.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f2283fff640 10 monclient(hunting): handle_auth_done global_id 144231 payload 824 2024-04-26T00:32:55.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f2280ff9640 1 --2- 172.21.15.37:0/2058210153 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2264035e80 0x7f2264038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:55.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f2283fff640 1 -- 172.21.15.37:0/2058210153 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f228404f510 con 0x7f2284139740 2024-04-26T00:32:55.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f2280ff9640 1 -- 172.21.15.37:0/2058210153 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(503..503 src has 370..503) ==== 3915+0+0 (secure 0 0 0) 0x7f2274047df0 con 0x7f228413c3e0 2024-04-26T00:32:55.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f22837fe640 1 --2- 172.21.15.37:0/2058210153 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2264035e80 0x7f2264038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:55.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.087+0000 7f22837fe640 10 monclient: get_auth_request con 0x7f2264035e80 auth_method 0 2024-04-26T00:32:55.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.091+0000 7f2283fff640 1 -- 172.21.15.37:0/2058210153 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2284139740 msgr2=0x7f2284139b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:55.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.091+0000 7f2283fff640 1 --2- 172.21.15.37:0/2058210153 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2284139740 0x7f2284139b20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f228409d140 tx=0x7f227c019550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:55.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.091+0000 7f2280ff9640 1 -- 172.21.15.37:0/2058210153 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f227c0bd070 con 0x7f2284139740 2024-04-26T00:32:55.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.091+0000 7f2280ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:55.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.091+0000 7f2280ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:55.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.091+0000 7f2280ff9640 10 monclient: dump: 2024-04-26T00:32:55.099 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:55.099 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:55.099 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.099 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.099 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:55.099 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:55.099 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:55.099 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:55.099 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:55.099 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:55.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.091+0000 7f2280ff9640 1 -- 172.21.15.37:0/2058210153 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f227c0b3040 con 0x7f2284139740 2024-04-26T00:32:55.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.091+0000 7f2280ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:55.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.091+0000 7f2280ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:32:55.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.091+0000 7f2280ff9640 1 -- 172.21.15.37:0/2058210153 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2284139740 msgr2=0x7f2284139b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:55.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.091+0000 7f2280ff9640 1 --2- 172.21.15.37:0/2058210153 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2284139740 0x7f2284139b20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f228409d140 tx=0x7f227c019550 comp rx=0 tx=0).stop 2024-04-26T00:32:55.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.091+0000 7f2280ff9640 1 -- 172.21.15.37:0/2058210153 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2284139740 msgr2=0x7f2284139b20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:32:55.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.091+0000 7f2280ff9640 1 --2- 172.21.15.37:0/2058210153 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2284139740 0x7f2284139b20 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:55.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.091+0000 7f22837fe640 1 --2- 172.21.15.37:0/2058210153 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2264035e80 0x7f2264038340 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f226c007d60 tx=0x7f226c007cb0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:55.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.099+0000 7f228a8ab640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:32:55.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.099+0000 7f228a8ab640 1 --2- 172.21.15.37:0/2058210153 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f228409d140 0x7f228409d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:55.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.099+0000 7f228a8ab640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:55.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.099+0000 7f228a8ab640 1 -- 172.21.15.37:0/2058210153 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f228413bc20 con 0x7f228409d140 2024-04-26T00:32:55.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.099+0000 7f2283fff640 1 --2- 172.21.15.37:0/2058210153 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f228409d140 0x7f228409d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:55.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.099+0000 7f2283fff640 10 monclient: get_auth_request con 0x7f228409d140 auth_method 0 2024-04-26T00:32:55.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.099+0000 7f2283fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:55.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.099+0000 7f2283fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:55.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.099+0000 7f2283fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:55.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.099+0000 7f2283fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:55.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.099+0000 7f2283fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:55.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.099+0000 7f2283fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:55.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.099+0000 7f2283fff640 10 monclient(hunting): handle_auth_done global_id 144234 payload 824 2024-04-26T00:32:55.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.099+0000 7f2283fff640 1 -- 172.21.15.37:0/2058210153 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f228411fda0 con 0x7f228409d140 2024-04-26T00:32:55.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.099+0000 7f2283fff640 1 -- 172.21.15.37:0/2058210153 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f228409d140 msgr2=0x7f228409d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:55.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.099+0000 7f2283fff640 1 --2- 172.21.15.37:0/2058210153 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f228409d140 0x7f228409d550 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f227c0193c0 tx=0x7f227c0a7000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:55.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.099+0000 7f2280ff9640 1 -- 172.21.15.37:0/2058210153 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f227c0b2040 con 0x7f228409d140 2024-04-26T00:32:55.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.099+0000 7f2280ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:55.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.099+0000 7f2280ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:55.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.099+0000 7f2280ff9640 10 monclient: dump: 2024-04-26T00:32:55.103 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:55.103 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:55.103 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.103 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.103 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:55.104 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:55.104 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:55.104 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:55.104 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:55.104 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:55.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.099+0000 7f2280ff9640 1 -- 172.21.15.37:0/2058210153 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4718 (secure 0 0 0) 0x7f227c004240 con 0x7f228409d140 2024-04-26T00:32:55.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.099+0000 7f2280ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:32:55.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.099+0000 7f2280ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:32:55.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.099+0000 7f2280ff9640 1 -- 172.21.15.37:0/2058210153 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f228409d140 msgr2=0x7f228409d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:55.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.099+0000 7f2280ff9640 1 --2- 172.21.15.37:0/2058210153 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f228409d140 0x7f228409d550 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f227c0193c0 tx=0x7f227c0a7000 comp rx=0 tx=0).stop 2024-04-26T00:32:55.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.099+0000 7f2280ff9640 1 -- 172.21.15.37:0/2058210153 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f228409d140 msgr2=0x7f228409d550 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:32:55.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.099+0000 7f2280ff9640 1 --2- 172.21.15.37:0/2058210153 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f228409d140 0x7f228409d550 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f227c0193c0 tx=0x7f227c0a7000 comp rx=0 tx=0).stop 2024-04-26T00:32:55.105 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:32:55.105 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:32:55.105 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:32:55.105 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:32:55.105 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 96, 2024-04-26T00:32:55.105 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 61643, 2024-04-26T00:32:55.105 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:32:55.105 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:32:55.105 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:32:55.106 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:32:55.106 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:55.106 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 7, 2024-04-26T00:32:55.106 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:32:55.106 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:32:55.106 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:32:55.106 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:55.106 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:55.106 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:55.106 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:55.106 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:55.106 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:55.106 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:55.106 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:55.107 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:55.107 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:55.107 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:55.107 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:55.107 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:32:55.107 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:32:55.107 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:55.107 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:55.107 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:55.107 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:55.107 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:55.107 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:55.107 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:55.107 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:55.108 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:55.108 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:55.108 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:55.108 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:55.108 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:55.108 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:32:55.108 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:32:55.108 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:32:55.108 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:32:55.108 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:32:55.108 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:32:55.108 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:32:55.108 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:32:55.108 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:32:55.108 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:32:55.109 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:32:55.110 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:32:55.110 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:32:55.110 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:32:55.110 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:32:55.110 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:32:55.110 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:32:55.110 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:55.110 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:55.110 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:55.110 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:55.110 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:55.110 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:55.110 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:55.111 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:55.111 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:55.111 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:55.111 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:55.111 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:55.111 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:32:55.112 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:55.112 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:32:55.112 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:55.112 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:32:55.112 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:32:55.112 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:55.112 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:55.112 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:55.112 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:55.112 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:32:55.112 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:55.112 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:55.113 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:55.113 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:55.113 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:32:55.113 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:55.113 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:55.113 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:55.113 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:55.113 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:32:55.113 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:32:55.113 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:55.113 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:55.113 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:55.113 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:55.113 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:55.114 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:32:55.114 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:32:55.114 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:55.114 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:55.114 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:55.114 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:55.114 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:32:55.114 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:55.114 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:55.114 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:55.114 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:55.114 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:32:55.114 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:55.114 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:55.115 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:55.115 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:55.115 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:32:55.115 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:32:55.115 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:55.115 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:55.115 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:55.115 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:55.115 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:55.115 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:32:55.115 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:32:55.116 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:55.116 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:55.116 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:55.116 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:55.116 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:32:55.116 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:55.116 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:55.116 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:55.116 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:55.116 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:32:55.116 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:55.116 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:55.116 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:55.116 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:55.116 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:32:55.117 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:32:55.117 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:55.117 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:55.117 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:55.117 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:55.117 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:55.117 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:55.117 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:32:55.117 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:32:55.117 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:55.117 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:55.117 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:55.117 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:32:55.117 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:55.117 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:55.118 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:32:55.118 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:55.118 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:55.118 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:55.118 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:32:55.118 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:55.118 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:55.118 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:32:55.118 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:55.118 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:55.118 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:55.119 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:32:55.119 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:55.119 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:55.119 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:55.119 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:32:55.119 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:32:55.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.099+0000 7f228a8ab640 1 -- 172.21.15.37:0/2058210153 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2264035e80 msgr2=0x7f2264038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:55.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.099+0000 7f228a8ab640 1 --2- 172.21.15.37:0/2058210153 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2264035e80 0x7f2264038340 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f226c007d60 tx=0x7f226c007cb0 comp rx=0 tx=0).stop 2024-04-26T00:32:55.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.099+0000 7f228a8ab640 10 monclient: shutdown 2024-04-26T00:32:55.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.099+0000 7f228a8ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:55.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.099+0000 7f228a8ab640 1 -- 172.21.15.37:0/2058210153 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f228413c3e0 msgr2=0x7f2284139200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:55.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.099+0000 7f228a8ab640 1 --2- 172.21.15.37:0/2058210153 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f228413c3e0 0x7f2284139200 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f227400c410 tx=0x7f227400c8d0 comp rx=0 tx=0).stop 2024-04-26T00:32:55.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.099+0000 7f2288e48640 1 -- 172.21.15.37:0/2058210153 reap_dead start 2024-04-26T00:32:55.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.099+0000 7f2288e48640 1 -- 172.21.15.37:0/2058210153 reap_dead start 2024-04-26T00:32:55.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.099+0000 7f228a8ab640 1 -- 172.21.15.37:0/2058210153 shutdown_connections 2024-04-26T00:32:55.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.099+0000 7f228a8ab640 1 -- 172.21.15.37:0/2058210153 >> 172.21.15.37:0/2058210153 conn(0x7f22840aa0a0 msgr2=0x7f22840a2580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:55.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.099+0000 7f228a8ab640 1 -- 172.21.15.37:0/2058210153 shutdown_connections 2024-04-26T00:32:55.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.099+0000 7f228a8ab640 1 -- 172.21.15.37:0/2058210153 wait complete. 2024-04-26T00:32:55.121 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:32:55.121 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:32:55.122 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:32:55.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.243+0000 7f147d30b640 10 monclient: get_monmap_and_config 2024-04-26T00:32:55.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.243+0000 7f147d30b640 10 monclient: build_initial_monmap 2024-04-26T00:32:55.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.243+0000 7f147d30b640 10 monclient: monmap: 2024-04-26T00:32:55.239 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:55.239 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:55.239 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:32:55.248145+0000 2024-04-26T00:32:55.239 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:32:55.248145+0000 2024-04-26T00:32:55.239 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:55.239 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:55.239 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:32:55.239 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:32:55.239 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:32:55.240 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:32:55.240 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:32:55.240 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:55.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.243+0000 7f147d30b640 1 Processor -- start 2024-04-26T00:32:55.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.243+0000 7f147d30b640 1 -- start start 2024-04-26T00:32:55.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.243+0000 7f147d30b640 10 monclient: init 2024-04-26T00:32:55.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.243+0000 7f147d30b640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:55.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.243+0000 7f147d30b640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-04-26T00:32:55.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.243+0000 7f147d30b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1478136f20 addr v1:172.21.15.37:6789/0 2024-04-26T00:32:55.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.243+0000 7f147d30b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1478137a10 0x7f147809bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:55.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.243+0000 7f147d30b640 10 monclient(hunting): picked mon.noname-c con 0x7f1478137a10 addr v2:172.21.15.45:3300/0 2024-04-26T00:32:55.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.243+0000 7f147d30b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f147813c450 addr v1:172.21.15.45:6789/0 2024-04-26T00:32:55.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.243+0000 7f147d30b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14780a50f0 con 0x7f1478136f20 2024-04-26T00:32:55.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.243+0000 7f147d30b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f147809b330 con 0x7f147813c450 2024-04-26T00:32:55.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.243+0000 7f147d30b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:55.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.243+0000 7f147d30b640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f147809b610 con 0x7f1478137a10 2024-04-26T00:32:55.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.243+0000 7f147d30b640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:55.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.243+0000 7f147d30b640 10 monclient(hunting): authenticate will time out at 1756.553775s 2024-04-26T00:32:55.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.243+0000 7f1476ffd640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f1478136f20 0x7f1478137300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:48972/0 (socket says 172.21.15.37:48972) 2024-04-26T00:32:55.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.243+0000 7f1476ffd640 1 -- 172.21.15.37:0/3608551396 learned_addr learned my addr 172.21.15.37:0/3608551396 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:32:55.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.243+0000 7f14767fc640 1 --2- 172.21.15.37:0/3608551396 >> v2:172.21.15.45:3300/0 conn(0x7f1478137a10 0x7f147809bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:55.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.243+0000 7f14767fc640 10 monclient(hunting): get_auth_request con 0x7f1478137a10 auth_method 0 2024-04-26T00:32:55.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.243+0000 7f14767fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:55.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.243+0000 7f14767fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:55.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.243+0000 7f14767fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:55.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.243+0000 7f14767fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:55.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.243+0000 7f14767fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:55.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.243+0000 7f14767fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:55.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.243+0000 7f14767fc640 10 monclient(hunting): handle_auth_done global_id 134299 payload 274 2024-04-26T00:32:55.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.243+0000 7f14767fc640 1 -- 172.21.15.37:0/3608551396 >> v1:172.21.15.45:6789/0 conn(0x7f147813c450 legacy=0x7f147813e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:55.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.243+0000 7f14767fc640 1 -- 172.21.15.37:0/3608551396 >> v1:172.21.15.37:6789/0 conn(0x7f1478136f20 legacy=0x7f1478137300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:55.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.243+0000 7f14767fc640 10 monclient: _finish_hunting 0 2024-04-26T00:32:55.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.243+0000 7f14767fc640 1 monclient: found mon.noname-c 2024-04-26T00:32:55.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.243+0000 7f14767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:55.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.243+0000 7f14767fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:32:55.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.243+0000 7f14767fc640 1 -- 172.21.15.37:0/3608551396 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f147809b8f0 con 0x7f1478137a10 2024-04-26T00:32:55.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.247+0000 7f14767fc640 1 -- 172.21.15.37:0/3608551396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1478137a10 msgr2=0x7f147809bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:55.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.247+0000 7f14767fc640 1 --2- 172.21.15.37:0/3608551396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1478137a10 0x7f147809bff0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f146c0097c0 tx=0x7f146c01c370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7086886e5eb5b71c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:55.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f1475ffb640 1 -- 172.21.15.37:0/3608551396 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f146c0b2070 con 0x7f1478137a10 2024-04-26T00:32:55.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f1475ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:55.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f1475ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:32:55.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f1475ffb640 10 monclient: dump: 2024-04-26T00:32:55.247 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:55.247 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:55.247 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.247 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.247 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:55.247 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:55.247 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:55.247 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:55.248 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:55.248 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:55.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f1475ffb640 10 monclient: _finish_auth 0 2024-04-26T00:32:55.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f1475ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:32:55.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f1475ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:55.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f1475ffb640 1 -- 172.21.15.37:0/3608551396 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f146c0ad640 con 0x7f1478137a10 2024-04-26T00:32:55.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f1475ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:55.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f1475ffb640 1 -- 172.21.15.37:0/3608551396 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f146c0ada10 con 0x7f1478137a10 2024-04-26T00:32:55.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f1475ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:55.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f1475ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:32:55.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f1475ffb640 10 monclient: dump: 2024-04-26T00:32:55.248 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:55.248 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:55.249 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.249 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.249 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:55.249 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:55.249 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:55.249 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:55.249 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:55.249 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:55.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f147d30b640 5 monclient: authenticate success, global_id 134299 2024-04-26T00:32:55.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f147d30b640 10 monclient: get_monmap_and_config success 2024-04-26T00:32:55.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f147d30b640 4 set_mon_vals no callback set 2024-04-26T00:32:55.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f147d30b640 10 monclient: shutdown 2024-04-26T00:32:55.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f147d30b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:55.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f147d30b640 1 -- 172.21.15.37:0/3608551396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1478137a10 msgr2=0x7f147809bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:55.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f147d30b640 1 --2- 172.21.15.37:0/3608551396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1478137a10 0x7f147809bff0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f146c0097c0 tx=0x7f146c01c370 comp rx=0 tx=0).stop 2024-04-26T00:32:55.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f147d30b640 1 -- 172.21.15.37:0/3608551396 shutdown_connections 2024-04-26T00:32:55.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f147d30b640 1 --2- 172.21.15.37:0/3608551396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1478137a10 0x7f147809bff0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:55.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f147d30b640 1 -- 172.21.15.37:0/3608551396 >> 172.21.15.37:0/3608551396 conn(0x7f14780a61b0 msgr2=0x7f14780a65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:55.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f147d30b640 1 -- 172.21.15.37:0/3608551396 shutdown_connections 2024-04-26T00:32:55.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f147d30b640 1 -- 172.21.15.37:0/3608551396 wait complete. 2024-04-26T00:32:55.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f147d30b640 10 monclient: build_initial_monmap 2024-04-26T00:32:55.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f147d30b640 10 monclient: monmap: 2024-04-26T00:32:55.255 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:55.256 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:32:55.256 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:32:55.256 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:32:55.256 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:55.256 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:32:55.256 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:32:55.256 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:32:55.256 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:32:55.256 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:55.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f147d30b640 1 Processor -- start 2024-04-26T00:32:55.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f147d30b640 1 -- start start 2024-04-26T00:32:55.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f147d30b640 10 monclient: init 2024-04-26T00:32:55.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f147d30b640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:55.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f147d30b640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:32:55.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f147d30b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1478136f20 0x7f1478137f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:55.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f147d30b640 10 monclient(hunting): picked mon.noname-a con 0x7f1478136f20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:32:55.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f147d30b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1478137a10 0x7f1478138470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:55.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f147d30b640 10 monclient(hunting): picked mon.noname-b con 0x7f1478137a10 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:32:55.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f147d30b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f147813c450 0x7f14781389b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:55.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f147d30b640 10 monclient(hunting): picked mon.noname-c con 0x7f147813c450 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:55.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f147d30b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:55.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f147d30b640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f1478142b10 con 0x7f1478136f20 2024-04-26T00:32:55.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f147d30b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:55.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f147d30b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1478142990 con 0x7f1478137a10 2024-04-26T00:32:55.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f147d30b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:55.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f147d30b640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f1478142c90 con 0x7f147813c450 2024-04-26T00:32:55.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f147d30b640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:55.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f147d30b640 10 monclient(hunting): authenticate will time out at 1756.563047s 2024-04-26T00:32:55.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f1476ffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1478136f20 0x7f1478137f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f1476ffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1478136f20 0x7f1478137f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44750/0 (socket says 172.21.15.37:44750) 2024-04-26T00:32:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f1476ffd640 1 -- 172.21.15.37:0/449888482 learned_addr learned my addr 172.21.15.37:0/449888482 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:32:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f1476ffd640 10 monclient(hunting): get_auth_request con 0x7f1478136f20 auth_method 0 2024-04-26T00:32:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f1476ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f1476ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f1476ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f14777fe640 1 --2- 172.21.15.37:0/449888482 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f147813c450 0x7f14781389b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f14777fe640 10 monclient(hunting): get_auth_request con 0x7f147813c450 auth_method 0 2024-04-26T00:32:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f14777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f14777fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f14777fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f1476ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f1476ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f1476ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:55.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f14767fc640 1 --2- 172.21.15.37:0/449888482 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1478137a10 0x7f1478138470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:55.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f14767fc640 10 monclient(hunting): get_auth_request con 0x7f1478137a10 auth_method 0 2024-04-26T00:32:55.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f14767fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:55.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f14767fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:55.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f14767fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:55.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f14777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:55.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f14777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:55.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f14777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:55.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f14767fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:55.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f14767fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:55.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f14767fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:55.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f1476ffd640 10 monclient(hunting): handle_auth_done global_id 144237 payload 824 2024-04-26T00:32:55.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f1476ffd640 1 -- 172.21.15.37:0/449888482 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f147813c450 msgr2=0x7f14781389b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:55.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f1476ffd640 1 --2- 172.21.15.37:0/449888482 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f147813c450 0x7f14781389b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:55.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f1476ffd640 1 -- 172.21.15.37:0/449888482 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1478137a10 msgr2=0x7f1478138470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:55.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f1476ffd640 1 --2- 172.21.15.37:0/449888482 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1478137a10 0x7f1478138470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:55.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f1476ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:32:55.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f1476ffd640 1 monclient: found mon.noname-a 2024-04-26T00:32:55.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f1476ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:55.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f1476ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:32:55.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.251+0000 7f1476ffd640 1 -- 172.21.15.37:0/449888482 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1478139470 con 0x7f1478136f20 2024-04-26T00:32:55.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.255+0000 7f1476ffd640 1 -- 172.21.15.37:0/449888482 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1478136f20 msgr2=0x7f1478137f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:55.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.255+0000 7f1476ffd640 1 --2- 172.21.15.37:0/449888482 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1478136f20 0x7f1478137f30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f146800c470 tx=0x7f146800c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:55.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.255+0000 7f145f7fe640 1 -- 172.21.15.37:0/449888482 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1468007070 con 0x7f1478136f20 2024-04-26T00:32:55.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.255+0000 7f145f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:55.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.255+0000 7f145f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:32:55.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.255+0000 7f145f7fe640 10 monclient: dump: 2024-04-26T00:32:55.262 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:55.262 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:55.262 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.262 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.262 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:55.262 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:55.262 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:55.262 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:55.262 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:55.262 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:55.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.255+0000 7f145f7fe640 10 monclient: _finish_auth 0 2024-04-26T00:32:55.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.255+0000 7f145f7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:32:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.255+0000 7f145f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.255+0000 7f147d30b640 5 monclient: authenticate success, global_id 144237 2024-04-26T00:32:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.255+0000 7f147d30b640 10 monclient: _renew_subs 2024-04-26T00:32:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.255+0000 7f145f7fe640 1 -- 172.21.15.37:0/449888482 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1468007210 con 0x7f1478136f20 2024-04-26T00:32:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.255+0000 7f147d30b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.255+0000 7f147d30b640 1 -- 172.21.15.37:0/449888482 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f147813a360 con 0x7f1478136f20 2024-04-26T00:32:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.255+0000 7f145f7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.255+0000 7f145f7fe640 1 -- 172.21.15.37:0/449888482 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1468007600 con 0x7f1478136f20 2024-04-26T00:32:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.255+0000 7f145f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.255+0000 7f145f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.255+0000 7f145f7fe640 10 monclient: dump: 2024-04-26T00:32:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:55.264 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.255+0000 7f14757fa640 4 set_mon_vals no callback set 2024-04-26T00:32:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.255+0000 7f147d30b640 10 monclient: _renew_subs 2024-04-26T00:32:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.255+0000 7f147d30b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.255+0000 7f147d30b640 1 -- 172.21.15.37:0/449888482 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f147809f3c0 con 0x7f1478136f20 2024-04-26T00:32:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.255+0000 7f147d30b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.255+0000 7f147d30b640 1 --2- 172.21.15.37:0/449888482 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f147813ba90 0x7f1478135560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:55.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.255+0000 7f147d30b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:55.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.255+0000 7f147d30b640 1 -- 172.21.15.37:0/449888482 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f147804f510 con 0x7f147813ba90 2024-04-26T00:32:55.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.255+0000 7f145f7fe640 1 -- 172.21.15.37:0/449888482 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f146801b400 con 0x7f1478136f20 2024-04-26T00:32:55.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.255+0000 7f14767fc640 1 --2- 172.21.15.37:0/449888482 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f147813ba90 0x7f1478135560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:55.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.255+0000 7f14767fc640 10 monclient: get_auth_request con 0x7f147813ba90 auth_method 0 2024-04-26T00:32:55.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.255+0000 7f14767fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:55.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.255+0000 7f14767fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:55.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.255+0000 7f14767fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:55.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.255+0000 7f14767fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:55.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.255+0000 7f14767fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:55.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.255+0000 7f14767fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:55.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.255+0000 7f145f7fe640 1 --2- 172.21.15.37:0/449888482 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1464035ed0 0x7f1464038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:55.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.255+0000 7f145f7fe640 1 -- 172.21.15.37:0/449888482 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(503..503 src has 370..503) ==== 3915+0+0 (secure 0 0 0) 0x7f1468048810 con 0x7f1478136f20 2024-04-26T00:32:55.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.255+0000 7f14767fc640 10 monclient(hunting): handle_auth_done global_id 154121 payload 824 2024-04-26T00:32:55.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.255+0000 7f14767fc640 1 -- 172.21.15.37:0/449888482 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f147813a810 con 0x7f147813ba90 2024-04-26T00:32:55.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.259+0000 7f14777fe640 1 --2- 172.21.15.37:0/449888482 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1464035ed0 0x7f1464038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:55.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.259+0000 7f14777fe640 10 monclient: get_auth_request con 0x7f1464035ed0 auth_method 0 2024-04-26T00:32:55.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.259+0000 7f14767fc640 1 -- 172.21.15.37:0/449888482 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f147813ba90 msgr2=0x7f1478135560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:55.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.259+0000 7f14767fc640 1 --2- 172.21.15.37:0/449888482 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f147813ba90 0x7f1478135560 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f146c0af000 tx=0x7f146c01abb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:55.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.259+0000 7f145f7fe640 1 -- 172.21.15.37:0/449888482 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f146c0bd070 con 0x7f147813ba90 2024-04-26T00:32:55.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.259+0000 7f145f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:55.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.259+0000 7f145f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:32:55.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.259+0000 7f145f7fe640 10 monclient: dump: 2024-04-26T00:32:55.268 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:55.268 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:55.268 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.269 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.269 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:55.269 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:55.269 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:55.269 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:55.269 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:55.269 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:55.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.259+0000 7f145f7fe640 1 -- 172.21.15.37:0/449888482 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f146c0b8640 con 0x7f147813ba90 2024-04-26T00:32:55.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.259+0000 7f145f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:55.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.259+0000 7f145f7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:32:55.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.259+0000 7f145f7fe640 1 -- 172.21.15.37:0/449888482 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f147813ba90 msgr2=0x7f1478135560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:55.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.259+0000 7f145f7fe640 1 --2- 172.21.15.37:0/449888482 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f147813ba90 0x7f1478135560 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f146c0af000 tx=0x7f146c01abb0 comp rx=0 tx=0).stop 2024-04-26T00:32:55.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.259+0000 7f145f7fe640 1 -- 172.21.15.37:0/449888482 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f147813ba90 msgr2=0x7f1478135560 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:32:55.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.259+0000 7f145f7fe640 1 --2- 172.21.15.37:0/449888482 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f147813ba90 0x7f1478135560 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:55.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.259+0000 7f14777fe640 1 --2- 172.21.15.37:0/449888482 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1464035ed0 0x7f1464038390 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1470005600 tx=0x7f1470003d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.267+0000 7f147d30b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:32:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.267+0000 7f147d30b640 1 --2- 172.21.15.37:0/449888482 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f147802c000 0x7f147802c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.267+0000 7f147d30b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.267+0000 7f147d30b640 1 -- 172.21.15.37:0/449888482 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f14780688c0 con 0x7f147802c000 2024-04-26T00:32:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.267+0000 7f14767fc640 1 --2- 172.21.15.37:0/449888482 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f147802c000 0x7f147802c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.267+0000 7f14767fc640 10 monclient: get_auth_request con 0x7f147802c000 auth_method 0 2024-04-26T00:32:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.267+0000 7f14767fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.267+0000 7f14767fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.267+0000 7f14767fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.267+0000 7f14767fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.267+0000 7f14767fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.267+0000 7f14767fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.267+0000 7f14767fc640 10 monclient(hunting): handle_auth_done global_id 154124 payload 824 2024-04-26T00:32:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.267+0000 7f14767fc640 1 -- 172.21.15.37:0/449888482 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f14780a1ff0 con 0x7f147802c000 2024-04-26T00:32:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.267+0000 7f14767fc640 1 -- 172.21.15.37:0/449888482 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f147802c000 msgr2=0x7f147802c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.267+0000 7f14767fc640 1 --2- 172.21.15.37:0/449888482 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f147802c000 0x7f147802c3e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f146c0097c0 tx=0x7f146c019110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.267+0000 7f145f7fe640 1 -- 172.21.15.37:0/449888482 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f146c0bd070 con 0x7f147802c000 2024-04-26T00:32:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.267+0000 7f145f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.267+0000 7f145f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:32:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.267+0000 7f145f7fe640 10 monclient: dump: 2024-04-26T00:32:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:55.272 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:55.272 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:55.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.267+0000 7f145f7fe640 1 -- 172.21.15.37:0/449888482 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7f146c003a30 con 0x7f147802c000 2024-04-26T00:32:55.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.267+0000 7f145f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:32:55.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.267+0000 7f145f7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:32:55.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.271+0000 7f145f7fe640 1 -- 172.21.15.37:0/449888482 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f147802c000 msgr2=0x7f147802c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:55.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.271+0000 7f145f7fe640 1 --2- 172.21.15.37:0/449888482 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f147802c000 0x7f147802c3e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f146c0097c0 tx=0x7f146c019110 comp rx=0 tx=0).stop 2024-04-26T00:32:55.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.271+0000 7f145f7fe640 1 -- 172.21.15.37:0/449888482 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f147802c000 msgr2=0x7f147802c3e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:32:55.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.271+0000 7f145f7fe640 1 --2- 172.21.15.37:0/449888482 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f147802c000 0x7f147802c3e0 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f146c0097c0 tx=0x7f146c019110 comp rx=0 tx=0).stop 2024-04-26T00:32:55.272 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:32:55.273 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:32:55.273 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:32:55.273 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:32:55.273 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 96, 2024-04-26T00:32:55.273 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 12787, 2024-04-26T00:32:55.273 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:32:55.273 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:32:55.273 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:32:55.273 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:32:55.273 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:55.273 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 7, 2024-04-26T00:32:55.273 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:32:55.273 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:32:55.274 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:32:55.274 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:55.274 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:55.274 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:55.274 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:55.274 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:55.274 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:55.274 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:55.274 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:55.274 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:55.274 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:55.274 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:55.274 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:55.274 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:32:55.275 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:32:55.275 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:55.275 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:55.275 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:55.275 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:55.275 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:55.275 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:55.275 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:55.275 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:55.275 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:55.275 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:55.275 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:55.275 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:55.276 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:55.276 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:32:55.276 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:32:55.276 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:32:55.276 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:32:55.276 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:32:55.276 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:32:55.276 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:32:55.276 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:32:55.276 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:32:55.276 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:32:55.276 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:32:55.276 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:32:55.277 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:32:55.277 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:32:55.277 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:32:55.277 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:32:55.277 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:32:55.277 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:55.277 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:55.277 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:55.277 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:55.277 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:55.277 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:55.277 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:55.277 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:55.277 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:55.278 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:55.278 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:55.278 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:55.278 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:32:55.278 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:55.278 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:32:55.278 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:55.278 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:32:55.278 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:32:55.278 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:55.278 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:55.278 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:55.278 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:55.278 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:32:55.279 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:55.279 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:55.279 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:55.279 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:55.279 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:32:55.279 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:55.279 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:55.279 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:55.279 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:55.279 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:32:55.279 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:32:55.279 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:55.279 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:55.279 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:55.280 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:55.280 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:55.280 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:32:55.280 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:32:55.280 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:55.280 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:55.280 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:55.280 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:55.280 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:32:55.280 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:55.280 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:55.280 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:55.280 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:55.280 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:32:55.281 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:55.281 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:55.281 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:55.281 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:55.281 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:32:55.281 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:32:55.281 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:55.281 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:55.281 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:55.281 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:55.281 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:55.281 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:32:55.281 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:32:55.281 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:55.282 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:55.282 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:55.282 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:55.282 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:32:55.282 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:55.282 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:55.282 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:55.282 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:55.282 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:32:55.282 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:55.282 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:55.282 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:55.283 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:55.283 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:32:55.283 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:32:55.283 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:55.283 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:55.283 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:55.283 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:55.283 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:55.283 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:55.283 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:32:55.283 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:32:55.283 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:55.283 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:55.283 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:55.284 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:32:55.284 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:55.284 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:55.284 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:32:55.284 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:55.284 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:55.284 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:55.284 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:32:55.284 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:55.284 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:55.284 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:55.284 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:32:55.284 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:32:55.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.271+0000 7f147d30b640 1 -- 172.21.15.37:0/449888482 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1464035ed0 msgr2=0x7f1464038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:55.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.271+0000 7f147d30b640 1 --2- 172.21.15.37:0/449888482 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1464035ed0 0x7f1464038390 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1470005600 tx=0x7f1470003d50 comp rx=0 tx=0).stop 2024-04-26T00:32:55.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.271+0000 7f147d30b640 10 monclient: shutdown 2024-04-26T00:32:55.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.271+0000 7f147d30b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:55.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.271+0000 7f147d30b640 1 -- 172.21.15.37:0/449888482 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1478136f20 msgr2=0x7f1478137f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:55.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.271+0000 7f147d30b640 1 --2- 172.21.15.37:0/449888482 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1478136f20 0x7f1478137f30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f146800c470 tx=0x7f146800c930 comp rx=0 tx=0).stop 2024-04-26T00:32:55.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.271+0000 7f14777fe640 1 -- 172.21.15.37:0/449888482 reap_dead start 2024-04-26T00:32:55.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.271+0000 7f14777fe640 1 -- 172.21.15.37:0/449888482 reap_dead start 2024-04-26T00:32:55.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.271+0000 7f147d30b640 1 -- 172.21.15.37:0/449888482 shutdown_connections 2024-04-26T00:32:55.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.271+0000 7f147d30b640 1 -- 172.21.15.37:0/449888482 >> 172.21.15.37:0/449888482 conn(0x7f14780a61b0 msgr2=0x7f14780a7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:55.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.271+0000 7f147d30b640 1 -- 172.21.15.37:0/449888482 shutdown_connections 2024-04-26T00:32:55.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.271+0000 7f147d30b640 1 -- 172.21.15.37:0/449888482 wait complete. 2024-04-26T00:32:55.288 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:32:55.288 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:32:55.288 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:32:55.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.411+0000 7f71fae0a640 10 monclient: get_monmap_and_config 2024-04-26T00:32:55.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.411+0000 7f71fae0a640 10 monclient: build_initial_monmap 2024-04-26T00:32:55.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.411+0000 7f71fae0a640 10 monclient: monmap: 2024-04-26T00:32:55.407 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:55.407 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:55.407 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:32:55.415957+0000 2024-04-26T00:32:55.407 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:32:55.415957+0000 2024-04-26T00:32:55.407 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:55.407 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:55.407 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:32:55.407 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:32:55.407 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:32:55.407 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:32:55.407 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:32:55.407 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:55.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.411+0000 7f71fae0a640 1 Processor -- start 2024-04-26T00:32:55.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.411+0000 7f71fae0a640 1 -- start start 2024-04-26T00:32:55.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.411+0000 7f71fae0a640 10 monclient: init 2024-04-26T00:32:55.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.411+0000 7f71fae0a640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:55.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.411+0000 7f71fae0a640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-04-26T00:32:55.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.411+0000 7f71fae0a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f71f40a0300 addr v1:172.21.15.45:6789/0 2024-04-26T00:32:55.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.411+0000 7f71fae0a640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f71f409fc50 0x7f71f4139ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:55.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.411+0000 7f71fae0a640 10 monclient(hunting): picked mon.noname-c con 0x7f71f409fc50 addr v2:172.21.15.45:3300/0 2024-04-26T00:32:55.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.411+0000 7f71fae0a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f71f413a400 addr v1:172.21.15.37:6789/0 2024-04-26T00:32:55.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.411+0000 7f71fae0a640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71f409c190 con 0x7f71f413a400 2024-04-26T00:32:55.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.411+0000 7f71fae0a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71f40a1110 con 0x7f71f40a0300 2024-04-26T00:32:55.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.411+0000 7f71fae0a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:55.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.411+0000 7f71fae0a640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f71f40a16d0 con 0x7f71f409fc50 2024-04-26T00:32:55.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.411+0000 7f71fae0a640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:55.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.411+0000 7f71fae0a640 10 monclient(hunting): authenticate will time out at 1756.721604s 2024-04-26T00:32:55.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.411+0000 7f71f93a7640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f71f413a400 0x7f71f413c7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:48988/0 (socket says 172.21.15.37:48988) 2024-04-26T00:32:55.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.411+0000 7f71f93a7640 1 -- 172.21.15.37:0/4008526283 learned_addr learned my addr 172.21.15.37:0/4008526283 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:32:55.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.411+0000 7f71f3fff640 1 --2- 172.21.15.37:0/4008526283 >> v2:172.21.15.45:3300/0 conn(0x7f71f409fc50 0x7f71f4139ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:55.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.411+0000 7f71f3fff640 10 monclient(hunting): get_auth_request con 0x7f71f409fc50 auth_method 0 2024-04-26T00:32:55.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.411+0000 7f71f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:55.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.411+0000 7f71f3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:55.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.411+0000 7f71f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:55.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.411+0000 7f71f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:55.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.411+0000 7f71f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:55.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.411+0000 7f71f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:55.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.411+0000 7f71f3fff640 10 monclient(hunting): handle_auth_done global_id 134305 payload 274 2024-04-26T00:32:55.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.411+0000 7f71f3fff640 1 -- 172.21.15.37:0/4008526283 >> v1:172.21.15.45:6789/0 conn(0x7f71f40a0300 legacy=0x7f71f40a06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:55.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.411+0000 7f71f3fff640 1 -- 172.21.15.37:0/4008526283 >> v1:172.21.15.37:6789/0 conn(0x7f71f413a400 legacy=0x7f71f413c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:55.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.411+0000 7f71f3fff640 10 monclient: _finish_hunting 0 2024-04-26T00:32:55.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.411+0000 7f71f3fff640 1 monclient: found mon.noname-c 2024-04-26T00:32:55.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.411+0000 7f71f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:55.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.411+0000 7f71f3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:32:55.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.411+0000 7f71f3fff640 1 -- 172.21.15.37:0/4008526283 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71f409f2a0 con 0x7f71f409fc50 2024-04-26T00:32:55.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.415+0000 7f71f3fff640 1 -- 172.21.15.37:0/4008526283 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71f409fc50 msgr2=0x7f71f4139ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:55.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.415+0000 7f71f3fff640 1 --2- 172.21.15.37:0/4008526283 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71f409fc50 0x7f71f4139ec0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f71f409f4d0 tx=0x7f71e801c370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae7b395ffdfd75ea server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:55.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f37fe640 1 -- 172.21.15.37:0/4008526283 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71e80b0070 con 0x7f71f409fc50 2024-04-26T00:32:55.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:55.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:32:55.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f37fe640 10 monclient: dump: 2024-04-26T00:32:55.415 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:55.415 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:55.415 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.415 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.415 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:55.415 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:55.415 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:55.415 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:55.415 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:55.415 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:55.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f37fe640 10 monclient: _finish_auth 0 2024-04-26T00:32:55.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f37fe640 10 monclient: _check_auth_tickets 2024-04-26T00:32:55.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:55.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f37fe640 1 -- 172.21.15.37:0/4008526283 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71e80ab600 con 0x7f71f409fc50 2024-04-26T00:32:55.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71fae0a640 5 monclient: authenticate success, global_id 134305 2024-04-26T00:32:55.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71fae0a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:32:55.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f37fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:55.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71fae0a640 10 monclient: get_monmap_and_config success 2024-04-26T00:32:55.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f37fe640 1 -- 172.21.15.37:0/4008526283 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71e80ab9d0 con 0x7f71f409fc50 2024-04-26T00:32:55.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71fae0a640 4 set_mon_vals no callback set 2024-04-26T00:32:55.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:55.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:32:55.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f37fe640 10 monclient: dump: 2024-04-26T00:32:55.416 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:55.416 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:55.416 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.416 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.416 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:55.417 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:55.417 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:55.417 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:55.417 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:55.417 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:55.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71fae0a640 10 monclient: shutdown 2024-04-26T00:32:55.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71fae0a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:55.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71fae0a640 1 -- 172.21.15.37:0/4008526283 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71f409fc50 msgr2=0x7f71f4139ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:55.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71fae0a640 1 --2- 172.21.15.37:0/4008526283 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71f409fc50 0x7f71f4139ec0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f71f409f4d0 tx=0x7f71e801c370 comp rx=0 tx=0).stop 2024-04-26T00:32:55.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71fae0a640 1 -- 172.21.15.37:0/4008526283 shutdown_connections 2024-04-26T00:32:55.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71fae0a640 1 --2- 172.21.15.37:0/4008526283 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71f409fc50 0x7f71f4139ec0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:55.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71fae0a640 1 -- 172.21.15.37:0/4008526283 >> 172.21.15.37:0/4008526283 conn(0x7f71f40a1d50 msgr2=0x7f71f40a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:55.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71fae0a640 1 -- 172.21.15.37:0/4008526283 shutdown_connections 2024-04-26T00:32:55.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71fae0a640 1 -- 172.21.15.37:0/4008526283 wait complete. 2024-04-26T00:32:55.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71fae0a640 10 monclient: build_initial_monmap 2024-04-26T00:32:55.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71fae0a640 10 monclient: monmap: 2024-04-26T00:32:55.427 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:55.428 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:32:55.428 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:32:55.428 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:32:55.428 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:55.428 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:32:55.428 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:32:55.428 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:32:55.429 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:32:55.429 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:55.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71fae0a640 1 Processor -- start 2024-04-26T00:32:55.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71fae0a640 1 -- start start 2024-04-26T00:32:55.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71fae0a640 10 monclient: init 2024-04-26T00:32:55.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71fae0a640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:55.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71fae0a640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:32:55.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71fae0a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f71f409fc50 0x7f71f411b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:55.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71fae0a640 10 monclient(hunting): picked mon.noname-a con 0x7f71f409fc50 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:32:55.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71fae0a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71f40a0300 0x7f71f4138fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:55.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71fae0a640 10 monclient(hunting): picked mon.noname-b con 0x7f71f40a0300 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:32:55.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71fae0a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f71f413a400 0x7f71f411c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:55.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71fae0a640 10 monclient(hunting): picked mon.noname-c con 0x7f71f413a400 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:55.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71fae0a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:55.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71fae0a640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f71f4140ac0 con 0x7f71f409fc50 2024-04-26T00:32:55.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71fae0a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:55.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71fae0a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f71f4140940 con 0x7f71f40a0300 2024-04-26T00:32:55.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71fae0a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:55.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71fae0a640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f71f4140c40 con 0x7f71f413a400 2024-04-26T00:32:55.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71fae0a640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:55.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71fae0a640 10 monclient(hunting): authenticate will time out at 1756.731063s 2024-04-26T00:32:55.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f8ba6640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f71f409fc50 0x7f71f411b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:55.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f8ba6640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f71f409fc50 0x7f71f411b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44752/0 (socket says 172.21.15.37:44752) 2024-04-26T00:32:55.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f93a7640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f71f413a400 0x7f71f411c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:55.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f8ba6640 1 -- 172.21.15.37:0/2848976317 learned_addr learned my addr 172.21.15.37:0/2848976317 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:32:55.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f8ba6640 10 monclient(hunting): get_auth_request con 0x7f71f409fc50 auth_method 0 2024-04-26T00:32:55.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f8ba6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:55.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f8ba6640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:55.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f8ba6640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:55.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f93a7640 10 monclient(hunting): get_auth_request con 0x7f71f413a400 auth_method 0 2024-04-26T00:32:55.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f93a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:55.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f93a7640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:55.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f93a7640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:55.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f8ba6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:55.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f8ba6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:55.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f8ba6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:55.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f93a7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:55.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f93a7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:55.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f93a7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:55.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f3fff640 1 --2- 172.21.15.37:0/2848976317 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71f40a0300 0x7f71f4138fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:55.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f3fff640 10 monclient(hunting): get_auth_request con 0x7f71f40a0300 auth_method 0 2024-04-26T00:32:55.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:55.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:55.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:55.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f8ba6640 10 monclient(hunting): handle_auth_done global_id 144240 payload 824 2024-04-26T00:32:55.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f8ba6640 1 -- 172.21.15.37:0/2848976317 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f71f413a400 msgr2=0x7f71f411c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:55.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f8ba6640 1 --2- 172.21.15.37:0/2848976317 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f71f413a400 0x7f71f411c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:55.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f8ba6640 1 -- 172.21.15.37:0/2848976317 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71f40a0300 msgr2=0x7f71f4138fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:55.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f8ba6640 1 --2- 172.21.15.37:0/2848976317 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71f40a0300 0x7f71f4138fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:55.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f8ba6640 10 monclient: _finish_hunting 0 2024-04-26T00:32:55.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f8ba6640 1 monclient: found mon.noname-a 2024-04-26T00:32:55.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f8ba6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:55.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f8ba6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:32:55.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f8ba6640 1 -- 172.21.15.37:0/2848976317 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71f413a990 con 0x7f71f409fc50 2024-04-26T00:32:55.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.419+0000 7f71f3fff640 1 --2- 172.21.15.37:0/2848976317 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71f40a0300 0x7f71f4138fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:32:55.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.423+0000 7f71f8ba6640 1 -- 172.21.15.37:0/2848976317 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f71f409fc50 msgr2=0x7f71f411b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:55.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.423+0000 7f71f8ba6640 1 --2- 172.21.15.37:0/2848976317 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f71f409fc50 0x7f71f411b8a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f71e400c450 tx=0x7f71e400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:55.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.423+0000 7f71f17fa640 1 -- 172.21.15.37:0/2848976317 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71e4012a40 con 0x7f71f409fc50 2024-04-26T00:32:55.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.423+0000 7f71f17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:55.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.423+0000 7f71f17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:32:55.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.423+0000 7f71f17fa640 10 monclient: dump: 2024-04-26T00:32:55.437 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:55.437 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:55.437 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.437 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.438 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:55.438 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:55.438 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:55.438 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:55.438 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:55.438 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:55.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.423+0000 7f71f17fa640 10 monclient: _finish_auth 0 2024-04-26T00:32:55.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.423+0000 7f71f17fa640 10 monclient: _check_auth_tickets 2024-04-26T00:32:55.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.423+0000 7f71f17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:55.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.423+0000 7f71f17fa640 1 -- 172.21.15.37:0/2848976317 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71e4017070 con 0x7f71f409fc50 2024-04-26T00:32:55.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.423+0000 7f71f17fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:55.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.423+0000 7f71f17fa640 1 -- 172.21.15.37:0/2848976317 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71e4012e40 con 0x7f71f409fc50 2024-04-26T00:32:55.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.423+0000 7f71f17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:55.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.423+0000 7f71f17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:55.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.423+0000 7f71f17fa640 10 monclient: dump: 2024-04-26T00:32:55.439 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:55.439 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:55.439 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.439 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.439 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:55.439 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:55.439 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:55.439 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:55.439 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:55.439 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:55.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.423+0000 7f71f2ffd640 4 set_mon_vals no callback set 2024-04-26T00:32:55.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.423+0000 7f71fae0a640 5 monclient: authenticate success, global_id 144240 2024-04-26T00:32:55.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.423+0000 7f71fae0a640 10 monclient: _renew_subs 2024-04-26T00:32:55.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.423+0000 7f71fae0a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:55.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.423+0000 7f71fae0a640 1 -- 172.21.15.37:0/2848976317 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71f409f550 con 0x7f71f409fc50 2024-04-26T00:32:55.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.423+0000 7f71fae0a640 10 monclient: _renew_subs 2024-04-26T00:32:55.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.423+0000 7f71fae0a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:55.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.423+0000 7f71fae0a640 1 -- 172.21.15.37:0/2848976317 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71f4135580 con 0x7f71f409fc50 2024-04-26T00:32:55.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.423+0000 7f71fae0a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:55.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.423+0000 7f71fae0a640 1 --2- 172.21.15.37:0/2848976317 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71f41cd9c0 0x7f71f402c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:55.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.423+0000 7f71fae0a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:55.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.423+0000 7f71fae0a640 1 -- 172.21.15.37:0/2848976317 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f71f411e560 con 0x7f71f41cd9c0 2024-04-26T00:32:55.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.423+0000 7f71f17fa640 1 -- 172.21.15.37:0/2848976317 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f71e401c460 con 0x7f71f409fc50 2024-04-26T00:32:55.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.423+0000 7f71f3fff640 1 --2- 172.21.15.37:0/2848976317 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71f41cd9c0 0x7f71f402c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:55.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.423+0000 7f71f3fff640 10 monclient: get_auth_request con 0x7f71f41cd9c0 auth_method 0 2024-04-26T00:32:55.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.423+0000 7f71f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:55.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.423+0000 7f71f3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:55.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.423+0000 7f71f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:55.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.427+0000 7f71f17fa640 1 --2- 172.21.15.37:0/2848976317 >> v2:172.21.15.45:6812/2379918623 conn(0x7f71e0035ed0 0x7f71e0038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:55.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.427+0000 7f71f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:55.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.427+0000 7f71f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:55.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.427+0000 7f71f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:55.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.427+0000 7f71f17fa640 1 -- 172.21.15.37:0/2848976317 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(503..503 src has 370..503) ==== 3915+0+0 (secure 0 0 0) 0x7f71e4048800 con 0x7f71f409fc50 2024-04-26T00:32:55.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.427+0000 7f71f3fff640 10 monclient(hunting): handle_auth_done global_id 134311 payload 824 2024-04-26T00:32:55.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.427+0000 7f71f3fff640 1 -- 172.21.15.37:0/2848976317 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f71f404f510 con 0x7f71f41cd9c0 2024-04-26T00:32:55.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.427+0000 7f71f93a7640 1 --2- 172.21.15.37:0/2848976317 >> v2:172.21.15.45:6812/2379918623 conn(0x7f71e0035ed0 0x7f71e0038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:55.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.427+0000 7f71f93a7640 10 monclient: get_auth_request con 0x7f71e0035ed0 auth_method 0 2024-04-26T00:32:55.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.427+0000 7f71f93a7640 1 -- 172.21.15.37:0/2848976317 >> v2:172.21.15.45:6812/2379918623 conn(0x7f71e0035ed0 msgr2=0x7f71e0038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:32:55.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.427+0000 7f71f93a7640 1 -- 172.21.15.37:0/2848976317 >> v2:172.21.15.45:6812/2379918623 conn(0x7f71e0035ed0 msgr2=0x7f71e0038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:32:55.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.427+0000 7f71f93a7640 1 --2- 172.21.15.37:0/2848976317 >> v2:172.21.15.45:6812/2379918623 conn(0x7f71e0035ed0 0x7f71e0038390 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f71ec0055b0 tx=0x7f71ec005540 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:32:55.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.427+0000 7f71f93a7640 1 --2- 172.21.15.37:0/2848976317 >> v2:172.21.15.45:6812/2379918623 conn(0x7f71e0035ed0 0x7f71e0038390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:32:55.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.427+0000 7f71f3fff640 1 -- 172.21.15.37:0/2848976317 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71f41cd9c0 msgr2=0x7f71f402c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:55.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.427+0000 7f71f3fff640 1 --2- 172.21.15.37:0/2848976317 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71f41cd9c0 0x7f71f402c000 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f71f406de30 tx=0x7f71e801abb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:55.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.427+0000 7f71f17fa640 1 -- 172.21.15.37:0/2848976317 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71e80ba070 con 0x7f71f41cd9c0 2024-04-26T00:32:55.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.427+0000 7f71f17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:55.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.427+0000 7f71f17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:32:55.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.427+0000 7f71f17fa640 10 monclient: dump: 2024-04-26T00:32:55.442 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:55.442 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:55.442 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.442 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.442 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:55.442 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:55.443 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:55.443 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:55.443 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:55.443 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:55.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.427+0000 7f71f17fa640 1 -- 172.21.15.37:0/2848976317 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f71e80be400 con 0x7f71f41cd9c0 2024-04-26T00:32:55.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.427+0000 7f71f17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:55.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.427+0000 7f71f17fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:32:55.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.427+0000 7f71f17fa640 1 -- 172.21.15.37:0/2848976317 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71f41cd9c0 msgr2=0x7f71f402c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:55.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.427+0000 7f71f17fa640 1 --2- 172.21.15.37:0/2848976317 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71f41cd9c0 0x7f71f402c000 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f71f406de30 tx=0x7f71e801abb0 comp rx=0 tx=0).stop 2024-04-26T00:32:55.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.427+0000 7f71f17fa640 1 -- 172.21.15.37:0/2848976317 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71f41cd9c0 msgr2=0x7f71f402c000 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:32:55.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.427+0000 7f71f17fa640 1 --2- 172.21.15.37:0/2848976317 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71f41cd9c0 0x7f71f402c000 secure :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f71f406de30 tx=0x7f71e801abb0 comp rx=0 tx=0).stop 2024-04-26T00:32:55.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.435+0000 7f71fae0a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:32:55.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.435+0000 7f71fae0a640 1 --2- 172.21.15.37:0/2848976317 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71f41cdda0 0x7f71f402c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:55.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.435+0000 7f71fae0a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:55.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.435+0000 7f71fae0a640 1 -- 172.21.15.37:0/2848976317 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f71f4139ff0 con 0x7f71f41cdda0 2024-04-26T00:32:55.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.435+0000 7f71f3fff640 1 --2- 172.21.15.37:0/2848976317 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71f41cdda0 0x7f71f402c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:55.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.435+0000 7f71f3fff640 10 monclient: get_auth_request con 0x7f71f41cdda0 auth_method 0 2024-04-26T00:32:55.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.435+0000 7f71f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:55.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.435+0000 7f71f3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:55.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.435+0000 7f71f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:55.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.435+0000 7f71f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:55.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.435+0000 7f71f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:55.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.435+0000 7f71f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:55.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.435+0000 7f71f3fff640 10 monclient(hunting): handle_auth_done global_id 134314 payload 824 2024-04-26T00:32:55.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.435+0000 7f71f3fff640 1 -- 172.21.15.37:0/2848976317 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f71f413d580 con 0x7f71f41cdda0 2024-04-26T00:32:55.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.439+0000 7f71f3fff640 1 -- 172.21.15.37:0/2848976317 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71f41cdda0 msgr2=0x7f71f402c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:55.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.439+0000 7f71f3fff640 1 --2- 172.21.15.37:0/2848976317 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71f41cdda0 0x7f71f402c540 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f71e8019fa0 tx=0x7f71e8019110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:55.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.439+0000 7f71f17fa640 1 -- 172.21.15.37:0/2848976317 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71e80ba040 con 0x7f71f41cdda0 2024-04-26T00:32:55.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.439+0000 7f71f17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:55.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.439+0000 7f71f17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:32:55.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.439+0000 7f71f17fa640 10 monclient: dump: 2024-04-26T00:32:55.447 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:55.447 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:55.447 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.447 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.447 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:55.447 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:55.447 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:55.447 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:55.447 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:55.447 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:55.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.439+0000 7f71f17fa640 1 -- 172.21.15.37:0/2848976317 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f71e80c9460 con 0x7f71f41cdda0 2024-04-26T00:32:55.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.439+0000 7f71f17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:32:55.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.439+0000 7f71f17fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:32:55.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.439+0000 7f71f17fa640 1 -- 172.21.15.37:0/2848976317 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71f41cdda0 msgr2=0x7f71f402c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:55.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.439+0000 7f71f17fa640 1 --2- 172.21.15.37:0/2848976317 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71f41cdda0 0x7f71f402c540 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f71e8019fa0 tx=0x7f71e8019110 comp rx=0 tx=0).stop 2024-04-26T00:32:55.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.439+0000 7f71f17fa640 1 -- 172.21.15.37:0/2848976317 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71f41cdda0 msgr2=0x7f71f402c540 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:32:55.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.439+0000 7f71f17fa640 1 --2- 172.21.15.37:0/2848976317 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71f41cdda0 0x7f71f402c540 secure :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f71e8019fa0 tx=0x7f71e8019110 comp rx=0 tx=0).stop 2024-04-26T00:32:55.448 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:32:55.448 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:32:55.448 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:32:55.448 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:32:55.448 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 96, 2024-04-26T00:32:55.448 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 105802, 2024-04-26T00:32:55.448 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:32:55.449 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:32:55.449 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:32:55.449 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:32:55.449 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:55.449 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 7, 2024-04-26T00:32:55.449 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:32:55.449 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:32:55.449 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:32:55.449 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:55.449 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:55.449 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:55.449 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:55.449 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:55.449 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:55.449 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:55.450 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:55.450 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:55.450 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:55.450 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:55.450 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:55.450 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:32:55.450 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:32:55.450 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:55.450 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:55.450 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:55.450 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:55.450 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:55.450 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:55.451 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:55.451 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:55.451 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:55.451 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:55.451 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:55.451 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:55.451 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:55.451 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:32:55.451 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:32:55.451 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:32:55.451 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:32:55.451 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:32:55.452 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:32:55.452 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:32:55.452 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:32:55.452 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:32:55.452 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:32:55.452 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:32:55.452 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:32:55.452 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:32:55.452 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:32:55.452 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:32:55.452 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:32:55.452 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:32:55.452 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:32:55.452 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:32:55.452 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:32:55.453 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:32:55.453 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:32:55.453 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:32:55.453 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:32:55.453 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:32:55.453 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:32:55.453 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:32:55.453 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:32:55.453 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:55.453 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:32:55.453 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:55.453 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:32:55.453 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:55.453 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:32:55.453 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:32:55.454 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:55.454 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:55.454 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:55.454 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:55.454 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:32:55.454 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:55.454 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:55.454 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:55.454 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:55.454 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:32:55.454 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:55.454 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:55.454 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:55.454 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:55.454 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:32:55.455 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:32:55.455 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:55.455 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:55.455 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:55.455 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:55.455 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:55.455 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:32:55.455 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:32:55.455 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:55.455 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:55.455 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:55.455 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:55.455 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:32:55.455 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:55.456 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:55.456 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:55.456 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:55.456 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:32:55.456 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:55.456 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:55.456 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:55.456 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:55.456 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:32:55.456 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:32:55.456 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:55.456 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:55.456 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:55.457 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:55.457 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:55.457 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:32:55.457 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:32:55.457 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:32:55.457 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:32:55.457 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:55.457 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:32:55.457 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:32:55.457 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:55.457 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:55.457 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:55.457 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:32:55.457 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:32:55.458 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:32:55.458 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:55.458 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:55.458 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:55.458 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:32:55.458 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:32:55.458 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:32:55.458 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:32:55.458 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:32:55.458 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:55.458 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:55.458 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:55.458 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:32:55.458 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:32:55.459 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:55.459 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:55.459 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:55.459 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:32:55.459 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:55.459 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:55.459 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:32:55.459 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:55.459 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:55.459 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:55.459 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:32:55.459 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:55.459 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:55.459 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:32:55.459 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:55.459 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:55.460 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:55.460 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:32:55.460 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:55.460 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:32:55.460 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:32:55.460 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:32:55.460 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:32:55.460 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:32:55.460 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:32:55.460 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:32:55.460 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:32:55.460 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:32:55.460 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:32:55.460 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:32:55.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.439+0000 7f71fae0a640 1 -- 172.21.15.37:0/2848976317 >> v2:172.21.15.45:6812/2379918623 conn(0x7f71e0035ed0 msgr2=0x7f71e0038390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:32:55.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.439+0000 7f71fae0a640 1 --2- 172.21.15.37:0/2848976317 >> v2:172.21.15.45:6812/2379918623 conn(0x7f71e0035ed0 0x7f71e0038390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:55.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.439+0000 7f71fae0a640 10 monclient: shutdown 2024-04-26T00:32:55.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.439+0000 7f71fae0a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:55.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.439+0000 7f71fae0a640 1 -- 172.21.15.37:0/2848976317 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f71f409fc50 msgr2=0x7f71f411b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:55.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.439+0000 7f71fae0a640 1 --2- 172.21.15.37:0/2848976317 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f71f409fc50 0x7f71f411b8a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f71e400c450 tx=0x7f71e400c910 comp rx=0 tx=0).stop 2024-04-26T00:32:55.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.439+0000 7f71f93a7640 1 -- 172.21.15.37:0/2848976317 reap_dead start 2024-04-26T00:32:55.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.439+0000 7f71f93a7640 1 -- 172.21.15.37:0/2848976317 reap_dead start 2024-04-26T00:32:55.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.439+0000 7f71fae0a640 1 -- 172.21.15.37:0/2848976317 shutdown_connections 2024-04-26T00:32:55.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.439+0000 7f71fae0a640 1 -- 172.21.15.37:0/2848976317 >> 172.21.15.37:0/2848976317 conn(0x7f71f40a1d50 msgr2=0x7f71f413c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:55.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.439+0000 7f71fae0a640 1 -- 172.21.15.37:0/2848976317 shutdown_connections 2024-04-26T00:32:55.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.439+0000 7f71fae0a640 1 -- 172.21.15.37:0/2848976317 wait complete. 2024-04-26T00:32:55.463 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:32:55.463 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-04-26T00:32:55.463 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-26T00:32:55.463 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-04-26T00:32:55.463 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-04-26T00:32:55.463 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-26T00:32:55.464 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T00:32:55.491 INFO:tasks.ceph.mon.b:Stopped 2024-04-26T00:32:55.491 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-26T00:32:55.491 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:32:55.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.611+0000 7fec07636640 10 monclient: get_monmap_and_config 2024-04-26T00:32:55.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.611+0000 7fec07636640 10 monclient: build_initial_monmap 2024-04-26T00:32:55.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.611+0000 7fec07636640 10 monclient: monmap: 2024-04-26T00:32:55.608 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:55.608 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:55.608 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:32:55.617705+0000 2024-04-26T00:32:55.609 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:32:55.617705+0000 2024-04-26T00:32:55.609 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:55.609 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:55.609 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:32:55.609 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:32:55.609 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:32:55.609 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:32:55.609 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:32:55.609 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:55.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.611+0000 7fec07636640 1 Processor -- start 2024-04-26T00:32:55.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.611+0000 7fec07636640 1 -- start start 2024-04-26T00:32:55.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.611+0000 7fec07636640 10 monclient: init 2024-04-26T00:32:55.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.611+0000 7fec07636640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:55.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.611+0000 7fec07636640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-04-26T00:32:55.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.611+0000 7fec07636640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fec00136da0 addr v1:172.21.15.37:6789/0 2024-04-26T00:32:55.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.611+0000 7fec07636640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fec00137180 addr v1:172.21.15.45:6789/0 2024-04-26T00:32:55.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.611+0000 7fec07636640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fec0013d2e0 0x7fec0013f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:55.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.611+0000 7fec07636640 10 monclient(hunting): picked mon.noname-a con 0x7fec0013d2e0 addr v2:172.21.15.37:3300/0 2024-04-26T00:32:55.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.611+0000 7fec07636640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec0009b4f0 con 0x7fec00136da0 2024-04-26T00:32:55.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.611+0000 7fec07636640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec0009b7d0 con 0x7fec00137180 2024-04-26T00:32:55.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.611+0000 7fec07636640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:55.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.611+0000 7fec07636640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fec0009bb70 con 0x7fec0013d2e0 2024-04-26T00:32:55.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.611+0000 7fec07636640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:55.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.611+0000 7fec07636640 10 monclient(hunting): authenticate will time out at 1756.923553s 2024-04-26T00:32:55.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.611+0000 7fec04bd1640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7fec00137180 legacy=0x7fec001375c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:32:55.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.611+0000 7fec053d2640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fec00136da0 0x7fec0009bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:49000/0 (socket says 172.21.15.37:49000) 2024-04-26T00:32:55.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.611+0000 7fec053d2640 1 -- 172.21.15.37:0/1236144354 learned_addr learned my addr 172.21.15.37:0/1236144354 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:32:55.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.611+0000 7fec05bd3640 1 --2- 172.21.15.37:0/1236144354 >> v2:172.21.15.37:3300/0 conn(0x7fec0013d2e0 0x7fec0013f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:55.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.611+0000 7fec05bd3640 10 monclient(hunting): get_auth_request con 0x7fec0013d2e0 auth_method 0 2024-04-26T00:32:55.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.611+0000 7fec05bd3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:55.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.611+0000 7fec05bd3640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:55.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.611+0000 7fec05bd3640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:55.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.611+0000 7fec05bd3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:55.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.611+0000 7fec05bd3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:55.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.611+0000 7fec05bd3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:55.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.615+0000 7fec05bd3640 10 monclient(hunting): handle_auth_done global_id 144243 payload 274 2024-04-26T00:32:55.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.615+0000 7fec05bd3640 1 -- 172.21.15.37:0/1236144354 >> v1:172.21.15.45:6789/0 conn(0x7fec00137180 legacy=0x7fec001375c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:32:55.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.615+0000 7fec05bd3640 1 -- 172.21.15.37:0/1236144354 >> v1:172.21.15.37:6789/0 conn(0x7fec00136da0 legacy=0x7fec0009bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:55.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.615+0000 7fec05bd3640 10 monclient: _finish_hunting 0 2024-04-26T00:32:55.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.615+0000 7fec05bd3640 1 monclient: found mon.noname-a 2024-04-26T00:32:55.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.615+0000 7fec05bd3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:55.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.615+0000 7fec05bd3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:32:55.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.615+0000 7fec05bd3640 1 -- 172.21.15.37:0/1236144354 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec0009c5e0 con 0x7fec0013d2e0 2024-04-26T00:32:55.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec05bd3640 1 -- 172.21.15.37:0/1236144354 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fec0013d2e0 msgr2=0x7fec0013f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:55.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec05bd3640 1 --2- 172.21.15.37:0/1236144354 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fec0013d2e0 0x7fec0013f6d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7febf8009d90 tx=0x7febf801c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40a39319e38fdac5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:55.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7febf7fff640 1 -- 172.21.15.37:0/1236144354 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf80ac070 con 0x7fec0013d2e0 2024-04-26T00:32:55.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7febf7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:55.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7febf7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:32:55.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7febf7fff640 10 monclient: dump: 2024-04-26T00:32:55.615 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:55.615 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:55.615 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.615 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.615 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:55.615 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:55.615 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:55.616 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:55.616 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:55.616 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:55.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7febf7fff640 10 monclient: _finish_auth 0 2024-04-26T00:32:55.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7febf7fff640 10 monclient: _check_auth_tickets 2024-04-26T00:32:55.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7febf7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:55.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7febf7fff640 1 -- 172.21.15.37:0/1236144354 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febf80a6760 con 0x7fec0013d2e0 2024-04-26T00:32:55.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7febf7fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:55.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7febf7fff640 1 -- 172.21.15.37:0/1236144354 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf80a6b30 con 0x7fec0013d2e0 2024-04-26T00:32:55.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec07636640 5 monclient: authenticate success, global_id 144243 2024-04-26T00:32:55.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec07636640 10 monclient: get_monmap_and_config success 2024-04-26T00:32:55.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec07636640 4 set_mon_vals no callback set 2024-04-26T00:32:55.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7febf7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:55.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7febf7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:55.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7febf7fff640 10 monclient: dump: 2024-04-26T00:32:55.617 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:55.617 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:55.617 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.617 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.617 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:55.617 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:55.617 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:55.617 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:55.617 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:55.617 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:55.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec07636640 10 monclient: shutdown 2024-04-26T00:32:55.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec07636640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:55.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec07636640 1 -- 172.21.15.37:0/1236144354 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fec0013d2e0 msgr2=0x7fec0013f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:55.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec07636640 1 --2- 172.21.15.37:0/1236144354 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fec0013d2e0 0x7fec0013f6d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7febf8009d90 tx=0x7febf801c7b0 comp rx=0 tx=0).stop 2024-04-26T00:32:55.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec07636640 1 -- 172.21.15.37:0/1236144354 shutdown_connections 2024-04-26T00:32:55.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec07636640 1 --2- 172.21.15.37:0/1236144354 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fec0013d2e0 0x7fec0013f6d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:55.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec07636640 1 -- 172.21.15.37:0/1236144354 >> 172.21.15.37:0/1236144354 conn(0x7fec000a5f00 msgr2=0x7fec000a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:55.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec07636640 1 -- 172.21.15.37:0/1236144354 shutdown_connections 2024-04-26T00:32:55.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec07636640 1 -- 172.21.15.37:0/1236144354 wait complete. 2024-04-26T00:32:55.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec07636640 10 monclient: build_initial_monmap 2024-04-26T00:32:55.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec07636640 10 monclient: monmap: 2024-04-26T00:32:55.623 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:55.624 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:32:55.624 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:32:55.624 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:32:55.624 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:55.624 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:32:55.624 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:32:55.624 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:32:55.624 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:32:55.625 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:55.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec07636640 1 Processor -- start 2024-04-26T00:32:55.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec07636640 1 -- start start 2024-04-26T00:32:55.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec07636640 10 monclient: init 2024-04-26T00:32:55.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec07636640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:55.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec07636640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:32:55.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec07636640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fec00136da0 0x7fec001cf150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:55.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec07636640 10 monclient(hunting): picked mon.noname-a con 0x7fec00136da0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:32:55.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec07636640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fec00137180 0x7fec00135c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:55.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec07636640 10 monclient(hunting): picked mon.noname-c con 0x7fec00137180 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:55.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec07636640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec0013d2e0 0x7fec00138ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:55.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec07636640 10 monclient(hunting): picked mon.noname-b con 0x7fec0013d2e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:32:55.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec07636640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:55.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec07636640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fec00142e20 con 0x7fec00136da0 2024-04-26T00:32:55.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec07636640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:55.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec07636640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fec00142ca0 con 0x7fec0013d2e0 2024-04-26T00:32:55.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec07636640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:55.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec07636640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fec00142fa0 con 0x7fec00137180 2024-04-26T00:32:55.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec07636640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:55.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec07636640 10 monclient(hunting): authenticate will time out at 1756.930919s 2024-04-26T00:32:55.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec04bd1640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fec00137180 0x7fec00135c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:55.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec053d2640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fec00136da0 0x7fec001cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:55.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec05bd3640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec0013d2e0 msgr2=0x7fec00138ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:32:55.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec04bd1640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fec00137180 0x7fec00135c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:48134/0 (socket says 172.21.15.37:48134) 2024-04-26T00:32:55.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec04bd1640 1 -- 172.21.15.37:0/1228538184 learned_addr learned my addr 172.21.15.37:0/1228538184 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:32:55.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec04bd1640 10 monclient(hunting): get_auth_request con 0x7fec00137180 auth_method 0 2024-04-26T00:32:55.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec04bd1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:55.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec04bd1640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:55.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec04bd1640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:55.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec05bd3640 1 --2- 172.21.15.37:0/1228538184 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec0013d2e0 0x7fec00138ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:32:55.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec053d2640 10 monclient(hunting): get_auth_request con 0x7fec00136da0 auth_method 0 2024-04-26T00:32:55.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec053d2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:55.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec053d2640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:55.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec053d2640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:55.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec04bd1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:55.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec04bd1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:55.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec04bd1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:55.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec053d2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:55.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec053d2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:55.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec053d2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:55.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec053d2640 10 monclient(hunting): handle_auth_done global_id 144246 payload 824 2024-04-26T00:32:55.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec053d2640 1 -- 172.21.15.37:0/1228538184 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fec00137180 msgr2=0x7fec00135c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:55.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec053d2640 1 --2- 172.21.15.37:0/1228538184 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fec00137180 0x7fec00135c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:55.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec053d2640 1 -- 172.21.15.37:0/1228538184 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec0013d2e0 msgr2=0x7fec00138ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:32:55.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec053d2640 1 --2- 172.21.15.37:0/1228538184 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec0013d2e0 0x7fec00138ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:55.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec053d2640 10 monclient: _finish_hunting 0 2024-04-26T00:32:55.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec053d2640 1 monclient: found mon.noname-a 2024-04-26T00:32:55.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec053d2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:55.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec053d2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:32:55.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec053d2640 1 -- 172.21.15.37:0/1228538184 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec0013b010 con 0x7fec00136da0 2024-04-26T00:32:55.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.619+0000 7fec04bd1640 1 --2- 172.21.15.37:0/1228538184 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fec00137180 0x7fec00135c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:32:55.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.623+0000 7fec053d2640 1 -- 172.21.15.37:0/1228538184 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fec00136da0 msgr2=0x7fec001cf150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:55.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.623+0000 7fec053d2640 1 --2- 172.21.15.37:0/1228538184 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fec00136da0 0x7fec001cf150 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7febfc00c410 tx=0x7febfc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:55.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.623+0000 7febf5ffb640 1 -- 172.21.15.37:0/1228538184 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febfc017070 con 0x7fec00136da0 2024-04-26T00:32:55.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.623+0000 7febf5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:55.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.623+0000 7febf5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:32:55.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.623+0000 7febf5ffb640 10 monclient: dump: 2024-04-26T00:32:55.629 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:55.629 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:55.629 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.629 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.630 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:55.630 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:55.630 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:55.630 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:55.630 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:55.630 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:55.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.623+0000 7febf5ffb640 10 monclient: _finish_auth 0 2024-04-26T00:32:55.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.623+0000 7febf5ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:32:55.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.623+0000 7febf5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:55.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.623+0000 7fec07636640 5 monclient: authenticate success, global_id 144246 2024-04-26T00:32:55.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.623+0000 7febf5ffb640 1 -- 172.21.15.37:0/1228538184 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febfc0070d0 con 0x7fec00136da0 2024-04-26T00:32:55.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.623+0000 7febf5ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:55.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.623+0000 7fec07636640 10 monclient: _renew_subs 2024-04-26T00:32:55.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.623+0000 7febf5ffb640 1 -- 172.21.15.37:0/1228538184 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febfc0074c0 con 0x7fec00136da0 2024-04-26T00:32:55.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.623+0000 7fec07636640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:55.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.623+0000 7fec07636640 1 -- 172.21.15.37:0/1228538184 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec0013c740 con 0x7fec00136da0 2024-04-26T00:32:55.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.623+0000 7febf5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:55.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.623+0000 7febf5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:55.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.623+0000 7febf5ffb640 10 monclient: dump: 2024-04-26T00:32:55.631 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:55.631 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:55.631 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.631 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:55.631 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:55.631 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:55.631 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:55.631 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:55.631 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:55.631 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:55.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.623+0000 7febf77fe640 4 set_mon_vals no callback set 2024-04-26T00:32:55.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.623+0000 7fec07636640 10 monclient: _renew_subs 2024-04-26T00:32:55.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.623+0000 7fec07636640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:55.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.623+0000 7fec07636640 1 -- 172.21.15.37:0/1228538184 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec0013b5d0 con 0x7fec00136da0 2024-04-26T00:32:55.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.623+0000 7fec07636640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:55.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.623+0000 7fec07636640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:55.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.623+0000 7fec07636640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:55.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.623+0000 7febf5ffb640 1 -- 172.21.15.37:0/1228538184 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7febfc01a070 con 0x7fec00136da0 2024-04-26T00:32:55.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.623+0000 7fec07636640 1 -- 172.21.15.37:0/1228538184 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec00066f40 con 0x7fec00136da0 2024-04-26T00:32:55.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.623+0000 7febf5ffb640 1 --2- 172.21.15.37:0/1228538184 >> v2:172.21.15.45:6812/2379918623 conn(0x7febe4035e80 0x7febe4038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:55.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.623+0000 7febf5ffb640 1 -- 172.21.15.37:0/1228538184 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(503..503 src has 370..503) ==== 3915+0+0 (secure 0 0 0) 0x7febfc047ee0 con 0x7fec00136da0 2024-04-26T00:32:55.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.623+0000 7fec04bd1640 1 --2- 172.21.15.37:0/1228538184 >> v2:172.21.15.45:6812/2379918623 conn(0x7febe4035e80 0x7febe4038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:55.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.623+0000 7fec04bd1640 10 monclient: get_auth_request con 0x7febe4035e80 auth_method 0 2024-04-26T00:32:55.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.627+0000 7fec04bd1640 1 --2- 172.21.15.37:0/1228538184 >> v2:172.21.15.45:6812/2379918623 conn(0x7febe4035e80 0x7febe4038340 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fec0013b9d0 tx=0x7febf0003b50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:55.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.631+0000 7febf5ffb640 1 -- 172.21.15.37:0/1228538184 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7febfc010db0 con 0x7fec00136da0 2024-04-26T00:32:55.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.631+0000 7febf5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:55.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.631+0000 7febf5ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:32:55.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.851+0000 7fec07636640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:32:55.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.851+0000 7fec07636640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:32:55.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.851+0000 7fec07636640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:32:55.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.851+0000 7fec07636640 1 -- 172.21.15.37:0/1228538184 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fec000a8210 con 0x7fec00136da0 2024-04-26T00:32:55.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.851+0000 7febf5ffb640 1 -- 172.21.15.37:0/1228538184 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7febfc046c40 con 0x7fec00136da0 2024-04-26T00:32:55.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.851+0000 7febf5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:32:55.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.851+0000 7febf5ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:32:55.849 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":96,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:32:55.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.855+0000 7fec07636640 1 -- 172.21.15.37:0/1228538184 >> v2:172.21.15.45:6812/2379918623 conn(0x7febe4035e80 msgr2=0x7febe4038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:55.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.855+0000 7fec07636640 1 --2- 172.21.15.37:0/1228538184 >> v2:172.21.15.45:6812/2379918623 conn(0x7febe4035e80 0x7febe4038340 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fec0013b9d0 tx=0x7febf0003b50 comp rx=0 tx=0).stop 2024-04-26T00:32:55.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.855+0000 7fec07636640 10 monclient: shutdown 2024-04-26T00:32:55.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.855+0000 7fec07636640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:55.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.855+0000 7fec07636640 1 -- 172.21.15.37:0/1228538184 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fec00136da0 msgr2=0x7fec001cf150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:55.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.855+0000 7fec07636640 1 --2- 172.21.15.37:0/1228538184 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fec00136da0 0x7fec001cf150 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7febfc00c410 tx=0x7febfc00c8d0 comp rx=0 tx=0).stop 2024-04-26T00:32:55.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.855+0000 7fec07636640 1 -- 172.21.15.37:0/1228538184 shutdown_connections 2024-04-26T00:32:55.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.855+0000 7fec07636640 1 --2- 172.21.15.37:0/1228538184 >> v2:172.21.15.45:6812/2379918623 conn(0x7febe4035e80 0x7febe4038340 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:55.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.855+0000 7fec07636640 1 --2- 172.21.15.37:0/1228538184 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec0013d2e0 0x7fec00138ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:55.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.855+0000 7fec07636640 1 --2- 172.21.15.37:0/1228538184 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fec00137180 0x7fec00135c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:55.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.855+0000 7fec07636640 1 --2- 172.21.15.37:0/1228538184 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fec00136da0 0x7fec001cf150 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:55.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.855+0000 7fec07636640 1 -- 172.21.15.37:0/1228538184 >> 172.21.15.37:0/1228538184 conn(0x7fec000a5f00 msgr2=0x7fec0013e150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:55.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.855+0000 7fec07636640 1 -- 172.21.15.37:0/1228538184 shutdown_connections 2024-04-26T00:32:55.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:55.855+0000 7fec07636640 1 -- 172.21.15.37:0/1228538184 wait complete. 2024-04-26T00:32:58.869 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:32:59.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab6a65640 10 monclient: get_monmap_and_config 2024-04-26T00:32:59.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab6a65640 10 monclient: build_initial_monmap 2024-04-26T00:32:59.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab6a65640 10 monclient: monmap: 2024-04-26T00:32:59.000 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:59.000 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:59.000 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:32:59.009473+0000 2024-04-26T00:32:59.001 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:32:59.009473+0000 2024-04-26T00:32:59.001 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:59.001 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:59.001 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:32:59.001 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:32:59.001 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:32:59.001 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:32:59.001 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:32:59.001 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:59.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab6a65640 1 Processor -- start 2024-04-26T00:32:59.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab6a65640 1 -- start start 2024-04-26T00:32:59.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab6a65640 10 monclient: init 2024-04-26T00:32:59.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab6a65640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:59.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab6a65640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-04-26T00:32:59.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab6a65640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feab00a0260 0x7feab00a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:59.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab6a65640 10 monclient(hunting): picked mon.noname-b con 0x7feab00a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:59.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab6a65640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feab0134b30 addr v1:172.21.15.37:6789/0 2024-04-26T00:32:59.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab6a65640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7feab0135660 0x7feab013ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:59.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab6a65640 10 monclient(hunting): picked mon.noname-a con 0x7feab0135660 addr v2:172.21.15.37:3300/0 2024-04-26T00:32:59.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab6a65640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feab00a1130 con 0x7feab0134b30 2024-04-26T00:32:59.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab6a65640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:59.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab6a65640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7feab009ec10 con 0x7feab0135660 2024-04-26T00:32:59.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab6a65640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:59.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab6a65640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7feab009f0f0 con 0x7feab00a0260 2024-04-26T00:32:59.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab6a65640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:59.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab6a65640 10 monclient(hunting): authenticate will time out at 1760.315266s 2024-04-26T00:32:59.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab5002640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7feab0135660 0x7feab013ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:59.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab4801640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feab00a0260 0x7feab00a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:59.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab5002640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7feab0135660 0x7feab013ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44764/0 (socket says 172.21.15.37:44764) 2024-04-26T00:32:59.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab5002640 1 -- 172.21.15.37:0/3392902731 learned_addr learned my addr 172.21.15.37:0/3392902731 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:32:59.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feaaffff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7feab0134b30 0x7feab0134f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:49010/0 (socket says 172.21.15.37:49010) 2024-04-26T00:32:59.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab5002640 10 monclient(hunting): get_auth_request con 0x7feab0135660 auth_method 0 2024-04-26T00:32:59.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab5002640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:59.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab5002640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:59.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab5002640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:59.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab4801640 10 monclient(hunting): get_auth_request con 0x7feab00a0260 auth_method 0 2024-04-26T00:32:59.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab4801640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:59.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab4801640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:59.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab4801640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:59.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab5002640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:59.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab5002640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:59.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab5002640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:59.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab4801640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:59.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab4801640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:59.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab4801640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:59.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab5002640 10 monclient(hunting): handle_auth_done global_id 144249 payload 274 2024-04-26T00:32:59.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab5002640 1 -- 172.21.15.37:0/3392902731 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feab00a0260 msgr2=0x7feab00a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:59.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab5002640 1 --2- 172.21.15.37:0/3392902731 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feab00a0260 0x7feab00a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:59.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab5002640 1 -- 172.21.15.37:0/3392902731 >> v1:172.21.15.37:6789/0 conn(0x7feab0134b30 legacy=0x7feab0134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:32:59.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab5002640 10 monclient: _finish_hunting 0 2024-04-26T00:32:59.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab5002640 1 monclient: found mon.noname-a 2024-04-26T00:32:59.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab5002640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:59.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab5002640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:32:59.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.003+0000 7feab5002640 1 -- 172.21.15.37:0/3392902731 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feab006cff0 con 0x7feab0135660 2024-04-26T00:32:59.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab5002640 1 -- 172.21.15.37:0/3392902731 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feab0135660 msgr2=0x7feab013ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:59.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab5002640 1 --2- 172.21.15.37:0/3392902731 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feab0135660 0x7feab013ce70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7feaa8009870 tx=0x7feaa801c200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa5f94f548de826c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:59.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feaaf7fe640 1 -- 172.21.15.37:0/3392902731 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa80b0070 con 0x7feab0135660 2024-04-26T00:32:59.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feaaf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:59.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feaaf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:32:59.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feaaf7fe640 10 monclient: dump: 2024-04-26T00:32:59.016 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:59.016 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:59.016 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:59.016 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:59.017 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:59.017 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:59.017 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:59.017 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:59.017 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:59.017 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:59.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feaaf7fe640 10 monclient: _finish_auth 0 2024-04-26T00:32:59.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feaaf7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:32:59.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feaaf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:59.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feaaf7fe640 1 -- 172.21.15.37:0/3392902731 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feaa80ab600 con 0x7feab0135660 2024-04-26T00:32:59.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab6a65640 5 monclient: authenticate success, global_id 144249 2024-04-26T00:32:59.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feaaf7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:59.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab6a65640 10 monclient: get_monmap_and_config success 2024-04-26T00:32:59.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab6a65640 4 set_mon_vals no callback set 2024-04-26T00:32:59.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feaaf7fe640 1 -- 172.21.15.37:0/3392902731 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa80ab9d0 con 0x7feab0135660 2024-04-26T00:32:59.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feaaf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:59.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feaaf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:32:59.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feaaf7fe640 10 monclient: dump: 2024-04-26T00:32:59.018 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:59.018 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:59.018 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:59.018 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:59.018 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:59.018 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:59.018 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:59.018 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:59.018 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:59.018 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:59.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab6a65640 10 monclient: shutdown 2024-04-26T00:32:59.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab6a65640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:59.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab6a65640 1 -- 172.21.15.37:0/3392902731 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feab0135660 msgr2=0x7feab013ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:59.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab6a65640 1 --2- 172.21.15.37:0/3392902731 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feab0135660 0x7feab013ce70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7feaa8009870 tx=0x7feaa801c200 comp rx=0 tx=0).stop 2024-04-26T00:32:59.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab6a65640 1 -- 172.21.15.37:0/3392902731 shutdown_connections 2024-04-26T00:32:59.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab6a65640 1 --2- 172.21.15.37:0/3392902731 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feab0135660 0x7feab013ce70 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:59.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab6a65640 1 --2- 172.21.15.37:0/3392902731 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feab00a0260 0x7feab00a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:59.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab6a65640 1 -- 172.21.15.37:0/3392902731 >> 172.21.15.37:0/3392902731 conn(0x7feab00a1ad0 msgr2=0x7feab00a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:59.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab6a65640 1 -- 172.21.15.37:0/3392902731 shutdown_connections 2024-04-26T00:32:59.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab6a65640 1 -- 172.21.15.37:0/3392902731 wait complete. 2024-04-26T00:32:59.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab6a65640 10 monclient: build_initial_monmap 2024-04-26T00:32:59.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab6a65640 10 monclient: monmap: 2024-04-26T00:32:59.019 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:32:59.019 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:32:59.019 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:32:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:32:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:32:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:32:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:32:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:32:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:32:59.020 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab6a65640 1 Processor -- start 2024-04-26T00:32:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab6a65640 1 -- start start 2024-04-26T00:32:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab6a65640 10 monclient: init 2024-04-26T00:32:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab6a65640 10 monclient: _reopen_session rank -1 2024-04-26T00:32:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab6a65640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:32:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab6a65640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feab00a0260 0x7feab0132240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab6a65640 10 monclient(hunting): picked mon.noname-c con 0x7feab00a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:32:59.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab6a65640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feab0134b30 0x7feab009c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:59.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab6a65640 10 monclient(hunting): picked mon.noname-b con 0x7feab0134b30 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:32:59.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab6a65640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feab0135660 0x7feab009ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:59.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab6a65640 10 monclient(hunting): picked mon.noname-a con 0x7feab0135660 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:32:59.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab6a65640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:59.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab6a65640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7feab00a0010 con 0x7feab0135660 2024-04-26T00:32:59.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab6a65640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:59.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab6a65640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7feab009fe90 con 0x7feab0134b30 2024-04-26T00:32:59.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab6a65640 10 monclient(hunting): start opening mon connection 2024-04-26T00:32:59.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab6a65640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7feab013fbd0 con 0x7feab00a0260 2024-04-26T00:32:59.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab6a65640 10 monclient(hunting): _renew_subs 2024-04-26T00:32:59.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab6a65640 10 monclient(hunting): authenticate will time out at 1760.322499s 2024-04-26T00:32:59.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab4801640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feab00a0260 0x7feab0132240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:59.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab5002640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feab0135660 0x7feab009ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab4801640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feab00a0260 0x7feab0132240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:48154/0 (socket says 172.21.15.37:48154) 2024-04-26T00:32:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab4801640 1 -- 172.21.15.37:0/357339752 learned_addr learned my addr 172.21.15.37:0/357339752 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:32:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab4801640 10 monclient(hunting): get_auth_request con 0x7feab00a0260 auth_method 0 2024-04-26T00:32:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab4801640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab4801640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab4801640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab5002640 10 monclient(hunting): get_auth_request con 0x7feab0135660 auth_method 0 2024-04-26T00:32:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab5002640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:32:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab5002640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:32:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab5002640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:32:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feaaffff640 1 -- 172.21.15.37:0/357339752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feab0134b30 msgr2=0x7feab009c500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:32:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feaaffff640 1 --2- 172.21.15.37:0/357339752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feab0134b30 0x7feab009c500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:32:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab4801640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab4801640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:59.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab4801640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:59.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab5002640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:32:59.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab5002640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:32:59.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab5002640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:32:59.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab4801640 10 monclient(hunting): handle_auth_done global_id 154136 payload 824 2024-04-26T00:32:59.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab4801640 1 -- 172.21.15.37:0/357339752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feab0134b30 msgr2=0x7feab009c500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:32:59.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab4801640 1 --2- 172.21.15.37:0/357339752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feab0134b30 0x7feab009c500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:59.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab4801640 1 -- 172.21.15.37:0/357339752 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feab0135660 msgr2=0x7feab009ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:59.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab4801640 1 --2- 172.21.15.37:0/357339752 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feab0135660 0x7feab009ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:59.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab4801640 10 monclient: _finish_hunting 0 2024-04-26T00:32:59.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab4801640 1 monclient: found mon.noname-c 2024-04-26T00:32:59.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab4801640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:32:59.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab4801640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:32:59.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.011+0000 7feab4801640 1 -- 172.21.15.37:0/357339752 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feab009e1a0 con 0x7feab00a0260 2024-04-26T00:32:59.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.015+0000 7feab4801640 1 -- 172.21.15.37:0/357339752 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feab00a0260 msgr2=0x7feab0132240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:32:59.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.015+0000 7feab4801640 1 --2- 172.21.15.37:0/357339752 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feab00a0260 0x7feab0132240 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7feaa000e870 tx=0x7feaa000ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:59.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.015+0000 7feaad7fa640 1 -- 172.21.15.37:0/357339752 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa0017070 con 0x7feab00a0260 2024-04-26T00:32:59.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.015+0000 7feaad7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:59.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.015+0000 7feaad7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:32:59.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.015+0000 7feaad7fa640 10 monclient: dump: 2024-04-26T00:32:59.024 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:59.024 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:59.024 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:59.024 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:59.024 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:59.024 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:59.025 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:59.025 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:59.025 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:59.025 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:59.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.015+0000 7feaad7fa640 10 monclient: _finish_auth 0 2024-04-26T00:32:59.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.015+0000 7feaad7fa640 10 monclient: _check_auth_tickets 2024-04-26T00:32:59.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.015+0000 7feaad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:32:59.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.015+0000 7feab6a65640 5 monclient: authenticate success, global_id 154136 2024-04-26T00:32:59.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.015+0000 7feab6a65640 10 monclient: _renew_subs 2024-04-26T00:32:59.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.015+0000 7feab6a65640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:32:59.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.015+0000 7feab6a65640 1 -- 172.21.15.37:0/357339752 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feab013b8f0 con 0x7feab00a0260 2024-04-26T00:32:59.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.015+0000 7feab6a65640 10 monclient: _renew_subs 2024-04-26T00:32:59.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.015+0000 7feab6a65640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:32:59.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.015+0000 7feab6a65640 1 -- 172.21.15.37:0/357339752 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7feab013b610 con 0x7feab00a0260 2024-04-26T00:32:59.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.015+0000 7feaad7fa640 1 -- 172.21.15.37:0/357339752 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feaa000ee60 con 0x7feab00a0260 2024-04-26T00:32:59.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.015+0000 7feaad7fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:32:59.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.015+0000 7feaad7fa640 1 -- 172.21.15.37:0/357339752 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa000cd70 con 0x7feab00a0260 2024-04-26T00:32:59.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.015+0000 7feaad7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:32:59.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.015+0000 7feaad7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:32:59.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.015+0000 7feaad7fa640 10 monclient: dump: 2024-04-26T00:32:59.026 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:32:59.026 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:32:59.026 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:59.026 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:32:59.026 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:32:59.026 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:32:59.026 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:32:59.026 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:32:59.026 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:32:59.027 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:32:59.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.015+0000 7feaae7fc640 4 set_mon_vals no callback set 2024-04-26T00:32:59.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.015+0000 7feab6a65640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:59.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.015+0000 7feab6a65640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:59.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.015+0000 7feab6a65640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:32:59.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.015+0000 7feab6a65640 1 -- 172.21.15.37:0/357339752 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea94005090 con 0x7feab00a0260 2024-04-26T00:32:59.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.015+0000 7feaad7fa640 1 -- 172.21.15.37:0/357339752 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7feaa001a070 con 0x7feab00a0260 2024-04-26T00:32:59.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.015+0000 7feaad7fa640 1 --2- 172.21.15.37:0/357339752 >> v2:172.21.15.45:6812/2379918623 conn(0x7fea90035e60 0x7fea90038320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:32:59.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.015+0000 7feaad7fa640 1 -- 172.21.15.37:0/357339752 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(503..503 src has 370..503) ==== 3915+0+0 (secure 0 0 0) 0x7feaa00490b0 con 0x7feab00a0260 2024-04-26T00:32:59.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.015+0000 7feaaffff640 1 --2- 172.21.15.37:0/357339752 >> v2:172.21.15.45:6812/2379918623 conn(0x7fea90035e60 0x7fea90038320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:32:59.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.015+0000 7feaaffff640 10 monclient: get_auth_request con 0x7fea90035e60 auth_method 0 2024-04-26T00:32:59.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.019+0000 7feaaffff640 1 --2- 172.21.15.37:0/357339752 >> v2:172.21.15.45:6812/2379918623 conn(0x7fea90035e60 0x7fea90038320 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7feab013bb00 tx=0x7fea98007df0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:32:59.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.019+0000 7feaad7fa640 1 -- 172.21.15.37:0/357339752 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7feaa0024380 con 0x7feab00a0260 2024-04-26T00:32:59.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.023+0000 7feaad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:32:59.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.023+0000 7feaad7fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:32:59.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.243+0000 7feab6a65640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:32:59.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.243+0000 7feab6a65640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:32:59.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.243+0000 7feab6a65640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:32:59.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.243+0000 7feab6a65640 1 -- 172.21.15.37:0/357339752 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fea94005650 con 0x7feab00a0260 2024-04-26T00:32:59.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.243+0000 7feaad7fa640 1 -- 172.21.15.37:0/357339752 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7feaa0013470 con 0x7feab00a0260 2024-04-26T00:32:59.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.243+0000 7feaad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:32:59.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.243+0000 7feaad7fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:32:59.241 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":96,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:32:59.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.247+0000 7feab6a65640 1 -- 172.21.15.37:0/357339752 >> v2:172.21.15.45:6812/2379918623 conn(0x7fea90035e60 msgr2=0x7fea90038320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.247+0000 7feab6a65640 1 --2- 172.21.15.37:0/357339752 >> v2:172.21.15.45:6812/2379918623 conn(0x7fea90035e60 0x7fea90038320 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7feab013bb00 tx=0x7fea98007df0 comp rx=0 tx=0).stop 2024-04-26T00:32:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.247+0000 7feab6a65640 10 monclient: shutdown 2024-04-26T00:32:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.247+0000 7feab6a65640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:32:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.247+0000 7feab6a65640 1 -- 172.21.15.37:0/357339752 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feab00a0260 msgr2=0x7feab0132240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:32:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.247+0000 7feab6a65640 1 --2- 172.21.15.37:0/357339752 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feab00a0260 0x7feab0132240 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7feaa000e870 tx=0x7feaa000ed30 comp rx=0 tx=0).stop 2024-04-26T00:32:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.247+0000 7feab6a65640 1 -- 172.21.15.37:0/357339752 shutdown_connections 2024-04-26T00:32:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.247+0000 7feab6a65640 1 --2- 172.21.15.37:0/357339752 >> v2:172.21.15.45:6812/2379918623 conn(0x7fea90035e60 0x7fea90038320 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.247+0000 7feab6a65640 1 --2- 172.21.15.37:0/357339752 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feab0135660 0x7feab009ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.247+0000 7feab6a65640 1 --2- 172.21.15.37:0/357339752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feab0134b30 0x7feab009c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.247+0000 7feab6a65640 1 --2- 172.21.15.37:0/357339752 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feab00a0260 0x7feab0132240 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:32:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.247+0000 7feab6a65640 1 -- 172.21.15.37:0/357339752 >> 172.21.15.37:0/357339752 conn(0x7feab00a1ad0 msgr2=0x7feab00a3d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:32:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.247+0000 7feab6a65640 1 -- 172.21.15.37:0/357339752 shutdown_connections 2024-04-26T00:32:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:32:59.247+0000 7feab6a65640 1 -- 172.21.15.37:0/357339752 wait complete. 2024-04-26T00:33:02.262 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:33:02.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.399+0000 7f891b9cc640 10 monclient: get_monmap_and_config 2024-04-26T00:33:02.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.399+0000 7f891b9cc640 10 monclient: build_initial_monmap 2024-04-26T00:33:02.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.399+0000 7f891b9cc640 10 monclient: monmap: 2024-04-26T00:33:02.393 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:33:02.393 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:02.393 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:33:02.401861+0000 2024-04-26T00:33:02.394 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:33:02.401861+0000 2024-04-26T00:33:02.394 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:33:02.394 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:02.394 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:33:02.394 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:33:02.394 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:33:02.394 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:33:02.394 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:33:02.394 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:02.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.399+0000 7f891b9cc640 1 Processor -- start 2024-04-26T00:33:02.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.399+0000 7f891b9cc640 1 -- start start 2024-04-26T00:33:02.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.399+0000 7f891b9cc640 10 monclient: init 2024-04-26T00:33:02.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.399+0000 7f891b9cc640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:02.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.399+0000 7f891b9cc640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-04-26T00:33:02.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.399+0000 7f891b9cc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f89140a0260 0x7f89140a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:02.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.399+0000 7f891b9cc640 10 monclient(hunting): picked mon.noname-b con 0x7f89140a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:33:02.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.399+0000 7f891b9cc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8914134b30 addr v1:172.21.15.45:6789/0 2024-04-26T00:33:02.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.399+0000 7f891b9cc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8914135660 addr v1:172.21.15.37:6789/0 2024-04-26T00:33:02.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.399+0000 7f891b9cc640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89140a1130 con 0x7f8914135660 2024-04-26T00:33:02.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.399+0000 7f891b9cc640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f891406cff0 con 0x7f8914134b30 2024-04-26T00:33:02.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.399+0000 7f891b9cc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:02.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.399+0000 7f891b9cc640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f891409ec10 con 0x7f89140a0260 2024-04-26T00:33:02.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.399+0000 7f891b9cc640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:02.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.399+0000 7f891b9cc640 10 monclient(hunting): authenticate will time out at 1763.707520s 2024-04-26T00:33:02.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.399+0000 7f8919768640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f89140a0260 0x7f89140a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:02.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.399+0000 7f8918f67640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f8914134b30 legacy=0x7f8914134f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:33:02.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.399+0000 7f8919f69640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f8914135660 0x7f891413ce70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:46072/0 (socket says 172.21.15.37:46072) 2024-04-26T00:33:02.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.399+0000 7f8919768640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f89140a0260 0x7f89140a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:52204/0 (socket says 172.21.15.37:52204) 2024-04-26T00:33:02.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.399+0000 7f8919f69640 1 -- 172.21.15.37:0/2975253690 learned_addr learned my addr 172.21.15.37:0/2975253690 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:33:02.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.399+0000 7f8919768640 10 monclient(hunting): get_auth_request con 0x7f89140a0260 auth_method 0 2024-04-26T00:33:02.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.399+0000 7f8919768640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:02.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.399+0000 7f8919768640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:02.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.399+0000 7f8919768640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:02.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.399+0000 7f8919768640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:02.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.399+0000 7f8919768640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:02.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.399+0000 7f8919768640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:02.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.403+0000 7f8919768640 10 monclient(hunting): handle_auth_done global_id 154139 payload 274 2024-04-26T00:33:02.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.403+0000 7f8919768640 1 -- 172.21.15.37:0/2975253690 >> v1:172.21.15.45:6789/0 conn(0x7f8914134b30 legacy=0x7f8914134f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:33:02.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.403+0000 7f8919768640 1 -- 172.21.15.37:0/2975253690 >> v1:172.21.15.37:6789/0 conn(0x7f8914135660 legacy=0x7f891413ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:33:02.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.403+0000 7f8919768640 10 monclient: _finish_hunting 0 2024-04-26T00:33:02.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.403+0000 7f8919768640 1 monclient: found mon.noname-b 2024-04-26T00:33:02.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.403+0000 7f8919768640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:02.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.403+0000 7f8919768640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:33:02.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.403+0000 7f8919768640 1 -- 172.21.15.37:0/2975253690 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f891409bc50 con 0x7f89140a0260 2024-04-26T00:33:02.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f8919768640 1 -- 172.21.15.37:0/2975253690 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f89140a0260 msgr2=0x7f89140a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:02.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f8919768640 1 --2- 172.21.15.37:0/2975253690 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f89140a0260 0x7f89140a0640 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f89100097f0 tx=0x7f891001c140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f21a1fab59ca524 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:02.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f890bfff640 1 -- 172.21.15.37:0/2975253690 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89100b1070 con 0x7f89140a0260 2024-04-26T00:33:02.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f890bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:02.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f890bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:33:02.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f890bfff640 10 monclient: dump: 2024-04-26T00:33:02.404 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:02.404 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:02.404 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:02.404 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:02.404 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:02.404 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:02.404 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:02.404 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:02.404 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:02.404 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:02.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f890bfff640 10 monclient: _finish_auth 0 2024-04-26T00:33:02.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f890bfff640 10 monclient: _check_auth_tickets 2024-04-26T00:33:02.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f890bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:02.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f890bfff640 1 -- 172.21.15.37:0/2975253690 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89100ac440 con 0x7f89140a0260 2024-04-26T00:33:02.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f891b9cc640 5 monclient: authenticate success, global_id 154139 2024-04-26T00:33:02.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f891b9cc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:33:02.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f890bfff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:33:02.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f891b9cc640 10 monclient: get_monmap_and_config success 2024-04-26T00:33:02.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f891b9cc640 4 set_mon_vals no callback set 2024-04-26T00:33:02.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f890bfff640 1 -- 172.21.15.37:0/2975253690 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89100ac810 con 0x7f89140a0260 2024-04-26T00:33:02.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f890bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:02.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f890bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:33:02.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f890bfff640 10 monclient: dump: 2024-04-26T00:33:02.405 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:02.405 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:02.405 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:02.406 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:02.406 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:02.406 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:02.406 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:02.406 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:02.406 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:02.406 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:02.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f891b9cc640 10 monclient: shutdown 2024-04-26T00:33:02.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f891b9cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:33:02.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f891b9cc640 1 -- 172.21.15.37:0/2975253690 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f89140a0260 msgr2=0x7f89140a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:02.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f891b9cc640 1 --2- 172.21.15.37:0/2975253690 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f89140a0260 0x7f89140a0640 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f89100097f0 tx=0x7f891001c140 comp rx=0 tx=0).stop 2024-04-26T00:33:02.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f891b9cc640 1 -- 172.21.15.37:0/2975253690 shutdown_connections 2024-04-26T00:33:02.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f891b9cc640 1 --2- 172.21.15.37:0/2975253690 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f89140a0260 0x7f89140a0640 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:02.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f891b9cc640 1 -- 172.21.15.37:0/2975253690 >> 172.21.15.37:0/2975253690 conn(0x7f89140a1ad0 msgr2=0x7f89140a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:33:02.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f891b9cc640 1 -- 172.21.15.37:0/2975253690 shutdown_connections 2024-04-26T00:33:02.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f891b9cc640 1 -- 172.21.15.37:0/2975253690 wait complete. 2024-04-26T00:33:02.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f891b9cc640 10 monclient: build_initial_monmap 2024-04-26T00:33:02.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f891b9cc640 10 monclient: monmap: 2024-04-26T00:33:02.407 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:33:02.407 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:33:02.407 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:33:02.407 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:33:02.407 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:33:02.407 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:33:02.407 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:33:02.407 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:33:02.407 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:33:02.407 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:02.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f891b9cc640 1 Processor -- start 2024-04-26T00:33:02.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f891b9cc640 1 -- start start 2024-04-26T00:33:02.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f891b9cc640 10 monclient: init 2024-04-26T00:33:02.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f891b9cc640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:02.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f891b9cc640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:33:02.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f891b9cc640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89140a0260 0x7f89141cee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:02.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f891b9cc640 10 monclient(hunting): picked mon.noname-a con 0x7f89140a0260 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:33:02.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f891b9cc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8914134b30 0x7f8914131900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:02.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f891b9cc640 10 monclient(hunting): picked mon.noname-c con 0x7f8914134b30 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:33:02.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f891b9cc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8914135660 0x7f8914131e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:02.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f891b9cc640 10 monclient(hunting): picked mon.noname-b con 0x7f8914135660 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:33:02.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f891b9cc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:02.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f891b9cc640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f891409f4c0 con 0x7f89140a0260 2024-04-26T00:33:02.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f891b9cc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:02.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f891b9cc640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f891409f0f0 con 0x7f8914135660 2024-04-26T00:33:02.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f891b9cc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:02.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f891b9cc640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8914140db0 con 0x7f8914134b30 2024-04-26T00:33:02.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f891b9cc640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:02.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f891b9cc640 10 monclient(hunting): authenticate will time out at 1763.714692s 2024-04-26T00:33:02.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f8919768640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89140a0260 0x7f89141cee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:02.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f8918f67640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8914134b30 0x7f8914131900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:02.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f8919768640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89140a0260 0x7f89141cee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51058/0 (socket says 172.21.15.37:51058) 2024-04-26T00:33:02.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f8919768640 1 -- 172.21.15.37:0/1621724363 learned_addr learned my addr 172.21.15.37:0/1621724363 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:33:02.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f8919768640 10 monclient(hunting): get_auth_request con 0x7f89140a0260 auth_method 0 2024-04-26T00:33:02.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f8919768640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:02.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f8919768640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:02.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f8919768640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:02.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f8918f67640 10 monclient(hunting): get_auth_request con 0x7f8914134b30 auth_method 0 2024-04-26T00:33:02.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f8918f67640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:02.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f8918f67640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:02.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f8918f67640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:02.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f8919f69640 1 -- 172.21.15.37:0/1621724363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8914135660 msgr2=0x7f8914131e40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:33:02.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f8919f69640 1 --2- 172.21.15.37:0/1621724363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8914135660 0x7f8914131e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:33:02.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f8919768640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:02.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f8919768640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:02.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f8919768640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:02.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f8918f67640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:02.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f8918f67640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:02.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f8918f67640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:02.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f8919768640 10 monclient(hunting): handle_auth_done global_id 144255 payload 824 2024-04-26T00:33:02.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f8919768640 1 -- 172.21.15.37:0/1621724363 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8914134b30 msgr2=0x7f8914131900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:02.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f8919768640 1 --2- 172.21.15.37:0/1621724363 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8914134b30 0x7f8914131900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:02.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f8919768640 1 -- 172.21.15.37:0/1621724363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8914135660 msgr2=0x7f8914131e40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:33:02.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f8919768640 1 --2- 172.21.15.37:0/1621724363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8914135660 0x7f8914131e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:02.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f8919768640 10 monclient: _finish_hunting 0 2024-04-26T00:33:02.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f8919768640 1 monclient: found mon.noname-a 2024-04-26T00:33:02.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f8919768640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:02.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f8919768640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:33:02.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f8919768640 1 -- 172.21.15.37:0/1621724363 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8914071ba0 con 0x7f89140a0260 2024-04-26T00:33:02.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.407+0000 7f8918f67640 1 --2- 172.21.15.37:0/1621724363 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8914134b30 0x7f8914131900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:33:02.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.411+0000 7f8919768640 1 -- 172.21.15.37:0/1621724363 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89140a0260 msgr2=0x7f89141cee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:02.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.411+0000 7f8919768640 1 --2- 172.21.15.37:0/1621724363 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89140a0260 0x7f89141cee40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f891001a7f0 tx=0x7f8910018ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:02.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.411+0000 7f8909ffb640 1 -- 172.21.15.37:0/1621724363 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89100ba020 con 0x7f89140a0260 2024-04-26T00:33:02.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.411+0000 7f8909ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:02.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.411+0000 7f8909ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:33:02.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.411+0000 7f8909ffb640 10 monclient: dump: 2024-04-26T00:33:02.414 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:02.414 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:02.414 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:02.414 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:02.415 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:02.415 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:02.415 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:02.415 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:02.415 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:02.415 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:02.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.411+0000 7f8909ffb640 10 monclient: _finish_auth 0 2024-04-26T00:33:02.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.411+0000 7f8909ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:33:02.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.411+0000 7f8909ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:02.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.411+0000 7f8909ffb640 1 -- 172.21.15.37:0/1621724363 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89100ace50 con 0x7f89140a0260 2024-04-26T00:33:02.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.411+0000 7f8909ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:33:02.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.411+0000 7f8909ffb640 1 -- 172.21.15.37:0/1621724363 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89100b1040 con 0x7f89140a0260 2024-04-26T00:33:02.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.411+0000 7f8909ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:02.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.411+0000 7f8909ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:33:02.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.411+0000 7f8909ffb640 10 monclient: dump: 2024-04-26T00:33:02.416 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:02.416 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:02.416 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:02.416 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:02.416 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:02.416 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:02.416 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:02.416 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:02.416 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:02.416 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:02.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.411+0000 7f890affd640 4 set_mon_vals no callback set 2024-04-26T00:33:02.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.411+0000 7f891b9cc640 5 monclient: authenticate success, global_id 144255 2024-04-26T00:33:02.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.411+0000 7f891b9cc640 10 monclient: _renew_subs 2024-04-26T00:33:02.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.411+0000 7f891b9cc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:33:02.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.411+0000 7f891b9cc640 1 -- 172.21.15.37:0/1621724363 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f891411d2c0 con 0x7f89140a0260 2024-04-26T00:33:02.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.411+0000 7f891b9cc640 10 monclient: _renew_subs 2024-04-26T00:33:02.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.411+0000 7f891b9cc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:33:02.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.411+0000 7f891b9cc640 1 -- 172.21.15.37:0/1621724363 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f891411bd50 con 0x7f89140a0260 2024-04-26T00:33:02.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.411+0000 7f891b9cc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:02.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.411+0000 7f891b9cc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:02.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.411+0000 7f891b9cc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:33:02.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.411+0000 7f891b9cc640 1 -- 172.21.15.37:0/1621724363 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8914066f40 con 0x7f89140a0260 2024-04-26T00:33:02.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.411+0000 7f8909ffb640 1 -- 172.21.15.37:0/1621724363 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f89100bd020 con 0x7f89140a0260 2024-04-26T00:33:02.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.411+0000 7f8909ffb640 1 --2- 172.21.15.37:0/1621724363 >> v2:172.21.15.45:6812/2379918623 conn(0x7f88f8035e80 0x7f88f8038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:02.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.411+0000 7f8909ffb640 1 -- 172.21.15.37:0/1621724363 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(503..503 src has 370..503) ==== 3915+0+0 (secure 0 0 0) 0x7f89100e2b10 con 0x7f89140a0260 2024-04-26T00:33:02.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.411+0000 7f8918f67640 1 --2- 172.21.15.37:0/1621724363 >> v2:172.21.15.45:6812/2379918623 conn(0x7f88f8035e80 0x7f88f8038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:02.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.411+0000 7f8918f67640 10 monclient: get_auth_request con 0x7f88f8035e80 auth_method 0 2024-04-26T00:33:02.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.415+0000 7f8918f67640 1 --2- 172.21.15.37:0/1621724363 >> v2:172.21.15.45:6812/2379918623 conn(0x7f88f8035e80 0x7f88f8038340 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f89140a06f0 tx=0x7f8904003b50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:02.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.415+0000 7f8909ffb640 1 -- 172.21.15.37:0/1621724363 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f89100c4030 con 0x7f89140a0260 2024-04-26T00:33:02.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.415+0000 7f8909ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:02.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.415+0000 7f8909ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:33:02.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.639+0000 7f891b9cc640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:33:02.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.639+0000 7f891b9cc640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:33:02.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.639+0000 7f891b9cc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:33:02.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.639+0000 7f891b9cc640 1 -- 172.21.15.37:0/1621724363 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f89140729d0 con 0x7f89140a0260 2024-04-26T00:33:02.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.639+0000 7f8909ffb640 1 -- 172.21.15.37:0/1621724363 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f89140a1130 con 0x7f89140a0260 2024-04-26T00:33:02.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.639+0000 7f8909ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:33:02.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.639+0000 7f8909ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:33:02.633 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":96,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:33:02.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.643+0000 7f891b9cc640 1 -- 172.21.15.37:0/1621724363 >> v2:172.21.15.45:6812/2379918623 conn(0x7f88f8035e80 msgr2=0x7f88f8038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:02.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.643+0000 7f891b9cc640 1 --2- 172.21.15.37:0/1621724363 >> v2:172.21.15.45:6812/2379918623 conn(0x7f88f8035e80 0x7f88f8038340 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f89140a06f0 tx=0x7f8904003b50 comp rx=0 tx=0).stop 2024-04-26T00:33:02.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.643+0000 7f891b9cc640 10 monclient: shutdown 2024-04-26T00:33:02.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.643+0000 7f891b9cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:33:02.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.643+0000 7f891b9cc640 1 -- 172.21.15.37:0/1621724363 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89140a0260 msgr2=0x7f89141cee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:02.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.643+0000 7f891b9cc640 1 --2- 172.21.15.37:0/1621724363 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89140a0260 0x7f89141cee40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f891001a7f0 tx=0x7f8910018ee0 comp rx=0 tx=0).stop 2024-04-26T00:33:02.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.643+0000 7f891b9cc640 1 -- 172.21.15.37:0/1621724363 shutdown_connections 2024-04-26T00:33:02.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.643+0000 7f891b9cc640 1 --2- 172.21.15.37:0/1621724363 >> v2:172.21.15.45:6812/2379918623 conn(0x7f88f8035e80 0x7f88f8038340 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:02.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.643+0000 7f891b9cc640 1 --2- 172.21.15.37:0/1621724363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8914135660 0x7f8914131e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:02.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.643+0000 7f891b9cc640 1 --2- 172.21.15.37:0/1621724363 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8914134b30 0x7f8914131900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:02.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.643+0000 7f891b9cc640 1 --2- 172.21.15.37:0/1621724363 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89140a0260 0x7f89141cee40 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:02.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.643+0000 7f891b9cc640 1 -- 172.21.15.37:0/1621724363 >> 172.21.15.37:0/1621724363 conn(0x7f89140a1ad0 msgr2=0x7f89140a3d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:33:02.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.643+0000 7f891b9cc640 1 -- 172.21.15.37:0/1621724363 shutdown_connections 2024-04-26T00:33:02.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:02.643+0000 7f891b9cc640 1 -- 172.21.15.37:0/1621724363 wait complete. 2024-04-26T00:33:05.653 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:33:05.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.795+0000 7fd54bbb5640 10 monclient: get_monmap_and_config 2024-04-26T00:33:05.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.795+0000 7fd54bbb5640 10 monclient: build_initial_monmap 2024-04-26T00:33:05.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.795+0000 7fd54bbb5640 10 monclient: monmap: 2024-04-26T00:33:05.788 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:33:05.788 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:05.788 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:33:05.797040+0000 2024-04-26T00:33:05.788 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:33:05.797040+0000 2024-04-26T00:33:05.788 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:33:05.788 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:05.788 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:33:05.788 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:33:05.789 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:33:05.789 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:33:05.789 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:33:05.789 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:05.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.795+0000 7fd54bbb5640 1 Processor -- start 2024-04-26T00:33:05.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.795+0000 7fd54bbb5640 1 -- start start 2024-04-26T00:33:05.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.795+0000 7fd54bbb5640 10 monclient: init 2024-04-26T00:33:05.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.795+0000 7fd54bbb5640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:05.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.795+0000 7fd54bbb5640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-04-26T00:33:05.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.795+0000 7fd54bbb5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd544133b20 addr v1:172.21.15.37:6789/0 2024-04-26T00:33:05.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.795+0000 7fd54bbb5640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd544133f00 0x7fd544134350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:05.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.795+0000 7fd54bbb5640 10 monclient(hunting): picked mon.noname-b con 0x7fd544133f00 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:33:05.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.795+0000 7fd54bbb5640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd544139050 0x7fd54413b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:05.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.795+0000 7fd54bbb5640 10 monclient(hunting): picked mon.noname-a con 0x7fd544139050 addr v2:172.21.15.37:3300/0 2024-04-26T00:33:05.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.795+0000 7fd54bbb5640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd54409fe20 con 0x7fd544133b20 2024-04-26T00:33:05.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.795+0000 7fd54bbb5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:05.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.795+0000 7fd54bbb5640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fd54409e620 con 0x7fd544139050 2024-04-26T00:33:05.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.795+0000 7fd54bbb5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:05.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.795+0000 7fd54bbb5640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd544072d00 con 0x7fd544133f00 2024-04-26T00:33:05.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.795+0000 7fd54bbb5640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:05.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.795+0000 7fd54bbb5640 10 monclient(hunting): authenticate will time out at 1767.102888s 2024-04-26T00:33:05.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.795+0000 7fd54a152640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd544139050 0x7fd54413b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:05.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.795+0000 7fd549951640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fd544133b20 0x7fd54409e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:46082/0 (socket says 172.21.15.37:46082) 2024-04-26T00:33:05.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.795+0000 7fd549951640 1 -- 172.21.15.37:0/1578853427 learned_addr learned my addr 172.21.15.37:0/1578853427 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:33:05.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.795+0000 7fd549150640 1 --2- 172.21.15.37:0/1578853427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd544133f00 0x7fd544134350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:05.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.795+0000 7fd54a152640 10 monclient(hunting): get_auth_request con 0x7fd544139050 auth_method 0 2024-04-26T00:33:05.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.795+0000 7fd54a152640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:05.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.795+0000 7fd54a152640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:05.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.795+0000 7fd54a152640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:05.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.795+0000 7fd549150640 10 monclient(hunting): get_auth_request con 0x7fd544133f00 auth_method 0 2024-04-26T00:33:05.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.795+0000 7fd549150640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:05.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.795+0000 7fd549150640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:05.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.795+0000 7fd549150640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:05.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.795+0000 7fd54a152640 1 -- 172.21.15.37:0/1578853427 >> v2:172.21.15.37:3300/0 conn(0x7fd544139050 msgr2=0x7fd54413b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:33:05.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.795+0000 7fd54a152640 1 -- 172.21.15.37:0/1578853427 >> v2:172.21.15.37:3300/0 conn(0x7fd544139050 msgr2=0x7fd54413b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:33:05.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.795+0000 7fd54a152640 1 --2- 172.21.15.37:0/1578853427 >> v2:172.21.15.37:3300/0 conn(0x7fd544139050 0x7fd54413b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:33:05.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.795+0000 7fd549150640 1 -- 172.21.15.37:0/1578853427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd544133f00 msgr2=0x7fd544134350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:33:05.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.795+0000 7fd549150640 1 -- 172.21.15.37:0/1578853427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd544133f00 msgr2=0x7fd544134350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:33:05.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.795+0000 7fd549150640 1 --2- 172.21.15.37:0/1578853427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd544133f00 0x7fd544134350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:33:05.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.795+0000 7fd54a152640 1 --2- 172.21.15.37:0/1578853427 >> v2:172.21.15.37:3300/0 conn(0x7fd544139050 0x7fd54413b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:33:05.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.795+0000 7fd549150640 1 --2- 172.21.15.37:0/1578853427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd544133f00 0x7fd544134350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:33:05.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.799+0000 7fd549951640 1 -- 172.21.15.37:0/1578853427 >> v1:172.21.15.37:6789/0 conn(0x7fd544133b20 legacy=0x7fd54409e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:05.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.999+0000 7fd549150640 1 --2- 172.21.15.37:0/1578853427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd544133f00 0x7fd544134350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:05.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.999+0000 7fd54a152640 1 --2- 172.21.15.37:0/1578853427 >> v2:172.21.15.37:3300/0 conn(0x7fd544139050 0x7fd54413b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:05.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.999+0000 7fd549150640 10 monclient(hunting): get_auth_request con 0x7fd544133f00 auth_method 0 2024-04-26T00:33:05.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.999+0000 7fd549150640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:05.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.999+0000 7fd549150640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:05.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.999+0000 7fd549150640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:33:05.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.999+0000 7fd54a152640 10 monclient(hunting): get_auth_request con 0x7fd544139050 auth_method 0 2024-04-26T00:33:05.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.999+0000 7fd54a152640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:05.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.999+0000 7fd54a152640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:05.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.999+0000 7fd54a152640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:33:05.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.999+0000 7fd549150640 1 -- 172.21.15.37:0/1578853427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd544133f00 msgr2=0x7fd544134350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:33:05.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.999+0000 7fd549150640 1 -- 172.21.15.37:0/1578853427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd544133f00 msgr2=0x7fd544134350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:33:05.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.999+0000 7fd549150640 1 --2- 172.21.15.37:0/1578853427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd544133f00 0x7fd544134350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:33:05.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.999+0000 7fd54a152640 1 -- 172.21.15.37:0/1578853427 >> v2:172.21.15.37:3300/0 conn(0x7fd544139050 msgr2=0x7fd54413b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:33:05.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.999+0000 7fd54a152640 1 -- 172.21.15.37:0/1578853427 >> v2:172.21.15.37:3300/0 conn(0x7fd544139050 msgr2=0x7fd54413b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:33:05.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.999+0000 7fd54a152640 1 --2- 172.21.15.37:0/1578853427 >> v2:172.21.15.37:3300/0 conn(0x7fd544139050 0x7fd54413b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:33:05.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.999+0000 7fd549150640 1 --2- 172.21.15.37:0/1578853427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd544133f00 0x7fd544134350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:33:05.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:05.999+0000 7fd54a152640 1 --2- 172.21.15.37:0/1578853427 >> v2:172.21.15.37:3300/0 conn(0x7fd544139050 0x7fd54413b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:33:06.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:06.403+0000 7fd54a152640 1 --2- 172.21.15.37:0/1578853427 >> v2:172.21.15.37:3300/0 conn(0x7fd544139050 0x7fd54413b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:06.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:06.403+0000 7fd549150640 1 --2- 172.21.15.37:0/1578853427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd544133f00 0x7fd544134350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:06.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:06.403+0000 7fd54a152640 10 monclient(hunting): get_auth_request con 0x7fd544139050 auth_method 0 2024-04-26T00:33:06.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:06.403+0000 7fd54a152640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:06.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:06.403+0000 7fd54a152640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:06.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:06.403+0000 7fd54a152640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:33:06.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:06.403+0000 7fd549150640 10 monclient(hunting): get_auth_request con 0x7fd544133f00 auth_method 0 2024-04-26T00:33:06.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:06.403+0000 7fd549150640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:06.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:06.403+0000 7fd549150640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:06.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:06.403+0000 7fd549150640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:33:06.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:06.403+0000 7fd54a152640 1 -- 172.21.15.37:0/1578853427 >> v2:172.21.15.37:3300/0 conn(0x7fd544139050 msgr2=0x7fd54413b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:33:06.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:06.403+0000 7fd54a152640 1 -- 172.21.15.37:0/1578853427 >> v2:172.21.15.37:3300/0 conn(0x7fd544139050 msgr2=0x7fd54413b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:33:06.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:06.403+0000 7fd54a152640 1 --2- 172.21.15.37:0/1578853427 >> v2:172.21.15.37:3300/0 conn(0x7fd544139050 0x7fd54413b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:33:06.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:06.403+0000 7fd54a152640 1 --2- 172.21.15.37:0/1578853427 >> v2:172.21.15.37:3300/0 conn(0x7fd544139050 0x7fd54413b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:33:06.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:06.403+0000 7fd549150640 1 -- 172.21.15.37:0/1578853427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd544133f00 msgr2=0x7fd544134350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:33:06.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:06.403+0000 7fd549150640 1 -- 172.21.15.37:0/1578853427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd544133f00 msgr2=0x7fd544134350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:33:06.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:06.403+0000 7fd549150640 1 --2- 172.21.15.37:0/1578853427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd544133f00 0x7fd544134350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:33:06.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:06.403+0000 7fd549150640 1 --2- 172.21.15.37:0/1578853427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd544133f00 0x7fd544134350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:33:07.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:07.207+0000 7fd54a152640 1 --2- 172.21.15.37:0/1578853427 >> v2:172.21.15.37:3300/0 conn(0x7fd544139050 0x7fd54413b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:07.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:07.207+0000 7fd549150640 1 --2- 172.21.15.37:0/1578853427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd544133f00 0x7fd544134350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:07.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:07.207+0000 7fd54a152640 10 monclient(hunting): get_auth_request con 0x7fd544139050 auth_method 0 2024-04-26T00:33:07.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:07.207+0000 7fd54a152640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:07.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:07.207+0000 7fd54a152640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:07.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:07.207+0000 7fd54a152640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:33:07.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:07.207+0000 7fd549150640 10 monclient(hunting): get_auth_request con 0x7fd544133f00 auth_method 0 2024-04-26T00:33:07.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:07.207+0000 7fd549150640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:07.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:07.207+0000 7fd549150640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:07.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:07.207+0000 7fd549150640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:33:07.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:07.207+0000 7fd54a152640 1 -- 172.21.15.37:0/1578853427 >> v2:172.21.15.37:3300/0 conn(0x7fd544139050 msgr2=0x7fd54413b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:33:07.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:07.207+0000 7fd549150640 1 -- 172.21.15.37:0/1578853427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd544133f00 msgr2=0x7fd544134350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:33:07.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:07.207+0000 7fd54a152640 1 -- 172.21.15.37:0/1578853427 >> v2:172.21.15.37:3300/0 conn(0x7fd544139050 msgr2=0x7fd54413b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:33:07.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:07.207+0000 7fd549150640 1 -- 172.21.15.37:0/1578853427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd544133f00 msgr2=0x7fd544134350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:33:07.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:07.207+0000 7fd54a152640 1 --2- 172.21.15.37:0/1578853427 >> v2:172.21.15.37:3300/0 conn(0x7fd544139050 0x7fd54413b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:33:07.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:07.207+0000 7fd549150640 1 --2- 172.21.15.37:0/1578853427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd544133f00 0x7fd544134350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:33:07.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:07.207+0000 7fd549150640 1 --2- 172.21.15.37:0/1578853427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd544133f00 0x7fd544134350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:33:07.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:07.207+0000 7fd54a152640 1 --2- 172.21.15.37:0/1578853427 >> v2:172.21.15.37:3300/0 conn(0x7fd544139050 0x7fd54413b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:33:08.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:08.795+0000 7fd5377fe640 10 monclient(hunting): tick 2024-04-26T00:33:08.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:08.795+0000 7fd5377fe640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:33:08.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:08.795+0000 7fd5377fe640 1 monclient(hunting): continuing hunt 2024-04-26T00:33:08.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:08.795+0000 7fd5377fe640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:33:08.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:08.795+0000 7fd5377fe640 1 -- 172.21.15.37:0/1578853427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd544133f00 msgr2=0x7fd544134350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:33:08.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:08.795+0000 7fd5377fe640 1 --2- 172.21.15.37:0/1578853427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd544133f00 0x7fd544134350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:08.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:08.795+0000 7fd5377fe640 1 -- 172.21.15.37:0/1578853427 >> v2:172.21.15.37:3300/0 conn(0x7fd544139050 msgr2=0x7fd54413b440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:33:08.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:08.795+0000 7fd5377fe640 1 --2- 172.21.15.37:0/1578853427 >> v2:172.21.15.37:3300/0 conn(0x7fd544139050 0x7fd54413b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:08.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:08.795+0000 7fd5377fe640 1 -- 172.21.15.37:0/1578853427 >> v1:172.21.15.37:6789/0 conn(0x7fd544133b20 legacy=0x7fd54409e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:08.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:08.795+0000 7fd5377fe640 10 monclient: _add_conns ranks=[4,1] 2024-04-26T00:33:08.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:08.795+0000 7fd5377fe640 1 --2- 172.21.15.37:0/1578853427 >> v2:172.21.15.45:3300/0 conn(0x7fd528005960 0x7fd528007de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:08.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:08.795+0000 7fd5377fe640 10 monclient(hunting): picked mon.noname-c con 0x7fd528005960 addr v2:172.21.15.45:3300/0 2024-04-26T00:33:08.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:08.795+0000 7fd5377fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd5280084d0 addr v1:172.21.15.45:6789/0 2024-04-26T00:33:08.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:08.795+0000 7fd5377fe640 1 -- 172.21.15.37:0/1578853427 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd54409ba60 con 0x7fd5280084d0 2024-04-26T00:33:08.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:08.795+0000 7fd5377fe640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:08.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:08.795+0000 7fd5377fe640 1 -- 172.21.15.37:0/1578853427 --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fd54409e620 con 0x7fd528005960 2024-04-26T00:33:08.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:08.795+0000 7fd5377fe640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:08.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:08.795+0000 7fd549150640 1 -- 172.21.15.37:0/1578853427 >> v1:172.21.15.45:6789/0 conn(0x7fd5280084d0 legacy=0x7fd52800a970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:33:08.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:08.795+0000 7fd549951640 1 -- 172.21.15.37:0/1578853427 >> v2:172.21.15.45:3300/0 conn(0x7fd528005960 msgr2=0x7fd528007de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:33:08.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:08.795+0000 7fd549951640 1 --2- 172.21.15.37:0/1578853427 >> v2:172.21.15.45:3300/0 conn(0x7fd528005960 0x7fd528007de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:33:08.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:08.999+0000 7fd549951640 1 -- 172.21.15.37:0/1578853427 >> v2:172.21.15.45:3300/0 conn(0x7fd528005960 msgr2=0x7fd528007de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:33:08.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:08.999+0000 7fd549150640 1 -- 172.21.15.37:0/1578853427 >> v1:172.21.15.45:6789/0 conn(0x7fd5280084d0 legacy=0x7fd52800a970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:33:08.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:08.999+0000 7fd549951640 1 --2- 172.21.15.37:0/1578853427 >> v2:172.21.15.45:3300/0 conn(0x7fd528005960 0x7fd528007de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:33:09.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:09.403+0000 7fd549951640 1 -- 172.21.15.37:0/1578853427 >> v2:172.21.15.45:3300/0 conn(0x7fd528005960 msgr2=0x7fd528007de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:33:09.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:09.403+0000 7fd549150640 1 -- 172.21.15.37:0/1578853427 >> v1:172.21.15.45:6789/0 conn(0x7fd5280084d0 legacy=0x7fd52800a970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:33:09.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:09.403+0000 7fd549951640 1 --2- 172.21.15.37:0/1578853427 >> v2:172.21.15.45:3300/0 conn(0x7fd528005960 0x7fd528007de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:33:10.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:10.207+0000 7fd549951640 1 -- 172.21.15.37:0/1578853427 >> v2:172.21.15.45:3300/0 conn(0x7fd528005960 msgr2=0x7fd528007de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:33:10.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:10.207+0000 7fd549150640 1 -- 172.21.15.37:0/1578853427 >> v1:172.21.15.45:6789/0 conn(0x7fd5280084d0 legacy=0x7fd52800a970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:33:10.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:10.207+0000 7fd549951640 1 --2- 172.21.15.37:0/1578853427 >> v2:172.21.15.45:3300/0 conn(0x7fd528005960 0x7fd528007de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:33:10.642 INFO:tasks.rados.rados.0.smithi045.stdout:2221: done (1 left) 2024-04-26T00:33:10.642 INFO:tasks.rados.rados.0.smithi045.stdout:2222: done (0 left) 2024-04-26T00:33:10.642 INFO:tasks.rados.rados.0.smithi045.stdout:2223: rollback oid 25 current snap is 249 2024-04-26T00:33:10.642 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 25 to 244 2024-04-26T00:33:10.642 INFO:tasks.rados.rados.0.smithi045.stdout:2224: write (excl) oid 31 current snap is 249 2024-04-26T00:33:10.642 INFO:tasks.rados.rados.0.smithi045.stdout:2224: seq_num 807 ranges {763249=659361,2090498=582616,3295061=268188} 2024-04-26T00:33:10.664 INFO:tasks.rados.rados.0.smithi045.stdout:2224: writing smithi04526579-31 from 763249 to 1422610 tid 1 2024-04-26T00:33:10.685 INFO:tasks.rados.rados.0.smithi045.stdout:2224: writing smithi04526579-31 from 2090498 to 2673114 tid 2 2024-04-26T00:33:10.698 INFO:tasks.rados.rados.0.smithi045.stdout:2224: writing smithi04526579-31 from 3295061 to 3563249 tid 3 2024-04-26T00:33:10.698 INFO:tasks.rados.rados.0.smithi045.stdout:2225: snap_remove snap 240 2024-04-26T00:33:11.645 INFO:tasks.rados.rados.0.smithi045.stdout:2225: done (2 left) 2024-04-26T00:33:11.646 INFO:tasks.rados.rados.0.smithi045.stdout:2226: copy_from oid 26 from oid 8 current snap is 249 2024-04-26T00:33:11.646 INFO:tasks.rados.rados.0.smithi045.stdout:2223: finishing rollback tid 0 to smithi04526579-25 2024-04-26T00:33:11.646 INFO:tasks.rados.rados.0.smithi045.stdout:2223: finishing rollback tid 1 to smithi04526579-25 2024-04-26T00:33:11.646 INFO:tasks.rados.rados.0.smithi045.stdout:2223: finishing rollback tid 2 to smithi04526579-25 2024-04-26T00:33:11.646 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 1911 (ObjNum 604 snap 186 seq_num 604) dirty exists 2024-04-26T00:33:11.646 INFO:tasks.rados.rados.0.smithi045.stdout:2224: finishing write tid 1 to smithi04526579-31 2024-04-26T00:33:11.646 INFO:tasks.rados.rados.0.smithi045.stdout:2224: finishing write tid 2 to smithi04526579-31 2024-04-26T00:33:11.646 INFO:tasks.rados.rados.0.smithi045.stdout:2224: finishing write tid 3 to smithi04526579-31 2024-04-26T00:33:11.646 INFO:tasks.rados.rados.0.smithi045.stdout:2224: finishing write tid 4 to smithi04526579-31 2024-04-26T00:33:11.646 INFO:tasks.rados.rados.0.smithi045.stdout:2224: finishing write tid 5 to smithi04526579-31 2024-04-26T00:33:11.646 INFO:tasks.rados.rados.0.smithi045.stdout:2224: oid 31 updating version 0 to 1534 2024-04-26T00:33:11.646 INFO:tasks.rados.rados.0.smithi045.stdout:2224: oid 31 updating version 1534 to 1536 2024-04-26T00:33:11.647 INFO:tasks.rados.rados.0.smithi045.stdout:2224: oid 31 version 1536 is already newer than 1533 2024-04-26T00:33:11.647 INFO:tasks.rados.rados.0.smithi045.stdout:2224: oid 31 version 1536 is already newer than 1535 2024-04-26T00:33:11.647 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 1536 (ObjNum 807 snap 249 seq_num 807) dirty exists 2024-04-26T00:33:11.647 INFO:tasks.rados.rados.0.smithi045.stdout:2224: left oid 31 (ObjNum 807 snap 249 seq_num 807) 2024-04-26T00:33:11.647 INFO:tasks.rados.rados.0.smithi045.stdout:2223: done (2 left) 2024-04-26T00:33:11.647 INFO:tasks.rados.rados.0.smithi045.stdout:2224: done (1 left) 2024-04-26T00:33:11.647 INFO:tasks.rados.rados.0.smithi045.stdout:2227: read oid 31 snap -1 2024-04-26T00:33:11.647 INFO:tasks.rados.rados.0.smithi045.stdout:2227: expect (ObjNum 807 snap 249 seq_num 807) 2024-04-26T00:33:11.647 INFO:tasks.rados.rados.0.smithi045.stdout:2228: rollback oid 49 current snap is 249 2024-04-26T00:33:11.647 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 49 to 238 2024-04-26T00:33:11.648 INFO:tasks.rados.rados.0.smithi045.stdout:2229: read oid 32 snap -1 2024-04-26T00:33:11.648 INFO:tasks.rados.rados.0.smithi045.stdout:2229: expect (ObjNum 769 snap 238 seq_num 769) 2024-04-26T00:33:11.648 INFO:tasks.rados.rados.0.smithi045.stdout:2230: read oid 16 snap -1 2024-04-26T00:33:11.648 INFO:tasks.rados.rados.0.smithi045.stdout:2230: expect (ObjNum 801 snap 246 seq_num 801) 2024-04-26T00:33:11.648 INFO:tasks.rados.rados.0.smithi045.stdout:2231: delete oid 1 current snap is 249 2024-04-26T00:33:11.659 INFO:tasks.rados.rados.0.smithi045.stdout:2226: finishing copy_from to smithi04526579-26 2024-04-26T00:33:11.659 INFO:tasks.rados.rados.0.smithi045.stdout:2226: got expected ENOENT (src dne) 2024-04-26T00:33:11.660 INFO:tasks.rados.rados.0.smithi045.stdout:2226: finishing copy_from racing read to smithi04526579-26 2024-04-26T00:33:11.665 INFO:tasks.rados.rados.0.smithi045.stdout:2226: done (5 left) 2024-04-26T00:33:11.665 INFO:tasks.rados.rados.0.smithi045.stdout:2231: done (4 left) 2024-04-26T00:33:11.665 INFO:tasks.rados.rados.0.smithi045.stdout:2232: write (excl) oid 29 current snap is 249 2024-04-26T00:33:11.666 INFO:tasks.rados.rados.0.smithi045.stdout:2232: seq_num 809 ranges {796863=409859,1807462=586285,3066260=530603} 2024-04-26T00:33:11.677 INFO:tasks.rados.rados.0.smithi045.stdout:2232: writing smithi04526579-29 from 796863 to 1206722 tid 1 2024-04-26T00:33:11.693 INFO:tasks.rados.rados.0.smithi045.stdout:2232: writing smithi04526579-29 from 1807462 to 2393747 tid 2 2024-04-26T00:33:11.712 INFO:tasks.rados.rados.0.smithi045.stdout:2232: writing smithi04526579-29 from 3066260 to 3596863 tid 3 2024-04-26T00:33:11.712 INFO:tasks.rados.rados.0.smithi045.stdout:2233: copy_from oid 45 from oid 1 current snap is 249 2024-04-26T00:33:11.712 INFO:tasks.rados.rados.0.smithi045.stdout:2228: finishing rollback tid 0 to smithi04526579-49 2024-04-26T00:33:11.712 INFO:tasks.rados.rados.0.smithi045.stdout:2228: finishing rollback tid 1 to smithi04526579-49 2024-04-26T00:33:11.713 INFO:tasks.rados.rados.0.smithi045.stdout:2227: expect (ObjNum 807 snap 249 seq_num 807) 2024-04-26T00:33:11.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.795+0000 7fd5377fe640 10 monclient(hunting): tick 2024-04-26T00:33:11.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.795+0000 7fd5377fe640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:33:11.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.795+0000 7fd5377fe640 1 monclient(hunting): continuing hunt 2024-04-26T00:33:11.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.795+0000 7fd5377fe640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:33:11.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.795+0000 7fd5377fe640 1 -- 172.21.15.37:0/1578853427 >> v2:172.21.15.45:3300/0 conn(0x7fd528005960 msgr2=0x7fd528007de0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:33:11.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.795+0000 7fd5377fe640 1 --2- 172.21.15.37:0/1578853427 >> v2:172.21.15.45:3300/0 conn(0x7fd528005960 0x7fd528007de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:11.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.795+0000 7fd5377fe640 1 -- 172.21.15.37:0/1578853427 >> v1:172.21.15.45:6789/0 conn(0x7fd5280084d0 legacy=0x7fd52800a970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:33:11.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.795+0000 7fd5377fe640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-04-26T00:33:11.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.795+0000 7fd54a152640 1 -- 172.21.15.37:0/1578853427 reap_dead start 2024-04-26T00:33:11.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.795+0000 7fd5377fe640 1 --2- 172.21.15.37:0/1578853427 >> v2:172.21.15.37:3300/0 conn(0x7fd544139050 0x7fd52800c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:11.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.795+0000 7fd5377fe640 10 monclient(hunting): picked mon.noname-a con 0x7fd544139050 addr v2:172.21.15.37:3300/0 2024-04-26T00:33:11.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.795+0000 7fd5377fe640 1 --2- 172.21.15.37:0/1578853427 >> v2:172.21.15.45:3300/0 conn(0x7fd528005960 0x7fd528007de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:11.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.795+0000 7fd5377fe640 10 monclient(hunting): picked mon.noname-c con 0x7fd528005960 addr v2:172.21.15.45:3300/0 2024-04-26T00:33:11.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.795+0000 7fd5377fe640 1 --2- 172.21.15.37:0/1578853427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd52800c760 0x7fd52800cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:11.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.795+0000 7fd5377fe640 10 monclient(hunting): picked mon.noname-b con 0x7fd52800c760 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:33:11.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.795+0000 7fd5377fe640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:11.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.795+0000 7fd5377fe640 1 -- 172.21.15.37:0/1578853427 --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fd54409e620 con 0x7fd544139050 2024-04-26T00:33:11.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.795+0000 7fd5377fe640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:11.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.795+0000 7fd5377fe640 1 -- 172.21.15.37:0/1578853427 --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fd544072d00 con 0x7fd528005960 2024-04-26T00:33:11.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.795+0000 7fd5377fe640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:11.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.795+0000 7fd5377fe640 1 -- 172.21.15.37:0/1578853427 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd52800aa10 con 0x7fd52800c760 2024-04-26T00:33:11.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.795+0000 7fd5377fe640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:11.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.795+0000 7fd549150640 1 -- 172.21.15.37:0/1578853427 >> v2:172.21.15.45:3300/0 conn(0x7fd528005960 msgr2=0x7fd528007de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:33:11.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.795+0000 7fd549150640 1 --2- 172.21.15.37:0/1578853427 >> v2:172.21.15.45:3300/0 conn(0x7fd528005960 0x7fd528007de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:33:11.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.795+0000 7fd549951640 1 --2- 172.21.15.37:0/1578853427 >> v2:172.21.15.37:3300/0 conn(0x7fd544139050 0x7fd52800c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:11.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.799+0000 7fd549951640 10 monclient(hunting): get_auth_request con 0x7fd544139050 auth_method 0 2024-04-26T00:33:11.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.799+0000 7fd549951640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:11.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.799+0000 7fd549951640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:11.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.799+0000 7fd54a152640 1 --2- 172.21.15.37:0/1578853427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd52800c760 0x7fd52800cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:11.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.799+0000 7fd549951640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:11.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.799+0000 7fd54a152640 10 monclient(hunting): get_auth_request con 0x7fd52800c760 auth_method 0 2024-04-26T00:33:11.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.799+0000 7fd54a152640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:11.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.799+0000 7fd54a152640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:11.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.799+0000 7fd54a152640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:11.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.799+0000 7fd54a152640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:11.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.799+0000 7fd54a152640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:11.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.799+0000 7fd54a152640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:11.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.799+0000 7fd549951640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:11.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.799+0000 7fd549951640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:11.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.799+0000 7fd549951640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:11.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.799+0000 7fd54a152640 10 monclient(hunting): handle_auth_done global_id 154145 payload 274 2024-04-26T00:33:11.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.799+0000 7fd54a152640 1 -- 172.21.15.37:0/1578853427 >> v2:172.21.15.45:3300/0 conn(0x7fd528005960 msgr2=0x7fd528007de0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:33:11.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.799+0000 7fd54a152640 1 --2- 172.21.15.37:0/1578853427 >> v2:172.21.15.45:3300/0 conn(0x7fd528005960 0x7fd528007de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:11.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.799+0000 7fd54a152640 1 -- 172.21.15.37:0/1578853427 >> v2:172.21.15.37:3300/0 conn(0x7fd544139050 msgr2=0x7fd52800c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:33:11.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.799+0000 7fd54a152640 1 --2- 172.21.15.37:0/1578853427 >> v2:172.21.15.37:3300/0 conn(0x7fd544139050 0x7fd52800c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:11.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.799+0000 7fd54a152640 10 monclient: _finish_hunting 0 2024-04-26T00:33:11.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.799+0000 7fd54a152640 1 monclient: found mon.noname-b 2024-04-26T00:33:11.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.799+0000 7fd54a152640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:11.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.799+0000 7fd54a152640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:33:11.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.799+0000 7fd54a152640 1 -- 172.21.15.37:0/1578853427 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd52800d260 con 0x7fd52800c760 2024-04-26T00:33:11.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.799+0000 7fd549951640 1 --2- 172.21.15.37:0/1578853427 >> v2:172.21.15.37:3300/0 conn(0x7fd544139050 0x7fd52800c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:33:11.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.807+0000 7fd54a152640 1 -- 172.21.15.37:0/1578853427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd52800c760 msgr2=0x7fd52800cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:11.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.807+0000 7fd54a152640 1 --2- 172.21.15.37:0/1578853427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd52800c760 0x7fd52800cbc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd544072e80 tx=0x7fd53c01d2a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c235622774b4e75 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:11.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.807+0000 7fd54894f640 1 -- 172.21.15.37:0/1578853427 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd53c0b3070 con 0x7fd52800c760 2024-04-26T00:33:11.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.807+0000 7fd54894f640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:11.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.807+0000 7fd54894f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:33:11.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.807+0000 7fd54894f640 10 monclient: dump: 2024-04-26T00:33:11.800 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:11.800 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:11.800 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:11.800 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:11.800 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:11.800 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:11.801 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:11.801 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:11.801 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:11.801 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:11.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.807+0000 7fd54894f640 10 monclient: _finish_auth 0 2024-04-26T00:33:11.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.807+0000 7fd54894f640 10 monclient: _check_auth_tickets 2024-04-26T00:33:11.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.807+0000 7fd54894f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:11.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.807+0000 7fd54894f640 1 -- 172.21.15.37:0/1578853427 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd53c0ae410 con 0x7fd52800c760 2024-04-26T00:33:11.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.807+0000 7fd54894f640 10 monclient: handle_config config(0 keys) 2024-04-26T00:33:11.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.807+0000 7fd54894f640 1 -- 172.21.15.37:0/1578853427 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd53c0ae600 con 0x7fd52800c760 2024-04-26T00:33:11.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.807+0000 7fd54894f640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:11.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.807+0000 7fd54894f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:33:11.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.807+0000 7fd54894f640 10 monclient: dump: 2024-04-26T00:33:11.802 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:11.802 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:11.802 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:11.802 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:11.802 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:11.802 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:11.802 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:11.802 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:11.802 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:11.802 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:11.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.807+0000 7fd54bbb5640 5 monclient: authenticate success, global_id 154145 2024-04-26T00:33:11.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.807+0000 7fd54bbb5640 10 monclient: get_monmap_and_config success 2024-04-26T00:33:11.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.807+0000 7fd54bbb5640 4 set_mon_vals no callback set 2024-04-26T00:33:11.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.807+0000 7fd54bbb5640 10 monclient: shutdown 2024-04-26T00:33:11.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.807+0000 7fd54bbb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:33:11.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.807+0000 7fd54bbb5640 1 -- 172.21.15.37:0/1578853427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd52800c760 msgr2=0x7fd52800cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:11.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.807+0000 7fd54bbb5640 1 --2- 172.21.15.37:0/1578853427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd52800c760 0x7fd52800cbc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd544072e80 tx=0x7fd53c01d2a0 comp rx=0 tx=0).stop 2024-04-26T00:33:11.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.807+0000 7fd54bbb5640 1 -- 172.21.15.37:0/1578853427 shutdown_connections 2024-04-26T00:33:11.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.807+0000 7fd54bbb5640 1 --2- 172.21.15.37:0/1578853427 >> v2:172.21.15.45:3300/0 conn(0x7fd528005960 0x7fd528007de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:11.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.807+0000 7fd54bbb5640 1 --2- 172.21.15.37:0/1578853427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd52800c760 0x7fd52800cbc0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:11.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.807+0000 7fd54bbb5640 1 --2- 172.21.15.37:0/1578853427 >> v2:172.21.15.37:3300/0 conn(0x7fd544139050 0x7fd52800c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:11.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.807+0000 7fd54bbb5640 1 -- 172.21.15.37:0/1578853427 >> 172.21.15.37:0/1578853427 conn(0x7fd5440a1ad0 msgr2=0x7fd5440a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:33:11.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.807+0000 7fd54bbb5640 1 -- 172.21.15.37:0/1578853427 shutdown_connections 2024-04-26T00:33:11.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.807+0000 7fd54bbb5640 1 -- 172.21.15.37:0/1578853427 wait complete. 2024-04-26T00:33:11.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd54bbb5640 10 monclient: build_initial_monmap 2024-04-26T00:33:11.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd54bbb5640 10 monclient: monmap: 2024-04-26T00:33:11.805 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:33:11.805 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:33:11.805 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:33:11.805 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:33:11.805 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:33:11.805 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:33:11.805 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:33:11.805 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:33:11.805 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:33:11.805 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:11.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd54bbb5640 1 Processor -- start 2024-04-26T00:33:11.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd54bbb5640 1 -- start start 2024-04-26T00:33:11.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd54bbb5640 10 monclient: init 2024-04-26T00:33:11.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd54bbb5640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:11.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd54bbb5640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:33:11.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd54bbb5640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd528005960 0x7fd54411bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:11.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd54bbb5640 10 monclient(hunting): picked mon.noname-c con 0x7fd528005960 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:33:11.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd54bbb5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd52800c760 0x7fd54412fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:11.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd54bbb5640 10 monclient(hunting): picked mon.noname-b con 0x7fd52800c760 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:33:11.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd54bbb5640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd544139050 0x7fd54412fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:11.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd54bbb5640 10 monclient(hunting): picked mon.noname-a con 0x7fd544139050 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:33:11.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd54bbb5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:11.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd54bbb5640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd5441308f0 con 0x7fd544139050 2024-04-26T00:33:11.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd54bbb5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:11.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd54bbb5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd544130770 con 0x7fd52800c760 2024-04-26T00:33:11.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd54bbb5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:11.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd54bbb5640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd544130a70 con 0x7fd528005960 2024-04-26T00:33:11.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd54bbb5640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:11.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd54bbb5640 10 monclient(hunting): authenticate will time out at 1773.117920s 2024-04-26T00:33:11.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd54a152640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd544139050 0x7fd54412fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:11.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd549150640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd52800c760 msgr2=0x7fd54412fab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:33:11.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd549951640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd528005960 0x7fd54411bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:11.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd54a152640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd544139050 0x7fd54412fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:45142/0 (socket says 172.21.15.37:45142) 2024-04-26T00:33:11.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd54a152640 1 -- 172.21.15.37:0/1827538983 learned_addr learned my addr 172.21.15.37:0/1827538983 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:33:11.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd54a152640 10 monclient(hunting): get_auth_request con 0x7fd544139050 auth_method 0 2024-04-26T00:33:11.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd54a152640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:11.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd54a152640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:11.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd54a152640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:11.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd549951640 10 monclient(hunting): get_auth_request con 0x7fd528005960 auth_method 0 2024-04-26T00:33:11.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd549951640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:11.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd549951640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:11.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd549951640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:11.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd549150640 1 --2- 172.21.15.37:0/1827538983 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd52800c760 0x7fd54412fab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:33:11.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd54a152640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:11.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd54a152640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:11.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd54a152640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:11.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd549951640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:11.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd549951640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:11.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd549951640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:11.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd549951640 10 monclient(hunting): handle_auth_done global_id 154148 payload 824 2024-04-26T00:33:11.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd549951640 1 -- 172.21.15.37:0/1827538983 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd52800c760 msgr2=0x7fd54412fab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:33:11.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd549951640 1 --2- 172.21.15.37:0/1827538983 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd52800c760 0x7fd54412fab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:11.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd549951640 1 -- 172.21.15.37:0/1827538983 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd544139050 msgr2=0x7fd54412fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:11.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd549951640 1 --2- 172.21.15.37:0/1827538983 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd544139050 0x7fd54412fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:11.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd549951640 10 monclient: _finish_hunting 0 2024-04-26T00:33:11.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd549951640 1 monclient: found mon.noname-c 2024-04-26T00:33:11.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd549951640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:11.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd549951640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:33:11.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd549951640 1 -- 172.21.15.37:0/1827538983 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd54411c010 con 0x7fd528005960 2024-04-26T00:33:11.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.811+0000 7fd54a152640 1 --2- 172.21.15.37:0/1827538983 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd544139050 0x7fd54412fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:33:11.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.815+0000 7fd549951640 1 -- 172.21.15.37:0/1827538983 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd528005960 msgr2=0x7fd54411bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:11.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.815+0000 7fd549951640 1 --2- 172.21.15.37:0/1827538983 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd528005960 0x7fd54411bad0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd540000c10 tx=0x7fd540007570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:11.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.815+0000 7fd5367fc640 1 -- 172.21.15.37:0/1827538983 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd54000fba0 con 0x7fd528005960 2024-04-26T00:33:11.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.815+0000 7fd5367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:11.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.815+0000 7fd5367fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:33:11.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.815+0000 7fd5367fc640 10 monclient: dump: 2024-04-26T00:33:11.812 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:11.812 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:11.812 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:11.812 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:11.812 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:11.813 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:11.813 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:11.813 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:11.813 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:11.813 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:11.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.815+0000 7fd5367fc640 10 monclient: _finish_auth 0 2024-04-26T00:33:11.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.815+0000 7fd5367fc640 10 monclient: _check_auth_tickets 2024-04-26T00:33:11.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.815+0000 7fd5367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:11.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.815+0000 7fd5367fc640 1 -- 172.21.15.37:0/1827538983 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd540018070 con 0x7fd528005960 2024-04-26T00:33:11.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.815+0000 7fd54bbb5640 5 monclient: authenticate success, global_id 154148 2024-04-26T00:33:11.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.815+0000 7fd54bbb5640 10 monclient: _renew_subs 2024-04-26T00:33:11.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.815+0000 7fd54bbb5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:33:11.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.815+0000 7fd54bbb5640 1 -- 172.21.15.37:0/1827538983 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd54411e480 con 0x7fd528005960 2024-04-26T00:33:11.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.815+0000 7fd5367fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:33:11.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.815+0000 7fd5367fc640 1 -- 172.21.15.37:0/1827538983 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5400133f0 con 0x7fd528005960 2024-04-26T00:33:11.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.815+0000 7fd5367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:11.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.815+0000 7fd5367fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:33:11.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.815+0000 7fd5367fc640 10 monclient: dump: 2024-04-26T00:33:11.814 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:11.814 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:11.814 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:11.814 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:11.814 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:11.814 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:11.814 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:11.814 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:11.814 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:11.814 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:11.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.815+0000 7fd537fff640 4 set_mon_vals no callback set 2024-04-26T00:33:11.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.815+0000 7fd54bbb5640 10 monclient: _renew_subs 2024-04-26T00:33:11.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.815+0000 7fd54bbb5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:33:11.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.815+0000 7fd54bbb5640 1 -- 172.21.15.37:0/1827538983 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd54411dbe0 con 0x7fd528005960 2024-04-26T00:33:11.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.815+0000 7fd5367fc640 1 -- 172.21.15.37:0/1827538983 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd540013590 con 0x7fd528005960 2024-04-26T00:33:11.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.815+0000 7fd54bbb5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:11.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.815+0000 7fd54bbb5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:11.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.815+0000 7fd54bbb5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:33:11.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.815+0000 7fd54bbb5640 1 -- 172.21.15.37:0/1827538983 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd544066f40 con 0x7fd528005960 2024-04-26T00:33:11.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.815+0000 7fd5367fc640 1 --2- 172.21.15.37:0/1827538983 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd528039470 0x7fd52803b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:11.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.815+0000 7fd5367fc640 1 -- 172.21.15.37:0/1827538983 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(506..506 src has 370..506) ==== 3959+0+0 (secure 0 0 0) 0x7fd540048a20 con 0x7fd528005960 2024-04-26T00:33:11.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.815+0000 7fd549150640 1 --2- 172.21.15.37:0/1827538983 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd528039470 0x7fd52803b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:11.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.815+0000 7fd549150640 10 monclient: get_auth_request con 0x7fd528039470 auth_method 0 2024-04-26T00:33:11.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.819+0000 7fd549150640 1 --2- 172.21.15.37:0/1827538983 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd528039470 0x7fd52803b930 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fd538006d30 tx=0x7fd538006a30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:11.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.819+0000 7fd5367fc640 1 -- 172.21.15.37:0/1827538983 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fd54409fe20 con 0x7fd528005960 2024-04-26T00:33:11.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.819+0000 7fd5367fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:11.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:11.819+0000 7fd5367fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:33:11.878 INFO:tasks.rados.rados.0.smithi045.stdout:2229: expect (ObjNum 769 snap 238 seq_num 769) 2024-04-26T00:33:12.011 INFO:tasks.rados.rados.0.smithi045.stdout:2228: finishing rollback tid 2 to smithi04526579-49 2024-04-26T00:33:12.012 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 1363 (ObjNum 724 snap 222 seq_num 724) dirty exists 2024-04-26T00:33:12.012 INFO:tasks.rados.rados.0.smithi045.stdout:2232: finishing write tid 1 to smithi04526579-29 2024-04-26T00:33:12.012 INFO:tasks.rados.rados.0.smithi045.stdout:2230: expect (ObjNum 801 snap 246 seq_num 801) 2024-04-26T00:33:12.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.051+0000 7fd54bbb5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:33:12.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.051+0000 7fd54bbb5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:33:12.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.055+0000 7fd54bbb5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:33:12.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.055+0000 7fd54bbb5640 1 -- 172.21.15.37:0/1827538983 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd54406b9a0 con 0x7fd528005960 2024-04-26T00:33:12.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.055+0000 7fd5367fc640 1 -- 172.21.15.37:0/1827538983 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7fd540047750 con 0x7fd528005960 2024-04-26T00:33:12.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.055+0000 7fd5367fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:33:12.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.055+0000 7fd5367fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:33:12.048 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":100,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:33:12.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.055+0000 7fd54bbb5640 1 -- 172.21.15.37:0/1827538983 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd528039470 msgr2=0x7fd52803b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:12.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.055+0000 7fd54bbb5640 1 --2- 172.21.15.37:0/1827538983 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd528039470 0x7fd52803b930 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fd538006d30 tx=0x7fd538006a30 comp rx=0 tx=0).stop 2024-04-26T00:33:12.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.055+0000 7fd54bbb5640 10 monclient: shutdown 2024-04-26T00:33:12.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.055+0000 7fd54bbb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:33:12.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.055+0000 7fd54bbb5640 1 -- 172.21.15.37:0/1827538983 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd528005960 msgr2=0x7fd54411bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:12.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.055+0000 7fd54bbb5640 1 --2- 172.21.15.37:0/1827538983 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd528005960 0x7fd54411bad0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd540000c10 tx=0x7fd540007570 comp rx=0 tx=0).stop 2024-04-26T00:33:12.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.055+0000 7fd54bbb5640 1 -- 172.21.15.37:0/1827538983 shutdown_connections 2024-04-26T00:33:12.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.055+0000 7fd54bbb5640 1 --2- 172.21.15.37:0/1827538983 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd528039470 0x7fd52803b930 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:12.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.055+0000 7fd54bbb5640 1 --2- 172.21.15.37:0/1827538983 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd544139050 0x7fd54412fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:12.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.055+0000 7fd54bbb5640 1 --2- 172.21.15.37:0/1827538983 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd52800c760 0x7fd54412fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:12.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.055+0000 7fd54bbb5640 1 --2- 172.21.15.37:0/1827538983 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd528005960 0x7fd54411bad0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:12.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.055+0000 7fd54bbb5640 1 -- 172.21.15.37:0/1827538983 >> 172.21.15.37:0/1827538983 conn(0x7fd5440a1ad0 msgr2=0x7fd54409c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:33:12.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.055+0000 7fd54bbb5640 1 -- 172.21.15.37:0/1827538983 shutdown_connections 2024-04-26T00:33:12.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.055+0000 7fd54bbb5640 1 -- 172.21.15.37:0/1827538983 wait complete. 2024-04-26T00:33:12.067 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-26T00:33:12.067 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-04-26T00:33:12.068 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:33:12.123 INFO:tasks.rados.rados.0.smithi045.stdout:2232: finishing write tid 2 to smithi04526579-29 2024-04-26T00:33:12.123 INFO:tasks.rados.rados.0.smithi045.stdout:2233: finishing copy_from to smithi04526579-45 2024-04-26T00:33:12.123 INFO:tasks.rados.rados.0.smithi045.stdout:2233: got expected ENOENT (src dne) 2024-04-26T00:33:12.123 INFO:tasks.rados.rados.0.smithi045.stdout:2233: finishing copy_from racing read to smithi04526579-45 2024-04-26T00:33:12.123 INFO:tasks.rados.rados.0.smithi045.stdout:2232: finishing write tid 3 to smithi04526579-29 2024-04-26T00:33:12.123 INFO:tasks.rados.rados.0.smithi045.stdout:2232: finishing write tid 4 to smithi04526579-29 2024-04-26T00:33:12.123 INFO:tasks.rados.rados.0.smithi045.stdout:2232: finishing write tid 5 to smithi04526579-29 2024-04-26T00:33:12.123 INFO:tasks.rados.rados.0.smithi045.stdout:2232: oid 29 updating version 0 to 1528 2024-04-26T00:33:12.124 INFO:tasks.rados.rados.0.smithi045.stdout:2232: oid 29 updating version 1528 to 1529 2024-04-26T00:33:12.124 INFO:tasks.rados.rados.0.smithi045.stdout:2232: oid 29 updating version 1529 to 1531 2024-04-26T00:33:12.124 INFO:tasks.rados.rados.0.smithi045.stdout:2232: oid 29 version 1531 is already newer than 1530 2024-04-26T00:33:12.124 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 1531 (ObjNum 809 snap 249 seq_num 809) dirty exists 2024-04-26T00:33:12.124 INFO:tasks.rados.rados.0.smithi045.stdout:2232: left oid 29 (ObjNum 809 snap 249 seq_num 809) 2024-04-26T00:33:12.124 INFO:tasks.rados.rados.0.smithi045.stdout:2227: done (5 left) 2024-04-26T00:33:12.124 INFO:tasks.rados.rados.0.smithi045.stdout:2228: done (4 left) 2024-04-26T00:33:12.124 INFO:tasks.rados.rados.0.smithi045.stdout:2229: done (3 left) 2024-04-26T00:33:12.124 INFO:tasks.rados.rados.0.smithi045.stdout:2230: done (2 left) 2024-04-26T00:33:12.124 INFO:tasks.rados.rados.0.smithi045.stdout:2232: done (1 left) 2024-04-26T00:33:12.124 INFO:tasks.rados.rados.0.smithi045.stdout:2233: done (0 left) 2024-04-26T00:33:12.124 INFO:tasks.rados.rados.0.smithi045.stdout:2234: rollback oid 1 current snap is 249 2024-04-26T00:33:12.124 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 1 to 234 2024-04-26T00:33:12.124 INFO:tasks.rados.rados.0.smithi045.stdout:2235: read oid 24 snap 219 2024-04-26T00:33:12.124 INFO:tasks.rados.rados.0.smithi045.stdout:2235: expect (ObjNum 582 snap 180 seq_num 582) 2024-04-26T00:33:12.125 INFO:tasks.rados.rados.0.smithi045.stdout:2236: snap_create 2024-04-26T00:33:12.129 INFO:tasks.rados.rados.0.smithi045.stdout:2234: finishing rollback tid 1 to smithi04526579-1 2024-04-26T00:33:12.129 INFO:tasks.rados.rados.0.smithi045.stdout:2234: finishing rollback tid 2 to smithi04526579-1 2024-04-26T00:33:12.129 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 1584 (ObjNum 661 snap 205 seq_num 661) dirty exists 2024-04-26T00:33:12.135 INFO:tasks.rados.rados.0.smithi045.stdout:2235: expect (ObjNum 582 snap 180 seq_num 582) 2024-04-26T00:33:12.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc822e0d640 10 monclient: get_monmap_and_config 2024-04-26T00:33:12.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc822e0d640 10 monclient: build_initial_monmap 2024-04-26T00:33:12.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc822e0d640 10 monclient: monmap: 2024-04-26T00:33:12.196 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:33:12.196 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:12.196 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:33:12.204952+0000 2024-04-26T00:33:12.196 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:33:12.204952+0000 2024-04-26T00:33:12.196 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:33:12.196 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:12.196 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:33:12.196 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:33:12.197 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:33:12.197 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:33:12.197 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:33:12.197 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:12.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc822e0d640 1 Processor -- start 2024-04-26T00:33:12.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc822e0d640 1 -- start start 2024-04-26T00:33:12.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc822e0d640 10 monclient: init 2024-04-26T00:33:12.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc822e0d640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:12.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc822e0d640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-04-26T00:33:12.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc822e0d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc81c0a3e70 0x7fc81c0a4250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:12.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc822e0d640 10 monclient(hunting): picked mon.noname-c con 0x7fc81c0a3e70 addr v2:172.21.15.45:3300/0 2024-04-26T00:33:12.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc822e0d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc81c09c620 addr v1:172.21.15.45:6789/0 2024-04-26T00:33:12.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc822e0d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc81c0a5a70 0x7fc81c0a4790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:12.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc822e0d640 10 monclient(hunting): picked mon.noname-a con 0x7fc81c0a5a70 addr v2:172.21.15.37:3300/0 2024-04-26T00:33:12.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc822e0d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc81c09b410 con 0x7fc81c09c620 2024-04-26T00:33:12.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc822e0d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:12.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc822e0d640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fc81c0a5460 con 0x7fc81c0a5a70 2024-04-26T00:33:12.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc822e0d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:12.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc822e0d640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc81c0a4cd0 con 0x7fc81c0a3e70 2024-04-26T00:33:12.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc822e0d640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:12.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc822e0d640 10 monclient(hunting): authenticate will time out at 1773.510923s 2024-04-26T00:33:12.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc820ba9640 1 -- >> v2:172.21.15.45:3300/0 conn(0x7fc81c0a3e70 msgr2=0x7fc81c0a4250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:33:12.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc820ba9640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc81c0a3e70 0x7fc81c0a4250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:33:12.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc81bfff640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7fc81c09c620 legacy=0x7fc81c133c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:33:12.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc8213aa640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc81c0a5a70 0x7fc81c0a4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:12.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc8213aa640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc81c0a5a70 0x7fc81c0a4790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:45152/0 (socket says 172.21.15.37:45152) 2024-04-26T00:33:12.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc8213aa640 1 -- 172.21.15.37:0/3493935214 learned_addr learned my addr 172.21.15.37:0/3493935214 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:33:12.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc8213aa640 10 monclient(hunting): get_auth_request con 0x7fc81c0a5a70 auth_method 0 2024-04-26T00:33:12.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc8213aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:12.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc8213aa640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:12.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc8213aa640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:12.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc8213aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:12.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc8213aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:12.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc8213aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:12.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc8213aa640 10 monclient(hunting): handle_auth_done global_id 144264 payload 274 2024-04-26T00:33:12.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc8213aa640 1 -- 172.21.15.37:0/3493935214 >> v2:172.21.15.45:3300/0 conn(0x7fc81c0a3e70 msgr2=0x7fc81c0a4250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:33:12.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc8213aa640 1 --2- 172.21.15.37:0/3493935214 >> v2:172.21.15.45:3300/0 conn(0x7fc81c0a3e70 0x7fc81c0a4250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:12.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc8213aa640 1 -- 172.21.15.37:0/3493935214 >> v1:172.21.15.45:6789/0 conn(0x7fc81c09c620 legacy=0x7fc81c133c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:33:12.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc8213aa640 10 monclient: _finish_hunting 0 2024-04-26T00:33:12.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc8213aa640 1 monclient: found mon.noname-a 2024-04-26T00:33:12.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc8213aa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:12.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc8213aa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:33:12.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.203+0000 7fc8213aa640 1 -- 172.21.15.37:0/3493935214 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc81c09b960 con 0x7fc81c0a5a70 2024-04-26T00:33:12.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc8213aa640 1 -- 172.21.15.37:0/3493935214 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc81c0a5a70 msgr2=0x7fc81c0a4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:12.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc8213aa640 1 --2- 172.21.15.37:0/3493935214 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc81c0a5a70 0x7fc81c0a4790 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc81c0a5a40 tx=0x7fc81401c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d6ede689ddd19bb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:12.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc81b7fe640 1 -- 172.21.15.37:0/3493935214 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8140b0070 con 0x7fc81c0a5a70 2024-04-26T00:33:12.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc81b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:12.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc81b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:33:12.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc81b7fe640 10 monclient: dump: 2024-04-26T00:33:12.203 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:12.203 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:12.204 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:12.204 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:12.204 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:12.204 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:12.204 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:12.204 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:12.204 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:12.204 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:12.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc81b7fe640 10 monclient: _finish_auth 0 2024-04-26T00:33:12.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc81b7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:33:12.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc81b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:12.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc81b7fe640 1 -- 172.21.15.37:0/3493935214 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8140ab600 con 0x7fc81c0a5a70 2024-04-26T00:33:12.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc822e0d640 5 monclient: authenticate success, global_id 144264 2024-04-26T00:33:12.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc822e0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:33:12.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc81b7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:33:12.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc81b7fe640 1 -- 172.21.15.37:0/3493935214 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8140ab9d0 con 0x7fc81c0a5a70 2024-04-26T00:33:12.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc822e0d640 10 monclient: get_monmap_and_config success 2024-04-26T00:33:12.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc822e0d640 4 set_mon_vals no callback set 2024-04-26T00:33:12.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc81b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:12.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc81b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:33:12.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc81b7fe640 10 monclient: dump: 2024-04-26T00:33:12.205 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:12.205 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:12.205 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:12.205 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:12.205 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:12.205 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:12.205 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:12.205 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:12.205 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:12.205 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:12.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc822e0d640 10 monclient: shutdown 2024-04-26T00:33:12.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc822e0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:33:12.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc822e0d640 1 -- 172.21.15.37:0/3493935214 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc81c0a5a70 msgr2=0x7fc81c0a4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:12.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc822e0d640 1 --2- 172.21.15.37:0/3493935214 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc81c0a5a70 0x7fc81c0a4790 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc81c0a5a40 tx=0x7fc81401c3c0 comp rx=0 tx=0).stop 2024-04-26T00:33:12.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc822e0d640 1 -- 172.21.15.37:0/3493935214 shutdown_connections 2024-04-26T00:33:12.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc822e0d640 1 --2- 172.21.15.37:0/3493935214 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc81c0a5a70 0x7fc81c0a4790 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:12.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc822e0d640 1 --2- 172.21.15.37:0/3493935214 >> v2:172.21.15.45:3300/0 conn(0x7fc81c0a3e70 0x7fc81c0a4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:12.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc822e0d640 1 -- 172.21.15.37:0/3493935214 >> 172.21.15.37:0/3493935214 conn(0x7fc81c0a6140 msgr2=0x7fc81c0a6550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:33:12.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc822e0d640 1 -- 172.21.15.37:0/3493935214 shutdown_connections 2024-04-26T00:33:12.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc822e0d640 1 -- 172.21.15.37:0/3493935214 wait complete. 2024-04-26T00:33:12.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc822e0d640 10 monclient: build_initial_monmap 2024-04-26T00:33:12.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc822e0d640 10 monclient: monmap: 2024-04-26T00:33:12.211 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:33:12.211 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:33:12.211 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:33:12.211 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:33:12.211 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:33:12.211 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:33:12.211 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:33:12.211 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:33:12.212 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:33:12.212 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:12.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc822e0d640 1 Processor -- start 2024-04-26T00:33:12.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc822e0d640 1 -- start start 2024-04-26T00:33:12.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc822e0d640 10 monclient: init 2024-04-26T00:33:12.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc822e0d640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:12.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc822e0d640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:33:12.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc822e0d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc81c0a3e70 0x7fc81c1cf7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:12.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc822e0d640 10 monclient(hunting): picked mon.noname-c con 0x7fc81c0a3e70 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:33:12.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc822e0d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc81c0a5a70 0x7fc81c13e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:12.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc822e0d640 10 monclient(hunting): picked mon.noname-a con 0x7fc81c0a5a70 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:33:12.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc822e0d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc81c13e7c0 0x7fc81c135bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:12.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc822e0d640 10 monclient(hunting): picked mon.noname-b con 0x7fc81c13e7c0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:33:12.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc822e0d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:12.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc822e0d640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fc81c141fb0 con 0x7fc81c0a5a70 2024-04-26T00:33:12.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc822e0d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:12.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc822e0d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc81c141e30 con 0x7fc81c13e7c0 2024-04-26T00:33:12.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc822e0d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:12.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc822e0d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc81c142130 con 0x7fc81c0a3e70 2024-04-26T00:33:12.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc822e0d640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:12.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc822e0d640 10 monclient(hunting): authenticate will time out at 1773.518991s 2024-04-26T00:33:12.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc81bfff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc81c0a5a70 0x7fc81c13e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:12.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc820ba9640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc81c0a3e70 0x7fc81c1cf7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:12.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc81bfff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc81c0a5a70 0x7fc81c13e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:45154/0 (socket says 172.21.15.37:45154) 2024-04-26T00:33:12.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc8213aa640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc81c13e7c0 msgr2=0x7fc81c135bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:33:12.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc820ba9640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc81c0a3e70 0x7fc81c1cf7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:34000/0 (socket says 172.21.15.37:34000) 2024-04-26T00:33:12.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc81bfff640 1 -- 172.21.15.37:0/3895116827 learned_addr learned my addr 172.21.15.37:0/3895116827 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:33:12.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc81bfff640 10 monclient(hunting): get_auth_request con 0x7fc81c0a5a70 auth_method 0 2024-04-26T00:33:12.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc81bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:12.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc81bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:12.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc81bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:12.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc820ba9640 10 monclient(hunting): get_auth_request con 0x7fc81c0a3e70 auth_method 0 2024-04-26T00:33:12.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc820ba9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:12.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc820ba9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:12.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc820ba9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:12.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc8213aa640 1 --2- 172.21.15.37:0/3895116827 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc81c13e7c0 0x7fc81c135bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:33:12.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc81bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:12.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc81bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:12.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc81bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:12.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc820ba9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:12.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc820ba9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:12.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc820ba9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:12.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc81bfff640 10 monclient(hunting): handle_auth_done global_id 144267 payload 824 2024-04-26T00:33:12.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc81bfff640 1 -- 172.21.15.37:0/3895116827 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc81c0a3e70 msgr2=0x7fc81c1cf7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:12.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc81bfff640 1 --2- 172.21.15.37:0/3895116827 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc81c0a3e70 0x7fc81c1cf7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:12.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc81bfff640 1 -- 172.21.15.37:0/3895116827 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc81c13e7c0 msgr2=0x7fc81c135bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:33:12.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc81bfff640 1 --2- 172.21.15.37:0/3895116827 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc81c13e7c0 0x7fc81c135bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:12.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc81bfff640 10 monclient: _finish_hunting 0 2024-04-26T00:33:12.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc81bfff640 1 monclient: found mon.noname-a 2024-04-26T00:33:12.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc81bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:12.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc81bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:33:12.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.211+0000 7fc81bfff640 1 -- 172.21.15.37:0/3895116827 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc81c09ada0 con 0x7fc81c0a5a70 2024-04-26T00:33:12.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc81bfff640 1 -- 172.21.15.37:0/3895116827 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc81c0a5a70 msgr2=0x7fc81c13e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:12.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc81bfff640 1 --2- 172.21.15.37:0/3895116827 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc81c0a5a70 0x7fc81c13e280 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc80c00b7d0 tx=0x7fc80c00bc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:12.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc8197fa640 1 -- 172.21.15.37:0/3895116827 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc80c00cbd0 con 0x7fc81c0a5a70 2024-04-26T00:33:12.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc8197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:12.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc8197fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:33:12.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc8197fa640 10 monclient: dump: 2024-04-26T00:33:12.216 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:12.216 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:12.216 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:12.216 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:12.216 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:12.216 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:12.216 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:12.216 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:12.216 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:12.216 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:12.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc8197fa640 10 monclient: _finish_auth 0 2024-04-26T00:33:12.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc8197fa640 10 monclient: _check_auth_tickets 2024-04-26T00:33:12.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc8197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:12.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc822e0d640 5 monclient: authenticate success, global_id 144267 2024-04-26T00:33:12.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc822e0d640 10 monclient: _renew_subs 2024-04-26T00:33:12.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc8197fa640 1 -- 172.21.15.37:0/3895116827 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc80c010070 con 0x7fc81c0a5a70 2024-04-26T00:33:12.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc822e0d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:33:12.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc822e0d640 1 -- 172.21.15.37:0/3895116827 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc81c0a9050 con 0x7fc81c0a5a70 2024-04-26T00:33:12.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc8197fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:33:12.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc8197fa640 1 -- 172.21.15.37:0/3895116827 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc80c015470 con 0x7fc81c0a5a70 2024-04-26T00:33:12.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc8197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:12.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc8197fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:33:12.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc8197fa640 10 monclient: dump: 2024-04-26T00:33:12.217 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:12.217 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:12.217 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:12.217 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:12.218 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:12.218 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:12.218 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:12.218 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:12.218 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:12.218 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:12.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc81affd640 4 set_mon_vals no callback set 2024-04-26T00:33:12.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc822e0d640 10 monclient: _renew_subs 2024-04-26T00:33:12.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc822e0d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:33:12.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc822e0d640 1 -- 172.21.15.37:0/3895116827 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc81c139810 con 0x7fc81c0a5a70 2024-04-26T00:33:12.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc822e0d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:12.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc822e0d640 1 --2- 172.21.15.37:0/3895116827 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc81c09c620 0x7fc81c1360f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:12.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc822e0d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:12.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc822e0d640 1 -- 172.21.15.37:0/3895116827 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fc81c04f510 con 0x7fc81c09c620 2024-04-26T00:33:12.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc820ba9640 1 --2- 172.21.15.37:0/3895116827 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc81c09c620 0x7fc81c1360f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:12.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc8197fa640 1 -- 172.21.15.37:0/3895116827 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc80c015690 con 0x7fc81c0a5a70 2024-04-26T00:33:12.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc820ba9640 10 monclient: get_auth_request con 0x7fc81c09c620 auth_method 0 2024-04-26T00:33:12.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc820ba9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:12.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc820ba9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:12.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc820ba9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:12.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc820ba9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:12.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc820ba9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:12.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc820ba9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:12.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc820ba9640 10 monclient(hunting): handle_auth_done global_id 144270 payload 824 2024-04-26T00:33:12.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc8197fa640 1 --2- 172.21.15.37:0/3895116827 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc7fc035e80 0x7fc7fc038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:12.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc820ba9640 1 -- 172.21.15.37:0/3895116827 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc81c0008e0 con 0x7fc81c09c620 2024-04-26T00:33:12.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc8197fa640 1 -- 172.21.15.37:0/3895116827 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(506..506 src has 370..506) ==== 3959+0+0 (secure 0 0 0) 0x7fc80c048ae0 con 0x7fc81c0a5a70 2024-04-26T00:33:12.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc8213aa640 1 --2- 172.21.15.37:0/3895116827 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc7fc035e80 0x7fc7fc038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:12.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.215+0000 7fc8213aa640 10 monclient: get_auth_request con 0x7fc7fc035e80 auth_method 0 2024-04-26T00:33:12.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.219+0000 7fc820ba9640 1 -- 172.21.15.37:0/3895116827 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc81c09c620 msgr2=0x7fc81c1360f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:12.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.219+0000 7fc820ba9640 1 --2- 172.21.15.37:0/3895116827 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc81c09c620 0x7fc81c1360f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc804003fd0 tx=0x7fc804009a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:12.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.219+0000 7fc8197fa640 1 -- 172.21.15.37:0/3895116827 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc804016070 con 0x7fc81c09c620 2024-04-26T00:33:12.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.219+0000 7fc8197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:12.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.219+0000 7fc8197fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:33:12.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.219+0000 7fc8197fa640 10 monclient: dump: 2024-04-26T00:33:12.222 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:12.222 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:12.222 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:12.223 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:12.223 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:12.223 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:12.223 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:12.223 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:12.223 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:12.223 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:12.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.219+0000 7fc8197fa640 1 -- 172.21.15.37:0/3895116827 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc80400e970 con 0x7fc81c09c620 2024-04-26T00:33:12.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.219+0000 7fc8197fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:12.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.219+0000 7fc8197fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:33:12.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.219+0000 7fc8197fa640 1 -- 172.21.15.37:0/3895116827 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc81c09c620 msgr2=0x7fc81c1360f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:12.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.219+0000 7fc8197fa640 1 --2- 172.21.15.37:0/3895116827 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc81c09c620 0x7fc81c1360f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc804003fd0 tx=0x7fc804009a40 comp rx=0 tx=0).stop 2024-04-26T00:33:12.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.219+0000 7fc8197fa640 1 -- 172.21.15.37:0/3895116827 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc81c09c620 msgr2=0x7fc81c1360f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:33:12.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.219+0000 7fc8197fa640 1 --2- 172.21.15.37:0/3895116827 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc81c09c620 0x7fc81c1360f0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:12.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.219+0000 7fc8213aa640 1 --2- 172.21.15.37:0/3895116827 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc7fc035e80 0x7fc7fc038340 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fc81c09acb0 tx=0x7fc814008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:12.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.227+0000 7fc822e0d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:33:12.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.227+0000 7fc822e0d640 1 --2- 172.21.15.37:0/3895116827 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc81c136630 0x7fc81c09d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:12.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.227+0000 7fc822e0d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:12.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.227+0000 7fc822e0d640 1 -- 172.21.15.37:0/3895116827 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fc81c136a40 con 0x7fc81c136630 2024-04-26T00:33:12.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.227+0000 7fc820ba9640 1 --2- 172.21.15.37:0/3895116827 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc81c136630 0x7fc81c09d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:12.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.227+0000 7fc820ba9640 10 monclient: get_auth_request con 0x7fc81c136630 auth_method 0 2024-04-26T00:33:12.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.227+0000 7fc820ba9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:12.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.227+0000 7fc820ba9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:12.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.227+0000 7fc820ba9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:12.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.227+0000 7fc820ba9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:12.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.227+0000 7fc820ba9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:12.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.227+0000 7fc820ba9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:12.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.227+0000 7fc820ba9640 10 monclient(hunting): handle_auth_done global_id 144273 payload 824 2024-04-26T00:33:12.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.227+0000 7fc820ba9640 1 -- 172.21.15.37:0/3895116827 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc81c13ebc0 con 0x7fc81c136630 2024-04-26T00:33:12.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.227+0000 7fc820ba9640 1 -- 172.21.15.37:0/3895116827 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc81c136630 msgr2=0x7fc81c09d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:12.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.227+0000 7fc820ba9640 1 --2- 172.21.15.37:0/3895116827 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc81c136630 0x7fc81c09d100 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc804018000 tx=0x7fc8040059e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:12.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.227+0000 7fc8197fa640 1 -- 172.21.15.37:0/3895116827 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc804016070 con 0x7fc81c136630 2024-04-26T00:33:12.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.227+0000 7fc8197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:12.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.227+0000 7fc8197fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:33:12.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.227+0000 7fc8197fa640 10 monclient: dump: 2024-04-26T00:33:12.225 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:12.225 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:12.225 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:12.225 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:12.225 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:12.225 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:12.225 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:12.225 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:12.225 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:12.226 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:12.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.227+0000 7fc8197fa640 1 -- 172.21.15.37:0/3895116827 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4878 (secure 0 0 0) 0x7fc8040067b0 con 0x7fc81c136630 2024-04-26T00:33:12.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.227+0000 7fc8197fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:33:12.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.227+0000 7fc8197fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:33:12.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.227+0000 7fc8197fa640 1 -- 172.21.15.37:0/3895116827 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc81c136630 msgr2=0x7fc81c09d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:12.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.227+0000 7fc8197fa640 1 --2- 172.21.15.37:0/3895116827 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc81c136630 0x7fc81c09d100 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc804018000 tx=0x7fc8040059e0 comp rx=0 tx=0).stop 2024-04-26T00:33:12.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.227+0000 7fc8197fa640 1 -- 172.21.15.37:0/3895116827 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc81c136630 msgr2=0x7fc81c09d100 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:33:12.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.227+0000 7fc8197fa640 1 --2- 172.21.15.37:0/3895116827 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc81c136630 0x7fc81c09d100 secure :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc804018000 tx=0x7fc8040059e0 comp rx=0 tx=0).stop 2024-04-26T00:33:12.226 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:33:12.226 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:33:12.227 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:33:12.227 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:33:12.227 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 100, 2024-04-26T00:33:12.227 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 78771, 2024-04-26T00:33:12.227 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:33:12.227 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:33:12.227 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:33:12.227 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:12.227 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 2, 2024-04-26T00:33:12.227 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:33:12.227 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:33:12.227 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:33:12.227 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:33:12.227 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:33:12.228 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:33:12.228 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:33:12.228 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:33:12.228 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:33:12.228 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:33:12.228 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:33:12.228 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:33:12.228 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:33:12.228 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:33:12.228 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:12.228 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:33:12.228 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:33:12.228 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:33:12.228 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:33:12.229 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:33:12.229 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:33:12.229 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:33:12.229 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:33:12.229 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:33:12.229 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:33:12.229 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:33:12.229 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:33:12.229 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:33:12.229 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:12.229 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:12.229 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:33:12.229 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:33:12.229 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:33:12.230 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:33:12.230 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:33:12.230 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:33:12.230 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:33:12.230 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:33:12.230 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:33:12.230 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:33:12.230 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:33:12.230 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:33:12.230 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:33:12.230 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:33:12.230 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:33:12.230 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:33:12.230 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:33:12.230 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:33:12.231 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:33:12.231 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:33:12.231 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:33:12.231 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:33:12.231 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:33:12.231 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:33:12.231 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:33:12.231 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:33:12.231 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:33:12.231 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:33:12.231 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:12.231 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:33:12.231 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:12.231 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:33:12.232 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:12.232 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:33:12.232 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:33:12.232 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:33:12.232 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:33:12.232 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:12.232 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:33:12.232 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:33:12.232 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:12.232 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:12.232 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:12.232 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:33:12.232 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:33:12.232 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:12.233 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:12.233 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:12.233 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:12.233 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:33:12.233 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:33:12.233 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:33:12.233 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:33:12.233 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:33:12.233 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:12.233 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:12.233 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:33:12.233 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:33:12.233 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:33:12.233 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:33:12.234 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:12.234 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:33:12.234 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:33:12.234 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:12.234 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:12.234 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:12.234 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:33:12.234 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:33:12.234 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:12.234 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:12.234 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:12.234 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:12.234 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:33:12.234 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:33:12.235 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:33:12.235 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:33:12.235 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:33:12.235 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:12.235 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:12.235 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:33:12.235 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:33:12.235 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:33:12.235 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:33:12.235 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:12.235 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:33:12.235 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:33:12.235 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:12.235 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:12.236 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:12.236 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:33:12.236 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:33:12.236 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:12.236 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:12.236 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:12.236 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:12.236 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:33:12.236 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:33:12.236 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:33:12.236 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:33:12.236 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:33:12.236 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:12.236 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:12.237 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:12.237 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:33:12.237 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:33:12.237 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:12.237 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:33:12.237 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:33:12.237 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:33:12.237 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:12.237 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:12.237 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:33:12.237 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:12.237 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:33:12.237 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:33:12.237 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:33:12.237 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:12.238 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:12.238 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:33:12.238 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:12.238 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:33:12.238 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:33:12.238 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:33:12.238 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:12.238 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:12.238 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:33:12.238 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:12.238 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:33:12.238 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:33:12.238 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:33:12.238 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:12.239 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:12.239 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:12.239 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:33:12.239 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:33:12.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.231+0000 7fc822e0d640 1 -- 172.21.15.37:0/3895116827 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc7fc035e80 msgr2=0x7fc7fc038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:12.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.231+0000 7fc822e0d640 1 --2- 172.21.15.37:0/3895116827 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc7fc035e80 0x7fc7fc038340 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fc81c09acb0 tx=0x7fc814008040 comp rx=0 tx=0).stop 2024-04-26T00:33:12.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.231+0000 7fc822e0d640 10 monclient: shutdown 2024-04-26T00:33:12.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.231+0000 7fc822e0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:33:12.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.231+0000 7fc822e0d640 1 -- 172.21.15.37:0/3895116827 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc81c0a5a70 msgr2=0x7fc81c13e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:12.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.231+0000 7fc822e0d640 1 --2- 172.21.15.37:0/3895116827 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc81c0a5a70 0x7fc81c13e280 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc80c00b7d0 tx=0x7fc80c00bc90 comp rx=0 tx=0).stop 2024-04-26T00:33:12.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.231+0000 7fc8213aa640 1 -- 172.21.15.37:0/3895116827 reap_dead start 2024-04-26T00:33:12.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.231+0000 7fc8213aa640 1 -- 172.21.15.37:0/3895116827 reap_dead start 2024-04-26T00:33:12.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.231+0000 7fc822e0d640 1 -- 172.21.15.37:0/3895116827 shutdown_connections 2024-04-26T00:33:12.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.231+0000 7fc822e0d640 1 -- 172.21.15.37:0/3895116827 >> 172.21.15.37:0/3895116827 conn(0x7fc81c0a6140 msgr2=0x7fc81c132df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:33:12.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.231+0000 7fc822e0d640 1 -- 172.21.15.37:0/3895116827 shutdown_connections 2024-04-26T00:33:12.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.231+0000 7fc822e0d640 1 -- 172.21.15.37:0/3895116827 wait complete. 2024-04-26T00:33:12.241 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:33:12.242 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-04-26T00:33:12.242 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:33:12.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.363+0000 7f07a093a640 10 monclient: get_monmap_and_config 2024-04-26T00:33:12.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.363+0000 7f07a093a640 10 monclient: build_initial_monmap 2024-04-26T00:33:12.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.363+0000 7f07a093a640 10 monclient: monmap: 2024-04-26T00:33:12.358 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:33:12.358 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:12.358 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:33:12.366869+0000 2024-04-26T00:33:12.358 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:33:12.366869+0000 2024-04-26T00:33:12.358 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:33:12.358 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:12.358 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:33:12.358 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:33:12.358 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:33:12.358 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:33:12.358 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:33:12.359 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:12.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.367+0000 7f07a093a640 1 Processor -- start 2024-04-26T00:33:12.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.367+0000 7f07a093a640 1 -- start start 2024-04-26T00:33:12.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.367+0000 7f07a093a640 10 monclient: init 2024-04-26T00:33:12.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.367+0000 7f07a093a640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:12.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.367+0000 7f07a093a640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-04-26T00:33:12.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.367+0000 7f07a093a640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f079c09c780 0x7f079c1350b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:12.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.367+0000 7f07a093a640 10 monclient(hunting): picked mon.noname-a con 0x7f079c09c780 addr v2:172.21.15.37:3300/0 2024-04-26T00:33:12.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.367+0000 7f07a093a640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f079c09cb60 0x7f079c0a1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:12.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.367+0000 7f07a093a640 10 monclient(hunting): picked mon.noname-c con 0x7f079c09cb60 addr v2:172.21.15.45:3300/0 2024-04-26T00:33:12.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.367+0000 7f07a093a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f079c0a20a0 addr v1:172.21.15.45:6789/0 2024-04-26T00:33:12.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.367+0000 7f07a093a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f079c06b1d0 con 0x7f079c0a20a0 2024-04-26T00:33:12.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.367+0000 7f07a093a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:12.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.367+0000 7f07a093a640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f079c1355f0 con 0x7f079c09c780 2024-04-26T00:33:12.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.367+0000 7f07a093a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:12.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.367+0000 7f07a093a640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f079c06b4f0 con 0x7f079c09cb60 2024-04-26T00:33:12.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.367+0000 7f07a093a640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:12.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.367+0000 7f07a093a640 10 monclient(hunting): authenticate will time out at 1773.673063s 2024-04-26T00:33:12.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.367+0000 7f079a59c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f079c09c780 0x7f079c1350b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:12.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.367+0000 7f0793fff640 1 -- >> v2:172.21.15.45:3300/0 conn(0x7f079c09cb60 msgr2=0x7f079c0a1b60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:33:12.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.367+0000 7f079ad9d640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f079c0a20a0 legacy=0x7f079c13bf00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:33:12.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.367+0000 7f079a59c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f079c09c780 0x7f079c1350b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:45168/0 (socket says 172.21.15.37:45168) 2024-04-26T00:33:12.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.367+0000 7f0793fff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f079c09cb60 0x7f079c0a1b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:33:12.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.367+0000 7f079a59c640 1 -- 172.21.15.37:0/1595426612 learned_addr learned my addr 172.21.15.37:0/1595426612 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:33:12.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.367+0000 7f079a59c640 10 monclient(hunting): get_auth_request con 0x7f079c09c780 auth_method 0 2024-04-26T00:33:12.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.367+0000 7f079a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:12.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.367+0000 7f079a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:12.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.367+0000 7f079a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:12.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.367+0000 7f079a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:12.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.367+0000 7f079a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:12.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.367+0000 7f079a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:12.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.367+0000 7f079a59c640 10 monclient(hunting): handle_auth_done global_id 144276 payload 274 2024-04-26T00:33:12.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.367+0000 7f079a59c640 1 -- 172.21.15.37:0/1595426612 >> v2:172.21.15.45:3300/0 conn(0x7f079c09cb60 msgr2=0x7f079c0a1b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:33:12.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.367+0000 7f079a59c640 1 --2- 172.21.15.37:0/1595426612 >> v2:172.21.15.45:3300/0 conn(0x7f079c09cb60 0x7f079c0a1b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:12.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.367+0000 7f079a59c640 1 -- 172.21.15.37:0/1595426612 >> v1:172.21.15.45:6789/0 conn(0x7f079c0a20a0 legacy=0x7f079c13bf00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:33:12.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.367+0000 7f079a59c640 10 monclient: _finish_hunting 0 2024-04-26T00:33:12.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.367+0000 7f079a59c640 1 monclient: found mon.noname-a 2024-04-26T00:33:12.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.367+0000 7f079a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:12.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.367+0000 7f079a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:33:12.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.367+0000 7f079a59c640 1 -- 172.21.15.37:0/1595426612 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f079c06b9c0 con 0x7f079c09c780 2024-04-26T00:33:12.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.371+0000 7f079a59c640 1 -- 172.21.15.37:0/1595426612 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f079c09c780 msgr2=0x7f079c1350b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:12.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.371+0000 7f079a59c640 1 --2- 172.21.15.37:0/1595426612 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f079c09c780 0x7f079c1350b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f079c0a4000 tx=0x7f078c01c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57fc51828ca6b812 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:12.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.371+0000 7f0799d9b640 1 -- 172.21.15.37:0/1595426612 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078c0b0070 con 0x7f079c09c780 2024-04-26T00:33:12.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.371+0000 7f0799d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:12.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.371+0000 7f0799d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:33:12.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.371+0000 7f0799d9b640 10 monclient: dump: 2024-04-26T00:33:12.372 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:12.372 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:12.372 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:12.372 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:12.373 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:12.373 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:12.373 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:12.373 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:12.373 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:12.373 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:12.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.371+0000 7f0799d9b640 10 monclient: _finish_auth 0 2024-04-26T00:33:12.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.371+0000 7f0799d9b640 10 monclient: _check_auth_tickets 2024-04-26T00:33:12.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.371+0000 7f0799d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:12.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.371+0000 7f0799d9b640 1 -- 172.21.15.37:0/1595426612 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f078c0a4e50 con 0x7f079c09c780 2024-04-26T00:33:12.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.371+0000 7f0799d9b640 10 monclient: handle_config config(0 keys) 2024-04-26T00:33:12.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.371+0000 7f0799d9b640 1 -- 172.21.15.37:0/1595426612 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078c0ab820 con 0x7f079c09c780 2024-04-26T00:33:12.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.371+0000 7f0799d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:12.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.371+0000 7f0799d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:33:12.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.371+0000 7f0799d9b640 10 monclient: dump: 2024-04-26T00:33:12.374 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:12.374 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:12.374 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:12.374 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:12.374 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:12.374 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:12.374 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:12.374 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:12.374 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:12.374 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:12.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.371+0000 7f07a093a640 5 monclient: authenticate success, global_id 144276 2024-04-26T00:33:12.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.371+0000 7f07a093a640 10 monclient: get_monmap_and_config success 2024-04-26T00:33:12.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.371+0000 7f07a093a640 4 set_mon_vals no callback set 2024-04-26T00:33:12.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.371+0000 7f07a093a640 10 monclient: shutdown 2024-04-26T00:33:12.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.371+0000 7f07a093a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:33:12.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.371+0000 7f07a093a640 1 -- 172.21.15.37:0/1595426612 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f079c09c780 msgr2=0x7f079c1350b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:12.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.371+0000 7f07a093a640 1 --2- 172.21.15.37:0/1595426612 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f079c09c780 0x7f079c1350b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f079c0a4000 tx=0x7f078c01c3c0 comp rx=0 tx=0).stop 2024-04-26T00:33:12.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.371+0000 7f07a093a640 1 -- 172.21.15.37:0/1595426612 shutdown_connections 2024-04-26T00:33:12.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.371+0000 7f07a093a640 1 --2- 172.21.15.37:0/1595426612 >> v2:172.21.15.45:3300/0 conn(0x7f079c09cb60 0x7f079c0a1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:12.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.371+0000 7f07a093a640 1 --2- 172.21.15.37:0/1595426612 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f079c09c780 0x7f079c1350b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:12.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.371+0000 7f07a093a640 1 -- 172.21.15.37:0/1595426612 >> 172.21.15.37:0/1595426612 conn(0x7f079c0a2e20 msgr2=0x7f079c0a3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:33:12.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.371+0000 7f07a093a640 1 -- 172.21.15.37:0/1595426612 shutdown_connections 2024-04-26T00:33:12.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.371+0000 7f07a093a640 1 -- 172.21.15.37:0/1595426612 wait complete. 2024-04-26T00:33:12.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.371+0000 7f07a093a640 10 monclient: build_initial_monmap 2024-04-26T00:33:12.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.371+0000 7f07a093a640 10 monclient: monmap: 2024-04-26T00:33:12.375 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:33:12.375 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:33:12.375 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:33:12.376 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:33:12.376 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:33:12.376 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:33:12.376 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:33:12.376 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:33:12.376 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:33:12.376 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:12.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f07a093a640 1 Processor -- start 2024-04-26T00:33:12.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f07a093a640 1 -- start start 2024-04-26T00:33:12.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f07a093a640 10 monclient: init 2024-04-26T00:33:12.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f07a093a640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:12.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f07a093a640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:33:12.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f07a093a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f079c09c780 0x7f079c12b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:12.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f07a093a640 10 monclient(hunting): picked mon.noname-b con 0x7f079c09c780 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:33:12.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f07a093a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f079c09cb60 0x7f079c12b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:12.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f07a093a640 10 monclient(hunting): picked mon.noname-a con 0x7f079c09cb60 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:33:12.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f07a093a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f079c0a20a0 0x7f079c12bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:12.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f07a093a640 10 monclient(hunting): picked mon.noname-c con 0x7f079c0a20a0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:33:12.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f07a093a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:12.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f07a093a640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f079c09d150 con 0x7f079c09cb60 2024-04-26T00:33:12.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f07a093a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:12.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f07a093a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f079c09cfd0 con 0x7f079c09c780 2024-04-26T00:33:12.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f07a093a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:12.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f07a093a640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f079c13fe40 con 0x7f079c0a20a0 2024-04-26T00:33:12.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f07a093a640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:12.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f07a093a640 10 monclient(hunting): authenticate will time out at 1773.680569s 2024-04-26T00:33:12.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f079ad9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f079c0a20a0 0x7f079c12bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:12.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f079a59c640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f079c09c780 msgr2=0x7f079c12b210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:33:12.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f079ad9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f079c0a20a0 0x7f079c12bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:34008/0 (socket says 172.21.15.37:34008) 2024-04-26T00:33:12.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f079ad9d640 1 -- 172.21.15.37:0/3810047564 learned_addr learned my addr 172.21.15.37:0/3810047564 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:33:12.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f0793fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f079c09cb60 0x7f079c12b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:12.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f079ad9d640 10 monclient(hunting): get_auth_request con 0x7f079c0a20a0 auth_method 0 2024-04-26T00:33:12.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f079ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:12.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f079ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:12.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f079ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:12.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f079a59c640 1 --2- 172.21.15.37:0/3810047564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f079c09c780 0x7f079c12b210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:33:12.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f0793fff640 10 monclient(hunting): get_auth_request con 0x7f079c09cb60 auth_method 0 2024-04-26T00:33:12.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f0793fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:12.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f0793fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:12.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f0793fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:12.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f079ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:12.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f079ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:12.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f079ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:12.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f0793fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:12.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f0793fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:12.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f0793fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:12.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f079ad9d640 10 monclient(hunting): handle_auth_done global_id 154154 payload 824 2024-04-26T00:33:12.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f079ad9d640 1 -- 172.21.15.37:0/3810047564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f079c09c780 msgr2=0x7f079c12b210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:33:12.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f079ad9d640 1 --2- 172.21.15.37:0/3810047564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f079c09c780 0x7f079c12b210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:12.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f079ad9d640 1 -- 172.21.15.37:0/3810047564 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f079c09cb60 msgr2=0x7f079c12b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:12.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f079ad9d640 1 --2- 172.21.15.37:0/3810047564 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f079c09cb60 0x7f079c12b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:12.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f079ad9d640 10 monclient: _finish_hunting 0 2024-04-26T00:33:12.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f079ad9d640 1 monclient: found mon.noname-c 2024-04-26T00:33:12.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f079ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:12.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f079ad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:33:12.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f079ad9d640 1 -- 172.21.15.37:0/3810047564 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f079c12e200 con 0x7f079c0a20a0 2024-04-26T00:33:12.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f0793fff640 1 --2- 172.21.15.37:0/3810047564 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f079c09cb60 0x7f079c12b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:33:12.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f079ad9d640 1 -- 172.21.15.37:0/3810047564 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f079c0a20a0 msgr2=0x7f079c12bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:12.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f079ad9d640 1 --2- 172.21.15.37:0/3810047564 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f079c0a20a0 0x7f079c12bc90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f079400be90 tx=0x7f079400bf80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:12.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f0792ffd640 1 -- 172.21.15.37:0/3810047564 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0794015670 con 0x7f079c0a20a0 2024-04-26T00:33:12.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f0792ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:12.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f0792ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:33:12.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f0792ffd640 10 monclient: dump: 2024-04-26T00:33:12.380 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:12.381 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:12.381 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:12.381 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:12.381 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:12.381 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:12.381 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:12.381 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:12.381 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:12.381 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:12.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f0792ffd640 10 monclient: _finish_auth 0 2024-04-26T00:33:12.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f0792ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:33:12.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f0792ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:12.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f0792ffd640 1 -- 172.21.15.37:0/3810047564 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0794010070 con 0x7f079c0a20a0 2024-04-26T00:33:12.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f0792ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:33:12.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f0792ffd640 1 -- 172.21.15.37:0/3810047564 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0794015a50 con 0x7f079c0a20a0 2024-04-26T00:33:12.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f0792ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:12.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f0792ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:33:12.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f0792ffd640 10 monclient: dump: 2024-04-26T00:33:12.382 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:12.382 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:12.382 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:12.382 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:12.382 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:12.382 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:12.382 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:12.382 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:12.382 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:12.382 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:12.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f07a093a640 5 monclient: authenticate success, global_id 154154 2024-04-26T00:33:12.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f07a093a640 10 monclient: _renew_subs 2024-04-26T00:33:12.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f079959a640 4 set_mon_vals no callback set 2024-04-26T00:33:12.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f07a093a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:33:12.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f07a093a640 1 -- 172.21.15.37:0/3810047564 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f079c12ca00 con 0x7f079c0a20a0 2024-04-26T00:33:12.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f07a093a640 10 monclient: _renew_subs 2024-04-26T00:33:12.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f07a093a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:33:12.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.375+0000 7f07a093a640 1 -- 172.21.15.37:0/3810047564 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f079c0728a0 con 0x7f079c0a20a0 2024-04-26T00:33:12.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.379+0000 7f07a093a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:12.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.379+0000 7f07a093a640 1 --2- 172.21.15.37:0/3810047564 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f079c12ebe0 0x7f079c099040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:12.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.379+0000 7f07a093a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:12.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.379+0000 7f07a093a640 1 -- 172.21.15.37:0/3810047564 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f079c04f510 con 0x7f079c12ebe0 2024-04-26T00:33:12.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.379+0000 7f079a59c640 1 --2- 172.21.15.37:0/3810047564 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f079c12ebe0 0x7f079c099040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:12.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.379+0000 7f079a59c640 10 monclient: get_auth_request con 0x7f079c12ebe0 auth_method 0 2024-04-26T00:33:12.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.379+0000 7f079a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:12.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.379+0000 7f079a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:12.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.379+0000 7f079a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:12.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.379+0000 7f079a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:12.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.379+0000 7f079a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:12.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.379+0000 7f079a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:12.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.379+0000 7f0792ffd640 1 -- 172.21.15.37:0/3810047564 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f07940050f0 con 0x7f079c0a20a0 2024-04-26T00:33:12.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.379+0000 7f079a59c640 10 monclient(hunting): handle_auth_done global_id 154157 payload 824 2024-04-26T00:33:12.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.379+0000 7f079a59c640 1 -- 172.21.15.37:0/3810047564 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f079c0730e0 con 0x7f079c12ebe0 2024-04-26T00:33:12.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.379+0000 7f0792ffd640 1 --2- 172.21.15.37:0/3810047564 >> v2:172.21.15.45:6812/2379918623 conn(0x7f077c035e80 0x7f077c038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:12.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.379+0000 7f0792ffd640 1 -- 172.21.15.37:0/3810047564 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(506..506 src has 370..506) ==== 3959+0+0 (secure 0 0 0) 0x7f0794049640 con 0x7f079c0a20a0 2024-04-26T00:33:12.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.379+0000 7f0793fff640 1 --2- 172.21.15.37:0/3810047564 >> v2:172.21.15.45:6812/2379918623 conn(0x7f077c035e80 0x7f077c038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:12.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.379+0000 7f0793fff640 10 monclient: get_auth_request con 0x7f077c035e80 auth_method 0 2024-04-26T00:33:12.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.379+0000 7f079a59c640 1 -- 172.21.15.37:0/3810047564 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f079c12ebe0 msgr2=0x7f079c099040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:12.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.379+0000 7f079a59c640 1 --2- 172.21.15.37:0/3810047564 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f079c12ebe0 0x7f079c099040 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f079c068000 tx=0x7f078c01ac00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:12.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.379+0000 7f0792ffd640 1 -- 172.21.15.37:0/3810047564 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078c0bb020 con 0x7f079c12ebe0 2024-04-26T00:33:12.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.379+0000 7f0792ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:12.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.379+0000 7f0792ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:33:12.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.379+0000 7f0792ffd640 10 monclient: dump: 2024-04-26T00:33:12.387 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:12.387 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:12.387 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:12.387 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:12.387 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:12.387 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:12.387 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:12.388 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:12.388 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:12.388 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:12.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.379+0000 7f0792ffd640 1 -- 172.21.15.37:0/3810047564 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f078c0ab810 con 0x7f079c12ebe0 2024-04-26T00:33:12.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.379+0000 7f0792ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:12.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.379+0000 7f0792ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:33:12.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.379+0000 7f0792ffd640 1 -- 172.21.15.37:0/3810047564 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f079c12ebe0 msgr2=0x7f079c099040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:12.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.379+0000 7f0792ffd640 1 --2- 172.21.15.37:0/3810047564 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f079c12ebe0 0x7f079c099040 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f079c068000 tx=0x7f078c01ac00 comp rx=0 tx=0).stop 2024-04-26T00:33:12.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.379+0000 7f0792ffd640 1 -- 172.21.15.37:0/3810047564 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f079c12ebe0 msgr2=0x7f079c099040 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:33:12.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.379+0000 7f0792ffd640 1 --2- 172.21.15.37:0/3810047564 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f079c12ebe0 0x7f079c099040 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f079c068000 tx=0x7f078c01ac00 comp rx=0 tx=0).stop 2024-04-26T00:33:12.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.383+0000 7f0793fff640 1 --2- 172.21.15.37:0/3810047564 >> v2:172.21.15.45:6812/2379918623 conn(0x7f077c035e80 0x7f077c038340 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0784006e80 tx=0x7f0784006dd0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:12.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.387+0000 7f07a093a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:33:12.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.387+0000 7f07a093a640 1 --2- 172.21.15.37:0/3810047564 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f079c099580 0x7f079c099960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:12.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.387+0000 7f07a093a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:12.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.387+0000 7f07a093a640 1 -- 172.21.15.37:0/3810047564 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f079c099ea0 con 0x7f079c099580 2024-04-26T00:33:12.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.387+0000 7f079a59c640 1 --2- 172.21.15.37:0/3810047564 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f079c099580 0x7f079c099960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:12.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.387+0000 7f079a59c640 10 monclient: get_auth_request con 0x7f079c099580 auth_method 0 2024-04-26T00:33:12.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.387+0000 7f079a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:12.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.387+0000 7f079a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:12.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.387+0000 7f079a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:12.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.387+0000 7f079a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:12.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.387+0000 7f079a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:12.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.387+0000 7f079a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:12.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.387+0000 7f079a59c640 10 monclient(hunting): handle_auth_done global_id 154160 payload 824 2024-04-26T00:33:12.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.387+0000 7f079a59c640 1 -- 172.21.15.37:0/3810047564 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f079c06ca10 con 0x7f079c099580 2024-04-26T00:33:12.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.391+0000 7f079a59c640 1 -- 172.21.15.37:0/3810047564 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f079c099580 msgr2=0x7f079c099960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:12.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.391+0000 7f079a59c640 1 --2- 172.21.15.37:0/3810047564 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f079c099580 0x7f079c099960 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f078c01aa70 tx=0x7f078c019160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:12.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.391+0000 7f0792ffd640 1 -- 172.21.15.37:0/3810047564 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078c0bb020 con 0x7f079c099580 2024-04-26T00:33:12.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.391+0000 7f0792ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:12.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.391+0000 7f0792ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:33:12.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.391+0000 7f0792ffd640 10 monclient: dump: 2024-04-26T00:33:12.390 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:12.390 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:12.390 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:12.390 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:12.390 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:12.390 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:12.390 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:12.390 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:12.390 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:12.390 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:12.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.391+0000 7f0792ffd640 1 -- 172.21.15.37:0/3810047564 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4706 (secure 0 0 0) 0x7f078c000e50 con 0x7f079c099580 2024-04-26T00:33:12.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.391+0000 7f0792ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:33:12.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.391+0000 7f0792ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:33:12.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.391+0000 7f0792ffd640 1 -- 172.21.15.37:0/3810047564 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f079c099580 msgr2=0x7f079c099960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:12.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.391+0000 7f0792ffd640 1 --2- 172.21.15.37:0/3810047564 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f079c099580 0x7f079c099960 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f078c01aa70 tx=0x7f078c019160 comp rx=0 tx=0).stop 2024-04-26T00:33:12.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.391+0000 7f0792ffd640 1 -- 172.21.15.37:0/3810047564 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f079c099580 msgr2=0x7f079c099960 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:33:12.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.391+0000 7f0792ffd640 1 --2- 172.21.15.37:0/3810047564 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f079c099580 0x7f079c099960 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f078c01aa70 tx=0x7f078c019160 comp rx=0 tx=0).stop 2024-04-26T00:33:12.391 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:33:12.391 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:33:12.391 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:33:12.391 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:33:12.391 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 100, 2024-04-26T00:33:12.391 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 29911, 2024-04-26T00:33:12.391 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:33:12.392 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:33:12.392 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:33:12.392 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:12.392 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 2, 2024-04-26T00:33:12.392 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:33:12.392 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:33:12.392 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:33:12.392 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:33:12.392 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:33:12.392 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:33:12.392 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:33:12.392 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:33:12.392 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:33:12.392 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:33:12.392 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:33:12.393 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:33:12.393 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:33:12.393 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:33:12.393 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:12.393 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:33:12.393 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:33:12.393 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:33:12.393 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:33:12.393 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:33:12.393 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:33:12.393 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:33:12.393 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:33:12.393 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:33:12.393 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:33:12.393 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:33:12.394 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:33:12.394 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:33:12.394 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:12.394 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:12.394 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:33:12.394 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:33:12.394 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:33:12.394 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:33:12.394 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:33:12.394 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:33:12.394 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:33:12.394 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:33:12.394 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:33:12.394 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:33:12.395 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:33:12.395 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:33:12.395 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:33:12.395 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:33:12.395 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:33:12.395 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:33:12.395 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:33:12.395 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:33:12.395 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:33:12.395 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:33:12.395 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:33:12.395 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:33:12.395 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:33:12.395 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:33:12.396 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:33:12.396 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:33:12.396 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:33:12.396 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:33:12.396 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:12.396 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:33:12.396 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:12.396 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:33:12.396 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:12.396 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:33:12.396 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:33:12.396 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:33:12.396 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:33:12.396 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:12.397 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:33:12.397 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:33:12.397 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:12.397 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:12.397 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:12.397 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:33:12.397 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:33:12.397 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:12.397 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:12.397 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:12.397 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:12.397 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:33:12.397 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:33:12.397 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:33:12.397 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:33:12.398 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:33:12.398 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:12.398 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:12.398 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:33:12.398 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:33:12.398 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:33:12.398 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:33:12.398 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:12.398 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:33:12.398 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:33:12.398 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:12.398 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:12.398 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:12.398 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:33:12.399 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:33:12.399 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:12.399 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:12.399 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:12.399 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:12.399 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:33:12.399 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:33:12.399 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:33:12.399 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:33:12.399 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:33:12.399 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:12.399 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:12.399 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:33:12.399 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:33:12.400 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:33:12.400 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:33:12.400 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:12.400 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:33:12.400 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:33:12.400 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:12.400 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:12.400 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:12.400 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:33:12.400 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:33:12.400 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:12.400 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:12.400 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:12.400 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:12.400 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:33:12.401 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:33:12.401 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:33:12.401 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:33:12.401 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:33:12.401 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:12.401 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:12.401 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:12.401 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:33:12.401 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:33:12.401 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:12.401 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:33:12.401 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:33:12.401 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:33:12.401 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:12.401 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:12.402 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:33:12.402 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:12.402 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:33:12.402 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:33:12.402 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:33:12.402 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:12.402 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:12.402 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:33:12.402 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:12.402 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:33:12.402 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:33:12.402 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:33:12.402 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:12.402 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:12.402 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:12.403 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:33:12.403 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:33:12.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.391+0000 7f07a093a640 1 -- 172.21.15.37:0/3810047564 >> v2:172.21.15.45:6812/2379918623 conn(0x7f077c035e80 msgr2=0x7f077c038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:12.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.391+0000 7f07a093a640 1 --2- 172.21.15.37:0/3810047564 >> v2:172.21.15.45:6812/2379918623 conn(0x7f077c035e80 0x7f077c038340 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0784006e80 tx=0x7f0784006dd0 comp rx=0 tx=0).stop 2024-04-26T00:33:12.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.391+0000 7f07a093a640 10 monclient: shutdown 2024-04-26T00:33:12.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.391+0000 7f07a093a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:33:12.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.391+0000 7f07a093a640 1 -- 172.21.15.37:0/3810047564 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f079c0a20a0 msgr2=0x7f079c12bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:12.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.391+0000 7f07a093a640 1 --2- 172.21.15.37:0/3810047564 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f079c0a20a0 0x7f079c12bc90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f079400be90 tx=0x7f079400bf80 comp rx=0 tx=0).stop 2024-04-26T00:33:12.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.391+0000 7f079ad9d640 1 -- 172.21.15.37:0/3810047564 reap_dead start 2024-04-26T00:33:12.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.391+0000 7f07a093a640 1 -- 172.21.15.37:0/3810047564 shutdown_connections 2024-04-26T00:33:12.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.391+0000 7f07a093a640 1 --2- 172.21.15.37:0/3810047564 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f079c0a20a0 0x7f079c12bc90 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:12.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.391+0000 7f07a093a640 1 -- 172.21.15.37:0/3810047564 >> 172.21.15.37:0/3810047564 conn(0x7f079c0a2e20 msgr2=0x7f079c0a4ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:33:12.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.391+0000 7f07a093a640 1 -- 172.21.15.37:0/3810047564 shutdown_connections 2024-04-26T00:33:12.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:12.391+0000 7f07a093a640 1 -- 172.21.15.37:0/3810047564 wait complete. 2024-04-26T00:33:12.406 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:33:12.406 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-04-26T00:33:12.654 INFO:tasks.rados.rados.0.smithi045.stdout:2234: done (2 left) 2024-04-26T00:33:12.654 INFO:tasks.rados.rados.0.smithi045.stdout:2235: done (1 left) 2024-04-26T00:33:12.654 INFO:tasks.rados.rados.0.smithi045.stdout:2236: done (0 left) 2024-04-26T00:33:12.655 INFO:tasks.rados.rados.0.smithi045.stdout:2237: read oid 21 snap 246 2024-04-26T00:33:12.655 INFO:tasks.rados.rados.0.smithi045.stdout:2237: expect (ObjNum 756 snap 232 seq_num 756) 2024-04-26T00:33:12.655 INFO:tasks.rados.rados.0.smithi045.stdout:2238: copy_from oid 14 from oid 30 current snap is 250 2024-04-26T00:33:12.655 INFO:tasks.rados.rados.0.smithi045.stdout:2239: read oid 35 snap -1 2024-04-26T00:33:12.655 INFO:tasks.rados.rados.0.smithi045.stdout:2239: expect (ObjNum 542 snap 167 seq_num 542) 2024-04-26T00:33:12.655 INFO:tasks.rados.rados.0.smithi045.stdout:2240: rollback oid 27 current snap is 250 2024-04-26T00:33:12.655 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 27 to 243 2024-04-26T00:33:12.655 INFO:tasks.rados.rados.0.smithi045.stdout:2241: snap_create 2024-04-26T00:33:12.661 INFO:tasks.rados.rados.0.smithi045.stdout:2238: finishing copy_from to smithi04526579-14 2024-04-26T00:33:12.662 INFO:tasks.rados.rados.0.smithi045.stdout:2238: got expected ENOENT (src dne) 2024-04-26T00:33:12.662 INFO:tasks.rados.rados.0.smithi045.stdout:2238: finishing copy_from racing read to smithi04526579-14 2024-04-26T00:33:12.675 INFO:tasks.rados.rados.0.smithi045.stdout:2237: expect (ObjNum 756 snap 232 seq_num 756) 2024-04-26T00:33:12.769 INFO:tasks.rados.rados.0.smithi045.stdout:2239: expect (ObjNum 542 snap 167 seq_num 542) 2024-04-26T00:33:12.915 INFO:tasks.rados.rados.0.smithi045.stdout:2240: finishing rollback tid 0 to smithi04526579-27 2024-04-26T00:33:12.915 INFO:tasks.rados.rados.0.smithi045.stdout:2240: finishing rollback tid 1 to smithi04526579-27 2024-04-26T00:33:12.915 INFO:tasks.rados.rados.0.smithi045.stdout:2240: finishing rollback tid 2 to smithi04526579-27 2024-04-26T00:33:12.915 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 1509 (ObjNum 750 snap 232 seq_num 750) dirty exists 2024-04-26T00:33:13.487 INFO:tasks.rados.rados.0.smithi045.stdout:2237: done (4 left) 2024-04-26T00:33:13.487 INFO:tasks.rados.rados.0.smithi045.stdout:2238: done (3 left) 2024-04-26T00:33:13.487 INFO:tasks.rados.rados.0.smithi045.stdout:2239: done (2 left) 2024-04-26T00:33:13.488 INFO:tasks.rados.rados.0.smithi045.stdout:2240: done (1 left) 2024-04-26T00:33:13.488 INFO:tasks.rados.rados.0.smithi045.stdout:2241: done (0 left) 2024-04-26T00:33:13.488 INFO:tasks.rados.rados.0.smithi045.stdout:2242: read oid 12 snap -1 2024-04-26T00:33:13.488 INFO:tasks.rados.rados.0.smithi045.stdout:2242: expect (ObjNum 777 snap 240 seq_num 777) 2024-04-26T00:33:13.488 INFO:tasks.rados.rados.0.smithi045.stdout:2243: snap_create 2024-04-26T00:33:13.497 INFO:tasks.rados.rados.0.smithi045.stdout:2242: expect (ObjNum 777 snap 240 seq_num 777) 2024-04-26T00:33:14.492 INFO:tasks.rados.rados.0.smithi045.stdout:2242: done (1 left) 2024-04-26T00:33:14.492 INFO:tasks.rados.rados.0.smithi045.stdout:2243: done (0 left) 2024-04-26T00:33:14.492 INFO:tasks.rados.rados.0.smithi045.stdout:2244: copy_from oid 19 from oid 44 current snap is 252 2024-04-26T00:33:14.492 INFO:tasks.rados.rados.0.smithi045.stdout:2245: delete oid 21 current snap is 252 2024-04-26T00:33:14.502 INFO:tasks.rados.rados.0.smithi045.stdout:2245: done (1 left) 2024-04-26T00:33:14.502 INFO:tasks.rados.rados.0.smithi045.stdout:2246: snap_remove snap 234 2024-04-26T00:33:15.493 INFO:tasks.rados.rados.0.smithi045.stdout:2246: done (1 left) 2024-04-26T00:33:15.493 INFO:tasks.rados.rados.0.smithi045.stdout:2247: write (excl) oid 1 current snap is 252 2024-04-26T00:33:15.494 INFO:tasks.rados.rados.0.smithi045.stdout:2244: finishing copy_from racing read to smithi04526579-19 2024-04-26T00:33:15.494 INFO:tasks.rados.rados.0.smithi045.stdout:2244: finishing copy_from to smithi04526579-19 2024-04-26T00:33:15.494 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 1095 (ObjNum 752 snap 232 seq_num 752) dirty exists 2024-04-26T00:33:15.494 INFO:tasks.rados.rados.0.smithi045.stdout:2247: seq_num 811 ranges {430477=560357,1408073=473601,2488400=736073} 2024-04-26T00:33:15.500 INFO:tasks.rados.rados.0.smithi045.stdout:2247: writing smithi04526579-1 from 430477 to 990834 tid 1 2024-04-26T00:33:15.506 INFO:tasks.rados.rados.0.smithi045.stdout:2247: writing smithi04526579-1 from 1408073 to 1881674 tid 2 2024-04-26T00:33:15.515 INFO:tasks.rados.rados.0.smithi045.stdout:2247: writing smithi04526579-1 from 2488400 to 3224473 tid 3 2024-04-26T00:33:15.515 INFO:tasks.rados.rados.0.smithi045.stdout:2244: done (1 left) 2024-04-26T00:33:15.515 INFO:tasks.rados.rados.0.smithi045.stdout:2248: read oid 24 snap 235 2024-04-26T00:33:15.515 INFO:tasks.rados.rados.0.smithi045.stdout:2248: expect (ObjNum 735 snap 227 seq_num 735) 2024-04-26T00:33:15.516 INFO:tasks.rados.rados.0.smithi045.stdout:2249: write oid 35 current snap is 252 2024-04-26T00:33:15.516 INFO:tasks.rados.rados.0.smithi045.stdout:2249: seq_num 812 ranges {447284=635606,1866555=733612,3315823=331461} 2024-04-26T00:33:15.522 INFO:tasks.rados.rados.0.smithi045.stdout:2249: writing smithi04526579-35 from 447284 to 1082890 tid 1 2024-04-26T00:33:15.531 INFO:tasks.rados.rados.0.smithi045.stdout:2249: writing smithi04526579-35 from 1866555 to 2600167 tid 2 2024-04-26T00:33:15.539 INFO:tasks.rados.rados.0.smithi045.stdout:2249: writing smithi04526579-35 from 3315823 to 3647284 tid 3 2024-04-26T00:33:15.539 INFO:tasks.rados.rados.0.smithi045.stdout:2250: rollback oid 2 current snap is 252 2024-04-26T00:33:15.539 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 2 to 140 2024-04-26T00:33:15.539 INFO:tasks.rados.rados.0.smithi045.stdout:2247: finishing write tid 1 to smithi04526579-1 2024-04-26T00:33:15.539 INFO:tasks.rados.rados.0.smithi045.stdout:2247: finishing write tid 2 to smithi04526579-1 2024-04-26T00:33:15.539 INFO:tasks.rados.rados.0.smithi045.stdout:2247: finishing write tid 3 to smithi04526579-1 2024-04-26T00:33:15.539 INFO:tasks.rados.rados.0.smithi045.stdout:2247: finishing write tid 4 to smithi04526579-1 2024-04-26T00:33:15.539 INFO:tasks.rados.rados.0.smithi045.stdout:2247: finishing write tid 5 to smithi04526579-1 2024-04-26T00:33:15.540 INFO:tasks.rados.rados.0.smithi045.stdout:2247: oid 1 updating version 0 to 1585 2024-04-26T00:33:15.540 INFO:tasks.rados.rados.0.smithi045.stdout:2247: oid 1 updating version 1585 to 1586 2024-04-26T00:33:15.540 INFO:tasks.rados.rados.0.smithi045.stdout:2247: oid 1 updating version 1586 to 1588 2024-04-26T00:33:15.540 INFO:tasks.rados.rados.0.smithi045.stdout:2247: oid 1 version 1588 is already newer than 1587 2024-04-26T00:33:15.540 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 1588 (ObjNum 811 snap 252 seq_num 811) dirty exists 2024-04-26T00:33:15.540 INFO:tasks.rados.rados.0.smithi045.stdout:2247: left oid 1 (ObjNum 811 snap 252 seq_num 811) 2024-04-26T00:33:15.540 INFO:tasks.rados.rados.0.smithi045.stdout:2247: done (3 left) 2024-04-26T00:33:15.540 INFO:tasks.rados.rados.0.smithi045.stdout:2251: snap_create 2024-04-26T00:33:15.543 INFO:tasks.rados.rados.0.smithi045.stdout:2248: expect (ObjNum 735 snap 227 seq_num 735) 2024-04-26T00:33:15.683 INFO:tasks.rados.rados.0.smithi045.stdout:2249: finishing write tid 1 to smithi04526579-35 2024-04-26T00:33:15.684 INFO:tasks.rados.rados.0.smithi045.stdout:2250: finishing rollback tid 1 to smithi04526579-2 2024-04-26T00:33:15.684 INFO:tasks.rados.rados.0.smithi045.stdout:2249: finishing write tid 2 to smithi04526579-35 2024-04-26T00:33:15.684 INFO:tasks.rados.rados.0.smithi045.stdout:2250: finishing rollback tid 2 to smithi04526579-2 2024-04-26T00:33:15.684 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 1538 (ObjNum 356 snap 105 seq_num 356) dirty exists 2024-04-26T00:33:15.684 INFO:tasks.rados.rados.0.smithi045.stdout:2249: finishing write tid 3 to smithi04526579-35 2024-04-26T00:33:15.684 INFO:tasks.rados.rados.0.smithi045.stdout:2249: finishing write tid 4 to smithi04526579-35 2024-04-26T00:33:15.684 INFO:tasks.rados.rados.0.smithi045.stdout:2249: finishing write tid 5 to smithi04526579-35 2024-04-26T00:33:15.684 INFO:tasks.rados.rados.0.smithi045.stdout:2249: oid 35 updating version 0 to 1525 2024-04-26T00:33:15.684 INFO:tasks.rados.rados.0.smithi045.stdout:2249: oid 35 version 1525 is already newer than 1523 2024-04-26T00:33:15.684 INFO:tasks.rados.rados.0.smithi045.stdout:2249: oid 35 version 1525 is already newer than 1524 2024-04-26T00:33:15.684 INFO:tasks.rados.rados.0.smithi045.stdout:2249: oid 35 version 1525 is already newer than 1520 2024-04-26T00:33:15.684 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 1525 (ObjNum 812 snap 252 seq_num 812) dirty exists 2024-04-26T00:33:15.685 INFO:tasks.rados.rados.0.smithi045.stdout:2249: left oid 35 (ObjNum 812 snap 252 seq_num 812) 2024-04-26T00:33:16.499 INFO:tasks.rados.rados.0.smithi045.stdout:2248: done (3 left) 2024-04-26T00:33:16.500 INFO:tasks.rados.rados.0.smithi045.stdout:2249: done (2 left) 2024-04-26T00:33:16.500 INFO:tasks.rados.rados.0.smithi045.stdout:2250: done (1 left) 2024-04-26T00:33:16.500 INFO:tasks.rados.rados.0.smithi045.stdout:2251: done (0 left) 2024-04-26T00:33:16.500 INFO:tasks.rados.rados.0.smithi045.stdout:2252: read oid 11 snap 235 2024-04-26T00:33:16.500 INFO:tasks.rados.rados.0.smithi045.stdout:2252: expect (ObjNum 759 snap 232 seq_num 759) 2024-04-26T00:33:16.500 INFO:tasks.rados.rados.0.smithi045.stdout:2253: read oid 33 snap 243 2024-04-26T00:33:16.500 INFO:tasks.rados.rados.0.smithi045.stdout:2253: expect (ObjNum 793 snap 242 seq_num 793) 2024-04-26T00:33:16.500 INFO:tasks.rados.rados.0.smithi045.stdout:2254: snap_create 2024-04-26T00:33:16.510 INFO:tasks.rados.rados.0.smithi045.stdout:2252: expect (ObjNum 759 snap 232 seq_num 759) 2024-04-26T00:33:16.634 INFO:tasks.rados.rados.0.smithi045.stdout:2253: expect (ObjNum 793 snap 242 seq_num 793) 2024-04-26T00:33:17.507 INFO:tasks.rados.rados.0.smithi045.stdout:2252: done (2 left) 2024-04-26T00:33:17.507 INFO:tasks.rados.rados.0.smithi045.stdout:2253: done (1 left) 2024-04-26T00:33:17.508 INFO:tasks.rados.rados.0.smithi045.stdout:2254: done (0 left) 2024-04-26T00:33:17.508 INFO:tasks.rados.rados.0.smithi045.stdout:2255: delete oid 29 current snap is 254 2024-04-26T00:33:17.520 INFO:tasks.rados.rados.0.smithi045.stdout:2255: done (0 left) 2024-04-26T00:33:17.521 INFO:tasks.rados.rados.0.smithi045.stdout:2256: rollback oid 27 current snap is 254 2024-04-26T00:33:17.521 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 27 to 236 2024-04-26T00:33:17.521 INFO:tasks.rados.rados.0.smithi045.stdout:2257: snap_create 2024-04-26T00:33:17.530 INFO:tasks.rados.rados.0.smithi045.stdout:2256: finishing rollback tid 0 to smithi04526579-27 2024-04-26T00:33:17.531 INFO:tasks.rados.rados.0.smithi045.stdout:2256: finishing rollback tid 1 to smithi04526579-27 2024-04-26T00:33:17.531 INFO:tasks.rados.rados.0.smithi045.stdout:2256: finishing rollback tid 2 to smithi04526579-27 2024-04-26T00:33:17.531 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 1529 (ObjNum 750 snap 232 seq_num 750) dirty exists 2024-04-26T00:33:18.489 INFO:tasks.rados.rados.0.smithi045.stdout:2256: done (1 left) 2024-04-26T00:33:18.489 INFO:tasks.rados.rados.0.smithi045.stdout:2257: done (0 left) 2024-04-26T00:33:18.489 INFO:tasks.rados.rados.0.smithi045.stdout:2258: read oid 30 snap -1 2024-04-26T00:33:18.489 INFO:tasks.rados.rados.0.smithi045.stdout:2258: expect deleted 2024-04-26T00:33:18.489 INFO:tasks.rados.rados.0.smithi045.stdout:2259: write oid 27 current snap is 255 2024-04-26T00:33:18.489 INFO:tasks.rados.rados.0.smithi045.stdout:2259: seq_num 814 ranges {480898=502457,1699872=737281,3087022=593876} 2024-04-26T00:33:18.505 INFO:tasks.rados.rados.0.smithi045.stdout:2259: writing smithi04526579-27 from 480898 to 983355 tid 1 2024-04-26T00:33:18.529 INFO:tasks.rados.rados.0.smithi045.stdout:2259: writing smithi04526579-27 from 1699872 to 2437153 tid 2 2024-04-26T00:33:18.547 INFO:tasks.rados.rados.0.smithi045.stdout:2259: writing smithi04526579-27 from 3087022 to 3680898 tid 3 2024-04-26T00:33:18.548 INFO:tasks.rados.rados.0.smithi045.stdout:2260: delete oid 33 current snap is 255 2024-04-26T00:33:18.548 INFO:tasks.rados.rados.0.smithi045.stdout:2259: finishing write tid 1 to smithi04526579-27 2024-04-26T00:33:18.548 INFO:tasks.rados.rados.0.smithi045.stdout:2259: finishing write tid 2 to smithi04526579-27 2024-04-26T00:33:18.554 INFO:tasks.rados.rados.0.smithi045.stdout:2258: done (2 left) 2024-04-26T00:33:18.554 INFO:tasks.rados.rados.0.smithi045.stdout:2260: done (1 left) 2024-04-26T00:33:18.554 INFO:tasks.rados.rados.0.smithi045.stdout:2261: read oid 33 snap -1 2024-04-26T00:33:18.554 INFO:tasks.rados.rados.0.smithi045.stdout:2261: expect deleted 2024-04-26T00:33:18.554 INFO:tasks.rados.rados.0.smithi045.stdout:2262: read oid 28 snap -1 2024-04-26T00:33:18.555 INFO:tasks.rados.rados.0.smithi045.stdout:2262: expect (ObjNum 802 snap 246 seq_num 802) 2024-04-26T00:33:18.555 INFO:tasks.rados.rados.0.smithi045.stdout:2263: read oid 4 snap 160 2024-04-26T00:33:18.555 INFO:tasks.rados.rados.0.smithi045.stdout:2263: expect (ObjNum 474 snap 147 seq_num 474) 2024-04-26T00:33:18.555 INFO:tasks.rados.rados.0.smithi045.stdout:2264: read oid 10 snap -1 2024-04-26T00:33:18.555 INFO:tasks.rados.rados.0.smithi045.stdout:2264: expect (ObjNum 788 snap 241 seq_num 788) 2024-04-26T00:33:18.555 INFO:tasks.rados.rados.0.smithi045.stdout:2265: rollback oid 14 current snap is 255 2024-04-26T00:33:18.555 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 14 to 236 2024-04-26T00:33:18.555 INFO:tasks.rados.rados.0.smithi045.stdout:2266: copy_from oid 6 from oid 20 current snap is 255 2024-04-26T00:33:18.556 INFO:tasks.rados.rados.0.smithi045.stdout:2267: rollback oid 7 current snap is 255 2024-04-26T00:33:18.556 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 7 to 254 2024-04-26T00:33:18.556 INFO:tasks.rados.rados.0.smithi045.stdout:2268: write (excl) oid 9 current snap is 255 2024-04-26T00:33:18.556 INFO:tasks.rados.rados.0.smithi045.stdout:2268: seq_num 816 ranges {514512=652955,1700483=740950,3025515=688997} 2024-04-26T00:33:18.579 INFO:tasks.rados.rados.0.smithi045.stdout:2268: writing smithi04526579-9 from 514512 to 1167467 tid 1 2024-04-26T00:33:18.594 INFO:tasks.rados.rados.0.smithi045.stdout:2268: writing smithi04526579-9 from 1700483 to 2441433 tid 2 2024-04-26T00:33:18.607 INFO:tasks.rados.rados.0.smithi045.stdout:2268: writing smithi04526579-9 from 3025515 to 3714512 tid 3 2024-04-26T00:33:18.608 INFO:tasks.rados.rados.0.smithi045.stdout:2269: copy_from oid 5 from oid 32 current snap is 255 2024-04-26T00:33:18.608 INFO:tasks.rados.rados.0.smithi045.stdout:2261: done (9 left) 2024-04-26T00:33:18.608 INFO:tasks.rados.rados.0.smithi045.stdout:2270: snap_remove snap 210 2024-04-26T00:33:19.516 INFO:tasks.rados.rados.0.smithi045.stdout:2270: done (9 left) 2024-04-26T00:33:19.516 INFO:tasks.rados.rados.0.smithi045.stdout:2271: rollback oid 30 current snap is 255 2024-04-26T00:33:19.516 INFO:tasks.rados.rados.0.smithi045.stdout:2265: finishing rollback tid 1 to smithi04526579-14 2024-04-26T00:33:19.516 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 1128 (ObjNum 33 snap 0 seq_num 2241941120) dirty dne 2024-04-26T00:33:19.516 INFO:tasks.rados.rados.0.smithi045.stdout:2259: finishing write tid 3 to smithi04526579-27 2024-04-26T00:33:19.516 INFO:tasks.rados.rados.0.smithi045.stdout:2259: finishing write tid 4 to smithi04526579-27 2024-04-26T00:33:19.517 INFO:tasks.rados.rados.0.smithi045.stdout:2259: finishing write tid 5 to smithi04526579-27 2024-04-26T00:33:19.517 INFO:tasks.rados.rados.0.smithi045.stdout:2259: oid 27 updating version 0 to 1531 2024-04-26T00:33:19.517 INFO:tasks.rados.rados.0.smithi045.stdout:2259: oid 27 updating version 1531 to 1532 2024-04-26T00:33:19.517 INFO:tasks.rados.rados.0.smithi045.stdout:2259: oid 27 updating version 1532 to 1533 2024-04-26T00:33:19.517 INFO:tasks.rados.rados.0.smithi045.stdout:2259: oid 27 updating version 1533 to 1534 2024-04-26T00:33:19.517 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 1534 (ObjNum 814 snap 255 seq_num 814) dirty exists 2024-04-26T00:33:19.517 INFO:tasks.rados.rados.0.smithi045.stdout:2259: left oid 27 (ObjNum 814 snap 255 seq_num 814) 2024-04-26T00:33:19.517 INFO:tasks.rados.rados.0.smithi045.stdout:2267: finishing rollback tid 1 to smithi04526579-7 2024-04-26T00:33:19.517 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 1787 (ObjNum 33 snap 0 seq_num 3717725323) dirty dne 2024-04-26T00:33:19.517 INFO:tasks.rados.rados.0.smithi045.stdout:2266: finishing copy_from racing read to smithi04526579-6 2024-04-26T00:33:19.517 INFO:tasks.rados.rados.0.smithi045.stdout:2262: expect (ObjNum 802 snap 246 seq_num 802) 2024-04-26T00:33:19.660 INFO:tasks.rados.rados.0.smithi045.stdout:2264: expect (ObjNum 788 snap 241 seq_num 788) 2024-04-26T00:33:19.891 INFO:tasks.rados.rados.0.smithi045.stdout:2266: finishing copy_from to smithi04526579-6 2024-04-26T00:33:19.892 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 1064 (ObjNum 542 snap 167 seq_num 542) dirty exists 2024-04-26T00:33:19.892 INFO:tasks.rados.rados.0.smithi045.stdout:2263: expect (ObjNum 474 snap 147 seq_num 474) 2024-04-26T00:33:20.141 INFO:tasks.rados.rados.0.smithi045.stdout:2268: finishing write tid 1 to smithi04526579-9 2024-04-26T00:33:20.142 INFO:tasks.rados.rados.0.smithi045.stdout:2268: finishing write tid 2 to smithi04526579-9 2024-04-26T00:33:20.142 INFO:tasks.rados.rados.0.smithi045.stdout:2268: finishing write tid 3 to smithi04526579-9 2024-04-26T00:33:20.142 INFO:tasks.rados.rados.0.smithi045.stdout:2268: finishing write tid 4 to smithi04526579-9 2024-04-26T00:33:20.142 INFO:tasks.rados.rados.0.smithi045.stdout:2268: finishing write tid 5 to smithi04526579-9 2024-04-26T00:33:20.142 INFO:tasks.rados.rados.0.smithi045.stdout:2268: oid 9 updating version 0 to 1542 2024-04-26T00:33:20.142 INFO:tasks.rados.rados.0.smithi045.stdout:2268: oid 9 version 1542 is already newer than 1539 2024-04-26T00:33:20.142 INFO:tasks.rados.rados.0.smithi045.stdout:2268: oid 9 version 1542 is already newer than 1541 2024-04-26T00:33:20.142 INFO:tasks.rados.rados.0.smithi045.stdout:2268: oid 9 version 1542 is already newer than 1540 2024-04-26T00:33:20.143 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 1542 (ObjNum 816 snap 255 seq_num 816) dirty exists 2024-04-26T00:33:20.143 INFO:tasks.rados.rados.0.smithi045.stdout:2268: left oid 9 (ObjNum 816 snap 255 seq_num 816) 2024-04-26T00:33:20.143 INFO:tasks.rados.rados.0.smithi045.stdout:2269: finishing copy_from racing read to smithi04526579-5 2024-04-26T00:33:20.143 INFO:tasks.rados.rados.0.smithi045.stdout:2269: finishing copy_from to smithi04526579-5 2024-04-26T00:33:20.143 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 1292 (ObjNum 769 snap 238 seq_num 769) dirty exists 2024-04-26T00:33:20.143 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 30 to 253 2024-04-26T00:33:20.144 INFO:tasks.rados.rados.0.smithi045.stdout:2259: done (9 left) 2024-04-26T00:33:20.144 INFO:tasks.rados.rados.0.smithi045.stdout:2262: done (8 left) 2024-04-26T00:33:20.144 INFO:tasks.rados.rados.0.smithi045.stdout:2263: done (7 left) 2024-04-26T00:33:20.144 INFO:tasks.rados.rados.0.smithi045.stdout:2264: done (6 left) 2024-04-26T00:33:20.144 INFO:tasks.rados.rados.0.smithi045.stdout:2265: done (5 left) 2024-04-26T00:33:20.144 INFO:tasks.rados.rados.0.smithi045.stdout:2266: done (4 left) 2024-04-26T00:33:20.144 INFO:tasks.rados.rados.0.smithi045.stdout:2267: done (3 left) 2024-04-26T00:33:20.145 INFO:tasks.rados.rados.0.smithi045.stdout:2268: done (2 left) 2024-04-26T00:33:20.145 INFO:tasks.rados.rados.0.smithi045.stdout:2269: done (1 left) 2024-04-26T00:33:20.145 INFO:tasks.rados.rados.0.smithi045.stdout:2272: snap_create 2024-04-26T00:33:20.145 INFO:tasks.rados.rados.0.smithi045.stdout:2271: finishing rollback tid 1 to smithi04526579-30 2024-04-26T00:33:20.145 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 1056 (ObjNum 129 snap 0 seq_num 1476942459) dirty dne 2024-04-26T00:33:20.518 INFO:tasks.rados.rados.0.smithi045.stdout:2271: done (1 left) 2024-04-26T00:33:20.518 INFO:tasks.rados.rados.0.smithi045.stdout:2272: done (0 left) 2024-04-26T00:33:20.518 INFO:tasks.rados.rados.0.smithi045.stdout:2273: delete oid 3 current snap is 256 2024-04-26T00:33:20.536 INFO:tasks.rados.rados.0.smithi045.stdout:2273: done (0 left) 2024-04-26T00:33:20.536 INFO:tasks.rados.rados.0.smithi045.stdout:2274: write oid 13 current snap is 256 2024-04-26T00:33:20.536 INFO:tasks.rados.rados.0.smithi045.stdout:2274: seq_num 818 ranges {548126=403453,1417447=744619,2680361=447683,3680450=67676} 2024-04-26T00:33:20.551 INFO:tasks.rados.rados.0.smithi045.stdout:2274: writing smithi04526579-13 from 548126 to 951579 tid 1 2024-04-26T00:33:20.572 INFO:tasks.rados.rados.0.smithi045.stdout:2274: writing smithi04526579-13 from 1417447 to 2162066 tid 2 2024-04-26T00:33:20.586 INFO:tasks.rados.rados.0.smithi045.stdout:2274: writing smithi04526579-13 from 2680361 to 3128044 tid 3 2024-04-26T00:33:20.594 INFO:tasks.rados.rados.0.smithi045.stdout:2274: writing smithi04526579-13 from 3680450 to 3748126 tid 4 2024-04-26T00:33:20.594 INFO:tasks.rados.rados.0.smithi045.stdout:2275: write (excl) oid 20 current snap is 256 2024-04-26T00:33:20.595 INFO:tasks.rados.rados.0.smithi045.stdout:2274: finishing write tid 1 to smithi04526579-13 2024-04-26T00:33:20.595 INFO:tasks.rados.rados.0.smithi045.stdout:2274: finishing write tid 2 to smithi04526579-13 2024-04-26T00:33:20.595 INFO:tasks.rados.rados.0.smithi045.stdout:2274: finishing write tid 3 to smithi04526579-13 2024-04-26T00:33:20.595 INFO:tasks.rados.rados.0.smithi045.stdout:2275: seq_num 819 ranges {564933=478702,1759576=488277,2991431=658955} 2024-04-26T00:33:20.607 INFO:tasks.rados.rados.0.smithi045.stdout:2275: writing smithi04526579-20 from 564933 to 1043635 tid 1 2024-04-26T00:33:20.622 INFO:tasks.rados.rados.0.smithi045.stdout:2275: writing smithi04526579-20 from 1759576 to 2247853 tid 2 2024-04-26T00:33:20.639 INFO:tasks.rados.rados.0.smithi045.stdout:2275: writing smithi04526579-20 from 2991431 to 3650386 tid 3 2024-04-26T00:33:20.640 INFO:tasks.rados.rados.0.smithi045.stdout:2276: copy_from oid 42 from oid 9 current snap is 256 2024-04-26T00:33:20.640 INFO:tasks.rados.rados.0.smithi045.stdout:2274: finishing write tid 4 to smithi04526579-13 2024-04-26T00:33:20.640 INFO:tasks.rados.rados.0.smithi045.stdout:2274: finishing write tid 5 to smithi04526579-13 2024-04-26T00:33:20.640 INFO:tasks.rados.rados.0.smithi045.stdout:2274: finishing write tid 6 to smithi04526579-13 2024-04-26T00:33:20.640 INFO:tasks.rados.rados.0.smithi045.stdout:2274: oid 13 updating version 0 to 1533 2024-04-26T00:33:20.640 INFO:tasks.rados.rados.0.smithi045.stdout:2274: oid 13 updating version 1533 to 1534 2024-04-26T00:33:20.641 INFO:tasks.rados.rados.0.smithi045.stdout:2274: oid 13 updating version 1534 to 1535 2024-04-26T00:33:20.641 INFO:tasks.rados.rados.0.smithi045.stdout:2274: oid 13 updating version 1535 to 1537 2024-04-26T00:33:20.641 INFO:tasks.rados.rados.0.smithi045.stdout:2274: oid 13 version 1537 is already newer than 1536 2024-04-26T00:33:20.641 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 1537 (ObjNum 818 snap 256 seq_num 818) dirty exists 2024-04-26T00:33:20.641 INFO:tasks.rados.rados.0.smithi045.stdout:2274: left oid 13 (ObjNum 818 snap 256 seq_num 818) 2024-04-26T00:33:20.641 INFO:tasks.rados.rados.0.smithi045.stdout:2275: finishing write tid 1 to smithi04526579-20 2024-04-26T00:33:20.642 INFO:tasks.rados.rados.0.smithi045.stdout:2275: finishing write tid 2 to smithi04526579-20 2024-04-26T00:33:20.642 INFO:tasks.rados.rados.0.smithi045.stdout:2274: done (2 left) 2024-04-26T00:33:20.642 INFO:tasks.rados.rados.0.smithi045.stdout:2277: snap_create 2024-04-26T00:33:20.647 INFO:tasks.rados.rados.0.smithi045.stdout:2275: finishing write tid 3 to smithi04526579-20 2024-04-26T00:33:20.648 INFO:tasks.rados.rados.0.smithi045.stdout:2275: finishing write tid 4 to smithi04526579-20 2024-04-26T00:33:20.652 INFO:tasks.rados.rados.0.smithi045.stdout:2275: finishing write tid 5 to smithi04526579-20 2024-04-26T00:33:20.653 INFO:tasks.rados.rados.0.smithi045.stdout:2275: oid 20 updating version 0 to 1294 2024-04-26T00:33:20.653 INFO:tasks.rados.rados.0.smithi045.stdout:2275: oid 20 updating version 1294 to 1296 2024-04-26T00:33:20.653 INFO:tasks.rados.rados.0.smithi045.stdout:2275: oid 20 version 1296 is already newer than 1295 2024-04-26T00:33:20.653 INFO:tasks.rados.rados.0.smithi045.stdout:2275: oid 20 updating version 1296 to 1297 2024-04-26T00:33:20.653 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 1297 (ObjNum 819 snap 256 seq_num 819) dirty exists 2024-04-26T00:33:20.654 INFO:tasks.rados.rados.0.smithi045.stdout:2275: left oid 20 (ObjNum 819 snap 256 seq_num 819) 2024-04-26T00:33:20.670 INFO:tasks.rados.rados.0.smithi045.stdout:2276: finishing copy_from racing read to smithi04526579-42 2024-04-26T00:33:20.672 INFO:tasks.rados.rados.0.smithi045.stdout:2276: finishing copy_from to smithi04526579-42 2024-04-26T00:33:20.673 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 1542 (ObjNum 816 snap 255 seq_num 816) dirty exists 2024-04-26T00:33:21.533 INFO:tasks.rados.rados.0.smithi045.stdout:2275: done (2 left) 2024-04-26T00:33:21.533 INFO:tasks.rados.rados.0.smithi045.stdout:2276: done (1 left) 2024-04-26T00:33:21.533 INFO:tasks.rados.rados.0.smithi045.stdout:2277: done (0 left) 2024-04-26T00:33:21.533 INFO:tasks.rados.rados.0.smithi045.stdout:2278: write oid 29 current snap is 257 2024-04-26T00:33:21.533 INFO:tasks.rados.rados.0.smithi045.stdout:2278: seq_num 820 ranges {581740=553951,1818058=748288,3018854=470227} 2024-04-26T00:33:21.547 INFO:tasks.rados.rados.0.smithi045.stdout:2278: writing smithi04526579-29 from 581740 to 1135691 tid 1 2024-04-26T00:33:21.564 INFO:tasks.rados.rados.0.smithi045.stdout:2278: writing smithi04526579-29 from 1818058 to 2566346 tid 2 2024-04-26T00:33:21.576 INFO:tasks.rados.rados.0.smithi045.stdout:2278: writing smithi04526579-29 from 3018854 to 3489081 tid 3 2024-04-26T00:33:21.576 INFO:tasks.rados.rados.0.smithi045.stdout:2279: snap_create 2024-04-26T00:33:21.576 INFO:tasks.rados.rados.0.smithi045.stdout:2278: finishing write tid 1 to smithi04526579-29 2024-04-26T00:33:21.576 INFO:tasks.rados.rados.0.smithi045.stdout:2278: finishing write tid 2 to smithi04526579-29 2024-04-26T00:33:21.582 INFO:tasks.rados.rados.0.smithi045.stdout:2278: finishing write tid 3 to smithi04526579-29 2024-04-26T00:33:21.584 INFO:tasks.rados.rados.0.smithi045.stdout:2278: finishing write tid 4 to smithi04526579-29 2024-04-26T00:33:21.584 INFO:tasks.rados.rados.0.smithi045.stdout:2278: finishing write tid 5 to smithi04526579-29 2024-04-26T00:33:21.584 INFO:tasks.rados.rados.0.smithi045.stdout:2278: oid 29 updating version 0 to 1541 2024-04-26T00:33:21.584 INFO:tasks.rados.rados.0.smithi045.stdout:2278: oid 29 version 1541 is already newer than 1539 2024-04-26T00:33:21.584 INFO:tasks.rados.rados.0.smithi045.stdout:2278: oid 29 version 1541 is already newer than 1540 2024-04-26T00:33:21.584 INFO:tasks.rados.rados.0.smithi045.stdout:2278: oid 29 version 1541 is already newer than 1538 2024-04-26T00:33:21.584 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 1541 (ObjNum 820 snap 257 seq_num 820) dirty exists 2024-04-26T00:33:21.584 INFO:tasks.rados.rados.0.smithi045.stdout:2278: left oid 29 (ObjNum 820 snap 257 seq_num 820) 2024-04-26T00:33:22.545 INFO:tasks.rados.rados.0.smithi045.stdout:2278: done (1 left) 2024-04-26T00:33:22.545 INFO:tasks.rados.rados.0.smithi045.stdout:2279: done (0 left) 2024-04-26T00:33:22.545 INFO:tasks.rados.rados.0.smithi045.stdout:2280: delete oid 11 current snap is 258 2024-04-26T00:33:22.556 INFO:tasks.rados.rados.0.smithi045.stdout:2280: done (0 left) 2024-04-26T00:33:22.556 INFO:tasks.rados.rados.0.smithi045.stdout:2281: delete oid 37 current snap is 258 2024-04-26T00:33:22.568 INFO:tasks.rados.rados.0.smithi045.stdout:2281: done (0 left) 2024-04-26T00:33:22.568 INFO:tasks.rados.rados.0.smithi045.stdout:2282: write oid 42 current snap is 258 2024-04-26T00:33:22.568 INFO:tasks.rados.rados.0.smithi045.stdout:2282: seq_num 823 ranges {632161=496051,1593504=495615,2701123=420396,3611645=220516} 2024-04-26T00:33:22.586 INFO:tasks.rados.rados.0.smithi045.stdout:2282: writing smithi04526579-42 from 632161 to 1128212 tid 1 2024-04-26T00:33:22.602 INFO:tasks.rados.rados.0.smithi045.stdout:2282: writing smithi04526579-42 from 1593504 to 2089119 tid 2 2024-04-26T00:33:22.619 INFO:tasks.rados.rados.0.smithi045.stdout:2282: writing smithi04526579-42 from 2701123 to 3121519 tid 3 2024-04-26T00:33:22.629 INFO:tasks.rados.rados.0.smithi045.stdout:2282: writing smithi04526579-42 from 3611645 to 3832161 tid 4 2024-04-26T00:33:22.629 INFO:tasks.rados.rados.0.smithi045.stdout:2283: write oid 12 current snap is 258 2024-04-26T00:33:22.629 INFO:tasks.rados.rados.0.smithi045.stdout:2283: seq_num 824 ranges {648968=571300,1651986=639273,2728546=631668} 2024-04-26T00:33:22.642 INFO:tasks.rados.rados.0.smithi045.stdout:2283: writing smithi04526579-12 from 648968 to 1220268 tid 1 2024-04-26T00:33:22.652 INFO:tasks.rados.rados.0.smithi045.stdout:2283: writing smithi04526579-12 from 1651986 to 2291259 tid 2 2024-04-26T00:33:22.662 INFO:tasks.rados.rados.0.smithi045.stdout:2283: writing smithi04526579-12 from 2728546 to 3360214 tid 3 2024-04-26T00:33:22.662 INFO:tasks.rados.rados.0.smithi045.stdout:2284: read oid 37 snap -1 2024-04-26T00:33:22.663 INFO:tasks.rados.rados.0.smithi045.stdout:2284: expect deleted 2024-04-26T00:33:22.663 INFO:tasks.rados.rados.0.smithi045.stdout:2285: delete oid 39 current snap is 258 2024-04-26T00:33:22.663 INFO:tasks.rados.rados.0.smithi045.stdout:2282: finishing write tid 1 to smithi04526579-42 2024-04-26T00:33:22.663 INFO:tasks.rados.rados.0.smithi045.stdout:2282: finishing write tid 2 to smithi04526579-42 2024-04-26T00:33:22.663 INFO:tasks.rados.rados.0.smithi045.stdout:2282: finishing write tid 3 to smithi04526579-42 2024-04-26T00:33:22.663 INFO:tasks.rados.rados.0.smithi045.stdout:2282: finishing write tid 4 to smithi04526579-42 2024-04-26T00:33:22.663 INFO:tasks.rados.rados.0.smithi045.stdout:2282: finishing write tid 5 to smithi04526579-42 2024-04-26T00:33:22.663 INFO:tasks.rados.rados.0.smithi045.stdout:2282: finishing write tid 6 to smithi04526579-42 2024-04-26T00:33:22.663 INFO:tasks.rados.rados.0.smithi045.stdout:2282: oid 42 updating version 0 to 1546 2024-04-26T00:33:22.663 INFO:tasks.rados.rados.0.smithi045.stdout:2282: oid 42 updating version 1546 to 1549 2024-04-26T00:33:22.663 INFO:tasks.rados.rados.0.smithi045.stdout:2282: oid 42 version 1549 is already newer than 1548 2024-04-26T00:33:22.663 INFO:tasks.rados.rados.0.smithi045.stdout:2282: oid 42 version 1549 is already newer than 1547 2024-04-26T00:33:22.663 INFO:tasks.rados.rados.0.smithi045.stdout:2282: oid 42 updating version 1549 to 1550 2024-04-26T00:33:22.664 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 1550 (ObjNum 823 snap 258 seq_num 823) dirty exists 2024-04-26T00:33:22.664 INFO:tasks.rados.rados.0.smithi045.stdout:2282: left oid 42 (ObjNum 823 snap 258 seq_num 823) 2024-04-26T00:33:22.664 INFO:tasks.rados.rados.0.smithi045.stdout:2283: finishing write tid 1 to smithi04526579-12 2024-04-26T00:33:22.664 INFO:tasks.rados.rados.0.smithi045.stdout:2283: finishing write tid 2 to smithi04526579-12 2024-04-26T00:33:22.670 INFO:tasks.rados.rados.0.smithi045.stdout:2282: done (3 left) 2024-04-26T00:33:22.670 INFO:tasks.rados.rados.0.smithi045.stdout:2284: done (2 left) 2024-04-26T00:33:22.670 INFO:tasks.rados.rados.0.smithi045.stdout:2285: done (1 left) 2024-04-26T00:33:22.670 INFO:tasks.rados.rados.0.smithi045.stdout:2286: rollback oid 30 current snap is 258 2024-04-26T00:33:22.671 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 30 to 247 2024-04-26T00:33:22.671 INFO:tasks.rados.rados.0.smithi045.stdout:2283: finishing write tid 3 to smithi04526579-12 2024-04-26T00:33:22.671 INFO:tasks.rados.rados.0.smithi045.stdout:2287: copy_from oid 28 from oid 25 current snap is 258 2024-04-26T00:33:22.671 INFO:tasks.rados.rados.0.smithi045.stdout:2288: write (excl) oid 44 current snap is 258 2024-04-26T00:33:22.671 INFO:tasks.rados.rados.0.smithi045.stdout:2288: seq_num 826 ranges {682582=721798,2168950=642942,3583392=299190} 2024-04-26T00:33:22.681 INFO:tasks.rados.rados.0.smithi045.stdout:2288: writing smithi04526579-44 from 682582 to 1404380 tid 1 2024-04-26T00:33:22.693 INFO:tasks.rados.rados.0.smithi045.stdout:2288: writing smithi04526579-44 from 2168950 to 2811892 tid 2 2024-04-26T00:33:22.702 INFO:tasks.rados.rados.0.smithi045.stdout:2288: writing smithi04526579-44 from 3583392 to 3882582 tid 3 2024-04-26T00:33:22.702 INFO:tasks.rados.rados.0.smithi045.stdout:2289: copy_from oid 40 from oid 47 current snap is 258 2024-04-26T00:33:22.702 INFO:tasks.rados.rados.0.smithi045.stdout:2283: finishing write tid 4 to smithi04526579-12 2024-04-26T00:33:22.702 INFO:tasks.rados.rados.0.smithi045.stdout:2283: finishing write tid 5 to smithi04526579-12 2024-04-26T00:33:22.702 INFO:tasks.rados.rados.0.smithi045.stdout:2283: oid 12 updating version 0 to 1454 2024-04-26T00:33:22.702 INFO:tasks.rados.rados.0.smithi045.stdout:2283: oid 12 updating version 1454 to 1455 2024-04-26T00:33:22.702 INFO:tasks.rados.rados.0.smithi045.stdout:2283: oid 12 version 1455 is already newer than 1453 2024-04-26T00:33:22.702 INFO:tasks.rados.rados.0.smithi045.stdout:2283: oid 12 updating version 1455 to 1456 2024-04-26T00:33:22.702 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 1456 (ObjNum 824 snap 258 seq_num 824) dirty exists 2024-04-26T00:33:22.702 INFO:tasks.rados.rados.0.smithi045.stdout:2283: left oid 12 (ObjNum 824 snap 258 seq_num 824) 2024-04-26T00:33:22.703 INFO:tasks.rados.rados.0.smithi045.stdout:2283: done (4 left) 2024-04-26T00:33:22.703 INFO:tasks.rados.rados.0.smithi045.stdout:2290: read oid 6 snap -1 2024-04-26T00:33:22.703 INFO:tasks.rados.rados.0.smithi045.stdout:2290: expect (ObjNum 542 snap 167 seq_num 542) 2024-04-26T00:33:22.703 INFO:tasks.rados.rados.0.smithi045.stdout:2291: delete oid 14 current snap is 258 2024-04-26T00:33:22.703 INFO:tasks.rados.rados.0.smithi045.stdout:2286: finishing rollback tid 1 to smithi04526579-30 2024-04-26T00:33:22.703 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 1056 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-04-26T00:33:22.703 INFO:tasks.rados.rados.0.smithi045.stdout:2287: finishing copy_from racing read to smithi04526579-28 2024-04-26T00:33:22.703 INFO:tasks.rados.rados.0.smithi045.stdout:2287: finishing copy_from to smithi04526579-28 2024-04-26T00:33:22.703 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 1911 (ObjNum 604 snap 186 seq_num 604) dirty exists 2024-04-26T00:33:22.703 INFO:tasks.rados.rados.0.smithi045.stdout:2288: finishing write tid 1 to smithi04526579-44 2024-04-26T00:33:22.703 INFO:tasks.rados.rados.0.smithi045.stdout:2288: finishing write tid 2 to smithi04526579-44 2024-04-26T00:33:22.706 INFO:tasks.rados.rados.0.smithi045.stdout:2286: done (5 left) 2024-04-26T00:33:22.706 INFO:tasks.rados.rados.0.smithi045.stdout:2287: done (4 left) 2024-04-26T00:33:22.706 INFO:tasks.rados.rados.0.smithi045.stdout:2291: done (3 left) 2024-04-26T00:33:22.706 INFO:tasks.rados.rados.0.smithi045.stdout:2292: read oid 24 snap 238 2024-04-26T00:33:22.706 INFO:tasks.rados.rados.0.smithi045.stdout:2292: expect (ObjNum 735 snap 227 seq_num 735) 2024-04-26T00:33:22.706 INFO:tasks.rados.rados.0.smithi045.stdout:2293: delete oid 23 current snap is 258 2024-04-26T00:33:22.712 INFO:tasks.rados.rados.0.smithi045.stdout:2288: finishing write tid 3 to smithi04526579-44 2024-04-26T00:33:22.712 INFO:tasks.rados.rados.0.smithi045.stdout:2288: finishing write tid 4 to smithi04526579-44 2024-04-26T00:33:22.714 INFO:tasks.rados.rados.0.smithi045.stdout:2293: done (4 left) 2024-04-26T00:33:22.714 INFO:tasks.rados.rados.0.smithi045.stdout:2294: read oid 21 snap -1 2024-04-26T00:33:22.714 INFO:tasks.rados.rados.0.smithi045.stdout:2294: expect deleted 2024-04-26T00:33:22.714 INFO:tasks.rados.rados.0.smithi045.stdout:2295: delete oid 43 current snap is 258 2024-04-26T00:33:22.714 INFO:tasks.rados.rados.0.smithi045.stdout:2288: finishing write tid 5 to smithi04526579-44 2024-04-26T00:33:22.714 INFO:tasks.rados.rados.0.smithi045.stdout:2288: oid 44 updating version 0 to 1301 2024-04-26T00:33:22.714 INFO:tasks.rados.rados.0.smithi045.stdout:2288: oid 44 version 1301 is already newer than 1300 2024-04-26T00:33:22.715 INFO:tasks.rados.rados.0.smithi045.stdout:2288: oid 44 updating version 1301 to 1302 2024-04-26T00:33:22.715 INFO:tasks.rados.rados.0.smithi045.stdout:2288: oid 44 version 1302 is already newer than 1299 2024-04-26T00:33:22.715 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 1302 (ObjNum 826 snap 258 seq_num 826) dirty exists 2024-04-26T00:33:22.715 INFO:tasks.rados.rados.0.smithi045.stdout:2288: left oid 44 (ObjNum 826 snap 258 seq_num 826) 2024-04-26T00:33:22.715 INFO:tasks.rados.rados.0.smithi045.stdout:2290: expect (ObjNum 542 snap 167 seq_num 542) 2024-04-26T00:33:22.912 INFO:tasks.rados.rados.0.smithi045.stdout:2292: expect (ObjNum 735 snap 227 seq_num 735) 2024-04-26T00:33:23.049 INFO:tasks.rados.rados.0.smithi045.stdout:2289: finishing copy_from racing read to smithi04526579-40 2024-04-26T00:33:23.049 INFO:tasks.rados.rados.0.smithi045.stdout:2289: finishing copy_from to smithi04526579-40 2024-04-26T00:33:23.049 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 1343 (ObjNum 782 snap 240 seq_num 782) dirty exists 2024-04-26T00:33:23.049 INFO:tasks.rados.rados.0.smithi045.stdout:2288: done (5 left) 2024-04-26T00:33:23.049 INFO:tasks.rados.rados.0.smithi045.stdout:2289: done (4 left) 2024-04-26T00:33:23.049 INFO:tasks.rados.rados.0.smithi045.stdout:2290: done (3 left) 2024-04-26T00:33:23.049 INFO:tasks.rados.rados.0.smithi045.stdout:2292: done (2 left) 2024-04-26T00:33:23.049 INFO:tasks.rados.rados.0.smithi045.stdout:2294: done (1 left) 2024-04-26T00:33:23.049 INFO:tasks.rados.rados.0.smithi045.stdout:2295: done (0 left) 2024-04-26T00:33:23.049 INFO:tasks.rados.rados.0.smithi045.stdout:2296: delete oid 22 current snap is 258 2024-04-26T00:33:23.056 INFO:tasks.rados.rados.0.smithi045.stdout:2296: done (0 left) 2024-04-26T00:33:23.056 INFO:tasks.rados.rados.0.smithi045.stdout:2297: write oid 14 current snap is 258 2024-04-26T00:33:23.056 INFO:tasks.rados.rados.0.smithi045.stdout:2297: seq_num 831 ranges {766617=414396,1945007=510291,3204154=626925} 2024-04-26T00:33:23.075 INFO:tasks.rados.rados.0.smithi045.stdout:2297: writing smithi04526579-14 from 766617 to 1181013 tid 1 2024-04-26T00:33:23.096 INFO:tasks.rados.rados.0.smithi045.stdout:2297: writing smithi04526579-14 from 1945007 to 2455298 tid 2 2024-04-26T00:33:23.117 INFO:tasks.rados.rados.0.smithi045.stdout:2297: writing smithi04526579-14 from 3204154 to 3831079 tid 3 2024-04-26T00:33:23.117 INFO:tasks.rados.rados.0.smithi045.stdout:2298: read oid 20 snap -1 2024-04-26T00:33:23.117 INFO:tasks.rados.rados.0.smithi045.stdout:2298: expect (ObjNum 819 snap 256 seq_num 819) 2024-04-26T00:33:23.117 INFO:tasks.rados.rados.0.smithi045.stdout:2299: snap_create 2024-04-26T00:33:23.117 INFO:tasks.rados.rados.0.smithi045.stdout:2297: finishing write tid 1 to smithi04526579-14 2024-04-26T00:33:23.117 INFO:tasks.rados.rados.0.smithi045.stdout:2297: finishing write tid 2 to smithi04526579-14 2024-04-26T00:33:23.124 INFO:tasks.rados.rados.0.smithi045.stdout:2297: finishing write tid 3 to smithi04526579-14 2024-04-26T00:33:23.125 INFO:tasks.rados.rados.0.smithi045.stdout:2297: finishing write tid 4 to smithi04526579-14 2024-04-26T00:33:23.126 INFO:tasks.rados.rados.0.smithi045.stdout:2297: finishing write tid 5 to smithi04526579-14 2024-04-26T00:33:23.126 INFO:tasks.rados.rados.0.smithi045.stdout:2297: oid 14 updating version 0 to 1143 2024-04-26T00:33:23.126 INFO:tasks.rados.rados.0.smithi045.stdout:2297: oid 14 version 1143 is already newer than 1140 2024-04-26T00:33:23.126 INFO:tasks.rados.rados.0.smithi045.stdout:2297: oid 14 version 1143 is already newer than 1142 2024-04-26T00:33:23.126 INFO:tasks.rados.rados.0.smithi045.stdout:2297: oid 14 version 1143 is already newer than 1141 2024-04-26T00:33:23.126 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 1143 (ObjNum 831 snap 258 seq_num 831) dirty exists 2024-04-26T00:33:23.126 INFO:tasks.rados.rados.0.smithi045.stdout:2297: left oid 14 (ObjNum 831 snap 258 seq_num 831) 2024-04-26T00:33:23.137 INFO:tasks.rados.rados.0.smithi045.stdout:2298: expect (ObjNum 819 snap 256 seq_num 819) 2024-04-26T00:33:23.491 INFO:tasks.rados.rados.0.smithi045.stdout:2297: done (2 left) 2024-04-26T00:33:23.491 INFO:tasks.rados.rados.0.smithi045.stdout:2298: done (1 left) 2024-04-26T00:33:23.491 INFO:tasks.rados.rados.0.smithi045.stdout:2299: done (0 left) 2024-04-26T00:33:23.491 INFO:tasks.rados.rados.0.smithi045.stdout:2300: snap_remove snap 244 2024-04-26T00:33:24.496 INFO:tasks.rados.rados.0.smithi045.stdout:2300: done (0 left) 2024-04-26T00:33:24.496 INFO:tasks.rados.rados.0.smithi045.stdout:2301: snap_remove snap 251 2024-04-26T00:33:25.504 INFO:tasks.rados.rados.0.smithi045.stdout:2301: done (0 left) 2024-04-26T00:33:25.504 INFO:tasks.rados.rados.0.smithi045.stdout:2302: snap_remove snap 233 2024-04-26T00:33:26.512 INFO:tasks.rados.rados.0.smithi045.stdout:2302: done (0 left) 2024-04-26T00:33:26.512 INFO:tasks.rados.rados.0.smithi045.stdout:2303: write oid 23 current snap is 259 2024-04-26T00:33:26.512 INFO:tasks.rados.rados.0.smithi045.stdout:2303: seq_num 832 ranges {783424=489645,2003489=653949,3231577=554550} 2024-04-26T00:33:26.521 INFO:tasks.rados.rados.0.smithi045.stdout:2303: writing smithi04526579-23 from 783424 to 1273069 tid 1 2024-04-26T00:33:26.528 INFO:tasks.rados.rados.0.smithi045.stdout:2303: writing smithi04526579-23 from 2003489 to 2657438 tid 2 2024-04-26T00:33:26.535 INFO:tasks.rados.rados.0.smithi045.stdout:2303: writing smithi04526579-23 from 3231577 to 3786127 tid 3 2024-04-26T00:33:26.535 INFO:tasks.rados.rados.0.smithi045.stdout:2304: delete oid 8 current snap is 259 2024-04-26T00:33:26.536 INFO:tasks.rados.rados.0.smithi045.stdout:2303: finishing write tid 1 to smithi04526579-23 2024-04-26T00:33:26.540 INFO:tasks.rados.rados.0.smithi045.stdout:2303: finishing write tid 2 to smithi04526579-23 2024-04-26T00:33:26.542 INFO:tasks.rados.rados.0.smithi045.stdout:2304: done (1 left) 2024-04-26T00:33:26.542 INFO:tasks.rados.rados.0.smithi045.stdout:2305: read oid 33 snap -1 2024-04-26T00:33:26.542 INFO:tasks.rados.rados.0.smithi045.stdout:2305: expect deleted 2024-04-26T00:33:26.542 INFO:tasks.rados.rados.0.smithi045.stdout:2306: delete oid 19 current snap is 259 2024-04-26T00:33:26.544 INFO:tasks.rados.rados.0.smithi045.stdout:2303: finishing write tid 3 to smithi04526579-23 2024-04-26T00:33:26.544 INFO:tasks.rados.rados.0.smithi045.stdout:2303: finishing write tid 4 to smithi04526579-23 2024-04-26T00:33:26.544 INFO:tasks.rados.rados.0.smithi045.stdout:2303: finishing write tid 5 to smithi04526579-23 2024-04-26T00:33:26.545 INFO:tasks.rados.rados.0.smithi045.stdout:2303: oid 23 updating version 0 to 1461 2024-04-26T00:33:26.545 INFO:tasks.rados.rados.0.smithi045.stdout:2303: oid 23 version 1461 is already newer than 1460 2024-04-26T00:33:26.545 INFO:tasks.rados.rados.0.smithi045.stdout:2303: oid 23 version 1461 is already newer than 1459 2024-04-26T00:33:26.545 INFO:tasks.rados.rados.0.smithi045.stdout:2303: oid 23 version 1461 is already newer than 1458 2024-04-26T00:33:26.545 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 1461 (ObjNum 832 snap 259 seq_num 832) dirty exists 2024-04-26T00:33:26.545 INFO:tasks.rados.rados.0.smithi045.stdout:2303: left oid 23 (ObjNum 832 snap 259 seq_num 832) 2024-04-26T00:33:26.555 INFO:tasks.rados.rados.0.smithi045.stdout:2303: done (2 left) 2024-04-26T00:33:26.555 INFO:tasks.rados.rados.0.smithi045.stdout:2305: done (1 left) 2024-04-26T00:33:26.555 INFO:tasks.rados.rados.0.smithi045.stdout:2306: done (0 left) 2024-04-26T00:33:26.555 INFO:tasks.rados.rados.0.smithi045.stdout:2307: rollback oid 40 current snap is 259 2024-04-26T00:33:26.555 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 40 to 195 2024-04-26T00:33:26.555 INFO:tasks.rados.rados.0.smithi045.stdout:2308: snap_create 2024-04-26T00:33:26.561 INFO:tasks.rados.rados.0.smithi045.stdout:2307: finishing rollback tid 0 to smithi04526579-40 2024-04-26T00:33:26.562 INFO:tasks.rados.rados.0.smithi045.stdout:2307: finishing rollback tid 1 to smithi04526579-40 2024-04-26T00:33:26.562 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 1463 (ObjNum 33 snap 0 seq_num 2214955312) dirty dne 2024-04-26T00:33:27.514 INFO:tasks.rados.rados.0.smithi045.stdout:2307: done (1 left) 2024-04-26T00:33:27.514 INFO:tasks.rados.rados.0.smithi045.stdout:2308: done (0 left) 2024-04-26T00:33:27.514 INFO:tasks.rados.rados.0.smithi045.stdout:2309: write oid 41 current snap is 260 2024-04-26T00:33:27.514 INFO:tasks.rados.rados.0.smithi045.stdout:2309: seq_num 835 ranges {433845=715392,1662582=371263} 2024-04-26T00:33:27.528 INFO:tasks.rados.rados.0.smithi045.stdout:2309: writing smithi04526579-41 from 433845 to 1149237 tid 1 2024-04-26T00:33:27.540 INFO:tasks.rados.rados.0.smithi045.stdout:2309: writing smithi04526579-41 from 1662582 to 2033845 tid 2 2024-04-26T00:33:27.540 INFO:tasks.rados.rados.0.smithi045.stdout:2310: write oid 5 current snap is 260 2024-04-26T00:33:27.540 INFO:tasks.rados.rados.0.smithi045.stdout:2310: seq_num 836 ranges {450652=790641,1721064=329588} 2024-04-26T00:33:27.556 INFO:tasks.rados.rados.0.smithi045.stdout:2310: writing smithi04526579-5 from 450652 to 1241293 tid 1 2024-04-26T00:33:27.567 INFO:tasks.rados.rados.0.smithi045.stdout:2310: writing smithi04526579-5 from 1721064 to 2050652 tid 2 2024-04-26T00:33:27.567 INFO:tasks.rados.rados.0.smithi045.stdout:2311: read oid 31 snap -1 2024-04-26T00:33:27.567 INFO:tasks.rados.rados.0.smithi045.stdout:2311: expect (ObjNum 807 snap 249 seq_num 807) 2024-04-26T00:33:27.567 INFO:tasks.rados.rados.0.smithi045.stdout:2312: rollback oid 44 current snap is 260 2024-04-26T00:33:27.567 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 44 to 237 2024-04-26T00:33:27.567 INFO:tasks.rados.rados.0.smithi045.stdout:2309: finishing write tid 1 to smithi04526579-41 2024-04-26T00:33:27.567 INFO:tasks.rados.rados.0.smithi045.stdout:2309: finishing write tid 2 to smithi04526579-41 2024-04-26T00:33:27.567 INFO:tasks.rados.rados.0.smithi045.stdout:2309: finishing write tid 3 to smithi04526579-41 2024-04-26T00:33:27.567 INFO:tasks.rados.rados.0.smithi045.stdout:2309: finishing write tid 4 to smithi04526579-41 2024-04-26T00:33:27.567 INFO:tasks.rados.rados.0.smithi045.stdout:2309: oid 41 updating version 0 to 1914 2024-04-26T00:33:27.567 INFO:tasks.rados.rados.0.smithi045.stdout:2309: oid 41 version 1914 is already newer than 1913 2024-04-26T00:33:27.568 INFO:tasks.rados.rados.0.smithi045.stdout:2309: oid 41 version 1914 is already newer than 1912 2024-04-26T00:33:27.568 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 1914 (ObjNum 835 snap 260 seq_num 835) dirty exists 2024-04-26T00:33:27.568 INFO:tasks.rados.rados.0.smithi045.stdout:2309: left oid 41 (ObjNum 835 snap 260 seq_num 835) 2024-04-26T00:33:27.568 INFO:tasks.rados.rados.0.smithi045.stdout:2309: done (3 left) 2024-04-26T00:33:27.568 INFO:tasks.rados.rados.0.smithi045.stdout:2313: delete oid 23 current snap is 260 2024-04-26T00:33:27.577 INFO:tasks.rados.rados.0.smithi045.stdout:2313: done (3 left) 2024-04-26T00:33:27.577 INFO:tasks.rados.rados.0.smithi045.stdout:2314: delete oid 8 current snap is 260 2024-04-26T00:33:27.583 INFO:tasks.rados.rados.0.smithi045.stdout:2312: finishing rollback tid 0 to smithi04526579-44 2024-04-26T00:33:27.584 INFO:tasks.rados.rados.0.smithi045.stdout:2312: finishing rollback tid 1 to smithi04526579-44 2024-04-26T00:33:27.584 INFO:tasks.rados.rados.0.smithi045.stdout:2312: finishing rollback tid 2 to smithi04526579-44 2024-04-26T00:33:27.584 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 1305 (ObjNum 752 snap 232 seq_num 752) dirty exists 2024-04-26T00:33:27.586 INFO:tasks.rados.rados.0.smithi045.stdout:2311: expect (ObjNum 807 snap 249 seq_num 807) 2024-04-26T00:33:27.751 INFO:tasks.rados.rados.0.smithi045.stdout:2310: finishing write tid 1 to smithi04526579-5 2024-04-26T00:33:27.752 INFO:tasks.rados.rados.0.smithi045.stdout:2310: finishing write tid 2 to smithi04526579-5 2024-04-26T00:33:27.752 INFO:tasks.rados.rados.0.smithi045.stdout:2310: finishing write tid 3 to smithi04526579-5 2024-04-26T00:33:27.752 INFO:tasks.rados.rados.0.smithi045.stdout:2310: finishing write tid 4 to smithi04526579-5 2024-04-26T00:33:27.752 INFO:tasks.rados.rados.0.smithi045.stdout:2310: oid 5 updating version 0 to 1977 2024-04-26T00:33:27.752 INFO:tasks.rados.rados.0.smithi045.stdout:2310: oid 5 version 1977 is already newer than 1975 2024-04-26T00:33:27.752 INFO:tasks.rados.rados.0.smithi045.stdout:2310: oid 5 version 1977 is already newer than 1976 2024-04-26T00:33:27.752 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 1977 (ObjNum 836 snap 260 seq_num 836) dirty exists 2024-04-26T00:33:27.752 INFO:tasks.rados.rados.0.smithi045.stdout:2310: left oid 5 (ObjNum 836 snap 260 seq_num 836) 2024-04-26T00:33:27.752 INFO:tasks.rados.rados.0.smithi045.stdout:2310: done (3 left) 2024-04-26T00:33:27.752 INFO:tasks.rados.rados.0.smithi045.stdout:2311: done (2 left) 2024-04-26T00:33:27.752 INFO:tasks.rados.rados.0.smithi045.stdout:2312: done (1 left) 2024-04-26T00:33:27.753 INFO:tasks.rados.rados.0.smithi045.stdout:2314: done (0 left) 2024-04-26T00:33:27.753 INFO:tasks.rados.rados.0.smithi045.stdout:2315: write oid 19 current snap is 260 2024-04-26T00:33:27.753 INFO:tasks.rados.rados.0.smithi045.stdout:2315: seq_num 839 ranges {501073=732741,1896510=204563} 2024-04-26T00:33:27.770 INFO:tasks.rados.rados.0.smithi045.stdout:2315: writing smithi04526579-19 from 501073 to 1233814 tid 1 2024-04-26T00:33:27.783 INFO:tasks.rados.rados.0.smithi045.stdout:2315: writing smithi04526579-19 from 1896510 to 2101073 tid 2 2024-04-26T00:33:27.783 INFO:tasks.rados.rados.0.smithi045.stdout:2316: delete oid 6 current snap is 260 2024-04-26T00:33:27.784 INFO:tasks.rados.rados.0.smithi045.stdout:2315: finishing write tid 1 to smithi04526579-19 2024-04-26T00:33:27.790 INFO:tasks.rados.rados.0.smithi045.stdout:2316: done (1 left) 2024-04-26T00:33:27.791 INFO:tasks.rados.rados.0.smithi045.stdout:2317: write (excl) oid 35 current snap is 260 2024-04-26T00:33:27.791 INFO:tasks.rados.rados.0.smithi045.stdout:2317: seq_num 841 ranges {534687=483239,1613474=412283} 2024-04-26T00:33:27.807 INFO:tasks.rados.rados.0.smithi045.stdout:2317: writing smithi04526579-35 from 534687 to 1017926 tid 1 2024-04-26T00:33:27.822 INFO:tasks.rados.rados.0.smithi045.stdout:2317: writing smithi04526579-35 from 1613474 to 2025757 tid 2 2024-04-26T00:33:27.822 INFO:tasks.rados.rados.0.smithi045.stdout:2318: write oid 16 current snap is 260 2024-04-26T00:33:27.822 INFO:tasks.rados.rados.0.smithi045.stdout:2318: seq_num 842 ranges {551494=558488,1671956=479538} 2024-04-26T00:33:27.835 INFO:tasks.rados.rados.0.smithi045.stdout:2318: writing smithi04526579-16 from 551494 to 1109982 tid 1 2024-04-26T00:33:27.847 INFO:tasks.rados.rados.0.smithi045.stdout:2318: writing smithi04526579-16 from 1671956 to 2151494 tid 2 2024-04-26T00:33:27.847 INFO:tasks.rados.rados.0.smithi045.stdout:2319: copy_from oid 4 from oid 6 current snap is 260 2024-04-26T00:33:27.847 INFO:tasks.rados.rados.0.smithi045.stdout:2315: finishing write tid 2 to smithi04526579-19 2024-04-26T00:33:27.847 INFO:tasks.rados.rados.0.smithi045.stdout:2315: finishing write tid 3 to smithi04526579-19 2024-04-26T00:33:27.847 INFO:tasks.rados.rados.0.smithi045.stdout:2315: finishing write tid 4 to smithi04526579-19 2024-04-26T00:33:27.847 INFO:tasks.rados.rados.0.smithi045.stdout:2315: oid 19 updating version 0 to 1577 2024-04-26T00:33:27.847 INFO:tasks.rados.rados.0.smithi045.stdout:2315: oid 19 version 1577 is already newer than 1576 2024-04-26T00:33:27.847 INFO:tasks.rados.rados.0.smithi045.stdout:2315: oid 19 version 1577 is already newer than 1575 2024-04-26T00:33:27.847 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 1577 (ObjNum 839 snap 260 seq_num 839) dirty exists 2024-04-26T00:33:27.847 INFO:tasks.rados.rados.0.smithi045.stdout:2315: left oid 19 (ObjNum 839 snap 260 seq_num 839) 2024-04-26T00:33:27.847 INFO:tasks.rados.rados.0.smithi045.stdout:2317: finishing write tid 1 to smithi04526579-35 2024-04-26T00:33:27.847 INFO:tasks.rados.rados.0.smithi045.stdout:2317: finishing write tid 2 to smithi04526579-35 2024-04-26T00:33:27.848 INFO:tasks.rados.rados.0.smithi045.stdout:2317: finishing write tid 3 to smithi04526579-35 2024-04-26T00:33:27.848 INFO:tasks.rados.rados.0.smithi045.stdout:2317: finishing write tid 4 to smithi04526579-35 2024-04-26T00:33:27.848 INFO:tasks.rados.rados.0.smithi045.stdout:2317: oid 35 updating version 0 to 1579 2024-04-26T00:33:27.848 INFO:tasks.rados.rados.0.smithi045.stdout:2317: oid 35 updating version 1579 to 1581 2024-04-26T00:33:27.848 INFO:tasks.rados.rados.0.smithi045.stdout:2317: oid 35 version 1581 is already newer than 1580 2024-04-26T00:33:27.848 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 1581 (ObjNum 841 snap 260 seq_num 841) dirty exists 2024-04-26T00:33:27.848 INFO:tasks.rados.rados.0.smithi045.stdout:2317: left oid 35 (ObjNum 841 snap 260 seq_num 841) 2024-04-26T00:33:27.848 INFO:tasks.rados.rados.0.smithi045.stdout:2318: finishing write tid 1 to smithi04526579-16 2024-04-26T00:33:27.848 INFO:tasks.rados.rados.0.smithi045.stdout:2315: done (3 left) 2024-04-26T00:33:27.848 INFO:tasks.rados.rados.0.smithi045.stdout:2317: done (2 left) 2024-04-26T00:33:27.848 INFO:tasks.rados.rados.0.smithi045.stdout:2320: snap_create 2024-04-26T00:33:27.854 INFO:tasks.rados.rados.0.smithi045.stdout:2318: finishing write tid 2 to smithi04526579-16 2024-04-26T00:33:27.855 INFO:tasks.rados.rados.0.smithi045.stdout:2319: finishing copy_from to smithi04526579-4 2024-04-26T00:33:27.855 INFO:tasks.rados.rados.0.smithi045.stdout:2319: got expected ENOENT (src dne) 2024-04-26T00:33:27.856 INFO:tasks.rados.rados.0.smithi045.stdout:2319: finishing copy_from racing read to smithi04526579-4 2024-04-26T00:33:27.856 INFO:tasks.rados.rados.0.smithi045.stdout:2318: finishing write tid 3 to smithi04526579-16 2024-04-26T00:33:27.857 INFO:tasks.rados.rados.0.smithi045.stdout:2318: finishing write tid 4 to smithi04526579-16 2024-04-26T00:33:27.857 INFO:tasks.rados.rados.0.smithi045.stdout:2318: oid 16 updating version 0 to 1144 2024-04-26T00:33:27.857 INFO:tasks.rados.rados.0.smithi045.stdout:2318: oid 16 updating version 1144 to 1145 2024-04-26T00:33:27.857 INFO:tasks.rados.rados.0.smithi045.stdout:2318: oid 16 updating version 1145 to 1146 2024-04-26T00:33:27.857 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 1146 (ObjNum 842 snap 260 seq_num 842) dirty exists 2024-04-26T00:33:27.857 INFO:tasks.rados.rados.0.smithi045.stdout:2318: left oid 16 (ObjNum 842 snap 260 seq_num 842) 2024-04-26T00:33:28.493 INFO:tasks.rados.rados.0.smithi045.stdout:2318: done (2 left) 2024-04-26T00:33:28.494 INFO:tasks.rados.rados.0.smithi045.stdout:2319: done (1 left) 2024-04-26T00:33:28.494 INFO:tasks.rados.rados.0.smithi045.stdout:2320: done (0 left) 2024-04-26T00:33:28.494 INFO:tasks.rados.rados.0.smithi045.stdout:2321: read oid 11 snap -1 2024-04-26T00:33:28.494 INFO:tasks.rados.rados.0.smithi045.stdout:2321: expect deleted 2024-04-26T00:33:28.494 INFO:tasks.rados.rados.0.smithi045.stdout:2322: read oid 49 snap -1 2024-04-26T00:33:28.494 INFO:tasks.rados.rados.0.smithi045.stdout:2322: expect (ObjNum 724 snap 222 seq_num 724) 2024-04-26T00:33:28.494 INFO:tasks.rados.rados.0.smithi045.stdout:2323: snap_create 2024-04-26T00:33:28.508 INFO:tasks.rados.rados.0.smithi045.stdout:2322: expect (ObjNum 724 snap 222 seq_num 724) 2024-04-26T00:33:29.506 INFO:tasks.rados.rados.0.smithi045.stdout:2321: done (2 left) 2024-04-26T00:33:29.506 INFO:tasks.rados.rados.0.smithi045.stdout:2322: done (1 left) 2024-04-26T00:33:29.506 INFO:tasks.rados.rados.0.smithi045.stdout:2323: done (0 left) 2024-04-26T00:33:29.506 INFO:tasks.rados.rados.0.smithi045.stdout:2324: delete oid 41 current snap is 262 2024-04-26T00:33:29.518 INFO:tasks.rados.rados.0.smithi045.stdout:2324: done (0 left) 2024-04-26T00:33:29.518 INFO:tasks.rados.rados.0.smithi045.stdout:2325: copy_from oid 33 from oid 25 current snap is 262 2024-04-26T00:33:29.518 INFO:tasks.rados.rados.0.smithi045.stdout:2326: read oid 5 snap -1 2024-04-26T00:33:29.518 INFO:tasks.rados.rados.0.smithi045.stdout:2326: expect (ObjNum 836 snap 260 seq_num 836) 2024-04-26T00:33:29.518 INFO:tasks.rados.rados.0.smithi045.stdout:2327: snap_remove snap 254 2024-04-26T00:33:31.322 INFO:tasks.rados.rados.0.smithi045.stdout:2327: done (2 left) 2024-04-26T00:33:31.322 INFO:tasks.rados.rados.0.smithi045.stdout:2328: delete oid 47 current snap is 262 2024-04-26T00:33:31.322 INFO:tasks.rados.rados.0.smithi045.stdout:2326: expect (ObjNum 836 snap 260 seq_num 836) 2024-04-26T00:33:31.455 INFO:tasks.rados.rados.0.smithi045.stdout:2325: finishing copy_from racing read to smithi04526579-33 2024-04-26T00:33:31.455 INFO:tasks.rados.rados.0.smithi045.stdout:2325: finishing copy_from to smithi04526579-33 2024-04-26T00:33:31.455 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 1911 (ObjNum 604 snap 186 seq_num 604) dirty exists 2024-04-26T00:33:31.455 INFO:tasks.rados.rados.0.smithi045.stdout:2325: done (2 left) 2024-04-26T00:33:31.455 INFO:tasks.rados.rados.0.smithi045.stdout:2326: done (1 left) 2024-04-26T00:33:31.455 INFO:tasks.rados.rados.0.smithi045.stdout:2328: done (0 left) 2024-04-26T00:33:31.455 INFO:tasks.rados.rados.0.smithi045.stdout:2329: delete oid 47 current snap is 262 2024-04-26T00:33:31.459 INFO:tasks.rados.rados.0.smithi045.stdout:2329: done (0 left) 2024-04-26T00:33:31.460 INFO:tasks.rados.rados.0.smithi045.stdout:2330: copy_from oid 24 from oid 41 current snap is 262 2024-04-26T00:33:31.460 INFO:tasks.rados.rados.0.smithi045.stdout:2331: delete oid 50 current snap is 262 2024-04-26T00:33:31.461 INFO:tasks.rados.rados.0.smithi045.stdout:2330: finishing copy_from to smithi04526579-24 2024-04-26T00:33:31.461 INFO:tasks.rados.rados.0.smithi045.stdout:2330: got expected ENOENT (src dne) 2024-04-26T00:33:31.462 INFO:tasks.rados.rados.0.smithi045.stdout:2330: finishing copy_from racing read to smithi04526579-24 2024-04-26T00:33:31.464 INFO:tasks.rados.rados.0.smithi045.stdout:2330: done (1 left) 2024-04-26T00:33:31.464 INFO:tasks.rados.rados.0.smithi045.stdout:2331: done (0 left) 2024-04-26T00:33:31.464 INFO:tasks.rados.rados.0.smithi045.stdout:2332: rollback oid 23 current snap is 262 2024-04-26T00:33:31.464 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 23 to 231 2024-04-26T00:33:31.464 INFO:tasks.rados.rados.0.smithi045.stdout:2333: write (excl) oid 35 current snap is 262 2024-04-26T00:33:31.464 INFO:tasks.rados.rados.0.smithi045.stdout:2333: seq_num 847 ranges {635529=651086,1848013=387516} 2024-04-26T00:33:31.485 INFO:tasks.rados.rados.0.smithi045.stdout:2333: writing smithi04526579-35 from 635529 to 1286615 tid 1 2024-04-26T00:33:31.501 INFO:tasks.rados.rados.0.smithi045.stdout:2333: writing smithi04526579-35 from 1848013 to 2235529 tid 2 2024-04-26T00:33:31.502 INFO:tasks.rados.rados.0.smithi045.stdout:2334: read oid 20 snap -1 2024-04-26T00:33:31.502 INFO:tasks.rados.rados.0.smithi045.stdout:2334: expect (ObjNum 819 snap 256 seq_num 819) 2024-04-26T00:33:31.502 INFO:tasks.rados.rados.0.smithi045.stdout:2335: delete oid 1 current snap is 262 2024-04-26T00:33:31.502 INFO:tasks.rados.rados.0.smithi045.stdout:2332: finishing rollback tid 1 to smithi04526579-23 2024-04-26T00:33:31.502 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 1464 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-04-26T00:33:31.502 INFO:tasks.rados.rados.0.smithi045.stdout:2333: finishing write tid 1 to smithi04526579-35 2024-04-26T00:33:31.510 INFO:tasks.rados.rados.0.smithi045.stdout:2332: done (3 left) 2024-04-26T00:33:31.510 INFO:tasks.rados.rados.0.smithi045.stdout:2335: done (2 left) 2024-04-26T00:33:31.510 INFO:tasks.rados.rados.0.smithi045.stdout:2336: write oid 27 current snap is 262 2024-04-26T00:33:31.510 INFO:tasks.rados.rados.0.smithi045.stdout:2336: seq_num 849 ranges {669143=401584,1564977=426959} 2024-04-26T00:33:31.528 INFO:tasks.rados.rados.0.smithi045.stdout:2336: writing smithi04526579-27 from 669143 to 1070727 tid 1 2024-04-26T00:33:31.543 INFO:tasks.rados.rados.0.smithi045.stdout:2336: writing smithi04526579-27 from 1564977 to 1991936 tid 2 2024-04-26T00:33:31.543 INFO:tasks.rados.rados.0.smithi045.stdout:2337: delete oid 39 current snap is 262 2024-04-26T00:33:31.543 INFO:tasks.rados.rados.0.smithi045.stdout:2333: finishing write tid 2 to smithi04526579-35 2024-04-26T00:33:31.543 INFO:tasks.rados.rados.0.smithi045.stdout:2334: expect (ObjNum 819 snap 256 seq_num 819) 2024-04-26T00:33:31.767 INFO:tasks.rados.rados.0.smithi045.stdout:2333: finishing write tid 3 to smithi04526579-35 2024-04-26T00:33:31.767 INFO:tasks.rados.rados.0.smithi045.stdout:2333: finishing write tid 4 to smithi04526579-35 2024-04-26T00:33:31.768 INFO:tasks.rados.rados.0.smithi045.stdout:2333: oid 35 updating version 0 to 1593 2024-04-26T00:33:31.768 INFO:tasks.rados.rados.0.smithi045.stdout:2333: oid 35 version 1593 is already newer than 1591 2024-04-26T00:33:31.768 INFO:tasks.rados.rados.0.smithi045.stdout:2333: oid 35 version 1593 is already newer than 1592 2024-04-26T00:33:31.768 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 1593 (ObjNum 847 snap 262 seq_num 847) dirty exists 2024-04-26T00:33:31.768 INFO:tasks.rados.rados.0.smithi045.stdout:2333: left oid 35 (ObjNum 847 snap 262 seq_num 847) 2024-04-26T00:33:31.768 INFO:tasks.rados.rados.0.smithi045.stdout:2336: finishing write tid 1 to smithi04526579-27 2024-04-26T00:33:31.768 INFO:tasks.rados.rados.0.smithi045.stdout:2336: finishing write tid 2 to smithi04526579-27 2024-04-26T00:33:31.768 INFO:tasks.rados.rados.0.smithi045.stdout:2336: finishing write tid 3 to smithi04526579-27 2024-04-26T00:33:31.768 INFO:tasks.rados.rados.0.smithi045.stdout:2336: finishing write tid 4 to smithi04526579-27 2024-04-26T00:33:31.768 INFO:tasks.rados.rados.0.smithi045.stdout:2336: oid 27 updating version 0 to 1596 2024-04-26T00:33:31.768 INFO:tasks.rados.rados.0.smithi045.stdout:2336: oid 27 version 1596 is already newer than 1595 2024-04-26T00:33:31.768 INFO:tasks.rados.rados.0.smithi045.stdout:2336: oid 27 updating version 1596 to 1597 2024-04-26T00:33:31.768 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 1597 (ObjNum 849 snap 262 seq_num 849) dirty exists 2024-04-26T00:33:31.769 INFO:tasks.rados.rados.0.smithi045.stdout:2336: left oid 27 (ObjNum 849 snap 262 seq_num 849) 2024-04-26T00:33:31.769 INFO:tasks.rados.rados.0.smithi045.stdout:2333: done (3 left) 2024-04-26T00:33:31.769 INFO:tasks.rados.rados.0.smithi045.stdout:2334: done (2 left) 2024-04-26T00:33:31.769 INFO:tasks.rados.rados.0.smithi045.stdout:2336: done (1 left) 2024-04-26T00:33:31.769 INFO:tasks.rados.rados.0.smithi045.stdout:2337: done (0 left) 2024-04-26T00:33:31.769 INFO:tasks.rados.rados.0.smithi045.stdout:2338: snap_create 2024-04-26T00:33:32.328 INFO:tasks.rados.rados.0.smithi045.stdout:2338: done (0 left) 2024-04-26T00:33:32.328 INFO:tasks.rados.rados.0.smithi045.stdout:2339: delete oid 20 current snap is 263 2024-04-26T00:33:32.340 INFO:tasks.rados.rados.0.smithi045.stdout:2339: done (0 left) 2024-04-26T00:33:32.340 INFO:tasks.rados.rados.0.smithi045.stdout:2340: delete oid 12 current snap is 263 2024-04-26T00:33:32.352 INFO:tasks.rados.rados.0.smithi045.stdout:2340: done (0 left) 2024-04-26T00:33:32.352 INFO:tasks.rados.rados.0.smithi045.stdout:2341: write (excl) oid 1 current snap is 263 2024-04-26T00:33:32.352 INFO:tasks.rados.rados.0.smithi045.stdout:2341: seq_num 853 ranges {736371=418933,1798905=434297} 2024-04-26T00:33:32.370 INFO:tasks.rados.rados.0.smithi045.stdout:2341: writing smithi04526579-1 from 736371 to 1155304 tid 1 2024-04-26T00:33:32.388 INFO:tasks.rados.rados.0.smithi045.stdout:2341: writing smithi04526579-1 from 1798905 to 2233202 tid 2 2024-04-26T00:33:32.388 INFO:tasks.rados.rados.0.smithi045.stdout:2342: delete oid 33 current snap is 263 2024-04-26T00:33:32.388 INFO:tasks.rados.rados.0.smithi045.stdout:2341: finishing write tid 1 to smithi04526579-1 2024-04-26T00:33:32.396 INFO:tasks.rados.rados.0.smithi045.stdout:2341: finishing write tid 2 to smithi04526579-1 2024-04-26T00:33:32.396 INFO:tasks.rados.rados.0.smithi045.stdout:2341: finishing write tid 3 to smithi04526579-1 2024-04-26T00:33:32.397 INFO:tasks.rados.rados.0.smithi045.stdout:2341: finishing write tid 4 to smithi04526579-1 2024-04-26T00:33:32.397 INFO:tasks.rados.rados.0.smithi045.stdout:2341: oid 1 updating version 0 to 1917 2024-04-26T00:33:32.397 INFO:tasks.rados.rados.0.smithi045.stdout:2341: oid 1 updating version 1917 to 1919 2024-04-26T00:33:32.397 INFO:tasks.rados.rados.0.smithi045.stdout:2341: oid 1 version 1919 is already newer than 1918 2024-04-26T00:33:32.397 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 1919 (ObjNum 853 snap 263 seq_num 853) dirty exists 2024-04-26T00:33:32.397 INFO:tasks.rados.rados.0.smithi045.stdout:2341: left oid 1 (ObjNum 853 snap 263 seq_num 853) 2024-04-26T00:33:32.398 INFO:tasks.rados.rados.0.smithi045.stdout:2341: done (1 left) 2024-04-26T00:33:32.398 INFO:tasks.rados.rados.0.smithi045.stdout:2342: done (0 left) 2024-04-26T00:33:32.398 INFO:tasks.rados.rados.0.smithi045.stdout:2343: read oid 43 snap -1 2024-04-26T00:33:32.398 INFO:tasks.rados.rados.0.smithi045.stdout:2343: expect deleted 2024-04-26T00:33:32.398 INFO:tasks.rados.rados.0.smithi045.stdout:2344: delete oid 48 current snap is 263 2024-04-26T00:33:32.404 INFO:tasks.rados.rados.0.smithi045.stdout:2343: done (1 left) 2024-04-26T00:33:32.405 INFO:tasks.rados.rados.0.smithi045.stdout:2344: done (0 left) 2024-04-26T00:33:32.405 INFO:tasks.rados.rados.0.smithi045.stdout:2345: snap_create 2024-04-26T00:33:32.407 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-04-26T00:33:32.407 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-04-26T00:33:32.407 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-26T00:33:32.408 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-26T00:33:32.410 INFO:tasks.ceph.mon.b:Started 2024-04-26T00:33:32.410 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:33:32.411 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:33:32.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7bf6289640 10 monclient: get_monmap_and_config 2024-04-26T00:33:32.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7bf6289640 10 monclient: build_initial_monmap 2024-04-26T00:33:32.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7bf6289640 10 monclient: monmap: 2024-04-26T00:33:32.541 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:33:32.541 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:32.541 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:33:32.549309+0000 2024-04-26T00:33:32.541 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:33:32.549309+0000 2024-04-26T00:33:32.541 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:33:32.541 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:32.541 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:33:32.541 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:33:32.541 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:33:32.541 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:33:32.541 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:33:32.541 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:32.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7bf6289640 1 Processor -- start 2024-04-26T00:33:32.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7bf6289640 1 -- start start 2024-04-26T00:33:32.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7bf6289640 10 monclient: init 2024-04-26T00:33:32.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7bf6289640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:32.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7bf6289640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-04-26T00:33:32.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7bf6289640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7bf00a0260 0x7f7bf00a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:32.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7bf6289640 10 monclient(hunting): picked mon.noname-c con 0x7f7bf00a0260 addr v2:172.21.15.45:3300/0 2024-04-26T00:33:32.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7bf6289640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bf0134b30 0x7f7bf0134f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:32.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7bf6289640 10 monclient(hunting): picked mon.noname-b con 0x7f7bf0134b30 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:33:32.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7bf6289640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7bf0135490 addr v1:172.21.15.37:6789/0 2024-04-26T00:33:32.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7bf6289640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7bf00a1130 con 0x7f7bf0135490 2024-04-26T00:33:32.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7bf6289640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:32.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7bf6289640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f7bf009ec10 con 0x7f7bf00a0260 2024-04-26T00:33:32.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7bf6289640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:32.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7bf6289640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f7bf009f0f0 con 0x7f7bf0134b30 2024-04-26T00:33:32.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7bf6289640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:32.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7bf6289640 10 monclient(hunting): authenticate will time out at 1793.855038s 2024-04-26T00:33:32.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7bef7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bf0134b30 0x7f7bf0134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:32.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7bf4826640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f7bf0135490 0x7f7bf013be60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:35114/0 (socket says 172.21.15.37:35114) 2024-04-26T00:33:32.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7bf4826640 1 -- 172.21.15.37:0/8830496 learned_addr learned my addr 172.21.15.37:0/8830496 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:33:32.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7bef7fe640 10 monclient(hunting): get_auth_request con 0x7f7bf0134b30 auth_method 0 2024-04-26T00:33:32.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7bef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:32.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7bef7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:32.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7bef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:32.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7beffff640 1 --2- 172.21.15.37:0/8830496 >> v2:172.21.15.45:3300/0 conn(0x7f7bf00a0260 0x7f7bf00a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:32.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7bef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:32.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7bef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:32.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7bef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:32.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7beffff640 10 monclient(hunting): get_auth_request con 0x7f7bf00a0260 auth_method 0 2024-04-26T00:33:32.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7beffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:32.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7beffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:32.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7beffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:32.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7beffff640 1 -- 172.21.15.37:0/8830496 >> v2:172.21.15.45:3300/0 conn(0x7f7bf00a0260 msgr2=0x7f7bf00a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:33:32.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7beffff640 1 -- 172.21.15.37:0/8830496 >> v2:172.21.15.45:3300/0 conn(0x7f7bf00a0260 msgr2=0x7f7bf00a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:33:32.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7beffff640 1 --2- 172.21.15.37:0/8830496 >> v2:172.21.15.45:3300/0 conn(0x7f7bf00a0260 0x7f7bf00a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:33:32.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7bef7fe640 10 monclient(hunting): handle_auth_done global_id 154163 payload 274 2024-04-26T00:33:32.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7beffff640 1 --2- 172.21.15.37:0/8830496 >> v2:172.21.15.45:3300/0 conn(0x7f7bf00a0260 0x7f7bf00a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:33:32.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7bef7fe640 1 -- 172.21.15.37:0/8830496 >> v2:172.21.15.45:3300/0 conn(0x7f7bf00a0260 msgr2=0x7f7bf00a0640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:33:32.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7bef7fe640 1 --2- 172.21.15.37:0/8830496 >> v2:172.21.15.45:3300/0 conn(0x7f7bf00a0260 0x7f7bf00a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:32.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7bef7fe640 1 -- 172.21.15.37:0/8830496 >> v1:172.21.15.37:6789/0 conn(0x7f7bf0135490 legacy=0x7f7bf013be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:33:32.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7bef7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:33:32.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7bef7fe640 1 monclient: found mon.noname-b 2024-04-26T00:33:32.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7bef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:32.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7bef7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:33:32.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.547+0000 7f7bef7fe640 1 -- 172.21.15.37:0/8830496 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bf006cff0 con 0x7f7bf0134b30 2024-04-26T00:33:32.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7bef7fe640 1 -- 172.21.15.37:0/8830496 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bf0134b30 msgr2=0x7f7bf0134f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:32.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7bef7fe640 1 --2- 172.21.15.37:0/8830496 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bf0134b30 0x7f7bf0134f50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8009d90 tx=0x7f7bd801c610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98d061e7ce586d5b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:32.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7beeffd640 1 -- 172.21.15.37:0/8830496 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd80a6040 con 0x7f7bf0134b30 2024-04-26T00:33:32.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7beeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:32.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7beeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:33:32.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7beeffd640 10 monclient: dump: 2024-04-26T00:33:32.549 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:32.549 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:32.549 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:32.549 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:32.549 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:32.549 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:32.549 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:32.550 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:32.550 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:32.550 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:32.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7beeffd640 10 monclient: _finish_auth 0 2024-04-26T00:33:32.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7beeffd640 10 monclient: _check_auth_tickets 2024-04-26T00:33:32.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7beeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:32.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7beeffd640 1 -- 172.21.15.37:0/8830496 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bd80af070 con 0x7f7bf0134b30 2024-04-26T00:33:32.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7beeffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:33:32.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7beeffd640 1 -- 172.21.15.37:0/8830496 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd80aa860 con 0x7f7bf0134b30 2024-04-26T00:33:32.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7beeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:32.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7beeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:33:32.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7beeffd640 10 monclient: dump: 2024-04-26T00:33:32.550 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:32.550 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:32.551 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:32.551 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:32.551 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:32.551 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:32.551 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:32.551 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:32.551 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:32.551 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:32.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7bf6289640 5 monclient: authenticate success, global_id 154163 2024-04-26T00:33:32.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7bf6289640 10 monclient: get_monmap_and_config success 2024-04-26T00:33:32.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7bf6289640 4 set_mon_vals no callback set 2024-04-26T00:33:32.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7bf6289640 10 monclient: shutdown 2024-04-26T00:33:32.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7bf6289640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:33:32.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7bf6289640 1 -- 172.21.15.37:0/8830496 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bf0134b30 msgr2=0x7f7bf0134f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:32.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7bf6289640 1 --2- 172.21.15.37:0/8830496 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bf0134b30 0x7f7bf0134f50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8009d90 tx=0x7f7bd801c610 comp rx=0 tx=0).stop 2024-04-26T00:33:32.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7bf6289640 1 -- 172.21.15.37:0/8830496 shutdown_connections 2024-04-26T00:33:32.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7bf6289640 1 --2- 172.21.15.37:0/8830496 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bf0134b30 0x7f7bf0134f50 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:32.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7bf6289640 1 --2- 172.21.15.37:0/8830496 >> v2:172.21.15.45:3300/0 conn(0x7f7bf00a0260 0x7f7bf00a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:32.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7bf6289640 1 -- 172.21.15.37:0/8830496 >> 172.21.15.37:0/8830496 conn(0x7f7bf00a1ad0 msgr2=0x7f7bf00a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:33:32.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7bf6289640 1 -- 172.21.15.37:0/8830496 shutdown_connections 2024-04-26T00:33:32.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7bf6289640 1 -- 172.21.15.37:0/8830496 wait complete. 2024-04-26T00:33:32.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7bf6289640 10 monclient: build_initial_monmap 2024-04-26T00:33:32.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7bf6289640 10 monclient: monmap: 2024-04-26T00:33:32.557 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:33:32.557 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:33:32.557 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:33:32.557 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:33:32.557 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:33:32.557 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:33:32.558 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:33:32.558 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:33:32.558 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:33:32.558 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:32.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7bf6289640 1 Processor -- start 2024-04-26T00:33:32.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7bf6289640 1 -- start start 2024-04-26T00:33:32.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7bf6289640 10 monclient: init 2024-04-26T00:33:32.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7bf6289640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:32.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7bf6289640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:33:32.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7bf6289640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bf00a0260 0x7f7bf0131110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:32.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7bf6289640 10 monclient(hunting): picked mon.noname-c con 0x7f7bf00a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:33:32.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7bf6289640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7bf0134b30 0x7f7bf0131650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:32.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7bf6289640 10 monclient(hunting): picked mon.noname-a con 0x7f7bf0134b30 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:33:32.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7bf6289640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bf0135490 0x7f7bf01398b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:32.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7bf6289640 10 monclient(hunting): picked mon.noname-b con 0x7f7bf0135490 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:33:32.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7bf6289640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:32.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7bf6289640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf009f430 con 0x7f7bf0134b30 2024-04-26T00:33:32.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7bf6289640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:32.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7bf6289640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf009f2b0 con 0x7f7bf0135490 2024-04-26T00:33:32.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7bf6289640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:32.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7bf6289640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f7bf009fe90 con 0x7f7bf00a0260 2024-04-26T00:33:32.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7bf6289640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:32.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.555+0000 7f7bf6289640 10 monclient(hunting): authenticate will time out at 1793.863846s 2024-04-26T00:33:32.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7bef7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7bf0134b30 0x7f7bf0131650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:32.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7beffff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bf00a0260 0x7f7bf0131110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:32.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7bef7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7bf0134b30 0x7f7bf0131650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:37582/0 (socket says 172.21.15.37:37582) 2024-04-26T00:33:32.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7bef7fe640 1 -- 172.21.15.37:0/518810616 learned_addr learned my addr 172.21.15.37:0/518810616 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:33:32.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7bef7fe640 10 monclient(hunting): get_auth_request con 0x7f7bf0134b30 auth_method 0 2024-04-26T00:33:32.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7beffff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bf00a0260 0x7f7bf0131110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:48326/0 (socket says 172.21.15.37:48326) 2024-04-26T00:33:32.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7bef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:32.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7bef7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:32.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7bef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:32.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7beffff640 10 monclient(hunting): get_auth_request con 0x7f7bf00a0260 auth_method 0 2024-04-26T00:33:32.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7beffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:32.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7beffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:32.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7beffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:32.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7bf4826640 1 --2- 172.21.15.37:0/518810616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bf0135490 0x7f7bf01398b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:32.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7bf4826640 10 monclient(hunting): get_auth_request con 0x7f7bf0135490 auth_method 0 2024-04-26T00:33:32.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7bf4826640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:32.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7bf4826640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:32.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7bf4826640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:32.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7beffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:32.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7beffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:32.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7beffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:32.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7bef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:32.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7bef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:32.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7bef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:32.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7bf4826640 1 -- 172.21.15.37:0/518810616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bf0135490 msgr2=0x7f7bf01398b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:33:32.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7bf4826640 1 -- 172.21.15.37:0/518810616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bf0135490 msgr2=0x7f7bf01398b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:33:32.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7bf4826640 1 --2- 172.21.15.37:0/518810616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bf0135490 0x7f7bf01398b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:33:32.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7bf4826640 1 --2- 172.21.15.37:0/518810616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bf0135490 0x7f7bf01398b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:33:32.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7beffff640 10 monclient(hunting): handle_auth_done global_id 154166 payload 824 2024-04-26T00:33:32.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7beffff640 1 -- 172.21.15.37:0/518810616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bf0135490 msgr2=0x7f7bf01398b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:33:32.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7beffff640 1 --2- 172.21.15.37:0/518810616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bf0135490 0x7f7bf01398b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:32.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7beffff640 1 -- 172.21.15.37:0/518810616 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7bf0134b30 msgr2=0x7f7bf0131650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:32.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7beffff640 1 --2- 172.21.15.37:0/518810616 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7bf0134b30 0x7f7bf0131650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:32.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7beffff640 10 monclient: _finish_hunting 0 2024-04-26T00:33:32.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7beffff640 1 monclient: found mon.noname-c 2024-04-26T00:33:32.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7beffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:32.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7beffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:33:32.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7beffff640 1 -- 172.21.15.37:0/518810616 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bf011c0d0 con 0x7f7bf00a0260 2024-04-26T00:33:32.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7beffff640 1 -- 172.21.15.37:0/518810616 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bf00a0260 msgr2=0x7f7bf0131110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:32.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7beffff640 1 --2- 172.21.15.37:0/518810616 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bf00a0260 0x7f7bf0131110 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7be800e810 tx=0x7f7be800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:32.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7becff9640 1 -- 172.21.15.37:0/518810616 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be8017030 con 0x7f7bf00a0260 2024-04-26T00:33:32.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7becff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:32.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7becff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:33:32.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7becff9640 10 monclient: dump: 2024-04-26T00:33:32.562 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:32.562 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:32.562 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:32.563 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:32.563 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:32.563 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:32.563 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:32.563 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:32.563 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:32.563 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:32.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7becff9640 10 monclient: _finish_auth 0 2024-04-26T00:33:32.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7becff9640 10 monclient: _check_auth_tickets 2024-04-26T00:33:32.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7becff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:32.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7becff9640 1 -- 172.21.15.37:0/518810616 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7be8004020 con 0x7f7bf00a0260 2024-04-26T00:33:32.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7becff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:33:32.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7becff9640 1 -- 172.21.15.37:0/518810616 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be800f040 con 0x7f7bf00a0260 2024-04-26T00:33:32.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7bf6289640 5 monclient: authenticate success, global_id 154166 2024-04-26T00:33:32.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7bf6289640 10 monclient: _renew_subs 2024-04-26T00:33:32.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7bf6289640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:33:32.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7bf6289640 1 -- 172.21.15.37:0/518810616 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bf01346e0 con 0x7f7bf00a0260 2024-04-26T00:33:32.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7bee7fc640 4 set_mon_vals no callback set 2024-04-26T00:33:32.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7becff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:32.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7becff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:33:32.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7becff9640 10 monclient: dump: 2024-04-26T00:33:32.564 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:32.564 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:32.564 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:32.564 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:32.564 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:32.564 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:32.564 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:32.564 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:32.565 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:32.565 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:32.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7bf6289640 10 monclient: _renew_subs 2024-04-26T00:33:32.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7bf6289640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:33:32.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7bf6289640 1 -- 172.21.15.37:0/518810616 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bf013bdf0 con 0x7f7bf00a0260 2024-04-26T00:33:32.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.559+0000 7f7bf6289640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:32.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.563+0000 7f7bf6289640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:32.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.563+0000 7f7bf6289640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:33:32.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.563+0000 7f7becff9640 1 -- 172.21.15.37:0/518810616 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f7be801a070 con 0x7f7bf00a0260 2024-04-26T00:33:32.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.563+0000 7f7bf6289640 1 -- 172.21.15.37:0/518810616 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bf0066f40 con 0x7f7bf00a0260 2024-04-26T00:33:32.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.563+0000 7f7becff9640 1 --2- 172.21.15.37:0/518810616 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7bd0035e80 0x7f7bd0038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:32.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.563+0000 7f7becff9640 1 -- 172.21.15.37:0/518810616 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(526..526 src has 370..526) ==== 3931+0+0 (secure 0 0 0) 0x7f7be80483e0 con 0x7f7bf00a0260 2024-04-26T00:33:32.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.563+0000 7f7bef7fe640 1 --2- 172.21.15.37:0/518810616 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7bd0035e80 0x7f7bd0038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:32.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.563+0000 7f7bef7fe640 10 monclient: get_auth_request con 0x7f7bd0035e80 auth_method 0 2024-04-26T00:33:32.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.563+0000 7f7bef7fe640 1 --2- 172.21.15.37:0/518810616 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7bd0035e80 0x7f7bd0038340 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8000f90 tx=0x7f7bd80b5040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:32.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.567+0000 7f7becff9640 1 -- 172.21.15.37:0/518810616 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7be8047130 con 0x7f7bf00a0260 2024-04-26T00:33:32.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.567+0000 7f7becff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:32.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.567+0000 7f7becff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:33:32.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.791+0000 7f7bf6289640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:33:32.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.791+0000 7f7bf6289640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:33:32.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.791+0000 7f7bf6289640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:33:32.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.791+0000 7f7bf6289640 1 -- 172.21.15.37:0/518810616 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7bf0131b90 con 0x7f7bf00a0260 2024-04-26T00:33:32.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.791+0000 7f7becff9640 1 -- 172.21.15.37:0/518810616 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7f7be800c780 con 0x7f7bf00a0260 2024-04-26T00:33:32.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.791+0000 7f7becff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:33:32.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.791+0000 7f7becff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:33:32.784 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":100,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":23,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:33:32.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.791+0000 7f7bf6289640 1 -- 172.21.15.37:0/518810616 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7bd0035e80 msgr2=0x7f7bd0038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:32.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.791+0000 7f7bf6289640 1 --2- 172.21.15.37:0/518810616 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7bd0035e80 0x7f7bd0038340 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8000f90 tx=0x7f7bd80b5040 comp rx=0 tx=0).stop 2024-04-26T00:33:32.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.791+0000 7f7bf6289640 10 monclient: shutdown 2024-04-26T00:33:32.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.791+0000 7f7bf6289640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:33:32.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.791+0000 7f7bf6289640 1 -- 172.21.15.37:0/518810616 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bf00a0260 msgr2=0x7f7bf0131110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:32.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.791+0000 7f7bf6289640 1 --2- 172.21.15.37:0/518810616 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bf00a0260 0x7f7bf0131110 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7be800e810 tx=0x7f7be800ecd0 comp rx=0 tx=0).stop 2024-04-26T00:33:32.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.791+0000 7f7bf6289640 1 -- 172.21.15.37:0/518810616 shutdown_connections 2024-04-26T00:33:32.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.791+0000 7f7bf6289640 1 --2- 172.21.15.37:0/518810616 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7bd0035e80 0x7f7bd0038340 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:32.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.791+0000 7f7bf6289640 1 --2- 172.21.15.37:0/518810616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bf0135490 0x7f7bf01398b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:32.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.791+0000 7f7bf6289640 1 --2- 172.21.15.37:0/518810616 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7bf0134b30 0x7f7bf0131650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:32.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.791+0000 7f7bf6289640 1 --2- 172.21.15.37:0/518810616 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7bf00a0260 0x7f7bf0131110 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:32.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.791+0000 7f7bf6289640 1 -- 172.21.15.37:0/518810616 >> 172.21.15.37:0/518810616 conn(0x7f7bf00a1ad0 msgr2=0x7f7bf00a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:33:32.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.791+0000 7f7bf6289640 1 -- 172.21.15.37:0/518810616 shutdown_connections 2024-04-26T00:33:32.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:32.791+0000 7f7bf6289640 1 -- 172.21.15.37:0/518810616 wait complete. 2024-04-26T00:33:33.331 INFO:tasks.rados.rados.0.smithi045.stdout:2345: done (0 left) 2024-04-26T00:33:33.331 INFO:tasks.rados.rados.0.smithi045.stdout:2346: copy_from oid 13 from oid 26 current snap is 264 2024-04-26T00:33:33.331 INFO:tasks.rados.rados.0.smithi045.stdout:2347: write (excl) oid 7 current snap is 264 2024-04-26T00:33:33.331 INFO:tasks.rados.rados.0.smithi045.stdout:2347: seq_num 856 ranges {786792=644680,1857998=528794} 2024-04-26T00:33:33.355 INFO:tasks.rados.rados.0.smithi045.stdout:2347: writing smithi04526579-7 from 786792 to 1431472 tid 1 2024-04-26T00:33:33.371 INFO:tasks.rados.rados.0.smithi045.stdout:2347: writing smithi04526579-7 from 1857998 to 2386792 tid 2 2024-04-26T00:33:33.371 INFO:tasks.rados.rados.0.smithi045.stdout:2348: write oid 48 current snap is 264 2024-04-26T00:33:33.371 INFO:tasks.rados.rados.0.smithi045.stdout:2347: finishing write tid 1 to smithi04526579-7 2024-04-26T00:33:33.371 INFO:tasks.rados.rados.0.smithi045.stdout:2346: finishing copy_from racing read to smithi04526579-13 2024-04-26T00:33:33.371 INFO:tasks.rados.rados.0.smithi045.stdout:2346: finishing copy_from to smithi04526579-13 2024-04-26T00:33:33.371 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 1304 (ObjNum 731 snap 224 seq_num 731) dirty exists 2024-04-26T00:33:33.371 INFO:tasks.rados.rados.0.smithi045.stdout:2348: seq_num 857 ranges {403599=719929,1916480=441635} 2024-04-26T00:33:33.389 INFO:tasks.rados.rados.0.smithi045.stdout:2348: writing smithi04526579-48 from 403599 to 1123528 tid 1 2024-04-26T00:33:33.408 INFO:tasks.rados.rados.0.smithi045.stdout:2348: writing smithi04526579-48 from 1916480 to 2358115 tid 2 2024-04-26T00:33:33.409 INFO:tasks.rados.rados.0.smithi045.stdout:2346: done (2 left) 2024-04-26T00:33:33.409 INFO:tasks.rados.rados.0.smithi045.stdout:2349: write (excl) oid 38 current snap is 264 2024-04-26T00:33:33.409 INFO:tasks.rados.rados.0.smithi045.stdout:2349: seq_num 858 ranges {420406=795178,1974962=445444} 2024-04-26T00:33:33.423 INFO:tasks.rados.rados.0.smithi045.stdout:2349: writing smithi04526579-38 from 420406 to 1215584 tid 1 2024-04-26T00:33:33.435 INFO:tasks.rados.rados.0.smithi045.stdout:2349: writing smithi04526579-38 from 1974962 to 2420406 tid 2 2024-04-26T00:33:33.435 INFO:tasks.rados.rados.0.smithi045.stdout:2350: read oid 40 snap 260 2024-04-26T00:33:33.435 INFO:tasks.rados.rados.0.smithi045.stdout:2350: expect deleted 2024-04-26T00:33:33.435 INFO:tasks.rados.rados.0.smithi045.stdout:2351: write oid 25 current snap is 264 2024-04-26T00:33:33.435 INFO:tasks.rados.rados.0.smithi045.stdout:2351: seq_num 859 ranges {437213=470427,1633444=728951} 2024-04-26T00:33:33.442 INFO:tasks.rados.rados.0.smithi045.stdout:2351: writing smithi04526579-25 from 437213 to 907640 tid 1 2024-04-26T00:33:33.453 INFO:tasks.rados.rados.0.smithi045.stdout:2351: writing smithi04526579-25 from 1633444 to 2362395 tid 2 2024-04-26T00:33:33.453 INFO:tasks.rados.rados.0.smithi045.stdout:2352: snap_remove snap 250 2024-04-26T00:33:33.793 INFO:tasks.rados.rados.0.smithi045.stdout:2352: done (5 left) 2024-04-26T00:33:33.793 INFO:tasks.rados.rados.0.smithi045.stdout:2353: snap_remove snap 249 2024-04-26T00:33:34.499 INFO:tasks.rados.rados.0.smithi045.stdout:2353: done (5 left) 2024-04-26T00:33:34.499 INFO:tasks.rados.rados.0.smithi045.stdout:2354: snap_create 2024-04-26T00:33:34.499 INFO:tasks.rados.rados.0.smithi045.stdout:2347: finishing write tid 2 to smithi04526579-7 2024-04-26T00:33:34.499 INFO:tasks.rados.rados.0.smithi045.stdout:2347: finishing write tid 3 to smithi04526579-7 2024-04-26T00:33:34.499 INFO:tasks.rados.rados.0.smithi045.stdout:2347: finishing write tid 4 to smithi04526579-7 2024-04-26T00:33:34.499 INFO:tasks.rados.rados.0.smithi045.stdout:2347: oid 7 updating version 0 to 1986 2024-04-26T00:33:34.500 INFO:tasks.rados.rados.0.smithi045.stdout:2347: oid 7 version 1986 is already newer than 1985 2024-04-26T00:33:34.500 INFO:tasks.rados.rados.0.smithi045.stdout:2347: oid 7 updating version 1986 to 1987 2024-04-26T00:33:34.500 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 1987 (ObjNum 856 snap 264 seq_num 856) dirty exists 2024-04-26T00:33:34.500 INFO:tasks.rados.rados.0.smithi045.stdout:2347: left oid 7 (ObjNum 856 snap 264 seq_num 856) 2024-04-26T00:33:34.500 INFO:tasks.rados.rados.0.smithi045.stdout:2348: finishing write tid 1 to smithi04526579-48 2024-04-26T00:33:34.500 INFO:tasks.rados.rados.0.smithi045.stdout:2348: finishing write tid 2 to smithi04526579-48 2024-04-26T00:33:34.500 INFO:tasks.rados.rados.0.smithi045.stdout:2348: finishing write tid 3 to smithi04526579-48 2024-04-26T00:33:34.500 INFO:tasks.rados.rados.0.smithi045.stdout:2348: finishing write tid 4 to smithi04526579-48 2024-04-26T00:33:34.500 INFO:tasks.rados.rados.0.smithi045.stdout:2348: oid 48 updating version 0 to 1403 2024-04-26T00:33:34.500 INFO:tasks.rados.rados.0.smithi045.stdout:2348: oid 48 version 1403 is already newer than 1402 2024-04-26T00:33:34.500 INFO:tasks.rados.rados.0.smithi045.stdout:2348: oid 48 version 1403 is already newer than 1401 2024-04-26T00:33:34.500 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 1403 (ObjNum 857 snap 264 seq_num 857) dirty exists 2024-04-26T00:33:34.500 INFO:tasks.rados.rados.0.smithi045.stdout:2348: left oid 48 (ObjNum 857 snap 264 seq_num 857) 2024-04-26T00:33:34.500 INFO:tasks.rados.rados.0.smithi045.stdout:2349: finishing write tid 1 to smithi04526579-38 2024-04-26T00:33:34.501 INFO:tasks.rados.rados.0.smithi045.stdout:2349: finishing write tid 2 to smithi04526579-38 2024-04-26T00:33:34.501 INFO:tasks.rados.rados.0.smithi045.stdout:2349: finishing write tid 3 to smithi04526579-38 2024-04-26T00:33:34.501 INFO:tasks.rados.rados.0.smithi045.stdout:2349: finishing write tid 4 to smithi04526579-38 2024-04-26T00:33:34.501 INFO:tasks.rados.rados.0.smithi045.stdout:2349: oid 38 updating version 0 to 1307 2024-04-26T00:33:34.501 INFO:tasks.rados.rados.0.smithi045.stdout:2349: oid 38 updating version 1307 to 1308 2024-04-26T00:33:34.501 INFO:tasks.rados.rados.0.smithi045.stdout:2349: oid 38 updating version 1308 to 1309 2024-04-26T00:33:34.501 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 1309 (ObjNum 858 snap 264 seq_num 858) dirty exists 2024-04-26T00:33:34.501 INFO:tasks.rados.rados.0.smithi045.stdout:2349: left oid 38 (ObjNum 858 snap 264 seq_num 858) 2024-04-26T00:33:34.501 INFO:tasks.rados.rados.0.smithi045.stdout:2351: finishing write tid 1 to smithi04526579-25 2024-04-26T00:33:34.501 INFO:tasks.rados.rados.0.smithi045.stdout:2351: finishing write tid 2 to smithi04526579-25 2024-04-26T00:33:34.501 INFO:tasks.rados.rados.0.smithi045.stdout:2351: finishing write tid 3 to smithi04526579-25 2024-04-26T00:33:34.501 INFO:tasks.rados.rados.0.smithi045.stdout:2351: finishing write tid 4 to smithi04526579-25 2024-04-26T00:33:34.501 INFO:tasks.rados.rados.0.smithi045.stdout:2351: oid 25 updating version 0 to 1990 2024-04-26T00:33:34.501 INFO:tasks.rados.rados.0.smithi045.stdout:2351: oid 25 version 1990 is already newer than 1989 2024-04-26T00:33:34.501 INFO:tasks.rados.rados.0.smithi045.stdout:2351: oid 25 updating version 1990 to 1991 2024-04-26T00:33:34.502 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 1991 (ObjNum 859 snap 264 seq_num 859) dirty exists 2024-04-26T00:33:34.502 INFO:tasks.rados.rados.0.smithi045.stdout:2351: left oid 25 (ObjNum 859 snap 264 seq_num 859) 2024-04-26T00:33:35.505 INFO:tasks.rados.rados.0.smithi045.stdout:2347: done (5 left) 2024-04-26T00:33:35.505 INFO:tasks.rados.rados.0.smithi045.stdout:2348: done (4 left) 2024-04-26T00:33:35.505 INFO:tasks.rados.rados.0.smithi045.stdout:2349: done (3 left) 2024-04-26T00:33:35.505 INFO:tasks.rados.rados.0.smithi045.stdout:2350: done (2 left) 2024-04-26T00:33:35.505 INFO:tasks.rados.rados.0.smithi045.stdout:2351: done (1 left) 2024-04-26T00:33:35.505 INFO:tasks.rados.rados.0.smithi045.stdout:2354: done (0 left) 2024-04-26T00:33:35.506 INFO:tasks.rados.rados.0.smithi045.stdout:2355: delete oid 8 current snap is 265 2024-04-26T00:33:35.513 INFO:tasks.rados.rados.0.smithi045.stdout:2355: done (0 left) 2024-04-26T00:33:35.513 INFO:tasks.rados.rados.0.smithi045.stdout:2356: rollback oid 40 current snap is 265 2024-04-26T00:33:35.513 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 40 to 246 2024-04-26T00:33:35.513 INFO:tasks.rados.rados.0.smithi045.stdout:2357: read oid 24 snap 160 2024-04-26T00:33:35.513 INFO:tasks.rados.rados.0.smithi045.stdout:2357: expect (ObjNum 505 snap 155 seq_num 505) 2024-04-26T00:33:35.513 INFO:tasks.rados.rados.0.smithi045.stdout:2358: write (excl) oid 35 current snap is 265 2024-04-26T00:33:35.514 INFO:tasks.rados.rados.0.smithi045.stdout:2358: seq_num 861 ranges {470827=737278,1750408=720419} 2024-04-26T00:33:35.529 INFO:tasks.rados.rados.0.smithi045.stdout:2358: writing smithi04526579-35 from 470827 to 1208105 tid 1 2024-04-26T00:33:35.545 INFO:tasks.rados.rados.0.smithi045.stdout:2358: writing smithi04526579-35 from 1750408 to 2470827 tid 2 2024-04-26T00:33:35.546 INFO:tasks.rados.rados.0.smithi045.stdout:2359: snap_create 2024-04-26T00:33:35.546 INFO:tasks.rados.rados.0.smithi045.stdout:2356: finishing rollback tid 1 to smithi04526579-40 2024-04-26T00:33:35.546 INFO:tasks.rados.rados.0.smithi045.stdout:2356: finishing rollback tid 2 to smithi04526579-40 2024-04-26T00:33:35.546 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 1467 (ObjNum 784 snap 241 seq_num 784) dirty exists 2024-04-26T00:33:35.546 INFO:tasks.rados.rados.0.smithi045.stdout:2357: expect (ObjNum 505 snap 155 seq_num 505) 2024-04-26T00:33:35.718 INFO:tasks.rados.rados.0.smithi045.stdout:2358: finishing write tid 1 to smithi04526579-35 2024-04-26T00:33:35.719 INFO:tasks.rados.rados.0.smithi045.stdout:2358: finishing write tid 2 to smithi04526579-35 2024-04-26T00:33:35.719 INFO:tasks.rados.rados.0.smithi045.stdout:2358: finishing write tid 3 to smithi04526579-35 2024-04-26T00:33:35.719 INFO:tasks.rados.rados.0.smithi045.stdout:2358: finishing write tid 4 to smithi04526579-35 2024-04-26T00:33:35.719 INFO:tasks.rados.rados.0.smithi045.stdout:2358: oid 35 updating version 0 to 1611 2024-04-26T00:33:35.719 INFO:tasks.rados.rados.0.smithi045.stdout:2358: oid 35 updating version 1611 to 1612 2024-04-26T00:33:35.719 INFO:tasks.rados.rados.0.smithi045.stdout:2358: oid 35 updating version 1612 to 1613 2024-04-26T00:33:35.719 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 1613 (ObjNum 861 snap 265 seq_num 861) dirty exists 2024-04-26T00:33:35.720 INFO:tasks.rados.rados.0.smithi045.stdout:2358: left oid 35 (ObjNum 861 snap 265 seq_num 861) 2024-04-26T00:33:35.803 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:33:35.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bbb329640 10 monclient: get_monmap_and_config 2024-04-26T00:33:35.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bbb329640 10 monclient: build_initial_monmap 2024-04-26T00:33:35.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bbb329640 10 monclient: monmap: 2024-04-26T00:33:35.937 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:33:35.937 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:35.937 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:33:35.945208+0000 2024-04-26T00:33:35.937 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:33:35.945208+0000 2024-04-26T00:33:35.937 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:33:35.937 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:35.937 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:33:35.937 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:33:35.937 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:33:35.937 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:33:35.937 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:33:35.937 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:35.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bbb329640 1 Processor -- start 2024-04-26T00:33:35.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bbb329640 1 -- start start 2024-04-26T00:33:35.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bbb329640 10 monclient: init 2024-04-26T00:33:35.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bbb329640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:35.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bbb329640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-04-26T00:33:35.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bbb329640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9bb40a0260 0x7f9bb40a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:35.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bbb329640 10 monclient(hunting): picked mon.noname-c con 0x7f9bb40a0260 addr v2:172.21.15.45:3300/0 2024-04-26T00:33:35.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bbb329640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9bb4134b30 0x7f9bb4134f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:35.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bbb329640 10 monclient(hunting): picked mon.noname-a con 0x7f9bb4134b30 addr v2:172.21.15.37:3300/0 2024-04-26T00:33:35.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bbb329640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9bb4135490 addr v1:172.21.15.45:6789/0 2024-04-26T00:33:35.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bbb329640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9bb40a1130 con 0x7f9bb4135490 2024-04-26T00:33:35.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bbb329640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:35.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bbb329640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f9bb409ec10 con 0x7f9bb4134b30 2024-04-26T00:33:35.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bbb329640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:35.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bbb329640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f9bb409f0f0 con 0x7f9bb40a0260 2024-04-26T00:33:35.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bbb329640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:35.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bbb329640 10 monclient(hunting): authenticate will time out at 1797.250667s 2024-04-26T00:33:35.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bb88c4640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9bb4134b30 0x7f9bb4134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:35.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bb88c4640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9bb4134b30 0x7f9bb4134f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:37596/0 (socket says 172.21.15.37:37596) 2024-04-26T00:33:35.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bb88c4640 1 -- 172.21.15.37:0/3188248568 learned_addr learned my addr 172.21.15.37:0/3188248568 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:33:35.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bb88c4640 10 monclient(hunting): get_auth_request con 0x7f9bb4134b30 auth_method 0 2024-04-26T00:33:35.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bb88c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:35.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bb88c4640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:35.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bb88c4640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:35.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bb88c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:35.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bb88c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:35.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bb90c5640 1 --2- 172.21.15.37:0/3188248568 >> v2:172.21.15.45:3300/0 conn(0x7f9bb40a0260 0x7f9bb40a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:35.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bb88c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:35.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bb90c5640 10 monclient(hunting): get_auth_request con 0x7f9bb40a0260 auth_method 0 2024-04-26T00:33:35.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bb90c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:35.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bb90c5640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:35.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bb90c5640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:35.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bb88c4640 10 monclient(hunting): handle_auth_done global_id 144285 payload 274 2024-04-26T00:33:35.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bb90c5640 1 -- 172.21.15.37:0/3188248568 >> v2:172.21.15.45:3300/0 conn(0x7f9bb40a0260 msgr2=0x7f9bb40a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:33:35.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bb90c5640 1 -- 172.21.15.37:0/3188248568 >> v2:172.21.15.45:3300/0 conn(0x7f9bb40a0260 msgr2=0x7f9bb40a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:33:35.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bb90c5640 1 --2- 172.21.15.37:0/3188248568 >> v2:172.21.15.45:3300/0 conn(0x7f9bb40a0260 0x7f9bb40a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:33:35.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bb88c4640 1 -- 172.21.15.37:0/3188248568 >> v2:172.21.15.45:3300/0 conn(0x7f9bb40a0260 msgr2=0x7f9bb40a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:33:35.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bb90c5640 1 --2- 172.21.15.37:0/3188248568 >> v2:172.21.15.45:3300/0 conn(0x7f9bb40a0260 0x7f9bb40a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:33:35.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bb88c4640 1 --2- 172.21.15.37:0/3188248568 >> v2:172.21.15.45:3300/0 conn(0x7f9bb40a0260 0x7f9bb40a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:35.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bb88c4640 1 -- 172.21.15.37:0/3188248568 >> v1:172.21.15.45:6789/0 conn(0x7f9bb4135490 legacy=0x7f9bb413be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:33:35.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bb88c4640 10 monclient: _finish_hunting 0 2024-04-26T00:33:35.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bb88c4640 1 monclient: found mon.noname-a 2024-04-26T00:33:35.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bb88c4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:35.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bb88c4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:33:35.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.943+0000 7f9bb88c4640 1 -- 172.21.15.37:0/3188248568 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bb406cff0 con 0x7f9bb4134b30 2024-04-26T00:33:35.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.951+0000 7f9bb88c4640 1 -- 172.21.15.37:0/3188248568 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bb4134b30 msgr2=0x7f9bb4134f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:35.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.951+0000 7f9bb88c4640 1 --2- 172.21.15.37:0/3188248568 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bb4134b30 0x7f9bb4134f50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9bb40a2ad0 tx=0x7f9ba401c1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b513b2636c1935e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:35.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9babfff640 1 -- 172.21.15.37:0/3188248568 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba40b0070 con 0x7f9bb4134b30 2024-04-26T00:33:35.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9babfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:35.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9babfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:33:35.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9babfff640 10 monclient: dump: 2024-04-26T00:33:35.961 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:35.961 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:35.961 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:35.961 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:35.961 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:35.961 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:35.961 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:35.961 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:35.961 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:35.961 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:35.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9babfff640 10 monclient: _finish_auth 0 2024-04-26T00:33:35.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9babfff640 10 monclient: _check_auth_tickets 2024-04-26T00:33:35.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9babfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:35.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9babfff640 1 -- 172.21.15.37:0/3188248568 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ba40ab600 con 0x7f9bb4134b30 2024-04-26T00:33:35.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bbb329640 5 monclient: authenticate success, global_id 144285 2024-04-26T00:33:35.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bbb329640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:33:35.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9babfff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:33:35.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bbb329640 10 monclient: get_monmap_and_config success 2024-04-26T00:33:35.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9babfff640 1 -- 172.21.15.37:0/3188248568 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba40ab9d0 con 0x7f9bb4134b30 2024-04-26T00:33:35.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bbb329640 4 set_mon_vals no callback set 2024-04-26T00:33:35.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9babfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:35.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9babfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:33:35.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9babfff640 10 monclient: dump: 2024-04-26T00:33:35.963 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:35.963 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:35.963 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:35.963 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:35.963 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:35.964 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:35.964 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:35.964 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:35.964 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:35.964 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:35.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bbb329640 10 monclient: shutdown 2024-04-26T00:33:35.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bbb329640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:33:35.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bbb329640 1 -- 172.21.15.37:0/3188248568 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bb4134b30 msgr2=0x7f9bb4134f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:35.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bbb329640 1 --2- 172.21.15.37:0/3188248568 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bb4134b30 0x7f9bb4134f50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9bb40a2ad0 tx=0x7f9ba401c1c0 comp rx=0 tx=0).stop 2024-04-26T00:33:35.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bbb329640 1 -- 172.21.15.37:0/3188248568 shutdown_connections 2024-04-26T00:33:35.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bbb329640 1 --2- 172.21.15.37:0/3188248568 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bb4134b30 0x7f9bb4134f50 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:35.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bbb329640 1 --2- 172.21.15.37:0/3188248568 >> v2:172.21.15.45:3300/0 conn(0x7f9bb40a0260 0x7f9bb40a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:35.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bbb329640 1 -- 172.21.15.37:0/3188248568 >> 172.21.15.37:0/3188248568 conn(0x7f9bb40a1ad0 msgr2=0x7f9bb40a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:33:35.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bbb329640 1 -- 172.21.15.37:0/3188248568 shutdown_connections 2024-04-26T00:33:35.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bbb329640 1 -- 172.21.15.37:0/3188248568 wait complete. 2024-04-26T00:33:35.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bbb329640 10 monclient: build_initial_monmap 2024-04-26T00:33:35.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bbb329640 10 monclient: monmap: 2024-04-26T00:33:35.969 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:33:35.969 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:33:35.969 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:33:35.969 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:33:35.970 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:33:35.970 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:33:35.970 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:33:35.970 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:33:35.970 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:33:35.970 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:35.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bbb329640 1 Processor -- start 2024-04-26T00:33:35.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bbb329640 1 -- start start 2024-04-26T00:33:35.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bbb329640 10 monclient: init 2024-04-26T00:33:35.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bbb329640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:35.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bbb329640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:33:35.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bbb329640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bb40a0260 0x7f9bb4071100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:35.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bbb329640 10 monclient(hunting): picked mon.noname-a con 0x7f9bb40a0260 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:33:35.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bbb329640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bb4134b30 0x7f9bb4131bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:35.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bbb329640 10 monclient(hunting): picked mon.noname-b con 0x7f9bb4134b30 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:33:35.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bbb329640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9bb4135490 0x7f9bb4132130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:35.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bbb329640 10 monclient(hunting): picked mon.noname-c con 0x7f9bb4135490 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:33:35.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bbb329640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:35.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bbb329640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb409f430 con 0x7f9bb40a0260 2024-04-26T00:33:35.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bbb329640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:35.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bbb329640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb409f2b0 con 0x7f9bb4134b30 2024-04-26T00:33:35.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bbb329640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:35.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bbb329640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9bb409fe90 con 0x7f9bb4135490 2024-04-26T00:33:35.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bbb329640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:35.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bbb329640 10 monclient(hunting): authenticate will time out at 1797.275139s 2024-04-26T00:33:35.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bb90c5640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bb40a0260 0x7f9bb4071100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:35.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bb98c6640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9bb4135490 0x7f9bb4132130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:35.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bb90c5640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bb40a0260 0x7f9bb4071100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:37610/0 (socket says 172.21.15.37:37610) 2024-04-26T00:33:35.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bb90c5640 1 -- 172.21.15.37:0/4043752074 learned_addr learned my addr 172.21.15.37:0/4043752074 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:33:35.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bb90c5640 10 monclient(hunting): get_auth_request con 0x7f9bb40a0260 auth_method 0 2024-04-26T00:33:35.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bb90c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:35.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bb90c5640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:35.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bb90c5640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:35.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bb98c6640 10 monclient(hunting): get_auth_request con 0x7f9bb4135490 auth_method 0 2024-04-26T00:33:35.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bb98c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:35.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bb98c6640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:35.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bb98c6640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:35.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bb90c5640 1 -- 172.21.15.37:0/4043752074 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bb40a0260 msgr2=0x7f9bb4071100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:33:35.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bb90c5640 1 -- 172.21.15.37:0/4043752074 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bb40a0260 msgr2=0x7f9bb4071100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:33:35.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bb90c5640 1 --2- 172.21.15.37:0/4043752074 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bb40a0260 0x7f9bb4071100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:33:35.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bb88c4640 1 --2- 172.21.15.37:0/4043752074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bb4134b30 0x7f9bb4131bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:35.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bb90c5640 1 --2- 172.21.15.37:0/4043752074 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bb40a0260 0x7f9bb4071100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:33:35.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bb98c6640 1 -- 172.21.15.37:0/4043752074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9bb4135490 msgr2=0x7f9bb4132130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:33:35.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bb98c6640 1 -- 172.21.15.37:0/4043752074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9bb4135490 msgr2=0x7f9bb4132130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:33:35.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bb98c6640 1 --2- 172.21.15.37:0/4043752074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9bb4135490 0x7f9bb4132130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:33:35.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bb98c6640 1 --2- 172.21.15.37:0/4043752074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9bb4135490 0x7f9bb4132130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:33:35.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bb88c4640 10 monclient(hunting): get_auth_request con 0x7f9bb4134b30 auth_method 0 2024-04-26T00:33:35.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bb88c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:35.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bb88c4640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:35.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bb88c4640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:35.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bb88c4640 1 -- 172.21.15.37:0/4043752074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bb4134b30 msgr2=0x7f9bb4131bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:33:35.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bb88c4640 1 -- 172.21.15.37:0/4043752074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bb4134b30 msgr2=0x7f9bb4131bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:33:35.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bb88c4640 1 --2- 172.21.15.37:0/4043752074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bb4134b30 0x7f9bb4131bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:33:35.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:35.967+0000 7f9bb88c4640 1 --2- 172.21.15.37:0/4043752074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bb4134b30 0x7f9bb4131bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:33:36.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.171+0000 7f9bb98c6640 1 --2- 172.21.15.37:0/4043752074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9bb4135490 0x7f9bb4132130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:36.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.171+0000 7f9bb90c5640 1 --2- 172.21.15.37:0/4043752074 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bb40a0260 0x7f9bb4071100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:36.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.171+0000 7f9bb98c6640 10 monclient(hunting): get_auth_request con 0x7f9bb4135490 auth_method 0 2024-04-26T00:33:36.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.171+0000 7f9bb98c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:36.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.171+0000 7f9bb98c6640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:36.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.171+0000 7f9bb98c6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:33:36.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.171+0000 7f9bb90c5640 10 monclient(hunting): get_auth_request con 0x7f9bb40a0260 auth_method 0 2024-04-26T00:33:36.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.171+0000 7f9bb90c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:36.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.171+0000 7f9bb90c5640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:36.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.171+0000 7f9bb90c5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:33:36.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.171+0000 7f9bb88c4640 1 --2- 172.21.15.37:0/4043752074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bb4134b30 0x7f9bb4131bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:36.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.171+0000 7f9bb88c4640 10 monclient(hunting): get_auth_request con 0x7f9bb4134b30 auth_method 0 2024-04-26T00:33:36.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.171+0000 7f9bb88c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:36.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.171+0000 7f9bb88c4640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:36.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.171+0000 7f9bb88c4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:33:36.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.171+0000 7f9bb98c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:36.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.171+0000 7f9bb98c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:36.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.171+0000 7f9bb98c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:36.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.171+0000 7f9bb90c5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:36.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.171+0000 7f9bb90c5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:36.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.171+0000 7f9bb90c5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:36.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.171+0000 7f9bb88c4640 1 -- 172.21.15.37:0/4043752074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bb4134b30 msgr2=0x7f9bb4131bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:33:36.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.171+0000 7f9bb88c4640 1 -- 172.21.15.37:0/4043752074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bb4134b30 msgr2=0x7f9bb4131bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:33:36.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.171+0000 7f9bb88c4640 1 --2- 172.21.15.37:0/4043752074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bb4134b30 0x7f9bb4131bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:33:36.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.171+0000 7f9bb88c4640 1 --2- 172.21.15.37:0/4043752074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bb4134b30 0x7f9bb4131bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:33:36.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.171+0000 7f9bb98c6640 10 monclient(hunting): handle_auth_done global_id 154169 payload 824 2024-04-26T00:33:36.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.171+0000 7f9bb98c6640 1 -- 172.21.15.37:0/4043752074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bb4134b30 msgr2=0x7f9bb4131bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:33:36.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.171+0000 7f9bb98c6640 1 --2- 172.21.15.37:0/4043752074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bb4134b30 0x7f9bb4131bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:36.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.171+0000 7f9bb98c6640 1 -- 172.21.15.37:0/4043752074 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bb40a0260 msgr2=0x7f9bb4071100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:36.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.171+0000 7f9bb98c6640 1 --2- 172.21.15.37:0/4043752074 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bb40a0260 0x7f9bb4071100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:36.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.171+0000 7f9bb98c6640 10 monclient: _finish_hunting 0 2024-04-26T00:33:36.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.171+0000 7f9bb98c6640 1 monclient: found mon.noname-c 2024-04-26T00:33:36.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.171+0000 7f9bb98c6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:36.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.171+0000 7f9bb98c6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:33:36.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.171+0000 7f9bb98c6640 1 -- 172.21.15.37:0/4043752074 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bb411e2c0 con 0x7f9bb4135490 2024-04-26T00:33:36.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.171+0000 7f9bb90c5640 1 --2- 172.21.15.37:0/4043752074 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bb40a0260 0x7f9bb4071100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:33:36.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.171+0000 7f9bb98c6640 1 -- 172.21.15.37:0/4043752074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9bb4135490 msgr2=0x7f9bb4132130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:36.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.175+0000 7f9bb98c6640 1 --2- 172.21.15.37:0/4043752074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9bb4135490 0x7f9bb4132130 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9bac007f60 tx=0x7f9bac00ebb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:36.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.175+0000 7f9ba9ffb640 1 -- 172.21.15.37:0/4043752074 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bac00ee00 con 0x7f9bb4135490 2024-04-26T00:33:36.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.175+0000 7f9ba9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:36.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.175+0000 7f9ba9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:33:36.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.175+0000 7f9ba9ffb640 10 monclient: dump: 2024-04-26T00:33:36.171 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:36.171 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:36.171 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:36.171 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:36.171 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:36.171 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:36.171 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:36.171 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:36.171 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:36.171 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:36.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.175+0000 7f9ba9ffb640 10 monclient: _finish_auth 0 2024-04-26T00:33:36.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.175+0000 7f9ba9ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:33:36.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.175+0000 7f9ba9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:36.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.175+0000 7f9ba9ffb640 1 -- 172.21.15.37:0/4043752074 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bac00cae0 con 0x7f9bb4135490 2024-04-26T00:33:36.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.175+0000 7f9ba9ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:33:36.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.175+0000 7f9ba9ffb640 1 -- 172.21.15.37:0/4043752074 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bac013420 con 0x7f9bb4135490 2024-04-26T00:33:36.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.175+0000 7f9ba9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:36.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.175+0000 7f9ba9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:33:36.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.175+0000 7f9ba9ffb640 10 monclient: dump: 2024-04-26T00:33:36.172 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:36.172 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:36.172 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:36.172 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:36.172 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:36.172 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:36.172 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:36.172 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:36.173 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:36.173 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:36.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.175+0000 7f9bbb329640 5 monclient: authenticate success, global_id 154169 2024-04-26T00:33:36.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.175+0000 7f9bbb329640 10 monclient: _renew_subs 2024-04-26T00:33:36.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.175+0000 7f9bbb329640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:33:36.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.175+0000 7f9bab7fe640 4 set_mon_vals no callback set 2024-04-26T00:33:36.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.175+0000 7f9bbb329640 1 -- 172.21.15.37:0/4043752074 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bb411e5a0 con 0x7f9bb4135490 2024-04-26T00:33:36.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.175+0000 7f9bbb329640 10 monclient: _renew_subs 2024-04-26T00:33:36.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.175+0000 7f9bbb329640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:33:36.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.175+0000 7f9bbb329640 1 -- 172.21.15.37:0/4043752074 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bb4071ff0 con 0x7f9bb4135490 2024-04-26T00:33:36.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.175+0000 7f9bbb329640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:36.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.175+0000 7f9bbb329640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:36.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.175+0000 7f9bbb329640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:33:36.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.175+0000 7f9bbb329640 1 -- 172.21.15.37:0/4043752074 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bb4066f40 con 0x7f9bb4135490 2024-04-26T00:33:36.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.175+0000 7f9ba9ffb640 1 -- 172.21.15.37:0/4043752074 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9bac013600 con 0x7f9bb4135490 2024-04-26T00:33:36.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.175+0000 7f9ba9ffb640 1 --2- 172.21.15.37:0/4043752074 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9b98035f00 0x7f9b980383c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:36.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.175+0000 7f9ba9ffb640 1 -- 172.21.15.37:0/4043752074 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(530..530 src has 370..530) ==== 3915+0+0 (secure 0 0 0) 0x7f9bac0492b0 con 0x7f9bb4135490 2024-04-26T00:33:36.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.175+0000 7f9bb90c5640 1 --2- 172.21.15.37:0/4043752074 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9b98035f00 0x7f9b980383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:36.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.175+0000 7f9bb90c5640 10 monclient: get_auth_request con 0x7f9b98035f00 auth_method 0 2024-04-26T00:33:36.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.179+0000 7f9bb90c5640 1 --2- 172.21.15.37:0/4043752074 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9b98035f00 0x7f9b980383c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f9bb4071990 tx=0x7f9bb00093c0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:36.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.183+0000 7f9ba9ffb640 1 -- 172.21.15.37:0/4043752074 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9bac0048c0 con 0x7f9bb4135490 2024-04-26T00:33:36.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.183+0000 7f9ba9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:36.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.183+0000 7f9ba9ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:33:36.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.411+0000 7f9bbb329640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:33:36.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.411+0000 7f9bbb329640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:33:36.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.411+0000 7f9bbb329640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:33:36.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.411+0000 7f9bbb329640 1 -- 172.21.15.37:0/4043752074 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9bb40a0640 con 0x7f9bb4135490 2024-04-26T00:33:36.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.411+0000 7f9ba9ffb640 1 -- 172.21.15.37:0/4043752074 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f9bac049060 con 0x7f9bb4135490 2024-04-26T00:33:36.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.411+0000 7f9ba9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:33:36.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.411+0000 7f9ba9ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:33:36.406 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":102,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:33:36.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.415+0000 7f9bbb329640 1 -- 172.21.15.37:0/4043752074 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9b98035f00 msgr2=0x7f9b980383c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:36.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.415+0000 7f9bbb329640 1 --2- 172.21.15.37:0/4043752074 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9b98035f00 0x7f9b980383c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f9bb4071990 tx=0x7f9bb00093c0 comp rx=0 tx=0).stop 2024-04-26T00:33:36.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.415+0000 7f9bbb329640 10 monclient: shutdown 2024-04-26T00:33:36.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.415+0000 7f9bbb329640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:33:36.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.415+0000 7f9bbb329640 1 -- 172.21.15.37:0/4043752074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9bb4135490 msgr2=0x7f9bb4132130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:36.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.415+0000 7f9bbb329640 1 --2- 172.21.15.37:0/4043752074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9bb4135490 0x7f9bb4132130 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9bac007f60 tx=0x7f9bac00ebb0 comp rx=0 tx=0).stop 2024-04-26T00:33:36.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.415+0000 7f9bbb329640 1 -- 172.21.15.37:0/4043752074 shutdown_connections 2024-04-26T00:33:36.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.415+0000 7f9bbb329640 1 --2- 172.21.15.37:0/4043752074 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9b98035f00 0x7f9b980383c0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:36.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.415+0000 7f9bbb329640 1 --2- 172.21.15.37:0/4043752074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9bb4135490 0x7f9bb4132130 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:36.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.415+0000 7f9bbb329640 1 --2- 172.21.15.37:0/4043752074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9bb4134b30 0x7f9bb4131bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:36.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.415+0000 7f9bbb329640 1 --2- 172.21.15.37:0/4043752074 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9bb40a0260 0x7f9bb4071100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:36.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.415+0000 7f9bbb329640 1 -- 172.21.15.37:0/4043752074 >> 172.21.15.37:0/4043752074 conn(0x7f9bb40a1ad0 msgr2=0x7f9bb413bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:33:36.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.415+0000 7f9bbb329640 1 -- 172.21.15.37:0/4043752074 shutdown_connections 2024-04-26T00:33:36.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.415+0000 7f9bbb329640 1 -- 172.21.15.37:0/4043752074 wait complete. 2024-04-26T00:33:36.425 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:33:36.425 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:33:36.425 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:33:36.508 INFO:tasks.rados.rados.0.smithi045.stdout:2356: done (3 left) 2024-04-26T00:33:36.508 INFO:tasks.rados.rados.0.smithi045.stdout:2357: done (2 left) 2024-04-26T00:33:36.508 INFO:tasks.rados.rados.0.smithi045.stdout:2358: done (1 left) 2024-04-26T00:33:36.508 INFO:tasks.rados.rados.0.smithi045.stdout:2359: done (0 left) 2024-04-26T00:33:36.509 INFO:tasks.rados.rados.0.smithi045.stdout:2360: copy_from oid 42 from oid 33 current snap is 266 2024-04-26T00:33:36.509 INFO:tasks.rados.rados.0.smithi045.stdout:2361: write (excl) oid 18 current snap is 266 2024-04-26T00:33:36.509 INFO:tasks.rados.rados.0.smithi045.stdout:2361: seq_num 862 ranges {487634=412527,1408890=592631} 2024-04-26T00:33:36.520 INFO:tasks.rados.rados.0.smithi045.stdout:2361: writing smithi04526579-18 from 487634 to 900161 tid 1 2024-04-26T00:33:36.535 INFO:tasks.rados.rados.0.smithi045.stdout:2361: writing smithi04526579-18 from 1408890 to 2001521 tid 2 2024-04-26T00:33:36.535 INFO:tasks.rados.rados.0.smithi045.stdout:2362: snap_remove snap 219 2024-04-26T00:33:36.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.563+0000 7f5a580ed640 10 monclient: get_monmap_and_config 2024-04-26T00:33:36.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.563+0000 7f5a580ed640 10 monclient: build_initial_monmap 2024-04-26T00:33:36.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.563+0000 7f5a580ed640 10 monclient: monmap: 2024-04-26T00:33:36.559 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:33:36.559 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:36.559 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:33:36.567775+0000 2024-04-26T00:33:36.559 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:33:36.567775+0000 2024-04-26T00:33:36.559 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:33:36.559 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:36.560 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:33:36.560 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:33:36.560 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:33:36.560 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:33:36.560 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:33:36.560 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:36.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.567+0000 7f5a580ed640 1 Processor -- start 2024-04-26T00:33:36.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.567+0000 7f5a580ed640 1 -- start start 2024-04-26T00:33:36.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.567+0000 7f5a580ed640 10 monclient: init 2024-04-26T00:33:36.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.567+0000 7f5a580ed640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:36.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.567+0000 7f5a580ed640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-04-26T00:33:36.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.567+0000 7f5a580ed640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5a500726b0 addr v1:172.21.15.45:6789/0 2024-04-26T00:33:36.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.567+0000 7f5a580ed640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5a500a0110 0x7f5a5013c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:36.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.567+0000 7f5a580ed640 10 monclient(hunting): picked mon.noname-c con 0x7f5a500a0110 addr v2:172.21.15.45:3300/0 2024-04-26T00:33:36.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.567+0000 7f5a580ed640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5a5013c630 addr v1:172.21.15.37:6789/0 2024-04-26T00:33:36.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.567+0000 7f5a580ed640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a500a1470 con 0x7f5a5013c630 2024-04-26T00:33:36.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.567+0000 7f5a580ed640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a500a0bd0 con 0x7f5a500726b0 2024-04-26T00:33:36.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.567+0000 7f5a580ed640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:36.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.567+0000 7f5a580ed640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f5a500a1bd0 con 0x7f5a500a0110 2024-04-26T00:33:36.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.567+0000 7f5a580ed640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:36.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.567+0000 7f5a580ed640 10 monclient(hunting): authenticate will time out at 1797.873380s 2024-04-26T00:33:36.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.567+0000 7f5a5668a640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f5a5013c630 0x7f5a5013ea20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:35128/0 (socket says 172.21.15.37:35128) 2024-04-26T00:33:36.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.567+0000 7f5a5668a640 1 -- 172.21.15.37:0/1132281097 learned_addr learned my addr 172.21.15.37:0/1132281097 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:33:36.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.567+0000 7f5a55688640 1 --2- 172.21.15.37:0/1132281097 >> v2:172.21.15.45:3300/0 conn(0x7f5a500a0110 0x7f5a5013c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:36.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.567+0000 7f5a55688640 10 monclient(hunting): get_auth_request con 0x7f5a500a0110 auth_method 0 2024-04-26T00:33:36.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.567+0000 7f5a55688640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:36.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.567+0000 7f5a55688640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:36.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.567+0000 7f5a55688640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:36.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.567+0000 7f5a55688640 1 -- 172.21.15.37:0/1132281097 >> v2:172.21.15.45:3300/0 conn(0x7f5a500a0110 msgr2=0x7f5a5013c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:33:36.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.567+0000 7f5a55688640 1 -- 172.21.15.37:0/1132281097 >> v2:172.21.15.45:3300/0 conn(0x7f5a500a0110 msgr2=0x7f5a5013c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:33:36.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.567+0000 7f5a55688640 1 --2- 172.21.15.37:0/1132281097 >> v2:172.21.15.45:3300/0 conn(0x7f5a500a0110 0x7f5a5013c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:33:36.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.567+0000 7f5a55688640 1 --2- 172.21.15.37:0/1132281097 >> v2:172.21.15.45:3300/0 conn(0x7f5a500a0110 0x7f5a5013c0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:33:36.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.567+0000 7f5a5668a640 1 -- 172.21.15.37:0/1132281097 >> v1:172.21.15.37:6789/0 conn(0x7f5a5013c630 legacy=0x7f5a5013ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:36.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.567+0000 7f5a54e87640 1 -- 172.21.15.37:0/1132281097 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3990343886 0 0) 0x7f5a48001460 con 0x7f5a5013c630 2024-04-26T00:33:36.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.567+0000 7f5a54e87640 10 monclient(hunting): _negotiate 2024-04-26T00:33:36.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.567+0000 7f5a54e87640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:36.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.567+0000 7f5a54e87640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:36.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.567+0000 7f5a54e87640 10 monclient(hunting): my global_id is 144291 2024-04-26T00:33:36.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.567+0000 7f5a54e87640 1 -- 172.21.15.37:0/1132281097 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a38003530 con 0x7f5a5013c630 2024-04-26T00:33:36.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a54e87640 1 -- 172.21.15.37:0/1132281097 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2391259434 0 0) 0x7f5a38003530 con 0x7f5a5013c630 2024-04-26T00:33:36.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a54e87640 1 -- 172.21.15.37:0/1132281097 >> v2:172.21.15.45:3300/0 conn(0x7f5a500a0110 msgr2=0x7f5a5013c0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:33:36.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a54e87640 1 --2- 172.21.15.37:0/1132281097 >> v2:172.21.15.45:3300/0 conn(0x7f5a500a0110 0x7f5a5013c0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:36.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a55e89640 1 -- 172.21.15.37:0/1132281097 >> v1:172.21.15.45:6789/0 conn(0x7f5a500726b0 legacy=0x7f5a50072a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:36.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a54e87640 1 -- 172.21.15.37:0/1132281097 >> v1:172.21.15.45:6789/0 conn(0x7f5a500726b0 legacy=0x7f5a50072a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:36.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a54e87640 10 monclient: _finish_hunting 0 2024-04-26T00:33:36.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a54e87640 1 monclient: found mon.noname-a-legacy 2024-04-26T00:33:36.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a54e87640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:36.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a54e87640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.37:6789/0 2024-04-26T00:33:36.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a54e87640 1 -- 172.21.15.37:0/1132281097 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a500a1190 con 0x7f5a5013c630 2024-04-26T00:33:36.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a54e87640 10 monclient: _finish_auth 0 2024-04-26T00:33:36.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a54e87640 10 monclient: _check_auth_tickets 2024-04-26T00:33:36.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a54e87640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:36.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a54e87640 1 -- 172.21.15.37:0/1132281097 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 866326386 0 0) 0x7f5a48001b00 con 0x7f5a5013c630 2024-04-26T00:33:36.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a580ed640 5 monclient: authenticate success, global_id 144291 2024-04-26T00:33:36.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a54e87640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:36.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a54e87640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-26T00:33:36.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a54e87640 10 monclient: dump: 2024-04-26T00:33:36.569 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:36.569 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:36.569 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:36.569 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:36.569 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:36.569 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:36.569 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:36.569 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:36.569 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:36.569 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:36.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a54e87640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] but i'm connected to v1:172.21.15.37:6789/0, reconnecting 2024-04-26T00:33:36.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a54e87640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:36.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a54e87640 1 -- 172.21.15.37:0/1132281097 >> v1:172.21.15.37:6789/0 conn(0x7f5a5013c630 legacy=0x7f5a5013ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:36.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a54e87640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:33:36.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a54e87640 1 --2- 172.21.15.37:0/1132281097 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5a38003a90 0x7f5a38003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:36.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a54e87640 10 monclient(hunting): picked mon.c con 0x7f5a38003a90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:33:36.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a54e87640 1 --2- 172.21.15.37:0/1132281097 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a38004810 0x7f5a38004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:36.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a54e87640 10 monclient(hunting): picked mon.a con 0x7f5a38004810 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:33:36.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a54e87640 1 --2- 172.21.15.37:0/1132281097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a38005420 0x7f5a3800ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:36.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a54e87640 10 monclient(hunting): picked mon.b con 0x7f5a38005420 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:33:36.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a54e87640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:36.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a54e87640 1 -- 172.21.15.37:0/1132281097 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f5a500a1bd0 con 0x7f5a38004810 2024-04-26T00:33:36.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a54e87640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:36.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a54e87640 1 -- 172.21.15.37:0/1132281097 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5a3800c330 con 0x7f5a38005420 2024-04-26T00:33:36.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a54e87640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:36.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a54e87640 1 -- 172.21.15.37:0/1132281097 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5a3800c530 con 0x7f5a38003a90 2024-04-26T00:33:36.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a54e87640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:36.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a54e87640 10 monclient(hunting): _finish_auth 0 2024-04-26T00:33:36.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a580ed640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-26T00:33:36.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a55688640 1 --2- 172.21.15.37:0/1132281097 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a38004810 0x7f5a38004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:36.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a55e89640 1 --2- 172.21.15.37:0/1132281097 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5a38003a90 0x7f5a38003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:36.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a55688640 10 monclient(hunting): get_auth_request con 0x7f5a38004810 auth_method 0 2024-04-26T00:33:36.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a55688640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:36.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a55688640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:36.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a55688640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:33:36.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a55e89640 10 monclient(hunting): get_auth_request con 0x7f5a38003a90 auth_method 0 2024-04-26T00:33:36.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a55e89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:36.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a55e89640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:36.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a55e89640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:33:36.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a55688640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:36.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a55688640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:36.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a55688640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:33:36.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a55e89640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:36.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a55e89640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:36.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a55e89640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:33:36.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a5668a640 1 --2- 172.21.15.37:0/1132281097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a38005420 0x7f5a3800ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:36.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a5668a640 10 monclient(hunting): get_auth_request con 0x7f5a38005420 auth_method 0 2024-04-26T00:33:36.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a5668a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:36.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a5668a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:36.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a5668a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:33:36.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a55688640 10 monclient(hunting): handle_auth_done global_id 144291 payload 293 2024-04-26T00:33:36.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a55688640 1 -- 172.21.15.37:0/1132281097 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5a38003a90 msgr2=0x7f5a38003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:33:36.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a55688640 1 --2- 172.21.15.37:0/1132281097 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5a38003a90 0x7f5a38003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:36.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a55688640 1 -- 172.21.15.37:0/1132281097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a38005420 msgr2=0x7f5a3800ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:33:36.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a55688640 1 --2- 172.21.15.37:0/1132281097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a38005420 0x7f5a3800ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:36.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a55688640 10 monclient: _finish_hunting 0 2024-04-26T00:33:36.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a55688640 1 monclient: found mon.a 2024-04-26T00:33:36.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a55688640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:36.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a55688640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:33:36.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a5668a640 1 -- 172.21.15.37:0/1132281097 reap_dead start 2024-04-26T00:33:36.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a55688640 1 -- 172.21.15.37:0/1132281097 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a38003530 con 0x7f5a38004810 2024-04-26T00:33:36.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a55688640 10 monclient: _finish_auth 0 2024-04-26T00:33:36.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a55688640 10 monclient: _check_auth_tickets 2024-04-26T00:33:36.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a55688640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:36.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.571+0000 7f5a55e89640 1 --2- 172.21.15.37:0/1132281097 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5a38003a90 0x7f5a38003eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:33:36.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a55688640 1 -- 172.21.15.37:0/1132281097 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a38004810 msgr2=0x7f5a38004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:36.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a55688640 1 --2- 172.21.15.37:0/1132281097 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a38004810 0x7f5a38004c90 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5a4400c820 tx=0x7f5a4401c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dc0ab8a117a8a3e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:36.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a54e87640 1 -- 172.21.15.37:0/1132281097 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a440b0070 con 0x7f5a38004810 2024-04-26T00:33:36.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a54e87640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:36.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a54e87640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:33:36.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a54e87640 10 monclient: dump: 2024-04-26T00:33:36.580 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:36.580 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:36.580 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:36.580 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:36.580 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:36.580 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:36.580 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:36.580 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:36.580 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:36.581 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:36.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a54e87640 1 -- 172.21.15.37:0/1132281097 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a440ab600 con 0x7f5a38004810 2024-04-26T00:33:36.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a54e87640 10 monclient: handle_config config(0 keys) 2024-04-26T00:33:36.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a54e87640 1 -- 172.21.15.37:0/1132281097 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a4400f040 con 0x7f5a38004810 2024-04-26T00:33:36.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a580ed640 10 monclient: get_monmap_and_config success 2024-04-26T00:33:36.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a580ed640 4 set_mon_vals no callback set 2024-04-26T00:33:36.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a54e87640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:36.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a54e87640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:33:36.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a54e87640 10 monclient: dump: 2024-04-26T00:33:36.581 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:36.581 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:36.581 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:36.581 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:36.581 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:36.582 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:36.582 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:36.582 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:36.582 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:36.582 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:36.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a580ed640 10 monclient: shutdown 2024-04-26T00:33:36.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a580ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:33:36.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a580ed640 1 -- 172.21.15.37:0/1132281097 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a38004810 msgr2=0x7f5a38004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:36.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a580ed640 1 --2- 172.21.15.37:0/1132281097 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a38004810 0x7f5a38004c90 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5a4400c820 tx=0x7f5a4401c4f0 comp rx=0 tx=0).stop 2024-04-26T00:33:36.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a580ed640 1 -- 172.21.15.37:0/1132281097 shutdown_connections 2024-04-26T00:33:36.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a580ed640 1 --2- 172.21.15.37:0/1132281097 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a38004810 0x7f5a38004c90 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:36.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a580ed640 1 -- 172.21.15.37:0/1132281097 >> 172.21.15.37:0/1132281097 conn(0x7f5a500a1d50 msgr2=0x7f5a500a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:33:36.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a580ed640 1 -- 172.21.15.37:0/1132281097 shutdown_connections 2024-04-26T00:33:36.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a580ed640 1 -- 172.21.15.37:0/1132281097 wait complete. 2024-04-26T00:33:36.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a580ed640 10 monclient: build_initial_monmap 2024-04-26T00:33:36.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a580ed640 10 monclient: monmap: 2024-04-26T00:33:36.583 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:33:36.583 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:33:36.583 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:33:36.583 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:33:36.583 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:33:36.583 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:33:36.583 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:33:36.583 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:33:36.583 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:33:36.583 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:36.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a580ed640 1 Processor -- start 2024-04-26T00:33:36.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a580ed640 1 -- start start 2024-04-26T00:33:36.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a580ed640 10 monclient: init 2024-04-26T00:33:36.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a580ed640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:36.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a580ed640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:33:36.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a580ed640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a38004810 0x7f5a501cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:36.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a580ed640 10 monclient(hunting): picked mon.noname-a con 0x7f5a38004810 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:33:36.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a580ed640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a50121320 0x7f5a501d0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:36.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a580ed640 10 monclient(hunting): picked mon.noname-b con 0x7f5a50121320 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:33:36.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a580ed640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5a5011d040 0x7f5a5011d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:36.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a580ed640 10 monclient(hunting): picked mon.noname-c con 0x7f5a5011d040 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:33:36.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a580ed640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:36.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a580ed640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f5a50137760 con 0x7f5a38004810 2024-04-26T00:33:36.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a580ed640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:36.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a580ed640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5a501375e0 con 0x7f5a50121320 2024-04-26T00:33:36.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a580ed640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:36.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a580ed640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5a501378e0 con 0x7f5a5011d040 2024-04-26T00:33:36.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a580ed640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:36.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a580ed640 10 monclient(hunting): authenticate will time out at 1797.883420s 2024-04-26T00:33:36.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a55e89640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a38004810 0x7f5a501cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:36.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a5668a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5a5011d040 0x7f5a5011d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:36.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a55e89640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a38004810 0x7f5a501cfb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:37638/0 (socket says 172.21.15.37:37638) 2024-04-26T00:33:36.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a55e89640 1 -- 172.21.15.37:0/3393880447 learned_addr learned my addr 172.21.15.37:0/3393880447 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:33:36.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a55e89640 10 monclient(hunting): get_auth_request con 0x7f5a38004810 auth_method 0 2024-04-26T00:33:36.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a55e89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:36.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a55e89640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:36.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a55e89640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:36.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a5668a640 10 monclient(hunting): get_auth_request con 0x7f5a5011d040 auth_method 0 2024-04-26T00:33:36.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a5668a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:36.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a5668a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:36.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a5668a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:36.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a55e89640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:36.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a55e89640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:36.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a55e89640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:36.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a5668a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:36.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a5668a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:36.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a5668a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:36.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a55688640 1 --2- 172.21.15.37:0/3393880447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a50121320 0x7f5a501d0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:36.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a55688640 10 monclient(hunting): get_auth_request con 0x7f5a50121320 auth_method 0 2024-04-26T00:33:36.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a55688640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:36.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a55688640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:36.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a55688640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:36.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a55e89640 10 monclient(hunting): handle_auth_done global_id 144294 payload 824 2024-04-26T00:33:36.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a55e89640 1 -- 172.21.15.37:0/3393880447 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5a5011d040 msgr2=0x7f5a5011d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:36.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a55e89640 1 --2- 172.21.15.37:0/3393880447 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5a5011d040 0x7f5a5011d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:36.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a55e89640 1 -- 172.21.15.37:0/3393880447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a50121320 msgr2=0x7f5a501d0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:36.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a55e89640 1 --2- 172.21.15.37:0/3393880447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5a50121320 0x7f5a501d0080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:36.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a55e89640 10 monclient: _finish_hunting 0 2024-04-26T00:33:36.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a55e89640 1 monclient: found mon.noname-a 2024-04-26T00:33:36.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a55e89640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:36.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a55e89640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:33:36.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a55e89640 1 -- 172.21.15.37:0/3393880447 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a50120f00 con 0x7f5a38004810 2024-04-26T00:33:36.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.575+0000 7f5a5668a640 1 --2- 172.21.15.37:0/3393880447 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5a5011d040 0x7f5a5011d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:33:36.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.579+0000 7f5a55e89640 1 -- 172.21.15.37:0/3393880447 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a38004810 msgr2=0x7f5a501cfb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:36.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.579+0000 7f5a55e89640 1 --2- 172.21.15.37:0/3393880447 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a38004810 0x7f5a501cfb40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5a4000fe10 tx=0x7f5a4000d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:36.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.579+0000 7f5a4e7fc640 1 -- 172.21.15.37:0/3393880447 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a40001d70 con 0x7f5a38004810 2024-04-26T00:33:36.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.579+0000 7f5a4e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:36.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.579+0000 7f5a4e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:33:36.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.579+0000 7f5a4e7fc640 10 monclient: dump: 2024-04-26T00:33:36.589 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:36.589 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:36.589 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:36.589 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:36.589 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:36.589 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:36.589 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:36.589 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:36.589 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:36.589 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:36.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.579+0000 7f5a4e7fc640 10 monclient: _finish_auth 0 2024-04-26T00:33:36.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.579+0000 7f5a4e7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:33:36.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.579+0000 7f5a4e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:36.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.579+0000 7f5a4e7fc640 1 -- 172.21.15.37:0/3393880447 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a40018070 con 0x7f5a38004810 2024-04-26T00:33:36.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.579+0000 7f5a4e7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:33:36.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.579+0000 7f5a4e7fc640 1 -- 172.21.15.37:0/3393880447 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a40005b50 con 0x7f5a38004810 2024-04-26T00:33:36.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.579+0000 7f5a4e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:36.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.579+0000 7f5a4e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:33:36.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.579+0000 7f5a4e7fc640 10 monclient: dump: 2024-04-26T00:33:36.590 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:36.590 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:36.590 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:36.590 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:36.590 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:36.590 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:36.591 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:36.591 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:36.591 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:36.591 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:36.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.579+0000 7f5a580ed640 5 monclient: authenticate success, global_id 144294 2024-04-26T00:33:36.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.579+0000 7f5a580ed640 10 monclient: _renew_subs 2024-04-26T00:33:36.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.579+0000 7f5a580ed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:33:36.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.579+0000 7f5a580ed640 1 -- 172.21.15.37:0/3393880447 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a5011ec10 con 0x7f5a38004810 2024-04-26T00:33:36.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.579+0000 7f5a4ffff640 4 set_mon_vals no callback set 2024-04-26T00:33:36.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.579+0000 7f5a580ed640 10 monclient: _renew_subs 2024-04-26T00:33:36.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.579+0000 7f5a580ed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:33:36.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.579+0000 7f5a580ed640 1 -- 172.21.15.37:0/3393880447 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a5011f4f0 con 0x7f5a38004810 2024-04-26T00:33:36.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.579+0000 7f5a580ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:36.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.579+0000 7f5a580ed640 1 --2- 172.21.15.37:0/3393880447 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a5011da00 0x7f5a5011dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:36.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.579+0000 7f5a580ed640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:36.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.579+0000 7f5a580ed640 1 -- 172.21.15.37:0/3393880447 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f5a5004f510 con 0x7f5a5011da00 2024-04-26T00:33:36.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.579+0000 7f5a55688640 1 --2- 172.21.15.37:0/3393880447 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a5011da00 0x7f5a5011dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:36.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.579+0000 7f5a55688640 10 monclient: get_auth_request con 0x7f5a5011da00 auth_method 0 2024-04-26T00:33:36.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.579+0000 7f5a55688640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:36.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.579+0000 7f5a55688640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:36.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.579+0000 7f5a55688640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:36.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.579+0000 7f5a55688640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:36.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.579+0000 7f5a55688640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:36.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.579+0000 7f5a55688640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:36.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.579+0000 7f5a4e7fc640 1 -- 172.21.15.37:0/3393880447 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f5a400070d0 con 0x7f5a38004810 2024-04-26T00:33:36.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.579+0000 7f5a55688640 10 monclient(hunting): handle_auth_done global_id 144297 payload 824 2024-04-26T00:33:36.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.579+0000 7f5a4e7fc640 1 --2- 172.21.15.37:0/3393880447 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5a3c035ed0 0x7f5a3c038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:36.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.579+0000 7f5a55688640 1 -- 172.21.15.37:0/3393880447 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5a5009d050 con 0x7f5a5011da00 2024-04-26T00:33:36.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.579+0000 7f5a4e7fc640 1 -- 172.21.15.37:0/3393880447 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(531..531 src has 370..531) ==== 3915+0+0 (secure 0 0 0) 0x7f5a40048b80 con 0x7f5a38004810 2024-04-26T00:33:36.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.583+0000 7f5a5668a640 1 --2- 172.21.15.37:0/3393880447 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5a3c035ed0 0x7f5a3c038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:36.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.583+0000 7f5a5668a640 10 monclient: get_auth_request con 0x7f5a3c035ed0 auth_method 0 2024-04-26T00:33:36.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.583+0000 7f5a55688640 1 -- 172.21.15.37:0/3393880447 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a5011da00 msgr2=0x7f5a5011dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:36.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.583+0000 7f5a55688640 1 --2- 172.21.15.37:0/3393880447 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a5011da00 0x7f5a5011dde0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5a44000ee0 tx=0x7f5a38005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:36.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.583+0000 7f5a4e7fc640 1 -- 172.21.15.37:0/3393880447 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a440bb030 con 0x7f5a5011da00 2024-04-26T00:33:36.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.583+0000 7f5a4e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:36.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.583+0000 7f5a4e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:33:36.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.583+0000 7f5a4e7fc640 10 monclient: dump: 2024-04-26T00:33:36.593 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:36.593 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:36.593 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:36.593 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:36.593 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:36.593 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:36.594 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:36.594 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:36.594 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:36.594 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:36.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.583+0000 7f5a4e7fc640 1 -- 172.21.15.37:0/3393880447 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5a4400f000 con 0x7f5a5011da00 2024-04-26T00:33:36.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.583+0000 7f5a4e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:36.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.583+0000 7f5a4e7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:33:36.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.583+0000 7f5a4e7fc640 1 -- 172.21.15.37:0/3393880447 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a5011da00 msgr2=0x7f5a5011dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:36.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.583+0000 7f5a4e7fc640 1 --2- 172.21.15.37:0/3393880447 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a5011da00 0x7f5a5011dde0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5a44000ee0 tx=0x7f5a38005840 comp rx=0 tx=0).stop 2024-04-26T00:33:36.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.583+0000 7f5a4e7fc640 1 -- 172.21.15.37:0/3393880447 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a5011da00 msgr2=0x7f5a5011dde0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:33:36.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.583+0000 7f5a4e7fc640 1 --2- 172.21.15.37:0/3393880447 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a5011da00 0x7f5a5011dde0 secure :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5a44000ee0 tx=0x7f5a38005840 comp rx=0 tx=0).stop 2024-04-26T00:33:36.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.583+0000 7f5a5668a640 1 --2- 172.21.15.37:0/3393880447 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5a3c035ed0 0x7f5a3c038390 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f5a500a3100 tx=0x7f5a48001450 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:36.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.591+0000 7f5a580ed640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:33:36.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.591+0000 7f5a580ed640 1 --2- 172.21.15.37:0/3393880447 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a50134210 0x7f5a50134640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:36.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.591+0000 7f5a580ed640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:36.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.591+0000 7f5a580ed640 1 -- 172.21.15.37:0/3393880447 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f5a501208a0 con 0x7f5a50134210 2024-04-26T00:33:36.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.591+0000 7f5a55688640 1 --2- 172.21.15.37:0/3393880447 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a50134210 0x7f5a50134640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:36.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.591+0000 7f5a55688640 10 monclient: get_auth_request con 0x7f5a50134210 auth_method 0 2024-04-26T00:33:36.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.591+0000 7f5a55688640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:36.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.591+0000 7f5a55688640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:36.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.591+0000 7f5a55688640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:36.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.591+0000 7f5a55688640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:36.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.591+0000 7f5a55688640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:36.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.591+0000 7f5a55688640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:36.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.591+0000 7f5a55688640 10 monclient(hunting): handle_auth_done global_id 144300 payload 824 2024-04-26T00:33:36.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.591+0000 7f5a55688640 1 -- 172.21.15.37:0/3393880447 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5a50073280 con 0x7f5a50134210 2024-04-26T00:33:36.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.591+0000 7f5a55688640 1 -- 172.21.15.37:0/3393880447 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a50134210 msgr2=0x7f5a50134640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:36.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.591+0000 7f5a55688640 1 --2- 172.21.15.37:0/3393880447 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a50134210 0x7f5a50134640 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5a44000c10 tx=0x7f5a440a6000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:36.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.591+0000 7f5a4e7fc640 1 -- 172.21.15.37:0/3393880447 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a440bb030 con 0x7f5a50134210 2024-04-26T00:33:36.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.591+0000 7f5a4e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:36.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.591+0000 7f5a4e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:33:36.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.591+0000 7f5a4e7fc640 10 monclient: dump: 2024-04-26T00:33:36.597 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:36.598 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:36.598 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:36.598 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:36.598 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:36.598 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:36.598 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:36.598 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:36.598 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:36.598 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:36.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.591+0000 7f5a4e7fc640 1 -- 172.21.15.37:0/3393880447 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4890 (secure 0 0 0) 0x7f5a440b7460 con 0x7f5a50134210 2024-04-26T00:33:36.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.591+0000 7f5a4e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:33:36.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.591+0000 7f5a4e7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:33:36.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.591+0000 7f5a4e7fc640 1 -- 172.21.15.37:0/3393880447 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a50134210 msgr2=0x7f5a50134640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:36.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.591+0000 7f5a4e7fc640 1 --2- 172.21.15.37:0/3393880447 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a50134210 0x7f5a50134640 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5a44000c10 tx=0x7f5a440a6000 comp rx=0 tx=0).stop 2024-04-26T00:33:36.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.591+0000 7f5a4e7fc640 1 -- 172.21.15.37:0/3393880447 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a50134210 msgr2=0x7f5a50134640 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:33:36.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.591+0000 7f5a4e7fc640 1 --2- 172.21.15.37:0/3393880447 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a50134210 0x7f5a50134640 secure :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5a44000c10 tx=0x7f5a440a6000 comp rx=0 tx=0).stop 2024-04-26T00:33:36.599 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:33:36.599 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:33:36.599 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:33:36.599 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:33:36.599 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 102, 2024-04-26T00:33:36.599 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 103134, 2024-04-26T00:33:36.599 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:33:36.599 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:33:36.600 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:33:36.600 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:33:36.600 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:36.600 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:33:36.600 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:33:36.600 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:33:36.600 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:33:36.600 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:33:36.600 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:33:36.600 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:33:36.600 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:33:36.600 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:33:36.600 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:33:36.600 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:33:36.601 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:33:36.601 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:33:36.601 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:33:36.601 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:33:36.601 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:36.601 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:33:36.601 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:33:36.601 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:33:36.601 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:33:36.601 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:33:36.601 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:33:36.601 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:33:36.601 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:33:36.601 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:33:36.601 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:33:36.602 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:33:36.602 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:33:36.602 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:33:36.602 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:36.602 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:36.602 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:33:36.602 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:33:36.602 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:33:36.602 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:33:36.602 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:33:36.602 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:33:36.602 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:33:36.602 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:33:36.602 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:33:36.603 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:33:36.603 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:33:36.603 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:33:36.603 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:33:36.603 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:33:36.603 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:33:36.603 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:33:36.603 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:33:36.603 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:33:36.603 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:33:36.603 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:33:36.603 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:33:36.603 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:33:36.603 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:33:36.604 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:33:36.604 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:33:36.604 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:33:36.604 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:33:36.604 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:33:36.604 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:36.604 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:33:36.604 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:36.604 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:33:36.604 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:36.604 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:33:36.604 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:33:36.604 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:33:36.604 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:33:36.604 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:36.605 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:33:36.605 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:33:36.605 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:36.605 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:36.605 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:36.605 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:33:36.605 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:33:36.605 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:36.605 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:36.605 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:36.605 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:36.605 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:33:36.605 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:33:36.605 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:33:36.606 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:33:36.606 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:33:36.606 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:36.606 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:36.606 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:33:36.606 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:33:36.606 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:33:36.606 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:33:36.606 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:36.606 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:33:36.606 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:33:36.606 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:36.606 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:36.607 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:36.607 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:33:36.607 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:33:36.607 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:36.607 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:36.607 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:36.607 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:36.607 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:33:36.607 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:33:36.607 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:33:36.607 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:33:36.607 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:33:36.607 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:36.607 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:36.607 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:33:36.608 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:33:36.608 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:33:36.608 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:33:36.608 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:36.608 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:33:36.608 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:33:36.608 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:36.608 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:36.608 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:36.608 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:33:36.608 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:33:36.608 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:36.608 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:36.608 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:36.608 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:36.609 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:33:36.609 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:33:36.609 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:33:36.609 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:33:36.609 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:33:36.609 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:36.609 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:36.609 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:36.609 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:33:36.609 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:33:36.609 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:36.609 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:33:36.609 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:33:36.609 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:33:36.610 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:36.610 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:36.610 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:33:36.610 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:36.610 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:33:36.610 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:33:36.610 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:33:36.610 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:36.610 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:36.610 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:33:36.610 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:36.610 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:33:36.610 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:33:36.610 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:33:36.611 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:36.611 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:36.611 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:33:36.611 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:36.611 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:33:36.611 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:33:36.611 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:33:36.611 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:36.611 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:36.611 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:36.611 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:33:36.611 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:33:36.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.595+0000 7f5a580ed640 1 -- 172.21.15.37:0/3393880447 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5a3c035ed0 msgr2=0x7f5a3c038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:36.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.595+0000 7f5a580ed640 1 --2- 172.21.15.37:0/3393880447 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5a3c035ed0 0x7f5a3c038390 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f5a500a3100 tx=0x7f5a48001450 comp rx=0 tx=0).stop 2024-04-26T00:33:36.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.595+0000 7f5a580ed640 10 monclient: shutdown 2024-04-26T00:33:36.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.595+0000 7f5a580ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:33:36.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.595+0000 7f5a580ed640 1 -- 172.21.15.37:0/3393880447 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a38004810 msgr2=0x7f5a501cfb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:36.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.595+0000 7f5a580ed640 1 --2- 172.21.15.37:0/3393880447 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a38004810 0x7f5a501cfb40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5a4000fe10 tx=0x7f5a4000d520 comp rx=0 tx=0).stop 2024-04-26T00:33:36.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.595+0000 7f5a5668a640 1 -- 172.21.15.37:0/3393880447 reap_dead start 2024-04-26T00:33:36.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.595+0000 7f5a5668a640 1 -- 172.21.15.37:0/3393880447 reap_dead start 2024-04-26T00:33:36.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.595+0000 7f5a580ed640 1 -- 172.21.15.37:0/3393880447 shutdown_connections 2024-04-26T00:33:36.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.595+0000 7f5a580ed640 1 -- 172.21.15.37:0/3393880447 >> 172.21.15.37:0/3393880447 conn(0x7f5a500a1d50 msgr2=0x7f5a501c1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:33:36.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.595+0000 7f5a580ed640 1 -- 172.21.15.37:0/3393880447 shutdown_connections 2024-04-26T00:33:36.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.595+0000 7f5a580ed640 1 -- 172.21.15.37:0/3393880447 wait complete. 2024-04-26T00:33:36.614 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:33:36.614 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:33:36.614 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:33:36.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.731+0000 7f1fecb31640 10 monclient: get_monmap_and_config 2024-04-26T00:33:36.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.731+0000 7f1fecb31640 10 monclient: build_initial_monmap 2024-04-26T00:33:36.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.731+0000 7f1fecb31640 10 monclient: monmap: 2024-04-26T00:33:36.727 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:33:36.727 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:36.727 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:33:36.735441+0000 2024-04-26T00:33:36.727 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:33:36.735441+0000 2024-04-26T00:33:36.728 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:33:36.728 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:36.728 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:33:36.728 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:33:36.728 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:33:36.728 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:33:36.728 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:33:36.728 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:36.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fecb31640 1 Processor -- start 2024-04-26T00:33:36.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fecb31640 1 -- start start 2024-04-26T00:33:36.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fecb31640 10 monclient: init 2024-04-26T00:33:36.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fecb31640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:36.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fecb31640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-04-26T00:33:36.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fecb31640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1fe80a2fe0 addr v1:172.21.15.37:6789/0 2024-04-26T00:33:36.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fecb31640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1fe8071da0 0x7f1fe80a1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:36.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fecb31640 10 monclient(hunting): picked mon.noname-c con 0x7f1fe8071da0 addr v2:172.21.15.45:3300/0 2024-04-26T00:33:36.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fecb31640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1fe81391c0 0x7f1fe813b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:36.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fecb31640 10 monclient(hunting): picked mon.noname-b con 0x7f1fe81391c0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:33:36.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fecb31640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fe8072d70 con 0x7f1fe80a2fe0 2024-04-26T00:33:36.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fecb31640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:36.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fecb31640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f1fe80710e0 con 0x7f1fe8071da0 2024-04-26T00:33:36.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fecb31640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:36.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fecb31640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f1fe80a33c0 con 0x7f1fe81391c0 2024-04-26T00:33:36.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fecb31640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:36.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fecb31640 10 monclient(hunting): authenticate will time out at 1798.041368s 2024-04-26T00:33:36.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fe6d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1fe81391c0 0x7f1fe813b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:36.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fe6d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1fe81391c0 0x7f1fe813b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:48358/0 (socket says 172.21.15.37:48358) 2024-04-26T00:33:36.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fe659c640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f1fe80a2fe0 0x7f1fe80a13d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:35138/0 (socket says 172.21.15.37:35138) 2024-04-26T00:33:36.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fe6d9d640 1 -- 172.21.15.37:0/2354057506 learned_addr learned my addr 172.21.15.37:0/2354057506 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:33:36.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fe6d9d640 10 monclient(hunting): get_auth_request con 0x7f1fe81391c0 auth_method 0 2024-04-26T00:33:36.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fe6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:36.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fe6d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:36.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fe6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:36.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fe6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:36.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fe6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:36.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fe6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:36.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fe5d9b640 1 --2- 172.21.15.37:0/2354057506 >> v2:172.21.15.45:3300/0 conn(0x7f1fe8071da0 0x7f1fe80a1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:36.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fe5d9b640 10 monclient(hunting): get_auth_request con 0x7f1fe8071da0 auth_method 0 2024-04-26T00:33:36.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fe5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:36.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fe5d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:36.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fe5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:36.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fe6d9d640 10 monclient(hunting): handle_auth_done global_id 154175 payload 274 2024-04-26T00:33:36.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fe6d9d640 1 -- 172.21.15.37:0/2354057506 >> v2:172.21.15.45:3300/0 conn(0x7f1fe8071da0 msgr2=0x7f1fe80a1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:33:36.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fe6d9d640 1 --2- 172.21.15.37:0/2354057506 >> v2:172.21.15.45:3300/0 conn(0x7f1fe8071da0 0x7f1fe80a1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:36.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fe6d9d640 1 -- 172.21.15.37:0/2354057506 >> v1:172.21.15.37:6789/0 conn(0x7f1fe80a2fe0 legacy=0x7f1fe80a13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:33:36.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fe6d9d640 10 monclient: _finish_hunting 0 2024-04-26T00:33:36.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fe6d9d640 1 monclient: found mon.noname-b 2024-04-26T00:33:36.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fe6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:36.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fe6d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:33:36.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.735+0000 7f1fe6d9d640 1 -- 172.21.15.37:0/2354057506 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fe8070b20 con 0x7f1fe81391c0 2024-04-26T00:33:36.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.739+0000 7f1fe6d9d640 1 -- 172.21.15.37:0/2354057506 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1fe81391c0 msgr2=0x7f1fe813b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:36.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.739+0000 7f1fe6d9d640 1 --2- 172.21.15.37:0/2354057506 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1fe81391c0 0x7f1fe813b5b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8071570 tx=0x7f1fe001c870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d456b69fc09b0e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:36.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.739+0000 7f1fe559a640 1 -- 172.21.15.37:0/2354057506 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe00a6500 con 0x7f1fe81391c0 2024-04-26T00:33:36.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.739+0000 7f1fe559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:36.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.739+0000 7f1fe559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:33:36.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.739+0000 7f1fe559a640 10 monclient: dump: 2024-04-26T00:33:36.748 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:36.748 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:36.748 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:36.748 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:36.748 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:36.748 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:36.748 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:36.748 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:36.748 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:36.748 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:36.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.739+0000 7f1fe559a640 10 monclient: _finish_auth 0 2024-04-26T00:33:36.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.739+0000 7f1fe559a640 10 monclient: _check_auth_tickets 2024-04-26T00:33:36.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.739+0000 7f1fe559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:36.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.739+0000 7f1fe559a640 1 -- 172.21.15.37:0/2354057506 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fe00a66a0 con 0x7f1fe81391c0 2024-04-26T00:33:36.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.739+0000 7f1fecb31640 5 monclient: authenticate success, global_id 154175 2024-04-26T00:33:36.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.739+0000 7f1fecb31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:33:36.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.739+0000 7f1fe559a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:33:36.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.739+0000 7f1fecb31640 10 monclient: get_monmap_and_config success 2024-04-26T00:33:36.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.739+0000 7f1fecb31640 4 set_mon_vals no callback set 2024-04-26T00:33:36.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.739+0000 7f1fe559a640 1 -- 172.21.15.37:0/2354057506 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe00a6a70 con 0x7f1fe81391c0 2024-04-26T00:33:36.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.739+0000 7f1fe559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:36.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.739+0000 7f1fe559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:33:36.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.739+0000 7f1fe559a640 10 monclient: dump: 2024-04-26T00:33:36.749 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:36.749 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:36.749 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:36.750 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:36.750 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:36.750 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:36.750 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:36.750 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:36.750 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:36.750 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:36.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.739+0000 7f1fecb31640 10 monclient: shutdown 2024-04-26T00:33:36.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.739+0000 7f1fecb31640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:33:36.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.739+0000 7f1fecb31640 1 -- 172.21.15.37:0/2354057506 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1fe81391c0 msgr2=0x7f1fe813b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:36.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.739+0000 7f1fecb31640 1 --2- 172.21.15.37:0/2354057506 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1fe81391c0 0x7f1fe813b5b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8071570 tx=0x7f1fe001c870 comp rx=0 tx=0).stop 2024-04-26T00:33:36.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.739+0000 7f1fecb31640 1 -- 172.21.15.37:0/2354057506 shutdown_connections 2024-04-26T00:33:36.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.739+0000 7f1fecb31640 1 --2- 172.21.15.37:0/2354057506 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1fe81391c0 0x7f1fe813b5b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:36.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.739+0000 7f1fecb31640 1 --2- 172.21.15.37:0/2354057506 >> v2:172.21.15.45:3300/0 conn(0x7f1fe8071da0 0x7f1fe80a1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:36.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.739+0000 7f1fecb31640 1 -- 172.21.15.37:0/2354057506 >> 172.21.15.37:0/2354057506 conn(0x7f1fe80a24c0 msgr2=0x7f1fe80a28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:33:36.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.739+0000 7f1fecb31640 1 -- 172.21.15.37:0/2354057506 shutdown_connections 2024-04-26T00:33:36.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.739+0000 7f1fecb31640 1 -- 172.21.15.37:0/2354057506 wait complete. 2024-04-26T00:33:36.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.739+0000 7f1fecb31640 10 monclient: build_initial_monmap 2024-04-26T00:33:36.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.739+0000 7f1fecb31640 10 monclient: monmap: 2024-04-26T00:33:36.751 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:33:36.751 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:33:36.751 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:33:36.751 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:33:36.752 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:33:36.752 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:33:36.752 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:33:36.752 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:33:36.752 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:33:36.752 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:36.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.739+0000 7f1fecb31640 1 Processor -- start 2024-04-26T00:33:36.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.739+0000 7f1fecb31640 1 -- start start 2024-04-26T00:33:36.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.739+0000 7f1fecb31640 10 monclient: init 2024-04-26T00:33:36.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fecb31640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:36.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fecb31640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:33:36.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fecb31640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fe8071da0 0x7f1fe811d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:36.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fecb31640 10 monclient(hunting): picked mon.noname-b con 0x7f1fe8071da0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:33:36.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fecb31640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1fe80a2fe0 0x7f1fe811dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:36.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fecb31640 10 monclient(hunting): picked mon.noname-c con 0x7f1fe80a2fe0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:33:36.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fecb31640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1fe81391c0 0x7f1fe811e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:36.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fecb31640 10 monclient(hunting): picked mon.noname-a con 0x7f1fe81391c0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:33:36.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fecb31640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:36.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fecb31640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe806e6d0 con 0x7f1fe81391c0 2024-04-26T00:33:36.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fecb31640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:36.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fecb31640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe806e550 con 0x7f1fe8071da0 2024-04-26T00:33:36.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fecb31640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:36.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fecb31640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f1fe806e850 con 0x7f1fe80a2fe0 2024-04-26T00:33:36.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fecb31640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:36.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fecb31640 10 monclient(hunting): authenticate will time out at 1798.048379s 2024-04-26T00:33:36.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fe6d9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1fe81391c0 0x7f1fe811e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:36.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fe6d9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1fe81391c0 0x7f1fe811e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:37652/0 (socket says 172.21.15.37:37652) 2024-04-26T00:33:36.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fe6d9d640 1 -- 172.21.15.37:0/4241511752 learned_addr learned my addr 172.21.15.37:0/4241511752 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:33:36.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fe6d9d640 10 monclient(hunting): get_auth_request con 0x7f1fe81391c0 auth_method 0 2024-04-26T00:33:36.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fe6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:36.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fe6d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:36.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fe6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:36.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fe6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:36.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fe5d9b640 1 --2- 172.21.15.37:0/4241511752 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1fe80a2fe0 0x7f1fe811dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:36.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fe6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:36.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fe6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:36.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fe5d9b640 10 monclient(hunting): get_auth_request con 0x7f1fe80a2fe0 auth_method 0 2024-04-26T00:33:36.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fe5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:36.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fe5d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:36.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fe5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:36.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fe659c640 1 --2- 172.21.15.37:0/4241511752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fe8071da0 0x7f1fe811d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:36.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fe659c640 10 monclient(hunting): get_auth_request con 0x7f1fe8071da0 auth_method 0 2024-04-26T00:33:36.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fe659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:36.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fe659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:36.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fe659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:36.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fe5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:36.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fe5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:36.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fe5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:36.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fe6d9d640 10 monclient(hunting): handle_auth_done global_id 144303 payload 824 2024-04-26T00:33:36.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fe6d9d640 1 -- 172.21.15.37:0/4241511752 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1fe80a2fe0 msgr2=0x7f1fe811dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:36.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fe6d9d640 1 --2- 172.21.15.37:0/4241511752 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1fe80a2fe0 0x7f1fe811dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:36.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fe6d9d640 1 -- 172.21.15.37:0/4241511752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fe8071da0 msgr2=0x7f1fe811d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:36.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fe6d9d640 1 --2- 172.21.15.37:0/4241511752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1fe8071da0 0x7f1fe811d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:36.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fe6d9d640 10 monclient: _finish_hunting 0 2024-04-26T00:33:36.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fe6d9d640 1 monclient: found mon.noname-a 2024-04-26T00:33:36.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fe6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:36.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fe6d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:33:36.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fe6d9d640 1 -- 172.21.15.37:0/4241511752 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fe80701d0 con 0x7f1fe81391c0 2024-04-26T00:33:36.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fe6d9d640 1 -- 172.21.15.37:0/4241511752 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1fe81391c0 msgr2=0x7f1fe811e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:36.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fe6d9d640 1 --2- 172.21.15.37:0/4241511752 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1fe81391c0 0x7f1fe811e260 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1fe001d110 tx=0x7f1fe001b0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:36.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fd6ffd640 1 -- 172.21.15.37:0/4241511752 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe00bf070 con 0x7f1fe81391c0 2024-04-26T00:33:36.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fd6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:36.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fd6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:33:36.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fd6ffd640 10 monclient: dump: 2024-04-26T00:33:36.756 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:36.757 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:36.757 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:36.757 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:36.757 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:36.757 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:36.757 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:36.757 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:36.757 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:36.757 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:36.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fd6ffd640 10 monclient: _finish_auth 0 2024-04-26T00:33:36.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fd6ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:33:36.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fd6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:36.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fd6ffd640 1 -- 172.21.15.37:0/4241511752 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fe00a6c60 con 0x7f1fe81391c0 2024-04-26T00:33:36.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fd6ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:33:36.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fd6ffd640 1 -- 172.21.15.37:0/4241511752 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe00af6c0 con 0x7f1fe81391c0 2024-04-26T00:33:36.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fecb31640 5 monclient: authenticate success, global_id 144303 2024-04-26T00:33:36.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fecb31640 10 monclient: _renew_subs 2024-04-26T00:33:36.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fecb31640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:33:36.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fecb31640 1 -- 172.21.15.37:0/4241511752 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fe811f220 con 0x7f1fe81391c0 2024-04-26T00:33:36.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fd6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:36.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fd6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:33:36.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fd6ffd640 10 monclient: dump: 2024-04-26T00:33:36.758 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:36.758 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:36.758 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:36.758 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:36.758 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:36.758 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:36.758 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:36.758 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:36.759 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:36.759 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:36.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fe4d99640 4 set_mon_vals no callback set 2024-04-26T00:33:36.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fecb31640 10 monclient: _renew_subs 2024-04-26T00:33:36.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fecb31640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:33:36.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.743+0000 7f1fecb31640 1 -- 172.21.15.37:0/4241511752 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fe811fb00 con 0x7f1fe81391c0 2024-04-26T00:33:36.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.747+0000 7f1fecb31640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:36.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.747+0000 7f1fecb31640 1 --2- 172.21.15.37:0/4241511752 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1fe813e310 0x7f1fe813e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:36.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.747+0000 7f1fecb31640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:36.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.747+0000 7f1fecb31640 1 -- 172.21.15.37:0/4241511752 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f1fe804f510 con 0x7f1fe813e310 2024-04-26T00:33:36.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.747+0000 7f1fe659c640 1 --2- 172.21.15.37:0/4241511752 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1fe813e310 0x7f1fe813e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:36.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.747+0000 7f1fe659c640 10 monclient: get_auth_request con 0x7f1fe813e310 auth_method 0 2024-04-26T00:33:36.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.747+0000 7f1fe659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:36.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.747+0000 7f1fe659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:36.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.747+0000 7f1fe659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:36.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.747+0000 7f1fd6ffd640 1 -- 172.21.15.37:0/4241511752 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f1fe00af860 con 0x7f1fe81391c0 2024-04-26T00:33:36.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.747+0000 7f1fe659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:36.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.747+0000 7f1fe659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:36.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.747+0000 7f1fe659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:36.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.747+0000 7f1fd6ffd640 1 --2- 172.21.15.37:0/4241511752 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1fc8035ed0 0x7f1fc8038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:36.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.747+0000 7f1fe659c640 10 monclient(hunting): handle_auth_done global_id 154181 payload 824 2024-04-26T00:33:36.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.747+0000 7f1fd6ffd640 1 -- 172.21.15.37:0/4241511752 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(531..531 src has 370..531) ==== 3915+0+0 (secure 0 0 0) 0x7f1fe00e2e60 con 0x7f1fe81391c0 2024-04-26T00:33:36.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.747+0000 7f1fe659c640 1 -- 172.21.15.37:0/4241511752 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1fe80008e0 con 0x7f1fe813e310 2024-04-26T00:33:36.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.747+0000 7f1fe5d9b640 1 --2- 172.21.15.37:0/4241511752 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1fc8035ed0 0x7f1fc8038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:36.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.747+0000 7f1fe5d9b640 10 monclient: get_auth_request con 0x7f1fc8035ed0 auth_method 0 2024-04-26T00:33:36.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.747+0000 7f1fe659c640 1 -- 172.21.15.37:0/4241511752 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1fe813e310 msgr2=0x7f1fe813e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:36.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.747+0000 7f1fe659c640 1 --2- 172.21.15.37:0/4241511752 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1fe813e310 0x7f1fe813e6f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1fd800e500 tx=0x7f1fd800e9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:36.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.747+0000 7f1fd6ffd640 1 -- 172.21.15.37:0/4241511752 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd8004e30 con 0x7f1fe813e310 2024-04-26T00:33:36.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.747+0000 7f1fd6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:36.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.747+0000 7f1fd6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:33:36.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.747+0000 7f1fd6ffd640 10 monclient: dump: 2024-04-26T00:33:36.761 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:36.761 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:36.761 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:36.761 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:36.761 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:36.761 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:36.761 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:36.761 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:36.761 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:36.761 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:36.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.747+0000 7f1fd6ffd640 1 -- 172.21.15.37:0/4241511752 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1fd8018070 con 0x7f1fe813e310 2024-04-26T00:33:36.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.747+0000 7f1fd6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:36.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.747+0000 7f1fd6ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:33:36.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.747+0000 7f1fd6ffd640 1 -- 172.21.15.37:0/4241511752 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1fe813e310 msgr2=0x7f1fe813e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:36.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.747+0000 7f1fd6ffd640 1 --2- 172.21.15.37:0/4241511752 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1fe813e310 0x7f1fe813e6f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1fd800e500 tx=0x7f1fd800e9c0 comp rx=0 tx=0).stop 2024-04-26T00:33:36.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.747+0000 7f1fd6ffd640 1 -- 172.21.15.37:0/4241511752 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1fe813e310 msgr2=0x7f1fe813e6f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:33:36.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.747+0000 7f1fd6ffd640 1 --2- 172.21.15.37:0/4241511752 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1fe813e310 0x7f1fe813e6f0 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1fd800e500 tx=0x7f1fd800e9c0 comp rx=0 tx=0).stop 2024-04-26T00:33:36.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.747+0000 7f1fe5d9b640 1 --2- 172.21.15.37:0/4241511752 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1fc8035ed0 0x7f1fc8038390 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f1fe811ee20 tx=0x7f1fdc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:36.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.755+0000 7f1fecb31640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:33:36.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.755+0000 7f1fecb31640 1 --2- 172.21.15.37:0/4241511752 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1fe813ec30 0x7f1fe81d9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:36.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.755+0000 7f1fecb31640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:36.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.755+0000 7f1fecb31640 1 -- 172.21.15.37:0/4241511752 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f1fe8120c70 con 0x7f1fe813ec30 2024-04-26T00:33:36.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.755+0000 7f1fe659c640 1 --2- 172.21.15.37:0/4241511752 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1fe813ec30 0x7f1fe81d9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:36.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.755+0000 7f1fe659c640 10 monclient: get_auth_request con 0x7f1fe813ec30 auth_method 0 2024-04-26T00:33:36.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.755+0000 7f1fe659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:36.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.755+0000 7f1fe659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:36.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.755+0000 7f1fe659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:36.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.755+0000 7f1fe659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:36.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.755+0000 7f1fe659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:36.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.755+0000 7f1fe659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:36.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.755+0000 7f1fe659c640 10 monclient(hunting): handle_auth_done global_id 154184 payload 824 2024-04-26T00:33:36.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.755+0000 7f1fe659c640 1 -- 172.21.15.37:0/4241511752 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1fe811e7a0 con 0x7f1fe813ec30 2024-04-26T00:33:36.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.759+0000 7f1fe659c640 1 -- 172.21.15.37:0/4241511752 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1fe813ec30 msgr2=0x7f1fe81d9320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:36.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.759+0000 7f1fe659c640 1 --2- 172.21.15.37:0/4241511752 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1fe813ec30 0x7f1fe81d9320 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1fd800e500 tx=0x7f1fd8004940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:36.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.759+0000 7f1fd6ffd640 1 -- 172.21.15.37:0/4241511752 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd80064e0 con 0x7f1fe813ec30 2024-04-26T00:33:36.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.759+0000 7f1fd6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:36.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.759+0000 7f1fd6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:33:36.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.759+0000 7f1fd6ffd640 10 monclient: dump: 2024-04-26T00:33:36.765 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:36.765 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:36.765 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:36.766 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:36.766 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:36.766 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:36.766 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:36.766 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:36.766 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:36.766 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:36.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.759+0000 7f1fd6ffd640 1 -- 172.21.15.37:0/4241511752 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4717 (secure 0 0 0) 0x7f1fd80064e0 con 0x7f1fe813ec30 2024-04-26T00:33:36.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.759+0000 7f1fd6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:33:36.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.759+0000 7f1fd6ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:33:36.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.759+0000 7f1fd6ffd640 1 -- 172.21.15.37:0/4241511752 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1fe813ec30 msgr2=0x7f1fe81d9320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:36.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.759+0000 7f1fd6ffd640 1 --2- 172.21.15.37:0/4241511752 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1fe813ec30 0x7f1fe81d9320 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1fd800e500 tx=0x7f1fd8004940 comp rx=0 tx=0).stop 2024-04-26T00:33:36.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.759+0000 7f1fd6ffd640 1 -- 172.21.15.37:0/4241511752 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1fe813ec30 msgr2=0x7f1fe81d9320 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:33:36.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.759+0000 7f1fd6ffd640 1 --2- 172.21.15.37:0/4241511752 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1fe813ec30 0x7f1fe81d9320 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1fd800e500 tx=0x7f1fd8004940 comp rx=0 tx=0).stop 2024-04-26T00:33:36.768 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:33:36.768 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:33:36.768 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:33:36.769 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:33:36.769 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 102, 2024-04-26T00:33:36.769 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 54279, 2024-04-26T00:33:36.769 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:33:36.769 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:33:36.769 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:33:36.769 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:33:36.770 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:36.770 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:33:36.770 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:33:36.770 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:33:36.770 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:33:36.770 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:33:36.771 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:33:36.771 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:33:36.771 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:33:36.771 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:33:36.771 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:33:36.771 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:33:36.772 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:33:36.772 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:33:36.772 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:33:36.772 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:33:36.772 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:36.772 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:33:36.772 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:33:36.773 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:33:36.773 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:33:36.773 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:33:36.773 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:33:36.773 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:33:36.773 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:33:36.773 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:33:36.773 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:33:36.774 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:33:36.774 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:33:36.774 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:33:36.774 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:36.774 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:36.774 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:33:36.774 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:33:36.775 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:33:36.775 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:33:36.775 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:33:36.775 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:33:36.775 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:33:36.775 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:33:36.775 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:33:36.776 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:33:36.776 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:33:36.776 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:33:36.776 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:33:36.776 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:33:36.776 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:33:36.776 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:33:36.777 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:33:36.777 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:33:36.777 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:33:36.777 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:33:36.777 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:33:36.777 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:33:36.777 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:33:36.778 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:33:36.778 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:33:36.778 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:33:36.778 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:33:36.778 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:33:36.778 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:36.778 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:33:36.778 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:36.779 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:33:36.779 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:36.779 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:33:36.779 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:33:36.779 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:33:36.779 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:33:36.779 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:36.780 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:33:36.780 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:33:36.780 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:36.780 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:36.780 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:36.780 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:33:36.780 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:33:36.781 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:36.781 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:36.781 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:36.781 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:36.781 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:33:36.781 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:33:36.781 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:33:36.781 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:33:36.782 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:33:36.782 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:36.782 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:36.782 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:33:36.782 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:33:36.782 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:33:36.782 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:33:36.783 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:36.783 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:33:36.783 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:33:36.783 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:36.783 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:36.783 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:36.783 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:33:36.784 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:33:36.784 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:36.784 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:36.784 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:36.784 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:36.784 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:33:36.784 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:33:36.785 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:33:36.785 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:33:36.785 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:33:36.785 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:36.785 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:36.785 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:33:36.785 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:33:36.785 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:33:36.786 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:33:36.786 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:36.786 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:33:36.786 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:33:36.786 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:36.787 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:36.787 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:36.787 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:33:36.787 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:33:36.787 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:36.789 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:36.789 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:36.789 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:36.789 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:33:36.790 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:33:36.790 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:33:36.790 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:33:36.790 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:33:36.790 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:36.790 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:36.790 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:36.790 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:33:36.790 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:33:36.791 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:36.791 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:33:36.791 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:33:36.791 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:33:36.791 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:36.791 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:36.791 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:33:36.791 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:36.792 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:33:36.792 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:33:36.792 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:33:36.792 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:36.792 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:36.792 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:33:36.792 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:36.793 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:33:36.793 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:33:36.793 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:33:36.793 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:36.793 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:36.793 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:36.793 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:33:36.793 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:33:36.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.759+0000 7f1fecb31640 1 -- 172.21.15.37:0/4241511752 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1fc8035ed0 msgr2=0x7f1fc8038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:36.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.759+0000 7f1fecb31640 1 --2- 172.21.15.37:0/4241511752 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1fc8035ed0 0x7f1fc8038390 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f1fe811ee20 tx=0x7f1fdc008040 comp rx=0 tx=0).stop 2024-04-26T00:33:36.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.759+0000 7f1fecb31640 10 monclient: shutdown 2024-04-26T00:33:36.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.759+0000 7f1fecb31640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:33:36.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.759+0000 7f1fecb31640 1 -- 172.21.15.37:0/4241511752 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1fe81391c0 msgr2=0x7f1fe811e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:36.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.759+0000 7f1fecb31640 1 --2- 172.21.15.37:0/4241511752 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1fe81391c0 0x7f1fe811e260 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1fe001d110 tx=0x7f1fe001b0b0 comp rx=0 tx=0).stop 2024-04-26T00:33:36.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.759+0000 7f1fe6d9d640 1 -- 172.21.15.37:0/4241511752 reap_dead start 2024-04-26T00:33:36.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.759+0000 7f1fe6d9d640 1 -- 172.21.15.37:0/4241511752 reap_dead start 2024-04-26T00:33:36.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.759+0000 7f1fecb31640 1 -- 172.21.15.37:0/4241511752 shutdown_connections 2024-04-26T00:33:36.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.759+0000 7f1fecb31640 1 -- 172.21.15.37:0/4241511752 >> 172.21.15.37:0/4241511752 conn(0x7f1fe80a24c0 msgr2=0x7f1fe81389b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:33:36.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.759+0000 7f1fecb31640 1 -- 172.21.15.37:0/4241511752 shutdown_connections 2024-04-26T00:33:36.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.759+0000 7f1fecb31640 1 -- 172.21.15.37:0/4241511752 wait complete. 2024-04-26T00:33:36.798 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:33:36.798 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:33:36.798 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:33:36.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f9392fcf640 10 monclient: get_monmap_and_config 2024-04-26T00:33:36.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f9392fcf640 10 monclient: build_initial_monmap 2024-04-26T00:33:36.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f9392fcf640 10 monclient: monmap: 2024-04-26T00:33:36.921 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:33:36.921 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:36.921 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:33:36.929368+0000 2024-04-26T00:33:36.921 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:33:36.929368+0000 2024-04-26T00:33:36.921 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:33:36.921 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:36.921 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:33:36.921 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:33:36.921 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:33:36.921 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:33:36.921 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:33:36.921 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:36.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f9392fcf640 1 Processor -- start 2024-04-26T00:33:36.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f9392fcf640 1 -- start start 2024-04-26T00:33:36.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f9392fcf640 10 monclient: init 2024-04-26T00:33:36.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f9392fcf640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:36.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f9392fcf640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-04-26T00:33:36.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f9392fcf640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f938c0a0300 0x7f938c09fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:36.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f9392fcf640 10 monclient(hunting): picked mon.noname-a con 0x7f938c0a0300 addr v2:172.21.15.37:3300/0 2024-04-26T00:33:36.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f9392fcf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f938c0a06e0 addr v1:172.21.15.37:6789/0 2024-04-26T00:33:36.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f9392fcf640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f938c1352b0 0x7f938c13cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:36.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f9392fcf640 10 monclient(hunting): picked mon.noname-c con 0x7f938c1352b0 addr v2:172.21.15.45:3300/0 2024-04-26T00:33:36.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f9392fcf640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f938c09c190 con 0x7f938c0a06e0 2024-04-26T00:33:36.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f9392fcf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:36.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f9392fcf640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f938c0a13f0 con 0x7f938c0a0300 2024-04-26T00:33:36.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f9392fcf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:36.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f9392fcf640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f938c0a1110 con 0x7f938c1352b0 2024-04-26T00:33:36.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f9392fcf640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:36.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f9392fcf640 10 monclient(hunting): authenticate will time out at 1798.234999s 2024-04-26T00:33:36.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f938bfff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f938c0a06e0 0x7f938c134ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:35146/0 (socket says 172.21.15.37:35146) 2024-04-26T00:33:36.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f938bfff640 1 -- 172.21.15.37:0/974418846 learned_addr learned my addr 172.21.15.37:0/974418846 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:33:36.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f9390d6b640 1 --2- 172.21.15.37:0/974418846 >> v2:172.21.15.37:3300/0 conn(0x7f938c0a0300 0x7f938c09fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:36.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f9390d6b640 10 monclient(hunting): get_auth_request con 0x7f938c0a0300 auth_method 0 2024-04-26T00:33:36.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f9390d6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:36.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f9390d6b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:36.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f9390d6b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:36.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f9390d6b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:36.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f9390d6b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:36.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f9390d6b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:36.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f939156c640 1 --2- 172.21.15.37:0/974418846 >> v2:172.21.15.45:3300/0 conn(0x7f938c1352b0 0x7f938c13cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:36.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f939156c640 10 monclient(hunting): get_auth_request con 0x7f938c1352b0 auth_method 0 2024-04-26T00:33:36.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f939156c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:36.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f939156c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:36.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f939156c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:36.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f9390d6b640 10 monclient(hunting): handle_auth_done global_id 144306 payload 274 2024-04-26T00:33:36.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f9390d6b640 1 -- 172.21.15.37:0/974418846 >> v2:172.21.15.45:3300/0 conn(0x7f938c1352b0 msgr2=0x7f938c13cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:33:36.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f9390d6b640 1 --2- 172.21.15.37:0/974418846 >> v2:172.21.15.45:3300/0 conn(0x7f938c1352b0 0x7f938c13cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:36.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f9390d6b640 1 -- 172.21.15.37:0/974418846 >> v1:172.21.15.37:6789/0 conn(0x7f938c0a06e0 legacy=0x7f938c134ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:33:36.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f9390d6b640 10 monclient: _finish_hunting 0 2024-04-26T00:33:36.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f9390d6b640 1 monclient: found mon.noname-a 2024-04-26T00:33:36.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f9390d6b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:36.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f9390d6b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:33:36.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.927+0000 7f9390d6b640 1 -- 172.21.15.37:0/974418846 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f938c0a16d0 con 0x7f938c0a0300 2024-04-26T00:33:36.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9390d6b640 1 -- 172.21.15.37:0/974418846 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f938c0a0300 msgr2=0x7f938c09fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:36.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9390d6b640 1 --2- 172.21.15.37:0/974418846 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f938c0a0300 0x7f938c09fc50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f938c06de30 tx=0x7f937c01c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abff5bce63cdf2cf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:36.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f938b7fe640 1 -- 172.21.15.37:0/974418846 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f937c0b2070 con 0x7f938c0a0300 2024-04-26T00:33:36.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f938b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:36.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f938b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:33:36.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f938b7fe640 10 monclient: dump: 2024-04-26T00:33:36.929 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:36.929 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:36.929 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:36.929 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:36.929 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:36.930 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:36.930 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:36.930 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:36.930 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:36.930 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:36.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f938b7fe640 10 monclient: _finish_auth 0 2024-04-26T00:33:36.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f938b7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:33:36.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f938b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:36.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f938b7fe640 1 -- 172.21.15.37:0/974418846 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f937c0a5850 con 0x7f938c0a0300 2024-04-26T00:33:36.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9392fcf640 5 monclient: authenticate success, global_id 144306 2024-04-26T00:33:36.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9392fcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:33:36.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f938b7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:33:36.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9392fcf640 10 monclient: get_monmap_and_config success 2024-04-26T00:33:36.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f938b7fe640 1 -- 172.21.15.37:0/974418846 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f937c0a5c20 con 0x7f938c0a0300 2024-04-26T00:33:36.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9392fcf640 4 set_mon_vals no callback set 2024-04-26T00:33:36.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f938b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:36.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f938b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:33:36.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f938b7fe640 10 monclient: dump: 2024-04-26T00:33:36.931 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:36.931 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:36.931 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:36.931 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:36.931 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:36.931 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:36.931 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:36.931 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:36.931 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:36.931 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:36.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9392fcf640 10 monclient: shutdown 2024-04-26T00:33:36.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9392fcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:33:36.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9392fcf640 1 -- 172.21.15.37:0/974418846 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f938c0a0300 msgr2=0x7f938c09fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:36.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9392fcf640 1 --2- 172.21.15.37:0/974418846 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f938c0a0300 0x7f938c09fc50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f938c06de30 tx=0x7f937c01c7b0 comp rx=0 tx=0).stop 2024-04-26T00:33:36.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9392fcf640 1 -- 172.21.15.37:0/974418846 shutdown_connections 2024-04-26T00:33:36.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9392fcf640 1 --2- 172.21.15.37:0/974418846 >> v2:172.21.15.45:3300/0 conn(0x7f938c1352b0 0x7f938c13cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:36.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9392fcf640 1 --2- 172.21.15.37:0/974418846 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f938c0a0300 0x7f938c09fc50 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:36.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9392fcf640 1 -- 172.21.15.37:0/974418846 >> 172.21.15.37:0/974418846 conn(0x7f938c0a1d50 msgr2=0x7f938c0a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:33:36.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9392fcf640 1 -- 172.21.15.37:0/974418846 shutdown_connections 2024-04-26T00:33:36.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9392fcf640 1 -- 172.21.15.37:0/974418846 wait complete. 2024-04-26T00:33:36.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9392fcf640 10 monclient: build_initial_monmap 2024-04-26T00:33:36.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9392fcf640 10 monclient: monmap: 2024-04-26T00:33:36.932 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:33:36.933 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:33:36.933 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:33:36.933 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:33:36.933 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:33:36.933 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:33:36.933 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:33:36.933 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:33:36.933 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:33:36.933 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:36.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9392fcf640 1 Processor -- start 2024-04-26T00:33:36.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9392fcf640 1 -- start start 2024-04-26T00:33:36.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9392fcf640 10 monclient: init 2024-04-26T00:33:36.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9392fcf640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:36.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9392fcf640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:33:36.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9392fcf640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f938c0a0300 0x7f938c1cf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:36.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9392fcf640 10 monclient(hunting): picked mon.noname-c con 0x7f938c0a0300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:33:36.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9392fcf640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f938c0a06e0 0x7f938c12fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:36.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9392fcf640 10 monclient(hunting): picked mon.noname-a con 0x7f938c0a06e0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:33:36.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9392fcf640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f938c1352b0 0x7f938c139ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:36.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9392fcf640 10 monclient(hunting): picked mon.noname-b con 0x7f938c1352b0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:33:36.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9392fcf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:36.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9392fcf640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f938c13fdc0 con 0x7f938c0a06e0 2024-04-26T00:33:36.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9392fcf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:36.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9392fcf640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f938c13fc40 con 0x7f938c1352b0 2024-04-26T00:33:36.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9392fcf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:36.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9392fcf640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f938c13ff40 con 0x7f938c0a0300 2024-04-26T00:33:36.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9392fcf640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:36.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9392fcf640 10 monclient(hunting): authenticate will time out at 1798.242571s 2024-04-26T00:33:36.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f938bfff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f938c0a06e0 0x7f938c12fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:36.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9390d6b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f938c0a0300 0x7f938c1cf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:36.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9390d6b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f938c0a0300 0x7f938c1cf1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:48400/0 (socket says 172.21.15.37:48400) 2024-04-26T00:33:36.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f938bfff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f938c0a06e0 0x7f938c12fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:37674/0 (socket says 172.21.15.37:37674) 2024-04-26T00:33:36.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9390d6b640 1 -- 172.21.15.37:0/1135351504 learned_addr learned my addr 172.21.15.37:0/1135351504 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:33:36.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9390d6b640 10 monclient(hunting): get_auth_request con 0x7f938c0a0300 auth_method 0 2024-04-26T00:33:36.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9390d6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:36.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9390d6b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:36.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9390d6b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:36.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f938bfff640 10 monclient(hunting): get_auth_request con 0x7f938c0a06e0 auth_method 0 2024-04-26T00:33:36.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f938bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:36.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f938bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:36.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f938bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:36.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f938bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:36.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f938bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:36.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f938bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:36.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9390d6b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:36.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9390d6b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:36.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9390d6b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:36.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f939156c640 1 --2- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f938c1352b0 0x7f938c139ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:36.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f939156c640 10 monclient(hunting): get_auth_request con 0x7f938c1352b0 auth_method 0 2024-04-26T00:33:36.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f939156c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:36.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f939156c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:36.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f939156c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:36.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f938bfff640 10 monclient(hunting): handle_auth_done global_id 144309 payload 824 2024-04-26T00:33:36.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f938bfff640 1 -- 172.21.15.37:0/1135351504 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f938c0a0300 msgr2=0x7f938c1cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:36.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f938bfff640 1 --2- 172.21.15.37:0/1135351504 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f938c0a0300 0x7f938c1cf1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:36.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f938bfff640 1 -- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f938c1352b0 msgr2=0x7f938c139ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:36.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f938bfff640 1 --2- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f938c1352b0 0x7f938c139ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:36.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f938bfff640 10 monclient: _finish_hunting 0 2024-04-26T00:33:36.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f938bfff640 1 monclient: found mon.noname-a 2024-04-26T00:33:36.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f938bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:36.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f938bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:33:36.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f938bfff640 1 -- 172.21.15.37:0/1135351504 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f938c1309c0 con 0x7f938c0a06e0 2024-04-26T00:33:36.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.935+0000 7f9390d6b640 1 --2- 172.21.15.37:0/1135351504 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f938c0a0300 0x7f938c1cf1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:33:36.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f938bfff640 1 -- 172.21.15.37:0/1135351504 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f938c0a06e0 msgr2=0x7f938c12fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:36.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f938bfff640 1 --2- 172.21.15.37:0/1135351504 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f938c0a06e0 0x7f938c12fb00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f938000c410 tx=0x7f938000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:36.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f93897fa640 1 -- 172.21.15.37:0/1135351504 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9380017070 con 0x7f938c0a06e0 2024-04-26T00:33:36.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f93897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:36.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f93897fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:33:36.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f93897fa640 10 monclient: dump: 2024-04-26T00:33:36.941 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:36.941 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:36.941 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:36.941 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:36.941 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:36.941 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:36.942 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:36.942 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:36.942 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:36.942 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:36.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f93897fa640 10 monclient: _finish_auth 0 2024-04-26T00:33:36.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f93897fa640 10 monclient: _check_auth_tickets 2024-04-26T00:33:36.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f93897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:36.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f93897fa640 1 -- 172.21.15.37:0/1135351504 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93800070d0 con 0x7f938c0a06e0 2024-04-26T00:33:36.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f9392fcf640 5 monclient: authenticate success, global_id 144309 2024-04-26T00:33:36.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f93897fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:33:36.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f93897fa640 1 -- 172.21.15.37:0/1135351504 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93800074c0 con 0x7f938c0a06e0 2024-04-26T00:33:36.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f9392fcf640 10 monclient: _renew_subs 2024-04-26T00:33:36.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f9392fcf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:33:36.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f9392fcf640 1 -- 172.21.15.37:0/1135351504 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f938c11bc10 con 0x7f938c0a06e0 2024-04-26T00:33:36.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f938affd640 4 set_mon_vals no callback set 2024-04-26T00:33:36.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f93897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:36.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f93897fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:33:36.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f93897fa640 10 monclient: dump: 2024-04-26T00:33:36.943 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:36.943 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:36.943 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:36.943 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:36.943 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:36.943 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:36.943 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:36.943 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:36.943 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:36.943 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:36.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f9392fcf640 10 monclient: _renew_subs 2024-04-26T00:33:36.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f9392fcf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:33:36.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f9392fcf640 1 -- 172.21.15.37:0/1135351504 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f938c1d0070 con 0x7f938c0a06e0 2024-04-26T00:33:36.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f9392fcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:36.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f9392fcf640 1 --2- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f938c130040 0x7f938c13a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:36.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f9392fcf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:36.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f9392fcf640 1 -- 172.21.15.37:0/1135351504 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f938c11c410 con 0x7f938c130040 2024-04-26T00:33:36.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f93897fa640 1 -- 172.21.15.37:0/1135351504 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f938001a070 con 0x7f938c0a06e0 2024-04-26T00:33:36.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f9390d6b640 1 --2- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f938c130040 0x7f938c13a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:36.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f93897fa640 1 --2- 172.21.15.37:0/1135351504 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9378035ed0 0x7f9378038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:36.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f9390d6b640 10 monclient: get_auth_request con 0x7f938c130040 auth_method 0 2024-04-26T00:33:36.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f9390d6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:36.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f9390d6b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:36.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f9390d6b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:36.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f93897fa640 1 -- 172.21.15.37:0/1135351504 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(531..531 src has 370..531) ==== 3915+0+0 (secure 0 0 0) 0x7f9380047f10 con 0x7f938c0a06e0 2024-04-26T00:33:36.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f9390d6b640 1 -- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f938c130040 msgr2=0x7f938c13a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:33:36.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f9390d6b640 1 -- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f938c130040 msgr2=0x7f938c13a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:33:36.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f9390d6b640 1 --2- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f938c130040 0x7f938c13a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:33:36.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f9390d6b640 1 --2- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f938c130040 0x7f938c13a0e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:33:36.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f939156c640 1 --2- 172.21.15.37:0/1135351504 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9378035ed0 0x7f9378038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:36.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.939+0000 7f939156c640 10 monclient: get_auth_request con 0x7f9378035ed0 auth_method 0 2024-04-26T00:33:36.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:36.943+0000 7f939156c640 1 --2- 172.21.15.37:0/1135351504 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9378035ed0 0x7f9378038390 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f938c09f6b0 tx=0x7f9384008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:37.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.143+0000 7f9390d6b640 1 --2- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f938c130040 0x7f938c13a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:37.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.143+0000 7f9390d6b640 10 monclient: get_auth_request con 0x7f938c130040 auth_method 0 2024-04-26T00:33:37.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.143+0000 7f9390d6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:37.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.143+0000 7f9390d6b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:37.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.143+0000 7f9390d6b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:33:37.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.143+0000 7f9390d6b640 1 -- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f938c130040 msgr2=0x7f938c13a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:33:37.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.143+0000 7f9390d6b640 1 -- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f938c130040 msgr2=0x7f938c13a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:33:37.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.143+0000 7f9390d6b640 1 --2- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f938c130040 0x7f938c13a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:33:37.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.143+0000 7f9390d6b640 1 --2- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f938c130040 0x7f938c13a0e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:33:37.530 INFO:tasks.rados.rados.0.smithi045.stdout:2362: done (2 left) 2024-04-26T00:33:37.530 INFO:tasks.rados.rados.0.smithi045.stdout:2363: snap_remove snap 246 2024-04-26T00:33:37.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.547+0000 7f9390d6b640 1 --2- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f938c130040 0x7f938c13a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:37.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.547+0000 7f9390d6b640 10 monclient: get_auth_request con 0x7f938c130040 auth_method 0 2024-04-26T00:33:37.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.547+0000 7f9390d6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:37.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.547+0000 7f9390d6b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:37.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.547+0000 7f9390d6b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:33:37.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.547+0000 7f9390d6b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:37.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.547+0000 7f9390d6b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:37.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.547+0000 7f9390d6b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:37.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.547+0000 7f9390d6b640 10 monclient(hunting): handle_auth_done global_id 164098 payload 824 2024-04-26T00:33:37.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.547+0000 7f9390d6b640 1 -- 172.21.15.37:0/1135351504 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f938c04f510 con 0x7f938c130040 2024-04-26T00:33:37.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.551+0000 7f9390d6b640 1 -- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f938c130040 msgr2=0x7f938c13a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:37.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.551+0000 7f9390d6b640 1 --2- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f938c130040 0x7f938c13a0e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f937c01ae60 tx=0x7f937c019550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:37.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.551+0000 7f93897fa640 1 -- 172.21.15.37:0/1135351504 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f937c0c6070 con 0x7f938c130040 2024-04-26T00:33:37.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.551+0000 7f93897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:37.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.551+0000 7f93897fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:33:37.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.551+0000 7f93897fa640 10 monclient: dump: 2024-04-26T00:33:37.546 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:37.546 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:37.546 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:37.546 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:37.546 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:37.546 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:37.546 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:37.546 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:37.546 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:37.546 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:37.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.551+0000 7f93897fa640 1 -- 172.21.15.37:0/1135351504 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f937c0b3040 con 0x7f938c130040 2024-04-26T00:33:37.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.551+0000 7f93897fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:37.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.551+0000 7f93897fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:33:37.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.551+0000 7f93897fa640 1 -- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f938c130040 msgr2=0x7f938c13a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:37.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.551+0000 7f93897fa640 1 --2- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f938c130040 0x7f938c13a0e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f937c01ae60 tx=0x7f937c019550 comp rx=0 tx=0).stop 2024-04-26T00:33:37.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.551+0000 7f93897fa640 1 -- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f938c130040 msgr2=0x7f938c13a0e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:33:37.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.551+0000 7f93897fa640 1 --2- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f938c130040 0x7f938c13a0e0 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f937c01ae60 tx=0x7f937c019550 comp rx=0 tx=0).stop 2024-04-26T00:33:37.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.563+0000 7f9392fcf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:33:37.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.563+0000 7f9392fcf640 1 --2- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f938c13a620 0x7f938c135c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:37.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.563+0000 7f9392fcf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:37.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.563+0000 7f9392fcf640 1 -- 172.21.15.37:0/1135351504 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f938c13aa30 con 0x7f938c13a620 2024-04-26T00:33:37.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.563+0000 7f9390d6b640 1 --2- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f938c13a620 0x7f938c135c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:37.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.563+0000 7f9390d6b640 10 monclient: get_auth_request con 0x7f938c13a620 auth_method 0 2024-04-26T00:33:37.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.563+0000 7f9390d6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:37.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.563+0000 7f9390d6b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:37.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.563+0000 7f9390d6b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:37.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.567+0000 7f9390d6b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:37.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.567+0000 7f9390d6b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:37.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.567+0000 7f9390d6b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:37.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.567+0000 7f9390d6b640 10 monclient(hunting): handle_auth_done global_id 164101 payload 824 2024-04-26T00:33:37.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.567+0000 7f9390d6b640 1 -- 172.21.15.37:0/1135351504 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f938c13b840 con 0x7f938c13a620 2024-04-26T00:33:37.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.567+0000 7f9390d6b640 1 -- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f938c13a620 msgr2=0x7f938c135c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:37.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.567+0000 7f9390d6b640 1 --2- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f938c13a620 0x7f938c135c90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f937c009bd0 tx=0x7f937c0a7000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:37.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.567+0000 7f9390d6b640 1 -- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f938c13a620 msgr2=0x7f938c135c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:33:37.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.567+0000 7f9390d6b640 1 -- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f938c13a620 msgr2=0x7f938c135c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:33:37.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.567+0000 7f9390d6b640 1 --2- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f938c13a620 0x7f938c135c90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f937c009bd0 tx=0x7f937c0a7000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:33:37.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.567+0000 7f9390d6b640 1 --2- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f938c13a620 0x7f938c135c90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f937c009bd0 tx=0x7f937c0a7000 comp rx=0 tx=0).stop 2024-04-26T00:33:37.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.567+0000 7f93897fa640 1 -- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f938c13a620 msgr2=0x7f938c135c90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:33:37.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.567+0000 7f93897fa640 1 --2- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f938c13a620 0x7f938c135c90 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:37.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.567+0000 7f93897fa640 1 --2- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f937803be20 0x7f937803e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:37.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.567+0000 7f93897fa640 1 -- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f938c13a620 msgr2=0x7f938c135c90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:33:37.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.567+0000 7f93897fa640 1 --2- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f938c13a620 0x7f938c135c90 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:37.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.567+0000 7f93897fa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:37.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.567+0000 7f93897fa640 1 -- 172.21.15.37:0/1135351504 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f937803e820 con 0x7f937803be20 2024-04-26T00:33:37.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.571+0000 7f9390d6b640 1 --2- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f937803be20 0x7f937803e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:37.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.571+0000 7f9390d6b640 10 monclient: get_auth_request con 0x7f937803be20 auth_method 0 2024-04-26T00:33:37.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.571+0000 7f9390d6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:37.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.571+0000 7f9390d6b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:37.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.571+0000 7f9390d6b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:37.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.571+0000 7f9390d6b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:37.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.571+0000 7f9390d6b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:37.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.571+0000 7f9390d6b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:37.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.571+0000 7f9390d6b640 10 monclient(hunting): handle_auth_done global_id 164104 payload 824 2024-04-26T00:33:37.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.571+0000 7f9390d6b640 1 -- 172.21.15.37:0/1135351504 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f937c0b3040 con 0x7f937803be20 2024-04-26T00:33:37.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.575+0000 7f9390d6b640 1 -- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f937803be20 msgr2=0x7f937803e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:37.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.575+0000 7f9390d6b640 1 --2- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f937803be20 0x7f937803e210 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f937c0070b0 tx=0x7f937c0a7070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:37.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.575+0000 7f93897fa640 1 -- 172.21.15.37:0/1135351504 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f937c0cc070 con 0x7f937803be20 2024-04-26T00:33:37.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.575+0000 7f93897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:37.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.575+0000 7f93897fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:33:37.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.575+0000 7f93897fa640 10 monclient: dump: 2024-04-26T00:33:37.571 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:37.571 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:37.571 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:37.571 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:37.571 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:37.571 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:37.571 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:37.571 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:37.572 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:37.572 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:37.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.575+0000 7f93897fa640 1 -- 172.21.15.37:0/1135351504 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7f937c0db440 con 0x7f937803be20 2024-04-26T00:33:37.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.575+0000 7f93897fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:33:37.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.575+0000 7f93897fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:33:37.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.575+0000 7f93897fa640 1 -- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f937803be20 msgr2=0x7f937803e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:37.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.575+0000 7f93897fa640 1 --2- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f937803be20 0x7f937803e210 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f937c0070b0 tx=0x7f937c0a7070 comp rx=0 tx=0).stop 2024-04-26T00:33:37.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.575+0000 7f93897fa640 1 -- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f937803be20 msgr2=0x7f937803e210 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:33:37.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.575+0000 7f93897fa640 1 --2- 172.21.15.37:0/1135351504 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f937803be20 0x7f937803e210 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f937c0070b0 tx=0x7f937c0a7070 comp rx=0 tx=0).stop 2024-04-26T00:33:37.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.575+0000 7f939156c640 1 -- 172.21.15.37:0/1135351504 reap_dead start 2024-04-26T00:33:37.572 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:33:37.573 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:33:37.573 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:33:37.573 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:33:37.573 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 102, 2024-04-26T00:33:37.573 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 5071, 2024-04-26T00:33:37.573 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:33:37.573 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:33:37.573 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:33:37.573 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:33:37.573 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:37.573 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 1, 2024-04-26T00:33:37.573 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:33:37.573 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:33:37.573 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:33:37.573 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:33:37.574 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:33:37.574 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:33:37.574 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:33:37.574 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:33:37.574 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:33:37.574 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:33:37.574 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:33:37.574 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:33:37.574 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:33:37.574 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:33:37.574 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:37.574 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:33:37.575 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:33:37.575 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:33:37.575 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:33:37.575 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:33:37.575 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:33:37.575 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:33:37.575 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:33:37.575 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:33:37.575 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:33:37.575 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:33:37.575 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:33:37.575 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:33:37.575 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:37.576 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:37.576 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:33:37.576 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:33:37.576 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:33:37.576 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:33:37.576 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:33:37.576 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:33:37.576 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:33:37.576 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:33:37.576 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:33:37.576 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:33:37.576 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:33:37.576 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:33:37.576 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:33:37.576 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:33:37.577 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:33:37.577 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:33:37.577 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:33:37.577 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:33:37.577 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:33:37.577 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:33:37.577 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:33:37.577 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:33:37.577 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:33:37.577 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:33:37.577 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:33:37.577 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:33:37.577 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:33:37.577 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:33:37.577 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:37.578 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:33:37.578 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:37.578 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:33:37.578 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:37.578 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:33:37.578 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:33:37.578 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:33:37.578 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:33:37.578 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:37.578 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:33:37.578 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:33:37.578 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:37.578 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:37.578 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:37.578 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:33:37.579 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:33:37.579 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:37.579 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:37.579 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:37.579 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:37.579 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:33:37.579 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:33:37.579 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:33:37.579 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:33:37.579 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:33:37.579 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:37.579 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:37.579 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:33:37.579 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:33:37.580 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:33:37.580 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:33:37.580 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:37.580 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:33:37.580 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:33:37.580 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:37.580 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:37.580 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:37.580 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:33:37.580 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:33:37.580 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:37.580 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:37.580 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:37.580 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:37.580 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:33:37.581 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:33:37.581 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:33:37.581 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:33:37.581 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:33:37.581 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:37.581 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:37.581 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:33:37.581 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:33:37.581 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:33:37.581 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:33:37.581 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:37.581 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:33:37.581 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:33:37.581 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:37.581 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:37.581 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:37.582 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:33:37.582 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:33:37.582 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:37.582 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:37.582 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:37.582 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:37.582 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:33:37.582 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:33:37.582 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:33:37.582 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:33:37.582 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:33:37.582 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:37.582 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:37.582 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:37.583 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:33:37.583 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:33:37.583 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:37.583 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:33:37.583 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:33:37.583 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:33:37.583 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:37.583 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:37.583 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:33:37.583 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:37.583 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:33:37.583 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:33:37.583 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:33:37.583 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:37.583 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:37.584 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:37.584 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:33:37.584 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:33:37.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.579+0000 7f9392fcf640 1 -- 172.21.15.37:0/1135351504 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9378035ed0 msgr2=0x7f9378038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:37.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.579+0000 7f9392fcf640 1 --2- 172.21.15.37:0/1135351504 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9378035ed0 0x7f9378038390 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f938c09f6b0 tx=0x7f9384008040 comp rx=0 tx=0).stop 2024-04-26T00:33:37.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.579+0000 7f9392fcf640 10 monclient: shutdown 2024-04-26T00:33:37.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.579+0000 7f9392fcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:33:37.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.579+0000 7f9392fcf640 1 -- 172.21.15.37:0/1135351504 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f938c0a06e0 msgr2=0x7f938c12fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:37.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.579+0000 7f9392fcf640 1 --2- 172.21.15.37:0/1135351504 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f938c0a06e0 0x7f938c12fb00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f938000c410 tx=0x7f938000c8d0 comp rx=0 tx=0).stop 2024-04-26T00:33:37.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.579+0000 7f9392fcf640 1 -- 172.21.15.37:0/1135351504 shutdown_connections 2024-04-26T00:33:37.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.579+0000 7f9392fcf640 1 --2- 172.21.15.37:0/1135351504 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9378035ed0 0x7f9378038390 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:37.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.579+0000 7f9392fcf640 1 --2- 172.21.15.37:0/1135351504 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f938c0a06e0 0x7f938c12fb00 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:37.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.579+0000 7f9392fcf640 1 -- 172.21.15.37:0/1135351504 >> 172.21.15.37:0/1135351504 conn(0x7f938c0a1d50 msgr2=0x7f938c0a3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:33:37.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.579+0000 7f9392fcf640 1 -- 172.21.15.37:0/1135351504 shutdown_connections 2024-04-26T00:33:37.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.579+0000 7f9392fcf640 1 -- 172.21.15.37:0/1135351504 wait complete. 2024-04-26T00:33:37.593 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:33:37.593 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-26T00:33:37.594 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-26T00:33:37.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.719+0000 7f785c4a8640 10 monclient: get_monmap_and_config 2024-04-26T00:33:37.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.719+0000 7f785c4a8640 10 monclient: build_initial_monmap 2024-04-26T00:33:37.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.719+0000 7f785c4a8640 10 monclient: monmap: 2024-04-26T00:33:37.714 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:33:37.714 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:37.714 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:33:37.722207+0000 2024-04-26T00:33:37.714 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:33:37.722207+0000 2024-04-26T00:33:37.714 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:33:37.714 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:37.714 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:33:37.714 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:33:37.714 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:33:37.714 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:33:37.714 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:33:37.715 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:37.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.719+0000 7f785c4a8640 1 Processor -- start 2024-04-26T00:33:37.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.719+0000 7f785c4a8640 1 -- start start 2024-04-26T00:33:37.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.719+0000 7f785c4a8640 10 monclient: init 2024-04-26T00:33:37.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.719+0000 7f785c4a8640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:37.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.719+0000 7f785c4a8640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-04-26T00:33:37.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.719+0000 7f785c4a8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f78540a0260 addr v1:172.21.15.45:6789/0 2024-04-26T00:33:37.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.719+0000 7f785c4a8640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7854137e40 0x7f785413a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:37.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.719+0000 7f785c4a8640 10 monclient(hunting): picked mon.noname-c con 0x7f7854137e40 addr v2:172.21.15.45:3300/0 2024-04-26T00:33:37.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.719+0000 7f785c4a8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f785413a770 0x7f785413cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:37.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.719+0000 7f785c4a8640 10 monclient(hunting): picked mon.noname-b con 0x7f785413a770 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:33:37.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.719+0000 7f785c4a8640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78540a1130 con 0x7f78540a0260 2024-04-26T00:33:37.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.719+0000 7f785c4a8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:37.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.719+0000 7f785c4a8640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f78540a0d50 con 0x7f7854137e40 2024-04-26T00:33:37.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.719+0000 7f785c4a8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:37.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.719+0000 7f785c4a8640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f785409ec10 con 0x7f785413a770 2024-04-26T00:33:37.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.719+0000 7f785c4a8640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:37.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.719+0000 7f785c4a8640 10 monclient(hunting): authenticate will time out at 1799.027848s 2024-04-26T00:33:37.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.723+0000 7f785aa45640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f785413a770 0x7f785413cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:37.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.723+0000 7f785aa45640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f785413a770 0x7f785413cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:48412/0 (socket says 172.21.15.37:48412) 2024-04-26T00:33:37.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.723+0000 7f785aa45640 1 -- 172.21.15.37:0/2273866267 learned_addr learned my addr 172.21.15.37:0/2273866267 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:33:37.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.723+0000 7f785aa45640 10 monclient(hunting): get_auth_request con 0x7f785413a770 auth_method 0 2024-04-26T00:33:37.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.723+0000 7f785aa45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:37.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.723+0000 7f785aa45640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:37.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.723+0000 7f785aa45640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:37.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.723+0000 7f7859a43640 1 --2- 172.21.15.37:0/2273866267 >> v2:172.21.15.45:3300/0 conn(0x7f7854137e40 0x7f785413a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:37.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.723+0000 7f785aa45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:37.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.723+0000 7f785aa45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:37.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.723+0000 7f785aa45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:37.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.723+0000 7f7859a43640 10 monclient(hunting): get_auth_request con 0x7f7854137e40 auth_method 0 2024-04-26T00:33:37.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.723+0000 7f7859a43640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:37.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.723+0000 7f7859a43640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:37.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.723+0000 7f7859a43640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:37.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.723+0000 7f785aa45640 10 monclient(hunting): handle_auth_done global_id 154190 payload 274 2024-04-26T00:33:37.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.723+0000 7f785aa45640 1 -- 172.21.15.37:0/2273866267 >> v2:172.21.15.45:3300/0 conn(0x7f7854137e40 msgr2=0x7f785413a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:33:37.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.723+0000 7f785aa45640 1 --2- 172.21.15.37:0/2273866267 >> v2:172.21.15.45:3300/0 conn(0x7f7854137e40 0x7f785413a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:37.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.723+0000 7f785aa45640 1 -- 172.21.15.37:0/2273866267 >> v1:172.21.15.45:6789/0 conn(0x7f78540a0260 legacy=0x7f78540a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:33:37.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.723+0000 7f785aa45640 10 monclient: _finish_hunting 0 2024-04-26T00:33:37.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.723+0000 7f785aa45640 1 monclient: found mon.noname-b 2024-04-26T00:33:37.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.723+0000 7f785aa45640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:37.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.723+0000 7f785aa45640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:33:37.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.723+0000 7f785aa45640 1 -- 172.21.15.37:0/2273866267 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f785406cff0 con 0x7f785413a770 2024-04-26T00:33:37.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.723+0000 7f7859a43640 1 --2- 172.21.15.37:0/2273866267 >> v2:172.21.15.45:3300/0 conn(0x7f7854137e40 0x7f785413a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:33:37.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.727+0000 7f785aa45640 1 -- 172.21.15.37:0/2273866267 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f785413a770 msgr2=0x7f785413cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:37.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.727+0000 7f785aa45640 1 --2- 172.21.15.37:0/2273866267 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f785413a770 0x7f785413cb60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f784c009e50 tx=0x7f784c01c6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41ebb682ac88d370 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:37.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.727+0000 7f7859242640 1 -- 172.21.15.37:0/2273866267 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f784c0a9460 con 0x7f785413a770 2024-04-26T00:33:37.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.727+0000 7f7859242640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:37.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.727+0000 7f7859242640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:33:37.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.727+0000 7f7859242640 10 monclient: dump: 2024-04-26T00:33:37.721 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:37.721 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:37.721 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:37.721 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:37.721 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:37.721 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:37.721 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:37.721 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:37.722 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:37.722 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:37.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.727+0000 7f7859242640 10 monclient: _finish_auth 0 2024-04-26T00:33:37.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.727+0000 7f7859242640 10 monclient: _check_auth_tickets 2024-04-26T00:33:37.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.727+0000 7f7859242640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:37.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.727+0000 7f785c4a8640 5 monclient: authenticate success, global_id 154190 2024-04-26T00:33:37.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.727+0000 7f785c4a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:33:37.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.727+0000 7f7859242640 1 -- 172.21.15.37:0/2273866267 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f784c0ae070 con 0x7f785413a770 2024-04-26T00:33:37.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.727+0000 7f7859242640 10 monclient: handle_config config(0 keys) 2024-04-26T00:33:37.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.727+0000 7f7859242640 1 -- 172.21.15.37:0/2273866267 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f784c0a9820 con 0x7f785413a770 2024-04-26T00:33:37.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.727+0000 7f7859242640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:37.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.727+0000 7f7859242640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:33:37.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.727+0000 7f7859242640 10 monclient: dump: 2024-04-26T00:33:37.723 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:37.723 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:37.723 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:37.723 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:37.723 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:37.723 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:37.723 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:37.723 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:37.723 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:37.723 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:37.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.727+0000 7f785c4a8640 10 monclient: get_monmap_and_config success 2024-04-26T00:33:37.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.727+0000 7f785c4a8640 4 set_mon_vals no callback set 2024-04-26T00:33:37.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.727+0000 7f785c4a8640 10 monclient: shutdown 2024-04-26T00:33:37.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.727+0000 7f785c4a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:33:37.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.727+0000 7f785c4a8640 1 -- 172.21.15.37:0/2273866267 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f785413a770 msgr2=0x7f785413cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:37.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.727+0000 7f785c4a8640 1 --2- 172.21.15.37:0/2273866267 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f785413a770 0x7f785413cb60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f784c009e50 tx=0x7f784c01c6d0 comp rx=0 tx=0).stop 2024-04-26T00:33:37.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.727+0000 7f785c4a8640 1 -- 172.21.15.37:0/2273866267 shutdown_connections 2024-04-26T00:33:37.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.727+0000 7f785c4a8640 1 --2- 172.21.15.37:0/2273866267 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f785413a770 0x7f785413cb60 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:37.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.727+0000 7f785c4a8640 1 --2- 172.21.15.37:0/2273866267 >> v2:172.21.15.45:3300/0 conn(0x7f7854137e40 0x7f785413a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:37.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.727+0000 7f785c4a8640 1 -- 172.21.15.37:0/2273866267 >> 172.21.15.37:0/2273866267 conn(0x7f78540a1ad0 msgr2=0x7f78540a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:33:37.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.727+0000 7f785c4a8640 1 -- 172.21.15.37:0/2273866267 shutdown_connections 2024-04-26T00:33:37.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.727+0000 7f785c4a8640 1 -- 172.21.15.37:0/2273866267 wait complete. 2024-04-26T00:33:37.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.727+0000 7f785c4a8640 10 monclient: build_initial_monmap 2024-04-26T00:33:37.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.727+0000 7f785c4a8640 10 monclient: monmap: 2024-04-26T00:33:37.729 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:33:37.729 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:33:37.729 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:33:37.729 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:33:37.729 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:33:37.729 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:33:37.729 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:33:37.729 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:33:37.729 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:33:37.729 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:37.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.727+0000 7f785c4a8640 1 Processor -- start 2024-04-26T00:33:37.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.727+0000 7f785c4a8640 1 -- start start 2024-04-26T00:33:37.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.727+0000 7f785c4a8640 10 monclient: init 2024-04-26T00:33:37.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f785c4a8640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:37.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f785c4a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:33:37.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f785c4a8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78540a0260 0x7f78541cece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:37.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f785c4a8640 10 monclient(hunting): picked mon.noname-b con 0x7f78540a0260 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:33:37.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f785c4a8640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7854137e40 0x7f7854134d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:37.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f785c4a8640 10 monclient(hunting): picked mon.noname-a con 0x7f7854137e40 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:33:37.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f785c4a8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f785413a770 0x7f7854135240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:37.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f785c4a8640 10 monclient(hunting): picked mon.noname-c con 0x7f785413a770 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:33:37.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f785c4a8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:37.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f785c4a8640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f785409f0f0 con 0x7f7854137e40 2024-04-26T00:33:37.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f785c4a8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:37.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f785c4a8640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f78540a00a0 con 0x7f78540a0260 2024-04-26T00:33:37.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f785c4a8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:37.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f785c4a8640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f785409f4c0 con 0x7f785413a770 2024-04-26T00:33:37.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f785c4a8640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:37.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f785c4a8640 10 monclient(hunting): authenticate will time out at 1799.036134s 2024-04-26T00:33:37.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f785aa45640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f785413a770 0x7f7854135240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:37.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f785aa45640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f785413a770 0x7f7854135240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:48414/0 (socket says 172.21.15.37:48414) 2024-04-26T00:33:37.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f785aa45640 1 -- 172.21.15.37:0/2084933072 learned_addr learned my addr 172.21.15.37:0/2084933072 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:33:37.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f785aa45640 10 monclient(hunting): get_auth_request con 0x7f785413a770 auth_method 0 2024-04-26T00:33:37.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f785aa45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:37.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f785aa45640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:37.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f785aa45640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:37.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f7859a43640 1 --2- 172.21.15.37:0/2084933072 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7854137e40 0x7f7854134d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:37.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f7859a43640 10 monclient(hunting): get_auth_request con 0x7f7854137e40 auth_method 0 2024-04-26T00:33:37.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f7859a43640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:37.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f7859a43640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:37.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f7859a43640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:37.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f785aa45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:37.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f785aa45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:37.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f785aa45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:37.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f785a244640 1 --2- 172.21.15.37:0/2084933072 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78540a0260 0x7f78541cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:37.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f7859a43640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:37.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f7859a43640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:37.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f7859a43640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:37.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f785a244640 10 monclient(hunting): get_auth_request con 0x7f78540a0260 auth_method 0 2024-04-26T00:33:37.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f785a244640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:37.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f785a244640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:37.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f785a244640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:37.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f7859a43640 10 monclient(hunting): handle_auth_done global_id 144312 payload 824 2024-04-26T00:33:37.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f7859a43640 1 -- 172.21.15.37:0/2084933072 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f785413a770 msgr2=0x7f7854135240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:37.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f7859a43640 1 --2- 172.21.15.37:0/2084933072 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f785413a770 0x7f7854135240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:37.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f7859a43640 1 -- 172.21.15.37:0/2084933072 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78540a0260 msgr2=0x7f78541cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:37.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f7859a43640 1 --2- 172.21.15.37:0/2084933072 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78540a0260 0x7f78541cece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:37.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f7859a43640 10 monclient: _finish_hunting 0 2024-04-26T00:33:37.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f7859a43640 1 monclient: found mon.noname-a 2024-04-26T00:33:37.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f7859a43640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:37.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f7859a43640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:33:37.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f7859a43640 1 -- 172.21.15.37:0/2084933072 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7854135e50 con 0x7f7854137e40 2024-04-26T00:33:37.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f785a244640 1 --2- 172.21.15.37:0/2084933072 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78540a0260 0x7f78541cece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:33:37.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f785aa45640 1 --2- 172.21.15.37:0/2084933072 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f785413a770 0x7f7854135240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:33:37.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f7859a43640 1 -- 172.21.15.37:0/2084933072 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7854137e40 msgr2=0x7f7854134d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:37.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f7859a43640 1 --2- 172.21.15.37:0/2084933072 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7854137e40 0x7f7854134d00 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f785000e7e0 tx=0x7f785000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:37.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f7846ffd640 1 -- 172.21.15.37:0/2084933072 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7850017070 con 0x7f7854137e40 2024-04-26T00:33:37.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f7846ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:37.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f7846ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:33:37.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f7846ffd640 10 monclient: dump: 2024-04-26T00:33:37.734 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:37.734 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:37.734 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:37.734 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:37.734 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:37.734 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:37.734 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:37.734 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:37.734 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:37.735 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:37.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f7846ffd640 10 monclient: _finish_auth 0 2024-04-26T00:33:37.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f7846ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:33:37.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f7846ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:37.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f7846ffd640 1 -- 172.21.15.37:0/2084933072 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f785000c8d0 con 0x7f7854137e40 2024-04-26T00:33:37.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f7846ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:33:37.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f7846ffd640 1 -- 172.21.15.37:0/2084933072 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785000f040 con 0x7f7854137e40 2024-04-26T00:33:37.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f785c4a8640 5 monclient: authenticate success, global_id 144312 2024-04-26T00:33:37.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f785c4a8640 10 monclient: _renew_subs 2024-04-26T00:33:37.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f785c4a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:33:37.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f785c4a8640 1 -- 172.21.15.37:0/2084933072 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f785409e070 con 0x7f7854137e40 2024-04-26T00:33:37.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f7846ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:37.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f7846ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:33:37.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f7847fff640 4 set_mon_vals no callback set 2024-04-26T00:33:37.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f7846ffd640 10 monclient: dump: 2024-04-26T00:33:37.736 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:37.736 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:37.736 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:37.736 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:37.736 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:37.736 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:37.736 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:37.736 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:37.736 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:37.736 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:37.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f785c4a8640 10 monclient: _renew_subs 2024-04-26T00:33:37.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f785c4a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:33:37.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.731+0000 7f785c4a8640 1 -- 172.21.15.37:0/2084933072 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7854120e30 con 0x7f7854137e40 2024-04-26T00:33:37.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.735+0000 7f785c4a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:37.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.735+0000 7f7846ffd640 1 -- 172.21.15.37:0/2084933072 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f785001a070 con 0x7f7854137e40 2024-04-26T00:33:37.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.735+0000 7f785c4a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:37.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.735+0000 7f785c4a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:33:37.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.735+0000 7f785c4a8640 1 -- 172.21.15.37:0/2084933072 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7854066f40 con 0x7f7854137e40 2024-04-26T00:33:37.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.735+0000 7f7846ffd640 1 --2- 172.21.15.37:0/2084933072 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7838035f00 0x7f78380383c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:37.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.735+0000 7f7846ffd640 1 -- 172.21.15.37:0/2084933072 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(532..532 src has 370..532) ==== 3959+0+0 (secure 0 0 0) 0x7f7850048660 con 0x7f7854137e40 2024-04-26T00:33:37.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.735+0000 7f785a244640 1 --2- 172.21.15.37:0/2084933072 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7838035f00 0x7f78380383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:37.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.735+0000 7f785a244640 10 monclient: get_auth_request con 0x7f7838035f00 auth_method 0 2024-04-26T00:33:37.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.735+0000 7f785a244640 1 --2- 172.21.15.37:0/2084933072 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7838035f00 0x7f78380383c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f78480098b0 tx=0x7f7848008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:37.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.739+0000 7f7846ffd640 1 -- 172.21.15.37:0/2084933072 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7854135e50 con 0x7f7854137e40 2024-04-26T00:33:37.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.739+0000 7f7846ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:37.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.739+0000 7f7846ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:33:37.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.963+0000 7f785c4a8640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-26T00:33:37.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.963+0000 7f785c4a8640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-26T00:33:37.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.963+0000 7f785c4a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:33:37.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.963+0000 7f785c4a8640 1 -- 172.21.15.37:0/2084933072 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f785406b9a0 con 0x7f7854137e40 2024-04-26T00:33:37.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.967+0000 7f7846ffd640 1 -- 172.21.15.37:0/2084933072 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f785000cc80 con 0x7f7854137e40 2024-04-26T00:33:37.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.967+0000 7f7846ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-26T00:33:37.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.967+0000 7f7846ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:33:37.960 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.967+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:37.960 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.967+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=11,kv=10,logm=53} crc {auth=3486560079,config=692619927,health=3007979612,kv=1034996044,logm=636988475}) 2024-04-26T00:33:37.960 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.967+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=23,config=2,health=12,kv=10,logm=53} crc {auth=1942369991,config=692619927,health=4252548296,kv=2259169355,logm=3775699367}) 2024-04-26T00:33:37.961 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.967+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:37.961 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.967+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=11,kv=10,logm=53} crc {auth=3486560079,config=692619927,health=3007979612,kv=1034996044,logm=636988475}) 2024-04-26T00:33:37.961 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.967+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=25,config=2,health=11,kv=10,logm=52} crc {auth=1467251052,config=692619927,health=3208788324,kv=1034996044,logm=3850748864}) 2024-04-26T00:33:37.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.967+0000 7f785c4a8640 1 -- 172.21.15.37:0/2084933072 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7838035f00 msgr2=0x7f78380383c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:37.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.967+0000 7f785c4a8640 1 --2- 172.21.15.37:0/2084933072 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7838035f00 0x7f78380383c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f78480098b0 tx=0x7f7848008040 comp rx=0 tx=0).stop 2024-04-26T00:33:37.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.967+0000 7f785c4a8640 10 monclient: shutdown 2024-04-26T00:33:37.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.967+0000 7f785c4a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:33:37.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.967+0000 7f785c4a8640 1 -- 172.21.15.37:0/2084933072 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7854137e40 msgr2=0x7f7854134d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:37.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.967+0000 7f785c4a8640 1 --2- 172.21.15.37:0/2084933072 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7854137e40 0x7f7854134d00 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f785000e7e0 tx=0x7f785000eca0 comp rx=0 tx=0).stop 2024-04-26T00:33:37.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.967+0000 7f785c4a8640 1 -- 172.21.15.37:0/2084933072 shutdown_connections 2024-04-26T00:33:37.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.967+0000 7f785c4a8640 1 --2- 172.21.15.37:0/2084933072 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7838035f00 0x7f78380383c0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:37.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.967+0000 7f785c4a8640 1 --2- 172.21.15.37:0/2084933072 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f785413a770 0x7f7854135240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:37.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.967+0000 7f785c4a8640 1 --2- 172.21.15.37:0/2084933072 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7854137e40 0x7f7854134d00 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:37.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.967+0000 7f785c4a8640 1 --2- 172.21.15.37:0/2084933072 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78540a0260 0x7f78541cece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:37.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.967+0000 7f785c4a8640 1 -- 172.21.15.37:0/2084933072 >> 172.21.15.37:0/2084933072 conn(0x7f78540a1ad0 msgr2=0x7f7854136060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:33:37.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.967+0000 7f785c4a8640 1 -- 172.21.15.37:0/2084933072 shutdown_connections 2024-04-26T00:33:37.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:37.967+0000 7f785c4a8640 1 -- 172.21.15.37:0/2084933072 wait complete. 2024-04-26T00:33:37.966 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.967+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:37.966 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.967+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1177376932}) 2024-04-26T00:33:37.966 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.967+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3866397399}) 2024-04-26T00:33:37.966 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.967+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:37.966 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.967+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1177376932}) 2024-04-26T00:33:37.966 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.967+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3351254212}) 2024-04-26T00:33:37.966 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.971+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:37.966 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.971+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1533850667}) 2024-04-26T00:33:37.966 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.971+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2660881887}) 2024-04-26T00:33:37.967 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.971+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:37.967 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.971+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1533850667}) 2024-04-26T00:33:37.967 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.971+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1162050989}) 2024-04-26T00:33:37.967 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.975+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:37.967 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.975+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2642715770}) 2024-04-26T00:33:37.967 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.975+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3344401944}) 2024-04-26T00:33:37.968 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.975+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:37.968 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.975+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2642715770}) 2024-04-26T00:33:37.968 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.975+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=522467769}) 2024-04-26T00:33:37.970 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.975+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:37.970 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.975+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3875982443}) 2024-04-26T00:33:37.970 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.975+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=106765190}) 2024-04-26T00:33:37.970 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.975+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:37.970 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.975+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3875982443}) 2024-04-26T00:33:37.970 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.975+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=810917861}) 2024-04-26T00:33:37.972 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.979+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:37.972 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.979+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=364599404}) 2024-04-26T00:33:37.972 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.979+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2709193655}) 2024-04-26T00:33:37.972 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.979+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:37.972 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.979+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=364599404}) 2024-04-26T00:33:37.973 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.979+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2639429302}) 2024-04-26T00:33:37.974 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.979+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:37.974 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.979+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3314413597}) 2024-04-26T00:33:37.974 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.979+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3826738858}) 2024-04-26T00:33:37.974 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.979+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:37.974 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.979+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3314413597}) 2024-04-26T00:33:37.974 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.979+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2294719853}) 2024-04-26T00:33:37.976 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.983+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:37.976 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.983+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1916863862}) 2024-04-26T00:33:37.977 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.983+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4021192299}) 2024-04-26T00:33:37.977 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.983+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:37.977 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.983+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1916863862}) 2024-04-26T00:33:37.977 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.983+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2640315603}) 2024-04-26T00:33:37.978 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.983+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:37.979 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.983+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2494482218}) 2024-04-26T00:33:37.979 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.983+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2165005973}) 2024-04-26T00:33:37.979 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.983+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:37.979 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.983+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2494482218}) 2024-04-26T00:33:37.979 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.983+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1921889825}) 2024-04-26T00:33:37.981 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-26T00:33:37.982 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.987+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:37.982 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.987+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1982639779}) 2024-04-26T00:33:37.982 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.987+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=238091812}) 2024-04-26T00:33:37.982 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.987+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:37.982 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.987+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1982639779}) 2024-04-26T00:33:37.982 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.987+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3091413321}) 2024-04-26T00:33:37.983 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.987+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:37.983 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.987+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=928205883}) 2024-04-26T00:33:37.983 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.987+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=849827514}) 2024-04-26T00:33:37.983 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.987+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:37.983 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.987+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=928205883}) 2024-04-26T00:33:37.983 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.987+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1295146831}) 2024-04-26T00:33:37.985 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.991+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:37.985 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.991+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=346152741}) 2024-04-26T00:33:37.985 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.991+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2055446866}) 2024-04-26T00:33:37.985 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.991+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:37.985 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.991+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=346152741}) 2024-04-26T00:33:37.985 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.991+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=958933477}) 2024-04-26T00:33:37.987 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.991+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:37.987 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.991+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3170346586}) 2024-04-26T00:33:37.987 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.991+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3204003218}) 2024-04-26T00:33:37.987 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.991+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:37.987 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.991+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3170346586}) 2024-04-26T00:33:37.988 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.991+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4128923392}) 2024-04-26T00:33:37.989 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.995+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:37.989 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.995+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2887274711}) 2024-04-26T00:33:37.989 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.995+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2868098096}) 2024-04-26T00:33:37.989 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.995+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:37.989 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.995+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2887274711}) 2024-04-26T00:33:37.989 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.995+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=977096077}) 2024-04-26T00:33:37.991 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.999+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:37.991 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.999+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2835883062}) 2024-04-26T00:33:37.991 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.999+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4066149291}) 2024-04-26T00:33:37.992 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.999+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:37.992 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.999+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2835883062}) 2024-04-26T00:33:37.992 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.999+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1003424162}) 2024-04-26T00:33:37.993 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.999+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:37.993 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.999+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3647884807}) 2024-04-26T00:33:37.994 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.999+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1255295573}) 2024-04-26T00:33:37.994 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.999+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:37.994 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.999+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3647884807}) 2024-04-26T00:33:37.994 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:37.999+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2849847910}) 2024-04-26T00:33:37.995 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.003+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:37.995 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.003+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1085076183}) 2024-04-26T00:33:37.996 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.003+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=496532565}) 2024-04-26T00:33:37.996 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.003+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:37.996 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.003+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1085076183}) 2024-04-26T00:33:37.996 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.003+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=677181241}) 2024-04-26T00:33:37.998 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.003+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:37.998 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.003+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=481767508}) 2024-04-26T00:33:37.998 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.003+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1877473805}) 2024-04-26T00:33:37.998 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.003+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:37.998 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.003+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=481767508}) 2024-04-26T00:33:37.999 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.003+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3139331790}) 2024-04-26T00:33:38.000 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.007+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.000 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.007+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1651432339}) 2024-04-26T00:33:38.000 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.007+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1283282712}) 2024-04-26T00:33:38.000 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.007+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.000 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.007+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1651432339}) 2024-04-26T00:33:38.000 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.007+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4269785976}) 2024-04-26T00:33:38.002 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.007+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.002 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.007+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1230685554}) 2024-04-26T00:33:38.002 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.007+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=936510182}) 2024-04-26T00:33:38.003 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.007+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.003 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.007+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1230685554}) 2024-04-26T00:33:38.003 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.007+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2791189342}) 2024-04-26T00:33:38.005 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.011+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.005 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.011+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1727302095}) 2024-04-26T00:33:38.005 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.011+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3494582144}) 2024-04-26T00:33:38.005 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.011+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.005 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.011+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1727302095}) 2024-04-26T00:33:38.005 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.011+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1019914272}) 2024-04-26T00:33:38.007 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.011+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.007 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.011+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3747883537}) 2024-04-26T00:33:38.007 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.011+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2026399570}) 2024-04-26T00:33:38.007 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.011+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.007 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.011+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3747883537}) 2024-04-26T00:33:38.007 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.011+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1742478191}) 2024-04-26T00:33:38.009 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.015+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.009 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.015+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3966665064}) 2024-04-26T00:33:38.009 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.015+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3852599315}) 2024-04-26T00:33:38.009 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.015+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.009 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.015+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3966665064}) 2024-04-26T00:33:38.009 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.015+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=285068824}) 2024-04-26T00:33:38.011 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.019+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.012 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.019+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3439408973}) 2024-04-26T00:33:38.012 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.019+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1034113080}) 2024-04-26T00:33:38.012 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.019+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.012 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.019+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3439408973}) 2024-04-26T00:33:38.012 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.019+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4130139629}) 2024-04-26T00:33:38.013 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.019+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.013 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.019+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3845944210}) 2024-04-26T00:33:38.014 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.019+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=174296647}) 2024-04-26T00:33:38.014 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.019+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.014 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.019+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3845944210}) 2024-04-26T00:33:38.014 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.019+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=366008830}) 2024-04-26T00:33:38.015 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.023+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.015 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.023+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3752305319}) 2024-04-26T00:33:38.016 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.023+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3706065040}) 2024-04-26T00:33:38.016 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.023+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.016 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.023+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3752305319}) 2024-04-26T00:33:38.016 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.023+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4037732004}) 2024-04-26T00:33:38.018 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.023+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.018 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.023+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1473367708}) 2024-04-26T00:33:38.018 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.023+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1681009498}) 2024-04-26T00:33:38.018 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.023+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.018 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.023+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1473367708}) 2024-04-26T00:33:38.018 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.023+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2769889840}) 2024-04-26T00:33:38.024 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.027+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.024 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.027+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1296272640}) 2024-04-26T00:33:38.024 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.027+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1318421212}) 2024-04-26T00:33:38.024 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.027+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.024 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.027+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1296272640}) 2024-04-26T00:33:38.024 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.027+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1447880007}) 2024-04-26T00:33:38.024 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.027+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.025 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.027+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2899238969}) 2024-04-26T00:33:38.025 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.027+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=704115228}) 2024-04-26T00:33:38.025 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.027+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.025 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.027+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2899238969}) 2024-04-26T00:33:38.025 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.027+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2797583052}) 2024-04-26T00:33:38.026 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.031+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.026 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.031+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=273637147}) 2024-04-26T00:33:38.026 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.031+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3551841308}) 2024-04-26T00:33:38.026 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.031+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.026 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.031+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=273637147}) 2024-04-26T00:33:38.026 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.031+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=161642670}) 2024-04-26T00:33:38.027 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.031+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.027 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.031+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1562595293}) 2024-04-26T00:33:38.027 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.031+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2059602675}) 2024-04-26T00:33:38.027 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.031+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.027 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.031+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1562595293}) 2024-04-26T00:33:38.027 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.031+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1386707861}) 2024-04-26T00:33:38.030 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.035+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.030 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.035+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3082709445}) 2024-04-26T00:33:38.031 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.035+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3180842736}) 2024-04-26T00:33:38.031 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.035+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.031 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.035+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3082709445}) 2024-04-26T00:33:38.031 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.035+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=837187431}) 2024-04-26T00:33:38.032 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.039+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.033 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.039+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3969570410}) 2024-04-26T00:33:38.033 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.039+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2451889915}) 2024-04-26T00:33:38.033 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.039+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.033 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.039+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3969570410}) 2024-04-26T00:33:38.033 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.039+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2527319513}) 2024-04-26T00:33:38.035 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.039+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.035 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.039+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1056823216}) 2024-04-26T00:33:38.035 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.039+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1032026292}) 2024-04-26T00:33:38.035 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.039+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.035 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.039+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1056823216}) 2024-04-26T00:33:38.035 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.039+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=343396895}) 2024-04-26T00:33:38.036 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.043+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.036 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.043+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1871813789}) 2024-04-26T00:33:38.036 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.043+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2968014001}) 2024-04-26T00:33:38.036 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.043+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.036 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.043+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1871813789}) 2024-04-26T00:33:38.036 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.043+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3577286781}) 2024-04-26T00:33:38.038 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.043+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.038 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.043+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1826747941}) 2024-04-26T00:33:38.038 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.043+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1624203505}) 2024-04-26T00:33:38.038 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.043+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.039 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.043+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1826747941}) 2024-04-26T00:33:38.039 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.043+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2887497135}) 2024-04-26T00:33:38.040 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.047+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.040 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.047+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=761321995}) 2024-04-26T00:33:38.040 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.047+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1761977626}) 2024-04-26T00:33:38.040 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.047+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.040 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.047+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=761321995}) 2024-04-26T00:33:38.040 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.047+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3018664451}) 2024-04-26T00:33:38.042 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.047+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.042 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.047+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3776337777}) 2024-04-26T00:33:38.042 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.047+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3787780720}) 2024-04-26T00:33:38.042 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.047+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.042 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.047+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3776337777}) 2024-04-26T00:33:38.042 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.047+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4016404960}) 2024-04-26T00:33:38.044 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.051+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.044 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.051+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1232503194}) 2024-04-26T00:33:38.044 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.051+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3942886516}) 2024-04-26T00:33:38.044 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.051+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.044 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.051+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1232503194}) 2024-04-26T00:33:38.044 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.051+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3320675440}) 2024-04-26T00:33:38.046 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.051+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.046 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.051+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2938247843}) 2024-04-26T00:33:38.046 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.051+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2546659926}) 2024-04-26T00:33:38.046 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.051+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.046 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.051+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2938247843}) 2024-04-26T00:33:38.046 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.051+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3970728513}) 2024-04-26T00:33:38.048 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.055+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.048 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.055+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=682884576}) 2024-04-26T00:33:38.048 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.055+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=64871399}) 2024-04-26T00:33:38.048 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.055+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.048 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.055+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=682884576}) 2024-04-26T00:33:38.048 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.055+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2964594171}) 2024-04-26T00:33:38.050 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.055+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.050 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.055+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1251423575}) 2024-04-26T00:33:38.050 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.055+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4026413431}) 2024-04-26T00:33:38.050 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.055+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.050 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.055+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1251423575}) 2024-04-26T00:33:38.050 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.055+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4275973598}) 2024-04-26T00:33:38.052 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.059+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.052 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.059+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1407994166}) 2024-04-26T00:33:38.052 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.059+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=435771970}) 2024-04-26T00:33:38.052 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.059+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.052 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.059+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1407994166}) 2024-04-26T00:33:38.052 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.059+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3809205580}) 2024-04-26T00:33:38.054 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.059+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.054 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.059+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1583216698}) 2024-04-26T00:33:38.054 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.059+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=133581624}) 2024-04-26T00:33:38.054 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.059+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.054 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.059+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1583216698}) 2024-04-26T00:33:38.054 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.059+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3078988807}) 2024-04-26T00:33:38.057 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.063+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.057 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.063+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=4,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=52} crc {logm=3873495173,mdsmap=2091496210,mgr=1137150072,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=3850851432,mon_config_key=4061063117,monmap=3876866194,osd_metadata=839604482,osd_pg_creating=2322578938,osd_snap=4267538056}) 2024-04-26T00:33:38.057 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.063+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=4,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=53} crc {logm=3873495173,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=1709480086,mon_config_key=1484518886,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=2322578938,osd_snap=2088895407}) 2024-04-26T00:33:38.057 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.063+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.057 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.063+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=4,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=52} crc {logm=3873495173,mdsmap=2091496210,mgr=1137150072,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=3850851432,mon_config_key=4061063117,monmap=3876866194,osd_metadata=839604482,osd_pg_creating=2322578938,osd_snap=4267538056}) 2024-04-26T00:33:38.057 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.063+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=4,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=49} crc {logm=3873495173,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=927355729,mon_config_key=4061063117,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=2322578938,osd_snap=2902488501}) 2024-04-26T00:33:38.490 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.495+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.490 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.495+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=41,osdmap=59} crc {osd_snap=3750274673,osdmap=2786517835}) 2024-04-26T00:33:38.491 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.495+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=41,osdmap=59} crc {osd_snap=1971279890,osdmap=3139263946}) 2024-04-26T00:33:38.491 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.495+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.491 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.495+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=41,osdmap=59} crc {osd_snap=3750274673,osdmap=2786517835}) 2024-04-26T00:33:38.491 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.495+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=40,osdmap=60} crc {osd_snap=1770134225,osdmap=3659243562}) 2024-04-26T00:33:38.500 INFO:tasks.rados.rados.0.smithi045.stdout:2363: done (2 left) 2024-04-26T00:33:38.500 INFO:tasks.rados.rados.0.smithi045.stdout:2364: rollback oid 36 current snap is 266 2024-04-26T00:33:38.500 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 36 to 238 2024-04-26T00:33:38.500 INFO:tasks.rados.rados.0.smithi045.stdout:2365: snap_remove snap 263 2024-04-26T00:33:38.500 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.507+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.501 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.507+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4112638491}) 2024-04-26T00:33:38.501 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.507+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3050933987}) 2024-04-26T00:33:38.501 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.507+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.501 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.507+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4112638491}) 2024-04-26T00:33:38.501 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.507+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=656748395}) 2024-04-26T00:33:38.573 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.579+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.573 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.579+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=90} crc {osdmap=1097968278}) 2024-04-26T00:33:38.573 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.579+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=90} crc {osdmap=3239813268}) 2024-04-26T00:33:38.573 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.579+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:33:38.573 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.579+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=90} crc {osdmap=1097968278}) 2024-04-26T00:33:38.573 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:33:38.579+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=82} crc {osdmap=1006581250}) 2024-04-26T00:33:38.982 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:33:38.983 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:33:39.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.119+0000 7f91a4c6c640 10 monclient: get_monmap_and_config 2024-04-26T00:33:39.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.119+0000 7f91a4c6c640 10 monclient: build_initial_monmap 2024-04-26T00:33:39.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.119+0000 7f91a4c6c640 10 monclient: monmap: 2024-04-26T00:33:39.114 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:33:39.114 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:39.114 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:33:39.121753+0000 2024-04-26T00:33:39.114 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:33:39.121753+0000 2024-04-26T00:33:39.114 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:33:39.114 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:39.114 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:33:39.114 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:33:39.114 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:33:39.114 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:33:39.115 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:33:39.115 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:39.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.119+0000 7f91a4c6c640 1 Processor -- start 2024-04-26T00:33:39.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.119+0000 7f91a4c6c640 1 -- start start 2024-04-26T00:33:39.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.119+0000 7f91a4c6c640 10 monclient: init 2024-04-26T00:33:39.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f91a4c6c640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:39.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f91a4c6c640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-04-26T00:33:39.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f91a4c6c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f91a0136e10 0x7f91a009c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:39.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f91a4c6c640 10 monclient(hunting): picked mon.noname-a con 0x7f91a0136e10 addr v2:172.21.15.37:3300/0 2024-04-26T00:33:39.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f91a4c6c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f91a01371f0 0x7f91a0137600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:39.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f91a4c6c640 10 monclient(hunting): picked mon.noname-c con 0x7f91a01371f0 addr v2:172.21.15.45:3300/0 2024-04-26T00:33:39.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f91a4c6c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f91a0135d90 addr v1:172.21.15.37:6789/0 2024-04-26T00:33:39.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f91a4c6c640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91a009b680 con 0x7f91a0135d90 2024-04-26T00:33:39.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f91a4c6c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:39.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f91a4c6c640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f91a009b8f0 con 0x7f91a0136e10 2024-04-26T00:33:39.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f91a4c6c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:39.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f91a4c6c640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f91a009bbd0 con 0x7f91a01371f0 2024-04-26T00:33:39.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f91a4c6c640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:39.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f91a4c6c640 10 monclient(hunting): authenticate will time out at 1800.428197s 2024-04-26T00:33:39.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f919e59c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f91a0136e10 0x7f91a009c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:39.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f919ed9d640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f91a0135d90 0x7f91a0136220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:35158/0 (socket says 172.21.15.37:35158) 2024-04-26T00:33:39.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f919ed9d640 1 -- 172.21.15.37:0/1121962057 learned_addr learned my addr 172.21.15.37:0/1121962057 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:33:39.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f919e59c640 10 monclient(hunting): get_auth_request con 0x7f91a0136e10 auth_method 0 2024-04-26T00:33:39.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f919e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:39.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f919e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:39.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f919e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:39.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f919dd9b640 1 --2- 172.21.15.37:0/1121962057 >> v2:172.21.15.45:3300/0 conn(0x7f91a01371f0 0x7f91a0137600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:39.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f919dd9b640 10 monclient(hunting): get_auth_request con 0x7f91a01371f0 auth_method 0 2024-04-26T00:33:39.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f919dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:39.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f919dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:39.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f919dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:39.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f919e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:39.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f919e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:39.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f919e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:39.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f919dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:39.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f919dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:39.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f919dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:39.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f919e59c640 10 monclient(hunting): handle_auth_done global_id 144315 payload 274 2024-04-26T00:33:39.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f919e59c640 1 -- 172.21.15.37:0/1121962057 >> v2:172.21.15.45:3300/0 conn(0x7f91a01371f0 msgr2=0x7f91a0137600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:33:39.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f919e59c640 1 --2- 172.21.15.37:0/1121962057 >> v2:172.21.15.45:3300/0 conn(0x7f91a01371f0 0x7f91a0137600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:39.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f919e59c640 1 -- 172.21.15.37:0/1121962057 >> v1:172.21.15.37:6789/0 conn(0x7f91a0135d90 legacy=0x7f91a0136220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:33:39.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f919e59c640 10 monclient: _finish_hunting 0 2024-04-26T00:33:39.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f919e59c640 1 monclient: found mon.noname-a 2024-04-26T00:33:39.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f919e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:39.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f919e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:33:39.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.123+0000 7f919e59c640 1 -- 172.21.15.37:0/1121962057 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91a0137b40 con 0x7f91a0136e10 2024-04-26T00:33:39.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f919e59c640 1 -- 172.21.15.37:0/1121962057 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91a0136e10 msgr2=0x7f91a009c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:39.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f919e59c640 1 --2- 172.21.15.37:0/1121962057 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91a0136e10 0x7f91a009c5b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f91a0068e60 tx=0x7f919001c300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4323eac5c986390e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:39.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f919d59a640 1 -- 172.21.15.37:0/1121962057 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91900b0070 con 0x7f91a0136e10 2024-04-26T00:33:39.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f919d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:39.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f919d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:33:39.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f919d59a640 10 monclient: dump: 2024-04-26T00:33:39.123 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:39.123 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:39.123 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:39.123 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:39.123 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:39.123 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:39.123 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:39.123 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:39.123 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:39.123 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:39.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f919d59a640 10 monclient: _finish_auth 0 2024-04-26T00:33:39.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f919d59a640 10 monclient: _check_auth_tickets 2024-04-26T00:33:39.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f919d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:39.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f919d59a640 1 -- 172.21.15.37:0/1121962057 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91900ab600 con 0x7f91a0136e10 2024-04-26T00:33:39.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f919d59a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:33:39.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f919d59a640 1 -- 172.21.15.37:0/1121962057 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91900ab9d0 con 0x7f91a0136e10 2024-04-26T00:33:39.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f91a4c6c640 5 monclient: authenticate success, global_id 144315 2024-04-26T00:33:39.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f91a4c6c640 10 monclient: get_monmap_and_config success 2024-04-26T00:33:39.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f91a4c6c640 4 set_mon_vals no callback set 2024-04-26T00:33:39.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f919d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:39.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f919d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:33:39.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f919d59a640 10 monclient: dump: 2024-04-26T00:33:39.124 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:39.124 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:39.124 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:39.124 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:39.124 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:39.124 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:39.124 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:39.124 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:39.125 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:39.125 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:39.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f91a4c6c640 10 monclient: shutdown 2024-04-26T00:33:39.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f91a4c6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:33:39.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f91a4c6c640 1 -- 172.21.15.37:0/1121962057 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91a0136e10 msgr2=0x7f91a009c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:39.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f91a4c6c640 1 --2- 172.21.15.37:0/1121962057 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91a0136e10 0x7f91a009c5b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f91a0068e60 tx=0x7f919001c300 comp rx=0 tx=0).stop 2024-04-26T00:33:39.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f91a4c6c640 1 -- 172.21.15.37:0/1121962057 shutdown_connections 2024-04-26T00:33:39.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f91a4c6c640 1 --2- 172.21.15.37:0/1121962057 >> v2:172.21.15.45:3300/0 conn(0x7f91a01371f0 0x7f91a0137600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:39.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f91a4c6c640 1 --2- 172.21.15.37:0/1121962057 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91a0136e10 0x7f91a009c5b0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:39.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f91a4c6c640 1 -- 172.21.15.37:0/1121962057 >> 172.21.15.37:0/1121962057 conn(0x7f91a00a64e0 msgr2=0x7f91a00a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:33:39.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f91a4c6c640 1 -- 172.21.15.37:0/1121962057 shutdown_connections 2024-04-26T00:33:39.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f91a4c6c640 1 -- 172.21.15.37:0/1121962057 wait complete. 2024-04-26T00:33:39.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f91a4c6c640 10 monclient: build_initial_monmap 2024-04-26T00:33:39.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f91a4c6c640 10 monclient: monmap: 2024-04-26T00:33:39.125 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:33:39.126 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:33:39.126 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:33:39.126 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:33:39.126 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:33:39.126 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:33:39.126 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:33:39.126 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:33:39.126 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:33:39.126 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:39.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f91a4c6c640 1 Processor -- start 2024-04-26T00:33:39.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f91a4c6c640 1 -- start start 2024-04-26T00:33:39.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f91a4c6c640 10 monclient: init 2024-04-26T00:33:39.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f91a4c6c640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:39.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f91a4c6c640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:33:39.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f91a4c6c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91a0135d90 0x7f91a01cf2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:39.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f91a4c6c640 10 monclient(hunting): picked mon.noname-a con 0x7f91a0135d90 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:33:39.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f91a4c6c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91a0136e10 0x7f91a0134d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:39.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f91a4c6c640 10 monclient(hunting): picked mon.noname-c con 0x7f91a0136e10 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:33:39.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f91a4c6c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91a01371f0 0x7f91a013a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:39.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f91a4c6c640 10 monclient(hunting): picked mon.noname-b con 0x7f91a01371f0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:33:39.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f91a4c6c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:39.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f91a4c6c640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f91a0141f10 con 0x7f91a0135d90 2024-04-26T00:33:39.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f91a4c6c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:39.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f91a4c6c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f91a0141d90 con 0x7f91a01371f0 2024-04-26T00:33:39.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f91a4c6c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:39.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f91a4c6c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f91a0142090 con 0x7f91a0136e10 2024-04-26T00:33:39.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f91a4c6c640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:39.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.127+0000 7f91a4c6c640 10 monclient(hunting): authenticate will time out at 1800.435948s 2024-04-26T00:33:39.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f919e59c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91a0135d90 0x7f91a01cf2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:39.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f919dd9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91a0136e10 0x7f91a0134d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:39.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f919e59c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91a0135d90 0x7f91a01cf2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:37706/0 (socket says 172.21.15.37:37706) 2024-04-26T00:33:39.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f919e59c640 1 -- 172.21.15.37:0/78287174 learned_addr learned my addr 172.21.15.37:0/78287174 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:33:39.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f919e59c640 10 monclient(hunting): get_auth_request con 0x7f91a0135d90 auth_method 0 2024-04-26T00:33:39.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f919e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:39.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f919e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:39.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f919e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:39.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f919dd9b640 10 monclient(hunting): get_auth_request con 0x7f91a0136e10 auth_method 0 2024-04-26T00:33:39.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f919dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:39.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f919dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:39.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f919dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:39.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f919e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:39.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f919e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:39.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f919e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:39.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f919dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:39.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f919dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:39.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f919dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:39.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f919ed9d640 1 --2- 172.21.15.37:0/78287174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91a01371f0 0x7f91a013a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:39.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f919ed9d640 10 monclient(hunting): get_auth_request con 0x7f91a01371f0 auth_method 0 2024-04-26T00:33:39.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f919ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:39.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f919ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:39.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f919ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:39.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f919e59c640 10 monclient(hunting): handle_auth_done global_id 144318 payload 824 2024-04-26T00:33:39.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f919e59c640 1 -- 172.21.15.37:0/78287174 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91a0136e10 msgr2=0x7f91a0134d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:39.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f919e59c640 1 --2- 172.21.15.37:0/78287174 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91a0136e10 0x7f91a0134d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:39.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f919e59c640 1 -- 172.21.15.37:0/78287174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91a01371f0 msgr2=0x7f91a013a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:39.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f919e59c640 1 --2- 172.21.15.37:0/78287174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91a01371f0 0x7f91a013a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:39.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f919e59c640 10 monclient: _finish_hunting 0 2024-04-26T00:33:39.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f919e59c640 1 monclient: found mon.noname-a 2024-04-26T00:33:39.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f919e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:39.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f919e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:33:39.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f919e59c640 1 -- 172.21.15.37:0/78287174 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91a01d01b0 con 0x7f91a0135d90 2024-04-26T00:33:39.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f919e59c640 1 -- 172.21.15.37:0/78287174 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91a0135d90 msgr2=0x7f91a01cf2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:39.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f919e59c640 1 --2- 172.21.15.37:0/78287174 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91a0135d90 0x7f91a01cf2c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f91900ae040 tx=0x7f91900190a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:39.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f918effd640 1 -- 172.21.15.37:0/78287174 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91900b8020 con 0x7f91a0135d90 2024-04-26T00:33:39.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f918effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:39.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f918effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:33:39.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f918effd640 10 monclient: dump: 2024-04-26T00:33:39.134 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:39.134 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:39.134 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:39.134 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:39.134 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:39.134 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:39.134 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:39.134 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:39.134 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:39.134 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:39.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f918effd640 10 monclient: _finish_auth 0 2024-04-26T00:33:39.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f918effd640 10 monclient: _check_auth_tickets 2024-04-26T00:33:39.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f918effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:39.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f918effd640 1 -- 172.21.15.37:0/78287174 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91900b4620 con 0x7f91a0135d90 2024-04-26T00:33:39.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f918effd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:33:39.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f918effd640 1 -- 172.21.15.37:0/78287174 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91900b0040 con 0x7f91a0135d90 2024-04-26T00:33:39.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f918effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:39.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f918effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:33:39.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f918effd640 10 monclient: dump: 2024-04-26T00:33:39.135 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:39.135 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:39.135 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:39.135 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:39.135 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:39.135 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:39.135 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:39.135 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:39.136 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:39.136 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:39.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f91a4c6c640 5 monclient: authenticate success, global_id 144318 2024-04-26T00:33:39.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f91a4c6c640 10 monclient: _renew_subs 2024-04-26T00:33:39.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f919cd99640 4 set_mon_vals no callback set 2024-04-26T00:33:39.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f91a4c6c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:33:39.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f91a4c6c640 1 -- 172.21.15.37:0/78287174 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91a009aca0 con 0x7f91a0135d90 2024-04-26T00:33:39.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f91a4c6c640 10 monclient: _renew_subs 2024-04-26T00:33:39.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f91a4c6c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:33:39.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f91a4c6c640 1 -- 172.21.15.37:0/78287174 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91a013e950 con 0x7f91a0135d90 2024-04-26T00:33:39.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f918effd640 1 -- 172.21.15.37:0/78287174 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f91900bb020 con 0x7f91a0135d90 2024-04-26T00:33:39.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f91a4c6c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:39.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f91a4c6c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:39.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f91a4c6c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:33:39.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.131+0000 7f91a4c6c640 1 -- 172.21.15.37:0/78287174 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91a0075910 con 0x7f91a0135d90 2024-04-26T00:33:39.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.135+0000 7f918effd640 1 --2- 172.21.15.37:0/78287174 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9180035ed0 0x7f9180038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:39.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.135+0000 7f918effd640 1 -- 172.21.15.37:0/78287174 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(533..533 src has 370..533) ==== 3987+0+0 (secure 0 0 0) 0x7f91900e2600 con 0x7f91a0135d90 2024-04-26T00:33:39.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.135+0000 7f919dd9b640 1 --2- 172.21.15.37:0/78287174 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9180035ed0 0x7f9180038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:39.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.135+0000 7f919dd9b640 10 monclient: get_auth_request con 0x7f9180035ed0 auth_method 0 2024-04-26T00:33:39.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.135+0000 7f919dd9b640 1 --2- 172.21.15.37:0/78287174 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9180035ed0 0x7f9180038390 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f91a013c140 tx=0x7f9194009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:39.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.139+0000 7f918effd640 1 -- 172.21.15.37:0/78287174 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f91900c2030 con 0x7f91a0135d90 2024-04-26T00:33:39.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.139+0000 7f918effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:39.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.139+0000 7f918effd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:33:39.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.363+0000 7f91a4c6c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:33:39.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.363+0000 7f91a4c6c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:33:39.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.363+0000 7f91a4c6c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:33:39.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.363+0000 7f91a4c6c640 1 -- 172.21.15.37:0/78287174 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f91a0066f40 con 0x7f91a0135d90 2024-04-26T00:33:39.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.363+0000 7f918effd640 1 -- 172.21.15.37:0/78287174 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f9190016010 con 0x7f91a0135d90 2024-04-26T00:33:39.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.363+0000 7f918effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:33:39.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.363+0000 7f918effd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:33:39.358 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":102,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:33:39.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.367+0000 7f91a4c6c640 1 -- 172.21.15.37:0/78287174 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9180035ed0 msgr2=0x7f9180038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:39.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.367+0000 7f91a4c6c640 1 --2- 172.21.15.37:0/78287174 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9180035ed0 0x7f9180038390 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f91a013c140 tx=0x7f9194009190 comp rx=0 tx=0).stop 2024-04-26T00:33:39.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.367+0000 7f91a4c6c640 10 monclient: shutdown 2024-04-26T00:33:39.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.367+0000 7f91a4c6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:33:39.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.367+0000 7f91a4c6c640 1 -- 172.21.15.37:0/78287174 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91a0135d90 msgr2=0x7f91a01cf2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:39.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.367+0000 7f91a4c6c640 1 --2- 172.21.15.37:0/78287174 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91a0135d90 0x7f91a01cf2c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f91900ae040 tx=0x7f91900190a0 comp rx=0 tx=0).stop 2024-04-26T00:33:39.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.367+0000 7f91a4c6c640 1 -- 172.21.15.37:0/78287174 shutdown_connections 2024-04-26T00:33:39.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.367+0000 7f91a4c6c640 1 --2- 172.21.15.37:0/78287174 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9180035ed0 0x7f9180038390 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:39.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.367+0000 7f91a4c6c640 1 --2- 172.21.15.37:0/78287174 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f91a01371f0 0x7f91a013a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:39.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.367+0000 7f91a4c6c640 1 --2- 172.21.15.37:0/78287174 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91a0136e10 0x7f91a0134d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:39.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.367+0000 7f91a4c6c640 1 --2- 172.21.15.37:0/78287174 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91a0135d90 0x7f91a01cf2c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:39.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.367+0000 7f91a4c6c640 1 -- 172.21.15.37:0/78287174 >> 172.21.15.37:0/78287174 conn(0x7f91a00a64e0 msgr2=0x7f91a00a4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:33:39.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.367+0000 7f91a4c6c640 1 -- 172.21.15.37:0/78287174 shutdown_connections 2024-04-26T00:33:39.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.367+0000 7f91a4c6c640 1 -- 172.21.15.37:0/78287174 wait complete. 2024-04-26T00:33:39.378 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:33:39.378 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-26T00:33:39.378 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:33:39.378 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:33:39.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.511+0000 7fa6ae3f8640 10 monclient: get_monmap_and_config 2024-04-26T00:33:39.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.511+0000 7fa6ae3f8640 10 monclient: build_initial_monmap 2024-04-26T00:33:39.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.511+0000 7fa6ae3f8640 10 monclient: monmap: 2024-04-26T00:33:39.505 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:33:39.505 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:39.505 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:33:39.513303+0000 2024-04-26T00:33:39.505 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:33:39.513303+0000 2024-04-26T00:33:39.505 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:33:39.505 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:39.505 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:33:39.505 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:33:39.505 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:33:39.505 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:33:39.505 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:33:39.506 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:39.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.511+0000 7fa6ae3f8640 1 Processor -- start 2024-04-26T00:33:39.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.511+0000 7fa6ae3f8640 1 -- start start 2024-04-26T00:33:39.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.511+0000 7fa6ae3f8640 10 monclient: init 2024-04-26T00:33:39.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.511+0000 7fa6ae3f8640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:39.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.511+0000 7fa6ae3f8640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-04-26T00:33:39.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.511+0000 7fa6ae3f8640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa6a8136f20 0x7fa6a809bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:39.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.511+0000 7fa6ae3f8640 10 monclient(hunting): picked mon.noname-c con 0x7fa6a8136f20 addr v2:172.21.15.45:3300/0 2024-04-26T00:33:39.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.511+0000 7fa6ae3f8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa6a8137300 addr v1:172.21.15.37:6789/0 2024-04-26T00:33:39.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.511+0000 7fa6ae3f8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa6a813c450 addr v1:172.21.15.45:6789/0 2024-04-26T00:33:39.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.511+0000 7fa6ae3f8640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6a80a50f0 con 0x7fa6a8137300 2024-04-26T00:33:39.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.511+0000 7fa6ae3f8640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6a809b330 con 0x7fa6a813c450 2024-04-26T00:33:39.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.511+0000 7fa6ae3f8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:39.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.511+0000 7fa6ae3f8640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fa6a809b610 con 0x7fa6a8136f20 2024-04-26T00:33:39.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.511+0000 7fa6ae3f8640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:39.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.511+0000 7fa6ae3f8640 10 monclient(hunting): authenticate will time out at 1800.819115s 2024-04-26T00:33:39.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.511+0000 7fa6a77fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fa6a8137300 0x7fa6a81377b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:35164/0 (socket says 172.21.15.37:35164) 2024-04-26T00:33:39.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.511+0000 7fa6a77fe640 1 -- 172.21.15.37:0/2974784452 learned_addr learned my addr 172.21.15.37:0/2974784452 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:33:39.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.511+0000 7fa6a7fff640 1 --2- 172.21.15.37:0/2974784452 >> v2:172.21.15.45:3300/0 conn(0x7fa6a8136f20 0x7fa6a809bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:39.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.511+0000 7fa6a7fff640 10 monclient(hunting): get_auth_request con 0x7fa6a8136f20 auth_method 0 2024-04-26T00:33:39.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.511+0000 7fa6a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:39.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.511+0000 7fa6a7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:39.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.511+0000 7fa6a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:39.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.511+0000 7fa6a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:39.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.511+0000 7fa6a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:39.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.511+0000 7fa6a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:39.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.515+0000 7fa6a77fe640 1 -- 172.21.15.37:0/2974784452 >> v1:172.21.15.37:6789/0 conn(0x7fa6a8137300 legacy=0x7fa6a81377b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:39.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.515+0000 7fa6a6ffd640 1 -- 172.21.15.37:0/2974784452 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1234051136 0 0) 0x7fa6a80a50f0 con 0x7fa6a8137300 2024-04-26T00:33:39.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.515+0000 7fa6a6ffd640 10 monclient(hunting): _negotiate 2024-04-26T00:33:39.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.515+0000 7fa6a6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:39.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.515+0000 7fa6a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:39.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.515+0000 7fa6a6ffd640 10 monclient(hunting): my global_id is 144321 2024-04-26T00:33:39.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.515+0000 7fa6a6ffd640 1 -- 172.21.15.37:0/2974784452 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa694003530 con 0x7fa6a8137300 2024-04-26T00:33:39.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.515+0000 7fa6ac995640 1 -- 172.21.15.37:0/2974784452 >> v1:172.21.15.45:6789/0 conn(0x7fa6a813c450 legacy=0x7fa6a813e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:39.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.515+0000 7fa6a7fff640 10 monclient(hunting): handle_auth_done global_id 164119 payload 274 2024-04-26T00:33:39.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.515+0000 7fa6a7fff640 1 -- 172.21.15.37:0/2974784452 >> v1:172.21.15.45:6789/0 conn(0x7fa6a813c450 legacy=0x7fa6a813e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:39.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.515+0000 7fa6a7fff640 1 -- 172.21.15.37:0/2974784452 >> v1:172.21.15.37:6789/0 conn(0x7fa6a8137300 legacy=0x7fa6a81377b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:39.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.515+0000 7fa6a6ffd640 1 -- 172.21.15.37:0/2974784452 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1514590388 0 0) 0x7fa694003530 con 0x7fa6a8137300 2024-04-26T00:33:39.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.515+0000 7fa6a7fff640 10 monclient: _finish_hunting 0 2024-04-26T00:33:39.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.515+0000 7fa6a7fff640 1 monclient: found mon.noname-c 2024-04-26T00:33:39.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.515+0000 7fa6a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:39.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.515+0000 7fa6a7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:33:39.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.515+0000 7fa6a7fff640 1 -- 172.21.15.37:0/2974784452 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6a809b8f0 con 0x7fa6a8136f20 2024-04-26T00:33:39.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.515+0000 7fa6a6ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-26T00:33:39.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.519+0000 7fa6a7fff640 1 -- 172.21.15.37:0/2974784452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6a8136f20 msgr2=0x7fa6a809bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:39.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.519+0000 7fa6a7fff640 1 --2- 172.21.15.37:0/2974784452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6a8136f20 0x7fa6a809bff0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa6980098c0 tx=0x7fa69801c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16f5fc2c0325c1ec server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:39.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.519+0000 7fa6a6ffd640 1 -- 172.21.15.37:0/2974784452 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6980b1070 con 0x7fa6a8136f20 2024-04-26T00:33:39.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.519+0000 7fa6a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:39.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.519+0000 7fa6a6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:33:39.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.519+0000 7fa6a6ffd640 10 monclient: dump: 2024-04-26T00:33:39.516 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:39.516 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:39.516 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:39.516 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:39.516 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:39.516 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:39.516 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:39.516 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:39.516 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:39.516 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:39.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.519+0000 7fa6a6ffd640 10 monclient: _finish_auth 0 2024-04-26T00:33:39.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.519+0000 7fa6a6ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:33:39.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.519+0000 7fa6a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:39.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.519+0000 7fa6a6ffd640 1 -- 172.21.15.37:0/2974784452 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6980ac410 con 0x7fa6a8136f20 2024-04-26T00:33:39.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.519+0000 7fa6ae3f8640 5 monclient: authenticate success, global_id 164119 2024-04-26T00:33:39.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.519+0000 7fa6ae3f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:33:39.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.519+0000 7fa6a6ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:33:39.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.519+0000 7fa6ae3f8640 10 monclient: get_monmap_and_config success 2024-04-26T00:33:39.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.519+0000 7fa6ae3f8640 4 set_mon_vals no callback set 2024-04-26T00:33:39.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.519+0000 7fa6a6ffd640 1 -- 172.21.15.37:0/2974784452 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa690001cb0 con 0x7fa6a8136f20 2024-04-26T00:33:39.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.519+0000 7fa6a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:39.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.519+0000 7fa6a6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:33:39.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.519+0000 7fa6a6ffd640 10 monclient: dump: 2024-04-26T00:33:39.517 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:39.517 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:39.517 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:39.518 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:39.518 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:39.518 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:39.518 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:39.518 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:39.518 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:39.518 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:39.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.519+0000 7fa6ae3f8640 10 monclient: shutdown 2024-04-26T00:33:39.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.519+0000 7fa6ae3f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:33:39.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.519+0000 7fa6ae3f8640 1 -- 172.21.15.37:0/2974784452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6a8136f20 msgr2=0x7fa6a809bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:39.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.519+0000 7fa6ae3f8640 1 --2- 172.21.15.37:0/2974784452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6a8136f20 0x7fa6a809bff0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa6980098c0 tx=0x7fa69801c410 comp rx=0 tx=0).stop 2024-04-26T00:33:39.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.519+0000 7fa6ae3f8640 1 -- 172.21.15.37:0/2974784452 shutdown_connections 2024-04-26T00:33:39.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.519+0000 7fa6ae3f8640 1 --2- 172.21.15.37:0/2974784452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6a8136f20 0x7fa6a809bff0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:39.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.519+0000 7fa6ae3f8640 1 -- 172.21.15.37:0/2974784452 >> 172.21.15.37:0/2974784452 conn(0x7fa6a80a61b0 msgr2=0x7fa6a80a65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:33:39.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6ae3f8640 1 -- 172.21.15.37:0/2974784452 shutdown_connections 2024-04-26T00:33:39.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6ae3f8640 1 -- 172.21.15.37:0/2974784452 wait complete. 2024-04-26T00:33:39.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6ae3f8640 10 monclient: build_initial_monmap 2024-04-26T00:33:39.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6ae3f8640 10 monclient: monmap: 2024-04-26T00:33:39.524 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:33:39.524 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:33:39.524 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:33:39.524 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:33:39.524 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:33:39.524 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:33:39.524 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:33:39.524 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:33:39.524 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:33:39.524 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:39.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6ae3f8640 1 Processor -- start 2024-04-26T00:33:39.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6ae3f8640 1 -- start start 2024-04-26T00:33:39.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6ae3f8640 10 monclient: init 2024-04-26T00:33:39.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6ae3f8640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:39.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6ae3f8640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:33:39.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6ae3f8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6a8136f20 0x7fa6a812be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:39.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6ae3f8640 10 monclient(hunting): picked mon.noname-b con 0x7fa6a8136f20 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:33:39.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6ae3f8640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6a8137300 0x7fa6a812c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:39.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6ae3f8640 10 monclient(hunting): picked mon.noname-a con 0x7fa6a8137300 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:33:39.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6ae3f8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa6a813c450 0x7fa6a80a0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:39.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6ae3f8640 10 monclient(hunting): picked mon.noname-c con 0x7fa6a813c450 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:33:39.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6ae3f8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:39.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6ae3f8640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa6a8142b10 con 0x7fa6a8137300 2024-04-26T00:33:39.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6ae3f8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:39.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6ae3f8640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa6a8142990 con 0x7fa6a8136f20 2024-04-26T00:33:39.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6ae3f8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:39.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6ae3f8640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa6a8142c90 con 0x7fa6a813c450 2024-04-26T00:33:39.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6ae3f8640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:39.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6ae3f8640 10 monclient(hunting): authenticate will time out at 1800.829894s 2024-04-26T00:33:39.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6a77fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6a8137300 0x7fa6a812c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:39.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6ac995640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa6a813c450 0x7fa6a80a0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:39.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6a77fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6a8137300 0x7fa6a812c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:37714/0 (socket says 172.21.15.37:37714) 2024-04-26T00:33:39.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6a77fe640 1 -- 172.21.15.37:0/1628183740 learned_addr learned my addr 172.21.15.37:0/1628183740 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:33:39.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6a77fe640 10 monclient(hunting): get_auth_request con 0x7fa6a8137300 auth_method 0 2024-04-26T00:33:39.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:39.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6a77fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:39.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:39.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6ac995640 10 monclient(hunting): get_auth_request con 0x7fa6a813c450 auth_method 0 2024-04-26T00:33:39.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6ac995640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:39.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6ac995640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:39.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6ac995640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:39.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6a77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:39.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6a77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:39.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6a77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:39.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6ac995640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:39.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6ac995640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:39.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6ac995640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:39.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6a7fff640 1 --2- 172.21.15.37:0/1628183740 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6a8136f20 0x7fa6a812be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:39.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6a7fff640 10 monclient(hunting): get_auth_request con 0x7fa6a8136f20 auth_method 0 2024-04-26T00:33:39.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:39.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6a7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:39.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:39.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6a77fe640 10 monclient(hunting): handle_auth_done global_id 144324 payload 824 2024-04-26T00:33:39.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6a77fe640 1 -- 172.21.15.37:0/1628183740 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa6a813c450 msgr2=0x7fa6a80a0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:39.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6a77fe640 1 --2- 172.21.15.37:0/1628183740 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa6a813c450 0x7fa6a80a0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:39.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6a77fe640 1 -- 172.21.15.37:0/1628183740 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6a8136f20 msgr2=0x7fa6a812be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:39.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6a77fe640 1 --2- 172.21.15.37:0/1628183740 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa6a8136f20 0x7fa6a812be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:39.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6a77fe640 10 monclient: _finish_hunting 0 2024-04-26T00:33:39.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6a77fe640 1 monclient: found mon.noname-a 2024-04-26T00:33:39.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:39.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6a77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:33:39.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.523+0000 7fa6a77fe640 1 -- 172.21.15.37:0/1628183740 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6a809dd30 con 0x7fa6a8137300 2024-04-26T00:33:39.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6a77fe640 1 -- 172.21.15.37:0/1628183740 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6a8137300 msgr2=0x7fa6a812c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:39.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6a77fe640 1 --2- 172.21.15.37:0/1628183740 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6a8137300 0x7fa6a812c390 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa69000ce10 tx=0x7fa69000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:39.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6a4ff9640 1 -- 172.21.15.37:0/1628183740 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa690017070 con 0x7fa6a8137300 2024-04-26T00:33:39.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6a4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:39.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6a4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:33:39.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6a4ff9640 10 monclient: dump: 2024-04-26T00:33:39.529 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:39.529 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:39.529 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:39.529 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:39.529 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:39.529 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:39.529 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:39.530 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:39.530 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:39.530 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:39.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6a4ff9640 10 monclient: _finish_auth 0 2024-04-26T00:33:39.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6a4ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:33:39.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6a4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:39.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6ae3f8640 5 monclient: authenticate success, global_id 144324 2024-04-26T00:33:39.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6a4ff9640 1 -- 172.21.15.37:0/1628183740 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa690012600 con 0x7fa6a8137300 2024-04-26T00:33:39.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6a4ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:33:39.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6a4ff9640 1 -- 172.21.15.37:0/1628183740 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6900129f0 con 0x7fa6a8137300 2024-04-26T00:33:39.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6ae3f8640 10 monclient: _renew_subs 2024-04-26T00:33:39.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6ae3f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:33:39.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6ae3f8640 1 -- 172.21.15.37:0/1628183740 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6a809e5d0 con 0x7fa6a8137300 2024-04-26T00:33:39.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6a4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:39.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6a4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:33:39.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6a4ff9640 10 monclient: dump: 2024-04-26T00:33:39.531 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:39.531 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:39.531 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:39.531 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:39.531 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:39.531 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:39.532 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:39.532 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:39.532 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:39.532 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:39.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6a5ffb640 4 set_mon_vals no callback set 2024-04-26T00:33:39.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6ae3f8640 10 monclient: _renew_subs 2024-04-26T00:33:39.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6ae3f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:33:39.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6ae3f8640 1 -- 172.21.15.37:0/1628183740 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6a809e010 con 0x7fa6a8137300 2024-04-26T00:33:39.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6ae3f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:39.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6ae3f8640 1 --2- 172.21.15.37:0/1628183740 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6a812c8d0 0x7fa6a80a1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:39.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6ae3f8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:39.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6ae3f8640 1 -- 172.21.15.37:0/1628183740 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa6a804f510 con 0x7fa6a812c8d0 2024-04-26T00:33:39.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6a7fff640 1 --2- 172.21.15.37:0/1628183740 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6a812c8d0 0x7fa6a80a1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:39.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6a7fff640 10 monclient: get_auth_request con 0x7fa6a812c8d0 auth_method 0 2024-04-26T00:33:39.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:39.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6a7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:39.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:39.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:39.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:39.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:39.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6a4ff9640 1 -- 172.21.15.37:0/1628183740 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa69001a070 con 0x7fa6a8137300 2024-04-26T00:33:39.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6a7fff640 10 monclient(hunting): handle_auth_done global_id 144327 payload 824 2024-04-26T00:33:39.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6a7fff640 1 -- 172.21.15.37:0/1628183740 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa6a80a30a0 con 0x7fa6a812c8d0 2024-04-26T00:33:39.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6a7fff640 1 -- 172.21.15.37:0/1628183740 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6a812c8d0 msgr2=0x7fa6a80a1380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:33:39.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6a7fff640 1 -- 172.21.15.37:0/1628183740 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6a812c8d0 msgr2=0x7fa6a80a1380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:33:39.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6a7fff640 1 --2- 172.21.15.37:0/1628183740 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6a812c8d0 0x7fa6a80a1380 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7fa6a8137ec0 tx=0x7fa69801ac50 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:33:39.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6a7fff640 1 --2- 172.21.15.37:0/1628183740 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6a812c8d0 0x7fa6a80a1380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:33:39.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6a4ff9640 1 --2- 172.21.15.37:0/1628183740 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa688035e80 0x7fa688038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:39.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6a4ff9640 1 -- 172.21.15.37:0/1628183740 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(533..533 src has 370..533) ==== 3987+0+0 (secure 0 0 0) 0x7fa690048ee0 con 0x7fa6a8137300 2024-04-26T00:33:39.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6ac995640 1 --2- 172.21.15.37:0/1628183740 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa688035e80 0x7fa688038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:39.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.527+0000 7fa6ac995640 10 monclient: get_auth_request con 0x7fa688035e80 auth_method 0 2024-04-26T00:33:39.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.531+0000 7fa6ac995640 1 --2- 172.21.15.37:0/1628183740 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa688035e80 0x7fa688038340 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fa6a0000f70 tx=0x7fa6a0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:39.574 INFO:tasks.rados.rados.0.smithi045.stdout:2365: done (3 left) 2024-04-26T00:33:39.574 INFO:tasks.rados.rados.0.smithi045.stdout:2366: rollback oid 3 current snap is 266 2024-04-26T00:33:39.574 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 3 to 253 2024-04-26T00:33:39.575 INFO:tasks.rados.rados.0.smithi045.stdout:2360: finishing copy_from to smithi04526579-42 2024-04-26T00:33:39.575 INFO:tasks.rados.rados.0.smithi045.stdout:2360: got expected ENOENT (src dne) 2024-04-26T00:33:39.575 INFO:tasks.rados.rados.0.smithi045.stdout:2360: finishing copy_from racing read to smithi04526579-42 2024-04-26T00:33:39.575 INFO:tasks.rados.rados.0.smithi045.stdout:2361: finishing write tid 1 to smithi04526579-18 2024-04-26T00:33:39.575 INFO:tasks.rados.rados.0.smithi045.stdout:2361: finishing write tid 2 to smithi04526579-18 2024-04-26T00:33:39.575 INFO:tasks.rados.rados.0.smithi045.stdout:2361: finishing write tid 3 to smithi04526579-18 2024-04-26T00:33:39.575 INFO:tasks.rados.rados.0.smithi045.stdout:2361: finishing write tid 4 to smithi04526579-18 2024-04-26T00:33:39.575 INFO:tasks.rados.rados.0.smithi045.stdout:2361: oid 18 updating version 0 to 2006 2024-04-26T00:33:39.575 INFO:tasks.rados.rados.0.smithi045.stdout:2361: oid 18 version 2006 is already newer than 2005 2024-04-26T00:33:39.575 INFO:tasks.rados.rados.0.smithi045.stdout:2361: oid 18 version 2006 is already newer than 2004 2024-04-26T00:33:39.575 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 2006 (ObjNum 862 snap 266 seq_num 862) dirty exists 2024-04-26T00:33:39.575 INFO:tasks.rados.rados.0.smithi045.stdout:2361: left oid 18 (ObjNum 862 snap 266 seq_num 862) 2024-04-26T00:33:39.576 INFO:tasks.rados.rados.0.smithi045.stdout:2364: finishing rollback tid 0 to smithi04526579-36 2024-04-26T00:33:39.576 INFO:tasks.rados.rados.0.smithi045.stdout:2364: finishing rollback tid 1 to smithi04526579-36 2024-04-26T00:33:39.576 INFO:tasks.rados.rados.0.smithi045.stdout:2364: finishing rollback tid 2 to smithi04526579-36 2024-04-26T00:33:39.576 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 1465 (ObjNum 703 snap 221 seq_num 703) dirty exists 2024-04-26T00:33:39.576 INFO:tasks.rados.rados.0.smithi045.stdout:2360: done (3 left) 2024-04-26T00:33:39.576 INFO:tasks.rados.rados.0.smithi045.stdout:2361: done (2 left) 2024-04-26T00:33:39.576 INFO:tasks.rados.rados.0.smithi045.stdout:2364: done (1 left) 2024-04-26T00:33:39.576 INFO:tasks.rados.rados.0.smithi045.stdout:2367: snap_create 2024-04-26T00:33:39.582 INFO:tasks.rados.rados.0.smithi045.stdout:2366: finishing rollback tid 1 to smithi04526579-3 2024-04-26T00:33:39.583 INFO:tasks.rados.rados.0.smithi045.stdout:2366: finishing rollback tid 2 to smithi04526579-3 2024-04-26T00:33:39.583 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 1433 (ObjNum 790 snap 241 seq_num 790) dirty exists 2024-04-26T00:33:39.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.731+0000 7fa6a7fff640 1 --2- 172.21.15.37:0/1628183740 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6a812c8d0 0x7fa6a80a1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:39.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.731+0000 7fa6a7fff640 10 monclient: get_auth_request con 0x7fa6a812c8d0 auth_method 0 2024-04-26T00:33:39.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.731+0000 7fa6a7fff640 10 monclient: get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:39.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.731+0000 7fa6a7fff640 10 monclient: _init_auth method 2 2024-04-26T00:33:39.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.731+0000 7fa6a7fff640 10 monclient: _init_auth already have auth, reseting 2024-04-26T00:33:39.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.731+0000 7fa6a7fff640 10 monclient: handle_auth_reply_more payload 9 2024-04-26T00:33:39.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.731+0000 7fa6a7fff640 10 monclient: handle_auth_reply_more payload_len 9 2024-04-26T00:33:39.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.731+0000 7fa6a7fff640 10 monclient: handle_auth_reply_more responding with 132 bytes 2024-04-26T00:33:39.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.731+0000 7fa6a7fff640 10 monclient: handle_auth_done global_id 144327 payload 293 2024-04-26T00:33:39.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.731+0000 7fa6a7fff640 1 -- 172.21.15.37:0/1628183740 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6a812c8d0 msgr2=0x7fa6a80a1380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:39.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.731+0000 7fa6a7fff640 1 --2- 172.21.15.37:0/1628183740 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6a812c8d0 0x7fa6a80a1380 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa69801c450 tx=0x7fa698008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:39.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.731+0000 7fa6a4ff9640 1 -- 172.21.15.37:0/1628183740 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6980b1040 con 0x7fa6a812c8d0 2024-04-26T00:33:39.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.731+0000 7fa6a4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:39.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.731+0000 7fa6a4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:33:39.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.731+0000 7fa6a4ff9640 10 monclient: dump: 2024-04-26T00:33:39.727 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:39.727 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:39.727 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:39.727 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:39.728 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:39.728 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:39.728 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:39.728 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:39.728 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:39.728 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:39.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.735+0000 7fa6a4ff9640 1 -- 172.21.15.37:0/1628183740 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa6980acd30 con 0x7fa6a812c8d0 2024-04-26T00:33:39.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.735+0000 7fa6a4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:39.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.735+0000 7fa6a4ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:33:39.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.735+0000 7fa6a4ff9640 1 -- 172.21.15.37:0/1628183740 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6a812c8d0 msgr2=0x7fa6a80a1380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:39.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.735+0000 7fa6a4ff9640 1 --2- 172.21.15.37:0/1628183740 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6a812c8d0 0x7fa6a80a1380 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa69801c450 tx=0x7fa698008040 comp rx=0 tx=0).stop 2024-04-26T00:33:39.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.735+0000 7fa6a4ff9640 1 -- 172.21.15.37:0/1628183740 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6a812c8d0 msgr2=0x7fa6a80a1380 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:33:39.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.735+0000 7fa6a4ff9640 1 --2- 172.21.15.37:0/1628183740 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6a812c8d0 0x7fa6a80a1380 secure :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa69801c450 tx=0x7fa698008040 comp rx=0 tx=0).stop 2024-04-26T00:33:39.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.743+0000 7fa6ae3f8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:33:39.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.743+0000 7fa6ae3f8640 1 --2- 172.21.15.37:0/1628183740 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6a80a18c0 0x7fa6a80a1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:39.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.743+0000 7fa6ae3f8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:39.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.743+0000 7fa6ae3f8640 1 -- 172.21.15.37:0/1628183740 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa6a80a2210 con 0x7fa6a80a18c0 2024-04-26T00:33:39.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.743+0000 7fa6a7fff640 1 --2- 172.21.15.37:0/1628183740 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6a80a18c0 0x7fa6a80a1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:39.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.743+0000 7fa6a7fff640 10 monclient: get_auth_request con 0x7fa6a80a18c0 auth_method 0 2024-04-26T00:33:39.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.743+0000 7fa6a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:39.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.743+0000 7fa6a7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:39.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.743+0000 7fa6a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:39.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.743+0000 7fa6a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:39.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.743+0000 7fa6a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:39.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.743+0000 7fa6a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:39.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.743+0000 7fa6a7fff640 10 monclient(hunting): handle_auth_done global_id 144330 payload 824 2024-04-26T00:33:39.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.743+0000 7fa6a7fff640 1 -- 172.21.15.37:0/1628183740 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa6a8075f10 con 0x7fa6a80a18c0 2024-04-26T00:33:39.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.747+0000 7fa6a7fff640 1 -- 172.21.15.37:0/1628183740 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6a80a18c0 msgr2=0x7fa6a80a1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:39.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.747+0000 7fa6a7fff640 1 --2- 172.21.15.37:0/1628183740 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6a80a18c0 0x7fa6a80a1cd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa69801a040 tx=0x7fa6980191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:39.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.747+0000 7fa6a4ff9640 1 -- 172.21.15.37:0/1628183740 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6980b1040 con 0x7fa6a80a18c0 2024-04-26T00:33:39.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.747+0000 7fa6a4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:39.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.747+0000 7fa6a4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:33:39.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.747+0000 7fa6a4ff9640 10 monclient: dump: 2024-04-26T00:33:39.741 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:39.742 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:39.742 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:39.742 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:39.742 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:39.742 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:39.742 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:39.742 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:39.742 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:39.742 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:39.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.747+0000 7fa6a4ff9640 1 -- 172.21.15.37:0/1628183740 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4890 (secure 0 0 0) 0x7fa698003a40 con 0x7fa6a80a18c0 2024-04-26T00:33:39.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.747+0000 7fa6a4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:33:39.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.747+0000 7fa6a4ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:33:39.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.747+0000 7fa6a4ff9640 1 -- 172.21.15.37:0/1628183740 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6a80a18c0 msgr2=0x7fa6a80a1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:39.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.747+0000 7fa6a4ff9640 1 --2- 172.21.15.37:0/1628183740 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6a80a18c0 0x7fa6a80a1cd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa69801a040 tx=0x7fa6980191b0 comp rx=0 tx=0).stop 2024-04-26T00:33:39.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.747+0000 7fa6a4ff9640 1 -- 172.21.15.37:0/1628183740 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6a80a18c0 msgr2=0x7fa6a80a1cd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:33:39.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.747+0000 7fa6a4ff9640 1 --2- 172.21.15.37:0/1628183740 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6a80a18c0 0x7fa6a80a1cd0 secure :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa69801a040 tx=0x7fa6980191b0 comp rx=0 tx=0).stop 2024-04-26T00:33:39.744 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:33:39.744 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:33:39.744 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:33:39.744 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:33:39.744 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 102, 2024-04-26T00:33:39.744 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 106290, 2024-04-26T00:33:39.744 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:33:39.744 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:33:39.744 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:33:39.744 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:33:39.744 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:39.745 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 3, 2024-04-26T00:33:39.745 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:33:39.745 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:33:39.745 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:33:39.745 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:33:39.745 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:33:39.745 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:33:39.745 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:33:39.745 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:33:39.745 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:33:39.745 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:33:39.745 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:33:39.745 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:33:39.745 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:33:39.746 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:33:39.746 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:39.746 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:33:39.746 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:33:39.746 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:33:39.746 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:33:39.746 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:33:39.746 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:33:39.746 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:33:39.746 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:33:39.746 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:33:39.746 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:33:39.746 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:33:39.746 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:33:39.747 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:33:39.747 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:39.747 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:39.747 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:33:39.747 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:33:39.747 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:33:39.747 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:33:39.747 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:33:39.747 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:33:39.747 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:33:39.747 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:33:39.747 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:33:39.747 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:33:39.747 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:33:39.748 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:33:39.748 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:33:39.748 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:33:39.748 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:33:39.748 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:33:39.748 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:33:39.748 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:33:39.748 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:33:39.748 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:33:39.748 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:33:39.748 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:33:39.748 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:33:39.748 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:33:39.748 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:33:39.749 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:33:39.749 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:33:39.749 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:33:39.749 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:39.749 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:33:39.749 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:39.749 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:33:39.749 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:39.749 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:33:39.749 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:33:39.749 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:33:39.749 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:33:39.749 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:39.749 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:33:39.749 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:33:39.750 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:39.750 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:39.750 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:39.750 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:33:39.750 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:33:39.750 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:39.750 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:39.750 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:39.750 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:39.750 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:33:39.750 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:33:39.750 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:33:39.750 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:33:39.751 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:33:39.751 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:39.751 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:39.751 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:33:39.751 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:33:39.751 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:33:39.751 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:33:39.751 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:39.751 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:33:39.751 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:33:39.751 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:39.751 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:39.751 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:39.751 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:33:39.752 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:33:39.752 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:39.752 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:39.752 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:39.752 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:39.752 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:33:39.752 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:33:39.752 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:33:39.752 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:33:39.752 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:33:39.752 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:39.752 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:39.752 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:33:39.752 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:33:39.753 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:33:39.753 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:33:39.753 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:39.753 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:33:39.753 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:33:39.753 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:39.753 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:39.753 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:39.753 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:33:39.753 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:33:39.753 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:39.753 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:39.754 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:39.754 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:39.754 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:33:39.754 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:33:39.754 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:33:39.754 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:33:39.754 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:33:39.754 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:39.754 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:39.754 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:39.754 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:33:39.754 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:33:39.754 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:39.755 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:33:39.755 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:33:39.755 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:33:39.755 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:39.755 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:39.755 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:33:39.755 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:39.755 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:33:39.755 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:33:39.755 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:33:39.755 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:39.755 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:39.755 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:33:39.755 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:39.755 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:33:39.756 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:33:39.756 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:33:39.756 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:39.756 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:39.756 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:33:39.756 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:39.756 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:33:39.756 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:33:39.756 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:33:39.756 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:39.756 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:39.756 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:39.756 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:33:39.756 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:33:39.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.747+0000 7fa6ae3f8640 1 -- 172.21.15.37:0/1628183740 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa688035e80 msgr2=0x7fa688038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:39.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.747+0000 7fa6ae3f8640 1 --2- 172.21.15.37:0/1628183740 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa688035e80 0x7fa688038340 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fa6a0000f70 tx=0x7fa6a0009040 comp rx=0 tx=0).stop 2024-04-26T00:33:39.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.747+0000 7fa6ae3f8640 10 monclient: shutdown 2024-04-26T00:33:39.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.747+0000 7fa6ae3f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:33:39.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.747+0000 7fa6ae3f8640 1 -- 172.21.15.37:0/1628183740 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6a8137300 msgr2=0x7fa6a812c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:39.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.747+0000 7fa6ae3f8640 1 --2- 172.21.15.37:0/1628183740 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa6a8137300 0x7fa6a812c390 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa69000ce10 tx=0x7fa69000a430 comp rx=0 tx=0).stop 2024-04-26T00:33:39.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.747+0000 7fa6ac995640 1 -- 172.21.15.37:0/1628183740 reap_dead start 2024-04-26T00:33:39.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.747+0000 7fa6ac995640 1 -- 172.21.15.37:0/1628183740 reap_dead start 2024-04-26T00:33:39.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.747+0000 7fa6ae3f8640 1 -- 172.21.15.37:0/1628183740 shutdown_connections 2024-04-26T00:33:39.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.747+0000 7fa6ae3f8640 1 -- 172.21.15.37:0/1628183740 >> 172.21.15.37:0/1628183740 conn(0x7fa6a80a61b0 msgr2=0x7fa6a80a8090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:33:39.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.747+0000 7fa6ae3f8640 1 -- 172.21.15.37:0/1628183740 shutdown_connections 2024-04-26T00:33:39.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.747+0000 7fa6ae3f8640 1 -- 172.21.15.37:0/1628183740 wait complete. 2024-04-26T00:33:39.770 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:33:39.770 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:33:39.770 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:33:39.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.907+0000 7fd06d87e640 10 monclient: get_monmap_and_config 2024-04-26T00:33:39.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.907+0000 7fd06d87e640 10 monclient: build_initial_monmap 2024-04-26T00:33:39.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.907+0000 7fd06d87e640 10 monclient: monmap: 2024-04-26T00:33:39.903 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:33:39.903 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:39.903 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:33:39.911798+0000 2024-04-26T00:33:39.904 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:33:39.911798+0000 2024-04-26T00:33:39.904 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:33:39.904 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:39.904 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:33:39.904 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:33:39.904 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:33:39.904 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:33:39.904 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:33:39.904 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:39.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd06d87e640 1 Processor -- start 2024-04-26T00:33:39.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd06d87e640 1 -- start start 2024-04-26T00:33:39.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd06d87e640 10 monclient: init 2024-04-26T00:33:39.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd06d87e640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:39.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd06d87e640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-04-26T00:33:39.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd06d87e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd0680a0300 0x7fd06809fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:39.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd06d87e640 10 monclient(hunting): picked mon.noname-c con 0x7fd0680a0300 addr v2:172.21.15.45:3300/0 2024-04-26T00:33:39.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd06d87e640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd0680a06e0 0x7fd068134ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:39.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd06d87e640 10 monclient(hunting): picked mon.noname-a con 0x7fd0680a06e0 addr v2:172.21.15.37:3300/0 2024-04-26T00:33:39.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd06d87e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0681350e0 0x7fd068135550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:39.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd06d87e640 10 monclient(hunting): picked mon.noname-b con 0x7fd0681350e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:33:39.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd06d87e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:39.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd06d87e640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fd0680a13f0 con 0x7fd0680a06e0 2024-04-26T00:33:39.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd06d87e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:39.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd06d87e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fd06809c190 con 0x7fd0680a0300 2024-04-26T00:33:39.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd06d87e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:39.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd06d87e640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd0680a1110 con 0x7fd0681350e0 2024-04-26T00:33:39.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd06d87e640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:39.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd06d87e640 10 monclient(hunting): authenticate will time out at 1801.217475s 2024-04-26T00:33:39.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd0677fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0681350e0 0x7fd068135550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:39.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd0667fc640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd0680a06e0 0x7fd068134ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:39.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd0677fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0681350e0 0x7fd068135550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:37286/0 (socket says 172.21.15.37:37286) 2024-04-26T00:33:39.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd0677fe640 1 -- 172.21.15.37:0/2903793094 learned_addr learned my addr 172.21.15.37:0/2903793094 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:33:39.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd0677fe640 10 monclient(hunting): get_auth_request con 0x7fd0681350e0 auth_method 0 2024-04-26T00:33:39.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd0677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:39.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd0677fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:39.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd0677fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:39.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd0667fc640 10 monclient(hunting): get_auth_request con 0x7fd0680a06e0 auth_method 0 2024-04-26T00:33:39.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd0667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:39.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd0667fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:39.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd0667fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:39.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd0677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:39.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd0677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:39.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd0677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:39.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd0667fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:39.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd0667fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:39.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd0667fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:39.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd066ffd640 1 --2- 172.21.15.37:0/2903793094 >> v2:172.21.15.45:3300/0 conn(0x7fd0680a0300 0x7fd06809fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:39.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd066ffd640 10 monclient(hunting): get_auth_request con 0x7fd0680a0300 auth_method 0 2024-04-26T00:33:39.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd066ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:39.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd066ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:39.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd066ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:39.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd0677fe640 10 monclient(hunting): handle_auth_done global_id 154202 payload 274 2024-04-26T00:33:39.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd0677fe640 1 -- 172.21.15.37:0/2903793094 >> v2:172.21.15.45:3300/0 conn(0x7fd0680a0300 msgr2=0x7fd06809fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:33:39.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd0677fe640 1 --2- 172.21.15.37:0/2903793094 >> v2:172.21.15.45:3300/0 conn(0x7fd0680a0300 0x7fd06809fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:39.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd0677fe640 1 -- 172.21.15.37:0/2903793094 >> v2:172.21.15.37:3300/0 conn(0x7fd0680a06e0 msgr2=0x7fd068134ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:33:39.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd0677fe640 1 --2- 172.21.15.37:0/2903793094 >> v2:172.21.15.37:3300/0 conn(0x7fd0680a06e0 0x7fd068134ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:39.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd0677fe640 10 monclient: _finish_hunting 0 2024-04-26T00:33:39.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd0677fe640 1 monclient: found mon.noname-b 2024-04-26T00:33:39.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd0677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:39.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd0677fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:33:39.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd0677fe640 1 -- 172.21.15.37:0/2903793094 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0680a16d0 con 0x7fd0681350e0 2024-04-26T00:33:39.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.911+0000 7fd0667fc640 1 --2- 172.21.15.37:0/2903793094 >> v2:172.21.15.37:3300/0 conn(0x7fd0680a06e0 0x7fd068134ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:33:39.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.915+0000 7fd0677fe640 1 -- 172.21.15.37:0/2903793094 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0681350e0 msgr2=0x7fd068135550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:39.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.915+0000 7fd0677fe640 1 --2- 172.21.15.37:0/2903793094 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0681350e0 0x7fd068135550 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd0680764d0 tx=0x7fd06001c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fccb5c03a2e27a99 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:39.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.915+0000 7fd065ffb640 1 -- 172.21.15.37:0/2903793094 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0600a5640 con 0x7fd0681350e0 2024-04-26T00:33:39.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.915+0000 7fd065ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:39.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.915+0000 7fd065ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:33:39.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.915+0000 7fd065ffb640 10 monclient: dump: 2024-04-26T00:33:39.912 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:39.912 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:39.912 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:39.912 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:39.912 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:39.912 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:39.913 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:39.913 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:39.913 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:39.913 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:39.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.915+0000 7fd065ffb640 10 monclient: _finish_auth 0 2024-04-26T00:33:39.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.915+0000 7fd065ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:33:39.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.915+0000 7fd065ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:39.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.915+0000 7fd065ffb640 1 -- 172.21.15.37:0/2903793094 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0600a57e0 con 0x7fd0681350e0 2024-04-26T00:33:39.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.915+0000 7fd065ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:33:39.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.915+0000 7fd065ffb640 1 -- 172.21.15.37:0/2903793094 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0600a5bb0 con 0x7fd0681350e0 2024-04-26T00:33:39.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.915+0000 7fd06d87e640 5 monclient: authenticate success, global_id 154202 2024-04-26T00:33:39.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.915+0000 7fd06d87e640 10 monclient: get_monmap_and_config success 2024-04-26T00:33:39.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.915+0000 7fd06d87e640 4 set_mon_vals no callback set 2024-04-26T00:33:39.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.915+0000 7fd065ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:39.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.915+0000 7fd065ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:33:39.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.915+0000 7fd065ffb640 10 monclient: dump: 2024-04-26T00:33:39.914 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:39.914 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:39.914 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:39.914 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:39.914 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:39.914 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:39.914 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:39.914 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:39.914 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:39.914 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:39.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.915+0000 7fd06d87e640 10 monclient: shutdown 2024-04-26T00:33:39.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.915+0000 7fd06d87e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:33:39.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.915+0000 7fd06d87e640 1 -- 172.21.15.37:0/2903793094 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0681350e0 msgr2=0x7fd068135550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:39.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.915+0000 7fd06d87e640 1 --2- 172.21.15.37:0/2903793094 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0681350e0 0x7fd068135550 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd0680764d0 tx=0x7fd06001c8b0 comp rx=0 tx=0).stop 2024-04-26T00:33:39.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.915+0000 7fd06d87e640 1 -- 172.21.15.37:0/2903793094 shutdown_connections 2024-04-26T00:33:39.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.915+0000 7fd06d87e640 1 --2- 172.21.15.37:0/2903793094 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0681350e0 0x7fd068135550 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:39.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.915+0000 7fd06d87e640 1 --2- 172.21.15.37:0/2903793094 >> v2:172.21.15.37:3300/0 conn(0x7fd0680a06e0 0x7fd068134ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:39.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.915+0000 7fd06d87e640 1 --2- 172.21.15.37:0/2903793094 >> v2:172.21.15.45:3300/0 conn(0x7fd0680a0300 0x7fd06809fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:39.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd06d87e640 1 -- 172.21.15.37:0/2903793094 >> 172.21.15.37:0/2903793094 conn(0x7fd0680a1d50 msgr2=0x7fd0680a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:33:39.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd06d87e640 1 -- 172.21.15.37:0/2903793094 shutdown_connections 2024-04-26T00:33:39.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd06d87e640 1 -- 172.21.15.37:0/2903793094 wait complete. 2024-04-26T00:33:39.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd06d87e640 10 monclient: build_initial_monmap 2024-04-26T00:33:39.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd06d87e640 10 monclient: monmap: 2024-04-26T00:33:39.915 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:33:39.915 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:33:39.915 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:33:39.915 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:33:39.915 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:33:39.916 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:33:39.916 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:33:39.916 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:33:39.916 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:33:39.916 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:39.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd06d87e640 1 Processor -- start 2024-04-26T00:33:39.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd06d87e640 1 -- start start 2024-04-26T00:33:39.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd06d87e640 10 monclient: init 2024-04-26T00:33:39.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd06d87e640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:39.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd06d87e640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:33:39.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd06d87e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0680a0300 0x7fd06812ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:39.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd06d87e640 10 monclient(hunting): picked mon.noname-b con 0x7fd0680a0300 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:33:39.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd06d87e640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd0680a06e0 0x7fd06812f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:39.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd06d87e640 10 monclient(hunting): picked mon.noname-a con 0x7fd0680a06e0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:33:39.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd06d87e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0681350e0 0x7fd0681cce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:39.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd06d87e640 10 monclient(hunting): picked mon.noname-c con 0x7fd0681350e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:33:39.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd06d87e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:39.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd06d87e640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd06813e6f0 con 0x7fd0680a06e0 2024-04-26T00:33:39.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd06d87e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:39.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd06d87e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd06813e870 con 0x7fd0680a0300 2024-04-26T00:33:39.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd06d87e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:39.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd06d87e640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd06813e9f0 con 0x7fd0681350e0 2024-04-26T00:33:39.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd06d87e640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:39.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd06d87e640 10 monclient(hunting): authenticate will time out at 1801.225700s 2024-04-26T00:33:39.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd0677fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0681350e0 0x7fd0681cce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:39.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd0667fc640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd0680a06e0 0x7fd06812f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:39.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd0677fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0681350e0 0x7fd0681cce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:37288/0 (socket says 172.21.15.37:37288) 2024-04-26T00:33:39.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd0677fe640 1 -- 172.21.15.37:0/673684016 learned_addr learned my addr 172.21.15.37:0/673684016 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:33:39.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd0677fe640 10 monclient(hunting): get_auth_request con 0x7fd0681350e0 auth_method 0 2024-04-26T00:33:39.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd0667fc640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd0680a06e0 0x7fd06812f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40128/0 (socket says 172.21.15.37:40128) 2024-04-26T00:33:39.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd0677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:39.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd0677fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:39.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd0677fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:39.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd0667fc640 10 monclient(hunting): get_auth_request con 0x7fd0680a06e0 auth_method 0 2024-04-26T00:33:39.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd0667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:39.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd0667fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:39.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd0667fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:39.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd0677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:39.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd0677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:39.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd0677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:39.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd0667fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:39.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd0667fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:39.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd0667fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:39.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd066ffd640 1 --2- 172.21.15.37:0/673684016 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0680a0300 0x7fd06812ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:39.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd066ffd640 10 monclient(hunting): get_auth_request con 0x7fd0680a0300 auth_method 0 2024-04-26T00:33:39.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd066ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:39.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd066ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:39.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd066ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:39.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd0677fe640 10 monclient(hunting): handle_auth_done global_id 154205 payload 824 2024-04-26T00:33:39.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd0677fe640 1 -- 172.21.15.37:0/673684016 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0680a0300 msgr2=0x7fd06812ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:39.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd0677fe640 1 --2- 172.21.15.37:0/673684016 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0680a0300 0x7fd06812ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:39.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd0677fe640 1 -- 172.21.15.37:0/673684016 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd0680a06e0 msgr2=0x7fd06812f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:39.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd0677fe640 1 --2- 172.21.15.37:0/673684016 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd0680a06e0 0x7fd06812f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:39.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd0677fe640 10 monclient: _finish_hunting 0 2024-04-26T00:33:39.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd0677fe640 1 monclient: found mon.noname-c 2024-04-26T00:33:39.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd0677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:39.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd0677fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:33:39.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.919+0000 7fd0677fe640 1 -- 172.21.15.37:0/673684016 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd068138940 con 0x7fd0681350e0 2024-04-26T00:33:39.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd0677fe640 1 -- 172.21.15.37:0/673684016 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0681350e0 msgr2=0x7fd0681cce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:39.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd0677fe640 1 --2- 172.21.15.37:0/673684016 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0681350e0 0x7fd0681cce60 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd0600af040 tx=0x7fd060019650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:39.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd04ffff640 1 -- 172.21.15.37:0/673684016 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0600bc020 con 0x7fd0681350e0 2024-04-26T00:33:39.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd04ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:39.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd04ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:33:39.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd04ffff640 10 monclient: dump: 2024-04-26T00:33:39.924 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:39.925 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:39.925 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:39.925 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:39.925 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:39.925 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:39.925 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:39.925 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:39.925 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:39.925 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:39.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd04ffff640 10 monclient: _finish_auth 0 2024-04-26T00:33:39.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd04ffff640 10 monclient: _check_auth_tickets 2024-04-26T00:33:39.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd04ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:39.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd04ffff640 1 -- 172.21.15.37:0/673684016 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0600b8880 con 0x7fd0681350e0 2024-04-26T00:33:39.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd06d87e640 5 monclient: authenticate success, global_id 154205 2024-04-26T00:33:39.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd06d87e640 10 monclient: _renew_subs 2024-04-26T00:33:39.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd06d87e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:33:39.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd06d87e640 1 -- 172.21.15.37:0/673684016 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd068071830 con 0x7fd0681350e0 2024-04-26T00:33:39.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd04ffff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:33:39.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd04ffff640 1 -- 172.21.15.37:0/673684016 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0600a5640 con 0x7fd0681350e0 2024-04-26T00:33:39.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd04ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:39.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd04ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:33:39.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd04ffff640 10 monclient: dump: 2024-04-26T00:33:39.926 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:39.926 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:39.926 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:39.926 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:39.926 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:39.926 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:39.926 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:39.927 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:39.927 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:39.927 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:39.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd06d87e640 10 monclient: _renew_subs 2024-04-26T00:33:39.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd0657fa640 4 set_mon_vals no callback set 2024-04-26T00:33:39.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd06d87e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:33:39.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd06d87e640 1 -- 172.21.15.37:0/673684016 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd06813bbb0 con 0x7fd0681350e0 2024-04-26T00:33:39.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd06d87e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:39.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd06d87e640 1 --2- 172.21.15.37:0/673684016 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0681cd3a0 0x7fd0681cd780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:39.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd06d87e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:39.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd06d87e640 1 -- 172.21.15.37:0/673684016 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd0680a0cd0 con 0x7fd0681cd3a0 2024-04-26T00:33:39.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd066ffd640 1 --2- 172.21.15.37:0/673684016 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0681cd3a0 0x7fd0681cd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:39.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd066ffd640 10 monclient: get_auth_request con 0x7fd0681cd3a0 auth_method 0 2024-04-26T00:33:39.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd066ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:39.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd066ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:39.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd066ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:39.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd04ffff640 1 -- 172.21.15.37:0/673684016 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd0600bf020 con 0x7fd0681350e0 2024-04-26T00:33:39.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd066ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:39.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd066ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:39.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd066ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:39.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd066ffd640 10 monclient(hunting): handle_auth_done global_id 154208 payload 824 2024-04-26T00:33:39.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd04ffff640 1 --2- 172.21.15.37:0/673684016 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd054035ed0 0x7fd054038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:39.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd066ffd640 1 -- 172.21.15.37:0/673684016 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd06809f550 con 0x7fd0681cd3a0 2024-04-26T00:33:39.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd04ffff640 1 -- 172.21.15.37:0/673684016 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(534..534 src has 370..534) ==== 3959+0+0 (secure 0 0 0) 0x7fd0600e39a0 con 0x7fd0681350e0 2024-04-26T00:33:39.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd0667fc640 1 --2- 172.21.15.37:0/673684016 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd054035ed0 0x7fd054038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:39.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.923+0000 7fd0667fc640 10 monclient: get_auth_request con 0x7fd054035ed0 auth_method 0 2024-04-26T00:33:39.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.927+0000 7fd066ffd640 1 -- 172.21.15.37:0/673684016 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0681cd3a0 msgr2=0x7fd0681cd780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:39.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.927+0000 7fd066ffd640 1 --2- 172.21.15.37:0/673684016 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0681cd3a0 0x7fd0681cd780 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd05800f510 tx=0x7fd05800f9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:39.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.927+0000 7fd04ffff640 1 -- 172.21.15.37:0/673684016 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd058019030 con 0x7fd0681cd3a0 2024-04-26T00:33:39.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.927+0000 7fd04ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:39.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.927+0000 7fd04ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:33:39.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.927+0000 7fd04ffff640 10 monclient: dump: 2024-04-26T00:33:39.929 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:39.929 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:39.929 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:39.929 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:39.929 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:39.929 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:39.929 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:39.929 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:39.929 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:39.929 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:39.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.927+0000 7fd04ffff640 1 -- 172.21.15.37:0/673684016 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd058004020 con 0x7fd0681cd3a0 2024-04-26T00:33:39.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.927+0000 7fd04ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:39.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.927+0000 7fd04ffff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:33:39.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.927+0000 7fd04ffff640 1 -- 172.21.15.37:0/673684016 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0681cd3a0 msgr2=0x7fd0681cd780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:39.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.927+0000 7fd04ffff640 1 --2- 172.21.15.37:0/673684016 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0681cd3a0 0x7fd0681cd780 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd05800f510 tx=0x7fd05800f9d0 comp rx=0 tx=0).stop 2024-04-26T00:33:39.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.927+0000 7fd04ffff640 1 -- 172.21.15.37:0/673684016 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0681cd3a0 msgr2=0x7fd0681cd780 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:33:39.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.927+0000 7fd04ffff640 1 --2- 172.21.15.37:0/673684016 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0681cd3a0 0x7fd0681cd780 secure :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd05800f510 tx=0x7fd05800f9d0 comp rx=0 tx=0).stop 2024-04-26T00:33:39.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.927+0000 7fd0667fc640 1 --2- 172.21.15.37:0/673684016 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd054035ed0 0x7fd054038390 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fd05c009580 tx=0x7fd05c012040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:39.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.931+0000 7fd06d87e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:33:39.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.931+0000 7fd06d87e640 1 --2- 172.21.15.37:0/673684016 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd06806eb70 0x7fd06806ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:39.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.931+0000 7fd06d87e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:39.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.931+0000 7fd06d87e640 1 -- 172.21.15.37:0/673684016 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd0681cdcc0 con 0x7fd06806eb70 2024-04-26T00:33:39.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.931+0000 7fd066ffd640 1 --2- 172.21.15.37:0/673684016 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd06806eb70 0x7fd06806ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:39.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.931+0000 7fd066ffd640 10 monclient: get_auth_request con 0x7fd06806eb70 auth_method 0 2024-04-26T00:33:39.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.931+0000 7fd066ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:39.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.931+0000 7fd066ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:39.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.931+0000 7fd066ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:39.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.931+0000 7fd066ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:39.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.931+0000 7fd066ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:39.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.931+0000 7fd066ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:39.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.931+0000 7fd066ffd640 10 monclient(hunting): handle_auth_done global_id 154211 payload 824 2024-04-26T00:33:39.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.931+0000 7fd066ffd640 1 -- 172.21.15.37:0/673684016 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd06809e760 con 0x7fd06806eb70 2024-04-26T00:33:39.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.935+0000 7fd066ffd640 1 -- 172.21.15.37:0/673684016 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd06806eb70 msgr2=0x7fd06806ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:39.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.935+0000 7fd066ffd640 1 --2- 172.21.15.37:0/673684016 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd06806eb70 0x7fd06806ef80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd05800f510 tx=0x7fd0580046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:39.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.935+0000 7fd04ffff640 1 -- 172.21.15.37:0/673684016 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05801fc20 con 0x7fd06806eb70 2024-04-26T00:33:39.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.935+0000 7fd04ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:39.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.935+0000 7fd04ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:33:39.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.935+0000 7fd04ffff640 10 monclient: dump: 2024-04-26T00:33:39.933 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:39.933 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:39.933 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:39.933 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:39.933 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:39.933 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:39.933 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:39.933 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:39.933 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:39.933 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:39.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.935+0000 7fd04ffff640 1 -- 172.21.15.37:0/673684016 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4717 (secure 0 0 0) 0x7fd05801fc20 con 0x7fd06806eb70 2024-04-26T00:33:39.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.935+0000 7fd04ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:33:39.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.935+0000 7fd04ffff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:33:39.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.935+0000 7fd04ffff640 1 -- 172.21.15.37:0/673684016 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd06806eb70 msgr2=0x7fd06806ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:39.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.935+0000 7fd04ffff640 1 --2- 172.21.15.37:0/673684016 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd06806eb70 0x7fd06806ef80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd05800f510 tx=0x7fd0580046d0 comp rx=0 tx=0).stop 2024-04-26T00:33:39.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.935+0000 7fd04ffff640 1 -- 172.21.15.37:0/673684016 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd06806eb70 msgr2=0x7fd06806ef80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:33:39.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.935+0000 7fd04ffff640 1 --2- 172.21.15.37:0/673684016 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd06806eb70 0x7fd06806ef80 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd05800f510 tx=0x7fd0580046d0 comp rx=0 tx=0).stop 2024-04-26T00:33:39.934 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:33:39.934 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:33:39.934 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:33:39.935 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:33:39.935 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 102, 2024-04-26T00:33:39.935 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 57455, 2024-04-26T00:33:39.935 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:33:39.935 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:33:39.935 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:33:39.935 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:33:39.935 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:39.935 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 3, 2024-04-26T00:33:39.935 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:33:39.935 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:33:39.935 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:33:39.935 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:33:39.935 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:33:39.935 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:33:39.936 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:33:39.936 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:33:39.936 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:33:39.936 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:33:39.936 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:33:39.936 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:33:39.936 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:33:39.936 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:33:39.936 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:39.936 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:33:39.936 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:33:39.936 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:33:39.936 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:33:39.936 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:33:39.936 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:33:39.937 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:33:39.937 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:33:39.937 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:33:39.937 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:33:39.937 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:33:39.937 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:33:39.937 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:33:39.937 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:39.937 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:39.937 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:33:39.937 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:33:39.937 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:33:39.937 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:33:39.937 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:33:39.937 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:33:39.938 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:33:39.938 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:33:39.938 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:33:39.938 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:33:39.938 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:33:39.938 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:33:39.938 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:33:39.938 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:33:39.938 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:33:39.938 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:33:39.938 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:33:39.938 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:33:39.938 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:33:39.938 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:33:39.938 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:33:39.939 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:33:39.939 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:33:39.939 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:33:39.939 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:33:39.939 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:33:39.939 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:33:39.939 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:33:39.939 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:39.939 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:33:39.939 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:39.939 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:33:39.939 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:39.939 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:33:39.939 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:33:39.939 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:33:39.940 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:33:39.940 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:39.940 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:33:39.940 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:33:39.940 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:39.940 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:39.940 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:39.940 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:33:39.940 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:33:39.940 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:39.940 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:39.940 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:39.940 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:39.940 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:33:39.940 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:33:39.941 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:33:39.941 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:33:39.941 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:33:39.941 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:39.941 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:39.941 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:33:39.941 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:33:39.941 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:33:39.941 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:33:39.941 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:39.941 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:33:39.941 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:33:39.941 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:39.941 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:39.941 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:39.942 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:33:39.942 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:33:39.942 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:39.942 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:39.942 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:39.942 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:39.942 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:33:39.942 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:33:39.942 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:33:39.942 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:33:39.942 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:33:39.942 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:39.942 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:39.942 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:33:39.942 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:33:39.943 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:33:39.943 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:33:39.943 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:39.943 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:33:39.943 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:33:39.943 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:39.943 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:39.943 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:39.943 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:33:39.943 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:33:39.943 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:39.943 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:39.943 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:39.943 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:39.943 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:33:39.944 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:33:39.944 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:33:39.944 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:33:39.944 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:33:39.944 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:39.944 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:39.944 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:39.944 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:33:39.944 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:33:39.944 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:39.944 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:33:39.944 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:33:39.944 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:33:39.944 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:39.944 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:39.944 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:33:39.945 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:39.945 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:33:39.945 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:33:39.945 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:33:39.945 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:39.945 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:39.945 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:33:39.945 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:39.945 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:33:39.945 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:33:39.945 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:33:39.945 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:39.948 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:39.948 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:39.948 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:33:39.948 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:33:39.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.935+0000 7fd06d87e640 1 -- 172.21.15.37:0/673684016 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd054035ed0 msgr2=0x7fd054038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:39.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.935+0000 7fd06d87e640 1 --2- 172.21.15.37:0/673684016 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd054035ed0 0x7fd054038390 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fd05c009580 tx=0x7fd05c012040 comp rx=0 tx=0).stop 2024-04-26T00:33:39.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.935+0000 7fd06d87e640 10 monclient: shutdown 2024-04-26T00:33:39.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.935+0000 7fd06d87e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:33:39.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.935+0000 7fd06d87e640 1 -- 172.21.15.37:0/673684016 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0681350e0 msgr2=0x7fd0681cce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:39.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.935+0000 7fd06d87e640 1 --2- 172.21.15.37:0/673684016 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0681350e0 0x7fd0681cce60 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd0600af040 tx=0x7fd060019650 comp rx=0 tx=0).stop 2024-04-26T00:33:39.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.935+0000 7fd0677fe640 1 -- 172.21.15.37:0/673684016 reap_dead start 2024-04-26T00:33:39.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.935+0000 7fd0677fe640 1 -- 172.21.15.37:0/673684016 reap_dead start 2024-04-26T00:33:39.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.935+0000 7fd06d87e640 1 -- 172.21.15.37:0/673684016 shutdown_connections 2024-04-26T00:33:39.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.935+0000 7fd06d87e640 1 -- 172.21.15.37:0/673684016 >> 172.21.15.37:0/673684016 conn(0x7fd0680a1d50 msgr2=0x7fd068131c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:33:39.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.939+0000 7fd06d87e640 1 -- 172.21.15.37:0/673684016 shutdown_connections 2024-04-26T00:33:39.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:39.939+0000 7fd06d87e640 1 -- 172.21.15.37:0/673684016 wait complete. 2024-04-26T00:33:39.950 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:33:39.950 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:33:39.951 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:33:40.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.075+0000 7fd254adc640 10 monclient: get_monmap_and_config 2024-04-26T00:33:40.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.075+0000 7fd254adc640 10 monclient: build_initial_monmap 2024-04-26T00:33:40.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.075+0000 7fd254adc640 10 monclient: monmap: 2024-04-26T00:33:40.072 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:33:40.072 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:40.072 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:33:40.076018+0000 2024-04-26T00:33:40.072 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:33:40.076018+0000 2024-04-26T00:33:40.072 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:33:40.072 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:40.072 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:33:40.072 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:33:40.072 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:33:40.072 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:33:40.072 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:33:40.073 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:40.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.075+0000 7fd254adc640 1 Processor -- start 2024-04-26T00:33:40.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.075+0000 7fd254adc640 1 -- start start 2024-04-26T00:33:40.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.075+0000 7fd254adc640 10 monclient: init 2024-04-26T00:33:40.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.075+0000 7fd254adc640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:40.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.075+0000 7fd254adc640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-04-26T00:33:40.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.075+0000 7fd254adc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd250136f20 addr v1:172.21.15.37:6789/0 2024-04-26T00:33:40.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.075+0000 7fd254adc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd250137a10 0x7fd25009bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:40.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.075+0000 7fd254adc640 10 monclient(hunting): picked mon.noname-b con 0x7fd250137a10 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:33:40.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.075+0000 7fd254adc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd25013c450 addr v1:172.21.15.45:6789/0 2024-04-26T00:33:40.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.075+0000 7fd254adc640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2500a50f0 con 0x7fd250136f20 2024-04-26T00:33:40.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.075+0000 7fd254adc640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd25009b330 con 0x7fd25013c450 2024-04-26T00:33:40.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.075+0000 7fd254adc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:40.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.075+0000 7fd254adc640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd25009b610 con 0x7fd250137a10 2024-04-26T00:33:40.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.075+0000 7fd254adc640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:40.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.075+0000 7fd254adc640 10 monclient(hunting): authenticate will time out at 1801.381921s 2024-04-26T00:33:40.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.075+0000 7fd24e59c640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fd250136f20 0x7fd250137300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:45002/0 (socket says 172.21.15.37:45002) 2024-04-26T00:33:40.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.075+0000 7fd24e59c640 1 -- 172.21.15.37:0/2484398361 learned_addr learned my addr 172.21.15.37:0/2484398361 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:33:40.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.075+0000 7fd24dd9b640 1 --2- 172.21.15.37:0/2484398361 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd250137a10 0x7fd25009bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:40.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.075+0000 7fd24dd9b640 10 monclient(hunting): get_auth_request con 0x7fd250137a10 auth_method 0 2024-04-26T00:33:40.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.075+0000 7fd24dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:40.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.075+0000 7fd24dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:40.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.075+0000 7fd24dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:40.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.075+0000 7fd24dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:40.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.075+0000 7fd24dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:40.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.075+0000 7fd24dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:40.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.075+0000 7fd24dd9b640 10 monclient(hunting): handle_auth_done global_id 154214 payload 274 2024-04-26T00:33:40.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.075+0000 7fd24dd9b640 1 -- 172.21.15.37:0/2484398361 >> v1:172.21.15.45:6789/0 conn(0x7fd25013c450 legacy=0x7fd25013e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:33:40.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.075+0000 7fd24dd9b640 1 -- 172.21.15.37:0/2484398361 >> v1:172.21.15.37:6789/0 conn(0x7fd250136f20 legacy=0x7fd250137300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:33:40.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.075+0000 7fd24dd9b640 10 monclient: _finish_hunting 0 2024-04-26T00:33:40.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.075+0000 7fd24dd9b640 1 monclient: found mon.noname-b 2024-04-26T00:33:40.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.075+0000 7fd24dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:40.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.075+0000 7fd24dd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:33:40.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.075+0000 7fd24dd9b640 1 -- 172.21.15.37:0/2484398361 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd25009b8f0 con 0x7fd250137a10 2024-04-26T00:33:40.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.079+0000 7fd24dd9b640 1 -- 172.21.15.37:0/2484398361 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd250137a10 msgr2=0x7fd25009bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:40.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.079+0000 7fd24dd9b640 1 --2- 172.21.15.37:0/2484398361 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd250137a10 0x7fd25009bff0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd250137e90 tx=0x7fd24401c370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d995fb18a4f2610 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:40.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.079+0000 7fd24d59a640 1 -- 172.21.15.37:0/2484398361 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2440b0070 con 0x7fd250137a10 2024-04-26T00:33:40.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.079+0000 7fd24d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:40.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.079+0000 7fd24d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:33:40.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.079+0000 7fd24d59a640 10 monclient: dump: 2024-04-26T00:33:40.082 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:40.082 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:40.082 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:40.082 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:40.082 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:40.082 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:40.082 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:40.082 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:40.082 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:40.082 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:40.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.079+0000 7fd24d59a640 10 monclient: _finish_auth 0 2024-04-26T00:33:40.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.079+0000 7fd24d59a640 10 monclient: _check_auth_tickets 2024-04-26T00:33:40.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.079+0000 7fd24d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:40.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.079+0000 7fd24d59a640 1 -- 172.21.15.37:0/2484398361 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2440ab600 con 0x7fd250137a10 2024-04-26T00:33:40.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.079+0000 7fd254adc640 5 monclient: authenticate success, global_id 154214 2024-04-26T00:33:40.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.079+0000 7fd254adc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:33:40.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.079+0000 7fd24d59a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:33:40.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.079+0000 7fd24d59a640 1 -- 172.21.15.37:0/2484398361 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2440ab9d0 con 0x7fd250137a10 2024-04-26T00:33:40.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.079+0000 7fd254adc640 10 monclient: get_monmap_and_config success 2024-04-26T00:33:40.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.079+0000 7fd254adc640 4 set_mon_vals no callback set 2024-04-26T00:33:40.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.079+0000 7fd24d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:40.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.079+0000 7fd24d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:33:40.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.079+0000 7fd24d59a640 10 monclient: dump: 2024-04-26T00:33:40.083 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:40.083 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:40.083 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:40.083 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:40.083 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:40.084 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:40.084 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:40.084 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:40.084 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:40.084 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:40.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.079+0000 7fd254adc640 10 monclient: shutdown 2024-04-26T00:33:40.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.079+0000 7fd254adc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:33:40.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.079+0000 7fd254adc640 1 -- 172.21.15.37:0/2484398361 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd250137a10 msgr2=0x7fd25009bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:40.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.079+0000 7fd254adc640 1 --2- 172.21.15.37:0/2484398361 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd250137a10 0x7fd25009bff0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd250137e90 tx=0x7fd24401c370 comp rx=0 tx=0).stop 2024-04-26T00:33:40.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.079+0000 7fd254adc640 1 -- 172.21.15.37:0/2484398361 shutdown_connections 2024-04-26T00:33:40.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.079+0000 7fd254adc640 1 --2- 172.21.15.37:0/2484398361 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd250137a10 0x7fd25009bff0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:40.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.079+0000 7fd254adc640 1 -- 172.21.15.37:0/2484398361 >> 172.21.15.37:0/2484398361 conn(0x7fd2500a61b0 msgr2=0x7fd2500a65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:33:40.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.079+0000 7fd254adc640 1 -- 172.21.15.37:0/2484398361 shutdown_connections 2024-04-26T00:33:40.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.079+0000 7fd254adc640 1 -- 172.21.15.37:0/2484398361 wait complete. 2024-04-26T00:33:40.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd254adc640 10 monclient: build_initial_monmap 2024-04-26T00:33:40.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd254adc640 10 monclient: monmap: 2024-04-26T00:33:40.085 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:33:40.085 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:33:40.085 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:33:40.085 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:33:40.085 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:33:40.085 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:33:40.085 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:33:40.085 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:33:40.085 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:33:40.085 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:40.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd254adc640 1 Processor -- start 2024-04-26T00:33:40.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd254adc640 1 -- start start 2024-04-26T00:33:40.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd254adc640 10 monclient: init 2024-04-26T00:33:40.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd254adc640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:40.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd254adc640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:33:40.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd254adc640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd250136f20 0x7fd25013b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:40.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd254adc640 10 monclient(hunting): picked mon.noname-a con 0x7fd250136f20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:33:40.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd254adc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd250137a10 0x7fd2501351d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:40.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd254adc640 10 monclient(hunting): picked mon.noname-b con 0x7fd250137a10 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:33:40.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd254adc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd25013c450 0x7fd250135710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:40.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd254adc640 10 monclient(hunting): picked mon.noname-c con 0x7fd25013c450 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:33:40.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd254adc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:40.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd254adc640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd250142b50 con 0x7fd250136f20 2024-04-26T00:33:40.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd254adc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:40.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd254adc640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd2501429d0 con 0x7fd250137a10 2024-04-26T00:33:40.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd254adc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:40.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd254adc640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd250142cd0 con 0x7fd25013c450 2024-04-26T00:33:40.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd254adc640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:40.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd254adc640 10 monclient(hunting): authenticate will time out at 1801.389426s 2024-04-26T00:33:40.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd24e59c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd250136f20 0x7fd25013b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:40.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd24ed9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd25013c450 0x7fd250135710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:40.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd24e59c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd250136f20 0x7fd25013b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40140/0 (socket says 172.21.15.37:40140) 2024-04-26T00:33:40.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd24ed9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd25013c450 0x7fd250135710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:37328/0 (socket says 172.21.15.37:37328) 2024-04-26T00:33:40.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd24e59c640 1 -- 172.21.15.37:0/2151340924 learned_addr learned my addr 172.21.15.37:0/2151340924 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:33:40.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd24ed9d640 10 monclient(hunting): get_auth_request con 0x7fd25013c450 auth_method 0 2024-04-26T00:33:40.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd24ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:40.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd24ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:40.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd24ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:40.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd24e59c640 10 monclient(hunting): get_auth_request con 0x7fd250136f20 auth_method 0 2024-04-26T00:33:40.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd24e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:40.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd24e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:40.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd24e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:40.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd24ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:40.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd24ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:40.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd24ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:40.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd24e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:40.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd24e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:40.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd24e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:40.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd24dd9b640 1 --2- 172.21.15.37:0/2151340924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd250137a10 0x7fd2501351d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:40.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd24dd9b640 10 monclient(hunting): get_auth_request con 0x7fd250137a10 auth_method 0 2024-04-26T00:33:40.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd24dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:40.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd24dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:40.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd24dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:40.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd24ed9d640 10 monclient(hunting): handle_auth_done global_id 154217 payload 824 2024-04-26T00:33:40.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd24ed9d640 1 -- 172.21.15.37:0/2151340924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd250137a10 msgr2=0x7fd2501351d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:40.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd24ed9d640 1 --2- 172.21.15.37:0/2151340924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd250137a10 0x7fd2501351d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:40.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd24ed9d640 1 -- 172.21.15.37:0/2151340924 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd250136f20 msgr2=0x7fd25013b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:40.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd24ed9d640 1 --2- 172.21.15.37:0/2151340924 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd250136f20 0x7fd25013b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:40.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd24ed9d640 10 monclient: _finish_hunting 0 2024-04-26T00:33:40.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd24ed9d640 1 monclient: found mon.noname-c 2024-04-26T00:33:40.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd24ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:40.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd24ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:33:40.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd24ed9d640 1 -- 172.21.15.37:0/2151340924 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2500a0040 con 0x7fd25013c450 2024-04-26T00:33:40.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.083+0000 7fd24e59c640 1 --2- 172.21.15.37:0/2151340924 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd250136f20 0x7fd25013b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:33:40.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd24ed9d640 1 -- 172.21.15.37:0/2151340924 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd25013c450 msgr2=0x7fd250135710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:40.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd24ed9d640 1 --2- 172.21.15.37:0/2151340924 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd25013c450 0x7fd250135710 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd24800c780 tx=0x7fd24800cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:40.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd23effd640 1 -- 172.21.15.37:0/2151340924 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd248016020 con 0x7fd25013c450 2024-04-26T00:33:40.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd23effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:40.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd23effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:33:40.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd23effd640 10 monclient: dump: 2024-04-26T00:33:40.090 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:40.090 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:40.090 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:40.090 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:40.090 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:40.090 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:40.090 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:40.090 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:40.090 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:40.090 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:40.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd23effd640 10 monclient: _finish_auth 0 2024-04-26T00:33:40.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd23effd640 10 monclient: _check_auth_tickets 2024-04-26T00:33:40.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd23effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:40.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd23effd640 1 -- 172.21.15.37:0/2151340924 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd24800ac60 con 0x7fd25013c450 2024-04-26T00:33:40.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd23effd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:33:40.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd23effd640 1 -- 172.21.15.37:0/2151340924 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd248012660 con 0x7fd25013c450 2024-04-26T00:33:40.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd23effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:40.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd23effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:33:40.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd23effd640 10 monclient: dump: 2024-04-26T00:33:40.091 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:40.091 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:40.091 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:40.091 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:40.091 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:40.091 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:40.091 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:40.091 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:40.091 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:40.091 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:40.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd254adc640 5 monclient: authenticate success, global_id 154217 2024-04-26T00:33:40.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd24cd99640 4 set_mon_vals no callback set 2024-04-26T00:33:40.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd254adc640 10 monclient: _renew_subs 2024-04-26T00:33:40.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd254adc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:33:40.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd254adc640 1 -- 172.21.15.37:0/2151340924 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd25009ae30 con 0x7fd25013c450 2024-04-26T00:33:40.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd254adc640 10 monclient: _renew_subs 2024-04-26T00:33:40.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd254adc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:33:40.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd254adc640 1 -- 172.21.15.37:0/2151340924 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd25013a5f0 con 0x7fd25013c450 2024-04-26T00:33:40.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd254adc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:40.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd254adc640 1 --2- 172.21.15.37:0/2151340924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2501386b0 0x7fd25002c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:40.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd254adc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:40.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd254adc640 1 -- 172.21.15.37:0/2151340924 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd250136010 con 0x7fd2501386b0 2024-04-26T00:33:40.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd23effd640 1 -- 172.21.15.37:0/2151340924 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd248019020 con 0x7fd25013c450 2024-04-26T00:33:40.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd23effd640 1 --2- 172.21.15.37:0/2151340924 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd230035ed0 0x7fd230038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:40.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd24e59c640 1 --2- 172.21.15.37:0/2151340924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2501386b0 0x7fd25002c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:40.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd23effd640 1 -- 172.21.15.37:0/2151340924 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(534..534 src has 370..534) ==== 3959+0+0 (secure 0 0 0) 0x7fd248048960 con 0x7fd25013c450 2024-04-26T00:33:40.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd24e59c640 10 monclient: get_auth_request con 0x7fd2501386b0 auth_method 0 2024-04-26T00:33:40.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd24e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:40.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd24e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:40.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd24e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:40.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd24e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:40.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd24e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:40.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd24e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:40.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd24dd9b640 1 --2- 172.21.15.37:0/2151340924 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd230035ed0 0x7fd230038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:40.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd24dd9b640 10 monclient: get_auth_request con 0x7fd230035ed0 auth_method 0 2024-04-26T00:33:40.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd24e59c640 10 monclient(hunting): handle_auth_done global_id 164134 payload 824 2024-04-26T00:33:40.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.087+0000 7fd24e59c640 1 -- 172.21.15.37:0/2151340924 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd25004f510 con 0x7fd2501386b0 2024-04-26T00:33:40.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.091+0000 7fd24e59c640 1 -- 172.21.15.37:0/2151340924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2501386b0 msgr2=0x7fd25002c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:40.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.091+0000 7fd24e59c640 1 --2- 172.21.15.37:0/2151340924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2501386b0 0x7fd25002c000 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd2400072a0 tx=0x7fd240007760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:40.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.091+0000 7fd24dd9b640 1 --2- 172.21.15.37:0/2151340924 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd230035ed0 0x7fd230038390 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd250068940 tx=0x7fd2440a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:40.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.091+0000 7fd23effd640 1 -- 172.21.15.37:0/2151340924 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd24000fde0 con 0x7fd2501386b0 2024-04-26T00:33:40.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.091+0000 7fd23effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:40.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.091+0000 7fd23effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:33:40.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.091+0000 7fd23effd640 10 monclient: dump: 2024-04-26T00:33:40.096 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:40.096 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:40.096 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:40.096 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:40.096 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:40.096 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:40.096 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:40.096 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:40.096 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:40.096 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:40.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.091+0000 7fd23effd640 1 -- 172.21.15.37:0/2151340924 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd24000fde0 con 0x7fd2501386b0 2024-04-26T00:33:40.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.091+0000 7fd23effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:40.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.091+0000 7fd23effd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:33:40.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.091+0000 7fd23effd640 1 -- 172.21.15.37:0/2151340924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2501386b0 msgr2=0x7fd25002c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:40.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.091+0000 7fd23effd640 1 --2- 172.21.15.37:0/2151340924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2501386b0 0x7fd25002c000 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd2400072a0 tx=0x7fd240007760 comp rx=0 tx=0).stop 2024-04-26T00:33:40.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.091+0000 7fd23effd640 1 -- 172.21.15.37:0/2151340924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2501386b0 msgr2=0x7fd25002c000 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:33:40.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.091+0000 7fd23effd640 1 --2- 172.21.15.37:0/2151340924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2501386b0 0x7fd25002c000 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd2400072a0 tx=0x7fd240007760 comp rx=0 tx=0).stop 2024-04-26T00:33:40.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.099+0000 7fd254adc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:33:40.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.099+0000 7fd254adc640 1 --2- 172.21.15.37:0/2151340924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd250138a90 0x7fd25002c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:40.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.099+0000 7fd254adc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:40.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.099+0000 7fd254adc640 1 -- 172.21.15.37:0/2151340924 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd2500a1ff0 con 0x7fd250138a90 2024-04-26T00:33:40.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.099+0000 7fd24e59c640 1 --2- 172.21.15.37:0/2151340924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd250138a90 0x7fd25002c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:40.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.099+0000 7fd24e59c640 10 monclient: get_auth_request con 0x7fd250138a90 auth_method 0 2024-04-26T00:33:40.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.099+0000 7fd24e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:40.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.099+0000 7fd24e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:40.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.099+0000 7fd24e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:40.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.099+0000 7fd24e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:40.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.099+0000 7fd24e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:40.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.099+0000 7fd24e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:40.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.099+0000 7fd24e59c640 10 monclient(hunting): handle_auth_done global_id 164137 payload 824 2024-04-26T00:33:40.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.099+0000 7fd24e59c640 1 -- 172.21.15.37:0/2151340924 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd250135c50 con 0x7fd250138a90 2024-04-26T00:33:40.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.103+0000 7fd24e59c640 1 -- 172.21.15.37:0/2151340924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd250138a90 msgr2=0x7fd25002c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:40.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.103+0000 7fd24e59c640 1 --2- 172.21.15.37:0/2151340924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd250138a90 0x7fd25002c540 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd240017000 tx=0x7fd240004950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:40.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.103+0000 7fd23effd640 1 -- 172.21.15.37:0/2151340924 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd24001ee40 con 0x7fd250138a90 2024-04-26T00:33:40.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.103+0000 7fd23effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:40.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.103+0000 7fd23effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:33:40.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.103+0000 7fd23effd640 10 monclient: dump: 2024-04-26T00:33:40.098 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:40.099 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:40.099 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:40.099 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:40.099 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:40.099 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:40.099 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:40.099 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:40.099 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:40.099 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:40.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.103+0000 7fd23effd640 1 -- 172.21.15.37:0/2151340924 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7fd240006240 con 0x7fd250138a90 2024-04-26T00:33:40.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.103+0000 7fd23effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:33:40.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.103+0000 7fd23effd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:33:40.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.103+0000 7fd23effd640 1 -- 172.21.15.37:0/2151340924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd250138a90 msgr2=0x7fd25002c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:40.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.103+0000 7fd23effd640 1 --2- 172.21.15.37:0/2151340924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd250138a90 0x7fd25002c540 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd240017000 tx=0x7fd240004950 comp rx=0 tx=0).stop 2024-04-26T00:33:40.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.103+0000 7fd23effd640 1 -- 172.21.15.37:0/2151340924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd250138a90 msgr2=0x7fd25002c540 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:33:40.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.103+0000 7fd23effd640 1 --2- 172.21.15.37:0/2151340924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd250138a90 0x7fd25002c540 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd240017000 tx=0x7fd240004950 comp rx=0 tx=0).stop 2024-04-26T00:33:40.100 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:33:40.100 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:33:40.100 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:33:40.101 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:33:40.101 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 102, 2024-04-26T00:33:40.101 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 7599, 2024-04-26T00:33:40.101 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:33:40.101 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:33:40.101 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:33:40.101 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:33:40.101 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:40.101 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 4, 2024-04-26T00:33:40.101 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:33:40.101 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:33:40.101 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:33:40.101 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:33:40.101 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:33:40.101 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:33:40.102 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:33:40.102 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:33:40.102 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:33:40.102 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:33:40.102 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:33:40.102 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:33:40.102 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:33:40.102 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:33:40.102 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:40.102 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:33:40.102 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:33:40.102 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:33:40.102 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:33:40.102 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:33:40.103 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:33:40.103 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:33:40.103 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:33:40.103 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:33:40.103 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:33:40.103 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:33:40.103 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:33:40.103 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:33:40.103 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:40.103 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:40.103 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:33:40.103 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:33:40.103 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:33:40.103 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:33:40.103 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:33:40.104 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:33:40.104 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:33:40.104 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:33:40.104 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:33:40.104 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:33:40.104 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:33:40.104 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:33:40.104 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:33:40.104 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:33:40.104 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:33:40.104 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:33:40.104 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:33:40.104 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:33:40.104 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:33:40.105 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:33:40.105 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:33:40.105 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:33:40.105 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:33:40.105 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:33:40.105 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:33:40.105 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:33:40.105 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:33:40.105 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:33:40.105 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:40.105 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:33:40.105 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:40.105 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:33:40.105 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:40.105 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:33:40.106 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:33:40.106 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:33:40.106 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:33:40.106 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:40.106 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:33:40.106 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:33:40.106 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:40.106 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:40.106 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:40.106 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:33:40.106 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:33:40.106 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:40.106 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:40.106 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:40.107 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:40.107 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:33:40.107 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:33:40.107 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:33:40.107 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:33:40.107 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:33:40.107 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:40.107 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:40.107 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:33:40.107 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:33:40.107 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:33:40.107 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:33:40.107 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:40.107 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:33:40.108 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:33:40.108 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:40.108 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:40.108 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:40.108 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:33:40.108 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:33:40.108 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:40.108 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:40.108 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:40.108 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:40.108 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:33:40.108 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:33:40.108 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:33:40.108 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:33:40.108 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:33:40.109 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:40.109 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:40.109 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:33:40.109 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:33:40.109 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:33:40.109 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:33:40.109 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:40.109 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:33:40.109 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:33:40.109 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:40.109 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:40.109 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:40.109 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:33:40.109 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:33:40.109 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:40.110 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:40.110 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:40.110 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:40.110 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:33:40.110 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:33:40.110 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:33:40.110 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:33:40.110 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:33:40.110 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:40.110 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:40.110 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:40.110 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:33:40.110 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:33:40.111 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:40.111 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:33:40.111 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:33:40.111 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:33:40.111 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:40.111 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:40.111 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:33:40.111 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:40.111 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:33:40.111 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:33:40.111 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:33:40.111 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:40.111 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:40.111 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:40.111 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:33:40.111 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:33:40.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.103+0000 7fd254adc640 1 -- 172.21.15.37:0/2151340924 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd230035ed0 msgr2=0x7fd230038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:40.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.103+0000 7fd254adc640 1 --2- 172.21.15.37:0/2151340924 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd230035ed0 0x7fd230038390 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd250068940 tx=0x7fd2440a6000 comp rx=0 tx=0).stop 2024-04-26T00:33:40.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.103+0000 7fd254adc640 10 monclient: shutdown 2024-04-26T00:33:40.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.103+0000 7fd254adc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:33:40.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.103+0000 7fd254adc640 1 -- 172.21.15.37:0/2151340924 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd25013c450 msgr2=0x7fd250135710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:40.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.103+0000 7fd254adc640 1 --2- 172.21.15.37:0/2151340924 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd25013c450 0x7fd250135710 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd24800c780 tx=0x7fd24800cc40 comp rx=0 tx=0).stop 2024-04-26T00:33:40.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.103+0000 7fd24ed9d640 1 -- 172.21.15.37:0/2151340924 reap_dead start 2024-04-26T00:33:40.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.103+0000 7fd24ed9d640 1 -- 172.21.15.37:0/2151340924 reap_dead start 2024-04-26T00:33:40.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.103+0000 7fd254adc640 1 -- 172.21.15.37:0/2151340924 shutdown_connections 2024-04-26T00:33:40.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.103+0000 7fd254adc640 1 -- 172.21.15.37:0/2151340924 >> 172.21.15.37:0/2151340924 conn(0x7fd2500a61b0 msgr2=0x7fd2500a8110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:33:40.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.103+0000 7fd254adc640 1 -- 172.21.15.37:0/2151340924 shutdown_connections 2024-04-26T00:33:40.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.103+0000 7fd254adc640 1 -- 172.21.15.37:0/2151340924 wait complete. 2024-04-26T00:33:40.114 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:33:40.114 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-04-26T00:33:40.114 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-26T00:33:40.114 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-04-26T00:33:40.114 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-04-26T00:33:40.115 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-26T00:33:40.115 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T00:33:40.133 INFO:tasks.ceph.mon.b:Stopped 2024-04-26T00:33:40.133 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-26T00:33:40.133 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:33:40.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.251+0000 7f97a0bd4640 10 monclient: get_monmap_and_config 2024-04-26T00:33:40.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.251+0000 7f97a0bd4640 10 monclient: build_initial_monmap 2024-04-26T00:33:40.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.251+0000 7f97a0bd4640 10 monclient: monmap: 2024-04-26T00:33:40.245 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:33:40.245 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:40.246 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:33:40.253840+0000 2024-04-26T00:33:40.246 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:33:40.253840+0000 2024-04-26T00:33:40.246 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:33:40.246 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:40.246 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:33:40.246 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:33:40.246 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:33:40.246 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:33:40.246 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:33:40.246 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:40.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.251+0000 7f97a0bd4640 1 Processor -- start 2024-04-26T00:33:40.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.251+0000 7f97a0bd4640 1 -- start start 2024-04-26T00:33:40.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.251+0000 7f97a0bd4640 10 monclient: init 2024-04-26T00:33:40.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.251+0000 7f97a0bd4640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:40.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.251+0000 7f97a0bd4640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-04-26T00:33:40.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.251+0000 7f97a0bd4640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f979c136e10 0x7f979c09c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:40.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.251+0000 7f97a0bd4640 10 monclient(hunting): picked mon.noname-a con 0x7f979c136e10 addr v2:172.21.15.37:3300/0 2024-04-26T00:33:40.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.251+0000 7f97a0bd4640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f979c1371f0 0x7f979c137600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:40.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.251+0000 7f97a0bd4640 10 monclient(hunting): picked mon.noname-c con 0x7f979c1371f0 addr v2:172.21.15.45:3300/0 2024-04-26T00:33:40.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.251+0000 7f97a0bd4640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f979c135d90 0x7f979c136240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:40.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.251+0000 7f97a0bd4640 10 monclient(hunting): picked mon.noname-b con 0x7f979c135d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:33:40.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.251+0000 7f97a0bd4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:40.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.251+0000 7f97a0bd4640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f979c09b8f0 con 0x7f979c136e10 2024-04-26T00:33:40.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.251+0000 7f97a0bd4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:40.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.251+0000 7f97a0bd4640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f979c09b680 con 0x7f979c1371f0 2024-04-26T00:33:40.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.251+0000 7f97a0bd4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:40.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.251+0000 7f97a0bd4640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f979c09bbd0 con 0x7f979c135d90 2024-04-26T00:33:40.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.251+0000 7f97a0bd4640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:40.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.251+0000 7f97a0bd4640 10 monclient(hunting): authenticate will time out at 1801.559541s 2024-04-26T00:33:40.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.251+0000 7f979ad9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f979c135d90 0x7f979c136240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:40.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.251+0000 7f979ad9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f979c135d90 0x7f979c136240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:37336/0 (socket says 172.21.15.37:37336) 2024-04-26T00:33:40.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.251+0000 7f979ad9d640 1 -- 172.21.15.37:0/2736173442 learned_addr learned my addr 172.21.15.37:0/2736173442 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:33:40.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.251+0000 7f979ad9d640 10 monclient(hunting): get_auth_request con 0x7f979c135d90 auth_method 0 2024-04-26T00:33:40.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.251+0000 7f979ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:40.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.251+0000 7f979ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:40.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.251+0000 7f979ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:40.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.251+0000 7f9799d9b640 1 -- 172.21.15.37:0/2736173442 >> v2:172.21.15.45:3300/0 conn(0x7f979c1371f0 msgr2=0x7f979c137600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:33:40.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.251+0000 7f979a59c640 1 --2- 172.21.15.37:0/2736173442 >> v2:172.21.15.37:3300/0 conn(0x7f979c136e10 0x7f979c09c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:40.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.251+0000 7f9799d9b640 1 --2- 172.21.15.37:0/2736173442 >> v2:172.21.15.45:3300/0 conn(0x7f979c1371f0 0x7f979c137600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:33:40.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.251+0000 7f979a59c640 10 monclient(hunting): get_auth_request con 0x7f979c136e10 auth_method 0 2024-04-26T00:33:40.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.251+0000 7f979a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:40.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.251+0000 7f979a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:40.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.251+0000 7f979a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:40.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.251+0000 7f979ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:40.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.251+0000 7f979ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:40.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.251+0000 7f979ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:40.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.255+0000 7f979a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:40.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.255+0000 7f979a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:40.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.255+0000 7f979a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:40.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.255+0000 7f979ad9d640 10 monclient(hunting): handle_auth_done global_id 154220 payload 274 2024-04-26T00:33:40.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.255+0000 7f979ad9d640 1 -- 172.21.15.37:0/2736173442 >> v2:172.21.15.45:3300/0 conn(0x7f979c1371f0 msgr2=0x7f979c137600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:33:40.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.255+0000 7f979ad9d640 1 --2- 172.21.15.37:0/2736173442 >> v2:172.21.15.45:3300/0 conn(0x7f979c1371f0 0x7f979c137600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:40.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.255+0000 7f979ad9d640 1 -- 172.21.15.37:0/2736173442 >> v2:172.21.15.37:3300/0 conn(0x7f979c136e10 msgr2=0x7f979c09c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:33:40.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.255+0000 7f979ad9d640 1 --2- 172.21.15.37:0/2736173442 >> v2:172.21.15.37:3300/0 conn(0x7f979c136e10 0x7f979c09c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:40.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.255+0000 7f979ad9d640 10 monclient: _finish_hunting 0 2024-04-26T00:33:40.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.255+0000 7f979ad9d640 1 monclient: found mon.noname-b 2024-04-26T00:33:40.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.255+0000 7f979ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:40.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.255+0000 7f979ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:33:40.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.255+0000 7f979ad9d640 1 -- 172.21.15.37:0/2736173442 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f979c137b40 con 0x7f979c135d90 2024-04-26T00:33:40.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979ad9d640 1 -- 172.21.15.37:0/2736173442 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f979c135d90 msgr2=0x7f979c136240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:40.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979ad9d640 1 --2- 172.21.15.37:0/2736173442 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f979c135d90 0x7f979c136240 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f979c068e60 tx=0x7f979401c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93d6e15db172a25d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:40.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979959a640 1 -- 172.21.15.37:0/2736173442 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97940b1070 con 0x7f979c135d90 2024-04-26T00:33:40.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:40.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:33:40.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979959a640 10 monclient: dump: 2024-04-26T00:33:40.259 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:40.259 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:40.259 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:40.260 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:40.260 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:40.260 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:40.260 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:40.260 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:40.260 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:40.260 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:40.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979959a640 10 monclient: _finish_auth 0 2024-04-26T00:33:40.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979959a640 10 monclient: _check_auth_tickets 2024-04-26T00:33:40.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:40.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979959a640 1 -- 172.21.15.37:0/2736173442 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97940a8050 con 0x7f979c135d90 2024-04-26T00:33:40.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979959a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:33:40.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979959a640 1 -- 172.21.15.37:0/2736173442 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97940ac860 con 0x7f979c135d90 2024-04-26T00:33:40.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:40.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:33:40.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979959a640 10 monclient: dump: 2024-04-26T00:33:40.261 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:40.261 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:40.261 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:40.261 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:40.261 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:40.261 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:40.261 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:40.261 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:40.261 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:40.261 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:40.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f97a0bd4640 5 monclient: authenticate success, global_id 154220 2024-04-26T00:33:40.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f97a0bd4640 10 monclient: get_monmap_and_config success 2024-04-26T00:33:40.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f97a0bd4640 4 set_mon_vals no callback set 2024-04-26T00:33:40.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f97a0bd4640 10 monclient: shutdown 2024-04-26T00:33:40.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f97a0bd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:33:40.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f97a0bd4640 1 -- 172.21.15.37:0/2736173442 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f979c135d90 msgr2=0x7f979c136240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:40.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f97a0bd4640 1 --2- 172.21.15.37:0/2736173442 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f979c135d90 0x7f979c136240 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f979c068e60 tx=0x7f979401c400 comp rx=0 tx=0).stop 2024-04-26T00:33:40.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f97a0bd4640 1 -- 172.21.15.37:0/2736173442 shutdown_connections 2024-04-26T00:33:40.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f97a0bd4640 1 --2- 172.21.15.37:0/2736173442 >> v2:172.21.15.45:3300/0 conn(0x7f979c1371f0 0x7f979c137600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:40.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f97a0bd4640 1 --2- 172.21.15.37:0/2736173442 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f979c135d90 0x7f979c136240 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:40.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f97a0bd4640 1 --2- 172.21.15.37:0/2736173442 >> v2:172.21.15.37:3300/0 conn(0x7f979c136e10 0x7f979c09c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:40.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f97a0bd4640 1 -- 172.21.15.37:0/2736173442 >> 172.21.15.37:0/2736173442 conn(0x7f979c0a64e0 msgr2=0x7f979c0a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:33:40.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f97a0bd4640 1 -- 172.21.15.37:0/2736173442 shutdown_connections 2024-04-26T00:33:40.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f97a0bd4640 1 -- 172.21.15.37:0/2736173442 wait complete. 2024-04-26T00:33:40.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f97a0bd4640 10 monclient: build_initial_monmap 2024-04-26T00:33:40.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f97a0bd4640 10 monclient: monmap: 2024-04-26T00:33:40.263 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:33:40.263 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:33:40.263 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:33:40.263 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:33:40.263 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:33:40.263 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:33:40.263 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:33:40.263 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:33:40.263 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:33:40.263 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:40.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f97a0bd4640 1 Processor -- start 2024-04-26T00:33:40.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f97a0bd4640 1 -- start start 2024-04-26T00:33:40.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f97a0bd4640 10 monclient: init 2024-04-26T00:33:40.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f97a0bd4640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:40.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f97a0bd4640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:33:40.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f97a0bd4640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f979c135d90 0x7f979c134eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:40.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f97a0bd4640 10 monclient(hunting): picked mon.noname-a con 0x7f979c135d90 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:33:40.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f97a0bd4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f979c136e10 0x7f979c1353f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:40.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f97a0bd4640 10 monclient(hunting): picked mon.noname-b con 0x7f979c136e10 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:33:40.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f97a0bd4640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f979c1371f0 0x7f979c13ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:40.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f97a0bd4640 10 monclient(hunting): picked mon.noname-c con 0x7f979c1371f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:33:40.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f97a0bd4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:40.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f97a0bd4640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f979c140cf0 con 0x7f979c135d90 2024-04-26T00:33:40.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f97a0bd4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:40.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f97a0bd4640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f979c140b70 con 0x7f979c136e10 2024-04-26T00:33:40.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f97a0bd4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:40.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f97a0bd4640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f979c140e70 con 0x7f979c1371f0 2024-04-26T00:33:40.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f97a0bd4640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:40.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f97a0bd4640 10 monclient(hunting): authenticate will time out at 1801.566938s 2024-04-26T00:33:40.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979a59c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f979c135d90 0x7f979c134eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:40.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979ad9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f979c1371f0 0x7f979c13ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:40.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979a59c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f979c135d90 0x7f979c134eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40154/0 (socket says 172.21.15.37:40154) 2024-04-26T00:33:40.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f9799d9b640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f979c136e10 msgr2=0x7f979c1353f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:33:40.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979a59c640 1 -- 172.21.15.37:0/2526696013 learned_addr learned my addr 172.21.15.37:0/2526696013 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:33:40.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979ad9d640 10 monclient(hunting): get_auth_request con 0x7f979c1371f0 auth_method 0 2024-04-26T00:33:40.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:40.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:40.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:40.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979a59c640 10 monclient(hunting): get_auth_request con 0x7f979c135d90 auth_method 0 2024-04-26T00:33:40.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:40.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:40.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:40.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f9799d9b640 1 --2- 172.21.15.37:0/2526696013 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f979c136e10 0x7f979c1353f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:33:40.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:40.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:40.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:40.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:40.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:40.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:40.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979ad9d640 10 monclient(hunting): handle_auth_done global_id 154223 payload 824 2024-04-26T00:33:40.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979ad9d640 1 -- 172.21.15.37:0/2526696013 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f979c136e10 msgr2=0x7f979c1353f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:33:40.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979ad9d640 1 --2- 172.21.15.37:0/2526696013 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f979c136e10 0x7f979c1353f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:40.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979ad9d640 1 -- 172.21.15.37:0/2526696013 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f979c135d90 msgr2=0x7f979c134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:40.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979ad9d640 1 --2- 172.21.15.37:0/2526696013 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f979c135d90 0x7f979c134eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:40.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979ad9d640 10 monclient: _finish_hunting 0 2024-04-26T00:33:40.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979ad9d640 1 monclient: found mon.noname-c 2024-04-26T00:33:40.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:40.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979ad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:33:40.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979ad9d640 1 -- 172.21.15.37:0/2526696013 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f979c06d380 con 0x7f979c1371f0 2024-04-26T00:33:40.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.259+0000 7f979a59c640 1 --2- 172.21.15.37:0/2526696013 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f979c135d90 0x7f979c134eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:33:40.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.263+0000 7f979ad9d640 1 -- 172.21.15.37:0/2526696013 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f979c1371f0 msgr2=0x7f979c13ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:40.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.263+0000 7f979ad9d640 1 --2- 172.21.15.37:0/2526696013 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f979c1371f0 0x7f979c13ac10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f97940ae000 tx=0x7f97940191a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:40.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.263+0000 7f9792ffd640 1 -- 172.21.15.37:0/2526696013 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97940bb070 con 0x7f979c1371f0 2024-04-26T00:33:40.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.263+0000 7f9792ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:40.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.263+0000 7f9792ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:33:40.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.263+0000 7f9792ffd640 10 monclient: dump: 2024-04-26T00:33:40.268 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:40.268 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:40.268 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:40.268 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:40.268 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:40.268 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:40.268 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:40.268 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:40.268 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:40.268 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:40.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.263+0000 7f9792ffd640 10 monclient: _finish_auth 0 2024-04-26T00:33:40.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.263+0000 7f9792ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:33:40.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.263+0000 7f9792ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:40.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.263+0000 7f9792ffd640 1 -- 172.21.15.37:0/2526696013 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97940b2040 con 0x7f979c1371f0 2024-04-26T00:33:40.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.263+0000 7f97a0bd4640 5 monclient: authenticate success, global_id 154223 2024-04-26T00:33:40.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.263+0000 7f97a0bd4640 10 monclient: _renew_subs 2024-04-26T00:33:40.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.263+0000 7f97a0bd4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:33:40.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.263+0000 7f97a0bd4640 1 -- 172.21.15.37:0/2526696013 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f979c120950 con 0x7f979c1371f0 2024-04-26T00:33:40.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.263+0000 7f9792ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:33:40.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.263+0000 7f9792ffd640 1 -- 172.21.15.37:0/2526696013 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97940b1070 con 0x7f979c1371f0 2024-04-26T00:33:40.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.263+0000 7f9792ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:40.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.263+0000 7f9792ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:33:40.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.263+0000 7f9792ffd640 10 monclient: dump: 2024-04-26T00:33:40.269 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:40.269 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:40.269 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:40.269 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:40.269 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:40.269 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:40.269 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:40.269 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:40.270 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:40.270 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:40.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.263+0000 7f9793fff640 4 set_mon_vals no callback set 2024-04-26T00:33:40.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.263+0000 7f97a0bd4640 10 monclient: _renew_subs 2024-04-26T00:33:40.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.263+0000 7f97a0bd4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:33:40.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.263+0000 7f97a0bd4640 1 -- 172.21.15.37:0/2526696013 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f979c1200b0 con 0x7f979c1371f0 2024-04-26T00:33:40.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.263+0000 7f9792ffd640 1 -- 172.21.15.37:0/2526696013 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f97940bf3f0 con 0x7f979c1371f0 2024-04-26T00:33:40.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.263+0000 7f97a0bd4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:40.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.263+0000 7f97a0bd4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:40.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.263+0000 7f97a0bd4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:33:40.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.263+0000 7f97a0bd4640 1 -- 172.21.15.37:0/2526696013 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f979c075910 con 0x7f979c1371f0 2024-04-26T00:33:40.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.263+0000 7f9792ffd640 1 --2- 172.21.15.37:0/2526696013 >> v2:172.21.15.45:6812/2379918623 conn(0x7f977c035e60 0x7f977c038320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:40.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.263+0000 7f9792ffd640 1 -- 172.21.15.37:0/2526696013 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(534..534 src has 370..534) ==== 3959+0+0 (secure 0 0 0) 0x7f97940e1be0 con 0x7f979c1371f0 2024-04-26T00:33:40.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.263+0000 7f979a59c640 1 --2- 172.21.15.37:0/2526696013 >> v2:172.21.15.45:6812/2379918623 conn(0x7f977c035e60 0x7f977c038320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:40.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.263+0000 7f979a59c640 10 monclient: get_auth_request con 0x7f977c035e60 auth_method 0 2024-04-26T00:33:40.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.267+0000 7f979a59c640 1 --2- 172.21.15.37:0/2526696013 >> v2:172.21.15.45:6812/2379918623 conn(0x7f977c035e60 0x7f977c038320 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f979c13de30 tx=0x7f978c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:40.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.267+0000 7f9792ffd640 1 -- 172.21.15.37:0/2526696013 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f979c06d380 con 0x7f979c1371f0 2024-04-26T00:33:40.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.267+0000 7f9792ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:40.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.267+0000 7f9792ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:33:40.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.491+0000 7f97a0bd4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:33:40.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.491+0000 7f97a0bd4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:33:40.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.491+0000 7f97a0bd4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:33:40.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.491+0000 7f97a0bd4640 1 -- 172.21.15.37:0/2526696013 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f979c066f40 con 0x7f979c1371f0 2024-04-26T00:33:40.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.491+0000 7f9792ffd640 1 -- 172.21.15.37:0/2526696013 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f9794016110 con 0x7f979c1371f0 2024-04-26T00:33:40.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.491+0000 7f9792ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:33:40.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.491+0000 7f9792ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:33:40.488 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":102,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:33:40.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.495+0000 7f97a0bd4640 1 -- 172.21.15.37:0/2526696013 >> v2:172.21.15.45:6812/2379918623 conn(0x7f977c035e60 msgr2=0x7f977c038320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:40.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.495+0000 7f97a0bd4640 1 --2- 172.21.15.37:0/2526696013 >> v2:172.21.15.45:6812/2379918623 conn(0x7f977c035e60 0x7f977c038320 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f979c13de30 tx=0x7f978c009190 comp rx=0 tx=0).stop 2024-04-26T00:33:40.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.495+0000 7f97a0bd4640 10 monclient: shutdown 2024-04-26T00:33:40.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.495+0000 7f97a0bd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:33:40.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.495+0000 7f97a0bd4640 1 -- 172.21.15.37:0/2526696013 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f979c1371f0 msgr2=0x7f979c13ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:40.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.495+0000 7f97a0bd4640 1 --2- 172.21.15.37:0/2526696013 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f979c1371f0 0x7f979c13ac10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f97940ae000 tx=0x7f97940191a0 comp rx=0 tx=0).stop 2024-04-26T00:33:40.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.495+0000 7f97a0bd4640 1 -- 172.21.15.37:0/2526696013 shutdown_connections 2024-04-26T00:33:40.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.495+0000 7f97a0bd4640 1 --2- 172.21.15.37:0/2526696013 >> v2:172.21.15.45:6812/2379918623 conn(0x7f977c035e60 0x7f977c038320 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:40.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.495+0000 7f97a0bd4640 1 --2- 172.21.15.37:0/2526696013 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f979c1371f0 0x7f979c13ac10 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:40.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.495+0000 7f97a0bd4640 1 --2- 172.21.15.37:0/2526696013 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f979c136e10 0x7f979c1353f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:40.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.495+0000 7f97a0bd4640 1 --2- 172.21.15.37:0/2526696013 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f979c135d90 0x7f979c134eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:40.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.495+0000 7f97a0bd4640 1 -- 172.21.15.37:0/2526696013 >> 172.21.15.37:0/2526696013 conn(0x7f979c0a64e0 msgr2=0x7f979c0a7c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:33:40.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.495+0000 7f97a0bd4640 1 -- 172.21.15.37:0/2526696013 shutdown_connections 2024-04-26T00:33:40.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:40.495+0000 7f97a0bd4640 1 -- 172.21.15.37:0/2526696013 wait complete. 2024-04-26T00:33:43.506 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:33:43.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.643+0000 7f252b98e640 10 monclient: get_monmap_and_config 2024-04-26T00:33:43.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.643+0000 7f252b98e640 10 monclient: build_initial_monmap 2024-04-26T00:33:43.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.643+0000 7f252b98e640 10 monclient: monmap: 2024-04-26T00:33:43.637 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:33:43.637 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:43.637 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:33:43.645143+0000 2024-04-26T00:33:43.637 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:33:43.645143+0000 2024-04-26T00:33:43.637 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:33:43.637 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:43.637 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:33:43.637 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:33:43.637 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:33:43.637 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:33:43.638 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:33:43.638 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:43.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.643+0000 7f252b98e640 1 Processor -- start 2024-04-26T00:33:43.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.643+0000 7f252b98e640 1 -- start start 2024-04-26T00:33:43.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.643+0000 7f252b98e640 10 monclient: init 2024-04-26T00:33:43.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.643+0000 7f252b98e640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:43.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.643+0000 7f252b98e640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-04-26T00:33:43.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.643+0000 7f252b98e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f25240a0260 addr v1:172.21.15.45:6789/0 2024-04-26T00:33:43.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.643+0000 7f252b98e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2524137e40 0x7f252413a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:43.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.643+0000 7f252b98e640 10 monclient(hunting): picked mon.noname-c con 0x7f2524137e40 addr v2:172.21.15.45:3300/0 2024-04-26T00:33:43.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.643+0000 7f252b98e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f252413a770 addr v1:172.21.15.37:6789/0 2024-04-26T00:33:43.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.643+0000 7f252b98e640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25240a1130 con 0x7f252413a770 2024-04-26T00:33:43.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.643+0000 7f252b98e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f252406cff0 con 0x7f25240a0260 2024-04-26T00:33:43.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.643+0000 7f252b98e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:43.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.643+0000 7f252b98e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f25240a0d50 con 0x7f2524137e40 2024-04-26T00:33:43.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.643+0000 7f252b98e640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:43.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.643+0000 7f252b98e640 10 monclient(hunting): authenticate will time out at 1804.951177s 2024-04-26T00:33:43.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.643+0000 7f2528f29640 1 -- >> v2:172.21.15.45:3300/0 conn(0x7f2524137e40 msgr2=0x7f252413a230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:33:43.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.643+0000 7f252972a640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f25240a0260 legacy=0x7f25240a0640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:33:43.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.643+0000 7f2528f29640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f2524137e40 0x7f252413a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:33:43.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.643+0000 7f2529f2b640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f252413a770 0x7f252413cb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:45014/0 (socket says 172.21.15.37:45014) 2024-04-26T00:33:43.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.643+0000 7f2529f2b640 1 -- 172.21.15.37:0/2709505720 learned_addr learned my addr 172.21.15.37:0/2709505720 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:33:43.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f2529f2b640 1 -- 172.21.15.37:0/2709505720 >> v1:172.21.15.37:6789/0 conn(0x7f252413a770 legacy=0x7f252413cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:43.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f251bfff640 1 -- 172.21.15.37:0/2709505720 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2642045326 0 0) 0x7f251c0017e0 con 0x7f252413a770 2024-04-26T00:33:43.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f251bfff640 10 monclient(hunting): _negotiate 2024-04-26T00:33:43.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f251bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:43.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f251bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:43.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f251bfff640 10 monclient(hunting): my global_id is 144348 2024-04-26T00:33:43.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f251bfff640 1 -- 172.21.15.37:0/2709505720 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2510003530 con 0x7f252413a770 2024-04-26T00:33:43.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f251bfff640 1 -- 172.21.15.37:0/2709505720 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1083582661 0 0) 0x7f2510003530 con 0x7f252413a770 2024-04-26T00:33:43.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f251bfff640 1 -- 172.21.15.37:0/2709505720 >> v2:172.21.15.45:3300/0 conn(0x7f2524137e40 msgr2=0x7f252413a230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:33:43.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f251bfff640 1 --2- 172.21.15.37:0/2709505720 >> v2:172.21.15.45:3300/0 conn(0x7f2524137e40 0x7f252413a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:43.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f251bfff640 1 -- 172.21.15.37:0/2709505720 >> v1:172.21.15.45:6789/0 conn(0x7f25240a0260 legacy=0x7f25240a0640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:33:43.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f251bfff640 10 monclient: _finish_hunting 0 2024-04-26T00:33:43.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f251bfff640 1 monclient: found mon.noname-a-legacy 2024-04-26T00:33:43.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f251bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:43.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f251bfff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.37:6789/0 2024-04-26T00:33:43.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f251bfff640 1 -- 172.21.15.37:0/2709505720 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f252409bc50 con 0x7f252413a770 2024-04-26T00:33:43.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f251bfff640 10 monclient: _finish_auth 0 2024-04-26T00:33:43.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f251bfff640 10 monclient: _check_auth_tickets 2024-04-26T00:33:43.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f251bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:43.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f251bfff640 1 -- 172.21.15.37:0/2709505720 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 866326386 0 0) 0x7f251c001f40 con 0x7f252413a770 2024-04-26T00:33:43.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f251bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:43.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f251bfff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-26T00:33:43.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f251bfff640 10 monclient: dump: 2024-04-26T00:33:43.645 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:43.645 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:43.645 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:43.645 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:43.645 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:43.645 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:43.645 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:43.645 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:43.645 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:43.645 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:43.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f251bfff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] but i'm connected to v1:172.21.15.37:6789/0, reconnecting 2024-04-26T00:33:43.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f251bfff640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:43.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f251bfff640 1 -- 172.21.15.37:0/2709505720 >> v1:172.21.15.37:6789/0 conn(0x7f252413a770 legacy=0x7f252413cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:43.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f251bfff640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:33:43.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f251bfff640 1 --2- 172.21.15.37:0/2709505720 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f25100039c0 0x7f2510003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:43.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f251bfff640 10 monclient(hunting): picked mon.c con 0x7f25100039c0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:33:43.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f251bfff640 1 --2- 172.21.15.37:0/2709505720 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2510004810 0x7f2510004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:43.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f251bfff640 10 monclient(hunting): picked mon.b con 0x7f2510004810 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:33:43.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f251bfff640 1 --2- 172.21.15.37:0/2709505720 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25100053d0 0x7f251000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:43.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f251bfff640 10 monclient(hunting): picked mon.a con 0x7f25100053d0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:33:43.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f251bfff640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:43.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f251bfff640 1 -- 172.21.15.37:0/2709505720 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f25240a0d50 con 0x7f25100053d0 2024-04-26T00:33:43.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f251bfff640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:43.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f251bfff640 1 -- 172.21.15.37:0/2709505720 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f251000c270 con 0x7f2510004810 2024-04-26T00:33:43.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f251bfff640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:43.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f251bfff640 1 -- 172.21.15.37:0/2709505720 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f251000c470 con 0x7f25100039c0 2024-04-26T00:33:43.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f251bfff640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:43.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f251bfff640 10 monclient(hunting): _finish_auth 0 2024-04-26T00:33:43.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f252972a640 1 --2- 172.21.15.37:0/2709505720 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f25100039c0 0x7f2510003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:43.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f2528f29640 1 -- 172.21.15.37:0/2709505720 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2510004810 msgr2=0x7f2510004c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:33:43.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f252972a640 10 monclient(hunting): get_auth_request con 0x7f25100039c0 auth_method 0 2024-04-26T00:33:43.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f252972a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:43.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f252972a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:43.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f252972a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:33:43.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f2529f2b640 1 --2- 172.21.15.37:0/2709505720 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25100053d0 0x7f251000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:43.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f2528f29640 1 --2- 172.21.15.37:0/2709505720 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2510004810 0x7f2510004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:33:43.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f2529f2b640 10 monclient(hunting): get_auth_request con 0x7f25100053d0 auth_method 0 2024-04-26T00:33:43.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f2529f2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:43.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f2529f2b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:43.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f2529f2b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:33:43.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f252972a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:43.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f252972a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:43.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f252972a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:33:43.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f2529f2b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:43.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f2529f2b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:43.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f2529f2b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:33:43.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f252972a640 10 monclient(hunting): handle_auth_done global_id 144348 payload 293 2024-04-26T00:33:43.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f252972a640 1 -- 172.21.15.37:0/2709505720 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2510004810 msgr2=0x7f2510004c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:33:43.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f252972a640 1 --2- 172.21.15.37:0/2709505720 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2510004810 0x7f2510004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:43.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f252972a640 1 -- 172.21.15.37:0/2709505720 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25100053d0 msgr2=0x7f251000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:33:43.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f252972a640 1 --2- 172.21.15.37:0/2709505720 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25100053d0 0x7f251000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:43.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f252972a640 10 monclient: _finish_hunting 0 2024-04-26T00:33:43.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f252972a640 1 monclient: found mon.c 2024-04-26T00:33:43.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f252972a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:43.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f252972a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:33:43.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f252972a640 1 -- 172.21.15.37:0/2709505720 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2510003530 con 0x7f25100039c0 2024-04-26T00:33:43.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f252972a640 10 monclient: _finish_auth 0 2024-04-26T00:33:43.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f252972a640 10 monclient: _check_auth_tickets 2024-04-26T00:33:43.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f2529f2b640 1 -- 172.21.15.37:0/2709505720 reap_dead start 2024-04-26T00:33:43.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f252972a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:43.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f252b98e640 5 monclient: authenticate success, global_id 144348 2024-04-26T00:33:43.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.647+0000 7f252b98e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:33:43.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.651+0000 7f252972a640 1 -- 172.21.15.37:0/2709505720 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f25100039c0 msgr2=0x7f2510003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:43.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.651+0000 7f252972a640 1 --2- 172.21.15.37:0/2709505720 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f25100039c0 0x7f2510003dc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2520003940 tx=0x7f252001c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de2b9d348a1104b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:43.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.651+0000 7f251bfff640 1 -- 172.21.15.37:0/2709505720 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2520009520 con 0x7f25100039c0 2024-04-26T00:33:43.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.651+0000 7f251bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:43.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.651+0000 7f251bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:33:43.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.651+0000 7f251bfff640 10 monclient: dump: 2024-04-26T00:33:43.656 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:43.656 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:43.656 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:43.656 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:43.656 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:43.656 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:43.656 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:43.656 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:43.656 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:43.656 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:43.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.651+0000 7f251bfff640 1 -- 172.21.15.37:0/2709505720 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2520003bc0 con 0x7f25100039c0 2024-04-26T00:33:43.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.651+0000 7f252b98e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:33:43.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.651+0000 7f251bfff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:33:43.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.651+0000 7f251bfff640 1 -- 172.21.15.37:0/2709505720 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2520005070 con 0x7f25100039c0 2024-04-26T00:33:43.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.651+0000 7f252b98e640 10 monclient: get_monmap_and_config success 2024-04-26T00:33:43.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.651+0000 7f252b98e640 4 set_mon_vals no callback set 2024-04-26T00:33:43.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.651+0000 7f251bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:43.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.651+0000 7f251bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:33:43.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.651+0000 7f251bfff640 10 monclient: dump: 2024-04-26T00:33:43.657 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:43.657 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:43.657 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:43.657 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:43.657 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:43.657 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:43.657 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:43.657 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:43.657 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:43.657 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:43.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.651+0000 7f252b98e640 10 monclient: shutdown 2024-04-26T00:33:43.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.651+0000 7f252b98e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:33:43.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.651+0000 7f252b98e640 1 -- 172.21.15.37:0/2709505720 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f25100039c0 msgr2=0x7f2510003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:43.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.651+0000 7f252b98e640 1 --2- 172.21.15.37:0/2709505720 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f25100039c0 0x7f2510003dc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2520003940 tx=0x7f252001c420 comp rx=0 tx=0).stop 2024-04-26T00:33:43.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.651+0000 7f252b98e640 1 -- 172.21.15.37:0/2709505720 shutdown_connections 2024-04-26T00:33:43.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.651+0000 7f252b98e640 1 --2- 172.21.15.37:0/2709505720 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f25100039c0 0x7f2510003dc0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:43.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.651+0000 7f252b98e640 1 -- 172.21.15.37:0/2709505720 >> 172.21.15.37:0/2709505720 conn(0x7f25240a1ad0 msgr2=0x7f25240a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:33:43.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.651+0000 7f252b98e640 1 -- 172.21.15.37:0/2709505720 shutdown_connections 2024-04-26T00:33:43.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.651+0000 7f252b98e640 1 -- 172.21.15.37:0/2709505720 wait complete. 2024-04-26T00:33:43.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f252b98e640 10 monclient: build_initial_monmap 2024-04-26T00:33:43.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f252b98e640 10 monclient: monmap: 2024-04-26T00:33:43.658 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:33:43.658 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:33:43.658 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:33:43.658 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:33:43.659 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:33:43.659 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:33:43.659 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:33:43.659 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:33:43.659 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:33:43.659 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:43.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f252b98e640 1 Processor -- start 2024-04-26T00:33:43.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f252b98e640 1 -- start start 2024-04-26T00:33:43.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f252b98e640 10 monclient: init 2024-04-26T00:33:43.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f252b98e640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:43.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f252b98e640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:33:43.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f252b98e640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25100039c0 0x7f25241ccf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:43.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f252b98e640 10 monclient(hunting): picked mon.noname-a con 0x7f25100039c0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:33:43.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f252b98e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2524131df0 0x7f2524132230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:43.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f252b98e640 10 monclient(hunting): picked mon.noname-b con 0x7f2524131df0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:33:43.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f252b98e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f252411d660 0x7f252411dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:43.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f252b98e640 10 monclient(hunting): picked mon.noname-c con 0x7f252411d660 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:33:43.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f252b98e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:43.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f252b98e640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f25240a00a0 con 0x7f25100039c0 2024-04-26T00:33:43.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f252b98e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:43.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f252b98e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f252409f0f0 con 0x7f2524131df0 2024-04-26T00:33:43.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f252b98e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:43.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f252b98e640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f25241cd4c0 con 0x7f252411d660 2024-04-26T00:33:43.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f252b98e640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:43.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f252b98e640 10 monclient(hunting): authenticate will time out at 1804.961340s 2024-04-26T00:33:43.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f252972a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25100039c0 0x7f25241ccf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:43.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f2529f2b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f252411d660 0x7f252411dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:43.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f252972a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25100039c0 0x7f25241ccf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40170/0 (socket says 172.21.15.37:40170) 2024-04-26T00:33:43.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f252972a640 1 -- 172.21.15.37:0/2707221463 learned_addr learned my addr 172.21.15.37:0/2707221463 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:33:43.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f2529f2b640 10 monclient(hunting): get_auth_request con 0x7f252411d660 auth_method 0 2024-04-26T00:33:43.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f2529f2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:43.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f2529f2b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:43.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f2529f2b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:43.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f252972a640 10 monclient(hunting): get_auth_request con 0x7f25100039c0 auth_method 0 2024-04-26T00:33:43.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f252972a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:43.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f252972a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:43.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f252972a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:43.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f2528f29640 1 -- 172.21.15.37:0/2707221463 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2524131df0 msgr2=0x7f2524132230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:33:43.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f2528f29640 1 --2- 172.21.15.37:0/2707221463 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2524131df0 0x7f2524132230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:33:43.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f2529f2b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:43.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f2529f2b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:43.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f2529f2b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:43.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f252972a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:43.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f252972a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:43.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f252972a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:43.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f2529f2b640 10 monclient(hunting): handle_auth_done global_id 154226 payload 824 2024-04-26T00:33:43.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f2529f2b640 1 -- 172.21.15.37:0/2707221463 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2524131df0 msgr2=0x7f2524132230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:33:43.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f2529f2b640 1 --2- 172.21.15.37:0/2707221463 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2524131df0 0x7f2524132230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:43.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f2529f2b640 1 -- 172.21.15.37:0/2707221463 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25100039c0 msgr2=0x7f25241ccf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:43.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f2529f2b640 1 --2- 172.21.15.37:0/2707221463 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25100039c0 0x7f25241ccf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:43.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f2529f2b640 10 monclient: _finish_hunting 0 2024-04-26T00:33:43.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f2529f2b640 1 monclient: found mon.noname-c 2024-04-26T00:33:43.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f2529f2b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:43.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f2529f2b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:33:43.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f2529f2b640 1 -- 172.21.15.37:0/2707221463 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25241215a0 con 0x7f252411d660 2024-04-26T00:33:43.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.655+0000 7f252972a640 1 --2- 172.21.15.37:0/2707221463 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25100039c0 0x7f25241ccf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:33:43.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.659+0000 7f2529f2b640 1 -- 172.21.15.37:0/2707221463 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f252411d660 msgr2=0x7f252411dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:43.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.659+0000 7f2529f2b640 1 --2- 172.21.15.37:0/2707221463 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f252411d660 0x7f252411dac0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f25240a15d0 tx=0x7f251c008430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:43.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.659+0000 7f2519ffb640 1 -- 172.21.15.37:0/2707221463 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f251c0143f0 con 0x7f252411d660 2024-04-26T00:33:43.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.659+0000 7f2519ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:43.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.659+0000 7f2519ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:33:43.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.659+0000 7f2519ffb640 10 monclient: dump: 2024-04-26T00:33:43.663 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:43.663 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:43.663 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:43.663 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:43.663 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:43.663 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:43.663 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:43.664 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:43.664 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:43.664 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:43.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.659+0000 7f2519ffb640 10 monclient: _finish_auth 0 2024-04-26T00:33:43.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.659+0000 7f2519ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:33:43.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.659+0000 7f2519ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:43.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.659+0000 7f252b98e640 5 monclient: authenticate success, global_id 154226 2024-04-26T00:33:43.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.659+0000 7f252b98e640 10 monclient: _renew_subs 2024-04-26T00:33:43.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.659+0000 7f2519ffb640 1 -- 172.21.15.37:0/2707221463 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f251c019070 con 0x7f252411d660 2024-04-26T00:33:43.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.659+0000 7f252b98e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:33:43.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.659+0000 7f252b98e640 1 -- 172.21.15.37:0/2707221463 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2524132770 con 0x7f252411d660 2024-04-26T00:33:43.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.659+0000 7f2519ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:33:43.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.659+0000 7f2519ffb640 1 -- 172.21.15.37:0/2707221463 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f251c0147b0 con 0x7f252411d660 2024-04-26T00:33:43.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.659+0000 7f2519ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:43.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.659+0000 7f2519ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:33:43.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.659+0000 7f2519ffb640 10 monclient: dump: 2024-04-26T00:33:43.665 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:43.665 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:43.665 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:43.665 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:43.665 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:43.665 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:43.665 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:43.665 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:43.665 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:43.665 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:43.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.659+0000 7f251affd640 4 set_mon_vals no callback set 2024-04-26T00:33:43.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.659+0000 7f252b98e640 10 monclient: _renew_subs 2024-04-26T00:33:43.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.659+0000 7f252b98e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:33:43.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.659+0000 7f252b98e640 1 -- 172.21.15.37:0/2707221463 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f252409b650 con 0x7f252411d660 2024-04-26T00:33:43.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.659+0000 7f2519ffb640 1 -- 172.21.15.37:0/2707221463 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f251c0149d0 con 0x7f252411d660 2024-04-26T00:33:43.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.659+0000 7f252b98e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:43.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.659+0000 7f252b98e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:43.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.659+0000 7f252b98e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:33:43.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.659+0000 7f252b98e640 1 -- 172.21.15.37:0/2707221463 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2524066f40 con 0x7f252411d660 2024-04-26T00:33:43.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.659+0000 7f2519ffb640 1 --2- 172.21.15.37:0/2707221463 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2508035eb0 0x7f2508038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:43.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.659+0000 7f2519ffb640 1 -- 172.21.15.37:0/2707221463 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(534..534 src has 370..534) ==== 3959+0+0 (secure 0 0 0) 0x7f251c0490f0 con 0x7f252411d660 2024-04-26T00:33:43.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.659+0000 7f252972a640 1 --2- 172.21.15.37:0/2707221463 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2508035eb0 0x7f2508038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:43.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.659+0000 7f252972a640 10 monclient: get_auth_request con 0x7f2508035eb0 auth_method 0 2024-04-26T00:33:43.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.663+0000 7f252972a640 1 --2- 172.21.15.37:0/2707221463 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2508035eb0 0x7f2508038370 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f2520003940 tx=0x7f2520005a90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:43.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.663+0000 7f2519ffb640 1 -- 172.21.15.37:0/2707221463 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f25240a1130 con 0x7f252411d660 2024-04-26T00:33:43.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.663+0000 7f2519ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:43.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.663+0000 7f2519ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:33:43.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.887+0000 7f252b98e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:33:43.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.887+0000 7f252b98e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:33:43.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.887+0000 7f252b98e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:33:43.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.887+0000 7f252b98e640 1 -- 172.21.15.37:0/2707221463 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f252411f2b0 con 0x7f252411d660 2024-04-26T00:33:43.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.887+0000 7f2519ffb640 1 -- 172.21.15.37:0/2707221463 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f251c01c360 con 0x7f252411d660 2024-04-26T00:33:43.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.887+0000 7f2519ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:33:43.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.887+0000 7f2519ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:33:43.881 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":102,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:33:43.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.887+0000 7f252b98e640 1 -- 172.21.15.37:0/2707221463 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2508035eb0 msgr2=0x7f2508038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:43.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.887+0000 7f252b98e640 1 --2- 172.21.15.37:0/2707221463 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2508035eb0 0x7f2508038370 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f2520003940 tx=0x7f2520005a90 comp rx=0 tx=0).stop 2024-04-26T00:33:43.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.887+0000 7f252b98e640 10 monclient: shutdown 2024-04-26T00:33:43.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.887+0000 7f252b98e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:33:43.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.887+0000 7f252b98e640 1 -- 172.21.15.37:0/2707221463 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f252411d660 msgr2=0x7f252411dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:43.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.887+0000 7f252b98e640 1 --2- 172.21.15.37:0/2707221463 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f252411d660 0x7f252411dac0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f25240a15d0 tx=0x7f251c008430 comp rx=0 tx=0).stop 2024-04-26T00:33:43.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.887+0000 7f252b98e640 1 -- 172.21.15.37:0/2707221463 shutdown_connections 2024-04-26T00:33:43.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.887+0000 7f252b98e640 1 --2- 172.21.15.37:0/2707221463 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2508035eb0 0x7f2508038370 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:43.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.887+0000 7f252b98e640 1 --2- 172.21.15.37:0/2707221463 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f252411d660 0x7f252411dac0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:43.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.887+0000 7f252b98e640 1 --2- 172.21.15.37:0/2707221463 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2524131df0 0x7f2524132230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:43.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.887+0000 7f252b98e640 1 --2- 172.21.15.37:0/2707221463 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25100039c0 0x7f25241ccf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:43.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.887+0000 7f252b98e640 1 -- 172.21.15.37:0/2707221463 >> 172.21.15.37:0/2707221463 conn(0x7f25240a1ad0 msgr2=0x7f25240a3d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:33:43.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.887+0000 7f252b98e640 1 -- 172.21.15.37:0/2707221463 shutdown_connections 2024-04-26T00:33:43.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:43.887+0000 7f252b98e640 1 -- 172.21.15.37:0/2707221463 wait complete. 2024-04-26T00:33:46.900 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:33:47.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa07028c640 10 monclient: get_monmap_and_config 2024-04-26T00:33:47.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa07028c640 10 monclient: build_initial_monmap 2024-04-26T00:33:47.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa07028c640 10 monclient: monmap: 2024-04-26T00:33:47.033 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:33:47.033 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:47.033 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:33:47.041399+0000 2024-04-26T00:33:47.033 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:33:47.041399+0000 2024-04-26T00:33:47.033 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:33:47.034 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:47.034 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:33:47.034 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:33:47.034 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:33:47.034 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:33:47.034 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:33:47.034 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:47.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa07028c640 1 Processor -- start 2024-04-26T00:33:47.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa07028c640 1 -- start start 2024-04-26T00:33:47.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa07028c640 10 monclient: init 2024-04-26T00:33:47.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa07028c640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:47.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa07028c640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-04-26T00:33:47.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa07028c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa068133b20 addr v1:172.21.15.45:6789/0 2024-04-26T00:33:47.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa07028c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa068133f00 0x7fa068134350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:47.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa07028c640 10 monclient(hunting): picked mon.noname-b con 0x7fa068133f00 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:33:47.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa07028c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa068139050 0x7fa06813b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:47.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa07028c640 10 monclient(hunting): picked mon.noname-c con 0x7fa068139050 addr v2:172.21.15.45:3300/0 2024-04-26T00:33:47.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa07028c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa06809fe20 con 0x7fa068133b20 2024-04-26T00:33:47.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa07028c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:47.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa07028c640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fa06809e620 con 0x7fa068139050 2024-04-26T00:33:47.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa07028c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:47.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa07028c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa068072d00 con 0x7fa068133f00 2024-04-26T00:33:47.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa07028c640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:47.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa07028c640 10 monclient(hunting): authenticate will time out at 1808.347005s 2024-04-26T00:33:47.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa06d827640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa068133f00 0x7fa068134350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:47.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa06e829640 1 -- >> v2:172.21.15.45:3300/0 conn(0x7fa068139050 msgr2=0x7fa06813b440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:33:47.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa06e028640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7fa068133b20 legacy=0x7fa06809e9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:33:47.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa06e829640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa068139050 0x7fa06813b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:33:47.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa06d827640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa068133f00 0x7fa068134350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:37382/0 (socket says 172.21.15.37:37382) 2024-04-26T00:33:47.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa06d827640 1 -- 172.21.15.37:0/2171315707 learned_addr learned my addr 172.21.15.37:0/2171315707 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:33:47.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa06d827640 10 monclient(hunting): get_auth_request con 0x7fa068133f00 auth_method 0 2024-04-26T00:33:47.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa06d827640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:47.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa06d827640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:47.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa06d827640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:47.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa06d827640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:47.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa06d827640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:47.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa06d827640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:47.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa06d827640 10 monclient(hunting): handle_auth_done global_id 154229 payload 274 2024-04-26T00:33:47.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa06d827640 1 -- 172.21.15.37:0/2171315707 >> v2:172.21.15.45:3300/0 conn(0x7fa068139050 msgr2=0x7fa06813b440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:33:47.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa06d827640 1 --2- 172.21.15.37:0/2171315707 >> v2:172.21.15.45:3300/0 conn(0x7fa068139050 0x7fa06813b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:47.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa06d827640 1 -- 172.21.15.37:0/2171315707 >> v1:172.21.15.45:6789/0 conn(0x7fa068133b20 legacy=0x7fa06809e9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:33:47.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa06d827640 10 monclient: _finish_hunting 0 2024-04-26T00:33:47.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa06d827640 1 monclient: found mon.noname-b 2024-04-26T00:33:47.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa06d827640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:47.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa06d827640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:33:47.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.039+0000 7fa06d827640 1 -- 172.21.15.37:0/2171315707 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa06809ba60 con 0x7fa068133f00 2024-04-26T00:33:47.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.043+0000 7fa06d827640 1 -- 172.21.15.37:0/2171315707 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa068133f00 msgr2=0x7fa068134350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:47.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.043+0000 7fa06d827640 1 --2- 172.21.15.37:0/2171315707 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa068133f00 0x7fa068134350 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa058009870 tx=0x7fa05801c220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ddd47ebddc4b50f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:47.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06d026640 1 -- 172.21.15.37:0/2171315707 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0580b0070 con 0x7fa068133f00 2024-04-26T00:33:47.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06d026640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:47.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06d026640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:33:47.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06d026640 10 monclient: dump: 2024-04-26T00:33:47.044 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:47.044 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:47.044 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:47.044 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:47.044 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:47.044 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:47.044 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:47.044 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:47.044 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:47.045 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:47.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06d026640 10 monclient: _finish_auth 0 2024-04-26T00:33:47.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06d026640 10 monclient: _check_auth_tickets 2024-04-26T00:33:47.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06d026640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:47.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa07028c640 5 monclient: authenticate success, global_id 154229 2024-04-26T00:33:47.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa07028c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:33:47.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06d026640 1 -- 172.21.15.37:0/2171315707 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0580ab600 con 0x7fa068133f00 2024-04-26T00:33:47.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06d026640 10 monclient: handle_config config(0 keys) 2024-04-26T00:33:47.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa07028c640 10 monclient: get_monmap_and_config success 2024-04-26T00:33:47.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06d026640 1 -- 172.21.15.37:0/2171315707 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0580ab9d0 con 0x7fa068133f00 2024-04-26T00:33:47.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06d026640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:47.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa07028c640 4 set_mon_vals no callback set 2024-04-26T00:33:47.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06d026640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:33:47.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06d026640 10 monclient: dump: 2024-04-26T00:33:47.046 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:47.046 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:47.046 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:47.046 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:47.046 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:47.046 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:47.046 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:47.046 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:47.046 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:47.046 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:47.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa07028c640 10 monclient: shutdown 2024-04-26T00:33:47.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa07028c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:33:47.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa07028c640 1 -- 172.21.15.37:0/2171315707 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa068133f00 msgr2=0x7fa068134350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:47.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa07028c640 1 --2- 172.21.15.37:0/2171315707 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa068133f00 0x7fa068134350 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa058009870 tx=0x7fa05801c220 comp rx=0 tx=0).stop 2024-04-26T00:33:47.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa07028c640 1 -- 172.21.15.37:0/2171315707 shutdown_connections 2024-04-26T00:33:47.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa07028c640 1 --2- 172.21.15.37:0/2171315707 >> v2:172.21.15.45:3300/0 conn(0x7fa068139050 0x7fa06813b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:47.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa07028c640 1 --2- 172.21.15.37:0/2171315707 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa068133f00 0x7fa068134350 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:47.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa07028c640 1 -- 172.21.15.37:0/2171315707 >> 172.21.15.37:0/2171315707 conn(0x7fa0680a1ad0 msgr2=0x7fa0680a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:33:47.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa07028c640 1 -- 172.21.15.37:0/2171315707 shutdown_connections 2024-04-26T00:33:47.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa07028c640 1 -- 172.21.15.37:0/2171315707 wait complete. 2024-04-26T00:33:47.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa07028c640 10 monclient: build_initial_monmap 2024-04-26T00:33:47.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa07028c640 10 monclient: monmap: 2024-04-26T00:33:47.047 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:33:47.048 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:33:47.048 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:33:47.048 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:33:47.048 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:33:47.048 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:33:47.048 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:33:47.048 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:33:47.048 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:33:47.048 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:47.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa07028c640 1 Processor -- start 2024-04-26T00:33:47.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa07028c640 1 -- start start 2024-04-26T00:33:47.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa07028c640 10 monclient: init 2024-04-26T00:33:47.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa07028c640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:47.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa07028c640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:33:47.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa07028c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa068133b20 0x7fa0681cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:47.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa07028c640 10 monclient(hunting): picked mon.noname-c con 0x7fa068133b20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:33:47.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa07028c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa068133f00 0x7fa06812fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:47.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa07028c640 10 monclient(hunting): picked mon.noname-b con 0x7fa068133f00 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:33:47.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa07028c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa068139050 0x7fa0681ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:47.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa07028c640 10 monclient(hunting): picked mon.noname-a con 0x7fa068139050 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:33:47.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa07028c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:47.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa07028c640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa06813e320 con 0x7fa068139050 2024-04-26T00:33:47.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa07028c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:47.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa07028c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa06813e1a0 con 0x7fa068133f00 2024-04-26T00:33:47.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa07028c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:47.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa07028c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa06813e4a0 con 0x7fa068133b20 2024-04-26T00:33:47.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa07028c640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:47.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa07028c640 10 monclient(hunting): authenticate will time out at 1808.354685s 2024-04-26T00:33:47.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06d827640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa068133f00 msgr2=0x7fa06812fe80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:33:47.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06e829640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa068139050 0x7fa0681ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:47.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06e028640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa068133b20 0x7fa0681cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:47.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06d827640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa068133f00 0x7fa06812fe80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:33:47.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06e829640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa068139050 0x7fa0681ceec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40186/0 (socket says 172.21.15.37:40186) 2024-04-26T00:33:47.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06e028640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa068133b20 0x7fa0681cb9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:37396/0 (socket says 172.21.15.37:37396) 2024-04-26T00:33:47.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06e829640 1 -- 172.21.15.37:0/135748505 learned_addr learned my addr 172.21.15.37:0/135748505 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:33:47.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06e829640 10 monclient(hunting): get_auth_request con 0x7fa068139050 auth_method 0 2024-04-26T00:33:47.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06e829640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:47.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06e829640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:47.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06e829640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:47.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06e028640 10 monclient(hunting): get_auth_request con 0x7fa068133b20 auth_method 0 2024-04-26T00:33:47.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06e028640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:47.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06e028640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:47.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06e028640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:47.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06e829640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:47.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06e829640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:47.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06e829640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:47.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06e028640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:47.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06e028640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:47.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06e028640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:47.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06e829640 10 monclient(hunting): handle_auth_done global_id 144354 payload 824 2024-04-26T00:33:47.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06e829640 1 -- 172.21.15.37:0/135748505 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa068133b20 msgr2=0x7fa0681cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:47.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06e829640 1 --2- 172.21.15.37:0/135748505 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa068133b20 0x7fa0681cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:47.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06e829640 1 -- 172.21.15.37:0/135748505 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa068133f00 msgr2=0x7fa06812fe80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:33:47.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06e829640 1 --2- 172.21.15.37:0/135748505 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa068133f00 0x7fa06812fe80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:47.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06e829640 10 monclient: _finish_hunting 0 2024-04-26T00:33:47.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06e829640 1 monclient: found mon.noname-a 2024-04-26T00:33:47.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06e829640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:47.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06e829640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:33:47.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.047+0000 7fa06e829640 1 -- 172.21.15.37:0/135748505 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0681cc400 con 0x7fa068139050 2024-04-26T00:33:47.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.051+0000 7fa06e829640 1 -- 172.21.15.37:0/135748505 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa068139050 msgr2=0x7fa0681ceec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:47.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.051+0000 7fa06e829640 1 --2- 172.21.15.37:0/135748505 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa068139050 0x7fa0681ceec0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa06000be20 tx=0x7fa060009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:47.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.051+0000 7fa05effd640 1 -- 172.21.15.37:0/135748505 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa060016630 con 0x7fa068139050 2024-04-26T00:33:47.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.051+0000 7fa05effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:47.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.051+0000 7fa05effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:33:47.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.051+0000 7fa05effd640 10 monclient: dump: 2024-04-26T00:33:47.056 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:47.056 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:47.056 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:47.056 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:47.056 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:47.056 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:47.056 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:47.056 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:47.056 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:47.057 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:47.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.051+0000 7fa05effd640 10 monclient: _finish_auth 0 2024-04-26T00:33:47.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.051+0000 7fa05effd640 10 monclient: _check_auth_tickets 2024-04-26T00:33:47.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.051+0000 7fa05effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:47.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.051+0000 7fa07028c640 5 monclient: authenticate success, global_id 144354 2024-04-26T00:33:47.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.051+0000 7fa07028c640 10 monclient: _renew_subs 2024-04-26T00:33:47.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.051+0000 7fa05effd640 1 -- 172.21.15.37:0/135748505 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa060010030 con 0x7fa068139050 2024-04-26T00:33:47.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.051+0000 7fa07028c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:33:47.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.051+0000 7fa07028c640 1 -- 172.21.15.37:0/135748505 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa06811be30 con 0x7fa068139050 2024-04-26T00:33:47.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.051+0000 7fa05effd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:33:47.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.051+0000 7fa05effd640 1 -- 172.21.15.37:0/135748505 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa060016a30 con 0x7fa068139050 2024-04-26T00:33:47.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.051+0000 7fa05effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:47.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.051+0000 7fa05effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:33:47.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.051+0000 7fa05effd640 10 monclient: dump: 2024-04-26T00:33:47.058 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:47.058 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:47.058 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:47.058 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:47.058 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:47.058 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:47.058 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:47.058 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:47.058 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:47.058 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:47.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.051+0000 7fa06c825640 4 set_mon_vals no callback set 2024-04-26T00:33:47.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.051+0000 7fa07028c640 10 monclient: _renew_subs 2024-04-26T00:33:47.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.051+0000 7fa07028c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:33:47.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.051+0000 7fa07028c640 1 -- 172.21.15.37:0/135748505 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0680a0000 con 0x7fa068139050 2024-04-26T00:33:47.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.051+0000 7fa07028c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:47.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.051+0000 7fa07028c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:47.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.051+0000 7fa07028c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:33:47.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.051+0000 7fa07028c640 1 -- 172.21.15.37:0/135748505 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa068066f40 con 0x7fa068139050 2024-04-26T00:33:47.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.051+0000 7fa05effd640 1 -- 172.21.15.37:0/135748505 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa0600153e0 con 0x7fa068139050 2024-04-26T00:33:47.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.051+0000 7fa05effd640 1 --2- 172.21.15.37:0/135748505 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa04c035e30 0x7fa04c0382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:47.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.051+0000 7fa05effd640 1 -- 172.21.15.37:0/135748505 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(534..534 src has 370..534) ==== 3959+0+0 (secure 0 0 0) 0x7fa060049570 con 0x7fa068139050 2024-04-26T00:33:47.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.051+0000 7fa06e028640 1 --2- 172.21.15.37:0/135748505 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa04c035e30 0x7fa04c0382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:47.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.051+0000 7fa06e028640 10 monclient: get_auth_request con 0x7fa04c035e30 auth_method 0 2024-04-26T00:33:47.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.055+0000 7fa06e028640 1 --2- 172.21.15.37:0/135748505 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa04c035e30 0x7fa04c0382f0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fa068072830 tx=0x7fa064009110 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:47.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.055+0000 7fa05effd640 1 -- 172.21.15.37:0/135748505 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa06001b020 con 0x7fa068139050 2024-04-26T00:33:47.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.055+0000 7fa05effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:47.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.055+0000 7fa05effd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:33:47.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.275+0000 7fa07028c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:33:47.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.275+0000 7fa07028c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:33:47.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.275+0000 7fa07028c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:33:47.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.275+0000 7fa07028c640 1 -- 172.21.15.37:0/135748505 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa06806b9a0 con 0x7fa068139050 2024-04-26T00:33:47.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.275+0000 7fa05effd640 1 -- 172.21.15.37:0/135748505 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1558 (secure 0 0 0) 0x7fa0681cc400 con 0x7fa068139050 2024-04-26T00:33:47.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.275+0000 7fa05effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:33:47.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.275+0000 7fa05effd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:33:47.271 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":102,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:33:47.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.279+0000 7fa07028c640 1 -- 172.21.15.37:0/135748505 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa04c035e30 msgr2=0x7fa04c0382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:47.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.279+0000 7fa07028c640 1 --2- 172.21.15.37:0/135748505 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa04c035e30 0x7fa04c0382f0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fa068072830 tx=0x7fa064009110 comp rx=0 tx=0).stop 2024-04-26T00:33:47.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.279+0000 7fa07028c640 10 monclient: shutdown 2024-04-26T00:33:47.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.279+0000 7fa07028c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:33:47.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.279+0000 7fa07028c640 1 -- 172.21.15.37:0/135748505 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa068139050 msgr2=0x7fa0681ceec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:47.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.279+0000 7fa07028c640 1 --2- 172.21.15.37:0/135748505 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa068139050 0x7fa0681ceec0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa06000be20 tx=0x7fa060009f10 comp rx=0 tx=0).stop 2024-04-26T00:33:47.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.279+0000 7fa07028c640 1 -- 172.21.15.37:0/135748505 shutdown_connections 2024-04-26T00:33:47.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.279+0000 7fa07028c640 1 --2- 172.21.15.37:0/135748505 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa04c035e30 0x7fa04c0382f0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:47.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.279+0000 7fa07028c640 1 --2- 172.21.15.37:0/135748505 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa068139050 0x7fa0681ceec0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:47.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.279+0000 7fa07028c640 1 --2- 172.21.15.37:0/135748505 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa068133f00 0x7fa06812fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:47.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.279+0000 7fa07028c640 1 --2- 172.21.15.37:0/135748505 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa068133b20 0x7fa0681cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:47.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.279+0000 7fa07028c640 1 -- 172.21.15.37:0/135748505 >> 172.21.15.37:0/135748505 conn(0x7fa0680a1ad0 msgr2=0x7fa0680a0260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:33:47.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.279+0000 7fa07028c640 1 -- 172.21.15.37:0/135748505 shutdown_connections 2024-04-26T00:33:47.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:47.279+0000 7fa07028c640 1 -- 172.21.15.37:0/135748505 wait complete. 2024-04-26T00:33:50.289 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:33:50.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.427+0000 7f85d668e640 10 monclient: get_monmap_and_config 2024-04-26T00:33:50.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.427+0000 7f85d668e640 10 monclient: build_initial_monmap 2024-04-26T00:33:50.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.427+0000 7f85d668e640 10 monclient: monmap: 2024-04-26T00:33:50.421 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:33:50.421 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:50.421 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:33:50.429368+0000 2024-04-26T00:33:50.421 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:33:50.429368+0000 2024-04-26T00:33:50.421 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:33:50.422 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:50.422 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:33:50.422 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:33:50.422 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:33:50.422 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:33:50.422 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:33:50.422 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:50.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.427+0000 7f85d668e640 1 Processor -- start 2024-04-26T00:33:50.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.427+0000 7f85d668e640 1 -- start start 2024-04-26T00:33:50.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.427+0000 7f85d668e640 10 monclient: init 2024-04-26T00:33:50.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.427+0000 7f85d668e640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:50.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.427+0000 7f85d668e640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-04-26T00:33:50.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.427+0000 7f85d668e640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f85d00a0260 0x7f85d00a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:50.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.427+0000 7f85d668e640 10 monclient(hunting): picked mon.noname-a con 0x7f85d00a0260 addr v2:172.21.15.37:3300/0 2024-04-26T00:33:50.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.427+0000 7f85d668e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f85d0134b30 addr v1:172.21.15.45:6789/0 2024-04-26T00:33:50.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.427+0000 7f85d668e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f85d0135660 addr v1:172.21.15.37:6789/0 2024-04-26T00:33:50.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.427+0000 7f85d668e640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85d00a1130 con 0x7f85d0135660 2024-04-26T00:33:50.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.427+0000 7f85d668e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85d006cff0 con 0x7f85d0134b30 2024-04-26T00:33:50.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.427+0000 7f85d668e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:50.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.427+0000 7f85d668e640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f85d009ec10 con 0x7f85d00a0260 2024-04-26T00:33:50.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.427+0000 7f85d668e640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:50.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.427+0000 7f85d668e640 10 monclient(hunting): authenticate will time out at 1811.735061s 2024-04-26T00:33:50.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.427+0000 7f85d4c2b640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f85d0135660 0x7f85d013ce70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:58960/0 (socket says 172.21.15.37:58960) 2024-04-26T00:33:50.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.427+0000 7f85cffff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f85d00a0260 0x7f85d00a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:50.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.427+0000 7f85cf7fe640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f85d0134b30 legacy=0x7f85d0134f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:33:50.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.427+0000 7f85d4c2b640 1 -- 172.21.15.37:0/1604420711 learned_addr learned my addr 172.21.15.37:0/1604420711 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:33:50.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.427+0000 7f85cffff640 10 monclient(hunting): get_auth_request con 0x7f85d00a0260 auth_method 0 2024-04-26T00:33:50.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.427+0000 7f85cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:50.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.427+0000 7f85cffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:50.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.427+0000 7f85cffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:50.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.427+0000 7f85cffff640 1 -- 172.21.15.37:0/1604420711 >> v2:172.21.15.37:3300/0 conn(0x7f85d00a0260 msgr2=0x7f85d00a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:33:50.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.427+0000 7f85cffff640 1 -- 172.21.15.37:0/1604420711 >> v2:172.21.15.37:3300/0 conn(0x7f85d00a0260 msgr2=0x7f85d00a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:33:50.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.427+0000 7f85cffff640 1 --2- 172.21.15.37:0/1604420711 >> v2:172.21.15.37:3300/0 conn(0x7f85d00a0260 0x7f85d00a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:33:50.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.427+0000 7f85cffff640 1 --2- 172.21.15.37:0/1604420711 >> v2:172.21.15.37:3300/0 conn(0x7f85d00a0260 0x7f85d00a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:33:50.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.431+0000 7f85d4c2b640 1 -- 172.21.15.37:0/1604420711 >> v1:172.21.15.37:6789/0 conn(0x7f85d0135660 legacy=0x7f85d013ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:50.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.631+0000 7f85cf7fe640 1 -- 172.21.15.37:0/1604420711 >> v1:172.21.15.45:6789/0 conn(0x7f85d0134b30 legacy=0x7f85d0134f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:33:50.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.631+0000 7f85cffff640 1 --2- 172.21.15.37:0/1604420711 >> v2:172.21.15.37:3300/0 conn(0x7f85d00a0260 0x7f85d00a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:50.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.631+0000 7f85cffff640 10 monclient(hunting): get_auth_request con 0x7f85d00a0260 auth_method 0 2024-04-26T00:33:50.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.631+0000 7f85cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:50.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.631+0000 7f85cffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:50.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.631+0000 7f85cffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:33:50.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.631+0000 7f85cffff640 1 -- 172.21.15.37:0/1604420711 >> v2:172.21.15.37:3300/0 conn(0x7f85d00a0260 msgr2=0x7f85d00a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:33:50.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.631+0000 7f85cffff640 1 -- 172.21.15.37:0/1604420711 >> v2:172.21.15.37:3300/0 conn(0x7f85d00a0260 msgr2=0x7f85d00a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:33:50.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.631+0000 7f85cffff640 1 --2- 172.21.15.37:0/1604420711 >> v2:172.21.15.37:3300/0 conn(0x7f85d00a0260 0x7f85d00a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:33:50.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:50.631+0000 7f85cffff640 1 --2- 172.21.15.37:0/1604420711 >> v2:172.21.15.37:3300/0 conn(0x7f85d00a0260 0x7f85d00a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:33:51.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:51.035+0000 7f85cf7fe640 1 -- 172.21.15.37:0/1604420711 >> v1:172.21.15.45:6789/0 conn(0x7f85d0134b30 legacy=0x7f85d0134f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:33:51.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:51.035+0000 7f85cffff640 1 --2- 172.21.15.37:0/1604420711 >> v2:172.21.15.37:3300/0 conn(0x7f85d00a0260 0x7f85d00a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:51.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:51.035+0000 7f85cffff640 10 monclient(hunting): get_auth_request con 0x7f85d00a0260 auth_method 0 2024-04-26T00:33:51.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:51.035+0000 7f85cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:51.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:51.035+0000 7f85cffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:51.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:51.035+0000 7f85cffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:33:51.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:51.035+0000 7f85cffff640 1 -- 172.21.15.37:0/1604420711 >> v2:172.21.15.37:3300/0 conn(0x7f85d00a0260 msgr2=0x7f85d00a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:33:51.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:51.035+0000 7f85cffff640 1 -- 172.21.15.37:0/1604420711 >> v2:172.21.15.37:3300/0 conn(0x7f85d00a0260 msgr2=0x7f85d00a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:33:51.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:51.035+0000 7f85cffff640 1 --2- 172.21.15.37:0/1604420711 >> v2:172.21.15.37:3300/0 conn(0x7f85d00a0260 0x7f85d00a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:33:51.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:51.035+0000 7f85cffff640 1 --2- 172.21.15.37:0/1604420711 >> v2:172.21.15.37:3300/0 conn(0x7f85d00a0260 0x7f85d00a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:33:51.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:51.839+0000 7f85cf7fe640 1 -- 172.21.15.37:0/1604420711 >> v1:172.21.15.45:6789/0 conn(0x7f85d0134b30 legacy=0x7f85d0134f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:33:51.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:51.839+0000 7f85cffff640 1 --2- 172.21.15.37:0/1604420711 >> v2:172.21.15.37:3300/0 conn(0x7f85d00a0260 0x7f85d00a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:51.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:51.839+0000 7f85cffff640 10 monclient(hunting): get_auth_request con 0x7f85d00a0260 auth_method 0 2024-04-26T00:33:51.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:51.839+0000 7f85cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:51.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:51.839+0000 7f85cffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:51.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:51.839+0000 7f85cffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:33:51.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:51.839+0000 7f85cffff640 1 -- 172.21.15.37:0/1604420711 >> v2:172.21.15.37:3300/0 conn(0x7f85d00a0260 msgr2=0x7f85d00a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:33:51.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:51.839+0000 7f85cffff640 1 -- 172.21.15.37:0/1604420711 >> v2:172.21.15.37:3300/0 conn(0x7f85d00a0260 msgr2=0x7f85d00a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:33:51.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:51.839+0000 7f85cffff640 1 --2- 172.21.15.37:0/1604420711 >> v2:172.21.15.37:3300/0 conn(0x7f85d00a0260 0x7f85d00a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:33:51.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:51.839+0000 7f85cffff640 1 --2- 172.21.15.37:0/1604420711 >> v2:172.21.15.37:3300/0 conn(0x7f85d00a0260 0x7f85d00a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:33:53.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:53.427+0000 7f85cdffb640 10 monclient(hunting): tick 2024-04-26T00:33:53.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:53.427+0000 7f85cdffb640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:33:53.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:53.427+0000 7f85cdffb640 1 monclient(hunting): continuing hunt 2024-04-26T00:33:53.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:53.427+0000 7f85cdffb640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:33:53.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:53.427+0000 7f85cdffb640 1 -- 172.21.15.37:0/1604420711 >> v2:172.21.15.37:3300/0 conn(0x7f85d00a0260 msgr2=0x7f85d00a0640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:33:53.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:53.427+0000 7f85cdffb640 1 --2- 172.21.15.37:0/1604420711 >> v2:172.21.15.37:3300/0 conn(0x7f85d00a0260 0x7f85d00a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:53.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:53.427+0000 7f85cdffb640 1 -- 172.21.15.37:0/1604420711 >> v1:172.21.15.45:6789/0 conn(0x7f85d0134b30 legacy=0x7f85d0134f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:33:53.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:53.427+0000 7f85cdffb640 1 -- 172.21.15.37:0/1604420711 >> v1:172.21.15.37:6789/0 conn(0x7f85d0135660 legacy=0x7f85d013ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:53.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:53.427+0000 7f85cdffb640 10 monclient: _add_conns ranks=[4,2] 2024-04-26T00:33:53.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:53.427+0000 7f85cdffb640 1 --2- 172.21.15.37:0/1604420711 >> v2:172.21.15.45:3300/0 conn(0x7f85b0003750 0x7f85b0005bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:53.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:53.427+0000 7f85cdffb640 10 monclient(hunting): picked mon.noname-c con 0x7f85b0003750 addr v2:172.21.15.45:3300/0 2024-04-26T00:33:53.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:53.427+0000 7f85cdffb640 1 --2- 172.21.15.37:0/1604420711 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b0006240 0x7f85b0008700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:53.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:53.427+0000 7f85cdffb640 10 monclient(hunting): picked mon.noname-b con 0x7f85b0006240 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:33:53.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:53.427+0000 7f85cdffb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:53.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:53.427+0000 7f85cdffb640 1 -- 172.21.15.37:0/1604420711 --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f85b0008e00 con 0x7f85b0003750 2024-04-26T00:33:53.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:53.427+0000 7f85cdffb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:53.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:53.427+0000 7f85cdffb640 1 -- 172.21.15.37:0/1604420711 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f85b0008fb0 con 0x7f85b0006240 2024-04-26T00:33:53.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:53.427+0000 7f85cdffb640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:53.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:53.427+0000 7f85cffff640 1 -- 172.21.15.37:0/1604420711 >> v2:172.21.15.45:3300/0 conn(0x7f85b0003750 msgr2=0x7f85b0005bb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:33:53.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:53.427+0000 7f85cffff640 1 --2- 172.21.15.37:0/1604420711 >> v2:172.21.15.45:3300/0 conn(0x7f85b0003750 0x7f85b0005bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:33:53.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:53.427+0000 7f85cf7fe640 1 --2- 172.21.15.37:0/1604420711 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b0006240 0x7f85b0008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:53.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:53.427+0000 7f85cf7fe640 10 monclient(hunting): get_auth_request con 0x7f85b0006240 auth_method 0 2024-04-26T00:33:53.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:53.427+0000 7f85cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:53.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:53.427+0000 7f85cf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:53.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:53.427+0000 7f85cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:53.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:53.427+0000 7f85cf7fe640 1 -- 172.21.15.37:0/1604420711 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b0006240 msgr2=0x7f85b0008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:33:53.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:53.427+0000 7f85cf7fe640 1 -- 172.21.15.37:0/1604420711 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b0006240 msgr2=0x7f85b0008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:33:53.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:53.431+0000 7f85cf7fe640 1 --2- 172.21.15.37:0/1604420711 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b0006240 0x7f85b0008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:33:53.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:53.431+0000 7f85cf7fe640 1 --2- 172.21.15.37:0/1604420711 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b0006240 0x7f85b0008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:33:53.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:53.631+0000 7f85cffff640 1 -- 172.21.15.37:0/1604420711 >> v2:172.21.15.45:3300/0 conn(0x7f85b0003750 msgr2=0x7f85b0005bb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:33:53.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:53.631+0000 7f85cffff640 1 --2- 172.21.15.37:0/1604420711 >> v2:172.21.15.45:3300/0 conn(0x7f85b0003750 0x7f85b0005bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:33:53.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:53.635+0000 7f85cf7fe640 1 --2- 172.21.15.37:0/1604420711 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b0006240 0x7f85b0008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:53.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:53.635+0000 7f85cf7fe640 10 monclient(hunting): get_auth_request con 0x7f85b0006240 auth_method 0 2024-04-26T00:33:53.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:53.635+0000 7f85cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:53.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:53.635+0000 7f85cf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:53.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:53.635+0000 7f85cf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:33:53.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:53.635+0000 7f85cf7fe640 1 -- 172.21.15.37:0/1604420711 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b0006240 msgr2=0x7f85b0008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:33:53.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:53.635+0000 7f85cf7fe640 1 -- 172.21.15.37:0/1604420711 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b0006240 msgr2=0x7f85b0008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:33:53.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:53.635+0000 7f85cf7fe640 1 --2- 172.21.15.37:0/1604420711 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b0006240 0x7f85b0008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:33:53.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:53.635+0000 7f85cf7fe640 1 --2- 172.21.15.37:0/1604420711 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b0006240 0x7f85b0008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:33:54.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:54.035+0000 7f85cffff640 1 -- 172.21.15.37:0/1604420711 >> v2:172.21.15.45:3300/0 conn(0x7f85b0003750 msgr2=0x7f85b0005bb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:33:54.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:54.035+0000 7f85cffff640 1 --2- 172.21.15.37:0/1604420711 >> v2:172.21.15.45:3300/0 conn(0x7f85b0003750 0x7f85b0005bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:33:54.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:54.039+0000 7f85cf7fe640 1 --2- 172.21.15.37:0/1604420711 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b0006240 0x7f85b0008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:54.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:54.039+0000 7f85cf7fe640 10 monclient(hunting): get_auth_request con 0x7f85b0006240 auth_method 0 2024-04-26T00:33:54.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:54.039+0000 7f85cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:54.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:54.039+0000 7f85cf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:54.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:54.039+0000 7f85cf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:33:54.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:54.039+0000 7f85cf7fe640 1 -- 172.21.15.37:0/1604420711 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b0006240 msgr2=0x7f85b0008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:33:54.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:54.039+0000 7f85cf7fe640 1 -- 172.21.15.37:0/1604420711 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b0006240 msgr2=0x7f85b0008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:33:54.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:54.039+0000 7f85cf7fe640 1 --2- 172.21.15.37:0/1604420711 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b0006240 0x7f85b0008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:33:54.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:54.039+0000 7f85cf7fe640 1 --2- 172.21.15.37:0/1604420711 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b0006240 0x7f85b0008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:33:54.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:54.839+0000 7f85cffff640 1 -- 172.21.15.37:0/1604420711 >> v2:172.21.15.45:3300/0 conn(0x7f85b0003750 msgr2=0x7f85b0005bb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:33:54.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:54.839+0000 7f85cffff640 1 --2- 172.21.15.37:0/1604420711 >> v2:172.21.15.45:3300/0 conn(0x7f85b0003750 0x7f85b0005bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:33:54.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:54.839+0000 7f85ceffd640 1 -- 172.21.15.37:0/1604420711 DispatchQueue::entry inject delay of 0.002000 2024-04-26T00:33:54.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:54.843+0000 7f85cf7fe640 1 --2- 172.21.15.37:0/1604420711 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b0006240 0x7f85b0008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:54.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:54.843+0000 7f85cf7fe640 10 monclient(hunting): get_auth_request con 0x7f85b0006240 auth_method 0 2024-04-26T00:33:54.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:54.843+0000 7f85cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:54.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:54.843+0000 7f85cf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:54.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:54.843+0000 7f85cf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:33:54.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:54.843+0000 7f85cf7fe640 1 -- 172.21.15.37:0/1604420711 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b0006240 msgr2=0x7f85b0008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:33:54.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:54.843+0000 7f85cf7fe640 1 -- 172.21.15.37:0/1604420711 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b0006240 msgr2=0x7f85b0008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:33:54.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:54.843+0000 7f85cf7fe640 1 --2- 172.21.15.37:0/1604420711 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b0006240 0x7f85b0008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:33:54.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:54.843+0000 7f85cf7fe640 1 --2- 172.21.15.37:0/1604420711 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b0006240 0x7f85b0008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:33:55.980 INFO:tasks.rados.rados.0.smithi045.stdout:2366: done (1 left) 2024-04-26T00:33:55.981 INFO:tasks.rados.rados.0.smithi045.stdout:2367: done (0 left) 2024-04-26T00:33:55.981 INFO:tasks.rados.rados.0.smithi045.stdout:2368: copy_from oid 38 from oid 31 current snap is 267 2024-04-26T00:33:55.981 INFO:tasks.rados.rados.0.smithi045.stdout:2369: copy_from oid 19 from oid 42 current snap is 267 2024-04-26T00:33:55.981 INFO:tasks.rados.rados.0.smithi045.stdout:2370: snap_remove snap 231 2024-04-26T00:33:56.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.427+0000 7f85cdffb640 10 monclient(hunting): tick 2024-04-26T00:33:56.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.427+0000 7f85cdffb640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:33:56.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.427+0000 7f85cdffb640 1 monclient(hunting): continuing hunt 2024-04-26T00:33:56.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.427+0000 7f85cdffb640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:33:56.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.427+0000 7f85cdffb640 1 -- 172.21.15.37:0/1604420711 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b0006240 msgr2=0x7f85b0008700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:33:56.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.427+0000 7f85cdffb640 1 --2- 172.21.15.37:0/1604420711 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b0006240 0x7f85b0008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:56.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.427+0000 7f85cdffb640 1 -- 172.21.15.37:0/1604420711 >> v2:172.21.15.45:3300/0 conn(0x7f85b0003750 msgr2=0x7f85b0005bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:33:56.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.427+0000 7f85cdffb640 1 --2- 172.21.15.37:0/1604420711 >> v2:172.21.15.45:3300/0 conn(0x7f85b0003750 0x7f85b0005bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:56.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.427+0000 7f85d4c2b640 1 -- 172.21.15.37:0/1604420711 reap_dead start 2024-04-26T00:33:56.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.427+0000 7f85cdffb640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-04-26T00:33:56.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.427+0000 7f85cdffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f85b0003ba0 addr v1:172.21.15.37:6789/0 2024-04-26T00:33:56.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.427+0000 7f85cdffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f85b00066f0 addr v1:172.21.15.45:6789/0 2024-04-26T00:33:56.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.427+0000 7f85cdffb640 1 --2- 172.21.15.37:0/1604420711 >> v2:172.21.15.37:3300/0 conn(0x7f85b000c210 0x7f85b000e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:56.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.427+0000 7f85cdffb640 10 monclient(hunting): picked mon.noname-a con 0x7f85b000c210 addr v2:172.21.15.37:3300/0 2024-04-26T00:33:56.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.427+0000 7f85cdffb640 1 -- 172.21.15.37:0/1604420711 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85d009bc50 con 0x7f85b0003ba0 2024-04-26T00:33:56.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.427+0000 7f85cdffb640 1 -- 172.21.15.37:0/1604420711 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85d006cff0 con 0x7f85b00066f0 2024-04-26T00:33:56.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.427+0000 7f85cdffb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:56.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.427+0000 7f85cdffb640 1 -- 172.21.15.37:0/1604420711 --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f85b0008e00 con 0x7f85b000c210 2024-04-26T00:33:56.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.427+0000 7f85cdffb640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:56.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.431+0000 7f85cf7fe640 1 -- 172.21.15.37:0/1604420711 >> v1:172.21.15.45:6789/0 conn(0x7f85b00066f0 legacy=0x7f85b000aaf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:33:56.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.431+0000 7f85d4c2b640 1 --2- 172.21.15.37:0/1604420711 >> v2:172.21.15.37:3300/0 conn(0x7f85b000c210 0x7f85b000e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:56.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.431+0000 7f85d4c2b640 10 monclient(hunting): get_auth_request con 0x7f85b000c210 auth_method 0 2024-04-26T00:33:56.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.431+0000 7f85d4c2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:56.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.431+0000 7f85d4c2b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:56.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.431+0000 7f85d4c2b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:56.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.431+0000 7f85d4c2b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:56.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.431+0000 7f85d4c2b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:56.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.431+0000 7f85d4c2b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:56.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.431+0000 7f85d4c2b640 10 monclient(hunting): handle_auth_done global_id 144357 payload 274 2024-04-26T00:33:56.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.431+0000 7f85d4c2b640 1 -- 172.21.15.37:0/1604420711 >> v1:172.21.15.45:6789/0 conn(0x7f85b00066f0 legacy=0x7f85b000aaf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:33:56.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.431+0000 7f85d4c2b640 1 -- 172.21.15.37:0/1604420711 >> v1:172.21.15.37:6789/0 conn(0x7f85b0003ba0 legacy=0x7f85b00093d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:33:56.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.431+0000 7f85d4c2b640 10 monclient: _finish_hunting 0 2024-04-26T00:33:56.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.431+0000 7f85d4c2b640 1 monclient: found mon.noname-a 2024-04-26T00:33:56.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.431+0000 7f85d4c2b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:56.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.431+0000 7f85d4c2b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:33:56.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.431+0000 7f85d4c2b640 1 -- 172.21.15.37:0/1604420711 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85b0010df0 con 0x7f85b000c210 2024-04-26T00:33:56.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.439+0000 7f85d4c2b640 1 -- 172.21.15.37:0/1604420711 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85b000c210 msgr2=0x7f85b000e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:56.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.439+0000 7f85d4c2b640 1 --2- 172.21.15.37:0/1604420711 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85b000c210 0x7f85b000e640 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f85d009eb50 tx=0x7f85c801c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc459d3f2b9a133b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:56.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.439+0000 7f85d4c2b640 1 --2- 172.21.15.37:0/1604420711 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85b000c210 0x7f85b000e640 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f85d009eb50 tx=0x7f85c801c890 comp rx=0 tx=0).queue_received will delay after 1.71409e+09 on 0x7f85c8007350 mon_map magic: 0 2024-04-26T00:33:56.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.439+0000 7f85ceffd640 1 -- 172.21.15.37:0/1604420711 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c8007350 con 0x7f85b000c210 2024-04-26T00:33:56.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.439+0000 7f85ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:56.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.439+0000 7f85ceffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:33:56.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.439+0000 7f85ceffd640 10 monclient: dump: 2024-04-26T00:33:56.433 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:56.433 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:56.433 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:56.433 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:56.433 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:56.433 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:56.433 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:56.433 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:56.433 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:56.434 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:56.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.439+0000 7f85ceffd640 10 monclient: _finish_auth 0 2024-04-26T00:33:56.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.439+0000 7f85ceffd640 10 monclient: _check_auth_tickets 2024-04-26T00:33:56.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.439+0000 7f85ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:56.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.439+0000 7f85ceffd640 1 -- 172.21.15.37:0/1604420711 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85c80ae070 con 0x7f85b000c210 2024-04-26T00:33:56.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.439+0000 7f85d668e640 5 monclient: authenticate success, global_id 144357 2024-04-26T00:33:56.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.439+0000 7f85d668e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:33:56.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.439+0000 7f85ceffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:33:56.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.439+0000 7f85d668e640 10 monclient: get_monmap_and_config success 2024-04-26T00:33:56.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.439+0000 7f85d668e640 4 set_mon_vals no callback set 2024-04-26T00:33:56.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.439+0000 7f85d668e640 10 monclient: shutdown 2024-04-26T00:33:56.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.439+0000 7f85d668e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:33:56.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.439+0000 7f85d668e640 1 -- 172.21.15.37:0/1604420711 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85b000c210 msgr2=0x7f85b000e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:56.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.439+0000 7f85d668e640 1 --2- 172.21.15.37:0/1604420711 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85b000c210 0x7f85b000e640 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f85d009eb50 tx=0x7f85c801c890 comp rx=0 tx=0).stop 2024-04-26T00:33:56.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.439+0000 7f85ceffd640 1 -- 172.21.15.37:0/1604420711 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 0 0 0) 0x7f85c80076f0 con 0x7f85b000c210 2024-04-26T00:33:56.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.439+0000 7f85ceffd640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-04-26T00:33:56.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.439+0000 7f85d668e640 1 -- 172.21.15.37:0/1604420711 shutdown_connections 2024-04-26T00:33:56.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.439+0000 7f85d668e640 1 --2- 172.21.15.37:0/1604420711 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85b000c210 0x7f85b000e640 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:56.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.439+0000 7f85d668e640 1 -- 172.21.15.37:0/1604420711 >> 172.21.15.37:0/1604420711 conn(0x7f85d00a1ad0 msgr2=0x7f85d00a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:33:56.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.439+0000 7f85d668e640 1 -- 172.21.15.37:0/1604420711 shutdown_connections 2024-04-26T00:33:56.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.439+0000 7f85d668e640 1 -- 172.21.15.37:0/1604420711 wait complete. 2024-04-26T00:33:56.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d668e640 10 monclient: build_initial_monmap 2024-04-26T00:33:56.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d668e640 10 monclient: monmap: 2024-04-26T00:33:56.437 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:33:56.437 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:33:56.437 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:33:56.437 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:33:56.437 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:33:56.437 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:33:56.437 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:33:56.437 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:33:56.437 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:33:56.438 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:56.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d668e640 1 Processor -- start 2024-04-26T00:33:56.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d668e640 1 -- start start 2024-04-26T00:33:56.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d668e640 10 monclient: init 2024-04-26T00:33:56.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d668e640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:56.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d668e640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:33:56.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d668e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85b0003ba0 0x7f85d0122360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:56.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d668e640 10 monclient(hunting): picked mon.noname-b con 0x7f85b0003ba0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:33:56.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d668e640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85b00066f0 0x7f85d01cf450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:56.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d668e640 10 monclient(hunting): picked mon.noname-a con 0x7f85b00066f0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:33:56.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d668e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b000c210 0x7f85d01cf990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:56.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d668e640 10 monclient(hunting): picked mon.noname-c con 0x7f85b000c210 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:33:56.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d668e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:56.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d668e640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f85d00a00a0 con 0x7f85b00066f0 2024-04-26T00:33:56.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d668e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:56.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d668e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f85d009f4c0 con 0x7f85b0003ba0 2024-04-26T00:33:56.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d668e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:56.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d668e640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f85d01228a0 con 0x7f85b000c210 2024-04-26T00:33:56.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d668e640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:56.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d668e640 10 monclient(hunting): authenticate will time out at 1817.750745s 2024-04-26T00:33:56.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85cffff640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85b0003ba0 msgr2=0x7f85d0122360 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:33:56.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85cffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85b0003ba0 0x7f85d0122360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:33:56.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d4c2b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b000c210 0x7f85d01cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:56.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d4c2b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b000c210 0x7f85d01cf990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:44928/0 (socket says 172.21.15.37:44928) 2024-04-26T00:33:56.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d4c2b640 1 -- 172.21.15.37:0/1584643987 learned_addr learned my addr 172.21.15.37:0/1584643987 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:33:56.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d4c2b640 10 monclient(hunting): get_auth_request con 0x7f85b000c210 auth_method 0 2024-04-26T00:33:56.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d4c2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:56.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d4c2b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:56.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d4c2b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:56.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85cf7fe640 1 --2- 172.21.15.37:0/1584643987 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85b00066f0 0x7f85d01cf450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:56.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85cf7fe640 10 monclient(hunting): get_auth_request con 0x7f85b00066f0 auth_method 0 2024-04-26T00:33:56.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:56.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85cf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:56.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:56.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d4c2b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:56.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d4c2b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:56.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d4c2b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:56.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:56.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:56.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:56.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d4c2b640 10 monclient(hunting): handle_auth_done global_id 154235 payload 824 2024-04-26T00:33:56.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d4c2b640 1 -- 172.21.15.37:0/1584643987 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85b0003ba0 msgr2=0x7f85d0122360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:33:56.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d4c2b640 1 --2- 172.21.15.37:0/1584643987 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85b0003ba0 0x7f85d0122360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:56.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d4c2b640 1 -- 172.21.15.37:0/1584643987 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85b00066f0 msgr2=0x7f85d01cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:56.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d4c2b640 1 --2- 172.21.15.37:0/1584643987 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85b00066f0 0x7f85d01cf450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:56.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d4c2b640 10 monclient: _finish_hunting 0 2024-04-26T00:33:56.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d4c2b640 1 monclient: found mon.noname-c 2024-04-26T00:33:56.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d4c2b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:56.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d4c2b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:33:56.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d4c2b640 1 -- 172.21.15.37:0/1584643987 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85d00729d0 con 0x7f85b000c210 2024-04-26T00:33:56.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85cf7fe640 1 --2- 172.21.15.37:0/1584643987 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85b00066f0 0x7f85d01cf450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:33:56.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d4c2b640 1 -- 172.21.15.37:0/1584643987 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b000c210 msgr2=0x7f85d01cf990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:33:56.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d4c2b640 1 -- 172.21.15.37:0/1584643987 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b000c210 msgr2=0x7f85d01cf990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:33:56.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d4c2b640 1 --2- 172.21.15.37:0/1584643987 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b000c210 0x7f85d01cf990 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f85c801cd70 tx=0x7f85c801b0d0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:33:56.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.443+0000 7f85d4c2b640 1 --2- 172.21.15.37:0/1584643987 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b000c210 0x7f85d01cf990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:33:56.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85d4c2b640 1 --2- 172.21.15.37:0/1584643987 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b000c210 0x7f85d01cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:56.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85d4c2b640 10 monclient: get_auth_request con 0x7f85b000c210 auth_method 0 2024-04-26T00:33:56.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85d4c2b640 0 --2- 172.21.15.37:0/1584643987 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b000c210 0x7f85d01cf990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-04-26T00:33:56.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85d4c2b640 1 --2- 172.21.15.37:0/1584643987 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b000c210 0x7f85d01cf990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:56.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85ccff9640 10 monclient: ms_handle_reset current mon [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:33:56.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85ccff9640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:56.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85ccff9640 1 -- 172.21.15.37:0/1584643987 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b000c210 msgr2=0x7f85d01cf990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:33:56.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85ccff9640 1 --2- 172.21.15.37:0/1584643987 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b000c210 0x7f85d01cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:56.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85ccff9640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:33:56.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85ccff9640 1 --2- 172.21.15.37:0/1584643987 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b0005690 0x7f85b000e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:56.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85ccff9640 10 monclient(hunting): picked mon.noname-c con 0x7f85b0005690 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:33:56.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85ccff9640 1 --2- 172.21.15.37:0/1584643987 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85b00066f0 0x7f85b0002b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:56.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85ccff9640 10 monclient(hunting): picked mon.noname-a con 0x7f85b00066f0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:33:56.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85ccff9640 1 --2- 172.21.15.37:0/1584643987 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85b0003ba0 0x7f85b000b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:56.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85ccff9640 10 monclient(hunting): picked mon.noname-b con 0x7f85b0003ba0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:33:56.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85ccff9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:56.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85ccff9640 1 -- 172.21.15.37:0/1584643987 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f85b000bb30 con 0x7f85b00066f0 2024-04-26T00:33:56.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85ccff9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:56.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85ccff9640 1 -- 172.21.15.37:0/1584643987 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f85b000bcb0 con 0x7f85b0003ba0 2024-04-26T00:33:56.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85ccff9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:56.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85ccff9640 1 -- 172.21.15.37:0/1584643987 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f85b000beb0 con 0x7f85b0005690 2024-04-26T00:33:56.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85ccff9640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:56.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85cffff640 1 --2- 172.21.15.37:0/1584643987 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b0005690 0x7f85b000e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:56.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85cffff640 10 monclient(hunting): get_auth_request con 0x7f85b0005690 auth_method 0 2024-04-26T00:33:56.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:56.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85cffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:56.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85cffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:33:56.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85d4c2b640 1 -- 172.21.15.37:0/1584643987 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85b0003ba0 msgr2=0x7f85b000b3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:33:56.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85cf7fe640 1 --2- 172.21.15.37:0/1584643987 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85b00066f0 0x7f85b0002b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:56.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85cf7fe640 10 monclient(hunting): get_auth_request con 0x7f85b00066f0 auth_method 0 2024-04-26T00:33:56.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:56.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85cf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:56.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85cf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:33:56.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85d4c2b640 1 --2- 172.21.15.37:0/1584643987 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85b0003ba0 0x7f85b000b3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:33:56.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:56.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:56.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85cffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:33:56.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:56.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:56.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:33:56.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85cffff640 10 monclient(hunting): handle_auth_done global_id 154235 payload 293 2024-04-26T00:33:56.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85cffff640 1 -- 172.21.15.37:0/1584643987 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85b0003ba0 msgr2=0x7f85b000b3c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:33:56.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85cffff640 1 --2- 172.21.15.37:0/1584643987 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85b0003ba0 0x7f85b000b3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:56.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85cffff640 1 -- 172.21.15.37:0/1584643987 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85b00066f0 msgr2=0x7f85b0002b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:56.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85cffff640 1 --2- 172.21.15.37:0/1584643987 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85b00066f0 0x7f85b0002b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:56.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85cffff640 10 monclient: _finish_hunting 0 2024-04-26T00:33:56.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85cffff640 1 monclient: found mon.noname-c 2024-04-26T00:33:56.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:56.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85cffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:33:56.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.647+0000 7f85cffff640 1 -- 172.21.15.37:0/1584643987 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85b0007c50 con 0x7f85b0005690 2024-04-26T00:33:56.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.651+0000 7f85cffff640 1 -- 172.21.15.37:0/1584643987 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b0005690 msgr2=0x7f85b000e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:56.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.651+0000 7f85cffff640 1 --2- 172.21.15.37:0/1584643987 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b0005690 0x7f85b000e600 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f85b8000ee0 tx=0x7f85c8009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:56.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.651+0000 7f85ccff9640 1 -- 172.21.15.37:0/1584643987 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b800f040 con 0x7f85b0005690 2024-04-26T00:33:56.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.651+0000 7f85ccff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:56.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.651+0000 7f85ccff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:33:56.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.651+0000 7f85ccff9640 10 monclient: dump: 2024-04-26T00:33:56.658 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:56.658 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:56.659 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:56.659 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:56.659 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:56.659 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:56.659 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:56.659 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:56.660 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:56.660 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:56.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.651+0000 7f85ccff9640 10 monclient: _finish_auth 0 2024-04-26T00:33:56.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.651+0000 7f85ccff9640 10 monclient: _check_auth_tickets 2024-04-26T00:33:56.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.651+0000 7f85ccff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:56.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.651+0000 7f85ccff9640 1 -- 172.21.15.37:0/1584643987 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85b8013ac0 con 0x7f85b0005690 2024-04-26T00:33:56.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.651+0000 7f85d668e640 5 monclient: authenticate success, global_id 154235 2024-04-26T00:33:56.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.651+0000 7f85d668e640 10 monclient: _renew_subs 2024-04-26T00:33:56.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.651+0000 7f85d668e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:33:56.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.651+0000 7f85d668e640 1 -- 172.21.15.37:0/1584643987 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85d0122360 con 0x7f85b0005690 2024-04-26T00:33:56.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.651+0000 7f85ccff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:33:56.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.651+0000 7f85ccff9640 1 -- 172.21.15.37:0/1584643987 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b8016460 con 0x7f85b0005690 2024-04-26T00:33:56.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.651+0000 7f85ccff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:56.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.651+0000 7f85ccff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:33:56.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.651+0000 7f85ccff9640 10 monclient: dump: 2024-04-26T00:33:56.662 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:56.663 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:56.663 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:56.663 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:56.663 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:56.663 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:56.663 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:56.663 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:56.664 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:56.664 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:56.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.651+0000 7f85ce7fc640 4 set_mon_vals no callback set 2024-04-26T00:33:56.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.651+0000 7f85d668e640 10 monclient: _renew_subs 2024-04-26T00:33:56.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.651+0000 7f85d668e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:33:56.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.651+0000 7f85d668e640 1 -- 172.21.15.37:0/1584643987 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f85d01cf450 con 0x7f85b0005690 2024-04-26T00:33:56.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.651+0000 7f85d668e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:56.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.651+0000 7f85d668e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:56.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.651+0000 7f85d668e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:33:56.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.651+0000 7f85d668e640 1 -- 172.21.15.37:0/1584643987 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85d0066f40 con 0x7f85b0005690 2024-04-26T00:33:56.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.651+0000 7f85ccff9640 1 -- 172.21.15.37:0/1584643987 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f85b8016600 con 0x7f85b0005690 2024-04-26T00:33:56.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.651+0000 7f85ccff9640 1 --2- 172.21.15.37:0/1584643987 >> v2:172.21.15.45:6812/2379918623 conn(0x7f85b000c210 0x7f85b0042530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:56.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.651+0000 7f85ccff9640 1 -- 172.21.15.37:0/1584643987 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(536..536 src has 370..536) ==== 3931+0+0 (secure 0 0 0) 0x7f85b8047970 con 0x7f85b0005690 2024-04-26T00:33:56.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.651+0000 7f85cf7fe640 1 --2- 172.21.15.37:0/1584643987 >> v2:172.21.15.45:6812/2379918623 conn(0x7f85b000c210 0x7f85b0042530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:56.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.651+0000 7f85cf7fe640 10 monclient: get_auth_request con 0x7f85b000c210 auth_method 0 2024-04-26T00:33:56.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.655+0000 7f85cf7fe640 1 --2- 172.21.15.37:0/1584643987 >> v2:172.21.15.45:6812/2379918623 conn(0x7f85b000c210 0x7f85b0042530 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f85c0005010 tx=0x7f85c000d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:56.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.659+0000 7f85ccff9640 1 -- 172.21.15.37:0/1584643987 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f85b801a020 con 0x7f85b0005690 2024-04-26T00:33:56.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.659+0000 7f85ccff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:56.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.659+0000 7f85ccff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:33:56.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.903+0000 7f85d668e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:33:56.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.903+0000 7f85d668e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:33:56.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.903+0000 7f85d668e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:33:56.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.903+0000 7f85d668e640 1 -- 172.21.15.37:0/1584643987 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f85d0121ea0 con 0x7f85b0005690 2024-04-26T00:33:56.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.907+0000 7f85ccff9640 1 -- 172.21.15.37:0/1584643987 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f85b801c4c0 con 0x7f85b0005690 2024-04-26T00:33:56.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.907+0000 7f85ccff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:33:56.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.907+0000 7f85ccff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:33:56.901 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":106,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:33:56.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.907+0000 7f85d668e640 1 -- 172.21.15.37:0/1584643987 >> v2:172.21.15.45:6812/2379918623 conn(0x7f85b000c210 msgr2=0x7f85b0042530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:56.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.907+0000 7f85d668e640 1 --2- 172.21.15.37:0/1584643987 >> v2:172.21.15.45:6812/2379918623 conn(0x7f85b000c210 0x7f85b0042530 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f85c0005010 tx=0x7f85c000d040 comp rx=0 tx=0).stop 2024-04-26T00:33:56.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.907+0000 7f85d668e640 10 monclient: shutdown 2024-04-26T00:33:56.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.907+0000 7f85d668e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:33:56.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.907+0000 7f85d668e640 1 -- 172.21.15.37:0/1584643987 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b0005690 msgr2=0x7f85b000e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:56.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.907+0000 7f85d668e640 1 --2- 172.21.15.37:0/1584643987 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b0005690 0x7f85b000e600 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f85b8000ee0 tx=0x7f85c8009040 comp rx=0 tx=0).stop 2024-04-26T00:33:56.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.907+0000 7f85d668e640 1 -- 172.21.15.37:0/1584643987 shutdown_connections 2024-04-26T00:33:56.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.907+0000 7f85d668e640 1 --2- 172.21.15.37:0/1584643987 >> v2:172.21.15.45:6812/2379918623 conn(0x7f85b000c210 0x7f85b0042530 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:56.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.907+0000 7f85d668e640 1 --2- 172.21.15.37:0/1584643987 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f85b0003ba0 0x7f85b000b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:56.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.907+0000 7f85d668e640 1 --2- 172.21.15.37:0/1584643987 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85b00066f0 0x7f85b0002b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:56.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.907+0000 7f85d668e640 1 --2- 172.21.15.37:0/1584643987 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85b0005690 0x7f85b000e600 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:56.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.907+0000 7f85d668e640 1 -- 172.21.15.37:0/1584643987 >> 172.21.15.37:0/1584643987 conn(0x7f85d00a1ad0 msgr2=0x7f85d0132140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:33:56.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.907+0000 7f85d668e640 1 -- 172.21.15.37:0/1584643987 shutdown_connections 2024-04-26T00:33:56.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:56.907+0000 7f85d668e640 1 -- 172.21.15.37:0/1584643987 wait complete. 2024-04-26T00:33:56.919 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-26T00:33:56.919 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-04-26T00:33:56.919 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:33:56.991 INFO:tasks.rados.rados.0.smithi045.stdout:2370: done (2 left) 2024-04-26T00:33:56.991 INFO:tasks.rados.rados.0.smithi045.stdout:2371: snap_remove snap 140 2024-04-26T00:33:57.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.059+0000 7f25e5a98640 10 monclient: get_monmap_and_config 2024-04-26T00:33:57.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.059+0000 7f25e5a98640 10 monclient: build_initial_monmap 2024-04-26T00:33:57.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.059+0000 7f25e5a98640 10 monclient: monmap: 2024-04-26T00:33:57.060 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:33:57.060 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:57.060 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:33:57.062887+0000 2024-04-26T00:33:57.060 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:33:57.062887+0000 2024-04-26T00:33:57.060 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:33:57.060 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:57.060 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:33:57.060 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:33:57.060 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:33:57.061 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:33:57.061 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:33:57.061 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:57.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.063+0000 7f25e5a98640 1 Processor -- start 2024-04-26T00:33:57.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.063+0000 7f25e5a98640 1 -- start start 2024-04-26T00:33:57.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.063+0000 7f25e5a98640 10 monclient: init 2024-04-26T00:33:57.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.063+0000 7f25e5a98640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:57.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.063+0000 7f25e5a98640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-04-26T00:33:57.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.063+0000 7f25e5a98640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f25e006b1d0 0x7f25e006b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:57.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.063+0000 7f25e5a98640 10 monclient(hunting): picked mon.noname-a con 0x7f25e006b1d0 addr v2:172.21.15.37:3300/0 2024-04-26T00:33:57.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.063+0000 7f25e5a98640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f25e009ef60 addr v1:172.21.15.45:6789/0 2024-04-26T00:33:57.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.063+0000 7f25e5a98640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f25e006baf0 0x7f25e0137ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:57.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.063+0000 7f25e5a98640 10 monclient(hunting): picked mon.noname-c con 0x7f25e006baf0 addr v2:172.21.15.45:3300/0 2024-04-26T00:33:57.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.063+0000 7f25e5a98640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25e00a28b0 con 0x7f25e009ef60 2024-04-26T00:33:57.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.063+0000 7f25e5a98640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:57.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.063+0000 7f25e5a98640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f25e009e280 con 0x7f25e006b1d0 2024-04-26T00:33:57.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.063+0000 7f25e5a98640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:57.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.063+0000 7f25e5a98640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f25e009dcc0 con 0x7f25e006baf0 2024-04-26T00:33:57.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.063+0000 7f25e5a98640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:57.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.063+0000 7f25e5a98640 10 monclient(hunting): authenticate will time out at 1818.368880s 2024-04-26T00:33:57.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.063+0000 7f25d6ffd640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f25e009ef60 legacy=0x7f25e01373b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:33:57.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.063+0000 7f25dffff640 1 -- >> v2:172.21.15.45:3300/0 conn(0x7f25e006baf0 msgr2=0x7f25e0137ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:33:57.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.063+0000 7f25df7fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f25e006b1d0 0x7f25e006b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:57.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.063+0000 7f25dffff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f25e006baf0 0x7f25e0137ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:33:57.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.063+0000 7f25df7fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f25e006b1d0 0x7f25e006b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:47062/0 (socket says 172.21.15.37:47062) 2024-04-26T00:33:57.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.063+0000 7f25df7fe640 1 -- 172.21.15.37:0/2162493693 learned_addr learned my addr 172.21.15.37:0/2162493693 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:33:57.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.063+0000 7f25df7fe640 10 monclient(hunting): get_auth_request con 0x7f25e006b1d0 auth_method 0 2024-04-26T00:33:57.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.063+0000 7f25df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:57.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.063+0000 7f25df7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:57.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.063+0000 7f25df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:57.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.063+0000 7f25df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:57.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.063+0000 7f25df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:57.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.063+0000 7f25df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:57.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.063+0000 7f25df7fe640 10 monclient(hunting): handle_auth_done global_id 144363 payload 274 2024-04-26T00:33:57.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.063+0000 7f25df7fe640 1 -- 172.21.15.37:0/2162493693 >> v2:172.21.15.45:3300/0 conn(0x7f25e006baf0 msgr2=0x7f25e0137ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:33:57.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.063+0000 7f25df7fe640 1 --2- 172.21.15.37:0/2162493693 >> v2:172.21.15.45:3300/0 conn(0x7f25e006baf0 0x7f25e0137ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:57.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.063+0000 7f25df7fe640 1 -- 172.21.15.37:0/2162493693 >> v1:172.21.15.45:6789/0 conn(0x7f25e009ef60 legacy=0x7f25e01373b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:33:57.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.063+0000 7f25df7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:33:57.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.063+0000 7f25df7fe640 1 monclient: found mon.noname-a 2024-04-26T00:33:57.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.063+0000 7f25df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:57.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.063+0000 7f25df7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:33:57.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.063+0000 7f25df7fe640 1 -- 172.21.15.37:0/2162493693 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25e009dfa0 con 0x7f25e006b1d0 2024-04-26T00:33:57.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.067+0000 7f25df7fe640 1 -- 172.21.15.37:0/2162493693 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25e006b1d0 msgr2=0x7f25e006b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:57.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.067+0000 7f25df7fe640 1 --2- 172.21.15.37:0/2162493693 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25e006b1d0 0x7f25e006b5b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f25e00a3fe0 tx=0x7f25d001c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2467e20c2eb4241c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:57.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.067+0000 7f25deffd640 1 -- 172.21.15.37:0/2162493693 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d00b0070 con 0x7f25e006b1d0 2024-04-26T00:33:57.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.067+0000 7f25deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:57.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.067+0000 7f25deffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:33:57.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.067+0000 7f25deffd640 10 monclient: dump: 2024-04-26T00:33:57.071 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:57.071 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:57.071 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:57.071 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:57.071 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:57.071 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:57.071 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:57.071 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:57.071 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:57.071 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:57.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.067+0000 7f25deffd640 10 monclient: _finish_auth 0 2024-04-26T00:33:57.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.067+0000 7f25deffd640 10 monclient: _check_auth_tickets 2024-04-26T00:33:57.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.067+0000 7f25deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:57.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.067+0000 7f25deffd640 1 -- 172.21.15.37:0/2162493693 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25d00ab600 con 0x7f25e006b1d0 2024-04-26T00:33:57.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.067+0000 7f25e5a98640 5 monclient: authenticate success, global_id 144363 2024-04-26T00:33:57.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.067+0000 7f25e5a98640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:33:57.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.067+0000 7f25deffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:33:57.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.067+0000 7f25e5a98640 10 monclient: get_monmap_and_config success 2024-04-26T00:33:57.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.067+0000 7f25deffd640 1 -- 172.21.15.37:0/2162493693 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d00ab9d0 con 0x7f25e006b1d0 2024-04-26T00:33:57.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.067+0000 7f25e5a98640 4 set_mon_vals no callback set 2024-04-26T00:33:57.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.067+0000 7f25deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:57.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.067+0000 7f25deffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:33:57.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.067+0000 7f25deffd640 10 monclient: dump: 2024-04-26T00:33:57.072 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:57.072 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:57.072 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:57.072 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:57.072 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:57.073 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:57.073 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:57.073 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:57.073 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:57.073 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:57.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.067+0000 7f25e5a98640 10 monclient: shutdown 2024-04-26T00:33:57.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.067+0000 7f25e5a98640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:33:57.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.067+0000 7f25e5a98640 1 -- 172.21.15.37:0/2162493693 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25e006b1d0 msgr2=0x7f25e006b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:57.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.067+0000 7f25e5a98640 1 --2- 172.21.15.37:0/2162493693 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25e006b1d0 0x7f25e006b5b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f25e00a3fe0 tx=0x7f25d001c3c0 comp rx=0 tx=0).stop 2024-04-26T00:33:57.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.067+0000 7f25e5a98640 1 -- 172.21.15.37:0/2162493693 shutdown_connections 2024-04-26T00:33:57.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.067+0000 7f25e5a98640 1 --2- 172.21.15.37:0/2162493693 >> v2:172.21.15.45:3300/0 conn(0x7f25e006baf0 0x7f25e0137ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:57.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.067+0000 7f25e5a98640 1 --2- 172.21.15.37:0/2162493693 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25e006b1d0 0x7f25e006b5b0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:57.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.067+0000 7f25e5a98640 1 -- 172.21.15.37:0/2162493693 >> 172.21.15.37:0/2162493693 conn(0x7f25e00a2c50 msgr2=0x7f25e00a3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:33:57.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.067+0000 7f25e5a98640 1 -- 172.21.15.37:0/2162493693 shutdown_connections 2024-04-26T00:33:57.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.067+0000 7f25e5a98640 1 -- 172.21.15.37:0/2162493693 wait complete. 2024-04-26T00:33:57.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25e5a98640 10 monclient: build_initial_monmap 2024-04-26T00:33:57.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25e5a98640 10 monclient: monmap: 2024-04-26T00:33:57.074 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:33:57.074 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:33:57.074 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:33:57.074 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:33:57.074 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:33:57.074 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:33:57.074 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:33:57.074 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:33:57.075 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:33:57.075 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:57.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25e5a98640 1 Processor -- start 2024-04-26T00:33:57.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25e5a98640 1 -- start start 2024-04-26T00:33:57.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25e5a98640 10 monclient: init 2024-04-26T00:33:57.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25e5a98640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:57.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25e5a98640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:33:57.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25e5a98640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f25e006b1d0 0x7f25e0136680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:57.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25e5a98640 10 monclient(hunting): picked mon.noname-c con 0x7f25e006b1d0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:33:57.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25e5a98640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25e006baf0 0x7f25e0136bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:57.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25e5a98640 10 monclient(hunting): picked mon.noname-a con 0x7f25e006baf0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:33:57.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25e5a98640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25e009ef60 0x7f25e01cf1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:57.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25e5a98640 10 monclient(hunting): picked mon.noname-b con 0x7f25e009ef60 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:33:57.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25e5a98640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:57.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25e5a98640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f25e0138180 con 0x7f25e006baf0 2024-04-26T00:33:57.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25e5a98640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:57.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25e5a98640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f25e0138000 con 0x7f25e009ef60 2024-04-26T00:33:57.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25e5a98640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:57.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25e5a98640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f25e0141ea0 con 0x7f25e006b1d0 2024-04-26T00:33:57.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25e5a98640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:57.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25e5a98640 10 monclient(hunting): authenticate will time out at 1818.377067s 2024-04-26T00:33:57.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25df7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f25e006b1d0 0x7f25e0136680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:57.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25dffff640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25e009ef60 msgr2=0x7f25e01cf1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:33:57.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25dffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25e009ef60 0x7f25e01cf1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:33:57.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25d6ffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25e006baf0 0x7f25e0136bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:57.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25df7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f25e006b1d0 0x7f25e0136680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:44950/0 (socket says 172.21.15.37:44950) 2024-04-26T00:33:57.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25df7fe640 1 -- 172.21.15.37:0/285146382 learned_addr learned my addr 172.21.15.37:0/285146382 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:33:57.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25df7fe640 10 monclient(hunting): get_auth_request con 0x7f25e006b1d0 auth_method 0 2024-04-26T00:33:57.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:57.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25df7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:57.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:57.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25d6ffd640 10 monclient(hunting): get_auth_request con 0x7f25e006baf0 auth_method 0 2024-04-26T00:33:57.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:57.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25d6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:57.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:57.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:57.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:57.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:57.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:57.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:57.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:57.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25d6ffd640 10 monclient(hunting): handle_auth_done global_id 144366 payload 824 2024-04-26T00:33:57.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25d6ffd640 1 -- 172.21.15.37:0/285146382 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f25e006b1d0 msgr2=0x7f25e0136680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:57.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25d6ffd640 1 --2- 172.21.15.37:0/285146382 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f25e006b1d0 0x7f25e0136680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:57.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25d6ffd640 1 -- 172.21.15.37:0/285146382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25e009ef60 msgr2=0x7f25e01cf1c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:33:57.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25d6ffd640 1 --2- 172.21.15.37:0/285146382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25e009ef60 0x7f25e01cf1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:57.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25d6ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:33:57.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25d6ffd640 1 monclient: found mon.noname-a 2024-04-26T00:33:57.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:57.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25d6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:33:57.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25d6ffd640 1 -- 172.21.15.37:0/285146382 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25e00a4a90 con 0x7f25e006baf0 2024-04-26T00:33:57.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25df7fe640 1 --2- 172.21.15.37:0/285146382 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f25e006b1d0 0x7f25e0136680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:33:57.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25d6ffd640 1 -- 172.21.15.37:0/285146382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25e006baf0 msgr2=0x7f25e0136bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:57.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.071+0000 7f25d6ffd640 1 --2- 172.21.15.37:0/285146382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25e006baf0 0x7f25e0136bc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f25c800b450 tx=0x7f25c800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:57.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25dcff9640 1 -- 172.21.15.37:0/285146382 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c8005070 con 0x7f25e006baf0 2024-04-26T00:33:57.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:57.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25dcff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:33:57.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25dcff9640 10 monclient: dump: 2024-04-26T00:33:57.079 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:57.079 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:57.079 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:57.079 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:57.079 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:57.079 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:57.080 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:57.080 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:57.080 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:57.080 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:57.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25dcff9640 10 monclient: _finish_auth 0 2024-04-26T00:33:57.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25dcff9640 10 monclient: _check_auth_tickets 2024-04-26T00:33:57.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25dcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:57.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25dcff9640 1 -- 172.21.15.37:0/285146382 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25c8005230 con 0x7f25e006baf0 2024-04-26T00:33:57.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25dcff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:33:57.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25dcff9640 1 -- 172.21.15.37:0/285146382 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c8005620 con 0x7f25e006baf0 2024-04-26T00:33:57.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:57.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25dcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:33:57.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25dcff9640 10 monclient: dump: 2024-04-26T00:33:57.080 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:57.081 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:57.081 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:57.081 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:57.081 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:57.081 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:57.081 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:57.081 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:57.081 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:57.081 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:57.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25e5a98640 5 monclient: authenticate success, global_id 144366 2024-04-26T00:33:57.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25e5a98640 10 monclient: _renew_subs 2024-04-26T00:33:57.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25e5a98640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:33:57.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25e5a98640 1 -- 172.21.15.37:0/285146382 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25e013c230 con 0x7f25e006baf0 2024-04-26T00:33:57.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25de7fc640 4 set_mon_vals no callback set 2024-04-26T00:33:57.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25e5a98640 10 monclient: _renew_subs 2024-04-26T00:33:57.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25e5a98640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:33:57.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25e5a98640 1 -- 172.21.15.37:0/285146382 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25e0137100 con 0x7f25e006baf0 2024-04-26T00:33:57.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25e5a98640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:57.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25e5a98640 1 --2- 172.21.15.37:0/285146382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25e01cf700 0x7f25e01cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:57.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25e5a98640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:57.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25e5a98640 1 -- 172.21.15.37:0/285146382 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f25e01d0020 con 0x7f25e01cf700 2024-04-26T00:33:57.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25df7fe640 1 --2- 172.21.15.37:0/285146382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25e01cf700 0x7f25e01cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:57.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25df7fe640 10 monclient: get_auth_request con 0x7f25e01cf700 auth_method 0 2024-04-26T00:33:57.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:57.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25df7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:57.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:57.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:57.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:57.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:57.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25dcff9640 1 -- 172.21.15.37:0/285146382 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f25c801a400 con 0x7f25e006baf0 2024-04-26T00:33:57.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25df7fe640 10 monclient(hunting): handle_auth_done global_id 144369 payload 824 2024-04-26T00:33:57.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25df7fe640 1 -- 172.21.15.37:0/285146382 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f25e013e6b0 con 0x7f25e01cf700 2024-04-26T00:33:57.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25dcff9640 1 --2- 172.21.15.37:0/285146382 >> v2:172.21.15.45:6812/2379918623 conn(0x7f25c0035e80 0x7f25c0038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:57.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25dcff9640 1 -- 172.21.15.37:0/285146382 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(537..537 src has 370..537) ==== 3975+0+0 (secure 0 0 0) 0x7f25c80485c0 con 0x7f25e006baf0 2024-04-26T00:33:57.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25dffff640 1 --2- 172.21.15.37:0/285146382 >> v2:172.21.15.45:6812/2379918623 conn(0x7f25c0035e80 0x7f25c0038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:57.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25dffff640 10 monclient: get_auth_request con 0x7f25c0035e80 auth_method 0 2024-04-26T00:33:57.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25df7fe640 1 -- 172.21.15.37:0/285146382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25e01cf700 msgr2=0x7f25e01cfae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:57.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25df7fe640 1 --2- 172.21.15.37:0/285146382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25e01cf700 0x7f25e01cfae0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f25d001aa70 tx=0x7f25d0019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:57.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25dcff9640 1 -- 172.21.15.37:0/285146382 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d00bc070 con 0x7f25e01cf700 2024-04-26T00:33:57.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:57.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25dcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:33:57.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.075+0000 7f25dcff9640 10 monclient: dump: 2024-04-26T00:33:57.086 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:57.086 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:57.086 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:57.086 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:57.086 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:57.086 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:57.086 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:57.086 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:57.087 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:57.087 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:57.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.079+0000 7f25dcff9640 1 -- 172.21.15.37:0/285146382 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f25d00b5640 con 0x7f25e01cf700 2024-04-26T00:33:57.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.079+0000 7f25dcff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:57.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.079+0000 7f25dcff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:33:57.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.079+0000 7f25dcff9640 1 -- 172.21.15.37:0/285146382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25e01cf700 msgr2=0x7f25e01cfae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:57.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.079+0000 7f25dcff9640 1 --2- 172.21.15.37:0/285146382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25e01cf700 0x7f25e01cfae0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f25d001aa70 tx=0x7f25d0019160 comp rx=0 tx=0).stop 2024-04-26T00:33:57.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.079+0000 7f25dcff9640 1 -- 172.21.15.37:0/285146382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25e01cf700 msgr2=0x7f25e01cfae0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:33:57.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.079+0000 7f25dcff9640 1 --2- 172.21.15.37:0/285146382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25e01cf700 0x7f25e01cfae0 secure :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f25d001aa70 tx=0x7f25d0019160 comp rx=0 tx=0).stop 2024-04-26T00:33:57.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.079+0000 7f25dffff640 1 --2- 172.21.15.37:0/285146382 >> v2:172.21.15.45:6812/2379918623 conn(0x7f25c0035e80 0x7f25c0038340 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f25e00a2880 tx=0x7f25d8009270 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:57.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.083+0000 7f25e5a98640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:33:57.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.083+0000 7f25e5a98640 1 --2- 172.21.15.37:0/285146382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25e0071430 0x7f25e0071840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:57.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.083+0000 7f25e5a98640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:57.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.083+0000 7f25e5a98640 1 -- 172.21.15.37:0/285146382 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f25e00a2560 con 0x7f25e0071430 2024-04-26T00:33:57.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.083+0000 7f25df7fe640 1 --2- 172.21.15.37:0/285146382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25e0071430 0x7f25e0071840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:57.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.083+0000 7f25df7fe640 10 monclient: get_auth_request con 0x7f25e0071430 auth_method 0 2024-04-26T00:33:57.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.083+0000 7f25df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:57.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.083+0000 7f25df7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:57.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.083+0000 7f25df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:57.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.083+0000 7f25df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:57.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.083+0000 7f25df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:57.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.083+0000 7f25df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:57.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.083+0000 7f25df7fe640 10 monclient(hunting): handle_auth_done global_id 144372 payload 824 2024-04-26T00:33:57.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.083+0000 7f25df7fe640 1 -- 172.21.15.37:0/285146382 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f25e004f510 con 0x7f25e0071430 2024-04-26T00:33:57.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.087+0000 7f25df7fe640 1 -- 172.21.15.37:0/285146382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25e0071430 msgr2=0x7f25e0071840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:57.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.087+0000 7f25df7fe640 1 --2- 172.21.15.37:0/285146382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25e0071430 0x7f25e0071840 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f25d0000c10 tx=0x7f25d0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:57.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.087+0000 7f25dcff9640 1 -- 172.21.15.37:0/285146382 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d00cf020 con 0x7f25e0071430 2024-04-26T00:33:57.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.087+0000 7f25dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:57.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.087+0000 7f25dcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:33:57.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.087+0000 7f25dcff9640 10 monclient: dump: 2024-04-26T00:33:57.089 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:57.089 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:57.089 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:57.089 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:57.089 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:57.090 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:57.090 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:57.090 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:57.090 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:57.090 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:57.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.087+0000 7f25dcff9640 1 -- 172.21.15.37:0/285146382 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7f25d00c1d60 con 0x7f25e0071430 2024-04-26T00:33:57.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.087+0000 7f25dcff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:33:57.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.087+0000 7f25dcff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:33:57.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.087+0000 7f25dcff9640 1 -- 172.21.15.37:0/285146382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25e0071430 msgr2=0x7f25e0071840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:57.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.087+0000 7f25dcff9640 1 --2- 172.21.15.37:0/285146382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25e0071430 0x7f25e0071840 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f25d0000c10 tx=0x7f25d0008040 comp rx=0 tx=0).stop 2024-04-26T00:33:57.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.087+0000 7f25dcff9640 1 -- 172.21.15.37:0/285146382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25e0071430 msgr2=0x7f25e0071840 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:33:57.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.087+0000 7f25dcff9640 1 --2- 172.21.15.37:0/285146382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25e0071430 0x7f25e0071840 secure :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f25d0000c10 tx=0x7f25d0008040 comp rx=0 tx=0).stop 2024-04-26T00:33:57.091 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:33:57.091 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:33:57.091 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:33:57.091 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:33:57.091 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 106, 2024-04-26T00:33:57.091 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 123630, 2024-04-26T00:33:57.091 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:33:57.091 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:33:57.091 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:33:57.091 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:57.091 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 2, 2024-04-26T00:33:57.092 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:33:57.092 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:33:57.092 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:33:57.092 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:33:57.092 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:33:57.092 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:33:57.092 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:33:57.092 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:33:57.092 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:33:57.092 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:33:57.092 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:33:57.092 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:33:57.092 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:33:57.092 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:33:57.092 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:57.093 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:33:57.093 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:33:57.093 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:33:57.093 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:33:57.093 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:33:57.093 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:33:57.093 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:33:57.093 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:33:57.093 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:33:57.093 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:33:57.093 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:33:57.093 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:33:57.093 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:33:57.093 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:57.094 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:57.094 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:33:57.094 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:33:57.094 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:33:57.094 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:33:57.094 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:33:57.094 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:33:57.095 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:33:57.095 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:33:57.095 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:33:57.095 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:33:57.095 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:33:57.095 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:33:57.095 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:33:57.095 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:33:57.096 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:33:57.096 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:33:57.096 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:33:57.096 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:33:57.096 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:33:57.096 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:33:57.097 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:33:57.097 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:33:57.097 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:33:57.097 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:33:57.097 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:33:57.097 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:33:57.097 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:33:57.098 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:33:57.098 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:57.098 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:33:57.098 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:57.098 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:33:57.098 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:57.099 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:33:57.099 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:33:57.099 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:33:57.099 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:33:57.099 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:57.099 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:33:57.100 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:33:57.100 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:57.100 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:57.100 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:57.100 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:33:57.100 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:33:57.100 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:57.101 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:57.101 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:57.101 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:57.101 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:33:57.101 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:33:57.101 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:33:57.101 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:33:57.101 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:33:57.102 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:57.102 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:57.102 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:33:57.102 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:33:57.102 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:33:57.102 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:33:57.103 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:57.103 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:33:57.103 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:33:57.103 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:57.103 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:57.103 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:57.104 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:33:57.104 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:33:57.104 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:57.104 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:57.104 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:57.104 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:57.105 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:33:57.105 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:33:57.105 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:33:57.105 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:33:57.105 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:33:57.105 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:57.105 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:57.106 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:33:57.106 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:33:57.106 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:33:57.106 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:33:57.106 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:57.106 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:33:57.106 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:33:57.106 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:57.106 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:57.106 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:57.107 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:33:57.107 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:33:57.107 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:57.107 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:57.107 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:57.107 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:57.107 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:33:57.107 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:33:57.107 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:33:57.107 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:33:57.107 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:33:57.107 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:57.107 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:57.107 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:57.107 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:33:57.108 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:33:57.108 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:57.108 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:33:57.108 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:33:57.108 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:33:57.108 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:57.108 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:57.108 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:33:57.108 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:57.108 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:33:57.108 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:33:57.108 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:33:57.108 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:57.108 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:57.108 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:33:57.109 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:57.109 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:33:57.109 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:33:57.109 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:33:57.109 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:57.109 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:57.109 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:33:57.109 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:57.109 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:33:57.109 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:33:57.109 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:33:57.109 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:57.109 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:57.109 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:57.110 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:33:57.110 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:33:57.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.087+0000 7f25e5a98640 1 -- 172.21.15.37:0/285146382 >> v2:172.21.15.45:6812/2379918623 conn(0x7f25c0035e80 msgr2=0x7f25c0038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:57.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.087+0000 7f25e5a98640 1 --2- 172.21.15.37:0/285146382 >> v2:172.21.15.45:6812/2379918623 conn(0x7f25c0035e80 0x7f25c0038340 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f25e00a2880 tx=0x7f25d8009270 comp rx=0 tx=0).stop 2024-04-26T00:33:57.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.087+0000 7f25e5a98640 10 monclient: shutdown 2024-04-26T00:33:57.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.087+0000 7f25e5a98640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:33:57.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.087+0000 7f25e5a98640 1 -- 172.21.15.37:0/285146382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25e006baf0 msgr2=0x7f25e0136bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:57.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.087+0000 7f25e5a98640 1 --2- 172.21.15.37:0/285146382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f25e006baf0 0x7f25e0136bc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f25c800b450 tx=0x7f25c800b910 comp rx=0 tx=0).stop 2024-04-26T00:33:57.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.087+0000 7f25dffff640 1 -- 172.21.15.37:0/285146382 reap_dead start 2024-04-26T00:33:57.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.087+0000 7f25dffff640 1 -- 172.21.15.37:0/285146382 reap_dead start 2024-04-26T00:33:57.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.087+0000 7f25e5a98640 1 -- 172.21.15.37:0/285146382 shutdown_connections 2024-04-26T00:33:57.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.087+0000 7f25e5a98640 1 -- 172.21.15.37:0/285146382 >> 172.21.15.37:0/285146382 conn(0x7f25e00a2c50 msgr2=0x7f25e009f410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:33:57.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.087+0000 7f25e5a98640 1 -- 172.21.15.37:0/285146382 shutdown_connections 2024-04-26T00:33:57.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.087+0000 7f25e5a98640 1 -- 172.21.15.37:0/285146382 wait complete. 2024-04-26T00:33:57.113 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:33:57.113 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-04-26T00:33:57.113 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:33:57.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5ee04b3640 10 monclient: get_monmap_and_config 2024-04-26T00:33:57.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5ee04b3640 10 monclient: build_initial_monmap 2024-04-26T00:33:57.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5ee04b3640 10 monclient: monmap: 2024-04-26T00:33:57.232 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:33:57.232 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:57.232 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:33:57.240371+0000 2024-04-26T00:33:57.232 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:33:57.240371+0000 2024-04-26T00:33:57.233 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:33:57.233 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:57.233 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:33:57.233 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:33:57.233 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:33:57.233 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:33:57.233 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:33:57.233 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:57.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5ee04b3640 1 Processor -- start 2024-04-26T00:33:57.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5ee04b3640 1 -- start start 2024-04-26T00:33:57.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5ee04b3640 10 monclient: init 2024-04-26T00:33:57.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5ee04b3640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:57.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5ee04b3640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-04-26T00:33:57.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5ee04b3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5ed80a2fe0 addr v1:172.21.15.37:6789/0 2024-04-26T00:33:57.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5ee04b3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ed8072c80 0x7f5ed8136f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:57.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5ee04b3640 10 monclient(hunting): picked mon.noname-b con 0x7f5ed8072c80 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:33:57.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5ee04b3640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f5ed813c050 0x7f5ed813e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:57.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5ee04b3640 10 monclient(hunting): picked mon.noname-a con 0x7f5ed813c050 addr v2:172.21.15.37:3300/0 2024-04-26T00:33:57.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5ee04b3640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ed809f010 con 0x7f5ed80a2fe0 2024-04-26T00:33:57.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5ee04b3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:57.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5ee04b3640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f5ed80a33c0 con 0x7f5ed813c050 2024-04-26T00:33:57.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5ee04b3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:57.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5ee04b3640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5ed806e780 con 0x7f5ed8072c80 2024-04-26T00:33:57.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5ee04b3640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:57.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5ee04b3640 10 monclient(hunting): authenticate will time out at 1818.546942s 2024-04-26T00:33:57.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5edda4e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ed8072c80 0x7f5ed8136f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:57.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5edda4e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ed8072c80 0x7f5ed8136f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:44956/0 (socket says 172.21.15.37:44956) 2024-04-26T00:33:57.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5edda4e640 1 -- 172.21.15.37:0/1927555269 learned_addr learned my addr 172.21.15.37:0/1927555269 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:33:57.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5edda4e640 10 monclient(hunting): get_auth_request con 0x7f5ed8072c80 auth_method 0 2024-04-26T00:33:57.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5ede24f640 1 --1- 172.21.15.37:0/1927555269 >> v1:172.21.15.37:6789/0 conn(0x7f5ed80a2fe0 0x7f5ed8072570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:58970/0 (socket says 172.21.15.37:58970) 2024-04-26T00:33:57.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5edda4e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:57.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5edda4e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:57.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5edda4e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:57.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5edea50640 1 --2- 172.21.15.37:0/1927555269 >> v2:172.21.15.37:3300/0 conn(0x7f5ed813c050 0x7f5ed813e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:57.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5edea50640 10 monclient(hunting): get_auth_request con 0x7f5ed813c050 auth_method 0 2024-04-26T00:33:57.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5edea50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:57.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5edea50640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:57.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5edea50640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:57.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5edda4e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:57.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5edda4e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:57.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5edda4e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:57.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5edea50640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:57.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5edea50640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:57.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5edea50640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:57.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5edea50640 10 monclient(hunting): handle_auth_done global_id 144375 payload 274 2024-04-26T00:33:57.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5edea50640 1 -- 172.21.15.37:0/1927555269 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ed8072c80 msgr2=0x7f5ed8136f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:33:57.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5edea50640 1 --2- 172.21.15.37:0/1927555269 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ed8072c80 0x7f5ed8136f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:57.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5edea50640 1 -- 172.21.15.37:0/1927555269 >> v1:172.21.15.37:6789/0 conn(0x7f5ed80a2fe0 legacy=0x7f5ed8072570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:33:57.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5edea50640 10 monclient: _finish_hunting 0 2024-04-26T00:33:57.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5edea50640 1 monclient: found mon.noname-a 2024-04-26T00:33:57.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5edea50640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:57.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5edea50640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:33:57.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5edea50640 1 -- 172.21.15.37:0/1927555269 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ed8137640 con 0x7f5ed813c050 2024-04-26T00:33:57.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.239+0000 7f5edda4e640 1 --2- 172.21.15.37:0/1927555269 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ed8072c80 0x7f5ed8136f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:33:57.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.243+0000 7f5edea50640 1 -- 172.21.15.37:0/1927555269 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5ed813c050 msgr2=0x7f5ed813e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:57.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.243+0000 7f5edea50640 1 --2- 172.21.15.37:0/1927555269 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5ed813c050 0x7f5ed813e440 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5ed0009dc0 tx=0x7f5ed001c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ac98a2c3ff7c807 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:57.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.243+0000 7f5edd24d640 1 -- 172.21.15.37:0/1927555269 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed00ac070 con 0x7f5ed813c050 2024-04-26T00:33:57.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.243+0000 7f5edd24d640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:57.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.243+0000 7f5edd24d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:33:57.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.243+0000 7f5edd24d640 10 monclient: dump: 2024-04-26T00:33:57.241 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:57.241 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:57.241 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:57.242 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:57.242 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:57.242 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:57.242 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:57.242 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:57.242 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:57.242 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:57.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.243+0000 7f5edd24d640 10 monclient: _finish_auth 0 2024-04-26T00:33:57.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.243+0000 7f5edd24d640 10 monclient: _check_auth_tickets 2024-04-26T00:33:57.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5edd24d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:57.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5edd24d640 1 -- 172.21.15.37:0/1927555269 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ed00a6660 con 0x7f5ed813c050 2024-04-26T00:33:57.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5ee04b3640 5 monclient: authenticate success, global_id 144375 2024-04-26T00:33:57.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5ee04b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:33:57.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5edd24d640 10 monclient: handle_config config(0 keys) 2024-04-26T00:33:57.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5edd24d640 1 -- 172.21.15.37:0/1927555269 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed00a6a30 con 0x7f5ed813c050 2024-04-26T00:33:57.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5ee04b3640 10 monclient: get_monmap_and_config success 2024-04-26T00:33:57.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5ee04b3640 4 set_mon_vals no callback set 2024-04-26T00:33:57.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5edd24d640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:57.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5edd24d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:33:57.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5edd24d640 10 monclient: dump: 2024-04-26T00:33:57.243 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:57.243 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:57.243 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:57.243 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:57.243 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:57.243 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:57.243 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:57.243 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:57.244 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:57.244 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:57.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5ee04b3640 10 monclient: shutdown 2024-04-26T00:33:57.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5ee04b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:33:57.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5ee04b3640 1 -- 172.21.15.37:0/1927555269 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5ed813c050 msgr2=0x7f5ed813e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:57.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5ee04b3640 1 --2- 172.21.15.37:0/1927555269 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5ed813c050 0x7f5ed813e440 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5ed0009dc0 tx=0x7f5ed001c7d0 comp rx=0 tx=0).stop 2024-04-26T00:33:57.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5ee04b3640 1 -- 172.21.15.37:0/1927555269 shutdown_connections 2024-04-26T00:33:57.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5ee04b3640 1 --2- 172.21.15.37:0/1927555269 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5ed813c050 0x7f5ed813e440 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:57.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5ee04b3640 1 --2- 172.21.15.37:0/1927555269 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ed8072c80 0x7f5ed8136f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:57.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5ee04b3640 1 -- 172.21.15.37:0/1927555269 >> 172.21.15.37:0/1927555269 conn(0x7f5ed80a24c0 msgr2=0x7f5ed80a28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:33:57.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5ee04b3640 1 -- 172.21.15.37:0/1927555269 shutdown_connections 2024-04-26T00:33:57.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5ee04b3640 1 -- 172.21.15.37:0/1927555269 wait complete. 2024-04-26T00:33:57.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5ee04b3640 10 monclient: build_initial_monmap 2024-04-26T00:33:57.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5ee04b3640 10 monclient: monmap: 2024-04-26T00:33:57.245 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:33:57.245 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:33:57.245 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:33:57.245 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:33:57.245 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:33:57.245 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:33:57.245 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:33:57.245 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:33:57.245 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:33:57.245 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:57.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5ee04b3640 1 Processor -- start 2024-04-26T00:33:57.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5ee04b3640 1 -- start start 2024-04-26T00:33:57.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5ee04b3640 10 monclient: init 2024-04-26T00:33:57.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5ee04b3640 10 monclient: _reopen_session rank -1 2024-04-26T00:33:57.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5ee04b3640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:33:57.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5ee04b3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ed8072c80 0x7f5ed813b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:57.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5ee04b3640 10 monclient(hunting): picked mon.noname-b con 0x7f5ed8072c80 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:33:57.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5ee04b3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ed80a2fe0 0x7f5ed8130510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:57.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5ee04b3640 10 monclient(hunting): picked mon.noname-c con 0x7f5ed80a2fe0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:33:57.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5ee04b3640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5ed813c050 0x7f5ed8130a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:57.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5ee04b3640 10 monclient(hunting): picked mon.noname-a con 0x7f5ed813c050 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:33:57.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5ee04b3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:57.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5ee04b3640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f5ed8141570 con 0x7f5ed813c050 2024-04-26T00:33:57.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5ee04b3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:57.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5ee04b3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5ed81413f0 con 0x7f5ed8072c80 2024-04-26T00:33:57.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5ee04b3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:57.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5ee04b3640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5ed81416f0 con 0x7f5ed80a2fe0 2024-04-26T00:33:57.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5ee04b3640 10 monclient(hunting): _renew_subs 2024-04-26T00:33:57.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5ee04b3640 10 monclient(hunting): authenticate will time out at 1818.554263s 2024-04-26T00:33:57.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5edea50640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5ed813c050 0x7f5ed8130a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:57.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5ede24f640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ed8072c80 msgr2=0x7f5ed813b3f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:33:57.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5ede24f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ed8072c80 0x7f5ed813b3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:33:57.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5edda4e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ed80a2fe0 0x7f5ed8130510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:57.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5edea50640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5ed813c050 0x7f5ed8130a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:47110/0 (socket says 172.21.15.37:47110) 2024-04-26T00:33:57.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5edea50640 1 -- 172.21.15.37:0/3896959074 learned_addr learned my addr 172.21.15.37:0/3896959074 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:33:57.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5edea50640 10 monclient(hunting): get_auth_request con 0x7f5ed813c050 auth_method 0 2024-04-26T00:33:57.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5edea50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:57.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5edea50640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:57.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5edea50640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:57.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5edda4e640 10 monclient(hunting): get_auth_request con 0x7f5ed80a2fe0 auth_method 0 2024-04-26T00:33:57.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5edda4e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:57.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5edda4e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:57.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5edda4e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:57.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5edea50640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:57.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5edea50640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:57.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5edea50640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:57.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5edda4e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:57.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5edda4e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:57.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5edda4e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:57.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5edea50640 10 monclient(hunting): handle_auth_done global_id 144378 payload 824 2024-04-26T00:33:57.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5edea50640 1 -- 172.21.15.37:0/3896959074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ed80a2fe0 msgr2=0x7f5ed8130510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:57.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5edea50640 1 --2- 172.21.15.37:0/3896959074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ed80a2fe0 0x7f5ed8130510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:57.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5edea50640 1 -- 172.21.15.37:0/3896959074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ed8072c80 msgr2=0x7f5ed813b3f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:33:57.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5edea50640 1 --2- 172.21.15.37:0/3896959074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ed8072c80 0x7f5ed813b3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:57.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5edea50640 10 monclient: _finish_hunting 0 2024-04-26T00:33:57.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5edea50640 1 monclient: found mon.noname-a 2024-04-26T00:33:57.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5edea50640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:33:57.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5edea50640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:33:57.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5edea50640 1 -- 172.21.15.37:0/3896959074 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ed8071f40 con 0x7f5ed813c050 2024-04-26T00:33:57.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.247+0000 7f5edda4e640 1 --2- 172.21.15.37:0/3896959074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ed80a2fe0 0x7f5ed8130510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:33:57.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5edea50640 1 -- 172.21.15.37:0/3896959074 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5ed813c050 msgr2=0x7f5ed8130a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:57.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5edea50640 1 --2- 172.21.15.37:0/3896959074 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5ed813c050 0x7f5ed8130a50 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5ed001ae80 tx=0x7f5ed0019570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:57.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5eceffd640 1 -- 172.21.15.37:0/3896959074 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed00c0070 con 0x7f5ed813c050 2024-04-26T00:33:57.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5eceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:57.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5eceffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:33:57.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5eceffd640 10 monclient: dump: 2024-04-26T00:33:57.255 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:57.255 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:57.255 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:57.255 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:57.255 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:57.255 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:57.255 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:57.255 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:57.256 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:57.256 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:57.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5eceffd640 10 monclient: _finish_auth 0 2024-04-26T00:33:57.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5eceffd640 10 monclient: _check_auth_tickets 2024-04-26T00:33:57.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5eceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:33:57.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5eceffd640 1 -- 172.21.15.37:0/3896959074 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ed00a6e30 con 0x7f5ed813c050 2024-04-26T00:33:57.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5ee04b3640 5 monclient: authenticate success, global_id 144378 2024-04-26T00:33:57.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5ee04b3640 10 monclient: _renew_subs 2024-04-26T00:33:57.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5ee04b3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:33:57.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5ee04b3640 1 -- 172.21.15.37:0/3896959074 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ed81391f0 con 0x7f5ed813c050 2024-04-26T00:33:57.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5eceffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:33:57.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5eceffd640 1 -- 172.21.15.37:0/3896959074 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed00ac050 con 0x7f5ed813c050 2024-04-26T00:33:57.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5eceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:57.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5eceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:33:57.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5eceffd640 10 monclient: dump: 2024-04-26T00:33:57.257 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:57.257 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:57.257 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:57.257 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:57.257 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:57.257 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:57.257 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:57.257 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:57.257 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:57.257 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:57.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5edca4c640 4 set_mon_vals no callback set 2024-04-26T00:33:57.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5ee04b3640 10 monclient: _renew_subs 2024-04-26T00:33:57.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5ee04b3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:33:57.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5ee04b3640 1 -- 172.21.15.37:0/3896959074 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ed813a430 con 0x7f5ed813c050 2024-04-26T00:33:57.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5ee04b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:57.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5ee04b3640 1 --2- 172.21.15.37:0/3896959074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ed8071600 0x7f5ed8137cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:57.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5ee04b3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:57.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5ee04b3640 1 -- 172.21.15.37:0/3896959074 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5ed8131050 con 0x7f5ed8071600 2024-04-26T00:33:57.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5ede24f640 1 --2- 172.21.15.37:0/3896959074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ed8071600 0x7f5ed8137cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:57.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5ede24f640 10 monclient: get_auth_request con 0x7f5ed8071600 auth_method 0 2024-04-26T00:33:57.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5ede24f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:57.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5ede24f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:57.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5ede24f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:57.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5ede24f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:57.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5ede24f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:57.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5ede24f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:57.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5eceffd640 1 -- 172.21.15.37:0/3896959074 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f5ed00af070 con 0x7f5ed813c050 2024-04-26T00:33:57.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5ede24f640 10 monclient(hunting): handle_auth_done global_id 154247 payload 824 2024-04-26T00:33:57.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5ede24f640 1 -- 172.21.15.37:0/3896959074 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5ed804f510 con 0x7f5ed8071600 2024-04-26T00:33:57.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5eceffd640 1 --2- 172.21.15.37:0/3896959074 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5ebc035e30 0x7f5ebc0382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:57.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5eceffd640 1 -- 172.21.15.37:0/3896959074 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(537..537 src has 370..537) ==== 3975+0+0 (secure 0 0 0) 0x7f5ed00e2f70 con 0x7f5ed813c050 2024-04-26T00:33:57.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5edda4e640 1 --2- 172.21.15.37:0/3896959074 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5ebc035e30 0x7f5ebc0382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:57.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.251+0000 7f5edda4e640 10 monclient: get_auth_request con 0x7f5ebc035e30 auth_method 0 2024-04-26T00:33:57.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.255+0000 7f5ede24f640 1 -- 172.21.15.37:0/3896959074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ed8071600 msgr2=0x7f5ed8137cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:57.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.255+0000 7f5ede24f640 1 --2- 172.21.15.37:0/3896959074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ed8071600 0x7f5ed8137cb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5ed400c360 tx=0x7f5ed400c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:57.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.255+0000 7f5eceffd640 1 -- 172.21.15.37:0/3896959074 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed4017070 con 0x7f5ed8071600 2024-04-26T00:33:57.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.255+0000 7f5eceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:57.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.255+0000 7f5eceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:33:57.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.255+0000 7f5eceffd640 10 monclient: dump: 2024-04-26T00:33:57.263 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:57.263 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:57.263 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:57.263 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:57.263 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:57.263 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:57.263 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:57.263 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:57.263 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:57.263 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:57.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.255+0000 7f5eceffd640 1 -- 172.21.15.37:0/3896959074 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5ed4005800 con 0x7f5ed8071600 2024-04-26T00:33:57.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.255+0000 7f5eceffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:33:57.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.255+0000 7f5eceffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:33:57.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.255+0000 7f5eceffd640 1 -- 172.21.15.37:0/3896959074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ed8071600 msgr2=0x7f5ed8137cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:57.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.255+0000 7f5eceffd640 1 --2- 172.21.15.37:0/3896959074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ed8071600 0x7f5ed8137cb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5ed400c360 tx=0x7f5ed400c820 comp rx=0 tx=0).stop 2024-04-26T00:33:57.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.255+0000 7f5eceffd640 1 -- 172.21.15.37:0/3896959074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ed8071600 msgr2=0x7f5ed8137cb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:33:57.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.255+0000 7f5eceffd640 1 --2- 172.21.15.37:0/3896959074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ed8071600 0x7f5ed8137cb0 secure :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5ed400c360 tx=0x7f5ed400c820 comp rx=0 tx=0).stop 2024-04-26T00:33:57.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.255+0000 7f5edda4e640 1 --2- 172.21.15.37:0/3896959074 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5ebc035e30 0x7f5ebc0382f0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f5ec80096f0 tx=0x7f5ec8009240 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:57.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.259+0000 7f5ee04b3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:33:57.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.259+0000 7f5ee04b3640 1 --2- 172.21.15.37:0/3896959074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ed81381f0 0x7f5ed8138600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:57.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.259+0000 7f5ee04b3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:57.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.259+0000 7f5ee04b3640 1 -- 172.21.15.37:0/3896959074 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5ed8138b40 con 0x7f5ed81381f0 2024-04-26T00:33:57.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.259+0000 7f5ede24f640 1 --2- 172.21.15.37:0/3896959074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ed81381f0 0x7f5ed8138600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:57.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.259+0000 7f5ede24f640 10 monclient: get_auth_request con 0x7f5ed81381f0 auth_method 0 2024-04-26T00:33:57.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.259+0000 7f5ede24f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:57.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.259+0000 7f5ede24f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:57.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.259+0000 7f5ede24f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:57.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.259+0000 7f5ede24f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:57.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.259+0000 7f5ede24f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:57.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.259+0000 7f5ede24f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:57.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.259+0000 7f5ede24f640 10 monclient(hunting): handle_auth_done global_id 154250 payload 824 2024-04-26T00:33:57.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.259+0000 7f5ede24f640 1 -- 172.21.15.37:0/3896959074 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5ed806c1e0 con 0x7f5ed81381f0 2024-04-26T00:33:57.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.263+0000 7f5ede24f640 1 -- 172.21.15.37:0/3896959074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ed81381f0 msgr2=0x7f5ed8138600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:57.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.263+0000 7f5ede24f640 1 --2- 172.21.15.37:0/3896959074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ed81381f0 0x7f5ed8138600 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5ed400c360 tx=0x7f5ed4004990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:57.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.263+0000 7f5ede24f640 1 -- 172.21.15.37:0/3896959074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ed81381f0 msgr2=0x7f5ed8138600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:33:57.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.263+0000 7f5ede24f640 1 -- 172.21.15.37:0/3896959074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ed81381f0 msgr2=0x7f5ed8138600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:33:57.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.263+0000 7f5ede24f640 1 --2- 172.21.15.37:0/3896959074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ed81381f0 0x7f5ed8138600 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5ed400c360 tx=0x7f5ed4004990 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:33:57.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.263+0000 7f5ede24f640 1 --2- 172.21.15.37:0/3896959074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ed81381f0 0x7f5ed8138600 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5ed400c360 tx=0x7f5ed4004990 comp rx=0 tx=0).stop 2024-04-26T00:33:57.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.263+0000 7f5eceffd640 1 -- 172.21.15.37:0/3896959074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ed81381f0 msgr2=0x7f5ed8138600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:33:57.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.263+0000 7f5eceffd640 1 --2- 172.21.15.37:0/3896959074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ed81381f0 0x7f5ed8138600 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:57.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.263+0000 7f5eceffd640 1 --2- 172.21.15.37:0/3896959074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ebc03bd70 0x7f5ebc03e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:33:57.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.263+0000 7f5eceffd640 1 -- 172.21.15.37:0/3896959074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ed81381f0 msgr2=0x7f5ed8138600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:33:57.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.263+0000 7f5eceffd640 1 --2- 172.21.15.37:0/3896959074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ed81381f0 0x7f5ed8138600 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:57.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.263+0000 7f5eceffd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:33:57.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.263+0000 7f5eceffd640 1 -- 172.21.15.37:0/3896959074 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5ebc03e770 con 0x7f5ebc03bd70 2024-04-26T00:33:57.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.263+0000 7f5ede24f640 1 --2- 172.21.15.37:0/3896959074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ebc03bd70 0x7f5ebc03e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:33:57.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.263+0000 7f5ede24f640 10 monclient: get_auth_request con 0x7f5ebc03bd70 auth_method 0 2024-04-26T00:33:57.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.263+0000 7f5ede24f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:33:57.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.263+0000 7f5ede24f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:33:57.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.263+0000 7f5ede24f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:33:57.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.263+0000 7f5ede24f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:33:57.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.263+0000 7f5ede24f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:33:57.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.263+0000 7f5ede24f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:33:57.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.263+0000 7f5ede24f640 10 monclient(hunting): handle_auth_done global_id 154253 payload 824 2024-04-26T00:33:57.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.263+0000 7f5ede24f640 1 -- 172.21.15.37:0/3896959074 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5ed4005800 con 0x7f5ebc03bd70 2024-04-26T00:33:57.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.267+0000 7f5ede24f640 1 -- 172.21.15.37:0/3896959074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ebc03bd70 msgr2=0x7f5ebc03e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:33:57.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.267+0000 7f5ede24f640 1 --2- 172.21.15.37:0/3896959074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ebc03bd70 0x7f5ebc03e160 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5ed4007810 tx=0x7f5ed4007e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:33:57.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.267+0000 7f5eceffd640 1 -- 172.21.15.37:0/3896959074 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed4017030 con 0x7f5ebc03bd70 2024-04-26T00:33:57.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.267+0000 7f5eceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:33:57.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.267+0000 7f5eceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:33:57.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.267+0000 7f5eceffd640 10 monclient: dump: 2024-04-26T00:33:57.271 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:33:57.271 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:33:57.271 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:57.271 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:33:57.271 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:33:57.271 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:33:57.271 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:33:57.271 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:33:57.271 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:33:57.271 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:33:57.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.267+0000 7f5eceffd640 1 -- 172.21.15.37:0/3896959074 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4706 (secure 0 0 0) 0x7f5ed4000e50 con 0x7f5ebc03bd70 2024-04-26T00:33:57.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.267+0000 7f5eceffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:33:57.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.267+0000 7f5eceffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:33:57.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.267+0000 7f5eceffd640 1 -- 172.21.15.37:0/3896959074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ebc03bd70 msgr2=0x7f5ebc03e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:57.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.267+0000 7f5eceffd640 1 --2- 172.21.15.37:0/3896959074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ebc03bd70 0x7f5ebc03e160 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5ed4007810 tx=0x7f5ed4007e70 comp rx=0 tx=0).stop 2024-04-26T00:33:57.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.267+0000 7f5eceffd640 1 -- 172.21.15.37:0/3896959074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ebc03bd70 msgr2=0x7f5ebc03e160 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:33:57.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.267+0000 7f5eceffd640 1 --2- 172.21.15.37:0/3896959074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ebc03bd70 0x7f5ebc03e160 secure :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5ed4007810 tx=0x7f5ed4007e70 comp rx=0 tx=0).stop 2024-04-26T00:33:57.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.267+0000 7f5edea50640 1 -- 172.21.15.37:0/3896959074 reap_dead start 2024-04-26T00:33:57.272 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:33:57.272 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:33:57.272 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:33:57.272 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:33:57.272 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 106, 2024-04-26T00:33:57.272 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 74787, 2024-04-26T00:33:57.273 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:33:57.273 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:33:57.273 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:33:57.273 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:57.273 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 2, 2024-04-26T00:33:57.273 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:33:57.273 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:33:57.273 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:33:57.273 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:33:57.273 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:33:57.273 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:33:57.273 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:33:57.273 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:33:57.273 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:33:57.274 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:33:57.274 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:33:57.274 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:33:57.274 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:33:57.274 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:33:57.274 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:57.274 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:33:57.274 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:33:57.274 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:33:57.274 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:33:57.274 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:33:57.274 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:33:57.274 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:33:57.274 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:33:57.275 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:33:57.275 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:33:57.275 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:33:57.275 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:33:57.275 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:33:57.275 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:57.275 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:57.275 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:33:57.275 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:33:57.275 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:33:57.275 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:33:57.275 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:33:57.275 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:33:57.275 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:33:57.276 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:33:57.276 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:33:57.276 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:33:57.276 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:33:57.276 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:33:57.276 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:33:57.276 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:33:57.276 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:33:57.276 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:33:57.276 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:33:57.276 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:33:57.276 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:33:57.276 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:33:57.276 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:33:57.277 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:33:57.277 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:33:57.277 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:33:57.277 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:33:57.277 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:33:57.277 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:33:57.277 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:33:57.277 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:57.277 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:33:57.277 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:57.277 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:33:57.277 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:57.277 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:33:57.277 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:33:57.277 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:33:57.278 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:33:57.278 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:57.278 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:33:57.278 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:33:57.278 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:57.278 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:57.278 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:57.278 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:33:57.278 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:33:57.278 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:57.278 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:57.278 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:57.278 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:57.279 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:33:57.279 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:33:57.279 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:33:57.279 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:33:57.279 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:33:57.279 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:57.279 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:57.279 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:33:57.279 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:33:57.279 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:33:57.279 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:33:57.279 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:57.279 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:33:57.279 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:33:57.279 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:57.280 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:57.280 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:57.280 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:33:57.280 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:33:57.280 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:57.280 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:57.280 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:57.280 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:57.280 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:33:57.280 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:33:57.280 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:33:57.280 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:33:57.280 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:33:57.280 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:57.281 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:57.281 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:33:57.281 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:33:57.281 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:33:57.281 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:33:57.281 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:57.281 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:33:57.281 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:33:57.281 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:57.281 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:57.281 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:57.281 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:33:57.281 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:33:57.281 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:33:57.282 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:57.282 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:57.282 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:57.282 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:33:57.282 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:33:57.282 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:33:57.282 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:33:57.282 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:33:57.282 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:57.282 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:57.282 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:57.282 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:33:57.282 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:33:57.282 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:57.282 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:33:57.283 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:33:57.283 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:33:57.283 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:57.283 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:57.283 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:33:57.283 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:57.283 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:33:57.283 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:33:57.283 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:33:57.283 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:57.283 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:33:57.283 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:33:57.283 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:33:57.283 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:33:57.284 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:33:57.284 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 3 2024-04-26T00:33:57.284 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:33:57.284 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:33:57.284 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:33:57.284 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:33:57.284 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:33:57.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.267+0000 7f5ee04b3640 1 -- 172.21.15.37:0/3896959074 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5ebc035e30 msgr2=0x7f5ebc0382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:57.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.267+0000 7f5ee04b3640 1 --2- 172.21.15.37:0/3896959074 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5ebc035e30 0x7f5ebc0382f0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f5ec80096f0 tx=0x7f5ec8009240 comp rx=0 tx=0).stop 2024-04-26T00:33:57.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.267+0000 7f5ee04b3640 10 monclient: shutdown 2024-04-26T00:33:57.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.267+0000 7f5ee04b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:33:57.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.267+0000 7f5ee04b3640 1 -- 172.21.15.37:0/3896959074 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5ed813c050 msgr2=0x7f5ed8130a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:33:57.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.267+0000 7f5ee04b3640 1 --2- 172.21.15.37:0/3896959074 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5ed813c050 0x7f5ed8130a50 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5ed001ae80 tx=0x7f5ed0019570 comp rx=0 tx=0).stop 2024-04-26T00:33:57.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.267+0000 7f5ee04b3640 1 -- 172.21.15.37:0/3896959074 shutdown_connections 2024-04-26T00:33:57.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.267+0000 7f5ee04b3640 1 --2- 172.21.15.37:0/3896959074 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5ebc035e30 0x7f5ebc0382f0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:57.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.267+0000 7f5ee04b3640 1 --2- 172.21.15.37:0/3896959074 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5ed813c050 0x7f5ed8130a50 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:33:57.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.267+0000 7f5ee04b3640 1 -- 172.21.15.37:0/3896959074 >> 172.21.15.37:0/3896959074 conn(0x7f5ed80a24c0 msgr2=0x7f5ed8135460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:33:57.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.267+0000 7f5ee04b3640 1 -- 172.21.15.37:0/3896959074 shutdown_connections 2024-04-26T00:33:57.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:33:57.267+0000 7f5ee04b3640 1 -- 172.21.15.37:0/3896959074 wait complete. 2024-04-26T00:33:57.286 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:33:57.287 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-04-26T00:33:58.004 INFO:tasks.rados.rados.0.smithi045.stdout:2371: done (2 left) 2024-04-26T00:33:58.004 INFO:tasks.rados.rados.0.smithi045.stdout:2372: snap_create 2024-04-26T00:33:58.004 INFO:tasks.rados.rados.0.smithi045.stdout:2368: finishing copy_from racing read to smithi04526579-38 2024-04-26T00:33:58.004 INFO:tasks.rados.rados.0.smithi045.stdout:2368: finishing copy_from to smithi04526579-38 2024-04-26T00:33:58.004 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 1536 (ObjNum 807 snap 249 seq_num 807) dirty exists 2024-04-26T00:33:58.004 INFO:tasks.rados.rados.0.smithi045.stdout:2369: finishing copy_from racing read to smithi04526579-19 2024-04-26T00:33:58.004 INFO:tasks.rados.rados.0.smithi045.stdout:2369: finishing copy_from to smithi04526579-19 2024-04-26T00:33:58.004 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 1550 (ObjNum 823 snap 258 seq_num 823) dirty exists 2024-04-26T00:33:58.501 INFO:tasks.rados.rados.0.smithi045.stdout:2368: done (2 left) 2024-04-26T00:33:58.501 INFO:tasks.rados.rados.0.smithi045.stdout:2369: done (1 left) 2024-04-26T00:33:58.501 INFO:tasks.rados.rados.0.smithi045.stdout:2372: done (0 left) 2024-04-26T00:33:58.501 INFO:tasks.rados.rados.0.smithi045.stdout:2373: read oid 5 snap -1 2024-04-26T00:33:58.501 INFO:tasks.rados.rados.0.smithi045.stdout:2373: expect (ObjNum 836 snap 260 seq_num 836) 2024-04-26T00:33:58.501 INFO:tasks.rados.rados.0.smithi045.stdout:2374: snap_create 2024-04-26T00:33:58.516 INFO:tasks.rados.rados.0.smithi045.stdout:2373: expect (ObjNum 836 snap 260 seq_num 836) 2024-04-26T00:33:59.509 INFO:tasks.rados.rados.0.smithi045.stdout:2373: done (1 left) 2024-04-26T00:33:59.509 INFO:tasks.rados.rados.0.smithi045.stdout:2374: done (0 left) 2024-04-26T00:33:59.509 INFO:tasks.rados.rados.0.smithi045.stdout:2375: copy_from oid 30 from oid 20 current snap is 269 2024-04-26T00:33:59.510 INFO:tasks.rados.rados.0.smithi045.stdout:2376: copy_from oid 40 from oid 50 current snap is 269 2024-04-26T00:33:59.510 INFO:tasks.rados.rados.0.smithi045.stdout:2377: snap_remove snap 160 2024-04-26T00:34:00.511 INFO:tasks.rados.rados.0.smithi045.stdout:2377: done (2 left) 2024-04-26T00:34:00.511 INFO:tasks.rados.rados.0.smithi045.stdout:2378: read oid 47 snap -1 2024-04-26T00:34:00.511 INFO:tasks.rados.rados.0.smithi045.stdout:2378: expect deleted 2024-04-26T00:34:00.511 INFO:tasks.rados.rados.0.smithi045.stdout:2379: write (excl) oid 2 current snap is 269 2024-04-26T00:34:00.511 INFO:tasks.rados.rados.0.smithi045.stdout:2379: seq_num 863 ranges {504441=487776,1467372=736289} 2024-04-26T00:34:00.518 INFO:tasks.rados.rados.0.smithi045.stdout:2379: writing smithi04526579-2 from 504441 to 992217 tid 1 2024-04-26T00:34:00.530 INFO:tasks.rados.rados.0.smithi045.stdout:2379: writing smithi04526579-2 from 1467372 to 2203661 tid 2 2024-04-26T00:34:00.531 INFO:tasks.rados.rados.0.smithi045.stdout:2380: write oid 38 current snap is 269 2024-04-26T00:34:00.531 INFO:tasks.rados.rados.0.smithi045.stdout:2380: seq_num 864 ranges {521248=563025,1809501=596300} 2024-04-26T00:34:00.541 INFO:tasks.rados.rados.0.smithi045.stdout:2380: writing smithi04526579-38 from 521248 to 1084273 tid 1 2024-04-26T00:34:00.557 INFO:tasks.rados.rados.0.smithi045.stdout:2380: writing smithi04526579-38 from 1809501 to 2405801 tid 2 2024-04-26T00:34:00.558 INFO:tasks.rados.rados.0.smithi045.stdout:2381: read oid 17 snap -1 2024-04-26T00:34:00.558 INFO:tasks.rados.rados.0.smithi045.stdout:2381: expect (ObjNum 758 snap 232 seq_num 758) 2024-04-26T00:34:00.558 INFO:tasks.rados.rados.0.smithi045.stdout:2382: delete oid 14 current snap is 269 2024-04-26T00:34:00.558 INFO:tasks.rados.rados.0.smithi045.stdout:2376: finishing copy_from to smithi04526579-40 2024-04-26T00:34:00.558 INFO:tasks.rados.rados.0.smithi045.stdout:2376: got expected ENOENT (src dne) 2024-04-26T00:34:00.558 INFO:tasks.rados.rados.0.smithi045.stdout:2376: finishing copy_from racing read to smithi04526579-40 2024-04-26T00:34:00.558 INFO:tasks.rados.rados.0.smithi045.stdout:2375: finishing copy_from to smithi04526579-30 2024-04-26T00:34:00.558 INFO:tasks.rados.rados.0.smithi045.stdout:2375: got expected ENOENT (src dne) 2024-04-26T00:34:00.558 INFO:tasks.rados.rados.0.smithi045.stdout:2375: finishing copy_from racing read to smithi04526579-30 2024-04-26T00:34:00.566 INFO:tasks.rados.rados.0.smithi045.stdout:2381: expect (ObjNum 758 snap 232 seq_num 758) 2024-04-26T00:34:00.727 INFO:tasks.rados.rados.0.smithi045.stdout:2380: finishing write tid 1 to smithi04526579-38 2024-04-26T00:34:00.728 INFO:tasks.rados.rados.0.smithi045.stdout:2380: finishing write tid 2 to smithi04526579-38 2024-04-26T00:34:00.728 INFO:tasks.rados.rados.0.smithi045.stdout:2375: done (6 left) 2024-04-26T00:34:00.728 INFO:tasks.rados.rados.0.smithi045.stdout:2376: done (5 left) 2024-04-26T00:34:00.728 INFO:tasks.rados.rados.0.smithi045.stdout:2378: done (4 left) 2024-04-26T00:34:00.728 INFO:tasks.rados.rados.0.smithi045.stdout:2381: done (3 left) 2024-04-26T00:34:00.728 INFO:tasks.rados.rados.0.smithi045.stdout:2382: done (2 left) 2024-04-26T00:34:00.728 INFO:tasks.rados.rados.0.smithi045.stdout:2383: copy_from oid 49 from oid 26 current snap is 269 2024-04-26T00:34:00.728 INFO:tasks.rados.rados.0.smithi045.stdout:2380: finishing write tid 3 to smithi04526579-38 2024-04-26T00:34:00.728 INFO:tasks.rados.rados.0.smithi045.stdout:2380: finishing write tid 4 to smithi04526579-38 2024-04-26T00:34:00.728 INFO:tasks.rados.rados.0.smithi045.stdout:2380: oid 38 updating version 0 to 1538 2024-04-26T00:34:00.728 INFO:tasks.rados.rados.0.smithi045.stdout:2380: oid 38 updating version 1538 to 1539 2024-04-26T00:34:00.728 INFO:tasks.rados.rados.0.smithi045.stdout:2380: oid 38 version 1539 is already newer than 1537 2024-04-26T00:34:00.728 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 1539 (ObjNum 864 snap 269 seq_num 864) dirty exists 2024-04-26T00:34:00.729 INFO:tasks.rados.rados.0.smithi045.stdout:2380: left oid 38 (ObjNum 864 snap 269 seq_num 864) 2024-04-26T00:34:00.729 INFO:tasks.rados.rados.0.smithi045.stdout:2379: finishing write tid 1 to smithi04526579-2 2024-04-26T00:34:00.729 INFO:tasks.rados.rados.0.smithi045.stdout:2379: finishing write tid 2 to smithi04526579-2 2024-04-26T00:34:00.729 INFO:tasks.rados.rados.0.smithi045.stdout:2379: finishing write tid 3 to smithi04526579-2 2024-04-26T00:34:00.729 INFO:tasks.rados.rados.0.smithi045.stdout:2379: finishing write tid 4 to smithi04526579-2 2024-04-26T00:34:00.729 INFO:tasks.rados.rados.0.smithi045.stdout:2379: oid 2 updating version 0 to 1545 2024-04-26T00:34:00.729 INFO:tasks.rados.rados.0.smithi045.stdout:2379: oid 2 version 1545 is already newer than 1544 2024-04-26T00:34:00.729 INFO:tasks.rados.rados.0.smithi045.stdout:2379: oid 2 version 1545 is already newer than 1543 2024-04-26T00:34:00.729 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 1545 (ObjNum 863 snap 269 seq_num 863) dirty exists 2024-04-26T00:34:00.729 INFO:tasks.rados.rados.0.smithi045.stdout:2379: left oid 2 (ObjNum 863 snap 269 seq_num 863) 2024-04-26T00:34:00.729 INFO:tasks.rados.rados.0.smithi045.stdout:2379: done (2 left) 2024-04-26T00:34:00.729 INFO:tasks.rados.rados.0.smithi045.stdout:2380: done (1 left) 2024-04-26T00:34:00.729 INFO:tasks.rados.rados.0.smithi045.stdout:2384: snap_remove snap 245 2024-04-26T00:34:01.521 INFO:tasks.rados.rados.0.smithi045.stdout:2384: done (1 left) 2024-04-26T00:34:01.521 INFO:tasks.rados.rados.0.smithi045.stdout:2385: write (excl) oid 14 current snap is 269 2024-04-26T00:34:01.521 INFO:tasks.rados.rados.0.smithi045.stdout:2385: seq_num 866 ranges {554862=713523,1926465=599969} 2024-04-26T00:34:01.541 INFO:tasks.rados.rados.0.smithi045.stdout:2385: writing smithi04526579-14 from 554862 to 1268385 tid 1 2024-04-26T00:34:01.561 INFO:tasks.rados.rados.0.smithi045.stdout:2385: writing smithi04526579-14 from 1926465 to 2526434 tid 2 2024-04-26T00:34:01.562 INFO:tasks.rados.rados.0.smithi045.stdout:2386: snap_create 2024-04-26T00:34:01.562 INFO:tasks.rados.rados.0.smithi045.stdout:2383: finishing copy_from racing read to smithi04526579-49 2024-04-26T00:34:01.562 INFO:tasks.rados.rados.0.smithi045.stdout:2383: finishing copy_from to smithi04526579-49 2024-04-26T00:34:01.562 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 1304 (ObjNum 731 snap 224 seq_num 731) dirty exists 2024-04-26T00:34:01.562 INFO:tasks.rados.rados.0.smithi045.stdout:2385: finishing write tid 1 to smithi04526579-14 2024-04-26T00:34:01.567 INFO:tasks.rados.rados.0.smithi045.stdout:2385: finishing write tid 2 to smithi04526579-14 2024-04-26T00:34:01.569 INFO:tasks.rados.rados.0.smithi045.stdout:2385: finishing write tid 3 to smithi04526579-14 2024-04-26T00:34:01.570 INFO:tasks.rados.rados.0.smithi045.stdout:2385: finishing write tid 4 to smithi04526579-14 2024-04-26T00:34:01.570 INFO:tasks.rados.rados.0.smithi045.stdout:2385: oid 14 updating version 0 to 1149 2024-04-26T00:34:01.570 INFO:tasks.rados.rados.0.smithi045.stdout:2385: oid 14 version 1149 is already newer than 1148 2024-04-26T00:34:01.570 INFO:tasks.rados.rados.0.smithi045.stdout:2385: oid 14 updating version 1149 to 1150 2024-04-26T00:34:01.570 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 1150 (ObjNum 866 snap 269 seq_num 866) dirty exists 2024-04-26T00:34:01.570 INFO:tasks.rados.rados.0.smithi045.stdout:2385: left oid 14 (ObjNum 866 snap 269 seq_num 866) 2024-04-26T00:34:02.525 INFO:tasks.rados.rados.0.smithi045.stdout:2383: done (2 left) 2024-04-26T00:34:02.526 INFO:tasks.rados.rados.0.smithi045.stdout:2385: done (1 left) 2024-04-26T00:34:02.526 INFO:tasks.rados.rados.0.smithi045.stdout:2386: done (0 left) 2024-04-26T00:34:02.526 INFO:tasks.rados.rados.0.smithi045.stdout:2387: snap_remove snap 252 2024-04-26T00:34:03.504 INFO:tasks.rados.rados.0.smithi045.stdout:2387: done (0 left) 2024-04-26T00:34:03.504 INFO:tasks.rados.rados.0.smithi045.stdout:2388: read oid 21 snap -1 2024-04-26T00:34:03.504 INFO:tasks.rados.rados.0.smithi045.stdout:2388: expect deleted 2024-04-26T00:34:03.504 INFO:tasks.rados.rados.0.smithi045.stdout:2389: snap_remove snap 258 2024-04-26T00:34:04.517 INFO:tasks.rados.rados.0.smithi045.stdout:2389: done (1 left) 2024-04-26T00:34:04.518 INFO:tasks.rados.rados.0.smithi045.stdout:2390: snap_create 2024-04-26T00:34:05.516 INFO:tasks.rados.rados.0.smithi045.stdout:2388: done (1 left) 2024-04-26T00:34:05.516 INFO:tasks.rados.rados.0.smithi045.stdout:2390: done (0 left) 2024-04-26T00:34:05.516 INFO:tasks.rados.rados.0.smithi045.stdout:2391: rollback oid 28 current snap is 271 2024-04-26T00:34:05.517 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 28 to 261 2024-04-26T00:34:05.517 INFO:tasks.rados.rados.0.smithi045.stdout:2392: read oid 25 snap -1 2024-04-26T00:34:05.517 INFO:tasks.rados.rados.0.smithi045.stdout:2392: expect (ObjNum 859 snap 264 seq_num 859) 2024-04-26T00:34:05.517 INFO:tasks.rados.rados.0.smithi045.stdout:2393: read oid 26 snap -1 2024-04-26T00:34:05.517 INFO:tasks.rados.rados.0.smithi045.stdout:2393: expect (ObjNum 731 snap 224 seq_num 731) 2024-04-26T00:34:05.517 INFO:tasks.rados.rados.0.smithi045.stdout:2394: snap_create 2024-04-26T00:34:05.532 INFO:tasks.rados.rados.0.smithi045.stdout:2392: expect (ObjNum 859 snap 264 seq_num 859) 2024-04-26T00:34:05.730 INFO:tasks.rados.rados.0.smithi045.stdout:2391: finishing rollback tid 0 to smithi04526579-28 2024-04-26T00:34:05.730 INFO:tasks.rados.rados.0.smithi045.stdout:2391: finishing rollback tid 1 to smithi04526579-28 2024-04-26T00:34:05.730 INFO:tasks.rados.rados.0.smithi045.stdout:2391: finishing rollback tid 2 to smithi04526579-28 2024-04-26T00:34:05.730 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 1922 (ObjNum 604 snap 186 seq_num 604) dirty exists 2024-04-26T00:34:05.730 INFO:tasks.rados.rados.0.smithi045.stdout:2393: expect (ObjNum 731 snap 224 seq_num 731) 2024-04-26T00:34:06.517 INFO:tasks.rados.rados.0.smithi045.stdout:2391: done (3 left) 2024-04-26T00:34:06.517 INFO:tasks.rados.rados.0.smithi045.stdout:2392: done (2 left) 2024-04-26T00:34:06.517 INFO:tasks.rados.rados.0.smithi045.stdout:2393: done (1 left) 2024-04-26T00:34:06.517 INFO:tasks.rados.rados.0.smithi045.stdout:2394: done (0 left) 2024-04-26T00:34:06.518 INFO:tasks.rados.rados.0.smithi045.stdout:2395: delete oid 30 current snap is 272 2024-04-26T00:34:06.520 INFO:tasks.rados.rados.0.smithi045.stdout:2395: done (0 left) 2024-04-26T00:34:06.520 INFO:tasks.rados.rados.0.smithi045.stdout:2396: snap_remove snap 247 2024-04-26T00:34:07.816 INFO:tasks.rados.rados.0.smithi045.stdout:2396: done (0 left) 2024-04-26T00:34:07.816 INFO:tasks.rados.rados.0.smithi045.stdout:2397: snap_remove snap 267 2024-04-26T00:34:08.507 INFO:tasks.rados.rados.0.smithi045.stdout:2397: done (0 left) 2024-04-26T00:34:08.507 INFO:tasks.rados.rados.0.smithi045.stdout:2398: read oid 14 snap -1 2024-04-26T00:34:08.507 INFO:tasks.rados.rados.0.smithi045.stdout:2398: expect (ObjNum 866 snap 269 seq_num 866) 2024-04-26T00:34:08.507 INFO:tasks.rados.rados.0.smithi045.stdout:2399: read oid 27 snap -1 2024-04-26T00:34:08.507 INFO:tasks.rados.rados.0.smithi045.stdout:2399: expect (ObjNum 849 snap 262 seq_num 849) 2024-04-26T00:34:08.508 INFO:tasks.rados.rados.0.smithi045.stdout:2400: write oid 13 current snap is 272 2024-04-26T00:34:08.508 INFO:tasks.rados.rados.0.smithi045.stdout:2400: seq_num 868 ranges {588476=580374,1643429=603638} 2024-04-26T00:34:08.527 INFO:tasks.rados.rados.0.smithi045.stdout:2400: writing smithi04526579-13 from 588476 to 1168850 tid 1 2024-04-26T00:34:08.548 INFO:tasks.rados.rados.0.smithi045.stdout:2400: writing smithi04526579-13 from 1643429 to 2247067 tid 2 2024-04-26T00:34:08.548 INFO:tasks.rados.rados.0.smithi045.stdout:2401: read oid 29 snap -1 2024-04-26T00:34:08.549 INFO:tasks.rados.rados.0.smithi045.stdout:2401: expect (ObjNum 820 snap 257 seq_num 820) 2024-04-26T00:34:08.549 INFO:tasks.rados.rados.0.smithi045.stdout:2402: copy_from oid 3 from oid 6 current snap is 272 2024-04-26T00:34:08.549 INFO:tasks.rados.rados.0.smithi045.stdout:2399: expect (ObjNum 849 snap 262 seq_num 849) 2024-04-26T00:34:08.716 INFO:tasks.rados.rados.0.smithi045.stdout:2398: expect (ObjNum 866 snap 269 seq_num 866) 2024-04-26T00:34:08.797 INFO:tasks.rados.rados.0.smithi045.stdout:2400: finishing write tid 1 to smithi04526579-13 2024-04-26T00:34:08.797 INFO:tasks.rados.rados.0.smithi045.stdout:2402: finishing copy_from to smithi04526579-3 2024-04-26T00:34:08.797 INFO:tasks.rados.rados.0.smithi045.stdout:2402: got expected ENOENT (src dne) 2024-04-26T00:34:08.798 INFO:tasks.rados.rados.0.smithi045.stdout:2402: finishing copy_from racing read to smithi04526579-3 2024-04-26T00:34:08.798 INFO:tasks.rados.rados.0.smithi045.stdout:2400: finishing write tid 2 to smithi04526579-13 2024-04-26T00:34:08.798 INFO:tasks.rados.rados.0.smithi045.stdout:2400: finishing write tid 3 to smithi04526579-13 2024-04-26T00:34:08.798 INFO:tasks.rados.rados.0.smithi045.stdout:2400: finishing write tid 4 to smithi04526579-13 2024-04-26T00:34:08.798 INFO:tasks.rados.rados.0.smithi045.stdout:2400: oid 13 updating version 0 to 1543 2024-04-26T00:34:08.798 INFO:tasks.rados.rados.0.smithi045.stdout:2400: oid 13 updating version 1543 to 1544 2024-04-26T00:34:08.798 INFO:tasks.rados.rados.0.smithi045.stdout:2400: oid 13 version 1544 is already newer than 1542 2024-04-26T00:34:08.798 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 1544 (ObjNum 868 snap 272 seq_num 868) dirty exists 2024-04-26T00:34:08.798 INFO:tasks.rados.rados.0.smithi045.stdout:2400: left oid 13 (ObjNum 868 snap 272 seq_num 868) 2024-04-26T00:34:08.798 INFO:tasks.rados.rados.0.smithi045.stdout:2401: expect (ObjNum 820 snap 257 seq_num 820) 2024-04-26T00:34:08.919 INFO:tasks.rados.rados.0.smithi045.stdout:2398: done (4 left) 2024-04-26T00:34:08.919 INFO:tasks.rados.rados.0.smithi045.stdout:2399: done (3 left) 2024-04-26T00:34:08.919 INFO:tasks.rados.rados.0.smithi045.stdout:2400: done (2 left) 2024-04-26T00:34:08.919 INFO:tasks.rados.rados.0.smithi045.stdout:2401: done (1 left) 2024-04-26T00:34:08.920 INFO:tasks.rados.rados.0.smithi045.stdout:2402: done (0 left) 2024-04-26T00:34:08.920 INFO:tasks.rados.rados.0.smithi045.stdout:2403: snap_remove snap 237 2024-04-26T00:34:09.826 INFO:tasks.rados.rados.0.smithi045.stdout:2403: done (0 left) 2024-04-26T00:34:09.826 INFO:tasks.rados.rados.0.smithi045.stdout:2404: snap_remove snap 265 2024-04-26T00:34:11.297 INFO:tasks.rados.rados.0.smithi045.stdout:2404: done (0 left) 2024-04-26T00:34:11.297 INFO:tasks.rados.rados.0.smithi045.stdout:2405: write oid 23 current snap is 272 2024-04-26T00:34:11.297 INFO:tasks.rados.rados.0.smithi045.stdout:2405: seq_num 869 ranges {605283=655623,1701911=747296} 2024-04-26T00:34:11.318 INFO:tasks.rados.rados.0.smithi045.stdout:2405: writing smithi04526579-23 from 605283 to 1260906 tid 1 2024-04-26T00:34:11.337 INFO:tasks.rados.rados.0.smithi045.stdout:2405: writing smithi04526579-23 from 1701911 to 2449207 tid 2 2024-04-26T00:34:11.338 INFO:tasks.rados.rados.0.smithi045.stdout:2406: snap_remove snap 271 2024-04-26T00:34:11.871 INFO:tasks.rados.rados.0.smithi045.stdout:2406: done (1 left) 2024-04-26T00:34:11.871 INFO:tasks.rados.rados.0.smithi045.stdout:2407: copy_from oid 13 from oid 14 current snap is 272 2024-04-26T00:34:11.871 INFO:tasks.rados.rados.0.smithi045.stdout:2405: finishing write tid 1 to smithi04526579-23 2024-04-26T00:34:11.872 INFO:tasks.rados.rados.0.smithi045.stdout:2405: finishing write tid 2 to smithi04526579-23 2024-04-26T00:34:11.872 INFO:tasks.rados.rados.0.smithi045.stdout:2405: finishing write tid 3 to smithi04526579-23 2024-04-26T00:34:11.872 INFO:tasks.rados.rados.0.smithi045.stdout:2405: finishing write tid 4 to smithi04526579-23 2024-04-26T00:34:11.872 INFO:tasks.rados.rados.0.smithi045.stdout:2405: oid 23 updating version 0 to 1469 2024-04-26T00:34:11.872 INFO:tasks.rados.rados.0.smithi045.stdout:2405: oid 23 updating version 1469 to 1470 2024-04-26T00:34:11.872 INFO:tasks.rados.rados.0.smithi045.stdout:2405: oid 23 version 1470 is already newer than 1468 2024-04-26T00:34:11.872 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 1470 (ObjNum 869 snap 272 seq_num 869) dirty exists 2024-04-26T00:34:11.872 INFO:tasks.rados.rados.0.smithi045.stdout:2405: left oid 23 (ObjNum 869 snap 272 seq_num 869) 2024-04-26T00:34:11.872 INFO:tasks.rados.rados.0.smithi045.stdout:2405: done (1 left) 2024-04-26T00:34:11.872 INFO:tasks.rados.rados.0.smithi045.stdout:2408: snap_remove snap 255 2024-04-26T00:34:12.875 INFO:tasks.rados.rados.0.smithi045.stdout:2407: finishing copy_from racing read to smithi04526579-13 2024-04-26T00:34:12.876 INFO:tasks.rados.rados.0.smithi045.stdout:2407: finishing copy_from to smithi04526579-13 2024-04-26T00:34:12.876 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 1150 (ObjNum 866 snap 269 seq_num 866) dirty exists 2024-04-26T00:34:12.876 INFO:tasks.rados.rados.0.smithi045.stdout:2407: done (1 left) 2024-04-26T00:34:12.876 INFO:tasks.rados.rados.0.smithi045.stdout:2408: done (0 left) 2024-04-26T00:34:12.876 INFO:tasks.rados.rados.0.smithi045.stdout:2409: write (excl) oid 17 current snap is 272 2024-04-26T00:34:12.876 INFO:tasks.rados.rados.0.smithi045.stdout:2409: seq_num 870 ranges {622090=730872,1760393=607307} 2024-04-26T00:34:12.899 INFO:tasks.rados.rados.0.smithi045.stdout:2409: writing smithi04526579-17 from 622090 to 1352962 tid 1 2024-04-26T00:34:12.918 INFO:tasks.rados.rados.0.smithi045.stdout:2409: writing smithi04526579-17 from 1760393 to 2367700 tid 2 2024-04-26T00:34:12.918 INFO:tasks.rados.rados.0.smithi045.stdout:2410: copy_from oid 9 from oid 28 current snap is 272 2024-04-26T00:34:12.918 INFO:tasks.rados.rados.0.smithi045.stdout:2409: finishing write tid 1 to smithi04526579-17 2024-04-26T00:34:12.918 INFO:tasks.rados.rados.0.smithi045.stdout:2411: delete oid 34 current snap is 272 2024-04-26T00:34:12.924 INFO:tasks.rados.rados.0.smithi045.stdout:2411: done (2 left) 2024-04-26T00:34:12.924 INFO:tasks.rados.rados.0.smithi045.stdout:2412: write oid 34 current snap is 272 2024-04-26T00:34:12.924 INFO:tasks.rados.rados.0.smithi045.stdout:2412: seq_num 872 ranges {655704=481370,1761004=494623} 2024-04-26T00:34:12.942 INFO:tasks.rados.rados.0.smithi045.stdout:2412: writing smithi04526579-34 from 655704 to 1137074 tid 1 2024-04-26T00:34:12.959 INFO:tasks.rados.rados.0.smithi045.stdout:2412: writing smithi04526579-34 from 1761004 to 2255627 tid 2 2024-04-26T00:34:12.959 INFO:tasks.rados.rados.0.smithi045.stdout:2413: read oid 49 snap -1 2024-04-26T00:34:12.959 INFO:tasks.rados.rados.0.smithi045.stdout:2413: expect (ObjNum 731 snap 224 seq_num 731) 2024-04-26T00:34:12.959 INFO:tasks.rados.rados.0.smithi045.stdout:2414: snap_remove snap 259 2024-04-26T00:34:13.523 INFO:tasks.rados.rados.0.smithi045.stdout:2414: done (4 left) 2024-04-26T00:34:13.523 INFO:tasks.rados.rados.0.smithi045.stdout:2415: delete oid 13 current snap is 272 2024-04-26T00:34:13.523 INFO:tasks.rados.rados.0.smithi045.stdout:2409: finishing write tid 2 to smithi04526579-17 2024-04-26T00:34:13.523 INFO:tasks.rados.rados.0.smithi045.stdout:2409: finishing write tid 3 to smithi04526579-17 2024-04-26T00:34:13.523 INFO:tasks.rados.rados.0.smithi045.stdout:2409: finishing write tid 4 to smithi04526579-17 2024-04-26T00:34:13.523 INFO:tasks.rados.rados.0.smithi045.stdout:2409: oid 17 updating version 0 to 1531 2024-04-26T00:34:13.523 INFO:tasks.rados.rados.0.smithi045.stdout:2409: oid 17 version 1531 is already newer than 1529 2024-04-26T00:34:13.523 INFO:tasks.rados.rados.0.smithi045.stdout:2409: oid 17 version 1531 is already newer than 1530 2024-04-26T00:34:13.523 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 1531 (ObjNum 870 snap 272 seq_num 870) dirty exists 2024-04-26T00:34:13.523 INFO:tasks.rados.rados.0.smithi045.stdout:2409: left oid 17 (ObjNum 870 snap 272 seq_num 870) 2024-04-26T00:34:13.524 INFO:tasks.rados.rados.0.smithi045.stdout:2410: finishing copy_from racing read to smithi04526579-9 2024-04-26T00:34:13.524 INFO:tasks.rados.rados.0.smithi045.stdout:2410: finishing copy_from to smithi04526579-9 2024-04-26T00:34:13.524 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 1922 (ObjNum 604 snap 186 seq_num 604) dirty exists 2024-04-26T00:34:13.524 INFO:tasks.rados.rados.0.smithi045.stdout:2412: finishing write tid 1 to smithi04526579-34 2024-04-26T00:34:13.524 INFO:tasks.rados.rados.0.smithi045.stdout:2412: finishing write tid 2 to smithi04526579-34 2024-04-26T00:34:13.524 INFO:tasks.rados.rados.0.smithi045.stdout:2412: finishing write tid 3 to smithi04526579-34 2024-04-26T00:34:13.524 INFO:tasks.rados.rados.0.smithi045.stdout:2412: finishing write tid 4 to smithi04526579-34 2024-04-26T00:34:13.524 INFO:tasks.rados.rados.0.smithi045.stdout:2412: oid 34 updating version 0 to 1511 2024-04-26T00:34:13.524 INFO:tasks.rados.rados.0.smithi045.stdout:2412: oid 34 updating version 1511 to 1513 2024-04-26T00:34:13.524 INFO:tasks.rados.rados.0.smithi045.stdout:2412: oid 34 version 1513 is already newer than 1512 2024-04-26T00:34:13.524 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 1513 (ObjNum 872 snap 272 seq_num 872) dirty exists 2024-04-26T00:34:13.524 INFO:tasks.rados.rados.0.smithi045.stdout:2412: left oid 34 (ObjNum 872 snap 272 seq_num 872) 2024-04-26T00:34:13.524 INFO:tasks.rados.rados.0.smithi045.stdout:2413: expect (ObjNum 731 snap 224 seq_num 731) 2024-04-26T00:34:13.661 INFO:tasks.rados.rados.0.smithi045.stdout:2409: done (4 left) 2024-04-26T00:34:13.661 INFO:tasks.rados.rados.0.smithi045.stdout:2410: done (3 left) 2024-04-26T00:34:13.661 INFO:tasks.rados.rados.0.smithi045.stdout:2412: done (2 left) 2024-04-26T00:34:13.661 INFO:tasks.rados.rados.0.smithi045.stdout:2413: done (1 left) 2024-04-26T00:34:13.661 INFO:tasks.rados.rados.0.smithi045.stdout:2415: done (0 left) 2024-04-26T00:34:13.661 INFO:tasks.rados.rados.0.smithi045.stdout:2416: read oid 32 snap -1 2024-04-26T00:34:13.661 INFO:tasks.rados.rados.0.smithi045.stdout:2416: expect (ObjNum 769 snap 238 seq_num 769) 2024-04-26T00:34:13.661 INFO:tasks.rados.rados.0.smithi045.stdout:2417: write oid 28 current snap is 272 2024-04-26T00:34:13.661 INFO:tasks.rados.rados.0.smithi045.stdout:2417: seq_num 874 ranges {689318=631868,1877968=498292} 2024-04-26T00:34:13.682 INFO:tasks.rados.rados.0.smithi045.stdout:2417: writing smithi04526579-28 from 689318 to 1321186 tid 1 2024-04-26T00:34:13.701 INFO:tasks.rados.rados.0.smithi045.stdout:2417: writing smithi04526579-28 from 1877968 to 2376260 tid 2 2024-04-26T00:34:13.701 INFO:tasks.rados.rados.0.smithi045.stdout:2418: read oid 46 snap 243 2024-04-26T00:34:13.701 INFO:tasks.rados.rados.0.smithi045.stdout:2418: expect (ObjNum 749 snap 231 seq_num 749) 2024-04-26T00:34:13.701 INFO:tasks.rados.rados.0.smithi045.stdout:2419: read oid 3 snap -1 2024-04-26T00:34:13.701 INFO:tasks.rados.rados.0.smithi045.stdout:2419: expect (ObjNum 790 snap 241 seq_num 790) 2024-04-26T00:34:13.701 INFO:tasks.rados.rados.0.smithi045.stdout:2420: snap_remove snap 262 2024-04-26T00:34:14.528 INFO:tasks.rados.rados.0.smithi045.stdout:2420: done (4 left) 2024-04-26T00:34:14.528 INFO:tasks.rados.rados.0.smithi045.stdout:2421: delete oid 26 current snap is 272 2024-04-26T00:34:14.528 INFO:tasks.rados.rados.0.smithi045.stdout:2416: expect (ObjNum 769 snap 238 seq_num 769) 2024-04-26T00:34:14.723 INFO:tasks.rados.rados.0.smithi045.stdout:2417: finishing write tid 1 to smithi04526579-28 2024-04-26T00:34:14.724 INFO:tasks.rados.rados.0.smithi045.stdout:2417: finishing write tid 2 to smithi04526579-28 2024-04-26T00:34:14.724 INFO:tasks.rados.rados.0.smithi045.stdout:2417: finishing write tid 3 to smithi04526579-28 2024-04-26T00:34:14.724 INFO:tasks.rados.rados.0.smithi045.stdout:2417: finishing write tid 4 to smithi04526579-28 2024-04-26T00:34:14.724 INFO:tasks.rados.rados.0.smithi045.stdout:2417: oid 28 updating version 0 to 1924 2024-04-26T00:34:14.724 INFO:tasks.rados.rados.0.smithi045.stdout:2417: oid 28 updating version 1924 to 1925 2024-04-26T00:34:14.724 INFO:tasks.rados.rados.0.smithi045.stdout:2417: oid 28 version 1925 is already newer than 1923 2024-04-26T00:34:14.724 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 1925 (ObjNum 874 snap 272 seq_num 874) dirty exists 2024-04-26T00:34:14.725 INFO:tasks.rados.rados.0.smithi045.stdout:2417: left oid 28 (ObjNum 874 snap 272 seq_num 874) 2024-04-26T00:34:14.725 INFO:tasks.rados.rados.0.smithi045.stdout:2418: expect (ObjNum 749 snap 231 seq_num 749) 2024-04-26T00:34:14.884 INFO:tasks.rados.rados.0.smithi045.stdout:2419: expect (ObjNum 790 snap 241 seq_num 790) 2024-04-26T00:34:15.085 INFO:tasks.rados.rados.0.smithi045.stdout:2416: done (4 left) 2024-04-26T00:34:15.085 INFO:tasks.rados.rados.0.smithi045.stdout:2417: done (3 left) 2024-04-26T00:34:15.085 INFO:tasks.rados.rados.0.smithi045.stdout:2418: done (2 left) 2024-04-26T00:34:15.085 INFO:tasks.rados.rados.0.smithi045.stdout:2419: done (1 left) 2024-04-26T00:34:15.086 INFO:tasks.rados.rados.0.smithi045.stdout:2421: done (0 left) 2024-04-26T00:34:15.086 INFO:tasks.rados.rados.0.smithi045.stdout:2422: read oid 27 snap -1 2024-04-26T00:34:15.086 INFO:tasks.rados.rados.0.smithi045.stdout:2422: expect (ObjNum 849 snap 262 seq_num 849) 2024-04-26T00:34:15.086 INFO:tasks.rados.rados.0.smithi045.stdout:2423: read oid 10 snap -1 2024-04-26T00:34:15.086 INFO:tasks.rados.rados.0.smithi045.stdout:2423: expect (ObjNum 788 snap 241 seq_num 788) 2024-04-26T00:34:15.086 INFO:tasks.rados.rados.0.smithi045.stdout:2424: copy_from oid 48 from oid 5 current snap is 272 2024-04-26T00:34:15.086 INFO:tasks.rados.rados.0.smithi045.stdout:2425: delete oid 39 current snap is 272 2024-04-26T00:34:15.090 INFO:tasks.rados.rados.0.smithi045.stdout:2425: done (3 left) 2024-04-26T00:34:15.091 INFO:tasks.rados.rados.0.smithi045.stdout:2426: write (excl) oid 16 current snap is 272 2024-04-26T00:34:15.091 INFO:tasks.rados.rados.0.smithi045.stdout:2426: seq_num 877 ranges {739739=573968,2053414=686325} 2024-04-26T00:34:15.114 INFO:tasks.rados.rados.0.smithi045.stdout:2426: writing smithi04526579-16 from 739739 to 1313707 tid 1 2024-04-26T00:34:15.137 INFO:tasks.rados.rados.0.smithi045.stdout:2426: writing smithi04526579-16 from 2053414 to 2739739 tid 2 2024-04-26T00:34:15.138 INFO:tasks.rados.rados.0.smithi045.stdout:2427: snap_create 2024-04-26T00:34:15.138 INFO:tasks.rados.rados.0.smithi045.stdout:2422: expect (ObjNum 849 snap 262 seq_num 849) 2024-04-26T00:34:15.287 INFO:tasks.rados.rados.0.smithi045.stdout:2423: expect (ObjNum 788 snap 241 seq_num 788) 2024-04-26T00:34:15.513 INFO:tasks.rados.rados.0.smithi045.stdout:2424: finishing copy_from racing read to smithi04526579-48 2024-04-26T00:34:15.513 INFO:tasks.rados.rados.0.smithi045.stdout:2424: finishing copy_from to smithi04526579-48 2024-04-26T00:34:15.513 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 1977 (ObjNum 836 snap 260 seq_num 836) dirty exists 2024-04-26T00:34:15.514 INFO:tasks.rados.rados.0.smithi045.stdout:2426: finishing write tid 1 to smithi04526579-16 2024-04-26T00:34:15.514 INFO:tasks.rados.rados.0.smithi045.stdout:2426: finishing write tid 2 to smithi04526579-16 2024-04-26T00:34:15.514 INFO:tasks.rados.rados.0.smithi045.stdout:2426: finishing write tid 3 to smithi04526579-16 2024-04-26T00:34:15.514 INFO:tasks.rados.rados.0.smithi045.stdout:2426: finishing write tid 4 to smithi04526579-16 2024-04-26T00:34:15.514 INFO:tasks.rados.rados.0.smithi045.stdout:2426: oid 16 updating version 0 to 1152 2024-04-26T00:34:15.514 INFO:tasks.rados.rados.0.smithi045.stdout:2426: oid 16 updating version 1152 to 1153 2024-04-26T00:34:15.514 INFO:tasks.rados.rados.0.smithi045.stdout:2426: oid 16 version 1153 is already newer than 1151 2024-04-26T00:34:15.514 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 1153 (ObjNum 877 snap 272 seq_num 877) dirty exists 2024-04-26T00:34:15.514 INFO:tasks.rados.rados.0.smithi045.stdout:2426: left oid 16 (ObjNum 877 snap 272 seq_num 877) 2024-04-26T00:34:15.902 INFO:tasks.rados.rados.0.smithi045.stdout:2422: done (4 left) 2024-04-26T00:34:15.902 INFO:tasks.rados.rados.0.smithi045.stdout:2423: done (3 left) 2024-04-26T00:34:15.902 INFO:tasks.rados.rados.0.smithi045.stdout:2424: done (2 left) 2024-04-26T00:34:15.902 INFO:tasks.rados.rados.0.smithi045.stdout:2426: done (1 left) 2024-04-26T00:34:15.902 INFO:tasks.rados.rados.0.smithi045.stdout:2427: done (0 left) 2024-04-26T00:34:15.902 INFO:tasks.rados.rados.0.smithi045.stdout:2428: write oid 3 current snap is 273 2024-04-26T00:34:15.902 INFO:tasks.rados.rados.0.smithi045.stdout:2428: seq_num 878 ranges {756546=649217,2111896=505630} 2024-04-26T00:34:15.921 INFO:tasks.rados.rados.0.smithi045.stdout:2428: writing smithi04526579-3 from 756546 to 1405763 tid 1 2024-04-26T00:34:15.940 INFO:tasks.rados.rados.0.smithi045.stdout:2428: writing smithi04526579-3 from 2111896 to 2617526 tid 2 2024-04-26T00:34:15.940 INFO:tasks.rados.rados.0.smithi045.stdout:2429: copy_from oid 1 from oid 30 current snap is 273 2024-04-26T00:34:15.940 INFO:tasks.rados.rados.0.smithi045.stdout:2428: finishing write tid 1 to smithi04526579-3 2024-04-26T00:34:15.940 INFO:tasks.rados.rados.0.smithi045.stdout:2430: write (excl) oid 39 current snap is 273 2024-04-26T00:34:15.940 INFO:tasks.rados.rados.0.smithi045.stdout:2430: seq_num 879 ranges {773353=724466,2170378=602975} 2024-04-26T00:34:15.961 INFO:tasks.rados.rados.0.smithi045.stdout:2430: writing smithi04526579-39 from 773353 to 1497819 tid 1 2024-04-26T00:34:15.974 INFO:tasks.rados.rados.0.smithi045.stdout:2430: writing smithi04526579-39 from 2170378 to 2773353 tid 2 2024-04-26T00:34:15.975 INFO:tasks.rados.rados.0.smithi045.stdout:2431: write oid 42 current snap is 273 2024-04-26T00:34:15.975 INFO:tasks.rados.rados.0.smithi045.stdout:2431: seq_num 880 ranges {790160=799715,2112507=509299} 2024-04-26T00:34:15.988 INFO:tasks.rados.rados.0.smithi045.stdout:2431: writing smithi04526579-42 from 790160 to 1589875 tid 1 2024-04-26T00:34:15.997 INFO:tasks.rados.rados.0.smithi045.stdout:2431: writing smithi04526579-42 from 2112507 to 2621806 tid 2 2024-04-26T00:34:15.997 INFO:tasks.rados.rados.0.smithi045.stdout:2432: read oid 8 snap -1 2024-04-26T00:34:15.997 INFO:tasks.rados.rados.0.smithi045.stdout:2432: expect deleted 2024-04-26T00:34:15.997 INFO:tasks.rados.rados.0.smithi045.stdout:2433: rollback oid 26 current snap is 273 2024-04-26T00:34:15.997 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 26 to 270 2024-04-26T00:34:15.997 INFO:tasks.rados.rados.0.smithi045.stdout:2429: finishing copy_from to smithi04526579-1 2024-04-26T00:34:15.997 INFO:tasks.rados.rados.0.smithi045.stdout:2429: got expected ENOENT (src dne) 2024-04-26T00:34:15.997 INFO:tasks.rados.rados.0.smithi045.stdout:2429: finishing copy_from racing read to smithi04526579-1 2024-04-26T00:34:15.997 INFO:tasks.rados.rados.0.smithi045.stdout:2429: done (5 left) 2024-04-26T00:34:15.998 INFO:tasks.rados.rados.0.smithi045.stdout:2434: write (excl) oid 4 current snap is 273 2024-04-26T00:34:15.998 INFO:tasks.rados.rados.0.smithi045.stdout:2434: seq_num 881 ranges {406967=474964,1370989=769310,2677186=129781} 2024-04-26T00:34:16.004 INFO:tasks.rados.rados.0.smithi045.stdout:2434: writing smithi04526579-4 from 406967 to 881931 tid 1 2024-04-26T00:34:16.015 INFO:tasks.rados.rados.0.smithi045.stdout:2434: writing smithi04526579-4 from 1370989 to 2140299 tid 2 2024-04-26T00:34:16.021 INFO:tasks.rados.rados.0.smithi045.stdout:2434: writing smithi04526579-4 from 2677186 to 2806967 tid 3 2024-04-26T00:34:16.021 INFO:tasks.rados.rados.0.smithi045.stdout:2435: copy_from oid 27 from oid 12 current snap is 273 2024-04-26T00:34:16.022 INFO:tasks.rados.rados.0.smithi045.stdout:2428: finishing write tid 2 to smithi04526579-3 2024-04-26T00:34:16.022 INFO:tasks.rados.rados.0.smithi045.stdout:2428: finishing write tid 3 to smithi04526579-3 2024-04-26T00:34:16.022 INFO:tasks.rados.rados.0.smithi045.stdout:2428: finishing write tid 4 to smithi04526579-3 2024-04-26T00:34:16.022 INFO:tasks.rados.rados.0.smithi045.stdout:2428: oid 3 updating version 0 to 1980 2024-04-26T00:34:16.022 INFO:tasks.rados.rados.0.smithi045.stdout:2428: oid 3 version 1980 is already newer than 1979 2024-04-26T00:34:16.022 INFO:tasks.rados.rados.0.smithi045.stdout:2428: oid 3 version 1980 is already newer than 1978 2024-04-26T00:34:16.022 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 1980 (ObjNum 878 snap 273 seq_num 878) dirty exists 2024-04-26T00:34:16.022 INFO:tasks.rados.rados.0.smithi045.stdout:2428: left oid 3 (ObjNum 878 snap 273 seq_num 878) 2024-04-26T00:34:16.022 INFO:tasks.rados.rados.0.smithi045.stdout:2430: finishing write tid 1 to smithi04526579-39 2024-04-26T00:34:16.022 INFO:tasks.rados.rados.0.smithi045.stdout:2430: finishing write tid 2 to smithi04526579-39 2024-04-26T00:34:16.022 INFO:tasks.rados.rados.0.smithi045.stdout:2430: finishing write tid 3 to smithi04526579-39 2024-04-26T00:34:16.022 INFO:tasks.rados.rados.0.smithi045.stdout:2428: done (6 left) 2024-04-26T00:34:16.022 INFO:tasks.rados.rados.0.smithi045.stdout:2436: snap_create 2024-04-26T00:34:16.023 INFO:tasks.rados.rados.0.smithi045.stdout:2430: finishing write tid 4 to smithi04526579-39 2024-04-26T00:34:16.023 INFO:tasks.rados.rados.0.smithi045.stdout:2430: oid 39 updating version 0 to 1541 2024-04-26T00:34:16.023 INFO:tasks.rados.rados.0.smithi045.stdout:2430: oid 39 version 1541 is already newer than 1540 2024-04-26T00:34:16.023 INFO:tasks.rados.rados.0.smithi045.stdout:2430: oid 39 updating version 1541 to 1542 2024-04-26T00:34:16.023 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 1542 (ObjNum 879 snap 273 seq_num 879) dirty exists 2024-04-26T00:34:16.023 INFO:tasks.rados.rados.0.smithi045.stdout:2430: left oid 39 (ObjNum 879 snap 273 seq_num 879) 2024-04-26T00:34:16.023 INFO:tasks.rados.rados.0.smithi045.stdout:2433: finishing rollback tid 1 to smithi04526579-26 2024-04-26T00:34:16.023 INFO:tasks.rados.rados.0.smithi045.stdout:2433: finishing rollback tid 2 to smithi04526579-26 2024-04-26T00:34:16.023 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 1928 (ObjNum 731 snap 224 seq_num 731) dirty exists 2024-04-26T00:34:16.023 INFO:tasks.rados.rados.0.smithi045.stdout:2431: finishing write tid 1 to smithi04526579-42 2024-04-26T00:34:16.023 INFO:tasks.rados.rados.0.smithi045.stdout:2431: finishing write tid 2 to smithi04526579-42 2024-04-26T00:34:16.023 INFO:tasks.rados.rados.0.smithi045.stdout:2431: finishing write tid 3 to smithi04526579-42 2024-04-26T00:34:16.024 INFO:tasks.rados.rados.0.smithi045.stdout:2431: finishing write tid 4 to smithi04526579-42 2024-04-26T00:34:16.024 INFO:tasks.rados.rados.0.smithi045.stdout:2431: oid 42 updating version 0 to 1718 2024-04-26T00:34:16.024 INFO:tasks.rados.rados.0.smithi045.stdout:2431: oid 42 version 1718 is already newer than 1717 2024-04-26T00:34:16.024 INFO:tasks.rados.rados.0.smithi045.stdout:2431: oid 42 updating version 1718 to 1719 2024-04-26T00:34:16.024 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 1719 (ObjNum 880 snap 273 seq_num 880) dirty exists 2024-04-26T00:34:16.024 INFO:tasks.rados.rados.0.smithi045.stdout:2431: left oid 42 (ObjNum 880 snap 273 seq_num 880) 2024-04-26T00:34:16.024 INFO:tasks.rados.rados.0.smithi045.stdout:2435: finishing copy_from to smithi04526579-27 2024-04-26T00:34:16.025 INFO:tasks.rados.rados.0.smithi045.stdout:2435: got expected ENOENT (src dne) 2024-04-26T00:34:16.025 INFO:tasks.rados.rados.0.smithi045.stdout:2434: finishing write tid 1 to smithi04526579-4 2024-04-26T00:34:16.025 INFO:tasks.rados.rados.0.smithi045.stdout:2435: finishing copy_from racing read to smithi04526579-27 2024-04-26T00:34:16.028 INFO:tasks.rados.rados.0.smithi045.stdout:2434: finishing write tid 2 to smithi04526579-4 2024-04-26T00:34:16.029 INFO:tasks.rados.rados.0.smithi045.stdout:2434: finishing write tid 3 to smithi04526579-4 2024-04-26T00:34:16.029 INFO:tasks.rados.rados.0.smithi045.stdout:2434: finishing write tid 4 to smithi04526579-4 2024-04-26T00:34:16.030 INFO:tasks.rados.rados.0.smithi045.stdout:2434: finishing write tid 5 to smithi04526579-4 2024-04-26T00:34:16.030 INFO:tasks.rados.rados.0.smithi045.stdout:2434: oid 4 updating version 0 to 1926 2024-04-26T00:34:16.030 INFO:tasks.rados.rados.0.smithi045.stdout:2434: oid 4 version 1926 is already newer than 1925 2024-04-26T00:34:16.030 INFO:tasks.rados.rados.0.smithi045.stdout:2434: oid 4 version 1926 is already newer than 1924 2024-04-26T00:34:16.030 INFO:tasks.rados.rados.0.smithi045.stdout:2434: oid 4 version 1926 is already newer than 1923 2024-04-26T00:34:16.030 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 1926 (ObjNum 881 snap 273 seq_num 881) dirty exists 2024-04-26T00:34:16.030 INFO:tasks.rados.rados.0.smithi045.stdout:2434: left oid 4 (ObjNum 881 snap 273 seq_num 881) 2024-04-26T00:34:16.916 INFO:tasks.rados.rados.0.smithi045.stdout:2430: done (6 left) 2024-04-26T00:34:16.917 INFO:tasks.rados.rados.0.smithi045.stdout:2431: done (5 left) 2024-04-26T00:34:16.917 INFO:tasks.rados.rados.0.smithi045.stdout:2432: done (4 left) 2024-04-26T00:34:16.917 INFO:tasks.rados.rados.0.smithi045.stdout:2433: done (3 left) 2024-04-26T00:34:16.917 INFO:tasks.rados.rados.0.smithi045.stdout:2434: done (2 left) 2024-04-26T00:34:16.917 INFO:tasks.rados.rados.0.smithi045.stdout:2435: done (1 left) 2024-04-26T00:34:16.917 INFO:tasks.rados.rados.0.smithi045.stdout:2436: done (0 left) 2024-04-26T00:34:16.917 INFO:tasks.rados.rados.0.smithi045.stdout:2437: write (excl) oid 6 current snap is 274 2024-04-26T00:34:16.917 INFO:tasks.rados.rados.0.smithi045.stdout:2437: seq_num 882 ranges {423774=666566,1545824=512968,2704609=119165} 2024-04-26T00:34:16.926 INFO:tasks.rados.rados.0.smithi045.stdout:2437: writing smithi04526579-6 from 423774 to 1090340 tid 1 2024-04-26T00:34:16.936 INFO:tasks.rados.rados.0.smithi045.stdout:2437: writing smithi04526579-6 from 1545824 to 2058792 tid 2 2024-04-26T00:34:16.944 INFO:tasks.rados.rados.0.smithi045.stdout:2437: writing smithi04526579-6 from 2704609 to 2823774 tid 3 2024-04-26T00:34:16.944 INFO:tasks.rados.rados.0.smithi045.stdout:2438: snap_remove snap 243 2024-04-26T00:34:17.288 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-04-26T00:34:17.288 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-04-26T00:34:17.288 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-26T00:34:17.288 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-26T00:34:17.291 INFO:tasks.ceph.mon.b:Started 2024-04-26T00:34:17.291 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:34:17.291 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:34:17.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f615f131640 10 monclient: get_monmap_and_config 2024-04-26T00:34:17.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f615f131640 10 monclient: build_initial_monmap 2024-04-26T00:34:17.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f615f131640 10 monclient: monmap: 2024-04-26T00:34:17.422 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:34:17.422 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:17.422 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:34:17.429101+0000 2024-04-26T00:34:17.422 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:34:17.429101+0000 2024-04-26T00:34:17.422 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:34:17.422 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:17.423 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:34:17.423 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:34:17.423 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:34:17.423 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:34:17.423 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:34:17.423 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:17.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f615f131640 1 Processor -- start 2024-04-26T00:34:17.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f615f131640 1 -- start start 2024-04-26T00:34:17.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f615f131640 10 monclient: init 2024-04-26T00:34:17.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f615f131640 10 monclient: _reopen_session rank -1 2024-04-26T00:34:17.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f615f131640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-04-26T00:34:17.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f615f131640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6158136da0 0x7f615809bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:17.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f615f131640 10 monclient(hunting): picked mon.noname-a con 0x7f6158136da0 addr v2:172.21.15.37:3300/0 2024-04-26T00:34:17.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f615f131640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f615809c410 0x7f6158137180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:17.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f615f131640 10 monclient(hunting): picked mon.noname-c con 0x7f615809c410 addr v2:172.21.15.45:3300/0 2024-04-26T00:34:17.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f615f131640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f61581376c0 addr v1:172.21.15.37:6789/0 2024-04-26T00:34:17.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f615f131640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f615809b4f0 con 0x7f61581376c0 2024-04-26T00:34:17.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f615f131640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:17.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f615f131640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f615809b210 con 0x7f6158136da0 2024-04-26T00:34:17.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f615f131640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:17.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f615f131640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f615809b7d0 con 0x7f615809c410 2024-04-26T00:34:17.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f615f131640 10 monclient(hunting): _renew_subs 2024-04-26T00:34:17.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f615f131640 10 monclient(hunting): authenticate will time out at 1838.735046s 2024-04-26T00:34:17.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f615cecd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6158136da0 0x7f615809bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:17.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f615d6ce640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f61581376c0 0x7f6158133b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:58324/0 (socket says 172.21.15.37:58324) 2024-04-26T00:34:17.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f615d6ce640 1 -- 172.21.15.37:0/723104578 learned_addr learned my addr 172.21.15.37:0/723104578 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:34:17.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f615cecd640 10 monclient(hunting): get_auth_request con 0x7f6158136da0 auth_method 0 2024-04-26T00:34:17.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f615cecd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:17.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f615cecd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:17.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f615cecd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:17.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f615cecd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:17.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f615cecd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:17.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f615cecd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:17.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f6157fff640 1 --2- 172.21.15.37:0/723104578 >> v2:172.21.15.45:3300/0 conn(0x7f615809c410 0x7f6158137180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:17.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f6157fff640 10 monclient(hunting): get_auth_request con 0x7f615809c410 auth_method 0 2024-04-26T00:34:17.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f6157fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:17.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f6157fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:17.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f6157fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:17.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f615cecd640 10 monclient(hunting): handle_auth_done global_id 144381 payload 274 2024-04-26T00:34:17.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f615cecd640 1 -- 172.21.15.37:0/723104578 >> v2:172.21.15.45:3300/0 conn(0x7f615809c410 msgr2=0x7f6158137180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:34:17.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f615cecd640 1 --2- 172.21.15.37:0/723104578 >> v2:172.21.15.45:3300/0 conn(0x7f615809c410 0x7f6158137180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:17.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f615cecd640 1 -- 172.21.15.37:0/723104578 >> v1:172.21.15.37:6789/0 conn(0x7f61581376c0 legacy=0x7f6158133b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:34:17.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f615cecd640 10 monclient: _finish_hunting 0 2024-04-26T00:34:17.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f615cecd640 1 monclient: found mon.noname-a 2024-04-26T00:34:17.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f615cecd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:34:17.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f615cecd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:34:17.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.426+0000 7f615cecd640 1 -- 172.21.15.37:0/723104578 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6158137b30 con 0x7f6158136da0 2024-04-26T00:34:17.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.430+0000 7f615cecd640 1 -- 172.21.15.37:0/723104578 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6158136da0 msgr2=0x7f615809bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:17.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.430+0000 7f615cecd640 1 --2- 172.21.15.37:0/723104578 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6158136da0 0x7f615809bed0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f61580a05c0 tx=0x7f614801c390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=748d1b6865d22581 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:17.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f61577fe640 1 -- 172.21.15.37:0/723104578 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61480b0070 con 0x7f6158136da0 2024-04-26T00:34:17.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f61577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:17.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f61577fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:34:17.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f61577fe640 10 monclient: dump: 2024-04-26T00:34:17.429 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:17.429 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:17.430 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:17.430 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:17.430 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:17.430 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:17.430 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:17.430 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:17.430 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:17.430 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:17.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f61577fe640 10 monclient: _finish_auth 0 2024-04-26T00:34:17.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f61577fe640 10 monclient: _check_auth_tickets 2024-04-26T00:34:17.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f61577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:34:17.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f61577fe640 1 -- 172.21.15.37:0/723104578 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61480ab600 con 0x7f6158136da0 2024-04-26T00:34:17.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f61577fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:34:17.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f61577fe640 1 -- 172.21.15.37:0/723104578 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61480ab9d0 con 0x7f6158136da0 2024-04-26T00:34:17.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615f131640 5 monclient: authenticate success, global_id 144381 2024-04-26T00:34:17.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615f131640 10 monclient: get_monmap_and_config success 2024-04-26T00:34:17.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f61577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:17.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615f131640 4 set_mon_vals no callback set 2024-04-26T00:34:17.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f61577fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:34:17.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f61577fe640 10 monclient: dump: 2024-04-26T00:34:17.432 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:17.432 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:17.432 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:17.432 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:17.432 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:17.432 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:17.432 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:17.432 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:17.432 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:17.432 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:17.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615f131640 10 monclient: shutdown 2024-04-26T00:34:17.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615f131640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:34:17.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615f131640 1 -- 172.21.15.37:0/723104578 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6158136da0 msgr2=0x7f615809bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:17.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615f131640 1 --2- 172.21.15.37:0/723104578 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6158136da0 0x7f615809bed0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f61580a05c0 tx=0x7f614801c390 comp rx=0 tx=0).stop 2024-04-26T00:34:17.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615f131640 1 -- 172.21.15.37:0/723104578 shutdown_connections 2024-04-26T00:34:17.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615f131640 1 --2- 172.21.15.37:0/723104578 >> v2:172.21.15.45:3300/0 conn(0x7f615809c410 0x7f6158137180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:17.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615f131640 1 --2- 172.21.15.37:0/723104578 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6158136da0 0x7f615809bed0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:17.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615f131640 1 -- 172.21.15.37:0/723104578 >> 172.21.15.37:0/723104578 conn(0x7f61580a5f00 msgr2=0x7f61580a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:34:17.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615f131640 1 -- 172.21.15.37:0/723104578 shutdown_connections 2024-04-26T00:34:17.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615f131640 1 -- 172.21.15.37:0/723104578 wait complete. 2024-04-26T00:34:17.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615f131640 10 monclient: build_initial_monmap 2024-04-26T00:34:17.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615f131640 10 monclient: monmap: 2024-04-26T00:34:17.433 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:34:17.434 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:34:17.434 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:34:17.434 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:34:17.434 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:34:17.434 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:34:17.434 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:34:17.434 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:34:17.434 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:34:17.434 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:17.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615f131640 1 Processor -- start 2024-04-26T00:34:17.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615f131640 1 -- start start 2024-04-26T00:34:17.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615f131640 10 monclient: init 2024-04-26T00:34:17.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615f131640 10 monclient: _reopen_session rank -1 2024-04-26T00:34:17.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615f131640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:34:17.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615f131640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f615809c410 0x7f61581cf460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:17.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615f131640 10 monclient(hunting): picked mon.noname-b con 0x7f615809c410 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:34:17.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615f131640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6158136da0 0x7f61581cf9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:17.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615f131640 10 monclient(hunting): picked mon.noname-c con 0x7f6158136da0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:34:17.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615f131640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f61581376c0 0x7f61581cfee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:17.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615f131640 10 monclient(hunting): picked mon.noname-a con 0x7f61581376c0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:34:17.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615f131640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:17.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615f131640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f6158141ea0 con 0x7f61581376c0 2024-04-26T00:34:17.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615f131640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:17.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615f131640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6158141d20 con 0x7f615809c410 2024-04-26T00:34:17.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615f131640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:17.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615f131640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f6158142020 con 0x7f6158136da0 2024-04-26T00:34:17.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615f131640 10 monclient(hunting): _renew_subs 2024-04-26T00:34:17.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615f131640 10 monclient(hunting): authenticate will time out at 1838.742514s 2024-04-26T00:34:17.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615d6ce640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f61581376c0 0x7f61581cfee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:17.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615d6ce640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f61581376c0 0x7f61581cfee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:59678/0 (socket says 172.21.15.37:59678) 2024-04-26T00:34:17.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f6157fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6158136da0 0x7f61581cf9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:17.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615d6ce640 1 -- 172.21.15.37:0/2825268061 learned_addr learned my addr 172.21.15.37:0/2825268061 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:34:17.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615d6ce640 10 monclient(hunting): get_auth_request con 0x7f61581376c0 auth_method 0 2024-04-26T00:34:17.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615d6ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:17.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615d6ce640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:17.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615d6ce640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:17.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f6157fff640 10 monclient(hunting): get_auth_request con 0x7f6158136da0 auth_method 0 2024-04-26T00:34:17.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f6157fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:17.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f6157fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:17.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f6157fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:17.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615d6ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:17.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615d6ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:17.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615d6ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:17.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f6157fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:17.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f6157fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:17.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615cecd640 1 --2- 172.21.15.37:0/2825268061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f615809c410 0x7f61581cf460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:17.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f6157fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:17.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615cecd640 10 monclient(hunting): get_auth_request con 0x7f615809c410 auth_method 0 2024-04-26T00:34:17.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615cecd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:17.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615cecd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:17.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615cecd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:17.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f6157fff640 10 monclient(hunting): handle_auth_done global_id 154256 payload 824 2024-04-26T00:34:17.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615cecd640 1 -- 172.21.15.37:0/2825268061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f615809c410 msgr2=0x7f61581cf460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:34:17.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615cecd640 1 -- 172.21.15.37:0/2825268061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f615809c410 msgr2=0x7f61581cf460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:34:17.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615cecd640 1 --2- 172.21.15.37:0/2825268061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f615809c410 0x7f61581cf460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:34:17.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f6157fff640 1 -- 172.21.15.37:0/2825268061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f615809c410 msgr2=0x7f61581cf460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:17.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615cecd640 1 --2- 172.21.15.37:0/2825268061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f615809c410 0x7f61581cf460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:34:17.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f6157fff640 1 --2- 172.21.15.37:0/2825268061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f615809c410 0x7f61581cf460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:17.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f6157fff640 1 -- 172.21.15.37:0/2825268061 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f61581376c0 msgr2=0x7f61581cfee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:17.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f6157fff640 1 --2- 172.21.15.37:0/2825268061 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f61581376c0 0x7f61581cfee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:17.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f6157fff640 10 monclient: _finish_hunting 0 2024-04-26T00:34:17.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f6157fff640 1 monclient: found mon.noname-c 2024-04-26T00:34:17.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f6157fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:34:17.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f6157fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:34:17.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f6157fff640 1 -- 172.21.15.37:0/2825268061 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f615809e770 con 0x7f6158136da0 2024-04-26T00:34:17.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.434+0000 7f615d6ce640 1 --2- 172.21.15.37:0/2825268061 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f61581376c0 0x7f61581cfee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:34:17.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.438+0000 7f6157fff640 1 -- 172.21.15.37:0/2825268061 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6158136da0 msgr2=0x7f61581cf9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:17.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.438+0000 7f6157fff640 1 --2- 172.21.15.37:0/2825268061 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6158136da0 0x7f61581cf9a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6140007e60 tx=0x7f614000e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:17.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.438+0000 7f61557fa640 1 -- 172.21.15.37:0/2825268061 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6140013790 con 0x7f6158136da0 2024-04-26T00:34:17.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.438+0000 7f61557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:17.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.438+0000 7f61557fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:34:17.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.438+0000 7f61557fa640 10 monclient: dump: 2024-04-26T00:34:17.444 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:17.444 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:17.444 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:17.444 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:17.445 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:17.445 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:17.445 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:17.445 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:17.445 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:17.445 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:17.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.438+0000 7f61557fa640 10 monclient: _finish_auth 0 2024-04-26T00:34:17.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.438+0000 7f61557fa640 10 monclient: _check_auth_tickets 2024-04-26T00:34:17.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.438+0000 7f61557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:34:17.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.438+0000 7f61557fa640 1 -- 172.21.15.37:0/2825268061 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6140019070 con 0x7f6158136da0 2024-04-26T00:34:17.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.438+0000 7f61557fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:34:17.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.438+0000 7f61557fa640 1 -- 172.21.15.37:0/2825268061 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6140013b50 con 0x7f6158136da0 2024-04-26T00:34:17.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.438+0000 7f615f131640 5 monclient: authenticate success, global_id 154256 2024-04-26T00:34:17.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.438+0000 7f615f131640 10 monclient: _renew_subs 2024-04-26T00:34:17.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.438+0000 7f615f131640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:34:17.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.438+0000 7f615f131640 1 -- 172.21.15.37:0/2825268061 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f615809ea50 con 0x7f6158136da0 2024-04-26T00:34:17.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.438+0000 7f6156ffd640 4 set_mon_vals no callback set 2024-04-26T00:34:17.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.438+0000 7f61557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:17.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.438+0000 7f61557fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:34:17.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.438+0000 7f61557fa640 10 monclient: dump: 2024-04-26T00:34:17.446 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:17.446 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:17.446 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:17.446 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:17.446 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:17.446 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:17.446 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:17.447 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:17.447 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:17.447 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:17.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.438+0000 7f615f131640 10 monclient: _renew_subs 2024-04-26T00:34:17.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.438+0000 7f615f131640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:34:17.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.438+0000 7f615f131640 1 -- 172.21.15.37:0/2825268061 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f61580a36d0 con 0x7f6158136da0 2024-04-26T00:34:17.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.438+0000 7f615f131640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:34:17.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.438+0000 7f615f131640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:34:17.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.438+0000 7f615f131640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:34:17.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.438+0000 7f615f131640 1 -- 172.21.15.37:0/2825268061 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6158066f40 con 0x7f6158136da0 2024-04-26T00:34:17.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.438+0000 7f61557fa640 1 -- 172.21.15.37:0/2825268061 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f614001e460 con 0x7f6158136da0 2024-04-26T00:34:17.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.438+0000 7f61557fa640 1 --2- 172.21.15.37:0/2825268061 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6138035ed0 0x7f6138038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:17.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.438+0000 7f61557fa640 1 -- 172.21.15.37:0/2825268061 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(557..557 src has 370..557) ==== 3963+0+0 (secure 0 0 0) 0x7f6140049310 con 0x7f6158136da0 2024-04-26T00:34:17.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.438+0000 7f615cecd640 1 --2- 172.21.15.37:0/2825268061 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6138035ed0 0x7f6138038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:17.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.438+0000 7f615cecd640 10 monclient: get_auth_request con 0x7f6138035ed0 auth_method 0 2024-04-26T00:34:17.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.442+0000 7f615cecd640 1 --2- 172.21.15.37:0/2825268061 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6138035ed0 0x7f6138038390 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f6158076070 tx=0x7f61480a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:17.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.446+0000 7f61557fa640 1 -- 172.21.15.37:0/2825268061 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f614001c520 con 0x7f6158136da0 2024-04-26T00:34:17.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.446+0000 7f61557fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:34:17.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.446+0000 7f61557fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:34:17.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.666+0000 7f615f131640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:34:17.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.666+0000 7f615f131640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:34:17.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.666+0000 7f615f131640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:34:17.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.666+0000 7f615f131640 1 -- 172.21.15.37:0/2825268061 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f61580a8cf0 con 0x7f6158136da0 2024-04-26T00:34:17.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.666+0000 7f61557fa640 1 -- 172.21.15.37:0/2825268061 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7f61400490e0 con 0x7f6158136da0 2024-04-26T00:34:17.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.666+0000 7f61557fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:34:17.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.666+0000 7f61557fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:34:17.663 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":106,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:34:17.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.670+0000 7f615f131640 1 -- 172.21.15.37:0/2825268061 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6138035ed0 msgr2=0x7f6138038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:17.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.670+0000 7f615f131640 1 --2- 172.21.15.37:0/2825268061 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6138035ed0 0x7f6138038390 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f6158076070 tx=0x7f61480a6000 comp rx=0 tx=0).stop 2024-04-26T00:34:17.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.670+0000 7f615f131640 10 monclient: shutdown 2024-04-26T00:34:17.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.670+0000 7f615f131640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:34:17.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.670+0000 7f615f131640 1 -- 172.21.15.37:0/2825268061 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6158136da0 msgr2=0x7f61581cf9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:17.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.670+0000 7f615f131640 1 --2- 172.21.15.37:0/2825268061 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6158136da0 0x7f61581cf9a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6140007e60 tx=0x7f614000e4b0 comp rx=0 tx=0).stop 2024-04-26T00:34:17.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.670+0000 7f615f131640 1 -- 172.21.15.37:0/2825268061 shutdown_connections 2024-04-26T00:34:17.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.670+0000 7f615f131640 1 --2- 172.21.15.37:0/2825268061 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6138035ed0 0x7f6138038390 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:17.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.670+0000 7f615f131640 1 --2- 172.21.15.37:0/2825268061 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f61581376c0 0x7f61581cfee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:17.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.670+0000 7f615f131640 1 --2- 172.21.15.37:0/2825268061 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6158136da0 0x7f61581cf9a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:17.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.670+0000 7f615f131640 1 --2- 172.21.15.37:0/2825268061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f615809c410 0x7f61581cf460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:17.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.670+0000 7f615f131640 1 -- 172.21.15.37:0/2825268061 >> 172.21.15.37:0/2825268061 conn(0x7f61580a5f00 msgr2=0x7f615812ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:34:17.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.670+0000 7f615f131640 1 -- 172.21.15.37:0/2825268061 shutdown_connections 2024-04-26T00:34:17.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:17.670+0000 7f615f131640 1 -- 172.21.15.37:0/2825268061 wait complete. 2024-04-26T00:34:17.927 INFO:tasks.rados.rados.0.smithi045.stdout:2438: done (1 left) 2024-04-26T00:34:17.927 INFO:tasks.rados.rados.0.smithi045.stdout:2439: write oid 46 current snap is 274 2024-04-26T00:34:17.928 INFO:tasks.rados.rados.0.smithi045.stdout:2437: finishing write tid 1 to smithi04526579-6 2024-04-26T00:34:17.928 INFO:tasks.rados.rados.0.smithi045.stdout:2439: seq_num 883 ranges {440581=741815,1604306=656626,2732032=108549} 2024-04-26T00:34:17.944 INFO:tasks.rados.rados.0.smithi045.stdout:2439: writing smithi04526579-46 from 440581 to 1182396 tid 1 2024-04-26T00:34:17.960 INFO:tasks.rados.rados.0.smithi045.stdout:2439: writing smithi04526579-46 from 1604306 to 2260932 tid 2 2024-04-26T00:34:17.969 INFO:tasks.rados.rados.0.smithi045.stdout:2439: writing smithi04526579-46 from 2732032 to 2840581 tid 3 2024-04-26T00:34:17.969 INFO:tasks.rados.rados.0.smithi045.stdout:2440: write oid 8 current snap is 274 2024-04-26T00:34:17.969 INFO:tasks.rados.rados.0.smithi045.stdout:2440: seq_num 884 ranges {457388=417064,1546435=516637,2643102=214286} 2024-04-26T00:34:17.983 INFO:tasks.rados.rados.0.smithi045.stdout:2440: writing smithi04526579-8 from 457388 to 874452 tid 1 2024-04-26T00:34:17.996 INFO:tasks.rados.rados.0.smithi045.stdout:2440: writing smithi04526579-8 from 1546435 to 2063072 tid 2 2024-04-26T00:34:18.005 INFO:tasks.rados.rados.0.smithi045.stdout:2440: writing smithi04526579-8 from 2643102 to 2857388 tid 3 2024-04-26T00:34:18.005 INFO:tasks.rados.rados.0.smithi045.stdout:2441: write oid 30 current snap is 274 2024-04-26T00:34:18.005 INFO:tasks.rados.rados.0.smithi045.stdout:2441: seq_num 885 ranges {474195=492313,1604917=660295,2670525=203670} 2024-04-26T00:34:18.013 INFO:tasks.rados.rados.0.smithi045.stdout:2441: writing smithi04526579-30 from 474195 to 966508 tid 1 2024-04-26T00:34:18.022 INFO:tasks.rados.rados.0.smithi045.stdout:2441: writing smithi04526579-30 from 1604917 to 2265212 tid 2 2024-04-26T00:34:18.027 INFO:tasks.rados.rados.0.smithi045.stdout:2441: writing smithi04526579-30 from 2670525 to 2874195 tid 3 2024-04-26T00:34:18.027 INFO:tasks.rados.rados.0.smithi045.stdout:2442: write oid 40 current snap is 274 2024-04-26T00:34:18.027 INFO:tasks.rados.rados.0.smithi045.stdout:2437: finishing write tid 2 to smithi04526579-6 2024-04-26T00:34:18.027 INFO:tasks.rados.rados.0.smithi045.stdout:2442: seq_num 886 ranges {491002=567562,1663399=520306,2814301=76701} 2024-04-26T00:34:18.035 INFO:tasks.rados.rados.0.smithi045.stdout:2442: writing smithi04526579-40 from 491002 to 1058564 tid 1 2024-04-26T00:34:18.043 INFO:tasks.rados.rados.0.smithi045.stdout:2442: writing smithi04526579-40 from 1663399 to 2183705 tid 2 2024-04-26T00:34:18.048 INFO:tasks.rados.rados.0.smithi045.stdout:2442: writing smithi04526579-40 from 2814301 to 2891002 tid 3 2024-04-26T00:34:18.048 INFO:tasks.rados.rados.0.smithi045.stdout:2443: snap_remove snap 264 2024-04-26T00:34:18.510 INFO:tasks.rados.rados.0.smithi045.stdout:2443: done (5 left) 2024-04-26T00:34:18.510 INFO:tasks.rados.rados.0.smithi045.stdout:2444: delete oid 17 current snap is 274 2024-04-26T00:34:18.510 INFO:tasks.rados.rados.0.smithi045.stdout:2437: finishing write tid 3 to smithi04526579-6 2024-04-26T00:34:18.510 INFO:tasks.rados.rados.0.smithi045.stdout:2437: finishing write tid 4 to smithi04526579-6 2024-04-26T00:34:18.511 INFO:tasks.rados.rados.0.smithi045.stdout:2437: finishing write tid 5 to smithi04526579-6 2024-04-26T00:34:18.511 INFO:tasks.rados.rados.0.smithi045.stdout:2437: oid 6 updating version 0 to 1068 2024-04-26T00:34:18.511 INFO:tasks.rados.rados.0.smithi045.stdout:2437: oid 6 version 1068 is already newer than 1066 2024-04-26T00:34:18.511 INFO:tasks.rados.rados.0.smithi045.stdout:2437: oid 6 version 1068 is already newer than 1067 2024-04-26T00:34:18.511 INFO:tasks.rados.rados.0.smithi045.stdout:2437: oid 6 updating version 1068 to 1069 2024-04-26T00:34:18.511 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 1069 (ObjNum 882 snap 274 seq_num 882) dirty exists 2024-04-26T00:34:18.511 INFO:tasks.rados.rados.0.smithi045.stdout:2437: left oid 6 (ObjNum 882 snap 274 seq_num 882) 2024-04-26T00:34:18.511 INFO:tasks.rados.rados.0.smithi045.stdout:2439: finishing write tid 1 to smithi04526579-46 2024-04-26T00:34:18.511 INFO:tasks.rados.rados.0.smithi045.stdout:2439: finishing write tid 2 to smithi04526579-46 2024-04-26T00:34:18.511 INFO:tasks.rados.rados.0.smithi045.stdout:2439: finishing write tid 3 to smithi04526579-46 2024-04-26T00:34:18.511 INFO:tasks.rados.rados.0.smithi045.stdout:2439: finishing write tid 4 to smithi04526579-46 2024-04-26T00:34:18.512 INFO:tasks.rados.rados.0.smithi045.stdout:2439: finishing write tid 5 to smithi04526579-46 2024-04-26T00:34:18.512 INFO:tasks.rados.rados.0.smithi045.stdout:2439: oid 46 updating version 0 to 1468 2024-04-26T00:34:18.512 INFO:tasks.rados.rados.0.smithi045.stdout:2439: oid 46 version 1468 is already newer than 1467 2024-04-26T00:34:18.512 INFO:tasks.rados.rados.0.smithi045.stdout:2439: oid 46 updating version 1468 to 1469 2024-04-26T00:34:18.512 INFO:tasks.rados.rados.0.smithi045.stdout:2439: oid 46 version 1469 is already newer than 1466 2024-04-26T00:34:18.512 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 1469 (ObjNum 883 snap 274 seq_num 883) dirty exists 2024-04-26T00:34:18.512 INFO:tasks.rados.rados.0.smithi045.stdout:2439: left oid 46 (ObjNum 883 snap 274 seq_num 883) 2024-04-26T00:34:18.512 INFO:tasks.rados.rados.0.smithi045.stdout:2440: finishing write tid 1 to smithi04526579-8 2024-04-26T00:34:18.512 INFO:tasks.rados.rados.0.smithi045.stdout:2440: finishing write tid 2 to smithi04526579-8 2024-04-26T00:34:18.512 INFO:tasks.rados.rados.0.smithi045.stdout:2440: finishing write tid 3 to smithi04526579-8 2024-04-26T00:34:18.512 INFO:tasks.rados.rados.0.smithi045.stdout:2440: finishing write tid 4 to smithi04526579-8 2024-04-26T00:34:18.512 INFO:tasks.rados.rados.0.smithi045.stdout:2440: finishing write tid 5 to smithi04526579-8 2024-04-26T00:34:18.512 INFO:tasks.rados.rados.0.smithi045.stdout:2440: oid 8 updating version 0 to 1546 2024-04-26T00:34:18.512 INFO:tasks.rados.rados.0.smithi045.stdout:2440: oid 8 updating version 1546 to 1547 2024-04-26T00:34:18.513 INFO:tasks.rados.rados.0.smithi045.stdout:2440: oid 8 updating version 1547 to 1548 2024-04-26T00:34:18.513 INFO:tasks.rados.rados.0.smithi045.stdout:2440: oid 8 updating version 1548 to 1549 2024-04-26T00:34:18.513 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 1549 (ObjNum 884 snap 274 seq_num 884) dirty exists 2024-04-26T00:34:18.513 INFO:tasks.rados.rados.0.smithi045.stdout:2440: left oid 8 (ObjNum 884 snap 274 seq_num 884) 2024-04-26T00:34:18.513 INFO:tasks.rados.rados.0.smithi045.stdout:2441: finishing write tid 1 to smithi04526579-30 2024-04-26T00:34:18.513 INFO:tasks.rados.rados.0.smithi045.stdout:2441: finishing write tid 2 to smithi04526579-30 2024-04-26T00:34:18.513 INFO:tasks.rados.rados.0.smithi045.stdout:2441: finishing write tid 3 to smithi04526579-30 2024-04-26T00:34:18.513 INFO:tasks.rados.rados.0.smithi045.stdout:2441: finishing write tid 4 to smithi04526579-30 2024-04-26T00:34:18.513 INFO:tasks.rados.rados.0.smithi045.stdout:2441: finishing write tid 5 to smithi04526579-30 2024-04-26T00:34:18.513 INFO:tasks.rados.rados.0.smithi045.stdout:2441: oid 30 updating version 0 to 1072 2024-04-26T00:34:18.513 INFO:tasks.rados.rados.0.smithi045.stdout:2441: oid 30 version 1072 is already newer than 1071 2024-04-26T00:34:18.513 INFO:tasks.rados.rados.0.smithi045.stdout:2441: oid 30 updating version 1072 to 1073 2024-04-26T00:34:18.513 INFO:tasks.rados.rados.0.smithi045.stdout:2441: oid 30 version 1073 is already newer than 1070 2024-04-26T00:34:18.514 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 1073 (ObjNum 885 snap 274 seq_num 885) dirty exists 2024-04-26T00:34:18.514 INFO:tasks.rados.rados.0.smithi045.stdout:2441: left oid 30 (ObjNum 885 snap 274 seq_num 885) 2024-04-26T00:34:18.514 INFO:tasks.rados.rados.0.smithi045.stdout:2442: finishing write tid 1 to smithi04526579-40 2024-04-26T00:34:18.514 INFO:tasks.rados.rados.0.smithi045.stdout:2442: finishing write tid 2 to smithi04526579-40 2024-04-26T00:34:18.514 INFO:tasks.rados.rados.0.smithi045.stdout:2442: finishing write tid 3 to smithi04526579-40 2024-04-26T00:34:18.514 INFO:tasks.rados.rados.0.smithi045.stdout:2442: finishing write tid 4 to smithi04526579-40 2024-04-26T00:34:18.514 INFO:tasks.rados.rados.0.smithi045.stdout:2442: finishing write tid 5 to smithi04526579-40 2024-04-26T00:34:18.514 INFO:tasks.rados.rados.0.smithi045.stdout:2442: oid 40 updating version 0 to 1474 2024-04-26T00:34:18.514 INFO:tasks.rados.rados.0.smithi045.stdout:2442: oid 40 version 1474 is already newer than 1473 2024-04-26T00:34:18.514 INFO:tasks.rados.rados.0.smithi045.stdout:2442: oid 40 version 1474 is already newer than 1471 2024-04-26T00:34:18.515 INFO:tasks.rados.rados.0.smithi045.stdout:2442: oid 40 version 1474 is already newer than 1472 2024-04-26T00:34:18.515 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 1474 (ObjNum 886 snap 274 seq_num 886) dirty exists 2024-04-26T00:34:18.515 INFO:tasks.rados.rados.0.smithi045.stdout:2442: left oid 40 (ObjNum 886 snap 274 seq_num 886) 2024-04-26T00:34:18.531 INFO:tasks.rados.rados.0.smithi045.stdout:2437: done (5 left) 2024-04-26T00:34:18.531 INFO:tasks.rados.rados.0.smithi045.stdout:2439: done (4 left) 2024-04-26T00:34:18.531 INFO:tasks.rados.rados.0.smithi045.stdout:2440: done (3 left) 2024-04-26T00:34:18.531 INFO:tasks.rados.rados.0.smithi045.stdout:2441: done (2 left) 2024-04-26T00:34:18.532 INFO:tasks.rados.rados.0.smithi045.stdout:2442: done (1 left) 2024-04-26T00:34:18.532 INFO:tasks.rados.rados.0.smithi045.stdout:2444: done (0 left) 2024-04-26T00:34:18.532 INFO:tasks.rados.rados.0.smithi045.stdout:2445: delete oid 9 current snap is 274 2024-04-26T00:34:18.543 INFO:tasks.rados.rados.0.smithi045.stdout:2445: done (0 left) 2024-04-26T00:34:18.543 INFO:tasks.rados.rados.0.smithi045.stdout:2446: snap_create 2024-04-26T00:34:19.514 INFO:tasks.rados.rados.0.smithi045.stdout:2446: done (0 left) 2024-04-26T00:34:19.514 INFO:tasks.rados.rados.0.smithi045.stdout:2447: snap_remove snap 269 2024-04-26T00:34:20.524 INFO:tasks.rados.rados.0.smithi045.stdout:2447: done (0 left) 2024-04-26T00:34:20.524 INFO:tasks.rados.rados.0.smithi045.stdout:2448: read oid 20 snap -1 2024-04-26T00:34:20.524 INFO:tasks.rados.rados.0.smithi045.stdout:2448: expect deleted 2024-04-26T00:34:20.524 INFO:tasks.rados.rados.0.smithi045.stdout:2449: write (excl) oid 33 current snap is 275 2024-04-26T00:34:20.524 INFO:tasks.rados.rados.0.smithi045.stdout:2449: seq_num 889 ranges {541423=793309,2122492=667633} 2024-04-26T00:34:20.548 INFO:tasks.rados.rados.0.smithi045.stdout:2449: writing smithi04526579-33 from 541423 to 1334732 tid 1 2024-04-26T00:34:20.573 INFO:tasks.rados.rados.0.smithi045.stdout:2449: writing smithi04526579-33 from 2122492 to 2790125 tid 2 2024-04-26T00:34:20.573 INFO:tasks.rados.rados.0.smithi045.stdout:2450: snap_remove snap 270 2024-04-26T00:34:20.682 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:34:20.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.818+0000 7fd9d49a1640 10 monclient: get_monmap_and_config 2024-04-26T00:34:20.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.818+0000 7fd9d49a1640 10 monclient: build_initial_monmap 2024-04-26T00:34:20.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.818+0000 7fd9d49a1640 10 monclient: monmap: 2024-04-26T00:34:20.814 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:34:20.815 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:20.815 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:34:20.821943+0000 2024-04-26T00:34:20.815 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:34:20.821943+0000 2024-04-26T00:34:20.815 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:34:20.815 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:20.815 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:34:20.815 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:34:20.815 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:34:20.815 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:34:20.815 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:34:20.815 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:20.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.818+0000 7fd9d49a1640 1 Processor -- start 2024-04-26T00:34:20.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.818+0000 7fd9d49a1640 1 -- start start 2024-04-26T00:34:20.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.818+0000 7fd9d49a1640 10 monclient: init 2024-04-26T00:34:20.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9d49a1640 10 monclient: _reopen_session rank -1 2024-04-26T00:34:20.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9d49a1640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-04-26T00:34:20.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9d49a1640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd9d0133b20 0x7fd9d009e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:20.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9d49a1640 10 monclient(hunting): picked mon.noname-a con 0x7fd9d0133b20 addr v2:172.21.15.37:3300/0 2024-04-26T00:34:20.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9d49a1640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd9d0133f00 0x7fd9d0134350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:20.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9d49a1640 10 monclient(hunting): picked mon.noname-b con 0x7fd9d0133f00 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:34:20.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9d49a1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd9d0130770 addr v1:172.21.15.45:6789/0 2024-04-26T00:34:20.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9d49a1640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9d009fe20 con 0x7fd9d0130770 2024-04-26T00:34:20.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9d49a1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:20.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9d49a1640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fd9d0072d00 con 0x7fd9d0133b20 2024-04-26T00:34:20.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9d49a1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:20.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9d49a1640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd9d009ba60 con 0x7fd9d0133f00 2024-04-26T00:34:20.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9d49a1640 10 monclient(hunting): _renew_subs 2024-04-26T00:34:20.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9d49a1640 10 monclient(hunting): authenticate will time out at 1842.128406s 2024-04-26T00:34:20.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9ce59c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd9d0133b20 0x7fd9d009e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:20.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9ce59c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd9d0133b20 0x7fd9d009e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33676/0 (socket says 172.21.15.37:33676) 2024-04-26T00:34:20.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9ce59c640 1 -- 172.21.15.37:0/2529680377 learned_addr learned my addr 172.21.15.37:0/2529680377 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:34:20.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9ce59c640 10 monclient(hunting): get_auth_request con 0x7fd9d0133b20 auth_method 0 2024-04-26T00:34:20.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9ce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:20.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9ce59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:20.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9c7fff640 1 --2- 172.21.15.37:0/2529680377 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd9d0133f00 0x7fd9d0134350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:20.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9ce59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:20.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9c7fff640 10 monclient(hunting): get_auth_request con 0x7fd9d0133f00 auth_method 0 2024-04-26T00:34:20.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:20.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9c7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:20.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:20.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9ce59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:20.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9ce59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:20.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9ce59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:20.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:20.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:20.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:20.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9ce59c640 10 monclient(hunting): handle_auth_done global_id 144387 payload 274 2024-04-26T00:34:20.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9ce59c640 1 -- 172.21.15.37:0/2529680377 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd9d0133f00 msgr2=0x7fd9d0134350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:34:20.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9ce59c640 1 --2- 172.21.15.37:0/2529680377 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd9d0133f00 0x7fd9d0134350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:20.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9ce59c640 1 -- 172.21.15.37:0/2529680377 >> v1:172.21.15.45:6789/0 conn(0x7fd9d0130770 legacy=0x7fd9d0130be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:34:20.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9ce59c640 10 monclient: _finish_hunting 0 2024-04-26T00:34:20.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9ce59c640 1 monclient: found mon.noname-a 2024-04-26T00:34:20.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9ce59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:34:20.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9ce59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:34:20.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.822+0000 7fd9ce59c640 1 -- 172.21.15.37:0/2529680377 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9d009dd10 con 0x7fd9d0133b20 2024-04-26T00:34:20.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9ce59c640 1 -- 172.21.15.37:0/2529680377 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd9d0133b20 msgr2=0x7fd9d009e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:20.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9ce59c640 1 --2- 172.21.15.37:0/2529680377 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd9d0133b20 0x7fd9d009e9d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd9c0009870 tx=0x7fd9c001c1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52b63926a7ec9411 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:20.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9cdd9b640 1 -- 172.21.15.37:0/2529680377 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c00b0070 con 0x7fd9d0133b20 2024-04-26T00:34:20.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9cdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:20.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9cdd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:34:20.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9cdd9b640 10 monclient: dump: 2024-04-26T00:34:20.823 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:20.823 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:20.823 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:20.824 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:20.824 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:20.824 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:20.824 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:20.824 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:20.824 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:20.824 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:20.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9cdd9b640 10 monclient: _finish_auth 0 2024-04-26T00:34:20.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9cdd9b640 10 monclient: _check_auth_tickets 2024-04-26T00:34:20.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9cdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:34:20.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9d49a1640 5 monclient: authenticate success, global_id 144387 2024-04-26T00:34:20.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9d49a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:34:20.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9cdd9b640 1 -- 172.21.15.37:0/2529680377 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9c00ab600 con 0x7fd9d0133b20 2024-04-26T00:34:20.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9cdd9b640 10 monclient: handle_config config(0 keys) 2024-04-26T00:34:20.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9d49a1640 10 monclient: get_monmap_and_config success 2024-04-26T00:34:20.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9d49a1640 4 set_mon_vals no callback set 2024-04-26T00:34:20.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9cdd9b640 1 -- 172.21.15.37:0/2529680377 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c00ab9d0 con 0x7fd9d0133b20 2024-04-26T00:34:20.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9cdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:20.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9cdd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:34:20.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9cdd9b640 10 monclient: dump: 2024-04-26T00:34:20.825 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:20.825 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:20.825 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:20.825 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:20.825 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:20.825 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:20.825 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:20.825 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:20.825 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:20.826 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:20.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9d49a1640 10 monclient: shutdown 2024-04-26T00:34:20.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9d49a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:34:20.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9d49a1640 1 -- 172.21.15.37:0/2529680377 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd9d0133b20 msgr2=0x7fd9d009e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:20.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9d49a1640 1 --2- 172.21.15.37:0/2529680377 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd9d0133b20 0x7fd9d009e9d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd9c0009870 tx=0x7fd9c001c1d0 comp rx=0 tx=0).stop 2024-04-26T00:34:20.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9d49a1640 1 -- 172.21.15.37:0/2529680377 shutdown_connections 2024-04-26T00:34:20.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9d49a1640 1 --2- 172.21.15.37:0/2529680377 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd9d0133f00 0x7fd9d0134350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:20.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9d49a1640 1 --2- 172.21.15.37:0/2529680377 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd9d0133b20 0x7fd9d009e9d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:20.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9d49a1640 1 -- 172.21.15.37:0/2529680377 >> 172.21.15.37:0/2529680377 conn(0x7fd9d00a1ad0 msgr2=0x7fd9d00a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:34:20.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9d49a1640 1 -- 172.21.15.37:0/2529680377 shutdown_connections 2024-04-26T00:34:20.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9d49a1640 1 -- 172.21.15.37:0/2529680377 wait complete. 2024-04-26T00:34:20.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9d49a1640 10 monclient: build_initial_monmap 2024-04-26T00:34:20.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9d49a1640 10 monclient: monmap: 2024-04-26T00:34:20.827 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:34:20.827 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:34:20.827 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:34:20.827 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:34:20.827 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:34:20.827 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:34:20.827 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:34:20.827 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:34:20.827 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:34:20.827 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:20.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9d49a1640 1 Processor -- start 2024-04-26T00:34:20.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9d49a1640 1 -- start start 2024-04-26T00:34:20.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9d49a1640 10 monclient: init 2024-04-26T00:34:20.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9d49a1640 10 monclient: _reopen_session rank -1 2024-04-26T00:34:20.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9d49a1640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:34:20.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9d49a1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9d0130770 0x7fd9d012def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:20.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9d49a1640 10 monclient(hunting): picked mon.noname-b con 0x7fd9d0130770 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:34:20.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9d49a1640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd9d0133b20 0x7fd9d012e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:20.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9d49a1640 10 monclient(hunting): picked mon.noname-a con 0x7fd9d0133b20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:34:20.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9d49a1640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd9d0133f00 0x7fd9d012e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:20.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9d49a1640 10 monclient(hunting): picked mon.noname-c con 0x7fd9d0133f00 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:34:20.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9d49a1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:20.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9d49a1640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd9d00a1750 con 0x7fd9d0133b20 2024-04-26T00:34:20.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9d49a1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:20.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9d49a1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd9d00a15d0 con 0x7fd9d0130770 2024-04-26T00:34:20.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9d49a1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:20.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9d49a1640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd9d013c880 con 0x7fd9d0133f00 2024-04-26T00:34:20.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9d49a1640 10 monclient(hunting): _renew_subs 2024-04-26T00:34:20.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9d49a1640 10 monclient(hunting): authenticate will time out at 1842.135802s 2024-04-26T00:34:20.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9ced9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd9d0133f00 0x7fd9d012e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:20.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9c7fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd9d0133b20 0x7fd9d012e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:20.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9c7fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd9d0133b20 0x7fd9d012e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33688/0 (socket says 172.21.15.37:33688) 2024-04-26T00:34:20.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9ced9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd9d0133f00 0x7fd9d012e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:57944/0 (socket says 172.21.15.37:57944) 2024-04-26T00:34:20.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9c7fff640 1 -- 172.21.15.37:0/3726020735 learned_addr learned my addr 172.21.15.37:0/3726020735 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:34:20.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9c7fff640 10 monclient(hunting): get_auth_request con 0x7fd9d0133b20 auth_method 0 2024-04-26T00:34:20.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.826+0000 7fd9c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:20.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9c7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:20.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:20.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9ced9d640 10 monclient(hunting): get_auth_request con 0x7fd9d0133f00 auth_method 0 2024-04-26T00:34:20.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9ced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:20.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9ced9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:20.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9ced9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:20.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9ced9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:20.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9ced9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:20.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9ced9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:20.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:20.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:20.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:20.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9ce59c640 1 --2- 172.21.15.37:0/3726020735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9d0130770 0x7fd9d012def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:20.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9ce59c640 10 monclient(hunting): get_auth_request con 0x7fd9d0130770 auth_method 0 2024-04-26T00:34:20.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9ce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:20.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9ce59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:20.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9ce59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:20.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9c7fff640 10 monclient(hunting): handle_auth_done global_id 144390 payload 824 2024-04-26T00:34:20.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9c7fff640 1 -- 172.21.15.37:0/3726020735 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd9d0133f00 msgr2=0x7fd9d012e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:20.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9c7fff640 1 --2- 172.21.15.37:0/3726020735 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd9d0133f00 0x7fd9d012e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:20.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9c7fff640 1 -- 172.21.15.37:0/3726020735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9d0130770 msgr2=0x7fd9d012def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:20.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9c7fff640 1 --2- 172.21.15.37:0/3726020735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9d0130770 0x7fd9d012def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:20.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9c7fff640 10 monclient: _finish_hunting 0 2024-04-26T00:34:20.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9c7fff640 1 monclient: found mon.noname-a 2024-04-26T00:34:20.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:34:20.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9c7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:34:20.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9c7fff640 1 -- 172.21.15.37:0/3726020735 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9d012f3d0 con 0x7fd9d0133b20 2024-04-26T00:34:20.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9ced9d640 1 --2- 172.21.15.37:0/3726020735 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd9d0133f00 0x7fd9d012e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:34:20.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9c7fff640 1 -- 172.21.15.37:0/3726020735 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd9d0133b20 msgr2=0x7fd9d012e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:20.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9c7fff640 1 --2- 172.21.15.37:0/3726020735 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd9d0133b20 0x7fd9d012e430 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd9b800e960 tx=0x7fd9b800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:20.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9c6ffd640 1 -- 172.21.15.37:0/3726020735 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b8017070 con 0x7fd9d0133b20 2024-04-26T00:34:20.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:20.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9c6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:34:20.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9c6ffd640 10 monclient: dump: 2024-04-26T00:34:20.835 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:20.835 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:20.835 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:20.835 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:20.835 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:20.835 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:20.835 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:20.836 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:20.836 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:20.836 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:20.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9c6ffd640 10 monclient: _finish_auth 0 2024-04-26T00:34:20.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9c6ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:34:20.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:34:20.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9c6ffd640 1 -- 172.21.15.37:0/3726020735 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9b800cc10 con 0x7fd9d0133b20 2024-04-26T00:34:20.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9c6ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:34:20.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9c6ffd640 1 -- 172.21.15.37:0/3726020735 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b8013470 con 0x7fd9d0133b20 2024-04-26T00:34:20.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:20.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9c6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:34:20.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9c6ffd640 10 monclient: dump: 2024-04-26T00:34:20.836 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:20.836 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:20.836 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:20.837 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:20.837 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:20.837 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:20.837 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:20.837 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:20.837 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:20.837 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:20.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9d49a1640 5 monclient: authenticate success, global_id 144390 2024-04-26T00:34:20.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9d49a1640 10 monclient: _renew_subs 2024-04-26T00:34:20.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9cd59a640 4 set_mon_vals no callback set 2024-04-26T00:34:20.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9d49a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:34:20.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9d49a1640 1 -- 172.21.15.37:0/3726020735 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9d0135340 con 0x7fd9d0133b20 2024-04-26T00:34:20.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9d49a1640 10 monclient: _renew_subs 2024-04-26T00:34:20.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9d49a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:34:20.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.830+0000 7fd9d49a1640 1 -- 172.21.15.37:0/3726020735 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9d00728a0 con 0x7fd9d0133b20 2024-04-26T00:34:20.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.834+0000 7fd9d49a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:34:20.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.834+0000 7fd9d49a1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:34:20.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.834+0000 7fd9d49a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:34:20.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.834+0000 7fd9d49a1640 1 -- 172.21.15.37:0/3726020735 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9d0066f40 con 0x7fd9d0133b20 2024-04-26T00:34:20.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.834+0000 7fd9c6ffd640 1 -- 172.21.15.37:0/3726020735 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd9b801a070 con 0x7fd9d0133b20 2024-04-26T00:34:20.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.834+0000 7fd9c6ffd640 1 --2- 172.21.15.37:0/3726020735 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd9b0035f00 0x7fd9b00383c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:20.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.834+0000 7fd9c6ffd640 1 -- 172.21.15.37:0/3726020735 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(561..561 src has 370..561) ==== 3959+0+0 (secure 0 0 0) 0x7fd9b8048b60 con 0x7fd9d0133b20 2024-04-26T00:34:20.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.834+0000 7fd9ce59c640 1 --2- 172.21.15.37:0/3726020735 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd9b0035f00 0x7fd9b00383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:20.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.834+0000 7fd9ce59c640 10 monclient: get_auth_request con 0x7fd9b0035f00 auth_method 0 2024-04-26T00:34:20.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.834+0000 7fd9ce59c640 1 --2- 172.21.15.37:0/3726020735 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd9b0035f00 0x7fd9b00383c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fd9c00ab000 tx=0x7fd9c00a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:20.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.838+0000 7fd9c6ffd640 1 -- 172.21.15.37:0/3726020735 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fd9b8013610 con 0x7fd9d0133b20 2024-04-26T00:34:20.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.838+0000 7fd9c6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:34:20.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:20.838+0000 7fd9c6ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:34:21.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.062+0000 7fd9d49a1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:34:21.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.062+0000 7fd9d49a1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:34:21.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.062+0000 7fd9d49a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:34:21.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.062+0000 7fd9d49a1640 1 -- 172.21.15.37:0/3726020735 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd9d012f7e0 con 0x7fd9d0133b20 2024-04-26T00:34:21.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.066+0000 7fd9c6ffd640 1 -- 172.21.15.37:0/3726020735 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fd9b801a520 con 0x7fd9d0133b20 2024-04-26T00:34:21.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.066+0000 7fd9c6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:34:21.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.066+0000 7fd9c6ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:34:21.063 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":108,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:34:21.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.070+0000 7fd9d49a1640 1 -- 172.21.15.37:0/3726020735 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd9b0035f00 msgr2=0x7fd9b00383c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:21.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.070+0000 7fd9d49a1640 1 --2- 172.21.15.37:0/3726020735 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd9b0035f00 0x7fd9b00383c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fd9c00ab000 tx=0x7fd9c00a6000 comp rx=0 tx=0).stop 2024-04-26T00:34:21.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.070+0000 7fd9d49a1640 10 monclient: shutdown 2024-04-26T00:34:21.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.070+0000 7fd9d49a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:34:21.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.070+0000 7fd9d49a1640 1 -- 172.21.15.37:0/3726020735 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd9d0133b20 msgr2=0x7fd9d012e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:21.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.070+0000 7fd9d49a1640 1 --2- 172.21.15.37:0/3726020735 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd9d0133b20 0x7fd9d012e430 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd9b800e960 tx=0x7fd9b800ee20 comp rx=0 tx=0).stop 2024-04-26T00:34:21.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.070+0000 7fd9d49a1640 1 -- 172.21.15.37:0/3726020735 shutdown_connections 2024-04-26T00:34:21.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.070+0000 7fd9d49a1640 1 --2- 172.21.15.37:0/3726020735 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd9b0035f00 0x7fd9b00383c0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:21.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.070+0000 7fd9d49a1640 1 --2- 172.21.15.37:0/3726020735 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd9d0133f00 0x7fd9d012e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:21.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.070+0000 7fd9d49a1640 1 --2- 172.21.15.37:0/3726020735 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd9d0133b20 0x7fd9d012e430 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:21.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.070+0000 7fd9d49a1640 1 --2- 172.21.15.37:0/3726020735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9d0130770 0x7fd9d012def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:21.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.070+0000 7fd9d49a1640 1 -- 172.21.15.37:0/3726020735 >> 172.21.15.37:0/3726020735 conn(0x7fd9d00a1ad0 msgr2=0x7fd9d012cbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:34:21.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.070+0000 7fd9d49a1640 1 -- 172.21.15.37:0/3726020735 shutdown_connections 2024-04-26T00:34:21.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.070+0000 7fd9d49a1640 1 -- 172.21.15.37:0/3726020735 wait complete. 2024-04-26T00:34:21.083 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:34:21.083 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:34:21.083 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:34:21.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.210+0000 7fe5b769f640 10 monclient: get_monmap_and_config 2024-04-26T00:34:21.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.210+0000 7fe5b769f640 10 monclient: build_initial_monmap 2024-04-26T00:34:21.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.210+0000 7fe5b769f640 10 monclient: monmap: 2024-04-26T00:34:21.207 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:34:21.207 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:21.207 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:34:21.214810+0000 2024-04-26T00:34:21.207 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:34:21.214810+0000 2024-04-26T00:34:21.207 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:34:21.207 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:21.208 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:34:21.208 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:34:21.208 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:34:21.208 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:34:21.208 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:34:21.208 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:21.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.214+0000 7fe5b769f640 1 Processor -- start 2024-04-26T00:34:21.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.214+0000 7fe5b769f640 1 -- start start 2024-04-26T00:34:21.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.214+0000 7fe5b769f640 10 monclient: init 2024-04-26T00:34:21.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.214+0000 7fe5b769f640 10 monclient: _reopen_session rank -1 2024-04-26T00:34:21.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.214+0000 7fe5b769f640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-04-26T00:34:21.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.214+0000 7fe5b769f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe5b00a3d90 addr v1:172.21.15.45:6789/0 2024-04-26T00:34:21.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.214+0000 7fe5b769f640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe5b00a4170 0x7fe5b00a45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:21.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.214+0000 7fe5b769f640 10 monclient(hunting): picked mon.noname-c con 0x7fe5b00a4170 addr v2:172.21.15.45:3300/0 2024-04-26T00:34:21.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.214+0000 7fe5b769f640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fe5b013c3e0 0x7fe5b013e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:21.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.214+0000 7fe5b769f640 10 monclient(hunting): picked mon.noname-a con 0x7fe5b013c3e0 addr v2:172.21.15.37:3300/0 2024-04-26T00:34:21.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.214+0000 7fe5b769f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5b00a5150 con 0x7fe5b00a3d90 2024-04-26T00:34:21.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.214+0000 7fe5b769f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:21.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.214+0000 7fe5b769f640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fe5b009c650 con 0x7fe5b013c3e0 2024-04-26T00:34:21.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.214+0000 7fe5b769f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:21.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.214+0000 7fe5b769f640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fe5b009b280 con 0x7fe5b00a4170 2024-04-26T00:34:21.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.214+0000 7fe5b769f640 10 monclient(hunting): _renew_subs 2024-04-26T00:34:21.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.214+0000 7fe5b769f640 10 monclient(hunting): authenticate will time out at 1842.520948s 2024-04-26T00:34:21.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.214+0000 7fe5b5c3c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fe5b013c3e0 0x7fe5b013e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:21.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.214+0000 7fe5b5c3c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fe5b013c3e0 0x7fe5b013e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33702/0 (socket says 172.21.15.37:33702) 2024-04-26T00:34:21.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.214+0000 7fe5b5c3c640 1 -- 172.21.15.37:0/2473578113 learned_addr learned my addr 172.21.15.37:0/2473578113 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:34:21.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.214+0000 7fe5b5c3c640 10 monclient(hunting): get_auth_request con 0x7fe5b013c3e0 auth_method 0 2024-04-26T00:34:21.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.214+0000 7fe5b5c3c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:21.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.214+0000 7fe5b5c3c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:21.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.214+0000 7fe5b5c3c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:21.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.214+0000 7fe5b5c3c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:21.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.214+0000 7fe5b5c3c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:21.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.214+0000 7fe5b5c3c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:21.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.214+0000 7fe5b5c3c640 10 monclient(hunting): handle_auth_done global_id 144393 payload 274 2024-04-26T00:34:21.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.214+0000 7fe5b5c3c640 1 -- 172.21.15.37:0/2473578113 >> v2:172.21.15.45:3300/0 conn(0x7fe5b00a4170 msgr2=0x7fe5b00a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:34:21.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.214+0000 7fe5b5c3c640 1 --2- 172.21.15.37:0/2473578113 >> v2:172.21.15.45:3300/0 conn(0x7fe5b00a4170 0x7fe5b00a45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:21.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.214+0000 7fe5b5c3c640 1 -- 172.21.15.37:0/2473578113 >> v1:172.21.15.45:6789/0 conn(0x7fe5b00a3d90 legacy=0x7fe5b009bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:34:21.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.214+0000 7fe5b5c3c640 10 monclient: _finish_hunting 0 2024-04-26T00:34:21.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.214+0000 7fe5b5c3c640 1 monclient: found mon.noname-a 2024-04-26T00:34:21.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.214+0000 7fe5b5c3c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:34:21.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.214+0000 7fe5b5c3c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:34:21.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.214+0000 7fe5b5c3c640 1 -- 172.21.15.37:0/2473578113 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5b009b560 con 0x7fe5b013c3e0 2024-04-26T00:34:21.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.218+0000 7fe5b5c3c640 1 -- 172.21.15.37:0/2473578113 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5b013c3e0 msgr2=0x7fe5b013e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:21.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.218+0000 7fe5b5c3c640 1 --2- 172.21.15.37:0/2473578113 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5b013c3e0 0x7fe5b013e7d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe5b009d140 tx=0x7fe5a801c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a98b734ade516fc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:21.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.218+0000 7fe5a3fff640 1 -- 172.21.15.37:0/2473578113 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5a80a5640 con 0x7fe5b013c3e0 2024-04-26T00:34:21.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.218+0000 7fe5a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:21.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.218+0000 7fe5a3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:34:21.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.218+0000 7fe5a3fff640 10 monclient: dump: 2024-04-26T00:34:21.228 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:21.228 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:21.228 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:21.228 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:21.228 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:21.228 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:21.228 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:21.228 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:21.228 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:21.228 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:21.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.218+0000 7fe5a3fff640 10 monclient: _finish_auth 0 2024-04-26T00:34:21.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.218+0000 7fe5a3fff640 10 monclient: _check_auth_tickets 2024-04-26T00:34:21.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.218+0000 7fe5a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:34:21.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.218+0000 7fe5a3fff640 1 -- 172.21.15.37:0/2473578113 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5a80a57e0 con 0x7fe5b013c3e0 2024-04-26T00:34:21.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.218+0000 7fe5b769f640 5 monclient: authenticate success, global_id 144393 2024-04-26T00:34:21.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.218+0000 7fe5b769f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:34:21.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.218+0000 7fe5a3fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:34:21.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.218+0000 7fe5b769f640 10 monclient: get_monmap_and_config success 2024-04-26T00:34:21.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.218+0000 7fe5b769f640 4 set_mon_vals no callback set 2024-04-26T00:34:21.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.218+0000 7fe5a3fff640 1 -- 172.21.15.37:0/2473578113 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5a80a5bb0 con 0x7fe5b013c3e0 2024-04-26T00:34:21.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.218+0000 7fe5a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:21.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.218+0000 7fe5a3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:34:21.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.218+0000 7fe5a3fff640 10 monclient: dump: 2024-04-26T00:34:21.229 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:21.229 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:21.229 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:21.229 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:21.229 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:21.229 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:21.229 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:21.230 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:21.230 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:21.230 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:21.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.218+0000 7fe5b769f640 10 monclient: shutdown 2024-04-26T00:34:21.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.218+0000 7fe5b769f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:34:21.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.218+0000 7fe5b769f640 1 -- 172.21.15.37:0/2473578113 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5b013c3e0 msgr2=0x7fe5b013e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:21.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.218+0000 7fe5b769f640 1 --2- 172.21.15.37:0/2473578113 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5b013c3e0 0x7fe5b013e7d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe5b009d140 tx=0x7fe5a801c870 comp rx=0 tx=0).stop 2024-04-26T00:34:21.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.218+0000 7fe5b769f640 1 -- 172.21.15.37:0/2473578113 shutdown_connections 2024-04-26T00:34:21.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.218+0000 7fe5b769f640 1 --2- 172.21.15.37:0/2473578113 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5b013c3e0 0x7fe5b013e7d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:21.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.218+0000 7fe5b769f640 1 --2- 172.21.15.37:0/2473578113 >> v2:172.21.15.45:3300/0 conn(0x7fe5b00a4170 0x7fe5b00a45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:21.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.218+0000 7fe5b769f640 1 -- 172.21.15.37:0/2473578113 >> 172.21.15.37:0/2473578113 conn(0x7fe5b00aa0a0 msgr2=0x7fe5b00a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:34:21.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.218+0000 7fe5b769f640 1 -- 172.21.15.37:0/2473578113 shutdown_connections 2024-04-26T00:34:21.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.218+0000 7fe5b769f640 1 -- 172.21.15.37:0/2473578113 wait complete. 2024-04-26T00:34:21.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.218+0000 7fe5b769f640 10 monclient: build_initial_monmap 2024-04-26T00:34:21.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.218+0000 7fe5b769f640 10 monclient: monmap: 2024-04-26T00:34:21.231 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:34:21.231 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:34:21.231 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:34:21.231 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:34:21.231 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:34:21.231 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:34:21.231 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:34:21.231 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:34:21.231 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:34:21.231 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:21.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.218+0000 7fe5b769f640 1 Processor -- start 2024-04-26T00:34:21.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.218+0000 7fe5b769f640 1 -- start start 2024-04-26T00:34:21.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.218+0000 7fe5b769f640 10 monclient: init 2024-04-26T00:34:21.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b769f640 10 monclient: _reopen_session rank -1 2024-04-26T00:34:21.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b769f640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:34:21.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b769f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5b00a3d90 0x7fe5b01cebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:21.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b769f640 10 monclient(hunting): picked mon.noname-a con 0x7fe5b00a3d90 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:34:21.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b769f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5b00a4170 0x7fe5b01cf280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:21.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b769f640 10 monclient(hunting): picked mon.noname-b con 0x7fe5b00a4170 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:34:21.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b769f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe5b013c3e0 0x7fe5b0139d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:21.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b769f640 10 monclient(hunting): picked mon.noname-c con 0x7fe5b013c3e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:34:21.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b769f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:21.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b769f640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe5b01418c0 con 0x7fe5b00a3d90 2024-04-26T00:34:21.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b769f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:21.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b769f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe5b0141740 con 0x7fe5b00a4170 2024-04-26T00:34:21.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b769f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:21.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b769f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe5b0141a40 con 0x7fe5b013c3e0 2024-04-26T00:34:21.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b769f640 10 monclient(hunting): _renew_subs 2024-04-26T00:34:21.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b769f640 10 monclient(hunting): authenticate will time out at 1842.528262s 2024-04-26T00:34:21.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b543b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5b00a3d90 0x7fe5b01cebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:21.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b5c3c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe5b013c3e0 0x7fe5b0139d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:21.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b543b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5b00a3d90 0x7fe5b01cebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33704/0 (socket says 172.21.15.37:33704) 2024-04-26T00:34:21.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b543b640 1 -- 172.21.15.37:0/3728651735 learned_addr learned my addr 172.21.15.37:0/3728651735 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:34:21.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b543b640 10 monclient(hunting): get_auth_request con 0x7fe5b00a3d90 auth_method 0 2024-04-26T00:34:21.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b543b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:21.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b543b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:21.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b543b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:21.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b5c3c640 10 monclient(hunting): get_auth_request con 0x7fe5b013c3e0 auth_method 0 2024-04-26T00:34:21.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b5c3c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:21.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b5c3c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:21.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b5c3c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:21.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b543b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:21.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b543b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:21.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b543b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:21.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b5c3c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:21.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b5c3c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:21.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b5c3c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:21.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b4c3a640 1 --2- 172.21.15.37:0/3728651735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5b00a4170 0x7fe5b01cf280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:21.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b4c3a640 10 monclient(hunting): get_auth_request con 0x7fe5b00a4170 auth_method 0 2024-04-26T00:34:21.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b4c3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:21.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b4c3a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:21.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b4c3a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:21.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b543b640 10 monclient(hunting): handle_auth_done global_id 144396 payload 824 2024-04-26T00:34:21.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b543b640 1 -- 172.21.15.37:0/3728651735 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe5b013c3e0 msgr2=0x7fe5b0139d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:21.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b543b640 1 --2- 172.21.15.37:0/3728651735 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe5b013c3e0 0x7fe5b0139d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:21.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b543b640 1 -- 172.21.15.37:0/3728651735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5b00a4170 msgr2=0x7fe5b01cf280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:21.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b543b640 1 --2- 172.21.15.37:0/3728651735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5b00a4170 0x7fe5b01cf280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:21.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b543b640 10 monclient: _finish_hunting 0 2024-04-26T00:34:21.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b543b640 1 monclient: found mon.noname-a 2024-04-26T00:34:21.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b543b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:34:21.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b543b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:34:21.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b543b640 1 -- 172.21.15.37:0/3728651735 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5b013c850 con 0x7fe5b00a3d90 2024-04-26T00:34:21.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b5c3c640 1 --2- 172.21.15.37:0/3728651735 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe5b013c3e0 0x7fe5b0139d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:34:21.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b543b640 1 -- 172.21.15.37:0/3728651735 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5b00a3d90 msgr2=0x7fe5b01cebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:21.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b543b640 1 --2- 172.21.15.37:0/3728651735 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5b00a3d90 0x7fe5b01cebe0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe5a400c450 tx=0x7fe5a400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:21.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5a1ffb640 1 -- 172.21.15.37:0/3728651735 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5a4012a40 con 0x7fe5b00a3d90 2024-04-26T00:34:21.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5a1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:21.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5a1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:34:21.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5a1ffb640 10 monclient: dump: 2024-04-26T00:34:21.236 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:21.236 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:21.236 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:21.236 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:21.236 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:21.236 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:21.236 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:21.236 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:21.236 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:21.236 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:21.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5a1ffb640 10 monclient: _finish_auth 0 2024-04-26T00:34:21.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5a1ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:34:21.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:34:21.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b769f640 5 monclient: authenticate success, global_id 144396 2024-04-26T00:34:21.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5a1ffb640 1 -- 172.21.15.37:0/3728651735 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5a4017070 con 0x7fe5b00a3d90 2024-04-26T00:34:21.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5a1ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:34:21.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b769f640 10 monclient: _renew_subs 2024-04-26T00:34:21.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5a1ffb640 1 -- 172.21.15.37:0/3728651735 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5a4012e40 con 0x7fe5b00a3d90 2024-04-26T00:34:21.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b769f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:34:21.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b769f640 1 -- 172.21.15.37:0/3728651735 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5b013b9b0 con 0x7fe5b00a3d90 2024-04-26T00:34:21.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5a37fe640 4 set_mon_vals no callback set 2024-04-26T00:34:21.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5a1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:21.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5a1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:34:21.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5a1ffb640 10 monclient: dump: 2024-04-26T00:34:21.237 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:21.237 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:21.237 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:21.237 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:21.237 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:21.238 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:21.238 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:21.238 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:21.238 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:21.238 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:21.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b769f640 10 monclient: _renew_subs 2024-04-26T00:34:21.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b769f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:34:21.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.222+0000 7fe5b769f640 1 -- 172.21.15.37:0/3728651735 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5b01373d0 con 0x7fe5b00a3d90 2024-04-26T00:34:21.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.226+0000 7fe5b769f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:34:21.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.226+0000 7fe5b769f640 1 --2- 172.21.15.37:0/3728651735 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5b01cf7c0 0x7fe5b013aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:21.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.226+0000 7fe5b769f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:21.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.226+0000 7fe5b769f640 1 -- 172.21.15.37:0/3728651735 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe5b004f510 con 0x7fe5b01cf7c0 2024-04-26T00:34:21.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.226+0000 7fe5b4c3a640 1 --2- 172.21.15.37:0/3728651735 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5b01cf7c0 0x7fe5b013aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:21.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.226+0000 7fe5b4c3a640 10 monclient: get_auth_request con 0x7fe5b01cf7c0 auth_method 0 2024-04-26T00:34:21.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.226+0000 7fe5b4c3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:21.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.226+0000 7fe5b4c3a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:21.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.226+0000 7fe5b4c3a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:21.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.226+0000 7fe5b4c3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:21.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.226+0000 7fe5b4c3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:21.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.226+0000 7fe5b4c3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:21.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.226+0000 7fe5a1ffb640 1 -- 172.21.15.37:0/3728651735 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe5a401c460 con 0x7fe5b00a3d90 2024-04-26T00:34:21.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.226+0000 7fe5b4c3a640 10 monclient(hunting): handle_auth_done global_id 144399 payload 824 2024-04-26T00:34:21.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.226+0000 7fe5b4c3a640 1 -- 172.21.15.37:0/3728651735 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe5b0121090 con 0x7fe5b01cf7c0 2024-04-26T00:34:21.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.226+0000 7fe5a1ffb640 1 --2- 172.21.15.37:0/3728651735 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe594035e80 0x7fe594038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:21.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.226+0000 7fe5a1ffb640 1 -- 172.21.15.37:0/3728651735 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(561..561 src has 370..561) ==== 3959+0+0 (secure 0 0 0) 0x7fe5a4048870 con 0x7fe5b00a3d90 2024-04-26T00:34:21.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.226+0000 7fe5b5c3c640 1 --2- 172.21.15.37:0/3728651735 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe594035e80 0x7fe594038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:21.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.226+0000 7fe5b5c3c640 10 monclient: get_auth_request con 0x7fe594035e80 auth_method 0 2024-04-26T00:34:21.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.226+0000 7fe5b4c3a640 1 -- 172.21.15.37:0/3728651735 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5b01cf7c0 msgr2=0x7fe5b013aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:21.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.226+0000 7fe5b4c3a640 1 --2- 172.21.15.37:0/3728651735 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5b01cf7c0 0x7fe5b013aa50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe5b01201a0 tx=0x7fe5ac00db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:21.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.226+0000 7fe5a1ffb640 1 -- 172.21.15.37:0/3728651735 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5ac018030 con 0x7fe5b01cf7c0 2024-04-26T00:34:21.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.226+0000 7fe5a1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:21.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.226+0000 7fe5a1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:34:21.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.226+0000 7fe5a1ffb640 10 monclient: dump: 2024-04-26T00:34:21.240 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:21.240 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:21.240 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:21.241 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:21.241 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:21.241 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:21.241 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:21.241 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:21.241 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:21.241 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:21.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.226+0000 7fe5a1ffb640 1 -- 172.21.15.37:0/3728651735 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe5ac0098f0 con 0x7fe5b01cf7c0 2024-04-26T00:34:21.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.226+0000 7fe5a1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:34:21.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.226+0000 7fe5a1ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:34:21.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.226+0000 7fe5a1ffb640 1 -- 172.21.15.37:0/3728651735 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5b01cf7c0 msgr2=0x7fe5b013aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:21.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.226+0000 7fe5a1ffb640 1 --2- 172.21.15.37:0/3728651735 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5b01cf7c0 0x7fe5b013aa50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe5b01201a0 tx=0x7fe5ac00db90 comp rx=0 tx=0).stop 2024-04-26T00:34:21.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.226+0000 7fe5a1ffb640 1 -- 172.21.15.37:0/3728651735 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5b01cf7c0 msgr2=0x7fe5b013aa50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:34:21.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.226+0000 7fe5a1ffb640 1 --2- 172.21.15.37:0/3728651735 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5b01cf7c0 0x7fe5b013aa50 secure :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe5b01201a0 tx=0x7fe5ac00db90 comp rx=0 tx=0).stop 2024-04-26T00:34:21.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.226+0000 7fe5b5c3c640 1 --2- 172.21.15.37:0/3728651735 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe594035e80 0x7fe594038340 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe5a80ab000 tx=0x7fe5a80a7070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:21.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.234+0000 7fe5b769f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:34:21.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.234+0000 7fe5b769f640 1 --2- 172.21.15.37:0/3728651735 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5b002c000 0x7fe5b002c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:21.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.234+0000 7fe5b769f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:21.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.234+0000 7fe5b769f640 1 -- 172.21.15.37:0/3728651735 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe5b00a1ea0 con 0x7fe5b002c000 2024-04-26T00:34:21.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.234+0000 7fe5b4c3a640 1 --2- 172.21.15.37:0/3728651735 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5b002c000 0x7fe5b002c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:21.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.234+0000 7fe5b4c3a640 10 monclient: get_auth_request con 0x7fe5b002c000 auth_method 0 2024-04-26T00:34:21.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.234+0000 7fe5b4c3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:21.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.234+0000 7fe5b4c3a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:21.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.234+0000 7fe5b4c3a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:21.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.234+0000 7fe5b4c3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:21.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.234+0000 7fe5b4c3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:21.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.234+0000 7fe5b4c3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:21.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.234+0000 7fe5b4c3a640 10 monclient(hunting): handle_auth_done global_id 144402 payload 824 2024-04-26T00:34:21.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.234+0000 7fe5b4c3a640 1 -- 172.21.15.37:0/3728651735 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe5b013af90 con 0x7fe5b002c000 2024-04-26T00:34:21.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.238+0000 7fe5b4c3a640 1 -- 172.21.15.37:0/3728651735 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5b002c000 msgr2=0x7fe5b002c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:21.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.238+0000 7fe5b4c3a640 1 --2- 172.21.15.37:0/3728651735 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5b002c000 0x7fe5b002c410 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe5b01201a0 tx=0x7fe5ac0047e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:21.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.238+0000 7fe5a1ffb640 1 -- 172.21.15.37:0/3728651735 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5ac028420 con 0x7fe5b002c000 2024-04-26T00:34:21.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.238+0000 7fe5a1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:21.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.238+0000 7fe5a1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:34:21.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.238+0000 7fe5a1ffb640 10 monclient: dump: 2024-04-26T00:34:21.245 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:21.245 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:21.245 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:21.245 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:21.246 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:21.246 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:21.246 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:21.246 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:21.246 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:21.246 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:21.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.238+0000 7fe5a1ffb640 1 -- 172.21.15.37:0/3728651735 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4890 (secure 0 0 0) 0x7fe5ac014410 con 0x7fe5b002c000 2024-04-26T00:34:21.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.238+0000 7fe5a1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:34:21.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.238+0000 7fe5a1ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:34:21.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.238+0000 7fe5a1ffb640 1 -- 172.21.15.37:0/3728651735 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5b002c000 msgr2=0x7fe5b002c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:21.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.238+0000 7fe5a1ffb640 1 --2- 172.21.15.37:0/3728651735 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5b002c000 0x7fe5b002c410 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe5b01201a0 tx=0x7fe5ac0047e0 comp rx=0 tx=0).stop 2024-04-26T00:34:21.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.238+0000 7fe5a1ffb640 1 -- 172.21.15.37:0/3728651735 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5b002c000 msgr2=0x7fe5b002c410 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:34:21.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.238+0000 7fe5a1ffb640 1 --2- 172.21.15.37:0/3728651735 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5b002c000 0x7fe5b002c410 secure :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe5b01201a0 tx=0x7fe5ac0047e0 comp rx=0 tx=0).stop 2024-04-26T00:34:21.247 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:34:21.247 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:34:21.247 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:34:21.247 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:34:21.247 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 108, 2024-04-26T00:34:21.247 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 147782, 2024-04-26T00:34:21.247 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:34:21.247 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:34:21.247 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:34:21.247 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:34:21.247 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:21.247 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:34:21.248 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:34:21.248 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:34:21.248 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:34:21.248 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:34:21.248 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:34:21.248 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:34:21.248 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:34:21.248 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:34:21.249 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:34:21.249 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:34:21.249 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:34:21.249 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:34:21.249 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:34:21.249 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:34:21.249 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:21.249 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:34:21.249 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:34:21.249 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:34:21.249 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:34:21.249 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:34:21.249 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:34:21.249 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:34:21.250 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:34:21.250 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:34:21.250 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:34:21.250 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:34:21.250 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:34:21.250 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:34:21.250 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:21.250 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:21.250 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:34:21.250 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:34:21.250 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:34:21.250 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:34:21.250 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:34:21.250 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:34:21.250 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:34:21.251 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:34:21.251 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:34:21.251 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:34:21.251 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:34:21.251 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:34:21.251 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:34:21.251 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:34:21.251 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:34:21.251 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:34:21.251 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:34:21.251 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:34:21.251 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:34:21.251 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:34:21.251 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:34:21.251 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:34:21.252 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:34:21.252 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:34:21.252 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:34:21.252 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:34:21.252 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:34:21.252 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:34:21.252 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:21.252 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:34:21.252 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:21.252 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:34:21.252 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:21.252 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:34:21.252 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:34:21.252 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:34:21.252 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:34:21.253 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:21.253 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:34:21.253 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:34:21.253 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:21.253 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:21.253 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:21.253 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:34:21.253 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:34:21.253 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:21.253 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:21.253 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:21.253 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:21.253 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:34:21.253 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:34:21.253 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:34:21.253 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:34:21.254 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:34:21.254 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:21.254 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:21.254 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:34:21.254 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:34:21.254 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:34:21.254 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:34:21.254 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:21.254 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:34:21.254 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:34:21.254 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:21.254 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:21.254 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:21.254 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:34:21.255 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:34:21.255 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:21.255 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:21.255 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:21.255 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:21.255 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:34:21.255 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:34:21.255 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:34:21.255 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:34:21.255 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:34:21.255 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:21.255 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:21.255 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:34:21.255 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:34:21.255 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:34:21.256 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:34:21.256 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:21.256 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:34:21.256 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:34:21.256 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:21.256 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:21.256 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:21.256 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:34:21.256 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:34:21.256 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:21.256 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:21.256 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:21.256 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:21.256 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:34:21.256 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:34:21.256 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:34:21.257 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:34:21.257 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:34:21.257 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:21.257 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:21.257 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:21.257 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:34:21.257 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:34:21.257 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:21.257 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:34:21.257 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:34:21.257 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:34:21.257 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:21.257 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:21.257 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:34:21.257 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:21.258 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:34:21.258 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:34:21.258 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:34:21.258 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:21.258 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:21.258 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:34:21.258 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:21.258 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:34:21.258 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:34:21.258 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:34:21.258 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:21.258 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:21.258 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:34:21.259 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:21.259 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:34:21.259 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:34:21.259 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:34:21.259 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:21.259 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:21.259 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:21.259 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:34:21.260 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:34:21.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.238+0000 7fe5b769f640 1 -- 172.21.15.37:0/3728651735 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe594035e80 msgr2=0x7fe594038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:21.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.238+0000 7fe5b769f640 1 --2- 172.21.15.37:0/3728651735 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe594035e80 0x7fe594038340 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe5a80ab000 tx=0x7fe5a80a7070 comp rx=0 tx=0).stop 2024-04-26T00:34:21.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.238+0000 7fe5b769f640 10 monclient: shutdown 2024-04-26T00:34:21.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.238+0000 7fe5b769f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:34:21.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.238+0000 7fe5b769f640 1 -- 172.21.15.37:0/3728651735 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5b00a3d90 msgr2=0x7fe5b01cebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:21.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.238+0000 7fe5b769f640 1 --2- 172.21.15.37:0/3728651735 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5b00a3d90 0x7fe5b01cebe0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe5a400c450 tx=0x7fe5a400c910 comp rx=0 tx=0).stop 2024-04-26T00:34:21.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.238+0000 7fe5b5c3c640 1 -- 172.21.15.37:0/3728651735 reap_dead start 2024-04-26T00:34:21.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.238+0000 7fe5b5c3c640 1 -- 172.21.15.37:0/3728651735 reap_dead start 2024-04-26T00:34:21.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.238+0000 7fe5b769f640 1 -- 172.21.15.37:0/3728651735 shutdown_connections 2024-04-26T00:34:21.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.238+0000 7fe5b769f640 1 -- 172.21.15.37:0/3728651735 >> 172.21.15.37:0/3728651735 conn(0x7fe5b00aa0a0 msgr2=0x7fe5b013d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:34:21.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.238+0000 7fe5b769f640 1 -- 172.21.15.37:0/3728651735 shutdown_connections 2024-04-26T00:34:21.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.238+0000 7fe5b769f640 1 -- 172.21.15.37:0/3728651735 wait complete. 2024-04-26T00:34:21.262 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:34:21.262 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:34:21.262 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:34:21.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.382+0000 7f309eaa8640 10 monclient: get_monmap_and_config 2024-04-26T00:34:21.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.382+0000 7f309eaa8640 10 monclient: build_initial_monmap 2024-04-26T00:34:21.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.382+0000 7f309eaa8640 10 monclient: monmap: 2024-04-26T00:34:21.380 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:34:21.380 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:21.380 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:34:21.387535+0000 2024-04-26T00:34:21.380 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:34:21.387535+0000 2024-04-26T00:34:21.380 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:34:21.380 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:21.380 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:34:21.380 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:34:21.380 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:34:21.380 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:34:21.380 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:34:21.380 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:21.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f309eaa8640 1 Processor -- start 2024-04-26T00:34:21.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f309eaa8640 1 -- start start 2024-04-26T00:34:21.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f309eaa8640 10 monclient: init 2024-04-26T00:34:21.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f309eaa8640 10 monclient: _reopen_session rank -1 2024-04-26T00:34:21.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f309eaa8640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-04-26T00:34:21.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f309eaa8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f30980a3d90 addr v1:172.21.15.37:6789/0 2024-04-26T00:34:21.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f309eaa8640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f30980a4170 0x7f30980a45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:21.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f309eaa8640 10 monclient(hunting): picked mon.noname-a con 0x7f30980a4170 addr v2:172.21.15.37:3300/0 2024-04-26T00:34:21.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f309eaa8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f309813c3e0 0x7f309813e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:21.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f309eaa8640 10 monclient(hunting): picked mon.noname-b con 0x7f309813c3e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:34:21.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f309eaa8640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30980a5150 con 0x7f30980a3d90 2024-04-26T00:34:21.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f309eaa8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:21.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f309eaa8640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f309809c650 con 0x7f30980a4170 2024-04-26T00:34:21.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f309eaa8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:21.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f309eaa8640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f309809b280 con 0x7f309813c3e0 2024-04-26T00:34:21.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f309eaa8640 10 monclient(hunting): _renew_subs 2024-04-26T00:34:21.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f309eaa8640 10 monclient(hunting): authenticate will time out at 1842.693592s 2024-04-26T00:34:21.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f309c844640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f30980a3d90 0x7f309809bf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:33370/0 (socket says 172.21.15.37:33370) 2024-04-26T00:34:21.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f309c844640 1 -- 172.21.15.37:0/2849162996 learned_addr learned my addr 172.21.15.37:0/2849162996 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:34:21.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f309d045640 1 --2- 172.21.15.37:0/2849162996 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f309813c3e0 0x7f309813e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:21.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f3097fff640 1 --2- 172.21.15.37:0/2849162996 >> v2:172.21.15.37:3300/0 conn(0x7f30980a4170 0x7f30980a45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:21.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f309d045640 10 monclient(hunting): get_auth_request con 0x7f309813c3e0 auth_method 0 2024-04-26T00:34:21.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f309d045640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:21.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f309d045640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:21.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f309d045640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:21.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f3097fff640 10 monclient(hunting): get_auth_request con 0x7f30980a4170 auth_method 0 2024-04-26T00:34:21.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f3097fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:21.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f3097fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:21.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f3097fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:21.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f309d045640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:21.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f309d045640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:21.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f309d045640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:21.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f3097fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:21.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f3097fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:21.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f3097fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:21.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f3097fff640 10 monclient(hunting): handle_auth_done global_id 144405 payload 274 2024-04-26T00:34:21.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f3097fff640 1 -- 172.21.15.37:0/2849162996 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f309813c3e0 msgr2=0x7f309813e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:34:21.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f3097fff640 1 --2- 172.21.15.37:0/2849162996 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f309813c3e0 0x7f309813e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:21.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f3097fff640 1 -- 172.21.15.37:0/2849162996 >> v1:172.21.15.37:6789/0 conn(0x7f30980a3d90 legacy=0x7f309809bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:34:21.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f3097fff640 10 monclient: _finish_hunting 0 2024-04-26T00:34:21.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f3097fff640 1 monclient: found mon.noname-a 2024-04-26T00:34:21.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f3097fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:34:21.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f3097fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:34:21.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f3097fff640 1 -- 172.21.15.37:0/2849162996 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f309809b560 con 0x7f30980a4170 2024-04-26T00:34:21.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.386+0000 7f309d045640 1 --2- 172.21.15.37:0/2849162996 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f309813c3e0 0x7f309813e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:34:21.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.390+0000 7f3097fff640 1 -- 172.21.15.37:0/2849162996 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f30980a4170 msgr2=0x7f30980a45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:21.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.390+0000 7f3097fff640 1 --2- 172.21.15.37:0/2849162996 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f30980a4170 0x7f30980a45e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f30800097c0 tx=0x7f308001c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e83e4258fdf67916 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:21.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.390+0000 7f30977fe640 1 -- 172.21.15.37:0/2849162996 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30800b0070 con 0x7f30980a4170 2024-04-26T00:34:21.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.390+0000 7f30977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:21.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.390+0000 7f30977fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:34:21.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.390+0000 7f30977fe640 10 monclient: dump: 2024-04-26T00:34:21.391 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:21.391 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:21.391 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:21.391 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:21.391 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:21.392 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:21.392 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:21.392 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:21.392 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:21.392 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:21.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.390+0000 7f30977fe640 10 monclient: _finish_auth 0 2024-04-26T00:34:21.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.390+0000 7f30977fe640 10 monclient: _check_auth_tickets 2024-04-26T00:34:21.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.390+0000 7f30977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:34:21.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.390+0000 7f30977fe640 1 -- 172.21.15.37:0/2849162996 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30800ab600 con 0x7f30980a4170 2024-04-26T00:34:21.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.390+0000 7f30977fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:34:21.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.390+0000 7f30977fe640 1 -- 172.21.15.37:0/2849162996 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30800ab9d0 con 0x7f30980a4170 2024-04-26T00:34:21.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.390+0000 7f30977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:21.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.390+0000 7f30977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:34:21.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.390+0000 7f30977fe640 10 monclient: dump: 2024-04-26T00:34:21.392 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:21.392 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:21.393 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:21.393 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:21.393 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:21.393 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:21.393 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:21.393 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:21.393 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:21.393 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:21.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.390+0000 7f309eaa8640 5 monclient: authenticate success, global_id 144405 2024-04-26T00:34:21.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.390+0000 7f309eaa8640 10 monclient: get_monmap_and_config success 2024-04-26T00:34:21.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.390+0000 7f309eaa8640 4 set_mon_vals no callback set 2024-04-26T00:34:21.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.390+0000 7f309eaa8640 10 monclient: shutdown 2024-04-26T00:34:21.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.390+0000 7f309eaa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:34:21.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.390+0000 7f309eaa8640 1 -- 172.21.15.37:0/2849162996 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f30980a4170 msgr2=0x7f30980a45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:21.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.390+0000 7f309eaa8640 1 --2- 172.21.15.37:0/2849162996 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f30980a4170 0x7f30980a45e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f30800097c0 tx=0x7f308001c320 comp rx=0 tx=0).stop 2024-04-26T00:34:21.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.390+0000 7f309eaa8640 1 -- 172.21.15.37:0/2849162996 shutdown_connections 2024-04-26T00:34:21.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.390+0000 7f309eaa8640 1 --2- 172.21.15.37:0/2849162996 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f309813c3e0 0x7f309813e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:21.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.390+0000 7f309eaa8640 1 --2- 172.21.15.37:0/2849162996 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f30980a4170 0x7f30980a45e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:21.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.390+0000 7f309eaa8640 1 -- 172.21.15.37:0/2849162996 >> 172.21.15.37:0/2849162996 conn(0x7f30980aa0a0 msgr2=0x7f30980a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:34:21.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.390+0000 7f309eaa8640 1 -- 172.21.15.37:0/2849162996 shutdown_connections 2024-04-26T00:34:21.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.390+0000 7f309eaa8640 1 -- 172.21.15.37:0/2849162996 wait complete. 2024-04-26T00:34:21.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.390+0000 7f309eaa8640 10 monclient: build_initial_monmap 2024-04-26T00:34:21.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.390+0000 7f309eaa8640 10 monclient: monmap: 2024-04-26T00:34:21.394 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:34:21.394 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:34:21.394 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:34:21.394 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:34:21.394 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:34:21.394 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:34:21.394 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:34:21.395 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:34:21.395 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:34:21.395 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:21.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.390+0000 7f309eaa8640 1 Processor -- start 2024-04-26T00:34:21.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f309eaa8640 1 -- start start 2024-04-26T00:34:21.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f309eaa8640 10 monclient: init 2024-04-26T00:34:21.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f309eaa8640 10 monclient: _reopen_session rank -1 2024-04-26T00:34:21.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f309eaa8640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:34:21.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f309eaa8640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f30980a3d90 0x7f309811ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:21.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f309eaa8640 10 monclient(hunting): picked mon.noname-a con 0x7f30980a3d90 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:34:21.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f309eaa8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f30980a4170 0x7f3098136d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:21.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f309eaa8640 10 monclient(hunting): picked mon.noname-c con 0x7f30980a4170 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:34:21.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f309eaa8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f309813c3e0 0x7f30981cec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:21.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f309eaa8640 10 monclient(hunting): picked mon.noname-b con 0x7f309813c3e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:34:21.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f309eaa8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:21.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f309eaa8640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f30981418c0 con 0x7f30980a3d90 2024-04-26T00:34:21.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f309eaa8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:21.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f309eaa8640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3098141740 con 0x7f309813c3e0 2024-04-26T00:34:21.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f309eaa8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:21.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f309eaa8640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f3098141a40 con 0x7f30980a4170 2024-04-26T00:34:21.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f309eaa8640 10 monclient(hunting): _renew_subs 2024-04-26T00:34:21.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f309eaa8640 10 monclient(hunting): authenticate will time out at 1842.700515s 2024-04-26T00:34:21.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f309c844640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f30980a3d90 0x7f309811ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:21.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f3097fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f30980a4170 0x7f3098136d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:21.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f309c844640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f30980a3d90 0x7f309811ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33740/0 (socket says 172.21.15.37:33740) 2024-04-26T00:34:21.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f309c844640 1 -- 172.21.15.37:0/96882090 learned_addr learned my addr 172.21.15.37:0/96882090 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:34:21.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f309c844640 10 monclient(hunting): get_auth_request con 0x7f30980a3d90 auth_method 0 2024-04-26T00:34:21.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f309c844640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:21.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f309c844640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:21.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f309c844640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:21.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f3097fff640 10 monclient(hunting): get_auth_request con 0x7f30980a4170 auth_method 0 2024-04-26T00:34:21.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f3097fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:21.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f3097fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:21.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f3097fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:21.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f309c844640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:21.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f309c844640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:21.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f309c844640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:21.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f3097fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:21.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f3097fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:21.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f3097fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:21.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f309d045640 1 --2- 172.21.15.37:0/96882090 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f309813c3e0 0x7f30981cec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:21.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f3097fff640 10 monclient(hunting): handle_auth_done global_id 154271 payload 824 2024-04-26T00:34:21.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f3097fff640 1 -- 172.21.15.37:0/96882090 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f309813c3e0 msgr2=0x7f30981cec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:21.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f3097fff640 1 --2- 172.21.15.37:0/96882090 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f309813c3e0 0x7f30981cec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:21.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f3097fff640 1 -- 172.21.15.37:0/96882090 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f30980a3d90 msgr2=0x7f309811ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:21.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f3097fff640 1 --2- 172.21.15.37:0/96882090 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f30980a3d90 0x7f309811ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:21.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f3097fff640 10 monclient: _finish_hunting 0 2024-04-26T00:34:21.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f3097fff640 1 monclient: found mon.noname-c 2024-04-26T00:34:21.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f3097fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:34:21.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f3097fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:34:21.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f3097fff640 1 -- 172.21.15.37:0/96882090 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f309809ada0 con 0x7f30980a4170 2024-04-26T00:34:21.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f309d045640 10 monclient: get_auth_request con 0x7f309813c3e0 auth_method 0 2024-04-26T00:34:21.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f309c844640 1 --2- 172.21.15.37:0/96882090 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f30980a3d90 0x7f309811ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:34:21.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f309d045640 1 --2- 172.21.15.37:0/96882090 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f309813c3e0 0x7f30981cec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:34:21.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f3097fff640 1 -- 172.21.15.37:0/96882090 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f30980a4170 msgr2=0x7f3098136d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:21.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f3097fff640 1 --2- 172.21.15.37:0/96882090 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f30980a4170 0x7f3098136d80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f309806e0e0 tx=0x7f30800190c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:21.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f30957fa640 1 -- 172.21.15.37:0/96882090 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30800b0040 con 0x7f30980a4170 2024-04-26T00:34:21.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f30957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:21.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f30957fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:34:21.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f30957fa640 10 monclient: dump: 2024-04-26T00:34:21.404 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:21.404 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:21.404 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:21.404 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:21.404 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:21.404 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:21.404 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:21.404 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:21.404 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:21.404 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:21.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f30957fa640 10 monclient: _finish_auth 0 2024-04-26T00:34:21.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f30957fa640 10 monclient: _check_auth_tickets 2024-04-26T00:34:21.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f30957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:34:21.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f30957fa640 1 -- 172.21.15.37:0/96882090 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30800b9070 con 0x7f30980a4170 2024-04-26T00:34:21.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f30957fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:34:21.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f30957fa640 1 -- 172.21.15.37:0/96882090 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30800aa9f0 con 0x7f30980a4170 2024-04-26T00:34:21.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f30957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:21.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f30957fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:34:21.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f30957fa640 10 monclient: dump: 2024-04-26T00:34:21.405 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:21.405 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:21.405 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:21.405 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:21.405 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:21.405 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:21.405 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:21.405 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:21.406 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:21.406 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:21.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f3096ffd640 4 set_mon_vals no callback set 2024-04-26T00:34:21.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f309eaa8640 5 monclient: authenticate success, global_id 154271 2024-04-26T00:34:21.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f309eaa8640 10 monclient: _renew_subs 2024-04-26T00:34:21.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f309eaa8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:34:21.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f309eaa8640 1 -- 172.21.15.37:0/96882090 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f309813b9a0 con 0x7f30980a4170 2024-04-26T00:34:21.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f309eaa8640 10 monclient: _renew_subs 2024-04-26T00:34:21.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f309eaa8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:34:21.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.394+0000 7f309eaa8640 1 -- 172.21.15.37:0/96882090 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3098121710 con 0x7f30980a4170 2024-04-26T00:34:21.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.398+0000 7f309eaa8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:34:21.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.398+0000 7f309eaa8640 1 --2- 172.21.15.37:0/96882090 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3098120e70 0x7f30981cf1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:21.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.398+0000 7f309eaa8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:21.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.398+0000 7f309eaa8640 1 -- 172.21.15.37:0/96882090 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f309804f510 con 0x7f3098120e70 2024-04-26T00:34:21.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.398+0000 7f309c844640 1 --2- 172.21.15.37:0/96882090 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3098120e70 0x7f30981cf1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:21.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.398+0000 7f309c844640 10 monclient: get_auth_request con 0x7f3098120e70 auth_method 0 2024-04-26T00:34:21.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.398+0000 7f309c844640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:21.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.398+0000 7f309c844640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:21.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.398+0000 7f309c844640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:21.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.398+0000 7f309c844640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:21.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.398+0000 7f309c844640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:21.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.398+0000 7f309c844640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:21.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.398+0000 7f30957fa640 1 -- 172.21.15.37:0/96882090 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f30800ab4c0 con 0x7f30980a4170 2024-04-26T00:34:21.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.398+0000 7f309c844640 10 monclient(hunting): handle_auth_done global_id 154274 payload 824 2024-04-26T00:34:21.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.398+0000 7f309c844640 1 -- 172.21.15.37:0/96882090 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f30980008e0 con 0x7f3098120e70 2024-04-26T00:34:21.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.398+0000 7f30957fa640 1 --2- 172.21.15.37:0/96882090 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3078035e80 0x7f3078038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:21.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.398+0000 7f30957fa640 1 -- 172.21.15.37:0/96882090 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(561..561 src has 370..561) ==== 3959+0+0 (secure 0 0 0) 0x7f30800e26a0 con 0x7f30980a4170 2024-04-26T00:34:21.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.398+0000 7f309d045640 1 --2- 172.21.15.37:0/96882090 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3078035e80 0x7f3078038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:21.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.398+0000 7f309d045640 10 monclient: get_auth_request con 0x7f3078035e80 auth_method 0 2024-04-26T00:34:21.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.398+0000 7f309c844640 1 -- 172.21.15.37:0/96882090 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3098120e70 msgr2=0x7f30981cf1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:21.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.398+0000 7f309c844640 1 --2- 172.21.15.37:0/96882090 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3098120e70 0x7f30981cf1d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f309000d3d0 tx=0x7f309000d890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:21.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.398+0000 7f30957fa640 1 -- 172.21.15.37:0/96882090 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30900075b0 con 0x7f3098120e70 2024-04-26T00:34:21.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.398+0000 7f30957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:21.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.398+0000 7f30957fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:34:21.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.398+0000 7f30957fa640 10 monclient: dump: 2024-04-26T00:34:21.408 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:21.408 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:21.408 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:21.408 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:21.408 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:21.408 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:21.408 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:21.408 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:21.408 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:21.408 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:21.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.398+0000 7f30957fa640 1 -- 172.21.15.37:0/96882090 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3090018070 con 0x7f3098120e70 2024-04-26T00:34:21.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.398+0000 7f30957fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:34:21.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.398+0000 7f30957fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:34:21.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.398+0000 7f30957fa640 1 -- 172.21.15.37:0/96882090 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3098120e70 msgr2=0x7f30981cf1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:21.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.398+0000 7f30957fa640 1 --2- 172.21.15.37:0/96882090 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3098120e70 0x7f30981cf1d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f309000d3d0 tx=0x7f309000d890 comp rx=0 tx=0).stop 2024-04-26T00:34:21.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.398+0000 7f30957fa640 1 -- 172.21.15.37:0/96882090 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3098120e70 msgr2=0x7f30981cf1d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:34:21.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.398+0000 7f30957fa640 1 --2- 172.21.15.37:0/96882090 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3098120e70 0x7f30981cf1d0 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f309000d3d0 tx=0x7f309000d890 comp rx=0 tx=0).stop 2024-04-26T00:34:21.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.402+0000 7f309d045640 1 --2- 172.21.15.37:0/96882090 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3078035e80 0x7f3078038340 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f3088009ac0 tx=0x7f3088008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:21.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.406+0000 7f309eaa8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:34:21.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.406+0000 7f309eaa8640 1 --2- 172.21.15.37:0/96882090 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f309813a300 0x7f30981cf710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:21.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.406+0000 7f309eaa8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:21.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.406+0000 7f309eaa8640 1 -- 172.21.15.37:0/96882090 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f309813b330 con 0x7f309813a300 2024-04-26T00:34:21.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.406+0000 7f309c844640 1 --2- 172.21.15.37:0/96882090 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f309813a300 0x7f30981cf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:21.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.406+0000 7f309c844640 10 monclient: get_auth_request con 0x7f309813a300 auth_method 0 2024-04-26T00:34:21.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.406+0000 7f309c844640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:21.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.406+0000 7f309c844640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:21.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.406+0000 7f309c844640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:21.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.406+0000 7f309c844640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:21.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.406+0000 7f309c844640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:21.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.406+0000 7f309c844640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:21.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.406+0000 7f309c844640 10 monclient(hunting): handle_auth_done global_id 154277 payload 824 2024-04-26T00:34:21.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.406+0000 7f309c844640 1 -- 172.21.15.37:0/96882090 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f30981372c0 con 0x7f309813a300 2024-04-26T00:34:21.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.410+0000 7f309c844640 1 -- 172.21.15.37:0/96882090 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f309813a300 msgr2=0x7f30981cf710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:21.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.410+0000 7f309c844640 1 --2- 172.21.15.37:0/96882090 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f309813a300 0x7f30981cf710 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f309000d3d0 tx=0x7f3090004cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:21.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.410+0000 7f30957fa640 1 -- 172.21.15.37:0/96882090 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3090013600 con 0x7f309813a300 2024-04-26T00:34:21.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.410+0000 7f30957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:21.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.410+0000 7f30957fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:34:21.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.410+0000 7f30957fa640 10 monclient: dump: 2024-04-26T00:34:21.412 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:21.412 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:21.412 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:21.412 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:21.412 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:21.412 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:21.412 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:21.413 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:21.413 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:21.413 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:21.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.410+0000 7f30957fa640 1 -- 172.21.15.37:0/96882090 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4717 (secure 0 0 0) 0x7f3090013600 con 0x7f309813a300 2024-04-26T00:34:21.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.410+0000 7f30957fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:34:21.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.410+0000 7f30957fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:34:21.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.410+0000 7f30957fa640 1 -- 172.21.15.37:0/96882090 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f309813a300 msgr2=0x7f30981cf710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:21.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.410+0000 7f30957fa640 1 --2- 172.21.15.37:0/96882090 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f309813a300 0x7f30981cf710 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f309000d3d0 tx=0x7f3090004cd0 comp rx=0 tx=0).stop 2024-04-26T00:34:21.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.410+0000 7f30957fa640 1 -- 172.21.15.37:0/96882090 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f309813a300 msgr2=0x7f30981cf710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:34:21.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.410+0000 7f30957fa640 1 --2- 172.21.15.37:0/96882090 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f309813a300 0x7f30981cf710 secure :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f309000d3d0 tx=0x7f3090004cd0 comp rx=0 tx=0).stop 2024-04-26T00:34:21.414 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:34:21.414 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:34:21.414 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:34:21.414 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:34:21.414 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 108, 2024-04-26T00:34:21.414 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 98930, 2024-04-26T00:34:21.414 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:34:21.414 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:34:21.415 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:34:21.415 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:34:21.415 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:21.415 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:34:21.415 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:34:21.415 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:34:21.415 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:34:21.415 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:34:21.415 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:34:21.415 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:34:21.415 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:34:21.415 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:34:21.415 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:34:21.415 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:34:21.415 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:34:21.416 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:34:21.416 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:34:21.416 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:34:21.416 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:21.416 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:34:21.416 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:34:21.416 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:34:21.416 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:34:21.416 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:34:21.416 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:34:21.416 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:34:21.416 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:34:21.416 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:34:21.416 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:34:21.416 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:34:21.417 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:34:21.417 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:34:21.417 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:21.417 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:21.417 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:34:21.417 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:34:21.417 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:34:21.417 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:34:21.417 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:34:21.417 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:34:21.417 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:34:21.417 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:34:21.417 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:34:21.417 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:34:21.417 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:34:21.417 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:34:21.418 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:34:21.418 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:34:21.418 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:34:21.418 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:34:21.418 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:34:21.418 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:34:21.418 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:34:21.418 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:34:21.418 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:34:21.418 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:34:21.418 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:34:21.418 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:34:21.418 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:34:21.419 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:34:21.419 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:34:21.419 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:34:21.419 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:21.419 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:34:21.419 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:21.419 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:34:21.419 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:21.420 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:34:21.420 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:34:21.420 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:34:21.420 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:34:21.420 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:21.420 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:34:21.420 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:34:21.420 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:21.420 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:21.420 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:21.420 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:34:21.420 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:34:21.420 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:21.420 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:21.421 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:21.421 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:21.421 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:34:21.421 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:34:21.421 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:34:21.421 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:34:21.421 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:34:21.421 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:21.421 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:21.421 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:34:21.421 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:34:21.421 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:34:21.421 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:34:21.421 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:21.421 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:34:21.422 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:34:21.422 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:21.422 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:21.422 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:21.422 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:34:21.422 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:34:21.422 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:21.422 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:21.422 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:21.422 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:21.422 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:34:21.422 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:34:21.422 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:34:21.422 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:34:21.422 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:34:21.423 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:21.425 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:21.425 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:34:21.425 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:34:21.425 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:34:21.425 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:34:21.425 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:21.425 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:34:21.426 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:34:21.426 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:21.426 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:21.426 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:21.426 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:34:21.426 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:34:21.426 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:21.426 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:21.426 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:21.426 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:21.426 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:34:21.427 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:34:21.427 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:34:21.427 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:34:21.427 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:34:21.427 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:21.427 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:21.428 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:21.428 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:34:21.428 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:34:21.428 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:21.428 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:34:21.428 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:34:21.428 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:34:21.428 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:21.428 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:21.428 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:34:21.428 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:21.428 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:34:21.428 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:34:21.428 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:34:21.428 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:21.429 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:21.429 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:34:21.429 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:21.429 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:34:21.429 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:34:21.429 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:34:21.429 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:21.429 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:21.429 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:21.429 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:34:21.429 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:34:21.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.410+0000 7f309eaa8640 1 -- 172.21.15.37:0/96882090 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3078035e80 msgr2=0x7f3078038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:21.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.410+0000 7f309eaa8640 1 --2- 172.21.15.37:0/96882090 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3078035e80 0x7f3078038340 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f3088009ac0 tx=0x7f3088008040 comp rx=0 tx=0).stop 2024-04-26T00:34:21.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.410+0000 7f309eaa8640 10 monclient: shutdown 2024-04-26T00:34:21.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.410+0000 7f309eaa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:34:21.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.410+0000 7f309eaa8640 1 -- 172.21.15.37:0/96882090 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f30980a4170 msgr2=0x7f3098136d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:21.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.410+0000 7f309eaa8640 1 --2- 172.21.15.37:0/96882090 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f30980a4170 0x7f3098136d80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f309806e0e0 tx=0x7f30800190c0 comp rx=0 tx=0).stop 2024-04-26T00:34:21.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.410+0000 7f309d045640 1 -- 172.21.15.37:0/96882090 reap_dead start 2024-04-26T00:34:21.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.410+0000 7f309d045640 1 -- 172.21.15.37:0/96882090 reap_dead start 2024-04-26T00:34:21.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.410+0000 7f309eaa8640 1 -- 172.21.15.37:0/96882090 shutdown_connections 2024-04-26T00:34:21.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.410+0000 7f309eaa8640 1 -- 172.21.15.37:0/96882090 >> 172.21.15.37:0/96882090 conn(0x7f30980aa0a0 msgr2=0x7f30980a8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:34:21.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.410+0000 7f309eaa8640 1 -- 172.21.15.37:0/96882090 shutdown_connections 2024-04-26T00:34:21.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.410+0000 7f309eaa8640 1 -- 172.21.15.37:0/96882090 wait complete. 2024-04-26T00:34:21.432 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:34:21.432 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:34:21.432 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:34:21.523 INFO:tasks.rados.rados.0.smithi045.stdout:2450: done (2 left) 2024-04-26T00:34:21.523 INFO:tasks.rados.rados.0.smithi045.stdout:2451: copy_from oid 28 from oid 27 current snap is 275 2024-04-26T00:34:21.523 INFO:tasks.rados.rados.0.smithi045.stdout:2449: finishing write tid 1 to smithi04526579-33 2024-04-26T00:34:21.523 INFO:tasks.rados.rados.0.smithi045.stdout:2449: finishing write tid 2 to smithi04526579-33 2024-04-26T00:34:21.523 INFO:tasks.rados.rados.0.smithi045.stdout:2449: finishing write tid 3 to smithi04526579-33 2024-04-26T00:34:21.523 INFO:tasks.rados.rados.0.smithi045.stdout:2449: finishing write tid 4 to smithi04526579-33 2024-04-26T00:34:21.523 INFO:tasks.rados.rados.0.smithi045.stdout:2449: oid 33 updating version 0 to 2119 2024-04-26T00:34:21.523 INFO:tasks.rados.rados.0.smithi045.stdout:2449: oid 33 version 2119 is already newer than 2118 2024-04-26T00:34:21.523 INFO:tasks.rados.rados.0.smithi045.stdout:2449: oid 33 version 2119 is already newer than 2117 2024-04-26T00:34:21.523 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 2119 (ObjNum 889 snap 275 seq_num 889) dirty exists 2024-04-26T00:34:21.523 INFO:tasks.rados.rados.0.smithi045.stdout:2449: left oid 33 (ObjNum 889 snap 275 seq_num 889) 2024-04-26T00:34:21.523 INFO:tasks.rados.rados.0.smithi045.stdout:2448: done (2 left) 2024-04-26T00:34:21.524 INFO:tasks.rados.rados.0.smithi045.stdout:2449: done (1 left) 2024-04-26T00:34:21.524 INFO:tasks.rados.rados.0.smithi045.stdout:2452: write (excl) oid 11 current snap is 275 2024-04-26T00:34:21.524 INFO:tasks.rados.rados.0.smithi045.stdout:2452: seq_num 890 ranges {558230=584911,1897327=527644,2923993=34237} 2024-04-26T00:34:21.542 INFO:tasks.rados.rados.0.smithi045.stdout:2452: writing smithi04526579-11 from 558230 to 1143141 tid 1 2024-04-26T00:34:21.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.554+0000 7f8a9cfde640 10 monclient: get_monmap_and_config 2024-04-26T00:34:21.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.554+0000 7f8a9cfde640 10 monclient: build_initial_monmap 2024-04-26T00:34:21.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.554+0000 7f8a9cfde640 10 monclient: monmap: 2024-04-26T00:34:21.551 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:34:21.551 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:21.551 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:34:21.559123+0000 2024-04-26T00:34:21.551 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:34:21.559123+0000 2024-04-26T00:34:21.552 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:34:21.552 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:21.552 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:34:21.552 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:34:21.552 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:34:21.552 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:34:21.552 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:34:21.552 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:21.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.558+0000 7f8a9cfde640 1 Processor -- start 2024-04-26T00:34:21.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.558+0000 7f8a9cfde640 1 -- start start 2024-04-26T00:34:21.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.558+0000 7f8a9cfde640 10 monclient: init 2024-04-26T00:34:21.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.558+0000 7f8a9cfde640 10 monclient: _reopen_session rank -1 2024-04-26T00:34:21.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.558+0000 7f8a9cfde640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-04-26T00:34:21.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.558+0000 7f8a9cfde640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8a980a2fe0 addr v1:172.21.15.37:6789/0 2024-04-26T00:34:21.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.558+0000 7f8a9cfde640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8a98137dd0 addr v1:172.21.15.45:6789/0 2024-04-26T00:34:21.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.558+0000 7f8a9cfde640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a9813b8e0 0x7f8a9813dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:21.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.558+0000 7f8a9cfde640 10 monclient(hunting): picked mon.noname-b con 0x7f8a9813b8e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:34:21.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.558+0000 7f8a9cfde640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a98072ec0 con 0x7f8a980a2fe0 2024-04-26T00:34:21.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.558+0000 7f8a9cfde640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a98134ac0 con 0x7f8a98137dd0 2024-04-26T00:34:21.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.558+0000 7f8a9cfde640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:21.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.558+0000 7f8a9cfde640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8a980a33c0 con 0x7f8a9813b8e0 2024-04-26T00:34:21.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.558+0000 7f8a9cfde640 10 monclient(hunting): _renew_subs 2024-04-26T00:34:21.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.558+0000 7f8a9cfde640 10 monclient(hunting): authenticate will time out at 1842.865295s 2024-04-26T00:34:21.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.558+0000 7f8a9659c640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f8a980a2fe0 0x7f8a981351b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:33376/0 (socket says 172.21.15.37:33376) 2024-04-26T00:34:21.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.558+0000 7f8a96d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a9813b8e0 0x7f8a9813dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:21.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.558+0000 7f8a9659c640 1 -- 172.21.15.37:0/3458664352 learned_addr learned my addr 172.21.15.37:0/3458664352 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:34:21.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.558+0000 7f8a96d9d640 10 monclient(hunting): get_auth_request con 0x7f8a9813b8e0 auth_method 0 2024-04-26T00:34:21.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.558+0000 7f8a96d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:21.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.558+0000 7f8a96d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:21.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.558+0000 7f8a96d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:21.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.558+0000 7f8a96d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:21.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.558+0000 7f8a96d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:21.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.558+0000 7f8a96d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:21.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.558+0000 7f8a96d9d640 10 monclient(hunting): handle_auth_done global_id 154280 payload 274 2024-04-26T00:34:21.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.558+0000 7f8a96d9d640 1 -- 172.21.15.37:0/3458664352 >> v1:172.21.15.45:6789/0 conn(0x7f8a98137dd0 legacy=0x7f8a9813a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:34:21.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.558+0000 7f8a96d9d640 1 -- 172.21.15.37:0/3458664352 >> v1:172.21.15.37:6789/0 conn(0x7f8a980a2fe0 legacy=0x7f8a981351b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:34:21.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.558+0000 7f8a96d9d640 10 monclient: _finish_hunting 0 2024-04-26T00:34:21.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.558+0000 7f8a96d9d640 1 monclient: found mon.noname-b 2024-04-26T00:34:21.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.558+0000 7f8a96d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:34:21.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.558+0000 7f8a96d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:34:21.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.558+0000 7f8a96d9d640 1 -- 172.21.15.37:0/3458664352 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a98134d60 con 0x7f8a9813b8e0 2024-04-26T00:34:21.557 INFO:tasks.rados.rados.0.smithi045.stdout:2452: writing smithi04526579-11 from 1897327 to 2424971 tid 2 2024-04-26T00:34:21.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.562+0000 7f8a96d9d640 1 -- 172.21.15.37:0/3458664352 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a9813b8e0 msgr2=0x7f8a9813dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:21.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.562+0000 7f8a96d9d640 1 --2- 172.21.15.37:0/3458664352 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a9813b8e0 0x7f8a9813dcd0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8a90009dc0 tx=0x7f8a9001c5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efb6edfae6ac0394 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:21.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.562+0000 7f8a9559a640 1 -- 172.21.15.37:0/3458664352 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a900ad070 con 0x7f8a9813b8e0 2024-04-26T00:34:21.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.562+0000 7f8a9559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:21.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.562+0000 7f8a9559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:34:21.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.562+0000 7f8a9559a640 10 monclient: dump: 2024-04-26T00:34:21.559 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:21.559 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:21.559 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:21.559 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:21.559 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:21.559 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:21.559 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:21.559 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:21.560 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:21.560 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:21.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.562+0000 7f8a9559a640 10 monclient: _finish_auth 0 2024-04-26T00:34:21.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.562+0000 7f8a9559a640 10 monclient: _check_auth_tickets 2024-04-26T00:34:21.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.562+0000 7f8a9559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:34:21.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.562+0000 7f8a9559a640 1 -- 172.21.15.37:0/3458664352 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a900a9750 con 0x7f8a9813b8e0 2024-04-26T00:34:21.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.562+0000 7f8a9cfde640 5 monclient: authenticate success, global_id 154280 2024-04-26T00:34:21.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.562+0000 7f8a9cfde640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:34:21.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.562+0000 7f8a9559a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:34:21.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.562+0000 7f8a9cfde640 10 monclient: get_monmap_and_config success 2024-04-26T00:34:21.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.562+0000 7f8a9559a640 1 -- 172.21.15.37:0/3458664352 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a900a9b20 con 0x7f8a9813b8e0 2024-04-26T00:34:21.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.562+0000 7f8a9559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:21.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.562+0000 7f8a9cfde640 4 set_mon_vals no callback set 2024-04-26T00:34:21.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.562+0000 7f8a9559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:34:21.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.562+0000 7f8a9559a640 10 monclient: dump: 2024-04-26T00:34:21.561 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:21.561 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:21.561 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:21.561 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:21.561 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:21.561 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:21.561 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:21.561 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:21.561 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:21.561 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:21.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.562+0000 7f8a9cfde640 10 monclient: shutdown 2024-04-26T00:34:21.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.562+0000 7f8a9cfde640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:34:21.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.562+0000 7f8a9cfde640 1 -- 172.21.15.37:0/3458664352 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a9813b8e0 msgr2=0x7f8a9813dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:21.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.562+0000 7f8a9cfde640 1 --2- 172.21.15.37:0/3458664352 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a9813b8e0 0x7f8a9813dcd0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8a90009dc0 tx=0x7f8a9001c5f0 comp rx=0 tx=0).stop 2024-04-26T00:34:21.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.562+0000 7f8a9cfde640 1 -- 172.21.15.37:0/3458664352 shutdown_connections 2024-04-26T00:34:21.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.562+0000 7f8a9cfde640 1 --2- 172.21.15.37:0/3458664352 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a9813b8e0 0x7f8a9813dcd0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:21.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.562+0000 7f8a9cfde640 1 -- 172.21.15.37:0/3458664352 >> 172.21.15.37:0/3458664352 conn(0x7f8a980a24c0 msgr2=0x7f8a980a28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:34:21.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.562+0000 7f8a9cfde640 1 -- 172.21.15.37:0/3458664352 shutdown_connections 2024-04-26T00:34:21.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.562+0000 7f8a9cfde640 1 -- 172.21.15.37:0/3458664352 wait complete. 2024-04-26T00:34:21.563 INFO:tasks.rados.rados.0.smithi045.stdout:2452: writing smithi04526579-11 from 2923993 to 2958230 tid 3 2024-04-26T00:34:21.563 INFO:tasks.rados.rados.0.smithi045.stdout:2453: delete oid 36 current snap is 275 2024-04-26T00:34:21.563 INFO:tasks.rados.rados.0.smithi045.stdout:2452: finishing write tid 1 to smithi04526579-11 2024-04-26T00:34:21.563 INFO:tasks.rados.rados.0.smithi045.stdout:2451: finishing copy_from racing read to smithi04526579-28 2024-04-26T00:34:21.563 INFO:tasks.rados.rados.0.smithi045.stdout:2451: finishing copy_from to smithi04526579-28 2024-04-26T00:34:21.563 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 1597 (ObjNum 849 snap 262 seq_num 849) dirty exists 2024-04-26T00:34:21.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.562+0000 7f8a9cfde640 10 monclient: build_initial_monmap 2024-04-26T00:34:21.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.562+0000 7f8a9cfde640 10 monclient: monmap: 2024-04-26T00:34:21.567 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:34:21.567 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:34:21.567 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:34:21.567 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:34:21.567 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:34:21.568 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:34:21.568 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:34:21.568 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:34:21.568 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:34:21.568 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:21.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a9cfde640 1 Processor -- start 2024-04-26T00:34:21.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a9cfde640 1 -- start start 2024-04-26T00:34:21.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a9cfde640 10 monclient: init 2024-04-26T00:34:21.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a9cfde640 10 monclient: _reopen_session rank -1 2024-04-26T00:34:21.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a9cfde640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:34:21.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a9cfde640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a980a2fe0 0x7f8a9812f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:21.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a9cfde640 10 monclient(hunting): picked mon.noname-a con 0x7f8a980a2fe0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:34:21.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a9cfde640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a98137dd0 0x7f8a9812f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:21.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a9cfde640 10 monclient(hunting): picked mon.noname-b con 0x7f8a98137dd0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:34:21.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a9cfde640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a9813b8e0 0x7f8a9812fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:21.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a9cfde640 10 monclient(hunting): picked mon.noname-c con 0x7f8a9813b8e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:34:21.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a9cfde640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:21.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a9cfde640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f8a9806ed70 con 0x7f8a980a2fe0 2024-04-26T00:34:21.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a9cfde640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:21.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a9cfde640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8a9806ebf0 con 0x7f8a98137dd0 2024-04-26T00:34:21.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a9cfde640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:21.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a9cfde640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8a98140a30 con 0x7f8a9813b8e0 2024-04-26T00:34:21.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a9cfde640 10 monclient(hunting): _renew_subs 2024-04-26T00:34:21.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a9cfde640 10 monclient(hunting): authenticate will time out at 1842.872645s 2024-04-26T00:34:21.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a96d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a9813b8e0 0x7f8a9812fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:21.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a96d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a9813b8e0 0x7f8a9812fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:58026/0 (socket says 172.21.15.37:58026) 2024-04-26T00:34:21.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a96d9d640 1 -- 172.21.15.37:0/1124860275 learned_addr learned my addr 172.21.15.37:0/1124860275 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:34:21.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a96d9d640 10 monclient(hunting): get_auth_request con 0x7f8a9813b8e0 auth_method 0 2024-04-26T00:34:21.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a9659c640 1 --2- 172.21.15.37:0/1124860275 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a980a2fe0 0x7f8a9812f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:21.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a96d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:21.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a96d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:21.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a96d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:21.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a9659c640 10 monclient(hunting): get_auth_request con 0x7f8a980a2fe0 auth_method 0 2024-04-26T00:34:21.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a9659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:21.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a9659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:21.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a9659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:21.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a96d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:21.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a96d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:21.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a96d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:21.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a95d9b640 1 --2- 172.21.15.37:0/1124860275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a98137dd0 0x7f8a9812f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:21.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a95d9b640 10 monclient(hunting): get_auth_request con 0x7f8a98137dd0 auth_method 0 2024-04-26T00:34:21.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a95d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:21.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a95d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:21.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a95d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:21.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a9659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:21.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a9659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:21.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a9659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:21.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a96d9d640 10 monclient(hunting): handle_auth_done global_id 154283 payload 824 2024-04-26T00:34:21.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a95d9b640 1 -- 172.21.15.37:0/1124860275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a98137dd0 msgr2=0x7f8a9812f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:34:21.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a96d9d640 1 -- 172.21.15.37:0/1124860275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a98137dd0 msgr2=0x7f8a9812f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:21.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a95d9b640 1 -- 172.21.15.37:0/1124860275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a98137dd0 msgr2=0x7f8a9812f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:34:21.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a95d9b640 1 --2- 172.21.15.37:0/1124860275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a98137dd0 0x7f8a9812f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:34:21.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a95d9b640 1 --2- 172.21.15.37:0/1124860275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a98137dd0 0x7f8a9812f8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:34:21.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a96d9d640 1 --2- 172.21.15.37:0/1124860275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a98137dd0 0x7f8a9812f8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:21.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a96d9d640 1 -- 172.21.15.37:0/1124860275 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a980a2fe0 msgr2=0x7f8a9812f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:21.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a96d9d640 1 --2- 172.21.15.37:0/1124860275 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a980a2fe0 0x7f8a9812f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:21.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a96d9d640 10 monclient: _finish_hunting 0 2024-04-26T00:34:21.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a96d9d640 1 monclient: found mon.noname-c 2024-04-26T00:34:21.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a96d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:34:21.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a96d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:34:21.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a96d9d640 1 -- 172.21.15.37:0/1124860275 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a980a1560 con 0x7f8a9813b8e0 2024-04-26T00:34:21.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a96d9d640 1 -- 172.21.15.37:0/1124860275 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a9813b8e0 msgr2=0x7f8a9812fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:21.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a96d9d640 1 --2- 172.21.15.37:0/1124860275 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a9813b8e0 0x7f8a9812fde0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8a9001aca0 tx=0x7f8a90019390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:21.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a86ffd640 1 -- 172.21.15.37:0/1124860275 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a900ad040 con 0x7f8a9813b8e0 2024-04-26T00:34:21.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:21.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a86ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:34:21.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a86ffd640 10 monclient: dump: 2024-04-26T00:34:21.572 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:21.572 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:21.572 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:21.572 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:21.572 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:21.572 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:21.573 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:21.573 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:21.573 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:21.573 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:21.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a86ffd640 10 monclient: _finish_auth 0 2024-04-26T00:34:21.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a86ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:34:21.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:34:21.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a86ffd640 1 -- 172.21.15.37:0/1124860275 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a900b2a70 con 0x7f8a9813b8e0 2024-04-26T00:34:21.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a86ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:34:21.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a86ffd640 1 -- 172.21.15.37:0/1124860275 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a900b2e60 con 0x7f8a9813b8e0 2024-04-26T00:34:21.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a9cfde640 5 monclient: authenticate success, global_id 154283 2024-04-26T00:34:21.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a9cfde640 10 monclient: _renew_subs 2024-04-26T00:34:21.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a9cfde640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:34:21.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a9cfde640 1 -- 172.21.15.37:0/1124860275 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a98131080 con 0x7f8a9813b8e0 2024-04-26T00:34:21.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:21.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a94d99640 4 set_mon_vals no callback set 2024-04-26T00:34:21.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a86ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:34:21.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a86ffd640 10 monclient: dump: 2024-04-26T00:34:21.574 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:21.574 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:21.574 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:21.574 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:21.574 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:21.574 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:21.574 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:21.574 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:21.575 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:21.575 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:21.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a9cfde640 10 monclient: _renew_subs 2024-04-26T00:34:21.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a9cfde640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:34:21.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.566+0000 7f8a9cfde640 1 -- 172.21.15.37:0/1124860275 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a9809b1d0 con 0x7f8a9813b8e0 2024-04-26T00:34:21.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.570+0000 7f8a9cfde640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:34:21.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.570+0000 7f8a9cfde640 1 --2- 172.21.15.37:0/1124860275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a981cd410 0x7f8a981cd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:21.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.570+0000 7f8a9cfde640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:21.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.570+0000 7f8a9cfde640 1 -- 172.21.15.37:0/1124860275 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8a9804f510 con 0x7f8a981cd410 2024-04-26T00:34:21.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.570+0000 7f8a86ffd640 1 -- 172.21.15.37:0/1124860275 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f8a900b0070 con 0x7f8a9813b8e0 2024-04-26T00:34:21.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.570+0000 7f8a86ffd640 1 --2- 172.21.15.37:0/1124860275 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8a78035eb0 0x7f8a78038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:21.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.570+0000 7f8a9659c640 1 --2- 172.21.15.37:0/1124860275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a981cd410 0x7f8a981cd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:21.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.570+0000 7f8a9659c640 10 monclient: get_auth_request con 0x7f8a981cd410 auth_method 0 2024-04-26T00:34:21.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.570+0000 7f8a9659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:21.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.570+0000 7f8a9659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:21.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.570+0000 7f8a9659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:21.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.570+0000 7f8a86ffd640 1 -- 172.21.15.37:0/1124860275 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(562..562 src has 370..562) ==== 3959+0+0 (secure 0 0 0) 0x7f8a900e3ba0 con 0x7f8a9813b8e0 2024-04-26T00:34:21.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.570+0000 7f8a9659c640 1 -- 172.21.15.37:0/1124860275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a981cd410 msgr2=0x7f8a981cd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:34:21.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.570+0000 7f8a9659c640 1 -- 172.21.15.37:0/1124860275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a981cd410 msgr2=0x7f8a981cd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:34:21.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.570+0000 7f8a9659c640 1 --2- 172.21.15.37:0/1124860275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a981cd410 0x7f8a981cd7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:34:21.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.570+0000 7f8a9659c640 1 --2- 172.21.15.37:0/1124860275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a981cd410 0x7f8a981cd7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:34:21.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.570+0000 7f8a95d9b640 1 --2- 172.21.15.37:0/1124860275 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8a78035eb0 0x7f8a78038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:21.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.570+0000 7f8a95d9b640 10 monclient: get_auth_request con 0x7f8a78035eb0 auth_method 0 2024-04-26T00:34:21.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.574+0000 7f8a95d9b640 1 --2- 172.21.15.37:0/1124860275 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8a78035eb0 0x7f8a78038370 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f8a98137c60 tx=0x7f8a8c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:21.577 INFO:tasks.rados.rados.0.smithi045.stdout:2452: finishing write tid 2 to smithi04526579-11 2024-04-26T00:34:21.577 INFO:tasks.rados.rados.0.smithi045.stdout:2452: finishing write tid 3 to smithi04526579-11 2024-04-26T00:34:21.577 INFO:tasks.rados.rados.0.smithi045.stdout:2451: done (2 left) 2024-04-26T00:34:21.578 INFO:tasks.rados.rados.0.smithi045.stdout:2453: done (1 left) 2024-04-26T00:34:21.578 INFO:tasks.rados.rados.0.smithi045.stdout:2454: rollback oid 42 current snap is 275 2024-04-26T00:34:21.578 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 42 to 235 2024-04-26T00:34:21.578 INFO:tasks.rados.rados.0.smithi045.stdout:2455: write (excl) oid 28 current snap is 275 2024-04-26T00:34:21.578 INFO:tasks.rados.rados.0.smithi045.stdout:2455: seq_num 892 ranges {591844=735409,1897938=531313,2862486=129358} 2024-04-26T00:34:21.578 INFO:tasks.rados.rados.0.smithi045.stdout:2455: writing smithi04526579-28 from 591844 to 1327253 tid 1 2024-04-26T00:34:21.584 INFO:tasks.rados.rados.0.smithi045.stdout:2455: writing smithi04526579-28 from 1897938 to 2429251 tid 2 2024-04-26T00:34:21.589 INFO:tasks.rados.rados.0.smithi045.stdout:2455: writing smithi04526579-28 from 2862486 to 2991844 tid 3 2024-04-26T00:34:21.589 INFO:tasks.rados.rados.0.smithi045.stdout:2456: snap_remove snap 272 2024-04-26T00:34:21.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.774+0000 7f8a9659c640 1 --2- 172.21.15.37:0/1124860275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a981cd410 0x7f8a981cd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:21.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.774+0000 7f8a9659c640 10 monclient: get_auth_request con 0x7f8a981cd410 auth_method 0 2024-04-26T00:34:21.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.774+0000 7f8a9659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:21.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.774+0000 7f8a9659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:21.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.774+0000 7f8a9659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:34:21.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.774+0000 7f8a9659c640 1 -- 172.21.15.37:0/1124860275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a981cd410 msgr2=0x7f8a981cd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:34:21.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.774+0000 7f8a9659c640 1 -- 172.21.15.37:0/1124860275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a981cd410 msgr2=0x7f8a981cd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:34:21.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.774+0000 7f8a9659c640 1 --2- 172.21.15.37:0/1124860275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a981cd410 0x7f8a981cd7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:34:21.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:21.774+0000 7f8a9659c640 1 --2- 172.21.15.37:0/1124860275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a981cd410 0x7f8a981cd7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:34:22.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.178+0000 7f8a9659c640 1 --2- 172.21.15.37:0/1124860275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a981cd410 0x7f8a981cd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:22.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.178+0000 7f8a9659c640 10 monclient: get_auth_request con 0x7f8a981cd410 auth_method 0 2024-04-26T00:34:22.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.178+0000 7f8a9659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:22.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.178+0000 7f8a9659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:22.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.178+0000 7f8a9659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:34:22.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.178+0000 7f8a9659c640 1 -- 172.21.15.37:0/1124860275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a981cd410 msgr2=0x7f8a981cd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:34:22.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.178+0000 7f8a9659c640 1 -- 172.21.15.37:0/1124860275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a981cd410 msgr2=0x7f8a981cd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:34:22.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.178+0000 7f8a9659c640 1 --2- 172.21.15.37:0/1124860275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a981cd410 0x7f8a981cd7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:34:22.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.178+0000 7f8a9659c640 1 --2- 172.21.15.37:0/1124860275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a981cd410 0x7f8a981cd7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:34:22.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.982+0000 7f8a9659c640 1 --2- 172.21.15.37:0/1124860275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a981cd410 0x7f8a981cd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:22.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.982+0000 7f8a9659c640 10 monclient: get_auth_request con 0x7f8a981cd410 auth_method 0 2024-04-26T00:34:22.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.982+0000 7f8a9659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:22.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.982+0000 7f8a9659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:22.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.982+0000 7f8a9659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:34:22.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.982+0000 7f8a9659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:22.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.982+0000 7f8a9659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:22.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.982+0000 7f8a9659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:22.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.982+0000 7f8a9659c640 10 monclient(hunting): handle_auth_done global_id 174100 payload 824 2024-04-26T00:34:22.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.982+0000 7f8a9659c640 1 -- 172.21.15.37:0/1124860275 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8a980008e0 con 0x7f8a981cd410 2024-04-26T00:34:22.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.982+0000 7f8a9659c640 1 -- 172.21.15.37:0/1124860275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a981cd410 msgr2=0x7f8a981cd7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:22.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.982+0000 7f8a9659c640 1 --2- 172.21.15.37:0/1124860275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a981cd410 0x7f8a981cd7f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8a88015590 tx=0x7f8a88015990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:22.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.986+0000 7f8a86ffd640 1 -- 172.21.15.37:0/1124860275 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a88013c60 con 0x7f8a981cd410 2024-04-26T00:34:22.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.986+0000 7f8a86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:22.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.986+0000 7f8a86ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:34:22.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.986+0000 7f8a86ffd640 10 monclient: dump: 2024-04-26T00:34:22.981 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:22.981 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:22.981 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:22.981 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:22.981 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:22.981 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:22.981 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:22.981 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:22.981 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:22.981 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:22.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.986+0000 7f8a86ffd640 1 -- 172.21.15.37:0/1124860275 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8a88013c60 con 0x7f8a981cd410 2024-04-26T00:34:22.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.986+0000 7f8a86ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:34:22.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.986+0000 7f8a86ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:34:22.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.986+0000 7f8a86ffd640 1 -- 172.21.15.37:0/1124860275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a981cd410 msgr2=0x7f8a981cd7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:22.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.986+0000 7f8a86ffd640 1 --2- 172.21.15.37:0/1124860275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a981cd410 0x7f8a981cd7f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8a88015590 tx=0x7f8a88015990 comp rx=0 tx=0).stop 2024-04-26T00:34:22.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.986+0000 7f8a86ffd640 1 -- 172.21.15.37:0/1124860275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a981cd410 msgr2=0x7f8a981cd7f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:34:22.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.986+0000 7f8a86ffd640 1 --2- 172.21.15.37:0/1124860275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a981cd410 0x7f8a981cd7f0 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8a88015590 tx=0x7f8a88015990 comp rx=0 tx=0).stop 2024-04-26T00:34:22.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.994+0000 7f8a9cfde640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:34:22.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.994+0000 7f8a9cfde640 1 --2- 172.21.15.37:0/1124860275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a9802c000 0x7f8a9802c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:22.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.994+0000 7f8a9cfde640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:22.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.994+0000 7f8a9cfde640 1 -- 172.21.15.37:0/1124860275 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8a9813d3d0 con 0x7f8a9802c000 2024-04-26T00:34:22.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.994+0000 7f8a9659c640 1 --2- 172.21.15.37:0/1124860275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a9802c000 0x7f8a9802c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:22.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.994+0000 7f8a9659c640 10 monclient: get_auth_request con 0x7f8a9802c000 auth_method 0 2024-04-26T00:34:22.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.994+0000 7f8a9659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:22.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.994+0000 7f8a9659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:22.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.994+0000 7f8a9659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:22.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.994+0000 7f8a9659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:22.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.994+0000 7f8a9659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:22.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.994+0000 7f8a9659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:22.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.994+0000 7f8a9659c640 10 monclient(hunting): handle_auth_done global_id 174103 payload 824 2024-04-26T00:34:22.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.994+0000 7f8a9659c640 1 -- 172.21.15.37:0/1124860275 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8a9809aa70 con 0x7f8a9802c000 2024-04-26T00:34:22.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.998+0000 7f8a9659c640 1 -- 172.21.15.37:0/1124860275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a9802c000 msgr2=0x7f8a9802c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:22.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.998+0000 7f8a9659c640 1 --2- 172.21.15.37:0/1124860275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a9802c000 0x7f8a9802c410 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8a88008010 tx=0x7f8a88009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:22.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.998+0000 7f8a86ffd640 1 -- 172.21.15.37:0/1124860275 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a88006660 con 0x7f8a9802c000 2024-04-26T00:34:22.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.998+0000 7f8a86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:22.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.998+0000 7f8a86ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:34:22.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.998+0000 7f8a86ffd640 10 monclient: dump: 2024-04-26T00:34:22.994 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:22.994 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:22.994 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:22.994 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:22.994 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:22.994 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:22.994 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:22.994 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:22.995 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:22.995 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:22.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.998+0000 7f8a86ffd640 1 -- 172.21.15.37:0/1124860275 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7f8a880262c0 con 0x7f8a9802c000 2024-04-26T00:34:22.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.998+0000 7f8a86ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:34:22.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.998+0000 7f8a86ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:34:22.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.998+0000 7f8a86ffd640 1 -- 172.21.15.37:0/1124860275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a9802c000 msgr2=0x7f8a9802c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:22.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.998+0000 7f8a86ffd640 1 --2- 172.21.15.37:0/1124860275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a9802c000 0x7f8a9802c410 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8a88008010 tx=0x7f8a88009060 comp rx=0 tx=0).stop 2024-04-26T00:34:22.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.998+0000 7f8a86ffd640 1 -- 172.21.15.37:0/1124860275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a9802c000 msgr2=0x7f8a9802c410 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:34:22.996 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:34:22.996 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:34:22.996 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:34:22.996 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:34:22.996 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 108, 2024-04-26T00:34:22.996 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 5619, 2024-04-26T00:34:22.996 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:34:22.996 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:34:22.996 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:34:22.997 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:34:22.997 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:22.997 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 1, 2024-04-26T00:34:22.997 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:34:22.997 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:34:22.997 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:34:22.997 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:34:22.997 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:34:22.997 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:34:22.997 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:34:22.997 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:34:22.997 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:34:22.997 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:34:22.997 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:34:22.997 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:34:22.998 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:34:22.998 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:34:22.998 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:22.998 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:34:22.998 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:34:22.998 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:34:22.998 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:34:22.998 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:34:22.998 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:34:22.998 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:34:22.998 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:34:22.998 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:34:22.998 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:34:22.998 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:34:22.999 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:34:22.999 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:34:22.999 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:22.999 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:22.999 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:34:22.999 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:34:22.999 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:34:22.999 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:34:22.999 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:34:22.999 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:34:23.000 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:34:23.000 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:34:23.000 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:34:23.000 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:34:23.000 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:34:23.000 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:34:23.000 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:34:23.000 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:34:23.000 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:34:23.000 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:34:23.000 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:34:23.000 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:34:23.000 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:34:23.000 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:34:23.001 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:34:23.001 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:34:23.001 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:34:23.001 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:34:23.001 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:34:23.001 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:34:23.001 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:34:23.001 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:34:23.001 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:23.001 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:34:23.001 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:23.001 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:34:23.002 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:23.002 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:34:23.002 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:34:23.002 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:34:23.002 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:34:23.002 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:23.002 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:34:23.002 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:34:23.002 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:23.002 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:23.002 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:23.002 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:34:23.002 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:34:23.002 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:23.003 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:23.003 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:23.003 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:23.003 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:34:23.003 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:34:23.003 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:34:23.003 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:34:23.003 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:34:23.003 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:23.003 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:23.003 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:34:23.003 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:34:23.003 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:34:23.003 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:34:23.004 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:23.004 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:34:23.004 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:34:23.004 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:23.004 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:23.004 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:23.004 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:34:23.004 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:34:23.004 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:23.004 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:23.004 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:23.004 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:23.004 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:34:23.004 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:34:23.004 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:34:23.005 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:34:23.005 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:34:23.005 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:23.005 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:23.005 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:34:23.005 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:34:23.005 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:34:23.005 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:34:23.005 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:23.005 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:34:23.005 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:34:23.005 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:23.005 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:23.006 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:23.006 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:34:23.006 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:34:23.006 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:23.006 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:23.006 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:23.006 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:23.006 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:34:23.006 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:34:23.007 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:34:23.007 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:34:23.007 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:34:23.007 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:23.007 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:23.007 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:23.008 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:34:23.008 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:34:23.008 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:23.008 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:34:23.008 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:34:23.008 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:34:23.008 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:23.008 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:23.009 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:34:23.009 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:23.009 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:34:23.009 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:34:23.009 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:34:23.009 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:23.009 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:23.009 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:23.009 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:34:23.009 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:34:23.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.998+0000 7f8a86ffd640 1 --2- 172.21.15.37:0/1124860275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a9802c000 0x7f8a9802c410 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:23.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.998+0000 7f8a9cfde640 1 -- 172.21.15.37:0/1124860275 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8a78035eb0 msgr2=0x7f8a78038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:23.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.998+0000 7f8a9cfde640 1 --2- 172.21.15.37:0/1124860275 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8a78035eb0 0x7f8a78038370 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f8a98137c60 tx=0x7f8a8c009040 comp rx=0 tx=0).stop 2024-04-26T00:34:23.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.998+0000 7f8a9cfde640 10 monclient: shutdown 2024-04-26T00:34:23.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.998+0000 7f8a9cfde640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:34:23.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.998+0000 7f8a9cfde640 1 -- 172.21.15.37:0/1124860275 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a9813b8e0 msgr2=0x7f8a9812fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:23.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.998+0000 7f8a9cfde640 1 --2- 172.21.15.37:0/1124860275 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a9813b8e0 0x7f8a9812fde0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8a9001aca0 tx=0x7f8a90019390 comp rx=0 tx=0).stop 2024-04-26T00:34:23.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.998+0000 7f8a96d9d640 1 -- 172.21.15.37:0/1124860275 reap_dead start 2024-04-26T00:34:23.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.998+0000 7f8a96d9d640 1 -- 172.21.15.37:0/1124860275 reap_dead start 2024-04-26T00:34:23.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.998+0000 7f8a9cfde640 1 -- 172.21.15.37:0/1124860275 shutdown_connections 2024-04-26T00:34:23.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.998+0000 7f8a9cfde640 1 -- 172.21.15.37:0/1124860275 >> 172.21.15.37:0/1124860275 conn(0x7f8a980a24c0 msgr2=0x7f8a98133750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:34:23.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.998+0000 7f8a9cfde640 1 -- 172.21.15.37:0/1124860275 shutdown_connections 2024-04-26T00:34:23.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:22.998+0000 7f8a9cfde640 1 -- 172.21.15.37:0/1124860275 wait complete. 2024-04-26T00:34:23.016 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:34:23.016 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-26T00:34:23.016 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-26T00:34:23.052 INFO:tasks.rados.rados.0.smithi045.stdout:2452: finishing write tid 4 to smithi04526579-11 2024-04-26T00:34:23.052 INFO:tasks.rados.rados.0.smithi045.stdout:2452: finishing write tid 5 to smithi04526579-11 2024-04-26T00:34:23.052 INFO:tasks.rados.rados.0.smithi045.stdout:2452: oid 11 updating version 0 to 1451 2024-04-26T00:34:23.052 INFO:tasks.rados.rados.0.smithi045.stdout:2452: oid 11 version 1451 is already newer than 1450 2024-04-26T00:34:23.052 INFO:tasks.rados.rados.0.smithi045.stdout:2452: oid 11 version 1451 is already newer than 1448 2024-04-26T00:34:23.052 INFO:tasks.rados.rados.0.smithi045.stdout:2452: oid 11 version 1451 is already newer than 1449 2024-04-26T00:34:23.052 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 1451 (ObjNum 890 snap 275 seq_num 890) dirty exists 2024-04-26T00:34:23.053 INFO:tasks.rados.rados.0.smithi045.stdout:2452: left oid 11 (ObjNum 890 snap 275 seq_num 890) 2024-04-26T00:34:23.053 INFO:tasks.rados.rados.0.smithi045.stdout:2454: finishing rollback tid 0 to smithi04526579-42 2024-04-26T00:34:23.053 INFO:tasks.rados.rados.0.smithi045.stdout:2454: finishing rollback tid 1 to smithi04526579-42 2024-04-26T00:34:23.053 INFO:tasks.rados.rados.0.smithi045.stdout:2454: finishing rollback tid 2 to smithi04526579-42 2024-04-26T00:34:23.053 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 1739 (ObjNum 734 snap 227 seq_num 734) dirty exists 2024-04-26T00:34:23.053 INFO:tasks.rados.rados.0.smithi045.stdout:2455: finishing write tid 1 to smithi04526579-28 2024-04-26T00:34:23.053 INFO:tasks.rados.rados.0.smithi045.stdout:2455: finishing write tid 2 to smithi04526579-28 2024-04-26T00:34:23.053 INFO:tasks.rados.rados.0.smithi045.stdout:2455: finishing write tid 3 to smithi04526579-28 2024-04-26T00:34:23.053 INFO:tasks.rados.rados.0.smithi045.stdout:2455: finishing write tid 4 to smithi04526579-28 2024-04-26T00:34:23.053 INFO:tasks.rados.rados.0.smithi045.stdout:2455: finishing write tid 5 to smithi04526579-28 2024-04-26T00:34:23.053 INFO:tasks.rados.rados.0.smithi045.stdout:2455: oid 28 updating version 0 to 1930 2024-04-26T00:34:23.053 INFO:tasks.rados.rados.0.smithi045.stdout:2455: oid 28 updating version 1930 to 1931 2024-04-26T00:34:23.053 INFO:tasks.rados.rados.0.smithi045.stdout:2455: oid 28 updating version 1931 to 1932 2024-04-26T00:34:23.053 INFO:tasks.rados.rados.0.smithi045.stdout:2455: oid 28 version 1932 is already newer than 1929 2024-04-26T00:34:23.053 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 1932 (ObjNum 892 snap 275 seq_num 892) dirty exists 2024-04-26T00:34:23.054 INFO:tasks.rados.rados.0.smithi045.stdout:2455: left oid 28 (ObjNum 892 snap 275 seq_num 892) 2024-04-26T00:34:23.054 INFO:tasks.rados.rados.0.smithi045.stdout:2452: done (3 left) 2024-04-26T00:34:23.054 INFO:tasks.rados.rados.0.smithi045.stdout:2454: done (2 left) 2024-04-26T00:34:23.054 INFO:tasks.rados.rados.0.smithi045.stdout:2455: done (1 left) 2024-04-26T00:34:23.054 INFO:tasks.rados.rados.0.smithi045.stdout:2456: done (0 left) 2024-04-26T00:34:23.054 INFO:tasks.rados.rados.0.smithi045.stdout:2457: snap_remove snap 273 2024-04-26T00:34:23.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.154+0000 7f97919c8640 10 monclient: get_monmap_and_config 2024-04-26T00:34:23.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.154+0000 7f97919c8640 10 monclient: build_initial_monmap 2024-04-26T00:34:23.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.154+0000 7f97919c8640 10 monclient: monmap: 2024-04-26T00:34:23.151 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:34:23.151 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:23.151 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:34:23.158589+0000 2024-04-26T00:34:23.152 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:34:23.158589+0000 2024-04-26T00:34:23.152 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:34:23.152 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:23.152 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:34:23.152 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:34:23.152 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:34:23.152 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:34:23.152 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:34:23.152 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:23.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.154+0000 7f97919c8640 1 Processor -- start 2024-04-26T00:34:23.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.154+0000 7f97919c8640 1 -- start start 2024-04-26T00:34:23.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.154+0000 7f97919c8640 10 monclient: init 2024-04-26T00:34:23.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.158+0000 7f97919c8640 10 monclient: _reopen_session rank -1 2024-04-26T00:34:23.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.158+0000 7f97919c8640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-04-26T00:34:23.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.158+0000 7f97919c8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f978c136da0 0x7f978c09bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:23.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.158+0000 7f97919c8640 10 monclient(hunting): picked mon.noname-b con 0x7f978c136da0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:34:23.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.158+0000 7f97919c8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f978c09c410 addr v1:172.21.15.37:6789/0 2024-04-26T00:34:23.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.158+0000 7f97919c8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f978c137890 addr v1:172.21.15.45:6789/0 2024-04-26T00:34:23.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.158+0000 7f97919c8640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f978c09b4f0 con 0x7f978c09c410 2024-04-26T00:34:23.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.158+0000 7f97919c8640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f978c09b7d0 con 0x7f978c137890 2024-04-26T00:34:23.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.158+0000 7f97919c8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:23.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.158+0000 7f97919c8640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f978c09bb40 con 0x7f978c136da0 2024-04-26T00:34:23.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.158+0000 7f97919c8640 10 monclient(hunting): _renew_subs 2024-04-26T00:34:23.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.158+0000 7f97919c8640 10 monclient(hunting): authenticate will time out at 1844.464465s 2024-04-26T00:34:23.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.158+0000 7f9783fff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f978c09c410 0x7f978c137180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:33388/0 (socket says 172.21.15.37:33388) 2024-04-26T00:34:23.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.158+0000 7f9783fff640 1 -- 172.21.15.37:0/1953340518 learned_addr learned my addr 172.21.15.37:0/1953340518 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:34:23.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.158+0000 7f978affd640 1 --2- 172.21.15.37:0/1953340518 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f978c136da0 0x7f978c09bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:23.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.158+0000 7f978affd640 10 monclient(hunting): get_auth_request con 0x7f978c136da0 auth_method 0 2024-04-26T00:34:23.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.158+0000 7f978affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:23.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.158+0000 7f978affd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:23.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.158+0000 7f978affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:23.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.158+0000 7f978affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:23.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.158+0000 7f978affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:23.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.158+0000 7f978affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:23.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.158+0000 7f978affd640 10 monclient(hunting): handle_auth_done global_id 154286 payload 274 2024-04-26T00:34:23.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.158+0000 7f978affd640 1 -- 172.21.15.37:0/1953340518 >> v1:172.21.15.45:6789/0 conn(0x7f978c137890 legacy=0x7f978c13e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:34:23.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.158+0000 7f978affd640 1 -- 172.21.15.37:0/1953340518 >> v1:172.21.15.37:6789/0 conn(0x7f978c09c410 legacy=0x7f978c137180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:34:23.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.158+0000 7f978affd640 10 monclient: _finish_hunting 0 2024-04-26T00:34:23.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.158+0000 7f978affd640 1 monclient: found mon.noname-b 2024-04-26T00:34:23.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.158+0000 7f978affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:34:23.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.158+0000 7f978affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:34:23.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.158+0000 7f978affd640 1 -- 172.21.15.37:0/1953340518 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f978c06dbb0 con 0x7f978c136da0 2024-04-26T00:34:23.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f978affd640 1 -- 172.21.15.37:0/1953340518 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f978c136da0 msgr2=0x7f978c09bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:23.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f978affd640 1 --2- 172.21.15.37:0/1953340518 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f978c136da0 0x7f978c09bed0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f977c0097f0 tx=0x7f977c01c320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90ea01f1a2a70e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:23.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f978a7fc640 1 -- 172.21.15.37:0/1953340518 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f977c0b0070 con 0x7f978c136da0 2024-04-26T00:34:23.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f978a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:23.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f978a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:34:23.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f978a7fc640 10 monclient: dump: 2024-04-26T00:34:23.163 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:23.163 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:23.164 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:23.164 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:23.164 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:23.164 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:23.164 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:23.164 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:23.164 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:23.164 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:23.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f978a7fc640 10 monclient: _finish_auth 0 2024-04-26T00:34:23.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f978a7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:34:23.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f978a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:34:23.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f978a7fc640 1 -- 172.21.15.37:0/1953340518 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f977c0a97f0 con 0x7f978c136da0 2024-04-26T00:34:23.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f978a7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:34:23.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f978a7fc640 1 -- 172.21.15.37:0/1953340518 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f977c0a9bc0 con 0x7f978c136da0 2024-04-26T00:34:23.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f97919c8640 5 monclient: authenticate success, global_id 154286 2024-04-26T00:34:23.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f97919c8640 10 monclient: get_monmap_and_config success 2024-04-26T00:34:23.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f97919c8640 4 set_mon_vals no callback set 2024-04-26T00:34:23.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f978a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:23.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f978a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:34:23.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f978a7fc640 10 monclient: dump: 2024-04-26T00:34:23.165 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:23.165 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:23.165 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:23.165 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:23.165 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:23.165 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:23.166 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:23.166 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:23.166 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:23.166 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:23.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f97919c8640 10 monclient: shutdown 2024-04-26T00:34:23.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f97919c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:34:23.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f97919c8640 1 -- 172.21.15.37:0/1953340518 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f978c136da0 msgr2=0x7f978c09bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:23.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f97919c8640 1 --2- 172.21.15.37:0/1953340518 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f978c136da0 0x7f978c09bed0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f977c0097f0 tx=0x7f977c01c320 comp rx=0 tx=0).stop 2024-04-26T00:34:23.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f97919c8640 1 -- 172.21.15.37:0/1953340518 shutdown_connections 2024-04-26T00:34:23.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f97919c8640 1 --2- 172.21.15.37:0/1953340518 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f978c136da0 0x7f978c09bed0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:23.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f97919c8640 1 -- 172.21.15.37:0/1953340518 >> 172.21.15.37:0/1953340518 conn(0x7f978c0a5f00 msgr2=0x7f978c0a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:34:23.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f97919c8640 1 -- 172.21.15.37:0/1953340518 shutdown_connections 2024-04-26T00:34:23.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f97919c8640 1 -- 172.21.15.37:0/1953340518 wait complete. 2024-04-26T00:34:23.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f97919c8640 10 monclient: build_initial_monmap 2024-04-26T00:34:23.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f97919c8640 10 monclient: monmap: 2024-04-26T00:34:23.168 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:34:23.168 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:34:23.168 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:34:23.168 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:34:23.168 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:34:23.168 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:34:23.168 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:34:23.168 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:34:23.168 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:34:23.168 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:23.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f97919c8640 1 Processor -- start 2024-04-26T00:34:23.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f97919c8640 1 -- start start 2024-04-26T00:34:23.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f97919c8640 10 monclient: init 2024-04-26T00:34:23.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f97919c8640 10 monclient: _reopen_session rank -1 2024-04-26T00:34:23.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f97919c8640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:34:23.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f97919c8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f978c09c410 0x7f978c1cef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:23.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f97919c8640 10 monclient(hunting): picked mon.noname-b con 0x7f978c09c410 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:34:23.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f97919c8640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f978c136da0 0x7f978c135df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:23.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f97919c8640 10 monclient(hunting): picked mon.noname-a con 0x7f978c136da0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:34:23.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f97919c8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f978c137890 0x7f978c13ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:23.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f97919c8640 10 monclient(hunting): picked mon.noname-c con 0x7f978c137890 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:34:23.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f97919c8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:23.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f97919c8640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f978c142a40 con 0x7f978c136da0 2024-04-26T00:34:23.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f97919c8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:23.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f97919c8640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f978c1428c0 con 0x7f978c09c410 2024-04-26T00:34:23.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f97919c8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:23.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f97919c8640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f978c142bc0 con 0x7f978c137890 2024-04-26T00:34:23.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f97919c8640 10 monclient(hunting): _renew_subs 2024-04-26T00:34:23.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f97919c8640 10 monclient(hunting): authenticate will time out at 1844.471769s 2024-04-26T00:34:23.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f9783fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f978c136da0 0x7f978c135df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:23.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f9783fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f978c136da0 0x7f978c135df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33770/0 (socket says 172.21.15.37:33770) 2024-04-26T00:34:23.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f9783fff640 1 -- 172.21.15.37:0/2466051171 learned_addr learned my addr 172.21.15.37:0/2466051171 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:34:23.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f9783fff640 10 monclient(hunting): get_auth_request con 0x7f978c136da0 auth_method 0 2024-04-26T00:34:23.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f9783fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:23.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f9783fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:23.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f9783fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:23.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f978b7fe640 1 --2- 172.21.15.37:0/2466051171 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f978c137890 0x7f978c13ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:23.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.162+0000 7f978b7fe640 10 monclient(hunting): get_auth_request con 0x7f978c137890 auth_method 0 2024-04-26T00:34:23.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f978b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:23.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f978b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:23.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f978b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:23.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f9783fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:23.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f9783fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:23.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f9783fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:23.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f978b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:23.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f978b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:23.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f978affd640 1 --2- 172.21.15.37:0/2466051171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f978c09c410 0x7f978c1cef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:23.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f978b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:23.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f978affd640 10 monclient(hunting): get_auth_request con 0x7f978c09c410 auth_method 0 2024-04-26T00:34:23.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f978affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:23.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f978affd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:23.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f978affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:23.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f978affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:23.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f978affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:23.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f978affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:23.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f978b7fe640 10 monclient(hunting): handle_auth_done global_id 154289 payload 824 2024-04-26T00:34:23.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f978b7fe640 1 -- 172.21.15.37:0/2466051171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f978c09c410 msgr2=0x7f978c1cef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:23.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f978b7fe640 1 --2- 172.21.15.37:0/2466051171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f978c09c410 0x7f978c1cef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:23.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f978b7fe640 1 -- 172.21.15.37:0/2466051171 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f978c136da0 msgr2=0x7f978c135df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:23.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f978b7fe640 1 --2- 172.21.15.37:0/2466051171 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f978c136da0 0x7f978c135df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:23.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f978b7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:34:23.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f978b7fe640 1 monclient: found mon.noname-c 2024-04-26T00:34:23.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f978b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:34:23.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f978b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:34:23.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f978b7fe640 1 -- 172.21.15.37:0/2466051171 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f978c13c8d0 con 0x7f978c137890 2024-04-26T00:34:23.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f9783fff640 1 --2- 172.21.15.37:0/2466051171 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f978c136da0 0x7f978c135df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:34:23.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f978b7fe640 1 -- 172.21.15.37:0/2466051171 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f978c137890 msgr2=0x7f978c13ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:23.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f978b7fe640 1 --2- 172.21.15.37:0/2466051171 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f978c137890 0x7f978c13ac30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f978400c780 tx=0x7f978400cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:23.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f9788ff9640 1 -- 172.21.15.37:0/2466051171 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9784016020 con 0x7f978c137890 2024-04-26T00:34:23.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f9788ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:23.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f9788ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:34:23.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f9788ff9640 10 monclient: dump: 2024-04-26T00:34:23.175 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:23.175 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:23.175 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:23.175 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:23.175 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:23.175 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:23.175 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:23.175 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:23.176 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:23.176 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:23.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f9788ff9640 10 monclient: _finish_auth 0 2024-04-26T00:34:23.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f9788ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:34:23.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f9788ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:34:23.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f9788ff9640 1 -- 172.21.15.37:0/2466051171 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f978400ac60 con 0x7f978c137890 2024-04-26T00:34:23.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f9788ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:34:23.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f9788ff9640 1 -- 172.21.15.37:0/2466051171 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9784012660 con 0x7f978c137890 2024-04-26T00:34:23.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f9788ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:23.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f9788ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:34:23.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f9788ff9640 10 monclient: dump: 2024-04-26T00:34:23.177 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:23.178 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:23.178 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:23.178 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:23.178 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:23.178 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:23.178 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:23.178 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:23.179 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:23.179 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:23.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f97919c8640 5 monclient: authenticate success, global_id 154289 2024-04-26T00:34:23.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f97919c8640 10 monclient: _renew_subs 2024-04-26T00:34:23.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f9789ffb640 4 set_mon_vals no callback set 2024-04-26T00:34:23.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f97919c8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:34:23.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f97919c8640 1 -- 172.21.15.37:0/2466051171 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f978c13ebc0 con 0x7f978c137890 2024-04-26T00:34:23.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f97919c8640 10 monclient: _renew_subs 2024-04-26T00:34:23.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f97919c8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:34:23.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f97919c8640 1 -- 172.21.15.37:0/2466051171 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f978c1cfe00 con 0x7f978c137890 2024-04-26T00:34:23.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f97919c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:34:23.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f97919c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:34:23.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f97919c8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:34:23.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f9788ff9640 1 -- 172.21.15.37:0/2466051171 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9784019020 con 0x7f978c137890 2024-04-26T00:34:23.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.166+0000 7f97919c8640 1 -- 172.21.15.37:0/2466051171 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f978c066f40 con 0x7f978c137890 2024-04-26T00:34:23.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.170+0000 7f9788ff9640 1 --2- 172.21.15.37:0/2466051171 >> v2:172.21.15.45:6812/2379918623 conn(0x7f976c035ed0 0x7f976c038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:23.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.170+0000 7f9788ff9640 1 -- 172.21.15.37:0/2466051171 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(563..563 src has 370..563) ==== 3975+0+0 (secure 0 0 0) 0x7f97840489c0 con 0x7f978c137890 2024-04-26T00:34:23.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.170+0000 7f978affd640 1 --2- 172.21.15.37:0/2466051171 >> v2:172.21.15.45:6812/2379918623 conn(0x7f976c035ed0 0x7f976c038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:23.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.170+0000 7f978affd640 10 monclient: get_auth_request con 0x7f976c035ed0 auth_method 0 2024-04-26T00:34:23.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.170+0000 7f978affd640 1 --2- 172.21.15.37:0/2466051171 >> v2:172.21.15.45:6812/2379918623 conn(0x7f976c035ed0 0x7f976c038390 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f977c0ab000 tx=0x7f977c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:23.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.174+0000 7f9788ff9640 1 -- 172.21.15.37:0/2466051171 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9784020030 con 0x7f978c137890 2024-04-26T00:34:23.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.174+0000 7f9788ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:34:23.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.174+0000 7f9788ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:34:23.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.394+0000 7f97919c8640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-26T00:34:23.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.394+0000 7f97919c8640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-26T00:34:23.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.394+0000 7f97919c8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:34:23.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.394+0000 7f97919c8640 1 -- 172.21.15.37:0/2466051171 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f978c075d70 con 0x7f978c137890 2024-04-26T00:34:23.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.398+0000 7f9788ff9640 1 -- 172.21.15.37:0/2466051171 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f978400ae10 con 0x7f978c137890 2024-04-26T00:34:23.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.398+0000 7f9788ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-26T00:34:23.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.398+0000 7f9788ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:34:23.396 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.398+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.396 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.398+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=12,kv=10,logm=50} crc {auth=4251416881,config=692619927,health=644399288,kv=1034996044,logm=1438736196}) 2024-04-26T00:34:23.396 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.398+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=24,config=2,health=10,kv=9,logm=55} crc {auth=3659988919,config=692619927,health=3849207400,kv=2491615610,logm=2468896442}) 2024-04-26T00:34:23.396 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.398+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.396 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.398+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=12,kv=10,logm=50} crc {auth=4251416881,config=692619927,health=644399288,kv=1034996044,logm=1438736196}) 2024-04-26T00:34:23.396 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.398+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=24,config=2,health=12,kv=10,logm=52} crc {auth=1926890655,config=692619927,health=644399288,kv=1034996044,logm=1336608834}) 2024-04-26T00:34:23.398 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.402+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.398 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.402+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2085351589}) 2024-04-26T00:34:23.398 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.402+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2417450614}) 2024-04-26T00:34:23.398 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.402+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.398 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.402+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2085351589}) 2024-04-26T00:34:23.398 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.402+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=556489786}) 2024-04-26T00:34:23.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.402+0000 7f97919c8640 1 -- 172.21.15.37:0/2466051171 >> v2:172.21.15.45:6812/2379918623 conn(0x7f976c035ed0 msgr2=0x7f976c038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:23.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.402+0000 7f97919c8640 1 --2- 172.21.15.37:0/2466051171 >> v2:172.21.15.45:6812/2379918623 conn(0x7f976c035ed0 0x7f976c038390 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f977c0ab000 tx=0x7f977c008040 comp rx=0 tx=0).stop 2024-04-26T00:34:23.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.402+0000 7f97919c8640 10 monclient: shutdown 2024-04-26T00:34:23.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.402+0000 7f97919c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:34:23.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.402+0000 7f97919c8640 1 -- 172.21.15.37:0/2466051171 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f978c137890 msgr2=0x7f978c13ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:23.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.402+0000 7f97919c8640 1 --2- 172.21.15.37:0/2466051171 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f978c137890 0x7f978c13ac30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f978400c780 tx=0x7f978400cc40 comp rx=0 tx=0).stop 2024-04-26T00:34:23.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.402+0000 7f97919c8640 1 -- 172.21.15.37:0/2466051171 shutdown_connections 2024-04-26T00:34:23.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.402+0000 7f97919c8640 1 --2- 172.21.15.37:0/2466051171 >> v2:172.21.15.45:6812/2379918623 conn(0x7f976c035ed0 0x7f976c038390 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:23.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.402+0000 7f97919c8640 1 --2- 172.21.15.37:0/2466051171 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f978c137890 0x7f978c13ac30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:23.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.402+0000 7f97919c8640 1 --2- 172.21.15.37:0/2466051171 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f978c136da0 0x7f978c135df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:23.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.402+0000 7f97919c8640 1 --2- 172.21.15.37:0/2466051171 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f978c09c410 0x7f978c1cef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:23.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.402+0000 7f97919c8640 1 -- 172.21.15.37:0/2466051171 >> 172.21.15.37:0/2466051171 conn(0x7f978c0a5f00 msgr2=0x7f978c09d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:34:23.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.402+0000 7f97919c8640 1 -- 172.21.15.37:0/2466051171 shutdown_connections 2024-04-26T00:34:23.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:23.402+0000 7f97919c8640 1 -- 172.21.15.37:0/2466051171 wait complete. 2024-04-26T00:34:23.400 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.406+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.401 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.406+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2949512392}) 2024-04-26T00:34:23.401 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.406+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4252619603}) 2024-04-26T00:34:23.401 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.406+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.401 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.406+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2949512392}) 2024-04-26T00:34:23.401 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.406+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2559713989}) 2024-04-26T00:34:23.403 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.406+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.403 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.406+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1874533287}) 2024-04-26T00:34:23.403 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.406+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2812533786}) 2024-04-26T00:34:23.403 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.406+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.403 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.406+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1874533287}) 2024-04-26T00:34:23.403 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.406+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2326204970}) 2024-04-26T00:34:23.405 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.410+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.405 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.410+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2172851572}) 2024-04-26T00:34:23.406 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.410+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=167205002}) 2024-04-26T00:34:23.406 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.410+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.406 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.410+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2172851572}) 2024-04-26T00:34:23.406 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.410+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2304747149}) 2024-04-26T00:34:23.407 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.410+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.407 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.410+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=613543514}) 2024-04-26T00:34:23.407 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.410+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2242524442}) 2024-04-26T00:34:23.407 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.410+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.408 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.410+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=613543514}) 2024-04-26T00:34:23.408 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.410+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2264472446}) 2024-04-26T00:34:23.409 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.414+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.409 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.414+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4110565084}) 2024-04-26T00:34:23.409 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.414+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1970668939}) 2024-04-26T00:34:23.410 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.414+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.410 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.414+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4110565084}) 2024-04-26T00:34:23.410 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.414+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4173775636}) 2024-04-26T00:34:23.411 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.414+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.411 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.414+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1811391505}) 2024-04-26T00:34:23.411 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.414+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3270996297}) 2024-04-26T00:34:23.411 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.414+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.412 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.414+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1811391505}) 2024-04-26T00:34:23.412 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.414+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3531530999}) 2024-04-26T00:34:23.413 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.418+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.413 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.418+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4015658216}) 2024-04-26T00:34:23.413 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.418+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2076360627}) 2024-04-26T00:34:23.413 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.418+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.414 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.418+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4015658216}) 2024-04-26T00:34:23.414 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.418+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1355909407}) 2024-04-26T00:34:23.415 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.418+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.415 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.418+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1282989053}) 2024-04-26T00:34:23.416 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.418+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1255954884}) 2024-04-26T00:34:23.416 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.418+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.416 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.418+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1282989053}) 2024-04-26T00:34:23.416 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.418+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1170297763}) 2024-04-26T00:34:23.417 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.422+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.418 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.422+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1804796731}) 2024-04-26T00:34:23.418 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.422+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=181470131}) 2024-04-26T00:34:23.418 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.422+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.418 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.422+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1804796731}) 2024-04-26T00:34:23.418 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.422+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1157115079}) 2024-04-26T00:34:23.419 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.422+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.419 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.422+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1842212566}) 2024-04-26T00:34:23.419 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.422+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4134075225}) 2024-04-26T00:34:23.419 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.422+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.419 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.422+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1842212566}) 2024-04-26T00:34:23.419 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.422+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1898254659}) 2024-04-26T00:34:23.420 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-26T00:34:23.420 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.426+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.421 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.426+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=14830558}) 2024-04-26T00:34:23.421 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.426+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2348757829}) 2024-04-26T00:34:23.421 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.426+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.421 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.426+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=14830558}) 2024-04-26T00:34:23.421 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.426+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=617343298}) 2024-04-26T00:34:23.422 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.426+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.423 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.426+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3954525988}) 2024-04-26T00:34:23.423 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.426+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4096775137}) 2024-04-26T00:34:23.423 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.426+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.423 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.426+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3954525988}) 2024-04-26T00:34:23.423 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.426+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2480040774}) 2024-04-26T00:34:23.424 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.430+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.424 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.430+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2814694462}) 2024-04-26T00:34:23.424 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.430+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2690894612}) 2024-04-26T00:34:23.424 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.430+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.424 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.430+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2814694462}) 2024-04-26T00:34:23.424 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.430+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2036057644}) 2024-04-26T00:34:23.426 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.430+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.426 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.430+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4136461037}) 2024-04-26T00:34:23.426 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.430+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3389425136}) 2024-04-26T00:34:23.426 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.430+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.427 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.430+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4136461037}) 2024-04-26T00:34:23.427 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.430+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=191689924}) 2024-04-26T00:34:23.428 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.434+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.428 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.434+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4274884094}) 2024-04-26T00:34:23.428 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.434+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1921985513}) 2024-04-26T00:34:23.428 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.434+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.428 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.434+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4274884094}) 2024-04-26T00:34:23.429 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.434+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3675376388}) 2024-04-26T00:34:23.505 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.510+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.505 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.510+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4017358443}) 2024-04-26T00:34:23.505 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.510+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4264333186}) 2024-04-26T00:34:23.505 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.510+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.505 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.510+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4017358443}) 2024-04-26T00:34:23.505 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.510+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3999627947}) 2024-04-26T00:34:23.515 INFO:tasks.rados.rados.0.smithi045.stdout:2457: done (0 left) 2024-04-26T00:34:23.515 INFO:tasks.rados.rados.0.smithi045.stdout:2458: snap_remove snap 248 2024-04-26T00:34:23.516 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.518+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.516 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.518+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=478395850}) 2024-04-26T00:34:23.516 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.518+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2022854518}) 2024-04-26T00:34:23.516 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.518+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.516 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.518+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=478395850}) 2024-04-26T00:34:23.517 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.518+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3582169103}) 2024-04-26T00:34:23.528 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.530+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.528 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.530+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1067873382}) 2024-04-26T00:34:23.528 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.530+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4188708740}) 2024-04-26T00:34:23.528 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.530+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.528 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.530+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1067873382}) 2024-04-26T00:34:23.528 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.530+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3977735044}) 2024-04-26T00:34:23.901 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.906+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.902 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.906+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3621068582}) 2024-04-26T00:34:23.902 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.906+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=974167587}) 2024-04-26T00:34:23.902 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.906+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:23.902 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.906+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3621068582}) 2024-04-26T00:34:23.902 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:23.906+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3062750432}) 2024-04-26T00:34:24.045 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.050+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.082 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.050+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=388566006}) 2024-04-26T00:34:24.082 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.050+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1373713643}) 2024-04-26T00:34:24.083 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.050+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.083 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.050+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=388566006}) 2024-04-26T00:34:24.083 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.050+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2491399774}) 2024-04-26T00:34:24.084 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.054+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.084 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.054+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1251949089}) 2024-04-26T00:34:24.084 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.054+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=236242803}) 2024-04-26T00:34:24.084 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.054+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.084 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.054+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1251949089}) 2024-04-26T00:34:24.084 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.054+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3742188690}) 2024-04-26T00:34:24.084 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.070+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.084 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.070+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1853231514}) 2024-04-26T00:34:24.084 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.070+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3506600644}) 2024-04-26T00:34:24.084 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.070+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.085 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.070+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1853231514}) 2024-04-26T00:34:24.085 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.070+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1612731988}) 2024-04-26T00:34:24.386 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.390+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.387 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.390+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4261969711}) 2024-04-26T00:34:24.387 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.390+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=199662879}) 2024-04-26T00:34:24.387 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.390+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.387 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.390+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4261969711}) 2024-04-26T00:34:24.387 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.390+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3556963054}) 2024-04-26T00:34:24.389 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.394+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.389 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.394+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2975129002}) 2024-04-26T00:34:24.389 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.394+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3770695616}) 2024-04-26T00:34:24.389 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.394+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.389 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.394+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2975129002}) 2024-04-26T00:34:24.389 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.394+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3010836719}) 2024-04-26T00:34:24.391 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.394+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.392 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.394+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2107032847}) 2024-04-26T00:34:24.392 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.394+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4053318617}) 2024-04-26T00:34:24.392 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.394+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.392 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.394+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2107032847}) 2024-04-26T00:34:24.392 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.394+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3638118125}) 2024-04-26T00:34:24.394 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.398+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.395 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.398+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1632326839}) 2024-04-26T00:34:24.395 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.398+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3572115597}) 2024-04-26T00:34:24.395 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.398+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.395 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.398+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1632326839}) 2024-04-26T00:34:24.395 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.398+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1623778670}) 2024-04-26T00:34:24.397 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.402+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.397 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.402+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=70907011}) 2024-04-26T00:34:24.397 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.402+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2108015784}) 2024-04-26T00:34:24.397 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.402+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.397 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.402+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=70907011}) 2024-04-26T00:34:24.397 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.402+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=338149840}) 2024-04-26T00:34:24.399 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.402+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.399 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.402+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=957370013}) 2024-04-26T00:34:24.399 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.402+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=465879213}) 2024-04-26T00:34:24.399 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.402+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.400 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.402+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=957370013}) 2024-04-26T00:34:24.400 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.402+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4173688864}) 2024-04-26T00:34:24.401 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.406+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.401 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.406+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1943849460}) 2024-04-26T00:34:24.402 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.406+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1697587938}) 2024-04-26T00:34:24.402 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.406+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.402 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.406+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1943849460}) 2024-04-26T00:34:24.402 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.406+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2115710634}) 2024-04-26T00:34:24.404 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.406+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.405 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.406+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3162436647}) 2024-04-26T00:34:24.405 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.406+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1501658297}) 2024-04-26T00:34:24.405 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.406+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.405 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.406+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3162436647}) 2024-04-26T00:34:24.405 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.410+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3804136150}) 2024-04-26T00:34:24.406 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.410+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.406 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.410+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2028879223}) 2024-04-26T00:34:24.407 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.410+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=140401328}) 2024-04-26T00:34:24.407 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.410+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.407 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.410+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2028879223}) 2024-04-26T00:34:24.407 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.410+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2946650230}) 2024-04-26T00:34:24.409 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.414+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.409 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.414+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3678181944}) 2024-04-26T00:34:24.409 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.414+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1353175094}) 2024-04-26T00:34:24.409 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.414+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.409 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.414+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3678181944}) 2024-04-26T00:34:24.409 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.414+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3162570503}) 2024-04-26T00:34:24.411 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.414+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.411 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.414+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2233730611}) 2024-04-26T00:34:24.412 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.414+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1445833597}) 2024-04-26T00:34:24.412 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.414+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.412 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.414+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2233730611}) 2024-04-26T00:34:24.412 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.414+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3567640130}) 2024-04-26T00:34:24.414 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.418+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.414 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.418+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2792058662}) 2024-04-26T00:34:24.414 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.418+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1569520763}) 2024-04-26T00:34:24.414 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.418+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.414 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.418+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2792058662}) 2024-04-26T00:34:24.414 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.418+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3159611471}) 2024-04-26T00:34:24.418 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.422+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.418 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.422+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3654580141}) 2024-04-26T00:34:24.418 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.422+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3043833281}) 2024-04-26T00:34:24.418 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.422+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.418 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.422+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3654580141}) 2024-04-26T00:34:24.419 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.422+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3803403867}) 2024-04-26T00:34:24.419 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.422+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.419 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.422+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3510122025}) 2024-04-26T00:34:24.420 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.422+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=819496810}) 2024-04-26T00:34:24.420 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.422+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.420 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.422+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3510122025}) 2024-04-26T00:34:24.420 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.422+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=491070769}) 2024-04-26T00:34:24.421 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:34:24.421 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:34:24.422 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.426+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.422 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.426+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2615910665}) 2024-04-26T00:34:24.422 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.426+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3449451217}) 2024-04-26T00:34:24.423 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.426+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.423 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.426+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2615910665}) 2024-04-26T00:34:24.423 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.426+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1429765198}) 2024-04-26T00:34:24.423 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.426+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.424 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.426+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2958571496}) 2024-04-26T00:34:24.425 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.426+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2043374488}) 2024-04-26T00:34:24.425 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.426+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.425 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.426+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2958571496}) 2024-04-26T00:34:24.425 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.430+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=745864463}) 2024-04-26T00:34:24.426 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.430+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.426 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.430+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1133688423}) 2024-04-26T00:34:24.426 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.430+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3355043001}) 2024-04-26T00:34:24.427 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.430+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.427 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.430+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1133688423}) 2024-04-26T00:34:24.427 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.430+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1514105389}) 2024-04-26T00:34:24.430 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.434+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.430 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.434+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3130126444}) 2024-04-26T00:34:24.430 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.434+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1871450971}) 2024-04-26T00:34:24.430 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.434+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.430 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.434+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3130126444}) 2024-04-26T00:34:24.430 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.434+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1034411455}) 2024-04-26T00:34:24.431 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.434+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.431 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.434+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2513840141}) 2024-04-26T00:34:24.432 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.434+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3669425318}) 2024-04-26T00:34:24.432 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.434+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.432 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.434+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2513840141}) 2024-04-26T00:34:24.432 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.434+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=204593000}) 2024-04-26T00:34:24.434 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.438+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.434 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.438+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1890075009}) 2024-04-26T00:34:24.434 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.438+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=155341217}) 2024-04-26T00:34:24.434 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.438+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.434 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.438+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1890075009}) 2024-04-26T00:34:24.434 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.438+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2482928832}) 2024-04-26T00:34:24.437 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.442+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.437 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.442+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2788119320}) 2024-04-26T00:34:24.437 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.442+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1512795453}) 2024-04-26T00:34:24.438 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.442+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.438 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.442+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2788119320}) 2024-04-26T00:34:24.438 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.442+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=114790774}) 2024-04-26T00:34:24.441 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.446+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.441 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.446+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2810052332}) 2024-04-26T00:34:24.441 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.446+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1352866640}) 2024-04-26T00:34:24.441 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.446+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.441 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.446+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2810052332}) 2024-04-26T00:34:24.441 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.446+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4139313844}) 2024-04-26T00:34:24.444 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.446+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.444 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.446+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=871542274}) 2024-04-26T00:34:24.444 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.446+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2687745228}) 2024-04-26T00:34:24.444 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.446+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.444 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.446+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=871542274}) 2024-04-26T00:34:24.444 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.446+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=152588428}) 2024-04-26T00:34:24.447 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.450+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.447 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.450+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2683025111}) 2024-04-26T00:34:24.447 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.450+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3543640904}) 2024-04-26T00:34:24.447 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.450+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.448 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.450+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2683025111}) 2024-04-26T00:34:24.448 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.450+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=653110589}) 2024-04-26T00:34:24.448 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.454+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.449 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.454+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=101474001}) 2024-04-26T00:34:24.449 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.454+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1018583737}) 2024-04-26T00:34:24.449 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.454+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.449 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.454+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=101474001}) 2024-04-26T00:34:24.449 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.454+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1806127561}) 2024-04-26T00:34:24.452 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.454+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.453 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.458+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=33,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=25} crc {logm=3139371076,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=148111919,mon_config_key=1720404874,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=1308449553,osd_snap=374037369}) 2024-04-26T00:34:24.453 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.458+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=35,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=22} crc {logm=2725590431,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=148111919,mon_config_key=4061063117,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=1308449553,osd_snap=144330155}) 2024-04-26T00:34:24.453 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.458+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.453 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.458+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=33,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=25} crc {logm=3139371076,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=148111919,mon_config_key=1720404874,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=1308449553,osd_snap=374037369}) 2024-04-26T00:34:24.453 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.458+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=34,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=11,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=25} crc {logm=4179517855,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=148111919,mon_config_key=4061063117,monmap=353912172,osd_metadata=2100554115,osd_pg_creating=1308449553,osd_snap=1042469308}) 2024-04-26T00:34:24.455 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.458+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.456 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.458+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=77,osdmap=23} crc {osd_snap=3842938332,osdmap=2356442859}) 2024-04-26T00:34:24.456 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.458+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=75,osdmap=25} crc {osd_snap=3897817404,osdmap=2780600525}) 2024-04-26T00:34:24.456 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.458+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.456 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.458+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=77,osdmap=23} crc {osd_snap=3842938332,osdmap=2356442859}) 2024-04-26T00:34:24.456 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.458+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=80,osdmap=20} crc {osd_snap=3951007547,osdmap=1410859736}) 2024-04-26T00:34:24.458 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.462+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.458 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.462+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1969953856}) 2024-04-26T00:34:24.458 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.462+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3258659334}) 2024-04-26T00:34:24.459 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.462+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.459 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.462+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1969953856}) 2024-04-26T00:34:24.459 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.462+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2392072890}) 2024-04-26T00:34:24.522 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.526+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.522 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.526+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2779209028}) 2024-04-26T00:34:24.522 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.526+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2476725289}) 2024-04-26T00:34:24.522 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.526+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.522 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.526+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2779209028}) 2024-04-26T00:34:24.522 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.526+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1721271314}) 2024-04-26T00:34:24.540 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.542+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.541 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.546+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=76} crc {osdmap=3195007047}) 2024-04-26T00:34:24.541 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.546+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=78} crc {osdmap=4015652182}) 2024-04-26T00:34:24.541 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.546+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:34:24.541 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.546+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=76} crc {osdmap=3195007047}) 2024-04-26T00:34:24.541 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:34:24.546+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=78} crc {osdmap=319972095}) 2024-04-26T00:34:24.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.562+0000 7fd458f58640 10 monclient: get_monmap_and_config 2024-04-26T00:34:24.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.562+0000 7fd458f58640 10 monclient: build_initial_monmap 2024-04-26T00:34:24.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.562+0000 7fd458f58640 10 monclient: monmap: 2024-04-26T00:34:24.560 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:34:24.561 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:24.561 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:34:24.567455+0000 2024-04-26T00:34:24.561 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:34:24.567455+0000 2024-04-26T00:34:24.561 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:34:24.561 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:24.561 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:34:24.561 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:34:24.561 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:34:24.561 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:34:24.561 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:34:24.561 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:24.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.570+0000 7fd458f58640 1 Processor -- start 2024-04-26T00:34:24.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.570+0000 7fd458f58640 1 -- start start 2024-04-26T00:34:24.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.570+0000 7fd458f58640 10 monclient: init 2024-04-26T00:34:24.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.570+0000 7fd458f58640 10 monclient: _reopen_session rank -1 2024-04-26T00:34:24.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.570+0000 7fd458f58640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-04-26T00:34:24.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.570+0000 7fd458f58640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd4540a3fe0 0x7fd45409c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:24.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.570+0000 7fd458f58640 10 monclient(hunting): picked mon.noname-a con 0x7fd4540a3fe0 addr v2:172.21.15.37:3300/0 2024-04-26T00:34:24.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.570+0000 7fd458f58640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd45409cda0 addr v1:172.21.15.45:6789/0 2024-04-26T00:34:24.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.570+0000 7fd458f58640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd4540a4ad0 addr v1:172.21.15.37:6789/0 2024-04-26T00:34:24.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.570+0000 7fd458f58640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd45409d210 con 0x7fd4540a4ad0 2024-04-26T00:34:24.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.570+0000 7fd458f58640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd45409bb20 con 0x7fd45409cda0 2024-04-26T00:34:24.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.570+0000 7fd458f58640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:24.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.570+0000 7fd458f58640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fd4540a5960 con 0x7fd4540a3fe0 2024-04-26T00:34:24.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.570+0000 7fd458f58640 10 monclient(hunting): _renew_subs 2024-04-26T00:34:24.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.570+0000 7fd458f58640 10 monclient(hunting): authenticate will time out at 1845.879440s 2024-04-26T00:34:24.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.570+0000 7fd45259c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd4540a3fe0 0x7fd45409c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:24.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.570+0000 7fd452d9d640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fd4540a4ad0 0x7fd4541427c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:33404/0 (socket says 172.21.15.37:33404) 2024-04-26T00:34:24.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.570+0000 7fd452d9d640 1 -- 172.21.15.37:0/3283890298 learned_addr learned my addr 172.21.15.37:0/3283890298 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:34:24.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.570+0000 7fd45259c640 10 monclient(hunting): get_auth_request con 0x7fd4540a3fe0 auth_method 0 2024-04-26T00:34:24.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.570+0000 7fd45259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:24.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.570+0000 7fd45259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:24.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.570+0000 7fd45259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:24.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.570+0000 7fd45259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:24.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.570+0000 7fd45259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:24.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.570+0000 7fd45259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:24.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.574+0000 7fd45259c640 10 monclient(hunting): handle_auth_done global_id 144417 payload 274 2024-04-26T00:34:24.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.574+0000 7fd45259c640 1 -- 172.21.15.37:0/3283890298 >> v1:172.21.15.45:6789/0 conn(0x7fd45409cda0 legacy=0x7fd4540a43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:34:24.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.574+0000 7fd45259c640 1 -- 172.21.15.37:0/3283890298 >> v1:172.21.15.37:6789/0 conn(0x7fd4540a4ad0 legacy=0x7fd4541427c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:34:24.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.574+0000 7fd45259c640 10 monclient: _finish_hunting 0 2024-04-26T00:34:24.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.574+0000 7fd45259c640 1 monclient: found mon.noname-a 2024-04-26T00:34:24.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.574+0000 7fd45259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:34:24.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.574+0000 7fd45259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:34:24.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.574+0000 7fd45259c640 1 -- 172.21.15.37:0/3283890298 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd45409b840 con 0x7fd4540a3fe0 2024-04-26T00:34:24.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.578+0000 7fd45259c640 1 -- 172.21.15.37:0/3283890298 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd4540a3fe0 msgr2=0x7fd45409c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:24.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.578+0000 7fd45259c640 1 --2- 172.21.15.37:0/3283890298 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd4540a3fe0 0x7fd45409c860 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd44c009d90 tx=0x7fd44c01c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd0e702209909ef1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:24.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.578+0000 7fd45159a640 1 -- 172.21.15.37:0/3283890298 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd44c0ac070 con 0x7fd4540a3fe0 2024-04-26T00:34:24.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.578+0000 7fd45159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:24.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.578+0000 7fd45159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:34:24.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.578+0000 7fd45159a640 10 monclient: dump: 2024-04-26T00:34:24.574 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:24.574 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:24.574 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:24.574 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:24.574 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:24.574 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:24.574 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:24.574 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:24.574 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:24.574 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:24.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.578+0000 7fd45159a640 10 monclient: _finish_auth 0 2024-04-26T00:34:24.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.578+0000 7fd45159a640 10 monclient: _check_auth_tickets 2024-04-26T00:34:24.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.578+0000 7fd45159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:34:24.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.578+0000 7fd45159a640 1 -- 172.21.15.37:0/3283890298 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd44c0a6930 con 0x7fd4540a3fe0 2024-04-26T00:34:24.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.578+0000 7fd45159a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:34:24.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.578+0000 7fd45159a640 1 -- 172.21.15.37:0/3283890298 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd44c0a6d00 con 0x7fd4540a3fe0 2024-04-26T00:34:24.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.578+0000 7fd45159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:24.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.578+0000 7fd45159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:34:24.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.578+0000 7fd45159a640 10 monclient: dump: 2024-04-26T00:34:24.576 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:24.576 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:24.576 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:24.576 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:24.576 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:24.576 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:24.576 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:24.576 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:24.576 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:24.576 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:24.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.578+0000 7fd458f58640 5 monclient: authenticate success, global_id 144417 2024-04-26T00:34:24.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.578+0000 7fd458f58640 10 monclient: get_monmap_and_config success 2024-04-26T00:34:24.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.578+0000 7fd458f58640 4 set_mon_vals no callback set 2024-04-26T00:34:24.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.578+0000 7fd458f58640 10 monclient: shutdown 2024-04-26T00:34:24.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.578+0000 7fd458f58640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:34:24.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.578+0000 7fd458f58640 1 -- 172.21.15.37:0/3283890298 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd4540a3fe0 msgr2=0x7fd45409c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:24.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.578+0000 7fd458f58640 1 --2- 172.21.15.37:0/3283890298 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd4540a3fe0 0x7fd45409c860 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd44c009d90 tx=0x7fd44c01c660 comp rx=0 tx=0).stop 2024-04-26T00:34:24.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.578+0000 7fd458f58640 1 -- 172.21.15.37:0/3283890298 shutdown_connections 2024-04-26T00:34:24.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.578+0000 7fd458f58640 1 --2- 172.21.15.37:0/3283890298 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd4540a3fe0 0x7fd45409c860 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:24.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.578+0000 7fd458f58640 1 -- 172.21.15.37:0/3283890298 >> 172.21.15.37:0/3283890298 conn(0x7fd4540a64e0 msgr2=0x7fd4540a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:34:24.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.578+0000 7fd458f58640 1 -- 172.21.15.37:0/3283890298 shutdown_connections 2024-04-26T00:34:24.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.578+0000 7fd458f58640 1 -- 172.21.15.37:0/3283890298 wait complete. 2024-04-26T00:34:24.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.578+0000 7fd458f58640 10 monclient: build_initial_monmap 2024-04-26T00:34:24.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.578+0000 7fd458f58640 10 monclient: monmap: 2024-04-26T00:34:24.582 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:34:24.582 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:34:24.582 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:34:24.582 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:34:24.583 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:34:24.583 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:34:24.583 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:34:24.583 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:34:24.583 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:34:24.583 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:24.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.578+0000 7fd458f58640 1 Processor -- start 2024-04-26T00:34:24.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.578+0000 7fd458f58640 1 -- start start 2024-04-26T00:34:24.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.578+0000 7fd458f58640 10 monclient: init 2024-04-26T00:34:24.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd458f58640 10 monclient: _reopen_session rank -1 2024-04-26T00:34:24.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd458f58640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:34:24.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd458f58640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd45409cda0 0x7fd4541d3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:24.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd458f58640 10 monclient(hunting): picked mon.noname-c con 0x7fd45409cda0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:34:24.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd458f58640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4540a3fe0 0x7fd45413fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:24.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd458f58640 10 monclient(hunting): picked mon.noname-b con 0x7fd4540a3fe0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:34:24.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd458f58640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd4540a4ad0 0x7fd45413ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:24.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd458f58640 10 monclient(hunting): picked mon.noname-a con 0x7fd4540a4ad0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:34:24.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd458f58640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:24.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd458f58640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd454146a90 con 0x7fd4540a4ad0 2024-04-26T00:34:24.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd458f58640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:24.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd458f58640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd454146910 con 0x7fd4540a3fe0 2024-04-26T00:34:24.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd458f58640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:24.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd458f58640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd454146c10 con 0x7fd45409cda0 2024-04-26T00:34:24.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd458f58640 10 monclient(hunting): _renew_subs 2024-04-26T00:34:24.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd458f58640 10 monclient(hunting): authenticate will time out at 1845.888174s 2024-04-26T00:34:24.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd45259c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd45409cda0 0x7fd4541d3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:24.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd45259c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd45409cda0 0x7fd4541d3490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:58048/0 (socket says 172.21.15.37:58048) 2024-04-26T00:34:24.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd452d9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd4540a4ad0 0x7fd45413ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:24.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd45259c640 1 -- 172.21.15.37:0/873204888 learned_addr learned my addr 172.21.15.37:0/873204888 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:34:24.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd45259c640 10 monclient(hunting): get_auth_request con 0x7fd45409cda0 auth_method 0 2024-04-26T00:34:24.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd45259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:24.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd45259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:24.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd45259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:24.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd452d9d640 10 monclient(hunting): get_auth_request con 0x7fd4540a4ad0 auth_method 0 2024-04-26T00:34:24.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd452d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:24.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd452d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:24.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd452d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:24.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd45259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:24.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd45259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:24.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd45259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:24.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd452d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:24.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd452d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:24.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd452d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:24.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd451d9b640 1 --2- 172.21.15.37:0/873204888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4540a3fe0 0x7fd45413fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:24.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd451d9b640 10 monclient(hunting): get_auth_request con 0x7fd4540a3fe0 auth_method 0 2024-04-26T00:34:24.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd451d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:24.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd451d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:24.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd451d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:24.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd45259c640 10 monclient(hunting): handle_auth_done global_id 154292 payload 824 2024-04-26T00:34:24.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd45259c640 1 -- 172.21.15.37:0/873204888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4540a3fe0 msgr2=0x7fd45413fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:24.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd45259c640 1 --2- 172.21.15.37:0/873204888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4540a3fe0 0x7fd45413fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:24.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd45259c640 1 -- 172.21.15.37:0/873204888 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd4540a4ad0 msgr2=0x7fd45413ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:24.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd45259c640 1 --2- 172.21.15.37:0/873204888 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd4540a4ad0 0x7fd45413ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:24.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd45259c640 10 monclient: _finish_hunting 0 2024-04-26T00:34:24.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd45259c640 1 monclient: found mon.noname-c 2024-04-26T00:34:24.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd45259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:34:24.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd45259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:34:24.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd45259c640 1 -- 172.21.15.37:0/873204888 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd454143230 con 0x7fd45409cda0 2024-04-26T00:34:24.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd452d9d640 1 --2- 172.21.15.37:0/873204888 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd4540a4ad0 0x7fd45413ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:34:24.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd45259c640 1 -- 172.21.15.37:0/873204888 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd45409cda0 msgr2=0x7fd4541d3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:24.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd45259c640 1 --2- 172.21.15.37:0/873204888 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd45409cda0 0x7fd4541d3490 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd44c01ad10 tx=0x7fd44c019400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:24.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd442ffd640 1 -- 172.21.15.37:0/873204888 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd44c0c0070 con 0x7fd45409cda0 2024-04-26T00:34:24.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd442ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:24.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd442ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:34:24.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd442ffd640 10 monclient: dump: 2024-04-26T00:34:24.587 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:24.587 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:24.587 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:24.587 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:24.587 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:24.588 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:24.588 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:24.588 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:24.588 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:24.588 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:24.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd442ffd640 10 monclient: _finish_auth 0 2024-04-26T00:34:24.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd442ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:34:24.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd442ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:34:24.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd442ffd640 1 -- 172.21.15.37:0/873204888 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd44c0b1870 con 0x7fd45409cda0 2024-04-26T00:34:24.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd458f58640 5 monclient: authenticate success, global_id 154292 2024-04-26T00:34:24.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd442ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:34:24.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd458f58640 10 monclient: _renew_subs 2024-04-26T00:34:24.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd442ffd640 1 -- 172.21.15.37:0/873204888 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd44c0ac050 con 0x7fd45409cda0 2024-04-26T00:34:24.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd458f58640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:34:24.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd458f58640 1 -- 172.21.15.37:0/873204888 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4540a10d0 con 0x7fd45409cda0 2024-04-26T00:34:24.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd442ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:24.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd442ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:34:24.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd442ffd640 10 monclient: dump: 2024-04-26T00:34:24.589 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:24.589 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:24.589 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:24.589 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:24.589 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:24.589 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:24.589 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:24.589 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:24.589 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:24.589 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:24.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd443fff640 4 set_mon_vals no callback set 2024-04-26T00:34:24.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd458f58640 10 monclient: _renew_subs 2024-04-26T00:34:24.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd458f58640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:34:24.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.582+0000 7fd458f58640 1 -- 172.21.15.37:0/873204888 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd454143510 con 0x7fd45409cda0 2024-04-26T00:34:24.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.586+0000 7fd442ffd640 1 -- 172.21.15.37:0/873204888 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd44c0c3070 con 0x7fd45409cda0 2024-04-26T00:34:24.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.586+0000 7fd458f58640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:34:24.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.586+0000 7fd458f58640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:34:24.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.586+0000 7fd458f58640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:34:24.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.586+0000 7fd458f58640 1 -- 172.21.15.37:0/873204888 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd454066f40 con 0x7fd45409cda0 2024-04-26T00:34:24.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.586+0000 7fd442ffd640 1 --2- 172.21.15.37:0/873204888 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd434035e80 0x7fd434038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:24.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.586+0000 7fd442ffd640 1 -- 172.21.15.37:0/873204888 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(565..565 src has 370..565) ==== 3959+0+0 (secure 0 0 0) 0x7fd44c0e3360 con 0x7fd45409cda0 2024-04-26T00:34:24.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.586+0000 7fd451d9b640 1 --2- 172.21.15.37:0/873204888 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd434035e80 0x7fd434038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:24.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.586+0000 7fd451d9b640 10 monclient: get_auth_request con 0x7fd434035e80 auth_method 0 2024-04-26T00:34:24.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.586+0000 7fd451d9b640 1 --2- 172.21.15.37:0/873204888 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd434035e80 0x7fd434038340 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fd454143e30 tx=0x7fd448009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:24.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.590+0000 7fd442ffd640 1 -- 172.21.15.37:0/873204888 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fd45409bb20 con 0x7fd45409cda0 2024-04-26T00:34:24.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.590+0000 7fd442ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:34:24.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.590+0000 7fd442ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:34:24.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.818+0000 7fd458f58640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:34:24.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.818+0000 7fd458f58640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:34:24.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.818+0000 7fd458f58640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:34:24.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.818+0000 7fd458f58640 1 -- 172.21.15.37:0/873204888 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd454141d70 con 0x7fd45409cda0 2024-04-26T00:34:24.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.818+0000 7fd442ffd640 1 -- 172.21.15.37:0/873204888 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fd44c0b1ce0 con 0x7fd45409cda0 2024-04-26T00:34:24.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.818+0000 7fd442ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:34:24.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.818+0000 7fd442ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:34:24.814 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":108,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:34:24.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.818+0000 7fd458f58640 1 -- 172.21.15.37:0/873204888 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd434035e80 msgr2=0x7fd434038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:24.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.818+0000 7fd458f58640 1 --2- 172.21.15.37:0/873204888 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd434035e80 0x7fd434038340 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fd454143e30 tx=0x7fd448009040 comp rx=0 tx=0).stop 2024-04-26T00:34:24.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.818+0000 7fd458f58640 10 monclient: shutdown 2024-04-26T00:34:24.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.818+0000 7fd458f58640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:34:24.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.818+0000 7fd458f58640 1 -- 172.21.15.37:0/873204888 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd45409cda0 msgr2=0x7fd4541d3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:24.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.818+0000 7fd458f58640 1 --2- 172.21.15.37:0/873204888 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd45409cda0 0x7fd4541d3490 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd44c01ad10 tx=0x7fd44c019400 comp rx=0 tx=0).stop 2024-04-26T00:34:24.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.818+0000 7fd458f58640 1 -- 172.21.15.37:0/873204888 shutdown_connections 2024-04-26T00:34:24.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.818+0000 7fd458f58640 1 --2- 172.21.15.37:0/873204888 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd434035e80 0x7fd434038340 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:24.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.818+0000 7fd458f58640 1 --2- 172.21.15.37:0/873204888 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd4540a4ad0 0x7fd45413ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:24.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.818+0000 7fd458f58640 1 --2- 172.21.15.37:0/873204888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4540a3fe0 0x7fd45413fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:24.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.818+0000 7fd458f58640 1 --2- 172.21.15.37:0/873204888 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd45409cda0 0x7fd4541d3490 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:24.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.818+0000 7fd458f58640 1 -- 172.21.15.37:0/873204888 >> 172.21.15.37:0/873204888 conn(0x7fd4540a64e0 msgr2=0x7fd45409f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:34:24.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.818+0000 7fd458f58640 1 -- 172.21.15.37:0/873204888 shutdown_connections 2024-04-26T00:34:24.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.818+0000 7fd458f58640 1 -- 172.21.15.37:0/873204888 wait complete. 2024-04-26T00:34:24.833 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:34:24.833 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-26T00:34:24.833 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:34:24.833 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:34:24.857 INFO:tasks.rados.rados.0.smithi045.stdout:2458: done (0 left) 2024-04-26T00:34:24.857 INFO:tasks.rados.rados.0.smithi045.stdout:2459: snap_remove snap 195 2024-04-26T00:34:24.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.998+0000 7f1c71a84640 10 monclient: get_monmap_and_config 2024-04-26T00:34:24.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.998+0000 7f1c71a84640 10 monclient: build_initial_monmap 2024-04-26T00:34:24.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:24.998+0000 7f1c71a84640 10 monclient: monmap: 2024-04-26T00:34:24.996 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:34:24.996 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:24.996 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:34:25.003801+0000 2024-04-26T00:34:24.996 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:34:25.003801+0000 2024-04-26T00:34:24.997 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:34:24.997 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:24.997 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:34:24.997 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:34:24.997 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:34:24.997 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:34:24.997 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:34:24.997 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:24.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c71a84640 1 Processor -- start 2024-04-26T00:34:24.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c71a84640 1 -- start start 2024-04-26T00:34:24.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c71a84640 10 monclient: init 2024-04-26T00:34:25.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c71a84640 10 monclient: _reopen_session rank -1 2024-04-26T00:34:25.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c71a84640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-04-26T00:34:25.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c71a84640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1c6c0a0300 0x7f1c6c09fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:25.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c71a84640 10 monclient(hunting): picked mon.noname-b con 0x7f1c6c0a0300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:34:25.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c71a84640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1c6c0a06e0 addr v1:172.21.15.37:6789/0 2024-04-26T00:34:25.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c71a84640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f1c6c1352b0 0x7f1c6c13cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:25.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c71a84640 10 monclient(hunting): picked mon.noname-a con 0x7f1c6c1352b0 addr v2:172.21.15.37:3300/0 2024-04-26T00:34:25.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c71a84640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c6c09c190 con 0x7f1c6c0a06e0 2024-04-26T00:34:25.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c71a84640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:25.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c71a84640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f1c6c0a13f0 con 0x7f1c6c1352b0 2024-04-26T00:34:25.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c71a84640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:25.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c71a84640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f1c6c0a1110 con 0x7f1c6c0a0300 2024-04-26T00:34:25.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c71a84640 10 monclient(hunting): _renew_subs 2024-04-26T00:34:25.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c71a84640 10 monclient(hunting): authenticate will time out at 1846.309459s 2024-04-26T00:34:25.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c6b7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1c6c0a0300 0x7f1c6c09fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:25.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c6b7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1c6c0a0300 0x7f1c6c09fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:58052/0 (socket says 172.21.15.37:58052) 2024-04-26T00:34:25.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c6b7fe640 1 -- 172.21.15.37:0/512672151 learned_addr learned my addr 172.21.15.37:0/512672151 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:34:25.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c6b7fe640 10 monclient(hunting): get_auth_request con 0x7f1c6c0a0300 auth_method 0 2024-04-26T00:34:25.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:25.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c6b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:25.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:25.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c6bfff640 1 --2- 172.21.15.37:0/512672151 >> v2:172.21.15.37:3300/0 conn(0x7f1c6c1352b0 0x7f1c6c13cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:25.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c6bfff640 10 monclient(hunting): get_auth_request con 0x7f1c6c1352b0 auth_method 0 2024-04-26T00:34:25.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:25.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c6bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:25.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:25.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c6b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:25.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c6b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:25.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c6b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:25.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:25.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:25.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:25.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c6b7fe640 10 monclient(hunting): handle_auth_done global_id 154295 payload 274 2024-04-26T00:34:25.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c6b7fe640 1 -- 172.21.15.37:0/512672151 >> v2:172.21.15.37:3300/0 conn(0x7f1c6c1352b0 msgr2=0x7f1c6c13cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:34:25.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c6b7fe640 1 --2- 172.21.15.37:0/512672151 >> v2:172.21.15.37:3300/0 conn(0x7f1c6c1352b0 0x7f1c6c13cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:25.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c6b7fe640 1 -- 172.21.15.37:0/512672151 >> v1:172.21.15.37:6789/0 conn(0x7f1c6c0a06e0 legacy=0x7f1c6c134ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:34:25.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c6b7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:34:25.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c6b7fe640 1 monclient: found mon.noname-b 2024-04-26T00:34:25.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c6b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:34:25.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c6b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:34:25.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c6b7fe640 1 -- 172.21.15.37:0/512672151 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c6c0a16d0 con 0x7f1c6c0a0300 2024-04-26T00:34:25.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.002+0000 7f1c6bfff640 1 --2- 172.21.15.37:0/512672151 >> v2:172.21.15.37:3300/0 conn(0x7f1c6c1352b0 0x7f1c6c13cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:34:25.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.006+0000 7f1c6b7fe640 1 -- 172.21.15.37:0/512672151 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1c6c0a0300 msgr2=0x7f1c6c09fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:25.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.006+0000 7f1c6b7fe640 1 --2- 172.21.15.37:0/512672151 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1c6c0a0300 0x7f1c6c09fc50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c009870 tx=0x7f1c5c01c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b00f058a9431005 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:25.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.006+0000 7f1c6a7fc640 1 -- 172.21.15.37:0/512672151 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c5c0b2070 con 0x7f1c6c0a0300 2024-04-26T00:34:25.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.006+0000 7f1c6a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:25.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.006+0000 7f1c6a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:34:25.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.006+0000 7f1c6a7fc640 10 monclient: dump: 2024-04-26T00:34:25.005 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:25.005 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:25.005 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.005 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.005 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:25.005 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:25.005 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:25.005 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:25.005 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:25.005 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:25.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.006+0000 7f1c6a7fc640 10 monclient: _finish_auth 0 2024-04-26T00:34:25.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.006+0000 7f1c6a7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:34:25.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.006+0000 7f1c6a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:34:25.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.006+0000 7f1c6a7fc640 1 -- 172.21.15.37:0/512672151 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c5c0ad680 con 0x7f1c6c0a0300 2024-04-26T00:34:25.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.006+0000 7f1c6a7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:34:25.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.006+0000 7f1c6a7fc640 1 -- 172.21.15.37:0/512672151 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c5c0ada50 con 0x7f1c6c0a0300 2024-04-26T00:34:25.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.006+0000 7f1c71a84640 5 monclient: authenticate success, global_id 154295 2024-04-26T00:34:25.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.006+0000 7f1c71a84640 10 monclient: get_monmap_and_config success 2024-04-26T00:34:25.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.006+0000 7f1c71a84640 4 set_mon_vals no callback set 2024-04-26T00:34:25.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.006+0000 7f1c6a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:25.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.006+0000 7f1c6a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:34:25.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.006+0000 7f1c6a7fc640 10 monclient: dump: 2024-04-26T00:34:25.006 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:25.006 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:25.006 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.007 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.007 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:25.007 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:25.007 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:25.007 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:25.007 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:25.007 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:25.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.006+0000 7f1c71a84640 10 monclient: shutdown 2024-04-26T00:34:25.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.006+0000 7f1c71a84640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:34:25.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.006+0000 7f1c71a84640 1 -- 172.21.15.37:0/512672151 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1c6c0a0300 msgr2=0x7f1c6c09fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:25.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.006+0000 7f1c71a84640 1 --2- 172.21.15.37:0/512672151 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1c6c0a0300 0x7f1c6c09fc50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c009870 tx=0x7f1c5c01c3e0 comp rx=0 tx=0).stop 2024-04-26T00:34:25.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.006+0000 7f1c71a84640 1 -- 172.21.15.37:0/512672151 shutdown_connections 2024-04-26T00:34:25.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.006+0000 7f1c71a84640 1 --2- 172.21.15.37:0/512672151 >> v2:172.21.15.37:3300/0 conn(0x7f1c6c1352b0 0x7f1c6c13cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:25.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.006+0000 7f1c71a84640 1 --2- 172.21.15.37:0/512672151 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1c6c0a0300 0x7f1c6c09fc50 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:25.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.006+0000 7f1c71a84640 1 -- 172.21.15.37:0/512672151 >> 172.21.15.37:0/512672151 conn(0x7f1c6c0a1d50 msgr2=0x7f1c6c0a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:34:25.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.006+0000 7f1c71a84640 1 -- 172.21.15.37:0/512672151 shutdown_connections 2024-04-26T00:34:25.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.006+0000 7f1c71a84640 1 -- 172.21.15.37:0/512672151 wait complete. 2024-04-26T00:34:25.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c71a84640 10 monclient: build_initial_monmap 2024-04-26T00:34:25.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c71a84640 10 monclient: monmap: 2024-04-26T00:34:25.008 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:34:25.008 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:34:25.008 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:34:25.008 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:34:25.008 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:34:25.008 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:34:25.008 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:34:25.008 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:34:25.008 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:34:25.008 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:25.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c71a84640 1 Processor -- start 2024-04-26T00:34:25.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c71a84640 1 -- start start 2024-04-26T00:34:25.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c71a84640 10 monclient: init 2024-04-26T00:34:25.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c71a84640 10 monclient: _reopen_session rank -1 2024-04-26T00:34:25.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c71a84640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:34:25.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c71a84640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c6c0a0300 0x7f1c6c1cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:25.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c71a84640 10 monclient(hunting): picked mon.noname-a con 0x7f1c6c0a0300 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:34:25.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c71a84640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1c6c0a06e0 0x7f1c6c1cf910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:25.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c71a84640 10 monclient(hunting): picked mon.noname-c con 0x7f1c6c0a06e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:34:25.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c71a84640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1c6c1352b0 0x7f1c6c1cfe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:25.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c71a84640 10 monclient(hunting): picked mon.noname-b con 0x7f1c6c1352b0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:34:25.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c71a84640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:25.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c71a84640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f1c6c13fc40 con 0x7f1c6c0a0300 2024-04-26T00:34:25.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c71a84640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:25.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c71a84640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1c6c13fdc0 con 0x7f1c6c1352b0 2024-04-26T00:34:25.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c71a84640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:25.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c71a84640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f1c6c13ff40 con 0x7f1c6c0a06e0 2024-04-26T00:34:25.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c71a84640 10 monclient(hunting): _renew_subs 2024-04-26T00:34:25.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c71a84640 10 monclient(hunting): authenticate will time out at 1846.317292s 2024-04-26T00:34:25.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c6b7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c6c0a0300 0x7f1c6c1cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:25.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c6b7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c6c0a0300 0x7f1c6c1cf3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33808/0 (socket says 172.21.15.37:33808) 2024-04-26T00:34:25.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c6affd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1c6c0a06e0 0x7f1c6c1cf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:25.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c6b7fe640 1 -- 172.21.15.37:0/1118602747 learned_addr learned my addr 172.21.15.37:0/1118602747 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:34:25.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c6b7fe640 10 monclient(hunting): get_auth_request con 0x7f1c6c0a0300 auth_method 0 2024-04-26T00:34:25.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:25.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c6b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:25.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:25.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c6affd640 10 monclient(hunting): get_auth_request con 0x7f1c6c0a06e0 auth_method 0 2024-04-26T00:34:25.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:25.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c6affd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:25.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c6affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:25.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c6b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:25.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c6b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:25.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c6b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:25.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c6affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:25.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c6affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:25.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c6affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:25.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c6bfff640 1 --2- 172.21.15.37:0/1118602747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1c6c1352b0 0x7f1c6c1cfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:25.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c6bfff640 10 monclient(hunting): get_auth_request con 0x7f1c6c1352b0 auth_method 0 2024-04-26T00:34:25.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:25.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c6bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:25.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:25.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c6b7fe640 10 monclient(hunting): handle_auth_done global_id 144426 payload 824 2024-04-26T00:34:25.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c6b7fe640 1 -- 172.21.15.37:0/1118602747 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1c6c0a06e0 msgr2=0x7f1c6c1cf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:25.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c6b7fe640 1 --2- 172.21.15.37:0/1118602747 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1c6c0a06e0 0x7f1c6c1cf910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:25.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c6b7fe640 1 -- 172.21.15.37:0/1118602747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1c6c1352b0 msgr2=0x7f1c6c1cfe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:25.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c6b7fe640 1 --2- 172.21.15.37:0/1118602747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1c6c1352b0 0x7f1c6c1cfe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:25.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c6b7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:34:25.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c6b7fe640 1 monclient: found mon.noname-a 2024-04-26T00:34:25.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c6b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:34:25.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c6b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:34:25.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c6b7fe640 1 -- 172.21.15.37:0/1118602747 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c6c13b8d0 con 0x7f1c6c0a0300 2024-04-26T00:34:25.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.010+0000 7f1c6affd640 1 --2- 172.21.15.37:0/1118602747 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1c6c0a06e0 0x7f1c6c1cf910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:34:25.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c6b7fe640 1 -- 172.21.15.37:0/1118602747 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c6c0a0300 msgr2=0x7f1c6c1cf3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:25.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c6b7fe640 1 --2- 172.21.15.37:0/1118602747 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c6c0a0300 0x7f1c6c1cf3d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c01c210 tx=0x7f1c5c01ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:25.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c53fff640 1 -- 172.21.15.37:0/1118602747 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c5c0bc020 con 0x7f1c6c0a0300 2024-04-26T00:34:25.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c53fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:25.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c53fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:34:25.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c53fff640 10 monclient: dump: 2024-04-26T00:34:25.018 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:25.018 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:25.018 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.018 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.018 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:25.019 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:25.019 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:25.019 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:25.019 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:25.019 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:25.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c53fff640 10 monclient: _finish_auth 0 2024-04-26T00:34:25.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c53fff640 10 monclient: _check_auth_tickets 2024-04-26T00:34:25.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c53fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:34:25.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c53fff640 1 -- 172.21.15.37:0/1118602747 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c5c0add20 con 0x7f1c6c0a0300 2024-04-26T00:34:25.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c71a84640 5 monclient: authenticate success, global_id 144426 2024-04-26T00:34:25.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c71a84640 10 monclient: _renew_subs 2024-04-26T00:34:25.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c71a84640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:34:25.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c71a84640 1 -- 172.21.15.37:0/1118602747 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c6c13bbb0 con 0x7f1c6c0a0300 2024-04-26T00:34:25.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c53fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:34:25.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c53fff640 1 -- 172.21.15.37:0/1118602747 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c5c0b2040 con 0x7f1c6c0a0300 2024-04-26T00:34:25.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c53fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:25.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c53fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:34:25.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c53fff640 10 monclient: dump: 2024-04-26T00:34:25.020 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:25.020 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:25.020 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.020 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.020 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:25.020 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:25.020 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:25.020 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:25.020 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:25.020 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:25.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c69ffb640 4 set_mon_vals no callback set 2024-04-26T00:34:25.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c71a84640 10 monclient: _renew_subs 2024-04-26T00:34:25.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c71a84640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:34:25.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c71a84640 1 -- 172.21.15.37:0/1118602747 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c6c09c750 con 0x7f1c6c0a0300 2024-04-26T00:34:25.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c71a84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:34:25.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c71a84640 1 --2- 172.21.15.37:0/1118602747 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c6c12d9c0 0x7f1c6c12dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:25.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c71a84640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:25.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c71a84640 1 -- 172.21.15.37:0/1118602747 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f1c6c0a0cd0 con 0x7f1c6c12d9c0 2024-04-26T00:34:25.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c53fff640 1 -- 172.21.15.37:0/1118602747 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f1c5c0bf020 con 0x7f1c6c0a0300 2024-04-26T00:34:25.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c6affd640 1 --2- 172.21.15.37:0/1118602747 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c6c12d9c0 0x7f1c6c12dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:25.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c6affd640 10 monclient: get_auth_request con 0x7f1c6c12d9c0 auth_method 0 2024-04-26T00:34:25.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:25.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c6affd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:25.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c6affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:25.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c6affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:25.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c6affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:25.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c53fff640 1 --2- 172.21.15.37:0/1118602747 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1c58035e80 0x7f1c58038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:25.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c6affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:25.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c53fff640 1 -- 172.21.15.37:0/1118602747 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(565..565 src has 370..565) ==== 3959+0+0 (secure 0 0 0) 0x7f1c5c0e2b60 con 0x7f1c6c0a0300 2024-04-26T00:34:25.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c6affd640 10 monclient(hunting): handle_auth_done global_id 144429 payload 824 2024-04-26T00:34:25.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c6affd640 1 -- 172.21.15.37:0/1118602747 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1c6c04f510 con 0x7f1c6c12d9c0 2024-04-26T00:34:25.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c6bfff640 1 --2- 172.21.15.37:0/1118602747 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1c58035e80 0x7f1c58038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:25.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.014+0000 7f1c6bfff640 10 monclient: get_auth_request con 0x7f1c58035e80 auth_method 0 2024-04-26T00:34:25.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.018+0000 7f1c6affd640 1 -- 172.21.15.37:0/1118602747 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c6c12d9c0 msgr2=0x7f1c6c12dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:25.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.018+0000 7f1c6affd640 1 --2- 172.21.15.37:0/1118602747 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c6c12d9c0 0x7f1c6c12dda0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1c60007fd0 tx=0x7f1c6000ff70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:25.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.018+0000 7f1c53fff640 1 -- 172.21.15.37:0/1118602747 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c60018020 con 0x7f1c6c12d9c0 2024-04-26T00:34:25.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.018+0000 7f1c53fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:25.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.018+0000 7f1c53fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:34:25.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.018+0000 7f1c53fff640 10 monclient: dump: 2024-04-26T00:34:25.023 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:25.023 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:25.023 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.023 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.023 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:25.023 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:25.023 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:25.023 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:25.023 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:25.023 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:25.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.018+0000 7f1c53fff640 1 -- 172.21.15.37:0/1118602747 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1c6000dd60 con 0x7f1c6c12d9c0 2024-04-26T00:34:25.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.018+0000 7f1c53fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:34:25.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.018+0000 7f1c53fff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:34:25.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.018+0000 7f1c53fff640 1 -- 172.21.15.37:0/1118602747 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c6c12d9c0 msgr2=0x7f1c6c12dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:25.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.018+0000 7f1c53fff640 1 --2- 172.21.15.37:0/1118602747 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c6c12d9c0 0x7f1c6c12dda0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1c60007fd0 tx=0x7f1c6000ff70 comp rx=0 tx=0).stop 2024-04-26T00:34:25.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.018+0000 7f1c53fff640 1 -- 172.21.15.37:0/1118602747 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c6c12d9c0 msgr2=0x7f1c6c12dda0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:34:25.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.018+0000 7f1c53fff640 1 --2- 172.21.15.37:0/1118602747 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c6c12d9c0 0x7f1c6c12dda0 secure :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1c60007fd0 tx=0x7f1c6000ff70 comp rx=0 tx=0).stop 2024-04-26T00:34:25.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.018+0000 7f1c6bfff640 1 --2- 172.21.15.37:0/1118602747 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1c58035e80 0x7f1c58038340 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f1c64007520 tx=0x7f1c6400b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:25.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.022+0000 7f1c71a84640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:34:25.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.022+0000 7f1c71a84640 1 --2- 172.21.15.37:0/1118602747 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c6c12e2e0 0x7f1c6c06f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:25.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.022+0000 7f1c71a84640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:25.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.022+0000 7f1c71a84640 1 -- 172.21.15.37:0/1118602747 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f1c6c09eb60 con 0x7f1c6c12e2e0 2024-04-26T00:34:25.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.022+0000 7f1c6affd640 1 --2- 172.21.15.37:0/1118602747 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c6c12e2e0 0x7f1c6c06f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:25.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.022+0000 7f1c6affd640 10 monclient: get_auth_request con 0x7f1c6c12e2e0 auth_method 0 2024-04-26T00:34:25.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.022+0000 7f1c6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:25.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.022+0000 7f1c6affd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:25.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.022+0000 7f1c6affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:25.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.022+0000 7f1c6affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:25.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.022+0000 7f1c6affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:25.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.022+0000 7f1c6affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:25.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.022+0000 7f1c6affd640 10 monclient(hunting): handle_auth_done global_id 144432 payload 824 2024-04-26T00:34:25.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.022+0000 7f1c6affd640 1 -- 172.21.15.37:0/1118602747 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1c6c06ba50 con 0x7f1c6c12e2e0 2024-04-26T00:34:25.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.026+0000 7f1c6affd640 1 -- 172.21.15.37:0/1118602747 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c6c12e2e0 msgr2=0x7f1c6c06f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:25.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.026+0000 7f1c6affd640 1 --2- 172.21.15.37:0/1118602747 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c6c12e2e0 0x7f1c6c06f630 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1c60019000 tx=0x7f1c60003700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:25.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.026+0000 7f1c53fff640 1 -- 172.21.15.37:0/1118602747 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c60018030 con 0x7f1c6c12e2e0 2024-04-26T00:34:25.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.026+0000 7f1c53fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:25.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.026+0000 7f1c53fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:34:25.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.026+0000 7f1c53fff640 10 monclient: dump: 2024-04-26T00:34:25.027 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:25.027 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:25.027 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.027 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.027 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:25.027 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:25.027 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:25.027 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:25.028 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:25.028 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:25.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.026+0000 7f1c53fff640 1 -- 172.21.15.37:0/1118602747 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4890 (secure 0 0 0) 0x7f1c60014b80 con 0x7f1c6c12e2e0 2024-04-26T00:34:25.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.026+0000 7f1c53fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:34:25.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.026+0000 7f1c53fff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:34:25.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.026+0000 7f1c53fff640 1 -- 172.21.15.37:0/1118602747 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c6c12e2e0 msgr2=0x7f1c6c06f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:25.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.026+0000 7f1c53fff640 1 --2- 172.21.15.37:0/1118602747 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c6c12e2e0 0x7f1c6c06f630 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1c60019000 tx=0x7f1c60003700 comp rx=0 tx=0).stop 2024-04-26T00:34:25.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.026+0000 7f1c53fff640 1 -- 172.21.15.37:0/1118602747 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c6c12e2e0 msgr2=0x7f1c6c06f630 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:34:25.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.026+0000 7f1c53fff640 1 --2- 172.21.15.37:0/1118602747 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c6c12e2e0 0x7f1c6c06f630 secure :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1c60019000 tx=0x7f1c60003700 comp rx=0 tx=0).stop 2024-04-26T00:34:25.028 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:34:25.028 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:34:25.029 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:34:25.029 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:34:25.029 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 108, 2024-04-26T00:34:25.029 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 151570, 2024-04-26T00:34:25.029 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:34:25.029 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:34:25.029 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:34:25.029 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:34:25.029 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:25.029 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 3, 2024-04-26T00:34:25.029 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:34:25.029 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:34:25.029 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:34:25.029 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:34:25.029 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:34:25.030 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:34:25.030 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:34:25.030 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:34:25.030 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:34:25.030 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:34:25.030 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:34:25.030 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:34:25.030 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:34:25.030 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:34:25.030 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:25.030 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:34:25.030 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:34:25.030 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:34:25.031 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:34:25.031 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:34:25.031 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:34:25.031 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:34:25.031 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:34:25.031 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:34:25.031 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:34:25.031 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:34:25.031 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:34:25.031 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:34:25.031 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:25.031 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:25.031 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:34:25.031 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:34:25.032 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:34:25.032 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:34:25.032 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:34:25.032 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:34:25.032 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:34:25.032 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:34:25.032 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:34:25.032 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:34:25.032 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:34:25.032 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:34:25.032 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:34:25.033 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:34:25.033 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:34:25.033 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:34:25.033 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:34:25.033 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:34:25.033 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:34:25.033 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:34:25.033 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:34:25.034 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:34:25.034 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:34:25.034 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:34:25.034 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:34:25.034 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:34:25.034 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:34:25.034 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:34:25.034 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:25.034 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:34:25.034 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:25.035 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:34:25.035 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:25.035 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:34:25.035 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:34:25.035 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:34:25.035 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:34:25.035 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:25.035 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:34:25.035 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:34:25.036 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:25.036 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:25.036 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:25.036 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:34:25.036 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:34:25.036 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:25.036 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:25.036 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:25.036 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:25.036 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:34:25.037 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:34:25.037 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:34:25.037 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:34:25.037 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:34:25.037 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:25.037 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:25.037 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:34:25.037 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:34:25.037 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:34:25.038 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:34:25.038 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:25.038 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:34:25.038 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:34:25.038 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:25.038 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:25.038 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:25.038 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:34:25.038 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:34:25.039 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:25.039 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:25.039 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:25.039 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:25.039 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:34:25.039 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:34:25.039 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:34:25.039 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:34:25.039 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:34:25.039 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:25.040 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:25.040 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:34:25.040 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:34:25.040 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:34:25.040 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:34:25.040 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:25.040 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:34:25.040 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:34:25.040 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:25.041 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:25.041 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:25.041 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:34:25.041 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:34:25.041 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:25.041 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:25.041 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:25.041 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:25.041 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:34:25.041 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:34:25.042 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:34:25.042 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:34:25.042 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:34:25.042 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:25.042 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:25.042 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:25.042 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:34:25.042 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:34:25.042 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:25.043 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:34:25.043 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:34:25.043 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:34:25.043 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:25.043 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:25.043 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:34:25.043 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:25.043 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:34:25.043 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:34:25.044 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:34:25.044 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:25.044 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:25.044 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:34:25.044 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:25.044 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:34:25.044 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:34:25.044 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:34:25.044 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:25.044 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:25.044 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:34:25.045 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:25.045 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:34:25.045 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:34:25.045 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:34:25.045 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:25.045 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:25.045 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:25.045 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:34:25.045 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:34:25.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.026+0000 7f1c71a84640 1 -- 172.21.15.37:0/1118602747 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1c58035e80 msgr2=0x7f1c58038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:25.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.026+0000 7f1c71a84640 1 --2- 172.21.15.37:0/1118602747 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1c58035e80 0x7f1c58038340 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f1c64007520 tx=0x7f1c6400b040 comp rx=0 tx=0).stop 2024-04-26T00:34:25.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.026+0000 7f1c71a84640 10 monclient: shutdown 2024-04-26T00:34:25.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.026+0000 7f1c71a84640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:34:25.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.026+0000 7f1c71a84640 1 -- 172.21.15.37:0/1118602747 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c6c0a0300 msgr2=0x7f1c6c1cf3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:25.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.026+0000 7f1c71a84640 1 --2- 172.21.15.37:0/1118602747 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1c6c0a0300 0x7f1c6c1cf3d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c01c210 tx=0x7f1c5c01ac20 comp rx=0 tx=0).stop 2024-04-26T00:34:25.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.026+0000 7f1c6bfff640 1 -- 172.21.15.37:0/1118602747 reap_dead start 2024-04-26T00:34:25.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.026+0000 7f1c6bfff640 1 -- 172.21.15.37:0/1118602747 reap_dead start 2024-04-26T00:34:25.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.026+0000 7f1c71a84640 1 -- 172.21.15.37:0/1118602747 shutdown_connections 2024-04-26T00:34:25.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.026+0000 7f1c71a84640 1 -- 172.21.15.37:0/1118602747 >> 172.21.15.37:0/1118602747 conn(0x7f1c6c0a1d50 msgr2=0x7f1c6c071130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:34:25.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.026+0000 7f1c71a84640 1 -- 172.21.15.37:0/1118602747 shutdown_connections 2024-04-26T00:34:25.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.026+0000 7f1c71a84640 1 -- 172.21.15.37:0/1118602747 wait complete. 2024-04-26T00:34:25.048 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:34:25.048 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:34:25.048 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:34:25.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f4337d4f640 10 monclient: get_monmap_and_config 2024-04-26T00:34:25.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f4337d4f640 10 monclient: build_initial_monmap 2024-04-26T00:34:25.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f4337d4f640 10 monclient: monmap: 2024-04-26T00:34:25.165 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:34:25.165 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:25.165 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:34:25.172770+0000 2024-04-26T00:34:25.165 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:34:25.172770+0000 2024-04-26T00:34:25.166 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:34:25.166 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:25.166 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:34:25.166 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:34:25.166 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:34:25.166 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:34:25.166 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:34:25.166 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:25.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f4337d4f640 1 Processor -- start 2024-04-26T00:34:25.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f4337d4f640 1 -- start start 2024-04-26T00:34:25.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f4337d4f640 10 monclient: init 2024-04-26T00:34:25.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f4337d4f640 10 monclient: _reopen_session rank -1 2024-04-26T00:34:25.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f4337d4f640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-04-26T00:34:25.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f4337d4f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f43300a3d90 0x7f433009bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:25.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f4337d4f640 10 monclient(hunting): picked mon.noname-b con 0x7f43300a3d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:34:25.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f4337d4f640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f43300a4170 0x7f43300a45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:25.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f4337d4f640 10 monclient(hunting): picked mon.noname-c con 0x7f43300a4170 addr v2:172.21.15.45:3300/0 2024-04-26T00:34:25.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f4337d4f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4330133b20 addr v1:172.21.15.45:6789/0 2024-04-26T00:34:25.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f4337d4f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43300a5150 con 0x7f4330133b20 2024-04-26T00:34:25.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f4337d4f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:25.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f4337d4f640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f433009b280 con 0x7f43300a4170 2024-04-26T00:34:25.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f4337d4f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:25.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f4337d4f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f433009b560 con 0x7f43300a3d90 2024-04-26T00:34:25.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f4337d4f640 10 monclient(hunting): _renew_subs 2024-04-26T00:34:25.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f4337d4f640 10 monclient(hunting): authenticate will time out at 1846.479173s 2024-04-26T00:34:25.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f4335aeb640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f43300a3d90 0x7f433009bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:25.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f4335aeb640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f43300a3d90 0x7f433009bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:58076/0 (socket says 172.21.15.37:58076) 2024-04-26T00:34:25.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f4335aeb640 1 -- 172.21.15.37:0/2711293325 learned_addr learned my addr 172.21.15.37:0/2711293325 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:34:25.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f4335aeb640 10 monclient(hunting): get_auth_request con 0x7f43300a3d90 auth_method 0 2024-04-26T00:34:25.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f4335aeb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:25.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f4335aeb640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:25.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f4335aeb640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:25.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f4335aeb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:25.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f4335aeb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:25.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f43352ea640 1 --2- 172.21.15.37:0/2711293325 >> v2:172.21.15.45:3300/0 conn(0x7f43300a4170 0x7f43300a45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:25.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f4335aeb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:25.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f43352ea640 10 monclient(hunting): get_auth_request con 0x7f43300a4170 auth_method 0 2024-04-26T00:34:25.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f43352ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:25.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f43352ea640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:25.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f43352ea640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:25.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f43352ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:25.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f43352ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:25.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f43352ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:25.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f4335aeb640 10 monclient(hunting): handle_auth_done global_id 154301 payload 274 2024-04-26T00:34:25.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f4335aeb640 1 -- 172.21.15.37:0/2711293325 >> v2:172.21.15.45:3300/0 conn(0x7f43300a4170 msgr2=0x7f43300a45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:34:25.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f4335aeb640 1 --2- 172.21.15.37:0/2711293325 >> v2:172.21.15.45:3300/0 conn(0x7f43300a4170 0x7f43300a45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:25.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f4335aeb640 1 -- 172.21.15.37:0/2711293325 >> v1:172.21.15.45:6789/0 conn(0x7f4330133b20 legacy=0x7f4330133f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:34:25.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f4335aeb640 10 monclient: _finish_hunting 0 2024-04-26T00:34:25.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f4335aeb640 1 monclient: found mon.noname-b 2024-04-26T00:34:25.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f4335aeb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:34:25.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f4335aeb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:34:25.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.170+0000 7f4335aeb640 1 -- 172.21.15.37:0/2711293325 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43300a4af0 con 0x7f43300a3d90 2024-04-26T00:34:25.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.174+0000 7f4335aeb640 1 -- 172.21.15.37:0/2711293325 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f43300a3d90 msgr2=0x7f433009bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:25.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.174+0000 7f4335aeb640 1 --2- 172.21.15.37:0/2711293325 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f43300a3d90 0x7f433009bf40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f432c009e50 tx=0x7f432c01c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4ca9477cc8bcddc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:25.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4334ae9640 1 -- 172.21.15.37:0/2711293325 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f432c0a5640 con 0x7f43300a3d90 2024-04-26T00:34:25.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4334ae9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:25.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4334ae9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:34:25.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4334ae9640 10 monclient: dump: 2024-04-26T00:34:25.174 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:25.174 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:25.174 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.174 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.174 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:25.174 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:25.174 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:25.174 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:25.174 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:25.174 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:25.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4334ae9640 10 monclient: _finish_auth 0 2024-04-26T00:34:25.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4334ae9640 10 monclient: _check_auth_tickets 2024-04-26T00:34:25.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4334ae9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:34:25.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4334ae9640 1 -- 172.21.15.37:0/2711293325 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f432c0a57e0 con 0x7f43300a3d90 2024-04-26T00:34:25.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4337d4f640 5 monclient: authenticate success, global_id 154301 2024-04-26T00:34:25.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4337d4f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:34:25.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4334ae9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:34:25.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4337d4f640 10 monclient: get_monmap_and_config success 2024-04-26T00:34:25.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4334ae9640 1 -- 172.21.15.37:0/2711293325 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f432c0a5bb0 con 0x7f43300a3d90 2024-04-26T00:34:25.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4334ae9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:25.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4337d4f640 4 set_mon_vals no callback set 2024-04-26T00:34:25.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4334ae9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:34:25.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4334ae9640 10 monclient: dump: 2024-04-26T00:34:25.175 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:25.175 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:25.176 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.176 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.176 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:25.176 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:25.176 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:25.176 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:25.176 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:25.176 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:25.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4337d4f640 10 monclient: shutdown 2024-04-26T00:34:25.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4337d4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:34:25.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4337d4f640 1 -- 172.21.15.37:0/2711293325 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f43300a3d90 msgr2=0x7f433009bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:25.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4337d4f640 1 --2- 172.21.15.37:0/2711293325 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f43300a3d90 0x7f433009bf40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f432c009e50 tx=0x7f432c01c8b0 comp rx=0 tx=0).stop 2024-04-26T00:34:25.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4337d4f640 1 -- 172.21.15.37:0/2711293325 shutdown_connections 2024-04-26T00:34:25.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4337d4f640 1 --2- 172.21.15.37:0/2711293325 >> v2:172.21.15.45:3300/0 conn(0x7f43300a4170 0x7f43300a45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:25.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4337d4f640 1 --2- 172.21.15.37:0/2711293325 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f43300a3d90 0x7f433009bf40 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:25.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4337d4f640 1 -- 172.21.15.37:0/2711293325 >> 172.21.15.37:0/2711293325 conn(0x7f43300aa0a0 msgr2=0x7f43300a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:34:25.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4337d4f640 1 -- 172.21.15.37:0/2711293325 shutdown_connections 2024-04-26T00:34:25.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4337d4f640 1 -- 172.21.15.37:0/2711293325 wait complete. 2024-04-26T00:34:25.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4337d4f640 10 monclient: build_initial_monmap 2024-04-26T00:34:25.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4337d4f640 10 monclient: monmap: 2024-04-26T00:34:25.177 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:34:25.177 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:34:25.177 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:34:25.177 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:34:25.177 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:34:25.177 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:34:25.177 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:34:25.177 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:34:25.177 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:34:25.177 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:25.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4337d4f640 1 Processor -- start 2024-04-26T00:34:25.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4337d4f640 1 -- start start 2024-04-26T00:34:25.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4337d4f640 10 monclient: init 2024-04-26T00:34:25.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4337d4f640 10 monclient: _reopen_session rank -1 2024-04-26T00:34:25.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4337d4f640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:34:25.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4337d4f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f43300a3d90 0x7f43301cf1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:25.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4337d4f640 10 monclient(hunting): picked mon.noname-a con 0x7f43300a3d90 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:34:25.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4337d4f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f43300a4170 0x7f433013e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:25.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4337d4f640 10 monclient(hunting): picked mon.noname-c con 0x7f43300a4170 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:34:25.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4337d4f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4330133b20 0x7f433013e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:25.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4337d4f640 10 monclient(hunting): picked mon.noname-b con 0x7f4330133b20 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:34:25.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4337d4f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:25.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4337d4f640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f4330141e30 con 0x7f43300a3d90 2024-04-26T00:34:25.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4337d4f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:25.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4337d4f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f433009c690 con 0x7f4330133b20 2024-04-26T00:34:25.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4337d4f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:25.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4337d4f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4330141fb0 con 0x7f43300a4170 2024-04-26T00:34:25.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4337d4f640 10 monclient(hunting): _renew_subs 2024-04-26T00:34:25.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4337d4f640 10 monclient(hunting): authenticate will time out at 1846.486382s 2024-04-26T00:34:25.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f43352ea640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f43300a4170 0x7f433013e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:25.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4335aeb640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f43300a3d90 0x7f43301cf1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:25.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f43352ea640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f43300a4170 0x7f433013e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:58088/0 (socket says 172.21.15.37:58088) 2024-04-26T00:34:25.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f43352ea640 1 -- 172.21.15.37:0/2437505556 learned_addr learned my addr 172.21.15.37:0/2437505556 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:34:25.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f43352ea640 10 monclient(hunting): get_auth_request con 0x7f43300a4170 auth_method 0 2024-04-26T00:34:25.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f43352ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:25.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f43352ea640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:25.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f43352ea640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:25.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4335aeb640 10 monclient(hunting): get_auth_request con 0x7f43300a3d90 auth_method 0 2024-04-26T00:34:25.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4335aeb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:25.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4335aeb640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:25.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4335aeb640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:25.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f43352ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:25.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f43352ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:25.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f43352ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:25.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4335aeb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:25.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4335aeb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:25.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4335aeb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:25.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f43362ec640 1 --2- 172.21.15.37:0/2437505556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4330133b20 0x7f433013e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:25.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f43352ea640 10 monclient(hunting): handle_auth_done global_id 154304 payload 824 2024-04-26T00:34:25.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f43352ea640 1 -- 172.21.15.37:0/2437505556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4330133b20 msgr2=0x7f433013e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:25.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f43352ea640 1 --2- 172.21.15.37:0/2437505556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4330133b20 0x7f433013e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:25.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f43352ea640 1 -- 172.21.15.37:0/2437505556 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f43300a3d90 msgr2=0x7f43301cf1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:25.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f43352ea640 1 --2- 172.21.15.37:0/2437505556 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f43300a3d90 0x7f43301cf1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:25.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f43352ea640 10 monclient: _finish_hunting 0 2024-04-26T00:34:25.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f43352ea640 1 monclient: found mon.noname-c 2024-04-26T00:34:25.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f43352ea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:34:25.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f43352ea640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:34:25.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f43352ea640 1 -- 172.21.15.37:0/2437505556 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f433013ccb0 con 0x7f43300a4170 2024-04-26T00:34:25.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f43362ec640 10 monclient: get_auth_request con 0x7f4330133b20 auth_method 0 2024-04-26T00:34:25.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f4335aeb640 1 --2- 172.21.15.37:0/2437505556 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f43300a3d90 0x7f43301cf1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:34:25.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.178+0000 7f43362ec640 1 --2- 172.21.15.37:0/2437505556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4330133b20 0x7f433013e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:34:25.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f43352ea640 1 -- 172.21.15.37:0/2437505556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f43300a4170 msgr2=0x7f433013e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:25.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f43352ea640 1 --2- 172.21.15.37:0/2437505556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f43300a4170 0x7f433013e410 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f432000e960 tx=0x7f432000ee20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:25.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f43267fc640 1 -- 172.21.15.37:0/2437505556 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4320017070 con 0x7f43300a4170 2024-04-26T00:34:25.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f43267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:25.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f43267fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:34:25.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f43267fc640 10 monclient: dump: 2024-04-26T00:34:25.186 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:25.186 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:25.186 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.187 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.187 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:25.187 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:25.187 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:25.187 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:25.187 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:25.187 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:25.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f43267fc640 10 monclient: _finish_auth 0 2024-04-26T00:34:25.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f43267fc640 10 monclient: _check_auth_tickets 2024-04-26T00:34:25.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f43267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:34:25.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f43267fc640 1 -- 172.21.15.37:0/2437505556 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f432000cc10 con 0x7f43300a4170 2024-04-26T00:34:25.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f43267fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:34:25.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f43267fc640 1 -- 172.21.15.37:0/2437505556 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4320013470 con 0x7f43300a4170 2024-04-26T00:34:25.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f4337d4f640 5 monclient: authenticate success, global_id 154304 2024-04-26T00:34:25.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f4337d4f640 10 monclient: _renew_subs 2024-04-26T00:34:25.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f4337d4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:34:25.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f4337d4f640 1 -- 172.21.15.37:0/2437505556 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f433009ada0 con 0x7f43300a4170 2024-04-26T00:34:25.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f43267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:25.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f43267fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:34:25.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f4327fff640 4 set_mon_vals no callback set 2024-04-26T00:34:25.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f43267fc640 10 monclient: dump: 2024-04-26T00:34:25.188 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:25.188 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:25.188 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.188 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.188 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:25.188 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:25.188 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:25.188 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:25.188 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:25.189 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:25.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f4337d4f640 10 monclient: _renew_subs 2024-04-26T00:34:25.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f4337d4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:34:25.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f4337d4f640 1 -- 172.21.15.37:0/2437505556 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f43301cf8e0 con 0x7f43300a4170 2024-04-26T00:34:25.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f4337d4f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:34:25.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f4337d4f640 1 --2- 172.21.15.37:0/2437505556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f433013c870 0x7f4330138d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:25.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f4337d4f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:25.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f4337d4f640 1 -- 172.21.15.37:0/2437505556 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f433004f510 con 0x7f433013c870 2024-04-26T00:34:25.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f4335aeb640 1 --2- 172.21.15.37:0/2437505556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f433013c870 0x7f4330138d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:25.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f4335aeb640 10 monclient: get_auth_request con 0x7f433013c870 auth_method 0 2024-04-26T00:34:25.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f4335aeb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:25.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f4335aeb640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:25.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f4335aeb640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:25.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f4335aeb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:25.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f4335aeb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:25.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f43267fc640 1 -- 172.21.15.37:0/2437505556 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f432001a070 con 0x7f43300a4170 2024-04-26T00:34:25.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f4335aeb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:25.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f4335aeb640 10 monclient(hunting): handle_auth_done global_id 154307 payload 824 2024-04-26T00:34:25.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f43267fc640 1 --2- 172.21.15.37:0/2437505556 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4314035ed0 0x7f4314038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:25.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f4335aeb640 1 -- 172.21.15.37:0/2437505556 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f433013a9c0 con 0x7f433013c870 2024-04-26T00:34:25.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f43267fc640 1 -- 172.21.15.37:0/2437505556 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(565..565 src has 370..565) ==== 3959+0+0 (secure 0 0 0) 0x7f4320048b90 con 0x7f43300a4170 2024-04-26T00:34:25.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f43362ec640 1 --2- 172.21.15.37:0/2437505556 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4314035ed0 0x7f4314038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:25.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.182+0000 7f43362ec640 10 monclient: get_auth_request con 0x7f4314035ed0 auth_method 0 2024-04-26T00:34:25.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.186+0000 7f4335aeb640 1 -- 172.21.15.37:0/2437505556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f433013c870 msgr2=0x7f4330138d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:25.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.186+0000 7f4335aeb640 1 --2- 172.21.15.37:0/2437505556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f433013c870 0x7f4330138d20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4330076100 tx=0x7f432c019650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:25.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.186+0000 7f43267fc640 1 -- 172.21.15.37:0/2437505556 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f432c0adb60 con 0x7f433013c870 2024-04-26T00:34:25.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.186+0000 7f43267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:25.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.186+0000 7f43267fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:34:25.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.186+0000 7f43267fc640 10 monclient: dump: 2024-04-26T00:34:25.190 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:25.191 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:25.191 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.191 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.191 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:25.191 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:25.191 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:25.191 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:25.191 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:25.191 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:25.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.186+0000 7f43267fc640 1 -- 172.21.15.37:0/2437505556 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f432c0adb60 con 0x7f433013c870 2024-04-26T00:34:25.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.186+0000 7f43267fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:34:25.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.186+0000 7f43267fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:34:25.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.186+0000 7f43267fc640 1 -- 172.21.15.37:0/2437505556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f433013c870 msgr2=0x7f4330138d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:25.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.186+0000 7f43267fc640 1 --2- 172.21.15.37:0/2437505556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f433013c870 0x7f4330138d20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4330076100 tx=0x7f432c019650 comp rx=0 tx=0).stop 2024-04-26T00:34:25.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.186+0000 7f43267fc640 1 -- 172.21.15.37:0/2437505556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f433013c870 msgr2=0x7f4330138d20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:34:25.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.186+0000 7f43267fc640 1 --2- 172.21.15.37:0/2437505556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f433013c870 0x7f4330138d20 secure :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4330076100 tx=0x7f432c019650 comp rx=0 tx=0).stop 2024-04-26T00:34:25.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.186+0000 7f43362ec640 1 --2- 172.21.15.37:0/2437505556 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4314035ed0 0x7f4314038390 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f433013b910 tx=0x7f4328009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:25.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.190+0000 7f4337d4f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:34:25.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.190+0000 7f4337d4f640 1 --2- 172.21.15.37:0/2437505556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4330139260 0x7f4330139670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:25.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.190+0000 7f4337d4f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:25.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.190+0000 7f4337d4f640 1 -- 172.21.15.37:0/2437505556 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4330139bb0 con 0x7f4330139260 2024-04-26T00:34:25.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.190+0000 7f4335aeb640 1 --2- 172.21.15.37:0/2437505556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4330139260 0x7f4330139670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:25.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.190+0000 7f4335aeb640 10 monclient: get_auth_request con 0x7f4330139260 auth_method 0 2024-04-26T00:34:25.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.190+0000 7f4335aeb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:25.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.190+0000 7f4335aeb640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:25.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.190+0000 7f4335aeb640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:25.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.190+0000 7f4335aeb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:25.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.190+0000 7f4335aeb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:25.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.190+0000 7f4335aeb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:25.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.190+0000 7f4335aeb640 10 monclient(hunting): handle_auth_done global_id 154310 payload 824 2024-04-26T00:34:25.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.190+0000 7f4335aeb640 1 -- 172.21.15.37:0/2437505556 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4330075dc0 con 0x7f4330139260 2024-04-26T00:34:25.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.194+0000 7f4335aeb640 1 -- 172.21.15.37:0/2437505556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4330139260 msgr2=0x7f4330139670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:25.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.194+0000 7f4335aeb640 1 --2- 172.21.15.37:0/2437505556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4330139260 0x7f4330139670 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f432c009e50 tx=0x7f432c0a7000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:25.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.194+0000 7f43267fc640 1 -- 172.21.15.37:0/2437505556 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f432c0c5040 con 0x7f4330139260 2024-04-26T00:34:25.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.194+0000 7f43267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:25.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.194+0000 7f43267fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:34:25.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.194+0000 7f43267fc640 10 monclient: dump: 2024-04-26T00:34:25.195 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:25.195 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:25.195 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.195 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.195 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:25.195 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:25.196 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:25.196 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:25.196 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:25.196 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:25.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.194+0000 7f43267fc640 1 -- 172.21.15.37:0/2437505556 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4718 (secure 0 0 0) 0x7f432c003db0 con 0x7f4330139260 2024-04-26T00:34:25.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.194+0000 7f43267fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:34:25.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.194+0000 7f43267fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:34:25.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.194+0000 7f43267fc640 1 -- 172.21.15.37:0/2437505556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4330139260 msgr2=0x7f4330139670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:25.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.194+0000 7f43267fc640 1 --2- 172.21.15.37:0/2437505556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4330139260 0x7f4330139670 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f432c009e50 tx=0x7f432c0a7000 comp rx=0 tx=0).stop 2024-04-26T00:34:25.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.194+0000 7f43267fc640 1 -- 172.21.15.37:0/2437505556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4330139260 msgr2=0x7f4330139670 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:34:25.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.194+0000 7f43267fc640 1 --2- 172.21.15.37:0/2437505556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4330139260 0x7f4330139670 secure :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f432c009e50 tx=0x7f432c0a7000 comp rx=0 tx=0).stop 2024-04-26T00:34:25.197 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:34:25.197 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:34:25.197 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:34:25.197 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:34:25.197 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 108, 2024-04-26T00:34:25.197 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 102714, 2024-04-26T00:34:25.197 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:34:25.197 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:34:25.197 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:34:25.197 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:34:25.197 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:25.197 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 4, 2024-04-26T00:34:25.198 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:34:25.198 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:34:25.198 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:34:25.198 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:34:25.198 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:34:25.198 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:34:25.198 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:34:25.198 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:34:25.198 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:34:25.198 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:34:25.198 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:34:25.198 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:34:25.198 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:34:25.198 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:34:25.198 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:25.199 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:34:25.199 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:34:25.199 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:34:25.199 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:34:25.199 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:34:25.199 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:34:25.199 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:34:25.199 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:34:25.199 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:34:25.199 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:34:25.199 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:34:25.199 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:34:25.199 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:34:25.199 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:25.199 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:25.200 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:34:25.200 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:34:25.200 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:34:25.200 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:34:25.200 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:34:25.200 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:34:25.200 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:34:25.200 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:34:25.200 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:34:25.200 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:34:25.200 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:34:25.200 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:34:25.200 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:34:25.200 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:34:25.200 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:34:25.201 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:34:25.201 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:34:25.201 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:34:25.201 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:34:25.201 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:34:25.201 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:34:25.201 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:34:25.201 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:34:25.201 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:34:25.201 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:34:25.201 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:34:25.201 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:34:25.201 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:34:25.201 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:25.201 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:34:25.202 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:25.202 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:34:25.202 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:25.202 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:34:25.202 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:34:25.202 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:34:25.202 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:34:25.202 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:25.202 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:34:25.202 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:34:25.202 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:25.202 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:25.202 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:25.202 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:34:25.203 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:34:25.203 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:25.203 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:25.203 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:25.203 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:25.203 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:34:25.203 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:34:25.203 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:34:25.203 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:34:25.203 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:34:25.203 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:25.203 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:25.203 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:34:25.203 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:34:25.203 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:34:25.204 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:34:25.204 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:25.204 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:34:25.204 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:34:25.204 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:25.204 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:25.204 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:25.204 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:34:25.204 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:34:25.205 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:25.205 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:25.205 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:25.205 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:25.205 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:34:25.205 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:34:25.205 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:34:25.205 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:34:25.205 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:34:25.205 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:25.205 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:25.205 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:34:25.205 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:34:25.205 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:34:25.205 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:34:25.206 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:25.206 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:34:25.206 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:34:25.206 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:25.206 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:25.206 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:25.206 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:34:25.206 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:34:25.206 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:25.206 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:25.206 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:25.206 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:25.206 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:34:25.206 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:34:25.207 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:34:25.207 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:34:25.207 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:34:25.207 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:25.207 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:25.207 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:25.207 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:34:25.207 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:34:25.207 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:25.207 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:34:25.207 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:34:25.207 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:34:25.207 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:25.207 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:25.207 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:34:25.208 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:25.208 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:34:25.208 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:34:25.208 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:34:25.208 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:25.208 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:25.208 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:34:25.208 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:25.208 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:34:25.208 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:34:25.208 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:34:25.208 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:25.208 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:25.208 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:25.208 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:34:25.209 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:34:25.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.194+0000 7f4337d4f640 1 -- 172.21.15.37:0/2437505556 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4314035ed0 msgr2=0x7f4314038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:25.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.194+0000 7f4337d4f640 1 --2- 172.21.15.37:0/2437505556 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4314035ed0 0x7f4314038390 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f433013b910 tx=0x7f4328009040 comp rx=0 tx=0).stop 2024-04-26T00:34:25.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.194+0000 7f4337d4f640 10 monclient: shutdown 2024-04-26T00:34:25.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.194+0000 7f4337d4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:34:25.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.194+0000 7f4337d4f640 1 -- 172.21.15.37:0/2437505556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f43300a4170 msgr2=0x7f433013e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:25.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.194+0000 7f4337d4f640 1 --2- 172.21.15.37:0/2437505556 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f43300a4170 0x7f433013e410 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f432000e960 tx=0x7f432000ee20 comp rx=0 tx=0).stop 2024-04-26T00:34:25.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.194+0000 7f43362ec640 1 -- 172.21.15.37:0/2437505556 reap_dead start 2024-04-26T00:34:25.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.194+0000 7f43362ec640 1 -- 172.21.15.37:0/2437505556 reap_dead start 2024-04-26T00:34:25.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.194+0000 7f4337d4f640 1 -- 172.21.15.37:0/2437505556 shutdown_connections 2024-04-26T00:34:25.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.194+0000 7f4337d4f640 1 -- 172.21.15.37:0/2437505556 >> 172.21.15.37:0/2437505556 conn(0x7f43300aa0a0 msgr2=0x7f43300a2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:34:25.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.194+0000 7f4337d4f640 1 -- 172.21.15.37:0/2437505556 shutdown_connections 2024-04-26T00:34:25.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.194+0000 7f4337d4f640 1 -- 172.21.15.37:0/2437505556 wait complete. 2024-04-26T00:34:25.211 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:34:25.211 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:34:25.211 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:34:25.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e2964d640 10 monclient: get_monmap_and_config 2024-04-26T00:34:25.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e2964d640 10 monclient: build_initial_monmap 2024-04-26T00:34:25.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e2964d640 10 monclient: monmap: 2024-04-26T00:34:25.326 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:34:25.326 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:25.326 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:34:25.333371+0000 2024-04-26T00:34:25.326 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:34:25.333371+0000 2024-04-26T00:34:25.326 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:34:25.326 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:25.326 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:34:25.326 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:34:25.326 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:34:25.326 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:34:25.327 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:34:25.327 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:25.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e2964d640 1 Processor -- start 2024-04-26T00:34:25.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e2964d640 1 -- start start 2024-04-26T00:34:25.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e2964d640 10 monclient: init 2024-04-26T00:34:25.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e2964d640 10 monclient: _reopen_session rank -1 2024-04-26T00:34:25.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e2964d640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-04-26T00:34:25.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e2964d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e24136f20 0x7f6e2409bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:25.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e2964d640 10 monclient(hunting): picked mon.noname-b con 0x7f6e24136f20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:34:25.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e2964d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6e24137300 0x7f6e24137780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:25.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e2964d640 10 monclient(hunting): picked mon.noname-a con 0x7f6e24137300 addr v2:172.21.15.37:3300/0 2024-04-26T00:34:25.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e2964d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6e24133c70 0x7f6e241340e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:25.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e2964d640 10 monclient(hunting): picked mon.noname-c con 0x7f6e24133c70 addr v2:172.21.15.45:3300/0 2024-04-26T00:34:25.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e2964d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:25.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e2964d640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f6e240a50f0 con 0x7f6e24137300 2024-04-26T00:34:25.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e2964d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:25.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e2964d640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f6e2409b330 con 0x7f6e24133c70 2024-04-26T00:34:25.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e2964d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:25.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e2964d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f6e2409b610 con 0x7f6e24136f20 2024-04-26T00:34:25.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e2964d640 10 monclient(hunting): _renew_subs 2024-04-26T00:34:25.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e2964d640 10 monclient(hunting): authenticate will time out at 1846.638974s 2024-04-26T00:34:25.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e227fc640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6e24137300 0x7f6e24137780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:25.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e22ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e24136f20 0x7f6e2409bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:25.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e227fc640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6e24137300 0x7f6e24137780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33840/0 (socket says 172.21.15.37:33840) 2024-04-26T00:34:25.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e227fc640 1 -- 172.21.15.37:0/4094589232 learned_addr learned my addr 172.21.15.37:0/4094589232 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:34:25.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e227fc640 10 monclient(hunting): get_auth_request con 0x7f6e24137300 auth_method 0 2024-04-26T00:34:25.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e227fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:25.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e227fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:25.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e227fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:25.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e22ffd640 10 monclient(hunting): get_auth_request con 0x7f6e24136f20 auth_method 0 2024-04-26T00:34:25.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e22ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:25.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e22ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:25.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:25.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e227fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:25.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e227fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:25.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e227fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:25.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e237fe640 1 --2- 172.21.15.37:0/4094589232 >> v2:172.21.15.45:3300/0 conn(0x7f6e24133c70 0x7f6e241340e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:25.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e22ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:25.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e22ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:25.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e22ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:25.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e237fe640 10 monclient(hunting): get_auth_request con 0x7f6e24133c70 auth_method 0 2024-04-26T00:34:25.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:25.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e237fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:25.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e237fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:25.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e227fc640 10 monclient(hunting): handle_auth_done global_id 144438 payload 274 2024-04-26T00:34:25.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e227fc640 1 -- 172.21.15.37:0/4094589232 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e24136f20 msgr2=0x7f6e2409bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:34:25.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e227fc640 1 --2- 172.21.15.37:0/4094589232 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e24136f20 0x7f6e2409bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:25.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e227fc640 1 -- 172.21.15.37:0/4094589232 >> v2:172.21.15.45:3300/0 conn(0x7f6e24133c70 msgr2=0x7f6e241340e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:34:25.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e227fc640 1 --2- 172.21.15.37:0/4094589232 >> v2:172.21.15.45:3300/0 conn(0x7f6e24133c70 0x7f6e241340e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:25.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e227fc640 10 monclient: _finish_hunting 0 2024-04-26T00:34:25.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e227fc640 1 monclient: found mon.noname-a 2024-04-26T00:34:25.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:34:25.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e227fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:34:25.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.330+0000 7f6e227fc640 1 -- 172.21.15.37:0/4094589232 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e2409b8f0 con 0x7f6e24137300 2024-04-26T00:34:25.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.334+0000 7f6e227fc640 1 -- 172.21.15.37:0/4094589232 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6e24137300 msgr2=0x7f6e24137780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:25.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.334+0000 7f6e227fc640 1 --2- 172.21.15.37:0/4094589232 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6e24137300 0x7f6e24137780 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6e2409bbd0 tx=0x7f6e1801c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d60ba39fd42e925 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:25.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.334+0000 7f6e21ffb640 1 -- 172.21.15.37:0/4094589232 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e180b0070 con 0x7f6e24137300 2024-04-26T00:34:25.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.334+0000 7f6e21ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:25.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.334+0000 7f6e21ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:34:25.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.334+0000 7f6e21ffb640 10 monclient: dump: 2024-04-26T00:34:25.338 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:25.338 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:25.338 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.339 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.339 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:25.339 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:25.339 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:25.339 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:25.339 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:25.339 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:25.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.334+0000 7f6e21ffb640 10 monclient: _finish_auth 0 2024-04-26T00:34:25.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.334+0000 7f6e21ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:34:25.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.334+0000 7f6e21ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:34:25.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.334+0000 7f6e2964d640 5 monclient: authenticate success, global_id 144438 2024-04-26T00:34:25.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.334+0000 7f6e21ffb640 1 -- 172.21.15.37:0/4094589232 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e180ab600 con 0x7f6e24137300 2024-04-26T00:34:25.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.334+0000 7f6e2964d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:34:25.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e21ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:34:25.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e21ffb640 1 -- 172.21.15.37:0/4094589232 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e180ab9d0 con 0x7f6e24137300 2024-04-26T00:34:25.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e21ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:25.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e21ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:34:25.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e21ffb640 10 monclient: dump: 2024-04-26T00:34:25.340 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:25.340 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:25.340 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.340 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.340 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:25.340 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:25.340 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:25.340 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:25.340 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:25.340 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:25.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e2964d640 10 monclient: get_monmap_and_config success 2024-04-26T00:34:25.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e2964d640 4 set_mon_vals no callback set 2024-04-26T00:34:25.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e2964d640 10 monclient: shutdown 2024-04-26T00:34:25.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e2964d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:34:25.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e2964d640 1 -- 172.21.15.37:0/4094589232 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6e24137300 msgr2=0x7f6e24137780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:25.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e2964d640 1 --2- 172.21.15.37:0/4094589232 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6e24137300 0x7f6e24137780 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6e2409bbd0 tx=0x7f6e1801c4e0 comp rx=0 tx=0).stop 2024-04-26T00:34:25.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e2964d640 1 -- 172.21.15.37:0/4094589232 shutdown_connections 2024-04-26T00:34:25.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e2964d640 1 --2- 172.21.15.37:0/4094589232 >> v2:172.21.15.45:3300/0 conn(0x7f6e24133c70 0x7f6e241340e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:25.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e2964d640 1 --2- 172.21.15.37:0/4094589232 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6e24137300 0x7f6e24137780 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:25.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e2964d640 1 --2- 172.21.15.37:0/4094589232 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e24136f20 0x7f6e2409bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:25.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e2964d640 1 -- 172.21.15.37:0/4094589232 >> 172.21.15.37:0/4094589232 conn(0x7f6e240a61b0 msgr2=0x7f6e240a65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:34:25.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e2964d640 1 -- 172.21.15.37:0/4094589232 shutdown_connections 2024-04-26T00:34:25.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e2964d640 1 -- 172.21.15.37:0/4094589232 wait complete. 2024-04-26T00:34:25.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e2964d640 10 monclient: build_initial_monmap 2024-04-26T00:34:25.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e2964d640 10 monclient: monmap: 2024-04-26T00:34:25.341 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:34:25.341 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:34:25.342 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:34:25.342 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:34:25.342 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:34:25.342 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:34:25.342 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:34:25.342 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:34:25.342 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:34:25.342 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:25.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e2964d640 1 Processor -- start 2024-04-26T00:34:25.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e2964d640 1 -- start start 2024-04-26T00:34:25.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e2964d640 10 monclient: init 2024-04-26T00:34:25.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e2964d640 10 monclient: _reopen_session rank -1 2024-04-26T00:34:25.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e2964d640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:34:25.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e2964d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6e24133c70 0x7f6e241cef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:25.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e2964d640 10 monclient(hunting): picked mon.noname-a con 0x7f6e24133c70 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:34:25.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e2964d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e24136f20 0x7f6e241cf480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:25.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e2964d640 10 monclient(hunting): picked mon.noname-b con 0x7f6e24136f20 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:34:25.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e2964d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e24137300 0x7f6e241cf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:25.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e2964d640 10 monclient(hunting): picked mon.noname-c con 0x7f6e24137300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:34:25.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e2964d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:25.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e2964d640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f6e2409ac60 con 0x7f6e24133c70 2024-04-26T00:34:25.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e2964d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:25.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e2964d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6e2409aae0 con 0x7f6e24136f20 2024-04-26T00:34:25.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e2964d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:25.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e2964d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f6e24140c80 con 0x7f6e24137300 2024-04-26T00:34:25.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e2964d640 10 monclient(hunting): _renew_subs 2024-04-26T00:34:25.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e2964d640 10 monclient(hunting): authenticate will time out at 1846.646146s 2024-04-26T00:34:25.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e22ffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6e24133c70 0x7f6e241cef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:25.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e237fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e24137300 0x7f6e241cf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:25.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e22ffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6e24133c70 0x7f6e241cef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33852/0 (socket says 172.21.15.37:33852) 2024-04-26T00:34:25.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e22ffd640 1 -- 172.21.15.37:0/3462680805 learned_addr learned my addr 172.21.15.37:0/3462680805 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:34:25.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e22ffd640 10 monclient(hunting): get_auth_request con 0x7f6e24133c70 auth_method 0 2024-04-26T00:34:25.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e22ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:25.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e22ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:25.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:25.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e237fe640 10 monclient(hunting): get_auth_request con 0x7f6e24137300 auth_method 0 2024-04-26T00:34:25.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:25.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e237fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:25.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e237fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:25.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:25.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:25.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:25.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e22ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:25.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e22ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:25.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e22ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:25.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e227fc640 1 --2- 172.21.15.37:0/3462680805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e24136f20 0x7f6e241cf480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:25.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e227fc640 10 monclient(hunting): get_auth_request con 0x7f6e24136f20 auth_method 0 2024-04-26T00:34:25.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e227fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:25.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e227fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:25.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e227fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:25.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e237fe640 10 monclient(hunting): handle_auth_done global_id 154316 payload 824 2024-04-26T00:34:25.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e237fe640 1 -- 172.21.15.37:0/3462680805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e24136f20 msgr2=0x7f6e241cf480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:25.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e237fe640 1 --2- 172.21.15.37:0/3462680805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e24136f20 0x7f6e241cf480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:25.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e237fe640 1 -- 172.21.15.37:0/3462680805 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6e24133c70 msgr2=0x7f6e241cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:25.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e237fe640 1 --2- 172.21.15.37:0/3462680805 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6e24133c70 0x7f6e241cef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:25.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e237fe640 10 monclient: _finish_hunting 0 2024-04-26T00:34:25.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e237fe640 1 monclient: found mon.noname-c 2024-04-26T00:34:25.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:34:25.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e237fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:34:25.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e237fe640 1 -- 172.21.15.37:0/3462680805 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e2409ae30 con 0x7f6e24137300 2024-04-26T00:34:25.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.338+0000 7f6e22ffd640 1 --2- 172.21.15.37:0/3462680805 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6e24133c70 0x7f6e241cef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:34:25.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e237fe640 1 -- 172.21.15.37:0/3462680805 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e24137300 msgr2=0x7f6e241cf9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:25.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e237fe640 1 --2- 172.21.15.37:0/3462680805 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e24137300 0x7f6e241cf9c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c007d90 tx=0x7f6e1c00e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:25.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e0b7fe640 1 -- 172.21.15.37:0/3462680805 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c0039a0 con 0x7f6e24137300 2024-04-26T00:34:25.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e0b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:25.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e0b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:34:25.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e0b7fe640 10 monclient: dump: 2024-04-26T00:34:25.346 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:25.346 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:25.346 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.346 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.346 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:25.346 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:25.347 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:25.347 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:25.347 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:25.347 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:25.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e0b7fe640 10 monclient: _finish_auth 0 2024-04-26T00:34:25.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e0b7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:34:25.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e0b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:34:25.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e0b7fe640 1 -- 172.21.15.37:0/3462680805 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e1c003b40 con 0x7f6e24137300 2024-04-26T00:34:25.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e0b7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:34:25.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e0b7fe640 1 -- 172.21.15.37:0/3462680805 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c004020 con 0x7f6e24137300 2024-04-26T00:34:25.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e0b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:25.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e0b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:34:25.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e0b7fe640 10 monclient: dump: 2024-04-26T00:34:25.348 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:25.348 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:25.348 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.348 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.348 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:25.348 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:25.348 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:25.348 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:25.348 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:25.348 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:25.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e20ff9640 4 set_mon_vals no callback set 2024-04-26T00:34:25.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e2964d640 5 monclient: authenticate success, global_id 154316 2024-04-26T00:34:25.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e2964d640 10 monclient: _renew_subs 2024-04-26T00:34:25.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e2964d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:34:25.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e2964d640 1 -- 172.21.15.37:0/3462680805 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e24135ea0 con 0x7f6e24137300 2024-04-26T00:34:25.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e2964d640 10 monclient: _renew_subs 2024-04-26T00:34:25.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e2964d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:34:25.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e2964d640 1 -- 172.21.15.37:0/3462680805 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e2413c610 con 0x7f6e24137300 2024-04-26T00:34:25.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e2964d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:34:25.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e2964d640 1 --2- 172.21.15.37:0/3462680805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e24132c60 0x7f6e24133040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:25.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e2964d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:25.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e2964d640 1 -- 172.21.15.37:0/3462680805 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6e2404f510 con 0x7f6e24132c60 2024-04-26T00:34:25.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e0b7fe640 1 -- 172.21.15.37:0/3462680805 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f6e1c01b020 con 0x7f6e24137300 2024-04-26T00:34:25.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e0b7fe640 1 --2- 172.21.15.37:0/3462680805 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6e10035e80 0x7f6e10038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:25.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e0b7fe640 1 -- 172.21.15.37:0/3462680805 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(565..565 src has 370..565) ==== 3959+0+0 (secure 0 0 0) 0x7f6e1c048d40 con 0x7f6e24137300 2024-04-26T00:34:25.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e22ffd640 1 --2- 172.21.15.37:0/3462680805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e24132c60 0x7f6e24133040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:25.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e22ffd640 10 monclient: get_auth_request con 0x7f6e24132c60 auth_method 0 2024-04-26T00:34:25.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e22ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:25.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e22ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:25.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:25.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e227fc640 1 --2- 172.21.15.37:0/3462680805 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6e10035e80 0x7f6e10038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:25.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e227fc640 10 monclient: get_auth_request con 0x7f6e10035e80 auth_method 0 2024-04-26T00:34:25.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e22ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:25.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e22ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:25.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e22ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:25.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e22ffd640 10 monclient(hunting): handle_auth_done global_id 174124 payload 824 2024-04-26T00:34:25.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.342+0000 7f6e22ffd640 1 -- 172.21.15.37:0/3462680805 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6e2413dba0 con 0x7f6e24132c60 2024-04-26T00:34:25.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.346+0000 7f6e22ffd640 1 -- 172.21.15.37:0/3462680805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e24132c60 msgr2=0x7f6e24133040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:25.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.346+0000 7f6e22ffd640 1 --2- 172.21.15.37:0/3462680805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e24132c60 0x7f6e24133040 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6e14011d30 tx=0x7f6e1400f610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:25.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.346+0000 7f6e227fc640 1 --2- 172.21.15.37:0/3462680805 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6e10035e80 0x7f6e10038340 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f6e180ab000 tx=0x7f6e180a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:25.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.346+0000 7f6e0b7fe640 1 -- 172.21.15.37:0/3462680805 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e1400fe70 con 0x7f6e24132c60 2024-04-26T00:34:25.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.346+0000 7f6e0b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:25.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.346+0000 7f6e0b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:34:25.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.346+0000 7f6e0b7fe640 10 monclient: dump: 2024-04-26T00:34:25.354 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:25.354 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:25.354 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.354 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.354 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:25.354 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:25.355 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:25.355 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:25.355 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:25.355 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:25.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.346+0000 7f6e0b7fe640 1 -- 172.21.15.37:0/3462680805 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6e1400fe70 con 0x7f6e24132c60 2024-04-26T00:34:25.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.346+0000 7f6e0b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:34:25.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.346+0000 7f6e0b7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:34:25.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.346+0000 7f6e0b7fe640 1 -- 172.21.15.37:0/3462680805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e24132c60 msgr2=0x7f6e24133040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:25.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.346+0000 7f6e0b7fe640 1 --2- 172.21.15.37:0/3462680805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e24132c60 0x7f6e24133040 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6e14011d30 tx=0x7f6e1400f610 comp rx=0 tx=0).stop 2024-04-26T00:34:25.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.346+0000 7f6e0b7fe640 1 -- 172.21.15.37:0/3462680805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e24132c60 msgr2=0x7f6e24133040 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:34:25.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.346+0000 7f6e0b7fe640 1 --2- 172.21.15.37:0/3462680805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e24132c60 0x7f6e24133040 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6e14011d30 tx=0x7f6e1400f610 comp rx=0 tx=0).stop 2024-04-26T00:34:25.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.354+0000 7f6e2964d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:34:25.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.354+0000 7f6e2964d640 1 --2- 172.21.15.37:0/3462680805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e24133580 0x7f6e240a08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:25.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.354+0000 7f6e2964d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:25.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.354+0000 7f6e2964d640 1 -- 172.21.15.37:0/3462680805 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6e24075f10 con 0x7f6e24133580 2024-04-26T00:34:25.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.354+0000 7f6e22ffd640 1 --2- 172.21.15.37:0/3462680805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e24133580 0x7f6e240a08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:25.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.354+0000 7f6e22ffd640 10 monclient: get_auth_request con 0x7f6e24133580 auth_method 0 2024-04-26T00:34:25.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.354+0000 7f6e22ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:25.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.354+0000 7f6e22ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:25.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.354+0000 7f6e22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:25.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.354+0000 7f6e22ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:25.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.354+0000 7f6e22ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:25.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.354+0000 7f6e22ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:25.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.354+0000 7f6e22ffd640 10 monclient(hunting): handle_auth_done global_id 174127 payload 824 2024-04-26T00:34:25.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.354+0000 7f6e22ffd640 1 -- 172.21.15.37:0/3462680805 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6e24133990 con 0x7f6e24133580 2024-04-26T00:34:25.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.358+0000 7f6e22ffd640 1 -- 172.21.15.37:0/3462680805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e24133580 msgr2=0x7f6e240a08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:25.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.358+0000 7f6e22ffd640 1 --2- 172.21.15.37:0/3462680805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e24133580 0x7f6e240a08e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6e14011d30 tx=0x7f6e140046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:25.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.358+0000 7f6e0b7fe640 1 -- 172.21.15.37:0/3462680805 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e14003a30 con 0x7f6e24133580 2024-04-26T00:34:25.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.358+0000 7f6e0b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:25.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.358+0000 7f6e0b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:34:25.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.358+0000 7f6e0b7fe640 10 monclient: dump: 2024-04-26T00:34:25.358 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:25.358 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:25.358 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.358 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.358 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:25.358 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:25.358 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:25.358 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:25.358 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:25.358 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:25.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.358+0000 7f6e0b7fe640 1 -- 172.21.15.37:0/3462680805 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7f6e14003a30 con 0x7f6e24133580 2024-04-26T00:34:25.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.358+0000 7f6e0b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:34:25.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.358+0000 7f6e0b7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:34:25.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.358+0000 7f6e0b7fe640 1 -- 172.21.15.37:0/3462680805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e24133580 msgr2=0x7f6e240a08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:25.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.358+0000 7f6e0b7fe640 1 --2- 172.21.15.37:0/3462680805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e24133580 0x7f6e240a08e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6e14011d30 tx=0x7f6e140046d0 comp rx=0 tx=0).stop 2024-04-26T00:34:25.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.358+0000 7f6e0b7fe640 1 -- 172.21.15.37:0/3462680805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e24133580 msgr2=0x7f6e240a08e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:34:25.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.358+0000 7f6e0b7fe640 1 --2- 172.21.15.37:0/3462680805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e24133580 0x7f6e240a08e0 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6e14011d30 tx=0x7f6e140046d0 comp rx=0 tx=0).stop 2024-04-26T00:34:25.359 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:34:25.359 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:34:25.359 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:34:25.359 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:34:25.360 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 108, 2024-04-26T00:34:25.360 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 7979, 2024-04-26T00:34:25.360 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:34:25.360 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:34:25.360 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:34:25.360 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:34:25.360 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:25.360 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 4, 2024-04-26T00:34:25.360 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:34:25.360 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:34:25.360 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:34:25.360 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:34:25.360 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:34:25.360 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:34:25.360 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:34:25.361 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:34:25.361 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:34:25.361 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:34:25.361 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:34:25.361 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:34:25.361 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:34:25.361 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:34:25.361 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:25.361 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:34:25.361 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:34:25.361 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:34:25.361 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:34:25.361 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:34:25.361 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:34:25.361 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:34:25.361 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:34:25.362 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:34:25.362 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:34:25.362 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:34:25.362 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:34:25.362 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:34:25.362 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:25.362 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:25.362 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:34:25.362 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:34:25.362 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:34:25.362 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:34:25.362 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:34:25.362 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:34:25.362 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:34:25.363 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:34:25.363 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:34:25.363 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:34:25.363 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:34:25.363 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:34:25.363 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:34:25.363 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:34:25.363 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:34:25.363 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:34:25.363 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:34:25.363 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:34:25.363 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:34:25.363 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:34:25.363 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:34:25.363 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:34:25.364 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:34:25.364 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:34:25.364 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:34:25.364 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:34:25.364 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:34:25.364 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:34:25.364 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:25.364 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:34:25.364 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:25.364 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:34:25.364 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:25.364 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:34:25.364 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:34:25.364 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:34:25.364 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:34:25.365 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:25.365 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:34:25.365 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:34:25.365 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:25.365 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:25.365 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:25.365 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:34:25.365 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:34:25.365 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:25.365 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:25.365 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:25.365 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:25.365 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:34:25.365 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:34:25.365 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:34:25.366 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:34:25.366 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:34:25.366 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:25.366 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:25.366 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:34:25.366 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:34:25.366 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:34:25.366 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:34:25.366 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:25.366 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:34:25.366 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:34:25.366 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:25.366 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:25.366 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:25.366 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:34:25.367 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:34:25.367 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:25.367 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:25.367 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:25.367 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:25.367 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:34:25.367 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:34:25.367 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:34:25.367 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:34:25.367 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:34:25.367 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:25.367 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:25.367 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:34:25.367 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:34:25.367 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:34:25.368 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:34:25.368 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:25.368 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:34:25.368 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:34:25.368 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:25.368 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:25.368 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:25.368 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:34:25.368 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:34:25.368 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:25.368 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:25.368 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:25.368 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:25.368 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:34:25.368 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:34:25.369 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:34:25.369 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:34:25.369 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:34:25.369 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:25.369 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:25.369 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:25.369 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:34:25.369 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:34:25.369 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:25.369 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:34:25.369 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:34:25.369 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:34:25.369 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:25.369 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:25.369 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:34:25.370 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:25.370 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:34:25.370 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:34:25.370 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:34:25.370 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:25.370 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:25.370 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:25.370 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:34:25.370 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:34:25.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.358+0000 7f6e2964d640 1 -- 172.21.15.37:0/3462680805 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6e10035e80 msgr2=0x7f6e10038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:25.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.358+0000 7f6e2964d640 1 --2- 172.21.15.37:0/3462680805 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6e10035e80 0x7f6e10038340 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f6e180ab000 tx=0x7f6e180a6000 comp rx=0 tx=0).stop 2024-04-26T00:34:25.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.358+0000 7f6e2964d640 10 monclient: shutdown 2024-04-26T00:34:25.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.358+0000 7f6e2964d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:34:25.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.358+0000 7f6e2964d640 1 -- 172.21.15.37:0/3462680805 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e24137300 msgr2=0x7f6e241cf9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:25.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.358+0000 7f6e237fe640 1 -- 172.21.15.37:0/3462680805 reap_dead start 2024-04-26T00:34:25.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.358+0000 7f6e2964d640 1 --2- 172.21.15.37:0/3462680805 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e24137300 0x7f6e241cf9c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c007d90 tx=0x7f6e1c00e4b0 comp rx=0 tx=0).stop 2024-04-26T00:34:25.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.358+0000 7f6e2964d640 1 -- 172.21.15.37:0/3462680805 shutdown_connections 2024-04-26T00:34:25.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.358+0000 7f6e2964d640 1 --2- 172.21.15.37:0/3462680805 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6e24137300 0x7f6e241cf9c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:25.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.358+0000 7f6e2964d640 1 -- 172.21.15.37:0/3462680805 >> 172.21.15.37:0/3462680805 conn(0x7f6e240a61b0 msgr2=0x7f6e240a7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:34:25.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.358+0000 7f6e2964d640 1 -- 172.21.15.37:0/3462680805 shutdown_connections 2024-04-26T00:34:25.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.358+0000 7f6e2964d640 1 -- 172.21.15.37:0/3462680805 wait complete. 2024-04-26T00:34:25.373 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:34:25.373 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-04-26T00:34:25.373 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-26T00:34:25.373 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-04-26T00:34:25.373 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-04-26T00:34:25.373 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-26T00:34:25.373 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T00:34:25.387 INFO:tasks.ceph.mon.c:Stopped 2024-04-26T00:34:25.388 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-26T00:34:25.388 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:34:25.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.554+0000 7ff0c2159640 10 monclient: get_monmap_and_config 2024-04-26T00:34:25.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.554+0000 7ff0c2159640 10 monclient: build_initial_monmap 2024-04-26T00:34:25.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.554+0000 7ff0c2159640 10 monclient: monmap: 2024-04-26T00:34:25.550 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:34:25.550 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:25.550 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:34:25.557006+0000 2024-04-26T00:34:25.550 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:34:25.557006+0000 2024-04-26T00:34:25.550 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:34:25.550 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:25.550 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:34:25.550 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:34:25.550 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:34:25.550 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:34:25.550 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:34:25.551 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:25.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.554+0000 7ff0c2159640 1 Processor -- start 2024-04-26T00:34:25.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.554+0000 7ff0c2159640 1 -- start start 2024-04-26T00:34:25.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.554+0000 7ff0c2159640 10 monclient: init 2024-04-26T00:34:25.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.554+0000 7ff0c2159640 10 monclient: _reopen_session rank -1 2024-04-26T00:34:25.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.554+0000 7ff0c2159640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-04-26T00:34:25.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.554+0000 7ff0c2159640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff0bc136e10 0x7ff0bc09c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:25.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.554+0000 7ff0c2159640 10 monclient(hunting): picked mon.noname-c con 0x7ff0bc136e10 addr v2:172.21.15.45:3300/0 2024-04-26T00:34:25.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.554+0000 7ff0c2159640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff0bc1371f0 addr v1:172.21.15.45:6789/0 2024-04-26T00:34:25.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.554+0000 7ff0c2159640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff0bc13c340 0x7ff0bc13e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:25.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.554+0000 7ff0c2159640 10 monclient(hunting): picked mon.noname-a con 0x7ff0bc13c340 addr v2:172.21.15.37:3300/0 2024-04-26T00:34:25.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.554+0000 7ff0c2159640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0bc09b680 con 0x7ff0bc1371f0 2024-04-26T00:34:25.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.554+0000 7ff0c2159640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:25.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.554+0000 7ff0c2159640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7ff0bc09b8f0 con 0x7ff0bc13c340 2024-04-26T00:34:25.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.554+0000 7ff0c2159640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:25.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.554+0000 7ff0c2159640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7ff0bc09bbd0 con 0x7ff0bc136e10 2024-04-26T00:34:25.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.554+0000 7ff0c2159640 10 monclient(hunting): _renew_subs 2024-04-26T00:34:25.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.554+0000 7ff0c2159640 10 monclient(hunting): authenticate will time out at 1846.863354s 2024-04-26T00:34:25.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.554+0000 7ff0bbfff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff0bc13c340 0x7ff0bc13e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:25.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.554+0000 7ff0bbfff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff0bc13c340 0x7ff0bc13e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33856/0 (socket says 172.21.15.37:33856) 2024-04-26T00:34:25.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.554+0000 7ff0bbfff640 1 -- 172.21.15.37:0/3782475898 learned_addr learned my addr 172.21.15.37:0/3782475898 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:34:25.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.554+0000 7ff0bbfff640 10 monclient(hunting): get_auth_request con 0x7ff0bc13c340 auth_method 0 2024-04-26T00:34:25.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.554+0000 7ff0bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:25.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.554+0000 7ff0bbfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:25.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.554+0000 7ff0bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:25.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.554+0000 7ff0bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:25.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.554+0000 7ff0bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:25.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.554+0000 7ff0bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:25.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.554+0000 7ff0bb7fe640 1 --2- 172.21.15.37:0/3782475898 >> v2:172.21.15.45:3300/0 conn(0x7ff0bc136e10 0x7ff0bc09c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:25.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.554+0000 7ff0bb7fe640 10 monclient(hunting): get_auth_request con 0x7ff0bc136e10 auth_method 0 2024-04-26T00:34:25.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.554+0000 7ff0bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:25.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.554+0000 7ff0bb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:25.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.554+0000 7ff0bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:25.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.554+0000 7ff0bbfff640 10 monclient(hunting): handle_auth_done global_id 144444 payload 274 2024-04-26T00:34:25.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.554+0000 7ff0bbfff640 1 -- 172.21.15.37:0/3782475898 >> v2:172.21.15.45:3300/0 conn(0x7ff0bc136e10 msgr2=0x7ff0bc09c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:34:25.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.554+0000 7ff0bbfff640 1 --2- 172.21.15.37:0/3782475898 >> v2:172.21.15.45:3300/0 conn(0x7ff0bc136e10 0x7ff0bc09c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:25.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.558+0000 7ff0bbfff640 1 -- 172.21.15.37:0/3782475898 >> v1:172.21.15.45:6789/0 conn(0x7ff0bc1371f0 legacy=0x7ff0bc137600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:34:25.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.558+0000 7ff0bbfff640 10 monclient: _finish_hunting 0 2024-04-26T00:34:25.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.558+0000 7ff0bbfff640 1 monclient: found mon.noname-a 2024-04-26T00:34:25.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.558+0000 7ff0bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:34:25.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.558+0000 7ff0bbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:34:25.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.558+0000 7ff0bbfff640 1 -- 172.21.15.37:0/3782475898 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0bc0a5d90 con 0x7ff0bc13c340 2024-04-26T00:34:25.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0bbfff640 1 -- 172.21.15.37:0/3782475898 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0bc13c340 msgr2=0x7ff0bc13e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:25.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0bbfff640 1 --2- 172.21.15.37:0/3782475898 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0bc13c340 0x7ff0bc13e730 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc0a5b00 tx=0x7ff0b401c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=287cc6e70dda609a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:25.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0baffd640 1 -- 172.21.15.37:0/3782475898 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b40a5530 con 0x7ff0bc13c340 2024-04-26T00:34:25.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:25.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0baffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:34:25.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0baffd640 10 monclient: dump: 2024-04-26T00:34:25.558 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:25.558 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:25.558 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.558 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.558 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:25.559 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:25.559 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:25.559 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:25.559 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:25.559 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:25.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0baffd640 10 monclient: _finish_auth 0 2024-04-26T00:34:25.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0baffd640 10 monclient: _check_auth_tickets 2024-04-26T00:34:25.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:34:25.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0baffd640 1 -- 172.21.15.37:0/3782475898 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0b40a56d0 con 0x7ff0bc13c340 2024-04-26T00:34:25.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0baffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:34:25.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0baffd640 1 -- 172.21.15.37:0/3782475898 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b40a5aa0 con 0x7ff0bc13c340 2024-04-26T00:34:25.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0c2159640 5 monclient: authenticate success, global_id 144444 2024-04-26T00:34:25.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0c2159640 10 monclient: get_monmap_and_config success 2024-04-26T00:34:25.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0c2159640 4 set_mon_vals no callback set 2024-04-26T00:34:25.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:25.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0baffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:34:25.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0baffd640 10 monclient: dump: 2024-04-26T00:34:25.560 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:25.560 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:25.560 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.560 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.560 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:25.560 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:25.560 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:25.560 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:25.560 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:25.560 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:25.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0c2159640 10 monclient: shutdown 2024-04-26T00:34:25.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0c2159640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:34:25.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0c2159640 1 -- 172.21.15.37:0/3782475898 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0bc13c340 msgr2=0x7ff0bc13e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:25.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0c2159640 1 --2- 172.21.15.37:0/3782475898 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0bc13c340 0x7ff0bc13e730 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc0a5b00 tx=0x7ff0b401c870 comp rx=0 tx=0).stop 2024-04-26T00:34:25.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0c2159640 1 -- 172.21.15.37:0/3782475898 shutdown_connections 2024-04-26T00:34:25.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0c2159640 1 --2- 172.21.15.37:0/3782475898 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0bc13c340 0x7ff0bc13e730 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:25.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0c2159640 1 --2- 172.21.15.37:0/3782475898 >> v2:172.21.15.45:3300/0 conn(0x7ff0bc136e10 0x7ff0bc09c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:25.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0c2159640 1 -- 172.21.15.37:0/3782475898 >> 172.21.15.37:0/3782475898 conn(0x7ff0bc0a64e0 msgr2=0x7ff0bc0a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:34:25.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0c2159640 1 -- 172.21.15.37:0/3782475898 shutdown_connections 2024-04-26T00:34:25.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0c2159640 1 -- 172.21.15.37:0/3782475898 wait complete. 2024-04-26T00:34:25.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0c2159640 10 monclient: build_initial_monmap 2024-04-26T00:34:25.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0c2159640 10 monclient: monmap: 2024-04-26T00:34:25.565 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:34:25.565 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:34:25.565 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:34:25.565 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:34:25.565 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:34:25.565 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:34:25.565 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:34:25.565 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:34:25.566 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:34:25.566 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:25.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0c2159640 1 Processor -- start 2024-04-26T00:34:25.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0c2159640 1 -- start start 2024-04-26T00:34:25.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0c2159640 10 monclient: init 2024-04-26T00:34:25.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0c2159640 10 monclient: _reopen_session rank -1 2024-04-26T00:34:25.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0c2159640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:34:25.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0c2159640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0bc136e10 0x7ff0bc1cede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:25.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0c2159640 10 monclient(hunting): picked mon.noname-a con 0x7ff0bc136e10 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:34:25.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0c2159640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0bc1371f0 0x7ff0bc135f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:25.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0c2159640 10 monclient(hunting): picked mon.noname-b con 0x7ff0bc1371f0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:34:25.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0c2159640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0bc13c340 0x7ff0bc138c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:25.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0c2159640 10 monclient(hunting): picked mon.noname-c con 0x7ff0bc13c340 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:34:25.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0c2159640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:25.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0c2159640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff0bc141820 con 0x7ff0bc136e10 2024-04-26T00:34:25.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0c2159640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:25.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0c2159640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff0bc1416a0 con 0x7ff0bc1371f0 2024-04-26T00:34:25.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0c2159640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:25.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0c2159640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff0bc1419a0 con 0x7ff0bc13c340 2024-04-26T00:34:25.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0c2159640 10 monclient(hunting): _renew_subs 2024-04-26T00:34:25.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0c2159640 10 monclient(hunting): authenticate will time out at 1846.871140s 2024-04-26T00:34:25.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0bbfff640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0bc13c340 msgr2=0x7ff0bc138c70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:34:25.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0bbfff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0bc13c340 0x7ff0bc138c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:34:25.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0bb7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0bc136e10 0x7ff0bc1cede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:25.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0bb7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0bc136e10 0x7ff0bc1cede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33870/0 (socket says 172.21.15.37:33870) 2024-04-26T00:34:25.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0bb7fe640 1 -- 172.21.15.37:0/1247293611 learned_addr learned my addr 172.21.15.37:0/1247293611 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:34:25.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0bb7fe640 10 monclient(hunting): get_auth_request con 0x7ff0bc136e10 auth_method 0 2024-04-26T00:34:25.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:25.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0bb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:25.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:25.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0bb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:25.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0bb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:25.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0bb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:25.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0b3fff640 1 --2- 172.21.15.37:0/1247293611 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0bc1371f0 0x7ff0bc135f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:25.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0b3fff640 10 monclient(hunting): get_auth_request con 0x7ff0bc1371f0 auth_method 0 2024-04-26T00:34:25.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:25.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0b3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:25.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:25.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0bb7fe640 10 monclient(hunting): handle_auth_done global_id 144447 payload 824 2024-04-26T00:34:25.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0bb7fe640 1 -- 172.21.15.37:0/1247293611 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0bc13c340 msgr2=0x7ff0bc138c70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:34:25.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0bb7fe640 1 --2- 172.21.15.37:0/1247293611 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0bc13c340 0x7ff0bc138c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:25.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0bb7fe640 1 -- 172.21.15.37:0/1247293611 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0bc1371f0 msgr2=0x7ff0bc135f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:25.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0bb7fe640 1 --2- 172.21.15.37:0/1247293611 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0bc1371f0 0x7ff0bc135f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:25.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0bb7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:34:25.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0bb7fe640 1 monclient: found mon.noname-a 2024-04-26T00:34:25.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:34:25.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0bb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:34:25.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.562+0000 7ff0bb7fe640 1 -- 172.21.15.37:0/1247293611 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0bc120640 con 0x7ff0bc136e10 2024-04-26T00:34:25.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.566+0000 7ff0bb7fe640 1 -- 172.21.15.37:0/1247293611 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0bc136e10 msgr2=0x7ff0bc1cede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:25.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.566+0000 7ff0bb7fe640 1 --2- 172.21.15.37:0/1247293611 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0bc136e10 0x7ff0bc1cede0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac00d800 tx=0x7ff0ac00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:25.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.566+0000 7ff0b8ff9640 1 -- 172.21.15.37:0/1247293611 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0ac00b8d0 con 0x7ff0bc136e10 2024-04-26T00:34:25.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.566+0000 7ff0b8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:25.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.566+0000 7ff0b8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:34:25.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.566+0000 7ff0b8ff9640 10 monclient: dump: 2024-04-26T00:34:25.570 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:25.570 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:25.570 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.570 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.571 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:25.571 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:25.571 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:25.571 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:25.571 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:25.571 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:25.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.566+0000 7ff0b8ff9640 10 monclient: _finish_auth 0 2024-04-26T00:34:25.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.566+0000 7ff0b8ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:34:25.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.566+0000 7ff0b8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:34:25.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.566+0000 7ff0b8ff9640 1 -- 172.21.15.37:0/1247293611 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0ac019070 con 0x7ff0bc136e10 2024-04-26T00:34:25.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.566+0000 7ff0c2159640 5 monclient: authenticate success, global_id 144447 2024-04-26T00:34:25.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.566+0000 7ff0c2159640 10 monclient: _renew_subs 2024-04-26T00:34:25.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.566+0000 7ff0c2159640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:34:25.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.566+0000 7ff0c2159640 1 -- 172.21.15.37:0/1247293611 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0bc120920 con 0x7ff0bc136e10 2024-04-26T00:34:25.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.566+0000 7ff0b8ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:34:25.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.566+0000 7ff0b8ff9640 1 -- 172.21.15.37:0/1247293611 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0ac00bc70 con 0x7ff0bc136e10 2024-04-26T00:34:25.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.566+0000 7ff0b8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:25.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.566+0000 7ff0b8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:34:25.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.566+0000 7ff0b8ff9640 10 monclient: dump: 2024-04-26T00:34:25.572 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:25.572 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:25.572 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.572 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:25.572 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:25.572 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:25.572 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:25.572 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:25.572 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:25.572 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:25.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.566+0000 7ff0c2159640 10 monclient: _renew_subs 2024-04-26T00:34:25.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.566+0000 7ff0c2159640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:34:25.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.566+0000 7ff0ba7fc640 4 set_mon_vals no callback set 2024-04-26T00:34:25.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.566+0000 7ff0c2159640 1 -- 172.21.15.37:0/1247293611 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0bc13b1a0 con 0x7ff0bc136e10 2024-04-26T00:34:25.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.566+0000 7ff0c2159640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:34:25.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.566+0000 7ff0b8ff9640 1 -- 172.21.15.37:0/1247293611 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff0ac01e460 con 0x7ff0bc136e10 2024-04-26T00:34:25.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.566+0000 7ff0c2159640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:34:25.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.566+0000 7ff0c2159640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:34:25.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.566+0000 7ff0c2159640 1 -- 172.21.15.37:0/1247293611 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0bc075910 con 0x7ff0bc136e10 2024-04-26T00:34:25.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.566+0000 7ff0b8ff9640 1 --2- 172.21.15.37:0/1247293611 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff09c035e30 0x7ff09c0382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:25.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.566+0000 7ff0b8ff9640 1 -- 172.21.15.37:0/1247293611 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(565..565 src has 370..565) ==== 3959+0+0 (secure 0 0 0) 0x7ff0ac048ff0 con 0x7ff0bc136e10 2024-04-26T00:34:25.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.566+0000 7ff0b3fff640 1 --2- 172.21.15.37:0/1247293611 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff09c035e30 0x7ff09c0382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:25.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.566+0000 7ff0b3fff640 10 monclient: get_auth_request con 0x7ff09c035e30 auth_method 0 2024-04-26T00:34:25.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.570+0000 7ff0b3fff640 1 --2- 172.21.15.37:0/1247293611 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff09c035e30 0x7ff09c0382f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc13b810 tx=0x7ff0a4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:25.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.574+0000 7ff0b8ff9640 1 -- 172.21.15.37:0/1247293611 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff0ac00be10 con 0x7ff0bc136e10 2024-04-26T00:34:25.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.574+0000 7ff0b8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:34:25.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.574+0000 7ff0b8ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:34:25.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.794+0000 7ff0c2159640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:34:25.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.794+0000 7ff0c2159640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:34:25.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.794+0000 7ff0c2159640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:34:25.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.794+0000 7ff0c2159640 1 -- 172.21.15.37:0/1247293611 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff0bc066f40 con 0x7ff0bc136e10 2024-04-26T00:34:25.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.794+0000 7ff0b8ff9640 1 -- 172.21.15.37:0/1247293611 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7ff0ac047d20 con 0x7ff0bc136e10 2024-04-26T00:34:25.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.794+0000 7ff0b8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:34:25.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.794+0000 7ff0b8ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:34:25.791 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":108,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:34:25.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.798+0000 7ff0c2159640 1 -- 172.21.15.37:0/1247293611 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff09c035e30 msgr2=0x7ff09c0382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:25.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.798+0000 7ff0c2159640 1 --2- 172.21.15.37:0/1247293611 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff09c035e30 0x7ff09c0382f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc13b810 tx=0x7ff0a4009040 comp rx=0 tx=0).stop 2024-04-26T00:34:25.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.798+0000 7ff0c2159640 10 monclient: shutdown 2024-04-26T00:34:25.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.798+0000 7ff0c2159640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:34:25.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.798+0000 7ff0c2159640 1 -- 172.21.15.37:0/1247293611 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0bc136e10 msgr2=0x7ff0bc1cede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:25.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.798+0000 7ff0c2159640 1 --2- 172.21.15.37:0/1247293611 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0bc136e10 0x7ff0bc1cede0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac00d800 tx=0x7ff0ac00dcc0 comp rx=0 tx=0).stop 2024-04-26T00:34:25.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.798+0000 7ff0c2159640 1 -- 172.21.15.37:0/1247293611 shutdown_connections 2024-04-26T00:34:25.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.798+0000 7ff0c2159640 1 --2- 172.21.15.37:0/1247293611 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff09c035e30 0x7ff09c0382f0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:25.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.798+0000 7ff0c2159640 1 --2- 172.21.15.37:0/1247293611 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0bc13c340 0x7ff0bc138c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:25.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.798+0000 7ff0c2159640 1 --2- 172.21.15.37:0/1247293611 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff0bc1371f0 0x7ff0bc135f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:25.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.798+0000 7ff0c2159640 1 --2- 172.21.15.37:0/1247293611 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0bc136e10 0x7ff0bc1cede0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:25.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.798+0000 7ff0c2159640 1 -- 172.21.15.37:0/1247293611 >> 172.21.15.37:0/1247293611 conn(0x7ff0bc0a64e0 msgr2=0x7ff0bc13d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:34:25.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.798+0000 7ff0c2159640 1 -- 172.21.15.37:0/1247293611 shutdown_connections 2024-04-26T00:34:25.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:25.798+0000 7ff0c2159640 1 -- 172.21.15.37:0/1247293611 wait complete. 2024-04-26T00:34:28.810 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:34:28.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.946+0000 7f450b9b2640 10 monclient: get_monmap_and_config 2024-04-26T00:34:28.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.946+0000 7f450b9b2640 10 monclient: build_initial_monmap 2024-04-26T00:34:28.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.946+0000 7f450b9b2640 10 monclient: monmap: 2024-04-26T00:34:28.942 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:34:28.942 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:28.942 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:34:28.949168+0000 2024-04-26T00:34:28.942 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:34:28.949168+0000 2024-04-26T00:34:28.942 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:34:28.942 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:28.942 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:34:28.942 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:34:28.942 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:34:28.942 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:34:28.942 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:34:28.943 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:28.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.946+0000 7f450b9b2640 1 Processor -- start 2024-04-26T00:34:28.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.946+0000 7f450b9b2640 1 -- start start 2024-04-26T00:34:28.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.946+0000 7f450b9b2640 10 monclient: init 2024-04-26T00:34:28.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.946+0000 7f450b9b2640 10 monclient: _reopen_session rank -1 2024-04-26T00:34:28.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.946+0000 7f450b9b2640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-04-26T00:34:28.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.946+0000 7f450b9b2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4504136da0 addr v1:172.21.15.37:6789/0 2024-04-26T00:34:28.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.946+0000 7f450b9b2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4504137180 addr v1:172.21.15.45:6789/0 2024-04-26T00:34:28.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.946+0000 7f450b9b2640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f450413d2e0 0x7f450413f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:28.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.946+0000 7f450b9b2640 10 monclient(hunting): picked mon.noname-c con 0x7f450413d2e0 addr v2:172.21.15.45:3300/0 2024-04-26T00:34:28.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.946+0000 7f450b9b2640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f450409b4f0 con 0x7f4504136da0 2024-04-26T00:34:28.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.946+0000 7f450b9b2640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f450409b7d0 con 0x7f4504137180 2024-04-26T00:34:28.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.946+0000 7f450b9b2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:28.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.946+0000 7f450b9b2640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f450409bb70 con 0x7f450413d2e0 2024-04-26T00:34:28.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.946+0000 7f450b9b2640 10 monclient(hunting): _renew_subs 2024-04-26T00:34:28.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.946+0000 7f450b9b2640 10 monclient(hunting): authenticate will time out at 1850.255084s 2024-04-26T00:34:28.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.946+0000 7f450974e640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f4504136da0 0x7f450409bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:33420/0 (socket says 172.21.15.37:33420) 2024-04-26T00:34:28.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.946+0000 7f450974e640 1 -- 172.21.15.37:0/84343702 learned_addr learned my addr 172.21.15.37:0/84343702 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:34:28.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.946+0000 7f4509f4f640 1 --2- 172.21.15.37:0/84343702 >> v2:172.21.15.45:3300/0 conn(0x7f450413d2e0 0x7f450413f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:28.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.946+0000 7f4509f4f640 10 monclient(hunting): get_auth_request con 0x7f450413d2e0 auth_method 0 2024-04-26T00:34:28.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.946+0000 7f4509f4f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:28.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.946+0000 7f4509f4f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:28.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.946+0000 7f4509f4f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:28.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.946+0000 7f4509f4f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:28.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.946+0000 7f4509f4f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:28.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.946+0000 7f4509f4f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:28.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.950+0000 7f4509f4f640 10 monclient(hunting): handle_auth_done global_id 174136 payload 274 2024-04-26T00:34:28.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.950+0000 7f4509f4f640 1 -- 172.21.15.37:0/84343702 >> v1:172.21.15.45:6789/0 conn(0x7f4504137180 legacy=0x7f45041375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:34:28.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.950+0000 7f4509f4f640 1 -- 172.21.15.37:0/84343702 >> v1:172.21.15.37:6789/0 conn(0x7f4504136da0 legacy=0x7f450409bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:34:28.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.950+0000 7f4509f4f640 10 monclient: _finish_hunting 0 2024-04-26T00:34:28.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.950+0000 7f4509f4f640 1 monclient: found mon.noname-c 2024-04-26T00:34:28.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.950+0000 7f4509f4f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:34:28.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.950+0000 7f4509f4f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:34:28.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.950+0000 7f4509f4f640 1 -- 172.21.15.37:0/84343702 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f450409c5e0 con 0x7f450413d2e0 2024-04-26T00:34:28.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f4509f4f640 1 -- 172.21.15.37:0/84343702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f450413d2e0 msgr2=0x7f450413f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:28.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f4509f4f640 1 --2- 172.21.15.37:0/84343702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f450413d2e0 0x7f450413f6d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f44fc009d90 tx=0x7f44fc01c7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1159c6f3f501029 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:28.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f44fbfff640 1 -- 172.21.15.37:0/84343702 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44fc0ac070 con 0x7f450413d2e0 2024-04-26T00:34:28.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f44fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:28.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f44fbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:34:28.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f44fbfff640 10 monclient: dump: 2024-04-26T00:34:28.951 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:28.951 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:28.951 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:28.951 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:28.951 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:28.951 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:28.951 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:28.952 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:28.952 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:28.952 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:28.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f44fbfff640 10 monclient: _finish_auth 0 2024-04-26T00:34:28.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f44fbfff640 10 monclient: _check_auth_tickets 2024-04-26T00:34:28.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f44fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:34:28.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f44fbfff640 1 -- 172.21.15.37:0/84343702 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44fc0a6760 con 0x7f450413d2e0 2024-04-26T00:34:28.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f450b9b2640 5 monclient: authenticate success, global_id 174136 2024-04-26T00:34:28.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f450b9b2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:34:28.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f44fbfff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:34:28.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f44fbfff640 1 -- 172.21.15.37:0/84343702 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44fc0a6b30 con 0x7f450413d2e0 2024-04-26T00:34:28.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f450b9b2640 10 monclient: get_monmap_and_config success 2024-04-26T00:34:28.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f450b9b2640 4 set_mon_vals no callback set 2024-04-26T00:34:28.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f44fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:28.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f44fbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:34:28.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f44fbfff640 10 monclient: dump: 2024-04-26T00:34:28.953 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:28.953 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:28.953 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:28.953 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:28.953 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:28.953 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:28.953 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:28.953 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:28.953 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:28.953 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:28.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f450b9b2640 10 monclient: shutdown 2024-04-26T00:34:28.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f450b9b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:34:28.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f450b9b2640 1 -- 172.21.15.37:0/84343702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f450413d2e0 msgr2=0x7f450413f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:28.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f450b9b2640 1 --2- 172.21.15.37:0/84343702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f450413d2e0 0x7f450413f6d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f44fc009d90 tx=0x7f44fc01c7b0 comp rx=0 tx=0).stop 2024-04-26T00:34:28.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f450b9b2640 1 -- 172.21.15.37:0/84343702 shutdown_connections 2024-04-26T00:34:28.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f450b9b2640 1 --2- 172.21.15.37:0/84343702 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f450413d2e0 0x7f450413f6d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:28.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f450b9b2640 1 -- 172.21.15.37:0/84343702 >> 172.21.15.37:0/84343702 conn(0x7f45040a5f00 msgr2=0x7f45040a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:34:28.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f450b9b2640 1 -- 172.21.15.37:0/84343702 shutdown_connections 2024-04-26T00:34:28.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f450b9b2640 1 -- 172.21.15.37:0/84343702 wait complete. 2024-04-26T00:34:28.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f450b9b2640 10 monclient: build_initial_monmap 2024-04-26T00:34:28.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f450b9b2640 10 monclient: monmap: 2024-04-26T00:34:28.954 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:34:28.954 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:34:28.954 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:34:28.954 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:34:28.954 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:34:28.955 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:34:28.955 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:34:28.955 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:34:28.955 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:34:28.955 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:28.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f450b9b2640 1 Processor -- start 2024-04-26T00:34:28.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f450b9b2640 1 -- start start 2024-04-26T00:34:28.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f450b9b2640 10 monclient: init 2024-04-26T00:34:28.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f450b9b2640 10 monclient: _reopen_session rank -1 2024-04-26T00:34:28.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f450b9b2640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:34:28.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f450b9b2640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4504136da0 0x7f45041cf150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:28.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f450b9b2640 10 monclient(hunting): picked mon.noname-c con 0x7f4504136da0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:34:28.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f450b9b2640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4504137180 0x7f4504135c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:28.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f450b9b2640 10 monclient(hunting): picked mon.noname-a con 0x7f4504137180 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:34:28.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f450b9b2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f450413d2e0 0x7f4504138ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:28.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f450b9b2640 10 monclient(hunting): picked mon.noname-b con 0x7f450413d2e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:34:28.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f450b9b2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:28.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f450b9b2640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f4504142e20 con 0x7f4504137180 2024-04-26T00:34:28.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f450b9b2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:28.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f450b9b2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4504142ca0 con 0x7f450413d2e0 2024-04-26T00:34:28.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f450b9b2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:28.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f450b9b2640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4504142fa0 con 0x7f4504136da0 2024-04-26T00:34:28.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f450b9b2640 10 monclient(hunting): _renew_subs 2024-04-26T00:34:28.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f450b9b2640 10 monclient(hunting): authenticate will time out at 1850.263734s 2024-04-26T00:34:28.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f450974e640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4504136da0 msgr2=0x7f45041cf150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:34:28.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f450974e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4504136da0 0x7f45041cf150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:34:28.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f4508f4d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4504137180 0x7f4504135c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:28.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f4508f4d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4504137180 0x7f4504135c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33884/0 (socket says 172.21.15.37:33884) 2024-04-26T00:34:28.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f4508f4d640 1 -- 172.21.15.37:0/2795943574 learned_addr learned my addr 172.21.15.37:0/2795943574 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:34:28.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f4508f4d640 10 monclient(hunting): get_auth_request con 0x7f4504137180 auth_method 0 2024-04-26T00:34:28.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f4508f4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:28.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f4508f4d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:28.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f4508f4d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:28.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f4508f4d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:28.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.954+0000 7f4508f4d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:28.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f4508f4d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:28.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f4509f4f640 1 --2- 172.21.15.37:0/2795943574 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f450413d2e0 0x7f4504138ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:28.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f4509f4f640 10 monclient(hunting): get_auth_request con 0x7f450413d2e0 auth_method 0 2024-04-26T00:34:28.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f4509f4f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:28.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f4509f4f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:28.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f4509f4f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:28.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f4508f4d640 10 monclient(hunting): handle_auth_done global_id 144450 payload 824 2024-04-26T00:34:28.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f4508f4d640 1 -- 172.21.15.37:0/2795943574 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4504136da0 msgr2=0x7f45041cf150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:34:28.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f4508f4d640 1 --2- 172.21.15.37:0/2795943574 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4504136da0 0x7f45041cf150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:28.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f4508f4d640 1 -- 172.21.15.37:0/2795943574 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f450413d2e0 msgr2=0x7f4504138ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:28.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f4508f4d640 1 --2- 172.21.15.37:0/2795943574 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f450413d2e0 0x7f4504138ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:28.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f4508f4d640 10 monclient: _finish_hunting 0 2024-04-26T00:34:28.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f4508f4d640 1 monclient: found mon.noname-a 2024-04-26T00:34:28.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f4508f4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:34:28.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f4508f4d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:34:28.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f4508f4d640 1 -- 172.21.15.37:0/2795943574 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f450413b010 con 0x7f4504137180 2024-04-26T00:34:28.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f4508f4d640 1 -- 172.21.15.37:0/2795943574 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4504137180 msgr2=0x7f4504135c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:28.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f4508f4d640 1 --2- 172.21.15.37:0/2795943574 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4504137180 0x7f4504135c40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f44f400c450 tx=0x7f44f400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:28.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f44f9ffb640 1 -- 172.21.15.37:0/2795943574 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f4012a40 con 0x7f4504137180 2024-04-26T00:34:28.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f44f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:28.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f44f9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:34:28.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f44f9ffb640 10 monclient: dump: 2024-04-26T00:34:28.963 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:28.963 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:28.963 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:28.963 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:28.963 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:28.963 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:28.963 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:28.963 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:28.963 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:28.963 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:28.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f44f9ffb640 10 monclient: _finish_auth 0 2024-04-26T00:34:28.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f44f9ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:34:28.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f44f9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:34:28.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f44f9ffb640 1 -- 172.21.15.37:0/2795943574 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44f4017070 con 0x7f4504137180 2024-04-26T00:34:28.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f450b9b2640 5 monclient: authenticate success, global_id 144450 2024-04-26T00:34:28.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f450b9b2640 10 monclient: _renew_subs 2024-04-26T00:34:28.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f450b9b2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:34:28.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f450b9b2640 1 -- 172.21.15.37:0/2795943574 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f450413c740 con 0x7f4504137180 2024-04-26T00:34:28.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f44f9ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:34:28.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f44f9ffb640 1 -- 172.21.15.37:0/2795943574 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f4012e40 con 0x7f4504137180 2024-04-26T00:34:28.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f44f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:28.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f44f9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:34:28.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f44f9ffb640 10 monclient: dump: 2024-04-26T00:34:28.964 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:28.964 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:28.964 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:28.964 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:28.964 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:28.965 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:28.965 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:28.965 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:28.965 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:28.965 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:28.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f44fb7fe640 4 set_mon_vals no callback set 2024-04-26T00:34:28.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f450b9b2640 10 monclient: _renew_subs 2024-04-26T00:34:28.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f450b9b2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:34:28.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f450b9b2640 1 -- 172.21.15.37:0/2795943574 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f450413b5d0 con 0x7f4504137180 2024-04-26T00:34:28.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f450b9b2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:34:28.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f450b9b2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:34:28.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f450b9b2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:34:28.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f450b9b2640 1 -- 172.21.15.37:0/2795943574 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4504066f40 con 0x7f4504137180 2024-04-26T00:34:28.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f44f9ffb640 1 -- 172.21.15.37:0/2795943574 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f44f401c460 con 0x7f4504137180 2024-04-26T00:34:28.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.958+0000 7f44f9ffb640 1 --2- 172.21.15.37:0/2795943574 >> v2:172.21.15.45:6812/2379918623 conn(0x7f44e8035e30 0x7f44e80382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:28.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.962+0000 7f44f9ffb640 1 -- 172.21.15.37:0/2795943574 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(565..565 src has 370..565) ==== 3959+0+0 (secure 0 0 0) 0x7f44f40488e0 con 0x7f4504137180 2024-04-26T00:34:28.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.962+0000 7f450974e640 1 --2- 172.21.15.37:0/2795943574 >> v2:172.21.15.45:6812/2379918623 conn(0x7f44e8035e30 0x7f44e80382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:28.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.962+0000 7f450974e640 10 monclient: get_auth_request con 0x7f44e8035e30 auth_method 0 2024-04-26T00:34:28.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.962+0000 7f450974e640 1 --2- 172.21.15.37:0/2795943574 >> v2:172.21.15.45:6812/2379918623 conn(0x7f44e8035e30 0x7f44e80382f0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f4500004820 tx=0x7f4500007b30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:28.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.966+0000 7f44f9ffb640 1 -- 172.21.15.37:0/2795943574 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f44f401b3c0 con 0x7f4504137180 2024-04-26T00:34:28.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.966+0000 7f44f9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:34:28.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:28.966+0000 7f44f9ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:34:29.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:29.190+0000 7f450b9b2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:34:29.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:29.190+0000 7f450b9b2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:34:29.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:29.190+0000 7f450b9b2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:34:29.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:29.190+0000 7f450b9b2640 1 -- 172.21.15.37:0/2795943574 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f45040a8210 con 0x7f4504137180 2024-04-26T00:34:29.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:29.190+0000 7f44f9ffb640 1 -- 172.21.15.37:0/2795943574 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f44f4037e20 con 0x7f4504137180 2024-04-26T00:34:29.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:29.190+0000 7f44f9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:34:29.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:29.190+0000 7f44f9ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:34:29.186 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":108,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:34:29.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:29.194+0000 7f450b9b2640 1 -- 172.21.15.37:0/2795943574 >> v2:172.21.15.45:6812/2379918623 conn(0x7f44e8035e30 msgr2=0x7f44e80382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:29.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:29.194+0000 7f450b9b2640 1 --2- 172.21.15.37:0/2795943574 >> v2:172.21.15.45:6812/2379918623 conn(0x7f44e8035e30 0x7f44e80382f0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f4500004820 tx=0x7f4500007b30 comp rx=0 tx=0).stop 2024-04-26T00:34:29.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:29.194+0000 7f450b9b2640 10 monclient: shutdown 2024-04-26T00:34:29.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:29.194+0000 7f450b9b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:34:29.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:29.194+0000 7f450b9b2640 1 -- 172.21.15.37:0/2795943574 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4504137180 msgr2=0x7f4504135c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:29.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:29.194+0000 7f450b9b2640 1 --2- 172.21.15.37:0/2795943574 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4504137180 0x7f4504135c40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f44f400c450 tx=0x7f44f400c910 comp rx=0 tx=0).stop 2024-04-26T00:34:29.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:29.194+0000 7f450b9b2640 1 -- 172.21.15.37:0/2795943574 shutdown_connections 2024-04-26T00:34:29.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:29.194+0000 7f450b9b2640 1 --2- 172.21.15.37:0/2795943574 >> v2:172.21.15.45:6812/2379918623 conn(0x7f44e8035e30 0x7f44e80382f0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:29.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:29.194+0000 7f450b9b2640 1 --2- 172.21.15.37:0/2795943574 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f450413d2e0 0x7f4504138ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:29.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:29.194+0000 7f450b9b2640 1 --2- 172.21.15.37:0/2795943574 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4504137180 0x7f4504135c40 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:29.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:29.194+0000 7f450b9b2640 1 --2- 172.21.15.37:0/2795943574 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4504136da0 0x7f45041cf150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:29.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:29.194+0000 7f450b9b2640 1 -- 172.21.15.37:0/2795943574 >> 172.21.15.37:0/2795943574 conn(0x7f45040a5f00 msgr2=0x7f450413e150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:34:29.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:29.194+0000 7f450b9b2640 1 -- 172.21.15.37:0/2795943574 shutdown_connections 2024-04-26T00:34:29.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:29.194+0000 7f450b9b2640 1 -- 172.21.15.37:0/2795943574 wait complete. 2024-04-26T00:34:32.206 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:34:32.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.342+0000 7ff9a143d640 10 monclient: get_monmap_and_config 2024-04-26T00:34:32.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.342+0000 7ff9a143d640 10 monclient: build_initial_monmap 2024-04-26T00:34:32.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.342+0000 7ff9a143d640 10 monclient: monmap: 2024-04-26T00:34:32.338 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:34:32.338 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:32.338 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:34:32.345605+0000 2024-04-26T00:34:32.338 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:34:32.345605+0000 2024-04-26T00:34:32.339 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:34:32.339 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:32.339 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:34:32.339 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:34:32.339 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:34:32.339 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:34:32.339 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:34:32.339 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:32.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.342+0000 7ff9a143d640 1 Processor -- start 2024-04-26T00:34:32.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.342+0000 7ff9a143d640 1 -- start start 2024-04-26T00:34:32.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.342+0000 7ff9a143d640 10 monclient: init 2024-04-26T00:34:32.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.342+0000 7ff9a143d640 10 monclient: _reopen_session rank -1 2024-04-26T00:34:32.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.342+0000 7ff9a143d640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-04-26T00:34:32.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.342+0000 7ff9a143d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff99c0a0260 0x7ff99c0a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:32.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.342+0000 7ff9a143d640 10 monclient(hunting): picked mon.noname-b con 0x7ff99c0a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:34:32.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.342+0000 7ff9a143d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff99c134b30 addr v1:172.21.15.37:6789/0 2024-04-26T00:34:32.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.342+0000 7ff9a143d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff99c135660 0x7ff99c13ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:32.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.342+0000 7ff9a143d640 10 monclient(hunting): picked mon.noname-c con 0x7ff99c135660 addr v2:172.21.15.45:3300/0 2024-04-26T00:34:32.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.342+0000 7ff9a143d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff99c0a1130 con 0x7ff99c134b30 2024-04-26T00:34:32.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.342+0000 7ff9a143d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:32.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.342+0000 7ff9a143d640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7ff99c09ec10 con 0x7ff99c135660 2024-04-26T00:34:32.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.342+0000 7ff9a143d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:32.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.342+0000 7ff9a143d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff99c09f0f0 con 0x7ff99c0a0260 2024-04-26T00:34:32.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.342+0000 7ff9a143d640 10 monclient(hunting): _renew_subs 2024-04-26T00:34:32.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.342+0000 7ff9a143d640 10 monclient(hunting): authenticate will time out at 1853.651385s 2024-04-26T00:34:32.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.342+0000 7ff99affd640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff99c0a0260 msgr2=0x7ff99c0a0640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:34:32.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.342+0000 7ff99affd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff99c0a0260 0x7ff99c0a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:34:32.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.342+0000 7ff99a7fc640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7ff99c134b30 0x7ff99c134f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:50292/0 (socket says 172.21.15.37:50292) 2024-04-26T00:34:32.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.342+0000 7ff99a7fc640 1 -- 172.21.15.37:0/2884735200 learned_addr learned my addr 172.21.15.37:0/2884735200 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:34:32.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.342+0000 7ff99b7fe640 1 --2- 172.21.15.37:0/2884735200 >> v2:172.21.15.45:3300/0 conn(0x7ff99c135660 0x7ff99c13ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:32.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.342+0000 7ff99b7fe640 10 monclient(hunting): get_auth_request con 0x7ff99c135660 auth_method 0 2024-04-26T00:34:32.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.342+0000 7ff99b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:32.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.342+0000 7ff99b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:32.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.342+0000 7ff99b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:32.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.346+0000 7ff99b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:32.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.346+0000 7ff99b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:32.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.346+0000 7ff99b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:32.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.346+0000 7ff99b7fe640 10 monclient(hunting): handle_auth_done global_id 174142 payload 274 2024-04-26T00:34:32.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.346+0000 7ff99b7fe640 1 -- 172.21.15.37:0/2884735200 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff99c0a0260 msgr2=0x7ff99c0a0640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:34:32.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.346+0000 7ff99b7fe640 1 --2- 172.21.15.37:0/2884735200 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff99c0a0260 0x7ff99c0a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:32.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.346+0000 7ff99b7fe640 1 -- 172.21.15.37:0/2884735200 >> v1:172.21.15.37:6789/0 conn(0x7ff99c134b30 legacy=0x7ff99c134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:34:32.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.346+0000 7ff99b7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:34:32.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.346+0000 7ff99b7fe640 1 monclient: found mon.noname-c 2024-04-26T00:34:32.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.346+0000 7ff99b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:34:32.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.346+0000 7ff99b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:34:32.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.346+0000 7ff99b7fe640 1 -- 172.21.15.37:0/2884735200 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff99c06cff0 con 0x7ff99c135660 2024-04-26T00:34:32.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff99b7fe640 1 -- 172.21.15.37:0/2884735200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff99c135660 msgr2=0x7ff99c13ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:32.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff99b7fe640 1 --2- 172.21.15.37:0/2884735200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff99c135660 0x7ff99c13ce70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff9940097c0 tx=0x7ff99401c160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3fa070367dcd17d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:32.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff999ffb640 1 -- 172.21.15.37:0/2884735200 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9940b1070 con 0x7ff99c135660 2024-04-26T00:34:32.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff999ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:32.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff999ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:34:32.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff999ffb640 10 monclient: dump: 2024-04-26T00:34:32.354 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:32.354 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:32.354 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:32.354 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:32.354 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:32.354 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:32.354 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:32.354 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:32.354 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:32.354 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:32.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff999ffb640 10 monclient: _finish_auth 0 2024-04-26T00:34:32.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff999ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:34:32.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff999ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:34:32.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff999ffb640 1 -- 172.21.15.37:0/2884735200 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9940ac440 con 0x7ff99c135660 2024-04-26T00:34:32.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff999ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:34:32.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff999ffb640 1 -- 172.21.15.37:0/2884735200 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9940ac810 con 0x7ff99c135660 2024-04-26T00:34:32.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff999ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:32.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff999ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:34:32.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff999ffb640 10 monclient: dump: 2024-04-26T00:34:32.355 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:32.355 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:32.355 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:32.355 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:32.355 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:32.356 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:32.356 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:32.356 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:32.356 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:32.356 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:32.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff9a143d640 5 monclient: authenticate success, global_id 174142 2024-04-26T00:34:32.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff9a143d640 10 monclient: get_monmap_and_config success 2024-04-26T00:34:32.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff9a143d640 4 set_mon_vals no callback set 2024-04-26T00:34:32.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff9a143d640 10 monclient: shutdown 2024-04-26T00:34:32.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff9a143d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:34:32.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff9a143d640 1 -- 172.21.15.37:0/2884735200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff99c135660 msgr2=0x7ff99c13ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:32.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff9a143d640 1 --2- 172.21.15.37:0/2884735200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff99c135660 0x7ff99c13ce70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff9940097c0 tx=0x7ff99401c160 comp rx=0 tx=0).stop 2024-04-26T00:34:32.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff9a143d640 1 -- 172.21.15.37:0/2884735200 shutdown_connections 2024-04-26T00:34:32.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff9a143d640 1 --2- 172.21.15.37:0/2884735200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff99c135660 0x7ff99c13ce70 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:32.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff9a143d640 1 --2- 172.21.15.37:0/2884735200 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff99c0a0260 0x7ff99c0a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:32.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff9a143d640 1 -- 172.21.15.37:0/2884735200 >> 172.21.15.37:0/2884735200 conn(0x7ff99c0a1ad0 msgr2=0x7ff99c0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:34:32.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff9a143d640 1 -- 172.21.15.37:0/2884735200 shutdown_connections 2024-04-26T00:34:32.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff9a143d640 1 -- 172.21.15.37:0/2884735200 wait complete. 2024-04-26T00:34:32.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff9a143d640 10 monclient: build_initial_monmap 2024-04-26T00:34:32.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff9a143d640 10 monclient: monmap: 2024-04-26T00:34:32.357 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:34:32.357 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:34:32.357 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:34:32.357 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:34:32.357 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:34:32.357 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:34:32.357 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:34:32.357 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:34:32.357 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:34:32.358 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:32.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff9a143d640 1 Processor -- start 2024-04-26T00:34:32.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff9a143d640 1 -- start start 2024-04-26T00:34:32.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff9a143d640 10 monclient: init 2024-04-26T00:34:32.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff9a143d640 10 monclient: _reopen_session rank -1 2024-04-26T00:34:32.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff9a143d640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:34:32.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff9a143d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff99c0a0260 0x7ff99c13bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:32.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff9a143d640 10 monclient(hunting): picked mon.noname-c con 0x7ff99c0a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:34:32.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff9a143d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff99c134b30 0x7ff99c137730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:32.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff9a143d640 10 monclient(hunting): picked mon.noname-b con 0x7ff99c134b30 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:34:32.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff9a143d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff99c135660 0x7ff99c137c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:32.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff9a143d640 10 monclient(hunting): picked mon.noname-a con 0x7ff99c135660 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:34:32.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff9a143d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:32.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff9a143d640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff99c0a0010 con 0x7ff99c135660 2024-04-26T00:34:32.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff9a143d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:32.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff9a143d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff99c09fe90 con 0x7ff99c134b30 2024-04-26T00:34:32.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff9a143d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:32.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff9a143d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff99c09c4d0 con 0x7ff99c0a0260 2024-04-26T00:34:32.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff9a143d640 10 monclient(hunting): _renew_subs 2024-04-26T00:34:32.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff9a143d640 10 monclient(hunting): authenticate will time out at 1853.659928s 2024-04-26T00:34:32.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff99affd640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff99c0a0260 msgr2=0x7ff99c13bd60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:34:32.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.350+0000 7ff99affd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff99c0a0260 0x7ff99c13bd60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:34:32.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff99b7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff99c135660 0x7ff99c137c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:32.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff99b7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff99c135660 0x7ff99c137c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:38636/0 (socket says 172.21.15.37:38636) 2024-04-26T00:34:32.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff99b7fe640 1 -- 172.21.15.37:0/3719748715 learned_addr learned my addr 172.21.15.37:0/3719748715 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:34:32.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff99b7fe640 10 monclient(hunting): get_auth_request con 0x7ff99c135660 auth_method 0 2024-04-26T00:34:32.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff99b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:32.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff99b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:32.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff99b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:32.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff99b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:32.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff99b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:32.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff99b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:32.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff99a7fc640 1 --2- 172.21.15.37:0/3719748715 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff99c134b30 0x7ff99c137730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:32.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff99a7fc640 10 monclient(hunting): get_auth_request con 0x7ff99c134b30 auth_method 0 2024-04-26T00:34:32.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff99a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:32.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff99a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:32.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff99a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:32.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff99b7fe640 10 monclient(hunting): handle_auth_done global_id 144453 payload 824 2024-04-26T00:34:32.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff99b7fe640 1 -- 172.21.15.37:0/3719748715 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff99c0a0260 msgr2=0x7ff99c13bd60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:34:32.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff99b7fe640 1 --2- 172.21.15.37:0/3719748715 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff99c0a0260 0x7ff99c13bd60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:32.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff99b7fe640 1 -- 172.21.15.37:0/3719748715 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff99c134b30 msgr2=0x7ff99c137730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:32.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff99b7fe640 1 --2- 172.21.15.37:0/3719748715 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff99c134b30 0x7ff99c137730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:32.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff99b7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:34:32.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff99b7fe640 1 monclient: found mon.noname-a 2024-04-26T00:34:32.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff99b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:34:32.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff99b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:34:32.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff99b7fe640 1 -- 172.21.15.37:0/3719748715 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff99c1392f0 con 0x7ff99c135660 2024-04-26T00:34:32.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff99a7fc640 1 --2- 172.21.15.37:0/3719748715 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff99c134b30 0x7ff99c137730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:34:32.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff99b7fe640 1 -- 172.21.15.37:0/3719748715 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff99c135660 msgr2=0x7ff99c137c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:32.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff99b7fe640 1 --2- 172.21.15.37:0/3719748715 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff99c135660 0x7ff99c137c70 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff9940af040 tx=0x7ff994018f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:32.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff9837fe640 1 -- 172.21.15.37:0/3719748715 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9940b9070 con 0x7ff99c135660 2024-04-26T00:34:32.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff9837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:32.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff9837fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:34:32.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff9837fe640 10 monclient: dump: 2024-04-26T00:34:32.362 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:32.362 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:32.362 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:32.362 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:32.362 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:32.362 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:32.362 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:32.362 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:32.362 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:32.362 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:32.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff9837fe640 10 monclient: _finish_auth 0 2024-04-26T00:34:32.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff9837fe640 10 monclient: _check_auth_tickets 2024-04-26T00:34:32.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff9837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:34:32.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff9837fe640 1 -- 172.21.15.37:0/3719748715 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9940b5820 con 0x7ff99c135660 2024-04-26T00:34:32.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff9837fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:34:32.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff9a143d640 5 monclient: authenticate success, global_id 144453 2024-04-26T00:34:32.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff9837fe640 1 -- 172.21.15.37:0/3719748715 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9940b1040 con 0x7ff99c135660 2024-04-26T00:34:32.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff9a143d640 10 monclient: _renew_subs 2024-04-26T00:34:32.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff9a143d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:34:32.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff9a143d640 1 -- 172.21.15.37:0/3719748715 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff99c0729d0 con 0x7ff99c135660 2024-04-26T00:34:32.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff9837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:32.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff9837fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:34:32.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff9837fe640 10 monclient: dump: 2024-04-26T00:34:32.363 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:32.363 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:32.363 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:32.363 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:32.363 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:32.364 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:32.364 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:32.364 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:32.364 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:32.364 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:32.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff9997fa640 4 set_mon_vals no callback set 2024-04-26T00:34:32.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff9a143d640 10 monclient: _renew_subs 2024-04-26T00:34:32.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff9a143d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:34:32.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.354+0000 7ff9a143d640 1 -- 172.21.15.37:0/3719748715 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff99c139e50 con 0x7ff99c135660 2024-04-26T00:34:32.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.358+0000 7ff9837fe640 1 -- 172.21.15.37:0/3719748715 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff9940bc070 con 0x7ff99c135660 2024-04-26T00:34:32.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.358+0000 7ff9a143d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:34:32.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.358+0000 7ff9a143d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:34:32.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.358+0000 7ff9a143d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:34:32.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.358+0000 7ff9a143d640 1 -- 172.21.15.37:0/3719748715 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff99c13b3e0 con 0x7ff99c135660 2024-04-26T00:34:32.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.358+0000 7ff9837fe640 1 --2- 172.21.15.37:0/3719748715 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff988035e30 0x7ff9880382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:32.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.358+0000 7ff9837fe640 1 -- 172.21.15.37:0/3719748715 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(565..565 src has 370..565) ==== 3959+0+0 (secure 0 0 0) 0x7ff9940e2570 con 0x7ff99c135660 2024-04-26T00:34:32.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.358+0000 7ff99affd640 1 --2- 172.21.15.37:0/3719748715 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff988035e30 0x7ff9880382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:32.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.358+0000 7ff99affd640 10 monclient: get_auth_request con 0x7ff988035e30 auth_method 0 2024-04-26T00:34:32.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.358+0000 7ff99affd640 1 --2- 172.21.15.37:0/3719748715 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff988035e30 0x7ff9880382f0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff98c003d10 tx=0x7ff98c009270 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:32.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.358+0000 7ff9837fe640 1 -- 172.21.15.37:0/3719748715 DispatchQueue::entry inject delay of 0.002000 2024-04-26T00:34:32.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.362+0000 7ff9837fe640 1 -- 172.21.15.37:0/3719748715 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff9940c3070 con 0x7ff99c135660 2024-04-26T00:34:32.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.362+0000 7ff9837fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:34:32.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.362+0000 7ff9837fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:34:32.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.586+0000 7ff9a143d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:34:32.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.586+0000 7ff9a143d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:34:32.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.586+0000 7ff9a143d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:34:32.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.586+0000 7ff9a143d640 1 -- 172.21.15.37:0/3719748715 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff99c066f40 con 0x7ff99c135660 2024-04-26T00:34:32.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.586+0000 7ff9837fe640 1 -- 172.21.15.37:0/3719748715 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1558 (secure 0 0 0) 0x7ff99c0a1130 con 0x7ff99c135660 2024-04-26T00:34:32.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.586+0000 7ff9837fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:34:32.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.586+0000 7ff9837fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:34:32.582 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":108,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:34:32.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.590+0000 7ff9a143d640 1 -- 172.21.15.37:0/3719748715 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff988035e30 msgr2=0x7ff9880382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:32.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.590+0000 7ff9a143d640 1 --2- 172.21.15.37:0/3719748715 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff988035e30 0x7ff9880382f0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff98c003d10 tx=0x7ff98c009270 comp rx=0 tx=0).stop 2024-04-26T00:34:32.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.590+0000 7ff9a143d640 10 monclient: shutdown 2024-04-26T00:34:32.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.590+0000 7ff9a143d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:34:32.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.590+0000 7ff9a143d640 1 -- 172.21.15.37:0/3719748715 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff99c135660 msgr2=0x7ff99c137c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:32.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.590+0000 7ff9a143d640 1 --2- 172.21.15.37:0/3719748715 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff99c135660 0x7ff99c137c70 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff9940af040 tx=0x7ff994018f00 comp rx=0 tx=0).stop 2024-04-26T00:34:32.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.590+0000 7ff9a143d640 1 -- 172.21.15.37:0/3719748715 shutdown_connections 2024-04-26T00:34:32.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.590+0000 7ff9a143d640 1 --2- 172.21.15.37:0/3719748715 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff988035e30 0x7ff9880382f0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:32.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.590+0000 7ff9a143d640 1 --2- 172.21.15.37:0/3719748715 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff99c135660 0x7ff99c137c70 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:32.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.590+0000 7ff9a143d640 1 --2- 172.21.15.37:0/3719748715 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff99c134b30 0x7ff99c137730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:32.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.590+0000 7ff9a143d640 1 --2- 172.21.15.37:0/3719748715 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff99c0a0260 0x7ff99c13bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:32.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.590+0000 7ff9a143d640 1 -- 172.21.15.37:0/3719748715 >> 172.21.15.37:0/3719748715 conn(0x7ff99c0a1ad0 msgr2=0x7ff99c0a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:34:32.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.590+0000 7ff9a143d640 1 -- 172.21.15.37:0/3719748715 shutdown_connections 2024-04-26T00:34:32.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:32.590+0000 7ff9a143d640 1 -- 172.21.15.37:0/3719748715 wait complete. 2024-04-26T00:34:35.603 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:34:35.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:35.738+0000 7fbaaca7c640 10 monclient: get_monmap_and_config 2024-04-26T00:34:35.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:35.738+0000 7fbaaca7c640 10 monclient: build_initial_monmap 2024-04-26T00:34:35.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:35.738+0000 7fbaaca7c640 10 monclient: monmap: 2024-04-26T00:34:35.736 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:34:35.736 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:35.736 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:34:35.742309+0000 2024-04-26T00:34:35.736 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:34:35.742309+0000 2024-04-26T00:34:35.736 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:34:35.736 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:35.736 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:34:35.736 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:34:35.736 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:34:35.736 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:34:35.736 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:34:35.736 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:35.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:35.738+0000 7fbaaca7c640 1 Processor -- start 2024-04-26T00:34:35.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:35.738+0000 7fbaaca7c640 1 -- start start 2024-04-26T00:34:35.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:35.738+0000 7fbaaca7c640 10 monclient: init 2024-04-26T00:34:35.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:35.738+0000 7fbaaca7c640 10 monclient: _reopen_session rank -1 2024-04-26T00:34:35.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:35.742+0000 7fbaaca7c640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-04-26T00:34:35.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:35.742+0000 7fbaaca7c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbaa80a0260 addr v1:172.21.15.37:6789/0 2024-04-26T00:34:35.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:35.742+0000 7fbaaca7c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fbaa8137e40 0x7fbaa813a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:35.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:35.742+0000 7fbaaca7c640 10 monclient(hunting): picked mon.noname-c con 0x7fbaa8137e40 addr v2:172.21.15.45:3300/0 2024-04-26T00:34:35.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:35.742+0000 7fbaaca7c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbaa813a770 addr v1:172.21.15.45:6789/0 2024-04-26T00:34:35.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:35.742+0000 7fbaaca7c640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbaa80a1130 con 0x7fbaa80a0260 2024-04-26T00:34:35.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:35.742+0000 7fbaaca7c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbaa806cff0 con 0x7fbaa813a770 2024-04-26T00:34:35.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:35.742+0000 7fbaaca7c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:35.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:35.742+0000 7fbaaca7c640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fbaa80a0d50 con 0x7fbaa8137e40 2024-04-26T00:34:35.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:35.742+0000 7fbaaca7c640 10 monclient(hunting): _renew_subs 2024-04-26T00:34:35.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:35.742+0000 7fbaaca7c640 10 monclient(hunting): authenticate will time out at 1857.048156s 2024-04-26T00:34:35.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:35.742+0000 7fbaa659c640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fbaa80a0260 0x7fbaa80a0640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:50304/0 (socket says 172.21.15.37:50304) 2024-04-26T00:34:35.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:35.742+0000 7fbaa659c640 1 -- 172.21.15.37:0/748420039 learned_addr learned my addr 172.21.15.37:0/748420039 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:34:35.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:35.742+0000 7fbaa5d9b640 1 --2- 172.21.15.37:0/748420039 >> v2:172.21.15.45:3300/0 conn(0x7fbaa8137e40 0x7fbaa813a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:35.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:35.742+0000 7fbaa5d9b640 10 monclient(hunting): get_auth_request con 0x7fbaa8137e40 auth_method 0 2024-04-26T00:34:35.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:35.742+0000 7fbaa5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:35.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:35.742+0000 7fbaa5d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:35.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:35.742+0000 7fbaa5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:35.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:35.742+0000 7fbaa5d9b640 1 -- 172.21.15.37:0/748420039 >> v2:172.21.15.45:3300/0 conn(0x7fbaa8137e40 msgr2=0x7fbaa813a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:34:35.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:35.742+0000 7fbaa5d9b640 1 -- 172.21.15.37:0/748420039 >> v2:172.21.15.45:3300/0 conn(0x7fbaa8137e40 msgr2=0x7fbaa813a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:34:35.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:35.742+0000 7fbaa5d9b640 1 --2- 172.21.15.37:0/748420039 >> v2:172.21.15.45:3300/0 conn(0x7fbaa8137e40 0x7fbaa813a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:34:35.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:35.742+0000 7fbaa5d9b640 1 --2- 172.21.15.37:0/748420039 >> v2:172.21.15.45:3300/0 conn(0x7fbaa8137e40 0x7fbaa813a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:34:35.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:35.742+0000 7fbaa659c640 1 -- 172.21.15.37:0/748420039 >> v1:172.21.15.37:6789/0 conn(0x7fbaa80a0260 legacy=0x7fbaa80a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:35.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:35.742+0000 7fbaa6d9d640 1 -- 172.21.15.37:0/748420039 >> v1:172.21.15.45:6789/0 conn(0x7fbaa813a770 legacy=0x7fbaa813cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:35.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:35.946+0000 7fbaa5d9b640 1 --2- 172.21.15.37:0/748420039 >> v2:172.21.15.45:3300/0 conn(0x7fbaa8137e40 0x7fbaa813a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:35.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:35.946+0000 7fbaa5d9b640 10 monclient(hunting): get_auth_request con 0x7fbaa8137e40 auth_method 0 2024-04-26T00:34:35.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:35.946+0000 7fbaa5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:35.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:35.946+0000 7fbaa5d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:35.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:35.946+0000 7fbaa5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:34:35.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:35.946+0000 7fbaa5d9b640 1 -- 172.21.15.37:0/748420039 >> v2:172.21.15.45:3300/0 conn(0x7fbaa8137e40 msgr2=0x7fbaa813a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:34:35.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:35.946+0000 7fbaa5d9b640 1 -- 172.21.15.37:0/748420039 >> v2:172.21.15.45:3300/0 conn(0x7fbaa8137e40 msgr2=0x7fbaa813a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:34:35.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:35.946+0000 7fbaa5d9b640 1 --2- 172.21.15.37:0/748420039 >> v2:172.21.15.45:3300/0 conn(0x7fbaa8137e40 0x7fbaa813a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:34:35.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:35.946+0000 7fbaa5d9b640 1 --2- 172.21.15.37:0/748420039 >> v2:172.21.15.45:3300/0 conn(0x7fbaa8137e40 0x7fbaa813a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:34:36.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:36.350+0000 7fbaa5d9b640 1 --2- 172.21.15.37:0/748420039 >> v2:172.21.15.45:3300/0 conn(0x7fbaa8137e40 0x7fbaa813a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:36.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:36.350+0000 7fbaa5d9b640 10 monclient(hunting): get_auth_request con 0x7fbaa8137e40 auth_method 0 2024-04-26T00:34:36.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:36.350+0000 7fbaa5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:36.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:36.350+0000 7fbaa5d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:36.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:36.350+0000 7fbaa5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:34:36.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:36.350+0000 7fbaa5d9b640 1 -- 172.21.15.37:0/748420039 >> v2:172.21.15.45:3300/0 conn(0x7fbaa8137e40 msgr2=0x7fbaa813a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:34:36.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:36.350+0000 7fbaa5d9b640 1 -- 172.21.15.37:0/748420039 >> v2:172.21.15.45:3300/0 conn(0x7fbaa8137e40 msgr2=0x7fbaa813a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:34:36.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:36.350+0000 7fbaa5d9b640 1 --2- 172.21.15.37:0/748420039 >> v2:172.21.15.45:3300/0 conn(0x7fbaa8137e40 0x7fbaa813a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:34:36.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:36.350+0000 7fbaa5d9b640 1 --2- 172.21.15.37:0/748420039 >> v2:172.21.15.45:3300/0 conn(0x7fbaa8137e40 0x7fbaa813a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:34:37.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:37.154+0000 7fbaa5d9b640 1 --2- 172.21.15.37:0/748420039 >> v2:172.21.15.45:3300/0 conn(0x7fbaa8137e40 0x7fbaa813a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:37.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:37.154+0000 7fbaa5d9b640 10 monclient(hunting): get_auth_request con 0x7fbaa8137e40 auth_method 0 2024-04-26T00:34:37.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:37.154+0000 7fbaa5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:37.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:37.154+0000 7fbaa5d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:37.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:37.154+0000 7fbaa5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:34:37.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:37.154+0000 7fbaa5d9b640 1 -- 172.21.15.37:0/748420039 >> v2:172.21.15.45:3300/0 conn(0x7fbaa8137e40 msgr2=0x7fbaa813a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:34:37.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:37.154+0000 7fbaa5d9b640 1 -- 172.21.15.37:0/748420039 >> v2:172.21.15.45:3300/0 conn(0x7fbaa8137e40 msgr2=0x7fbaa813a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:34:37.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:37.154+0000 7fbaa5d9b640 1 --2- 172.21.15.37:0/748420039 >> v2:172.21.15.45:3300/0 conn(0x7fbaa8137e40 0x7fbaa813a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:34:37.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:37.154+0000 7fbaa5d9b640 1 --2- 172.21.15.37:0/748420039 >> v2:172.21.15.45:3300/0 conn(0x7fbaa8137e40 0x7fbaa813a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:34:38.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:38.742+0000 7fba9ffff640 10 monclient(hunting): tick 2024-04-26T00:34:38.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:38.742+0000 7fba9ffff640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:34:38.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:38.742+0000 7fba9ffff640 1 monclient(hunting): continuing hunt 2024-04-26T00:34:38.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:38.742+0000 7fba9ffff640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:34:38.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:38.742+0000 7fba9ffff640 1 -- 172.21.15.37:0/748420039 >> v2:172.21.15.45:3300/0 conn(0x7fbaa8137e40 msgr2=0x7fbaa813a230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:34:38.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:38.742+0000 7fba9ffff640 1 --2- 172.21.15.37:0/748420039 >> v2:172.21.15.45:3300/0 conn(0x7fbaa8137e40 0x7fbaa813a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:38.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:38.742+0000 7fba9ffff640 1 -- 172.21.15.37:0/748420039 >> v1:172.21.15.45:6789/0 conn(0x7fbaa813a770 legacy=0x7fbaa813cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:38.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:38.742+0000 7fba9ffff640 1 -- 172.21.15.37:0/748420039 >> v1:172.21.15.37:6789/0 conn(0x7fbaa80a0260 legacy=0x7fbaa80a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:38.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:38.742+0000 7fba9ffff640 10 monclient: _add_conns ranks=[2,3] 2024-04-26T00:34:38.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:38.742+0000 7fba9ffff640 1 --2- 172.21.15.37:0/748420039 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fba88003840 0x7fba88005c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:38.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:38.742+0000 7fba9ffff640 10 monclient(hunting): picked mon.noname-b con 0x7fba88003840 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:34:38.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:38.742+0000 7fba9ffff640 1 --2- 172.21.15.37:0/748420039 >> v2:172.21.15.37:3300/0 conn(0x7fba880063c0 0x7fba88008880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:38.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:38.742+0000 7fba9ffff640 10 monclient(hunting): picked mon.noname-a con 0x7fba880063c0 addr v2:172.21.15.37:3300/0 2024-04-26T00:34:38.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:38.742+0000 7fba9ffff640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:38.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:38.742+0000 7fba9ffff640 1 -- 172.21.15.37:0/748420039 --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fbaa80a0d50 con 0x7fba880063c0 2024-04-26T00:34:38.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:38.742+0000 7fba9ffff640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:38.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:38.742+0000 7fba9ffff640 1 -- 172.21.15.37:0/748420039 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fba88008f40 con 0x7fba88003840 2024-04-26T00:34:38.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:38.742+0000 7fba9ffff640 10 monclient(hunting): _renew_subs 2024-04-26T00:34:38.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:38.742+0000 7fbaa659c640 1 -- 172.21.15.37:0/748420039 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fba88003840 msgr2=0x7fba88005c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:34:38.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:38.742+0000 7fbaa659c640 1 --2- 172.21.15.37:0/748420039 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fba88003840 0x7fba88005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:34:38.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:38.742+0000 7fbaa5d9b640 1 --2- 172.21.15.37:0/748420039 >> v2:172.21.15.37:3300/0 conn(0x7fba880063c0 0x7fba88008880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:38.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:38.742+0000 7fbaa5d9b640 10 monclient(hunting): get_auth_request con 0x7fba880063c0 auth_method 0 2024-04-26T00:34:38.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:38.742+0000 7fbaa5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:38.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:38.742+0000 7fbaa5d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:38.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:38.742+0000 7fbaa5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:38.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:38.742+0000 7fbaa5d9b640 1 -- 172.21.15.37:0/748420039 >> v2:172.21.15.37:3300/0 conn(0x7fba880063c0 msgr2=0x7fba88008880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:34:38.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:38.742+0000 7fbaa5d9b640 1 -- 172.21.15.37:0/748420039 >> v2:172.21.15.37:3300/0 conn(0x7fba880063c0 msgr2=0x7fba88008880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:34:38.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:38.742+0000 7fbaa5d9b640 1 --2- 172.21.15.37:0/748420039 >> v2:172.21.15.37:3300/0 conn(0x7fba880063c0 0x7fba88008880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:34:38.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:38.742+0000 7fbaa5d9b640 1 --2- 172.21.15.37:0/748420039 >> v2:172.21.15.37:3300/0 conn(0x7fba880063c0 0x7fba88008880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:34:38.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:38.946+0000 7fbaa659c640 1 -- 172.21.15.37:0/748420039 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fba88003840 msgr2=0x7fba88005c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:34:38.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:38.946+0000 7fbaa659c640 1 --2- 172.21.15.37:0/748420039 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fba88003840 0x7fba88005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:34:38.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:38.946+0000 7fbaa5d9b640 1 --2- 172.21.15.37:0/748420039 >> v2:172.21.15.37:3300/0 conn(0x7fba880063c0 0x7fba88008880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:38.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:38.946+0000 7fbaa5d9b640 10 monclient(hunting): get_auth_request con 0x7fba880063c0 auth_method 0 2024-04-26T00:34:38.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:38.946+0000 7fbaa5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:38.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:38.946+0000 7fbaa5d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:38.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:38.946+0000 7fbaa5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:34:38.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:38.946+0000 7fbaa5d9b640 1 -- 172.21.15.37:0/748420039 >> v2:172.21.15.37:3300/0 conn(0x7fba880063c0 msgr2=0x7fba88008880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:34:38.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:38.946+0000 7fbaa5d9b640 1 -- 172.21.15.37:0/748420039 >> v2:172.21.15.37:3300/0 conn(0x7fba880063c0 msgr2=0x7fba88008880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:34:38.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:38.946+0000 7fbaa5d9b640 1 --2- 172.21.15.37:0/748420039 >> v2:172.21.15.37:3300/0 conn(0x7fba880063c0 0x7fba88008880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:34:38.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:38.946+0000 7fbaa5d9b640 1 --2- 172.21.15.37:0/748420039 >> v2:172.21.15.37:3300/0 conn(0x7fba880063c0 0x7fba88008880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:34:39.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:39.350+0000 7fbaa659c640 1 -- 172.21.15.37:0/748420039 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fba88003840 msgr2=0x7fba88005c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:34:39.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:39.350+0000 7fbaa659c640 1 --2- 172.21.15.37:0/748420039 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fba88003840 0x7fba88005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:34:39.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:39.350+0000 7fbaa5d9b640 1 --2- 172.21.15.37:0/748420039 >> v2:172.21.15.37:3300/0 conn(0x7fba880063c0 0x7fba88008880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:39.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:39.350+0000 7fbaa5d9b640 10 monclient(hunting): get_auth_request con 0x7fba880063c0 auth_method 0 2024-04-26T00:34:39.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:39.350+0000 7fbaa5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:39.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:39.350+0000 7fbaa5d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:39.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:39.350+0000 7fbaa5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:34:39.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:39.350+0000 7fbaa5d9b640 1 -- 172.21.15.37:0/748420039 >> v2:172.21.15.37:3300/0 conn(0x7fba880063c0 msgr2=0x7fba88008880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:34:39.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:39.350+0000 7fbaa5d9b640 1 -- 172.21.15.37:0/748420039 >> v2:172.21.15.37:3300/0 conn(0x7fba880063c0 msgr2=0x7fba88008880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:34:39.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:39.350+0000 7fbaa5d9b640 1 --2- 172.21.15.37:0/748420039 >> v2:172.21.15.37:3300/0 conn(0x7fba880063c0 0x7fba88008880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:34:39.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:39.350+0000 7fbaa5d9b640 1 --2- 172.21.15.37:0/748420039 >> v2:172.21.15.37:3300/0 conn(0x7fba880063c0 0x7fba88008880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:34:40.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.154+0000 7fbaa659c640 1 -- 172.21.15.37:0/748420039 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fba88003840 msgr2=0x7fba88005c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:34:40.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.154+0000 7fbaa659c640 1 --2- 172.21.15.37:0/748420039 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fba88003840 0x7fba88005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:34:40.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.154+0000 7fbaa5d9b640 1 --2- 172.21.15.37:0/748420039 >> v2:172.21.15.37:3300/0 conn(0x7fba880063c0 0x7fba88008880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:40.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.154+0000 7fbaa5d9b640 10 monclient(hunting): get_auth_request con 0x7fba880063c0 auth_method 0 2024-04-26T00:34:40.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.154+0000 7fbaa5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:40.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.154+0000 7fbaa5d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:40.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.154+0000 7fbaa5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:34:40.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.154+0000 7fbaa5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:40.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.154+0000 7fbaa5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:40.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.154+0000 7fbaa5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:40.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.154+0000 7fbaa5d9b640 10 monclient(hunting): handle_auth_done global_id 144456 payload 274 2024-04-26T00:34:40.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.154+0000 7fbaa5d9b640 1 -- 172.21.15.37:0/748420039 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fba88003840 msgr2=0x7fba88005c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:34:40.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.154+0000 7fbaa5d9b640 1 --2- 172.21.15.37:0/748420039 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fba88003840 0x7fba88005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:40.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.154+0000 7fbaa5d9b640 10 monclient: _finish_hunting 0 2024-04-26T00:34:40.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.154+0000 7fbaa5d9b640 1 monclient: found mon.noname-a 2024-04-26T00:34:40.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.154+0000 7fbaa5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:34:40.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.154+0000 7fbaa5d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:34:40.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.154+0000 7fbaa5d9b640 1 -- 172.21.15.37:0/748420039 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaa809bc50 con 0x7fba880063c0 2024-04-26T00:34:40.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.162+0000 7fbaa5d9b640 1 -- 172.21.15.37:0/748420039 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba880063c0 msgr2=0x7fba88008880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:40.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.162+0000 7fbaa5d9b640 1 --2- 172.21.15.37:0/748420039 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba880063c0 0x7fba88008880 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fba90000ee0 tx=0x7fba9001da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae6f50896266645d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:40.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.162+0000 7fbaa559a640 1 -- 172.21.15.37:0/748420039 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba9000f040 con 0x7fba880063c0 2024-04-26T00:34:40.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.162+0000 7fbaa559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:40.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.162+0000 7fbaa559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:34:40.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.162+0000 7fbaa559a640 10 monclient: dump: 2024-04-26T00:34:40.158 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:40.158 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:40.159 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:40.159 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:40.159 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:40.159 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:40.159 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:40.159 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:40.159 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:40.159 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:40.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.162+0000 7fbaa559a640 10 monclient: _finish_auth 0 2024-04-26T00:34:40.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.162+0000 7fbaa559a640 10 monclient: _check_auth_tickets 2024-04-26T00:34:40.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.162+0000 7fbaa559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:34:40.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.162+0000 7fbaa559a640 1 -- 172.21.15.37:0/748420039 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba900a68d0 con 0x7fba880063c0 2024-04-26T00:34:40.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.162+0000 7fbaa559a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:34:40.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.162+0000 7fbaa559a640 1 -- 172.21.15.37:0/748420039 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba900a6cc0 con 0x7fba880063c0 2024-04-26T00:34:40.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.162+0000 7fbaa559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:40.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.162+0000 7fbaa559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:34:40.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.162+0000 7fbaa559a640 10 monclient: dump: 2024-04-26T00:34:40.160 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:40.160 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:40.160 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:40.160 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:40.160 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:40.160 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:40.160 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:40.160 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:40.160 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:40.160 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:40.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.162+0000 7fbaaca7c640 5 monclient: authenticate success, global_id 144456 2024-04-26T00:34:40.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.162+0000 7fbaaca7c640 10 monclient: get_monmap_and_config success 2024-04-26T00:34:40.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.162+0000 7fbaaca7c640 4 set_mon_vals no callback set 2024-04-26T00:34:40.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.162+0000 7fbaaca7c640 10 monclient: shutdown 2024-04-26T00:34:40.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.162+0000 7fbaaca7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:34:40.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.162+0000 7fbaaca7c640 1 -- 172.21.15.37:0/748420039 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba880063c0 msgr2=0x7fba88008880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:40.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.162+0000 7fbaaca7c640 1 --2- 172.21.15.37:0/748420039 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba880063c0 0x7fba88008880 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fba90000ee0 tx=0x7fba9001da90 comp rx=0 tx=0).stop 2024-04-26T00:34:40.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.162+0000 7fbaa6d9d640 1 -- 172.21.15.37:0/748420039 reap_dead start 2024-04-26T00:34:40.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.162+0000 7fbaaca7c640 1 -- 172.21.15.37:0/748420039 shutdown_connections 2024-04-26T00:34:40.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.162+0000 7fbaaca7c640 1 --2- 172.21.15.37:0/748420039 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba880063c0 0x7fba88008880 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:40.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.162+0000 7fbaaca7c640 1 -- 172.21.15.37:0/748420039 >> 172.21.15.37:0/748420039 conn(0x7fbaa80a1ad0 msgr2=0x7fbaa80a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:34:40.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.162+0000 7fbaaca7c640 1 -- 172.21.15.37:0/748420039 shutdown_connections 2024-04-26T00:34:40.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.162+0000 7fbaaca7c640 1 -- 172.21.15.37:0/748420039 wait complete. 2024-04-26T00:34:40.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.166+0000 7fbaaca7c640 10 monclient: build_initial_monmap 2024-04-26T00:34:40.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.166+0000 7fbaaca7c640 10 monclient: monmap: 2024-04-26T00:34:40.162 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:34:40.163 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:34:40.163 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:34:40.163 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:34:40.163 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:34:40.163 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:34:40.163 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:34:40.163 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:34:40.163 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:34:40.163 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:40.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.166+0000 7fbaaca7c640 1 Processor -- start 2024-04-26T00:34:40.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.166+0000 7fbaaca7c640 1 -- start start 2024-04-26T00:34:40.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.166+0000 7fbaaca7c640 10 monclient: init 2024-04-26T00:34:40.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.166+0000 7fbaaca7c640 10 monclient: _reopen_session rank -1 2024-04-26T00:34:40.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.166+0000 7fbaaca7c640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:34:40.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.166+0000 7fbaaca7c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba880063c0 0x7fbaa81ced40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:40.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.166+0000 7fbaaca7c640 10 monclient(hunting): picked mon.noname-a con 0x7fba880063c0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:34:40.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.166+0000 7fbaaca7c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbaa81319b0 0x7fbaa8131e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:40.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.166+0000 7fbaaca7c640 10 monclient(hunting): picked mon.noname-c con 0x7fbaa81319b0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:34:40.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.166+0000 7fbaaca7c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaa8070060 0x7fbaa80704e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:40.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.166+0000 7fbaaca7c640 10 monclient(hunting): picked mon.noname-b con 0x7fbaa8070060 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:34:40.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.166+0000 7fbaaca7c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:40.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.166+0000 7fbaaca7c640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fbaa809f4c0 con 0x7fba880063c0 2024-04-26T00:34:40.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.166+0000 7fbaaca7c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:40.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.166+0000 7fbaaca7c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbaa809f0d0 con 0x7fbaa8070060 2024-04-26T00:34:40.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.166+0000 7fbaaca7c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:40.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.166+0000 7fbaaca7c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fbaa80a00a0 con 0x7fbaa81319b0 2024-04-26T00:34:40.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.166+0000 7fbaaca7c640 10 monclient(hunting): _renew_subs 2024-04-26T00:34:40.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.166+0000 7fbaa5d9b640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbaa81319b0 msgr2=0x7fbaa8131e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:34:40.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.166+0000 7fbaaca7c640 10 monclient(hunting): authenticate will time out at 1861.475880s 2024-04-26T00:34:40.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.166+0000 7fbaa5d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbaa81319b0 0x7fbaa8131e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:34:40.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fbaa659c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba880063c0 0x7fbaa81ced40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:40.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fbaa659c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba880063c0 0x7fbaa81ced40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:43792/0 (socket says 172.21.15.37:43792) 2024-04-26T00:34:40.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fbaa659c640 1 -- 172.21.15.37:0/2565391890 learned_addr learned my addr 172.21.15.37:0/2565391890 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:34:40.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fbaa659c640 10 monclient(hunting): get_auth_request con 0x7fba880063c0 auth_method 0 2024-04-26T00:34:40.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fbaa659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:40.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fbaa659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:40.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fbaa659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:40.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fbaa659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:40.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fbaa659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:40.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fbaa659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:40.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fbaa6d9d640 1 --2- 172.21.15.37:0/2565391890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaa8070060 0x7fbaa80704e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:40.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fbaa6d9d640 10 monclient(hunting): get_auth_request con 0x7fbaa8070060 auth_method 0 2024-04-26T00:34:40.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fbaa6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:40.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fbaa6d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:40.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fbaa6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:40.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fbaa659c640 10 monclient(hunting): handle_auth_done global_id 144459 payload 824 2024-04-26T00:34:40.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fbaa659c640 1 -- 172.21.15.37:0/2565391890 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbaa81319b0 msgr2=0x7fbaa8131e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:34:40.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fbaa659c640 1 --2- 172.21.15.37:0/2565391890 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbaa81319b0 0x7fbaa8131e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:40.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fbaa659c640 1 -- 172.21.15.37:0/2565391890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaa8070060 msgr2=0x7fbaa80704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:40.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fbaa659c640 1 --2- 172.21.15.37:0/2565391890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaa8070060 0x7fbaa80704e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:40.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fbaa659c640 10 monclient: _finish_hunting 0 2024-04-26T00:34:40.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fbaa659c640 1 monclient: found mon.noname-a 2024-04-26T00:34:40.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fbaa659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:34:40.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fbaa659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:34:40.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fbaa659c640 1 -- 172.21.15.37:0/2565391890 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaa80a4130 con 0x7fba880063c0 2024-04-26T00:34:40.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fbaa659c640 1 -- 172.21.15.37:0/2565391890 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba880063c0 msgr2=0x7fbaa81ced40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:40.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fbaa659c640 1 --2- 172.21.15.37:0/2565391890 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba880063c0 0x7fbaa81ced40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbaa000ce20 tx=0x7fbaa000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:40.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fba9effd640 1 -- 172.21.15.37:0/2565391890 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaa0017070 con 0x7fba880063c0 2024-04-26T00:34:40.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fba9effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:40.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fba9effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:34:40.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fba9effd640 10 monclient: dump: 2024-04-26T00:34:40.170 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:40.170 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:40.170 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:40.171 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:40.171 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:40.171 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:40.171 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:40.171 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:40.171 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:40.171 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:40.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fba9effd640 10 monclient: _finish_auth 0 2024-04-26T00:34:40.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fba9effd640 10 monclient: _check_auth_tickets 2024-04-26T00:34:40.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fba9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:34:40.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fba9effd640 1 -- 172.21.15.37:0/2565391890 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbaa0012600 con 0x7fba880063c0 2024-04-26T00:34:40.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fba9effd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:34:40.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fba9effd640 1 -- 172.21.15.37:0/2565391890 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaa00129b0 con 0x7fba880063c0 2024-04-26T00:34:40.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fba9effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:40.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fba9effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:34:40.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fba9effd640 10 monclient: dump: 2024-04-26T00:34:40.172 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:40.172 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:40.172 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:40.172 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:40.172 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:40.172 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:40.172 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:40.172 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:40.172 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:40.172 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:40.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fbaaca7c640 5 monclient: authenticate success, global_id 144459 2024-04-26T00:34:40.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fbaa4d99640 4 set_mon_vals no callback set 2024-04-26T00:34:40.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fbaaca7c640 10 monclient: _renew_subs 2024-04-26T00:34:40.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fbaaca7c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:34:40.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fbaaca7c640 1 -- 172.21.15.37:0/2565391890 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbaa811d350 con 0x7fba880063c0 2024-04-26T00:34:40.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fbaaca7c640 10 monclient: _renew_subs 2024-04-26T00:34:40.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fbaaca7c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:34:40.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.170+0000 7fbaaca7c640 1 -- 172.21.15.37:0/2565391890 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbaa811cab0 con 0x7fba880063c0 2024-04-26T00:34:40.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.174+0000 7fbaaca7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:34:40.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.174+0000 7fbaaca7c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:34:40.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.174+0000 7fbaaca7c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:34:40.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.174+0000 7fba9effd640 1 -- 172.21.15.37:0/2565391890 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fbaa001a070 con 0x7fba880063c0 2024-04-26T00:34:40.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.174+0000 7fbaaca7c640 1 -- 172.21.15.37:0/2565391890 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbaa8066f40 con 0x7fba880063c0 2024-04-26T00:34:40.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.174+0000 7fba9effd640 1 --2- 172.21.15.37:0/2565391890 >> v2:172.21.15.45:6812/2379918623 conn(0x7fba88036e20 0x7fba880392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:40.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.174+0000 7fba9effd640 1 -- 172.21.15.37:0/2565391890 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(565..565 src has 370..565) ==== 3959+0+0 (secure 0 0 0) 0x7fbaa0048900 con 0x7fba880063c0 2024-04-26T00:34:40.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.174+0000 7fbaa5d9b640 1 --2- 172.21.15.37:0/2565391890 >> v2:172.21.15.45:6812/2379918623 conn(0x7fba88036e20 0x7fba880392e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:40.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.174+0000 7fbaa5d9b640 10 monclient: get_auth_request con 0x7fba88036e20 auth_method 0 2024-04-26T00:34:40.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.178+0000 7fbaa5d9b640 1 --2- 172.21.15.37:0/2565391890 >> v2:172.21.15.45:6812/2379918623 conn(0x7fba88036e20 0x7fba880392e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fba900ad000 tx=0x7fba900a8070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:40.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.182+0000 7fba9effd640 1 -- 172.21.15.37:0/2565391890 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fbaa80a4130 con 0x7fba880063c0 2024-04-26T00:34:40.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.182+0000 7fba9effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:34:40.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.182+0000 7fba9effd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:34:40.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.402+0000 7fbaaca7c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:34:40.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.402+0000 7fbaaca7c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:34:40.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.402+0000 7fbaaca7c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:34:40.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.402+0000 7fbaaca7c640 1 -- 172.21.15.37:0/2565391890 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fbaa813c260 con 0x7fba880063c0 2024-04-26T00:34:40.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.402+0000 7fba9effd640 1 -- 172.21.15.37:0/2565391890 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7fbaa80a1130 con 0x7fba880063c0 2024-04-26T00:34:40.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.402+0000 7fba9effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:34:40.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.402+0000 7fba9effd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:34:40.399 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":112,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:34:40.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.402+0000 7fbaaca7c640 1 -- 172.21.15.37:0/2565391890 >> v2:172.21.15.45:6812/2379918623 conn(0x7fba88036e20 msgr2=0x7fba880392e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:40.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.402+0000 7fbaaca7c640 1 --2- 172.21.15.37:0/2565391890 >> v2:172.21.15.45:6812/2379918623 conn(0x7fba88036e20 0x7fba880392e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fba900ad000 tx=0x7fba900a8070 comp rx=0 tx=0).stop 2024-04-26T00:34:40.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.402+0000 7fbaaca7c640 10 monclient: shutdown 2024-04-26T00:34:40.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.402+0000 7fbaaca7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:34:40.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.402+0000 7fbaaca7c640 1 -- 172.21.15.37:0/2565391890 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba880063c0 msgr2=0x7fbaa81ced40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:40.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.402+0000 7fbaaca7c640 1 --2- 172.21.15.37:0/2565391890 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba880063c0 0x7fbaa81ced40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbaa000ce20 tx=0x7fbaa000a430 comp rx=0 tx=0).stop 2024-04-26T00:34:40.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.402+0000 7fbaaca7c640 1 -- 172.21.15.37:0/2565391890 shutdown_connections 2024-04-26T00:34:40.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.402+0000 7fbaaca7c640 1 --2- 172.21.15.37:0/2565391890 >> v2:172.21.15.45:6812/2379918623 conn(0x7fba88036e20 0x7fba880392e0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:40.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.402+0000 7fbaaca7c640 1 --2- 172.21.15.37:0/2565391890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaa8070060 0x7fbaa80704e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:40.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.402+0000 7fbaaca7c640 1 --2- 172.21.15.37:0/2565391890 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbaa81319b0 0x7fbaa8131e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:40.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.402+0000 7fbaaca7c640 1 --2- 172.21.15.37:0/2565391890 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba880063c0 0x7fbaa81ced40 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:40.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.402+0000 7fbaaca7c640 1 -- 172.21.15.37:0/2565391890 >> 172.21.15.37:0/2565391890 conn(0x7fbaa80a1ad0 msgr2=0x7fbaa809d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:34:40.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.406+0000 7fbaaca7c640 1 -- 172.21.15.37:0/2565391890 shutdown_connections 2024-04-26T00:34:40.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.406+0000 7fbaaca7c640 1 -- 172.21.15.37:0/2565391890 wait complete. 2024-04-26T00:34:40.417 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-26T00:34:40.417 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-04-26T00:34:40.417 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:34:40.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.530+0000 7f89c2afb640 10 monclient: get_monmap_and_config 2024-04-26T00:34:40.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.530+0000 7f89c2afb640 10 monclient: build_initial_monmap 2024-04-26T00:34:40.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.530+0000 7f89c2afb640 10 monclient: monmap: 2024-04-26T00:34:40.529 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:34:40.529 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:40.529 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:34:40.535841+0000 2024-04-26T00:34:40.529 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:34:40.535841+0000 2024-04-26T00:34:40.530 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:34:40.530 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:40.530 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:34:40.530 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:34:40.530 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:34:40.530 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:34:40.530 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:34:40.530 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:40.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c2afb640 1 Processor -- start 2024-04-26T00:34:40.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c2afb640 1 -- start start 2024-04-26T00:34:40.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c2afb640 10 monclient: init 2024-04-26T00:34:40.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c2afb640 10 monclient: _reopen_session rank -1 2024-04-26T00:34:40.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c2afb640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-04-26T00:34:40.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c2afb640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f89bc130eb0 0x7f89bc09ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:40.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c2afb640 10 monclient(hunting): picked mon.noname-b con 0x7f89bc130eb0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:34:40.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c2afb640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f89bc131290 0x7f89bc133b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:40.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c2afb640 10 monclient(hunting): picked mon.noname-c con 0x7f89bc131290 addr v2:172.21.15.45:3300/0 2024-04-26T00:34:40.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c2afb640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f89bc1340d0 0x7f89bc134580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:40.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c2afb640 10 monclient(hunting): picked mon.noname-a con 0x7f89bc1340d0 addr v2:172.21.15.37:3300/0 2024-04-26T00:34:40.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c2afb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:40.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c2afb640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f89bc09df60 con 0x7f89bc1340d0 2024-04-26T00:34:40.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c2afb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:40.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c2afb640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f89bc09c190 con 0x7f89bc131290 2024-04-26T00:34:40.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c2afb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:40.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c2afb640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f89bc0a11c0 con 0x7f89bc130eb0 2024-04-26T00:34:40.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c2afb640 10 monclient(hunting): _renew_subs 2024-04-26T00:34:40.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c2afb640 10 monclient(hunting): authenticate will time out at 1861.841433s 2024-04-26T00:34:40.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c0897640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f89bc130eb0 msgr2=0x7f89bc09ef20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:34:40.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c0897640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f89bc130eb0 0x7f89bc09ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:34:40.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c1098640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f89bc1340d0 0x7f89bc134580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:40.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c1098640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f89bc1340d0 0x7f89bc134580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:43804/0 (socket says 172.21.15.37:43804) 2024-04-26T00:34:40.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c1098640 1 -- 172.21.15.37:0/1808082726 learned_addr learned my addr 172.21.15.37:0/1808082726 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:34:40.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c1098640 10 monclient(hunting): get_auth_request con 0x7f89bc1340d0 auth_method 0 2024-04-26T00:34:40.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c1098640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:40.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c1098640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:40.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c1098640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:40.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c1098640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:40.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c1098640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:40.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c1098640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:40.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89bbfff640 1 --2- 172.21.15.37:0/1808082726 >> v2:172.21.15.45:3300/0 conn(0x7f89bc131290 0x7f89bc133b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:40.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89bbfff640 10 monclient(hunting): get_auth_request con 0x7f89bc131290 auth_method 0 2024-04-26T00:34:40.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:40.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89bbfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:40.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:40.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c1098640 10 monclient(hunting): handle_auth_done global_id 144462 payload 274 2024-04-26T00:34:40.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c1098640 1 -- 172.21.15.37:0/1808082726 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f89bc130eb0 msgr2=0x7f89bc09ef20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:34:40.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c1098640 1 --2- 172.21.15.37:0/1808082726 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f89bc130eb0 0x7f89bc09ef20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:40.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c1098640 1 -- 172.21.15.37:0/1808082726 >> v2:172.21.15.45:3300/0 conn(0x7f89bc131290 msgr2=0x7f89bc133b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:34:40.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c1098640 1 --2- 172.21.15.37:0/1808082726 >> v2:172.21.15.45:3300/0 conn(0x7f89bc131290 0x7f89bc133b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:40.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c1098640 10 monclient: _finish_hunting 0 2024-04-26T00:34:40.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c1098640 1 monclient: found mon.noname-a 2024-04-26T00:34:40.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c1098640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:34:40.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c1098640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:34:40.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.534+0000 7f89c1098640 1 -- 172.21.15.37:0/1808082726 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89bc09dc80 con 0x7f89bc1340d0 2024-04-26T00:34:40.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.538+0000 7f89c1098640 1 -- 172.21.15.37:0/1808082726 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89bc1340d0 msgr2=0x7f89bc134580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:40.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.538+0000 7f89c1098640 1 --2- 172.21.15.37:0/1808082726 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89bc1340d0 0x7f89bc134580 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f89bc06d330 tx=0x7f89b401ead0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78a10a39c6ca66bf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:40.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.538+0000 7f89bb7fe640 1 -- 172.21.15.37:0/1808082726 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b400b2d0 con 0x7f89bc1340d0 2024-04-26T00:34:40.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.538+0000 7f89bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:40.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.538+0000 7f89bb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:34:40.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.538+0000 7f89bb7fe640 10 monclient: dump: 2024-04-26T00:34:40.544 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:40.544 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:40.545 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:40.545 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:40.545 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:40.545 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:40.545 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:40.545 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:40.545 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:40.545 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:40.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.538+0000 7f89bb7fe640 10 monclient: _finish_auth 0 2024-04-26T00:34:40.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.538+0000 7f89bb7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:34:40.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.538+0000 7f89bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:34:40.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.538+0000 7f89bb7fe640 1 -- 172.21.15.37:0/1808082726 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89b400b470 con 0x7f89bc1340d0 2024-04-26T00:34:40.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.538+0000 7f89bb7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:34:40.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.538+0000 7f89bb7fe640 1 -- 172.21.15.37:0/1808082726 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b40a7990 con 0x7f89bc1340d0 2024-04-26T00:34:40.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.538+0000 7f89bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:40.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.538+0000 7f89bb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:34:40.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.538+0000 7f89bb7fe640 10 monclient: dump: 2024-04-26T00:34:40.546 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:40.546 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:40.546 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:40.546 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:40.546 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:40.547 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:40.547 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:40.547 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:40.547 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:40.547 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:40.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.538+0000 7f89c2afb640 5 monclient: authenticate success, global_id 144462 2024-04-26T00:34:40.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.538+0000 7f89c2afb640 10 monclient: get_monmap_and_config success 2024-04-26T00:34:40.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.538+0000 7f89c2afb640 4 set_mon_vals no callback set 2024-04-26T00:34:40.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.538+0000 7f89c2afb640 10 monclient: shutdown 2024-04-26T00:34:40.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.538+0000 7f89c2afb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:34:40.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.538+0000 7f89c2afb640 1 -- 172.21.15.37:0/1808082726 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89bc1340d0 msgr2=0x7f89bc134580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:40.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.538+0000 7f89c2afb640 1 --2- 172.21.15.37:0/1808082726 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89bc1340d0 0x7f89bc134580 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f89bc06d330 tx=0x7f89b401ead0 comp rx=0 tx=0).stop 2024-04-26T00:34:40.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.538+0000 7f89c2afb640 1 -- 172.21.15.37:0/1808082726 shutdown_connections 2024-04-26T00:34:40.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.538+0000 7f89c2afb640 1 --2- 172.21.15.37:0/1808082726 >> v2:172.21.15.45:3300/0 conn(0x7f89bc131290 0x7f89bc133b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:40.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.538+0000 7f89c2afb640 1 --2- 172.21.15.37:0/1808082726 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89bc1340d0 0x7f89bc134580 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:40.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.538+0000 7f89c2afb640 1 --2- 172.21.15.37:0/1808082726 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f89bc130eb0 0x7f89bc09ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:40.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c2afb640 1 -- 172.21.15.37:0/1808082726 >> 172.21.15.37:0/1808082726 conn(0x7f89bc0a1d50 msgr2=0x7f89bc0a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:34:40.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c2afb640 1 -- 172.21.15.37:0/1808082726 shutdown_connections 2024-04-26T00:34:40.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c2afb640 1 -- 172.21.15.37:0/1808082726 wait complete. 2024-04-26T00:34:40.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c2afb640 10 monclient: build_initial_monmap 2024-04-26T00:34:40.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c2afb640 10 monclient: monmap: 2024-04-26T00:34:40.549 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:34:40.549 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:34:40.549 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:34:40.549 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:34:40.549 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:34:40.549 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:34:40.549 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:34:40.549 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:34:40.549 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:34:40.550 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:40.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c2afb640 1 Processor -- start 2024-04-26T00:34:40.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c2afb640 1 -- start start 2024-04-26T00:34:40.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c2afb640 10 monclient: init 2024-04-26T00:34:40.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c2afb640 10 monclient: _reopen_session rank -1 2024-04-26T00:34:40.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c2afb640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:34:40.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c2afb640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f89bc130eb0 0x7f89bc1cae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:40.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c2afb640 10 monclient(hunting): picked mon.noname-c con 0x7f89bc130eb0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:34:40.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c2afb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89bc131290 0x7f89bc13a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:40.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c2afb640 10 monclient(hunting): picked mon.noname-b con 0x7f89bc131290 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:34:40.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c2afb640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89bc1340d0 0x7f89bc13a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:40.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c2afb640 10 monclient(hunting): picked mon.noname-a con 0x7f89bc1340d0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:34:40.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c2afb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:40.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c2afb640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f89bc0a0460 con 0x7f89bc1340d0 2024-04-26T00:34:40.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c2afb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:40.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c2afb640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f89bc131880 con 0x7f89bc131290 2024-04-26T00:34:40.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c2afb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:40.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c2afb640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f89bc0a05e0 con 0x7f89bc130eb0 2024-04-26T00:34:40.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c2afb640 10 monclient(hunting): _renew_subs 2024-04-26T00:34:40.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c2afb640 10 monclient(hunting): authenticate will time out at 1861.849513s 2024-04-26T00:34:40.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c0897640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f89bc130eb0 msgr2=0x7f89bc1cae50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:34:40.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c0897640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f89bc130eb0 0x7f89bc1cae50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:34:40.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c1098640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89bc1340d0 0x7f89bc13a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:40.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c1098640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89bc1340d0 0x7f89bc13a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:43806/0 (socket says 172.21.15.37:43806) 2024-04-26T00:34:40.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c1098640 1 -- 172.21.15.37:0/3055537316 learned_addr learned my addr 172.21.15.37:0/3055537316 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:34:40.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c1098640 10 monclient(hunting): get_auth_request con 0x7f89bc1340d0 auth_method 0 2024-04-26T00:34:40.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c1098640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:40.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c1098640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:40.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c1098640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:40.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c1098640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:40.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c1098640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:40.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c1098640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:40.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89bbfff640 1 --2- 172.21.15.37:0/3055537316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89bc131290 0x7f89bc13a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:40.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c1098640 10 monclient(hunting): handle_auth_done global_id 144465 payload 824 2024-04-26T00:34:40.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c1098640 1 -- 172.21.15.37:0/3055537316 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f89bc130eb0 msgr2=0x7f89bc1cae50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:34:40.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c1098640 1 --2- 172.21.15.37:0/3055537316 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f89bc130eb0 0x7f89bc1cae50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:40.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c1098640 1 -- 172.21.15.37:0/3055537316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89bc131290 msgr2=0x7f89bc13a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:40.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c1098640 1 --2- 172.21.15.37:0/3055537316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89bc131290 0x7f89bc13a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:40.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c1098640 10 monclient: _finish_hunting 0 2024-04-26T00:34:40.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c1098640 1 monclient: found mon.noname-a 2024-04-26T00:34:40.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c1098640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:34:40.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c1098640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:34:40.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89c1098640 1 -- 172.21.15.37:0/3055537316 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89bc12f2d0 con 0x7f89bc1340d0 2024-04-26T00:34:40.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89bbfff640 10 monclient: get_auth_request con 0x7f89bc131290 auth_method 0 2024-04-26T00:34:40.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.542+0000 7f89bbfff640 1 --2- 172.21.15.37:0/3055537316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89bc131290 0x7f89bc13a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:34:40.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89c1098640 1 -- 172.21.15.37:0/3055537316 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89bc1340d0 msgr2=0x7f89bc13a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:40.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89c1098640 1 --2- 172.21.15.37:0/3055537316 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89bc1340d0 0x7f89bc13a940 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f89b40b1040 tx=0x7f89b401b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:40.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89b97fa640 1 -- 172.21.15.37:0/3055537316 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b40be030 con 0x7f89bc1340d0 2024-04-26T00:34:40.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:40.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89b97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:34:40.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89b97fa640 10 monclient: dump: 2024-04-26T00:34:40.554 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:40.554 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:40.554 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:40.554 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:40.554 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:40.554 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:40.554 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:40.555 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:40.555 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:40.555 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:40.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89b97fa640 10 monclient: _finish_auth 0 2024-04-26T00:34:40.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89b97fa640 10 monclient: _check_auth_tickets 2024-04-26T00:34:40.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:34:40.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89b97fa640 1 -- 172.21.15.37:0/3055537316 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89b4009810 con 0x7f89bc1340d0 2024-04-26T00:34:40.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89b97fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:34:40.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89b97fa640 1 -- 172.21.15.37:0/3055537316 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b4009be0 con 0x7f89bc1340d0 2024-04-26T00:34:40.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:40.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89b97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:34:40.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89b97fa640 10 monclient: dump: 2024-04-26T00:34:40.556 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:40.556 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:40.556 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:40.556 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:40.556 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:40.556 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:40.556 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:40.556 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:40.556 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:40.556 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:40.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89baffd640 4 set_mon_vals no callback set 2024-04-26T00:34:40.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89c2afb640 5 monclient: authenticate success, global_id 144465 2024-04-26T00:34:40.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89c2afb640 10 monclient: _renew_subs 2024-04-26T00:34:40.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89c2afb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:34:40.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89c2afb640 1 -- 172.21.15.37:0/3055537316 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89bc136b70 con 0x7f89bc1340d0 2024-04-26T00:34:40.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89c2afb640 10 monclient: _renew_subs 2024-04-26T00:34:40.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89c2afb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:34:40.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89c2afb640 1 -- 172.21.15.37:0/3055537316 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89bc138ed0 con 0x7f89bc1340d0 2024-04-26T00:34:40.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89c2afb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:34:40.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89c2afb640 1 --2- 172.21.15.37:0/3055537316 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89bc137860 0x7f89bc135120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:40.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89c2afb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:40.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89c2afb640 1 -- 172.21.15.37:0/3055537316 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f89bc0a14e0 con 0x7f89bc137860 2024-04-26T00:34:40.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89c0897640 1 --2- 172.21.15.37:0/3055537316 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89bc137860 0x7f89bc135120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:40.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89c0897640 10 monclient: get_auth_request con 0x7f89bc137860 auth_method 0 2024-04-26T00:34:40.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89c0897640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:40.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89c0897640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:40.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89b97fa640 1 -- 172.21.15.37:0/3055537316 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f89b40c1070 con 0x7f89bc1340d0 2024-04-26T00:34:40.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89c0897640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:40.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89c0897640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:40.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89c0897640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:40.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89c0897640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:40.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89b97fa640 1 --2- 172.21.15.37:0/3055537316 >> v2:172.21.15.45:6812/2379918623 conn(0x7f89a8035e80 0x7f89a8038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:40.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89b97fa640 1 -- 172.21.15.37:0/3055537316 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(565..565 src has 370..565) ==== 3959+0+0 (secure 0 0 0) 0x7f89b40e52c0 con 0x7f89bc1340d0 2024-04-26T00:34:40.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89c0897640 10 monclient(hunting): handle_auth_done global_id 144468 payload 824 2024-04-26T00:34:40.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89c0897640 1 -- 172.21.15.37:0/3055537316 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f89bc04f510 con 0x7f89bc137860 2024-04-26T00:34:40.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89bbfff640 1 --2- 172.21.15.37:0/3055537316 >> v2:172.21.15.45:6812/2379918623 conn(0x7f89a8035e80 0x7f89a8038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:40.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.546+0000 7f89bbfff640 10 monclient: get_auth_request con 0x7f89a8035e80 auth_method 0 2024-04-26T00:34:40.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.550+0000 7f89c0897640 1 -- 172.21.15.37:0/3055537316 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89bc137860 msgr2=0x7f89bc135120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:40.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.550+0000 7f89c0897640 1 --2- 172.21.15.37:0/3055537316 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89bc137860 0x7f89bc135120 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f89ac00b450 tx=0x7f89ac00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:40.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.550+0000 7f89b97fa640 1 -- 172.21.15.37:0/3055537316 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89ac005270 con 0x7f89bc137860 2024-04-26T00:34:40.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.550+0000 7f89b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:40.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.550+0000 7f89b97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:34:40.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.550+0000 7f89b97fa640 10 monclient: dump: 2024-04-26T00:34:40.562 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:40.562 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:40.563 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:40.563 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:40.563 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:40.563 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:40.563 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:40.563 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:40.563 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:40.563 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:40.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.550+0000 7f89b97fa640 1 -- 172.21.15.37:0/3055537316 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f89ac015070 con 0x7f89bc137860 2024-04-26T00:34:40.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.550+0000 7f89b97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:34:40.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.550+0000 7f89b97fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:34:40.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.550+0000 7f89b97fa640 1 -- 172.21.15.37:0/3055537316 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89bc137860 msgr2=0x7f89bc135120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:40.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.550+0000 7f89b97fa640 1 --2- 172.21.15.37:0/3055537316 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89bc137860 0x7f89bc135120 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f89ac00b450 tx=0x7f89ac00b910 comp rx=0 tx=0).stop 2024-04-26T00:34:40.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.550+0000 7f89b97fa640 1 -- 172.21.15.37:0/3055537316 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89bc137860 msgr2=0x7f89bc135120 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:34:40.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.550+0000 7f89b97fa640 1 --2- 172.21.15.37:0/3055537316 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89bc137860 0x7f89bc135120 secure :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f89ac00b450 tx=0x7f89ac00b910 comp rx=0 tx=0).stop 2024-04-26T00:34:40.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.550+0000 7f89bbfff640 1 --2- 172.21.15.37:0/3055537316 >> v2:172.21.15.45:6812/2379918623 conn(0x7f89a8035e80 0x7f89a8038340 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f89bc130760 tx=0x7f89b0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:40.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.554+0000 7f89c2afb640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:34:40.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.554+0000 7f89c2afb640 1 --2- 172.21.15.37:0/3055537316 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89bc135660 0x7f89bc135a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:40.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.554+0000 7f89c2afb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:40.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.554+0000 7f89c2afb640 1 -- 172.21.15.37:0/3055537316 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f89bc135fb0 con 0x7f89bc135660 2024-04-26T00:34:40.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.554+0000 7f89c0897640 1 --2- 172.21.15.37:0/3055537316 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89bc135660 0x7f89bc135a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:40.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.554+0000 7f89c0897640 10 monclient: get_auth_request con 0x7f89bc135660 auth_method 0 2024-04-26T00:34:40.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.554+0000 7f89c0897640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:40.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.554+0000 7f89c0897640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:40.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.554+0000 7f89c0897640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:40.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.554+0000 7f89c0897640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:40.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.554+0000 7f89c0897640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:40.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.554+0000 7f89c0897640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:40.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.554+0000 7f89c0897640 10 monclient(hunting): handle_auth_done global_id 144471 payload 824 2024-04-26T00:34:40.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.554+0000 7f89c0897640 1 -- 172.21.15.37:0/3055537316 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f89bc0a0f60 con 0x7f89bc135660 2024-04-26T00:34:40.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.558+0000 7f89c0897640 1 -- 172.21.15.37:0/3055537316 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89bc135660 msgr2=0x7f89bc135a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:40.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.558+0000 7f89c0897640 1 --2- 172.21.15.37:0/3055537316 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89bc135660 0x7f89bc135a70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f89ac00b450 tx=0x7f89ac0059e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:40.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.558+0000 7f89b97fa640 1 -- 172.21.15.37:0/3055537316 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89ac01ae60 con 0x7f89bc135660 2024-04-26T00:34:40.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.558+0000 7f89b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:40.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.558+0000 7f89b97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:34:40.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.558+0000 7f89b97fa640 10 monclient: dump: 2024-04-26T00:34:40.566 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:40.566 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:40.566 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:40.566 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:40.566 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:40.566 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:40.566 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:40.566 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:40.566 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:40.566 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:40.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.558+0000 7f89b97fa640 1 -- 172.21.15.37:0/3055537316 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7f89ac010480 con 0x7f89bc135660 2024-04-26T00:34:40.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.558+0000 7f89b97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:34:40.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.558+0000 7f89b97fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:34:40.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.558+0000 7f89b97fa640 1 -- 172.21.15.37:0/3055537316 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89bc135660 msgr2=0x7f89bc135a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:40.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.558+0000 7f89b97fa640 1 --2- 172.21.15.37:0/3055537316 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89bc135660 0x7f89bc135a70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f89ac00b450 tx=0x7f89ac0059e0 comp rx=0 tx=0).stop 2024-04-26T00:34:40.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.558+0000 7f89b97fa640 1 -- 172.21.15.37:0/3055537316 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89bc135660 msgr2=0x7f89bc135a70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:34:40.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.558+0000 7f89b97fa640 1 --2- 172.21.15.37:0/3055537316 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89bc135660 0x7f89bc135a70 secure :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f89ac00b450 tx=0x7f89ac0059e0 comp rx=0 tx=0).stop 2024-04-26T00:34:40.567 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:34:40.567 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:34:40.568 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:34:40.568 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:34:40.568 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 112, 2024-04-26T00:34:40.568 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 167101, 2024-04-26T00:34:40.568 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:34:40.568 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:34:40.568 INFO:teuthology.orchestra.run.smithi037.stdout: 1 2024-04-26T00:34:40.568 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:40.568 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:34:40.568 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:34:40.568 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:34:40.568 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:34:40.569 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:34:40.569 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:34:40.569 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:34:40.569 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:34:40.569 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:34:40.569 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:34:40.569 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:34:40.569 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:34:40.569 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:34:40.569 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:34:40.569 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:34:40.569 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:40.569 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:34:40.570 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:34:40.570 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:34:40.570 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:34:40.570 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:34:40.570 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:34:40.570 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:34:40.570 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:34:40.570 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:34:40.570 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:34:40.570 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:34:40.570 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:34:40.570 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:34:40.570 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:40.571 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:40.571 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:34:40.571 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:34:40.571 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:34:40.571 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:34:40.571 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:34:40.571 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:34:40.571 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:34:40.571 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:34:40.571 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:34:40.571 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:34:40.572 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:34:40.572 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:34:40.572 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:34:40.572 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:34:40.572 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:34:40.572 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:34:40.572 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:34:40.572 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:34:40.572 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:34:40.572 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:34:40.572 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:34:40.572 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:34:40.573 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:34:40.573 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:34:40.573 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:34:40.573 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:34:40.573 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:34:40.573 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:34:40.573 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:40.573 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:34:40.573 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:40.573 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:34:40.573 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:40.573 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:34:40.573 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:34:40.574 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:34:40.574 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:34:40.574 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:40.574 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:34:40.574 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:34:40.574 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:40.574 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:40.574 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:40.574 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:34:40.574 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:34:40.574 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:40.575 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:40.575 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:40.575 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:40.575 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:34:40.575 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:34:40.575 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:34:40.575 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:34:40.575 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:34:40.575 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:40.575 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:40.575 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:34:40.575 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:34:40.575 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:34:40.576 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:34:40.576 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:40.576 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:34:40.576 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:34:40.576 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:40.576 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:40.576 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:40.576 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:34:40.576 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:34:40.576 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:40.576 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:40.576 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:40.576 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:40.577 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:34:40.577 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:34:40.577 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:34:40.577 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:34:40.577 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:34:40.577 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:40.577 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:40.577 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:34:40.577 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:34:40.577 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:34:40.577 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:34:40.577 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:40.577 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:34:40.578 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:34:40.578 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:40.578 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:40.578 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:40.578 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:34:40.578 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:34:40.578 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:40.578 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:40.578 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:40.578 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:40.578 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:34:40.578 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:34:40.579 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:34:40.579 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:34:40.579 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:34:40.579 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:40.579 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:40.579 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:40.579 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:34:40.579 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:34:40.579 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:40.579 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:34:40.579 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:34:40.579 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:34:40.579 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:40.580 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:40.580 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:34:40.580 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:40.580 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:34:40.580 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:34:40.580 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:34:40.580 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:40.580 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:40.580 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:34:40.580 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:40.580 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:34:40.580 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:34:40.580 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 7 2024-04-26T00:34:40.581 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:40.581 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:40.581 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:34:40.581 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:40.581 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:34:40.581 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:34:40.581 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:34:40.581 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:40.581 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:40.581 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:40.581 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:34:40.581 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:34:40.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.558+0000 7f89c2afb640 1 -- 172.21.15.37:0/3055537316 >> v2:172.21.15.45:6812/2379918623 conn(0x7f89a8035e80 msgr2=0x7f89a8038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:40.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.558+0000 7f89c2afb640 1 --2- 172.21.15.37:0/3055537316 >> v2:172.21.15.45:6812/2379918623 conn(0x7f89a8035e80 0x7f89a8038340 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f89bc130760 tx=0x7f89b0008040 comp rx=0 tx=0).stop 2024-04-26T00:34:40.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.558+0000 7f89c2afb640 10 monclient: shutdown 2024-04-26T00:34:40.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.558+0000 7f89c2afb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:34:40.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.558+0000 7f89c2afb640 1 -- 172.21.15.37:0/3055537316 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89bc1340d0 msgr2=0x7f89bc13a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:40.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.558+0000 7f89c2afb640 1 --2- 172.21.15.37:0/3055537316 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89bc1340d0 0x7f89bc13a940 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f89b40b1040 tx=0x7f89b401b870 comp rx=0 tx=0).stop 2024-04-26T00:34:40.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.558+0000 7f89c1098640 1 -- 172.21.15.37:0/3055537316 reap_dead start 2024-04-26T00:34:40.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.558+0000 7f89c1098640 1 -- 172.21.15.37:0/3055537316 reap_dead start 2024-04-26T00:34:40.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.558+0000 7f89c2afb640 1 -- 172.21.15.37:0/3055537316 shutdown_connections 2024-04-26T00:34:40.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.558+0000 7f89c2afb640 1 -- 172.21.15.37:0/3055537316 >> 172.21.15.37:0/3055537316 conn(0x7f89bc0a1d50 msgr2=0x7f89bc09cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:34:40.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.558+0000 7f89c2afb640 1 -- 172.21.15.37:0/3055537316 shutdown_connections 2024-04-26T00:34:40.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.558+0000 7f89c2afb640 1 -- 172.21.15.37:0/3055537316 wait complete. 2024-04-26T00:34:40.584 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:34:40.584 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-04-26T00:34:40.585 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:34:40.586 INFO:tasks.rados.rados.0.smithi045.stdout:2459: done (0 left) 2024-04-26T00:34:40.587 INFO:tasks.rados.rados.0.smithi045.stdout:2460: rollback oid 32 current snap is 275 2024-04-26T00:34:40.587 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 32 to 253 2024-04-26T00:34:40.587 INFO:tasks.rados.rados.0.smithi045.stdout:2461: read oid 8 snap -1 2024-04-26T00:34:40.587 INFO:tasks.rados.rados.0.smithi045.stdout:2461: expect (ObjNum 884 snap 274 seq_num 884) 2024-04-26T00:34:40.587 INFO:tasks.rados.rados.0.smithi045.stdout:2462: write (excl) oid 34 current snap is 275 2024-04-26T00:34:40.587 INFO:tasks.rados.rados.0.smithi045.stdout:2462: seq_num 893 ranges {608651=410658,1556420=674971,2773556=235095} 2024-04-26T00:34:40.606 INFO:tasks.rados.rados.0.smithi045.stdout:2462: writing smithi04526579-34 from 608651 to 1019309 tid 1 2024-04-26T00:34:40.629 INFO:tasks.rados.rados.0.smithi045.stdout:2462: writing smithi04526579-34 from 1556420 to 2231391 tid 2 2024-04-26T00:34:40.641 INFO:tasks.rados.rados.0.smithi045.stdout:2462: writing smithi04526579-34 from 2773556 to 3008651 tid 3 2024-04-26T00:34:40.642 INFO:tasks.rados.rados.0.smithi045.stdout:2463: write (excl) oid 38 current snap is 275 2024-04-26T00:34:40.642 INFO:tasks.rados.rados.0.smithi045.stdout:2461: expect (ObjNum 884 snap 274 seq_num 884) 2024-04-26T00:34:40.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.714+0000 7fc5e02d3640 10 monclient: get_monmap_and_config 2024-04-26T00:34:40.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.714+0000 7fc5e02d3640 10 monclient: build_initial_monmap 2024-04-26T00:34:40.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.714+0000 7fc5e02d3640 10 monclient: monmap: 2024-04-26T00:34:40.709 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:34:40.710 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:40.710 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:34:40.716683+0000 2024-04-26T00:34:40.710 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:34:40.716683+0000 2024-04-26T00:34:40.710 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:34:40.710 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:40.710 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:34:40.710 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:34:40.710 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:34:40.710 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:34:40.710 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:34:40.710 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:40.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.714+0000 7fc5e02d3640 1 Processor -- start 2024-04-26T00:34:40.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.714+0000 7fc5e02d3640 1 -- start start 2024-04-26T00:34:40.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.714+0000 7fc5e02d3640 10 monclient: init 2024-04-26T00:34:40.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.714+0000 7fc5e02d3640 10 monclient: _reopen_session rank -1 2024-04-26T00:34:40.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.714+0000 7fc5e02d3640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-04-26T00:34:40.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.714+0000 7fc5e02d3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc5d80a1050 addr v1:172.21.15.37:6789/0 2024-04-26T00:34:40.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.714+0000 7fc5e02d3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc5d80a1430 addr v1:172.21.15.45:6789/0 2024-04-26T00:34:40.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.714+0000 7fc5e02d3640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc5d809fa10 0x7fc5d813fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:40.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.714+0000 7fc5e02d3640 10 monclient(hunting): picked mon.noname-c con 0x7fc5d809fa10 addr v2:172.21.15.45:3300/0 2024-04-26T00:34:40.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.714+0000 7fc5e02d3640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5d8073510 con 0x7fc5d80a1050 2024-04-26T00:34:40.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.714+0000 7fc5e02d3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5d806d100 con 0x7fc5d80a1430 2024-04-26T00:34:40.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.714+0000 7fc5e02d3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:40.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.714+0000 7fc5e02d3640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc5d809f040 con 0x7fc5d809fa10 2024-04-26T00:34:40.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.714+0000 7fc5e02d3640 10 monclient(hunting): _renew_subs 2024-04-26T00:34:40.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.714+0000 7fc5e02d3640 10 monclient(hunting): authenticate will time out at 1862.022286s 2024-04-26T00:34:40.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.714+0000 7fc5de06f640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fc5d80a1050 0x7fc5d806c2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:46808/0 (socket says 172.21.15.37:46808) 2024-04-26T00:34:40.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.714+0000 7fc5de06f640 1 -- 172.21.15.37:0/12995651 learned_addr learned my addr 172.21.15.37:0/12995651 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:34:40.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.714+0000 7fc5de870640 1 --2- 172.21.15.37:0/12995651 >> v2:172.21.15.45:3300/0 conn(0x7fc5d809fa10 0x7fc5d813fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:40.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.714+0000 7fc5de870640 10 monclient(hunting): get_auth_request con 0x7fc5d809fa10 auth_method 0 2024-04-26T00:34:40.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.714+0000 7fc5de870640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:40.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.714+0000 7fc5de870640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:40.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.714+0000 7fc5de870640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:40.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.714+0000 7fc5de870640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:40.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.714+0000 7fc5de870640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:40.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.714+0000 7fc5de870640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:40.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.714+0000 7fc5de870640 10 monclient(hunting): handle_auth_done global_id 174154 payload 274 2024-04-26T00:34:40.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.714+0000 7fc5de870640 1 -- 172.21.15.37:0/12995651 >> v1:172.21.15.45:6789/0 conn(0x7fc5d80a1430 legacy=0x7fc5d806c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:34:40.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.714+0000 7fc5de870640 1 -- 172.21.15.37:0/12995651 >> v1:172.21.15.37:6789/0 conn(0x7fc5d80a1050 legacy=0x7fc5d806c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:34:40.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.714+0000 7fc5de870640 10 monclient: _finish_hunting 0 2024-04-26T00:34:40.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.714+0000 7fc5de870640 1 monclient: found mon.noname-c 2024-04-26T00:34:40.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.714+0000 7fc5de870640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:34:40.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.714+0000 7fc5de870640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:34:40.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.714+0000 7fc5de870640 1 -- 172.21.15.37:0/12995651 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5d809e000 con 0x7fc5d809fa10 2024-04-26T00:34:40.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.718+0000 7fc5de870640 1 -- 172.21.15.37:0/12995651 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d809fa10 msgr2=0x7fc5d813fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:40.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.718+0000 7fc5de870640 1 --2- 172.21.15.37:0/12995651 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d809fa10 0x7fc5d813fe80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc5c80097c0 tx=0x7fc5c801c2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54785a60810adea3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:40.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5dd06d640 1 -- 172.21.15.37:0/12995651 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c80b0070 con 0x7fc5d809fa10 2024-04-26T00:34:40.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5dd06d640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:40.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5dd06d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:34:40.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5dd06d640 10 monclient: dump: 2024-04-26T00:34:40.721 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:40.721 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:40.721 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:40.721 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:40.721 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:40.721 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:40.722 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:40.722 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:40.722 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:40.722 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:40.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5dd06d640 10 monclient: _finish_auth 0 2024-04-26T00:34:40.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5dd06d640 10 monclient: _check_auth_tickets 2024-04-26T00:34:40.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5dd06d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:34:40.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5e02d3640 5 monclient: authenticate success, global_id 174154 2024-04-26T00:34:40.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5e02d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:34:40.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5dd06d640 1 -- 172.21.15.37:0/12995651 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5c80ab600 con 0x7fc5d809fa10 2024-04-26T00:34:40.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5dd06d640 10 monclient: handle_config config(0 keys) 2024-04-26T00:34:40.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5e02d3640 10 monclient: get_monmap_and_config success 2024-04-26T00:34:40.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5e02d3640 4 set_mon_vals no callback set 2024-04-26T00:34:40.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5dd06d640 1 -- 172.21.15.37:0/12995651 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c80ab9d0 con 0x7fc5d809fa10 2024-04-26T00:34:40.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5dd06d640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:40.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5dd06d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:34:40.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5dd06d640 10 monclient: dump: 2024-04-26T00:34:40.723 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:40.723 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:40.723 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:40.723 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:40.723 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:40.723 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:40.723 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:40.723 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:40.723 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:40.723 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:40.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5e02d3640 10 monclient: shutdown 2024-04-26T00:34:40.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5e02d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:34:40.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5e02d3640 1 -- 172.21.15.37:0/12995651 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d809fa10 msgr2=0x7fc5d813fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:40.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5e02d3640 1 --2- 172.21.15.37:0/12995651 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d809fa10 0x7fc5d813fe80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc5c80097c0 tx=0x7fc5c801c2e0 comp rx=0 tx=0).stop 2024-04-26T00:34:40.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5e02d3640 1 -- 172.21.15.37:0/12995651 shutdown_connections 2024-04-26T00:34:40.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5e02d3640 1 --2- 172.21.15.37:0/12995651 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d809fa10 0x7fc5d813fe80 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:40.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5e02d3640 1 -- 172.21.15.37:0/12995651 >> 172.21.15.37:0/12995651 conn(0x7fc5d80a7e00 msgr2=0x7fc5d80a3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:34:40.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5e02d3640 1 -- 172.21.15.37:0/12995651 shutdown_connections 2024-04-26T00:34:40.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5e02d3640 1 -- 172.21.15.37:0/12995651 wait complete. 2024-04-26T00:34:40.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5e02d3640 10 monclient: build_initial_monmap 2024-04-26T00:34:40.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5e02d3640 10 monclient: monmap: 2024-04-26T00:34:40.724 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:34:40.724 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:34:40.724 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:34:40.724 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:34:40.724 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:34:40.725 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:34:40.725 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:34:40.725 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:34:40.725 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:34:40.725 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:40.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5e02d3640 1 Processor -- start 2024-04-26T00:34:40.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5e02d3640 1 -- start start 2024-04-26T00:34:40.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5e02d3640 10 monclient: init 2024-04-26T00:34:40.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5e02d3640 10 monclient: _reopen_session rank -1 2024-04-26T00:34:40.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5e02d3640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:34:40.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5e02d3640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5d809fa10 0x7fc5d81cf100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:40.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5e02d3640 10 monclient(hunting): picked mon.noname-a con 0x7fc5d809fa10 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:34:40.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5e02d3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc5d80a1050 0x7fc5d81cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:40.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5e02d3640 10 monclient(hunting): picked mon.noname-c con 0x7fc5d80a1050 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:34:40.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5e02d3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d80a1430 0x7fc5d813e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:40.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5e02d3640 10 monclient(hunting): picked mon.noname-b con 0x7fc5d80a1430 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:34:40.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5e02d3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:40.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5e02d3640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d8142df0 con 0x7fc5d809fa10 2024-04-26T00:34:40.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5e02d3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:40.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5e02d3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d80a37f0 con 0x7fc5d80a1430 2024-04-26T00:34:40.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5e02d3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:40.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5e02d3640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc5d80a30f0 con 0x7fc5d80a1050 2024-04-26T00:34:40.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5e02d3640 10 monclient(hunting): _renew_subs 2024-04-26T00:34:40.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5e02d3640 10 monclient(hunting): authenticate will time out at 1862.030849s 2024-04-26T00:34:40.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5dd86e640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc5d80a1050 msgr2=0x7fc5d81cf8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:34:40.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5dd86e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc5d80a1050 0x7fc5d81cf8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:34:40.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5de06f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5d809fa10 0x7fc5d81cf100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:40.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5de06f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5d809fa10 0x7fc5d81cf100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:43830/0 (socket says 172.21.15.37:43830) 2024-04-26T00:34:40.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5de06f640 1 -- 172.21.15.37:0/1132491133 learned_addr learned my addr 172.21.15.37:0/1132491133 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:34:40.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5de06f640 10 monclient(hunting): get_auth_request con 0x7fc5d809fa10 auth_method 0 2024-04-26T00:34:40.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5de06f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:40.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5de06f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:40.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5de06f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:40.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5de06f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:40.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5de06f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:40.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5de06f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:40.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5de870640 1 --2- 172.21.15.37:0/1132491133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d80a1430 0x7fc5d813e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:40.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5de870640 10 monclient(hunting): get_auth_request con 0x7fc5d80a1430 auth_method 0 2024-04-26T00:34:40.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5de870640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:40.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5de870640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:40.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5de870640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:40.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5de870640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:40.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5de870640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:40.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5de870640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:40.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5de06f640 10 monclient(hunting): handle_auth_done global_id 144474 payload 824 2024-04-26T00:34:40.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5de06f640 1 -- 172.21.15.37:0/1132491133 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc5d80a1050 msgr2=0x7fc5d81cf8d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:34:40.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5de06f640 1 --2- 172.21.15.37:0/1132491133 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc5d80a1050 0x7fc5d81cf8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:40.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5de06f640 1 -- 172.21.15.37:0/1132491133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d80a1430 msgr2=0x7fc5d813e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:40.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5de06f640 1 --2- 172.21.15.37:0/1132491133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d80a1430 0x7fc5d813e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:34:40.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5de06f640 10 monclient: _finish_hunting 0 2024-04-26T00:34:40.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5de06f640 1 monclient: found mon.noname-a 2024-04-26T00:34:40.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5de06f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:34:40.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5de06f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:34:40.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.722+0000 7fc5de06f640 1 -- 172.21.15.37:0/1132491133 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5d80a5bd0 con 0x7fc5d809fa10 2024-04-26T00:34:40.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5de06f640 1 -- 172.21.15.37:0/1132491133 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5d809fa10 msgr2=0x7fc5d81cf100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:40.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5de06f640 1 --2- 172.21.15.37:0/1132491133 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5d809fa10 0x7fc5d81cf100 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc5d000cc10 tx=0x7fc5d000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:40.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5d6ffd640 1 -- 172.21.15.37:0/1132491133 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5d0012400 con 0x7fc5d809fa10 2024-04-26T00:34:40.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:40.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5d6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:34:40.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5d6ffd640 10 monclient: dump: 2024-04-26T00:34:40.731 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:40.731 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:40.731 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:40.731 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:40.731 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:40.731 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:40.732 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:40.732 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:40.732 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:40.732 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:40.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5d6ffd640 10 monclient: _finish_auth 0 2024-04-26T00:34:40.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5d6ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:34:40.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:34:40.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5d6ffd640 1 -- 172.21.15.37:0/1132491133 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5d00125e0 con 0x7fc5d809fa10 2024-04-26T00:34:40.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5d6ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:34:40.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5d6ffd640 1 -- 172.21.15.37:0/1132491133 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5d00129d0 con 0x7fc5d809fa10 2024-04-26T00:34:40.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:40.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5d6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:34:40.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5d6ffd640 10 monclient: dump: 2024-04-26T00:34:40.732 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:40.732 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:40.733 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:40.733 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:40.733 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:40.733 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:40.733 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:40.733 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:40.733 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:40.733 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:40.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5e02d3640 5 monclient: authenticate success, global_id 144474 2024-04-26T00:34:40.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5e02d3640 10 monclient: _renew_subs 2024-04-26T00:34:40.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5d7fff640 4 set_mon_vals no callback set 2024-04-26T00:34:40.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5e02d3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:34:40.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5e02d3640 1 -- 172.21.15.37:0/1132491133 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5d813fd40 con 0x7fc5d809fa10 2024-04-26T00:34:40.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5e02d3640 10 monclient: _renew_subs 2024-04-26T00:34:40.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5e02d3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:34:40.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5e02d3640 1 -- 172.21.15.37:0/1132491133 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5d8123990 con 0x7fc5d809fa10 2024-04-26T00:34:40.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5e02d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:34:40.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5e02d3640 1 --2- 172.21.15.37:0/1132491133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d81cfe10 0x7fc5d813e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:40.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5e02d3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:40.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5e02d3640 1 -- 172.21.15.37:0/1132491133 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d81222c0 con 0x7fc5d81cfe10 2024-04-26T00:34:40.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5d6ffd640 1 -- 172.21.15.37:0/1132491133 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc5d0012bb0 con 0x7fc5d809fa10 2024-04-26T00:34:40.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5d6ffd640 1 --2- 172.21.15.37:0/1132491133 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc5c4035e80 0x7fc5c4038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:40.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5dd86e640 1 --2- 172.21.15.37:0/1132491133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d81cfe10 0x7fc5d813e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:40.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5dd86e640 10 monclient: get_auth_request con 0x7fc5d81cfe10 auth_method 0 2024-04-26T00:34:40.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5d6ffd640 1 -- 172.21.15.37:0/1132491133 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(566..566 src has 370..566) ==== 3975+0+0 (secure 0 0 0) 0x7fc5d0049190 con 0x7fc5d809fa10 2024-04-26T00:34:40.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5dd86e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:40.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5dd86e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:40.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5dd86e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:40.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5dd86e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:40.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5dd86e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:40.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5dd86e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:40.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5de870640 1 --2- 172.21.15.37:0/1132491133 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc5c4035e80 0x7fc5c4038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:40.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5de870640 10 monclient: get_auth_request con 0x7fc5c4035e80 auth_method 0 2024-04-26T00:34:40.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5dd86e640 10 monclient(hunting): handle_auth_done global_id 174160 payload 824 2024-04-26T00:34:40.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.726+0000 7fc5dd86e640 1 -- 172.21.15.37:0/1132491133 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc5d804f510 con 0x7fc5d81cfe10 2024-04-26T00:34:40.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.730+0000 7fc5dd86e640 1 -- 172.21.15.37:0/1132491133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d81cfe10 msgr2=0x7fc5d813e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:40.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.730+0000 7fc5dd86e640 1 --2- 172.21.15.37:0/1132491133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d81cfe10 0x7fc5d813e680 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc007fd0 tx=0x7fc5cc00c810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:40.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.730+0000 7fc5de870640 1 --2- 172.21.15.37:0/1132491133 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc5c4035e80 0x7fc5c4038340 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fc5c80ad000 tx=0x7fc5c80a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:40.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.730+0000 7fc5d6ffd640 1 -- 172.21.15.37:0/1132491133 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc017070 con 0x7fc5d81cfe10 2024-04-26T00:34:40.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.730+0000 7fc5d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:40.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.730+0000 7fc5d6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:34:40.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.730+0000 7fc5d6ffd640 10 monclient: dump: 2024-04-26T00:34:40.735 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:40.736 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:40.736 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:40.736 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:40.736 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:40.736 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:40.736 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:40.736 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:40.736 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:40.736 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:40.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.730+0000 7fc5d6ffd640 1 -- 172.21.15.37:0/1132491133 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc5cc0057d0 con 0x7fc5d81cfe10 2024-04-26T00:34:40.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.730+0000 7fc5d6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:34:40.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.730+0000 7fc5d6ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:34:40.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.730+0000 7fc5d6ffd640 1 -- 172.21.15.37:0/1132491133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d81cfe10 msgr2=0x7fc5d813e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:40.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.730+0000 7fc5d6ffd640 1 --2- 172.21.15.37:0/1132491133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d81cfe10 0x7fc5d813e680 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc007fd0 tx=0x7fc5cc00c810 comp rx=0 tx=0).stop 2024-04-26T00:34:40.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.730+0000 7fc5d6ffd640 1 -- 172.21.15.37:0/1132491133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d81cfe10 msgr2=0x7fc5d813e680 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:34:40.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.730+0000 7fc5d6ffd640 1 --2- 172.21.15.37:0/1132491133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d81cfe10 0x7fc5d813e680 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc007fd0 tx=0x7fc5cc00c810 comp rx=0 tx=0).stop 2024-04-26T00:34:40.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.738+0000 7fc5e02d3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:34:40.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.738+0000 7fc5e02d3640 1 --2- 172.21.15.37:0/1132491133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d813ebc0 0x7fc5d809ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:34:40.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.738+0000 7fc5e02d3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:34:40.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.738+0000 7fc5e02d3640 1 -- 172.21.15.37:0/1132491133 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d813efd0 con 0x7fc5d813ebc0 2024-04-26T00:34:40.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.738+0000 7fc5dd86e640 1 --2- 172.21.15.37:0/1132491133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d813ebc0 0x7fc5d809ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:34:40.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.738+0000 7fc5dd86e640 10 monclient: get_auth_request con 0x7fc5d813ebc0 auth_method 0 2024-04-26T00:34:40.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.738+0000 7fc5dd86e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:34:40.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.738+0000 7fc5dd86e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:34:40.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.738+0000 7fc5dd86e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:34:40.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.738+0000 7fc5dd86e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:34:40.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.738+0000 7fc5dd86e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:34:40.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.738+0000 7fc5dd86e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:34:40.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.738+0000 7fc5dd86e640 10 monclient(hunting): handle_auth_done global_id 174163 payload 824 2024-04-26T00:34:40.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.738+0000 7fc5dd86e640 1 -- 172.21.15.37:0/1132491133 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc5d80a5580 con 0x7fc5d813ebc0 2024-04-26T00:34:40.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.742+0000 7fc5dd86e640 1 -- 172.21.15.37:0/1132491133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d813ebc0 msgr2=0x7fc5d809ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:34:40.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.742+0000 7fc5dd86e640 1 --2- 172.21.15.37:0/1132491133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d813ebc0 0x7fc5d809ad10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc004600 tx=0x7fc5cc0037f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:34:40.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.742+0000 7fc5d6ffd640 1 -- 172.21.15.37:0/1132491133 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc017070 con 0x7fc5d813ebc0 2024-04-26T00:34:40.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.742+0000 7fc5d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:34:40.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.742+0000 7fc5d6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:34:40.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.742+0000 7fc5d6ffd640 10 monclient: dump: 2024-04-26T00:34:40.740 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:34:40.740 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:34:40.740 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:40.740 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:34:40.740 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:34:40.740 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:34:40.740 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:34:40.740 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:34:40.740 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:34:40.740 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:34:40.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.742+0000 7fc5d6ffd640 1 -- 172.21.15.37:0/1132491133 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4706 (secure 0 0 0) 0x7fc5cc022400 con 0x7fc5d813ebc0 2024-04-26T00:34:40.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.742+0000 7fc5d6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:34:40.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.742+0000 7fc5d6ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:34:40.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.742+0000 7fc5d6ffd640 1 -- 172.21.15.37:0/1132491133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d813ebc0 msgr2=0x7fc5d809ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:40.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.742+0000 7fc5d6ffd640 1 --2- 172.21.15.37:0/1132491133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d813ebc0 0x7fc5d809ad10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc004600 tx=0x7fc5cc0037f0 comp rx=0 tx=0).stop 2024-04-26T00:34:40.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.742+0000 7fc5d6ffd640 1 -- 172.21.15.37:0/1132491133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d813ebc0 msgr2=0x7fc5d809ad10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:34:40.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.742+0000 7fc5d6ffd640 1 --2- 172.21.15.37:0/1132491133 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5d813ebc0 0x7fc5d809ad10 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc004600 tx=0x7fc5cc0037f0 comp rx=0 tx=0).stop 2024-04-26T00:34:40.741 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:34:40.741 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:34:40.741 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:34:40.741 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:34:40.741 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 112, 2024-04-26T00:34:40.741 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 23363, 2024-04-26T00:34:40.742 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:34:40.742 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:34:40.742 INFO:teuthology.orchestra.run.smithi037.stdout: 1 2024-04-26T00:34:40.742 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:40.742 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 1, 2024-04-26T00:34:40.742 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:34:40.742 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:34:40.742 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:34:40.742 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:34:40.742 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:34:40.742 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:34:40.742 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:34:40.742 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:34:40.742 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:34:40.743 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:34:40.743 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:34:40.743 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:34:40.743 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:34:40.743 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:34:40.743 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:40.743 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:34:40.743 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:34:40.743 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:34:40.743 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:34:40.743 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:34:40.744 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:34:40.744 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:34:40.744 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:34:40.744 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:34:40.744 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:34:40.744 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:34:40.744 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:34:40.744 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:34:40.744 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:40.744 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:40.744 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:34:40.744 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:34:40.744 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:34:40.744 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:34:40.745 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:34:40.745 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:34:40.745 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:34:40.745 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:34:40.745 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:34:40.745 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:34:40.745 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:34:40.745 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:34:40.745 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:34:40.745 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:34:40.745 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:34:40.745 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:34:40.745 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:34:40.745 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:34:40.745 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:34:40.746 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:34:40.746 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:34:40.746 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:34:40.746 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:34:40.746 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:34:40.746 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:34:40.746 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:34:40.746 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:34:40.746 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:34:40.746 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:40.746 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:34:40.746 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:40.746 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:34:40.747 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:40.747 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:34:40.747 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:34:40.747 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:34:40.747 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:34:40.747 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:40.747 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:34:40.747 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:34:40.747 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:40.747 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:40.747 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:40.747 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:34:40.747 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:34:40.747 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:40.748 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:40.748 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:40.748 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:40.748 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:34:40.748 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:34:40.748 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:34:40.748 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:34:40.748 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:34:40.748 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:40.748 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:40.748 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:34:40.748 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:34:40.748 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:34:40.748 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:34:40.749 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:40.749 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:34:40.749 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:34:40.749 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:40.749 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:40.749 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:40.749 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:34:40.749 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:34:40.749 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:40.749 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:40.749 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:40.749 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:40.749 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:34:40.749 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:34:40.750 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:34:40.750 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:34:40.750 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:34:40.750 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:40.750 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:40.750 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:34:40.750 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:34:40.750 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:34:40.750 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:34:40.750 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:40.750 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:34:40.750 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:34:40.750 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:40.750 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:40.751 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:40.751 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:34:40.751 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:34:40.751 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:34:40.751 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:40.751 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:40.751 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:40.751 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:34:40.751 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:34:40.751 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:34:40.751 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:34:40.751 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:34:40.751 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:40.751 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:40.751 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:40.752 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:34:40.752 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:34:40.752 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:40.752 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:34:40.752 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:34:40.752 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:34:40.752 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:40.752 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:40.752 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:34:40.752 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:40.752 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:34:40.752 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:34:40.752 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:34:40.752 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:40.753 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:34:40.753 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:34:40.753 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:34:40.753 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:34:40.753 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:34:40.753 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:34:40.754 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:34:40.754 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:34:40.754 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:34:40.754 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:34:40.754 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:34:40.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.742+0000 7fc5e02d3640 1 -- 172.21.15.37:0/1132491133 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc5c4035e80 msgr2=0x7fc5c4038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:40.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.742+0000 7fc5e02d3640 1 --2- 172.21.15.37:0/1132491133 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc5c4035e80 0x7fc5c4038340 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fc5c80ad000 tx=0x7fc5c80a6000 comp rx=0 tx=0).stop 2024-04-26T00:34:40.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.742+0000 7fc5e02d3640 10 monclient: shutdown 2024-04-26T00:34:40.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.742+0000 7fc5e02d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:34:40.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.742+0000 7fc5e02d3640 1 -- 172.21.15.37:0/1132491133 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5d809fa10 msgr2=0x7fc5d81cf100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:34:40.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.742+0000 7fc5e02d3640 1 --2- 172.21.15.37:0/1132491133 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5d809fa10 0x7fc5d81cf100 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc5d000cc10 tx=0x7fc5d000a430 comp rx=0 tx=0).stop 2024-04-26T00:34:40.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.742+0000 7fc5de870640 1 -- 172.21.15.37:0/1132491133 reap_dead start 2024-04-26T00:34:40.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.742+0000 7fc5de870640 1 -- 172.21.15.37:0/1132491133 reap_dead start 2024-04-26T00:34:40.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.742+0000 7fc5e02d3640 1 -- 172.21.15.37:0/1132491133 shutdown_connections 2024-04-26T00:34:40.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.742+0000 7fc5e02d3640 1 -- 172.21.15.37:0/1132491133 >> 172.21.15.37:0/1132491133 conn(0x7fc5d80a7e00 msgr2=0x7fc5d8071e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:34:40.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.742+0000 7fc5e02d3640 1 -- 172.21.15.37:0/1132491133 shutdown_connections 2024-04-26T00:34:40.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:34:40.742+0000 7fc5e02d3640 1 -- 172.21.15.37:0/1132491133 wait complete. 2024-04-26T00:34:40.759 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:34:40.759 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-04-26T00:34:40.782 INFO:tasks.rados.rados.0.smithi045.stdout:2460: finishing rollback tid 0 to smithi04526579-32 2024-04-26T00:34:40.782 INFO:tasks.rados.rados.0.smithi045.stdout:2460: finishing rollback tid 1 to smithi04526579-32 2024-04-26T00:34:40.782 INFO:tasks.rados.rados.0.smithi045.stdout:2460: finishing rollback tid 2 to smithi04526579-32 2024-04-26T00:34:40.782 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 1757 (ObjNum 769 snap 238 seq_num 769) dirty exists 2024-04-26T00:34:40.782 INFO:tasks.rados.rados.0.smithi045.stdout:2462: finishing write tid 1 to smithi04526579-34 2024-04-26T00:34:40.782 INFO:tasks.rados.rados.0.smithi045.stdout:2462: finishing write tid 2 to smithi04526579-34 2024-04-26T00:34:40.782 INFO:tasks.rados.rados.0.smithi045.stdout:2463: seq_num 894 ranges {625458=485907,1614902=534982,2917332=108126} 2024-04-26T00:34:40.799 INFO:tasks.rados.rados.0.smithi045.stdout:2463: writing smithi04526579-38 from 625458 to 1111365 tid 1 2024-04-26T00:34:40.814 INFO:tasks.rados.rados.0.smithi045.stdout:2463: writing smithi04526579-38 from 1614902 to 2149884 tid 2 2024-04-26T00:34:40.828 INFO:tasks.rados.rados.0.smithi045.stdout:2463: writing smithi04526579-38 from 2917332 to 3025458 tid 3 2024-04-26T00:34:40.828 INFO:tasks.rados.rados.0.smithi045.stdout:2460: done (3 left) 2024-04-26T00:34:40.829 INFO:tasks.rados.rados.0.smithi045.stdout:2461: done (2 left) 2024-04-26T00:34:40.829 INFO:tasks.rados.rados.0.smithi045.stdout:2464: read oid 22 snap 260 2024-04-26T00:34:40.829 INFO:tasks.rados.rados.0.smithi045.stdout:2464: expect deleted 2024-04-26T00:34:40.829 INFO:tasks.rados.rados.0.smithi045.stdout:2465: snap_remove snap 238 2024-04-26T00:34:41.591 INFO:tasks.rados.rados.0.smithi045.stdout:2465: done (3 left) 2024-04-26T00:34:41.592 INFO:tasks.rados.rados.0.smithi045.stdout:2466: rollback oid 21 current snap is 275 2024-04-26T00:34:41.592 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 21 to 235 2024-04-26T00:34:41.592 INFO:tasks.rados.rados.0.smithi045.stdout:2462: finishing write tid 3 to smithi04526579-34 2024-04-26T00:34:41.592 INFO:tasks.rados.rados.0.smithi045.stdout:2462: finishing write tid 4 to smithi04526579-34 2024-04-26T00:34:41.592 INFO:tasks.rados.rados.0.smithi045.stdout:2462: finishing write tid 5 to smithi04526579-34 2024-04-26T00:34:41.592 INFO:tasks.rados.rados.0.smithi045.stdout:2462: oid 34 updating version 0 to 1981 2024-04-26T00:34:41.592 INFO:tasks.rados.rados.0.smithi045.stdout:2462: oid 34 updating version 1981 to 1983 2024-04-26T00:34:41.592 INFO:tasks.rados.rados.0.smithi045.stdout:2462: oid 34 version 1983 is already newer than 1982 2024-04-26T00:34:41.592 INFO:tasks.rados.rados.0.smithi045.stdout:2462: oid 34 updating version 1983 to 1984 2024-04-26T00:34:41.592 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 1984 (ObjNum 893 snap 275 seq_num 893) dirty exists 2024-04-26T00:34:41.592 INFO:tasks.rados.rados.0.smithi045.stdout:2462: left oid 34 (ObjNum 893 snap 275 seq_num 893) 2024-04-26T00:34:41.592 INFO:tasks.rados.rados.0.smithi045.stdout:2463: finishing write tid 1 to smithi04526579-38 2024-04-26T00:34:41.592 INFO:tasks.rados.rados.0.smithi045.stdout:2463: finishing write tid 2 to smithi04526579-38 2024-04-26T00:34:41.593 INFO:tasks.rados.rados.0.smithi045.stdout:2463: finishing write tid 3 to smithi04526579-38 2024-04-26T00:34:41.593 INFO:tasks.rados.rados.0.smithi045.stdout:2463: finishing write tid 4 to smithi04526579-38 2024-04-26T00:34:41.593 INFO:tasks.rados.rados.0.smithi045.stdout:2463: finishing write tid 5 to smithi04526579-38 2024-04-26T00:34:41.593 INFO:tasks.rados.rados.0.smithi045.stdout:2463: oid 38 updating version 0 to 1544 2024-04-26T00:34:41.593 INFO:tasks.rados.rados.0.smithi045.stdout:2463: oid 38 updating version 1544 to 1545 2024-04-26T00:34:41.593 INFO:tasks.rados.rados.0.smithi045.stdout:2463: oid 38 version 1545 is already newer than 1543 2024-04-26T00:34:41.593 INFO:tasks.rados.rados.0.smithi045.stdout:2463: oid 38 updating version 1545 to 1546 2024-04-26T00:34:41.593 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 1546 (ObjNum 894 snap 275 seq_num 894) dirty exists 2024-04-26T00:34:41.593 INFO:tasks.rados.rados.0.smithi045.stdout:2463: left oid 38 (ObjNum 894 snap 275 seq_num 894) 2024-04-26T00:34:41.593 INFO:tasks.rados.rados.0.smithi045.stdout:2462: done (3 left) 2024-04-26T00:34:41.593 INFO:tasks.rados.rados.0.smithi045.stdout:2463: done (2 left) 2024-04-26T00:34:41.593 INFO:tasks.rados.rados.0.smithi045.stdout:2464: done (1 left) 2024-04-26T00:34:41.594 INFO:tasks.rados.rados.0.smithi045.stdout:2467: snap_create 2024-04-26T00:34:41.612 INFO:tasks.rados.rados.0.smithi045.stdout:2466: finishing rollback tid 1 to smithi04526579-21 2024-04-26T00:34:41.612 INFO:tasks.rados.rados.0.smithi045.stdout:2466: finishing rollback tid 2 to smithi04526579-21 2024-04-26T00:34:41.612 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 1099 (ObjNum 756 snap 232 seq_num 756) dirty exists 2024-04-26T00:34:42.595 INFO:tasks.rados.rados.0.smithi045.stdout:2466: done (1 left) 2024-04-26T00:34:42.595 INFO:tasks.rados.rados.0.smithi045.stdout:2467: done (0 left) 2024-04-26T00:34:42.595 INFO:tasks.rados.rados.0.smithi045.stdout:2468: delete oid 17 current snap is 276 2024-04-26T00:34:42.604 INFO:tasks.rados.rados.0.smithi045.stdout:2468: done (0 left) 2024-04-26T00:34:42.604 INFO:tasks.rados.rados.0.smithi045.stdout:2469: read oid 43 snap -1 2024-04-26T00:34:42.604 INFO:tasks.rados.rados.0.smithi045.stdout:2469: expect deleted 2024-04-26T00:34:42.604 INFO:tasks.rados.rados.0.smithi045.stdout:2470: snap_remove snap 268 2024-04-26T00:34:43.516 INFO:tasks.rados.rados.0.smithi045.stdout:2469: done (1 left) 2024-04-26T00:34:43.516 INFO:tasks.rados.rados.0.smithi045.stdout:2470: done (0 left) 2024-04-26T00:34:43.516 INFO:tasks.rados.rados.0.smithi045.stdout:2471: write oid 36 current snap is 276 2024-04-26T00:34:43.516 INFO:tasks.rados.rados.0.smithi045.stdout:2471: seq_num 896 ranges {659072=636405,1731866=422298,2855825=203247} 2024-04-26T00:34:43.527 INFO:tasks.rados.rados.0.smithi045.stdout:2471: writing smithi04526579-36 from 659072 to 1295477 tid 1 2024-04-26T00:34:43.534 INFO:tasks.rados.rados.0.smithi045.stdout:2471: writing smithi04526579-36 from 1731866 to 2154164 tid 2 2024-04-26T00:34:43.541 INFO:tasks.rados.rados.0.smithi045.stdout:2471: writing smithi04526579-36 from 2855825 to 3059072 tid 3 2024-04-26T00:34:43.542 INFO:tasks.rados.rados.0.smithi045.stdout:2472: read oid 45 snap 235 2024-04-26T00:34:43.542 INFO:tasks.rados.rados.0.smithi045.stdout:2472: expect (ObjNum 690 snap 215 seq_num 690) 2024-04-26T00:34:43.542 INFO:tasks.rados.rados.0.smithi045.stdout:2473: write (excl) oid 15 current snap is 276 2024-04-26T00:34:43.542 INFO:tasks.rados.rados.0.smithi045.stdout:2473: seq_num 897 ranges {675879=711654,2073995=682309} 2024-04-26T00:34:43.553 INFO:tasks.rados.rados.0.smithi045.stdout:2473: writing smithi04526579-15 from 675879 to 1387533 tid 1 2024-04-26T00:34:43.565 INFO:tasks.rados.rados.0.smithi045.stdout:2473: writing smithi04526579-15 from 2073995 to 2756304 tid 2 2024-04-26T00:34:43.565 INFO:tasks.rados.rados.0.smithi045.stdout:2474: copy_from oid 3 from oid 13 current snap is 276 2024-04-26T00:34:43.565 INFO:tasks.rados.rados.0.smithi045.stdout:2471: finishing write tid 1 to smithi04526579-36 2024-04-26T00:34:43.565 INFO:tasks.rados.rados.0.smithi045.stdout:2471: finishing write tid 2 to smithi04526579-36 2024-04-26T00:34:43.565 INFO:tasks.rados.rados.0.smithi045.stdout:2471: finishing write tid 3 to smithi04526579-36 2024-04-26T00:34:43.565 INFO:tasks.rados.rados.0.smithi045.stdout:2471: finishing write tid 4 to smithi04526579-36 2024-04-26T00:34:43.565 INFO:tasks.rados.rados.0.smithi045.stdout:2471: finishing write tid 5 to smithi04526579-36 2024-04-26T00:34:43.565 INFO:tasks.rados.rados.0.smithi045.stdout:2471: oid 36 updating version 0 to 1473 2024-04-26T00:34:43.566 INFO:tasks.rados.rados.0.smithi045.stdout:2471: oid 36 version 1473 is already newer than 1472 2024-04-26T00:34:43.566 INFO:tasks.rados.rados.0.smithi045.stdout:2471: oid 36 version 1473 is already newer than 1471 2024-04-26T00:34:43.566 INFO:tasks.rados.rados.0.smithi045.stdout:2471: oid 36 updating version 1473 to 1474 2024-04-26T00:34:43.566 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 1474 (ObjNum 896 snap 276 seq_num 896) dirty exists 2024-04-26T00:34:43.566 INFO:tasks.rados.rados.0.smithi045.stdout:2471: left oid 36 (ObjNum 896 snap 276 seq_num 896) 2024-04-26T00:34:43.566 INFO:tasks.rados.rados.0.smithi045.stdout:2472: expect (ObjNum 690 snap 215 seq_num 690) 2024-04-26T00:34:43.818 INFO:tasks.rados.rados.0.smithi045.stdout:2473: finishing write tid 1 to smithi04526579-15 2024-04-26T00:34:43.818 INFO:tasks.rados.rados.0.smithi045.stdout:2471: done (3 left) 2024-04-26T00:34:43.818 INFO:tasks.rados.rados.0.smithi045.stdout:2472: done (2 left) 2024-04-26T00:34:43.818 INFO:tasks.rados.rados.0.smithi045.stdout:2475: write (excl) oid 25 current snap is 276 2024-04-26T00:34:43.818 INFO:tasks.rados.rados.0.smithi045.stdout:2475: seq_num 898 ranges {692686=503256,1848830=425967,2910671=182015} 2024-04-26T00:34:43.824 INFO:tasks.rados.rados.0.smithi045.stdout:2475: writing smithi04526579-25 from 692686 to 1195942 tid 1 2024-04-26T00:34:43.830 INFO:tasks.rados.rados.0.smithi045.stdout:2475: writing smithi04526579-25 from 1848830 to 2274797 tid 2 2024-04-26T00:34:43.835 INFO:tasks.rados.rados.0.smithi045.stdout:2475: writing smithi04526579-25 from 2910671 to 3092686 tid 3 2024-04-26T00:34:43.835 INFO:tasks.rados.rados.0.smithi045.stdout:2476: snap_remove snap 236 2024-04-26T00:34:44.606 INFO:tasks.rados.rados.0.smithi045.stdout:2476: done (3 left) 2024-04-26T00:34:44.606 INFO:tasks.rados.rados.0.smithi045.stdout:2477: snap_remove snap 261 2024-04-26T00:34:45.615 INFO:tasks.rados.rados.0.smithi045.stdout:2477: done (3 left) 2024-04-26T00:34:45.615 INFO:tasks.rados.rados.0.smithi045.stdout:2478: delete oid 26 current snap is 276 2024-04-26T00:34:45.615 INFO:tasks.rados.rados.0.smithi045.stdout:2474: finishing copy_from to smithi04526579-3 2024-04-26T00:34:45.615 INFO:tasks.rados.rados.0.smithi045.stdout:2474: got expected ENOENT (src dne) 2024-04-26T00:34:45.616 INFO:tasks.rados.rados.0.smithi045.stdout:2473: finishing write tid 2 to smithi04526579-15 2024-04-26T00:34:45.616 INFO:tasks.rados.rados.0.smithi045.stdout:2474: finishing copy_from racing read to smithi04526579-3 2024-04-26T00:34:45.616 INFO:tasks.rados.rados.0.smithi045.stdout:2473: finishing write tid 3 to smithi04526579-15 2024-04-26T00:34:45.616 INFO:tasks.rados.rados.0.smithi045.stdout:2473: finishing write tid 4 to smithi04526579-15 2024-04-26T00:34:45.616 INFO:tasks.rados.rados.0.smithi045.stdout:2473: oid 15 updating version 0 to 1934 2024-04-26T00:34:45.616 INFO:tasks.rados.rados.0.smithi045.stdout:2473: oid 15 version 1934 is already newer than 1933 2024-04-26T00:34:45.616 INFO:tasks.rados.rados.0.smithi045.stdout:2473: oid 15 updating version 1934 to 1935 2024-04-26T00:34:45.616 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 1935 (ObjNum 897 snap 276 seq_num 897) dirty exists 2024-04-26T00:34:45.616 INFO:tasks.rados.rados.0.smithi045.stdout:2473: left oid 15 (ObjNum 897 snap 276 seq_num 897) 2024-04-26T00:34:45.616 INFO:tasks.rados.rados.0.smithi045.stdout:2475: finishing write tid 1 to smithi04526579-25 2024-04-26T00:34:45.616 INFO:tasks.rados.rados.0.smithi045.stdout:2475: finishing write tid 2 to smithi04526579-25 2024-04-26T00:34:45.616 INFO:tasks.rados.rados.0.smithi045.stdout:2475: finishing write tid 3 to smithi04526579-25 2024-04-26T00:34:45.616 INFO:tasks.rados.rados.0.smithi045.stdout:2475: finishing write tid 4 to smithi04526579-25 2024-04-26T00:34:45.616 INFO:tasks.rados.rados.0.smithi045.stdout:2475: finishing write tid 5 to smithi04526579-25 2024-04-26T00:34:45.616 INFO:tasks.rados.rados.0.smithi045.stdout:2475: oid 25 updating version 0 to 2151 2024-04-26T00:34:45.617 INFO:tasks.rados.rados.0.smithi045.stdout:2475: oid 25 updating version 2151 to 2153 2024-04-26T00:34:45.617 INFO:tasks.rados.rados.0.smithi045.stdout:2475: oid 25 updating version 2153 to 2154 2024-04-26T00:34:45.617 INFO:tasks.rados.rados.0.smithi045.stdout:2475: oid 25 version 2154 is already newer than 2152 2024-04-26T00:34:45.617 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 2154 (ObjNum 898 snap 276 seq_num 898) dirty exists 2024-04-26T00:34:45.617 INFO:tasks.rados.rados.0.smithi045.stdout:2475: left oid 25 (ObjNum 898 snap 276 seq_num 898) 2024-04-26T00:34:45.627 INFO:tasks.rados.rados.0.smithi045.stdout:2473: done (3 left) 2024-04-26T00:34:45.627 INFO:tasks.rados.rados.0.smithi045.stdout:2474: done (2 left) 2024-04-26T00:34:45.627 INFO:tasks.rados.rados.0.smithi045.stdout:2475: done (1 left) 2024-04-26T00:34:45.628 INFO:tasks.rados.rados.0.smithi045.stdout:2478: done (0 left) 2024-04-26T00:34:45.628 INFO:tasks.rados.rados.0.smithi045.stdout:2479: copy_from oid 13 from oid 5 current snap is 276 2024-04-26T00:34:45.628 INFO:tasks.rados.rados.0.smithi045.stdout:2480: snap_remove snap 274 2024-04-26T00:34:46.627 INFO:tasks.rados.rados.0.smithi045.stdout:2480: done (1 left) 2024-04-26T00:34:46.627 INFO:tasks.rados.rados.0.smithi045.stdout:2481: read oid 40 snap -1 2024-04-26T00:34:46.627 INFO:tasks.rados.rados.0.smithi045.stdout:2481: expect (ObjNum 886 snap 274 seq_num 886) 2024-04-26T00:34:46.627 INFO:tasks.rados.rados.0.smithi045.stdout:2482: snap_remove snap 260 2024-04-26T00:34:47.637 INFO:tasks.rados.rados.0.smithi045.stdout:2482: done (2 left) 2024-04-26T00:34:47.637 INFO:tasks.rados.rados.0.smithi045.stdout:2483: copy_from oid 42 from oid 23 current snap is 276 2024-04-26T00:34:47.637 INFO:tasks.rados.rados.0.smithi045.stdout:2479: finishing copy_from racing read to smithi04526579-13 2024-04-26T00:34:47.637 INFO:tasks.rados.rados.0.smithi045.stdout:2479: finishing copy_from to smithi04526579-13 2024-04-26T00:34:47.637 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 1977 (ObjNum 836 snap 260 seq_num 836) dirty exists 2024-04-26T00:34:47.637 INFO:tasks.rados.rados.0.smithi045.stdout:2481: expect (ObjNum 886 snap 274 seq_num 886) 2024-04-26T00:34:47.827 INFO:tasks.rados.rados.0.smithi045.stdout:2483: finishing copy_from racing read to smithi04526579-42 2024-04-26T00:34:47.828 INFO:tasks.rados.rados.0.smithi045.stdout:2483: finishing copy_from to smithi04526579-42 2024-04-26T00:34:47.828 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 1470 (ObjNum 869 snap 272 seq_num 869) dirty exists 2024-04-26T00:34:47.828 INFO:tasks.rados.rados.0.smithi045.stdout:2479: done (2 left) 2024-04-26T00:34:47.828 INFO:tasks.rados.rados.0.smithi045.stdout:2481: done (1 left) 2024-04-26T00:34:47.828 INFO:tasks.rados.rados.0.smithi045.stdout:2483: done (0 left) 2024-04-26T00:34:47.828 INFO:tasks.rados.rados.0.smithi045.stdout:2484: write oid 17 current snap is 276 2024-04-26T00:34:47.828 INFO:tasks.rados.rados.0.smithi045.stdout:2484: seq_num 900 ranges {726300=653754,1965794=429636,2965517=160783} 2024-04-26T00:34:47.850 INFO:tasks.rados.rados.0.smithi045.stdout:2484: writing smithi04526579-17 from 726300 to 1380054 tid 1 2024-04-26T00:34:47.867 INFO:tasks.rados.rados.0.smithi045.stdout:2484: writing smithi04526579-17 from 1965794 to 2395430 tid 2 2024-04-26T00:34:47.878 INFO:tasks.rados.rados.0.smithi045.stdout:2484: writing smithi04526579-17 from 2965517 to 3126300 tid 3 2024-04-26T00:34:47.879 INFO:tasks.rados.rados.0.smithi045.stdout:2485: write oid 23 current snap is 276 2024-04-26T00:34:47.879 INFO:tasks.rados.rados.0.smithi045.stdout:2485: seq_num 901 ranges {743107=729003,1907923=689647} 2024-04-26T00:34:47.897 INFO:tasks.rados.rados.0.smithi045.stdout:2485: writing smithi04526579-23 from 743107 to 1472110 tid 1 2024-04-26T00:34:47.909 INFO:tasks.rados.rados.0.smithi045.stdout:2485: writing smithi04526579-23 from 1907923 to 2597570 tid 2 2024-04-26T00:34:47.909 INFO:tasks.rados.rados.0.smithi045.stdout:2486: snap_create 2024-04-26T00:34:47.909 INFO:tasks.rados.rados.0.smithi045.stdout:2484: finishing write tid 1 to smithi04526579-17 2024-04-26T00:34:47.909 INFO:tasks.rados.rados.0.smithi045.stdout:2484: finishing write tid 2 to smithi04526579-17 2024-04-26T00:34:47.909 INFO:tasks.rados.rados.0.smithi045.stdout:2484: finishing write tid 3 to smithi04526579-17 2024-04-26T00:34:47.909 INFO:tasks.rados.rados.0.smithi045.stdout:2484: finishing write tid 4 to smithi04526579-17 2024-04-26T00:34:47.909 INFO:tasks.rados.rados.0.smithi045.stdout:2484: finishing write tid 5 to smithi04526579-17 2024-04-26T00:34:47.910 INFO:tasks.rados.rados.0.smithi045.stdout:2484: oid 17 updating version 0 to 1535 2024-04-26T00:34:47.910 INFO:tasks.rados.rados.0.smithi045.stdout:2484: oid 17 updating version 1535 to 1536 2024-04-26T00:34:47.910 INFO:tasks.rados.rados.0.smithi045.stdout:2484: oid 17 version 1536 is already newer than 1533 2024-04-26T00:34:47.910 INFO:tasks.rados.rados.0.smithi045.stdout:2484: oid 17 version 1536 is already newer than 1534 2024-04-26T00:34:47.910 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 1536 (ObjNum 900 snap 276 seq_num 900) dirty exists 2024-04-26T00:34:47.910 INFO:tasks.rados.rados.0.smithi045.stdout:2484: left oid 17 (ObjNum 900 snap 276 seq_num 900) 2024-04-26T00:34:47.910 INFO:tasks.rados.rados.0.smithi045.stdout:2485: finishing write tid 1 to smithi04526579-23 2024-04-26T00:34:47.917 INFO:tasks.rados.rados.0.smithi045.stdout:2485: finishing write tid 2 to smithi04526579-23 2024-04-26T00:34:47.919 INFO:tasks.rados.rados.0.smithi045.stdout:2485: finishing write tid 3 to smithi04526579-23 2024-04-26T00:34:47.919 INFO:tasks.rados.rados.0.smithi045.stdout:2485: finishing write tid 4 to smithi04526579-23 2024-04-26T00:34:47.919 INFO:tasks.rados.rados.0.smithi045.stdout:2485: oid 23 updating version 0 to 1475 2024-04-26T00:34:47.919 INFO:tasks.rados.rados.0.smithi045.stdout:2485: oid 23 updating version 1475 to 1477 2024-04-26T00:34:47.919 INFO:tasks.rados.rados.0.smithi045.stdout:2485: oid 23 version 1477 is already newer than 1476 2024-04-26T00:34:47.919 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 1477 (ObjNum 901 snap 276 seq_num 901) dirty exists 2024-04-26T00:34:47.919 INFO:tasks.rados.rados.0.smithi045.stdout:2485: left oid 23 (ObjNum 901 snap 276 seq_num 901) 2024-04-26T00:34:48.520 INFO:tasks.rados.rados.0.smithi045.stdout:2484: done (2 left) 2024-04-26T00:34:48.520 INFO:tasks.rados.rados.0.smithi045.stdout:2485: done (1 left) 2024-04-26T00:34:48.520 INFO:tasks.rados.rados.0.smithi045.stdout:2486: done (0 left) 2024-04-26T00:34:48.520 INFO:tasks.rados.rados.0.smithi045.stdout:2487: read oid 38 snap -1 2024-04-26T00:34:48.520 INFO:tasks.rados.rados.0.smithi045.stdout:2487: expect (ObjNum 894 snap 275 seq_num 894) 2024-04-26T00:34:48.520 INFO:tasks.rados.rados.0.smithi045.stdout:2488: copy_from oid 50 from oid 32 current snap is 277 2024-04-26T00:34:48.520 INFO:tasks.rados.rados.0.smithi045.stdout:2489: read oid 19 snap -1 2024-04-26T00:34:48.520 INFO:tasks.rados.rados.0.smithi045.stdout:2489: expect (ObjNum 823 snap 258 seq_num 823) 2024-04-26T00:34:48.520 INFO:tasks.rados.rados.0.smithi045.stdout:2490: read oid 40 snap -1 2024-04-26T00:34:48.521 INFO:tasks.rados.rados.0.smithi045.stdout:2490: expect (ObjNum 886 snap 274 seq_num 886) 2024-04-26T00:34:48.521 INFO:tasks.rados.rados.0.smithi045.stdout:2491: write oid 37 current snap is 277 2024-04-26T00:34:48.521 INFO:tasks.rados.rados.0.smithi045.stdout:2491: seq_num 902 ranges {759914=404252,1566405=433305,2504010=655904} 2024-04-26T00:34:48.532 INFO:tasks.rados.rados.0.smithi045.stdout:2491: writing smithi04526579-37 from 759914 to 1164166 tid 1 2024-04-26T00:34:48.540 INFO:tasks.rados.rados.0.smithi045.stdout:2491: writing smithi04526579-37 from 1566405 to 1999710 tid 2 2024-04-26T00:34:48.551 INFO:tasks.rados.rados.0.smithi045.stdout:2491: writing smithi04526579-37 from 2504010 to 3159914 tid 3 2024-04-26T00:34:48.551 INFO:tasks.rados.rados.0.smithi045.stdout:2492: write oid 41 current snap is 277 2024-04-26T00:34:48.551 INFO:tasks.rados.rados.0.smithi045.stdout:2492: seq_num 903 ranges {776721=479501,2024887=693316} 2024-04-26T00:34:48.562 INFO:tasks.rados.rados.0.smithi045.stdout:2492: writing smithi04526579-41 from 776721 to 1256222 tid 1 2024-04-26T00:34:48.575 INFO:tasks.rados.rados.0.smithi045.stdout:2492: writing smithi04526579-41 from 2024887 to 2718203 tid 2 2024-04-26T00:34:48.575 INFO:tasks.rados.rados.0.smithi045.stdout:2493: delete oid 26 current snap is 277 2024-04-26T00:34:48.575 INFO:tasks.rados.rados.0.smithi045.stdout:2490: expect (ObjNum 886 snap 274 seq_num 886) 2024-04-26T00:34:48.736 INFO:tasks.rados.rados.0.smithi045.stdout:2487: expect (ObjNum 894 snap 275 seq_num 894) 2024-04-26T00:34:48.913 INFO:tasks.rados.rados.0.smithi045.stdout:2491: finishing write tid 1 to smithi04526579-37 2024-04-26T00:34:48.913 INFO:tasks.rados.rados.0.smithi045.stdout:2491: finishing write tid 2 to smithi04526579-37 2024-04-26T00:34:48.913 INFO:tasks.rados.rados.0.smithi045.stdout:2489: expect (ObjNum 823 snap 258 seq_num 823) 2024-04-26T00:34:49.088 INFO:tasks.rados.rados.0.smithi045.stdout:2491: finishing write tid 3 to smithi04526579-37 2024-04-26T00:34:49.089 INFO:tasks.rados.rados.0.smithi045.stdout:2488: finishing copy_from racing read to smithi04526579-50 2024-04-26T00:34:49.089 INFO:tasks.rados.rados.0.smithi045.stdout:2491: finishing write tid 4 to smithi04526579-37 2024-04-26T00:34:49.089 INFO:tasks.rados.rados.0.smithi045.stdout:2491: finishing write tid 5 to smithi04526579-37 2024-04-26T00:34:49.089 INFO:tasks.rados.rados.0.smithi045.stdout:2491: oid 37 updating version 0 to 2176 2024-04-26T00:34:49.089 INFO:tasks.rados.rados.0.smithi045.stdout:2491: oid 37 version 2176 is already newer than 2175 2024-04-26T00:34:49.089 INFO:tasks.rados.rados.0.smithi045.stdout:2491: oid 37 updating version 2176 to 2177 2024-04-26T00:34:49.089 INFO:tasks.rados.rados.0.smithi045.stdout:2491: oid 37 updating version 2177 to 2178 2024-04-26T00:34:49.089 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 2178 (ObjNum 902 snap 277 seq_num 902) dirty exists 2024-04-26T00:34:49.089 INFO:tasks.rados.rados.0.smithi045.stdout:2491: left oid 37 (ObjNum 902 snap 277 seq_num 902) 2024-04-26T00:34:49.089 INFO:tasks.rados.rados.0.smithi045.stdout:2492: finishing write tid 1 to smithi04526579-41 2024-04-26T00:34:49.089 INFO:tasks.rados.rados.0.smithi045.stdout:2488: finishing copy_from to smithi04526579-50 2024-04-26T00:34:49.089 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 1757 (ObjNum 769 snap 238 seq_num 769) dirty exists 2024-04-26T00:34:49.090 INFO:tasks.rados.rados.0.smithi045.stdout:2492: finishing write tid 2 to smithi04526579-41 2024-04-26T00:34:49.090 INFO:tasks.rados.rados.0.smithi045.stdout:2492: finishing write tid 3 to smithi04526579-41 2024-04-26T00:34:49.090 INFO:tasks.rados.rados.0.smithi045.stdout:2492: finishing write tid 4 to smithi04526579-41 2024-04-26T00:34:49.090 INFO:tasks.rados.rados.0.smithi045.stdout:2492: oid 41 updating version 0 to 1938 2024-04-26T00:34:49.090 INFO:tasks.rados.rados.0.smithi045.stdout:2492: oid 41 updating version 1938 to 1939 2024-04-26T00:34:49.090 INFO:tasks.rados.rados.0.smithi045.stdout:2492: oid 41 version 1939 is already newer than 1937 2024-04-26T00:34:49.090 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 1939 (ObjNum 903 snap 277 seq_num 903) dirty exists 2024-04-26T00:34:49.090 INFO:tasks.rados.rados.0.smithi045.stdout:2492: left oid 41 (ObjNum 903 snap 277 seq_num 903) 2024-04-26T00:34:49.090 INFO:tasks.rados.rados.0.smithi045.stdout:2487: done (6 left) 2024-04-26T00:34:49.090 INFO:tasks.rados.rados.0.smithi045.stdout:2488: done (5 left) 2024-04-26T00:34:49.090 INFO:tasks.rados.rados.0.smithi045.stdout:2489: done (4 left) 2024-04-26T00:34:49.090 INFO:tasks.rados.rados.0.smithi045.stdout:2490: done (3 left) 2024-04-26T00:34:49.090 INFO:tasks.rados.rados.0.smithi045.stdout:2491: done (2 left) 2024-04-26T00:34:49.090 INFO:tasks.rados.rados.0.smithi045.stdout:2492: done (1 left) 2024-04-26T00:34:49.091 INFO:tasks.rados.rados.0.smithi045.stdout:2493: done (0 left) 2024-04-26T00:34:49.091 INFO:tasks.rados.rados.0.smithi045.stdout:2494: snap_remove snap 276 2024-04-26T00:34:50.025 INFO:tasks.rados.rados.0.smithi045.stdout:2494: done (0 left) 2024-04-26T00:34:50.025 INFO:tasks.rados.rados.0.smithi045.stdout:2495: read oid 47 snap -1 2024-04-26T00:34:50.025 INFO:tasks.rados.rados.0.smithi045.stdout:2495: expect deleted 2024-04-26T00:34:50.025 INFO:tasks.rados.rados.0.smithi045.stdout:2496: snap_remove snap 257 2024-04-26T00:34:51.027 INFO:tasks.rados.rados.0.smithi045.stdout:2496: done (1 left) 2024-04-26T00:34:51.027 INFO:tasks.rados.rados.0.smithi045.stdout:2497: read oid 49 snap -1 2024-04-26T00:34:51.028 INFO:tasks.rados.rados.0.smithi045.stdout:2497: expect (ObjNum 731 snap 224 seq_num 731) 2024-04-26T00:34:51.028 INFO:tasks.rados.rados.0.smithi045.stdout:2498: snap_remove snap 205 2024-04-26T00:34:52.030 INFO:tasks.rados.rados.0.smithi045.stdout:2495: done (2 left) 2024-04-26T00:34:52.030 INFO:tasks.rados.rados.0.smithi045.stdout:2498: done (1 left) 2024-04-26T00:34:52.030 INFO:tasks.rados.rados.0.smithi045.stdout:2499: delete oid 16 current snap is 277 2024-04-26T00:34:52.030 INFO:tasks.rados.rados.0.smithi045.stdout:2497: expect (ObjNum 731 snap 224 seq_num 731) 2024-04-26T00:34:52.206 INFO:tasks.rados.rados.0.smithi045.stdout:2497: done (1 left) 2024-04-26T00:34:52.206 INFO:tasks.rados.rados.0.smithi045.stdout:2499: done (0 left) 2024-04-26T00:34:52.206 INFO:tasks.rados.rados.0.smithi045.stdout:2500: snap_remove snap 275 2024-04-26T00:34:53.053 INFO:tasks.rados.rados.0.smithi045.stdout:2500: done (0 left) 2024-04-26T00:34:53.053 INFO:tasks.rados.rados.0.smithi045.stdout:2501: read oid 27 snap 253 2024-04-26T00:34:53.053 INFO:tasks.rados.rados.0.smithi045.stdout:2501: expect (ObjNum 750 snap 232 seq_num 750) 2024-04-26T00:34:53.053 INFO:tasks.rados.rados.0.smithi045.stdout:2502: read oid 21 snap -1 2024-04-26T00:34:53.053 INFO:tasks.rados.rados.0.smithi045.stdout:2502: expect (ObjNum 756 snap 232 seq_num 756) 2024-04-26T00:34:53.053 INFO:tasks.rados.rados.0.smithi045.stdout:2503: write oid 6 current snap is 277 2024-04-26T00:34:53.053 INFO:tasks.rados.rados.0.smithi045.stdout:2503: seq_num 906 ranges {427142=421601,1400333=440643,2613702=450326} 2024-04-26T00:34:53.067 INFO:tasks.rados.rados.0.smithi045.stdout:2503: writing smithi04526579-6 from 427142 to 848743 tid 1 2024-04-26T00:34:53.086 INFO:tasks.rados.rados.0.smithi045.stdout:2503: writing smithi04526579-6 from 1400333 to 1840976 tid 2 2024-04-26T00:34:53.106 INFO:tasks.rados.rados.0.smithi045.stdout:2503: writing smithi04526579-6 from 2613702 to 3064028 tid 3 2024-04-26T00:34:53.106 INFO:tasks.rados.rados.0.smithi045.stdout:2504: read oid 39 snap 266 2024-04-26T00:34:53.106 INFO:tasks.rados.rados.0.smithi045.stdout:2504: expect deleted 2024-04-26T00:34:53.106 INFO:tasks.rados.rados.0.smithi045.stdout:2505: snap_remove snap 235 2024-04-26T00:34:53.519 INFO:tasks.rados.rados.0.smithi045.stdout:2505: done (4 left) 2024-04-26T00:34:53.519 INFO:tasks.rados.rados.0.smithi045.stdout:2506: read oid 25 snap -1 2024-04-26T00:34:53.519 INFO:tasks.rados.rados.0.smithi045.stdout:2506: expect (ObjNum 898 snap 276 seq_num 898) 2024-04-26T00:34:53.519 INFO:tasks.rados.rados.0.smithi045.stdout:2503: finishing write tid 1 to smithi04526579-6 2024-04-26T00:34:53.519 INFO:tasks.rados.rados.0.smithi045.stdout:2502: expect (ObjNum 756 snap 232 seq_num 756) 2024-04-26T00:34:53.621 INFO:tasks.rados.rados.0.smithi045.stdout:2501: expect (ObjNum 750 snap 232 seq_num 750) 2024-04-26T00:34:53.804 INFO:tasks.rados.rados.0.smithi045.stdout:2503: finishing write tid 2 to smithi04526579-6 2024-04-26T00:34:53.804 INFO:tasks.rados.rados.0.smithi045.stdout:2503: finishing write tid 3 to smithi04526579-6 2024-04-26T00:34:53.805 INFO:tasks.rados.rados.0.smithi045.stdout:2503: finishing write tid 4 to smithi04526579-6 2024-04-26T00:34:53.805 INFO:tasks.rados.rados.0.smithi045.stdout:2503: finishing write tid 5 to smithi04526579-6 2024-04-26T00:34:53.805 INFO:tasks.rados.rados.0.smithi045.stdout:2503: oid 6 updating version 0 to 1076 2024-04-26T00:34:53.805 INFO:tasks.rados.rados.0.smithi045.stdout:2503: oid 6 version 1076 is already newer than 1075 2024-04-26T00:34:53.805 INFO:tasks.rados.rados.0.smithi045.stdout:2503: oid 6 version 1076 is already newer than 1074 2024-04-26T00:34:53.805 INFO:tasks.rados.rados.0.smithi045.stdout:2503: oid 6 updating version 1076 to 1077 2024-04-26T00:34:53.805 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 1077 (ObjNum 906 snap 277 seq_num 906) dirty exists 2024-04-26T00:34:53.806 INFO:tasks.rados.rados.0.smithi045.stdout:2503: left oid 6 (ObjNum 906 snap 277 seq_num 906) 2024-04-26T00:34:53.806 INFO:tasks.rados.rados.0.smithi045.stdout:2506: expect (ObjNum 898 snap 276 seq_num 898) 2024-04-26T00:34:54.002 INFO:tasks.rados.rados.0.smithi045.stdout:2501: done (4 left) 2024-04-26T00:34:54.003 INFO:tasks.rados.rados.0.smithi045.stdout:2502: done (3 left) 2024-04-26T00:34:54.003 INFO:tasks.rados.rados.0.smithi045.stdout:2503: done (2 left) 2024-04-26T00:34:54.003 INFO:tasks.rados.rados.0.smithi045.stdout:2504: done (1 left) 2024-04-26T00:34:54.003 INFO:tasks.rados.rados.0.smithi045.stdout:2506: done (0 left) 2024-04-26T00:34:54.003 INFO:tasks.rados.rados.0.smithi045.stdout:2507: write (excl) oid 1 current snap is 277 2024-04-26T00:34:54.003 INFO:tasks.rados.rados.0.smithi045.stdout:2507: seq_num 907 ranges {443949=496850,1458815=584301,2641125=602824} 2024-04-26T00:34:54.017 INFO:tasks.rados.rados.0.smithi045.stdout:2507: writing smithi04526579-1 from 443949 to 940799 tid 1 2024-04-26T00:34:54.036 INFO:tasks.rados.rados.0.smithi045.stdout:2507: writing smithi04526579-1 from 1458815 to 2043116 tid 2 2024-04-26T00:34:54.055 INFO:tasks.rados.rados.0.smithi045.stdout:2507: writing smithi04526579-1 from 2641125 to 3243949 tid 3 2024-04-26T00:34:54.055 INFO:tasks.rados.rados.0.smithi045.stdout:2508: snap_create 2024-04-26T00:34:54.055 INFO:tasks.rados.rados.0.smithi045.stdout:2507: finishing write tid 1 to smithi04526579-1 2024-04-26T00:34:54.055 INFO:tasks.rados.rados.0.smithi045.stdout:2507: finishing write tid 2 to smithi04526579-1 2024-04-26T00:34:54.062 INFO:tasks.rados.rados.0.smithi045.stdout:2507: finishing write tid 3 to smithi04526579-1 2024-04-26T00:34:54.063 INFO:tasks.rados.rados.0.smithi045.stdout:2507: finishing write tid 4 to smithi04526579-1 2024-04-26T00:34:54.063 INFO:tasks.rados.rados.0.smithi045.stdout:2507: finishing write tid 5 to smithi04526579-1 2024-04-26T00:34:54.063 INFO:tasks.rados.rados.0.smithi045.stdout:2507: oid 1 updating version 0 to 1942 2024-04-26T00:34:54.064 INFO:tasks.rados.rados.0.smithi045.stdout:2507: oid 1 version 1942 is already newer than 1941 2024-04-26T00:34:54.064 INFO:tasks.rados.rados.0.smithi045.stdout:2507: oid 1 updating version 1942 to 1943 2024-04-26T00:34:54.064 INFO:tasks.rados.rados.0.smithi045.stdout:2507: oid 1 version 1943 is already newer than 1940 2024-04-26T00:34:54.064 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 1943 (ObjNum 907 snap 277 seq_num 907) dirty exists 2024-04-26T00:34:54.064 INFO:tasks.rados.rados.0.smithi045.stdout:2507: left oid 1 (ObjNum 907 snap 277 seq_num 907) 2024-04-26T00:34:54.524 INFO:tasks.rados.rados.0.smithi045.stdout:2507: done (1 left) 2024-04-26T00:34:54.525 INFO:tasks.rados.rados.0.smithi045.stdout:2508: done (0 left) 2024-04-26T00:34:54.525 INFO:tasks.rados.rados.0.smithi045.stdout:2509: snap_remove snap 266 2024-04-26T00:34:55.528 INFO:tasks.rados.rados.0.smithi045.stdout:2509: done (0 left) 2024-04-26T00:34:55.528 INFO:tasks.rados.rados.0.smithi045.stdout:2510: write oid 33 current snap is 278 2024-04-26T00:34:55.528 INFO:tasks.rados.rados.0.smithi045.stdout:2510: seq_num 908 ranges {460756=572099,1517297=444312,2668548=472870} 2024-04-26T00:34:55.534 INFO:tasks.rados.rados.0.smithi045.stdout:2510: writing smithi04526579-33 from 460756 to 1032855 tid 1 2024-04-26T00:34:55.540 INFO:tasks.rados.rados.0.smithi045.stdout:2510: writing smithi04526579-33 from 1517297 to 1961609 tid 2 2024-04-26T00:34:55.547 INFO:tasks.rados.rados.0.smithi045.stdout:2510: writing smithi04526579-33 from 2668548 to 3141418 tid 3 2024-04-26T00:34:55.547 INFO:tasks.rados.rados.0.smithi045.stdout:2511: snap_remove snap 277 2024-04-26T00:34:56.533 INFO:tasks.rados.rados.0.smithi045.stdout:2511: done (1 left) 2024-04-26T00:34:56.533 INFO:tasks.rados.rados.0.smithi045.stdout:2512: snap_remove snap 253 2024-04-26T00:34:57.542 INFO:tasks.rados.rados.0.smithi045.stdout:2512: done (1 left) 2024-04-26T00:34:57.543 INFO:tasks.rados.rados.0.smithi045.stdout:2513: copy_from oid 5 from oid 22 current snap is 278 2024-04-26T00:34:57.543 INFO:tasks.rados.rados.0.smithi045.stdout:2510: finishing write tid 1 to smithi04526579-33 2024-04-26T00:34:57.543 INFO:tasks.rados.rados.0.smithi045.stdout:2510: finishing write tid 2 to smithi04526579-33 2024-04-26T00:34:57.543 INFO:tasks.rados.rados.0.smithi045.stdout:2510: finishing write tid 3 to smithi04526579-33 2024-04-26T00:34:57.543 INFO:tasks.rados.rados.0.smithi045.stdout:2510: finishing write tid 4 to smithi04526579-33 2024-04-26T00:34:57.543 INFO:tasks.rados.rados.0.smithi045.stdout:2510: finishing write tid 5 to smithi04526579-33 2024-04-26T00:34:57.543 INFO:tasks.rados.rados.0.smithi045.stdout:2510: oid 33 updating version 0 to 2214 2024-04-26T00:34:57.543 INFO:tasks.rados.rados.0.smithi045.stdout:2510: oid 33 updating version 2214 to 2217 2024-04-26T00:34:57.543 INFO:tasks.rados.rados.0.smithi045.stdout:2510: oid 33 version 2217 is already newer than 2216 2024-04-26T00:34:57.543 INFO:tasks.rados.rados.0.smithi045.stdout:2510: oid 33 version 2217 is already newer than 2215 2024-04-26T00:34:57.543 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 2217 (ObjNum 908 snap 278 seq_num 908) dirty exists 2024-04-26T00:34:57.543 INFO:tasks.rados.rados.0.smithi045.stdout:2510: left oid 33 (ObjNum 908 snap 278 seq_num 908) 2024-04-26T00:34:57.543 INFO:tasks.rados.rados.0.smithi045.stdout:2510: done (1 left) 2024-04-26T00:34:57.543 INFO:tasks.rados.rados.0.smithi045.stdout:2514: copy_from oid 2 from oid 45 current snap is 278 2024-04-26T00:34:57.544 INFO:tasks.rados.rados.0.smithi045.stdout:2515: delete oid 20 current snap is 278 2024-04-26T00:34:57.557 INFO:tasks.rados.rados.0.smithi045.stdout:2515: done (2 left) 2024-04-26T00:34:57.557 INFO:tasks.rados.rados.0.smithi045.stdout:2516: copy_from oid 7 from oid 38 current snap is 278 2024-04-26T00:34:57.557 INFO:tasks.rados.rados.0.smithi045.stdout:2517: write (excl) oid 13 current snap is 278 2024-04-26T00:34:57.558 INFO:tasks.rados.rados.0.smithi045.stdout:2517: seq_num 910 ranges {494370=722597,1917908=447981,3007041=287329} 2024-04-26T00:34:57.580 INFO:tasks.rados.rados.0.smithi045.stdout:2517: writing smithi04526579-13 from 494370 to 1216967 tid 1 2024-04-26T00:34:57.600 INFO:tasks.rados.rados.0.smithi045.stdout:2517: writing smithi04526579-13 from 1917908 to 2365889 tid 2 2024-04-26T00:34:57.614 INFO:tasks.rados.rados.0.smithi045.stdout:2517: writing smithi04526579-13 from 3007041 to 3294370 tid 3 2024-04-26T00:34:57.614 INFO:tasks.rados.rados.0.smithi045.stdout:2518: write (excl) oid 44 current snap is 278 2024-04-26T00:34:57.614 INFO:tasks.rados.rados.0.smithi045.stdout:2518: seq_num 911 ranges {511177=797846,1976390=591639,3034464=276713} 2024-04-26T00:34:57.629 INFO:tasks.rados.rados.0.smithi045.stdout:2518: writing smithi04526579-44 from 511177 to 1309023 tid 1 2024-04-26T00:34:57.640 INFO:tasks.rados.rados.0.smithi045.stdout:2518: writing smithi04526579-44 from 1976390 to 2568029 tid 2 2024-04-26T00:34:57.647 INFO:tasks.rados.rados.0.smithi045.stdout:2518: writing smithi04526579-44 from 3034464 to 3311177 tid 3 2024-04-26T00:34:57.647 INFO:tasks.rados.rados.0.smithi045.stdout:2519: write (excl) oid 9 current snap is 278 2024-04-26T00:34:57.647 INFO:tasks.rados.rados.0.smithi045.stdout:2519: seq_num 912 ranges {527984=473095,1634872=451650,2661887=634311} 2024-04-26T00:34:57.654 INFO:tasks.rados.rados.0.smithi045.stdout:2519: writing smithi04526579-9 from 527984 to 1001079 tid 1 2024-04-26T00:34:57.662 INFO:tasks.rados.rados.0.smithi045.stdout:2519: writing smithi04526579-9 from 1634872 to 2086522 tid 2 2024-04-26T00:34:57.671 INFO:tasks.rados.rados.0.smithi045.stdout:2519: writing smithi04526579-9 from 2661887 to 3296198 tid 3 2024-04-26T00:34:57.671 INFO:tasks.rados.rados.0.smithi045.stdout:2520: snap_create 2024-04-26T00:34:57.671 INFO:tasks.rados.rados.0.smithi045.stdout:2513: finishing copy_from to smithi04526579-5 2024-04-26T00:34:57.671 INFO:tasks.rados.rados.0.smithi045.stdout:2513: got expected ENOENT (src dne) 2024-04-26T00:34:57.671 INFO:tasks.rados.rados.0.smithi045.stdout:2513: finishing copy_from racing read to smithi04526579-5 2024-04-26T00:34:57.671 INFO:tasks.rados.rados.0.smithi045.stdout:2517: finishing write tid 1 to smithi04526579-13 2024-04-26T00:34:57.671 INFO:tasks.rados.rados.0.smithi045.stdout:2514: finishing copy_from racing read to smithi04526579-2 2024-04-26T00:34:57.672 INFO:tasks.rados.rados.0.smithi045.stdout:2514: finishing copy_from to smithi04526579-2 2024-04-26T00:34:57.672 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 1446 (ObjNum 690 snap 215 seq_num 690) dirty exists 2024-04-26T00:34:57.672 INFO:tasks.rados.rados.0.smithi045.stdout:2517: finishing write tid 2 to smithi04526579-13 2024-04-26T00:34:57.672 INFO:tasks.rados.rados.0.smithi045.stdout:2516: finishing copy_from racing read to smithi04526579-7 2024-04-26T00:34:57.672 INFO:tasks.rados.rados.0.smithi045.stdout:2516: finishing copy_from to smithi04526579-7 2024-04-26T00:34:57.672 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 1546 (ObjNum 894 snap 275 seq_num 894) dirty exists 2024-04-26T00:34:57.672 INFO:tasks.rados.rados.0.smithi045.stdout:2517: finishing write tid 3 to smithi04526579-13 2024-04-26T00:34:57.672 INFO:tasks.rados.rados.0.smithi045.stdout:2517: finishing write tid 4 to smithi04526579-13 2024-04-26T00:34:57.672 INFO:tasks.rados.rados.0.smithi045.stdout:2517: finishing write tid 5 to smithi04526579-13 2024-04-26T00:34:57.672 INFO:tasks.rados.rados.0.smithi045.stdout:2517: oid 13 updating version 0 to 1979 2024-04-26T00:34:57.672 INFO:tasks.rados.rados.0.smithi045.stdout:2517: oid 13 updating version 1979 to 1980 2024-04-26T00:34:57.672 INFO:tasks.rados.rados.0.smithi045.stdout:2517: oid 13 updating version 1980 to 1981 2024-04-26T00:34:57.672 INFO:tasks.rados.rados.0.smithi045.stdout:2517: oid 13 version 1981 is already newer than 1978 2024-04-26T00:34:57.672 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 1981 (ObjNum 910 snap 278 seq_num 910) dirty exists 2024-04-26T00:34:57.673 INFO:tasks.rados.rados.0.smithi045.stdout:2517: left oid 13 (ObjNum 910 snap 278 seq_num 910) 2024-04-26T00:34:57.673 INFO:tasks.rados.rados.0.smithi045.stdout:2518: finishing write tid 1 to smithi04526579-44 2024-04-26T00:34:57.673 INFO:tasks.rados.rados.0.smithi045.stdout:2518: finishing write tid 2 to smithi04526579-44 2024-04-26T00:34:57.673 INFO:tasks.rados.rados.0.smithi045.stdout:2518: finishing write tid 3 to smithi04526579-44 2024-04-26T00:34:57.673 INFO:tasks.rados.rados.0.smithi045.stdout:2518: finishing write tid 4 to smithi04526579-44 2024-04-26T00:34:57.673 INFO:tasks.rados.rados.0.smithi045.stdout:2518: finishing write tid 5 to smithi04526579-44 2024-04-26T00:34:57.673 INFO:tasks.rados.rados.0.smithi045.stdout:2518: oid 44 updating version 0 to 1548 2024-04-26T00:34:57.673 INFO:tasks.rados.rados.0.smithi045.stdout:2518: oid 44 updating version 1548 to 1549 2024-04-26T00:34:57.673 INFO:tasks.rados.rados.0.smithi045.stdout:2518: oid 44 updating version 1549 to 1550 2024-04-26T00:34:57.673 INFO:tasks.rados.rados.0.smithi045.stdout:2518: oid 44 version 1550 is already newer than 1547 2024-04-26T00:34:57.673 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 1550 (ObjNum 911 snap 278 seq_num 911) dirty exists 2024-04-26T00:34:57.673 INFO:tasks.rados.rados.0.smithi045.stdout:2518: left oid 44 (ObjNum 911 snap 278 seq_num 911) 2024-04-26T00:34:57.673 INFO:tasks.rados.rados.0.smithi045.stdout:2519: finishing write tid 1 to smithi04526579-9 2024-04-26T00:34:57.673 INFO:tasks.rados.rados.0.smithi045.stdout:2519: finishing write tid 2 to smithi04526579-9 2024-04-26T00:34:57.677 INFO:tasks.rados.rados.0.smithi045.stdout:2519: finishing write tid 3 to smithi04526579-9 2024-04-26T00:34:57.678 INFO:tasks.rados.rados.0.smithi045.stdout:2519: finishing write tid 4 to smithi04526579-9 2024-04-26T00:34:57.679 INFO:tasks.rados.rados.0.smithi045.stdout:2519: finishing write tid 5 to smithi04526579-9 2024-04-26T00:34:57.679 INFO:tasks.rados.rados.0.smithi045.stdout:2519: oid 9 updating version 0 to 1931 2024-04-26T00:34:57.680 INFO:tasks.rados.rados.0.smithi045.stdout:2519: oid 9 version 1931 is already newer than 1929 2024-04-26T00:34:57.680 INFO:tasks.rados.rados.0.smithi045.stdout:2519: oid 9 version 1931 is already newer than 1930 2024-04-26T00:34:57.680 INFO:tasks.rados.rados.0.smithi045.stdout:2519: oid 9 version 1931 is already newer than 1928 2024-04-26T00:34:57.680 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 1931 (ObjNum 912 snap 278 seq_num 912) dirty exists 2024-04-26T00:34:57.680 INFO:tasks.rados.rados.0.smithi045.stdout:2519: left oid 9 (ObjNum 912 snap 278 seq_num 912) 2024-04-26T00:34:59.288 INFO:tasks.rados.rados.0.smithi045.stdout:2513: done (6 left) 2024-04-26T00:34:59.288 INFO:tasks.rados.rados.0.smithi045.stdout:2514: done (5 left) 2024-04-26T00:34:59.288 INFO:tasks.rados.rados.0.smithi045.stdout:2516: done (4 left) 2024-04-26T00:34:59.288 INFO:tasks.rados.rados.0.smithi045.stdout:2517: done (3 left) 2024-04-26T00:34:59.288 INFO:tasks.rados.rados.0.smithi045.stdout:2518: done (2 left) 2024-04-26T00:34:59.289 INFO:tasks.rados.rados.0.smithi045.stdout:2519: done (1 left) 2024-04-26T00:34:59.289 INFO:tasks.rados.rados.0.smithi045.stdout:2520: done (0 left) 2024-04-26T00:34:59.289 INFO:tasks.rados.rados.0.smithi045.stdout:2521: snap_create 2024-04-26T00:34:59.550 INFO:tasks.rados.rados.0.smithi045.stdout:2521: done (0 left) 2024-04-26T00:34:59.550 INFO:tasks.rados.rados.0.smithi045.stdout:2522: rollback oid 1 current snap is 280 2024-04-26T00:34:59.550 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 1 to 278 2024-04-26T00:34:59.551 INFO:tasks.rados.rados.0.smithi045.stdout:2523: copy_from oid 18 from oid 34 current snap is 280 2024-04-26T00:34:59.551 INFO:tasks.rados.rados.0.smithi045.stdout:2524: rollback oid 19 current snap is 280 2024-04-26T00:34:59.551 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 19 to 279 2024-04-26T00:34:59.551 INFO:tasks.rados.rados.0.smithi045.stdout:2525: snap_create 2024-04-26T00:34:59.568 INFO:tasks.rados.rados.0.smithi045.stdout:2522: finishing rollback tid 0 to smithi04526579-1 2024-04-26T00:34:59.574 INFO:tasks.rados.rados.0.smithi045.stdout:2524: finishing rollback tid 0 to smithi04526579-19 2024-04-26T00:34:59.574 INFO:tasks.rados.rados.0.smithi045.stdout:2524: finishing rollback tid 1 to smithi04526579-19 2024-04-26T00:34:59.574 INFO:tasks.rados.rados.0.smithi045.stdout:2524: finishing rollback tid 2 to smithi04526579-19 2024-04-26T00:34:59.575 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 1861 (ObjNum 823 snap 258 seq_num 823) dirty exists 2024-04-26T00:34:59.576 INFO:tasks.rados.rados.0.smithi045.stdout:2522: finishing rollback tid 1 to smithi04526579-1 2024-04-26T00:34:59.576 INFO:tasks.rados.rados.0.smithi045.stdout:2522: finishing rollback tid 2 to smithi04526579-1 2024-04-26T00:34:59.576 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 1946 (ObjNum 907 snap 277 seq_num 907) dirty exists 2024-04-26T00:34:59.587 INFO:tasks.rados.rados.0.smithi045.stdout:2523: finishing copy_from racing read to smithi04526579-18 2024-04-26T00:34:59.587 INFO:tasks.rados.rados.0.smithi045.stdout:2523: finishing copy_from to smithi04526579-18 2024-04-26T00:34:59.587 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 1984 (ObjNum 893 snap 275 seq_num 893) dirty exists 2024-04-26T00:35:00.554 INFO:tasks.rados.rados.0.smithi045.stdout:2522: done (3 left) 2024-04-26T00:35:00.554 INFO:tasks.rados.rados.0.smithi045.stdout:2523: done (2 left) 2024-04-26T00:35:00.554 INFO:tasks.rados.rados.0.smithi045.stdout:2524: done (1 left) 2024-04-26T00:35:00.554 INFO:tasks.rados.rados.0.smithi045.stdout:2525: done (0 left) 2024-04-26T00:35:00.554 INFO:tasks.rados.rados.0.smithi045.stdout:2526: copy_from oid 21 from oid 27 current snap is 281 2024-04-26T00:35:00.554 INFO:tasks.rados.rados.0.smithi045.stdout:2527: snap_create 2024-04-26T00:35:00.581 INFO:tasks.rados.rados.0.smithi045.stdout:2526: finishing copy_from racing read to smithi04526579-21 2024-04-26T00:35:00.588 INFO:tasks.rados.rados.0.smithi045.stdout:2526: finishing copy_from to smithi04526579-21 2024-04-26T00:35:00.588 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 1597 (ObjNum 849 snap 262 seq_num 849) dirty exists 2024-04-26T00:35:00.760 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-04-26T00:35:00.760 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-04-26T00:35:00.760 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-26T00:35:00.760 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-26T00:35:00.763 INFO:tasks.ceph.mon.c:Started 2024-04-26T00:35:00.763 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:35:00.764 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:35:00.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.902+0000 7fb8f4204640 10 monclient: get_monmap_and_config 2024-04-26T00:35:00.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.902+0000 7fb8f4204640 10 monclient: build_initial_monmap 2024-04-26T00:35:00.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.902+0000 7fb8f4204640 10 monclient: monmap: 2024-04-26T00:35:00.900 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:00.900 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:00.900 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:35:00.907034+0000 2024-04-26T00:35:00.900 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:35:00.907034+0000 2024-04-26T00:35:00.900 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:00.900 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:00.900 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:35:00.901 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:35:00.901 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:35:00.901 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:35:00.901 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:35:00.901 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:00.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.902+0000 7fb8f4204640 1 Processor -- start 2024-04-26T00:35:00.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.902+0000 7fb8f4204640 1 -- start start 2024-04-26T00:35:00.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.902+0000 7fb8f4204640 10 monclient: init 2024-04-26T00:35:00.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f4204640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:00.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f4204640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-04-26T00:35:00.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f4204640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb8ec136e10 addr v1:172.21.15.37:6789/0 2024-04-26T00:35:00.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f4204640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8ec137900 0x7fb8ec09c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:00.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f4204640 10 monclient(hunting): picked mon.noname-b con 0x7fb8ec137900 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:00.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f4204640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb8ec13c340 addr v1:172.21.15.45:6789/0 2024-04-26T00:35:00.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f4204640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8ec09b680 con 0x7fb8ec136e10 2024-04-26T00:35:00.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f4204640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8ec09bbd0 con 0x7fb8ec13c340 2024-04-26T00:35:00.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f4204640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:00.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f4204640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb8ec09b8f0 con 0x7fb8ec137900 2024-04-26T00:35:00.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f4204640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:00.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f4204640 10 monclient(hunting): authenticate will time out at 1882.212472s 2024-04-26T00:35:00.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f179f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8ec137900 0x7fb8ec09c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:00.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f1fa0640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fb8ec136e10 0x7fb8ec1371f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:42258/0 (socket says 172.21.15.37:42258) 2024-04-26T00:35:00.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f1fa0640 1 -- 172.21.15.37:0/1282687217 learned_addr learned my addr 172.21.15.37:0/1282687217 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:35:00.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f179f640 10 monclient(hunting): get_auth_request con 0x7fb8ec137900 auth_method 0 2024-04-26T00:35:00.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f179f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:00.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f179f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:00.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f179f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:00.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f179f640 1 -- 172.21.15.37:0/1282687217 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8ec137900 msgr2=0x7fb8ec09c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:35:00.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f179f640 1 -- 172.21.15.37:0/1282687217 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8ec137900 msgr2=0x7fb8ec09c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:35:00.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f179f640 1 --2- 172.21.15.37:0/1282687217 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8ec137900 0x7fb8ec09c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:35:00.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f179f640 1 --2- 172.21.15.37:0/1282687217 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8ec137900 0x7fb8ec09c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:35:00.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f1fa0640 1 -- 172.21.15.37:0/1282687217 >> v1:172.21.15.37:6789/0 conn(0x7fb8ec136e10 legacy=0x7fb8ec1371f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:00.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f0f9e640 1 -- 172.21.15.37:0/1282687217 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 876802903 0 0) 0x7fb8ec09b680 con 0x7fb8ec136e10 2024-04-26T00:35:00.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f0f9e640 10 monclient(hunting): _negotiate 2024-04-26T00:35:00.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f0f9e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:00.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f0f9e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:00.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f0f9e640 10 monclient(hunting): my global_id is 144477 2024-04-26T00:35:00.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f0f9e640 1 -- 172.21.15.37:0/1282687217 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8d8003530 con 0x7fb8ec136e10 2024-04-26T00:35:00.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f27a1640 1 -- 172.21.15.37:0/1282687217 >> v1:172.21.15.45:6789/0 conn(0x7fb8ec13c340 legacy=0x7fb8ec13e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:00.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f0f9e640 1 -- 172.21.15.37:0/1282687217 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4072890241 0 0) 0x7fb8d8003530 con 0x7fb8ec136e10 2024-04-26T00:35:00.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f0f9e640 1 -- 172.21.15.37:0/1282687217 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8ec137900 msgr2=0x7fb8ec09c5b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:35:00.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f0f9e640 1 --2- 172.21.15.37:0/1282687217 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8ec137900 0x7fb8ec09c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:00.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f0f9e640 1 -- 172.21.15.37:0/1282687217 >> v1:172.21.15.45:6789/0 conn(0x7fb8ec13c340 legacy=0x7fb8ec13e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:00.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f0f9e640 10 monclient: _finish_hunting 0 2024-04-26T00:35:00.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f0f9e640 1 monclient: found mon.noname-a-legacy 2024-04-26T00:35:00.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f0f9e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:00.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f0f9e640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.37:6789/0 2024-04-26T00:35:00.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f0f9e640 1 -- 172.21.15.37:0/1282687217 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8ec0a5d90 con 0x7fb8ec136e10 2024-04-26T00:35:00.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f0f9e640 10 monclient: _finish_auth 0 2024-04-26T00:35:00.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f0f9e640 10 monclient: _check_auth_tickets 2024-04-26T00:35:00.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f0f9e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:00.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f0f9e640 1 -- 172.21.15.37:0/1282687217 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 866326386 0 0) 0x7fb8e8001930 con 0x7fb8ec136e10 2024-04-26T00:35:00.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f0f9e640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:00.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f0f9e640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-26T00:35:00.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f0f9e640 10 monclient: dump: 2024-04-26T00:35:00.909 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:00.909 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:00.909 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:00.909 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:00.909 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:00.909 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:00.910 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:00.910 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:00.910 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:00.910 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:00.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f0f9e640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] but i'm connected to v1:172.21.15.37:6789/0, reconnecting 2024-04-26T00:35:00.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f0f9e640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:00.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f0f9e640 1 -- 172.21.15.37:0/1282687217 >> v1:172.21.15.37:6789/0 conn(0x7fb8ec136e10 legacy=0x7fb8ec1371f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:00.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f0f9e640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:35:00.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f0f9e640 1 --2- 172.21.15.37:0/1282687217 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8ec137900 0x7fb8d8003a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:00.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f0f9e640 10 monclient(hunting): picked mon.c con 0x7fb8ec137900 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:00.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f0f9e640 1 --2- 172.21.15.37:0/1282687217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8d80040f0 0x7fb8d8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:00.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f0f9e640 10 monclient(hunting): picked mon.b con 0x7fb8d80040f0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:35:00.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f0f9e640 1 --2- 172.21.15.37:0/1282687217 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8d8004e70 0x7fb8d800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:00.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f0f9e640 10 monclient(hunting): picked mon.a con 0x7fb8d8004e70 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:35:00.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f0f9e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:00.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f0f9e640 1 -- 172.21.15.37:0/1282687217 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ec09b8f0 con 0x7fb8d8004e70 2024-04-26T00:35:00.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f0f9e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:00.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f0f9e640 1 -- 172.21.15.37:0/1282687217 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb8d8005690 con 0x7fb8d80040f0 2024-04-26T00:35:00.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f0f9e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:00.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f0f9e640 1 -- 172.21.15.37:0/1282687217 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb8d800bf90 con 0x7fb8ec137900 2024-04-26T00:35:00.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f0f9e640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:00.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f0f9e640 10 monclient(hunting): _finish_auth 0 2024-04-26T00:35:00.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f1fa0640 1 --2- 172.21.15.37:0/1282687217 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8ec137900 0x7fb8d8003a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:00.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f1fa0640 10 monclient(hunting): get_auth_request con 0x7fb8ec137900 auth_method 0 2024-04-26T00:35:00.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f1fa0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:00.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f1fa0640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:00.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f1fa0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:35:00.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f27a1640 1 --2- 172.21.15.37:0/1282687217 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8d8004e70 0x7fb8d800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:00.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f1fa0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:00.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f1fa0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:00.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f1fa0640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:35:00.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f27a1640 10 monclient(hunting): get_auth_request con 0x7fb8d8004e70 auth_method 0 2024-04-26T00:35:00.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f27a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:00.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f27a1640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:00.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f27a1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:35:00.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f179f640 1 --2- 172.21.15.37:0/1282687217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8d80040f0 0x7fb8d8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:00.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f27a1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:00.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f27a1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:00.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f27a1640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:35:00.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f179f640 10 monclient(hunting): get_auth_request con 0x7fb8d80040f0 auth_method 0 2024-04-26T00:35:00.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f179f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:00.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f179f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:00.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f179f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:35:00.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f1fa0640 10 monclient(hunting): handle_auth_done global_id 144477 payload 293 2024-04-26T00:35:00.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f1fa0640 1 -- 172.21.15.37:0/1282687217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8d80040f0 msgr2=0x7fb8d8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:00.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f1fa0640 1 --2- 172.21.15.37:0/1282687217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8d80040f0 0x7fb8d8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:00.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f1fa0640 1 -- 172.21.15.37:0/1282687217 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8d8004e70 msgr2=0x7fb8d800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:00.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f1fa0640 1 --2- 172.21.15.37:0/1282687217 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8d8004e70 0x7fb8d800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:00.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f1fa0640 10 monclient: _finish_hunting 0 2024-04-26T00:35:00.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f1fa0640 1 monclient: found mon.c 2024-04-26T00:35:00.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f1fa0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:00.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f1fa0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:35:00.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f1fa0640 1 -- 172.21.15.37:0/1282687217 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8d8003530 con 0x7fb8ec137900 2024-04-26T00:35:00.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f1fa0640 10 monclient: _finish_auth 0 2024-04-26T00:35:00.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f1fa0640 10 monclient: _check_auth_tickets 2024-04-26T00:35:00.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f1fa0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:00.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f4204640 5 monclient: authenticate success, global_id 144477 2024-04-26T00:35:00.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.906+0000 7fb8f4204640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:35:00.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f1fa0640 1 -- 172.21.15.37:0/1282687217 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8ec137900 msgr2=0x7fb8d8003a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:00.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f1fa0640 1 --2- 172.21.15.37:0/1282687217 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8ec137900 0x7fb8d8003a70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8001b50 tx=0x7fb8e801cd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c978a684fa199098 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:00.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f0f9e640 1 -- 172.21.15.37:0/1282687217 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e800c0e0 con 0x7fb8ec137900 2024-04-26T00:35:00.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f0f9e640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:00.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f0f9e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:35:00.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f0f9e640 10 monclient: dump: 2024-04-26T00:35:00.928 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:00.928 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:00.928 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:00.928 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:00.928 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:00.928 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:00.928 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:00.928 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:00.928 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:00.928 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:00.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f0f9e640 1 -- 172.21.15.37:0/1282687217 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8e80a9070 con 0x7fb8ec137900 2024-04-26T00:35:00.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f4204640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:35:00.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f0f9e640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:00.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f4204640 10 monclient: get_monmap_and_config success 2024-04-26T00:35:00.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f4204640 4 set_mon_vals no callback set 2024-04-26T00:35:00.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f0f9e640 1 -- 172.21.15.37:0/1282687217 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e8005d80 con 0x7fb8ec137900 2024-04-26T00:35:00.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f0f9e640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:00.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f0f9e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:35:00.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f0f9e640 10 monclient: dump: 2024-04-26T00:35:00.929 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:00.929 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:00.929 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:00.929 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:00.929 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:00.929 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:00.929 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:00.929 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:00.930 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:00.930 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:00.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f4204640 10 monclient: shutdown 2024-04-26T00:35:00.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f4204640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:00.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f4204640 1 -- 172.21.15.37:0/1282687217 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8ec137900 msgr2=0x7fb8d8003a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:00.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f4204640 1 --2- 172.21.15.37:0/1282687217 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8ec137900 0x7fb8d8003a70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8001b50 tx=0x7fb8e801cd50 comp rx=0 tx=0).stop 2024-04-26T00:35:00.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f27a1640 1 -- 172.21.15.37:0/1282687217 reap_dead start 2024-04-26T00:35:00.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f4204640 1 -- 172.21.15.37:0/1282687217 shutdown_connections 2024-04-26T00:35:00.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f4204640 1 -- 172.21.15.37:0/1282687217 >> 172.21.15.37:0/1282687217 conn(0x7fb8ec0a64e0 msgr2=0x7fb8ec0a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:00.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f4204640 1 -- 172.21.15.37:0/1282687217 shutdown_connections 2024-04-26T00:35:00.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f4204640 1 -- 172.21.15.37:0/1282687217 wait complete. 2024-04-26T00:35:00.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f4204640 10 monclient: build_initial_monmap 2024-04-26T00:35:00.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f4204640 10 monclient: monmap: 2024-04-26T00:35:00.931 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:00.931 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:35:00.931 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:35:00.931 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:35:00.931 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:00.931 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:35:00.931 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:35:00.931 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:35:00.931 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:35:00.931 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:00.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f4204640 1 Processor -- start 2024-04-26T00:35:00.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f4204640 1 -- start start 2024-04-26T00:35:00.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f4204640 10 monclient: init 2024-04-26T00:35:00.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f4204640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:00.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f4204640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:35:00.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f4204640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8ec137900 0x7fb8ec134200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:00.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f4204640 10 monclient(hunting): picked mon.noname-a con 0x7fb8ec137900 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:35:00.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f4204640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8ec1cc020 0x7fb8ec09d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:00.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f4204640 10 monclient(hunting): picked mon.noname-b con 0x7fb8ec1cc020 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:35:00.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f4204640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8ec134740 0x7fb8ec09da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:00.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f4204640 10 monclient(hunting): picked mon.noname-c con 0x7fb8ec134740 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:00.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f4204640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:00.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f4204640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ec140870 con 0x7fb8ec137900 2024-04-26T00:35:00.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f4204640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:00.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f4204640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ec09c9c0 con 0x7fb8ec1cc020 2024-04-26T00:35:00.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f4204640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:00.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f4204640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb8ec1409f0 con 0x7fb8ec134740 2024-04-26T00:35:00.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f4204640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:00.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f4204640 10 monclient(hunting): authenticate will time out at 1882.222420s 2024-04-26T00:35:00.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f1fa0640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8ec137900 0x7fb8ec134200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:00.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f27a1640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8ec134740 0x7fb8ec09da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:00.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f1fa0640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8ec137900 0x7fb8ec134200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52760/0 (socket says 172.21.15.37:52760) 2024-04-26T00:35:00.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f1fa0640 1 -- 172.21.15.37:0/115677039 learned_addr learned my addr 172.21.15.37:0/115677039 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:00.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f27a1640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8ec134740 0x7fb8ec09da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:57744/0 (socket says 172.21.15.37:57744) 2024-04-26T00:35:00.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f1fa0640 10 monclient(hunting): get_auth_request con 0x7fb8ec137900 auth_method 0 2024-04-26T00:35:00.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f1fa0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:00.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f1fa0640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:00.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f1fa0640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:00.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f27a1640 10 monclient(hunting): get_auth_request con 0x7fb8ec134740 auth_method 0 2024-04-26T00:35:00.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f27a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:00.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f27a1640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:00.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f27a1640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:00.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f1fa0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:00.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f1fa0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:00.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f27a1640 1 -- 172.21.15.37:0/115677039 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8ec134740 msgr2=0x7fb8ec09da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:35:00.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f27a1640 1 -- 172.21.15.37:0/115677039 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8ec134740 msgr2=0x7fb8ec09da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:35:00.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f27a1640 1 --2- 172.21.15.37:0/115677039 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8ec134740 0x7fb8ec09da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:35:00.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f1fa0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:00.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f27a1640 1 --2- 172.21.15.37:0/115677039 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8ec134740 0x7fb8ec09da00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:35:00.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f179f640 1 --2- 172.21.15.37:0/115677039 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8ec1cc020 0x7fb8ec09d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:00.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f179f640 10 monclient(hunting): get_auth_request con 0x7fb8ec1cc020 auth_method 0 2024-04-26T00:35:00.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f179f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:00.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f179f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:00.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f179f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:00.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f1fa0640 10 monclient(hunting): handle_auth_done global_id 144480 payload 824 2024-04-26T00:35:00.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f1fa0640 1 -- 172.21.15.37:0/115677039 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8ec134740 msgr2=0x7fb8ec09da00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:35:00.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f1fa0640 1 --2- 172.21.15.37:0/115677039 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8ec134740 0x7fb8ec09da00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:00.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f1fa0640 1 -- 172.21.15.37:0/115677039 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8ec1cc020 msgr2=0x7fb8ec09d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:00.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f1fa0640 1 --2- 172.21.15.37:0/115677039 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8ec1cc020 0x7fb8ec09d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:00.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f1fa0640 10 monclient: _finish_hunting 0 2024-04-26T00:35:00.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f1fa0640 1 monclient: found mon.noname-a 2024-04-26T00:35:00.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f1fa0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:00.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f1fa0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:35:00.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f1fa0640 1 -- 172.21.15.37:0/115677039 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8ec0a8bf0 con 0x7fb8ec137900 2024-04-26T00:35:00.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.914+0000 7fb8f179f640 1 --2- 172.21.15.37:0/115677039 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8ec1cc020 0x7fb8ec09d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:35:00.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.918+0000 7fb8f1fa0640 1 -- 172.21.15.37:0/115677039 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8ec137900 msgr2=0x7fb8ec134200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:00.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.918+0000 7fb8f1fa0640 1 --2- 172.21.15.37:0/115677039 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8ec137900 0x7fb8ec134200 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb8e801b450 tx=0x7fb8d8005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:00.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.918+0000 7fb8e27fc640 1 -- 172.21.15.37:0/115677039 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e8001730 con 0x7fb8ec137900 2024-04-26T00:35:00.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.918+0000 7fb8e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:00.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.918+0000 7fb8e27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:35:00.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.918+0000 7fb8e27fc640 10 monclient: dump: 2024-04-26T00:35:00.936 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:00.936 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:00.936 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:00.936 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:00.936 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:00.936 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:00.936 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:00.936 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:00.936 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:00.936 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:00.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.918+0000 7fb8e27fc640 10 monclient: _finish_auth 0 2024-04-26T00:35:00.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.918+0000 7fb8e27fc640 10 monclient: _check_auth_tickets 2024-04-26T00:35:00.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.918+0000 7fb8e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:00.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.918+0000 7fb8f4204640 5 monclient: authenticate success, global_id 144480 2024-04-26T00:35:00.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.918+0000 7fb8e27fc640 1 -- 172.21.15.37:0/115677039 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8e80018d0 con 0x7fb8ec137900 2024-04-26T00:35:00.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.918+0000 7fb8e27fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:00.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.918+0000 7fb8e27fc640 1 -- 172.21.15.37:0/115677039 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e80072d0 con 0x7fb8ec137900 2024-04-26T00:35:00.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.918+0000 7fb8e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:00.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.918+0000 7fb8e27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:00.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.918+0000 7fb8e37fe640 4 set_mon_vals no callback set 2024-04-26T00:35:00.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.918+0000 7fb8e27fc640 10 monclient: dump: 2024-04-26T00:35:00.937 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:00.937 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:00.937 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:00.937 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:00.938 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:00.938 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:00.938 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:00.938 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:00.938 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:00.938 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:00.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.918+0000 7fb8f4204640 10 monclient: _renew_subs 2024-04-26T00:35:00.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.918+0000 7fb8f4204640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:00.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.918+0000 7fb8f4204640 1 -- 172.21.15.37:0/115677039 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8ec09f700 con 0x7fb8ec137900 2024-04-26T00:35:00.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.918+0000 7fb8f4204640 10 monclient: _renew_subs 2024-04-26T00:35:00.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.918+0000 7fb8f4204640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:00.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.918+0000 7fb8f4204640 1 -- 172.21.15.37:0/115677039 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8ec0a00c0 con 0x7fb8ec137900 2024-04-26T00:35:00.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.918+0000 7fb8f4204640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:00.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.918+0000 7fb8f4204640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:00.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.918+0000 7fb8f4204640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:00.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.918+0000 7fb8f4204640 1 -- 172.21.15.37:0/115677039 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8ec075910 con 0x7fb8ec137900 2024-04-26T00:35:00.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.918+0000 7fb8e27fc640 1 -- 172.21.15.37:0/115677039 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb8e8007470 con 0x7fb8ec137900 2024-04-26T00:35:00.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.918+0000 7fb8e27fc640 1 --2- 172.21.15.37:0/115677039 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb8d0035ed0 0x7fb8d0038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:00.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.918+0000 7fb8e27fc640 1 -- 172.21.15.37:0/115677039 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(586..586 src has 370..586) ==== 3931+0+0 (secure 0 0 0) 0x7fb8e80e2400 con 0x7fb8ec137900 2024-04-26T00:35:00.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.918+0000 7fb8f179f640 1 --2- 172.21.15.37:0/115677039 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb8d0035ed0 0x7fb8d0038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:00.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.918+0000 7fb8f179f640 10 monclient: get_auth_request con 0x7fb8d0035ed0 auth_method 0 2024-04-26T00:35:00.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.922+0000 7fb8f179f640 1 --2- 172.21.15.37:0/115677039 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb8d0035ed0 0x7fb8d0038390 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec09f1e0 tx=0x7fb8dc00c3b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:00.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.922+0000 7fb8e27fc640 1 -- 172.21.15.37:0/115677039 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fb8e8016ab0 con 0x7fb8ec137900 2024-04-26T00:35:00.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.922+0000 7fb8e27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:00.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:00.922+0000 7fb8e27fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:35:01.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:01.146+0000 7fb8f4204640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:35:01.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:01.146+0000 7fb8f4204640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:35:01.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:01.146+0000 7fb8f4204640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:01.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:01.146+0000 7fb8f4204640 1 -- 172.21.15.37:0/115677039 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb8ec066f40 con 0x7fb8ec137900 2024-04-26T00:35:01.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:01.150+0000 7fb8e27fc640 1 -- 172.21.15.37:0/115677039 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7fb8e80c1560 con 0x7fb8ec137900 2024-04-26T00:35:01.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:01.150+0000 7fb8e27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:35:01.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:01.150+0000 7fb8e27fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:35:01.145 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":112,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:35:01.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:01.150+0000 7fb8f4204640 1 -- 172.21.15.37:0/115677039 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb8d0035ed0 msgr2=0x7fb8d0038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:01.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:01.150+0000 7fb8f4204640 1 --2- 172.21.15.37:0/115677039 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb8d0035ed0 0x7fb8d0038390 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec09f1e0 tx=0x7fb8dc00c3b0 comp rx=0 tx=0).stop 2024-04-26T00:35:01.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:01.150+0000 7fb8f4204640 10 monclient: shutdown 2024-04-26T00:35:01.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:01.150+0000 7fb8f4204640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:01.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:01.150+0000 7fb8f4204640 1 -- 172.21.15.37:0/115677039 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8ec137900 msgr2=0x7fb8ec134200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:01.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:01.150+0000 7fb8f4204640 1 --2- 172.21.15.37:0/115677039 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8ec137900 0x7fb8ec134200 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb8e801b450 tx=0x7fb8d8005840 comp rx=0 tx=0).stop 2024-04-26T00:35:01.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:01.150+0000 7fb8f4204640 1 -- 172.21.15.37:0/115677039 shutdown_connections 2024-04-26T00:35:01.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:01.150+0000 7fb8f4204640 1 --2- 172.21.15.37:0/115677039 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb8d0035ed0 0x7fb8d0038390 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:01.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:01.150+0000 7fb8f4204640 1 --2- 172.21.15.37:0/115677039 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8ec134740 0x7fb8ec09da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:01.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:01.150+0000 7fb8f4204640 1 --2- 172.21.15.37:0/115677039 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb8ec1cc020 0x7fb8ec09d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:01.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:01.150+0000 7fb8f4204640 1 --2- 172.21.15.37:0/115677039 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8ec137900 0x7fb8ec134200 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:01.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:01.150+0000 7fb8f4204640 1 -- 172.21.15.37:0/115677039 >> 172.21.15.37:0/115677039 conn(0x7fb8ec0a64e0 msgr2=0x7fb8ec13e730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:01.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:01.150+0000 7fb8f4204640 1 -- 172.21.15.37:0/115677039 shutdown_connections 2024-04-26T00:35:01.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:01.150+0000 7fb8f4204640 1 -- 172.21.15.37:0/115677039 wait complete. 2024-04-26T00:35:01.556 INFO:tasks.rados.rados.0.smithi045.stdout:2526: done (1 left) 2024-04-26T00:35:01.557 INFO:tasks.rados.rados.0.smithi045.stdout:2527: done (0 left) 2024-04-26T00:35:01.557 INFO:tasks.rados.rados.0.smithi045.stdout:2528: write (excl) oid 46 current snap is 282 2024-04-26T00:35:01.557 INFO:tasks.rados.rados.0.smithi045.stdout:2528: seq_num 913 ranges {544791=664697,1693354=595308,2689310=445583} 2024-04-26T00:35:01.566 INFO:tasks.rados.rados.0.smithi045.stdout:2528: writing smithi04526579-46 from 544791 to 1209488 tid 1 2024-04-26T00:35:01.576 INFO:tasks.rados.rados.0.smithi045.stdout:2528: writing smithi04526579-46 from 1693354 to 2288662 tid 2 2024-04-26T00:35:01.583 INFO:tasks.rados.rados.0.smithi045.stdout:2528: writing smithi04526579-46 from 2689310 to 3134893 tid 3 2024-04-26T00:35:01.584 INFO:tasks.rados.rados.0.smithi045.stdout:2528: finishing write tid 1 to smithi04526579-46 2024-04-26T00:35:01.584 INFO:tasks.rados.rados.0.smithi045.stdout:2528: finishing write tid 2 to smithi04526579-46 2024-04-26T00:35:01.584 INFO:tasks.rados.rados.0.smithi045.stdout:2529: read oid 42 snap 278 2024-04-26T00:35:01.584 INFO:tasks.rados.rados.0.smithi045.stdout:2529: expect (ObjNum 869 snap 272 seq_num 869) 2024-04-26T00:35:01.584 INFO:tasks.rados.rados.0.smithi045.stdout:2530: write oid 16 current snap is 282 2024-04-26T00:35:01.584 INFO:tasks.rados.rados.0.smithi045.stdout:2530: seq_num 914 ranges {561598=739946,1751836=455319,2716733=644865} 2024-04-26T00:35:01.594 INFO:tasks.rados.rados.0.smithi045.stdout:2530: writing smithi04526579-16 from 561598 to 1301544 tid 1 2024-04-26T00:35:01.602 INFO:tasks.rados.rados.0.smithi045.stdout:2530: writing smithi04526579-16 from 1751836 to 2207155 tid 2 2024-04-26T00:35:01.612 INFO:tasks.rados.rados.0.smithi045.stdout:2530: writing smithi04526579-16 from 2716733 to 3361598 tid 3 2024-04-26T00:35:01.613 INFO:tasks.rados.rados.0.smithi045.stdout:2531: rollback oid 6 current snap is 282 2024-04-26T00:35:01.613 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 6 to 256 2024-04-26T00:35:01.613 INFO:tasks.rados.rados.0.smithi045.stdout:2528: finishing write tid 3 to smithi04526579-46 2024-04-26T00:35:01.613 INFO:tasks.rados.rados.0.smithi045.stdout:2532: rollback oid 49 current snap is 282 2024-04-26T00:35:01.613 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 49 to 280 2024-04-26T00:35:01.613 INFO:tasks.rados.rados.0.smithi045.stdout:2528: finishing write tid 4 to smithi04526579-46 2024-04-26T00:35:01.613 INFO:tasks.rados.rados.0.smithi045.stdout:2528: finishing write tid 5 to smithi04526579-46 2024-04-26T00:35:01.613 INFO:tasks.rados.rados.0.smithi045.stdout:2528: oid 46 updating version 0 to 1476 2024-04-26T00:35:01.613 INFO:tasks.rados.rados.0.smithi045.stdout:2528: oid 46 updating version 1476 to 1478 2024-04-26T00:35:01.613 INFO:tasks.rados.rados.0.smithi045.stdout:2528: oid 46 version 1478 is already newer than 1475 2024-04-26T00:35:01.613 INFO:tasks.rados.rados.0.smithi045.stdout:2528: oid 46 version 1478 is already newer than 1477 2024-04-26T00:35:01.613 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 1478 (ObjNum 913 snap 282 seq_num 913) dirty exists 2024-04-26T00:35:01.613 INFO:tasks.rados.rados.0.smithi045.stdout:2528: left oid 46 (ObjNum 913 snap 282 seq_num 913) 2024-04-26T00:35:01.614 INFO:tasks.rados.rados.0.smithi045.stdout:2530: finishing write tid 1 to smithi04526579-16 2024-04-26T00:35:01.614 INFO:tasks.rados.rados.0.smithi045.stdout:2529: expect (ObjNum 869 snap 272 seq_num 869) 2024-04-26T00:35:01.751 INFO:tasks.rados.rados.0.smithi045.stdout:2530: finishing write tid 2 to smithi04526579-16 2024-04-26T00:35:01.752 INFO:tasks.rados.rados.0.smithi045.stdout:2531: finishing rollback tid 0 to smithi04526579-6 2024-04-26T00:35:01.752 INFO:tasks.rados.rados.0.smithi045.stdout:2531: finishing rollback tid 1 to smithi04526579-6 2024-04-26T00:35:01.752 INFO:tasks.rados.rados.0.smithi045.stdout:2531: finishing rollback tid 2 to smithi04526579-6 2024-04-26T00:35:01.752 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 1080 (ObjNum 542 snap 167 seq_num 542) dirty exists 2024-04-26T00:35:01.752 INFO:tasks.rados.rados.0.smithi045.stdout:2530: finishing write tid 3 to smithi04526579-16 2024-04-26T00:35:01.752 INFO:tasks.rados.rados.0.smithi045.stdout:2532: finishing rollback tid 0 to smithi04526579-49 2024-04-26T00:35:01.752 INFO:tasks.rados.rados.0.smithi045.stdout:2530: finishing write tid 4 to smithi04526579-16 2024-04-26T00:35:01.752 INFO:tasks.rados.rados.0.smithi045.stdout:2530: finishing write tid 5 to smithi04526579-16 2024-04-26T00:35:01.752 INFO:tasks.rados.rados.0.smithi045.stdout:2530: oid 16 updating version 0 to 1156 2024-04-26T00:35:01.752 INFO:tasks.rados.rados.0.smithi045.stdout:2530: oid 16 updating version 1156 to 1158 2024-04-26T00:35:01.752 INFO:tasks.rados.rados.0.smithi045.stdout:2530: oid 16 version 1158 is already newer than 1155 2024-04-26T00:35:01.752 INFO:tasks.rados.rados.0.smithi045.stdout:2530: oid 16 version 1158 is already newer than 1157 2024-04-26T00:35:01.753 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 1158 (ObjNum 914 snap 282 seq_num 914) dirty exists 2024-04-26T00:35:01.753 INFO:tasks.rados.rados.0.smithi045.stdout:2530: left oid 16 (ObjNum 914 snap 282 seq_num 914) 2024-04-26T00:35:01.753 INFO:tasks.rados.rados.0.smithi045.stdout:2532: finishing rollback tid 1 to smithi04526579-49 2024-04-26T00:35:01.753 INFO:tasks.rados.rados.0.smithi045.stdout:2532: finishing rollback tid 2 to smithi04526579-49 2024-04-26T00:35:01.753 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 1987 (ObjNum 731 snap 224 seq_num 731) dirty exists 2024-04-26T00:35:01.753 INFO:tasks.rados.rados.0.smithi045.stdout:2528: done (4 left) 2024-04-26T00:35:01.753 INFO:tasks.rados.rados.0.smithi045.stdout:2529: done (3 left) 2024-04-26T00:35:01.753 INFO:tasks.rados.rados.0.smithi045.stdout:2530: done (2 left) 2024-04-26T00:35:01.753 INFO:tasks.rados.rados.0.smithi045.stdout:2531: done (1 left) 2024-04-26T00:35:01.753 INFO:tasks.rados.rados.0.smithi045.stdout:2532: done (0 left) 2024-04-26T00:35:01.753 INFO:tasks.rados.rados.0.smithi045.stdout:2533: snap_remove snap 279 2024-04-26T00:35:02.573 INFO:tasks.rados.rados.0.smithi045.stdout:2533: done (0 left) 2024-04-26T00:35:02.573 INFO:tasks.rados.rados.0.smithi045.stdout:2534: write (excl) oid 26 current snap is 282 2024-04-26T00:35:02.573 INFO:tasks.rados.rados.0.smithi045.stdout:2534: seq_num 915 ranges {578405=415195,1410318=598977,2744156=584480} 2024-04-26T00:35:02.584 INFO:tasks.rados.rados.0.smithi045.stdout:2534: writing smithi04526579-26 from 578405 to 993600 tid 1 2024-04-26T00:35:02.594 INFO:tasks.rados.rados.0.smithi045.stdout:2534: writing smithi04526579-26 from 1410318 to 2009295 tid 2 2024-04-26T00:35:02.605 INFO:tasks.rados.rados.0.smithi045.stdout:2534: writing smithi04526579-26 from 2744156 to 3328636 tid 3 2024-04-26T00:35:02.605 INFO:tasks.rados.rados.0.smithi045.stdout:2535: snap_remove snap 280 2024-04-26T00:35:03.527 INFO:tasks.rados.rados.0.smithi045.stdout:2535: done (1 left) 2024-04-26T00:35:03.527 INFO:tasks.rados.rados.0.smithi045.stdout:2536: write oid 43 current snap is 282 2024-04-26T00:35:03.527 INFO:tasks.rados.rados.0.smithi045.stdout:2536: seq_num 916 ranges {595212=490444,1868800=458988,2887932=507280} 2024-04-26T00:35:03.540 INFO:tasks.rados.rados.0.smithi045.stdout:2536: writing smithi04526579-43 from 595212 to 1085656 tid 1 2024-04-26T00:35:03.555 INFO:tasks.rados.rados.0.smithi045.stdout:2536: writing smithi04526579-43 from 1868800 to 2327788 tid 2 2024-04-26T00:35:03.567 INFO:tasks.rados.rados.0.smithi045.stdout:2536: writing smithi04526579-43 from 2887932 to 3395212 tid 3 2024-04-26T00:35:03.567 INFO:tasks.rados.rados.0.smithi045.stdout:2537: write oid 8 current snap is 282 2024-04-26T00:35:03.567 INFO:tasks.rados.rados.0.smithi045.stdout:2537: seq_num 917 ranges {612019=565693,1810929=602646,3082649=329370} 2024-04-26T00:35:03.581 INFO:tasks.rados.rados.0.smithi045.stdout:2537: writing smithi04526579-8 from 612019 to 1177712 tid 1 2024-04-26T00:35:03.593 INFO:tasks.rados.rados.0.smithi045.stdout:2537: writing smithi04526579-8 from 1810929 to 2413575 tid 2 2024-04-26T00:35:03.602 INFO:tasks.rados.rados.0.smithi045.stdout:2537: writing smithi04526579-8 from 3082649 to 3412019 tid 3 2024-04-26T00:35:03.602 INFO:tasks.rados.rados.0.smithi045.stdout:2538: write (excl) oid 23 current snap is 282 2024-04-26T00:35:03.602 INFO:tasks.rados.rados.0.smithi045.stdout:2534: finishing write tid 1 to smithi04526579-26 2024-04-26T00:35:03.603 INFO:tasks.rados.rados.0.smithi045.stdout:2538: seq_num 918 ranges {628826=640942,1869411=462657,2826425=534649} 2024-04-26T00:35:03.612 INFO:tasks.rados.rados.0.smithi045.stdout:2538: writing smithi04526579-23 from 628826 to 1269768 tid 1 2024-04-26T00:35:03.621 INFO:tasks.rados.rados.0.smithi045.stdout:2538: writing smithi04526579-23 from 1869411 to 2332068 tid 2 2024-04-26T00:35:03.631 INFO:tasks.rados.rados.0.smithi045.stdout:2538: writing smithi04526579-23 from 2826425 to 3361074 tid 3 2024-04-26T00:35:03.631 INFO:tasks.rados.rados.0.smithi045.stdout:2539: rollback oid 35 current snap is 282 2024-04-26T00:35:03.631 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 35 to 256 2024-04-26T00:35:03.631 INFO:tasks.rados.rados.0.smithi045.stdout:2534: finishing write tid 2 to smithi04526579-26 2024-04-26T00:35:03.631 INFO:tasks.rados.rados.0.smithi045.stdout:2534: finishing write tid 3 to smithi04526579-26 2024-04-26T00:35:03.631 INFO:tasks.rados.rados.0.smithi045.stdout:2534: finishing write tid 4 to smithi04526579-26 2024-04-26T00:35:03.631 INFO:tasks.rados.rados.0.smithi045.stdout:2534: finishing write tid 5 to smithi04526579-26 2024-04-26T00:35:03.631 INFO:tasks.rados.rados.0.smithi045.stdout:2534: oid 26 updating version 0 to 1948 2024-04-26T00:35:03.631 INFO:tasks.rados.rados.0.smithi045.stdout:2534: oid 26 updating version 1948 to 1950 2024-04-26T00:35:03.631 INFO:tasks.rados.rados.0.smithi045.stdout:2534: oid 26 version 1950 is already newer than 1949 2024-04-26T00:35:03.631 INFO:tasks.rados.rados.0.smithi045.stdout:2534: oid 26 version 1950 is already newer than 1947 2024-04-26T00:35:03.632 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 1950 (ObjNum 915 snap 282 seq_num 915) dirty exists 2024-04-26T00:35:03.632 INFO:tasks.rados.rados.0.smithi045.stdout:2534: left oid 26 (ObjNum 915 snap 282 seq_num 915) 2024-04-26T00:35:03.632 INFO:tasks.rados.rados.0.smithi045.stdout:2536: finishing write tid 1 to smithi04526579-43 2024-04-26T00:35:03.632 INFO:tasks.rados.rados.0.smithi045.stdout:2536: finishing write tid 2 to smithi04526579-43 2024-04-26T00:35:03.632 INFO:tasks.rados.rados.0.smithi045.stdout:2536: finishing write tid 3 to smithi04526579-43 2024-04-26T00:35:03.632 INFO:tasks.rados.rados.0.smithi045.stdout:2536: finishing write tid 4 to smithi04526579-43 2024-04-26T00:35:03.632 INFO:tasks.rados.rados.0.smithi045.stdout:2536: finishing write tid 5 to smithi04526579-43 2024-04-26T00:35:03.632 INFO:tasks.rados.rados.0.smithi045.stdout:2536: oid 43 updating version 0 to 1758 2024-04-26T00:35:03.632 INFO:tasks.rados.rados.0.smithi045.stdout:2536: oid 43 updating version 1758 to 1760 2024-04-26T00:35:03.632 INFO:tasks.rados.rados.0.smithi045.stdout:2536: oid 43 version 1760 is already newer than 1759 2024-04-26T00:35:03.632 INFO:tasks.rados.rados.0.smithi045.stdout:2536: oid 43 updating version 1760 to 1761 2024-04-26T00:35:03.632 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 1761 (ObjNum 916 snap 282 seq_num 916) dirty exists 2024-04-26T00:35:03.632 INFO:tasks.rados.rados.0.smithi045.stdout:2536: left oid 43 (ObjNum 916 snap 282 seq_num 916) 2024-04-26T00:35:03.632 INFO:tasks.rados.rados.0.smithi045.stdout:2537: finishing write tid 1 to smithi04526579-8 2024-04-26T00:35:03.633 INFO:tasks.rados.rados.0.smithi045.stdout:2537: finishing write tid 2 to smithi04526579-8 2024-04-26T00:35:03.633 INFO:tasks.rados.rados.0.smithi045.stdout:2534: done (4 left) 2024-04-26T00:35:03.633 INFO:tasks.rados.rados.0.smithi045.stdout:2536: done (3 left) 2024-04-26T00:35:03.633 INFO:tasks.rados.rados.0.smithi045.stdout:2540: copy_from oid 39 from oid 48 current snap is 282 2024-04-26T00:35:03.633 INFO:tasks.rados.rados.0.smithi045.stdout:2537: finishing write tid 3 to smithi04526579-8 2024-04-26T00:35:03.633 INFO:tasks.rados.rados.0.smithi045.stdout:2537: finishing write tid 4 to smithi04526579-8 2024-04-26T00:35:03.633 INFO:tasks.rados.rados.0.smithi045.stdout:2537: finishing write tid 5 to smithi04526579-8 2024-04-26T00:35:03.633 INFO:tasks.rados.rados.0.smithi045.stdout:2537: oid 8 updating version 0 to 1982 2024-04-26T00:35:03.633 INFO:tasks.rados.rados.0.smithi045.stdout:2537: oid 8 updating version 1982 to 1983 2024-04-26T00:35:03.633 INFO:tasks.rados.rados.0.smithi045.stdout:2537: oid 8 updating version 1983 to 1984 2024-04-26T00:35:03.633 INFO:tasks.rados.rados.0.smithi045.stdout:2537: oid 8 updating version 1984 to 1985 2024-04-26T00:35:03.633 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 1985 (ObjNum 917 snap 282 seq_num 917) dirty exists 2024-04-26T00:35:03.633 INFO:tasks.rados.rados.0.smithi045.stdout:2537: left oid 8 (ObjNum 917 snap 282 seq_num 917) 2024-04-26T00:35:03.633 INFO:tasks.rados.rados.0.smithi045.stdout:2538: finishing write tid 1 to smithi04526579-23 2024-04-26T00:35:03.634 INFO:tasks.rados.rados.0.smithi045.stdout:2538: finishing write tid 2 to smithi04526579-23 2024-04-26T00:35:03.634 INFO:tasks.rados.rados.0.smithi045.stdout:2537: done (3 left) 2024-04-26T00:35:03.634 INFO:tasks.rados.rados.0.smithi045.stdout:2541: rollback oid 16 current snap is 282 2024-04-26T00:35:03.634 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 16 to 256 2024-04-26T00:35:03.634 INFO:tasks.rados.rados.0.smithi045.stdout:2542: copy_from oid 47 from oid 28 current snap is 282 2024-04-26T00:35:03.634 INFO:tasks.rados.rados.0.smithi045.stdout:2543: read oid 4 snap -1 2024-04-26T00:35:03.634 INFO:tasks.rados.rados.0.smithi045.stdout:2543: expect (ObjNum 881 snap 273 seq_num 881) 2024-04-26T00:35:03.634 INFO:tasks.rados.rados.0.smithi045.stdout:2544: write oid 33 current snap is 282 2024-04-26T00:35:03.634 INFO:tasks.rados.rados.0.smithi045.stdout:2544: seq_num 919 ranges {645633=716191,1927893=606315,3137495=308138} 2024-04-26T00:35:03.641 INFO:tasks.rados.rados.0.smithi045.stdout:2544: writing smithi04526579-33 from 645633 to 1361824 tid 1 2024-04-26T00:35:03.651 INFO:tasks.rados.rados.0.smithi045.stdout:2544: writing smithi04526579-33 from 1927893 to 2534208 tid 2 2024-04-26T00:35:03.659 INFO:tasks.rados.rados.0.smithi045.stdout:2544: writing smithi04526579-33 from 3137495 to 3445633 tid 3 2024-04-26T00:35:03.659 INFO:tasks.rados.rados.0.smithi045.stdout:2545: read oid 3 snap -1 2024-04-26T00:35:03.659 INFO:tasks.rados.rados.0.smithi045.stdout:2545: expect (ObjNum 878 snap 273 seq_num 878) 2024-04-26T00:35:03.659 INFO:tasks.rados.rados.0.smithi045.stdout:2546: delete oid 38 current snap is 282 2024-04-26T00:35:03.659 INFO:tasks.rados.rados.0.smithi045.stdout:2538: finishing write tid 3 to smithi04526579-23 2024-04-26T00:35:03.659 INFO:tasks.rados.rados.0.smithi045.stdout:2541: finishing rollback tid 0 to smithi04526579-16 2024-04-26T00:35:03.659 INFO:tasks.rados.rados.0.smithi045.stdout:2538: finishing write tid 4 to smithi04526579-23 2024-04-26T00:35:03.659 INFO:tasks.rados.rados.0.smithi045.stdout:2538: finishing write tid 5 to smithi04526579-23 2024-04-26T00:35:03.659 INFO:tasks.rados.rados.0.smithi045.stdout:2538: oid 23 updating version 0 to 1480 2024-04-26T00:35:03.660 INFO:tasks.rados.rados.0.smithi045.stdout:2538: oid 23 version 1480 is already newer than 1479 2024-04-26T00:35:03.660 INFO:tasks.rados.rados.0.smithi045.stdout:2538: oid 23 version 1480 is already newer than 1478 2024-04-26T00:35:03.660 INFO:tasks.rados.rados.0.smithi045.stdout:2538: oid 23 updating version 1480 to 1481 2024-04-26T00:35:03.660 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 1481 (ObjNum 918 snap 282 seq_num 918) dirty exists 2024-04-26T00:35:03.660 INFO:tasks.rados.rados.0.smithi045.stdout:2538: left oid 23 (ObjNum 918 snap 282 seq_num 918) 2024-04-26T00:35:03.660 INFO:tasks.rados.rados.0.smithi045.stdout:2541: finishing rollback tid 1 to smithi04526579-16 2024-04-26T00:35:03.660 INFO:tasks.rados.rados.0.smithi045.stdout:2543: expect (ObjNum 881 snap 273 seq_num 881) 2024-04-26T00:35:03.842 INFO:tasks.rados.rados.0.smithi045.stdout:2539: finishing rollback tid 0 to smithi04526579-35 2024-04-26T00:35:03.842 INFO:tasks.rados.rados.0.smithi045.stdout:2541: finishing rollback tid 2 to smithi04526579-16 2024-04-26T00:35:03.842 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 1161 (ObjNum 801 snap 246 seq_num 801) dirty exists 2024-04-26T00:35:03.842 INFO:tasks.rados.rados.0.smithi045.stdout:2542: finishing copy_from racing read to smithi04526579-47 2024-04-26T00:35:03.842 INFO:tasks.rados.rados.0.smithi045.stdout:2539: finishing rollback tid 1 to smithi04526579-35 2024-04-26T00:35:03.843 INFO:tasks.rados.rados.0.smithi045.stdout:2540: finishing copy_from racing read to smithi04526579-39 2024-04-26T00:35:03.843 INFO:tasks.rados.rados.0.smithi045.stdout:2539: finishing rollback tid 2 to smithi04526579-35 2024-04-26T00:35:03.843 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 1867 (ObjNum 812 snap 252 seq_num 812) dirty exists 2024-04-26T00:35:03.843 INFO:tasks.rados.rados.0.smithi045.stdout:2540: finishing copy_from to smithi04526579-39 2024-04-26T00:35:03.843 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 1977 (ObjNum 836 snap 260 seq_num 836) dirty exists 2024-04-26T00:35:03.843 INFO:tasks.rados.rados.0.smithi045.stdout:2538: done (8 left) 2024-04-26T00:35:03.843 INFO:tasks.rados.rados.0.smithi045.stdout:2539: done (7 left) 2024-04-26T00:35:03.843 INFO:tasks.rados.rados.0.smithi045.stdout:2540: done (6 left) 2024-04-26T00:35:03.843 INFO:tasks.rados.rados.0.smithi045.stdout:2541: done (5 left) 2024-04-26T00:35:03.843 INFO:tasks.rados.rados.0.smithi045.stdout:2543: done (4 left) 2024-04-26T00:35:03.843 INFO:tasks.rados.rados.0.smithi045.stdout:2546: done (3 left) 2024-04-26T00:35:03.843 INFO:tasks.rados.rados.0.smithi045.stdout:2547: read oid 15 snap -1 2024-04-26T00:35:03.844 INFO:tasks.rados.rados.0.smithi045.stdout:2547: expect (ObjNum 897 snap 276 seq_num 897) 2024-04-26T00:35:03.844 INFO:tasks.rados.rados.0.smithi045.stdout:2548: copy_from oid 26 from oid 30 current snap is 282 2024-04-26T00:35:03.844 INFO:tasks.rados.rados.0.smithi045.stdout:2544: finishing write tid 1 to smithi04526579-33 2024-04-26T00:35:03.844 INFO:tasks.rados.rados.0.smithi045.stdout:2544: finishing write tid 2 to smithi04526579-33 2024-04-26T00:35:03.844 INFO:tasks.rados.rados.0.smithi045.stdout:2544: finishing write tid 3 to smithi04526579-33 2024-04-26T00:35:03.844 INFO:tasks.rados.rados.0.smithi045.stdout:2545: expect (ObjNum 878 snap 273 seq_num 878) 2024-04-26T00:35:04.041 INFO:tasks.rados.rados.0.smithi045.stdout:2542: finishing copy_from to smithi04526579-47 2024-04-26T00:35:04.042 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 1932 (ObjNum 892 snap 275 seq_num 892) dirty exists 2024-04-26T00:35:04.042 INFO:tasks.rados.rados.0.smithi045.stdout:2544: finishing write tid 4 to smithi04526579-33 2024-04-26T00:35:04.042 INFO:tasks.rados.rados.0.smithi045.stdout:2544: finishing write tid 5 to smithi04526579-33 2024-04-26T00:35:04.042 INFO:tasks.rados.rados.0.smithi045.stdout:2544: oid 33 updating version 0 to 2237 2024-04-26T00:35:04.042 INFO:tasks.rados.rados.0.smithi045.stdout:2544: oid 33 version 2237 is already newer than 2236 2024-04-26T00:35:04.042 INFO:tasks.rados.rados.0.smithi045.stdout:2544: oid 33 version 2237 is already newer than 2234 2024-04-26T00:35:04.042 INFO:tasks.rados.rados.0.smithi045.stdout:2544: oid 33 version 2237 is already newer than 2235 2024-04-26T00:35:04.042 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 2237 (ObjNum 919 snap 282 seq_num 919) dirty exists 2024-04-26T00:35:04.042 INFO:tasks.rados.rados.0.smithi045.stdout:2544: left oid 33 (ObjNum 919 snap 282 seq_num 919) 2024-04-26T00:35:04.042 INFO:tasks.rados.rados.0.smithi045.stdout:2547: expect (ObjNum 897 snap 276 seq_num 897) 2024-04-26T00:35:04.165 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:35:04.216 INFO:tasks.rados.rados.0.smithi045.stdout:2548: finishing copy_from racing read to smithi04526579-26 2024-04-26T00:35:04.216 INFO:tasks.rados.rados.0.smithi045.stdout:2548: finishing copy_from to smithi04526579-26 2024-04-26T00:35:04.216 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 1073 (ObjNum 885 snap 274 seq_num 885) dirty exists 2024-04-26T00:35:04.216 INFO:tasks.rados.rados.0.smithi045.stdout:2542: done (4 left) 2024-04-26T00:35:04.216 INFO:tasks.rados.rados.0.smithi045.stdout:2544: done (3 left) 2024-04-26T00:35:04.216 INFO:tasks.rados.rados.0.smithi045.stdout:2545: done (2 left) 2024-04-26T00:35:04.216 INFO:tasks.rados.rados.0.smithi045.stdout:2547: done (1 left) 2024-04-26T00:35:04.217 INFO:tasks.rados.rados.0.smithi045.stdout:2548: done (0 left) 2024-04-26T00:35:04.217 INFO:tasks.rados.rados.0.smithi045.stdout:2549: snap_create 2024-04-26T00:35:04.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.302+0000 7fc4fee22640 10 monclient: get_monmap_and_config 2024-04-26T00:35:04.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.302+0000 7fc4fee22640 10 monclient: build_initial_monmap 2024-04-26T00:35:04.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.302+0000 7fc4fee22640 10 monclient: monmap: 2024-04-26T00:35:04.299 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:04.299 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:04.299 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:35:04.305512+0000 2024-04-26T00:35:04.299 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:35:04.305512+0000 2024-04-26T00:35:04.299 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:04.299 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:04.299 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:35:04.299 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:35:04.299 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:35:04.299 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:35:04.299 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:35:04.299 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:04.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.302+0000 7fc4fee22640 1 Processor -- start 2024-04-26T00:35:04.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.302+0000 7fc4fee22640 1 -- start start 2024-04-26T00:35:04.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.302+0000 7fc4fee22640 10 monclient: init 2024-04-26T00:35:04.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.302+0000 7fc4fee22640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:04.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.302+0000 7fc4fee22640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-04-26T00:35:04.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.302+0000 7fc4fee22640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc4f8136da0 0x7fc4f809bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:04.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.302+0000 7fc4fee22640 10 monclient(hunting): picked mon.noname-a con 0x7fc4f8136da0 addr v2:172.21.15.37:3300/0 2024-04-26T00:35:04.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.302+0000 7fc4fee22640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc4f809c410 addr v1:172.21.15.37:6789/0 2024-04-26T00:35:04.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.302+0000 7fc4fee22640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc4f8137890 addr v1:172.21.15.45:6789/0 2024-04-26T00:35:04.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.302+0000 7fc4fee22640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4f809b4f0 con 0x7fc4f809c410 2024-04-26T00:35:04.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.302+0000 7fc4fee22640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4f809b7d0 con 0x7fc4f8137890 2024-04-26T00:35:04.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.302+0000 7fc4fee22640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:04.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.302+0000 7fc4fee22640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fc4f809bb40 con 0x7fc4f8136da0 2024-04-26T00:35:04.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.302+0000 7fc4fee22640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:04.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.302+0000 7fc4fee22640 10 monclient(hunting): authenticate will time out at 1885.611229s 2024-04-26T00:35:04.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.302+0000 7fc4fcbbe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc4f8136da0 0x7fc4f809bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:04.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.302+0000 7fc4f7fff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fc4f809c410 0x7fc4f8137180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:42264/0 (socket says 172.21.15.37:42264) 2024-04-26T00:35:04.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.302+0000 7fc4fcbbe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc4f8136da0 0x7fc4f809bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52814/0 (socket says 172.21.15.37:52814) 2024-04-26T00:35:04.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.302+0000 7fc4f7fff640 1 -- 172.21.15.37:0/1013834508 learned_addr learned my addr 172.21.15.37:0/1013834508 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:35:04.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.302+0000 7fc4fcbbe640 10 monclient(hunting): get_auth_request con 0x7fc4f8136da0 auth_method 0 2024-04-26T00:35:04.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.302+0000 7fc4fcbbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:04.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.302+0000 7fc4fcbbe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:04.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.302+0000 7fc4fcbbe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:04.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.302+0000 7fc4fcbbe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:04.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.302+0000 7fc4fcbbe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:04.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.302+0000 7fc4fcbbe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:04.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.302+0000 7fc4fcbbe640 10 monclient(hunting): handle_auth_done global_id 144483 payload 274 2024-04-26T00:35:04.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.302+0000 7fc4fcbbe640 1 -- 172.21.15.37:0/1013834508 >> v1:172.21.15.45:6789/0 conn(0x7fc4f8137890 legacy=0x7fc4f813e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:04.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.302+0000 7fc4fcbbe640 1 -- 172.21.15.37:0/1013834508 >> v1:172.21.15.37:6789/0 conn(0x7fc4f809c410 legacy=0x7fc4f8137180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:04.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.302+0000 7fc4fcbbe640 10 monclient: _finish_hunting 0 2024-04-26T00:35:04.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.302+0000 7fc4fcbbe640 1 monclient: found mon.noname-a 2024-04-26T00:35:04.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.302+0000 7fc4fcbbe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:04.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.302+0000 7fc4fcbbe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:35:04.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.302+0000 7fc4fcbbe640 1 -- 172.21.15.37:0/1013834508 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4f806dbb0 con 0x7fc4f8136da0 2024-04-26T00:35:04.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fcbbe640 1 -- 172.21.15.37:0/1013834508 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc4f8136da0 msgr2=0x7fc4f809bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:04.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fcbbe640 1 --2- 172.21.15.37:0/1013834508 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc4f8136da0 0x7fc4f809bed0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc4e00097c0 tx=0x7fc4e001c2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=283b3ce9714bc490 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:04.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4f77fe640 1 -- 172.21.15.37:0/1013834508 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e00b0070 con 0x7fc4f8136da0 2024-04-26T00:35:04.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:04.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4f77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:35:04.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4f77fe640 10 monclient: dump: 2024-04-26T00:35:04.305 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:04.305 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:04.305 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:04.305 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:04.305 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:04.306 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:04.306 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:04.306 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:04.306 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:04.306 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:04.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4f77fe640 10 monclient: _finish_auth 0 2024-04-26T00:35:04.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4f77fe640 10 monclient: _check_auth_tickets 2024-04-26T00:35:04.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:04.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4f77fe640 1 -- 172.21.15.37:0/1013834508 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4e00a97f0 con 0x7fc4f8136da0 2024-04-26T00:35:04.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fee22640 5 monclient: authenticate success, global_id 144483 2024-04-26T00:35:04.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fee22640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:35:04.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4f77fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:04.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4f77fe640 1 -- 172.21.15.37:0/1013834508 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e00a9bc0 con 0x7fc4f8136da0 2024-04-26T00:35:04.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:04.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4f77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:04.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4f77fe640 10 monclient: dump: 2024-04-26T00:35:04.307 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:04.307 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:04.307 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:04.307 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:04.307 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:04.307 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:04.307 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:04.307 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:04.307 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:04.307 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:04.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fee22640 10 monclient: get_monmap_and_config success 2024-04-26T00:35:04.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fee22640 4 set_mon_vals no callback set 2024-04-26T00:35:04.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fee22640 10 monclient: shutdown 2024-04-26T00:35:04.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fee22640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:04.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fee22640 1 -- 172.21.15.37:0/1013834508 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc4f8136da0 msgr2=0x7fc4f809bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:04.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fee22640 1 --2- 172.21.15.37:0/1013834508 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc4f8136da0 0x7fc4f809bed0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc4e00097c0 tx=0x7fc4e001c2e0 comp rx=0 tx=0).stop 2024-04-26T00:35:04.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fee22640 1 -- 172.21.15.37:0/1013834508 shutdown_connections 2024-04-26T00:35:04.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fee22640 1 --2- 172.21.15.37:0/1013834508 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc4f8136da0 0x7fc4f809bed0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:04.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fee22640 1 -- 172.21.15.37:0/1013834508 >> 172.21.15.37:0/1013834508 conn(0x7fc4f80a5f00 msgr2=0x7fc4f80a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:04.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fee22640 1 -- 172.21.15.37:0/1013834508 shutdown_connections 2024-04-26T00:35:04.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fee22640 1 -- 172.21.15.37:0/1013834508 wait complete. 2024-04-26T00:35:04.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fee22640 10 monclient: build_initial_monmap 2024-04-26T00:35:04.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fee22640 10 monclient: monmap: 2024-04-26T00:35:04.313 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:04.313 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:35:04.313 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:35:04.313 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:35:04.313 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:04.313 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:35:04.313 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:35:04.313 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:35:04.313 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:35:04.314 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:04.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fee22640 1 Processor -- start 2024-04-26T00:35:04.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fee22640 1 -- start start 2024-04-26T00:35:04.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fee22640 10 monclient: init 2024-04-26T00:35:04.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fee22640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:04.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fee22640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:35:04.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fee22640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc4f809c410 0x7fc4f81cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:04.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fee22640 10 monclient(hunting): picked mon.noname-a con 0x7fc4f809c410 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:35:04.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fee22640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc4f8136da0 0x7fc4f8135cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:04.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fee22640 10 monclient(hunting): picked mon.noname-c con 0x7fc4f8136da0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:04.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fee22640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4f8137890 0x7fc4f813ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:04.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fee22640 10 monclient(hunting): picked mon.noname-b con 0x7fc4f8137890 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:35:04.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fee22640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:04.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fee22640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f8142a80 con 0x7fc4f809c410 2024-04-26T00:35:04.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fee22640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:04.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fee22640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f8142900 con 0x7fc4f8137890 2024-04-26T00:35:04.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fee22640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:04.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fee22640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc4f8142c00 con 0x7fc4f8136da0 2024-04-26T00:35:04.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fee22640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:04.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fee22640 10 monclient(hunting): authenticate will time out at 1885.618449s 2024-04-26T00:35:04.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4f7fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc4f8136da0 0x7fc4f8135cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:04.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fcbbe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc4f809c410 0x7fc4f81cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:04.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4f7fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc4f8136da0 0x7fc4f8135cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:57760/0 (socket says 172.21.15.37:57760) 2024-04-26T00:35:04.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fcbbe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc4f809c410 0x7fc4f81cedb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52818/0 (socket says 172.21.15.37:52818) 2024-04-26T00:35:04.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4f7fff640 1 -- 172.21.15.37:0/2822175061 learned_addr learned my addr 172.21.15.37:0/2822175061 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:04.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4f7fff640 10 monclient(hunting): get_auth_request con 0x7fc4f8136da0 auth_method 0 2024-04-26T00:35:04.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:04.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4f7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:04.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:04.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fcbbe640 10 monclient(hunting): get_auth_request con 0x7fc4f809c410 auth_method 0 2024-04-26T00:35:04.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fcbbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:04.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fcbbe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:04.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fcbbe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:04.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fcbbe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:04.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fcbbe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:04.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fcbbe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:04.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4f7fff640 1 -- 172.21.15.37:0/2822175061 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc4f8136da0 msgr2=0x7fc4f8135cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:35:04.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4f7fff640 1 -- 172.21.15.37:0/2822175061 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc4f8136da0 msgr2=0x7fc4f8135cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:35:04.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4f7fff640 1 --2- 172.21.15.37:0/2822175061 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc4f8136da0 0x7fc4f8135cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:35:04.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4f7fff640 1 --2- 172.21.15.37:0/2822175061 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc4f8136da0 0x7fc4f8135cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:35:04.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fd3bf640 1 --2- 172.21.15.37:0/2822175061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4f8137890 0x7fc4f813ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:04.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fd3bf640 10 monclient(hunting): get_auth_request con 0x7fc4f8137890 auth_method 0 2024-04-26T00:35:04.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fd3bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:04.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fd3bf640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:04.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fd3bf640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:04.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fcbbe640 10 monclient(hunting): handle_auth_done global_id 144486 payload 824 2024-04-26T00:35:04.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fcbbe640 1 -- 172.21.15.37:0/2822175061 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc4f8136da0 msgr2=0x7fc4f8135cd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:35:04.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fcbbe640 1 --2- 172.21.15.37:0/2822175061 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc4f8136da0 0x7fc4f8135cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:04.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fcbbe640 1 -- 172.21.15.37:0/2822175061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4f8137890 msgr2=0x7fc4f813ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:04.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fcbbe640 1 --2- 172.21.15.37:0/2822175061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4f8137890 0x7fc4f813ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:04.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fcbbe640 10 monclient: _finish_hunting 0 2024-04-26T00:35:04.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fcbbe640 1 monclient: found mon.noname-a 2024-04-26T00:35:04.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fcbbe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:04.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fcbbe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:35:04.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.310+0000 7fc4fcbbe640 1 -- 172.21.15.37:0/2822175061 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4f813c8a0 con 0x7fc4f809c410 2024-04-26T00:35:04.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.314+0000 7fc4fcbbe640 1 -- 172.21.15.37:0/2822175061 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc4f809c410 msgr2=0x7fc4f81cedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:04.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.314+0000 7fc4fcbbe640 1 --2- 172.21.15.37:0/2822175061 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc4f809c410 0x7fc4f81cedb0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc4e00ab000 tx=0x7fc4e001ab20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:04.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.314+0000 7fc4f57fa640 1 -- 172.21.15.37:0/2822175061 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e00b0050 con 0x7fc4f809c410 2024-04-26T00:35:04.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.314+0000 7fc4f57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:04.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.314+0000 7fc4f57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:35:04.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.314+0000 7fc4f57fa640 10 monclient: dump: 2024-04-26T00:35:04.318 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:04.318 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:04.318 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:04.318 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:04.318 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:04.318 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:04.318 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:04.318 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:04.318 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:04.318 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:04.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.314+0000 7fc4f57fa640 10 monclient: _finish_auth 0 2024-04-26T00:35:04.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.314+0000 7fc4f57fa640 10 monclient: _check_auth_tickets 2024-04-26T00:35:04.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.314+0000 7fc4f57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:04.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.314+0000 7fc4fee22640 5 monclient: authenticate success, global_id 144486 2024-04-26T00:35:04.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.314+0000 7fc4f57fa640 1 -- 172.21.15.37:0/2822175061 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4e00bc070 con 0x7fc4f809c410 2024-04-26T00:35:04.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.314+0000 7fc4fee22640 10 monclient: _renew_subs 2024-04-26T00:35:04.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.314+0000 7fc4fee22640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:04.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.314+0000 7fc4fee22640 1 -- 172.21.15.37:0/2822175061 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4f81cfca0 con 0x7fc4f809c410 2024-04-26T00:35:04.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.314+0000 7fc4f57fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:04.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.314+0000 7fc4f57fa640 1 -- 172.21.15.37:0/2822175061 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e00b7450 con 0x7fc4f809c410 2024-04-26T00:35:04.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.314+0000 7fc4f57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:04.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.314+0000 7fc4f57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:04.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.314+0000 7fc4f57fa640 10 monclient: dump: 2024-04-26T00:35:04.319 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:04.319 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:04.319 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:04.319 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:04.319 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:04.320 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:04.320 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:04.320 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:04.320 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:04.320 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:04.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.314+0000 7fc4f6ffd640 4 set_mon_vals no callback set 2024-04-26T00:35:04.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.314+0000 7fc4fee22640 10 monclient: _renew_subs 2024-04-26T00:35:04.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.314+0000 7fc4fee22640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:04.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.314+0000 7fc4fee22640 1 -- 172.21.15.37:0/2822175061 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4f813e2f0 con 0x7fc4f809c410 2024-04-26T00:35:04.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.314+0000 7fc4fee22640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:04.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.314+0000 7fc4fee22640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:04.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.314+0000 7fc4fee22640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:04.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.314+0000 7fc4fee22640 1 -- 172.21.15.37:0/2822175061 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4f8066f40 con 0x7fc4f809c410 2024-04-26T00:35:04.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.314+0000 7fc4f57fa640 1 -- 172.21.15.37:0/2822175061 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc4e00a96b0 con 0x7fc4f809c410 2024-04-26T00:35:04.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.314+0000 7fc4f57fa640 1 --2- 172.21.15.37:0/2822175061 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc4d8035e80 0x7fc4d8038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:04.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.314+0000 7fc4f57fa640 1 -- 172.21.15.37:0/2822175061 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(589..589 src has 370..589) ==== 3987+0+0 (secure 0 0 0) 0x7fc4e00e37e0 con 0x7fc4f809c410 2024-04-26T00:35:04.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.314+0000 7fc4f7fff640 1 --2- 172.21.15.37:0/2822175061 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc4d8035e80 0x7fc4d8038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:04.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.314+0000 7fc4f7fff640 10 monclient: get_auth_request con 0x7fc4d8035e80 auth_method 0 2024-04-26T00:35:04.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.318+0000 7fc4f7fff640 1 --2- 172.21.15.37:0/2822175061 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc4d8035e80 0x7fc4d8038340 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fc4f813e030 tx=0x7fc4e8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:04.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.318+0000 7fc4f57fa640 1 -- 172.21.15.37:0/2822175061 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc4e0015ff0 con 0x7fc4f809c410 2024-04-26T00:35:04.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.318+0000 7fc4f57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:04.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.318+0000 7fc4f57fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:35:04.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.542+0000 7fc4fee22640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:35:04.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.542+0000 7fc4fee22640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:35:04.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.542+0000 7fc4fee22640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:04.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.542+0000 7fc4fee22640 1 -- 172.21.15.37:0/2822175061 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc4f8075d70 con 0x7fc4f809c410 2024-04-26T00:35:04.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.542+0000 7fc4f57fa640 1 -- 172.21.15.37:0/2822175061 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7fc4e00c07c0 con 0x7fc4f809c410 2024-04-26T00:35:04.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.542+0000 7fc4f57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:35:04.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.542+0000 7fc4f57fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:35:04.541 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":112,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:35:04.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.546+0000 7fc4fee22640 1 -- 172.21.15.37:0/2822175061 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc4d8035e80 msgr2=0x7fc4d8038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:04.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.546+0000 7fc4fee22640 1 --2- 172.21.15.37:0/2822175061 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc4d8035e80 0x7fc4d8038340 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fc4f813e030 tx=0x7fc4e8009040 comp rx=0 tx=0).stop 2024-04-26T00:35:04.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.546+0000 7fc4fee22640 10 monclient: shutdown 2024-04-26T00:35:04.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.546+0000 7fc4fee22640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:04.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.546+0000 7fc4fee22640 1 -- 172.21.15.37:0/2822175061 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc4f809c410 msgr2=0x7fc4f81cedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:04.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.546+0000 7fc4fee22640 1 --2- 172.21.15.37:0/2822175061 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc4f809c410 0x7fc4f81cedb0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc4e00ab000 tx=0x7fc4e001ab20 comp rx=0 tx=0).stop 2024-04-26T00:35:04.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.546+0000 7fc4fee22640 1 -- 172.21.15.37:0/2822175061 shutdown_connections 2024-04-26T00:35:04.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.546+0000 7fc4fee22640 1 --2- 172.21.15.37:0/2822175061 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc4d8035e80 0x7fc4d8038340 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:04.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.546+0000 7fc4fee22640 1 --2- 172.21.15.37:0/2822175061 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4f8137890 0x7fc4f813ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:04.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.546+0000 7fc4fee22640 1 --2- 172.21.15.37:0/2822175061 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc4f8136da0 0x7fc4f8135cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:04.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.546+0000 7fc4fee22640 1 --2- 172.21.15.37:0/2822175061 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc4f809c410 0x7fc4f81cedb0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:04.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.546+0000 7fc4fee22640 1 -- 172.21.15.37:0/2822175061 >> 172.21.15.37:0/2822175061 conn(0x7fc4f80a5f00 msgr2=0x7fc4f80a7a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:04.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.546+0000 7fc4fee22640 1 -- 172.21.15.37:0/2822175061 shutdown_connections 2024-04-26T00:35:04.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:04.546+0000 7fc4fee22640 1 -- 172.21.15.37:0/2822175061 wait complete. 2024-04-26T00:35:04.590 INFO:tasks.rados.rados.0.smithi045.stdout:2549: done (0 left) 2024-04-26T00:35:04.590 INFO:tasks.rados.rados.0.smithi045.stdout:2550: snap_remove snap 282 2024-04-26T00:35:05.595 INFO:tasks.rados.rados.0.smithi045.stdout:2550: done (0 left) 2024-04-26T00:35:05.596 INFO:tasks.rados.rados.0.smithi045.stdout:2551: read oid 48 snap 278 2024-04-26T00:35:05.596 INFO:tasks.rados.rados.0.smithi045.stdout:2551: expect (ObjNum 836 snap 260 seq_num 836) 2024-04-26T00:35:05.596 INFO:tasks.rados.rados.0.smithi045.stdout:2552: rollback oid 36 current snap is 283 2024-04-26T00:35:05.596 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 36 to 281 2024-04-26T00:35:05.596 INFO:tasks.rados.rados.0.smithi045.stdout:2553: write (excl) oid 16 current snap is 283 2024-04-26T00:35:05.596 INFO:tasks.rados.rados.0.smithi045.stdout:2553: seq_num 921 ranges {679247=583042,2044857=609984,3192341=286906} 2024-04-26T00:35:05.603 INFO:tasks.rados.rados.0.smithi045.stdout:2553: writing smithi04526579-16 from 679247 to 1262289 tid 1 2024-04-26T00:35:05.622 INFO:tasks.rados.rados.0.smithi045.stdout:2553: writing smithi04526579-16 from 2044857 to 2654841 tid 2 2024-04-26T00:35:05.634 INFO:tasks.rados.rados.0.smithi045.stdout:2553: writing smithi04526579-16 from 3192341 to 3479247 tid 3 2024-04-26T00:35:05.635 INFO:tasks.rados.rados.0.smithi045.stdout:2554: snap_create 2024-04-26T00:35:05.635 INFO:tasks.rados.rados.0.smithi045.stdout:2552: finishing rollback tid 0 to smithi04526579-36 2024-04-26T00:35:05.635 INFO:tasks.rados.rados.0.smithi045.stdout:2552: finishing rollback tid 1 to smithi04526579-36 2024-04-26T00:35:05.635 INFO:tasks.rados.rados.0.smithi045.stdout:2552: finishing rollback tid 2 to smithi04526579-36 2024-04-26T00:35:05.635 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 1481 (ObjNum 896 snap 276 seq_num 896) dirty exists 2024-04-26T00:35:05.635 INFO:tasks.rados.rados.0.smithi045.stdout:2551: expect (ObjNum 836 snap 260 seq_num 836) 2024-04-26T00:35:05.811 INFO:tasks.rados.rados.0.smithi045.stdout:2553: finishing write tid 1 to smithi04526579-16 2024-04-26T00:35:05.811 INFO:tasks.rados.rados.0.smithi045.stdout:2553: finishing write tid 2 to smithi04526579-16 2024-04-26T00:35:05.811 INFO:tasks.rados.rados.0.smithi045.stdout:2553: finishing write tid 3 to smithi04526579-16 2024-04-26T00:35:05.811 INFO:tasks.rados.rados.0.smithi045.stdout:2553: finishing write tid 4 to smithi04526579-16 2024-04-26T00:35:05.811 INFO:tasks.rados.rados.0.smithi045.stdout:2553: finishing write tid 5 to smithi04526579-16 2024-04-26T00:35:05.811 INFO:tasks.rados.rados.0.smithi045.stdout:2553: oid 16 updating version 0 to 1164 2024-04-26T00:35:05.811 INFO:tasks.rados.rados.0.smithi045.stdout:2553: oid 16 updating version 1164 to 1165 2024-04-26T00:35:05.811 INFO:tasks.rados.rados.0.smithi045.stdout:2553: oid 16 version 1165 is already newer than 1163 2024-04-26T00:35:05.811 INFO:tasks.rados.rados.0.smithi045.stdout:2553: oid 16 version 1165 is already newer than 1162 2024-04-26T00:35:05.811 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 1165 (ObjNum 921 snap 283 seq_num 921) dirty exists 2024-04-26T00:35:05.811 INFO:tasks.rados.rados.0.smithi045.stdout:2553: left oid 16 (ObjNum 921 snap 283 seq_num 921) 2024-04-26T00:35:06.611 INFO:tasks.rados.rados.0.smithi045.stdout:2551: done (3 left) 2024-04-26T00:35:06.611 INFO:tasks.rados.rados.0.smithi045.stdout:2552: done (2 left) 2024-04-26T00:35:06.611 INFO:tasks.rados.rados.0.smithi045.stdout:2553: done (1 left) 2024-04-26T00:35:06.611 INFO:tasks.rados.rados.0.smithi045.stdout:2554: done (0 left) 2024-04-26T00:35:06.611 INFO:tasks.rados.rados.0.smithi045.stdout:2555: snap_create 2024-04-26T00:35:07.562 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:35:07.620 INFO:tasks.rados.rados.0.smithi045.stdout:2555: done (0 left) 2024-04-26T00:35:07.620 INFO:tasks.rados.rados.0.smithi045.stdout:2556: copy_from oid 23 from oid 49 current snap is 285 2024-04-26T00:35:07.620 INFO:tasks.rados.rados.0.smithi045.stdout:2557: snap_create 2024-04-26T00:35:07.651 INFO:tasks.rados.rados.0.smithi045.stdout:2556: finishing copy_from racing read to smithi04526579-23 2024-04-26T00:35:07.654 INFO:tasks.rados.rados.0.smithi045.stdout:2556: finishing copy_from to smithi04526579-23 2024-04-26T00:35:07.654 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 1987 (ObjNum 731 snap 224 seq_num 731) dirty exists 2024-04-26T00:35:07.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.701+0000 7f9e7f295640 10 monclient: get_monmap_and_config 2024-04-26T00:35:07.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.701+0000 7f9e7f295640 10 monclient: build_initial_monmap 2024-04-26T00:35:07.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.701+0000 7f9e7f295640 10 monclient: monmap: 2024-04-26T00:35:07.700 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:07.700 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:07.700 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:35:07.707098+0000 2024-04-26T00:35:07.700 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:35:07.707098+0000 2024-04-26T00:35:07.701 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:07.701 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:07.701 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:35:07.701 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:35:07.701 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:35:07.701 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:35:07.701 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:35:07.701 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:07.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7f295640 1 Processor -- start 2024-04-26T00:35:07.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7f295640 1 -- start start 2024-04-26T00:35:07.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7f295640 10 monclient: init 2024-04-26T00:35:07.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7f295640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:07.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7f295640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-04-26T00:35:07.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7f295640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e78136da0 0x7f9e7809bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:07.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7f295640 10 monclient(hunting): picked mon.noname-b con 0x7f9e78136da0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:07.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7f295640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9e7809c410 0x7f9e78137180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:07.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7f295640 10 monclient(hunting): picked mon.noname-a con 0x7f9e7809c410 addr v2:172.21.15.37:3300/0 2024-04-26T00:35:07.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7f295640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9e781376c0 addr v1:172.21.15.37:6789/0 2024-04-26T00:35:07.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7f295640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e7809b4f0 con 0x7f9e781376c0 2024-04-26T00:35:07.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7f295640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:07.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7f295640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f9e7809b210 con 0x7f9e7809c410 2024-04-26T00:35:07.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7f295640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:07.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7f295640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9e7809b7d0 con 0x7f9e78136da0 2024-04-26T00:35:07.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7f295640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:07.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7f295640 10 monclient(hunting): authenticate will time out at 1889.012886s 2024-04-26T00:35:07.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7c830640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9e7809c410 0x7f9e78137180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:07.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7d031640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e78136da0 0x7f9e7809bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:07.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7c830640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9e7809c410 0x7f9e78137180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52824/0 (socket says 172.21.15.37:52824) 2024-04-26T00:35:07.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7c830640 1 -- 172.21.15.37:0/2943873426 learned_addr learned my addr 172.21.15.37:0/2943873426 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:07.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7d031640 10 monclient(hunting): get_auth_request con 0x7f9e78136da0 auth_method 0 2024-04-26T00:35:07.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7d031640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:07.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7d031640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:07.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7d031640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:07.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7d832640 1 --1- 172.21.15.37:0/2943873426 >> v1:172.21.15.37:6789/0 conn(0x7f9e781376c0 0x7f9e78133b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:42276/0 (socket says 172.21.15.37:42276) 2024-04-26T00:35:07.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7c830640 10 monclient(hunting): get_auth_request con 0x7f9e7809c410 auth_method 0 2024-04-26T00:35:07.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7c830640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:07.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7c830640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:07.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7c830640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:07.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7d031640 1 -- 172.21.15.37:0/2943873426 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e78136da0 msgr2=0x7f9e7809bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:35:07.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7d031640 1 -- 172.21.15.37:0/2943873426 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e78136da0 msgr2=0x7f9e7809bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:35:07.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7d031640 1 --2- 172.21.15.37:0/2943873426 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e78136da0 0x7f9e7809bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:35:07.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7c830640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:07.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7c830640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:07.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7c830640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:07.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7d031640 1 --2- 172.21.15.37:0/2943873426 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e78136da0 0x7f9e7809bed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:35:07.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7c830640 10 monclient(hunting): handle_auth_done global_id 144489 payload 274 2024-04-26T00:35:07.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7c830640 1 -- 172.21.15.37:0/2943873426 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e78136da0 msgr2=0x7f9e7809bed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:35:07.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7c830640 1 --2- 172.21.15.37:0/2943873426 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e78136da0 0x7f9e7809bed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:07.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7c830640 1 -- 172.21.15.37:0/2943873426 >> v1:172.21.15.37:6789/0 conn(0x7f9e781376c0 legacy=0x7f9e78133b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:07.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7c830640 10 monclient: _finish_hunting 0 2024-04-26T00:35:07.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7c830640 1 monclient: found mon.noname-a 2024-04-26T00:35:07.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7c830640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:07.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7c830640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:35:07.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.705+0000 7f9e7c830640 1 -- 172.21.15.37:0/2943873426 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e78137b30 con 0x7f9e7809c410 2024-04-26T00:35:07.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.709+0000 7f9e7c830640 1 -- 172.21.15.37:0/2943873426 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e7809c410 msgr2=0x7f9e78137180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:07.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.709+0000 7f9e7c830640 1 --2- 172.21.15.37:0/2943873426 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e7809c410 0x7f9e78137180 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9e74009870 tx=0x7f9e7401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e32cd3359b264ab server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:07.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.709+0000 7f9e6bfff640 1 -- 172.21.15.37:0/2943873426 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e740b0070 con 0x7f9e7809c410 2024-04-26T00:35:07.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.709+0000 7f9e6bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:07.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.709+0000 7f9e6bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:35:07.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.709+0000 7f9e6bfff640 10 monclient: dump: 2024-04-26T00:35:07.711 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:07.711 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:07.711 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:07.711 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:07.711 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:07.711 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:07.711 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:07.712 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:07.712 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:07.712 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:07.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.709+0000 7f9e6bfff640 10 monclient: _finish_auth 0 2024-04-26T00:35:07.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.709+0000 7f9e6bfff640 10 monclient: _check_auth_tickets 2024-04-26T00:35:07.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.709+0000 7f9e6bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:07.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.709+0000 7f9e6bfff640 1 -- 172.21.15.37:0/2943873426 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e740ab600 con 0x7f9e7809c410 2024-04-26T00:35:07.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.709+0000 7f9e6bfff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:07.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.709+0000 7f9e6bfff640 1 -- 172.21.15.37:0/2943873426 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e740ab9d0 con 0x7f9e7809c410 2024-04-26T00:35:07.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.709+0000 7f9e7f295640 5 monclient: authenticate success, global_id 144489 2024-04-26T00:35:07.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.709+0000 7f9e7f295640 10 monclient: get_monmap_and_config success 2024-04-26T00:35:07.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.709+0000 7f9e7f295640 4 set_mon_vals no callback set 2024-04-26T00:35:07.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.709+0000 7f9e6bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:07.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.709+0000 7f9e6bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:07.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.709+0000 7f9e6bfff640 10 monclient: dump: 2024-04-26T00:35:07.713 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:07.713 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:07.713 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:07.713 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:07.713 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:07.713 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:07.713 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:07.713 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:07.713 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:07.713 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:07.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.709+0000 7f9e7f295640 10 monclient: shutdown 2024-04-26T00:35:07.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.709+0000 7f9e7f295640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:07.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.709+0000 7f9e7f295640 1 -- 172.21.15.37:0/2943873426 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e7809c410 msgr2=0x7f9e78137180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:07.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.709+0000 7f9e7f295640 1 --2- 172.21.15.37:0/2943873426 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e7809c410 0x7f9e78137180 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9e74009870 tx=0x7f9e7401c3e0 comp rx=0 tx=0).stop 2024-04-26T00:35:07.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.709+0000 7f9e7f295640 1 -- 172.21.15.37:0/2943873426 shutdown_connections 2024-04-26T00:35:07.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.709+0000 7f9e7f295640 1 --2- 172.21.15.37:0/2943873426 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e7809c410 0x7f9e78137180 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:07.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.709+0000 7f9e7f295640 1 --2- 172.21.15.37:0/2943873426 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e78136da0 0x7f9e7809bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:07.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.709+0000 7f9e7f295640 1 -- 172.21.15.37:0/2943873426 >> 172.21.15.37:0/2943873426 conn(0x7f9e780a5f00 msgr2=0x7f9e780a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:07.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.709+0000 7f9e7f295640 1 -- 172.21.15.37:0/2943873426 shutdown_connections 2024-04-26T00:35:07.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.709+0000 7f9e7f295640 1 -- 172.21.15.37:0/2943873426 wait complete. 2024-04-26T00:35:07.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.709+0000 7f9e7f295640 10 monclient: build_initial_monmap 2024-04-26T00:35:07.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.709+0000 7f9e7f295640 10 monclient: monmap: 2024-04-26T00:35:07.714 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:07.714 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:35:07.714 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:35:07.714 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:35:07.714 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:07.714 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:35:07.714 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:35:07.715 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:35:07.715 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:35:07.715 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:07.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.709+0000 7f9e7f295640 1 Processor -- start 2024-04-26T00:35:07.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.709+0000 7f9e7f295640 1 -- start start 2024-04-26T00:35:07.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.709+0000 7f9e7f295640 10 monclient: init 2024-04-26T00:35:07.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.709+0000 7f9e7f295640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:07.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.709+0000 7f9e7f295640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:35:07.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.709+0000 7f9e7f295640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e7809c410 0x7f9e7812cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:07.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7f295640 10 monclient(hunting): picked mon.noname-b con 0x7f9e7809c410 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:35:07.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7f295640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e78136da0 0x7f9e7812d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:07.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7f295640 10 monclient(hunting): picked mon.noname-a con 0x7f9e78136da0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:35:07.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7f295640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e781376c0 0x7f9e781cf2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:07.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7f295640 10 monclient(hunting): picked mon.noname-c con 0x7f9e781376c0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:07.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7f295640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:07.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7f295640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9e78141ea0 con 0x7f9e78136da0 2024-04-26T00:35:07.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7f295640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:07.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7f295640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9e78141d20 con 0x7f9e7809c410 2024-04-26T00:35:07.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7f295640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:07.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7f295640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9e78142020 con 0x7f9e781376c0 2024-04-26T00:35:07.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7f295640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:07.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7f295640 10 monclient(hunting): authenticate will time out at 1889.020085s 2024-04-26T00:35:07.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7d832640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e781376c0 0x7f9e781cf2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:07.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7d832640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e781376c0 0x7f9e781cf2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:57794/0 (socket says 172.21.15.37:57794) 2024-04-26T00:35:07.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7d832640 1 -- 172.21.15.37:0/2764477907 learned_addr learned my addr 172.21.15.37:0/2764477907 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:07.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7d832640 10 monclient(hunting): get_auth_request con 0x7f9e781376c0 auth_method 0 2024-04-26T00:35:07.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7d832640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:07.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7d832640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:07.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7d832640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:07.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7c830640 1 --2- 172.21.15.37:0/2764477907 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e78136da0 0x7f9e7812d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:07.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7c830640 10 monclient(hunting): get_auth_request con 0x7f9e78136da0 auth_method 0 2024-04-26T00:35:07.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7c830640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:07.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7c830640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:07.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7c830640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:07.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7c830640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:07.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7c830640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:07.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7c830640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:07.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7d832640 1 -- 172.21.15.37:0/2764477907 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e781376c0 msgr2=0x7f9e781cf2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:35:07.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7d832640 1 -- 172.21.15.37:0/2764477907 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e781376c0 msgr2=0x7f9e781cf2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:35:07.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7d832640 1 --2- 172.21.15.37:0/2764477907 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e781376c0 0x7f9e781cf2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:35:07.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7d031640 1 --2- 172.21.15.37:0/2764477907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e7809c410 0x7f9e7812cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:07.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7d832640 1 --2- 172.21.15.37:0/2764477907 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e781376c0 0x7f9e781cf2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:35:07.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7d031640 10 monclient(hunting): get_auth_request con 0x7f9e7809c410 auth_method 0 2024-04-26T00:35:07.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7d031640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:07.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7d031640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:07.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7d031640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:07.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7c830640 10 monclient(hunting): handle_auth_done global_id 144492 payload 824 2024-04-26T00:35:07.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7c830640 1 -- 172.21.15.37:0/2764477907 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e781376c0 msgr2=0x7f9e781cf2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:35:07.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7c830640 1 --2- 172.21.15.37:0/2764477907 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e781376c0 0x7f9e781cf2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:07.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7c830640 1 -- 172.21.15.37:0/2764477907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e7809c410 msgr2=0x7f9e7812cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:07.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7c830640 1 --2- 172.21.15.37:0/2764477907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e7809c410 0x7f9e7812cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:07.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7c830640 10 monclient: _finish_hunting 0 2024-04-26T00:35:07.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7c830640 1 monclient: found mon.noname-a 2024-04-26T00:35:07.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7c830640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:07.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7c830640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:35:07.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7c830640 1 -- 172.21.15.37:0/2764477907 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e7809aca0 con 0x7f9e78136da0 2024-04-26T00:35:07.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7c830640 1 -- 172.21.15.37:0/2764477907 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e78136da0 msgr2=0x7f9e7812d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:07.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7c830640 1 --2- 172.21.15.37:0/2764477907 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e78136da0 0x7f9e7812d440 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9e740ab000 tx=0x7f9e74019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:07.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e69ffb640 1 -- 172.21.15.37:0/2764477907 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e740b0040 con 0x7f9e78136da0 2024-04-26T00:35:07.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e69ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:07.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e69ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:35:07.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e69ffb640 10 monclient: dump: 2024-04-26T00:35:07.721 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:07.721 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:07.721 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:07.721 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:07.721 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:07.721 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:07.722 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:07.722 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:07.722 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:07.722 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:07.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e69ffb640 10 monclient: _finish_auth 0 2024-04-26T00:35:07.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e69ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:35:07.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e69ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:07.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e69ffb640 1 -- 172.21.15.37:0/2764477907 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e740b37b0 con 0x7f9e78136da0 2024-04-26T00:35:07.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e69ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:07.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e69ffb640 1 -- 172.21.15.37:0/2764477907 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e740b3ba0 con 0x7f9e78136da0 2024-04-26T00:35:07.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e69ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:07.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e69ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:07.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e69ffb640 10 monclient: dump: 2024-04-26T00:35:07.722 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:07.722 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:07.723 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:07.723 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:07.723 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:07.723 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:07.723 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:07.723 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:07.723 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:07.723 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:07.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7f295640 5 monclient: authenticate success, global_id 144492 2024-04-26T00:35:07.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7f295640 10 monclient: _renew_subs 2024-04-26T00:35:07.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e6affd640 4 set_mon_vals no callback set 2024-04-26T00:35:07.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7f295640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:07.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7f295640 1 -- 172.21.15.37:0/2764477907 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e780a8cf0 con 0x7f9e78136da0 2024-04-26T00:35:07.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7f295640 10 monclient: _renew_subs 2024-04-26T00:35:07.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7f295640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:07.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7f295640 1 -- 172.21.15.37:0/2764477907 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e7809ee00 con 0x7f9e78136da0 2024-04-26T00:35:07.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.713+0000 7f9e7f295640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:07.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.717+0000 7f9e7f295640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:07.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.717+0000 7f9e7f295640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:07.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.717+0000 7f9e7f295640 1 -- 172.21.15.37:0/2764477907 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e78066f40 con 0x7f9e78136da0 2024-04-26T00:35:07.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.717+0000 7f9e69ffb640 1 -- 172.21.15.37:0/2764477907 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9e740bb060 con 0x7f9e78136da0 2024-04-26T00:35:07.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.717+0000 7f9e69ffb640 1 --2- 172.21.15.37:0/2764477907 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9e5c035eb0 0x7f9e5c038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:07.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.717+0000 7f9e69ffb640 1 -- 172.21.15.37:0/2764477907 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(593..593 src has 370..593) ==== 3915+0+0 (secure 0 0 0) 0x7f9e740e1bb0 con 0x7f9e78136da0 2024-04-26T00:35:07.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.717+0000 7f9e7d031640 1 --2- 172.21.15.37:0/2764477907 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9e5c035eb0 0x7f9e5c038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:07.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.717+0000 7f9e7d031640 10 monclient: get_auth_request con 0x7f9e5c035eb0 auth_method 0 2024-04-26T00:35:07.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.717+0000 7f9e7d031640 1 --2- 172.21.15.37:0/2764477907 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9e5c035eb0 0x7f9e5c038370 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c009610 tx=0x7f9e6c00b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:07.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.721+0000 7f9e69ffb640 1 -- 172.21.15.37:0/2764477907 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9e7809aca0 con 0x7f9e78136da0 2024-04-26T00:35:07.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.721+0000 7f9e69ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:07.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.721+0000 7f9e69ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:35:07.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.945+0000 7f9e7f295640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:35:07.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.945+0000 7f9e7f295640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:35:07.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.945+0000 7f9e7f295640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:07.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.945+0000 7f9e7f295640 1 -- 172.21.15.37:0/2764477907 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9e78075d70 con 0x7f9e78136da0 2024-04-26T00:35:07.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.945+0000 7f9e69ffb640 1 -- 172.21.15.37:0/2764477907 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f9e740160f0 con 0x7f9e78136da0 2024-04-26T00:35:07.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.945+0000 7f9e69ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:35:07.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.945+0000 7f9e69ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:35:07.943 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":114,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:35:07.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.945+0000 7f9e7f295640 1 -- 172.21.15.37:0/2764477907 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9e5c035eb0 msgr2=0x7f9e5c038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:07.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.945+0000 7f9e7f295640 1 --2- 172.21.15.37:0/2764477907 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9e5c035eb0 0x7f9e5c038370 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c009610 tx=0x7f9e6c00b040 comp rx=0 tx=0).stop 2024-04-26T00:35:07.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.945+0000 7f9e7f295640 10 monclient: shutdown 2024-04-26T00:35:07.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.945+0000 7f9e7f295640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:07.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.945+0000 7f9e7f295640 1 -- 172.21.15.37:0/2764477907 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e78136da0 msgr2=0x7f9e7812d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:07.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.945+0000 7f9e7f295640 1 --2- 172.21.15.37:0/2764477907 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e78136da0 0x7f9e7812d440 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9e740ab000 tx=0x7f9e74019180 comp rx=0 tx=0).stop 2024-04-26T00:35:07.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.945+0000 7f9e7f295640 1 -- 172.21.15.37:0/2764477907 shutdown_connections 2024-04-26T00:35:07.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.945+0000 7f9e7f295640 1 --2- 172.21.15.37:0/2764477907 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9e5c035eb0 0x7f9e5c038370 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:07.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.945+0000 7f9e7f295640 1 --2- 172.21.15.37:0/2764477907 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e781376c0 0x7f9e781cf2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:07.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.945+0000 7f9e7f295640 1 --2- 172.21.15.37:0/2764477907 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e78136da0 0x7f9e7812d440 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:07.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.945+0000 7f9e7f295640 1 --2- 172.21.15.37:0/2764477907 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e7809c410 0x7f9e7812cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:07.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.945+0000 7f9e7f295640 1 -- 172.21.15.37:0/2764477907 >> 172.21.15.37:0/2764477907 conn(0x7f9e780a5f00 msgr2=0x7f9e78135550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:07.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.945+0000 7f9e7f295640 1 -- 172.21.15.37:0/2764477907 shutdown_connections 2024-04-26T00:35:07.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:07.945+0000 7f9e7f295640 1 -- 172.21.15.37:0/2764477907 wait complete. 2024-04-26T00:35:07.962 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:35:07.962 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:35:07.962 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:35:08.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.089+0000 7f7f14dc5640 10 monclient: get_monmap_and_config 2024-04-26T00:35:08.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.089+0000 7f7f14dc5640 10 monclient: build_initial_monmap 2024-04-26T00:35:08.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.089+0000 7f7f14dc5640 10 monclient: monmap: 2024-04-26T00:35:08.088 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:08.089 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:08.089 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:35:08.094984+0000 2024-04-26T00:35:08.089 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:35:08.094984+0000 2024-04-26T00:35:08.089 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:08.089 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:08.089 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:35:08.089 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:35:08.089 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:35:08.089 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:35:08.089 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:35:08.089 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:08.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f14dc5640 1 Processor -- start 2024-04-26T00:35:08.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f14dc5640 1 -- start start 2024-04-26T00:35:08.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f14dc5640 10 monclient: init 2024-04-26T00:35:08.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f14dc5640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:08.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f14dc5640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-04-26T00:35:08.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f14dc5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7f100a3d90 addr v1:172.21.15.45:6789/0 2024-04-26T00:35:08.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f14dc5640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f7f100a4170 0x7f7f100a45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:08.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f14dc5640 10 monclient(hunting): picked mon.noname-a con 0x7f7f100a4170 addr v2:172.21.15.37:3300/0 2024-04-26T00:35:08.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f14dc5640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7f1013c3e0 0x7f7f1013e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:08.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f14dc5640 10 monclient(hunting): picked mon.noname-b con 0x7f7f1013c3e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:08.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f14dc5640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f100a5150 con 0x7f7f100a3d90 2024-04-26T00:35:08.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f14dc5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:08.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f14dc5640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f7f1009c650 con 0x7f7f100a4170 2024-04-26T00:35:08.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f14dc5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:08.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f14dc5640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f7f1009b280 con 0x7f7f1013c3e0 2024-04-26T00:35:08.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f14dc5640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:08.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f14dc5640 10 monclient(hunting): authenticate will time out at 1889.401192s 2024-04-26T00:35:08.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f0ed9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7f1013c3e0 0x7f7f1013e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:08.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f07fff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f7f100a4170 0x7f7f100a45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:08.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f0ed9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7f1013c3e0 0x7f7f1013e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:57802/0 (socket says 172.21.15.37:57802) 2024-04-26T00:35:08.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f0ed9d640 1 -- 172.21.15.37:0/3002766140 learned_addr learned my addr 172.21.15.37:0/3002766140 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:08.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f0ed9d640 10 monclient(hunting): get_auth_request con 0x7f7f1013c3e0 auth_method 0 2024-04-26T00:35:08.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f0ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:08.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f0ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:08.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f0ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:08.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f07fff640 10 monclient(hunting): get_auth_request con 0x7f7f100a4170 auth_method 0 2024-04-26T00:35:08.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:08.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f07fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:08.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f07fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:08.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f0ed9d640 1 -- 172.21.15.37:0/3002766140 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7f1013c3e0 msgr2=0x7f7f1013e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:35:08.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f0ed9d640 1 -- 172.21.15.37:0/3002766140 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7f1013c3e0 msgr2=0x7f7f1013e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:35:08.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:08.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:08.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f0ed9d640 1 --2- 172.21.15.37:0/3002766140 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7f1013c3e0 0x7f7f1013e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:35:08.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:08.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f0ed9d640 1 --2- 172.21.15.37:0/3002766140 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7f1013c3e0 0x7f7f1013e7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:35:08.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f07fff640 10 monclient(hunting): handle_auth_done global_id 144495 payload 274 2024-04-26T00:35:08.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f07fff640 1 -- 172.21.15.37:0/3002766140 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7f1013c3e0 msgr2=0x7f7f1013e7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:35:08.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f07fff640 1 --2- 172.21.15.37:0/3002766140 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7f1013c3e0 0x7f7f1013e7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:08.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f07fff640 1 -- 172.21.15.37:0/3002766140 >> v1:172.21.15.45:6789/0 conn(0x7f7f100a3d90 legacy=0x7f7f1009bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:08.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f07fff640 10 monclient: _finish_hunting 0 2024-04-26T00:35:08.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f07fff640 1 monclient: found mon.noname-a 2024-04-26T00:35:08.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:08.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f07fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:35:08.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.093+0000 7f7f07fff640 1 -- 172.21.15.37:0/3002766140 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f1009b560 con 0x7f7f100a4170 2024-04-26T00:35:08.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.097+0000 7f7f07fff640 1 -- 172.21.15.37:0/3002766140 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f100a4170 msgr2=0x7f7f100a45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:08.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.097+0000 7f7f07fff640 1 --2- 172.21.15.37:0/3002766140 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f100a4170 0x7f7f100a45e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7ef80097c0 tx=0x7f7ef801c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11c526cf85d9ba5c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:08.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.097+0000 7f7f0dd9b640 1 -- 172.21.15.37:0/3002766140 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef80b0070 con 0x7f7f100a4170 2024-04-26T00:35:08.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.097+0000 7f7f0dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:08.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.097+0000 7f7f0dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:35:08.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.097+0000 7f7f0dd9b640 10 monclient: dump: 2024-04-26T00:35:08.100 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:08.100 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:08.100 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:08.100 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:08.100 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:08.100 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:08.100 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:08.100 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:08.100 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:08.101 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:08.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.097+0000 7f7f0dd9b640 10 monclient: _finish_auth 0 2024-04-26T00:35:08.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.097+0000 7f7f0dd9b640 10 monclient: _check_auth_tickets 2024-04-26T00:35:08.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.097+0000 7f7f0dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:08.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.097+0000 7f7f0dd9b640 1 -- 172.21.15.37:0/3002766140 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ef80ab600 con 0x7f7f100a4170 2024-04-26T00:35:08.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.097+0000 7f7f14dc5640 5 monclient: authenticate success, global_id 144495 2024-04-26T00:35:08.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.097+0000 7f7f14dc5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:35:08.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.097+0000 7f7f0dd9b640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:08.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.097+0000 7f7f14dc5640 10 monclient: get_monmap_and_config success 2024-04-26T00:35:08.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.097+0000 7f7f14dc5640 4 set_mon_vals no callback set 2024-04-26T00:35:08.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.097+0000 7f7f0dd9b640 1 -- 172.21.15.37:0/3002766140 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef80ab9d0 con 0x7f7f100a4170 2024-04-26T00:35:08.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.097+0000 7f7f0dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:08.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.097+0000 7f7f0dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:08.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.097+0000 7f7f0dd9b640 10 monclient: dump: 2024-04-26T00:35:08.101 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:08.102 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:08.102 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:08.102 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:08.102 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:08.102 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:08.102 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:08.102 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:08.102 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:08.102 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:08.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.097+0000 7f7f14dc5640 10 monclient: shutdown 2024-04-26T00:35:08.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.097+0000 7f7f14dc5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:08.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.097+0000 7f7f14dc5640 1 -- 172.21.15.37:0/3002766140 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f100a4170 msgr2=0x7f7f100a45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:08.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.097+0000 7f7f14dc5640 1 --2- 172.21.15.37:0/3002766140 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f100a4170 0x7f7f100a45e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7ef80097c0 tx=0x7f7ef801c320 comp rx=0 tx=0).stop 2024-04-26T00:35:08.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.097+0000 7f7f14dc5640 1 -- 172.21.15.37:0/3002766140 shutdown_connections 2024-04-26T00:35:08.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.097+0000 7f7f14dc5640 1 --2- 172.21.15.37:0/3002766140 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7f1013c3e0 0x7f7f1013e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:08.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.097+0000 7f7f14dc5640 1 --2- 172.21.15.37:0/3002766140 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f100a4170 0x7f7f100a45e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:08.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.097+0000 7f7f14dc5640 1 -- 172.21.15.37:0/3002766140 >> 172.21.15.37:0/3002766140 conn(0x7f7f100aa0a0 msgr2=0x7f7f100a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:08.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.097+0000 7f7f14dc5640 1 -- 172.21.15.37:0/3002766140 shutdown_connections 2024-04-26T00:35:08.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.097+0000 7f7f14dc5640 1 -- 172.21.15.37:0/3002766140 wait complete. 2024-04-26T00:35:08.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.097+0000 7f7f14dc5640 10 monclient: build_initial_monmap 2024-04-26T00:35:08.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.097+0000 7f7f14dc5640 10 monclient: monmap: 2024-04-26T00:35:08.103 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:08.103 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:35:08.103 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:35:08.104 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:35:08.104 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:08.104 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:35:08.104 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:35:08.104 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:35:08.104 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:35:08.104 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:08.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.097+0000 7f7f14dc5640 1 Processor -- start 2024-04-26T00:35:08.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.097+0000 7f7f14dc5640 1 -- start start 2024-04-26T00:35:08.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.097+0000 7f7f14dc5640 10 monclient: init 2024-04-26T00:35:08.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f14dc5640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:08.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f14dc5640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:35:08.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f14dc5640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7f100a3d90 0x7f7f101ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:08.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f14dc5640 10 monclient(hunting): picked mon.noname-c con 0x7f7f100a3d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:08.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f14dc5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f100a4170 0x7f7f10136fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:08.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f14dc5640 10 monclient(hunting): picked mon.noname-b con 0x7f7f100a4170 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:35:08.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f14dc5640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f1013c3e0 0x7f7f1013a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:08.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f14dc5640 10 monclient(hunting): picked mon.noname-a con 0x7f7f1013c3e0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:35:08.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f14dc5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:08.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f14dc5640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f7f101418c0 con 0x7f7f1013c3e0 2024-04-26T00:35:08.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f14dc5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:08.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f14dc5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7f10141740 con 0x7f7f100a4170 2024-04-26T00:35:08.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f14dc5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:08.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f14dc5640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f7f10141a40 con 0x7f7f100a3d90 2024-04-26T00:35:08.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f14dc5640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:08.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f14dc5640 10 monclient(hunting): authenticate will time out at 1889.408293s 2024-04-26T00:35:08.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f0e59c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7f100a3d90 0x7f7f101ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:08.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f0ed9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f1013c3e0 0x7f7f1013a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:08.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f0e59c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7f100a3d90 0x7f7f101ceea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:57804/0 (socket says 172.21.15.37:57804) 2024-04-26T00:35:08.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f0e59c640 1 -- 172.21.15.37:0/386172562 learned_addr learned my addr 172.21.15.37:0/386172562 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:08.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f0e59c640 10 monclient(hunting): get_auth_request con 0x7f7f100a3d90 auth_method 0 2024-04-26T00:35:08.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:08.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f0e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:08.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f0e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:08.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f0ed9d640 10 monclient(hunting): get_auth_request con 0x7f7f1013c3e0 auth_method 0 2024-04-26T00:35:08.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f0ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:08.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f0ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:08.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f0ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:08.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f0e59c640 1 -- 172.21.15.37:0/386172562 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7f100a3d90 msgr2=0x7f7f101ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:35:08.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f0e59c640 1 -- 172.21.15.37:0/386172562 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7f100a3d90 msgr2=0x7f7f101ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:35:08.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f0e59c640 1 --2- 172.21.15.37:0/386172562 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7f100a3d90 0x7f7f101ceea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:35:08.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f0ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:08.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f0ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:08.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f0ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:08.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f0e59c640 1 --2- 172.21.15.37:0/386172562 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7f100a3d90 0x7f7f101ceea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:35:08.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f07fff640 1 --2- 172.21.15.37:0/386172562 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f100a4170 0x7f7f10136fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:08.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f07fff640 10 monclient(hunting): get_auth_request con 0x7f7f100a4170 auth_method 0 2024-04-26T00:35:08.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:08.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f07fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:08.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f07fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:08.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f0ed9d640 10 monclient(hunting): handle_auth_done global_id 144498 payload 824 2024-04-26T00:35:08.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f0ed9d640 1 -- 172.21.15.37:0/386172562 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7f100a3d90 msgr2=0x7f7f101ceea0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:35:08.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f0ed9d640 1 --2- 172.21.15.37:0/386172562 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7f100a3d90 0x7f7f101ceea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:08.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f0ed9d640 1 -- 172.21.15.37:0/386172562 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f100a4170 msgr2=0x7f7f10136fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:08.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f0ed9d640 1 --2- 172.21.15.37:0/386172562 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f100a4170 0x7f7f10136fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:08.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f0ed9d640 10 monclient: _finish_hunting 0 2024-04-26T00:35:08.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f0ed9d640 1 monclient: found mon.noname-a 2024-04-26T00:35:08.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f0ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:08.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f0ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:35:08.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f0ed9d640 1 -- 172.21.15.37:0/386172562 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f10137c60 con 0x7f7f1013c3e0 2024-04-26T00:35:08.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f0ed9d640 1 -- 172.21.15.37:0/386172562 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f1013c3e0 msgr2=0x7f7f1013a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:08.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f0ed9d640 1 --2- 172.21.15.37:0/386172562 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f1013c3e0 0x7f7f1013a120 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7f08009fd0 tx=0x7f7f08007ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:08.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f06ffd640 1 -- 172.21.15.37:0/386172562 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0801a070 con 0x7f7f1013c3e0 2024-04-26T00:35:08.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:08.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f06ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:35:08.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f06ffd640 10 monclient: dump: 2024-04-26T00:35:08.113 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:08.113 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:08.113 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:08.113 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:08.113 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:08.113 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:08.113 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:08.113 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:08.113 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:08.113 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:08.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f06ffd640 10 monclient: _finish_auth 0 2024-04-26T00:35:08.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f06ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:35:08.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:08.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f06ffd640 1 -- 172.21.15.37:0/386172562 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f0800ddf0 con 0x7f7f1013c3e0 2024-04-26T00:35:08.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f06ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:08.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f06ffd640 1 -- 172.21.15.37:0/386172562 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f08015640 con 0x7f7f1013c3e0 2024-04-26T00:35:08.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f14dc5640 5 monclient: authenticate success, global_id 144498 2024-04-26T00:35:08.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f14dc5640 10 monclient: _renew_subs 2024-04-26T00:35:08.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f14dc5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:08.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f14dc5640 1 -- 172.21.15.37:0/386172562 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f1009ada0 con 0x7f7f1013c3e0 2024-04-26T00:35:08.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:08.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f06ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:08.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f06ffd640 10 monclient: dump: 2024-04-26T00:35:08.114 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:08.114 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:08.115 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:08.115 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:08.115 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:08.115 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:08.115 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:08.115 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:08.115 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:08.115 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:08.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f0d59a640 4 set_mon_vals no callback set 2024-04-26T00:35:08.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f14dc5640 10 monclient: _renew_subs 2024-04-26T00:35:08.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f14dc5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:08.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.101+0000 7f7f14dc5640 1 -- 172.21.15.37:0/386172562 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f10121670 con 0x7f7f1013c3e0 2024-04-26T00:35:08.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.105+0000 7f7f14dc5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:08.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.105+0000 7f7f14dc5640 1 --2- 172.21.15.37:0/386172562 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f10137500 0x7f7f1013a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:08.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.105+0000 7f7f14dc5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:08.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.105+0000 7f7f14dc5640 1 -- 172.21.15.37:0/386172562 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f7f101202b0 con 0x7f7f10137500 2024-04-26T00:35:08.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.105+0000 7f7f0e59c640 1 --2- 172.21.15.37:0/386172562 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f10137500 0x7f7f1013a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:08.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.105+0000 7f7f0e59c640 10 monclient: get_auth_request con 0x7f7f10137500 auth_method 0 2024-04-26T00:35:08.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.105+0000 7f7f0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:08.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.105+0000 7f7f0e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:08.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.105+0000 7f7f0e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:08.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.105+0000 7f7f0e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:08.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.105+0000 7f7f0e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:08.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.105+0000 7f7f0e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:08.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.105+0000 7f7f0ed9d640 1 --2- 172.21.15.37:0/386172562 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f1013c3e0 0x7f7f1013a120 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7f08009fd0 tx=0x7f7f08007ef0 comp rx=0 tx=0).queue_received will delay after 1.71409e+09 on 0x7f7f08048960 osd_map(593..593 src has 370..593) 2024-04-26T00:35:08.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.105+0000 7f7f06ffd640 1 -- 172.21.15.37:0/386172562 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f7f0801d070 con 0x7f7f1013c3e0 2024-04-26T00:35:08.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.105+0000 7f7f0e59c640 10 monclient(hunting): handle_auth_done global_id 144501 payload 824 2024-04-26T00:35:08.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.105+0000 7f7f0e59c640 1 -- 172.21.15.37:0/386172562 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7f1004f510 con 0x7f7f10137500 2024-04-26T00:35:08.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.105+0000 7f7f06ffd640 1 --2- 172.21.15.37:0/386172562 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7ef0035e80 0x7f7ef0038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:08.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.105+0000 7f7f07fff640 1 --2- 172.21.15.37:0/386172562 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7ef0035e80 0x7f7ef0038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:08.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.105+0000 7f7f07fff640 10 monclient: get_auth_request con 0x7f7ef0035e80 auth_method 0 2024-04-26T00:35:08.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.105+0000 7f7f0e59c640 1 -- 172.21.15.37:0/386172562 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f10137500 msgr2=0x7f7f1013a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:08.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.105+0000 7f7f0e59c640 1 --2- 172.21.15.37:0/386172562 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f10137500 0x7f7f1013a660 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7f0000e4f0 tx=0x7f7f0000e9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:08.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.105+0000 7f7f06ffd640 1 -- 172.21.15.37:0/386172562 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f00007450 con 0x7f7f10137500 2024-04-26T00:35:08.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.105+0000 7f7f06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:08.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.105+0000 7f7f06ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:08.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.105+0000 7f7f06ffd640 10 monclient: dump: 2024-04-26T00:35:08.117 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:08.117 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:08.117 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:08.118 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:08.118 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:08.118 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:08.118 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:08.118 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:08.118 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:08.118 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:08.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.105+0000 7f7f06ffd640 1 -- 172.21.15.37:0/386172562 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7f00007450 con 0x7f7f10137500 2024-04-26T00:35:08.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.105+0000 7f7f06ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:08.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.105+0000 7f7f06ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:35:08.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.105+0000 7f7f06ffd640 1 -- 172.21.15.37:0/386172562 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f10137500 msgr2=0x7f7f1013a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:08.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.105+0000 7f7f06ffd640 1 --2- 172.21.15.37:0/386172562 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f10137500 0x7f7f1013a660 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7f0000e4f0 tx=0x7f7f0000e9b0 comp rx=0 tx=0).stop 2024-04-26T00:35:08.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.105+0000 7f7f06ffd640 1 -- 172.21.15.37:0/386172562 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f10137500 msgr2=0x7f7f1013a660 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:35:08.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.105+0000 7f7f06ffd640 1 --2- 172.21.15.37:0/386172562 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f10137500 0x7f7f1013a660 secure :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7f0000e4f0 tx=0x7f7f0000e9b0 comp rx=0 tx=0).stop 2024-04-26T00:35:08.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.109+0000 7f7f07fff640 1 --2- 172.21.15.37:0/386172562 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7ef0035e80 0x7f7ef0038340 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7f100a57e0 tx=0x7f7ef80a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:08.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.113+0000 7f7f14dc5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:35:08.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.113+0000 7f7f14dc5640 1 --2- 172.21.15.37:0/386172562 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f1013aba0 0x7f7f1009e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:08.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.113+0000 7f7f14dc5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:08.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.113+0000 7f7f14dc5640 1 -- 172.21.15.37:0/386172562 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f7f1013af80 con 0x7f7f1013aba0 2024-04-26T00:35:08.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.113+0000 7f7f0e59c640 1 --2- 172.21.15.37:0/386172562 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f1013aba0 0x7f7f1009e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:08.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.113+0000 7f7f0e59c640 10 monclient: get_auth_request con 0x7f7f1013aba0 auth_method 0 2024-04-26T00:35:08.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.113+0000 7f7f0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:08.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.113+0000 7f7f0e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:08.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.113+0000 7f7f0e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:08.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.113+0000 7f7f0e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:08.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.113+0000 7f7f0e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:08.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.113+0000 7f7f0e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:08.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.113+0000 7f7f0e59c640 10 monclient(hunting): handle_auth_done global_id 144504 payload 824 2024-04-26T00:35:08.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.113+0000 7f7f0e59c640 1 -- 172.21.15.37:0/386172562 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7f10075dc0 con 0x7f7f1013aba0 2024-04-26T00:35:08.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.117+0000 7f7f0e59c640 1 -- 172.21.15.37:0/386172562 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f1013aba0 msgr2=0x7f7f1009e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:08.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.117+0000 7f7f0e59c640 1 --2- 172.21.15.37:0/386172562 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f1013aba0 0x7f7f1009e080 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7f0000e4f0 tx=0x7f7f00003770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:08.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.117+0000 7f7f06ffd640 1 -- 172.21.15.37:0/386172562 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0001eb70 con 0x7f7f1013aba0 2024-04-26T00:35:08.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.117+0000 7f7f06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:08.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.117+0000 7f7f06ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:08.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.117+0000 7f7f06ffd640 10 monclient: dump: 2024-04-26T00:35:08.121 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:08.121 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:08.121 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:08.121 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:08.121 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:08.121 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:08.121 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:08.122 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:08.122 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:08.122 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:08.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.117+0000 7f7f06ffd640 1 -- 172.21.15.37:0/386172562 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4890 (secure 0 0 0) 0x7f7f00006e40 con 0x7f7f1013aba0 2024-04-26T00:35:08.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.117+0000 7f7f06ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:35:08.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.117+0000 7f7f06ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:35:08.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.117+0000 7f7f06ffd640 1 -- 172.21.15.37:0/386172562 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f1013aba0 msgr2=0x7f7f1009e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:08.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.117+0000 7f7f06ffd640 1 --2- 172.21.15.37:0/386172562 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f1013aba0 0x7f7f1009e080 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7f0000e4f0 tx=0x7f7f00003770 comp rx=0 tx=0).stop 2024-04-26T00:35:08.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.117+0000 7f7f06ffd640 1 -- 172.21.15.37:0/386172562 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f1013aba0 msgr2=0x7f7f1009e080 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:35:08.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.117+0000 7f7f06ffd640 1 --2- 172.21.15.37:0/386172562 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f1013aba0 0x7f7f1009e080 secure :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7f0000e4f0 tx=0x7f7f00003770 comp rx=0 tx=0).stop 2024-04-26T00:35:08.122 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:35:08.123 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:35:08.123 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:35:08.123 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:35:08.123 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 114, 2024-04-26T00:35:08.123 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 194661, 2024-04-26T00:35:08.123 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:35:08.123 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:35:08.123 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:35:08.123 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:35:08.123 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:08.123 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 1, 2024-04-26T00:35:08.123 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:35:08.123 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:35:08.123 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:35:08.124 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:08.124 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:08.124 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:08.124 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:08.124 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:08.124 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:08.124 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:08.124 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:08.124 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:08.124 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:08.124 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:08.124 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:08.124 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:35:08.124 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:35:08.125 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:08.125 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:08.125 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:08.125 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:08.125 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:08.125 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:08.125 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:08.125 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:08.125 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:08.125 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:08.125 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:08.125 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:08.125 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:08.125 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:35:08.125 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:35:08.125 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:35:08.126 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:35:08.126 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:35:08.126 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:35:08.126 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:35:08.126 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:35:08.126 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:35:08.126 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:35:08.126 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:35:08.126 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:35:08.126 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:35:08.126 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:35:08.127 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:35:08.127 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:35:08.127 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:35:08.127 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:08.127 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:08.127 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:08.127 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:08.127 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:08.127 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:08.127 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:08.127 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:08.127 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:08.127 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:08.127 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:08.128 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:08.128 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:35:08.128 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:08.128 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:35:08.128 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:08.128 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:35:08.128 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:35:08.128 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:08.128 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:08.128 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:08.128 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:08.128 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:35:08.128 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:08.128 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:08.128 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:08.128 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:08.129 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:35:08.129 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:08.129 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:08.129 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:08.129 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:08.129 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:35:08.129 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:35:08.129 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:08.129 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:08.129 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:08.129 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:08.129 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:08.129 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:35:08.129 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:35:08.130 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:08.130 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:08.130 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:08.130 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:08.130 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:35:08.130 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:08.130 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:08.130 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:08.130 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:08.130 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:35:08.130 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:08.130 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:08.130 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:08.130 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:08.131 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:35:08.131 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:35:08.131 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:08.131 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:08.131 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:08.131 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:08.131 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:08.131 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:35:08.131 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:35:08.131 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:08.131 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:08.131 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:08.131 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:08.132 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:35:08.132 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:08.132 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:08.132 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:08.132 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:08.132 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:35:08.132 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:08.132 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:08.132 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:08.132 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:08.132 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:35:08.132 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:35:08.132 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:08.132 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:08.133 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:08.133 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:08.133 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:08.133 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:08.133 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:35:08.133 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:35:08.133 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:08.133 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:08.133 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:08.133 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:35:08.133 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:08.133 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:08.133 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:35:08.133 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:08.133 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:08.134 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:08.134 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:35:08.134 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:08.134 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:08.134 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:35:08.134 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:08.134 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:08.134 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:08.134 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 7 2024-04-26T00:35:08.134 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:08.134 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:08.134 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:35:08.134 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:08.134 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:08.134 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:08.135 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:35:08.135 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:08.135 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:08.135 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:08.135 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:35:08.135 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:35:08.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.117+0000 7f7f14dc5640 1 -- 172.21.15.37:0/386172562 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7ef0035e80 msgr2=0x7f7ef0038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:08.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.117+0000 7f7f14dc5640 1 --2- 172.21.15.37:0/386172562 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7ef0035e80 0x7f7ef0038340 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7f100a57e0 tx=0x7f7ef80a6000 comp rx=0 tx=0).stop 2024-04-26T00:35:08.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.117+0000 7f7f14dc5640 10 monclient: shutdown 2024-04-26T00:35:08.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.117+0000 7f7f14dc5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:08.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.117+0000 7f7f14dc5640 1 -- 172.21.15.37:0/386172562 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f1013c3e0 msgr2=0x7f7f1013a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:08.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.117+0000 7f7f14dc5640 1 --2- 172.21.15.37:0/386172562 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f1013c3e0 0x7f7f1013a120 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7f08009fd0 tx=0x7f7f08007ef0 comp rx=0 tx=0).stop 2024-04-26T00:35:08.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.117+0000 7f7f0ed9d640 1 -- 172.21.15.37:0/386172562 reap_dead start 2024-04-26T00:35:08.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.117+0000 7f7f0ed9d640 1 -- 172.21.15.37:0/386172562 reap_dead start 2024-04-26T00:35:08.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.117+0000 7f7f06ffd640 1 -- 172.21.15.37:0/386172562 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(593..593 src has 370..593) ==== 3915+0+0 (unknown 0 0 0) 0x7f7f08048960 con 0x7f7f1013c3e0 2024-04-26T00:35:08.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.117+0000 7f7f14dc5640 1 -- 172.21.15.37:0/386172562 shutdown_connections 2024-04-26T00:35:08.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.117+0000 7f7f14dc5640 1 -- 172.21.15.37:0/386172562 >> 172.21.15.37:0/386172562 conn(0x7f7f100aa0a0 msgr2=0x7f7f100a8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:08.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.117+0000 7f7f14dc5640 1 -- 172.21.15.37:0/386172562 shutdown_connections 2024-04-26T00:35:08.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.117+0000 7f7f14dc5640 1 -- 172.21.15.37:0/386172562 wait complete. 2024-04-26T00:35:08.138 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:35:08.138 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:35:08.138 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:35:08.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.253+0000 7f8eb129f640 10 monclient: get_monmap_and_config 2024-04-26T00:35:08.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.253+0000 7f8eb129f640 10 monclient: build_initial_monmap 2024-04-26T00:35:08.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.253+0000 7f8eb129f640 10 monclient: monmap: 2024-04-26T00:35:08.254 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:08.254 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:08.254 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:35:08.259843+0000 2024-04-26T00:35:08.254 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:35:08.259843+0000 2024-04-26T00:35:08.254 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:08.254 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:08.254 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:35:08.254 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:35:08.254 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:35:08.254 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:35:08.255 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:35:08.255 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:08.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.257+0000 7f8eb129f640 1 Processor -- start 2024-04-26T00:35:08.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.257+0000 7f8eb129f640 1 -- start start 2024-04-26T00:35:08.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.257+0000 7f8eb129f640 10 monclient: init 2024-04-26T00:35:08.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.257+0000 7f8eb129f640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:08.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.257+0000 7f8eb129f640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-04-26T00:35:08.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.257+0000 7f8eb129f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8eac130eb0 addr v1:172.21.15.37:6789/0 2024-04-26T00:35:08.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.257+0000 7f8eb129f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8eac09ef20 addr v1:172.21.15.45:6789/0 2024-04-26T00:35:08.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.257+0000 7f8eb129f640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f8eac13a5d0 0x7f8eac13c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:08.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.257+0000 7f8eb129f640 10 monclient(hunting): picked mon.noname-a con 0x7f8eac13a5d0 addr v2:172.21.15.37:3300/0 2024-04-26T00:35:08.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.257+0000 7f8eb129f640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8eac09c190 con 0x7f8eac130eb0 2024-04-26T00:35:08.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.257+0000 7f8eb129f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8eac0a11c0 con 0x7f8eac09ef20 2024-04-26T00:35:08.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.257+0000 7f8eb129f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:08.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.257+0000 7f8eb129f640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f8eac09df60 con 0x7f8eac13a5d0 2024-04-26T00:35:08.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.257+0000 7f8eb129f640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:08.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.257+0000 7f8eb129f640 10 monclient(hunting): authenticate will time out at 1889.565498s 2024-04-26T00:35:08.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.257+0000 7f8eaaffd640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f8eac130eb0 0x7f8eac131290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:42284/0 (socket says 172.21.15.37:42284) 2024-04-26T00:35:08.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.257+0000 7f8eaaffd640 1 -- 172.21.15.37:0/3096710097 learned_addr learned my addr 172.21.15.37:0/3096710097 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:35:08.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.257+0000 7f8eab7fe640 1 --2- 172.21.15.37:0/3096710097 >> v2:172.21.15.37:3300/0 conn(0x7f8eac13a5d0 0x7f8eac13c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:08.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.257+0000 7f8eab7fe640 10 monclient(hunting): get_auth_request con 0x7f8eac13a5d0 auth_method 0 2024-04-26T00:35:08.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.257+0000 7f8eab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:08.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.257+0000 7f8eab7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:08.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.257+0000 7f8eab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:08.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.257+0000 7f8eab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:08.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.257+0000 7f8eab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:08.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.257+0000 7f8eab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:08.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.257+0000 7f8eab7fe640 10 monclient(hunting): handle_auth_done global_id 144507 payload 274 2024-04-26T00:35:08.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.257+0000 7f8eab7fe640 1 -- 172.21.15.37:0/3096710097 >> v1:172.21.15.45:6789/0 conn(0x7f8eac09ef20 legacy=0x7f8eac138eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:08.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.257+0000 7f8eab7fe640 1 -- 172.21.15.37:0/3096710097 >> v1:172.21.15.37:6789/0 conn(0x7f8eac130eb0 legacy=0x7f8eac131290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:08.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.257+0000 7f8eab7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:35:08.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.257+0000 7f8eab7fe640 1 monclient: found mon.noname-a 2024-04-26T00:35:08.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.257+0000 7f8eab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:08.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.257+0000 7f8eab7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:35:08.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.257+0000 7f8eab7fe640 1 -- 172.21.15.37:0/3096710097 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8eac09dc80 con 0x7f8eac13a5d0 2024-04-26T00:35:08.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.261+0000 7f8eab7fe640 1 -- 172.21.15.37:0/3096710097 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8eac13a5d0 msgr2=0x7f8eac13c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:08.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.261+0000 7f8eab7fe640 1 --2- 172.21.15.37:0/3096710097 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8eac13a5d0 0x7f8eac13c9c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4009d90 tx=0x7f8ea401c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe0ea79c70055f79 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:08.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.261+0000 7f8ea9ffb640 1 -- 172.21.15.37:0/3096710097 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ea40b2070 con 0x7f8eac13a5d0 2024-04-26T00:35:08.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.261+0000 7f8ea9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:08.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.261+0000 7f8ea9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:35:08.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.261+0000 7f8ea9ffb640 10 monclient: dump: 2024-04-26T00:35:08.262 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:08.262 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:08.262 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:08.262 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:08.262 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:08.262 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:08.262 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:08.262 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:08.262 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:08.262 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:08.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.261+0000 7f8ea9ffb640 10 monclient: _finish_auth 0 2024-04-26T00:35:08.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.261+0000 7f8ea9ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:35:08.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.261+0000 7f8ea9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:08.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.261+0000 7f8ea9ffb640 1 -- 172.21.15.37:0/3096710097 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ea40a5940 con 0x7f8eac13a5d0 2024-04-26T00:35:08.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.261+0000 7f8ea9ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:08.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.261+0000 7f8ea9ffb640 1 -- 172.21.15.37:0/3096710097 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ea40a5d10 con 0x7f8eac13a5d0 2024-04-26T00:35:08.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.261+0000 7f8ea9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:08.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.261+0000 7f8ea9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:08.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.261+0000 7f8ea9ffb640 10 monclient: dump: 2024-04-26T00:35:08.263 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:08.263 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:08.264 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:08.264 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:08.264 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:08.264 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:08.264 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:08.264 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:08.264 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:08.264 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:08.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.261+0000 7f8eb129f640 5 monclient: authenticate success, global_id 144507 2024-04-26T00:35:08.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.261+0000 7f8eb129f640 10 monclient: get_monmap_and_config success 2024-04-26T00:35:08.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.261+0000 7f8eb129f640 4 set_mon_vals no callback set 2024-04-26T00:35:08.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.261+0000 7f8eb129f640 10 monclient: shutdown 2024-04-26T00:35:08.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.261+0000 7f8eb129f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:08.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.261+0000 7f8eb129f640 1 -- 172.21.15.37:0/3096710097 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8eac13a5d0 msgr2=0x7f8eac13c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:08.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.261+0000 7f8eb129f640 1 --2- 172.21.15.37:0/3096710097 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8eac13a5d0 0x7f8eac13c9c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4009d90 tx=0x7f8ea401c780 comp rx=0 tx=0).stop 2024-04-26T00:35:08.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.261+0000 7f8eb129f640 1 -- 172.21.15.37:0/3096710097 shutdown_connections 2024-04-26T00:35:08.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.261+0000 7f8eb129f640 1 --2- 172.21.15.37:0/3096710097 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8eac13a5d0 0x7f8eac13c9c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:08.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.261+0000 7f8eb129f640 1 -- 172.21.15.37:0/3096710097 >> 172.21.15.37:0/3096710097 conn(0x7f8eac0a1d50 msgr2=0x7f8eac0a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:08.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.261+0000 7f8eb129f640 1 -- 172.21.15.37:0/3096710097 shutdown_connections 2024-04-26T00:35:08.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.261+0000 7f8eb129f640 1 -- 172.21.15.37:0/3096710097 wait complete. 2024-04-26T00:35:08.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eb129f640 10 monclient: build_initial_monmap 2024-04-26T00:35:08.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eb129f640 10 monclient: monmap: 2024-04-26T00:35:08.273 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:08.274 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:35:08.274 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:35:08.274 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:35:08.274 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:08.274 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:35:08.274 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:35:08.274 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:35:08.275 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:35:08.275 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:08.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eb129f640 1 Processor -- start 2024-04-26T00:35:08.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eb129f640 1 -- start start 2024-04-26T00:35:08.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eb129f640 10 monclient: init 2024-04-26T00:35:08.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eb129f640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:08.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eb129f640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:35:08.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eb129f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac09ef20 0x7f8eac138bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:08.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eb129f640 10 monclient(hunting): picked mon.noname-c con 0x7f8eac09ef20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:08.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eb129f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8eac130eb0 0x7f8eac12ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:08.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eb129f640 10 monclient(hunting): picked mon.noname-a con 0x7f8eac130eb0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:35:08.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eb129f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8eac13a5d0 0x7f8eac12f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:08.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eb129f640 10 monclient(hunting): picked mon.noname-b con 0x7f8eac13a5d0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:35:08.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eb129f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:08.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eb129f640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f8eac13f8a0 con 0x7f8eac130eb0 2024-04-26T00:35:08.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eb129f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:08.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eb129f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8eac13f720 con 0x7f8eac13a5d0 2024-04-26T00:35:08.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eb129f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:08.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eb129f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8eac13fa20 con 0x7f8eac09ef20 2024-04-26T00:35:08.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eb129f640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:08.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eb129f640 10 monclient(hunting): authenticate will time out at 1889.573705s 2024-04-26T00:35:08.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eaaffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac09ef20 0x7f8eac138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:08.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eaa7fc640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8eac130eb0 0x7f8eac12ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:08.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eaa7fc640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8eac130eb0 0x7f8eac12ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52890/0 (socket says 172.21.15.37:52890) 2024-04-26T00:35:08.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eaa7fc640 1 -- 172.21.15.37:0/1215210130 learned_addr learned my addr 172.21.15.37:0/1215210130 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:08.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eaa7fc640 10 monclient(hunting): get_auth_request con 0x7f8eac130eb0 auth_method 0 2024-04-26T00:35:08.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eaa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:08.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eaa7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:08.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:08.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eaaffd640 1 --2- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac09ef20 0x7f8eac138bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:57812/0 (socket says 172.21.15.37:57812) 2024-04-26T00:35:08.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eaaffd640 10 monclient(hunting): get_auth_request con 0x7f8eac09ef20 auth_method 0 2024-04-26T00:35:08.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eaaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:08.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eaaffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:08.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:08.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eaa7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:08.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eaa7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:08.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eaa7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:08.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eaaffd640 1 -- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac09ef20 msgr2=0x7f8eac138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:35:08.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eaaffd640 1 -- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac09ef20 msgr2=0x7f8eac138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:35:08.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eaaffd640 1 --2- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac09ef20 0x7f8eac138bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:35:08.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eab7fe640 1 --2- 172.21.15.37:0/1215210130 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8eac13a5d0 0x7f8eac12f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:08.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eaaffd640 1 --2- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac09ef20 0x7f8eac138bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:35:08.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eab7fe640 10 monclient(hunting): get_auth_request con 0x7f8eac13a5d0 auth_method 0 2024-04-26T00:35:08.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:08.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eab7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:08.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:08.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eaa7fc640 10 monclient(hunting): handle_auth_done global_id 144510 payload 824 2024-04-26T00:35:08.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eaa7fc640 1 -- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac09ef20 msgr2=0x7f8eac138bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:35:08.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eaa7fc640 1 --2- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac09ef20 0x7f8eac138bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:08.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eaa7fc640 1 -- 172.21.15.37:0/1215210130 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8eac13a5d0 msgr2=0x7f8eac12f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:08.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eaa7fc640 1 --2- 172.21.15.37:0/1215210130 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8eac13a5d0 0x7f8eac12f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:08.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eaa7fc640 10 monclient: _finish_hunting 0 2024-04-26T00:35:08.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eaa7fc640 1 monclient: found mon.noname-a 2024-04-26T00:35:08.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eaa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:08.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eaa7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:35:08.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eaa7fc640 1 -- 172.21.15.37:0/1215210130 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8eac1395d0 con 0x7f8eac130eb0 2024-04-26T00:35:08.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.265+0000 7f8eab7fe640 1 --2- 172.21.15.37:0/1215210130 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8eac13a5d0 0x7f8eac12f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:35:08.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8eaa7fc640 1 -- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8eac130eb0 msgr2=0x7f8eac12ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:08.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8eaa7fc640 1 --2- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8eac130eb0 0x7f8eac12ec70 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8ea000c480 tx=0x7f8ea000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:08.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8e937fe640 1 -- 172.21.15.37:0/1215210130 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ea0012a40 con 0x7f8eac130eb0 2024-04-26T00:35:08.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8e937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:08.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8e937fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:35:08.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8e937fe640 10 monclient: dump: 2024-04-26T00:35:08.282 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:08.282 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:08.282 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:08.282 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:08.282 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:08.282 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:08.283 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:08.283 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:08.283 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:08.283 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:08.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8e937fe640 10 monclient: _finish_auth 0 2024-04-26T00:35:08.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8e937fe640 10 monclient: _check_auth_tickets 2024-04-26T00:35:08.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8e937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:08.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8e937fe640 1 -- 172.21.15.37:0/1215210130 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ea0017070 con 0x7f8eac130eb0 2024-04-26T00:35:08.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8e937fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:08.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8e937fe640 1 -- 172.21.15.37:0/1215210130 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ea0012e40 con 0x7f8eac130eb0 2024-04-26T00:35:08.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8e937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:08.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8e937fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:08.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8e937fe640 10 monclient: dump: 2024-04-26T00:35:08.283 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:08.283 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:08.284 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:08.284 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:08.284 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:08.284 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:08.284 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:08.284 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:08.284 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:08.284 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:08.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8ea8ff9640 4 set_mon_vals no callback set 2024-04-26T00:35:08.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8eb129f640 5 monclient: authenticate success, global_id 144510 2024-04-26T00:35:08.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8eb129f640 10 monclient: _renew_subs 2024-04-26T00:35:08.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8eb129f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:08.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8eb129f640 1 -- 172.21.15.37:0/1215210130 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8eac133cb0 con 0x7f8eac130eb0 2024-04-26T00:35:08.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8eb129f640 10 monclient: _renew_subs 2024-04-26T00:35:08.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8eb129f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:08.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8eb129f640 1 -- 172.21.15.37:0/1215210130 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8eac1ceeb0 con 0x7f8eac130eb0 2024-04-26T00:35:08.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8eb129f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:08.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8eb129f640 1 --2- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac09fc50 0x7f8eac0a0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:08.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8eb129f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:08.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8eb129f640 1 -- 172.21.15.37:0/1215210130 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8eac11b9b0 con 0x7f8eac09fc50 2024-04-26T00:35:08.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8eaaffd640 1 --2- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac09fc50 0x7f8eac0a0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:08.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8eaaffd640 10 monclient: get_auth_request con 0x7f8eac09fc50 auth_method 0 2024-04-26T00:35:08.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8eaaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:08.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8eaaffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:08.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8eaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:08.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8eaaffd640 1 -- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac09fc50 msgr2=0x7f8eac0a0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:35:08.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8eaaffd640 1 -- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac09fc50 msgr2=0x7f8eac0a0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:35:08.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8eaaffd640 1 --2- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac09fc50 0x7f8eac0a0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:35:08.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8eaaffd640 1 --2- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac09fc50 0x7f8eac0a0030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:35:08.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8e937fe640 1 -- 172.21.15.37:0/1215210130 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f8ea001c460 con 0x7f8eac130eb0 2024-04-26T00:35:08.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8e937fe640 1 --2- 172.21.15.37:0/1215210130 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8e98035ed0 0x7f8e98038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:08.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8e937fe640 1 -- 172.21.15.37:0/1215210130 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(593..593 src has 370..593) ==== 3915+0+0 (secure 0 0 0) 0x7f8ea0048940 con 0x7f8eac130eb0 2024-04-26T00:35:08.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8eab7fe640 1 --2- 172.21.15.37:0/1215210130 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8e98035ed0 0x7f8e98038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:08.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.269+0000 7f8eab7fe640 10 monclient: get_auth_request con 0x7f8e98035ed0 auth_method 0 2024-04-26T00:35:08.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.273+0000 7f8eab7fe640 1 --2- 172.21.15.37:0/1215210130 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8e98035ed0 0x7f8e98038390 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f8eac0764d0 tx=0x7f8ea40a7070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:08.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.473+0000 7f8eaaffd640 1 --2- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac09fc50 0x7f8eac0a0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:08.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.473+0000 7f8eaaffd640 10 monclient: get_auth_request con 0x7f8eac09fc50 auth_method 0 2024-04-26T00:35:08.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.473+0000 7f8eaaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:08.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.473+0000 7f8eaaffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:08.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.473+0000 7f8eaaffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:35:08.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.473+0000 7f8eaaffd640 1 -- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac09fc50 msgr2=0x7f8eac0a0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:35:08.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.473+0000 7f8eaaffd640 1 -- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac09fc50 msgr2=0x7f8eac0a0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:35:08.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.473+0000 7f8eaaffd640 1 --2- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac09fc50 0x7f8eac0a0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:35:08.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.473+0000 7f8eaaffd640 1 --2- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac09fc50 0x7f8eac0a0030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:35:08.529 INFO:tasks.rados.rados.0.smithi045.stdout:2556: done (1 left) 2024-04-26T00:35:08.529 INFO:tasks.rados.rados.0.smithi045.stdout:2557: done (0 left) 2024-04-26T00:35:08.530 INFO:tasks.rados.rados.0.smithi045.stdout:2558: write (excl) oid 7 current snap is 286 2024-04-26T00:35:08.530 INFO:tasks.rados.rados.0.smithi045.stdout:2558: seq_num 922 ranges {696054=658291,2103339=753642} 2024-04-26T00:35:08.551 INFO:tasks.rados.rados.0.smithi045.stdout:2558: writing smithi04526579-7 from 696054 to 1354345 tid 1 2024-04-26T00:35:08.576 INFO:tasks.rados.rados.0.smithi045.stdout:2558: writing smithi04526579-7 from 2103339 to 2856981 tid 2 2024-04-26T00:35:08.576 INFO:tasks.rados.rados.0.smithi045.stdout:2559: copy_from oid 30 from oid 5 current snap is 286 2024-04-26T00:35:08.576 INFO:tasks.rados.rados.0.smithi045.stdout:2558: finishing write tid 1 to smithi04526579-7 2024-04-26T00:35:08.576 INFO:tasks.rados.rados.0.smithi045.stdout:2560: delete oid 50 current snap is 286 2024-04-26T00:35:08.590 INFO:tasks.rados.rados.0.smithi045.stdout:2558: finishing write tid 2 to smithi04526579-7 2024-04-26T00:35:08.590 INFO:tasks.rados.rados.0.smithi045.stdout:2558: finishing write tid 3 to smithi04526579-7 2024-04-26T00:35:08.590 INFO:tasks.rados.rados.0.smithi045.stdout:2558: finishing write tid 4 to smithi04526579-7 2024-04-26T00:35:08.590 INFO:tasks.rados.rados.0.smithi045.stdout:2558: oid 7 updating version 0 to 2240 2024-04-26T00:35:08.590 INFO:tasks.rados.rados.0.smithi045.stdout:2558: oid 7 updating version 2240 to 2241 2024-04-26T00:35:08.590 INFO:tasks.rados.rados.0.smithi045.stdout:2558: oid 7 version 2241 is already newer than 2239 2024-04-26T00:35:08.590 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 2241 (ObjNum 922 snap 286 seq_num 922) dirty exists 2024-04-26T00:35:08.591 INFO:tasks.rados.rados.0.smithi045.stdout:2558: left oid 7 (ObjNum 922 snap 286 seq_num 922) 2024-04-26T00:35:08.595 INFO:tasks.rados.rados.0.smithi045.stdout:2558: done (2 left) 2024-04-26T00:35:08.595 INFO:tasks.rados.rados.0.smithi045.stdout:2560: done (1 left) 2024-04-26T00:35:08.595 INFO:tasks.rados.rados.0.smithi045.stdout:2561: read oid 7 snap -1 2024-04-26T00:35:08.595 INFO:tasks.rados.rados.0.smithi045.stdout:2561: expect (ObjNum 922 snap 286 seq_num 922) 2024-04-26T00:35:08.595 INFO:tasks.rados.rados.0.smithi045.stdout:2562: snap_create 2024-04-26T00:35:08.598 INFO:tasks.rados.rados.0.smithi045.stdout:2559: finishing copy_from racing read to smithi04526579-30 2024-04-26T00:35:08.599 INFO:tasks.rados.rados.0.smithi045.stdout:2559: finishing copy_from to smithi04526579-30 2024-04-26T00:35:08.600 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 1977 (ObjNum 836 snap 260 seq_num 836) dirty exists 2024-04-26T00:35:08.611 INFO:tasks.rados.rados.0.smithi045.stdout:2561: expect (ObjNum 922 snap 286 seq_num 922) 2024-04-26T00:35:08.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.877+0000 7f8eaaffd640 1 --2- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac09fc50 0x7f8eac0a0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:08.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.877+0000 7f8eaaffd640 10 monclient: get_auth_request con 0x7f8eac09fc50 auth_method 0 2024-04-26T00:35:08.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.877+0000 7f8eaaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:08.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.877+0000 7f8eaaffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:08.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.877+0000 7f8eaaffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:35:08.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.877+0000 7f8eaaffd640 1 -- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac09fc50 msgr2=0x7f8eac0a0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:35:08.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.877+0000 7f8eaaffd640 1 -- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac09fc50 msgr2=0x7f8eac0a0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:35:08.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.877+0000 7f8eaaffd640 1 --2- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac09fc50 0x7f8eac0a0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:35:08.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:08.877+0000 7f8eaaffd640 1 --2- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac09fc50 0x7f8eac0a0030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:35:09.546 INFO:tasks.rados.rados.0.smithi045.stdout:2559: done (2 left) 2024-04-26T00:35:09.546 INFO:tasks.rados.rados.0.smithi045.stdout:2561: done (1 left) 2024-04-26T00:35:09.546 INFO:tasks.rados.rados.0.smithi045.stdout:2562: done (0 left) 2024-04-26T00:35:09.546 INFO:tasks.rados.rados.0.smithi045.stdout:2563: copy_from oid 26 from oid 40 current snap is 287 2024-04-26T00:35:09.546 INFO:tasks.rados.rados.0.smithi045.stdout:2564: write (excl) oid 8 current snap is 287 2024-04-26T00:35:09.546 INFO:tasks.rados.rados.0.smithi045.stdout:2564: seq_num 924 ranges {729668=408789,1820303=757311,3274610=255058} 2024-04-26T00:35:09.564 INFO:tasks.rados.rados.0.smithi045.stdout:2564: writing smithi04526579-8 from 729668 to 1138457 tid 1 2024-04-26T00:35:09.588 INFO:tasks.rados.rados.0.smithi045.stdout:2564: writing smithi04526579-8 from 1820303 to 2577614 tid 2 2024-04-26T00:35:09.602 INFO:tasks.rados.rados.0.smithi045.stdout:2564: writing smithi04526579-8 from 3274610 to 3529668 tid 3 2024-04-26T00:35:09.603 INFO:tasks.rados.rados.0.smithi045.stdout:2565: read oid 48 snap -1 2024-04-26T00:35:09.603 INFO:tasks.rados.rados.0.smithi045.stdout:2565: expect (ObjNum 836 snap 260 seq_num 836) 2024-04-26T00:35:09.603 INFO:tasks.rados.rados.0.smithi045.stdout:2566: copy_from oid 44 from oid 25 current snap is 287 2024-04-26T00:35:09.603 INFO:tasks.rados.rados.0.smithi045.stdout:2563: finishing copy_from racing read to smithi04526579-26 2024-04-26T00:35:09.603 INFO:tasks.rados.rados.0.smithi045.stdout:2563: finishing copy_from to smithi04526579-26 2024-04-26T00:35:09.603 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 1474 (ObjNum 886 snap 274 seq_num 886) dirty exists 2024-04-26T00:35:09.603 INFO:tasks.rados.rados.0.smithi045.stdout:2564: finishing write tid 1 to smithi04526579-8 2024-04-26T00:35:09.603 INFO:tasks.rados.rados.0.smithi045.stdout:2564: finishing write tid 2 to smithi04526579-8 2024-04-26T00:35:09.603 INFO:tasks.rados.rados.0.smithi045.stdout:2563: done (3 left) 2024-04-26T00:35:09.603 INFO:tasks.rados.rados.0.smithi045.stdout:2567: write (excl) oid 10 current snap is 287 2024-04-26T00:35:09.603 INFO:tasks.rados.rados.0.smithi045.stdout:2567: seq_num 925 ranges {746475=484038,1762432=617322,2785680=529906} 2024-04-26T00:35:09.617 INFO:tasks.rados.rados.0.smithi045.stdout:2567: writing smithi04526579-10 from 746475 to 1230513 tid 1 2024-04-26T00:35:09.628 INFO:tasks.rados.rados.0.smithi045.stdout:2567: writing smithi04526579-10 from 1762432 to 2379754 tid 2 2024-04-26T00:35:09.636 INFO:tasks.rados.rados.0.smithi045.stdout:2567: writing smithi04526579-10 from 2785680 to 3315586 tid 3 2024-04-26T00:35:09.636 INFO:tasks.rados.rados.0.smithi045.stdout:2568: rollback oid 38 current snap is 287 2024-04-26T00:35:09.636 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 38 to 281 2024-04-26T00:35:09.636 INFO:tasks.rados.rados.0.smithi045.stdout:2564: finishing write tid 3 to smithi04526579-8 2024-04-26T00:35:09.636 INFO:tasks.rados.rados.0.smithi045.stdout:2569: snap_remove snap 284 2024-04-26T00:35:09.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:09.681+0000 7f8eaaffd640 1 --2- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac09fc50 0x7f8eac0a0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:09.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:09.681+0000 7f8eaaffd640 10 monclient: get_auth_request con 0x7f8eac09fc50 auth_method 0 2024-04-26T00:35:09.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:09.681+0000 7f8eaaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:09.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:09.681+0000 7f8eaaffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:09.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:09.681+0000 7f8eaaffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:35:09.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:09.681+0000 7f8eaaffd640 1 -- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac09fc50 msgr2=0x7f8eac0a0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:35:09.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:09.681+0000 7f8eaaffd640 1 -- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac09fc50 msgr2=0x7f8eac0a0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:35:09.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:09.681+0000 7f8eaaffd640 1 --2- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac09fc50 0x7f8eac0a0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:35:09.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:09.681+0000 7f8eaaffd640 1 --2- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac09fc50 0x7f8eac0a0030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:35:10.625 INFO:tasks.rados.rados.0.smithi045.stdout:2569: done (5 left) 2024-04-26T00:35:10.625 INFO:tasks.rados.rados.0.smithi045.stdout:2570: read oid 26 snap -1 2024-04-26T00:35:10.625 INFO:tasks.rados.rados.0.smithi045.stdout:2570: expect (ObjNum 886 snap 274 seq_num 886) 2024-04-26T00:35:10.626 INFO:tasks.rados.rados.0.smithi045.stdout:2571: read oid 14 snap -1 2024-04-26T00:35:10.626 INFO:tasks.rados.rados.0.smithi045.stdout:2571: expect (ObjNum 866 snap 269 seq_num 866) 2024-04-26T00:35:10.626 INFO:tasks.rados.rados.0.smithi045.stdout:2564: finishing write tid 4 to smithi04526579-8 2024-04-26T00:35:10.626 INFO:tasks.rados.rados.0.smithi045.stdout:2564: finishing write tid 5 to smithi04526579-8 2024-04-26T00:35:10.626 INFO:tasks.rados.rados.0.smithi045.stdout:2564: oid 8 updating version 0 to 1988 2024-04-26T00:35:10.626 INFO:tasks.rados.rados.0.smithi045.stdout:2564: oid 8 updating version 1988 to 1989 2024-04-26T00:35:10.626 INFO:tasks.rados.rados.0.smithi045.stdout:2564: oid 8 version 1989 is already newer than 1987 2024-04-26T00:35:10.626 INFO:tasks.rados.rados.0.smithi045.stdout:2564: oid 8 version 1989 is already newer than 1986 2024-04-26T00:35:10.626 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 1989 (ObjNum 924 snap 287 seq_num 924) dirty exists 2024-04-26T00:35:10.626 INFO:tasks.rados.rados.0.smithi045.stdout:2564: left oid 8 (ObjNum 924 snap 287 seq_num 924) 2024-04-26T00:35:10.626 INFO:tasks.rados.rados.0.smithi045.stdout:2565: expect (ObjNum 836 snap 260 seq_num 836) 2024-04-26T00:35:10.749 INFO:tasks.rados.rados.0.smithi045.stdout:2566: finishing copy_from racing read to smithi04526579-44 2024-04-26T00:35:10.750 INFO:tasks.rados.rados.0.smithi045.stdout:2567: finishing write tid 1 to smithi04526579-10 2024-04-26T00:35:10.750 INFO:tasks.rados.rados.0.smithi045.stdout:2567: finishing write tid 2 to smithi04526579-10 2024-04-26T00:35:10.750 INFO:tasks.rados.rados.0.smithi045.stdout:2566: finishing copy_from to smithi04526579-44 2024-04-26T00:35:10.750 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 2154 (ObjNum 898 snap 276 seq_num 898) dirty exists 2024-04-26T00:35:10.750 INFO:tasks.rados.rados.0.smithi045.stdout:2568: finishing rollback tid 1 to smithi04526579-38 2024-04-26T00:35:10.750 INFO:tasks.rados.rados.0.smithi045.stdout:2568: finishing rollback tid 2 to smithi04526579-38 2024-04-26T00:35:10.750 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 2156 (ObjNum 894 snap 275 seq_num 894) dirty exists 2024-04-26T00:35:10.750 INFO:tasks.rados.rados.0.smithi045.stdout:2567: finishing write tid 3 to smithi04526579-10 2024-04-26T00:35:10.750 INFO:tasks.rados.rados.0.smithi045.stdout:2567: finishing write tid 4 to smithi04526579-10 2024-04-26T00:35:10.750 INFO:tasks.rados.rados.0.smithi045.stdout:2567: finishing write tid 5 to smithi04526579-10 2024-04-26T00:35:10.750 INFO:tasks.rados.rados.0.smithi045.stdout:2567: oid 10 updating version 0 to 1766 2024-04-26T00:35:10.751 INFO:tasks.rados.rados.0.smithi045.stdout:2567: oid 10 version 1766 is already newer than 1765 2024-04-26T00:35:10.751 INFO:tasks.rados.rados.0.smithi045.stdout:2567: oid 10 version 1766 is already newer than 1764 2024-04-26T00:35:10.751 INFO:tasks.rados.rados.0.smithi045.stdout:2567: oid 10 version 1766 is already newer than 1763 2024-04-26T00:35:10.751 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 1766 (ObjNum 925 snap 287 seq_num 925) dirty exists 2024-04-26T00:35:10.751 INFO:tasks.rados.rados.0.smithi045.stdout:2567: left oid 10 (ObjNum 925 snap 287 seq_num 925) 2024-04-26T00:35:10.751 INFO:tasks.rados.rados.0.smithi045.stdout:2570: expect (ObjNum 886 snap 274 seq_num 886) 2024-04-26T00:35:10.881 INFO:tasks.rados.rados.0.smithi045.stdout:2564: done (6 left) 2024-04-26T00:35:10.881 INFO:tasks.rados.rados.0.smithi045.stdout:2565: done (5 left) 2024-04-26T00:35:10.881 INFO:tasks.rados.rados.0.smithi045.stdout:2566: done (4 left) 2024-04-26T00:35:10.882 INFO:tasks.rados.rados.0.smithi045.stdout:2567: done (3 left) 2024-04-26T00:35:10.882 INFO:tasks.rados.rados.0.smithi045.stdout:2568: done (2 left) 2024-04-26T00:35:10.882 INFO:tasks.rados.rados.0.smithi045.stdout:2570: done (1 left) 2024-04-26T00:35:10.882 INFO:tasks.rados.rados.0.smithi045.stdout:2572: snap_remove snap 283 2024-04-26T00:35:11.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.265+0000 7f8e927fc640 10 monclient: tick 2024-04-26T00:35:11.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.265+0000 7f8e927fc640 10 monclient: _check_auth_tickets 2024-04-26T00:35:11.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.265+0000 7f8e927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:11.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.265+0000 7f8e927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:11.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.285+0000 7f8eaaffd640 1 --2- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac09fc50 0x7f8eac0a0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:11.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.285+0000 7f8eaaffd640 10 monclient: get_auth_request con 0x7f8eac09fc50 auth_method 0 2024-04-26T00:35:11.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.285+0000 7f8eaaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:11.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.285+0000 7f8eaaffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:11.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.285+0000 7f8eaaffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:35:11.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.285+0000 7f8eaaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:11.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.285+0000 7f8eaaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:11.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.285+0000 7f8eaaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:11.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.285+0000 7f8eaaffd640 10 monclient(hunting): handle_auth_done global_id 184100 payload 824 2024-04-26T00:35:11.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.285+0000 7f8eaaffd640 1 -- 172.21.15.37:0/1215210130 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8eac04f510 con 0x7f8eac09fc50 2024-04-26T00:35:11.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.285+0000 7f8eaaffd640 1 -- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac09fc50 msgr2=0x7f8eac0a0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:11.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.285+0000 7f8eaaffd640 1 --2- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac09fc50 0x7f8eac0a0030 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c007d50 tx=0x7f8e9c012d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:11.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.289+0000 7f8e937fe640 1 -- 172.21.15.37:0/1215210130 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e9c010cd0 con 0x7f8eac09fc50 2024-04-26T00:35:11.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.289+0000 7f8e937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:11.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.289+0000 7f8e937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:35:11.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.289+0000 7f8e937fe640 10 monclient: dump: 2024-04-26T00:35:11.285 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:11.285 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:11.285 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:11.285 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:11.286 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:11.286 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:11.286 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:11.286 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:11.286 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:11.286 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:11.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.289+0000 7f8e937fe640 1 -- 172.21.15.37:0/1215210130 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8e9c010cd0 con 0x7f8eac09fc50 2024-04-26T00:35:11.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.289+0000 7f8e937fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:11.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.289+0000 7f8e937fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:35:11.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.289+0000 7f8e937fe640 1 -- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac09fc50 msgr2=0x7f8eac0a0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:11.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.289+0000 7f8e937fe640 1 --2- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac09fc50 0x7f8eac0a0030 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c007d50 tx=0x7f8e9c012d50 comp rx=0 tx=0).stop 2024-04-26T00:35:11.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.289+0000 7f8e937fe640 1 -- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac09fc50 msgr2=0x7f8eac0a0030 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:35:11.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.289+0000 7f8e937fe640 1 --2- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac09fc50 0x7f8eac0a0030 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c007d50 tx=0x7f8e9c012d50 comp rx=0 tx=0).stop 2024-04-26T00:35:11.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.297+0000 7f8eb129f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:35:11.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.297+0000 7f8eb129f640 1 --2- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac0a0570 0x7f8eac06fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:11.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.297+0000 7f8eb129f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:11.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.297+0000 7f8eb129f640 1 -- 172.21.15.37:0/1215210130 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8eac0a1470 con 0x7f8eac0a0570 2024-04-26T00:35:11.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.297+0000 7f8eaaffd640 1 --2- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac0a0570 0x7f8eac06fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:11.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.297+0000 7f8eaaffd640 10 monclient: get_auth_request con 0x7f8eac0a0570 auth_method 0 2024-04-26T00:35:11.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.297+0000 7f8eaaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:11.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.297+0000 7f8eaaffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:11.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.297+0000 7f8eaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:11.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.297+0000 7f8eaaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:11.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.297+0000 7f8eaaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:11.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.297+0000 7f8eaaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:11.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.297+0000 7f8eaaffd640 10 monclient(hunting): handle_auth_done global_id 184103 payload 824 2024-04-26T00:35:11.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.297+0000 7f8eaaffd640 1 -- 172.21.15.37:0/1215210130 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8eac0a0f60 con 0x7f8eac0a0570 2024-04-26T00:35:11.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.301+0000 7f8eaaffd640 1 -- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac0a0570 msgr2=0x7f8eac06fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:11.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.301+0000 7f8eaaffd640 1 --2- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac0a0570 0x7f8eac06fc50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c007d50 tx=0x7f8e9c004950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:11.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.301+0000 7f8e937fe640 1 -- 172.21.15.37:0/1215210130 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e9c01c810 con 0x7f8eac0a0570 2024-04-26T00:35:11.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.301+0000 7f8e937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:11.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.301+0000 7f8e937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:35:11.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.301+0000 7f8e937fe640 10 monclient: dump: 2024-04-26T00:35:11.299 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:11.299 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:11.299 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:11.299 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:11.299 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:11.299 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:11.299 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:11.300 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:11.300 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:11.300 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:11.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.301+0000 7f8e937fe640 1 -- 172.21.15.37:0/1215210130 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4547 (secure 0 0 0) 0x7f8e9c010e80 con 0x7f8eac0a0570 2024-04-26T00:35:11.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.301+0000 7f8e937fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:35:11.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.301+0000 7f8e937fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:35:11.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.301+0000 7f8e937fe640 1 -- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac0a0570 msgr2=0x7f8eac06fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:11.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.301+0000 7f8e937fe640 1 --2- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac0a0570 0x7f8eac06fc50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c007d50 tx=0x7f8e9c004950 comp rx=0 tx=0).stop 2024-04-26T00:35:11.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.301+0000 7f8e937fe640 1 -- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac0a0570 msgr2=0x7f8eac06fc50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:35:11.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.301+0000 7f8e937fe640 1 --2- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8eac0a0570 0x7f8eac06fc50 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c007d50 tx=0x7f8e9c004950 comp rx=0 tx=0).stop 2024-04-26T00:35:11.301 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:35:11.301 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:35:11.301 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:35:11.301 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:35:11.301 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 114, 2024-04-26T00:35:11.301 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 10447, 2024-04-26T00:35:11.301 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:35:11.301 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:35:11.302 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:35:11.302 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:35:11.302 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:11.302 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 4, 2024-04-26T00:35:11.302 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:35:11.302 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:35:11.302 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:35:11.302 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:11.302 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:11.302 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:11.302 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:11.302 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:11.302 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:11.302 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:11.302 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:11.303 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:11.303 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:11.303 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:11.303 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:11.303 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:35:11.303 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:35:11.303 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:11.303 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:11.303 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:11.303 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:11.303 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:11.303 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:11.303 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:11.303 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:11.303 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:11.304 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:11.304 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:11.304 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:11.304 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:11.304 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:35:11.304 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:35:11.304 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:35:11.304 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:35:11.304 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:35:11.304 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:35:11.304 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:35:11.305 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:35:11.305 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:35:11.305 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:35:11.305 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:35:11.305 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:35:11.305 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:35:11.305 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:35:11.305 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:35:11.305 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:35:11.305 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:35:11.305 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:11.305 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:11.305 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:11.305 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:11.306 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:11.306 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:11.306 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:11.306 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:11.306 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:11.306 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:11.306 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:11.306 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:11.306 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:35:11.306 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:11.306 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:35:11.306 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:11.306 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:35:11.306 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:35:11.306 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:11.307 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:11.307 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:11.307 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:11.307 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:35:11.307 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:11.307 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:11.307 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:11.307 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:11.307 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:35:11.307 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:11.307 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:11.307 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:11.307 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:11.307 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:35:11.307 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:35:11.308 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:11.308 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:11.308 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:11.308 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:11.308 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:11.309 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:35:11.309 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:35:11.309 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:11.309 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:11.309 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:11.309 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:11.309 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:35:11.309 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:11.309 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:11.309 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:11.309 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:11.309 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:35:11.309 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:11.309 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:11.309 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:11.310 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:11.310 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:35:11.310 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:35:11.310 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:11.310 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:11.310 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:11.310 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:11.310 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:11.310 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:35:11.310 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:35:11.310 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:11.310 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:11.310 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:11.310 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:11.311 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:35:11.311 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:11.311 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:11.311 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:11.311 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:11.311 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:35:11.311 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:11.311 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:11.311 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:11.311 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:11.311 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:35:11.311 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:35:11.311 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:11.311 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:11.311 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:11.312 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:11.312 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:11.312 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:11.312 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:35:11.312 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:35:11.312 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:11.312 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:11.312 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:11.312 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:35:11.312 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:11.312 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:11.312 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:35:11.312 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:11.312 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:11.312 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:11.313 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:35:11.313 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:11.313 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:11.313 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:11.313 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:35:11.313 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:35:11.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.301+0000 7f8eb129f640 1 -- 172.21.15.37:0/1215210130 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8e98035ed0 msgr2=0x7f8e98038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:11.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.301+0000 7f8eb129f640 1 --2- 172.21.15.37:0/1215210130 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8e98035ed0 0x7f8e98038390 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f8eac0764d0 tx=0x7f8ea40a7070 comp rx=0 tx=0).stop 2024-04-26T00:35:11.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.301+0000 7f8eb129f640 10 monclient: shutdown 2024-04-26T00:35:11.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.301+0000 7f8eb129f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:11.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.301+0000 7f8eb129f640 1 -- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8eac130eb0 msgr2=0x7f8eac12ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:11.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.301+0000 7f8eb129f640 1 --2- 172.21.15.37:0/1215210130 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8eac130eb0 0x7f8eac12ec70 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8ea000c480 tx=0x7f8ea000c940 comp rx=0 tx=0).stop 2024-04-26T00:35:11.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.301+0000 7f8eab7fe640 1 -- 172.21.15.37:0/1215210130 reap_dead start 2024-04-26T00:35:11.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.301+0000 7f8eab7fe640 1 -- 172.21.15.37:0/1215210130 reap_dead start 2024-04-26T00:35:11.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.301+0000 7f8eb129f640 1 -- 172.21.15.37:0/1215210130 shutdown_connections 2024-04-26T00:35:11.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.301+0000 7f8eb129f640 1 -- 172.21.15.37:0/1215210130 >> 172.21.15.37:0/1215210130 conn(0x7f8eac0a1d50 msgr2=0x7f8eac0a3930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:11.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.301+0000 7f8eb129f640 1 -- 172.21.15.37:0/1215210130 shutdown_connections 2024-04-26T00:35:11.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.301+0000 7f8eb129f640 1 -- 172.21.15.37:0/1215210130 wait complete. 2024-04-26T00:35:11.321 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:35:11.321 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:35:11.321 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:35:11.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.449+0000 7fda5cc5d640 10 monclient: get_monmap_and_config 2024-04-26T00:35:11.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.449+0000 7fda5cc5d640 10 monclient: build_initial_monmap 2024-04-26T00:35:11.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.449+0000 7fda5cc5d640 10 monclient: monmap: 2024-04-26T00:35:11.446 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:11.446 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:11.446 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:35:11.452679+0000 2024-04-26T00:35:11.446 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:35:11.452679+0000 2024-04-26T00:35:11.446 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:11.446 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:11.446 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:35:11.447 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:35:11.447 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:35:11.447 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:35:11.447 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:35:11.447 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:11.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.449+0000 7fda5cc5d640 1 Processor -- start 2024-04-26T00:35:11.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.449+0000 7fda5cc5d640 1 -- start start 2024-04-26T00:35:11.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.449+0000 7fda5cc5d640 10 monclient: init 2024-04-26T00:35:11.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.449+0000 7fda5cc5d640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:11.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.449+0000 7fda5cc5d640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-04-26T00:35:11.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.449+0000 7fda5cc5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fda5813b6e0 addr v1:172.21.15.37:6789/0 2024-04-26T00:35:11.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.449+0000 7fda5cc5d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fda5809de30 0x7fda581404d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:11.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.449+0000 7fda5cc5d640 10 monclient(hunting): picked mon.noname-b con 0x7fda5809de30 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:11.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.449+0000 7fda5cc5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fda58140a10 addr v1:172.21.15.45:6789/0 2024-04-26T00:35:11.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.449+0000 7fda5cc5d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda5809e210 con 0x7fda5813b6e0 2024-04-26T00:35:11.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.449+0000 7fda5cc5d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda580732a0 con 0x7fda58140a10 2024-04-26T00:35:11.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.449+0000 7fda5cc5d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:11.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.449+0000 7fda5cc5d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fda5809cb90 con 0x7fda5809de30 2024-04-26T00:35:11.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.449+0000 7fda5cc5d640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:11.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.449+0000 7fda5cc5d640 10 monclient(hunting): authenticate will time out at 1892.758388s 2024-04-26T00:35:11.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.449+0000 7fda55d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fda5809de30 0x7fda581404d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:11.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.449+0000 7fda5659c640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fda5813b6e0 0x7fda5813bac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:36022/0 (socket says 172.21.15.37:36022) 2024-04-26T00:35:11.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.449+0000 7fda5659c640 1 -- 172.21.15.37:0/2118180 learned_addr learned my addr 172.21.15.37:0/2118180 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:35:11.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.449+0000 7fda55d9b640 10 monclient(hunting): get_auth_request con 0x7fda5809de30 auth_method 0 2024-04-26T00:35:11.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.449+0000 7fda55d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:11.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.449+0000 7fda55d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:11.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.449+0000 7fda55d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:11.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.449+0000 7fda55d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:11.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.449+0000 7fda55d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:11.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.449+0000 7fda55d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:11.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.449+0000 7fda55d9b640 10 monclient(hunting): handle_auth_done global_id 184106 payload 274 2024-04-26T00:35:11.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.449+0000 7fda55d9b640 1 -- 172.21.15.37:0/2118180 >> v1:172.21.15.45:6789/0 conn(0x7fda58140a10 legacy=0x7fda58142e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:11.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.449+0000 7fda55d9b640 1 -- 172.21.15.37:0/2118180 >> v1:172.21.15.37:6789/0 conn(0x7fda5813b6e0 legacy=0x7fda5813bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:11.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.449+0000 7fda55d9b640 10 monclient: _finish_hunting 0 2024-04-26T00:35:11.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.449+0000 7fda55d9b640 1 monclient: found mon.noname-b 2024-04-26T00:35:11.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.449+0000 7fda55d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:11.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.449+0000 7fda55d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:35:11.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.449+0000 7fda55d9b640 1 -- 172.21.15.37:0/2118180 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda5809c140 con 0x7fda5809de30 2024-04-26T00:35:11.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.453+0000 7fda55d9b640 1 -- 172.21.15.37:0/2118180 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fda5809de30 msgr2=0x7fda581404d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:11.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.453+0000 7fda55d9b640 1 --2- 172.21.15.37:0/2118180 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fda5809de30 0x7fda581404d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fda40009df0 tx=0x7fda4001c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc92535c02f78eec server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:11.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.453+0000 7fda5559a640 1 -- 172.21.15.37:0/2118180 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda400ac070 con 0x7fda5809de30 2024-04-26T00:35:11.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.453+0000 7fda5559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:11.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.453+0000 7fda5559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:35:11.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.453+0000 7fda5559a640 10 monclient: dump: 2024-04-26T00:35:11.454 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:11.454 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:11.454 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:11.454 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:11.454 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:11.454 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:11.455 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:11.455 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:11.455 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:11.455 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:11.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.453+0000 7fda5559a640 10 monclient: _finish_auth 0 2024-04-26T00:35:11.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.453+0000 7fda5559a640 10 monclient: _check_auth_tickets 2024-04-26T00:35:11.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.453+0000 7fda5559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:11.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.453+0000 7fda5cc5d640 5 monclient: authenticate success, global_id 184106 2024-04-26T00:35:11.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.453+0000 7fda5559a640 1 -- 172.21.15.37:0/2118180 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda400a6700 con 0x7fda5809de30 2024-04-26T00:35:11.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.453+0000 7fda5559a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:11.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.453+0000 7fda5cc5d640 10 monclient: get_monmap_and_config success 2024-04-26T00:35:11.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.453+0000 7fda5cc5d640 4 set_mon_vals no callback set 2024-04-26T00:35:11.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.453+0000 7fda5559a640 1 -- 172.21.15.37:0/2118180 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda400a6ad0 con 0x7fda5809de30 2024-04-26T00:35:11.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.453+0000 7fda5559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:11.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.453+0000 7fda5559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:35:11.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.453+0000 7fda5559a640 10 monclient: dump: 2024-04-26T00:35:11.456 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:11.456 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:11.456 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:11.456 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:11.456 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:11.456 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:11.456 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:11.456 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:11.456 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:11.456 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:11.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda5cc5d640 10 monclient: shutdown 2024-04-26T00:35:11.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda5cc5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:11.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda5cc5d640 1 -- 172.21.15.37:0/2118180 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fda5809de30 msgr2=0x7fda581404d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:11.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda5cc5d640 1 --2- 172.21.15.37:0/2118180 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fda5809de30 0x7fda581404d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fda40009df0 tx=0x7fda4001c680 comp rx=0 tx=0).stop 2024-04-26T00:35:11.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda5cc5d640 1 -- 172.21.15.37:0/2118180 shutdown_connections 2024-04-26T00:35:11.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda5cc5d640 1 --2- 172.21.15.37:0/2118180 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fda5809de30 0x7fda581404d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:11.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda5cc5d640 1 -- 172.21.15.37:0/2118180 >> 172.21.15.37:0/2118180 conn(0x7fda580a1d50 msgr2=0x7fda580a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:11.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda5cc5d640 1 -- 172.21.15.37:0/2118180 shutdown_connections 2024-04-26T00:35:11.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda5cc5d640 1 -- 172.21.15.37:0/2118180 wait complete. 2024-04-26T00:35:11.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda5cc5d640 10 monclient: build_initial_monmap 2024-04-26T00:35:11.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda5cc5d640 10 monclient: monmap: 2024-04-26T00:35:11.457 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:11.457 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:35:11.457 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:35:11.457 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:35:11.457 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:11.457 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:35:11.457 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:35:11.457 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:35:11.458 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:35:11.458 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:11.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda5cc5d640 1 Processor -- start 2024-04-26T00:35:11.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda5cc5d640 1 -- start start 2024-04-26T00:35:11.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda5cc5d640 10 monclient: init 2024-04-26T00:35:11.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda5cc5d640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:11.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda5cc5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:35:11.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda5cc5d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fda5809de30 0x7fda58139650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:11.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda5cc5d640 10 monclient(hunting): picked mon.noname-a con 0x7fda5809de30 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:35:11.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda5cc5d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda5813b6e0 0x7fda58139b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:11.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda5cc5d640 10 monclient(hunting): picked mon.noname-b con 0x7fda5813b6e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:35:11.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda5cc5d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fda58140a10 0x7fda581d34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:11.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda5cc5d640 10 monclient(hunting): picked mon.noname-c con 0x7fda58140a10 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:11.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda5cc5d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:11.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda5cc5d640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fda581470d0 con 0x7fda5809de30 2024-04-26T00:35:11.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda5cc5d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:11.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda5cc5d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fda58146f50 con 0x7fda5813b6e0 2024-04-26T00:35:11.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda5cc5d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:11.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda5cc5d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fda58147250 con 0x7fda58140a10 2024-04-26T00:35:11.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda5cc5d640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:11.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda5cc5d640 10 monclient(hunting): authenticate will time out at 1892.765815s 2024-04-26T00:35:11.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda56d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fda58140a10 0x7fda581d34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:11.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda56d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fda58140a10 0x7fda581d34c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:60310/0 (socket says 172.21.15.37:60310) 2024-04-26T00:35:11.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda56d9d640 1 -- 172.21.15.37:0/555119991 learned_addr learned my addr 172.21.15.37:0/555119991 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:11.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda56d9d640 10 monclient(hunting): get_auth_request con 0x7fda58140a10 auth_method 0 2024-04-26T00:35:11.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda56d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:11.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda56d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:11.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda56d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:11.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda56d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:11.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda56d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:11.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda56d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:11.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda5659c640 1 --2- 172.21.15.37:0/555119991 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fda5809de30 0x7fda58139650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:11.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda55d9b640 1 --2- 172.21.15.37:0/555119991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda5813b6e0 0x7fda58139b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:11.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda5659c640 10 monclient(hunting): get_auth_request con 0x7fda5809de30 auth_method 0 2024-04-26T00:35:11.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda5659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:11.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda5659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:11.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda5659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:11.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda55d9b640 10 monclient(hunting): get_auth_request con 0x7fda5813b6e0 auth_method 0 2024-04-26T00:35:11.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda55d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:11.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda55d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:11.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda55d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:11.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda56d9d640 10 monclient(hunting): handle_auth_done global_id 184109 payload 824 2024-04-26T00:35:11.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda56d9d640 1 -- 172.21.15.37:0/555119991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda5813b6e0 msgr2=0x7fda58139b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:11.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda56d9d640 1 --2- 172.21.15.37:0/555119991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda5813b6e0 0x7fda58139b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:11.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda56d9d640 1 -- 172.21.15.37:0/555119991 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fda5809de30 msgr2=0x7fda58139650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:11.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda56d9d640 1 --2- 172.21.15.37:0/555119991 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fda5809de30 0x7fda58139650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:11.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda56d9d640 10 monclient: _finish_hunting 0 2024-04-26T00:35:11.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda56d9d640 1 monclient: found mon.noname-c 2024-04-26T00:35:11.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda56d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:11.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda56d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:35:11.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.457+0000 7fda56d9d640 1 -- 172.21.15.37:0/555119991 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda5806f5a0 con 0x7fda58140a10 2024-04-26T00:35:11.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda56d9d640 1 -- 172.21.15.37:0/555119991 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fda58140a10 msgr2=0x7fda581d34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:11.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda56d9d640 1 --2- 172.21.15.37:0/555119991 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fda58140a10 0x7fda581d34c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fda5000c390 tx=0x7fda5000c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:11.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda4effd640 1 -- 172.21.15.37:0/555119991 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda500119a0 con 0x7fda58140a10 2024-04-26T00:35:11.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:11.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda4effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:35:11.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda4effd640 10 monclient: dump: 2024-04-26T00:35:11.465 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:11.465 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:11.465 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:11.465 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:11.465 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:11.465 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:11.465 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:11.465 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:11.465 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:11.465 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:11.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda4effd640 10 monclient: _finish_auth 0 2024-04-26T00:35:11.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda4effd640 10 monclient: _check_auth_tickets 2024-04-26T00:35:11.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:11.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda4effd640 1 -- 172.21.15.37:0/555119991 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda50017070 con 0x7fda58140a10 2024-04-26T00:35:11.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda5cc5d640 5 monclient: authenticate success, global_id 184109 2024-04-26T00:35:11.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda5cc5d640 10 monclient: _renew_subs 2024-04-26T00:35:11.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda5cc5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:35:11.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda5cc5d640 1 -- 172.21.15.37:0/555119991 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda58071890 con 0x7fda58140a10 2024-04-26T00:35:11.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda4effd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:11.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda4effd640 1 -- 172.21.15.37:0/555119991 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda50011d80 con 0x7fda58140a10 2024-04-26T00:35:11.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:11.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda4effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:35:11.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda4effd640 10 monclient: dump: 2024-04-26T00:35:11.466 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:11.466 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:11.466 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:11.467 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:11.467 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:11.467 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:11.467 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:11.467 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:11.467 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:11.467 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:11.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda4ffff640 4 set_mon_vals no callback set 2024-04-26T00:35:11.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda5cc5d640 10 monclient: _renew_subs 2024-04-26T00:35:11.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda5cc5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:35:11.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda5cc5d640 1 -- 172.21.15.37:0/555119991 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fda5813a0d0 con 0x7fda58140a10 2024-04-26T00:35:11.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda5cc5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:11.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda5cc5d640 1 --2- 172.21.15.37:0/555119991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda581d3a00 0x7fda581d3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:11.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda5cc5d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:11.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda5cc5d640 1 -- 172.21.15.37:0/555119991 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fda581402e0 con 0x7fda581d3a00 2024-04-26T00:35:11.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda4effd640 1 -- 172.21.15.37:0/555119991 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fda5001c460 con 0x7fda58140a10 2024-04-26T00:35:11.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda5659c640 1 --2- 172.21.15.37:0/555119991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda581d3a00 0x7fda581d3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:11.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda4effd640 1 --2- 172.21.15.37:0/555119991 >> v2:172.21.15.45:6812/2379918623 conn(0x7fda38035ed0 0x7fda38038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:11.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda5659c640 10 monclient: get_auth_request con 0x7fda581d3a00 auth_method 0 2024-04-26T00:35:11.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda5659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:11.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda5659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:11.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda5659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:11.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda4effd640 1 -- 172.21.15.37:0/555119991 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(596..596 src has 370..596) ==== 3943+0+0 (secure 0 0 0) 0x7fda500486b0 con 0x7fda58140a10 2024-04-26T00:35:11.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda5659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:11.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda5659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:11.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda5659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:11.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda55d9b640 1 --2- 172.21.15.37:0/555119991 >> v2:172.21.15.45:6812/2379918623 conn(0x7fda38035ed0 0x7fda38038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:11.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda55d9b640 10 monclient: get_auth_request con 0x7fda38035ed0 auth_method 0 2024-04-26T00:35:11.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda5659c640 10 monclient(hunting): handle_auth_done global_id 174187 payload 824 2024-04-26T00:35:11.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.461+0000 7fda5659c640 1 -- 172.21.15.37:0/555119991 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fda581d4320 con 0x7fda581d3a00 2024-04-26T00:35:11.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.465+0000 7fda5659c640 1 -- 172.21.15.37:0/555119991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda581d3a00 msgr2=0x7fda581d3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:11.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.465+0000 7fda5659c640 1 --2- 172.21.15.37:0/555119991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda581d3a00 0x7fda581d3de0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fda4800e500 tx=0x7fda4800e9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:11.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.465+0000 7fda55d9b640 1 --2- 172.21.15.37:0/555119991 >> v2:172.21.15.45:6812/2379918623 conn(0x7fda38035ed0 0x7fda38038390 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fda580764d0 tx=0x7fda400a7000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:11.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.465+0000 7fda4effd640 1 -- 172.21.15.37:0/555119991 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda48004e40 con 0x7fda581d3a00 2024-04-26T00:35:11.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.465+0000 7fda4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:11.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.465+0000 7fda4effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:35:11.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.465+0000 7fda4effd640 10 monclient: dump: 2024-04-26T00:35:11.471 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:11.471 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:11.471 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:11.471 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:11.471 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:11.471 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:11.471 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:11.471 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:11.471 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:11.472 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:11.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.465+0000 7fda4effd640 1 -- 172.21.15.37:0/555119991 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fda48018070 con 0x7fda581d3a00 2024-04-26T00:35:11.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.465+0000 7fda4effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:11.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.465+0000 7fda4effd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:35:11.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.465+0000 7fda4effd640 1 -- 172.21.15.37:0/555119991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda581d3a00 msgr2=0x7fda581d3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:11.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.465+0000 7fda4effd640 1 --2- 172.21.15.37:0/555119991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda581d3a00 0x7fda581d3de0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fda4800e500 tx=0x7fda4800e9c0 comp rx=0 tx=0).stop 2024-04-26T00:35:11.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.465+0000 7fda4effd640 1 -- 172.21.15.37:0/555119991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda581d3a00 msgr2=0x7fda581d3de0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:35:11.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.465+0000 7fda4effd640 1 --2- 172.21.15.37:0/555119991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda581d3a00 0x7fda581d3de0 secure :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fda4800e500 tx=0x7fda4800e9c0 comp rx=0 tx=0).stop 2024-04-26T00:35:11.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.473+0000 7fda5cc5d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:35:11.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.473+0000 7fda5cc5d640 1 --2- 172.21.15.37:0/555119991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda5813c2a0 0x7fda5813c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:11.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.473+0000 7fda5cc5d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:11.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.473+0000 7fda5cc5d640 1 -- 172.21.15.37:0/555119991 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fda5804f510 con 0x7fda5813c2a0 2024-04-26T00:35:11.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.473+0000 7fda5659c640 1 --2- 172.21.15.37:0/555119991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda5813c2a0 0x7fda5813c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:11.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.473+0000 7fda5659c640 10 monclient: get_auth_request con 0x7fda5813c2a0 auth_method 0 2024-04-26T00:35:11.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.473+0000 7fda5659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:11.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.473+0000 7fda5659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:11.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.473+0000 7fda5659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:11.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.473+0000 7fda5659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:11.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.473+0000 7fda5659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:11.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.473+0000 7fda5659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:11.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.473+0000 7fda5659c640 10 monclient(hunting): handle_auth_done global_id 174190 payload 824 2024-04-26T00:35:11.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.473+0000 7fda5659c640 1 -- 172.21.15.37:0/555119991 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fda581407f0 con 0x7fda5813c2a0 2024-04-26T00:35:11.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.477+0000 7fda5659c640 1 -- 172.21.15.37:0/555119991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda5813c2a0 msgr2=0x7fda5813c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:11.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.477+0000 7fda5659c640 1 --2- 172.21.15.37:0/555119991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda5813c2a0 0x7fda5813c680 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fda4800e4d0 tx=0x7fda48004950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:11.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.477+0000 7fda4effd640 1 -- 172.21.15.37:0/555119991 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda480064e0 con 0x7fda5813c2a0 2024-04-26T00:35:11.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.477+0000 7fda4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:11.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.477+0000 7fda4effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:35:11.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.477+0000 7fda4effd640 10 monclient: dump: 2024-04-26T00:35:11.474 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:11.474 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:11.474 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:11.474 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:11.474 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:11.474 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:11.474 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:11.474 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:11.474 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:11.475 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:11.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.477+0000 7fda4effd640 1 -- 172.21.15.37:0/555119991 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4717 (secure 0 0 0) 0x7fda480064e0 con 0x7fda5813c2a0 2024-04-26T00:35:11.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.477+0000 7fda4effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:35:11.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.477+0000 7fda4effd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:35:11.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.477+0000 7fda4effd640 1 -- 172.21.15.37:0/555119991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda5813c2a0 msgr2=0x7fda5813c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:11.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.477+0000 7fda4effd640 1 --2- 172.21.15.37:0/555119991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda5813c2a0 0x7fda5813c680 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fda4800e4d0 tx=0x7fda48004950 comp rx=0 tx=0).stop 2024-04-26T00:35:11.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.477+0000 7fda4effd640 1 -- 172.21.15.37:0/555119991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda5813c2a0 msgr2=0x7fda5813c680 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:35:11.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.477+0000 7fda4effd640 1 --2- 172.21.15.37:0/555119991 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda5813c2a0 0x7fda5813c680 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fda4800e4d0 tx=0x7fda48004950 comp rx=0 tx=0).stop 2024-04-26T00:35:11.476 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:35:11.476 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:35:11.476 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:35:11.476 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:35:11.476 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 114, 2024-04-26T00:35:11.476 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 54095, 2024-04-26T00:35:11.476 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:35:11.476 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:35:11.476 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:35:11.476 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:35:11.477 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:11.477 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 5, 2024-04-26T00:35:11.477 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:35:11.477 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:35:11.477 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:35:11.477 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:11.477 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:11.477 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:11.477 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:11.477 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:11.477 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:11.477 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:11.477 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:11.477 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:11.477 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:11.478 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:11.478 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:11.478 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:35:11.478 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:35:11.478 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:11.478 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:11.478 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:11.478 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:11.478 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:11.478 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:11.478 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:11.478 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:11.478 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:11.478 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:11.479 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:11.479 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:11.479 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:11.479 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:35:11.479 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:35:11.479 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:35:11.479 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:35:11.479 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:35:11.479 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:35:11.479 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:35:11.479 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:35:11.479 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:35:11.479 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:35:11.479 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:35:11.479 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:35:11.480 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:35:11.480 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:35:11.480 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:35:11.480 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:35:11.480 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:35:11.480 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:11.480 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:11.480 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:11.480 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:11.480 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:11.480 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:11.480 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:11.480 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:11.480 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:11.480 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:11.481 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:11.481 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:11.481 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:35:11.481 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:11.481 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:35:11.481 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:11.481 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:35:11.481 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:35:11.481 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:11.481 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:11.481 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:11.481 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:11.481 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:35:11.481 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:11.481 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:11.482 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:11.482 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:11.482 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:35:11.482 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:11.482 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:11.482 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:11.482 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:11.482 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:35:11.482 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:35:11.482 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:11.482 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:11.482 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:11.482 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:11.482 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:11.483 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:35:11.483 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:35:11.483 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:11.483 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:11.483 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:11.483 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:11.483 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:35:11.483 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:11.483 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:11.483 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:11.483 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:11.483 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:35:11.483 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:11.483 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:11.483 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:11.484 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:11.484 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:35:11.484 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:35:11.484 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:11.484 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:11.484 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:11.484 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:11.484 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:11.484 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:35:11.485 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:35:11.485 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:11.485 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:11.485 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:11.485 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:11.485 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:35:11.485 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:11.485 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:11.485 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:11.485 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:11.485 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:35:11.485 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:11.485 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:11.485 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:11.486 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:11.486 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:35:11.486 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:35:11.486 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:11.486 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:11.486 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:11.486 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:11.486 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:11.486 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:11.486 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:35:11.486 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:35:11.486 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:11.486 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:11.486 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:11.487 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:35:11.487 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:11.487 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:11.487 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:35:11.487 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:11.487 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:11.487 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:11.487 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:35:11.487 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:11.487 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:11.487 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:35:11.487 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:11.487 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:11.487 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:11.487 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:35:11.488 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:11.488 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:11.488 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:11.488 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:35:11.488 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:35:11.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.477+0000 7fda5cc5d640 1 -- 172.21.15.37:0/555119991 >> v2:172.21.15.45:6812/2379918623 conn(0x7fda38035ed0 msgr2=0x7fda38038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:11.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.477+0000 7fda5cc5d640 1 --2- 172.21.15.37:0/555119991 >> v2:172.21.15.45:6812/2379918623 conn(0x7fda38035ed0 0x7fda38038390 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fda580764d0 tx=0x7fda400a7000 comp rx=0 tx=0).stop 2024-04-26T00:35:11.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.477+0000 7fda5cc5d640 10 monclient: shutdown 2024-04-26T00:35:11.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.477+0000 7fda5cc5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:11.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.477+0000 7fda5cc5d640 1 -- 172.21.15.37:0/555119991 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fda58140a10 msgr2=0x7fda581d34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:11.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.477+0000 7fda5cc5d640 1 --2- 172.21.15.37:0/555119991 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fda58140a10 0x7fda581d34c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fda5000c390 tx=0x7fda5000c850 comp rx=0 tx=0).stop 2024-04-26T00:35:11.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.477+0000 7fda56d9d640 1 -- 172.21.15.37:0/555119991 reap_dead start 2024-04-26T00:35:11.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.477+0000 7fda56d9d640 1 -- 172.21.15.37:0/555119991 reap_dead start 2024-04-26T00:35:11.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.477+0000 7fda5cc5d640 1 -- 172.21.15.37:0/555119991 shutdown_connections 2024-04-26T00:35:11.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.477+0000 7fda5cc5d640 1 -- 172.21.15.37:0/555119991 >> 172.21.15.37:0/555119991 conn(0x7fda580a1d50 msgr2=0x7fda580a3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:11.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.477+0000 7fda5cc5d640 1 -- 172.21.15.37:0/555119991 shutdown_connections 2024-04-26T00:35:11.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.477+0000 7fda5cc5d640 1 -- 172.21.15.37:0/555119991 wait complete. 2024-04-26T00:35:11.491 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:35:11.491 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-26T00:35:11.491 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-26T00:35:11.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.605+0000 7fcb58917640 10 monclient: get_monmap_and_config 2024-04-26T00:35:11.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.605+0000 7fcb58917640 10 monclient: build_initial_monmap 2024-04-26T00:35:11.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.605+0000 7fcb58917640 10 monclient: monmap: 2024-04-26T00:35:11.604 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:11.605 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:11.605 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:35:11.611199+0000 2024-04-26T00:35:11.605 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:35:11.611199+0000 2024-04-26T00:35:11.605 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:11.605 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:11.605 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:35:11.605 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:35:11.605 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:35:11.605 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:35:11.605 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:35:11.605 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:11.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb58917640 1 Processor -- start 2024-04-26T00:35:11.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb58917640 1 -- start start 2024-04-26T00:35:11.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb58917640 10 monclient: init 2024-04-26T00:35:11.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb58917640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:11.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb58917640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-04-26T00:35:11.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb58917640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcb54136da0 addr v1:172.21.15.37:6789/0 2024-04-26T00:35:11.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb58917640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fcb54137180 0x7fcb541375c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:11.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb58917640 10 monclient(hunting): picked mon.noname-c con 0x7fcb54137180 addr v2:172.21.15.45:3300/0 2024-04-26T00:35:11.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb58917640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb5413c2d0 0x7fcb5413e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:11.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb58917640 10 monclient(hunting): picked mon.noname-b con 0x7fcb5413c2d0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:11.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb58917640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb5409b4f0 con 0x7fcb54136da0 2024-04-26T00:35:11.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb58917640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:11.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb58917640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fcb5409b210 con 0x7fcb54137180 2024-04-26T00:35:11.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb58917640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:11.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb58917640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fcb5409b7d0 con 0x7fcb5413c2d0 2024-04-26T00:35:11.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb58917640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:11.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb58917640 10 monclient(hunting): authenticate will time out at 1892.917196s 2024-04-26T00:35:11.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb52d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb5413c2d0 0x7fcb5413e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:11.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb5259c640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fcb54136da0 0x7fcb5409bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:36024/0 (socket says 172.21.15.37:36024) 2024-04-26T00:35:11.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb5259c640 1 -- 172.21.15.37:0/3207511211 learned_addr learned my addr 172.21.15.37:0/3207511211 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:35:11.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb52d9d640 10 monclient(hunting): get_auth_request con 0x7fcb5413c2d0 auth_method 0 2024-04-26T00:35:11.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb52d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:11.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb52d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:11.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb52d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:11.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb52d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:11.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb52d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:11.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb52d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:11.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb49d9b640 1 --2- 172.21.15.37:0/3207511211 >> v2:172.21.15.45:3300/0 conn(0x7fcb54137180 0x7fcb541375c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:11.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb49d9b640 10 monclient(hunting): get_auth_request con 0x7fcb54137180 auth_method 0 2024-04-26T00:35:11.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb49d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:11.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb49d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:11.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb49d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:11.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb52d9d640 10 monclient(hunting): handle_auth_done global_id 184112 payload 274 2024-04-26T00:35:11.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb52d9d640 1 -- 172.21.15.37:0/3207511211 >> v2:172.21.15.45:3300/0 conn(0x7fcb54137180 msgr2=0x7fcb541375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:11.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb52d9d640 1 --2- 172.21.15.37:0/3207511211 >> v2:172.21.15.45:3300/0 conn(0x7fcb54137180 0x7fcb541375c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:11.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb52d9d640 1 -- 172.21.15.37:0/3207511211 >> v1:172.21.15.37:6789/0 conn(0x7fcb54136da0 legacy=0x7fcb5409bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:11.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb52d9d640 10 monclient: _finish_hunting 0 2024-04-26T00:35:11.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb52d9d640 1 monclient: found mon.noname-b 2024-04-26T00:35:11.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb52d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:11.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb52d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:35:11.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.609+0000 7fcb52d9d640 1 -- 172.21.15.37:0/3207511211 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb54137b00 con 0x7fcb5413c2d0 2024-04-26T00:35:11.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.613+0000 7fcb52d9d640 1 -- 172.21.15.37:0/3207511211 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb5413c2d0 msgr2=0x7fcb5413e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:11.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.613+0000 7fcb52d9d640 1 --2- 172.21.15.37:0/3207511211 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb5413c2d0 0x7fcb5413e6c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c009d90 tx=0x7fcb4c01c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9636a60fe2c89640 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:11.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.613+0000 7fcb51d9b640 1 -- 172.21.15.37:0/3207511211 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c0b2070 con 0x7fcb5413c2d0 2024-04-26T00:35:11.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.613+0000 7fcb51d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:11.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.613+0000 7fcb51d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:35:11.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.613+0000 7fcb51d9b640 10 monclient: dump: 2024-04-26T00:35:11.613 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:11.613 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:11.613 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:11.613 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:11.613 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:11.613 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:11.613 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:11.614 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:11.614 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:11.614 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:11.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.613+0000 7fcb51d9b640 10 monclient: _finish_auth 0 2024-04-26T00:35:11.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.613+0000 7fcb51d9b640 10 monclient: _check_auth_tickets 2024-04-26T00:35:11.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.613+0000 7fcb51d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:11.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.613+0000 7fcb58917640 5 monclient: authenticate success, global_id 184112 2024-04-26T00:35:11.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.613+0000 7fcb51d9b640 1 -- 172.21.15.37:0/3207511211 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb4c0a5890 con 0x7fcb5413c2d0 2024-04-26T00:35:11.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.613+0000 7fcb58917640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:35:11.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.613+0000 7fcb51d9b640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:11.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.613+0000 7fcb58917640 10 monclient: get_monmap_and_config success 2024-04-26T00:35:11.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.613+0000 7fcb51d9b640 1 -- 172.21.15.37:0/3207511211 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c0a5c60 con 0x7fcb5413c2d0 2024-04-26T00:35:11.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.613+0000 7fcb58917640 4 set_mon_vals no callback set 2024-04-26T00:35:11.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.613+0000 7fcb51d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:11.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.613+0000 7fcb51d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:35:11.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.613+0000 7fcb51d9b640 10 monclient: dump: 2024-04-26T00:35:11.615 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:11.615 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:11.615 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:11.615 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:11.615 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:11.615 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:11.615 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:11.615 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:11.615 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:11.615 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:11.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.613+0000 7fcb58917640 10 monclient: shutdown 2024-04-26T00:35:11.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.613+0000 7fcb58917640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:11.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.613+0000 7fcb58917640 1 -- 172.21.15.37:0/3207511211 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb5413c2d0 msgr2=0x7fcb5413e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:11.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.613+0000 7fcb58917640 1 --2- 172.21.15.37:0/3207511211 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb5413c2d0 0x7fcb5413e6c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c009d90 tx=0x7fcb4c01c7f0 comp rx=0 tx=0).stop 2024-04-26T00:35:11.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.613+0000 7fcb58917640 1 -- 172.21.15.37:0/3207511211 shutdown_connections 2024-04-26T00:35:11.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.613+0000 7fcb58917640 1 --2- 172.21.15.37:0/3207511211 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb5413c2d0 0x7fcb5413e6c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:11.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.613+0000 7fcb58917640 1 --2- 172.21.15.37:0/3207511211 >> v2:172.21.15.45:3300/0 conn(0x7fcb54137180 0x7fcb541375c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:11.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.613+0000 7fcb58917640 1 -- 172.21.15.37:0/3207511211 >> 172.21.15.37:0/3207511211 conn(0x7fcb540a5f00 msgr2=0x7fcb540a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:11.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.613+0000 7fcb58917640 1 -- 172.21.15.37:0/3207511211 shutdown_connections 2024-04-26T00:35:11.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.613+0000 7fcb58917640 1 -- 172.21.15.37:0/3207511211 wait complete. 2024-04-26T00:35:11.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.613+0000 7fcb58917640 10 monclient: build_initial_monmap 2024-04-26T00:35:11.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.613+0000 7fcb58917640 10 monclient: monmap: 2024-04-26T00:35:11.616 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:11.616 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:35:11.616 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:35:11.616 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:35:11.617 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:11.617 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:35:11.617 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:35:11.617 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:35:11.617 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:35:11.617 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:11.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.613+0000 7fcb58917640 1 Processor -- start 2024-04-26T00:35:11.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.613+0000 7fcb58917640 1 -- start start 2024-04-26T00:35:11.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.613+0000 7fcb58917640 10 monclient: init 2024-04-26T00:35:11.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb58917640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:11.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb58917640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:35:11.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb58917640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcb54136da0 0x7fcb5411fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:11.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb58917640 10 monclient(hunting): picked mon.noname-a con 0x7fcb54136da0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:35:11.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb58917640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb54137180 0x7fcb54136050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:11.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb58917640 10 monclient(hunting): picked mon.noname-b con 0x7fcb54137180 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:35:11.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb58917640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb5413c2d0 0x7fcb541cee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:11.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb58917640 10 monclient(hunting): picked mon.noname-c con 0x7fcb5413c2d0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:11.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb58917640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:11.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb58917640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fcb541417b0 con 0x7fcb54136da0 2024-04-26T00:35:11.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb58917640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:11.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb58917640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcb54141630 con 0x7fcb54137180 2024-04-26T00:35:11.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb58917640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:11.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb58917640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fcb54141930 con 0x7fcb5413c2d0 2024-04-26T00:35:11.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb58917640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:11.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb58917640 10 monclient(hunting): authenticate will time out at 1892.924397s 2024-04-26T00:35:11.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb5259c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcb54136da0 0x7fcb5411fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:11.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb5259c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcb54136da0 0x7fcb5411fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:34400/0 (socket says 172.21.15.37:34400) 2024-04-26T00:35:11.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb52d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb5413c2d0 0x7fcb541cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:11.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb5259c640 1 -- 172.21.15.37:0/2167702039 learned_addr learned my addr 172.21.15.37:0/2167702039 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:11.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb5259c640 10 monclient(hunting): get_auth_request con 0x7fcb54136da0 auth_method 0 2024-04-26T00:35:11.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb5259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:11.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb5259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:11.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb5259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:11.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb52d9d640 10 monclient(hunting): get_auth_request con 0x7fcb5413c2d0 auth_method 0 2024-04-26T00:35:11.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb52d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:11.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb52d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:11.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb52d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:11.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb5259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:11.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb5259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:11.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb5259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:11.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb52d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:11.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb52d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:11.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb52d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:11.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb49d9b640 1 --2- 172.21.15.37:0/2167702039 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb54137180 0x7fcb54136050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:11.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb49d9b640 10 monclient(hunting): get_auth_request con 0x7fcb54137180 auth_method 0 2024-04-26T00:35:11.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb49d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:11.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb49d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:11.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb49d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:11.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb5259c640 10 monclient(hunting): handle_auth_done global_id 144516 payload 824 2024-04-26T00:35:11.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb5259c640 1 -- 172.21.15.37:0/2167702039 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb5413c2d0 msgr2=0x7fcb541cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:11.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb5259c640 1 --2- 172.21.15.37:0/2167702039 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb5413c2d0 0x7fcb541cee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:11.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb5259c640 1 -- 172.21.15.37:0/2167702039 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb54137180 msgr2=0x7fcb54136050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:11.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb5259c640 1 --2- 172.21.15.37:0/2167702039 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb54137180 0x7fcb54136050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:11.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb5259c640 10 monclient: _finish_hunting 0 2024-04-26T00:35:11.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb5259c640 1 monclient: found mon.noname-a 2024-04-26T00:35:11.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb5259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:11.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb5259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:35:11.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb5259c640 1 -- 172.21.15.37:0/2167702039 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb54120700 con 0x7fcb54136da0 2024-04-26T00:35:11.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb52d9d640 1 --2- 172.21.15.37:0/2167702039 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb5413c2d0 0x7fcb541cee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:35:11.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb5259c640 1 -- 172.21.15.37:0/2167702039 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcb54136da0 msgr2=0x7fcb5411fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:11.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.617+0000 7fcb5259c640 1 --2- 172.21.15.37:0/2167702039 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcb54136da0 0x7fcb5411fa50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fcb4400c470 tx=0x7fcb4400c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:11.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.621+0000 7fcb4b7fe640 1 -- 172.21.15.37:0/2167702039 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb44007070 con 0x7fcb54136da0 2024-04-26T00:35:11.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.621+0000 7fcb4b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:11.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.621+0000 7fcb4b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:35:11.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.621+0000 7fcb4b7fe640 10 monclient: dump: 2024-04-26T00:35:11.626 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:11.626 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:11.626 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:11.626 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:11.626 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:11.627 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:11.627 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:11.627 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:11.627 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:11.627 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:11.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.621+0000 7fcb4b7fe640 10 monclient: _finish_auth 0 2024-04-26T00:35:11.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.621+0000 7fcb4b7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:35:11.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.621+0000 7fcb4b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:11.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.621+0000 7fcb4b7fe640 1 -- 172.21.15.37:0/2167702039 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb44007210 con 0x7fcb54136da0 2024-04-26T00:35:11.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.621+0000 7fcb4b7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:11.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.621+0000 7fcb4b7fe640 1 -- 172.21.15.37:0/2167702039 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb44007600 con 0x7fcb54136da0 2024-04-26T00:35:11.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.621+0000 7fcb4b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:11.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.621+0000 7fcb4b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:11.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.621+0000 7fcb4b7fe640 10 monclient: dump: 2024-04-26T00:35:11.628 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:11.628 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:11.628 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:11.628 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:11.628 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:11.628 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:11.628 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:11.629 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:11.629 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:11.629 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:11.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.621+0000 7fcb58917640 5 monclient: authenticate success, global_id 144516 2024-04-26T00:35:11.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.621+0000 7fcb58917640 10 monclient: _renew_subs 2024-04-26T00:35:11.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.621+0000 7fcb5159a640 4 set_mon_vals no callback set 2024-04-26T00:35:11.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.621+0000 7fcb58917640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:11.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.621+0000 7fcb58917640 1 -- 172.21.15.37:0/2167702039 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb541209e0 con 0x7fcb54136da0 2024-04-26T00:35:11.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.621+0000 7fcb58917640 10 monclient: _renew_subs 2024-04-26T00:35:11.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.621+0000 7fcb58917640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:11.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.621+0000 7fcb58917640 1 -- 172.21.15.37:0/2167702039 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb5413a230 con 0x7fcb54136da0 2024-04-26T00:35:11.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.621+0000 7fcb58917640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:11.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.621+0000 7fcb58917640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:11.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.621+0000 7fcb58917640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:11.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.621+0000 7fcb58917640 1 -- 172.21.15.37:0/2167702039 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb54066f40 con 0x7fcb54136da0 2024-04-26T00:35:11.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.621+0000 7fcb4b7fe640 1 -- 172.21.15.37:0/2167702039 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fcb4401b400 con 0x7fcb54136da0 2024-04-26T00:35:11.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.621+0000 7fcb4b7fe640 1 --2- 172.21.15.37:0/2167702039 >> v2:172.21.15.45:6812/2379918623 conn(0x7fcb34035ed0 0x7fcb34038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:11.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.621+0000 7fcb4b7fe640 1 -- 172.21.15.37:0/2167702039 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(596..596 src has 370..596) ==== 3943+0+0 (secure 0 0 0) 0x7fcb44048790 con 0x7fcb54136da0 2024-04-26T00:35:11.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.621+0000 7fcb49d9b640 1 --2- 172.21.15.37:0/2167702039 >> v2:172.21.15.45:6812/2379918623 conn(0x7fcb34035ed0 0x7fcb34038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:11.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.621+0000 7fcb49d9b640 10 monclient: get_auth_request con 0x7fcb34035ed0 auth_method 0 2024-04-26T00:35:11.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.625+0000 7fcb49d9b640 1 --2- 172.21.15.37:0/2167702039 >> v2:172.21.15.45:6812/2379918623 conn(0x7fcb34035ed0 0x7fcb34038390 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fcb5413a730 tx=0x7fcb3c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:11.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.625+0000 7fcb4b7fe640 1 -- 172.21.15.37:0/2167702039 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fcb44012400 con 0x7fcb54136da0 2024-04-26T00:35:11.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.625+0000 7fcb4b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:11.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.625+0000 7fcb4b7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:35:11.634 INFO:tasks.rados.rados.0.smithi045.stdout:2572: done (1 left) 2024-04-26T00:35:11.634 INFO:tasks.rados.rados.0.smithi045.stdout:2573: snap_remove snap 285 2024-04-26T00:35:11.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.857+0000 7fcb58917640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-26T00:35:11.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.857+0000 7fcb58917640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-26T00:35:11.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.857+0000 7fcb58917640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:11.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.857+0000 7fcb58917640 1 -- 172.21.15.37:0/2167702039 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fcb5409c250 con 0x7fcb54136da0 2024-04-26T00:35:11.856 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:11.857+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:11.856 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:11.857+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=12,kv=10,logm=50} crc {auth=2313935944,config=692619927,health=2114909644,kv=1034996044,logm=1264843049}) 2024-04-26T00:35:11.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.857+0000 7fcb4b7fe640 1 -- 172.21.15.37:0/2167702039 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fcb4401bb60 con 0x7fcb54136da0 2024-04-26T00:35:11.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.857+0000 7fcb4b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-26T00:35:11.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.857+0000 7fcb4b7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:35:11.857 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:11.857+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=28,config=2,health=11,kv=10,logm=49} crc {auth=1973083993,config=692619927,health=726458928,kv=1034996044,logm=1916304353}) 2024-04-26T00:35:11.857 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:11.857+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:11.857 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:11.857+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=12,kv=10,logm=50} crc {auth=2313935944,config=692619927,health=2114909644,kv=1034996044,logm=1264843049}) 2024-04-26T00:35:11.857 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:11.857+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=12,kv=10,logm=50} crc {auth=533718893,config=692619927,health=2114909644,kv=1034996044,logm=4048062781}) 2024-04-26T00:35:11.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.861+0000 7fcb58917640 1 -- 172.21.15.37:0/2167702039 >> v2:172.21.15.45:6812/2379918623 conn(0x7fcb34035ed0 msgr2=0x7fcb34038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:11.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.861+0000 7fcb58917640 1 --2- 172.21.15.37:0/2167702039 >> v2:172.21.15.45:6812/2379918623 conn(0x7fcb34035ed0 0x7fcb34038390 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fcb5413a730 tx=0x7fcb3c008040 comp rx=0 tx=0).stop 2024-04-26T00:35:11.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.861+0000 7fcb58917640 10 monclient: shutdown 2024-04-26T00:35:11.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.861+0000 7fcb58917640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:11.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.861+0000 7fcb58917640 1 -- 172.21.15.37:0/2167702039 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcb54136da0 msgr2=0x7fcb5411fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:11.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.861+0000 7fcb58917640 1 --2- 172.21.15.37:0/2167702039 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcb54136da0 0x7fcb5411fa50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fcb4400c470 tx=0x7fcb4400c930 comp rx=0 tx=0).stop 2024-04-26T00:35:11.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.861+0000 7fcb58917640 1 -- 172.21.15.37:0/2167702039 shutdown_connections 2024-04-26T00:35:11.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.861+0000 7fcb58917640 1 --2- 172.21.15.37:0/2167702039 >> v2:172.21.15.45:6812/2379918623 conn(0x7fcb34035ed0 0x7fcb34038390 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:11.859 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:11.861+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:11.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.861+0000 7fcb58917640 1 --2- 172.21.15.37:0/2167702039 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb5413c2d0 0x7fcb541cee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:11.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.861+0000 7fcb58917640 1 --2- 172.21.15.37:0/2167702039 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb54137180 0x7fcb54136050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:11.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.861+0000 7fcb58917640 1 --2- 172.21.15.37:0/2167702039 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcb54136da0 0x7fcb5411fa50 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:11.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.861+0000 7fcb58917640 1 -- 172.21.15.37:0/2167702039 >> 172.21.15.37:0/2167702039 conn(0x7fcb540a5f00 msgr2=0x7fcb5413ce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:11.860 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:11.861+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2812721374}) 2024-04-26T00:35:11.860 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:11.861+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4206636708}) 2024-04-26T00:35:11.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.861+0000 7fcb58917640 1 -- 172.21.15.37:0/2167702039 shutdown_connections 2024-04-26T00:35:11.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:11.861+0000 7fcb58917640 1 -- 172.21.15.37:0/2167702039 wait complete. 2024-04-26T00:35:11.862 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:11.861+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:11.862 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:11.861+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2812721374}) 2024-04-26T00:35:11.862 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:11.861+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2319768445}) 2024-04-26T00:35:11.862 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:11.865+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:11.862 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:11.865+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2156139226}) 2024-04-26T00:35:11.862 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:11.865+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1354899956}) 2024-04-26T00:35:11.862 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:11.865+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:11.862 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:11.865+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2156139226}) 2024-04-26T00:35:11.862 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:11.865+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=39800905}) 2024-04-26T00:35:11.877 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-26T00:35:11.937 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:11.941+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:11.937 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:11.941+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1492533786}) 2024-04-26T00:35:11.938 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:11.941+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2894714280}) 2024-04-26T00:35:11.938 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:11.941+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:11.938 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:11.941+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1492533786}) 2024-04-26T00:35:11.938 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:11.941+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3346395117}) 2024-04-26T00:35:11.943 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:11.945+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:11.943 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:11.945+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3655610530}) 2024-04-26T00:35:11.943 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:11.945+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1427579376}) 2024-04-26T00:35:11.943 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:11.945+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:11.943 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:11.945+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3655610530}) 2024-04-26T00:35:11.943 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:11.945+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4111986543}) 2024-04-26T00:35:12.025 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.025+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.025 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.025+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1158129565}) 2024-04-26T00:35:12.025 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.025+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2332561779}) 2024-04-26T00:35:12.025 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.025+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.025 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.025+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1158129565}) 2024-04-26T00:35:12.025 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.025+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2204870687}) 2024-04-26T00:35:12.635 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.637+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.635 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.637+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3137804039}) 2024-04-26T00:35:12.635 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.637+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=825914158}) 2024-04-26T00:35:12.635 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.637+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.636 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.637+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3137804039}) 2024-04-26T00:35:12.636 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.637+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3475549844}) 2024-04-26T00:35:12.646 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.649+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.646 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.649+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=630237994}) 2024-04-26T00:35:12.646 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.649+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3997974442}) 2024-04-26T00:35:12.646 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.649+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.647 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.649+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=630237994}) 2024-04-26T00:35:12.647 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.649+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=180547545}) 2024-04-26T00:35:12.654 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.657+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.654 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.657+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=779582813}) 2024-04-26T00:35:12.655 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.657+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1945162189}) 2024-04-26T00:35:12.655 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.657+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.655 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.657+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=779582813}) 2024-04-26T00:35:12.655 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.657+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3285191085}) 2024-04-26T00:35:12.658 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.661+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.658 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.661+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1192811972}) 2024-04-26T00:35:12.658 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.661+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=95549966}) 2024-04-26T00:35:12.658 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.661+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.658 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.661+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1192811972}) 2024-04-26T00:35:12.658 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.661+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4216045109}) 2024-04-26T00:35:12.659 INFO:tasks.rados.rados.0.smithi045.stdout:2573: done (1 left) 2024-04-26T00:35:12.659 INFO:tasks.rados.rados.0.smithi045.stdout:2574: delete oid 23 current snap is 287 2024-04-26T00:35:12.659 INFO:tasks.rados.rados.0.smithi045.stdout:2571: expect (ObjNum 866 snap 269 seq_num 866) 2024-04-26T00:35:12.661 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.665+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.661 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.665+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3574909307}) 2024-04-26T00:35:12.661 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.665+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4196802379}) 2024-04-26T00:35:12.662 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.665+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.662 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.665+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3574909307}) 2024-04-26T00:35:12.662 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.665+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=787706929}) 2024-04-26T00:35:12.665 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.665+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.665 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.665+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=933087748}) 2024-04-26T00:35:12.665 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.665+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2901569942}) 2024-04-26T00:35:12.665 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.665+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.665 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.665+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=933087748}) 2024-04-26T00:35:12.665 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.665+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=416777701}) 2024-04-26T00:35:12.668 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.669+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.668 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.669+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=701101825}) 2024-04-26T00:35:12.668 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.669+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1882669940}) 2024-04-26T00:35:12.669 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.669+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.669 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.669+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=701101825}) 2024-04-26T00:35:12.669 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.669+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2726733026}) 2024-04-26T00:35:12.672 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.673+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.672 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.673+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=143273794}) 2024-04-26T00:35:12.672 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.673+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2996197687}) 2024-04-26T00:35:12.672 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.673+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.672 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.673+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=143273794}) 2024-04-26T00:35:12.672 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.673+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1268371796}) 2024-04-26T00:35:12.677 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.677+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.677 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.677+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3136636021}) 2024-04-26T00:35:12.677 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.677+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3027336277}) 2024-04-26T00:35:12.677 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.677+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.677 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.677+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3136636021}) 2024-04-26T00:35:12.677 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.677+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1110991667}) 2024-04-26T00:35:12.681 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.685+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.681 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.685+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1151736974}) 2024-04-26T00:35:12.681 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.685+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3091509067}) 2024-04-26T00:35:12.681 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.685+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.681 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.685+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1151736974}) 2024-04-26T00:35:12.681 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.685+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2610041296}) 2024-04-26T00:35:12.687 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.689+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.687 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.689+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2407963768}) 2024-04-26T00:35:12.687 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.689+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=48568991}) 2024-04-26T00:35:12.687 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.689+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.687 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.689+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2407963768}) 2024-04-26T00:35:12.687 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.689+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3239151479}) 2024-04-26T00:35:12.689 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.693+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.689 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.693+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2225348004}) 2024-04-26T00:35:12.689 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.693+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3269055936}) 2024-04-26T00:35:12.689 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.693+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.689 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.693+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2225348004}) 2024-04-26T00:35:12.689 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.693+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2561506031}) 2024-04-26T00:35:12.692 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.693+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.692 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.693+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=570583213}) 2024-04-26T00:35:12.692 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.693+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1926809994}) 2024-04-26T00:35:12.692 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.693+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.692 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.693+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=570583213}) 2024-04-26T00:35:12.693 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.693+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1179462213}) 2024-04-26T00:35:12.695 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.697+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.695 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.697+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2801023394}) 2024-04-26T00:35:12.695 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.697+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=838475727}) 2024-04-26T00:35:12.695 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.697+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.695 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.697+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2801023394}) 2024-04-26T00:35:12.695 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.697+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2096323824}) 2024-04-26T00:35:12.697 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.701+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.697 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.701+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1011165371}) 2024-04-26T00:35:12.697 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.701+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3019240162}) 2024-04-26T00:35:12.697 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.701+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.697 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.701+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1011165371}) 2024-04-26T00:35:12.697 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.701+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2099085167}) 2024-04-26T00:35:12.699 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.701+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.699 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.701+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=357504048}) 2024-04-26T00:35:12.699 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.701+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1742842388}) 2024-04-26T00:35:12.700 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.701+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.700 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.701+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=357504048}) 2024-04-26T00:35:12.700 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.701+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=120092830}) 2024-04-26T00:35:12.702 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.705+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.702 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.705+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2751092807}) 2024-04-26T00:35:12.702 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.705+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=389312329}) 2024-04-26T00:35:12.702 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.705+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.702 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.705+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2751092807}) 2024-04-26T00:35:12.702 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.705+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4263753336}) 2024-04-26T00:35:12.704 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.705+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.704 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.705+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3511134284}) 2024-04-26T00:35:12.704 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.705+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1589151055}) 2024-04-26T00:35:12.704 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.705+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.704 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.705+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3511134284}) 2024-04-26T00:35:12.704 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.705+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1598599399}) 2024-04-26T00:35:12.706 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.709+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.706 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.709+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=264933515}) 2024-04-26T00:35:12.706 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.709+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4165920742}) 2024-04-26T00:35:12.706 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.709+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.706 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.709+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=264933515}) 2024-04-26T00:35:12.707 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.709+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2805344720}) 2024-04-26T00:35:12.708 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.709+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.708 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.709+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2159324308}) 2024-04-26T00:35:12.709 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.709+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3660293058}) 2024-04-26T00:35:12.709 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.709+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.709 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.709+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2159324308}) 2024-04-26T00:35:12.709 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.709+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2834373041}) 2024-04-26T00:35:12.711 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.713+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.711 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.713+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1744535304}) 2024-04-26T00:35:12.711 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.713+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=219538740}) 2024-04-26T00:35:12.711 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.713+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.711 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.713+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1744535304}) 2024-04-26T00:35:12.711 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.713+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1853287900}) 2024-04-26T00:35:12.713 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.717+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.713 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.717+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2597444936}) 2024-04-26T00:35:12.713 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.717+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2755426664}) 2024-04-26T00:35:12.713 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.717+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.713 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.717+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2597444936}) 2024-04-26T00:35:12.713 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.717+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2112766601}) 2024-04-26T00:35:12.715 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.717+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.715 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.717+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=396365712}) 2024-04-26T00:35:12.715 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.717+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1237287953}) 2024-04-26T00:35:12.715 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.717+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.716 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.717+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=396365712}) 2024-04-26T00:35:12.716 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.717+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1304015749}) 2024-04-26T00:35:12.718 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.721+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.718 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.721+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2639963012}) 2024-04-26T00:35:12.718 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.721+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1155708017}) 2024-04-26T00:35:12.718 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.721+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.718 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.721+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2639963012}) 2024-04-26T00:35:12.718 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.721+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2662533439}) 2024-04-26T00:35:12.720 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.721+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.720 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.721+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3845860384}) 2024-04-26T00:35:12.720 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.721+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2488481197}) 2024-04-26T00:35:12.720 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.721+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.720 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.721+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3845860384}) 2024-04-26T00:35:12.720 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.721+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1983175915}) 2024-04-26T00:35:12.722 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.725+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.722 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.725+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3715265523}) 2024-04-26T00:35:12.722 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.725+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4183709056}) 2024-04-26T00:35:12.722 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.725+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.722 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.725+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3715265523}) 2024-04-26T00:35:12.722 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.725+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2741240939}) 2024-04-26T00:35:12.724 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.725+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.724 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.725+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1706915224}) 2024-04-26T00:35:12.724 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.725+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1156829059}) 2024-04-26T00:35:12.725 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.725+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.725 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.725+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1706915224}) 2024-04-26T00:35:12.725 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.725+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=409794037}) 2024-04-26T00:35:12.726 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.729+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.727 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.729+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2600718819}) 2024-04-26T00:35:12.727 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.729+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3584500522}) 2024-04-26T00:35:12.727 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.729+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.727 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.729+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2600718819}) 2024-04-26T00:35:12.727 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.729+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2307740683}) 2024-04-26T00:35:12.729 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.733+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.729 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.733+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1280451629}) 2024-04-26T00:35:12.729 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.733+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=109532359}) 2024-04-26T00:35:12.729 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.733+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.729 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.733+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1280451629}) 2024-04-26T00:35:12.729 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.733+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2997501745}) 2024-04-26T00:35:12.731 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.733+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.731 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.733+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=390471259}) 2024-04-26T00:35:12.731 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.733+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=445679731}) 2024-04-26T00:35:12.731 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.733+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.731 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.733+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=390471259}) 2024-04-26T00:35:12.731 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.733+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4159784075}) 2024-04-26T00:35:12.733 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.737+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.733 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.737+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3857950583}) 2024-04-26T00:35:12.733 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.737+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2494968520}) 2024-04-26T00:35:12.733 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.737+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.733 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.737+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3857950583}) 2024-04-26T00:35:12.733 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.737+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2840543567}) 2024-04-26T00:35:12.735 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.737+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.735 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.737+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=268110903}) 2024-04-26T00:35:12.735 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.737+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=640233531}) 2024-04-26T00:35:12.735 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.737+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.735 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.737+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=268110903}) 2024-04-26T00:35:12.735 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.737+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2154148914}) 2024-04-26T00:35:12.737 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.741+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.737 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.741+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1000066573}) 2024-04-26T00:35:12.738 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.741+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1903393069}) 2024-04-26T00:35:12.738 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.741+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.738 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.741+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1000066573}) 2024-04-26T00:35:12.738 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.741+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=355477271}) 2024-04-26T00:35:12.739 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.741+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.740 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.741+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=454056794}) 2024-04-26T00:35:12.740 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.741+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1108097511}) 2024-04-26T00:35:12.740 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.741+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.740 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.741+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=454056794}) 2024-04-26T00:35:12.740 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.741+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=169381443}) 2024-04-26T00:35:12.742 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.745+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.742 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.745+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4153363325}) 2024-04-26T00:35:12.742 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.745+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1627958292}) 2024-04-26T00:35:12.742 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.745+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.742 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.745+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4153363325}) 2024-04-26T00:35:12.742 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.745+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1840859280}) 2024-04-26T00:35:12.744 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.745+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.744 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.745+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3263713651}) 2024-04-26T00:35:12.744 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.745+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4168970954}) 2024-04-26T00:35:12.744 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.745+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.744 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.745+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3263713651}) 2024-04-26T00:35:12.744 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.745+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3661894508}) 2024-04-26T00:35:12.746 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.749+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.746 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.749+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1505701506}) 2024-04-26T00:35:12.746 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.749+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=673201931}) 2024-04-26T00:35:12.746 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.749+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.746 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.749+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1505701506}) 2024-04-26T00:35:12.746 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.749+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=632582897}) 2024-04-26T00:35:12.748 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.749+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.748 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.749+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2124760311}) 2024-04-26T00:35:12.748 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.749+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=272443400}) 2024-04-26T00:35:12.748 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.749+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.748 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.749+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2124760311}) 2024-04-26T00:35:12.748 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.749+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=598266942}) 2024-04-26T00:35:12.750 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.753+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.750 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.753+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4060339883}) 2024-04-26T00:35:12.750 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.753+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2655999598}) 2024-04-26T00:35:12.750 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.753+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.750 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.753+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4060339883}) 2024-04-26T00:35:12.750 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.753+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=939803914}) 2024-04-26T00:35:12.752 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.753+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.752 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.753+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=316479332}) 2024-04-26T00:35:12.752 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.753+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1772048083}) 2024-04-26T00:35:12.752 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.753+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.752 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.753+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=316479332}) 2024-04-26T00:35:12.752 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.753+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1295447389}) 2024-04-26T00:35:12.754 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.757+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.754 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.757+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=666637054}) 2024-04-26T00:35:12.754 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.757+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=247879359}) 2024-04-26T00:35:12.754 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.757+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.755 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.757+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=666637054}) 2024-04-26T00:35:12.755 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.757+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3733863303}) 2024-04-26T00:35:12.756 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.757+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.756 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.757+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3857410797}) 2024-04-26T00:35:12.756 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.757+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4220240665}) 2024-04-26T00:35:12.756 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.757+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.757 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.757+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3857410797}) 2024-04-26T00:35:12.757 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.757+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3288880532}) 2024-04-26T00:35:12.759 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.761+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.759 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.761+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3580054967}) 2024-04-26T00:35:12.759 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.761+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1191434606}) 2024-04-26T00:35:12.759 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.761+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.759 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.761+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3580054967}) 2024-04-26T00:35:12.759 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.761+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4074990023}) 2024-04-26T00:35:12.761 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.765+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.761 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.765+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2398490532}) 2024-04-26T00:35:12.761 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.765+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=283061501}) 2024-04-26T00:35:12.761 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.765+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.761 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.765+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2398490532}) 2024-04-26T00:35:12.761 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.765+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3242071783}) 2024-04-26T00:35:12.763 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.765+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.763 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.765+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1931076340}) 2024-04-26T00:35:12.763 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.765+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=599526906}) 2024-04-26T00:35:12.763 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.765+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.763 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.765+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1931076340}) 2024-04-26T00:35:12.763 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.765+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1072276288}) 2024-04-26T00:35:12.765 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.769+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.765 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.769+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2612154949}) 2024-04-26T00:35:12.765 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.769+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1581884642}) 2024-04-26T00:35:12.766 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.769+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.766 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.769+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2612154949}) 2024-04-26T00:35:12.766 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.769+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1836223641}) 2024-04-26T00:35:12.767 INFO:tasks.rados.rados.0.smithi045.stdout:2571: done (1 left) 2024-04-26T00:35:12.767 INFO:tasks.rados.rados.0.smithi045.stdout:2574: done (0 left) 2024-04-26T00:35:12.767 INFO:tasks.rados.rados.0.smithi045.stdout:2575: snap_create 2024-04-26T00:35:12.768 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.769+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.768 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.769+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=88,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=1} crc {logm=1066897652,mdsmap=2840952586,mgr=3303164479,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=4030192881}) 2024-04-26T00:35:12.768 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.769+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=86,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=3} crc {logm=1412140751,mdsmap=1401145748,mgr=1214999089,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=3357681507}) 2024-04-26T00:35:12.768 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.769+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.768 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.769+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=88,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=1} crc {logm=1066897652,mdsmap=2840952586,mgr=3303164479,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=4030192881}) 2024-04-26T00:35:12.769 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.769+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=85,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=2} crc {logm=345089585,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=3770728276}) 2024-04-26T00:35:12.770 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.773+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.771 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.773+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=9,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=71} crc {mgrstat=3680103511,mon_config_key=3794053260,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=3355350469,osd_snap=4025940532}) 2024-04-26T00:35:12.771 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.773+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=9,mon_config_key=10,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=73} crc {mgrstat=3680103511,mon_config_key=752439442,monmap=3876866194,osd_metadata=206931372,osd_pg_creating=3355350469,osd_snap=7041311}) 2024-04-26T00:35:12.771 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.773+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.771 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.773+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=9,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=71} crc {mgrstat=3680103511,mon_config_key=3794053260,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=3355350469,osd_snap=4025940532}) 2024-04-26T00:35:12.771 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.773+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=9,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=70} crc {mgrstat=3680103511,mon_config_key=4061063117,monmap=3876866194,osd_metadata=3306121364,osd_pg_creating=3355350469,osd_snap=2990431404}) 2024-04-26T00:35:12.773 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.773+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.773 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.773+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=35,osdmap=65} crc {osd_snap=447710144,osdmap=142415243}) 2024-04-26T00:35:12.773 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.773+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=37,osdmap=63} crc {osd_snap=1367835090,osdmap=2976738798}) 2024-04-26T00:35:12.773 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.773+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.773 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.773+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=35,osdmap=65} crc {osd_snap=447710144,osdmap=142415243}) 2024-04-26T00:35:12.773 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.773+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=35,osdmap=65} crc {osd_snap=2085339658,osdmap=431090818}) 2024-04-26T00:35:12.775 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.777+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.775 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.777+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1819336785}) 2024-04-26T00:35:12.775 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.777+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2915713116}) 2024-04-26T00:35:12.775 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.777+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.775 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.777+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1819336785}) 2024-04-26T00:35:12.776 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.777+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4142932092}) 2024-04-26T00:35:12.778 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.781+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.778 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.781+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=913287231}) 2024-04-26T00:35:12.779 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.781+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1949783186}) 2024-04-26T00:35:12.779 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.781+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.779 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.781+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=913287231}) 2024-04-26T00:35:12.779 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.781+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2749997056}) 2024-04-26T00:35:12.790 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.793+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.790 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.793+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=77} crc {osdmap=3393597937}) 2024-04-26T00:35:12.790 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.793+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=79} crc {osdmap=3876186660}) 2024-04-26T00:35:12.790 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.793+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:12.790 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.793+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=77} crc {osdmap=3393597937}) 2024-04-26T00:35:12.790 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:12.793+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=78} crc {osdmap=2865119261}) 2024-04-26T00:35:12.878 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:35:12.879 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:35:13.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a3a4ca640 10 monclient: get_monmap_and_config 2024-04-26T00:35:13.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a3a4ca640 10 monclient: build_initial_monmap 2024-04-26T00:35:13.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a3a4ca640 10 monclient: monmap: 2024-04-26T00:35:13.011 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:13.011 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:13.011 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:35:13.017493+0000 2024-04-26T00:35:13.011 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:35:13.017493+0000 2024-04-26T00:35:13.011 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:13.011 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:13.011 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:35:13.011 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:35:13.012 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:35:13.012 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:35:13.012 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:35:13.012 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:13.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a3a4ca640 1 Processor -- start 2024-04-26T00:35:13.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a3a4ca640 1 -- start start 2024-04-26T00:35:13.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a3a4ca640 10 monclient: init 2024-04-26T00:35:13.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a3a4ca640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:13.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a3a4ca640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-04-26T00:35:13.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a3a4ca640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6a340a0260 addr v1:172.21.15.45:6789/0 2024-04-26T00:35:13.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a3a4ca640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a34137e40 0x7f6a3413a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:13.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a3a4ca640 10 monclient(hunting): picked mon.noname-b con 0x7f6a34137e40 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:13.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a3a4ca640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6a3413a770 0x7f6a3413cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:13.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a3a4ca640 10 monclient(hunting): picked mon.noname-a con 0x7f6a3413a770 addr v2:172.21.15.37:3300/0 2024-04-26T00:35:13.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a3a4ca640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a340a1130 con 0x7f6a340a0260 2024-04-26T00:35:13.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a3a4ca640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:13.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a3a4ca640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f6a340a0d50 con 0x7f6a3413a770 2024-04-26T00:35:13.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a3a4ca640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:13.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a3a4ca640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f6a3409ec10 con 0x7f6a34137e40 2024-04-26T00:35:13.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a3a4ca640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:13.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a3a4ca640 10 monclient(hunting): authenticate will time out at 1894.323205s 2024-04-26T00:35:13.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a337fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a34137e40 0x7f6a3413a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:13.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a38a67640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6a3413a770 0x7f6a3413cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:13.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a337fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a34137e40 0x7f6a3413a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:60332/0 (socket says 172.21.15.37:60332) 2024-04-26T00:35:13.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a337fe640 1 -- 172.21.15.37:0/3723848542 learned_addr learned my addr 172.21.15.37:0/3723848542 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:13.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a337fe640 10 monclient(hunting): get_auth_request con 0x7f6a34137e40 auth_method 0 2024-04-26T00:35:13.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:13.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a337fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:13.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a337fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:13.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a38a67640 10 monclient(hunting): get_auth_request con 0x7f6a3413a770 auth_method 0 2024-04-26T00:35:13.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a38a67640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:13.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a38a67640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:13.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a38a67640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:13.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:13.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:13.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:13.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a38a67640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:13.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a38a67640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:13.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a38a67640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:13.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a38a67640 10 monclient(hunting): handle_auth_done global_id 144519 payload 274 2024-04-26T00:35:13.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a38a67640 1 -- 172.21.15.37:0/3723848542 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a34137e40 msgr2=0x7f6a3413a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:13.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a38a67640 1 --2- 172.21.15.37:0/3723848542 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a34137e40 0x7f6a3413a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:13.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a38a67640 1 -- 172.21.15.37:0/3723848542 >> v1:172.21.15.45:6789/0 conn(0x7f6a340a0260 legacy=0x7f6a340a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:13.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a38a67640 10 monclient: _finish_hunting 0 2024-04-26T00:35:13.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a38a67640 1 monclient: found mon.noname-a 2024-04-26T00:35:13.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a38a67640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:13.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a38a67640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:35:13.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a38a67640 1 -- 172.21.15.37:0/3723848542 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a3406cff0 con 0x7f6a3413a770 2024-04-26T00:35:13.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.013+0000 7f6a337fe640 1 --2- 172.21.15.37:0/3723848542 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a34137e40 0x7f6a3413a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:35:13.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.017+0000 7f6a38a67640 1 -- 172.21.15.37:0/3723848542 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a3413a770 msgr2=0x7f6a3413cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:13.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.017+0000 7f6a38a67640 1 --2- 172.21.15.37:0/3723848542 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a3413a770 0x7f6a3413cb60 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6a240097c0 tx=0x7f6a2401c140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=663925eecaefb38a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:13.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a32ffd640 1 -- 172.21.15.37:0/3723848542 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a240b1070 con 0x7f6a3413a770 2024-04-26T00:35:13.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:13.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a32ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:35:13.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a32ffd640 10 monclient: dump: 2024-04-26T00:35:13.021 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:13.021 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:13.021 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.021 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.021 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:13.021 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:13.021 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:13.021 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:13.021 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:13.021 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:13.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a32ffd640 10 monclient: _finish_auth 0 2024-04-26T00:35:13.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a32ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:35:13.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:13.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a32ffd640 1 -- 172.21.15.37:0/3723848542 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a240ac440 con 0x7f6a3413a770 2024-04-26T00:35:13.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a32ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:13.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a32ffd640 1 -- 172.21.15.37:0/3723848542 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a240ac810 con 0x7f6a3413a770 2024-04-26T00:35:13.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a3a4ca640 5 monclient: authenticate success, global_id 144519 2024-04-26T00:35:13.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a3a4ca640 10 monclient: get_monmap_and_config success 2024-04-26T00:35:13.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a3a4ca640 4 set_mon_vals no callback set 2024-04-26T00:35:13.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:13.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a32ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:13.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a32ffd640 10 monclient: dump: 2024-04-26T00:35:13.022 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:13.023 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:13.023 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.023 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.023 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:13.023 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:13.023 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:13.023 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:13.023 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:13.023 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:13.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a3a4ca640 10 monclient: shutdown 2024-04-26T00:35:13.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a3a4ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:13.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a3a4ca640 1 -- 172.21.15.37:0/3723848542 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a3413a770 msgr2=0x7f6a3413cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:13.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a3a4ca640 1 --2- 172.21.15.37:0/3723848542 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a3413a770 0x7f6a3413cb60 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6a240097c0 tx=0x7f6a2401c140 comp rx=0 tx=0).stop 2024-04-26T00:35:13.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a3a4ca640 1 -- 172.21.15.37:0/3723848542 shutdown_connections 2024-04-26T00:35:13.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a3a4ca640 1 --2- 172.21.15.37:0/3723848542 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a3413a770 0x7f6a3413cb60 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:13.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a3a4ca640 1 --2- 172.21.15.37:0/3723848542 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a34137e40 0x7f6a3413a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:13.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a3a4ca640 1 -- 172.21.15.37:0/3723848542 >> 172.21.15.37:0/3723848542 conn(0x7f6a340a1ad0 msgr2=0x7f6a340a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:13.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a3a4ca640 1 -- 172.21.15.37:0/3723848542 shutdown_connections 2024-04-26T00:35:13.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a3a4ca640 1 -- 172.21.15.37:0/3723848542 wait complete. 2024-04-26T00:35:13.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a3a4ca640 10 monclient: build_initial_monmap 2024-04-26T00:35:13.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a3a4ca640 10 monclient: monmap: 2024-04-26T00:35:13.025 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:13.025 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:35:13.025 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:35:13.025 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:35:13.025 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:13.025 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:35:13.025 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:35:13.025 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:35:13.025 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:35:13.025 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:13.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a3a4ca640 1 Processor -- start 2024-04-26T00:35:13.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a3a4ca640 1 -- start start 2024-04-26T00:35:13.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a3a4ca640 10 monclient: init 2024-04-26T00:35:13.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a3a4ca640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:13.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a3a4ca640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:35:13.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a3a4ca640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a340a0260 0x7f6a341ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:13.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a3a4ca640 10 monclient(hunting): picked mon.noname-a con 0x7f6a340a0260 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:35:13.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a3a4ca640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a34137e40 0x7f6a34136d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:13.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a3a4ca640 10 monclient(hunting): picked mon.noname-c con 0x7f6a34137e40 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:13.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a3a4ca640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a3413a770 0x7f6a3411d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:13.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a3a4ca640 10 monclient(hunting): picked mon.noname-b con 0x7f6a3413a770 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:35:13.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a3a4ca640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:13.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a3a4ca640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f6a3409f4c0 con 0x7f6a340a0260 2024-04-26T00:35:13.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a3a4ca640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:13.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a3a4ca640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6a3409f0f0 con 0x7f6a3413a770 2024-04-26T00:35:13.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a3a4ca640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:13.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a3a4ca640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f6a3413f8c0 con 0x7f6a34137e40 2024-04-26T00:35:13.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a3a4ca640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:13.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a3a4ca640 10 monclient(hunting): authenticate will time out at 1894.330327s 2024-04-26T00:35:13.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a33fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a340a0260 0x7f6a341ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:13.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a337fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a34137e40 0x7f6a34136d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:13.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a33fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a340a0260 0x7f6a341ccbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:34424/0 (socket says 172.21.15.37:34424) 2024-04-26T00:35:13.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a33fff640 1 -- 172.21.15.37:0/202244576 learned_addr learned my addr 172.21.15.37:0/202244576 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:13.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a33fff640 10 monclient(hunting): get_auth_request con 0x7f6a340a0260 auth_method 0 2024-04-26T00:35:13.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:13.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a33fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:13.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a33fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:13.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a337fe640 1 --2- 172.21.15.37:0/202244576 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a34137e40 0x7f6a34136d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:60344/0 (socket says 172.21.15.37:60344) 2024-04-26T00:35:13.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a337fe640 10 monclient(hunting): get_auth_request con 0x7f6a34137e40 auth_method 0 2024-04-26T00:35:13.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:13.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a337fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:13.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a337fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:13.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:13.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:13.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:13.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:13.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:13.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:13.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a38a67640 1 --2- 172.21.15.37:0/202244576 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a3413a770 0x7f6a3411d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:13.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a38a67640 10 monclient(hunting): get_auth_request con 0x7f6a3413a770 auth_method 0 2024-04-26T00:35:13.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a38a67640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:13.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a38a67640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:13.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a38a67640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:13.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a337fe640 10 monclient(hunting): handle_auth_done global_id 184121 payload 824 2024-04-26T00:35:13.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a337fe640 1 -- 172.21.15.37:0/202244576 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a3413a770 msgr2=0x7f6a3411d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:13.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a337fe640 1 --2- 172.21.15.37:0/202244576 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a3413a770 0x7f6a3411d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:13.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a337fe640 1 -- 172.21.15.37:0/202244576 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a340a0260 msgr2=0x7f6a341ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:13.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a337fe640 1 --2- 172.21.15.37:0/202244576 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a340a0260 0x7f6a341ccbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:13.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a337fe640 10 monclient: _finish_hunting 0 2024-04-26T00:35:13.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a337fe640 1 monclient: found mon.noname-c 2024-04-26T00:35:13.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:13.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a337fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:35:13.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a337fe640 1 -- 172.21.15.37:0/202244576 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a34120e30 con 0x7f6a34137e40 2024-04-26T00:35:13.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.021+0000 7f6a33fff640 1 --2- 172.21.15.37:0/202244576 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a340a0260 0x7f6a341ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:35:13.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.025+0000 7f6a337fe640 1 -- 172.21.15.37:0/202244576 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a34137e40 msgr2=0x7f6a34136d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:13.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.025+0000 7f6a337fe640 1 --2- 172.21.15.37:0/202244576 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a34137e40 0x7f6a34136d40 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c00e910 tx=0x7f6a1c00edd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:13.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.025+0000 7f6a30ff9640 1 -- 172.21.15.37:0/202244576 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a1c00cc10 con 0x7f6a34137e40 2024-04-26T00:35:13.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.025+0000 7f6a30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:13.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.025+0000 7f6a30ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:35:13.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.025+0000 7f6a30ff9640 10 monclient: dump: 2024-04-26T00:35:13.032 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:13.032 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:13.032 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.032 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.032 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:13.032 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:13.032 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:13.032 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:13.032 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:13.032 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:13.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.025+0000 7f6a30ff9640 10 monclient: _finish_auth 0 2024-04-26T00:35:13.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.025+0000 7f6a30ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:35:13.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.025+0000 7f6a30ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:13.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.025+0000 7f6a30ff9640 1 -- 172.21.15.37:0/202244576 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a1c00cdb0 con 0x7f6a34137e40 2024-04-26T00:35:13.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.025+0000 7f6a3a4ca640 5 monclient: authenticate success, global_id 184121 2024-04-26T00:35:13.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.025+0000 7f6a3a4ca640 10 monclient: _renew_subs 2024-04-26T00:35:13.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.025+0000 7f6a3a4ca640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:35:13.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.025+0000 7f6a3a4ca640 1 -- 172.21.15.37:0/202244576 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a341383b0 con 0x7f6a34137e40 2024-04-26T00:35:13.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.025+0000 7f6a30ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:13.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.025+0000 7f6a30ff9640 1 -- 172.21.15.37:0/202244576 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a1c004250 con 0x7f6a34137e40 2024-04-26T00:35:13.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.025+0000 7f6a30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:13.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.025+0000 7f6a30ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:35:13.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.025+0000 7f6a30ff9640 10 monclient: dump: 2024-04-26T00:35:13.033 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:13.034 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:13.034 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.034 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.034 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:13.034 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:13.034 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:13.034 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:13.034 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:13.034 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:13.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.025+0000 7f6a3a4ca640 10 monclient: _renew_subs 2024-04-26T00:35:13.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.025+0000 7f6a327fc640 4 set_mon_vals no callback set 2024-04-26T00:35:13.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.025+0000 7f6a3a4ca640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:35:13.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.025+0000 7f6a3a4ca640 1 -- 172.21.15.37:0/202244576 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a341cda50 con 0x7f6a34137e40 2024-04-26T00:35:13.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.025+0000 7f6a3a4ca640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:13.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.025+0000 7f6a3a4ca640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:13.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.025+0000 7f6a3a4ca640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:35:13.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.025+0000 7f6a30ff9640 1 -- 172.21.15.37:0/202244576 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f6a1c003620 con 0x7f6a34137e40 2024-04-26T00:35:13.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.025+0000 7f6a3a4ca640 1 -- 172.21.15.37:0/202244576 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a34066f40 con 0x7f6a34137e40 2024-04-26T00:35:13.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.025+0000 7f6a30ff9640 1 --2- 172.21.15.37:0/202244576 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6a14035ed0 0x7f6a14038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:13.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.025+0000 7f6a30ff9640 1 -- 172.21.15.37:0/202244576 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(598..598 src has 370..598) ==== 3927+0+0 (secure 0 0 0) 0x7f6a1c048a90 con 0x7f6a34137e40 2024-04-26T00:35:13.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.025+0000 7f6a33fff640 1 --2- 172.21.15.37:0/202244576 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6a14035ed0 0x7f6a14038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:13.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.025+0000 7f6a33fff640 10 monclient: get_auth_request con 0x7f6a14035ed0 auth_method 0 2024-04-26T00:35:13.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.029+0000 7f6a33fff640 1 --2- 172.21.15.37:0/202244576 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6a14035ed0 0x7f6a14038390 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c00f3c0 tx=0x7f6a2c003e70 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:13.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.029+0000 7f6a30ff9640 1 -- 172.21.15.37:0/202244576 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6a34120e30 con 0x7f6a34137e40 2024-04-26T00:35:13.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.029+0000 7f6a30ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:13.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.029+0000 7f6a30ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:35:13.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.261+0000 7f6a3a4ca640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:35:13.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.261+0000 7f6a3a4ca640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:35:13.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.261+0000 7f6a3a4ca640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:35:13.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.261+0000 7f6a3a4ca640 1 -- 172.21.15.37:0/202244576 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6a3406b9a0 con 0x7f6a34137e40 2024-04-26T00:35:13.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.261+0000 7f6a30ff9640 1 -- 172.21.15.37:0/202244576 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f6a1c017020 con 0x7f6a34137e40 2024-04-26T00:35:13.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.261+0000 7f6a30ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:35:13.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.261+0000 7f6a30ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:35:13.259 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":114,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:35:13.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.265+0000 7f6a3a4ca640 1 -- 172.21.15.37:0/202244576 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6a14035ed0 msgr2=0x7f6a14038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:13.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.265+0000 7f6a3a4ca640 1 --2- 172.21.15.37:0/202244576 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6a14035ed0 0x7f6a14038390 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c00f3c0 tx=0x7f6a2c003e70 comp rx=0 tx=0).stop 2024-04-26T00:35:13.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.265+0000 7f6a3a4ca640 10 monclient: shutdown 2024-04-26T00:35:13.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.265+0000 7f6a3a4ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:13.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.265+0000 7f6a3a4ca640 1 -- 172.21.15.37:0/202244576 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a34137e40 msgr2=0x7f6a34136d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:13.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.265+0000 7f6a3a4ca640 1 --2- 172.21.15.37:0/202244576 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a34137e40 0x7f6a34136d40 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c00e910 tx=0x7f6a1c00edd0 comp rx=0 tx=0).stop 2024-04-26T00:35:13.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.265+0000 7f6a3a4ca640 1 -- 172.21.15.37:0/202244576 shutdown_connections 2024-04-26T00:35:13.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.265+0000 7f6a3a4ca640 1 --2- 172.21.15.37:0/202244576 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6a14035ed0 0x7f6a14038390 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:13.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.265+0000 7f6a3a4ca640 1 --2- 172.21.15.37:0/202244576 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a3413a770 0x7f6a3411d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:13.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.265+0000 7f6a3a4ca640 1 --2- 172.21.15.37:0/202244576 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a34137e40 0x7f6a34136d40 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:13.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.265+0000 7f6a3a4ca640 1 --2- 172.21.15.37:0/202244576 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a340a0260 0x7f6a341ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:13.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.265+0000 7f6a3a4ca640 1 -- 172.21.15.37:0/202244576 >> 172.21.15.37:0/202244576 conn(0x7f6a340a1ad0 msgr2=0x7f6a34138750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:13.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.265+0000 7f6a3a4ca640 1 -- 172.21.15.37:0/202244576 shutdown_connections 2024-04-26T00:35:13.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.265+0000 7f6a3a4ca640 1 -- 172.21.15.37:0/202244576 wait complete. 2024-04-26T00:35:13.279 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:35:13.279 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-26T00:35:13.279 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:35:13.279 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:35:13.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f5510164640 10 monclient: get_monmap_and_config 2024-04-26T00:35:13.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f5510164640 10 monclient: build_initial_monmap 2024-04-26T00:35:13.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f5510164640 10 monclient: monmap: 2024-04-26T00:35:13.399 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:13.399 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:13.399 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:35:13.405466+0000 2024-04-26T00:35:13.399 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:35:13.405466+0000 2024-04-26T00:35:13.399 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:13.400 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:13.400 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:35:13.400 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:35:13.400 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:35:13.400 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:35:13.400 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:35:13.400 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:13.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f5510164640 1 Processor -- start 2024-04-26T00:35:13.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f5510164640 1 -- start start 2024-04-26T00:35:13.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f5510164640 10 monclient: init 2024-04-26T00:35:13.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f5510164640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:13.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f5510164640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-04-26T00:35:13.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f5510164640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f55080a3d90 0x7f550809bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:13.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f5510164640 10 monclient(hunting): picked mon.noname-b con 0x7f55080a3d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:13.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f5510164640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f55080a4170 0x7f55080a45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:13.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f5510164640 10 monclient(hunting): picked mon.noname-c con 0x7f55080a4170 addr v2:172.21.15.45:3300/0 2024-04-26T00:35:13.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f5510164640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5508133b20 addr v1:172.21.15.37:6789/0 2024-04-26T00:35:13.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f5510164640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55080a5150 con 0x7f5508133b20 2024-04-26T00:35:13.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f5510164640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:13.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f5510164640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f550809b280 con 0x7f55080a4170 2024-04-26T00:35:13.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f5510164640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:13.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f5510164640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f550809b560 con 0x7f55080a3d90 2024-04-26T00:35:13.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f5510164640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:13.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f5510164640 10 monclient(hunting): authenticate will time out at 1894.711392s 2024-04-26T00:35:13.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f550df00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f55080a3d90 0x7f550809bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:13.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f550df00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f55080a3d90 0x7f550809bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:60348/0 (socket says 172.21.15.37:60348) 2024-04-26T00:35:13.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f550df00640 1 -- 172.21.15.37:0/1162848491 learned_addr learned my addr 172.21.15.37:0/1162848491 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:13.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f550df00640 10 monclient(hunting): get_auth_request con 0x7f55080a3d90 auth_method 0 2024-04-26T00:35:13.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f550df00640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:13.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f550df00640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:13.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f550df00640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:13.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f550e701640 1 --1- 172.21.15.37:0/1162848491 >> v1:172.21.15.37:6789/0 conn(0x7f5508133b20 0x7f5508133f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:36038/0 (socket says 172.21.15.37:36038) 2024-04-26T00:35:13.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f550df00640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:13.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f550df00640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:13.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f550df00640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:13.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f550d6ff640 1 --2- 172.21.15.37:0/1162848491 >> v2:172.21.15.45:3300/0 conn(0x7f55080a4170 0x7f55080a45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:13.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f550d6ff640 10 monclient(hunting): get_auth_request con 0x7f55080a4170 auth_method 0 2024-04-26T00:35:13.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f550d6ff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:13.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f550d6ff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:13.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f550d6ff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:13.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f550df00640 10 monclient(hunting): handle_auth_done global_id 184124 payload 274 2024-04-26T00:35:13.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f550df00640 1 -- 172.21.15.37:0/1162848491 >> v2:172.21.15.45:3300/0 conn(0x7f55080a4170 msgr2=0x7f55080a45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:13.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f550df00640 1 --2- 172.21.15.37:0/1162848491 >> v2:172.21.15.45:3300/0 conn(0x7f55080a4170 0x7f55080a45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:13.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f550df00640 1 -- 172.21.15.37:0/1162848491 >> v1:172.21.15.37:6789/0 conn(0x7f5508133b20 legacy=0x7f5508133f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:13.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f550df00640 10 monclient: _finish_hunting 0 2024-04-26T00:35:13.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f550df00640 1 monclient: found mon.noname-b 2024-04-26T00:35:13.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f550df00640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:13.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f550df00640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:35:13.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.401+0000 7f550df00640 1 -- 172.21.15.37:0/1162848491 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55080a4af0 con 0x7f55080a3d90 2024-04-26T00:35:13.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f550df00640 1 -- 172.21.15.37:0/1162848491 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f55080a3d90 msgr2=0x7f550809bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:13.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f550df00640 1 --2- 172.21.15.37:0/1162848491 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f55080a3d90 0x7f550809bf40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f54f0009e50 tx=0x7f54f001c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4dff27981d39e0e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:13.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f550cefe640 1 -- 172.21.15.37:0/1162848491 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54f00a5640 con 0x7f55080a3d90 2024-04-26T00:35:13.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f550cefe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:13.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f550cefe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:35:13.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f550cefe640 10 monclient: dump: 2024-04-26T00:35:13.407 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:13.407 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:13.407 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.407 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.407 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:13.407 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:13.407 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:13.408 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:13.408 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:13.408 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:13.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f550cefe640 10 monclient: _finish_auth 0 2024-04-26T00:35:13.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f550cefe640 10 monclient: _check_auth_tickets 2024-04-26T00:35:13.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f550cefe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:13.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f550cefe640 1 -- 172.21.15.37:0/1162848491 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54f00a57e0 con 0x7f55080a3d90 2024-04-26T00:35:13.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f5510164640 5 monclient: authenticate success, global_id 184124 2024-04-26T00:35:13.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f550cefe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:13.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f5510164640 10 monclient: get_monmap_and_config success 2024-04-26T00:35:13.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f550cefe640 1 -- 172.21.15.37:0/1162848491 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54f00a5bb0 con 0x7f55080a3d90 2024-04-26T00:35:13.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f5510164640 4 set_mon_vals no callback set 2024-04-26T00:35:13.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f550cefe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:13.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f550cefe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:35:13.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f550cefe640 10 monclient: dump: 2024-04-26T00:35:13.409 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:13.409 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:13.409 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.409 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.409 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:13.409 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:13.409 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:13.409 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:13.409 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:13.409 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:13.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f5510164640 10 monclient: shutdown 2024-04-26T00:35:13.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f5510164640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:13.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f5510164640 1 -- 172.21.15.37:0/1162848491 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f55080a3d90 msgr2=0x7f550809bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:13.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f5510164640 1 --2- 172.21.15.37:0/1162848491 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f55080a3d90 0x7f550809bf40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f54f0009e50 tx=0x7f54f001c8b0 comp rx=0 tx=0).stop 2024-04-26T00:35:13.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f5510164640 1 -- 172.21.15.37:0/1162848491 shutdown_connections 2024-04-26T00:35:13.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f5510164640 1 --2- 172.21.15.37:0/1162848491 >> v2:172.21.15.45:3300/0 conn(0x7f55080a4170 0x7f55080a45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:13.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f5510164640 1 --2- 172.21.15.37:0/1162848491 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f55080a3d90 0x7f550809bf40 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:13.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f5510164640 1 -- 172.21.15.37:0/1162848491 >> 172.21.15.37:0/1162848491 conn(0x7f55080aa0a0 msgr2=0x7f55080a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:13.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f5510164640 1 -- 172.21.15.37:0/1162848491 shutdown_connections 2024-04-26T00:35:13.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f5510164640 1 -- 172.21.15.37:0/1162848491 wait complete. 2024-04-26T00:35:13.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f5510164640 10 monclient: build_initial_monmap 2024-04-26T00:35:13.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f5510164640 10 monclient: monmap: 2024-04-26T00:35:13.417 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:13.417 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:35:13.417 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:35:13.417 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:35:13.417 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:13.417 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:35:13.417 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:35:13.417 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:35:13.417 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:35:13.417 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:13.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f5510164640 1 Processor -- start 2024-04-26T00:35:13.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f5510164640 1 -- start start 2024-04-26T00:35:13.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f5510164640 10 monclient: init 2024-04-26T00:35:13.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f5510164640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:13.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f5510164640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:35:13.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f5510164640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f55080a3d90 0x7f550813e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:13.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f5510164640 10 monclient(hunting): picked mon.noname-c con 0x7f55080a3d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:13.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f5510164640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55080a4170 0x7f550813e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:13.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f5510164640 10 monclient(hunting): picked mon.noname-a con 0x7f55080a4170 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:35:13.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f5510164640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5508133b20 0x7f55081cf5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:13.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f5510164640 10 monclient(hunting): picked mon.noname-b con 0x7f5508133b20 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:35:13.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f5510164640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:13.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f5510164640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f5508141e30 con 0x7f55080a4170 2024-04-26T00:35:13.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f5510164640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:13.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f5510164640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f550809c690 con 0x7f5508133b20 2024-04-26T00:35:13.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f5510164640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:13.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f5510164640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5508141fb0 con 0x7f55080a3d90 2024-04-26T00:35:13.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f5510164640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:13.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f5510164640 10 monclient(hunting): authenticate will time out at 1894.719711s 2024-04-26T00:35:13.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f550d6ff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55080a4170 0x7f550813e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:13.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f550df00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f55080a3d90 0x7f550813e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:13.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f550d6ff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55080a4170 0x7f550813e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:34432/0 (socket says 172.21.15.37:34432) 2024-04-26T00:35:13.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f550d6ff640 1 -- 172.21.15.37:0/2565926325 learned_addr learned my addr 172.21.15.37:0/2565926325 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:13.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f550d6ff640 10 monclient(hunting): get_auth_request con 0x7f55080a4170 auth_method 0 2024-04-26T00:35:13.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f550d6ff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:13.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f550d6ff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:13.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f550d6ff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:13.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f550df00640 10 monclient(hunting): get_auth_request con 0x7f55080a3d90 auth_method 0 2024-04-26T00:35:13.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f550df00640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:13.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f550df00640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:13.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f550df00640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:13.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.409+0000 7f550d6ff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:13.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f550d6ff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:13.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f550d6ff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:13.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f550df00640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:13.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f550df00640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:13.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f550df00640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:13.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f550e701640 1 --2- 172.21.15.37:0/2565926325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5508133b20 0x7f55081cf5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:13.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f550e701640 10 monclient(hunting): get_auth_request con 0x7f5508133b20 auth_method 0 2024-04-26T00:35:13.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f550e701640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:13.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f550e701640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:13.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f550e701640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:13.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f550d6ff640 10 monclient(hunting): handle_auth_done global_id 144525 payload 824 2024-04-26T00:35:13.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f550d6ff640 1 -- 172.21.15.37:0/2565926325 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f55080a3d90 msgr2=0x7f550813e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:13.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f550d6ff640 1 --2- 172.21.15.37:0/2565926325 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f55080a3d90 0x7f550813e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:13.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f550d6ff640 1 -- 172.21.15.37:0/2565926325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5508133b20 msgr2=0x7f55081cf5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:13.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f550d6ff640 1 --2- 172.21.15.37:0/2565926325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5508133b20 0x7f55081cf5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:13.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f550d6ff640 10 monclient: _finish_hunting 0 2024-04-26T00:35:13.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f550d6ff640 1 monclient: found mon.noname-a 2024-04-26T00:35:13.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f550d6ff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:13.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f550d6ff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:35:13.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f550d6ff640 1 -- 172.21.15.37:0/2565926325 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f550813ac50 con 0x7f55080a4170 2024-04-26T00:35:13.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f550df00640 1 --2- 172.21.15.37:0/2565926325 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f55080a3d90 0x7f550813e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:35:13.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f550d6ff640 1 -- 172.21.15.37:0/2565926325 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55080a4170 msgr2=0x7f550813e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:13.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f550d6ff640 1 --2- 172.21.15.37:0/2565926325 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55080a4170 0x7f550813e990 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f54f800d800 tx=0x7f54f800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:13.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f55067fc640 1 -- 172.21.15.37:0/2565926325 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54f800b8d0 con 0x7f55080a4170 2024-04-26T00:35:13.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f55067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:13.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f55067fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:35:13.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f55067fc640 10 monclient: dump: 2024-04-26T00:35:13.421 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:13.421 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:13.422 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.422 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.422 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:13.422 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:13.422 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:13.422 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:13.422 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:13.422 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:13.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f55067fc640 10 monclient: _finish_auth 0 2024-04-26T00:35:13.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f55067fc640 10 monclient: _check_auth_tickets 2024-04-26T00:35:13.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f55067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:13.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f5510164640 5 monclient: authenticate success, global_id 144525 2024-04-26T00:35:13.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f5510164640 10 monclient: _renew_subs 2024-04-26T00:35:13.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f55067fc640 1 -- 172.21.15.37:0/2565926325 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54f8019070 con 0x7f55080a4170 2024-04-26T00:35:13.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f5510164640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:13.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f5510164640 1 -- 172.21.15.37:0/2565926325 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f550813cf40 con 0x7f55080a4170 2024-04-26T00:35:13.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f55067fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:13.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f55067fc640 1 -- 172.21.15.37:0/2565926325 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54f800bc70 con 0x7f55080a4170 2024-04-26T00:35:13.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f55067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:13.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f55067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:13.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f55067fc640 10 monclient: dump: 2024-04-26T00:35:13.423 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:13.423 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:13.423 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.423 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.423 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:13.424 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:13.424 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:13.424 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:13.424 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:13.424 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:13.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f5507fff640 4 set_mon_vals no callback set 2024-04-26T00:35:13.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f5510164640 10 monclient: _renew_subs 2024-04-26T00:35:13.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f5510164640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:13.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f5510164640 1 -- 172.21.15.37:0/2565926325 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f550813b9d0 con 0x7f55080a4170 2024-04-26T00:35:13.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f5510164640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:13.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f5510164640 1 --2- 172.21.15.37:0/2565926325 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55081cfb30 0x7f55081cff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:13.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f5510164640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:13.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f5510164640 1 -- 172.21.15.37:0/2565926325 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f55080a90f0 con 0x7f55081cfb30 2024-04-26T00:35:13.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f55067fc640 1 -- 172.21.15.37:0/2565926325 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f54f801e460 con 0x7f55080a4170 2024-04-26T00:35:13.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f550df00640 1 --2- 172.21.15.37:0/2565926325 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55081cfb30 0x7f55081cff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:13.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f55067fc640 1 --2- 172.21.15.37:0/2565926325 >> v2:172.21.15.45:6812/2379918623 conn(0x7f54e8035e80 0x7f54e8038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:13.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.413+0000 7f550df00640 10 monclient: get_auth_request con 0x7f55081cfb30 auth_method 0 2024-04-26T00:35:13.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.417+0000 7f550df00640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:13.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.417+0000 7f550df00640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:13.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.417+0000 7f550df00640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:13.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.417+0000 7f55067fc640 1 -- 172.21.15.37:0/2565926325 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(598..598 src has 370..598) ==== 3927+0+0 (secure 0 0 0) 0x7f54f8048f70 con 0x7f55080a4170 2024-04-26T00:35:13.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.417+0000 7f550df00640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:13.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.417+0000 7f550df00640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:13.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.417+0000 7f550df00640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:13.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.417+0000 7f550df00640 10 monclient(hunting): handle_auth_done global_id 144528 payload 824 2024-04-26T00:35:13.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.417+0000 7f550e701640 1 --2- 172.21.15.37:0/2565926325 >> v2:172.21.15.45:6812/2379918623 conn(0x7f54e8035e80 0x7f54e8038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:13.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.417+0000 7f550df00640 1 -- 172.21.15.37:0/2565926325 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f550809ada0 con 0x7f55081cfb30 2024-04-26T00:35:13.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.417+0000 7f550e701640 10 monclient: get_auth_request con 0x7f54e8035e80 auth_method 0 2024-04-26T00:35:13.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.417+0000 7f550df00640 1 -- 172.21.15.37:0/2565926325 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55081cfb30 msgr2=0x7f55081cff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:13.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.417+0000 7f550df00640 1 --2- 172.21.15.37:0/2565926325 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55081cfb30 0x7f55081cff10 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f54f00ae000 tx=0x7f54f0019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:13.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.417+0000 7f55067fc640 1 -- 172.21.15.37:0/2565926325 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54f00adde0 con 0x7f55081cfb30 2024-04-26T00:35:13.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.417+0000 7f55067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:13.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.417+0000 7f55067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:13.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.417+0000 7f55067fc640 10 monclient: dump: 2024-04-26T00:35:13.426 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:13.426 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:13.426 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.426 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.426 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:13.427 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:13.427 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:13.427 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:13.427 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:13.427 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:13.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.417+0000 7f550e701640 1 --2- 172.21.15.37:0/2565926325 >> v2:172.21.15.45:6812/2379918623 conn(0x7f54e8035e80 0x7f54e8038340 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f5500007020 tx=0x7f5500009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:13.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.417+0000 7f55067fc640 1 -- 172.21.15.37:0/2565926325 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f54f00adde0 con 0x7f55081cfb30 2024-04-26T00:35:13.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.417+0000 7f55067fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:13.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.417+0000 7f55067fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:35:13.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.417+0000 7f55067fc640 1 -- 172.21.15.37:0/2565926325 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55081cfb30 msgr2=0x7f55081cff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:13.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.417+0000 7f55067fc640 1 --2- 172.21.15.37:0/2565926325 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55081cfb30 0x7f55081cff10 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f54f00ae000 tx=0x7f54f0019650 comp rx=0 tx=0).stop 2024-04-26T00:35:13.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.417+0000 7f55067fc640 1 -- 172.21.15.37:0/2565926325 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55081cfb30 msgr2=0x7f55081cff10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:35:13.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.417+0000 7f55067fc640 1 --2- 172.21.15.37:0/2565926325 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55081cfb30 0x7f55081cff10 secure :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f54f00ae000 tx=0x7f54f0019650 comp rx=0 tx=0).stop 2024-04-26T00:35:13.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.425+0000 7f5510164640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:35:13.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.425+0000 7f5510164640 1 --2- 172.21.15.37:0/2565926325 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5508134b30 0x7f5508134f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:13.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.425+0000 7f5510164640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:13.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.425+0000 7f5510164640 1 -- 172.21.15.37:0/2565926325 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f550813cb10 con 0x7f5508134b30 2024-04-26T00:35:13.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.425+0000 7f550df00640 1 --2- 172.21.15.37:0/2565926325 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5508134b30 0x7f5508134f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:13.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.425+0000 7f550df00640 10 monclient: get_auth_request con 0x7f5508134b30 auth_method 0 2024-04-26T00:35:13.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.425+0000 7f550df00640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:13.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.425+0000 7f550df00640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:13.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.425+0000 7f550df00640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:13.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.425+0000 7f550df00640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:13.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.425+0000 7f550df00640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:13.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.425+0000 7f550df00640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:13.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.425+0000 7f550df00640 10 monclient(hunting): handle_auth_done global_id 144531 payload 824 2024-04-26T00:35:13.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.425+0000 7f550df00640 1 -- 172.21.15.37:0/2565926325 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f550804f510 con 0x7f5508134b30 2024-04-26T00:35:13.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.429+0000 7f550df00640 1 -- 172.21.15.37:0/2565926325 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5508134b30 msgr2=0x7f5508134f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:13.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.429+0000 7f550df00640 1 --2- 172.21.15.37:0/2565926325 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5508134b30 0x7f5508134f10 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f54f0009e50 tx=0x7f54f00a7000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:13.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.429+0000 7f55067fc640 1 -- 172.21.15.37:0/2565926325 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54f0003e60 con 0x7f5508134b30 2024-04-26T00:35:13.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.429+0000 7f55067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:13.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.429+0000 7f55067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:13.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.429+0000 7f55067fc640 10 monclient: dump: 2024-04-26T00:35:13.430 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:13.430 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:13.430 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.431 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.431 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:13.431 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:13.431 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:13.431 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:13.431 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:13.431 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:13.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.429+0000 7f55067fc640 1 -- 172.21.15.37:0/2565926325 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4890 (secure 0 0 0) 0x7f54f00c5290 con 0x7f5508134b30 2024-04-26T00:35:13.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.429+0000 7f55067fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:35:13.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.429+0000 7f55067fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:35:13.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.429+0000 7f55067fc640 1 -- 172.21.15.37:0/2565926325 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5508134b30 msgr2=0x7f5508134f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:13.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.429+0000 7f55067fc640 1 --2- 172.21.15.37:0/2565926325 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5508134b30 0x7f5508134f10 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f54f0009e50 tx=0x7f54f00a7000 comp rx=0 tx=0).stop 2024-04-26T00:35:13.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.429+0000 7f55067fc640 1 -- 172.21.15.37:0/2565926325 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5508134b30 msgr2=0x7f5508134f10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:35:13.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.429+0000 7f55067fc640 1 --2- 172.21.15.37:0/2565926325 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5508134b30 0x7f5508134f10 secure :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f54f0009e50 tx=0x7f54f00a7000 comp rx=0 tx=0).stop 2024-04-26T00:35:13.432 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:35:13.432 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:35:13.432 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:35:13.432 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:35:13.432 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 114, 2024-04-26T00:35:13.432 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 199973, 2024-04-26T00:35:13.432 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:35:13.432 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:35:13.432 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:35:13.433 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:35:13.433 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:13.433 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 7, 2024-04-26T00:35:13.433 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:35:13.433 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:35:13.433 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:35:13.433 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:13.433 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:13.433 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:13.433 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:13.433 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:13.433 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:13.433 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:13.433 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:13.433 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:13.434 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:13.434 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:13.434 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:13.434 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:35:13.434 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:35:13.434 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:13.434 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:13.434 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:13.434 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:13.434 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:13.434 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:13.434 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:13.434 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:13.434 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:13.435 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:13.435 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:13.435 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:13.435 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:13.435 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:35:13.435 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:35:13.435 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:35:13.435 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:35:13.435 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:35:13.435 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:35:13.435 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:35:13.435 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:35:13.435 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:35:13.436 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:35:13.436 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:35:13.436 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:35:13.436 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:35:13.436 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:35:13.436 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:35:13.436 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:35:13.436 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:35:13.436 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:13.436 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:13.436 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:13.436 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:13.436 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:13.436 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:13.437 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:13.437 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:13.437 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:13.437 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:13.437 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:13.437 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:13.437 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:35:13.437 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:13.437 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:35:13.437 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:13.437 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:35:13.437 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:35:13.437 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:13.437 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:13.437 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:13.438 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:13.438 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:35:13.438 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:13.438 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:13.438 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:13.438 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:13.438 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:35:13.438 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:13.438 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:13.438 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:13.438 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:13.438 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:35:13.438 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:35:13.438 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:13.438 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:13.439 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:13.439 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:13.439 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:13.439 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:35:13.439 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:35:13.439 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:13.439 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:13.439 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:13.439 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:13.440 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:35:13.440 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:13.440 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:13.440 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:13.440 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:13.440 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:35:13.440 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:13.440 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:13.440 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:13.440 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:13.440 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:35:13.440 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:35:13.440 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:13.441 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:13.441 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:13.441 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:13.441 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:13.441 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:35:13.441 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:35:13.441 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:13.441 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:13.441 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:13.441 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:13.441 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:35:13.441 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:13.441 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:13.441 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:13.441 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:13.442 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:35:13.442 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:13.442 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:13.442 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:13.442 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:13.442 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:35:13.442 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:35:13.442 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:13.442 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:13.442 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:13.442 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:13.442 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:13.442 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:13.443 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:35:13.443 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:35:13.443 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:13.443 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:13.443 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:13.443 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:35:13.443 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:13.443 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:13.443 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:35:13.443 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:13.443 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:13.443 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:13.443 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:35:13.443 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:13.444 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:13.444 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:35:13.444 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:13.444 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:13.444 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:13.444 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 7 2024-04-26T00:35:13.444 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:13.444 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:13.444 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:35:13.444 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:13.444 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:13.444 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:13.444 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:35:13.444 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:13.444 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:13.445 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:13.445 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:35:13.445 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:35:13.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.429+0000 7f5510164640 1 -- 172.21.15.37:0/2565926325 >> v2:172.21.15.45:6812/2379918623 conn(0x7f54e8035e80 msgr2=0x7f54e8038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:13.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.429+0000 7f5510164640 1 --2- 172.21.15.37:0/2565926325 >> v2:172.21.15.45:6812/2379918623 conn(0x7f54e8035e80 0x7f54e8038340 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f5500007020 tx=0x7f5500009040 comp rx=0 tx=0).stop 2024-04-26T00:35:13.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.429+0000 7f5510164640 10 monclient: shutdown 2024-04-26T00:35:13.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.429+0000 7f5510164640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:13.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.429+0000 7f5510164640 1 -- 172.21.15.37:0/2565926325 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55080a4170 msgr2=0x7f550813e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:13.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.429+0000 7f5510164640 1 --2- 172.21.15.37:0/2565926325 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55080a4170 0x7f550813e990 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f54f800d800 tx=0x7f54f800dcc0 comp rx=0 tx=0).stop 2024-04-26T00:35:13.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.429+0000 7f550e701640 1 -- 172.21.15.37:0/2565926325 reap_dead start 2024-04-26T00:35:13.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.429+0000 7f550e701640 1 -- 172.21.15.37:0/2565926325 reap_dead start 2024-04-26T00:35:13.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.429+0000 7f5510164640 1 -- 172.21.15.37:0/2565926325 shutdown_connections 2024-04-26T00:35:13.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.429+0000 7f5510164640 1 -- 172.21.15.37:0/2565926325 >> 172.21.15.37:0/2565926325 conn(0x7f55080aa0a0 msgr2=0x7f55080a2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:13.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.429+0000 7f5510164640 1 -- 172.21.15.37:0/2565926325 shutdown_connections 2024-04-26T00:35:13.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.429+0000 7f5510164640 1 -- 172.21.15.37:0/2565926325 wait complete. 2024-04-26T00:35:13.448 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:35:13.448 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:35:13.448 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:35:13.525 INFO:tasks.rados.rados.0.smithi045.stdout:2575: done (0 left) 2024-04-26T00:35:13.525 INFO:tasks.rados.rados.0.smithi045.stdout:2576: write oid 37 current snap is 288 2024-04-26T00:35:13.525 INFO:tasks.rados.rados.0.smithi045.stdout:2576: seq_num 927 ranges {780089=634536,1879396=620991,3240526=339563} 2024-04-26T00:35:13.548 INFO:tasks.rados.rados.0.smithi045.stdout:2576: writing smithi04526579-37 from 780089 to 1414625 tid 1 2024-04-26T00:35:13.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.561+0000 7f968a045640 10 monclient: get_monmap_and_config 2024-04-26T00:35:13.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.561+0000 7f968a045640 10 monclient: build_initial_monmap 2024-04-26T00:35:13.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.561+0000 7f968a045640 10 monclient: monmap: 2024-04-26T00:35:13.561 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:13.561 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:13.561 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:35:13.567800+0000 2024-04-26T00:35:13.561 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:35:13.567800+0000 2024-04-26T00:35:13.561 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:13.561 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:13.561 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:35:13.561 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:35:13.561 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:35:13.561 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:35:13.561 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:35:13.562 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:13.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.565+0000 7f968a045640 1 Processor -- start 2024-04-26T00:35:13.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.565+0000 7f968a045640 1 -- start start 2024-04-26T00:35:13.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.565+0000 7f968a045640 10 monclient: init 2024-04-26T00:35:13.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.565+0000 7f968a045640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:13.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.565+0000 7f968a045640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-04-26T00:35:13.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.565+0000 7f968a045640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f968409c8d0 addr v1:172.21.15.37:6789/0 2024-04-26T00:35:13.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.565+0000 7f968a045640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f968409ccb0 0x7f9684131ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:13.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.565+0000 7f968a045640 10 monclient(hunting): picked mon.noname-c con 0x7f968409ccb0 addr v2:172.21.15.45:3300/0 2024-04-26T00:35:13.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.565+0000 7f968a045640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f96840a2270 addr v1:172.21.15.45:6789/0 2024-04-26T00:35:13.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.565+0000 7f968a045640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f968406ba00 con 0x7f968409c8d0 2024-04-26T00:35:13.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.565+0000 7f968a045640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f968406bce0 con 0x7f96840a2270 2024-04-26T00:35:13.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.565+0000 7f968a045640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:13.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.565+0000 7f968a045640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f96840a0130 con 0x7f968409ccb0 2024-04-26T00:35:13.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.565+0000 7f968a045640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:13.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.565+0000 7f968a045640 10 monclient(hunting): authenticate will time out at 1894.873296s 2024-04-26T00:35:13.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.565+0000 7f96837fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f968409c8d0 0x7f96840a1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:36046/0 (socket says 172.21.15.37:36046) 2024-04-26T00:35:13.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.565+0000 7f96837fe640 1 -- 172.21.15.37:0/677351396 learned_addr learned my addr 172.21.15.37:0/677351396 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:35:13.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.565+0000 7f9682ffd640 1 --2- 172.21.15.37:0/677351396 >> v2:172.21.15.45:3300/0 conn(0x7f968409ccb0 0x7f9684131ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:13.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.565+0000 7f9682ffd640 10 monclient(hunting): get_auth_request con 0x7f968409ccb0 auth_method 0 2024-04-26T00:35:13.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.565+0000 7f9682ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:13.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.565+0000 7f9682ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:13.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.565+0000 7f9682ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:13.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.565+0000 7f9682ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:13.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.565+0000 7f9682ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:13.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.565+0000 7f9682ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:13.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.565+0000 7f9682ffd640 10 monclient(hunting): handle_auth_done global_id 174208 payload 274 2024-04-26T00:35:13.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.565+0000 7f9682ffd640 1 -- 172.21.15.37:0/677351396 >> v1:172.21.15.45:6789/0 conn(0x7f96840a2270 legacy=0x7f968413ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:13.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.565+0000 7f9682ffd640 1 -- 172.21.15.37:0/677351396 >> v1:172.21.15.37:6789/0 conn(0x7f968409c8d0 legacy=0x7f96840a1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:13.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.565+0000 7f9682ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:35:13.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.565+0000 7f9682ffd640 1 monclient: found mon.noname-c 2024-04-26T00:35:13.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.565+0000 7f9682ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:13.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.565+0000 7f9682ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:35:13.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.565+0000 7f9682ffd640 1 -- 172.21.15.37:0/677351396 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f968406bfc0 con 0x7f968409ccb0 2024-04-26T00:35:13.567 INFO:tasks.rados.rados.0.smithi045.stdout:2576: writing smithi04526579-37 from 1879396 to 2500387 tid 2 2024-04-26T00:35:13.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.569+0000 7f9682ffd640 1 -- 172.21.15.37:0/677351396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f968409ccb0 msgr2=0x7f9684131ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:13.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.569+0000 7f9682ffd640 1 --2- 172.21.15.37:0/677351396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f968409ccb0 0x7f9684131ab0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f96780097c0 tx=0x7f967801c2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3fcbe11fc5580ba server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:13.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.569+0000 7f96827fc640 1 -- 172.21.15.37:0/677351396 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96780b0070 con 0x7f968409ccb0 2024-04-26T00:35:13.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.569+0000 7f96827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:13.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.569+0000 7f96827fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:35:13.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.569+0000 7f96827fc640 10 monclient: dump: 2024-04-26T00:35:13.569 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:13.569 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:13.569 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.569 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.569 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:13.569 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:13.569 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:13.569 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:13.569 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:13.569 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:13.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.569+0000 7f96827fc640 10 monclient: _finish_auth 0 2024-04-26T00:35:13.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.569+0000 7f96827fc640 10 monclient: _check_auth_tickets 2024-04-26T00:35:13.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.569+0000 7f96827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:13.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.569+0000 7f96827fc640 1 -- 172.21.15.37:0/677351396 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96780ab600 con 0x7f968409ccb0 2024-04-26T00:35:13.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.569+0000 7f968a045640 5 monclient: authenticate success, global_id 174208 2024-04-26T00:35:13.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.569+0000 7f968a045640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:35:13.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.569+0000 7f96827fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:13.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.569+0000 7f96827fc640 1 -- 172.21.15.37:0/677351396 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96780ab9d0 con 0x7f968409ccb0 2024-04-26T00:35:13.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.569+0000 7f96827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:13.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.569+0000 7f96827fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:35:13.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.569+0000 7f96827fc640 10 monclient: dump: 2024-04-26T00:35:13.570 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:13.570 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:13.570 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.570 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.570 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:13.570 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:13.571 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:13.571 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:13.571 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:13.571 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:13.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.569+0000 7f968a045640 10 monclient: get_monmap_and_config success 2024-04-26T00:35:13.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.569+0000 7f968a045640 4 set_mon_vals no callback set 2024-04-26T00:35:13.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f968a045640 10 monclient: shutdown 2024-04-26T00:35:13.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f968a045640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:13.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f968a045640 1 -- 172.21.15.37:0/677351396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f968409ccb0 msgr2=0x7f9684131ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:13.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f968a045640 1 --2- 172.21.15.37:0/677351396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f968409ccb0 0x7f9684131ab0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f96780097c0 tx=0x7f967801c2e0 comp rx=0 tx=0).stop 2024-04-26T00:35:13.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f968a045640 1 -- 172.21.15.37:0/677351396 shutdown_connections 2024-04-26T00:35:13.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f968a045640 1 --2- 172.21.15.37:0/677351396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f968409ccb0 0x7f9684131ab0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:13.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f968a045640 1 -- 172.21.15.37:0/677351396 >> 172.21.15.37:0/677351396 conn(0x7f96840a2c50 msgr2=0x7f96840a3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:13.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f968a045640 1 -- 172.21.15.37:0/677351396 shutdown_connections 2024-04-26T00:35:13.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f968a045640 1 -- 172.21.15.37:0/677351396 wait complete. 2024-04-26T00:35:13.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f968a045640 10 monclient: build_initial_monmap 2024-04-26T00:35:13.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f968a045640 10 monclient: monmap: 2024-04-26T00:35:13.575 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:13.575 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:35:13.575 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:35:13.575 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:35:13.575 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:13.576 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:35:13.576 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:35:13.576 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:35:13.576 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:35:13.576 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:13.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f968a045640 1 Processor -- start 2024-04-26T00:35:13.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f968a045640 1 -- start start 2024-04-26T00:35:13.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f968a045640 10 monclient: init 2024-04-26T00:35:13.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f968a045640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:13.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f968a045640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:35:13.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f968a045640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f968409c8d0 0x7f968413cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:13.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f968a045640 10 monclient(hunting): picked mon.noname-a con 0x7f968409c8d0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:35:13.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f968a045640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f968409ccb0 0x7f968413d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:13.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f968a045640 10 monclient(hunting): picked mon.noname-b con 0x7f968409ccb0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:35:13.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f968a045640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f96840a2270 0x7f968409d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:13.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f968a045640 10 monclient(hunting): picked mon.noname-c con 0x7f96840a2270 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:13.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f968a045640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:13.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f968a045640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9684141160 con 0x7f968409c8d0 2024-04-26T00:35:13.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f968a045640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:13.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f968a045640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9684140fe0 con 0x7f968409ccb0 2024-04-26T00:35:13.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f968a045640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:13.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f968a045640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f96841412e0 con 0x7f96840a2270 2024-04-26T00:35:13.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f968a045640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:13.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f968a045640 10 monclient(hunting): authenticate will time out at 1894.882443s 2024-04-26T00:35:13.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f96837fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f968409c8d0 0x7f968413cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:13.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f9683fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f96840a2270 0x7f968409d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:13.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f96837fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f968409c8d0 0x7f968413cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:34464/0 (socket says 172.21.15.37:34464) 2024-04-26T00:35:13.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f96837fe640 1 -- 172.21.15.37:0/2272532095 learned_addr learned my addr 172.21.15.37:0/2272532095 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:13.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f96837fe640 10 monclient(hunting): get_auth_request con 0x7f968409c8d0 auth_method 0 2024-04-26T00:35:13.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f96837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:13.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f96837fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:13.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f96837fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:13.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f9683fff640 10 monclient(hunting): get_auth_request con 0x7f96840a2270 auth_method 0 2024-04-26T00:35:13.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f9683fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:13.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f9683fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:13.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f9683fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:13.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f96837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:13.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f96837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:13.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f96837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:13.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f9683fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:13.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f9683fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:13.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f9683fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:13.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f9682ffd640 1 --2- 172.21.15.37:0/2272532095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f968409ccb0 0x7f968413d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:13.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f9682ffd640 10 monclient(hunting): get_auth_request con 0x7f968409ccb0 auth_method 0 2024-04-26T00:35:13.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f9682ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:13.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f9682ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:13.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f9682ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:13.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f9683fff640 10 monclient(hunting): handle_auth_done global_id 184130 payload 824 2024-04-26T00:35:13.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f9683fff640 1 -- 172.21.15.37:0/2272532095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f968409ccb0 msgr2=0x7f968413d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:13.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f9683fff640 1 --2- 172.21.15.37:0/2272532095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f968409ccb0 0x7f968413d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:13.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f9683fff640 1 -- 172.21.15.37:0/2272532095 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f968409c8d0 msgr2=0x7f968413cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:13.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f9683fff640 1 --2- 172.21.15.37:0/2272532095 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f968409c8d0 0x7f968413cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:13.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f9683fff640 10 monclient: _finish_hunting 0 2024-04-26T00:35:13.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f9683fff640 1 monclient: found mon.noname-c 2024-04-26T00:35:13.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f9683fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:13.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f9683fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:35:13.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f9683fff640 1 -- 172.21.15.37:0/2272532095 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9684090db0 con 0x7f96840a2270 2024-04-26T00:35:13.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f96837fe640 1 --2- 172.21.15.37:0/2272532095 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f968409c8d0 0x7f968413cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:35:13.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.573+0000 7f9682ffd640 1 --2- 172.21.15.37:0/2272532095 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f968409ccb0 0x7f968413d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:35:13.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f9683fff640 1 -- 172.21.15.37:0/2272532095 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f96840a2270 msgr2=0x7f968409d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:13.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f9683fff640 1 --2- 172.21.15.37:0/2272532095 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f96840a2270 0x7f968409d650 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f967c00c7b0 tx=0x7f967c00cc70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:13.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f966bfff640 1 -- 172.21.15.37:0/2272532095 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f967c016020 con 0x7f96840a2270 2024-04-26T00:35:13.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f966bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:13.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f966bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:35:13.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f966bfff640 10 monclient: dump: 2024-04-26T00:35:13.580 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:13.580 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:13.580 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.580 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.580 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:13.580 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:13.580 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:13.580 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:13.580 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:13.581 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:13.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f966bfff640 10 monclient: _finish_auth 0 2024-04-26T00:35:13.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f966bfff640 10 monclient: _check_auth_tickets 2024-04-26T00:35:13.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f966bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:13.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f966bfff640 1 -- 172.21.15.37:0/2272532095 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f967c00ac60 con 0x7f96840a2270 2024-04-26T00:35:13.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f966bfff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:13.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f966bfff640 1 -- 172.21.15.37:0/2272532095 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f967c00ce60 con 0x7f96840a2270 2024-04-26T00:35:13.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f968a045640 5 monclient: authenticate success, global_id 184130 2024-04-26T00:35:13.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f968a045640 10 monclient: _renew_subs 2024-04-26T00:35:13.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f968a045640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:35:13.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f968a045640 1 -- 172.21.15.37:0/2272532095 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f968413d9a0 con 0x7f96840a2270 2024-04-26T00:35:13.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f966bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:13.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f9681ffb640 4 set_mon_vals no callback set 2024-04-26T00:35:13.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f966bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:35:13.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f966bfff640 10 monclient: dump: 2024-04-26T00:35:13.581 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:13.582 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:13.582 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.582 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.582 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:13.582 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:13.582 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:13.582 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:13.582 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:13.582 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:13.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f968a045640 10 monclient: _renew_subs 2024-04-26T00:35:13.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f968a045640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:35:13.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f968a045640 1 -- 172.21.15.37:0/2272532095 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f968408c6a0 con 0x7f96840a2270 2024-04-26T00:35:13.583 INFO:tasks.rados.rados.0.smithi045.stdout:2576: writing smithi04526579-37 from 3240526 to 3580089 tid 3 2024-04-26T00:35:13.583 INFO:tasks.rados.rados.0.smithi045.stdout:2577: delete oid 25 current snap is 288 2024-04-26T00:35:13.583 INFO:tasks.rados.rados.0.smithi045.stdout:2576: finishing write tid 1 to smithi04526579-37 2024-04-26T00:35:13.584 INFO:tasks.rados.rados.0.smithi045.stdout:2576: finishing write tid 2 to smithi04526579-37 2024-04-26T00:35:13.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f968a045640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:13.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f968a045640 1 --2- 172.21.15.37:0/2272532095 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f968409db90 0x7f968409df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:13.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f968a045640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:13.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f968a045640 1 -- 172.21.15.37:0/2272532095 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f968409e4b0 con 0x7f968409db90 2024-04-26T00:35:13.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f966bfff640 1 -- 172.21.15.37:0/2272532095 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f967c019020 con 0x7f96840a2270 2024-04-26T00:35:13.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f96837fe640 1 --2- 172.21.15.37:0/2272532095 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f968409db90 0x7f968409df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:13.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f96837fe640 10 monclient: get_auth_request con 0x7f968409db90 auth_method 0 2024-04-26T00:35:13.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f96837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:13.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f96837fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:13.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f96837fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:13.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f966bfff640 1 --2- 172.21.15.37:0/2272532095 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9670035e80 0x7f9670038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:13.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f96837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:13.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f96837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:13.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f96837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:13.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f966bfff640 1 -- 172.21.15.37:0/2272532095 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(599..599 src has 370..599) ==== 3927+0+0 (secure 0 0 0) 0x7f967c0489e0 con 0x7f96840a2270 2024-04-26T00:35:13.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f96837fe640 10 monclient(hunting): handle_auth_done global_id 184133 payload 824 2024-04-26T00:35:13.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f96837fe640 1 -- 172.21.15.37:0/2272532095 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f968404f510 con 0x7f968409db90 2024-04-26T00:35:13.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f9682ffd640 1 --2- 172.21.15.37:0/2272532095 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9670035e80 0x7f9670038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:13.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.577+0000 7f9682ffd640 10 monclient: get_auth_request con 0x7f9670035e80 auth_method 0 2024-04-26T00:35:13.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.581+0000 7f96837fe640 1 -- 172.21.15.37:0/2272532095 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f968409db90 msgr2=0x7f968409df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:13.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.581+0000 7f96837fe640 1 --2- 172.21.15.37:0/2272532095 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f968409db90 0x7f968409df70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f96740072a0 tx=0x7f9674007760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:13.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.581+0000 7f966bfff640 1 -- 172.21.15.37:0/2272532095 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f967400fde0 con 0x7f968409db90 2024-04-26T00:35:13.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.581+0000 7f966bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:13.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.581+0000 7f966bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:35:13.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.581+0000 7f966bfff640 10 monclient: dump: 2024-04-26T00:35:13.588 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:13.588 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:13.588 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.588 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.588 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:13.588 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:13.588 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:13.589 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:13.589 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:13.589 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:13.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.581+0000 7f966bfff640 1 -- 172.21.15.37:0/2272532095 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f967400fde0 con 0x7f968409db90 2024-04-26T00:35:13.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.581+0000 7f966bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:13.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.581+0000 7f966bfff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:35:13.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.581+0000 7f966bfff640 1 -- 172.21.15.37:0/2272532095 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f968409db90 msgr2=0x7f968409df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:13.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.581+0000 7f966bfff640 1 --2- 172.21.15.37:0/2272532095 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f968409db90 0x7f968409df70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f96740072a0 tx=0x7f9674007760 comp rx=0 tx=0).stop 2024-04-26T00:35:13.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.581+0000 7f966bfff640 1 -- 172.21.15.37:0/2272532095 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f968409db90 msgr2=0x7f968409df70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:35:13.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.581+0000 7f966bfff640 1 --2- 172.21.15.37:0/2272532095 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f968409db90 0x7f968409df70 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:13.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.581+0000 7f9682ffd640 1 --2- 172.21.15.37:0/2272532095 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9670035e80 0x7f9670038340 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f96780ad000 tx=0x7f96780a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:13.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.589+0000 7f968a045640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:35:13.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.589+0000 7f968a045640 1 --2- 172.21.15.37:0/2272532095 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9684092750 0x7f9684092b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:13.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.589+0000 7f968a045640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:13.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.589+0000 7f968a045640 1 -- 172.21.15.37:0/2272532095 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f968409ed20 con 0x7f9684092750 2024-04-26T00:35:13.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.589+0000 7f96837fe640 1 --2- 172.21.15.37:0/2272532095 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9684092750 0x7f9684092b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:13.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.589+0000 7f96837fe640 10 monclient: get_auth_request con 0x7f9684092750 auth_method 0 2024-04-26T00:35:13.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.589+0000 7f96837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:13.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.589+0000 7f96837fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:13.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.589+0000 7f96837fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:13.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.589+0000 7f96837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:13.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.589+0000 7f96837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:13.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.589+0000 7f96837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:13.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.589+0000 7f96837fe640 10 monclient(hunting): handle_auth_done global_id 184136 payload 824 2024-04-26T00:35:13.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.589+0000 7f96837fe640 1 -- 172.21.15.37:0/2272532095 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f968408d640 con 0x7f9684092750 2024-04-26T00:35:13.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.589+0000 7f96837fe640 1 -- 172.21.15.37:0/2272532095 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9684092750 msgr2=0x7f9684092b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:13.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.589+0000 7f96837fe640 1 --2- 172.21.15.37:0/2272532095 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9684092750 0x7f9684092b60 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9674017000 tx=0x7f9674004950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:13.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.589+0000 7f966bfff640 1 -- 172.21.15.37:0/2272532095 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9674006450 con 0x7f9684092750 2024-04-26T00:35:13.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.589+0000 7f966bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:13.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.593+0000 7f966bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:35:13.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.593+0000 7f966bfff640 10 monclient: dump: 2024-04-26T00:35:13.591 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:13.591 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:13.591 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.591 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.591 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:13.592 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:13.592 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:13.592 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:13.592 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:13.592 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:13.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.593+0000 7f966bfff640 1 -- 172.21.15.37:0/2272532095 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4547 (secure 0 0 0) 0x7f9674006450 con 0x7f9684092750 2024-04-26T00:35:13.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.593+0000 7f966bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:35:13.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.593+0000 7f966bfff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:35:13.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.593+0000 7f966bfff640 1 -- 172.21.15.37:0/2272532095 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9684092750 msgr2=0x7f9684092b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:13.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.593+0000 7f966bfff640 1 --2- 172.21.15.37:0/2272532095 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9684092750 0x7f9684092b60 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9674017000 tx=0x7f9674004950 comp rx=0 tx=0).stop 2024-04-26T00:35:13.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.593+0000 7f966bfff640 1 -- 172.21.15.37:0/2272532095 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9684092750 msgr2=0x7f9684092b60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:35:13.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.593+0000 7f966bfff640 1 --2- 172.21.15.37:0/2272532095 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9684092750 0x7f9684092b60 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9674017000 tx=0x7f9674004950 comp rx=0 tx=0).stop 2024-04-26T00:35:13.593 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:35:13.593 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:35:13.593 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:35:13.593 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:35:13.593 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 114, 2024-04-26T00:35:13.593 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 12739, 2024-04-26T00:35:13.593 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:35:13.593 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:35:13.593 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:35:13.593 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:35:13.593 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:13.593 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 7, 2024-04-26T00:35:13.593 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:35:13.593 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:35:13.594 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:35:13.594 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:13.594 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:13.594 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:13.594 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:13.594 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:13.594 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:13.594 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:13.594 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:13.594 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:13.594 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:13.594 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:13.594 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:13.594 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:35:13.594 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:35:13.594 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:13.595 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:13.595 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:13.595 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:13.595 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:13.595 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:13.595 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:13.595 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:13.595 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:13.595 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:13.595 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:13.595 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:13.595 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:13.595 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:35:13.595 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:35:13.595 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:35:13.596 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:35:13.596 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:35:13.596 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:35:13.596 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:35:13.596 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:35:13.596 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:35:13.596 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:35:13.596 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:35:13.596 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:35:13.596 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:35:13.596 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:35:13.596 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:35:13.596 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:35:13.596 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:35:13.597 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:13.597 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:13.597 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:13.597 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:13.597 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:13.597 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:13.597 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:13.597 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:13.597 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:13.597 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:13.597 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:13.597 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:13.597 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:35:13.597 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:13.598 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:35:13.598 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:13.598 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:35:13.598 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:35:13.598 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:13.598 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:13.598 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:13.598 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:13.598 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:35:13.598 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:13.598 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:13.598 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:13.598 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:13.598 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:35:13.599 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:13.599 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:13.599 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:13.599 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:13.599 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:35:13.599 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:35:13.599 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:13.599 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:13.599 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:13.599 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:13.599 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:13.599 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:35:13.599 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:35:13.599 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:13.599 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:13.600 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:13.600 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:13.600 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:35:13.600 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:13.600 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:13.600 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:13.600 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:13.600 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:35:13.600 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:13.600 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:13.600 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:13.600 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:13.600 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:35:13.600 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:35:13.600 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:13.601 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:13.601 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:13.601 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:13.601 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:13.601 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:35:13.601 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:35:13.601 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:13.601 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:13.601 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:13.601 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:13.601 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:35:13.601 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:13.601 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:13.601 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:13.601 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:13.601 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:35:13.602 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:13.602 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:13.602 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:13.602 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:13.602 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:35:13.602 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:35:13.602 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:13.602 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:13.602 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:13.602 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:13.602 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:13.602 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:13.602 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:35:13.602 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:35:13.603 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:13.603 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:13.603 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:13.603 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:35:13.603 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:13.603 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:13.603 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:35:13.603 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:13.603 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:13.603 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:13.603 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:35:13.603 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:13.603 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:13.603 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:13.603 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:35:13.604 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:35:13.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.593+0000 7f968a045640 1 -- 172.21.15.37:0/2272532095 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9670035e80 msgr2=0x7f9670038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:13.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.593+0000 7f968a045640 1 --2- 172.21.15.37:0/2272532095 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9670035e80 0x7f9670038340 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f96780ad000 tx=0x7f96780a6000 comp rx=0 tx=0).stop 2024-04-26T00:35:13.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.593+0000 7f968a045640 10 monclient: shutdown 2024-04-26T00:35:13.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.593+0000 7f968a045640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:13.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.593+0000 7f968a045640 1 -- 172.21.15.37:0/2272532095 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f96840a2270 msgr2=0x7f968409d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:13.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.593+0000 7f968a045640 1 --2- 172.21.15.37:0/2272532095 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f96840a2270 0x7f968409d650 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f967c00c7b0 tx=0x7f967c00cc70 comp rx=0 tx=0).stop 2024-04-26T00:35:13.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.593+0000 7f9683fff640 1 -- 172.21.15.37:0/2272532095 reap_dead start 2024-04-26T00:35:13.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.593+0000 7f9683fff640 1 -- 172.21.15.37:0/2272532095 reap_dead start 2024-04-26T00:35:13.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.593+0000 7f968a045640 1 -- 172.21.15.37:0/2272532095 shutdown_connections 2024-04-26T00:35:13.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.593+0000 7f968a045640 1 -- 172.21.15.37:0/2272532095 >> 172.21.15.37:0/2272532095 conn(0x7f96840a2c50 msgr2=0x7f96840a4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:13.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.593+0000 7f968a045640 1 -- 172.21.15.37:0/2272532095 shutdown_connections 2024-04-26T00:35:13.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.593+0000 7f968a045640 1 -- 172.21.15.37:0/2272532095 wait complete. 2024-04-26T00:35:13.605 INFO:tasks.rados.rados.0.smithi045.stdout:2576: finishing write tid 3 to smithi04526579-37 2024-04-26T00:35:13.605 INFO:tasks.rados.rados.0.smithi045.stdout:2576: finishing write tid 4 to smithi04526579-37 2024-04-26T00:35:13.605 INFO:tasks.rados.rados.0.smithi045.stdout:2576: finishing write tid 5 to smithi04526579-37 2024-04-26T00:35:13.606 INFO:tasks.rados.rados.0.smithi045.stdout:2576: oid 37 updating version 0 to 2251 2024-04-26T00:35:13.606 INFO:tasks.rados.rados.0.smithi045.stdout:2576: oid 37 updating version 2251 to 2252 2024-04-26T00:35:13.606 INFO:tasks.rados.rados.0.smithi045.stdout:2576: oid 37 version 2252 is already newer than 2249 2024-04-26T00:35:13.606 INFO:tasks.rados.rados.0.smithi045.stdout:2576: oid 37 version 2252 is already newer than 2250 2024-04-26T00:35:13.606 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 2252 (ObjNum 927 snap 288 seq_num 927) dirty exists 2024-04-26T00:35:13.606 INFO:tasks.rados.rados.0.smithi045.stdout:2576: left oid 37 (ObjNum 927 snap 288 seq_num 927) 2024-04-26T00:35:13.606 INFO:tasks.rados.rados.0.smithi045.stdout:2576: done (1 left) 2024-04-26T00:35:13.606 INFO:tasks.rados.rados.0.smithi045.stdout:2577: done (0 left) 2024-04-26T00:35:13.606 INFO:tasks.rados.rados.0.smithi045.stdout:2578: snap_remove snap 281 2024-04-26T00:35:13.607 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:35:13.607 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:35:13.607 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:35:13.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.733+0000 7f7386a38640 10 monclient: get_monmap_and_config 2024-04-26T00:35:13.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.733+0000 7f7386a38640 10 monclient: build_initial_monmap 2024-04-26T00:35:13.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.733+0000 7f7386a38640 10 monclient: monmap: 2024-04-26T00:35:13.730 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:13.730 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:13.730 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:35:13.736735+0000 2024-04-26T00:35:13.731 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:35:13.736735+0000 2024-04-26T00:35:13.731 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:13.731 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:13.731 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:35:13.731 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:35:13.731 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:35:13.731 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:35:13.731 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:35:13.731 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:13.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.733+0000 7f7386a38640 1 Processor -- start 2024-04-26T00:35:13.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.733+0000 7f7386a38640 1 -- start start 2024-04-26T00:35:13.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.733+0000 7f7386a38640 10 monclient: init 2024-04-26T00:35:13.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.733+0000 7f7386a38640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:13.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.733+0000 7f7386a38640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-04-26T00:35:13.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.733+0000 7f7386a38640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f73800a0300 addr v1:172.21.15.45:6789/0 2024-04-26T00:35:13.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.733+0000 7f7386a38640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f738009fc50 0x7f7380139ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:13.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.733+0000 7f7386a38640 10 monclient(hunting): picked mon.noname-a con 0x7f738009fc50 addr v2:172.21.15.37:3300/0 2024-04-26T00:35:13.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.733+0000 7f7386a38640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f738013a400 addr v1:172.21.15.37:6789/0 2024-04-26T00:35:13.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.733+0000 7f7386a38640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f738009c190 con 0x7f738013a400 2024-04-26T00:35:13.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.733+0000 7f7386a38640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73800a1110 con 0x7f73800a0300 2024-04-26T00:35:13.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.733+0000 7f7386a38640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:13.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.733+0000 7f7386a38640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f73800a16d0 con 0x7f738009fc50 2024-04-26T00:35:13.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.733+0000 7f7386a38640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:13.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.733+0000 7f7386a38640 10 monclient(hunting): authenticate will time out at 1895.042460s 2024-04-26T00:35:13.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.733+0000 7f737f7fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f738009fc50 0x7f7380139ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:13.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.733+0000 7f7384fd5640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f738013a400 0x7f738013c7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:36048/0 (socket says 172.21.15.37:36048) 2024-04-26T00:35:13.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.733+0000 7f7384fd5640 1 -- 172.21.15.37:0/3786760141 learned_addr learned my addr 172.21.15.37:0/3786760141 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:35:13.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.733+0000 7f737f7fe640 10 monclient(hunting): get_auth_request con 0x7f738009fc50 auth_method 0 2024-04-26T00:35:13.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.733+0000 7f737f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:13.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.733+0000 7f737f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:13.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.733+0000 7f737f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:13.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.733+0000 7f737f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:13.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.733+0000 7f737f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:13.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.733+0000 7f737f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:13.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.733+0000 7f737f7fe640 10 monclient(hunting): handle_auth_done global_id 144537 payload 274 2024-04-26T00:35:13.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.733+0000 7f737f7fe640 1 -- 172.21.15.37:0/3786760141 >> v1:172.21.15.45:6789/0 conn(0x7f73800a0300 legacy=0x7f73800a06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:13.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.733+0000 7f737f7fe640 1 -- 172.21.15.37:0/3786760141 >> v1:172.21.15.37:6789/0 conn(0x7f738013a400 legacy=0x7f738013c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:13.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.733+0000 7f737f7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:35:13.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.733+0000 7f737f7fe640 1 monclient: found mon.noname-a 2024-04-26T00:35:13.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.733+0000 7f737f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:13.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.733+0000 7f737f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:35:13.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.733+0000 7f737f7fe640 1 -- 172.21.15.37:0/3786760141 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f738009f2a0 con 0x7f738009fc50 2024-04-26T00:35:13.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.737+0000 7f737f7fe640 1 -- 172.21.15.37:0/3786760141 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f738009fc50 msgr2=0x7f7380139ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:13.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.737+0000 7f737f7fe640 1 --2- 172.21.15.37:0/3786760141 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f738009fc50 0x7f7380139ec0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f738009f4d0 tx=0x7f737401c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e584d0d74f48f2e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:13.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.737+0000 7f737effd640 1 -- 172.21.15.37:0/3786760141 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73740b2070 con 0x7f738009fc50 2024-04-26T00:35:13.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.737+0000 7f737effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:13.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.737+0000 7f737effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:35:13.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.737+0000 7f737effd640 10 monclient: dump: 2024-04-26T00:35:13.738 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:13.738 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:13.739 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.739 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.739 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:13.739 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:13.739 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:13.739 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:13.739 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:13.739 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:13.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.737+0000 7f737effd640 10 monclient: _finish_auth 0 2024-04-26T00:35:13.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.737+0000 7f737effd640 10 monclient: _check_auth_tickets 2024-04-26T00:35:13.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.737+0000 7f737effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:13.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.737+0000 7f737effd640 1 -- 172.21.15.37:0/3786760141 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73740a58d0 con 0x7f738009fc50 2024-04-26T00:35:13.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.737+0000 7f737effd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:13.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.737+0000 7f737effd640 1 -- 172.21.15.37:0/3786760141 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73740a5ca0 con 0x7f738009fc50 2024-04-26T00:35:13.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.737+0000 7f737effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:13.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.737+0000 7f737effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:13.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.737+0000 7f737effd640 10 monclient: dump: 2024-04-26T00:35:13.740 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:13.740 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:13.740 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.740 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.740 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:13.740 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:13.740 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:13.740 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:13.740 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:13.740 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:13.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.737+0000 7f7386a38640 5 monclient: authenticate success, global_id 144537 2024-04-26T00:35:13.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.737+0000 7f7386a38640 10 monclient: get_monmap_and_config success 2024-04-26T00:35:13.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.737+0000 7f7386a38640 4 set_mon_vals no callback set 2024-04-26T00:35:13.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f7386a38640 10 monclient: shutdown 2024-04-26T00:35:13.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f7386a38640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:13.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f7386a38640 1 -- 172.21.15.37:0/3786760141 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f738009fc50 msgr2=0x7f7380139ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:13.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f7386a38640 1 --2- 172.21.15.37:0/3786760141 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f738009fc50 0x7f7380139ec0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f738009f4d0 tx=0x7f737401c820 comp rx=0 tx=0).stop 2024-04-26T00:35:13.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f7386a38640 1 -- 172.21.15.37:0/3786760141 shutdown_connections 2024-04-26T00:35:13.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f7386a38640 1 --2- 172.21.15.37:0/3786760141 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f738009fc50 0x7f7380139ec0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:13.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f7386a38640 1 -- 172.21.15.37:0/3786760141 >> 172.21.15.37:0/3786760141 conn(0x7f73800a1d50 msgr2=0x7f73800a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:13.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f7386a38640 1 -- 172.21.15.37:0/3786760141 shutdown_connections 2024-04-26T00:35:13.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f7386a38640 1 -- 172.21.15.37:0/3786760141 wait complete. 2024-04-26T00:35:13.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f7386a38640 10 monclient: build_initial_monmap 2024-04-26T00:35:13.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f7386a38640 10 monclient: monmap: 2024-04-26T00:35:13.741 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:13.741 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:35:13.741 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:35:13.742 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:35:13.742 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:13.742 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:35:13.742 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:35:13.742 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:35:13.742 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:35:13.742 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:13.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f7386a38640 1 Processor -- start 2024-04-26T00:35:13.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f7386a38640 1 -- start start 2024-04-26T00:35:13.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f7386a38640 10 monclient: init 2024-04-26T00:35:13.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f7386a38640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:13.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f7386a38640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:35:13.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f7386a38640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f738009fc50 0x7f738011b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:13.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f7386a38640 10 monclient(hunting): picked mon.noname-a con 0x7f738009fc50 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:35:13.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f7386a38640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73800a0300 0x7f7380138fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:13.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f7386a38640 10 monclient(hunting): picked mon.noname-c con 0x7f73800a0300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:13.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f7386a38640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f738013a400 0x7f738011c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:13.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f7386a38640 10 monclient(hunting): picked mon.noname-b con 0x7f738013a400 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:35:13.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f7386a38640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:13.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f7386a38640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f7380140ac0 con 0x7f738009fc50 2024-04-26T00:35:13.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f7386a38640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:13.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f7386a38640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7380140940 con 0x7f738013a400 2024-04-26T00:35:13.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f7386a38640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:13.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f7386a38640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f7380140c40 con 0x7f73800a0300 2024-04-26T00:35:13.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f7386a38640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:13.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f7386a38640 10 monclient(hunting): authenticate will time out at 1895.050024s 2024-04-26T00:35:13.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f737f7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73800a0300 0x7f7380138fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:13.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f737ffff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f738009fc50 0x7f738011b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:13.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f737f7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73800a0300 0x7f7380138fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:60392/0 (socket says 172.21.15.37:60392) 2024-04-26T00:35:13.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f737f7fe640 1 -- 172.21.15.37:0/796226540 learned_addr learned my addr 172.21.15.37:0/796226540 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:13.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f737f7fe640 10 monclient(hunting): get_auth_request con 0x7f73800a0300 auth_method 0 2024-04-26T00:35:13.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f737f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:13.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f737f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:13.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f737f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:13.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f737ffff640 10 monclient(hunting): get_auth_request con 0x7f738009fc50 auth_method 0 2024-04-26T00:35:13.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f737ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:13.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f737ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:13.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f737ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:13.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f737f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:13.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f737f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:13.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f737f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:13.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f737ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:13.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f737ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:13.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f737ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:13.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f737f7fe640 10 monclient(hunting): handle_auth_done global_id 184139 payload 824 2024-04-26T00:35:13.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f7384fd5640 1 --2- 172.21.15.37:0/796226540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f738013a400 0x7f738011c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:13.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f737f7fe640 1 -- 172.21.15.37:0/796226540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f738013a400 msgr2=0x7f738011c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:13.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f737f7fe640 1 --2- 172.21.15.37:0/796226540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f738013a400 0x7f738011c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:13.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f737f7fe640 1 -- 172.21.15.37:0/796226540 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f738009fc50 msgr2=0x7f738011b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:13.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f737f7fe640 1 --2- 172.21.15.37:0/796226540 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f738009fc50 0x7f738011b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:13.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f737f7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:35:13.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f737f7fe640 1 monclient: found mon.noname-c 2024-04-26T00:35:13.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f737f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:13.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f737f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:35:13.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f737f7fe640 1 -- 172.21.15.37:0/796226540 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f738013a990 con 0x7f73800a0300 2024-04-26T00:35:13.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f7384fd5640 10 monclient: get_auth_request con 0x7f738013a400 auth_method 0 2024-04-26T00:35:13.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f737ffff640 1 --2- 172.21.15.37:0/796226540 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f738009fc50 0x7f738011b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:35:13.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.741+0000 7f7384fd5640 1 --2- 172.21.15.37:0/796226540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f738013a400 0x7f738011c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:35:13.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f737f7fe640 1 -- 172.21.15.37:0/796226540 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73800a0300 msgr2=0x7f7380138fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:13.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f737f7fe640 1 --2- 172.21.15.37:0/796226540 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73800a0300 0x7f7380138fa0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f73740b0040 tx=0x7f73740195c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:13.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f737cff9640 1 -- 172.21.15.37:0/796226540 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73740bd020 con 0x7f73800a0300 2024-04-26T00:35:13.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f737cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:13.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f737cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:35:13.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f737cff9640 10 monclient: dump: 2024-04-26T00:35:13.750 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:13.750 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:13.750 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.750 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.751 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:13.751 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:13.751 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:13.751 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:13.751 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:13.751 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:13.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f737cff9640 10 monclient: _finish_auth 0 2024-04-26T00:35:13.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f737cff9640 10 monclient: _check_auth_tickets 2024-04-26T00:35:13.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f737cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:13.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f737cff9640 1 -- 172.21.15.37:0/796226540 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73740b8880 con 0x7f73800a0300 2024-04-26T00:35:13.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f737cff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:13.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f737cff9640 1 -- 172.21.15.37:0/796226540 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73740b2040 con 0x7f73800a0300 2024-04-26T00:35:13.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f737cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:13.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f737cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:35:13.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f737cff9640 10 monclient: dump: 2024-04-26T00:35:13.752 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:13.752 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:13.752 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.752 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.752 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:13.752 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:13.752 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:13.752 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:13.752 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:13.752 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:13.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f7386a38640 5 monclient: authenticate success, global_id 184139 2024-04-26T00:35:13.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f7386a38640 10 monclient: _renew_subs 2024-04-26T00:35:13.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f737dffb640 4 set_mon_vals no callback set 2024-04-26T00:35:13.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f7386a38640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:35:13.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f7386a38640 1 -- 172.21.15.37:0/796226540 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f738009f550 con 0x7f73800a0300 2024-04-26T00:35:13.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f7386a38640 10 monclient: _renew_subs 2024-04-26T00:35:13.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f7386a38640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:35:13.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f7386a38640 1 -- 172.21.15.37:0/796226540 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7380135580 con 0x7f73800a0300 2024-04-26T00:35:13.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f7386a38640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:13.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f7386a38640 1 --2- 172.21.15.37:0/796226540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73801cd9c0 0x7f738002c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:13.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f7386a38640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:13.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f7386a38640 1 -- 172.21.15.37:0/796226540 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f738011e560 con 0x7f73801cd9c0 2024-04-26T00:35:13.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f737cff9640 1 -- 172.21.15.37:0/796226540 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f73740c0020 con 0x7f73800a0300 2024-04-26T00:35:13.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f737cff9640 1 --2- 172.21.15.37:0/796226540 >> v2:172.21.15.45:6812/2379918623 conn(0x7f736c035e80 0x7f736c038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:13.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f737cff9640 1 -- 172.21.15.37:0/796226540 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(599..599 src has 370..599) ==== 3927+0+0 (secure 0 0 0) 0x7f73740e3a70 con 0x7f73800a0300 2024-04-26T00:35:13.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f737ffff640 1 --2- 172.21.15.37:0/796226540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73801cd9c0 0x7f738002c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:13.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f737ffff640 10 monclient: get_auth_request con 0x7f73801cd9c0 auth_method 0 2024-04-26T00:35:13.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f737ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:13.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f737ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:13.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f737ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:13.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f737ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:13.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f737ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:13.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f737ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:13.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f7384fd5640 1 --2- 172.21.15.37:0/796226540 >> v2:172.21.15.45:6812/2379918623 conn(0x7f736c035e80 0x7f736c038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:13.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f7384fd5640 10 monclient: get_auth_request con 0x7f736c035e80 auth_method 0 2024-04-26T00:35:13.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f737ffff640 10 monclient(hunting): handle_auth_done global_id 174214 payload 824 2024-04-26T00:35:13.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.745+0000 7f737ffff640 1 -- 172.21.15.37:0/796226540 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f738004f510 con 0x7f73801cd9c0 2024-04-26T00:35:13.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.749+0000 7f737ffff640 1 -- 172.21.15.37:0/796226540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73801cd9c0 msgr2=0x7f738002c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:13.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.749+0000 7f737ffff640 1 --2- 172.21.15.37:0/796226540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73801cd9c0 0x7f738002c000 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f737000fae0 tx=0x7f7370004510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:13.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.749+0000 7f7384fd5640 1 --2- 172.21.15.37:0/796226540 >> v2:172.21.15.45:6812/2379918623 conn(0x7f736c035e80 0x7f736c038340 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f738011df70 tx=0x7f7378009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:13.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.749+0000 7f737cff9640 1 -- 172.21.15.37:0/796226540 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f737000ddc0 con 0x7f73801cd9c0 2024-04-26T00:35:13.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.749+0000 7f737cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:13.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.749+0000 7f737cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:35:13.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.749+0000 7f737cff9640 10 monclient: dump: 2024-04-26T00:35:13.755 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:13.755 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:13.755 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.755 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.755 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:13.755 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:13.755 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:13.755 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:13.755 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:13.755 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:13.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.749+0000 7f737cff9640 1 -- 172.21.15.37:0/796226540 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f737000ddc0 con 0x7f73801cd9c0 2024-04-26T00:35:13.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.749+0000 7f737cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:13.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.749+0000 7f737cff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:35:13.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.749+0000 7f737cff9640 1 -- 172.21.15.37:0/796226540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73801cd9c0 msgr2=0x7f738002c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:13.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.749+0000 7f737cff9640 1 --2- 172.21.15.37:0/796226540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73801cd9c0 0x7f738002c000 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f737000fae0 tx=0x7f7370004510 comp rx=0 tx=0).stop 2024-04-26T00:35:13.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.749+0000 7f737cff9640 1 -- 172.21.15.37:0/796226540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73801cd9c0 msgr2=0x7f738002c000 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:35:13.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.749+0000 7f737cff9640 1 --2- 172.21.15.37:0/796226540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73801cd9c0 0x7f738002c000 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f737000fae0 tx=0x7f7370004510 comp rx=0 tx=0).stop 2024-04-26T00:35:13.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.757+0000 7f7386a38640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:35:13.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.757+0000 7f7386a38640 1 --2- 172.21.15.37:0/796226540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73801cdda0 0x7f738002c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:13.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.757+0000 7f7386a38640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:13.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.757+0000 7f7386a38640 1 -- 172.21.15.37:0/796226540 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f738013d580 con 0x7f73801cdda0 2024-04-26T00:35:13.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.757+0000 7f737ffff640 1 --2- 172.21.15.37:0/796226540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73801cdda0 0x7f738002c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:13.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.757+0000 7f737ffff640 10 monclient: get_auth_request con 0x7f73801cdda0 auth_method 0 2024-04-26T00:35:13.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.757+0000 7f737ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:13.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.757+0000 7f737ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:13.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.757+0000 7f737ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:13.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.757+0000 7f737ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:13.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.757+0000 7f737ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:13.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.757+0000 7f737ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:13.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.757+0000 7f737ffff640 10 monclient(hunting): handle_auth_done global_id 174217 payload 824 2024-04-26T00:35:13.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.757+0000 7f737ffff640 1 -- 172.21.15.37:0/796226540 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f738009eb60 con 0x7f73801cdda0 2024-04-26T00:35:13.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.761+0000 7f737ffff640 1 -- 172.21.15.37:0/796226540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73801cdda0 msgr2=0x7f738002c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:13.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.761+0000 7f737ffff640 1 --2- 172.21.15.37:0/796226540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73801cdda0 0x7f738002c540 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f737000fae0 tx=0x7f7370003770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:13.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.761+0000 7f737cff9640 1 -- 172.21.15.37:0/796226540 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73700047f0 con 0x7f73801cdda0 2024-04-26T00:35:13.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.761+0000 7f737cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:13.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.761+0000 7f737cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:35:13.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.761+0000 7f737cff9640 10 monclient: dump: 2024-04-26T00:35:13.760 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:13.760 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:13.760 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.760 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.760 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:13.761 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:13.761 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:13.761 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:13.761 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:13.761 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:13.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.761+0000 7f737cff9640 1 -- 172.21.15.37:0/796226540 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4717 (secure 0 0 0) 0x7f73700047f0 con 0x7f73801cdda0 2024-04-26T00:35:13.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.761+0000 7f737cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:35:13.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.761+0000 7f737cff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:35:13.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.761+0000 7f737cff9640 1 -- 172.21.15.37:0/796226540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73801cdda0 msgr2=0x7f738002c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:13.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.761+0000 7f737cff9640 1 --2- 172.21.15.37:0/796226540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73801cdda0 0x7f738002c540 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f737000fae0 tx=0x7f7370003770 comp rx=0 tx=0).stop 2024-04-26T00:35:13.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.761+0000 7f737cff9640 1 -- 172.21.15.37:0/796226540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73801cdda0 msgr2=0x7f738002c540 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:35:13.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.761+0000 7f737cff9640 1 --2- 172.21.15.37:0/796226540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73801cdda0 0x7f738002c540 secure :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f737000fae0 tx=0x7f7370003770 comp rx=0 tx=0).stop 2024-04-26T00:35:13.762 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:35:13.762 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:35:13.762 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:35:13.763 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:35:13.763 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 114, 2024-04-26T00:35:13.763 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 56379, 2024-04-26T00:35:13.763 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:35:13.763 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:35:13.763 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:35:13.763 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:35:13.763 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:13.763 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 7, 2024-04-26T00:35:13.763 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:35:13.763 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:35:13.764 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:35:13.764 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:13.764 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:13.764 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:13.764 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:13.764 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:13.764 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:13.764 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:13.764 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:13.764 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:13.764 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:13.764 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:13.764 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:13.764 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:35:13.764 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:35:13.765 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:13.765 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:13.765 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:13.765 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:13.765 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:13.765 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:13.765 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:13.765 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:13.765 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:13.765 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:13.765 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:13.765 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:13.765 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:13.765 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:35:13.765 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:35:13.766 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:35:13.766 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:35:13.766 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:35:13.766 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:35:13.766 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:35:13.766 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:35:13.766 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:35:13.766 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:35:13.766 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:35:13.766 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:35:13.766 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:35:13.767 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:35:13.767 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:35:13.767 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:35:13.767 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:35:13.767 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:13.767 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:13.767 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:13.767 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:13.767 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:13.767 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:13.767 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:13.767 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:13.768 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:13.768 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:13.768 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:13.768 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:13.768 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:35:13.768 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:13.768 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:35:13.768 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:13.768 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:35:13.768 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:35:13.768 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:13.768 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:13.768 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:13.769 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:13.769 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:35:13.769 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:13.769 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:13.769 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:13.769 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:13.769 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:35:13.769 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:13.769 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:13.769 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:13.769 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:13.769 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:35:13.769 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:35:13.770 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:13.770 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:13.770 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:13.770 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:13.770 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:13.770 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:35:13.770 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:35:13.770 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:13.770 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:13.770 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:13.770 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:13.771 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:35:13.771 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:13.771 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:13.771 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:13.771 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:13.771 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:35:13.771 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:13.771 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:13.771 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:13.771 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:13.771 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:35:13.771 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:35:13.771 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:13.771 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:13.772 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:13.772 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:13.772 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:13.772 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:35:13.772 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:35:13.772 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:13.772 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:13.772 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:13.772 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:13.772 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:35:13.772 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:13.772 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:13.772 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:13.772 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:13.773 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:35:13.773 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:13.773 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:13.773 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:13.773 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:13.773 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:35:13.773 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:35:13.773 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:13.773 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:13.773 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:13.773 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:13.773 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:13.773 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:13.773 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:35:13.774 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:35:13.774 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:13.774 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:13.774 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:13.774 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:35:13.774 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:13.774 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:13.774 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:35:13.774 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:13.774 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:13.774 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:13.774 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:35:13.774 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:13.774 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:13.775 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:35:13.775 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:13.775 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:13.775 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:13.775 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:35:13.775 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:13.775 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:13.775 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:13.775 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:35:13.775 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:35:13.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.761+0000 7f7386a38640 1 -- 172.21.15.37:0/796226540 >> v2:172.21.15.45:6812/2379918623 conn(0x7f736c035e80 msgr2=0x7f736c038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:13.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.761+0000 7f7386a38640 1 --2- 172.21.15.37:0/796226540 >> v2:172.21.15.45:6812/2379918623 conn(0x7f736c035e80 0x7f736c038340 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f738011df70 tx=0x7f7378009040 comp rx=0 tx=0).stop 2024-04-26T00:35:13.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.761+0000 7f7386a38640 10 monclient: shutdown 2024-04-26T00:35:13.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.761+0000 7f7386a38640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:13.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.761+0000 7f7386a38640 1 -- 172.21.15.37:0/796226540 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73800a0300 msgr2=0x7f7380138fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:13.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.761+0000 7f7386a38640 1 --2- 172.21.15.37:0/796226540 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73800a0300 0x7f7380138fa0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f73740b0040 tx=0x7f73740195c0 comp rx=0 tx=0).stop 2024-04-26T00:35:13.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.761+0000 7f7384fd5640 1 -- 172.21.15.37:0/796226540 reap_dead start 2024-04-26T00:35:13.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.761+0000 7f7384fd5640 1 -- 172.21.15.37:0/796226540 reap_dead start 2024-04-26T00:35:13.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.761+0000 7f7386a38640 1 -- 172.21.15.37:0/796226540 shutdown_connections 2024-04-26T00:35:13.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.761+0000 7f7386a38640 1 -- 172.21.15.37:0/796226540 >> 172.21.15.37:0/796226540 conn(0x7f73800a1d50 msgr2=0x7f738013c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:13.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.761+0000 7f7386a38640 1 -- 172.21.15.37:0/796226540 shutdown_connections 2024-04-26T00:35:13.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.761+0000 7f7386a38640 1 -- 172.21.15.37:0/796226540 wait complete. 2024-04-26T00:35:13.777 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:35:13.778 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-04-26T00:35:13.778 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-26T00:35:13.778 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-04-26T00:35:13.778 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-04-26T00:35:13.778 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-26T00:35:13.778 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T00:35:13.796 INFO:tasks.ceph.mon.b:Stopped 2024-04-26T00:35:13.796 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-26T00:35:13.796 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:35:13.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f752ecac640 10 monclient: get_monmap_and_config 2024-04-26T00:35:13.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f752ecac640 10 monclient: build_initial_monmap 2024-04-26T00:35:13.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f752ecac640 10 monclient: monmap: 2024-04-26T00:35:13.911 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:13.911 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:13.911 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:35:13.917653+0000 2024-04-26T00:35:13.911 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:35:13.917653+0000 2024-04-26T00:35:13.911 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:13.911 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:13.911 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:35:13.912 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:35:13.912 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:35:13.912 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:35:13.912 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:35:13.912 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:13.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f752ecac640 1 Processor -- start 2024-04-26T00:35:13.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f752ecac640 1 -- start start 2024-04-26T00:35:13.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f752ecac640 10 monclient: init 2024-04-26T00:35:13.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f752ecac640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:13.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f752ecac640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-04-26T00:35:13.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f752ecac640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f75280a0260 0x7f75280a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:13.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f752ecac640 10 monclient(hunting): picked mon.noname-b con 0x7f75280a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:13.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f752ecac640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7528134b30 addr v1:172.21.15.37:6789/0 2024-04-26T00:35:13.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f752ecac640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7528135660 0x7f752813ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:13.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f752ecac640 10 monclient(hunting): picked mon.noname-c con 0x7f7528135660 addr v2:172.21.15.45:3300/0 2024-04-26T00:35:13.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f752ecac640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75280a1130 con 0x7f7528134b30 2024-04-26T00:35:13.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f752ecac640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:13.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f752ecac640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f752809ec10 con 0x7f7528135660 2024-04-26T00:35:13.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f752ecac640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:13.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f752ecac640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f752809f0f0 con 0x7f75280a0260 2024-04-26T00:35:13.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f752ecac640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:13.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f752ecac640 10 monclient(hunting): authenticate will time out at 1895.223365s 2024-04-26T00:35:13.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f7527fff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f7528134b30 0x7f7528134f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:36056/0 (socket says 172.21.15.37:36056) 2024-04-26T00:35:13.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f752ca48640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f75280a0260 0x7f75280a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:13.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f7527fff640 1 -- 172.21.15.37:0/3997498478 learned_addr learned my addr 172.21.15.37:0/3997498478 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:35:13.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f752d249640 1 -- 172.21.15.37:0/3997498478 >> v2:172.21.15.45:3300/0 conn(0x7f7528135660 msgr2=0x7f752813ce70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:35:13.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f752ca48640 10 monclient(hunting): get_auth_request con 0x7f75280a0260 auth_method 0 2024-04-26T00:35:13.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f752ca48640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:13.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f752ca48640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:13.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f752ca48640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:13.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f752d249640 1 --2- 172.21.15.37:0/3997498478 >> v2:172.21.15.45:3300/0 conn(0x7f7528135660 0x7f752813ce70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:35:13.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f752ca48640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:13.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f752ca48640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:13.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f752ca48640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:13.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f752ca48640 10 monclient(hunting): handle_auth_done global_id 184142 payload 274 2024-04-26T00:35:13.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f752ca48640 1 -- 172.21.15.37:0/3997498478 >> v2:172.21.15.45:3300/0 conn(0x7f7528135660 msgr2=0x7f752813ce70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:35:13.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f752ca48640 1 --2- 172.21.15.37:0/3997498478 >> v2:172.21.15.45:3300/0 conn(0x7f7528135660 0x7f752813ce70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:13.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f752ca48640 1 -- 172.21.15.37:0/3997498478 >> v1:172.21.15.37:6789/0 conn(0x7f7528134b30 legacy=0x7f7528134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:13.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f752ca48640 10 monclient: _finish_hunting 0 2024-04-26T00:35:13.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f752ca48640 1 monclient: found mon.noname-b 2024-04-26T00:35:13.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f752ca48640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:13.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f752ca48640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:35:13.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.913+0000 7f752ca48640 1 -- 172.21.15.37:0/3997498478 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f752806cff0 con 0x7f75280a0260 2024-04-26T00:35:13.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.921+0000 7f752ca48640 1 -- 172.21.15.37:0/3997498478 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f75280a0260 msgr2=0x7f75280a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:13.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.921+0000 7f752ca48640 1 --2- 172.21.15.37:0/3997498478 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f75280a0260 0x7f75280a0640 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f751c0097c0 tx=0x7f751c01c160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ede6ed59f04a1b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:13.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.921+0000 7f75277fe640 1 -- 172.21.15.37:0/3997498478 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751c0b1070 con 0x7f75280a0260 2024-04-26T00:35:13.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.921+0000 7f75277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:13.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.921+0000 7f75277fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:35:13.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.921+0000 7f75277fe640 10 monclient: dump: 2024-04-26T00:35:13.921 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:13.921 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:13.921 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.921 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.921 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:13.921 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:13.921 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:13.921 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:13.921 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:13.921 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:13.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.921+0000 7f75277fe640 10 monclient: _finish_auth 0 2024-04-26T00:35:13.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.921+0000 7f75277fe640 10 monclient: _check_auth_tickets 2024-04-26T00:35:13.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.921+0000 7f75277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:13.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.921+0000 7f75277fe640 1 -- 172.21.15.37:0/3997498478 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f751c0ac440 con 0x7f75280a0260 2024-04-26T00:35:13.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.921+0000 7f752ecac640 5 monclient: authenticate success, global_id 184142 2024-04-26T00:35:13.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.921+0000 7f752ecac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:35:13.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.921+0000 7f75277fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:13.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.921+0000 7f752ecac640 10 monclient: get_monmap_and_config success 2024-04-26T00:35:13.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.921+0000 7f752ecac640 4 set_mon_vals no callback set 2024-04-26T00:35:13.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.921+0000 7f75277fe640 1 -- 172.21.15.37:0/3997498478 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751c0ac810 con 0x7f75280a0260 2024-04-26T00:35:13.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.921+0000 7f75277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:13.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.921+0000 7f75277fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:35:13.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.921+0000 7f75277fe640 10 monclient: dump: 2024-04-26T00:35:13.923 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:13.923 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:13.923 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.923 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.923 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:13.924 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:13.924 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:13.924 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:13.924 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:13.924 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:13.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.921+0000 7f752ecac640 10 monclient: shutdown 2024-04-26T00:35:13.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.921+0000 7f752ecac640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:13.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.921+0000 7f752ecac640 1 -- 172.21.15.37:0/3997498478 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f75280a0260 msgr2=0x7f75280a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:13.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.921+0000 7f752ecac640 1 --2- 172.21.15.37:0/3997498478 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f75280a0260 0x7f75280a0640 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f751c0097c0 tx=0x7f751c01c160 comp rx=0 tx=0).stop 2024-04-26T00:35:13.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.921+0000 7f752ecac640 1 -- 172.21.15.37:0/3997498478 shutdown_connections 2024-04-26T00:35:13.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.921+0000 7f752ecac640 1 --2- 172.21.15.37:0/3997498478 >> v2:172.21.15.45:3300/0 conn(0x7f7528135660 0x7f752813ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:13.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.921+0000 7f752ecac640 1 --2- 172.21.15.37:0/3997498478 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f75280a0260 0x7f75280a0640 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:13.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.921+0000 7f752ecac640 1 -- 172.21.15.37:0/3997498478 >> 172.21.15.37:0/3997498478 conn(0x7f75280a1ad0 msgr2=0x7f75280a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:13.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.921+0000 7f752ecac640 1 -- 172.21.15.37:0/3997498478 shutdown_connections 2024-04-26T00:35:13.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.921+0000 7f752ecac640 1 -- 172.21.15.37:0/3997498478 wait complete. 2024-04-26T00:35:13.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.921+0000 7f752ecac640 10 monclient: build_initial_monmap 2024-04-26T00:35:13.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.921+0000 7f752ecac640 10 monclient: monmap: 2024-04-26T00:35:13.933 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:13.933 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:35:13.933 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:35:13.933 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:35:13.934 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:13.934 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:35:13.934 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:35:13.934 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:35:13.934 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:35:13.934 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:13.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f752ecac640 1 Processor -- start 2024-04-26T00:35:13.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f752ecac640 1 -- start start 2024-04-26T00:35:13.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f752ecac640 10 monclient: init 2024-04-26T00:35:13.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f752ecac640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:13.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f752ecac640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:35:13.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f752ecac640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75280a0260 0x7f75281ccc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:13.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f752ecac640 10 monclient(hunting): picked mon.noname-b con 0x7f75280a0260 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:35:13.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f752ecac640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7528134b30 0x7f75281319b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:13.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f752ecac640 10 monclient(hunting): picked mon.noname-a con 0x7f7528134b30 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:35:13.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f752ecac640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7528135660 0x7f7528139af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:13.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f752ecac640 10 monclient(hunting): picked mon.noname-c con 0x7f7528135660 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:13.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f752ecac640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:13.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f752ecac640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f75280a0010 con 0x7f7528134b30 2024-04-26T00:35:13.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f752ecac640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:13.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f752ecac640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f752809fe90 con 0x7f75280a0260 2024-04-26T00:35:13.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f752ecac640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:13.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f752ecac640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f752813fbd0 con 0x7f7528135660 2024-04-26T00:35:13.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f752ecac640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:13.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f752ecac640 10 monclient(hunting): authenticate will time out at 1895.232757s 2024-04-26T00:35:13.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f752ca48640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75280a0260 msgr2=0x7f75281ccc50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:35:13.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f7527fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7528134b30 0x7f75281319b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:13.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f752ca48640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75280a0260 0x7f75281ccc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:35:13.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f752d249640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7528135660 0x7f7528139af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:13.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f7527fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7528134b30 0x7f75281319b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:34492/0 (socket says 172.21.15.37:34492) 2024-04-26T00:35:13.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f7527fff640 1 -- 172.21.15.37:0/2955111624 learned_addr learned my addr 172.21.15.37:0/2955111624 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:13.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f7527fff640 10 monclient(hunting): get_auth_request con 0x7f7528134b30 auth_method 0 2024-04-26T00:35:13.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f7527fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:13.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f7527fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:13.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f7527fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:13.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f752d249640 10 monclient(hunting): get_auth_request con 0x7f7528135660 auth_method 0 2024-04-26T00:35:13.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f752d249640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:13.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f752d249640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:13.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f752d249640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:13.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f7527fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:13.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f7527fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:13.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f7527fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:13.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f752d249640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:13.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f752d249640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:13.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f752d249640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:13.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f7527fff640 10 monclient(hunting): handle_auth_done global_id 144543 payload 824 2024-04-26T00:35:13.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f7527fff640 1 -- 172.21.15.37:0/2955111624 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7528135660 msgr2=0x7f7528139af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:13.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f7527fff640 1 --2- 172.21.15.37:0/2955111624 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7528135660 0x7f7528139af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:13.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f7527fff640 1 -- 172.21.15.37:0/2955111624 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75280a0260 msgr2=0x7f75281ccc50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:35:13.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f7527fff640 1 --2- 172.21.15.37:0/2955111624 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75280a0260 0x7f75281ccc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:13.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f7527fff640 10 monclient: _finish_hunting 0 2024-04-26T00:35:13.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f7527fff640 1 monclient: found mon.noname-a 2024-04-26T00:35:13.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f7527fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:13.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f7527fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:35:13.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f7527fff640 1 -- 172.21.15.37:0/2955111624 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f752811c340 con 0x7f7528134b30 2024-04-26T00:35:13.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f7527fff640 1 -- 172.21.15.37:0/2955111624 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7528134b30 msgr2=0x7f75281319b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:13.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f7527fff640 1 --2- 172.21.15.37:0/2955111624 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7528134b30 0x7f75281319b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f751800cc10 tx=0x7f751800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:13.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f7527fff640 1 --2- 172.21.15.37:0/2955111624 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7528134b30 0x7f75281319b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f751800cc10 tx=0x7f751800a430 comp rx=0 tx=0).queue_received will delay after 1.71409e+09 on 0x7f7518012400 mon_map magic: 0 2024-04-26T00:35:13.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f75257fa640 1 -- 172.21.15.37:0/2955111624 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7518012400 con 0x7f7528134b30 2024-04-26T00:35:13.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f75257fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:13.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f75257fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:35:13.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f75257fa640 10 monclient: dump: 2024-04-26T00:35:13.941 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:13.941 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:13.942 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.942 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.942 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:13.942 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:13.942 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:13.942 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:13.942 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:13.943 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:13.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f75257fa640 10 monclient: _finish_auth 0 2024-04-26T00:35:13.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f75257fa640 10 monclient: _check_auth_tickets 2024-04-26T00:35:13.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f75257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:13.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f75257fa640 1 -- 172.21.15.37:0/2955111624 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75180125e0 con 0x7f7528134b30 2024-04-26T00:35:13.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f752ecac640 5 monclient: authenticate success, global_id 144543 2024-04-26T00:35:13.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f752ecac640 10 monclient: _renew_subs 2024-04-26T00:35:13.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f752ecac640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:13.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f752ecac640 1 -- 172.21.15.37:0/2955111624 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75280729d0 con 0x7f7528134b30 2024-04-26T00:35:13.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f75257fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:13.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f7526ffd640 4 set_mon_vals no callback set 2024-04-26T00:35:13.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f752ecac640 10 monclient: _renew_subs 2024-04-26T00:35:13.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f752ecac640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:13.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.925+0000 7f752ecac640 1 -- 172.21.15.37:0/2955111624 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f752811c060 con 0x7f7528134b30 2024-04-26T00:35:13.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.929+0000 7f75257fa640 1 -- 172.21.15.37:0/2955111624 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75180129d0 con 0x7f7528134b30 2024-04-26T00:35:13.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.929+0000 7f75257fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:13.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.929+0000 7f75257fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:13.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.929+0000 7f75257fa640 10 monclient: dump: 2024-04-26T00:35:13.945 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:13.945 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:13.945 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.945 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:13.945 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:13.945 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:13.945 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:13.945 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:13.945 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:13.945 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:13.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.929+0000 7f75257fa640 1 -- 172.21.15.37:0/2955111624 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f7518012bf0 con 0x7f7528134b30 2024-04-26T00:35:13.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.929+0000 7f752ecac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:13.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.929+0000 7f752ecac640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:13.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.929+0000 7f752ecac640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:13.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.929+0000 7f752ecac640 1 -- 172.21.15.37:0/2955111624 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7528066f40 con 0x7f7528134b30 2024-04-26T00:35:13.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.929+0000 7f75257fa640 1 --2- 172.21.15.37:0/2955111624 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7514035e30 0x7f75140382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:13.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.929+0000 7f752ca48640 1 --2- 172.21.15.37:0/2955111624 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7514035e30 0x7f75140382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:13.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.929+0000 7f752ca48640 10 monclient: get_auth_request con 0x7f7514035e30 auth_method 0 2024-04-26T00:35:13.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.933+0000 7f752ca48640 1 --2- 172.21.15.37:0/2955111624 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7514035e30 0x7f75140382f0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f751c0ab000 tx=0x7f751c0a6070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:13.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:13.933+0000 7f75257fa640 1 -- 172.21.15.37:0/2955111624 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(599..599 src has 370..599) ==== 3927+0+0 (secure 0 0 0) 0x7f7518049260 con 0x7f7528134b30 2024-04-26T00:35:14.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:14.549+0000 7f75257fa640 1 -- 172.21.15.37:0/2955111624 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f751801a360 con 0x7f7528134b30 2024-04-26T00:35:14.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:14.549+0000 7f75257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:14.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:14.549+0000 7f75257fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:35:14.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:14.777+0000 7f752ecac640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:35:14.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:14.777+0000 7f752ecac640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:35:14.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:14.777+0000 7f752ecac640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:14.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:14.777+0000 7f752ecac640 1 -- 172.21.15.37:0/2955111624 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f752811bb60 con 0x7f7528134b30 2024-04-26T00:35:14.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:14.777+0000 7f75257fa640 1 -- 172.21.15.37:0/2955111624 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f7518047fb0 con 0x7f7528134b30 2024-04-26T00:35:14.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:14.777+0000 7f75257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:35:14.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:14.777+0000 7f75257fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:35:14.777 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":114,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:35:14.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:14.781+0000 7f752ecac640 1 -- 172.21.15.37:0/2955111624 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7514035e30 msgr2=0x7f75140382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:14.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:14.781+0000 7f752ecac640 1 --2- 172.21.15.37:0/2955111624 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7514035e30 0x7f75140382f0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f751c0ab000 tx=0x7f751c0a6070 comp rx=0 tx=0).stop 2024-04-26T00:35:14.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:14.781+0000 7f752ecac640 10 monclient: shutdown 2024-04-26T00:35:14.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:14.781+0000 7f752ecac640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:14.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:14.781+0000 7f752ecac640 1 -- 172.21.15.37:0/2955111624 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7528134b30 msgr2=0x7f75281319b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:14.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:14.781+0000 7f752ecac640 1 --2- 172.21.15.37:0/2955111624 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7528134b30 0x7f75281319b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f751800cc10 tx=0x7f751800a430 comp rx=0 tx=0).stop 2024-04-26T00:35:14.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:14.781+0000 7f752ecac640 1 -- 172.21.15.37:0/2955111624 shutdown_connections 2024-04-26T00:35:14.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:14.781+0000 7f752ecac640 1 --2- 172.21.15.37:0/2955111624 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7514035e30 0x7f75140382f0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:14.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:14.781+0000 7f752ecac640 1 --2- 172.21.15.37:0/2955111624 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7528135660 0x7f7528139af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:14.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:14.781+0000 7f752ecac640 1 --2- 172.21.15.37:0/2955111624 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7528134b30 0x7f75281319b0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:14.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:14.781+0000 7f752ecac640 1 --2- 172.21.15.37:0/2955111624 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f75280a0260 0x7f75281ccc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:14.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:14.781+0000 7f752ecac640 1 -- 172.21.15.37:0/2955111624 >> 172.21.15.37:0/2955111624 conn(0x7f75280a1ad0 msgr2=0x7f7528132df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:14.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:14.781+0000 7f752ecac640 1 -- 172.21.15.37:0/2955111624 shutdown_connections 2024-04-26T00:35:14.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:14.781+0000 7f752ecac640 1 -- 172.21.15.37:0/2955111624 wait complete. 2024-04-26T00:35:17.797 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:35:17.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.933+0000 7f8fc210e640 10 monclient: get_monmap_and_config 2024-04-26T00:35:17.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.933+0000 7f8fc210e640 10 monclient: build_initial_monmap 2024-04-26T00:35:17.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.933+0000 7f8fc210e640 10 monclient: monmap: 2024-04-26T00:35:17.931 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:17.931 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:17.931 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:35:17.937260+0000 2024-04-26T00:35:17.931 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:35:17.937260+0000 2024-04-26T00:35:17.931 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:17.931 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:17.931 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:35:17.931 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:35:17.931 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:35:17.932 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:35:17.932 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:35:17.932 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:17.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.933+0000 7f8fc210e640 1 Processor -- start 2024-04-26T00:35:17.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.933+0000 7f8fc210e640 1 -- start start 2024-04-26T00:35:17.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.933+0000 7f8fc210e640 10 monclient: init 2024-04-26T00:35:17.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.933+0000 7f8fc210e640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:17.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.933+0000 7f8fc210e640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-04-26T00:35:17.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.933+0000 7f8fc210e640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f8fbc0a0260 0x7f8fbc0a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:17.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.933+0000 7f8fc210e640 10 monclient(hunting): picked mon.noname-a con 0x7f8fbc0a0260 addr v2:172.21.15.37:3300/0 2024-04-26T00:35:17.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.933+0000 7f8fc210e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8fbc134b30 addr v1:172.21.15.37:6789/0 2024-04-26T00:35:17.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.933+0000 7f8fc210e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8fbc135660 addr v1:172.21.15.45:6789/0 2024-04-26T00:35:17.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.933+0000 7f8fc210e640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fbc0a1130 con 0x7f8fbc134b30 2024-04-26T00:35:17.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.933+0000 7f8fc210e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fbc06cff0 con 0x7f8fbc135660 2024-04-26T00:35:17.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.933+0000 7f8fc210e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:17.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.933+0000 7f8fc210e640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f8fbc09ec10 con 0x7f8fbc0a0260 2024-04-26T00:35:17.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.933+0000 7f8fc210e640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:17.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.933+0000 7f8fc210e640 10 monclient(hunting): authenticate will time out at 1899.243066s 2024-04-26T00:35:17.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.933+0000 7f8fbbfff640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f8fbc135660 legacy=0x7f8fbc13ce70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:35:17.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.933+0000 7f8fbb7fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f8fbc0a0260 0x7f8fbc0a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:17.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.933+0000 7f8fbaffd640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f8fbc134b30 0x7f8fbc134f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:36066/0 (socket says 172.21.15.37:36066) 2024-04-26T00:35:17.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.933+0000 7f8fbaffd640 1 -- 172.21.15.37:0/2442894808 learned_addr learned my addr 172.21.15.37:0/2442894808 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:35:17.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.933+0000 7f8fbb7fe640 10 monclient(hunting): get_auth_request con 0x7f8fbc0a0260 auth_method 0 2024-04-26T00:35:17.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.933+0000 7f8fbb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:17.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.933+0000 7f8fbb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:17.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.933+0000 7f8fbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:17.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.933+0000 7f8fbb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:17.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.933+0000 7f8fbb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:17.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.933+0000 7f8fbb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:17.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.933+0000 7f8fbb7fe640 10 monclient(hunting): handle_auth_done global_id 144546 payload 274 2024-04-26T00:35:17.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.933+0000 7f8fbb7fe640 1 -- 172.21.15.37:0/2442894808 >> v1:172.21.15.45:6789/0 conn(0x7f8fbc135660 legacy=0x7f8fbc13ce70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:35:17.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.933+0000 7f8fbb7fe640 1 -- 172.21.15.37:0/2442894808 >> v1:172.21.15.37:6789/0 conn(0x7f8fbc134b30 legacy=0x7f8fbc134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:17.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.933+0000 7f8fbb7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:35:17.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.933+0000 7f8fbb7fe640 1 monclient: found mon.noname-a 2024-04-26T00:35:17.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.933+0000 7f8fbb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:17.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.933+0000 7f8fbb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:35:17.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.933+0000 7f8fbb7fe640 1 -- 172.21.15.37:0/2442894808 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fbc09bc50 con 0x7f8fbc0a0260 2024-04-26T00:35:17.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fbb7fe640 1 -- 172.21.15.37:0/2442894808 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8fbc0a0260 msgr2=0x7f8fbc0a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:17.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fbb7fe640 1 --2- 172.21.15.37:0/2442894808 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8fbc0a0260 0x7f8fbc0a0640 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f8fac0097c0 tx=0x7f8fac01c100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b413eb423f27cea1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:17.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fba7fc640 1 -- 172.21.15.37:0/2442894808 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fac0b1070 con 0x7f8fbc0a0260 2024-04-26T00:35:17.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:17.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fba7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:35:17.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fba7fc640 10 monclient: dump: 2024-04-26T00:35:17.939 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:17.939 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:17.939 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:17.939 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:17.939 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:17.939 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:17.939 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:17.939 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:17.939 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:17.939 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:17.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fba7fc640 10 monclient: _finish_auth 0 2024-04-26T00:35:17.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fba7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:35:17.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:17.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fba7fc640 1 -- 172.21.15.37:0/2442894808 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fac0ac440 con 0x7f8fbc0a0260 2024-04-26T00:35:17.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fc210e640 5 monclient: authenticate success, global_id 144546 2024-04-26T00:35:17.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fc210e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:35:17.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fba7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:17.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fc210e640 10 monclient: get_monmap_and_config success 2024-04-26T00:35:17.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fc210e640 4 set_mon_vals no callback set 2024-04-26T00:35:17.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fba7fc640 1 -- 172.21.15.37:0/2442894808 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fac0ac810 con 0x7f8fbc0a0260 2024-04-26T00:35:17.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:17.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fba7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:17.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fba7fc640 10 monclient: dump: 2024-04-26T00:35:17.940 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:17.940 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:17.941 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:17.941 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:17.941 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:17.941 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:17.941 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:17.941 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:17.941 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:17.941 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:17.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fc210e640 10 monclient: shutdown 2024-04-26T00:35:17.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fc210e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:17.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fc210e640 1 -- 172.21.15.37:0/2442894808 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8fbc0a0260 msgr2=0x7f8fbc0a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:17.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fc210e640 1 --2- 172.21.15.37:0/2442894808 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8fbc0a0260 0x7f8fbc0a0640 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f8fac0097c0 tx=0x7f8fac01c100 comp rx=0 tx=0).stop 2024-04-26T00:35:17.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fc210e640 1 -- 172.21.15.37:0/2442894808 shutdown_connections 2024-04-26T00:35:17.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fc210e640 1 --2- 172.21.15.37:0/2442894808 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8fbc0a0260 0x7f8fbc0a0640 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:17.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fc210e640 1 -- 172.21.15.37:0/2442894808 >> 172.21.15.37:0/2442894808 conn(0x7f8fbc0a1ad0 msgr2=0x7f8fbc0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:17.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fc210e640 1 -- 172.21.15.37:0/2442894808 shutdown_connections 2024-04-26T00:35:17.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fc210e640 1 -- 172.21.15.37:0/2442894808 wait complete. 2024-04-26T00:35:17.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fc210e640 10 monclient: build_initial_monmap 2024-04-26T00:35:17.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fc210e640 10 monclient: monmap: 2024-04-26T00:35:17.946 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:17.946 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:35:17.946 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:35:17.946 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:35:17.946 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:17.946 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:35:17.946 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:35:17.947 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:35:17.947 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:35:17.947 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:17.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fc210e640 1 Processor -- start 2024-04-26T00:35:17.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fc210e640 1 -- start start 2024-04-26T00:35:17.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fc210e640 10 monclient: init 2024-04-26T00:35:17.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fc210e640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:17.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fc210e640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:35:17.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fc210e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fbc0a0260 0x7f8fbc131810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:17.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fc210e640 10 monclient(hunting): picked mon.noname-b con 0x7f8fbc0a0260 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:35:17.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fc210e640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8fbc134b30 0x7f8fbc1398d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:17.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fc210e640 10 monclient(hunting): picked mon.noname-a con 0x7f8fbc134b30 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:35:17.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fc210e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8fbc135660 0x7f8fbc139e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:17.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fc210e640 10 monclient(hunting): picked mon.noname-c con 0x7f8fbc135660 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:17.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fc210e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:17.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fc210e640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f8fbc09f4c0 con 0x7f8fbc134b30 2024-04-26T00:35:17.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fc210e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:17.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fc210e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8fbc09f0f0 con 0x7f8fbc0a0260 2024-04-26T00:35:17.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fc210e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:17.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fc210e640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8fbc140db0 con 0x7f8fbc135660 2024-04-26T00:35:17.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fc210e640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:17.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fc210e640 10 monclient(hunting): authenticate will time out at 1899.250851s 2024-04-26T00:35:17.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fbbfff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8fbc135660 0x7f8fbc139e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:17.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fbaffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8fbc134b30 0x7f8fbc1398d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:17.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fbbfff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8fbc135660 0x7f8fbc139e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:60408/0 (socket says 172.21.15.37:60408) 2024-04-26T00:35:17.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fbbfff640 1 -- 172.21.15.37:0/4099415992 learned_addr learned my addr 172.21.15.37:0/4099415992 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:17.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fbbfff640 10 monclient(hunting): get_auth_request con 0x7f8fbc135660 auth_method 0 2024-04-26T00:35:17.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fbbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:17.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fbbfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:17.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:17.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fbb7fe640 1 -- 172.21.15.37:0/4099415992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fbc0a0260 msgr2=0x7f8fbc131810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:35:17.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fbb7fe640 1 --2- 172.21.15.37:0/4099415992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fbc0a0260 0x7f8fbc131810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:35:17.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fbaffd640 10 monclient(hunting): get_auth_request con 0x7f8fbc134b30 auth_method 0 2024-04-26T00:35:17.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fbaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:17.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fbaffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:17.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fbaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:17.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fbbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:17.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fbbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:17.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fbbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:17.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fbaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:17.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fbaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:17.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fbaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:17.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fbbfff640 10 monclient(hunting): handle_auth_done global_id 184148 payload 824 2024-04-26T00:35:17.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fbbfff640 1 -- 172.21.15.37:0/4099415992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fbc0a0260 msgr2=0x7f8fbc131810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:35:17.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fbbfff640 1 --2- 172.21.15.37:0/4099415992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fbc0a0260 0x7f8fbc131810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:17.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fbbfff640 1 -- 172.21.15.37:0/4099415992 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8fbc134b30 msgr2=0x7f8fbc1398d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:17.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fbbfff640 1 --2- 172.21.15.37:0/4099415992 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8fbc134b30 0x7f8fbc1398d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:17.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fbbfff640 10 monclient: _finish_hunting 0 2024-04-26T00:35:17.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fbbfff640 1 monclient: found mon.noname-c 2024-04-26T00:35:17.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fbbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:17.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fbbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:35:17.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fbbfff640 1 -- 172.21.15.37:0/4099415992 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fbc13d810 con 0x7f8fbc135660 2024-04-26T00:35:17.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.941+0000 7f8fbaffd640 1 --2- 172.21.15.37:0/4099415992 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8fbc134b30 0x7f8fbc1398d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:35:17.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.945+0000 7f8fbbfff640 1 -- 172.21.15.37:0/4099415992 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8fbc135660 msgr2=0x7f8fbc139e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:17.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.945+0000 7f8fbbfff640 1 --2- 172.21.15.37:0/4099415992 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8fbc135660 0x7f8fbc139e10 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8fb400b760 tx=0x7f8fb400bc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:17.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.945+0000 7f8fa3fff640 1 -- 172.21.15.37:0/4099415992 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fb400cb70 con 0x7f8fbc135660 2024-04-26T00:35:17.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.945+0000 7f8fa3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:17.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.945+0000 7f8fa3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:35:17.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.945+0000 7f8fa3fff640 10 monclient: dump: 2024-04-26T00:35:17.952 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:17.952 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:17.952 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:17.952 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:17.952 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:17.953 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:17.953 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:17.953 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:17.953 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:17.953 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:17.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.945+0000 7f8fa3fff640 10 monclient: _finish_auth 0 2024-04-26T00:35:17.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.945+0000 7f8fa3fff640 10 monclient: _check_auth_tickets 2024-04-26T00:35:17.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.945+0000 7f8fa3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:17.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.945+0000 7f8fc210e640 5 monclient: authenticate success, global_id 184148 2024-04-26T00:35:17.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.945+0000 7f8fa3fff640 1 -- 172.21.15.37:0/4099415992 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fb4010070 con 0x7f8fbc135660 2024-04-26T00:35:17.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.945+0000 7f8fa3fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:17.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.945+0000 7f8fa3fff640 1 -- 172.21.15.37:0/4099415992 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fb4015470 con 0x7f8fbc135660 2024-04-26T00:35:17.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.945+0000 7f8fc210e640 10 monclient: _renew_subs 2024-04-26T00:35:17.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.945+0000 7f8fc210e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:35:17.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.945+0000 7f8fc210e640 1 -- 172.21.15.37:0/4099415992 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fbc13be00 con 0x7f8fbc135660 2024-04-26T00:35:17.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.945+0000 7f8fa3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:17.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.945+0000 7f8fb9ffb640 4 set_mon_vals no callback set 2024-04-26T00:35:17.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.945+0000 7f8fa3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:35:17.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.945+0000 7f8fa3fff640 10 monclient: dump: 2024-04-26T00:35:17.954 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:17.954 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:17.954 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:17.954 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:17.954 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:17.954 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:17.954 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:17.954 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:17.954 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:17.954 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:17.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.945+0000 7f8fc210e640 10 monclient: _renew_subs 2024-04-26T00:35:17.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.945+0000 7f8fc210e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:35:17.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.945+0000 7f8fc210e640 1 -- 172.21.15.37:0/4099415992 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fbc13b520 con 0x7f8fbc135660 2024-04-26T00:35:17.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.945+0000 7f8fc210e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:17.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.945+0000 7f8fc210e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:17.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.945+0000 7f8fc210e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:35:17.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.945+0000 7f8fa3fff640 1 -- 172.21.15.37:0/4099415992 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f8fb40156d0 con 0x7f8fbc135660 2024-04-26T00:35:17.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.945+0000 7f8fc210e640 1 -- 172.21.15.37:0/4099415992 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fbc066f40 con 0x7f8fbc135660 2024-04-26T00:35:17.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.945+0000 7f8fa3fff640 1 --2- 172.21.15.37:0/4099415992 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8fa8035e80 0x7f8fa8038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:17.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.945+0000 7f8fa3fff640 1 -- 172.21.15.37:0/4099415992 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(599..599 src has 370..599) ==== 3927+0+0 (secure 0 0 0) 0x7f8fb4048960 con 0x7f8fbc135660 2024-04-26T00:35:17.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.945+0000 7f8fbb7fe640 1 --2- 172.21.15.37:0/4099415992 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8fa8035e80 0x7f8fa8038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:17.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.945+0000 7f8fbb7fe640 10 monclient: get_auth_request con 0x7f8fa8035e80 auth_method 0 2024-04-26T00:35:17.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.949+0000 7f8fbb7fe640 1 --2- 172.21.15.37:0/4099415992 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8fa8035e80 0x7f8fa8038340 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc09f690 tx=0x7f8fac008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:17.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.953+0000 7f8fa3fff640 1 -- 172.21.15.37:0/4099415992 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f8fb40476b0 con 0x7f8fbc135660 2024-04-26T00:35:17.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.953+0000 7f8fa3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:17.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:17.953+0000 7f8fa3fff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:35:18.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:18.177+0000 7f8fc210e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:35:18.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:18.177+0000 7f8fc210e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:35:18.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:18.177+0000 7f8fc210e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:35:18.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:18.177+0000 7f8fc210e640 1 -- 172.21.15.37:0/4099415992 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8fbc13d490 con 0x7f8fbc135660 2024-04-26T00:35:18.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:18.177+0000 7f8fa3fff640 1 -- 172.21.15.37:0/4099415992 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1558 (secure 0 0 0) 0x7f8fb4014790 con 0x7f8fbc135660 2024-04-26T00:35:18.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:18.177+0000 7f8fa3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:35:18.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:18.177+0000 7f8fa3fff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:35:18.176 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":114,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:35:18.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:18.181+0000 7f8fc210e640 1 -- 172.21.15.37:0/4099415992 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8fa8035e80 msgr2=0x7f8fa8038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:18.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:18.181+0000 7f8fc210e640 1 --2- 172.21.15.37:0/4099415992 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8fa8035e80 0x7f8fa8038340 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc09f690 tx=0x7f8fac008040 comp rx=0 tx=0).stop 2024-04-26T00:35:18.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:18.181+0000 7f8fc210e640 10 monclient: shutdown 2024-04-26T00:35:18.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:18.181+0000 7f8fc210e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:18.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:18.181+0000 7f8fc210e640 1 -- 172.21.15.37:0/4099415992 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8fbc135660 msgr2=0x7f8fbc139e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:18.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:18.181+0000 7f8fc210e640 1 --2- 172.21.15.37:0/4099415992 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8fbc135660 0x7f8fbc139e10 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8fb400b760 tx=0x7f8fb400bc20 comp rx=0 tx=0).stop 2024-04-26T00:35:18.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:18.181+0000 7f8fc210e640 1 -- 172.21.15.37:0/4099415992 shutdown_connections 2024-04-26T00:35:18.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:18.181+0000 7f8fc210e640 1 --2- 172.21.15.37:0/4099415992 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8fa8035e80 0x7f8fa8038340 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:18.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:18.181+0000 7f8fc210e640 1 --2- 172.21.15.37:0/4099415992 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8fbc135660 0x7f8fbc139e10 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:18.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:18.181+0000 7f8fc210e640 1 --2- 172.21.15.37:0/4099415992 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8fbc134b30 0x7f8fbc1398d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:18.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:18.181+0000 7f8fc210e640 1 --2- 172.21.15.37:0/4099415992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fbc0a0260 0x7f8fbc131810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:18.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:18.181+0000 7f8fc210e640 1 -- 172.21.15.37:0/4099415992 >> 172.21.15.37:0/4099415992 conn(0x7f8fbc0a1ad0 msgr2=0x7f8fbc0a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:18.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:18.181+0000 7f8fc210e640 1 -- 172.21.15.37:0/4099415992 shutdown_connections 2024-04-26T00:35:18.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:18.181+0000 7f8fc210e640 1 -- 172.21.15.37:0/4099415992 wait complete. 2024-04-26T00:35:21.195 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:35:21.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.329+0000 7fb662829640 10 monclient: get_monmap_and_config 2024-04-26T00:35:21.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.329+0000 7fb662829640 10 monclient: build_initial_monmap 2024-04-26T00:35:21.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.329+0000 7fb662829640 10 monclient: monmap: 2024-04-26T00:35:21.327 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:21.327 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:21.327 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:35:21.333206+0000 2024-04-26T00:35:21.327 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:35:21.333206+0000 2024-04-26T00:35:21.327 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:21.327 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:21.327 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:35:21.327 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:35:21.327 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:35:21.327 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:35:21.327 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:35:21.328 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:21.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.329+0000 7fb662829640 1 Processor -- start 2024-04-26T00:35:21.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.329+0000 7fb662829640 1 -- start start 2024-04-26T00:35:21.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.329+0000 7fb662829640 10 monclient: init 2024-04-26T00:35:21.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.329+0000 7fb662829640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:21.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.329+0000 7fb662829640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-04-26T00:35:21.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.329+0000 7fb662829640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb65c0724d0 addr v1:172.21.15.37:6789/0 2024-04-26T00:35:21.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.329+0000 7fb662829640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb65c133990 0x7fb65c13c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:21.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.329+0000 7fb662829640 10 monclient(hunting): picked mon.noname-a con 0x7fb65c133990 addr v2:172.21.15.37:3300/0 2024-04-26T00:35:21.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.329+0000 7fb662829640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb65c133da0 addr v1:172.21.15.45:6789/0 2024-04-26T00:35:21.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.329+0000 7fb662829640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb65c06e9e0 con 0x7fb65c0724d0 2024-04-26T00:35:21.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.329+0000 7fb662829640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb65c06efa0 con 0x7fb65c133da0 2024-04-26T00:35:21.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.329+0000 7fb662829640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:21.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.329+0000 7fb662829640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fb65c072fc0 con 0x7fb65c133990 2024-04-26T00:35:21.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.329+0000 7fb662829640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:21.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.329+0000 7fb662829640 10 monclient(hunting): authenticate will time out at 1902.639197s 2024-04-26T00:35:21.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.329+0000 7fb660dc6640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7fb65c133da0 legacy=0x7fb65c13e5e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:35:21.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.329+0000 7fb65b7fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb65c133990 0x7fb65c13c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:21.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.329+0000 7fb65bfff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fb65c0724d0 0x7fb65c0728b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:59088/0 (socket says 172.21.15.37:59088) 2024-04-26T00:35:21.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.329+0000 7fb65bfff640 1 -- 172.21.15.37:0/1096374315 learned_addr learned my addr 172.21.15.37:0/1096374315 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:35:21.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.329+0000 7fb65affd640 1 -- 172.21.15.37:0/1096374315 DispatchQueue::entry inject delay of 0.002000 2024-04-26T00:35:21.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.329+0000 7fb65b7fe640 10 monclient(hunting): get_auth_request con 0x7fb65c133990 auth_method 0 2024-04-26T00:35:21.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.329+0000 7fb65b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:21.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.329+0000 7fb65b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:21.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.329+0000 7fb65b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:21.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.329+0000 7fb65b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:21.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.329+0000 7fb65b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:21.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.329+0000 7fb65b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:21.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.329+0000 7fb65b7fe640 10 monclient(hunting): handle_auth_done global_id 144552 payload 274 2024-04-26T00:35:21.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.329+0000 7fb65b7fe640 1 -- 172.21.15.37:0/1096374315 >> v1:172.21.15.45:6789/0 conn(0x7fb65c133da0 legacy=0x7fb65c13e5e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:35:21.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.329+0000 7fb65b7fe640 1 -- 172.21.15.37:0/1096374315 >> v1:172.21.15.37:6789/0 conn(0x7fb65c0724d0 legacy=0x7fb65c0728b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:21.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.329+0000 7fb65b7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:35:21.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.329+0000 7fb65b7fe640 1 monclient: found mon.noname-a 2024-04-26T00:35:21.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.329+0000 7fb65b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:21.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.329+0000 7fb65b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:35:21.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.329+0000 7fb65b7fe640 1 -- 172.21.15.37:0/1096374315 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb65c09f9e0 con 0x7fb65c133990 2024-04-26T00:35:21.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.333+0000 7fb65b7fe640 1 -- 172.21.15.37:0/1096374315 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb65c133990 msgr2=0x7fb65c13c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:21.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.333+0000 7fb65b7fe640 1 --2- 172.21.15.37:0/1096374315 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb65c133990 0x7fb65c13c090 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb644009d90 tx=0x7fb64401c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a75f41a82048924 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:21.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb65affd640 1 -- 172.21.15.37:0/1096374315 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6440ac070 con 0x7fb65c133990 2024-04-26T00:35:21.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb65affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:21.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb65affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:35:21.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb65affd640 10 monclient: dump: 2024-04-26T00:35:21.337 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:21.337 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:21.337 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:21.337 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:21.337 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:21.337 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:21.337 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:21.337 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:21.338 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:21.338 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:21.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb65affd640 10 monclient: _finish_auth 0 2024-04-26T00:35:21.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb65affd640 10 monclient: _check_auth_tickets 2024-04-26T00:35:21.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb65affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:21.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb65affd640 1 -- 172.21.15.37:0/1096374315 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6440a6760 con 0x7fb65c133990 2024-04-26T00:35:21.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb65affd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:21.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb65affd640 1 -- 172.21.15.37:0/1096374315 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6440a6b30 con 0x7fb65c133990 2024-04-26T00:35:21.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb662829640 5 monclient: authenticate success, global_id 144552 2024-04-26T00:35:21.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb662829640 10 monclient: get_monmap_and_config success 2024-04-26T00:35:21.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb662829640 4 set_mon_vals no callback set 2024-04-26T00:35:21.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb65affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:21.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb65affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:21.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb65affd640 10 monclient: dump: 2024-04-26T00:35:21.339 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:21.339 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:21.339 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:21.339 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:21.339 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:21.339 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:21.339 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:21.339 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:21.339 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:21.339 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:21.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb662829640 10 monclient: shutdown 2024-04-26T00:35:21.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb662829640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:21.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb662829640 1 -- 172.21.15.37:0/1096374315 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb65c133990 msgr2=0x7fb65c13c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:21.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb662829640 1 --2- 172.21.15.37:0/1096374315 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb65c133990 0x7fb65c13c090 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb644009d90 tx=0x7fb64401c7b0 comp rx=0 tx=0).stop 2024-04-26T00:35:21.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb662829640 1 -- 172.21.15.37:0/1096374315 shutdown_connections 2024-04-26T00:35:21.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb662829640 1 --2- 172.21.15.37:0/1096374315 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb65c133990 0x7fb65c13c090 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:21.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb662829640 1 -- 172.21.15.37:0/1096374315 >> 172.21.15.37:0/1096374315 conn(0x7fb65c0a24c0 msgr2=0x7fb65c0a28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:21.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb662829640 1 -- 172.21.15.37:0/1096374315 shutdown_connections 2024-04-26T00:35:21.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb662829640 1 -- 172.21.15.37:0/1096374315 wait complete. 2024-04-26T00:35:21.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb662829640 10 monclient: build_initial_monmap 2024-04-26T00:35:21.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb662829640 10 monclient: monmap: 2024-04-26T00:35:21.340 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:21.340 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:35:21.340 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:35:21.340 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:35:21.341 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:21.341 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:35:21.341 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:35:21.341 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:35:21.341 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:35:21.341 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:21.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb662829640 1 Processor -- start 2024-04-26T00:35:21.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb662829640 1 -- start start 2024-04-26T00:35:21.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb662829640 10 monclient: init 2024-04-26T00:35:21.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb662829640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:21.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb662829640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:35:21.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb662829640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb65c0724d0 0x7fb65c1cf560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:21.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb662829640 10 monclient(hunting): picked mon.noname-b con 0x7fb65c0724d0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:35:21.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb662829640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb65c133990 0x7fb65c13df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:21.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb662829640 10 monclient(hunting): picked mon.noname-c con 0x7fb65c133990 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:21.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb662829640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb65c133da0 0x7fb65c13e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:21.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb662829640 10 monclient(hunting): picked mon.noname-a con 0x7fb65c133da0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:35:21.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb662829640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:21.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb662829640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb65c142ff0 con 0x7fb65c133da0 2024-04-26T00:35:21.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb662829640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:21.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb662829640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb65c142e70 con 0x7fb65c0724d0 2024-04-26T00:35:21.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb662829640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:21.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb662829640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb65c143170 con 0x7fb65c133990 2024-04-26T00:35:21.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb662829640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:21.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb662829640 10 monclient(hunting): authenticate will time out at 1902.646333s 2024-04-26T00:35:21.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb65b7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb65c133990 0x7fb65c13df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:21.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb65bfff640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb65c0724d0 msgr2=0x7fb65c1cf560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:35:21.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb660dc6640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb65c133da0 0x7fb65c13e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:21.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb65bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb65c0724d0 0x7fb65c1cf560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:35:21.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb65b7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb65c133990 0x7fb65c13df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:59260/0 (socket says 172.21.15.37:59260) 2024-04-26T00:35:21.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb65b7fe640 1 -- 172.21.15.37:0/3766487118 learned_addr learned my addr 172.21.15.37:0/3766487118 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:21.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb65b7fe640 10 monclient(hunting): get_auth_request con 0x7fb65c133990 auth_method 0 2024-04-26T00:35:21.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb65b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:21.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb65b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:21.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb65b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:21.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb660dc6640 10 monclient(hunting): get_auth_request con 0x7fb65c133da0 auth_method 0 2024-04-26T00:35:21.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb660dc6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:21.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb660dc6640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:21.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb660dc6640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:21.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb660dc6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:21.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb660dc6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:21.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb660dc6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:21.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb65b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:21.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb65b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:21.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb65b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:21.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb660dc6640 10 monclient(hunting): handle_auth_done global_id 144555 payload 824 2024-04-26T00:35:21.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb660dc6640 1 -- 172.21.15.37:0/3766487118 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb65c133990 msgr2=0x7fb65c13df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:21.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb660dc6640 1 --2- 172.21.15.37:0/3766487118 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb65c133990 0x7fb65c13df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:21.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb660dc6640 1 -- 172.21.15.37:0/3766487118 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb65c0724d0 msgr2=0x7fb65c1cf560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:35:21.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb660dc6640 1 --2- 172.21.15.37:0/3766487118 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb65c0724d0 0x7fb65c1cf560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:21.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb660dc6640 10 monclient: _finish_hunting 0 2024-04-26T00:35:21.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb660dc6640 1 monclient: found mon.noname-a 2024-04-26T00:35:21.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb660dc6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:21.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb660dc6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:35:21.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb660dc6640 1 -- 172.21.15.37:0/3766487118 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb65c09fcc0 con 0x7fb65c133da0 2024-04-26T00:35:21.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.337+0000 7fb65b7fe640 1 --2- 172.21.15.37:0/3766487118 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb65c133990 0x7fb65c13df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:35:21.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.341+0000 7fb660dc6640 1 -- 172.21.15.37:0/3766487118 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb65c133da0 msgr2=0x7fb65c13e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:21.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.341+0000 7fb660dc6640 1 --2- 172.21.15.37:0/3766487118 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb65c133da0 0x7fb65c13e4a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb65400b3c0 tx=0x7fb65400b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:21.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.341+0000 7fb658ff9640 1 -- 172.21.15.37:0/3766487118 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb654016070 con 0x7fb65c133da0 2024-04-26T00:35:21.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.341+0000 7fb658ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:21.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.341+0000 7fb658ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:35:21.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.341+0000 7fb658ff9640 10 monclient: dump: 2024-04-26T00:35:21.347 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:21.347 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:21.347 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:21.348 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:21.348 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:21.348 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:21.348 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:21.348 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:21.348 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:21.348 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:21.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.341+0000 7fb658ff9640 10 monclient: _finish_auth 0 2024-04-26T00:35:21.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.341+0000 7fb658ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:35:21.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.341+0000 7fb658ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:21.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.341+0000 7fb658ff9640 1 -- 172.21.15.37:0/3766487118 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb654005070 con 0x7fb65c133da0 2024-04-26T00:35:21.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.341+0000 7fb658ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:21.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.341+0000 7fb658ff9640 1 -- 172.21.15.37:0/3766487118 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb654005460 con 0x7fb65c133da0 2024-04-26T00:35:21.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.341+0000 7fb662829640 5 monclient: authenticate success, global_id 144555 2024-04-26T00:35:21.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.341+0000 7fb662829640 10 monclient: _renew_subs 2024-04-26T00:35:21.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.341+0000 7fb662829640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:21.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.341+0000 7fb662829640 1 -- 172.21.15.37:0/3766487118 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb65c13c470 con 0x7fb65c133da0 2024-04-26T00:35:21.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.341+0000 7fb658ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:21.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.341+0000 7fb658ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:21.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.341+0000 7fb658ff9640 10 monclient: dump: 2024-04-26T00:35:21.349 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:21.349 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:21.349 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:21.349 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:21.349 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:21.349 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:21.349 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:21.349 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:21.349 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:21.350 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:21.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.341+0000 7fb65a7fc640 4 set_mon_vals no callback set 2024-04-26T00:35:21.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.341+0000 7fb662829640 10 monclient: _renew_subs 2024-04-26T00:35:21.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.341+0000 7fb662829640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:21.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.341+0000 7fb662829640 1 -- 172.21.15.37:0/3766487118 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb65c1cfaa0 con 0x7fb65c133da0 2024-04-26T00:35:21.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.341+0000 7fb6527fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:21.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.341+0000 7fb6527fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:21.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.341+0000 7fb6527fc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:21.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.341+0000 7fb6527fc640 1 -- 172.21.15.37:0/3766487118 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb65c066f40 con 0x7fb65c133da0 2024-04-26T00:35:21.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.341+0000 7fb658ff9640 1 -- 172.21.15.37:0/3766487118 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb654019070 con 0x7fb65c133da0 2024-04-26T00:35:21.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.341+0000 7fb658ff9640 1 --2- 172.21.15.37:0/3766487118 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb63c035e80 0x7fb63c038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:21.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.341+0000 7fb658ff9640 1 -- 172.21.15.37:0/3766487118 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(599..599 src has 370..599) ==== 3927+0+0 (secure 0 0 0) 0x7fb654047cc0 con 0x7fb65c133da0 2024-04-26T00:35:21.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.341+0000 7fb65bfff640 1 --2- 172.21.15.37:0/3766487118 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb63c035e80 0x7fb63c038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:21.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.341+0000 7fb65bfff640 10 monclient: get_auth_request con 0x7fb63c035e80 auth_method 0 2024-04-26T00:35:21.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.345+0000 7fb65bfff640 1 --2- 172.21.15.37:0/3766487118 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb63c035e80 0x7fb63c038340 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fb64c004820 tx=0x7fb64c007b30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:21.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.345+0000 7fb658ff9640 1 -- 172.21.15.37:0/3766487118 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fb65c09fcc0 con 0x7fb65c133da0 2024-04-26T00:35:21.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.345+0000 7fb658ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:21.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.345+0000 7fb658ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:35:21.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.565+0000 7fb6527fc640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:35:21.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.565+0000 7fb6527fc640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:35:21.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.565+0000 7fb6527fc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:21.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.565+0000 7fb6527fc640 1 -- 172.21.15.37:0/3766487118 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb65c13b020 con 0x7fb65c133da0 2024-04-26T00:35:21.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.565+0000 7fb658ff9640 1 -- 172.21.15.37:0/3766487118 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1558 (secure 0 0 0) 0x7fb6540182c0 con 0x7fb65c133da0 2024-04-26T00:35:21.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.565+0000 7fb658ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:35:21.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.565+0000 7fb658ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:35:21.563 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":114,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:35:21.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.565+0000 7fb6527fc640 1 -- 172.21.15.37:0/3766487118 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb63c035e80 msgr2=0x7fb63c038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:21.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.565+0000 7fb6527fc640 1 --2- 172.21.15.37:0/3766487118 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb63c035e80 0x7fb63c038340 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fb64c004820 tx=0x7fb64c007b30 comp rx=0 tx=0).stop 2024-04-26T00:35:21.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.565+0000 7fb6527fc640 10 monclient: shutdown 2024-04-26T00:35:21.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.565+0000 7fb6527fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:21.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.565+0000 7fb6527fc640 1 -- 172.21.15.37:0/3766487118 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb65c133da0 msgr2=0x7fb65c13e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:21.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.565+0000 7fb6527fc640 1 --2- 172.21.15.37:0/3766487118 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb65c133da0 0x7fb65c13e4a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb65400b3c0 tx=0x7fb65400b880 comp rx=0 tx=0).stop 2024-04-26T00:35:21.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.565+0000 7fb6527fc640 1 -- 172.21.15.37:0/3766487118 shutdown_connections 2024-04-26T00:35:21.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.565+0000 7fb6527fc640 1 --2- 172.21.15.37:0/3766487118 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb63c035e80 0x7fb63c038340 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:21.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.565+0000 7fb6527fc640 1 --2- 172.21.15.37:0/3766487118 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb65c133da0 0x7fb65c13e4a0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:21.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.565+0000 7fb6527fc640 1 --2- 172.21.15.37:0/3766487118 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb65c133990 0x7fb65c13df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:21.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.565+0000 7fb6527fc640 1 --2- 172.21.15.37:0/3766487118 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb65c0724d0 0x7fb65c1cf560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:21.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.565+0000 7fb6527fc640 1 -- 172.21.15.37:0/3766487118 >> 172.21.15.37:0/3766487118 conn(0x7fb65c0a24c0 msgr2=0x7fb65c132100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:21.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.569+0000 7fb6527fc640 1 -- 172.21.15.37:0/3766487118 shutdown_connections 2024-04-26T00:35:21.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:21.569+0000 7fb6527fc640 1 -- 172.21.15.37:0/3766487118 wait complete. 2024-04-26T00:35:24.582 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:35:24.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.717+0000 7fce6e1b1640 10 monclient: get_monmap_and_config 2024-04-26T00:35:24.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.717+0000 7fce6e1b1640 10 monclient: build_initial_monmap 2024-04-26T00:35:24.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.717+0000 7fce6e1b1640 10 monclient: monmap: 2024-04-26T00:35:24.715 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:24.715 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:24.715 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:35:24.721284+0000 2024-04-26T00:35:24.715 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:35:24.721284+0000 2024-04-26T00:35:24.715 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:24.715 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:24.715 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:35:24.715 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:35:24.716 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:35:24.716 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:35:24.716 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:35:24.716 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:24.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.717+0000 7fce6e1b1640 1 Processor -- start 2024-04-26T00:35:24.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.717+0000 7fce6e1b1640 1 -- start start 2024-04-26T00:35:24.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.717+0000 7fce6e1b1640 10 monclient: init 2024-04-26T00:35:24.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.717+0000 7fce6e1b1640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:24.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.717+0000 7fce6e1b1640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-04-26T00:35:24.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.717+0000 7fce6e1b1640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce680a0260 0x7fce680a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:24.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.717+0000 7fce6e1b1640 10 monclient(hunting): picked mon.noname-b con 0x7fce680a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:24.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.717+0000 7fce6e1b1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fce68134b30 addr v1:172.21.15.37:6789/0 2024-04-26T00:35:24.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.717+0000 7fce6e1b1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fce68135660 addr v1:172.21.15.45:6789/0 2024-04-26T00:35:24.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.717+0000 7fce6e1b1640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce680a1130 con 0x7fce68134b30 2024-04-26T00:35:24.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.717+0000 7fce6e1b1640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce6806cff0 con 0x7fce68135660 2024-04-26T00:35:24.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.717+0000 7fce6e1b1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:24.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.717+0000 7fce6e1b1640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fce6809ec10 con 0x7fce680a0260 2024-04-26T00:35:24.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.717+0000 7fce6e1b1640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:24.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.717+0000 7fce6e1b1640 10 monclient(hunting): authenticate will time out at 1906.027465s 2024-04-26T00:35:24.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.717+0000 7fce677fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce680a0260 0x7fce680a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:24.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.717+0000 7fce67fff640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7fce68135660 legacy=0x7fce6813ce70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:35:24.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.717+0000 7fce66ffd640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fce68134b30 0x7fce68134f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:59096/0 (socket says 172.21.15.37:59096) 2024-04-26T00:35:24.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.717+0000 7fce66ffd640 1 -- 172.21.15.37:0/422919626 learned_addr learned my addr 172.21.15.37:0/422919626 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:35:24.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.717+0000 7fce677fe640 10 monclient(hunting): get_auth_request con 0x7fce680a0260 auth_method 0 2024-04-26T00:35:24.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.717+0000 7fce677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:24.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.717+0000 7fce677fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:24.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.717+0000 7fce677fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:24.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.717+0000 7fce677fe640 1 -- 172.21.15.37:0/422919626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce680a0260 msgr2=0x7fce680a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:35:24.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.717+0000 7fce677fe640 1 -- 172.21.15.37:0/422919626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce680a0260 msgr2=0x7fce680a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:35:24.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.717+0000 7fce677fe640 1 --2- 172.21.15.37:0/422919626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce680a0260 0x7fce680a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:35:24.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.717+0000 7fce677fe640 1 --2- 172.21.15.37:0/422919626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce680a0260 0x7fce680a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:35:24.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.721+0000 7fce66ffd640 1 -- 172.21.15.37:0/422919626 >> v1:172.21.15.37:6789/0 conn(0x7fce68134b30 legacy=0x7fce68134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:24.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.921+0000 7fce67fff640 1 -- 172.21.15.37:0/422919626 >> v1:172.21.15.45:6789/0 conn(0x7fce68135660 legacy=0x7fce6813ce70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:35:24.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.921+0000 7fce677fe640 1 --2- 172.21.15.37:0/422919626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce680a0260 0x7fce680a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:24.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.921+0000 7fce677fe640 10 monclient(hunting): get_auth_request con 0x7fce680a0260 auth_method 0 2024-04-26T00:35:24.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.921+0000 7fce677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:24.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.921+0000 7fce677fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:24.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.921+0000 7fce677fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:35:24.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.921+0000 7fce677fe640 1 -- 172.21.15.37:0/422919626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce680a0260 msgr2=0x7fce680a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:35:24.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.921+0000 7fce677fe640 1 -- 172.21.15.37:0/422919626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce680a0260 msgr2=0x7fce680a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:35:24.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.921+0000 7fce677fe640 1 --2- 172.21.15.37:0/422919626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce680a0260 0x7fce680a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:35:24.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:24.921+0000 7fce677fe640 1 --2- 172.21.15.37:0/422919626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce680a0260 0x7fce680a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:35:25.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:25.325+0000 7fce67fff640 1 -- 172.21.15.37:0/422919626 >> v1:172.21.15.45:6789/0 conn(0x7fce68135660 legacy=0x7fce6813ce70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:35:25.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:25.325+0000 7fce677fe640 1 --2- 172.21.15.37:0/422919626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce680a0260 0x7fce680a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:25.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:25.325+0000 7fce677fe640 10 monclient(hunting): get_auth_request con 0x7fce680a0260 auth_method 0 2024-04-26T00:35:25.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:25.325+0000 7fce677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:25.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:25.325+0000 7fce677fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:25.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:25.325+0000 7fce677fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:35:25.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:25.325+0000 7fce677fe640 1 -- 172.21.15.37:0/422919626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce680a0260 msgr2=0x7fce680a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:35:25.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:25.325+0000 7fce677fe640 1 -- 172.21.15.37:0/422919626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce680a0260 msgr2=0x7fce680a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:35:25.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:25.325+0000 7fce677fe640 1 --2- 172.21.15.37:0/422919626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce680a0260 0x7fce680a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:35:25.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:25.325+0000 7fce677fe640 1 --2- 172.21.15.37:0/422919626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce680a0260 0x7fce680a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:35:26.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:26.129+0000 7fce67fff640 1 -- 172.21.15.37:0/422919626 >> v1:172.21.15.45:6789/0 conn(0x7fce68135660 legacy=0x7fce6813ce70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:35:26.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:26.129+0000 7fce677fe640 1 --2- 172.21.15.37:0/422919626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce680a0260 0x7fce680a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:26.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:26.129+0000 7fce677fe640 10 monclient(hunting): get_auth_request con 0x7fce680a0260 auth_method 0 2024-04-26T00:35:26.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:26.129+0000 7fce677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:26.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:26.129+0000 7fce677fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:26.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:26.129+0000 7fce677fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:35:26.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:26.129+0000 7fce677fe640 1 -- 172.21.15.37:0/422919626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce680a0260 msgr2=0x7fce680a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:35:26.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:26.129+0000 7fce677fe640 1 -- 172.21.15.37:0/422919626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce680a0260 msgr2=0x7fce680a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:35:26.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:26.129+0000 7fce677fe640 1 --2- 172.21.15.37:0/422919626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce680a0260 0x7fce680a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:35:26.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:26.129+0000 7fce677fe640 1 --2- 172.21.15.37:0/422919626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce680a0260 0x7fce680a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:35:27.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:27.717+0000 7fce657fa640 10 monclient(hunting): tick 2024-04-26T00:35:27.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:27.717+0000 7fce657fa640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:35:27.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:27.717+0000 7fce657fa640 1 monclient(hunting): continuing hunt 2024-04-26T00:35:27.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:27.717+0000 7fce657fa640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:35:27.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:27.717+0000 7fce657fa640 1 -- 172.21.15.37:0/422919626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce680a0260 msgr2=0x7fce680a0640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:35:27.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:27.717+0000 7fce657fa640 1 --2- 172.21.15.37:0/422919626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce680a0260 0x7fce680a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:27.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:27.717+0000 7fce657fa640 1 -- 172.21.15.37:0/422919626 >> v1:172.21.15.45:6789/0 conn(0x7fce68135660 legacy=0x7fce6813ce70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:35:27.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:27.717+0000 7fce657fa640 1 -- 172.21.15.37:0/422919626 >> v1:172.21.15.37:6789/0 conn(0x7fce68134b30 legacy=0x7fce68134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:27.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:27.717+0000 7fce657fa640 10 monclient: _add_conns ranks=[3,4] 2024-04-26T00:35:27.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:27.717+0000 7fce657fa640 1 --2- 172.21.15.37:0/422919626 >> v2:172.21.15.37:3300/0 conn(0x7fce54003760 0x7fce54005be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:27.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:27.717+0000 7fce657fa640 10 monclient(hunting): picked mon.noname-a con 0x7fce54003760 addr v2:172.21.15.37:3300/0 2024-04-26T00:35:27.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:27.717+0000 7fce657fa640 1 --2- 172.21.15.37:0/422919626 >> v2:172.21.15.45:3300/0 conn(0x7fce540062d0 0x7fce54008790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:27.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:27.717+0000 7fce657fa640 10 monclient(hunting): picked mon.noname-c con 0x7fce540062d0 addr v2:172.21.15.45:3300/0 2024-04-26T00:35:27.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:27.717+0000 7fce657fa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:27.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:27.717+0000 7fce657fa640 1 -- 172.21.15.37:0/422919626 --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fce54008e20 con 0x7fce54003760 2024-04-26T00:35:27.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:27.717+0000 7fce657fa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:27.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:27.717+0000 7fce657fa640 1 -- 172.21.15.37:0/422919626 --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fce54008fd0 con 0x7fce540062d0 2024-04-26T00:35:27.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:27.717+0000 7fce657fa640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:27.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:27.717+0000 7fce66ffd640 1 -- 172.21.15.37:0/422919626 >> v2:172.21.15.45:3300/0 conn(0x7fce540062d0 msgr2=0x7fce54008790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:35:27.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:27.721+0000 7fce66ffd640 1 --2- 172.21.15.37:0/422919626 >> v2:172.21.15.45:3300/0 conn(0x7fce540062d0 0x7fce54008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:35:27.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:27.721+0000 7fce677fe640 1 --2- 172.21.15.37:0/422919626 >> v2:172.21.15.37:3300/0 conn(0x7fce54003760 0x7fce54005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:27.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:27.721+0000 7fce677fe640 10 monclient(hunting): get_auth_request con 0x7fce54003760 auth_method 0 2024-04-26T00:35:27.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:27.721+0000 7fce677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:27.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:27.721+0000 7fce677fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:27.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:27.721+0000 7fce677fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:27.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:27.721+0000 7fce677fe640 1 -- 172.21.15.37:0/422919626 >> v2:172.21.15.37:3300/0 conn(0x7fce54003760 msgr2=0x7fce54005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:35:27.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:27.721+0000 7fce677fe640 1 -- 172.21.15.37:0/422919626 >> v2:172.21.15.37:3300/0 conn(0x7fce54003760 msgr2=0x7fce54005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:35:27.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:27.721+0000 7fce677fe640 1 --2- 172.21.15.37:0/422919626 >> v2:172.21.15.37:3300/0 conn(0x7fce54003760 0x7fce54005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:35:27.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:27.721+0000 7fce677fe640 1 --2- 172.21.15.37:0/422919626 >> v2:172.21.15.37:3300/0 conn(0x7fce54003760 0x7fce54005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:35:27.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:27.925+0000 7fce66ffd640 1 -- 172.21.15.37:0/422919626 >> v2:172.21.15.45:3300/0 conn(0x7fce540062d0 msgr2=0x7fce54008790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:35:27.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:27.925+0000 7fce66ffd640 1 --2- 172.21.15.37:0/422919626 >> v2:172.21.15.45:3300/0 conn(0x7fce540062d0 0x7fce54008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:35:27.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:27.925+0000 7fce677fe640 1 --2- 172.21.15.37:0/422919626 >> v2:172.21.15.37:3300/0 conn(0x7fce54003760 0x7fce54005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:27.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:27.925+0000 7fce677fe640 10 monclient(hunting): get_auth_request con 0x7fce54003760 auth_method 0 2024-04-26T00:35:27.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:27.925+0000 7fce677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:27.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:27.925+0000 7fce677fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:27.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:27.925+0000 7fce677fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:35:27.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:27.925+0000 7fce677fe640 1 -- 172.21.15.37:0/422919626 >> v2:172.21.15.37:3300/0 conn(0x7fce54003760 msgr2=0x7fce54005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:35:27.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:27.925+0000 7fce677fe640 1 -- 172.21.15.37:0/422919626 >> v2:172.21.15.37:3300/0 conn(0x7fce54003760 msgr2=0x7fce54005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:35:27.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:27.925+0000 7fce677fe640 1 --2- 172.21.15.37:0/422919626 >> v2:172.21.15.37:3300/0 conn(0x7fce54003760 0x7fce54005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:35:27.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:27.925+0000 7fce677fe640 1 --2- 172.21.15.37:0/422919626 >> v2:172.21.15.37:3300/0 conn(0x7fce54003760 0x7fce54005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:35:28.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:28.329+0000 7fce66ffd640 1 -- 172.21.15.37:0/422919626 >> v2:172.21.15.45:3300/0 conn(0x7fce540062d0 msgr2=0x7fce54008790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:35:28.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:28.329+0000 7fce677fe640 1 --2- 172.21.15.37:0/422919626 >> v2:172.21.15.37:3300/0 conn(0x7fce54003760 0x7fce54005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:28.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:28.329+0000 7fce66ffd640 1 --2- 172.21.15.37:0/422919626 >> v2:172.21.15.45:3300/0 conn(0x7fce540062d0 0x7fce54008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:35:28.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:28.329+0000 7fce677fe640 10 monclient(hunting): get_auth_request con 0x7fce54003760 auth_method 0 2024-04-26T00:35:28.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:28.329+0000 7fce677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:28.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:28.329+0000 7fce677fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:28.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:28.329+0000 7fce677fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:35:28.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:28.329+0000 7fce677fe640 1 -- 172.21.15.37:0/422919626 >> v2:172.21.15.37:3300/0 conn(0x7fce54003760 msgr2=0x7fce54005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:35:28.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:28.329+0000 7fce677fe640 1 -- 172.21.15.37:0/422919626 >> v2:172.21.15.37:3300/0 conn(0x7fce54003760 msgr2=0x7fce54005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:35:28.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:28.329+0000 7fce677fe640 1 --2- 172.21.15.37:0/422919626 >> v2:172.21.15.37:3300/0 conn(0x7fce54003760 0x7fce54005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:35:28.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:28.329+0000 7fce677fe640 1 --2- 172.21.15.37:0/422919626 >> v2:172.21.15.37:3300/0 conn(0x7fce54003760 0x7fce54005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:35:29.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:29.133+0000 7fce66ffd640 1 -- 172.21.15.37:0/422919626 >> v2:172.21.15.45:3300/0 conn(0x7fce540062d0 msgr2=0x7fce54008790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:35:29.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:29.133+0000 7fce66ffd640 1 --2- 172.21.15.37:0/422919626 >> v2:172.21.15.45:3300/0 conn(0x7fce540062d0 0x7fce54008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:35:29.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:29.133+0000 7fce677fe640 1 --2- 172.21.15.37:0/422919626 >> v2:172.21.15.37:3300/0 conn(0x7fce54003760 0x7fce54005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:29.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:29.133+0000 7fce677fe640 10 monclient(hunting): get_auth_request con 0x7fce54003760 auth_method 0 2024-04-26T00:35:29.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:29.133+0000 7fce677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:29.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:29.133+0000 7fce677fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:29.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:29.133+0000 7fce677fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:35:29.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:29.133+0000 7fce677fe640 1 -- 172.21.15.37:0/422919626 >> v2:172.21.15.37:3300/0 conn(0x7fce54003760 msgr2=0x7fce54005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:35:29.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:29.133+0000 7fce677fe640 1 -- 172.21.15.37:0/422919626 >> v2:172.21.15.37:3300/0 conn(0x7fce54003760 msgr2=0x7fce54005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:35:29.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:29.133+0000 7fce677fe640 1 --2- 172.21.15.37:0/422919626 >> v2:172.21.15.37:3300/0 conn(0x7fce54003760 0x7fce54005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:35:29.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:29.133+0000 7fce677fe640 1 --2- 172.21.15.37:0/422919626 >> v2:172.21.15.37:3300/0 conn(0x7fce54003760 0x7fce54005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:35:29.373 INFO:tasks.rados.rados.0.smithi045.stdout:2578: done (0 left) 2024-04-26T00:35:29.373 INFO:tasks.rados.rados.0.smithi045.stdout:2579: write oid 17 current snap is 288 2024-04-26T00:35:29.373 INFO:tasks.rados.rados.0.smithi045.stdout:2579: seq_num 929 ranges {413703=501387,1596360=624660,2895372=691347} 2024-04-26T00:35:29.386 INFO:tasks.rados.rados.0.smithi045.stdout:2579: writing smithi04526579-17 from 413703 to 915090 tid 1 2024-04-26T00:35:29.407 INFO:tasks.rados.rados.0.smithi045.stdout:2579: writing smithi04526579-17 from 1596360 to 2221020 tid 2 2024-04-26T00:35:29.427 INFO:tasks.rados.rados.0.smithi045.stdout:2579: writing smithi04526579-17 from 2895372 to 3586719 tid 3 2024-04-26T00:35:29.427 INFO:tasks.rados.rados.0.smithi045.stdout:2580: rollback oid 2 current snap is 288 2024-04-26T00:35:29.427 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 2 to 287 2024-04-26T00:35:29.428 INFO:tasks.rados.rados.0.smithi045.stdout:2579: finishing write tid 1 to smithi04526579-17 2024-04-26T00:35:29.428 INFO:tasks.rados.rados.0.smithi045.stdout:2579: finishing write tid 2 to smithi04526579-17 2024-04-26T00:35:29.428 INFO:tasks.rados.rados.0.smithi045.stdout:2581: write oid 9 current snap is 288 2024-04-26T00:35:29.428 INFO:tasks.rados.rados.0.smithi045.stdout:2581: seq_num 930 ranges {430510=576636,1654842=768318,2922795=502619} 2024-04-26T00:35:29.438 INFO:tasks.rados.rados.0.smithi045.stdout:2581: writing smithi04526579-9 from 430510 to 1007146 tid 1 2024-04-26T00:35:29.452 INFO:tasks.rados.rados.0.smithi045.stdout:2581: writing smithi04526579-9 from 1654842 to 2423160 tid 2 2024-04-26T00:35:29.461 INFO:tasks.rados.rados.0.smithi045.stdout:2581: writing smithi04526579-9 from 2922795 to 3425414 tid 3 2024-04-26T00:35:29.461 INFO:tasks.rados.rados.0.smithi045.stdout:2582: delete oid 4 current snap is 288 2024-04-26T00:35:29.461 INFO:tasks.rados.rados.0.smithi045.stdout:2579: finishing write tid 3 to smithi04526579-17 2024-04-26T00:35:29.461 INFO:tasks.rados.rados.0.smithi045.stdout:2579: finishing write tid 4 to smithi04526579-17 2024-04-26T00:35:29.461 INFO:tasks.rados.rados.0.smithi045.stdout:2579: finishing write tid 5 to smithi04526579-17 2024-04-26T00:35:29.462 INFO:tasks.rados.rados.0.smithi045.stdout:2579: oid 17 updating version 0 to 1768 2024-04-26T00:35:29.462 INFO:tasks.rados.rados.0.smithi045.stdout:2579: oid 17 version 1768 is already newer than 1767 2024-04-26T00:35:29.462 INFO:tasks.rados.rados.0.smithi045.stdout:2579: oid 17 updating version 1768 to 1769 2024-04-26T00:35:29.462 INFO:tasks.rados.rados.0.smithi045.stdout:2579: oid 17 updating version 1769 to 1770 2024-04-26T00:35:29.462 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 1770 (ObjNum 929 snap 288 seq_num 929) dirty exists 2024-04-26T00:35:29.462 INFO:tasks.rados.rados.0.smithi045.stdout:2579: left oid 17 (ObjNum 929 snap 288 seq_num 929) 2024-04-26T00:35:29.463 INFO:tasks.rados.rados.0.smithi045.stdout:2580: finishing rollback tid 0 to smithi04526579-2 2024-04-26T00:35:29.463 INFO:tasks.rados.rados.0.smithi045.stdout:2580: finishing rollback tid 1 to smithi04526579-2 2024-04-26T00:35:29.463 INFO:tasks.rados.rados.0.smithi045.stdout:2580: finishing rollback tid 2 to smithi04526579-2 2024-04-26T00:35:29.463 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 1934 (ObjNum 690 snap 215 seq_num 690) dirty exists 2024-04-26T00:35:29.463 INFO:tasks.rados.rados.0.smithi045.stdout:2581: finishing write tid 1 to smithi04526579-9 2024-04-26T00:35:29.465 INFO:tasks.rados.rados.0.smithi045.stdout:2581: finishing write tid 2 to smithi04526579-9 2024-04-26T00:35:29.472 INFO:tasks.rados.rados.0.smithi045.stdout:2581: finishing write tid 3 to smithi04526579-9 2024-04-26T00:35:29.476 INFO:tasks.rados.rados.0.smithi045.stdout:2581: finishing write tid 4 to smithi04526579-9 2024-04-26T00:35:29.476 INFO:tasks.rados.rados.0.smithi045.stdout:2581: finishing write tid 5 to smithi04526579-9 2024-04-26T00:35:29.476 INFO:tasks.rados.rados.0.smithi045.stdout:2581: oid 9 updating version 0 to 1938 2024-04-26T00:35:29.476 INFO:tasks.rados.rados.0.smithi045.stdout:2581: oid 9 version 1938 is already newer than 1935 2024-04-26T00:35:29.476 INFO:tasks.rados.rados.0.smithi045.stdout:2581: oid 9 version 1938 is already newer than 1936 2024-04-26T00:35:29.477 INFO:tasks.rados.rados.0.smithi045.stdout:2581: oid 9 version 1938 is already newer than 1937 2024-04-26T00:35:29.477 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 1938 (ObjNum 930 snap 288 seq_num 930) dirty exists 2024-04-26T00:35:29.477 INFO:tasks.rados.rados.0.smithi045.stdout:2581: left oid 9 (ObjNum 930 snap 288 seq_num 930) 2024-04-26T00:35:29.477 INFO:tasks.rados.rados.0.smithi045.stdout:2579: done (3 left) 2024-04-26T00:35:29.477 INFO:tasks.rados.rados.0.smithi045.stdout:2580: done (2 left) 2024-04-26T00:35:29.477 INFO:tasks.rados.rados.0.smithi045.stdout:2581: done (1 left) 2024-04-26T00:35:29.477 INFO:tasks.rados.rados.0.smithi045.stdout:2582: done (0 left) 2024-04-26T00:35:29.478 INFO:tasks.rados.rados.0.smithi045.stdout:2583: write oid 23 current snap is 288 2024-04-26T00:35:29.478 INFO:tasks.rados.rados.0.smithi045.stdout:2583: seq_num 932 ranges {464124=727134,1771806=771987,2977641=641516} 2024-04-26T00:35:29.486 INFO:tasks.rados.rados.0.smithi045.stdout:2583: writing smithi04526579-23 from 464124 to 1191258 tid 1 2024-04-26T00:35:29.498 INFO:tasks.rados.rados.0.smithi045.stdout:2583: writing smithi04526579-23 from 1771806 to 2543793 tid 2 2024-04-26T00:35:29.506 INFO:tasks.rados.rados.0.smithi045.stdout:2583: writing smithi04526579-23 from 2977641 to 3619157 tid 3 2024-04-26T00:35:29.506 INFO:tasks.rados.rados.0.smithi045.stdout:2584: snap_remove snap 278 2024-04-26T00:35:30.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.717+0000 7fce657fa640 10 monclient(hunting): tick 2024-04-26T00:35:30.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.717+0000 7fce657fa640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:35:30.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.717+0000 7fce657fa640 1 monclient(hunting): continuing hunt 2024-04-26T00:35:30.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.717+0000 7fce657fa640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:35:30.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.717+0000 7fce657fa640 1 -- 172.21.15.37:0/422919626 >> v2:172.21.15.45:3300/0 conn(0x7fce540062d0 msgr2=0x7fce54008790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:35:30.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.717+0000 7fce657fa640 1 --2- 172.21.15.37:0/422919626 >> v2:172.21.15.45:3300/0 conn(0x7fce540062d0 0x7fce54008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:30.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.717+0000 7fce657fa640 1 -- 172.21.15.37:0/422919626 >> v2:172.21.15.37:3300/0 conn(0x7fce54003760 msgr2=0x7fce54005be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:35:30.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.717+0000 7fce657fa640 1 --2- 172.21.15.37:0/422919626 >> v2:172.21.15.37:3300/0 conn(0x7fce54003760 0x7fce54005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:30.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.721+0000 7fce67fff640 1 -- 172.21.15.37:0/422919626 reap_dead start 2024-04-26T00:35:30.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.721+0000 7fce657fa640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-04-26T00:35:30.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.721+0000 7fce657fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fce54003bd0 addr v1:172.21.15.45:6789/0 2024-04-26T00:35:30.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.721+0000 7fce657fa640 1 --2- 172.21.15.37:0/422919626 >> v2:172.21.15.45:3300/0 conn(0x7fce54007e80 0x7fce54008300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:30.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.721+0000 7fce657fa640 10 monclient(hunting): picked mon.noname-c con 0x7fce54007e80 addr v2:172.21.15.45:3300/0 2024-04-26T00:35:30.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.721+0000 7fce657fa640 1 --2- 172.21.15.37:0/422919626 >> v2:172.21.15.37:3300/0 conn(0x7fce54008840 0x7fce5400d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:30.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.721+0000 7fce657fa640 10 monclient(hunting): picked mon.noname-a con 0x7fce54008840 addr v2:172.21.15.37:3300/0 2024-04-26T00:35:30.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.721+0000 7fce657fa640 1 -- 172.21.15.37:0/422919626 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce6809bc50 con 0x7fce54003bd0 2024-04-26T00:35:30.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.721+0000 7fce657fa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:30.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.721+0000 7fce657fa640 1 -- 172.21.15.37:0/422919626 --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fce54008e20 con 0x7fce54008840 2024-04-26T00:35:30.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.721+0000 7fce657fa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:30.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.721+0000 7fce657fa640 1 -- 172.21.15.37:0/422919626 --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fce54008fd0 con 0x7fce54007e80 2024-04-26T00:35:30.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.721+0000 7fce657fa640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:30.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.721+0000 7fce677fe640 1 -- 172.21.15.37:0/422919626 >> v1:172.21.15.45:6789/0 conn(0x7fce54003bd0 legacy=0x7fce54006760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:35:30.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.721+0000 7fce66ffd640 1 -- 172.21.15.37:0/422919626 >> v2:172.21.15.45:3300/0 conn(0x7fce54007e80 msgr2=0x7fce54008300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:35:30.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.721+0000 7fce66ffd640 1 --2- 172.21.15.37:0/422919626 >> v2:172.21.15.45:3300/0 conn(0x7fce54007e80 0x7fce54008300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:35:30.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.721+0000 7fce67fff640 1 --2- 172.21.15.37:0/422919626 >> v2:172.21.15.37:3300/0 conn(0x7fce54008840 0x7fce5400d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:30.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.721+0000 7fce67fff640 10 monclient(hunting): get_auth_request con 0x7fce54008840 auth_method 0 2024-04-26T00:35:30.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.721+0000 7fce67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:30.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.721+0000 7fce67fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:30.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.721+0000 7fce67fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:30.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.721+0000 7fce67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:30.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.721+0000 7fce67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:30.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.721+0000 7fce67fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:30.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.721+0000 7fce67fff640 10 monclient(hunting): handle_auth_done global_id 144558 payload 274 2024-04-26T00:35:30.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.721+0000 7fce67fff640 1 -- 172.21.15.37:0/422919626 >> v2:172.21.15.45:3300/0 conn(0x7fce54007e80 msgr2=0x7fce54008300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:35:30.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.721+0000 7fce67fff640 1 --2- 172.21.15.37:0/422919626 >> v2:172.21.15.45:3300/0 conn(0x7fce54007e80 0x7fce54008300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:30.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.721+0000 7fce67fff640 1 -- 172.21.15.37:0/422919626 >> v1:172.21.15.45:6789/0 conn(0x7fce54003bd0 legacy=0x7fce54006760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:35:30.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.721+0000 7fce67fff640 10 monclient: _finish_hunting 0 2024-04-26T00:35:30.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.721+0000 7fce67fff640 1 monclient: found mon.noname-a 2024-04-26T00:35:30.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.721+0000 7fce67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:30.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.721+0000 7fce67fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:35:30.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.721+0000 7fce67fff640 1 -- 172.21.15.37:0/422919626 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce6806cff0 con 0x7fce54008840 2024-04-26T00:35:30.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.729+0000 7fce67fff640 1 -- 172.21.15.37:0/422919626 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce54008840 msgr2=0x7fce5400d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:30.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.729+0000 7fce67fff640 1 --2- 172.21.15.37:0/422919626 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce54008840 0x7fce5400d3a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fce68135a40 tx=0x7fce6001cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=750d3dcf1c768b25 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:30.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.729+0000 7fce67fff640 1 --2- 172.21.15.37:0/422919626 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce54008840 0x7fce5400d3a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fce68135a40 tx=0x7fce6001cc90 comp rx=0 tx=0).queue_received will delay after 1.71409e+09 on 0x7fce600a9070 config(0 keys) 2024-04-26T00:35:30.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.729+0000 7fce667fc640 1 -- 172.21.15.37:0/422919626 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce600a5ac0 con 0x7fce54008840 2024-04-26T00:35:30.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.729+0000 7fce667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:30.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.729+0000 7fce667fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:35:30.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.729+0000 7fce667fc640 10 monclient: dump: 2024-04-26T00:35:30.729 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:30.729 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:30.729 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:30.729 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:30.729 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:30.729 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:30.729 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:30.729 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:30.729 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:30.730 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:30.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.729+0000 7fce667fc640 10 monclient: _finish_auth 0 2024-04-26T00:35:30.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.729+0000 7fce667fc640 10 monclient: _check_auth_tickets 2024-04-26T00:35:30.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.729+0000 7fce667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:30.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.729+0000 7fce667fc640 1 -- 172.21.15.37:0/422919626 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce600a9070 con 0x7fce54008840 2024-04-26T00:35:30.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.729+0000 7fce667fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:30.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.729+0000 7fce6e1b1640 5 monclient: authenticate success, global_id 144558 2024-04-26T00:35:30.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.729+0000 7fce6e1b1640 10 monclient: get_monmap_and_config success 2024-04-26T00:35:30.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.729+0000 7fce6e1b1640 4 set_mon_vals no callback set 2024-04-26T00:35:30.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.729+0000 7fce6e1b1640 10 monclient: shutdown 2024-04-26T00:35:30.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.729+0000 7fce6e1b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:30.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.729+0000 7fce6e1b1640 1 -- 172.21.15.37:0/422919626 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce54008840 msgr2=0x7fce5400d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:30.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.729+0000 7fce6e1b1640 1 --2- 172.21.15.37:0/422919626 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce54008840 0x7fce5400d3a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fce68135a40 tx=0x7fce6001cc90 comp rx=0 tx=0).stop 2024-04-26T00:35:30.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.729+0000 7fce667fc640 1 -- 172.21.15.37:0/422919626 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 0 0 0) 0x7fce600ae470 con 0x7fce54008840 2024-04-26T00:35:30.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.729+0000 7fce6e1b1640 1 -- 172.21.15.37:0/422919626 shutdown_connections 2024-04-26T00:35:30.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.729+0000 7fce667fc640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-04-26T00:35:30.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.729+0000 7fce6e1b1640 1 --2- 172.21.15.37:0/422919626 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce54008840 0x7fce5400d3a0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:30.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.729+0000 7fce6e1b1640 1 --2- 172.21.15.37:0/422919626 >> v2:172.21.15.45:3300/0 conn(0x7fce54007e80 0x7fce54008300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:30.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.729+0000 7fce6e1b1640 1 -- 172.21.15.37:0/422919626 >> 172.21.15.37:0/422919626 conn(0x7fce680a1ad0 msgr2=0x7fce680a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:30.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce6e1b1640 1 -- 172.21.15.37:0/422919626 shutdown_connections 2024-04-26T00:35:30.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce6e1b1640 1 -- 172.21.15.37:0/422919626 wait complete. 2024-04-26T00:35:30.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce6e1b1640 10 monclient: build_initial_monmap 2024-04-26T00:35:30.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce6e1b1640 10 monclient: monmap: 2024-04-26T00:35:30.733 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:30.733 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:35:30.733 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:35:30.733 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:35:30.733 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:30.733 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:35:30.733 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:35:30.733 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:35:30.733 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:35:30.733 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:30.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce6e1b1640 1 Processor -- start 2024-04-26T00:35:30.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce6e1b1640 1 -- start start 2024-04-26T00:35:30.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce6e1b1640 10 monclient: init 2024-04-26T00:35:30.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce6e1b1640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:30.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce6e1b1640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:35:30.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce6e1b1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce54003bd0 0x7fce681cf250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:30.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce6e1b1640 10 monclient(hunting): picked mon.noname-b con 0x7fce54003bd0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:35:30.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce6e1b1640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce54007e80 0x7fce68130b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:30.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce6e1b1640 10 monclient(hunting): picked mon.noname-c con 0x7fce54007e80 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:30.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce6e1b1640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce54008840 0x7fce681310b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:30.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce6e1b1640 10 monclient(hunting): picked mon.noname-a con 0x7fce54008840 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:35:30.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce6e1b1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:30.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce6e1b1640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fce6809f0d0 con 0x7fce54008840 2024-04-26T00:35:30.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce6e1b1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:30.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce6e1b1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fce6809f4c0 con 0x7fce54003bd0 2024-04-26T00:35:30.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce6e1b1640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:30.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce6e1b1640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fce6809fe90 con 0x7fce54007e80 2024-04-26T00:35:30.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce6e1b1640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:30.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce6e1b1640 10 monclient(hunting): authenticate will time out at 1912.043131s 2024-04-26T00:35:30.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce677fe640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce54003bd0 msgr2=0x7fce681cf250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:35:30.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce677fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce54003bd0 0x7fce681cf250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:35:30.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce66ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce54007e80 0x7fce68130b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:30.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce67fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce54008840 0x7fce681310b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:30.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce66ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce54007e80 0x7fce68130b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:44452/0 (socket says 172.21.15.37:44452) 2024-04-26T00:35:30.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce66ffd640 1 -- 172.21.15.37:0/952331189 learned_addr learned my addr 172.21.15.37:0/952331189 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:30.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce66ffd640 10 monclient(hunting): get_auth_request con 0x7fce54007e80 auth_method 0 2024-04-26T00:35:30.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce66ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:30.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce66ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:30.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:30.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce67fff640 10 monclient(hunting): get_auth_request con 0x7fce54008840 auth_method 0 2024-04-26T00:35:30.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:30.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce67fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:30.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce67fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:30.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce66ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:30.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce66ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:30.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce66ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:30.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:30.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:30.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce67fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:30.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce66ffd640 10 monclient(hunting): handle_auth_done global_id 184154 payload 824 2024-04-26T00:35:30.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce66ffd640 1 -- 172.21.15.37:0/952331189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce54003bd0 msgr2=0x7fce681cf250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:35:30.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.733+0000 7fce66ffd640 1 --2- 172.21.15.37:0/952331189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce54003bd0 0x7fce681cf250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:30.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.737+0000 7fce66ffd640 1 -- 172.21.15.37:0/952331189 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce54008840 msgr2=0x7fce681310b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:30.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.737+0000 7fce66ffd640 1 --2- 172.21.15.37:0/952331189 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce54008840 0x7fce681310b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:30.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.737+0000 7fce66ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:35:30.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.737+0000 7fce66ffd640 1 monclient: found mon.noname-c 2024-04-26T00:35:30.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.737+0000 7fce66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:30.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.737+0000 7fce66ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:35:30.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.737+0000 7fce66ffd640 1 -- 172.21.15.37:0/952331189 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce68123b90 con 0x7fce54007e80 2024-04-26T00:35:30.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.737+0000 7fce67fff640 1 --2- 172.21.15.37:0/952331189 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce54008840 0x7fce681310b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:35:30.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.737+0000 7fce66ffd640 1 -- 172.21.15.37:0/952331189 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce54007e80 msgr2=0x7fce68130b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:30.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.737+0000 7fce66ffd640 1 --2- 172.21.15.37:0/952331189 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce54007e80 0x7fce68130b70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fce5c00c4a0 tx=0x7fce5c00c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:30.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.737+0000 7fce4ffff640 1 -- 172.21.15.37:0/952331189 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce5c017070 con 0x7fce54007e80 2024-04-26T00:35:30.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.737+0000 7fce4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:30.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.737+0000 7fce4ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:35:30.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.737+0000 7fce4ffff640 10 monclient: dump: 2024-04-26T00:35:30.742 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:30.742 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:30.742 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:30.742 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:30.742 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:30.742 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:30.742 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:30.742 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:30.742 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:30.742 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:30.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.737+0000 7fce4ffff640 10 monclient: _finish_auth 0 2024-04-26T00:35:30.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.737+0000 7fce4ffff640 10 monclient: _check_auth_tickets 2024-04-26T00:35:30.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.737+0000 7fce4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:30.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.737+0000 7fce6e1b1640 5 monclient: authenticate success, global_id 184154 2024-04-26T00:35:30.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.737+0000 7fce6e1b1640 10 monclient: _renew_subs 2024-04-26T00:35:30.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.737+0000 7fce6e1b1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:35:30.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.737+0000 7fce6e1b1640 1 -- 172.21.15.37:0/952331189 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce68131ce0 con 0x7fce54007e80 2024-04-26T00:35:30.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.737+0000 7fce4ffff640 1 -- 172.21.15.37:0/952331189 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce5c007070 con 0x7fce54007e80 2024-04-26T00:35:30.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.737+0000 7fce4ffff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:30.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.737+0000 7fce4ffff640 1 -- 172.21.15.37:0/952331189 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce5c007420 con 0x7fce54007e80 2024-04-26T00:35:30.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.737+0000 7fce4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:30.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.737+0000 7fce4ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:35:30.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.737+0000 7fce4ffff640 10 monclient: dump: 2024-04-26T00:35:30.744 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:30.744 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:30.744 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:30.744 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:30.744 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:30.744 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:30.744 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:30.745 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:30.745 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:30.745 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:30.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.737+0000 7fce6e1b1640 10 monclient: _renew_subs 2024-04-26T00:35:30.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.737+0000 7fce6e1b1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:35:30.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.737+0000 7fce65ffb640 4 set_mon_vals no callback set 2024-04-26T00:35:30.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.737+0000 7fce6e1b1640 1 -- 172.21.15.37:0/952331189 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fce681252c0 con 0x7fce54007e80 2024-04-26T00:35:30.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.737+0000 7fce4ffff640 1 -- 172.21.15.37:0/952331189 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fce5c01a070 con 0x7fce54007e80 2024-04-26T00:35:30.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.741+0000 7fce4ffff640 1 --2- 172.21.15.37:0/952331189 >> v2:172.21.15.45:6812/2379918623 conn(0x7fce54037a70 0x7fce54039f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:30.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.741+0000 7fce4ffff640 1 -- 172.21.15.37:0/952331189 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(600..600 src has 370..600) ==== 3943+0+0 (secure 0 0 0) 0x7fce5c047e50 con 0x7fce54007e80 2024-04-26T00:35:30.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.741+0000 7fce6e1b1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:30.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.741+0000 7fce6e1b1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:30.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.741+0000 7fce6e1b1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:35:30.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.741+0000 7fce6e1b1640 1 -- 172.21.15.37:0/952331189 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce68066f40 con 0x7fce54007e80 2024-04-26T00:35:30.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.741+0000 7fce677fe640 1 --2- 172.21.15.37:0/952331189 >> v2:172.21.15.45:6812/2379918623 conn(0x7fce54037a70 0x7fce54039f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:30.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.741+0000 7fce677fe640 10 monclient: get_auth_request con 0x7fce54037a70 auth_method 0 2024-04-26T00:35:30.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.741+0000 7fce677fe640 1 --2- 172.21.15.37:0/952331189 >> v2:172.21.15.45:6812/2379918623 conn(0x7fce54037a70 0x7fce54039f30 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fce68122eb0 tx=0x7fce58008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:30.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.745+0000 7fce4ffff640 1 -- 172.21.15.37:0/952331189 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fce680a1130 con 0x7fce54007e80 2024-04-26T00:35:30.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.745+0000 7fce4ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:30.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.745+0000 7fce4ffff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:35:30.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.973+0000 7fce6e1b1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:35:30.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.973+0000 7fce6e1b1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:35:30.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.973+0000 7fce6e1b1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:35:30.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.973+0000 7fce6e1b1640 1 -- 172.21.15.37:0/952331189 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fce6806b9a0 con 0x7fce54007e80 2024-04-26T00:35:30.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.973+0000 7fce4ffff640 1 -- 172.21.15.37:0/952331189 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7fce5c007680 con 0x7fce54007e80 2024-04-26T00:35:30.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.973+0000 7fce4ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:35:30.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.973+0000 7fce4ffff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:35:30.971 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":118,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:35:30.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.977+0000 7fce6e1b1640 1 -- 172.21.15.37:0/952331189 >> v2:172.21.15.45:6812/2379918623 conn(0x7fce54037a70 msgr2=0x7fce54039f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:30.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.977+0000 7fce6e1b1640 1 --2- 172.21.15.37:0/952331189 >> v2:172.21.15.45:6812/2379918623 conn(0x7fce54037a70 0x7fce54039f30 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fce68122eb0 tx=0x7fce58008040 comp rx=0 tx=0).stop 2024-04-26T00:35:30.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.977+0000 7fce6e1b1640 10 monclient: shutdown 2024-04-26T00:35:30.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.977+0000 7fce6e1b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:30.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.977+0000 7fce6e1b1640 1 -- 172.21.15.37:0/952331189 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce54007e80 msgr2=0x7fce68130b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:30.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.977+0000 7fce6e1b1640 1 --2- 172.21.15.37:0/952331189 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce54007e80 0x7fce68130b70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fce5c00c4a0 tx=0x7fce5c00c960 comp rx=0 tx=0).stop 2024-04-26T00:35:30.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.977+0000 7fce6e1b1640 1 -- 172.21.15.37:0/952331189 shutdown_connections 2024-04-26T00:35:30.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.977+0000 7fce6e1b1640 1 --2- 172.21.15.37:0/952331189 >> v2:172.21.15.45:6812/2379918623 conn(0x7fce54037a70 0x7fce54039f30 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:30.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.977+0000 7fce6e1b1640 1 --2- 172.21.15.37:0/952331189 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce54008840 0x7fce681310b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:30.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.977+0000 7fce6e1b1640 1 --2- 172.21.15.37:0/952331189 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce54007e80 0x7fce68130b70 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:30.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.977+0000 7fce6e1b1640 1 --2- 172.21.15.37:0/952331189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce54003bd0 0x7fce681cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:30.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.977+0000 7fce6e1b1640 1 -- 172.21.15.37:0/952331189 >> 172.21.15.37:0/952331189 conn(0x7fce680a1ad0 msgr2=0x7fce680a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:30.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.977+0000 7fce6e1b1640 1 -- 172.21.15.37:0/952331189 shutdown_connections 2024-04-26T00:35:30.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:30.977+0000 7fce6e1b1640 1 -- 172.21.15.37:0/952331189 wait complete. 2024-04-26T00:35:30.992 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-26T00:35:30.992 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-04-26T00:35:30.992 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:35:31.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0dadf5d640 10 monclient: get_monmap_and_config 2024-04-26T00:35:31.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0dadf5d640 10 monclient: build_initial_monmap 2024-04-26T00:35:31.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0dadf5d640 10 monclient: monmap: 2024-04-26T00:35:31.119 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:31.119 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:31.119 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:35:31.125433+0000 2024-04-26T00:35:31.119 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:35:31.125433+0000 2024-04-26T00:35:31.119 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:31.119 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:31.120 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:35:31.120 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:35:31.120 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:35:31.120 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:35:31.120 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:35:31.120 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:31.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0dadf5d640 1 Processor -- start 2024-04-26T00:35:31.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0dadf5d640 1 -- start start 2024-04-26T00:35:31.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0dadf5d640 10 monclient: init 2024-04-26T00:35:31.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0dadf5d640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:31.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0dadf5d640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-04-26T00:35:31.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0dadf5d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f0da80a3d90 0x7f0da809bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:31.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0dadf5d640 10 monclient(hunting): picked mon.noname-a con 0x7f0da80a3d90 addr v2:172.21.15.37:3300/0 2024-04-26T00:35:31.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0dadf5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0da80a4170 addr v1:172.21.15.45:6789/0 2024-04-26T00:35:31.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0dadf5d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0da813c3e0 0x7f0da813e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:31.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0dadf5d640 10 monclient(hunting): picked mon.noname-b con 0x7f0da813c3e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:31.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0dadf5d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0da80a5150 con 0x7f0da80a4170 2024-04-26T00:35:31.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0dadf5d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:31.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0dadf5d640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f0da809b280 con 0x7f0da80a3d90 2024-04-26T00:35:31.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0dadf5d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:31.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0dadf5d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0da809b560 con 0x7f0da813c3e0 2024-04-26T00:35:31.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0dadf5d640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:31.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0dadf5d640 10 monclient(hunting): authenticate will time out at 1912.430989s 2024-04-26T00:35:31.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0da77fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f0da80a3d90 0x7f0da809bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:31.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0da77fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f0da80a3d90 0x7f0da809bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:49226/0 (socket says 172.21.15.37:49226) 2024-04-26T00:35:31.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0da77fe640 1 -- 172.21.15.37:0/4085870078 learned_addr learned my addr 172.21.15.37:0/4085870078 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:31.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0da77fe640 10 monclient(hunting): get_auth_request con 0x7f0da80a3d90 auth_method 0 2024-04-26T00:35:31.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0da77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:31.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0da77fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:31.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0da77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:31.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0da6ffd640 1 -- 172.21.15.37:0/4085870078 >> v1:172.21.15.45:6789/0 conn(0x7f0da80a4170 legacy=0x7f0da80a45e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:35:31.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0da7fff640 1 --2- 172.21.15.37:0/4085870078 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0da813c3e0 0x7f0da813e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:31.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0da77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:31.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0da77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:31.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0da77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:31.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0da7fff640 10 monclient(hunting): get_auth_request con 0x7f0da813c3e0 auth_method 0 2024-04-26T00:35:31.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0da7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:31.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0da7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:31.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0da7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:31.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0da7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:31.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0da7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:31.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0da7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:31.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0da77fe640 10 monclient(hunting): handle_auth_done global_id 144564 payload 274 2024-04-26T00:35:31.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0da77fe640 1 -- 172.21.15.37:0/4085870078 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0da813c3e0 msgr2=0x7f0da813e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:31.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0da77fe640 1 --2- 172.21.15.37:0/4085870078 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0da813c3e0 0x7f0da813e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:31.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0da77fe640 1 -- 172.21.15.37:0/4085870078 >> v1:172.21.15.45:6789/0 conn(0x7f0da80a4170 legacy=0x7f0da80a45e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:35:31.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0da77fe640 10 monclient: _finish_hunting 0 2024-04-26T00:35:31.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0da77fe640 1 monclient: found mon.noname-a 2024-04-26T00:35:31.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0da77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:31.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0da77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:35:31.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.121+0000 7f0da77fe640 1 -- 172.21.15.37:0/4085870078 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0da806dc60 con 0x7f0da80a3d90 2024-04-26T00:35:31.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.125+0000 7f0da77fe640 1 -- 172.21.15.37:0/4085870078 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0da80a3d90 msgr2=0x7f0da809bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:31.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.125+0000 7f0da77fe640 1 --2- 172.21.15.37:0/4085870078 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0da80a3d90 0x7f0da809bf40 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0d90009870 tx=0x7f0d9001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2599edaaa31d9ca7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:31.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.125+0000 7f0da67fc640 1 -- 172.21.15.37:0/4085870078 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d900b0070 con 0x7f0da80a3d90 2024-04-26T00:35:31.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.125+0000 7f0da67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:31.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.125+0000 7f0da67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:35:31.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.125+0000 7f0da67fc640 10 monclient: dump: 2024-04-26T00:35:31.129 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:31.129 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:31.129 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:31.129 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:31.129 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:31.129 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:31.130 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:31.130 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:31.130 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:31.130 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:31.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.125+0000 7f0da67fc640 10 monclient: _finish_auth 0 2024-04-26T00:35:31.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.125+0000 7f0da67fc640 10 monclient: _check_auth_tickets 2024-04-26T00:35:31.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.125+0000 7f0da67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:31.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.125+0000 7f0da67fc640 1 -- 172.21.15.37:0/4085870078 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d900ab600 con 0x7f0da80a3d90 2024-04-26T00:35:31.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.125+0000 7f0dadf5d640 5 monclient: authenticate success, global_id 144564 2024-04-26T00:35:31.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.125+0000 7f0dadf5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:35:31.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.125+0000 7f0da67fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:31.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.125+0000 7f0dadf5d640 10 monclient: get_monmap_and_config success 2024-04-26T00:35:31.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.125+0000 7f0da67fc640 1 -- 172.21.15.37:0/4085870078 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d900ab9d0 con 0x7f0da80a3d90 2024-04-26T00:35:31.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.125+0000 7f0da67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:31.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.125+0000 7f0dadf5d640 4 set_mon_vals no callback set 2024-04-26T00:35:31.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.125+0000 7f0da67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:31.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.125+0000 7f0da67fc640 10 monclient: dump: 2024-04-26T00:35:31.131 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:31.131 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:31.131 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:31.131 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:31.131 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:31.131 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:31.131 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:31.131 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:31.131 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:31.131 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:31.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0dadf5d640 10 monclient: shutdown 2024-04-26T00:35:31.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0dadf5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:31.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0dadf5d640 1 -- 172.21.15.37:0/4085870078 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0da80a3d90 msgr2=0x7f0da809bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:31.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0dadf5d640 1 --2- 172.21.15.37:0/4085870078 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0da80a3d90 0x7f0da809bf40 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0d90009870 tx=0x7f0d9001c3e0 comp rx=0 tx=0).stop 2024-04-26T00:35:31.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0dadf5d640 1 -- 172.21.15.37:0/4085870078 shutdown_connections 2024-04-26T00:35:31.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0dadf5d640 1 --2- 172.21.15.37:0/4085870078 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0da813c3e0 0x7f0da813e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:31.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0dadf5d640 1 --2- 172.21.15.37:0/4085870078 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0da80a3d90 0x7f0da809bf40 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:31.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0dadf5d640 1 -- 172.21.15.37:0/4085870078 >> 172.21.15.37:0/4085870078 conn(0x7f0da80aa0a0 msgr2=0x7f0da80a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:31.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0dadf5d640 1 -- 172.21.15.37:0/4085870078 shutdown_connections 2024-04-26T00:35:31.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0dadf5d640 1 -- 172.21.15.37:0/4085870078 wait complete. 2024-04-26T00:35:31.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0dadf5d640 10 monclient: build_initial_monmap 2024-04-26T00:35:31.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0dadf5d640 10 monclient: monmap: 2024-04-26T00:35:31.132 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:31.132 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:35:31.132 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:35:31.132 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:35:31.132 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:31.133 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:35:31.133 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:35:31.133 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:35:31.133 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:35:31.133 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:31.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0dadf5d640 1 Processor -- start 2024-04-26T00:35:31.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0dadf5d640 1 -- start start 2024-04-26T00:35:31.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0dadf5d640 10 monclient: init 2024-04-26T00:35:31.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0dadf5d640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:31.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0dadf5d640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:35:31.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0dadf5d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0da80a3d90 0x7f0da81cee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:31.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0dadf5d640 10 monclient(hunting): picked mon.noname-a con 0x7f0da80a3d90 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:35:31.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0dadf5d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0da80a4170 0x7f0da8136e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:31.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0dadf5d640 10 monclient(hunting): picked mon.noname-c con 0x7f0da80a4170 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:31.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0dadf5d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0da813c3e0 0x7f0da81373b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:31.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0dadf5d640 10 monclient(hunting): picked mon.noname-b con 0x7f0da813c3e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:35:31.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0dadf5d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:31.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0dadf5d640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f0da8141740 con 0x7f0da80a3d90 2024-04-26T00:35:31.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0dadf5d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:31.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0dadf5d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0da809c690 con 0x7f0da813c3e0 2024-04-26T00:35:31.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0dadf5d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:31.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0dadf5d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0da81418c0 con 0x7f0da80a4170 2024-04-26T00:35:31.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0dadf5d640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:31.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0dadf5d640 10 monclient(hunting): authenticate will time out at 1912.438122s 2024-04-26T00:35:31.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0da77fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0da80a3d90 0x7f0da81cee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:31.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0da6ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0da80a4170 0x7f0da8136e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:31.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0da77fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0da80a3d90 0x7f0da81cee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:49230/0 (socket says 172.21.15.37:49230) 2024-04-26T00:35:31.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0da6ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0da80a4170 0x7f0da8136e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:44468/0 (socket says 172.21.15.37:44468) 2024-04-26T00:35:31.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0da77fe640 1 -- 172.21.15.37:0/1432310783 learned_addr learned my addr 172.21.15.37:0/1432310783 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:31.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0da77fe640 10 monclient(hunting): get_auth_request con 0x7f0da80a3d90 auth_method 0 2024-04-26T00:35:31.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0da77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:31.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0da77fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:31.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0da77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:31.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0da7fff640 1 -- 172.21.15.37:0/1432310783 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0da813c3e0 msgr2=0x7f0da81373b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:35:31.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0da6ffd640 10 monclient(hunting): get_auth_request con 0x7f0da80a4170 auth_method 0 2024-04-26T00:35:31.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0da6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:31.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0da6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:31.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0da6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:31.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0da7fff640 1 --2- 172.21.15.37:0/1432310783 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0da813c3e0 0x7f0da81373b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:35:31.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0da77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:31.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0da77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:31.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0da77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:31.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0da6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:31.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0da6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:31.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0da6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:31.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0da6ffd640 10 monclient(hunting): handle_auth_done global_id 184160 payload 824 2024-04-26T00:35:31.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0da6ffd640 1 -- 172.21.15.37:0/1432310783 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0da813c3e0 msgr2=0x7f0da81373b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:35:31.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0da6ffd640 1 --2- 172.21.15.37:0/1432310783 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0da813c3e0 0x7f0da81373b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:31.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0da6ffd640 1 -- 172.21.15.37:0/1432310783 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0da80a3d90 msgr2=0x7f0da81cee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:31.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0da6ffd640 1 --2- 172.21.15.37:0/1432310783 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0da80a3d90 0x7f0da81cee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:31.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0da6ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:35:31.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0da6ffd640 1 monclient: found mon.noname-c 2024-04-26T00:35:31.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0da6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:31.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0da6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:35:31.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0da6ffd640 1 -- 172.21.15.37:0/1432310783 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0da8133360 con 0x7f0da80a4170 2024-04-26T00:35:31.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.129+0000 7f0da77fe640 1 --2- 172.21.15.37:0/1432310783 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0da80a3d90 0x7f0da81cee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:35:31.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0da6ffd640 1 -- 172.21.15.37:0/1432310783 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0da80a4170 msgr2=0x7f0da8136e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:31.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0da6ffd640 1 --2- 172.21.15.37:0/1432310783 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0da80a4170 0x7f0da8136e70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0d9800be00 tx=0x7f0d98009f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:31.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0d9ffff640 1 -- 172.21.15.37:0/1432310783 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d98015670 con 0x7f0da80a4170 2024-04-26T00:35:31.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0d9ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:31.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0d9ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:35:31.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0d9ffff640 10 monclient: dump: 2024-04-26T00:35:31.140 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:31.140 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:31.140 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:31.140 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:31.140 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:31.140 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:31.140 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:31.140 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:31.140 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:31.140 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:31.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0d9ffff640 10 monclient: _finish_auth 0 2024-04-26T00:35:31.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0d9ffff640 10 monclient: _check_auth_tickets 2024-04-26T00:35:31.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0d9ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:31.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0d9ffff640 1 -- 172.21.15.37:0/1432310783 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d98010070 con 0x7f0da80a4170 2024-04-26T00:35:31.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0dadf5d640 5 monclient: authenticate success, global_id 184160 2024-04-26T00:35:31.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0dadf5d640 10 monclient: _renew_subs 2024-04-26T00:35:31.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0dadf5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:35:31.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0dadf5d640 1 -- 172.21.15.37:0/1432310783 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0da8132190 con 0x7f0da80a4170 2024-04-26T00:35:31.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0d9ffff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:31.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0d9ffff640 1 -- 172.21.15.37:0/1432310783 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d98015a50 con 0x7f0da80a4170 2024-04-26T00:35:31.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0d9ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:31.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0d9ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:35:31.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0d9ffff640 10 monclient: dump: 2024-04-26T00:35:31.141 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:31.141 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:31.141 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:31.141 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:31.141 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:31.141 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:31.142 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:31.142 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:31.142 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:31.142 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:31.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0da5ffb640 4 set_mon_vals no callback set 2024-04-26T00:35:31.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0dadf5d640 10 monclient: _renew_subs 2024-04-26T00:35:31.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0dadf5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:35:31.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0dadf5d640 1 -- 172.21.15.37:0/1432310783 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0da81cf350 con 0x7f0da80a4170 2024-04-26T00:35:31.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0dadf5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:31.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0dadf5d640 1 --2- 172.21.15.37:0/1432310783 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0da8139c60 0x7f0da813a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:31.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0dadf5d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:31.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0dadf5d640 1 -- 172.21.15.37:0/1432310783 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f0da804f510 con 0x7f0da8139c60 2024-04-26T00:35:31.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0d9ffff640 1 -- 172.21.15.37:0/1432310783 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f0d98015cb0 con 0x7f0da80a4170 2024-04-26T00:35:31.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0da77fe640 1 --2- 172.21.15.37:0/1432310783 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0da8139c60 0x7f0da813a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:31.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0da77fe640 10 monclient: get_auth_request con 0x7f0da8139c60 auth_method 0 2024-04-26T00:35:31.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0da77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:31.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0da77fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:31.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0da77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:31.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0d9ffff640 1 --2- 172.21.15.37:0/1432310783 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0d88035e30 0x7f0d880382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:31.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0da77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:31.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0da77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:31.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0da77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:31.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0d9ffff640 1 -- 172.21.15.37:0/1432310783 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(600..600 src has 370..600) ==== 3943+0+0 (secure 0 0 0) 0x7f0d98048e80 con 0x7f0da80a4170 2024-04-26T00:35:31.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0da77fe640 10 monclient(hunting): handle_auth_done global_id 144570 payload 824 2024-04-26T00:35:31.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0da77fe640 1 -- 172.21.15.37:0/1432310783 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0da8131f10 con 0x7f0da8139c60 2024-04-26T00:35:31.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0da7fff640 1 --2- 172.21.15.37:0/1432310783 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0d88035e30 0x7f0d880382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:31.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.133+0000 7f0da7fff640 10 monclient: get_auth_request con 0x7f0d88035e30 auth_method 0 2024-04-26T00:35:31.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.137+0000 7f0da77fe640 1 -- 172.21.15.37:0/1432310783 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0da8139c60 msgr2=0x7f0da813a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:31.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.137+0000 7f0da77fe640 1 --2- 172.21.15.37:0/1432310783 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0da8139c60 0x7f0da813a040 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0d9001cc80 tx=0x7f0d9001ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:31.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.137+0000 7f0d9ffff640 1 -- 172.21.15.37:0/1432310783 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d900bb020 con 0x7f0da8139c60 2024-04-26T00:35:31.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.137+0000 7f0d9ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:31.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.137+0000 7f0d9ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:31.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.137+0000 7f0d9ffff640 10 monclient: dump: 2024-04-26T00:35:31.144 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:31.144 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:31.144 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:31.144 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:31.144 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:31.144 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:31.144 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:31.144 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:31.144 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:31.144 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:31.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.137+0000 7f0d9ffff640 1 -- 172.21.15.37:0/1432310783 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0d900b6880 con 0x7f0da8139c60 2024-04-26T00:35:31.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.137+0000 7f0d9ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:31.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.137+0000 7f0d9ffff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:35:31.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.137+0000 7f0d9ffff640 1 -- 172.21.15.37:0/1432310783 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0da8139c60 msgr2=0x7f0da813a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:31.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.137+0000 7f0d9ffff640 1 --2- 172.21.15.37:0/1432310783 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0da8139c60 0x7f0da813a040 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0d9001cc80 tx=0x7f0d9001ac20 comp rx=0 tx=0).stop 2024-04-26T00:35:31.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.137+0000 7f0d9ffff640 1 -- 172.21.15.37:0/1432310783 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0da8139c60 msgr2=0x7f0da813a040 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:35:31.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.137+0000 7f0d9ffff640 1 --2- 172.21.15.37:0/1432310783 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0da8139c60 0x7f0da813a040 secure :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0d9001cc80 tx=0x7f0d9001ac20 comp rx=0 tx=0).stop 2024-04-26T00:35:31.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.137+0000 7f0da7fff640 1 --2- 172.21.15.37:0/1432310783 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0d88035e30 0x7f0d880382f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0da00094c0 tx=0x7f0da0009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:31.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.141+0000 7f0dadf5d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:35:31.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.141+0000 7f0dadf5d640 1 --2- 172.21.15.37:0/1432310783 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0da813a580 0x7f0da809e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:31.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.141+0000 7f0dadf5d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:31.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.141+0000 7f0dadf5d640 1 -- 172.21.15.37:0/1432310783 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f0da813a990 con 0x7f0da813a580 2024-04-26T00:35:31.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.141+0000 7f0da77fe640 1 --2- 172.21.15.37:0/1432310783 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0da813a580 0x7f0da809e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:31.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.141+0000 7f0da77fe640 10 monclient: get_auth_request con 0x7f0da813a580 auth_method 0 2024-04-26T00:35:31.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.141+0000 7f0da77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:31.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.141+0000 7f0da77fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:31.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.141+0000 7f0da77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:31.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.141+0000 7f0da77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:31.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.141+0000 7f0da77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:31.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.141+0000 7f0da77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:31.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.141+0000 7f0da77fe640 10 monclient(hunting): handle_auth_done global_id 144573 payload 824 2024-04-26T00:35:31.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.141+0000 7f0da77fe640 1 -- 172.21.15.37:0/1432310783 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0da81378f0 con 0x7f0da813a580 2024-04-26T00:35:31.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.145+0000 7f0da77fe640 1 -- 172.21.15.37:0/1432310783 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0da813a580 msgr2=0x7f0da809e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:31.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.145+0000 7f0da77fe640 1 --2- 172.21.15.37:0/1432310783 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0da813a580 0x7f0da809e180 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0d90000c10 tx=0x7f0d900a8000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:31.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.145+0000 7f0d9ffff640 1 -- 172.21.15.37:0/1432310783 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d900bb030 con 0x7f0da813a580 2024-04-26T00:35:31.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.145+0000 7f0d9ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:31.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.145+0000 7f0d9ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:31.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.145+0000 7f0d9ffff640 10 monclient: dump: 2024-04-26T00:35:31.148 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:31.148 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:31.148 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:31.148 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:31.148 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:31.148 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:31.148 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:31.148 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:31.148 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:31.148 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:31.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.145+0000 7f0d9ffff640 1 -- 172.21.15.37:0/1432310783 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7f0d900cc450 con 0x7f0da813a580 2024-04-26T00:35:31.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.145+0000 7f0d9ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:35:31.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.145+0000 7f0d9ffff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:35:31.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.145+0000 7f0d9ffff640 1 -- 172.21.15.37:0/1432310783 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0da813a580 msgr2=0x7f0da809e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:31.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.145+0000 7f0d9ffff640 1 --2- 172.21.15.37:0/1432310783 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0da813a580 0x7f0da809e180 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0d90000c10 tx=0x7f0d900a8000 comp rx=0 tx=0).stop 2024-04-26T00:35:31.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.145+0000 7f0d9ffff640 1 -- 172.21.15.37:0/1432310783 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0da813a580 msgr2=0x7f0da809e180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:35:31.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.145+0000 7f0d9ffff640 1 --2- 172.21.15.37:0/1432310783 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0da813a580 0x7f0da809e180 secure :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0d90000c10 tx=0x7f0d900a8000 comp rx=0 tx=0).stop 2024-04-26T00:35:31.149 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:35:31.149 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:35:31.149 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:35:31.149 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:35:31.149 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 118, 2024-04-26T00:35:31.149 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 217689, 2024-04-26T00:35:31.149 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:35:31.150 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:35:31.150 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:35:31.150 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:31.150 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 1, 2024-04-26T00:35:31.150 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:35:31.150 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:35:31.150 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:35:31.150 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:31.150 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:31.150 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:31.150 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:31.150 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:31.150 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:31.150 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:31.151 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:31.151 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:31.151 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:31.151 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:31.151 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:31.151 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:35:31.151 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:35:31.151 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:31.151 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:31.151 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:31.151 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:31.151 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:31.151 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:31.151 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:31.151 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:31.152 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:31.152 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:31.152 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:31.152 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:31.152 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:31.152 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:35:31.152 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:35:31.152 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:35:31.152 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:35:31.152 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:35:31.152 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:35:31.152 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:35:31.152 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:35:31.152 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:35:31.153 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:35:31.153 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:35:31.153 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:35:31.153 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:35:31.153 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:35:31.153 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:35:31.153 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:35:31.153 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:35:31.153 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:31.153 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:31.153 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:31.153 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:31.153 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:31.153 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:31.154 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:31.154 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:31.154 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:31.154 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:31.154 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:31.154 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:31.154 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:35:31.154 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:31.154 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:35:31.154 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:31.154 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:35:31.154 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:35:31.154 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:31.154 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:31.154 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:31.155 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:31.155 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:35:31.155 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:31.155 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:31.155 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:31.155 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:31.155 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:35:31.155 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:31.155 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:31.155 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:31.155 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:31.155 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:35:31.155 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:35:31.155 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:31.156 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:31.156 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:31.156 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:31.156 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:31.156 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:35:31.156 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:35:31.156 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:31.156 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:31.156 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:31.156 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:31.156 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:35:31.156 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:31.156 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:31.156 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:31.156 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:31.157 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:35:31.157 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:31.157 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:31.157 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:31.157 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:31.157 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:35:31.157 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:35:31.157 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:31.157 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:31.157 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:31.157 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:31.157 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:31.157 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:35:31.157 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:35:31.157 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:31.158 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:31.158 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:31.158 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:31.158 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:35:31.158 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:31.158 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:31.158 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:31.158 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:31.158 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:35:31.158 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:31.158 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:31.158 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:31.158 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:31.158 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:35:31.159 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:35:31.159 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:31.159 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:31.159 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:31.159 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:31.159 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:31.159 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:31.159 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:35:31.159 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:35:31.159 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:31.159 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:31.159 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:31.159 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:35:31.159 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:31.159 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:31.160 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:35:31.160 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:31.160 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:31.160 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:31.160 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:35:31.160 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:31.160 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:31.160 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:35:31.160 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:31.160 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:31.160 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:31.160 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:35:31.160 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:31.160 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:31.160 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:35:31.161 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:31.161 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:31.161 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:31.161 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:35:31.161 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:31.161 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:31.161 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:31.161 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:35:31.161 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:35:31.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.145+0000 7f0dadf5d640 1 -- 172.21.15.37:0/1432310783 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0d88035e30 msgr2=0x7f0d880382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:31.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.145+0000 7f0dadf5d640 1 --2- 172.21.15.37:0/1432310783 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0d88035e30 0x7f0d880382f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0da00094c0 tx=0x7f0da0009190 comp rx=0 tx=0).stop 2024-04-26T00:35:31.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.145+0000 7f0dadf5d640 10 monclient: shutdown 2024-04-26T00:35:31.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.145+0000 7f0dadf5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:31.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.145+0000 7f0dadf5d640 1 -- 172.21.15.37:0/1432310783 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0da80a4170 msgr2=0x7f0da8136e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:31.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.145+0000 7f0dadf5d640 1 --2- 172.21.15.37:0/1432310783 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0da80a4170 0x7f0da8136e70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0d9800be00 tx=0x7f0d98009f70 comp rx=0 tx=0).stop 2024-04-26T00:35:31.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.145+0000 7f0da7fff640 1 -- 172.21.15.37:0/1432310783 reap_dead start 2024-04-26T00:35:31.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.145+0000 7f0da7fff640 1 -- 172.21.15.37:0/1432310783 reap_dead start 2024-04-26T00:35:31.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.145+0000 7f0dadf5d640 1 -- 172.21.15.37:0/1432310783 shutdown_connections 2024-04-26T00:35:31.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.145+0000 7f0dadf5d640 1 -- 172.21.15.37:0/1432310783 >> 172.21.15.37:0/1432310783 conn(0x7f0da80aa0a0 msgr2=0x7f0da80a8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:31.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.145+0000 7f0dadf5d640 1 -- 172.21.15.37:0/1432310783 shutdown_connections 2024-04-26T00:35:31.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.145+0000 7f0dadf5d640 1 -- 172.21.15.37:0/1432310783 wait complete. 2024-04-26T00:35:31.163 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:35:31.164 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-04-26T00:35:31.164 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:35:31.279 INFO:tasks.rados.rados.0.smithi045.stdout:2584: done (1 left) 2024-04-26T00:35:31.279 INFO:tasks.rados.rados.0.smithi045.stdout:2585: read oid 42 snap -1 2024-04-26T00:35:31.279 INFO:tasks.rados.rados.0.smithi045.stdout:2585: expect (ObjNum 869 snap 272 seq_num 869) 2024-04-26T00:35:31.279 INFO:tasks.rados.rados.0.smithi045.stdout:2586: rollback oid 14 current snap is 288 2024-04-26T00:35:31.279 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 14 to 286 2024-04-26T00:35:31.279 INFO:tasks.rados.rados.0.smithi045.stdout:2583: finishing write tid 1 to smithi04526579-23 2024-04-26T00:35:31.279 INFO:tasks.rados.rados.0.smithi045.stdout:2583: finishing write tid 2 to smithi04526579-23 2024-04-26T00:35:31.279 INFO:tasks.rados.rados.0.smithi045.stdout:2583: finishing write tid 3 to smithi04526579-23 2024-04-26T00:35:31.279 INFO:tasks.rados.rados.0.smithi045.stdout:2583: finishing write tid 4 to smithi04526579-23 2024-04-26T00:35:31.280 INFO:tasks.rados.rados.0.smithi045.stdout:2583: finishing write tid 5 to smithi04526579-23 2024-04-26T00:35:31.280 INFO:tasks.rados.rados.0.smithi045.stdout:2583: oid 23 updating version 0 to 1990 2024-04-26T00:35:31.280 INFO:tasks.rados.rados.0.smithi045.stdout:2583: oid 23 updating version 1990 to 1992 2024-04-26T00:35:31.280 INFO:tasks.rados.rados.0.smithi045.stdout:2583: oid 23 version 1992 is already newer than 1989 2024-04-26T00:35:31.280 INFO:tasks.rados.rados.0.smithi045.stdout:2583: oid 23 version 1992 is already newer than 1991 2024-04-26T00:35:31.280 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 1992 (ObjNum 932 snap 288 seq_num 932) dirty exists 2024-04-26T00:35:31.280 INFO:tasks.rados.rados.0.smithi045.stdout:2583: left oid 23 (ObjNum 932 snap 288 seq_num 932) 2024-04-26T00:35:31.280 INFO:tasks.rados.rados.0.smithi045.stdout:2583: done (2 left) 2024-04-26T00:35:31.280 INFO:tasks.rados.rados.0.smithi045.stdout:2587: snap_create 2024-04-26T00:35:31.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.285+0000 7f82aac47640 10 monclient: get_monmap_and_config 2024-04-26T00:35:31.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.285+0000 7f82aac47640 10 monclient: build_initial_monmap 2024-04-26T00:35:31.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.285+0000 7f82aac47640 10 monclient: monmap: 2024-04-26T00:35:31.286 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:31.286 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:31.286 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:35:31.291456+0000 2024-04-26T00:35:31.286 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:35:31.291456+0000 2024-04-26T00:35:31.287 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:31.287 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:31.287 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:35:31.287 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:35:31.287 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:35:31.287 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:35:31.287 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:35:31.287 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:31.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.289+0000 7f82aac47640 1 Processor -- start 2024-04-26T00:35:31.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.289+0000 7f82aac47640 1 -- start start 2024-04-26T00:35:31.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.289+0000 7f82aac47640 10 monclient: init 2024-04-26T00:35:31.288 INFO:tasks.rados.rados.0.smithi045.stdout:2586: finishing rollback tid 0 to smithi04526579-14 2024-04-26T00:35:31.289 INFO:tasks.rados.rados.0.smithi045.stdout:2586: finishing rollback tid 1 to smithi04526579-14 2024-04-26T00:35:31.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.289+0000 7f82aac47640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:31.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.289+0000 7f82aac47640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-04-26T00:35:31.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.289+0000 7f82aac47640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f82a40a0220 0x7f82a409fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:31.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.289+0000 7f82aac47640 10 monclient(hunting): picked mon.noname-a con 0x7f82a40a0220 addr v2:172.21.15.37:3300/0 2024-04-26T00:35:31.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.289+0000 7f82aac47640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f82a40a18b0 addr v1:172.21.15.37:6789/0 2024-04-26T00:35:31.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.289+0000 7f82aac47640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f82a413c1c0 0x7f82a413e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:31.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.289+0000 7f82aac47640 10 monclient(hunting): picked mon.noname-c con 0x7f82a413c1c0 addr v2:172.21.15.45:3300/0 2024-04-26T00:35:31.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.289+0000 7f82aac47640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82a40a0ed0 con 0x7f82a40a18b0 2024-04-26T00:35:31.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.289+0000 7f82aac47640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:31.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.289+0000 7f82aac47640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f82a40a11b0 con 0x7f82a40a0220 2024-04-26T00:35:31.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.289+0000 7f82aac47640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:31.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.289+0000 7f82aac47640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f82a40a1490 con 0x7f82a413c1c0 2024-04-26T00:35:31.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.289+0000 7f82aac47640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:31.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.289+0000 7f82aac47640 10 monclient(hunting): authenticate will time out at 1912.597628s 2024-04-26T00:35:31.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.289+0000 7f82a91e4640 1 -- >> v2:172.21.15.45:3300/0 conn(0x7f82a413c1c0 msgr2=0x7f82a413e5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:35:31.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.289+0000 7f82a91e4640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f82a413c1c0 0x7f82a413e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:35:31.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.289+0000 7f82a3fff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f82a40a18b0 0x7f82a40a0600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:44946/0 (socket says 172.21.15.37:44946) 2024-04-26T00:35:31.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.289+0000 7f82a3fff640 1 -- 172.21.15.37:0/3127432772 learned_addr learned my addr 172.21.15.37:0/3127432772 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:35:31.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.289+0000 7f82a89e3640 1 --2- 172.21.15.37:0/3127432772 >> v2:172.21.15.37:3300/0 conn(0x7f82a40a0220 0x7f82a409fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:31.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.289+0000 7f82a89e3640 10 monclient(hunting): get_auth_request con 0x7f82a40a0220 auth_method 0 2024-04-26T00:35:31.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.289+0000 7f82a89e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:31.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.289+0000 7f82a89e3640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:31.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.289+0000 7f82a89e3640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:31.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.289+0000 7f82a89e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:31.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.289+0000 7f82a89e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:31.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.289+0000 7f82a89e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:31.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.289+0000 7f82a89e3640 10 monclient(hunting): handle_auth_done global_id 144576 payload 274 2024-04-26T00:35:31.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.289+0000 7f82a89e3640 1 -- 172.21.15.37:0/3127432772 >> v2:172.21.15.45:3300/0 conn(0x7f82a413c1c0 msgr2=0x7f82a413e5b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:35:31.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.289+0000 7f82a89e3640 1 --2- 172.21.15.37:0/3127432772 >> v2:172.21.15.45:3300/0 conn(0x7f82a413c1c0 0x7f82a413e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:31.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.289+0000 7f82a89e3640 1 -- 172.21.15.37:0/3127432772 >> v1:172.21.15.37:6789/0 conn(0x7f82a40a18b0 legacy=0x7f82a40a0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:31.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.289+0000 7f82a89e3640 10 monclient: _finish_hunting 0 2024-04-26T00:35:31.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.289+0000 7f82a89e3640 1 monclient: found mon.noname-a 2024-04-26T00:35:31.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.289+0000 7f82a89e3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:31.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.289+0000 7f82a89e3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:35:31.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.289+0000 7f82a89e3640 1 -- 172.21.15.37:0/3127432772 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82a4072d50 con 0x7f82a40a0220 2024-04-26T00:35:31.293 INFO:tasks.rados.rados.0.smithi045.stdout:2586: finishing rollback tid 2 to smithi04526579-14 2024-04-26T00:35:31.293 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 1168 (ObjNum 866 snap 269 seq_num 866) dirty exists 2024-04-26T00:35:31.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.297+0000 7f82a89e3640 1 -- 172.21.15.37:0/3127432772 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f82a40a0220 msgr2=0x7f82a409fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:31.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.297+0000 7f82a89e3640 1 --2- 172.21.15.37:0/3127432772 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f82a40a0220 0x7f82a409fc50 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f82a406dd00 tx=0x7f829401c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8204b47c7dba3266 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:31.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.297+0000 7f82a37fe640 1 -- 172.21.15.37:0/3127432772 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82940b0070 con 0x7f82a40a0220 2024-04-26T00:35:31.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.297+0000 7f82a37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:31.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.297+0000 7f82a37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:35:31.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.297+0000 7f82a37fe640 10 monclient: dump: 2024-04-26T00:35:31.296 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:31.296 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:31.296 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:31.296 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:31.296 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:31.296 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:31.296 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:31.297 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:31.297 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:31.297 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:31.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.297+0000 7f82a37fe640 10 monclient: _finish_auth 0 2024-04-26T00:35:31.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.297+0000 7f82a37fe640 10 monclient: _check_auth_tickets 2024-04-26T00:35:31.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.297+0000 7f82a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:31.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.297+0000 7f82a37fe640 1 -- 172.21.15.37:0/3127432772 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82940ab600 con 0x7f82a40a0220 2024-04-26T00:35:31.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.297+0000 7f82aac47640 5 monclient: authenticate success, global_id 144576 2024-04-26T00:35:31.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.297+0000 7f82aac47640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:35:31.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.297+0000 7f82a37fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:31.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.297+0000 7f82aac47640 10 monclient: get_monmap_and_config success 2024-04-26T00:35:31.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.297+0000 7f82a37fe640 1 -- 172.21.15.37:0/3127432772 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82940ab9d0 con 0x7f82a40a0220 2024-04-26T00:35:31.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.297+0000 7f82aac47640 4 set_mon_vals no callback set 2024-04-26T00:35:31.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.297+0000 7f82a37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:31.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.297+0000 7f82a37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:31.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.297+0000 7f82a37fe640 10 monclient: dump: 2024-04-26T00:35:31.298 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:31.298 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:31.298 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:31.298 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:31.298 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:31.298 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:31.298 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:31.298 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:31.298 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:31.298 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:31.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.297+0000 7f82aac47640 10 monclient: shutdown 2024-04-26T00:35:31.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.297+0000 7f82aac47640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:31.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.297+0000 7f82aac47640 1 -- 172.21.15.37:0/3127432772 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f82a40a0220 msgr2=0x7f82a409fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:31.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.297+0000 7f82aac47640 1 --2- 172.21.15.37:0/3127432772 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f82a40a0220 0x7f82a409fc50 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f82a406dd00 tx=0x7f829401c320 comp rx=0 tx=0).stop 2024-04-26T00:35:31.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.297+0000 7f82aac47640 1 -- 172.21.15.37:0/3127432772 shutdown_connections 2024-04-26T00:35:31.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.297+0000 7f82aac47640 1 --2- 172.21.15.37:0/3127432772 >> v2:172.21.15.45:3300/0 conn(0x7f82a413c1c0 0x7f82a413e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:31.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.297+0000 7f82aac47640 1 --2- 172.21.15.37:0/3127432772 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f82a40a0220 0x7f82a409fc50 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:31.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.297+0000 7f82aac47640 1 -- 172.21.15.37:0/3127432772 >> 172.21.15.37:0/3127432772 conn(0x7f82a40a1d50 msgr2=0x7f82a40a2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:31.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.297+0000 7f82aac47640 1 -- 172.21.15.37:0/3127432772 shutdown_connections 2024-04-26T00:35:31.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.297+0000 7f82aac47640 1 -- 172.21.15.37:0/3127432772 wait complete. 2024-04-26T00:35:31.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.297+0000 7f82aac47640 10 monclient: build_initial_monmap 2024-04-26T00:35:31.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.297+0000 7f82aac47640 10 monclient: monmap: 2024-04-26T00:35:31.301 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:31.301 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:35:31.301 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:35:31.301 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:35:31.301 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:31.301 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:35:31.301 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:35:31.302 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:35:31.302 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:35:31.302 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:31.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.297+0000 7f82aac47640 1 Processor -- start 2024-04-26T00:35:31.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.297+0000 7f82aac47640 1 -- start start 2024-04-26T00:35:31.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.297+0000 7f82aac47640 10 monclient: init 2024-04-26T00:35:31.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82aac47640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:31.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82aac47640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:35:31.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82aac47640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82a40a0220 0x7f82a41ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:31.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82aac47640 10 monclient(hunting): picked mon.noname-b con 0x7f82a40a0220 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:35:31.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82aac47640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82a40a18b0 0x7f82a4130e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:31.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82aac47640 10 monclient(hunting): picked mon.noname-c con 0x7f82a40a18b0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:31.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82aac47640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f82a413c1c0 0x7f82a41cb980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:31.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82aac47640 10 monclient(hunting): picked mon.noname-a con 0x7f82a413c1c0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:35:31.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82aac47640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:31.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82aac47640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f82a41416a0 con 0x7f82a413c1c0 2024-04-26T00:35:31.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82aac47640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:31.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82aac47640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f82a4141520 con 0x7f82a40a0220 2024-04-26T00:35:31.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82aac47640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:31.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82aac47640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f82a4141820 con 0x7f82a40a18b0 2024-04-26T00:35:31.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82aac47640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:31.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82aac47640 10 monclient(hunting): authenticate will time out at 1912.608830s 2024-04-26T00:35:31.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82a89e3640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82a40a0220 msgr2=0x7f82a41ceec0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:35:31.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82a89e3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82a40a0220 0x7f82a41ceec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:35:31.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82a3fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82a40a18b0 0x7f82a4130e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:31.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82a91e4640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f82a413c1c0 0x7f82a41cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:31.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82a3fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82a40a18b0 0x7f82a4130e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:44478/0 (socket says 172.21.15.37:44478) 2024-04-26T00:35:31.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82a3fff640 1 -- 172.21.15.37:0/298959645 learned_addr learned my addr 172.21.15.37:0/298959645 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:31.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82a3fff640 10 monclient(hunting): get_auth_request con 0x7f82a40a18b0 auth_method 0 2024-04-26T00:35:31.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:31.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82a3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:31.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:31.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:31.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:31.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:31.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82a3fff640 10 monclient(hunting): handle_auth_done global_id 184163 payload 824 2024-04-26T00:35:31.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82a3fff640 1 -- 172.21.15.37:0/298959645 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82a40a0220 msgr2=0x7f82a41ceec0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:35:31.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82a3fff640 1 --2- 172.21.15.37:0/298959645 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f82a40a0220 0x7f82a41ceec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:31.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82a3fff640 1 -- 172.21.15.37:0/298959645 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f82a413c1c0 msgr2=0x7f82a41cb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:31.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82a3fff640 1 --2- 172.21.15.37:0/298959645 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f82a413c1c0 0x7f82a41cb980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:31.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82a3fff640 10 monclient: _finish_hunting 0 2024-04-26T00:35:31.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82a3fff640 1 monclient: found mon.noname-c 2024-04-26T00:35:31.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:31.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82a3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:35:31.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82a3fff640 1 -- 172.21.15.37:0/298959645 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82a4137ae0 con 0x7f82a40a18b0 2024-04-26T00:35:31.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82a91e4640 10 monclient: get_auth_request con 0x7f82a413c1c0 auth_method 0 2024-04-26T00:35:31.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.301+0000 7f82a91e4640 1 --2- 172.21.15.37:0/298959645 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f82a413c1c0 0x7f82a41cb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:35:31.306 INFO:tasks.rados.rados.0.smithi045.stdout:2585: expect (ObjNum 869 snap 272 seq_num 869) 2024-04-26T00:35:31.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.305+0000 7f82a3fff640 1 -- 172.21.15.37:0/298959645 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82a40a18b0 msgr2=0x7f82a4130e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:31.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.305+0000 7f82a3fff640 1 --2- 172.21.15.37:0/298959645 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82a40a18b0 0x7f82a4130e20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f829800c470 tx=0x7f829800c930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:31.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.305+0000 7f82a17fa640 1 -- 172.21.15.37:0/298959645 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8298017070 con 0x7f82a40a18b0 2024-04-26T00:35:31.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.305+0000 7f82a17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:31.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.305+0000 7f82a3fff640 1 --2- 172.21.15.37:0/298959645 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82a40a18b0 0x7f82a4130e20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f829800c470 tx=0x7f829800c930 comp rx=0 tx=0).queue_received will delay after 1.71409e+09 on 0x7f82980070d0 config(0 keys) 2024-04-26T00:35:31.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.305+0000 7f82a17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:35:31.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.305+0000 7f82a17fa640 10 monclient: dump: 2024-04-26T00:35:31.309 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:31.309 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:31.309 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:31.309 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:31.309 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:31.309 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:31.309 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:31.309 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:31.309 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:31.309 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:31.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.305+0000 7f82a17fa640 10 monclient: _finish_auth 0 2024-04-26T00:35:31.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.305+0000 7f82a17fa640 10 monclient: _check_auth_tickets 2024-04-26T00:35:31.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.305+0000 7f82a17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:31.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.305+0000 7f82a17fa640 1 -- 172.21.15.37:0/298959645 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82980070d0 con 0x7f82a40a18b0 2024-04-26T00:35:31.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.305+0000 7f82aac47640 5 monclient: authenticate success, global_id 184163 2024-04-26T00:35:31.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.305+0000 7f82aac47640 10 monclient: _renew_subs 2024-04-26T00:35:31.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.305+0000 7f82aac47640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:35:31.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.305+0000 7f82aac47640 1 -- 172.21.15.37:0/298959645 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82a409eeb0 con 0x7f82a40a18b0 2024-04-26T00:35:31.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.305+0000 7f82a17fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:31.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.305+0000 7f82aac47640 10 monclient: _renew_subs 2024-04-26T00:35:31.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.305+0000 7f82aac47640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:35:31.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.305+0000 7f82aac47640 1 -- 172.21.15.37:0/298959645 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f82a4137800 con 0x7f82a40a18b0 2024-04-26T00:35:31.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.305+0000 7f82aac47640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:31.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.305+0000 7f82aac47640 1 --2- 172.21.15.37:0/298959645 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82a4131360 0x7f82a41cbec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:31.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.305+0000 7f82aac47640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:31.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.305+0000 7f82aac47640 1 -- 172.21.15.37:0/298959645 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f82a404f510 con 0x7f82a4131360 2024-04-26T00:35:31.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.305+0000 7f82a89e3640 1 --2- 172.21.15.37:0/298959645 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82a4131360 0x7f82a41cbec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:31.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.305+0000 7f82a89e3640 10 monclient: get_auth_request con 0x7f82a4131360 auth_method 0 2024-04-26T00:35:31.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.305+0000 7f82a89e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:31.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.305+0000 7f82a89e3640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:31.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.305+0000 7f82a89e3640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:31.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.305+0000 7f82a17fa640 1 -- 172.21.15.37:0/298959645 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82980074c0 con 0x7f82a40a18b0 2024-04-26T00:35:31.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.305+0000 7f82a17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:31.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.305+0000 7f82a17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:35:31.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.305+0000 7f82a17fa640 10 monclient: dump: 2024-04-26T00:35:31.311 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:31.311 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:31.311 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:31.311 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:31.311 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:31.312 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:31.312 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:31.312 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:31.312 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:31.312 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:31.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.305+0000 7f82a17fa640 1 -- 172.21.15.37:0/298959645 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f829801a070 con 0x7f82a40a18b0 2024-04-26T00:35:31.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.305+0000 7f82a89e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:31.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.305+0000 7f82a89e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:31.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.305+0000 7f82a17fa640 1 --2- 172.21.15.37:0/298959645 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8290035e30 0x7f82900382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:31.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.305+0000 7f82a89e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:31.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.305+0000 7f82a27fc640 4 set_mon_vals no callback set 2024-04-26T00:35:31.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.305+0000 7f82a89e3640 10 monclient(hunting): handle_auth_done global_id 184166 payload 824 2024-04-26T00:35:31.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.305+0000 7f82a89e3640 1 -- 172.21.15.37:0/298959645 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f82a409b790 con 0x7f82a4131360 2024-04-26T00:35:31.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.309+0000 7f82a91e4640 1 --2- 172.21.15.37:0/298959645 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8290035e30 0x7f82900382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:31.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.309+0000 7f82a91e4640 10 monclient: get_auth_request con 0x7f8290035e30 auth_method 0 2024-04-26T00:35:31.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.309+0000 7f82a89e3640 1 -- 172.21.15.37:0/298959645 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82a4131360 msgr2=0x7f82a41cbec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:31.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.309+0000 7f82a89e3640 1 --2- 172.21.15.37:0/298959645 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82a4131360 0x7f82a41cbec0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f82940ab000 tx=0x7f829401ab60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:31.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.309+0000 7f82a17fa640 1 -- 172.21.15.37:0/298959645 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82940b9070 con 0x7f82a4131360 2024-04-26T00:35:31.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.309+0000 7f82a17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:31.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.309+0000 7f82a17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:35:31.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.309+0000 7f82a17fa640 10 monclient: dump: 2024-04-26T00:35:31.313 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:31.313 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:31.313 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:31.313 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:31.313 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:31.313 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:31.313 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:31.314 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:31.314 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:31.314 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:31.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.309+0000 7f82a17fa640 1 -- 172.21.15.37:0/298959645 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f82940b5680 con 0x7f82a4131360 2024-04-26T00:35:31.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.309+0000 7f82a17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:31.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.309+0000 7f82a17fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:35:31.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.309+0000 7f82a17fa640 1 -- 172.21.15.37:0/298959645 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82a4131360 msgr2=0x7f82a41cbec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:31.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.309+0000 7f82a17fa640 1 --2- 172.21.15.37:0/298959645 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82a4131360 0x7f82a41cbec0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f82940ab000 tx=0x7f829401ab60 comp rx=0 tx=0).stop 2024-04-26T00:35:31.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.309+0000 7f82a17fa640 1 -- 172.21.15.37:0/298959645 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82a4131360 msgr2=0x7f82a41cbec0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:35:31.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.309+0000 7f82a17fa640 1 --2- 172.21.15.37:0/298959645 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82a4131360 0x7f82a41cbec0 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f82940ab000 tx=0x7f829401ab60 comp rx=0 tx=0).stop 2024-04-26T00:35:31.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.309+0000 7f82a91e4640 1 --2- 172.21.15.37:0/298959645 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8290035e30 0x7f82900382f0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f829c003cc0 tx=0x7f829c0072e0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:31.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.317+0000 7f82aac47640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:35:31.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.317+0000 7f82aac47640 1 --2- 172.21.15.37:0/298959645 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82a4136850 0x7f82a41cc400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:31.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.317+0000 7f82aac47640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:31.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.317+0000 7f82aac47640 1 -- 172.21.15.37:0/298959645 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f82a41363b0 con 0x7f82a4136850 2024-04-26T00:35:31.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.317+0000 7f82a89e3640 1 --2- 172.21.15.37:0/298959645 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82a4136850 0x7f82a41cc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:31.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.317+0000 7f82a89e3640 10 monclient: get_auth_request con 0x7f82a4136850 auth_method 0 2024-04-26T00:35:31.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.317+0000 7f82a89e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:31.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.317+0000 7f82a89e3640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:31.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.317+0000 7f82a89e3640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:31.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.317+0000 7f82a89e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:31.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.317+0000 7f82a89e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:31.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.317+0000 7f82a89e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:31.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.317+0000 7f82a89e3640 10 monclient(hunting): handle_auth_done global_id 184169 payload 824 2024-04-26T00:35:31.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.317+0000 7f82a89e3640 1 -- 172.21.15.37:0/298959645 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f82a406ba50 con 0x7f82a4136850 2024-04-26T00:35:31.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.321+0000 7f82a89e3640 1 -- 172.21.15.37:0/298959645 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82a4136850 msgr2=0x7f82a41cc400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:31.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.321+0000 7f82a89e3640 1 --2- 172.21.15.37:0/298959645 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82a4136850 0x7f82a41cc400 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8294019f50 tx=0x7f8294009190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:31.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.321+0000 7f82a17fa640 1 -- 172.21.15.37:0/298959645 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82940b9040 con 0x7f82a4136850 2024-04-26T00:35:31.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.321+0000 7f82a17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:31.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.321+0000 7f82a17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:35:31.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.321+0000 7f82a17fa640 10 monclient: dump: 2024-04-26T00:35:31.318 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:31.319 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:31.319 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:31.319 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:31.319 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:31.319 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:31.319 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:31.319 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:31.319 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:31.319 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:31.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.321+0000 7f82a17fa640 1 -- 172.21.15.37:0/298959645 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4706 (secure 0 0 0) 0x7f82940c8430 con 0x7f82a4136850 2024-04-26T00:35:31.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.321+0000 7f82a17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:35:31.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.321+0000 7f82a17fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:35:31.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.321+0000 7f82a17fa640 1 -- 172.21.15.37:0/298959645 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82a4136850 msgr2=0x7f82a41cc400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:31.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.321+0000 7f82a17fa640 1 --2- 172.21.15.37:0/298959645 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82a4136850 0x7f82a41cc400 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8294019f50 tx=0x7f8294009190 comp rx=0 tx=0).stop 2024-04-26T00:35:31.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.321+0000 7f82a17fa640 1 -- 172.21.15.37:0/298959645 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82a4136850 msgr2=0x7f82a41cc400 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:35:31.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.321+0000 7f82a17fa640 1 --2- 172.21.15.37:0/298959645 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82a4136850 0x7f82a41cc400 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8294019f50 tx=0x7f8294009190 comp rx=0 tx=0).stop 2024-04-26T00:35:31.320 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:35:31.320 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:35:31.320 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:35:31.320 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:35:31.320 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 118, 2024-04-26T00:35:31.320 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 30467, 2024-04-26T00:35:31.320 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:35:31.320 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:35:31.321 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:35:31.321 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:31.321 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 1, 2024-04-26T00:35:31.321 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:35:31.321 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:35:31.321 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:35:31.321 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:31.321 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:31.321 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:31.321 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:31.321 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:31.321 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:31.321 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:31.321 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:31.321 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:31.322 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:31.322 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:31.322 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:31.322 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:35:31.322 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:35:31.322 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:31.322 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:31.322 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:31.322 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:31.322 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:31.322 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:31.322 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:31.322 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:31.322 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:31.323 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:31.323 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:31.323 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:31.323 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:31.323 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:35:31.323 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:35:31.323 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:35:31.323 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:35:31.323 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:35:31.323 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:35:31.323 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:35:31.323 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:35:31.323 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:35:31.323 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:35:31.324 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:35:31.324 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:35:31.324 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:35:31.324 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:35:31.324 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:35:31.324 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:35:31.324 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:35:31.324 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:31.324 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:31.324 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:31.324 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:31.325 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:31.325 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:31.325 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:31.325 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:31.325 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:31.325 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:31.325 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:31.325 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:31.325 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:35:31.325 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:31.325 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:35:31.325 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:31.326 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:35:31.326 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:35:31.326 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:31.326 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:31.326 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:31.326 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:31.326 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:35:31.326 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:31.326 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:31.326 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:31.326 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:31.326 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:35:31.327 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:31.327 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:31.327 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:31.327 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:31.327 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:35:31.327 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:35:31.327 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:31.327 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:31.327 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:31.327 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:31.327 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:31.327 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:35:31.327 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:35:31.327 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:31.328 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:31.328 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:31.328 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:31.328 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:35:31.328 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:31.328 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:31.328 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:31.328 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:31.328 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:35:31.328 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:31.328 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:31.328 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:31.328 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:31.328 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:35:31.329 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:35:31.329 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:31.329 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:31.329 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:31.329 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:31.329 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:31.329 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:35:31.329 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:35:31.329 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:31.329 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:31.329 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:31.329 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:31.329 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:35:31.329 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:31.329 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:31.330 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:31.330 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:31.330 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:35:31.330 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:31.330 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:31.330 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:31.330 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:31.330 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:35:31.330 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:35:31.330 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:31.330 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:31.330 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:31.330 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:31.331 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:31.331 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:31.331 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:35:31.331 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:35:31.331 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:31.331 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:31.331 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:31.331 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:35:31.331 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:31.331 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:31.331 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:35:31.331 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:31.331 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:31.331 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:31.331 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:35:31.332 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:31.332 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:31.332 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:35:31.332 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:31.332 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:31.332 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:31.332 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:35:31.332 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:31.332 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:31.332 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:31.332 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:35:31.332 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:35:31.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.321+0000 7f82aac47640 1 -- 172.21.15.37:0/298959645 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8290035e30 msgr2=0x7f82900382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:31.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.321+0000 7f82aac47640 1 --2- 172.21.15.37:0/298959645 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8290035e30 0x7f82900382f0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f829c003cc0 tx=0x7f829c0072e0 comp rx=0 tx=0).stop 2024-04-26T00:35:31.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.321+0000 7f82aac47640 10 monclient: shutdown 2024-04-26T00:35:31.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.321+0000 7f82aac47640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:31.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.321+0000 7f82aac47640 1 -- 172.21.15.37:0/298959645 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82a40a18b0 msgr2=0x7f82a4130e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:31.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.321+0000 7f82aac47640 1 --2- 172.21.15.37:0/298959645 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82a40a18b0 0x7f82a4130e20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f829800c470 tx=0x7f829800c930 comp rx=0 tx=0).stop 2024-04-26T00:35:31.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.321+0000 7f82a91e4640 1 -- 172.21.15.37:0/298959645 reap_dead start 2024-04-26T00:35:31.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.321+0000 7f82a17fa640 1 -- 172.21.15.37:0/298959645 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(601..601 src has 370..601) ==== 3959+0+0 (unknown 0 0 0) 0x7f8298048120 con 0x7f82a40a18b0 2024-04-26T00:35:31.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.321+0000 7f82aac47640 1 -- 172.21.15.37:0/298959645 shutdown_connections 2024-04-26T00:35:31.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.321+0000 7f82aac47640 1 --2- 172.21.15.37:0/298959645 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82a40a18b0 0x7f82a4130e20 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:31.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.321+0000 7f82aac47640 1 -- 172.21.15.37:0/298959645 >> 172.21.15.37:0/298959645 conn(0x7f82a40a1d50 msgr2=0x7f82a409cbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:31.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.321+0000 7f82aac47640 1 -- 172.21.15.37:0/298959645 shutdown_connections 2024-04-26T00:35:31.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:31.321+0000 7f82aac47640 1 -- 172.21.15.37:0/298959645 wait complete. 2024-04-26T00:35:31.335 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:35:31.335 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-04-26T00:35:32.288 INFO:tasks.rados.rados.0.smithi045.stdout:2585: done (2 left) 2024-04-26T00:35:32.289 INFO:tasks.rados.rados.0.smithi045.stdout:2586: done (1 left) 2024-04-26T00:35:32.289 INFO:tasks.rados.rados.0.smithi045.stdout:2587: done (0 left) 2024-04-26T00:35:32.289 INFO:tasks.rados.rados.0.smithi045.stdout:2588: read oid 13 snap 256 2024-04-26T00:35:32.289 INFO:tasks.rados.rados.0.smithi045.stdout:2588: expect (ObjNum 818 snap 256 seq_num 818) 2024-04-26T00:35:32.289 INFO:tasks.rados.rados.0.smithi045.stdout:2589: copy_from oid 4 from oid 14 current snap is 289 2024-04-26T00:35:32.289 INFO:tasks.rados.rados.0.smithi045.stdout:2590: rollback oid 34 current snap is 289 2024-04-26T00:35:32.289 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 34 to 288 2024-04-26T00:35:32.289 INFO:tasks.rados.rados.0.smithi045.stdout:2591: copy_from oid 45 from oid 44 current snap is 289 2024-04-26T00:35:32.289 INFO:tasks.rados.rados.0.smithi045.stdout:2592: rollback oid 6 current snap is 289 2024-04-26T00:35:32.289 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 6 to 288 2024-04-26T00:35:32.289 INFO:tasks.rados.rados.0.smithi045.stdout:2593: write oid 20 current snap is 289 2024-04-26T00:35:32.289 INFO:tasks.rados.rados.0.smithi045.stdout:2593: seq_num 933 ranges {480931=402383,1313935=515645,2488711=452788,3443230=237701} 2024-04-26T00:35:32.299 INFO:tasks.rados.rados.0.smithi045.stdout:2593: writing smithi04526579-20 from 480931 to 883314 tid 1 2024-04-26T00:35:32.310 INFO:tasks.rados.rados.0.smithi045.stdout:2593: writing smithi04526579-20 from 1313935 to 1829580 tid 2 2024-04-26T00:35:32.326 INFO:tasks.rados.rados.0.smithi045.stdout:2593: writing smithi04526579-20 from 2488711 to 2941499 tid 3 2024-04-26T00:35:32.338 INFO:tasks.rados.rados.0.smithi045.stdout:2593: writing smithi04526579-20 from 3443230 to 3680931 tid 4 2024-04-26T00:35:32.338 INFO:tasks.rados.rados.0.smithi045.stdout:2594: read oid 46 snap -1 2024-04-26T00:35:32.339 INFO:tasks.rados.rados.0.smithi045.stdout:2594: expect (ObjNum 913 snap 282 seq_num 913) 2024-04-26T00:35:32.339 INFO:tasks.rados.rados.0.smithi045.stdout:2595: rollback oid 39 current snap is 289 2024-04-26T00:35:32.339 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 39 to 256 2024-04-26T00:35:32.339 INFO:tasks.rados.rados.0.smithi045.stdout:2592: finishing rollback tid 0 to smithi04526579-6 2024-04-26T00:35:32.339 INFO:tasks.rados.rados.0.smithi045.stdout:2592: finishing rollback tid 1 to smithi04526579-6 2024-04-26T00:35:32.339 INFO:tasks.rados.rados.0.smithi045.stdout:2592: finishing rollback tid 2 to smithi04526579-6 2024-04-26T00:35:32.339 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 1980 (ObjNum 542 snap 167 seq_num 542) dirty exists 2024-04-26T00:35:32.339 INFO:tasks.rados.rados.0.smithi045.stdout:2590: finishing rollback tid 0 to smithi04526579-34 2024-04-26T00:35:32.339 INFO:tasks.rados.rados.0.smithi045.stdout:2590: finishing rollback tid 1 to smithi04526579-34 2024-04-26T00:35:32.339 INFO:tasks.rados.rados.0.smithi045.stdout:2590: finishing rollback tid 2 to smithi04526579-34 2024-04-26T00:35:32.339 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 1990 (ObjNum 893 snap 275 seq_num 893) dirty exists 2024-04-26T00:35:32.340 INFO:tasks.rados.rados.0.smithi045.stdout:2588: expect (ObjNum 818 snap 256 seq_num 818) 2024-04-26T00:35:32.594 INFO:tasks.rados.rados.0.smithi045.stdout:2593: finishing write tid 1 to smithi04526579-20 2024-04-26T00:35:32.594 INFO:tasks.rados.rados.0.smithi045.stdout:2589: finishing copy_from racing read to smithi04526579-4 2024-04-26T00:35:32.594 INFO:tasks.rados.rados.0.smithi045.stdout:2593: finishing write tid 2 to smithi04526579-20 2024-04-26T00:35:32.594 INFO:tasks.rados.rados.0.smithi045.stdout:2589: finishing copy_from to smithi04526579-4 2024-04-26T00:35:32.594 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 1168 (ObjNum 866 snap 269 seq_num 866) dirty exists 2024-04-26T00:35:32.594 INFO:tasks.rados.rados.0.smithi045.stdout:2591: finishing copy_from racing read to smithi04526579-45 2024-04-26T00:35:32.594 INFO:tasks.rados.rados.0.smithi045.stdout:2593: finishing write tid 3 to smithi04526579-20 2024-04-26T00:35:32.594 INFO:tasks.rados.rados.0.smithi045.stdout:2591: finishing copy_from to smithi04526579-45 2024-04-26T00:35:32.594 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 2154 (ObjNum 898 snap 276 seq_num 898) dirty exists 2024-04-26T00:35:32.594 INFO:tasks.rados.rados.0.smithi045.stdout:2594: expect (ObjNum 913 snap 282 seq_num 913) 2024-04-26T00:35:32.832 INFO:tasks.rados.rados.0.smithi045.stdout:2593: finishing write tid 4 to smithi04526579-20 2024-04-26T00:35:32.832 INFO:tasks.rados.rados.0.smithi045.stdout:2593: finishing write tid 5 to smithi04526579-20 2024-04-26T00:35:32.833 INFO:tasks.rados.rados.0.smithi045.stdout:2593: finishing write tid 6 to smithi04526579-20 2024-04-26T00:35:32.833 INFO:tasks.rados.rados.0.smithi045.stdout:2593: oid 20 updating version 0 to 2159 2024-04-26T00:35:32.833 INFO:tasks.rados.rados.0.smithi045.stdout:2593: oid 20 version 2159 is already newer than 2158 2024-04-26T00:35:32.833 INFO:tasks.rados.rados.0.smithi045.stdout:2593: oid 20 version 2159 is already newer than 2157 2024-04-26T00:35:32.833 INFO:tasks.rados.rados.0.smithi045.stdout:2593: oid 20 updating version 2159 to 2160 2024-04-26T00:35:32.833 INFO:tasks.rados.rados.0.smithi045.stdout:2593: oid 20 updating version 2160 to 2161 2024-04-26T00:35:32.833 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 2161 (ObjNum 933 snap 289 seq_num 933) dirty exists 2024-04-26T00:35:32.833 INFO:tasks.rados.rados.0.smithi045.stdout:2593: left oid 20 (ObjNum 933 snap 289 seq_num 933) 2024-04-26T00:35:32.833 INFO:tasks.rados.rados.0.smithi045.stdout:2595: finishing rollback tid 0 to smithi04526579-39 2024-04-26T00:35:32.833 INFO:tasks.rados.rados.0.smithi045.stdout:2595: finishing rollback tid 1 to smithi04526579-39 2024-04-26T00:35:32.833 INFO:tasks.rados.rados.0.smithi045.stdout:2595: finishing rollback tid 2 to smithi04526579-39 2024-04-26T00:35:32.833 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 2164 (ObjNum 656 snap 203 seq_num 656) dirty exists 2024-04-26T00:35:32.833 INFO:tasks.rados.rados.0.smithi045.stdout:2588: done (7 left) 2024-04-26T00:35:32.833 INFO:tasks.rados.rados.0.smithi045.stdout:2589: done (6 left) 2024-04-26T00:35:32.834 INFO:tasks.rados.rados.0.smithi045.stdout:2590: done (5 left) 2024-04-26T00:35:32.834 INFO:tasks.rados.rados.0.smithi045.stdout:2591: done (4 left) 2024-04-26T00:35:32.834 INFO:tasks.rados.rados.0.smithi045.stdout:2592: done (3 left) 2024-04-26T00:35:32.834 INFO:tasks.rados.rados.0.smithi045.stdout:2593: done (2 left) 2024-04-26T00:35:32.834 INFO:tasks.rados.rados.0.smithi045.stdout:2594: done (1 left) 2024-04-26T00:35:32.834 INFO:tasks.rados.rados.0.smithi045.stdout:2595: done (0 left) 2024-04-26T00:35:32.834 INFO:tasks.rados.rados.0.smithi045.stdout:2596: rollback oid 34 current snap is 289 2024-04-26T00:35:32.834 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 34 to 288 2024-04-26T00:35:32.834 INFO:tasks.rados.rados.0.smithi045.stdout:2597: write oid 44 current snap is 289 2024-04-26T00:35:32.834 INFO:tasks.rados.rados.0.smithi045.stdout:2597: seq_num 934 ranges {497738=477632,1772417=775656,3316134=381604} 2024-04-26T00:35:32.847 INFO:tasks.rados.rados.0.smithi045.stdout:2597: writing smithi04526579-44 from 497738 to 975370 tid 1 2024-04-26T00:35:32.869 INFO:tasks.rados.rados.0.smithi045.stdout:2597: writing smithi04526579-44 from 1772417 to 2548073 tid 2 2024-04-26T00:35:32.885 INFO:tasks.rados.rados.0.smithi045.stdout:2597: writing smithi04526579-44 from 3316134 to 3697738 tid 3 2024-04-26T00:35:32.885 INFO:tasks.rados.rados.0.smithi045.stdout:2598: snap_create 2024-04-26T00:35:32.886 INFO:tasks.rados.rados.0.smithi045.stdout:2596: finishing rollback tid 0 to smithi04526579-34 2024-04-26T00:35:32.886 INFO:tasks.rados.rados.0.smithi045.stdout:2596: finishing rollback tid 1 to smithi04526579-34 2024-04-26T00:35:32.886 INFO:tasks.rados.rados.0.smithi045.stdout:2596: finishing rollback tid 2 to smithi04526579-34 2024-04-26T00:35:32.886 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 1993 (ObjNum 893 snap 275 seq_num 893) dirty exists 2024-04-26T00:35:32.886 INFO:tasks.rados.rados.0.smithi045.stdout:2597: finishing write tid 1 to smithi04526579-44 2024-04-26T00:35:32.886 INFO:tasks.rados.rados.0.smithi045.stdout:2597: finishing write tid 2 to smithi04526579-44 2024-04-26T00:35:32.893 INFO:tasks.rados.rados.0.smithi045.stdout:2597: finishing write tid 3 to smithi04526579-44 2024-04-26T00:35:32.893 INFO:tasks.rados.rados.0.smithi045.stdout:2597: finishing write tid 4 to smithi04526579-44 2024-04-26T00:35:32.894 INFO:tasks.rados.rados.0.smithi045.stdout:2597: finishing write tid 5 to smithi04526579-44 2024-04-26T00:35:32.894 INFO:tasks.rados.rados.0.smithi045.stdout:2597: oid 44 updating version 0 to 2166 2024-04-26T00:35:32.894 INFO:tasks.rados.rados.0.smithi045.stdout:2597: oid 44 version 2166 is already newer than 2165 2024-04-26T00:35:32.894 INFO:tasks.rados.rados.0.smithi045.stdout:2597: oid 44 updating version 2166 to 2167 2024-04-26T00:35:32.894 INFO:tasks.rados.rados.0.smithi045.stdout:2597: oid 44 updating version 2167 to 2168 2024-04-26T00:35:32.894 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 2168 (ObjNum 934 snap 289 seq_num 934) dirty exists 2024-04-26T00:35:32.894 INFO:tasks.rados.rados.0.smithi045.stdout:2597: left oid 44 (ObjNum 934 snap 289 seq_num 934) 2024-04-26T00:35:33.300 INFO:tasks.rados.rados.0.smithi045.stdout:2596: done (2 left) 2024-04-26T00:35:33.300 INFO:tasks.rados.rados.0.smithi045.stdout:2597: done (1 left) 2024-04-26T00:35:33.300 INFO:tasks.rados.rados.0.smithi045.stdout:2598: done (0 left) 2024-04-26T00:35:33.300 INFO:tasks.rados.rados.0.smithi045.stdout:2599: delete oid 47 current snap is 290 2024-04-26T00:35:33.306 INFO:tasks.rados.rados.0.smithi045.stdout:2599: done (0 left) 2024-04-26T00:35:33.306 INFO:tasks.rados.rados.0.smithi045.stdout:2600: write oid 28 current snap is 290 2024-04-26T00:35:33.307 INFO:tasks.rados.rados.0.smithi045.stdout:2600: seq_num 936 ranges {531352=744483,2005734=779325,3487333=244019} 2024-04-26T00:35:33.327 INFO:tasks.rados.rados.0.smithi045.stdout:2600: writing smithi04526579-28 from 531352 to 1275835 tid 1 2024-04-26T00:35:33.354 INFO:tasks.rados.rados.0.smithi045.stdout:2600: writing smithi04526579-28 from 2005734 to 2785059 tid 2 2024-04-26T00:35:33.368 INFO:tasks.rados.rados.0.smithi045.stdout:2600: writing smithi04526579-28 from 3487333 to 3731352 tid 3 2024-04-26T00:35:33.368 INFO:tasks.rados.rados.0.smithi045.stdout:2601: rollback oid 22 current snap is 290 2024-04-26T00:35:33.368 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 22 to 256 2024-04-26T00:35:33.368 INFO:tasks.rados.rados.0.smithi045.stdout:2600: finishing write tid 1 to smithi04526579-28 2024-04-26T00:35:33.368 INFO:tasks.rados.rados.0.smithi045.stdout:2600: finishing write tid 2 to smithi04526579-28 2024-04-26T00:35:33.368 INFO:tasks.rados.rados.0.smithi045.stdout:2602: delete oid 34 current snap is 290 2024-04-26T00:35:33.373 INFO:tasks.rados.rados.0.smithi045.stdout:2601: finishing rollback tid 1 to smithi04526579-22 2024-04-26T00:35:33.374 INFO:tasks.rados.rados.0.smithi045.stdout:2600: finishing write tid 3 to smithi04526579-28 2024-04-26T00:35:33.374 INFO:tasks.rados.rados.0.smithi045.stdout:2600: finishing write tid 4 to smithi04526579-28 2024-04-26T00:35:33.375 INFO:tasks.rados.rados.0.smithi045.stdout:2600: finishing write tid 5 to smithi04526579-28 2024-04-26T00:35:33.375 INFO:tasks.rados.rados.0.smithi045.stdout:2600: oid 28 updating version 0 to 1952 2024-04-26T00:35:33.375 INFO:tasks.rados.rados.0.smithi045.stdout:2600: oid 28 updating version 1952 to 1953 2024-04-26T00:35:33.375 INFO:tasks.rados.rados.0.smithi045.stdout:2600: oid 28 updating version 1953 to 1954 2024-04-26T00:35:33.375 INFO:tasks.rados.rados.0.smithi045.stdout:2600: oid 28 version 1954 is already newer than 1951 2024-04-26T00:35:33.375 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 1954 (ObjNum 936 snap 290 seq_num 936) dirty exists 2024-04-26T00:35:33.375 INFO:tasks.rados.rados.0.smithi045.stdout:2600: left oid 28 (ObjNum 936 snap 290 seq_num 936) 2024-04-26T00:35:33.376 INFO:tasks.rados.rados.0.smithi045.stdout:2600: done (2 left) 2024-04-26T00:35:33.376 INFO:tasks.rados.rados.0.smithi045.stdout:2602: done (1 left) 2024-04-26T00:35:33.376 INFO:tasks.rados.rados.0.smithi045.stdout:2603: write oid 15 current snap is 290 2024-04-26T00:35:33.376 INFO:tasks.rados.rados.0.smithi045.stdout:2603: seq_num 938 ranges {564966=494981,1606345=782994,3025826=425501} 2024-04-26T00:35:33.390 INFO:tasks.rados.rados.0.smithi045.stdout:2603: writing smithi04526579-15 from 564966 to 1059947 tid 1 2024-04-26T00:35:33.405 INFO:tasks.rados.rados.0.smithi045.stdout:2603: writing smithi04526579-15 from 1606345 to 2389339 tid 2 2024-04-26T00:35:33.418 INFO:tasks.rados.rados.0.smithi045.stdout:2603: writing smithi04526579-15 from 3025826 to 3451327 tid 3 2024-04-26T00:35:33.419 INFO:tasks.rados.rados.0.smithi045.stdout:2604: write oid 37 current snap is 290 2024-04-26T00:35:33.419 INFO:tasks.rados.rados.0.smithi045.stdout:2601: finishing rollback tid 2 to smithi04526579-22 2024-04-26T00:35:33.419 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 1483 (ObjNum 690 snap 215 seq_num 690) dirty exists 2024-04-26T00:35:33.419 INFO:tasks.rados.rados.0.smithi045.stdout:2603: finishing write tid 1 to smithi04526579-15 2024-04-26T00:35:33.419 INFO:tasks.rados.rados.0.smithi045.stdout:2603: finishing write tid 2 to smithi04526579-15 2024-04-26T00:35:33.419 INFO:tasks.rados.rados.0.smithi045.stdout:2604: seq_num 939 ranges {581773=570230,1664827=526652,2653249=636773} 2024-04-26T00:35:33.430 INFO:tasks.rados.rados.0.smithi045.stdout:2604: writing smithi04526579-37 from 581773 to 1152003 tid 1 2024-04-26T00:35:33.441 INFO:tasks.rados.rados.0.smithi045.stdout:2604: writing smithi04526579-37 from 1664827 to 2191479 tid 2 2024-04-26T00:35:33.453 INFO:tasks.rados.rados.0.smithi045.stdout:2604: writing smithi04526579-37 from 2653249 to 3290022 tid 3 2024-04-26T00:35:33.453 INFO:tasks.rados.rados.0.smithi045.stdout:2601: done (2 left) 2024-04-26T00:35:33.453 INFO:tasks.rados.rados.0.smithi045.stdout:2605: copy_from oid 23 from oid 22 current snap is 290 2024-04-26T00:35:33.453 INFO:tasks.rados.rados.0.smithi045.stdout:2603: finishing write tid 3 to smithi04526579-15 2024-04-26T00:35:33.453 INFO:tasks.rados.rados.0.smithi045.stdout:2603: finishing write tid 4 to smithi04526579-15 2024-04-26T00:35:33.453 INFO:tasks.rados.rados.0.smithi045.stdout:2603: finishing write tid 5 to smithi04526579-15 2024-04-26T00:35:33.454 INFO:tasks.rados.rados.0.smithi045.stdout:2603: oid 15 updating version 0 to 1956 2024-04-26T00:35:33.454 INFO:tasks.rados.rados.0.smithi045.stdout:2603: oid 15 updating version 1956 to 1957 2024-04-26T00:35:33.454 INFO:tasks.rados.rados.0.smithi045.stdout:2603: oid 15 updating version 1957 to 1958 2024-04-26T00:35:33.454 INFO:tasks.rados.rados.0.smithi045.stdout:2603: oid 15 version 1958 is already newer than 1955 2024-04-26T00:35:33.454 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 1958 (ObjNum 938 snap 290 seq_num 938) dirty exists 2024-04-26T00:35:33.454 INFO:tasks.rados.rados.0.smithi045.stdout:2603: left oid 15 (ObjNum 938 snap 290 seq_num 938) 2024-04-26T00:35:33.454 INFO:tasks.rados.rados.0.smithi045.stdout:2603: done (2 left) 2024-04-26T00:35:33.454 INFO:tasks.rados.rados.0.smithi045.stdout:2606: write (excl) oid 29 current snap is 290 2024-04-26T00:35:33.454 INFO:tasks.rados.rados.0.smithi045.stdout:2606: seq_num 940 ranges {598580=645479,1723309=786663,3080672=448045} 2024-04-26T00:35:33.463 INFO:tasks.rados.rados.0.smithi045.stdout:2606: writing smithi04526579-29 from 598580 to 1244059 tid 1 2024-04-26T00:35:33.475 INFO:tasks.rados.rados.0.smithi045.stdout:2606: writing smithi04526579-29 from 1723309 to 2509972 tid 2 2024-04-26T00:35:33.484 INFO:tasks.rados.rados.0.smithi045.stdout:2606: writing smithi04526579-29 from 3080672 to 3528717 tid 3 2024-04-26T00:35:33.484 INFO:tasks.rados.rados.0.smithi045.stdout:2607: rollback oid 25 current snap is 290 2024-04-26T00:35:33.484 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 25 to 288 2024-04-26T00:35:33.484 INFO:tasks.rados.rados.0.smithi045.stdout:2604: finishing write tid 1 to smithi04526579-37 2024-04-26T00:35:33.484 INFO:tasks.rados.rados.0.smithi045.stdout:2608: snap_remove snap 256 2024-04-26T00:35:33.537 INFO:tasks.rados.rados.0.smithi045.stdout:2608: done (4 left) 2024-04-26T00:35:33.537 INFO:tasks.rados.rados.0.smithi045.stdout:2609: snap_create 2024-04-26T00:35:33.537 INFO:tasks.rados.rados.0.smithi045.stdout:2604: finishing write tid 2 to smithi04526579-37 2024-04-26T00:35:33.537 INFO:tasks.rados.rados.0.smithi045.stdout:2604: finishing write tid 3 to smithi04526579-37 2024-04-26T00:35:33.537 INFO:tasks.rados.rados.0.smithi045.stdout:2604: finishing write tid 4 to smithi04526579-37 2024-04-26T00:35:33.537 INFO:tasks.rados.rados.0.smithi045.stdout:2604: finishing write tid 5 to smithi04526579-37 2024-04-26T00:35:33.537 INFO:tasks.rados.rados.0.smithi045.stdout:2604: oid 37 updating version 0 to 2277 2024-04-26T00:35:33.537 INFO:tasks.rados.rados.0.smithi045.stdout:2604: oid 37 version 2277 is already newer than 2275 2024-04-26T00:35:33.537 INFO:tasks.rados.rados.0.smithi045.stdout:2604: oid 37 version 2277 is already newer than 2274 2024-04-26T00:35:33.537 INFO:tasks.rados.rados.0.smithi045.stdout:2604: oid 37 version 2277 is already newer than 2276 2024-04-26T00:35:33.538 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 2277 (ObjNum 939 snap 290 seq_num 939) dirty exists 2024-04-26T00:35:33.538 INFO:tasks.rados.rados.0.smithi045.stdout:2604: left oid 37 (ObjNum 939 snap 290 seq_num 939) 2024-04-26T00:35:33.538 INFO:tasks.rados.rados.0.smithi045.stdout:2605: finishing copy_from racing read to smithi04526579-23 2024-04-26T00:35:33.538 INFO:tasks.rados.rados.0.smithi045.stdout:2605: finishing copy_from to smithi04526579-23 2024-04-26T00:35:33.538 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 1483 (ObjNum 690 snap 215 seq_num 690) dirty exists 2024-04-26T00:35:33.538 INFO:tasks.rados.rados.0.smithi045.stdout:2606: finishing write tid 1 to smithi04526579-29 2024-04-26T00:35:33.538 INFO:tasks.rados.rados.0.smithi045.stdout:2606: finishing write tid 2 to smithi04526579-29 2024-04-26T00:35:33.538 INFO:tasks.rados.rados.0.smithi045.stdout:2607: finishing rollback tid 1 to smithi04526579-25 2024-04-26T00:35:33.538 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 2254 (ObjNum 33 snap 0 seq_num 2241801856) dirty dne 2024-04-26T00:35:33.538 INFO:tasks.rados.rados.0.smithi045.stdout:2606: finishing write tid 3 to smithi04526579-29 2024-04-26T00:35:33.538 INFO:tasks.rados.rados.0.smithi045.stdout:2606: finishing write tid 4 to smithi04526579-29 2024-04-26T00:35:33.538 INFO:tasks.rados.rados.0.smithi045.stdout:2606: finishing write tid 5 to smithi04526579-29 2024-04-26T00:35:33.538 INFO:tasks.rados.rados.0.smithi045.stdout:2606: oid 29 updating version 0 to 1993 2024-04-26T00:35:33.539 INFO:tasks.rados.rados.0.smithi045.stdout:2606: oid 29 version 1993 is already newer than 1990 2024-04-26T00:35:33.539 INFO:tasks.rados.rados.0.smithi045.stdout:2606: oid 29 version 1993 is already newer than 1992 2024-04-26T00:35:33.539 INFO:tasks.rados.rados.0.smithi045.stdout:2606: oid 29 version 1993 is already newer than 1991 2024-04-26T00:35:33.539 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 1993 (ObjNum 940 snap 290 seq_num 940) dirty exists 2024-04-26T00:35:33.539 INFO:tasks.rados.rados.0.smithi045.stdout:2606: left oid 29 (ObjNum 940 snap 290 seq_num 940) 2024-04-26T00:35:34.538 INFO:tasks.rados.rados.0.smithi045.stdout:2604: done (4 left) 2024-04-26T00:35:34.538 INFO:tasks.rados.rados.0.smithi045.stdout:2605: done (3 left) 2024-04-26T00:35:34.538 INFO:tasks.rados.rados.0.smithi045.stdout:2606: done (2 left) 2024-04-26T00:35:34.538 INFO:tasks.rados.rados.0.smithi045.stdout:2607: done (1 left) 2024-04-26T00:35:34.538 INFO:tasks.rados.rados.0.smithi045.stdout:2609: done (0 left) 2024-04-26T00:35:34.539 INFO:tasks.rados.rados.0.smithi045.stdout:2610: copy_from oid 19 from oid 27 current snap is 291 2024-04-26T00:35:34.539 INFO:tasks.rados.rados.0.smithi045.stdout:2611: read oid 35 snap -1 2024-04-26T00:35:34.539 INFO:tasks.rados.rados.0.smithi045.stdout:2611: expect (ObjNum 812 snap 252 seq_num 812) 2024-04-26T00:35:34.539 INFO:tasks.rados.rados.0.smithi045.stdout:2612: snap_remove snap 286 2024-04-26T00:35:35.549 INFO:tasks.rados.rados.0.smithi045.stdout:2612: done (2 left) 2024-04-26T00:35:35.549 INFO:tasks.rados.rados.0.smithi045.stdout:2613: snap_remove snap 289 2024-04-26T00:35:36.549 INFO:tasks.rados.rados.0.smithi045.stdout:2613: done (2 left) 2024-04-26T00:35:36.549 INFO:tasks.rados.rados.0.smithi045.stdout:2614: snap_remove snap 288 2024-04-26T00:35:37.551 INFO:tasks.rados.rados.0.smithi045.stdout:2614: done (2 left) 2024-04-26T00:35:37.551 INFO:tasks.rados.rados.0.smithi045.stdout:2615: rollback oid 46 current snap is 291 2024-04-26T00:35:37.551 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 46 to 287 2024-04-26T00:35:37.551 INFO:tasks.rados.rados.0.smithi045.stdout:2611: expect (ObjNum 812 snap 252 seq_num 812) 2024-04-26T00:35:37.822 INFO:tasks.rados.rados.0.smithi045.stdout:2610: finishing copy_from racing read to smithi04526579-19 2024-04-26T00:35:37.822 INFO:tasks.rados.rados.0.smithi045.stdout:2610: finishing copy_from to smithi04526579-19 2024-04-26T00:35:37.823 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 1597 (ObjNum 849 snap 262 seq_num 849) dirty exists 2024-04-26T00:35:37.823 INFO:tasks.rados.rados.0.smithi045.stdout:2615: finishing rollback tid 0 to smithi04526579-46 2024-04-26T00:35:37.823 INFO:tasks.rados.rados.0.smithi045.stdout:2615: finishing rollback tid 1 to smithi04526579-46 2024-04-26T00:35:37.823 INFO:tasks.rados.rados.0.smithi045.stdout:2615: finishing rollback tid 2 to smithi04526579-46 2024-04-26T00:35:37.823 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 1486 (ObjNum 913 snap 282 seq_num 913) dirty exists 2024-04-26T00:35:37.823 INFO:tasks.rados.rados.0.smithi045.stdout:2610: done (2 left) 2024-04-26T00:35:37.823 INFO:tasks.rados.rados.0.smithi045.stdout:2611: done (1 left) 2024-04-26T00:35:37.823 INFO:tasks.rados.rados.0.smithi045.stdout:2615: done (0 left) 2024-04-26T00:35:37.823 INFO:tasks.rados.rados.0.smithi045.stdout:2616: write oid 27 current snap is 291 2024-04-26T00:35:37.823 INFO:tasks.rados.rados.0.smithi045.stdout:2616: seq_num 941 ranges {615387=720728,1781791=530321,3108095=707292} 2024-04-26T00:35:37.838 INFO:tasks.rados.rados.0.smithi045.stdout:2616: writing smithi04526579-27 from 615387 to 1336115 tid 1 2024-04-26T00:35:37.850 INFO:tasks.rados.rados.0.smithi045.stdout:2616: writing smithi04526579-27 from 1781791 to 2312112 tid 2 2024-04-26T00:35:37.869 INFO:tasks.rados.rados.0.smithi045.stdout:2616: writing smithi04526579-27 from 3108095 to 3815387 tid 3 2024-04-26T00:35:37.869 INFO:tasks.rados.rados.0.smithi045.stdout:2617: snap_remove snap 287 2024-04-26T00:35:38.537 INFO:tasks.rados.rados.0.smithi045.stdout:2617: done (1 left) 2024-04-26T00:35:38.537 INFO:tasks.rados.rados.0.smithi045.stdout:2618: copy_from oid 33 from oid 2 current snap is 291 2024-04-26T00:35:38.537 INFO:tasks.rados.rados.0.smithi045.stdout:2616: finishing write tid 1 to smithi04526579-27 2024-04-26T00:35:38.537 INFO:tasks.rados.rados.0.smithi045.stdout:2616: finishing write tid 2 to smithi04526579-27 2024-04-26T00:35:38.538 INFO:tasks.rados.rados.0.smithi045.stdout:2616: finishing write tid 3 to smithi04526579-27 2024-04-26T00:35:38.538 INFO:tasks.rados.rados.0.smithi045.stdout:2616: finishing write tid 4 to smithi04526579-27 2024-04-26T00:35:38.538 INFO:tasks.rados.rados.0.smithi045.stdout:2616: finishing write tid 5 to smithi04526579-27 2024-04-26T00:35:38.538 INFO:tasks.rados.rados.0.smithi045.stdout:2616: oid 27 updating version 0 to 1894 2024-04-26T00:35:38.538 INFO:tasks.rados.rados.0.smithi045.stdout:2616: oid 27 updating version 1894 to 1895 2024-04-26T00:35:38.538 INFO:tasks.rados.rados.0.smithi045.stdout:2616: oid 27 updating version 1895 to 1896 2024-04-26T00:35:38.538 INFO:tasks.rados.rados.0.smithi045.stdout:2616: oid 27 version 1896 is already newer than 1893 2024-04-26T00:35:38.538 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 1896 (ObjNum 941 snap 291 seq_num 941) dirty exists 2024-04-26T00:35:38.538 INFO:tasks.rados.rados.0.smithi045.stdout:2616: left oid 27 (ObjNum 941 snap 291 seq_num 941) 2024-04-26T00:35:38.538 INFO:tasks.rados.rados.0.smithi045.stdout:2616: done (1 left) 2024-04-26T00:35:38.538 INFO:tasks.rados.rados.0.smithi045.stdout:2619: read oid 3 snap 290 2024-04-26T00:35:38.538 INFO:tasks.rados.rados.0.smithi045.stdout:2619: expect (ObjNum 878 snap 273 seq_num 878) 2024-04-26T00:35:38.538 INFO:tasks.rados.rados.0.smithi045.stdout:2620: write (excl) oid 22 current snap is 291 2024-04-26T00:35:38.539 INFO:tasks.rados.rados.0.smithi045.stdout:2620: seq_num 942 ranges {632194=795977,2123920=790332,3419165=413029} 2024-04-26T00:35:38.547 INFO:tasks.rados.rados.0.smithi045.stdout:2620: writing smithi04526579-22 from 632194 to 1428171 tid 1 2024-04-26T00:35:38.557 INFO:tasks.rados.rados.0.smithi045.stdout:2620: writing smithi04526579-22 from 2123920 to 2914252 tid 2 2024-04-26T00:35:38.564 INFO:tasks.rados.rados.0.smithi045.stdout:2620: writing smithi04526579-22 from 3419165 to 3832194 tid 3 2024-04-26T00:35:38.564 INFO:tasks.rados.rados.0.smithi045.stdout:2621: read oid 46 snap -1 2024-04-26T00:35:38.564 INFO:tasks.rados.rados.0.smithi045.stdout:2621: expect (ObjNum 913 snap 282 seq_num 913) 2024-04-26T00:35:38.564 INFO:tasks.rados.rados.0.smithi045.stdout:2622: rollback oid 16 current snap is 291 2024-04-26T00:35:38.564 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 16 to 290 2024-04-26T00:35:38.564 INFO:tasks.rados.rados.0.smithi045.stdout:2619: expect (ObjNum 878 snap 273 seq_num 878) 2024-04-26T00:35:38.762 INFO:tasks.rados.rados.0.smithi045.stdout:2621: expect (ObjNum 913 snap 282 seq_num 913) 2024-04-26T00:35:38.999 INFO:tasks.rados.rados.0.smithi045.stdout:2620: finishing write tid 1 to smithi04526579-22 2024-04-26T00:35:38.999 INFO:tasks.rados.rados.0.smithi045.stdout:2620: finishing write tid 2 to smithi04526579-22 2024-04-26T00:35:39.000 INFO:tasks.rados.rados.0.smithi045.stdout:2619: done (4 left) 2024-04-26T00:35:39.000 INFO:tasks.rados.rados.0.smithi045.stdout:2621: done (3 left) 2024-04-26T00:35:39.000 INFO:tasks.rados.rados.0.smithi045.stdout:2623: snap_create 2024-04-26T00:35:39.000 INFO:tasks.rados.rados.0.smithi045.stdout:2620: finishing write tid 3 to smithi04526579-22 2024-04-26T00:35:39.000 INFO:tasks.rados.rados.0.smithi045.stdout:2620: finishing write tid 4 to smithi04526579-22 2024-04-26T00:35:39.000 INFO:tasks.rados.rados.0.smithi045.stdout:2620: finishing write tid 5 to smithi04526579-22 2024-04-26T00:35:39.000 INFO:tasks.rados.rados.0.smithi045.stdout:2620: oid 22 updating version 0 to 1487 2024-04-26T00:35:39.000 INFO:tasks.rados.rados.0.smithi045.stdout:2620: oid 22 updating version 1487 to 1488 2024-04-26T00:35:39.000 INFO:tasks.rados.rados.0.smithi045.stdout:2620: oid 22 updating version 1488 to 1490 2024-04-26T00:35:39.000 INFO:tasks.rados.rados.0.smithi045.stdout:2620: oid 22 version 1490 is already newer than 1489 2024-04-26T00:35:39.000 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 1490 (ObjNum 942 snap 291 seq_num 942) dirty exists 2024-04-26T00:35:39.000 INFO:tasks.rados.rados.0.smithi045.stdout:2620: left oid 22 (ObjNum 942 snap 291 seq_num 942) 2024-04-26T00:35:39.000 INFO:tasks.rados.rados.0.smithi045.stdout:2622: finishing rollback tid 0 to smithi04526579-16 2024-04-26T00:35:39.000 INFO:tasks.rados.rados.0.smithi045.stdout:2622: finishing rollback tid 1 to smithi04526579-16 2024-04-26T00:35:39.001 INFO:tasks.rados.rados.0.smithi045.stdout:2622: finishing rollback tid 2 to smithi04526579-16 2024-04-26T00:35:39.001 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 1171 (ObjNum 921 snap 283 seq_num 921) dirty exists 2024-04-26T00:35:39.001 INFO:tasks.rados.rados.0.smithi045.stdout:2618: finishing copy_from racing read to smithi04526579-33 2024-04-26T00:35:39.001 INFO:tasks.rados.rados.0.smithi045.stdout:2618: finishing copy_from to smithi04526579-33 2024-04-26T00:35:39.001 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 1934 (ObjNum 690 snap 215 seq_num 690) dirty exists 2024-04-26T00:35:39.580 INFO:tasks.rados.rados.0.smithi045.stdout:2618: done (3 left) 2024-04-26T00:35:39.580 INFO:tasks.rados.rados.0.smithi045.stdout:2620: done (2 left) 2024-04-26T00:35:39.580 INFO:tasks.rados.rados.0.smithi045.stdout:2622: done (1 left) 2024-04-26T00:35:39.580 INFO:tasks.rados.rados.0.smithi045.stdout:2623: done (0 left) 2024-04-26T00:35:39.580 INFO:tasks.rados.rados.0.smithi045.stdout:2624: read oid 38 snap 290 2024-04-26T00:35:39.581 INFO:tasks.rados.rados.0.smithi045.stdout:2624: expect (ObjNum 894 snap 275 seq_num 894) 2024-04-26T00:35:39.581 INFO:tasks.rados.rados.0.smithi045.stdout:2625: delete oid 50 current snap is 292 2024-04-26T00:35:39.588 INFO:tasks.rados.rados.0.smithi045.stdout:2625: done (1 left) 2024-04-26T00:35:39.589 INFO:tasks.rados.rados.0.smithi045.stdout:2626: delete oid 37 current snap is 292 2024-04-26T00:35:39.597 INFO:tasks.rados.rados.0.smithi045.stdout:2624: expect (ObjNum 894 snap 275 seq_num 894) 2024-04-26T00:35:39.769 INFO:tasks.rados.rados.0.smithi045.stdout:2624: done (1 left) 2024-04-26T00:35:39.769 INFO:tasks.rados.rados.0.smithi045.stdout:2626: done (0 left) 2024-04-26T00:35:39.770 INFO:tasks.rados.rados.0.smithi045.stdout:2627: rollback oid 19 current snap is 292 2024-04-26T00:35:39.770 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 19 to 291 2024-04-26T00:35:39.770 INFO:tasks.rados.rados.0.smithi045.stdout:2628: rollback oid 37 current snap is 292 2024-04-26T00:35:39.770 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 37 to 291 2024-04-26T00:35:39.770 INFO:tasks.rados.rados.0.smithi045.stdout:2629: write oid 23 current snap is 292 2024-04-26T00:35:39.770 INFO:tasks.rados.rados.0.smithi045.stdout:2629: seq_num 945 ranges {682615=738077,2015719=537659,3217787=420758} 2024-04-26T00:35:39.777 INFO:tasks.rados.rados.0.smithi045.stdout:2629: writing smithi04526579-23 from 682615 to 1420692 tid 1 2024-04-26T00:35:39.784 INFO:tasks.rados.rados.0.smithi045.stdout:2629: writing smithi04526579-23 from 2015719 to 2553378 tid 2 2024-04-26T00:35:39.790 INFO:tasks.rados.rados.0.smithi045.stdout:2629: writing smithi04526579-23 from 3217787 to 3638545 tid 3 2024-04-26T00:35:39.790 INFO:tasks.rados.rados.0.smithi045.stdout:2630: delete oid 17 current snap is 292 2024-04-26T00:35:39.790 INFO:tasks.rados.rados.0.smithi045.stdout:2628: finishing rollback tid 1 to smithi04526579-37 2024-04-26T00:35:39.790 INFO:tasks.rados.rados.0.smithi045.stdout:2627: finishing rollback tid 0 to smithi04526579-19 2024-04-26T00:35:39.790 INFO:tasks.rados.rados.0.smithi045.stdout:2628: finishing rollback tid 2 to smithi04526579-37 2024-04-26T00:35:39.790 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 2302 (ObjNum 939 snap 290 seq_num 939) dirty exists 2024-04-26T00:35:39.790 INFO:tasks.rados.rados.0.smithi045.stdout:2627: finishing rollback tid 1 to smithi04526579-19 2024-04-26T00:35:39.791 INFO:tasks.rados.rados.0.smithi045.stdout:2627: finishing rollback tid 2 to smithi04526579-19 2024-04-26T00:35:39.791 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 1904 (ObjNum 849 snap 262 seq_num 849) dirty exists 2024-04-26T00:35:39.796 INFO:tasks.rados.rados.0.smithi045.stdout:2629: finishing write tid 1 to smithi04526579-23 2024-04-26T00:35:39.797 INFO:tasks.rados.rados.0.smithi045.stdout:2629: finishing write tid 2 to smithi04526579-23 2024-04-26T00:35:39.797 INFO:tasks.rados.rados.0.smithi045.stdout:2629: finishing write tid 3 to smithi04526579-23 2024-04-26T00:35:39.797 INFO:tasks.rados.rados.0.smithi045.stdout:2629: finishing write tid 4 to smithi04526579-23 2024-04-26T00:35:39.797 INFO:tasks.rados.rados.0.smithi045.stdout:2629: finishing write tid 5 to smithi04526579-23 2024-04-26T00:35:39.797 INFO:tasks.rados.rados.0.smithi045.stdout:2629: oid 23 updating version 0 to 1994 2024-04-26T00:35:39.797 INFO:tasks.rados.rados.0.smithi045.stdout:2629: oid 23 updating version 1994 to 1996 2024-04-26T00:35:39.798 INFO:tasks.rados.rados.0.smithi045.stdout:2629: oid 23 version 1996 is already newer than 1993 2024-04-26T00:35:39.798 INFO:tasks.rados.rados.0.smithi045.stdout:2629: oid 23 version 1996 is already newer than 1995 2024-04-26T00:35:39.798 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 1996 (ObjNum 945 snap 292 seq_num 945) dirty exists 2024-04-26T00:35:39.798 INFO:tasks.rados.rados.0.smithi045.stdout:2629: left oid 23 (ObjNum 945 snap 292 seq_num 945) 2024-04-26T00:35:39.800 INFO:tasks.rados.rados.0.smithi045.stdout:2627: done (3 left) 2024-04-26T00:35:39.800 INFO:tasks.rados.rados.0.smithi045.stdout:2628: done (2 left) 2024-04-26T00:35:39.801 INFO:tasks.rados.rados.0.smithi045.stdout:2629: done (1 left) 2024-04-26T00:35:39.801 INFO:tasks.rados.rados.0.smithi045.stdout:2630: done (0 left) 2024-04-26T00:35:39.801 INFO:tasks.rados.rados.0.smithi045.stdout:2631: snap_create 2024-04-26T00:35:40.589 INFO:tasks.rados.rados.0.smithi045.stdout:2631: done (0 left) 2024-04-26T00:35:40.589 INFO:tasks.rados.rados.0.smithi045.stdout:2632: snap_remove snap 292 2024-04-26T00:35:41.602 INFO:tasks.rados.rados.0.smithi045.stdout:2632: done (0 left) 2024-04-26T00:35:41.602 INFO:tasks.rados.rados.0.smithi045.stdout:2633: read oid 38 snap -1 2024-04-26T00:35:41.602 INFO:tasks.rados.rados.0.smithi045.stdout:2633: expect (ObjNum 894 snap 275 seq_num 894) 2024-04-26T00:35:41.602 INFO:tasks.rados.rados.0.smithi045.stdout:2634: read oid 47 snap -1 2024-04-26T00:35:41.602 INFO:tasks.rados.rados.0.smithi045.stdout:2634: expect deleted 2024-04-26T00:35:41.602 INFO:tasks.rados.rados.0.smithi045.stdout:2635: rollback oid 22 current snap is 293 2024-04-26T00:35:41.602 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 22 to 291 2024-04-26T00:35:41.602 INFO:tasks.rados.rados.0.smithi045.stdout:2636: write oid 39 current snap is 293 2024-04-26T00:35:41.603 INFO:tasks.rados.rados.0.smithi045.stdout:2636: seq_num 947 ranges {716229=488575,1616330=541328,2756280=559655} 2024-04-26T00:35:41.624 INFO:tasks.rados.rados.0.smithi045.stdout:2636: writing smithi04526579-39 from 716229 to 1204804 tid 1 2024-04-26T00:35:41.639 INFO:tasks.rados.rados.0.smithi045.stdout:2636: writing smithi04526579-39 from 1616330 to 2157658 tid 2 2024-04-26T00:35:41.657 INFO:tasks.rados.rados.0.smithi045.stdout:2636: writing smithi04526579-39 from 2756280 to 3315935 tid 3 2024-04-26T00:35:41.657 INFO:tasks.rados.rados.0.smithi045.stdout:2637: rollback oid 49 current snap is 293 2024-04-26T00:35:41.657 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 49 to 290 2024-04-26T00:35:41.657 INFO:tasks.rados.rados.0.smithi045.stdout:2635: finishing rollback tid 0 to smithi04526579-22 2024-04-26T00:35:41.658 INFO:tasks.rados.rados.0.smithi045.stdout:2634: done (4 left) 2024-04-26T00:35:41.658 INFO:tasks.rados.rados.0.smithi045.stdout:2638: copy_from oid 48 from oid 15 current snap is 293 2024-04-26T00:35:41.658 INFO:tasks.rados.rados.0.smithi045.stdout:2635: finishing rollback tid 1 to smithi04526579-22 2024-04-26T00:35:41.658 INFO:tasks.rados.rados.0.smithi045.stdout:2635: finishing rollback tid 2 to smithi04526579-22 2024-04-26T00:35:41.658 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 1493 (ObjNum 942 snap 291 seq_num 942) dirty exists 2024-04-26T00:35:41.658 INFO:tasks.rados.rados.0.smithi045.stdout:2633: expect (ObjNum 894 snap 275 seq_num 894) 2024-04-26T00:35:41.892 INFO:tasks.rados.rados.0.smithi045.stdout:2636: finishing write tid 1 to smithi04526579-39 2024-04-26T00:35:41.892 INFO:tasks.rados.rados.0.smithi045.stdout:2636: finishing write tid 2 to smithi04526579-39 2024-04-26T00:35:41.893 INFO:tasks.rados.rados.0.smithi045.stdout:2636: finishing write tid 3 to smithi04526579-39 2024-04-26T00:35:41.893 INFO:tasks.rados.rados.0.smithi045.stdout:2636: finishing write tid 4 to smithi04526579-39 2024-04-26T00:35:41.893 INFO:tasks.rados.rados.0.smithi045.stdout:2636: finishing write tid 5 to smithi04526579-39 2024-04-26T00:35:41.893 INFO:tasks.rados.rados.0.smithi045.stdout:2636: oid 39 updating version 0 to 2172 2024-04-26T00:35:41.893 INFO:tasks.rados.rados.0.smithi045.stdout:2636: oid 39 version 2172 is already newer than 2170 2024-04-26T00:35:41.893 INFO:tasks.rados.rados.0.smithi045.stdout:2636: oid 39 version 2172 is already newer than 2171 2024-04-26T00:35:41.893 INFO:tasks.rados.rados.0.smithi045.stdout:2636: oid 39 version 2172 is already newer than 2169 2024-04-26T00:35:41.893 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 2172 (ObjNum 947 snap 293 seq_num 947) dirty exists 2024-04-26T00:35:41.893 INFO:tasks.rados.rados.0.smithi045.stdout:2636: left oid 39 (ObjNum 947 snap 293 seq_num 947) 2024-04-26T00:35:41.893 INFO:tasks.rados.rados.0.smithi045.stdout:2637: finishing rollback tid 0 to smithi04526579-49 2024-04-26T00:35:41.893 INFO:tasks.rados.rados.0.smithi045.stdout:2637: finishing rollback tid 1 to smithi04526579-49 2024-04-26T00:35:41.893 INFO:tasks.rados.rados.0.smithi045.stdout:2637: finishing rollback tid 2 to smithi04526579-49 2024-04-26T00:35:41.893 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 1998 (ObjNum 731 snap 224 seq_num 731) dirty exists 2024-04-26T00:35:41.894 INFO:tasks.rados.rados.0.smithi045.stdout:2638: finishing copy_from racing read to smithi04526579-48 2024-04-26T00:35:41.894 INFO:tasks.rados.rados.0.smithi045.stdout:2638: finishing copy_from to smithi04526579-48 2024-04-26T00:35:41.894 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 1958 (ObjNum 938 snap 290 seq_num 938) dirty exists 2024-04-26T00:35:41.894 INFO:tasks.rados.rados.0.smithi045.stdout:2633: done (4 left) 2024-04-26T00:35:41.894 INFO:tasks.rados.rados.0.smithi045.stdout:2635: done (3 left) 2024-04-26T00:35:41.894 INFO:tasks.rados.rados.0.smithi045.stdout:2636: done (2 left) 2024-04-26T00:35:41.894 INFO:tasks.rados.rados.0.smithi045.stdout:2637: done (1 left) 2024-04-26T00:35:41.894 INFO:tasks.rados.rados.0.smithi045.stdout:2638: done (0 left) 2024-04-26T00:35:41.894 INFO:tasks.rados.rados.0.smithi045.stdout:2639: delete oid 29 current snap is 293 2024-04-26T00:35:41.900 INFO:tasks.rados.rados.0.smithi045.stdout:2639: done (0 left) 2024-04-26T00:35:41.900 INFO:tasks.rados.rados.0.smithi045.stdout:2640: read oid 46 snap -1 2024-04-26T00:35:41.900 INFO:tasks.rados.rados.0.smithi045.stdout:2640: expect (ObjNum 913 snap 282 seq_num 913) 2024-04-26T00:35:41.900 INFO:tasks.rados.rados.0.smithi045.stdout:2641: write (excl) oid 26 current snap is 293 2024-04-26T00:35:41.900 INFO:tasks.rados.rados.0.smithi045.stdout:2641: seq_num 949 ranges {749843=639073,2133294=544997,3211126=582199} 2024-04-26T00:35:41.922 INFO:tasks.rados.rados.0.smithi045.stdout:2641: writing smithi04526579-26 from 749843 to 1388916 tid 1 2024-04-26T00:35:41.944 INFO:tasks.rados.rados.0.smithi045.stdout:2641: writing smithi04526579-26 from 2133294 to 2678291 tid 2 2024-04-26T00:35:41.960 INFO:tasks.rados.rados.0.smithi045.stdout:2641: writing smithi04526579-26 from 3211126 to 3793325 tid 3 2024-04-26T00:35:41.960 INFO:tasks.rados.rados.0.smithi045.stdout:2642: write (excl) oid 39 current snap is 293 2024-04-26T00:35:41.960 INFO:tasks.rados.rados.0.smithi045.stdout:2642: seq_num 950 ranges {766650=714322,2075423=688655,3522196=444454} 2024-04-26T00:35:41.975 INFO:tasks.rados.rados.0.smithi045.stdout:2642: writing smithi04526579-39 from 766650 to 1480972 tid 1 2024-04-26T00:35:41.986 INFO:tasks.rados.rados.0.smithi045.stdout:2642: writing smithi04526579-39 from 2075423 to 2764078 tid 2 2024-04-26T00:35:41.996 INFO:tasks.rados.rados.0.smithi045.stdout:2642: writing smithi04526579-39 from 3522196 to 3966650 tid 3 2024-04-26T00:35:41.996 INFO:tasks.rados.rados.0.smithi045.stdout:2643: rollback oid 25 current snap is 293 2024-04-26T00:35:41.996 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 25 to 290 2024-04-26T00:35:41.996 INFO:tasks.rados.rados.0.smithi045.stdout:2640: expect (ObjNum 913 snap 282 seq_num 913) 2024-04-26T00:35:42.236 INFO:tasks.rados.rados.0.smithi045.stdout:2641: finishing write tid 1 to smithi04526579-26 2024-04-26T00:35:42.236 INFO:tasks.rados.rados.0.smithi045.stdout:2641: finishing write tid 2 to smithi04526579-26 2024-04-26T00:35:42.236 INFO:tasks.rados.rados.0.smithi045.stdout:2641: finishing write tid 3 to smithi04526579-26 2024-04-26T00:35:42.236 INFO:tasks.rados.rados.0.smithi045.stdout:2641: finishing write tid 4 to smithi04526579-26 2024-04-26T00:35:42.236 INFO:tasks.rados.rados.0.smithi045.stdout:2641: finishing write tid 5 to smithi04526579-26 2024-04-26T00:35:42.236 INFO:tasks.rados.rados.0.smithi045.stdout:2641: oid 26 updating version 0 to 1959 2024-04-26T00:35:42.236 INFO:tasks.rados.rados.0.smithi045.stdout:2641: oid 26 updating version 1959 to 1961 2024-04-26T00:35:42.236 INFO:tasks.rados.rados.0.smithi045.stdout:2641: oid 26 updating version 1961 to 1962 2024-04-26T00:35:42.236 INFO:tasks.rados.rados.0.smithi045.stdout:2641: oid 26 version 1962 is already newer than 1960 2024-04-26T00:35:42.236 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 1962 (ObjNum 949 snap 293 seq_num 949) dirty exists 2024-04-26T00:35:42.237 INFO:tasks.rados.rados.0.smithi045.stdout:2641: left oid 26 (ObjNum 949 snap 293 seq_num 949) 2024-04-26T00:35:42.237 INFO:tasks.rados.rados.0.smithi045.stdout:2642: finishing write tid 1 to smithi04526579-39 2024-04-26T00:35:42.237 INFO:tasks.rados.rados.0.smithi045.stdout:2642: finishing write tid 2 to smithi04526579-39 2024-04-26T00:35:42.237 INFO:tasks.rados.rados.0.smithi045.stdout:2643: finishing rollback tid 1 to smithi04526579-25 2024-04-26T00:35:42.237 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 0 (ObjNum 49 snap 0 seq_num 470032400) dirty dne 2024-04-26T00:35:42.237 INFO:tasks.rados.rados.0.smithi045.stdout:2642: finishing write tid 3 to smithi04526579-39 2024-04-26T00:35:42.237 INFO:tasks.rados.rados.0.smithi045.stdout:2642: finishing write tid 4 to smithi04526579-39 2024-04-26T00:35:42.237 INFO:tasks.rados.rados.0.smithi045.stdout:2642: finishing write tid 5 to smithi04526579-39 2024-04-26T00:35:42.237 INFO:tasks.rados.rados.0.smithi045.stdout:2642: oid 39 updating version 0 to 2175 2024-04-26T00:35:42.237 INFO:tasks.rados.rados.0.smithi045.stdout:2642: oid 39 version 2175 is already newer than 2174 2024-04-26T00:35:42.237 INFO:tasks.rados.rados.0.smithi045.stdout:2642: oid 39 version 2175 is already newer than 2173 2024-04-26T00:35:42.237 INFO:tasks.rados.rados.0.smithi045.stdout:2642: oid 39 updating version 2175 to 2176 2024-04-26T00:35:42.237 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 2176 (ObjNum 950 snap 293 seq_num 950) dirty exists 2024-04-26T00:35:42.237 INFO:tasks.rados.rados.0.smithi045.stdout:2642: left oid 39 (ObjNum 950 snap 293 seq_num 950) 2024-04-26T00:35:42.237 INFO:tasks.rados.rados.0.smithi045.stdout:2640: done (3 left) 2024-04-26T00:35:42.237 INFO:tasks.rados.rados.0.smithi045.stdout:2641: done (2 left) 2024-04-26T00:35:42.238 INFO:tasks.rados.rados.0.smithi045.stdout:2642: done (1 left) 2024-04-26T00:35:42.238 INFO:tasks.rados.rados.0.smithi045.stdout:2643: done (0 left) 2024-04-26T00:35:42.238 INFO:tasks.rados.rados.0.smithi045.stdout:2644: snap_create 2024-04-26T00:35:42.607 INFO:tasks.rados.rados.0.smithi045.stdout:2644: done (0 left) 2024-04-26T00:35:42.607 INFO:tasks.rados.rados.0.smithi045.stdout:2645: copy_from oid 26 from oid 27 current snap is 294 2024-04-26T00:35:42.607 INFO:tasks.rados.rados.0.smithi045.stdout:2646: rollback oid 45 current snap is 294 2024-04-26T00:35:42.607 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 45 to 290 2024-04-26T00:35:42.607 INFO:tasks.rados.rados.0.smithi045.stdout:2647: snap_remove snap 293 2024-04-26T00:35:43.537 INFO:tasks.rados.rados.0.smithi045.stdout:2647: done (2 left) 2024-04-26T00:35:43.537 INFO:tasks.rados.rados.0.smithi045.stdout:2648: snap_create 2024-04-26T00:35:43.537 INFO:tasks.rados.rados.0.smithi045.stdout:2646: finishing rollback tid 0 to smithi04526579-45 2024-04-26T00:35:43.537 INFO:tasks.rados.rados.0.smithi045.stdout:2646: finishing rollback tid 1 to smithi04526579-45 2024-04-26T00:35:43.537 INFO:tasks.rados.rados.0.smithi045.stdout:2646: finishing rollback tid 2 to smithi04526579-45 2024-04-26T00:35:43.537 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 2157 (ObjNum 898 snap 276 seq_num 898) dirty exists 2024-04-26T00:35:43.538 INFO:tasks.rados.rados.0.smithi045.stdout:2645: finishing copy_from racing read to smithi04526579-26 2024-04-26T00:35:43.538 INFO:tasks.rados.rados.0.smithi045.stdout:2645: finishing copy_from to smithi04526579-26 2024-04-26T00:35:43.538 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 1896 (ObjNum 941 snap 291 seq_num 941) dirty exists 2024-04-26T00:35:44.545 INFO:tasks.rados.rados.0.smithi045.stdout:2645: done (2 left) 2024-04-26T00:35:44.545 INFO:tasks.rados.rados.0.smithi045.stdout:2646: done (1 left) 2024-04-26T00:35:44.545 INFO:tasks.rados.rados.0.smithi045.stdout:2648: done (0 left) 2024-04-26T00:35:44.546 INFO:tasks.rados.rados.0.smithi045.stdout:2649: write oid 35 current snap is 295 2024-04-26T00:35:44.546 INFO:tasks.rados.rados.0.smithi045.stdout:2649: seq_num 951 ranges {783457=505924,1850258=548666,2865972=721096} 2024-04-26T00:35:44.564 INFO:tasks.rados.rados.0.smithi045.stdout:2649: writing smithi04526579-35 from 783457 to 1289381 tid 1 2024-04-26T00:35:44.579 INFO:tasks.rados.rados.0.smithi045.stdout:2649: writing smithi04526579-35 from 1850258 to 2398924 tid 2 2024-04-26T00:35:44.589 INFO:tasks.rados.rados.0.smithi045.stdout:2649: writing smithi04526579-35 from 2865972 to 3587068 tid 3 2024-04-26T00:35:44.589 INFO:tasks.rados.rados.0.smithi045.stdout:2650: write (excl) oid 44 current snap is 295 2024-04-26T00:35:44.589 INFO:tasks.rados.rados.0.smithi045.stdout:2649: finishing write tid 1 to smithi04526579-35 2024-04-26T00:35:44.589 INFO:tasks.rados.rados.0.smithi045.stdout:2649: finishing write tid 2 to smithi04526579-35 2024-04-26T00:35:44.589 INFO:tasks.rados.rados.0.smithi045.stdout:2650: seq_num 952 ranges {400264=581173,1508740=491524} 2024-04-26T00:35:44.598 INFO:tasks.rados.rados.0.smithi045.stdout:2650: writing smithi04526579-44 from 400264 to 981437 tid 1 2024-04-26T00:35:44.610 INFO:tasks.rados.rados.0.smithi045.stdout:2650: writing smithi04526579-44 from 1508740 to 2000264 tid 2 2024-04-26T00:35:44.610 INFO:tasks.rados.rados.0.smithi045.stdout:2651: write oid 27 current snap is 295 2024-04-26T00:35:44.610 INFO:tasks.rados.rados.0.smithi045.stdout:2651: seq_num 953 ranges {417071=656422,1567222=449849} 2024-04-26T00:35:44.621 INFO:tasks.rados.rados.0.smithi045.stdout:2651: writing smithi04526579-27 from 417071 to 1073493 tid 1 2024-04-26T00:35:44.630 INFO:tasks.rados.rados.0.smithi045.stdout:2651: writing smithi04526579-27 from 1567222 to 2017071 tid 2 2024-04-26T00:35:44.631 INFO:tasks.rados.rados.0.smithi045.stdout:2652: write oid 33 current snap is 295 2024-04-26T00:35:44.631 INFO:tasks.rados.rados.0.smithi045.stdout:2652: seq_num 954 ranges {433878=731671,1909351=124527} 2024-04-26T00:35:44.640 INFO:tasks.rados.rados.0.smithi045.stdout:2652: writing smithi04526579-33 from 433878 to 1165549 tid 1 2024-04-26T00:35:44.647 INFO:tasks.rados.rados.0.smithi045.stdout:2652: writing smithi04526579-33 from 1909351 to 2033878 tid 2 2024-04-26T00:35:44.647 INFO:tasks.rados.rados.0.smithi045.stdout:2653: copy_from oid 45 from oid 4 current snap is 295 2024-04-26T00:35:44.647 INFO:tasks.rados.rados.0.smithi045.stdout:2649: finishing write tid 3 to smithi04526579-35 2024-04-26T00:35:44.647 INFO:tasks.rados.rados.0.smithi045.stdout:2649: finishing write tid 4 to smithi04526579-35 2024-04-26T00:35:44.647 INFO:tasks.rados.rados.0.smithi045.stdout:2649: finishing write tid 5 to smithi04526579-35 2024-04-26T00:35:44.647 INFO:tasks.rados.rados.0.smithi045.stdout:2649: oid 35 updating version 0 to 1909 2024-04-26T00:35:44.648 INFO:tasks.rados.rados.0.smithi045.stdout:2649: oid 35 version 1909 is already newer than 1908 2024-04-26T00:35:44.648 INFO:tasks.rados.rados.0.smithi045.stdout:2649: oid 35 version 1909 is already newer than 1907 2024-04-26T00:35:44.648 INFO:tasks.rados.rados.0.smithi045.stdout:2649: oid 35 version 1909 is already newer than 1906 2024-04-26T00:35:44.648 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 1909 (ObjNum 951 snap 295 seq_num 951) dirty exists 2024-04-26T00:35:44.648 INFO:tasks.rados.rados.0.smithi045.stdout:2649: left oid 35 (ObjNum 951 snap 295 seq_num 951) 2024-04-26T00:35:44.648 INFO:tasks.rados.rados.0.smithi045.stdout:2650: finishing write tid 1 to smithi04526579-44 2024-04-26T00:35:44.648 INFO:tasks.rados.rados.0.smithi045.stdout:2650: finishing write tid 2 to smithi04526579-44 2024-04-26T00:35:44.648 INFO:tasks.rados.rados.0.smithi045.stdout:2650: finishing write tid 3 to smithi04526579-44 2024-04-26T00:35:44.648 INFO:tasks.rados.rados.0.smithi045.stdout:2650: finishing write tid 4 to smithi04526579-44 2024-04-26T00:35:44.648 INFO:tasks.rados.rados.0.smithi045.stdout:2650: oid 44 updating version 0 to 2179 2024-04-26T00:35:44.648 INFO:tasks.rados.rados.0.smithi045.stdout:2650: oid 44 version 2179 is already newer than 2177 2024-04-26T00:35:44.648 INFO:tasks.rados.rados.0.smithi045.stdout:2650: oid 44 version 2179 is already newer than 2178 2024-04-26T00:35:44.648 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 2179 (ObjNum 952 snap 295 seq_num 952) dirty exists 2024-04-26T00:35:44.648 INFO:tasks.rados.rados.0.smithi045.stdout:2650: left oid 44 (ObjNum 952 snap 295 seq_num 952) 2024-04-26T00:35:44.649 INFO:tasks.rados.rados.0.smithi045.stdout:2651: finishing write tid 1 to smithi04526579-27 2024-04-26T00:35:44.649 INFO:tasks.rados.rados.0.smithi045.stdout:2651: finishing write tid 2 to smithi04526579-27 2024-04-26T00:35:44.649 INFO:tasks.rados.rados.0.smithi045.stdout:2651: finishing write tid 3 to smithi04526579-27 2024-04-26T00:35:44.649 INFO:tasks.rados.rados.0.smithi045.stdout:2651: finishing write tid 4 to smithi04526579-27 2024-04-26T00:35:44.649 INFO:tasks.rados.rados.0.smithi045.stdout:2651: oid 27 updating version 0 to 1913 2024-04-26T00:35:44.649 INFO:tasks.rados.rados.0.smithi045.stdout:2651: oid 27 version 1913 is already newer than 1912 2024-04-26T00:35:44.649 INFO:tasks.rados.rados.0.smithi045.stdout:2651: oid 27 version 1913 is already newer than 1911 2024-04-26T00:35:44.649 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 1913 (ObjNum 953 snap 295 seq_num 953) dirty exists 2024-04-26T00:35:44.649 INFO:tasks.rados.rados.0.smithi045.stdout:2651: left oid 27 (ObjNum 953 snap 295 seq_num 953) 2024-04-26T00:35:44.649 INFO:tasks.rados.rados.0.smithi045.stdout:2649: done (4 left) 2024-04-26T00:35:44.649 INFO:tasks.rados.rados.0.smithi045.stdout:2650: done (3 left) 2024-04-26T00:35:44.649 INFO:tasks.rados.rados.0.smithi045.stdout:2651: done (2 left) 2024-04-26T00:35:44.649 INFO:tasks.rados.rados.0.smithi045.stdout:2654: write (excl) oid 5 current snap is 295 2024-04-26T00:35:44.649 INFO:tasks.rados.rados.0.smithi045.stdout:2654: seq_num 955 ranges {450685=406920,1567833=482852} 2024-04-26T00:35:44.653 INFO:tasks.rados.rados.0.smithi045.stdout:2654: writing smithi04526579-5 from 450685 to 857605 tid 1 2024-04-26T00:35:44.663 INFO:tasks.rados.rados.0.smithi045.stdout:2654: writing smithi04526579-5 from 1567833 to 2050685 tid 2 2024-04-26T00:35:44.663 INFO:tasks.rados.rados.0.smithi045.stdout:2655: copy_from oid 13 from oid 44 current snap is 295 2024-04-26T00:35:44.663 INFO:tasks.rados.rados.0.smithi045.stdout:2652: finishing write tid 1 to smithi04526579-33 2024-04-26T00:35:44.663 INFO:tasks.rados.rados.0.smithi045.stdout:2656: snap_create 2024-04-26T00:35:44.665 INFO:tasks.rados.rados.0.smithi045.stdout:2652: finishing write tid 2 to smithi04526579-33 2024-04-26T00:35:44.678 INFO:tasks.rados.rados.0.smithi045.stdout:2652: finishing write tid 3 to smithi04526579-33 2024-04-26T00:35:44.678 INFO:tasks.rados.rados.0.smithi045.stdout:2652: finishing write tid 4 to smithi04526579-33 2024-04-26T00:35:44.679 INFO:tasks.rados.rados.0.smithi045.stdout:2652: oid 33 updating version 0 to 2305 2024-04-26T00:35:44.679 INFO:tasks.rados.rados.0.smithi045.stdout:2652: oid 33 updating version 2305 to 2306 2024-04-26T00:35:44.679 INFO:tasks.rados.rados.0.smithi045.stdout:2652: oid 33 version 2306 is already newer than 2304 2024-04-26T00:35:44.679 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 2306 (ObjNum 954 snap 295 seq_num 954) dirty exists 2024-04-26T00:35:44.679 INFO:tasks.rados.rados.0.smithi045.stdout:2652: left oid 33 (ObjNum 954 snap 295 seq_num 954) 2024-04-26T00:35:44.679 INFO:tasks.rados.rados.0.smithi045.stdout:2654: finishing write tid 1 to smithi04526579-5 2024-04-26T00:35:44.679 INFO:tasks.rados.rados.0.smithi045.stdout:2653: finishing copy_from racing read to smithi04526579-45 2024-04-26T00:35:44.681 INFO:tasks.rados.rados.0.smithi045.stdout:2654: finishing write tid 2 to smithi04526579-5 2024-04-26T00:35:44.682 INFO:tasks.rados.rados.0.smithi045.stdout:2654: finishing write tid 3 to smithi04526579-5 2024-04-26T00:35:44.682 INFO:tasks.rados.rados.0.smithi045.stdout:2654: finishing write tid 4 to smithi04526579-5 2024-04-26T00:35:44.682 INFO:tasks.rados.rados.0.smithi045.stdout:2654: oid 5 updating version 0 to 2309 2024-04-26T00:35:44.682 INFO:tasks.rados.rados.0.smithi045.stdout:2654: oid 5 updating version 2309 to 2310 2024-04-26T00:35:44.682 INFO:tasks.rados.rados.0.smithi045.stdout:2654: oid 5 version 2310 is already newer than 2308 2024-04-26T00:35:44.683 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 2310 (ObjNum 955 snap 295 seq_num 955) dirty exists 2024-04-26T00:35:44.683 INFO:tasks.rados.rados.0.smithi045.stdout:2654: left oid 5 (ObjNum 955 snap 295 seq_num 955) 2024-04-26T00:35:44.683 INFO:tasks.rados.rados.0.smithi045.stdout:2655: finishing copy_from racing read to smithi04526579-13 2024-04-26T00:35:44.683 INFO:tasks.rados.rados.0.smithi045.stdout:2653: finishing copy_from to smithi04526579-45 2024-04-26T00:35:44.683 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 1168 (ObjNum 866 snap 269 seq_num 866) dirty exists 2024-04-26T00:35:44.683 INFO:tasks.rados.rados.0.smithi045.stdout:2655: finishing copy_from to smithi04526579-13 2024-04-26T00:35:44.683 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 2179 (ObjNum 952 snap 295 seq_num 952) dirty exists 2024-04-26T00:35:45.626 INFO:tasks.rados.rados.0.smithi045.stdout:2652: done (4 left) 2024-04-26T00:35:45.626 INFO:tasks.rados.rados.0.smithi045.stdout:2653: done (3 left) 2024-04-26T00:35:45.626 INFO:tasks.rados.rados.0.smithi045.stdout:2654: done (2 left) 2024-04-26T00:35:45.626 INFO:tasks.rados.rados.0.smithi045.stdout:2655: done (1 left) 2024-04-26T00:35:45.626 INFO:tasks.rados.rados.0.smithi045.stdout:2656: done (0 left) 2024-04-26T00:35:45.626 INFO:tasks.rados.rados.0.smithi045.stdout:2657: write (excl) oid 1 current snap is 296 2024-04-26T00:35:45.626 INFO:tasks.rados.rados.0.smithi045.stdout:2657: seq_num 956 ranges {467492=482169,1626315=441177} 2024-04-26T00:35:45.634 INFO:tasks.rados.rados.0.smithi045.stdout:2657: writing smithi04526579-1 from 467492 to 949661 tid 1 2024-04-26T00:35:45.645 INFO:tasks.rados.rados.0.smithi045.stdout:2657: writing smithi04526579-1 from 1626315 to 2067492 tid 2 2024-04-26T00:35:45.645 INFO:tasks.rados.rados.0.smithi045.stdout:2658: delete oid 9 current snap is 296 2024-04-26T00:35:45.658 INFO:tasks.rados.rados.0.smithi045.stdout:2658: done (1 left) 2024-04-26T00:35:45.658 INFO:tasks.rados.rados.0.smithi045.stdout:2659: copy_from oid 33 from oid 44 current snap is 296 2024-04-26T00:35:45.659 INFO:tasks.rados.rados.0.smithi045.stdout:2660: rollback oid 17 current snap is 296 2024-04-26T00:35:45.659 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 17 to 294 2024-04-26T00:35:45.659 INFO:tasks.rados.rados.0.smithi045.stdout:2661: rollback oid 15 current snap is 296 2024-04-26T00:35:45.659 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 15 to 295 2024-04-26T00:35:45.659 INFO:tasks.rados.rados.0.smithi045.stdout:2662: delete oid 18 current snap is 296 2024-04-26T00:35:45.659 INFO:tasks.rados.rados.0.smithi045.stdout:2657: finishing write tid 1 to smithi04526579-1 2024-04-26T00:35:45.664 INFO:tasks.rados.rados.0.smithi045.stdout:2660: finishing rollback tid 1 to smithi04526579-17 2024-04-26T00:35:45.664 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 1771 (ObjNum 129 snap 0 seq_num 2242049552) dirty dne 2024-04-26T00:35:45.671 INFO:tasks.rados.rados.0.smithi045.stdout:2657: finishing write tid 2 to smithi04526579-1 2024-04-26T00:35:45.671 INFO:tasks.rados.rados.0.smithi045.stdout:2657: finishing write tid 3 to smithi04526579-1 2024-04-26T00:35:45.672 INFO:tasks.rados.rados.0.smithi045.stdout:2657: finishing write tid 4 to smithi04526579-1 2024-04-26T00:35:45.672 INFO:tasks.rados.rados.0.smithi045.stdout:2657: oid 1 updating version 0 to 1965 2024-04-26T00:35:45.672 INFO:tasks.rados.rados.0.smithi045.stdout:2657: oid 1 version 1965 is already newer than 1964 2024-04-26T00:35:45.672 INFO:tasks.rados.rados.0.smithi045.stdout:2657: oid 1 version 1965 is already newer than 1963 2024-04-26T00:35:45.672 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 1965 (ObjNum 956 snap 296 seq_num 956) dirty exists 2024-04-26T00:35:45.672 INFO:tasks.rados.rados.0.smithi045.stdout:2657: left oid 1 (ObjNum 956 snap 296 seq_num 956) 2024-04-26T00:35:45.672 INFO:tasks.rados.rados.0.smithi045.stdout:2661: finishing rollback tid 0 to smithi04526579-15 2024-04-26T00:35:45.673 INFO:tasks.rados.rados.0.smithi045.stdout:2661: finishing rollback tid 1 to smithi04526579-15 2024-04-26T00:35:45.675 INFO:tasks.rados.rados.0.smithi045.stdout:2661: finishing rollback tid 2 to smithi04526579-15 2024-04-26T00:35:45.675 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 1968 (ObjNum 938 snap 290 seq_num 938) dirty exists 2024-04-26T00:35:45.677 INFO:tasks.rados.rados.0.smithi045.stdout:2659: finishing copy_from racing read to smithi04526579-33 2024-04-26T00:35:45.677 INFO:tasks.rados.rados.0.smithi045.stdout:2657: done (4 left) 2024-04-26T00:35:45.678 INFO:tasks.rados.rados.0.smithi045.stdout:2660: done (3 left) 2024-04-26T00:35:45.678 INFO:tasks.rados.rados.0.smithi045.stdout:2661: done (2 left) 2024-04-26T00:35:45.678 INFO:tasks.rados.rados.0.smithi045.stdout:2662: done (1 left) 2024-04-26T00:35:45.678 INFO:tasks.rados.rados.0.smithi045.stdout:2663: copy_from oid 47 from oid 23 current snap is 296 2024-04-26T00:35:45.678 INFO:tasks.rados.rados.0.smithi045.stdout:2664: delete oid 38 current snap is 296 2024-04-26T00:35:45.681 INFO:tasks.rados.rados.0.smithi045.stdout:2659: finishing copy_from to smithi04526579-33 2024-04-26T00:35:45.681 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 2179 (ObjNum 952 snap 295 seq_num 952) dirty exists 2024-04-26T00:35:45.686 INFO:tasks.rados.rados.0.smithi045.stdout:2659: done (2 left) 2024-04-26T00:35:45.687 INFO:tasks.rados.rados.0.smithi045.stdout:2664: done (1 left) 2024-04-26T00:35:45.687 INFO:tasks.rados.rados.0.smithi045.stdout:2665: rollback oid 10 current snap is 296 2024-04-26T00:35:45.687 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 10 to 290 2024-04-26T00:35:45.687 INFO:tasks.rados.rados.0.smithi045.stdout:2666: write (excl) oid 27 current snap is 296 2024-04-26T00:35:45.687 INFO:tasks.rados.rados.0.smithi045.stdout:2666: seq_num 960 ranges {534720=499518,1460243=674477} 2024-04-26T00:35:45.701 INFO:tasks.rados.rados.0.smithi045.stdout:2666: writing smithi04526579-27 from 534720 to 1034238 tid 1 2024-04-26T00:35:45.714 INFO:tasks.rados.rados.0.smithi045.stdout:2666: writing smithi04526579-27 from 1460243 to 2134720 tid 2 2024-04-26T00:35:45.715 INFO:tasks.rados.rados.0.smithi045.stdout:2667: rollback oid 26 current snap is 296 2024-04-26T00:35:45.715 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 26 to 294 2024-04-26T00:35:45.715 INFO:tasks.rados.rados.0.smithi045.stdout:2665: finishing rollback tid 0 to smithi04526579-10 2024-04-26T00:35:45.715 INFO:tasks.rados.rados.0.smithi045.stdout:2665: finishing rollback tid 1 to smithi04526579-10 2024-04-26T00:35:45.715 INFO:tasks.rados.rados.0.smithi045.stdout:2665: finishing rollback tid 2 to smithi04526579-10 2024-04-26T00:35:45.715 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 1774 (ObjNum 925 snap 287 seq_num 925) dirty exists 2024-04-26T00:35:45.715 INFO:tasks.rados.rados.0.smithi045.stdout:2663: finishing copy_from racing read to smithi04526579-47 2024-04-26T00:35:45.715 INFO:tasks.rados.rados.0.smithi045.stdout:2666: finishing write tid 1 to smithi04526579-27 2024-04-26T00:35:45.715 INFO:tasks.rados.rados.0.smithi045.stdout:2665: done (3 left) 2024-04-26T00:35:45.715 INFO:tasks.rados.rados.0.smithi045.stdout:2668: read oid 35 snap -1 2024-04-26T00:35:45.715 INFO:tasks.rados.rados.0.smithi045.stdout:2668: expect (ObjNum 951 snap 295 seq_num 951) 2024-04-26T00:35:45.715 INFO:tasks.rados.rados.0.smithi045.stdout:2669: write oid 11 current snap is 296 2024-04-26T00:35:45.715 INFO:tasks.rados.rados.0.smithi045.stdout:2669: seq_num 961 ranges {551527=574767,1918725=232802} 2024-04-26T00:35:45.728 INFO:tasks.rados.rados.0.smithi045.stdout:2669: writing smithi04526579-11 from 551527 to 1126294 tid 1 2024-04-26T00:35:45.740 INFO:tasks.rados.rados.0.smithi045.stdout:2669: writing smithi04526579-11 from 1918725 to 2151527 tid 2 2024-04-26T00:35:45.740 INFO:tasks.rados.rados.0.smithi045.stdout:2670: delete oid 13 current snap is 296 2024-04-26T00:35:45.740 INFO:tasks.rados.rados.0.smithi045.stdout:2663: finishing copy_from to smithi04526579-47 2024-04-26T00:35:45.740 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 1996 (ObjNum 945 snap 292 seq_num 945) dirty exists 2024-04-26T00:35:45.740 INFO:tasks.rados.rados.0.smithi045.stdout:2667: finishing rollback tid 0 to smithi04526579-26 2024-04-26T00:35:45.740 INFO:tasks.rados.rados.0.smithi045.stdout:2667: finishing rollback tid 1 to smithi04526579-26 2024-04-26T00:35:45.740 INFO:tasks.rados.rados.0.smithi045.stdout:2667: finishing rollback tid 2 to smithi04526579-26 2024-04-26T00:35:45.740 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 1971 (ObjNum 941 snap 291 seq_num 941) dirty exists 2024-04-26T00:35:45.741 INFO:tasks.rados.rados.0.smithi045.stdout:2666: finishing write tid 2 to smithi04526579-27 2024-04-26T00:35:45.741 INFO:tasks.rados.rados.0.smithi045.stdout:2666: finishing write tid 3 to smithi04526579-27 2024-04-26T00:35:45.741 INFO:tasks.rados.rados.0.smithi045.stdout:2666: finishing write tid 4 to smithi04526579-27 2024-04-26T00:35:45.741 INFO:tasks.rados.rados.0.smithi045.stdout:2666: oid 27 updating version 0 to 1915 2024-04-26T00:35:45.741 INFO:tasks.rados.rados.0.smithi045.stdout:2666: oid 27 updating version 1915 to 1916 2024-04-26T00:35:45.741 INFO:tasks.rados.rados.0.smithi045.stdout:2666: oid 27 updating version 1916 to 1917 2024-04-26T00:35:45.741 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 1917 (ObjNum 960 snap 296 seq_num 960) dirty exists 2024-04-26T00:35:45.741 INFO:tasks.rados.rados.0.smithi045.stdout:2666: left oid 27 (ObjNum 960 snap 296 seq_num 960) 2024-04-26T00:35:45.741 INFO:tasks.rados.rados.0.smithi045.stdout:2669: finishing write tid 1 to smithi04526579-11 2024-04-26T00:35:45.743 INFO:tasks.rados.rados.0.smithi045.stdout:2668: expect (ObjNum 951 snap 295 seq_num 951) 2024-04-26T00:35:46.076 INFO:tasks.rados.rados.0.smithi045.stdout:2669: finishing write tid 2 to smithi04526579-11 2024-04-26T00:35:46.076 INFO:tasks.rados.rados.0.smithi045.stdout:2669: finishing write tid 3 to smithi04526579-11 2024-04-26T00:35:46.076 INFO:tasks.rados.rados.0.smithi045.stdout:2669: finishing write tid 4 to smithi04526579-11 2024-04-26T00:35:46.076 INFO:tasks.rados.rados.0.smithi045.stdout:2669: oid 11 updating version 0 to 2160 2024-04-26T00:35:46.077 INFO:tasks.rados.rados.0.smithi045.stdout:2669: oid 11 version 2160 is already newer than 2158 2024-04-26T00:35:46.077 INFO:tasks.rados.rados.0.smithi045.stdout:2669: oid 11 version 2160 is already newer than 2159 2024-04-26T00:35:46.077 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 2160 (ObjNum 961 snap 296 seq_num 961) dirty exists 2024-04-26T00:35:46.077 INFO:tasks.rados.rados.0.smithi045.stdout:2669: left oid 11 (ObjNum 961 snap 296 seq_num 961) 2024-04-26T00:35:46.077 INFO:tasks.rados.rados.0.smithi045.stdout:2663: done (5 left) 2024-04-26T00:35:46.077 INFO:tasks.rados.rados.0.smithi045.stdout:2666: done (4 left) 2024-04-26T00:35:46.077 INFO:tasks.rados.rados.0.smithi045.stdout:2667: done (3 left) 2024-04-26T00:35:46.077 INFO:tasks.rados.rados.0.smithi045.stdout:2668: done (2 left) 2024-04-26T00:35:46.077 INFO:tasks.rados.rados.0.smithi045.stdout:2669: done (1 left) 2024-04-26T00:35:46.077 INFO:tasks.rados.rados.0.smithi045.stdout:2670: done (0 left) 2024-04-26T00:35:46.077 INFO:tasks.rados.rados.0.smithi045.stdout:2671: read oid 15 snap -1 2024-04-26T00:35:46.077 INFO:tasks.rados.rados.0.smithi045.stdout:2671: expect (ObjNum 938 snap 290 seq_num 938) 2024-04-26T00:35:46.077 INFO:tasks.rados.rados.0.smithi045.stdout:2672: copy_from oid 10 from oid 1 current snap is 296 2024-04-26T00:35:46.077 INFO:tasks.rados.rados.0.smithi045.stdout:2673: snap_create 2024-04-26T00:35:46.089 INFO:tasks.rados.rados.0.smithi045.stdout:2671: expect (ObjNum 938 snap 290 seq_num 938) 2024-04-26T00:35:46.366 INFO:tasks.rados.rados.0.smithi045.stdout:2672: finishing copy_from racing read to smithi04526579-10 2024-04-26T00:35:46.366 INFO:tasks.rados.rados.0.smithi045.stdout:2672: finishing copy_from to smithi04526579-10 2024-04-26T00:35:46.366 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 1965 (ObjNum 956 snap 296 seq_num 956) dirty exists 2024-04-26T00:35:46.637 INFO:tasks.rados.rados.0.smithi045.stdout:2671: done (2 left) 2024-04-26T00:35:46.638 INFO:tasks.rados.rados.0.smithi045.stdout:2672: done (1 left) 2024-04-26T00:35:46.638 INFO:tasks.rados.rados.0.smithi045.stdout:2673: done (0 left) 2024-04-26T00:35:46.638 INFO:tasks.rados.rados.0.smithi045.stdout:2674: snap_remove snap 291 2024-04-26T00:35:47.656 INFO:tasks.rados.rados.0.smithi045.stdout:2674: done (0 left) 2024-04-26T00:35:47.656 INFO:tasks.rados.rados.0.smithi045.stdout:2675: read oid 42 snap -1 2024-04-26T00:35:47.656 INFO:tasks.rados.rados.0.smithi045.stdout:2675: expect (ObjNum 869 snap 272 seq_num 869) 2024-04-26T00:35:47.656 INFO:tasks.rados.rados.0.smithi045.stdout:2676: copy_from oid 36 from oid 41 current snap is 297 2024-04-26T00:35:47.656 INFO:tasks.rados.rados.0.smithi045.stdout:2677: snap_remove snap 296 2024-04-26T00:35:48.538 INFO:tasks.rados.rados.0.smithi045.stdout:2677: done (2 left) 2024-04-26T00:35:48.538 INFO:tasks.rados.rados.0.smithi045.stdout:2678: delete oid 40 current snap is 297 2024-04-26T00:35:48.538 INFO:tasks.rados.rados.0.smithi045.stdout:2675: expect (ObjNum 869 snap 272 seq_num 869) 2024-04-26T00:35:48.625 INFO:tasks.rados.rados.0.smithi045.stdout:2676: finishing copy_from racing read to smithi04526579-36 2024-04-26T00:35:48.625 INFO:tasks.rados.rados.0.smithi045.stdout:2676: finishing copy_from to smithi04526579-36 2024-04-26T00:35:48.625 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 1939 (ObjNum 903 snap 277 seq_num 903) dirty exists 2024-04-26T00:35:48.625 INFO:tasks.rados.rados.0.smithi045.stdout:2675: done (2 left) 2024-04-26T00:35:48.625 INFO:tasks.rados.rados.0.smithi045.stdout:2676: done (1 left) 2024-04-26T00:35:48.625 INFO:tasks.rados.rados.0.smithi045.stdout:2678: done (0 left) 2024-04-26T00:35:48.625 INFO:tasks.rados.rados.0.smithi045.stdout:2679: snap_create 2024-04-26T00:35:49.540 INFO:tasks.rados.rados.0.smithi045.stdout:2679: done (0 left) 2024-04-26T00:35:49.540 INFO:tasks.rados.rados.0.smithi045.stdout:2680: write (excl) oid 49 current snap is 298 2024-04-26T00:35:49.540 INFO:tasks.rados.rados.0.smithi045.stdout:2680: seq_num 964 ranges {601948=400514,1577818=624130} 2024-04-26T00:35:49.556 INFO:tasks.rados.rados.0.smithi045.stdout:2680: writing smithi04526579-49 from 601948 to 1002462 tid 1 2024-04-26T00:35:49.576 INFO:tasks.rados.rados.0.smithi045.stdout:2680: writing smithi04526579-49 from 1577818 to 2201948 tid 2 2024-04-26T00:35:49.576 INFO:tasks.rados.rados.0.smithi045.stdout:2681: write (excl) oid 2 current snap is 298 2024-04-26T00:35:49.576 INFO:tasks.rados.rados.0.smithi045.stdout:2681: seq_num 965 ranges {618755=475763,1636300=457996} 2024-04-26T00:35:49.593 INFO:tasks.rados.rados.0.smithi045.stdout:2681: writing smithi04526579-2 from 618755 to 1094518 tid 1 2024-04-26T00:35:49.606 INFO:tasks.rados.rados.0.smithi045.stdout:2681: writing smithi04526579-2 from 1636300 to 2094296 tid 2 2024-04-26T00:35:49.606 INFO:tasks.rados.rados.0.smithi045.stdout:2682: delete oid 47 current snap is 298 2024-04-26T00:35:49.606 INFO:tasks.rados.rados.0.smithi045.stdout:2680: finishing write tid 1 to smithi04526579-49 2024-04-26T00:35:49.606 INFO:tasks.rados.rados.0.smithi045.stdout:2680: finishing write tid 2 to smithi04526579-49 2024-04-26T00:35:49.607 INFO:tasks.rados.rados.0.smithi045.stdout:2680: finishing write tid 3 to smithi04526579-49 2024-04-26T00:35:49.607 INFO:tasks.rados.rados.0.smithi045.stdout:2680: finishing write tid 4 to smithi04526579-49 2024-04-26T00:35:49.607 INFO:tasks.rados.rados.0.smithi045.stdout:2680: oid 49 updating version 0 to 2001 2024-04-26T00:35:49.607 INFO:tasks.rados.rados.0.smithi045.stdout:2680: oid 49 version 2001 is already newer than 1999 2024-04-26T00:35:49.607 INFO:tasks.rados.rados.0.smithi045.stdout:2680: oid 49 version 2001 is already newer than 2000 2024-04-26T00:35:49.607 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 2001 (ObjNum 964 snap 298 seq_num 964) dirty exists 2024-04-26T00:35:49.607 INFO:tasks.rados.rados.0.smithi045.stdout:2680: left oid 49 (ObjNum 964 snap 298 seq_num 964) 2024-04-26T00:35:49.613 INFO:tasks.rados.rados.0.smithi045.stdout:2680: done (2 left) 2024-04-26T00:35:49.613 INFO:tasks.rados.rados.0.smithi045.stdout:2682: done (1 left) 2024-04-26T00:35:49.613 INFO:tasks.rados.rados.0.smithi045.stdout:2683: read oid 48 snap -1 2024-04-26T00:35:49.613 INFO:tasks.rados.rados.0.smithi045.stdout:2683: expect (ObjNum 938 snap 290 seq_num 938) 2024-04-26T00:35:49.613 INFO:tasks.rados.rados.0.smithi045.stdout:2684: copy_from oid 19 from oid 26 current snap is 298 2024-04-26T00:35:49.614 INFO:tasks.rados.rados.0.smithi045.stdout:2685: snap_create 2024-04-26T00:35:49.631 INFO:tasks.rados.rados.0.smithi045.stdout:2683: expect (ObjNum 938 snap 290 seq_num 938) 2024-04-26T00:35:49.885 INFO:tasks.rados.rados.0.smithi045.stdout:2681: finishing write tid 1 to smithi04526579-2 2024-04-26T00:35:49.885 INFO:tasks.rados.rados.0.smithi045.stdout:2684: finishing copy_from racing read to smithi04526579-19 2024-04-26T00:35:49.885 INFO:tasks.rados.rados.0.smithi045.stdout:2681: finishing write tid 2 to smithi04526579-2 2024-04-26T00:35:49.885 INFO:tasks.rados.rados.0.smithi045.stdout:2681: finishing write tid 3 to smithi04526579-2 2024-04-26T00:35:49.885 INFO:tasks.rados.rados.0.smithi045.stdout:2681: finishing write tid 4 to smithi04526579-2 2024-04-26T00:35:49.885 INFO:tasks.rados.rados.0.smithi045.stdout:2681: oid 2 updating version 0 to 1941 2024-04-26T00:35:49.885 INFO:tasks.rados.rados.0.smithi045.stdout:2681: oid 2 updating version 1941 to 1943 2024-04-26T00:35:49.885 INFO:tasks.rados.rados.0.smithi045.stdout:2681: oid 2 version 1943 is already newer than 1942 2024-04-26T00:35:49.885 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 1943 (ObjNum 965 snap 298 seq_num 965) dirty exists 2024-04-26T00:35:49.885 INFO:tasks.rados.rados.0.smithi045.stdout:2681: left oid 2 (ObjNum 965 snap 298 seq_num 965) 2024-04-26T00:35:49.885 INFO:tasks.rados.rados.0.smithi045.stdout:2684: finishing copy_from to smithi04526579-19 2024-04-26T00:35:49.885 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 1971 (ObjNum 941 snap 291 seq_num 941) dirty exists 2024-04-26T00:35:51.250 INFO:tasks.rados.rados.0.smithi045.stdout:2681: done (3 left) 2024-04-26T00:35:51.250 INFO:tasks.rados.rados.0.smithi045.stdout:2683: done (2 left) 2024-04-26T00:35:51.250 INFO:tasks.rados.rados.0.smithi045.stdout:2684: done (1 left) 2024-04-26T00:35:51.250 INFO:tasks.rados.rados.0.smithi045.stdout:2685: done (0 left) 2024-04-26T00:35:51.250 INFO:tasks.rados.rados.0.smithi045.stdout:2686: snap_create 2024-04-26T00:35:51.336 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-04-26T00:35:51.336 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-04-26T00:35:51.336 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-26T00:35:51.337 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-26T00:35:51.340 INFO:tasks.ceph.mon.b:Started 2024-04-26T00:35:51.340 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:35:51.340 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:35:51.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06ba6eb640 10 monclient: get_monmap_and_config 2024-04-26T00:35:51.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06ba6eb640 10 monclient: build_initial_monmap 2024-04-26T00:35:51.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06ba6eb640 10 monclient: monmap: 2024-04-26T00:35:51.472 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:51.472 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:51.472 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:35:51.477736+0000 2024-04-26T00:35:51.472 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:35:51.477736+0000 2024-04-26T00:35:51.472 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:51.473 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:51.473 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:35:51.473 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:35:51.473 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:35:51.473 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:35:51.473 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:35:51.473 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:51.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06ba6eb640 1 Processor -- start 2024-04-26T00:35:51.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06ba6eb640 1 -- start start 2024-04-26T00:35:51.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06ba6eb640 10 monclient: init 2024-04-26T00:35:51.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06ba6eb640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:51.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06ba6eb640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-04-26T00:35:51.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06ba6eb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f06b4133b20 addr v1:172.21.15.45:6789/0 2024-04-26T00:35:51.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06ba6eb640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f06b4133f00 0x7f06b4134350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:51.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06ba6eb640 10 monclient(hunting): picked mon.noname-a con 0x7f06b4133f00 addr v2:172.21.15.37:3300/0 2024-04-26T00:35:51.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06ba6eb640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f06b4139050 0x7f06b413b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:51.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06ba6eb640 10 monclient(hunting): picked mon.noname-c con 0x7f06b4139050 addr v2:172.21.15.45:3300/0 2024-04-26T00:35:51.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06ba6eb640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06b409fe20 con 0x7f06b4133b20 2024-04-26T00:35:51.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06ba6eb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:51.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06ba6eb640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f06b4072d00 con 0x7f06b4133f00 2024-04-26T00:35:51.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06ba6eb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:51.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06ba6eb640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f06b409ba60 con 0x7f06b4139050 2024-04-26T00:35:51.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06ba6eb640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:51.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06ba6eb640 10 monclient(hunting): authenticate will time out at 1932.783383s 2024-04-26T00:35:51.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06ab7fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f06b4133f00 0x7f06b4134350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:51.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06ab7fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f06b4133f00 0x7f06b4134350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:60722/0 (socket says 172.21.15.37:60722) 2024-04-26T00:35:51.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06ab7fe640 1 -- 172.21.15.37:0/685932935 learned_addr learned my addr 172.21.15.37:0/685932935 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:51.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06ab7fe640 10 monclient(hunting): get_auth_request con 0x7f06b4133f00 auth_method 0 2024-04-26T00:35:51.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:51.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06ab7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:51.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:51.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06b8c88640 1 --2- 172.21.15.37:0/685932935 >> v2:172.21.15.45:3300/0 conn(0x7f06b4139050 0x7f06b413b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:51.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:51.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:51.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:51.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06b8c88640 10 monclient(hunting): get_auth_request con 0x7f06b4139050 auth_method 0 2024-04-26T00:35:51.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06b8c88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:51.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06b8c88640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:51.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06b8c88640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:51.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06ab7fe640 10 monclient(hunting): handle_auth_done global_id 144579 payload 274 2024-04-26T00:35:51.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06b8c88640 1 -- 172.21.15.37:0/685932935 >> v2:172.21.15.45:3300/0 conn(0x7f06b4139050 msgr2=0x7f06b413b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:35:51.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06b8c88640 1 -- 172.21.15.37:0/685932935 >> v2:172.21.15.45:3300/0 conn(0x7f06b4139050 msgr2=0x7f06b413b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:35:51.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06b8c88640 1 --2- 172.21.15.37:0/685932935 >> v2:172.21.15.45:3300/0 conn(0x7f06b4139050 0x7f06b413b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:35:51.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06ab7fe640 1 -- 172.21.15.37:0/685932935 >> v2:172.21.15.45:3300/0 conn(0x7f06b4139050 msgr2=0x7f06b413b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:51.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.473+0000 7f06b8c88640 1 --2- 172.21.15.37:0/685932935 >> v2:172.21.15.45:3300/0 conn(0x7f06b4139050 0x7f06b413b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:35:51.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.477+0000 7f06ab7fe640 1 --2- 172.21.15.37:0/685932935 >> v2:172.21.15.45:3300/0 conn(0x7f06b4139050 0x7f06b413b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:51.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.477+0000 7f06ab7fe640 1 -- 172.21.15.37:0/685932935 >> v1:172.21.15.45:6789/0 conn(0x7f06b4133b20 legacy=0x7f06b409e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:51.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.477+0000 7f06ab7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:35:51.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.477+0000 7f06ab7fe640 1 monclient: found mon.noname-a 2024-04-26T00:35:51.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.477+0000 7f06ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:51.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.477+0000 7f06ab7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:35:51.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.477+0000 7f06ab7fe640 1 -- 172.21.15.37:0/685932935 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06b409dd10 con 0x7f06b4133f00 2024-04-26T00:35:51.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ab7fe640 1 -- 172.21.15.37:0/685932935 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f06b4133f00 msgr2=0x7f06b4134350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:51.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ab7fe640 1 --2- 172.21.15.37:0/685932935 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f06b4133f00 0x7f06b4134350 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f06b409f1c0 tx=0x7f069c01c190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afb96fbd2fcac512 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:51.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b37fe640 1 -- 172.21.15.37:0/685932935 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069c0b0070 con 0x7f06b4133f00 2024-04-26T00:35:51.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:51.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:35:51.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b37fe640 10 monclient: dump: 2024-04-26T00:35:51.484 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:51.484 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:51.484 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:51.484 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:51.484 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:51.484 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:51.484 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:51.484 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:51.484 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:51.484 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:51.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b37fe640 10 monclient: _finish_auth 0 2024-04-26T00:35:51.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b37fe640 10 monclient: _check_auth_tickets 2024-04-26T00:35:51.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:51.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b37fe640 1 -- 172.21.15.37:0/685932935 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f069c0ab600 con 0x7f06b4133f00 2024-04-26T00:35:51.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b37fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:51.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b37fe640 1 -- 172.21.15.37:0/685932935 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069c0ab9d0 con 0x7f06b4133f00 2024-04-26T00:35:51.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ba6eb640 5 monclient: authenticate success, global_id 144579 2024-04-26T00:35:51.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ba6eb640 10 monclient: get_monmap_and_config success 2024-04-26T00:35:51.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ba6eb640 4 set_mon_vals no callback set 2024-04-26T00:35:51.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:51.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:51.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b37fe640 10 monclient: dump: 2024-04-26T00:35:51.485 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:51.485 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:51.485 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:51.486 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:51.486 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:51.486 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:51.486 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:51.486 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:51.486 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:51.486 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:51.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ba6eb640 10 monclient: shutdown 2024-04-26T00:35:51.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ba6eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:51.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ba6eb640 1 -- 172.21.15.37:0/685932935 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f06b4133f00 msgr2=0x7f06b4134350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:51.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ba6eb640 1 --2- 172.21.15.37:0/685932935 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f06b4133f00 0x7f06b4134350 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f06b409f1c0 tx=0x7f069c01c190 comp rx=0 tx=0).stop 2024-04-26T00:35:51.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ba6eb640 1 -- 172.21.15.37:0/685932935 shutdown_connections 2024-04-26T00:35:51.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ba6eb640 1 --2- 172.21.15.37:0/685932935 >> v2:172.21.15.45:3300/0 conn(0x7f06b4139050 0x7f06b413b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:51.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ba6eb640 1 --2- 172.21.15.37:0/685932935 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f06b4133f00 0x7f06b4134350 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:51.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ba6eb640 1 -- 172.21.15.37:0/685932935 >> 172.21.15.37:0/685932935 conn(0x7f06b40a1ad0 msgr2=0x7f06b40a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:51.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ba6eb640 1 -- 172.21.15.37:0/685932935 shutdown_connections 2024-04-26T00:35:51.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ba6eb640 1 -- 172.21.15.37:0/685932935 wait complete. 2024-04-26T00:35:51.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ba6eb640 10 monclient: build_initial_monmap 2024-04-26T00:35:51.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ba6eb640 10 monclient: monmap: 2024-04-26T00:35:51.487 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:51.487 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:35:51.487 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:35:51.487 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:35:51.487 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:51.488 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:35:51.488 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:35:51.488 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:35:51.488 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:35:51.488 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:51.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ba6eb640 1 Processor -- start 2024-04-26T00:35:51.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ba6eb640 1 -- start start 2024-04-26T00:35:51.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ba6eb640 10 monclient: init 2024-04-26T00:35:51.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ba6eb640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:51.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ba6eb640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:35:51.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ba6eb640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f06b4133b20 0x7f06b41cec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:51.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ba6eb640 10 monclient(hunting): picked mon.noname-a con 0x7f06b4133b20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:35:51.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ba6eb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06b4133f00 0x7f06b4130cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:51.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ba6eb640 10 monclient(hunting): picked mon.noname-b con 0x7f06b4133f00 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:35:51.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ba6eb640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f06b4139050 0x7f06b406ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:51.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ba6eb640 10 monclient(hunting): picked mon.noname-c con 0x7f06b4139050 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:51.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ba6eb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:51.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ba6eb640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f06b40a0bf0 con 0x7f06b4133b20 2024-04-26T00:35:51.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ba6eb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:51.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ba6eb640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f06b40a0a70 con 0x7f06b4133f00 2024-04-26T00:35:51.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ba6eb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:51.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ba6eb640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f06b40a0d70 con 0x7f06b4139050 2024-04-26T00:35:51.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ba6eb640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:51.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ba6eb640 10 monclient(hunting): authenticate will time out at 1932.790935s 2024-04-26T00:35:51.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b3fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f06b4133b20 0x7f06b41cec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:51.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b3fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f06b4133b20 0x7f06b41cec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:60736/0 (socket says 172.21.15.37:60736) 2024-04-26T00:35:51.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b3fff640 1 -- 172.21.15.37:0/2082052899 learned_addr learned my addr 172.21.15.37:0/2082052899 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:51.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b3fff640 10 monclient(hunting): get_auth_request con 0x7f06b4133b20 auth_method 0 2024-04-26T00:35:51.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:51.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:51.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:51.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b8c88640 1 --2- 172.21.15.37:0/2082052899 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f06b4139050 0x7f06b406ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:51.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b8c88640 10 monclient(hunting): get_auth_request con 0x7f06b4139050 auth_method 0 2024-04-26T00:35:51.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b8c88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:51.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b8c88640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:51.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b8c88640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:51.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:51.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:51.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:51.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b8c88640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:51.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b8c88640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:51.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b8c88640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:51.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ab7fe640 1 --2- 172.21.15.37:0/2082052899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06b4133f00 0x7f06b4130cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:51.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ab7fe640 10 monclient(hunting): get_auth_request con 0x7f06b4133f00 auth_method 0 2024-04-26T00:35:51.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:51.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ab7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:51.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:51.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b3fff640 10 monclient(hunting): handle_auth_done global_id 144582 payload 824 2024-04-26T00:35:51.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b3fff640 1 -- 172.21.15.37:0/2082052899 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f06b4139050 msgr2=0x7f06b406ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:51.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b3fff640 1 --2- 172.21.15.37:0/2082052899 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f06b4139050 0x7f06b406ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:51.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b3fff640 1 -- 172.21.15.37:0/2082052899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06b4133f00 msgr2=0x7f06b4130cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:51.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b3fff640 1 --2- 172.21.15.37:0/2082052899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06b4133f00 0x7f06b4130cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:51.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b3fff640 10 monclient: _finish_hunting 0 2024-04-26T00:35:51.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b3fff640 1 monclient: found mon.noname-a 2024-04-26T00:35:51.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:51.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:35:51.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.481+0000 7f06b3fff640 1 -- 172.21.15.37:0/2082052899 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06b4070690 con 0x7f06b4133b20 2024-04-26T00:35:51.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.485+0000 7f06b3fff640 1 -- 172.21.15.37:0/2082052899 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f06b4133b20 msgr2=0x7f06b41cec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:51.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.485+0000 7f06b3fff640 1 --2- 172.21.15.37:0/2082052899 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f06b4133b20 0x7f06b41cec90 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f06a400c480 tx=0x7f06a400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:51.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.485+0000 7f06b17fa640 1 -- 172.21.15.37:0/2082052899 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a4012a40 con 0x7f06b4133b20 2024-04-26T00:35:51.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.485+0000 7f06b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:51.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.485+0000 7f06b17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:35:51.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.485+0000 7f06b17fa640 10 monclient: dump: 2024-04-26T00:35:51.492 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:51.492 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:51.492 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:51.492 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:51.493 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:51.493 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:51.493 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:51.493 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:51.493 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:51.493 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:51.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.485+0000 7f06b17fa640 10 monclient: _finish_auth 0 2024-04-26T00:35:51.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.485+0000 7f06b17fa640 10 monclient: _check_auth_tickets 2024-04-26T00:35:51.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.485+0000 7f06b17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:51.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.485+0000 7f06ba6eb640 5 monclient: authenticate success, global_id 144582 2024-04-26T00:35:51.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.485+0000 7f06b17fa640 1 -- 172.21.15.37:0/2082052899 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06a4017070 con 0x7f06b4133b20 2024-04-26T00:35:51.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.485+0000 7f06b17fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:51.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.485+0000 7f06b17fa640 1 -- 172.21.15.37:0/2082052899 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a4012e40 con 0x7f06b4133b20 2024-04-26T00:35:51.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.485+0000 7f06b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:51.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.485+0000 7f06b17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:51.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.485+0000 7f06b17fa640 10 monclient: dump: 2024-04-26T00:35:51.494 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:51.494 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:51.494 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:51.494 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:51.494 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:51.494 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:51.494 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:51.494 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:51.494 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:51.494 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:51.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.485+0000 7f06b27fc640 4 set_mon_vals no callback set 2024-04-26T00:35:51.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.485+0000 7f06ba6eb640 10 monclient: _renew_subs 2024-04-26T00:35:51.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.485+0000 7f06ba6eb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:51.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.485+0000 7f06ba6eb640 1 -- 172.21.15.37:0/2082052899 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06b41cfb10 con 0x7f06b4133b20 2024-04-26T00:35:51.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.485+0000 7f06ba6eb640 10 monclient: _renew_subs 2024-04-26T00:35:51.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.485+0000 7f06ba6eb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:51.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.485+0000 7f06ba6eb640 1 -- 172.21.15.37:0/2082052899 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06b40720e0 con 0x7f06b4133b20 2024-04-26T00:35:51.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.485+0000 7f06ba6eb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:51.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.485+0000 7f06ba6eb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:51.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.485+0000 7f06ba6eb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:51.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.485+0000 7f06ba6eb640 1 -- 172.21.15.37:0/2082052899 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06b4072320 con 0x7f06b4133b20 2024-04-26T00:35:51.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.485+0000 7f06b17fa640 1 -- 172.21.15.37:0/2082052899 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f06a401c460 con 0x7f06b4133b20 2024-04-26T00:35:51.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.485+0000 7f06b17fa640 1 --2- 172.21.15.37:0/2082052899 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0694035e80 0x7f0694038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:51.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.485+0000 7f06b17fa640 1 -- 172.21.15.37:0/2082052899 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(621..621 src has 370..621) ==== 3915+0+0 (secure 0 0 0) 0x7f06a40488a0 con 0x7f06b4133b20 2024-04-26T00:35:51.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.485+0000 7f06ab7fe640 1 --2- 172.21.15.37:0/2082052899 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0694035e80 0x7f0694038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:51.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.485+0000 7f06ab7fe640 10 monclient: get_auth_request con 0x7f0694035e80 auth_method 0 2024-04-26T00:35:51.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.489+0000 7f06ab7fe640 1 --2- 172.21.15.37:0/2082052899 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0694035e80 0x7f0694038340 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f069c0ab000 tx=0x7f069c0a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:51.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.493+0000 7f06b17fa640 1 -- 172.21.15.37:0/2082052899 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f06a4047600 con 0x7f06b4133b20 2024-04-26T00:35:51.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.493+0000 7f06b17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:51.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.493+0000 7f06b17fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:35:51.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.713+0000 7f06ba6eb640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:35:51.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.713+0000 7f06ba6eb640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:35:51.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.713+0000 7f06ba6eb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:51.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.713+0000 7f06ba6eb640 1 -- 172.21.15.37:0/2082052899 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f06b4066f40 con 0x7f06b4133b20 2024-04-26T00:35:51.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.713+0000 7f06b17fa640 1 -- 172.21.15.37:0/2082052899 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7f06a401b410 con 0x7f06b4133b20 2024-04-26T00:35:51.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.713+0000 7f06b17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:35:51.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.713+0000 7f06b17fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:35:51.714 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":118,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:35:51.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.717+0000 7f06ba6eb640 1 -- 172.21.15.37:0/2082052899 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0694035e80 msgr2=0x7f0694038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:51.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.717+0000 7f06ba6eb640 1 --2- 172.21.15.37:0/2082052899 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0694035e80 0x7f0694038340 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f069c0ab000 tx=0x7f069c0a6000 comp rx=0 tx=0).stop 2024-04-26T00:35:51.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.717+0000 7f06ba6eb640 10 monclient: shutdown 2024-04-26T00:35:51.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.717+0000 7f06ba6eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:51.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.717+0000 7f06ba6eb640 1 -- 172.21.15.37:0/2082052899 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f06b4133b20 msgr2=0x7f06b41cec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:51.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.717+0000 7f06ba6eb640 1 --2- 172.21.15.37:0/2082052899 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f06b4133b20 0x7f06b41cec90 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f06a400c480 tx=0x7f06a400c940 comp rx=0 tx=0).stop 2024-04-26T00:35:51.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.717+0000 7f06ba6eb640 1 -- 172.21.15.37:0/2082052899 shutdown_connections 2024-04-26T00:35:51.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.717+0000 7f06ba6eb640 1 --2- 172.21.15.37:0/2082052899 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0694035e80 0x7f0694038340 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:51.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.717+0000 7f06ba6eb640 1 --2- 172.21.15.37:0/2082052899 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f06b4139050 0x7f06b406ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:51.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.717+0000 7f06ba6eb640 1 --2- 172.21.15.37:0/2082052899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06b4133f00 0x7f06b4130cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:51.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.717+0000 7f06ba6eb640 1 --2- 172.21.15.37:0/2082052899 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f06b4133b20 0x7f06b41cec90 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:51.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.717+0000 7f06ba6eb640 1 -- 172.21.15.37:0/2082052899 >> 172.21.15.37:0/2082052899 conn(0x7f06b40a1ad0 msgr2=0x7f06b4131c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:51.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.717+0000 7f06ba6eb640 1 -- 172.21.15.37:0/2082052899 shutdown_connections 2024-04-26T00:35:51.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:51.717+0000 7f06ba6eb640 1 -- 172.21.15.37:0/2082052899 wait complete. 2024-04-26T00:35:52.259 INFO:tasks.rados.rados.0.smithi045.stdout:2686: done (0 left) 2024-04-26T00:35:52.260 INFO:tasks.rados.rados.0.smithi045.stdout:2687: snap_create 2024-04-26T00:35:53.261 INFO:tasks.rados.rados.0.smithi045.stdout:2687: done (0 left) 2024-04-26T00:35:53.261 INFO:tasks.rados.rados.0.smithi045.stdout:2688: rollback oid 20 current snap is 301 2024-04-26T00:35:53.261 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 20 to 297 2024-04-26T00:35:53.261 INFO:tasks.rados.rados.0.smithi045.stdout:2689: read oid 24 snap -1 2024-04-26T00:35:53.261 INFO:tasks.rados.rados.0.smithi045.stdout:2689: expect deleted 2024-04-26T00:35:53.261 INFO:tasks.rados.rados.0.smithi045.stdout:2690: read oid 13 snap -1 2024-04-26T00:35:53.261 INFO:tasks.rados.rados.0.smithi045.stdout:2690: expect deleted 2024-04-26T00:35:53.261 INFO:tasks.rados.rados.0.smithi045.stdout:2691: read oid 8 snap -1 2024-04-26T00:35:53.262 INFO:tasks.rados.rados.0.smithi045.stdout:2691: expect (ObjNum 924 snap 287 seq_num 924) 2024-04-26T00:35:53.262 INFO:tasks.rados.rados.0.smithi045.stdout:2692: delete oid 25 current snap is 301 2024-04-26T00:35:53.280 INFO:tasks.rados.rados.0.smithi045.stdout:2689: done (4 left) 2024-04-26T00:35:53.280 INFO:tasks.rados.rados.0.smithi045.stdout:2690: done (3 left) 2024-04-26T00:35:53.280 INFO:tasks.rados.rados.0.smithi045.stdout:2692: done (2 left) 2024-04-26T00:35:53.280 INFO:tasks.rados.rados.0.smithi045.stdout:2693: copy_from oid 18 from oid 42 current snap is 301 2024-04-26T00:35:53.280 INFO:tasks.rados.rados.0.smithi045.stdout:2694: read oid 7 snap -1 2024-04-26T00:35:53.280 INFO:tasks.rados.rados.0.smithi045.stdout:2694: expect (ObjNum 922 snap 286 seq_num 922) 2024-04-26T00:35:53.280 INFO:tasks.rados.rados.0.smithi045.stdout:2695: write (excl) oid 33 current snap is 301 2024-04-26T00:35:53.280 INFO:tasks.rados.rados.0.smithi045.stdout:2695: seq_num 968 ranges {669176=417863,1811746=457430} 2024-04-26T00:35:53.297 INFO:tasks.rados.rados.0.smithi045.stdout:2695: writing smithi04526579-33 from 669176 to 1087039 tid 1 2024-04-26T00:35:53.317 INFO:tasks.rados.rados.0.smithi045.stdout:2695: writing smithi04526579-33 from 1811746 to 2269176 tid 2 2024-04-26T00:35:53.317 INFO:tasks.rados.rados.0.smithi045.stdout:2696: delete oid 32 current snap is 301 2024-04-26T00:35:53.317 INFO:tasks.rados.rados.0.smithi045.stdout:2691: expect (ObjNum 924 snap 287 seq_num 924) 2024-04-26T00:35:53.583 INFO:tasks.rados.rados.0.smithi045.stdout:2688: finishing rollback tid 0 to smithi04526579-20 2024-04-26T00:35:53.583 INFO:tasks.rados.rados.0.smithi045.stdout:2688: finishing rollback tid 1 to smithi04526579-20 2024-04-26T00:35:53.583 INFO:tasks.rados.rados.0.smithi045.stdout:2688: finishing rollback tid 2 to smithi04526579-20 2024-04-26T00:35:53.584 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 2183 (ObjNum 933 snap 289 seq_num 933) dirty exists 2024-04-26T00:35:53.584 INFO:tasks.rados.rados.0.smithi045.stdout:2694: expect (ObjNum 922 snap 286 seq_num 922) 2024-04-26T00:35:53.812 INFO:tasks.rados.rados.0.smithi045.stdout:2693: finishing copy_from racing read to smithi04526579-18 2024-04-26T00:35:53.812 INFO:tasks.rados.rados.0.smithi045.stdout:2693: finishing copy_from to smithi04526579-18 2024-04-26T00:35:53.812 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 1470 (ObjNum 869 snap 272 seq_num 869) dirty exists 2024-04-26T00:35:53.812 INFO:tasks.rados.rados.0.smithi045.stdout:2695: finishing write tid 1 to smithi04526579-33 2024-04-26T00:35:53.812 INFO:tasks.rados.rados.0.smithi045.stdout:2695: finishing write tid 2 to smithi04526579-33 2024-04-26T00:35:53.812 INFO:tasks.rados.rados.0.smithi045.stdout:2695: finishing write tid 3 to smithi04526579-33 2024-04-26T00:35:53.812 INFO:tasks.rados.rados.0.smithi045.stdout:2695: finishing write tid 4 to smithi04526579-33 2024-04-26T00:35:53.813 INFO:tasks.rados.rados.0.smithi045.stdout:2695: oid 33 updating version 0 to 2327 2024-04-26T00:35:53.813 INFO:tasks.rados.rados.0.smithi045.stdout:2695: oid 33 version 2327 is already newer than 2326 2024-04-26T00:35:53.813 INFO:tasks.rados.rados.0.smithi045.stdout:2695: oid 33 updating version 2327 to 2328 2024-04-26T00:35:53.813 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 2328 (ObjNum 968 snap 301 seq_num 968) dirty exists 2024-04-26T00:35:53.813 INFO:tasks.rados.rados.0.smithi045.stdout:2695: left oid 33 (ObjNum 968 snap 301 seq_num 968) 2024-04-26T00:35:53.813 INFO:tasks.rados.rados.0.smithi045.stdout:2688: done (5 left) 2024-04-26T00:35:53.813 INFO:tasks.rados.rados.0.smithi045.stdout:2691: done (4 left) 2024-04-26T00:35:53.813 INFO:tasks.rados.rados.0.smithi045.stdout:2693: done (3 left) 2024-04-26T00:35:53.813 INFO:tasks.rados.rados.0.smithi045.stdout:2694: done (2 left) 2024-04-26T00:35:53.813 INFO:tasks.rados.rados.0.smithi045.stdout:2695: done (1 left) 2024-04-26T00:35:53.813 INFO:tasks.rados.rados.0.smithi045.stdout:2696: done (0 left) 2024-04-26T00:35:53.813 INFO:tasks.rados.rados.0.smithi045.stdout:2697: read oid 32 snap -1 2024-04-26T00:35:53.813 INFO:tasks.rados.rados.0.smithi045.stdout:2697: expect deleted 2024-04-26T00:35:53.814 INFO:tasks.rados.rados.0.smithi045.stdout:2698: snap_remove snap 295 2024-04-26T00:35:54.560 INFO:tasks.rados.rados.0.smithi045.stdout:2698: done (1 left) 2024-04-26T00:35:54.560 INFO:tasks.rados.rados.0.smithi045.stdout:2699: write oid 33 current snap is 301 2024-04-26T00:35:54.560 INFO:tasks.rados.rados.0.smithi045.stdout:2699: seq_num 970 ranges {702790=568361,1812357=490433} 2024-04-26T00:35:54.567 INFO:tasks.rados.rados.0.smithi045.stdout:2699: writing smithi04526579-33 from 702790 to 1271151 tid 1 2024-04-26T00:35:54.573 INFO:tasks.rados.rados.0.smithi045.stdout:2699: writing smithi04526579-33 from 1812357 to 2302790 tid 2 2024-04-26T00:35:54.574 INFO:tasks.rados.rados.0.smithi045.stdout:2700: read oid 7 snap -1 2024-04-26T00:35:54.574 INFO:tasks.rados.rados.0.smithi045.stdout:2700: expect (ObjNum 922 snap 286 seq_num 922) 2024-04-26T00:35:54.574 INFO:tasks.rados.rados.0.smithi045.stdout:2701: copy_from oid 2 from oid 49 current snap is 301 2024-04-26T00:35:54.574 INFO:tasks.rados.rados.0.smithi045.stdout:2697: done (3 left) 2024-04-26T00:35:54.574 INFO:tasks.rados.rados.0.smithi045.stdout:2702: read oid 30 snap -1 2024-04-26T00:35:54.574 INFO:tasks.rados.rados.0.smithi045.stdout:2702: expect (ObjNum 836 snap 260 seq_num 836) 2024-04-26T00:35:54.574 INFO:tasks.rados.rados.0.smithi045.stdout:2703: write oid 46 current snap is 301 2024-04-26T00:35:54.574 INFO:tasks.rados.rados.0.smithi045.stdout:2703: seq_num 971 ranges {719597=643610,1870839=448758} 2024-04-26T00:35:54.582 INFO:tasks.rados.rados.0.smithi045.stdout:2703: writing smithi04526579-46 from 719597 to 1363207 tid 1 2024-04-26T00:35:54.588 INFO:tasks.rados.rados.0.smithi045.stdout:2703: writing smithi04526579-46 from 1870839 to 2319597 tid 2 2024-04-26T00:35:54.588 INFO:tasks.rados.rados.0.smithi045.stdout:2704: write (excl) oid 36 current snap is 301 2024-04-26T00:35:54.588 INFO:tasks.rados.rados.0.smithi045.stdout:2704: seq_num 972 ranges {736404=718859,1929321=407083} 2024-04-26T00:35:54.596 INFO:tasks.rados.rados.0.smithi045.stdout:2704: writing smithi04526579-36 from 736404 to 1455263 tid 1 2024-04-26T00:35:54.602 INFO:tasks.rados.rados.0.smithi045.stdout:2704: writing smithi04526579-36 from 1929321 to 2336404 tid 2 2024-04-26T00:35:54.602 INFO:tasks.rados.rados.0.smithi045.stdout:2705: delete oid 19 current snap is 301 2024-04-26T00:35:54.602 INFO:tasks.rados.rados.0.smithi045.stdout:2702: expect (ObjNum 836 snap 260 seq_num 836) 2024-04-26T00:35:54.723 INFO:tasks.rados.rados.0.smithi045.stdout:2703: finishing write tid 1 to smithi04526579-46 2024-04-26T00:35:54.723 INFO:tasks.rados.rados.0.smithi045.stdout:2700: expect (ObjNum 922 snap 286 seq_num 922) 2024-04-26T00:35:54.734 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:35:54.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f84a2024640 10 monclient: get_monmap_and_config 2024-04-26T00:35:54.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f84a2024640 10 monclient: build_initial_monmap 2024-04-26T00:35:54.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f84a2024640 10 monclient: monmap: 2024-04-26T00:35:54.863 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:54.863 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:54.863 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:35:54.869142+0000 2024-04-26T00:35:54.864 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:35:54.869142+0000 2024-04-26T00:35:54.864 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:54.864 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:54.864 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:35:54.864 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:35:54.864 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:35:54.864 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:35:54.864 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:35:54.864 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:54.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f84a2024640 1 Processor -- start 2024-04-26T00:35:54.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f84a2024640 1 -- start start 2024-04-26T00:35:54.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f84a2024640 10 monclient: init 2024-04-26T00:35:54.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f84a2024640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:54.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f84a2024640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-04-26T00:35:54.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f84a2024640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f849c136eb0 0x7f849c137290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:54.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f84a2024640 10 monclient(hunting): picked mon.noname-b con 0x7f849c136eb0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:54.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f84a2024640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f849c1377d0 0x7f849c133b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:54.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f84a2024640 10 monclient(hunting): picked mon.noname-a con 0x7f849c1377d0 addr v2:172.21.15.37:3300/0 2024-04-26T00:35:54.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f84a2024640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f849c1340d0 addr v1:172.21.15.45:6789/0 2024-04-26T00:35:54.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f84a2024640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f849c09b8c0 con 0x7f849c1340d0 2024-04-26T00:35:54.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f84a2024640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:54.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f84a2024640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f849c09c7d0 con 0x7f849c1377d0 2024-04-26T00:35:54.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f84a2024640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:54.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f84a2024640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f849c09b300 con 0x7f849c136eb0 2024-04-26T00:35:54.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f84a2024640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:54.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f84a2024640 10 monclient(hunting): authenticate will time out at 1936.175030s 2024-04-26T00:35:54.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f849b7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f849c136eb0 0x7f849c137290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:54.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f849b7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f849c136eb0 0x7f849c137290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:50818/0 (socket says 172.21.15.37:50818) 2024-04-26T00:35:54.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f849b7fe640 1 -- 172.21.15.37:0/3052185557 learned_addr learned my addr 172.21.15.37:0/3052185557 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:54.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f849b7fe640 10 monclient(hunting): get_auth_request con 0x7f849c136eb0 auth_method 0 2024-04-26T00:35:54.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f849b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:54.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f849b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:54.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f849b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:54.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f849affd640 1 --2- 172.21.15.37:0/3052185557 >> v2:172.21.15.37:3300/0 conn(0x7f849c1377d0 0x7f849c133b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:54.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f849affd640 10 monclient(hunting): get_auth_request con 0x7f849c1377d0 auth_method 0 2024-04-26T00:35:54.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f849affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:54.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f849affd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:54.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f849affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:54.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f849b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:54.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f849b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:54.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f849b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:54.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f849affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:54.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f849affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:54.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f849affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:54.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f849affd640 10 monclient(hunting): handle_auth_done global_id 144585 payload 274 2024-04-26T00:35:54.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f849affd640 1 -- 172.21.15.37:0/3052185557 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f849c136eb0 msgr2=0x7f849c137290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:54.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f849affd640 1 --2- 172.21.15.37:0/3052185557 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f849c136eb0 0x7f849c137290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:54.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f849affd640 1 -- 172.21.15.37:0/3052185557 >> v1:172.21.15.45:6789/0 conn(0x7f849c1340d0 legacy=0x7f849c13e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:54.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f849affd640 10 monclient: _finish_hunting 0 2024-04-26T00:35:54.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f849affd640 1 monclient: found mon.noname-a 2024-04-26T00:35:54.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f849affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:54.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f849affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:35:54.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f849affd640 1 -- 172.21.15.37:0/3052185557 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f849c09b5e0 con 0x7f849c1377d0 2024-04-26T00:35:54.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.865+0000 7f849b7fe640 1 --2- 172.21.15.37:0/3052185557 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f849c136eb0 0x7f849c137290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:35:54.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.869+0000 7f849affd640 1 -- 172.21.15.37:0/3052185557 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f849c1377d0 msgr2=0x7f849c133b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:54.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.869+0000 7f849affd640 1 --2- 172.21.15.37:0/3052185557 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f849c1377d0 0x7f849c133b90 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f84840097c0 tx=0x7f848401c140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a321412e2bfa15d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:54.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.869+0000 7f849a7fc640 1 -- 172.21.15.37:0/3052185557 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84840b0070 con 0x7f849c1377d0 2024-04-26T00:35:54.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.869+0000 7f849a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:54.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.869+0000 7f849a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:35:54.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.869+0000 7f849a7fc640 10 monclient: dump: 2024-04-26T00:35:54.886 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:54.886 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:54.886 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:54.886 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:54.886 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:54.886 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:54.886 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:54.886 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:54.886 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:54.886 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:54.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.869+0000 7f849a7fc640 10 monclient: _finish_auth 0 2024-04-26T00:35:54.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.869+0000 7f849a7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:35:54.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.869+0000 7f849a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:54.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.869+0000 7f84a2024640 5 monclient: authenticate success, global_id 144585 2024-04-26T00:35:54.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.869+0000 7f849a7fc640 1 -- 172.21.15.37:0/3052185557 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84840ab600 con 0x7f849c1377d0 2024-04-26T00:35:54.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.869+0000 7f849a7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:54.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.869+0000 7f849a7fc640 1 -- 172.21.15.37:0/3052185557 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84840ab9d0 con 0x7f849c1377d0 2024-04-26T00:35:54.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.869+0000 7f849a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:54.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.869+0000 7f849a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:54.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.869+0000 7f849a7fc640 10 monclient: dump: 2024-04-26T00:35:54.887 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:54.887 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:54.887 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:54.888 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:54.888 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:54.888 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:54.888 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:54.888 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:54.888 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:54.888 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:54.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.869+0000 7f84a2024640 10 monclient: get_monmap_and_config success 2024-04-26T00:35:54.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.869+0000 7f84a2024640 4 set_mon_vals no callback set 2024-04-26T00:35:54.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f84a2024640 10 monclient: shutdown 2024-04-26T00:35:54.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f84a2024640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:54.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f84a2024640 1 -- 172.21.15.37:0/3052185557 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f849c1377d0 msgr2=0x7f849c133b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:54.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f84a2024640 1 --2- 172.21.15.37:0/3052185557 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f849c1377d0 0x7f849c133b90 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f84840097c0 tx=0x7f848401c140 comp rx=0 tx=0).stop 2024-04-26T00:35:54.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f84a2024640 1 -- 172.21.15.37:0/3052185557 shutdown_connections 2024-04-26T00:35:54.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f84a2024640 1 --2- 172.21.15.37:0/3052185557 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f849c1377d0 0x7f849c133b90 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:54.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f84a2024640 1 --2- 172.21.15.37:0/3052185557 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f849c136eb0 0x7f849c137290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:54.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f84a2024640 1 -- 172.21.15.37:0/3052185557 >> 172.21.15.37:0/3052185557 conn(0x7f849c0a6300 msgr2=0x7f849c0a6710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:54.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f84a2024640 1 -- 172.21.15.37:0/3052185557 shutdown_connections 2024-04-26T00:35:54.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f84a2024640 1 -- 172.21.15.37:0/3052185557 wait complete. 2024-04-26T00:35:54.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f84a2024640 10 monclient: build_initial_monmap 2024-04-26T00:35:54.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f84a2024640 10 monclient: monmap: 2024-04-26T00:35:54.890 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:54.890 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:35:54.890 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:35:54.890 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:35:54.890 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:54.891 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:35:54.891 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:35:54.891 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:35:54.891 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:35:54.891 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:54.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f84a2024640 1 Processor -- start 2024-04-26T00:35:54.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f84a2024640 1 -- start start 2024-04-26T00:35:54.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f84a2024640 10 monclient: init 2024-04-26T00:35:54.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f84a2024640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:54.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f84a2024640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:35:54.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f84a2024640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f849c1340d0 0x7f849c1cf470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:54.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f84a2024640 10 monclient(hunting): picked mon.noname-b con 0x7f849c1340d0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:35:54.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f84a2024640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f849c136eb0 0x7f849c13e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:54.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f84a2024640 10 monclient(hunting): picked mon.noname-a con 0x7f849c136eb0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:35:54.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f84a2024640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f849c1377d0 0x7f849c13ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:54.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f84a2024640 10 monclient(hunting): picked mon.noname-c con 0x7f849c1377d0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:54.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f84a2024640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:54.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f84a2024640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f849c0a5b40 con 0x7f849c136eb0 2024-04-26T00:35:54.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f84a2024640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:54.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f84a2024640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f849c0a59c0 con 0x7f849c1340d0 2024-04-26T00:35:54.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f84a2024640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:54.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f84a2024640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f849c09a9e0 con 0x7f849c1377d0 2024-04-26T00:35:54.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f84a2024640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:54.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f84a2024640 10 monclient(hunting): authenticate will time out at 1936.182885s 2024-04-26T00:35:54.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f849affd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f849c136eb0 0x7f849c13e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:54.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f849affd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f849c136eb0 0x7f849c13e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:60754/0 (socket says 172.21.15.37:60754) 2024-04-26T00:35:54.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f849affd640 1 -- 172.21.15.37:0/3618083271 learned_addr learned my addr 172.21.15.37:0/3618083271 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:54.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f849affd640 10 monclient(hunting): get_auth_request con 0x7f849c136eb0 auth_method 0 2024-04-26T00:35:54.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f849affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:54.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f849affd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:54.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f849affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:54.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f849bfff640 1 --2- 172.21.15.37:0/3618083271 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f849c1377d0 0x7f849c13ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:54.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f849bfff640 10 monclient(hunting): get_auth_request con 0x7f849c1377d0 auth_method 0 2024-04-26T00:35:54.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f849bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:54.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f849bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:54.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f849bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:54.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f849affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:54.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f849affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:54.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f849affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:54.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f849b7fe640 1 --2- 172.21.15.37:0/3618083271 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f849c1340d0 0x7f849c1cf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:54.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f849b7fe640 10 monclient(hunting): get_auth_request con 0x7f849c1340d0 auth_method 0 2024-04-26T00:35:54.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f849b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:54.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f849b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:54.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f849b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:54.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f849bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:54.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f849bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:54.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f849bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:54.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f849affd640 10 monclient(hunting): handle_auth_done global_id 144588 payload 824 2024-04-26T00:35:54.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f849affd640 1 -- 172.21.15.37:0/3618083271 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f849c1377d0 msgr2=0x7f849c13ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:54.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f849affd640 1 --2- 172.21.15.37:0/3618083271 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f849c1377d0 0x7f849c13ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:54.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f849affd640 1 -- 172.21.15.37:0/3618083271 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f849c1340d0 msgr2=0x7f849c1cf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:54.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f849affd640 1 --2- 172.21.15.37:0/3618083271 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f849c1340d0 0x7f849c1cf470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:54.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f849affd640 10 monclient: _finish_hunting 0 2024-04-26T00:35:54.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f849affd640 1 monclient: found mon.noname-a 2024-04-26T00:35:54.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f849affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:54.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f849affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:35:54.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.873+0000 7f849affd640 1 -- 172.21.15.37:0/3618083271 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f849c13cff0 con 0x7f849c136eb0 2024-04-26T00:35:54.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.877+0000 7f849affd640 1 -- 172.21.15.37:0/3618083271 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f849c136eb0 msgr2=0x7f849c13e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:54.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.877+0000 7f849affd640 1 --2- 172.21.15.37:0/3618083271 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f849c136eb0 0x7f849c13e500 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f84840ae040 tx=0x7f848401a980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:54.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.877+0000 7f8493fff640 1 -- 172.21.15.37:0/3618083271 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84840b0050 con 0x7f849c136eb0 2024-04-26T00:35:54.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.877+0000 7f8493fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:54.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.877+0000 7f8493fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:35:54.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.877+0000 7f8493fff640 10 monclient: dump: 2024-04-26T00:35:54.899 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:54.899 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:54.899 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:54.899 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:54.899 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:54.899 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:54.899 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:54.899 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:54.899 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:54.899 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:54.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.877+0000 7f8493fff640 10 monclient: _finish_auth 0 2024-04-26T00:35:54.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.877+0000 7f8493fff640 10 monclient: _check_auth_tickets 2024-04-26T00:35:54.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.877+0000 7f8493fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:54.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.877+0000 7f8493fff640 1 -- 172.21.15.37:0/3618083271 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84840bc070 con 0x7f849c136eb0 2024-04-26T00:35:54.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.877+0000 7f84a2024640 5 monclient: authenticate success, global_id 144588 2024-04-26T00:35:54.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.877+0000 7f84a2024640 10 monclient: _renew_subs 2024-04-26T00:35:54.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.877+0000 7f84a2024640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:54.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.877+0000 7f84a2024640 1 -- 172.21.15.37:0/3618083271 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f849c09ae10 con 0x7f849c136eb0 2024-04-26T00:35:54.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.877+0000 7f8493fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:54.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.877+0000 7f8493fff640 1 -- 172.21.15.37:0/3618083271 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84840b5a40 con 0x7f849c136eb0 2024-04-26T00:35:54.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.877+0000 7f8493fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:54.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.877+0000 7f8493fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:54.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.877+0000 7f8493fff640 10 monclient: dump: 2024-04-26T00:35:54.900 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:54.900 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:54.900 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:54.900 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:54.900 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:54.900 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:54.900 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:54.900 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:54.901 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:54.901 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:54.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.877+0000 7f8499ffb640 4 set_mon_vals no callback set 2024-04-26T00:35:54.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.877+0000 7f84a2024640 10 monclient: _renew_subs 2024-04-26T00:35:54.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.877+0000 7f84a2024640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:54.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.877+0000 7f84a2024640 1 -- 172.21.15.37:0/3618083271 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f849c13cd10 con 0x7f849c136eb0 2024-04-26T00:35:54.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.877+0000 7f84a2024640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:54.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.877+0000 7f84a2024640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:54.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.877+0000 7f84a2024640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:54.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.877+0000 7f8493fff640 1 -- 172.21.15.37:0/3618083271 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f84840b5c20 con 0x7f849c136eb0 2024-04-26T00:35:54.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.877+0000 7f84a2024640 1 -- 172.21.15.37:0/3618083271 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f849c066f40 con 0x7f849c136eb0 2024-04-26T00:35:54.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.877+0000 7f8493fff640 1 --2- 172.21.15.37:0/3618083271 >> v2:172.21.15.45:6812/2379918623 conn(0x7f847c035ed0 0x7f847c038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:54.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.877+0000 7f8493fff640 1 -- 172.21.15.37:0/3618083271 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(625..625 src has 370..625) ==== 3927+0+0 (secure 0 0 0) 0x7f84840e30d0 con 0x7f849c136eb0 2024-04-26T00:35:54.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.877+0000 7f849b7fe640 1 --2- 172.21.15.37:0/3618083271 >> v2:172.21.15.45:6812/2379918623 conn(0x7f847c035ed0 0x7f847c038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:54.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.877+0000 7f849b7fe640 10 monclient: get_auth_request con 0x7f847c035ed0 auth_method 0 2024-04-26T00:35:54.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.881+0000 7f849b7fe640 1 --2- 172.21.15.37:0/3618083271 >> v2:172.21.15.45:6812/2379918623 conn(0x7f847c035ed0 0x7f847c038390 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f848c009a40 tx=0x7f848c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:54.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.881+0000 7f8493fff640 1 -- 172.21.15.37:0/3618083271 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f849c09b8c0 con 0x7f849c136eb0 2024-04-26T00:35:54.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.881+0000 7f8493fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:54.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:54.881+0000 7f8493fff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:35:54.947 INFO:tasks.rados.rados.0.smithi045.stdout:2699: finishing write tid 1 to smithi04526579-33 2024-04-26T00:35:54.947 INFO:tasks.rados.rados.0.smithi045.stdout:2699: finishing write tid 2 to smithi04526579-33 2024-04-26T00:35:54.947 INFO:tasks.rados.rados.0.smithi045.stdout:2699: finishing write tid 3 to smithi04526579-33 2024-04-26T00:35:54.948 INFO:tasks.rados.rados.0.smithi045.stdout:2699: finishing write tid 4 to smithi04526579-33 2024-04-26T00:35:54.948 INFO:tasks.rados.rados.0.smithi045.stdout:2699: oid 33 updating version 0 to 2331 2024-04-26T00:35:54.948 INFO:tasks.rados.rados.0.smithi045.stdout:2699: oid 33 version 2331 is already newer than 2329 2024-04-26T00:35:54.948 INFO:tasks.rados.rados.0.smithi045.stdout:2699: oid 33 version 2331 is already newer than 2330 2024-04-26T00:35:54.948 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 2331 (ObjNum 970 snap 301 seq_num 970) dirty exists 2024-04-26T00:35:54.948 INFO:tasks.rados.rados.0.smithi045.stdout:2699: left oid 33 (ObjNum 970 snap 301 seq_num 970) 2024-04-26T00:35:54.948 INFO:tasks.rados.rados.0.smithi045.stdout:2701: finishing copy_from racing read to smithi04526579-2 2024-04-26T00:35:54.948 INFO:tasks.rados.rados.0.smithi045.stdout:2701: finishing copy_from to smithi04526579-2 2024-04-26T00:35:54.948 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 2001 (ObjNum 964 snap 298 seq_num 964) dirty exists 2024-04-26T00:35:54.948 INFO:tasks.rados.rados.0.smithi045.stdout:2703: finishing write tid 2 to smithi04526579-46 2024-04-26T00:35:54.948 INFO:tasks.rados.rados.0.smithi045.stdout:2703: finishing write tid 3 to smithi04526579-46 2024-04-26T00:35:54.948 INFO:tasks.rados.rados.0.smithi045.stdout:2703: finishing write tid 4 to smithi04526579-46 2024-04-26T00:35:54.948 INFO:tasks.rados.rados.0.smithi045.stdout:2703: oid 46 updating version 0 to 1942 2024-04-26T00:35:54.948 INFO:tasks.rados.rados.0.smithi045.stdout:2703: oid 46 version 1942 is already newer than 1940 2024-04-26T00:35:54.949 INFO:tasks.rados.rados.0.smithi045.stdout:2703: oid 46 version 1942 is already newer than 1941 2024-04-26T00:35:54.949 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 1942 (ObjNum 971 snap 301 seq_num 971) dirty exists 2024-04-26T00:35:54.949 INFO:tasks.rados.rados.0.smithi045.stdout:2703: left oid 46 (ObjNum 971 snap 301 seq_num 971) 2024-04-26T00:35:54.949 INFO:tasks.rados.rados.0.smithi045.stdout:2704: finishing write tid 1 to smithi04526579-36 2024-04-26T00:35:54.949 INFO:tasks.rados.rados.0.smithi045.stdout:2704: finishing write tid 2 to smithi04526579-36 2024-04-26T00:35:54.949 INFO:tasks.rados.rados.0.smithi045.stdout:2699: done (6 left) 2024-04-26T00:35:54.949 INFO:tasks.rados.rados.0.smithi045.stdout:2700: done (5 left) 2024-04-26T00:35:54.949 INFO:tasks.rados.rados.0.smithi045.stdout:2701: done (4 left) 2024-04-26T00:35:54.949 INFO:tasks.rados.rados.0.smithi045.stdout:2702: done (3 left) 2024-04-26T00:35:54.949 INFO:tasks.rados.rados.0.smithi045.stdout:2703: done (2 left) 2024-04-26T00:35:54.949 INFO:tasks.rados.rados.0.smithi045.stdout:2705: done (1 left) 2024-04-26T00:35:54.949 INFO:tasks.rados.rados.0.smithi045.stdout:2706: delete oid 24 current snap is 301 2024-04-26T00:35:54.949 INFO:tasks.rados.rados.0.smithi045.stdout:2704: finishing write tid 3 to smithi04526579-36 2024-04-26T00:35:54.949 INFO:tasks.rados.rados.0.smithi045.stdout:2704: finishing write tid 4 to smithi04526579-36 2024-04-26T00:35:54.949 INFO:tasks.rados.rados.0.smithi045.stdout:2704: oid 36 updating version 0 to 1944 2024-04-26T00:35:54.949 INFO:tasks.rados.rados.0.smithi045.stdout:2704: oid 36 version 1944 is already newer than 1943 2024-04-26T00:35:54.950 INFO:tasks.rados.rados.0.smithi045.stdout:2704: oid 36 updating version 1944 to 1945 2024-04-26T00:35:54.950 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 1945 (ObjNum 972 snap 301 seq_num 972) dirty exists 2024-04-26T00:35:54.950 INFO:tasks.rados.rados.0.smithi045.stdout:2704: left oid 36 (ObjNum 972 snap 301 seq_num 972) 2024-04-26T00:35:54.952 INFO:tasks.rados.rados.0.smithi045.stdout:2704: done (1 left) 2024-04-26T00:35:54.952 INFO:tasks.rados.rados.0.smithi045.stdout:2706: done (0 left) 2024-04-26T00:35:54.952 INFO:tasks.rados.rados.0.smithi045.stdout:2707: write oid 19 current snap is 301 2024-04-26T00:35:54.952 INFO:tasks.rados.rados.0.smithi045.stdout:2707: seq_num 975 ranges {786825=660959,2104767=282058} 2024-04-26T00:35:54.971 INFO:tasks.rados.rados.0.smithi045.stdout:2707: writing smithi04526579-19 from 786825 to 1447784 tid 1 2024-04-26T00:35:54.983 INFO:tasks.rados.rados.0.smithi045.stdout:2707: writing smithi04526579-19 from 2104767 to 2386825 tid 2 2024-04-26T00:35:54.983 INFO:tasks.rados.rados.0.smithi045.stdout:2708: snap_remove snap 299 2024-04-26T00:35:55.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.113+0000 7f84a2024640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:35:55.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.113+0000 7f84a2024640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:35:55.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.113+0000 7f84a2024640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:55.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.113+0000 7f84a2024640 1 -- 172.21.15.37:0/3618083271 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f849c13b5e0 con 0x7f849c136eb0 2024-04-26T00:35:55.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.113+0000 7f8493fff640 1 -- 172.21.15.37:0/3618083271 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f84840e1e20 con 0x7f849c136eb0 2024-04-26T00:35:55.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.113+0000 7f8493fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:35:55.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.113+0000 7f8493fff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:35:55.114 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":120,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:35:55.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.117+0000 7f84a2024640 1 -- 172.21.15.37:0/3618083271 >> v2:172.21.15.45:6812/2379918623 conn(0x7f847c035ed0 msgr2=0x7f847c038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:55.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.117+0000 7f84a2024640 1 --2- 172.21.15.37:0/3618083271 >> v2:172.21.15.45:6812/2379918623 conn(0x7f847c035ed0 0x7f847c038390 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f848c009a40 tx=0x7f848c008040 comp rx=0 tx=0).stop 2024-04-26T00:35:55.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.117+0000 7f84a2024640 10 monclient: shutdown 2024-04-26T00:35:55.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.117+0000 7f84a2024640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:55.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.117+0000 7f84a2024640 1 -- 172.21.15.37:0/3618083271 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f849c136eb0 msgr2=0x7f849c13e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:55.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.117+0000 7f84a2024640 1 --2- 172.21.15.37:0/3618083271 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f849c136eb0 0x7f849c13e500 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f84840ae040 tx=0x7f848401a980 comp rx=0 tx=0).stop 2024-04-26T00:35:55.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.117+0000 7f84a2024640 1 -- 172.21.15.37:0/3618083271 shutdown_connections 2024-04-26T00:35:55.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.117+0000 7f84a2024640 1 --2- 172.21.15.37:0/3618083271 >> v2:172.21.15.45:6812/2379918623 conn(0x7f847c035ed0 0x7f847c038390 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:55.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.117+0000 7f84a2024640 1 --2- 172.21.15.37:0/3618083271 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f849c1377d0 0x7f849c13ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:55.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.117+0000 7f84a2024640 1 --2- 172.21.15.37:0/3618083271 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f849c136eb0 0x7f849c13e500 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:55.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.117+0000 7f84a2024640 1 --2- 172.21.15.37:0/3618083271 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f849c1340d0 0x7f849c1cf470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:55.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.117+0000 7f84a2024640 1 -- 172.21.15.37:0/3618083271 >> 172.21.15.37:0/3618083271 conn(0x7f849c0a6300 msgr2=0x7f849c0a7300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:55.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.117+0000 7f84a2024640 1 -- 172.21.15.37:0/3618083271 shutdown_connections 2024-04-26T00:35:55.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.117+0000 7f84a2024640 1 -- 172.21.15.37:0/3618083271 wait complete. 2024-04-26T00:35:55.133 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:35:55.133 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:35:55.133 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:35:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a7a544640 10 monclient: get_monmap_and_config 2024-04-26T00:35:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a7a544640 10 monclient: build_initial_monmap 2024-04-26T00:35:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a7a544640 10 monclient: monmap: 2024-04-26T00:35:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:35:55.264862+0000 2024-04-26T00:35:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:35:55.264862+0000 2024-04-26T00:35:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:35:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:35:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:35:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:35:55.260 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:35:55.260 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:55.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a7a544640 1 Processor -- start 2024-04-26T00:35:55.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a7a544640 1 -- start start 2024-04-26T00:35:55.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a7a544640 10 monclient: init 2024-04-26T00:35:55.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a7a544640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:55.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a7a544640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-04-26T00:35:55.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a7a544640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a740a0300 0x7f8a7409fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:55.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a7a544640 10 monclient(hunting): picked mon.noname-b con 0x7f8a740a0300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:55.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a7a544640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8a740a06e0 0x7f8a74134ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:55.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a7a544640 10 monclient(hunting): picked mon.noname-c con 0x7f8a740a06e0 addr v2:172.21.15.45:3300/0 2024-04-26T00:35:55.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a7a544640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f8a741350e0 0x7f8a74135570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a7a544640 10 monclient(hunting): picked mon.noname-a con 0x7f8a741350e0 addr v2:172.21.15.37:3300/0 2024-04-26T00:35:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a7a544640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a7a544640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f8a740a13f0 con 0x7f8a741350e0 2024-04-26T00:35:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a7a544640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a7a544640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f8a7409c190 con 0x7f8a740a06e0 2024-04-26T00:35:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a7a544640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a7a544640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8a740a1110 con 0x7f8a740a0300 2024-04-26T00:35:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a7a544640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a7a544640 10 monclient(hunting): authenticate will time out at 1936.570581s 2024-04-26T00:35:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a78ae1640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f8a741350e0 0x7f8a74135570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a73fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a740a0300 0x7f8a7409fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a78ae1640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f8a741350e0 0x7f8a74135570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:60770/0 (socket says 172.21.15.37:60770) 2024-04-26T00:35:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a78ae1640 1 -- 172.21.15.37:0/4125927223 learned_addr learned my addr 172.21.15.37:0/4125927223 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a78ae1640 10 monclient(hunting): get_auth_request con 0x7f8a741350e0 auth_method 0 2024-04-26T00:35:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a78ae1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a78ae1640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a78ae1640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a73fff640 10 monclient(hunting): get_auth_request con 0x7f8a740a0300 auth_method 0 2024-04-26T00:35:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a73fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a73fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a78ae1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a78ae1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a78ae1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a73fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a73fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a73fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a737fe640 1 --2- 172.21.15.37:0/4125927223 >> v2:172.21.15.45:3300/0 conn(0x7f8a740a06e0 0x7f8a74134ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a737fe640 10 monclient(hunting): get_auth_request con 0x7f8a740a06e0 auth_method 0 2024-04-26T00:35:55.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:55.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a737fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:55.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a737fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:55.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a78ae1640 10 monclient(hunting): handle_auth_done global_id 144591 payload 274 2024-04-26T00:35:55.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a78ae1640 1 -- 172.21.15.37:0/4125927223 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a740a0300 msgr2=0x7f8a7409fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:55.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a78ae1640 1 --2- 172.21.15.37:0/4125927223 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a740a0300 0x7f8a7409fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:55.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a78ae1640 1 -- 172.21.15.37:0/4125927223 >> v2:172.21.15.45:3300/0 conn(0x7f8a740a06e0 msgr2=0x7f8a74134ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:55.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a78ae1640 1 --2- 172.21.15.37:0/4125927223 >> v2:172.21.15.45:3300/0 conn(0x7f8a740a06e0 0x7f8a74134ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:55.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a78ae1640 10 monclient: _finish_hunting 0 2024-04-26T00:35:55.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a78ae1640 1 monclient: found mon.noname-a 2024-04-26T00:35:55.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a78ae1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:55.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a78ae1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:35:55.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.261+0000 7f8a78ae1640 1 -- 172.21.15.37:0/4125927223 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a740a16d0 con 0x7f8a741350e0 2024-04-26T00:35:55.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.265+0000 7f8a78ae1640 1 -- 172.21.15.37:0/4125927223 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a741350e0 msgr2=0x7f8a74135570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:55.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.265+0000 7f8a78ae1640 1 --2- 172.21.15.37:0/4125927223 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a741350e0 0x7f8a74135570 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8a7406d330 tx=0x7f8a6c01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e66484e00932e28 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:55.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.265+0000 7f8a72ffd640 1 -- 172.21.15.37:0/4125927223 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c0b0070 con 0x7f8a741350e0 2024-04-26T00:35:55.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.265+0000 7f8a72ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:55.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.265+0000 7f8a72ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:35:55.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.265+0000 7f8a72ffd640 10 monclient: dump: 2024-04-26T00:35:55.269 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:55.270 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.265+0000 7f8a72ffd640 10 monclient: _finish_auth 0 2024-04-26T00:35:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.265+0000 7f8a72ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:35:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.265+0000 7f8a72ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.265+0000 7f8a72ffd640 1 -- 172.21.15.37:0/4125927223 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a6c0ab600 con 0x7f8a741350e0 2024-04-26T00:35:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.265+0000 7f8a7a544640 5 monclient: authenticate success, global_id 144591 2024-04-26T00:35:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.265+0000 7f8a7a544640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:35:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.265+0000 7f8a72ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.265+0000 7f8a7a544640 10 monclient: get_monmap_and_config success 2024-04-26T00:35:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.265+0000 7f8a72ffd640 1 -- 172.21.15.37:0/4125927223 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c0ab9d0 con 0x7f8a741350e0 2024-04-26T00:35:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.265+0000 7f8a7a544640 4 set_mon_vals no callback set 2024-04-26T00:35:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.265+0000 7f8a72ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.265+0000 7f8a72ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.265+0000 7f8a72ffd640 10 monclient: dump: 2024-04-26T00:35:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:55.272 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:55.272 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:55.272 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:55.272 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:55.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a7a544640 10 monclient: shutdown 2024-04-26T00:35:55.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a7a544640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:55.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a7a544640 1 -- 172.21.15.37:0/4125927223 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a741350e0 msgr2=0x7f8a74135570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:55.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a7a544640 1 --2- 172.21.15.37:0/4125927223 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a741350e0 0x7f8a74135570 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8a7406d330 tx=0x7f8a6c01c3e0 comp rx=0 tx=0).stop 2024-04-26T00:35:55.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a7a544640 1 -- 172.21.15.37:0/4125927223 shutdown_connections 2024-04-26T00:35:55.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a7a544640 1 --2- 172.21.15.37:0/4125927223 >> v2:172.21.15.45:3300/0 conn(0x7f8a740a06e0 0x7f8a74134ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:55.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a7a544640 1 --2- 172.21.15.37:0/4125927223 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a741350e0 0x7f8a74135570 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:55.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a7a544640 1 --2- 172.21.15.37:0/4125927223 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a740a0300 0x7f8a7409fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:55.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a7a544640 1 -- 172.21.15.37:0/4125927223 >> 172.21.15.37:0/4125927223 conn(0x7f8a740a1d50 msgr2=0x7f8a740a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:55.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a7a544640 1 -- 172.21.15.37:0/4125927223 shutdown_connections 2024-04-26T00:35:55.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a7a544640 1 -- 172.21.15.37:0/4125927223 wait complete. 2024-04-26T00:35:55.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a7a544640 10 monclient: build_initial_monmap 2024-04-26T00:35:55.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a7a544640 10 monclient: monmap: 2024-04-26T00:35:55.273 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:55.273 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:35:55.273 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:35:55.273 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:35:55.273 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:55.273 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:35:55.273 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:35:55.273 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:35:55.273 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:35:55.273 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:55.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a7a544640 1 Processor -- start 2024-04-26T00:35:55.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a7a544640 1 -- start start 2024-04-26T00:35:55.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a7a544640 10 monclient: init 2024-04-26T00:35:55.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a7a544640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:55.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a7a544640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:35:55.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a7a544640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a740a0300 0x7f8a741cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:55.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a7a544640 10 monclient(hunting): picked mon.noname-b con 0x7f8a740a0300 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:35:55.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a7a544640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a740a06e0 0x7f8a741cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:55.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a7a544640 10 monclient(hunting): picked mon.noname-c con 0x7f8a740a06e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:55.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a7a544640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a741350e0 0x7f8a74138960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:55.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a7a544640 10 monclient(hunting): picked mon.noname-a con 0x7f8a741350e0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:35:55.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a7a544640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:55.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a7a544640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7413e6f0 con 0x7f8a741350e0 2024-04-26T00:35:55.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a7a544640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:55.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a7a544640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7413e870 con 0x7f8a740a0300 2024-04-26T00:35:55.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a7a544640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:55.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a7a544640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8a7413e9f0 con 0x7f8a740a06e0 2024-04-26T00:35:55.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a7a544640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:55.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a7a544640 10 monclient(hunting): authenticate will time out at 1936.577744s 2024-04-26T00:35:55.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a737fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a740a06e0 0x7f8a741cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:55.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a737fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a740a06e0 0x7f8a741cd4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:50838/0 (socket says 172.21.15.37:50838) 2024-04-26T00:35:55.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a737fe640 1 -- 172.21.15.37:0/1285244385 learned_addr learned my addr 172.21.15.37:0/1285244385 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:55.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a737fe640 10 monclient(hunting): get_auth_request con 0x7f8a740a06e0 auth_method 0 2024-04-26T00:35:55.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:55.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a737fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:55.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a737fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:55.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a78ae1640 1 --2- 172.21.15.37:0/1285244385 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a741350e0 0x7f8a74138960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:55.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:55.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:55.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:55.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a78ae1640 10 monclient(hunting): get_auth_request con 0x7f8a741350e0 auth_method 0 2024-04-26T00:35:55.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a78ae1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:55.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a78ae1640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:55.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a78ae1640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:55.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a73fff640 1 --2- 172.21.15.37:0/1285244385 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a740a0300 0x7f8a741cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:55.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a78ae1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:55.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a78ae1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:55.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a78ae1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:55.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a73fff640 10 monclient(hunting): get_auth_request con 0x7f8a740a0300 auth_method 0 2024-04-26T00:35:55.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:55.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a73fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:55.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a73fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:55.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a737fe640 10 monclient(hunting): handle_auth_done global_id 184184 payload 824 2024-04-26T00:35:55.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a737fe640 1 -- 172.21.15.37:0/1285244385 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a740a0300 msgr2=0x7f8a741cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:55.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a737fe640 1 --2- 172.21.15.37:0/1285244385 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a740a0300 0x7f8a741cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:55.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a737fe640 1 -- 172.21.15.37:0/1285244385 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a741350e0 msgr2=0x7f8a74138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:55.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a737fe640 1 --2- 172.21.15.37:0/1285244385 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a741350e0 0x7f8a74138960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:55.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a737fe640 10 monclient: _finish_hunting 0 2024-04-26T00:35:55.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a737fe640 1 monclient: found mon.noname-c 2024-04-26T00:35:55.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:55.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a737fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:35:55.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.269+0000 7f8a737fe640 1 -- 172.21.15.37:0/1285244385 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a7409f550 con 0x7f8a740a06e0 2024-04-26T00:35:55.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a737fe640 1 -- 172.21.15.37:0/1285244385 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a740a06e0 msgr2=0x7f8a741cd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:55.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a737fe640 1 --2- 172.21.15.37:0/1285244385 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a740a06e0 0x7f8a741cd4a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c007ef0 tx=0x7f8a5c00e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:55.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a70ff9640 1 -- 172.21.15.37:0/1285244385 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5c013790 con 0x7f8a740a06e0 2024-04-26T00:35:55.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a70ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:55.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a70ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:35:55.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a70ff9640 10 monclient: dump: 2024-04-26T00:35:55.281 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:55.281 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:55.281 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:55.281 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:55.282 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:55.282 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:55.282 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:55.282 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:55.282 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:55.282 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:55.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a70ff9640 10 monclient: _finish_auth 0 2024-04-26T00:35:55.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a70ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:35:55.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a70ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:55.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a70ff9640 1 -- 172.21.15.37:0/1285244385 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a5c019070 con 0x7f8a740a06e0 2024-04-26T00:35:55.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a7a544640 5 monclient: authenticate success, global_id 184184 2024-04-26T00:35:55.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a7a544640 10 monclient: _renew_subs 2024-04-26T00:35:55.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a7a544640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:35:55.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a7a544640 1 -- 172.21.15.37:0/1285244385 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a740715f0 con 0x7f8a740a06e0 2024-04-26T00:35:55.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a70ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:55.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a70ff9640 1 -- 172.21.15.37:0/1285244385 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5c013b70 con 0x7f8a740a06e0 2024-04-26T00:35:55.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a70ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:55.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a70ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:35:55.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a70ff9640 10 monclient: dump: 2024-04-26T00:35:55.283 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:55.283 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:55.283 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:55.283 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:55.283 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:55.283 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:55.283 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:55.283 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:55.283 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:55.283 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:55.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a7a544640 10 monclient: _renew_subs 2024-04-26T00:35:55.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a7a544640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:35:55.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a727fc640 4 set_mon_vals no callback set 2024-04-26T00:35:55.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a7a544640 1 -- 172.21.15.37:0/1285244385 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a74130fa0 con 0x7f8a740a06e0 2024-04-26T00:35:55.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a7a544640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:55.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a7a544640 1 --2- 172.21.15.37:0/1285244385 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a741cd9e0 0x7f8a74138ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:55.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a7a544640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:55.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a7a544640 1 -- 172.21.15.37:0/1285244385 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f8a740a0cd0 con 0x7f8a741cd9e0 2024-04-26T00:35:55.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a70ff9640 1 -- 172.21.15.37:0/1285244385 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f8a5c013d10 con 0x7f8a740a06e0 2024-04-26T00:35:55.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a73fff640 1 --2- 172.21.15.37:0/1285244385 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a741cd9e0 0x7f8a74138ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:55.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a73fff640 10 monclient: get_auth_request con 0x7f8a741cd9e0 auth_method 0 2024-04-26T00:35:55.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:55.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a73fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:55.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a73fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:55.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a73fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:55.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a73fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:55.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a73fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:55.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a70ff9640 1 --2- 172.21.15.37:0/1285244385 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8a54035ed0 0x7f8a54038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:55.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a73fff640 10 monclient(hunting): handle_auth_done global_id 144597 payload 824 2024-04-26T00:35:55.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a70ff9640 1 -- 172.21.15.37:0/1285244385 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(625..625 src has 370..625) ==== 3927+0+0 (secure 0 0 0) 0x7f8a5c0492c0 con 0x7f8a740a06e0 2024-04-26T00:35:55.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a73fff640 1 -- 172.21.15.37:0/1285244385 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8a7411c440 con 0x7f8a741cd9e0 2024-04-26T00:35:55.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a78ae1640 1 --2- 172.21.15.37:0/1285244385 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8a54035ed0 0x7f8a54038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:55.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.273+0000 7f8a78ae1640 10 monclient: get_auth_request con 0x7f8a54035ed0 auth_method 0 2024-04-26T00:35:55.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.277+0000 7f8a73fff640 1 -- 172.21.15.37:0/1285244385 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a741cd9e0 msgr2=0x7f8a74138ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:55.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.277+0000 7f8a73fff640 1 --2- 172.21.15.37:0/1285244385 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a741cd9e0 0x7f8a74138ea0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8a64011600 tx=0x7f8a64011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:55.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.277+0000 7f8a70ff9640 1 -- 172.21.15.37:0/1285244385 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a6400fc30 con 0x7f8a741cd9e0 2024-04-26T00:35:55.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.277+0000 7f8a70ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:55.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.277+0000 7f8a70ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:55.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.277+0000 7f8a70ff9640 10 monclient: dump: 2024-04-26T00:35:55.285 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:55.286 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:55.286 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:55.286 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:55.286 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:55.286 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:55.286 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:55.286 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:55.286 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:55.286 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:55.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.277+0000 7f8a70ff9640 1 -- 172.21.15.37:0/1285244385 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8a6400fc30 con 0x7f8a741cd9e0 2024-04-26T00:35:55.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.277+0000 7f8a70ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:55.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.277+0000 7f8a70ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:35:55.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.277+0000 7f8a70ff9640 1 -- 172.21.15.37:0/1285244385 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a741cd9e0 msgr2=0x7f8a74138ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:55.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.277+0000 7f8a70ff9640 1 --2- 172.21.15.37:0/1285244385 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a741cd9e0 0x7f8a74138ea0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8a64011600 tx=0x7f8a64011ac0 comp rx=0 tx=0).stop 2024-04-26T00:35:55.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.277+0000 7f8a70ff9640 1 -- 172.21.15.37:0/1285244385 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a741cd9e0 msgr2=0x7f8a74138ea0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:35:55.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.277+0000 7f8a70ff9640 1 --2- 172.21.15.37:0/1285244385 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a741cd9e0 0x7f8a74138ea0 secure :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8a64011600 tx=0x7f8a64011ac0 comp rx=0 tx=0).stop 2024-04-26T00:35:55.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.277+0000 7f8a78ae1640 1 --2- 172.21.15.37:0/1285244385 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8a54035ed0 0x7f8a54038390 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f8a740a2da0 tx=0x7f8a6c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:55.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.281+0000 7f8a7a544640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:35:55.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.281+0000 7f8a7a544640 1 --2- 172.21.15.37:0/1285244385 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a741393e0 0x7f8a741397f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:55.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.281+0000 7f8a7a544640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:55.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.281+0000 7f8a7a544640 1 -- 172.21.15.37:0/1285244385 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7404f510 con 0x7f8a741393e0 2024-04-26T00:35:55.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.281+0000 7f8a73fff640 1 --2- 172.21.15.37:0/1285244385 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a741393e0 0x7f8a741397f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:55.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.281+0000 7f8a73fff640 10 monclient: get_auth_request con 0x7f8a741393e0 auth_method 0 2024-04-26T00:35:55.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.281+0000 7f8a73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:55.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.281+0000 7f8a73fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:55.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.281+0000 7f8a73fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:55.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.281+0000 7f8a73fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:55.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.281+0000 7f8a73fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:55.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.281+0000 7f8a73fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:55.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.281+0000 7f8a73fff640 10 monclient(hunting): handle_auth_done global_id 144600 payload 824 2024-04-26T00:35:55.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.281+0000 7f8a73fff640 1 -- 172.21.15.37:0/1285244385 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8a7409eb60 con 0x7f8a741393e0 2024-04-26T00:35:55.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.285+0000 7f8a73fff640 1 -- 172.21.15.37:0/1285244385 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a741393e0 msgr2=0x7f8a741397f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:55.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.285+0000 7f8a73fff640 1 --2- 172.21.15.37:0/1285244385 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a741393e0 0x7f8a741397f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8a64011600 tx=0x7f8a640046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:55.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.285+0000 7f8a70ff9640 1 -- 172.21.15.37:0/1285244385 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a64003830 con 0x7f8a741393e0 2024-04-26T00:35:55.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.285+0000 7f8a70ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:55.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.285+0000 7f8a70ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:55.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.285+0000 7f8a70ff9640 10 monclient: dump: 2024-04-26T00:35:55.289 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:55.289 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:55.289 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:55.289 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:55.289 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:55.289 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:55.290 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:55.290 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:55.290 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:55.290 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:55.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.285+0000 7f8a70ff9640 1 -- 172.21.15.37:0/1285244385 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4890 (secure 0 0 0) 0x7f8a64003830 con 0x7f8a741393e0 2024-04-26T00:35:55.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.285+0000 7f8a70ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:35:55.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.285+0000 7f8a70ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:35:55.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.285+0000 7f8a70ff9640 1 -- 172.21.15.37:0/1285244385 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a741393e0 msgr2=0x7f8a741397f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:55.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.285+0000 7f8a70ff9640 1 --2- 172.21.15.37:0/1285244385 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a741393e0 0x7f8a741397f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8a64011600 tx=0x7f8a640046d0 comp rx=0 tx=0).stop 2024-04-26T00:35:55.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.285+0000 7f8a70ff9640 1 -- 172.21.15.37:0/1285244385 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a741393e0 msgr2=0x7f8a741397f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:35:55.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.285+0000 7f8a70ff9640 1 --2- 172.21.15.37:0/1285244385 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a741393e0 0x7f8a741397f0 secure :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8a64011600 tx=0x7f8a640046d0 comp rx=0 tx=0).stop 2024-04-26T00:35:55.290 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:35:55.291 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:35:55.291 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:35:55.291 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:35:55.291 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 120, 2024-04-26T00:35:55.291 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 241828, 2024-04-26T00:35:55.291 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:35:55.291 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:35:55.291 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:35:55.291 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:35:55.291 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:55.291 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 1, 2024-04-26T00:35:55.291 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:35:55.291 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:35:55.292 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:35:55.292 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:55.292 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:55.292 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:55.292 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:55.292 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:55.292 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:55.292 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:55.292 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:55.292 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:55.292 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:55.292 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:55.292 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:55.292 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:35:55.293 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:35:55.293 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:55.293 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:55.293 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:55.293 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:55.293 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:55.293 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:55.293 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:55.293 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:55.293 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:55.293 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:55.293 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:55.293 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:55.293 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:55.294 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:35:55.294 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:35:55.294 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:35:55.294 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:35:55.294 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:35:55.294 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:35:55.294 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:35:55.294 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:35:55.294 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:35:55.294 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:35:55.294 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:35:55.294 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:35:55.294 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:35:55.294 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:35:55.294 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:35:55.295 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:35:55.295 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:35:55.295 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:55.295 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:55.295 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:55.295 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:55.295 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:55.295 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:55.295 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:55.295 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:55.295 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:55.295 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:55.295 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:55.295 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:55.296 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:35:55.296 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:55.296 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:35:55.296 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:55.296 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:35:55.296 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:35:55.296 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:55.296 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:55.296 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:55.296 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:55.296 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:35:55.296 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:55.296 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:55.296 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:55.296 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:55.297 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:35:55.297 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:55.297 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:55.297 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:55.297 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:55.297 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:35:55.297 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:35:55.297 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:55.297 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:55.297 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:55.297 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:55.297 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:55.297 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:35:55.297 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:35:55.297 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:55.298 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:55.298 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:55.298 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:55.298 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:35:55.298 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:55.298 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:55.298 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:55.298 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:55.298 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:35:55.298 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:55.298 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:55.298 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:55.298 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:55.298 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:35:55.299 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:35:55.299 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:55.299 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:55.299 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:55.299 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:55.299 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:55.299 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:35:55.299 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:35:55.299 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:55.299 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:55.299 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:55.299 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:55.299 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:35:55.299 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:55.299 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:55.300 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:55.301 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:55.301 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:35:55.301 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:55.301 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:55.301 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:55.301 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:55.301 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:35:55.302 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:35:55.302 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:55.302 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:55.302 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:55.302 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:55.302 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:55.302 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:55.302 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:35:55.302 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:35:55.302 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:55.302 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:55.302 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:55.302 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:35:55.302 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:55.302 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:55.303 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:35:55.303 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:55.303 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:55.303 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:55.303 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:35:55.303 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:55.303 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:55.303 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:35:55.303 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:55.303 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:55.303 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:55.303 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:35:55.303 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:55.303 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:55.303 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:35:55.304 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:55.304 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:55.304 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:55.304 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:35:55.304 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:55.304 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:55.304 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:55.304 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:35:55.304 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:35:55.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.285+0000 7f8a7a544640 1 -- 172.21.15.37:0/1285244385 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8a54035ed0 msgr2=0x7f8a54038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:55.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.285+0000 7f8a7a544640 1 --2- 172.21.15.37:0/1285244385 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8a54035ed0 0x7f8a54038390 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f8a740a2da0 tx=0x7f8a6c008040 comp rx=0 tx=0).stop 2024-04-26T00:35:55.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.285+0000 7f8a7a544640 10 monclient: shutdown 2024-04-26T00:35:55.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.285+0000 7f8a7a544640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:55.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.285+0000 7f8a7a544640 1 -- 172.21.15.37:0/1285244385 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a740a06e0 msgr2=0x7f8a741cd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:55.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.285+0000 7f8a7a544640 1 --2- 172.21.15.37:0/1285244385 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a740a06e0 0x7f8a741cd4a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c007ef0 tx=0x7f8a5c00e4b0 comp rx=0 tx=0).stop 2024-04-26T00:35:55.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.285+0000 7f8a78ae1640 1 -- 172.21.15.37:0/1285244385 reap_dead start 2024-04-26T00:35:55.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.285+0000 7f8a78ae1640 1 -- 172.21.15.37:0/1285244385 reap_dead start 2024-04-26T00:35:55.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.285+0000 7f8a7a544640 1 -- 172.21.15.37:0/1285244385 shutdown_connections 2024-04-26T00:35:55.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.285+0000 7f8a7a544640 1 -- 172.21.15.37:0/1285244385 >> 172.21.15.37:0/1285244385 conn(0x7f8a740a1d50 msgr2=0x7f8a74131a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:55.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.285+0000 7f8a7a544640 1 -- 172.21.15.37:0/1285244385 shutdown_connections 2024-04-26T00:35:55.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.285+0000 7f8a7a544640 1 -- 172.21.15.37:0/1285244385 wait complete. 2024-04-26T00:35:55.307 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:35:55.307 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:35:55.307 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:35:55.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.429+0000 7ff6a6731640 10 monclient: get_monmap_and_config 2024-04-26T00:35:55.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.429+0000 7ff6a6731640 10 monclient: build_initial_monmap 2024-04-26T00:35:55.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.429+0000 7ff6a6731640 10 monclient: monmap: 2024-04-26T00:35:55.426 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:55.426 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:55.426 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:35:55.432161+0000 2024-04-26T00:35:55.426 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:35:55.432161+0000 2024-04-26T00:35:55.426 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:55.427 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:55.427 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:35:55.427 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:35:55.427 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:35:55.427 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:35:55.427 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:35:55.427 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:55.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.429+0000 7ff6a6731640 1 Processor -- start 2024-04-26T00:35:55.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.429+0000 7ff6a6731640 1 -- start start 2024-04-26T00:35:55.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.429+0000 7ff6a6731640 10 monclient: init 2024-04-26T00:35:55.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.429+0000 7ff6a6731640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:55.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.429+0000 7ff6a6731640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-04-26T00:35:55.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.429+0000 7ff6a6731640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff6a0136eb0 addr v1:172.21.15.45:6789/0 2024-04-26T00:35:55.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.429+0000 7ff6a6731640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6a01379a0 0x7ff6a00a1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:55.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.429+0000 7ff6a6731640 10 monclient(hunting): picked mon.noname-b con 0x7ff6a01379a0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:55.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.429+0000 7ff6a6731640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff6a013c3e0 addr v1:172.21.15.37:6789/0 2024-04-26T00:35:55.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.429+0000 7ff6a6731640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6a00a3770 con 0x7ff6a013c3e0 2024-04-26T00:35:55.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.429+0000 7ff6a6731640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6a006da90 con 0x7ff6a0136eb0 2024-04-26T00:35:55.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.429+0000 7ff6a6731640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:55.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.429+0000 7ff6a6731640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff6a00a3970 con 0x7ff6a01379a0 2024-04-26T00:35:55.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.429+0000 7ff6a6731640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:55.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.429+0000 7ff6a6731640 10 monclient(hunting): authenticate will time out at 1936.737882s 2024-04-26T00:35:55.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.429+0000 7ff6a4cce640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7ff6a013c3e0 0x7ff6a013e7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:44002/0 (socket says 172.21.15.37:44002) 2024-04-26T00:35:55.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.429+0000 7ff6a4cce640 1 -- 172.21.15.37:0/1472492922 learned_addr learned my addr 172.21.15.37:0/1472492922 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:35:55.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.429+0000 7ff69f7fe640 1 --2- 172.21.15.37:0/1472492922 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6a01379a0 0x7ff6a00a1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:55.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.429+0000 7ff69f7fe640 10 monclient(hunting): get_auth_request con 0x7ff6a01379a0 auth_method 0 2024-04-26T00:35:55.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.429+0000 7ff69f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:55.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.429+0000 7ff69f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:55.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.429+0000 7ff69f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:55.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.429+0000 7ff69f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:55.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.429+0000 7ff69f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:55.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.429+0000 7ff69f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:55.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.429+0000 7ff69f7fe640 10 monclient(hunting): handle_auth_done global_id 184187 payload 274 2024-04-26T00:35:55.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.429+0000 7ff69f7fe640 1 -- 172.21.15.37:0/1472492922 >> v1:172.21.15.45:6789/0 conn(0x7ff6a0136eb0 legacy=0x7ff6a0137290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:55.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.429+0000 7ff69f7fe640 1 -- 172.21.15.37:0/1472492922 >> v1:172.21.15.37:6789/0 conn(0x7ff6a013c3e0 legacy=0x7ff6a013e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:55.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.429+0000 7ff69f7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:35:55.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.429+0000 7ff69f7fe640 1 monclient: found mon.noname-b 2024-04-26T00:35:55.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.429+0000 7ff69f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:55.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.429+0000 7ff69f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:35:55.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.429+0000 7ff69f7fe640 1 -- 172.21.15.37:0/1472492922 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6a00a34f0 con 0x7ff6a01379a0 2024-04-26T00:35:55.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff69f7fe640 1 -- 172.21.15.37:0/1472492922 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6a01379a0 msgr2=0x7ff6a00a1c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:55.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff69f7fe640 1 --2- 172.21.15.37:0/1472492922 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6a01379a0 0x7ff6a00a1c90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff690009dc0 tx=0x7ff69001c7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aac9e8b6b07564a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:55.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff69effd640 1 -- 172.21.15.37:0/1472492922 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6900b2070 con 0x7ff6a01379a0 2024-04-26T00:35:55.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff69effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:55.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff69effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:35:55.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff69effd640 10 monclient: dump: 2024-04-26T00:35:55.434 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:55.434 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:55.434 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:55.435 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:55.435 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:55.435 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:55.435 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:55.435 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:55.435 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:55.435 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:55.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff69effd640 10 monclient: _finish_auth 0 2024-04-26T00:35:55.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff69effd640 10 monclient: _check_auth_tickets 2024-04-26T00:35:55.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff69effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:55.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff69effd640 1 -- 172.21.15.37:0/1472492922 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6900a5840 con 0x7ff6a01379a0 2024-04-26T00:35:55.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a6731640 5 monclient: authenticate success, global_id 184187 2024-04-26T00:35:55.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff69effd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:55.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a6731640 10 monclient: get_monmap_and_config success 2024-04-26T00:35:55.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff69effd640 1 -- 172.21.15.37:0/1472492922 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6900a5c10 con 0x7ff6a01379a0 2024-04-26T00:35:55.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a6731640 4 set_mon_vals no callback set 2024-04-26T00:35:55.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff69effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:55.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff69effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:35:55.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff69effd640 10 monclient: dump: 2024-04-26T00:35:55.436 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:55.436 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:55.436 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:55.436 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:55.436 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:55.436 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:55.436 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:55.436 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:55.436 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:55.436 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:55.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a6731640 10 monclient: shutdown 2024-04-26T00:35:55.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a6731640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:55.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a6731640 1 -- 172.21.15.37:0/1472492922 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6a01379a0 msgr2=0x7ff6a00a1c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:55.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a6731640 1 --2- 172.21.15.37:0/1472492922 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6a01379a0 0x7ff6a00a1c90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff690009dc0 tx=0x7ff69001c7d0 comp rx=0 tx=0).stop 2024-04-26T00:35:55.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a6731640 1 -- 172.21.15.37:0/1472492922 shutdown_connections 2024-04-26T00:35:55.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a6731640 1 --2- 172.21.15.37:0/1472492922 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6a01379a0 0x7ff6a00a1c90 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:55.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a6731640 1 -- 172.21.15.37:0/1472492922 >> 172.21.15.37:0/1472492922 conn(0x7ff6a00a7290 msgr2=0x7ff6a00a76a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:55.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a6731640 1 -- 172.21.15.37:0/1472492922 shutdown_connections 2024-04-26T00:35:55.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a6731640 1 -- 172.21.15.37:0/1472492922 wait complete. 2024-04-26T00:35:55.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a6731640 10 monclient: build_initial_monmap 2024-04-26T00:35:55.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a6731640 10 monclient: monmap: 2024-04-26T00:35:55.445 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:55.445 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:35:55.445 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:35:55.445 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:35:55.445 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:55.445 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:35:55.445 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:35:55.445 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:35:55.446 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:35:55.446 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:55.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a6731640 1 Processor -- start 2024-04-26T00:35:55.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a6731640 1 -- start start 2024-04-26T00:35:55.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a6731640 10 monclient: init 2024-04-26T00:35:55.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a6731640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:55.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a6731640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:35:55.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a6731640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6a0136eb0 0x7ff6a013d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:55.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a6731640 10 monclient(hunting): picked mon.noname-b con 0x7ff6a0136eb0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:35:55.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a6731640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6a01379a0 0x7ff6a013da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:55.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a6731640 10 monclient(hunting): picked mon.noname-c con 0x7ff6a01379a0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:55.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a6731640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff6a013c3e0 0x7ff6a013df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:55.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a6731640 10 monclient(hunting): picked mon.noname-a con 0x7ff6a013c3e0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:35:55.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a6731640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:55.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a6731640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a0142aa0 con 0x7ff6a013c3e0 2024-04-26T00:35:55.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a6731640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:55.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a6731640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a0142920 con 0x7ff6a0136eb0 2024-04-26T00:35:55.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a6731640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:55.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a6731640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff6a0142c20 con 0x7ff6a01379a0 2024-04-26T00:35:55.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a6731640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:55.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a6731640 10 monclient(hunting): authenticate will time out at 1936.746621s 2024-04-26T00:35:55.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff69f7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6a01379a0 0x7ff6a013da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:55.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a4cce640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff6a013c3e0 0x7ff6a013df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:55.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a4cce640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff6a013c3e0 0x7ff6a013df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:60816/0 (socket says 172.21.15.37:60816) 2024-04-26T00:35:55.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a4cce640 1 -- 172.21.15.37:0/2676849720 learned_addr learned my addr 172.21.15.37:0/2676849720 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:55.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff69f7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6a01379a0 0x7ff6a013da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:50854/0 (socket says 172.21.15.37:50854) 2024-04-26T00:35:55.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a4cce640 10 monclient(hunting): get_auth_request con 0x7ff6a013c3e0 auth_method 0 2024-04-26T00:35:55.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a4cce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:55.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a4cce640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:55.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a4cce640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:55.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff69f7fe640 10 monclient(hunting): get_auth_request con 0x7ff6a01379a0 auth_method 0 2024-04-26T00:35:55.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff69f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:55.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff69f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:55.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff69f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:55.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a4cce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:55.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a4cce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:55.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a4cce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:55.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff69f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:55.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff69f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:55.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff69f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:55.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff69ffff640 1 --2- 172.21.15.37:0/2676849720 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6a0136eb0 0x7ff6a013d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:55.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff69ffff640 10 monclient(hunting): get_auth_request con 0x7ff6a0136eb0 auth_method 0 2024-04-26T00:35:55.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff69ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:55.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff69ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:55.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff69ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:55.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a4cce640 10 monclient(hunting): handle_auth_done global_id 144603 payload 824 2024-04-26T00:35:55.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a4cce640 1 -- 172.21.15.37:0/2676849720 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6a01379a0 msgr2=0x7ff6a013da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:55.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a4cce640 1 --2- 172.21.15.37:0/2676849720 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6a01379a0 0x7ff6a013da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:55.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a4cce640 1 -- 172.21.15.37:0/2676849720 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6a0136eb0 msgr2=0x7ff6a013d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:55.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a4cce640 1 --2- 172.21.15.37:0/2676849720 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6a0136eb0 0x7ff6a013d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:55.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a4cce640 10 monclient: _finish_hunting 0 2024-04-26T00:35:55.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a4cce640 1 monclient: found mon.noname-a 2024-04-26T00:35:55.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a4cce640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:55.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a4cce640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:35:55.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.437+0000 7ff6a4cce640 1 -- 172.21.15.37:0/2676849720 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6a0072920 con 0x7ff6a013c3e0 2024-04-26T00:35:55.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff6a4cce640 1 -- 172.21.15.37:0/2676849720 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff6a013c3e0 msgr2=0x7ff6a013df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:55.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff6a4cce640 1 --2- 172.21.15.37:0/2676849720 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff6a013c3e0 0x7ff6a013df90 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff69800c3f0 tx=0x7ff69800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:55.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff69cff9640 1 -- 172.21.15.37:0/2676849720 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff698017070 con 0x7ff6a013c3e0 2024-04-26T00:35:55.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff69cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:55.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff69cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:35:55.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff69cff9640 10 monclient: dump: 2024-04-26T00:35:55.450 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:55.450 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:55.450 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:55.450 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:55.451 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:55.451 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:55.451 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:55.451 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:55.451 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:55.451 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:55.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff69cff9640 10 monclient: _finish_auth 0 2024-04-26T00:35:55.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff69cff9640 10 monclient: _check_auth_tickets 2024-04-26T00:35:55.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff69cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:55.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff69cff9640 1 -- 172.21.15.37:0/2676849720 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6980070d0 con 0x7ff6a013c3e0 2024-04-26T00:35:55.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff69cff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:55.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff69cff9640 1 -- 172.21.15.37:0/2676849720 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6980074c0 con 0x7ff6a013c3e0 2024-04-26T00:35:55.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff6a6731640 5 monclient: authenticate success, global_id 144603 2024-04-26T00:35:55.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff6a6731640 10 monclient: _renew_subs 2024-04-26T00:35:55.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff69e7fc640 4 set_mon_vals no callback set 2024-04-26T00:35:55.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff6a6731640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:55.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff6a6731640 1 -- 172.21.15.37:0/2676849720 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6a0071570 con 0x7ff6a013c3e0 2024-04-26T00:35:55.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff69cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:55.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff69cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:55.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff69cff9640 10 monclient: dump: 2024-04-26T00:35:55.453 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:55.453 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:55.453 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:55.453 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:55.453 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:55.453 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:55.453 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:55.453 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:55.453 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:55.453 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:55.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff6a6731640 10 monclient: _renew_subs 2024-04-26T00:35:55.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff6a6731640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:55.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff6a6731640 1 -- 172.21.15.37:0/2676849720 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6a0138300 con 0x7ff6a013c3e0 2024-04-26T00:35:55.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff6a6731640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:55.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff6a6731640 1 --2- 172.21.15.37:0/2676849720 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6a01cedb0 0x7ff6a01cf190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:55.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff6a6731640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:55.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff6a6731640 1 -- 172.21.15.37:0/2676849720 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff6a004f510 con 0x7ff6a01cedb0 2024-04-26T00:35:55.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff69ffff640 1 --2- 172.21.15.37:0/2676849720 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6a01cedb0 0x7ff6a01cf190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:55.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff69cff9640 1 -- 172.21.15.37:0/2676849720 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff69801a070 con 0x7ff6a013c3e0 2024-04-26T00:35:55.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff69ffff640 10 monclient: get_auth_request con 0x7ff6a01cedb0 auth_method 0 2024-04-26T00:35:55.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff69ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:55.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff69ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:55.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff69ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:55.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff69ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:55.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff69ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:55.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff69ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:55.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff69cff9640 1 --2- 172.21.15.37:0/2676849720 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff680035e80 0x7ff680038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:55.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff69cff9640 1 -- 172.21.15.37:0/2676849720 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(625..625 src has 370..625) ==== 3927+0+0 (secure 0 0 0) 0x7ff698047f60 con 0x7ff6a013c3e0 2024-04-26T00:35:55.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff69ffff640 10 monclient(hunting): handle_auth_done global_id 184193 payload 824 2024-04-26T00:35:55.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff69ffff640 1 -- 172.21.15.37:0/2676849720 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff6a0138050 con 0x7ff6a01cedb0 2024-04-26T00:35:55.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff69f7fe640 1 --2- 172.21.15.37:0/2676849720 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff680035e80 0x7ff680038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:55.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.441+0000 7ff69f7fe640 10 monclient: get_auth_request con 0x7ff680035e80 auth_method 0 2024-04-26T00:35:55.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.445+0000 7ff69ffff640 1 -- 172.21.15.37:0/2676849720 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6a01cedb0 msgr2=0x7ff6a01cf190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:55.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.445+0000 7ff69ffff640 1 --2- 172.21.15.37:0/2676849720 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6a01cedb0 0x7ff6a01cf190 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff68800e450 tx=0x7ff68800e910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:55.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.445+0000 7ff69cff9640 1 -- 172.21.15.37:0/2676849720 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff688007450 con 0x7ff6a01cedb0 2024-04-26T00:35:55.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.445+0000 7ff69cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:55.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.445+0000 7ff69cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:35:55.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.445+0000 7ff69cff9640 10 monclient: dump: 2024-04-26T00:35:55.458 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:55.458 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:55.458 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:55.458 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:55.458 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:55.458 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:55.458 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:55.458 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:55.458 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:55.458 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:55.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.445+0000 7ff69cff9640 1 -- 172.21.15.37:0/2676849720 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff688007450 con 0x7ff6a01cedb0 2024-04-26T00:35:55.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.445+0000 7ff69cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:55.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.445+0000 7ff69cff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:35:55.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.445+0000 7ff69cff9640 1 -- 172.21.15.37:0/2676849720 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6a01cedb0 msgr2=0x7ff6a01cf190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:55.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.445+0000 7ff69cff9640 1 --2- 172.21.15.37:0/2676849720 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6a01cedb0 0x7ff6a01cf190 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff68800e450 tx=0x7ff68800e910 comp rx=0 tx=0).stop 2024-04-26T00:35:55.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.445+0000 7ff69cff9640 1 -- 172.21.15.37:0/2676849720 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6a01cedb0 msgr2=0x7ff6a01cf190 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:35:55.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.445+0000 7ff69cff9640 1 --2- 172.21.15.37:0/2676849720 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6a01cedb0 0x7ff6a01cf190 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff68800e450 tx=0x7ff68800e910 comp rx=0 tx=0).stop 2024-04-26T00:35:55.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.445+0000 7ff69f7fe640 1 --2- 172.21.15.37:0/2676849720 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff680035e80 0x7ff680038340 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7ff6a006e4c0 tx=0x7ff6900a7070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:55.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.449+0000 7ff6a6731640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:35:55.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.449+0000 7ff6a6731640 1 --2- 172.21.15.37:0/2676849720 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6a01cf6d0 0x7ff6a009b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:55.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.449+0000 7ff6a6731640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:55.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.449+0000 7ff6a6731640 1 -- 172.21.15.37:0/2676849720 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff6a0067850 con 0x7ff6a01cf6d0 2024-04-26T00:35:55.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.453+0000 7ff69ffff640 1 --2- 172.21.15.37:0/2676849720 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6a01cf6d0 0x7ff6a009b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:55.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.453+0000 7ff69ffff640 10 monclient: get_auth_request con 0x7ff6a01cf6d0 auth_method 0 2024-04-26T00:35:55.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.453+0000 7ff69ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:55.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.453+0000 7ff69ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:55.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.453+0000 7ff69ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:55.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.453+0000 7ff69ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:55.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.453+0000 7ff69ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:55.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.453+0000 7ff69ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:55.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.453+0000 7ff69ffff640 10 monclient(hunting): handle_auth_done global_id 184196 payload 824 2024-04-26T00:35:55.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.453+0000 7ff69ffff640 1 -- 172.21.15.37:0/2676849720 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff6a013b930 con 0x7ff6a01cf6d0 2024-04-26T00:35:55.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.453+0000 7ff69ffff640 1 -- 172.21.15.37:0/2676849720 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6a01cf6d0 msgr2=0x7ff6a009b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:55.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.453+0000 7ff69ffff640 1 --2- 172.21.15.37:0/2676849720 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6a01cf6d0 0x7ff6a009b270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff68800e450 tx=0x7ff688003770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:55.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.453+0000 7ff69cff9640 1 -- 172.21.15.37:0/2676849720 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff68801e960 con 0x7ff6a01cf6d0 2024-04-26T00:35:55.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.453+0000 7ff69cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:55.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.453+0000 7ff69cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:35:55.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.453+0000 7ff69cff9640 10 monclient: dump: 2024-04-26T00:35:55.461 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:55.461 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:55.461 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:55.461 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:55.461 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:55.461 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:55.461 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:55.461 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:55.462 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:55.462 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:55.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.453+0000 7ff69cff9640 1 -- 172.21.15.37:0/2676849720 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4717 (secure 0 0 0) 0x7ff688028460 con 0x7ff6a01cf6d0 2024-04-26T00:35:55.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.453+0000 7ff69cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:35:55.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.453+0000 7ff69cff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:35:55.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.453+0000 7ff69cff9640 1 -- 172.21.15.37:0/2676849720 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6a01cf6d0 msgr2=0x7ff6a009b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:55.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.453+0000 7ff69cff9640 1 --2- 172.21.15.37:0/2676849720 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6a01cf6d0 0x7ff6a009b270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff68800e450 tx=0x7ff688003770 comp rx=0 tx=0).stop 2024-04-26T00:35:55.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.453+0000 7ff69cff9640 1 -- 172.21.15.37:0/2676849720 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6a01cf6d0 msgr2=0x7ff6a009b270 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:35:55.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.453+0000 7ff69cff9640 1 --2- 172.21.15.37:0/2676849720 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6a01cf6d0 0x7ff6a009b270 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff68800e450 tx=0x7ff688003770 comp rx=0 tx=0).stop 2024-04-26T00:35:55.462 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:35:55.463 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:35:55.463 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:35:55.463 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:35:55.463 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 120, 2024-04-26T00:35:55.463 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 54599, 2024-04-26T00:35:55.463 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:35:55.463 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:35:55.463 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:35:55.463 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:35:55.463 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:55.464 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 1, 2024-04-26T00:35:55.464 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:35:55.464 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:35:55.464 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:35:55.464 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:55.464 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:55.464 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:55.464 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:55.464 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:55.464 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:55.464 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:55.464 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:55.464 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:55.465 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:55.465 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:55.465 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:55.465 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:35:55.465 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:35:55.465 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:55.465 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:55.465 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:55.465 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:55.465 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:55.465 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:55.465 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:55.465 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:55.465 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:55.466 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:55.466 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:55.466 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:55.466 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:55.466 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:35:55.466 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:35:55.466 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:35:55.466 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:35:55.466 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:35:55.466 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:35:55.466 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:35:55.466 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:35:55.467 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:35:55.467 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:35:55.467 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:35:55.467 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:35:55.467 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:35:55.467 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:35:55.467 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:35:55.467 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:35:55.467 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:35:55.467 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:55.467 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:55.467 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:55.467 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:55.467 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:55.467 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:55.468 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:55.468 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:55.468 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:55.468 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:55.468 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:55.468 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:55.468 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:35:55.468 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:55.468 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:35:55.469 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:55.469 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:35:55.469 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:35:55.469 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:55.469 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:55.469 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:55.469 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:55.469 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:35:55.469 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:55.469 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:55.469 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:55.469 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:55.469 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:35:55.469 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:55.469 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:55.470 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:55.470 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:55.470 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:35:55.470 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:35:55.470 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:55.470 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:55.470 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:55.470 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:55.471 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:55.471 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:35:55.471 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:35:55.471 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:55.471 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:55.471 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:55.471 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:55.471 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:35:55.471 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:55.471 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:55.471 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:55.472 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:55.472 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:35:55.472 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:55.472 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:55.472 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:55.472 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:55.472 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:35:55.472 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:35:55.472 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:55.472 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:55.472 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:55.472 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:55.472 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:55.472 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:35:55.472 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:35:55.473 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:55.473 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:55.473 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:55.473 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:55.473 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:35:55.473 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:55.473 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:55.473 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:55.473 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:55.473 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:35:55.473 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:55.473 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:55.473 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:55.473 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:55.473 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:35:55.474 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:35:55.474 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:55.474 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:55.474 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:55.474 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:55.474 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:55.474 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:55.474 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:35:55.474 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:35:55.474 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:55.474 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:55.474 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:55.474 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:35:55.475 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:55.475 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:55.475 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:35:55.475 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:55.475 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:55.475 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:55.475 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:35:55.475 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:55.475 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:55.475 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:35:55.475 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:55.475 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:55.475 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:55.475 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:35:55.476 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:55.476 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:55.476 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:55.476 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:35:55.476 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:35:55.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.453+0000 7ff6a6731640 1 -- 172.21.15.37:0/2676849720 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff680035e80 msgr2=0x7ff680038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:55.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.453+0000 7ff6a6731640 1 --2- 172.21.15.37:0/2676849720 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff680035e80 0x7ff680038340 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7ff6a006e4c0 tx=0x7ff6900a7070 comp rx=0 tx=0).stop 2024-04-26T00:35:55.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.453+0000 7ff6a6731640 10 monclient: shutdown 2024-04-26T00:35:55.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.453+0000 7ff6a6731640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:55.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.453+0000 7ff6a6731640 1 -- 172.21.15.37:0/2676849720 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff6a013c3e0 msgr2=0x7ff6a013df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:55.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.453+0000 7ff6a6731640 1 --2- 172.21.15.37:0/2676849720 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff6a013c3e0 0x7ff6a013df90 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff69800c3f0 tx=0x7ff69800c8b0 comp rx=0 tx=0).stop 2024-04-26T00:35:55.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.453+0000 7ff6a4cce640 1 -- 172.21.15.37:0/2676849720 reap_dead start 2024-04-26T00:35:55.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.453+0000 7ff6a4cce640 1 -- 172.21.15.37:0/2676849720 reap_dead start 2024-04-26T00:35:55.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.453+0000 7ff6a6731640 1 -- 172.21.15.37:0/2676849720 shutdown_connections 2024-04-26T00:35:55.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.453+0000 7ff6a6731640 1 -- 172.21.15.37:0/2676849720 >> 172.21.15.37:0/2676849720 conn(0x7ff6a00a7290 msgr2=0x7ff6a00a5cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:55.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.453+0000 7ff6a6731640 1 -- 172.21.15.37:0/2676849720 shutdown_connections 2024-04-26T00:35:55.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.453+0000 7ff6a6731640 1 -- 172.21.15.37:0/2676849720 wait complete. 2024-04-26T00:35:55.478 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:35:55.478 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:35:55.479 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:35:55.570 INFO:tasks.rados.rados.0.smithi045.stdout:2708: done (1 left) 2024-04-26T00:35:55.570 INFO:tasks.rados.rados.0.smithi045.stdout:2709: snap_remove snap 298 2024-04-26T00:35:55.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb9909b640 10 monclient: get_monmap_and_config 2024-04-26T00:35:55.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb9909b640 10 monclient: build_initial_monmap 2024-04-26T00:35:55.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb9909b640 10 monclient: monmap: 2024-04-26T00:35:55.595 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:55.595 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:55.595 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:35:55.601153+0000 2024-04-26T00:35:55.595 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:35:55.601153+0000 2024-04-26T00:35:55.595 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:55.596 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:55.596 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:35:55.596 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:35:55.596 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:35:55.596 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:35:55.596 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:35:55.596 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:55.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb9909b640 1 Processor -- start 2024-04-26T00:35:55.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb9909b640 1 -- start start 2024-04-26T00:35:55.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb9909b640 10 monclient: init 2024-04-26T00:35:55.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb9909b640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:55.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb9909b640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-04-26T00:35:55.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb9909b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb940a3d90 0x7fcb9409bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:55.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb9909b640 10 monclient(hunting): picked mon.noname-b con 0x7fcb940a3d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:55.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb9909b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcb940a4170 addr v1:172.21.15.45:6789/0 2024-04-26T00:35:55.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb9909b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fcb9413c3e0 0x7fcb9413e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:55.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb9909b640 10 monclient(hunting): picked mon.noname-c con 0x7fcb9413c3e0 addr v2:172.21.15.45:3300/0 2024-04-26T00:35:55.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb9909b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb940a5150 con 0x7fcb940a4170 2024-04-26T00:35:55.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb9909b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:55.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb9909b640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fcb9409b280 con 0x7fcb9413c3e0 2024-04-26T00:35:55.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb9909b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:55.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb9909b640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fcb9409b560 con 0x7fcb940a3d90 2024-04-26T00:35:55.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb9909b640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:55.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb9909b640 10 monclient(hunting): authenticate will time out at 1936.907110s 2024-04-26T00:35:55.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb92d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb940a3d90 0x7fcb9409bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:55.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb92d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb940a3d90 0x7fcb9409bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:50882/0 (socket says 172.21.15.37:50882) 2024-04-26T00:35:55.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb92d9d640 1 -- 172.21.15.37:0/2102063545 learned_addr learned my addr 172.21.15.37:0/2102063545 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:55.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb92d9d640 10 monclient(hunting): get_auth_request con 0x7fcb940a3d90 auth_method 0 2024-04-26T00:35:55.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb92d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:55.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb92d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:55.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb92d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:55.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb92d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:55.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb92d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:55.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb92d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:55.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb9359e640 1 --2- 172.21.15.37:0/2102063545 >> v2:172.21.15.45:3300/0 conn(0x7fcb9413c3e0 0x7fcb9413e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:55.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb92d9d640 10 monclient(hunting): handle_auth_done global_id 184199 payload 274 2024-04-26T00:35:55.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb92d9d640 1 -- 172.21.15.37:0/2102063545 >> v2:172.21.15.45:3300/0 conn(0x7fcb9413c3e0 msgr2=0x7fcb9413e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:55.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb92d9d640 1 --2- 172.21.15.37:0/2102063545 >> v2:172.21.15.45:3300/0 conn(0x7fcb9413c3e0 0x7fcb9413e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:55.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb92d9d640 1 -- 172.21.15.37:0/2102063545 >> v1:172.21.15.45:6789/0 conn(0x7fcb940a4170 legacy=0x7fcb940a45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:55.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb92d9d640 10 monclient: _finish_hunting 0 2024-04-26T00:35:55.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb92d9d640 1 monclient: found mon.noname-b 2024-04-26T00:35:55.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb92d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:55.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb92d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:35:55.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb92d9d640 1 -- 172.21.15.37:0/2102063545 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb9406dc60 con 0x7fcb940a3d90 2024-04-26T00:35:55.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb9359e640 10 monclient: get_auth_request con 0x7fcb9413c3e0 auth_method 0 2024-04-26T00:35:55.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.597+0000 7fcb9359e640 1 --2- 172.21.15.37:0/2102063545 >> v2:172.21.15.45:3300/0 conn(0x7fcb9413c3e0 0x7fcb9413e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:35:55.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb92d9d640 1 -- 172.21.15.37:0/2102063545 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb940a3d90 msgr2=0x7fcb9409bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:55.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb92d9d640 1 --2- 172.21.15.37:0/2102063545 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb940a3d90 0x7fcb9409bf40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcb84009870 tx=0x7fcb8401c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=278217dafdd2af6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:55.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb91d9b640 1 -- 172.21.15.37:0/2102063545 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb840b0070 con 0x7fcb940a3d90 2024-04-26T00:35:55.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb91d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:55.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb91d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:35:55.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb91d9b640 10 monclient: dump: 2024-04-26T00:35:55.604 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:55.604 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:55.604 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:55.604 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:55.604 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:55.604 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:55.604 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:55.604 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:55.604 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:55.604 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:55.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb91d9b640 10 monclient: _finish_auth 0 2024-04-26T00:35:55.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb91d9b640 10 monclient: _check_auth_tickets 2024-04-26T00:35:55.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb91d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:55.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb91d9b640 1 -- 172.21.15.37:0/2102063545 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb840ab600 con 0x7fcb940a3d90 2024-04-26T00:35:55.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9909b640 5 monclient: authenticate success, global_id 184199 2024-04-26T00:35:55.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9909b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:35:55.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb91d9b640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:55.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9909b640 10 monclient: get_monmap_and_config success 2024-04-26T00:35:55.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb91d9b640 1 -- 172.21.15.37:0/2102063545 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb840ab9d0 con 0x7fcb940a3d90 2024-04-26T00:35:55.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9909b640 4 set_mon_vals no callback set 2024-04-26T00:35:55.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb91d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:55.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb91d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:35:55.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb91d9b640 10 monclient: dump: 2024-04-26T00:35:55.605 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:55.605 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:55.605 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:55.606 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:55.606 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:55.606 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:55.606 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:55.606 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:55.606 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:55.606 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:55.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9909b640 10 monclient: shutdown 2024-04-26T00:35:55.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9909b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:55.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9909b640 1 -- 172.21.15.37:0/2102063545 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb940a3d90 msgr2=0x7fcb9409bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:55.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9909b640 1 --2- 172.21.15.37:0/2102063545 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb940a3d90 0x7fcb9409bf40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcb84009870 tx=0x7fcb8401c3e0 comp rx=0 tx=0).stop 2024-04-26T00:35:55.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9909b640 1 -- 172.21.15.37:0/2102063545 shutdown_connections 2024-04-26T00:35:55.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9909b640 1 --2- 172.21.15.37:0/2102063545 >> v2:172.21.15.45:3300/0 conn(0x7fcb9413c3e0 0x7fcb9413e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:55.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9909b640 1 --2- 172.21.15.37:0/2102063545 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb940a3d90 0x7fcb9409bf40 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:55.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9909b640 1 -- 172.21.15.37:0/2102063545 >> 172.21.15.37:0/2102063545 conn(0x7fcb940aa0a0 msgr2=0x7fcb940a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:55.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9909b640 1 -- 172.21.15.37:0/2102063545 shutdown_connections 2024-04-26T00:35:55.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9909b640 1 -- 172.21.15.37:0/2102063545 wait complete. 2024-04-26T00:35:55.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9909b640 10 monclient: build_initial_monmap 2024-04-26T00:35:55.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9909b640 10 monclient: monmap: 2024-04-26T00:35:55.607 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:55.607 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:35:55.607 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:35:55.607 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:35:55.607 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:55.607 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:35:55.607 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:35:55.607 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:35:55.607 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:35:55.607 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:55.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9909b640 1 Processor -- start 2024-04-26T00:35:55.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9909b640 1 -- start start 2024-04-26T00:35:55.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9909b640 10 monclient: init 2024-04-26T00:35:55.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9909b640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:55.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9909b640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:35:55.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9909b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb940a3d90 0x7fcb94136ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:55.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9909b640 10 monclient(hunting): picked mon.noname-b con 0x7fcb940a3d90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:35:55.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9909b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb940a4170 0x7fcb94137020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:55.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9909b640 10 monclient(hunting): picked mon.noname-c con 0x7fcb940a4170 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:55.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9909b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcb9413c3e0 0x7fcb9413a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:55.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9909b640 10 monclient(hunting): picked mon.noname-a con 0x7fcb9413c3e0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:35:55.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9909b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:55.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9909b640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fcb94141740 con 0x7fcb9413c3e0 2024-04-26T00:35:55.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9909b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:55.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9909b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcb9409c690 con 0x7fcb940a3d90 2024-04-26T00:35:55.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9909b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:55.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9909b640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fcb941418c0 con 0x7fcb940a4170 2024-04-26T00:35:55.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9909b640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:55.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9909b640 10 monclient(hunting): authenticate will time out at 1936.914489s 2024-04-26T00:35:55.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9259c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb940a4170 0x7fcb94137020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:55.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9359e640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcb9413c3e0 0x7fcb9413a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:55.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9259c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb940a4170 0x7fcb94137020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:50888/0 (socket says 172.21.15.37:50888) 2024-04-26T00:35:55.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9259c640 1 -- 172.21.15.37:0/3110668719 learned_addr learned my addr 172.21.15.37:0/3110668719 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:55.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9259c640 10 monclient(hunting): get_auth_request con 0x7fcb940a4170 auth_method 0 2024-04-26T00:35:55.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9359e640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcb9413c3e0 0x7fcb9413a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:60832/0 (socket says 172.21.15.37:60832) 2024-04-26T00:35:55.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:55.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:55.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:55.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9359e640 10 monclient(hunting): get_auth_request con 0x7fcb9413c3e0 auth_method 0 2024-04-26T00:35:55.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:55.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9359e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:55.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9359e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:55.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:55.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:55.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:55.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:55.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:55.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:55.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb92d9d640 1 --2- 172.21.15.37:0/3110668719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb940a3d90 0x7fcb94136ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:55.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb92d9d640 10 monclient(hunting): get_auth_request con 0x7fcb940a3d90 auth_method 0 2024-04-26T00:35:55.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb92d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:55.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb92d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:55.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb92d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:55.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9259c640 10 monclient(hunting): handle_auth_done global_id 184202 payload 824 2024-04-26T00:35:55.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9259c640 1 -- 172.21.15.37:0/3110668719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb940a3d90 msgr2=0x7fcb94136ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:55.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9259c640 1 --2- 172.21.15.37:0/3110668719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb940a3d90 0x7fcb94136ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:55.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9259c640 1 -- 172.21.15.37:0/3110668719 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcb9413c3e0 msgr2=0x7fcb9413a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:55.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9259c640 1 --2- 172.21.15.37:0/3110668719 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcb9413c3e0 0x7fcb9413a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:55.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9259c640 10 monclient: _finish_hunting 0 2024-04-26T00:35:55.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9259c640 1 monclient: found mon.noname-c 2024-04-26T00:35:55.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:55.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:35:55.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9259c640 1 -- 172.21.15.37:0/3110668719 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb9409ada0 con 0x7fcb940a4170 2024-04-26T00:35:55.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.605+0000 7fcb9359e640 1 --2- 172.21.15.37:0/3110668719 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcb9413c3e0 0x7fcb9413a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:35:55.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb9259c640 1 -- 172.21.15.37:0/3110668719 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb940a4170 msgr2=0x7fcb94137020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:55.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb9259c640 1 --2- 172.21.15.37:0/3110668719 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb940a4170 0x7fcb94137020 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcb8800c480 tx=0x7fcb8800c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:55.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb7b7fe640 1 -- 172.21.15.37:0/3110668719 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb88012a40 con 0x7fcb940a4170 2024-04-26T00:35:55.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb7b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:55.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb7b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:35:55.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb7b7fe640 10 monclient: dump: 2024-04-26T00:35:55.616 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:55.616 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:55.616 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:55.617 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:55.617 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:55.617 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:55.617 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:55.617 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:55.617 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:55.617 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:55.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb7b7fe640 10 monclient: _finish_auth 0 2024-04-26T00:35:55.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb7b7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:35:55.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb7b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:55.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb7b7fe640 1 -- 172.21.15.37:0/3110668719 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb88017070 con 0x7fcb940a4170 2024-04-26T00:35:55.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb7b7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:55.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb7b7fe640 1 -- 172.21.15.37:0/3110668719 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb88012e40 con 0x7fcb940a4170 2024-04-26T00:35:55.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb9909b640 5 monclient: authenticate success, global_id 184202 2024-04-26T00:35:55.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb9909b640 10 monclient: _renew_subs 2024-04-26T00:35:55.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb9909b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:35:55.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb9909b640 1 -- 172.21.15.37:0/3110668719 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb94121910 con 0x7fcb940a4170 2024-04-26T00:35:55.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb7b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:55.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb7b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:35:55.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb9159a640 4 set_mon_vals no callback set 2024-04-26T00:35:55.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb7b7fe640 10 monclient: dump: 2024-04-26T00:35:55.618 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:55.618 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:55.618 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:55.618 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:55.618 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:55.618 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:55.618 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:55.618 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:55.619 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:55.619 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:55.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb9909b640 10 monclient: _renew_subs 2024-04-26T00:35:55.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb9909b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:35:55.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb9909b640 1 -- 172.21.15.37:0/3110668719 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb94121630 con 0x7fcb940a4170 2024-04-26T00:35:55.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb9909b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:55.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb9909b640 1 --2- 172.21.15.37:0/3110668719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9413ad10 0x7fcb9402c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:55.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb9909b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:55.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb9909b640 1 -- 172.21.15.37:0/3110668719 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcb9404f510 con 0x7fcb9413ad10 2024-04-26T00:35:55.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb7b7fe640 1 -- 172.21.15.37:0/3110668719 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fcb8801c460 con 0x7fcb940a4170 2024-04-26T00:35:55.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb92d9d640 1 --2- 172.21.15.37:0/3110668719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9413ad10 0x7fcb9402c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:55.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb92d9d640 10 monclient: get_auth_request con 0x7fcb9413ad10 auth_method 0 2024-04-26T00:35:55.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb7b7fe640 1 --2- 172.21.15.37:0/3110668719 >> v2:172.21.15.45:6812/2379918623 conn(0x7fcb80035ed0 0x7fcb80038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:55.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb92d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:55.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb92d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:55.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb92d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:55.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb7b7fe640 1 -- 172.21.15.37:0/3110668719 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(626..626 src has 370..626) ==== 3943+0+0 (secure 0 0 0) 0x7fcb880488d0 con 0x7fcb940a4170 2024-04-26T00:35:55.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb92d9d640 1 -- 172.21.15.37:0/3110668719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9413ad10 msgr2=0x7fcb9402c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:35:55.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb92d9d640 1 -- 172.21.15.37:0/3110668719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9413ad10 msgr2=0x7fcb9402c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:35:55.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb92d9d640 1 --2- 172.21.15.37:0/3110668719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9413ad10 0x7fcb9402c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:35:55.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb92d9d640 1 --2- 172.21.15.37:0/3110668719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9413ad10 0x7fcb9402c000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:35:55.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb9359e640 1 --2- 172.21.15.37:0/3110668719 >> v2:172.21.15.45:6812/2379918623 conn(0x7fcb80035ed0 0x7fcb80038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:55.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.609+0000 7fcb9359e640 10 monclient: get_auth_request con 0x7fcb80035ed0 auth_method 0 2024-04-26T00:35:55.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.613+0000 7fcb9359e640 1 --2- 172.21.15.37:0/3110668719 >> v2:172.21.15.45:6812/2379918623 conn(0x7fcb80035ed0 0x7fcb80038390 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fcb94120350 tx=0x7fcb8c007780 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:55.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.813+0000 7fcb92d9d640 1 --2- 172.21.15.37:0/3110668719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9413ad10 0x7fcb9402c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:55.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.813+0000 7fcb92d9d640 10 monclient: get_auth_request con 0x7fcb9413ad10 auth_method 0 2024-04-26T00:35:55.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.813+0000 7fcb92d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:55.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.813+0000 7fcb92d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:55.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.813+0000 7fcb92d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:35:55.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.813+0000 7fcb92d9d640 1 -- 172.21.15.37:0/3110668719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9413ad10 msgr2=0x7fcb9402c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:35:55.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.813+0000 7fcb92d9d640 1 -- 172.21.15.37:0/3110668719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9413ad10 msgr2=0x7fcb9402c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:35:55.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.813+0000 7fcb92d9d640 1 --2- 172.21.15.37:0/3110668719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9413ad10 0x7fcb9402c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:35:55.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:55.813+0000 7fcb92d9d640 1 --2- 172.21.15.37:0/3110668719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9413ad10 0x7fcb9402c000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:35:56.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:56.217+0000 7fcb92d9d640 1 --2- 172.21.15.37:0/3110668719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9413ad10 0x7fcb9402c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:56.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:56.217+0000 7fcb92d9d640 10 monclient: get_auth_request con 0x7fcb9413ad10 auth_method 0 2024-04-26T00:35:56.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:56.217+0000 7fcb92d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:56.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:56.217+0000 7fcb92d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:56.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:56.217+0000 7fcb92d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:35:56.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:56.217+0000 7fcb92d9d640 1 -- 172.21.15.37:0/3110668719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9413ad10 msgr2=0x7fcb9402c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:35:56.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:56.217+0000 7fcb92d9d640 1 -- 172.21.15.37:0/3110668719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9413ad10 msgr2=0x7fcb9402c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:35:56.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:56.217+0000 7fcb92d9d640 1 --2- 172.21.15.37:0/3110668719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9413ad10 0x7fcb9402c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:35:56.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:56.217+0000 7fcb92d9d640 1 --2- 172.21.15.37:0/3110668719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9413ad10 0x7fcb9402c000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:35:56.575 INFO:tasks.rados.rados.0.smithi045.stdout:2709: done (1 left) 2024-04-26T00:35:56.575 INFO:tasks.rados.rados.0.smithi045.stdout:2710: write oid 25 current snap is 301 2024-04-26T00:35:56.575 INFO:tasks.rados.rados.0.smithi045.stdout:2710: seq_num 976 ranges {403632=736208,1763249=619999} 2024-04-26T00:35:56.594 INFO:tasks.rados.rados.0.smithi045.stdout:2710: writing smithi04526579-25 from 403632 to 1139840 tid 1 2024-04-26T00:35:56.614 INFO:tasks.rados.rados.0.smithi045.stdout:2710: writing smithi04526579-25 from 1763249 to 2383248 tid 2 2024-04-26T00:35:56.614 INFO:tasks.rados.rados.0.smithi045.stdout:2711: snap_remove snap 290 2024-04-26T00:35:57.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.021+0000 7fcb92d9d640 1 --2- 172.21.15.37:0/3110668719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9413ad10 0x7fcb9402c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:57.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.021+0000 7fcb92d9d640 10 monclient: get_auth_request con 0x7fcb9413ad10 auth_method 0 2024-04-26T00:35:57.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.021+0000 7fcb92d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:57.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.021+0000 7fcb92d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:57.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.021+0000 7fcb92d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:35:57.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.021+0000 7fcb92d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:57.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.021+0000 7fcb92d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:57.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.021+0000 7fcb92d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:57.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.021+0000 7fcb92d9d640 10 monclient(hunting): handle_auth_done global_id 194098 payload 824 2024-04-26T00:35:57.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.021+0000 7fcb92d9d640 1 -- 172.21.15.37:0/3110668719 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcb940a1ea0 con 0x7fcb9413ad10 2024-04-26T00:35:57.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.025+0000 7fcb92d9d640 1 -- 172.21.15.37:0/3110668719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9413ad10 msgr2=0x7fcb9402c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:57.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.025+0000 7fcb92d9d640 1 --2- 172.21.15.37:0/3110668719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9413ad10 0x7fcb9402c000 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fcb840ad000 tx=0x7fcb8401ac20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:57.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.025+0000 7fcb7b7fe640 1 -- 172.21.15.37:0/3110668719 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb840b0040 con 0x7fcb9413ad10 2024-04-26T00:35:57.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.025+0000 7fcb7b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:57.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.025+0000 7fcb7b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:35:57.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.025+0000 7fcb7b7fe640 10 monclient: dump: 2024-04-26T00:35:57.023 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:57.023 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:57.023 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:57.023 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:57.023 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:57.023 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:57.023 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:57.023 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:57.023 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:57.023 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:57.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.025+0000 7fcb7b7fe640 1 -- 172.21.15.37:0/3110668719 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcb84009490 con 0x7fcb9413ad10 2024-04-26T00:35:57.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.025+0000 7fcb7b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:57.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.025+0000 7fcb7b7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:35:57.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.025+0000 7fcb7b7fe640 1 -- 172.21.15.37:0/3110668719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9413ad10 msgr2=0x7fcb9402c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:57.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.025+0000 7fcb7b7fe640 1 --2- 172.21.15.37:0/3110668719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9413ad10 0x7fcb9402c000 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fcb840ad000 tx=0x7fcb8401ac20 comp rx=0 tx=0).stop 2024-04-26T00:35:57.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.025+0000 7fcb7b7fe640 1 -- 172.21.15.37:0/3110668719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9413ad10 msgr2=0x7fcb9402c000 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:35:57.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.025+0000 7fcb7b7fe640 1 --2- 172.21.15.37:0/3110668719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9413ad10 0x7fcb9402c000 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fcb840ad000 tx=0x7fcb8401ac20 comp rx=0 tx=0).stop 2024-04-26T00:35:57.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.033+0000 7fcb9909b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:35:57.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.033+0000 7fcb9909b640 1 --2- 172.21.15.37:0/3110668719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9413b0f0 0x7fcb9402c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:57.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.033+0000 7fcb9909b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:57.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.033+0000 7fcb9909b640 1 -- 172.21.15.37:0/3110668719 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcb94075dc0 con 0x7fcb9413b0f0 2024-04-26T00:35:57.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.033+0000 7fcb92d9d640 1 --2- 172.21.15.37:0/3110668719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9413b0f0 0x7fcb9402c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:57.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.033+0000 7fcb92d9d640 10 monclient: get_auth_request con 0x7fcb9413b0f0 auth_method 0 2024-04-26T00:35:57.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.033+0000 7fcb92d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:57.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.033+0000 7fcb92d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:57.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.033+0000 7fcb92d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:57.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.033+0000 7fcb92d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:57.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.033+0000 7fcb92d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:57.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.033+0000 7fcb92d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:57.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.033+0000 7fcb92d9d640 10 monclient(hunting): handle_auth_done global_id 194101 payload 824 2024-04-26T00:35:57.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.033+0000 7fcb92d9d640 1 -- 172.21.15.37:0/3110668719 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcb94137560 con 0x7fcb9413b0f0 2024-04-26T00:35:57.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.037+0000 7fcb92d9d640 1 -- 172.21.15.37:0/3110668719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9413b0f0 msgr2=0x7fcb9402c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:57.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.037+0000 7fcb92d9d640 1 --2- 172.21.15.37:0/3110668719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9413b0f0 0x7fcb9402c540 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcb84000c10 tx=0x7fcb84019180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:57.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.037+0000 7fcb7b7fe640 1 -- 172.21.15.37:0/3110668719 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb840c9070 con 0x7fcb9413b0f0 2024-04-26T00:35:57.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.037+0000 7fcb7b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:57.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.037+0000 7fcb7b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:35:57.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.037+0000 7fcb7b7fe640 10 monclient: dump: 2024-04-26T00:35:57.035 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:57.035 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:57.035 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:57.035 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:57.035 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:57.035 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:57.036 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:57.036 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:57.036 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:57.036 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:57.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.037+0000 7fcb7b7fe640 1 -- 172.21.15.37:0/3110668719 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7fcb84003c60 con 0x7fcb9413b0f0 2024-04-26T00:35:57.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.037+0000 7fcb7b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:35:57.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.037+0000 7fcb7b7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:35:57.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.037+0000 7fcb7b7fe640 1 -- 172.21.15.37:0/3110668719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9413b0f0 msgr2=0x7fcb9402c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:57.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.037+0000 7fcb7b7fe640 1 --2- 172.21.15.37:0/3110668719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9413b0f0 0x7fcb9402c540 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcb84000c10 tx=0x7fcb84019180 comp rx=0 tx=0).stop 2024-04-26T00:35:57.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.037+0000 7fcb7b7fe640 1 -- 172.21.15.37:0/3110668719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9413b0f0 msgr2=0x7fcb9402c540 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:35:57.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.037+0000 7fcb7b7fe640 1 --2- 172.21.15.37:0/3110668719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9413b0f0 0x7fcb9402c540 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcb84000c10 tx=0x7fcb84019180 comp rx=0 tx=0).stop 2024-04-26T00:35:57.037 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:35:57.037 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:35:57.037 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:35:57.037 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:35:57.037 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 120, 2024-04-26T00:35:57.038 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 5611, 2024-04-26T00:35:57.038 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:35:57.038 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:35:57.038 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:35:57.038 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:35:57.038 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:57.038 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 2, 2024-04-26T00:35:57.038 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:35:57.038 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:35:57.038 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:35:57.038 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:57.038 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:57.038 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:57.038 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:57.039 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:57.039 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:57.039 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:57.039 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:57.039 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:57.039 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:57.039 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:57.039 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:57.039 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:35:57.039 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:35:57.039 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:57.039 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:57.039 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:57.039 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:57.040 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:57.040 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:57.040 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:57.040 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:57.040 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:57.040 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:57.040 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:57.040 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:57.040 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:57.040 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:35:57.040 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:35:57.040 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:35:57.040 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:35:57.040 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:35:57.041 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:35:57.041 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:35:57.041 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:35:57.041 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:35:57.041 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:35:57.041 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:35:57.041 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:35:57.041 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:35:57.041 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:35:57.041 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:35:57.041 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:35:57.041 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:35:57.041 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:57.041 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:57.041 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:57.042 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:57.042 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:57.042 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:57.042 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:57.042 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:57.042 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:57.042 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:57.042 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:57.042 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:57.042 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:35:57.042 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:57.042 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:35:57.042 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:57.042 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:35:57.042 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:35:57.043 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:57.043 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:57.043 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:57.043 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:57.043 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:35:57.043 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:57.043 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:57.043 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:57.043 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:57.043 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:35:57.043 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:57.043 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:57.043 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:57.043 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:57.044 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:35:57.044 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:35:57.044 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:57.044 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:57.044 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:57.044 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:57.044 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:57.044 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:35:57.044 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:35:57.044 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:57.044 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:57.044 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:57.044 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:57.044 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:35:57.045 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:57.045 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:57.045 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:57.045 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:57.045 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:35:57.045 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:57.045 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:57.045 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:57.045 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:57.045 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:35:57.045 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:35:57.045 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:57.045 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:57.045 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:57.045 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:57.045 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:57.046 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:35:57.046 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:35:57.046 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:57.046 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:57.046 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:57.046 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:57.046 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:35:57.046 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:57.046 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:57.046 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:57.046 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:57.046 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:35:57.046 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:57.046 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:57.047 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:57.047 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:57.047 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:35:57.047 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:35:57.047 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:57.047 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:57.047 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:57.047 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:57.047 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:57.047 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:57.047 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:35:57.047 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:35:57.047 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:57.047 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:57.047 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:57.048 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:35:57.048 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:57.048 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:57.048 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:35:57.048 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:57.048 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:57.048 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:57.048 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:35:57.048 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:57.048 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:57.048 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:57.048 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:35:57.048 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:35:57.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.037+0000 7fcb9909b640 1 -- 172.21.15.37:0/3110668719 >> v2:172.21.15.45:6812/2379918623 conn(0x7fcb80035ed0 msgr2=0x7fcb80038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:57.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.037+0000 7fcb9909b640 1 --2- 172.21.15.37:0/3110668719 >> v2:172.21.15.45:6812/2379918623 conn(0x7fcb80035ed0 0x7fcb80038390 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fcb94120350 tx=0x7fcb8c007780 comp rx=0 tx=0).stop 2024-04-26T00:35:57.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.037+0000 7fcb9909b640 10 monclient: shutdown 2024-04-26T00:35:57.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.037+0000 7fcb9909b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:57.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.037+0000 7fcb9909b640 1 -- 172.21.15.37:0/3110668719 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb940a4170 msgr2=0x7fcb94137020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:57.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.037+0000 7fcb9909b640 1 --2- 172.21.15.37:0/3110668719 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb940a4170 0x7fcb94137020 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcb8800c480 tx=0x7fcb8800c940 comp rx=0 tx=0).stop 2024-04-26T00:35:57.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.037+0000 7fcb9359e640 1 -- 172.21.15.37:0/3110668719 reap_dead start 2024-04-26T00:35:57.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.037+0000 7fcb9359e640 1 -- 172.21.15.37:0/3110668719 reap_dead start 2024-04-26T00:35:57.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.037+0000 7fcb9909b640 1 -- 172.21.15.37:0/3110668719 shutdown_connections 2024-04-26T00:35:57.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.037+0000 7fcb9909b640 1 -- 172.21.15.37:0/3110668719 >> 172.21.15.37:0/3110668719 conn(0x7fcb940aa0a0 msgr2=0x7fcb9409f080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:57.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.037+0000 7fcb9909b640 1 -- 172.21.15.37:0/3110668719 shutdown_connections 2024-04-26T00:35:57.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.037+0000 7fcb9909b640 1 -- 172.21.15.37:0/3110668719 wait complete. 2024-04-26T00:35:57.061 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:35:57.061 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-26T00:35:57.061 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-26T00:35:57.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.189+0000 7f5eec574640 10 monclient: get_monmap_and_config 2024-04-26T00:35:57.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.189+0000 7f5eec574640 10 monclient: build_initial_monmap 2024-04-26T00:35:57.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.189+0000 7f5eec574640 10 monclient: monmap: 2024-04-26T00:35:57.189 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:57.189 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:57.189 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:35:57.195044+0000 2024-04-26T00:35:57.189 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:35:57.195044+0000 2024-04-26T00:35:57.189 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:57.190 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:57.190 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:35:57.190 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:35:57.190 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:35:57.190 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:35:57.190 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:35:57.190 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:57.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5eec574640 1 Processor -- start 2024-04-26T00:35:57.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5eec574640 1 -- start start 2024-04-26T00:35:57.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5eec574640 10 monclient: init 2024-04-26T00:35:57.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5eec574640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:57.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5eec574640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-04-26T00:35:57.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5eec574640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ee4136da0 0x7f5ee409bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:57.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5eec574640 10 monclient(hunting): picked mon.noname-b con 0x7f5ee4136da0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:57.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5eec574640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5ee409c410 0x7f5ee4137180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:57.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5eec574640 10 monclient(hunting): picked mon.noname-c con 0x7f5ee409c410 addr v2:172.21.15.45:3300/0 2024-04-26T00:35:57.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5eec574640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5ee41376c0 addr v1:172.21.15.45:6789/0 2024-04-26T00:35:57.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5eec574640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ee409b4f0 con 0x7f5ee41376c0 2024-04-26T00:35:57.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5eec574640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:57.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5eec574640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f5ee409b210 con 0x7f5ee409c410 2024-04-26T00:35:57.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5eec574640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:57.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5eec574640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5ee409b7d0 con 0x7f5ee4136da0 2024-04-26T00:35:57.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5eec574640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:57.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5eec574640 10 monclient(hunting): authenticate will time out at 1938.501191s 2024-04-26T00:35:57.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5eea310640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ee4136da0 0x7f5ee409bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:57.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5eea310640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ee4136da0 0x7f5ee409bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:50904/0 (socket says 172.21.15.37:50904) 2024-04-26T00:35:57.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5eea310640 1 -- 172.21.15.37:0/2501357722 learned_addr learned my addr 172.21.15.37:0/2501357722 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:57.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5eea310640 10 monclient(hunting): get_auth_request con 0x7f5ee4136da0 auth_method 0 2024-04-26T00:35:57.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5eea310640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:57.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5eea310640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:57.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5eea310640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:57.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5eea310640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:57.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5eea310640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:57.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5eea310640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:57.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5ee9b0f640 1 --2- 172.21.15.37:0/2501357722 >> v2:172.21.15.45:3300/0 conn(0x7f5ee409c410 0x7f5ee4137180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:57.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5ee9b0f640 10 monclient(hunting): get_auth_request con 0x7f5ee409c410 auth_method 0 2024-04-26T00:35:57.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5ee9b0f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:57.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5ee9b0f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:57.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5ee9b0f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:57.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5ee9b0f640 1 -- 172.21.15.37:0/2501357722 >> v2:172.21.15.45:3300/0 conn(0x7f5ee409c410 msgr2=0x7f5ee4137180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:35:57.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5ee9b0f640 1 -- 172.21.15.37:0/2501357722 >> v2:172.21.15.45:3300/0 conn(0x7f5ee409c410 msgr2=0x7f5ee4137180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:35:57.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5ee9b0f640 1 --2- 172.21.15.37:0/2501357722 >> v2:172.21.15.45:3300/0 conn(0x7f5ee409c410 0x7f5ee4137180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:35:57.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5eea310640 10 monclient(hunting): handle_auth_done global_id 184205 payload 274 2024-04-26T00:35:57.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5eea310640 1 -- 172.21.15.37:0/2501357722 >> v2:172.21.15.45:3300/0 conn(0x7f5ee409c410 msgr2=0x7f5ee4137180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:57.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5ee9b0f640 1 --2- 172.21.15.37:0/2501357722 >> v2:172.21.15.45:3300/0 conn(0x7f5ee409c410 0x7f5ee4137180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:35:57.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5eea310640 1 --2- 172.21.15.37:0/2501357722 >> v2:172.21.15.45:3300/0 conn(0x7f5ee409c410 0x7f5ee4137180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:57.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5eea310640 1 -- 172.21.15.37:0/2501357722 >> v1:172.21.15.45:6789/0 conn(0x7f5ee41376c0 legacy=0x7f5ee4133b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:57.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5eea310640 10 monclient: _finish_hunting 0 2024-04-26T00:35:57.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5eea310640 1 monclient: found mon.noname-b 2024-04-26T00:35:57.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5eea310640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:57.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5eea310640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:35:57.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.193+0000 7f5eea310640 1 -- 172.21.15.37:0/2501357722 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ee4137b30 con 0x7f5ee4136da0 2024-04-26T00:35:57.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.197+0000 7f5eea310640 1 -- 172.21.15.37:0/2501357722 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ee4136da0 msgr2=0x7f5ee409bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:57.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.197+0000 7f5eea310640 1 --2- 172.21.15.37:0/2501357722 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ee4136da0 0x7f5ee409bed0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5ee0009870 tx=0x7f5ee001c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8975d4c007f0180b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:57.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.197+0000 7f5ee930e640 1 -- 172.21.15.37:0/2501357722 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee00b0070 con 0x7f5ee4136da0 2024-04-26T00:35:57.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.197+0000 7f5ee930e640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:57.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.197+0000 7f5ee930e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:35:57.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.197+0000 7f5ee930e640 10 monclient: dump: 2024-04-26T00:35:57.198 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:57.198 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:57.198 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:57.198 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:57.198 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:57.198 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:57.199 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:57.199 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:57.199 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:57.199 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:57.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.197+0000 7f5ee930e640 10 monclient: _finish_auth 0 2024-04-26T00:35:57.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.197+0000 7f5ee930e640 10 monclient: _check_auth_tickets 2024-04-26T00:35:57.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.197+0000 7f5ee930e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:57.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.197+0000 7f5ee930e640 1 -- 172.21.15.37:0/2501357722 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ee00ab600 con 0x7f5ee4136da0 2024-04-26T00:35:57.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.197+0000 7f5ee930e640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:57.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.197+0000 7f5ee930e640 1 -- 172.21.15.37:0/2501357722 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee00ab9d0 con 0x7f5ee4136da0 2024-04-26T00:35:57.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.197+0000 7f5eec574640 5 monclient: authenticate success, global_id 184205 2024-04-26T00:35:57.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.197+0000 7f5eec574640 10 monclient: get_monmap_and_config success 2024-04-26T00:35:57.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.197+0000 7f5eec574640 4 set_mon_vals no callback set 2024-04-26T00:35:57.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.197+0000 7f5ee930e640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:57.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.197+0000 7f5ee930e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:35:57.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.197+0000 7f5ee930e640 10 monclient: dump: 2024-04-26T00:35:57.200 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:57.200 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:57.200 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:57.200 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:57.200 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:57.200 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:57.200 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:57.200 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:57.200 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:57.201 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:57.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.197+0000 7f5eec574640 10 monclient: shutdown 2024-04-26T00:35:57.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.197+0000 7f5eec574640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:57.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.197+0000 7f5eec574640 1 -- 172.21.15.37:0/2501357722 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ee4136da0 msgr2=0x7f5ee409bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:57.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.197+0000 7f5eec574640 1 --2- 172.21.15.37:0/2501357722 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ee4136da0 0x7f5ee409bed0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5ee0009870 tx=0x7f5ee001c3e0 comp rx=0 tx=0).stop 2024-04-26T00:35:57.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.197+0000 7f5eec574640 1 -- 172.21.15.37:0/2501357722 shutdown_connections 2024-04-26T00:35:57.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.197+0000 7f5eec574640 1 --2- 172.21.15.37:0/2501357722 >> v2:172.21.15.45:3300/0 conn(0x7f5ee409c410 0x7f5ee4137180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:57.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.197+0000 7f5eec574640 1 --2- 172.21.15.37:0/2501357722 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ee4136da0 0x7f5ee409bed0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:57.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.197+0000 7f5eec574640 1 -- 172.21.15.37:0/2501357722 >> 172.21.15.37:0/2501357722 conn(0x7f5ee40a5f00 msgr2=0x7f5ee40a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:57.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.197+0000 7f5eec574640 1 -- 172.21.15.37:0/2501357722 shutdown_connections 2024-04-26T00:35:57.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.197+0000 7f5eec574640 1 -- 172.21.15.37:0/2501357722 wait complete. 2024-04-26T00:35:57.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.197+0000 7f5eec574640 10 monclient: build_initial_monmap 2024-04-26T00:35:57.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.197+0000 7f5eec574640 10 monclient: monmap: 2024-04-26T00:35:57.202 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:57.202 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:35:57.202 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:35:57.202 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:35:57.202 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:57.202 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:35:57.202 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:35:57.202 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:35:57.202 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:35:57.202 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:57.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.197+0000 7f5eec574640 1 Processor -- start 2024-04-26T00:35:57.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.197+0000 7f5eec574640 1 -- start start 2024-04-26T00:35:57.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.197+0000 7f5eec574640 10 monclient: init 2024-04-26T00:35:57.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eec574640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:57.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eec574640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:35:57.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eec574640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ee409c410 0x7f5ee41cf640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:57.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eec574640 10 monclient(hunting): picked mon.noname-c con 0x7f5ee409c410 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:57.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eec574640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ee4136da0 0x7f5ee413de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:57.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eec574640 10 monclient(hunting): picked mon.noname-b con 0x7f5ee4136da0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:35:57.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eec574640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5ee41376c0 0x7f5ee413e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:57.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eec574640 10 monclient(hunting): picked mon.noname-a con 0x7f5ee41376c0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:35:57.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eec574640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:57.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eec574640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f5ee4141ea0 con 0x7f5ee41376c0 2024-04-26T00:35:57.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eec574640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:57.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eec574640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5ee4141d20 con 0x7f5ee4136da0 2024-04-26T00:35:57.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eec574640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:57.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eec574640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5ee4142020 con 0x7f5ee409c410 2024-04-26T00:35:57.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eec574640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:57.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eec574640 10 monclient(hunting): authenticate will time out at 1938.508388s 2024-04-26T00:35:57.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eea310640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ee409c410 0x7f5ee41cf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:57.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eea310640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ee409c410 0x7f5ee41cf640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:50914/0 (socket says 172.21.15.37:50914) 2024-04-26T00:35:57.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eea310640 1 -- 172.21.15.37:0/2873393720 learned_addr learned my addr 172.21.15.37:0/2873393720 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:57.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eea310640 10 monclient(hunting): get_auth_request con 0x7f5ee409c410 auth_method 0 2024-04-26T00:35:57.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eea310640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:57.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eea310640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:57.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eea310640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:57.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eeab11640 1 --2- 172.21.15.37:0/2873393720 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5ee41376c0 0x7f5ee413e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:57.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eeab11640 10 monclient(hunting): get_auth_request con 0x7f5ee41376c0 auth_method 0 2024-04-26T00:35:57.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eeab11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:57.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eeab11640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:57.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eeab11640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:57.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eea310640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:57.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eea310640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:57.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eea310640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:57.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eeab11640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:57.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eeab11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:57.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eeab11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:57.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5ee9b0f640 1 --2- 172.21.15.37:0/2873393720 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ee4136da0 0x7f5ee413de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:57.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eea310640 10 monclient(hunting): handle_auth_done global_id 184208 payload 824 2024-04-26T00:35:57.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eea310640 1 -- 172.21.15.37:0/2873393720 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ee4136da0 msgr2=0x7f5ee413de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:57.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eea310640 1 --2- 172.21.15.37:0/2873393720 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ee4136da0 0x7f5ee413de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:57.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eea310640 1 -- 172.21.15.37:0/2873393720 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5ee41376c0 msgr2=0x7f5ee413e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:57.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eea310640 1 --2- 172.21.15.37:0/2873393720 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5ee41376c0 0x7f5ee413e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:57.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eea310640 10 monclient: _finish_hunting 0 2024-04-26T00:35:57.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eea310640 1 monclient: found mon.noname-c 2024-04-26T00:35:57.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eea310640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:57.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eea310640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:35:57.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eea310640 1 -- 172.21.15.37:0/2873393720 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ee4138630 con 0x7f5ee409c410 2024-04-26T00:35:57.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5ee9b0f640 10 monclient: get_auth_request con 0x7f5ee4136da0 auth_method 0 2024-04-26T00:35:57.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5ee9b0f640 1 --2- 172.21.15.37:0/2873393720 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ee4136da0 0x7f5ee413de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:35:57.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eea310640 1 -- 172.21.15.37:0/2873393720 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ee409c410 msgr2=0x7f5ee41cf640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:57.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eea310640 1 --2- 172.21.15.37:0/2873393720 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ee409c410 0x7f5ee41cf640 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5ee00ae040 tx=0x7f5ee001ac20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:57.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5edaffd640 1 -- 172.21.15.37:0/2873393720 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee00ba020 con 0x7f5ee409c410 2024-04-26T00:35:57.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5edaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:57.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5edaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:35:57.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5edaffd640 10 monclient: dump: 2024-04-26T00:35:57.211 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:57.211 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:57.211 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:57.211 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:57.211 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:57.211 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:57.211 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:57.211 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:57.211 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:57.211 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:57.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5edaffd640 10 monclient: _finish_auth 0 2024-04-26T00:35:57.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5edaffd640 10 monclient: _check_auth_tickets 2024-04-26T00:35:57.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5edaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:57.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5edaffd640 1 -- 172.21.15.37:0/2873393720 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ee00b6850 con 0x7f5ee409c410 2024-04-26T00:35:57.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eec574640 5 monclient: authenticate success, global_id 184208 2024-04-26T00:35:57.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eec574640 10 monclient: _renew_subs 2024-04-26T00:35:57.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eec574640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:35:57.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eec574640 1 -- 172.21.15.37:0/2873393720 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ee4138910 con 0x7f5ee409c410 2024-04-26T00:35:57.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5edaffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:57.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5edaffd640 1 -- 172.21.15.37:0/2873393720 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee00b0040 con 0x7f5ee409c410 2024-04-26T00:35:57.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5edaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:57.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5edaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:35:57.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5edaffd640 10 monclient: dump: 2024-04-26T00:35:57.212 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:57.212 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:57.212 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:57.212 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:57.212 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:57.212 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:57.213 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:57.213 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:57.213 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:57.213 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:57.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eec574640 10 monclient: _renew_subs 2024-04-26T00:35:57.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eec574640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:35:57.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5ee8b0d640 4 set_mon_vals no callback set 2024-04-26T00:35:57.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.201+0000 7f5eec574640 1 -- 172.21.15.37:0/2873393720 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ee4137db0 con 0x7f5ee409c410 2024-04-26T00:35:57.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.205+0000 7f5eec574640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:57.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.205+0000 7f5edaffd640 1 -- 172.21.15.37:0/2873393720 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f5ee00bd020 con 0x7f5ee409c410 2024-04-26T00:35:57.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.205+0000 7f5eec574640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:57.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.205+0000 7f5eec574640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:35:57.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.205+0000 7f5eec574640 1 -- 172.21.15.37:0/2873393720 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ee4066f40 con 0x7f5ee409c410 2024-04-26T00:35:57.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.205+0000 7f5edaffd640 1 --2- 172.21.15.37:0/2873393720 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5ec8035ed0 0x7f5ec8038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:57.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.205+0000 7f5edaffd640 1 -- 172.21.15.37:0/2873393720 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(627..627 src has 370..627) ==== 3943+0+0 (secure 0 0 0) 0x7f5ee00e2e70 con 0x7f5ee409c410 2024-04-26T00:35:57.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.205+0000 7f5ee9b0f640 1 --2- 172.21.15.37:0/2873393720 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5ec8035ed0 0x7f5ec8038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:57.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.205+0000 7f5ee9b0f640 10 monclient: get_auth_request con 0x7f5ec8035ed0 auth_method 0 2024-04-26T00:35:57.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.205+0000 7f5ee9b0f640 1 --2- 172.21.15.37:0/2873393720 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5ec8035ed0 0x7f5ec8038390 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f5ed4006aa0 tx=0x7f5ed4006a30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:57.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.209+0000 7f5edaffd640 1 -- 172.21.15.37:0/2873393720 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5ee00c4030 con 0x7f5ee409c410 2024-04-26T00:35:57.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.209+0000 7f5edaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:57.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.209+0000 7f5edaffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:35:57.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.437+0000 7f5eec574640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-26T00:35:57.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.437+0000 7f5eec574640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-26T00:35:57.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.437+0000 7f5eec574640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:35:57.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.437+0000 7f5eec574640 1 -- 172.21.15.37:0/2873393720 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f5ee41325e0 con 0x7f5ee409c410 2024-04-26T00:35:57.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.437+0000 7f5edaffd640 1 -- 172.21.15.37:0/2873393720 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f5ee00160f0 con 0x7f5ee409c410 2024-04-26T00:35:57.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.437+0000 7f5edaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-26T00:35:57.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.437+0000 7f5edaffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:35:57.439 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.441+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.439 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.441+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=1,health=12,kv=10,logm=51} crc {auth=3219220428,config=3678701417,health=2770241980,kv=1034996044,logm=2981416742}) 2024-04-26T00:35:57.439 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.441+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=28,config=2,health=11,kv=9,logm=50} crc {auth=4112304187,config=692619927,health=3884750207,kv=2700185179,logm=1838142264}) 2024-04-26T00:35:57.439 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.441+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.439 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.441+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=1,health=12,kv=10,logm=51} crc {auth=3219220428,config=3678701417,health=2770241980,kv=1034996044,logm=2981416742}) 2024-04-26T00:35:57.439 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.441+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=30,config=2,health=10,kv=9,logm=49} crc {auth=3487931948,config=692619927,health=3109683543,kv=2700185179,logm=2664770413}) 2024-04-26T00:35:57.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.441+0000 7f5eec574640 1 -- 172.21.15.37:0/2873393720 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5ec8035ed0 msgr2=0x7f5ec8038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:57.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.441+0000 7f5eec574640 1 --2- 172.21.15.37:0/2873393720 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5ec8035ed0 0x7f5ec8038390 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f5ed4006aa0 tx=0x7f5ed4006a30 comp rx=0 tx=0).stop 2024-04-26T00:35:57.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.441+0000 7f5eec574640 10 monclient: shutdown 2024-04-26T00:35:57.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.441+0000 7f5eec574640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:57.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.441+0000 7f5eec574640 1 -- 172.21.15.37:0/2873393720 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ee409c410 msgr2=0x7f5ee41cf640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:57.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.441+0000 7f5eec574640 1 --2- 172.21.15.37:0/2873393720 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ee409c410 0x7f5ee41cf640 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5ee00ae040 tx=0x7f5ee001ac20 comp rx=0 tx=0).stop 2024-04-26T00:35:57.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.441+0000 7f5eec574640 1 -- 172.21.15.37:0/2873393720 shutdown_connections 2024-04-26T00:35:57.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.441+0000 7f5eec574640 1 --2- 172.21.15.37:0/2873393720 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5ec8035ed0 0x7f5ec8038390 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:57.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.441+0000 7f5eec574640 1 --2- 172.21.15.37:0/2873393720 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5ee41376c0 0x7f5ee413e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:57.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.441+0000 7f5eec574640 1 --2- 172.21.15.37:0/2873393720 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5ee4136da0 0x7f5ee413de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:57.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.441+0000 7f5eec574640 1 --2- 172.21.15.37:0/2873393720 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ee409c410 0x7f5ee41cf640 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:57.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.441+0000 7f5eec574640 1 -- 172.21.15.37:0/2873393720 >> 172.21.15.37:0/2873393720 conn(0x7f5ee40a5f00 msgr2=0x7f5ee40a2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:57.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.441+0000 7f5eec574640 1 -- 172.21.15.37:0/2873393720 shutdown_connections 2024-04-26T00:35:57.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:57.441+0000 7f5eec574640 1 -- 172.21.15.37:0/2873393720 wait complete. 2024-04-26T00:35:57.445 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.441+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.445 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.441+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2191777750}) 2024-04-26T00:35:57.445 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.441+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3967951656}) 2024-04-26T00:35:57.445 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.441+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.446 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.441+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2191777750}) 2024-04-26T00:35:57.446 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.441+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2942118015}) 2024-04-26T00:35:57.446 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.445+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.446 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.445+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2865191216}) 2024-04-26T00:35:57.446 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.445+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3398613911}) 2024-04-26T00:35:57.446 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.445+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.446 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.445+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2865191216}) 2024-04-26T00:35:57.447 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.445+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1842615874}) 2024-04-26T00:35:57.448 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.449+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.448 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.449+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=238943811}) 2024-04-26T00:35:57.448 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.449+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3613289228}) 2024-04-26T00:35:57.448 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.449+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.449 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.449+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=238943811}) 2024-04-26T00:35:57.449 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.449+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3774771375}) 2024-04-26T00:35:57.450 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.449+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.450 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.449+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3797303894}) 2024-04-26T00:35:57.450 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.449+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=582127286}) 2024-04-26T00:35:57.451 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.449+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.451 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.449+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3797303894}) 2024-04-26T00:35:57.451 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.449+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2435937866}) 2024-04-26T00:35:57.455 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.453+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.455 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.453+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4117888977}) 2024-04-26T00:35:57.455 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.453+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2187226367}) 2024-04-26T00:35:57.455 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.453+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.455 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.453+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4117888977}) 2024-04-26T00:35:57.455 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.453+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3642176669}) 2024-04-26T00:35:57.455 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.457+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.455 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.457+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3643327930}) 2024-04-26T00:35:57.455 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.457+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3201995058}) 2024-04-26T00:35:57.456 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.457+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.456 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.457+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3643327930}) 2024-04-26T00:35:57.456 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.457+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2436287079}) 2024-04-26T00:35:57.457 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.457+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.457 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.457+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3023730600}) 2024-04-26T00:35:57.457 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.457+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1663846801}) 2024-04-26T00:35:57.457 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.457+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.457 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.457+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3023730600}) 2024-04-26T00:35:57.457 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.457+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2129581376}) 2024-04-26T00:35:57.459 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-26T00:35:57.573 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.573+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.573 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.573+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1251515517}) 2024-04-26T00:35:57.573 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.573+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4215360613}) 2024-04-26T00:35:57.573 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.573+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.573 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.573+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1251515517}) 2024-04-26T00:35:57.573 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.573+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3163453912}) 2024-04-26T00:35:57.579 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.581+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.579 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.581+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1962972185}) 2024-04-26T00:35:57.579 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.581+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1340720382}) 2024-04-26T00:35:57.580 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.581+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.580 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.581+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1962972185}) 2024-04-26T00:35:57.580 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.581+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1236174504}) 2024-04-26T00:35:57.596 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.597+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.596 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.597+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3304540353}) 2024-04-26T00:35:57.596 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.597+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2341544601}) 2024-04-26T00:35:57.596 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.597+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.596 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.597+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3304540353}) 2024-04-26T00:35:57.596 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.597+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3703307425}) 2024-04-26T00:35:57.602 INFO:tasks.rados.rados.0.smithi045.stdout:2711: done (2 left) 2024-04-26T00:35:57.602 INFO:tasks.rados.rados.0.smithi045.stdout:2712: copy_from oid 4 from oid 26 current snap is 301 2024-04-26T00:35:57.602 INFO:tasks.rados.rados.0.smithi045.stdout:2707: finishing write tid 1 to smithi04526579-19 2024-04-26T00:35:57.602 INFO:tasks.rados.rados.0.smithi045.stdout:2707: finishing write tid 2 to smithi04526579-19 2024-04-26T00:35:57.602 INFO:tasks.rados.rados.0.smithi045.stdout:2707: finishing write tid 3 to smithi04526579-19 2024-04-26T00:35:57.603 INFO:tasks.rados.rados.0.smithi045.stdout:2707: finishing write tid 4 to smithi04526579-19 2024-04-26T00:35:57.603 INFO:tasks.rados.rados.0.smithi045.stdout:2707: oid 19 updating version 0 to 1976 2024-04-26T00:35:57.603 INFO:tasks.rados.rados.0.smithi045.stdout:2707: oid 19 version 1976 is already newer than 1975 2024-04-26T00:35:57.603 INFO:tasks.rados.rados.0.smithi045.stdout:2707: oid 19 version 1976 is already newer than 1974 2024-04-26T00:35:57.603 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 1976 (ObjNum 975 snap 301 seq_num 975) dirty exists 2024-04-26T00:35:57.603 INFO:tasks.rados.rados.0.smithi045.stdout:2707: left oid 19 (ObjNum 975 snap 301 seq_num 975) 2024-04-26T00:35:57.603 INFO:tasks.rados.rados.0.smithi045.stdout:2710: finishing write tid 1 to smithi04526579-25 2024-04-26T00:35:57.603 INFO:tasks.rados.rados.0.smithi045.stdout:2710: finishing write tid 2 to smithi04526579-25 2024-04-26T00:35:57.603 INFO:tasks.rados.rados.0.smithi045.stdout:2710: finishing write tid 3 to smithi04526579-25 2024-04-26T00:35:57.603 INFO:tasks.rados.rados.0.smithi045.stdout:2710: finishing write tid 4 to smithi04526579-25 2024-04-26T00:35:57.603 INFO:tasks.rados.rados.0.smithi045.stdout:2710: oid 25 updating version 0 to 2341 2024-04-26T00:35:57.603 INFO:tasks.rados.rados.0.smithi045.stdout:2710: oid 25 updating version 2341 to 2343 2024-04-26T00:35:57.604 INFO:tasks.rados.rados.0.smithi045.stdout:2710: oid 25 version 2343 is already newer than 2342 2024-04-26T00:35:57.604 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 2343 (ObjNum 976 snap 301 seq_num 976) dirty exists 2024-04-26T00:35:57.604 INFO:tasks.rados.rados.0.smithi045.stdout:2710: left oid 25 (ObjNum 976 snap 301 seq_num 976) 2024-04-26T00:35:57.604 INFO:tasks.rados.rados.0.smithi045.stdout:2707: done (2 left) 2024-04-26T00:35:57.604 INFO:tasks.rados.rados.0.smithi045.stdout:2710: done (1 left) 2024-04-26T00:35:57.604 INFO:tasks.rados.rados.0.smithi045.stdout:2713: copy_from oid 23 from oid 9 current snap is 301 2024-04-26T00:35:57.604 INFO:tasks.rados.rados.0.smithi045.stdout:2714: copy_from oid 17 from oid 16 current snap is 301 2024-04-26T00:35:57.604 INFO:tasks.rados.rados.0.smithi045.stdout:2715: snap_remove snap 297 2024-04-26T00:35:57.612 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.613+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.612 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.613+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1366900221}) 2024-04-26T00:35:57.612 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.613+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2141030127}) 2024-04-26T00:35:57.612 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.613+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.612 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.613+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1366900221}) 2024-04-26T00:35:57.613 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.613+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1970861048}) 2024-04-26T00:35:57.619 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.621+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.620 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.621+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1464243881}) 2024-04-26T00:35:57.620 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.621+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3786224865}) 2024-04-26T00:35:57.620 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.621+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.620 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.621+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1464243881}) 2024-04-26T00:35:57.620 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.621+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1371526822}) 2024-04-26T00:35:57.629 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.625+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.629 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.625+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3174643970}) 2024-04-26T00:35:57.629 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.625+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4112776039}) 2024-04-26T00:35:57.629 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.625+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.629 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.625+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3174643970}) 2024-04-26T00:35:57.630 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.625+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=561859981}) 2024-04-26T00:35:57.637 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.637+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.638 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.637+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=424467218}) 2024-04-26T00:35:57.638 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.637+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1593645520}) 2024-04-26T00:35:57.638 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.637+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.638 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.637+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=424467218}) 2024-04-26T00:35:57.638 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.637+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4016241548}) 2024-04-26T00:35:57.641 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.641+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.641 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.641+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3716687406}) 2024-04-26T00:35:57.641 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.641+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3752903799}) 2024-04-26T00:35:57.641 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.641+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.641 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.641+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3716687406}) 2024-04-26T00:35:57.641 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.641+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1229446565}) 2024-04-26T00:35:57.644 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.645+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.644 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.645+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3838100606}) 2024-04-26T00:35:57.645 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.645+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=930146900}) 2024-04-26T00:35:57.645 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.645+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.645 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.645+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3838100606}) 2024-04-26T00:35:57.645 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.645+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3712181620}) 2024-04-26T00:35:57.647 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.649+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.647 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.649+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=636318394}) 2024-04-26T00:35:57.647 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.649+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4122975230}) 2024-04-26T00:35:57.648 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.649+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.648 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.649+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=636318394}) 2024-04-26T00:35:57.648 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.649+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=420849763}) 2024-04-26T00:35:57.652 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.653+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.652 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.653+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1759128993}) 2024-04-26T00:35:57.652 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.653+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=630939773}) 2024-04-26T00:35:57.652 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.653+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.652 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.653+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1759128993}) 2024-04-26T00:35:57.652 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.653+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4271096493}) 2024-04-26T00:35:57.657 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.657+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.657 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.657+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2057786016}) 2024-04-26T00:35:57.657 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.657+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2127289903}) 2024-04-26T00:35:57.657 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.657+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.657 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.657+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2057786016}) 2024-04-26T00:35:57.657 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.657+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1428568278}) 2024-04-26T00:35:57.662 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.665+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.662 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.665+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1545011796}) 2024-04-26T00:35:57.662 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.665+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2066897927}) 2024-04-26T00:35:57.662 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.665+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.662 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.665+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1545011796}) 2024-04-26T00:35:57.662 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.665+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2125933450}) 2024-04-26T00:35:57.664 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.665+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.664 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.665+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1689118361}) 2024-04-26T00:35:57.664 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.665+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4105941041}) 2024-04-26T00:35:57.664 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.665+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.664 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.665+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1689118361}) 2024-04-26T00:35:57.664 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.665+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4153081366}) 2024-04-26T00:35:57.666 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.669+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.666 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.669+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1328889619}) 2024-04-26T00:35:57.666 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.669+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3390707414}) 2024-04-26T00:35:57.666 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.669+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.666 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.669+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1328889619}) 2024-04-26T00:35:57.666 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.669+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=9783829}) 2024-04-26T00:35:57.668 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.669+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.669 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.669+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=582327717}) 2024-04-26T00:35:57.669 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.669+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4074348001}) 2024-04-26T00:35:57.670 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.669+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.670 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.669+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=582327717}) 2024-04-26T00:35:57.670 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.669+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1854909155}) 2024-04-26T00:35:57.671 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.673+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.671 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.673+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=959870196}) 2024-04-26T00:35:57.671 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.673+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3028863530}) 2024-04-26T00:35:57.672 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.673+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.672 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.673+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=959870196}) 2024-04-26T00:35:57.672 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.673+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3588944199}) 2024-04-26T00:35:57.674 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.673+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.674 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.673+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1114699647}) 2024-04-26T00:35:57.674 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.673+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=538483601}) 2024-04-26T00:35:57.674 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.673+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.674 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.673+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1114699647}) 2024-04-26T00:35:57.674 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.673+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3886939063}) 2024-04-26T00:35:57.674 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.677+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.674 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.677+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4119830904}) 2024-04-26T00:35:57.674 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.677+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1533171168}) 2024-04-26T00:35:57.674 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.677+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.675 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.677+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4119830904}) 2024-04-26T00:35:57.675 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.677+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2028419974}) 2024-04-26T00:35:57.676 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.677+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.676 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.677+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1029881922}) 2024-04-26T00:35:57.676 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.677+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2108376108}) 2024-04-26T00:35:57.676 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.677+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.676 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.677+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1029881922}) 2024-04-26T00:35:57.676 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.677+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1629173438}) 2024-04-26T00:35:57.678 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.681+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.678 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.681+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1449098344}) 2024-04-26T00:35:57.678 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.681+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1490961725}) 2024-04-26T00:35:57.678 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.681+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.678 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.681+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1449098344}) 2024-04-26T00:35:57.678 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.681+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3108105580}) 2024-04-26T00:35:57.680 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.681+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.680 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.681+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=684238341}) 2024-04-26T00:35:57.680 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.681+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1283227780}) 2024-04-26T00:35:57.680 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.681+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.680 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.681+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=684238341}) 2024-04-26T00:35:57.680 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.681+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3425029752}) 2024-04-26T00:35:57.682 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.685+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.682 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.685+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=632374067}) 2024-04-26T00:35:57.682 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.685+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1819516071}) 2024-04-26T00:35:57.682 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.685+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.682 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.685+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=632374067}) 2024-04-26T00:35:57.682 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.685+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1088198489}) 2024-04-26T00:35:57.684 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.685+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.684 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.685+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=456450287}) 2024-04-26T00:35:57.684 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.685+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=827652213}) 2024-04-26T00:35:57.684 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.685+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.684 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.685+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=456450287}) 2024-04-26T00:35:57.684 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.685+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3436883228}) 2024-04-26T00:35:57.686 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.689+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.686 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.689+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=109845168}) 2024-04-26T00:35:57.686 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.689+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1488007992}) 2024-04-26T00:35:57.686 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.689+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.686 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.689+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=109845168}) 2024-04-26T00:35:57.686 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.689+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3276096826}) 2024-04-26T00:35:57.688 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.689+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.688 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.689+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2282607884}) 2024-04-26T00:35:57.688 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.689+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=244862449}) 2024-04-26T00:35:57.688 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.689+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.688 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.689+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2282607884}) 2024-04-26T00:35:57.688 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.689+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=66914179}) 2024-04-26T00:35:57.690 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.693+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.690 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.693+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3796104679}) 2024-04-26T00:35:57.690 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.693+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=513678273}) 2024-04-26T00:35:57.690 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.693+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.691 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.693+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3796104679}) 2024-04-26T00:35:57.691 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.693+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2492557003}) 2024-04-26T00:35:57.692 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.693+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.692 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.693+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3403402137}) 2024-04-26T00:35:57.692 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.693+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=882242789}) 2024-04-26T00:35:57.692 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.693+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.693 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.693+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3403402137}) 2024-04-26T00:35:57.693 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.693+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1427498284}) 2024-04-26T00:35:57.694 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.697+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.694 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.697+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3872071151}) 2024-04-26T00:35:57.695 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.697+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2942498376}) 2024-04-26T00:35:57.695 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.697+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.695 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.697+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3872071151}) 2024-04-26T00:35:57.695 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.697+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3361118206}) 2024-04-26T00:35:57.696 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.697+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.697 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.697+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2060704154}) 2024-04-26T00:35:57.697 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.697+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=225321456}) 2024-04-26T00:35:57.697 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.697+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.697 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.697+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2060704154}) 2024-04-26T00:35:57.697 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.697+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4015576722}) 2024-04-26T00:35:57.698 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.701+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.699 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.701+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2951849463}) 2024-04-26T00:35:57.699 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.701+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1466563202}) 2024-04-26T00:35:57.699 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.701+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.699 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.701+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2951849463}) 2024-04-26T00:35:57.699 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.701+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=542136452}) 2024-04-26T00:35:57.700 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.701+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.700 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.701+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=611449074}) 2024-04-26T00:35:57.701 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.701+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2342826158}) 2024-04-26T00:35:57.701 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.701+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.701 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.701+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=611449074}) 2024-04-26T00:35:57.701 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.701+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=427779867}) 2024-04-26T00:35:57.702 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.705+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.703 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.705+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3366229216}) 2024-04-26T00:35:57.703 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.705+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2478542305}) 2024-04-26T00:35:57.703 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.705+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.703 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.705+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3366229216}) 2024-04-26T00:35:57.703 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.705+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1960804862}) 2024-04-26T00:35:57.705 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.705+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.705 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.705+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3527201890}) 2024-04-26T00:35:57.705 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.705+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2124644021}) 2024-04-26T00:35:57.705 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.705+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.705 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.705+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3527201890}) 2024-04-26T00:35:57.705 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.705+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1401308128}) 2024-04-26T00:35:57.707 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.709+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.707 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.709+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3808061034}) 2024-04-26T00:35:57.707 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.709+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=447419473}) 2024-04-26T00:35:57.707 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.709+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.707 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.709+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3808061034}) 2024-04-26T00:35:57.708 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.709+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=32915088}) 2024-04-26T00:35:57.708 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.709+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.708 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.709+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3244139927}) 2024-04-26T00:35:57.709 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.709+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3964151536}) 2024-04-26T00:35:57.709 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.709+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.709 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.709+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3244139927}) 2024-04-26T00:35:57.709 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.709+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2944671071}) 2024-04-26T00:35:57.711 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.713+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.711 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.713+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1411357719}) 2024-04-26T00:35:57.711 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.713+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2764823319}) 2024-04-26T00:35:57.711 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.713+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.711 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.713+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1411357719}) 2024-04-26T00:35:57.711 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.713+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2880932541}) 2024-04-26T00:35:57.713 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.713+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.713 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.713+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3718061787}) 2024-04-26T00:35:57.713 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.713+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1663663830}) 2024-04-26T00:35:57.713 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.713+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.713 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.713+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3718061787}) 2024-04-26T00:35:57.713 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.713+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2480395922}) 2024-04-26T00:35:57.715 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.717+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.715 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.717+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3141892388}) 2024-04-26T00:35:57.715 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.717+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2666214761}) 2024-04-26T00:35:57.715 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.717+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.715 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.717+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3141892388}) 2024-04-26T00:35:57.715 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.717+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4047787362}) 2024-04-26T00:35:57.717 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.717+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.717 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.717+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2354683604}) 2024-04-26T00:35:57.717 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.717+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1325382101}) 2024-04-26T00:35:57.717 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.717+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.717 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.717+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2354683604}) 2024-04-26T00:35:57.717 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.717+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3090569238}) 2024-04-26T00:35:57.719 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.721+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.719 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.721+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4107546194}) 2024-04-26T00:35:57.719 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.721+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1775676968}) 2024-04-26T00:35:57.719 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.721+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.719 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.721+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4107546194}) 2024-04-26T00:35:57.719 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.721+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=207393027}) 2024-04-26T00:35:57.721 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.721+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.721 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.721+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=972252809}) 2024-04-26T00:35:57.721 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.721+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1340372374}) 2024-04-26T00:35:57.721 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.721+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.721 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.721+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=972252809}) 2024-04-26T00:35:57.721 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.721+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2162476093}) 2024-04-26T00:35:57.723 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.725+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.723 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.725+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1296349718}) 2024-04-26T00:35:57.723 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.725+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=562903355}) 2024-04-26T00:35:57.723 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.725+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.723 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.725+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1296349718}) 2024-04-26T00:35:57.723 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.725+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2577850249}) 2024-04-26T00:35:57.724 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.725+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.725 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.725+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3422105805}) 2024-04-26T00:35:57.725 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.725+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3911030887}) 2024-04-26T00:35:57.725 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.725+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.725 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.725+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3422105805}) 2024-04-26T00:35:57.725 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.725+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3230548351}) 2024-04-26T00:35:57.727 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.729+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.727 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.729+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3708587192}) 2024-04-26T00:35:57.727 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.729+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2518859591}) 2024-04-26T00:35:57.727 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.729+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.727 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.729+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3708587192}) 2024-04-26T00:35:57.727 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.729+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=936923869}) 2024-04-26T00:35:57.729 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.729+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.729 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.729+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1743929829}) 2024-04-26T00:35:57.729 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.729+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4293651657}) 2024-04-26T00:35:57.729 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.729+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.729 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.729+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1743929829}) 2024-04-26T00:35:57.729 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.729+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3688131399}) 2024-04-26T00:35:57.731 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.733+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.731 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.733+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2534435248}) 2024-04-26T00:35:57.731 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.733+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=243170140}) 2024-04-26T00:35:57.731 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.733+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.731 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.733+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2534435248}) 2024-04-26T00:35:57.731 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.733+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2007134781}) 2024-04-26T00:35:57.733 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.733+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.733 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.733+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=390215786}) 2024-04-26T00:35:57.733 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.733+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4057275437}) 2024-04-26T00:35:57.733 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.733+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.733 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.733+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=390215786}) 2024-04-26T00:35:57.733 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.733+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3776031602}) 2024-04-26T00:35:57.734 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.737+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.735 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.737+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2161753577}) 2024-04-26T00:35:57.735 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.737+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3644278019}) 2024-04-26T00:35:57.735 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.737+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.735 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.737+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2161753577}) 2024-04-26T00:35:57.735 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.737+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1288132977}) 2024-04-26T00:35:57.737 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.737+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.737 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.737+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=19,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=39} crc {logm=2997900438,mdsmap=2840952586,mgr=1427520485,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=3326375313,mon_config_key=4061063118,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=3342131255,osd_snap=500002545}) 2024-04-26T00:35:57.738 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.737+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=18,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=38} crc {logm=911032574,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=3326375313,mon_config_key=4061063117,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=3342131254,osd_snap=3316538061}) 2024-04-26T00:35:57.738 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.737+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.738 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.737+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=19,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=39} crc {logm=2997900438,mdsmap=2840952586,mgr=1427520485,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=3326375313,mon_config_key=4061063118,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=3342131255,osd_snap=500002545}) 2024-04-26T00:35:57.738 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.737+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=16,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=42} crc {logm=1617260255,mdsmap=2840952586,mgr=3781331069,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=1021216742,mon_config_key=752439442,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=3342131254,osd_snap=3608767148}) 2024-04-26T00:35:57.739 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.741+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.739 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.741+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=76,osdmap=24} crc {osd_snap=504885564,osdmap=2255923461}) 2024-04-26T00:35:57.740 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.741+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=77,osdmap=23} crc {osd_snap=1022328171,osdmap=4182941842}) 2024-04-26T00:35:57.740 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.741+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.740 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.741+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=76,osdmap=24} crc {osd_snap=504885564,osdmap=2255923461}) 2024-04-26T00:35:57.740 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.741+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=70,osdmap=30} crc {osd_snap=2875192652,osdmap=2817973258}) 2024-04-26T00:35:57.742 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.741+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.742 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.741+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2870904100}) 2024-04-26T00:35:57.742 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.741+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1789360037}) 2024-04-26T00:35:57.742 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.741+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.742 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.741+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2870904100}) 2024-04-26T00:35:57.742 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.745+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1064507545}) 2024-04-26T00:35:57.744 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.745+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.744 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.745+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3403735221}) 2024-04-26T00:35:57.744 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.745+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1618039907}) 2024-04-26T00:35:57.744 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.745+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.744 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.745+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3403735221}) 2024-04-26T00:35:57.744 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.745+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2920707198}) 2024-04-26T00:35:57.748 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.749+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.748 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.749+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1498661931}) 2024-04-26T00:35:57.748 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.749+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2639624423}) 2024-04-26T00:35:57.748 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.749+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.748 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.749+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1498661931}) 2024-04-26T00:35:57.748 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.749+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=735061188}) 2024-04-26T00:35:57.760 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.761+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.760 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.761+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=62} crc {osdmap=3264904875}) 2024-04-26T00:35:57.760 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.761+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=62} crc {osdmap=3750258379}) 2024-04-26T00:35:57.761 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.761+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:35:57.761 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.761+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=62} crc {osdmap=3264904875}) 2024-04-26T00:35:57.761 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:35:57.761+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=62} crc {osdmap=2500718419}) 2024-04-26T00:35:58.460 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:35:58.461 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:35:58.545 INFO:tasks.rados.rados.0.smithi045.stdout:2715: done (3 left) 2024-04-26T00:35:58.545 INFO:tasks.rados.rados.0.smithi045.stdout:2716: rollback oid 27 current snap is 301 2024-04-26T00:35:58.545 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 27 to 294 2024-04-26T00:35:58.545 INFO:tasks.rados.rados.0.smithi045.stdout:2713: finishing copy_from to smithi04526579-23 2024-04-26T00:35:58.545 INFO:tasks.rados.rados.0.smithi045.stdout:2713: got expected ENOENT (src dne) 2024-04-26T00:35:58.545 INFO:tasks.rados.rados.0.smithi045.stdout:2713: finishing copy_from racing read to smithi04526579-23 2024-04-26T00:35:58.545 INFO:tasks.rados.rados.0.smithi045.stdout:2714: finishing copy_from racing read to smithi04526579-17 2024-04-26T00:35:58.545 INFO:tasks.rados.rados.0.smithi045.stdout:2712: finishing copy_from racing read to smithi04526579-4 2024-04-26T00:35:58.545 INFO:tasks.rados.rados.0.smithi045.stdout:2712: finishing copy_from to smithi04526579-4 2024-04-26T00:35:58.545 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 1971 (ObjNum 941 snap 291 seq_num 941) dirty exists 2024-04-26T00:35:58.546 INFO:tasks.rados.rados.0.smithi045.stdout:2714: finishing copy_from to smithi04526579-17 2024-04-26T00:35:58.546 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 1171 (ObjNum 921 snap 283 seq_num 921) dirty exists 2024-04-26T00:35:58.546 INFO:tasks.rados.rados.0.smithi045.stdout:2712: done (3 left) 2024-04-26T00:35:58.546 INFO:tasks.rados.rados.0.smithi045.stdout:2713: done (2 left) 2024-04-26T00:35:58.546 INFO:tasks.rados.rados.0.smithi045.stdout:2714: done (1 left) 2024-04-26T00:35:58.546 INFO:tasks.rados.rados.0.smithi045.stdout:2717: snap_remove snap 300 2024-04-26T00:35:58.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.605+0000 7fb855603640 10 monclient: get_monmap_and_config 2024-04-26T00:35:58.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.605+0000 7fb855603640 10 monclient: build_initial_monmap 2024-04-26T00:35:58.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.605+0000 7fb855603640 10 monclient: monmap: 2024-04-26T00:35:58.603 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:58.603 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:58.603 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:35:58.608122+0000 2024-04-26T00:35:58.603 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:35:58.608122+0000 2024-04-26T00:35:58.603 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:58.603 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:58.603 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:35:58.603 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:35:58.603 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:35:58.603 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:35:58.603 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:35:58.603 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:58.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.605+0000 7fb855603640 1 Processor -- start 2024-04-26T00:35:58.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.605+0000 7fb855603640 1 -- start start 2024-04-26T00:35:58.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.605+0000 7fb855603640 10 monclient: init 2024-04-26T00:35:58.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.605+0000 7fb855603640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:58.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.605+0000 7fb855603640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-04-26T00:35:58.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.605+0000 7fb855603640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb850136e10 0x7fb85009c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:58.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.605+0000 7fb855603640 10 monclient(hunting): picked mon.noname-a con 0x7fb850136e10 addr v2:172.21.15.37:3300/0 2024-04-26T00:35:58.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.605+0000 7fb855603640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb8501371f0 0x7fb850137600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:58.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.605+0000 7fb855603640 10 monclient(hunting): picked mon.noname-c con 0x7fb8501371f0 addr v2:172.21.15.45:3300/0 2024-04-26T00:35:58.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.605+0000 7fb855603640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb850135d90 addr v1:172.21.15.45:6789/0 2024-04-26T00:35:58.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.605+0000 7fb855603640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb85009b680 con 0x7fb850135d90 2024-04-26T00:35:58.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.605+0000 7fb855603640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:58.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.605+0000 7fb855603640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fb85009b8f0 con 0x7fb850136e10 2024-04-26T00:35:58.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.605+0000 7fb855603640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:58.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.605+0000 7fb855603640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fb85009bbd0 con 0x7fb8501371f0 2024-04-26T00:35:58.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.605+0000 7fb855603640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:58.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.605+0000 7fb855603640 10 monclient(hunting): authenticate will time out at 1939.913848s 2024-04-26T00:35:58.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.605+0000 7fb84effd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb850136e10 0x7fb85009c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:58.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.605+0000 7fb84effd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb850136e10 0x7fb85009c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:60854/0 (socket says 172.21.15.37:60854) 2024-04-26T00:35:58.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.605+0000 7fb84effd640 1 -- 172.21.15.37:0/3661830331 learned_addr learned my addr 172.21.15.37:0/3661830331 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:58.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.605+0000 7fb84effd640 10 monclient(hunting): get_auth_request con 0x7fb850136e10 auth_method 0 2024-04-26T00:35:58.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.605+0000 7fb84effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:58.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.605+0000 7fb84effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:58.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.605+0000 7fb84effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:58.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.605+0000 7fb84effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:58.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.605+0000 7fb84effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:58.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.605+0000 7fb84effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:58.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.605+0000 7fb84effd640 10 monclient(hunting): handle_auth_done global_id 144612 payload 274 2024-04-26T00:35:58.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.605+0000 7fb84effd640 1 -- 172.21.15.37:0/3661830331 >> v2:172.21.15.45:3300/0 conn(0x7fb8501371f0 msgr2=0x7fb850137600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:58.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.605+0000 7fb84effd640 1 --2- 172.21.15.37:0/3661830331 >> v2:172.21.15.45:3300/0 conn(0x7fb8501371f0 0x7fb850137600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:58.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.605+0000 7fb84effd640 1 -- 172.21.15.37:0/3661830331 >> v1:172.21.15.45:6789/0 conn(0x7fb850135d90 legacy=0x7fb850136220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:58.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.605+0000 7fb84effd640 10 monclient: _finish_hunting 0 2024-04-26T00:35:58.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.605+0000 7fb84effd640 1 monclient: found mon.noname-a 2024-04-26T00:35:58.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.605+0000 7fb84effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:58.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.605+0000 7fb84effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:35:58.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.605+0000 7fb84effd640 1 -- 172.21.15.37:0/3661830331 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb850137b40 con 0x7fb850136e10 2024-04-26T00:35:58.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.609+0000 7fb84effd640 1 -- 172.21.15.37:0/3661830331 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb850136e10 msgr2=0x7fb85009c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:58.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.609+0000 7fb84effd640 1 --2- 172.21.15.37:0/3661830331 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb850136e10 0x7fb85009c5b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb850068e60 tx=0x7fb84001c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebb7a70e9d96799d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:58.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.613+0000 7fb84dffb640 1 -- 172.21.15.37:0/3661830331 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8400b1070 con 0x7fb850136e10 2024-04-26T00:35:58.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.613+0000 7fb84dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:58.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.613+0000 7fb84dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:35:58.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.613+0000 7fb84dffb640 10 monclient: dump: 2024-04-26T00:35:58.612 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:58.612 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:58.612 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:58.612 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:58.612 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:58.612 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:58.612 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:58.612 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:58.612 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:58.612 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:58.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.613+0000 7fb84dffb640 10 monclient: _finish_auth 0 2024-04-26T00:35:58.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.613+0000 7fb84dffb640 10 monclient: _check_auth_tickets 2024-04-26T00:35:58.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.613+0000 7fb84dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:58.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.613+0000 7fb84dffb640 1 -- 172.21.15.37:0/3661830331 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8400ac620 con 0x7fb850136e10 2024-04-26T00:35:58.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.613+0000 7fb855603640 5 monclient: authenticate success, global_id 144612 2024-04-26T00:35:58.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.613+0000 7fb855603640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:35:58.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.613+0000 7fb84dffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:58.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.613+0000 7fb855603640 10 monclient: get_monmap_and_config success 2024-04-26T00:35:58.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.613+0000 7fb855603640 4 set_mon_vals no callback set 2024-04-26T00:35:58.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.613+0000 7fb84dffb640 1 -- 172.21.15.37:0/3661830331 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8400ac9f0 con 0x7fb850136e10 2024-04-26T00:35:58.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.613+0000 7fb84dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:58.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.613+0000 7fb84dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:58.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.613+0000 7fb84dffb640 10 monclient: dump: 2024-04-26T00:35:58.613 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:58.613 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:58.613 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:58.613 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:58.614 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:58.614 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:58.614 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:58.614 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:58.614 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:58.614 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:58.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.613+0000 7fb855603640 10 monclient: shutdown 2024-04-26T00:35:58.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.613+0000 7fb855603640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:58.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.613+0000 7fb855603640 1 -- 172.21.15.37:0/3661830331 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb850136e10 msgr2=0x7fb85009c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:58.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.613+0000 7fb855603640 1 --2- 172.21.15.37:0/3661830331 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb850136e10 0x7fb85009c5b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb850068e60 tx=0x7fb84001c3a0 comp rx=0 tx=0).stop 2024-04-26T00:35:58.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.613+0000 7fb855603640 1 -- 172.21.15.37:0/3661830331 shutdown_connections 2024-04-26T00:35:58.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.613+0000 7fb855603640 1 --2- 172.21.15.37:0/3661830331 >> v2:172.21.15.45:3300/0 conn(0x7fb8501371f0 0x7fb850137600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:58.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.613+0000 7fb855603640 1 --2- 172.21.15.37:0/3661830331 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb850136e10 0x7fb85009c5b0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:58.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.613+0000 7fb855603640 1 -- 172.21.15.37:0/3661830331 >> 172.21.15.37:0/3661830331 conn(0x7fb8500a64e0 msgr2=0x7fb8500a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:58.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.613+0000 7fb855603640 1 -- 172.21.15.37:0/3661830331 shutdown_connections 2024-04-26T00:35:58.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.613+0000 7fb855603640 1 -- 172.21.15.37:0/3661830331 wait complete. 2024-04-26T00:35:58.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.613+0000 7fb855603640 10 monclient: build_initial_monmap 2024-04-26T00:35:58.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.613+0000 7fb855603640 10 monclient: monmap: 2024-04-26T00:35:58.623 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:58.623 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:35:58.623 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:35:58.623 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:35:58.623 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:58.623 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:35:58.623 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:35:58.623 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:35:58.623 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:35:58.623 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:58.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.613+0000 7fb855603640 1 Processor -- start 2024-04-26T00:35:58.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.613+0000 7fb855603640 1 -- start start 2024-04-26T00:35:58.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.613+0000 7fb855603640 10 monclient: init 2024-04-26T00:35:58.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb855603640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:58.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb855603640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:35:58.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb855603640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb850135d90 0x7fb8501cf440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:58.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb855603640 10 monclient(hunting): picked mon.noname-b con 0x7fb850135d90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:35:58.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb855603640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb850136e10 0x7fb850134ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:58.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb855603640 10 monclient(hunting): picked mon.noname-a con 0x7fb850136e10 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:35:58.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb855603640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8501371f0 0x7fb85013abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:58.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb855603640 10 monclient(hunting): picked mon.noname-c con 0x7fb8501371f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:58.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb855603640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:58.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb855603640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb850141f10 con 0x7fb850136e10 2024-04-26T00:35:58.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb855603640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:58.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb855603640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb850141d90 con 0x7fb850135d90 2024-04-26T00:35:58.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb855603640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:58.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb855603640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb850142090 con 0x7fb8501371f0 2024-04-26T00:35:58.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb855603640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:58.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb855603640 10 monclient(hunting): authenticate will time out at 1939.924270s 2024-04-26T00:35:58.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb84e7fc640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb850136e10 0x7fb850134ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:58.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb84f7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8501371f0 0x7fb85013abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:58.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb84e7fc640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb850136e10 0x7fb850134ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:60870/0 (socket says 172.21.15.37:60870) 2024-04-26T00:35:58.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb84e7fc640 1 -- 172.21.15.37:0/1695573452 learned_addr learned my addr 172.21.15.37:0/1695573452 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:58.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb84e7fc640 10 monclient(hunting): get_auth_request con 0x7fb850136e10 auth_method 0 2024-04-26T00:35:58.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb84e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:58.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb84e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:58.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb84e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:58.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb84f7fe640 10 monclient(hunting): get_auth_request con 0x7fb8501371f0 auth_method 0 2024-04-26T00:35:58.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb84f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:58.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb84f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:58.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb84f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:58.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb84e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:58.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb84e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:58.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb84e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:58.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb84f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:58.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb84f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:58.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb84effd640 1 --2- 172.21.15.37:0/1695573452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb850135d90 0x7fb8501cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:58.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb84f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:58.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb84effd640 10 monclient(hunting): get_auth_request con 0x7fb850135d90 auth_method 0 2024-04-26T00:35:58.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb84effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:58.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb84effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:58.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb84effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:58.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb84e7fc640 10 monclient(hunting): handle_auth_done global_id 144615 payload 824 2024-04-26T00:35:58.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb84e7fc640 1 -- 172.21.15.37:0/1695573452 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8501371f0 msgr2=0x7fb85013abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:58.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb84e7fc640 1 --2- 172.21.15.37:0/1695573452 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8501371f0 0x7fb85013abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:58.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb84e7fc640 1 -- 172.21.15.37:0/1695573452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb850135d90 msgr2=0x7fb8501cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:58.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb84e7fc640 1 --2- 172.21.15.37:0/1695573452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb850135d90 0x7fb8501cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:58.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb84e7fc640 10 monclient: _finish_hunting 0 2024-04-26T00:35:58.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb84e7fc640 1 monclient: found mon.noname-a 2024-04-26T00:35:58.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb84e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:58.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb84e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:35:58.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb84e7fc640 1 -- 172.21.15.37:0/1695573452 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8500a34e0 con 0x7fb850136e10 2024-04-26T00:35:58.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb84f7fe640 1 --2- 172.21.15.37:0/1695573452 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8501371f0 0x7fb85013abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:35:58.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb84e7fc640 1 -- 172.21.15.37:0/1695573452 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb850136e10 msgr2=0x7fb850134ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:58.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb84e7fc640 1 --2- 172.21.15.37:0/1695573452 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb850136e10 0x7fb850134ec0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb84400b550 tx=0x7fb84400ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:58.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb8377fe640 1 -- 172.21.15.37:0/1695573452 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb844014070 con 0x7fb850136e10 2024-04-26T00:35:58.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb8377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:58.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb8377fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:35:58.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb8377fe640 10 monclient: dump: 2024-04-26T00:35:58.628 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:58.628 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:58.628 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:58.628 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:58.628 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:58.628 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:58.628 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:58.628 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:58.628 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:58.628 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:58.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb8377fe640 10 monclient: _finish_auth 0 2024-04-26T00:35:58.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb8377fe640 10 monclient: _check_auth_tickets 2024-04-26T00:35:58.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb8377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:58.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb855603640 5 monclient: authenticate success, global_id 144615 2024-04-26T00:35:58.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb855603640 10 monclient: _renew_subs 2024-04-26T00:35:58.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb8377fe640 1 -- 172.21.15.37:0/1695573452 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb844010aa0 con 0x7fb850136e10 2024-04-26T00:35:58.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb855603640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:58.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb855603640 1 -- 172.21.15.37:0/1695573452 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb85013e8a0 con 0x7fb850136e10 2024-04-26T00:35:58.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb8377fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:58.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb8377fe640 1 -- 172.21.15.37:0/1695573452 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb84400f3c0 con 0x7fb850136e10 2024-04-26T00:35:58.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb8377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:58.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb8377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:58.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb8377fe640 10 monclient: dump: 2024-04-26T00:35:58.630 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:58.630 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:58.630 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:58.630 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:58.630 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:58.630 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:58.630 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:58.630 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:58.630 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:58.630 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:58.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb84d7fa640 4 set_mon_vals no callback set 2024-04-26T00:35:58.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb855603640 10 monclient: _renew_subs 2024-04-26T00:35:58.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb855603640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:58.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.617+0000 7fb855603640 1 -- 172.21.15.37:0/1695573452 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb85013d480 con 0x7fb850136e10 2024-04-26T00:35:58.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.621+0000 7fb8377fe640 1 -- 172.21.15.37:0/1695573452 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb844017070 con 0x7fb850136e10 2024-04-26T00:35:58.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.621+0000 7fb855603640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:58.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.621+0000 7fb855603640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:58.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.621+0000 7fb855603640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:58.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.621+0000 7fb855603640 1 -- 172.21.15.37:0/1695573452 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb850066f40 con 0x7fb850136e10 2024-04-26T00:35:58.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.621+0000 7fb8377fe640 1 --2- 172.21.15.37:0/1695573452 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb83c035e80 0x7fb83c038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:58.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.621+0000 7fb8377fe640 1 -- 172.21.15.37:0/1695573452 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(629..629 src has 370..629) ==== 3971+0+0 (secure 0 0 0) 0x7fb8440488f0 con 0x7fb850136e10 2024-04-26T00:35:58.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.621+0000 7fb84effd640 1 --2- 172.21.15.37:0/1695573452 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb83c035e80 0x7fb83c038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:58.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.621+0000 7fb84effd640 10 monclient: get_auth_request con 0x7fb83c035e80 auth_method 0 2024-04-26T00:35:58.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.621+0000 7fb84effd640 1 --2- 172.21.15.37:0/1695573452 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb83c035e80 0x7fb83c038340 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fb8400ac000 tx=0x7fb8400a7000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:58.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.625+0000 7fb8377fe640 1 -- 172.21.15.37:0/1695573452 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fb8500a34e0 con 0x7fb850136e10 2024-04-26T00:35:58.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.625+0000 7fb8377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:58.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.625+0000 7fb8377fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:35:58.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.849+0000 7fb855603640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:35:58.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.849+0000 7fb855603640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:35:58.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.849+0000 7fb855603640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:58.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.849+0000 7fb855603640 1 -- 172.21.15.37:0/1695573452 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb8500008e0 con 0x7fb850136e10 2024-04-26T00:35:58.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.849+0000 7fb8377fe640 1 -- 172.21.15.37:0/1695573452 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fb844037e20 con 0x7fb850136e10 2024-04-26T00:35:58.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.849+0000 7fb8377fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:35:58.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.849+0000 7fb8377fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:35:58.850 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":120,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:35:58.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.853+0000 7fb855603640 1 -- 172.21.15.37:0/1695573452 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb83c035e80 msgr2=0x7fb83c038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:58.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.853+0000 7fb855603640 1 --2- 172.21.15.37:0/1695573452 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb83c035e80 0x7fb83c038340 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fb8400ac000 tx=0x7fb8400a7000 comp rx=0 tx=0).stop 2024-04-26T00:35:58.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.853+0000 7fb855603640 10 monclient: shutdown 2024-04-26T00:35:58.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.853+0000 7fb855603640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:58.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.853+0000 7fb855603640 1 -- 172.21.15.37:0/1695573452 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb850136e10 msgr2=0x7fb850134ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:58.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.853+0000 7fb855603640 1 --2- 172.21.15.37:0/1695573452 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb850136e10 0x7fb850134ec0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb84400b550 tx=0x7fb84400ba10 comp rx=0 tx=0).stop 2024-04-26T00:35:58.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.853+0000 7fb855603640 1 -- 172.21.15.37:0/1695573452 shutdown_connections 2024-04-26T00:35:58.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.853+0000 7fb855603640 1 --2- 172.21.15.37:0/1695573452 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb83c035e80 0x7fb83c038340 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:58.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.853+0000 7fb855603640 1 --2- 172.21.15.37:0/1695573452 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8501371f0 0x7fb85013abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:58.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.853+0000 7fb855603640 1 --2- 172.21.15.37:0/1695573452 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb850136e10 0x7fb850134ec0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:58.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.853+0000 7fb855603640 1 --2- 172.21.15.37:0/1695573452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb850135d90 0x7fb8501cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:58.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.853+0000 7fb855603640 1 -- 172.21.15.37:0/1695573452 >> 172.21.15.37:0/1695573452 conn(0x7fb8500a64e0 msgr2=0x7fb8500a7de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:58.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.853+0000 7fb855603640 1 -- 172.21.15.37:0/1695573452 shutdown_connections 2024-04-26T00:35:58.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.853+0000 7fb855603640 1 -- 172.21.15.37:0/1695573452 wait complete. 2024-04-26T00:35:58.869 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:35:58.869 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-26T00:35:58.869 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:35:58.869 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:35:58.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.997+0000 7f07bb35e640 10 monclient: get_monmap_and_config 2024-04-26T00:35:58.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.997+0000 7f07bb35e640 10 monclient: build_initial_monmap 2024-04-26T00:35:58.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.997+0000 7f07bb35e640 10 monclient: monmap: 2024-04-26T00:35:58.996 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:58.997 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:58.997 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:35:59.002153+0000 2024-04-26T00:35:58.997 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:35:59.002153+0000 2024-04-26T00:35:58.997 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:58.997 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:58.997 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:35:58.997 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:35:58.997 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:35:58.997 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:35:58.997 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:35:58.997 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:59.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.997+0000 7f07bb35e640 1 Processor -- start 2024-04-26T00:35:59.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.997+0000 7f07bb35e640 1 -- start start 2024-04-26T00:35:59.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:58.997+0000 7f07bb35e640 10 monclient: init 2024-04-26T00:35:59.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07bb35e640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:59.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07bb35e640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-04-26T00:35:59.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07bb35e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f07b409e220 0x7f07b409fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:59.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07bb35e640 10 monclient(hunting): picked mon.noname-b con 0x7f07b409e220 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:59.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07bb35e640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f07b40a14f0 0x7f07b40a0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:59.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07bb35e640 10 monclient(hunting): picked mon.noname-a con 0x7f07b40a14f0 addr v2:172.21.15.37:3300/0 2024-04-26T00:35:59.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07bb35e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f07b409e600 0x7f07b40a06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:59.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07bb35e640 10 monclient(hunting): picked mon.noname-c con 0x7f07b409e600 addr v2:172.21.15.45:3300/0 2024-04-26T00:35:59.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07bb35e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:59.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07bb35e640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f07b409cd10 con 0x7f07b40a14f0 2024-04-26T00:35:59.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07bb35e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:59.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07bb35e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f07b40a12b0 con 0x7f07b409e600 2024-04-26T00:35:59.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07bb35e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:59.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07bb35e640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f07b409da20 con 0x7f07b409e220 2024-04-26T00:35:59.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07bb35e640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:59.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07bb35e640 10 monclient(hunting): authenticate will time out at 1940.308428s 2024-04-26T00:35:59.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07b88f9640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f07b40a14f0 0x7f07b40a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:59.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07b90fa640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f07b409e220 0x7f07b409fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:59.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07b90fa640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f07b409e220 0x7f07b409fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:50932/0 (socket says 172.21.15.37:50932) 2024-04-26T00:35:59.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07b88f9640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f07b40a14f0 0x7f07b40a0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:60876/0 (socket says 172.21.15.37:60876) 2024-04-26T00:35:59.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07b90fa640 1 -- 172.21.15.37:0/3086203149 learned_addr learned my addr 172.21.15.37:0/3086203149 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:59.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07b90fa640 10 monclient(hunting): get_auth_request con 0x7f07b409e220 auth_method 0 2024-04-26T00:35:59.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07b90fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:59.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07b90fa640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:59.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07b90fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:59.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07b88f9640 10 monclient(hunting): get_auth_request con 0x7f07b40a14f0 auth_method 0 2024-04-26T00:35:59.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07b88f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:59.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07b88f9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:59.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07b88f9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:59.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07b88f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:59.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07b88f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:59.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07b88f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:59.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07b90fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:59.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07b90fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:59.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07b90fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:59.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07b98fb640 1 --2- 172.21.15.37:0/3086203149 >> v2:172.21.15.45:3300/0 conn(0x7f07b409e600 0x7f07b40a06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:59.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07b88f9640 10 monclient(hunting): handle_auth_done global_id 144618 payload 274 2024-04-26T00:35:59.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07b88f9640 1 -- 172.21.15.37:0/3086203149 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f07b409e220 msgr2=0x7f07b409fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:59.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07b88f9640 1 --2- 172.21.15.37:0/3086203149 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f07b409e220 0x7f07b409fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:59.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07b88f9640 1 -- 172.21.15.37:0/3086203149 >> v2:172.21.15.45:3300/0 conn(0x7f07b409e600 msgr2=0x7f07b40a06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:59.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07b88f9640 1 --2- 172.21.15.37:0/3086203149 >> v2:172.21.15.45:3300/0 conn(0x7f07b409e600 0x7f07b40a06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:59.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07b88f9640 10 monclient: _finish_hunting 0 2024-04-26T00:35:59.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07b88f9640 1 monclient: found mon.noname-a 2024-04-26T00:35:59.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07b88f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:59.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07b88f9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:35:59.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07b88f9640 1 -- 172.21.15.37:0/3086203149 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07b409dd00 con 0x7f07b40a14f0 2024-04-26T00:35:59.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07b98fb640 10 monclient: get_auth_request con 0x7f07b409e600 auth_method 0 2024-04-26T00:35:59.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.001+0000 7f07b98fb640 1 --2- 172.21.15.37:0/3086203149 >> v2:172.21.15.45:3300/0 conn(0x7f07b409e600 0x7f07b40a06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:35:59.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07b88f9640 1 -- 172.21.15.37:0/3086203149 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07b40a14f0 msgr2=0x7f07b40a0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:59.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07b88f9640 1 --2- 172.21.15.37:0/3086203149 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07b40a14f0 0x7f07b40a0190 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f07b406d140 tx=0x7f07b001c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e53f3f8ba1d51bd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:59.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07abfff640 1 -- 172.21.15.37:0/3086203149 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b00b0070 con 0x7f07b40a14f0 2024-04-26T00:35:59.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:59.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07abfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:35:59.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07abfff640 10 monclient: dump: 2024-04-26T00:35:59.008 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:59.008 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:59.008 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.008 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.008 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:59.008 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:59.008 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:59.008 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:59.008 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:59.008 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:59.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07abfff640 10 monclient: _finish_auth 0 2024-04-26T00:35:59.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07abfff640 10 monclient: _check_auth_tickets 2024-04-26T00:35:59.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:59.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07abfff640 1 -- 172.21.15.37:0/3086203149 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07b00ab400 con 0x7f07b40a14f0 2024-04-26T00:35:59.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07abfff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:59.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07abfff640 1 -- 172.21.15.37:0/3086203149 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b00ab7d0 con 0x7f07b40a14f0 2024-04-26T00:35:59.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07bb35e640 5 monclient: authenticate success, global_id 144618 2024-04-26T00:35:59.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07bb35e640 10 monclient: get_monmap_and_config success 2024-04-26T00:35:59.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07bb35e640 4 set_mon_vals no callback set 2024-04-26T00:35:59.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:59.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07abfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:59.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07abfff640 10 monclient: dump: 2024-04-26T00:35:59.010 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:59.010 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:59.010 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.010 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.010 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:59.010 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:59.010 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:59.010 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:59.010 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:59.010 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:59.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07bb35e640 10 monclient: shutdown 2024-04-26T00:35:59.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07bb35e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:59.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07bb35e640 1 -- 172.21.15.37:0/3086203149 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07b40a14f0 msgr2=0x7f07b40a0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:59.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07bb35e640 1 --2- 172.21.15.37:0/3086203149 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07b40a14f0 0x7f07b40a0190 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f07b406d140 tx=0x7f07b001c320 comp rx=0 tx=0).stop 2024-04-26T00:35:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07bb35e640 1 -- 172.21.15.37:0/3086203149 shutdown_connections 2024-04-26T00:35:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07bb35e640 1 --2- 172.21.15.37:0/3086203149 >> v2:172.21.15.45:3300/0 conn(0x7f07b409e600 0x7f07b40a06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07bb35e640 1 --2- 172.21.15.37:0/3086203149 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07b40a14f0 0x7f07b40a0190 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07bb35e640 1 --2- 172.21.15.37:0/3086203149 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f07b409e220 0x7f07b409fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07bb35e640 1 -- 172.21.15.37:0/3086203149 >> 172.21.15.37:0/3086203149 conn(0x7f07b40a1d50 msgr2=0x7f07b40a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07bb35e640 1 -- 172.21.15.37:0/3086203149 shutdown_connections 2024-04-26T00:35:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07bb35e640 1 -- 172.21.15.37:0/3086203149 wait complete. 2024-04-26T00:35:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07bb35e640 10 monclient: build_initial_monmap 2024-04-26T00:35:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07bb35e640 10 monclient: monmap: 2024-04-26T00:35:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:35:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:35:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:35:59.011 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:59.012 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:35:59.012 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:35:59.012 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:35:59.012 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:35:59.012 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:59.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07bb35e640 1 Processor -- start 2024-04-26T00:35:59.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07bb35e640 1 -- start start 2024-04-26T00:35:59.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07bb35e640 10 monclient: init 2024-04-26T00:35:59.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07bb35e640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:59.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07bb35e640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:35:59.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07bb35e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f07b409e220 0x7f07b4136fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:59.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07bb35e640 10 monclient(hunting): picked mon.noname-c con 0x7f07b409e220 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:59.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07bb35e640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07b409e600 0x7f07b41cbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:59.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07bb35e640 10 monclient(hunting): picked mon.noname-a con 0x7f07b409e600 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:35:59.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07bb35e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07b40a14f0 0x7f07b41cc110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:59.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07bb35e640 10 monclient(hunting): picked mon.noname-b con 0x7f07b40a14f0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:35:59.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07bb35e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:59.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07bb35e640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f07b413c660 con 0x7f07b409e600 2024-04-26T00:35:59.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07bb35e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:59.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07bb35e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f07b413c4e0 con 0x7f07b40a14f0 2024-04-26T00:35:59.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07bb35e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:59.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07bb35e640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f07b413c7e0 con 0x7f07b409e220 2024-04-26T00:35:59.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07bb35e640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:59.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07bb35e640 10 monclient(hunting): authenticate will time out at 1940.315686s 2024-04-26T00:35:59.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07b90fa640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f07b409e220 0x7f07b4136fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:59.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07b88f9640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07b409e600 0x7f07b41cbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:59.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07b90fa640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f07b409e220 0x7f07b4136fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:50948/0 (socket says 172.21.15.37:50948) 2024-04-26T00:35:59.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07b88f9640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07b409e600 0x7f07b41cbbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:60892/0 (socket says 172.21.15.37:60892) 2024-04-26T00:35:59.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07b90fa640 1 -- 172.21.15.37:0/3633987446 learned_addr learned my addr 172.21.15.37:0/3633987446 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:59.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07b90fa640 10 monclient(hunting): get_auth_request con 0x7f07b409e220 auth_method 0 2024-04-26T00:35:59.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07b90fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:59.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07b90fa640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:59.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07b90fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:59.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07b88f9640 10 monclient(hunting): get_auth_request con 0x7f07b409e600 auth_method 0 2024-04-26T00:35:59.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07b88f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:59.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07b88f9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:59.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07b88f9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:59.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07b90fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:59.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07b90fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:59.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07b90fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:59.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07b88f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:59.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07b88f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:59.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.005+0000 7f07b88f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:59.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07b98fb640 1 --2- 172.21.15.37:0/3633987446 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07b40a14f0 0x7f07b41cc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:59.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07b90fa640 10 monclient(hunting): handle_auth_done global_id 184217 payload 824 2024-04-26T00:35:59.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07b90fa640 1 -- 172.21.15.37:0/3633987446 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07b40a14f0 msgr2=0x7f07b41cc110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:59.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07b90fa640 1 --2- 172.21.15.37:0/3633987446 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07b40a14f0 0x7f07b41cc110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:59.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07b90fa640 1 -- 172.21.15.37:0/3633987446 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07b409e600 msgr2=0x7f07b41cbbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:59.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07b90fa640 1 --2- 172.21.15.37:0/3633987446 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07b409e600 0x7f07b41cbbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:59.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07b90fa640 10 monclient: _finish_hunting 0 2024-04-26T00:35:59.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07b90fa640 1 monclient: found mon.noname-c 2024-04-26T00:35:59.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07b90fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:59.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07b90fa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:35:59.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07b90fa640 1 -- 172.21.15.37:0/3633987446 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07b409c970 con 0x7f07b409e220 2024-04-26T00:35:59.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07b98fb640 10 monclient: get_auth_request con 0x7f07b40a14f0 auth_method 0 2024-04-26T00:35:59.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07b98fb640 1 --2- 172.21.15.37:0/3633987446 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07b40a14f0 0x7f07b41cc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:35:59.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07b88f9640 1 --2- 172.21.15.37:0/3633987446 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07b409e600 0x7f07b41cbbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:35:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07b90fa640 1 -- 172.21.15.37:0/3633987446 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f07b409e220 msgr2=0x7f07b4136fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07b90fa640 1 --2- 172.21.15.37:0/3633987446 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f07b409e220 0x7f07b4136fb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f07a400b340 tx=0x7f07a4009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07a9ffb640 1 -- 172.21.15.37:0/3633987446 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a4009d70 con 0x7f07b409e220 2024-04-26T00:35:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07a9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07a9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:35:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07a9ffb640 10 monclient: dump: 2024-04-26T00:35:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:59.020 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.021 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.021 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:59.021 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:59.021 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:59.021 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:59.021 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:59.021 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:59.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07a9ffb640 10 monclient: _finish_auth 0 2024-04-26T00:35:59.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07a9ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:35:59.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:59.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07a9ffb640 1 -- 172.21.15.37:0/3633987446 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07a40173f0 con 0x7f07b409e220 2024-04-26T00:35:59.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07a9ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07bb35e640 5 monclient: authenticate success, global_id 184217 2024-04-26T00:35:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07bb35e640 10 monclient: _renew_subs 2024-04-26T00:35:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07a9ffb640 1 -- 172.21.15.37:0/3633987446 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a40177a0 con 0x7f07b409e220 2024-04-26T00:35:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07bb35e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:35:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07bb35e640 1 -- 172.21.15.37:0/3633987446 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07b40725c0 con 0x7f07b409e220 2024-04-26T00:35:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07a9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07ab7fe640 4 set_mon_vals no callback set 2024-04-26T00:35:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07a9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:35:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07a9ffb640 10 monclient: dump: 2024-04-26T00:35:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.022 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:59.023 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:59.023 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:59.023 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:59.023 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:59.023 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:59.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07bb35e640 10 monclient: _renew_subs 2024-04-26T00:35:59.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07bb35e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:35:59.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07bb35e640 1 -- 172.21.15.37:0/3633987446 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f07b4126110 con 0x7f07b409e220 2024-04-26T00:35:59.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07bb35e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:59.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07bb35e640 1 --2- 172.21.15.37:0/3633987446 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07b4125440 0x7f07b402d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:59.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07bb35e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:59.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07bb35e640 1 -- 172.21.15.37:0/3633987446 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f07b41cfb50 con 0x7f07b4125440 2024-04-26T00:35:59.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07b88f9640 1 --2- 172.21.15.37:0/3633987446 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07b4125440 0x7f07b402d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:59.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07b88f9640 10 monclient: get_auth_request con 0x7f07b4125440 auth_method 0 2024-04-26T00:35:59.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07b88f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:59.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07b88f9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:59.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07b88f9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:59.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07b90fa640 1 --2- 172.21.15.37:0/3633987446 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f07b409e220 0x7f07b4136fb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f07a400b340 tx=0x7f07a4009120 comp rx=0 tx=0).queue_received will delay after 1.71409e+09 on 0x7f07a4017940 mgrmap(e 5) 2024-04-26T00:35:59.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07b88f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:59.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07b88f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:59.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07b88f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:59.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07a9ffb640 1 -- 172.21.15.37:0/3633987446 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f07a4017940 con 0x7f07b409e220 2024-04-26T00:35:59.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.009+0000 7f07b88f9640 10 monclient(hunting): handle_auth_done global_id 144624 payload 824 2024-04-26T00:35:59.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.013+0000 7f07b88f9640 1 -- 172.21.15.37:0/3633987446 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f07b4126560 con 0x7f07b4125440 2024-04-26T00:35:59.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.013+0000 7f07a9ffb640 1 --2- 172.21.15.37:0/3633987446 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0798035ed0 0x7f0798038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:59.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.013+0000 7f07b98fb640 1 --2- 172.21.15.37:0/3633987446 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0798035ed0 0x7f0798038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:59.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.013+0000 7f07b98fb640 10 monclient: get_auth_request con 0x7f0798035ed0 auth_method 0 2024-04-26T00:35:59.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.013+0000 7f07b88f9640 1 -- 172.21.15.37:0/3633987446 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07b4125440 msgr2=0x7f07b402d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:59.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.013+0000 7f07b88f9640 1 --2- 172.21.15.37:0/3633987446 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07b4125440 0x7f07b402d180 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f07b4076500 tx=0x7f07b001ab60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:59.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.013+0000 7f07a9ffb640 1 -- 172.21.15.37:0/3633987446 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b00bb070 con 0x7f07b4125440 2024-04-26T00:35:59.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.013+0000 7f07a9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:59.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.013+0000 7f07a9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:59.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.013+0000 7f07a9ffb640 10 monclient: dump: 2024-04-26T00:35:59.025 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:59.025 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:59.025 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.025 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.025 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:59.025 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:59.025 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:59.025 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:59.025 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:59.025 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:59.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.013+0000 7f07a9ffb640 1 -- 172.21.15.37:0/3633987446 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f07b00b6850 con 0x7f07b4125440 2024-04-26T00:35:59.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.013+0000 7f07a9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:59.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.013+0000 7f07a9ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:35:59.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.013+0000 7f07a9ffb640 1 -- 172.21.15.37:0/3633987446 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07b4125440 msgr2=0x7f07b402d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:59.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.013+0000 7f07a9ffb640 1 --2- 172.21.15.37:0/3633987446 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07b4125440 0x7f07b402d180 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f07b4076500 tx=0x7f07b001ab60 comp rx=0 tx=0).stop 2024-04-26T00:35:59.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.013+0000 7f07a9ffb640 1 -- 172.21.15.37:0/3633987446 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07b4125440 msgr2=0x7f07b402d180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:35:59.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.013+0000 7f07a9ffb640 1 --2- 172.21.15.37:0/3633987446 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07b4125440 0x7f07b402d180 secure :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f07b4076500 tx=0x7f07b001ab60 comp rx=0 tx=0).stop 2024-04-26T00:35:59.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.013+0000 7f07b98fb640 1 --2- 172.21.15.37:0/3633987446 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0798035ed0 0x7f0798038390 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f07b409ea50 tx=0x7f07ac007440 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:59.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.021+0000 7f07bb35e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:35:59.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.021+0000 7f07bb35e640 1 --2- 172.21.15.37:0/3633987446 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07b41374f0 0x7f07b41cc650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:59.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.021+0000 7f07bb35e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:59.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.021+0000 7f07bb35e640 1 -- 172.21.15.37:0/3633987446 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f07b4139380 con 0x7f07b41374f0 2024-04-26T00:35:59.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.021+0000 7f07b88f9640 1 --2- 172.21.15.37:0/3633987446 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07b41374f0 0x7f07b41cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:59.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.021+0000 7f07b88f9640 10 monclient: get_auth_request con 0x7f07b41374f0 auth_method 0 2024-04-26T00:35:59.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.021+0000 7f07b88f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:59.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.021+0000 7f07b88f9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:59.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.021+0000 7f07b88f9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:59.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.021+0000 7f07b88f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:59.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.021+0000 7f07b88f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:59.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.021+0000 7f07b88f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:59.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.021+0000 7f07b88f9640 10 monclient(hunting): handle_auth_done global_id 144627 payload 824 2024-04-26T00:35:59.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.021+0000 7f07b88f9640 1 -- 172.21.15.37:0/3633987446 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f07b406ba50 con 0x7f07b41374f0 2024-04-26T00:35:59.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.021+0000 7f07b88f9640 1 -- 172.21.15.37:0/3633987446 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07b41374f0 msgr2=0x7f07b41cc650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:59.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.021+0000 7f07b88f9640 1 --2- 172.21.15.37:0/3633987446 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07b41374f0 0x7f07b41cc650 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f07b00097c0 tx=0x7f07b00190c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:59.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.021+0000 7f07a9ffb640 1 -- 172.21.15.37:0/3633987446 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b00bb050 con 0x7f07b41374f0 2024-04-26T00:35:59.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.021+0000 7f07a9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:59.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.021+0000 7f07a9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:59.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.021+0000 7f07a9ffb640 10 monclient: dump: 2024-04-26T00:35:59.028 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:59.028 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:59.028 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.028 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.029 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:59.029 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:59.029 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:59.029 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:59.029 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:59.029 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:59.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.021+0000 7f07a9ffb640 1 -- 172.21.15.37:0/3633987446 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4890 (secure 0 0 0) 0x7f07b00044e0 con 0x7f07b41374f0 2024-04-26T00:35:59.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.021+0000 7f07a9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:35:59.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.021+0000 7f07a9ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:35:59.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.021+0000 7f07a9ffb640 1 -- 172.21.15.37:0/3633987446 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07b41374f0 msgr2=0x7f07b41cc650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:59.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.021+0000 7f07a9ffb640 1 --2- 172.21.15.37:0/3633987446 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07b41374f0 0x7f07b41cc650 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f07b00097c0 tx=0x7f07b00190c0 comp rx=0 tx=0).stop 2024-04-26T00:35:59.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.025+0000 7f07a9ffb640 1 -- 172.21.15.37:0/3633987446 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07b41374f0 msgr2=0x7f07b41cc650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:35:59.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.025+0000 7f07a9ffb640 1 --2- 172.21.15.37:0/3633987446 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07b41374f0 0x7f07b41cc650 secure :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f07b00097c0 tx=0x7f07b00190c0 comp rx=0 tx=0).stop 2024-04-26T00:35:59.030 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:35:59.030 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:35:59.030 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:35:59.030 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:35:59.030 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 120, 2024-04-26T00:35:59.030 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 245564, 2024-04-26T00:35:59.030 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:35:59.030 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:35:59.030 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:35:59.030 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:35:59.030 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:59.030 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 4, 2024-04-26T00:35:59.031 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:35:59.031 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:35:59.031 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:35:59.031 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:59.031 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:59.031 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:59.031 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:59.031 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:59.031 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:59.031 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:59.031 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:59.031 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:59.031 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:59.031 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:59.031 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:59.032 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:35:59.032 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:35:59.032 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:59.032 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:59.032 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:59.032 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:59.032 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:59.032 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:59.032 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:59.032 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:59.032 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:59.032 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:59.032 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:59.032 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:59.032 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:59.033 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:35:59.033 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:35:59.033 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:35:59.033 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:35:59.033 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:35:59.033 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:35:59.033 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:35:59.033 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:35:59.033 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:35:59.033 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:35:59.033 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:35:59.033 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:35:59.033 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:35:59.033 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:35:59.033 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:35:59.034 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:35:59.034 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:35:59.034 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:59.034 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:59.034 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:59.034 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:59.034 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:59.034 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:59.034 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:59.034 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:59.034 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:59.034 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:59.034 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:59.034 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:59.034 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:35:59.035 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:59.035 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:35:59.035 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:59.035 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:35:59.035 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:35:59.035 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:59.035 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:59.035 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:59.035 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:59.035 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:35:59.035 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:59.036 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:59.036 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:59.036 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:59.036 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:35:59.036 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:59.036 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:59.036 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:59.036 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:59.036 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:35:59.036 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:35:59.036 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:59.036 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:59.036 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:59.036 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:59.037 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:59.037 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:35:59.037 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:35:59.037 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:59.037 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:59.037 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:59.037 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:59.037 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:35:59.037 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:59.037 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:59.037 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:59.037 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:59.037 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:35:59.037 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:59.038 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:59.038 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:59.038 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:59.038 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:35:59.038 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:35:59.038 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:59.038 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:59.038 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:59.038 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:59.038 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:59.038 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:35:59.038 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:35:59.038 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:59.038 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:59.039 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:59.039 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:59.039 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:35:59.039 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:59.039 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:59.039 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:59.039 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:59.039 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:35:59.039 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:59.039 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:59.039 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:59.039 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:59.039 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:35:59.039 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:35:59.040 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:59.040 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:59.040 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:59.040 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:59.040 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:59.040 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:59.040 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:35:59.040 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:35:59.040 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:59.040 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:59.040 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:59.040 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:35:59.040 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:59.040 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:59.040 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:35:59.040 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:59.041 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:59.041 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:59.041 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:35:59.041 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:59.041 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:59.041 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:35:59.041 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:59.041 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:59.041 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:59.041 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:35:59.041 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:59.041 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:59.041 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:35:59.041 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:59.042 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:59.042 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:59.042 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:35:59.042 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:59.042 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:59.042 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:59.042 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:35:59.042 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:35:59.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.025+0000 7f07bb35e640 1 -- 172.21.15.37:0/3633987446 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0798035ed0 msgr2=0x7f0798038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:59.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.025+0000 7f07bb35e640 1 --2- 172.21.15.37:0/3633987446 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0798035ed0 0x7f0798038390 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f07b409ea50 tx=0x7f07ac007440 comp rx=0 tx=0).stop 2024-04-26T00:35:59.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.025+0000 7f07bb35e640 10 monclient: shutdown 2024-04-26T00:35:59.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.025+0000 7f07bb35e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:59.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.025+0000 7f07bb35e640 1 -- 172.21.15.37:0/3633987446 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f07b409e220 msgr2=0x7f07b4136fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:59.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.025+0000 7f07bb35e640 1 --2- 172.21.15.37:0/3633987446 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f07b409e220 0x7f07b4136fb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f07a400b340 tx=0x7f07a4009120 comp rx=0 tx=0).stop 2024-04-26T00:35:59.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.025+0000 7f07b98fb640 1 -- 172.21.15.37:0/3633987446 reap_dead start 2024-04-26T00:35:59.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.025+0000 7f07a9ffb640 1 -- 172.21.15.37:0/3633987446 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(629..629 src has 370..629) ==== 3971+0+0 (unknown 0 0 0) 0x7f07a4048db0 con 0x7f07b409e220 2024-04-26T00:35:59.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.025+0000 7f07b98fb640 1 -- 172.21.15.37:0/3633987446 reap_dead start 2024-04-26T00:35:59.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.025+0000 7f07bb35e640 1 -- 172.21.15.37:0/3633987446 shutdown_connections 2024-04-26T00:35:59.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.025+0000 7f07bb35e640 1 -- 172.21.15.37:0/3633987446 >> 172.21.15.37:0/3633987446 conn(0x7f07b40a1d50 msgr2=0x7f07b40a39e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:59.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.025+0000 7f07bb35e640 1 -- 172.21.15.37:0/3633987446 shutdown_connections 2024-04-26T00:35:59.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.025+0000 7f07bb35e640 1 -- 172.21.15.37:0/3633987446 wait complete. 2024-04-26T00:35:59.045 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:35:59.045 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:35:59.045 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:35:59.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f73a0891640 10 monclient: get_monmap_and_config 2024-04-26T00:35:59.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f73a0891640 10 monclient: build_initial_monmap 2024-04-26T00:35:59.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f73a0891640 10 monclient: monmap: 2024-04-26T00:35:59.162 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:59.162 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:59.163 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:35:59.168116+0000 2024-04-26T00:35:59.163 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:35:59.168116+0000 2024-04-26T00:35:59.163 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:59.163 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:59.163 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:35:59.163 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:35:59.163 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:35:59.163 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:35:59.163 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:35:59.163 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:59.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f73a0891640 1 Processor -- start 2024-04-26T00:35:59.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f73a0891640 1 -- start start 2024-04-26T00:35:59.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f73a0891640 10 monclient: init 2024-04-26T00:35:59.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f73a0891640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:59.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f73a0891640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-04-26T00:35:59.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f73a0891640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f739c131dc0 0x7f739c1321a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:59.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f73a0891640 10 monclient(hunting): picked mon.noname-c con 0x7f739c131dc0 addr v2:172.21.15.45:3300/0 2024-04-26T00:35:59.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f73a0891640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f739c072710 0x7f739c134c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:59.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f73a0891640 10 monclient(hunting): picked mon.noname-b con 0x7f739c072710 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:59.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f73a0891640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f739c135170 0x7f739c135600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:59.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f73a0891640 10 monclient(hunting): picked mon.noname-a con 0x7f739c135170 addr v2:172.21.15.37:3300/0 2024-04-26T00:35:59.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f73a0891640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:59.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f73a0891640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f739c072cc0 con 0x7f739c135170 2024-04-26T00:35:59.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f73a0891640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:59.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f73a0891640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f739c071da0 con 0x7f739c131dc0 2024-04-26T00:35:59.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f73a0891640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:59.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f73a0891640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f739c071a50 con 0x7f739c072710 2024-04-26T00:35:59.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f73a0891640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:59.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f73a0891640 10 monclient(hunting): authenticate will time out at 1940.473776s 2024-04-26T00:35:59.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f7399d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f739c072710 0x7f739c134c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:59.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f7399d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f739c072710 0x7f739c134c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:50960/0 (socket says 172.21.15.37:50960) 2024-04-26T00:35:59.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f7399d9b640 1 -- 172.21.15.37:0/1622403994 learned_addr learned my addr 172.21.15.37:0/1622403994 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:59.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f7399d9b640 10 monclient(hunting): get_auth_request con 0x7f739c072710 auth_method 0 2024-04-26T00:35:59.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f7399d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:59.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f7399d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:59.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f7399d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:59.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f739ad9d640 1 --2- 172.21.15.37:0/1622403994 >> v2:172.21.15.37:3300/0 conn(0x7f739c135170 0x7f739c135600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:59.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f7399d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:59.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f7399d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:59.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f7399d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:59.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f739ad9d640 10 monclient(hunting): get_auth_request con 0x7f739c135170 auth_method 0 2024-04-26T00:35:59.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f739ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:59.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f739ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:59.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f739ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:59.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f739a59c640 1 --2- 172.21.15.37:0/1622403994 >> v2:172.21.15.45:3300/0 conn(0x7f739c131dc0 0x7f739c1321a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:59.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f739ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:59.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f739ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:59.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f739ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:59.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f739a59c640 10 monclient(hunting): get_auth_request con 0x7f739c131dc0 auth_method 0 2024-04-26T00:35:59.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f739a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:59.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f739a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:59.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f739a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:59.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f7399d9b640 10 monclient(hunting): handle_auth_done global_id 184220 payload 274 2024-04-26T00:35:59.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f7399d9b640 1 -- 172.21.15.37:0/1622403994 >> v2:172.21.15.45:3300/0 conn(0x7f739c131dc0 msgr2=0x7f739c1321a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:59.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f7399d9b640 1 --2- 172.21.15.37:0/1622403994 >> v2:172.21.15.45:3300/0 conn(0x7f739c131dc0 0x7f739c1321a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:59.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f7399d9b640 1 -- 172.21.15.37:0/1622403994 >> v2:172.21.15.37:3300/0 conn(0x7f739c135170 msgr2=0x7f739c135600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:59.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f7399d9b640 1 --2- 172.21.15.37:0/1622403994 >> v2:172.21.15.37:3300/0 conn(0x7f739c135170 0x7f739c135600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:59.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f7399d9b640 10 monclient: _finish_hunting 0 2024-04-26T00:35:59.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f7399d9b640 1 monclient: found mon.noname-b 2024-04-26T00:35:59.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f7399d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:59.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f7399d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:35:59.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.165+0000 7f7399d9b640 1 -- 172.21.15.37:0/1622403994 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f739c0a3360 con 0x7f739c072710 2024-04-26T00:35:59.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.169+0000 7f7399d9b640 1 -- 172.21.15.37:0/1622403994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f739c072710 msgr2=0x7f739c134c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:59.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.169+0000 7f7399d9b640 1 --2- 172.21.15.37:0/1622403994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f739c072710 0x7f739c134c30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f739c0a4ea0 tx=0x7f739001c3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57001cd2b4c9add5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:59.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.169+0000 7f739959a640 1 -- 172.21.15.37:0/1622403994 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73900b0070 con 0x7f739c072710 2024-04-26T00:35:59.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.169+0000 7f739959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:59.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.169+0000 7f739959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:35:59.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.169+0000 7f739959a640 10 monclient: dump: 2024-04-26T00:35:59.177 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:59.177 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:59.177 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.177 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.178 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:59.178 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:59.178 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:59.178 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:59.178 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:59.178 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:59.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.169+0000 7f739959a640 10 monclient: _finish_auth 0 2024-04-26T00:35:59.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.169+0000 7f739959a640 10 monclient: _check_auth_tickets 2024-04-26T00:35:59.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.169+0000 7f739959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:59.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.169+0000 7f73a0891640 5 monclient: authenticate success, global_id 184220 2024-04-26T00:35:59.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.169+0000 7f739959a640 1 -- 172.21.15.37:0/1622403994 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73900ab600 con 0x7f739c072710 2024-04-26T00:35:59.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.169+0000 7f739959a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:59.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.169+0000 7f73a0891640 10 monclient: get_monmap_and_config success 2024-04-26T00:35:59.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.169+0000 7f73a0891640 4 set_mon_vals no callback set 2024-04-26T00:35:59.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.169+0000 7f739959a640 1 -- 172.21.15.37:0/1622403994 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73900ab9d0 con 0x7f739c072710 2024-04-26T00:35:59.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.169+0000 7f739959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:59.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.169+0000 7f739959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:35:59.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.169+0000 7f739959a640 10 monclient: dump: 2024-04-26T00:35:59.179 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:59.179 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:59.179 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.179 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.179 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:59.179 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:59.179 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:59.179 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:59.179 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:59.179 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:59.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.169+0000 7f73a0891640 10 monclient: shutdown 2024-04-26T00:35:59.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.169+0000 7f73a0891640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:59.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.169+0000 7f73a0891640 1 -- 172.21.15.37:0/1622403994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f739c072710 msgr2=0x7f739c134c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:59.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.169+0000 7f73a0891640 1 --2- 172.21.15.37:0/1622403994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f739c072710 0x7f739c134c30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f739c0a4ea0 tx=0x7f739001c3b0 comp rx=0 tx=0).stop 2024-04-26T00:35:59.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.169+0000 7f73a0891640 1 -- 172.21.15.37:0/1622403994 shutdown_connections 2024-04-26T00:35:59.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.169+0000 7f73a0891640 1 --2- 172.21.15.37:0/1622403994 >> v2:172.21.15.37:3300/0 conn(0x7f739c135170 0x7f739c135600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:59.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.169+0000 7f73a0891640 1 --2- 172.21.15.37:0/1622403994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f739c072710 0x7f739c134c30 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:59.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.169+0000 7f73a0891640 1 --2- 172.21.15.37:0/1622403994 >> v2:172.21.15.45:3300/0 conn(0x7f739c131dc0 0x7f739c1321a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:59.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.169+0000 7f73a0891640 1 -- 172.21.15.37:0/1622403994 >> 172.21.15.37:0/1622403994 conn(0x7f739c0a7300 msgr2=0x7f739c0a76e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:59.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.169+0000 7f73a0891640 1 -- 172.21.15.37:0/1622403994 shutdown_connections 2024-04-26T00:35:59.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.169+0000 7f73a0891640 1 -- 172.21.15.37:0/1622403994 wait complete. 2024-04-26T00:35:59.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f73a0891640 10 monclient: build_initial_monmap 2024-04-26T00:35:59.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f73a0891640 10 monclient: monmap: 2024-04-26T00:35:59.180 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:59.180 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:35:59.180 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:35:59.181 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:35:59.181 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:59.181 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:35:59.181 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:35:59.181 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:35:59.181 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:35:59.181 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:59.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f73a0891640 1 Processor -- start 2024-04-26T00:35:59.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f73a0891640 1 -- start start 2024-04-26T00:35:59.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f73a0891640 10 monclient: init 2024-04-26T00:35:59.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f73a0891640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:59.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f73a0891640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:35:59.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f73a0891640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f739c131dc0 0x7f739c130e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:59.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f73a0891640 10 monclient(hunting): picked mon.noname-c con 0x7f739c131dc0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:59.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f73a0891640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f739c135170 0x7f739c131380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:59.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f73a0891640 10 monclient(hunting): picked mon.noname-b con 0x7f739c135170 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:35:59.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f73a0891640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f739c1cc9a0 0x7f739c1cce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:59.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f73a0891640 10 monclient(hunting): picked mon.noname-a con 0x7f739c1cc9a0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:35:59.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f73a0891640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:59.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f73a0891640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f739c132860 con 0x7f739c1cc9a0 2024-04-26T00:35:59.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f73a0891640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:59.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f73a0891640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f739c1326e0 con 0x7f739c135170 2024-04-26T00:35:59.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f73a0891640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:59.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f73a0891640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f739c13e780 con 0x7f739c131dc0 2024-04-26T00:35:59.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f73a0891640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:59.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f73a0891640 10 monclient(hunting): authenticate will time out at 1940.481353s 2024-04-26T00:35:59.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f739a59c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f739c131dc0 0x7f739c130e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:59.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f739ad9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f739c1cc9a0 0x7f739c1cce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:59.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f739a59c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f739c131dc0 0x7f739c130e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:50964/0 (socket says 172.21.15.37:50964) 2024-04-26T00:35:59.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f739a59c640 1 -- 172.21.15.37:0/607306063 learned_addr learned my addr 172.21.15.37:0/607306063 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:59.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f739ad9d640 10 monclient(hunting): get_auth_request con 0x7f739c1cc9a0 auth_method 0 2024-04-26T00:35:59.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f739ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:59.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f739ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:59.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f739ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:59.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f739a59c640 10 monclient(hunting): get_auth_request con 0x7f739c131dc0 auth_method 0 2024-04-26T00:35:59.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f739a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:59.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f739a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:59.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f739a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:59.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f739ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:59.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f739ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:59.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f739ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:59.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f739a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:59.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f739a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:59.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f739a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:59.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f7399d9b640 1 --2- 172.21.15.37:0/607306063 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f739c135170 0x7f739c131380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:59.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f7399d9b640 10 monclient(hunting): get_auth_request con 0x7f739c135170 auth_method 0 2024-04-26T00:35:59.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f7399d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:59.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f7399d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:59.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f7399d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:59.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f739ad9d640 10 monclient(hunting): handle_auth_done global_id 144633 payload 824 2024-04-26T00:35:59.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f739ad9d640 1 -- 172.21.15.37:0/607306063 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f739c131dc0 msgr2=0x7f739c130e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:59.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f739ad9d640 1 --2- 172.21.15.37:0/607306063 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f739c131dc0 0x7f739c130e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:59.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f739ad9d640 1 -- 172.21.15.37:0/607306063 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f739c135170 msgr2=0x7f739c131380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:59.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f739ad9d640 1 --2- 172.21.15.37:0/607306063 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f739c135170 0x7f739c131380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:59.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f739ad9d640 10 monclient: _finish_hunting 0 2024-04-26T00:35:59.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f739ad9d640 1 monclient: found mon.noname-a 2024-04-26T00:35:59.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f739ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:59.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f739ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:35:59.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.173+0000 7f739ad9d640 1 -- 172.21.15.37:0/607306063 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f739c13a8b0 con 0x7f739c1cc9a0 2024-04-26T00:35:59.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f739ad9d640 1 -- 172.21.15.37:0/607306063 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f739c1cc9a0 msgr2=0x7f739c1cce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:59.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f739ad9d640 1 --2- 172.21.15.37:0/607306063 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f739c1cc9a0 0x7f739c1cce50 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f739400efc0 tx=0x7f739400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:59.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f738affd640 1 -- 172.21.15.37:0/607306063 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7394018020 con 0x7f739c1cc9a0 2024-04-26T00:35:59.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f738affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:59.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f738affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:35:59.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f738affd640 10 monclient: dump: 2024-04-26T00:35:59.185 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:59.185 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:59.185 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.185 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.185 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:59.185 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:59.185 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:59.185 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:59.185 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:59.186 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:59.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f738affd640 10 monclient: _finish_auth 0 2024-04-26T00:35:59.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f738affd640 10 monclient: _check_auth_tickets 2024-04-26T00:35:59.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f738affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:59.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f738affd640 1 -- 172.21.15.37:0/607306063 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73940146a0 con 0x7f739c1cc9a0 2024-04-26T00:35:59.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f73a0891640 5 monclient: authenticate success, global_id 144633 2024-04-26T00:35:59.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f73a0891640 10 monclient: _renew_subs 2024-04-26T00:35:59.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f73a0891640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:59.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f73a0891640 1 -- 172.21.15.37:0/607306063 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f739c138ea0 con 0x7f739c1cc9a0 2024-04-26T00:35:59.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f738affd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:59.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f738affd640 1 -- 172.21.15.37:0/607306063 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7394014a90 con 0x7f739c1cc9a0 2024-04-26T00:35:59.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f738affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:59.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f738affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:59.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f738affd640 10 monclient: dump: 2024-04-26T00:35:59.187 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:59.187 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:59.187 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.187 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.187 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:59.187 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:59.187 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:59.187 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:59.187 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:59.187 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:59.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f7398d99640 4 set_mon_vals no callback set 2024-04-26T00:35:59.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f73a0891640 10 monclient: _renew_subs 2024-04-26T00:35:59.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f73a0891640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:59.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f73a0891640 1 -- 172.21.15.37:0/607306063 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f739c1385c0 con 0x7f739c1cc9a0 2024-04-26T00:35:59.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f73a0891640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:59.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f73a0891640 1 --2- 172.21.15.37:0/607306063 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f739c072710 0x7f739c1cd390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:59.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f73a0891640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:59.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f73a0891640 1 -- 172.21.15.37:0/607306063 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f739c04f510 con 0x7f739c072710 2024-04-26T00:35:59.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f739a59c640 1 --2- 172.21.15.37:0/607306063 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f739c072710 0x7f739c1cd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:59.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f739a59c640 10 monclient: get_auth_request con 0x7f739c072710 auth_method 0 2024-04-26T00:35:59.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f739a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:59.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f739a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:59.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f739a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:59.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f739a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:59.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f739a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:59.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f739a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:59.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f738affd640 1 -- 172.21.15.37:0/607306063 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f739401b020 con 0x7f739c1cc9a0 2024-04-26T00:35:59.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f739a59c640 10 monclient(hunting): handle_auth_done global_id 184226 payload 824 2024-04-26T00:35:59.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f739a59c640 1 -- 172.21.15.37:0/607306063 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f739c0008e0 con 0x7f739c072710 2024-04-26T00:35:59.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f738affd640 1 --2- 172.21.15.37:0/607306063 >> v2:172.21.15.45:6812/2379918623 conn(0x7f737c035e80 0x7f737c038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:59.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f738affd640 1 -- 172.21.15.37:0/607306063 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(629..629 src has 370..629) ==== 3971+0+0 (secure 0 0 0) 0x7f7394049830 con 0x7f739c1cc9a0 2024-04-26T00:35:59.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f7399d9b640 1 --2- 172.21.15.37:0/607306063 >> v2:172.21.15.45:6812/2379918623 conn(0x7f737c035e80 0x7f737c038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:59.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f7399d9b640 10 monclient: get_auth_request con 0x7f737c035e80 auth_method 0 2024-04-26T00:35:59.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f739a59c640 1 -- 172.21.15.37:0/607306063 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f739c072710 msgr2=0x7f739c1cd390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:59.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f739a59c640 1 --2- 172.21.15.37:0/607306063 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f739c072710 0x7f739c1cd390 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f738c010890 tx=0x7f738c010d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:59.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f738affd640 1 -- 172.21.15.37:0/607306063 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f738c018070 con 0x7f739c072710 2024-04-26T00:35:59.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f738affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:59.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f738affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:35:59.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.177+0000 7f738affd640 10 monclient: dump: 2024-04-26T00:35:59.191 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:59.192 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:59.192 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.192 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.192 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:59.192 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:59.192 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:59.192 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:59.192 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:59.192 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:59.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.181+0000 7f738affd640 1 -- 172.21.15.37:0/607306063 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f738c00ecb0 con 0x7f739c072710 2024-04-26T00:35:59.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.181+0000 7f738affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:59.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.181+0000 7f738affd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:35:59.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.181+0000 7f738affd640 1 -- 172.21.15.37:0/607306063 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f739c072710 msgr2=0x7f739c1cd390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:59.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.181+0000 7f738affd640 1 --2- 172.21.15.37:0/607306063 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f739c072710 0x7f739c1cd390 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f738c010890 tx=0x7f738c010d50 comp rx=0 tx=0).stop 2024-04-26T00:35:59.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.181+0000 7f738affd640 1 -- 172.21.15.37:0/607306063 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f739c072710 msgr2=0x7f739c1cd390 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:35:59.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.181+0000 7f738affd640 1 --2- 172.21.15.37:0/607306063 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f739c072710 0x7f739c1cd390 secure :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f738c010890 tx=0x7f738c010d50 comp rx=0 tx=0).stop 2024-04-26T00:35:59.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.181+0000 7f7399d9b640 1 --2- 172.21.15.37:0/607306063 >> v2:172.21.15.45:6812/2379918623 conn(0x7f737c035e80 0x7f737c038340 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f739c070610 tx=0x7f7390008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:59.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.185+0000 7f73a0891640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:35:59.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.185+0000 7f73a0891640 1 --2- 172.21.15.37:0/607306063 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f739c09b270 0x7f739c09b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:59.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.185+0000 7f73a0891640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:59.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.185+0000 7f73a0891640 1 -- 172.21.15.37:0/607306063 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f739c1318c0 con 0x7f739c09b270 2024-04-26T00:35:59.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.185+0000 7f739a59c640 1 --2- 172.21.15.37:0/607306063 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f739c09b270 0x7f739c09b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:59.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.185+0000 7f739a59c640 10 monclient: get_auth_request con 0x7f739c09b270 auth_method 0 2024-04-26T00:35:59.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.185+0000 7f739a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:59.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.185+0000 7f739a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:59.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.185+0000 7f739a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:59.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.185+0000 7f739a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:59.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.185+0000 7f739a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:59.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.185+0000 7f739a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:59.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.185+0000 7f739a59c640 10 monclient(hunting): handle_auth_done global_id 184229 payload 824 2024-04-26T00:35:59.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.185+0000 7f739a59c640 1 -- 172.21.15.37:0/607306063 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f739c13b030 con 0x7f739c09b270 2024-04-26T00:35:59.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.189+0000 7f739a59c640 1 -- 172.21.15.37:0/607306063 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f739c09b270 msgr2=0x7f739c09b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:59.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.189+0000 7f739a59c640 1 --2- 172.21.15.37:0/607306063 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f739c09b270 0x7f739c09b680 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f738c010890 tx=0x7f738c0046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:59.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.189+0000 7f738affd640 1 -- 172.21.15.37:0/607306063 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f738c014850 con 0x7f739c09b270 2024-04-26T00:35:59.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.189+0000 7f738affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:59.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.189+0000 7f738affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:35:59.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.189+0000 7f738affd640 10 monclient: dump: 2024-04-26T00:35:59.194 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:59.194 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:59.194 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.194 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.194 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:59.194 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:59.194 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:59.194 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:59.195 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:59.195 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:59.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.189+0000 7f738affd640 1 -- 172.21.15.37:0/607306063 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4717 (secure 0 0 0) 0x7f738c014850 con 0x7f739c09b270 2024-04-26T00:35:59.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.189+0000 7f738affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:35:59.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.189+0000 7f738affd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:35:59.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.189+0000 7f738affd640 1 -- 172.21.15.37:0/607306063 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f739c09b270 msgr2=0x7f739c09b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:59.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.189+0000 7f738affd640 1 --2- 172.21.15.37:0/607306063 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f739c09b270 0x7f739c09b680 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f738c010890 tx=0x7f738c0046d0 comp rx=0 tx=0).stop 2024-04-26T00:35:59.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.189+0000 7f738affd640 1 -- 172.21.15.37:0/607306063 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f739c09b270 msgr2=0x7f739c09b680 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:35:59.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.189+0000 7f738affd640 1 --2- 172.21.15.37:0/607306063 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f739c09b270 0x7f739c09b680 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f738c010890 tx=0x7f738c0046d0 comp rx=0 tx=0).stop 2024-04-26T00:35:59.195 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:35:59.196 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:35:59.196 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:35:59.196 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:35:59.196 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 120, 2024-04-26T00:35:59.196 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 58335, 2024-04-26T00:35:59.196 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:35:59.196 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:35:59.196 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:35:59.196 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:35:59.196 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:59.196 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 4, 2024-04-26T00:35:59.196 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:35:59.196 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:35:59.196 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:35:59.196 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:59.197 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:59.197 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:59.197 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:59.197 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:59.197 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:59.197 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:59.197 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:59.197 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:59.197 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:59.197 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:59.197 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:59.197 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:35:59.197 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:35:59.197 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:59.198 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:59.198 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:59.198 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:59.198 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:59.198 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:59.198 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:59.198 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:59.198 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:59.198 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:59.198 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:59.198 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:59.198 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:59.198 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:35:59.198 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:35:59.198 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:35:59.199 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:35:59.199 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:35:59.199 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:35:59.199 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:35:59.199 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:35:59.199 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:35:59.199 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:35:59.199 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:35:59.199 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:35:59.199 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:35:59.199 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:35:59.199 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:35:59.199 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:35:59.199 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:35:59.200 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:59.200 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:59.200 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:59.200 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:59.200 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:59.200 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:59.200 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:59.200 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:59.200 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:59.200 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:59.200 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:59.200 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:59.200 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:35:59.200 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:59.200 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:35:59.201 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:59.201 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:35:59.201 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:35:59.201 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:59.201 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:59.201 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:59.201 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:59.201 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:35:59.201 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:59.201 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:59.201 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:59.201 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:59.201 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:35:59.201 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:59.201 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:59.202 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:59.202 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:59.202 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:35:59.202 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:35:59.202 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:59.202 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:59.202 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:59.202 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:59.202 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:59.202 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:35:59.202 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:35:59.202 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:59.202 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:59.202 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:59.202 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:59.203 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:35:59.203 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:59.203 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:59.203 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:59.203 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:59.203 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:35:59.203 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:59.203 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:59.203 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:59.203 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:59.203 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:35:59.203 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:35:59.203 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:59.203 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:59.203 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:59.204 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:59.204 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:59.204 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:35:59.204 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:35:59.204 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:59.204 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:59.204 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:59.204 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:59.204 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:35:59.204 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:59.204 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:59.204 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:59.204 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:59.204 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:35:59.204 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:59.205 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:59.205 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:59.205 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:59.205 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:35:59.205 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:35:59.205 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:59.205 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:59.205 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:59.205 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:59.205 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:59.205 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:59.205 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:35:59.205 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:35:59.205 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:59.206 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:59.206 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:59.206 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:35:59.206 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:59.206 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:59.206 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:35:59.206 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:59.206 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:59.206 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:59.206 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:35:59.206 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:59.206 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:59.206 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:35:59.206 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:59.207 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:59.207 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:59.207 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:35:59.207 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:59.207 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:59.207 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:59.207 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:35:59.207 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:35:59.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.189+0000 7f73a0891640 1 -- 172.21.15.37:0/607306063 >> v2:172.21.15.45:6812/2379918623 conn(0x7f737c035e80 msgr2=0x7f737c038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:59.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.189+0000 7f73a0891640 1 --2- 172.21.15.37:0/607306063 >> v2:172.21.15.45:6812/2379918623 conn(0x7f737c035e80 0x7f737c038340 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f739c070610 tx=0x7f7390008040 comp rx=0 tx=0).stop 2024-04-26T00:35:59.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.189+0000 7f73a0891640 10 monclient: shutdown 2024-04-26T00:35:59.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.189+0000 7f73a0891640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:59.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.189+0000 7f73a0891640 1 -- 172.21.15.37:0/607306063 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f739c1cc9a0 msgr2=0x7f739c1cce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:59.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.189+0000 7f73a0891640 1 --2- 172.21.15.37:0/607306063 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f739c1cc9a0 0x7f739c1cce50 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f739400efc0 tx=0x7f739400c490 comp rx=0 tx=0).stop 2024-04-26T00:35:59.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.189+0000 7f739ad9d640 1 -- 172.21.15.37:0/607306063 reap_dead start 2024-04-26T00:35:59.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.189+0000 7f739ad9d640 1 -- 172.21.15.37:0/607306063 reap_dead start 2024-04-26T00:35:59.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.189+0000 7f73a0891640 1 -- 172.21.15.37:0/607306063 shutdown_connections 2024-04-26T00:35:59.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.189+0000 7f73a0891640 1 -- 172.21.15.37:0/607306063 >> 172.21.15.37:0/607306063 conn(0x7f739c0a7300 msgr2=0x7f739c133640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:59.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.189+0000 7f73a0891640 1 -- 172.21.15.37:0/607306063 shutdown_connections 2024-04-26T00:35:59.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.189+0000 7f73a0891640 1 -- 172.21.15.37:0/607306063 wait complete. 2024-04-26T00:35:59.210 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:35:59.210 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:35:59.210 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:35:59.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf4d77d640 10 monclient: get_monmap_and_config 2024-04-26T00:35:59.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf4d77d640 10 monclient: build_initial_monmap 2024-04-26T00:35:59.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf4d77d640 10 monclient: monmap: 2024-04-26T00:35:59.330 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:59.330 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:59.330 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:35:59.331999+0000 2024-04-26T00:35:59.330 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:35:59.331999+0000 2024-04-26T00:35:59.330 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:59.330 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:59.330 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:35:59.331 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:35:59.331 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:35:59.331 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:35:59.331 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:35:59.331 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:59.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf4d77d640 1 Processor -- start 2024-04-26T00:35:59.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf4d77d640 1 -- start start 2024-04-26T00:35:59.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf4d77d640 10 monclient: init 2024-04-26T00:35:59.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf4d77d640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:59.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf4d77d640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-04-26T00:35:59.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf4d77d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcf480a3d90 addr v1:172.21.15.45:6789/0 2024-04-26T00:35:59.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf4d77d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fcf480a4170 0x7fcf480a45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:59.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf4d77d640 10 monclient(hunting): picked mon.noname-c con 0x7fcf480a4170 addr v2:172.21.15.45:3300/0 2024-04-26T00:35:59.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf4d77d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcf4813c3e0 0x7fcf4813e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:59.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf4d77d640 10 monclient(hunting): picked mon.noname-b con 0x7fcf4813c3e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:59.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf4d77d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf480a5150 con 0x7fcf480a3d90 2024-04-26T00:35:59.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf4d77d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:59.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf4d77d640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fcf4809c650 con 0x7fcf480a4170 2024-04-26T00:35:59.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf4d77d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:59.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf4d77d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fcf4809b280 con 0x7fcf4813c3e0 2024-04-26T00:35:59.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf4d77d640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:59.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf4d77d640 10 monclient(hunting): authenticate will time out at 1940.637686s 2024-04-26T00:35:59.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf477fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcf4813c3e0 0x7fcf4813e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:59.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf477fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcf4813c3e0 0x7fcf4813e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:50978/0 (socket says 172.21.15.37:50978) 2024-04-26T00:35:59.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf477fe640 1 -- 172.21.15.37:0/2535170163 learned_addr learned my addr 172.21.15.37:0/2535170163 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:59.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf477fe640 10 monclient(hunting): get_auth_request con 0x7fcf4813c3e0 auth_method 0 2024-04-26T00:35:59.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:59.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf477fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:59.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf477fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:59.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:59.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:59.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:59.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf467fc640 1 --2- 172.21.15.37:0/2535170163 >> v2:172.21.15.45:3300/0 conn(0x7fcf480a4170 0x7fcf480a45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:59.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf467fc640 10 monclient(hunting): get_auth_request con 0x7fcf480a4170 auth_method 0 2024-04-26T00:35:59.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:59.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf467fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:59.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf467fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:59.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf477fe640 10 monclient(hunting): handle_auth_done global_id 184232 payload 274 2024-04-26T00:35:59.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf477fe640 1 -- 172.21.15.37:0/2535170163 >> v2:172.21.15.45:3300/0 conn(0x7fcf480a4170 msgr2=0x7fcf480a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:59.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf477fe640 1 --2- 172.21.15.37:0/2535170163 >> v2:172.21.15.45:3300/0 conn(0x7fcf480a4170 0x7fcf480a45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:59.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf477fe640 1 -- 172.21.15.37:0/2535170163 >> v1:172.21.15.45:6789/0 conn(0x7fcf480a3d90 legacy=0x7fcf4809bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:59.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf477fe640 10 monclient: _finish_hunting 0 2024-04-26T00:35:59.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf477fe640 1 monclient: found mon.noname-b 2024-04-26T00:35:59.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:59.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf477fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:35:59.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.329+0000 7fcf477fe640 1 -- 172.21.15.37:0/2535170163 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf4809b560 con 0x7fcf4813c3e0 2024-04-26T00:35:59.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.333+0000 7fcf477fe640 1 -- 172.21.15.37:0/2535170163 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcf4813c3e0 msgr2=0x7fcf4813e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:59.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.333+0000 7fcf477fe640 1 --2- 172.21.15.37:0/2535170163 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcf4813c3e0 0x7fcf4813e7d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcf40009e50 tx=0x7fcf4001c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c1f5f524eda3ada server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:59.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.333+0000 7fcf45ffb640 1 -- 172.21.15.37:0/2535170163 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf400a5640 con 0x7fcf4813c3e0 2024-04-26T00:35:59.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.333+0000 7fcf45ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:59.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.333+0000 7fcf45ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:35:59.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.333+0000 7fcf45ffb640 10 monclient: dump: 2024-04-26T00:35:59.341 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:59.341 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:59.341 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.342 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.342 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:59.342 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:59.342 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:59.342 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:59.342 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:59.342 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:59.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.333+0000 7fcf45ffb640 10 monclient: _finish_auth 0 2024-04-26T00:35:59.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.333+0000 7fcf45ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:35:59.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.333+0000 7fcf45ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:59.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.333+0000 7fcf45ffb640 1 -- 172.21.15.37:0/2535170163 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf400a57e0 con 0x7fcf4813c3e0 2024-04-26T00:35:59.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.333+0000 7fcf45ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:59.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.333+0000 7fcf45ffb640 1 -- 172.21.15.37:0/2535170163 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf400a5bb0 con 0x7fcf4813c3e0 2024-04-26T00:35:59.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.333+0000 7fcf4d77d640 5 monclient: authenticate success, global_id 184232 2024-04-26T00:35:59.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.333+0000 7fcf4d77d640 10 monclient: get_monmap_and_config success 2024-04-26T00:35:59.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.333+0000 7fcf4d77d640 4 set_mon_vals no callback set 2024-04-26T00:35:59.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.333+0000 7fcf45ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:59.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.333+0000 7fcf45ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:35:59.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.333+0000 7fcf45ffb640 10 monclient: dump: 2024-04-26T00:35:59.343 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:59.343 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:59.343 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.343 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.343 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:59.344 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:59.344 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:59.344 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:59.344 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:59.344 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:59.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.333+0000 7fcf4d77d640 10 monclient: shutdown 2024-04-26T00:35:59.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.333+0000 7fcf4d77d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:59.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.333+0000 7fcf4d77d640 1 -- 172.21.15.37:0/2535170163 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcf4813c3e0 msgr2=0x7fcf4813e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:59.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.333+0000 7fcf4d77d640 1 --2- 172.21.15.37:0/2535170163 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcf4813c3e0 0x7fcf4813e7d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcf40009e50 tx=0x7fcf4001c8b0 comp rx=0 tx=0).stop 2024-04-26T00:35:59.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf4d77d640 1 -- 172.21.15.37:0/2535170163 shutdown_connections 2024-04-26T00:35:59.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf4d77d640 1 --2- 172.21.15.37:0/2535170163 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcf4813c3e0 0x7fcf4813e7d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:59.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf4d77d640 1 --2- 172.21.15.37:0/2535170163 >> v2:172.21.15.45:3300/0 conn(0x7fcf480a4170 0x7fcf480a45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:59.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf4d77d640 1 -- 172.21.15.37:0/2535170163 >> 172.21.15.37:0/2535170163 conn(0x7fcf480aa0a0 msgr2=0x7fcf480a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:59.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf4d77d640 1 -- 172.21.15.37:0/2535170163 shutdown_connections 2024-04-26T00:35:59.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf4d77d640 1 -- 172.21.15.37:0/2535170163 wait complete. 2024-04-26T00:35:59.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf4d77d640 10 monclient: build_initial_monmap 2024-04-26T00:35:59.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf4d77d640 10 monclient: monmap: 2024-04-26T00:35:59.345 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:59.345 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:35:59.345 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:35:59.345 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:35:59.345 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:59.345 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:35:59.345 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:35:59.345 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:35:59.345 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:35:59.346 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:59.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf4d77d640 1 Processor -- start 2024-04-26T00:35:59.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf4d77d640 1 -- start start 2024-04-26T00:35:59.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf4d77d640 10 monclient: init 2024-04-26T00:35:59.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf4d77d640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:59.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf4d77d640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:35:59.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf4d77d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcf480a3d90 0x7fcf481ced80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:59.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf4d77d640 10 monclient(hunting): picked mon.noname-a con 0x7fcf480a3d90 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:35:59.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf4d77d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf480a4170 0x7fcf481364c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:59.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf4d77d640 10 monclient(hunting): picked mon.noname-b con 0x7fcf480a4170 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:35:59.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf4d77d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcf4813c3e0 0x7fcf48139590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:59.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf4d77d640 10 monclient(hunting): picked mon.noname-c con 0x7fcf4813c3e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:59.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf4d77d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:59.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf4d77d640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fcf481418c0 con 0x7fcf480a3d90 2024-04-26T00:35:59.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf4d77d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:59.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf4d77d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcf48141740 con 0x7fcf480a4170 2024-04-26T00:35:59.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf4d77d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:59.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf4d77d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fcf48141a40 con 0x7fcf4813c3e0 2024-04-26T00:35:59.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf4d77d640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:59.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf4d77d640 10 monclient(hunting): authenticate will time out at 1940.645789s 2024-04-26T00:35:59.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf46ffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcf480a3d90 0x7fcf481ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:59.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf477fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcf4813c3e0 0x7fcf48139590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:59.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf46ffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcf480a3d90 0x7fcf481ced80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:60942/0 (socket says 172.21.15.37:60942) 2024-04-26T00:35:59.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf46ffd640 1 -- 172.21.15.37:0/1071100575 learned_addr learned my addr 172.21.15.37:0/1071100575 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:59.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf46ffd640 10 monclient(hunting): get_auth_request con 0x7fcf480a3d90 auth_method 0 2024-04-26T00:35:59.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf46ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:59.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf46ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:59.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:59.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf477fe640 10 monclient(hunting): get_auth_request con 0x7fcf4813c3e0 auth_method 0 2024-04-26T00:35:59.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:59.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf477fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:59.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf477fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:59.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf46ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:59.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf46ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:59.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf46ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:59.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:59.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:59.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:59.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf467fc640 1 --2- 172.21.15.37:0/1071100575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf480a4170 0x7fcf481364c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:59.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf467fc640 10 monclient(hunting): get_auth_request con 0x7fcf480a4170 auth_method 0 2024-04-26T00:35:59.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:59.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf467fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:59.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf467fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:59.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf477fe640 10 monclient(hunting): handle_auth_done global_id 184235 payload 824 2024-04-26T00:35:59.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf477fe640 1 -- 172.21.15.37:0/1071100575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf480a4170 msgr2=0x7fcf481364c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:59.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf477fe640 1 --2- 172.21.15.37:0/1071100575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf480a4170 0x7fcf481364c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:59.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf477fe640 1 -- 172.21.15.37:0/1071100575 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcf480a3d90 msgr2=0x7fcf481ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:59.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf477fe640 1 --2- 172.21.15.37:0/1071100575 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcf480a3d90 0x7fcf481ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:59.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf477fe640 10 monclient: _finish_hunting 0 2024-04-26T00:35:59.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf477fe640 1 monclient: found mon.noname-c 2024-04-26T00:35:59.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:59.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf477fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:35:59.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf477fe640 1 -- 172.21.15.37:0/1071100575 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf4811f9e0 con 0x7fcf4813c3e0 2024-04-26T00:35:59.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.337+0000 7fcf467fc640 1 --2- 172.21.15.37:0/1071100575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf480a4170 0x7fcf481364c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:35:59.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf477fe640 1 -- 172.21.15.37:0/1071100575 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcf4813c3e0 msgr2=0x7fcf48139590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:59.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf477fe640 1 --2- 172.21.15.37:0/1071100575 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcf4813c3e0 0x7fcf48139590 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcf400ae000 tx=0x7fcf40019650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:59.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf3f7fe640 1 -- 172.21.15.37:0/1071100575 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf400bc030 con 0x7fcf4813c3e0 2024-04-26T00:35:59.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf3f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:59.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf3f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:35:59.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf3f7fe640 10 monclient: dump: 2024-04-26T00:35:59.351 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:59.351 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:59.351 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.351 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.351 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:59.351 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:59.351 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:59.351 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:59.351 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:59.351 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:59.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf3f7fe640 10 monclient: _finish_auth 0 2024-04-26T00:35:59.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf3f7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:35:59.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:59.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf3f7fe640 1 -- 172.21.15.37:0/1071100575 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf400b8680 con 0x7fcf4813c3e0 2024-04-26T00:35:59.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf4d77d640 5 monclient: authenticate success, global_id 184235 2024-04-26T00:35:59.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf4d77d640 10 monclient: _renew_subs 2024-04-26T00:35:59.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf4d77d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:35:59.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf4d77d640 1 -- 172.21.15.37:0/1071100575 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf4809ada0 con 0x7fcf4813c3e0 2024-04-26T00:35:59.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf3f7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:59.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf3f7fe640 1 -- 172.21.15.37:0/1071100575 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf400b8a30 con 0x7fcf4813c3e0 2024-04-26T00:35:59.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf3f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:59.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf3f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:35:59.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf3f7fe640 10 monclient: dump: 2024-04-26T00:35:59.352 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:59.352 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:59.353 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.353 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.353 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:59.353 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:59.353 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:59.353 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:59.353 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:59.353 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:59.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf4d77d640 10 monclient: _renew_subs 2024-04-26T00:35:59.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf4d77d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:35:59.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf4d77d640 1 -- 172.21.15.37:0/1071100575 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf4813b1b0 con 0x7fcf4813c3e0 2024-04-26T00:35:59.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf44ff9640 4 set_mon_vals no callback set 2024-04-26T00:35:59.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf4d77d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:59.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf4d77d640 1 --2- 172.21.15.37:0/1071100575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf48136a00 0x7fcf48139ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:59.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf4d77d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:59.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf4d77d640 1 -- 172.21.15.37:0/1071100575 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcf4809c4c0 con 0x7fcf48136a00 2024-04-26T00:35:59.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf46ffd640 1 --2- 172.21.15.37:0/1071100575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf48136a00 0x7fcf48139ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:59.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf46ffd640 10 monclient: get_auth_request con 0x7fcf48136a00 auth_method 0 2024-04-26T00:35:59.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf46ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:59.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf46ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:59.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:59.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf46ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:59.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf46ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:59.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf46ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:59.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf3f7fe640 1 -- 172.21.15.37:0/1071100575 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fcf400bf070 con 0x7fcf4813c3e0 2024-04-26T00:35:59.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf46ffd640 10 monclient(hunting): handle_auth_done global_id 194119 payload 824 2024-04-26T00:35:59.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf46ffd640 1 -- 172.21.15.37:0/1071100575 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcf48120c90 con 0x7fcf48136a00 2024-04-26T00:35:59.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf3f7fe640 1 --2- 172.21.15.37:0/1071100575 >> v2:172.21.15.45:6812/2379918623 conn(0x7fcf28035eb0 0x7fcf28038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:59.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf3f7fe640 1 -- 172.21.15.37:0/1071100575 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(629..629 src has 370..629) ==== 3971+0+0 (secure 0 0 0) 0x7fcf400e37d0 con 0x7fcf4813c3e0 2024-04-26T00:35:59.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf467fc640 1 --2- 172.21.15.37:0/1071100575 >> v2:172.21.15.45:6812/2379918623 conn(0x7fcf28035eb0 0x7fcf28038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:59.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.341+0000 7fcf467fc640 10 monclient: get_auth_request con 0x7fcf28035eb0 auth_method 0 2024-04-26T00:35:59.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.345+0000 7fcf46ffd640 1 -- 172.21.15.37:0/1071100575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf48136a00 msgr2=0x7fcf48139ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:59.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.345+0000 7fcf46ffd640 1 --2- 172.21.15.37:0/1071100575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf48136a00 0x7fcf48139ad0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fcf3000f7a0 tx=0x7fcf3000fc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:59.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.345+0000 7fcf3f7fe640 1 -- 172.21.15.37:0/1071100575 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf30018030 con 0x7fcf48136a00 2024-04-26T00:35:59.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.345+0000 7fcf3f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:59.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.345+0000 7fcf3f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:35:59.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.345+0000 7fcf3f7fe640 10 monclient: dump: 2024-04-26T00:35:59.358 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:59.358 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:59.359 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.359 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.359 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:59.359 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:59.359 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:59.359 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:59.359 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:59.359 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:59.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.345+0000 7fcf467fc640 1 --2- 172.21.15.37:0/1071100575 >> v2:172.21.15.45:6812/2379918623 conn(0x7fcf28035eb0 0x7fcf28038370 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fcf38004660 tx=0x7fcf38008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:59.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.345+0000 7fcf3f7fe640 1 -- 172.21.15.37:0/1071100575 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcf3000dbc0 con 0x7fcf48136a00 2024-04-26T00:35:59.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.345+0000 7fcf3f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:59.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.345+0000 7fcf3f7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:35:59.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.345+0000 7fcf3f7fe640 1 -- 172.21.15.37:0/1071100575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf48136a00 msgr2=0x7fcf48139ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:59.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.345+0000 7fcf3f7fe640 1 --2- 172.21.15.37:0/1071100575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf48136a00 0x7fcf48139ad0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fcf3000f7a0 tx=0x7fcf3000fc60 comp rx=0 tx=0).stop 2024-04-26T00:35:59.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.345+0000 7fcf3f7fe640 1 -- 172.21.15.37:0/1071100575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf48136a00 msgr2=0x7fcf48139ad0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:35:59.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.345+0000 7fcf3f7fe640 1 --2- 172.21.15.37:0/1071100575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf48136a00 0x7fcf48139ad0 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fcf3000f7a0 tx=0x7fcf3000fc60 comp rx=0 tx=0).stop 2024-04-26T00:35:59.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.353+0000 7fcf4d77d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:35:59.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.353+0000 7fcf4d77d640 1 --2- 172.21.15.37:0/1071100575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf4813a010 0x7fcf480a0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:59.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.353+0000 7fcf4d77d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:59.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.353+0000 7fcf4d77d640 1 -- 172.21.15.37:0/1071100575 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcf4813a420 con 0x7fcf4813a010 2024-04-26T00:35:59.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.353+0000 7fcf46ffd640 1 --2- 172.21.15.37:0/1071100575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf4813a010 0x7fcf480a0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:59.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.353+0000 7fcf46ffd640 10 monclient: get_auth_request con 0x7fcf4813a010 auth_method 0 2024-04-26T00:35:59.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.353+0000 7fcf46ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:59.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.353+0000 7fcf46ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:59.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.353+0000 7fcf46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:59.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.353+0000 7fcf46ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:59.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.353+0000 7fcf46ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:59.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.353+0000 7fcf46ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:59.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.353+0000 7fcf46ffd640 10 monclient(hunting): handle_auth_done global_id 194122 payload 824 2024-04-26T00:35:59.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.353+0000 7fcf46ffd640 1 -- 172.21.15.37:0/1071100575 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcf4804f510 con 0x7fcf4813a010 2024-04-26T00:35:59.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.357+0000 7fcf46ffd640 1 -- 172.21.15.37:0/1071100575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf4813a010 msgr2=0x7fcf480a0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:59.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.357+0000 7fcf46ffd640 1 --2- 172.21.15.37:0/1071100575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf4813a010 0x7fcf480a0790 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fcf3000f7a0 tx=0x7fcf30007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:59.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.357+0000 7fcf3f7fe640 1 -- 172.21.15.37:0/1071100575 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf3001dd80 con 0x7fcf4813a010 2024-04-26T00:35:59.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.357+0000 7fcf3f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:59.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.357+0000 7fcf3f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:35:59.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.357+0000 7fcf3f7fe640 10 monclient: dump: 2024-04-26T00:35:59.361 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:59.361 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:59.361 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.361 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.361 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:59.361 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:59.361 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:59.362 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:59.362 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:59.362 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:59.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.357+0000 7fcf3f7fe640 1 -- 172.21.15.37:0/1071100575 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7fcf30004000 con 0x7fcf4813a010 2024-04-26T00:35:59.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.357+0000 7fcf3f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:35:59.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.357+0000 7fcf3f7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:35:59.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.357+0000 7fcf3f7fe640 1 -- 172.21.15.37:0/1071100575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf4813a010 msgr2=0x7fcf480a0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:59.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.357+0000 7fcf3f7fe640 1 --2- 172.21.15.37:0/1071100575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf4813a010 0x7fcf480a0790 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fcf3000f7a0 tx=0x7fcf30007f90 comp rx=0 tx=0).stop 2024-04-26T00:35:59.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.357+0000 7fcf3f7fe640 1 -- 172.21.15.37:0/1071100575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf4813a010 msgr2=0x7fcf480a0790 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:35:59.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.357+0000 7fcf3f7fe640 1 --2- 172.21.15.37:0/1071100575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf4813a010 0x7fcf480a0790 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fcf3000f7a0 tx=0x7fcf30007f90 comp rx=0 tx=0).stop 2024-04-26T00:35:59.363 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:35:59.363 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:35:59.364 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:35:59.364 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:35:59.364 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 120, 2024-04-26T00:35:59.364 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 7927, 2024-04-26T00:35:59.364 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:35:59.364 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:35:59.365 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:35:59.365 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:35:59.365 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:59.365 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 5, 2024-04-26T00:35:59.365 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:35:59.365 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:35:59.365 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:35:59.365 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:59.365 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:59.365 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:59.365 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:59.365 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:59.365 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:59.365 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:59.366 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:59.366 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:59.366 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:59.366 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:59.366 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:59.366 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:35:59.366 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:35:59.366 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:59.366 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:59.366 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:59.366 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:59.366 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:59.366 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:59.366 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:59.367 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:59.367 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:59.367 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:59.367 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:59.367 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:59.367 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:59.367 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:35:59.367 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:35:59.367 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:35:59.368 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:35:59.368 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:35:59.368 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:35:59.368 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:35:59.368 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:35:59.368 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:35:59.368 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:35:59.368 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:35:59.369 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:35:59.369 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:35:59.369 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:35:59.369 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:35:59.369 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:35:59.369 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:35:59.369 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:35:59.369 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:35:59.369 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:35:59.369 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:35:59.369 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:35:59.369 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:35:59.369 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:35:59.369 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:35:59.369 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:35:59.370 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:35:59.370 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:35:59.370 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:59.370 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:35:59.370 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:59.370 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:35:59.370 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:59.371 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:35:59.371 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:35:59.371 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:59.371 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:59.371 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:59.371 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:59.371 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:35:59.371 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:59.371 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:59.371 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:59.371 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:59.372 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:35:59.372 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:59.372 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:59.372 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:59.372 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:59.372 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:35:59.372 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:35:59.372 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:59.372 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:59.372 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:59.372 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:59.372 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:59.372 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:35:59.372 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:35:59.372 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:59.373 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:59.373 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:59.373 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:59.373 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:35:59.373 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:59.373 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:59.373 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:59.373 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:59.373 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:35:59.373 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:59.373 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:59.373 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:59.373 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:59.373 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:35:59.373 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:35:59.374 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:59.374 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:59.374 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:59.374 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:59.374 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:59.374 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:35:59.374 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:35:59.374 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:35:59.374 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:35:59.374 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:59.374 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:35:59.374 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:35:59.374 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:59.374 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:59.375 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:59.375 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:35:59.375 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:35:59.375 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:35:59.375 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:59.375 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:59.375 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:59.375 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:35:59.375 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:35:59.375 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:35:59.375 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:35:59.375 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:35:59.375 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:59.375 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:59.376 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:59.376 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:35:59.376 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:35:59.376 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:59.376 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:59.376 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:59.376 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:35:59.376 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:59.376 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:35:59.376 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:35:59.376 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:35:59.376 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:35:59.376 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:35:59.376 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:35:59.376 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:35:59.377 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:35:59.377 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:35:59.377 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:35:59.377 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:35:59.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.357+0000 7fcf4d77d640 1 -- 172.21.15.37:0/1071100575 >> v2:172.21.15.45:6812/2379918623 conn(0x7fcf28035eb0 msgr2=0x7fcf28038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:59.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.357+0000 7fcf4d77d640 1 --2- 172.21.15.37:0/1071100575 >> v2:172.21.15.45:6812/2379918623 conn(0x7fcf28035eb0 0x7fcf28038370 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fcf38004660 tx=0x7fcf38008040 comp rx=0 tx=0).stop 2024-04-26T00:35:59.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.357+0000 7fcf4d77d640 10 monclient: shutdown 2024-04-26T00:35:59.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.357+0000 7fcf4d77d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:59.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.357+0000 7fcf4d77d640 1 -- 172.21.15.37:0/1071100575 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcf4813c3e0 msgr2=0x7fcf48139590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:59.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.357+0000 7fcf4d77d640 1 --2- 172.21.15.37:0/1071100575 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcf4813c3e0 0x7fcf48139590 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcf400ae000 tx=0x7fcf40019650 comp rx=0 tx=0).stop 2024-04-26T00:35:59.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.357+0000 7fcf477fe640 1 -- 172.21.15.37:0/1071100575 reap_dead start 2024-04-26T00:35:59.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.357+0000 7fcf477fe640 1 -- 172.21.15.37:0/1071100575 reap_dead start 2024-04-26T00:35:59.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.357+0000 7fcf4d77d640 1 -- 172.21.15.37:0/1071100575 shutdown_connections 2024-04-26T00:35:59.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.357+0000 7fcf4d77d640 1 -- 172.21.15.37:0/1071100575 >> 172.21.15.37:0/1071100575 conn(0x7fcf480aa0a0 msgr2=0x7fcf4809f080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:59.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.357+0000 7fcf4d77d640 1 -- 172.21.15.37:0/1071100575 shutdown_connections 2024-04-26T00:35:59.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.357+0000 7fcf4d77d640 1 -- 172.21.15.37:0/1071100575 wait complete. 2024-04-26T00:35:59.380 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:35:59.380 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-04-26T00:35:59.380 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-26T00:35:59.381 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-04-26T00:35:59.381 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-04-26T00:35:59.381 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-26T00:35:59.381 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T00:35:59.393 INFO:tasks.ceph.mon.c:Stopped 2024-04-26T00:35:59.393 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-26T00:35:59.393 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:35:59.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.549+0000 7f86497e5640 10 monclient: get_monmap_and_config 2024-04-26T00:35:59.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.549+0000 7f86497e5640 10 monclient: build_initial_monmap 2024-04-26T00:35:59.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.549+0000 7f86497e5640 10 monclient: monmap: 2024-04-26T00:35:59.548 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:59.548 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:59.548 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:35:59.553470+0000 2024-04-26T00:35:59.548 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:35:59.553470+0000 2024-04-26T00:35:59.548 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:59.548 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:59.548 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:35:59.548 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:35:59.548 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:35:59.548 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:35:59.548 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:35:59.548 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:59.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.549+0000 7f86497e5640 1 Processor -- start 2024-04-26T00:35:59.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.549+0000 7f86497e5640 1 -- start start 2024-04-26T00:35:59.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.549+0000 7f86497e5640 10 monclient: init 2024-04-26T00:35:59.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.549+0000 7f86497e5640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:59.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.549+0000 7f86497e5640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-04-26T00:35:59.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.549+0000 7f86497e5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f86440a0260 addr v1:172.21.15.37:6789/0 2024-04-26T00:35:59.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.549+0000 7f86497e5640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f8644137e40 0x7f864413a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:59.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.549+0000 7f86497e5640 10 monclient(hunting): picked mon.noname-a con 0x7f8644137e40 addr v2:172.21.15.37:3300/0 2024-04-26T00:35:59.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.549+0000 7f86497e5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f864413a770 addr v1:172.21.15.45:6789/0 2024-04-26T00:35:59.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.549+0000 7f86497e5640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86440a1130 con 0x7f86440a0260 2024-04-26T00:35:59.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.549+0000 7f86497e5640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f864406cff0 con 0x7f864413a770 2024-04-26T00:35:59.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.549+0000 7f86497e5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:59.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.549+0000 7f86497e5640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f86440a0d50 con 0x7f8644137e40 2024-04-26T00:35:59.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.549+0000 7f86497e5640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:59.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.549+0000 7f86497e5640 10 monclient(hunting): authenticate will time out at 1940.859173s 2024-04-26T00:35:59.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.549+0000 7f86427fc640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f8644137e40 0x7f864413a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:59.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.549+0000 7f86427fc640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f8644137e40 0x7f864413a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:60944/0 (socket says 172.21.15.37:60944) 2024-04-26T00:35:59.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.549+0000 7f86427fc640 1 -- 172.21.15.37:0/2425235506 learned_addr learned my addr 172.21.15.37:0/2425235506 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:59.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.549+0000 7f86427fc640 10 monclient(hunting): get_auth_request con 0x7f8644137e40 auth_method 0 2024-04-26T00:35:59.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.549+0000 7f8642ffd640 1 --1- 172.21.15.37:0/2425235506 >> v1:172.21.15.37:6789/0 conn(0x7f86440a0260 0x7f86440a0640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:44014/0 (socket says 172.21.15.37:44014) 2024-04-26T00:35:59.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.549+0000 7f86427fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:59.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.549+0000 7f86427fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:59.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.549+0000 7f86427fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:59.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.549+0000 7f86427fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:59.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.549+0000 7f86427fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:59.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.549+0000 7f86427fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:59.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.549+0000 7f86427fc640 10 monclient(hunting): handle_auth_done global_id 144639 payload 274 2024-04-26T00:35:59.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.549+0000 7f86427fc640 1 -- 172.21.15.37:0/2425235506 >> v1:172.21.15.45:6789/0 conn(0x7f864413a770 legacy=0x7f864413cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:59.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.549+0000 7f86427fc640 1 -- 172.21.15.37:0/2425235506 >> v1:172.21.15.37:6789/0 conn(0x7f86440a0260 legacy=0x7f86440a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:35:59.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.549+0000 7f86427fc640 10 monclient: _finish_hunting 0 2024-04-26T00:35:59.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.549+0000 7f86427fc640 1 monclient: found mon.noname-a 2024-04-26T00:35:59.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.549+0000 7f86427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:59.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.549+0000 7f86427fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:35:59.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.549+0000 7f86427fc640 1 -- 172.21.15.37:0/2425235506 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f864409bc50 con 0x7f8644137e40 2024-04-26T00:35:59.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.553+0000 7f86427fc640 1 -- 172.21.15.37:0/2425235506 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8644137e40 msgr2=0x7f864413a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:59.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86427fc640 1 --2- 172.21.15.37:0/2425235506 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8644137e40 0x7f864413a230 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f862c009e50 tx=0x7f862c01c690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce72602234449414 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:59.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f8641ffb640 1 -- 172.21.15.37:0/2425235506 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f862c0a9460 con 0x7f8644137e40 2024-04-26T00:35:59.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f8641ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:59.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f8641ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:35:59.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f8641ffb640 10 monclient: dump: 2024-04-26T00:35:59.557 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:59.557 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:59.557 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.557 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.557 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:59.557 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:59.557 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:59.558 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:59.558 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:59.558 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:59.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f8641ffb640 10 monclient: _finish_auth 0 2024-04-26T00:35:59.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f8641ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:35:59.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f8641ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:59.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f8641ffb640 1 -- 172.21.15.37:0/2425235506 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f862c0ae070 con 0x7f8644137e40 2024-04-26T00:35:59.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f8641ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:59.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f8641ffb640 1 -- 172.21.15.37:0/2425235506 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f862c0a9820 con 0x7f8644137e40 2024-04-26T00:35:59.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86497e5640 5 monclient: authenticate success, global_id 144639 2024-04-26T00:35:59.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86497e5640 10 monclient: get_monmap_and_config success 2024-04-26T00:35:59.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86497e5640 4 set_mon_vals no callback set 2024-04-26T00:35:59.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f8641ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:59.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f8641ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:59.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f8641ffb640 10 monclient: dump: 2024-04-26T00:35:59.559 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:59.559 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:59.559 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.559 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.559 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:59.559 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:59.559 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:59.559 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:59.559 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:59.560 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:59.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86497e5640 10 monclient: shutdown 2024-04-26T00:35:59.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86497e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:59.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86497e5640 1 -- 172.21.15.37:0/2425235506 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8644137e40 msgr2=0x7f864413a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:59.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86497e5640 1 --2- 172.21.15.37:0/2425235506 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8644137e40 0x7f864413a230 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f862c009e50 tx=0x7f862c01c690 comp rx=0 tx=0).stop 2024-04-26T00:35:59.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86497e5640 1 -- 172.21.15.37:0/2425235506 shutdown_connections 2024-04-26T00:35:59.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86497e5640 1 --2- 172.21.15.37:0/2425235506 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8644137e40 0x7f864413a230 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:59.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86497e5640 1 -- 172.21.15.37:0/2425235506 >> 172.21.15.37:0/2425235506 conn(0x7f86440a1ad0 msgr2=0x7f86440a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:59.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86497e5640 1 -- 172.21.15.37:0/2425235506 shutdown_connections 2024-04-26T00:35:59.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86497e5640 1 -- 172.21.15.37:0/2425235506 wait complete. 2024-04-26T00:35:59.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86497e5640 10 monclient: build_initial_monmap 2024-04-26T00:35:59.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86497e5640 10 monclient: monmap: 2024-04-26T00:35:59.560 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:35:59.560 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:35:59.560 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:35:59.561 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:35:59.561 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:35:59.561 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:35:59.561 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:35:59.561 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:35:59.561 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:35:59.561 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:59.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86497e5640 1 Processor -- start 2024-04-26T00:35:59.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86497e5640 1 -- start start 2024-04-26T00:35:59.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86497e5640 10 monclient: init 2024-04-26T00:35:59.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86497e5640 10 monclient: _reopen_session rank -1 2024-04-26T00:35:59.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86497e5640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:35:59.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86497e5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86440a0260 0x7f8644071490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:59.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86497e5640 10 monclient(hunting): picked mon.noname-b con 0x7f86440a0260 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:35:59.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86497e5640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8644137e40 0x7f8644134f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:59.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86497e5640 10 monclient(hunting): picked mon.noname-a con 0x7f8644137e40 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:35:59.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86497e5640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f864413a770 0x7f86441cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:59.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86497e5640 10 monclient(hunting): picked mon.noname-c con 0x7f864413a770 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:35:59.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86497e5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:59.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86497e5640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f864409f0f0 con 0x7f8644137e40 2024-04-26T00:35:59.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86497e5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:59.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86497e5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f864409f4c0 con 0x7f86440a0260 2024-04-26T00:35:59.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86497e5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:35:59.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86497e5640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f86440a00a0 con 0x7f864413a770 2024-04-26T00:35:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86497e5640 10 monclient(hunting): _renew_subs 2024-04-26T00:35:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86497e5640 10 monclient(hunting): authenticate will time out at 1940.866884s 2024-04-26T00:35:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86437fe640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f864413a770 msgr2=0x7f86441cf0b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:35:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86427fc640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8644137e40 0x7f8644134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86437fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f864413a770 0x7f86441cf0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:35:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86427fc640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8644137e40 0x7f8644134f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:60958/0 (socket says 172.21.15.37:60958) 2024-04-26T00:35:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86427fc640 1 -- 172.21.15.37:0/3480485132 learned_addr learned my addr 172.21.15.37:0/3480485132 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:35:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86427fc640 10 monclient(hunting): get_auth_request con 0x7f8644137e40 auth_method 0 2024-04-26T00:35:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86427fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:35:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86427fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:35:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86427fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:35:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86427fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:35:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86427fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:35:59.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86427fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:35:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f8642ffd640 1 --2- 172.21.15.37:0/3480485132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86440a0260 0x7f8644071490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86427fc640 10 monclient(hunting): handle_auth_done global_id 144642 payload 824 2024-04-26T00:35:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86427fc640 1 -- 172.21.15.37:0/3480485132 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f864413a770 msgr2=0x7f86441cf0b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:35:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86427fc640 1 --2- 172.21.15.37:0/3480485132 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f864413a770 0x7f86441cf0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86427fc640 1 -- 172.21.15.37:0/3480485132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86440a0260 msgr2=0x7f8644071490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86427fc640 1 --2- 172.21.15.37:0/3480485132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86440a0260 0x7f8644071490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86427fc640 10 monclient: _finish_hunting 0 2024-04-26T00:35:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86427fc640 1 monclient: found mon.noname-a 2024-04-26T00:35:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:35:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86427fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:35:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f86427fc640 1 -- 172.21.15.37:0/3480485132 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8644120070 con 0x7f8644137e40 2024-04-26T00:35:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f8642ffd640 10 monclient: get_auth_request con 0x7f86440a0260 auth_method 0 2024-04-26T00:35:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.557+0000 7f8642ffd640 1 --2- 172.21.15.37:0/3480485132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86440a0260 0x7f8644071490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:35:59.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.561+0000 7f86427fc640 1 -- 172.21.15.37:0/3480485132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8644137e40 msgr2=0x7f8644134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:35:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.561+0000 7f86427fc640 1 --2- 172.21.15.37:0/3480485132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8644137e40 0x7f8644134f70 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f862c0ab040 tx=0x7f862c019430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.561+0000 7f863b7fe640 1 -- 172.21.15.37:0/3480485132 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f862c0b8070 con 0x7f8644137e40 2024-04-26T00:35:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.561+0000 7f863b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.561+0000 7f863b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:35:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.561+0000 7f863b7fe640 10 monclient: dump: 2024-04-26T00:35:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:59.569 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:59.569 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:59.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.561+0000 7f863b7fe640 10 monclient: _finish_auth 0 2024-04-26T00:35:59.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.561+0000 7f863b7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:35:59.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.561+0000 7f863b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:35:59.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.561+0000 7f863b7fe640 1 -- 172.21.15.37:0/3480485132 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f862c0a9c20 con 0x7f8644137e40 2024-04-26T00:35:59.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.561+0000 7f863b7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:35:59.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.561+0000 7f863b7fe640 1 -- 172.21.15.37:0/3480485132 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f862c0bc400 con 0x7f8644137e40 2024-04-26T00:35:59.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.561+0000 7f863b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:35:59.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.561+0000 7f863b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:35:59.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.561+0000 7f863b7fe640 10 monclient: dump: 2024-04-26T00:35:59.570 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:35:59.570 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:35:59.571 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.571 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:35:59.571 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:35:59.571 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:35:59.571 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:35:59.571 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:35:59.571 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:35:59.571 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:35:59.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.561+0000 7f86497e5640 5 monclient: authenticate success, global_id 144642 2024-04-26T00:35:59.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.561+0000 7f86417fa640 4 set_mon_vals no callback set 2024-04-26T00:35:59.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.561+0000 7f86497e5640 10 monclient: _renew_subs 2024-04-26T00:35:59.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.561+0000 7f86497e5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:59.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.561+0000 7f86497e5640 1 -- 172.21.15.37:0/3480485132 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8644120350 con 0x7f8644137e40 2024-04-26T00:35:59.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.561+0000 7f86497e5640 10 monclient: _renew_subs 2024-04-26T00:35:59.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.561+0000 7f86497e5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:59.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.561+0000 7f86497e5640 1 -- 172.21.15.37:0/3480485132 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f864411fbc0 con 0x7f8644137e40 2024-04-26T00:35:59.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.561+0000 7f86497e5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:59.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.561+0000 7f86497e5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:59.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.561+0000 7f86497e5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:59.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.561+0000 7f86497e5640 1 -- 172.21.15.37:0/3480485132 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8644072080 con 0x7f8644137e40 2024-04-26T00:35:59.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.561+0000 7f863b7fe640 1 -- 172.21.15.37:0/3480485132 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f862c0bc5c0 con 0x7f8644137e40 2024-04-26T00:35:59.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.561+0000 7f863b7fe640 1 --2- 172.21.15.37:0/3480485132 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8624035e80 0x7f8624038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:35:59.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.561+0000 7f863b7fe640 1 -- 172.21.15.37:0/3480485132 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(629..629 src has 370..629) ==== 3971+0+0 (secure 0 0 0) 0x7f862c0e3140 con 0x7f8644137e40 2024-04-26T00:35:59.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.561+0000 7f8642ffd640 1 --2- 172.21.15.37:0/3480485132 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8624035e80 0x7f8624038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:35:59.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.561+0000 7f8642ffd640 10 monclient: get_auth_request con 0x7f8624035e80 auth_method 0 2024-04-26T00:35:59.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.565+0000 7f8642ffd640 1 --2- 172.21.15.37:0/3480485132 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8624035e80 0x7f8624038340 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f863c007fd0 tx=0x7f863c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:35:59.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.565+0000 7f863b7fe640 1 -- 172.21.15.37:0/3480485132 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f8644120070 con 0x7f8644137e40 2024-04-26T00:35:59.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.565+0000 7f863b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:35:59.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.565+0000 7f863b7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:35:59.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.789+0000 7f86497e5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:35:59.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.789+0000 7f86497e5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:35:59.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.789+0000 7f86497e5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:35:59.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.789+0000 7f86497e5640 1 -- 172.21.15.37:0/3480485132 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8644066f40 con 0x7f8644137e40 2024-04-26T00:35:59.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.789+0000 7f863b7fe640 1 -- 172.21.15.37:0/3480485132 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f86440a1130 con 0x7f8644137e40 2024-04-26T00:35:59.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.789+0000 7f863b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:35:59.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.789+0000 7f863b7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:35:59.788 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":120,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:35:59.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.793+0000 7f86497e5640 1 -- 172.21.15.37:0/3480485132 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8624035e80 msgr2=0x7f8624038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:59.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.793+0000 7f86497e5640 1 --2- 172.21.15.37:0/3480485132 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8624035e80 0x7f8624038340 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f863c007fd0 tx=0x7f863c009040 comp rx=0 tx=0).stop 2024-04-26T00:35:59.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.793+0000 7f86497e5640 10 monclient: shutdown 2024-04-26T00:35:59.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.793+0000 7f86497e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:35:59.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.793+0000 7f86497e5640 1 -- 172.21.15.37:0/3480485132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8644137e40 msgr2=0x7f8644134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:35:59.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.793+0000 7f86497e5640 1 --2- 172.21.15.37:0/3480485132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8644137e40 0x7f8644134f70 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f862c0ab040 tx=0x7f862c019430 comp rx=0 tx=0).stop 2024-04-26T00:35:59.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.793+0000 7f86497e5640 1 -- 172.21.15.37:0/3480485132 shutdown_connections 2024-04-26T00:35:59.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.793+0000 7f86497e5640 1 --2- 172.21.15.37:0/3480485132 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8624035e80 0x7f8624038340 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:59.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.793+0000 7f86497e5640 1 --2- 172.21.15.37:0/3480485132 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f864413a770 0x7f86441cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:59.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.793+0000 7f86497e5640 1 --2- 172.21.15.37:0/3480485132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8644137e40 0x7f8644134f70 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:59.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.793+0000 7f86497e5640 1 --2- 172.21.15.37:0/3480485132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f86440a0260 0x7f8644071490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:35:59.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.793+0000 7f86497e5640 1 -- 172.21.15.37:0/3480485132 >> 172.21.15.37:0/3480485132 conn(0x7f86440a1ad0 msgr2=0x7f8644137600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:35:59.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.793+0000 7f86497e5640 1 -- 172.21.15.37:0/3480485132 shutdown_connections 2024-04-26T00:35:59.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:35:59.793+0000 7f86497e5640 1 -- 172.21.15.37:0/3480485132 wait complete. 2024-04-26T00:36:02.808 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:36:02.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.941+0000 7f62f4291640 10 monclient: get_monmap_and_config 2024-04-26T00:36:02.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.941+0000 7f62f4291640 10 monclient: build_initial_monmap 2024-04-26T00:36:02.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.941+0000 7f62f4291640 10 monclient: monmap: 2024-04-26T00:36:02.940 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:36:02.940 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:02.940 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:36:02.945518+0000 2024-04-26T00:36:02.940 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:36:02.945518+0000 2024-04-26T00:36:02.940 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:36:02.940 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:02.940 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:36:02.940 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:36:02.940 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:36:02.940 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:36:02.940 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:36:02.941 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:02.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.941+0000 7f62f4291640 1 Processor -- start 2024-04-26T00:36:02.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.941+0000 7f62f4291640 1 -- start start 2024-04-26T00:36:02.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.941+0000 7f62f4291640 10 monclient: init 2024-04-26T00:36:02.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.941+0000 7f62f4291640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:02.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.941+0000 7f62f4291640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-04-26T00:36:02.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.941+0000 7f62f4291640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f62ec0a0260 addr v1:172.21.15.45:6789/0 2024-04-26T00:36:02.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.941+0000 7f62f4291640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f62ec137e40 0x7f62ec13a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:02.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.941+0000 7f62f4291640 10 monclient(hunting): picked mon.noname-c con 0x7f62ec137e40 addr v2:172.21.15.45:3300/0 2024-04-26T00:36:02.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.941+0000 7f62f4291640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f62ec13a770 addr v1:172.21.15.37:6789/0 2024-04-26T00:36:02.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.941+0000 7f62f4291640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62ec0a1130 con 0x7f62ec13a770 2024-04-26T00:36:02.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.941+0000 7f62f4291640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62ec06cff0 con 0x7f62ec0a0260 2024-04-26T00:36:02.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.941+0000 7f62f4291640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:02.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.941+0000 7f62f4291640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f62ec0a0d50 con 0x7f62ec137e40 2024-04-26T00:36:02.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.941+0000 7f62f4291640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:02.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.941+0000 7f62f4291640 10 monclient(hunting): authenticate will time out at 1944.251427s 2024-04-26T00:36:02.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.941+0000 7f62f282e640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f62ec13a770 0x7f62ec13cb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:55686/0 (socket says 172.21.15.37:55686) 2024-04-26T00:36:02.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.941+0000 7f62f282e640 1 -- 172.21.15.37:0/237034945 learned_addr learned my addr 172.21.15.37:0/237034945 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:36:02.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.941+0000 7f62f182c640 1 --2- 172.21.15.37:0/237034945 >> v2:172.21.15.45:3300/0 conn(0x7f62ec137e40 0x7f62ec13a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:02.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.941+0000 7f62f182c640 10 monclient(hunting): get_auth_request con 0x7f62ec137e40 auth_method 0 2024-04-26T00:36:02.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.941+0000 7f62f182c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:02.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.941+0000 7f62f182c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:02.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.941+0000 7f62f182c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:02.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.945+0000 7f62f182c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:02.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.945+0000 7f62f182c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:02.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.945+0000 7f62f182c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:02.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.945+0000 7f62f182c640 10 monclient(hunting): handle_auth_done global_id 194125 payload 274 2024-04-26T00:36:02.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.945+0000 7f62f182c640 1 -- 172.21.15.37:0/237034945 >> v1:172.21.15.45:6789/0 conn(0x7f62ec0a0260 legacy=0x7f62ec0a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:36:02.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.945+0000 7f62f182c640 1 -- 172.21.15.37:0/237034945 >> v1:172.21.15.37:6789/0 conn(0x7f62ec13a770 legacy=0x7f62ec13cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:36:02.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.945+0000 7f62f182c640 10 monclient: _finish_hunting 0 2024-04-26T00:36:02.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.945+0000 7f62f182c640 1 monclient: found mon.noname-c 2024-04-26T00:36:02.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.945+0000 7f62f182c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:02.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.945+0000 7f62f182c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:36:02.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.945+0000 7f62f182c640 1 -- 172.21.15.37:0/237034945 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62ec09bc50 con 0x7f62ec137e40 2024-04-26T00:36:02.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.949+0000 7f62f182c640 1 -- 172.21.15.37:0/237034945 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62ec137e40 msgr2=0x7f62ec13a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:02.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.949+0000 7f62f182c640 1 --2- 172.21.15.37:0/237034945 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62ec137e40 0x7f62ec13a230 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f62dc0097c0 tx=0x7f62dc01c100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f5cc54de9c94a8d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:02.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.949+0000 7f62f102b640 1 -- 172.21.15.37:0/237034945 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62dc0b1070 con 0x7f62ec137e40 2024-04-26T00:36:02.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.949+0000 7f62f102b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:02.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.949+0000 7f62f102b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:36:02.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.949+0000 7f62f102b640 10 monclient: dump: 2024-04-26T00:36:02.948 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:02.948 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:02.948 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:02.948 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:02.948 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:02.948 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:02.948 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:02.948 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:02.948 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:02.948 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:02.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.949+0000 7f62f102b640 10 monclient: _finish_auth 0 2024-04-26T00:36:02.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.949+0000 7f62f102b640 10 monclient: _check_auth_tickets 2024-04-26T00:36:02.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.949+0000 7f62f102b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:02.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.949+0000 7f62f102b640 1 -- 172.21.15.37:0/237034945 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62dc0ac440 con 0x7f62ec137e40 2024-04-26T00:36:02.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.949+0000 7f62f102b640 10 monclient: handle_config config(0 keys) 2024-04-26T00:36:02.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.949+0000 7f62f102b640 1 -- 172.21.15.37:0/237034945 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62dc0ac810 con 0x7f62ec137e40 2024-04-26T00:36:02.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.949+0000 7f62f102b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:02.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.949+0000 7f62f102b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:36:02.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.949+0000 7f62f102b640 10 monclient: dump: 2024-04-26T00:36:02.949 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:02.949 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:02.950 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:02.950 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:02.950 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:02.950 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:02.950 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:02.950 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:02.950 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:02.950 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:02.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.949+0000 7f62f4291640 5 monclient: authenticate success, global_id 194125 2024-04-26T00:36:02.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.949+0000 7f62f4291640 10 monclient: get_monmap_and_config success 2024-04-26T00:36:02.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.949+0000 7f62f4291640 4 set_mon_vals no callback set 2024-04-26T00:36:02.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.949+0000 7f62f4291640 10 monclient: shutdown 2024-04-26T00:36:02.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.949+0000 7f62f4291640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:36:02.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.949+0000 7f62f4291640 1 -- 172.21.15.37:0/237034945 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62ec137e40 msgr2=0x7f62ec13a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:02.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.949+0000 7f62f4291640 1 --2- 172.21.15.37:0/237034945 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62ec137e40 0x7f62ec13a230 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f62dc0097c0 tx=0x7f62dc01c100 comp rx=0 tx=0).stop 2024-04-26T00:36:02.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.949+0000 7f62f4291640 1 -- 172.21.15.37:0/237034945 shutdown_connections 2024-04-26T00:36:02.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.949+0000 7f62f4291640 1 --2- 172.21.15.37:0/237034945 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62ec137e40 0x7f62ec13a230 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:02.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.949+0000 7f62f4291640 1 -- 172.21.15.37:0/237034945 >> 172.21.15.37:0/237034945 conn(0x7f62ec0a1ad0 msgr2=0x7f62ec0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:36:02.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.949+0000 7f62f4291640 1 -- 172.21.15.37:0/237034945 shutdown_connections 2024-04-26T00:36:02.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.949+0000 7f62f4291640 1 -- 172.21.15.37:0/237034945 wait complete. 2024-04-26T00:36:02.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.949+0000 7f62f4291640 10 monclient: build_initial_monmap 2024-04-26T00:36:02.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.949+0000 7f62f4291640 10 monclient: monmap: 2024-04-26T00:36:02.956 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:36:02.956 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:36:02.956 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:36:02.956 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:36:02.956 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:36:02.956 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:36:02.956 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:36:02.956 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:36:02.956 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:36:02.956 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:02.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.949+0000 7f62f4291640 1 Processor -- start 2024-04-26T00:36:02.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.949+0000 7f62f4291640 1 -- start start 2024-04-26T00:36:02.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.949+0000 7f62f4291640 10 monclient: init 2024-04-26T00:36:02.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f4291640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:02.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f4291640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:36:02.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f4291640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62ec0a0260 0x7f62ec1349c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:02.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f4291640 10 monclient(hunting): picked mon.noname-b con 0x7f62ec0a0260 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:36:02.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f4291640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f62ec137e40 0x7f62ec134f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:02.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f4291640 10 monclient(hunting): picked mon.noname-a con 0x7f62ec137e40 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:36:02.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f4291640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec13a770 0x7f62ec09c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:02.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f4291640 10 monclient(hunting): picked mon.noname-c con 0x7f62ec13a770 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:36:02.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f4291640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:02.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f4291640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec09f0f0 con 0x7f62ec137e40 2024-04-26T00:36:02.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f4291640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:02.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f4291640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec09f4c0 con 0x7f62ec0a0260 2024-04-26T00:36:02.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f4291640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:02.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f4291640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f62ec0a00a0 con 0x7f62ec13a770 2024-04-26T00:36:02.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f4291640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:02.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f4291640 10 monclient(hunting): authenticate will time out at 1944.260584s 2024-04-26T00:36:02.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f282e640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec13a770 msgr2=0x7f62ec09c4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:36:02.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f282e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec13a770 0x7f62ec09c4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:36:02.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f182c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f62ec137e40 0x7f62ec134f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:02.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f182c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f62ec137e40 0x7f62ec134f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52020/0 (socket says 172.21.15.37:52020) 2024-04-26T00:36:02.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f182c640 1 -- 172.21.15.37:0/3027377431 learned_addr learned my addr 172.21.15.37:0/3027377431 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:36:02.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f182c640 10 monclient(hunting): get_auth_request con 0x7f62ec137e40 auth_method 0 2024-04-26T00:36:02.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f182c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:02.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f182c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:02.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f182c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:02.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f182c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:02.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f182c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:02.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f182c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:02.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f202d640 1 --2- 172.21.15.37:0/3027377431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62ec0a0260 0x7f62ec1349c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:02.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f202d640 10 monclient(hunting): get_auth_request con 0x7f62ec0a0260 auth_method 0 2024-04-26T00:36:02.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f202d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:02.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f202d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:02.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f202d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:02.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f182c640 10 monclient(hunting): handle_auth_done global_id 144645 payload 824 2024-04-26T00:36:02.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f182c640 1 -- 172.21.15.37:0/3027377431 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec13a770 msgr2=0x7f62ec09c4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:36:02.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f182c640 1 --2- 172.21.15.37:0/3027377431 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec13a770 0x7f62ec09c4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:02.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f182c640 1 -- 172.21.15.37:0/3027377431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62ec0a0260 msgr2=0x7f62ec1349c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:02.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f182c640 1 --2- 172.21.15.37:0/3027377431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62ec0a0260 0x7f62ec1349c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:02.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f182c640 10 monclient: _finish_hunting 0 2024-04-26T00:36:02.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f182c640 1 monclient: found mon.noname-a 2024-04-26T00:36:02.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f182c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:02.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f182c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:36:02.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f182c640 1 -- 172.21.15.37:0/3027377431 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62ec11fef0 con 0x7f62ec137e40 2024-04-26T00:36:02.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f182c640 1 -- 172.21.15.37:0/3027377431 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f62ec137e40 msgr2=0x7f62ec134f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:02.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f182c640 1 --2- 172.21.15.37:0/3027377431 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f62ec137e40 0x7f62ec134f00 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f62dc018d10 tx=0x7f62dc018ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:02.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62e2ffd640 1 -- 172.21.15.37:0/3027377431 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62dc0ba020 con 0x7f62ec137e40 2024-04-26T00:36:02.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:02.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62e2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:36:02.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62e2ffd640 10 monclient: dump: 2024-04-26T00:36:02.961 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:02.961 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:02.961 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:02.961 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:02.961 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:02.961 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:02.961 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:02.962 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:02.962 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:02.962 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:02.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62e2ffd640 10 monclient: _finish_auth 0 2024-04-26T00:36:02.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62e2ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:36:02.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:02.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62e2ffd640 1 -- 172.21.15.37:0/3027377431 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62dc0ace50 con 0x7f62ec137e40 2024-04-26T00:36:02.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62e2ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:36:02.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62e2ffd640 1 -- 172.21.15.37:0/3027377431 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62dc0b1040 con 0x7f62ec137e40 2024-04-26T00:36:02.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f4291640 5 monclient: authenticate success, global_id 144645 2024-04-26T00:36:02.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f4291640 10 monclient: _renew_subs 2024-04-26T00:36:02.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f4291640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:02.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f4291640 1 -- 172.21.15.37:0/3027377431 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62ec071e80 con 0x7f62ec137e40 2024-04-26T00:36:02.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f082a640 4 set_mon_vals no callback set 2024-04-26T00:36:02.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:02.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62e2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:36:02.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62e2ffd640 10 monclient: dump: 2024-04-26T00:36:02.963 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:02.963 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:02.963 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:02.963 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:02.963 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:02.963 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:02.963 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:02.963 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:02.963 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:02.963 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:02.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f4291640 10 monclient: _renew_subs 2024-04-26T00:36:02.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f4291640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:02.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.953+0000 7f62f4291640 1 -- 172.21.15.37:0/3027377431 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62ec135440 con 0x7f62ec137e40 2024-04-26T00:36:02.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.957+0000 7f62e2ffd640 1 -- 172.21.15.37:0/3027377431 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f62dc0bd020 con 0x7f62ec137e40 2024-04-26T00:36:02.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.957+0000 7f62f4291640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:02.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.957+0000 7f62f4291640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:02.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.957+0000 7f62f4291640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:02.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.957+0000 7f62f4291640 1 -- 172.21.15.37:0/3027377431 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62ec066f40 con 0x7f62ec137e40 2024-04-26T00:36:02.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.957+0000 7f62e2ffd640 1 --2- 172.21.15.37:0/3027377431 >> v2:172.21.15.45:6812/2379918623 conn(0x7f62d0035e30 0x7f62d00382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:02.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.957+0000 7f62e2ffd640 1 -- 172.21.15.37:0/3027377431 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(629..629 src has 370..629) ==== 3971+0+0 (secure 0 0 0) 0x7f62dc0e2bc0 con 0x7f62ec137e40 2024-04-26T00:36:02.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.957+0000 7f62f202d640 1 --2- 172.21.15.37:0/3027377431 >> v2:172.21.15.45:6812/2379918623 conn(0x7f62d0035e30 0x7f62d00382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:02.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.957+0000 7f62f202d640 10 monclient: get_auth_request con 0x7f62d0035e30 auth_method 0 2024-04-26T00:36:02.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.961+0000 7f62f202d640 1 --2- 172.21.15.37:0/3027377431 >> v2:172.21.15.45:6812/2379918623 conn(0x7f62d0035e30 0x7f62d00382f0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f62ec135ac0 tx=0x7f62e8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:02.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.961+0000 7f62e2ffd640 1 -- 172.21.15.37:0/3027377431 DispatchQueue::entry inject delay of 0.002000 2024-04-26T00:36:02.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.961+0000 7f62e2ffd640 1 -- 172.21.15.37:0/3027377431 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f62dc0c4030 con 0x7f62ec137e40 2024-04-26T00:36:02.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.961+0000 7f62e2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:02.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:02.961+0000 7f62e2ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:36:03.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:03.181+0000 7f62f4291640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:36:03.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:03.181+0000 7f62f4291640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:36:03.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:03.181+0000 7f62f4291640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:03.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:03.181+0000 7f62f4291640 1 -- 172.21.15.37:0/3027377431 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f62ec06b9a0 con 0x7f62ec137e40 2024-04-26T00:36:03.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:03.181+0000 7f62e2ffd640 1 -- 172.21.15.37:0/3027377431 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f62ec0a1130 con 0x7f62ec137e40 2024-04-26T00:36:03.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:03.181+0000 7f62e2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:36:03.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:03.181+0000 7f62e2ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:36:03.181 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":120,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:36:03.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:03.185+0000 7f62f4291640 1 -- 172.21.15.37:0/3027377431 >> v2:172.21.15.45:6812/2379918623 conn(0x7f62d0035e30 msgr2=0x7f62d00382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:03.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:03.185+0000 7f62f4291640 1 --2- 172.21.15.37:0/3027377431 >> v2:172.21.15.45:6812/2379918623 conn(0x7f62d0035e30 0x7f62d00382f0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f62ec135ac0 tx=0x7f62e8009040 comp rx=0 tx=0).stop 2024-04-26T00:36:03.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:03.185+0000 7f62f4291640 10 monclient: shutdown 2024-04-26T00:36:03.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:03.185+0000 7f62f4291640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:36:03.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:03.185+0000 7f62f4291640 1 -- 172.21.15.37:0/3027377431 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f62ec137e40 msgr2=0x7f62ec134f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:03.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:03.185+0000 7f62f4291640 1 --2- 172.21.15.37:0/3027377431 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f62ec137e40 0x7f62ec134f00 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f62dc018d10 tx=0x7f62dc018ea0 comp rx=0 tx=0).stop 2024-04-26T00:36:03.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:03.185+0000 7f62f4291640 1 -- 172.21.15.37:0/3027377431 shutdown_connections 2024-04-26T00:36:03.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:03.185+0000 7f62f4291640 1 --2- 172.21.15.37:0/3027377431 >> v2:172.21.15.45:6812/2379918623 conn(0x7f62d0035e30 0x7f62d00382f0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:03.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:03.185+0000 7f62f4291640 1 --2- 172.21.15.37:0/3027377431 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62ec13a770 0x7f62ec09c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:03.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:03.185+0000 7f62f4291640 1 --2- 172.21.15.37:0/3027377431 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f62ec137e40 0x7f62ec134f00 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:03.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:03.185+0000 7f62f4291640 1 --2- 172.21.15.37:0/3027377431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f62ec0a0260 0x7f62ec1349c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:03.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:03.185+0000 7f62f4291640 1 -- 172.21.15.37:0/3027377431 >> 172.21.15.37:0/3027377431 conn(0x7f62ec0a1ad0 msgr2=0x7f62ec0a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:36:03.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:03.185+0000 7f62f4291640 1 -- 172.21.15.37:0/3027377431 shutdown_connections 2024-04-26T00:36:03.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:03.185+0000 7f62f4291640 1 -- 172.21.15.37:0/3027377431 wait complete. 2024-04-26T00:36:06.202 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:36:06.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.337+0000 7ff757a7e640 10 monclient: get_monmap_and_config 2024-04-26T00:36:06.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.337+0000 7ff757a7e640 10 monclient: build_initial_monmap 2024-04-26T00:36:06.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.337+0000 7ff757a7e640 10 monclient: monmap: 2024-04-26T00:36:06.335 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:36:06.336 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:06.336 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:36:06.341331+0000 2024-04-26T00:36:06.336 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:36:06.341331+0000 2024-04-26T00:36:06.336 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:36:06.337 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:06.337 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:36:06.337 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:36:06.337 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:36:06.337 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:36:06.337 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:36:06.337 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:06.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.337+0000 7ff757a7e640 1 Processor -- start 2024-04-26T00:36:06.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.337+0000 7ff757a7e640 1 -- start start 2024-04-26T00:36:06.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.337+0000 7ff757a7e640 10 monclient: init 2024-04-26T00:36:06.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.337+0000 7ff757a7e640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:06.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.337+0000 7ff757a7e640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-04-26T00:36:06.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.337+0000 7ff757a7e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff750136da0 addr v1:172.21.15.45:6789/0 2024-04-26T00:36:06.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.337+0000 7ff757a7e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff750137180 addr v1:172.21.15.37:6789/0 2024-04-26T00:36:06.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.337+0000 7ff757a7e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff75013d2e0 0x7ff75013f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:06.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.337+0000 7ff757a7e640 10 monclient(hunting): picked mon.noname-c con 0x7ff75013d2e0 addr v2:172.21.15.45:3300/0 2024-04-26T00:36:06.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.337+0000 7ff757a7e640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff75009b4f0 con 0x7ff750137180 2024-04-26T00:36:06.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.337+0000 7ff757a7e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff75009b7d0 con 0x7ff750136da0 2024-04-26T00:36:06.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.337+0000 7ff757a7e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:06.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.337+0000 7ff757a7e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7ff75009bb70 con 0x7ff75013d2e0 2024-04-26T00:36:06.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.337+0000 7ff757a7e640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:06.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.337+0000 7ff757a7e640 10 monclient(hunting): authenticate will time out at 1947.647189s 2024-04-26T00:36:06.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.337+0000 7ff755019640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7ff750137180 0x7ff7501375c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:55698/0 (socket says 172.21.15.37:55698) 2024-04-26T00:36:06.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.337+0000 7ff755019640 1 -- 172.21.15.37:0/3946276988 learned_addr learned my addr 172.21.15.37:0/3946276988 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:36:06.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.337+0000 7ff75601b640 1 --2- 172.21.15.37:0/3946276988 >> v2:172.21.15.45:3300/0 conn(0x7ff75013d2e0 0x7ff75013f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:06.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.337+0000 7ff75601b640 10 monclient(hunting): get_auth_request con 0x7ff75013d2e0 auth_method 0 2024-04-26T00:36:06.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.337+0000 7ff75601b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:06.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.337+0000 7ff75601b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:06.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.337+0000 7ff75601b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:06.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.337+0000 7ff75601b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:06.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.337+0000 7ff75601b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:06.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.337+0000 7ff75601b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:06.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.341+0000 7ff75601b640 10 monclient(hunting): handle_auth_done global_id 194131 payload 274 2024-04-26T00:36:06.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.341+0000 7ff75601b640 1 -- 172.21.15.37:0/3946276988 >> v1:172.21.15.45:6789/0 conn(0x7ff750136da0 legacy=0x7ff75009bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:36:06.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.341+0000 7ff75601b640 1 -- 172.21.15.37:0/3946276988 >> v1:172.21.15.37:6789/0 conn(0x7ff750137180 legacy=0x7ff7501375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:36:06.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.341+0000 7ff75601b640 10 monclient: _finish_hunting 0 2024-04-26T00:36:06.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.341+0000 7ff75601b640 1 monclient: found mon.noname-c 2024-04-26T00:36:06.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.341+0000 7ff75601b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:06.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.341+0000 7ff75601b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:36:06.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.341+0000 7ff75601b640 1 -- 172.21.15.37:0/3946276988 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff75009c5e0 con 0x7ff75013d2e0 2024-04-26T00:36:06.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.345+0000 7ff75601b640 1 -- 172.21.15.37:0/3946276988 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff75013d2e0 msgr2=0x7ff75013f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:06.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.345+0000 7ff75601b640 1 --2- 172.21.15.37:0/3946276988 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff75013d2e0 0x7ff75013f6d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff7480097c0 tx=0x7ff74801c2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1589db1cc4f98521 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:06.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.345+0000 7ff754818640 1 -- 172.21.15.37:0/3946276988 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7480b0070 con 0x7ff75013d2e0 2024-04-26T00:36:06.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.345+0000 7ff754818640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:06.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.345+0000 7ff754818640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:36:06.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.345+0000 7ff754818640 10 monclient: dump: 2024-04-26T00:36:06.345 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:06.345 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:06.345 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:06.345 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:06.346 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:06.346 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:06.346 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:06.346 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:06.346 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:06.346 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:06.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.345+0000 7ff754818640 10 monclient: _finish_auth 0 2024-04-26T00:36:06.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.345+0000 7ff754818640 10 monclient: _check_auth_tickets 2024-04-26T00:36:06.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.345+0000 7ff754818640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:06.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.345+0000 7ff754818640 1 -- 172.21.15.37:0/3946276988 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7480ab600 con 0x7ff75013d2e0 2024-04-26T00:36:06.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.345+0000 7ff754818640 10 monclient: handle_config config(0 keys) 2024-04-26T00:36:06.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.345+0000 7ff754818640 1 -- 172.21.15.37:0/3946276988 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7480ab9d0 con 0x7ff75013d2e0 2024-04-26T00:36:06.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.345+0000 7ff754818640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:06.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.345+0000 7ff754818640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:36:06.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.345+0000 7ff754818640 10 monclient: dump: 2024-04-26T00:36:06.347 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:06.347 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:06.347 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:06.347 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:06.347 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:06.348 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:06.348 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:06.348 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:06.348 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:06.348 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:06.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.345+0000 7ff757a7e640 5 monclient: authenticate success, global_id 194131 2024-04-26T00:36:06.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.345+0000 7ff757a7e640 10 monclient: get_monmap_and_config success 2024-04-26T00:36:06.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.345+0000 7ff757a7e640 4 set_mon_vals no callback set 2024-04-26T00:36:06.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.345+0000 7ff757a7e640 10 monclient: shutdown 2024-04-26T00:36:06.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.345+0000 7ff757a7e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:36:06.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.345+0000 7ff757a7e640 1 -- 172.21.15.37:0/3946276988 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff75013d2e0 msgr2=0x7ff75013f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:06.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.345+0000 7ff757a7e640 1 --2- 172.21.15.37:0/3946276988 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff75013d2e0 0x7ff75013f6d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff7480097c0 tx=0x7ff74801c2e0 comp rx=0 tx=0).stop 2024-04-26T00:36:06.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.345+0000 7ff757a7e640 1 -- 172.21.15.37:0/3946276988 shutdown_connections 2024-04-26T00:36:06.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.345+0000 7ff757a7e640 1 --2- 172.21.15.37:0/3946276988 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff75013d2e0 0x7ff75013f6d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:06.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.345+0000 7ff757a7e640 1 -- 172.21.15.37:0/3946276988 >> 172.21.15.37:0/3946276988 conn(0x7ff7500a5f00 msgr2=0x7ff7500a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:36:06.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.345+0000 7ff757a7e640 1 -- 172.21.15.37:0/3946276988 shutdown_connections 2024-04-26T00:36:06.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.345+0000 7ff757a7e640 1 -- 172.21.15.37:0/3946276988 wait complete. 2024-04-26T00:36:06.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.345+0000 7ff757a7e640 10 monclient: build_initial_monmap 2024-04-26T00:36:06.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.345+0000 7ff757a7e640 10 monclient: monmap: 2024-04-26T00:36:06.356 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:36:06.356 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:36:06.356 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:36:06.357 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:36:06.357 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:36:06.357 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:36:06.357 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:36:06.357 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:36:06.357 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:36:06.357 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:06.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff757a7e640 1 Processor -- start 2024-04-26T00:36:06.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff757a7e640 1 -- start start 2024-04-26T00:36:06.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff757a7e640 10 monclient: init 2024-04-26T00:36:06.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff757a7e640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:06.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff757a7e640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:36:06.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff757a7e640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff750136da0 0x7ff7501cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:06.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff757a7e640 10 monclient(hunting): picked mon.noname-a con 0x7ff750136da0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:36:06.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff757a7e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff750137180 0x7ff750138bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:06.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff757a7e640 10 monclient(hunting): picked mon.noname-c con 0x7ff750137180 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:36:06.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff757a7e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff75013d2e0 0x7ff750139110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:06.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff757a7e640 10 monclient(hunting): picked mon.noname-b con 0x7ff75013d2e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:36:06.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff757a7e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:06.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff757a7e640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff750142e20 con 0x7ff750136da0 2024-04-26T00:36:06.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff755019640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff750137180 msgr2=0x7ff750138bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:36:06.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff757a7e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:06.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff757a7e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff750142ca0 con 0x7ff75013d2e0 2024-04-26T00:36:06.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff757a7e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:06.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff757a7e640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff750142fa0 con 0x7ff750137180 2024-04-26T00:36:06.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff757a7e640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:06.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff755019640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff750137180 0x7ff750138bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:36:06.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff75581a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff750136da0 0x7ff7501cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:06.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff757a7e640 10 monclient(hunting): authenticate will time out at 1947.657283s 2024-04-26T00:36:06.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff75581a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff750136da0 0x7ff7501cf8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52024/0 (socket says 172.21.15.37:52024) 2024-04-26T00:36:06.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff75581a640 1 -- 172.21.15.37:0/4077856953 learned_addr learned my addr 172.21.15.37:0/4077856953 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:36:06.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff75581a640 10 monclient(hunting): get_auth_request con 0x7ff750136da0 auth_method 0 2024-04-26T00:36:06.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff75581a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:06.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff75581a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:06.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff75581a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:06.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff75581a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:06.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff75581a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:06.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff75581a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:06.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff75601b640 1 --2- 172.21.15.37:0/4077856953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff75013d2e0 0x7ff750139110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:06.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff75601b640 10 monclient(hunting): get_auth_request con 0x7ff75013d2e0 auth_method 0 2024-04-26T00:36:06.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff75601b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:06.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff75601b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:06.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff75601b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:06.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff75581a640 10 monclient(hunting): handle_auth_done global_id 144648 payload 824 2024-04-26T00:36:06.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff75581a640 1 -- 172.21.15.37:0/4077856953 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff750137180 msgr2=0x7ff750138bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:36:06.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff75581a640 1 --2- 172.21.15.37:0/4077856953 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff750137180 0x7ff750138bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:06.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff75581a640 1 -- 172.21.15.37:0/4077856953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff75013d2e0 msgr2=0x7ff750139110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:06.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff75581a640 1 --2- 172.21.15.37:0/4077856953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff75013d2e0 0x7ff750139110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:06.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff75581a640 10 monclient: _finish_hunting 0 2024-04-26T00:36:06.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff75581a640 1 monclient: found mon.noname-a 2024-04-26T00:36:06.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff75581a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:06.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff75581a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:36:06.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff75581a640 1 -- 172.21.15.37:0/4077856953 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff75009aca0 con 0x7ff750136da0 2024-04-26T00:36:06.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff75581a640 1 -- 172.21.15.37:0/4077856953 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff750136da0 msgr2=0x7ff7501cf8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:06.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.349+0000 7ff75581a640 1 --2- 172.21.15.37:0/4077856953 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff750136da0 0x7ff7501cf8f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7ff74c00c7c0 tx=0x7ff74c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:06.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.353+0000 7ff7467fc640 1 -- 172.21.15.37:0/4077856953 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff74c016020 con 0x7ff750136da0 2024-04-26T00:36:06.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.353+0000 7ff7467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:06.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.353+0000 7ff7467fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:36:06.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.353+0000 7ff7467fc640 10 monclient: dump: 2024-04-26T00:36:06.361 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:06.361 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:06.361 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:06.361 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:06.362 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:06.362 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:06.362 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:06.362 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:06.362 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:06.362 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:06.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.353+0000 7ff7467fc640 10 monclient: _finish_auth 0 2024-04-26T00:36:06.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.353+0000 7ff7467fc640 10 monclient: _check_auth_tickets 2024-04-26T00:36:06.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.353+0000 7ff7467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:06.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.353+0000 7ff7467fc640 1 -- 172.21.15.37:0/4077856953 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff74c00ac60 con 0x7ff750136da0 2024-04-26T00:36:06.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.353+0000 7ff7467fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:36:06.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.353+0000 7ff7467fc640 1 -- 172.21.15.37:0/4077856953 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff74c012660 con 0x7ff750136da0 2024-04-26T00:36:06.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.353+0000 7ff757a7e640 5 monclient: authenticate success, global_id 144648 2024-04-26T00:36:06.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.353+0000 7ff757a7e640 10 monclient: _renew_subs 2024-04-26T00:36:06.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.353+0000 7ff757a7e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:06.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.353+0000 7ff7477fe640 4 set_mon_vals no callback set 2024-04-26T00:36:06.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.353+0000 7ff757a7e640 1 -- 172.21.15.37:0/4077856953 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7500a8210 con 0x7ff750136da0 2024-04-26T00:36:06.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.353+0000 7ff7467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:06.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.353+0000 7ff7467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:36:06.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.353+0000 7ff7467fc640 10 monclient: dump: 2024-04-26T00:36:06.363 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:06.363 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:06.363 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:06.363 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:06.363 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:06.363 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:06.363 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:06.363 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:06.363 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:06.364 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:06.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.353+0000 7ff757a7e640 10 monclient: _renew_subs 2024-04-26T00:36:06.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.353+0000 7ff757a7e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:06.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.353+0000 7ff757a7e640 1 -- 172.21.15.37:0/4077856953 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff75013caa0 con 0x7ff750136da0 2024-04-26T00:36:06.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.353+0000 7ff757a7e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:06.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.353+0000 7ff757a7e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:06.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.353+0000 7ff757a7e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:06.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.353+0000 7ff7467fc640 1 -- 172.21.15.37:0/4077856953 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff74c019020 con 0x7ff750136da0 2024-04-26T00:36:06.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.353+0000 7ff757a7e640 1 -- 172.21.15.37:0/4077856953 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff750066f40 con 0x7ff750136da0 2024-04-26T00:36:06.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.353+0000 7ff7467fc640 1 --2- 172.21.15.37:0/4077856953 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff734035e80 0x7ff734038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:06.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.353+0000 7ff7467fc640 1 -- 172.21.15.37:0/4077856953 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(629..629 src has 370..629) ==== 3971+0+0 (secure 0 0 0) 0x7ff74c048b70 con 0x7ff750136da0 2024-04-26T00:36:06.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.353+0000 7ff755019640 1 --2- 172.21.15.37:0/4077856953 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff734035e80 0x7ff734038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:06.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.353+0000 7ff755019640 10 monclient: get_auth_request con 0x7ff734035e80 auth_method 0 2024-04-26T00:36:06.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.357+0000 7ff755019640 1 --2- 172.21.15.37:0/4077856953 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff734035e80 0x7ff734038340 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff75009c980 tx=0x7ff740007b30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:06.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.357+0000 7ff7467fc640 1 -- 172.21.15.37:0/4077856953 DispatchQueue::entry inject delay of 0.002000 2024-04-26T00:36:06.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.357+0000 7ff7467fc640 1 -- 172.21.15.37:0/4077856953 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff74c020030 con 0x7ff750136da0 2024-04-26T00:36:06.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.357+0000 7ff7467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:06.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.357+0000 7ff7467fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:36:06.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.573+0000 7ff757a7e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:36:06.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.573+0000 7ff757a7e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:36:06.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.573+0000 7ff757a7e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:06.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.573+0000 7ff757a7e640 1 -- 172.21.15.37:0/4077856953 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff7501362b0 con 0x7ff750136da0 2024-04-26T00:36:06.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.577+0000 7ff7467fc640 1 -- 172.21.15.37:0/4077856953 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1558 (secure 0 0 0) 0x7ff74c00ae10 con 0x7ff750136da0 2024-04-26T00:36:06.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.577+0000 7ff7467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:36:06.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.577+0000 7ff7467fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:36:06.575 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":120,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:36:06.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.577+0000 7ff757a7e640 1 -- 172.21.15.37:0/4077856953 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff734035e80 msgr2=0x7ff734038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:06.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.577+0000 7ff757a7e640 1 --2- 172.21.15.37:0/4077856953 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff734035e80 0x7ff734038340 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff75009c980 tx=0x7ff740007b30 comp rx=0 tx=0).stop 2024-04-26T00:36:06.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.577+0000 7ff757a7e640 10 monclient: shutdown 2024-04-26T00:36:06.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.577+0000 7ff757a7e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:36:06.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.577+0000 7ff757a7e640 1 -- 172.21.15.37:0/4077856953 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff750136da0 msgr2=0x7ff7501cf8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:06.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.577+0000 7ff757a7e640 1 --2- 172.21.15.37:0/4077856953 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff750136da0 0x7ff7501cf8f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7ff74c00c7c0 tx=0x7ff74c00cc80 comp rx=0 tx=0).stop 2024-04-26T00:36:06.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.577+0000 7ff757a7e640 1 -- 172.21.15.37:0/4077856953 shutdown_connections 2024-04-26T00:36:06.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.577+0000 7ff757a7e640 1 --2- 172.21.15.37:0/4077856953 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff734035e80 0x7ff734038340 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:06.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.577+0000 7ff757a7e640 1 --2- 172.21.15.37:0/4077856953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff75013d2e0 0x7ff750139110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:06.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.577+0000 7ff757a7e640 1 --2- 172.21.15.37:0/4077856953 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff750137180 0x7ff750138bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:06.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.577+0000 7ff757a7e640 1 --2- 172.21.15.37:0/4077856953 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff750136da0 0x7ff7501cf8f0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:06.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.577+0000 7ff757a7e640 1 -- 172.21.15.37:0/4077856953 >> 172.21.15.37:0/4077856953 conn(0x7ff7500a5f00 msgr2=0x7ff75013e320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:36:06.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.577+0000 7ff757a7e640 1 -- 172.21.15.37:0/4077856953 shutdown_connections 2024-04-26T00:36:06.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:06.577+0000 7ff757a7e640 1 -- 172.21.15.37:0/4077856953 wait complete. 2024-04-26T00:36:09.594 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:36:09.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.725+0000 7f640e063640 10 monclient: get_monmap_and_config 2024-04-26T00:36:09.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.725+0000 7f640e063640 10 monclient: build_initial_monmap 2024-04-26T00:36:09.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.725+0000 7f640e063640 10 monclient: monmap: 2024-04-26T00:36:09.724 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:36:09.724 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:09.724 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:36:09.729855+0000 2024-04-26T00:36:09.725 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:36:09.729855+0000 2024-04-26T00:36:09.725 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:36:09.725 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:09.725 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:36:09.725 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:36:09.725 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:36:09.725 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:36:09.726 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:36:09.726 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:09.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.725+0000 7f640e063640 1 Processor -- start 2024-04-26T00:36:09.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.725+0000 7f640e063640 1 -- start start 2024-04-26T00:36:09.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.725+0000 7f640e063640 10 monclient: init 2024-04-26T00:36:09.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.725+0000 7f640e063640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:09.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.725+0000 7f640e063640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-04-26T00:36:09.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.725+0000 7f640e063640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f64080a0260 0x7f64080a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:09.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.725+0000 7f640e063640 10 monclient(hunting): picked mon.noname-c con 0x7f64080a0260 addr v2:172.21.15.45:3300/0 2024-04-26T00:36:09.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.725+0000 7f640e063640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6408134b30 0x7f6408134f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:09.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.725+0000 7f640e063640 10 monclient(hunting): picked mon.noname-a con 0x7f6408134b30 addr v2:172.21.15.37:3300/0 2024-04-26T00:36:09.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.725+0000 7f640e063640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6408135490 addr v1:172.21.15.45:6789/0 2024-04-26T00:36:09.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.725+0000 7f640e063640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64080a1130 con 0x7f6408135490 2024-04-26T00:36:09.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.725+0000 7f640e063640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:09.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.725+0000 7f640e063640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f640809ec10 con 0x7f6408134b30 2024-04-26T00:36:09.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.725+0000 7f640e063640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:09.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.725+0000 7f640e063640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f640809f0f0 con 0x7f64080a0260 2024-04-26T00:36:09.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.725+0000 7f640e063640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:09.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.725+0000 7f640e063640 10 monclient(hunting): authenticate will time out at 1951.035702s 2024-04-26T00:36:09.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.725+0000 7f6406ffd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6408134b30 0x7f6408134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:09.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.725+0000 7f6406ffd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6408134b30 0x7f6408134f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:55050/0 (socket says 172.21.15.37:55050) 2024-04-26T00:36:09.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.725+0000 7f6406ffd640 1 -- 172.21.15.37:0/2873716811 learned_addr learned my addr 172.21.15.37:0/2873716811 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:36:09.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.725+0000 7f6406ffd640 10 monclient(hunting): get_auth_request con 0x7f6408134b30 auth_method 0 2024-04-26T00:36:09.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.725+0000 7f6406ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:09.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.725+0000 7f6406ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:09.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.725+0000 7f6406ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:09.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.729+0000 7f6406ffd640 1 -- 172.21.15.37:0/2873716811 >> v2:172.21.15.37:3300/0 conn(0x7f6408134b30 msgr2=0x7f6408134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:36:09.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.729+0000 7f6406ffd640 1 -- 172.21.15.37:0/2873716811 >> v2:172.21.15.37:3300/0 conn(0x7f6408134b30 msgr2=0x7f6408134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:36:09.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.729+0000 7f6406ffd640 1 --2- 172.21.15.37:0/2873716811 >> v2:172.21.15.37:3300/0 conn(0x7f6408134b30 0x7f6408134f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:36:09.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.729+0000 7f6406ffd640 1 --2- 172.21.15.37:0/2873716811 >> v2:172.21.15.37:3300/0 conn(0x7f6408134b30 0x7f6408134f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:36:09.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.729+0000 7f64077fe640 1 --2- 172.21.15.37:0/2873716811 >> v2:172.21.15.45:3300/0 conn(0x7f64080a0260 0x7f64080a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:09.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.729+0000 7f64077fe640 10 monclient(hunting): get_auth_request con 0x7f64080a0260 auth_method 0 2024-04-26T00:36:09.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.729+0000 7f64077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:09.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.729+0000 7f64077fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:09.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.729+0000 7f64077fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:09.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.729+0000 7f64077fe640 1 -- 172.21.15.37:0/2873716811 >> v2:172.21.15.45:3300/0 conn(0x7f64080a0260 msgr2=0x7f64080a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:36:09.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.729+0000 7f64077fe640 1 -- 172.21.15.37:0/2873716811 >> v2:172.21.15.45:3300/0 conn(0x7f64080a0260 msgr2=0x7f64080a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:36:09.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.729+0000 7f64077fe640 1 --2- 172.21.15.37:0/2873716811 >> v2:172.21.15.45:3300/0 conn(0x7f64080a0260 0x7f64080a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:36:09.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.729+0000 7f64077fe640 1 --2- 172.21.15.37:0/2873716811 >> v2:172.21.15.45:3300/0 conn(0x7f64080a0260 0x7f64080a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:36:09.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.729+0000 7f6407fff640 1 -- 172.21.15.37:0/2873716811 >> v1:172.21.15.45:6789/0 conn(0x7f6408135490 legacy=0x7f640813be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:09.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.933+0000 7f6406ffd640 1 --2- 172.21.15.37:0/2873716811 >> v2:172.21.15.37:3300/0 conn(0x7f6408134b30 0x7f6408134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:09.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.933+0000 7f6406ffd640 10 monclient(hunting): get_auth_request con 0x7f6408134b30 auth_method 0 2024-04-26T00:36:09.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.933+0000 7f6406ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:09.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.933+0000 7f6406ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:09.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.933+0000 7f6406ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:36:09.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.933+0000 7f64077fe640 1 --2- 172.21.15.37:0/2873716811 >> v2:172.21.15.45:3300/0 conn(0x7f64080a0260 0x7f64080a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:09.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.933+0000 7f64077fe640 10 monclient(hunting): get_auth_request con 0x7f64080a0260 auth_method 0 2024-04-26T00:36:09.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.933+0000 7f64077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:09.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.933+0000 7f64077fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:09.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.933+0000 7f64077fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:36:09.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.933+0000 7f6406ffd640 1 -- 172.21.15.37:0/2873716811 >> v2:172.21.15.37:3300/0 conn(0x7f6408134b30 msgr2=0x7f6408134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:36:09.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.933+0000 7f6406ffd640 1 -- 172.21.15.37:0/2873716811 >> v2:172.21.15.37:3300/0 conn(0x7f6408134b30 msgr2=0x7f6408134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:36:09.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.933+0000 7f6406ffd640 1 --2- 172.21.15.37:0/2873716811 >> v2:172.21.15.37:3300/0 conn(0x7f6408134b30 0x7f6408134f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:36:09.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.933+0000 7f6406ffd640 1 --2- 172.21.15.37:0/2873716811 >> v2:172.21.15.37:3300/0 conn(0x7f6408134b30 0x7f6408134f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:36:09.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.933+0000 7f64077fe640 1 -- 172.21.15.37:0/2873716811 >> v2:172.21.15.45:3300/0 conn(0x7f64080a0260 msgr2=0x7f64080a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:36:09.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.933+0000 7f64077fe640 1 -- 172.21.15.37:0/2873716811 >> v2:172.21.15.45:3300/0 conn(0x7f64080a0260 msgr2=0x7f64080a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:36:09.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.933+0000 7f64077fe640 1 --2- 172.21.15.37:0/2873716811 >> v2:172.21.15.45:3300/0 conn(0x7f64080a0260 0x7f64080a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:36:09.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:09.933+0000 7f64077fe640 1 --2- 172.21.15.37:0/2873716811 >> v2:172.21.15.45:3300/0 conn(0x7f64080a0260 0x7f64080a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:36:10.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:10.337+0000 7f6406ffd640 1 --2- 172.21.15.37:0/2873716811 >> v2:172.21.15.37:3300/0 conn(0x7f6408134b30 0x7f6408134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:10.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:10.337+0000 7f6406ffd640 10 monclient(hunting): get_auth_request con 0x7f6408134b30 auth_method 0 2024-04-26T00:36:10.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:10.337+0000 7f6406ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:10.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:10.337+0000 7f6406ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:10.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:10.337+0000 7f6406ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:36:10.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:10.337+0000 7f64077fe640 1 --2- 172.21.15.37:0/2873716811 >> v2:172.21.15.45:3300/0 conn(0x7f64080a0260 0x7f64080a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:10.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:10.337+0000 7f64077fe640 10 monclient(hunting): get_auth_request con 0x7f64080a0260 auth_method 0 2024-04-26T00:36:10.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:10.337+0000 7f64077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:10.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:10.337+0000 7f64077fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:10.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:10.337+0000 7f64077fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:36:10.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:10.337+0000 7f6406ffd640 1 -- 172.21.15.37:0/2873716811 >> v2:172.21.15.37:3300/0 conn(0x7f6408134b30 msgr2=0x7f6408134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:36:10.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:10.337+0000 7f6406ffd640 1 -- 172.21.15.37:0/2873716811 >> v2:172.21.15.37:3300/0 conn(0x7f6408134b30 msgr2=0x7f6408134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:36:10.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:10.337+0000 7f6406ffd640 1 --2- 172.21.15.37:0/2873716811 >> v2:172.21.15.37:3300/0 conn(0x7f6408134b30 0x7f6408134f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:36:10.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:10.337+0000 7f6406ffd640 1 --2- 172.21.15.37:0/2873716811 >> v2:172.21.15.37:3300/0 conn(0x7f6408134b30 0x7f6408134f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:36:10.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:10.337+0000 7f64077fe640 1 -- 172.21.15.37:0/2873716811 >> v2:172.21.15.45:3300/0 conn(0x7f64080a0260 msgr2=0x7f64080a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:36:10.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:10.337+0000 7f64077fe640 1 -- 172.21.15.37:0/2873716811 >> v2:172.21.15.45:3300/0 conn(0x7f64080a0260 msgr2=0x7f64080a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:36:10.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:10.337+0000 7f64077fe640 1 --2- 172.21.15.37:0/2873716811 >> v2:172.21.15.45:3300/0 conn(0x7f64080a0260 0x7f64080a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:36:10.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:10.337+0000 7f64077fe640 1 --2- 172.21.15.37:0/2873716811 >> v2:172.21.15.45:3300/0 conn(0x7f64080a0260 0x7f64080a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:36:11.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:11.141+0000 7f6406ffd640 1 --2- 172.21.15.37:0/2873716811 >> v2:172.21.15.37:3300/0 conn(0x7f6408134b30 0x7f6408134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:11.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:11.141+0000 7f6406ffd640 10 monclient(hunting): get_auth_request con 0x7f6408134b30 auth_method 0 2024-04-26T00:36:11.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:11.141+0000 7f6406ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:11.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:11.141+0000 7f6406ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:11.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:11.141+0000 7f6406ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:36:11.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:11.141+0000 7f64077fe640 1 --2- 172.21.15.37:0/2873716811 >> v2:172.21.15.45:3300/0 conn(0x7f64080a0260 0x7f64080a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:11.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:11.141+0000 7f6406ffd640 1 -- 172.21.15.37:0/2873716811 >> v2:172.21.15.37:3300/0 conn(0x7f6408134b30 msgr2=0x7f6408134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:36:11.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:11.141+0000 7f64077fe640 10 monclient(hunting): get_auth_request con 0x7f64080a0260 auth_method 0 2024-04-26T00:36:11.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:11.141+0000 7f6406ffd640 1 -- 172.21.15.37:0/2873716811 >> v2:172.21.15.37:3300/0 conn(0x7f6408134b30 msgr2=0x7f6408134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:36:11.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:11.141+0000 7f64077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:11.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:11.141+0000 7f64077fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:11.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:11.141+0000 7f64077fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:36:11.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:11.141+0000 7f6406ffd640 1 --2- 172.21.15.37:0/2873716811 >> v2:172.21.15.37:3300/0 conn(0x7f6408134b30 0x7f6408134f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:36:11.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:11.141+0000 7f6406ffd640 1 --2- 172.21.15.37:0/2873716811 >> v2:172.21.15.37:3300/0 conn(0x7f6408134b30 0x7f6408134f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:36:11.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:11.141+0000 7f64077fe640 1 -- 172.21.15.37:0/2873716811 >> v2:172.21.15.45:3300/0 conn(0x7f64080a0260 msgr2=0x7f64080a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:36:11.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:11.141+0000 7f64077fe640 1 -- 172.21.15.37:0/2873716811 >> v2:172.21.15.45:3300/0 conn(0x7f64080a0260 msgr2=0x7f64080a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:36:11.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:11.141+0000 7f64077fe640 1 --2- 172.21.15.37:0/2873716811 >> v2:172.21.15.45:3300/0 conn(0x7f64080a0260 0x7f64080a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:36:11.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:11.141+0000 7f64077fe640 1 --2- 172.21.15.37:0/2873716811 >> v2:172.21.15.45:3300/0 conn(0x7f64080a0260 0x7f64080a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:36:12.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:12.729+0000 7f64057fa640 10 monclient(hunting): tick 2024-04-26T00:36:12.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:12.729+0000 7f64057fa640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:36:12.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:12.729+0000 7f64057fa640 1 monclient(hunting): continuing hunt 2024-04-26T00:36:12.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:12.729+0000 7f64057fa640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:36:12.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:12.729+0000 7f64057fa640 1 -- 172.21.15.37:0/2873716811 >> v2:172.21.15.45:3300/0 conn(0x7f64080a0260 msgr2=0x7f64080a0640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:36:12.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:12.729+0000 7f64057fa640 1 --2- 172.21.15.37:0/2873716811 >> v2:172.21.15.45:3300/0 conn(0x7f64080a0260 0x7f64080a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:12.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:12.729+0000 7f64057fa640 1 -- 172.21.15.37:0/2873716811 >> v2:172.21.15.37:3300/0 conn(0x7f6408134b30 msgr2=0x7f6408134f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:36:12.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:12.729+0000 7f64057fa640 1 --2- 172.21.15.37:0/2873716811 >> v2:172.21.15.37:3300/0 conn(0x7f6408134b30 0x7f6408134f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:12.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:12.729+0000 7f64057fa640 1 -- 172.21.15.37:0/2873716811 >> v1:172.21.15.45:6789/0 conn(0x7f6408135490 legacy=0x7f640813be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:12.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:12.729+0000 7f64057fa640 10 monclient: _add_conns ranks=[2,0] 2024-04-26T00:36:12.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:12.729+0000 7f64057fa640 1 --2- 172.21.15.37:0/2873716811 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f63e8003700 0x7f63e8005b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:12.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:12.729+0000 7f64057fa640 10 monclient(hunting): picked mon.noname-b con 0x7f63e8003700 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:36:12.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:12.729+0000 7f64057fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f63e8006260 addr v1:172.21.15.37:6789/0 2024-04-26T00:36:12.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:12.729+0000 7f64057fa640 1 -- 172.21.15.37:0/2873716811 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f640806cff0 con 0x7f63e8006260 2024-04-26T00:36:12.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:12.729+0000 7f64057fa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:12.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:12.729+0000 7f64057fa640 1 -- 172.21.15.37:0/2873716811 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f640809f0f0 con 0x7f63e8003700 2024-04-26T00:36:12.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:12.729+0000 7f64057fa640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:12.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:12.729+0000 7f64077fe640 1 -- 172.21.15.37:0/2873716811 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f63e8003700 msgr2=0x7f63e8005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:36:12.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:12.729+0000 7f64077fe640 1 --2- 172.21.15.37:0/2873716811 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f63e8003700 0x7f63e8005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:36:12.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:12.729+0000 7f6406ffd640 1 -- 172.21.15.37:0/2873716811 >> v1:172.21.15.37:6789/0 conn(0x7f63e8006260 legacy=0x7f63e8008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:12.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:12.933+0000 7f64077fe640 1 -- 172.21.15.37:0/2873716811 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f63e8003700 msgr2=0x7f63e8005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:36:12.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:12.933+0000 7f64077fe640 1 --2- 172.21.15.37:0/2873716811 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f63e8003700 0x7f63e8005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:36:13.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.337+0000 7f64077fe640 1 -- 172.21.15.37:0/2873716811 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f63e8003700 msgr2=0x7f63e8005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:36:13.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.337+0000 7f64077fe640 1 --2- 172.21.15.37:0/2873716811 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f63e8003700 0x7f63e8005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:36:13.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.677+0000 7f64067fc640 1 -- 172.21.15.37:0/2873716811 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1195601779 0 0) 0x7f640806cff0 con 0x7f63e8006260 2024-04-26T00:36:13.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.677+0000 7f64067fc640 10 monclient(hunting): _negotiate 2024-04-26T00:36:13.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.677+0000 7f64067fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:13.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.677+0000 7f64067fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:13.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.677+0000 7f64067fc640 10 monclient(hunting): my global_id is 144651 2024-04-26T00:36:13.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.677+0000 7f64067fc640 1 -- 172.21.15.37:0/2873716811 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63f4003530 con 0x7f63e8006260 2024-04-26T00:36:13.681 INFO:tasks.rados.rados.0.smithi045.stdout:2717: done (1 left) 2024-04-26T00:36:13.682 INFO:tasks.rados.rados.0.smithi045.stdout:2718: write (excl) oid 34 current snap is 301 2024-04-26T00:36:13.682 INFO:tasks.rados.rados.0.smithi045.stdout:2716: finishing rollback tid 0 to smithi04526579-27 2024-04-26T00:36:13.682 INFO:tasks.rados.rados.0.smithi045.stdout:2718: seq_num 977 ranges {420439=411457,1421731=480010} 2024-04-26T00:36:13.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.681+0000 7f64067fc640 1 -- 172.21.15.37:0/2873716811 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1278128053 0 0) 0x7f63f4003530 con 0x7f63e8006260 2024-04-26T00:36:13.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.681+0000 7f64067fc640 1 -- 172.21.15.37:0/2873716811 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f63e8003700 msgr2=0x7f63e8005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:36:13.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.681+0000 7f64067fc640 1 --2- 172.21.15.37:0/2873716811 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f63e8003700 0x7f63e8005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:13.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.681+0000 7f64067fc640 10 monclient: _finish_hunting 0 2024-04-26T00:36:13.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.681+0000 7f64067fc640 1 monclient: found mon.noname-a-legacy 2024-04-26T00:36:13.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.681+0000 7f64067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:13.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.681+0000 7f64067fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.37:6789/0 2024-04-26T00:36:13.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.681+0000 7f64067fc640 1 -- 172.21.15.37:0/2873716811 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63e800b060 con 0x7f63e8006260 2024-04-26T00:36:13.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.681+0000 7f64067fc640 10 monclient: _finish_auth 0 2024-04-26T00:36:13.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.681+0000 7f64067fc640 10 monclient: _check_auth_tickets 2024-04-26T00:36:13.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.681+0000 7f64067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:13.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.681+0000 7f64067fc640 1 -- 172.21.15.37:0/2873716811 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 866326386 0 0) 0x7f63f800f040 con 0x7f63e8006260 2024-04-26T00:36:13.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.681+0000 7f640e063640 5 monclient: authenticate success, global_id 144651 2024-04-26T00:36:13.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.681+0000 7f640e063640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:36:13.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.681+0000 7f64067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:13.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.681+0000 7f64067fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-26T00:36:13.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.681+0000 7f64067fc640 10 monclient: dump: 2024-04-26T00:36:13.685 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:13.685 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:13.685 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:13.685 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:13.685 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:13.685 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:13.685 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:13.685 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:13.685 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:13.685 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:13.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.681+0000 7f64067fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] but i'm connected to v1:172.21.15.37:6789/0, reconnecting 2024-04-26T00:36:13.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.681+0000 7f64067fc640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:13.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.681+0000 7f64067fc640 1 -- 172.21.15.37:0/2873716811 >> v1:172.21.15.37:6789/0 conn(0x7f63e8006260 legacy=0x7f63e8008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:13.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.681+0000 7f64067fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:36:13.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.681+0000 7f6407fff640 1 -- 172.21.15.37:0/2873716811 reap_dead start 2024-04-26T00:36:13.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.681+0000 7f64067fc640 1 --2- 172.21.15.37:0/2873716811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63f40039c0 0x7f63f4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:13.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.681+0000 7f64067fc640 10 monclient(hunting): picked mon.b con 0x7f63f40039c0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:36:13.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.681+0000 7f64067fc640 1 --2- 172.21.15.37:0/2873716811 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f63f4004810 0x7f63f4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:13.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.681+0000 7f64067fc640 10 monclient(hunting): picked mon.a con 0x7f63f4004810 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:36:13.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.681+0000 7f64067fc640 1 --2- 172.21.15.37:0/2873716811 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f63e8003700 0x7f63f400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:13.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.681+0000 7f64067fc640 10 monclient(hunting): picked mon.c con 0x7f63e8003700 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:36:13.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.681+0000 7f64067fc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:13.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.681+0000 7f64067fc640 1 -- 172.21.15.37:0/2873716811 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f640809f0f0 con 0x7f63f4004810 2024-04-26T00:36:13.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.681+0000 7f64067fc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:13.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.681+0000 7f64067fc640 1 -- 172.21.15.37:0/2873716811 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f63f4005570 con 0x7f63f40039c0 2024-04-26T00:36:13.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.681+0000 7f64067fc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:13.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.681+0000 7f64067fc640 1 -- 172.21.15.37:0/2873716811 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f63f400bf10 con 0x7f63e8003700 2024-04-26T00:36:13.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.681+0000 7f64067fc640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:13.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.681+0000 7f64067fc640 10 monclient(hunting): _finish_auth 0 2024-04-26T00:36:13.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.681+0000 7f640e063640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-26T00:36:13.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.681+0000 7f6407fff640 1 -- 172.21.15.37:0/2873716811 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f63e8003700 msgr2=0x7f63f400b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:36:13.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.681+0000 7f6407fff640 1 --2- 172.21.15.37:0/2873716811 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f63e8003700 0x7f63f400b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:36:13.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.681+0000 7f6406ffd640 1 --2- 172.21.15.37:0/2873716811 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f63f4004810 0x7f63f4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:13.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.685+0000 7f64077fe640 1 --2- 172.21.15.37:0/2873716811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63f40039c0 0x7f63f4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:13.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.685+0000 7f64077fe640 10 monclient(hunting): get_auth_request con 0x7f63f40039c0 auth_method 0 2024-04-26T00:36:13.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.685+0000 7f64077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:13.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.685+0000 7f64077fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:13.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.685+0000 7f64077fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:36:13.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.685+0000 7f6406ffd640 10 monclient(hunting): get_auth_request con 0x7f63f4004810 auth_method 0 2024-04-26T00:36:13.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.685+0000 7f6406ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:13.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.685+0000 7f6406ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:13.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.685+0000 7f6406ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:36:13.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.685+0000 7f64077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:13.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.685+0000 7f64077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:13.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.685+0000 7f64077fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:36:13.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.685+0000 7f6406ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:13.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.685+0000 7f6406ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:13.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.685+0000 7f6406ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:36:13.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.685+0000 7f64077fe640 10 monclient(hunting): handle_auth_done global_id 144651 payload 293 2024-04-26T00:36:13.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.685+0000 7f64077fe640 1 -- 172.21.15.37:0/2873716811 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f63e8003700 msgr2=0x7f63f400b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:36:13.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.685+0000 7f64077fe640 1 --2- 172.21.15.37:0/2873716811 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f63e8003700 0x7f63f400b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:13.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.685+0000 7f64077fe640 1 -- 172.21.15.37:0/2873716811 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f63f4004810 msgr2=0x7f63f4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:36:13.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.685+0000 7f64077fe640 1 --2- 172.21.15.37:0/2873716811 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f63f4004810 0x7f63f4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:13.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.685+0000 7f64077fe640 10 monclient: _finish_hunting 0 2024-04-26T00:36:13.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.685+0000 7f64077fe640 1 monclient: found mon.b 2024-04-26T00:36:13.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.685+0000 7f64077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:13.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.685+0000 7f64077fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.45:3300/0 2024-04-26T00:36:13.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.685+0000 7f64077fe640 1 -- 172.21.15.37:0/2873716811 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63f4003530 con 0x7f63f40039c0 2024-04-26T00:36:13.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.685+0000 7f64077fe640 10 monclient: _finish_auth 0 2024-04-26T00:36:13.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.685+0000 7f64077fe640 10 monclient: _check_auth_tickets 2024-04-26T00:36:13.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.685+0000 7f64077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:13.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.685+0000 7f6406ffd640 1 --2- 172.21.15.37:0/2873716811 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f63f4004810 0x7f63f4004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:36:13.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.689+0000 7f64077fe640 1 -- 172.21.15.37:0/2873716811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63f40039c0 msgr2=0x7f63f4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:13.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.689+0000 7f64077fe640 1 --2- 172.21.15.37:0/2873716811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63f40039c0 0x7f63f4003dc0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f63f000eae0 tx=0x7f63f001d3d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73b5e1e7852e825a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:13.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.689+0000 7f64067fc640 1 -- 172.21.15.37:0/2873716811 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f00b3070 con 0x7f63f40039c0 2024-04-26T00:36:13.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.689+0000 7f64067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:13.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.689+0000 7f64067fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:36:13.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.689+0000 7f64067fc640 10 monclient: dump: 2024-04-26T00:36:13.692 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:13.692 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:13.692 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:13.692 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:13.692 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:13.692 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:13.692 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:13.692 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:13.692 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:13.692 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:13.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.689+0000 7f640e063640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:36:13.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.689+0000 7f64067fc640 1 -- 172.21.15.37:0/2873716811 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63f00ae410 con 0x7f63f40039c0 2024-04-26T00:36:13.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.689+0000 7f64067fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:36:13.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.689+0000 7f640e063640 10 monclient: get_monmap_and_config success 2024-04-26T00:36:13.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.689+0000 7f640e063640 4 set_mon_vals no callback set 2024-04-26T00:36:13.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.689+0000 7f64067fc640 1 -- 172.21.15.37:0/2873716811 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f000f040 con 0x7f63f40039c0 2024-04-26T00:36:13.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.689+0000 7f64067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:13.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.689+0000 7f64067fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:36:13.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.689+0000 7f64067fc640 10 monclient: dump: 2024-04-26T00:36:13.693 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:13.693 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:13.693 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:13.693 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:13.693 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:13.693 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:13.694 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:13.694 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:13.694 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:13.694 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:13.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.693+0000 7f640e063640 10 monclient: shutdown 2024-04-26T00:36:13.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.693+0000 7f640e063640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:36:13.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.693+0000 7f640e063640 1 -- 172.21.15.37:0/2873716811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63f40039c0 msgr2=0x7f63f4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:13.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.693+0000 7f640e063640 1 --2- 172.21.15.37:0/2873716811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63f40039c0 0x7f63f4003dc0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f63f000eae0 tx=0x7f63f001d3d0 comp rx=0 tx=0).stop 2024-04-26T00:36:13.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.693+0000 7f640e063640 1 -- 172.21.15.37:0/2873716811 shutdown_connections 2024-04-26T00:36:13.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.693+0000 7f640e063640 1 --2- 172.21.15.37:0/2873716811 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f63f4004810 0x7f63f4004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:13.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.693+0000 7f640e063640 1 --2- 172.21.15.37:0/2873716811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63f40039c0 0x7f63f4003dc0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:13.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.693+0000 7f640e063640 1 --2- 172.21.15.37:0/2873716811 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f63e8003700 0x7f63f400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:13.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.693+0000 7f640e063640 1 -- 172.21.15.37:0/2873716811 >> 172.21.15.37:0/2873716811 conn(0x7f64080a1ad0 msgr2=0x7f64080a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:36:13.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.693+0000 7f640e063640 1 -- 172.21.15.37:0/2873716811 shutdown_connections 2024-04-26T00:36:13.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.693+0000 7f640e063640 1 -- 172.21.15.37:0/2873716811 wait complete. 2024-04-26T00:36:13.695 INFO:tasks.rados.rados.0.smithi045.stdout:2718: writing smithi04526579-34 from 420439 to 831896 tid 1 2024-04-26T00:36:13.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.693+0000 7f640e063640 10 monclient: build_initial_monmap 2024-04-26T00:36:13.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.693+0000 7f640e063640 10 monclient: monmap: 2024-04-26T00:36:13.700 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:36:13.700 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:36:13.700 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:36:13.700 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:36:13.700 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:36:13.700 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:36:13.700 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:36:13.700 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:36:13.700 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:36:13.701 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:13.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.693+0000 7f640e063640 1 Processor -- start 2024-04-26T00:36:13.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.693+0000 7f640e063640 1 -- start start 2024-04-26T00:36:13.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.693+0000 7f640e063640 10 monclient: init 2024-04-26T00:36:13.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.693+0000 7f640e063640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:13.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.693+0000 7f640e063640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:36:13.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.693+0000 7f640e063640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63e8003700 0x7f640809c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:13.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.693+0000 7f640e063640 10 monclient(hunting): picked mon.noname-b con 0x7f63e8003700 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:36:13.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.693+0000 7f640e063640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f63f40039c0 0x7f640809ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:13.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f640e063640 10 monclient(hunting): picked mon.noname-c con 0x7f63f40039c0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:36:13.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f640e063640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f63f4004810 0x7f640809cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:13.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f640e063640 10 monclient(hunting): picked mon.noname-a con 0x7f63f4004810 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:36:13.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f640e063640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:13.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f640e063640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f64080a0010 con 0x7f63f4004810 2024-04-26T00:36:13.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f640e063640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:13.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f640e063640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f640809fe90 con 0x7f63e8003700 2024-04-26T00:36:13.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f640e063640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:13.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f640e063640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f6408137e40 con 0x7f63f40039c0 2024-04-26T00:36:13.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f640e063640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:13.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f640e063640 10 monclient(hunting): authenticate will time out at 1955.004110s 2024-04-26T00:36:13.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f6406ffd640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f63f40039c0 msgr2=0x7f640809ca10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:36:13.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f6406ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f63f40039c0 0x7f640809ca10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:36:13.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f6407fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f63f4004810 0x7f640809cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:13.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f64077fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63e8003700 0x7f640809c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:13.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f6407fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f63f4004810 0x7f640809cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:55090/0 (socket says 172.21.15.37:55090) 2024-04-26T00:36:13.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f64077fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63e8003700 0x7f640809c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.37:51940/0 (socket says 172.21.15.37:51940) 2024-04-26T00:36:13.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f6407fff640 1 -- 172.21.15.37:0/1205339337 learned_addr learned my addr 172.21.15.37:0/1205339337 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:36:13.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f6407fff640 10 monclient(hunting): get_auth_request con 0x7f63f4004810 auth_method 0 2024-04-26T00:36:13.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f6407fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:13.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f6407fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:13.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f6407fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:13.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f64077fe640 10 monclient(hunting): get_auth_request con 0x7f63e8003700 auth_method 0 2024-04-26T00:36:13.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f64077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:13.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f64077fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:13.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f64077fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:13.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f6407fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:13.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f6407fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:13.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f6407fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:13.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f64077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:13.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f64077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:13.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f64077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:13.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f6407fff640 10 monclient(hunting): handle_auth_done global_id 144654 payload 824 2024-04-26T00:36:13.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f6407fff640 1 -- 172.21.15.37:0/1205339337 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f63f40039c0 msgr2=0x7f640809ca10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:36:13.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f6407fff640 1 --2- 172.21.15.37:0/1205339337 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f63f40039c0 0x7f640809ca10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:13.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f6407fff640 1 -- 172.21.15.37:0/1205339337 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63e8003700 msgr2=0x7f640809c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:13.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f6407fff640 1 --2- 172.21.15.37:0/1205339337 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63e8003700 0x7f640809c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:13.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f6407fff640 10 monclient: _finish_hunting 0 2024-04-26T00:36:13.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f6407fff640 1 monclient: found mon.noname-a 2024-04-26T00:36:13.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f6407fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:13.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f6407fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:36:13.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f6407fff640 1 -- 172.21.15.37:0/1205339337 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6408123a50 con 0x7f63f4004810 2024-04-26T00:36:13.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f6407fff640 1 -- 172.21.15.37:0/1205339337 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f63f4004810 msgr2=0x7f640809cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:13.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f6407fff640 1 --2- 172.21.15.37:0/1205339337 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f63f4004810 0x7f640809cf50 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f64000012f0 tx=0x7f640000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:13.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f63fffff640 1 -- 172.21.15.37:0/1205339337 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6400017070 con 0x7f63f4004810 2024-04-26T00:36:13.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f63fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:13.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f63fffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:36:13.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f63fffff640 10 monclient: dump: 2024-04-26T00:36:13.705 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:13.705 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:13.705 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:13.705 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:13.705 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:13.705 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:13.705 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:13.705 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:13.705 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:13.705 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:13.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f63fffff640 10 monclient: _finish_auth 0 2024-04-26T00:36:13.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f63fffff640 10 monclient: _check_auth_tickets 2024-04-26T00:36:13.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f63fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:13.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f63fffff640 1 -- 172.21.15.37:0/1205339337 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f640000a5b0 con 0x7f63f4004810 2024-04-26T00:36:13.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f63fffff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:36:13.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f63fffff640 1 -- 172.21.15.37:0/1205339337 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f640000a9a0 con 0x7f63f4004810 2024-04-26T00:36:13.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f63fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:13.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f63fffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:36:13.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f63fffff640 10 monclient: dump: 2024-04-26T00:36:13.706 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:13.706 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:13.706 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:13.706 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:13.706 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:13.706 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:13.706 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:13.706 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:13.706 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:13.706 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:13.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.697+0000 7f6405ffb640 4 set_mon_vals no callback set 2024-04-26T00:36:13.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.701+0000 7f640e063640 5 monclient: authenticate success, global_id 144654 2024-04-26T00:36:13.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.701+0000 7f640e063640 10 monclient: _renew_subs 2024-04-26T00:36:13.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.701+0000 7f640e063640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:13.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.701+0000 7f640e063640 1 -- 172.21.15.37:0/1205339337 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6408123d30 con 0x7f63f4004810 2024-04-26T00:36:13.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.701+0000 7f640e063640 10 monclient: _renew_subs 2024-04-26T00:36:13.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.701+0000 7f640e063640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:13.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.701+0000 7f640e063640 1 -- 172.21.15.37:0/1205339337 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6408123770 con 0x7f63f4004810 2024-04-26T00:36:13.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.701+0000 7f640e063640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:13.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.701+0000 7f640e063640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:13.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.701+0000 7f640e063640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:13.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.701+0000 7f640e063640 1 -- 172.21.15.37:0/1205339337 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6408066f40 con 0x7f63f4004810 2024-04-26T00:36:13.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.701+0000 7f63fffff640 1 -- 172.21.15.37:0/1205339337 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f640001a070 con 0x7f63f4004810 2024-04-26T00:36:13.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.701+0000 7f63fffff640 1 --2- 172.21.15.37:0/1205339337 >> v2:172.21.15.45:6812/2379918623 conn(0x7f63e8036820 0x7f63e8038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:13.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.701+0000 7f63fffff640 1 -- 172.21.15.37:0/1205339337 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(630..630 src has 370..630) ==== 3943+0+0 (secure 0 0 0) 0x7f6400047e30 con 0x7f63f4004810 2024-04-26T00:36:13.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.701+0000 7f64077fe640 1 --2- 172.21.15.37:0/1205339337 >> v2:172.21.15.45:6812/2379918623 conn(0x7f63e8036820 0x7f63e8038ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:13.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.701+0000 7f64077fe640 10 monclient: get_auth_request con 0x7f63e8036820 auth_method 0 2024-04-26T00:36:13.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.705+0000 7f64077fe640 1 --2- 172.21.15.37:0/1205339337 >> v2:172.21.15.45:6812/2379918623 conn(0x7f63e8036820 0x7f63e8038ce0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f63f0000ee0 tx=0x7f63f0004740 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:13.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.705+0000 7f63fffff640 1 -- 172.21.15.37:0/1205339337 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f64080a1130 con 0x7f63f4004810 2024-04-26T00:36:13.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.709+0000 7f63fffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:13.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.709+0000 7f63fffff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:36:13.715 INFO:tasks.rados.rados.0.smithi045.stdout:2718: writing smithi04526579-34 from 1421731 to 1901741 tid 2 2024-04-26T00:36:13.715 INFO:tasks.rados.rados.0.smithi045.stdout:2719: copy_from oid 21 from oid 8 current snap is 301 2024-04-26T00:36:13.715 INFO:tasks.rados.rados.0.smithi045.stdout:2716: finishing rollback tid 1 to smithi04526579-27 2024-04-26T00:36:13.715 INFO:tasks.rados.rados.0.smithi045.stdout:2716: finishing rollback tid 2 to smithi04526579-27 2024-04-26T00:36:13.716 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 1979 (ObjNum 941 snap 291 seq_num 941) dirty exists 2024-04-26T00:36:13.716 INFO:tasks.rados.rados.0.smithi045.stdout:2718: finishing write tid 1 to smithi04526579-34 2024-04-26T00:36:13.716 INFO:tasks.rados.rados.0.smithi045.stdout:2716: done (2 left) 2024-04-26T00:36:13.716 INFO:tasks.rados.rados.0.smithi045.stdout:2720: copy_from oid 42 from oid 25 current snap is 301 2024-04-26T00:36:13.716 INFO:tasks.rados.rados.0.smithi045.stdout:2721: copy_from oid 36 from oid 41 current snap is 301 2024-04-26T00:36:13.716 INFO:tasks.rados.rados.0.smithi045.stdout:2722: read oid 14 snap 294 2024-04-26T00:36:13.716 INFO:tasks.rados.rados.0.smithi045.stdout:2722: expect (ObjNum 866 snap 269 seq_num 866) 2024-04-26T00:36:13.716 INFO:tasks.rados.rados.0.smithi045.stdout:2723: rollback oid 20 current snap is 301 2024-04-26T00:36:13.716 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 20 to 294 2024-04-26T00:36:13.716 INFO:tasks.rados.rados.0.smithi045.stdout:2724: write oid 2 current snap is 301 2024-04-26T00:36:13.716 INFO:tasks.rados.rados.0.smithi045.stdout:2724: seq_num 978 ranges {437246=486706,1363860=623668} 2024-04-26T00:36:13.733 INFO:tasks.rados.rados.0.smithi045.stdout:2724: writing smithi04526579-2 from 437246 to 923952 tid 1 2024-04-26T00:36:13.748 INFO:tasks.rados.rados.0.smithi045.stdout:2724: writing smithi04526579-2 from 1363860 to 1987528 tid 2 2024-04-26T00:36:13.748 INFO:tasks.rados.rados.0.smithi045.stdout:2725: write oid 11 current snap is 301 2024-04-26T00:36:13.748 INFO:tasks.rados.rados.0.smithi045.stdout:2725: seq_num 979 ranges {454053=561955,1422342=483679,2368404=85649} 2024-04-26T00:36:13.757 INFO:tasks.rados.rados.0.smithi045.stdout:2725: writing smithi04526579-11 from 454053 to 1016008 tid 1 2024-04-26T00:36:13.766 INFO:tasks.rados.rados.0.smithi045.stdout:2725: writing smithi04526579-11 from 1422342 to 1906021 tid 2 2024-04-26T00:36:13.771 INFO:tasks.rados.rados.0.smithi045.stdout:2725: writing smithi04526579-11 from 2368404 to 2454053 tid 3 2024-04-26T00:36:13.771 INFO:tasks.rados.rados.0.smithi045.stdout:2726: write (excl) oid 33 current snap is 301 2024-04-26T00:36:13.771 INFO:tasks.rados.rados.0.smithi045.stdout:2723: finishing rollback tid 0 to smithi04526579-20 2024-04-26T00:36:13.771 INFO:tasks.rados.rados.0.smithi045.stdout:2722: expect (ObjNum 866 snap 269 seq_num 866) 2024-04-26T00:36:13.894 INFO:tasks.rados.rados.0.smithi045.stdout:2723: finishing rollback tid 1 to smithi04526579-20 2024-04-26T00:36:13.894 INFO:tasks.rados.rados.0.smithi045.stdout:2723: finishing rollback tid 2 to smithi04526579-20 2024-04-26T00:36:13.894 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 2186 (ObjNum 933 snap 289 seq_num 933) dirty exists 2024-04-26T00:36:13.895 INFO:tasks.rados.rados.0.smithi045.stdout:2718: finishing write tid 2 to smithi04526579-34 2024-04-26T00:36:13.895 INFO:tasks.rados.rados.0.smithi045.stdout:2718: finishing write tid 3 to smithi04526579-34 2024-04-26T00:36:13.895 INFO:tasks.rados.rados.0.smithi045.stdout:2718: finishing write tid 4 to smithi04526579-34 2024-04-26T00:36:13.895 INFO:tasks.rados.rados.0.smithi045.stdout:2718: oid 34 updating version 0 to 2005 2024-04-26T00:36:13.895 INFO:tasks.rados.rados.0.smithi045.stdout:2718: oid 34 version 2005 is already newer than 2003 2024-04-26T00:36:13.895 INFO:tasks.rados.rados.0.smithi045.stdout:2718: oid 34 version 2005 is already newer than 2004 2024-04-26T00:36:13.895 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 2005 (ObjNum 977 snap 301 seq_num 977) dirty exists 2024-04-26T00:36:13.895 INFO:tasks.rados.rados.0.smithi045.stdout:2718: left oid 34 (ObjNum 977 snap 301 seq_num 977) 2024-04-26T00:36:13.895 INFO:tasks.rados.rados.0.smithi045.stdout:2720: finishing copy_from racing read to smithi04526579-42 2024-04-26T00:36:13.895 INFO:tasks.rados.rados.0.smithi045.stdout:2719: finishing copy_from racing read to smithi04526579-21 2024-04-26T00:36:13.895 INFO:tasks.rados.rados.0.smithi045.stdout:2720: finishing copy_from to smithi04526579-42 2024-04-26T00:36:13.895 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 2343 (ObjNum 976 snap 301 seq_num 976) dirty exists 2024-04-26T00:36:13.895 INFO:tasks.rados.rados.0.smithi045.stdout:2721: finishing copy_from racing read to smithi04526579-36 2024-04-26T00:36:13.896 INFO:tasks.rados.rados.0.smithi045.stdout:2726: seq_num 980 ranges {470860=637204,1880824=590036} 2024-04-26T00:36:13.900 INFO:tasks.rados.rados.0.smithi045.stdout:2726: writing smithi04526579-33 from 470860 to 1108064 tid 1 2024-04-26T00:36:13.908 INFO:tasks.rados.rados.0.smithi045.stdout:2726: writing smithi04526579-33 from 1880824 to 2470860 tid 2 2024-04-26T00:36:13.908 INFO:tasks.rados.rados.0.smithi045.stdout:2718: done (8 left) 2024-04-26T00:36:13.909 INFO:tasks.rados.rados.0.smithi045.stdout:2720: done (7 left) 2024-04-26T00:36:13.909 INFO:tasks.rados.rados.0.smithi045.stdout:2722: done (6 left) 2024-04-26T00:36:13.909 INFO:tasks.rados.rados.0.smithi045.stdout:2723: done (5 left) 2024-04-26T00:36:13.909 INFO:tasks.rados.rados.0.smithi045.stdout:2727: read oid 40 snap 294 2024-04-26T00:36:13.909 INFO:tasks.rados.rados.0.smithi045.stdout:2727: expect (ObjNum 886 snap 274 seq_num 886) 2024-04-26T00:36:13.909 INFO:tasks.rados.rados.0.smithi045.stdout:2728: rollback oid 9 current snap is 301 2024-04-26T00:36:13.909 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 9 to 294 2024-04-26T00:36:13.909 INFO:tasks.rados.rados.0.smithi045.stdout:2724: finishing write tid 1 to smithi04526579-2 2024-04-26T00:36:13.909 INFO:tasks.rados.rados.0.smithi045.stdout:2721: finishing copy_from to smithi04526579-36 2024-04-26T00:36:13.909 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 1939 (ObjNum 903 snap 277 seq_num 903) dirty exists 2024-04-26T00:36:13.909 INFO:tasks.rados.rados.0.smithi045.stdout:2719: finishing copy_from to smithi04526579-21 2024-04-26T00:36:13.909 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 1989 (ObjNum 924 snap 287 seq_num 924) dirty exists 2024-04-26T00:36:13.909 INFO:tasks.rados.rados.0.smithi045.stdout:2724: finishing write tid 2 to smithi04526579-2 2024-04-26T00:36:13.909 INFO:tasks.rados.rados.0.smithi045.stdout:2724: finishing write tid 3 to smithi04526579-2 2024-04-26T00:36:13.909 INFO:tasks.rados.rados.0.smithi045.stdout:2719: done (6 left) 2024-04-26T00:36:13.910 INFO:tasks.rados.rados.0.smithi045.stdout:2721: done (5 left) 2024-04-26T00:36:13.910 INFO:tasks.rados.rados.0.smithi045.stdout:2729: rollback oid 24 current snap is 301 2024-04-26T00:36:13.910 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 24 to 294 2024-04-26T00:36:13.910 INFO:tasks.rados.rados.0.smithi045.stdout:2724: finishing write tid 4 to smithi04526579-2 2024-04-26T00:36:13.910 INFO:tasks.rados.rados.0.smithi045.stdout:2724: oid 2 updating version 0 to 2004 2024-04-26T00:36:13.910 INFO:tasks.rados.rados.0.smithi045.stdout:2724: oid 2 version 2004 is already newer than 2002 2024-04-26T00:36:13.910 INFO:tasks.rados.rados.0.smithi045.stdout:2724: oid 2 version 2004 is already newer than 2003 2024-04-26T00:36:13.910 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 2004 (ObjNum 978 snap 301 seq_num 978) dirty exists 2024-04-26T00:36:13.910 INFO:tasks.rados.rados.0.smithi045.stdout:2724: left oid 2 (ObjNum 978 snap 301 seq_num 978) 2024-04-26T00:36:13.910 INFO:tasks.rados.rados.0.smithi045.stdout:2725: finishing write tid 1 to smithi04526579-11 2024-04-26T00:36:13.910 INFO:tasks.rados.rados.0.smithi045.stdout:2725: finishing write tid 2 to smithi04526579-11 2024-04-26T00:36:13.910 INFO:tasks.rados.rados.0.smithi045.stdout:2725: finishing write tid 3 to smithi04526579-11 2024-04-26T00:36:13.910 INFO:tasks.rados.rados.0.smithi045.stdout:2724: done (5 left) 2024-04-26T00:36:13.910 INFO:tasks.rados.rados.0.smithi045.stdout:2730: delete oid 38 current snap is 301 2024-04-26T00:36:13.911 INFO:tasks.rados.rados.0.smithi045.stdout:2725: finishing write tid 4 to smithi04526579-11 2024-04-26T00:36:13.911 INFO:tasks.rados.rados.0.smithi045.stdout:2725: finishing write tid 5 to smithi04526579-11 2024-04-26T00:36:13.911 INFO:tasks.rados.rados.0.smithi045.stdout:2725: oid 11 updating version 0 to 2163 2024-04-26T00:36:13.911 INFO:tasks.rados.rados.0.smithi045.stdout:2725: oid 11 version 2163 is already newer than 2161 2024-04-26T00:36:13.911 INFO:tasks.rados.rados.0.smithi045.stdout:2725: oid 11 updating version 2163 to 2164 2024-04-26T00:36:13.911 INFO:tasks.rados.rados.0.smithi045.stdout:2725: oid 11 version 2164 is already newer than 2162 2024-04-26T00:36:13.911 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 2164 (ObjNum 979 snap 301 seq_num 979) dirty exists 2024-04-26T00:36:13.911 INFO:tasks.rados.rados.0.smithi045.stdout:2725: left oid 11 (ObjNum 979 snap 301 seq_num 979) 2024-04-26T00:36:13.919 INFO:tasks.rados.rados.0.smithi045.stdout:2726: finishing write tid 1 to smithi04526579-33 2024-04-26T00:36:13.920 INFO:tasks.rados.rados.0.smithi045.stdout:2729: finishing rollback tid 1 to smithi04526579-24 2024-04-26T00:36:13.920 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 0 (ObjNum 65 snap 0 seq_num 1174903418) dirty dne 2024-04-26T00:36:13.920 INFO:tasks.rados.rados.0.smithi045.stdout:2727: expect (ObjNum 886 snap 274 seq_num 886) 2024-04-26T00:36:13.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.949+0000 7f640e063640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:36:13.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.949+0000 7f640e063640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:36:13.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.949+0000 7f640e063640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:13.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.949+0000 7f640e063640 1 -- 172.21.15.37:0/1205339337 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6408124150 con 0x7f63f4004810 2024-04-26T00:36:13.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.949+0000 7f63fffff640 1 -- 172.21.15.37:0/1205339337 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f640001a360 con 0x7f63f4004810 2024-04-26T00:36:13.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.949+0000 7f63fffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:36:13.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.949+0000 7f63fffff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:36:13.951 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":124,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:36:13.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.953+0000 7f640e063640 1 -- 172.21.15.37:0/1205339337 >> v2:172.21.15.45:6812/2379918623 conn(0x7f63e8036820 msgr2=0x7f63e8038ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:13.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.953+0000 7f640e063640 1 --2- 172.21.15.37:0/1205339337 >> v2:172.21.15.45:6812/2379918623 conn(0x7f63e8036820 0x7f63e8038ce0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f63f0000ee0 tx=0x7f63f0004740 comp rx=0 tx=0).stop 2024-04-26T00:36:13.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.953+0000 7f640e063640 10 monclient: shutdown 2024-04-26T00:36:13.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.953+0000 7f640e063640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:36:13.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.953+0000 7f640e063640 1 -- 172.21.15.37:0/1205339337 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f63f4004810 msgr2=0x7f640809cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:13.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.953+0000 7f640e063640 1 --2- 172.21.15.37:0/1205339337 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f63f4004810 0x7f640809cf50 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f64000012f0 tx=0x7f640000cb90 comp rx=0 tx=0).stop 2024-04-26T00:36:13.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.953+0000 7f640e063640 1 -- 172.21.15.37:0/1205339337 shutdown_connections 2024-04-26T00:36:13.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.953+0000 7f640e063640 1 --2- 172.21.15.37:0/1205339337 >> v2:172.21.15.45:6812/2379918623 conn(0x7f63e8036820 0x7f63e8038ce0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:13.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.953+0000 7f640e063640 1 --2- 172.21.15.37:0/1205339337 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f63f4004810 0x7f640809cf50 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:13.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.953+0000 7f640e063640 1 --2- 172.21.15.37:0/1205339337 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f63f40039c0 0x7f640809ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:13.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.953+0000 7f640e063640 1 --2- 172.21.15.37:0/1205339337 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f63e8003700 0x7f640809c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:13.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.953+0000 7f640e063640 1 -- 172.21.15.37:0/1205339337 >> 172.21.15.37:0/1205339337 conn(0x7f64080a1ad0 msgr2=0x7f6408132170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:36:13.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.953+0000 7f640e063640 1 -- 172.21.15.37:0/1205339337 shutdown_connections 2024-04-26T00:36:13.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:13.953+0000 7f640e063640 1 -- 172.21.15.37:0/1205339337 wait complete. 2024-04-26T00:36:13.968 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-26T00:36:13.968 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-04-26T00:36:13.968 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:36:14.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb79517640 10 monclient: get_monmap_and_config 2024-04-26T00:36:14.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb79517640 10 monclient: build_initial_monmap 2024-04-26T00:36:14.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb79517640 10 monclient: monmap: 2024-04-26T00:36:14.091 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:36:14.091 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:14.091 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:36:14.096515+0000 2024-04-26T00:36:14.091 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:36:14.096515+0000 2024-04-26T00:36:14.091 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:36:14.091 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:14.091 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:36:14.092 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:36:14.092 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:36:14.092 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:36:14.092 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:36:14.092 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:14.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb79517640 1 Processor -- start 2024-04-26T00:36:14.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb79517640 1 -- start start 2024-04-26T00:36:14.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb79517640 10 monclient: init 2024-04-26T00:36:14.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb79517640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:14.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb79517640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-04-26T00:36:14.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb79517640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb740a0300 0x7fbb7409fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:14.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb79517640 10 monclient(hunting): picked mon.noname-b con 0x7fbb740a0300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:36:14.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb79517640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbb740a06e0 addr v1:172.21.15.45:6789/0 2024-04-26T00:36:14.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb79517640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fbb741352b0 0x7fbb7413cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:14.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb79517640 10 monclient(hunting): picked mon.noname-a con 0x7fbb741352b0 addr v2:172.21.15.37:3300/0 2024-04-26T00:36:14.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb79517640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb7409c190 con 0x7fbb740a06e0 2024-04-26T00:36:14.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb79517640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:14.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb79517640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fbb740a13f0 con 0x7fbb741352b0 2024-04-26T00:36:14.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb79517640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:14.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb79517640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fbb740a1110 con 0x7fbb740a0300 2024-04-26T00:36:14.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb79517640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:14.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb79517640 10 monclient(hunting): authenticate will time out at 1955.402819s 2024-04-26T00:36:14.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb72ffd640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb740a0300 msgr2=0x7fbb7409fc50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:36:14.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb72ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb740a0300 0x7fbb7409fc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:36:14.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb737fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fbb741352b0 0x7fbb7413cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:14.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb737fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fbb741352b0 0x7fbb7413cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:55106/0 (socket says 172.21.15.37:55106) 2024-04-26T00:36:14.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb737fe640 1 -- 172.21.15.37:0/89225912 learned_addr learned my addr 172.21.15.37:0/89225912 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:36:14.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb737fe640 10 monclient(hunting): get_auth_request con 0x7fbb741352b0 auth_method 0 2024-04-26T00:36:14.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:14.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb737fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:14.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb737fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:14.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:14.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:14.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:14.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb737fe640 10 monclient(hunting): handle_auth_done global_id 144657 payload 274 2024-04-26T00:36:14.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb737fe640 1 -- 172.21.15.37:0/89225912 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb740a0300 msgr2=0x7fbb7409fc50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:36:14.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb737fe640 1 --2- 172.21.15.37:0/89225912 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb740a0300 0x7fbb7409fc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:14.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb737fe640 1 -- 172.21.15.37:0/89225912 >> v1:172.21.15.45:6789/0 conn(0x7fbb740a06e0 legacy=0x7fbb74134ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:36:14.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb737fe640 10 monclient: _finish_hunting 0 2024-04-26T00:36:14.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb737fe640 1 monclient: found mon.noname-a 2024-04-26T00:36:14.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:14.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb737fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:36:14.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.093+0000 7fbb737fe640 1 -- 172.21.15.37:0/89225912 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb740a16d0 con 0x7fbb741352b0 2024-04-26T00:36:14.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb737fe640 1 -- 172.21.15.37:0/89225912 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb741352b0 msgr2=0x7fbb7413cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:14.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb737fe640 1 --2- 172.21.15.37:0/89225912 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb741352b0 0x7fbb7413cee0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c009e50 tx=0x7fbb6c01c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5b045f138c29a76 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:14.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb727fc640 1 -- 172.21.15.37:0/89225912 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c0a5730 con 0x7fbb741352b0 2024-04-26T00:36:14.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:14.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb727fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:36:14.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb727fc640 10 monclient: dump: 2024-04-26T00:36:14.100 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:14.101 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:14.101 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:14.101 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:14.101 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:14.101 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:14.101 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:14.101 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:14.101 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:14.101 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:14.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb727fc640 10 monclient: _finish_auth 0 2024-04-26T00:36:14.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb727fc640 10 monclient: _check_auth_tickets 2024-04-26T00:36:14.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:14.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb727fc640 1 -- 172.21.15.37:0/89225912 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb6c0a58d0 con 0x7fbb741352b0 2024-04-26T00:36:14.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb79517640 5 monclient: authenticate success, global_id 144657 2024-04-26T00:36:14.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb79517640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:36:14.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb727fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:36:14.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb79517640 10 monclient: get_monmap_and_config success 2024-04-26T00:36:14.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb79517640 4 set_mon_vals no callback set 2024-04-26T00:36:14.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb727fc640 1 -- 172.21.15.37:0/89225912 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c0a5ca0 con 0x7fbb741352b0 2024-04-26T00:36:14.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:14.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:36:14.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb727fc640 10 monclient: dump: 2024-04-26T00:36:14.102 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:14.102 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:14.102 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:14.102 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:14.102 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:14.103 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:14.103 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:14.103 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:14.103 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:14.103 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:14.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb79517640 10 monclient: shutdown 2024-04-26T00:36:14.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb79517640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:36:14.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb79517640 1 -- 172.21.15.37:0/89225912 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb741352b0 msgr2=0x7fbb7413cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:14.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb79517640 1 --2- 172.21.15.37:0/89225912 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb741352b0 0x7fbb7413cee0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c009e50 tx=0x7fbb6c01c8d0 comp rx=0 tx=0).stop 2024-04-26T00:36:14.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb79517640 1 -- 172.21.15.37:0/89225912 shutdown_connections 2024-04-26T00:36:14.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb79517640 1 --2- 172.21.15.37:0/89225912 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb741352b0 0x7fbb7413cee0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:14.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb79517640 1 --2- 172.21.15.37:0/89225912 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb740a0300 0x7fbb7409fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:14.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb79517640 1 -- 172.21.15.37:0/89225912 >> 172.21.15.37:0/89225912 conn(0x7fbb740a1d50 msgr2=0x7fbb740a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:36:14.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb79517640 1 -- 172.21.15.37:0/89225912 shutdown_connections 2024-04-26T00:36:14.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb79517640 1 -- 172.21.15.37:0/89225912 wait complete. 2024-04-26T00:36:14.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb79517640 10 monclient: build_initial_monmap 2024-04-26T00:36:14.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb79517640 10 monclient: monmap: 2024-04-26T00:36:14.109 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:36:14.109 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:36:14.109 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:36:14.109 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:36:14.109 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:36:14.109 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:36:14.109 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:36:14.109 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:36:14.109 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:36:14.109 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:14.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb79517640 1 Processor -- start 2024-04-26T00:36:14.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb79517640 1 -- start start 2024-04-26T00:36:14.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb79517640 10 monclient: init 2024-04-26T00:36:14.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb79517640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:14.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb79517640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:36:14.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb79517640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb740a0300 0x7fbb7413bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:14.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb79517640 10 monclient(hunting): picked mon.noname-b con 0x7fbb740a0300 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:36:14.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb79517640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb740a06e0 0x7fbb7409c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:14.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb79517640 10 monclient(hunting): picked mon.noname-a con 0x7fbb740a06e0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:36:14.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb79517640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb741352b0 0x7fbb7412fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:14.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb79517640 10 monclient(hunting): picked mon.noname-c con 0x7fbb741352b0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:36:14.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb79517640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:14.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb79517640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7413fc40 con 0x7fbb740a06e0 2024-04-26T00:36:14.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb79517640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:14.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb79517640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7413fdc0 con 0x7fbb740a0300 2024-04-26T00:36:14.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb79517640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:14.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb79517640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fbb7413ff40 con 0x7fbb741352b0 2024-04-26T00:36:14.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb79517640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:14.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb79517640 10 monclient(hunting): authenticate will time out at 1955.411874s 2024-04-26T00:36:14.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb737fe640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb741352b0 msgr2=0x7fbb7412fb70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:36:14.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb6bfff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb740a06e0 0x7fbb7409c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:14.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.101+0000 7fbb737fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb741352b0 0x7fbb7412fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:36:14.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb6bfff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb740a06e0 0x7fbb7409c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:55108/0 (socket says 172.21.15.37:55108) 2024-04-26T00:36:14.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb6bfff640 1 -- 172.21.15.37:0/396120084 learned_addr learned my addr 172.21.15.37:0/396120084 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:36:14.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb6bfff640 10 monclient(hunting): get_auth_request con 0x7fbb740a06e0 auth_method 0 2024-04-26T00:36:14.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:14.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb6bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:14.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:14.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:14.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:14.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:14.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb72ffd640 1 --2- 172.21.15.37:0/396120084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb740a0300 0x7fbb7413bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:14.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb72ffd640 10 monclient(hunting): get_auth_request con 0x7fbb740a0300 auth_method 0 2024-04-26T00:36:14.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:14.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb72ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:14.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:14.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb6bfff640 10 monclient(hunting): handle_auth_done global_id 144660 payload 824 2024-04-26T00:36:14.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb6bfff640 1 -- 172.21.15.37:0/396120084 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb741352b0 msgr2=0x7fbb7412fb70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:36:14.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb6bfff640 1 --2- 172.21.15.37:0/396120084 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb741352b0 0x7fbb7412fb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:14.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb6bfff640 1 -- 172.21.15.37:0/396120084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb740a0300 msgr2=0x7fbb7413bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:14.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb6bfff640 1 --2- 172.21.15.37:0/396120084 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbb740a0300 0x7fbb7413bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:14.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb6bfff640 10 monclient: _finish_hunting 0 2024-04-26T00:36:14.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb6bfff640 1 monclient: found mon.noname-a 2024-04-26T00:36:14.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:14.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb6bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:36:14.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb6bfff640 1 -- 172.21.15.37:0/396120084 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb7411bc80 con 0x7fbb740a06e0 2024-04-26T00:36:14.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb6bfff640 1 -- 172.21.15.37:0/396120084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb740a06e0 msgr2=0x7fbb7409c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:14.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb6bfff640 1 --2- 172.21.15.37:0/396120084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb740a06e0 0x7fbb7409c750 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c00c480 tx=0x7fbb5c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:14.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb6b7fe640 1 -- 172.21.15.37:0/396120084 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c012a40 con 0x7fbb740a06e0 2024-04-26T00:36:14.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb6b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:14.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb6b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:36:14.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb6b7fe640 10 monclient: dump: 2024-04-26T00:36:14.113 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:14.113 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:14.113 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:14.113 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:14.113 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:14.114 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:14.114 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:14.114 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:14.114 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:14.114 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:14.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb6b7fe640 10 monclient: _finish_auth 0 2024-04-26T00:36:14.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb6b7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:36:14.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb6b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:14.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb6b7fe640 1 -- 172.21.15.37:0/396120084 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb5c017070 con 0x7fbb740a06e0 2024-04-26T00:36:14.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb6b7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:36:14.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb6b7fe640 1 -- 172.21.15.37:0/396120084 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c012e40 con 0x7fbb740a06e0 2024-04-26T00:36:14.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb6b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:14.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb6b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:36:14.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb6b7fe640 10 monclient: dump: 2024-04-26T00:36:14.115 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:14.115 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:14.115 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:14.115 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:14.115 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:14.115 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:14.115 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:14.115 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:14.115 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:14.115 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:14.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb79517640 5 monclient: authenticate success, global_id 144660 2024-04-26T00:36:14.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb79517640 10 monclient: _renew_subs 2024-04-26T00:36:14.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb717fa640 4 set_mon_vals no callback set 2024-04-26T00:36:14.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb79517640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:14.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb79517640 1 -- 172.21.15.37:0/396120084 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb7411bf60 con 0x7fbb740a06e0 2024-04-26T00:36:14.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb79517640 10 monclient: _renew_subs 2024-04-26T00:36:14.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb79517640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:14.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb79517640 1 -- 172.21.15.37:0/396120084 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb7411b930 con 0x7fbb740a06e0 2024-04-26T00:36:14.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb79517640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:14.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb79517640 1 --2- 172.21.15.37:0/396120084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb741300b0 0x7fbb741cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:14.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb79517640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:14.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb79517640 1 -- 172.21.15.37:0/396120084 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fbb740a0cd0 con 0x7fbb741300b0 2024-04-26T00:36:14.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb72ffd640 1 --2- 172.21.15.37:0/396120084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb741300b0 0x7fbb741cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:14.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb72ffd640 10 monclient: get_auth_request con 0x7fbb741300b0 auth_method 0 2024-04-26T00:36:14.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:14.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb72ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:14.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:14.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb72ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:14.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb72ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:14.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb72ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:14.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.105+0000 7fbb6b7fe640 1 -- 172.21.15.37:0/396120084 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fbb5c01c460 con 0x7fbb740a06e0 2024-04-26T00:36:14.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.109+0000 7fbb72ffd640 10 monclient(hunting): handle_auth_done global_id 144663 payload 824 2024-04-26T00:36:14.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.109+0000 7fbb72ffd640 1 -- 172.21.15.37:0/396120084 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbb7404f510 con 0x7fbb741300b0 2024-04-26T00:36:14.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.109+0000 7fbb6b7fe640 1 --2- 172.21.15.37:0/396120084 >> v2:172.21.15.45:6812/2379918623 conn(0x7fbb54035e30 0x7fbb540382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:14.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.109+0000 7fbb6b7fe640 1 -- 172.21.15.37:0/396120084 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(630..630 src has 370..630) ==== 3943+0+0 (secure 0 0 0) 0x7fbb5c048940 con 0x7fbb740a06e0 2024-04-26T00:36:14.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.109+0000 7fbb737fe640 1 --2- 172.21.15.37:0/396120084 >> v2:172.21.15.45:6812/2379918623 conn(0x7fbb54035e30 0x7fbb540382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:14.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.109+0000 7fbb737fe640 10 monclient: get_auth_request con 0x7fbb54035e30 auth_method 0 2024-04-26T00:36:14.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.109+0000 7fbb72ffd640 1 -- 172.21.15.37:0/396120084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb741300b0 msgr2=0x7fbb741cf3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:14.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.109+0000 7fbb72ffd640 1 --2- 172.21.15.37:0/396120084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb741300b0 0x7fbb741cf3a0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fbb64003990 tx=0x7fbb6400db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:14.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.109+0000 7fbb6b7fe640 1 -- 172.21.15.37:0/396120084 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb64018070 con 0x7fbb741300b0 2024-04-26T00:36:14.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.109+0000 7fbb6b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:14.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.109+0000 7fbb6b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:36:14.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.109+0000 7fbb6b7fe640 10 monclient: dump: 2024-04-26T00:36:14.118 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:14.118 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:14.118 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:14.118 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:14.118 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:14.118 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:14.118 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:14.118 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:14.118 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:14.118 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:14.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.109+0000 7fbb6b7fe640 1 -- 172.21.15.37:0/396120084 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbb64005c60 con 0x7fbb741300b0 2024-04-26T00:36:14.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.109+0000 7fbb6b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:14.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.109+0000 7fbb6b7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:36:14.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.109+0000 7fbb6b7fe640 1 -- 172.21.15.37:0/396120084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb741300b0 msgr2=0x7fbb741cf3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:14.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.109+0000 7fbb6b7fe640 1 --2- 172.21.15.37:0/396120084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb741300b0 0x7fbb741cf3a0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fbb64003990 tx=0x7fbb6400db90 comp rx=0 tx=0).stop 2024-04-26T00:36:14.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.109+0000 7fbb6b7fe640 1 -- 172.21.15.37:0/396120084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb741300b0 msgr2=0x7fbb741cf3a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:36:14.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.109+0000 7fbb6b7fe640 1 --2- 172.21.15.37:0/396120084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb741300b0 0x7fbb741cf3a0 secure :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fbb64003990 tx=0x7fbb6400db90 comp rx=0 tx=0).stop 2024-04-26T00:36:14.120 INFO:tasks.rados.rados.0.smithi045.stdout:2726: finishing write tid 2 to smithi04526579-33 2024-04-26T00:36:14.120 INFO:tasks.rados.rados.0.smithi045.stdout:2728: finishing rollback tid 1 to smithi04526579-9 2024-04-26T00:36:14.120 INFO:tasks.rados.rados.0.smithi045.stdout:2728: finishing rollback tid 2 to smithi04526579-9 2024-04-26T00:36:14.120 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 2006 (ObjNum 930 snap 288 seq_num 930) dirty exists 2024-04-26T00:36:14.120 INFO:tasks.rados.rados.0.smithi045.stdout:2726: finishing write tid 3 to smithi04526579-33 2024-04-26T00:36:14.120 INFO:tasks.rados.rados.0.smithi045.stdout:2726: finishing write tid 4 to smithi04526579-33 2024-04-26T00:36:14.120 INFO:tasks.rados.rados.0.smithi045.stdout:2726: oid 33 updating version 0 to 2358 2024-04-26T00:36:14.120 INFO:tasks.rados.rados.0.smithi045.stdout:2726: oid 33 version 2358 is already newer than 2357 2024-04-26T00:36:14.121 INFO:tasks.rados.rados.0.smithi045.stdout:2726: oid 33 version 2358 is already newer than 2356 2024-04-26T00:36:14.121 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 2358 (ObjNum 980 snap 301 seq_num 980) dirty exists 2024-04-26T00:36:14.121 INFO:tasks.rados.rados.0.smithi045.stdout:2726: left oid 33 (ObjNum 980 snap 301 seq_num 980) 2024-04-26T00:36:14.121 INFO:tasks.rados.rados.0.smithi045.stdout:2725: done (5 left) 2024-04-26T00:36:14.121 INFO:tasks.rados.rados.0.smithi045.stdout:2726: done (4 left) 2024-04-26T00:36:14.121 INFO:tasks.rados.rados.0.smithi045.stdout:2727: done (3 left) 2024-04-26T00:36:14.121 INFO:tasks.rados.rados.0.smithi045.stdout:2728: done (2 left) 2024-04-26T00:36:14.121 INFO:tasks.rados.rados.0.smithi045.stdout:2729: done (1 left) 2024-04-26T00:36:14.121 INFO:tasks.rados.rados.0.smithi045.stdout:2730: done (0 left) 2024-04-26T00:36:14.121 INFO:tasks.rados.rados.0.smithi045.stdout:2731: snap_remove snap 294 2024-04-26T00:36:14.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.109+0000 7fbb737fe640 1 --2- 172.21.15.37:0/396120084 >> v2:172.21.15.45:6812/2379918623 conn(0x7fbb54035e30 0x7fbb540382f0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fbb7406d330 tx=0x7fbb6c0a7070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:14.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.117+0000 7fbb79517640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:36:14.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.117+0000 7fbb79517640 1 --2- 172.21.15.37:0/396120084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb741cf8e0 0x7fbb741cfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:14.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.117+0000 7fbb79517640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:14.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.117+0000 7fbb79517640 1 -- 172.21.15.37:0/396120084 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fbb741d0230 con 0x7fbb741cf8e0 2024-04-26T00:36:14.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.117+0000 7fbb72ffd640 1 --2- 172.21.15.37:0/396120084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb741cf8e0 0x7fbb741cfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:14.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.117+0000 7fbb72ffd640 10 monclient: get_auth_request con 0x7fbb741cf8e0 auth_method 0 2024-04-26T00:36:14.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.117+0000 7fbb72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:14.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.117+0000 7fbb72ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:14.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.117+0000 7fbb72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:14.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.117+0000 7fbb72ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:14.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.117+0000 7fbb72ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:14.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.117+0000 7fbb72ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:14.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.117+0000 7fbb72ffd640 10 monclient(hunting): handle_auth_done global_id 144666 payload 824 2024-04-26T00:36:14.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.117+0000 7fbb72ffd640 1 -- 172.21.15.37:0/396120084 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbb7409eb60 con 0x7fbb741cf8e0 2024-04-26T00:36:14.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.117+0000 7fbb72ffd640 1 -- 172.21.15.37:0/396120084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb741cf8e0 msgr2=0x7fbb741cfcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:14.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.117+0000 7fbb72ffd640 1 --2- 172.21.15.37:0/396120084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb741cf8e0 0x7fbb741cfcf0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fbb6400b130 tx=0x7fbb640059e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:14.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.117+0000 7fbb6b7fe640 1 -- 172.21.15.37:0/396120084 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb64018040 con 0x7fbb741cf8e0 2024-04-26T00:36:14.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.117+0000 7fbb6b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:14.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.117+0000 7fbb6b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:36:14.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.117+0000 7fbb6b7fe640 10 monclient: dump: 2024-04-26T00:36:14.124 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:14.124 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:14.124 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:14.124 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:14.124 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:14.124 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:14.124 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:14.124 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:14.125 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:14.125 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:14.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.117+0000 7fbb6b7fe640 1 -- 172.21.15.37:0/396120084 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7fbb64005110 con 0x7fbb741cf8e0 2024-04-26T00:36:14.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.117+0000 7fbb6b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:36:14.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.117+0000 7fbb6b7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:36:14.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.117+0000 7fbb6b7fe640 1 -- 172.21.15.37:0/396120084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb741cf8e0 msgr2=0x7fbb741cfcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:14.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.117+0000 7fbb6b7fe640 1 --2- 172.21.15.37:0/396120084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb741cf8e0 0x7fbb741cfcf0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fbb6400b130 tx=0x7fbb640059e0 comp rx=0 tx=0).stop 2024-04-26T00:36:14.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.117+0000 7fbb6b7fe640 1 -- 172.21.15.37:0/396120084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb741cf8e0 msgr2=0x7fbb741cfcf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:36:14.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.117+0000 7fbb6b7fe640 1 --2- 172.21.15.37:0/396120084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb741cf8e0 0x7fbb741cfcf0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:14.126 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:36:14.126 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:36:14.126 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:36:14.126 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:36:14.126 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 124, 2024-04-26T00:36:14.126 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 260660, 2024-04-26T00:36:14.126 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:36:14.127 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:36:14.127 INFO:teuthology.orchestra.run.smithi037.stdout: 1 2024-04-26T00:36:14.127 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:36:14.127 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:36:14.127 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:36:14.127 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:36:14.127 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:36:14.127 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:36:14.127 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:36:14.127 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:36:14.127 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:36:14.127 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:36:14.127 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:36:14.127 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:36:14.127 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:36:14.128 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:36:14.128 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:36:14.128 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:36:14.128 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:36:14.128 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:36:14.128 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:36:14.128 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:36:14.128 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:36:14.128 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:36:14.128 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:36:14.128 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:36:14.128 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:36:14.128 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:36:14.128 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:36:14.128 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:36:14.129 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:36:14.129 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:36:14.129 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:14.129 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:14.129 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:36:14.129 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:36:14.129 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:36:14.129 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:36:14.129 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:36:14.129 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:36:14.129 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:36:14.129 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:36:14.129 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:36:14.129 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:36:14.129 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:36:14.130 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:36:14.130 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:36:14.130 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:36:14.130 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:36:14.130 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:36:14.130 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:36:14.130 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:36:14.130 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:36:14.130 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:36:14.130 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:36:14.130 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:36:14.130 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:36:14.130 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:36:14.130 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:36:14.131 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:36:14.131 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:36:14.131 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:36:14.131 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:36:14.131 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:36:14.131 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:14.131 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:36:14.131 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:14.131 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:36:14.131 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:36:14.131 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:36:14.131 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:36:14.131 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:14.131 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:36:14.131 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:36:14.131 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:14.132 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:14.132 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:14.132 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:36:14.132 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:36:14.132 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:14.132 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:14.132 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:14.132 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:14.132 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:36:14.132 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:36:14.132 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:36:14.132 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:36:14.132 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:36:14.132 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:14.132 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:14.133 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:36:14.133 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:36:14.133 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:36:14.133 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:36:14.133 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:14.133 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:36:14.133 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:36:14.133 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:14.133 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:14.133 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:14.133 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:36:14.133 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:36:14.133 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:14.133 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:14.134 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:14.134 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:14.134 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:36:14.134 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:36:14.134 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:36:14.134 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:36:14.134 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:36:14.134 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:14.134 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:14.134 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:36:14.134 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:36:14.134 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:36:14.134 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:36:14.134 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:14.134 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:36:14.135 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:36:14.135 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:14.135 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:14.135 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:14.135 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:36:14.135 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:36:14.135 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:14.135 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:14.135 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:14.135 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:14.135 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:36:14.135 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:36:14.135 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:36:14.135 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:36:14.136 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:36:14.136 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:14.136 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:14.136 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:14.136 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:36:14.136 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:36:14.136 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:14.136 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:36:14.136 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:36:14.136 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:36:14.136 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:14.136 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:36:14.136 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:36:14.136 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:14.136 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:36:14.137 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:36:14.137 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:36:14.137 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:14.137 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:36:14.137 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:36:14.137 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:14.137 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:36:14.137 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:36:14.137 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 7 2024-04-26T00:36:14.137 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:14.137 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:36:14.137 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:36:14.137 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:14.137 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:36:14.138 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:36:14.138 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:36:14.138 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:14.138 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:14.138 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:14.138 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:36:14.138 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:36:14.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.121+0000 7fbb79517640 1 -- 172.21.15.37:0/396120084 >> v2:172.21.15.45:6812/2379918623 conn(0x7fbb54035e30 msgr2=0x7fbb540382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:14.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.121+0000 7fbb79517640 1 --2- 172.21.15.37:0/396120084 >> v2:172.21.15.45:6812/2379918623 conn(0x7fbb54035e30 0x7fbb540382f0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fbb7406d330 tx=0x7fbb6c0a7070 comp rx=0 tx=0).stop 2024-04-26T00:36:14.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.121+0000 7fbb79517640 10 monclient: shutdown 2024-04-26T00:36:14.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.121+0000 7fbb79517640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:36:14.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.121+0000 7fbb79517640 1 -- 172.21.15.37:0/396120084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb740a06e0 msgr2=0x7fbb7409c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:14.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.121+0000 7fbb79517640 1 --2- 172.21.15.37:0/396120084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb740a06e0 0x7fbb7409c750 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c00c480 tx=0x7fbb5c00c940 comp rx=0 tx=0).stop 2024-04-26T00:36:14.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.121+0000 7fbb737fe640 1 -- 172.21.15.37:0/396120084 reap_dead start 2024-04-26T00:36:14.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.121+0000 7fbb737fe640 1 -- 172.21.15.37:0/396120084 reap_dead start 2024-04-26T00:36:14.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.121+0000 7fbb79517640 1 -- 172.21.15.37:0/396120084 shutdown_connections 2024-04-26T00:36:14.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.121+0000 7fbb79517640 1 -- 172.21.15.37:0/396120084 >> 172.21.15.37:0/396120084 conn(0x7fbb740a1d50 msgr2=0x7fbb740a3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:36:14.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.121+0000 7fbb79517640 1 -- 172.21.15.37:0/396120084 shutdown_connections 2024-04-26T00:36:14.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.121+0000 7fbb79517640 1 -- 172.21.15.37:0/396120084 wait complete. 2024-04-26T00:36:14.140 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:36:14.141 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-04-26T00:36:14.141 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:36:14.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.257+0000 7f8c98fe6640 10 monclient: get_monmap_and_config 2024-04-26T00:36:14.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.257+0000 7f8c98fe6640 10 monclient: build_initial_monmap 2024-04-26T00:36:14.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.257+0000 7f8c98fe6640 10 monclient: monmap: 2024-04-26T00:36:14.258 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:36:14.258 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:14.258 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:36:14.263155+0000 2024-04-26T00:36:14.258 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:36:14.263155+0000 2024-04-26T00:36:14.258 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:36:14.258 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:14.258 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:36:14.258 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:36:14.258 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:36:14.258 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:36:14.258 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:36:14.258 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:14.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c98fe6640 1 Processor -- start 2024-04-26T00:36:14.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c98fe6640 1 -- start start 2024-04-26T00:36:14.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c98fe6640 10 monclient: init 2024-04-26T00:36:14.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c98fe6640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:14.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c98fe6640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-04-26T00:36:14.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c98fe6640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f8c940a3d90 0x7f8c9409bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:14.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c98fe6640 10 monclient(hunting): picked mon.noname-a con 0x7f8c940a3d90 addr v2:172.21.15.37:3300/0 2024-04-26T00:36:14.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c98fe6640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8c940a4170 0x7f8c940a45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:14.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c98fe6640 10 monclient(hunting): picked mon.noname-b con 0x7f8c940a4170 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:36:14.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c98fe6640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8c94133b20 0x7f8c94133f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:14.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c98fe6640 10 monclient(hunting): picked mon.noname-c con 0x7f8c94133b20 addr v2:172.21.15.45:3300/0 2024-04-26T00:36:14.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c98fe6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:14.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c98fe6640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f8c940a5150 con 0x7f8c940a3d90 2024-04-26T00:36:14.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c98fe6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:14.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c98fe6640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f8c9409b280 con 0x7f8c94133b20 2024-04-26T00:36:14.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c98fe6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:14.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c98fe6640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8c9409b560 con 0x7f8c940a4170 2024-04-26T00:36:14.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c98fe6640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:14.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c98fe6640 10 monclient(hunting): authenticate will time out at 1955.568761s 2024-04-26T00:36:14.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c91d9b640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8c940a4170 msgr2=0x7f8c940a45b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:36:14.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c91d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8c940a4170 0x7f8c940a45b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:36:14.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c9259c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f8c940a3d90 0x7f8c9409bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:14.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c9259c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f8c940a3d90 0x7f8c9409bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:55132/0 (socket says 172.21.15.37:55132) 2024-04-26T00:36:14.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c9259c640 1 -- 172.21.15.37:0/284703508 learned_addr learned my addr 172.21.15.37:0/284703508 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:36:14.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c9259c640 10 monclient(hunting): get_auth_request con 0x7f8c940a3d90 auth_method 0 2024-04-26T00:36:14.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c9259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:14.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c9259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:14.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c9259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:14.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c9259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:14.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c9259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:14.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c9259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:14.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c92d9d640 1 --2- 172.21.15.37:0/284703508 >> v2:172.21.15.45:3300/0 conn(0x7f8c94133b20 0x7f8c94133f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:14.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c92d9d640 10 monclient(hunting): get_auth_request con 0x7f8c94133b20 auth_method 0 2024-04-26T00:36:14.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c92d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:14.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c92d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:14.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c92d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:14.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c9259c640 10 monclient(hunting): handle_auth_done global_id 144669 payload 274 2024-04-26T00:36:14.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c9259c640 1 -- 172.21.15.37:0/284703508 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8c940a4170 msgr2=0x7f8c940a45b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:36:14.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c9259c640 1 --2- 172.21.15.37:0/284703508 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8c940a4170 0x7f8c940a45b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:14.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c9259c640 1 -- 172.21.15.37:0/284703508 >> v2:172.21.15.45:3300/0 conn(0x7f8c94133b20 msgr2=0x7f8c94133f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:36:14.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c9259c640 1 --2- 172.21.15.37:0/284703508 >> v2:172.21.15.45:3300/0 conn(0x7f8c94133b20 0x7f8c94133f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:14.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c9259c640 10 monclient: _finish_hunting 0 2024-04-26T00:36:14.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c9259c640 1 monclient: found mon.noname-a 2024-04-26T00:36:14.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c9259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:14.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c9259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:36:14.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.261+0000 7f8c9259c640 1 -- 172.21.15.37:0/284703508 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c940a4af0 con 0x7f8c940a3d90 2024-04-26T00:36:14.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.265+0000 7f8c9259c640 1 -- 172.21.15.37:0/284703508 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8c940a3d90 msgr2=0x7f8c9409bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:14.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.265+0000 7f8c9259c640 1 --2- 172.21.15.37:0/284703508 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8c940a3d90 0x7f8c9409bf40 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f8c94076100 tx=0x7f8c8401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4d4a9191804797f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:14.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.265+0000 7f8c9159a640 1 -- 172.21.15.37:0/284703508 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c840b0070 con 0x7f8c940a3d90 2024-04-26T00:36:14.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.265+0000 7f8c9159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:14.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.265+0000 7f8c9159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:36:14.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.265+0000 7f8c9159a640 10 monclient: dump: 2024-04-26T00:36:14.266 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:14.266 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:14.266 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:14.266 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:14.266 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:14.266 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:14.266 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:14.266 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:14.267 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:14.267 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:14.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.265+0000 7f8c9159a640 10 monclient: _finish_auth 0 2024-04-26T00:36:14.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.265+0000 7f8c9159a640 10 monclient: _check_auth_tickets 2024-04-26T00:36:14.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.265+0000 7f8c9159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:14.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.265+0000 7f8c9159a640 1 -- 172.21.15.37:0/284703508 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c840ab600 con 0x7f8c940a3d90 2024-04-26T00:36:14.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.265+0000 7f8c9159a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:36:14.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.265+0000 7f8c9159a640 1 -- 172.21.15.37:0/284703508 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c840ab9d0 con 0x7f8c940a3d90 2024-04-26T00:36:14.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.265+0000 7f8c98fe6640 5 monclient: authenticate success, global_id 144669 2024-04-26T00:36:14.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.265+0000 7f8c98fe6640 10 monclient: get_monmap_and_config success 2024-04-26T00:36:14.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.265+0000 7f8c98fe6640 4 set_mon_vals no callback set 2024-04-26T00:36:14.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.265+0000 7f8c9159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:14.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.265+0000 7f8c9159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:36:14.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.265+0000 7f8c9159a640 10 monclient: dump: 2024-04-26T00:36:14.267 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:14.268 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:14.268 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:14.268 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:14.268 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:14.268 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:14.268 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:14.268 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:14.268 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:14.268 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:14.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.265+0000 7f8c98fe6640 10 monclient: shutdown 2024-04-26T00:36:14.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.265+0000 7f8c98fe6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:36:14.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.265+0000 7f8c98fe6640 1 -- 172.21.15.37:0/284703508 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8c940a3d90 msgr2=0x7f8c9409bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:14.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.265+0000 7f8c98fe6640 1 --2- 172.21.15.37:0/284703508 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8c940a3d90 0x7f8c9409bf40 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f8c94076100 tx=0x7f8c8401c400 comp rx=0 tx=0).stop 2024-04-26T00:36:14.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.265+0000 7f8c98fe6640 1 -- 172.21.15.37:0/284703508 shutdown_connections 2024-04-26T00:36:14.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.265+0000 7f8c98fe6640 1 --2- 172.21.15.37:0/284703508 >> v2:172.21.15.45:3300/0 conn(0x7f8c94133b20 0x7f8c94133f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:14.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.265+0000 7f8c98fe6640 1 --2- 172.21.15.37:0/284703508 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8c940a4170 0x7f8c940a45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:14.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.265+0000 7f8c98fe6640 1 --2- 172.21.15.37:0/284703508 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8c940a3d90 0x7f8c9409bf40 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:14.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.265+0000 7f8c98fe6640 1 -- 172.21.15.37:0/284703508 >> 172.21.15.37:0/284703508 conn(0x7f8c940aa0a0 msgr2=0x7f8c940a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:36:14.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.265+0000 7f8c98fe6640 1 -- 172.21.15.37:0/284703508 shutdown_connections 2024-04-26T00:36:14.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.265+0000 7f8c98fe6640 1 -- 172.21.15.37:0/284703508 wait complete. 2024-04-26T00:36:14.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.265+0000 7f8c98fe6640 10 monclient: build_initial_monmap 2024-04-26T00:36:14.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.265+0000 7f8c98fe6640 10 monclient: monmap: 2024-04-26T00:36:14.269 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:36:14.269 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:36:14.269 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:36:14.269 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:36:14.269 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:36:14.269 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:36:14.269 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:36:14.269 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:36:14.270 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:36:14.270 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:14.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.265+0000 7f8c98fe6640 1 Processor -- start 2024-04-26T00:36:14.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.265+0000 7f8c98fe6640 1 -- start start 2024-04-26T00:36:14.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.265+0000 7f8c98fe6640 10 monclient: init 2024-04-26T00:36:14.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c98fe6640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:14.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c98fe6640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:36:14.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c98fe6640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8c940a3d90 0x7f8c94132e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:14.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c98fe6640 10 monclient(hunting): picked mon.noname-a con 0x7f8c940a3d90 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:36:14.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c98fe6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c940a4170 0x7f8c941333a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:14.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c98fe6640 10 monclient(hunting): picked mon.noname-b con 0x7f8c940a4170 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:36:14.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c98fe6640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8c94133b20 0x7f8c941cf1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:14.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c98fe6640 10 monclient(hunting): picked mon.noname-c con 0x7f8c94133b20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:36:14.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c98fe6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:14.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c98fe6640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f8c94140c10 con 0x7f8c940a3d90 2024-04-26T00:36:14.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c98fe6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:14.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c98fe6640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8c9409c690 con 0x7f8c940a4170 2024-04-26T00:36:14.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c98fe6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:14.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c98fe6640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8c94140d90 con 0x7f8c94133b20 2024-04-26T00:36:14.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c98fe6640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:14.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c98fe6640 10 monclient(hunting): authenticate will time out at 1955.576478s 2024-04-26T00:36:14.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c92d9d640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8c94133b20 msgr2=0x7f8c941cf1e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:36:14.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c92d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8c94133b20 0x7f8c941cf1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:36:14.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c9259c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8c940a3d90 0x7f8c94132e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:14.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c9259c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8c940a3d90 0x7f8c94132e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:55138/0 (socket says 172.21.15.37:55138) 2024-04-26T00:36:14.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c9259c640 1 -- 172.21.15.37:0/197457524 learned_addr learned my addr 172.21.15.37:0/197457524 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:36:14.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c9259c640 10 monclient(hunting): get_auth_request con 0x7f8c940a3d90 auth_method 0 2024-04-26T00:36:14.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c9259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:14.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c9259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:14.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c9259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:14.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c9259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:14.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c9259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:14.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c9259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:14.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c91d9b640 1 --2- 172.21.15.37:0/197457524 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c940a4170 0x7f8c941333a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:14.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c91d9b640 10 monclient(hunting): get_auth_request con 0x7f8c940a4170 auth_method 0 2024-04-26T00:36:14.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c91d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:14.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c91d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:14.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c91d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:14.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c9259c640 10 monclient(hunting): handle_auth_done global_id 144672 payload 824 2024-04-26T00:36:14.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c9259c640 1 -- 172.21.15.37:0/197457524 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8c94133b20 msgr2=0x7f8c941cf1e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:36:14.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c9259c640 1 --2- 172.21.15.37:0/197457524 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8c94133b20 0x7f8c941cf1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:14.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c9259c640 1 -- 172.21.15.37:0/197457524 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c940a4170 msgr2=0x7f8c941333a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:14.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c9259c640 1 --2- 172.21.15.37:0/197457524 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c940a4170 0x7f8c941333a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:14.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c9259c640 10 monclient: _finish_hunting 0 2024-04-26T00:36:14.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c9259c640 1 monclient: found mon.noname-a 2024-04-26T00:36:14.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c9259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:14.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c9259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:36:14.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c9259c640 1 -- 172.21.15.37:0/197457524 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c9413b480 con 0x7f8c940a3d90 2024-04-26T00:36:14.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c9259c640 1 -- 172.21.15.37:0/197457524 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8c940a3d90 msgr2=0x7f8c94132e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:14.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c9259c640 1 --2- 172.21.15.37:0/197457524 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8c940a3d90 0x7f8c94132e60 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f8c840ae040 tx=0x7f8c8401ac40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:14.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c8affd640 1 -- 172.21.15.37:0/197457524 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c840ba020 con 0x7f8c940a3d90 2024-04-26T00:36:14.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:14.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c8affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:36:14.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c8affd640 10 monclient: dump: 2024-04-26T00:36:14.278 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:14.278 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:14.278 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:14.278 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:14.278 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:14.278 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:14.279 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:14.279 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:14.279 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:14.279 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:14.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c8affd640 10 monclient: _finish_auth 0 2024-04-26T00:36:14.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c8affd640 10 monclient: _check_auth_tickets 2024-04-26T00:36:14.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:14.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c8affd640 1 -- 172.21.15.37:0/197457524 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c840b6630 con 0x7f8c940a3d90 2024-04-26T00:36:14.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c8affd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:36:14.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c8affd640 1 -- 172.21.15.37:0/197457524 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c840b0040 con 0x7f8c940a3d90 2024-04-26T00:36:14.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c98fe6640 5 monclient: authenticate success, global_id 144672 2024-04-26T00:36:14.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:14.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c8affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:36:14.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c8affd640 10 monclient: dump: 2024-04-26T00:36:14.279 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:14.280 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:14.280 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:14.280 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:14.280 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:14.280 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:14.280 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:14.280 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:14.280 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:14.280 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:14.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c98fe6640 10 monclient: _renew_subs 2024-04-26T00:36:14.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c98fe6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:14.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c98fe6640 1 -- 172.21.15.37:0/197457524 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c9413cc10 con 0x7f8c940a3d90 2024-04-26T00:36:14.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c8bfff640 4 set_mon_vals no callback set 2024-04-26T00:36:14.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c98fe6640 10 monclient: _renew_subs 2024-04-26T00:36:14.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c98fe6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:14.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.269+0000 7f8c98fe6640 1 -- 172.21.15.37:0/197457524 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c940a08b0 con 0x7f8c940a3d90 2024-04-26T00:36:14.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.273+0000 7f8c98fe6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:14.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.273+0000 7f8c98fe6640 1 --2- 172.21.15.37:0/197457524 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c9413c5b0 0x7f8c941cf720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:14.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.273+0000 7f8c98fe6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:14.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.273+0000 7f8c98fe6640 1 -- 172.21.15.37:0/197457524 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8c9404f510 con 0x7f8c9413c5b0 2024-04-26T00:36:14.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.273+0000 7f8c8affd640 1 -- 172.21.15.37:0/197457524 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f8c840bd020 con 0x7f8c940a3d90 2024-04-26T00:36:14.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.273+0000 7f8c8affd640 1 --2- 172.21.15.37:0/197457524 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8c74035e30 0x7f8c740382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:14.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.273+0000 7f8c91d9b640 1 --2- 172.21.15.37:0/197457524 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c9413c5b0 0x7f8c941cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:14.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.273+0000 7f8c8affd640 1 -- 172.21.15.37:0/197457524 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(630..630 src has 370..630) ==== 3943+0+0 (secure 0 0 0) 0x7f8c840e2a30 con 0x7f8c940a3d90 2024-04-26T00:36:14.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.273+0000 7f8c91d9b640 10 monclient: get_auth_request con 0x7f8c9413c5b0 auth_method 0 2024-04-26T00:36:14.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.273+0000 7f8c91d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:14.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.273+0000 7f8c91d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:14.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.273+0000 7f8c91d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:14.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.273+0000 7f8c91d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:14.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.273+0000 7f8c91d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:14.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.273+0000 7f8c91d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:14.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.273+0000 7f8c92d9d640 1 --2- 172.21.15.37:0/197457524 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8c74035e30 0x7f8c740382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:14.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.273+0000 7f8c92d9d640 10 monclient: get_auth_request con 0x7f8c74035e30 auth_method 0 2024-04-26T00:36:14.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.273+0000 7f8c91d9b640 10 monclient(hunting): handle_auth_done global_id 194149 payload 824 2024-04-26T00:36:14.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.273+0000 7f8c91d9b640 1 -- 172.21.15.37:0/197457524 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8c9413b200 con 0x7f8c9413c5b0 2024-04-26T00:36:14.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.273+0000 7f8c91d9b640 1 -- 172.21.15.37:0/197457524 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c9413c5b0 msgr2=0x7f8c941cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:14.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.273+0000 7f8c91d9b640 1 --2- 172.21.15.37:0/197457524 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c9413c5b0 0x7f8c941cf720 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8c9413c580 tx=0x7f8c7c00db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:14.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.273+0000 7f8c92d9d640 1 --2- 172.21.15.37:0/197457524 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8c74035e30 0x7f8c740382f0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c009660 tx=0x7f8c8c0098b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:14.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.277+0000 7f8c8affd640 1 -- 172.21.15.37:0/197457524 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c018070 con 0x7f8c9413c5b0 2024-04-26T00:36:14.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.277+0000 7f8c8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:14.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.277+0000 7f8c8affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:36:14.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.277+0000 7f8c8affd640 10 monclient: dump: 2024-04-26T00:36:14.282 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:14.282 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:14.283 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:14.283 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:14.283 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:14.283 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:14.283 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:14.283 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:14.283 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:14.283 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:14.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.277+0000 7f8c8affd640 1 -- 172.21.15.37:0/197457524 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8c7c005c60 con 0x7f8c9413c5b0 2024-04-26T00:36:14.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.277+0000 7f8c8affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:14.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.277+0000 7f8c8affd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:36:14.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.277+0000 7f8c8affd640 1 -- 172.21.15.37:0/197457524 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c9413c5b0 msgr2=0x7f8c941cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:14.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.277+0000 7f8c8affd640 1 --2- 172.21.15.37:0/197457524 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c9413c5b0 0x7f8c941cf720 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8c9413c580 tx=0x7f8c7c00db90 comp rx=0 tx=0).stop 2024-04-26T00:36:14.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.277+0000 7f8c8affd640 1 -- 172.21.15.37:0/197457524 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c9413c5b0 msgr2=0x7f8c941cf720 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:36:14.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.277+0000 7f8c8affd640 1 --2- 172.21.15.37:0/197457524 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c9413c5b0 0x7f8c941cf720 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8c9413c580 tx=0x7f8c7c00db90 comp rx=0 tx=0).stop 2024-04-26T00:36:14.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.281+0000 7f8c98fe6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:36:14.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.281+0000 7f8c98fe6640 1 --2- 172.21.15.37:0/197457524 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c941cfc60 0x7f8c94134b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:14.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.281+0000 7f8c98fe6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:14.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.281+0000 7f8c98fe6640 1 -- 172.21.15.37:0/197457524 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8c941d0040 con 0x7f8c941cfc60 2024-04-26T00:36:14.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.281+0000 7f8c91d9b640 1 --2- 172.21.15.37:0/197457524 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c941cfc60 0x7f8c94134b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:14.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.281+0000 7f8c91d9b640 10 monclient: get_auth_request con 0x7f8c941cfc60 auth_method 0 2024-04-26T00:36:14.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.281+0000 7f8c91d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:14.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.281+0000 7f8c91d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:14.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.281+0000 7f8c91d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:14.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.281+0000 7f8c91d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:14.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.281+0000 7f8c91d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:14.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.281+0000 7f8c91d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:14.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.281+0000 7f8c91d9b640 10 monclient(hunting): handle_auth_done global_id 194152 payload 824 2024-04-26T00:36:14.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.281+0000 7f8c91d9b640 1 -- 172.21.15.37:0/197457524 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8c941338e0 con 0x7f8c941cfc60 2024-04-26T00:36:14.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.285+0000 7f8c91d9b640 1 -- 172.21.15.37:0/197457524 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c941cfc60 msgr2=0x7f8c94134b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:14.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.285+0000 7f8c91d9b640 1 --2- 172.21.15.37:0/197457524 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c941cfc60 0x7f8c94134b30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c006fd0 tx=0x7f8c7c0059e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:14.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.285+0000 7f8c8affd640 1 -- 172.21.15.37:0/197457524 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c018040 con 0x7f8c941cfc60 2024-04-26T00:36:14.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.285+0000 7f8c8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:14.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.285+0000 7f8c8affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:36:14.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.285+0000 7f8c8affd640 10 monclient: dump: 2024-04-26T00:36:14.287 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:14.287 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:14.287 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:14.287 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:14.287 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:14.287 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:14.288 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:14.288 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:14.288 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:14.288 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:14.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.285+0000 7f8c8affd640 1 -- 172.21.15.37:0/197457524 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4536 (secure 0 0 0) 0x7f8c7c005110 con 0x7f8c941cfc60 2024-04-26T00:36:14.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.285+0000 7f8c8affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:36:14.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.285+0000 7f8c8affd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:36:14.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.285+0000 7f8c8affd640 1 -- 172.21.15.37:0/197457524 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c941cfc60 msgr2=0x7f8c94134b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:14.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.285+0000 7f8c8affd640 1 --2- 172.21.15.37:0/197457524 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c941cfc60 0x7f8c94134b30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c006fd0 tx=0x7f8c7c0059e0 comp rx=0 tx=0).stop 2024-04-26T00:36:14.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.285+0000 7f8c8affd640 1 -- 172.21.15.37:0/197457524 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c941cfc60 msgr2=0x7f8c94134b30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:36:14.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.285+0000 7f8c8affd640 1 --2- 172.21.15.37:0/197457524 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8c941cfc60 0x7f8c94134b30 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c006fd0 tx=0x7f8c7c0059e0 comp rx=0 tx=0).stop 2024-04-26T00:36:14.289 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:36:14.289 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:36:14.289 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:36:14.289 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:36:14.289 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 124, 2024-04-26T00:36:14.289 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 22859, 2024-04-26T00:36:14.289 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:36:14.289 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:36:14.289 INFO:teuthology.orchestra.run.smithi037.stdout: 1 2024-04-26T00:36:14.289 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:36:14.289 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:36:14.289 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:36:14.289 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:36:14.289 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:36:14.290 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:36:14.290 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:36:14.290 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:36:14.290 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:36:14.290 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:36:14.290 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:36:14.290 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:36:14.290 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:36:14.290 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:36:14.290 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:36:14.290 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:36:14.290 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:36:14.290 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:36:14.290 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:36:14.291 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:36:14.291 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:36:14.291 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:36:14.291 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:36:14.291 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:36:14.291 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:36:14.291 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:36:14.291 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:36:14.291 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:36:14.291 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:36:14.291 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:36:14.291 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:14.291 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:14.291 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:36:14.291 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:36:14.292 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:36:14.292 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:36:14.292 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:36:14.292 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:36:14.292 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:36:14.292 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:36:14.292 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:36:14.292 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:36:14.292 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:36:14.292 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:36:14.292 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:36:14.292 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:36:14.292 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:36:14.292 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:36:14.292 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:36:14.292 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:36:14.293 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:36:14.293 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:36:14.293 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:36:14.293 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:36:14.293 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:36:14.293 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:36:14.293 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:36:14.293 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:36:14.293 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:36:14.293 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:36:14.293 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:36:14.293 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:36:14.293 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:14.293 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:36:14.294 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:14.294 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:36:14.294 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:36:14.294 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:36:14.294 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:36:14.294 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:14.294 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:36:14.294 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:36:14.294 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:14.294 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:14.294 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:14.294 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:36:14.294 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:36:14.294 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:14.294 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:14.295 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:14.295 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:14.295 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:36:14.295 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:36:14.295 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:36:14.295 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:36:14.295 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:36:14.295 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:14.295 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:14.295 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:36:14.295 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:36:14.295 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:36:14.295 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:36:14.295 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:14.295 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:36:14.296 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:36:14.296 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:14.296 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:14.296 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:14.296 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:36:14.296 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:36:14.296 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:14.296 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:14.296 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:14.296 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:14.296 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:36:14.296 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:36:14.296 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:36:14.296 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:36:14.296 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:36:14.297 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:14.297 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:14.297 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:36:14.297 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:36:14.297 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:36:14.297 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:36:14.297 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:14.297 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:36:14.297 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:36:14.297 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:14.297 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:14.297 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:14.297 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:36:14.297 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:36:14.297 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:14.298 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:14.298 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:14.298 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:14.298 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:36:14.298 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:36:14.298 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:36:14.298 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:36:14.298 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:36:14.298 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:14.298 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:14.298 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:14.298 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:36:14.298 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:36:14.298 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:14.298 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:36:14.299 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:36:14.299 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:36:14.299 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:14.299 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:36:14.299 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:36:14.299 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:14.299 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:36:14.299 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:36:14.299 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:36:14.299 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:14.299 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:14.299 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:14.299 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:36:14.299 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:36:14.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.285+0000 7f8c98fe6640 1 -- 172.21.15.37:0/197457524 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8c74035e30 msgr2=0x7f8c740382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:14.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.285+0000 7f8c98fe6640 1 --2- 172.21.15.37:0/197457524 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8c74035e30 0x7f8c740382f0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c009660 tx=0x7f8c8c0098b0 comp rx=0 tx=0).stop 2024-04-26T00:36:14.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.285+0000 7f8c98fe6640 10 monclient: shutdown 2024-04-26T00:36:14.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.285+0000 7f8c98fe6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:36:14.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.285+0000 7f8c98fe6640 1 -- 172.21.15.37:0/197457524 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8c940a3d90 msgr2=0x7f8c94132e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:14.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.285+0000 7f8c98fe6640 1 --2- 172.21.15.37:0/197457524 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8c940a3d90 0x7f8c94132e60 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f8c840ae040 tx=0x7f8c8401ac40 comp rx=0 tx=0).stop 2024-04-26T00:36:14.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.285+0000 7f8c92d9d640 1 -- 172.21.15.37:0/197457524 reap_dead start 2024-04-26T00:36:14.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.285+0000 7f8c92d9d640 1 -- 172.21.15.37:0/197457524 reap_dead start 2024-04-26T00:36:14.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.285+0000 7f8c98fe6640 1 -- 172.21.15.37:0/197457524 shutdown_connections 2024-04-26T00:36:14.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.285+0000 7f8c98fe6640 1 -- 172.21.15.37:0/197457524 >> 172.21.15.37:0/197457524 conn(0x7f8c940aa0a0 msgr2=0x7f8c940a22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:36:14.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.285+0000 7f8c98fe6640 1 -- 172.21.15.37:0/197457524 shutdown_connections 2024-04-26T00:36:14.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:14.285+0000 7f8c98fe6640 1 -- 172.21.15.37:0/197457524 wait complete. 2024-04-26T00:36:14.302 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:36:14.302 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-04-26T00:36:14.685 INFO:tasks.rados.rados.0.smithi045.stdout:2731: done (0 left) 2024-04-26T00:36:14.686 INFO:tasks.rados.rados.0.smithi045.stdout:2732: copy_from oid 36 from oid 33 current snap is 301 2024-04-26T00:36:14.686 INFO:tasks.rados.rados.0.smithi045.stdout:2733: write (excl) oid 21 current snap is 301 2024-04-26T00:36:14.686 INFO:tasks.rados.rados.0.smithi045.stdout:2733: seq_num 982 ranges {504474=504055,1714141=631006} 2024-04-26T00:36:14.705 INFO:tasks.rados.rados.0.smithi045.stdout:2733: writing smithi04526579-21 from 504474 to 1008529 tid 1 2024-04-26T00:36:14.730 INFO:tasks.rados.rados.0.smithi045.stdout:2733: writing smithi04526579-21 from 1714141 to 2345147 tid 2 2024-04-26T00:36:14.730 INFO:tasks.rados.rados.0.smithi045.stdout:2734: write (excl) oid 42 current snap is 301 2024-04-26T00:36:14.730 INFO:tasks.rados.rados.0.smithi045.stdout:2732: finishing copy_from racing read to smithi04526579-36 2024-04-26T00:36:14.730 INFO:tasks.rados.rados.0.smithi045.stdout:2733: finishing write tid 1 to smithi04526579-21 2024-04-26T00:36:14.730 INFO:tasks.rados.rados.0.smithi045.stdout:2734: seq_num 983 ranges {521281=579304,1656270=774664} 2024-04-26T00:36:14.749 INFO:tasks.rados.rados.0.smithi045.stdout:2734: writing smithi04526579-42 from 521281 to 1100585 tid 1 2024-04-26T00:36:14.768 INFO:tasks.rados.rados.0.smithi045.stdout:2734: writing smithi04526579-42 from 1656270 to 2430934 tid 2 2024-04-26T00:36:14.768 INFO:tasks.rados.rados.0.smithi045.stdout:2735: read oid 16 snap -1 2024-04-26T00:36:14.768 INFO:tasks.rados.rados.0.smithi045.stdout:2735: expect (ObjNum 921 snap 283 seq_num 921) 2024-04-26T00:36:14.768 INFO:tasks.rados.rados.0.smithi045.stdout:2736: snap_create 2024-04-26T00:36:14.768 INFO:tasks.rados.rados.0.smithi045.stdout:2732: finishing copy_from to smithi04526579-36 2024-04-26T00:36:14.768 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 2358 (ObjNum 980 snap 301 seq_num 980) dirty exists 2024-04-26T00:36:14.768 INFO:tasks.rados.rados.0.smithi045.stdout:2733: finishing write tid 2 to smithi04526579-21 2024-04-26T00:36:14.768 INFO:tasks.rados.rados.0.smithi045.stdout:2733: finishing write tid 3 to smithi04526579-21 2024-04-26T00:36:14.769 INFO:tasks.rados.rados.0.smithi045.stdout:2733: finishing write tid 4 to smithi04526579-21 2024-04-26T00:36:14.769 INFO:tasks.rados.rados.0.smithi045.stdout:2733: oid 21 updating version 0 to 1992 2024-04-26T00:36:14.769 INFO:tasks.rados.rados.0.smithi045.stdout:2733: oid 21 version 1992 is already newer than 1991 2024-04-26T00:36:14.769 INFO:tasks.rados.rados.0.smithi045.stdout:2733: oid 21 version 1992 is already newer than 1990 2024-04-26T00:36:14.769 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 1992 (ObjNum 982 snap 301 seq_num 982) dirty exists 2024-04-26T00:36:14.769 INFO:tasks.rados.rados.0.smithi045.stdout:2733: left oid 21 (ObjNum 982 snap 301 seq_num 982) 2024-04-26T00:36:14.769 INFO:tasks.rados.rados.0.smithi045.stdout:2734: finishing write tid 1 to smithi04526579-42 2024-04-26T00:36:14.780 INFO:tasks.rados.rados.0.smithi045.stdout:2734: finishing write tid 2 to smithi04526579-42 2024-04-26T00:36:14.780 INFO:tasks.rados.rados.0.smithi045.stdout:2734: finishing write tid 3 to smithi04526579-42 2024-04-26T00:36:14.781 INFO:tasks.rados.rados.0.smithi045.stdout:2734: finishing write tid 4 to smithi04526579-42 2024-04-26T00:36:14.781 INFO:tasks.rados.rados.0.smithi045.stdout:2734: oid 42 updating version 0 to 2346 2024-04-26T00:36:14.781 INFO:tasks.rados.rados.0.smithi045.stdout:2734: oid 42 version 2346 is already newer than 2345 2024-04-26T00:36:14.781 INFO:tasks.rados.rados.0.smithi045.stdout:2734: oid 42 version 2346 is already newer than 2344 2024-04-26T00:36:14.781 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 2346 (ObjNum 983 snap 301 seq_num 983) dirty exists 2024-04-26T00:36:14.781 INFO:tasks.rados.rados.0.smithi045.stdout:2734: left oid 42 (ObjNum 983 snap 301 seq_num 983) 2024-04-26T00:36:14.781 INFO:tasks.rados.rados.0.smithi045.stdout:2735: expect (ObjNum 921 snap 283 seq_num 921) 2024-04-26T00:36:15.998 INFO:tasks.rados.rados.0.smithi045.stdout:2732: done (4 left) 2024-04-26T00:36:15.998 INFO:tasks.rados.rados.0.smithi045.stdout:2733: done (3 left) 2024-04-26T00:36:15.998 INFO:tasks.rados.rados.0.smithi045.stdout:2734: done (2 left) 2024-04-26T00:36:15.998 INFO:tasks.rados.rados.0.smithi045.stdout:2735: done (1 left) 2024-04-26T00:36:15.998 INFO:tasks.rados.rados.0.smithi045.stdout:2736: done (0 left) 2024-04-26T00:36:15.998 INFO:tasks.rados.rados.0.smithi045.stdout:2737: write (excl) oid 46 current snap is 302 2024-04-26T00:36:15.998 INFO:tasks.rados.rados.0.smithi045.stdout:2737: seq_num 984 ranges {538088=654553,1714752=634675} 2024-04-26T00:36:16.009 INFO:tasks.rados.rados.0.smithi045.stdout:2737: writing smithi04526579-46 from 538088 to 1192641 tid 1 2024-04-26T00:36:16.024 INFO:tasks.rados.rados.0.smithi045.stdout:2737: writing smithi04526579-46 from 1714752 to 2349427 tid 2 2024-04-26T00:36:16.024 INFO:tasks.rados.rados.0.smithi045.stdout:2738: read oid 50 snap -1 2024-04-26T00:36:16.024 INFO:tasks.rados.rados.0.smithi045.stdout:2738: expect deleted 2024-04-26T00:36:16.025 INFO:tasks.rados.rados.0.smithi045.stdout:2739: delete oid 11 current snap is 302 2024-04-26T00:36:16.030 INFO:tasks.rados.rados.0.smithi045.stdout:2738: done (2 left) 2024-04-26T00:36:16.030 INFO:tasks.rados.rados.0.smithi045.stdout:2739: done (1 left) 2024-04-26T00:36:16.031 INFO:tasks.rados.rados.0.smithi045.stdout:2740: write (excl) oid 21 current snap is 302 2024-04-26T00:36:16.031 INFO:tasks.rados.rados.0.smithi045.stdout:2740: seq_num 986 ranges {571702=405051,1431716=638344,2560365=11337} 2024-04-26T00:36:16.036 INFO:tasks.rados.rados.0.smithi045.stdout:2740: writing smithi04526579-21 from 571702 to 976753 tid 1 2024-04-26T00:36:16.042 INFO:tasks.rados.rados.0.smithi045.stdout:2740: writing smithi04526579-21 from 1431716 to 2070060 tid 2 2024-04-26T00:36:16.045 INFO:tasks.rados.rados.0.smithi045.stdout:2740: writing smithi04526579-21 from 2560365 to 2571702 tid 3 2024-04-26T00:36:16.045 INFO:tasks.rados.rados.0.smithi045.stdout:2741: write oid 34 current snap is 302 2024-04-26T00:36:16.045 INFO:tasks.rados.rados.0.smithi045.stdout:2737: finishing write tid 1 to smithi04526579-46 2024-04-26T00:36:16.045 INFO:tasks.rados.rados.0.smithi045.stdout:2737: finishing write tid 2 to smithi04526579-46 2024-04-26T00:36:16.045 INFO:tasks.rados.rados.0.smithi045.stdout:2737: finishing write tid 3 to smithi04526579-46 2024-04-26T00:36:16.045 INFO:tasks.rados.rados.0.smithi045.stdout:2737: finishing write tid 4 to smithi04526579-46 2024-04-26T00:36:16.045 INFO:tasks.rados.rados.0.smithi045.stdout:2737: oid 46 updating version 0 to 2361 2024-04-26T00:36:16.045 INFO:tasks.rados.rados.0.smithi045.stdout:2737: oid 46 version 2361 is already newer than 2360 2024-04-26T00:36:16.045 INFO:tasks.rados.rados.0.smithi045.stdout:2737: oid 46 version 2361 is already newer than 2359 2024-04-26T00:36:16.045 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 2361 (ObjNum 984 snap 302 seq_num 984) dirty exists 2024-04-26T00:36:16.046 INFO:tasks.rados.rados.0.smithi045.stdout:2737: left oid 46 (ObjNum 984 snap 302 seq_num 984) 2024-04-26T00:36:16.046 INFO:tasks.rados.rados.0.smithi045.stdout:2741: seq_num 987 ranges {588509=480300,1773845=782002} 2024-04-26T00:36:16.051 INFO:tasks.rados.rados.0.smithi045.stdout:2741: writing smithi04526579-34 from 588509 to 1068809 tid 1 2024-04-26T00:36:16.059 INFO:tasks.rados.rados.0.smithi045.stdout:2741: writing smithi04526579-34 from 1773845 to 2555847 tid 2 2024-04-26T00:36:16.060 INFO:tasks.rados.rados.0.smithi045.stdout:2737: done (2 left) 2024-04-26T00:36:16.060 INFO:tasks.rados.rados.0.smithi045.stdout:2742: snap_remove snap 301 2024-04-26T00:36:17.000 INFO:tasks.rados.rados.0.smithi045.stdout:2742: done (2 left) 2024-04-26T00:36:17.000 INFO:tasks.rados.rados.0.smithi045.stdout:2743: read oid 33 snap -1 2024-04-26T00:36:17.000 INFO:tasks.rados.rados.0.smithi045.stdout:2743: expect (ObjNum 980 snap 301 seq_num 980) 2024-04-26T00:36:17.000 INFO:tasks.rados.rados.0.smithi045.stdout:2744: rollback oid 4 current snap is 302 2024-04-26T00:36:17.001 INFO:tasks.rados.rados.0.smithi045.stdout:2744: done (3 left) 2024-04-26T00:36:17.001 INFO:tasks.rados.rados.0.smithi045.stdout:2745: read oid 6 snap -1 2024-04-26T00:36:17.001 INFO:tasks.rados.rados.0.smithi045.stdout:2745: expect (ObjNum 542 snap 167 seq_num 542) 2024-04-26T00:36:17.001 INFO:tasks.rados.rados.0.smithi045.stdout:2746: snap_create 2024-04-26T00:36:17.001 INFO:tasks.rados.rados.0.smithi045.stdout:2740: finishing write tid 1 to smithi04526579-21 2024-04-26T00:36:17.001 INFO:tasks.rados.rados.0.smithi045.stdout:2740: finishing write tid 2 to smithi04526579-21 2024-04-26T00:36:17.001 INFO:tasks.rados.rados.0.smithi045.stdout:2740: finishing write tid 3 to smithi04526579-21 2024-04-26T00:36:17.001 INFO:tasks.rados.rados.0.smithi045.stdout:2740: finishing write tid 4 to smithi04526579-21 2024-04-26T00:36:17.001 INFO:tasks.rados.rados.0.smithi045.stdout:2740: finishing write tid 5 to smithi04526579-21 2024-04-26T00:36:17.001 INFO:tasks.rados.rados.0.smithi045.stdout:2740: oid 21 updating version 0 to 1995 2024-04-26T00:36:17.001 INFO:tasks.rados.rados.0.smithi045.stdout:2740: oid 21 updating version 1995 to 1996 2024-04-26T00:36:17.001 INFO:tasks.rados.rados.0.smithi045.stdout:2740: oid 21 version 1996 is already newer than 1994 2024-04-26T00:36:17.001 INFO:tasks.rados.rados.0.smithi045.stdout:2740: oid 21 version 1996 is already newer than 1993 2024-04-26T00:36:17.001 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 1996 (ObjNum 986 snap 302 seq_num 986) dirty exists 2024-04-26T00:36:17.002 INFO:tasks.rados.rados.0.smithi045.stdout:2740: left oid 21 (ObjNum 986 snap 302 seq_num 986) 2024-04-26T00:36:17.002 INFO:tasks.rados.rados.0.smithi045.stdout:2741: finishing write tid 1 to smithi04526579-34 2024-04-26T00:36:17.002 INFO:tasks.rados.rados.0.smithi045.stdout:2741: finishing write tid 2 to smithi04526579-34 2024-04-26T00:36:17.002 INFO:tasks.rados.rados.0.smithi045.stdout:2741: finishing write tid 3 to smithi04526579-34 2024-04-26T00:36:17.002 INFO:tasks.rados.rados.0.smithi045.stdout:2741: finishing write tid 4 to smithi04526579-34 2024-04-26T00:36:17.002 INFO:tasks.rados.rados.0.smithi045.stdout:2741: oid 34 updating version 0 to 2007 2024-04-26T00:36:17.002 INFO:tasks.rados.rados.0.smithi045.stdout:2741: oid 34 version 2007 is already newer than 2006 2024-04-26T00:36:17.002 INFO:tasks.rados.rados.0.smithi045.stdout:2741: oid 34 updating version 2007 to 2008 2024-04-26T00:36:17.002 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 2008 (ObjNum 987 snap 302 seq_num 987) dirty exists 2024-04-26T00:36:17.002 INFO:tasks.rados.rados.0.smithi045.stdout:2741: left oid 34 (ObjNum 987 snap 302 seq_num 987) 2024-04-26T00:36:17.018 INFO:tasks.rados.rados.0.smithi045.stdout:2743: expect (ObjNum 980 snap 301 seq_num 980) 2024-04-26T00:36:17.266 INFO:tasks.rados.rados.0.smithi045.stdout:2745: expect (ObjNum 542 snap 167 seq_num 542) 2024-04-26T00:36:18.014 INFO:tasks.rados.rados.0.smithi045.stdout:2740: done (4 left) 2024-04-26T00:36:18.014 INFO:tasks.rados.rados.0.smithi045.stdout:2741: done (3 left) 2024-04-26T00:36:18.014 INFO:tasks.rados.rados.0.smithi045.stdout:2743: done (2 left) 2024-04-26T00:36:18.014 INFO:tasks.rados.rados.0.smithi045.stdout:2745: done (1 left) 2024-04-26T00:36:18.014 INFO:tasks.rados.rados.0.smithi045.stdout:2746: done (0 left) 2024-04-26T00:36:18.014 INFO:tasks.rados.rados.0.smithi045.stdout:2747: read oid 48 snap 302 2024-04-26T00:36:18.014 INFO:tasks.rados.rados.0.smithi045.stdout:2747: expect (ObjNum 938 snap 290 seq_num 938) 2024-04-26T00:36:18.014 INFO:tasks.rados.rados.0.smithi045.stdout:2748: rollback oid 34 current snap is 303 2024-04-26T00:36:18.015 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 34 to 302 2024-04-26T00:36:18.015 INFO:tasks.rados.rados.0.smithi045.stdout:2749: write (excl) oid 19 current snap is 303 2024-04-26T00:36:18.015 INFO:tasks.rados.rados.0.smithi045.stdout:2749: seq_num 988 ranges {605316=555549,1832327=642013} 2024-04-26T00:36:18.026 INFO:tasks.rados.rados.0.smithi045.stdout:2749: writing smithi04526579-19 from 605316 to 1160865 tid 1 2024-04-26T00:36:18.041 INFO:tasks.rados.rados.0.smithi045.stdout:2749: writing smithi04526579-19 from 1832327 to 2474340 tid 2 2024-04-26T00:36:18.041 INFO:tasks.rados.rados.0.smithi045.stdout:2750: delete oid 39 current snap is 303 2024-04-26T00:36:18.041 INFO:tasks.rados.rados.0.smithi045.stdout:2747: expect (ObjNum 938 snap 290 seq_num 938) 2024-04-26T00:36:18.260 INFO:tasks.rados.rados.0.smithi045.stdout:2748: finishing rollback tid 0 to smithi04526579-34 2024-04-26T00:36:18.260 INFO:tasks.rados.rados.0.smithi045.stdout:2748: finishing rollback tid 1 to smithi04526579-34 2024-04-26T00:36:18.260 INFO:tasks.rados.rados.0.smithi045.stdout:2748: finishing rollback tid 2 to smithi04526579-34 2024-04-26T00:36:18.260 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 2011 (ObjNum 987 snap 302 seq_num 987) dirty exists 2024-04-26T00:36:18.261 INFO:tasks.rados.rados.0.smithi045.stdout:2749: finishing write tid 1 to smithi04526579-19 2024-04-26T00:36:18.261 INFO:tasks.rados.rados.0.smithi045.stdout:2749: finishing write tid 2 to smithi04526579-19 2024-04-26T00:36:18.261 INFO:tasks.rados.rados.0.smithi045.stdout:2749: finishing write tid 3 to smithi04526579-19 2024-04-26T00:36:18.261 INFO:tasks.rados.rados.0.smithi045.stdout:2749: finishing write tid 4 to smithi04526579-19 2024-04-26T00:36:18.261 INFO:tasks.rados.rados.0.smithi045.stdout:2749: oid 19 updating version 0 to 2347 2024-04-26T00:36:18.261 INFO:tasks.rados.rados.0.smithi045.stdout:2749: oid 19 updating version 2347 to 2349 2024-04-26T00:36:18.261 INFO:tasks.rados.rados.0.smithi045.stdout:2749: oid 19 version 2349 is already newer than 2348 2024-04-26T00:36:18.261 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 2349 (ObjNum 988 snap 303 seq_num 988) dirty exists 2024-04-26T00:36:18.261 INFO:tasks.rados.rados.0.smithi045.stdout:2749: left oid 19 (ObjNum 988 snap 303 seq_num 988) 2024-04-26T00:36:18.261 INFO:tasks.rados.rados.0.smithi045.stdout:2747: done (3 left) 2024-04-26T00:36:18.261 INFO:tasks.rados.rados.0.smithi045.stdout:2748: done (2 left) 2024-04-26T00:36:18.261 INFO:tasks.rados.rados.0.smithi045.stdout:2749: done (1 left) 2024-04-26T00:36:18.262 INFO:tasks.rados.rados.0.smithi045.stdout:2750: done (0 left) 2024-04-26T00:36:18.262 INFO:tasks.rados.rados.0.smithi045.stdout:2751: read oid 44 snap -1 2024-04-26T00:36:18.262 INFO:tasks.rados.rados.0.smithi045.stdout:2751: expect (ObjNum 952 snap 295 seq_num 952) 2024-04-26T00:36:18.262 INFO:tasks.rados.rados.0.smithi045.stdout:2752: write oid 8 current snap is 303 2024-04-26T00:36:18.262 INFO:tasks.rados.rados.0.smithi045.stdout:2752: seq_num 990 ranges {638930=422400,1665644=645682} 2024-04-26T00:36:18.275 INFO:tasks.rados.rados.0.smithi045.stdout:2752: writing smithi04526579-8 from 638930 to 1061330 tid 1 2024-04-26T00:36:18.293 INFO:tasks.rados.rados.0.smithi045.stdout:2752: writing smithi04526579-8 from 1665644 to 2311326 tid 2 2024-04-26T00:36:18.293 INFO:tasks.rados.rados.0.smithi045.stdout:2753: read oid 6 snap -1 2024-04-26T00:36:18.293 INFO:tasks.rados.rados.0.smithi045.stdout:2753: expect (ObjNum 542 snap 167 seq_num 542) 2024-04-26T00:36:18.293 INFO:tasks.rados.rados.0.smithi045.stdout:2754: snap_remove snap 302 2024-04-26T00:36:18.549 INFO:tasks.rados.rados.0.smithi045.stdout:2754: done (3 left) 2024-04-26T00:36:18.549 INFO:tasks.rados.rados.0.smithi045.stdout:2755: copy_from oid 26 from oid 23 current snap is 303 2024-04-26T00:36:18.549 INFO:tasks.rados.rados.0.smithi045.stdout:2751: expect (ObjNum 952 snap 295 seq_num 952) 2024-04-26T00:36:18.702 INFO:tasks.rados.rados.0.smithi045.stdout:2752: finishing write tid 1 to smithi04526579-8 2024-04-26T00:36:18.703 INFO:tasks.rados.rados.0.smithi045.stdout:2752: finishing write tid 2 to smithi04526579-8 2024-04-26T00:36:18.703 INFO:tasks.rados.rados.0.smithi045.stdout:2752: finishing write tid 3 to smithi04526579-8 2024-04-26T00:36:18.703 INFO:tasks.rados.rados.0.smithi045.stdout:2752: finishing write tid 4 to smithi04526579-8 2024-04-26T00:36:18.703 INFO:tasks.rados.rados.0.smithi045.stdout:2752: oid 8 updating version 0 to 2183 2024-04-26T00:36:18.703 INFO:tasks.rados.rados.0.smithi045.stdout:2752: oid 8 version 2183 is already newer than 2181 2024-04-26T00:36:18.703 INFO:tasks.rados.rados.0.smithi045.stdout:2752: oid 8 version 2183 is already newer than 2182 2024-04-26T00:36:18.703 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 2183 (ObjNum 990 snap 303 seq_num 990) dirty exists 2024-04-26T00:36:18.704 INFO:tasks.rados.rados.0.smithi045.stdout:2752: left oid 8 (ObjNum 990 snap 303 seq_num 990) 2024-04-26T00:36:18.704 INFO:tasks.rados.rados.0.smithi045.stdout:2753: expect (ObjNum 542 snap 167 seq_num 542) 2024-04-26T00:36:18.847 INFO:tasks.rados.rados.0.smithi045.stdout:2755: finishing copy_from racing read to smithi04526579-26 2024-04-26T00:36:18.848 INFO:tasks.rados.rados.0.smithi045.stdout:2755: finishing copy_from to smithi04526579-26 2024-04-26T00:36:18.848 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 1996 (ObjNum 945 snap 292 seq_num 945) dirty exists 2024-04-26T00:36:18.848 INFO:tasks.rados.rados.0.smithi045.stdout:2751: done (3 left) 2024-04-26T00:36:18.848 INFO:tasks.rados.rados.0.smithi045.stdout:2752: done (2 left) 2024-04-26T00:36:18.848 INFO:tasks.rados.rados.0.smithi045.stdout:2753: done (1 left) 2024-04-26T00:36:18.848 INFO:tasks.rados.rados.0.smithi045.stdout:2755: done (0 left) 2024-04-26T00:36:18.848 INFO:tasks.rados.rados.0.smithi045.stdout:2756: write (excl) oid 13 current snap is 303 2024-04-26T00:36:18.848 INFO:tasks.rados.rados.0.smithi045.stdout:2756: seq_num 991 ranges {655737=497649,1607773=789340} 2024-04-26T00:36:18.866 INFO:tasks.rados.rados.0.smithi045.stdout:2756: writing smithi04526579-13 from 655737 to 1153386 tid 1 2024-04-26T00:36:18.887 INFO:tasks.rados.rados.0.smithi045.stdout:2756: writing smithi04526579-13 from 1607773 to 2397113 tid 2 2024-04-26T00:36:18.887 INFO:tasks.rados.rados.0.smithi045.stdout:2757: read oid 22 snap -1 2024-04-26T00:36:18.887 INFO:tasks.rados.rados.0.smithi045.stdout:2757: expect (ObjNum 942 snap 291 seq_num 942) 2024-04-26T00:36:18.887 INFO:tasks.rados.rados.0.smithi045.stdout:2758: delete oid 9 current snap is 303 2024-04-26T00:36:18.887 INFO:tasks.rados.rados.0.smithi045.stdout:2756: finishing write tid 1 to smithi04526579-13 2024-04-26T00:36:18.896 INFO:tasks.rados.rados.0.smithi045.stdout:2758: done (2 left) 2024-04-26T00:36:18.896 INFO:tasks.rados.rados.0.smithi045.stdout:2759: write (excl) oid 16 current snap is 303 2024-04-26T00:36:18.896 INFO:tasks.rados.rados.0.smithi045.stdout:2759: seq_num 993 ranges {689351=648147,2124737=564614} 2024-04-26T00:36:18.904 INFO:tasks.rados.rados.0.smithi045.stdout:2759: writing smithi04526579-16 from 689351 to 1337498 tid 1 2024-04-26T00:36:18.912 INFO:tasks.rados.rados.0.smithi045.stdout:2759: writing smithi04526579-16 from 2124737 to 2689351 tid 2 2024-04-26T00:36:18.912 INFO:tasks.rados.rados.0.smithi045.stdout:2760: copy_from oid 42 from oid 41 current snap is 303 2024-04-26T00:36:18.912 INFO:tasks.rados.rados.0.smithi045.stdout:2756: finishing write tid 2 to smithi04526579-13 2024-04-26T00:36:18.912 INFO:tasks.rados.rados.0.smithi045.stdout:2756: finishing write tid 3 to smithi04526579-13 2024-04-26T00:36:18.912 INFO:tasks.rados.rados.0.smithi045.stdout:2756: finishing write tid 4 to smithi04526579-13 2024-04-26T00:36:18.912 INFO:tasks.rados.rados.0.smithi045.stdout:2756: oid 13 updating version 0 to 2185 2024-04-26T00:36:18.912 INFO:tasks.rados.rados.0.smithi045.stdout:2756: oid 13 version 2185 is already newer than 2184 2024-04-26T00:36:18.912 INFO:tasks.rados.rados.0.smithi045.stdout:2756: oid 13 updating version 2185 to 2186 2024-04-26T00:36:18.912 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 2186 (ObjNum 991 snap 303 seq_num 991) dirty exists 2024-04-26T00:36:18.912 INFO:tasks.rados.rados.0.smithi045.stdout:2756: left oid 13 (ObjNum 991 snap 303 seq_num 991) 2024-04-26T00:36:18.912 INFO:tasks.rados.rados.0.smithi045.stdout:2757: expect (ObjNum 942 snap 291 seq_num 942) 2024-04-26T00:36:19.154 INFO:tasks.rados.rados.0.smithi045.stdout:2759: finishing write tid 1 to smithi04526579-16 2024-04-26T00:36:19.155 INFO:tasks.rados.rados.0.smithi045.stdout:2759: finishing write tid 2 to smithi04526579-16 2024-04-26T00:36:19.155 INFO:tasks.rados.rados.0.smithi045.stdout:2756: done (3 left) 2024-04-26T00:36:19.155 INFO:tasks.rados.rados.0.smithi045.stdout:2757: done (2 left) 2024-04-26T00:36:19.155 INFO:tasks.rados.rados.0.smithi045.stdout:2761: rollback oid 20 current snap is 303 2024-04-26T00:36:19.155 INFO:tasks.rados.rados.0.smithi045.stdout:2761: done (2 left) 2024-04-26T00:36:19.155 INFO:tasks.rados.rados.0.smithi045.stdout:2762: write (excl) oid 40 current snap is 303 2024-04-26T00:36:19.155 INFO:tasks.rados.rados.0.smithi045.stdout:2759: finishing write tid 3 to smithi04526579-16 2024-04-26T00:36:19.155 INFO:tasks.rados.rados.0.smithi045.stdout:2759: finishing write tid 4 to smithi04526579-16 2024-04-26T00:36:19.155 INFO:tasks.rados.rados.0.smithi045.stdout:2759: oid 16 updating version 0 to 1174 2024-04-26T00:36:19.155 INFO:tasks.rados.rados.0.smithi045.stdout:2759: oid 16 version 1174 is already newer than 1172 2024-04-26T00:36:19.156 INFO:tasks.rados.rados.0.smithi045.stdout:2759: oid 16 version 1174 is already newer than 1173 2024-04-26T00:36:19.156 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 1174 (ObjNum 993 snap 303 seq_num 993) dirty exists 2024-04-26T00:36:19.156 INFO:tasks.rados.rados.0.smithi045.stdout:2759: left oid 16 (ObjNum 993 snap 303 seq_num 993) 2024-04-26T00:36:19.156 INFO:tasks.rados.rados.0.smithi045.stdout:2760: finishing copy_from racing read to smithi04526579-42 2024-04-26T00:36:19.156 INFO:tasks.rados.rados.0.smithi045.stdout:2762: seq_num 994 ranges {706158=723396,2183219=522939} 2024-04-26T00:36:19.162 INFO:tasks.rados.rados.0.smithi045.stdout:2762: writing smithi04526579-40 from 706158 to 1429554 tid 1 2024-04-26T00:36:19.169 INFO:tasks.rados.rados.0.smithi045.stdout:2762: writing smithi04526579-40 from 2183219 to 2706158 tid 2 2024-04-26T00:36:19.169 INFO:tasks.rados.rados.0.smithi045.stdout:2759: done (2 left) 2024-04-26T00:36:19.169 INFO:tasks.rados.rados.0.smithi045.stdout:2763: write oid 13 current snap is 303 2024-04-26T00:36:19.169 INFO:tasks.rados.rados.0.smithi045.stdout:2763: seq_num 995 ranges {722965=798645,2125348=597617} 2024-04-26T00:36:19.187 INFO:tasks.rados.rados.0.smithi045.stdout:2763: writing smithi04526579-13 from 722965 to 1521610 tid 1 2024-04-26T00:36:19.199 INFO:tasks.rados.rados.0.smithi045.stdout:2763: writing smithi04526579-13 from 2125348 to 2722965 tid 2 2024-04-26T00:36:19.199 INFO:tasks.rados.rados.0.smithi045.stdout:2764: rollback oid 29 current snap is 303 2024-04-26T00:36:19.199 INFO:tasks.rados.rados.0.smithi045.stdout:2764: done (3 left) 2024-04-26T00:36:19.199 INFO:tasks.rados.rados.0.smithi045.stdout:2765: write (excl) oid 10 current snap is 303 2024-04-26T00:36:19.199 INFO:tasks.rados.rados.0.smithi045.stdout:2765: seq_num 996 ranges {739772=590247,1900183=540336} 2024-04-26T00:36:19.210 INFO:tasks.rados.rados.0.smithi045.stdout:2765: writing smithi04526579-10 from 739772 to 1330019 tid 1 2024-04-26T00:36:19.220 INFO:tasks.rados.rados.0.smithi045.stdout:2765: writing smithi04526579-10 from 1900183 to 2440519 tid 2 2024-04-26T00:36:19.220 INFO:tasks.rados.rados.0.smithi045.stdout:2766: copy_from oid 26 from oid 9 current snap is 303 2024-04-26T00:36:19.220 INFO:tasks.rados.rados.0.smithi045.stdout:2760: finishing copy_from to smithi04526579-42 2024-04-26T00:36:19.220 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 1939 (ObjNum 903 snap 277 seq_num 903) dirty exists 2024-04-26T00:36:19.220 INFO:tasks.rados.rados.0.smithi045.stdout:2762: finishing write tid 1 to smithi04526579-40 2024-04-26T00:36:19.221 INFO:tasks.rados.rados.0.smithi045.stdout:2762: finishing write tid 2 to smithi04526579-40 2024-04-26T00:36:19.221 INFO:tasks.rados.rados.0.smithi045.stdout:2762: finishing write tid 3 to smithi04526579-40 2024-04-26T00:36:19.221 INFO:tasks.rados.rados.0.smithi045.stdout:2762: finishing write tid 4 to smithi04526579-40 2024-04-26T00:36:19.221 INFO:tasks.rados.rados.0.smithi045.stdout:2762: oid 40 updating version 0 to 1999 2024-04-26T00:36:19.221 INFO:tasks.rados.rados.0.smithi045.stdout:2762: oid 40 updating version 1999 to 2000 2024-04-26T00:36:19.221 INFO:tasks.rados.rados.0.smithi045.stdout:2762: oid 40 version 2000 is already newer than 1998 2024-04-26T00:36:19.221 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 2000 (ObjNum 994 snap 303 seq_num 994) dirty exists 2024-04-26T00:36:19.221 INFO:tasks.rados.rados.0.smithi045.stdout:2762: left oid 40 (ObjNum 994 snap 303 seq_num 994) 2024-04-26T00:36:19.221 INFO:tasks.rados.rados.0.smithi045.stdout:2763: finishing write tid 1 to smithi04526579-13 2024-04-26T00:36:19.221 INFO:tasks.rados.rados.0.smithi045.stdout:2763: finishing write tid 2 to smithi04526579-13 2024-04-26T00:36:19.221 INFO:tasks.rados.rados.0.smithi045.stdout:2763: finishing write tid 3 to smithi04526579-13 2024-04-26T00:36:19.221 INFO:tasks.rados.rados.0.smithi045.stdout:2763: finishing write tid 4 to smithi04526579-13 2024-04-26T00:36:19.221 INFO:tasks.rados.rados.0.smithi045.stdout:2763: oid 13 updating version 0 to 2188 2024-04-26T00:36:19.222 INFO:tasks.rados.rados.0.smithi045.stdout:2763: oid 13 updating version 2188 to 2189 2024-04-26T00:36:19.222 INFO:tasks.rados.rados.0.smithi045.stdout:2763: oid 13 version 2189 is already newer than 2187 2024-04-26T00:36:19.222 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 2189 (ObjNum 995 snap 303 seq_num 995) dirty exists 2024-04-26T00:36:19.222 INFO:tasks.rados.rados.0.smithi045.stdout:2763: left oid 13 (ObjNum 995 snap 303 seq_num 995) 2024-04-26T00:36:19.222 INFO:tasks.rados.rados.0.smithi045.stdout:2765: finishing write tid 1 to smithi04526579-10 2024-04-26T00:36:19.222 INFO:tasks.rados.rados.0.smithi045.stdout:2760: done (4 left) 2024-04-26T00:36:19.222 INFO:tasks.rados.rados.0.smithi045.stdout:2762: done (3 left) 2024-04-26T00:36:19.222 INFO:tasks.rados.rados.0.smithi045.stdout:2763: done (2 left) 2024-04-26T00:36:19.222 INFO:tasks.rados.rados.0.smithi045.stdout:2767: read oid 38 snap -1 2024-04-26T00:36:19.222 INFO:tasks.rados.rados.0.smithi045.stdout:2767: expect deleted 2024-04-26T00:36:19.222 INFO:tasks.rados.rados.0.smithi045.stdout:2768: copy_from oid 35 from oid 4 current snap is 303 2024-04-26T00:36:19.222 INFO:tasks.rados.rados.0.smithi045.stdout:2769: write (excl) oid 3 current snap is 303 2024-04-26T00:36:19.222 INFO:tasks.rados.rados.0.smithi045.stdout:2769: seq_num 997 ranges {756579=665496,1958665=400347} 2024-04-26T00:36:19.230 INFO:tasks.rados.rados.0.smithi045.stdout:2769: writing smithi04526579-3 from 756579 to 1422075 tid 1 2024-04-26T00:36:19.238 INFO:tasks.rados.rados.0.smithi045.stdout:2769: writing smithi04526579-3 from 1958665 to 2359012 tid 2 2024-04-26T00:36:19.238 INFO:tasks.rados.rados.0.smithi045.stdout:2770: write (excl) oid 7 current snap is 303 2024-04-26T00:36:19.238 INFO:tasks.rados.rados.0.smithi045.stdout:2770: seq_num 998 ranges {773386=740745,2017147=544005} 2024-04-26T00:36:19.249 INFO:tasks.rados.rados.0.smithi045.stdout:2770: writing smithi04526579-7 from 773386 to 1514131 tid 1 2024-04-26T00:36:19.256 INFO:tasks.rados.rados.0.smithi045.stdout:2770: writing smithi04526579-7 from 2017147 to 2561152 tid 2 2024-04-26T00:36:19.256 INFO:tasks.rados.rados.0.smithi045.stdout:2771: copy_from oid 29 from oid 34 current snap is 303 2024-04-26T00:36:19.256 INFO:tasks.rados.rados.0.smithi045.stdout:2772: read oid 49 snap -1 2024-04-26T00:36:19.256 INFO:tasks.rados.rados.0.smithi045.stdout:2772: expect (ObjNum 964 snap 298 seq_num 964) 2024-04-26T00:36:19.256 INFO:tasks.rados.rados.0.smithi045.stdout:2773: write (excl) oid 45 current snap is 303 2024-04-26T00:36:19.256 INFO:tasks.rados.rados.0.smithi045.stdout:2773: seq_num 999 ranges {790193=415994,1959276=404016} 2024-04-26T00:36:19.263 INFO:tasks.rados.rados.0.smithi045.stdout:2773: writing smithi04526579-45 from 790193 to 1206187 tid 1 2024-04-26T00:36:19.271 INFO:tasks.rados.rados.0.smithi045.stdout:2773: writing smithi04526579-45 from 1959276 to 2363292 tid 2 2024-04-26T00:36:19.271 INFO:tasks.rados.rados.0.smithi045.stdout:2774: read oid 20 snap -1 2024-04-26T00:36:19.271 INFO:tasks.rados.rados.0.smithi045.stdout:2774: expect (ObjNum 933 snap 289 seq_num 933) 2024-04-26T00:36:19.271 INFO:tasks.rados.rados.0.smithi045.stdout:2775: snap_create 2024-04-26T00:36:19.272 INFO:tasks.rados.rados.0.smithi045.stdout:2766: finishing copy_from to smithi04526579-26 2024-04-26T00:36:19.272 INFO:tasks.rados.rados.0.smithi045.stdout:2766: got expected ENOENT (src dne) 2024-04-26T00:36:19.272 INFO:tasks.rados.rados.0.smithi045.stdout:2766: finishing copy_from racing read to smithi04526579-26 2024-04-26T00:36:19.272 INFO:tasks.rados.rados.0.smithi045.stdout:2765: finishing write tid 2 to smithi04526579-10 2024-04-26T00:36:19.272 INFO:tasks.rados.rados.0.smithi045.stdout:2765: finishing write tid 3 to smithi04526579-10 2024-04-26T00:36:19.272 INFO:tasks.rados.rados.0.smithi045.stdout:2765: finishing write tid 4 to smithi04526579-10 2024-04-26T00:36:19.272 INFO:tasks.rados.rados.0.smithi045.stdout:2765: oid 10 updating version 0 to 1966 2024-04-26T00:36:19.272 INFO:tasks.rados.rados.0.smithi045.stdout:2765: oid 10 updating version 1966 to 1967 2024-04-26T00:36:19.272 INFO:tasks.rados.rados.0.smithi045.stdout:2765: oid 10 updating version 1967 to 1968 2024-04-26T00:36:19.272 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 1968 (ObjNum 996 snap 303 seq_num 996) dirty exists 2024-04-26T00:36:19.272 INFO:tasks.rados.rados.0.smithi045.stdout:2765: left oid 10 (ObjNum 996 snap 303 seq_num 996) 2024-04-26T00:36:19.273 INFO:tasks.rados.rados.0.smithi045.stdout:2769: finishing write tid 1 to smithi04526579-3 2024-04-26T00:36:19.273 INFO:tasks.rados.rados.0.smithi045.stdout:2769: finishing write tid 2 to smithi04526579-3 2024-04-26T00:36:19.273 INFO:tasks.rados.rados.0.smithi045.stdout:2769: finishing write tid 3 to smithi04526579-3 2024-04-26T00:36:19.273 INFO:tasks.rados.rados.0.smithi045.stdout:2769: finishing write tid 4 to smithi04526579-3 2024-04-26T00:36:19.273 INFO:tasks.rados.rados.0.smithi045.stdout:2769: oid 3 updating version 0 to 2014 2024-04-26T00:36:19.273 INFO:tasks.rados.rados.0.smithi045.stdout:2769: oid 3 version 2014 is already newer than 2013 2024-04-26T00:36:19.273 INFO:tasks.rados.rados.0.smithi045.stdout:2769: oid 3 version 2014 is already newer than 2012 2024-04-26T00:36:19.273 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 2014 (ObjNum 997 snap 303 seq_num 997) dirty exists 2024-04-26T00:36:19.273 INFO:tasks.rados.rados.0.smithi045.stdout:2769: left oid 3 (ObjNum 997 snap 303 seq_num 997) 2024-04-26T00:36:19.273 INFO:tasks.rados.rados.0.smithi045.stdout:2768: finishing copy_from racing read to smithi04526579-35 2024-04-26T00:36:19.273 INFO:tasks.rados.rados.0.smithi045.stdout:2768: finishing copy_from to smithi04526579-35 2024-04-26T00:36:19.273 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 1971 (ObjNum 941 snap 291 seq_num 941) dirty exists 2024-04-26T00:36:19.273 INFO:tasks.rados.rados.0.smithi045.stdout:2772: expect (ObjNum 964 snap 298 seq_num 964) 2024-04-26T00:36:19.418 INFO:tasks.rados.rados.0.smithi045.stdout:2770: finishing write tid 1 to smithi04526579-7 2024-04-26T00:36:19.418 INFO:tasks.rados.rados.0.smithi045.stdout:2770: finishing write tid 2 to smithi04526579-7 2024-04-26T00:36:19.418 INFO:tasks.rados.rados.0.smithi045.stdout:2770: finishing write tid 3 to smithi04526579-7 2024-04-26T00:36:19.418 INFO:tasks.rados.rados.0.smithi045.stdout:2770: finishing write tid 4 to smithi04526579-7 2024-04-26T00:36:19.418 INFO:tasks.rados.rados.0.smithi045.stdout:2770: oid 7 updating version 0 to 2366 2024-04-26T00:36:19.418 INFO:tasks.rados.rados.0.smithi045.stdout:2770: oid 7 updating version 2366 to 2367 2024-04-26T00:36:19.418 INFO:tasks.rados.rados.0.smithi045.stdout:2770: oid 7 version 2367 is already newer than 2365 2024-04-26T00:36:19.418 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 2367 (ObjNum 998 snap 303 seq_num 998) dirty exists 2024-04-26T00:36:19.419 INFO:tasks.rados.rados.0.smithi045.stdout:2770: left oid 7 (ObjNum 998 snap 303 seq_num 998) 2024-04-26T00:36:19.419 INFO:tasks.rados.rados.0.smithi045.stdout:2773: finishing write tid 1 to smithi04526579-45 2024-04-26T00:36:19.419 INFO:tasks.rados.rados.0.smithi045.stdout:2773: finishing write tid 2 to smithi04526579-45 2024-04-26T00:36:19.419 INFO:tasks.rados.rados.0.smithi045.stdout:2773: finishing write tid 3 to smithi04526579-45 2024-04-26T00:36:19.419 INFO:tasks.rados.rados.0.smithi045.stdout:2773: finishing write tid 4 to smithi04526579-45 2024-04-26T00:36:19.419 INFO:tasks.rados.rados.0.smithi045.stdout:2773: oid 45 updating version 0 to 2166 2024-04-26T00:36:19.419 INFO:tasks.rados.rados.0.smithi045.stdout:2773: oid 45 updating version 2166 to 2167 2024-04-26T00:36:19.419 INFO:tasks.rados.rados.0.smithi045.stdout:2773: oid 45 updating version 2167 to 2168 2024-04-26T00:36:19.419 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 2168 (ObjNum 999 snap 303 seq_num 999) dirty exists 2024-04-26T00:36:19.419 INFO:tasks.rados.rados.0.smithi045.stdout:2773: left oid 45 (ObjNum 999 snap 303 seq_num 999) 2024-04-26T00:36:19.419 INFO:tasks.rados.rados.0.smithi045.stdout:2771: finishing copy_from racing read to smithi04526579-29 2024-04-26T00:36:19.419 INFO:tasks.rados.rados.0.smithi045.stdout:2771: finishing copy_from to smithi04526579-29 2024-04-26T00:36:19.419 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 2011 (ObjNum 987 snap 302 seq_num 987) dirty exists 2024-04-26T00:36:19.419 INFO:tasks.rados.rados.0.smithi045.stdout:2774: expect (ObjNum 933 snap 289 seq_num 933) 2024-04-26T00:36:20.025 INFO:tasks.rados.rados.0.smithi045.stdout:2765: done (10 left) 2024-04-26T00:36:20.026 INFO:tasks.rados.rados.0.smithi045.stdout:2766: done (9 left) 2024-04-26T00:36:20.026 INFO:tasks.rados.rados.0.smithi045.stdout:2767: done (8 left) 2024-04-26T00:36:20.026 INFO:tasks.rados.rados.0.smithi045.stdout:2768: done (7 left) 2024-04-26T00:36:20.026 INFO:tasks.rados.rados.0.smithi045.stdout:2769: done (6 left) 2024-04-26T00:36:20.026 INFO:tasks.rados.rados.0.smithi045.stdout:2770: done (5 left) 2024-04-26T00:36:20.026 INFO:tasks.rados.rados.0.smithi045.stdout:2771: done (4 left) 2024-04-26T00:36:20.026 INFO:tasks.rados.rados.0.smithi045.stdout:2772: done (3 left) 2024-04-26T00:36:20.026 INFO:tasks.rados.rados.0.smithi045.stdout:2773: done (2 left) 2024-04-26T00:36:20.026 INFO:tasks.rados.rados.0.smithi045.stdout:2774: done (1 left) 2024-04-26T00:36:20.026 INFO:tasks.rados.rados.0.smithi045.stdout:2775: done (0 left) 2024-04-26T00:36:20.026 INFO:tasks.rados.rados.0.smithi045.stdout:2776: copy_from oid 11 from oid 23 current snap is 304 2024-04-26T00:36:20.026 INFO:tasks.rados.rados.0.smithi045.stdout:2777: copy_from oid 32 from oid 45 current snap is 304 2024-04-26T00:36:20.027 INFO:tasks.rados.rados.0.smithi045.stdout:2778: delete oid 4 current snap is 304 2024-04-26T00:36:20.036 INFO:tasks.rados.rados.0.smithi045.stdout:2778: done (2 left) 2024-04-26T00:36:20.036 INFO:tasks.rados.rados.0.smithi045.stdout:2779: write oid 38 current snap is 304 2024-04-26T00:36:20.036 INFO:tasks.rados.rados.0.smithi045.stdout:2779: seq_num 1001 ranges {423807=566492,1676240=407685,2739004=84803} 2024-04-26T00:36:20.046 INFO:tasks.rados.rados.0.smithi045.stdout:2779: writing smithi04526579-38 from 423807 to 990299 tid 1 2024-04-26T00:36:20.053 INFO:tasks.rados.rados.0.smithi045.stdout:2779: writing smithi04526579-38 from 1676240 to 2083925 tid 2 2024-04-26T00:36:20.060 INFO:tasks.rados.rados.0.smithi045.stdout:2779: writing smithi04526579-38 from 2739004 to 2823807 tid 3 2024-04-26T00:36:20.060 INFO:tasks.rados.rados.0.smithi045.stdout:2780: write (excl) oid 24 current snap is 304 2024-04-26T00:36:20.060 INFO:tasks.rados.rados.0.smithi045.stdout:2780: seq_num 1002 ranges {440614=641741,1734722=551343} 2024-04-26T00:36:20.068 INFO:tasks.rados.rados.0.smithi045.stdout:2780: writing smithi04526579-24 from 440614 to 1082355 tid 1 2024-04-26T00:36:20.079 INFO:tasks.rados.rados.0.smithi045.stdout:2780: writing smithi04526579-24 from 1734722 to 2286065 tid 2 2024-04-26T00:36:20.079 INFO:tasks.rados.rados.0.smithi045.stdout:2781: snap_create 2024-04-26T00:36:20.079 INFO:tasks.rados.rados.0.smithi045.stdout:2776: finishing copy_from racing read to smithi04526579-11 2024-04-26T00:36:20.079 INFO:tasks.rados.rados.0.smithi045.stdout:2779: finishing write tid 1 to smithi04526579-38 2024-04-26T00:36:20.079 INFO:tasks.rados.rados.0.smithi045.stdout:2776: finishing copy_from to smithi04526579-11 2024-04-26T00:36:20.079 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 1996 (ObjNum 945 snap 292 seq_num 945) dirty exists 2024-04-26T00:36:20.079 INFO:tasks.rados.rados.0.smithi045.stdout:2777: finishing copy_from racing read to smithi04526579-32 2024-04-26T00:36:20.079 INFO:tasks.rados.rados.0.smithi045.stdout:2779: finishing write tid 2 to smithi04526579-38 2024-04-26T00:36:20.079 INFO:tasks.rados.rados.0.smithi045.stdout:2777: finishing copy_from to smithi04526579-32 2024-04-26T00:36:20.080 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 2168 (ObjNum 999 snap 303 seq_num 999) dirty exists 2024-04-26T00:36:20.080 INFO:tasks.rados.rados.0.smithi045.stdout:2779: finishing write tid 3 to smithi04526579-38 2024-04-26T00:36:20.080 INFO:tasks.rados.rados.0.smithi045.stdout:2779: finishing write tid 4 to smithi04526579-38 2024-04-26T00:36:20.080 INFO:tasks.rados.rados.0.smithi045.stdout:2779: finishing write tid 5 to smithi04526579-38 2024-04-26T00:36:20.080 INFO:tasks.rados.rados.0.smithi045.stdout:2779: oid 38 updating version 0 to 2188 2024-04-26T00:36:20.080 INFO:tasks.rados.rados.0.smithi045.stdout:2779: oid 38 updating version 2188 to 2191 2024-04-26T00:36:20.080 INFO:tasks.rados.rados.0.smithi045.stdout:2779: oid 38 version 2191 is already newer than 2190 2024-04-26T00:36:20.080 INFO:tasks.rados.rados.0.smithi045.stdout:2779: oid 38 version 2191 is already newer than 2189 2024-04-26T00:36:20.080 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 2191 (ObjNum 1001 snap 304 seq_num 1001) dirty exists 2024-04-26T00:36:20.080 INFO:tasks.rados.rados.0.smithi045.stdout:2779: left oid 38 (ObjNum 1001 snap 304 seq_num 1001) 2024-04-26T00:36:20.080 INFO:tasks.rados.rados.0.smithi045.stdout:2780: finishing write tid 1 to smithi04526579-24 2024-04-26T00:36:20.086 INFO:tasks.rados.rados.0.smithi045.stdout:2780: finishing write tid 2 to smithi04526579-24 2024-04-26T00:36:20.087 INFO:tasks.rados.rados.0.smithi045.stdout:2780: finishing write tid 3 to smithi04526579-24 2024-04-26T00:36:20.087 INFO:tasks.rados.rados.0.smithi045.stdout:2780: finishing write tid 4 to smithi04526579-24 2024-04-26T00:36:20.087 INFO:tasks.rados.rados.0.smithi045.stdout:2780: oid 24 updating version 0 to 2369 2024-04-26T00:36:20.087 INFO:tasks.rados.rados.0.smithi045.stdout:2780: oid 24 version 2369 is already newer than 2368 2024-04-26T00:36:20.087 INFO:tasks.rados.rados.0.smithi045.stdout:2780: oid 24 updating version 2369 to 2370 2024-04-26T00:36:20.088 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 2370 (ObjNum 1002 snap 304 seq_num 1002) dirty exists 2024-04-26T00:36:20.088 INFO:tasks.rados.rados.0.smithi045.stdout:2780: left oid 24 (ObjNum 1002 snap 304 seq_num 1002) 2024-04-26T00:36:21.035 INFO:tasks.rados.rados.0.smithi045.stdout:2776: done (4 left) 2024-04-26T00:36:21.035 INFO:tasks.rados.rados.0.smithi045.stdout:2777: done (3 left) 2024-04-26T00:36:21.035 INFO:tasks.rados.rados.0.smithi045.stdout:2779: done (2 left) 2024-04-26T00:36:21.035 INFO:tasks.rados.rados.0.smithi045.stdout:2780: done (1 left) 2024-04-26T00:36:21.035 INFO:tasks.rados.rados.0.smithi045.stdout:2781: done (0 left) 2024-04-26T00:36:21.035 INFO:tasks.rados.rados.0.smithi045.stdout:2782: delete oid 49 current snap is 305 2024-04-26T00:36:21.045 INFO:tasks.rados.rados.0.smithi045.stdout:2782: done (0 left) 2024-04-26T00:36:21.046 INFO:tasks.rados.rados.0.smithi045.stdout:2783: copy_from oid 12 from oid 11 current snap is 305 2024-04-26T00:36:21.046 INFO:tasks.rados.rados.0.smithi045.stdout:2784: delete oid 30 current snap is 305 2024-04-26T00:36:21.054 INFO:tasks.rados.rados.0.smithi045.stdout:2784: done (1 left) 2024-04-26T00:36:21.054 INFO:tasks.rados.rados.0.smithi045.stdout:2785: delete oid 27 current snap is 305 2024-04-26T00:36:21.069 INFO:tasks.rados.rados.0.smithi045.stdout:2785: done (1 left) 2024-04-26T00:36:21.070 INFO:tasks.rados.rados.0.smithi045.stdout:2786: snap_remove snap 303 2024-04-26T00:36:22.042 INFO:tasks.rados.rados.0.smithi045.stdout:2783: finishing copy_from racing read to smithi04526579-12 2024-04-26T00:36:22.042 INFO:tasks.rados.rados.0.smithi045.stdout:2783: finishing copy_from to smithi04526579-12 2024-04-26T00:36:22.042 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 1996 (ObjNum 945 snap 292 seq_num 945) dirty exists 2024-04-26T00:36:22.043 INFO:tasks.rados.rados.0.smithi045.stdout:2783: done (1 left) 2024-04-26T00:36:22.043 INFO:tasks.rados.rados.0.smithi045.stdout:2786: done (0 left) 2024-04-26T00:36:22.043 INFO:tasks.rados.rados.0.smithi045.stdout:2787: read oid 1 snap 304 2024-04-26T00:36:22.043 INFO:tasks.rados.rados.0.smithi045.stdout:2787: expect (ObjNum 956 snap 296 seq_num 956) 2024-04-26T00:36:22.043 INFO:tasks.rados.rados.0.smithi045.stdout:2788: read oid 37 snap -1 2024-04-26T00:36:22.043 INFO:tasks.rados.rados.0.smithi045.stdout:2788: expect (ObjNum 939 snap 290 seq_num 939) 2024-04-26T00:36:22.043 INFO:tasks.rados.rados.0.smithi045.stdout:2789: snap_create 2024-04-26T00:36:22.055 INFO:tasks.rados.rados.0.smithi045.stdout:2787: expect (ObjNum 956 snap 296 seq_num 956) 2024-04-26T00:36:22.222 INFO:tasks.rados.rados.0.smithi045.stdout:2788: expect (ObjNum 939 snap 290 seq_num 939) 2024-04-26T00:36:23.047 INFO:tasks.rados.rados.0.smithi045.stdout:2787: done (2 left) 2024-04-26T00:36:23.047 INFO:tasks.rados.rados.0.smithi045.stdout:2788: done (1 left) 2024-04-26T00:36:23.047 INFO:tasks.rados.rados.0.smithi045.stdout:2789: done (0 left) 2024-04-26T00:36:23.047 INFO:tasks.rados.rados.0.smithi045.stdout:2790: read oid 27 snap -1 2024-04-26T00:36:23.047 INFO:tasks.rados.rados.0.smithi045.stdout:2790: expect deleted 2024-04-26T00:36:23.047 INFO:tasks.rados.rados.0.smithi045.stdout:2791: read oid 23 snap 305 2024-04-26T00:36:23.047 INFO:tasks.rados.rados.0.smithi045.stdout:2791: expect (ObjNum 945 snap 292 seq_num 945) 2024-04-26T00:36:23.047 INFO:tasks.rados.rados.0.smithi045.stdout:2792: snap_create 2024-04-26T00:36:23.058 INFO:tasks.rados.rados.0.smithi045.stdout:2791: expect (ObjNum 945 snap 292 seq_num 945) 2024-04-26T00:36:23.551 INFO:tasks.rados.rados.0.smithi045.stdout:2790: done (2 left) 2024-04-26T00:36:23.551 INFO:tasks.rados.rados.0.smithi045.stdout:2791: done (1 left) 2024-04-26T00:36:23.551 INFO:tasks.rados.rados.0.smithi045.stdout:2792: done (0 left) 2024-04-26T00:36:23.551 INFO:tasks.rados.rados.0.smithi045.stdout:2793: write (excl) oid 49 current snap is 307 2024-04-26T00:36:23.551 INFO:tasks.rados.rados.0.smithi045.stdout:2793: seq_num 1006 ranges {507842=659090,1568650=558681,2759766=148076} 2024-04-26T00:36:23.569 INFO:tasks.rados.rados.0.smithi045.stdout:2793: writing smithi04526579-49 from 507842 to 1166932 tid 1 2024-04-26T00:36:23.587 INFO:tasks.rados.rados.0.smithi045.stdout:2793: writing smithi04526579-49 from 1568650 to 2127331 tid 2 2024-04-26T00:36:23.601 INFO:tasks.rados.rados.0.smithi045.stdout:2793: writing smithi04526579-49 from 2759766 to 2907842 tid 3 2024-04-26T00:36:23.601 INFO:tasks.rados.rados.0.smithi045.stdout:2794: read oid 44 snap 306 2024-04-26T00:36:23.601 INFO:tasks.rados.rados.0.smithi045.stdout:2794: expect (ObjNum 952 snap 295 seq_num 952) 2024-04-26T00:36:23.601 INFO:tasks.rados.rados.0.smithi045.stdout:2795: snap_remove snap 305 2024-04-26T00:36:24.552 INFO:tasks.rados.rados.0.smithi045.stdout:2795: done (2 left) 2024-04-26T00:36:24.552 INFO:tasks.rados.rados.0.smithi045.stdout:2796: delete oid 30 current snap is 307 2024-04-26T00:36:24.552 INFO:tasks.rados.rados.0.smithi045.stdout:2793: finishing write tid 1 to smithi04526579-49 2024-04-26T00:36:24.552 INFO:tasks.rados.rados.0.smithi045.stdout:2793: finishing write tid 2 to smithi04526579-49 2024-04-26T00:36:24.552 INFO:tasks.rados.rados.0.smithi045.stdout:2794: expect (ObjNum 952 snap 295 seq_num 952) 2024-04-26T00:36:24.696 INFO:tasks.rados.rados.0.smithi045.stdout:2793: finishing write tid 3 to smithi04526579-49 2024-04-26T00:36:24.696 INFO:tasks.rados.rados.0.smithi045.stdout:2793: finishing write tid 4 to smithi04526579-49 2024-04-26T00:36:24.696 INFO:tasks.rados.rados.0.smithi045.stdout:2793: finishing write tid 5 to smithi04526579-49 2024-04-26T00:36:24.696 INFO:tasks.rados.rados.0.smithi045.stdout:2793: oid 49 updating version 0 to 2018 2024-04-26T00:36:24.696 INFO:tasks.rados.rados.0.smithi045.stdout:2793: oid 49 version 2018 is already newer than 2017 2024-04-26T00:36:24.696 INFO:tasks.rados.rados.0.smithi045.stdout:2793: oid 49 version 2018 is already newer than 2016 2024-04-26T00:36:24.696 INFO:tasks.rados.rados.0.smithi045.stdout:2793: oid 49 updating version 2018 to 2019 2024-04-26T00:36:24.696 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 2019 (ObjNum 1006 snap 307 seq_num 1006) dirty exists 2024-04-26T00:36:24.696 INFO:tasks.rados.rados.0.smithi045.stdout:2793: left oid 49 (ObjNum 1006 snap 307 seq_num 1006) 2024-04-26T00:36:24.696 INFO:tasks.rados.rados.0.smithi045.stdout:2793: done (2 left) 2024-04-26T00:36:24.696 INFO:tasks.rados.rados.0.smithi045.stdout:2794: done (1 left) 2024-04-26T00:36:24.696 INFO:tasks.rados.rados.0.smithi045.stdout:2796: done (0 left) 2024-04-26T00:36:24.696 INFO:tasks.rados.rados.0.smithi045.stdout:2797: snap_create 2024-04-26T00:36:25.553 INFO:tasks.rados.rados.0.smithi045.stdout:2797: done (0 left) 2024-04-26T00:36:25.553 INFO:tasks.rados.rados.0.smithi045.stdout:2798: delete oid 41 current snap is 308 2024-04-26T00:36:25.561 INFO:tasks.rados.rados.0.smithi045.stdout:2798: done (0 left) 2024-04-26T00:36:25.562 INFO:tasks.rados.rados.0.smithi045.stdout:2799: write (excl) oid 14 current snap is 308 2024-04-26T00:36:25.562 INFO:tasks.rados.rados.0.smithi045.stdout:2799: seq_num 1009 ranges {558263=484837,1627743=706008} 2024-04-26T00:36:25.578 INFO:tasks.rados.rados.0.smithi045.stdout:2799: writing smithi04526579-14 from 558263 to 1043100 tid 1 2024-04-26T00:36:25.600 INFO:tasks.rados.rados.0.smithi045.stdout:2799: writing smithi04526579-14 from 1627743 to 2333751 tid 2 2024-04-26T00:36:25.600 INFO:tasks.rados.rados.0.smithi045.stdout:2800: snap_create 2024-04-26T00:36:25.600 INFO:tasks.rados.rados.0.smithi045.stdout:2799: finishing write tid 1 to smithi04526579-14 2024-04-26T00:36:25.608 INFO:tasks.rados.rados.0.smithi045.stdout:2799: finishing write tid 2 to smithi04526579-14 2024-04-26T00:36:25.608 INFO:tasks.rados.rados.0.smithi045.stdout:2799: finishing write tid 3 to smithi04526579-14 2024-04-26T00:36:25.609 INFO:tasks.rados.rados.0.smithi045.stdout:2799: finishing write tid 4 to smithi04526579-14 2024-04-26T00:36:25.609 INFO:tasks.rados.rados.0.smithi045.stdout:2799: oid 14 updating version 0 to 1177 2024-04-26T00:36:25.609 INFO:tasks.rados.rados.0.smithi045.stdout:2799: oid 14 version 1177 is already newer than 1175 2024-04-26T00:36:25.609 INFO:tasks.rados.rados.0.smithi045.stdout:2799: oid 14 version 1177 is already newer than 1176 2024-04-26T00:36:25.609 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 1177 (ObjNum 1009 snap 308 seq_num 1009) dirty exists 2024-04-26T00:36:25.609 INFO:tasks.rados.rados.0.smithi045.stdout:2799: left oid 14 (ObjNum 1009 snap 308 seq_num 1009) 2024-04-26T00:36:26.560 INFO:tasks.rados.rados.0.smithi045.stdout:2799: done (1 left) 2024-04-26T00:36:26.561 INFO:tasks.rados.rados.0.smithi045.stdout:2800: done (0 left) 2024-04-26T00:36:26.561 INFO:tasks.rados.rados.0.smithi045.stdout:2801: delete oid 33 current snap is 309 2024-04-26T00:36:26.573 INFO:tasks.rados.rados.0.smithi045.stdout:2801: done (0 left) 2024-04-26T00:36:26.573 INFO:tasks.rados.rados.0.smithi045.stdout:2802: write (excl) oid 3 current snap is 309 2024-04-26T00:36:26.573 INFO:tasks.rados.rados.0.smithi045.stdout:2802: seq_num 1011 ranges {591877=635335,1628354=709677} 2024-04-26T00:36:26.593 INFO:tasks.rados.rados.0.smithi045.stdout:2802: writing smithi04526579-3 from 591877 to 1227212 tid 1 2024-04-26T00:36:26.613 INFO:tasks.rados.rados.0.smithi045.stdout:2802: writing smithi04526579-3 from 1628354 to 2338031 tid 2 2024-04-26T00:36:26.614 INFO:tasks.rados.rados.0.smithi045.stdout:2803: write (excl) oid 16 current snap is 309 2024-04-26T00:36:26.614 INFO:tasks.rados.rados.0.smithi045.stdout:2803: seq_num 1012 ranges {608684=426937,1803189=569688,2924304=84380} 2024-04-26T00:36:26.630 INFO:tasks.rados.rados.0.smithi045.stdout:2803: writing smithi04526579-16 from 608684 to 1035621 tid 1 2024-04-26T00:36:26.650 INFO:tasks.rados.rados.0.smithi045.stdout:2803: writing smithi04526579-16 from 1803189 to 2372877 tid 2 2024-04-26T00:36:26.659 INFO:tasks.rados.rados.0.smithi045.stdout:2803: writing smithi04526579-16 from 2924304 to 3008684 tid 3 2024-04-26T00:36:26.659 INFO:tasks.rados.rados.0.smithi045.stdout:2804: read oid 9 snap -1 2024-04-26T00:36:26.659 INFO:tasks.rados.rados.0.smithi045.stdout:2804: expect deleted 2024-04-26T00:36:26.659 INFO:tasks.rados.rados.0.smithi045.stdout:2805: rollback oid 44 current snap is 309 2024-04-26T00:36:26.659 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 44 to 304 2024-04-26T00:36:26.659 INFO:tasks.rados.rados.0.smithi045.stdout:2802: finishing write tid 1 to smithi04526579-3 2024-04-26T00:36:26.659 INFO:tasks.rados.rados.0.smithi045.stdout:2802: finishing write tid 2 to smithi04526579-3 2024-04-26T00:36:26.659 INFO:tasks.rados.rados.0.smithi045.stdout:2802: finishing write tid 3 to smithi04526579-3 2024-04-26T00:36:26.659 INFO:tasks.rados.rados.0.smithi045.stdout:2802: finishing write tid 4 to smithi04526579-3 2024-04-26T00:36:26.659 INFO:tasks.rados.rados.0.smithi045.stdout:2802: oid 3 updating version 0 to 2020 2024-04-26T00:36:26.660 INFO:tasks.rados.rados.0.smithi045.stdout:2802: oid 3 updating version 2020 to 2022 2024-04-26T00:36:26.660 INFO:tasks.rados.rados.0.smithi045.stdout:2802: oid 3 version 2022 is already newer than 2021 2024-04-26T00:36:26.660 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 2022 (ObjNum 1011 snap 309 seq_num 1011) dirty exists 2024-04-26T00:36:26.660 INFO:tasks.rados.rados.0.smithi045.stdout:2802: left oid 3 (ObjNum 1011 snap 309 seq_num 1011) 2024-04-26T00:36:26.660 INFO:tasks.rados.rados.0.smithi045.stdout:2803: finishing write tid 1 to smithi04526579-16 2024-04-26T00:36:26.660 INFO:tasks.rados.rados.0.smithi045.stdout:2803: finishing write tid 2 to smithi04526579-16 2024-04-26T00:36:26.660 INFO:tasks.rados.rados.0.smithi045.stdout:2802: done (3 left) 2024-04-26T00:36:26.660 INFO:tasks.rados.rados.0.smithi045.stdout:2806: snap_remove snap 307 2024-04-26T00:36:27.569 INFO:tasks.rados.rados.0.smithi045.stdout:2806: done (3 left) 2024-04-26T00:36:27.569 INFO:tasks.rados.rados.0.smithi045.stdout:2807: rollback oid 45 current snap is 309 2024-04-26T00:36:27.569 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 45 to 308 2024-04-26T00:36:27.569 INFO:tasks.rados.rados.0.smithi045.stdout:2803: finishing write tid 3 to smithi04526579-16 2024-04-26T00:36:27.569 INFO:tasks.rados.rados.0.smithi045.stdout:2803: finishing write tid 4 to smithi04526579-16 2024-04-26T00:36:27.570 INFO:tasks.rados.rados.0.smithi045.stdout:2803: finishing write tid 5 to smithi04526579-16 2024-04-26T00:36:27.570 INFO:tasks.rados.rados.0.smithi045.stdout:2803: oid 16 updating version 0 to 1179 2024-04-26T00:36:27.570 INFO:tasks.rados.rados.0.smithi045.stdout:2803: oid 16 updating version 1179 to 1180 2024-04-26T00:36:27.570 INFO:tasks.rados.rados.0.smithi045.stdout:2803: oid 16 updating version 1180 to 1181 2024-04-26T00:36:27.570 INFO:tasks.rados.rados.0.smithi045.stdout:2803: oid 16 version 1181 is already newer than 1178 2024-04-26T00:36:27.570 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 1181 (ObjNum 1012 snap 309 seq_num 1012) dirty exists 2024-04-26T00:36:27.570 INFO:tasks.rados.rados.0.smithi045.stdout:2803: left oid 16 (ObjNum 1012 snap 309 seq_num 1012) 2024-04-26T00:36:27.570 INFO:tasks.rados.rados.0.smithi045.stdout:2805: finishing rollback tid 0 to smithi04526579-44 2024-04-26T00:36:27.570 INFO:tasks.rados.rados.0.smithi045.stdout:2805: finishing rollback tid 1 to smithi04526579-44 2024-04-26T00:36:27.570 INFO:tasks.rados.rados.0.smithi045.stdout:2805: finishing rollback tid 2 to smithi04526579-44 2024-04-26T00:36:27.570 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 2194 (ObjNum 952 snap 295 seq_num 952) dirty exists 2024-04-26T00:36:27.570 INFO:tasks.rados.rados.0.smithi045.stdout:2803: done (3 left) 2024-04-26T00:36:27.570 INFO:tasks.rados.rados.0.smithi045.stdout:2804: done (2 left) 2024-04-26T00:36:27.570 INFO:tasks.rados.rados.0.smithi045.stdout:2805: done (1 left) 2024-04-26T00:36:27.571 INFO:tasks.rados.rados.0.smithi045.stdout:2808: rollback oid 2 current snap is 309 2024-04-26T00:36:27.571 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 2 to 308 2024-04-26T00:36:27.571 INFO:tasks.rados.rados.0.smithi045.stdout:2809: write oid 49 current snap is 309 2024-04-26T00:36:27.571 INFO:tasks.rados.rados.0.smithi045.stdout:2809: seq_num 1013 ranges {625491=502186,1861671=713346} 2024-04-26T00:36:27.583 INFO:tasks.rados.rados.0.smithi045.stdout:2809: writing smithi04526579-49 from 625491 to 1127677 tid 1 2024-04-26T00:36:27.603 INFO:tasks.rados.rados.0.smithi045.stdout:2809: writing smithi04526579-49 from 1861671 to 2575017 tid 2 2024-04-26T00:36:27.604 INFO:tasks.rados.rados.0.smithi045.stdout:2810: snap_remove snap 304 2024-04-26T00:36:28.550 INFO:tasks.rados.rados.0.smithi045.stdout:2810: done (3 left) 2024-04-26T00:36:28.550 INFO:tasks.rados.rados.0.smithi045.stdout:2811: copy_from oid 35 from oid 25 current snap is 309 2024-04-26T00:36:28.550 INFO:tasks.rados.rados.0.smithi045.stdout:2807: finishing rollback tid 0 to smithi04526579-45 2024-04-26T00:36:28.550 INFO:tasks.rados.rados.0.smithi045.stdout:2807: finishing rollback tid 1 to smithi04526579-45 2024-04-26T00:36:28.551 INFO:tasks.rados.rados.0.smithi045.stdout:2807: finishing rollback tid 2 to smithi04526579-45 2024-04-26T00:36:28.551 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 2171 (ObjNum 999 snap 303 seq_num 999) dirty exists 2024-04-26T00:36:28.551 INFO:tasks.rados.rados.0.smithi045.stdout:2808: finishing rollback tid 0 to smithi04526579-2 2024-04-26T00:36:28.551 INFO:tasks.rados.rados.0.smithi045.stdout:2808: finishing rollback tid 1 to smithi04526579-2 2024-04-26T00:36:28.551 INFO:tasks.rados.rados.0.smithi045.stdout:2808: finishing rollback tid 2 to smithi04526579-2 2024-04-26T00:36:28.551 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 2011 (ObjNum 978 snap 301 seq_num 978) dirty exists 2024-04-26T00:36:28.551 INFO:tasks.rados.rados.0.smithi045.stdout:2809: finishing write tid 1 to smithi04526579-49 2024-04-26T00:36:28.551 INFO:tasks.rados.rados.0.smithi045.stdout:2809: finishing write tid 2 to smithi04526579-49 2024-04-26T00:36:28.551 INFO:tasks.rados.rados.0.smithi045.stdout:2809: finishing write tid 3 to smithi04526579-49 2024-04-26T00:36:28.551 INFO:tasks.rados.rados.0.smithi045.stdout:2809: finishing write tid 4 to smithi04526579-49 2024-04-26T00:36:28.551 INFO:tasks.rados.rados.0.smithi045.stdout:2809: oid 49 updating version 0 to 2025 2024-04-26T00:36:28.551 INFO:tasks.rados.rados.0.smithi045.stdout:2809: oid 49 version 2025 is already newer than 2024 2024-04-26T00:36:28.551 INFO:tasks.rados.rados.0.smithi045.stdout:2809: oid 49 version 2025 is already newer than 2023 2024-04-26T00:36:28.551 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 2025 (ObjNum 1013 snap 309 seq_num 1013) dirty exists 2024-04-26T00:36:28.552 INFO:tasks.rados.rados.0.smithi045.stdout:2809: left oid 49 (ObjNum 1013 snap 309 seq_num 1013) 2024-04-26T00:36:28.552 INFO:tasks.rados.rados.0.smithi045.stdout:2807: done (3 left) 2024-04-26T00:36:28.552 INFO:tasks.rados.rados.0.smithi045.stdout:2808: done (2 left) 2024-04-26T00:36:28.552 INFO:tasks.rados.rados.0.smithi045.stdout:2809: done (1 left) 2024-04-26T00:36:28.552 INFO:tasks.rados.rados.0.smithi045.stdout:2812: snap_remove snap 306 2024-04-26T00:36:29.553 INFO:tasks.rados.rados.0.smithi045.stdout:2812: done (1 left) 2024-04-26T00:36:29.554 INFO:tasks.rados.rados.0.smithi045.stdout:2813: rollback oid 10 current snap is 309 2024-04-26T00:36:29.554 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 10 to 308 2024-04-26T00:36:29.554 INFO:tasks.rados.rados.0.smithi045.stdout:2811: finishing copy_from racing read to smithi04526579-35 2024-04-26T00:36:29.554 INFO:tasks.rados.rados.0.smithi045.stdout:2811: finishing copy_from to smithi04526579-35 2024-04-26T00:36:29.554 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 2343 (ObjNum 976 snap 301 seq_num 976) dirty exists 2024-04-26T00:36:29.554 INFO:tasks.rados.rados.0.smithi045.stdout:2811: done (1 left) 2024-04-26T00:36:29.554 INFO:tasks.rados.rados.0.smithi045.stdout:2814: read oid 34 snap -1 2024-04-26T00:36:29.554 INFO:tasks.rados.rados.0.smithi045.stdout:2814: expect (ObjNum 987 snap 302 seq_num 987) 2024-04-26T00:36:29.554 INFO:tasks.rados.rados.0.smithi045.stdout:2815: snap_create 2024-04-26T00:36:29.572 INFO:tasks.rados.rados.0.smithi045.stdout:2814: expect (ObjNum 987 snap 302 seq_num 987) 2024-04-26T00:36:29.750 INFO:tasks.rados.rados.0.smithi045.stdout:2813: finishing rollback tid 0 to smithi04526579-10 2024-04-26T00:36:29.751 INFO:tasks.rados.rados.0.smithi045.stdout:2813: finishing rollback tid 1 to smithi04526579-10 2024-04-26T00:36:29.751 INFO:tasks.rados.rados.0.smithi045.stdout:2813: finishing rollback tid 2 to smithi04526579-10 2024-04-26T00:36:29.751 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 1971 (ObjNum 996 snap 303 seq_num 996) dirty exists 2024-04-26T00:36:30.559 INFO:tasks.rados.rados.0.smithi045.stdout:2813: done (2 left) 2024-04-26T00:36:30.560 INFO:tasks.rados.rados.0.smithi045.stdout:2814: done (1 left) 2024-04-26T00:36:30.560 INFO:tasks.rados.rados.0.smithi045.stdout:2815: done (0 left) 2024-04-26T00:36:30.560 INFO:tasks.rados.rados.0.smithi045.stdout:2816: rollback oid 50 current snap is 310 2024-04-26T00:36:30.560 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 50 to 309 2024-04-26T00:36:30.560 INFO:tasks.rados.rados.0.smithi045.stdout:2817: snap_remove snap 308 2024-04-26T00:36:31.571 INFO:tasks.rados.rados.0.smithi045.stdout:2817: done (1 left) 2024-04-26T00:36:31.571 INFO:tasks.rados.rados.0.smithi045.stdout:2818: rollback oid 45 current snap is 310 2024-04-26T00:36:31.571 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 45 to 309 2024-04-26T00:36:31.571 INFO:tasks.rados.rados.0.smithi045.stdout:2816: finishing rollback tid 1 to smithi04526579-50 2024-04-26T00:36:31.571 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 0 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-04-26T00:36:31.571 INFO:tasks.rados.rados.0.smithi045.stdout:2816: done (1 left) 2024-04-26T00:36:31.572 INFO:tasks.rados.rados.0.smithi045.stdout:2819: read oid 7 snap -1 2024-04-26T00:36:31.572 INFO:tasks.rados.rados.0.smithi045.stdout:2819: expect (ObjNum 998 snap 303 seq_num 998) 2024-04-26T00:36:31.572 INFO:tasks.rados.rados.0.smithi045.stdout:2820: read oid 25 snap -1 2024-04-26T00:36:31.572 INFO:tasks.rados.rados.0.smithi045.stdout:2820: expect (ObjNum 976 snap 301 seq_num 976) 2024-04-26T00:36:31.572 INFO:tasks.rados.rados.0.smithi045.stdout:2821: read oid 36 snap -1 2024-04-26T00:36:31.572 INFO:tasks.rados.rados.0.smithi045.stdout:2821: expect (ObjNum 980 snap 301 seq_num 980) 2024-04-26T00:36:31.572 INFO:tasks.rados.rados.0.smithi045.stdout:2822: read oid 41 snap -1 2024-04-26T00:36:31.572 INFO:tasks.rados.rados.0.smithi045.stdout:2822: expect deleted 2024-04-26T00:36:31.572 INFO:tasks.rados.rados.0.smithi045.stdout:2823: write oid 24 current snap is 310 2024-04-26T00:36:31.572 INFO:tasks.rados.rados.0.smithi045.stdout:2823: seq_num 1014 ranges {642298=577435,1920153=573357,2979150=63148} 2024-04-26T00:36:31.594 INFO:tasks.rados.rados.0.smithi045.stdout:2823: writing smithi04526579-24 from 642298 to 1219733 tid 1 2024-04-26T00:36:31.615 INFO:tasks.rados.rados.0.smithi045.stdout:2823: writing smithi04526579-24 from 1920153 to 2493510 tid 2 2024-04-26T00:36:31.623 INFO:tasks.rados.rados.0.smithi045.stdout:2823: writing smithi04526579-24 from 2979150 to 3042298 tid 3 2024-04-26T00:36:31.624 INFO:tasks.rados.rados.0.smithi045.stdout:2824: copy_from oid 23 from oid 14 current snap is 310 2024-04-26T00:36:31.624 INFO:tasks.rados.rados.0.smithi045.stdout:2821: expect (ObjNum 980 snap 301 seq_num 980) 2024-04-26T00:36:31.867 INFO:tasks.rados.rados.0.smithi045.stdout:2818: finishing rollback tid 0 to smithi04526579-45 2024-04-26T00:36:31.867 INFO:tasks.rados.rados.0.smithi045.stdout:2819: expect (ObjNum 998 snap 303 seq_num 998) 2024-04-26T00:36:32.037 INFO:tasks.rados.rados.0.smithi045.stdout:2818: finishing rollback tid 1 to smithi04526579-45 2024-04-26T00:36:32.037 INFO:tasks.rados.rados.0.smithi045.stdout:2818: finishing rollback tid 2 to smithi04526579-45 2024-04-26T00:36:32.037 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 2174 (ObjNum 999 snap 303 seq_num 999) dirty exists 2024-04-26T00:36:32.037 INFO:tasks.rados.rados.0.smithi045.stdout:2820: expect (ObjNum 976 snap 301 seq_num 976) 2024-04-26T00:36:32.112 INFO:tasks.rados.rados.0.smithi045.stdout:2823: finishing write tid 1 to smithi04526579-24 2024-04-26T00:36:32.113 INFO:tasks.rados.rados.0.smithi045.stdout:2823: finishing write tid 2 to smithi04526579-24 2024-04-26T00:36:32.113 INFO:tasks.rados.rados.0.smithi045.stdout:2823: finishing write tid 3 to smithi04526579-24 2024-04-26T00:36:32.113 INFO:tasks.rados.rados.0.smithi045.stdout:2823: finishing write tid 4 to smithi04526579-24 2024-04-26T00:36:32.113 INFO:tasks.rados.rados.0.smithi045.stdout:2823: finishing write tid 5 to smithi04526579-24 2024-04-26T00:36:32.113 INFO:tasks.rados.rados.0.smithi045.stdout:2823: oid 24 updating version 0 to 2384 2024-04-26T00:36:32.113 INFO:tasks.rados.rados.0.smithi045.stdout:2823: oid 24 version 2384 is already newer than 2382 2024-04-26T00:36:32.113 INFO:tasks.rados.rados.0.smithi045.stdout:2823: oid 24 updating version 2384 to 2385 2024-04-26T00:36:32.113 INFO:tasks.rados.rados.0.smithi045.stdout:2823: oid 24 version 2385 is already newer than 2383 2024-04-26T00:36:32.113 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 2385 (ObjNum 1014 snap 310 seq_num 1014) dirty exists 2024-04-26T00:36:32.113 INFO:tasks.rados.rados.0.smithi045.stdout:2823: left oid 24 (ObjNum 1014 snap 310 seq_num 1014) 2024-04-26T00:36:32.113 INFO:tasks.rados.rados.0.smithi045.stdout:2824: finishing copy_from racing read to smithi04526579-23 2024-04-26T00:36:32.113 INFO:tasks.rados.rados.0.smithi045.stdout:2824: finishing copy_from to smithi04526579-23 2024-04-26T00:36:32.113 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 1177 (ObjNum 1009 snap 308 seq_num 1009) dirty exists 2024-04-26T00:36:32.114 INFO:tasks.rados.rados.0.smithi045.stdout:2818: done (6 left) 2024-04-26T00:36:32.114 INFO:tasks.rados.rados.0.smithi045.stdout:2819: done (5 left) 2024-04-26T00:36:32.114 INFO:tasks.rados.rados.0.smithi045.stdout:2820: done (4 left) 2024-04-26T00:36:32.114 INFO:tasks.rados.rados.0.smithi045.stdout:2821: done (3 left) 2024-04-26T00:36:32.114 INFO:tasks.rados.rados.0.smithi045.stdout:2822: done (2 left) 2024-04-26T00:36:32.114 INFO:tasks.rados.rados.0.smithi045.stdout:2823: done (1 left) 2024-04-26T00:36:32.114 INFO:tasks.rados.rados.0.smithi045.stdout:2824: done (0 left) 2024-04-26T00:36:32.114 INFO:tasks.rados.rados.0.smithi045.stdout:2825: read oid 38 snap -1 2024-04-26T00:36:32.114 INFO:tasks.rados.rados.0.smithi045.stdout:2825: expect (ObjNum 1001 snap 304 seq_num 1001) 2024-04-26T00:36:32.114 INFO:tasks.rados.rados.0.smithi045.stdout:2826: read oid 48 snap -1 2024-04-26T00:36:32.114 INFO:tasks.rados.rados.0.smithi045.stdout:2826: expect (ObjNum 938 snap 290 seq_num 938) 2024-04-26T00:36:32.114 INFO:tasks.rados.rados.0.smithi045.stdout:2827: write oid 15 current snap is 310 2024-04-26T00:36:32.114 INFO:tasks.rados.rados.0.smithi045.stdout:2827: seq_num 1015 ranges {659105=652684,1862282=717015} 2024-04-26T00:36:32.138 INFO:tasks.rados.rados.0.smithi045.stdout:2827: writing smithi04526579-15 from 659105 to 1311789 tid 1 2024-04-26T00:36:32.160 INFO:tasks.rados.rados.0.smithi045.stdout:2827: writing smithi04526579-15 from 1862282 to 2579297 tid 2 2024-04-26T00:36:32.160 INFO:tasks.rados.rados.0.smithi045.stdout:2828: copy_from oid 40 from oid 12 current snap is 310 2024-04-26T00:36:32.160 INFO:tasks.rados.rados.0.smithi045.stdout:2826: expect (ObjNum 938 snap 290 seq_num 938) 2024-04-26T00:36:32.418 INFO:tasks.rados.rados.0.smithi045.stdout:2825: expect (ObjNum 1001 snap 304 seq_num 1001) 2024-04-26T00:36:32.513 INFO:tasks.rados.rados.0.smithi045.stdout:2827: finishing write tid 1 to smithi04526579-15 2024-04-26T00:36:32.513 INFO:tasks.rados.rados.0.smithi045.stdout:2825: done (3 left) 2024-04-26T00:36:32.513 INFO:tasks.rados.rados.0.smithi045.stdout:2826: done (2 left) 2024-04-26T00:36:32.513 INFO:tasks.rados.rados.0.smithi045.stdout:2829: write oid 41 current snap is 310 2024-04-26T00:36:32.513 INFO:tasks.rados.rados.0.smithi045.stdout:2827: finishing write tid 2 to smithi04526579-15 2024-04-26T00:36:32.513 INFO:tasks.rados.rados.0.smithi045.stdout:2827: finishing write tid 3 to smithi04526579-15 2024-04-26T00:36:32.513 INFO:tasks.rados.rados.0.smithi045.stdout:2827: finishing write tid 4 to smithi04526579-15 2024-04-26T00:36:32.513 INFO:tasks.rados.rados.0.smithi045.stdout:2827: oid 15 updating version 0 to 1998 2024-04-26T00:36:32.513 INFO:tasks.rados.rados.0.smithi045.stdout:2827: oid 15 updating version 1998 to 2000 2024-04-26T00:36:32.513 INFO:tasks.rados.rados.0.smithi045.stdout:2827: oid 15 version 2000 is already newer than 1999 2024-04-26T00:36:32.513 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 2000 (ObjNum 1015 snap 310 seq_num 1015) dirty exists 2024-04-26T00:36:32.514 INFO:tasks.rados.rados.0.smithi045.stdout:2827: left oid 15 (ObjNum 1015 snap 310 seq_num 1015) 2024-04-26T00:36:32.514 INFO:tasks.rados.rados.0.smithi045.stdout:2828: finishing copy_from racing read to smithi04526579-40 2024-04-26T00:36:32.514 INFO:tasks.rados.rados.0.smithi045.stdout:2829: seq_num 1016 ranges {675912=727933,1920764=460673,2801290=274622} 2024-04-26T00:36:32.520 INFO:tasks.rados.rados.0.smithi045.stdout:2829: writing smithi04526579-41 from 675912 to 1403845 tid 1 2024-04-26T00:36:32.526 INFO:tasks.rados.rados.0.smithi045.stdout:2829: writing smithi04526579-41 from 1920764 to 2381437 tid 2 2024-04-26T00:36:32.531 INFO:tasks.rados.rados.0.smithi045.stdout:2829: writing smithi04526579-41 from 2801290 to 3075912 tid 3 2024-04-26T00:36:32.531 INFO:tasks.rados.rados.0.smithi045.stdout:2827: done (2 left) 2024-04-26T00:36:32.531 INFO:tasks.rados.rados.0.smithi045.stdout:2830: read oid 35 snap 309 2024-04-26T00:36:32.531 INFO:tasks.rados.rados.0.smithi045.stdout:2830: expect (ObjNum 976 snap 301 seq_num 976) 2024-04-26T00:36:32.531 INFO:tasks.rados.rados.0.smithi045.stdout:2831: read oid 46 snap -1 2024-04-26T00:36:32.531 INFO:tasks.rados.rados.0.smithi045.stdout:2831: expect (ObjNum 984 snap 302 seq_num 984) 2024-04-26T00:36:32.531 INFO:tasks.rados.rados.0.smithi045.stdout:2832: rollback oid 2 current snap is 310 2024-04-26T00:36:32.532 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 2 to 309 2024-04-26T00:36:32.532 INFO:tasks.rados.rados.0.smithi045.stdout:2828: finishing copy_from to smithi04526579-40 2024-04-26T00:36:32.532 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 1996 (ObjNum 945 snap 292 seq_num 945) dirty exists 2024-04-26T00:36:32.532 INFO:tasks.rados.rados.0.smithi045.stdout:2828: done (4 left) 2024-04-26T00:36:32.532 INFO:tasks.rados.rados.0.smithi045.stdout:2833: read oid 47 snap 309 2024-04-26T00:36:32.532 INFO:tasks.rados.rados.0.smithi045.stdout:2833: expect deleted 2024-04-26T00:36:32.532 INFO:tasks.rados.rados.0.smithi045.stdout:2834: write (excl) oid 1 current snap is 310 2024-04-26T00:36:32.532 INFO:tasks.rados.rados.0.smithi045.stdout:2834: seq_num 1017 ranges {692719=403182,1579246=720684,2828713=264006} 2024-04-26T00:36:32.537 INFO:tasks.rados.rados.0.smithi045.stdout:2834: writing smithi04526579-1 from 692719 to 1095901 tid 1 2024-04-26T00:36:32.546 INFO:tasks.rados.rados.0.smithi045.stdout:2834: writing smithi04526579-1 from 1579246 to 2299930 tid 2 2024-04-26T00:36:32.550 INFO:tasks.rados.rados.0.smithi045.stdout:2834: writing smithi04526579-1 from 2828713 to 3092719 tid 3 2024-04-26T00:36:32.551 INFO:tasks.rados.rados.0.smithi045.stdout:2835: rollback oid 23 current snap is 310 2024-04-26T00:36:32.551 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 23 to 309 2024-04-26T00:36:32.551 INFO:tasks.rados.rados.0.smithi045.stdout:2831: expect (ObjNum 984 snap 302 seq_num 984) 2024-04-26T00:36:32.728 INFO:tasks.rados.rados.0.smithi045.stdout:2829: finishing write tid 1 to smithi04526579-41 2024-04-26T00:36:32.728 INFO:tasks.rados.rados.0.smithi045.stdout:2829: finishing write tid 2 to smithi04526579-41 2024-04-26T00:36:32.728 INFO:tasks.rados.rados.0.smithi045.stdout:2829: finishing write tid 3 to smithi04526579-41 2024-04-26T00:36:32.729 INFO:tasks.rados.rados.0.smithi045.stdout:2832: finishing rollback tid 0 to smithi04526579-2 2024-04-26T00:36:32.729 INFO:tasks.rados.rados.0.smithi045.stdout:2829: finishing write tid 4 to smithi04526579-41 2024-04-26T00:36:32.729 INFO:tasks.rados.rados.0.smithi045.stdout:2832: finishing rollback tid 1 to smithi04526579-2 2024-04-26T00:36:32.729 INFO:tasks.rados.rados.0.smithi045.stdout:2832: finishing rollback tid 2 to smithi04526579-2 2024-04-26T00:36:32.729 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 2014 (ObjNum 978 snap 301 seq_num 978) dirty exists 2024-04-26T00:36:32.729 INFO:tasks.rados.rados.0.smithi045.stdout:2830: expect (ObjNum 976 snap 301 seq_num 976) 2024-04-26T00:36:32.803 INFO:tasks.rados.rados.0.smithi045.stdout:2829: finishing write tid 5 to smithi04526579-41 2024-04-26T00:36:32.803 INFO:tasks.rados.rados.0.smithi045.stdout:2829: oid 41 updating version 0 to 2004 2024-04-26T00:36:32.803 INFO:tasks.rados.rados.0.smithi045.stdout:2829: oid 41 version 2004 is already newer than 2003 2024-04-26T00:36:32.803 INFO:tasks.rados.rados.0.smithi045.stdout:2829: oid 41 version 2004 is already newer than 2002 2024-04-26T00:36:32.803 INFO:tasks.rados.rados.0.smithi045.stdout:2829: oid 41 version 2004 is already newer than 2001 2024-04-26T00:36:32.803 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 2004 (ObjNum 1016 snap 310 seq_num 1016) dirty exists 2024-04-26T00:36:32.803 INFO:tasks.rados.rados.0.smithi045.stdout:2829: left oid 41 (ObjNum 1016 snap 310 seq_num 1016) 2024-04-26T00:36:32.803 INFO:tasks.rados.rados.0.smithi045.stdout:2834: finishing write tid 1 to smithi04526579-1 2024-04-26T00:36:32.803 INFO:tasks.rados.rados.0.smithi045.stdout:2834: finishing write tid 2 to smithi04526579-1 2024-04-26T00:36:32.803 INFO:tasks.rados.rados.0.smithi045.stdout:2834: finishing write tid 3 to smithi04526579-1 2024-04-26T00:36:32.803 INFO:tasks.rados.rados.0.smithi045.stdout:2834: finishing write tid 4 to smithi04526579-1 2024-04-26T00:36:32.804 INFO:tasks.rados.rados.0.smithi045.stdout:2834: finishing write tid 5 to smithi04526579-1 2024-04-26T00:36:32.804 INFO:tasks.rados.rados.0.smithi045.stdout:2834: oid 1 updating version 0 to 2005 2024-04-26T00:36:32.804 INFO:tasks.rados.rados.0.smithi045.stdout:2834: oid 1 updating version 2005 to 2007 2024-04-26T00:36:32.804 INFO:tasks.rados.rados.0.smithi045.stdout:2834: oid 1 updating version 2007 to 2008 2024-04-26T00:36:32.804 INFO:tasks.rados.rados.0.smithi045.stdout:2834: oid 1 version 2008 is already newer than 2006 2024-04-26T00:36:32.804 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 2008 (ObjNum 1017 snap 310 seq_num 1017) dirty exists 2024-04-26T00:36:32.804 INFO:tasks.rados.rados.0.smithi045.stdout:2834: left oid 1 (ObjNum 1017 snap 310 seq_num 1017) 2024-04-26T00:36:32.804 INFO:tasks.rados.rados.0.smithi045.stdout:2835: finishing rollback tid 0 to smithi04526579-23 2024-04-26T00:36:32.804 INFO:tasks.rados.rados.0.smithi045.stdout:2835: finishing rollback tid 1 to smithi04526579-23 2024-04-26T00:36:32.804 INFO:tasks.rados.rados.0.smithi045.stdout:2829: done (6 left) 2024-04-26T00:36:32.804 INFO:tasks.rados.rados.0.smithi045.stdout:2830: done (5 left) 2024-04-26T00:36:32.804 INFO:tasks.rados.rados.0.smithi045.stdout:2831: done (4 left) 2024-04-26T00:36:32.804 INFO:tasks.rados.rados.0.smithi045.stdout:2832: done (3 left) 2024-04-26T00:36:32.804 INFO:tasks.rados.rados.0.smithi045.stdout:2833: done (2 left) 2024-04-26T00:36:32.804 INFO:tasks.rados.rados.0.smithi045.stdout:2834: done (1 left) 2024-04-26T00:36:32.805 INFO:tasks.rados.rados.0.smithi045.stdout:2836: delete oid 19 current snap is 310 2024-04-26T00:36:32.805 INFO:tasks.rados.rados.0.smithi045.stdout:2835: finishing rollback tid 2 to smithi04526579-23 2024-04-26T00:36:32.805 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 2003 (ObjNum 945 snap 292 seq_num 945) dirty exists 2024-04-26T00:36:32.811 INFO:tasks.rados.rados.0.smithi045.stdout:2835: done (1 left) 2024-04-26T00:36:32.811 INFO:tasks.rados.rados.0.smithi045.stdout:2836: done (0 left) 2024-04-26T00:36:32.811 INFO:tasks.rados.rados.0.smithi045.stdout:2837: write (excl) oid 36 current snap is 310 2024-04-26T00:36:32.811 INFO:tasks.rados.rados.0.smithi045.stdout:2837: seq_num 1019 ranges {726333=670033,2096210=724353} 2024-04-26T00:36:32.830 INFO:tasks.rados.rados.0.smithi045.stdout:2837: writing smithi04526579-36 from 726333 to 1396366 tid 1 2024-04-26T00:36:32.851 INFO:tasks.rados.rados.0.smithi045.stdout:2837: writing smithi04526579-36 from 2096210 to 2820563 tid 2 2024-04-26T00:36:32.851 INFO:tasks.rados.rados.0.smithi045.stdout:2838: snap_remove snap 309 2024-04-26T00:36:33.551 INFO:tasks.rados.rados.0.smithi045.stdout:2838: done (1 left) 2024-04-26T00:36:33.551 INFO:tasks.rados.rados.0.smithi045.stdout:2839: rollback oid 24 current snap is 310 2024-04-26T00:36:33.551 INFO:tasks.rados.rados.0.smithi045.stdout:2839: done (1 left) 2024-04-26T00:36:33.552 INFO:tasks.rados.rados.0.smithi045.stdout:2840: write oid 47 current snap is 310 2024-04-26T00:36:33.552 INFO:tasks.rados.rados.0.smithi045.stdout:2840: seq_num 1020 ranges {743140=745282,2154692=468011} 2024-04-26T00:36:33.574 INFO:tasks.rados.rados.0.smithi045.stdout:2840: writing smithi04526579-47 from 743140 to 1488422 tid 1 2024-04-26T00:36:33.593 INFO:tasks.rados.rados.0.smithi045.stdout:2840: writing smithi04526579-47 from 2154692 to 2622703 tid 2 2024-04-26T00:36:33.593 INFO:tasks.rados.rados.0.smithi045.stdout:2841: write (excl) oid 46 current snap is 310 2024-04-26T00:36:33.593 INFO:tasks.rados.rados.0.smithi045.stdout:2841: seq_num 1021 ranges {759947=420531,1813174=728022} 2024-04-26T00:36:33.602 INFO:tasks.rados.rados.0.smithi045.stdout:2841: writing smithi04526579-46 from 759947 to 1180478 tid 1 2024-04-26T00:36:33.613 INFO:tasks.rados.rados.0.smithi045.stdout:2841: writing smithi04526579-46 from 1813174 to 2541196 tid 2 2024-04-26T00:36:33.613 INFO:tasks.rados.rados.0.smithi045.stdout:2842: read oid 8 snap -1 2024-04-26T00:36:33.613 INFO:tasks.rados.rados.0.smithi045.stdout:2842: expect (ObjNum 990 snap 303 seq_num 990) 2024-04-26T00:36:33.613 INFO:tasks.rados.rados.0.smithi045.stdout:2843: read oid 22 snap -1 2024-04-26T00:36:33.613 INFO:tasks.rados.rados.0.smithi045.stdout:2843: expect (ObjNum 942 snap 291 seq_num 942) 2024-04-26T00:36:33.613 INFO:tasks.rados.rados.0.smithi045.stdout:2844: write oid 50 current snap is 310 2024-04-26T00:36:33.613 INFO:tasks.rados.rados.0.smithi045.stdout:2844: seq_num 1022 ranges {776754=495780,1871656=471680,2965828=210926} 2024-04-26T00:36:33.624 INFO:tasks.rados.rados.0.smithi045.stdout:2844: writing smithi04526579-50 from 776754 to 1272534 tid 1 2024-04-26T00:36:33.632 INFO:tasks.rados.rados.0.smithi045.stdout:2844: writing smithi04526579-50 from 1871656 to 2343336 tid 2 2024-04-26T00:36:33.639 INFO:tasks.rados.rados.0.smithi045.stdout:2844: writing smithi04526579-50 from 2965828 to 3176754 tid 3 2024-04-26T00:36:33.639 INFO:tasks.rados.rados.0.smithi045.stdout:2845: write (excl) oid 40 current snap is 310 2024-04-26T00:36:33.639 INFO:tasks.rados.rados.0.smithi045.stdout:2845: seq_num 1023 ranges {793561=571029,1813785=731691,2993251=200310} 2024-04-26T00:36:33.649 INFO:tasks.rados.rados.0.smithi045.stdout:2845: writing smithi04526579-40 from 793561 to 1364590 tid 1 2024-04-26T00:36:33.656 INFO:tasks.rados.rados.0.smithi045.stdout:2845: writing smithi04526579-40 from 1813785 to 2545476 tid 2 2024-04-26T00:36:33.660 INFO:tasks.rados.rados.0.smithi045.stdout:2845: writing smithi04526579-40 from 2993251 to 3193561 tid 3 2024-04-26T00:36:33.660 INFO:tasks.rados.rados.0.smithi045.stdout:2846: write oid 6 current snap is 310 2024-04-26T00:36:33.660 INFO:tasks.rados.rados.0.smithi045.stdout:2846: seq_num 1024 ranges {410368=646278,1472267=475349,2504321=405600} 2024-04-26T00:36:33.666 INFO:tasks.rados.rados.0.smithi045.stdout:2846: writing smithi04526579-6 from 410368 to 1056646 tid 1 2024-04-26T00:36:33.672 INFO:tasks.rados.rados.0.smithi045.stdout:2846: writing smithi04526579-6 from 1472267 to 1947616 tid 2 2024-04-26T00:36:33.678 INFO:tasks.rados.rados.0.smithi045.stdout:2846: writing smithi04526579-6 from 2504321 to 2909921 tid 3 2024-04-26T00:36:33.678 INFO:tasks.rados.rados.0.smithi045.stdout:2847: read oid 44 snap -1 2024-04-26T00:36:33.678 INFO:tasks.rados.rados.0.smithi045.stdout:2847: expect (ObjNum 952 snap 295 seq_num 952) 2024-04-26T00:36:33.678 INFO:tasks.rados.rados.0.smithi045.stdout:2848: write oid 2 current snap is 310 2024-04-26T00:36:33.678 INFO:tasks.rados.rados.0.smithi045.stdout:2837: finishing write tid 1 to smithi04526579-36 2024-04-26T00:36:33.678 INFO:tasks.rados.rados.0.smithi045.stdout:2848: seq_num 1025 ranges {427175=721527,1930749=735360} 2024-04-26T00:36:33.685 INFO:tasks.rados.rados.0.smithi045.stdout:2848: writing smithi04526579-2 from 427175 to 1148702 tid 1 2024-04-26T00:36:33.694 INFO:tasks.rados.rados.0.smithi045.stdout:2848: writing smithi04526579-2 from 1930749 to 2666109 tid 2 2024-04-26T00:36:33.694 INFO:tasks.rados.rados.0.smithi045.stdout:2849: write (excl) oid 9 current snap is 310 2024-04-26T00:36:33.694 INFO:tasks.rados.rados.0.smithi045.stdout:2849: seq_num 1026 ranges {443982=796776,1989231=479018,2959167=284815} 2024-04-26T00:36:33.700 INFO:tasks.rados.rados.0.smithi045.stdout:2849: writing smithi04526579-9 from 443982 to 1240758 tid 1 2024-04-26T00:36:33.707 INFO:tasks.rados.rados.0.smithi045.stdout:2849: writing smithi04526579-9 from 1989231 to 2468249 tid 2 2024-04-26T00:36:33.712 INFO:tasks.rados.rados.0.smithi045.stdout:2849: writing smithi04526579-9 from 2959167 to 3243982 tid 3 2024-04-26T00:36:33.712 INFO:tasks.rados.rados.0.smithi045.stdout:2850: rollback oid 37 current snap is 310 2024-04-26T00:36:33.712 INFO:tasks.rados.rados.0.smithi045.stdout:2850: done (11 left) 2024-04-26T00:36:33.712 INFO:tasks.rados.rados.0.smithi045.stdout:2851: write oid 26 current snap is 310 2024-04-26T00:36:33.712 INFO:tasks.rados.rados.0.smithi045.stdout:2851: seq_num 1027 ranges {460789=588378,1764066=739029,3219296=41493} 2024-04-26T00:36:33.718 INFO:tasks.rados.rados.0.smithi045.stdout:2851: writing smithi04526579-26 from 460789 to 1049167 tid 1 2024-04-26T00:36:33.726 INFO:tasks.rados.rados.0.smithi045.stdout:2851: writing smithi04526579-26 from 1764066 to 2503095 tid 2 2024-04-26T00:36:33.731 INFO:tasks.rados.rados.0.smithi045.stdout:2851: writing smithi04526579-26 from 3219296 to 3260789 tid 3 2024-04-26T00:36:33.731 INFO:tasks.rados.rados.0.smithi045.stdout:2852: write (excl) oid 10 current snap is 310 2024-04-26T00:36:33.731 INFO:tasks.rados.rados.0.smithi045.stdout:2852: seq_num 1028 ranges {477596=663627,1706195=482687,2614013=567041} 2024-04-26T00:36:33.738 INFO:tasks.rados.rados.0.smithi045.stdout:2852: writing smithi04526579-10 from 477596 to 1141223 tid 1 2024-04-26T00:36:33.744 INFO:tasks.rados.rados.0.smithi045.stdout:2852: writing smithi04526579-10 from 1706195 to 2188882 tid 2 2024-04-26T00:36:33.750 INFO:tasks.rados.rados.0.smithi045.stdout:2852: writing smithi04526579-10 from 2614013 to 3181054 tid 3 2024-04-26T00:36:33.750 INFO:tasks.rados.rados.0.smithi045.stdout:2853: rollback oid 19 current snap is 310 2024-04-26T00:36:33.750 INFO:tasks.rados.rados.0.smithi045.stdout:2853: done (13 left) 2024-04-26T00:36:33.750 INFO:tasks.rados.rados.0.smithi045.stdout:2854: copy_from oid 13 from oid 33 current snap is 310 2024-04-26T00:36:33.750 INFO:tasks.rados.rados.0.smithi045.stdout:2837: finishing write tid 2 to smithi04526579-36 2024-04-26T00:36:33.750 INFO:tasks.rados.rados.0.smithi045.stdout:2837: finishing write tid 3 to smithi04526579-36 2024-04-26T00:36:33.750 INFO:tasks.rados.rados.0.smithi045.stdout:2837: finishing write tid 4 to smithi04526579-36 2024-04-26T00:36:33.750 INFO:tasks.rados.rados.0.smithi045.stdout:2837: oid 36 updating version 0 to 2364 2024-04-26T00:36:33.751 INFO:tasks.rados.rados.0.smithi045.stdout:2837: oid 36 version 2364 is already newer than 2363 2024-04-26T00:36:33.751 INFO:tasks.rados.rados.0.smithi045.stdout:2837: oid 36 version 2364 is already newer than 2362 2024-04-26T00:36:33.751 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 2364 (ObjNum 1019 snap 310 seq_num 1019) dirty exists 2024-04-26T00:36:33.751 INFO:tasks.rados.rados.0.smithi045.stdout:2837: left oid 36 (ObjNum 1019 snap 310 seq_num 1019) 2024-04-26T00:36:33.751 INFO:tasks.rados.rados.0.smithi045.stdout:2840: finishing write tid 1 to smithi04526579-47 2024-04-26T00:36:33.751 INFO:tasks.rados.rados.0.smithi045.stdout:2840: finishing write tid 2 to smithi04526579-47 2024-04-26T00:36:33.751 INFO:tasks.rados.rados.0.smithi045.stdout:2840: finishing write tid 3 to smithi04526579-47 2024-04-26T00:36:33.751 INFO:tasks.rados.rados.0.smithi045.stdout:2840: finishing write tid 4 to smithi04526579-47 2024-04-26T00:36:33.751 INFO:tasks.rados.rados.0.smithi045.stdout:2840: oid 47 updating version 0 to 2028 2024-04-26T00:36:33.751 INFO:tasks.rados.rados.0.smithi045.stdout:2840: oid 47 version 2028 is already newer than 2026 2024-04-26T00:36:33.751 INFO:tasks.rados.rados.0.smithi045.stdout:2840: oid 47 version 2028 is already newer than 2027 2024-04-26T00:36:33.751 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 2028 (ObjNum 1020 snap 310 seq_num 1020) dirty exists 2024-04-26T00:36:33.751 INFO:tasks.rados.rados.0.smithi045.stdout:2840: left oid 47 (ObjNum 1020 snap 310 seq_num 1020) 2024-04-26T00:36:33.751 INFO:tasks.rados.rados.0.smithi045.stdout:2837: done (13 left) 2024-04-26T00:36:33.751 INFO:tasks.rados.rados.0.smithi045.stdout:2840: done (12 left) 2024-04-26T00:36:33.752 INFO:tasks.rados.rados.0.smithi045.stdout:2855: write (excl) oid 49 current snap is 310 2024-04-26T00:36:33.752 INFO:tasks.rados.rados.0.smithi045.stdout:2855: seq_num 1029 ranges {494403=738876,1764677=626345,3041436=252967} 2024-04-26T00:36:33.756 INFO:tasks.rados.rados.0.smithi045.stdout:2855: writing smithi04526579-49 from 494403 to 1233279 tid 1 2024-04-26T00:36:33.765 INFO:tasks.rados.rados.0.smithi045.stdout:2855: writing smithi04526579-49 from 1764677 to 2391022 tid 2 2024-04-26T00:36:33.770 INFO:tasks.rados.rados.0.smithi045.stdout:2855: writing smithi04526579-49 from 3041436 to 3294403 tid 3 2024-04-26T00:36:33.770 INFO:tasks.rados.rados.0.smithi045.stdout:2856: read oid 3 snap -1 2024-04-26T00:36:33.770 INFO:tasks.rados.rados.0.smithi045.stdout:2856: expect (ObjNum 1011 snap 309 seq_num 1011) 2024-04-26T00:36:33.770 INFO:tasks.rados.rados.0.smithi045.stdout:2857: copy_from oid 25 from oid 23 current snap is 310 2024-04-26T00:36:33.770 INFO:tasks.rados.rados.0.smithi045.stdout:2841: finishing write tid 1 to smithi04526579-46 2024-04-26T00:36:33.770 INFO:tasks.rados.rados.0.smithi045.stdout:2841: finishing write tid 2 to smithi04526579-46 2024-04-26T00:36:33.770 INFO:tasks.rados.rados.0.smithi045.stdout:2843: expect (ObjNum 942 snap 291 seq_num 942) 2024-04-26T00:36:34.067 INFO:tasks.rados.rados.0.smithi045.stdout:2841: finishing write tid 3 to smithi04526579-46 2024-04-26T00:36:34.067 INFO:tasks.rados.rados.0.smithi045.stdout:2841: finishing write tid 4 to smithi04526579-46 2024-04-26T00:36:34.067 INFO:tasks.rados.rados.0.smithi045.stdout:2841: oid 46 updating version 0 to 2367 2024-04-26T00:36:34.067 INFO:tasks.rados.rados.0.smithi045.stdout:2841: oid 46 version 2367 is already newer than 2365 2024-04-26T00:36:34.067 INFO:tasks.rados.rados.0.smithi045.stdout:2841: oid 46 version 2367 is already newer than 2366 2024-04-26T00:36:34.068 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 2367 (ObjNum 1021 snap 310 seq_num 1021) dirty exists 2024-04-26T00:36:34.068 INFO:tasks.rados.rados.0.smithi045.stdout:2841: left oid 46 (ObjNum 1021 snap 310 seq_num 1021) 2024-04-26T00:36:34.068 INFO:tasks.rados.rados.0.smithi045.stdout:2842: expect (ObjNum 990 snap 303 seq_num 990) 2024-04-26T00:36:34.237 INFO:tasks.rados.rados.0.smithi045.stdout:2844: finishing write tid 1 to smithi04526579-50 2024-04-26T00:36:34.237 INFO:tasks.rados.rados.0.smithi045.stdout:2844: finishing write tid 2 to smithi04526579-50 2024-04-26T00:36:34.237 INFO:tasks.rados.rados.0.smithi045.stdout:2844: finishing write tid 3 to smithi04526579-50 2024-04-26T00:36:34.237 INFO:tasks.rados.rados.0.smithi045.stdout:2844: finishing write tid 4 to smithi04526579-50 2024-04-26T00:36:34.237 INFO:tasks.rados.rados.0.smithi045.stdout:2844: finishing write tid 5 to smithi04526579-50 2024-04-26T00:36:34.238 INFO:tasks.rados.rados.0.smithi045.stdout:2844: oid 50 updating version 0 to 1975 2024-04-26T00:36:34.238 INFO:tasks.rados.rados.0.smithi045.stdout:2844: oid 50 version 1975 is already newer than 1972 2024-04-26T00:36:34.238 INFO:tasks.rados.rados.0.smithi045.stdout:2844: oid 50 version 1975 is already newer than 1973 2024-04-26T00:36:34.238 INFO:tasks.rados.rados.0.smithi045.stdout:2844: oid 50 version 1975 is already newer than 1974 2024-04-26T00:36:34.238 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 1975 (ObjNum 1022 snap 310 seq_num 1022) dirty exists 2024-04-26T00:36:34.238 INFO:tasks.rados.rados.0.smithi045.stdout:2844: left oid 50 (ObjNum 1022 snap 310 seq_num 1022) 2024-04-26T00:36:34.238 INFO:tasks.rados.rados.0.smithi045.stdout:2845: finishing write tid 1 to smithi04526579-40 2024-04-26T00:36:34.238 INFO:tasks.rados.rados.0.smithi045.stdout:2845: finishing write tid 2 to smithi04526579-40 2024-04-26T00:36:34.238 INFO:tasks.rados.rados.0.smithi045.stdout:2845: finishing write tid 3 to smithi04526579-40 2024-04-26T00:36:34.238 INFO:tasks.rados.rados.0.smithi045.stdout:2845: finishing write tid 4 to smithi04526579-40 2024-04-26T00:36:34.238 INFO:tasks.rados.rados.0.smithi045.stdout:2845: finishing write tid 5 to smithi04526579-40 2024-04-26T00:36:34.238 INFO:tasks.rados.rados.0.smithi045.stdout:2845: oid 40 updating version 0 to 2004 2024-04-26T00:36:34.238 INFO:tasks.rados.rados.0.smithi045.stdout:2845: oid 40 updating version 2004 to 2005 2024-04-26T00:36:34.238 INFO:tasks.rados.rados.0.smithi045.stdout:2845: oid 40 updating version 2005 to 2006 2024-04-26T00:36:34.239 INFO:tasks.rados.rados.0.smithi045.stdout:2845: oid 40 updating version 2006 to 2007 2024-04-26T00:36:34.239 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 2007 (ObjNum 1023 snap 310 seq_num 1023) dirty exists 2024-04-26T00:36:34.239 INFO:tasks.rados.rados.0.smithi045.stdout:2845: left oid 40 (ObjNum 1023 snap 310 seq_num 1023) 2024-04-26T00:36:34.239 INFO:tasks.rados.rados.0.smithi045.stdout:2846: finishing write tid 1 to smithi04526579-6 2024-04-26T00:36:34.239 INFO:tasks.rados.rados.0.smithi045.stdout:2841: done (14 left) 2024-04-26T00:36:34.239 INFO:tasks.rados.rados.0.smithi045.stdout:2842: done (13 left) 2024-04-26T00:36:34.239 INFO:tasks.rados.rados.0.smithi045.stdout:2843: done (12 left) 2024-04-26T00:36:34.239 INFO:tasks.rados.rados.0.smithi045.stdout:2844: done (11 left) 2024-04-26T00:36:34.239 INFO:tasks.rados.rados.0.smithi045.stdout:2845: done (10 left) 2024-04-26T00:36:34.239 INFO:tasks.rados.rados.0.smithi045.stdout:2858: rollback oid 15 current snap is 310 2024-04-26T00:36:34.239 INFO:tasks.rados.rados.0.smithi045.stdout:2858: done (10 left) 2024-04-26T00:36:34.239 INFO:tasks.rados.rados.0.smithi045.stdout:2859: snap_create 2024-04-26T00:36:34.239 INFO:tasks.rados.rados.0.smithi045.stdout:2846: finishing write tid 2 to smithi04526579-6 2024-04-26T00:36:34.239 INFO:tasks.rados.rados.0.smithi045.stdout:2846: finishing write tid 3 to smithi04526579-6 2024-04-26T00:36:34.239 INFO:tasks.rados.rados.0.smithi045.stdout:2846: finishing write tid 4 to smithi04526579-6 2024-04-26T00:36:34.240 INFO:tasks.rados.rados.0.smithi045.stdout:2846: finishing write tid 5 to smithi04526579-6 2024-04-26T00:36:34.240 INFO:tasks.rados.rados.0.smithi045.stdout:2846: oid 6 updating version 0 to 1982 2024-04-26T00:36:34.240 INFO:tasks.rados.rados.0.smithi045.stdout:2846: oid 6 updating version 1982 to 1984 2024-04-26T00:36:34.240 INFO:tasks.rados.rados.0.smithi045.stdout:2846: oid 6 version 1984 is already newer than 1983 2024-04-26T00:36:34.240 INFO:tasks.rados.rados.0.smithi045.stdout:2846: oid 6 updating version 1984 to 1985 2024-04-26T00:36:34.240 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 1985 (ObjNum 1024 snap 310 seq_num 1024) dirty exists 2024-04-26T00:36:34.240 INFO:tasks.rados.rados.0.smithi045.stdout:2846: left oid 6 (ObjNum 1024 snap 310 seq_num 1024) 2024-04-26T00:36:34.240 INFO:tasks.rados.rados.0.smithi045.stdout:2847: expect (ObjNum 952 snap 295 seq_num 952) 2024-04-26T00:36:34.303 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-04-26T00:36:34.303 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-04-26T00:36:34.303 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-26T00:36:34.303 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-26T00:36:34.306 INFO:tasks.ceph.mon.c:Started 2024-04-26T00:36:34.306 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:36:34.306 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:36:34.343 INFO:tasks.rados.rados.0.smithi045.stdout:2848: finishing write tid 1 to smithi04526579-2 2024-04-26T00:36:34.343 INFO:tasks.rados.rados.0.smithi045.stdout:2848: finishing write tid 2 to smithi04526579-2 2024-04-26T00:36:34.343 INFO:tasks.rados.rados.0.smithi045.stdout:2848: finishing write tid 3 to smithi04526579-2 2024-04-26T00:36:34.343 INFO:tasks.rados.rados.0.smithi045.stdout:2848: finishing write tid 4 to smithi04526579-2 2024-04-26T00:36:34.343 INFO:tasks.rados.rados.0.smithi045.stdout:2848: oid 2 updating version 0 to 2015 2024-04-26T00:36:34.343 INFO:tasks.rados.rados.0.smithi045.stdout:2848: oid 2 updating version 2015 to 2017 2024-04-26T00:36:34.343 INFO:tasks.rados.rados.0.smithi045.stdout:2848: oid 2 version 2017 is already newer than 2016 2024-04-26T00:36:34.343 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 2017 (ObjNum 1025 snap 310 seq_num 1025) dirty exists 2024-04-26T00:36:34.343 INFO:tasks.rados.rados.0.smithi045.stdout:2848: left oid 2 (ObjNum 1025 snap 310 seq_num 1025) 2024-04-26T00:36:34.343 INFO:tasks.rados.rados.0.smithi045.stdout:2849: finishing write tid 1 to smithi04526579-9 2024-04-26T00:36:34.343 INFO:tasks.rados.rados.0.smithi045.stdout:2849: finishing write tid 2 to smithi04526579-9 2024-04-26T00:36:34.344 INFO:tasks.rados.rados.0.smithi045.stdout:2849: finishing write tid 3 to smithi04526579-9 2024-04-26T00:36:34.344 INFO:tasks.rados.rados.0.smithi045.stdout:2849: finishing write tid 4 to smithi04526579-9 2024-04-26T00:36:34.344 INFO:tasks.rados.rados.0.smithi045.stdout:2849: finishing write tid 5 to smithi04526579-9 2024-04-26T00:36:34.344 INFO:tasks.rados.rados.0.smithi045.stdout:2849: oid 9 updating version 0 to 2018 2024-04-26T00:36:34.344 INFO:tasks.rados.rados.0.smithi045.stdout:2849: oid 9 updating version 2018 to 2020 2024-04-26T00:36:34.344 INFO:tasks.rados.rados.0.smithi045.stdout:2849: oid 9 updating version 2020 to 2021 2024-04-26T00:36:34.344 INFO:tasks.rados.rados.0.smithi045.stdout:2849: oid 9 version 2021 is already newer than 2019 2024-04-26T00:36:34.344 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 2021 (ObjNum 1026 snap 310 seq_num 1026) dirty exists 2024-04-26T00:36:34.344 INFO:tasks.rados.rados.0.smithi045.stdout:2849: left oid 9 (ObjNum 1026 snap 310 seq_num 1026) 2024-04-26T00:36:34.344 INFO:tasks.rados.rados.0.smithi045.stdout:2851: finishing write tid 1 to smithi04526579-26 2024-04-26T00:36:34.344 INFO:tasks.rados.rados.0.smithi045.stdout:2851: finishing write tid 2 to smithi04526579-26 2024-04-26T00:36:34.344 INFO:tasks.rados.rados.0.smithi045.stdout:2851: finishing write tid 3 to smithi04526579-26 2024-04-26T00:36:34.344 INFO:tasks.rados.rados.0.smithi045.stdout:2851: finishing write tid 4 to smithi04526579-26 2024-04-26T00:36:34.344 INFO:tasks.rados.rados.0.smithi045.stdout:2851: finishing write tid 5 to smithi04526579-26 2024-04-26T00:36:34.345 INFO:tasks.rados.rados.0.smithi045.stdout:2851: oid 26 updating version 0 to 2011 2024-04-26T00:36:34.345 INFO:tasks.rados.rados.0.smithi045.stdout:2851: oid 26 updating version 2011 to 2012 2024-04-26T00:36:34.345 INFO:tasks.rados.rados.0.smithi045.stdout:2851: oid 26 version 2012 is already newer than 2009 2024-04-26T00:36:34.345 INFO:tasks.rados.rados.0.smithi045.stdout:2851: oid 26 version 2012 is already newer than 2010 2024-04-26T00:36:34.345 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 2012 (ObjNum 1027 snap 310 seq_num 1027) dirty exists 2024-04-26T00:36:34.345 INFO:tasks.rados.rados.0.smithi045.stdout:2851: left oid 26 (ObjNum 1027 snap 310 seq_num 1027) 2024-04-26T00:36:34.345 INFO:tasks.rados.rados.0.smithi045.stdout:2852: finishing write tid 1 to smithi04526579-10 2024-04-26T00:36:34.345 INFO:tasks.rados.rados.0.smithi045.stdout:2852: finishing write tid 2 to smithi04526579-10 2024-04-26T00:36:34.345 INFO:tasks.rados.rados.0.smithi045.stdout:2854: finishing copy_from to smithi04526579-13 2024-04-26T00:36:34.345 INFO:tasks.rados.rados.0.smithi045.stdout:2854: got expected ENOENT (src dne) 2024-04-26T00:36:34.345 INFO:tasks.rados.rados.0.smithi045.stdout:2854: finishing copy_from racing read to smithi04526579-13 2024-04-26T00:36:34.345 INFO:tasks.rados.rados.0.smithi045.stdout:2852: finishing write tid 3 to smithi04526579-10 2024-04-26T00:36:34.345 INFO:tasks.rados.rados.0.smithi045.stdout:2852: finishing write tid 4 to smithi04526579-10 2024-04-26T00:36:34.345 INFO:tasks.rados.rados.0.smithi045.stdout:2852: finishing write tid 5 to smithi04526579-10 2024-04-26T00:36:34.345 INFO:tasks.rados.rados.0.smithi045.stdout:2852: oid 10 updating version 0 to 1977 2024-04-26T00:36:34.346 INFO:tasks.rados.rados.0.smithi045.stdout:2852: oid 10 version 1977 is already newer than 1976 2024-04-26T00:36:34.346 INFO:tasks.rados.rados.0.smithi045.stdout:2852: oid 10 updating version 1977 to 1979 2024-04-26T00:36:34.346 INFO:tasks.rados.rados.0.smithi045.stdout:2852: oid 10 version 1979 is already newer than 1978 2024-04-26T00:36:34.346 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 1979 (ObjNum 1028 snap 310 seq_num 1028) dirty exists 2024-04-26T00:36:34.346 INFO:tasks.rados.rados.0.smithi045.stdout:2852: left oid 10 (ObjNum 1028 snap 310 seq_num 1028) 2024-04-26T00:36:34.346 INFO:tasks.rados.rados.0.smithi045.stdout:2855: finishing write tid 1 to smithi04526579-49 2024-04-26T00:36:34.346 INFO:tasks.rados.rados.0.smithi045.stdout:2855: finishing write tid 2 to smithi04526579-49 2024-04-26T00:36:34.346 INFO:tasks.rados.rados.0.smithi045.stdout:2855: finishing write tid 3 to smithi04526579-49 2024-04-26T00:36:34.346 INFO:tasks.rados.rados.0.smithi045.stdout:2855: finishing write tid 4 to smithi04526579-49 2024-04-26T00:36:34.346 INFO:tasks.rados.rados.0.smithi045.stdout:2855: finishing write tid 5 to smithi04526579-49 2024-04-26T00:36:34.346 INFO:tasks.rados.rados.0.smithi045.stdout:2855: oid 49 updating version 0 to 2032 2024-04-26T00:36:34.346 INFO:tasks.rados.rados.0.smithi045.stdout:2855: oid 49 version 2032 is already newer than 2030 2024-04-26T00:36:34.346 INFO:tasks.rados.rados.0.smithi045.stdout:2855: oid 49 version 2032 is already newer than 2029 2024-04-26T00:36:34.346 INFO:tasks.rados.rados.0.smithi045.stdout:2855: oid 49 version 2032 is already newer than 2031 2024-04-26T00:36:34.347 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 2032 (ObjNum 1029 snap 310 seq_num 1029) dirty exists 2024-04-26T00:36:34.347 INFO:tasks.rados.rados.0.smithi045.stdout:2855: left oid 49 (ObjNum 1029 snap 310 seq_num 1029) 2024-04-26T00:36:34.347 INFO:tasks.rados.rados.0.smithi045.stdout:2856: expect (ObjNum 1011 snap 309 seq_num 1011) 2024-04-26T00:36:34.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.424+0000 7fc51bf9d640 10 monclient: get_monmap_and_config 2024-04-26T00:36:34.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.424+0000 7fc51bf9d640 10 monclient: build_initial_monmap 2024-04-26T00:36:34.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.424+0000 7fc51bf9d640 10 monclient: monmap: 2024-04-26T00:36:34.426 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:36:34.426 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:34.426 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:36:34.431691+0000 2024-04-26T00:36:34.427 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:36:34.431691+0000 2024-04-26T00:36:34.427 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:36:34.427 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:34.427 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:36:34.427 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:36:34.427 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:36:34.427 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:36:34.427 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:36:34.427 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:34.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc51bf9d640 1 Processor -- start 2024-04-26T00:36:34.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc51bf9d640 1 -- start start 2024-04-26T00:36:34.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc51bf9d640 10 monclient: init 2024-04-26T00:36:34.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc51bf9d640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:34.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc51bf9d640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-04-26T00:36:34.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc51bf9d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc514136da0 0x7fc51409bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:34.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc51bf9d640 10 monclient(hunting): picked mon.noname-a con 0x7fc514136da0 addr v2:172.21.15.37:3300/0 2024-04-26T00:36:34.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc51bf9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc51409c410 0x7fc514137180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:34.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc51bf9d640 10 monclient(hunting): picked mon.noname-c con 0x7fc51409c410 addr v2:172.21.15.45:3300/0 2024-04-26T00:36:34.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc51bf9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc5141376c0 0x7fc514133b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:34.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc51bf9d640 10 monclient(hunting): picked mon.noname-b con 0x7fc5141376c0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:36:34.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc51bf9d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:34.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc51bf9d640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fc51409b210 con 0x7fc514136da0 2024-04-26T00:36:34.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc51bf9d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:34.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc51bf9d640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc51409b4f0 con 0x7fc51409c410 2024-04-26T00:36:34.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc51bf9d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:34.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc51bf9d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc51409b7d0 con 0x7fc5141376c0 2024-04-26T00:36:34.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc51bf9d640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:34.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc51bf9d640 10 monclient(hunting): authenticate will time out at 1975.736964s 2024-04-26T00:36:34.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc519d39640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc514136da0 0x7fc51409bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:34.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc51a53a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc5141376c0 0x7fc514133b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:34.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc519d39640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc514136da0 0x7fc51409bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:34198/0 (socket says 172.21.15.37:34198) 2024-04-26T00:36:34.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc519d39640 1 -- 172.21.15.37:0/3074850072 learned_addr learned my addr 172.21.15.37:0/3074850072 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:36:34.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc519d39640 10 monclient(hunting): get_auth_request con 0x7fc514136da0 auth_method 0 2024-04-26T00:36:34.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc519d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:34.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc519d39640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:34.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc519d39640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:34.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc51a53a640 10 monclient(hunting): get_auth_request con 0x7fc5141376c0 auth_method 0 2024-04-26T00:36:34.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc51a53a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:34.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc51a53a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:34.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc51a53a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:34.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc51a53a640 1 -- 172.21.15.37:0/3074850072 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc5141376c0 msgr2=0x7fc514133b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:36:34.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc51a53a640 1 -- 172.21.15.37:0/3074850072 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc5141376c0 msgr2=0x7fc514133b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:36:34.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc51a53a640 1 --2- 172.21.15.37:0/3074850072 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc5141376c0 0x7fc514133b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:36:34.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc519d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:34.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc519d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:34.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc519d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:34.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc51a53a640 1 --2- 172.21.15.37:0/3074850072 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc5141376c0 0x7fc514133b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:36:34.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc519538640 1 --2- 172.21.15.37:0/3074850072 >> v2:172.21.15.45:3300/0 conn(0x7fc51409c410 0x7fc514137180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:34.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc519538640 10 monclient(hunting): get_auth_request con 0x7fc51409c410 auth_method 0 2024-04-26T00:36:34.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc519538640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:34.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc519538640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:34.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc519538640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:34.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc519d39640 10 monclient(hunting): handle_auth_done global_id 144675 payload 274 2024-04-26T00:36:34.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc519d39640 1 -- 172.21.15.37:0/3074850072 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc5141376c0 msgr2=0x7fc514133b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:36:34.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc519d39640 1 --2- 172.21.15.37:0/3074850072 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc5141376c0 0x7fc514133b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:34.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc519d39640 1 -- 172.21.15.37:0/3074850072 >> v2:172.21.15.45:3300/0 conn(0x7fc51409c410 msgr2=0x7fc514137180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:36:34.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc519d39640 1 --2- 172.21.15.37:0/3074850072 >> v2:172.21.15.45:3300/0 conn(0x7fc51409c410 0x7fc514137180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:34.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc519d39640 10 monclient: _finish_hunting 0 2024-04-26T00:36:34.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc519d39640 1 monclient: found mon.noname-a 2024-04-26T00:36:34.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc519d39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:34.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc519d39640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:36:34.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.428+0000 7fc519d39640 1 -- 172.21.15.37:0/3074850072 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc514137b00 con 0x7fc514136da0 2024-04-26T00:36:34.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.432+0000 7fc519d39640 1 -- 172.21.15.37:0/3074850072 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc514136da0 msgr2=0x7fc51409bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:34.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.432+0000 7fc519d39640 1 --2- 172.21.15.37:0/3074850072 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc514136da0 0x7fc51409bed0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fc5140a05c0 tx=0x7fc51001c120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de88a7a3414c9484 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:34.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.432+0000 7fc518d37640 1 -- 172.21.15.37:0/3074850072 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5100b0070 con 0x7fc514136da0 2024-04-26T00:36:34.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.432+0000 7fc518d37640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:34.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.432+0000 7fc518d37640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:36:34.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.432+0000 7fc518d37640 10 monclient: dump: 2024-04-26T00:36:34.438 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:34.438 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:34.438 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:34.438 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:34.438 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:34.438 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:34.438 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:34.439 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:34.439 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:34.439 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:34.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.432+0000 7fc518d37640 10 monclient: _finish_auth 0 2024-04-26T00:36:34.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.432+0000 7fc518d37640 10 monclient: _check_auth_tickets 2024-04-26T00:36:34.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.432+0000 7fc518d37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:34.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.432+0000 7fc51bf9d640 5 monclient: authenticate success, global_id 144675 2024-04-26T00:36:34.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.432+0000 7fc51bf9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:36:34.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.432+0000 7fc518d37640 1 -- 172.21.15.37:0/3074850072 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5100ab600 con 0x7fc514136da0 2024-04-26T00:36:34.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.432+0000 7fc518d37640 10 monclient: handle_config config(0 keys) 2024-04-26T00:36:34.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.432+0000 7fc518d37640 1 -- 172.21.15.37:0/3074850072 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5100ab9d0 con 0x7fc514136da0 2024-04-26T00:36:34.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.432+0000 7fc51bf9d640 10 monclient: get_monmap_and_config success 2024-04-26T00:36:34.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.432+0000 7fc518d37640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:34.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.432+0000 7fc51bf9d640 4 set_mon_vals no callback set 2024-04-26T00:36:34.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.432+0000 7fc518d37640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:36:34.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.432+0000 7fc518d37640 10 monclient: dump: 2024-04-26T00:36:34.440 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:34.440 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:34.440 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:34.440 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:34.440 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:34.440 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:34.440 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:34.440 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:34.440 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:34.440 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:34.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.432+0000 7fc51bf9d640 10 monclient: shutdown 2024-04-26T00:36:34.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.432+0000 7fc51bf9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:36:34.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.432+0000 7fc51bf9d640 1 -- 172.21.15.37:0/3074850072 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc514136da0 msgr2=0x7fc51409bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:34.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.432+0000 7fc51bf9d640 1 --2- 172.21.15.37:0/3074850072 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc514136da0 0x7fc51409bed0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fc5140a05c0 tx=0x7fc51001c120 comp rx=0 tx=0).stop 2024-04-26T00:36:34.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.432+0000 7fc51bf9d640 1 -- 172.21.15.37:0/3074850072 shutdown_connections 2024-04-26T00:36:34.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.432+0000 7fc51bf9d640 1 --2- 172.21.15.37:0/3074850072 >> v2:172.21.15.45:3300/0 conn(0x7fc51409c410 0x7fc514137180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:34.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.432+0000 7fc51bf9d640 1 --2- 172.21.15.37:0/3074850072 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc5141376c0 0x7fc514133b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:34.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.432+0000 7fc51bf9d640 1 --2- 172.21.15.37:0/3074850072 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc514136da0 0x7fc51409bed0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:34.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.432+0000 7fc51bf9d640 1 -- 172.21.15.37:0/3074850072 >> 172.21.15.37:0/3074850072 conn(0x7fc5140a5f00 msgr2=0x7fc5140a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:36:34.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.432+0000 7fc51bf9d640 1 -- 172.21.15.37:0/3074850072 shutdown_connections 2024-04-26T00:36:34.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.432+0000 7fc51bf9d640 1 -- 172.21.15.37:0/3074850072 wait complete. 2024-04-26T00:36:34.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.432+0000 7fc51bf9d640 10 monclient: build_initial_monmap 2024-04-26T00:36:34.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.432+0000 7fc51bf9d640 10 monclient: monmap: 2024-04-26T00:36:34.441 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:36:34.441 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:36:34.442 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:36:34.442 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:36:34.442 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:36:34.442 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:36:34.442 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:36:34.442 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:36:34.442 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:36:34.442 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:34.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.432+0000 7fc51bf9d640 1 Processor -- start 2024-04-26T00:36:34.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.432+0000 7fc51bf9d640 1 -- start start 2024-04-26T00:36:34.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.432+0000 7fc51bf9d640 10 monclient: init 2024-04-26T00:36:34.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.432+0000 7fc51bf9d640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:34.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.432+0000 7fc51bf9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:36:34.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.432+0000 7fc51bf9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc51409c410 0x7fc5141d0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:34.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.432+0000 7fc51bf9d640 10 monclient(hunting): picked mon.noname-a con 0x7fc51409c410 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:36:34.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.432+0000 7fc51bf9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc514136da0 0x7fc5141d0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:34.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc51bf9d640 10 monclient(hunting): picked mon.noname-c con 0x7fc514136da0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:36:34.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc51bf9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5141376c0 0x7fc51409d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:34.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc51bf9d640 10 monclient(hunting): picked mon.noname-b con 0x7fc5141376c0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:36:34.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc51bf9d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:34.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc51bf9d640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fc514140e90 con 0x7fc51409c410 2024-04-26T00:36:34.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc51bf9d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:34.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc51bf9d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc514140d10 con 0x7fc5141376c0 2024-04-26T00:36:34.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc51bf9d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:34.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc51bf9d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc514141010 con 0x7fc514136da0 2024-04-26T00:36:34.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc51bf9d640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:34.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc51bf9d640 10 monclient(hunting): authenticate will time out at 1975.744061s 2024-04-26T00:36:34.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc519538640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc514136da0 0x7fc5141d0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:34.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc519d39640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc51409c410 0x7fc5141d0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:34.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc519538640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc514136da0 0x7fc5141d0770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:39054/0 (socket says 172.21.15.37:39054) 2024-04-26T00:36:34.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc519538640 1 -- 172.21.15.37:0/3988539204 learned_addr learned my addr 172.21.15.37:0/3988539204 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:36:34.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc519538640 10 monclient(hunting): get_auth_request con 0x7fc514136da0 auth_method 0 2024-04-26T00:36:34.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc519538640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:34.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc519538640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:34.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc519538640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:34.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc519d39640 10 monclient(hunting): get_auth_request con 0x7fc51409c410 auth_method 0 2024-04-26T00:36:34.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc519d39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:34.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc519d39640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:34.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc519d39640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:34.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc519d39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:34.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc519d39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:34.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc519d39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:34.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc519538640 1 -- 172.21.15.37:0/3988539204 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc514136da0 msgr2=0x7fc5141d0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:36:34.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc519538640 1 -- 172.21.15.37:0/3988539204 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc514136da0 msgr2=0x7fc5141d0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:36:34.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc519538640 1 --2- 172.21.15.37:0/3988539204 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc514136da0 0x7fc5141d0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:36:34.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc51a53a640 1 --2- 172.21.15.37:0/3988539204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5141376c0 0x7fc51409d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:34.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc519538640 1 --2- 172.21.15.37:0/3988539204 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc514136da0 0x7fc5141d0770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:36:34.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc51a53a640 10 monclient(hunting): get_auth_request con 0x7fc5141376c0 auth_method 0 2024-04-26T00:36:34.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc51a53a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:34.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc51a53a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:34.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc51a53a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:34.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc519d39640 10 monclient(hunting): handle_auth_done global_id 144678 payload 824 2024-04-26T00:36:34.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc519d39640 1 -- 172.21.15.37:0/3988539204 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc514136da0 msgr2=0x7fc5141d0770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:36:34.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc519d39640 1 --2- 172.21.15.37:0/3988539204 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc514136da0 0x7fc5141d0770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:34.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc519d39640 1 -- 172.21.15.37:0/3988539204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5141376c0 msgr2=0x7fc51409d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:34.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc519d39640 1 --2- 172.21.15.37:0/3988539204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5141376c0 0x7fc51409d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:34.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc519d39640 10 monclient: _finish_hunting 0 2024-04-26T00:36:34.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc519d39640 1 monclient: found mon.noname-a 2024-04-26T00:36:34.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc519d39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:34.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc519d39640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:36:34.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc519d39640 1 -- 172.21.15.37:0/3988539204 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc514137e30 con 0x7fc51409c410 2024-04-26T00:36:34.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc519d39640 1 -- 172.21.15.37:0/3988539204 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc51409c410 msgr2=0x7fc5141d0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:34.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc519d39640 1 --2- 172.21.15.37:0/3988539204 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc51409c410 0x7fc5141d0230 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fc5100ab000 tx=0x7fc51001a960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:34.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc50a7fc640 1 -- 172.21.15.37:0/3988539204 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5100ba030 con 0x7fc51409c410 2024-04-26T00:36:34.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc50a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:34.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc50a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:36:34.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc50a7fc640 10 monclient: dump: 2024-04-26T00:36:34.446 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:34.446 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:34.446 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:34.447 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:34.447 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:34.447 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:34.447 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:34.447 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:34.447 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:34.447 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:34.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc50a7fc640 10 monclient: _finish_auth 0 2024-04-26T00:36:34.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc50a7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:36:34.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc50a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:34.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc50a7fc640 1 -- 172.21.15.37:0/3988539204 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5100abe30 con 0x7fc51409c410 2024-04-26T00:36:34.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc50a7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:36:34.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc50a7fc640 1 -- 172.21.15.37:0/3988539204 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5100b0040 con 0x7fc51409c410 2024-04-26T00:36:34.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc51bf9d640 5 monclient: authenticate success, global_id 144678 2024-04-26T00:36:34.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc50a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:34.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc50a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:36:34.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc50a7fc640 10 monclient: dump: 2024-04-26T00:36:34.448 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:34.448 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:34.448 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:34.448 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:34.448 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:34.448 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:34.448 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:34.449 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:34.449 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:34.449 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:34.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc50b7fe640 4 set_mon_vals no callback set 2024-04-26T00:36:34.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc51bf9d640 10 monclient: _renew_subs 2024-04-26T00:36:34.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc51bf9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:34.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc51bf9d640 1 -- 172.21.15.37:0/3988539204 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5140a8d70 con 0x7fc51409c410 2024-04-26T00:36:34.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc51bf9d640 10 monclient: _renew_subs 2024-04-26T00:36:34.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc51bf9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:34.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.436+0000 7fc51bf9d640 1 -- 172.21.15.37:0/3988539204 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5141d0cb0 con 0x7fc51409c410 2024-04-26T00:36:34.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.440+0000 7fc51bf9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:34.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.440+0000 7fc51bf9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:34.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.440+0000 7fc51bf9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:34.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.440+0000 7fc51bf9d640 1 -- 172.21.15.37:0/3988539204 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc514066f40 con 0x7fc51409c410 2024-04-26T00:36:34.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.440+0000 7fc50a7fc640 1 -- 172.21.15.37:0/3988539204 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc5100bd070 con 0x7fc51409c410 2024-04-26T00:36:34.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.440+0000 7fc50a7fc640 1 --2- 172.21.15.37:0/3988539204 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc4f8035e80 0x7fc4f8038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:34.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.440+0000 7fc50a7fc640 1 -- 172.21.15.37:0/3988539204 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(649..649 src has 370..649) ==== 3971+0+0 (secure 0 0 0) 0x7fc5100e3660 con 0x7fc51409c410 2024-04-26T00:36:34.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.440+0000 7fc519538640 1 --2- 172.21.15.37:0/3988539204 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc4f8035e80 0x7fc4f8038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:34.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.440+0000 7fc519538640 10 monclient: get_auth_request con 0x7fc4f8035e80 auth_method 0 2024-04-26T00:36:34.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.444+0000 7fc519538640 1 --2- 172.21.15.37:0/3988539204 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc4f8035e80 0x7fc4f8038340 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fc51413df80 tx=0x7fc504008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:34.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.444+0000 7fc50a7fc640 1 -- 172.21.15.37:0/3988539204 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc5100160f0 con 0x7fc51409c410 2024-04-26T00:36:34.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.444+0000 7fc50a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:34.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.444+0000 7fc50a7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:36:34.566 INFO:tasks.rados.rados.0.smithi045.stdout:2857: finishing copy_from racing read to smithi04526579-25 2024-04-26T00:36:34.566 INFO:tasks.rados.rados.0.smithi045.stdout:2857: finishing copy_from to smithi04526579-25 2024-04-26T00:36:34.566 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 2003 (ObjNum 945 snap 292 seq_num 945) dirty exists 2024-04-26T00:36:34.579 INFO:tasks.rados.rados.0.smithi045.stdout:2846: done (10 left) 2024-04-26T00:36:34.579 INFO:tasks.rados.rados.0.smithi045.stdout:2847: done (9 left) 2024-04-26T00:36:34.579 INFO:tasks.rados.rados.0.smithi045.stdout:2848: done (8 left) 2024-04-26T00:36:34.579 INFO:tasks.rados.rados.0.smithi045.stdout:2849: done (7 left) 2024-04-26T00:36:34.579 INFO:tasks.rados.rados.0.smithi045.stdout:2851: done (6 left) 2024-04-26T00:36:34.579 INFO:tasks.rados.rados.0.smithi045.stdout:2852: done (5 left) 2024-04-26T00:36:34.579 INFO:tasks.rados.rados.0.smithi045.stdout:2854: done (4 left) 2024-04-26T00:36:34.579 INFO:tasks.rados.rados.0.smithi045.stdout:2855: done (3 left) 2024-04-26T00:36:34.580 INFO:tasks.rados.rados.0.smithi045.stdout:2856: done (2 left) 2024-04-26T00:36:34.580 INFO:tasks.rados.rados.0.smithi045.stdout:2857: done (1 left) 2024-04-26T00:36:34.580 INFO:tasks.rados.rados.0.smithi045.stdout:2859: done (0 left) 2024-04-26T00:36:34.580 INFO:tasks.rados.rados.0.smithi045.stdout:2860: read oid 49 snap -1 2024-04-26T00:36:34.580 INFO:tasks.rados.rados.0.smithi045.stdout:2860: expect (ObjNum 1029 snap 310 seq_num 1029) 2024-04-26T00:36:34.580 INFO:tasks.rados.rados.0.smithi045.stdout:2861: write (excl) oid 47 current snap is 311 2024-04-26T00:36:34.580 INFO:tasks.rados.rados.0.smithi045.stdout:2861: seq_num 1030 ranges {511210=414125,1423159=486356,2668859=589585} 2024-04-26T00:36:34.583 INFO:tasks.rados.rados.0.smithi045.stdout:2861: writing smithi04526579-47 from 511210 to 925335 tid 1 2024-04-26T00:36:34.588 INFO:tasks.rados.rados.0.smithi045.stdout:2861: writing smithi04526579-47 from 1423159 to 1909515 tid 2 2024-04-26T00:36:34.597 INFO:tasks.rados.rados.0.smithi045.stdout:2861: writing smithi04526579-47 from 2668859 to 3258444 tid 3 2024-04-26T00:36:34.597 INFO:tasks.rados.rados.0.smithi045.stdout:2862: write oid 43 current snap is 311 2024-04-26T00:36:34.597 INFO:tasks.rados.rados.0.smithi045.stdout:2862: seq_num 1031 ranges {528017=489374,1481641=630014,2696282=400857} 2024-04-26T00:36:34.601 INFO:tasks.rados.rados.0.smithi045.stdout:2862: writing smithi04526579-43 from 528017 to 1017391 tid 1 2024-04-26T00:36:34.608 INFO:tasks.rados.rados.0.smithi045.stdout:2862: writing smithi04526579-43 from 1481641 to 2111655 tid 2 2024-04-26T00:36:34.614 INFO:tasks.rados.rados.0.smithi045.stdout:2862: writing smithi04526579-43 from 2696282 to 3097139 tid 3 2024-04-26T00:36:34.614 INFO:tasks.rados.rados.0.smithi045.stdout:2863: snap_remove snap 310 2024-04-26T00:36:34.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.676+0000 7fc51bf9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:36:34.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.676+0000 7fc51bf9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:36:34.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.676+0000 7fc51bf9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:34.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.676+0000 7fc51bf9d640 1 -- 172.21.15.37:0/3988539204 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc514136680 con 0x7fc51409c410 2024-04-26T00:36:34.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.676+0000 7fc50a7fc640 1 -- 172.21.15.37:0/3988539204 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7fc5100e13d0 con 0x7fc51409c410 2024-04-26T00:36:34.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.676+0000 7fc50a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:36:34.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.676+0000 7fc50a7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:36:34.677 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":124,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:36:34.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.680+0000 7fc51bf9d640 1 -- 172.21.15.37:0/3988539204 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc4f8035e80 msgr2=0x7fc4f8038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:34.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.680+0000 7fc51bf9d640 1 --2- 172.21.15.37:0/3988539204 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc4f8035e80 0x7fc4f8038340 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fc51413df80 tx=0x7fc504008040 comp rx=0 tx=0).stop 2024-04-26T00:36:34.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.680+0000 7fc51bf9d640 10 monclient: shutdown 2024-04-26T00:36:34.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.680+0000 7fc51bf9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:36:34.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.680+0000 7fc51bf9d640 1 -- 172.21.15.37:0/3988539204 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc51409c410 msgr2=0x7fc5141d0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:34.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.680+0000 7fc51bf9d640 1 --2- 172.21.15.37:0/3988539204 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc51409c410 0x7fc5141d0230 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fc5100ab000 tx=0x7fc51001a960 comp rx=0 tx=0).stop 2024-04-26T00:36:34.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.680+0000 7fc51bf9d640 1 -- 172.21.15.37:0/3988539204 shutdown_connections 2024-04-26T00:36:34.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.680+0000 7fc51bf9d640 1 --2- 172.21.15.37:0/3988539204 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc4f8035e80 0x7fc4f8038340 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:34.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.680+0000 7fc51bf9d640 1 --2- 172.21.15.37:0/3988539204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5141376c0 0x7fc51409d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:34.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.680+0000 7fc51bf9d640 1 --2- 172.21.15.37:0/3988539204 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc514136da0 0x7fc5141d0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:34.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.680+0000 7fc51bf9d640 1 --2- 172.21.15.37:0/3988539204 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc51409c410 0x7fc5141d0230 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:34.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.680+0000 7fc51bf9d640 1 -- 172.21.15.37:0/3988539204 >> 172.21.15.37:0/3988539204 conn(0x7fc5140a5f00 msgr2=0x7fc514130f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:36:34.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.680+0000 7fc51bf9d640 1 -- 172.21.15.37:0/3988539204 shutdown_connections 2024-04-26T00:36:34.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:34.680+0000 7fc51bf9d640 1 -- 172.21.15.37:0/3988539204 wait complete. 2024-04-26T00:36:35.587 INFO:tasks.rados.rados.0.smithi045.stdout:2863: done (3 left) 2024-04-26T00:36:35.588 INFO:tasks.rados.rados.0.smithi045.stdout:2864: read oid 50 snap -1 2024-04-26T00:36:35.588 INFO:tasks.rados.rados.0.smithi045.stdout:2864: expect (ObjNum 1022 snap 310 seq_num 1022) 2024-04-26T00:36:35.588 INFO:tasks.rados.rados.0.smithi045.stdout:2865: write oid 14 current snap is 311 2024-04-26T00:36:35.588 INFO:tasks.rados.rados.0.smithi045.stdout:2865: seq_num 1032 ranges {544824=564623,1823770=490025,3007352=337472} 2024-04-26T00:36:35.598 INFO:tasks.rados.rados.0.smithi045.stdout:2865: writing smithi04526579-14 from 544824 to 1109447 tid 1 2024-04-26T00:36:35.609 INFO:tasks.rados.rados.0.smithi045.stdout:2865: writing smithi04526579-14 from 1823770 to 2313795 tid 2 2024-04-26T00:36:35.617 INFO:tasks.rados.rados.0.smithi045.stdout:2865: writing smithi04526579-14 from 3007352 to 3344824 tid 3 2024-04-26T00:36:35.617 INFO:tasks.rados.rados.0.smithi045.stdout:2866: rollback oid 10 current snap is 311 2024-04-26T00:36:35.617 INFO:tasks.rados.rados.0.smithi045.stdout:2866: done (5 left) 2024-04-26T00:36:35.617 INFO:tasks.rados.rados.0.smithi045.stdout:2867: snap_create 2024-04-26T00:36:35.617 INFO:tasks.rados.rados.0.smithi045.stdout:2860: expect (ObjNum 1029 snap 310 seq_num 1029) 2024-04-26T00:36:35.844 INFO:tasks.rados.rados.0.smithi045.stdout:2861: finishing write tid 1 to smithi04526579-47 2024-04-26T00:36:35.844 INFO:tasks.rados.rados.0.smithi045.stdout:2861: finishing write tid 2 to smithi04526579-47 2024-04-26T00:36:35.844 INFO:tasks.rados.rados.0.smithi045.stdout:2861: finishing write tid 3 to smithi04526579-47 2024-04-26T00:36:35.844 INFO:tasks.rados.rados.0.smithi045.stdout:2861: finishing write tid 4 to smithi04526579-47 2024-04-26T00:36:35.844 INFO:tasks.rados.rados.0.smithi045.stdout:2861: finishing write tid 5 to smithi04526579-47 2024-04-26T00:36:35.845 INFO:tasks.rados.rados.0.smithi045.stdout:2861: oid 47 updating version 0 to 2033 2024-04-26T00:36:35.845 INFO:tasks.rados.rados.0.smithi045.stdout:2861: oid 47 updating version 2033 to 2035 2024-04-26T00:36:35.845 INFO:tasks.rados.rados.0.smithi045.stdout:2861: oid 47 version 2035 is already newer than 2034 2024-04-26T00:36:35.845 INFO:tasks.rados.rados.0.smithi045.stdout:2861: oid 47 updating version 2035 to 2036 2024-04-26T00:36:35.845 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 2036 (ObjNum 1030 snap 311 seq_num 1030) dirty exists 2024-04-26T00:36:35.845 INFO:tasks.rados.rados.0.smithi045.stdout:2861: left oid 47 (ObjNum 1030 snap 311 seq_num 1030) 2024-04-26T00:36:35.845 INFO:tasks.rados.rados.0.smithi045.stdout:2862: finishing write tid 1 to smithi04526579-43 2024-04-26T00:36:35.845 INFO:tasks.rados.rados.0.smithi045.stdout:2862: finishing write tid 2 to smithi04526579-43 2024-04-26T00:36:35.845 INFO:tasks.rados.rados.0.smithi045.stdout:2862: finishing write tid 3 to smithi04526579-43 2024-04-26T00:36:35.845 INFO:tasks.rados.rados.0.smithi045.stdout:2862: finishing write tid 4 to smithi04526579-43 2024-04-26T00:36:35.845 INFO:tasks.rados.rados.0.smithi045.stdout:2862: finishing write tid 5 to smithi04526579-43 2024-04-26T00:36:35.845 INFO:tasks.rados.rados.0.smithi045.stdout:2862: oid 43 updating version 0 to 1980 2024-04-26T00:36:35.845 INFO:tasks.rados.rados.0.smithi045.stdout:2862: oid 43 updating version 1980 to 1982 2024-04-26T00:36:35.845 INFO:tasks.rados.rados.0.smithi045.stdout:2862: oid 43 updating version 1982 to 1983 2024-04-26T00:36:35.846 INFO:tasks.rados.rados.0.smithi045.stdout:2862: oid 43 version 1983 is already newer than 1981 2024-04-26T00:36:35.846 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 1983 (ObjNum 1031 snap 311 seq_num 1031) dirty exists 2024-04-26T00:36:35.846 INFO:tasks.rados.rados.0.smithi045.stdout:2862: left oid 43 (ObjNum 1031 snap 311 seq_num 1031) 2024-04-26T00:36:35.846 INFO:tasks.rados.rados.0.smithi045.stdout:2864: expect (ObjNum 1022 snap 310 seq_num 1022) 2024-04-26T00:36:35.950 INFO:tasks.rados.rados.0.smithi045.stdout:2865: finishing write tid 1 to smithi04526579-14 2024-04-26T00:36:35.950 INFO:tasks.rados.rados.0.smithi045.stdout:2865: finishing write tid 2 to smithi04526579-14 2024-04-26T00:36:35.950 INFO:tasks.rados.rados.0.smithi045.stdout:2865: finishing write tid 3 to smithi04526579-14 2024-04-26T00:36:35.951 INFO:tasks.rados.rados.0.smithi045.stdout:2865: finishing write tid 4 to smithi04526579-14 2024-04-26T00:36:35.951 INFO:tasks.rados.rados.0.smithi045.stdout:2865: finishing write tid 5 to smithi04526579-14 2024-04-26T00:36:35.951 INFO:tasks.rados.rados.0.smithi045.stdout:2865: oid 14 updating version 0 to 1183 2024-04-26T00:36:35.951 INFO:tasks.rados.rados.0.smithi045.stdout:2865: oid 14 version 1183 is already newer than 1182 2024-04-26T00:36:35.951 INFO:tasks.rados.rados.0.smithi045.stdout:2865: oid 14 updating version 1183 to 1185 2024-04-26T00:36:35.951 INFO:tasks.rados.rados.0.smithi045.stdout:2865: oid 14 version 1185 is already newer than 1184 2024-04-26T00:36:35.951 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 1185 (ObjNum 1032 snap 311 seq_num 1032) dirty exists 2024-04-26T00:36:35.951 INFO:tasks.rados.rados.0.smithi045.stdout:2865: left oid 14 (ObjNum 1032 snap 311 seq_num 1032) 2024-04-26T00:36:36.597 INFO:tasks.rados.rados.0.smithi045.stdout:2860: done (5 left) 2024-04-26T00:36:36.597 INFO:tasks.rados.rados.0.smithi045.stdout:2861: done (4 left) 2024-04-26T00:36:36.597 INFO:tasks.rados.rados.0.smithi045.stdout:2862: done (3 left) 2024-04-26T00:36:36.598 INFO:tasks.rados.rados.0.smithi045.stdout:2864: done (2 left) 2024-04-26T00:36:36.598 INFO:tasks.rados.rados.0.smithi045.stdout:2865: done (1 left) 2024-04-26T00:36:36.598 INFO:tasks.rados.rados.0.smithi045.stdout:2867: done (0 left) 2024-04-26T00:36:36.598 INFO:tasks.rados.rados.0.smithi045.stdout:2868: snap_remove snap 311 2024-04-26T00:36:37.603 INFO:tasks.rados.rados.0.smithi045.stdout:2868: done (0 left) 2024-04-26T00:36:37.604 INFO:tasks.rados.rados.0.smithi045.stdout:2869: write (excl) oid 17 current snap is 312 2024-04-26T00:36:37.604 INFO:tasks.rados.rados.0.smithi045.stdout:2869: seq_num 1033 ranges {561631=639872,1882252=633683,3034775=326856} 2024-04-26T00:36:37.617 INFO:tasks.rados.rados.0.smithi045.stdout:2869: writing smithi04526579-17 from 561631 to 1201503 tid 1 2024-04-26T00:36:37.633 INFO:tasks.rados.rados.0.smithi045.stdout:2869: writing smithi04526579-17 from 1882252 to 2515935 tid 2 2024-04-26T00:36:37.644 INFO:tasks.rados.rados.0.smithi045.stdout:2869: writing smithi04526579-17 from 3034775 to 3361631 tid 3 2024-04-26T00:36:37.644 INFO:tasks.rados.rados.0.smithi045.stdout:2870: read oid 49 snap -1 2024-04-26T00:36:37.644 INFO:tasks.rados.rados.0.smithi045.stdout:2870: expect (ObjNum 1029 snap 310 seq_num 1029) 2024-04-26T00:36:37.644 INFO:tasks.rados.rados.0.smithi045.stdout:2871: read oid 38 snap -1 2024-04-26T00:36:37.644 INFO:tasks.rados.rados.0.smithi045.stdout:2871: expect (ObjNum 1001 snap 304 seq_num 1001) 2024-04-26T00:36:37.644 INFO:tasks.rados.rados.0.smithi045.stdout:2872: read oid 22 snap -1 2024-04-26T00:36:37.644 INFO:tasks.rados.rados.0.smithi045.stdout:2872: expect (ObjNum 942 snap 291 seq_num 942) 2024-04-26T00:36:37.644 INFO:tasks.rados.rados.0.smithi045.stdout:2873: write (excl) oid 36 current snap is 312 2024-04-26T00:36:37.644 INFO:tasks.rados.rados.0.smithi045.stdout:2869: finishing write tid 1 to smithi04526579-17 2024-04-26T00:36:37.644 INFO:tasks.rados.rados.0.smithi045.stdout:2869: finishing write tid 2 to smithi04526579-17 2024-04-26T00:36:37.644 INFO:tasks.rados.rados.0.smithi045.stdout:2873: seq_num 1034 ranges {578438=431474,1657087=493694,2778551=599887} 2024-04-26T00:36:37.650 INFO:tasks.rados.rados.0.smithi045.stdout:2873: writing smithi04526579-36 from 578438 to 1009912 tid 1 2024-04-26T00:36:37.657 INFO:tasks.rados.rados.0.smithi045.stdout:2873: writing smithi04526579-36 from 1657087 to 2150781 tid 2 2024-04-26T00:36:37.665 INFO:tasks.rados.rados.0.smithi045.stdout:2873: writing smithi04526579-36 from 2778551 to 3378438 tid 3 2024-04-26T00:36:37.665 INFO:tasks.rados.rados.0.smithi045.stdout:2874: rollback oid 10 current snap is 312 2024-04-26T00:36:37.665 INFO:tasks.rados.rados.0.smithi045.stdout:2874: done (5 left) 2024-04-26T00:36:37.665 INFO:tasks.rados.rados.0.smithi045.stdout:2875: snap_create 2024-04-26T00:36:37.665 INFO:tasks.rados.rados.0.smithi045.stdout:2871: expect (ObjNum 1001 snap 304 seq_num 1001) 2024-04-26T00:36:37.698 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:36:37.759 INFO:tasks.rados.rados.0.smithi045.stdout:2869: finishing write tid 3 to smithi04526579-17 2024-04-26T00:36:37.759 INFO:tasks.rados.rados.0.smithi045.stdout:2869: finishing write tid 4 to smithi04526579-17 2024-04-26T00:36:37.759 INFO:tasks.rados.rados.0.smithi045.stdout:2869: finishing write tid 5 to smithi04526579-17 2024-04-26T00:36:37.759 INFO:tasks.rados.rados.0.smithi045.stdout:2869: oid 17 updating version 0 to 1984 2024-04-26T00:36:37.760 INFO:tasks.rados.rados.0.smithi045.stdout:2869: oid 17 updating version 1984 to 1987 2024-04-26T00:36:37.760 INFO:tasks.rados.rados.0.smithi045.stdout:2869: oid 17 version 1987 is already newer than 1986 2024-04-26T00:36:37.760 INFO:tasks.rados.rados.0.smithi045.stdout:2869: oid 17 version 1987 is already newer than 1985 2024-04-26T00:36:37.760 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 1987 (ObjNum 1033 snap 312 seq_num 1033) dirty exists 2024-04-26T00:36:37.760 INFO:tasks.rados.rados.0.smithi045.stdout:2869: left oid 17 (ObjNum 1033 snap 312 seq_num 1033) 2024-04-26T00:36:37.760 INFO:tasks.rados.rados.0.smithi045.stdout:2872: expect (ObjNum 942 snap 291 seq_num 942) 2024-04-26T00:36:37.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f731adc2640 10 monclient: get_monmap_and_config 2024-04-26T00:36:37.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f731adc2640 10 monclient: build_initial_monmap 2024-04-26T00:36:37.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f731adc2640 10 monclient: monmap: 2024-04-26T00:36:37.828 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:36:37.828 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:37.828 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:36:37.833625+0000 2024-04-26T00:36:37.829 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:36:37.833625+0000 2024-04-26T00:36:37.829 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:36:37.829 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:37.829 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:36:37.829 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:36:37.829 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:36:37.829 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:36:37.829 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:36:37.829 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:37.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f731adc2640 1 Processor -- start 2024-04-26T00:36:37.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f731adc2640 1 -- start start 2024-04-26T00:36:37.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f731adc2640 10 monclient: init 2024-04-26T00:36:37.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f731adc2640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:37.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f731adc2640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-04-26T00:36:37.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f731adc2640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7314071fc0 0x7f7314137240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:37.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f731adc2640 10 monclient(hunting): picked mon.noname-c con 0x7f7314071fc0 addr v2:172.21.15.45:3300/0 2024-04-26T00:36:37.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f731adc2640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f7314137780 0x7f73140723a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:37.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f731adc2640 10 monclient(hunting): picked mon.noname-a con 0x7f7314137780 addr v2:172.21.15.37:3300/0 2024-04-26T00:36:37.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f731adc2640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73140728e0 0x7f7314130770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:37.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f731adc2640 10 monclient(hunting): picked mon.noname-b con 0x7f73140728e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:36:37.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f731adc2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:37.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f731adc2640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f7314073200 con 0x7f7314137780 2024-04-26T00:36:37.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f731adc2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:37.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f731adc2640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f73140a0c20 con 0x7f7314071fc0 2024-04-26T00:36:37.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f731adc2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:37.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f731adc2640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f73140a0f00 con 0x7f73140728e0 2024-04-26T00:36:37.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f731adc2640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:37.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f731adc2640 10 monclient(hunting): authenticate will time out at 1979.139367s 2024-04-26T00:36:37.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f731935f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73140728e0 0x7f7314130770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:37.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f7313fff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f7314137780 0x7f73140723a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:37.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f731935f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73140728e0 0x7f7314130770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:39066/0 (socket says 172.21.15.37:39066) 2024-04-26T00:36:37.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f731935f640 1 -- 172.21.15.37:0/2932055748 learned_addr learned my addr 172.21.15.37:0/2932055748 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:36:37.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f731935f640 10 monclient(hunting): get_auth_request con 0x7f73140728e0 auth_method 0 2024-04-26T00:36:37.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f731935f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:37.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f731935f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:37.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f731935f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:37.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f7313fff640 1 --2- 172.21.15.37:0/2932055748 >> v2:172.21.15.37:3300/0 conn(0x7f7314137780 0x7f73140723a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:34258/0 (socket says 172.21.15.37:34258) 2024-04-26T00:36:37.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f7313fff640 10 monclient(hunting): get_auth_request con 0x7f7314137780 auth_method 0 2024-04-26T00:36:37.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f7313fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:37.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f7313fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:37.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f7313fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:37.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f7313fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:37.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f731935f640 1 -- 172.21.15.37:0/2932055748 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73140728e0 msgr2=0x7f7314130770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:36:37.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f7313fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:37.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f731935f640 1 -- 172.21.15.37:0/2932055748 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73140728e0 msgr2=0x7f7314130770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:36:37.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f7318b5e640 1 --2- 172.21.15.37:0/2932055748 >> v2:172.21.15.45:3300/0 conn(0x7f7314071fc0 0x7f7314137240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:37.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f731935f640 1 --2- 172.21.15.37:0/2932055748 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73140728e0 0x7f7314130770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:36:37.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f7313fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:37.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f7318b5e640 10 monclient(hunting): get_auth_request con 0x7f7314071fc0 auth_method 0 2024-04-26T00:36:37.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f7318b5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:37.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f7318b5e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:37.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f7318b5e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:37.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f731935f640 1 --2- 172.21.15.37:0/2932055748 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73140728e0 0x7f7314130770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:36:37.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f7313fff640 10 monclient(hunting): handle_auth_done global_id 144681 payload 274 2024-04-26T00:36:37.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f7313fff640 1 -- 172.21.15.37:0/2932055748 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73140728e0 msgr2=0x7f7314130770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:36:37.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f7313fff640 1 --2- 172.21.15.37:0/2932055748 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73140728e0 0x7f7314130770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:37.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f7313fff640 1 -- 172.21.15.37:0/2932055748 >> v2:172.21.15.45:3300/0 conn(0x7f7314071fc0 msgr2=0x7f7314137240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:36:37.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f7313fff640 1 --2- 172.21.15.37:0/2932055748 >> v2:172.21.15.45:3300/0 conn(0x7f7314071fc0 0x7f7314137240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:37.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f7313fff640 10 monclient: _finish_hunting 0 2024-04-26T00:36:37.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f7313fff640 1 monclient: found mon.noname-a 2024-04-26T00:36:37.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.828+0000 7f7313fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:37.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.832+0000 7f7313fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:36:37.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.832+0000 7f7313fff640 1 -- 172.21.15.37:0/2932055748 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7314072d70 con 0x7f7314137780 2024-04-26T00:36:37.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f7313fff640 1 -- 172.21.15.37:0/2932055748 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7314137780 msgr2=0x7f73140723a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:37.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f7313fff640 1 --2- 172.21.15.37:0/2932055748 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7314137780 0x7f73140723a0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7314072fa0 tx=0x7f730801c060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2421851a18cd6fe0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:37.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f73137fe640 1 -- 172.21.15.37:0/2932055748 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73080b1070 con 0x7f7314137780 2024-04-26T00:36:37.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f73137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:37.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f73137fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:36:37.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f73137fe640 10 monclient: dump: 2024-04-26T00:36:37.843 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:37.843 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:37.843 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:37.843 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:37.843 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:37.843 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:37.843 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:37.843 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:37.844 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:37.844 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:37.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f73137fe640 10 monclient: _finish_auth 0 2024-04-26T00:36:37.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f73137fe640 10 monclient: _check_auth_tickets 2024-04-26T00:36:37.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f73137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:37.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731adc2640 5 monclient: authenticate success, global_id 144681 2024-04-26T00:36:37.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731adc2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:36:37.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f73137fe640 1 -- 172.21.15.37:0/2932055748 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73080a4dd0 con 0x7f7314137780 2024-04-26T00:36:37.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f73137fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:36:37.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731adc2640 10 monclient: get_monmap_and_config success 2024-04-26T00:36:37.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731adc2640 4 set_mon_vals no callback set 2024-04-26T00:36:37.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f73137fe640 1 -- 172.21.15.37:0/2932055748 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73080ac660 con 0x7f7314137780 2024-04-26T00:36:37.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f73137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:37.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f73137fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:36:37.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f73137fe640 10 monclient: dump: 2024-04-26T00:36:37.845 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:37.845 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:37.845 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:37.845 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:37.845 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:37.846 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:37.846 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:37.846 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:37.846 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:37.846 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:37.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731adc2640 10 monclient: shutdown 2024-04-26T00:36:37.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731adc2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:36:37.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731adc2640 1 -- 172.21.15.37:0/2932055748 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7314137780 msgr2=0x7f73140723a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:37.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731adc2640 1 --2- 172.21.15.37:0/2932055748 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7314137780 0x7f73140723a0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7314072fa0 tx=0x7f730801c060 comp rx=0 tx=0).stop 2024-04-26T00:36:37.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731adc2640 1 -- 172.21.15.37:0/2932055748 shutdown_connections 2024-04-26T00:36:37.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731adc2640 1 --2- 172.21.15.37:0/2932055748 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73140728e0 0x7f7314130770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:37.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731adc2640 1 --2- 172.21.15.37:0/2932055748 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7314137780 0x7f73140723a0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:37.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731adc2640 1 --2- 172.21.15.37:0/2932055748 >> v2:172.21.15.45:3300/0 conn(0x7f7314071fc0 0x7f7314137240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:37.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731adc2640 1 -- 172.21.15.37:0/2932055748 >> 172.21.15.37:0/2932055748 conn(0x7f73140a1ad0 msgr2=0x7f73140a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:36:37.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731adc2640 1 -- 172.21.15.37:0/2932055748 shutdown_connections 2024-04-26T00:36:37.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731adc2640 1 -- 172.21.15.37:0/2932055748 wait complete. 2024-04-26T00:36:37.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731adc2640 10 monclient: build_initial_monmap 2024-04-26T00:36:37.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731adc2640 10 monclient: monmap: 2024-04-26T00:36:37.848 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:36:37.848 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:36:37.848 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:36:37.848 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:36:37.848 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:36:37.848 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:36:37.848 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:36:37.848 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:36:37.848 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:36:37.848 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:37.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731adc2640 1 Processor -- start 2024-04-26T00:36:37.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731adc2640 1 -- start start 2024-04-26T00:36:37.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731adc2640 10 monclient: init 2024-04-26T00:36:37.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731adc2640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:37.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731adc2640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:36:37.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731adc2640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7314071fc0 0x7f731409e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:37.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731adc2640 10 monclient(hunting): picked mon.noname-a con 0x7f7314071fc0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:36:37.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731adc2640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73140728e0 0x7f7314138790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:37.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731adc2640 10 monclient(hunting): picked mon.noname-c con 0x7f73140728e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:36:37.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731adc2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7314137780 0x7f7314138cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:37.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731adc2640 10 monclient(hunting): picked mon.noname-b con 0x7f7314137780 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:36:37.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731adc2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:37.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731adc2640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f7314140f00 con 0x7f7314071fc0 2024-04-26T00:36:37.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731adc2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:37.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731adc2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7314140d80 con 0x7f7314137780 2024-04-26T00:36:37.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731adc2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:37.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731adc2640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f7314141080 con 0x7f73140728e0 2024-04-26T00:36:37.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731adc2640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:37.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731adc2640 10 monclient(hunting): authenticate will time out at 1979.146668s 2024-04-26T00:36:37.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f7313fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73140728e0 0x7f7314138790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:37.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f7313fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73140728e0 0x7f7314138790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:39070/0 (socket says 172.21.15.37:39070) 2024-04-26T00:36:37.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f7313fff640 1 -- 172.21.15.37:0/1396140339 learned_addr learned my addr 172.21.15.37:0/1396140339 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:36:37.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f7313fff640 10 monclient(hunting): get_auth_request con 0x7f73140728e0 auth_method 0 2024-04-26T00:36:37.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f7313fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:37.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f7313fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:37.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f7313fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:37.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f7318b5e640 1 --2- 172.21.15.37:0/1396140339 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7314071fc0 0x7f731409e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:37.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f7318b5e640 10 monclient(hunting): get_auth_request con 0x7f7314071fc0 auth_method 0 2024-04-26T00:36:37.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f7318b5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:37.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f7318b5e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:37.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f7318b5e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:37.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f7313fff640 1 -- 172.21.15.37:0/1396140339 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73140728e0 msgr2=0x7f7314138790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:36:37.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f7313fff640 1 -- 172.21.15.37:0/1396140339 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73140728e0 msgr2=0x7f7314138790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:36:37.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f7313fff640 1 --2- 172.21.15.37:0/1396140339 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73140728e0 0x7f7314138790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:36:37.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f7313fff640 1 --2- 172.21.15.37:0/1396140339 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73140728e0 0x7f7314138790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:36:37.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f7318b5e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:37.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f7318b5e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:37.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f7318b5e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:37.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731935f640 1 --2- 172.21.15.37:0/1396140339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7314137780 0x7f7314138cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:37.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731935f640 10 monclient(hunting): get_auth_request con 0x7f7314137780 auth_method 0 2024-04-26T00:36:37.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731935f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:37.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731935f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:37.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731935f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:37.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f7318b5e640 10 monclient(hunting): handle_auth_done global_id 144684 payload 824 2024-04-26T00:36:37.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f7318b5e640 1 -- 172.21.15.37:0/1396140339 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73140728e0 msgr2=0x7f7314138790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:36:37.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f7318b5e640 1 --2- 172.21.15.37:0/1396140339 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73140728e0 0x7f7314138790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:37.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f7318b5e640 1 -- 172.21.15.37:0/1396140339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7314137780 msgr2=0x7f7314138cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:37.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f7318b5e640 1 --2- 172.21.15.37:0/1396140339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7314137780 0x7f7314138cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:37.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f7318b5e640 10 monclient: _finish_hunting 0 2024-04-26T00:36:37.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f7318b5e640 1 monclient: found mon.noname-a 2024-04-26T00:36:37.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f7318b5e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:37.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f7318b5e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:36:37.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f7318b5e640 1 -- 172.21.15.37:0/1396140339 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7314139520 con 0x7f7314071fc0 2024-04-26T00:36:37.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.836+0000 7f731935f640 1 --2- 172.21.15.37:0/1396140339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7314137780 0x7f7314138cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:36:37.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.840+0000 7f7318b5e640 1 -- 172.21.15.37:0/1396140339 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7314071fc0 msgr2=0x7f731409e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:37.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.840+0000 7f7318b5e640 1 --2- 172.21.15.37:0/1396140339 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7314071fc0 0x7f731409e530 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f730400d5f0 tx=0x7f730400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:37.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.840+0000 7f73117fa640 1 -- 172.21.15.37:0/1396140339 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7304012800 con 0x7f7314071fc0 2024-04-26T00:36:37.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.840+0000 7f73117fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:37.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.840+0000 7f73117fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:36:37.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.840+0000 7f73117fa640 10 monclient: dump: 2024-04-26T00:36:37.853 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:37.853 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:37.853 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:37.853 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:37.853 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:37.853 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:37.853 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:37.853 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:37.854 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:37.854 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:37.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.840+0000 7f73117fa640 10 monclient: _finish_auth 0 2024-04-26T00:36:37.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.840+0000 7f73117fa640 10 monclient: _check_auth_tickets 2024-04-26T00:36:37.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.840+0000 7f73117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:37.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.840+0000 7f73117fa640 1 -- 172.21.15.37:0/1396140339 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7304019070 con 0x7f7314071fc0 2024-04-26T00:36:37.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.840+0000 7f73117fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:36:37.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.840+0000 7f73117fa640 1 -- 172.21.15.37:0/1396140339 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7304012bc0 con 0x7f7314071fc0 2024-04-26T00:36:37.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.840+0000 7f73117fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:37.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.840+0000 7f73117fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:36:37.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.840+0000 7f73117fa640 10 monclient: dump: 2024-04-26T00:36:37.854 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:37.854 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:37.855 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:37.855 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:37.855 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:37.855 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:37.855 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:37.855 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:37.855 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:37.855 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:37.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.840+0000 7f731adc2640 5 monclient: authenticate success, global_id 144684 2024-04-26T00:36:37.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.840+0000 7f731adc2640 10 monclient: _renew_subs 2024-04-26T00:36:37.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.840+0000 7f731adc2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:37.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.840+0000 7f73127fc640 4 set_mon_vals no callback set 2024-04-26T00:36:37.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.840+0000 7f731adc2640 1 -- 172.21.15.37:0/1396140339 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73140a0160 con 0x7f7314071fc0 2024-04-26T00:36:37.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.840+0000 7f731adc2640 10 monclient: _renew_subs 2024-04-26T00:36:37.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.840+0000 7f731adc2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:37.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.840+0000 7f731adc2640 1 -- 172.21.15.37:0/1396140339 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f731409f120 con 0x7f7314071fc0 2024-04-26T00:36:37.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.840+0000 7f731adc2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:37.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.840+0000 7f731adc2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:37.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.840+0000 7f731adc2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:37.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.840+0000 7f731adc2640 1 -- 172.21.15.37:0/1396140339 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7314066f40 con 0x7f7314071fc0 2024-04-26T00:36:37.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.840+0000 7f73117fa640 1 -- 172.21.15.37:0/1396140339 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f730401e460 con 0x7f7314071fc0 2024-04-26T00:36:37.856 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.840+0000 7f73117fa640 1 --2- 172.21.15.37:0/1396140339 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7300035f00 0x7f73000383c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:37.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.840+0000 7f73117fa640 1 -- 172.21.15.37:0/1396140339 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(653..653 src has 370..653) ==== 3927+0+0 (secure 0 0 0) 0x7f7304048f20 con 0x7f7314071fc0 2024-04-26T00:36:37.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.840+0000 7f7313fff640 1 --2- 172.21.15.37:0/1396140339 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7300035f00 0x7f73000383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:37.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.840+0000 7f7313fff640 10 monclient: get_auth_request con 0x7f7300035f00 auth_method 0 2024-04-26T00:36:37.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.844+0000 7f7313fff640 1 --2- 172.21.15.37:0/1396140339 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7300035f00 0x7f73000383c0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f7314139cb0 tx=0x7f73080a6070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:37.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.844+0000 7f73117fa640 1 -- 172.21.15.37:0/1396140339 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f730401d580 con 0x7f7314071fc0 2024-04-26T00:36:37.857 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.844+0000 7f73117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:37.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:37.844+0000 7f73117fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:36:38.001 INFO:tasks.rados.rados.0.smithi045.stdout:2870: expect (ObjNum 1029 snap 310 seq_num 1029) 2024-04-26T00:36:38.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:38.068+0000 7f731adc2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:36:38.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:38.068+0000 7f731adc2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:36:38.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:38.068+0000 7f731adc2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:38.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:38.068+0000 7f731adc2640 1 -- 172.21.15.37:0/1396140339 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f731413db60 con 0x7f7314071fc0 2024-04-26T00:36:38.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:38.072+0000 7f73117fa640 1 -- 172.21.15.37:0/1396140339 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7f730401d7a0 con 0x7f7314071fc0 2024-04-26T00:36:38.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:38.072+0000 7f73117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:36:38.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:38.072+0000 7f73117fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:36:38.071 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":124,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:36:38.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:38.072+0000 7f731adc2640 1 -- 172.21.15.37:0/1396140339 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7300035f00 msgr2=0x7f73000383c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:38.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:38.072+0000 7f731adc2640 1 --2- 172.21.15.37:0/1396140339 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7300035f00 0x7f73000383c0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f7314139cb0 tx=0x7f73080a6070 comp rx=0 tx=0).stop 2024-04-26T00:36:38.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:38.072+0000 7f731adc2640 10 monclient: shutdown 2024-04-26T00:36:38.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:38.072+0000 7f731adc2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:36:38.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:38.072+0000 7f731adc2640 1 -- 172.21.15.37:0/1396140339 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7314071fc0 msgr2=0x7f731409e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:38.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:38.072+0000 7f731adc2640 1 --2- 172.21.15.37:0/1396140339 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7314071fc0 0x7f731409e530 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f730400d5f0 tx=0x7f730400dab0 comp rx=0 tx=0).stop 2024-04-26T00:36:38.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:38.072+0000 7f731adc2640 1 -- 172.21.15.37:0/1396140339 shutdown_connections 2024-04-26T00:36:38.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:38.072+0000 7f731adc2640 1 --2- 172.21.15.37:0/1396140339 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7300035f00 0x7f73000383c0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:38.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:38.072+0000 7f731adc2640 1 --2- 172.21.15.37:0/1396140339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7314137780 0x7f7314138cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:38.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:38.072+0000 7f731adc2640 1 --2- 172.21.15.37:0/1396140339 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73140728e0 0x7f7314138790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:38.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:38.072+0000 7f731adc2640 1 --2- 172.21.15.37:0/1396140339 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7314071fc0 0x7f731409e530 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:38.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:38.072+0000 7f731adc2640 1 -- 172.21.15.37:0/1396140339 >> 172.21.15.37:0/1396140339 conn(0x7f73140a1ad0 msgr2=0x7f73140a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:36:38.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:38.072+0000 7f731adc2640 1 -- 172.21.15.37:0/1396140339 shutdown_connections 2024-04-26T00:36:38.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:38.072+0000 7f731adc2640 1 -- 172.21.15.37:0/1396140339 wait complete. 2024-04-26T00:36:38.195 INFO:tasks.rados.rados.0.smithi045.stdout:2873: finishing write tid 1 to smithi04526579-36 2024-04-26T00:36:38.195 INFO:tasks.rados.rados.0.smithi045.stdout:2873: finishing write tid 2 to smithi04526579-36 2024-04-26T00:36:38.195 INFO:tasks.rados.rados.0.smithi045.stdout:2873: finishing write tid 3 to smithi04526579-36 2024-04-26T00:36:38.196 INFO:tasks.rados.rados.0.smithi045.stdout:2873: finishing write tid 4 to smithi04526579-36 2024-04-26T00:36:38.196 INFO:tasks.rados.rados.0.smithi045.stdout:2873: finishing write tid 5 to smithi04526579-36 2024-04-26T00:36:38.196 INFO:tasks.rados.rados.0.smithi045.stdout:2873: oid 36 updating version 0 to 2371 2024-04-26T00:36:38.196 INFO:tasks.rados.rados.0.smithi045.stdout:2873: oid 36 version 2371 is already newer than 2368 2024-04-26T00:36:38.196 INFO:tasks.rados.rados.0.smithi045.stdout:2873: oid 36 version 2371 is already newer than 2369 2024-04-26T00:36:38.196 INFO:tasks.rados.rados.0.smithi045.stdout:2873: oid 36 version 2371 is already newer than 2370 2024-04-26T00:36:38.196 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 2371 (ObjNum 1034 snap 312 seq_num 1034) dirty exists 2024-04-26T00:36:38.196 INFO:tasks.rados.rados.0.smithi045.stdout:2873: left oid 36 (ObjNum 1034 snap 312 seq_num 1034) 2024-04-26T00:36:38.557 INFO:tasks.rados.rados.0.smithi045.stdout:2869: done (5 left) 2024-04-26T00:36:38.557 INFO:tasks.rados.rados.0.smithi045.stdout:2870: done (4 left) 2024-04-26T00:36:38.557 INFO:tasks.rados.rados.0.smithi045.stdout:2871: done (3 left) 2024-04-26T00:36:38.557 INFO:tasks.rados.rados.0.smithi045.stdout:2872: done (2 left) 2024-04-26T00:36:38.558 INFO:tasks.rados.rados.0.smithi045.stdout:2873: done (1 left) 2024-04-26T00:36:38.558 INFO:tasks.rados.rados.0.smithi045.stdout:2875: done (0 left) 2024-04-26T00:36:38.558 INFO:tasks.rados.rados.0.smithi045.stdout:2876: snap_remove snap 312 2024-04-26T00:36:39.556 INFO:tasks.rados.rados.0.smithi045.stdout:2876: done (0 left) 2024-04-26T00:36:39.557 INFO:tasks.rados.rados.0.smithi045.stdout:2877: copy_from oid 4 from oid 17 current snap is 313 2024-04-26T00:36:39.557 INFO:tasks.rados.rados.0.smithi045.stdout:2878: read oid 5 snap -1 2024-04-26T00:36:39.557 INFO:tasks.rados.rados.0.smithi045.stdout:2878: expect (ObjNum 955 snap 295 seq_num 955) 2024-04-26T00:36:39.557 INFO:tasks.rados.rados.0.smithi045.stdout:2879: delete oid 25 current snap is 313 2024-04-26T00:36:39.576 INFO:tasks.rados.rados.0.smithi045.stdout:2878: expect (ObjNum 955 snap 295 seq_num 955) 2024-04-26T00:36:39.688 INFO:tasks.rados.rados.0.smithi045.stdout:2877: finishing copy_from racing read to smithi04526579-4 2024-04-26T00:36:39.689 INFO:tasks.rados.rados.0.smithi045.stdout:2877: finishing copy_from to smithi04526579-4 2024-04-26T00:36:39.689 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 1987 (ObjNum 1033 snap 312 seq_num 1033) dirty exists 2024-04-26T00:36:39.689 INFO:tasks.rados.rados.0.smithi045.stdout:2877: done (2 left) 2024-04-26T00:36:39.689 INFO:tasks.rados.rados.0.smithi045.stdout:2878: done (1 left) 2024-04-26T00:36:39.689 INFO:tasks.rados.rados.0.smithi045.stdout:2879: done (0 left) 2024-04-26T00:36:39.689 INFO:tasks.rados.rados.0.smithi045.stdout:2880: rollback oid 44 current snap is 313 2024-04-26T00:36:39.689 INFO:tasks.rados.rados.0.smithi045.stdout:2880: done (0 left) 2024-04-26T00:36:39.689 INFO:tasks.rados.rados.0.smithi045.stdout:2881: read oid 28 snap -1 2024-04-26T00:36:39.689 INFO:tasks.rados.rados.0.smithi045.stdout:2881: expect (ObjNum 936 snap 290 seq_num 936) 2024-04-26T00:36:39.689 INFO:tasks.rados.rados.0.smithi045.stdout:2882: copy_from oid 49 from oid 50 current snap is 313 2024-04-26T00:36:39.689 INFO:tasks.rados.rados.0.smithi045.stdout:2883: delete oid 15 current snap is 313 2024-04-26T00:36:39.701 INFO:tasks.rados.rados.0.smithi045.stdout:2881: expect (ObjNum 936 snap 290 seq_num 936) 2024-04-26T00:36:39.951 INFO:tasks.rados.rados.0.smithi045.stdout:2882: finishing copy_from racing read to smithi04526579-49 2024-04-26T00:36:39.951 INFO:tasks.rados.rados.0.smithi045.stdout:2882: finishing copy_from to smithi04526579-49 2024-04-26T00:36:39.951 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 1975 (ObjNum 1022 snap 310 seq_num 1022) dirty exists 2024-04-26T00:36:39.951 INFO:tasks.rados.rados.0.smithi045.stdout:2881: done (2 left) 2024-04-26T00:36:39.951 INFO:tasks.rados.rados.0.smithi045.stdout:2882: done (1 left) 2024-04-26T00:36:39.951 INFO:tasks.rados.rados.0.smithi045.stdout:2883: done (0 left) 2024-04-26T00:36:39.951 INFO:tasks.rados.rados.0.smithi045.stdout:2884: write (excl) oid 41 current snap is 313 2024-04-26T00:36:39.951 INFO:tasks.rados.rados.0.smithi045.stdout:2884: seq_num 1037 ranges {628859=657221,1716180=641021,3144467=284392} 2024-04-26T00:36:39.971 INFO:tasks.rados.rados.0.smithi045.stdout:2884: writing smithi04526579-41 from 628859 to 1286080 tid 1 2024-04-26T00:36:39.989 INFO:tasks.rados.rados.0.smithi045.stdout:2884: writing smithi04526579-41 from 1716180 to 2357201 tid 2 2024-04-26T00:36:40.007 INFO:tasks.rados.rados.0.smithi045.stdout:2884: writing smithi04526579-41 from 3144467 to 3428859 tid 3 2024-04-26T00:36:40.008 INFO:tasks.rados.rados.0.smithi045.stdout:2885: read oid 38 snap -1 2024-04-26T00:36:40.008 INFO:tasks.rados.rados.0.smithi045.stdout:2885: expect (ObjNum 1001 snap 304 seq_num 1001) 2024-04-26T00:36:40.008 INFO:tasks.rados.rados.0.smithi045.stdout:2886: rollback oid 27 current snap is 313 2024-04-26T00:36:40.008 INFO:tasks.rados.rados.0.smithi045.stdout:2886: done (2 left) 2024-04-26T00:36:40.008 INFO:tasks.rados.rados.0.smithi045.stdout:2887: delete oid 28 current snap is 313 2024-04-26T00:36:40.008 INFO:tasks.rados.rados.0.smithi045.stdout:2884: finishing write tid 1 to smithi04526579-41 2024-04-26T00:36:40.008 INFO:tasks.rados.rados.0.smithi045.stdout:2884: finishing write tid 2 to smithi04526579-41 2024-04-26T00:36:40.014 INFO:tasks.rados.rados.0.smithi045.stdout:2884: finishing write tid 3 to smithi04526579-41 2024-04-26T00:36:40.014 INFO:tasks.rados.rados.0.smithi045.stdout:2884: finishing write tid 4 to smithi04526579-41 2024-04-26T00:36:40.015 INFO:tasks.rados.rados.0.smithi045.stdout:2884: finishing write tid 5 to smithi04526579-41 2024-04-26T00:36:40.015 INFO:tasks.rados.rados.0.smithi045.stdout:2884: oid 41 updating version 0 to 2014 2024-04-26T00:36:40.015 INFO:tasks.rados.rados.0.smithi045.stdout:2884: oid 41 updating version 2014 to 2016 2024-04-26T00:36:40.015 INFO:tasks.rados.rados.0.smithi045.stdout:2884: oid 41 version 2016 is already newer than 2015 2024-04-26T00:36:40.015 INFO:tasks.rados.rados.0.smithi045.stdout:2884: oid 41 updating version 2016 to 2017 2024-04-26T00:36:40.015 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 2017 (ObjNum 1037 snap 313 seq_num 1037) dirty exists 2024-04-26T00:36:40.015 INFO:tasks.rados.rados.0.smithi045.stdout:2884: left oid 41 (ObjNum 1037 snap 313 seq_num 1037) 2024-04-26T00:36:40.015 INFO:tasks.rados.rados.0.smithi045.stdout:2885: expect (ObjNum 1001 snap 304 seq_num 1001) 2024-04-26T00:36:40.151 INFO:tasks.rados.rados.0.smithi045.stdout:2884: done (2 left) 2024-04-26T00:36:40.151 INFO:tasks.rados.rados.0.smithi045.stdout:2885: done (1 left) 2024-04-26T00:36:40.151 INFO:tasks.rados.rados.0.smithi045.stdout:2887: done (0 left) 2024-04-26T00:36:40.151 INFO:tasks.rados.rados.0.smithi045.stdout:2888: write (excl) oid 3 current snap is 313 2024-04-26T00:36:40.151 INFO:tasks.rados.rados.0.smithi045.stdout:2888: seq_num 1039 ranges {662473=407719,1833144=644690,3199313=263160} 2024-04-26T00:36:40.167 INFO:tasks.rados.rados.0.smithi045.stdout:2888: writing smithi04526579-3 from 662473 to 1070192 tid 1 2024-04-26T00:36:40.188 INFO:tasks.rados.rados.0.smithi045.stdout:2888: writing smithi04526579-3 from 1833144 to 2477834 tid 2 2024-04-26T00:36:40.203 INFO:tasks.rados.rados.0.smithi045.stdout:2888: writing smithi04526579-3 from 3199313 to 3462473 tid 3 2024-04-26T00:36:40.203 INFO:tasks.rados.rados.0.smithi045.stdout:2889: rollback oid 2 current snap is 313 2024-04-26T00:36:40.203 INFO:tasks.rados.rados.0.smithi045.stdout:2889: done (1 left) 2024-04-26T00:36:40.203 INFO:tasks.rados.rados.0.smithi045.stdout:2890: read oid 40 snap -1 2024-04-26T00:36:40.203 INFO:tasks.rados.rados.0.smithi045.stdout:2890: expect (ObjNum 1023 snap 310 seq_num 1023) 2024-04-26T00:36:40.204 INFO:tasks.rados.rados.0.smithi045.stdout:2891: read oid 37 snap -1 2024-04-26T00:36:40.204 INFO:tasks.rados.rados.0.smithi045.stdout:2891: expect (ObjNum 939 snap 290 seq_num 939) 2024-04-26T00:36:40.204 INFO:tasks.rados.rados.0.smithi045.stdout:2892: write (excl) oid 36 current snap is 313 2024-04-26T00:36:40.204 INFO:tasks.rados.rados.0.smithi045.stdout:2892: seq_num 1040 ranges {679280=482968,1775273=788348,3110383=368897} 2024-04-26T00:36:40.217 INFO:tasks.rados.rados.0.smithi045.stdout:2892: writing smithi04526579-36 from 679280 to 1162248 tid 1 2024-04-26T00:36:40.231 INFO:tasks.rados.rados.0.smithi045.stdout:2892: writing smithi04526579-36 from 1775273 to 2563621 tid 2 2024-04-26T00:36:40.239 INFO:tasks.rados.rados.0.smithi045.stdout:2892: writing smithi04526579-36 from 3110383 to 3479280 tid 3 2024-04-26T00:36:40.239 INFO:tasks.rados.rados.0.smithi045.stdout:2893: snap_create 2024-04-26T00:36:40.239 INFO:tasks.rados.rados.0.smithi045.stdout:2888: finishing write tid 1 to smithi04526579-3 2024-04-26T00:36:40.239 INFO:tasks.rados.rados.0.smithi045.stdout:2888: finishing write tid 2 to smithi04526579-3 2024-04-26T00:36:40.239 INFO:tasks.rados.rados.0.smithi045.stdout:2888: finishing write tid 3 to smithi04526579-3 2024-04-26T00:36:40.239 INFO:tasks.rados.rados.0.smithi045.stdout:2888: finishing write tid 4 to smithi04526579-3 2024-04-26T00:36:40.239 INFO:tasks.rados.rados.0.smithi045.stdout:2888: finishing write tid 5 to smithi04526579-3 2024-04-26T00:36:40.239 INFO:tasks.rados.rados.0.smithi045.stdout:2888: oid 3 updating version 0 to 2039 2024-04-26T00:36:40.239 INFO:tasks.rados.rados.0.smithi045.stdout:2888: oid 3 version 2039 is already newer than 2037 2024-04-26T00:36:40.239 INFO:tasks.rados.rados.0.smithi045.stdout:2888: oid 3 updating version 2039 to 2040 2024-04-26T00:36:40.240 INFO:tasks.rados.rados.0.smithi045.stdout:2888: oid 3 version 2040 is already newer than 2038 2024-04-26T00:36:40.240 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 2040 (ObjNum 1039 snap 313 seq_num 1039) dirty exists 2024-04-26T00:36:40.240 INFO:tasks.rados.rados.0.smithi045.stdout:2888: left oid 3 (ObjNum 1039 snap 313 seq_num 1039) 2024-04-26T00:36:40.240 INFO:tasks.rados.rados.0.smithi045.stdout:2891: expect (ObjNum 939 snap 290 seq_num 939) 2024-04-26T00:36:40.505 INFO:tasks.rados.rados.0.smithi045.stdout:2890: expect (ObjNum 1023 snap 310 seq_num 1023) 2024-04-26T00:36:40.750 INFO:tasks.rados.rados.0.smithi045.stdout:2892: finishing write tid 1 to smithi04526579-36 2024-04-26T00:36:40.751 INFO:tasks.rados.rados.0.smithi045.stdout:2892: finishing write tid 2 to smithi04526579-36 2024-04-26T00:36:40.751 INFO:tasks.rados.rados.0.smithi045.stdout:2892: finishing write tid 3 to smithi04526579-36 2024-04-26T00:36:40.751 INFO:tasks.rados.rados.0.smithi045.stdout:2892: finishing write tid 4 to smithi04526579-36 2024-04-26T00:36:40.751 INFO:tasks.rados.rados.0.smithi045.stdout:2892: finishing write tid 5 to smithi04526579-36 2024-04-26T00:36:40.751 INFO:tasks.rados.rados.0.smithi045.stdout:2892: oid 36 updating version 0 to 2373 2024-04-26T00:36:40.751 INFO:tasks.rados.rados.0.smithi045.stdout:2892: oid 36 updating version 2373 to 2374 2024-04-26T00:36:40.751 INFO:tasks.rados.rados.0.smithi045.stdout:2892: oid 36 updating version 2374 to 2375 2024-04-26T00:36:40.751 INFO:tasks.rados.rados.0.smithi045.stdout:2892: oid 36 version 2375 is already newer than 2372 2024-04-26T00:36:40.751 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 2375 (ObjNum 1040 snap 313 seq_num 1040) dirty exists 2024-04-26T00:36:40.751 INFO:tasks.rados.rados.0.smithi045.stdout:2892: left oid 36 (ObjNum 1040 snap 313 seq_num 1040) 2024-04-26T00:36:41.072 INFO:tasks.rados.rados.0.smithi045.stdout:2888: done (4 left) 2024-04-26T00:36:41.072 INFO:tasks.rados.rados.0.smithi045.stdout:2890: done (3 left) 2024-04-26T00:36:41.072 INFO:tasks.rados.rados.0.smithi045.stdout:2891: done (2 left) 2024-04-26T00:36:41.072 INFO:tasks.rados.rados.0.smithi045.stdout:2892: done (1 left) 2024-04-26T00:36:41.072 INFO:tasks.rados.rados.0.smithi045.stdout:2893: done (0 left) 2024-04-26T00:36:41.073 INFO:tasks.rados.rados.0.smithi045.stdout:2894: read oid 6 snap -1 2024-04-26T00:36:41.073 INFO:tasks.rados.rados.0.smithi045.stdout:2894: expect (ObjNum 1024 snap 310 seq_num 1024) 2024-04-26T00:36:41.073 INFO:tasks.rados.rados.0.smithi045.stdout:2895: rollback oid 42 current snap is 314 2024-04-26T00:36:41.073 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 42 to 313 2024-04-26T00:36:41.073 INFO:tasks.rados.rados.0.smithi045.stdout:2896: snap_create 2024-04-26T00:36:41.086 INFO:tasks.rados.rados.0.smithi045.stdout:2894: expect (ObjNum 1024 snap 310 seq_num 1024) 2024-04-26T00:36:41.092 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:36:41.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.224+0000 7fa7d5cda640 10 monclient: get_monmap_and_config 2024-04-26T00:36:41.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.224+0000 7fa7d5cda640 10 monclient: build_initial_monmap 2024-04-26T00:36:41.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.224+0000 7fa7d5cda640 10 monclient: monmap: 2024-04-26T00:36:41.226 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:36:41.226 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:41.226 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:36:41.230828+0000 2024-04-26T00:36:41.226 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:36:41.230828+0000 2024-04-26T00:36:41.226 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:36:41.226 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:41.226 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:36:41.226 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:36:41.226 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:36:41.226 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:36:41.226 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:36:41.227 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:41.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.224+0000 7fa7d5cda640 1 Processor -- start 2024-04-26T00:36:41.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.224+0000 7fa7d5cda640 1 -- start start 2024-04-26T00:36:41.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.224+0000 7fa7d5cda640 10 monclient: init 2024-04-26T00:36:41.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7d5cda640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:41.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7d5cda640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-04-26T00:36:41.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7d5cda640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa7d00a0260 addr v1:172.21.15.37:6789/0 2024-04-26T00:36:41.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7d5cda640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa7d0137e40 0x7fa7d013a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:41.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7d5cda640 10 monclient(hunting): picked mon.noname-c con 0x7fa7d0137e40 addr v2:172.21.15.45:3300/0 2024-04-26T00:36:41.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7d5cda640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa7d013a770 addr v1:172.21.15.45:6789/0 2024-04-26T00:36:41.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7d5cda640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7d00a1130 con 0x7fa7d00a0260 2024-04-26T00:36:41.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7d5cda640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7d006cff0 con 0x7fa7d013a770 2024-04-26T00:36:41.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7d5cda640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:41.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7d5cda640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fa7d00a0d50 con 0x7fa7d0137e40 2024-04-26T00:36:41.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7d5cda640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:41.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7d5cda640 10 monclient(hunting): authenticate will time out at 1982.536749s 2024-04-26T00:36:41.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7cf7fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fa7d00a0260 0x7fa7d00a0640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:33982/0 (socket says 172.21.15.37:33982) 2024-04-26T00:36:41.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7cf7fe640 1 -- 172.21.15.37:0/1968667205 learned_addr learned my addr 172.21.15.37:0/1968667205 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:36:41.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ceffd640 1 --2- 172.21.15.37:0/1968667205 >> v2:172.21.15.45:3300/0 conn(0x7fa7d0137e40 0x7fa7d013a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:41.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ceffd640 10 monclient(hunting): get_auth_request con 0x7fa7d0137e40 auth_method 0 2024-04-26T00:36:41.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:41.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ceffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:41.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:41.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:41.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:41.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:41.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7cf7fe640 1 -- 172.21.15.37:0/1968667205 >> v1:172.21.15.37:6789/0 conn(0x7fa7d00a0260 legacy=0x7fa7d00a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:41.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ce7fc640 1 -- 172.21.15.37:0/1968667205 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2371532833 0 0) 0x7fa7b8001460 con 0x7fa7d00a0260 2024-04-26T00:36:41.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ce7fc640 10 monclient(hunting): _negotiate 2024-04-26T00:36:41.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ce7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:41.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:41.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ce7fc640 10 monclient(hunting): my global_id is 144687 2024-04-26T00:36:41.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ce7fc640 1 -- 172.21.15.37:0/1968667205 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7bc003530 con 0x7fa7d00a0260 2024-04-26T00:36:41.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7cffff640 1 -- 172.21.15.37:0/1968667205 >> v1:172.21.15.45:6789/0 conn(0x7fa7d013a770 legacy=0x7fa7d013cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:41.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ce7fc640 1 -- 172.21.15.37:0/1968667205 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1148740636 0 0) 0x7fa7bc003530 con 0x7fa7d00a0260 2024-04-26T00:36:41.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ce7fc640 1 -- 172.21.15.37:0/1968667205 >> v2:172.21.15.45:3300/0 conn(0x7fa7d0137e40 msgr2=0x7fa7d013a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:36:41.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ce7fc640 1 --2- 172.21.15.37:0/1968667205 >> v2:172.21.15.45:3300/0 conn(0x7fa7d0137e40 0x7fa7d013a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:41.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ce7fc640 1 -- 172.21.15.37:0/1968667205 >> v1:172.21.15.45:6789/0 conn(0x7fa7d013a770 legacy=0x7fa7d013cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:41.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ce7fc640 10 monclient: _finish_hunting 0 2024-04-26T00:36:41.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ce7fc640 1 monclient: found mon.noname-a-legacy 2024-04-26T00:36:41.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:41.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ce7fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.37:6789/0 2024-04-26T00:36:41.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ce7fc640 1 -- 172.21.15.37:0/1968667205 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7d009bc50 con 0x7fa7d00a0260 2024-04-26T00:36:41.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ce7fc640 10 monclient: _finish_auth 0 2024-04-26T00:36:41.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ce7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:36:41.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:41.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ce7fc640 1 -- 172.21.15.37:0/1968667205 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 866326386 0 0) 0x7fa7b8001b00 con 0x7fa7d00a0260 2024-04-26T00:36:41.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7d5cda640 5 monclient: authenticate success, global_id 144687 2024-04-26T00:36:41.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7d5cda640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:36:41.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:41.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ce7fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-26T00:36:41.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ce7fc640 10 monclient: dump: 2024-04-26T00:36:41.235 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:41.235 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:41.235 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:41.236 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:41.236 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:41.236 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:41.236 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:41.236 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:41.236 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:41.236 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:41.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ce7fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] but i'm connected to v1:172.21.15.37:6789/0, reconnecting 2024-04-26T00:36:41.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ce7fc640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:41.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ce7fc640 1 -- 172.21.15.37:0/1968667205 >> v1:172.21.15.37:6789/0 conn(0x7fa7d00a0260 legacy=0x7fa7d00a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:41.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ce7fc640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:36:41.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ce7fc640 1 --2- 172.21.15.37:0/1968667205 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7bc003a90 0x7fa7bc003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:41.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ce7fc640 10 monclient(hunting): picked mon.c con 0x7fa7bc003a90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:36:41.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ce7fc640 1 --2- 172.21.15.37:0/1968667205 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7bc004810 0x7fa7bc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:41.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ce7fc640 10 monclient(hunting): picked mon.a con 0x7fa7bc004810 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:36:41.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ce7fc640 1 --2- 172.21.15.37:0/1968667205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7bc0053d0 0x7fa7bc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:41.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ce7fc640 10 monclient(hunting): picked mon.b con 0x7fa7bc0053d0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:36:41.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ce7fc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:41.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ce7fc640 1 -- 172.21.15.37:0/1968667205 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d00a0d50 con 0x7fa7bc004810 2024-04-26T00:36:41.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ce7fc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:41.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ce7fc640 1 -- 172.21.15.37:0/1968667205 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa7bc00c270 con 0x7fa7bc0053d0 2024-04-26T00:36:41.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ce7fc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:41.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ce7fc640 1 -- 172.21.15.37:0/1968667205 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa7bc00c470 con 0x7fa7bc003a90 2024-04-26T00:36:41.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ce7fc640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:41.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7ce7fc640 10 monclient(hunting): _finish_auth 0 2024-04-26T00:36:41.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.228+0000 7fa7d5cda640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-26T00:36:41.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.232+0000 7fa7cf7fe640 1 --2- 172.21.15.37:0/1968667205 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7bc003a90 0x7fa7bc003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:41.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.232+0000 7fa7ceffd640 1 --2- 172.21.15.37:0/1968667205 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7bc004810 0x7fa7bc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:41.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.232+0000 7fa7cf7fe640 10 monclient(hunting): get_auth_request con 0x7fa7bc003a90 auth_method 0 2024-04-26T00:36:41.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.232+0000 7fa7cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:41.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.232+0000 7fa7cf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:41.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.232+0000 7fa7cf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:36:41.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.232+0000 7fa7ceffd640 10 monclient(hunting): get_auth_request con 0x7fa7bc004810 auth_method 0 2024-04-26T00:36:41.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.232+0000 7fa7ceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:41.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.232+0000 7fa7ceffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:41.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.232+0000 7fa7ceffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:36:41.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.232+0000 7fa7ceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:41.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.232+0000 7fa7ceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:41.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.232+0000 7fa7ceffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:36:41.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.232+0000 7fa7cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:41.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.232+0000 7fa7cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:41.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.232+0000 7fa7cffff640 1 --2- 172.21.15.37:0/1968667205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7bc0053d0 0x7fa7bc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:41.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.232+0000 7fa7cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:36:41.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.232+0000 7fa7cffff640 10 monclient(hunting): get_auth_request con 0x7fa7bc0053d0 auth_method 0 2024-04-26T00:36:41.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.232+0000 7fa7cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:41.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.232+0000 7fa7cffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:41.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.232+0000 7fa7cffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:36:41.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.232+0000 7fa7ceffd640 10 monclient(hunting): handle_auth_done global_id 144687 payload 293 2024-04-26T00:36:41.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.232+0000 7fa7ceffd640 1 -- 172.21.15.37:0/1968667205 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7bc003a90 msgr2=0x7fa7bc003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:36:41.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.232+0000 7fa7ceffd640 1 --2- 172.21.15.37:0/1968667205 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7bc003a90 0x7fa7bc003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:41.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.232+0000 7fa7ceffd640 1 -- 172.21.15.37:0/1968667205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7bc0053d0 msgr2=0x7fa7bc00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:36:41.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.232+0000 7fa7ceffd640 1 --2- 172.21.15.37:0/1968667205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7bc0053d0 0x7fa7bc00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:41.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.232+0000 7fa7ceffd640 10 monclient: _finish_hunting 0 2024-04-26T00:36:41.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.232+0000 7fa7ceffd640 1 monclient: found mon.a 2024-04-26T00:36:41.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.232+0000 7fa7ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:41.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.232+0000 7fa7ceffd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:41.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.232+0000 7fa7ceffd640 1 -- 172.21.15.37:0/1968667205 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7bc003530 con 0x7fa7bc004810 2024-04-26T00:36:41.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.232+0000 7fa7ceffd640 10 monclient: _finish_auth 0 2024-04-26T00:36:41.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.232+0000 7fa7ceffd640 10 monclient: _check_auth_tickets 2024-04-26T00:36:41.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.232+0000 7fa7ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:41.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.232+0000 7fa7cffff640 1 -- 172.21.15.37:0/1968667205 reap_dead start 2024-04-26T00:36:41.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7ceffd640 1 -- 172.21.15.37:0/1968667205 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7bc004810 msgr2=0x7fa7bc004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:41.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7ceffd640 1 --2- 172.21.15.37:0/1968667205 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7bc004810 0x7fa7bc004c90 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa7c0009fc0 tx=0x7fa7c001cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f40d23ef453c764 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:41.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7ce7fc640 1 -- 172.21.15.37:0/1968667205 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c00b2070 con 0x7fa7bc004810 2024-04-26T00:36:41.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:41.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7ce7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:36:41.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7ce7fc640 10 monclient: dump: 2024-04-26T00:36:41.246 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:41.246 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:41.246 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:41.246 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:41.247 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:41.247 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:41.247 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:41.247 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:41.247 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:41.247 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:41.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7ce7fc640 1 -- 172.21.15.37:0/1968667205 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7c00a5a50 con 0x7fa7bc004810 2024-04-26T00:36:41.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7ce7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:36:41.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7ce7fc640 1 -- 172.21.15.37:0/1968667205 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c0009c10 con 0x7fa7bc004810 2024-04-26T00:36:41.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7d5cda640 10 monclient: get_monmap_and_config success 2024-04-26T00:36:41.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7d5cda640 4 set_mon_vals no callback set 2024-04-26T00:36:41.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:41.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7ce7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:36:41.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7ce7fc640 10 monclient: dump: 2024-04-26T00:36:41.248 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:41.248 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:41.248 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:41.248 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:41.248 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:41.248 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:41.248 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:41.248 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:41.248 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:41.248 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:41.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7d5cda640 10 monclient: shutdown 2024-04-26T00:36:41.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7d5cda640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:36:41.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7d5cda640 1 -- 172.21.15.37:0/1968667205 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7bc004810 msgr2=0x7fa7bc004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:41.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7d5cda640 1 --2- 172.21.15.37:0/1968667205 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7bc004810 0x7fa7bc004c90 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa7c0009fc0 tx=0x7fa7c001cc60 comp rx=0 tx=0).stop 2024-04-26T00:36:41.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7d5cda640 1 -- 172.21.15.37:0/1968667205 shutdown_connections 2024-04-26T00:36:41.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7d5cda640 1 --2- 172.21.15.37:0/1968667205 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7bc004810 0x7fa7bc004c90 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:41.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7d5cda640 1 -- 172.21.15.37:0/1968667205 >> 172.21.15.37:0/1968667205 conn(0x7fa7d00a1ad0 msgr2=0x7fa7d00a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:36:41.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7d5cda640 1 -- 172.21.15.37:0/1968667205 shutdown_connections 2024-04-26T00:36:41.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7d5cda640 1 -- 172.21.15.37:0/1968667205 wait complete. 2024-04-26T00:36:41.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7d5cda640 10 monclient: build_initial_monmap 2024-04-26T00:36:41.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7d5cda640 10 monclient: monmap: 2024-04-26T00:36:41.249 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:36:41.249 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:36:41.249 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:36:41.249 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:36:41.249 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:36:41.249 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:36:41.249 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:36:41.249 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:36:41.250 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:36:41.250 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:41.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7d5cda640 1 Processor -- start 2024-04-26T00:36:41.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7d5cda640 1 -- start start 2024-04-26T00:36:41.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7d5cda640 10 monclient: init 2024-04-26T00:36:41.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7d5cda640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:41.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7d5cda640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:36:41.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7d5cda640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7bc004810 0x7fa7d0131f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:41.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7d5cda640 10 monclient(hunting): picked mon.noname-c con 0x7fa7bc004810 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:36:41.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7d5cda640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7d0132490 0x7fa7d01cf290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:41.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7d5cda640 10 monclient(hunting): picked mon.noname-a con 0x7fa7d0132490 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:36:41.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7d5cda640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7d01cf7d0 0x7fa7d01cbd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:41.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7d5cda640 10 monclient(hunting): picked mon.noname-b con 0x7fa7d01cf7d0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:36:41.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7d5cda640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:41.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7d5cda640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d00a00e0 con 0x7fa7d0132490 2024-04-26T00:36:41.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7d5cda640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:41.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7d5cda640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d009f4c0 con 0x7fa7d01cf7d0 2024-04-26T00:36:41.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7d5cda640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:41.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7d5cda640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa7d0132e80 con 0x7fa7bc004810 2024-04-26T00:36:41.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7d5cda640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:41.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7d5cda640 10 monclient(hunting): authenticate will time out at 1982.547016s 2024-04-26T00:36:41.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7cf7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7bc004810 0x7fa7d0131f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:41.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7ceffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7d0132490 0x7fa7d01cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:41.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7ceffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7d0132490 0x7fa7d01cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50246/0 (socket says 172.21.15.37:50246) 2024-04-26T00:36:41.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7cf7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7bc004810 0x7fa7d0131f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:59340/0 (socket says 172.21.15.37:59340) 2024-04-26T00:36:41.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7ceffd640 1 -- 172.21.15.37:0/3139630891 learned_addr learned my addr 172.21.15.37:0/3139630891 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:36:41.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7ceffd640 10 monclient(hunting): get_auth_request con 0x7fa7d0132490 auth_method 0 2024-04-26T00:36:41.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7ceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:41.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7ceffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:41.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:41.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7cf7fe640 10 monclient(hunting): get_auth_request con 0x7fa7bc004810 auth_method 0 2024-04-26T00:36:41.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:41.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7cf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:41.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:41.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7ceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:41.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7ceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:41.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7ceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:41.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7cf7fe640 1 -- 172.21.15.37:0/3139630891 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7bc004810 msgr2=0x7fa7d0131f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:36:41.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7cf7fe640 1 -- 172.21.15.37:0/3139630891 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7bc004810 msgr2=0x7fa7d0131f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:36:41.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7cf7fe640 1 --2- 172.21.15.37:0/3139630891 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7bc004810 0x7fa7d0131f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:36:41.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7cf7fe640 1 --2- 172.21.15.37:0/3139630891 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7bc004810 0x7fa7d0131f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:36:41.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7cffff640 1 --2- 172.21.15.37:0/3139630891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7d01cf7d0 0x7fa7d01cbd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:41.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7cffff640 10 monclient(hunting): get_auth_request con 0x7fa7d01cf7d0 auth_method 0 2024-04-26T00:36:41.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:41.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7cffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:41.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7cffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:41.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7ceffd640 10 monclient(hunting): handle_auth_done global_id 144690 payload 824 2024-04-26T00:36:41.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7ceffd640 1 -- 172.21.15.37:0/3139630891 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7bc004810 msgr2=0x7fa7d0131f50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:36:41.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7ceffd640 1 --2- 172.21.15.37:0/3139630891 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7bc004810 0x7fa7d0131f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:41.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7ceffd640 1 -- 172.21.15.37:0/3139630891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7d01cf7d0 msgr2=0x7fa7d01cbd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:41.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7ceffd640 1 --2- 172.21.15.37:0/3139630891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7d01cf7d0 0x7fa7d01cbd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:41.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7ceffd640 10 monclient: _finish_hunting 0 2024-04-26T00:36:41.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7ceffd640 1 monclient: found mon.noname-a 2024-04-26T00:36:41.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:41.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7ceffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:36:41.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.236+0000 7fa7ceffd640 1 -- 172.21.15.37:0/3139630891 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7d0121950 con 0x7fa7d0132490 2024-04-26T00:36:41.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.240+0000 7fa7ceffd640 1 -- 172.21.15.37:0/3139630891 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7d0132490 msgr2=0x7fa7d01cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:41.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.240+0000 7fa7ceffd640 1 --2- 172.21.15.37:0/3139630891 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7d0132490 0x7fa7d01cf290 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fa7c001c6d0 tx=0x7fa7c0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:41.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.240+0000 7fa7c7fff640 1 -- 172.21.15.37:0/3139630891 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c00ba070 con 0x7fa7d0132490 2024-04-26T00:36:41.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.240+0000 7fa7c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:41.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.240+0000 7fa7c7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:36:41.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.240+0000 7fa7c7fff640 10 monclient: dump: 2024-04-26T00:36:41.254 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:41.254 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:41.254 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:41.254 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:41.254 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:41.254 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:41.254 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:41.254 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:41.255 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:41.255 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:41.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.240+0000 7fa7c7fff640 10 monclient: _finish_auth 0 2024-04-26T00:36:41.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.240+0000 7fa7c7fff640 10 monclient: _check_auth_tickets 2024-04-26T00:36:41.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.240+0000 7fa7c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:41.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.240+0000 7fa7c7fff640 1 -- 172.21.15.37:0/3139630891 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7c00039a0 con 0x7fa7d0132490 2024-04-26T00:36:41.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.240+0000 7fa7c7fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:36:41.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.240+0000 7fa7c7fff640 1 -- 172.21.15.37:0/3139630891 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c00b2040 con 0x7fa7d0132490 2024-04-26T00:36:41.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.240+0000 7fa7c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:41.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.240+0000 7fa7c7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:36:41.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.240+0000 7fa7c7fff640 10 monclient: dump: 2024-04-26T00:36:41.255 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:41.255 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:41.255 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:41.256 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:41.256 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:41.256 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:41.256 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:41.256 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:41.256 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:41.256 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:41.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.240+0000 7fa7d5cda640 5 monclient: authenticate success, global_id 144690 2024-04-26T00:36:41.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.240+0000 7fa7d5cda640 10 monclient: _renew_subs 2024-04-26T00:36:41.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.240+0000 7fa7d5cda640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:41.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.240+0000 7fa7d5cda640 1 -- 172.21.15.37:0/3139630891 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7d009e080 con 0x7fa7d0132490 2024-04-26T00:36:41.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.240+0000 7fa7cdffb640 4 set_mon_vals no callback set 2024-04-26T00:36:41.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.240+0000 7fa7d5cda640 10 monclient: _renew_subs 2024-04-26T00:36:41.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.240+0000 7fa7d5cda640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:41.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.240+0000 7fa7d5cda640 1 -- 172.21.15.37:0/3139630891 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7d01cfc10 con 0x7fa7d0132490 2024-04-26T00:36:41.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.240+0000 7fa7d5cda640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:41.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.240+0000 7fa7d5cda640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:41.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.240+0000 7fa7d5cda640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:41.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.240+0000 7fa7d5cda640 1 -- 172.21.15.37:0/3139630891 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7d0066f40 con 0x7fa7d0132490 2024-04-26T00:36:41.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.240+0000 7fa7c7fff640 1 -- 172.21.15.37:0/3139630891 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa7c00ad940 con 0x7fa7d0132490 2024-04-26T00:36:41.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.240+0000 7fa7c7fff640 1 --2- 172.21.15.37:0/3139630891 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa7b0035eb0 0x7fa7b0038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:41.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.240+0000 7fa7c7fff640 1 -- 172.21.15.37:0/3139630891 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(656..656 src has 370..656) ==== 3899+0+0 (secure 0 0 0) 0x7fa7c00e3360 con 0x7fa7d0132490 2024-04-26T00:36:41.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.240+0000 7fa7cf7fe640 1 --2- 172.21.15.37:0/3139630891 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa7b0035eb0 0x7fa7b0038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:41.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.240+0000 7fa7cf7fe640 10 monclient: get_auth_request con 0x7fa7b0035eb0 auth_method 0 2024-04-26T00:36:41.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.244+0000 7fa7cf7fe640 1 --2- 172.21.15.37:0/3139630891 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa7b0035eb0 0x7fa7b0038370 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa7b8000c10 tx=0x7fa7b8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:41.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.248+0000 7fa7c7fff640 1 -- 172.21.15.37:0/3139630891 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa7c0016970 con 0x7fa7d0132490 2024-04-26T00:36:41.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.248+0000 7fa7c7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:41.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.248+0000 7fa7c7fff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:36:41.284 INFO:tasks.rados.rados.0.smithi045.stdout:2895: finishing rollback tid 0 to smithi04526579-42 2024-04-26T00:36:41.285 INFO:tasks.rados.rados.0.smithi045.stdout:2895: finishing rollback tid 1 to smithi04526579-42 2024-04-26T00:36:41.285 INFO:tasks.rados.rados.0.smithi045.stdout:2895: finishing rollback tid 2 to smithi04526579-42 2024-04-26T00:36:41.285 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 2354 (ObjNum 903 snap 277 seq_num 903) dirty exists 2024-04-26T00:36:41.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.468+0000 7fa7d5cda640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:36:41.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.468+0000 7fa7d5cda640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:36:41.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.468+0000 7fa7d5cda640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:41.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.468+0000 7fa7d5cda640 1 -- 172.21.15.37:0/3139630891 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa7d006b9a0 con 0x7fa7d0132490 2024-04-26T00:36:41.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.472+0000 7fa7c7fff640 1 -- 172.21.15.37:0/3139630891 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fa7c00b6410 con 0x7fa7d0132490 2024-04-26T00:36:41.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.472+0000 7fa7c7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:36:41.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.472+0000 7fa7c7fff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:36:41.472 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":126,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:36:41.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.472+0000 7fa7d5cda640 1 -- 172.21.15.37:0/3139630891 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa7b0035eb0 msgr2=0x7fa7b0038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:41.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.472+0000 7fa7d5cda640 1 --2- 172.21.15.37:0/3139630891 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa7b0035eb0 0x7fa7b0038370 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa7b8000c10 tx=0x7fa7b8009040 comp rx=0 tx=0).stop 2024-04-26T00:36:41.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.472+0000 7fa7d5cda640 10 monclient: shutdown 2024-04-26T00:36:41.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.472+0000 7fa7d5cda640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:36:41.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.472+0000 7fa7d5cda640 1 -- 172.21.15.37:0/3139630891 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7d0132490 msgr2=0x7fa7d01cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:41.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.472+0000 7fa7d5cda640 1 --2- 172.21.15.37:0/3139630891 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7d0132490 0x7fa7d01cf290 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fa7c001c6d0 tx=0x7fa7c0009750 comp rx=0 tx=0).stop 2024-04-26T00:36:41.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.472+0000 7fa7d5cda640 1 -- 172.21.15.37:0/3139630891 shutdown_connections 2024-04-26T00:36:41.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.472+0000 7fa7d5cda640 1 --2- 172.21.15.37:0/3139630891 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa7b0035eb0 0x7fa7b0038370 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:41.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.472+0000 7fa7d5cda640 1 --2- 172.21.15.37:0/3139630891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7d01cf7d0 0x7fa7d01cbd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:41.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.472+0000 7fa7d5cda640 1 --2- 172.21.15.37:0/3139630891 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7d0132490 0x7fa7d01cf290 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:41.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.472+0000 7fa7d5cda640 1 --2- 172.21.15.37:0/3139630891 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7bc004810 0x7fa7d0131f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:41.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.472+0000 7fa7d5cda640 1 -- 172.21.15.37:0/3139630891 >> 172.21.15.37:0/3139630891 conn(0x7fa7d00a1ad0 msgr2=0x7fa7d00a2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:36:41.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.472+0000 7fa7d5cda640 1 -- 172.21.15.37:0/3139630891 shutdown_connections 2024-04-26T00:36:41.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.472+0000 7fa7d5cda640 1 -- 172.21.15.37:0/3139630891 wait complete. 2024-04-26T00:36:41.490 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:36:41.490 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:36:41.490 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:36:41.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.612+0000 7fd1881cc640 10 monclient: get_monmap_and_config 2024-04-26T00:36:41.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.612+0000 7fd1881cc640 10 monclient: build_initial_monmap 2024-04-26T00:36:41.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.612+0000 7fd1881cc640 10 monclient: monmap: 2024-04-26T00:36:41.615 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:36:41.615 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:41.615 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:36:41.619285+0000 2024-04-26T00:36:41.615 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:36:41.619285+0000 2024-04-26T00:36:41.615 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:36:41.615 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:41.615 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:36:41.616 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:36:41.616 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:36:41.616 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:36:41.616 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:36:41.616 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:41.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.616+0000 7fd1881cc640 1 Processor -- start 2024-04-26T00:36:41.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.616+0000 7fd1881cc640 1 -- start start 2024-04-26T00:36:41.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.616+0000 7fd1881cc640 10 monclient: init 2024-04-26T00:36:41.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.616+0000 7fd1881cc640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:41.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.616+0000 7fd1881cc640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-04-26T00:36:41.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.616+0000 7fd1881cc640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd1800a0300 0x7fd18009fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:41.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.616+0000 7fd1881cc640 10 monclient(hunting): picked mon.noname-a con 0x7fd1800a0300 addr v2:172.21.15.37:3300/0 2024-04-26T00:36:41.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.616+0000 7fd1881cc640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd1800a06e0 0x7fd180134ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:41.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.616+0000 7fd1881cc640 10 monclient(hunting): picked mon.noname-c con 0x7fd1800a06e0 addr v2:172.21.15.45:3300/0 2024-04-26T00:36:41.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.616+0000 7fd1881cc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd1801350e0 addr v1:172.21.15.37:6789/0 2024-04-26T00:36:41.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.616+0000 7fd1881cc640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd18009c190 con 0x7fd1801350e0 2024-04-26T00:36:41.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.616+0000 7fd1881cc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:41.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.616+0000 7fd1881cc640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fd1800a13f0 con 0x7fd1800a0300 2024-04-26T00:36:41.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.616+0000 7fd1881cc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:41.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.616+0000 7fd1881cc640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fd1800a1110 con 0x7fd1800a06e0 2024-04-26T00:36:41.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.616+0000 7fd1881cc640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:41.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.616+0000 7fd1881cc640 10 monclient(hunting): authenticate will time out at 1982.924911s 2024-04-26T00:36:41.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.616+0000 7fd185f68640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd1800a0300 0x7fd18009fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:41.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.616+0000 7fd185f68640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd1800a0300 0x7fd18009fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50260/0 (socket says 172.21.15.37:50260) 2024-04-26T00:36:41.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.616+0000 7fd186769640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fd1801350e0 0x7fd18013bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:33994/0 (socket says 172.21.15.37:33994) 2024-04-26T00:36:41.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.616+0000 7fd185f68640 1 -- 172.21.15.37:0/3303913594 learned_addr learned my addr 172.21.15.37:0/3303913594 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:36:41.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.616+0000 7fd185f68640 10 monclient(hunting): get_auth_request con 0x7fd1800a0300 auth_method 0 2024-04-26T00:36:41.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.616+0000 7fd185f68640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:41.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.616+0000 7fd185f68640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:41.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.616+0000 7fd185f68640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:41.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.616+0000 7fd185f68640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:41.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.616+0000 7fd185f68640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:41.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.616+0000 7fd185f68640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:41.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.616+0000 7fd185767640 1 --2- 172.21.15.37:0/3303913594 >> v2:172.21.15.45:3300/0 conn(0x7fd1800a06e0 0x7fd180134ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:41.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.616+0000 7fd185f68640 10 monclient(hunting): handle_auth_done global_id 144693 payload 274 2024-04-26T00:36:41.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.616+0000 7fd185f68640 1 -- 172.21.15.37:0/3303913594 >> v2:172.21.15.45:3300/0 conn(0x7fd1800a06e0 msgr2=0x7fd180134ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:36:41.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.616+0000 7fd185f68640 1 --2- 172.21.15.37:0/3303913594 >> v2:172.21.15.45:3300/0 conn(0x7fd1800a06e0 0x7fd180134ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:41.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.616+0000 7fd185f68640 1 -- 172.21.15.37:0/3303913594 >> v1:172.21.15.37:6789/0 conn(0x7fd1801350e0 legacy=0x7fd18013bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:36:41.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.616+0000 7fd185f68640 10 monclient: _finish_hunting 0 2024-04-26T00:36:41.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.616+0000 7fd185f68640 1 monclient: found mon.noname-a 2024-04-26T00:36:41.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.616+0000 7fd185f68640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:41.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.616+0000 7fd185f68640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:36:41.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.616+0000 7fd185f68640 1 -- 172.21.15.37:0/3303913594 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1800a16d0 con 0x7fd1800a0300 2024-04-26T00:36:41.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.616+0000 7fd185767640 10 monclient: get_auth_request con 0x7fd1800a06e0 auth_method 0 2024-04-26T00:36:41.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.616+0000 7fd185767640 1 --2- 172.21.15.37:0/3303913594 >> v2:172.21.15.45:3300/0 conn(0x7fd1800a06e0 0x7fd180134ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:36:41.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.620+0000 7fd185f68640 1 -- 172.21.15.37:0/3303913594 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd1800a0300 msgr2=0x7fd18009fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:41.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.620+0000 7fd185f68640 1 --2- 172.21.15.37:0/3303913594 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd1800a0300 0x7fd18009fc50 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd18006caa0 tx=0x7fd17801c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=608e8442d6e1ee02 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:41.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.620+0000 7fd184f66640 1 -- 172.21.15.37:0/3303913594 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1780a55f0 con 0x7fd1800a0300 2024-04-26T00:36:41.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.620+0000 7fd184f66640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:41.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.620+0000 7fd184f66640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:36:41.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.620+0000 7fd184f66640 10 monclient: dump: 2024-04-26T00:36:41.632 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:41.632 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:41.632 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:41.632 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:41.632 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:41.632 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:41.633 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:41.633 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:41.633 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:41.633 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:41.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.620+0000 7fd184f66640 10 monclient: _finish_auth 0 2024-04-26T00:36:41.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.620+0000 7fd184f66640 10 monclient: _check_auth_tickets 2024-04-26T00:36:41.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.620+0000 7fd184f66640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:41.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.620+0000 7fd184f66640 1 -- 172.21.15.37:0/3303913594 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1780a5790 con 0x7fd1800a0300 2024-04-26T00:36:41.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.620+0000 7fd184f66640 10 monclient: handle_config config(0 keys) 2024-04-26T00:36:41.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.620+0000 7fd184f66640 1 -- 172.21.15.37:0/3303913594 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1780a5b60 con 0x7fd1800a0300 2024-04-26T00:36:41.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.620+0000 7fd184f66640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:41.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.620+0000 7fd184f66640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:36:41.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.620+0000 7fd184f66640 10 monclient: dump: 2024-04-26T00:36:41.633 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:41.634 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:41.634 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:41.634 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:41.634 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:41.634 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:41.634 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:41.634 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:41.634 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:41.634 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:41.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.620+0000 7fd1881cc640 5 monclient: authenticate success, global_id 144693 2024-04-26T00:36:41.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.620+0000 7fd1881cc640 10 monclient: get_monmap_and_config success 2024-04-26T00:36:41.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.620+0000 7fd1881cc640 4 set_mon_vals no callback set 2024-04-26T00:36:41.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.620+0000 7fd1881cc640 10 monclient: shutdown 2024-04-26T00:36:41.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.620+0000 7fd1881cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:36:41.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.620+0000 7fd1881cc640 1 -- 172.21.15.37:0/3303913594 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd1800a0300 msgr2=0x7fd18009fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:41.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.620+0000 7fd1881cc640 1 --2- 172.21.15.37:0/3303913594 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd1800a0300 0x7fd18009fc50 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd18006caa0 tx=0x7fd17801c8a0 comp rx=0 tx=0).stop 2024-04-26T00:36:41.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.620+0000 7fd1881cc640 1 -- 172.21.15.37:0/3303913594 shutdown_connections 2024-04-26T00:36:41.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.620+0000 7fd1881cc640 1 --2- 172.21.15.37:0/3303913594 >> v2:172.21.15.45:3300/0 conn(0x7fd1800a06e0 0x7fd180134ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:41.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.620+0000 7fd1881cc640 1 --2- 172.21.15.37:0/3303913594 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd1800a0300 0x7fd18009fc50 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:41.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.620+0000 7fd1881cc640 1 -- 172.21.15.37:0/3303913594 >> 172.21.15.37:0/3303913594 conn(0x7fd1800a1d50 msgr2=0x7fd1800a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:36:41.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.620+0000 7fd1881cc640 1 -- 172.21.15.37:0/3303913594 shutdown_connections 2024-04-26T00:36:41.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.620+0000 7fd1881cc640 1 -- 172.21.15.37:0/3303913594 wait complete. 2024-04-26T00:36:41.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.620+0000 7fd1881cc640 10 monclient: build_initial_monmap 2024-04-26T00:36:41.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.620+0000 7fd1881cc640 10 monclient: monmap: 2024-04-26T00:36:41.636 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:36:41.636 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:36:41.637 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:36:41.637 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:36:41.637 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:36:41.637 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:36:41.637 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:36:41.637 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:36:41.637 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:36:41.637 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:41.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.620+0000 7fd1881cc640 1 Processor -- start 2024-04-26T00:36:41.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd1881cc640 1 -- start start 2024-04-26T00:36:41.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd1881cc640 10 monclient: init 2024-04-26T00:36:41.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd1881cc640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:41.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd1881cc640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:36:41.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd1881cc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd1800a0300 0x7fd18013c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:41.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd1881cc640 10 monclient(hunting): picked mon.noname-c con 0x7fd1800a0300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:36:41.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd1881cc640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd1800a06e0 0x7fd1801cf4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:41.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd1881cc640 10 monclient(hunting): picked mon.noname-a con 0x7fd1800a06e0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:36:41.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd1881cc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1801350e0 0x7fd1801cf9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:41.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd1881cc640 10 monclient(hunting): picked mon.noname-b con 0x7fd1801350e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:36:41.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd1881cc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:41.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd1881cc640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd1800a0c40 con 0x7fd1800a06e0 2024-04-26T00:36:41.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd1881cc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:41.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd1881cc640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd1800a0ac0 con 0x7fd1801350e0 2024-04-26T00:36:41.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd1881cc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:41.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd1881cc640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd18013fe10 con 0x7fd1800a0300 2024-04-26T00:36:41.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd1881cc640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:41.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd1881cc640 10 monclient(hunting): authenticate will time out at 1982.932637s 2024-04-26T00:36:41.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd185767640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd1800a06e0 0x7fd1801cf4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:41.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd185f68640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd1800a0300 0x7fd18013c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:41.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd185767640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd1800a06e0 0x7fd1801cf4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50266/0 (socket says 172.21.15.37:50266) 2024-04-26T00:36:41.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd185767640 1 -- 172.21.15.37:0/2979121732 learned_addr learned my addr 172.21.15.37:0/2979121732 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:36:41.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd185767640 10 monclient(hunting): get_auth_request con 0x7fd1800a06e0 auth_method 0 2024-04-26T00:36:41.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd185767640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:41.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd185767640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:41.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd185767640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:41.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd185f68640 10 monclient(hunting): get_auth_request con 0x7fd1800a0300 auth_method 0 2024-04-26T00:36:41.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd185f68640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:41.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd185f68640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:41.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd185f68640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:41.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd185767640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:41.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd185767640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:41.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd185767640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:41.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd185f68640 1 -- 172.21.15.37:0/2979121732 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd1800a0300 msgr2=0x7fd18013c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:36:41.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd185f68640 1 -- 172.21.15.37:0/2979121732 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd1800a0300 msgr2=0x7fd18013c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:36:41.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd185f68640 1 --2- 172.21.15.37:0/2979121732 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd1800a0300 0x7fd18013c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:36:41.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd185f68640 1 --2- 172.21.15.37:0/2979121732 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd1800a0300 0x7fd18013c4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:36:41.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd186769640 1 --2- 172.21.15.37:0/2979121732 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1801350e0 0x7fd1801cf9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:41.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd186769640 10 monclient(hunting): get_auth_request con 0x7fd1801350e0 auth_method 0 2024-04-26T00:36:41.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd186769640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:41.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd186769640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:41.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd186769640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:41.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd185767640 10 monclient(hunting): handle_auth_done global_id 144696 payload 824 2024-04-26T00:36:41.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd185767640 1 -- 172.21.15.37:0/2979121732 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd1800a0300 msgr2=0x7fd18013c4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:36:41.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd185767640 1 --2- 172.21.15.37:0/2979121732 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd1800a0300 0x7fd18013c4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:41.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd185767640 1 -- 172.21.15.37:0/2979121732 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1801350e0 msgr2=0x7fd1801cf9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:41.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd185767640 1 --2- 172.21.15.37:0/2979121732 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd1801350e0 0x7fd1801cf9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:41.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd185767640 10 monclient: _finish_hunting 0 2024-04-26T00:36:41.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd185767640 1 monclient: found mon.noname-a 2024-04-26T00:36:41.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd185767640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:41.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd185767640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:36:41.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd185767640 1 -- 172.21.15.37:0/2979121732 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd18009f550 con 0x7fd1800a06e0 2024-04-26T00:36:41.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd185767640 1 -- 172.21.15.37:0/2979121732 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd1800a06e0 msgr2=0x7fd1801cf4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:41.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd185767640 1 --2- 172.21.15.37:0/2979121732 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd1800a06e0 0x7fd1801cf4a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd17400c7c0 tx=0x7fd17400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:41.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd1727fc640 1 -- 172.21.15.37:0/2979121732 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd174007c40 con 0x7fd1800a06e0 2024-04-26T00:36:41.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd1727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:41.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd1727fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:36:41.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd1727fc640 10 monclient: dump: 2024-04-26T00:36:41.642 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:41.642 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:41.642 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:41.642 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:41.642 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:41.642 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:41.642 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:41.642 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:41.643 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:41.643 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:41.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd1727fc640 10 monclient: _finish_auth 0 2024-04-26T00:36:41.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd1727fc640 10 monclient: _check_auth_tickets 2024-04-26T00:36:41.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd1727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:41.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd1881cc640 5 monclient: authenticate success, global_id 144696 2024-04-26T00:36:41.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd1881cc640 10 monclient: _renew_subs 2024-04-26T00:36:41.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd1881cc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:41.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd1727fc640 1 -- 172.21.15.37:0/2979121732 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd174007de0 con 0x7fd1800a06e0 2024-04-26T00:36:41.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd1881cc640 1 -- 172.21.15.37:0/2979121732 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd18006e560 con 0x7fd1800a06e0 2024-04-26T00:36:41.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd1727fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:36:41.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd1727fc640 1 -- 172.21.15.37:0/2979121732 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd174011630 con 0x7fd1800a06e0 2024-04-26T00:36:41.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd1727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:41.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd1727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:36:41.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd1727fc640 10 monclient: dump: 2024-04-26T00:36:41.644 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:41.644 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:41.644 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:41.644 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:41.644 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:41.644 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:41.644 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:41.644 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:41.644 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:41.644 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:41.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd173fff640 4 set_mon_vals no callback set 2024-04-26T00:36:41.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd1881cc640 10 monclient: _renew_subs 2024-04-26T00:36:41.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd1881cc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:41.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.624+0000 7fd1881cc640 1 -- 172.21.15.37:0/2979121732 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd18013a7a0 con 0x7fd1800a06e0 2024-04-26T00:36:41.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.628+0000 7fd1881cc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:41.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.628+0000 7fd1881cc640 1 --2- 172.21.15.37:0/2979121732 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd1801cff20 0x7fd18012da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:41.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.628+0000 7fd1881cc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:41.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.628+0000 7fd1881cc640 1 -- 172.21.15.37:0/2979121732 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd1801d0300 con 0x7fd1801cff20 2024-04-26T00:36:41.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.628+0000 7fd185f68640 1 --2- 172.21.15.37:0/2979121732 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd1801cff20 0x7fd18012da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:41.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.628+0000 7fd185f68640 10 monclient: get_auth_request con 0x7fd1801cff20 auth_method 0 2024-04-26T00:36:41.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.628+0000 7fd185f68640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:41.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.628+0000 7fd185f68640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:41.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.628+0000 7fd185f68640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:41.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.628+0000 7fd1727fc640 1 -- 172.21.15.37:0/2979121732 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd1740117d0 con 0x7fd1800a06e0 2024-04-26T00:36:41.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.628+0000 7fd185f68640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:41.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.628+0000 7fd185f68640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:41.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.628+0000 7fd185f68640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:41.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.628+0000 7fd1727fc640 1 --2- 172.21.15.37:0/2979121732 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd164035e80 0x7fd164038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:41.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.628+0000 7fd1727fc640 1 -- 172.21.15.37:0/2979121732 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(656..656 src has 370..656) ==== 3899+0+0 (secure 0 0 0) 0x7fd174048cd0 con 0x7fd1800a06e0 2024-04-26T00:36:41.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.628+0000 7fd185f68640 10 monclient(hunting): handle_auth_done global_id 144699 payload 824 2024-04-26T00:36:41.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.628+0000 7fd185f68640 1 -- 172.21.15.37:0/2979121732 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd18004f510 con 0x7fd1801cff20 2024-04-26T00:36:41.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.628+0000 7fd186769640 1 --2- 172.21.15.37:0/2979121732 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd164035e80 0x7fd164038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:41.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.628+0000 7fd186769640 10 monclient: get_auth_request con 0x7fd164035e80 auth_method 0 2024-04-26T00:36:41.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.628+0000 7fd185f68640 1 -- 172.21.15.37:0/2979121732 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd1801cff20 msgr2=0x7fd18012da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:41.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.628+0000 7fd185f68640 1 --2- 172.21.15.37:0/2979121732 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd1801cff20 0x7fd18012da50 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fd1780ae000 tx=0x7fd1780196a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:41.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.628+0000 7fd1727fc640 1 -- 172.21.15.37:0/2979121732 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1780b6620 con 0x7fd1801cff20 2024-04-26T00:36:41.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.628+0000 7fd1727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:41.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.628+0000 7fd1727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:36:41.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.628+0000 7fd1727fc640 10 monclient: dump: 2024-04-26T00:36:41.649 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:41.649 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:41.649 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:41.649 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:41.649 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:41.649 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:41.649 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:41.649 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:41.649 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:41.649 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:41.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.628+0000 7fd1727fc640 1 -- 172.21.15.37:0/2979121732 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd1780b6620 con 0x7fd1801cff20 2024-04-26T00:36:41.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.628+0000 7fd1727fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:41.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.628+0000 7fd1727fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:36:41.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.628+0000 7fd1727fc640 1 -- 172.21.15.37:0/2979121732 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd1801cff20 msgr2=0x7fd18012da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:41.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.628+0000 7fd1727fc640 1 --2- 172.21.15.37:0/2979121732 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd1801cff20 0x7fd18012da50 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fd1780ae000 tx=0x7fd1780196a0 comp rx=0 tx=0).stop 2024-04-26T00:36:41.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.628+0000 7fd1727fc640 1 -- 172.21.15.37:0/2979121732 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd1801cff20 msgr2=0x7fd18012da50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:36:41.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.628+0000 7fd1727fc640 1 --2- 172.21.15.37:0/2979121732 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd1801cff20 0x7fd18012da50 secure :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fd1780ae000 tx=0x7fd1780196a0 comp rx=0 tx=0).stop 2024-04-26T00:36:41.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.632+0000 7fd186769640 1 --2- 172.21.15.37:0/2979121732 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd164035e80 0x7fd164038340 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd18009e700 tx=0x7fd17c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:41.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.636+0000 7fd1881cc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:36:41.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.636+0000 7fd1881cc640 1 --2- 172.21.15.37:0/2979121732 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd18012df90 0x7fd18012e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:41.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.636+0000 7fd1881cc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:41.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.636+0000 7fd1881cc640 1 -- 172.21.15.37:0/2979121732 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd18012e8e0 con 0x7fd18012df90 2024-04-26T00:36:41.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.636+0000 7fd185f68640 1 --2- 172.21.15.37:0/2979121732 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd18012df90 0x7fd18012e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:41.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.636+0000 7fd185f68640 10 monclient: get_auth_request con 0x7fd18012df90 auth_method 0 2024-04-26T00:36:41.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.636+0000 7fd185f68640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:41.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.636+0000 7fd185f68640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:41.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.636+0000 7fd185f68640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:41.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.636+0000 7fd185f68640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:41.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.636+0000 7fd185f68640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:41.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.636+0000 7fd185f68640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:41.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.636+0000 7fd185f68640 10 monclient(hunting): handle_auth_done global_id 144702 payload 824 2024-04-26T00:36:41.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.636+0000 7fd185f68640 1 -- 172.21.15.37:0/2979121732 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd18013cce0 con 0x7fd18012df90 2024-04-26T00:36:41.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.640+0000 7fd185f68640 1 -- 172.21.15.37:0/2979121732 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd18012df90 msgr2=0x7fd18012e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:41.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.640+0000 7fd185f68640 1 --2- 172.21.15.37:0/2979121732 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd18012df90 0x7fd18012e3a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd17801c8e0 tx=0x7fd1780a7000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:41.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.640+0000 7fd1727fc640 1 -- 172.21.15.37:0/2979121732 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1780b69d0 con 0x7fd18012df90 2024-04-26T00:36:41.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.640+0000 7fd1727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:41.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.640+0000 7fd1727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:36:41.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.640+0000 7fd1727fc640 10 monclient: dump: 2024-04-26T00:36:41.652 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:41.652 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:41.652 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:41.652 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:41.652 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:41.652 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:41.652 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:41.652 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:41.652 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:41.652 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:41.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.640+0000 7fd1727fc640 1 -- 172.21.15.37:0/2979121732 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4890 (secure 0 0 0) 0x7fd1780b69d0 con 0x7fd18012df90 2024-04-26T00:36:41.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.640+0000 7fd1727fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:36:41.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.640+0000 7fd1727fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:36:41.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.640+0000 7fd1727fc640 1 -- 172.21.15.37:0/2979121732 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd18012df90 msgr2=0x7fd18012e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:41.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.640+0000 7fd1727fc640 1 --2- 172.21.15.37:0/2979121732 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd18012df90 0x7fd18012e3a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd17801c8e0 tx=0x7fd1780a7000 comp rx=0 tx=0).stop 2024-04-26T00:36:41.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.640+0000 7fd1727fc640 1 -- 172.21.15.37:0/2979121732 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd18012df90 msgr2=0x7fd18012e3a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:36:41.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.640+0000 7fd1727fc640 1 --2- 172.21.15.37:0/2979121732 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd18012df90 0x7fd18012e3a0 secure :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd17801c8e0 tx=0x7fd1780a7000 comp rx=0 tx=0).stop 2024-04-26T00:36:41.653 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:36:41.654 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:36:41.654 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:36:41.654 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:36:41.654 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 126, 2024-04-26T00:36:41.654 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 288184, 2024-04-26T00:36:41.654 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:36:41.654 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:36:41.654 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:36:41.654 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:36:41.654 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:36:41.654 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 1, 2024-04-26T00:36:41.654 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:36:41.655 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:36:41.655 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:36:41.655 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:36:41.655 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:36:41.655 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:36:41.655 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:36:41.655 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:36:41.655 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:36:41.655 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:36:41.655 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:36:41.655 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:36:41.655 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:36:41.655 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:36:41.655 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:36:41.656 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:36:41.656 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:36:41.656 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:36:41.656 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:36:41.656 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:36:41.656 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:36:41.656 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:36:41.656 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:36:41.656 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:36:41.657 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:36:41.657 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:36:41.657 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:36:41.657 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:36:41.657 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:41.657 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:41.658 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:36:41.658 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:36:41.658 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:36:41.658 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:36:41.658 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:36:41.658 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:36:41.658 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:36:41.658 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:36:41.658 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:36:41.658 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:36:41.658 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:36:41.659 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:36:41.659 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:36:41.659 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:36:41.659 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:36:41.659 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:36:41.659 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:36:41.659 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:36:41.659 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:36:41.659 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:36:41.659 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:36:41.659 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:36:41.659 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:36:41.659 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:36:41.659 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:36:41.660 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:36:41.660 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:36:41.660 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:36:41.660 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:36:41.660 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:36:41.660 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:41.660 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:36:41.660 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:41.660 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:36:41.660 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:36:41.660 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:36:41.660 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:36:41.661 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:41.661 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:36:41.661 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:36:41.661 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:41.661 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:41.661 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:41.661 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:36:41.662 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:36:41.662 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:41.662 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:41.662 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:41.662 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:41.662 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:36:41.662 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:36:41.662 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:36:41.662 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:36:41.662 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:36:41.662 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:41.662 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:41.662 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:36:41.663 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:36:41.663 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:36:41.663 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:36:41.663 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:41.663 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:36:41.663 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:36:41.663 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:41.663 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:41.663 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:41.663 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:36:41.663 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:36:41.663 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:41.664 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:41.664 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:41.664 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:41.664 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:36:41.664 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:36:41.664 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:36:41.664 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:36:41.664 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:36:41.664 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:41.664 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:41.664 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:36:41.664 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:36:41.664 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:36:41.664 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:36:41.665 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:41.665 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:36:41.665 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:36:41.665 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:41.665 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:41.665 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:41.665 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:36:41.665 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:36:41.665 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:41.665 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:41.665 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:41.665 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:41.665 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:36:41.665 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:36:41.665 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:36:41.666 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:36:41.666 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:36:41.666 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:41.666 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:41.666 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:41.666 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:36:41.666 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:36:41.666 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:41.666 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:36:41.666 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:36:41.666 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:36:41.666 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:41.666 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:36:41.667 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:36:41.667 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:41.667 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:36:41.667 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:36:41.667 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:36:41.667 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:41.667 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:36:41.667 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:36:41.667 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:41.667 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:36:41.667 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:36:41.667 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 7 2024-04-26T00:36:41.667 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:41.668 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:36:41.668 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:36:41.668 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:41.668 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:36:41.668 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:36:41.668 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:36:41.668 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:41.669 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:41.669 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:41.669 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:36:41.669 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:36:41.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.640+0000 7fd1881cc640 1 -- 172.21.15.37:0/2979121732 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd164035e80 msgr2=0x7fd164038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:41.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.640+0000 7fd1881cc640 1 --2- 172.21.15.37:0/2979121732 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd164035e80 0x7fd164038340 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd18009e700 tx=0x7fd17c009040 comp rx=0 tx=0).stop 2024-04-26T00:36:41.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.640+0000 7fd1881cc640 10 monclient: shutdown 2024-04-26T00:36:41.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.640+0000 7fd1881cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:36:41.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.640+0000 7fd1881cc640 1 -- 172.21.15.37:0/2979121732 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd1800a06e0 msgr2=0x7fd1801cf4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:41.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.640+0000 7fd1881cc640 1 --2- 172.21.15.37:0/2979121732 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd1800a06e0 0x7fd1801cf4a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd17400c7c0 tx=0x7fd17400cc80 comp rx=0 tx=0).stop 2024-04-26T00:36:41.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.640+0000 7fd186769640 1 -- 172.21.15.37:0/2979121732 reap_dead start 2024-04-26T00:36:41.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.640+0000 7fd186769640 1 -- 172.21.15.37:0/2979121732 reap_dead start 2024-04-26T00:36:41.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.640+0000 7fd1881cc640 1 -- 172.21.15.37:0/2979121732 shutdown_connections 2024-04-26T00:36:41.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.640+0000 7fd1881cc640 1 -- 172.21.15.37:0/2979121732 >> 172.21.15.37:0/2979121732 conn(0x7fd1800a1d50 msgr2=0x7fd1800a3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:36:41.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.640+0000 7fd1881cc640 1 -- 172.21.15.37:0/2979121732 shutdown_connections 2024-04-26T00:36:41.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.640+0000 7fd1881cc640 1 -- 172.21.15.37:0/2979121732 wait complete. 2024-04-26T00:36:41.672 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:36:41.672 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:36:41.672 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:36:41.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.784+0000 7ff3ad57b640 10 monclient: get_monmap_and_config 2024-04-26T00:36:41.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.784+0000 7ff3ad57b640 10 monclient: build_initial_monmap 2024-04-26T00:36:41.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.784+0000 7ff3ad57b640 10 monclient: monmap: 2024-04-26T00:36:41.787 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:36:41.787 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:41.787 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:36:41.791817+0000 2024-04-26T00:36:41.787 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:36:41.791817+0000 2024-04-26T00:36:41.787 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:36:41.787 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:41.787 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:36:41.788 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:36:41.788 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:36:41.788 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:36:41.788 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:36:41.788 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:41.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.788+0000 7ff3ad57b640 1 Processor -- start 2024-04-26T00:36:41.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.788+0000 7ff3ad57b640 1 -- start start 2024-04-26T00:36:41.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.788+0000 7ff3ad57b640 10 monclient: init 2024-04-26T00:36:41.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.788+0000 7ff3ad57b640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:41.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.788+0000 7ff3ad57b640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-04-26T00:36:41.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.788+0000 7ff3ad57b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff3a8136f20 addr v1:172.21.15.45:6789/0 2024-04-26T00:36:41.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.788+0000 7ff3ad57b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a8137a10 0x7ff3a809bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:41.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.788+0000 7ff3ad57b640 10 monclient(hunting): picked mon.noname-b con 0x7ff3a8137a10 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:36:41.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.788+0000 7ff3ad57b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff3a813c450 addr v1:172.21.15.37:6789/0 2024-04-26T00:36:41.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.788+0000 7ff3ad57b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3a80a50f0 con 0x7ff3a813c450 2024-04-26T00:36:41.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.788+0000 7ff3ad57b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3a809b330 con 0x7ff3a8136f20 2024-04-26T00:36:41.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.788+0000 7ff3ad57b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:41.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.788+0000 7ff3ad57b640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff3a809b610 con 0x7ff3a8137a10 2024-04-26T00:36:41.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.788+0000 7ff3ad57b640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:41.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.788+0000 7ff3ad57b640 10 monclient(hunting): authenticate will time out at 1983.097672s 2024-04-26T00:36:41.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.788+0000 7ff3a67fc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a8137a10 0x7ff3a809bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:41.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.788+0000 7ff3a77fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7ff3a813c450 0x7ff3a813e840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:33998/0 (socket says 172.21.15.37:33998) 2024-04-26T00:36:41.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.788+0000 7ff3a77fe640 1 -- 172.21.15.37:0/1904292319 learned_addr learned my addr 172.21.15.37:0/1904292319 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:36:41.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.788+0000 7ff3a67fc640 10 monclient(hunting): get_auth_request con 0x7ff3a8137a10 auth_method 0 2024-04-26T00:36:41.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.788+0000 7ff3a67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:41.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.788+0000 7ff3a67fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:41.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.788+0000 7ff3a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:41.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.788+0000 7ff3a67fc640 1 -- 172.21.15.37:0/1904292319 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a8137a10 msgr2=0x7ff3a809bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:36:41.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.788+0000 7ff3a67fc640 1 -- 172.21.15.37:0/1904292319 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a8137a10 msgr2=0x7ff3a809bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:36:41.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.788+0000 7ff3a67fc640 1 --2- 172.21.15.37:0/1904292319 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a8137a10 0x7ff3a809bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:36:41.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.788+0000 7ff3a67fc640 1 --2- 172.21.15.37:0/1904292319 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a8137a10 0x7ff3a809bff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:36:41.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a77fe640 1 -- 172.21.15.37:0/1904292319 >> v1:172.21.15.37:6789/0 conn(0x7ff3a813c450 legacy=0x7ff3a813e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:41.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a5ffb640 1 -- 172.21.15.37:0/1904292319 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2488807434 0 0) 0x7ff3a80a50f0 con 0x7ff3a813c450 2024-04-26T00:36:41.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a5ffb640 10 monclient(hunting): _negotiate 2024-04-26T00:36:41.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a5ffb640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:41.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:41.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a5ffb640 10 monclient(hunting): my global_id is 144705 2024-04-26T00:36:41.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a5ffb640 1 -- 172.21.15.37:0/1904292319 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff390003530 con 0x7ff3a813c450 2024-04-26T00:36:41.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a6ffd640 1 -- 172.21.15.37:0/1904292319 >> v1:172.21.15.45:6789/0 conn(0x7ff3a8136f20 legacy=0x7ff3a8137300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:41.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a5ffb640 1 -- 172.21.15.37:0/1904292319 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2875780496 0 0) 0x7ff390003530 con 0x7ff3a813c450 2024-04-26T00:36:41.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a5ffb640 1 -- 172.21.15.37:0/1904292319 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a8137a10 msgr2=0x7ff3a809bff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:36:41.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a5ffb640 1 --2- 172.21.15.37:0/1904292319 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a8137a10 0x7ff3a809bff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:41.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a5ffb640 1 -- 172.21.15.37:0/1904292319 >> v1:172.21.15.45:6789/0 conn(0x7ff3a8136f20 legacy=0x7ff3a8137300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:41.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a5ffb640 10 monclient: _finish_hunting 0 2024-04-26T00:36:41.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a5ffb640 1 monclient: found mon.noname-a-legacy 2024-04-26T00:36:41.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:41.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a5ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.37:6789/0 2024-04-26T00:36:41.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a5ffb640 1 -- 172.21.15.37:0/1904292319 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3a809b8f0 con 0x7ff3a813c450 2024-04-26T00:36:41.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a5ffb640 10 monclient: _finish_auth 0 2024-04-26T00:36:41.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a5ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:36:41.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:41.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a5ffb640 1 -- 172.21.15.37:0/1904292319 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 866326386 0 0) 0x7ff3a0001d60 con 0x7ff3a813c450 2024-04-26T00:36:41.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3ad57b640 5 monclient: authenticate success, global_id 144705 2024-04-26T00:36:41.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3ad57b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:36:41.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:41.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a5ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-26T00:36:41.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a5ffb640 10 monclient: dump: 2024-04-26T00:36:41.797 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:41.797 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:41.797 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:41.797 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:41.797 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:41.797 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:41.797 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:41.797 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:41.797 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:41.797 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:41.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a5ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] but i'm connected to v1:172.21.15.37:6789/0, reconnecting 2024-04-26T00:36:41.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a5ffb640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:41.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a5ffb640 1 -- 172.21.15.37:0/1904292319 >> v1:172.21.15.37:6789/0 conn(0x7ff3a813c450 legacy=0x7ff3a813e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:41.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a5ffb640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:36:41.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a5ffb640 1 --2- 172.21.15.37:0/1904292319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3900039f0 0x7ff390003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:41.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a5ffb640 10 monclient(hunting): picked mon.b con 0x7ff3900039f0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:36:41.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a5ffb640 1 --2- 172.21.15.37:0/1904292319 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff390004810 0x7ff390004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:41.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a5ffb640 10 monclient(hunting): picked mon.a con 0x7ff390004810 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:36:41.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a5ffb640 1 --2- 172.21.15.37:0/1904292319 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a8137a10 0x7ff39000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:41.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a5ffb640 10 monclient(hunting): picked mon.c con 0x7ff3a8137a10 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:36:41.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a5ffb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:41.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a5ffb640 1 -- 172.21.15.37:0/1904292319 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a809b610 con 0x7ff390004810 2024-04-26T00:36:41.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a5ffb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:41.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a5ffb640 1 -- 172.21.15.37:0/1904292319 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff390005530 con 0x7ff3900039f0 2024-04-26T00:36:41.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a5ffb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:41.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a5ffb640 1 -- 172.21.15.37:0/1904292319 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff39000bf10 con 0x7ff3a8137a10 2024-04-26T00:36:41.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a5ffb640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:41.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a5ffb640 10 monclient(hunting): _finish_auth 0 2024-04-26T00:36:41.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3ad57b640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-26T00:36:41.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a67fc640 1 --2- 172.21.15.37:0/1904292319 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff390004810 0x7ff390004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:41.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a77fe640 1 --2- 172.21.15.37:0/1904292319 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a8137a10 0x7ff39000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:41.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a67fc640 10 monclient(hunting): get_auth_request con 0x7ff390004810 auth_method 0 2024-04-26T00:36:41.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:41.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a67fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:41.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a67fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:36:41.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a77fe640 10 monclient(hunting): get_auth_request con 0x7ff3a8137a10 auth_method 0 2024-04-26T00:36:41.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:41.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a77fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:41.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:36:41.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:41.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:41.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a77fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:36:41.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a6ffd640 1 --2- 172.21.15.37:0/1904292319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3900039f0 0x7ff390003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:41.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:41.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:41.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a67fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:36:41.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a6ffd640 10 monclient(hunting): get_auth_request con 0x7ff3900039f0 auth_method 0 2024-04-26T00:36:41.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:41.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:41.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:36:41.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a77fe640 10 monclient(hunting): handle_auth_done global_id 144705 payload 293 2024-04-26T00:36:41.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a77fe640 1 -- 172.21.15.37:0/1904292319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3900039f0 msgr2=0x7ff390003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:36:41.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a77fe640 1 --2- 172.21.15.37:0/1904292319 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3900039f0 0x7ff390003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:41.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a77fe640 1 -- 172.21.15.37:0/1904292319 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff390004810 msgr2=0x7ff390004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:36:41.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a77fe640 1 --2- 172.21.15.37:0/1904292319 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff390004810 0x7ff390004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:41.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a77fe640 10 monclient: _finish_hunting 0 2024-04-26T00:36:41.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a77fe640 1 monclient: found mon.c 2024-04-26T00:36:41.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:41.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a77fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:36:41.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a77fe640 1 -- 172.21.15.37:0/1904292319 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff390003530 con 0x7ff3a8137a10 2024-04-26T00:36:41.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a77fe640 10 monclient: _finish_auth 0 2024-04-26T00:36:41.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a77fe640 10 monclient: _check_auth_tickets 2024-04-26T00:36:41.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.792+0000 7ff3a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:41.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3a77fe640 1 -- 172.21.15.37:0/1904292319 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a8137a10 msgr2=0x7ff39000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:41.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3a77fe640 1 --2- 172.21.15.37:0/1904292319 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a8137a10 0x7ff39000b9d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff3a809bca0 tx=0x7ff3a001d690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b896cbf905fa2b82 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:41.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3a5ffb640 1 -- 172.21.15.37:0/1904292319 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a000a820 con 0x7ff3a8137a10 2024-04-26T00:36:41.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:41.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3a5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:36:41.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3a5ffb640 10 monclient: dump: 2024-04-26T00:36:41.807 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:41.807 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:41.807 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:41.807 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:41.807 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:41.807 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:41.808 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:41.808 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:41.808 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:41.808 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:41.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3ad57b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:36:41.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3a5ffb640 1 -- 172.21.15.37:0/1904292319 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3a000d040 con 0x7ff3a8137a10 2024-04-26T00:36:41.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3a5ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:36:41.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3a5ffb640 1 -- 172.21.15.37:0/1904292319 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a00a78c0 con 0x7ff3a8137a10 2024-04-26T00:36:41.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:41.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3a5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:36:41.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3a5ffb640 10 monclient: dump: 2024-04-26T00:36:41.808 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:41.808 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:41.808 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:41.809 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:41.809 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:41.809 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:41.809 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:41.809 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:41.809 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:41.809 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:41.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3ad57b640 10 monclient: get_monmap_and_config success 2024-04-26T00:36:41.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3ad57b640 4 set_mon_vals no callback set 2024-04-26T00:36:41.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3ad57b640 10 monclient: shutdown 2024-04-26T00:36:41.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3ad57b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:36:41.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3ad57b640 1 -- 172.21.15.37:0/1904292319 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a8137a10 msgr2=0x7ff39000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:41.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3ad57b640 1 --2- 172.21.15.37:0/1904292319 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a8137a10 0x7ff39000b9d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff3a809bca0 tx=0x7ff3a001d690 comp rx=0 tx=0).stop 2024-04-26T00:36:41.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3a77fe640 1 -- 172.21.15.37:0/1904292319 reap_dead start 2024-04-26T00:36:41.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3ad57b640 1 -- 172.21.15.37:0/1904292319 shutdown_connections 2024-04-26T00:36:41.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3ad57b640 1 -- 172.21.15.37:0/1904292319 >> 172.21.15.37:0/1904292319 conn(0x7ff3a80a61b0 msgr2=0x7ff3a80a65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:36:41.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3ad57b640 1 -- 172.21.15.37:0/1904292319 shutdown_connections 2024-04-26T00:36:41.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3ad57b640 1 -- 172.21.15.37:0/1904292319 wait complete. 2024-04-26T00:36:41.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3ad57b640 10 monclient: build_initial_monmap 2024-04-26T00:36:41.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3ad57b640 10 monclient: monmap: 2024-04-26T00:36:41.810 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:36:41.810 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:36:41.810 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:36:41.810 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:36:41.810 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:36:41.810 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:36:41.810 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:36:41.810 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:36:41.810 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:36:41.810 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:41.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3ad57b640 1 Processor -- start 2024-04-26T00:36:41.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3ad57b640 1 -- start start 2024-04-26T00:36:41.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3ad57b640 10 monclient: init 2024-04-26T00:36:41.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3ad57b640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:41.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3ad57b640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:36:41.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3ad57b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff3a81353b0 0x7ff3a8135790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:41.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3ad57b640 10 monclient(hunting): picked mon.noname-a con 0x7ff3a81353b0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:36:41.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3ad57b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3a81cbc90 0x7ff3a81cc110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:41.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3ad57b640 10 monclient(hunting): picked mon.noname-b con 0x7ff3a81cbc90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:36:41.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3ad57b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a81cc650 0x7ff3a81d6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:41.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3ad57b640 10 monclient(hunting): picked mon.noname-c con 0x7ff3a81cc650 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:36:41.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3ad57b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:41.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3ad57b640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a81408f0 con 0x7ff3a81353b0 2024-04-26T00:36:41.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3ad57b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:41.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3ad57b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a8140770 con 0x7ff3a81cbc90 2024-04-26T00:36:41.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3ad57b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:41.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3ad57b640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff3a8140a70 con 0x7ff3a81cc650 2024-04-26T00:36:41.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3ad57b640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:41.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3ad57b640 10 monclient(hunting): authenticate will time out at 1983.107733s 2024-04-26T00:36:41.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3a6ffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff3a81353b0 0x7ff3a8135790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:41.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3a77fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a81cc650 0x7ff3a81d6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:41.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3a6ffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff3a81353b0 0x7ff3a8135790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50300/0 (socket says 172.21.15.37:50300) 2024-04-26T00:36:41.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3a6ffd640 1 -- 172.21.15.37:0/1445358479 learned_addr learned my addr 172.21.15.37:0/1445358479 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:36:41.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3a6ffd640 10 monclient(hunting): get_auth_request con 0x7ff3a81353b0 auth_method 0 2024-04-26T00:36:41.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:41.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3a6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:41.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:41.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3a77fe640 10 monclient(hunting): get_auth_request con 0x7ff3a81cc650 auth_method 0 2024-04-26T00:36:41.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:41.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3a77fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:41.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:41.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3a6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:41.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3a6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:41.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.796+0000 7ff3a6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:41.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff3a77fe640 1 -- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a81cc650 msgr2=0x7ff3a81d6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:36:41.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff3a77fe640 1 -- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a81cc650 msgr2=0x7ff3a81d6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:36:41.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff3a77fe640 1 --2- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a81cc650 0x7ff3a81d6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:36:41.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff3a77fe640 1 --2- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a81cc650 0x7ff3a81d6540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:36:41.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff3a6ffd640 10 monclient(hunting): handle_auth_done global_id 144708 payload 824 2024-04-26T00:36:41.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff3a67fc640 1 --2- 172.21.15.37:0/1445358479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3a81cbc90 0x7ff3a81cc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:41.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff3a6ffd640 1 -- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a81cc650 msgr2=0x7ff3a81d6540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:36:41.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff3a6ffd640 1 --2- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a81cc650 0x7ff3a81d6540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:41.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff3a6ffd640 1 -- 172.21.15.37:0/1445358479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3a81cbc90 msgr2=0x7ff3a81cc110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:41.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff3a6ffd640 1 --2- 172.21.15.37:0/1445358479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3a81cbc90 0x7ff3a81cc110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:41.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff3a6ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:36:41.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff3a6ffd640 1 monclient: found mon.noname-a 2024-04-26T00:36:41.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff3a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:41.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff3a6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:36:41.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff3a6ffd640 1 -- 172.21.15.37:0/1445358479 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3a81ccad0 con 0x7ff3a81353b0 2024-04-26T00:36:41.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff3a67fc640 10 monclient: get_auth_request con 0x7ff3a81cbc90 auth_method 0 2024-04-26T00:36:41.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff3a67fc640 1 --2- 172.21.15.37:0/1445358479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff3a81cbc90 0x7ff3a81cc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:36:41.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff3a6ffd640 1 -- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff3a81353b0 msgr2=0x7ff3a8135790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:41.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff3a6ffd640 1 --2- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff3a81353b0 0x7ff3a8135790 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff39800e470 tx=0x7ff39800e930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:41.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff38f7fe640 1 -- 172.21.15.37:0/1445358479 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3980074f0 con 0x7ff3a81353b0 2024-04-26T00:36:41.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff38f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:41.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff38f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:36:41.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff38f7fe640 10 monclient: dump: 2024-04-26T00:36:41.815 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:41.815 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:41.815 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:41.815 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:41.815 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:41.815 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:41.815 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:41.815 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:41.815 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:41.815 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:41.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff38f7fe640 10 monclient: _finish_auth 0 2024-04-26T00:36:41.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff38f7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:36:41.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff38f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:41.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff38f7fe640 1 -- 172.21.15.37:0/1445358479 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff398018070 con 0x7ff3a81353b0 2024-04-26T00:36:41.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff3ad57b640 5 monclient: authenticate success, global_id 144708 2024-04-26T00:36:41.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff3ad57b640 10 monclient: _renew_subs 2024-04-26T00:36:41.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff3ad57b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:41.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff3ad57b640 1 -- 172.21.15.37:0/1445358479 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3a8120b70 con 0x7ff3a81353b0 2024-04-26T00:36:41.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff38f7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:36:41.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff38f7fe640 1 -- 172.21.15.37:0/1445358479 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff398005a10 con 0x7ff3a81353b0 2024-04-26T00:36:41.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff38f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:41.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff38f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:36:41.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff38f7fe640 10 monclient: dump: 2024-04-26T00:36:41.816 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:41.816 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:41.816 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:41.816 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:41.816 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:41.817 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:41.817 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:41.817 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:41.817 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:41.817 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:41.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff3a4ff9640 4 set_mon_vals no callback set 2024-04-26T00:36:41.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff3ad57b640 10 monclient: _renew_subs 2024-04-26T00:36:41.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff3ad57b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:41.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff3ad57b640 1 -- 172.21.15.37:0/1445358479 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3a8136150 con 0x7ff3a81353b0 2024-04-26T00:36:41.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff3ad57b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:41.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff3ad57b640 1 --2- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a802d180 0x7ff3a80623f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:41.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff3ad57b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:41.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff3ad57b640 1 -- 172.21.15.37:0/1445358479 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff3a809aea0 con 0x7ff3a802d180 2024-04-26T00:36:41.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff3a67fc640 1 --2- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a802d180 0x7ff3a80623f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:41.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff3a67fc640 10 monclient: get_auth_request con 0x7ff3a802d180 auth_method 0 2024-04-26T00:36:41.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff3a67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:41.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff3a67fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:41.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff3a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:41.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff3a67fc640 1 -- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a802d180 msgr2=0x7ff3a80623f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:36:41.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff3a67fc640 1 -- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a802d180 msgr2=0x7ff3a80623f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:36:41.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff3a67fc640 1 --2- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a802d180 0x7ff3a80623f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:36:41.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff38f7fe640 1 -- 172.21.15.37:0/1445358479 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff398005bb0 con 0x7ff3a81353b0 2024-04-26T00:36:41.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff3a67fc640 1 --2- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a802d180 0x7ff3a80623f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:36:41.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff38f7fe640 1 --2- 172.21.15.37:0/1445358479 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff394035e80 0x7ff394038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:41.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.800+0000 7ff38f7fe640 1 -- 172.21.15.37:0/1445358479 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(656..656 src has 370..656) ==== 3899+0+0 (secure 0 0 0) 0x7ff3980487c0 con 0x7ff3a81353b0 2024-04-26T00:36:41.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.804+0000 7ff3a77fe640 1 --2- 172.21.15.37:0/1445358479 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff394035e80 0x7ff394038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:41.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.804+0000 7ff3a77fe640 10 monclient: get_auth_request con 0x7ff394035e80 auth_method 0 2024-04-26T00:36:41.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:41.804+0000 7ff3a77fe640 1 --2- 172.21.15.37:0/1445358479 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff394035e80 0x7ff394038340 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0002240 tx=0x7ff3a00b2040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:42.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:42.004+0000 7ff3a67fc640 1 --2- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a802d180 0x7ff3a80623f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:42.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:42.004+0000 7ff3a67fc640 10 monclient: get_auth_request con 0x7ff3a802d180 auth_method 0 2024-04-26T00:36:42.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:42.004+0000 7ff3a67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:42.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:42.004+0000 7ff3a67fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:42.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:42.004+0000 7ff3a67fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:36:42.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:42.004+0000 7ff3a67fc640 1 -- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a802d180 msgr2=0x7ff3a80623f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:36:42.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:42.004+0000 7ff3a67fc640 1 -- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a802d180 msgr2=0x7ff3a80623f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:36:42.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:42.004+0000 7ff3a67fc640 1 --2- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a802d180 0x7ff3a80623f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:36:42.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:42.004+0000 7ff3a67fc640 1 --2- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a802d180 0x7ff3a80623f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:36:42.081 INFO:tasks.rados.rados.0.smithi045.stdout:2894: done (2 left) 2024-04-26T00:36:42.082 INFO:tasks.rados.rados.0.smithi045.stdout:2895: done (1 left) 2024-04-26T00:36:42.082 INFO:tasks.rados.rados.0.smithi045.stdout:2896: done (0 left) 2024-04-26T00:36:42.082 INFO:tasks.rados.rados.0.smithi045.stdout:2897: rollback oid 32 current snap is 315 2024-04-26T00:36:42.082 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 32 to 314 2024-04-26T00:36:42.082 INFO:tasks.rados.rados.0.smithi045.stdout:2898: write oid 28 current snap is 315 2024-04-26T00:36:42.082 INFO:tasks.rados.rados.0.smithi045.stdout:2898: seq_num 1041 ranges {696087=558217,1833755=648359,3137806=358281} 2024-04-26T00:36:42.095 INFO:tasks.rados.rados.0.smithi045.stdout:2898: writing smithi04526579-28 from 696087 to 1254304 tid 1 2024-04-26T00:36:42.107 INFO:tasks.rados.rados.0.smithi045.stdout:2898: writing smithi04526579-28 from 1833755 to 2482114 tid 2 2024-04-26T00:36:42.118 INFO:tasks.rados.rados.0.smithi045.stdout:2898: writing smithi04526579-28 from 3137806 to 3496087 tid 3 2024-04-26T00:36:42.118 INFO:tasks.rados.rados.0.smithi045.stdout:2899: snap_create 2024-04-26T00:36:42.118 INFO:tasks.rados.rados.0.smithi045.stdout:2897: finishing rollback tid 0 to smithi04526579-32 2024-04-26T00:36:42.118 INFO:tasks.rados.rados.0.smithi045.stdout:2897: finishing rollback tid 1 to smithi04526579-32 2024-04-26T00:36:42.118 INFO:tasks.rados.rados.0.smithi045.stdout:2897: finishing rollback tid 2 to smithi04526579-32 2024-04-26T00:36:42.118 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 2357 (ObjNum 999 snap 303 seq_num 999) dirty exists 2024-04-26T00:36:42.118 INFO:tasks.rados.rados.0.smithi045.stdout:2898: finishing write tid 1 to smithi04526579-28 2024-04-26T00:36:42.118 INFO:tasks.rados.rados.0.smithi045.stdout:2898: finishing write tid 2 to smithi04526579-28 2024-04-26T00:36:42.125 INFO:tasks.rados.rados.0.smithi045.stdout:2898: finishing write tid 3 to smithi04526579-28 2024-04-26T00:36:42.126 INFO:tasks.rados.rados.0.smithi045.stdout:2898: finishing write tid 4 to smithi04526579-28 2024-04-26T00:36:42.126 INFO:tasks.rados.rados.0.smithi045.stdout:2898: finishing write tid 5 to smithi04526579-28 2024-04-26T00:36:42.126 INFO:tasks.rados.rados.0.smithi045.stdout:2898: oid 28 updating version 0 to 2019 2024-04-26T00:36:42.126 INFO:tasks.rados.rados.0.smithi045.stdout:2898: oid 28 updating version 2019 to 2021 2024-04-26T00:36:42.127 INFO:tasks.rados.rados.0.smithi045.stdout:2898: oid 28 updating version 2021 to 2022 2024-04-26T00:36:42.127 INFO:tasks.rados.rados.0.smithi045.stdout:2898: oid 28 version 2022 is already newer than 2020 2024-04-26T00:36:42.127 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 2022 (ObjNum 1041 snap 315 seq_num 1041) dirty exists 2024-04-26T00:36:42.127 INFO:tasks.rados.rados.0.smithi045.stdout:2898: left oid 28 (ObjNum 1041 snap 315 seq_num 1041) 2024-04-26T00:36:42.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:42.408+0000 7ff3a67fc640 1 --2- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a802d180 0x7ff3a80623f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:42.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:42.408+0000 7ff3a67fc640 10 monclient: get_auth_request con 0x7ff3a802d180 auth_method 0 2024-04-26T00:36:42.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:42.408+0000 7ff3a67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:42.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:42.408+0000 7ff3a67fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:42.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:42.408+0000 7ff3a67fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:36:42.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:42.408+0000 7ff3a67fc640 1 -- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a802d180 msgr2=0x7ff3a80623f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:36:42.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:42.408+0000 7ff3a67fc640 1 -- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a802d180 msgr2=0x7ff3a80623f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:36:42.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:42.408+0000 7ff3a67fc640 1 --2- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a802d180 0x7ff3a80623f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:36:42.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:42.408+0000 7ff3a67fc640 1 --2- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a802d180 0x7ff3a80623f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:36:43.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:43.212+0000 7ff3a67fc640 1 --2- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a802d180 0x7ff3a80623f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:43.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:43.212+0000 7ff3a67fc640 10 monclient: get_auth_request con 0x7ff3a802d180 auth_method 0 2024-04-26T00:36:43.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:43.212+0000 7ff3a67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:43.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:43.212+0000 7ff3a67fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:43.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:43.212+0000 7ff3a67fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:36:43.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:43.212+0000 7ff3a67fc640 1 -- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a802d180 msgr2=0x7ff3a80623f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:36:43.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:43.212+0000 7ff3a67fc640 1 -- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a802d180 msgr2=0x7ff3a80623f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:36:43.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:43.212+0000 7ff3a67fc640 1 --2- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a802d180 0x7ff3a80623f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:36:43.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:43.212+0000 7ff3a67fc640 1 --2- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a802d180 0x7ff3a80623f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:36:43.676 INFO:tasks.rados.rados.0.smithi045.stdout:2897: done (2 left) 2024-04-26T00:36:43.676 INFO:tasks.rados.rados.0.smithi045.stdout:2898: done (1 left) 2024-04-26T00:36:43.676 INFO:tasks.rados.rados.0.smithi045.stdout:2899: done (0 left) 2024-04-26T00:36:43.676 INFO:tasks.rados.rados.0.smithi045.stdout:2900: delete oid 9 current snap is 316 2024-04-26T00:36:43.691 INFO:tasks.rados.rados.0.smithi045.stdout:2900: done (0 left) 2024-04-26T00:36:43.692 INFO:tasks.rados.rados.0.smithi045.stdout:2901: copy_from oid 21 from oid 42 current snap is 316 2024-04-26T00:36:43.693 INFO:tasks.rados.rados.0.smithi045.stdout:2902: write (excl) oid 13 current snap is 316 2024-04-26T00:36:43.693 INFO:tasks.rados.rados.0.smithi045.stdout:2902: seq_num 1043 ranges {729701=425068,1667072=652028,2909005=485180} 2024-04-26T00:36:43.708 INFO:tasks.rados.rados.0.smithi045.stdout:2902: writing smithi04526579-13 from 729701 to 1154769 tid 1 2024-04-26T00:36:43.723 INFO:tasks.rados.rados.0.smithi045.stdout:2902: writing smithi04526579-13 from 1667072 to 2319100 tid 2 2024-04-26T00:36:43.738 INFO:tasks.rados.rados.0.smithi045.stdout:2902: writing smithi04526579-13 from 2909005 to 3394185 tid 3 2024-04-26T00:36:43.738 INFO:tasks.rados.rados.0.smithi045.stdout:2903: write (excl) oid 18 current snap is 316 2024-04-26T00:36:43.738 INFO:tasks.rados.rados.0.smithi045.stdout:2903: seq_num 1044 ranges {746508=500317,2009201=795686,3220075=326433} 2024-04-26T00:36:43.753 INFO:tasks.rados.rados.0.smithi045.stdout:2903: writing smithi04526579-18 from 746508 to 1246825 tid 1 2024-04-26T00:36:43.768 INFO:tasks.rados.rados.0.smithi045.stdout:2903: writing smithi04526579-18 from 2009201 to 2804887 tid 2 2024-04-26T00:36:43.775 INFO:tasks.rados.rados.0.smithi045.stdout:2903: writing smithi04526579-18 from 3220075 to 3546508 tid 3 2024-04-26T00:36:43.775 INFO:tasks.rados.rados.0.smithi045.stdout:2904: read oid 23 snap -1 2024-04-26T00:36:43.775 INFO:tasks.rados.rados.0.smithi045.stdout:2904: expect (ObjNum 945 snap 292 seq_num 945) 2024-04-26T00:36:43.776 INFO:tasks.rados.rados.0.smithi045.stdout:2905: rollback oid 19 current snap is 316 2024-04-26T00:36:43.776 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 19 to 314 2024-04-26T00:36:43.776 INFO:tasks.rados.rados.0.smithi045.stdout:2902: finishing write tid 1 to smithi04526579-13 2024-04-26T00:36:43.776 INFO:tasks.rados.rados.0.smithi045.stdout:2901: finishing copy_from racing read to smithi04526579-21 2024-04-26T00:36:43.776 INFO:tasks.rados.rados.0.smithi045.stdout:2901: finishing copy_from to smithi04526579-21 2024-04-26T00:36:43.776 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 2354 (ObjNum 903 snap 277 seq_num 903) dirty exists 2024-04-26T00:36:43.776 INFO:tasks.rados.rados.0.smithi045.stdout:2902: finishing write tid 2 to smithi04526579-13 2024-04-26T00:36:43.776 INFO:tasks.rados.rados.0.smithi045.stdout:2902: finishing write tid 3 to smithi04526579-13 2024-04-26T00:36:43.776 INFO:tasks.rados.rados.0.smithi045.stdout:2902: finishing write tid 4 to smithi04526579-13 2024-04-26T00:36:43.776 INFO:tasks.rados.rados.0.smithi045.stdout:2902: finishing write tid 5 to smithi04526579-13 2024-04-26T00:36:43.776 INFO:tasks.rados.rados.0.smithi045.stdout:2902: oid 13 updating version 0 to 2191 2024-04-26T00:36:43.776 INFO:tasks.rados.rados.0.smithi045.stdout:2902: oid 13 updating version 2191 to 2192 2024-04-26T00:36:43.776 INFO:tasks.rados.rados.0.smithi045.stdout:2902: oid 13 version 2192 is already newer than 2190 2024-04-26T00:36:43.776 INFO:tasks.rados.rados.0.smithi045.stdout:2902: oid 13 updating version 2192 to 2193 2024-04-26T00:36:43.777 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 2193 (ObjNum 1043 snap 316 seq_num 1043) dirty exists 2024-04-26T00:36:43.777 INFO:tasks.rados.rados.0.smithi045.stdout:2902: left oid 13 (ObjNum 1043 snap 316 seq_num 1043) 2024-04-26T00:36:43.777 INFO:tasks.rados.rados.0.smithi045.stdout:2903: finishing write tid 1 to smithi04526579-18 2024-04-26T00:36:43.777 INFO:tasks.rados.rados.0.smithi045.stdout:2901: done (4 left) 2024-04-26T00:36:43.777 INFO:tasks.rados.rados.0.smithi045.stdout:2902: done (3 left) 2024-04-26T00:36:43.777 INFO:tasks.rados.rados.0.smithi045.stdout:2906: snap_create 2024-04-26T00:36:43.777 INFO:tasks.rados.rados.0.smithi045.stdout:2905: finishing rollback tid 1 to smithi04526579-19 2024-04-26T00:36:43.777 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 0 (ObjNum 129 snap 0 seq_num 2240327888) dirty dne 2024-04-26T00:36:43.786 INFO:tasks.rados.rados.0.smithi045.stdout:2903: finishing write tid 2 to smithi04526579-18 2024-04-26T00:36:43.787 INFO:tasks.rados.rados.0.smithi045.stdout:2903: finishing write tid 3 to smithi04526579-18 2024-04-26T00:36:43.787 INFO:tasks.rados.rados.0.smithi045.stdout:2903: finishing write tid 4 to smithi04526579-18 2024-04-26T00:36:43.787 INFO:tasks.rados.rados.0.smithi045.stdout:2903: finishing write tid 5 to smithi04526579-18 2024-04-26T00:36:43.787 INFO:tasks.rados.rados.0.smithi045.stdout:2903: oid 18 updating version 0 to 2393 2024-04-26T00:36:43.787 INFO:tasks.rados.rados.0.smithi045.stdout:2903: oid 18 version 2393 is already newer than 2392 2024-04-26T00:36:43.787 INFO:tasks.rados.rados.0.smithi045.stdout:2903: oid 18 updating version 2393 to 2394 2024-04-26T00:36:43.787 INFO:tasks.rados.rados.0.smithi045.stdout:2903: oid 18 version 2394 is already newer than 2391 2024-04-26T00:36:43.787 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 2394 (ObjNum 1044 snap 316 seq_num 1044) dirty exists 2024-04-26T00:36:43.787 INFO:tasks.rados.rados.0.smithi045.stdout:2903: left oid 18 (ObjNum 1044 snap 316 seq_num 1044) 2024-04-26T00:36:43.788 INFO:tasks.rados.rados.0.smithi045.stdout:2904: expect (ObjNum 945 snap 292 seq_num 945) 2024-04-26T00:36:44.561 INFO:tasks.rados.rados.0.smithi045.stdout:2903: done (3 left) 2024-04-26T00:36:44.561 INFO:tasks.rados.rados.0.smithi045.stdout:2904: done (2 left) 2024-04-26T00:36:44.561 INFO:tasks.rados.rados.0.smithi045.stdout:2905: done (1 left) 2024-04-26T00:36:44.561 INFO:tasks.rados.rados.0.smithi045.stdout:2906: done (0 left) 2024-04-26T00:36:44.561 INFO:tasks.rados.rados.0.smithi045.stdout:2907: write oid 39 current snap is 317 2024-04-26T00:36:44.561 INFO:tasks.rados.rados.0.smithi045.stdout:2907: seq_num 1045 ranges {763315=575566,2067683=655697,3247498=315817} 2024-04-26T00:36:44.573 INFO:tasks.rados.rados.0.smithi045.stdout:2907: writing smithi04526579-39 from 763315 to 1338881 tid 1 2024-04-26T00:36:44.586 INFO:tasks.rados.rados.0.smithi045.stdout:2907: writing smithi04526579-39 from 2067683 to 2723380 tid 2 2024-04-26T00:36:44.594 INFO:tasks.rados.rados.0.smithi045.stdout:2907: writing smithi04526579-39 from 3247498 to 3563315 tid 3 2024-04-26T00:36:44.594 INFO:tasks.rados.rados.0.smithi045.stdout:2908: rollback oid 11 current snap is 317 2024-04-26T00:36:44.594 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 11 to 314 2024-04-26T00:36:44.595 INFO:tasks.rados.rados.0.smithi045.stdout:2907: finishing write tid 1 to smithi04526579-39 2024-04-26T00:36:44.595 INFO:tasks.rados.rados.0.smithi045.stdout:2909: write (excl) oid 9 current snap is 317 2024-04-26T00:36:44.595 INFO:tasks.rados.rados.0.smithi045.stdout:2909: seq_num 1046 ranges {780122=650815,2126165=799355} 2024-04-26T00:36:44.609 INFO:tasks.rados.rados.0.smithi045.stdout:2909: writing smithi04526579-9 from 780122 to 1430937 tid 1 2024-04-26T00:36:44.623 INFO:tasks.rados.rados.0.smithi045.stdout:2909: writing smithi04526579-9 from 2126165 to 2925520 tid 2 2024-04-26T00:36:44.623 INFO:tasks.rados.rados.0.smithi045.stdout:2910: copy_from oid 32 from oid 49 current snap is 317 2024-04-26T00:36:44.623 INFO:tasks.rados.rados.0.smithi045.stdout:2907: finishing write tid 2 to smithi04526579-39 2024-04-26T00:36:44.623 INFO:tasks.rados.rados.0.smithi045.stdout:2907: finishing write tid 3 to smithi04526579-39 2024-04-26T00:36:44.623 INFO:tasks.rados.rados.0.smithi045.stdout:2907: finishing write tid 4 to smithi04526579-39 2024-04-26T00:36:44.623 INFO:tasks.rados.rados.0.smithi045.stdout:2911: read oid 17 snap -1 2024-04-26T00:36:44.623 INFO:tasks.rados.rados.0.smithi045.stdout:2911: expect (ObjNum 1033 snap 312 seq_num 1033) 2024-04-26T00:36:44.623 INFO:tasks.rados.rados.0.smithi045.stdout:2912: write oid 16 current snap is 317 2024-04-26T00:36:44.623 INFO:tasks.rados.rados.0.smithi045.stdout:2912: seq_num 1047 ranges {796929=726064,2184647=659366,3302344=294585} 2024-04-26T00:36:44.635 INFO:tasks.rados.rados.0.smithi045.stdout:2912: writing smithi04526579-16 from 796929 to 1522993 tid 1 2024-04-26T00:36:44.646 INFO:tasks.rados.rados.0.smithi045.stdout:2912: writing smithi04526579-16 from 2184647 to 2844013 tid 2 2024-04-26T00:36:44.652 INFO:tasks.rados.rados.0.smithi045.stdout:2912: writing smithi04526579-16 from 3302344 to 3596929 tid 3 2024-04-26T00:36:44.652 INFO:tasks.rados.rados.0.smithi045.stdout:2913: snap_remove snap 313 2024-04-26T00:36:44.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.796+0000 7ff38e7fc640 10 monclient: tick 2024-04-26T00:36:44.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.796+0000 7ff38e7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:36:44.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.796+0000 7ff38e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:44.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.796+0000 7ff38e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:44.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.816+0000 7ff3a67fc640 1 --2- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a802d180 0x7ff3a80623f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:44.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.816+0000 7ff3a67fc640 10 monclient: get_auth_request con 0x7ff3a802d180 auth_method 0 2024-04-26T00:36:44.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.816+0000 7ff3a67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:44.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.816+0000 7ff3a67fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:44.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.816+0000 7ff3a67fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:36:44.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.816+0000 7ff3a67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:44.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.816+0000 7ff3a67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:44.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.816+0000 7ff3a67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:44.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.816+0000 7ff3a67fc640 10 monclient(hunting): handle_auth_done global_id 204101 payload 824 2024-04-26T00:36:44.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.816+0000 7ff3a67fc640 1 -- 172.21.15.37:0/1445358479 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff3a802d560 con 0x7ff3a802d180 2024-04-26T00:36:44.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.816+0000 7ff3a67fc640 1 -- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a802d180 msgr2=0x7ff3a80623f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:44.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.816+0000 7ff3a67fc640 1 --2- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a802d180 0x7ff3a80623f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff39c000ee0 tx=0x7ff39c0197d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:44.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.820+0000 7ff38f7fe640 1 -- 172.21.15.37:0/1445358479 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff39c00f040 con 0x7ff3a802d180 2024-04-26T00:36:44.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.820+0000 7ff38f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:44.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.820+0000 7ff38f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:36:44.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.820+0000 7ff38f7fe640 10 monclient: dump: 2024-04-26T00:36:44.819 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:44.819 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:44.819 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:44.819 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:44.819 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:44.819 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:44.819 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:44.819 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:44.819 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:44.819 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:44.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.820+0000 7ff38f7fe640 1 -- 172.21.15.37:0/1445358479 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff39c017810 con 0x7ff3a802d180 2024-04-26T00:36:44.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.820+0000 7ff38f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:44.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.820+0000 7ff38f7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:36:44.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.820+0000 7ff38f7fe640 1 -- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a802d180 msgr2=0x7ff3a80623f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:44.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.820+0000 7ff38f7fe640 1 --2- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a802d180 0x7ff3a80623f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff39c000ee0 tx=0x7ff39c0197d0 comp rx=0 tx=0).stop 2024-04-26T00:36:44.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.820+0000 7ff38f7fe640 1 -- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a802d180 msgr2=0x7ff3a80623f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:36:44.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.820+0000 7ff38f7fe640 1 --2- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a802d180 0x7ff3a80623f0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:44.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.828+0000 7ff3ad57b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:36:44.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.828+0000 7ff3ad57b640 1 --2- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a8062930 0x7ff3a81daaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:44.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.828+0000 7ff3ad57b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:44.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.828+0000 7ff3ad57b640 1 -- 172.21.15.37:0/1445358479 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff3a804f510 con 0x7ff3a8062930 2024-04-26T00:36:44.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.828+0000 7ff3a67fc640 1 --2- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a8062930 0x7ff3a81daaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:44.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.828+0000 7ff3a67fc640 10 monclient: get_auth_request con 0x7ff3a8062930 auth_method 0 2024-04-26T00:36:44.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.828+0000 7ff3a67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:44.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.828+0000 7ff3a67fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:44.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.828+0000 7ff3a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:44.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.828+0000 7ff3a67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:44.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.828+0000 7ff3a67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:44.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.828+0000 7ff3a67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:44.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.828+0000 7ff3a67fc640 10 monclient(hunting): handle_auth_done global_id 204104 payload 824 2024-04-26T00:36:44.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.828+0000 7ff3a67fc640 1 -- 172.21.15.37:0/1445358479 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff3a80a2b40 con 0x7ff3a8062930 2024-04-26T00:36:44.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.828+0000 7ff3a67fc640 1 -- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a8062930 msgr2=0x7ff3a81daaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:44.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.828+0000 7ff3a67fc640 1 --2- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a8062930 0x7ff3a81daaa0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff39c000ee0 tx=0x7ff39c004740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:44.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.832+0000 7ff38f7fe640 1 -- 172.21.15.37:0/1445358479 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff39c004030 con 0x7ff3a8062930 2024-04-26T00:36:44.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.832+0000 7ff38f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:44.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.832+0000 7ff38f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:36:44.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.832+0000 7ff38f7fe640 10 monclient: dump: 2024-04-26T00:36:44.831 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:44.831 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:44.831 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:44.831 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:44.831 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:44.831 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:44.831 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:44.831 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:44.831 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:44.831 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:44.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.832+0000 7ff38f7fe640 1 -- 172.21.15.37:0/1445358479 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4547 (secure 0 0 0) 0x7ff39c029040 con 0x7ff3a8062930 2024-04-26T00:36:44.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.832+0000 7ff38f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:36:44.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.832+0000 7ff38f7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:36:44.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.832+0000 7ff38f7fe640 1 -- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a8062930 msgr2=0x7ff3a81daaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:44.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.832+0000 7ff38f7fe640 1 --2- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a8062930 0x7ff3a81daaa0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff39c000ee0 tx=0x7ff39c004740 comp rx=0 tx=0).stop 2024-04-26T00:36:44.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.832+0000 7ff38f7fe640 1 -- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a8062930 msgr2=0x7ff3a81daaa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:36:44.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.832+0000 7ff38f7fe640 1 --2- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff3a8062930 0x7ff3a81daaa0 secure :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff39c000ee0 tx=0x7ff39c004740 comp rx=0 tx=0).stop 2024-04-26T00:36:44.833 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:36:44.833 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:36:44.833 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:36:44.833 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:36:44.833 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 126, 2024-04-26T00:36:44.833 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 10439, 2024-04-26T00:36:44.833 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:36:44.833 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:36:44.833 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:36:44.833 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:36:44.834 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:36:44.834 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 4, 2024-04-26T00:36:44.834 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:36:44.834 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:36:44.834 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:36:44.834 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:36:44.834 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:36:44.834 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:36:44.835 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:36:44.835 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:36:44.835 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:36:44.835 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:36:44.835 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:36:44.835 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:36:44.835 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:36:44.835 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:36:44.835 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:36:44.835 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:36:44.835 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:36:44.835 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:36:44.835 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:36:44.835 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:36:44.835 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:36:44.836 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:36:44.836 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:36:44.836 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:36:44.836 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:36:44.836 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:36:44.836 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:36:44.836 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:36:44.836 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:44.836 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:44.836 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:36:44.836 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:36:44.836 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:36:44.836 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:36:44.836 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:36:44.836 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:36:44.837 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:36:44.837 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:36:44.837 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:36:44.837 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:36:44.837 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:36:44.837 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:36:44.837 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:36:44.837 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:36:44.837 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:36:44.837 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:36:44.837 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:36:44.837 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:36:44.837 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:36:44.837 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:36:44.838 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:36:44.838 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:36:44.838 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:36:44.838 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:36:44.838 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:36:44.838 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:36:44.838 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:36:44.838 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:36:44.838 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:36:44.838 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:36:44.838 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:44.838 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:36:44.838 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:44.838 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:36:44.839 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:36:44.839 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:36:44.839 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:36:44.839 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:44.839 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:36:44.839 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:36:44.839 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:44.839 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:44.839 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:44.839 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:36:44.839 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:36:44.839 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:44.839 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:44.839 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:44.839 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:44.840 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:36:44.840 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:36:44.840 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:36:44.840 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:36:44.840 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:36:44.840 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:44.840 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:44.840 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:36:44.840 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:36:44.840 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:36:44.840 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:36:44.840 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:44.840 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:36:44.840 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:36:44.840 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:44.841 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:44.841 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:44.841 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:36:44.841 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:36:44.841 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:44.841 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:44.841 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:44.841 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:44.841 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:36:44.841 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:36:44.841 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:36:44.841 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:36:44.841 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:36:44.841 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:44.842 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:44.842 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:36:44.842 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:36:44.842 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:36:44.842 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:36:44.842 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:44.842 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:36:44.842 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:36:44.842 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:44.842 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:44.842 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:44.842 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:36:44.842 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:36:44.842 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:44.842 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:44.843 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:44.843 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:44.843 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:36:44.843 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:36:44.843 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:36:44.843 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:36:44.843 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:36:44.843 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:44.843 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:44.843 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:44.843 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:36:44.843 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:36:44.843 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:44.843 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:36:44.843 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:36:44.844 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:36:44.844 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:44.844 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:36:44.844 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:36:44.844 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:44.844 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:36:44.844 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:36:44.844 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:36:44.844 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:44.845 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:44.845 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:44.845 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:36:44.845 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:36:44.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.832+0000 7ff3ad57b640 1 -- 172.21.15.37:0/1445358479 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff394035e80 msgr2=0x7ff394038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:44.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.832+0000 7ff3ad57b640 1 --2- 172.21.15.37:0/1445358479 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff394035e80 0x7ff394038340 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0002240 tx=0x7ff3a00b2040 comp rx=0 tx=0).stop 2024-04-26T00:36:44.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.832+0000 7ff3ad57b640 10 monclient: shutdown 2024-04-26T00:36:44.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.832+0000 7ff3ad57b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:36:44.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.832+0000 7ff3ad57b640 1 -- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff3a81353b0 msgr2=0x7ff3a8135790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:44.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.832+0000 7ff3ad57b640 1 --2- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff3a81353b0 0x7ff3a8135790 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff39800e470 tx=0x7ff39800e930 comp rx=0 tx=0).stop 2024-04-26T00:36:44.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.832+0000 7ff3a77fe640 1 -- 172.21.15.37:0/1445358479 reap_dead start 2024-04-26T00:36:44.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.832+0000 7ff3ad57b640 1 -- 172.21.15.37:0/1445358479 shutdown_connections 2024-04-26T00:36:44.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.832+0000 7ff3ad57b640 1 --2- 172.21.15.37:0/1445358479 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff3a81353b0 0x7ff3a8135790 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:44.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.832+0000 7ff3ad57b640 1 -- 172.21.15.37:0/1445358479 >> 172.21.15.37:0/1445358479 conn(0x7ff3a80a61b0 msgr2=0x7ff3a80a8150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:36:44.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.832+0000 7ff3ad57b640 1 -- 172.21.15.37:0/1445358479 shutdown_connections 2024-04-26T00:36:44.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.832+0000 7ff3ad57b640 1 -- 172.21.15.37:0/1445358479 wait complete. 2024-04-26T00:36:44.859 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:36:44.859 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:36:44.859 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:36:44.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b39060640 10 monclient: get_monmap_and_config 2024-04-26T00:36:44.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b39060640 10 monclient: build_initial_monmap 2024-04-26T00:36:44.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b39060640 10 monclient: monmap: 2024-04-26T00:36:44.991 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:36:44.991 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:44.991 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:36:44.996246+0000 2024-04-26T00:36:44.991 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:36:44.996246+0000 2024-04-26T00:36:44.992 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:36:44.992 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:44.992 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:36:44.992 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:36:44.992 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:36:44.992 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:36:44.992 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:36:44.992 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:44.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b39060640 1 Processor -- start 2024-04-26T00:36:44.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b39060640 1 -- start start 2024-04-26T00:36:44.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b39060640 10 monclient: init 2024-04-26T00:36:44.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b39060640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:44.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b39060640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-04-26T00:36:44.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b39060640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9b340a0300 0x7f9b3409fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:44.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b39060640 10 monclient(hunting): picked mon.noname-c con 0x7f9b340a0300 addr v2:172.21.15.45:3300/0 2024-04-26T00:36:44.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b39060640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9b340a06e0 0x7f9b34134ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:44.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b39060640 10 monclient(hunting): picked mon.noname-b con 0x7f9b340a06e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:36:44.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b39060640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9b341350e0 0x7f9b34135570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:44.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b39060640 10 monclient(hunting): picked mon.noname-a con 0x7f9b341350e0 addr v2:172.21.15.37:3300/0 2024-04-26T00:36:44.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b39060640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:44.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b39060640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f9b340a13f0 con 0x7f9b341350e0 2024-04-26T00:36:44.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b39060640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:44.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b39060640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f9b3409c190 con 0x7f9b340a0300 2024-04-26T00:36:44.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b39060640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:44.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b39060640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9b340a1110 con 0x7f9b340a06e0 2024-04-26T00:36:44.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b39060640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:44.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b39060640 10 monclient(hunting): authenticate will time out at 1986.302064s 2024-04-26T00:36:44.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b3259c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9b340a06e0 0x7f9b34134ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:44.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b3359e640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9b341350e0 0x7f9b34135570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:44.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b3259c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9b340a06e0 0x7f9b34134ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:59462/0 (socket says 172.21.15.37:59462) 2024-04-26T00:36:44.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b3359e640 10 monclient(hunting): get_auth_request con 0x7f9b341350e0 auth_method 0 2024-04-26T00:36:44.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b3359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:44.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b3359e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:44.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b3259c640 1 -- 172.21.15.37:0/3682470847 learned_addr learned my addr 172.21.15.37:0/3682470847 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:36:44.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b3359e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:44.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b3259c640 10 monclient(hunting): get_auth_request con 0x7f9b340a06e0 auth_method 0 2024-04-26T00:36:44.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b3259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:44.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b3259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:44.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b3259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:44.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b3359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:44.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b3359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:44.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b3359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:44.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b3259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:44.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b3259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:44.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b3259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:44.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b32d9d640 1 --2- 172.21.15.37:0/3682470847 >> v2:172.21.15.45:3300/0 conn(0x7f9b340a0300 0x7f9b3409fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:44.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b32d9d640 10 monclient(hunting): get_auth_request con 0x7f9b340a0300 auth_method 0 2024-04-26T00:36:44.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b32d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:44.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b32d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:44.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b32d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:44.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b3359e640 10 monclient(hunting): handle_auth_done global_id 144711 payload 274 2024-04-26T00:36:44.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b3359e640 1 -- 172.21.15.37:0/3682470847 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9b340a06e0 msgr2=0x7f9b34134ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:36:44.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b3359e640 1 --2- 172.21.15.37:0/3682470847 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9b340a06e0 0x7f9b34134ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:44.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b3359e640 1 -- 172.21.15.37:0/3682470847 >> v2:172.21.15.45:3300/0 conn(0x7f9b340a0300 msgr2=0x7f9b3409fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:36:44.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b3359e640 1 --2- 172.21.15.37:0/3682470847 >> v2:172.21.15.45:3300/0 conn(0x7f9b340a0300 0x7f9b3409fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:44.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b3359e640 10 monclient: _finish_hunting 0 2024-04-26T00:36:44.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b3359e640 1 monclient: found mon.noname-a 2024-04-26T00:36:44.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b3359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:44.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b3359e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:36:44.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b3359e640 1 -- 172.21.15.37:0/3682470847 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b340a16d0 con 0x7f9b341350e0 2024-04-26T00:36:44.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.992+0000 7f9b3259c640 1 --2- 172.21.15.37:0/3682470847 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9b340a06e0 0x7f9b34134ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:36:45.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.996+0000 7f9b3359e640 1 -- 172.21.15.37:0/3682470847 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9b341350e0 msgr2=0x7f9b34135570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:45.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.996+0000 7f9b3359e640 1 --2- 172.21.15.37:0/3682470847 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9b341350e0 0x7f9b34135570 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f9b340764d0 tx=0x7f9b2c01c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd680dcd1f274771 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:45.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.996+0000 7f9b31d9b640 1 -- 172.21.15.37:0/3682470847 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b2c0b2070 con 0x7f9b341350e0 2024-04-26T00:36:45.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.996+0000 7f9b31d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:45.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.996+0000 7f9b31d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:36:45.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.996+0000 7f9b31d9b640 10 monclient: dump: 2024-04-26T00:36:45.002 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:45.002 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:45.002 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:45.002 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:45.003 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:45.003 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:45.003 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:45.003 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:45.003 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:45.003 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:45.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.996+0000 7f9b31d9b640 10 monclient: _finish_auth 0 2024-04-26T00:36:45.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.996+0000 7f9b31d9b640 10 monclient: _check_auth_tickets 2024-04-26T00:36:45.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.996+0000 7f9b31d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:45.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.996+0000 7f9b31d9b640 1 -- 172.21.15.37:0/3682470847 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b2c0a5780 con 0x7f9b341350e0 2024-04-26T00:36:45.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.996+0000 7f9b31d9b640 10 monclient: handle_config config(0 keys) 2024-04-26T00:36:45.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.996+0000 7f9b31d9b640 1 -- 172.21.15.37:0/3682470847 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b2c0a5b50 con 0x7f9b341350e0 2024-04-26T00:36:45.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.996+0000 7f9b31d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:45.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.996+0000 7f9b31d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:36:45.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.996+0000 7f9b31d9b640 10 monclient: dump: 2024-04-26T00:36:45.004 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:45.004 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:45.004 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:45.004 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:45.004 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:45.004 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:45.004 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:45.004 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:45.004 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:45.004 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:45.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.996+0000 7f9b39060640 5 monclient: authenticate success, global_id 144711 2024-04-26T00:36:45.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.996+0000 7f9b39060640 10 monclient: get_monmap_and_config success 2024-04-26T00:36:45.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.996+0000 7f9b39060640 4 set_mon_vals no callback set 2024-04-26T00:36:45.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.996+0000 7f9b39060640 10 monclient: shutdown 2024-04-26T00:36:45.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.996+0000 7f9b39060640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:36:45.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.996+0000 7f9b39060640 1 -- 172.21.15.37:0/3682470847 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9b341350e0 msgr2=0x7f9b34135570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:45.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.996+0000 7f9b39060640 1 --2- 172.21.15.37:0/3682470847 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9b341350e0 0x7f9b34135570 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f9b340764d0 tx=0x7f9b2c01c7f0 comp rx=0 tx=0).stop 2024-04-26T00:36:45.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.996+0000 7f9b39060640 1 -- 172.21.15.37:0/3682470847 shutdown_connections 2024-04-26T00:36:45.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.996+0000 7f9b39060640 1 --2- 172.21.15.37:0/3682470847 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9b341350e0 0x7f9b34135570 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:45.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.996+0000 7f9b39060640 1 --2- 172.21.15.37:0/3682470847 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9b340a06e0 0x7f9b34134ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:45.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.996+0000 7f9b39060640 1 --2- 172.21.15.37:0/3682470847 >> v2:172.21.15.45:3300/0 conn(0x7f9b340a0300 0x7f9b3409fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:45.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.996+0000 7f9b39060640 1 -- 172.21.15.37:0/3682470847 >> 172.21.15.37:0/3682470847 conn(0x7f9b340a1d50 msgr2=0x7f9b340a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:36:45.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.996+0000 7f9b39060640 1 -- 172.21.15.37:0/3682470847 shutdown_connections 2024-04-26T00:36:45.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:44.996+0000 7f9b39060640 1 -- 172.21.15.37:0/3682470847 wait complete. 2024-04-26T00:36:45.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b39060640 10 monclient: build_initial_monmap 2024-04-26T00:36:45.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b39060640 10 monclient: monmap: 2024-04-26T00:36:45.006 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:36:45.006 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:36:45.006 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:36:45.006 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:36:45.006 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:36:45.006 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:36:45.006 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:36:45.006 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:36:45.006 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:36:45.006 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:45.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b39060640 1 Processor -- start 2024-04-26T00:36:45.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b39060640 1 -- start start 2024-04-26T00:36:45.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b39060640 10 monclient: init 2024-04-26T00:36:45.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b39060640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:45.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b39060640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:36:45.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b39060640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9b340a0300 0x7f9b341cef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:45.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b39060640 10 monclient(hunting): picked mon.noname-a con 0x7f9b340a0300 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:36:45.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b39060640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b340a06e0 0x7f9b341cf530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:45.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b39060640 10 monclient(hunting): picked mon.noname-b con 0x7f9b340a06e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:36:45.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b39060640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9b341350e0 0x7f9b341cfa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:45.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b39060640 10 monclient(hunting): picked mon.noname-c con 0x7f9b341350e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:36:45.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b39060640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:45.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b39060640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9b3413e6f0 con 0x7f9b340a0300 2024-04-26T00:36:45.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b39060640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:45.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b39060640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9b3413e870 con 0x7f9b340a06e0 2024-04-26T00:36:45.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b39060640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:45.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b39060640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9b3413e9f0 con 0x7f9b341350e0 2024-04-26T00:36:45.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b39060640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:45.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b39060640 10 monclient(hunting): authenticate will time out at 1986.309467s 2024-04-26T00:36:45.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b32d9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9b340a0300 0x7f9b341cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:45.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b32d9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9b340a0300 0x7f9b341cef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50322/0 (socket says 172.21.15.37:50322) 2024-04-26T00:36:45.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b32d9d640 1 -- 172.21.15.37:0/3718511291 learned_addr learned my addr 172.21.15.37:0/3718511291 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:36:45.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b32d9d640 10 monclient(hunting): get_auth_request con 0x7f9b340a0300 auth_method 0 2024-04-26T00:36:45.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b32d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:45.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b32d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:45.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b32d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:45.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b3359e640 1 --2- 172.21.15.37:0/3718511291 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9b341350e0 0x7f9b341cfa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:45.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b3359e640 10 monclient(hunting): get_auth_request con 0x7f9b341350e0 auth_method 0 2024-04-26T00:36:45.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b3359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:45.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b3359e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:45.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b3359e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:45.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b32d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:45.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b32d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:45.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b32d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:45.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b3359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:45.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b3359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:45.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b3359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:45.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b3259c640 1 --2- 172.21.15.37:0/3718511291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b340a06e0 0x7f9b341cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:45.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b3259c640 10 monclient(hunting): get_auth_request con 0x7f9b340a06e0 auth_method 0 2024-04-26T00:36:45.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b3259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:45.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b3259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:45.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b3259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:45.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b32d9d640 10 monclient(hunting): handle_auth_done global_id 144714 payload 824 2024-04-26T00:36:45.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b32d9d640 1 -- 172.21.15.37:0/3718511291 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9b341350e0 msgr2=0x7f9b341cfa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:45.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b32d9d640 1 --2- 172.21.15.37:0/3718511291 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9b341350e0 0x7f9b341cfa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:45.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b32d9d640 1 -- 172.21.15.37:0/3718511291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b340a06e0 msgr2=0x7f9b341cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:45.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b32d9d640 1 --2- 172.21.15.37:0/3718511291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b340a06e0 0x7f9b341cf530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:45.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b32d9d640 10 monclient: _finish_hunting 0 2024-04-26T00:36:45.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b32d9d640 1 monclient: found mon.noname-a 2024-04-26T00:36:45.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b32d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:45.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b32d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:36:45.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b32d9d640 1 -- 172.21.15.37:0/3718511291 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b34139ae0 con 0x7f9b340a0300 2024-04-26T00:36:45.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.000+0000 7f9b3359e640 1 --2- 172.21.15.37:0/3718511291 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9b341350e0 0x7f9b341cfa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:36:45.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b32d9d640 1 -- 172.21.15.37:0/3718511291 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9b340a0300 msgr2=0x7f9b341cef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:45.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b32d9d640 1 --2- 172.21.15.37:0/3718511291 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9b340a0300 0x7f9b341cef00 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f9b2400d800 tx=0x7f9b2400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:45.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b1b7fe640 1 -- 172.21.15.37:0/3718511291 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b2400b8d0 con 0x7f9b340a0300 2024-04-26T00:36:45.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:45.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b1b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:36:45.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b1b7fe640 10 monclient: dump: 2024-04-26T00:36:45.014 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:45.014 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:45.014 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:45.014 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:45.014 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:45.014 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:45.014 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:45.014 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:45.014 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:45.014 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:45.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b1b7fe640 10 monclient: _finish_auth 0 2024-04-26T00:36:45.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b1b7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:36:45.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:45.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b39060640 5 monclient: authenticate success, global_id 144714 2024-04-26T00:36:45.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b1b7fe640 1 -- 172.21.15.37:0/3718511291 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b24019070 con 0x7f9b340a0300 2024-04-26T00:36:45.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b39060640 10 monclient: _renew_subs 2024-04-26T00:36:45.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b39060640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:45.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b39060640 1 -- 172.21.15.37:0/3718511291 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b340715c0 con 0x7f9b340a0300 2024-04-26T00:36:45.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b1b7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:36:45.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b1b7fe640 1 -- 172.21.15.37:0/3718511291 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b2400bc70 con 0x7f9b340a0300 2024-04-26T00:36:45.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:45.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b1b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:36:45.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b1b7fe640 10 monclient: dump: 2024-04-26T00:36:45.016 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:45.016 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:45.016 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:45.016 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:45.016 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:45.016 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:45.016 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:45.016 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:45.016 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:45.016 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:45.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b39060640 10 monclient: _renew_subs 2024-04-26T00:36:45.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b39060640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:45.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b39060640 1 -- 172.21.15.37:0/3718511291 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b3413b720 con 0x7f9b340a0300 2024-04-26T00:36:45.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b3159a640 4 set_mon_vals no callback set 2024-04-26T00:36:45.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b39060640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:45.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b39060640 1 --2- 172.21.15.37:0/3718511291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b3409c750 0x7f9b3409cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:45.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b39060640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:45.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b39060640 1 -- 172.21.15.37:0/3718511291 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9b340a0cd0 con 0x7f9b3409c750 2024-04-26T00:36:45.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b1b7fe640 1 -- 172.21.15.37:0/3718511291 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9b2401e460 con 0x7f9b340a0300 2024-04-26T00:36:45.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b3259c640 1 --2- 172.21.15.37:0/3718511291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b3409c750 0x7f9b3409cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:45.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b1b7fe640 1 --2- 172.21.15.37:0/3718511291 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9b20035ed0 0x7f9b20038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:45.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b3259c640 10 monclient: get_auth_request con 0x7f9b3409c750 auth_method 0 2024-04-26T00:36:45.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b3259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:45.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b3259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:45.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b3259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:45.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b1b7fe640 1 -- 172.21.15.37:0/3718511291 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(660..660 src has 370..660) ==== 3871+0+0 (secure 0 0 0) 0x7f9b24048ea0 con 0x7f9b340a0300 2024-04-26T00:36:45.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b3259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:45.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b3259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:45.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b3259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:45.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b3359e640 1 --2- 172.21.15.37:0/3718511291 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9b20035ed0 0x7f9b20038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:45.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b3359e640 10 monclient: get_auth_request con 0x7f9b20035ed0 auth_method 0 2024-04-26T00:36:45.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b3259c640 10 monclient(hunting): handle_auth_done global_id 194188 payload 824 2024-04-26T00:36:45.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.004+0000 7f9b3259c640 1 -- 172.21.15.37:0/3718511291 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9b3404f510 con 0x7f9b3409c750 2024-04-26T00:36:45.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.008+0000 7f9b3259c640 1 -- 172.21.15.37:0/3718511291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b3409c750 msgr2=0x7f9b3409cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:45.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.008+0000 7f9b3259c640 1 --2- 172.21.15.37:0/3718511291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b3409c750 0x7f9b3409cb30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9b3409deb0 tx=0x7f9b28011a60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:45.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.008+0000 7f9b3359e640 1 --2- 172.21.15.37:0/3718511291 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9b20035ed0 0x7f9b20038390 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f9b340a2da0 tx=0x7f9b2c0a7070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:45.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.008+0000 7f9b1b7fe640 1 -- 172.21.15.37:0/3718511291 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b28015b60 con 0x7f9b3409c750 2024-04-26T00:36:45.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.008+0000 7f9b1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:45.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.008+0000 7f9b1b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:36:45.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.008+0000 7f9b1b7fe640 10 monclient: dump: 2024-04-26T00:36:45.018 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:45.018 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:45.018 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:45.018 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:45.019 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:45.019 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:45.019 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:45.019 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:45.019 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:45.019 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:45.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.008+0000 7f9b1b7fe640 1 -- 172.21.15.37:0/3718511291 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9b28015b60 con 0x7f9b3409c750 2024-04-26T00:36:45.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.008+0000 7f9b1b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:45.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.008+0000 7f9b1b7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:36:45.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.008+0000 7f9b1b7fe640 1 -- 172.21.15.37:0/3718511291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b3409c750 msgr2=0x7f9b3409cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:45.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.008+0000 7f9b1b7fe640 1 --2- 172.21.15.37:0/3718511291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b3409c750 0x7f9b3409cb30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9b3409deb0 tx=0x7f9b28011a60 comp rx=0 tx=0).stop 2024-04-26T00:36:45.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.008+0000 7f9b1b7fe640 1 -- 172.21.15.37:0/3718511291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b3409c750 msgr2=0x7f9b3409cb30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:36:45.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.008+0000 7f9b1b7fe640 1 --2- 172.21.15.37:0/3718511291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b3409c750 0x7f9b3409cb30 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9b3409deb0 tx=0x7f9b28011a60 comp rx=0 tx=0).stop 2024-04-26T00:36:45.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.016+0000 7f9b39060640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:36:45.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.016+0000 7f9b39060640 1 --2- 172.21.15.37:0/3718511291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b3409d070 0x7f9b34135c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:45.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.016+0000 7f9b39060640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:45.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.016+0000 7f9b39060640 1 -- 172.21.15.37:0/3718511291 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9b3409eb60 con 0x7f9b3409d070 2024-04-26T00:36:45.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.016+0000 7f9b3259c640 1 --2- 172.21.15.37:0/3718511291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b3409d070 0x7f9b34135c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:45.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.016+0000 7f9b3259c640 10 monclient: get_auth_request con 0x7f9b3409d070 auth_method 0 2024-04-26T00:36:45.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.016+0000 7f9b3259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:45.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.016+0000 7f9b3259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:45.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.016+0000 7f9b3259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:45.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.016+0000 7f9b3259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:45.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.016+0000 7f9b3259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:45.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.016+0000 7f9b3259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:45.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.016+0000 7f9b3259c640 10 monclient(hunting): handle_auth_done global_id 194191 payload 824 2024-04-26T00:36:45.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.016+0000 7f9b3259c640 1 -- 172.21.15.37:0/3718511291 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9b34139f90 con 0x7f9b3409d070 2024-04-26T00:36:45.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.020+0000 7f9b3259c640 1 -- 172.21.15.37:0/3718511291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b3409d070 msgr2=0x7f9b34135c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:45.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.020+0000 7f9b3259c640 1 --2- 172.21.15.37:0/3718511291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b3409d070 0x7f9b34135c40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9b3409deb0 tx=0x7f9b280046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:45.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.020+0000 7f9b1b7fe640 1 -- 172.21.15.37:0/3718511291 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b28004230 con 0x7f9b3409d070 2024-04-26T00:36:45.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.020+0000 7f9b1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:45.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.020+0000 7f9b1b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:36:45.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.020+0000 7f9b1b7fe640 10 monclient: dump: 2024-04-26T00:36:45.023 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:45.023 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:45.024 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:45.024 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:45.024 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:45.024 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:45.024 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:45.024 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:45.024 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:45.024 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:45.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.020+0000 7f9b1b7fe640 1 -- 172.21.15.37:0/3718511291 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4547 (secure 0 0 0) 0x7f9b28000e50 con 0x7f9b3409d070 2024-04-26T00:36:45.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.020+0000 7f9b1b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:36:45.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.020+0000 7f9b1b7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:36:45.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.020+0000 7f9b1b7fe640 1 -- 172.21.15.37:0/3718511291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b3409d070 msgr2=0x7f9b34135c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:45.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.020+0000 7f9b1b7fe640 1 --2- 172.21.15.37:0/3718511291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b3409d070 0x7f9b34135c40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9b3409deb0 tx=0x7f9b280046d0 comp rx=0 tx=0).stop 2024-04-26T00:36:45.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.020+0000 7f9b1b7fe640 1 -- 172.21.15.37:0/3718511291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b3409d070 msgr2=0x7f9b34135c40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:36:45.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.020+0000 7f9b1b7fe640 1 --2- 172.21.15.37:0/3718511291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b3409d070 0x7f9b34135c40 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9b3409deb0 tx=0x7f9b280046d0 comp rx=0 tx=0).stop 2024-04-26T00:36:45.025 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:36:45.025 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:36:45.025 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:36:45.025 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:36:45.025 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 126, 2024-04-26T00:36:45.025 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 53591, 2024-04-26T00:36:45.025 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:36:45.025 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:36:45.025 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:36:45.025 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:36:45.025 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:36:45.026 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 4, 2024-04-26T00:36:45.026 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:36:45.026 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:36:45.026 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:36:45.026 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:36:45.026 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:36:45.026 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:36:45.026 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:36:45.026 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:36:45.026 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:36:45.026 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:36:45.026 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:36:45.026 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:36:45.026 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:36:45.026 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:36:45.027 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:36:45.027 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:36:45.027 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:36:45.027 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:36:45.027 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:36:45.027 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:36:45.027 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:36:45.027 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:36:45.027 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:36:45.027 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:36:45.027 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:36:45.027 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:36:45.027 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:36:45.027 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:36:45.027 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:45.028 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:45.028 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:36:45.028 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:36:45.028 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:36:45.028 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:36:45.028 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:36:45.028 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:36:45.028 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:36:45.028 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:36:45.028 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:36:45.028 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:36:45.028 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:36:45.028 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:36:45.028 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:36:45.029 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:36:45.029 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:36:45.029 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:36:45.029 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:36:45.029 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:36:45.029 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:36:45.029 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:36:45.029 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:36:45.029 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:36:45.029 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:36:45.029 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:36:45.029 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:36:45.029 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:36:45.029 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:36:45.029 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:36:45.030 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:36:45.030 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:36:45.030 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:45.030 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:36:45.030 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:45.030 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:36:45.030 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:36:45.030 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:36:45.030 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:36:45.030 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:45.030 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:36:45.030 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:36:45.030 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:45.030 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:45.030 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:45.031 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:36:45.031 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:36:45.031 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:45.031 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:45.031 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:45.031 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:45.031 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:36:45.031 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:36:45.031 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:36:45.031 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:36:45.031 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:36:45.031 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:45.031 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:45.031 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:36:45.031 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:36:45.032 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:36:45.032 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:36:45.032 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:45.032 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:36:45.032 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:36:45.032 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:45.032 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:45.032 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:45.032 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:36:45.032 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:36:45.032 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:45.032 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:45.032 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:45.032 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:45.032 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:36:45.033 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:36:45.033 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:36:45.033 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:36:45.033 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:36:45.033 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:45.033 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:45.033 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:36:45.033 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:36:45.033 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:36:45.033 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:36:45.033 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:45.033 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:36:45.033 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:36:45.033 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:45.033 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:45.034 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:45.034 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:36:45.034 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:36:45.034 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:45.034 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:45.034 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:45.034 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:45.034 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:36:45.034 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:36:45.034 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:36:45.034 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:36:45.034 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:36:45.034 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:45.034 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:45.035 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:45.035 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:36:45.035 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:36:45.035 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:45.035 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:36:45.035 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:36:45.035 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:36:45.035 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:45.035 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:36:45.035 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:36:45.035 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:45.035 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:36:45.035 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:36:45.035 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:36:45.035 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:45.036 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:45.036 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:45.036 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:36:45.036 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:36:45.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.020+0000 7f9b39060640 1 -- 172.21.15.37:0/3718511291 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9b20035ed0 msgr2=0x7f9b20038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:45.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.020+0000 7f9b39060640 1 --2- 172.21.15.37:0/3718511291 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9b20035ed0 0x7f9b20038390 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f9b340a2da0 tx=0x7f9b2c0a7070 comp rx=0 tx=0).stop 2024-04-26T00:36:45.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.020+0000 7f9b39060640 10 monclient: shutdown 2024-04-26T00:36:45.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.020+0000 7f9b39060640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:36:45.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.020+0000 7f9b39060640 1 -- 172.21.15.37:0/3718511291 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9b340a0300 msgr2=0x7f9b341cef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:45.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.020+0000 7f9b39060640 1 --2- 172.21.15.37:0/3718511291 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9b340a0300 0x7f9b341cef00 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f9b2400d800 tx=0x7f9b2400dcc0 comp rx=0 tx=0).stop 2024-04-26T00:36:45.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.020+0000 7f9b3359e640 1 -- 172.21.15.37:0/3718511291 reap_dead start 2024-04-26T00:36:45.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.020+0000 7f9b3359e640 1 -- 172.21.15.37:0/3718511291 reap_dead start 2024-04-26T00:36:45.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.020+0000 7f9b39060640 1 -- 172.21.15.37:0/3718511291 shutdown_connections 2024-04-26T00:36:45.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.020+0000 7f9b39060640 1 -- 172.21.15.37:0/3718511291 >> 172.21.15.37:0/3718511291 conn(0x7f9b340a1d50 msgr2=0x7f9b340a4870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:36:45.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.020+0000 7f9b39060640 1 -- 172.21.15.37:0/3718511291 shutdown_connections 2024-04-26T00:36:45.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.020+0000 7f9b39060640 1 -- 172.21.15.37:0/3718511291 wait complete. 2024-04-26T00:36:45.038 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:36:45.038 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-26T00:36:45.038 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-26T00:36:45.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.156+0000 7f983e928640 10 monclient: get_monmap_and_config 2024-04-26T00:36:45.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.156+0000 7f983e928640 10 monclient: build_initial_monmap 2024-04-26T00:36:45.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.156+0000 7f983e928640 10 monclient: monmap: 2024-04-26T00:36:45.157 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:36:45.157 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:45.157 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:36:45.162089+0000 2024-04-26T00:36:45.157 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:36:45.162089+0000 2024-04-26T00:36:45.157 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:36:45.157 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:45.157 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:36:45.157 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:36:45.157 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:36:45.157 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:36:45.157 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:36:45.157 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:45.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.156+0000 7f983e928640 1 Processor -- start 2024-04-26T00:36:45.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.156+0000 7f983e928640 1 -- start start 2024-04-26T00:36:45.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.156+0000 7f983e928640 10 monclient: init 2024-04-26T00:36:45.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f983e928640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:45.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f983e928640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-04-26T00:36:45.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f983e928640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9838071f00 0x7f98380722e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:45.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f983e928640 10 monclient(hunting): picked mon.noname-a con 0x7f9838071f00 addr v2:172.21.15.37:3300/0 2024-04-26T00:36:45.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f983e928640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f98380713a0 addr v1:172.21.15.37:6789/0 2024-04-26T00:36:45.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f983e928640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f98380717f0 0x7f98380a1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:45.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f983e928640 10 monclient(hunting): picked mon.noname-b con 0x7f98380717f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:36:45.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f983e928640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f983806ee70 con 0x7f98380713a0 2024-04-26T00:36:45.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f983e928640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:45.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f983e928640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f983806ea30 con 0x7f9838071f00 2024-04-26T00:36:45.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f983e928640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:45.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f983e928640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9838070120 con 0x7f98380717f0 2024-04-26T00:36:45.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f983e928640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:45.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f983e928640 10 monclient(hunting): authenticate will time out at 1986.468603s 2024-04-26T00:36:45.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f983cec5640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f98380717f0 0x7f98380a1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:45.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f9837fff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9838071f00 0x7f98380722e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:45.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f983cec5640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f98380717f0 0x7f98380a1bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:59482/0 (socket says 172.21.15.37:59482) 2024-04-26T00:36:45.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f983cec5640 1 -- 172.21.15.37:0/3799860768 learned_addr learned my addr 172.21.15.37:0/3799860768 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:36:45.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f983cec5640 10 monclient(hunting): get_auth_request con 0x7f98380717f0 auth_method 0 2024-04-26T00:36:45.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f983cec5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:45.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f983cec5640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:45.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f983cec5640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:45.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f9837fff640 10 monclient(hunting): get_auth_request con 0x7f9838071f00 auth_method 0 2024-04-26T00:36:45.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f9837fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:45.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f9837fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:45.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f9837fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:45.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f983cec5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:45.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f983cec5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:45.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f983cec5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:45.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f9837fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:45.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f9837fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:45.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f9837fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:45.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f983cec5640 10 monclient(hunting): handle_auth_done global_id 204113 payload 274 2024-04-26T00:36:45.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f983cec5640 1 -- 172.21.15.37:0/3799860768 >> v2:172.21.15.37:3300/0 conn(0x7f9838071f00 msgr2=0x7f98380722e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:36:45.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f983cec5640 1 --2- 172.21.15.37:0/3799860768 >> v2:172.21.15.37:3300/0 conn(0x7f9838071f00 0x7f98380722e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:45.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f983cec5640 1 -- 172.21.15.37:0/3799860768 >> v1:172.21.15.37:6789/0 conn(0x7f98380713a0 legacy=0x7f98380a14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:36:45.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f983cec5640 10 monclient: _finish_hunting 0 2024-04-26T00:36:45.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f983cec5640 1 monclient: found mon.noname-b 2024-04-26T00:36:45.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f983cec5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:45.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f983cec5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:36:45.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f983cec5640 1 -- 172.21.15.37:0/3799860768 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9838070400 con 0x7f98380717f0 2024-04-26T00:36:45.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.160+0000 7f9837fff640 1 --2- 172.21.15.37:0/3799860768 >> v2:172.21.15.37:3300/0 conn(0x7f9838071f00 0x7f98380722e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:36:45.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.164+0000 7f983cec5640 1 -- 172.21.15.37:0/3799860768 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f98380717f0 msgr2=0x7f98380a1bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:45.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.164+0000 7f983cec5640 1 --2- 172.21.15.37:0/3799860768 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f98380717f0 0x7f98380a1bc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9830009e50 tx=0x7f983001c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17e194a4d645c0d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:45.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.164+0000 7f9836ffd640 1 -- 172.21.15.37:0/3799860768 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98300ac020 con 0x7f98380717f0 2024-04-26T00:36:45.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.164+0000 7f9836ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:45.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.164+0000 7f9836ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:36:45.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.164+0000 7f9836ffd640 10 monclient: dump: 2024-04-26T00:36:45.165 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:45.165 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:45.165 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:45.165 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:45.165 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:45.165 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:45.165 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:45.165 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:45.165 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:45.166 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:45.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.164+0000 7f9836ffd640 10 monclient: _finish_auth 0 2024-04-26T00:36:45.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.164+0000 7f9836ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:36:45.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.164+0000 7f9836ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:45.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.164+0000 7f9836ffd640 1 -- 172.21.15.37:0/3799860768 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98300a6440 con 0x7f98380717f0 2024-04-26T00:36:45.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.164+0000 7f983e928640 5 monclient: authenticate success, global_id 204113 2024-04-26T00:36:45.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.164+0000 7f983e928640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:36:45.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.164+0000 7f9836ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:36:45.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.164+0000 7f983e928640 10 monclient: get_monmap_and_config success 2024-04-26T00:36:45.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.164+0000 7f983e928640 4 set_mon_vals no callback set 2024-04-26T00:36:45.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.164+0000 7f9836ffd640 1 -- 172.21.15.37:0/3799860768 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98300a6810 con 0x7f98380717f0 2024-04-26T00:36:45.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.164+0000 7f9836ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:45.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.164+0000 7f9836ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:36:45.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.164+0000 7f9836ffd640 10 monclient: dump: 2024-04-26T00:36:45.166 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:45.167 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:45.167 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:45.167 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:45.168 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:45.168 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:45.168 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:45.168 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:45.168 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:45.168 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:45.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.164+0000 7f983e928640 10 monclient: shutdown 2024-04-26T00:36:45.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.164+0000 7f983e928640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:36:45.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.164+0000 7f983e928640 1 -- 172.21.15.37:0/3799860768 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f98380717f0 msgr2=0x7f98380a1bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:45.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.164+0000 7f983e928640 1 --2- 172.21.15.37:0/3799860768 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f98380717f0 0x7f98380a1bc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9830009e50 tx=0x7f983001c940 comp rx=0 tx=0).stop 2024-04-26T00:36:45.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.164+0000 7f983e928640 1 -- 172.21.15.37:0/3799860768 shutdown_connections 2024-04-26T00:36:45.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.164+0000 7f983e928640 1 --2- 172.21.15.37:0/3799860768 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f98380717f0 0x7f98380a1bc0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:45.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.164+0000 7f983e928640 1 --2- 172.21.15.37:0/3799860768 >> v2:172.21.15.37:3300/0 conn(0x7f9838071f00 0x7f98380722e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:45.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.164+0000 7f983e928640 1 -- 172.21.15.37:0/3799860768 >> 172.21.15.37:0/3799860768 conn(0x7f98380a6250 msgr2=0x7f98380a2610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:36:45.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.164+0000 7f983e928640 1 -- 172.21.15.37:0/3799860768 shutdown_connections 2024-04-26T00:36:45.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.164+0000 7f983e928640 1 -- 172.21.15.37:0/3799860768 wait complete. 2024-04-26T00:36:45.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.164+0000 7f983e928640 10 monclient: build_initial_monmap 2024-04-26T00:36:45.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.164+0000 7f983e928640 10 monclient: monmap: 2024-04-26T00:36:45.174 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:36:45.174 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:36:45.174 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:36:45.174 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:36:45.174 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:36:45.174 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:36:45.175 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:36:45.175 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:36:45.175 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:36:45.175 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:45.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.164+0000 7f983e928640 1 Processor -- start 2024-04-26T00:36:45.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.164+0000 7f983e928640 1 -- start start 2024-04-26T00:36:45.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.164+0000 7f983e928640 10 monclient: init 2024-04-26T00:36:45.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f983e928640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:45.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f983e928640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:36:45.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f983e928640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f98380713a0 0x7f98381cee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:45.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f983e928640 10 monclient(hunting): picked mon.noname-a con 0x7f98380713a0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:36:45.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f983e928640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f98380717f0 0x7f983813a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:45.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f983e928640 10 monclient(hunting): picked mon.noname-c con 0x7f98380717f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:36:45.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f983e928640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9838071f00 0x7f983813abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:45.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f983e928640 10 monclient(hunting): picked mon.noname-b con 0x7f9838071f00 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:36:45.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f983e928640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:45.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f983e928640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f983813fa50 con 0x7f98380713a0 2024-04-26T00:36:45.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f983e928640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:45.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f983e928640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f983813f8d0 con 0x7f9838071f00 2024-04-26T00:36:45.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f983e928640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:45.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f983e928640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f983813fbd0 con 0x7f98380717f0 2024-04-26T00:36:45.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f983e928640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:45.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f983e928640 10 monclient(hunting): authenticate will time out at 1986.476383s 2024-04-26T00:36:45.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f98377fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f98380717f0 0x7f983813a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:45.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f9837fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f98380713a0 0x7f98381cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:45.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f9837fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f98380713a0 0x7f98381cee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50342/0 (socket says 172.21.15.37:50342) 2024-04-26T00:36:45.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f9837fff640 1 -- 172.21.15.37:0/3412883799 learned_addr learned my addr 172.21.15.37:0/3412883799 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:36:45.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f9837fff640 10 monclient(hunting): get_auth_request con 0x7f98380713a0 auth_method 0 2024-04-26T00:36:45.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f9837fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:45.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f9837fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:45.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f9837fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:45.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f98377fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f98380717f0 0x7f983813a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:59488/0 (socket says 172.21.15.37:59488) 2024-04-26T00:36:45.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f98377fe640 10 monclient(hunting): get_auth_request con 0x7f98380717f0 auth_method 0 2024-04-26T00:36:45.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f98377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:45.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f98377fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:45.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f98377fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:45.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f9837fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:45.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f9837fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:45.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f9837fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:45.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f98377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:45.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f98377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:45.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f98377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:45.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f983cec5640 1 --2- 172.21.15.37:0/3412883799 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9838071f00 0x7f983813abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:45.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f9837fff640 10 monclient(hunting): handle_auth_done global_id 144720 payload 824 2024-04-26T00:36:45.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f9837fff640 1 -- 172.21.15.37:0/3412883799 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f98380717f0 msgr2=0x7f983813a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:45.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f9837fff640 1 --2- 172.21.15.37:0/3412883799 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f98380717f0 0x7f983813a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:45.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f9837fff640 1 -- 172.21.15.37:0/3412883799 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9838071f00 msgr2=0x7f983813abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:45.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f9837fff640 1 --2- 172.21.15.37:0/3412883799 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9838071f00 0x7f983813abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:45.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f9837fff640 10 monclient: _finish_hunting 0 2024-04-26T00:36:45.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f9837fff640 1 monclient: found mon.noname-a 2024-04-26T00:36:45.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f9837fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:45.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f9837fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:36:45.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f9837fff640 1 -- 172.21.15.37:0/3412883799 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9838124d60 con 0x7f98380713a0 2024-04-26T00:36:45.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f983cec5640 10 monclient: get_auth_request con 0x7f9838071f00 auth_method 0 2024-04-26T00:36:45.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f983cec5640 1 --2- 172.21.15.37:0/3412883799 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9838071f00 0x7f983813abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:36:45.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f9837fff640 1 -- 172.21.15.37:0/3412883799 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f98380713a0 msgr2=0x7f98381cee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:45.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f9837fff640 1 --2- 172.21.15.37:0/3412883799 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f98380713a0 0x7f98381cee80 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f982800ebe0 tx=0x7f982800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:45.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f9834ff9640 1 -- 172.21.15.37:0/3412883799 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9828017070 con 0x7f98380713a0 2024-04-26T00:36:45.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f9834ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:45.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f9834ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:36:45.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f9834ff9640 10 monclient: dump: 2024-04-26T00:36:45.179 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:45.179 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:45.179 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:45.179 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:45.179 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:45.179 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:45.180 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:45.180 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:45.180 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:45.180 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:45.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f9834ff9640 10 monclient: _finish_auth 0 2024-04-26T00:36:45.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f9834ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:36:45.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f9834ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:45.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f9834ff9640 1 -- 172.21.15.37:0/3412883799 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f982800ce40 con 0x7f98380713a0 2024-04-26T00:36:45.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f9834ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:36:45.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f9834ff9640 1 -- 172.21.15.37:0/3412883799 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9828013690 con 0x7f98380713a0 2024-04-26T00:36:45.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f9834ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:45.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f9834ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:36:45.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f9834ff9640 10 monclient: dump: 2024-04-26T00:36:45.181 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:45.181 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:45.181 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:45.181 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:45.181 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:45.181 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:45.181 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:45.181 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:45.181 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:45.182 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:45.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f983e928640 5 monclient: authenticate success, global_id 144720 2024-04-26T00:36:45.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f983e928640 10 monclient: _renew_subs 2024-04-26T00:36:45.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f983e928640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:45.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f983e928640 1 -- 172.21.15.37:0/3412883799 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98381cf3c0 con 0x7f98380713a0 2024-04-26T00:36:45.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f98367fc640 4 set_mon_vals no callback set 2024-04-26T00:36:45.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f983e928640 10 monclient: _renew_subs 2024-04-26T00:36:45.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f983e928640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:45.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.168+0000 7f983e928640 1 -- 172.21.15.37:0/3412883799 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98381267b0 con 0x7f98380713a0 2024-04-26T00:36:45.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.172+0000 7f983e928640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:45.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.172+0000 7f983e928640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:45.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.172+0000 7f983e928640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:45.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.172+0000 7f9834ff9640 1 -- 172.21.15.37:0/3412883799 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f982801a070 con 0x7f98380713a0 2024-04-26T00:36:45.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.172+0000 7f983e928640 1 -- 172.21.15.37:0/3412883799 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9838066f40 con 0x7f98380713a0 2024-04-26T00:36:45.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.172+0000 7f9834ff9640 1 --2- 172.21.15.37:0/3412883799 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9824035e80 0x7f9824038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:45.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.172+0000 7f9834ff9640 1 -- 172.21.15.37:0/3412883799 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(660..660 src has 370..660) ==== 3871+0+0 (secure 0 0 0) 0x7f9828048ac0 con 0x7f98380713a0 2024-04-26T00:36:45.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.172+0000 7f98377fe640 1 --2- 172.21.15.37:0/3412883799 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9824035e80 0x7f9824038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:45.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.172+0000 7f98377fe640 10 monclient: get_auth_request con 0x7f9824035e80 auth_method 0 2024-04-26T00:36:45.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.176+0000 7f98377fe640 1 --2- 172.21.15.37:0/3412883799 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9824035e80 0x7f9824038340 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f98381250e0 tx=0x7f982c009060 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:45.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.176+0000 7f9834ff9640 1 -- 172.21.15.37:0/3412883799 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9828047850 con 0x7f98380713a0 2024-04-26T00:36:45.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.176+0000 7f9834ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:45.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.176+0000 7f9834ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:36:45.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.404+0000 7f983e928640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-26T00:36:45.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.404+0000 7f983e928640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-26T00:36:45.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.404+0000 7f983e928640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:45.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.404+0000 7f983e928640 1 -- 172.21.15.37:0/3412883799 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f983806c1e0 con 0x7f98380713a0 2024-04-26T00:36:45.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.404+0000 7f9834ff9640 1 -- 172.21.15.37:0/3412883799 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f9828011510 con 0x7f98380713a0 2024-04-26T00:36:45.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.404+0000 7f9834ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-26T00:36:45.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.404+0000 7f9834ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:36:45.406 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.404+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:45.406 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.404+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=11,kv=10,logm=48} crc {auth=2801402904,config=692619927,health=101084904,kv=1034996044,logm=650333477}) 2024-04-26T00:36:45.406 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.404+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=29,config=2,health=12,kv=9,logm=48} crc {auth=10178223,config=692619927,health=949105977,kv=3913313004,logm=610562616}) 2024-04-26T00:36:45.406 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.404+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:45.406 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.404+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=11,kv=10,logm=48} crc {auth=2801402904,config=692619927,health=101084904,kv=1034996044,logm=650333477}) 2024-04-26T00:36:45.406 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.404+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=27,config=2,health=11,kv=10,logm=50} crc {auth=301750000,config=692619927,health=2214895885,kv=1034996044,logm=3319316877}) 2024-04-26T00:36:45.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.408+0000 7f983e928640 1 -- 172.21.15.37:0/3412883799 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9824035e80 msgr2=0x7f9824038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:45.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.408+0000 7f983e928640 1 --2- 172.21.15.37:0/3412883799 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9824035e80 0x7f9824038340 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f98381250e0 tx=0x7f982c009060 comp rx=0 tx=0).stop 2024-04-26T00:36:45.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.408+0000 7f983e928640 10 monclient: shutdown 2024-04-26T00:36:45.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.408+0000 7f983e928640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:36:45.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.408+0000 7f983e928640 1 -- 172.21.15.37:0/3412883799 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f98380713a0 msgr2=0x7f98381cee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:45.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.408+0000 7f983e928640 1 --2- 172.21.15.37:0/3412883799 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f98380713a0 0x7f98381cee80 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f982800ebe0 tx=0x7f982800c520 comp rx=0 tx=0).stop 2024-04-26T00:36:45.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.408+0000 7f983e928640 1 -- 172.21.15.37:0/3412883799 shutdown_connections 2024-04-26T00:36:45.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.408+0000 7f983e928640 1 --2- 172.21.15.37:0/3412883799 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9824035e80 0x7f9824038340 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:45.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.408+0000 7f983e928640 1 --2- 172.21.15.37:0/3412883799 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9838071f00 0x7f983813abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:45.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.408+0000 7f983e928640 1 --2- 172.21.15.37:0/3412883799 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f98380717f0 0x7f983813a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:45.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.408+0000 7f983e928640 1 --2- 172.21.15.37:0/3412883799 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f98380713a0 0x7f98381cee80 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:45.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.408+0000 7f983e928640 1 -- 172.21.15.37:0/3412883799 >> 172.21.15.37:0/3412883799 conn(0x7f98380a6250 msgr2=0x7f98380a1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:36:45.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.408+0000 7f983e928640 1 -- 172.21.15.37:0/3412883799 shutdown_connections 2024-04-26T00:36:45.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:45.408+0000 7f983e928640 1 -- 172.21.15.37:0/3412883799 wait complete. 2024-04-26T00:36:45.409 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.408+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:45.410 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.408+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=120065797}) 2024-04-26T00:36:45.410 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.408+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=182092316}) 2024-04-26T00:36:45.410 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.408+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:45.410 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.408+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=120065797}) 2024-04-26T00:36:45.410 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.408+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=775978065}) 2024-04-26T00:36:45.411 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.412+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:45.411 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.412+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=775495765}) 2024-04-26T00:36:45.411 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.412+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1195171188}) 2024-04-26T00:36:45.411 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.412+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:45.411 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.412+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=775495765}) 2024-04-26T00:36:45.411 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.412+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3349658389}) 2024-04-26T00:36:45.414 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.412+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:45.414 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.412+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3758912148}) 2024-04-26T00:36:45.414 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.412+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2149242713}) 2024-04-26T00:36:45.414 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.412+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:45.414 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.412+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3758912148}) 2024-04-26T00:36:45.414 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.412+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1804063288}) 2024-04-26T00:36:45.417 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.416+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:45.417 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.416+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4081756250}) 2024-04-26T00:36:45.417 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.416+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2294423719}) 2024-04-26T00:36:45.417 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.416+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:45.417 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.416+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4081756250}) 2024-04-26T00:36:45.417 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.416+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1712760454}) 2024-04-26T00:36:45.419 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.420+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:45.419 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.420+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1145841051}) 2024-04-26T00:36:45.419 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.420+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2525440383}) 2024-04-26T00:36:45.419 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.420+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:45.419 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.420+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1145841051}) 2024-04-26T00:36:45.419 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.420+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=725910075}) 2024-04-26T00:36:45.421 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.420+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:45.421 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.420+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1495613679}) 2024-04-26T00:36:45.421 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.420+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2400766410}) 2024-04-26T00:36:45.421 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.420+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:45.421 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.420+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1495613679}) 2024-04-26T00:36:45.421 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.420+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1274834889}) 2024-04-26T00:36:45.423 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.424+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:45.423 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.424+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=161651064}) 2024-04-26T00:36:45.423 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.424+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1485951454}) 2024-04-26T00:36:45.423 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.424+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:45.423 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.424+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=161651064}) 2024-04-26T00:36:45.424 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.424+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=376149715}) 2024-04-26T00:36:45.425 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-26T00:36:45.426 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.424+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:45.426 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.424+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2378560493}) 2024-04-26T00:36:45.426 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.424+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1437776142}) 2024-04-26T00:36:45.426 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.424+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:45.426 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.424+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2378560493}) 2024-04-26T00:36:45.426 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.424+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1122611801}) 2024-04-26T00:36:45.432 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.428+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:45.432 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.428+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2487632045}) 2024-04-26T00:36:45.432 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.428+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2066715089}) 2024-04-26T00:36:45.432 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.428+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:45.432 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.428+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2487632045}) 2024-04-26T00:36:45.432 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.428+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=124343527}) 2024-04-26T00:36:45.432 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.428+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:45.432 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.428+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3918247175}) 2024-04-26T00:36:45.433 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.428+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=651109523}) 2024-04-26T00:36:45.433 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.428+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:45.433 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.428+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3918247175}) 2024-04-26T00:36:45.433 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.428+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2050682368}) 2024-04-26T00:36:45.433 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.432+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:45.433 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.432+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=338373218}) 2024-04-26T00:36:45.433 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.432+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=703222044}) 2024-04-26T00:36:45.433 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.432+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:45.433 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.432+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=338373218}) 2024-04-26T00:36:45.433 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.432+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3931359538}) 2024-04-26T00:36:45.557 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.556+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:45.557 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.556+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3061119623}) 2024-04-26T00:36:45.557 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.556+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=610203204}) 2024-04-26T00:36:45.557 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.556+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:45.557 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.556+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3061119623}) 2024-04-26T00:36:45.558 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.556+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=769809599}) 2024-04-26T00:36:45.565 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.564+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:45.565 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.564+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=215450555}) 2024-04-26T00:36:45.565 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.564+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1272110274}) 2024-04-26T00:36:45.565 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.564+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:45.565 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.564+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=215450555}) 2024-04-26T00:36:45.565 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:45.564+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2342090210}) 2024-04-26T00:36:45.570 INFO:tasks.rados.rados.0.smithi045.stdout:2913: done (6 left) 2024-04-26T00:36:45.570 INFO:tasks.rados.rados.0.smithi045.stdout:2914: read oid 14 snap -1 2024-04-26T00:36:45.570 INFO:tasks.rados.rados.0.smithi045.stdout:2914: expect (ObjNum 1032 snap 311 seq_num 1032) 2024-04-26T00:36:45.570 INFO:tasks.rados.rados.0.smithi045.stdout:2915: write oid 45 current snap is 317 2024-04-26T00:36:45.570 INFO:tasks.rados.rados.0.smithi045.stdout:2907: finishing write tid 5 to smithi04526579-39 2024-04-26T00:36:45.570 INFO:tasks.rados.rados.0.smithi045.stdout:2907: oid 39 updating version 0 to 2198 2024-04-26T00:36:45.570 INFO:tasks.rados.rados.0.smithi045.stdout:2907: oid 39 version 2198 is already newer than 2197 2024-04-26T00:36:45.570 INFO:tasks.rados.rados.0.smithi045.stdout:2907: oid 39 version 2198 is already newer than 2196 2024-04-26T00:36:45.570 INFO:tasks.rados.rados.0.smithi045.stdout:2907: oid 39 version 2198 is already newer than 2195 2024-04-26T00:36:45.570 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 2198 (ObjNum 1045 snap 317 seq_num 1045) dirty exists 2024-04-26T00:36:45.570 INFO:tasks.rados.rados.0.smithi045.stdout:2907: left oid 39 (ObjNum 1045 snap 317 seq_num 1045) 2024-04-26T00:36:45.570 INFO:tasks.rados.rados.0.smithi045.stdout:2908: finishing rollback tid 0 to smithi04526579-11 2024-04-26T00:36:45.571 INFO:tasks.rados.rados.0.smithi045.stdout:2908: finishing rollback tid 1 to smithi04526579-11 2024-04-26T00:36:45.571 INFO:tasks.rados.rados.0.smithi045.stdout:2908: finishing rollback tid 2 to smithi04526579-11 2024-04-26T00:36:45.571 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 2177 (ObjNum 945 snap 292 seq_num 945) dirty exists 2024-04-26T00:36:45.571 INFO:tasks.rados.rados.0.smithi045.stdout:2909: finishing write tid 1 to smithi04526579-9 2024-04-26T00:36:45.571 INFO:tasks.rados.rados.0.smithi045.stdout:2915: seq_num 1048 ranges {413736=401313,1326776=403024,2413414=457893,3438716=175020} 2024-04-26T00:36:45.576 INFO:tasks.rados.rados.0.smithi045.stdout:2915: writing smithi04526579-45 from 413736 to 815049 tid 1 2024-04-26T00:36:45.584 INFO:tasks.rados.rados.0.smithi045.stdout:2915: writing smithi04526579-45 from 1326776 to 1729800 tid 2 2024-04-26T00:36:45.594 INFO:tasks.rados.rados.0.smithi045.stdout:2915: writing smithi04526579-45 from 2413414 to 2871307 tid 3 2024-04-26T00:36:45.600 INFO:tasks.rados.rados.0.smithi045.stdout:2915: writing smithi04526579-45 from 3438716 to 3613736 tid 4 2024-04-26T00:36:45.600 INFO:tasks.rados.rados.0.smithi045.stdout:2907: done (7 left) 2024-04-26T00:36:45.600 INFO:tasks.rados.rados.0.smithi045.stdout:2908: done (6 left) 2024-04-26T00:36:45.601 INFO:tasks.rados.rados.0.smithi045.stdout:2916: rollback oid 19 current snap is 317 2024-04-26T00:36:45.601 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 19 to 315 2024-04-26T00:36:45.601 INFO:tasks.rados.rados.0.smithi045.stdout:2909: finishing write tid 2 to smithi04526579-9 2024-04-26T00:36:45.601 INFO:tasks.rados.rados.0.smithi045.stdout:2909: finishing write tid 3 to smithi04526579-9 2024-04-26T00:36:45.601 INFO:tasks.rados.rados.0.smithi045.stdout:2909: finishing write tid 4 to smithi04526579-9 2024-04-26T00:36:45.601 INFO:tasks.rados.rados.0.smithi045.stdout:2909: oid 9 updating version 0 to 2023 2024-04-26T00:36:45.601 INFO:tasks.rados.rados.0.smithi045.stdout:2909: oid 9 updating version 2023 to 2025 2024-04-26T00:36:45.601 INFO:tasks.rados.rados.0.smithi045.stdout:2909: oid 9 version 2025 is already newer than 2024 2024-04-26T00:36:45.602 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 2025 (ObjNum 1046 snap 317 seq_num 1046) dirty exists 2024-04-26T00:36:45.602 INFO:tasks.rados.rados.0.smithi045.stdout:2909: left oid 9 (ObjNum 1046 snap 317 seq_num 1046) 2024-04-26T00:36:45.602 INFO:tasks.rados.rados.0.smithi045.stdout:2911: expect (ObjNum 1033 snap 312 seq_num 1033) 2024-04-26T00:36:45.856 INFO:tasks.rados.rados.0.smithi045.stdout:2910: finishing copy_from racing read to smithi04526579-32 2024-04-26T00:36:45.856 INFO:tasks.rados.rados.0.smithi045.stdout:2910: finishing copy_from to smithi04526579-32 2024-04-26T00:36:45.856 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 1975 (ObjNum 1022 snap 310 seq_num 1022) dirty exists 2024-04-26T00:36:45.856 INFO:tasks.rados.rados.0.smithi045.stdout:2912: finishing write tid 1 to smithi04526579-16 2024-04-26T00:36:45.856 INFO:tasks.rados.rados.0.smithi045.stdout:2912: finishing write tid 2 to smithi04526579-16 2024-04-26T00:36:45.856 INFO:tasks.rados.rados.0.smithi045.stdout:2912: finishing write tid 3 to smithi04526579-16 2024-04-26T00:36:45.856 INFO:tasks.rados.rados.0.smithi045.stdout:2912: finishing write tid 4 to smithi04526579-16 2024-04-26T00:36:45.856 INFO:tasks.rados.rados.0.smithi045.stdout:2912: finishing write tid 5 to smithi04526579-16 2024-04-26T00:36:45.856 INFO:tasks.rados.rados.0.smithi045.stdout:2912: oid 16 updating version 0 to 1189 2024-04-26T00:36:45.857 INFO:tasks.rados.rados.0.smithi045.stdout:2912: oid 16 version 1189 is already newer than 1187 2024-04-26T00:36:45.857 INFO:tasks.rados.rados.0.smithi045.stdout:2912: oid 16 version 1189 is already newer than 1186 2024-04-26T00:36:45.857 INFO:tasks.rados.rados.0.smithi045.stdout:2912: oid 16 version 1189 is already newer than 1188 2024-04-26T00:36:45.857 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 1189 (ObjNum 1047 snap 317 seq_num 1047) dirty exists 2024-04-26T00:36:45.857 INFO:tasks.rados.rados.0.smithi045.stdout:2912: left oid 16 (ObjNum 1047 snap 317 seq_num 1047) 2024-04-26T00:36:45.857 INFO:tasks.rados.rados.0.smithi045.stdout:2914: expect (ObjNum 1032 snap 311 seq_num 1032) 2024-04-26T00:36:46.053 INFO:tasks.rados.rados.0.smithi045.stdout:2915: finishing write tid 1 to smithi04526579-45 2024-04-26T00:36:46.053 INFO:tasks.rados.rados.0.smithi045.stdout:2915: finishing write tid 2 to smithi04526579-45 2024-04-26T00:36:46.053 INFO:tasks.rados.rados.0.smithi045.stdout:2915: finishing write tid 3 to smithi04526579-45 2024-04-26T00:36:46.053 INFO:tasks.rados.rados.0.smithi045.stdout:2915: finishing write tid 4 to smithi04526579-45 2024-04-26T00:36:46.053 INFO:tasks.rados.rados.0.smithi045.stdout:2915: finishing write tid 5 to smithi04526579-45 2024-04-26T00:36:46.053 INFO:tasks.rados.rados.0.smithi045.stdout:2916: finishing rollback tid 1 to smithi04526579-19 2024-04-26T00:36:46.053 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 0 (ObjNum 129 snap 0 seq_num 2242028456) dirty dne 2024-04-26T00:36:46.053 INFO:tasks.rados.rados.0.smithi045.stdout:2915: finishing write tid 6 to smithi04526579-45 2024-04-26T00:36:46.054 INFO:tasks.rados.rados.0.smithi045.stdout:2915: oid 45 updating version 0 to 2181 2024-04-26T00:36:46.054 INFO:tasks.rados.rados.0.smithi045.stdout:2915: oid 45 version 2181 is already newer than 2178 2024-04-26T00:36:46.054 INFO:tasks.rados.rados.0.smithi045.stdout:2915: oid 45 version 2181 is already newer than 2180 2024-04-26T00:36:46.054 INFO:tasks.rados.rados.0.smithi045.stdout:2915: oid 45 version 2181 is already newer than 2179 2024-04-26T00:36:46.054 INFO:tasks.rados.rados.0.smithi045.stdout:2915: oid 45 updating version 2181 to 2182 2024-04-26T00:36:46.054 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 2182 (ObjNum 1048 snap 317 seq_num 1048) dirty exists 2024-04-26T00:36:46.054 INFO:tasks.rados.rados.0.smithi045.stdout:2915: left oid 45 (ObjNum 1048 snap 317 seq_num 1048) 2024-04-26T00:36:46.054 INFO:tasks.rados.rados.0.smithi045.stdout:2909: done (6 left) 2024-04-26T00:36:46.054 INFO:tasks.rados.rados.0.smithi045.stdout:2910: done (5 left) 2024-04-26T00:36:46.054 INFO:tasks.rados.rados.0.smithi045.stdout:2911: done (4 left) 2024-04-26T00:36:46.054 INFO:tasks.rados.rados.0.smithi045.stdout:2912: done (3 left) 2024-04-26T00:36:46.054 INFO:tasks.rados.rados.0.smithi045.stdout:2914: done (2 left) 2024-04-26T00:36:46.054 INFO:tasks.rados.rados.0.smithi045.stdout:2915: done (1 left) 2024-04-26T00:36:46.055 INFO:tasks.rados.rados.0.smithi045.stdout:2916: done (0 left) 2024-04-26T00:36:46.055 INFO:tasks.rados.rados.0.smithi045.stdout:2917: read oid 42 snap -1 2024-04-26T00:36:46.055 INFO:tasks.rados.rados.0.smithi045.stdout:2917: expect (ObjNum 903 snap 277 seq_num 903) 2024-04-26T00:36:46.055 INFO:tasks.rados.rados.0.smithi045.stdout:2918: copy_from oid 31 from oid 43 current snap is 317 2024-04-26T00:36:46.055 INFO:tasks.rados.rados.0.smithi045.stdout:2919: delete oid 6 current snap is 317 2024-04-26T00:36:46.060 INFO:tasks.rados.rados.0.smithi045.stdout:2919: done (2 left) 2024-04-26T00:36:46.060 INFO:tasks.rados.rados.0.smithi045.stdout:2920: copy_from oid 35 from oid 21 current snap is 317 2024-04-26T00:36:46.060 INFO:tasks.rados.rados.0.smithi045.stdout:2921: delete oid 26 current snap is 317 2024-04-26T00:36:46.063 INFO:tasks.rados.rados.0.smithi045.stdout:2917: expect (ObjNum 903 snap 277 seq_num 903) 2024-04-26T00:36:46.097 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.096+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.097 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.096+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3473685225}) 2024-04-26T00:36:46.098 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.096+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1164017384}) 2024-04-26T00:36:46.098 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.096+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.098 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.096+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3473685225}) 2024-04-26T00:36:46.098 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.096+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=42012409}) 2024-04-26T00:36:46.105 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.104+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.105 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.104+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3363946777}) 2024-04-26T00:36:46.105 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.104+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=131629590}) 2024-04-26T00:36:46.105 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.104+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.105 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.104+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3363946777}) 2024-04-26T00:36:46.105 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.104+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1956710247}) 2024-04-26T00:36:46.115 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.116+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.115 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.116+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1552359016}) 2024-04-26T00:36:46.115 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.116+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4220632479}) 2024-04-26T00:36:46.115 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.116+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.115 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.116+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1552359016}) 2024-04-26T00:36:46.115 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.116+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2865152948}) 2024-04-26T00:36:46.172 INFO:tasks.rados.rados.0.smithi045.stdout:2918: finishing copy_from racing read to smithi04526579-31 2024-04-26T00:36:46.172 INFO:tasks.rados.rados.0.smithi045.stdout:2920: finishing copy_from racing read to smithi04526579-35 2024-04-26T00:36:46.172 INFO:tasks.rados.rados.0.smithi045.stdout:2918: finishing copy_from to smithi04526579-31 2024-04-26T00:36:46.172 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 1983 (ObjNum 1031 snap 311 seq_num 1031) dirty exists 2024-04-26T00:36:46.172 INFO:tasks.rados.rados.0.smithi045.stdout:2920: finishing copy_from to smithi04526579-35 2024-04-26T00:36:46.173 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 2354 (ObjNum 903 snap 277 seq_num 903) dirty exists 2024-04-26T00:36:46.173 INFO:tasks.rados.rados.0.smithi045.stdout:2917: done (3 left) 2024-04-26T00:36:46.173 INFO:tasks.rados.rados.0.smithi045.stdout:2918: done (2 left) 2024-04-26T00:36:46.173 INFO:tasks.rados.rados.0.smithi045.stdout:2920: done (1 left) 2024-04-26T00:36:46.173 INFO:tasks.rados.rados.0.smithi045.stdout:2921: done (0 left) 2024-04-26T00:36:46.173 INFO:tasks.rados.rados.0.smithi045.stdout:2922: write oid 24 current snap is 317 2024-04-26T00:36:46.173 INFO:tasks.rados.rados.0.smithi045.stdout:2922: seq_num 1051 ranges {464157=743413,1618575=666704,3012036=408062} 2024-04-26T00:36:46.193 INFO:tasks.rados.rados.0.smithi045.stdout:2922: writing smithi04526579-24 from 464157 to 1207570 tid 1 2024-04-26T00:36:46.209 INFO:tasks.rados.rados.0.smithi045.stdout:2922: writing smithi04526579-24 from 1618575 to 2285279 tid 2 2024-04-26T00:36:46.227 INFO:tasks.rados.rados.0.smithi045.stdout:2922: writing smithi04526579-24 from 3012036 to 3420098 tid 3 2024-04-26T00:36:46.227 INFO:tasks.rados.rados.0.smithi045.stdout:2923: snap_create 2024-04-26T00:36:46.227 INFO:tasks.rados.rados.0.smithi045.stdout:2922: finishing write tid 1 to smithi04526579-24 2024-04-26T00:36:46.228 INFO:tasks.rados.rados.0.smithi045.stdout:2922: finishing write tid 2 to smithi04526579-24 2024-04-26T00:36:46.236 INFO:tasks.rados.rados.0.smithi045.stdout:2922: finishing write tid 3 to smithi04526579-24 2024-04-26T00:36:46.237 INFO:tasks.rados.rados.0.smithi045.stdout:2922: finishing write tid 4 to smithi04526579-24 2024-04-26T00:36:46.238 INFO:tasks.rados.rados.0.smithi045.stdout:2922: finishing write tid 5 to smithi04526579-24 2024-04-26T00:36:46.238 INFO:tasks.rados.rados.0.smithi045.stdout:2922: oid 24 updating version 0 to 2399 2024-04-26T00:36:46.238 INFO:tasks.rados.rados.0.smithi045.stdout:2922: oid 24 updating version 2399 to 2400 2024-04-26T00:36:46.238 INFO:tasks.rados.rados.0.smithi045.stdout:2922: oid 24 updating version 2400 to 2401 2024-04-26T00:36:46.238 INFO:tasks.rados.rados.0.smithi045.stdout:2922: oid 24 version 2401 is already newer than 2398 2024-04-26T00:36:46.238 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 2401 (ObjNum 1051 snap 317 seq_num 1051) dirty exists 2024-04-26T00:36:46.238 INFO:tasks.rados.rados.0.smithi045.stdout:2922: left oid 24 (ObjNum 1051 snap 317 seq_num 1051) 2024-04-26T00:36:46.244 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.244+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.244 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.244+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2270610376}) 2024-04-26T00:36:46.244 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.244+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1666148225}) 2024-04-26T00:36:46.244 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.244+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.244 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.244+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2270610376}) 2024-04-26T00:36:46.244 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.244+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2238078917}) 2024-04-26T00:36:46.251 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.244+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.252 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.244+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1562710607}) 2024-04-26T00:36:46.252 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.244+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=305404715}) 2024-04-26T00:36:46.252 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.244+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.252 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.244+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1562710607}) 2024-04-26T00:36:46.252 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.244+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3981225712}) 2024-04-26T00:36:46.252 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.248+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.252 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.248+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1818682905}) 2024-04-26T00:36:46.253 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.248+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3572113739}) 2024-04-26T00:36:46.253 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.248+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.253 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.248+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1818682905}) 2024-04-26T00:36:46.253 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.248+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3253005896}) 2024-04-26T00:36:46.253 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.252+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.253 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.252+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1807229609}) 2024-04-26T00:36:46.253 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.252+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=62393873}) 2024-04-26T00:36:46.253 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.252+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.253 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.252+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1807229609}) 2024-04-26T00:36:46.253 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.252+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1107063031}) 2024-04-26T00:36:46.259 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.252+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.259 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.252+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1822335843}) 2024-04-26T00:36:46.259 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.252+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1091312140}) 2024-04-26T00:36:46.259 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.252+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.260 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.252+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1822335843}) 2024-04-26T00:36:46.260 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.252+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2896628869}) 2024-04-26T00:36:46.260 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.256+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.260 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.256+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=646277894}) 2024-04-26T00:36:46.260 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.256+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2458129505}) 2024-04-26T00:36:46.260 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.256+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.260 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.256+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=646277894}) 2024-04-26T00:36:46.260 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.256+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1160517247}) 2024-04-26T00:36:46.262 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.256+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.263 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.256+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3042712182}) 2024-04-26T00:36:46.263 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.256+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1257956405}) 2024-04-26T00:36:46.263 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.256+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.263 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.256+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3042712182}) 2024-04-26T00:36:46.263 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.256+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3517267851}) 2024-04-26T00:36:46.263 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.260+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.263 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.260+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3701102623}) 2024-04-26T00:36:46.263 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.260+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1554220038}) 2024-04-26T00:36:46.263 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.260+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.263 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.260+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3701102623}) 2024-04-26T00:36:46.263 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.260+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=808729360}) 2024-04-26T00:36:46.263 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.260+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.263 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.260+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=720945663}) 2024-04-26T00:36:46.263 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.260+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3748338972}) 2024-04-26T00:36:46.264 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.260+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.264 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.260+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=720945663}) 2024-04-26T00:36:46.264 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.260+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2116382871}) 2024-04-26T00:36:46.265 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.264+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.265 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.264+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3350611991}) 2024-04-26T00:36:46.265 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.264+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3793327641}) 2024-04-26T00:36:46.265 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.264+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.265 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.264+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3350611991}) 2024-04-26T00:36:46.266 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.264+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1587710992}) 2024-04-26T00:36:46.266 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.264+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.267 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.264+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3255339525}) 2024-04-26T00:36:46.267 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.264+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1597899790}) 2024-04-26T00:36:46.267 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.264+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.267 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.264+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3255339525}) 2024-04-26T00:36:46.267 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.264+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1347546710}) 2024-04-26T00:36:46.268 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.268+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.268 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.268+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=900716200}) 2024-04-26T00:36:46.268 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.268+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=31862057}) 2024-04-26T00:36:46.268 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.268+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.268 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.268+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=900716200}) 2024-04-26T00:36:46.268 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.268+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1445095683}) 2024-04-26T00:36:46.270 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.268+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.270 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.268+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2539507580}) 2024-04-26T00:36:46.270 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.268+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1420686264}) 2024-04-26T00:36:46.270 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.268+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.270 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.268+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2539507580}) 2024-04-26T00:36:46.270 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.268+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=69575835}) 2024-04-26T00:36:46.273 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.272+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.274 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.272+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2508495389}) 2024-04-26T00:36:46.274 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.272+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3167514857}) 2024-04-26T00:36:46.274 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.272+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.274 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.272+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2508495389}) 2024-04-26T00:36:46.274 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.272+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3931298408}) 2024-04-26T00:36:46.275 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.276+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.275 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.276+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=829909104}) 2024-04-26T00:36:46.275 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.276+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=393200967}) 2024-04-26T00:36:46.275 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.276+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.275 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.276+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=829909104}) 2024-04-26T00:36:46.275 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.276+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2119832515}) 2024-04-26T00:36:46.277 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.276+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.277 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.276+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=154589287}) 2024-04-26T00:36:46.277 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.276+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4061489015}) 2024-04-26T00:36:46.277 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.276+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.277 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.276+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=154589287}) 2024-04-26T00:36:46.277 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.276+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4284674616}) 2024-04-26T00:36:46.279 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.280+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.279 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.280+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2398106829}) 2024-04-26T00:36:46.279 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.280+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3779519402}) 2024-04-26T00:36:46.279 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.280+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.279 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.280+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2398106829}) 2024-04-26T00:36:46.279 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.280+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1335481483}) 2024-04-26T00:36:46.281 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.280+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.281 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.280+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=167688307}) 2024-04-26T00:36:46.281 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.280+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3019213805}) 2024-04-26T00:36:46.281 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.280+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.281 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.280+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=167688307}) 2024-04-26T00:36:46.281 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.280+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=587012084}) 2024-04-26T00:36:46.283 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.284+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.284 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.284+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1465371758}) 2024-04-26T00:36:46.284 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.284+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1096841654}) 2024-04-26T00:36:46.284 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.284+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.284 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.284+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1465371758}) 2024-04-26T00:36:46.284 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.284+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2813951379}) 2024-04-26T00:36:46.286 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.284+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.286 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.284+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=760993200}) 2024-04-26T00:36:46.286 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.284+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=465418560}) 2024-04-26T00:36:46.286 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.284+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.286 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.284+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=760993200}) 2024-04-26T00:36:46.286 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.284+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3582963827}) 2024-04-26T00:36:46.288 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.288+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.288 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.288+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3374235772}) 2024-04-26T00:36:46.288 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.288+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4086434098}) 2024-04-26T00:36:46.288 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.288+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.288 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.288+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3374235772}) 2024-04-26T00:36:46.288 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.288+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2949674126}) 2024-04-26T00:36:46.290 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.288+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.290 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.288+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3688518250}) 2024-04-26T00:36:46.290 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.288+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2043786183}) 2024-04-26T00:36:46.290 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.288+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.290 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.288+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3688518250}) 2024-04-26T00:36:46.291 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.288+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2081205907}) 2024-04-26T00:36:46.292 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.292+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.292 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.292+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2226998682}) 2024-04-26T00:36:46.292 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.292+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3079239942}) 2024-04-26T00:36:46.293 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.292+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.293 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.292+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2226998682}) 2024-04-26T00:36:46.293 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.292+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=53405156}) 2024-04-26T00:36:46.295 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.296+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.295 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.296+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3230700013}) 2024-04-26T00:36:46.295 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.296+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3923836846}) 2024-04-26T00:36:46.295 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.296+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.295 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.296+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3230700013}) 2024-04-26T00:36:46.295 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.296+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=951064579}) 2024-04-26T00:36:46.297 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.296+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.297 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.296+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3675744277}) 2024-04-26T00:36:46.297 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.296+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1407200801}) 2024-04-26T00:36:46.297 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.296+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.297 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.296+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3675744277}) 2024-04-26T00:36:46.297 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.296+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2780970556}) 2024-04-26T00:36:46.299 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.300+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.299 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.300+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1375983838}) 2024-04-26T00:36:46.299 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.300+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3734357609}) 2024-04-26T00:36:46.299 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.300+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.299 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.300+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1375983838}) 2024-04-26T00:36:46.300 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.300+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1099596897}) 2024-04-26T00:36:46.301 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.300+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.301 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.300+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2142172761}) 2024-04-26T00:36:46.302 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.300+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=486726351}) 2024-04-26T00:36:46.302 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.300+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.302 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.300+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2142172761}) 2024-04-26T00:36:46.302 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.300+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3627430753}) 2024-04-26T00:36:46.304 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.304+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.304 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.304+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2315283227}) 2024-04-26T00:36:46.304 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.304+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2405629358}) 2024-04-26T00:36:46.304 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.304+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.304 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.304+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2315283227}) 2024-04-26T00:36:46.304 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.304+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=202267418}) 2024-04-26T00:36:46.306 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.304+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.306 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.304+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2380393480}) 2024-04-26T00:36:46.306 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.304+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=390271650}) 2024-04-26T00:36:46.306 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.304+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.306 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.304+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2380393480}) 2024-04-26T00:36:46.306 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.304+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1368051355}) 2024-04-26T00:36:46.308 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.308+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.308 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.308+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2386798888}) 2024-04-26T00:36:46.308 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.308+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4017414060}) 2024-04-26T00:36:46.308 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.308+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.309 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.308+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2386798888}) 2024-04-26T00:36:46.309 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.308+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2438065100}) 2024-04-26T00:36:46.311 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.312+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.311 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.312+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1768388305}) 2024-04-26T00:36:46.311 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.312+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1008480709}) 2024-04-26T00:36:46.311 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.312+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.311 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.312+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1768388305}) 2024-04-26T00:36:46.311 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.312+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2317179456}) 2024-04-26T00:36:46.313 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.312+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.313 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.312+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3523167091}) 2024-04-26T00:36:46.313 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.312+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2899993590}) 2024-04-26T00:36:46.313 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.312+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.313 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.312+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3523167091}) 2024-04-26T00:36:46.313 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.312+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=394578587}) 2024-04-26T00:36:46.315 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.316+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.315 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.316+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3072489206}) 2024-04-26T00:36:46.315 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.316+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1830807709}) 2024-04-26T00:36:46.315 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.316+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.315 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.316+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3072489206}) 2024-04-26T00:36:46.315 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.316+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2908747733}) 2024-04-26T00:36:46.317 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.316+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.317 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.316+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3791128043}) 2024-04-26T00:36:46.317 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.316+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2474437322}) 2024-04-26T00:36:46.317 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.316+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.317 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.316+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3791128043}) 2024-04-26T00:36:46.317 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.316+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2832423175}) 2024-04-26T00:36:46.319 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.320+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.319 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.320+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=39841765}) 2024-04-26T00:36:46.319 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.320+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=766939439}) 2024-04-26T00:36:46.320 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.320+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.320 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.320+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=39841765}) 2024-04-26T00:36:46.320 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.320+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2804083555}) 2024-04-26T00:36:46.322 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.320+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.322 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.320+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1547116773}) 2024-04-26T00:36:46.322 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.320+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3080964975}) 2024-04-26T00:36:46.322 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.320+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.322 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.320+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1547116773}) 2024-04-26T00:36:46.322 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.320+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=233239262}) 2024-04-26T00:36:46.324 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.324+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.324 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.324+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1335459836}) 2024-04-26T00:36:46.324 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.324+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1956065638}) 2024-04-26T00:36:46.324 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.324+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.324 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.324+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1335459836}) 2024-04-26T00:36:46.324 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.324+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3089194786}) 2024-04-26T00:36:46.326 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.324+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.326 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.324+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=396335296}) 2024-04-26T00:36:46.326 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.324+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2135264045}) 2024-04-26T00:36:46.326 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.324+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.326 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.324+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=396335296}) 2024-04-26T00:36:46.326 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.324+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1183595442}) 2024-04-26T00:36:46.328 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.328+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.328 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.328+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3569572450}) 2024-04-26T00:36:46.328 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.328+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2058525436}) 2024-04-26T00:36:46.328 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.328+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.328 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.328+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3569572450}) 2024-04-26T00:36:46.328 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.328+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2801692657}) 2024-04-26T00:36:46.330 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.328+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.330 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.328+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2810794781}) 2024-04-26T00:36:46.330 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.328+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1744683971}) 2024-04-26T00:36:46.330 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.328+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.330 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.328+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2810794781}) 2024-04-26T00:36:46.330 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.328+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2491355886}) 2024-04-26T00:36:46.332 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.332+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.332 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.332+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2638164407}) 2024-04-26T00:36:46.332 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.332+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1623811166}) 2024-04-26T00:36:46.332 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.332+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.332 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.332+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2638164407}) 2024-04-26T00:36:46.333 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.332+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4021818270}) 2024-04-26T00:36:46.334 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.336+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.334 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.336+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2177908624}) 2024-04-26T00:36:46.334 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.336+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1112632467}) 2024-04-26T00:36:46.335 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.336+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.335 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.336+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2177908624}) 2024-04-26T00:36:46.335 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.336+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2672517069}) 2024-04-26T00:36:46.337 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.336+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.337 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.336+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1559799220}) 2024-04-26T00:36:46.337 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.336+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2223898054}) 2024-04-26T00:36:46.337 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.336+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.337 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.336+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1559799220}) 2024-04-26T00:36:46.337 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.336+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2302191088}) 2024-04-26T00:36:46.339 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.340+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.339 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.340+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4275386221}) 2024-04-26T00:36:46.339 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.340+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4131413613}) 2024-04-26T00:36:46.339 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.340+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.339 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.340+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4275386221}) 2024-04-26T00:36:46.339 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.340+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1706627341}) 2024-04-26T00:36:46.340 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.340+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.340 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.340+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1075295491}) 2024-04-26T00:36:46.340 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.340+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3936438894}) 2024-04-26T00:36:46.340 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.340+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.340 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.340+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1075295491}) 2024-04-26T00:36:46.341 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.340+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2677217208}) 2024-04-26T00:36:46.343 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.344+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.343 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.344+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=27,mdsmap=3,mgr=7,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29} crc {logm=3617073518,mdsmap=2840952586,mgr=1137150072,mgr_metadata=3686404438,mgrstat=3075843522,mon_config_key=4061063117,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=2156411766,osd_snap=3027231052}) 2024-04-26T00:36:46.343 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.344+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=29,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=12,mon_config_key=10,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=28} crc {logm=2543694477,mdsmap=2840952586,mgr=1682877445,mgr_metadata=3686404438,mgrstat=173617542,mon_config_key=752439442,monmap=3876866194,osd_metadata=2387245979,osd_pg_creating=2156411766,osd_snap=2167756900}) 2024-04-26T00:36:46.343 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.344+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.343 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.344+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=27,mdsmap=3,mgr=7,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29} crc {logm=3617073518,mdsmap=2840952586,mgr=1137150072,mgr_metadata=3686404438,mgrstat=3075843522,mon_config_key=4061063117,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=2156411766,osd_snap=3027231052}) 2024-04-26T00:36:46.343 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.344+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=28,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=28} crc {logm=286580910,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=3285793343,mon_config_key=4210576201,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=2156411766,osd_snap=1923760120}) 2024-04-26T00:36:46.345 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.344+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.345 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.344+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=88,osdmap=12} crc {osd_snap=1089378201,osdmap=3702102149}) 2024-04-26T00:36:46.345 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.344+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=90,osdmap=10} crc {osd_snap=3321928337,osdmap=1300804929}) 2024-04-26T00:36:46.345 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.344+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.345 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.344+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=88,osdmap=12} crc {osd_snap=1089378201,osdmap=3702102149}) 2024-04-26T00:36:46.345 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.344+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=92,osdmap=8} crc {osd_snap=823639870,osdmap=3952470050}) 2024-04-26T00:36:46.348 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.348+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.348 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.348+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2375232553}) 2024-04-26T00:36:46.348 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.348+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=568505963}) 2024-04-26T00:36:46.348 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.348+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.348 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.348+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2375232553}) 2024-04-26T00:36:46.348 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.348+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1110543964}) 2024-04-26T00:36:46.350 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.348+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.350 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.348+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=724011278}) 2024-04-26T00:36:46.350 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.348+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2212785272}) 2024-04-26T00:36:46.350 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.348+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.350 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.348+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=724011278}) 2024-04-26T00:36:46.350 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.348+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4083344231}) 2024-04-26T00:36:46.352 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.352+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.352 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.352+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2380272946}) 2024-04-26T00:36:46.352 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.352+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=36424404}) 2024-04-26T00:36:46.352 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.352+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.353 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.352+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2380272946}) 2024-04-26T00:36:46.353 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.352+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2748196124}) 2024-04-26T00:36:46.356 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.356+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.356 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.356+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4260103152}) 2024-04-26T00:36:46.356 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.356+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2056814617}) 2024-04-26T00:36:46.356 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.356+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.356 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.356+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4260103152}) 2024-04-26T00:36:46.357 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.356+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=879395720}) 2024-04-26T00:36:46.366 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.364+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.366 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.364+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=21} crc {osdmap=1560892524}) 2024-04-26T00:36:46.366 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.364+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=19} crc {osdmap=3482988122}) 2024-04-26T00:36:46.366 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.364+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:36:46.366 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.364+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=21} crc {osdmap=1560892524}) 2024-04-26T00:36:46.366 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:36:46.364+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=20} crc {osdmap=2282755904}) 2024-04-26T00:36:46.426 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:36:46.426 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:36:46.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.556+0000 7f2df22a7640 10 monclient: get_monmap_and_config 2024-04-26T00:36:46.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.556+0000 7f2df22a7640 10 monclient: build_initial_monmap 2024-04-26T00:36:46.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.556+0000 7f2df22a7640 10 monclient: monmap: 2024-04-26T00:36:46.559 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:36:46.559 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:46.559 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:36:46.563763+0000 2024-04-26T00:36:46.559 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:36:46.563763+0000 2024-04-26T00:36:46.559 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:36:46.559 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:46.559 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:36:46.559 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:36:46.559 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:36:46.559 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:36:46.559 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:36:46.559 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:46.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.560+0000 7f2df22a7640 1 Processor -- start 2024-04-26T00:36:46.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.560+0000 7f2df22a7640 1 -- start start 2024-04-26T00:36:46.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.560+0000 7f2df22a7640 10 monclient: init 2024-04-26T00:36:46.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.560+0000 7f2df22a7640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:46.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.560+0000 7f2df22a7640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-04-26T00:36:46.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.560+0000 7f2df22a7640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f2dec0a0260 0x7f2dec0a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:46.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.560+0000 7f2df22a7640 10 monclient(hunting): picked mon.noname-a con 0x7f2dec0a0260 addr v2:172.21.15.37:3300/0 2024-04-26T00:36:46.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.560+0000 7f2df22a7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2dec134b30 addr v1:172.21.15.45:6789/0 2024-04-26T00:36:46.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.560+0000 7f2df22a7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2dec135660 addr v1:172.21.15.37:6789/0 2024-04-26T00:36:46.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.560+0000 7f2df22a7640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2dec0a1130 con 0x7f2dec135660 2024-04-26T00:36:46.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.560+0000 7f2df22a7640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2dec06cff0 con 0x7f2dec134b30 2024-04-26T00:36:46.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.560+0000 7f2df22a7640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:46.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.560+0000 7f2df22a7640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f2dec09ec10 con 0x7f2dec0a0260 2024-04-26T00:36:46.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.560+0000 7f2df22a7640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:46.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.560+0000 7f2df22a7640 10 monclient(hunting): authenticate will time out at 1987.869708s 2024-04-26T00:36:46.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.560+0000 7f2debfff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f2dec0a0260 0x7f2dec0a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:46.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.560+0000 7f2df0844640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f2dec135660 0x7f2dec13ce70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:34022/0 (socket says 172.21.15.37:34022) 2024-04-26T00:36:46.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.560+0000 7f2df0844640 1 -- 172.21.15.37:0/1065638949 learned_addr learned my addr 172.21.15.37:0/1065638949 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:36:46.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.560+0000 7f2debfff640 10 monclient(hunting): get_auth_request con 0x7f2dec0a0260 auth_method 0 2024-04-26T00:36:46.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.560+0000 7f2debfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:46.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.560+0000 7f2debfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:46.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.560+0000 7f2debfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:46.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.560+0000 7f2debfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:46.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.560+0000 7f2debfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:46.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.560+0000 7f2debfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:46.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.560+0000 7f2debfff640 10 monclient(hunting): handle_auth_done global_id 144723 payload 274 2024-04-26T00:36:46.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.560+0000 7f2debfff640 1 -- 172.21.15.37:0/1065638949 >> v1:172.21.15.45:6789/0 conn(0x7f2dec134b30 legacy=0x7f2dec134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:36:46.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.560+0000 7f2debfff640 1 -- 172.21.15.37:0/1065638949 >> v1:172.21.15.37:6789/0 conn(0x7f2dec135660 legacy=0x7f2dec13ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:36:46.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.560+0000 7f2debfff640 10 monclient: _finish_hunting 0 2024-04-26T00:36:46.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.560+0000 7f2debfff640 1 monclient: found mon.noname-a 2024-04-26T00:36:46.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.560+0000 7f2debfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:46.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.560+0000 7f2debfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:36:46.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.560+0000 7f2debfff640 1 -- 172.21.15.37:0/1065638949 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dec09bc50 con 0x7f2dec0a0260 2024-04-26T00:36:46.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.564+0000 7f2debfff640 1 -- 172.21.15.37:0/1065638949 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2dec0a0260 msgr2=0x7f2dec0a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:46.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.564+0000 7f2debfff640 1 --2- 172.21.15.37:0/1065638949 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2dec0a0260 0x7f2dec0a0640 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc009d90 tx=0x7f2ddc01c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=656e20512c607ab3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:46.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.564+0000 7f2deaffd640 1 -- 172.21.15.37:0/1065638949 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ddc0a6040 con 0x7f2dec0a0260 2024-04-26T00:36:46.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.564+0000 7f2deaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:46.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.564+0000 7f2deaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:36:46.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.564+0000 7f2deaffd640 10 monclient: dump: 2024-04-26T00:36:46.571 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:46.571 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:46.571 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:46.571 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:46.571 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:46.571 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:46.571 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:46.571 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:46.572 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:46.572 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:46.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.564+0000 7f2deaffd640 10 monclient: _finish_auth 0 2024-04-26T00:36:46.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.564+0000 7f2deaffd640 10 monclient: _check_auth_tickets 2024-04-26T00:36:46.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.564+0000 7f2deaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:46.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.564+0000 7f2deaffd640 1 -- 172.21.15.37:0/1065638949 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ddc0af070 con 0x7f2dec0a0260 2024-04-26T00:36:46.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.564+0000 7f2deaffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:36:46.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.564+0000 7f2deaffd640 1 -- 172.21.15.37:0/1065638949 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ddc0aa940 con 0x7f2dec0a0260 2024-04-26T00:36:46.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.564+0000 7f2deaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:46.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.564+0000 7f2deaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:36:46.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.564+0000 7f2deaffd640 10 monclient: dump: 2024-04-26T00:36:46.572 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:46.572 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:46.573 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:46.573 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:46.573 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:46.573 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:46.573 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:46.573 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:46.573 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:46.573 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:46.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.564+0000 7f2df22a7640 5 monclient: authenticate success, global_id 144723 2024-04-26T00:36:46.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.564+0000 7f2df22a7640 10 monclient: get_monmap_and_config success 2024-04-26T00:36:46.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.564+0000 7f2df22a7640 4 set_mon_vals no callback set 2024-04-26T00:36:46.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2df22a7640 10 monclient: shutdown 2024-04-26T00:36:46.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2df22a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:36:46.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2df22a7640 1 -- 172.21.15.37:0/1065638949 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2dec0a0260 msgr2=0x7f2dec0a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:46.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2df22a7640 1 --2- 172.21.15.37:0/1065638949 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2dec0a0260 0x7f2dec0a0640 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc009d90 tx=0x7f2ddc01c5d0 comp rx=0 tx=0).stop 2024-04-26T00:36:46.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2df22a7640 1 -- 172.21.15.37:0/1065638949 shutdown_connections 2024-04-26T00:36:46.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2df22a7640 1 --2- 172.21.15.37:0/1065638949 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2dec0a0260 0x7f2dec0a0640 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:46.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2df22a7640 1 -- 172.21.15.37:0/1065638949 >> 172.21.15.37:0/1065638949 conn(0x7f2dec0a1ad0 msgr2=0x7f2dec0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:36:46.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2df22a7640 1 -- 172.21.15.37:0/1065638949 shutdown_connections 2024-04-26T00:36:46.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2df22a7640 1 -- 172.21.15.37:0/1065638949 wait complete. 2024-04-26T00:36:46.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2df22a7640 10 monclient: build_initial_monmap 2024-04-26T00:36:46.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2df22a7640 10 monclient: monmap: 2024-04-26T00:36:46.574 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:36:46.574 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:36:46.574 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:36:46.575 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:36:46.575 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:36:46.575 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:36:46.575 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:36:46.575 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:36:46.575 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:36:46.575 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:46.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2df22a7640 1 Processor -- start 2024-04-26T00:36:46.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2df22a7640 1 -- start start 2024-04-26T00:36:46.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2df22a7640 10 monclient: init 2024-04-26T00:36:46.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2df22a7640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:46.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2df22a7640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:36:46.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2df22a7640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2dec0a0260 0x7f2dec1cf0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:46.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2df22a7640 10 monclient(hunting): picked mon.noname-a con 0x7f2dec0a0260 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:36:46.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2df22a7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dec134b30 0x7f2dec131940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:46.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2df22a7640 10 monclient(hunting): picked mon.noname-b con 0x7f2dec134b30 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:36:46.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2df22a7640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2dec135660 0x7f2dec09c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:46.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2df22a7640 10 monclient(hunting): picked mon.noname-c con 0x7f2dec135660 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:36:46.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2df22a7640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:46.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2df22a7640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f2dec09f4c0 con 0x7f2dec0a0260 2024-04-26T00:36:46.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2df22a7640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:46.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2df22a7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2dec09f0f0 con 0x7f2dec134b30 2024-04-26T00:36:46.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2df22a7640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:46.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2df22a7640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2dec140db0 con 0x7f2dec135660 2024-04-26T00:36:46.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2df22a7640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:46.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2df22a7640 10 monclient(hunting): authenticate will time out at 1987.878142s 2024-04-26T00:36:46.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2debfff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2dec0a0260 0x7f2dec1cf0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:46.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2debfff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2dec0a0260 0x7f2dec1cf0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50362/0 (socket says 172.21.15.37:50362) 2024-04-26T00:36:46.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2debfff640 1 -- 172.21.15.37:0/1770057108 learned_addr learned my addr 172.21.15.37:0/1770057108 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:36:46.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2debfff640 10 monclient(hunting): get_auth_request con 0x7f2dec0a0260 auth_method 0 2024-04-26T00:36:46.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2debfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:46.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2debfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:46.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2debfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:46.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2df0844640 1 --2- 172.21.15.37:0/1770057108 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2dec135660 0x7f2dec09c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:46.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2debfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:46.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2debfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:46.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2deb7fe640 1 --2- 172.21.15.37:0/1770057108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dec134b30 0x7f2dec131940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:46.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2debfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:46.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2df0844640 10 monclient(hunting): get_auth_request con 0x7f2dec135660 auth_method 0 2024-04-26T00:36:46.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2df0844640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:46.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2df0844640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:46.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2df0844640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:46.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2deb7fe640 10 monclient(hunting): get_auth_request con 0x7f2dec134b30 auth_method 0 2024-04-26T00:36:46.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2deb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:46.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2deb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:46.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2deb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:46.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2df0844640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:46.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2df0844640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:46.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2df0844640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:46.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2deb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:46.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2deb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:46.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2deb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:46.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2debfff640 10 monclient(hunting): handle_auth_done global_id 144726 payload 824 2024-04-26T00:36:46.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2debfff640 1 -- 172.21.15.37:0/1770057108 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2dec135660 msgr2=0x7f2dec09c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:46.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2debfff640 1 --2- 172.21.15.37:0/1770057108 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2dec135660 0x7f2dec09c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:46.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2debfff640 1 -- 172.21.15.37:0/1770057108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dec134b30 msgr2=0x7f2dec131940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:46.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2debfff640 1 --2- 172.21.15.37:0/1770057108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dec134b30 0x7f2dec131940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:46.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2debfff640 10 monclient: _finish_hunting 0 2024-04-26T00:36:46.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2debfff640 1 monclient: found mon.noname-a 2024-04-26T00:36:46.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2debfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:46.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2debfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:36:46.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2debfff640 1 -- 172.21.15.37:0/1770057108 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dec1348a0 con 0x7f2dec0a0260 2024-04-26T00:36:46.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.568+0000 7f2df0844640 1 --2- 172.21.15.37:0/1770057108 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2dec135660 0x7f2dec09c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:36:46.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.572+0000 7f2debfff640 1 -- 172.21.15.37:0/1770057108 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2dec0a0260 msgr2=0x7f2dec1cf0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:46.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.572+0000 7f2debfff640 1 --2- 172.21.15.37:0/1770057108 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2dec0a0260 0x7f2dec1cf0a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc01a200 tx=0x7f2ddc019370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:46.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.572+0000 7f2de8ff9640 1 -- 172.21.15.37:0/1770057108 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ddc0aab60 con 0x7f2dec0a0260 2024-04-26T00:36:46.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.572+0000 7f2de8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:46.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.572+0000 7f2de8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:36:46.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.572+0000 7f2de8ff9640 10 monclient: dump: 2024-04-26T00:36:46.582 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:46.582 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:46.582 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:46.582 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:46.582 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:46.582 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:46.582 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:46.582 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:46.582 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:46.582 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:46.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.572+0000 7f2de8ff9640 10 monclient: _finish_auth 0 2024-04-26T00:36:46.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.572+0000 7f2de8ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:36:46.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.572+0000 7f2de8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:46.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.572+0000 7f2de8ff9640 1 -- 172.21.15.37:0/1770057108 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ddc0a6030 con 0x7f2dec0a0260 2024-04-26T00:36:46.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.572+0000 7f2de8ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:36:46.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.572+0000 7f2de8ff9640 1 -- 172.21.15.37:0/1770057108 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ddc0be430 con 0x7f2dec0a0260 2024-04-26T00:36:46.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.572+0000 7f2de8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:46.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.572+0000 7f2de8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:36:46.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.572+0000 7f2de8ff9640 10 monclient: dump: 2024-04-26T00:36:46.583 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:46.583 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:46.583 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:46.583 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:46.583 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:46.583 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:46.583 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:46.583 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:46.583 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:46.584 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:46.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.572+0000 7f2df22a7640 5 monclient: authenticate success, global_id 144726 2024-04-26T00:36:46.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.572+0000 7f2df22a7640 10 monclient: _renew_subs 2024-04-26T00:36:46.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.572+0000 7f2df22a7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:46.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.572+0000 7f2de9ffb640 4 set_mon_vals no callback set 2024-04-26T00:36:46.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.572+0000 7f2df22a7640 1 -- 172.21.15.37:0/1770057108 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dec0729d0 con 0x7f2dec0a0260 2024-04-26T00:36:46.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.572+0000 7f2df22a7640 10 monclient: _renew_subs 2024-04-26T00:36:46.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.572+0000 7f2df22a7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:46.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.572+0000 7f2df22a7640 1 -- 172.21.15.37:0/1770057108 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dec11d2e0 con 0x7f2dec0a0260 2024-04-26T00:36:46.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.572+0000 7f2df22a7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:46.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.572+0000 7f2df22a7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:46.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.572+0000 7f2df22a7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:46.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.572+0000 7f2df22a7640 1 -- 172.21.15.37:0/1770057108 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dec066f40 con 0x7f2dec0a0260 2024-04-26T00:36:46.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.572+0000 7f2de8ff9640 1 -- 172.21.15.37:0/1770057108 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2ddc0c7440 con 0x7f2dec0a0260 2024-04-26T00:36:46.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.572+0000 7f2de8ff9640 1 --2- 172.21.15.37:0/1770057108 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2dcc035ed0 0x7f2dcc038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:46.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.572+0000 7f2de8ff9640 1 -- 172.21.15.37:0/1770057108 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(661..661 src has 370..661) ==== 3927+0+0 (secure 0 0 0) 0x7f2ddc0e34b0 con 0x7f2dec0a0260 2024-04-26T00:36:46.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.572+0000 7f2deb7fe640 1 --2- 172.21.15.37:0/1770057108 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2dcc035ed0 0x7f2dcc038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:46.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.572+0000 7f2deb7fe640 10 monclient: get_auth_request con 0x7f2dcc035ed0 auth_method 0 2024-04-26T00:36:46.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.576+0000 7f2deb7fe640 1 --2- 172.21.15.37:0/1770057108 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2dcc035ed0 0x7f2dcc038390 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f2dec132a70 tx=0x7f2dd4007e40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:46.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.576+0000 7f2de8ff9640 1 -- 172.21.15.37:0/1770057108 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f2dec0a1130 con 0x7f2dec0a0260 2024-04-26T00:36:46.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.576+0000 7f2de8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:46.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.576+0000 7f2de8ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:36:46.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.800+0000 7f2df22a7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:36:46.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.800+0000 7f2df22a7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:36:46.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.800+0000 7f2df22a7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:46.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.800+0000 7f2df22a7640 1 -- 172.21.15.37:0/1770057108 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2dec11cf60 con 0x7f2dec0a0260 2024-04-26T00:36:46.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.800+0000 7f2de8ff9640 1 -- 172.21.15.37:0/1770057108 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f2ddc0e3250 con 0x7f2dec0a0260 2024-04-26T00:36:46.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.800+0000 7f2de8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:36:46.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.800+0000 7f2de8ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:36:46.800 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":126,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:36:46.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.800+0000 7f2df22a7640 1 -- 172.21.15.37:0/1770057108 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2dcc035ed0 msgr2=0x7f2dcc038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:46.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.800+0000 7f2df22a7640 1 --2- 172.21.15.37:0/1770057108 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2dcc035ed0 0x7f2dcc038390 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f2dec132a70 tx=0x7f2dd4007e40 comp rx=0 tx=0).stop 2024-04-26T00:36:46.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.800+0000 7f2df22a7640 10 monclient: shutdown 2024-04-26T00:36:46.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.800+0000 7f2df22a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:36:46.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.800+0000 7f2df22a7640 1 -- 172.21.15.37:0/1770057108 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2dec0a0260 msgr2=0x7f2dec1cf0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:46.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.800+0000 7f2df22a7640 1 --2- 172.21.15.37:0/1770057108 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2dec0a0260 0x7f2dec1cf0a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc01a200 tx=0x7f2ddc019370 comp rx=0 tx=0).stop 2024-04-26T00:36:46.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.800+0000 7f2df22a7640 1 -- 172.21.15.37:0/1770057108 shutdown_connections 2024-04-26T00:36:46.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.800+0000 7f2df22a7640 1 --2- 172.21.15.37:0/1770057108 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2dcc035ed0 0x7f2dcc038390 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:46.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.800+0000 7f2df22a7640 1 --2- 172.21.15.37:0/1770057108 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2dec135660 0x7f2dec09c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:46.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.800+0000 7f2df22a7640 1 --2- 172.21.15.37:0/1770057108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2dec134b30 0x7f2dec131940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:46.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.800+0000 7f2df22a7640 1 --2- 172.21.15.37:0/1770057108 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2dec0a0260 0x7f2dec1cf0a0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:46.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.800+0000 7f2df22a7640 1 -- 172.21.15.37:0/1770057108 >> 172.21.15.37:0/1770057108 conn(0x7f2dec0a1ad0 msgr2=0x7f2dec0a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:36:46.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.800+0000 7f2df22a7640 1 -- 172.21.15.37:0/1770057108 shutdown_connections 2024-04-26T00:36:46.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.800+0000 7f2df22a7640 1 -- 172.21.15.37:0/1770057108 wait complete. 2024-04-26T00:36:46.819 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:36:46.819 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-26T00:36:46.819 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:36:46.819 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:36:46.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.936+0000 7fb2d629d640 10 monclient: get_monmap_and_config 2024-04-26T00:36:46.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.936+0000 7fb2d629d640 10 monclient: build_initial_monmap 2024-04-26T00:36:46.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2d629d640 10 monclient: monmap: 2024-04-26T00:36:46.939 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:36:46.939 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:46.939 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:36:46.943901+0000 2024-04-26T00:36:46.939 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:36:46.943901+0000 2024-04-26T00:36:46.939 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:36:46.939 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:46.939 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:36:46.939 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:36:46.939 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:36:46.939 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:36:46.939 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:36:46.940 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:46.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2d629d640 1 Processor -- start 2024-04-26T00:36:46.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2d629d640 1 -- start start 2024-04-26T00:36:46.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2d629d640 10 monclient: init 2024-04-26T00:36:46.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2d629d640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:46.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2d629d640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-04-26T00:36:46.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2d629d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb2d00726b0 0x7fb2d0072a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:46.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2d629d640 10 monclient(hunting): picked mon.noname-c con 0x7fb2d00726b0 addr v2:172.21.15.45:3300/0 2024-04-26T00:36:46.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2d629d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb2d00a0110 0x7fb2d0136dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:46.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2d629d640 10 monclient(hunting): picked mon.noname-a con 0x7fb2d00a0110 addr v2:172.21.15.37:3300/0 2024-04-26T00:36:46.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2d629d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb2d0137310 addr v1:172.21.15.37:6789/0 2024-04-26T00:36:46.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2d629d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2d00a1470 con 0x7fb2d0137310 2024-04-26T00:36:46.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2d629d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:46.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2d629d640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fb2d00a1bd0 con 0x7fb2d00a0110 2024-04-26T00:36:46.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2d629d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:46.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2d629d640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fb2d00a0eb0 con 0x7fb2d00726b0 2024-04-26T00:36:46.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2d629d640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:46.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2d629d640 10 monclient(hunting): authenticate will time out at 1988.249561s 2024-04-26T00:36:46.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2cf7fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb2d00a0110 0x7fb2d0136dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:46.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2d483a640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fb2d0137310 0x7fb2d013e100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:34028/0 (socket says 172.21.15.37:34028) 2024-04-26T00:36:46.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2d483a640 1 -- 172.21.15.37:0/2026675309 learned_addr learned my addr 172.21.15.37:0/2026675309 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:36:46.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2cf7fe640 10 monclient(hunting): get_auth_request con 0x7fb2d00a0110 auth_method 0 2024-04-26T00:36:46.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:46.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2cf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:46.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:46.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:46.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:46.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:46.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2cffff640 1 --2- 172.21.15.37:0/2026675309 >> v2:172.21.15.45:3300/0 conn(0x7fb2d00726b0 0x7fb2d0072a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:46.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2cffff640 10 monclient(hunting): get_auth_request con 0x7fb2d00726b0 auth_method 0 2024-04-26T00:36:46.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:46.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2cffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:46.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2cffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:46.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2cf7fe640 10 monclient(hunting): handle_auth_done global_id 144729 payload 274 2024-04-26T00:36:46.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2cf7fe640 1 -- 172.21.15.37:0/2026675309 >> v2:172.21.15.45:3300/0 conn(0x7fb2d00726b0 msgr2=0x7fb2d0072a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:36:46.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2cf7fe640 1 --2- 172.21.15.37:0/2026675309 >> v2:172.21.15.45:3300/0 conn(0x7fb2d00726b0 0x7fb2d0072a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:46.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2cf7fe640 1 -- 172.21.15.37:0/2026675309 >> v1:172.21.15.37:6789/0 conn(0x7fb2d0137310 legacy=0x7fb2d013e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:36:46.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2cf7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:36:46.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2cf7fe640 1 monclient: found mon.noname-a 2024-04-26T00:36:46.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:46.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2cf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:36:46.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.940+0000 7fb2cf7fe640 1 -- 172.21.15.37:0/2026675309 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2d00a0bd0 con 0x7fb2d00a0110 2024-04-26T00:36:46.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.944+0000 7fb2cf7fe640 1 -- 172.21.15.37:0/2026675309 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb2d00a0110 msgr2=0x7fb2d0136dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:46.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.944+0000 7fb2cf7fe640 1 --2- 172.21.15.37:0/2026675309 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb2d00a0110 0x7fb2d0136dd0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb2d00764d0 tx=0x7fb2c401cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=621accfe92a729f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:46.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.944+0000 7fb2ceffd640 1 -- 172.21.15.37:0/2026675309 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c40b0070 con 0x7fb2d00a0110 2024-04-26T00:36:46.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.944+0000 7fb2ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:46.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.944+0000 7fb2ceffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:36:46.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.944+0000 7fb2ceffd640 10 monclient: dump: 2024-04-26T00:36:46.946 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:46.946 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:46.946 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:46.946 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:46.946 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:46.947 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:46.947 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:46.947 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:46.947 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:46.947 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:46.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.944+0000 7fb2ceffd640 10 monclient: _finish_auth 0 2024-04-26T00:36:46.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.944+0000 7fb2ceffd640 10 monclient: _check_auth_tickets 2024-04-26T00:36:46.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.944+0000 7fb2ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:46.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.944+0000 7fb2ceffd640 1 -- 172.21.15.37:0/2026675309 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2c40ab600 con 0x7fb2d00a0110 2024-04-26T00:36:46.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.944+0000 7fb2ceffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:36:46.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.944+0000 7fb2ceffd640 1 -- 172.21.15.37:0/2026675309 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c40ab9d0 con 0x7fb2d00a0110 2024-04-26T00:36:46.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.944+0000 7fb2d629d640 5 monclient: authenticate success, global_id 144729 2024-04-26T00:36:46.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.944+0000 7fb2d629d640 10 monclient: get_monmap_and_config success 2024-04-26T00:36:46.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.944+0000 7fb2d629d640 4 set_mon_vals no callback set 2024-04-26T00:36:46.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.944+0000 7fb2ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:46.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.944+0000 7fb2ceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:36:46.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.944+0000 7fb2ceffd640 10 monclient: dump: 2024-04-26T00:36:46.948 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:46.948 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:46.948 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:46.948 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:46.948 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:46.948 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:46.948 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:46.948 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:46.948 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:46.948 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:46.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.944+0000 7fb2d629d640 10 monclient: shutdown 2024-04-26T00:36:46.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.944+0000 7fb2d629d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:36:46.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.944+0000 7fb2d629d640 1 -- 172.21.15.37:0/2026675309 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb2d00a0110 msgr2=0x7fb2d0136dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:46.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.944+0000 7fb2d629d640 1 --2- 172.21.15.37:0/2026675309 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb2d00a0110 0x7fb2d0136dd0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb2d00764d0 tx=0x7fb2c401cb30 comp rx=0 tx=0).stop 2024-04-26T00:36:46.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.944+0000 7fb2d629d640 1 -- 172.21.15.37:0/2026675309 shutdown_connections 2024-04-26T00:36:46.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.944+0000 7fb2d629d640 1 --2- 172.21.15.37:0/2026675309 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb2d00a0110 0x7fb2d0136dd0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:46.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.944+0000 7fb2d629d640 1 --2- 172.21.15.37:0/2026675309 >> v2:172.21.15.45:3300/0 conn(0x7fb2d00726b0 0x7fb2d0072a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:46.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.944+0000 7fb2d629d640 1 -- 172.21.15.37:0/2026675309 >> 172.21.15.37:0/2026675309 conn(0x7fb2d00a1d50 msgr2=0x7fb2d00a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:36:46.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.944+0000 7fb2d629d640 1 -- 172.21.15.37:0/2026675309 shutdown_connections 2024-04-26T00:36:46.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.944+0000 7fb2d629d640 1 -- 172.21.15.37:0/2026675309 wait complete. 2024-04-26T00:36:46.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2d629d640 10 monclient: build_initial_monmap 2024-04-26T00:36:46.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2d629d640 10 monclient: monmap: 2024-04-26T00:36:46.954 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:36:46.954 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:36:46.954 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:36:46.954 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:36:46.954 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:36:46.954 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:36:46.954 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:36:46.954 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:36:46.954 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:36:46.954 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:46.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2d629d640 1 Processor -- start 2024-04-26T00:36:46.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2d629d640 1 -- start start 2024-04-26T00:36:46.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2d629d640 10 monclient: init 2024-04-26T00:36:46.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2d629d640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:46.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2d629d640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:36:46.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2d629d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb2d00726b0 0x7fb2d01cbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:46.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2d629d640 10 monclient(hunting): picked mon.noname-a con 0x7fb2d00726b0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:36:46.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2d629d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb2d0137310 0x7fb2d01cc500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:46.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2d629d640 10 monclient(hunting): picked mon.noname-c con 0x7fb2d0137310 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:36:46.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2d629d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2d01cca40 0x7fb2d01cf4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:46.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2d629d640 10 monclient(hunting): picked mon.noname-b con 0x7fb2d01cca40 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:36:46.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2d629d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:46.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2d629d640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d009bc50 con 0x7fb2d00726b0 2024-04-26T00:36:46.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2d629d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:46.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2d629d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d00730c0 con 0x7fb2d01cca40 2024-04-26T00:36:46.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2d629d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:46.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2d629d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb2d009bdd0 con 0x7fb2d0137310 2024-04-26T00:36:46.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2d629d640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:46.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2d629d640 10 monclient(hunting): authenticate will time out at 1988.257208s 2024-04-26T00:36:46.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2cf7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb2d0137310 0x7fb2d01cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:46.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2cffff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb2d00726b0 0x7fb2d01cbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:46.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2cf7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb2d0137310 0x7fb2d01cc500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:59504/0 (socket says 172.21.15.37:59504) 2024-04-26T00:36:46.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2cf7fe640 1 -- 172.21.15.37:0/661492814 learned_addr learned my addr 172.21.15.37:0/661492814 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:36:46.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2cffff640 10 monclient(hunting): get_auth_request con 0x7fb2d00726b0 auth_method 0 2024-04-26T00:36:46.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:46.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2cffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:46.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2cffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:46.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2cf7fe640 10 monclient(hunting): get_auth_request con 0x7fb2d0137310 auth_method 0 2024-04-26T00:36:46.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:46.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2cf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:46.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:46.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:46.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:46.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:46.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:46.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:46.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:46.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2d483a640 1 --2- 172.21.15.37:0/661492814 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2d01cca40 0x7fb2d01cf4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:46.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2d483a640 10 monclient(hunting): get_auth_request con 0x7fb2d01cca40 auth_method 0 2024-04-26T00:36:46.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2d483a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:46.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2d483a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:46.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2d483a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:46.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2cffff640 10 monclient(hunting): handle_auth_done global_id 144732 payload 824 2024-04-26T00:36:46.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2cffff640 1 -- 172.21.15.37:0/661492814 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb2d0137310 msgr2=0x7fb2d01cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:46.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2cffff640 1 --2- 172.21.15.37:0/661492814 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb2d0137310 0x7fb2d01cc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:46.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2cffff640 1 -- 172.21.15.37:0/661492814 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2d01cca40 msgr2=0x7fb2d01cf4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:46.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2cffff640 1 --2- 172.21.15.37:0/661492814 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb2d01cca40 0x7fb2d01cf4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:46.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2cffff640 10 monclient: _finish_hunting 0 2024-04-26T00:36:46.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2cffff640 1 monclient: found mon.noname-a 2024-04-26T00:36:46.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:46.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2cffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:36:46.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2cffff640 1 -- 172.21.15.37:0/661492814 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2d013d660 con 0x7fb2d00726b0 2024-04-26T00:36:46.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.948+0000 7fb2cf7fe640 1 --2- 172.21.15.37:0/661492814 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb2d0137310 0x7fb2d01cc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:36:46.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2cffff640 1 -- 172.21.15.37:0/661492814 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb2d00726b0 msgr2=0x7fb2d01cbfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:46.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2cffff640 1 --2- 172.21.15.37:0/661492814 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb2d00726b0 0x7fb2d01cbfc0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb2c000d5f0 tx=0x7fb2c000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:46.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2ccff9640 1 -- 172.21.15.37:0/661492814 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c0012800 con 0x7fb2d00726b0 2024-04-26T00:36:46.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2ccff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:46.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2ccff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:36:46.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2ccff9640 10 monclient: dump: 2024-04-26T00:36:46.958 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:46.958 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:46.958 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:46.959 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:46.959 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:46.959 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:46.959 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:46.959 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:46.959 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:46.959 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:46.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2ccff9640 10 monclient: _finish_auth 0 2024-04-26T00:36:46.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2ccff9640 10 monclient: _check_auth_tickets 2024-04-26T00:36:46.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2ccff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:46.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2ccff9640 1 -- 172.21.15.37:0/661492814 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2c0019070 con 0x7fb2d00726b0 2024-04-26T00:36:46.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2ccff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:36:46.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2ccff9640 1 -- 172.21.15.37:0/661492814 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c0012bc0 con 0x7fb2d00726b0 2024-04-26T00:36:46.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2d629d640 5 monclient: authenticate success, global_id 144732 2024-04-26T00:36:46.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2d629d640 10 monclient: _renew_subs 2024-04-26T00:36:46.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2d629d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:46.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2d629d640 1 -- 172.21.15.37:0/661492814 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2d013d940 con 0x7fb2d00726b0 2024-04-26T00:36:46.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2ccff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:46.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2ccff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:36:46.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2ccff9640 10 monclient: dump: 2024-04-26T00:36:46.960 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:46.960 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:46.960 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:46.960 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:46.960 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:46.960 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:46.960 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:46.960 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:46.960 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:46.960 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:46.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2ce7fc640 4 set_mon_vals no callback set 2024-04-26T00:36:46.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2d629d640 10 monclient: _renew_subs 2024-04-26T00:36:46.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2d629d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:46.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2d629d640 1 -- 172.21.15.37:0/661492814 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2d00a05d0 con 0x7fb2d00726b0 2024-04-26T00:36:46.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2d629d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:46.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2d629d640 1 --2- 172.21.15.37:0/661492814 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb2d01cfa20 0x7fb2d01cfe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:46.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2d629d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:46.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2d629d640 1 -- 172.21.15.37:0/661492814 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d004f510 con 0x7fb2d01cfa20 2024-04-26T00:36:46.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2cf7fe640 1 --2- 172.21.15.37:0/661492814 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb2d01cfa20 0x7fb2d01cfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:46.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2ccff9640 1 -- 172.21.15.37:0/661492814 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb2c001e460 con 0x7fb2d00726b0 2024-04-26T00:36:46.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2cf7fe640 10 monclient: get_auth_request con 0x7fb2d01cfa20 auth_method 0 2024-04-26T00:36:46.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:46.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2cf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:46.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:46.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:46.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:46.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:46.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2ccff9640 1 --2- 172.21.15.37:0/661492814 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb2bc035f30 0x7fb2bc0383f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:46.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2cf7fe640 10 monclient(hunting): handle_auth_done global_id 144735 payload 824 2024-04-26T00:36:46.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2ccff9640 1 -- 172.21.15.37:0/661492814 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(661..661 src has 370..661) ==== 3927+0+0 (secure 0 0 0) 0x7fb2c0048e80 con 0x7fb2d00726b0 2024-04-26T00:36:46.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2cf7fe640 1 -- 172.21.15.37:0/661492814 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb2d013b650 con 0x7fb2d01cfa20 2024-04-26T00:36:46.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2d483a640 1 --2- 172.21.15.37:0/661492814 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb2bc035f30 0x7fb2bc0383f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:46.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2d483a640 10 monclient: get_auth_request con 0x7fb2bc035f30 auth_method 0 2024-04-26T00:36:46.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2cf7fe640 1 -- 172.21.15.37:0/661492814 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb2d01cfa20 msgr2=0x7fb2d01cfe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:46.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.952+0000 7fb2cf7fe640 1 --2- 172.21.15.37:0/661492814 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb2d01cfa20 0x7fb2d01cfe00 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb2c4018ce0 tx=0x7fb2c4018e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:46.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.956+0000 7fb2ccff9640 1 -- 172.21.15.37:0/661492814 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c40ba070 con 0x7fb2d01cfa20 2024-04-26T00:36:46.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.956+0000 7fb2ccff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:46.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.956+0000 7fb2ccff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:36:46.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.956+0000 7fb2ccff9640 10 monclient: dump: 2024-04-26T00:36:46.965 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:46.965 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:46.965 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:46.965 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:46.965 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:46.965 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:46.965 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:46.965 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:46.965 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:46.966 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:46.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.956+0000 7fb2ccff9640 1 -- 172.21.15.37:0/661492814 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb2c4004450 con 0x7fb2d01cfa20 2024-04-26T00:36:46.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.956+0000 7fb2ccff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:46.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.956+0000 7fb2ccff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:36:46.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.956+0000 7fb2ccff9640 1 -- 172.21.15.37:0/661492814 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb2d01cfa20 msgr2=0x7fb2d01cfe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:46.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.956+0000 7fb2ccff9640 1 --2- 172.21.15.37:0/661492814 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb2d01cfa20 0x7fb2d01cfe00 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb2c4018ce0 tx=0x7fb2c4018e70 comp rx=0 tx=0).stop 2024-04-26T00:36:46.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.956+0000 7fb2ccff9640 1 -- 172.21.15.37:0/661492814 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb2d01cfa20 msgr2=0x7fb2d01cfe00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:36:46.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.956+0000 7fb2ccff9640 1 --2- 172.21.15.37:0/661492814 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb2d01cfa20 0x7fb2d01cfe00 secure :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb2c4018ce0 tx=0x7fb2c4018e70 comp rx=0 tx=0).stop 2024-04-26T00:36:46.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.956+0000 7fb2d483a640 1 --2- 172.21.15.37:0/661492814 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb2bc035f30 0x7fb2bc0383f0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8007af0 tx=0x7fb2c8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:46.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.960+0000 7fb2d629d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:36:46.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.960+0000 7fb2d629d640 1 --2- 172.21.15.37:0/661492814 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb2d0133ff0 0x7fb2d0134400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:46.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.960+0000 7fb2d629d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:46.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.960+0000 7fb2d629d640 1 -- 172.21.15.37:0/661492814 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d0073280 con 0x7fb2d0133ff0 2024-04-26T00:36:46.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.960+0000 7fb2cf7fe640 1 --2- 172.21.15.37:0/661492814 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb2d0133ff0 0x7fb2d0134400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:46.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.960+0000 7fb2cf7fe640 10 monclient: get_auth_request con 0x7fb2d0133ff0 auth_method 0 2024-04-26T00:36:46.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.960+0000 7fb2cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:46.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.960+0000 7fb2cf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:46.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.960+0000 7fb2cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:46.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.960+0000 7fb2cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:46.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.960+0000 7fb2cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:46.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.960+0000 7fb2cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:46.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.960+0000 7fb2cf7fe640 10 monclient(hunting): handle_auth_done global_id 144738 payload 824 2024-04-26T00:36:46.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.960+0000 7fb2cf7fe640 1 -- 172.21.15.37:0/661492814 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb2d013d190 con 0x7fb2d0133ff0 2024-04-26T00:36:46.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.964+0000 7fb2cf7fe640 1 -- 172.21.15.37:0/661492814 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb2d0133ff0 msgr2=0x7fb2d0134400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:46.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.964+0000 7fb2cf7fe640 1 --2- 172.21.15.37:0/661492814 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb2d0133ff0 0x7fb2d0134400 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb2c4019d00 tx=0x7fb2c40a6000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:46.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.964+0000 7fb2ccff9640 1 -- 172.21.15.37:0/661492814 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c40b0040 con 0x7fb2d0133ff0 2024-04-26T00:36:46.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.964+0000 7fb2ccff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:46.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.964+0000 7fb2ccff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:36:46.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.964+0000 7fb2ccff9640 10 monclient: dump: 2024-04-26T00:36:46.967 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:46.968 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:46.968 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:46.968 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:46.968 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:46.968 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:46.968 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:46.968 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:46.968 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:46.968 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:46.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.964+0000 7fb2ccff9640 1 -- 172.21.15.37:0/661492814 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4890 (secure 0 0 0) 0x7fb2c40ca450 con 0x7fb2d0133ff0 2024-04-26T00:36:46.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.964+0000 7fb2ccff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:36:46.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.964+0000 7fb2ccff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:36:46.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.964+0000 7fb2ccff9640 1 -- 172.21.15.37:0/661492814 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb2d0133ff0 msgr2=0x7fb2d0134400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:46.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.964+0000 7fb2ccff9640 1 --2- 172.21.15.37:0/661492814 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb2d0133ff0 0x7fb2d0134400 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb2c4019d00 tx=0x7fb2c40a6000 comp rx=0 tx=0).stop 2024-04-26T00:36:46.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.964+0000 7fb2ccff9640 1 -- 172.21.15.37:0/661492814 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb2d0133ff0 msgr2=0x7fb2d0134400 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:36:46.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.964+0000 7fb2ccff9640 1 --2- 172.21.15.37:0/661492814 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb2d0133ff0 0x7fb2d0134400 secure :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb2c4019d00 tx=0x7fb2c40a6000 comp rx=0 tx=0).stop 2024-04-26T00:36:46.969 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:36:46.969 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:36:46.969 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:36:46.969 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:36:46.969 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 126, 2024-04-26T00:36:46.969 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 293507, 2024-04-26T00:36:46.969 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:36:46.969 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:36:46.969 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:36:46.969 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:36:46.970 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:36:46.970 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 6, 2024-04-26T00:36:46.970 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:36:46.970 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:36:46.970 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:36:46.970 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:36:46.970 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:36:46.970 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:36:46.970 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:36:46.970 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:36:46.970 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:36:46.970 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:36:46.970 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:36:46.970 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:36:46.971 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:36:46.971 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:36:46.971 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:36:46.971 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:36:46.971 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:36:46.971 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:36:46.971 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:36:46.971 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:36:46.971 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:36:46.971 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:36:46.971 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:36:46.971 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:36:46.971 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:36:46.971 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:36:46.971 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:36:46.972 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:36:46.972 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:46.972 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:46.972 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:36:46.972 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:36:46.972 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:36:46.972 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:36:46.972 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:36:46.972 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:36:46.972 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:36:46.972 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:36:46.973 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:36:46.973 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:36:46.973 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:36:46.973 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:36:46.973 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:36:46.973 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:36:46.973 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:36:46.973 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:36:46.973 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:36:46.973 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:36:46.973 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:36:46.973 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:36:46.973 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:36:46.973 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:36:46.974 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:36:46.974 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:36:46.974 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:36:46.974 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:36:46.974 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:36:46.974 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:36:46.974 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:36:46.974 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:36:46.974 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:46.974 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:36:46.974 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:46.974 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:36:46.974 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:36:46.974 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:36:46.974 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:36:46.975 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:46.975 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:36:46.975 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:36:46.975 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:46.975 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:46.975 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:46.975 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:36:46.975 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:36:46.975 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:46.975 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:46.975 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:46.975 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:46.975 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:36:46.975 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:36:46.975 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:36:46.976 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:36:46.976 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:36:46.976 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:46.976 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:46.976 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:36:46.976 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:36:46.976 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:36:46.976 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:36:46.976 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:46.976 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:36:46.976 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:36:46.976 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:46.976 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:46.976 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:46.976 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:36:46.977 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:36:46.977 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:46.977 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:46.977 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:46.977 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:46.977 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:36:46.977 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:36:46.977 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:36:46.977 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:36:46.977 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:36:46.977 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:46.977 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:46.977 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:36:46.977 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:36:46.977 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:36:46.978 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:36:46.978 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:46.978 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:36:46.978 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:36:46.978 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:46.978 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:46.978 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:46.978 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:36:46.978 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:36:46.978 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:46.978 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:46.978 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:46.978 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:46.978 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:36:46.978 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:36:46.979 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:36:46.979 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:36:46.979 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:36:46.979 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:46.979 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:46.979 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:46.979 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:36:46.979 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:36:46.979 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:46.979 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:36:46.979 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:36:46.979 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:36:46.979 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:46.979 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:36:46.980 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:36:46.980 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:46.980 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:36:46.980 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:36:46.980 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:36:46.980 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:46.980 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:36:46.980 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:36:46.980 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:46.980 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:36:46.980 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:36:46.980 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 7 2024-04-26T00:36:46.980 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:46.980 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:36:46.980 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:36:46.981 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:46.981 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:36:46.981 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:36:46.981 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:36:46.981 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:46.981 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:46.981 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:46.981 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:36:46.981 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:36:46.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.964+0000 7fb2d629d640 1 -- 172.21.15.37:0/661492814 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb2bc035f30 msgr2=0x7fb2bc0383f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:46.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.964+0000 7fb2d629d640 1 --2- 172.21.15.37:0/661492814 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb2bc035f30 0x7fb2bc0383f0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8007af0 tx=0x7fb2c8009040 comp rx=0 tx=0).stop 2024-04-26T00:36:46.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.964+0000 7fb2d629d640 10 monclient: shutdown 2024-04-26T00:36:46.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.964+0000 7fb2d629d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:36:46.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.964+0000 7fb2d629d640 1 -- 172.21.15.37:0/661492814 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb2d00726b0 msgr2=0x7fb2d01cbfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:46.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.964+0000 7fb2d629d640 1 --2- 172.21.15.37:0/661492814 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb2d00726b0 0x7fb2d01cbfc0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb2c000d5f0 tx=0x7fb2c000dab0 comp rx=0 tx=0).stop 2024-04-26T00:36:46.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.964+0000 7fb2d483a640 1 -- 172.21.15.37:0/661492814 reap_dead start 2024-04-26T00:36:46.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.964+0000 7fb2d483a640 1 -- 172.21.15.37:0/661492814 reap_dead start 2024-04-26T00:36:46.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.964+0000 7fb2d629d640 1 -- 172.21.15.37:0/661492814 shutdown_connections 2024-04-26T00:36:46.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.964+0000 7fb2d629d640 1 -- 172.21.15.37:0/661492814 >> 172.21.15.37:0/661492814 conn(0x7fb2d00a1d50 msgr2=0x7fb2d00a3fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:36:46.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.964+0000 7fb2d629d640 1 -- 172.21.15.37:0/661492814 shutdown_connections 2024-04-26T00:36:46.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:46.964+0000 7fb2d629d640 1 -- 172.21.15.37:0/661492814 wait complete. 2024-04-26T00:36:46.983 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:36:46.984 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:36:46.984 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:36:47.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc80c821640 10 monclient: get_monmap_and_config 2024-04-26T00:36:47.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc80c821640 10 monclient: build_initial_monmap 2024-04-26T00:36:47.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc80c821640 10 monclient: monmap: 2024-04-26T00:36:47.099 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:36:47.099 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:47.099 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:36:47.104317+0000 2024-04-26T00:36:47.099 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:36:47.104317+0000 2024-04-26T00:36:47.099 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:36:47.100 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:47.100 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:36:47.100 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:36:47.100 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:36:47.100 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:36:47.100 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:36:47.100 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:47.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc80c821640 1 Processor -- start 2024-04-26T00:36:47.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc80c821640 1 -- start start 2024-04-26T00:36:47.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc80c821640 10 monclient: init 2024-04-26T00:36:47.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc80c821640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:47.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc80c821640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-04-26T00:36:47.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc80c821640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc808130eb0 0x7fc80809ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:47.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc80c821640 10 monclient(hunting): picked mon.noname-c con 0x7fc808130eb0 addr v2:172.21.15.45:3300/0 2024-04-26T00:36:47.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc80c821640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc808131290 0x7fc808133b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:47.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc80c821640 10 monclient(hunting): picked mon.noname-a con 0x7fc808131290 addr v2:172.21.15.37:3300/0 2024-04-26T00:36:47.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc80c821640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8081340d0 0x7fc808134580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:47.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc80c821640 10 monclient(hunting): picked mon.noname-b con 0x7fc8081340d0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:36:47.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc80c821640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:47.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc80c821640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fc80809df60 con 0x7fc808131290 2024-04-26T00:36:47.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc80c821640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:47.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc80c821640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc80809c190 con 0x7fc808130eb0 2024-04-26T00:36:47.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc80c821640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:47.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc80c821640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc8080a11c0 con 0x7fc8081340d0 2024-04-26T00:36:47.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc80c821640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:47.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc80c821640 10 monclient(hunting): authenticate will time out at 1988.410485s 2024-04-26T00:36:47.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc806d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8081340d0 0x7fc808134580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:47.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc806d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8081340d0 0x7fc808134580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:59506/0 (socket says 172.21.15.37:59506) 2024-04-26T00:36:47.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc806d9d640 1 -- 172.21.15.37:0/1580559244 learned_addr learned my addr 172.21.15.37:0/1580559244 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:36:47.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc806d9d640 10 monclient(hunting): get_auth_request con 0x7fc8081340d0 auth_method 0 2024-04-26T00:36:47.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc806d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:47.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc806d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:47.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc806d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:47.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc805d9b640 1 --2- 172.21.15.37:0/1580559244 >> v2:172.21.15.37:3300/0 conn(0x7fc808131290 0x7fc808133b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:47.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc805d9b640 10 monclient(hunting): get_auth_request con 0x7fc808131290 auth_method 0 2024-04-26T00:36:47.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc805d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:47.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc80659c640 1 --2- 172.21.15.37:0/1580559244 >> v2:172.21.15.45:3300/0 conn(0x7fc808130eb0 0x7fc80809ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:47.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc805d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:47.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc805d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:47.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc80659c640 10 monclient(hunting): get_auth_request con 0x7fc808130eb0 auth_method 0 2024-04-26T00:36:47.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc80659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:47.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc80659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:47.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc80659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:47.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc806d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:47.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc806d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:47.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc806d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:47.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc805d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:47.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc805d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:47.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc805d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:47.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc80659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:47.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc80659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:47.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc80659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:47.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc806d9d640 10 monclient(hunting): handle_auth_done global_id 204125 payload 274 2024-04-26T00:36:47.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc806d9d640 1 -- 172.21.15.37:0/1580559244 >> v2:172.21.15.45:3300/0 conn(0x7fc808130eb0 msgr2=0x7fc80809ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:36:47.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc806d9d640 1 --2- 172.21.15.37:0/1580559244 >> v2:172.21.15.45:3300/0 conn(0x7fc808130eb0 0x7fc80809ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:47.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc806d9d640 1 -- 172.21.15.37:0/1580559244 >> v2:172.21.15.37:3300/0 conn(0x7fc808131290 msgr2=0x7fc808133b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:36:47.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc806d9d640 1 --2- 172.21.15.37:0/1580559244 >> v2:172.21.15.37:3300/0 conn(0x7fc808131290 0x7fc808133b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:47.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc806d9d640 10 monclient: _finish_hunting 0 2024-04-26T00:36:47.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc806d9d640 1 monclient: found mon.noname-b 2024-04-26T00:36:47.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc806d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:47.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc806d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:36:47.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.100+0000 7fc806d9d640 1 -- 172.21.15.37:0/1580559244 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc80809dc80 con 0x7fc8081340d0 2024-04-26T00:36:47.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.104+0000 7fc806d9d640 1 -- 172.21.15.37:0/1580559244 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8081340d0 msgr2=0x7fc808134580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:47.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.104+0000 7fc806d9d640 1 --2- 172.21.15.37:0/1580559244 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8081340d0 0x7fc808134580 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc8080764d0 tx=0x7fc80001c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6f31d548a4b600 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:47.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.104+0000 7fc80559a640 1 -- 172.21.15.37:0/1580559244 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8000a5640 con 0x7fc8081340d0 2024-04-26T00:36:47.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.104+0000 7fc80559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:47.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.104+0000 7fc80559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:36:47.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.104+0000 7fc80559a640 10 monclient: dump: 2024-04-26T00:36:47.116 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:47.116 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:47.116 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:47.116 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:47.116 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:47.116 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:47.116 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:47.116 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:47.116 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:47.116 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:47.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.104+0000 7fc80559a640 10 monclient: _finish_auth 0 2024-04-26T00:36:47.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.104+0000 7fc80559a640 10 monclient: _check_auth_tickets 2024-04-26T00:36:47.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.104+0000 7fc80559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:47.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.104+0000 7fc80559a640 1 -- 172.21.15.37:0/1580559244 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8000a57e0 con 0x7fc8081340d0 2024-04-26T00:36:47.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.104+0000 7fc80559a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:36:47.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.104+0000 7fc80559a640 1 -- 172.21.15.37:0/1580559244 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8000a5bb0 con 0x7fc8081340d0 2024-04-26T00:36:47.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.104+0000 7fc80c821640 5 monclient: authenticate success, global_id 204125 2024-04-26T00:36:47.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.104+0000 7fc80c821640 10 monclient: get_monmap_and_config success 2024-04-26T00:36:47.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.104+0000 7fc80c821640 4 set_mon_vals no callback set 2024-04-26T00:36:47.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.104+0000 7fc80559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:47.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.104+0000 7fc80559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:36:47.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.104+0000 7fc80559a640 10 monclient: dump: 2024-04-26T00:36:47.118 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:47.118 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:47.118 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:47.118 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:47.118 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:47.118 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:47.118 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:47.118 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:47.118 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:47.118 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:47.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc80c821640 10 monclient: shutdown 2024-04-26T00:36:47.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc80c821640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:36:47.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc80c821640 1 -- 172.21.15.37:0/1580559244 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8081340d0 msgr2=0x7fc808134580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:47.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc80c821640 1 --2- 172.21.15.37:0/1580559244 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8081340d0 0x7fc808134580 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc8080764d0 tx=0x7fc80001c8b0 comp rx=0 tx=0).stop 2024-04-26T00:36:47.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc80c821640 1 -- 172.21.15.37:0/1580559244 shutdown_connections 2024-04-26T00:36:47.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc80c821640 1 --2- 172.21.15.37:0/1580559244 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8081340d0 0x7fc808134580 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:47.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc80c821640 1 --2- 172.21.15.37:0/1580559244 >> v2:172.21.15.37:3300/0 conn(0x7fc808131290 0x7fc808133b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:47.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc80c821640 1 --2- 172.21.15.37:0/1580559244 >> v2:172.21.15.45:3300/0 conn(0x7fc808130eb0 0x7fc80809ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:47.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc80c821640 1 -- 172.21.15.37:0/1580559244 >> 172.21.15.37:0/1580559244 conn(0x7fc8080a1d50 msgr2=0x7fc8080a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:36:47.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc80c821640 1 -- 172.21.15.37:0/1580559244 shutdown_connections 2024-04-26T00:36:47.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc80c821640 1 -- 172.21.15.37:0/1580559244 wait complete. 2024-04-26T00:36:47.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc80c821640 10 monclient: build_initial_monmap 2024-04-26T00:36:47.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc80c821640 10 monclient: monmap: 2024-04-26T00:36:47.120 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:36:47.120 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:36:47.120 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:36:47.121 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:36:47.121 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:36:47.121 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:36:47.121 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:36:47.121 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:36:47.122 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:36:47.122 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:47.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc80c821640 1 Processor -- start 2024-04-26T00:36:47.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc80c821640 1 -- start start 2024-04-26T00:36:47.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc80c821640 10 monclient: init 2024-04-26T00:36:47.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc80c821640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:47.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc80c821640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:36:47.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc80c821640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc808130eb0 0x7fc80812de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:47.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc80c821640 10 monclient(hunting): picked mon.noname-b con 0x7fc808130eb0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:36:47.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc80c821640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc808131290 0x7fc80812e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:47.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc80c821640 10 monclient(hunting): picked mon.noname-a con 0x7fc808131290 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:36:47.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc80c821640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8081340d0 0x7fc8081ccba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:47.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc80c821640 10 monclient(hunting): picked mon.noname-c con 0x7fc8081340d0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:36:47.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc80c821640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:47.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc80c821640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fc80813d860 con 0x7fc808131290 2024-04-26T00:36:47.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc80c821640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:47.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc80c821640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc80813d6e0 con 0x7fc808130eb0 2024-04-26T00:36:47.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc80c821640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:47.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc80c821640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc80813d9e0 con 0x7fc8081340d0 2024-04-26T00:36:47.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc80c821640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:47.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc80c821640 10 monclient(hunting): authenticate will time out at 1988.418541s 2024-04-26T00:36:47.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc805d9b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc808131290 0x7fc80812e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:47.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc806d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8081340d0 0x7fc8081ccba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:47.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc80659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc808130eb0 0x7fc80812de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:47.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc805d9b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc808131290 0x7fc80812e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50406/0 (socket says 172.21.15.37:50406) 2024-04-26T00:36:47.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc806d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8081340d0 0x7fc8081ccba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:59522/0 (socket says 172.21.15.37:59522) 2024-04-26T00:36:47.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc805d9b640 1 -- 172.21.15.37:0/1072237718 learned_addr learned my addr 172.21.15.37:0/1072237718 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:36:47.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc80659c640 10 monclient(hunting): get_auth_request con 0x7fc808130eb0 auth_method 0 2024-04-26T00:36:47.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc80659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:47.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc80659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:47.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc80659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:47.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc806d9d640 10 monclient(hunting): get_auth_request con 0x7fc8081340d0 auth_method 0 2024-04-26T00:36:47.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc806d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:47.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc806d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:47.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc806d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:47.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc805d9b640 10 monclient(hunting): get_auth_request con 0x7fc808131290 auth_method 0 2024-04-26T00:36:47.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc805d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:47.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc805d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:47.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc805d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:47.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc80659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:47.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc80659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:47.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc80659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:47.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc806d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:47.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc806d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:47.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc806d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:47.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc805d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:47.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc805d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:47.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc805d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:47.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc806d9d640 10 monclient(hunting): handle_auth_done global_id 204128 payload 824 2024-04-26T00:36:47.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc806d9d640 1 -- 172.21.15.37:0/1072237718 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc808130eb0 msgr2=0x7fc80812de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:47.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc806d9d640 1 --2- 172.21.15.37:0/1072237718 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc808130eb0 0x7fc80812de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:47.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc806d9d640 1 -- 172.21.15.37:0/1072237718 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc808131290 msgr2=0x7fc80812e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:47.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc806d9d640 1 --2- 172.21.15.37:0/1072237718 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc808131290 0x7fc80812e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:47.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc806d9d640 10 monclient: _finish_hunting 0 2024-04-26T00:36:47.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc806d9d640 1 monclient: found mon.noname-c 2024-04-26T00:36:47.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc806d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:47.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc806d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:36:47.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc806d9d640 1 -- 172.21.15.37:0/1072237718 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8080a1470 con 0x7fc8081340d0 2024-04-26T00:36:47.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.108+0000 7fc805d9b640 1 --2- 172.21.15.37:0/1072237718 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc808131290 0x7fc80812e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:36:47.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc806d9d640 1 -- 172.21.15.37:0/1072237718 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8081340d0 msgr2=0x7fc8081ccba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:47.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc806d9d640 1 --2- 172.21.15.37:0/1072237718 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8081340d0 0x7fc8081ccba0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc8000ae000 tx=0x7fc800019650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:47.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc7f6ffd640 1 -- 172.21.15.37:0/1072237718 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8000bc070 con 0x7fc8081340d0 2024-04-26T00:36:47.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc7f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:47.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc7f6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:36:47.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc7f6ffd640 10 monclient: dump: 2024-04-26T00:36:47.128 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:47.128 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:47.128 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:47.128 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:47.128 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:47.128 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:47.128 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:47.128 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:47.128 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:47.128 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:47.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc7f6ffd640 10 monclient: _finish_auth 0 2024-04-26T00:36:47.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc7f6ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:36:47.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc7f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:47.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc80c821640 5 monclient: authenticate success, global_id 204128 2024-04-26T00:36:47.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc7f6ffd640 1 -- 172.21.15.37:0/1072237718 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8000b8850 con 0x7fc8081340d0 2024-04-26T00:36:47.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc80c821640 10 monclient: _renew_subs 2024-04-26T00:36:47.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc80c821640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:36:47.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc80c821640 1 -- 172.21.15.37:0/1072237718 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc808130130 con 0x7fc8081340d0 2024-04-26T00:36:47.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc7f6ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:36:47.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc7f6ffd640 1 -- 172.21.15.37:0/1072237718 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8000a5640 con 0x7fc8081340d0 2024-04-26T00:36:47.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc7f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:47.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc7f6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:36:47.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc804d99640 4 set_mon_vals no callback set 2024-04-26T00:36:47.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc7f6ffd640 10 monclient: dump: 2024-04-26T00:36:47.129 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:47.129 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:47.129 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:47.129 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:47.129 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:47.130 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:47.130 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:47.130 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:47.130 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:47.130 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:47.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc80c821640 10 monclient: _renew_subs 2024-04-26T00:36:47.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc80c821640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:36:47.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc80c821640 1 -- 172.21.15.37:0/1072237718 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8081396c0 con 0x7fc8081340d0 2024-04-26T00:36:47.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc80c821640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:47.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc80c821640 1 --2- 172.21.15.37:0/1072237718 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8081cd0e0 0x7fc8081cd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:47.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc80c821640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:47.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc80c821640 1 -- 172.21.15.37:0/1072237718 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc808135a50 con 0x7fc8081cd0e0 2024-04-26T00:36:47.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc7f6ffd640 1 -- 172.21.15.37:0/1072237718 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc8000b8c60 con 0x7fc8081340d0 2024-04-26T00:36:47.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc80659c640 1 --2- 172.21.15.37:0/1072237718 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8081cd0e0 0x7fc8081cd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:47.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc80659c640 10 monclient: get_auth_request con 0x7fc8081cd0e0 auth_method 0 2024-04-26T00:36:47.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc80659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:47.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc80659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:47.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc80659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:47.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc80659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:47.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc80659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:47.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc80659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:47.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc7f6ffd640 1 --2- 172.21.15.37:0/1072237718 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc7e8035e80 0x7fc7e8038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:47.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc7f6ffd640 1 -- 172.21.15.37:0/1072237718 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(661..661 src has 370..661) ==== 3927+0+0 (secure 0 0 0) 0x7fc8000e32a0 con 0x7fc8081340d0 2024-04-26T00:36:47.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc80659c640 10 monclient(hunting): handle_auth_done global_id 204131 payload 824 2024-04-26T00:36:47.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc80659c640 1 -- 172.21.15.37:0/1072237718 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc80812e890 con 0x7fc8081cd0e0 2024-04-26T00:36:47.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc805d9b640 1 --2- 172.21.15.37:0/1072237718 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc7e8035e80 0x7fc7e8038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:47.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.112+0000 7fc805d9b640 10 monclient: get_auth_request con 0x7fc7e8035e80 auth_method 0 2024-04-26T00:36:47.133 INFO:tasks.rados.rados.0.smithi045.stdout:2922: done (1 left) 2024-04-26T00:36:47.133 INFO:tasks.rados.rados.0.smithi045.stdout:2923: done (0 left) 2024-04-26T00:36:47.133 INFO:tasks.rados.rados.0.smithi045.stdout:2924: delete oid 39 current snap is 318 2024-04-26T00:36:47.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.116+0000 7fc80659c640 1 -- 172.21.15.37:0/1072237718 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8081cd0e0 msgr2=0x7fc8081cd4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:47.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.116+0000 7fc80659c640 1 --2- 172.21.15.37:0/1072237718 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8081cd0e0 0x7fc8081cd4c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8011900 tx=0x7fc7f8011dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:47.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.116+0000 7fc7f6ffd640 1 -- 172.21.15.37:0/1072237718 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f8019070 con 0x7fc8081cd0e0 2024-04-26T00:36:47.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.116+0000 7fc7f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:47.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.116+0000 7fc7f6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:36:47.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.116+0000 7fc7f6ffd640 10 monclient: dump: 2024-04-26T00:36:47.135 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:47.135 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:47.136 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:47.136 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:47.136 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:47.136 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:47.136 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:47.136 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:47.136 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:47.136 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:47.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.116+0000 7fc805d9b640 1 --2- 172.21.15.37:0/1072237718 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc7e8035e80 0x7fc7e8038340 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fc8081372b0 tx=0x7fc7fc009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:47.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.120+0000 7fc7f6ffd640 1 -- 172.21.15.37:0/1072237718 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc7f800f890 con 0x7fc8081cd0e0 2024-04-26T00:36:47.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.120+0000 7fc7f6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:47.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.120+0000 7fc7f6ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:36:47.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.120+0000 7fc7f6ffd640 1 -- 172.21.15.37:0/1072237718 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8081cd0e0 msgr2=0x7fc8081cd4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:47.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.120+0000 7fc7f6ffd640 1 --2- 172.21.15.37:0/1072237718 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8081cd0e0 0x7fc8081cd4c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8011900 tx=0x7fc7f8011dc0 comp rx=0 tx=0).stop 2024-04-26T00:36:47.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.120+0000 7fc7f6ffd640 1 -- 172.21.15.37:0/1072237718 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8081cd0e0 msgr2=0x7fc8081cd4c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:36:47.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.120+0000 7fc7f6ffd640 1 --2- 172.21.15.37:0/1072237718 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8081cd0e0 0x7fc8081cd4c0 secure :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8011900 tx=0x7fc7f8011dc0 comp rx=0 tx=0).stop 2024-04-26T00:36:47.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.128+0000 7fc80c821640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:36:47.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.128+0000 7fc80c821640 1 --2- 172.21.15.37:0/1072237718 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc80806fe50 0x7fc808070230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:47.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.128+0000 7fc80c821640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:47.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.128+0000 7fc80c821640 1 -- 172.21.15.37:0/1072237718 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc8081cda00 con 0x7fc80806fe50 2024-04-26T00:36:47.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.128+0000 7fc80659c640 1 --2- 172.21.15.37:0/1072237718 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc80806fe50 0x7fc808070230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:47.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.128+0000 7fc80659c640 10 monclient: get_auth_request con 0x7fc80806fe50 auth_method 0 2024-04-26T00:36:47.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.128+0000 7fc80659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:47.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.128+0000 7fc80659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:47.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.128+0000 7fc80659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:47.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.128+0000 7fc80659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:47.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.128+0000 7fc80659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:47.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.128+0000 7fc80659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:47.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.128+0000 7fc80659c640 10 monclient(hunting): handle_auth_done global_id 204134 payload 824 2024-04-26T00:36:47.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.128+0000 7fc80659c640 1 -- 172.21.15.37:0/1072237718 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc80804f510 con 0x7fc80806fe50 2024-04-26T00:36:47.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.132+0000 7fc80659c640 1 -- 172.21.15.37:0/1072237718 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc80806fe50 msgr2=0x7fc808070230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:47.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.132+0000 7fc80659c640 1 --2- 172.21.15.37:0/1072237718 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc80806fe50 0x7fc808070230 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8011900 tx=0x7fc7f80046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:47.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.132+0000 7fc7f6ffd640 1 -- 172.21.15.37:0/1072237718 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f8019040 con 0x7fc80806fe50 2024-04-26T00:36:47.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.132+0000 7fc7f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:47.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.132+0000 7fc7f6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:36:47.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.132+0000 7fc7f6ffd640 10 monclient: dump: 2024-04-26T00:36:47.139 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:47.139 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:47.139 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:47.140 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:47.140 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:47.140 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:47.140 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:47.140 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:47.140 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:47.140 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:47.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.132+0000 7fc7f6ffd640 1 -- 172.21.15.37:0/1072237718 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4547 (secure 0 0 0) 0x7fc7f801e450 con 0x7fc80806fe50 2024-04-26T00:36:47.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.132+0000 7fc7f6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:36:47.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.132+0000 7fc7f6ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:36:47.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.132+0000 7fc7f6ffd640 1 -- 172.21.15.37:0/1072237718 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc80806fe50 msgr2=0x7fc808070230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:47.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.132+0000 7fc7f6ffd640 1 --2- 172.21.15.37:0/1072237718 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc80806fe50 0x7fc808070230 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8011900 tx=0x7fc7f80046d0 comp rx=0 tx=0).stop 2024-04-26T00:36:47.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.132+0000 7fc7f6ffd640 1 -- 172.21.15.37:0/1072237718 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc80806fe50 msgr2=0x7fc808070230 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:36:47.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.132+0000 7fc7f6ffd640 1 --2- 172.21.15.37:0/1072237718 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc80806fe50 0x7fc808070230 secure :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8011900 tx=0x7fc7f80046d0 comp rx=0 tx=0).stop 2024-04-26T00:36:47.141 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:36:47.141 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:36:47.141 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:36:47.141 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:36:47.141 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 126, 2024-04-26T00:36:47.141 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 12739, 2024-04-26T00:36:47.142 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:36:47.142 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:36:47.142 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:36:47.142 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:36:47.142 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:36:47.142 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 7, 2024-04-26T00:36:47.142 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:36:47.142 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:36:47.142 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:36:47.142 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:36:47.142 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:36:47.143 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:36:47.143 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:36:47.143 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:36:47.143 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:36:47.143 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:36:47.143 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:36:47.143 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:36:47.143 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:36:47.143 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:36:47.143 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:36:47.143 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:36:47.143 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:36:47.143 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:36:47.143 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:36:47.143 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:36:47.144 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:36:47.144 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:36:47.144 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:36:47.144 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:36:47.144 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:36:47.144 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:36:47.144 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:36:47.144 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:36:47.144 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:47.144 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:47.144 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:36:47.145 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:36:47.145 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:36:47.145 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:36:47.145 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:36:47.145 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:36:47.145 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:36:47.146 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:36:47.146 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:36:47.146 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:36:47.146 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:36:47.146 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:36:47.146 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:36:47.146 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:36:47.146 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:36:47.146 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:36:47.147 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:36:47.147 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:36:47.147 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:36:47.147 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:36:47.147 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:36:47.147 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:36:47.147 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:36:47.147 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:36:47.147 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:36:47.147 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:36:47.148 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:36:47.148 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:36:47.148 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:36:47.148 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:36:47.148 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:47.148 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:36:47.148 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:47.148 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:36:47.148 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:36:47.148 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:36:47.148 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:36:47.148 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:47.148 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:36:47.148 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:36:47.149 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:47.149 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:47.149 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:47.149 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:36:47.149 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:36:47.149 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:47.149 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:47.149 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:47.150 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:47.150 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:36:47.150 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:36:47.150 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:36:47.150 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:36:47.150 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:36:47.151 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:47.151 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:47.151 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:36:47.151 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:36:47.151 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:36:47.151 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:36:47.152 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:47.152 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:36:47.152 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:36:47.152 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:47.152 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:47.152 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:47.153 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:36:47.153 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:36:47.153 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:47.153 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:47.153 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:47.153 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:47.154 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:36:47.154 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:36:47.154 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:36:47.154 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:36:47.154 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:36:47.154 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:47.155 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:47.155 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:36:47.155 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:36:47.155 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:36:47.155 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:36:47.156 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:47.156 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:36:47.156 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:36:47.156 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:47.156 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:47.156 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:47.157 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:36:47.157 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:36:47.157 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:47.157 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:47.157 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:47.157 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:47.157 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:36:47.158 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:36:47.158 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:36:47.158 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:36:47.158 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:36:47.158 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:47.158 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:47.159 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:47.159 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:36:47.159 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:36:47.159 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:47.159 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:36:47.159 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:36:47.159 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:36:47.159 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:47.160 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:36:47.160 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:36:47.160 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:47.160 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:36:47.160 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:36:47.160 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:36:47.160 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:47.161 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:47.161 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:47.161 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:36:47.161 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:36:47.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.136+0000 7fc80c821640 1 -- 172.21.15.37:0/1072237718 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc7e8035e80 msgr2=0x7fc7e8038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:47.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.136+0000 7fc80c821640 1 --2- 172.21.15.37:0/1072237718 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc7e8035e80 0x7fc7e8038340 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fc8081372b0 tx=0x7fc7fc009190 comp rx=0 tx=0).stop 2024-04-26T00:36:47.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.136+0000 7fc80c821640 10 monclient: shutdown 2024-04-26T00:36:47.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.136+0000 7fc80c821640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:36:47.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.136+0000 7fc80c821640 1 -- 172.21.15.37:0/1072237718 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8081340d0 msgr2=0x7fc8081ccba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:47.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.136+0000 7fc80c821640 1 --2- 172.21.15.37:0/1072237718 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc8081340d0 0x7fc8081ccba0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc8000ae000 tx=0x7fc800019650 comp rx=0 tx=0).stop 2024-04-26T00:36:47.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.136+0000 7fc806d9d640 1 -- 172.21.15.37:0/1072237718 reap_dead start 2024-04-26T00:36:47.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.136+0000 7fc806d9d640 1 -- 172.21.15.37:0/1072237718 reap_dead start 2024-04-26T00:36:47.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.136+0000 7fc80c821640 1 -- 172.21.15.37:0/1072237718 shutdown_connections 2024-04-26T00:36:47.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.136+0000 7fc80c821640 1 -- 172.21.15.37:0/1072237718 >> 172.21.15.37:0/1072237718 conn(0x7fc8080a1d50 msgr2=0x7fc808130340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:36:47.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.136+0000 7fc80c821640 1 -- 172.21.15.37:0/1072237718 shutdown_connections 2024-04-26T00:36:47.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.136+0000 7fc80c821640 1 -- 172.21.15.37:0/1072237718 wait complete. 2024-04-26T00:36:47.165 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:36:47.165 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:36:47.165 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:36:47.167 INFO:tasks.rados.rados.0.smithi045.stdout:2924: done (0 left) 2024-04-26T00:36:47.167 INFO:tasks.rados.rados.0.smithi045.stdout:2925: rollback oid 45 current snap is 318 2024-04-26T00:36:47.167 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 45 to 314 2024-04-26T00:36:47.167 INFO:tasks.rados.rados.0.smithi045.stdout:2926: copy_from oid 40 from oid 15 current snap is 318 2024-04-26T00:36:47.167 INFO:tasks.rados.rados.0.smithi045.stdout:2927: read oid 12 snap -1 2024-04-26T00:36:47.167 INFO:tasks.rados.rados.0.smithi045.stdout:2927: expect (ObjNum 945 snap 292 seq_num 945) 2024-04-26T00:36:47.167 INFO:tasks.rados.rados.0.smithi045.stdout:2928: copy_from oid 22 from oid 29 current snap is 318 2024-04-26T00:36:47.167 INFO:tasks.rados.rados.0.smithi045.stdout:2929: copy_from oid 4 from oid 26 current snap is 318 2024-04-26T00:36:47.167 INFO:tasks.rados.rados.0.smithi045.stdout:2930: copy_from oid 38 from oid 30 current snap is 318 2024-04-26T00:36:47.167 INFO:tasks.rados.rados.0.smithi045.stdout:2931: read oid 19 snap -1 2024-04-26T00:36:47.167 INFO:tasks.rados.rados.0.smithi045.stdout:2931: expect deleted 2024-04-26T00:36:47.168 INFO:tasks.rados.rados.0.smithi045.stdout:2932: rollback oid 23 current snap is 318 2024-04-26T00:36:47.168 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 23 to 316 2024-04-26T00:36:47.168 INFO:tasks.rados.rados.0.smithi045.stdout:2933: copy_from oid 49 from oid 13 current snap is 318 2024-04-26T00:36:47.168 INFO:tasks.rados.rados.0.smithi045.stdout:2934: read oid 41 snap 316 2024-04-26T00:36:47.168 INFO:tasks.rados.rados.0.smithi045.stdout:2934: expect (ObjNum 1037 snap 313 seq_num 1037) 2024-04-26T00:36:47.168 INFO:tasks.rados.rados.0.smithi045.stdout:2935: copy_from oid 3 from oid 5 current snap is 318 2024-04-26T00:36:47.168 INFO:tasks.rados.rados.0.smithi045.stdout:2936: copy_from oid 36 from oid 24 current snap is 318 2024-04-26T00:36:47.168 INFO:tasks.rados.rados.0.smithi045.stdout:2937: write oid 43 current snap is 318 2024-04-26T00:36:47.168 INFO:tasks.rados.rados.0.smithi045.stdout:2937: seq_num 1053 ranges {497771=493911,1619186=554020,2950529=430606} 2024-04-26T00:36:47.168 INFO:tasks.rados.rados.0.smithi045.stdout:2937: writing smithi04526579-43 from 497771 to 991682 tid 1 2024-04-26T00:36:47.173 INFO:tasks.rados.rados.0.smithi045.stdout:2937: writing smithi04526579-43 from 1619186 to 2173206 tid 2 2024-04-26T00:36:47.191 INFO:tasks.rados.rados.0.smithi045.stdout:2937: writing smithi04526579-43 from 2950529 to 3381135 tid 3 2024-04-26T00:36:47.192 INFO:tasks.rados.rados.0.smithi045.stdout:2938: rollback oid 25 current snap is 318 2024-04-26T00:36:47.192 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 25 to 316 2024-04-26T00:36:47.192 INFO:tasks.rados.rados.0.smithi045.stdout:2930: finishing copy_from to smithi04526579-38 2024-04-26T00:36:47.192 INFO:tasks.rados.rados.0.smithi045.stdout:2930: got expected ENOENT (src dne) 2024-04-26T00:36:47.192 INFO:tasks.rados.rados.0.smithi045.stdout:2930: finishing copy_from racing read to smithi04526579-38 2024-04-26T00:36:47.192 INFO:tasks.rados.rados.0.smithi045.stdout:2929: finishing copy_from to smithi04526579-4 2024-04-26T00:36:47.192 INFO:tasks.rados.rados.0.smithi045.stdout:2929: got expected ENOENT (src dne) 2024-04-26T00:36:47.192 INFO:tasks.rados.rados.0.smithi045.stdout:2929: finishing copy_from racing read to smithi04526579-4 2024-04-26T00:36:47.192 INFO:tasks.rados.rados.0.smithi045.stdout:2935: finishing copy_from racing read to smithi04526579-3 2024-04-26T00:36:47.192 INFO:tasks.rados.rados.0.smithi045.stdout:2933: finishing copy_from racing read to smithi04526579-49 2024-04-26T00:36:47.192 INFO:tasks.rados.rados.0.smithi045.stdout:2929: done (13 left) 2024-04-26T00:36:47.192 INFO:tasks.rados.rados.0.smithi045.stdout:2930: done (12 left) 2024-04-26T00:36:47.192 INFO:tasks.rados.rados.0.smithi045.stdout:2931: done (11 left) 2024-04-26T00:36:47.192 INFO:tasks.rados.rados.0.smithi045.stdout:2939: read oid 7 snap -1 2024-04-26T00:36:47.193 INFO:tasks.rados.rados.0.smithi045.stdout:2939: expect (ObjNum 998 snap 303 seq_num 998) 2024-04-26T00:36:47.193 INFO:tasks.rados.rados.0.smithi045.stdout:2940: snap_remove snap 317 2024-04-26T00:36:47.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e838a640 10 monclient: get_monmap_and_config 2024-04-26T00:36:47.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e838a640 10 monclient: build_initial_monmap 2024-04-26T00:36:47.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e838a640 10 monclient: monmap: 2024-04-26T00:36:47.285 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:36:47.285 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:47.285 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:36:47.288632+0000 2024-04-26T00:36:47.285 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:36:47.288632+0000 2024-04-26T00:36:47.285 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:36:47.286 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:47.286 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:36:47.286 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:36:47.286 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:36:47.286 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:36:47.287 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:36:47.287 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:47.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e838a640 1 Processor -- start 2024-04-26T00:36:47.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e838a640 1 -- start start 2024-04-26T00:36:47.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e838a640 10 monclient: init 2024-04-26T00:36:47.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e838a640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:47.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e838a640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-04-26T00:36:47.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e838a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb5e0130eb0 addr v1:172.21.15.37:6789/0 2024-04-26T00:36:47.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e838a640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb5e009ef20 0x7fb5e0138eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:47.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e838a640 10 monclient(hunting): picked mon.noname-a con 0x7fb5e009ef20 addr v2:172.21.15.37:3300/0 2024-04-26T00:36:47.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e838a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5e01393f0 0x7fb5e013b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:47.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e838a640 10 monclient(hunting): picked mon.noname-b con 0x7fb5e01393f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:36:47.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e838a640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5e009c190 con 0x7fb5e0130eb0 2024-04-26T00:36:47.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e838a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:47.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e838a640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fb5e009df60 con 0x7fb5e009ef20 2024-04-26T00:36:47.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e838a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:47.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e838a640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb5e00a11c0 con 0x7fb5e01393f0 2024-04-26T00:36:47.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e838a640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:47.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e838a640 10 monclient(hunting): authenticate will time out at 1988.594392s 2024-04-26T00:36:47.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e5925640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb5e009ef20 0x7fb5e0138eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:47.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e6927640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5e01393f0 0x7fb5e013b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:47.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e5925640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb5e009ef20 0x7fb5e0138eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50414/0 (socket says 172.21.15.37:50414) 2024-04-26T00:36:47.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e5925640 1 -- 172.21.15.37:0/2422587116 learned_addr learned my addr 172.21.15.37:0/2422587116 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:36:47.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e5925640 10 monclient(hunting): get_auth_request con 0x7fb5e009ef20 auth_method 0 2024-04-26T00:36:47.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e5925640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:47.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e5925640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:47.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e5925640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:47.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e6927640 10 monclient(hunting): get_auth_request con 0x7fb5e01393f0 auth_method 0 2024-04-26T00:36:47.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e6927640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:47.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e6927640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:47.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e6927640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:47.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e5925640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:47.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e5925640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:47.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e5925640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:47.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e6927640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:47.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e6927640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:47.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e6927640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:47.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e5925640 10 monclient(hunting): handle_auth_done global_id 144747 payload 274 2024-04-26T00:36:47.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e5925640 1 -- 172.21.15.37:0/2422587116 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5e01393f0 msgr2=0x7fb5e013b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:36:47.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e5925640 1 --2- 172.21.15.37:0/2422587116 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5e01393f0 0x7fb5e013b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:47.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e5925640 1 -- 172.21.15.37:0/2422587116 >> v1:172.21.15.37:6789/0 conn(0x7fb5e0130eb0 legacy=0x7fb5e0131290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:36:47.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e5925640 10 monclient: _finish_hunting 0 2024-04-26T00:36:47.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e5925640 1 monclient: found mon.noname-a 2024-04-26T00:36:47.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e5925640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:47.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e5925640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:36:47.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e5925640 1 -- 172.21.15.37:0/2422587116 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5e009dc80 con 0x7fb5e009ef20 2024-04-26T00:36:47.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.284+0000 7fb5e6927640 1 --2- 172.21.15.37:0/2422587116 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5e01393f0 0x7fb5e013b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:36:47.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.288+0000 7fb5e5925640 1 -- 172.21.15.37:0/2422587116 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5e009ef20 msgr2=0x7fb5e0138eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:47.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.288+0000 7fb5e5925640 1 --2- 172.21.15.37:0/2422587116 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5e009ef20 0x7fb5e0138eb0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0009870 tx=0x7fb5d001c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8120a25086806cc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:47.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.288+0000 7fb5e5124640 1 -- 172.21.15.37:0/2422587116 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d00b2070 con 0x7fb5e009ef20 2024-04-26T00:36:47.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.288+0000 7fb5e5124640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:47.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.288+0000 7fb5e5124640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:36:47.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.288+0000 7fb5e5124640 10 monclient: dump: 2024-04-26T00:36:47.298 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:47.298 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:47.298 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:47.298 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:47.298 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:47.298 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:47.299 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:47.299 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:47.299 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:47.299 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:47.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.288+0000 7fb5e5124640 10 monclient: _finish_auth 0 2024-04-26T00:36:47.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.288+0000 7fb5e5124640 10 monclient: _check_auth_tickets 2024-04-26T00:36:47.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.288+0000 7fb5e5124640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:47.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.288+0000 7fb5e5124640 1 -- 172.21.15.37:0/2422587116 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5d00ad640 con 0x7fb5e009ef20 2024-04-26T00:36:47.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.288+0000 7fb5e5124640 10 monclient: handle_config config(0 keys) 2024-04-26T00:36:47.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.288+0000 7fb5e5124640 1 -- 172.21.15.37:0/2422587116 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d00ada10 con 0x7fb5e009ef20 2024-04-26T00:36:47.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.288+0000 7fb5e5124640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:47.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.288+0000 7fb5e5124640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:36:47.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.288+0000 7fb5e5124640 10 monclient: dump: 2024-04-26T00:36:47.300 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:47.300 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:47.300 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:47.300 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:47.300 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:47.300 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:47.300 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:47.300 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:47.300 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:47.301 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:47.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.288+0000 7fb5e838a640 5 monclient: authenticate success, global_id 144747 2024-04-26T00:36:47.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.288+0000 7fb5e838a640 10 monclient: get_monmap_and_config success 2024-04-26T00:36:47.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.288+0000 7fb5e838a640 4 set_mon_vals no callback set 2024-04-26T00:36:47.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e838a640 10 monclient: shutdown 2024-04-26T00:36:47.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e838a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:36:47.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e838a640 1 -- 172.21.15.37:0/2422587116 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5e009ef20 msgr2=0x7fb5e0138eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:47.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e838a640 1 --2- 172.21.15.37:0/2422587116 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5e009ef20 0x7fb5e0138eb0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0009870 tx=0x7fb5d001c3b0 comp rx=0 tx=0).stop 2024-04-26T00:36:47.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e838a640 1 -- 172.21.15.37:0/2422587116 shutdown_connections 2024-04-26T00:36:47.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e838a640 1 --2- 172.21.15.37:0/2422587116 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5e01393f0 0x7fb5e013b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:47.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e838a640 1 --2- 172.21.15.37:0/2422587116 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5e009ef20 0x7fb5e0138eb0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:47.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e838a640 1 -- 172.21.15.37:0/2422587116 >> 172.21.15.37:0/2422587116 conn(0x7fb5e00a1d50 msgr2=0x7fb5e00a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:36:47.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e838a640 1 -- 172.21.15.37:0/2422587116 shutdown_connections 2024-04-26T00:36:47.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e838a640 1 -- 172.21.15.37:0/2422587116 wait complete. 2024-04-26T00:36:47.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e838a640 10 monclient: build_initial_monmap 2024-04-26T00:36:47.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e838a640 10 monclient: monmap: 2024-04-26T00:36:47.302 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:36:47.303 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:36:47.303 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:36:47.303 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:36:47.303 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:36:47.303 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:36:47.303 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:36:47.303 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:36:47.304 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:36:47.304 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:47.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e838a640 1 Processor -- start 2024-04-26T00:36:47.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e838a640 1 -- start start 2024-04-26T00:36:47.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e838a640 10 monclient: init 2024-04-26T00:36:47.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e838a640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:47.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e838a640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:36:47.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e838a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5e009ef20 0x7fb5e0136100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:47.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e838a640 10 monclient(hunting): picked mon.noname-c con 0x7fb5e009ef20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:36:47.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e838a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5e0130eb0 0x7fb5e0136640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:47.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e838a640 10 monclient(hunting): picked mon.noname-a con 0x7fb5e0130eb0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:36:47.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e838a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e01393f0 0x7fb5e0136b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:47.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e838a640 10 monclient(hunting): picked mon.noname-b con 0x7fb5e01393f0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:36:47.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e838a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:47.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e838a640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb5e009fdd0 con 0x7fb5e0130eb0 2024-04-26T00:36:47.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e838a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:47.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e838a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb5e009fc50 con 0x7fb5e01393f0 2024-04-26T00:36:47.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e838a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:47.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e838a640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb5e009ff50 con 0x7fb5e009ef20 2024-04-26T00:36:47.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e838a640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:47.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e838a640 10 monclient(hunting): authenticate will time out at 1988.602414s 2024-04-26T00:36:47.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e5925640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5e0130eb0 0x7fb5e0136640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:47.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e5925640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5e0130eb0 0x7fb5e0136640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50416/0 (socket says 172.21.15.37:50416) 2024-04-26T00:36:47.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e5925640 1 -- 172.21.15.37:0/408291998 learned_addr learned my addr 172.21.15.37:0/408291998 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:36:47.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e5925640 10 monclient(hunting): get_auth_request con 0x7fb5e0130eb0 auth_method 0 2024-04-26T00:36:47.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e5925640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:47.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e5925640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:47.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e5925640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:47.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e6126640 1 --2- 172.21.15.37:0/408291998 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5e009ef20 0x7fb5e0136100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:47.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e6126640 10 monclient(hunting): get_auth_request con 0x7fb5e009ef20 auth_method 0 2024-04-26T00:36:47.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e6126640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:47.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e6126640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:47.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e6126640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:47.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e5925640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:47.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e5925640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:47.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e5925640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:47.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e6126640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:47.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e6126640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:47.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e6126640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:47.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e6927640 1 --2- 172.21.15.37:0/408291998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e01393f0 0x7fb5e0136b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:47.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e5925640 10 monclient(hunting): handle_auth_done global_id 144750 payload 824 2024-04-26T00:36:47.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e5925640 1 -- 172.21.15.37:0/408291998 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5e009ef20 msgr2=0x7fb5e0136100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:47.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e5925640 1 --2- 172.21.15.37:0/408291998 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb5e009ef20 0x7fb5e0136100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:47.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e5925640 1 -- 172.21.15.37:0/408291998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e01393f0 msgr2=0x7fb5e0136b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:47.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e5925640 1 --2- 172.21.15.37:0/408291998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e01393f0 0x7fb5e0136b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:47.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e5925640 10 monclient: _finish_hunting 0 2024-04-26T00:36:47.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e5925640 1 monclient: found mon.noname-a 2024-04-26T00:36:47.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e5925640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:47.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e5925640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:36:47.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e5925640 1 -- 172.21.15.37:0/408291998 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5e01388b0 con 0x7fb5e0130eb0 2024-04-26T00:36:47.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e6927640 10 monclient: get_auth_request con 0x7fb5e01393f0 auth_method 0 2024-04-26T00:36:47.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.292+0000 7fb5e6927640 1 --2- 172.21.15.37:0/408291998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e01393f0 0x7fb5e0136b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:36:47.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5e5925640 1 -- 172.21.15.37:0/408291998 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5e0130eb0 msgr2=0x7fb5e0136640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:47.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5e5925640 1 --2- 172.21.15.37:0/408291998 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5e0130eb0 0x7fb5e0136640 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb5d00b0040 tx=0x7fb5d0019150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:47.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5d6ffd640 1 -- 172.21.15.37:0/408291998 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d00b2040 con 0x7fb5e0130eb0 2024-04-26T00:36:47.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:47.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5d6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:36:47.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5d6ffd640 10 monclient: dump: 2024-04-26T00:36:47.316 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:47.316 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:47.316 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:47.316 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:47.316 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:47.317 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:47.317 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:47.317 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:47.317 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:47.317 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:47.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5d6ffd640 10 monclient: _finish_auth 0 2024-04-26T00:36:47.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5d6ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:36:47.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:47.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5d6ffd640 1 -- 172.21.15.37:0/408291998 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5d00ab550 con 0x7fb5e0130eb0 2024-04-26T00:36:47.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5e838a640 5 monclient: authenticate success, global_id 144750 2024-04-26T00:36:47.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5e838a640 10 monclient: _renew_subs 2024-04-26T00:36:47.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5e838a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:47.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5e838a640 1 -- 172.21.15.37:0/408291998 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5e00a1470 con 0x7fb5e0130eb0 2024-04-26T00:36:47.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5d6ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:36:47.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5d6ffd640 1 -- 172.21.15.37:0/408291998 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d00ab920 con 0x7fb5e0130eb0 2024-04-26T00:36:47.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:47.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5d6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:36:47.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5d6ffd640 10 monclient: dump: 2024-04-26T00:36:47.318 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:47.318 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:47.318 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:47.318 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:47.318 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:47.318 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:47.318 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:47.318 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:47.318 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:47.319 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:47.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5e838a640 10 monclient: _renew_subs 2024-04-26T00:36:47.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5e838a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:47.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5e4923640 4 set_mon_vals no callback set 2024-04-26T00:36:47.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5e838a640 1 -- 172.21.15.37:0/408291998 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5e006dc40 con 0x7fb5e0130eb0 2024-04-26T00:36:47.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5e838a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:47.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5e838a640 1 --2- 172.21.15.37:0/408291998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e01370c0 0x7fb5e006ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:47.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5e838a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:47.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5e838a640 1 -- 172.21.15.37:0/408291998 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb5e004f510 con 0x7fb5e01370c0 2024-04-26T00:36:47.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5d6ffd640 1 -- 172.21.15.37:0/408291998 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb5d00abb00 con 0x7fb5e0130eb0 2024-04-26T00:36:47.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5d6ffd640 1 --2- 172.21.15.37:0/408291998 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb5c4035ed0 0x7fb5c4038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:47.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5d6ffd640 1 -- 172.21.15.37:0/408291998 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(662..662 src has 370..662) ==== 3899+0+0 (secure 0 0 0) 0x7fb5d00e24a0 con 0x7fb5e0130eb0 2024-04-26T00:36:47.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5e6126640 1 --2- 172.21.15.37:0/408291998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e01370c0 0x7fb5e006ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:47.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5e6126640 10 monclient: get_auth_request con 0x7fb5e01370c0 auth_method 0 2024-04-26T00:36:47.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5e6126640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:47.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5e6126640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:47.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5e6126640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:47.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5e6126640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:47.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5e6126640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:47.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5e6126640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:47.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5e6927640 1 --2- 172.21.15.37:0/408291998 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb5c4035ed0 0x7fb5c4038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:47.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5e6927640 10 monclient: get_auth_request con 0x7fb5c4035ed0 auth_method 0 2024-04-26T00:36:47.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5e6126640 10 monclient(hunting): handle_auth_done global_id 194209 payload 824 2024-04-26T00:36:47.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.296+0000 7fb5e6126640 1 -- 172.21.15.37:0/408291998 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb5e01374a0 con 0x7fb5e01370c0 2024-04-26T00:36:47.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.300+0000 7fb5e6126640 1 -- 172.21.15.37:0/408291998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e01370c0 msgr2=0x7fb5e006ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:47.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.300+0000 7fb5e6126640 1 --2- 172.21.15.37:0/408291998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e01370c0 0x7fb5e006ec10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc00f770 tx=0x7fb5dc00fc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:47.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.300+0000 7fb5e6927640 1 --2- 172.21.15.37:0/408291998 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb5c4035ed0 0x7fb5c4038390 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fb5e01387a0 tx=0x7fb5d800b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:47.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.300+0000 7fb5d6ffd640 1 -- 172.21.15.37:0/408291998 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5dc00fe40 con 0x7fb5e01370c0 2024-04-26T00:36:47.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.300+0000 7fb5d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:47.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.300+0000 7fb5d6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:36:47.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.300+0000 7fb5d6ffd640 10 monclient: dump: 2024-04-26T00:36:47.321 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:47.321 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:47.321 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:47.321 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:47.321 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:47.321 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:47.322 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:47.322 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:47.322 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:47.322 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:47.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.300+0000 7fb5d6ffd640 1 -- 172.21.15.37:0/408291998 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb5dc019070 con 0x7fb5e01370c0 2024-04-26T00:36:47.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.300+0000 7fb5d6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:47.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.300+0000 7fb5d6ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:36:47.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.300+0000 7fb5d6ffd640 1 -- 172.21.15.37:0/408291998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e01370c0 msgr2=0x7fb5e006ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:47.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.300+0000 7fb5d6ffd640 1 --2- 172.21.15.37:0/408291998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e01370c0 0x7fb5e006ec10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc00f770 tx=0x7fb5dc00fc30 comp rx=0 tx=0).stop 2024-04-26T00:36:47.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.300+0000 7fb5d6ffd640 1 -- 172.21.15.37:0/408291998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e01370c0 msgr2=0x7fb5e006ec10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:36:47.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.300+0000 7fb5d6ffd640 1 --2- 172.21.15.37:0/408291998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e01370c0 0x7fb5e006ec10 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc00f770 tx=0x7fb5dc00fc30 comp rx=0 tx=0).stop 2024-04-26T00:36:47.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.308+0000 7fb5e838a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:36:47.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.308+0000 7fb5e838a640 1 --2- 172.21.15.37:0/408291998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e01319a0 0x7fb5e0131db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:47.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.308+0000 7fb5e838a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:47.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.308+0000 7fb5e838a640 1 -- 172.21.15.37:0/408291998 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb5e00a0f60 con 0x7fb5e01319a0 2024-04-26T00:36:47.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.308+0000 7fb5e6126640 1 --2- 172.21.15.37:0/408291998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e01319a0 0x7fb5e0131db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:47.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.308+0000 7fb5e6126640 10 monclient: get_auth_request con 0x7fb5e01319a0 auth_method 0 2024-04-26T00:36:47.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.308+0000 7fb5e6126640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:47.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.308+0000 7fb5e6126640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:47.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.308+0000 7fb5e6126640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:47.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.308+0000 7fb5e6126640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:47.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.308+0000 7fb5e6126640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:47.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.308+0000 7fb5e6126640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:47.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.308+0000 7fb5e6126640 10 monclient(hunting): handle_auth_done global_id 194212 payload 824 2024-04-26T00:36:47.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.308+0000 7fb5e6126640 1 -- 172.21.15.37:0/408291998 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb5e006e930 con 0x7fb5e01319a0 2024-04-26T00:36:47.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.312+0000 7fb5e6126640 1 -- 172.21.15.37:0/408291998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e01319a0 msgr2=0x7fb5e0131db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:47.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.312+0000 7fb5e6126640 1 --2- 172.21.15.37:0/408291998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e01319a0 0x7fb5e0131db0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc00f770 tx=0x7fb5dc003770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:47.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.312+0000 7fb5d6ffd640 1 -- 172.21.15.37:0/408291998 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5dc006e00 con 0x7fb5e01319a0 2024-04-26T00:36:47.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.312+0000 7fb5d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:47.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.312+0000 7fb5d6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:36:47.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.312+0000 7fb5d6ffd640 10 monclient: dump: 2024-04-26T00:36:47.324 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:47.324 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:47.324 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:47.324 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:47.324 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:47.324 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:47.324 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:47.325 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:47.325 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:47.325 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:47.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.312+0000 7fb5d6ffd640 1 -- 172.21.15.37:0/408291998 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4547 (secure 0 0 0) 0x7fb5dc006e00 con 0x7fb5e01319a0 2024-04-26T00:36:47.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.312+0000 7fb5d6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:36:47.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.312+0000 7fb5d6ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:36:47.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.312+0000 7fb5d6ffd640 1 -- 172.21.15.37:0/408291998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e01319a0 msgr2=0x7fb5e0131db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:47.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.312+0000 7fb5d6ffd640 1 --2- 172.21.15.37:0/408291998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e01319a0 0x7fb5e0131db0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc00f770 tx=0x7fb5dc003770 comp rx=0 tx=0).stop 2024-04-26T00:36:47.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.312+0000 7fb5d6ffd640 1 -- 172.21.15.37:0/408291998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e01319a0 msgr2=0x7fb5e0131db0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:36:47.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.312+0000 7fb5d6ffd640 1 --2- 172.21.15.37:0/408291998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5e01319a0 0x7fb5e0131db0 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc00f770 tx=0x7fb5dc003770 comp rx=0 tx=0).stop 2024-04-26T00:36:47.326 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:36:47.326 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:36:47.326 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:36:47.326 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:36:47.326 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 126, 2024-04-26T00:36:47.326 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 55887, 2024-04-26T00:36:47.326 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:36:47.326 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:36:47.326 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:36:47.326 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:36:47.326 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:36:47.326 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 7, 2024-04-26T00:36:47.327 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:36:47.327 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:36:47.327 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:36:47.327 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:36:47.327 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:36:47.327 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:36:47.327 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:36:47.327 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:36:47.327 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:36:47.327 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:36:47.327 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:36:47.327 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:36:47.327 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:36:47.327 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:36:47.328 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:36:47.328 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:36:47.328 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:36:47.328 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:36:47.328 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:36:47.328 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:36:47.328 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:36:47.328 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:36:47.328 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:36:47.328 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:36:47.328 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:36:47.328 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:36:47.328 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:36:47.328 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:36:47.329 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:47.329 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:47.329 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:36:47.329 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:36:47.329 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:36:47.329 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:36:47.329 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:36:47.329 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:36:47.329 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:36:47.329 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:36:47.329 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:36:47.329 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:36:47.329 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:36:47.329 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:36:47.329 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:36:47.330 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:36:47.330 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:36:47.330 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:36:47.330 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:36:47.330 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:36:47.330 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:36:47.330 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:36:47.330 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:36:47.330 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:36:47.330 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:36:47.330 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:36:47.330 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:36:47.330 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:36:47.331 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:36:47.331 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:36:47.331 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:36:47.331 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:36:47.331 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:47.331 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:36:47.331 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:47.331 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:36:47.331 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:36:47.331 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:36:47.331 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:36:47.331 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:47.331 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:36:47.332 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:36:47.332 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:47.332 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:47.332 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:47.332 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:36:47.332 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:36:47.332 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:47.332 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:47.332 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:47.332 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:47.332 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:36:47.332 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:36:47.333 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:36:47.333 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:36:47.333 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:36:47.333 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:47.333 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:47.333 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:36:47.333 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:36:47.333 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:36:47.333 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:36:47.333 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:47.333 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:36:47.333 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:36:47.333 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:47.333 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:47.334 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:47.334 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:36:47.334 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:36:47.334 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:47.334 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:47.334 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:47.334 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:47.334 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:36:47.334 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:36:47.334 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:36:47.334 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:36:47.334 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:36:47.334 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:47.335 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:47.335 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:36:47.335 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:36:47.335 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:36:47.335 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:36:47.335 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:47.335 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:36:47.335 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:36:47.335 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:47.335 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:47.335 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:47.335 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:36:47.335 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:36:47.335 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:36:47.335 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:47.336 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:47.336 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:47.336 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:36:47.336 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:36:47.336 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:36:47.336 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:36:47.336 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:36:47.336 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:47.336 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:47.336 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:47.336 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:36:47.336 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:36:47.337 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:47.337 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:36:47.337 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:36:47.337 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:36:47.337 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:47.337 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:36:47.337 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:36:47.337 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:36:47.337 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:36:47.338 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:36:47.338 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:36:47.338 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:36:47.338 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:36:47.338 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:36:47.338 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:36:47.338 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:36:47.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.312+0000 7fb5e838a640 1 -- 172.21.15.37:0/408291998 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb5c4035ed0 msgr2=0x7fb5c4038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:47.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.312+0000 7fb5e838a640 1 --2- 172.21.15.37:0/408291998 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb5c4035ed0 0x7fb5c4038390 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fb5e01387a0 tx=0x7fb5d800b040 comp rx=0 tx=0).stop 2024-04-26T00:36:47.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.312+0000 7fb5e838a640 10 monclient: shutdown 2024-04-26T00:36:47.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.312+0000 7fb5e838a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:36:47.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.312+0000 7fb5e838a640 1 -- 172.21.15.37:0/408291998 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5e0130eb0 msgr2=0x7fb5e0136640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:47.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.312+0000 7fb5e838a640 1 --2- 172.21.15.37:0/408291998 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5e0130eb0 0x7fb5e0136640 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb5d00b0040 tx=0x7fb5d0019150 comp rx=0 tx=0).stop 2024-04-26T00:36:47.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.312+0000 7fb5e6927640 1 -- 172.21.15.37:0/408291998 reap_dead start 2024-04-26T00:36:47.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.312+0000 7fb5e6927640 1 -- 172.21.15.37:0/408291998 reap_dead start 2024-04-26T00:36:47.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.312+0000 7fb5e838a640 1 -- 172.21.15.37:0/408291998 shutdown_connections 2024-04-26T00:36:47.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.312+0000 7fb5e838a640 1 -- 172.21.15.37:0/408291998 >> 172.21.15.37:0/408291998 conn(0x7fb5e00a1d50 msgr2=0x7fb5e00a3050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:36:47.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.312+0000 7fb5e838a640 1 -- 172.21.15.37:0/408291998 shutdown_connections 2024-04-26T00:36:47.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.312+0000 7fb5e838a640 1 -- 172.21.15.37:0/408291998 wait complete. 2024-04-26T00:36:47.342 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:36:47.342 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-04-26T00:36:47.342 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-26T00:36:47.342 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-04-26T00:36:47.342 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-04-26T00:36:47.342 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-26T00:36:47.343 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T00:36:47.366 INFO:tasks.ceph.mon.b:Stopped 2024-04-26T00:36:47.366 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-26T00:36:47.366 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:36:47.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.488+0000 7ff832713640 10 monclient: get_monmap_and_config 2024-04-26T00:36:47.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.488+0000 7ff832713640 10 monclient: build_initial_monmap 2024-04-26T00:36:47.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.488+0000 7ff832713640 10 monclient: monmap: 2024-04-26T00:36:47.488 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:36:47.488 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:47.488 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:36:47.492934+0000 2024-04-26T00:36:47.488 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:36:47.492934+0000 2024-04-26T00:36:47.488 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:36:47.488 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:47.488 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:36:47.488 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:36:47.488 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:36:47.488 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:36:47.488 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:36:47.489 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:47.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.488+0000 7ff832713640 1 Processor -- start 2024-04-26T00:36:47.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.488+0000 7ff832713640 1 -- start start 2024-04-26T00:36:47.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.488+0000 7ff832713640 10 monclient: init 2024-04-26T00:36:47.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.488+0000 7ff832713640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:47.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.488+0000 7ff832713640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-04-26T00:36:47.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.488+0000 7ff832713640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff82c136e10 0x7ff82c09c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:47.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.488+0000 7ff832713640 10 monclient(hunting): picked mon.noname-c con 0x7ff82c136e10 addr v2:172.21.15.45:3300/0 2024-04-26T00:36:47.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.488+0000 7ff832713640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff82c1371f0 addr v1:172.21.15.45:6789/0 2024-04-26T00:36:47.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.488+0000 7ff832713640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff82c13c340 addr v1:172.21.15.37:6789/0 2024-04-26T00:36:47.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.488+0000 7ff832713640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff82c09b680 con 0x7ff82c13c340 2024-04-26T00:36:47.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.488+0000 7ff832713640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff82c09bbd0 con 0x7ff82c1371f0 2024-04-26T00:36:47.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.488+0000 7ff832713640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:47.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.488+0000 7ff832713640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7ff82c09b8f0 con 0x7ff82c136e10 2024-04-26T00:36:47.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.488+0000 7ff832713640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:47.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.488+0000 7ff832713640 10 monclient(hunting): authenticate will time out at 1988.798614s 2024-04-26T00:36:47.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.488+0000 7ff82bfff640 1 -- >> v2:172.21.15.45:3300/0 conn(0x7ff82c136e10 msgr2=0x7ff82c09c5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:36:47.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.488+0000 7ff82bfff640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff82c136e10 0x7ff82c09c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:36:47.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.488+0000 7ff82b7fe640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7ff82c1371f0 legacy=0x7ff82c137600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:36:47.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.488+0000 7ff830cb0640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7ff82c13c340 0x7ff82c13e730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:34058/0 (socket says 172.21.15.37:34058) 2024-04-26T00:36:47.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.488+0000 7ff830cb0640 1 -- 172.21.15.37:0/2015778479 learned_addr learned my addr 172.21.15.37:0/2015778479 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:36:47.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff830cb0640 1 -- 172.21.15.37:0/2015778479 >> v1:172.21.15.37:6789/0 conn(0x7ff82c13c340 legacy=0x7ff82c13e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:47.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82affd640 1 -- 172.21.15.37:0/2015778479 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1792492573 0 0) 0x7ff82c09b680 con 0x7ff82c13c340 2024-04-26T00:36:47.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82affd640 10 monclient(hunting): _negotiate 2024-04-26T00:36:47.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82affd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:47.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:47.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82affd640 10 monclient(hunting): my global_id is 144753 2024-04-26T00:36:47.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82affd640 1 -- 172.21.15.37:0/2015778479 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff818003530 con 0x7ff82c13c340 2024-04-26T00:36:47.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82affd640 1 -- 172.21.15.37:0/2015778479 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2721392383 0 0) 0x7ff818003530 con 0x7ff82c13c340 2024-04-26T00:36:47.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82affd640 1 -- 172.21.15.37:0/2015778479 >> v2:172.21.15.45:3300/0 conn(0x7ff82c136e10 msgr2=0x7ff82c09c5b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:36:47.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82affd640 1 --2- 172.21.15.37:0/2015778479 >> v2:172.21.15.45:3300/0 conn(0x7ff82c136e10 0x7ff82c09c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:47.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82affd640 1 -- 172.21.15.37:0/2015778479 >> v1:172.21.15.45:6789/0 conn(0x7ff82c1371f0 legacy=0x7ff82c137600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:36:47.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82affd640 10 monclient: _finish_hunting 0 2024-04-26T00:36:47.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82affd640 1 monclient: found mon.noname-a-legacy 2024-04-26T00:36:47.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:47.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82affd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.37:6789/0 2024-04-26T00:36:47.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82affd640 1 -- 172.21.15.37:0/2015778479 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff82c0a5d90 con 0x7ff82c13c340 2024-04-26T00:36:47.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82affd640 10 monclient: _finish_auth 0 2024-04-26T00:36:47.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82affd640 10 monclient: _check_auth_tickets 2024-04-26T00:36:47.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:47.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82affd640 1 -- 172.21.15.37:0/2015778479 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 866326386 0 0) 0x7ff81c001930 con 0x7ff82c13c340 2024-04-26T00:36:47.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:47.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82affd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-26T00:36:47.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82affd640 10 monclient: dump: 2024-04-26T00:36:47.497 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:47.497 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:47.498 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:47.498 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:47.498 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:47.498 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:47.498 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:47.498 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:47.498 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:47.498 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:47.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82affd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] but i'm connected to v1:172.21.15.37:6789/0, reconnecting 2024-04-26T00:36:47.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82affd640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:47.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82affd640 1 -- 172.21.15.37:0/2015778479 >> v1:172.21.15.37:6789/0 conn(0x7ff82c13c340 legacy=0x7ff82c13e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:47.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82affd640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:36:47.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82affd640 1 --2- 172.21.15.37:0/2015778479 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff8180039c0 0x7ff818003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:47.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82affd640 10 monclient(hunting): picked mon.a con 0x7ff8180039c0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:36:47.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82affd640 1 --2- 172.21.15.37:0/2015778479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff818004810 0x7ff818004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:47.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82affd640 10 monclient(hunting): picked mon.b con 0x7ff818004810 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:36:47.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82affd640 1 --2- 172.21.15.37:0/2015778479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff818005380 0x7ff81800b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:47.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82affd640 10 monclient(hunting): picked mon.c con 0x7ff818005380 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:36:47.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82affd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:47.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82affd640 1 -- 172.21.15.37:0/2015778479 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff82c09b8f0 con 0x7ff8180039c0 2024-04-26T00:36:47.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82affd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:47.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82affd640 1 -- 172.21.15.37:0/2015778479 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff81800c2d0 con 0x7ff818004810 2024-04-26T00:36:47.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82affd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:47.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82affd640 1 -- 172.21.15.37:0/2015778479 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff81800c4d0 con 0x7ff818005380 2024-04-26T00:36:47.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82affd640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:47.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82affd640 10 monclient(hunting): _finish_auth 0 2024-04-26T00:36:47.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82bfff640 1 --2- 172.21.15.37:0/2015778479 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff8180039c0 0x7ff818003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:47.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82bfff640 10 monclient(hunting): get_auth_request con 0x7ff8180039c0 auth_method 0 2024-04-26T00:36:47.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:47.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:47.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:36:47.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff830cb0640 1 --2- 172.21.15.37:0/2015778479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff818005380 0x7ff81800b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:47.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:47.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:47.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:36:47.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff830cb0640 10 monclient(hunting): get_auth_request con 0x7ff818005380 auth_method 0 2024-04-26T00:36:47.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82b7fe640 1 -- 172.21.15.37:0/2015778479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff818004810 msgr2=0x7ff818004c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:36:47.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff830cb0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:47.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff830cb0640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:47.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff830cb0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:36:47.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82b7fe640 1 --2- 172.21.15.37:0/2015778479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff818004810 0x7ff818004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:36:47.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff830cb0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:47.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff830cb0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:47.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff830cb0640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:36:47.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82bfff640 10 monclient(hunting): handle_auth_done global_id 144753 payload 293 2024-04-26T00:36:47.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82bfff640 1 -- 172.21.15.37:0/2015778479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff818005380 msgr2=0x7ff81800b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:36:47.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82bfff640 1 --2- 172.21.15.37:0/2015778479 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff818005380 0x7ff81800b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:47.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82bfff640 1 -- 172.21.15.37:0/2015778479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff818004810 msgr2=0x7ff818004c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:36:47.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82bfff640 1 --2- 172.21.15.37:0/2015778479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff818004810 0x7ff818004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:47.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82bfff640 10 monclient: _finish_hunting 0 2024-04-26T00:36:47.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82bfff640 1 monclient: found mon.a 2024-04-26T00:36:47.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:47.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82bfff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:47.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82bfff640 1 -- 172.21.15.37:0/2015778479 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff818003530 con 0x7ff8180039c0 2024-04-26T00:36:47.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82bfff640 10 monclient: _finish_auth 0 2024-04-26T00:36:47.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff830cb0640 1 -- 172.21.15.37:0/2015778479 reap_dead start 2024-04-26T00:36:47.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82bfff640 10 monclient: _check_auth_tickets 2024-04-26T00:36:47.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff82bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:47.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff832713640 5 monclient: authenticate success, global_id 144753 2024-04-26T00:36:47.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.492+0000 7ff832713640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:36:47.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.496+0000 7ff82bfff640 1 -- 172.21.15.37:0/2015778479 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff8180039c0 msgr2=0x7ff818003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:47.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.496+0000 7ff82bfff640 1 --2- 172.21.15.37:0/2015778479 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff8180039c0 0x7ff818003dc0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff824003940 tx=0x7ff82401c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40d21c1c411fbd2e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:47.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.496+0000 7ff82affd640 1 -- 172.21.15.37:0/2015778479 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff824009520 con 0x7ff8180039c0 2024-04-26T00:36:47.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.496+0000 7ff82affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:47.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.496+0000 7ff82affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:36:47.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.496+0000 7ff82affd640 10 monclient: dump: 2024-04-26T00:36:47.510 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:47.510 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:47.510 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:47.510 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:47.510 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:47.510 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:47.510 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:47.510 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:47.510 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:47.511 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:47.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.496+0000 7ff82affd640 1 -- 172.21.15.37:0/2015778479 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff824003bc0 con 0x7ff8180039c0 2024-04-26T00:36:47.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.496+0000 7ff832713640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:36:47.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.496+0000 7ff82affd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:36:47.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.496+0000 7ff832713640 10 monclient: get_monmap_and_config success 2024-04-26T00:36:47.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.496+0000 7ff82affd640 1 -- 172.21.15.37:0/2015778479 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff824005070 con 0x7ff8180039c0 2024-04-26T00:36:47.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.496+0000 7ff832713640 4 set_mon_vals no callback set 2024-04-26T00:36:47.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.496+0000 7ff82affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:47.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.496+0000 7ff82affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:36:47.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.496+0000 7ff82affd640 10 monclient: dump: 2024-04-26T00:36:47.511 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:47.511 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:47.511 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:47.511 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:47.511 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:47.512 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:47.512 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:47.512 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:47.512 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:47.513 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:47.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff832713640 10 monclient: shutdown 2024-04-26T00:36:47.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff832713640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:36:47.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff832713640 1 -- 172.21.15.37:0/2015778479 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff8180039c0 msgr2=0x7ff818003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:47.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff832713640 1 --2- 172.21.15.37:0/2015778479 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff8180039c0 0x7ff818003dc0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff824003940 tx=0x7ff82401c420 comp rx=0 tx=0).stop 2024-04-26T00:36:47.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff832713640 1 -- 172.21.15.37:0/2015778479 shutdown_connections 2024-04-26T00:36:47.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff832713640 1 --2- 172.21.15.37:0/2015778479 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff8180039c0 0x7ff818003dc0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:47.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff832713640 1 -- 172.21.15.37:0/2015778479 >> 172.21.15.37:0/2015778479 conn(0x7ff82c0a64e0 msgr2=0x7ff82c0a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:36:47.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff832713640 1 -- 172.21.15.37:0/2015778479 shutdown_connections 2024-04-26T00:36:47.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff832713640 1 -- 172.21.15.37:0/2015778479 wait complete. 2024-04-26T00:36:47.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff832713640 10 monclient: build_initial_monmap 2024-04-26T00:36:47.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff832713640 10 monclient: monmap: 2024-04-26T00:36:47.514 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:36:47.514 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:36:47.514 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:36:47.514 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:36:47.514 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:36:47.515 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:36:47.515 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:36:47.515 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:36:47.515 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:36:47.515 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:47.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff832713640 1 Processor -- start 2024-04-26T00:36:47.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff832713640 1 -- start start 2024-04-26T00:36:47.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff832713640 10 monclient: init 2024-04-26T00:36:47.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff832713640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:47.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff832713640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:36:47.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff832713640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8180039c0 0x7ff82c1202f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:47.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff832713640 10 monclient(hunting): picked mon.noname-b con 0x7ff8180039c0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:36:47.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff832713640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff82c134980 0x7ff82c120830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:47.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff832713640 10 monclient(hunting): picked mon.noname-c con 0x7ff82c134980 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:36:47.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff832713640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff82c120d70 0x7ff82c1d5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:47.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff832713640 10 monclient(hunting): picked mon.noname-a con 0x7ff82c120d70 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:36:47.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff832713640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:47.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff832713640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff82c136120 con 0x7ff82c120d70 2024-04-26T00:36:47.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff832713640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:47.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff832713640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff82c135fa0 con 0x7ff8180039c0 2024-04-26T00:36:47.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff832713640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:47.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff832713640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff82c1362a0 con 0x7ff82c134980 2024-04-26T00:36:47.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff832713640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:47.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff832713640 10 monclient(hunting): authenticate will time out at 1988.810689s 2024-04-26T00:36:47.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff82bfff640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8180039c0 msgr2=0x7ff82c1202f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:36:47.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff830cb0640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff82c120d70 0x7ff82c1d5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:47.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff82bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8180039c0 0x7ff82c1202f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:36:47.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff830cb0640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff82c120d70 0x7ff82c1d5540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:50440/0 (socket says 172.21.15.37:50440) 2024-04-26T00:36:47.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff830cb0640 1 -- 172.21.15.37:0/2218335676 learned_addr learned my addr 172.21.15.37:0/2218335676 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:36:47.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff830cb0640 10 monclient(hunting): get_auth_request con 0x7ff82c120d70 auth_method 0 2024-04-26T00:36:47.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff830cb0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:47.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff830cb0640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:47.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff830cb0640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:47.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff82b7fe640 1 --2- 172.21.15.37:0/2218335676 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff82c134980 0x7ff82c120830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:47.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff830cb0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:47.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff830cb0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:47.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff830cb0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:47.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff82b7fe640 10 monclient(hunting): get_auth_request con 0x7ff82c134980 auth_method 0 2024-04-26T00:36:47.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff82b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:47.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff82b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:47.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff82b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:47.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff82b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:47.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff82b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:47.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff82b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:47.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff830cb0640 10 monclient(hunting): handle_auth_done global_id 144756 payload 824 2024-04-26T00:36:47.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff830cb0640 1 -- 172.21.15.37:0/2218335676 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff82c134980 msgr2=0x7ff82c120830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:47.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff830cb0640 1 --2- 172.21.15.37:0/2218335676 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff82c134980 0x7ff82c120830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:47.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff830cb0640 1 -- 172.21.15.37:0/2218335676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8180039c0 msgr2=0x7ff82c1202f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:36:47.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff830cb0640 1 --2- 172.21.15.37:0/2218335676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8180039c0 0x7ff82c1202f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:47.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff830cb0640 10 monclient: _finish_hunting 0 2024-04-26T00:36:47.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff830cb0640 1 monclient: found mon.noname-a 2024-04-26T00:36:47.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff830cb0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:47.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff830cb0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:36:47.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.500+0000 7ff830cb0640 1 -- 172.21.15.37:0/2218335676 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff82c123ee0 con 0x7ff82c120d70 2024-04-26T00:36:47.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.504+0000 7ff830cb0640 1 -- 172.21.15.37:0/2218335676 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff82c120d70 msgr2=0x7ff82c1d5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:47.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.504+0000 7ff830cb0640 1 --2- 172.21.15.37:0/2218335676 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff82c120d70 0x7ff82c1d5540 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff82c09d4b0 tx=0x7ff81c007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:47.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.504+0000 7ff828ff9640 1 -- 172.21.15.37:0/2218335676 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff81c007850 con 0x7ff82c120d70 2024-04-26T00:36:47.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.504+0000 7ff828ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:47.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.504+0000 7ff828ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:36:47.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.504+0000 7ff830cb0640 1 --2- 172.21.15.37:0/2218335676 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff82c120d70 0x7ff82c1d5540 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff82c09d4b0 tx=0x7ff81c007720 comp rx=0 tx=0).queue_received will delay after 1.71409e+09 on 0x7ff81c018070 config(0 keys) 2024-04-26T00:36:47.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.504+0000 7ff828ff9640 10 monclient: dump: 2024-04-26T00:36:47.520 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:47.520 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:47.520 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:47.520 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:47.520 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:47.520 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:47.520 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:47.520 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:47.521 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:47.521 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:47.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.504+0000 7ff828ff9640 10 monclient: _finish_auth 0 2024-04-26T00:36:47.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.504+0000 7ff828ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:36:47.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.504+0000 7ff828ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:47.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.504+0000 7ff832713640 5 monclient: authenticate success, global_id 144756 2024-04-26T00:36:47.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.504+0000 7ff828ff9640 1 -- 172.21.15.37:0/2218335676 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff81c018070 con 0x7ff82c120d70 2024-04-26T00:36:47.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.504+0000 7ff832713640 10 monclient: _renew_subs 2024-04-26T00:36:47.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.504+0000 7ff832713640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:47.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.504+0000 7ff832713640 1 -- 172.21.15.37:0/2218335676 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff82c122090 con 0x7ff82c120d70 2024-04-26T00:36:47.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.504+0000 7ff828ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:36:47.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.504+0000 7ff82a7fc640 4 set_mon_vals no callback set 2024-04-26T00:36:47.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.504+0000 7ff832713640 10 monclient: _renew_subs 2024-04-26T00:36:47.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.504+0000 7ff832713640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:47.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.504+0000 7ff832713640 1 -- 172.21.15.37:0/2218335676 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff82c06d5f0 con 0x7ff82c120d70 2024-04-26T00:36:47.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.504+0000 7ff832713640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:47.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.504+0000 7ff832713640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:47.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.504+0000 7ff832713640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:47.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.504+0000 7ff828ff9640 1 -- 172.21.15.37:0/2218335676 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff81c00fd60 con 0x7ff82c120d70 2024-04-26T00:36:47.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.504+0000 7ff832713640 1 -- 172.21.15.37:0/2218335676 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff82c075910 con 0x7ff82c120d70 2024-04-26T00:36:47.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.504+0000 7ff828ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:47.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.504+0000 7ff828ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:36:47.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.504+0000 7ff828ff9640 10 monclient: dump: 2024-04-26T00:36:47.522 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:47.522 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:47.523 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:47.523 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:47.523 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:47.523 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:47.523 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:47.523 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:47.523 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:47.523 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:47.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.504+0000 7ff828ff9640 1 -- 172.21.15.37:0/2218335676 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff81c01d460 con 0x7ff82c120d70 2024-04-26T00:36:47.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.504+0000 7ff828ff9640 1 --2- 172.21.15.37:0/2218335676 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff80c035e60 0x7ff80c038320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:47.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.504+0000 7ff82bfff640 1 --2- 172.21.15.37:0/2218335676 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff80c035e60 0x7ff80c038320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:47.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.504+0000 7ff82bfff640 10 monclient: get_auth_request con 0x7ff80c035e60 auth_method 0 2024-04-26T00:36:47.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.508+0000 7ff82bfff640 1 --2- 172.21.15.37:0/2218335676 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff80c035e60 0x7ff80c038320 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7ff824008000 tx=0x7ff824005a90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:47.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:47.512+0000 7ff828ff9640 1 -- 172.21.15.37:0/2218335676 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(662..662 src has 370..662) ==== 3899+0+0 (secure 0 0 0) 0x7ff81c048970 con 0x7ff82c120d70 2024-04-26T00:36:48.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:48.332+0000 7ff828ff9640 1 -- 172.21.15.37:0/2218335676 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff81c0476e0 con 0x7ff82c120d70 2024-04-26T00:36:48.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:48.332+0000 7ff828ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:48.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:48.332+0000 7ff828ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:36:48.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:48.568+0000 7ff832713640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:36:48.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:48.568+0000 7ff832713640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:36:48.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:48.568+0000 7ff832713640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:48.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:48.568+0000 7ff832713640 1 -- 172.21.15.37:0/2218335676 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff82c066f40 con 0x7ff82c120d70 2024-04-26T00:36:48.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:48.568+0000 7ff828ff9640 1 -- 172.21.15.37:0/2218335676 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7ff81c01b520 con 0x7ff82c120d70 2024-04-26T00:36:48.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:48.568+0000 7ff828ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:36:48.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:48.568+0000 7ff828ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:36:48.568 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":126,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:36:48.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:48.568+0000 7ff832713640 1 -- 172.21.15.37:0/2218335676 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff80c035e60 msgr2=0x7ff80c038320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:48.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:48.568+0000 7ff832713640 1 --2- 172.21.15.37:0/2218335676 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff80c035e60 0x7ff80c038320 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7ff824008000 tx=0x7ff824005a90 comp rx=0 tx=0).stop 2024-04-26T00:36:48.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:48.568+0000 7ff832713640 10 monclient: shutdown 2024-04-26T00:36:48.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:48.568+0000 7ff832713640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:36:48.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:48.568+0000 7ff832713640 1 -- 172.21.15.37:0/2218335676 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff82c120d70 msgr2=0x7ff82c1d5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:48.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:48.568+0000 7ff832713640 1 --2- 172.21.15.37:0/2218335676 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff82c120d70 0x7ff82c1d5540 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff82c09d4b0 tx=0x7ff81c007720 comp rx=0 tx=0).stop 2024-04-26T00:36:48.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:48.568+0000 7ff832713640 1 -- 172.21.15.37:0/2218335676 shutdown_connections 2024-04-26T00:36:48.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:48.568+0000 7ff832713640 1 --2- 172.21.15.37:0/2218335676 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff80c035e60 0x7ff80c038320 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:48.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:48.568+0000 7ff832713640 1 --2- 172.21.15.37:0/2218335676 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff82c120d70 0x7ff82c1d5540 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:48.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:48.568+0000 7ff832713640 1 --2- 172.21.15.37:0/2218335676 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff82c134980 0x7ff82c120830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:48.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:48.568+0000 7ff832713640 1 --2- 172.21.15.37:0/2218335676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8180039c0 0x7ff82c1202f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:48.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:48.568+0000 7ff832713640 1 -- 172.21.15.37:0/2218335676 >> 172.21.15.37:0/2218335676 conn(0x7ff82c0a64e0 msgr2=0x7ff82c1353e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:36:48.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:48.568+0000 7ff832713640 1 -- 172.21.15.37:0/2218335676 shutdown_connections 2024-04-26T00:36:48.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:48.568+0000 7ff832713640 1 -- 172.21.15.37:0/2218335676 wait complete. 2024-04-26T00:36:51.587 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:36:51.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.720+0000 7f5559f88640 10 monclient: get_monmap_and_config 2024-04-26T00:36:51.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.720+0000 7f5559f88640 10 monclient: build_initial_monmap 2024-04-26T00:36:51.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.720+0000 7f5559f88640 10 monclient: monmap: 2024-04-26T00:36:51.720 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:36:51.720 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:51.720 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:36:51.725159+0000 2024-04-26T00:36:51.720 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:36:51.725159+0000 2024-04-26T00:36:51.720 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:36:51.720 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:51.721 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:36:51.721 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:36:51.721 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:36:51.721 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:36:51.721 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:36:51.721 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:51.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.720+0000 7f5559f88640 1 Processor -- start 2024-04-26T00:36:51.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.720+0000 7f5559f88640 1 -- start start 2024-04-26T00:36:51.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.720+0000 7f5559f88640 10 monclient: init 2024-04-26T00:36:51.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.720+0000 7f5559f88640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:51.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.720+0000 7f5559f88640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-04-26T00:36:51.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.720+0000 7f5559f88640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5554133b20 addr v1:172.21.15.45:6789/0 2024-04-26T00:36:51.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.720+0000 7f5559f88640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5554133f00 0x7f5554134350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:51.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.720+0000 7f5559f88640 10 monclient(hunting): picked mon.noname-c con 0x7f5554133f00 addr v2:172.21.15.45:3300/0 2024-04-26T00:36:51.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.720+0000 7f5559f88640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5554139050 addr v1:172.21.15.37:6789/0 2024-04-26T00:36:51.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.720+0000 7f5559f88640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f555409fe20 con 0x7f5554139050 2024-04-26T00:36:51.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.720+0000 7f5559f88640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5554072d00 con 0x7f5554133b20 2024-04-26T00:36:51.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.720+0000 7f5559f88640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:51.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.720+0000 7f5559f88640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f555409ba60 con 0x7f5554133f00 2024-04-26T00:36:51.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.720+0000 7f5559f88640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:51.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.720+0000 7f5559f88640 10 monclient(hunting): authenticate will time out at 1993.031119s 2024-04-26T00:36:51.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.720+0000 7f55537fe640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f5554133b20 legacy=0x7f555409e9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:36:51.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.720+0000 7f5552ffd640 1 -- >> v2:172.21.15.45:3300/0 conn(0x7f5554133f00 msgr2=0x7f5554134350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:36:51.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.720+0000 7f5552ffd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5554133f00 0x7f5554134350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:36:51.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.720+0000 7f5553fff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f5554139050 0x7f555413b440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:47896/0 (socket says 172.21.15.37:47896) 2024-04-26T00:36:51.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.720+0000 7f5553fff640 1 -- 172.21.15.37:0/1495230781 learned_addr learned my addr 172.21.15.37:0/1495230781 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:36:51.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f5553fff640 1 -- 172.21.15.37:0/1495230781 >> v1:172.21.15.37:6789/0 conn(0x7f5554139050 legacy=0x7f555413b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:51.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55527fc640 1 -- 172.21.15.37:0/1495230781 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2698544307 0 0) 0x7f554c0017e0 con 0x7f5554139050 2024-04-26T00:36:51.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55527fc640 10 monclient(hunting): _negotiate 2024-04-26T00:36:51.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55527fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:51.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55527fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:51.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55527fc640 10 monclient(hunting): my global_id is 144759 2024-04-26T00:36:51.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55527fc640 1 -- 172.21.15.37:0/1495230781 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f553c003530 con 0x7f5554139050 2024-04-26T00:36:51.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55527fc640 1 -- 172.21.15.37:0/1495230781 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 515707285 0 0) 0x7f553c003530 con 0x7f5554139050 2024-04-26T00:36:51.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55527fc640 1 -- 172.21.15.37:0/1495230781 >> v2:172.21.15.45:3300/0 conn(0x7f5554133f00 msgr2=0x7f5554134350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:36:51.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55527fc640 1 --2- 172.21.15.37:0/1495230781 >> v2:172.21.15.45:3300/0 conn(0x7f5554133f00 0x7f5554134350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:51.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55527fc640 1 -- 172.21.15.37:0/1495230781 >> v1:172.21.15.45:6789/0 conn(0x7f5554133b20 legacy=0x7f555409e9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:36:51.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55527fc640 10 monclient: _finish_hunting 0 2024-04-26T00:36:51.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55527fc640 1 monclient: found mon.noname-a-legacy 2024-04-26T00:36:51.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:51.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55527fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.37:6789/0 2024-04-26T00:36:51.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55527fc640 1 -- 172.21.15.37:0/1495230781 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f555409dd10 con 0x7f5554139050 2024-04-26T00:36:51.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55527fc640 10 monclient: _finish_auth 0 2024-04-26T00:36:51.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55527fc640 10 monclient: _check_auth_tickets 2024-04-26T00:36:51.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:51.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55527fc640 1 -- 172.21.15.37:0/1495230781 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 866326386 0 0) 0x7f554c001e80 con 0x7f5554139050 2024-04-26T00:36:51.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:51.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55527fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-26T00:36:51.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55527fc640 10 monclient: dump: 2024-04-26T00:36:51.728 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:51.728 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:51.728 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:51.728 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:51.728 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:51.728 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:51.729 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:51.729 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:51.729 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:51.729 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:51.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55527fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] but i'm connected to v1:172.21.15.37:6789/0, reconnecting 2024-04-26T00:36:51.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55527fc640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:51.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55527fc640 1 -- 172.21.15.37:0/1495230781 >> v1:172.21.15.37:6789/0 conn(0x7f5554139050 legacy=0x7f555413b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:51.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55527fc640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:36:51.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55527fc640 1 --2- 172.21.15.37:0/1495230781 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f553c0039c0 0x7f553c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:51.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55527fc640 10 monclient(hunting): picked mon.c con 0x7f553c0039c0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:36:51.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55527fc640 1 --2- 172.21.15.37:0/1495230781 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f553c004810 0x7f553c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:51.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55527fc640 10 monclient(hunting): picked mon.a con 0x7f553c004810 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:36:51.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55527fc640 1 --2- 172.21.15.37:0/1495230781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f553c005380 0x7f553c00dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:51.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55527fc640 10 monclient(hunting): picked mon.b con 0x7f553c005380 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:36:51.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55527fc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:51.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55527fc640 1 -- 172.21.15.37:0/1495230781 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f555409ba60 con 0x7f553c004810 2024-04-26T00:36:51.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55527fc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:51.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55527fc640 1 -- 172.21.15.37:0/1495230781 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f553c00e4e0 con 0x7f553c005380 2024-04-26T00:36:51.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55527fc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:51.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55527fc640 1 -- 172.21.15.37:0/1495230781 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f553c00e6e0 con 0x7f553c0039c0 2024-04-26T00:36:51.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55527fc640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:51.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55527fc640 10 monclient(hunting): _finish_auth 0 2024-04-26T00:36:51.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f5552ffd640 1 --2- 172.21.15.37:0/1495230781 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f553c004810 0x7f553c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:51.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55537fe640 1 --2- 172.21.15.37:0/1495230781 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f553c0039c0 0x7f553c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:51.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f5552ffd640 10 monclient(hunting): get_auth_request con 0x7f553c004810 auth_method 0 2024-04-26T00:36:51.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f5552ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:51.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f5552ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:51.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f5552ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:36:51.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f5553fff640 1 -- 172.21.15.37:0/1495230781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f553c005380 msgr2=0x7f553c00dc00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:36:51.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55537fe640 10 monclient(hunting): get_auth_request con 0x7f553c0039c0 auth_method 0 2024-04-26T00:36:51.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:51.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55537fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:51.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55537fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:36:51.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f5553fff640 1 --2- 172.21.15.37:0/1495230781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f553c005380 0x7f553c00dc00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:36:51.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f5552ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:51.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f5552ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:51.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f5552ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:36:51.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:51.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:51.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55537fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:36:51.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55537fe640 10 monclient(hunting): handle_auth_done global_id 144759 payload 293 2024-04-26T00:36:51.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55537fe640 1 -- 172.21.15.37:0/1495230781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f553c005380 msgr2=0x7f553c00dc00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:36:51.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55537fe640 1 --2- 172.21.15.37:0/1495230781 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f553c005380 0x7f553c00dc00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:51.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55537fe640 1 -- 172.21.15.37:0/1495230781 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f553c004810 msgr2=0x7f553c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:36:51.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55537fe640 1 --2- 172.21.15.37:0/1495230781 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f553c004810 0x7f553c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:51.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55537fe640 10 monclient: _finish_hunting 0 2024-04-26T00:36:51.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55537fe640 1 monclient: found mon.c 2024-04-26T00:36:51.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f5553fff640 1 -- 172.21.15.37:0/1495230781 reap_dead start 2024-04-26T00:36:51.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:51.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55537fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:36:51.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55537fe640 1 -- 172.21.15.37:0/1495230781 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f553c003530 con 0x7f553c0039c0 2024-04-26T00:36:51.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55537fe640 10 monclient: _finish_auth 0 2024-04-26T00:36:51.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55537fe640 10 monclient: _check_auth_tickets 2024-04-26T00:36:51.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f55537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:51.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f5552ffd640 1 --2- 172.21.15.37:0/1495230781 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f553c004810 0x7f553c004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:36:51.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f5559f88640 5 monclient: authenticate success, global_id 144759 2024-04-26T00:36:51.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.724+0000 7f5559f88640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:36:51.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.728+0000 7f55537fe640 1 -- 172.21.15.37:0/1495230781 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f553c0039c0 msgr2=0x7f553c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:51.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.728+0000 7f55537fe640 1 --2- 172.21.15.37:0/1495230781 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f553c0039c0 0x7f553c003dc0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5544003940 tx=0x7f554401c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9b44a6a5035827f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:51.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.728+0000 7f55537fe640 1 --2- 172.21.15.37:0/1495230781 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f553c0039c0 0x7f553c003dc0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5544003940 tx=0x7f554401c3f0 comp rx=0 tx=0).queue_received will delay after 1.71409e+09 on 0x7f5544005070 mon_map magic: 0 2024-04-26T00:36:51.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.728+0000 7f55527fc640 1 -- 172.21.15.37:0/1495230781 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5544009520 con 0x7f553c0039c0 2024-04-26T00:36:51.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.728+0000 7f55527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:51.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.728+0000 7f55527fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:36:51.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.728+0000 7f55527fc640 10 monclient: dump: 2024-04-26T00:36:51.738 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:51.738 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:51.738 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:51.738 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:51.738 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:51.738 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:51.738 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:51.738 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:51.738 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:51.739 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:51.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.728+0000 7f55527fc640 1 -- 172.21.15.37:0/1495230781 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5544003bc0 con 0x7f553c0039c0 2024-04-26T00:36:51.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.728+0000 7f55527fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:36:51.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.728+0000 7f5559f88640 10 monclient: get_monmap_and_config success 2024-04-26T00:36:51.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.728+0000 7f5559f88640 4 set_mon_vals no callback set 2024-04-26T00:36:51.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.728+0000 7f5559f88640 10 monclient: shutdown 2024-04-26T00:36:51.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.728+0000 7f5559f88640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:36:51.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.728+0000 7f5559f88640 1 -- 172.21.15.37:0/1495230781 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f553c0039c0 msgr2=0x7f553c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:51.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.728+0000 7f5559f88640 1 --2- 172.21.15.37:0/1495230781 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f553c0039c0 0x7f553c003dc0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5544003940 tx=0x7f554401c3f0 comp rx=0 tx=0).stop 2024-04-26T00:36:51.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.728+0000 7f55527fc640 1 -- 172.21.15.37:0/1495230781 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 0 0 0) 0x7f5544005070 con 0x7f553c0039c0 2024-04-26T00:36:51.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.728+0000 7f55527fc640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-04-26T00:36:51.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.728+0000 7f5559f88640 1 -- 172.21.15.37:0/1495230781 shutdown_connections 2024-04-26T00:36:51.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.728+0000 7f5559f88640 1 --2- 172.21.15.37:0/1495230781 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f553c0039c0 0x7f553c003dc0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:51.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.728+0000 7f5559f88640 1 -- 172.21.15.37:0/1495230781 >> 172.21.15.37:0/1495230781 conn(0x7f55540a1ad0 msgr2=0x7f55540a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:36:51.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.728+0000 7f5559f88640 1 -- 172.21.15.37:0/1495230781 shutdown_connections 2024-04-26T00:36:51.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.728+0000 7f5559f88640 1 -- 172.21.15.37:0/1495230781 wait complete. 2024-04-26T00:36:51.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f5559f88640 10 monclient: build_initial_monmap 2024-04-26T00:36:51.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f5559f88640 10 monclient: monmap: 2024-04-26T00:36:51.740 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:36:51.740 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:36:51.740 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:36:51.740 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:36:51.740 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:36:51.740 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:36:51.740 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:36:51.740 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:36:51.740 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:36:51.740 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:51.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f5559f88640 1 Processor -- start 2024-04-26T00:36:51.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f5559f88640 1 -- start start 2024-04-26T00:36:51.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f5559f88640 10 monclient: init 2024-04-26T00:36:51.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f5559f88640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:51.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f5559f88640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:36:51.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f5559f88640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f553c0039c0 0x7f55541cb470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:51.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f5559f88640 10 monclient(hunting): picked mon.noname-c con 0x7f553c0039c0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:36:51.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f5559f88640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55541cb9b0 0x7f5554071250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:51.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f5559f88640 10 monclient(hunting): picked mon.noname-b con 0x7f55541cb9b0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:36:51.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f5559f88640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55541cbdb0 0x7f5554071790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:51.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f5559f88640 10 monclient(hunting): picked mon.noname-a con 0x7f55541cbdb0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:36:51.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f5559f88640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:51.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f5559f88640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f55541348c0 con 0x7f55541cbdb0 2024-04-26T00:36:51.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f5559f88640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:51.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f5559f88640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5554134740 con 0x7f55541cb9b0 2024-04-26T00:36:51.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f5559f88640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:51.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f5559f88640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5554130f80 con 0x7f553c0039c0 2024-04-26T00:36:51.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f5559f88640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:51.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f5559f88640 10 monclient(hunting): authenticate will time out at 1993.041384s 2024-04-26T00:36:51.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f55537fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f553c0039c0 0x7f55541cb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:51.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f5552ffd640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55541cb9b0 msgr2=0x7f5554071250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:36:51.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f55537fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f553c0039c0 0x7f55541cb470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:47422/0 (socket says 172.21.15.37:47422) 2024-04-26T00:36:51.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f55537fe640 1 -- 172.21.15.37:0/1550005643 learned_addr learned my addr 172.21.15.37:0/1550005643 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:36:51.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f55537fe640 10 monclient(hunting): get_auth_request con 0x7f553c0039c0 auth_method 0 2024-04-26T00:36:51.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f55537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:51.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f55537fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:51.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f5552ffd640 1 --2- 172.21.15.37:0/1550005643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55541cb9b0 0x7f5554071250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:36:51.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f55537fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:51.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f5553fff640 1 --2- 172.21.15.37:0/1550005643 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55541cbdb0 0x7f5554071790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:51.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f55537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:51.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f55537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:51.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f55537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:51.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f5553fff640 10 monclient(hunting): get_auth_request con 0x7f55541cbdb0 auth_method 0 2024-04-26T00:36:51.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f5553fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:51.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f5553fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:51.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f5553fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:51.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f5553fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:51.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f5553fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:51.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f5553fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:51.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f55537fe640 10 monclient(hunting): handle_auth_done global_id 204146 payload 824 2024-04-26T00:36:51.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f55537fe640 1 -- 172.21.15.37:0/1550005643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55541cb9b0 msgr2=0x7f5554071250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:36:51.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f55537fe640 1 --2- 172.21.15.37:0/1550005643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55541cb9b0 0x7f5554071250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:51.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f55537fe640 1 -- 172.21.15.37:0/1550005643 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55541cbdb0 msgr2=0x7f5554071790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:51.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f55537fe640 1 --2- 172.21.15.37:0/1550005643 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55541cbdb0 0x7f5554071790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:51.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f55537fe640 10 monclient: _finish_hunting 0 2024-04-26T00:36:51.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f55537fe640 1 monclient: found mon.noname-c 2024-04-26T00:36:51.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f55537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:51.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f55537fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:36:51.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.732+0000 7f55537fe640 1 -- 172.21.15.37:0/1550005643 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f555411d9c0 con 0x7f553c0039c0 2024-04-26T00:36:51.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.736+0000 7f55537fe640 1 -- 172.21.15.37:0/1550005643 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f553c0039c0 msgr2=0x7f55541cb470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:51.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.736+0000 7f55537fe640 1 --2- 172.21.15.37:0/1550005643 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f553c0039c0 0x7f55541cb470 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5544003940 tx=0x7f553c005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:51.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.736+0000 7f553bfff640 1 -- 172.21.15.37:0/1550005643 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55440bb070 con 0x7f553c0039c0 2024-04-26T00:36:51.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.736+0000 7f553bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:51.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.736+0000 7f553bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:36:51.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.736+0000 7f553bfff640 10 monclient: dump: 2024-04-26T00:36:51.744 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:51.744 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:51.745 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:51.745 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:51.745 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:51.745 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:51.745 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:51.745 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:51.745 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:51.745 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:51.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.736+0000 7f553bfff640 10 monclient: _finish_auth 0 2024-04-26T00:36:51.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.736+0000 7f553bfff640 10 monclient: _check_auth_tickets 2024-04-26T00:36:51.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.736+0000 7f553bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:51.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.736+0000 7f553bfff640 1 -- 172.21.15.37:0/1550005643 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5544009500 con 0x7f553c0039c0 2024-04-26T00:36:51.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.736+0000 7f553bfff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:36:51.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.736+0000 7f553bfff640 1 -- 172.21.15.37:0/1550005643 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55440ac420 con 0x7f553c0039c0 2024-04-26T00:36:51.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.736+0000 7f553bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:51.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.736+0000 7f553bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:36:51.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.736+0000 7f553bfff640 10 monclient: dump: 2024-04-26T00:36:51.746 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:51.746 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:51.746 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:51.746 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:51.746 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:51.746 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:51.746 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:51.746 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:51.746 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:51.746 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:51.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.736+0000 7f55517fa640 4 set_mon_vals no callback set 2024-04-26T00:36:51.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.736+0000 7f5559f88640 5 monclient: authenticate success, global_id 204146 2024-04-26T00:36:51.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.736+0000 7f5559f88640 10 monclient: _renew_subs 2024-04-26T00:36:51.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.736+0000 7f5559f88640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:36:51.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.736+0000 7f5559f88640 1 -- 172.21.15.37:0/1550005643 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f555411c6c0 con 0x7f553c0039c0 2024-04-26T00:36:51.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.736+0000 7f5559f88640 10 monclient: _renew_subs 2024-04-26T00:36:51.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.736+0000 7f5559f88640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:36:51.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.736+0000 7f5559f88640 1 -- 172.21.15.37:0/1550005643 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f555412f8d0 con 0x7f553c0039c0 2024-04-26T00:36:51.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.736+0000 7f5559f88640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:51.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.736+0000 7f5559f88640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:51.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.736+0000 7f5559f88640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:36:51.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.736+0000 7f553bfff640 1 -- 172.21.15.37:0/1550005643 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f55440ac600 con 0x7f553c0039c0 2024-04-26T00:36:51.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.736+0000 7f5559f88640 1 -- 172.21.15.37:0/1550005643 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f555411b940 con 0x7f553c0039c0 2024-04-26T00:36:51.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.736+0000 7f553bfff640 1 --2- 172.21.15.37:0/1550005643 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5540035e90 0x7f5540038350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:51.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.736+0000 7f553bfff640 1 -- 172.21.15.37:0/1550005643 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(662..662 src has 370..662) ==== 3899+0+0 (secure 0 0 0) 0x7f55440e1e20 con 0x7f553c0039c0 2024-04-26T00:36:51.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.736+0000 7f5552ffd640 1 --2- 172.21.15.37:0/1550005643 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5540035e90 0x7f5540038350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:51.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.736+0000 7f5552ffd640 10 monclient: get_auth_request con 0x7f5540035e90 auth_method 0 2024-04-26T00:36:51.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.740+0000 7f5552ffd640 1 --2- 172.21.15.37:0/1550005643 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5540035e90 0x7f5540038350 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f5554062db0 tx=0x7f55480090e0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:51.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.740+0000 7f553bfff640 1 -- 172.21.15.37:0/1550005643 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5544016130 con 0x7f553c0039c0 2024-04-26T00:36:51.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.740+0000 7f553bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:51.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.740+0000 7f553bfff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:36:51.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.964+0000 7f5559f88640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:36:51.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.964+0000 7f5559f88640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:36:51.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.964+0000 7f5559f88640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:36:51.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.964+0000 7f5559f88640 1 -- 172.21.15.37:0/1550005643 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5554066f40 con 0x7f553c0039c0 2024-04-26T00:36:51.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.964+0000 7f553bfff640 1 -- 172.21.15.37:0/1550005643 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1558 (secure 0 0 0) 0x7f55440c4630 con 0x7f553c0039c0 2024-04-26T00:36:51.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.964+0000 7f553bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:36:51.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.964+0000 7f553bfff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:36:51.966 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":126,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:36:51.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.968+0000 7f5559f88640 1 -- 172.21.15.37:0/1550005643 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5540035e90 msgr2=0x7f5540038350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:51.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.968+0000 7f5559f88640 1 --2- 172.21.15.37:0/1550005643 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5540035e90 0x7f5540038350 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f5554062db0 tx=0x7f55480090e0 comp rx=0 tx=0).stop 2024-04-26T00:36:51.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.968+0000 7f5559f88640 10 monclient: shutdown 2024-04-26T00:36:51.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.968+0000 7f5559f88640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:36:51.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.968+0000 7f5559f88640 1 -- 172.21.15.37:0/1550005643 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f553c0039c0 msgr2=0x7f55541cb470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:51.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.968+0000 7f5559f88640 1 --2- 172.21.15.37:0/1550005643 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f553c0039c0 0x7f55541cb470 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5544003940 tx=0x7f553c005840 comp rx=0 tx=0).stop 2024-04-26T00:36:51.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.968+0000 7f5559f88640 1 -- 172.21.15.37:0/1550005643 shutdown_connections 2024-04-26T00:36:51.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.968+0000 7f5559f88640 1 --2- 172.21.15.37:0/1550005643 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5540035e90 0x7f5540038350 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:51.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.968+0000 7f5559f88640 1 --2- 172.21.15.37:0/1550005643 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55541cbdb0 0x7f5554071790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:51.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.968+0000 7f5559f88640 1 --2- 172.21.15.37:0/1550005643 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55541cb9b0 0x7f5554071250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:51.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.968+0000 7f5559f88640 1 --2- 172.21.15.37:0/1550005643 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f553c0039c0 0x7f55541cb470 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:51.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.968+0000 7f5559f88640 1 -- 172.21.15.37:0/1550005643 >> 172.21.15.37:0/1550005643 conn(0x7f55540a1ad0 msgr2=0x7f55540a46d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:36:51.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.968+0000 7f5559f88640 1 -- 172.21.15.37:0/1550005643 shutdown_connections 2024-04-26T00:36:51.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:51.968+0000 7f5559f88640 1 -- 172.21.15.37:0/1550005643 wait complete. 2024-04-26T00:36:54.986 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:36:55.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.120+0000 7f80bab9f640 10 monclient: get_monmap_and_config 2024-04-26T00:36:55.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.120+0000 7f80bab9f640 10 monclient: build_initial_monmap 2024-04-26T00:36:55.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.120+0000 7f80bab9f640 10 monclient: monmap: 2024-04-26T00:36:55.121 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:36:55.121 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:55.121 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:36:55.125483+0000 2024-04-26T00:36:55.121 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:36:55.125483+0000 2024-04-26T00:36:55.121 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:36:55.122 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:55.122 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:36:55.122 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:36:55.122 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:36:55.122 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:36:55.122 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:36:55.122 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:55.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.120+0000 7f80bab9f640 1 Processor -- start 2024-04-26T00:36:55.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.120+0000 7f80bab9f640 1 -- start start 2024-04-26T00:36:55.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.120+0000 7f80bab9f640 10 monclient: init 2024-04-26T00:36:55.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.120+0000 7f80bab9f640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:55.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.120+0000 7f80bab9f640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-04-26T00:36:55.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.120+0000 7f80bab9f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f80b4136e10 addr v1:172.21.15.37:6789/0 2024-04-26T00:36:55.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.120+0000 7f80bab9f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f80b4137900 addr v1:172.21.15.45:6789/0 2024-04-26T00:36:55.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.120+0000 7f80bab9f640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f80b409c5b0 0x7f80b413fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:55.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.120+0000 7f80bab9f640 10 monclient(hunting): picked mon.noname-a con 0x7f80b409c5b0 addr v2:172.21.15.37:3300/0 2024-04-26T00:36:55.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.120+0000 7f80bab9f640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80b409b680 con 0x7f80b4136e10 2024-04-26T00:36:55.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.120+0000 7f80bab9f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80b409bbd0 con 0x7f80b4137900 2024-04-26T00:36:55.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.120+0000 7f80bab9f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:55.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.120+0000 7f80bab9f640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f80b409b8f0 con 0x7f80b409c5b0 2024-04-26T00:36:55.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.120+0000 7f80bab9f640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:55.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.120+0000 7f80bab9f640 10 monclient(hunting): authenticate will time out at 1996.431236s 2024-04-26T00:36:55.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.120+0000 7f80b3fff640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f80b4137900 legacy=0x7f80b413c340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:36:55.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.120+0000 7f80b893b640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f80b4136e10 0x7f80b41371f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:47902/0 (socket says 172.21.15.37:47902) 2024-04-26T00:36:55.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.120+0000 7f80b893b640 1 -- 172.21.15.37:0/1701235230 learned_addr learned my addr 172.21.15.37:0/1701235230 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:36:55.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.120+0000 7f80b913c640 1 --2- 172.21.15.37:0/1701235230 >> v2:172.21.15.37:3300/0 conn(0x7f80b409c5b0 0x7f80b413fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:55.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.120+0000 7f80b913c640 10 monclient(hunting): get_auth_request con 0x7f80b409c5b0 auth_method 0 2024-04-26T00:36:55.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.120+0000 7f80b913c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:55.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.120+0000 7f80b913c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:55.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.120+0000 7f80b913c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:55.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.120+0000 7f80b913c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:55.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.120+0000 7f80b913c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:55.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.120+0000 7f80b913c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:55.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.120+0000 7f80b913c640 10 monclient(hunting): handle_auth_done global_id 144765 payload 274 2024-04-26T00:36:55.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.120+0000 7f80b913c640 1 -- 172.21.15.37:0/1701235230 >> v1:172.21.15.45:6789/0 conn(0x7f80b4137900 legacy=0x7f80b413c340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:36:55.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.120+0000 7f80b913c640 1 -- 172.21.15.37:0/1701235230 >> v1:172.21.15.37:6789/0 conn(0x7f80b4136e10 legacy=0x7f80b41371f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:36:55.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.120+0000 7f80b913c640 10 monclient: _finish_hunting 0 2024-04-26T00:36:55.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.120+0000 7f80b913c640 1 monclient: found mon.noname-a 2024-04-26T00:36:55.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.120+0000 7f80b913c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:55.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.120+0000 7f80b913c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:36:55.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.120+0000 7f80b913c640 1 -- 172.21.15.37:0/1701235230 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80b40a5d90 con 0x7f80b409c5b0 2024-04-26T00:36:55.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b913c640 1 -- 172.21.15.37:0/1701235230 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80b409c5b0 msgr2=0x7f80b413fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:55.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b913c640 1 --2- 172.21.15.37:0/1701235230 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80b409c5b0 0x7f80b413fa70 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f80ac009d90 tx=0x7f80ac01c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a909b788d5ced6a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:55.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b37fe640 1 -- 172.21.15.37:0/1701235230 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80ac0b2070 con 0x7f80b409c5b0 2024-04-26T00:36:55.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:55.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:36:55.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b37fe640 10 monclient: dump: 2024-04-26T00:36:55.129 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:55.129 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:55.129 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:55.129 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:55.129 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:55.129 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:55.129 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:55.129 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:55.129 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:55.130 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:55.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b37fe640 10 monclient: _finish_auth 0 2024-04-26T00:36:55.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b37fe640 10 monclient: _check_auth_tickets 2024-04-26T00:36:55.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:55.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80bab9f640 5 monclient: authenticate success, global_id 144765 2024-04-26T00:36:55.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80bab9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:36:55.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b37fe640 1 -- 172.21.15.37:0/1701235230 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80ac0a5860 con 0x7f80b409c5b0 2024-04-26T00:36:55.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b37fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:36:55.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80bab9f640 10 monclient: get_monmap_and_config success 2024-04-26T00:36:55.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80bab9f640 4 set_mon_vals no callback set 2024-04-26T00:36:55.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b37fe640 1 -- 172.21.15.37:0/1701235230 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80ac0a5c30 con 0x7f80b409c5b0 2024-04-26T00:36:55.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:55.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:36:55.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b37fe640 10 monclient: dump: 2024-04-26T00:36:55.131 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:55.131 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:55.131 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:55.131 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:55.131 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:55.131 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:55.131 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:55.131 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:55.131 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:55.131 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:55.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80bab9f640 10 monclient: shutdown 2024-04-26T00:36:55.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80bab9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:36:55.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80bab9f640 1 -- 172.21.15.37:0/1701235230 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80b409c5b0 msgr2=0x7f80b413fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:55.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80bab9f640 1 --2- 172.21.15.37:0/1701235230 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80b409c5b0 0x7f80b413fa70 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f80ac009d90 tx=0x7f80ac01c7b0 comp rx=0 tx=0).stop 2024-04-26T00:36:55.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80bab9f640 1 -- 172.21.15.37:0/1701235230 shutdown_connections 2024-04-26T00:36:55.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80bab9f640 1 --2- 172.21.15.37:0/1701235230 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80b409c5b0 0x7f80b413fa70 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:55.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80bab9f640 1 -- 172.21.15.37:0/1701235230 >> 172.21.15.37:0/1701235230 conn(0x7f80b40a64e0 msgr2=0x7f80b40a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:36:55.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80bab9f640 1 -- 172.21.15.37:0/1701235230 shutdown_connections 2024-04-26T00:36:55.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80bab9f640 1 -- 172.21.15.37:0/1701235230 wait complete. 2024-04-26T00:36:55.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80bab9f640 10 monclient: build_initial_monmap 2024-04-26T00:36:55.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80bab9f640 10 monclient: monmap: 2024-04-26T00:36:55.132 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:36:55.132 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:36:55.132 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:36:55.132 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:36:55.133 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:36:55.133 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:36:55.133 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:36:55.133 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:36:55.133 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:36:55.133 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:55.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80bab9f640 1 Processor -- start 2024-04-26T00:36:55.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80bab9f640 1 -- start start 2024-04-26T00:36:55.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80bab9f640 10 monclient: init 2024-04-26T00:36:55.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80bab9f640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:55.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80bab9f640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:36:55.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80bab9f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80b409c5b0 0x7f80b4135a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:55.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80bab9f640 10 monclient(hunting): picked mon.noname-b con 0x7f80b409c5b0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:36:55.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80bab9f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f80b4136e10 0x7f80b413b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:55.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80bab9f640 10 monclient(hunting): picked mon.noname-c con 0x7f80b4136e10 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:36:55.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80bab9f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80b4137900 0x7f80b413bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:55.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80bab9f640 10 monclient(hunting): picked mon.noname-a con 0x7f80b4137900 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:36:55.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80bab9f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:55.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80bab9f640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f80b4142b60 con 0x7f80b4137900 2024-04-26T00:36:55.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80bab9f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:55.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80bab9f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f80b41429e0 con 0x7f80b409c5b0 2024-04-26T00:36:55.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80bab9f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:55.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80bab9f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f80b4142ce0 con 0x7f80b4136e10 2024-04-26T00:36:55.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80bab9f640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:55.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80bab9f640 10 monclient(hunting): authenticate will time out at 1996.439024s 2024-04-26T00:36:55.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b913c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80b4137900 0x7f80b413bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:55.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b913c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80b4137900 0x7f80b413bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:46828/0 (socket says 172.21.15.37:46828) 2024-04-26T00:36:55.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b893b640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80b409c5b0 msgr2=0x7f80b4135a80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:36:55.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b913c640 1 -- 172.21.15.37:0/3350202330 learned_addr learned my addr 172.21.15.37:0/3350202330 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:36:55.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b913c640 10 monclient(hunting): get_auth_request con 0x7f80b4137900 auth_method 0 2024-04-26T00:36:55.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b913c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:55.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b913c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:55.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b913c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:55.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b893b640 1 --2- 172.21.15.37:0/3350202330 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80b409c5b0 0x7f80b4135a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:36:55.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b3fff640 1 --2- 172.21.15.37:0/3350202330 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f80b4136e10 0x7f80b413b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:55.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b913c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:55.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b913c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:55.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b913c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:55.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b3fff640 10 monclient(hunting): get_auth_request con 0x7f80b4136e10 auth_method 0 2024-04-26T00:36:55.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:55.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:55.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:55.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:36:55.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:36:55.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:36:55.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b913c640 10 monclient(hunting): handle_auth_done global_id 144768 payload 824 2024-04-26T00:36:55.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b913c640 1 -- 172.21.15.37:0/3350202330 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f80b4136e10 msgr2=0x7f80b413b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:55.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b913c640 1 --2- 172.21.15.37:0/3350202330 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f80b4136e10 0x7f80b413b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:55.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b913c640 1 -- 172.21.15.37:0/3350202330 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80b409c5b0 msgr2=0x7f80b4135a80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:36:55.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b913c640 1 --2- 172.21.15.37:0/3350202330 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80b409c5b0 0x7f80b4135a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:55.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b913c640 10 monclient: _finish_hunting 0 2024-04-26T00:36:55.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b913c640 1 monclient: found mon.noname-a 2024-04-26T00:36:55.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b913c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:36:55.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b913c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:36:55.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.128+0000 7f80b913c640 1 -- 172.21.15.37:0/3350202330 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80b413f890 con 0x7f80b4137900 2024-04-26T00:36:55.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.132+0000 7f80b913c640 1 -- 172.21.15.37:0/3350202330 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80b4137900 msgr2=0x7f80b413bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:36:55.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.132+0000 7f80b913c640 1 --2- 172.21.15.37:0/3350202330 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80b4137900 0x7f80b413bf10 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f80ac0b0040 tx=0x7f80ac019550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:55.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.132+0000 7f80b913c640 1 --2- 172.21.15.37:0/3350202330 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80b4137900 0x7f80b413bf10 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f80ac0b0040 tx=0x7f80ac019550 comp rx=0 tx=0).queue_received will delay after 1.71409e+09 on 0x7f80ac0bd020 mon_map magic: 0 2024-04-26T00:36:55.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.132+0000 7f80b17fa640 1 -- 172.21.15.37:0/3350202330 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80ac0bd020 con 0x7f80b4137900 2024-04-26T00:36:55.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.132+0000 7f80b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:55.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.132+0000 7f80b17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:36:55.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.132+0000 7f80b17fa640 10 monclient: dump: 2024-04-26T00:36:55.141 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:55.141 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:55.141 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:55.141 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:55.141 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:55.141 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:55.142 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:55.142 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:55.142 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:55.142 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:55.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.132+0000 7f80b17fa640 10 monclient: _finish_auth 0 2024-04-26T00:36:55.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.132+0000 7f80b17fa640 10 monclient: _check_auth_tickets 2024-04-26T00:36:55.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.132+0000 7f80b17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:36:55.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.132+0000 7f80bab9f640 5 monclient: authenticate success, global_id 144768 2024-04-26T00:36:55.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.132+0000 7f80bab9f640 10 monclient: _renew_subs 2024-04-26T00:36:55.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.132+0000 7f80b17fa640 1 -- 172.21.15.37:0/3350202330 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80ac0ad900 con 0x7f80b4137900 2024-04-26T00:36:55.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.132+0000 7f80bab9f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:55.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.132+0000 7f80bab9f640 1 -- 172.21.15.37:0/3350202330 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80b413de10 con 0x7f80b4137900 2024-04-26T00:36:55.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.132+0000 7f80b17fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:36:55.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.132+0000 7f80b2ffd640 4 set_mon_vals no callback set 2024-04-26T00:36:55.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.132+0000 7f80bab9f640 10 monclient: _renew_subs 2024-04-26T00:36:55.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.132+0000 7f80bab9f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:55.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.132+0000 7f80bab9f640 1 -- 172.21.15.37:0/3350202330 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80b413f2d0 con 0x7f80b4137900 2024-04-26T00:36:55.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.132+0000 7f80bab9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:55.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.132+0000 7f80bab9f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:55.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.132+0000 7f80bab9f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:55.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.132+0000 7f80bab9f640 1 -- 172.21.15.37:0/3350202330 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80b4075910 con 0x7f80b4137900 2024-04-26T00:36:55.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.132+0000 7f80b17fa640 1 -- 172.21.15.37:0/3350202330 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80ac0b2040 con 0x7f80b4137900 2024-04-26T00:36:55.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.132+0000 7f80b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:36:55.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.132+0000 7f80b17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:36:55.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.132+0000 7f80b17fa640 10 monclient: dump: 2024-04-26T00:36:55.143 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:36:55.143 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:55.143 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:55.144 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:36:55.144 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:36:55.144 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:55.144 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:36:55.144 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:36:55.144 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:36:55.144 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:55.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.132+0000 7f80b17fa640 1 -- 172.21.15.37:0/3350202330 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f80ac0c0020 con 0x7f80b4137900 2024-04-26T00:36:55.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.132+0000 7f80b17fa640 1 --2- 172.21.15.37:0/3350202330 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8094035e30 0x7f80940382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:55.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.132+0000 7f80b893b640 1 --2- 172.21.15.37:0/3350202330 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8094035e30 0x7f80940382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:55.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.136+0000 7f80b893b640 10 monclient: get_auth_request con 0x7f8094035e30 auth_method 0 2024-04-26T00:36:55.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.136+0000 7f80b893b640 1 --2- 172.21.15.37:0/3350202330 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8094035e30 0x7f80940382f0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f809c004820 tx=0x7f809c007b30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:36:55.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.136+0000 7f80b17fa640 1 -- 172.21.15.37:0/3350202330 DispatchQueue::entry inject delay of 0.002000 2024-04-26T00:36:55.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.140+0000 7f80b17fa640 1 -- 172.21.15.37:0/3350202330 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(662..662 src has 370..662) ==== 3899+0+0 (secure 0 0 0) 0x7f80ac0e3a70 con 0x7f80b4137900 2024-04-26T00:36:55.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.756+0000 7f80b17fa640 1 -- 172.21.15.37:0/3350202330 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f80ac0c7030 con 0x7f80b4137900 2024-04-26T00:36:55.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.756+0000 7f80b17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:36:55.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.756+0000 7f80b17fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:36:55.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.992+0000 7f80bab9f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:36:55.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.992+0000 7f80bab9f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:36:55.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.992+0000 7f80bab9f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:36:55.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.992+0000 7f80bab9f640 1 -- 172.21.15.37:0/3350202330 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f80b4075910 con 0x7f80b4137900 2024-04-26T00:36:55.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.992+0000 7f80b17fa640 1 -- 172.21.15.37:0/3350202330 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1558 (secure 0 0 0) 0x7f80ac0164c0 con 0x7f80b4137900 2024-04-26T00:36:55.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.992+0000 7f80b17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:36:55.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.992+0000 7f80b17fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:36:55.992 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":126,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:36:55.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.992+0000 7f80bab9f640 1 -- 172.21.15.37:0/3350202330 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8094035e30 msgr2=0x7f80940382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:55.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.992+0000 7f80bab9f640 1 --2- 172.21.15.37:0/3350202330 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8094035e30 0x7f80940382f0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f809c004820 tx=0x7f809c007b30 comp rx=0 tx=0).stop 2024-04-26T00:36:55.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.992+0000 7f80bab9f640 10 monclient: shutdown 2024-04-26T00:36:55.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.992+0000 7f80bab9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:36:55.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.992+0000 7f80bab9f640 1 -- 172.21.15.37:0/3350202330 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80b4137900 msgr2=0x7f80b413bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:36:55.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.992+0000 7f80bab9f640 1 --2- 172.21.15.37:0/3350202330 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80b4137900 0x7f80b413bf10 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f80ac0b0040 tx=0x7f80ac019550 comp rx=0 tx=0).stop 2024-04-26T00:36:55.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.992+0000 7f80bab9f640 1 -- 172.21.15.37:0/3350202330 shutdown_connections 2024-04-26T00:36:55.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.992+0000 7f80bab9f640 1 --2- 172.21.15.37:0/3350202330 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8094035e30 0x7f80940382f0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:55.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.992+0000 7f80bab9f640 1 --2- 172.21.15.37:0/3350202330 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80b4137900 0x7f80b413bf10 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:55.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.992+0000 7f80bab9f640 1 --2- 172.21.15.37:0/3350202330 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f80b4136e10 0x7f80b413b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:55.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.992+0000 7f80bab9f640 1 --2- 172.21.15.37:0/3350202330 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80b409c5b0 0x7f80b4135a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:36:55.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.992+0000 7f80bab9f640 1 -- 172.21.15.37:0/3350202330 >> 172.21.15.37:0/3350202330 conn(0x7f80b40a64e0 msgr2=0x7f80b40a1570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:36:55.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.992+0000 7f80bab9f640 1 -- 172.21.15.37:0/3350202330 shutdown_connections 2024-04-26T00:36:55.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:55.992+0000 7f80bab9f640 1 -- 172.21.15.37:0/3350202330 wait complete. 2024-04-26T00:36:59.013 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:36:59.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd04bccf640 10 monclient: get_monmap_and_config 2024-04-26T00:36:59.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd04bccf640 10 monclient: build_initial_monmap 2024-04-26T00:36:59.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd04bccf640 10 monclient: monmap: 2024-04-26T00:36:59.148 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:36:59.148 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:36:59.148 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:36:59.152912+0000 2024-04-26T00:36:59.148 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:36:59.152912+0000 2024-04-26T00:36:59.148 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:36:59.148 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:36:59.148 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:36:59.148 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:36:59.149 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:36:59.149 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:36:59.149 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:36:59.149 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:36:59.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd04bccf640 1 Processor -- start 2024-04-26T00:36:59.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd04bccf640 1 -- start start 2024-04-26T00:36:59.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd04bccf640 10 monclient: init 2024-04-26T00:36:59.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd04bccf640 10 monclient: _reopen_session rank -1 2024-04-26T00:36:59.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd04bccf640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-04-26T00:36:59.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd04bccf640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd0440a0260 0x7fd0440a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:59.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd04bccf640 10 monclient(hunting): picked mon.noname-a con 0x7fd0440a0260 addr v2:172.21.15.37:3300/0 2024-04-26T00:36:59.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd04bccf640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd044134b30 0x7fd044134f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:59.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd04bccf640 10 monclient(hunting): picked mon.noname-b con 0x7fd044134b30 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:36:59.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd04bccf640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd044135490 0x7fd04413be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:36:59.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd04bccf640 10 monclient(hunting): picked mon.noname-c con 0x7fd044135490 addr v2:172.21.15.45:3300/0 2024-04-26T00:36:59.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd04bccf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:59.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd04bccf640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fd04409ec10 con 0x7fd0440a0260 2024-04-26T00:36:59.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd04bccf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:59.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd04bccf640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fd04409f0f0 con 0x7fd044135490 2024-04-26T00:36:59.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd04bccf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:36:59.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd04bccf640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd0440a1130 con 0x7fd044134b30 2024-04-26T00:36:59.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd04bccf640 10 monclient(hunting): _renew_subs 2024-04-26T00:36:59.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd04bccf640 10 monclient(hunting): authenticate will time out at 2000.458569s 2024-04-26T00:36:59.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd04926a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd044134b30 0x7fd044134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:59.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd049a6b640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd0440a0260 0x7fd0440a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:59.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd04a26c640 1 -- >> v2:172.21.15.45:3300/0 conn(0x7fd044135490 msgr2=0x7fd04413be60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:36:59.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd04a26c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd044135490 0x7fd04413be60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:36:59.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd049a6b640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd0440a0260 0x7fd0440a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:46838/0 (socket says 172.21.15.37:46838) 2024-04-26T00:36:59.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd049a6b640 1 -- 172.21.15.37:0/3819765401 learned_addr learned my addr 172.21.15.37:0/3819765401 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:36:59.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd049a6b640 10 monclient(hunting): get_auth_request con 0x7fd0440a0260 auth_method 0 2024-04-26T00:36:59.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd049a6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:59.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd049a6b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:59.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd049a6b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:59.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd04926a640 10 monclient(hunting): get_auth_request con 0x7fd044134b30 auth_method 0 2024-04-26T00:36:59.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd04926a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:59.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd04926a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:59.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd04926a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:36:59.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd049a6b640 1 -- 172.21.15.37:0/3819765401 >> v2:172.21.15.37:3300/0 conn(0x7fd0440a0260 msgr2=0x7fd0440a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:36:59.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd049a6b640 1 -- 172.21.15.37:0/3819765401 >> v2:172.21.15.37:3300/0 conn(0x7fd0440a0260 msgr2=0x7fd0440a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:36:59.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd049a6b640 1 --2- 172.21.15.37:0/3819765401 >> v2:172.21.15.37:3300/0 conn(0x7fd0440a0260 0x7fd0440a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:36:59.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd04926a640 1 -- 172.21.15.37:0/3819765401 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd044134b30 msgr2=0x7fd044134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:36:59.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd04926a640 1 -- 172.21.15.37:0/3819765401 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd044134b30 msgr2=0x7fd044134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:36:59.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd04926a640 1 --2- 172.21.15.37:0/3819765401 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd044134b30 0x7fd044134f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:36:59.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd049a6b640 1 --2- 172.21.15.37:0/3819765401 >> v2:172.21.15.37:3300/0 conn(0x7fd0440a0260 0x7fd0440a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:36:59.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.148+0000 7fd04926a640 1 --2- 172.21.15.37:0/3819765401 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd044134b30 0x7fd044134f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:36:59.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.352+0000 7fd04a26c640 1 -- 172.21.15.37:0/3819765401 >> v2:172.21.15.45:3300/0 conn(0x7fd044135490 msgr2=0x7fd04413be60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:36:59.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.352+0000 7fd04926a640 1 --2- 172.21.15.37:0/3819765401 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd044134b30 0x7fd044134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:59.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.352+0000 7fd049a6b640 1 --2- 172.21.15.37:0/3819765401 >> v2:172.21.15.37:3300/0 conn(0x7fd0440a0260 0x7fd0440a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:59.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.352+0000 7fd04a26c640 1 --2- 172.21.15.37:0/3819765401 >> v2:172.21.15.45:3300/0 conn(0x7fd044135490 0x7fd04413be60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:36:59.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.352+0000 7fd04926a640 10 monclient(hunting): get_auth_request con 0x7fd044134b30 auth_method 0 2024-04-26T00:36:59.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.352+0000 7fd04926a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:59.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.352+0000 7fd04926a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:59.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.352+0000 7fd04926a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:36:59.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.352+0000 7fd049a6b640 10 monclient(hunting): get_auth_request con 0x7fd0440a0260 auth_method 0 2024-04-26T00:36:59.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.352+0000 7fd049a6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:59.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.352+0000 7fd049a6b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:59.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.352+0000 7fd049a6b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:36:59.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.352+0000 7fd04926a640 1 -- 172.21.15.37:0/3819765401 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd044134b30 msgr2=0x7fd044134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:36:59.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.352+0000 7fd04926a640 1 -- 172.21.15.37:0/3819765401 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd044134b30 msgr2=0x7fd044134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:36:59.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.352+0000 7fd049a6b640 1 -- 172.21.15.37:0/3819765401 >> v2:172.21.15.37:3300/0 conn(0x7fd0440a0260 msgr2=0x7fd0440a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:36:59.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.352+0000 7fd04926a640 1 --2- 172.21.15.37:0/3819765401 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd044134b30 0x7fd044134f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:36:59.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.352+0000 7fd049a6b640 1 -- 172.21.15.37:0/3819765401 >> v2:172.21.15.37:3300/0 conn(0x7fd0440a0260 msgr2=0x7fd0440a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:36:59.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.352+0000 7fd049a6b640 1 --2- 172.21.15.37:0/3819765401 >> v2:172.21.15.37:3300/0 conn(0x7fd0440a0260 0x7fd0440a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:36:59.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.352+0000 7fd049a6b640 1 --2- 172.21.15.37:0/3819765401 >> v2:172.21.15.37:3300/0 conn(0x7fd0440a0260 0x7fd0440a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:36:59.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.352+0000 7fd04926a640 1 --2- 172.21.15.37:0/3819765401 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd044134b30 0x7fd044134f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:36:59.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.756+0000 7fd04a26c640 1 -- 172.21.15.37:0/3819765401 >> v2:172.21.15.45:3300/0 conn(0x7fd044135490 msgr2=0x7fd04413be60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:36:59.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.756+0000 7fd04a26c640 1 --2- 172.21.15.37:0/3819765401 >> v2:172.21.15.45:3300/0 conn(0x7fd044135490 0x7fd04413be60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:36:59.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.756+0000 7fd049a6b640 1 --2- 172.21.15.37:0/3819765401 >> v2:172.21.15.37:3300/0 conn(0x7fd0440a0260 0x7fd0440a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:59.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.756+0000 7fd04926a640 1 --2- 172.21.15.37:0/3819765401 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd044134b30 0x7fd044134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:36:59.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.756+0000 7fd04926a640 10 monclient(hunting): get_auth_request con 0x7fd044134b30 auth_method 0 2024-04-26T00:36:59.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.756+0000 7fd04926a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:59.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.756+0000 7fd04926a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:59.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.756+0000 7fd04926a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:36:59.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.756+0000 7fd049a6b640 10 monclient(hunting): get_auth_request con 0x7fd0440a0260 auth_method 0 2024-04-26T00:36:59.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.756+0000 7fd049a6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:36:59.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.756+0000 7fd049a6b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:36:59.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.756+0000 7fd049a6b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:36:59.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.756+0000 7fd04926a640 1 -- 172.21.15.37:0/3819765401 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd044134b30 msgr2=0x7fd044134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:36:59.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.756+0000 7fd04926a640 1 -- 172.21.15.37:0/3819765401 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd044134b30 msgr2=0x7fd044134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:36:59.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.756+0000 7fd04926a640 1 --2- 172.21.15.37:0/3819765401 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd044134b30 0x7fd044134f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:36:59.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.756+0000 7fd049a6b640 1 -- 172.21.15.37:0/3819765401 >> v2:172.21.15.37:3300/0 conn(0x7fd0440a0260 msgr2=0x7fd0440a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:36:59.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.756+0000 7fd049a6b640 1 -- 172.21.15.37:0/3819765401 >> v2:172.21.15.37:3300/0 conn(0x7fd0440a0260 msgr2=0x7fd0440a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:36:59.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.756+0000 7fd049a6b640 1 --2- 172.21.15.37:0/3819765401 >> v2:172.21.15.37:3300/0 conn(0x7fd0440a0260 0x7fd0440a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:36:59.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.756+0000 7fd04926a640 1 --2- 172.21.15.37:0/3819765401 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd044134b30 0x7fd044134f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:36:59.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:36:59.756+0000 7fd049a6b640 1 --2- 172.21.15.37:0/3819765401 >> v2:172.21.15.37:3300/0 conn(0x7fd0440a0260 0x7fd0440a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:37:00.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:00.560+0000 7fd04a26c640 1 -- 172.21.15.37:0/3819765401 >> v2:172.21.15.45:3300/0 conn(0x7fd044135490 msgr2=0x7fd04413be60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:37:00.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:00.560+0000 7fd04a26c640 1 --2- 172.21.15.37:0/3819765401 >> v2:172.21.15.45:3300/0 conn(0x7fd044135490 0x7fd04413be60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:37:00.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:00.560+0000 7fd04926a640 1 --2- 172.21.15.37:0/3819765401 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd044134b30 0x7fd044134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:00.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:00.560+0000 7fd049a6b640 1 --2- 172.21.15.37:0/3819765401 >> v2:172.21.15.37:3300/0 conn(0x7fd0440a0260 0x7fd0440a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:00.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:00.560+0000 7fd04926a640 10 monclient(hunting): get_auth_request con 0x7fd044134b30 auth_method 0 2024-04-26T00:37:00.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:00.560+0000 7fd04926a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:00.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:00.560+0000 7fd04926a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:00.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:00.560+0000 7fd04926a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:37:00.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:00.560+0000 7fd049a6b640 10 monclient(hunting): get_auth_request con 0x7fd0440a0260 auth_method 0 2024-04-26T00:37:00.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:00.560+0000 7fd049a6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:00.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:00.560+0000 7fd049a6b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:00.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:00.560+0000 7fd049a6b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:37:00.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:00.560+0000 7fd04926a640 1 -- 172.21.15.37:0/3819765401 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd044134b30 msgr2=0x7fd044134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:37:00.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:00.560+0000 7fd04926a640 1 -- 172.21.15.37:0/3819765401 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd044134b30 msgr2=0x7fd044134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:37:00.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:00.560+0000 7fd04926a640 1 --2- 172.21.15.37:0/3819765401 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd044134b30 0x7fd044134f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:37:00.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:00.560+0000 7fd049a6b640 1 -- 172.21.15.37:0/3819765401 >> v2:172.21.15.37:3300/0 conn(0x7fd0440a0260 msgr2=0x7fd0440a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:37:00.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:00.560+0000 7fd049a6b640 1 -- 172.21.15.37:0/3819765401 >> v2:172.21.15.37:3300/0 conn(0x7fd0440a0260 msgr2=0x7fd0440a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:37:00.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:00.560+0000 7fd049a6b640 1 --2- 172.21.15.37:0/3819765401 >> v2:172.21.15.37:3300/0 conn(0x7fd0440a0260 0x7fd0440a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:37:00.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:00.560+0000 7fd04926a640 1 --2- 172.21.15.37:0/3819765401 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd044134b30 0x7fd044134f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:37:00.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:00.560+0000 7fd049a6b640 1 --2- 172.21.15.37:0/3819765401 >> v2:172.21.15.37:3300/0 conn(0x7fd0440a0260 0x7fd0440a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:37:02.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.148+0000 7fd03b7fe640 10 monclient(hunting): tick 2024-04-26T00:37:02.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.148+0000 7fd03b7fe640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:37:02.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.148+0000 7fd03b7fe640 1 monclient(hunting): continuing hunt 2024-04-26T00:37:02.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.148+0000 7fd03b7fe640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:37:02.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.148+0000 7fd03b7fe640 1 -- 172.21.15.37:0/3819765401 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd044134b30 msgr2=0x7fd044134f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:37:02.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.148+0000 7fd03b7fe640 1 --2- 172.21.15.37:0/3819765401 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd044134b30 0x7fd044134f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:02.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.148+0000 7fd03b7fe640 1 -- 172.21.15.37:0/3819765401 >> v2:172.21.15.45:3300/0 conn(0x7fd044135490 msgr2=0x7fd04413be60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:37:02.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.148+0000 7fd03b7fe640 1 --2- 172.21.15.37:0/3819765401 >> v2:172.21.15.45:3300/0 conn(0x7fd044135490 0x7fd04413be60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:02.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.148+0000 7fd03b7fe640 1 -- 172.21.15.37:0/3819765401 >> v2:172.21.15.37:3300/0 conn(0x7fd0440a0260 msgr2=0x7fd0440a0640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:37:02.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.148+0000 7fd03b7fe640 1 --2- 172.21.15.37:0/3819765401 >> v2:172.21.15.37:3300/0 conn(0x7fd0440a0260 0x7fd0440a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:02.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.148+0000 7fd03b7fe640 10 monclient: _add_conns ranks=[1,0] 2024-04-26T00:37:02.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.148+0000 7fd03b7fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd0280036f0 addr v1:172.21.15.45:6789/0 2024-04-26T00:37:02.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.148+0000 7fd03b7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd028007400 addr v1:172.21.15.37:6789/0 2024-04-26T00:37:02.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.148+0000 7fd03b7fe640 1 -- 172.21.15.37:0/3819765401 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd04406cff0 con 0x7fd028007400 2024-04-26T00:37:02.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.148+0000 7fd03b7fe640 1 -- 172.21.15.37:0/3819765401 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd02800c140 con 0x7fd0280036f0 2024-04-26T00:37:02.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.148+0000 7fd03b7fe640 10 monclient(hunting): _renew_subs 2024-04-26T00:37:02.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.148+0000 7fd049a6b640 1 -- 172.21.15.37:0/3819765401 >> v1:172.21.15.45:6789/0 conn(0x7fd0280036f0 legacy=0x7fd028005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:37:02.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.152+0000 7fd04926a640 1 -- 172.21.15.37:0/3819765401 >> v1:172.21.15.37:6789/0 conn(0x7fd028007400 legacy=0x7fd0280098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:02.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.192+0000 7fd048a69640 1 -- 172.21.15.37:0/3819765401 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1213994859 0 0) 0x7fd04406cff0 con 0x7fd028007400 2024-04-26T00:37:02.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.192+0000 7fd048a69640 10 monclient(hunting): _negotiate 2024-04-26T00:37:02.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.192+0000 7fd048a69640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:02.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.192+0000 7fd048a69640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:02.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.192+0000 7fd048a69640 10 monclient(hunting): my global_id is 144771 2024-04-26T00:37:02.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.192+0000 7fd048a69640 1 -- 172.21.15.37:0/3819765401 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd030003530 con 0x7fd028007400 2024-04-26T00:37:02.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.196+0000 7fd048a69640 1 -- 172.21.15.37:0/3819765401 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4090156785 0 0) 0x7fd030003530 con 0x7fd028007400 2024-04-26T00:37:02.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.196+0000 7fd048a69640 1 -- 172.21.15.37:0/3819765401 >> v1:172.21.15.45:6789/0 conn(0x7fd0280036f0 legacy=0x7fd028005b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:37:02.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.196+0000 7fd048a69640 10 monclient: _finish_hunting 0 2024-04-26T00:37:02.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.196+0000 7fd048a69640 1 monclient: found mon.noname-a-legacy 2024-04-26T00:37:02.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.196+0000 7fd048a69640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:37:02.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.196+0000 7fd048a69640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.37:6789/0 2024-04-26T00:37:02.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.196+0000 7fd048a69640 1 -- 172.21.15.37:0/3819765401 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd02800d380 con 0x7fd028007400 2024-04-26T00:37:02.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.196+0000 7fd048a69640 10 monclient: _finish_auth 0 2024-04-26T00:37:02.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.196+0000 7fd048a69640 10 monclient: _check_auth_tickets 2024-04-26T00:37:02.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.196+0000 7fd048a69640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:37:02.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.196+0000 7fd048a69640 1 -- 172.21.15.37:0/3819765401 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 866326386 0 0) 0x7fd03400f040 con 0x7fd028007400 2024-04-26T00:37:02.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.196+0000 7fd04bccf640 5 monclient: authenticate success, global_id 144771 2024-04-26T00:37:02.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.196+0000 7fd04bccf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:37:02.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.196+0000 7fd048a69640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:02.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.196+0000 7fd048a69640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-26T00:37:02.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.196+0000 7fd048a69640 10 monclient: dump: 2024-04-26T00:37:02.201 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:02.202 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:02.202 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:02.202 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:02.202 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:02.202 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:02.202 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:02.202 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:02.202 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:02.202 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:02.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.196+0000 7fd048a69640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] but i'm connected to v1:172.21.15.37:6789/0, reconnecting 2024-04-26T00:37:02.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.196+0000 7fd048a69640 10 monclient: _reopen_session rank -1 2024-04-26T00:37:02.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.196+0000 7fd048a69640 1 -- 172.21.15.37:0/3819765401 >> v1:172.21.15.37:6789/0 conn(0x7fd028007400 legacy=0x7fd0280098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:02.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.196+0000 7fd04a26c640 1 -- 172.21.15.37:0/3819765401 reap_dead start 2024-04-26T00:37:02.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.196+0000 7fd048a69640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:37:02.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.196+0000 7fd048a69640 1 --2- 172.21.15.37:0/3819765401 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0300039f0 0x7fd030003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:02.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.196+0000 7fd048a69640 10 monclient(hunting): picked mon.c con 0x7fd0300039f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:37:02.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.196+0000 7fd048a69640 1 --2- 172.21.15.37:0/3819765401 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd030004810 0x7fd030004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:02.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.196+0000 7fd048a69640 10 monclient(hunting): picked mon.a con 0x7fd030004810 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:37:02.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.196+0000 7fd048a69640 1 --2- 172.21.15.37:0/3819765401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0300052f0 0x7fd03000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:02.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd048a69640 10 monclient(hunting): picked mon.b con 0x7fd0300052f0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:37:02.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd048a69640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:02.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd048a69640 1 -- 172.21.15.37:0/3819765401 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd03000c1e0 con 0x7fd030004810 2024-04-26T00:37:02.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd048a69640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:02.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd048a69640 1 -- 172.21.15.37:0/3819765401 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd03000c390 con 0x7fd0300052f0 2024-04-26T00:37:02.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd048a69640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:02.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd048a69640 1 -- 172.21.15.37:0/3819765401 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd03000c590 con 0x7fd0300039f0 2024-04-26T00:37:02.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd048a69640 10 monclient(hunting): _renew_subs 2024-04-26T00:37:02.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd048a69640 10 monclient(hunting): _finish_auth 0 2024-04-26T00:37:02.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd04bccf640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-26T00:37:02.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd04a26c640 1 -- 172.21.15.37:0/3819765401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0300052f0 msgr2=0x7fd03000b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:37:02.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd04926a640 1 --2- 172.21.15.37:0/3819765401 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd030004810 0x7fd030004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:02.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd049a6b640 1 --2- 172.21.15.37:0/3819765401 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0300039f0 0x7fd030003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:02.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd04a26c640 1 --2- 172.21.15.37:0/3819765401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0300052f0 0x7fd03000b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:37:02.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd04926a640 10 monclient(hunting): get_auth_request con 0x7fd030004810 auth_method 0 2024-04-26T00:37:02.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd04926a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:02.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd04926a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:02.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd04926a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:37:02.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd049a6b640 10 monclient(hunting): get_auth_request con 0x7fd0300039f0 auth_method 0 2024-04-26T00:37:02.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd049a6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:02.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd049a6b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:02.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd049a6b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:37:02.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd049a6b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:02.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd049a6b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:02.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd049a6b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:37:02.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd04926a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:02.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd04926a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:02.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd04926a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:37:02.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd049a6b640 10 monclient(hunting): handle_auth_done global_id 144771 payload 293 2024-04-26T00:37:02.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd049a6b640 1 -- 172.21.15.37:0/3819765401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0300052f0 msgr2=0x7fd03000b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:37:02.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd049a6b640 1 --2- 172.21.15.37:0/3819765401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0300052f0 0x7fd03000b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:02.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd049a6b640 1 -- 172.21.15.37:0/3819765401 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd030004810 msgr2=0x7fd030004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:37:02.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd049a6b640 1 --2- 172.21.15.37:0/3819765401 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd030004810 0x7fd030004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:02.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd049a6b640 10 monclient: _finish_hunting 0 2024-04-26T00:37:02.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd049a6b640 1 monclient: found mon.c 2024-04-26T00:37:02.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd049a6b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:37:02.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd049a6b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:37:02.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd049a6b640 1 -- 172.21.15.37:0/3819765401 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd030003530 con 0x7fd0300039f0 2024-04-26T00:37:02.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd049a6b640 10 monclient: _finish_auth 0 2024-04-26T00:37:02.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd049a6b640 10 monclient: _check_auth_tickets 2024-04-26T00:37:02.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd049a6b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:37:02.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.200+0000 7fd04926a640 1 --2- 172.21.15.37:0/3819765401 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd030004810 0x7fd030004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:37:02.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.208+0000 7fd049a6b640 1 -- 172.21.15.37:0/3819765401 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0300039f0 msgr2=0x7fd030003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:02.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.208+0000 7fd049a6b640 1 --2- 172.21.15.37:0/3819765401 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0300039f0 0x7fd030003dd0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd040003750 tx=0x7fd04001d0f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd04d7989dbd72e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:02.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.208+0000 7fd048a69640 1 -- 172.21.15.37:0/3819765401 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0400ab070 con 0x7fd0300039f0 2024-04-26T00:37:02.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.208+0000 7fd048a69640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:02.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.208+0000 7fd048a69640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:37:02.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.208+0000 7fd048a69640 10 monclient: dump: 2024-04-26T00:37:02.208 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:02.209 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:02.209 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:02.209 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:02.209 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:02.209 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:02.209 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:02.209 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:02.209 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:02.209 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:02.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.208+0000 7fd048a69640 1 -- 172.21.15.37:0/3819765401 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0400a5d60 con 0x7fd0300039f0 2024-04-26T00:37:02.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.208+0000 7fd048a69640 10 monclient: handle_config config(0 keys) 2024-04-26T00:37:02.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.208+0000 7fd048a69640 1 -- 172.21.15.37:0/3819765401 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd04000f040 con 0x7fd0300039f0 2024-04-26T00:37:02.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.208+0000 7fd04bccf640 10 monclient: get_monmap_and_config success 2024-04-26T00:37:02.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.208+0000 7fd04bccf640 4 set_mon_vals no callback set 2024-04-26T00:37:02.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.208+0000 7fd048a69640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:02.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.208+0000 7fd048a69640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:37:02.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.208+0000 7fd048a69640 10 monclient: dump: 2024-04-26T00:37:02.210 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:02.210 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:02.210 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:02.210 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:02.210 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:02.210 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:02.210 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:02.210 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:02.210 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:02.211 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:02.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.208+0000 7fd04bccf640 10 monclient: shutdown 2024-04-26T00:37:02.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.208+0000 7fd04bccf640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:37:02.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.208+0000 7fd04bccf640 1 -- 172.21.15.37:0/3819765401 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0300039f0 msgr2=0x7fd030003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:02.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.208+0000 7fd04bccf640 1 --2- 172.21.15.37:0/3819765401 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0300039f0 0x7fd030003dd0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd040003750 tx=0x7fd04001d0f0 comp rx=0 tx=0).stop 2024-04-26T00:37:02.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.208+0000 7fd04bccf640 1 -- 172.21.15.37:0/3819765401 shutdown_connections 2024-04-26T00:37:02.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.208+0000 7fd04bccf640 1 --2- 172.21.15.37:0/3819765401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0300052f0 0x7fd03000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:02.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.208+0000 7fd04bccf640 1 --2- 172.21.15.37:0/3819765401 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd030004810 0x7fd030004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:02.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.208+0000 7fd04bccf640 1 --2- 172.21.15.37:0/3819765401 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0300039f0 0x7fd030003dd0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:02.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.208+0000 7fd04bccf640 1 -- 172.21.15.37:0/3819765401 >> 172.21.15.37:0/3819765401 conn(0x7fd0440a1ad0 msgr2=0x7fd0440a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:37:02.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.208+0000 7fd04bccf640 1 -- 172.21.15.37:0/3819765401 shutdown_connections 2024-04-26T00:37:02.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.208+0000 7fd04bccf640 1 -- 172.21.15.37:0/3819765401 wait complete. 2024-04-26T00:37:02.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04bccf640 10 monclient: build_initial_monmap 2024-04-26T00:37:02.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04bccf640 10 monclient: monmap: 2024-04-26T00:37:02.213 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:37:02.213 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:37:02.213 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:37:02.213 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:37:02.213 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:37:02.213 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:37:02.213 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:37:02.213 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:37:02.213 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:37:02.213 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:02.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04bccf640 1 Processor -- start 2024-04-26T00:37:02.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04bccf640 1 -- start start 2024-04-26T00:37:02.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04bccf640 10 monclient: init 2024-04-26T00:37:02.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04bccf640 10 monclient: _reopen_session rank -1 2024-04-26T00:37:02.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04bccf640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:37:02.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04bccf640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0300039f0 0x7fd044131e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:02.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04bccf640 10 monclient(hunting): picked mon.noname-b con 0x7fd0300039f0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:37:02.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04bccf640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd030004810 0x7fd0441ce7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:02.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04bccf640 10 monclient(hunting): picked mon.noname-c con 0x7fd030004810 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:37:02.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04bccf640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd0300052f0 0x7fd0441cb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:02.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04bccf640 10 monclient(hunting): picked mon.noname-a con 0x7fd0300052f0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:37:02.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04bccf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:02.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04bccf640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd04409fe90 con 0x7fd0300052f0 2024-04-26T00:37:02.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04bccf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:02.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04bccf640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd04409f4c0 con 0x7fd0300039f0 2024-04-26T00:37:02.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04bccf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:02.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04bccf640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd0440a0010 con 0x7fd030004810 2024-04-26T00:37:02.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04bccf640 10 monclient(hunting): _renew_subs 2024-04-26T00:37:02.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04bccf640 10 monclient(hunting): authenticate will time out at 2003.523174s 2024-04-26T00:37:02.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd049a6b640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0300039f0 msgr2=0x7fd044131e40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:37:02.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04a26c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd0300052f0 0x7fd0441cb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:02.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04926a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd030004810 0x7fd0441ce7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:02.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd049a6b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0300039f0 0x7fd044131e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:37:02.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04a26c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd0300052f0 0x7fd0441cb2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:47292/0 (socket says 172.21.15.37:47292) 2024-04-26T00:37:02.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04a26c640 1 -- 172.21.15.37:0/1565689459 learned_addr learned my addr 172.21.15.37:0/1565689459 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:37:02.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04a26c640 10 monclient(hunting): get_auth_request con 0x7fd0300052f0 auth_method 0 2024-04-26T00:37:02.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04a26c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:02.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04a26c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:02.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04a26c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:02.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04926a640 10 monclient(hunting): get_auth_request con 0x7fd030004810 auth_method 0 2024-04-26T00:37:02.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04926a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:02.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04926a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:02.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04926a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:02.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04a26c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:02.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04a26c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:02.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04a26c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:02.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04926a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:02.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04926a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:02.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04926a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:02.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04926a640 10 monclient(hunting): handle_auth_done global_id 204152 payload 824 2024-04-26T00:37:02.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04926a640 1 -- 172.21.15.37:0/1565689459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0300039f0 msgr2=0x7fd044131e40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:37:02.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04926a640 1 --2- 172.21.15.37:0/1565689459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0300039f0 0x7fd044131e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:02.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04926a640 1 -- 172.21.15.37:0/1565689459 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd0300052f0 msgr2=0x7fd0441cb2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:02.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04926a640 1 --2- 172.21.15.37:0/1565689459 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd0300052f0 0x7fd0441cb2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:02.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04926a640 10 monclient: _finish_hunting 0 2024-04-26T00:37:02.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04926a640 1 monclient: found mon.noname-c 2024-04-26T00:37:02.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04926a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:37:02.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04926a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:37:02.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04926a640 1 -- 172.21.15.37:0/1565689459 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0440729d0 con 0x7fd030004810 2024-04-26T00:37:02.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.212+0000 7fd04a26c640 1 --2- 172.21.15.37:0/1565689459 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd0300052f0 0x7fd0441cb2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:37:02.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.216+0000 7fd04926a640 1 -- 172.21.15.37:0/1565689459 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd030004810 msgr2=0x7fd0441ce7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:02.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.216+0000 7fd04926a640 1 --2- 172.21.15.37:0/1565689459 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd030004810 0x7fd0441ce7a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd03400e8a0 tx=0x7fd0340146c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:02.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.216+0000 7fd03a7fc640 1 -- 172.21.15.37:0/1565689459 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03401b070 con 0x7fd030004810 2024-04-26T00:37:02.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.216+0000 7fd03a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:02.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.216+0000 7fd03a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:37:02.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.216+0000 7fd03a7fc640 10 monclient: dump: 2024-04-26T00:37:02.222 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:02.222 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:02.222 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:02.223 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:02.223 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:02.223 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:02.223 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:02.223 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:02.223 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:02.223 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:02.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.216+0000 7fd03a7fc640 10 monclient: _finish_auth 0 2024-04-26T00:37:02.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.216+0000 7fd03a7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:37:02.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.216+0000 7fd03a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:37:02.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.216+0000 7fd03a7fc640 1 -- 172.21.15.37:0/1565689459 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd034004440 con 0x7fd030004810 2024-04-26T00:37:02.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.216+0000 7fd03a7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:37:02.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.216+0000 7fd03a7fc640 1 -- 172.21.15.37:0/1565689459 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd034003820 con 0x7fd030004810 2024-04-26T00:37:02.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.216+0000 7fd04bccf640 5 monclient: authenticate success, global_id 204152 2024-04-26T00:37:02.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.216+0000 7fd03a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:02.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.216+0000 7fd03a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:37:02.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.216+0000 7fd03a7fc640 10 monclient: dump: 2024-04-26T00:37:02.224 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:02.224 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:02.224 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:02.224 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:02.224 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:02.224 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:02.224 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:02.225 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:02.225 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:02.225 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:02.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.216+0000 7fd04bccf640 10 monclient: _renew_subs 2024-04-26T00:37:02.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.216+0000 7fd03bfff640 4 set_mon_vals no callback set 2024-04-26T00:37:02.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.216+0000 7fd04bccf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:37:02.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.216+0000 7fd04bccf640 1 -- 172.21.15.37:0/1565689459 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd044121820 con 0x7fd030004810 2024-04-26T00:37:02.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.216+0000 7fd04bccf640 10 monclient: _renew_subs 2024-04-26T00:37:02.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.216+0000 7fd04bccf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:37:02.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.216+0000 7fd04bccf640 1 -- 172.21.15.37:0/1565689459 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd044121540 con 0x7fd030004810 2024-04-26T00:37:02.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.216+0000 7fd03a7fc640 1 -- 172.21.15.37:0/1565689459 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd03401e070 con 0x7fd030004810 2024-04-26T00:37:02.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.216+0000 7fd04bccf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:02.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.216+0000 7fd04bccf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:02.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.216+0000 7fd04bccf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:37:02.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.216+0000 7fd04bccf640 1 -- 172.21.15.37:0/1565689459 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd044066f40 con 0x7fd030004810 2024-04-26T00:37:02.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.216+0000 7fd03a7fc640 1 --2- 172.21.15.37:0/1565689459 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd0280366d0 0x7fd028038b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:02.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.220+0000 7fd03a7fc640 1 -- 172.21.15.37:0/1565689459 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(662..662 src has 370..662) ==== 3899+0+0 (secure 0 0 0) 0x7fd034049480 con 0x7fd030004810 2024-04-26T00:37:02.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.220+0000 7fd049a6b640 1 --2- 172.21.15.37:0/1565689459 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd0280366d0 0x7fd028038b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:02.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.220+0000 7fd049a6b640 10 monclient: get_auth_request con 0x7fd0280366d0 auth_method 0 2024-04-26T00:37:02.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.220+0000 7fd049a6b640 1 --2- 172.21.15.37:0/1565689459 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd0280366d0 0x7fd028038b90 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fd0400ad000 tx=0x7fd040010040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:02.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.224+0000 7fd03a7fc640 1 -- 172.21.15.37:0/1565689459 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fd0440729d0 con 0x7fd030004810 2024-04-26T00:37:02.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.224+0000 7fd03a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:02.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.224+0000 7fd03a7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:37:02.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.452+0000 7fd04bccf640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:37:02.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.452+0000 7fd04bccf640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:37:02.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.452+0000 7fd04bccf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:37:02.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.452+0000 7fd04bccf640 1 -- 172.21.15.37:0/1565689459 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd04411fde0 con 0x7fd030004810 2024-04-26T00:37:02.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.452+0000 7fd03a7fc640 1 -- 172.21.15.37:0/1565689459 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7fd0340481f0 con 0x7fd030004810 2024-04-26T00:37:02.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.452+0000 7fd03a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:37:02.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.452+0000 7fd03a7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:37:02.452 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":130,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:37:02.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.452+0000 7fd04bccf640 1 -- 172.21.15.37:0/1565689459 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd0280366d0 msgr2=0x7fd028038b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:02.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.452+0000 7fd04bccf640 1 --2- 172.21.15.37:0/1565689459 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd0280366d0 0x7fd028038b90 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fd0400ad000 tx=0x7fd040010040 comp rx=0 tx=0).stop 2024-04-26T00:37:02.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.452+0000 7fd04bccf640 10 monclient: shutdown 2024-04-26T00:37:02.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.452+0000 7fd04bccf640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:37:02.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.452+0000 7fd04bccf640 1 -- 172.21.15.37:0/1565689459 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd030004810 msgr2=0x7fd0441ce7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:02.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.452+0000 7fd04bccf640 1 --2- 172.21.15.37:0/1565689459 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd030004810 0x7fd0441ce7a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd03400e8a0 tx=0x7fd0340146c0 comp rx=0 tx=0).stop 2024-04-26T00:37:02.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.452+0000 7fd04bccf640 1 -- 172.21.15.37:0/1565689459 shutdown_connections 2024-04-26T00:37:02.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.452+0000 7fd04bccf640 1 --2- 172.21.15.37:0/1565689459 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd0280366d0 0x7fd028038b90 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:02.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.452+0000 7fd04bccf640 1 --2- 172.21.15.37:0/1565689459 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd0300052f0 0x7fd0441cb2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:02.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.452+0000 7fd04bccf640 1 --2- 172.21.15.37:0/1565689459 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd030004810 0x7fd0441ce7a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:02.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.452+0000 7fd04bccf640 1 --2- 172.21.15.37:0/1565689459 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd0300039f0 0x7fd044131e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:02.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.452+0000 7fd04bccf640 1 -- 172.21.15.37:0/1565689459 >> 172.21.15.37:0/1565689459 conn(0x7fd0440a1ad0 msgr2=0x7fd0440a3d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:37:02.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.452+0000 7fd04bccf640 1 -- 172.21.15.37:0/1565689459 shutdown_connections 2024-04-26T00:37:02.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.452+0000 7fd04bccf640 1 -- 172.21.15.37:0/1565689459 wait complete. 2024-04-26T00:37:02.471 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-26T00:37:02.471 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-04-26T00:37:02.472 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:37:02.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.596+0000 7fe295515640 10 monclient: get_monmap_and_config 2024-04-26T00:37:02.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.596+0000 7fe295515640 10 monclient: build_initial_monmap 2024-04-26T00:37:02.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.596+0000 7fe295515640 10 monclient: monmap: 2024-04-26T00:37:02.596 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:37:02.596 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:02.596 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:37:02.601027+0000 2024-04-26T00:37:02.596 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:37:02.601027+0000 2024-04-26T00:37:02.596 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:37:02.596 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:02.596 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:37:02.597 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:37:02.597 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:37:02.597 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:37:02.597 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:37:02.597 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:02.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.596+0000 7fe295515640 1 Processor -- start 2024-04-26T00:37:02.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.596+0000 7fe295515640 1 -- start start 2024-04-26T00:37:02.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.596+0000 7fe295515640 10 monclient: init 2024-04-26T00:37:02.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.596+0000 7fe295515640 10 monclient: _reopen_session rank -1 2024-04-26T00:37:02.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.596+0000 7fe295515640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-04-26T00:37:02.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.596+0000 7fe295515640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe290136f20 addr v1:172.21.15.37:6789/0 2024-04-26T00:37:02.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.596+0000 7fe295515640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe290137a10 0x7fe29009bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:02.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.596+0000 7fe295515640 10 monclient(hunting): picked mon.noname-b con 0x7fe290137a10 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:37:02.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.596+0000 7fe295515640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe29013c450 addr v1:172.21.15.45:6789/0 2024-04-26T00:37:02.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.596+0000 7fe295515640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2900a50f0 con 0x7fe290136f20 2024-04-26T00:37:02.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.596+0000 7fe295515640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe29009b330 con 0x7fe29013c450 2024-04-26T00:37:02.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.596+0000 7fe295515640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:02.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.596+0000 7fe295515640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe29009b610 con 0x7fe290137a10 2024-04-26T00:37:02.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.596+0000 7fe295515640 10 monclient(hunting): _renew_subs 2024-04-26T00:37:02.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.596+0000 7fe295515640 10 monclient(hunting): authenticate will time out at 2003.906798s 2024-04-26T00:37:02.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.596+0000 7fe28e7fc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe290137a10 0x7fe29009bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:02.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.596+0000 7fe28f7fe640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7fe29013c450 legacy=0x7fe29013e840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:37:02.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.596+0000 7fe28e7fc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe290137a10 0x7fe29009bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:45228/0 (socket says 172.21.15.37:45228) 2024-04-26T00:37:02.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.596+0000 7fe28e7fc640 1 -- 172.21.15.37:0/1394669205 learned_addr learned my addr 172.21.15.37:0/1394669205 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:37:02.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.596+0000 7fe28e7fc640 10 monclient(hunting): get_auth_request con 0x7fe290137a10 auth_method 0 2024-04-26T00:37:02.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.596+0000 7fe28e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:02.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.596+0000 7fe28e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:02.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.596+0000 7fe28e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:02.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.596+0000 7fe28e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:02.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.596+0000 7fe28e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:02.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.596+0000 7fe28e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:02.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.596+0000 7fe28e7fc640 10 monclient(hunting): handle_auth_done global_id 204155 payload 274 2024-04-26T00:37:02.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.596+0000 7fe28e7fc640 1 -- 172.21.15.37:0/1394669205 >> v1:172.21.15.45:6789/0 conn(0x7fe29013c450 legacy=0x7fe29013e840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:37:02.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.596+0000 7fe28e7fc640 1 -- 172.21.15.37:0/1394669205 >> v1:172.21.15.37:6789/0 conn(0x7fe290136f20 legacy=0x7fe290137300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:37:02.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.596+0000 7fe28e7fc640 10 monclient: _finish_hunting 0 2024-04-26T00:37:02.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.596+0000 7fe28e7fc640 1 monclient: found mon.noname-b 2024-04-26T00:37:02.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.596+0000 7fe28e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:37:02.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.596+0000 7fe28e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:37:02.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.596+0000 7fe28e7fc640 1 -- 172.21.15.37:0/1394669205 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe29009b8f0 con 0x7fe290137a10 2024-04-26T00:37:02.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28e7fc640 1 -- 172.21.15.37:0/1394669205 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe290137a10 msgr2=0x7fe29009bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:02.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28e7fc640 1 --2- 172.21.15.37:0/1394669205 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe290137a10 0x7fe29009bff0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe290137e90 tx=0x7fe27801c430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6131cdb494e2670 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:02.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28dffb640 1 -- 172.21.15.37:0/1394669205 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2780b0070 con 0x7fe290137a10 2024-04-26T00:37:02.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:02.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:37:02.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28dffb640 10 monclient: dump: 2024-04-26T00:37:02.604 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:02.604 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:02.604 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:02.604 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:02.604 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:02.604 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:02.604 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:02.604 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:02.604 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:02.604 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:02.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28dffb640 10 monclient: _finish_auth 0 2024-04-26T00:37:02.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28dffb640 10 monclient: _check_auth_tickets 2024-04-26T00:37:02.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:37:02.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28dffb640 1 -- 172.21.15.37:0/1394669205 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2780ab600 con 0x7fe290137a10 2024-04-26T00:37:02.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28dffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:37:02.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28dffb640 1 -- 172.21.15.37:0/1394669205 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2780ab9d0 con 0x7fe290137a10 2024-04-26T00:37:02.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe295515640 5 monclient: authenticate success, global_id 204155 2024-04-26T00:37:02.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe295515640 10 monclient: get_monmap_and_config success 2024-04-26T00:37:02.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe295515640 4 set_mon_vals no callback set 2024-04-26T00:37:02.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:02.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:37:02.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28dffb640 10 monclient: dump: 2024-04-26T00:37:02.605 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:02.605 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:02.605 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:02.606 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:02.606 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:02.606 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:02.606 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:02.606 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:02.606 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:02.606 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:02.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe295515640 10 monclient: shutdown 2024-04-26T00:37:02.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe295515640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:37:02.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe295515640 1 -- 172.21.15.37:0/1394669205 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe290137a10 msgr2=0x7fe29009bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:02.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe295515640 1 --2- 172.21.15.37:0/1394669205 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe290137a10 0x7fe29009bff0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe290137e90 tx=0x7fe27801c430 comp rx=0 tx=0).stop 2024-04-26T00:37:02.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe295515640 1 -- 172.21.15.37:0/1394669205 shutdown_connections 2024-04-26T00:37:02.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe295515640 1 --2- 172.21.15.37:0/1394669205 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe290137a10 0x7fe29009bff0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:02.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe295515640 1 -- 172.21.15.37:0/1394669205 >> 172.21.15.37:0/1394669205 conn(0x7fe2900a61b0 msgr2=0x7fe2900a65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:37:02.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe295515640 1 -- 172.21.15.37:0/1394669205 shutdown_connections 2024-04-26T00:37:02.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe295515640 1 -- 172.21.15.37:0/1394669205 wait complete. 2024-04-26T00:37:02.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe295515640 10 monclient: build_initial_monmap 2024-04-26T00:37:02.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe295515640 10 monclient: monmap: 2024-04-26T00:37:02.611 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:37:02.611 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:37:02.611 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:37:02.611 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:37:02.611 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:37:02.611 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:37:02.611 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:37:02.611 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:37:02.611 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:37:02.611 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:02.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe295515640 1 Processor -- start 2024-04-26T00:37:02.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe295515640 1 -- start start 2024-04-26T00:37:02.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe295515640 10 monclient: init 2024-04-26T00:37:02.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe295515640 10 monclient: _reopen_session rank -1 2024-04-26T00:37:02.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe295515640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:37:02.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe295515640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe290136f20 0x7fe29013b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:02.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe295515640 10 monclient(hunting): picked mon.noname-a con 0x7fe290136f20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:37:02.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe295515640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe290137a10 0x7fe2901cef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:02.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe295515640 10 monclient(hunting): picked mon.noname-c con 0x7fe290137a10 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:37:02.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe295515640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe29013c450 0x7fe2901cf460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:02.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe295515640 10 monclient(hunting): picked mon.noname-b con 0x7fe29013c450 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:37:02.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe295515640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:02.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe295515640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe290142b50 con 0x7fe290136f20 2024-04-26T00:37:02.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe295515640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:02.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe295515640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe2901429d0 con 0x7fe29013c450 2024-04-26T00:37:02.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe295515640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:02.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe295515640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe290142cd0 con 0x7fe290137a10 2024-04-26T00:37:02.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe295515640 10 monclient(hunting): _renew_subs 2024-04-26T00:37:02.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe295515640 10 monclient(hunting): authenticate will time out at 2003.914596s 2024-04-26T00:37:02.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28effd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe290136f20 0x7fe29013b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:02.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28e7fc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe290137a10 0x7fe2901cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:02.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28f7fe640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe29013c450 msgr2=0x7fe2901cf460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:37:02.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe29013c450 0x7fe2901cf460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:37:02.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28effd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe290136f20 0x7fe29013b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:47298/0 (socket says 172.21.15.37:47298) 2024-04-26T00:37:02.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28e7fc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe290137a10 0x7fe2901cef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:45236/0 (socket says 172.21.15.37:45236) 2024-04-26T00:37:02.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28effd640 1 -- 172.21.15.37:0/3099812382 learned_addr learned my addr 172.21.15.37:0/3099812382 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:37:02.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28effd640 10 monclient(hunting): get_auth_request con 0x7fe290136f20 auth_method 0 2024-04-26T00:37:02.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:02.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:02.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:02.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28e7fc640 10 monclient(hunting): get_auth_request con 0x7fe290137a10 auth_method 0 2024-04-26T00:37:02.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:02.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:02.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:02.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:02.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:02.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:02.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:02.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:02.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:02.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28e7fc640 10 monclient(hunting): handle_auth_done global_id 204158 payload 824 2024-04-26T00:37:02.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28e7fc640 1 -- 172.21.15.37:0/3099812382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe29013c450 msgr2=0x7fe2901cf460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:37:02.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28e7fc640 1 --2- 172.21.15.37:0/3099812382 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe29013c450 0x7fe2901cf460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:02.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28e7fc640 1 -- 172.21.15.37:0/3099812382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe290136f20 msgr2=0x7fe29013b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:02.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28e7fc640 1 --2- 172.21.15.37:0/3099812382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe290136f20 0x7fe29013b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:02.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28e7fc640 10 monclient: _finish_hunting 0 2024-04-26T00:37:02.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28e7fc640 1 monclient: found mon.noname-c 2024-04-26T00:37:02.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:37:02.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28e7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:37:02.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.604+0000 7fe28e7fc640 1 -- 172.21.15.37:0/3099812382 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe290136180 con 0x7fe290137a10 2024-04-26T00:37:02.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe28e7fc640 1 -- 172.21.15.37:0/3099812382 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe290137a10 msgr2=0x7fe2901cef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:02.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe28e7fc640 1 --2- 172.21.15.37:0/3099812382 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe290137a10 0x7fe2901cef20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe2780ad000 tx=0x7fe2780191d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:02.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe2877fe640 1 -- 172.21.15.37:0/3099812382 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2780b9020 con 0x7fe290137a10 2024-04-26T00:37:02.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe2877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:02.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe2877fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:37:02.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe2877fe640 10 monclient: dump: 2024-04-26T00:37:02.615 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:02.615 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:02.615 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:02.615 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:02.616 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:02.616 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:02.616 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:02.616 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:02.616 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:02.616 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:02.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe2877fe640 10 monclient: _finish_auth 0 2024-04-26T00:37:02.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe2877fe640 10 monclient: _check_auth_tickets 2024-04-26T00:37:02.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe2877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:37:02.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe2877fe640 1 -- 172.21.15.37:0/3099812382 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2780b1040 con 0x7fe290137a10 2024-04-26T00:37:02.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe2877fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:37:02.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe2877fe640 1 -- 172.21.15.37:0/3099812382 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2780b0050 con 0x7fe290137a10 2024-04-26T00:37:02.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe295515640 5 monclient: authenticate success, global_id 204158 2024-04-26T00:37:02.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe295515640 10 monclient: _renew_subs 2024-04-26T00:37:02.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe295515640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:37:02.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe295515640 1 -- 172.21.15.37:0/3099812382 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2900a35b0 con 0x7fe290137a10 2024-04-26T00:37:02.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe2877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:02.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe28d7fa640 4 set_mon_vals no callback set 2024-04-26T00:37:02.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe2877fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:37:02.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe2877fe640 10 monclient: dump: 2024-04-26T00:37:02.617 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:02.617 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:02.617 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:02.617 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:02.617 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:02.617 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:02.617 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:02.617 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:02.617 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:02.617 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:02.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe295515640 10 monclient: _renew_subs 2024-04-26T00:37:02.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe295515640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:37:02.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe295515640 1 -- 172.21.15.37:0/3099812382 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2900a2b80 con 0x7fe290137a10 2024-04-26T00:37:02.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe295515640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:02.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe295515640 1 --2- 172.21.15.37:0/3099812382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2901cf9a0 0x7fe290135440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:02.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe295515640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:02.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe295515640 1 -- 172.21.15.37:0/3099812382 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe29013a440 con 0x7fe2901cf9a0 2024-04-26T00:37:02.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe28effd640 1 --2- 172.21.15.37:0/3099812382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2901cf9a0 0x7fe290135440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:02.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe28effd640 10 monclient: get_auth_request con 0x7fe2901cf9a0 auth_method 0 2024-04-26T00:37:02.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe28effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:02.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe28effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:02.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe28effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:02.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe28effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:02.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe28effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:02.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe28effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:02.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe2877fe640 1 -- 172.21.15.37:0/3099812382 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe2780bc020 con 0x7fe290137a10 2024-04-26T00:37:02.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe28effd640 10 monclient(hunting): handle_auth_done global_id 144780 payload 824 2024-04-26T00:37:02.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe28effd640 1 -- 172.21.15.37:0/3099812382 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe2901cfd80 con 0x7fe2901cf9a0 2024-04-26T00:37:02.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe2877fe640 1 --2- 172.21.15.37:0/3099812382 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe270035e80 0x7fe270038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:02.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe2877fe640 1 -- 172.21.15.37:0/3099812382 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(662..662 src has 370..662) ==== 3899+0+0 (secure 0 0 0) 0x7fe2780e2870 con 0x7fe290137a10 2024-04-26T00:37:02.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe28f7fe640 1 --2- 172.21.15.37:0/3099812382 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe270035e80 0x7fe270038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:02.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.608+0000 7fe28f7fe640 10 monclient: get_auth_request con 0x7fe270035e80 auth_method 0 2024-04-26T00:37:02.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.612+0000 7fe28effd640 1 -- 172.21.15.37:0/3099812382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2901cf9a0 msgr2=0x7fe290135440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:02.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.612+0000 7fe28effd640 1 --2- 172.21.15.37:0/3099812382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2901cf9a0 0x7fe290135440 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fe28800fb10 tx=0x7fe2880058e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:02.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.612+0000 7fe2877fe640 1 -- 172.21.15.37:0/3099812382 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe288018030 con 0x7fe2901cf9a0 2024-04-26T00:37:02.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.612+0000 7fe2877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:02.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.612+0000 7fe2877fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:37:02.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.612+0000 7fe2877fe640 10 monclient: dump: 2024-04-26T00:37:02.621 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:02.621 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:02.621 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:02.621 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:02.621 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:02.621 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:02.621 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:02.621 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:02.621 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:02.622 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:02.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.612+0000 7fe2877fe640 1 -- 172.21.15.37:0/3099812382 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe28800ddf0 con 0x7fe2901cf9a0 2024-04-26T00:37:02.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.612+0000 7fe2877fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:02.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.612+0000 7fe2877fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:37:02.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.612+0000 7fe2877fe640 1 -- 172.21.15.37:0/3099812382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2901cf9a0 msgr2=0x7fe290135440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:02.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.612+0000 7fe2877fe640 1 --2- 172.21.15.37:0/3099812382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2901cf9a0 0x7fe290135440 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fe28800fb10 tx=0x7fe2880058e0 comp rx=0 tx=0).stop 2024-04-26T00:37:02.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.612+0000 7fe2877fe640 1 -- 172.21.15.37:0/3099812382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2901cf9a0 msgr2=0x7fe290135440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:37:02.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.612+0000 7fe2877fe640 1 --2- 172.21.15.37:0/3099812382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2901cf9a0 0x7fe290135440 secure :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fe28800fb10 tx=0x7fe2880058e0 comp rx=0 tx=0).stop 2024-04-26T00:37:02.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.612+0000 7fe28f7fe640 1 --2- 172.21.15.37:0/3099812382 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe270035e80 0x7fe270038340 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fe290139110 tx=0x7fe280006c70 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:02.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.616+0000 7fe295515640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:37:02.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.616+0000 7fe295515640 1 --2- 172.21.15.37:0/3099812382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe29002c000 0x7fe29002c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:02.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.616+0000 7fe295515640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:02.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.616+0000 7fe295515640 1 -- 172.21.15.37:0/3099812382 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe29013aa20 con 0x7fe29002c000 2024-04-26T00:37:02.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.620+0000 7fe28effd640 1 --2- 172.21.15.37:0/3099812382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe29002c000 0x7fe29002c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:02.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.620+0000 7fe28effd640 10 monclient: get_auth_request con 0x7fe29002c000 auth_method 0 2024-04-26T00:37:02.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.620+0000 7fe28effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:02.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.620+0000 7fe28effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:02.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.620+0000 7fe28effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:02.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.620+0000 7fe28effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:02.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.620+0000 7fe28effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:02.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.620+0000 7fe28effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:02.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.620+0000 7fe28effd640 10 monclient(hunting): handle_auth_done global_id 144783 payload 824 2024-04-26T00:37:02.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.620+0000 7fe28effd640 1 -- 172.21.15.37:0/3099812382 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe2900688c0 con 0x7fe29002c000 2024-04-26T00:37:02.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.620+0000 7fe28effd640 1 -- 172.21.15.37:0/3099812382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe29002c000 msgr2=0x7fe29002c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:02.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.620+0000 7fe28effd640 1 --2- 172.21.15.37:0/3099812382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe29002c000 0x7fe29002c410 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fe2880059b0 tx=0x7fe288003af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:02.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.620+0000 7fe2877fe640 1 -- 172.21.15.37:0/3099812382 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe288014890 con 0x7fe29002c000 2024-04-26T00:37:02.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.620+0000 7fe2877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:02.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.620+0000 7fe2877fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:37:02.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.620+0000 7fe2877fe640 10 monclient: dump: 2024-04-26T00:37:02.628 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:02.628 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:02.628 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:02.628 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:02.628 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:02.628 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:02.628 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:02.628 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:02.628 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:02.628 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:02.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.620+0000 7fe2877fe640 1 -- 172.21.15.37:0/3099812382 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7fe288014890 con 0x7fe29002c000 2024-04-26T00:37:02.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.620+0000 7fe2877fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:37:02.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.620+0000 7fe2877fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:37:02.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.620+0000 7fe2877fe640 1 -- 172.21.15.37:0/3099812382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe29002c000 msgr2=0x7fe29002c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:02.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.620+0000 7fe2877fe640 1 --2- 172.21.15.37:0/3099812382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe29002c000 0x7fe29002c410 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fe2880059b0 tx=0x7fe288003af0 comp rx=0 tx=0).stop 2024-04-26T00:37:02.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.620+0000 7fe2877fe640 1 -- 172.21.15.37:0/3099812382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe29002c000 msgr2=0x7fe29002c410 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:37:02.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.620+0000 7fe2877fe640 1 --2- 172.21.15.37:0/3099812382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe29002c000 0x7fe29002c410 secure :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fe2880059b0 tx=0x7fe288003af0 comp rx=0 tx=0).stop 2024-04-26T00:37:02.629 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:37:02.629 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:37:02.629 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:37:02.629 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:37:02.629 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 130, 2024-04-26T00:37:02.629 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 309163, 2024-04-26T00:37:02.629 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:37:02.629 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:37:02.630 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:37:02.630 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:02.630 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:37:02.630 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:37:02.630 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:37:02.630 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:37:02.630 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:37:02.630 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:37:02.630 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:37:02.630 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:37:02.630 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:37:02.630 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:37:02.630 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:37:02.630 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:37:02.631 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:37:02.631 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:37:02.631 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:37:02.631 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:02.631 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:37:02.631 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:37:02.631 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:37:02.631 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:37:02.631 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:37:02.631 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:37:02.631 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:37:02.631 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:37:02.631 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:37:02.631 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:37:02.631 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:37:02.632 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:37:02.632 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:37:02.632 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:02.632 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:02.632 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:37:02.632 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:37:02.632 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:37:02.632 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:37:02.632 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:37:02.632 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:37:02.632 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:37:02.632 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:37:02.632 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:37:02.632 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:37:02.633 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:37:02.633 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:37:02.633 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:37:02.633 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:37:02.633 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:37:02.633 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:37:02.633 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:37:02.633 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:37:02.633 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:37:02.633 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:37:02.633 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:37:02.633 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:37:02.633 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:37:02.633 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:37:02.633 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:37:02.634 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:37:02.634 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:37:02.634 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:37:02.634 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:02.634 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:37:02.634 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:02.634 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:37:02.634 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:02.634 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:37:02.634 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:37:02.634 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:37:02.634 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:37:02.634 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:02.634 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:37:02.635 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:37:02.635 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:02.635 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:02.635 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:02.635 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:37:02.635 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:37:02.635 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:02.635 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:02.635 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:02.635 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:02.635 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:37:02.635 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:37:02.635 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:37:02.635 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:37:02.636 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:37:02.636 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:02.636 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:02.636 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:37:02.636 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:37:02.636 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:37:02.636 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:37:02.636 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:02.636 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:37:02.636 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:37:02.636 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:02.636 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:02.636 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:02.636 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:37:02.636 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:37:02.637 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:02.637 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:02.637 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:02.637 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:02.637 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:37:02.637 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:37:02.637 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:37:02.637 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:37:02.637 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:37:02.637 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:02.637 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:02.637 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:37:02.637 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:37:02.637 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:37:02.638 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:37:02.638 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:02.638 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:37:02.638 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:37:02.638 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:02.638 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:02.638 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:02.638 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:37:02.638 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:37:02.638 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:02.638 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:02.638 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:02.638 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:02.638 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:37:02.638 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:37:02.639 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:37:02.639 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:37:02.639 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:37:02.639 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:02.639 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:02.639 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:02.639 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:37:02.639 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:37:02.639 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:02.639 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:37:02.639 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:37:02.639 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:37:02.639 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:02.639 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:02.639 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:37:02.640 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:02.640 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:37:02.640 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:37:02.640 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:37:02.640 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:02.640 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:02.640 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:37:02.640 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:02.640 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:37:02.640 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:37:02.640 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:37:02.640 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:02.640 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:02.640 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:37:02.640 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:02.641 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:37:02.641 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:37:02.641 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:37:02.641 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:02.641 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:02.641 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:02.641 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:37:02.641 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:37:02.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.620+0000 7fe295515640 1 -- 172.21.15.37:0/3099812382 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe270035e80 msgr2=0x7fe270038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:02.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.620+0000 7fe295515640 1 --2- 172.21.15.37:0/3099812382 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe270035e80 0x7fe270038340 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fe290139110 tx=0x7fe280006c70 comp rx=0 tx=0).stop 2024-04-26T00:37:02.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.620+0000 7fe295515640 10 monclient: shutdown 2024-04-26T00:37:02.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.620+0000 7fe295515640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:37:02.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.620+0000 7fe295515640 1 -- 172.21.15.37:0/3099812382 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe290137a10 msgr2=0x7fe2901cef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:02.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.620+0000 7fe295515640 1 --2- 172.21.15.37:0/3099812382 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe290137a10 0x7fe2901cef20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe2780ad000 tx=0x7fe2780191d0 comp rx=0 tx=0).stop 2024-04-26T00:37:02.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.620+0000 7fe28f7fe640 1 -- 172.21.15.37:0/3099812382 reap_dead start 2024-04-26T00:37:02.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.620+0000 7fe28f7fe640 1 -- 172.21.15.37:0/3099812382 reap_dead start 2024-04-26T00:37:02.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.620+0000 7fe295515640 1 -- 172.21.15.37:0/3099812382 shutdown_connections 2024-04-26T00:37:02.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.620+0000 7fe295515640 1 -- 172.21.15.37:0/3099812382 >> 172.21.15.37:0/3099812382 conn(0x7fe2900a61b0 msgr2=0x7fe2900a8060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:37:02.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.620+0000 7fe295515640 1 -- 172.21.15.37:0/3099812382 shutdown_connections 2024-04-26T00:37:02.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.620+0000 7fe295515640 1 -- 172.21.15.37:0/3099812382 wait complete. 2024-04-26T00:37:02.643 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:37:02.643 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-04-26T00:37:02.644 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:37:02.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe877a37640 10 monclient: get_monmap_and_config 2024-04-26T00:37:02.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe877a37640 10 monclient: build_initial_monmap 2024-04-26T00:37:02.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe877a37640 10 monclient: monmap: 2024-04-26T00:37:02.759 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:37:02.759 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:02.759 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:37:02.764076+0000 2024-04-26T00:37:02.759 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:37:02.764076+0000 2024-04-26T00:37:02.759 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:37:02.759 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:02.759 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:37:02.760 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:37:02.760 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:37:02.760 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:37:02.760 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:37:02.760 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:02.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe877a37640 1 Processor -- start 2024-04-26T00:37:02.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe877a37640 1 -- start start 2024-04-26T00:37:02.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe877a37640 10 monclient: init 2024-04-26T00:37:02.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe877a37640 10 monclient: _reopen_session rank -1 2024-04-26T00:37:02.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe877a37640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-04-26T00:37:02.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe877a37640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe8700a2fe0 0x7fe8701351b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:02.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe877a37640 10 monclient(hunting): picked mon.noname-b con 0x7fe8700a2fe0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:37:02.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe877a37640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe8701356f0 0x7fe8700a13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:02.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe877a37640 10 monclient(hunting): picked mon.noname-c con 0x7fe8701356f0 addr v2:172.21.15.45:3300/0 2024-04-26T00:37:02.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe877a37640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe8700a1910 addr v1:172.21.15.45:6789/0 2024-04-26T00:37:02.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe877a37640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe870072ec0 con 0x7fe8700a1910 2024-04-26T00:37:02.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe877a37640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:02.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe877a37640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fe8700a33c0 con 0x7fe8701356f0 2024-04-26T00:37:02.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe877a37640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:02.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe877a37640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe870071f00 con 0x7fe8700a2fe0 2024-04-26T00:37:02.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe877a37640 10 monclient(hunting): _renew_subs 2024-04-26T00:37:02.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe877a37640 10 monclient(hunting): authenticate will time out at 2004.069872s 2024-04-26T00:37:02.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe8757d3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe8700a2fe0 0x7fe8701351b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:02.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe875fd4640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7fe8700a1910 legacy=0x7fe87013bdf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:37:02.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe874fd2640 1 -- >> v2:172.21.15.45:3300/0 conn(0x7fe8701356f0 msgr2=0x7fe8700a13d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:37:02.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe8757d3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe8700a2fe0 0x7fe8701351b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:45246/0 (socket says 172.21.15.37:45246) 2024-04-26T00:37:02.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe8757d3640 1 -- 172.21.15.37:0/3009321182 learned_addr learned my addr 172.21.15.37:0/3009321182 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:37:02.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe8757d3640 10 monclient(hunting): get_auth_request con 0x7fe8700a2fe0 auth_method 0 2024-04-26T00:37:02.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe8757d3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:02.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe8757d3640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:02.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe8757d3640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:02.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe874fd2640 1 --2- 172.21.15.37:0/3009321182 >> v2:172.21.15.45:3300/0 conn(0x7fe8701356f0 0x7fe8700a13d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:37:02.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe8757d3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:02.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe8757d3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:02.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe8757d3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:02.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe8757d3640 10 monclient(hunting): handle_auth_done global_id 204161 payload 274 2024-04-26T00:37:02.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe8757d3640 1 -- 172.21.15.37:0/3009321182 >> v2:172.21.15.45:3300/0 conn(0x7fe8701356f0 msgr2=0x7fe8700a13d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:37:02.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe8757d3640 1 --2- 172.21.15.37:0/3009321182 >> v2:172.21.15.45:3300/0 conn(0x7fe8701356f0 0x7fe8700a13d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:02.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe8757d3640 1 -- 172.21.15.37:0/3009321182 >> v1:172.21.15.45:6789/0 conn(0x7fe8700a1910 legacy=0x7fe87013bdf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:37:02.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe8757d3640 10 monclient: _finish_hunting 0 2024-04-26T00:37:02.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe8757d3640 1 monclient: found mon.noname-b 2024-04-26T00:37:02.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe8757d3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:37:02.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe8757d3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:37:02.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.760+0000 7fe8757d3640 1 -- 172.21.15.37:0/3009321182 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe870134ac0 con 0x7fe8700a2fe0 2024-04-26T00:37:02.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe8757d3640 1 -- 172.21.15.37:0/3009321182 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe8700a2fe0 msgr2=0x7fe8701351b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:02.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe8757d3640 1 --2- 172.21.15.37:0/3009321182 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe8700a2fe0 0x7fe8701351b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe860009870 tx=0x7fe86001c2b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9261ff4775f7c2dd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:02.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe867fff640 1 -- 172.21.15.37:0/3009321182 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8600b0070 con 0x7fe8700a2fe0 2024-04-26T00:37:02.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe867fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:02.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe867fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:37:02.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe867fff640 10 monclient: dump: 2024-04-26T00:37:02.768 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:02.768 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:02.768 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:02.768 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:02.768 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:02.768 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:02.768 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:02.769 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:02.769 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:02.769 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:02.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe867fff640 10 monclient: _finish_auth 0 2024-04-26T00:37:02.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe867fff640 10 monclient: _check_auth_tickets 2024-04-26T00:37:02.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe867fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:37:02.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe867fff640 1 -- 172.21.15.37:0/3009321182 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8600ab600 con 0x7fe8700a2fe0 2024-04-26T00:37:02.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe877a37640 5 monclient: authenticate success, global_id 204161 2024-04-26T00:37:02.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe877a37640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:37:02.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe867fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:37:02.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe877a37640 10 monclient: get_monmap_and_config success 2024-04-26T00:37:02.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe867fff640 1 -- 172.21.15.37:0/3009321182 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8600ab9d0 con 0x7fe8700a2fe0 2024-04-26T00:37:02.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe877a37640 4 set_mon_vals no callback set 2024-04-26T00:37:02.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe867fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:02.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe867fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:37:02.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe867fff640 10 monclient: dump: 2024-04-26T00:37:02.770 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:02.770 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:02.770 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:02.770 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:02.770 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:02.770 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:02.770 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:02.770 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:02.770 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:02.770 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:02.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe877a37640 10 monclient: shutdown 2024-04-26T00:37:02.770 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe877a37640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:37:02.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe877a37640 1 -- 172.21.15.37:0/3009321182 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe8700a2fe0 msgr2=0x7fe8701351b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:02.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe877a37640 1 --2- 172.21.15.37:0/3009321182 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe8700a2fe0 0x7fe8701351b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe860009870 tx=0x7fe86001c2b0 comp rx=0 tx=0).stop 2024-04-26T00:37:02.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe877a37640 1 -- 172.21.15.37:0/3009321182 shutdown_connections 2024-04-26T00:37:02.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe877a37640 1 --2- 172.21.15.37:0/3009321182 >> v2:172.21.15.45:3300/0 conn(0x7fe8701356f0 0x7fe8700a13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:02.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe877a37640 1 --2- 172.21.15.37:0/3009321182 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe8700a2fe0 0x7fe8701351b0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:02.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe877a37640 1 -- 172.21.15.37:0/3009321182 >> 172.21.15.37:0/3009321182 conn(0x7fe8700a24c0 msgr2=0x7fe8700a28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:37:02.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe877a37640 1 -- 172.21.15.37:0/3009321182 shutdown_connections 2024-04-26T00:37:02.771 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe877a37640 1 -- 172.21.15.37:0/3009321182 wait complete. 2024-04-26T00:37:02.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe877a37640 10 monclient: build_initial_monmap 2024-04-26T00:37:02.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe877a37640 10 monclient: monmap: 2024-04-26T00:37:02.780 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:37:02.780 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:37:02.780 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:37:02.780 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:37:02.780 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:37:02.780 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:37:02.780 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:37:02.780 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:37:02.780 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:37:02.781 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:02.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe877a37640 1 Processor -- start 2024-04-26T00:37:02.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe877a37640 1 -- start start 2024-04-26T00:37:02.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe877a37640 10 monclient: init 2024-04-26T00:37:02.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe877a37640 10 monclient: _reopen_session rank -1 2024-04-26T00:37:02.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe877a37640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:37:02.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe877a37640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe8700a1910 0x7fe8701cd260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:02.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe877a37640 10 monclient(hunting): picked mon.noname-c con 0x7fe8700a1910 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:37:02.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe877a37640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8700a2fe0 0x7fe87012e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:02.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe877a37640 10 monclient(hunting): picked mon.noname-b con 0x7fe8700a2fe0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:37:02.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe877a37640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe8701356f0 0x7fe87012ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:02.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe877a37640 10 monclient(hunting): picked mon.noname-a con 0x7fe8701356f0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:37:02.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe877a37640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:02.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe877a37640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe87006ed70 con 0x7fe8701356f0 2024-04-26T00:37:02.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe877a37640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:02.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe877a37640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe87006ebf0 con 0x7fe8700a2fe0 2024-04-26T00:37:02.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe877a37640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:02.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe877a37640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe87013fd30 con 0x7fe8700a1910 2024-04-26T00:37:02.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe877a37640 10 monclient(hunting): _renew_subs 2024-04-26T00:37:02.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe877a37640 10 monclient(hunting): authenticate will time out at 2004.079258s 2024-04-26T00:37:02.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe8757d3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe8700a1910 0x7fe8701cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:02.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe874fd2640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8700a2fe0 msgr2=0x7fe87012e4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:37:02.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe875fd4640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe8701356f0 0x7fe87012ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:02.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe874fd2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8700a2fe0 0x7fe87012e4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:37:02.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe8757d3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe8700a1910 0x7fe8701cd260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:45262/0 (socket says 172.21.15.37:45262) 2024-04-26T00:37:02.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe8757d3640 1 -- 172.21.15.37:0/3725827689 learned_addr learned my addr 172.21.15.37:0/3725827689 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:37:02.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe8757d3640 10 monclient(hunting): get_auth_request con 0x7fe8700a1910 auth_method 0 2024-04-26T00:37:02.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe875fd4640 1 --2- 172.21.15.37:0/3725827689 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe8701356f0 0x7fe87012ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:47316/0 (socket says 172.21.15.37:47316) 2024-04-26T00:37:02.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe8757d3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:02.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe8757d3640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:02.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe8757d3640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:02.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe875fd4640 10 monclient(hunting): get_auth_request con 0x7fe8701356f0 auth_method 0 2024-04-26T00:37:02.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe875fd4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:02.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe875fd4640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:02.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe875fd4640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:02.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe875fd4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:02.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe875fd4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:02.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe875fd4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:02.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe8757d3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:02.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe8757d3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:02.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe8757d3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:02.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe875fd4640 10 monclient(hunting): handle_auth_done global_id 144786 payload 824 2024-04-26T00:37:02.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe875fd4640 1 -- 172.21.15.37:0/3725827689 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe8700a1910 msgr2=0x7fe8701cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:02.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe875fd4640 1 --2- 172.21.15.37:0/3725827689 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe8700a1910 0x7fe8701cd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:02.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe875fd4640 1 -- 172.21.15.37:0/3725827689 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8700a2fe0 msgr2=0x7fe87012e4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:37:02.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe875fd4640 1 --2- 172.21.15.37:0/3725827689 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8700a2fe0 0x7fe87012e4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:02.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe875fd4640 10 monclient: _finish_hunting 0 2024-04-26T00:37:02.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe875fd4640 1 monclient: found mon.noname-a 2024-04-26T00:37:02.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe875fd4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:37:02.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe875fd4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:37:02.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.768+0000 7fe875fd4640 1 -- 172.21.15.37:0/3725827689 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8701398d0 con 0x7fe8701356f0 2024-04-26T00:37:02.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe875fd4640 1 -- 172.21.15.37:0/3725827689 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe8701356f0 msgr2=0x7fe87012ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:02.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe875fd4640 1 --2- 172.21.15.37:0/3725827689 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe8701356f0 0x7fe87012ea30 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe86800b390 tx=0x7fe86800b850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:02.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe865ffb640 1 -- 172.21.15.37:0/3725827689 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe868016070 con 0x7fe8701356f0 2024-04-26T00:37:02.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe865ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:02.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe865ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:37:02.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe865ffb640 10 monclient: dump: 2024-04-26T00:37:02.785 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:02.785 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:02.785 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:02.785 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:02.785 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:02.786 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:02.786 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:02.786 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:02.786 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:02.786 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:02.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe865ffb640 10 monclient: _finish_auth 0 2024-04-26T00:37:02.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe865ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:37:02.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe865ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:37:02.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe877a37640 5 monclient: authenticate success, global_id 144786 2024-04-26T00:37:02.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe865ffb640 1 -- 172.21.15.37:0/3725827689 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe868005070 con 0x7fe8701356f0 2024-04-26T00:37:02.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe877a37640 10 monclient: _renew_subs 2024-04-26T00:37:02.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe877a37640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:37:02.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe877a37640 1 -- 172.21.15.37:0/3725827689 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe87006ef50 con 0x7fe8701356f0 2024-04-26T00:37:02.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe865ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:37:02.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe865ffb640 1 -- 172.21.15.37:0/3725827689 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe868005460 con 0x7fe8701356f0 2024-04-26T00:37:02.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe865ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:02.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe865ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:37:02.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe865ffb640 10 monclient: dump: 2024-04-26T00:37:02.787 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:02.787 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:02.787 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:02.787 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:02.787 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:02.787 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:02.787 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:02.787 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:02.787 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:02.787 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:02.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe877a37640 10 monclient: _renew_subs 2024-04-26T00:37:02.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe877a37640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:37:02.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe8677fe640 4 set_mon_vals no callback set 2024-04-26T00:37:02.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe877a37640 1 -- 172.21.15.37:0/3725827689 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe870137ec0 con 0x7fe8701356f0 2024-04-26T00:37:02.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe877a37640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:02.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe877a37640 1 --2- 172.21.15.37:0/3725827689 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe870139030 0x7fe870135bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:02.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe877a37640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:02.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe877a37640 1 -- 172.21.15.37:0/3725827689 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe8700688f0 con 0x7fe870139030 2024-04-26T00:37:02.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe8757d3640 1 --2- 172.21.15.37:0/3725827689 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe870139030 0x7fe870135bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:02.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe8757d3640 10 monclient: get_auth_request con 0x7fe870139030 auth_method 0 2024-04-26T00:37:02.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe8757d3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:02.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe8757d3640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:02.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe8757d3640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:02.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe865ffb640 1 -- 172.21.15.37:0/3725827689 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe868019070 con 0x7fe8701356f0 2024-04-26T00:37:02.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe8757d3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:02.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe8757d3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:02.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe8757d3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:02.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe8757d3640 10 monclient(hunting): handle_auth_done global_id 204167 payload 824 2024-04-26T00:37:02.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe865ffb640 1 --2- 172.21.15.37:0/3725827689 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe85c035e60 0x7fe85c038320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:02.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe8757d3640 1 -- 172.21.15.37:0/3725827689 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe87004f510 con 0x7fe870139030 2024-04-26T00:37:02.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe865ffb640 1 -- 172.21.15.37:0/3725827689 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(662..662 src has 370..662) ==== 3899+0+0 (secure 0 0 0) 0x7fe868047c30 con 0x7fe8701356f0 2024-04-26T00:37:02.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe874fd2640 1 --2- 172.21.15.37:0/3725827689 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe85c035e60 0x7fe85c038320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:02.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.772+0000 7fe874fd2640 10 monclient: get_auth_request con 0x7fe85c035e60 auth_method 0 2024-04-26T00:37:02.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.776+0000 7fe8757d3640 1 -- 172.21.15.37:0/3725827689 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe870139030 msgr2=0x7fe870135bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:02.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.776+0000 7fe8757d3640 1 --2- 172.21.15.37:0/3725827689 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe870139030 0x7fe870135bb0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe860009870 tx=0x7fe860019050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:02.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.776+0000 7fe865ffb640 1 -- 172.21.15.37:0/3725827689 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8600ba030 con 0x7fe870139030 2024-04-26T00:37:02.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.776+0000 7fe865ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:02.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.776+0000 7fe865ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:37:02.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.776+0000 7fe865ffb640 10 monclient: dump: 2024-04-26T00:37:02.790 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:02.790 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:02.790 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:02.790 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:02.790 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:02.790 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:02.790 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:02.790 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:02.790 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:02.790 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:02.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.776+0000 7fe865ffb640 1 -- 172.21.15.37:0/3725827689 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe8600b1040 con 0x7fe870139030 2024-04-26T00:37:02.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.776+0000 7fe865ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:02.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.776+0000 7fe865ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:37:02.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.776+0000 7fe865ffb640 1 -- 172.21.15.37:0/3725827689 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe870139030 msgr2=0x7fe870135bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:02.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.776+0000 7fe865ffb640 1 --2- 172.21.15.37:0/3725827689 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe870139030 0x7fe870135bb0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe860009870 tx=0x7fe860019050 comp rx=0 tx=0).stop 2024-04-26T00:37:02.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.776+0000 7fe865ffb640 1 -- 172.21.15.37:0/3725827689 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe870139030 msgr2=0x7fe870135bb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:37:02.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.776+0000 7fe865ffb640 1 --2- 172.21.15.37:0/3725827689 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe870139030 0x7fe870135bb0 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe860009870 tx=0x7fe860019050 comp rx=0 tx=0).stop 2024-04-26T00:37:02.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.776+0000 7fe874fd2640 1 --2- 172.21.15.37:0/3725827689 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe85c035e60 0x7fe85c038320 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fe870138610 tx=0x7fe86c009270 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:02.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.784+0000 7fe877a37640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:37:02.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.784+0000 7fe877a37640 1 --2- 172.21.15.37:0/3725827689 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe8701360f0 0x7fe870136500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:02.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.784+0000 7fe877a37640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:02.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.784+0000 7fe877a37640 1 -- 172.21.15.37:0/3725827689 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe870136a40 con 0x7fe8701360f0 2024-04-26T00:37:02.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.784+0000 7fe8757d3640 1 --2- 172.21.15.37:0/3725827689 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe8701360f0 0x7fe870136500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:02.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.784+0000 7fe8757d3640 10 monclient: get_auth_request con 0x7fe8701360f0 auth_method 0 2024-04-26T00:37:02.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.784+0000 7fe8757d3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:02.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.784+0000 7fe8757d3640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:02.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.784+0000 7fe8757d3640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:02.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.784+0000 7fe8757d3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:02.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.784+0000 7fe8757d3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:02.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.784+0000 7fe8757d3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:02.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.784+0000 7fe8757d3640 10 monclient(hunting): handle_auth_done global_id 204170 payload 824 2024-04-26T00:37:02.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.784+0000 7fe8757d3640 1 -- 172.21.15.37:0/3725827689 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe87006c1e0 con 0x7fe8701360f0 2024-04-26T00:37:02.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.784+0000 7fe8757d3640 1 -- 172.21.15.37:0/3725827689 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe8701360f0 msgr2=0x7fe870136500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:02.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.784+0000 7fe8757d3640 1 --2- 172.21.15.37:0/3725827689 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe8701360f0 0x7fe870136500 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe86001c2f0 tx=0x7fe8600a8000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:02.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.784+0000 7fe865ffb640 1 -- 172.21.15.37:0/3725827689 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8600ba070 con 0x7fe8701360f0 2024-04-26T00:37:02.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.784+0000 7fe865ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:02.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.784+0000 7fe865ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:37:02.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.784+0000 7fe865ffb640 10 monclient: dump: 2024-04-26T00:37:02.794 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:02.794 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:02.794 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:02.794 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:02.794 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:02.795 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:02.795 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:02.795 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:02.795 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:02.795 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:02.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.784+0000 7fe865ffb640 1 -- 172.21.15.37:0/3725827689 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4536 (secure 0 0 0) 0x7fe860003e10 con 0x7fe8701360f0 2024-04-26T00:37:02.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.784+0000 7fe865ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:37:02.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.784+0000 7fe865ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:37:02.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.784+0000 7fe865ffb640 1 -- 172.21.15.37:0/3725827689 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe8701360f0 msgr2=0x7fe870136500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:02.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.784+0000 7fe865ffb640 1 --2- 172.21.15.37:0/3725827689 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe8701360f0 0x7fe870136500 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe86001c2f0 tx=0x7fe8600a8000 comp rx=0 tx=0).stop 2024-04-26T00:37:02.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.784+0000 7fe865ffb640 1 -- 172.21.15.37:0/3725827689 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe8701360f0 msgr2=0x7fe870136500 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:37:02.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.784+0000 7fe865ffb640 1 --2- 172.21.15.37:0/3725827689 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe8701360f0 0x7fe870136500 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe86001c2f0 tx=0x7fe8600a8000 comp rx=0 tx=0).stop 2024-04-26T00:37:02.796 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:37:02.796 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:37:02.796 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:37:02.796 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:37:02.796 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 130, 2024-04-26T00:37:02.796 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 28391, 2024-04-26T00:37:02.796 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:37:02.796 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:37:02.796 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:37:02.796 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:02.796 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:37:02.796 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:37:02.796 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:37:02.797 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:37:02.797 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:37:02.797 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:37:02.797 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:37:02.797 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:37:02.797 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:37:02.797 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:37:02.797 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:37:02.797 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:37:02.797 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:37:02.797 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:37:02.797 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:37:02.797 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:02.797 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:37:02.798 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:37:02.798 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:37:02.798 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:37:02.798 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:37:02.798 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:37:02.798 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:37:02.798 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:37:02.798 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:37:02.798 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:37:02.798 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:37:02.798 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:37:02.798 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:37:02.798 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:02.798 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:02.799 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:37:02.799 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:37:02.799 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:37:02.799 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:37:02.799 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:37:02.799 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:37:02.799 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:37:02.799 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:37:02.799 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:37:02.799 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:37:02.799 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:37:02.799 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:37:02.799 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:37:02.799 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:37:02.799 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:37:02.800 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:37:02.800 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:37:02.800 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:37:02.800 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:37:02.800 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:37:02.800 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:37:02.800 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:37:02.800 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:37:02.800 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:37:02.800 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:37:02.800 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:37:02.800 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:37:02.800 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:37:02.800 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:02.801 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:37:02.801 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:02.801 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:37:02.801 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:02.801 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:37:02.801 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:37:02.801 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:37:02.801 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:37:02.801 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:02.801 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:37:02.801 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:37:02.801 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:02.801 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:02.801 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:02.801 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:37:02.802 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:37:02.802 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:02.802 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:02.802 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:02.802 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:02.802 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:37:02.802 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:37:02.802 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:37:02.802 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:37:02.802 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:37:02.802 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:02.802 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:02.802 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:37:02.802 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:37:02.803 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:37:02.803 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:37:02.803 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:02.803 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:37:02.803 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:37:02.803 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:02.803 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:02.803 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:02.803 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:37:02.803 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:37:02.803 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:02.803 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:02.803 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:02.803 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:02.804 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:37:02.804 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:37:02.804 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:37:02.804 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:37:02.804 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:37:02.804 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:02.804 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:02.804 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:37:02.804 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:37:02.804 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:37:02.804 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:37:02.804 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:02.804 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:37:02.804 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:37:02.804 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:02.805 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:02.805 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:02.805 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:37:02.805 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:37:02.805 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:02.805 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:02.805 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:02.805 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:02.805 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:37:02.805 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:37:02.805 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:37:02.805 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:37:02.805 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:37:02.805 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:02.806 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:02.806 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:02.806 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:37:02.806 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:37:02.806 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:02.806 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:37:02.806 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:37:02.806 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:37:02.806 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:02.806 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:02.806 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:37:02.806 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:02.806 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:37:02.806 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:37:02.807 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:37:02.807 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:02.807 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:02.807 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:02.807 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:37:02.807 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:37:02.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.784+0000 7fe877a37640 1 -- 172.21.15.37:0/3725827689 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe85c035e60 msgr2=0x7fe85c038320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:02.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.784+0000 7fe877a37640 1 --2- 172.21.15.37:0/3725827689 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe85c035e60 0x7fe85c038320 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fe870138610 tx=0x7fe86c009270 comp rx=0 tx=0).stop 2024-04-26T00:37:02.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.784+0000 7fe877a37640 10 monclient: shutdown 2024-04-26T00:37:02.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.784+0000 7fe877a37640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:37:02.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.784+0000 7fe877a37640 1 -- 172.21.15.37:0/3725827689 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe8701356f0 msgr2=0x7fe87012ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:02.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.784+0000 7fe877a37640 1 --2- 172.21.15.37:0/3725827689 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe8701356f0 0x7fe87012ea30 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe86800b390 tx=0x7fe86800b850 comp rx=0 tx=0).stop 2024-04-26T00:37:02.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.784+0000 7fe875fd4640 1 -- 172.21.15.37:0/3725827689 reap_dead start 2024-04-26T00:37:02.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.788+0000 7fe875fd4640 1 -- 172.21.15.37:0/3725827689 reap_dead start 2024-04-26T00:37:02.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.788+0000 7fe877a37640 1 -- 172.21.15.37:0/3725827689 shutdown_connections 2024-04-26T00:37:02.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.788+0000 7fe877a37640 1 -- 172.21.15.37:0/3725827689 >> 172.21.15.37:0/3725827689 conn(0x7fe8700a24c0 msgr2=0x7fe8700a4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:37:02.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.788+0000 7fe877a37640 1 -- 172.21.15.37:0/3725827689 shutdown_connections 2024-04-26T00:37:02.809 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:02.788+0000 7fe877a37640 1 -- 172.21.15.37:0/3725827689 wait complete. 2024-04-26T00:37:02.809 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:37:02.809 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-04-26T00:37:03.177 INFO:tasks.rados.rados.0.smithi045.stdout:2940: done (12 left) 2024-04-26T00:37:03.177 INFO:tasks.rados.rados.0.smithi045.stdout:2941: snap_remove snap 315 2024-04-26T00:37:03.565 INFO:tasks.rados.rados.0.smithi045.stdout:2941: done (12 left) 2024-04-26T00:37:03.565 INFO:tasks.rados.rados.0.smithi045.stdout:2942: copy_from oid 47 from oid 37 current snap is 318 2024-04-26T00:37:03.565 INFO:tasks.rados.rados.0.smithi045.stdout:2925: finishing rollback tid 0 to smithi04526579-45 2024-04-26T00:37:03.565 INFO:tasks.rados.rados.0.smithi045.stdout:2925: finishing rollback tid 1 to smithi04526579-45 2024-04-26T00:37:03.565 INFO:tasks.rados.rados.0.smithi045.stdout:2927: expect (ObjNum 945 snap 292 seq_num 945) 2024-04-26T00:37:03.862 INFO:tasks.rados.rados.0.smithi045.stdout:2934: expect (ObjNum 1037 snap 313 seq_num 1037) 2024-04-26T00:37:04.020 INFO:tasks.rados.rados.0.smithi045.stdout:2926: finishing copy_from to smithi04526579-40 2024-04-26T00:37:04.020 INFO:tasks.rados.rados.0.smithi045.stdout:2926: got expected ENOENT (src dne) 2024-04-26T00:37:04.020 INFO:tasks.rados.rados.0.smithi045.stdout:2926: finishing copy_from racing read to smithi04526579-40 2024-04-26T00:37:04.020 INFO:tasks.rados.rados.0.smithi045.stdout:2925: finishing rollback tid 2 to smithi04526579-45 2024-04-26T00:37:04.020 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 2185 (ObjNum 999 snap 303 seq_num 999) dirty exists 2024-04-26T00:37:04.020 INFO:tasks.rados.rados.0.smithi045.stdout:2932: finishing rollback tid 0 to smithi04526579-23 2024-04-26T00:37:04.020 INFO:tasks.rados.rados.0.smithi045.stdout:2932: finishing rollback tid 1 to smithi04526579-23 2024-04-26T00:37:04.020 INFO:tasks.rados.rados.0.smithi045.stdout:2925: done (12 left) 2024-04-26T00:37:04.020 INFO:tasks.rados.rados.0.smithi045.stdout:2926: done (11 left) 2024-04-26T00:37:04.021 INFO:tasks.rados.rados.0.smithi045.stdout:2927: done (10 left) 2024-04-26T00:37:04.021 INFO:tasks.rados.rados.0.smithi045.stdout:2934: done (9 left) 2024-04-26T00:37:04.021 INFO:tasks.rados.rados.0.smithi045.stdout:2943: write (excl) oid 20 current snap is 318 2024-04-26T00:37:04.021 INFO:tasks.rados.rados.0.smithi045.stdout:2932: finishing rollback tid 2 to smithi04526579-23 2024-04-26T00:37:04.021 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 2010 (ObjNum 945 snap 292 seq_num 945) dirty exists 2024-04-26T00:37:04.021 INFO:tasks.rados.rados.0.smithi045.stdout:2937: finishing write tid 1 to smithi04526579-43 2024-04-26T00:37:04.021 INFO:tasks.rados.rados.0.smithi045.stdout:2935: finishing copy_from to smithi04526579-3 2024-04-26T00:37:04.021 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 2310 (ObjNum 955 snap 295 seq_num 955) dirty exists 2024-04-26T00:37:04.021 INFO:tasks.rados.rados.0.smithi045.stdout:2937: finishing write tid 2 to smithi04526579-43 2024-04-26T00:37:04.021 INFO:tasks.rados.rados.0.smithi045.stdout:2933: finishing copy_from to smithi04526579-49 2024-04-26T00:37:04.021 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 2193 (ObjNum 1043 snap 316 seq_num 1043) dirty exists 2024-04-26T00:37:04.021 INFO:tasks.rados.rados.0.smithi045.stdout:2943: seq_num 1054 ranges {514578=569160,1677668=414031,2577952=641878} 2024-04-26T00:37:04.038 INFO:tasks.rados.rados.0.smithi045.stdout:2943: writing smithi04526579-20 from 514578 to 1083738 tid 1 2024-04-26T00:37:04.054 INFO:tasks.rados.rados.0.smithi045.stdout:2943: writing smithi04526579-20 from 1677668 to 2091699 tid 2 2024-04-26T00:37:04.073 INFO:tasks.rados.rados.0.smithi045.stdout:2943: writing smithi04526579-20 from 2577952 to 3219830 tid 3 2024-04-26T00:37:04.073 INFO:tasks.rados.rados.0.smithi045.stdout:2932: done (9 left) 2024-04-26T00:37:04.073 INFO:tasks.rados.rados.0.smithi045.stdout:2933: done (8 left) 2024-04-26T00:37:04.073 INFO:tasks.rados.rados.0.smithi045.stdout:2935: done (7 left) 2024-04-26T00:37:04.073 INFO:tasks.rados.rados.0.smithi045.stdout:2944: write oid 19 current snap is 318 2024-04-26T00:37:04.073 INFO:tasks.rados.rados.0.smithi045.stdout:2936: finishing copy_from racing read to smithi04526579-36 2024-04-26T00:37:04.073 INFO:tasks.rados.rados.0.smithi045.stdout:2928: finishing copy_from racing read to smithi04526579-22 2024-04-26T00:37:04.073 INFO:tasks.rados.rados.0.smithi045.stdout:2928: finishing copy_from to smithi04526579-22 2024-04-26T00:37:04.074 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 2011 (ObjNum 987 snap 302 seq_num 987) dirty exists 2024-04-26T00:37:04.074 INFO:tasks.rados.rados.0.smithi045.stdout:2936: finishing copy_from to smithi04526579-36 2024-04-26T00:37:04.074 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 2401 (ObjNum 1051 snap 317 seq_num 1051) dirty exists 2024-04-26T00:37:04.074 INFO:tasks.rados.rados.0.smithi045.stdout:2937: finishing write tid 3 to smithi04526579-43 2024-04-26T00:37:04.074 INFO:tasks.rados.rados.0.smithi045.stdout:2937: finishing write tid 4 to smithi04526579-43 2024-04-26T00:37:04.074 INFO:tasks.rados.rados.0.smithi045.stdout:2937: finishing write tid 5 to smithi04526579-43 2024-04-26T00:37:04.074 INFO:tasks.rados.rados.0.smithi045.stdout:2937: oid 43 updating version 0 to 1988 2024-04-26T00:37:04.074 INFO:tasks.rados.rados.0.smithi045.stdout:2937: oid 43 updating version 1988 to 1991 2024-04-26T00:37:04.074 INFO:tasks.rados.rados.0.smithi045.stdout:2937: oid 43 version 1991 is already newer than 1990 2024-04-26T00:37:04.074 INFO:tasks.rados.rados.0.smithi045.stdout:2937: oid 43 version 1991 is already newer than 1989 2024-04-26T00:37:04.074 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 1991 (ObjNum 1053 snap 318 seq_num 1053) dirty exists 2024-04-26T00:37:04.074 INFO:tasks.rados.rados.0.smithi045.stdout:2937: left oid 43 (ObjNum 1053 snap 318 seq_num 1053) 2024-04-26T00:37:04.074 INFO:tasks.rados.rados.0.smithi045.stdout:2939: expect (ObjNum 998 snap 303 seq_num 998) 2024-04-26T00:37:04.243 INFO:tasks.rados.rados.0.smithi045.stdout:2938: finishing rollback tid 1 to smithi04526579-25 2024-04-26T00:37:04.243 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 0 (ObjNum 65 snap 0 seq_num 0) dirty dne 2024-04-26T00:37:04.243 INFO:tasks.rados.rados.0.smithi045.stdout:2942: finishing copy_from racing read to smithi04526579-47 2024-04-26T00:37:04.243 INFO:tasks.rados.rados.0.smithi045.stdout:2942: finishing copy_from to smithi04526579-47 2024-04-26T00:37:04.244 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 2302 (ObjNum 939 snap 290 seq_num 939) dirty exists 2024-04-26T00:37:04.244 INFO:tasks.rados.rados.0.smithi045.stdout:2943: finishing write tid 1 to smithi04526579-20 2024-04-26T00:37:04.244 INFO:tasks.rados.rados.0.smithi045.stdout:2943: finishing write tid 2 to smithi04526579-20 2024-04-26T00:37:04.244 INFO:tasks.rados.rados.0.smithi045.stdout:2943: finishing write tid 3 to smithi04526579-20 2024-04-26T00:37:04.244 INFO:tasks.rados.rados.0.smithi045.stdout:2943: finishing write tid 4 to smithi04526579-20 2024-04-26T00:37:04.244 INFO:tasks.rados.rados.0.smithi045.stdout:2943: finishing write tid 5 to smithi04526579-20 2024-04-26T00:37:04.244 INFO:tasks.rados.rados.0.smithi045.stdout:2943: oid 20 updating version 0 to 2203 2024-04-26T00:37:04.244 INFO:tasks.rados.rados.0.smithi045.stdout:2943: oid 20 version 2203 is already newer than 2200 2024-04-26T00:37:04.244 INFO:tasks.rados.rados.0.smithi045.stdout:2943: oid 20 version 2203 is already newer than 2202 2024-04-26T00:37:04.244 INFO:tasks.rados.rados.0.smithi045.stdout:2943: oid 20 version 2203 is already newer than 2201 2024-04-26T00:37:04.244 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 2203 (ObjNum 1054 snap 318 seq_num 1054) dirty exists 2024-04-26T00:37:04.244 INFO:tasks.rados.rados.0.smithi045.stdout:2943: left oid 20 (ObjNum 1054 snap 318 seq_num 1054) 2024-04-26T00:37:04.244 INFO:tasks.rados.rados.0.smithi045.stdout:2944: seq_num 1055 ranges {531385=644409,1736150=557689,3005375=569503} 2024-04-26T00:37:04.262 INFO:tasks.rados.rados.0.smithi045.stdout:2944: writing smithi04526579-19 from 531385 to 1175794 tid 1 2024-04-26T00:37:04.280 INFO:tasks.rados.rados.0.smithi045.stdout:2944: writing smithi04526579-19 from 1736150 to 2293839 tid 2 2024-04-26T00:37:04.301 INFO:tasks.rados.rados.0.smithi045.stdout:2944: writing smithi04526579-19 from 3005375 to 3574878 tid 3 2024-04-26T00:37:04.301 INFO:tasks.rados.rados.0.smithi045.stdout:2928: done (7 left) 2024-04-26T00:37:04.301 INFO:tasks.rados.rados.0.smithi045.stdout:2936: done (6 left) 2024-04-26T00:37:04.302 INFO:tasks.rados.rados.0.smithi045.stdout:2937: done (5 left) 2024-04-26T00:37:04.302 INFO:tasks.rados.rados.0.smithi045.stdout:2938: done (4 left) 2024-04-26T00:37:04.302 INFO:tasks.rados.rados.0.smithi045.stdout:2939: done (3 left) 2024-04-26T00:37:04.302 INFO:tasks.rados.rados.0.smithi045.stdout:2942: done (2 left) 2024-04-26T00:37:04.302 INFO:tasks.rados.rados.0.smithi045.stdout:2943: done (1 left) 2024-04-26T00:37:04.302 INFO:tasks.rados.rados.0.smithi045.stdout:2945: snap_remove snap 316 2024-04-26T00:37:05.189 INFO:tasks.rados.rados.0.smithi045.stdout:2945: done (1 left) 2024-04-26T00:37:05.189 INFO:tasks.rados.rados.0.smithi045.stdout:2946: delete oid 43 current snap is 318 2024-04-26T00:37:05.189 INFO:tasks.rados.rados.0.smithi045.stdout:2944: finishing write tid 1 to smithi04526579-19 2024-04-26T00:37:05.189 INFO:tasks.rados.rados.0.smithi045.stdout:2944: finishing write tid 2 to smithi04526579-19 2024-04-26T00:37:05.189 INFO:tasks.rados.rados.0.smithi045.stdout:2944: finishing write tid 3 to smithi04526579-19 2024-04-26T00:37:05.189 INFO:tasks.rados.rados.0.smithi045.stdout:2944: finishing write tid 4 to smithi04526579-19 2024-04-26T00:37:05.189 INFO:tasks.rados.rados.0.smithi045.stdout:2944: finishing write tid 5 to smithi04526579-19 2024-04-26T00:37:05.189 INFO:tasks.rados.rados.0.smithi045.stdout:2944: oid 19 updating version 0 to 2358 2024-04-26T00:37:05.189 INFO:tasks.rados.rados.0.smithi045.stdout:2944: oid 19 updating version 2358 to 2360 2024-04-26T00:37:05.189 INFO:tasks.rados.rados.0.smithi045.stdout:2944: oid 19 version 2360 is already newer than 2359 2024-04-26T00:37:05.189 INFO:tasks.rados.rados.0.smithi045.stdout:2944: oid 19 updating version 2360 to 2361 2024-04-26T00:37:05.190 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 2361 (ObjNum 1055 snap 318 seq_num 1055) dirty exists 2024-04-26T00:37:05.190 INFO:tasks.rados.rados.0.smithi045.stdout:2944: left oid 19 (ObjNum 1055 snap 318 seq_num 1055) 2024-04-26T00:37:05.212 INFO:tasks.rados.rados.0.smithi045.stdout:2944: done (1 left) 2024-04-26T00:37:05.212 INFO:tasks.rados.rados.0.smithi045.stdout:2946: done (0 left) 2024-04-26T00:37:05.213 INFO:tasks.rados.rados.0.smithi045.stdout:2947: read oid 19 snap -1 2024-04-26T00:37:05.213 INFO:tasks.rados.rados.0.smithi045.stdout:2947: expect (ObjNum 1055 snap 318 seq_num 1055) 2024-04-26T00:37:05.213 INFO:tasks.rados.rados.0.smithi045.stdout:2948: write (excl) oid 31 current snap is 318 2024-04-26T00:37:05.213 INFO:tasks.rados.rados.0.smithi045.stdout:2948: seq_num 1057 ranges {564999=511260,1853114=561358,3060221=592047} 2024-04-26T00:37:05.228 INFO:tasks.rados.rados.0.smithi045.stdout:2948: writing smithi04526579-31 from 564999 to 1076259 tid 1 2024-04-26T00:37:05.246 INFO:tasks.rados.rados.0.smithi045.stdout:2948: writing smithi04526579-31 from 1853114 to 2414472 tid 2 2024-04-26T00:37:05.262 INFO:tasks.rados.rados.0.smithi045.stdout:2948: writing smithi04526579-31 from 3060221 to 3652268 tid 3 2024-04-26T00:37:05.262 INFO:tasks.rados.rados.0.smithi045.stdout:2949: write oid 22 current snap is 318 2024-04-26T00:37:05.262 INFO:tasks.rados.rados.0.smithi045.stdout:2949: seq_num 1058 ranges {581806=586509,1911596=421369,3087644=403319} 2024-04-26T00:37:05.275 INFO:tasks.rados.rados.0.smithi045.stdout:2949: writing smithi04526579-22 from 581806 to 1168315 tid 1 2024-04-26T00:37:05.286 INFO:tasks.rados.rados.0.smithi045.stdout:2949: writing smithi04526579-22 from 1911596 to 2332965 tid 2 2024-04-26T00:37:05.295 INFO:tasks.rados.rados.0.smithi045.stdout:2949: writing smithi04526579-22 from 3087644 to 3490963 tid 3 2024-04-26T00:37:05.295 INFO:tasks.rados.rados.0.smithi045.stdout:2950: write (excl) oid 2 current snap is 318 2024-04-26T00:37:05.295 INFO:tasks.rados.rados.0.smithi045.stdout:2950: seq_num 1059 ranges {598613=661758,1970078=565027,3115067=614591} 2024-04-26T00:37:05.304 INFO:tasks.rados.rados.0.smithi045.stdout:2950: writing smithi04526579-2 from 598613 to 1260371 tid 1 2024-04-26T00:37:05.313 INFO:tasks.rados.rados.0.smithi045.stdout:2950: writing smithi04526579-2 from 1970078 to 2535105 tid 2 2024-04-26T00:37:05.321 INFO:tasks.rados.rados.0.smithi045.stdout:2950: writing smithi04526579-2 from 3115067 to 3729658 tid 3 2024-04-26T00:37:05.321 INFO:tasks.rados.rados.0.smithi045.stdout:2951: write (excl) oid 6 current snap is 318 2024-04-26T00:37:05.321 INFO:tasks.rados.rados.0.smithi045.stdout:2951: seq_num 1060 ranges {615420=737007,1912207=425038,3026137=542216} 2024-04-26T00:37:05.328 INFO:tasks.rados.rados.0.smithi045.stdout:2951: writing smithi04526579-6 from 615420 to 1352427 tid 1 2024-04-26T00:37:05.334 INFO:tasks.rados.rados.0.smithi045.stdout:2951: writing smithi04526579-6 from 1912207 to 2337245 tid 2 2024-04-26T00:37:05.342 INFO:tasks.rados.rados.0.smithi045.stdout:2951: writing smithi04526579-6 from 3026137 to 3568353 tid 3 2024-04-26T00:37:05.342 INFO:tasks.rados.rados.0.smithi045.stdout:2952: read oid 1 snap -1 2024-04-26T00:37:05.342 INFO:tasks.rados.rados.0.smithi045.stdout:2952: expect (ObjNum 1017 snap 310 seq_num 1017) 2024-04-26T00:37:05.342 INFO:tasks.rados.rados.0.smithi045.stdout:2953: write oid 18 current snap is 318 2024-04-26T00:37:05.342 INFO:tasks.rados.rados.0.smithi045.stdout:2947: expect (ObjNum 1055 snap 318 seq_num 1055) 2024-04-26T00:37:05.512 INFO:tasks.rados.rados.0.smithi045.stdout:2948: finishing write tid 1 to smithi04526579-31 2024-04-26T00:37:05.512 INFO:tasks.rados.rados.0.smithi045.stdout:2948: finishing write tid 2 to smithi04526579-31 2024-04-26T00:37:05.512 INFO:tasks.rados.rados.0.smithi045.stdout:2948: finishing write tid 3 to smithi04526579-31 2024-04-26T00:37:05.512 INFO:tasks.rados.rados.0.smithi045.stdout:2948: finishing write tid 4 to smithi04526579-31 2024-04-26T00:37:05.512 INFO:tasks.rados.rados.0.smithi045.stdout:2948: finishing write tid 5 to smithi04526579-31 2024-04-26T00:37:05.512 INFO:tasks.rados.rados.0.smithi045.stdout:2948: oid 31 updating version 0 to 2028 2024-04-26T00:37:05.512 INFO:tasks.rados.rados.0.smithi045.stdout:2948: oid 31 version 2028 is already newer than 2027 2024-04-26T00:37:05.512 INFO:tasks.rados.rados.0.smithi045.stdout:2948: oid 31 updating version 2028 to 2029 2024-04-26T00:37:05.512 INFO:tasks.rados.rados.0.smithi045.stdout:2948: oid 31 version 2029 is already newer than 2026 2024-04-26T00:37:05.512 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 2029 (ObjNum 1057 snap 318 seq_num 1057) dirty exists 2024-04-26T00:37:05.513 INFO:tasks.rados.rados.0.smithi045.stdout:2948: left oid 31 (ObjNum 1057 snap 318 seq_num 1057) 2024-04-26T00:37:05.513 INFO:tasks.rados.rados.0.smithi045.stdout:2949: finishing write tid 1 to smithi04526579-22 2024-04-26T00:37:05.513 INFO:tasks.rados.rados.0.smithi045.stdout:2949: finishing write tid 2 to smithi04526579-22 2024-04-26T00:37:05.513 INFO:tasks.rados.rados.0.smithi045.stdout:2949: finishing write tid 3 to smithi04526579-22 2024-04-26T00:37:05.513 INFO:tasks.rados.rados.0.smithi045.stdout:2949: finishing write tid 4 to smithi04526579-22 2024-04-26T00:37:05.513 INFO:tasks.rados.rados.0.smithi045.stdout:2949: finishing write tid 5 to smithi04526579-22 2024-04-26T00:37:05.513 INFO:tasks.rados.rados.0.smithi045.stdout:2949: oid 22 updating version 0 to 2402 2024-04-26T00:37:05.513 INFO:tasks.rados.rados.0.smithi045.stdout:2949: oid 22 updating version 2402 to 2405 2024-04-26T00:37:05.513 INFO:tasks.rados.rados.0.smithi045.stdout:2949: oid 22 version 2405 is already newer than 2404 2024-04-26T00:37:05.513 INFO:tasks.rados.rados.0.smithi045.stdout:2949: oid 22 version 2405 is already newer than 2403 2024-04-26T00:37:05.513 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 2405 (ObjNum 1058 snap 318 seq_num 1058) dirty exists 2024-04-26T00:37:05.513 INFO:tasks.rados.rados.0.smithi045.stdout:2949: left oid 22 (ObjNum 1058 snap 318 seq_num 1058) 2024-04-26T00:37:05.513 INFO:tasks.rados.rados.0.smithi045.stdout:2950: finishing write tid 1 to smithi04526579-2 2024-04-26T00:37:05.514 INFO:tasks.rados.rados.0.smithi045.stdout:2950: finishing write tid 2 to smithi04526579-2 2024-04-26T00:37:05.514 INFO:tasks.rados.rados.0.smithi045.stdout:2950: finishing write tid 3 to smithi04526579-2 2024-04-26T00:37:05.514 INFO:tasks.rados.rados.0.smithi045.stdout:2950: finishing write tid 4 to smithi04526579-2 2024-04-26T00:37:05.514 INFO:tasks.rados.rados.0.smithi045.stdout:2953: seq_num 1061 ranges {632227=412256,1570689=568696,2653560=753488} 2024-04-26T00:37:05.517 INFO:tasks.rados.rados.0.smithi045.stdout:2953: writing smithi04526579-18 from 632227 to 1044483 tid 1 2024-04-26T00:37:05.524 INFO:tasks.rados.rados.0.smithi045.stdout:2953: writing smithi04526579-18 from 1570689 to 2139385 tid 2 2024-04-26T00:37:05.531 INFO:tasks.rados.rados.0.smithi045.stdout:2953: writing smithi04526579-18 from 2653560 to 3407048 tid 3 2024-04-26T00:37:05.531 INFO:tasks.rados.rados.0.smithi045.stdout:2947: done (6 left) 2024-04-26T00:37:05.531 INFO:tasks.rados.rados.0.smithi045.stdout:2948: done (5 left) 2024-04-26T00:37:05.531 INFO:tasks.rados.rados.0.smithi045.stdout:2949: done (4 left) 2024-04-26T00:37:05.531 INFO:tasks.rados.rados.0.smithi045.stdout:2954: write (excl) oid 45 current snap is 318 2024-04-26T00:37:05.531 INFO:tasks.rados.rados.0.smithi045.stdout:2954: seq_num 1062 ranges {649034=487505,1629171=428707,2680983=564760,3798768=50266} 2024-04-26T00:37:05.537 INFO:tasks.rados.rados.0.smithi045.stdout:2954: writing smithi04526579-45 from 649034 to 1136539 tid 1 2024-04-26T00:37:05.543 INFO:tasks.rados.rados.0.smithi045.stdout:2954: writing smithi04526579-45 from 1629171 to 2057878 tid 2 2024-04-26T00:37:05.549 INFO:tasks.rados.rados.0.smithi045.stdout:2954: writing smithi04526579-45 from 2680983 to 3245743 tid 3 2024-04-26T00:37:05.554 INFO:tasks.rados.rados.0.smithi045.stdout:2954: writing smithi04526579-45 from 3798768 to 3849034 tid 4 2024-04-26T00:37:05.554 INFO:tasks.rados.rados.0.smithi045.stdout:2955: read oid 12 snap 314 2024-04-26T00:37:05.554 INFO:tasks.rados.rados.0.smithi045.stdout:2955: expect (ObjNum 945 snap 292 seq_num 945) 2024-04-26T00:37:05.554 INFO:tasks.rados.rados.0.smithi045.stdout:2956: rollback oid 17 current snap is 318 2024-04-26T00:37:05.554 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 17 to 314 2024-04-26T00:37:05.554 INFO:tasks.rados.rados.0.smithi045.stdout:2957: delete oid 41 current snap is 318 2024-04-26T00:37:05.555 INFO:tasks.rados.rados.0.smithi045.stdout:2950: finishing write tid 5 to smithi04526579-2 2024-04-26T00:37:05.555 INFO:tasks.rados.rados.0.smithi045.stdout:2950: oid 2 updating version 0 to 2030 2024-04-26T00:37:05.555 INFO:tasks.rados.rados.0.smithi045.stdout:2950: oid 2 updating version 2030 to 2033 2024-04-26T00:37:05.555 INFO:tasks.rados.rados.0.smithi045.stdout:2950: oid 2 version 2033 is already newer than 2032 2024-04-26T00:37:05.555 INFO:tasks.rados.rados.0.smithi045.stdout:2950: oid 2 version 2033 is already newer than 2031 2024-04-26T00:37:05.555 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 2033 (ObjNum 1059 snap 318 seq_num 1059) dirty exists 2024-04-26T00:37:05.555 INFO:tasks.rados.rados.0.smithi045.stdout:2950: left oid 2 (ObjNum 1059 snap 318 seq_num 1059) 2024-04-26T00:37:05.555 INFO:tasks.rados.rados.0.smithi045.stdout:2951: finishing write tid 1 to smithi04526579-6 2024-04-26T00:37:05.555 INFO:tasks.rados.rados.0.smithi045.stdout:2951: finishing write tid 2 to smithi04526579-6 2024-04-26T00:37:05.555 INFO:tasks.rados.rados.0.smithi045.stdout:2951: finishing write tid 3 to smithi04526579-6 2024-04-26T00:37:05.556 INFO:tasks.rados.rados.0.smithi045.stdout:2951: finishing write tid 4 to smithi04526579-6 2024-04-26T00:37:05.556 INFO:tasks.rados.rados.0.smithi045.stdout:2951: finishing write tid 5 to smithi04526579-6 2024-04-26T00:37:05.556 INFO:tasks.rados.rados.0.smithi045.stdout:2951: oid 6 updating version 0 to 1988 2024-04-26T00:37:05.556 INFO:tasks.rados.rados.0.smithi045.stdout:2951: oid 6 version 1988 is already newer than 1987 2024-04-26T00:37:05.556 INFO:tasks.rados.rados.0.smithi045.stdout:2951: oid 6 updating version 1988 to 1990 2024-04-26T00:37:05.556 INFO:tasks.rados.rados.0.smithi045.stdout:2951: oid 6 version 1990 is already newer than 1989 2024-04-26T00:37:05.556 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 1990 (ObjNum 1060 snap 318 seq_num 1060) dirty exists 2024-04-26T00:37:05.556 INFO:tasks.rados.rados.0.smithi045.stdout:2951: left oid 6 (ObjNum 1060 snap 318 seq_num 1060) 2024-04-26T00:37:05.556 INFO:tasks.rados.rados.0.smithi045.stdout:2952: expect (ObjNum 1017 snap 310 seq_num 1017) 2024-04-26T00:37:05.738 INFO:tasks.rados.rados.0.smithi045.stdout:2953: finishing write tid 1 to smithi04526579-18 2024-04-26T00:37:05.738 INFO:tasks.rados.rados.0.smithi045.stdout:2953: finishing write tid 2 to smithi04526579-18 2024-04-26T00:37:05.738 INFO:tasks.rados.rados.0.smithi045.stdout:2953: finishing write tid 3 to smithi04526579-18 2024-04-26T00:37:05.738 INFO:tasks.rados.rados.0.smithi045.stdout:2953: finishing write tid 4 to smithi04526579-18 2024-04-26T00:37:05.738 INFO:tasks.rados.rados.0.smithi045.stdout:2953: finishing write tid 5 to smithi04526579-18 2024-04-26T00:37:05.738 INFO:tasks.rados.rados.0.smithi045.stdout:2953: oid 18 updating version 0 to 2410 2024-04-26T00:37:05.739 INFO:tasks.rados.rados.0.smithi045.stdout:2953: oid 18 updating version 2410 to 2411 2024-04-26T00:37:05.739 INFO:tasks.rados.rados.0.smithi045.stdout:2953: oid 18 version 2411 is already newer than 2409 2024-04-26T00:37:05.739 INFO:tasks.rados.rados.0.smithi045.stdout:2953: oid 18 updating version 2411 to 2412 2024-04-26T00:37:05.739 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 2412 (ObjNum 1061 snap 318 seq_num 1061) dirty exists 2024-04-26T00:37:05.739 INFO:tasks.rados.rados.0.smithi045.stdout:2953: left oid 18 (ObjNum 1061 snap 318 seq_num 1061) 2024-04-26T00:37:05.739 INFO:tasks.rados.rados.0.smithi045.stdout:2954: finishing write tid 1 to smithi04526579-45 2024-04-26T00:37:05.739 INFO:tasks.rados.rados.0.smithi045.stdout:2954: finishing write tid 2 to smithi04526579-45 2024-04-26T00:37:05.739 INFO:tasks.rados.rados.0.smithi045.stdout:2954: finishing write tid 3 to smithi04526579-45 2024-04-26T00:37:05.739 INFO:tasks.rados.rados.0.smithi045.stdout:2954: finishing write tid 4 to smithi04526579-45 2024-04-26T00:37:05.739 INFO:tasks.rados.rados.0.smithi045.stdout:2954: finishing write tid 5 to smithi04526579-45 2024-04-26T00:37:05.739 INFO:tasks.rados.rados.0.smithi045.stdout:2954: finishing write tid 6 to smithi04526579-45 2024-04-26T00:37:05.739 INFO:tasks.rados.rados.0.smithi045.stdout:2954: oid 45 updating version 0 to 2187 2024-04-26T00:37:05.739 INFO:tasks.rados.rados.0.smithi045.stdout:2954: oid 45 updating version 2187 to 2189 2024-04-26T00:37:05.739 INFO:tasks.rados.rados.0.smithi045.stdout:2954: oid 45 updating version 2189 to 2190 2024-04-26T00:37:05.739 INFO:tasks.rados.rados.0.smithi045.stdout:2954: oid 45 version 2190 is already newer than 2186 2024-04-26T00:37:05.740 INFO:tasks.rados.rados.0.smithi045.stdout:2954: oid 45 version 2190 is already newer than 2188 2024-04-26T00:37:05.740 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 2190 (ObjNum 1062 snap 318 seq_num 1062) dirty exists 2024-04-26T00:37:05.740 INFO:tasks.rados.rados.0.smithi045.stdout:2954: left oid 45 (ObjNum 1062 snap 318 seq_num 1062) 2024-04-26T00:37:05.740 INFO:tasks.rados.rados.0.smithi045.stdout:2955: expect (ObjNum 945 snap 292 seq_num 945) 2024-04-26T00:37:05.994 INFO:tasks.rados.rados.0.smithi045.stdout:2956: finishing rollback tid 0 to smithi04526579-17 2024-04-26T00:37:05.994 INFO:tasks.rados.rados.0.smithi045.stdout:2956: finishing rollback tid 1 to smithi04526579-17 2024-04-26T00:37:05.994 INFO:tasks.rados.rados.0.smithi045.stdout:2956: finishing rollback tid 2 to smithi04526579-17 2024-04-26T00:37:05.994 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 1995 (ObjNum 1033 snap 312 seq_num 1033) dirty exists 2024-04-26T00:37:05.994 INFO:tasks.rados.rados.0.smithi045.stdout:2950: done (7 left) 2024-04-26T00:37:05.994 INFO:tasks.rados.rados.0.smithi045.stdout:2951: done (6 left) 2024-04-26T00:37:05.994 INFO:tasks.rados.rados.0.smithi045.stdout:2952: done (5 left) 2024-04-26T00:37:05.994 INFO:tasks.rados.rados.0.smithi045.stdout:2953: done (4 left) 2024-04-26T00:37:05.995 INFO:tasks.rados.rados.0.smithi045.stdout:2954: done (3 left) 2024-04-26T00:37:05.995 INFO:tasks.rados.rados.0.smithi045.stdout:2955: done (2 left) 2024-04-26T00:37:05.995 INFO:tasks.rados.rados.0.smithi045.stdout:2956: done (1 left) 2024-04-26T00:37:05.995 INFO:tasks.rados.rados.0.smithi045.stdout:2957: done (0 left) 2024-04-26T00:37:05.995 INFO:tasks.rados.rados.0.smithi045.stdout:2958: read oid 40 snap -1 2024-04-26T00:37:05.995 INFO:tasks.rados.rados.0.smithi045.stdout:2958: expect (ObjNum 1023 snap 310 seq_num 1023) 2024-04-26T00:37:05.995 INFO:tasks.rados.rados.0.smithi045.stdout:2959: snap_create 2024-04-26T00:37:06.000 INFO:tasks.rados.rados.0.smithi045.stdout:2958: expect (ObjNum 1023 snap 310 seq_num 1023) 2024-04-26T00:37:06.246 INFO:tasks.rados.rados.0.smithi045.stdout:2958: done (1 left) 2024-04-26T00:37:06.247 INFO:tasks.rados.rados.0.smithi045.stdout:2959: done (0 left) 2024-04-26T00:37:06.247 INFO:tasks.rados.rados.0.smithi045.stdout:2960: snap_create 2024-04-26T00:37:07.211 INFO:tasks.rados.rados.0.smithi045.stdout:2960: done (0 left) 2024-04-26T00:37:07.211 INFO:tasks.rados.rados.0.smithi045.stdout:2961: read oid 22 snap -1 2024-04-26T00:37:07.211 INFO:tasks.rados.rados.0.smithi045.stdout:2961: expect (ObjNum 1058 snap 318 seq_num 1058) 2024-04-26T00:37:07.211 INFO:tasks.rados.rados.0.smithi045.stdout:2962: copy_from oid 32 from oid 49 current snap is 320 2024-04-26T00:37:07.212 INFO:tasks.rados.rados.0.smithi045.stdout:2963: read oid 8 snap 318 2024-04-26T00:37:07.212 INFO:tasks.rados.rados.0.smithi045.stdout:2963: expect (ObjNum 990 snap 303 seq_num 990) 2024-04-26T00:37:07.212 INFO:tasks.rados.rados.0.smithi045.stdout:2964: write (excl) oid 34 current snap is 320 2024-04-26T00:37:07.212 INFO:tasks.rados.rados.0.smithi045.stdout:2964: seq_num 1064 ranges {682648=638003,2029782=432376,3019476=587304} 2024-04-26T00:37:07.235 INFO:tasks.rados.rados.0.smithi045.stdout:2964: writing smithi04526579-34 from 682648 to 1320651 tid 1 2024-04-26T00:37:07.254 INFO:tasks.rados.rados.0.smithi045.stdout:2964: writing smithi04526579-34 from 2029782 to 2462158 tid 2 2024-04-26T00:37:07.272 INFO:tasks.rados.rados.0.smithi045.stdout:2964: writing smithi04526579-34 from 3019476 to 3606780 tid 3 2024-04-26T00:37:07.272 INFO:tasks.rados.rados.0.smithi045.stdout:2965: read oid 4 snap -1 2024-04-26T00:37:07.272 INFO:tasks.rados.rados.0.smithi045.stdout:2965: expect (ObjNum 1033 snap 312 seq_num 1033) 2024-04-26T00:37:07.272 INFO:tasks.rados.rados.0.smithi045.stdout:2966: snap_remove snap 314 2024-04-26T00:37:08.229 INFO:tasks.rados.rados.0.smithi045.stdout:2966: done (5 left) 2024-04-26T00:37:08.229 INFO:tasks.rados.rados.0.smithi045.stdout:2967: write (excl) oid 44 current snap is 320 2024-04-26T00:37:08.229 INFO:tasks.rados.rados.0.smithi045.stdout:2967: seq_num 1065 ranges {699455=429605,1804617=576034,3163252=514929} 2024-04-26T00:37:08.248 INFO:tasks.rados.rados.0.smithi045.stdout:2967: writing smithi04526579-44 from 699455 to 1129060 tid 1 2024-04-26T00:37:08.268 INFO:tasks.rados.rados.0.smithi045.stdout:2967: writing smithi04526579-44 from 1804617 to 2380651 tid 2 2024-04-26T00:37:08.287 INFO:tasks.rados.rados.0.smithi045.stdout:2967: writing smithi04526579-44 from 3163252 to 3678181 tid 3 2024-04-26T00:37:08.287 INFO:tasks.rados.rados.0.smithi045.stdout:2968: snap_create 2024-04-26T00:37:08.287 INFO:tasks.rados.rados.0.smithi045.stdout:2961: expect (ObjNum 1058 snap 318 seq_num 1058) 2024-04-26T00:37:08.508 INFO:tasks.rados.rados.0.smithi045.stdout:2963: expect (ObjNum 990 snap 303 seq_num 990) 2024-04-26T00:37:08.678 INFO:tasks.rados.rados.0.smithi045.stdout:2964: finishing write tid 1 to smithi04526579-34 2024-04-26T00:37:08.678 INFO:tasks.rados.rados.0.smithi045.stdout:2962: finishing copy_from racing read to smithi04526579-32 2024-04-26T00:37:08.679 INFO:tasks.rados.rados.0.smithi045.stdout:2962: finishing copy_from to smithi04526579-32 2024-04-26T00:37:08.679 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 2193 (ObjNum 1043 snap 316 seq_num 1043) dirty exists 2024-04-26T00:37:08.679 INFO:tasks.rados.rados.0.smithi045.stdout:2964: finishing write tid 2 to smithi04526579-34 2024-04-26T00:37:08.679 INFO:tasks.rados.rados.0.smithi045.stdout:2964: finishing write tid 3 to smithi04526579-34 2024-04-26T00:37:08.679 INFO:tasks.rados.rados.0.smithi045.stdout:2964: finishing write tid 4 to smithi04526579-34 2024-04-26T00:37:08.679 INFO:tasks.rados.rados.0.smithi045.stdout:2964: finishing write tid 5 to smithi04526579-34 2024-04-26T00:37:08.679 INFO:tasks.rados.rados.0.smithi045.stdout:2964: oid 34 updating version 0 to 2312 2024-04-26T00:37:08.679 INFO:tasks.rados.rados.0.smithi045.stdout:2964: oid 34 version 2312 is already newer than 2311 2024-04-26T00:37:08.679 INFO:tasks.rados.rados.0.smithi045.stdout:2964: oid 34 updating version 2312 to 2314 2024-04-26T00:37:08.679 INFO:tasks.rados.rados.0.smithi045.stdout:2964: oid 34 version 2314 is already newer than 2313 2024-04-26T00:37:08.679 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 2314 (ObjNum 1064 snap 320 seq_num 1064) dirty exists 2024-04-26T00:37:08.679 INFO:tasks.rados.rados.0.smithi045.stdout:2964: left oid 34 (ObjNum 1064 snap 320 seq_num 1064) 2024-04-26T00:37:08.679 INFO:tasks.rados.rados.0.smithi045.stdout:2965: expect (ObjNum 1033 snap 312 seq_num 1033) 2024-04-26T00:37:08.878 INFO:tasks.rados.rados.0.smithi045.stdout:2967: finishing write tid 1 to smithi04526579-44 2024-04-26T00:37:08.878 INFO:tasks.rados.rados.0.smithi045.stdout:2967: finishing write tid 2 to smithi04526579-44 2024-04-26T00:37:08.878 INFO:tasks.rados.rados.0.smithi045.stdout:2967: finishing write tid 3 to smithi04526579-44 2024-04-26T00:37:08.878 INFO:tasks.rados.rados.0.smithi045.stdout:2967: finishing write tid 4 to smithi04526579-44 2024-04-26T00:37:08.878 INFO:tasks.rados.rados.0.smithi045.stdout:2967: finishing write tid 5 to smithi04526579-44 2024-04-26T00:37:08.878 INFO:tasks.rados.rados.0.smithi045.stdout:2967: oid 44 updating version 0 to 2206 2024-04-26T00:37:08.878 INFO:tasks.rados.rados.0.smithi045.stdout:2967: oid 44 updating version 2206 to 2207 2024-04-26T00:37:08.878 INFO:tasks.rados.rados.0.smithi045.stdout:2967: oid 44 version 2207 is already newer than 2205 2024-04-26T00:37:08.878 INFO:tasks.rados.rados.0.smithi045.stdout:2967: oid 44 version 2207 is already newer than 2204 2024-04-26T00:37:08.879 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 2207 (ObjNum 1065 snap 320 seq_num 1065) dirty exists 2024-04-26T00:37:08.879 INFO:tasks.rados.rados.0.smithi045.stdout:2967: left oid 44 (ObjNum 1065 snap 320 seq_num 1065) 2024-04-26T00:37:08.879 INFO:tasks.rados.rados.0.smithi045.stdout:2961: done (6 left) 2024-04-26T00:37:08.879 INFO:tasks.rados.rados.0.smithi045.stdout:2962: done (5 left) 2024-04-26T00:37:08.879 INFO:tasks.rados.rados.0.smithi045.stdout:2963: done (4 left) 2024-04-26T00:37:08.879 INFO:tasks.rados.rados.0.smithi045.stdout:2964: done (3 left) 2024-04-26T00:37:08.879 INFO:tasks.rados.rados.0.smithi045.stdout:2965: done (2 left) 2024-04-26T00:37:08.879 INFO:tasks.rados.rados.0.smithi045.stdout:2967: done (1 left) 2024-04-26T00:37:08.879 INFO:tasks.rados.rados.0.smithi045.stdout:2968: done (0 left) 2024-04-26T00:37:08.879 INFO:tasks.rados.rados.0.smithi045.stdout:2969: read oid 2 snap -1 2024-04-26T00:37:08.879 INFO:tasks.rados.rados.0.smithi045.stdout:2969: expect (ObjNum 1059 snap 318 seq_num 1059) 2024-04-26T00:37:08.879 INFO:tasks.rados.rados.0.smithi045.stdout:2970: snap_remove snap 318 2024-04-26T00:37:09.572 INFO:tasks.rados.rados.0.smithi045.stdout:2970: done (1 left) 2024-04-26T00:37:09.572 INFO:tasks.rados.rados.0.smithi045.stdout:2971: write (excl) oid 11 current snap is 321 2024-04-26T00:37:09.572 INFO:tasks.rados.rados.0.smithi045.stdout:2971: seq_num 1066 ranges {716262=504854,1863099=719692,3074322=726201} 2024-04-26T00:37:09.591 INFO:tasks.rados.rados.0.smithi045.stdout:2971: writing smithi04526579-11 from 716262 to 1221116 tid 1 2024-04-26T00:37:09.614 INFO:tasks.rados.rados.0.smithi045.stdout:2971: writing smithi04526579-11 from 1863099 to 2582791 tid 2 2024-04-26T00:37:09.631 INFO:tasks.rados.rados.0.smithi045.stdout:2971: writing smithi04526579-11 from 3074322 to 3800523 tid 3 2024-04-26T00:37:09.631 INFO:tasks.rados.rados.0.smithi045.stdout:2972: write (excl) oid 42 current snap is 321 2024-04-26T00:37:09.631 INFO:tasks.rados.rados.0.smithi045.stdout:2972: seq_num 1067 ranges {733069=580103,1921581=579703,3218098=537473} 2024-04-26T00:37:09.645 INFO:tasks.rados.rados.0.smithi045.stdout:2972: writing smithi04526579-42 from 733069 to 1313172 tid 1 2024-04-26T00:37:09.656 INFO:tasks.rados.rados.0.smithi045.stdout:2972: writing smithi04526579-42 from 1921581 to 2501284 tid 2 2024-04-26T00:37:09.667 INFO:tasks.rados.rados.0.smithi045.stdout:2972: writing smithi04526579-42 from 3218098 to 3755571 tid 3 2024-04-26T00:37:09.667 INFO:tasks.rados.rados.0.smithi045.stdout:2973: read oid 5 snap -1 2024-04-26T00:37:09.667 INFO:tasks.rados.rados.0.smithi045.stdout:2973: expect (ObjNum 955 snap 295 seq_num 955) 2024-04-26T00:37:09.667 INFO:tasks.rados.rados.0.smithi045.stdout:2974: read oid 38 snap 320 2024-04-26T00:37:09.667 INFO:tasks.rados.rados.0.smithi045.stdout:2974: expect (ObjNum 1001 snap 304 seq_num 1001) 2024-04-26T00:37:09.667 INFO:tasks.rados.rados.0.smithi045.stdout:2975: read oid 19 snap -1 2024-04-26T00:37:09.667 INFO:tasks.rados.rados.0.smithi045.stdout:2975: expect (ObjNum 1055 snap 318 seq_num 1055) 2024-04-26T00:37:09.667 INFO:tasks.rados.rados.0.smithi045.stdout:2976: rollback oid 24 current snap is 321 2024-04-26T00:37:09.668 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 24 to 320 2024-04-26T00:37:09.668 INFO:tasks.rados.rados.0.smithi045.stdout:2969: expect (ObjNum 1059 snap 318 seq_num 1059) 2024-04-26T00:37:09.968 INFO:tasks.rados.rados.0.smithi045.stdout:2971: finishing write tid 1 to smithi04526579-11 2024-04-26T00:37:09.968 INFO:tasks.rados.rados.0.smithi045.stdout:2971: finishing write tid 2 to smithi04526579-11 2024-04-26T00:37:09.968 INFO:tasks.rados.rados.0.smithi045.stdout:2971: finishing write tid 3 to smithi04526579-11 2024-04-26T00:37:09.968 INFO:tasks.rados.rados.0.smithi045.stdout:2971: finishing write tid 4 to smithi04526579-11 2024-04-26T00:37:09.968 INFO:tasks.rados.rados.0.smithi045.stdout:2971: finishing write tid 5 to smithi04526579-11 2024-04-26T00:37:09.968 INFO:tasks.rados.rados.0.smithi045.stdout:2971: oid 11 updating version 0 to 2193 2024-04-26T00:37:09.968 INFO:tasks.rados.rados.0.smithi045.stdout:2971: oid 11 updating version 2193 to 2194 2024-04-26T00:37:09.968 INFO:tasks.rados.rados.0.smithi045.stdout:2971: oid 11 version 2194 is already newer than 2192 2024-04-26T00:37:09.968 INFO:tasks.rados.rados.0.smithi045.stdout:2971: oid 11 version 2194 is already newer than 2191 2024-04-26T00:37:09.968 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 2194 (ObjNum 1066 snap 321 seq_num 1066) dirty exists 2024-04-26T00:37:09.969 INFO:tasks.rados.rados.0.smithi045.stdout:2971: left oid 11 (ObjNum 1066 snap 321 seq_num 1066) 2024-04-26T00:37:09.969 INFO:tasks.rados.rados.0.smithi045.stdout:2972: finishing write tid 1 to smithi04526579-42 2024-04-26T00:37:09.969 INFO:tasks.rados.rados.0.smithi045.stdout:2973: expect (ObjNum 955 snap 295 seq_num 955) 2024-04-26T00:37:10.061 INFO:tasks.rados.rados.0.smithi045.stdout:2974: expect (ObjNum 1001 snap 304 seq_num 1001) 2024-04-26T00:37:10.155 INFO:tasks.rados.rados.0.smithi045.stdout:2972: finishing write tid 2 to smithi04526579-42 2024-04-26T00:37:10.155 INFO:tasks.rados.rados.0.smithi045.stdout:2972: finishing write tid 3 to smithi04526579-42 2024-04-26T00:37:10.155 INFO:tasks.rados.rados.0.smithi045.stdout:2972: finishing write tid 4 to smithi04526579-42 2024-04-26T00:37:10.155 INFO:tasks.rados.rados.0.smithi045.stdout:2972: finishing write tid 5 to smithi04526579-42 2024-04-26T00:37:10.155 INFO:tasks.rados.rados.0.smithi045.stdout:2972: oid 42 updating version 0 to 2365 2024-04-26T00:37:10.155 INFO:tasks.rados.rados.0.smithi045.stdout:2972: oid 42 version 2365 is already newer than 2362 2024-04-26T00:37:10.155 INFO:tasks.rados.rados.0.smithi045.stdout:2972: oid 42 version 2365 is already newer than 2364 2024-04-26T00:37:10.156 INFO:tasks.rados.rados.0.smithi045.stdout:2972: oid 42 version 2365 is already newer than 2363 2024-04-26T00:37:10.156 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 2365 (ObjNum 1067 snap 321 seq_num 1067) dirty exists 2024-04-26T00:37:10.156 INFO:tasks.rados.rados.0.smithi045.stdout:2972: left oid 42 (ObjNum 1067 snap 321 seq_num 1067) 2024-04-26T00:37:10.156 INFO:tasks.rados.rados.0.smithi045.stdout:2975: expect (ObjNum 1055 snap 318 seq_num 1055) 2024-04-26T00:37:10.279 INFO:tasks.rados.rados.0.smithi045.stdout:2976: finishing rollback tid 0 to smithi04526579-24 2024-04-26T00:37:10.279 INFO:tasks.rados.rados.0.smithi045.stdout:2976: finishing rollback tid 1 to smithi04526579-24 2024-04-26T00:37:10.279 INFO:tasks.rados.rados.0.smithi045.stdout:2976: finishing rollback tid 2 to smithi04526579-24 2024-04-26T00:37:10.279 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 2420 (ObjNum 1051 snap 317 seq_num 1051) dirty exists 2024-04-26T00:37:10.279 INFO:tasks.rados.rados.0.smithi045.stdout:2969: done (6 left) 2024-04-26T00:37:10.279 INFO:tasks.rados.rados.0.smithi045.stdout:2971: done (5 left) 2024-04-26T00:37:10.279 INFO:tasks.rados.rados.0.smithi045.stdout:2972: done (4 left) 2024-04-26T00:37:10.279 INFO:tasks.rados.rados.0.smithi045.stdout:2973: done (3 left) 2024-04-26T00:37:10.279 INFO:tasks.rados.rados.0.smithi045.stdout:2974: done (2 left) 2024-04-26T00:37:10.279 INFO:tasks.rados.rados.0.smithi045.stdout:2975: done (1 left) 2024-04-26T00:37:10.279 INFO:tasks.rados.rados.0.smithi045.stdout:2976: done (0 left) 2024-04-26T00:37:10.279 INFO:tasks.rados.rados.0.smithi045.stdout:2977: read oid 48 snap -1 2024-04-26T00:37:10.280 INFO:tasks.rados.rados.0.smithi045.stdout:2977: expect (ObjNum 938 snap 290 seq_num 938) 2024-04-26T00:37:10.280 INFO:tasks.rados.rados.0.smithi045.stdout:2978: copy_from oid 34 from oid 30 current snap is 321 2024-04-26T00:37:10.280 INFO:tasks.rados.rados.0.smithi045.stdout:2979: snap_remove snap 319 2024-04-26T00:37:11.225 INFO:tasks.rados.rados.0.smithi045.stdout:2979: done (2 left) 2024-04-26T00:37:11.226 INFO:tasks.rados.rados.0.smithi045.stdout:2980: copy_from oid 25 from oid 39 current snap is 321 2024-04-26T00:37:11.226 INFO:tasks.rados.rados.0.smithi045.stdout:2977: expect (ObjNum 938 snap 290 seq_num 938) 2024-04-26T00:37:11.450 INFO:tasks.rados.rados.0.smithi045.stdout:2978: finishing copy_from to smithi04526579-34 2024-04-26T00:37:11.450 INFO:tasks.rados.rados.0.smithi045.stdout:2978: got expected ENOENT (src dne) 2024-04-26T00:37:11.450 INFO:tasks.rados.rados.0.smithi045.stdout:2978: finishing copy_from racing read to smithi04526579-34 2024-04-26T00:37:11.450 INFO:tasks.rados.rados.0.smithi045.stdout:2980: finishing copy_from to smithi04526579-25 2024-04-26T00:37:11.450 INFO:tasks.rados.rados.0.smithi045.stdout:2980: got expected ENOENT (src dne) 2024-04-26T00:37:11.450 INFO:tasks.rados.rados.0.smithi045.stdout:2980: finishing copy_from racing read to smithi04526579-25 2024-04-26T00:37:11.450 INFO:tasks.rados.rados.0.smithi045.stdout:2977: done (2 left) 2024-04-26T00:37:11.450 INFO:tasks.rados.rados.0.smithi045.stdout:2978: done (1 left) 2024-04-26T00:37:11.450 INFO:tasks.rados.rados.0.smithi045.stdout:2980: done (0 left) 2024-04-26T00:37:11.450 INFO:tasks.rados.rados.0.smithi045.stdout:2981: delete oid 17 current snap is 321 2024-04-26T00:37:11.454 INFO:tasks.rados.rados.0.smithi045.stdout:2981: done (0 left) 2024-04-26T00:37:11.454 INFO:tasks.rados.rados.0.smithi045.stdout:2982: snap_remove snap 320 2024-04-26T00:37:12.234 INFO:tasks.rados.rados.0.smithi045.stdout:2982: done (0 left) 2024-04-26T00:37:12.234 INFO:tasks.rados.rados.0.smithi045.stdout:2983: read oid 12 snap -1 2024-04-26T00:37:12.234 INFO:tasks.rados.rados.0.smithi045.stdout:2983: expect (ObjNum 945 snap 292 seq_num 945) 2024-04-26T00:37:12.234 INFO:tasks.rados.rados.0.smithi045.stdout:2984: write oid 3 current snap is 321 2024-04-26T00:37:12.234 INFO:tasks.rados.rados.0.smithi045.stdout:2984: seq_num 1069 ranges {766683=730601,1922192=583372,3156591=676370} 2024-04-26T00:37:12.261 INFO:tasks.rados.rados.0.smithi045.stdout:2984: writing smithi04526579-3 from 766683 to 1497284 tid 1 2024-04-26T00:37:12.278 INFO:tasks.rados.rados.0.smithi045.stdout:2984: writing smithi04526579-3 from 1922192 to 2505564 tid 2 2024-04-26T00:37:12.294 INFO:tasks.rados.rados.0.smithi045.stdout:2984: writing smithi04526579-3 from 3156591 to 3832961 tid 3 2024-04-26T00:37:12.295 INFO:tasks.rados.rados.0.smithi045.stdout:2985: read oid 43 snap -1 2024-04-26T00:37:12.295 INFO:tasks.rados.rados.0.smithi045.stdout:2985: expect deleted 2024-04-26T00:37:12.295 INFO:tasks.rados.rados.0.smithi045.stdout:2986: snap_create 2024-04-26T00:37:12.295 INFO:tasks.rados.rados.0.smithi045.stdout:2983: expect (ObjNum 945 snap 292 seq_num 945) 2024-04-26T00:37:12.555 INFO:tasks.rados.rados.0.smithi045.stdout:2984: finishing write tid 1 to smithi04526579-3 2024-04-26T00:37:12.556 INFO:tasks.rados.rados.0.smithi045.stdout:2984: finishing write tid 2 to smithi04526579-3 2024-04-26T00:37:12.556 INFO:tasks.rados.rados.0.smithi045.stdout:2984: finishing write tid 3 to smithi04526579-3 2024-04-26T00:37:12.556 INFO:tasks.rados.rados.0.smithi045.stdout:2984: finishing write tid 4 to smithi04526579-3 2024-04-26T00:37:12.556 INFO:tasks.rados.rados.0.smithi045.stdout:2984: finishing write tid 5 to smithi04526579-3 2024-04-26T00:37:12.556 INFO:tasks.rados.rados.0.smithi045.stdout:2984: oid 3 updating version 0 to 2316 2024-04-26T00:37:12.556 INFO:tasks.rados.rados.0.smithi045.stdout:2984: oid 3 version 2316 is already newer than 2315 2024-04-26T00:37:12.556 INFO:tasks.rados.rados.0.smithi045.stdout:2984: oid 3 updating version 2316 to 2318 2024-04-26T00:37:12.556 INFO:tasks.rados.rados.0.smithi045.stdout:2984: oid 3 version 2318 is already newer than 2317 2024-04-26T00:37:12.556 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 2318 (ObjNum 1069 snap 321 seq_num 1069) dirty exists 2024-04-26T00:37:12.556 INFO:tasks.rados.rados.0.smithi045.stdout:2984: left oid 3 (ObjNum 1069 snap 321 seq_num 1069) 2024-04-26T00:37:13.236 INFO:tasks.rados.rados.0.smithi045.stdout:2983: done (3 left) 2024-04-26T00:37:13.236 INFO:tasks.rados.rados.0.smithi045.stdout:2984: done (2 left) 2024-04-26T00:37:13.236 INFO:tasks.rados.rados.0.smithi045.stdout:2985: done (1 left) 2024-04-26T00:37:13.236 INFO:tasks.rados.rados.0.smithi045.stdout:2986: done (0 left) 2024-04-26T00:37:13.236 INFO:tasks.rados.rados.0.smithi045.stdout:2987: delete oid 44 current snap is 322 2024-04-26T00:37:13.251 INFO:tasks.rados.rados.0.smithi045.stdout:2987: done (0 left) 2024-04-26T00:37:13.251 INFO:tasks.rados.rados.0.smithi045.stdout:2988: read oid 20 snap -1 2024-04-26T00:37:13.251 INFO:tasks.rados.rados.0.smithi045.stdout:2988: expect (ObjNum 1054 snap 318 seq_num 1054) 2024-04-26T00:37:13.251 INFO:tasks.rados.rados.0.smithi045.stdout:2989: copy_from oid 37 from oid 24 current snap is 322 2024-04-26T00:37:13.251 INFO:tasks.rados.rados.0.smithi045.stdout:2990: read oid 45 snap -1 2024-04-26T00:37:13.252 INFO:tasks.rados.rados.0.smithi045.stdout:2990: expect (ObjNum 1062 snap 318 seq_num 1062) 2024-04-26T00:37:13.252 INFO:tasks.rados.rados.0.smithi045.stdout:2991: copy_from oid 11 from oid 13 current snap is 322 2024-04-26T00:37:13.252 INFO:tasks.rados.rados.0.smithi045.stdout:2992: write (excl) oid 18 current snap is 322 2024-04-26T00:37:13.252 INFO:tasks.rados.rados.0.smithi045.stdout:2992: seq_num 1071 ranges {400297=481099,1639156=361141} 2024-04-26T00:37:13.264 INFO:tasks.rados.rados.0.smithi045.stdout:2992: writing smithi04526579-18 from 400297 to 881396 tid 1 2024-04-26T00:37:13.282 INFO:tasks.rados.rados.0.smithi045.stdout:2992: writing smithi04526579-18 from 1639156 to 2000297 tid 2 2024-04-26T00:37:13.282 INFO:tasks.rados.rados.0.smithi045.stdout:2993: delete oid 46 current snap is 322 2024-04-26T00:37:13.282 INFO:tasks.rados.rados.0.smithi045.stdout:2988: expect (ObjNum 1054 snap 318 seq_num 1054) 2024-04-26T00:37:13.457 INFO:tasks.rados.rados.0.smithi045.stdout:2990: expect (ObjNum 1062 snap 318 seq_num 1062) 2024-04-26T00:37:13.668 INFO:tasks.rados.rados.0.smithi045.stdout:2989: finishing copy_from racing read to smithi04526579-37 2024-04-26T00:37:13.668 INFO:tasks.rados.rados.0.smithi045.stdout:2991: finishing copy_from racing read to smithi04526579-11 2024-04-26T00:37:13.668 INFO:tasks.rados.rados.0.smithi045.stdout:2989: finishing copy_from to smithi04526579-37 2024-04-26T00:37:13.668 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 2420 (ObjNum 1051 snap 317 seq_num 1051) dirty exists 2024-04-26T00:37:13.668 INFO:tasks.rados.rados.0.smithi045.stdout:2991: finishing copy_from to smithi04526579-11 2024-04-26T00:37:13.668 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 2193 (ObjNum 1043 snap 316 seq_num 1043) dirty exists 2024-04-26T00:37:13.668 INFO:tasks.rados.rados.0.smithi045.stdout:2992: finishing write tid 1 to smithi04526579-18 2024-04-26T00:37:13.668 INFO:tasks.rados.rados.0.smithi045.stdout:2992: finishing write tid 2 to smithi04526579-18 2024-04-26T00:37:13.668 INFO:tasks.rados.rados.0.smithi045.stdout:2992: finishing write tid 3 to smithi04526579-18 2024-04-26T00:37:13.668 INFO:tasks.rados.rados.0.smithi045.stdout:2992: finishing write tid 4 to smithi04526579-18 2024-04-26T00:37:13.668 INFO:tasks.rados.rados.0.smithi045.stdout:2992: oid 18 updating version 0 to 2430 2024-04-26T00:37:13.669 INFO:tasks.rados.rados.0.smithi045.stdout:2992: oid 18 version 2430 is already newer than 2428 2024-04-26T00:37:13.669 INFO:tasks.rados.rados.0.smithi045.stdout:2992: oid 18 version 2430 is already newer than 2429 2024-04-26T00:37:13.669 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 2430 (ObjNum 1071 snap 322 seq_num 1071) dirty exists 2024-04-26T00:37:13.669 INFO:tasks.rados.rados.0.smithi045.stdout:2992: left oid 18 (ObjNum 1071 snap 322 seq_num 1071) 2024-04-26T00:37:13.669 INFO:tasks.rados.rados.0.smithi045.stdout:2988: done (5 left) 2024-04-26T00:37:13.669 INFO:tasks.rados.rados.0.smithi045.stdout:2989: done (4 left) 2024-04-26T00:37:13.669 INFO:tasks.rados.rados.0.smithi045.stdout:2990: done (3 left) 2024-04-26T00:37:13.669 INFO:tasks.rados.rados.0.smithi045.stdout:2991: done (2 left) 2024-04-26T00:37:13.669 INFO:tasks.rados.rados.0.smithi045.stdout:2992: done (1 left) 2024-04-26T00:37:13.669 INFO:tasks.rados.rados.0.smithi045.stdout:2993: done (0 left) 2024-04-26T00:37:13.669 INFO:tasks.rados.rados.0.smithi045.stdout:2994: write (excl) oid 49 current snap is 322 2024-04-26T00:37:13.669 INFO:tasks.rados.rados.0.smithi045.stdout:2994: seq_num 1073 ranges {433911=747950,1756120=277791} 2024-04-26T00:37:13.686 INFO:tasks.rados.rados.0.smithi045.stdout:2994: writing smithi04526579-49 from 433911 to 1181861 tid 1 2024-04-26T00:37:13.701 INFO:tasks.rados.rados.0.smithi045.stdout:2994: writing smithi04526579-49 from 1756120 to 2033911 tid 2 2024-04-26T00:37:13.702 INFO:tasks.rados.rados.0.smithi045.stdout:2995: read oid 1 snap 321 2024-04-26T00:37:13.702 INFO:tasks.rados.rados.0.smithi045.stdout:2995: expect (ObjNum 1017 snap 310 seq_num 1017) 2024-04-26T00:37:13.702 INFO:tasks.rados.rados.0.smithi045.stdout:2996: write (excl) oid 2 current snap is 322 2024-04-26T00:37:13.702 INFO:tasks.rados.rados.0.smithi045.stdout:2996: seq_num 1074 ranges {450718=423199,1414602=636116} 2024-04-26T00:37:13.716 INFO:tasks.rados.rados.0.smithi045.stdout:2996: writing smithi04526579-2 from 450718 to 873917 tid 1 2024-04-26T00:37:13.732 INFO:tasks.rados.rados.0.smithi045.stdout:2996: writing smithi04526579-2 from 1414602 to 2050718 tid 2 2024-04-26T00:37:13.733 INFO:tasks.rados.rados.0.smithi045.stdout:2997: write (excl) oid 38 current snap is 322 2024-04-26T00:37:13.733 INFO:tasks.rados.rados.0.smithi045.stdout:2997: seq_num 1075 ranges {467525=498448,1473084=594379} 2024-04-26T00:37:13.743 INFO:tasks.rados.rados.0.smithi045.stdout:2997: writing smithi04526579-38 from 467525 to 965973 tid 1 2024-04-26T00:37:13.753 INFO:tasks.rados.rados.0.smithi045.stdout:2997: writing smithi04526579-38 from 1473084 to 2067463 tid 2 2024-04-26T00:37:13.753 INFO:tasks.rados.rados.0.smithi045.stdout:2998: copy_from oid 11 from oid 29 current snap is 322 2024-04-26T00:37:13.753 INFO:tasks.rados.rados.0.smithi045.stdout:2994: finishing write tid 1 to smithi04526579-49 2024-04-26T00:37:13.753 INFO:tasks.rados.rados.0.smithi045.stdout:2999: write oid 50 current snap is 322 2024-04-26T00:37:13.754 INFO:tasks.rados.rados.0.smithi045.stdout:2999: seq_num 1076 ranges {484332=573697,1531566=552766} 2024-04-26T00:37:13.762 INFO:tasks.rados.rados.0.smithi045.stdout:2999: writing smithi04526579-50 from 484332 to 1058029 tid 1 2024-04-26T00:37:13.771 INFO:tasks.rados.rados.0.smithi045.stdout:2999: writing smithi04526579-50 from 1531566 to 2084332 tid 2 2024-04-26T00:37:13.772 INFO:tasks.rados.rados.0.smithi045.stdout:3000: copy_from oid 15 from oid 41 current snap is 322 2024-04-26T00:37:13.772 INFO:tasks.rados.rados.0.smithi045.stdout:2995: expect (ObjNum 1017 snap 310 seq_num 1017) 2024-04-26T00:37:13.988 INFO:tasks.rados.rados.0.smithi045.stdout:2994: finishing write tid 2 to smithi04526579-49 2024-04-26T00:37:13.988 INFO:tasks.rados.rados.0.smithi045.stdout:2994: finishing write tid 3 to smithi04526579-49 2024-04-26T00:37:13.988 INFO:tasks.rados.rados.0.smithi045.stdout:2994: finishing write tid 4 to smithi04526579-49 2024-04-26T00:37:13.988 INFO:tasks.rados.rados.0.smithi045.stdout:2994: oid 49 updating version 0 to 2320 2024-04-26T00:37:13.988 INFO:tasks.rados.rados.0.smithi045.stdout:2994: oid 49 updating version 2320 to 2321 2024-04-26T00:37:13.988 INFO:tasks.rados.rados.0.smithi045.stdout:2994: oid 49 version 2321 is already newer than 2319 2024-04-26T00:37:13.988 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 2321 (ObjNum 1073 snap 322 seq_num 1073) dirty exists 2024-04-26T00:37:13.988 INFO:tasks.rados.rados.0.smithi045.stdout:2994: left oid 49 (ObjNum 1073 snap 322 seq_num 1073) 2024-04-26T00:37:13.988 INFO:tasks.rados.rados.0.smithi045.stdout:2996: finishing write tid 1 to smithi04526579-2 2024-04-26T00:37:13.988 INFO:tasks.rados.rados.0.smithi045.stdout:2996: finishing write tid 2 to smithi04526579-2 2024-04-26T00:37:13.988 INFO:tasks.rados.rados.0.smithi045.stdout:2996: finishing write tid 3 to smithi04526579-2 2024-04-26T00:37:13.988 INFO:tasks.rados.rados.0.smithi045.stdout:2996: finishing write tid 4 to smithi04526579-2 2024-04-26T00:37:13.989 INFO:tasks.rados.rados.0.smithi045.stdout:2996: oid 2 updating version 0 to 2034 2024-04-26T00:37:13.989 INFO:tasks.rados.rados.0.smithi045.stdout:2996: oid 2 updating version 2034 to 2035 2024-04-26T00:37:13.989 INFO:tasks.rados.rados.0.smithi045.stdout:2996: oid 2 updating version 2035 to 2036 2024-04-26T00:37:13.989 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 2036 (ObjNum 1074 snap 322 seq_num 1074) dirty exists 2024-04-26T00:37:13.989 INFO:tasks.rados.rados.0.smithi045.stdout:2996: left oid 2 (ObjNum 1074 snap 322 seq_num 1074) 2024-04-26T00:37:13.989 INFO:tasks.rados.rados.0.smithi045.stdout:2997: finishing write tid 1 to smithi04526579-38 2024-04-26T00:37:13.989 INFO:tasks.rados.rados.0.smithi045.stdout:2997: finishing write tid 2 to smithi04526579-38 2024-04-26T00:37:13.989 INFO:tasks.rados.rados.0.smithi045.stdout:2997: finishing write tid 3 to smithi04526579-38 2024-04-26T00:37:13.989 INFO:tasks.rados.rados.0.smithi045.stdout:2997: finishing write tid 4 to smithi04526579-38 2024-04-26T00:37:13.989 INFO:tasks.rados.rados.0.smithi045.stdout:2997: oid 38 updating version 0 to 2209 2024-04-26T00:37:13.989 INFO:tasks.rados.rados.0.smithi045.stdout:2997: oid 38 updating version 2209 to 2211 2024-04-26T00:37:13.989 INFO:tasks.rados.rados.0.smithi045.stdout:2997: oid 38 version 2211 is already newer than 2210 2024-04-26T00:37:13.989 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 2211 (ObjNum 1075 snap 322 seq_num 1075) dirty exists 2024-04-26T00:37:13.989 INFO:tasks.rados.rados.0.smithi045.stdout:2997: left oid 38 (ObjNum 1075 snap 322 seq_num 1075) 2024-04-26T00:37:13.989 INFO:tasks.rados.rados.0.smithi045.stdout:2998: finishing copy_from racing read to smithi04526579-11 2024-04-26T00:37:13.990 INFO:tasks.rados.rados.0.smithi045.stdout:3000: finishing copy_from to smithi04526579-15 2024-04-26T00:37:13.990 INFO:tasks.rados.rados.0.smithi045.stdout:3000: got expected ENOENT (src dne) 2024-04-26T00:37:13.990 INFO:tasks.rados.rados.0.smithi045.stdout:3000: finishing copy_from racing read to smithi04526579-15 2024-04-26T00:37:13.990 INFO:tasks.rados.rados.0.smithi045.stdout:2998: finishing copy_from to smithi04526579-11 2024-04-26T00:37:13.990 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 2011 (ObjNum 987 snap 302 seq_num 987) dirty exists 2024-04-26T00:37:13.990 INFO:tasks.rados.rados.0.smithi045.stdout:2999: finishing write tid 1 to smithi04526579-50 2024-04-26T00:37:13.990 INFO:tasks.rados.rados.0.smithi045.stdout:2999: finishing write tid 2 to smithi04526579-50 2024-04-26T00:37:13.990 INFO:tasks.rados.rados.0.smithi045.stdout:2999: finishing write tid 3 to smithi04526579-50 2024-04-26T00:37:13.990 INFO:tasks.rados.rados.0.smithi045.stdout:2999: finishing write tid 4 to smithi04526579-50 2024-04-26T00:37:13.990 INFO:tasks.rados.rados.0.smithi045.stdout:2999: oid 50 updating version 0 to 1999 2024-04-26T00:37:13.990 INFO:tasks.rados.rados.0.smithi045.stdout:2999: oid 50 version 1999 is already newer than 1998 2024-04-26T00:37:13.990 INFO:tasks.rados.rados.0.smithi045.stdout:2999: oid 50 version 1999 is already newer than 1997 2024-04-26T00:37:13.990 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 1999 (ObjNum 1076 snap 322 seq_num 1076) dirty exists 2024-04-26T00:37:13.991 INFO:tasks.rados.rados.0.smithi045.stdout:2999: left oid 50 (ObjNum 1076 snap 322 seq_num 1076) 2024-04-26T00:37:13.991 INFO:tasks.rados.rados.0.smithi045.stdout:2994: done (6 left) 2024-04-26T00:37:13.991 INFO:tasks.rados.rados.0.smithi045.stdout:2995: done (5 left) 2024-04-26T00:37:13.991 INFO:tasks.rados.rados.0.smithi045.stdout:2996: done (4 left) 2024-04-26T00:37:13.991 INFO:tasks.rados.rados.0.smithi045.stdout:2997: done (3 left) 2024-04-26T00:37:13.991 INFO:tasks.rados.rados.0.smithi045.stdout:2998: done (2 left) 2024-04-26T00:37:13.991 INFO:tasks.rados.rados.0.smithi045.stdout:2999: done (1 left) 2024-04-26T00:37:13.991 INFO:tasks.rados.rados.0.smithi045.stdout:3000: done (0 left) 2024-04-26T00:37:13.991 INFO:tasks.rados.rados.0.smithi045.stdout:3001: write oid 35 current snap is 322 2024-04-26T00:37:13.991 INFO:tasks.rados.rados.0.smithi045.stdout:3001: seq_num 1077 ranges {501139=648946,1873695=227444} 2024-04-26T00:37:13.995 INFO:tasks.rados.rados.0.smithi045.stdout:3001: writing smithi04526579-35 from 501139 to 1150085 tid 1 2024-04-26T00:37:14.002 INFO:tasks.rados.rados.0.smithi045.stdout:3001: writing smithi04526579-35 from 1873695 to 2101139 tid 2 2024-04-26T00:37:14.002 INFO:tasks.rados.rados.0.smithi045.stdout:3002: read oid 39 snap -1 2024-04-26T00:37:14.002 INFO:tasks.rados.rados.0.smithi045.stdout:3002: expect deleted 2024-04-26T00:37:14.002 INFO:tasks.rados.rados.0.smithi045.stdout:3003: copy_from oid 20 from oid 42 current snap is 322 2024-04-26T00:37:14.003 INFO:tasks.rados.rados.0.smithi045.stdout:3004: rollback oid 36 current snap is 322 2024-04-26T00:37:14.003 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 36 to 321 2024-04-26T00:37:14.003 INFO:tasks.rados.rados.0.smithi045.stdout:3005: read oid 37 snap -1 2024-04-26T00:37:14.003 INFO:tasks.rados.rados.0.smithi045.stdout:3005: expect (ObjNum 1051 snap 317 seq_num 1051) 2024-04-26T00:37:14.003 INFO:tasks.rados.rados.0.smithi045.stdout:3006: read oid 34 snap -1 2024-04-26T00:37:14.003 INFO:tasks.rados.rados.0.smithi045.stdout:3006: expect (ObjNum 1064 snap 320 seq_num 1064) 2024-04-26T00:37:14.003 INFO:tasks.rados.rados.0.smithi045.stdout:3007: rollback oid 44 current snap is 322 2024-04-26T00:37:14.003 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 44 to 321 2024-04-26T00:37:14.003 INFO:tasks.rados.rados.0.smithi045.stdout:3008: write oid 10 current snap is 322 2024-04-26T00:37:14.003 INFO:tasks.rados.rados.0.smithi045.stdout:3008: seq_num 1078 ranges {517946=724195,1932177=185769} 2024-04-26T00:37:14.010 INFO:tasks.rados.rados.0.smithi045.stdout:3008: writing smithi04526579-10 from 517946 to 1242141 tid 1 2024-04-26T00:37:14.017 INFO:tasks.rados.rados.0.smithi045.stdout:3008: writing smithi04526579-10 from 1932177 to 2117946 tid 2 2024-04-26T00:37:14.017 INFO:tasks.rados.rados.0.smithi045.stdout:3009: read oid 25 snap -1 2024-04-26T00:37:14.017 INFO:tasks.rados.rados.0.smithi045.stdout:3009: expect deleted 2024-04-26T00:37:14.017 INFO:tasks.rados.rados.0.smithi045.stdout:3010: write oid 18 current snap is 322 2024-04-26T00:37:14.017 INFO:tasks.rados.rados.0.smithi045.stdout:3010: seq_num 1079 ranges {534753=799444,1990659=144094} 2024-04-26T00:37:14.027 INFO:tasks.rados.rados.0.smithi045.stdout:3010: writing smithi04526579-18 from 534753 to 1334197 tid 1 2024-04-26T00:37:14.034 INFO:tasks.rados.rados.0.smithi045.stdout:3010: writing smithi04526579-18 from 1990659 to 2134753 tid 2 2024-04-26T00:37:14.034 INFO:tasks.rados.rados.0.smithi045.stdout:3011: snap_create 2024-04-26T00:37:14.035 INFO:tasks.rados.rados.0.smithi045.stdout:3007: finishing rollback tid 1 to smithi04526579-44 2024-04-26T00:37:14.035 INFO:tasks.rados.rados.0.smithi045.stdout:3004: finishing rollback tid 0 to smithi04526579-36 2024-04-26T00:37:14.035 INFO:tasks.rados.rados.0.smithi045.stdout:3004: finishing rollback tid 1 to smithi04526579-36 2024-04-26T00:37:14.035 INFO:tasks.rados.rados.0.smithi045.stdout:3004: finishing rollback tid 2 to smithi04526579-36 2024-04-26T00:37:14.035 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 2409 (ObjNum 1051 snap 317 seq_num 1051) dirty exists 2024-04-26T00:37:14.035 INFO:tasks.rados.rados.0.smithi045.stdout:3001: finishing write tid 1 to smithi04526579-35 2024-04-26T00:37:14.035 INFO:tasks.rados.rados.0.smithi045.stdout:3001: finishing write tid 2 to smithi04526579-35 2024-04-26T00:37:14.035 INFO:tasks.rados.rados.0.smithi045.stdout:3007: finishing rollback tid 2 to smithi04526579-44 2024-04-26T00:37:14.035 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 2213 (ObjNum 1065 snap 320 seq_num 1065) dirty exists 2024-04-26T00:37:14.035 INFO:tasks.rados.rados.0.smithi045.stdout:3001: finishing write tid 3 to smithi04526579-35 2024-04-26T00:37:14.035 INFO:tasks.rados.rados.0.smithi045.stdout:3001: finishing write tid 4 to smithi04526579-35 2024-04-26T00:37:14.035 INFO:tasks.rados.rados.0.smithi045.stdout:3001: oid 35 updating version 0 to 2368 2024-04-26T00:37:14.035 INFO:tasks.rados.rados.0.smithi045.stdout:3001: oid 35 version 2368 is already newer than 2367 2024-04-26T00:37:14.035 INFO:tasks.rados.rados.0.smithi045.stdout:3001: oid 35 version 2368 is already newer than 2366 2024-04-26T00:37:14.035 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 2368 (ObjNum 1077 snap 322 seq_num 1077) dirty exists 2024-04-26T00:37:14.036 INFO:tasks.rados.rados.0.smithi045.stdout:3001: left oid 35 (ObjNum 1077 snap 322 seq_num 1077) 2024-04-26T00:37:14.036 INFO:tasks.rados.rados.0.smithi045.stdout:3008: finishing write tid 1 to smithi04526579-10 2024-04-26T00:37:14.036 INFO:tasks.rados.rados.0.smithi045.stdout:3008: finishing write tid 2 to smithi04526579-10 2024-04-26T00:37:14.036 INFO:tasks.rados.rados.0.smithi045.stdout:3008: finishing write tid 3 to smithi04526579-10 2024-04-26T00:37:14.036 INFO:tasks.rados.rados.0.smithi045.stdout:3008: finishing write tid 4 to smithi04526579-10 2024-04-26T00:37:14.036 INFO:tasks.rados.rados.0.smithi045.stdout:3008: oid 10 updating version 0 to 2000 2024-04-26T00:37:14.036 INFO:tasks.rados.rados.0.smithi045.stdout:3008: oid 10 updating version 2000 to 2001 2024-04-26T00:37:14.036 INFO:tasks.rados.rados.0.smithi045.stdout:3008: oid 10 updating version 2001 to 2002 2024-04-26T00:37:14.036 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 2002 (ObjNum 1078 snap 322 seq_num 1078) dirty exists 2024-04-26T00:37:14.036 INFO:tasks.rados.rados.0.smithi045.stdout:3008: left oid 10 (ObjNum 1078 snap 322 seq_num 1078) 2024-04-26T00:37:14.036 INFO:tasks.rados.rados.0.smithi045.stdout:3005: expect (ObjNum 1051 snap 317 seq_num 1051) 2024-04-26T00:37:14.251 INFO:tasks.rados.rados.0.smithi045.stdout:3006: expect (ObjNum 1064 snap 320 seq_num 1064) 2024-04-26T00:37:14.465 INFO:tasks.rados.rados.0.smithi045.stdout:3010: finishing write tid 1 to smithi04526579-18 2024-04-26T00:37:14.465 INFO:tasks.rados.rados.0.smithi045.stdout:3010: finishing write tid 2 to smithi04526579-18 2024-04-26T00:37:14.465 INFO:tasks.rados.rados.0.smithi045.stdout:3010: finishing write tid 3 to smithi04526579-18 2024-04-26T00:37:14.465 INFO:tasks.rados.rados.0.smithi045.stdout:3010: finishing write tid 4 to smithi04526579-18 2024-04-26T00:37:14.465 INFO:tasks.rados.rados.0.smithi045.stdout:3010: oid 18 updating version 0 to 2433 2024-04-26T00:37:14.465 INFO:tasks.rados.rados.0.smithi045.stdout:3010: oid 18 version 2433 is already newer than 2431 2024-04-26T00:37:14.465 INFO:tasks.rados.rados.0.smithi045.stdout:3010: oid 18 version 2433 is already newer than 2432 2024-04-26T00:37:14.465 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 2433 (ObjNum 1079 snap 322 seq_num 1079) dirty exists 2024-04-26T00:37:14.465 INFO:tasks.rados.rados.0.smithi045.stdout:3010: left oid 18 (ObjNum 1079 snap 322 seq_num 1079) 2024-04-26T00:37:14.465 INFO:tasks.rados.rados.0.smithi045.stdout:3003: finishing copy_from racing read to smithi04526579-20 2024-04-26T00:37:14.465 INFO:tasks.rados.rados.0.smithi045.stdout:3003: finishing copy_from to smithi04526579-20 2024-04-26T00:37:14.466 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 2365 (ObjNum 1067 snap 321 seq_num 1067) dirty exists 2024-04-26T00:37:14.580 INFO:tasks.rados.rados.0.smithi045.stdout:3001: done (10 left) 2024-04-26T00:37:14.580 INFO:tasks.rados.rados.0.smithi045.stdout:3002: done (9 left) 2024-04-26T00:37:14.580 INFO:tasks.rados.rados.0.smithi045.stdout:3003: done (8 left) 2024-04-26T00:37:14.580 INFO:tasks.rados.rados.0.smithi045.stdout:3004: done (7 left) 2024-04-26T00:37:14.580 INFO:tasks.rados.rados.0.smithi045.stdout:3005: done (6 left) 2024-04-26T00:37:14.580 INFO:tasks.rados.rados.0.smithi045.stdout:3006: done (5 left) 2024-04-26T00:37:14.580 INFO:tasks.rados.rados.0.smithi045.stdout:3007: done (4 left) 2024-04-26T00:37:14.580 INFO:tasks.rados.rados.0.smithi045.stdout:3008: done (3 left) 2024-04-26T00:37:14.580 INFO:tasks.rados.rados.0.smithi045.stdout:3009: done (2 left) 2024-04-26T00:37:14.581 INFO:tasks.rados.rados.0.smithi045.stdout:3010: done (1 left) 2024-04-26T00:37:14.581 INFO:tasks.rados.rados.0.smithi045.stdout:3011: done (0 left) 2024-04-26T00:37:14.581 INFO:tasks.rados.rados.0.smithi045.stdout:3012: snap_remove snap 321 2024-04-26T00:37:15.578 INFO:tasks.rados.rados.0.smithi045.stdout:3012: done (0 left) 2024-04-26T00:37:15.578 INFO:tasks.rados.rados.0.smithi045.stdout:3013: write (excl) oid 4 current snap is 323 2024-04-26T00:37:15.578 INFO:tasks.rados.rados.0.smithi045.stdout:3013: seq_num 1080 ranges {551560=591046,1765494=386066} 2024-04-26T00:37:15.590 INFO:tasks.rados.rados.0.smithi045.stdout:3013: writing smithi04526579-4 from 551560 to 1142606 tid 1 2024-04-26T00:37:15.601 INFO:tasks.rados.rados.0.smithi045.stdout:3013: writing smithi04526579-4 from 1765494 to 2151560 tid 2 2024-04-26T00:37:15.601 INFO:tasks.rados.rados.0.smithi045.stdout:3014: delete oid 39 current snap is 323 2024-04-26T00:37:15.609 INFO:tasks.rados.rados.0.smithi045.stdout:3014: done (1 left) 2024-04-26T00:37:15.609 INFO:tasks.rados.rados.0.smithi045.stdout:3015: snap_remove snap 322 2024-04-26T00:37:16.587 INFO:tasks.rados.rados.0.smithi045.stdout:3015: done (1 left) 2024-04-26T00:37:16.587 INFO:tasks.rados.rados.0.smithi045.stdout:3016: delete oid 11 current snap is 323 2024-04-26T00:37:16.588 INFO:tasks.rados.rados.0.smithi045.stdout:3013: finishing write tid 1 to smithi04526579-4 2024-04-26T00:37:16.588 INFO:tasks.rados.rados.0.smithi045.stdout:3013: finishing write tid 2 to smithi04526579-4 2024-04-26T00:37:16.588 INFO:tasks.rados.rados.0.smithi045.stdout:3013: finishing write tid 3 to smithi04526579-4 2024-04-26T00:37:16.588 INFO:tasks.rados.rados.0.smithi045.stdout:3013: finishing write tid 4 to smithi04526579-4 2024-04-26T00:37:16.588 INFO:tasks.rados.rados.0.smithi045.stdout:3013: oid 4 updating version 0 to 2038 2024-04-26T00:37:16.588 INFO:tasks.rados.rados.0.smithi045.stdout:3013: oid 4 version 2038 is already newer than 2037 2024-04-26T00:37:16.589 INFO:tasks.rados.rados.0.smithi045.stdout:3013: oid 4 updating version 2038 to 2039 2024-04-26T00:37:16.589 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 2039 (ObjNum 1080 snap 323 seq_num 1080) dirty exists 2024-04-26T00:37:16.589 INFO:tasks.rados.rados.0.smithi045.stdout:3013: left oid 4 (ObjNum 1080 snap 323 seq_num 1080) 2024-04-26T00:37:16.595 INFO:tasks.rados.rados.0.smithi045.stdout:3013: done (1 left) 2024-04-26T00:37:16.596 INFO:tasks.rados.rados.0.smithi045.stdout:3016: done (0 left) 2024-04-26T00:37:16.596 INFO:tasks.rados.rados.0.smithi045.stdout:3017: delete oid 26 current snap is 323 2024-04-26T00:37:16.608 INFO:tasks.rados.rados.0.smithi045.stdout:3017: done (0 left) 2024-04-26T00:37:16.608 INFO:tasks.rados.rados.0.smithi045.stdout:3018: copy_from oid 1 from oid 18 current snap is 323 2024-04-26T00:37:16.608 INFO:tasks.rados.rados.0.smithi045.stdout:3019: delete oid 50 current snap is 323 2024-04-26T00:37:16.612 INFO:tasks.rados.rados.0.smithi045.stdout:3019: done (1 left) 2024-04-26T00:37:16.612 INFO:tasks.rados.rados.0.smithi045.stdout:3020: snap_create 2024-04-26T00:37:16.630 INFO:tasks.rados.rados.0.smithi045.stdout:3018: finishing copy_from racing read to smithi04526579-1 2024-04-26T00:37:16.630 INFO:tasks.rados.rados.0.smithi045.stdout:3018: finishing copy_from to smithi04526579-1 2024-04-26T00:37:16.631 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 2433 (ObjNum 1079 snap 322 seq_num 1079) dirty exists 2024-04-26T00:37:17.611 INFO:tasks.rados.rados.0.smithi045.stdout:3018: done (1 left) 2024-04-26T00:37:17.611 INFO:tasks.rados.rados.0.smithi045.stdout:3020: done (0 left) 2024-04-26T00:37:17.611 INFO:tasks.rados.rados.0.smithi045.stdout:3021: write oid 49 current snap is 324 2024-04-26T00:37:17.611 INFO:tasks.rados.rados.0.smithi045.stdout:3021: seq_num 1085 ranges {635595=567291,1825198=410397} 2024-04-26T00:37:17.630 INFO:tasks.rados.rados.0.smithi045.stdout:3021: writing smithi04526579-49 from 635595 to 1202886 tid 1 2024-04-26T00:37:17.647 INFO:tasks.rados.rados.0.smithi045.stdout:3021: writing smithi04526579-49 from 1825198 to 2235595 tid 2 2024-04-26T00:37:17.647 INFO:tasks.rados.rados.0.smithi045.stdout:3022: read oid 36 snap -1 2024-04-26T00:37:17.647 INFO:tasks.rados.rados.0.smithi045.stdout:3022: expect (ObjNum 1051 snap 317 seq_num 1051) 2024-04-26T00:37:17.647 INFO:tasks.rados.rados.0.smithi045.stdout:3023: write (excl) oid 42 current snap is 324 2024-04-26T00:37:17.647 INFO:tasks.rados.rados.0.smithi045.stdout:3023: seq_num 1086 ranges {652402=642540,1883680=368722} 2024-04-26T00:37:17.667 INFO:tasks.rados.rados.0.smithi045.stdout:3023: writing smithi04526579-42 from 652402 to 1294942 tid 1 2024-04-26T00:37:17.678 INFO:tasks.rados.rados.0.smithi045.stdout:3023: writing smithi04526579-42 from 1883680 to 2252402 tid 2 2024-04-26T00:37:17.679 INFO:tasks.rados.rados.0.smithi045.stdout:3024: read oid 38 snap -1 2024-04-26T00:37:17.679 INFO:tasks.rados.rados.0.smithi045.stdout:3024: expect (ObjNum 1075 snap 322 seq_num 1075) 2024-04-26T00:37:17.679 INFO:tasks.rados.rados.0.smithi045.stdout:3025: write (excl) oid 14 current snap is 324 2024-04-26T00:37:17.679 INFO:tasks.rados.rados.0.smithi045.stdout:3021: finishing write tid 1 to smithi04526579-49 2024-04-26T00:37:17.679 INFO:tasks.rados.rados.0.smithi045.stdout:3025: seq_num 1087 ranges {669209=717789,1942162=327047} 2024-04-26T00:37:17.694 INFO:tasks.rados.rados.0.smithi045.stdout:3025: writing smithi04526579-14 from 669209 to 1386998 tid 1 2024-04-26T00:37:17.704 INFO:tasks.rados.rados.0.smithi045.stdout:3025: writing smithi04526579-14 from 1942162 to 2269209 tid 2 2024-04-26T00:37:17.704 INFO:tasks.rados.rados.0.smithi045.stdout:3026: delete oid 22 current snap is 324 2024-04-26T00:37:17.704 INFO:tasks.rados.rados.0.smithi045.stdout:3021: finishing write tid 2 to smithi04526579-49 2024-04-26T00:37:17.704 INFO:tasks.rados.rados.0.smithi045.stdout:3021: finishing write tid 3 to smithi04526579-49 2024-04-26T00:37:17.704 INFO:tasks.rados.rados.0.smithi045.stdout:3021: finishing write tid 4 to smithi04526579-49 2024-04-26T00:37:17.704 INFO:tasks.rados.rados.0.smithi045.stdout:3021: oid 49 updating version 0 to 2322 2024-04-26T00:37:17.704 INFO:tasks.rados.rados.0.smithi045.stdout:3021: oid 49 updating version 2322 to 2323 2024-04-26T00:37:17.704 INFO:tasks.rados.rados.0.smithi045.stdout:3021: oid 49 updating version 2323 to 2324 2024-04-26T00:37:17.704 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 2324 (ObjNum 1085 snap 324 seq_num 1085) dirty exists 2024-04-26T00:37:17.704 INFO:tasks.rados.rados.0.smithi045.stdout:3021: left oid 49 (ObjNum 1085 snap 324 seq_num 1085) 2024-04-26T00:37:17.705 INFO:tasks.rados.rados.0.smithi045.stdout:3022: expect (ObjNum 1051 snap 317 seq_num 1051) 2024-04-26T00:37:17.965 INFO:tasks.rados.rados.0.smithi045.stdout:3024: expect (ObjNum 1075 snap 322 seq_num 1075) 2024-04-26T00:37:18.064 INFO:tasks.rados.rados.0.smithi045.stdout:3023: finishing write tid 1 to smithi04526579-42 2024-04-26T00:37:18.064 INFO:tasks.rados.rados.0.smithi045.stdout:3023: finishing write tid 2 to smithi04526579-42 2024-04-26T00:37:18.065 INFO:tasks.rados.rados.0.smithi045.stdout:3023: finishing write tid 3 to smithi04526579-42 2024-04-26T00:37:18.065 INFO:tasks.rados.rados.0.smithi045.stdout:3023: finishing write tid 4 to smithi04526579-42 2024-04-26T00:37:18.065 INFO:tasks.rados.rados.0.smithi045.stdout:3023: oid 42 updating version 0 to 2371 2024-04-26T00:37:18.065 INFO:tasks.rados.rados.0.smithi045.stdout:3023: oid 42 version 2371 is already newer than 2370 2024-04-26T00:37:18.065 INFO:tasks.rados.rados.0.smithi045.stdout:3023: oid 42 version 2371 is already newer than 2369 2024-04-26T00:37:18.065 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 2371 (ObjNum 1086 snap 324 seq_num 1086) dirty exists 2024-04-26T00:37:18.065 INFO:tasks.rados.rados.0.smithi045.stdout:3023: left oid 42 (ObjNum 1086 snap 324 seq_num 1086) 2024-04-26T00:37:18.065 INFO:tasks.rados.rados.0.smithi045.stdout:3025: finishing write tid 1 to smithi04526579-14 2024-04-26T00:37:18.065 INFO:tasks.rados.rados.0.smithi045.stdout:3025: finishing write tid 2 to smithi04526579-14 2024-04-26T00:37:18.065 INFO:tasks.rados.rados.0.smithi045.stdout:3021: done (5 left) 2024-04-26T00:37:18.065 INFO:tasks.rados.rados.0.smithi045.stdout:3022: done (4 left) 2024-04-26T00:37:18.065 INFO:tasks.rados.rados.0.smithi045.stdout:3023: done (3 left) 2024-04-26T00:37:18.065 INFO:tasks.rados.rados.0.smithi045.stdout:3024: done (2 left) 2024-04-26T00:37:18.065 INFO:tasks.rados.rados.0.smithi045.stdout:3026: done (1 left) 2024-04-26T00:37:18.066 INFO:tasks.rados.rados.0.smithi045.stdout:3027: delete oid 24 current snap is 324 2024-04-26T00:37:18.066 INFO:tasks.rados.rados.0.smithi045.stdout:3025: finishing write tid 3 to smithi04526579-14 2024-04-26T00:37:18.066 INFO:tasks.rados.rados.0.smithi045.stdout:3025: finishing write tid 4 to smithi04526579-14 2024-04-26T00:37:18.066 INFO:tasks.rados.rados.0.smithi045.stdout:3025: oid 14 updating version 0 to 1190 2024-04-26T00:37:18.066 INFO:tasks.rados.rados.0.smithi045.stdout:3025: oid 14 updating version 1190 to 1192 2024-04-26T00:37:18.066 INFO:tasks.rados.rados.0.smithi045.stdout:3025: oid 14 version 1192 is already newer than 1191 2024-04-26T00:37:18.066 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 1192 (ObjNum 1087 snap 324 seq_num 1087) dirty exists 2024-04-26T00:37:18.066 INFO:tasks.rados.rados.0.smithi045.stdout:3025: left oid 14 (ObjNum 1087 snap 324 seq_num 1087) 2024-04-26T00:37:18.074 INFO:tasks.rados.rados.0.smithi045.stdout:3025: done (1 left) 2024-04-26T00:37:18.074 INFO:tasks.rados.rados.0.smithi045.stdout:3027: done (0 left) 2024-04-26T00:37:18.074 INFO:tasks.rados.rados.0.smithi045.stdout:3028: write oid 47 current snap is 324 2024-04-26T00:37:18.074 INFO:tasks.rados.rados.0.smithi045.stdout:3028: seq_num 1090 ranges {719630=659889,2117608=202022} 2024-04-26T00:37:18.096 INFO:tasks.rados.rados.0.smithi045.stdout:3028: writing smithi04526579-47 from 719630 to 1379519 tid 1 2024-04-26T00:37:18.102 INFO:tasks.rados.rados.0.smithi045.stdout:3028: writing smithi04526579-47 from 2117608 to 2319630 tid 2 2024-04-26T00:37:18.102 INFO:tasks.rados.rados.0.smithi045.stdout:3029: delete oid 9 current snap is 324 2024-04-26T00:37:18.110 INFO:tasks.rados.rados.0.smithi045.stdout:3029: done (1 left) 2024-04-26T00:37:18.110 INFO:tasks.rados.rados.0.smithi045.stdout:3030: snap_remove snap 323 2024-04-26T00:37:18.568 INFO:tasks.rados.rados.0.smithi045.stdout:3030: done (1 left) 2024-04-26T00:37:18.568 INFO:tasks.rados.rados.0.smithi045.stdout:3031: delete oid 50 current snap is 324 2024-04-26T00:37:18.568 INFO:tasks.rados.rados.0.smithi045.stdout:3028: finishing write tid 1 to smithi04526579-47 2024-04-26T00:37:18.568 INFO:tasks.rados.rados.0.smithi045.stdout:3028: finishing write tid 2 to smithi04526579-47 2024-04-26T00:37:18.569 INFO:tasks.rados.rados.0.smithi045.stdout:3028: finishing write tid 3 to smithi04526579-47 2024-04-26T00:37:18.569 INFO:tasks.rados.rados.0.smithi045.stdout:3028: finishing write tid 4 to smithi04526579-47 2024-04-26T00:37:18.569 INFO:tasks.rados.rados.0.smithi045.stdout:3028: oid 47 updating version 0 to 2327 2024-04-26T00:37:18.569 INFO:tasks.rados.rados.0.smithi045.stdout:3028: oid 47 version 2327 is already newer than 2325 2024-04-26T00:37:18.569 INFO:tasks.rados.rados.0.smithi045.stdout:3028: oid 47 version 2327 is already newer than 2326 2024-04-26T00:37:18.569 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 2327 (ObjNum 1090 snap 324 seq_num 1090) dirty exists 2024-04-26T00:37:18.569 INFO:tasks.rados.rados.0.smithi045.stdout:3028: left oid 47 (ObjNum 1090 snap 324 seq_num 1090) 2024-04-26T00:37:18.585 INFO:tasks.rados.rados.0.smithi045.stdout:3028: done (1 left) 2024-04-26T00:37:18.585 INFO:tasks.rados.rados.0.smithi045.stdout:3031: done (0 left) 2024-04-26T00:37:18.585 INFO:tasks.rados.rados.0.smithi045.stdout:3032: rollback oid 4 current snap is 324 2024-04-26T00:37:18.585 INFO:tasks.rados.rados.0.smithi045.stdout:3032: done (0 left) 2024-04-26T00:37:18.585 INFO:tasks.rados.rados.0.smithi045.stdout:3033: write (excl) oid 39 current snap is 324 2024-04-26T00:37:18.586 INFO:tasks.rados.rados.0.smithi045.stdout:3033: seq_num 1093 ranges {770051=485636,1776701=511047} 2024-04-26T00:37:18.605 INFO:tasks.rados.rados.0.smithi045.stdout:3033: writing smithi04526579-39 from 770051 to 1255687 tid 1 2024-04-26T00:37:18.619 INFO:tasks.rados.rados.0.smithi045.stdout:3033: writing smithi04526579-39 from 1776701 to 2287748 tid 2 2024-04-26T00:37:18.619 INFO:tasks.rados.rados.0.smithi045.stdout:3034: write (excl) oid 42 current snap is 324 2024-04-26T00:37:18.619 INFO:tasks.rados.rados.0.smithi045.stdout:3034: seq_num 1094 ranges {786858=560885,1835183=551675} 2024-04-26T00:37:18.626 INFO:tasks.rados.rados.0.smithi045.stdout:3034: writing smithi04526579-42 from 786858 to 1347743 tid 1 2024-04-26T00:37:18.633 INFO:tasks.rados.rados.0.smithi045.stdout:3034: writing smithi04526579-42 from 1835183 to 2386858 tid 2 2024-04-26T00:37:18.633 INFO:tasks.rados.rados.0.smithi045.stdout:3035: write (excl) oid 32 current snap is 324 2024-04-26T00:37:18.633 INFO:tasks.rados.rados.0.smithi045.stdout:3035: seq_num 1095 ranges {403665=752487,1610018=514716} 2024-04-26T00:37:18.644 INFO:tasks.rados.rados.0.smithi045.stdout:3035: writing smithi04526579-32 from 403665 to 1156152 tid 1 2024-04-26T00:37:18.653 INFO:tasks.rados.rados.0.smithi045.stdout:3035: writing smithi04526579-32 from 1610018 to 2124734 tid 2 2024-04-26T00:37:18.653 INFO:tasks.rados.rados.0.smithi045.stdout:3036: copy_from oid 40 from oid 44 current snap is 324 2024-04-26T00:37:18.653 INFO:tasks.rados.rados.0.smithi045.stdout:3033: finishing write tid 1 to smithi04526579-39 2024-04-26T00:37:18.653 INFO:tasks.rados.rados.0.smithi045.stdout:3033: finishing write tid 2 to smithi04526579-39 2024-04-26T00:37:18.653 INFO:tasks.rados.rados.0.smithi045.stdout:3033: finishing write tid 3 to smithi04526579-39 2024-04-26T00:37:18.653 INFO:tasks.rados.rados.0.smithi045.stdout:3033: finishing write tid 4 to smithi04526579-39 2024-04-26T00:37:18.653 INFO:tasks.rados.rados.0.smithi045.stdout:3033: oid 39 updating version 0 to 2367 2024-04-26T00:37:18.653 INFO:tasks.rados.rados.0.smithi045.stdout:3033: oid 39 updating version 2367 to 2368 2024-04-26T00:37:18.653 INFO:tasks.rados.rados.0.smithi045.stdout:3033: oid 39 version 2368 is already newer than 2366 2024-04-26T00:37:18.653 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 2368 (ObjNum 1093 snap 324 seq_num 1093) dirty exists 2024-04-26T00:37:18.653 INFO:tasks.rados.rados.0.smithi045.stdout:3033: left oid 39 (ObjNum 1093 snap 324 seq_num 1093) 2024-04-26T00:37:18.653 INFO:tasks.rados.rados.0.smithi045.stdout:3034: finishing write tid 1 to smithi04526579-42 2024-04-26T00:37:18.653 INFO:tasks.rados.rados.0.smithi045.stdout:3034: finishing write tid 2 to smithi04526579-42 2024-04-26T00:37:18.654 INFO:tasks.rados.rados.0.smithi045.stdout:3033: done (3 left) 2024-04-26T00:37:18.654 INFO:tasks.rados.rados.0.smithi045.stdout:3037: rollback oid 15 current snap is 324 2024-04-26T00:37:18.654 INFO:tasks.rados.rados.0.smithi045.stdout:3037: done (3 left) 2024-04-26T00:37:18.654 INFO:tasks.rados.rados.0.smithi045.stdout:3038: read oid 36 snap -1 2024-04-26T00:37:18.654 INFO:tasks.rados.rados.0.smithi045.stdout:3038: expect (ObjNum 1051 snap 317 seq_num 1051) 2024-04-26T00:37:18.654 INFO:tasks.rados.rados.0.smithi045.stdout:3039: rollback oid 35 current snap is 324 2024-04-26T00:37:18.654 INFO:tasks.rados.rados.0.smithi045.stdout:3039: done (4 left) 2024-04-26T00:37:18.654 INFO:tasks.rados.rados.0.smithi045.stdout:3040: snap_create 2024-04-26T00:37:18.660 INFO:tasks.rados.rados.0.smithi045.stdout:3034: finishing write tid 3 to smithi04526579-42 2024-04-26T00:37:18.661 INFO:tasks.rados.rados.0.smithi045.stdout:3034: finishing write tid 4 to smithi04526579-42 2024-04-26T00:37:18.661 INFO:tasks.rados.rados.0.smithi045.stdout:3034: oid 42 updating version 0 to 2374 2024-04-26T00:37:18.661 INFO:tasks.rados.rados.0.smithi045.stdout:3034: oid 42 version 2374 is already newer than 2372 2024-04-26T00:37:18.661 INFO:tasks.rados.rados.0.smithi045.stdout:3034: oid 42 version 2374 is already newer than 2373 2024-04-26T00:37:18.661 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 2374 (ObjNum 1094 snap 324 seq_num 1094) dirty exists 2024-04-26T00:37:18.661 INFO:tasks.rados.rados.0.smithi045.stdout:3034: left oid 42 (ObjNum 1094 snap 324 seq_num 1094) 2024-04-26T00:37:18.667 INFO:tasks.rados.rados.0.smithi045.stdout:3035: finishing write tid 1 to smithi04526579-32 2024-04-26T00:37:18.667 INFO:tasks.rados.rados.0.smithi045.stdout:3035: finishing write tid 2 to smithi04526579-32 2024-04-26T00:37:18.668 INFO:tasks.rados.rados.0.smithi045.stdout:3038: expect (ObjNum 1051 snap 317 seq_num 1051) 2024-04-26T00:37:18.945 INFO:tasks.rados.rados.0.smithi045.stdout:3035: finishing write tid 3 to smithi04526579-32 2024-04-26T00:37:18.945 INFO:tasks.rados.rados.0.smithi045.stdout:3035: finishing write tid 4 to smithi04526579-32 2024-04-26T00:37:18.946 INFO:tasks.rados.rados.0.smithi045.stdout:3035: oid 32 updating version 0 to 2376 2024-04-26T00:37:18.946 INFO:tasks.rados.rados.0.smithi045.stdout:3035: oid 32 updating version 2376 to 2377 2024-04-26T00:37:18.946 INFO:tasks.rados.rados.0.smithi045.stdout:3035: oid 32 version 2377 is already newer than 2375 2024-04-26T00:37:18.946 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 2377 (ObjNum 1095 snap 324 seq_num 1095) dirty exists 2024-04-26T00:37:18.946 INFO:tasks.rados.rados.0.smithi045.stdout:3035: left oid 32 (ObjNum 1095 snap 324 seq_num 1095) 2024-04-26T00:37:18.947 INFO:tasks.rados.rados.0.smithi045.stdout:3036: finishing copy_from racing read to smithi04526579-40 2024-04-26T00:37:18.947 INFO:tasks.rados.rados.0.smithi045.stdout:3036: finishing copy_from to smithi04526579-40 2024-04-26T00:37:18.947 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 2213 (ObjNum 1065 snap 320 seq_num 1065) dirty exists 2024-04-26T00:37:19.610 INFO:tasks.rados.rados.0.smithi045.stdout:3034: done (4 left) 2024-04-26T00:37:19.610 INFO:tasks.rados.rados.0.smithi045.stdout:3035: done (3 left) 2024-04-26T00:37:19.610 INFO:tasks.rados.rados.0.smithi045.stdout:3036: done (2 left) 2024-04-26T00:37:19.610 INFO:tasks.rados.rados.0.smithi045.stdout:3038: done (1 left) 2024-04-26T00:37:19.610 INFO:tasks.rados.rados.0.smithi045.stdout:3040: done (0 left) 2024-04-26T00:37:19.610 INFO:tasks.rados.rados.0.smithi045.stdout:3041: snap_remove snap 324 2024-04-26T00:37:20.611 INFO:tasks.rados.rados.0.smithi045.stdout:3041: done (0 left) 2024-04-26T00:37:20.611 INFO:tasks.rados.rados.0.smithi045.stdout:3042: rollback oid 5 current snap is 325 2024-04-26T00:37:20.611 INFO:tasks.rados.rados.0.smithi045.stdout:3042: done (0 left) 2024-04-26T00:37:20.611 INFO:tasks.rados.rados.0.smithi045.stdout:3043: read oid 17 snap -1 2024-04-26T00:37:20.611 INFO:tasks.rados.rados.0.smithi045.stdout:3043: expect deleted 2024-04-26T00:37:20.611 INFO:tasks.rados.rados.0.smithi045.stdout:3044: write oid 18 current snap is 325 2024-04-26T00:37:20.611 INFO:tasks.rados.rados.0.smithi045.stdout:3044: seq_num 1096 ranges {420472=427736,1268500=658374,2347953=72519} 2024-04-26T00:37:20.620 INFO:tasks.rados.rados.0.smithi045.stdout:3044: writing smithi04526579-18 from 420472 to 848208 tid 1 2024-04-26T00:37:20.631 INFO:tasks.rados.rados.0.smithi045.stdout:3044: writing smithi04526579-18 from 1268500 to 1926874 tid 2 2024-04-26T00:37:20.636 INFO:tasks.rados.rados.0.smithi045.stdout:3044: writing smithi04526579-18 from 2347953 to 2420472 tid 3 2024-04-26T00:37:20.636 INFO:tasks.rados.rados.0.smithi045.stdout:3045: read oid 2 snap -1 2024-04-26T00:37:20.636 INFO:tasks.rados.rados.0.smithi045.stdout:3045: expect (ObjNum 1074 snap 322 seq_num 1074) 2024-04-26T00:37:20.636 INFO:tasks.rados.rados.0.smithi045.stdout:3046: snap_create 2024-04-26T00:37:20.641 INFO:tasks.rados.rados.0.smithi045.stdout:3044: finishing write tid 1 to smithi04526579-18 2024-04-26T00:37:20.653 INFO:tasks.rados.rados.0.smithi045.stdout:3045: expect (ObjNum 1074 snap 322 seq_num 1074) 2024-04-26T00:37:20.842 INFO:tasks.rados.rados.0.smithi045.stdout:3044: finishing write tid 2 to smithi04526579-18 2024-04-26T00:37:20.843 INFO:tasks.rados.rados.0.smithi045.stdout:3044: finishing write tid 3 to smithi04526579-18 2024-04-26T00:37:20.843 INFO:tasks.rados.rados.0.smithi045.stdout:3044: finishing write tid 4 to smithi04526579-18 2024-04-26T00:37:20.843 INFO:tasks.rados.rados.0.smithi045.stdout:3044: finishing write tid 5 to smithi04526579-18 2024-04-26T00:37:20.843 INFO:tasks.rados.rados.0.smithi045.stdout:3044: oid 18 updating version 0 to 2442 2024-04-26T00:37:20.843 INFO:tasks.rados.rados.0.smithi045.stdout:3044: oid 18 updating version 2442 to 2445 2024-04-26T00:37:20.843 INFO:tasks.rados.rados.0.smithi045.stdout:3044: oid 18 version 2445 is already newer than 2444 2024-04-26T00:37:20.843 INFO:tasks.rados.rados.0.smithi045.stdout:3044: oid 18 version 2445 is already newer than 2443 2024-04-26T00:37:20.843 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 2445 (ObjNum 1096 snap 325 seq_num 1096) dirty exists 2024-04-26T00:37:20.843 INFO:tasks.rados.rados.0.smithi045.stdout:3044: left oid 18 (ObjNum 1096 snap 325 seq_num 1096) 2024-04-26T00:37:21.630 INFO:tasks.rados.rados.0.smithi045.stdout:3043: done (3 left) 2024-04-26T00:37:21.630 INFO:tasks.rados.rados.0.smithi045.stdout:3044: done (2 left) 2024-04-26T00:37:21.630 INFO:tasks.rados.rados.0.smithi045.stdout:3045: done (1 left) 2024-04-26T00:37:21.630 INFO:tasks.rados.rados.0.smithi045.stdout:3046: done (0 left) 2024-04-26T00:37:21.630 INFO:tasks.rados.rados.0.smithi045.stdout:3047: snap_remove snap 325 2024-04-26T00:37:22.637 INFO:tasks.rados.rados.0.smithi045.stdout:3047: done (0 left) 2024-04-26T00:37:22.637 INFO:tasks.rados.rados.0.smithi045.stdout:3048: read oid 22 snap -1 2024-04-26T00:37:22.637 INFO:tasks.rados.rados.0.smithi045.stdout:3048: expect deleted 2024-04-26T00:37:22.637 INFO:tasks.rados.rados.0.smithi045.stdout:3049: write (excl) oid 47 current snap is 326 2024-04-26T00:37:22.637 INFO:tasks.rados.rados.0.smithi045.stdout:3049: seq_num 1097 ranges {437279=502985,1610629=518385} 2024-04-26T00:37:22.651 INFO:tasks.rados.rados.0.smithi045.stdout:3049: writing smithi04526579-47 from 437279 to 940264 tid 1 2024-04-26T00:37:22.670 INFO:tasks.rados.rados.0.smithi045.stdout:3049: writing smithi04526579-47 from 1610629 to 2129014 tid 2 2024-04-26T00:37:22.670 INFO:tasks.rados.rados.0.smithi045.stdout:3050: copy_from oid 30 from oid 14 current snap is 326 2024-04-26T00:37:22.670 INFO:tasks.rados.rados.0.smithi045.stdout:3049: finishing write tid 1 to smithi04526579-47 2024-04-26T00:37:22.670 INFO:tasks.rados.rados.0.smithi045.stdout:3048: done (2 left) 2024-04-26T00:37:22.670 INFO:tasks.rados.rados.0.smithi045.stdout:3051: write oid 20 current snap is 326 2024-04-26T00:37:22.670 INFO:tasks.rados.rados.0.smithi045.stdout:3051: seq_num 1098 ranges {454086=578234,1669111=662043} 2024-04-26T00:37:22.687 INFO:tasks.rados.rados.0.smithi045.stdout:3051: writing smithi04526579-20 from 454086 to 1032320 tid 1 2024-04-26T00:37:22.701 INFO:tasks.rados.rados.0.smithi045.stdout:3051: writing smithi04526579-20 from 1669111 to 2331154 tid 2 2024-04-26T00:37:22.701 INFO:tasks.rados.rados.0.smithi045.stdout:3052: read oid 45 snap -1 2024-04-26T00:37:22.701 INFO:tasks.rados.rados.0.smithi045.stdout:3052: expect (ObjNum 1062 snap 318 seq_num 1062) 2024-04-26T00:37:22.701 INFO:tasks.rados.rados.0.smithi045.stdout:3053: read oid 28 snap -1 2024-04-26T00:37:22.702 INFO:tasks.rados.rados.0.smithi045.stdout:3053: expect (ObjNum 1041 snap 315 seq_num 1041) 2024-04-26T00:37:22.702 INFO:tasks.rados.rados.0.smithi045.stdout:3054: write oid 9 current snap is 326 2024-04-26T00:37:22.702 INFO:tasks.rados.rados.0.smithi045.stdout:3049: finishing write tid 2 to smithi04526579-47 2024-04-26T00:37:22.702 INFO:tasks.rados.rados.0.smithi045.stdout:3054: seq_num 1099 ranges {470893=653483,1727593=522054} 2024-04-26T00:37:22.710 INFO:tasks.rados.rados.0.smithi045.stdout:3054: writing smithi04526579-9 from 470893 to 1124376 tid 1 2024-04-26T00:37:22.722 INFO:tasks.rados.rados.0.smithi045.stdout:3054: writing smithi04526579-9 from 1727593 to 2249647 tid 2 2024-04-26T00:37:22.722 INFO:tasks.rados.rados.0.smithi045.stdout:3055: snap_create 2024-04-26T00:37:22.722 INFO:tasks.rados.rados.0.smithi045.stdout:3049: finishing write tid 3 to smithi04526579-47 2024-04-26T00:37:22.722 INFO:tasks.rados.rados.0.smithi045.stdout:3049: finishing write tid 4 to smithi04526579-47 2024-04-26T00:37:22.722 INFO:tasks.rados.rados.0.smithi045.stdout:3049: oid 47 updating version 0 to 2330 2024-04-26T00:37:22.722 INFO:tasks.rados.rados.0.smithi045.stdout:3049: oid 47 version 2330 is already newer than 2329 2024-04-26T00:37:22.722 INFO:tasks.rados.rados.0.smithi045.stdout:3049: oid 47 version 2330 is already newer than 2328 2024-04-26T00:37:22.722 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 2330 (ObjNum 1097 snap 326 seq_num 1097) dirty exists 2024-04-26T00:37:22.722 INFO:tasks.rados.rados.0.smithi045.stdout:3049: left oid 47 (ObjNum 1097 snap 326 seq_num 1097) 2024-04-26T00:37:22.722 INFO:tasks.rados.rados.0.smithi045.stdout:3050: finishing copy_from racing read to smithi04526579-30 2024-04-26T00:37:22.722 INFO:tasks.rados.rados.0.smithi045.stdout:3050: finishing copy_from to smithi04526579-30 2024-04-26T00:37:22.723 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 1192 (ObjNum 1087 snap 324 seq_num 1087) dirty exists 2024-04-26T00:37:22.723 INFO:tasks.rados.rados.0.smithi045.stdout:3051: finishing write tid 1 to smithi04526579-20 2024-04-26T00:37:22.723 INFO:tasks.rados.rados.0.smithi045.stdout:3052: expect (ObjNum 1062 snap 318 seq_num 1062) 2024-04-26T00:37:22.810 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-04-26T00:37:22.810 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-04-26T00:37:22.811 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-26T00:37:22.811 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-26T00:37:22.814 INFO:tasks.ceph.mon.b:Started 2024-04-26T00:37:22.814 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:37:22.814 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:37:22.942 INFO:tasks.rados.rados.0.smithi045.stdout:3053: expect (ObjNum 1041 snap 315 seq_num 1041) 2024-04-26T00:37:22.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.944+0000 7f3a9bb26640 10 monclient: get_monmap_and_config 2024-04-26T00:37:22.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.944+0000 7f3a9bb26640 10 monclient: build_initial_monmap 2024-04-26T00:37:22.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.944+0000 7f3a9bb26640 10 monclient: monmap: 2024-04-26T00:37:22.945 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:37:22.945 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:22.945 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:37:22.949967+0000 2024-04-26T00:37:22.945 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:37:22.949967+0000 2024-04-26T00:37:22.945 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:37:22.945 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:22.946 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:37:22.946 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:37:22.946 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:37:22.946 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:37:22.946 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:37:22.946 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:22.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.944+0000 7f3a9bb26640 1 Processor -- start 2024-04-26T00:37:22.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.944+0000 7f3a9bb26640 1 -- start start 2024-04-26T00:37:22.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.944+0000 7f3a9bb26640 10 monclient: init 2024-04-26T00:37:22.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.948+0000 7f3a9bb26640 10 monclient: _reopen_session rank -1 2024-04-26T00:37:22.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.948+0000 7f3a9bb26640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-04-26T00:37:22.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.948+0000 7f3a9bb26640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3a940a0260 addr v1:172.21.15.37:6789/0 2024-04-26T00:37:22.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.948+0000 7f3a9bb26640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f3a94137e40 0x7f3a9413a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:22.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.948+0000 7f3a9bb26640 10 monclient(hunting): picked mon.noname-a con 0x7f3a94137e40 addr v2:172.21.15.37:3300/0 2024-04-26T00:37:22.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.948+0000 7f3a9bb26640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3a9413a770 0x7f3a9413cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:22.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.948+0000 7f3a9bb26640 10 monclient(hunting): picked mon.noname-c con 0x7f3a9413a770 addr v2:172.21.15.45:3300/0 2024-04-26T00:37:22.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.948+0000 7f3a9bb26640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a940a1130 con 0x7f3a940a0260 2024-04-26T00:37:22.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.948+0000 7f3a9bb26640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:22.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.948+0000 7f3a9bb26640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f3a940a0d50 con 0x7f3a94137e40 2024-04-26T00:37:22.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.948+0000 7f3a9bb26640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:22.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.948+0000 7f3a9bb26640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f3a9409ec10 con 0x7f3a9413a770 2024-04-26T00:37:22.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.948+0000 7f3a9bb26640 10 monclient(hunting): _renew_subs 2024-04-26T00:37:22.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.948+0000 7f3a9bb26640 10 monclient(hunting): authenticate will time out at 2024.256197s 2024-04-26T00:37:22.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.948+0000 7f3a990c1640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f3a94137e40 0x7f3a9413a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:22.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.948+0000 7f3a998c2640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f3a940a0260 0x7f3a940a0640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:47128/0 (socket says 172.21.15.37:47128) 2024-04-26T00:37:22.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.948+0000 7f3a998c2640 1 -- 172.21.15.37:0/3810858148 learned_addr learned my addr 172.21.15.37:0/3810858148 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:37:22.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.948+0000 7f3a990c1640 10 monclient(hunting): get_auth_request con 0x7f3a94137e40 auth_method 0 2024-04-26T00:37:22.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.948+0000 7f3a990c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:22.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.948+0000 7f3a990c1640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:22.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.948+0000 7f3a990c1640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:22.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.948+0000 7f3a990c1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:22.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.948+0000 7f3a990c1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:22.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.948+0000 7f3a990c1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:22.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.948+0000 7f3a9a0c3640 1 --2- 172.21.15.37:0/3810858148 >> v2:172.21.15.45:3300/0 conn(0x7f3a9413a770 0x7f3a9413cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:22.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.948+0000 7f3a9a0c3640 10 monclient(hunting): get_auth_request con 0x7f3a9413a770 auth_method 0 2024-04-26T00:37:22.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.948+0000 7f3a9a0c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:22.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.948+0000 7f3a9a0c3640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:22.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.948+0000 7f3a9a0c3640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:22.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.948+0000 7f3a990c1640 10 monclient(hunting): handle_auth_done global_id 144789 payload 274 2024-04-26T00:37:22.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.948+0000 7f3a990c1640 1 -- 172.21.15.37:0/3810858148 >> v2:172.21.15.45:3300/0 conn(0x7f3a9413a770 msgr2=0x7f3a9413cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:37:22.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.948+0000 7f3a990c1640 1 --2- 172.21.15.37:0/3810858148 >> v2:172.21.15.45:3300/0 conn(0x7f3a9413a770 0x7f3a9413cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:22.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.948+0000 7f3a990c1640 1 -- 172.21.15.37:0/3810858148 >> v1:172.21.15.37:6789/0 conn(0x7f3a940a0260 legacy=0x7f3a940a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:37:22.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.948+0000 7f3a990c1640 10 monclient: _finish_hunting 0 2024-04-26T00:37:22.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.948+0000 7f3a990c1640 1 monclient: found mon.noname-a 2024-04-26T00:37:22.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.948+0000 7f3a990c1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:37:22.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.948+0000 7f3a990c1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:37:22.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.948+0000 7f3a990c1640 1 -- 172.21.15.37:0/3810858148 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a9406cff0 con 0x7f3a94137e40 2024-04-26T00:37:22.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.952+0000 7f3a990c1640 1 -- 172.21.15.37:0/3810858148 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a94137e40 msgr2=0x7f3a9413a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:22.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.952+0000 7f3a990c1640 1 --2- 172.21.15.37:0/3810858148 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a94137e40 0x7f3a9413a230 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f3a940a1a60 tx=0x7f3a9001c120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9d5c9bffe0d537e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:22.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.952+0000 7f3a988c0640 1 -- 172.21.15.37:0/3810858148 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a900b1070 con 0x7f3a94137e40 2024-04-26T00:37:22.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.952+0000 7f3a988c0640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:22.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.952+0000 7f3a988c0640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:37:22.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.952+0000 7f3a988c0640 10 monclient: dump: 2024-04-26T00:37:22.952 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:22.953 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:22.953 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:22.953 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:22.953 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:22.953 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:22.953 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:22.953 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:22.953 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:22.953 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:22.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.952+0000 7f3a988c0640 10 monclient: _finish_auth 0 2024-04-26T00:37:22.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.952+0000 7f3a988c0640 10 monclient: _check_auth_tickets 2024-04-26T00:37:22.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.952+0000 7f3a988c0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:37:22.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.952+0000 7f3a988c0640 1 -- 172.21.15.37:0/3810858148 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a900ac440 con 0x7f3a94137e40 2024-04-26T00:37:22.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.952+0000 7f3a988c0640 10 monclient: handle_config config(0 keys) 2024-04-26T00:37:22.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.952+0000 7f3a988c0640 1 -- 172.21.15.37:0/3810858148 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a900ac810 con 0x7f3a94137e40 2024-04-26T00:37:22.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.952+0000 7f3a988c0640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:22.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.952+0000 7f3a988c0640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:37:22.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.952+0000 7f3a988c0640 10 monclient: dump: 2024-04-26T00:37:22.954 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:22.954 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:22.954 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:22.954 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:22.954 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:22.954 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:22.954 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:22.954 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:22.954 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:22.954 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:22.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.952+0000 7f3a9bb26640 5 monclient: authenticate success, global_id 144789 2024-04-26T00:37:22.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.952+0000 7f3a9bb26640 10 monclient: get_monmap_and_config success 2024-04-26T00:37:22.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.952+0000 7f3a9bb26640 4 set_mon_vals no callback set 2024-04-26T00:37:22.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.952+0000 7f3a9bb26640 10 monclient: shutdown 2024-04-26T00:37:22.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.952+0000 7f3a9bb26640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:37:22.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.952+0000 7f3a9bb26640 1 -- 172.21.15.37:0/3810858148 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a94137e40 msgr2=0x7f3a9413a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:22.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.952+0000 7f3a9bb26640 1 --2- 172.21.15.37:0/3810858148 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a94137e40 0x7f3a9413a230 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f3a940a1a60 tx=0x7f3a9001c120 comp rx=0 tx=0).stop 2024-04-26T00:37:22.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.952+0000 7f3a9bb26640 1 -- 172.21.15.37:0/3810858148 shutdown_connections 2024-04-26T00:37:22.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.952+0000 7f3a9bb26640 1 --2- 172.21.15.37:0/3810858148 >> v2:172.21.15.45:3300/0 conn(0x7f3a9413a770 0x7f3a9413cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:22.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.952+0000 7f3a9bb26640 1 --2- 172.21.15.37:0/3810858148 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a94137e40 0x7f3a9413a230 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:22.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.952+0000 7f3a9bb26640 1 -- 172.21.15.37:0/3810858148 >> 172.21.15.37:0/3810858148 conn(0x7f3a940a1ad0 msgr2=0x7f3a940a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:37:22.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.952+0000 7f3a9bb26640 1 -- 172.21.15.37:0/3810858148 shutdown_connections 2024-04-26T00:37:22.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.952+0000 7f3a9bb26640 1 -- 172.21.15.37:0/3810858148 wait complete. 2024-04-26T00:37:22.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.952+0000 7f3a9bb26640 10 monclient: build_initial_monmap 2024-04-26T00:37:22.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.952+0000 7f3a9bb26640 10 monclient: monmap: 2024-04-26T00:37:22.960 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:37:22.961 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:37:22.961 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:37:22.961 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:37:22.961 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:37:22.961 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:37:22.961 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:37:22.961 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:37:22.961 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:37:22.961 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:22.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.952+0000 7f3a9bb26640 1 Processor -- start 2024-04-26T00:37:22.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.952+0000 7f3a9bb26640 1 -- start start 2024-04-26T00:37:22.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.952+0000 7f3a9bb26640 10 monclient: init 2024-04-26T00:37:22.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9bb26640 10 monclient: _reopen_session rank -1 2024-04-26T00:37:22.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9bb26640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:37:22.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9bb26640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a940a0260 0x7f3a941350c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:22.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9bb26640 10 monclient(hunting): picked mon.noname-b con 0x7f3a940a0260 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:37:22.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9bb26640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3a94137e40 0x7f3a94135600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:22.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9bb26640 10 monclient(hunting): picked mon.noname-c con 0x7f3a94137e40 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:37:22.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9bb26640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a9413a770 0x7f3a941cb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:22.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9bb26640 10 monclient(hunting): picked mon.noname-a con 0x7f3a9413a770 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:37:22.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9bb26640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:22.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9bb26640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f3a9409f0f0 con 0x7f3a9413a770 2024-04-26T00:37:22.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9bb26640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:22.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9bb26640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3a940a00a0 con 0x7f3a940a0260 2024-04-26T00:37:22.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9bb26640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:22.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9bb26640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f3a9409f4c0 con 0x7f3a94137e40 2024-04-26T00:37:22.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9bb26640 10 monclient(hunting): _renew_subs 2024-04-26T00:37:22.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9bb26640 10 monclient(hunting): authenticate will time out at 2024.264434s 2024-04-26T00:37:22.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a990c1640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3a94137e40 0x7f3a94135600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:22.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9a0c3640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a9413a770 0x7f3a941cb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:22.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a990c1640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3a94137e40 0x7f3a94135600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:34476/0 (socket says 172.21.15.37:34476) 2024-04-26T00:37:22.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9a0c3640 10 monclient(hunting): get_auth_request con 0x7f3a9413a770 auth_method 0 2024-04-26T00:37:22.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a990c1640 1 -- 172.21.15.37:0/3060223878 learned_addr learned my addr 172.21.15.37:0/3060223878 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:37:22.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9a0c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:22.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9a0c3640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:22.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9a0c3640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:22.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a990c1640 10 monclient(hunting): get_auth_request con 0x7f3a94137e40 auth_method 0 2024-04-26T00:37:22.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a990c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:22.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a990c1640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:22.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a990c1640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:22.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9a0c3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:22.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9a0c3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:22.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9a0c3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:22.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a998c2640 1 --2- 172.21.15.37:0/3060223878 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a940a0260 0x7f3a941350c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:22.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a990c1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:22.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a990c1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:22.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a990c1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:22.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a998c2640 10 monclient(hunting): get_auth_request con 0x7f3a940a0260 auth_method 0 2024-04-26T00:37:22.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a998c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:22.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a998c2640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:22.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a998c2640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:22.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9a0c3640 10 monclient(hunting): handle_auth_done global_id 144792 payload 824 2024-04-26T00:37:22.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a998c2640 1 -- 172.21.15.37:0/3060223878 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a940a0260 msgr2=0x7f3a941350c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:37:22.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a998c2640 1 -- 172.21.15.37:0/3060223878 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a940a0260 msgr2=0x7f3a941350c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:37:22.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9a0c3640 1 -- 172.21.15.37:0/3060223878 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3a94137e40 msgr2=0x7f3a94135600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:22.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a998c2640 1 --2- 172.21.15.37:0/3060223878 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a940a0260 0x7f3a941350c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:37:22.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9a0c3640 1 --2- 172.21.15.37:0/3060223878 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3a94137e40 0x7f3a94135600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:22.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9a0c3640 1 -- 172.21.15.37:0/3060223878 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a940a0260 msgr2=0x7f3a941350c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:22.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a998c2640 1 --2- 172.21.15.37:0/3060223878 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a940a0260 0x7f3a941350c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:37:22.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9a0c3640 1 --2- 172.21.15.37:0/3060223878 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a940a0260 0x7f3a941350c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:22.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9a0c3640 10 monclient: _finish_hunting 0 2024-04-26T00:37:22.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9a0c3640 1 monclient: found mon.noname-a 2024-04-26T00:37:22.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9a0c3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:37:22.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9a0c3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:37:22.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9a0c3640 1 -- 172.21.15.37:0/3060223878 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a94120b60 con 0x7f3a9413a770 2024-04-26T00:37:22.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9a0c3640 1 -- 172.21.15.37:0/3060223878 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a9413a770 msgr2=0x7f3a941cb8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:22.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9a0c3640 1 --2- 172.21.15.37:0/3060223878 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a9413a770 0x7f3a941cb8d0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c007d90 tx=0x7f3a8c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:22.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a8a7fc640 1 -- 172.21.15.37:0/3060223878 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a8c0039a0 con 0x7f3a9413a770 2024-04-26T00:37:22.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a8a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:22.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a8a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:37:22.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a8a7fc640 10 monclient: dump: 2024-04-26T00:37:22.966 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:22.966 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:22.966 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:22.966 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:22.966 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:22.966 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:22.966 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:22.966 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:22.966 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:22.966 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:22.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a8a7fc640 10 monclient: _finish_auth 0 2024-04-26T00:37:22.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a8a7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:37:22.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a8a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:37:22.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a8a7fc640 1 -- 172.21.15.37:0/3060223878 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a8c003b40 con 0x7f3a9413a770 2024-04-26T00:37:22.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a8a7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:37:22.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a8a7fc640 1 -- 172.21.15.37:0/3060223878 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a8c004020 con 0x7f3a9413a770 2024-04-26T00:37:22.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9bb26640 5 monclient: authenticate success, global_id 144792 2024-04-26T00:37:22.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9bb26640 10 monclient: _renew_subs 2024-04-26T00:37:22.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9bb26640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:37:22.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9bb26640 1 -- 172.21.15.37:0/3060223878 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a941211d0 con 0x7f3a9413a770 2024-04-26T00:37:22.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a8bfff640 4 set_mon_vals no callback set 2024-04-26T00:37:22.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a8a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:22.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a8a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:37:22.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a8a7fc640 10 monclient: dump: 2024-04-26T00:37:22.967 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:22.967 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:22.967 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:22.967 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:22.968 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:22.968 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:22.968 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:22.968 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:22.968 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:22.968 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:22.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9bb26640 10 monclient: _renew_subs 2024-04-26T00:37:22.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9bb26640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:37:22.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.956+0000 7f3a9bb26640 1 -- 172.21.15.37:0/3060223878 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a941cbe10 con 0x7f3a9413a770 2024-04-26T00:37:22.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.960+0000 7f3a9bb26640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:22.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.960+0000 7f3a9bb26640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:22.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.960+0000 7f3a9bb26640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:37:22.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.960+0000 7f3a9bb26640 1 -- 172.21.15.37:0/3060223878 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a94066f40 con 0x7f3a9413a770 2024-04-26T00:37:22.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.960+0000 7f3a8a7fc640 1 -- 172.21.15.37:0/3060223878 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f3a8c01b020 con 0x7f3a9413a770 2024-04-26T00:37:22.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.960+0000 7f3a8a7fc640 1 --2- 172.21.15.37:0/3060223878 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3a78035e80 0x7f3a78038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:22.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.960+0000 7f3a8a7fc640 1 -- 172.21.15.37:0/3060223878 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(683..683 src has 370..683) ==== 3927+0+0 (secure 0 0 0) 0x7f3a8c048d20 con 0x7f3a9413a770 2024-04-26T00:37:22.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.960+0000 7f3a998c2640 1 --2- 172.21.15.37:0/3060223878 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3a78035e80 0x7f3a78038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:22.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.960+0000 7f3a998c2640 10 monclient: get_auth_request con 0x7f3a78035e80 auth_method 0 2024-04-26T00:37:22.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.960+0000 7f3a998c2640 1 --2- 172.21.15.37:0/3060223878 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3a78035e80 0x7f3a78038340 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f3a84007d50 tx=0x7f3a84009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:22.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.964+0000 7f3a8a7fc640 1 -- 172.21.15.37:0/3060223878 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f3a8c014420 con 0x7f3a9413a770 2024-04-26T00:37:22.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.964+0000 7f3a8a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:22.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:22.964+0000 7f3a8a7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:37:23.056 INFO:tasks.rados.rados.0.smithi045.stdout:3051: finishing write tid 2 to smithi04526579-20 2024-04-26T00:37:23.056 INFO:tasks.rados.rados.0.smithi045.stdout:3051: finishing write tid 3 to smithi04526579-20 2024-04-26T00:37:23.056 INFO:tasks.rados.rados.0.smithi045.stdout:3051: finishing write tid 4 to smithi04526579-20 2024-04-26T00:37:23.056 INFO:tasks.rados.rados.0.smithi045.stdout:3051: oid 20 updating version 0 to 2370 2024-04-26T00:37:23.056 INFO:tasks.rados.rados.0.smithi045.stdout:3051: oid 20 updating version 2370 to 2371 2024-04-26T00:37:23.056 INFO:tasks.rados.rados.0.smithi045.stdout:3051: oid 20 version 2371 is already newer than 2369 2024-04-26T00:37:23.056 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 2371 (ObjNum 1098 snap 326 seq_num 1098) dirty exists 2024-04-26T00:37:23.057 INFO:tasks.rados.rados.0.smithi045.stdout:3051: left oid 20 (ObjNum 1098 snap 326 seq_num 1098) 2024-04-26T00:37:23.057 INFO:tasks.rados.rados.0.smithi045.stdout:3054: finishing write tid 1 to smithi04526579-9 2024-04-26T00:37:23.057 INFO:tasks.rados.rados.0.smithi045.stdout:3054: finishing write tid 2 to smithi04526579-9 2024-04-26T00:37:23.057 INFO:tasks.rados.rados.0.smithi045.stdout:3054: finishing write tid 3 to smithi04526579-9 2024-04-26T00:37:23.057 INFO:tasks.rados.rados.0.smithi045.stdout:3054: finishing write tid 4 to smithi04526579-9 2024-04-26T00:37:23.057 INFO:tasks.rados.rados.0.smithi045.stdout:3054: oid 9 updating version 0 to 2041 2024-04-26T00:37:23.057 INFO:tasks.rados.rados.0.smithi045.stdout:3054: oid 9 updating version 2041 to 2043 2024-04-26T00:37:23.057 INFO:tasks.rados.rados.0.smithi045.stdout:3054: oid 9 version 2043 is already newer than 2042 2024-04-26T00:37:23.057 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 2043 (ObjNum 1099 snap 326 seq_num 1099) dirty exists 2024-04-26T00:37:23.057 INFO:tasks.rados.rados.0.smithi045.stdout:3054: left oid 9 (ObjNum 1099 snap 326 seq_num 1099) 2024-04-26T00:37:23.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:23.184+0000 7f3a9bb26640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:37:23.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:23.184+0000 7f3a9bb26640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:37:23.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:23.184+0000 7f3a9bb26640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:37:23.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:23.184+0000 7f3a9bb26640 1 -- 172.21.15.37:0/3060223878 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3a9409e490 con 0x7f3a9413a770 2024-04-26T00:37:23.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:23.184+0000 7f3a8a7fc640 1 -- 172.21.15.37:0/3060223878 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7f3a8c01cc70 con 0x7f3a9413a770 2024-04-26T00:37:23.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:23.188+0000 7f3a8a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:37:23.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:23.188+0000 7f3a8a7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:37:23.189 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":130,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:37:23.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:23.188+0000 7f3a9bb26640 1 -- 172.21.15.37:0/3060223878 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3a78035e80 msgr2=0x7f3a78038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:23.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:23.188+0000 7f3a9bb26640 1 --2- 172.21.15.37:0/3060223878 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3a78035e80 0x7f3a78038340 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f3a84007d50 tx=0x7f3a84009040 comp rx=0 tx=0).stop 2024-04-26T00:37:23.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:23.188+0000 7f3a9bb26640 10 monclient: shutdown 2024-04-26T00:37:23.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:23.188+0000 7f3a9bb26640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:37:23.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:23.188+0000 7f3a9bb26640 1 -- 172.21.15.37:0/3060223878 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a9413a770 msgr2=0x7f3a941cb8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:23.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:23.188+0000 7f3a9bb26640 1 --2- 172.21.15.37:0/3060223878 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a9413a770 0x7f3a941cb8d0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c007d90 tx=0x7f3a8c00e4b0 comp rx=0 tx=0).stop 2024-04-26T00:37:23.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:23.188+0000 7f3a9bb26640 1 -- 172.21.15.37:0/3060223878 shutdown_connections 2024-04-26T00:37:23.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:23.188+0000 7f3a9bb26640 1 --2- 172.21.15.37:0/3060223878 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3a78035e80 0x7f3a78038340 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:23.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:23.188+0000 7f3a9bb26640 1 --2- 172.21.15.37:0/3060223878 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a9413a770 0x7f3a941cb8d0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:23.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:23.188+0000 7f3a9bb26640 1 --2- 172.21.15.37:0/3060223878 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3a94137e40 0x7f3a94135600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:23.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:23.188+0000 7f3a9bb26640 1 --2- 172.21.15.37:0/3060223878 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a940a0260 0x7f3a941350c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:23.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:23.188+0000 7f3a9bb26640 1 -- 172.21.15.37:0/3060223878 >> 172.21.15.37:0/3060223878 conn(0x7f3a940a1ad0 msgr2=0x7f3a940a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:37:23.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:23.188+0000 7f3a9bb26640 1 -- 172.21.15.37:0/3060223878 shutdown_connections 2024-04-26T00:37:23.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:23.188+0000 7f3a9bb26640 1 -- 172.21.15.37:0/3060223878 wait complete. 2024-04-26T00:37:23.569 INFO:tasks.rados.rados.0.smithi045.stdout:3049: done (6 left) 2024-04-26T00:37:23.569 INFO:tasks.rados.rados.0.smithi045.stdout:3050: done (5 left) 2024-04-26T00:37:23.569 INFO:tasks.rados.rados.0.smithi045.stdout:3051: done (4 left) 2024-04-26T00:37:23.569 INFO:tasks.rados.rados.0.smithi045.stdout:3052: done (3 left) 2024-04-26T00:37:23.570 INFO:tasks.rados.rados.0.smithi045.stdout:3053: done (2 left) 2024-04-26T00:37:23.570 INFO:tasks.rados.rados.0.smithi045.stdout:3054: done (1 left) 2024-04-26T00:37:23.570 INFO:tasks.rados.rados.0.smithi045.stdout:3055: done (0 left) 2024-04-26T00:37:23.570 INFO:tasks.rados.rados.0.smithi045.stdout:3056: read oid 45 snap -1 2024-04-26T00:37:23.570 INFO:tasks.rados.rados.0.smithi045.stdout:3056: expect (ObjNum 1062 snap 318 seq_num 1062) 2024-04-26T00:37:23.570 INFO:tasks.rados.rados.0.smithi045.stdout:3057: read oid 25 snap -1 2024-04-26T00:37:23.570 INFO:tasks.rados.rados.0.smithi045.stdout:3057: expect deleted 2024-04-26T00:37:23.570 INFO:tasks.rados.rados.0.smithi045.stdout:3058: delete oid 17 current snap is 327 2024-04-26T00:37:23.579 INFO:tasks.rados.rados.0.smithi045.stdout:3057: done (2 left) 2024-04-26T00:37:23.579 INFO:tasks.rados.rados.0.smithi045.stdout:3058: done (1 left) 2024-04-26T00:37:23.579 INFO:tasks.rados.rados.0.smithi045.stdout:3059: rollback oid 11 current snap is 327 2024-04-26T00:37:23.579 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 11 to 326 2024-04-26T00:37:23.579 INFO:tasks.rados.rados.0.smithi045.stdout:3060: snap_create 2024-04-26T00:37:23.580 INFO:tasks.rados.rados.0.smithi045.stdout:3056: expect (ObjNum 1062 snap 318 seq_num 1062) 2024-04-26T00:37:23.855 INFO:tasks.rados.rados.0.smithi045.stdout:3059: finishing rollback tid 1 to smithi04526579-11 2024-04-26T00:37:23.855 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 0 (ObjNum 289 snap 0 seq_num 3719237521) dirty dne 2024-04-26T00:37:24.572 INFO:tasks.rados.rados.0.smithi045.stdout:3056: done (2 left) 2024-04-26T00:37:24.572 INFO:tasks.rados.rados.0.smithi045.stdout:3059: done (1 left) 2024-04-26T00:37:24.572 INFO:tasks.rados.rados.0.smithi045.stdout:3060: done (0 left) 2024-04-26T00:37:24.572 INFO:tasks.rados.rados.0.smithi045.stdout:3061: read oid 28 snap 327 2024-04-26T00:37:24.572 INFO:tasks.rados.rados.0.smithi045.stdout:3061: expect (ObjNum 1041 snap 315 seq_num 1041) 2024-04-26T00:37:24.572 INFO:tasks.rados.rados.0.smithi045.stdout:3062: rollback oid 17 current snap is 328 2024-04-26T00:37:24.572 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 17 to 326 2024-04-26T00:37:24.573 INFO:tasks.rados.rados.0.smithi045.stdout:3063: snap_create 2024-04-26T00:37:24.577 INFO:tasks.rados.rados.0.smithi045.stdout:3062: finishing rollback tid 1 to smithi04526579-17 2024-04-26T00:37:24.577 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 0 (ObjNum 209 snap 0 seq_num 2241836384) dirty dne 2024-04-26T00:37:24.586 INFO:tasks.rados.rados.0.smithi045.stdout:3061: expect (ObjNum 1041 snap 315 seq_num 1041) 2024-04-26T00:37:25.574 INFO:tasks.rados.rados.0.smithi045.stdout:3061: done (2 left) 2024-04-26T00:37:25.574 INFO:tasks.rados.rados.0.smithi045.stdout:3062: done (1 left) 2024-04-26T00:37:25.574 INFO:tasks.rados.rados.0.smithi045.stdout:3063: done (0 left) 2024-04-26T00:37:25.574 INFO:tasks.rados.rados.0.smithi045.stdout:3064: read oid 47 snap 327 2024-04-26T00:37:25.574 INFO:tasks.rados.rados.0.smithi045.stdout:3064: expect (ObjNum 1097 snap 326 seq_num 1097) 2024-04-26T00:37:25.575 INFO:tasks.rados.rados.0.smithi045.stdout:3065: read oid 20 snap -1 2024-04-26T00:37:25.575 INFO:tasks.rados.rados.0.smithi045.stdout:3065: expect (ObjNum 1098 snap 326 seq_num 1098) 2024-04-26T00:37:25.575 INFO:tasks.rados.rados.0.smithi045.stdout:3066: read oid 13 snap 327 2024-04-26T00:37:25.575 INFO:tasks.rados.rados.0.smithi045.stdout:3066: expect (ObjNum 1043 snap 316 seq_num 1043) 2024-04-26T00:37:25.575 INFO:tasks.rados.rados.0.smithi045.stdout:3067: write (excl) oid 32 current snap is 329 2024-04-26T00:37:25.575 INFO:tasks.rados.rados.0.smithi045.stdout:3067: seq_num 1101 ranges {504507=403981,1328204=409370,2252362=252145} 2024-04-26T00:37:25.591 INFO:tasks.rados.rados.0.smithi045.stdout:3067: writing smithi04526579-32 from 504507 to 908488 tid 1 2024-04-26T00:37:25.606 INFO:tasks.rados.rados.0.smithi045.stdout:3067: writing smithi04526579-32 from 1328204 to 1737574 tid 2 2024-04-26T00:37:25.619 INFO:tasks.rados.rados.0.smithi045.stdout:3067: writing smithi04526579-32 from 2252362 to 2504507 tid 3 2024-04-26T00:37:25.619 INFO:tasks.rados.rados.0.smithi045.stdout:3068: copy_from oid 27 from oid 36 current snap is 329 2024-04-26T00:37:25.620 INFO:tasks.rados.rados.0.smithi045.stdout:3064: expect (ObjNum 1097 snap 326 seq_num 1097) 2024-04-26T00:37:25.808 INFO:tasks.rados.rados.0.smithi045.stdout:3066: expect (ObjNum 1043 snap 316 seq_num 1043) 2024-04-26T00:37:25.999 INFO:tasks.rados.rados.0.smithi045.stdout:3065: expect (ObjNum 1098 snap 326 seq_num 1098) 2024-04-26T00:37:26.113 INFO:tasks.rados.rados.0.smithi045.stdout:3067: finishing write tid 1 to smithi04526579-32 2024-04-26T00:37:26.113 INFO:tasks.rados.rados.0.smithi045.stdout:3067: finishing write tid 2 to smithi04526579-32 2024-04-26T00:37:26.113 INFO:tasks.rados.rados.0.smithi045.stdout:3067: finishing write tid 3 to smithi04526579-32 2024-04-26T00:37:26.113 INFO:tasks.rados.rados.0.smithi045.stdout:3067: finishing write tid 4 to smithi04526579-32 2024-04-26T00:37:26.113 INFO:tasks.rados.rados.0.smithi045.stdout:3067: finishing write tid 5 to smithi04526579-32 2024-04-26T00:37:26.114 INFO:tasks.rados.rados.0.smithi045.stdout:3067: oid 32 updating version 0 to 2381 2024-04-26T00:37:26.114 INFO:tasks.rados.rados.0.smithi045.stdout:3067: oid 32 version 2381 is already newer than 2378 2024-04-26T00:37:26.114 INFO:tasks.rados.rados.0.smithi045.stdout:3067: oid 32 version 2381 is already newer than 2379 2024-04-26T00:37:26.114 INFO:tasks.rados.rados.0.smithi045.stdout:3067: oid 32 version 2381 is already newer than 2380 2024-04-26T00:37:26.114 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 2381 (ObjNum 1101 snap 329 seq_num 1101) dirty exists 2024-04-26T00:37:26.114 INFO:tasks.rados.rados.0.smithi045.stdout:3067: left oid 32 (ObjNum 1101 snap 329 seq_num 1101) 2024-04-26T00:37:26.114 INFO:tasks.rados.rados.0.smithi045.stdout:3068: finishing copy_from racing read to smithi04526579-27 2024-04-26T00:37:26.114 INFO:tasks.rados.rados.0.smithi045.stdout:3068: finishing copy_from to smithi04526579-27 2024-04-26T00:37:26.114 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 2409 (ObjNum 1051 snap 317 seq_num 1051) dirty exists 2024-04-26T00:37:26.114 INFO:tasks.rados.rados.0.smithi045.stdout:3064: done (4 left) 2024-04-26T00:37:26.114 INFO:tasks.rados.rados.0.smithi045.stdout:3065: done (3 left) 2024-04-26T00:37:26.114 INFO:tasks.rados.rados.0.smithi045.stdout:3066: done (2 left) 2024-04-26T00:37:26.114 INFO:tasks.rados.rados.0.smithi045.stdout:3067: done (1 left) 2024-04-26T00:37:26.114 INFO:tasks.rados.rados.0.smithi045.stdout:3068: done (0 left) 2024-04-26T00:37:26.115 INFO:tasks.rados.rados.0.smithi045.stdout:3069: write oid 16 current snap is 329 2024-04-26T00:37:26.115 INFO:tasks.rados.rados.0.smithi045.stdout:3069: seq_num 1102 ranges {521314=479230,1786686=669381} 2024-04-26T00:37:26.129 INFO:tasks.rados.rados.0.smithi045.stdout:3069: writing smithi04526579-16 from 521314 to 1000544 tid 1 2024-04-26T00:37:26.153 INFO:tasks.rados.rados.0.smithi045.stdout:3069: writing smithi04526579-16 from 1786686 to 2456067 tid 2 2024-04-26T00:37:26.154 INFO:tasks.rados.rados.0.smithi045.stdout:3070: copy_from oid 20 from oid 44 current snap is 329 2024-04-26T00:37:26.154 INFO:tasks.rados.rados.0.smithi045.stdout:3069: finishing write tid 1 to smithi04526579-16 2024-04-26T00:37:26.154 INFO:tasks.rados.rados.0.smithi045.stdout:3071: rollback oid 11 current snap is 329 2024-04-26T00:37:26.154 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 11 to 327 2024-04-26T00:37:26.154 INFO:tasks.rados.rados.0.smithi045.stdout:3072: write (excl) oid 8 current snap is 329 2024-04-26T00:37:26.154 INFO:tasks.rados.rados.0.smithi045.stdout:3072: seq_num 1103 ranges {538121=670832,1961521=413039} 2024-04-26T00:37:26.171 INFO:tasks.rados.rados.0.smithi045.stdout:3072: writing smithi04526579-8 from 538121 to 1208953 tid 1 2024-04-26T00:37:26.186 INFO:tasks.rados.rados.0.smithi045.stdout:3072: writing smithi04526579-8 from 1961521 to 2374560 tid 2 2024-04-26T00:37:26.186 INFO:tasks.rados.rados.0.smithi045.stdout:3073: read oid 31 snap 326 2024-04-26T00:37:26.186 INFO:tasks.rados.rados.0.smithi045.stdout:3073: expect (ObjNum 1057 snap 318 seq_num 1057) 2024-04-26T00:37:26.186 INFO:tasks.rados.rados.0.smithi045.stdout:3074: rollback oid 19 current snap is 329 2024-04-26T00:37:26.186 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 19 to 327 2024-04-26T00:37:26.186 INFO:tasks.rados.rados.0.smithi045.stdout:3071: finishing rollback tid 1 to smithi04526579-11 2024-04-26T00:37:26.186 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-04-26T00:37:26.186 INFO:tasks.rados.rados.0.smithi045.stdout:3069: finishing write tid 2 to smithi04526579-16 2024-04-26T00:37:26.186 INFO:tasks.rados.rados.0.smithi045.stdout:3069: finishing write tid 3 to smithi04526579-16 2024-04-26T00:37:26.187 INFO:tasks.rados.rados.0.smithi045.stdout:3069: finishing write tid 4 to smithi04526579-16 2024-04-26T00:37:26.187 INFO:tasks.rados.rados.0.smithi045.stdout:3069: oid 16 updating version 0 to 1193 2024-04-26T00:37:26.187 INFO:tasks.rados.rados.0.smithi045.stdout:3069: oid 16 updating version 1193 to 1194 2024-04-26T00:37:26.187 INFO:tasks.rados.rados.0.smithi045.stdout:3069: oid 16 updating version 1194 to 1195 2024-04-26T00:37:26.187 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 1195 (ObjNum 1102 snap 329 seq_num 1102) dirty exists 2024-04-26T00:37:26.187 INFO:tasks.rados.rados.0.smithi045.stdout:3069: left oid 16 (ObjNum 1102 snap 329 seq_num 1102) 2024-04-26T00:37:26.187 INFO:tasks.rados.rados.0.smithi045.stdout:3070: finishing copy_from racing read to smithi04526579-20 2024-04-26T00:37:26.188 INFO:tasks.rados.rados.0.smithi045.stdout:3072: finishing write tid 1 to smithi04526579-8 2024-04-26T00:37:26.188 INFO:tasks.rados.rados.0.smithi045.stdout:3069: done (5 left) 2024-04-26T00:37:26.188 INFO:tasks.rados.rados.0.smithi045.stdout:3071: done (4 left) 2024-04-26T00:37:26.188 INFO:tasks.rados.rados.0.smithi045.stdout:3075: write (excl) oid 34 current snap is 329 2024-04-26T00:37:26.188 INFO:tasks.rados.rados.0.smithi045.stdout:3075: seq_num 1104 ranges {554928=746081,2020003=534925} 2024-04-26T00:37:26.201 INFO:tasks.rados.rados.0.smithi045.stdout:3075: writing smithi04526579-34 from 554928 to 1301009 tid 1 2024-04-26T00:37:26.207 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:37:26.217 INFO:tasks.rados.rados.0.smithi045.stdout:3075: writing smithi04526579-34 from 2020003 to 2554928 tid 2 2024-04-26T00:37:26.218 INFO:tasks.rados.rados.0.smithi045.stdout:3076: snap_remove snap 328 2024-04-26T00:37:26.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.340+0000 7f89ea23b640 10 monclient: get_monmap_and_config 2024-04-26T00:37:26.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.340+0000 7f89ea23b640 10 monclient: build_initial_monmap 2024-04-26T00:37:26.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.340+0000 7f89ea23b640 10 monclient: monmap: 2024-04-26T00:37:26.341 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:37:26.341 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:26.341 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:37:26.345390+0000 2024-04-26T00:37:26.341 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:37:26.345390+0000 2024-04-26T00:37:26.341 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:37:26.341 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:26.341 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:37:26.342 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:37:26.342 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:37:26.342 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:37:26.342 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:37:26.342 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:26.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.340+0000 7f89ea23b640 1 Processor -- start 2024-04-26T00:37:26.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.340+0000 7f89ea23b640 1 -- start start 2024-04-26T00:37:26.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.340+0000 7f89ea23b640 10 monclient: init 2024-04-26T00:37:26.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.340+0000 7f89ea23b640 10 monclient: _reopen_session rank -1 2024-04-26T00:37:26.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.340+0000 7f89ea23b640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-04-26T00:37:26.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.340+0000 7f89ea23b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f89e40a0260 addr v1:172.21.15.45:6789/0 2024-04-26T00:37:26.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.340+0000 7f89ea23b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f89e4137e40 addr v1:172.21.15.37:6789/0 2024-04-26T00:37:26.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.340+0000 7f89ea23b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f89e413b950 0x7f89e413dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:26.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.340+0000 7f89ea23b640 10 monclient(hunting): picked mon.noname-b con 0x7f89e413b950 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:37:26.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.340+0000 7f89ea23b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89e40a1130 con 0x7f89e4137e40 2024-04-26T00:37:26.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.340+0000 7f89ea23b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89e406cff0 con 0x7f89e40a0260 2024-04-26T00:37:26.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.340+0000 7f89ea23b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:26.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.340+0000 7f89ea23b640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f89e40a0d50 con 0x7f89e413b950 2024-04-26T00:37:26.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.340+0000 7f89ea23b640 10 monclient(hunting): _renew_subs 2024-04-26T00:37:26.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.340+0000 7f89ea23b640 10 monclient(hunting): authenticate will time out at 2027.651055s 2024-04-26T00:37:26.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.340+0000 7f89e3fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f89e413b950 0x7f89e413dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:26.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.340+0000 7f89e3fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f89e413b950 0x7f89e413dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:34488/0 (socket says 172.21.15.37:34488) 2024-04-26T00:37:26.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.340+0000 7f89e3fff640 1 -- 172.21.15.37:0/403108296 learned_addr learned my addr 172.21.15.37:0/403108296 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:37:26.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.340+0000 7f89e3fff640 10 monclient(hunting): get_auth_request con 0x7f89e413b950 auth_method 0 2024-04-26T00:37:26.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.340+0000 7f89e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:26.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.340+0000 7f89e3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:26.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.340+0000 7f89e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:26.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.340+0000 7f89e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:26.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.340+0000 7f89e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:26.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.340+0000 7f89e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:26.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.340+0000 7f89e3fff640 10 monclient(hunting): handle_auth_done global_id 204176 payload 274 2024-04-26T00:37:26.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.340+0000 7f89e3fff640 1 -- 172.21.15.37:0/403108296 >> v1:172.21.15.45:6789/0 conn(0x7f89e40a0260 legacy=0x7f89e40a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:37:26.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.340+0000 7f89e3fff640 1 -- 172.21.15.37:0/403108296 >> v1:172.21.15.37:6789/0 conn(0x7f89e4137e40 legacy=0x7f89e413a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:37:26.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.340+0000 7f89e3fff640 10 monclient: _finish_hunting 0 2024-04-26T00:37:26.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.340+0000 7f89e3fff640 1 monclient: found mon.noname-b 2024-04-26T00:37:26.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.340+0000 7f89e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:37:26.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.340+0000 7f89e3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:37:26.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.340+0000 7f89e3fff640 1 -- 172.21.15.37:0/403108296 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89e409bc50 con 0x7f89e413b950 2024-04-26T00:37:26.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.344+0000 7f89e3fff640 1 -- 172.21.15.37:0/403108296 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f89e413b950 msgr2=0x7f89e413dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:26.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.344+0000 7f89e3fff640 1 --2- 172.21.15.37:0/403108296 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f89e413b950 0x7f89e413dd40 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f89dc009e80 tx=0x7f89dc01c6b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92d7c751dd60f5df server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:26.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e27fc640 1 -- 172.21.15.37:0/403108296 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89dc0a9440 con 0x7f89e413b950 2024-04-26T00:37:26.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:26.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:37:26.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e27fc640 10 monclient: dump: 2024-04-26T00:37:26.347 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:26.347 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:26.348 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:26.348 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:26.348 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:26.348 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:26.348 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:26.348 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:26.348 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:26.348 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:26.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e27fc640 10 monclient: _finish_auth 0 2024-04-26T00:37:26.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e27fc640 10 monclient: _check_auth_tickets 2024-04-26T00:37:26.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:37:26.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89ea23b640 5 monclient: authenticate success, global_id 204176 2024-04-26T00:37:26.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89ea23b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:37:26.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e27fc640 1 -- 172.21.15.37:0/403108296 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89dc0ae070 con 0x7f89e413b950 2024-04-26T00:37:26.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e27fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:37:26.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e27fc640 1 -- 172.21.15.37:0/403108296 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89dc0a9800 con 0x7f89e413b950 2024-04-26T00:37:26.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89ea23b640 10 monclient: get_monmap_and_config success 2024-04-26T00:37:26.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89ea23b640 4 set_mon_vals no callback set 2024-04-26T00:37:26.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:26.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:37:26.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e27fc640 10 monclient: dump: 2024-04-26T00:37:26.349 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:26.349 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:26.349 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:26.350 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:26.350 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:26.350 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:26.350 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:26.350 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:26.350 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:26.350 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:26.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89ea23b640 10 monclient: shutdown 2024-04-26T00:37:26.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89ea23b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:37:26.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89ea23b640 1 -- 172.21.15.37:0/403108296 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f89e413b950 msgr2=0x7f89e413dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:26.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89ea23b640 1 --2- 172.21.15.37:0/403108296 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f89e413b950 0x7f89e413dd40 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f89dc009e80 tx=0x7f89dc01c6b0 comp rx=0 tx=0).stop 2024-04-26T00:37:26.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89ea23b640 1 -- 172.21.15.37:0/403108296 shutdown_connections 2024-04-26T00:37:26.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89ea23b640 1 --2- 172.21.15.37:0/403108296 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f89e413b950 0x7f89e413dd40 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:26.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89ea23b640 1 -- 172.21.15.37:0/403108296 >> 172.21.15.37:0/403108296 conn(0x7f89e40a1ad0 msgr2=0x7f89e40a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:37:26.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89ea23b640 1 -- 172.21.15.37:0/403108296 shutdown_connections 2024-04-26T00:37:26.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89ea23b640 1 -- 172.21.15.37:0/403108296 wait complete. 2024-04-26T00:37:26.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89ea23b640 10 monclient: build_initial_monmap 2024-04-26T00:37:26.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89ea23b640 10 monclient: monmap: 2024-04-26T00:37:26.356 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:37:26.356 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:37:26.356 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:37:26.356 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:37:26.356 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:37:26.356 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:37:26.356 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:37:26.356 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:37:26.356 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:37:26.356 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:26.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89ea23b640 1 Processor -- start 2024-04-26T00:37:26.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89ea23b640 1 -- start start 2024-04-26T00:37:26.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89ea23b640 10 monclient: init 2024-04-26T00:37:26.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89ea23b640 10 monclient: _reopen_session rank -1 2024-04-26T00:37:26.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89ea23b640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:37:26.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89ea23b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89e40a0260 0x7f89e41ccdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:26.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89ea23b640 10 monclient(hunting): picked mon.noname-b con 0x7f89e40a0260 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:37:26.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89ea23b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f89e4137e40 0x7f89e41cd310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:26.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89ea23b640 10 monclient(hunting): picked mon.noname-c con 0x7f89e4137e40 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:37:26.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89ea23b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89e413b950 0x7f89e41cd850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:26.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89ea23b640 10 monclient(hunting): picked mon.noname-a con 0x7f89e413b950 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:37:26.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89ea23b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:26.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89ea23b640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f89e409f0f0 con 0x7f89e413b950 2024-04-26T00:37:26.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89ea23b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:26.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89ea23b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f89e409f4c0 con 0x7f89e40a0260 2024-04-26T00:37:26.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89ea23b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:26.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89ea23b640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f89e40a00a0 con 0x7f89e4137e40 2024-04-26T00:37:26.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89ea23b640 10 monclient(hunting): _renew_subs 2024-04-26T00:37:26.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89ea23b640 10 monclient(hunting): authenticate will time out at 2027.658856s 2024-04-26T00:37:26.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e2ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f89e4137e40 0x7f89e41cd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:26.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e3fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89e413b950 0x7f89e41cd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:26.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e2ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f89e4137e40 0x7f89e41cd310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:34498/0 (socket says 172.21.15.37:34498) 2024-04-26T00:37:26.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e2ffd640 1 -- 172.21.15.37:0/3461217207 learned_addr learned my addr 172.21.15.37:0/3461217207 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:37:26.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e2ffd640 10 monclient(hunting): get_auth_request con 0x7f89e4137e40 auth_method 0 2024-04-26T00:37:26.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:26.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:26.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:26.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e3fff640 10 monclient(hunting): get_auth_request con 0x7f89e413b950 auth_method 0 2024-04-26T00:37:26.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:26.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:26.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:26.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:26.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:26.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:26.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:26.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:26.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:26.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e37fe640 1 --2- 172.21.15.37:0/3461217207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89e40a0260 0x7f89e41ccdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:26.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e37fe640 10 monclient(hunting): get_auth_request con 0x7f89e40a0260 auth_method 0 2024-04-26T00:37:26.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:26.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:26.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:26.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e2ffd640 10 monclient(hunting): handle_auth_done global_id 204179 payload 824 2024-04-26T00:37:26.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e2ffd640 1 -- 172.21.15.37:0/3461217207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89e40a0260 msgr2=0x7f89e41ccdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:26.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e2ffd640 1 --2- 172.21.15.37:0/3461217207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89e40a0260 0x7f89e41ccdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:26.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e2ffd640 1 -- 172.21.15.37:0/3461217207 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89e413b950 msgr2=0x7f89e41cd850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:26.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e2ffd640 1 --2- 172.21.15.37:0/3461217207 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89e413b950 0x7f89e41cd850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:26.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e2ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:37:26.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e2ffd640 1 monclient: found mon.noname-c 2024-04-26T00:37:26.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:37:26.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e2ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:37:26.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.348+0000 7f89e2ffd640 1 -- 172.21.15.37:0/3461217207 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89e411e3a0 con 0x7f89e4137e40 2024-04-26T00:37:26.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.352+0000 7f89e2ffd640 1 -- 172.21.15.37:0/3461217207 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f89e4137e40 msgr2=0x7f89e41cd310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:26.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.352+0000 7f89e2ffd640 1 --2- 172.21.15.37:0/3461217207 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f89e4137e40 0x7f89e41cd310 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f89cc00c450 tx=0x7f89cc00c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:26.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.352+0000 7f89dbfff640 1 -- 172.21.15.37:0/3461217207 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89cc012a40 con 0x7f89e4137e40 2024-04-26T00:37:26.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.352+0000 7f89dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:26.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.352+0000 7f89dbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:37:26.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.352+0000 7f89dbfff640 10 monclient: dump: 2024-04-26T00:37:26.361 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:26.361 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:26.361 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:26.361 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:26.361 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:26.361 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:26.361 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:26.361 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:26.361 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:26.361 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:26.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.352+0000 7f89dbfff640 10 monclient: _finish_auth 0 2024-04-26T00:37:26.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.352+0000 7f89dbfff640 10 monclient: _check_auth_tickets 2024-04-26T00:37:26.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.352+0000 7f89dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:37:26.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.352+0000 7f89dbfff640 1 -- 172.21.15.37:0/3461217207 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89cc017070 con 0x7f89e4137e40 2024-04-26T00:37:26.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.352+0000 7f89dbfff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:37:26.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.352+0000 7f89dbfff640 1 -- 172.21.15.37:0/3461217207 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89cc012e40 con 0x7f89e4137e40 2024-04-26T00:37:26.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.352+0000 7f89ea23b640 5 monclient: authenticate success, global_id 204179 2024-04-26T00:37:26.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.352+0000 7f89ea23b640 10 monclient: _renew_subs 2024-04-26T00:37:26.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.352+0000 7f89ea23b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:37:26.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.352+0000 7f89ea23b640 1 -- 172.21.15.37:0/3461217207 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89e411e680 con 0x7f89e4137e40 2024-04-26T00:37:26.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.352+0000 7f89e17fa640 4 set_mon_vals no callback set 2024-04-26T00:37:26.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.352+0000 7f89dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:26.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.352+0000 7f89dbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:37:26.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.352+0000 7f89dbfff640 10 monclient: dump: 2024-04-26T00:37:26.362 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:26.362 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:26.362 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:26.363 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:26.363 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:26.363 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:26.363 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:26.363 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:26.363 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:26.363 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:26.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.352+0000 7f89ea23b640 10 monclient: _renew_subs 2024-04-26T00:37:26.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.352+0000 7f89ea23b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:37:26.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.352+0000 7f89ea23b640 1 -- 172.21.15.37:0/3461217207 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f89e4071e40 con 0x7f89e4137e40 2024-04-26T00:37:26.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.352+0000 7f89dbfff640 1 -- 172.21.15.37:0/3461217207 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f89cc01c460 con 0x7f89e4137e40 2024-04-26T00:37:26.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.352+0000 7f89ea23b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:26.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.352+0000 7f89ea23b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:26.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.352+0000 7f89ea23b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:37:26.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.352+0000 7f89ea23b640 1 -- 172.21.15.37:0/3461217207 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89e4066f40 con 0x7f89e4137e40 2024-04-26T00:37:26.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.352+0000 7f89dbfff640 1 --2- 172.21.15.37:0/3461217207 >> v2:172.21.15.45:6812/2379918623 conn(0x7f89c4035ed0 0x7f89c4038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:26.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.352+0000 7f89dbfff640 1 -- 172.21.15.37:0/3461217207 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(686..686 src has 370..686) ==== 3899+0+0 (secure 0 0 0) 0x7f89cc0488a0 con 0x7f89e4137e40 2024-04-26T00:37:26.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.352+0000 7f89e37fe640 1 --2- 172.21.15.37:0/3461217207 >> v2:172.21.15.45:6812/2379918623 conn(0x7f89c4035ed0 0x7f89c4038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:26.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.352+0000 7f89e37fe640 10 monclient: get_auth_request con 0x7f89c4035ed0 auth_method 0 2024-04-26T00:37:26.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.356+0000 7f89e37fe640 1 --2- 172.21.15.37:0/3461217207 >> v2:172.21.15.45:6812/2379918623 conn(0x7f89c4035ed0 0x7f89c4038390 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f89d4004e30 tx=0x7f89d4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:26.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.360+0000 7f89dbfff640 1 -- 172.21.15.37:0/3461217207 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f89cc047610 con 0x7f89e4137e40 2024-04-26T00:37:26.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.360+0000 7f89dbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:26.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.360+0000 7f89dbfff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:37:26.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.580+0000 7f89ea23b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:37:26.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.580+0000 7f89ea23b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:37:26.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.580+0000 7f89ea23b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:37:26.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.580+0000 7f89ea23b640 1 -- 172.21.15.37:0/3461217207 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f89e4068ee0 con 0x7f89e4137e40 2024-04-26T00:37:26.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.584+0000 7f89dbfff640 1 -- 172.21.15.37:0/3461217207 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f89cc037e20 con 0x7f89e4137e40 2024-04-26T00:37:26.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.584+0000 7f89dbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:37:26.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.584+0000 7f89dbfff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:37:26.584 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":132,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:37:26.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.584+0000 7f89ea23b640 1 -- 172.21.15.37:0/3461217207 >> v2:172.21.15.45:6812/2379918623 conn(0x7f89c4035ed0 msgr2=0x7f89c4038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:26.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.584+0000 7f89ea23b640 1 --2- 172.21.15.37:0/3461217207 >> v2:172.21.15.45:6812/2379918623 conn(0x7f89c4035ed0 0x7f89c4038390 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f89d4004e30 tx=0x7f89d4009040 comp rx=0 tx=0).stop 2024-04-26T00:37:26.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.584+0000 7f89ea23b640 10 monclient: shutdown 2024-04-26T00:37:26.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.584+0000 7f89ea23b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:37:26.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.584+0000 7f89ea23b640 1 -- 172.21.15.37:0/3461217207 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f89e4137e40 msgr2=0x7f89e41cd310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:26.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.584+0000 7f89ea23b640 1 --2- 172.21.15.37:0/3461217207 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f89e4137e40 0x7f89e41cd310 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f89cc00c450 tx=0x7f89cc00c910 comp rx=0 tx=0).stop 2024-04-26T00:37:26.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.584+0000 7f89ea23b640 1 -- 172.21.15.37:0/3461217207 shutdown_connections 2024-04-26T00:37:26.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.584+0000 7f89ea23b640 1 --2- 172.21.15.37:0/3461217207 >> v2:172.21.15.45:6812/2379918623 conn(0x7f89c4035ed0 0x7f89c4038390 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:26.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.584+0000 7f89ea23b640 1 --2- 172.21.15.37:0/3461217207 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89e413b950 0x7f89e41cd850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:26.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.584+0000 7f89ea23b640 1 --2- 172.21.15.37:0/3461217207 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f89e4137e40 0x7f89e41cd310 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:26.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.584+0000 7f89ea23b640 1 --2- 172.21.15.37:0/3461217207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89e40a0260 0x7f89e41ccdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:26.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.584+0000 7f89ea23b640 1 -- 172.21.15.37:0/3461217207 >> 172.21.15.37:0/3461217207 conn(0x7f89e40a1ad0 msgr2=0x7f89e40a3d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:37:26.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.584+0000 7f89ea23b640 1 -- 172.21.15.37:0/3461217207 shutdown_connections 2024-04-26T00:37:26.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.584+0000 7f89ea23b640 1 -- 172.21.15.37:0/3461217207 wait complete. 2024-04-26T00:37:26.604 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:37:26.604 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:37:26.604 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:37:26.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.720+0000 7f8bf7678640 10 monclient: get_monmap_and_config 2024-04-26T00:37:26.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.720+0000 7f8bf7678640 10 monclient: build_initial_monmap 2024-04-26T00:37:26.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.720+0000 7f8bf7678640 10 monclient: monmap: 2024-04-26T00:37:26.723 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:37:26.723 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:26.723 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:37:26.727850+0000 2024-04-26T00:37:26.724 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:37:26.727850+0000 2024-04-26T00:37:26.724 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:37:26.724 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:26.724 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:37:26.724 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:37:26.724 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:37:26.724 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:37:26.724 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:37:26.724 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:26.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf7678640 1 Processor -- start 2024-04-26T00:37:26.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf7678640 1 -- start start 2024-04-26T00:37:26.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf7678640 10 monclient: init 2024-04-26T00:37:26.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf7678640 10 monclient: _reopen_session rank -1 2024-04-26T00:37:26.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf7678640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-04-26T00:37:26.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf7678640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f8bf00a0300 0x7f8bf009fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:26.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf7678640 10 monclient(hunting): picked mon.noname-a con 0x7f8bf00a0300 addr v2:172.21.15.37:3300/0 2024-04-26T00:37:26.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf7678640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8bf00a06e0 0x7f8bf0134ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:26.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf7678640 10 monclient(hunting): picked mon.noname-b con 0x7f8bf00a06e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:37:26.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf7678640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8bf01350e0 0x7f8bf0135570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:26.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf7678640 10 monclient(hunting): picked mon.noname-c con 0x7f8bf01350e0 addr v2:172.21.15.45:3300/0 2024-04-26T00:37:26.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf7678640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:26.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf7678640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f8bf00a13f0 con 0x7f8bf00a0300 2024-04-26T00:37:26.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf7678640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:26.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf7678640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f8bf009c190 con 0x7f8bf01350e0 2024-04-26T00:37:26.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf7678640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:26.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf7678640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8bf00a1110 con 0x7f8bf00a06e0 2024-04-26T00:37:26.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf7678640 10 monclient(hunting): _renew_subs 2024-04-26T00:37:26.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf7678640 10 monclient(hunting): authenticate will time out at 2028.033546s 2024-04-26T00:37:26.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf4c13640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8bf00a06e0 0x7f8bf0134ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:26.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf5414640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f8bf00a0300 0x7f8bf009fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:26.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf4c13640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8bf00a06e0 0x7f8bf0134ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:34500/0 (socket says 172.21.15.37:34500) 2024-04-26T00:37:26.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf4c13640 1 -- 172.21.15.37:0/1153106219 learned_addr learned my addr 172.21.15.37:0/1153106219 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:37:26.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf4c13640 10 monclient(hunting): get_auth_request con 0x7f8bf00a06e0 auth_method 0 2024-04-26T00:37:26.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf4c13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:26.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf4c13640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:26.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf4c13640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:26.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf5414640 10 monclient(hunting): get_auth_request con 0x7f8bf00a0300 auth_method 0 2024-04-26T00:37:26.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf5414640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:26.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf5414640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:26.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf5414640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:26.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf4c13640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:26.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf4c13640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:26.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf4c13640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:26.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf5414640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:26.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf5414640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:26.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf5414640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:26.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf5c15640 1 --2- 172.21.15.37:0/1153106219 >> v2:172.21.15.45:3300/0 conn(0x7f8bf01350e0 0x7f8bf0135570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:26.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf5c15640 10 monclient(hunting): get_auth_request con 0x7f8bf01350e0 auth_method 0 2024-04-26T00:37:26.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf5c15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:26.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf5c15640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:26.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf5c15640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:26.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf4c13640 10 monclient(hunting): handle_auth_done global_id 204182 payload 274 2024-04-26T00:37:26.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf4c13640 1 -- 172.21.15.37:0/1153106219 >> v2:172.21.15.45:3300/0 conn(0x7f8bf01350e0 msgr2=0x7f8bf0135570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:37:26.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf4c13640 1 --2- 172.21.15.37:0/1153106219 >> v2:172.21.15.45:3300/0 conn(0x7f8bf01350e0 0x7f8bf0135570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:26.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf4c13640 1 -- 172.21.15.37:0/1153106219 >> v2:172.21.15.37:3300/0 conn(0x7f8bf00a0300 msgr2=0x7f8bf009fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:37:26.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf4c13640 1 --2- 172.21.15.37:0/1153106219 >> v2:172.21.15.37:3300/0 conn(0x7f8bf00a0300 0x7f8bf009fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:26.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf4c13640 10 monclient: _finish_hunting 0 2024-04-26T00:37:26.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf4c13640 1 monclient: found mon.noname-b 2024-04-26T00:37:26.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf4c13640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:37:26.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf4c13640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:37:26.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf4c13640 1 -- 172.21.15.37:0/1153106219 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bf00a16d0 con 0x7f8bf00a06e0 2024-04-26T00:37:26.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.724+0000 7f8bf5414640 1 --2- 172.21.15.37:0/1153106219 >> v2:172.21.15.37:3300/0 conn(0x7f8bf00a0300 0x7f8bf009fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:37:26.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.728+0000 7f8bf4c13640 1 -- 172.21.15.37:0/1153106219 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8bf00a06e0 msgr2=0x7f8bf0134ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:26.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.728+0000 7f8bf4c13640 1 --2- 172.21.15.37:0/1153106219 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8bf00a06e0 0x7f8bf0134ba0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8bf00a2da0 tx=0x7f8be001c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f81cde607e3e0b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:26.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.728+0000 7f8be7fff640 1 -- 172.21.15.37:0/1153106219 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be00b0070 con 0x7f8bf00a06e0 2024-04-26T00:37:26.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.728+0000 7f8be7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:26.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.728+0000 7f8be7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:37:26.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.728+0000 7f8be7fff640 10 monclient: dump: 2024-04-26T00:37:26.734 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:26.734 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:26.734 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:26.734 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:26.734 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:26.734 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:26.735 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:26.735 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:26.735 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:26.735 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:26.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.728+0000 7f8be7fff640 10 monclient: _finish_auth 0 2024-04-26T00:37:26.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.728+0000 7f8be7fff640 10 monclient: _check_auth_tickets 2024-04-26T00:37:26.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.728+0000 7f8be7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:37:26.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.728+0000 7f8bf7678640 5 monclient: authenticate success, global_id 204182 2024-04-26T00:37:26.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.728+0000 7f8be7fff640 1 -- 172.21.15.37:0/1153106219 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8be00ab600 con 0x7f8bf00a06e0 2024-04-26T00:37:26.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.728+0000 7f8bf7678640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:37:26.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.728+0000 7f8be7fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:37:26.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.728+0000 7f8bf7678640 10 monclient: get_monmap_and_config success 2024-04-26T00:37:26.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.728+0000 7f8bf7678640 4 set_mon_vals no callback set 2024-04-26T00:37:26.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.728+0000 7f8be7fff640 1 -- 172.21.15.37:0/1153106219 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be00ab9d0 con 0x7f8bf00a06e0 2024-04-26T00:37:26.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.728+0000 7f8be7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:26.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.728+0000 7f8be7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:37:26.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.728+0000 7f8be7fff640 10 monclient: dump: 2024-04-26T00:37:26.736 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:26.736 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:26.736 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:26.736 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:26.736 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:26.736 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:26.736 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:26.736 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:26.736 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:26.736 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:26.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.728+0000 7f8bf7678640 10 monclient: shutdown 2024-04-26T00:37:26.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.728+0000 7f8bf7678640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:37:26.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.728+0000 7f8bf7678640 1 -- 172.21.15.37:0/1153106219 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8bf00a06e0 msgr2=0x7f8bf0134ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:26.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.728+0000 7f8bf7678640 1 --2- 172.21.15.37:0/1153106219 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8bf00a06e0 0x7f8bf0134ba0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8bf00a2da0 tx=0x7f8be001c3e0 comp rx=0 tx=0).stop 2024-04-26T00:37:26.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.728+0000 7f8bf7678640 1 -- 172.21.15.37:0/1153106219 shutdown_connections 2024-04-26T00:37:26.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.728+0000 7f8bf7678640 1 --2- 172.21.15.37:0/1153106219 >> v2:172.21.15.45:3300/0 conn(0x7f8bf01350e0 0x7f8bf0135570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:26.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.728+0000 7f8bf7678640 1 --2- 172.21.15.37:0/1153106219 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8bf00a06e0 0x7f8bf0134ba0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:26.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.728+0000 7f8bf7678640 1 --2- 172.21.15.37:0/1153106219 >> v2:172.21.15.37:3300/0 conn(0x7f8bf00a0300 0x7f8bf009fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:26.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.728+0000 7f8bf7678640 1 -- 172.21.15.37:0/1153106219 >> 172.21.15.37:0/1153106219 conn(0x7f8bf00a1d50 msgr2=0x7f8bf00a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:37:26.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.728+0000 7f8bf7678640 1 -- 172.21.15.37:0/1153106219 shutdown_connections 2024-04-26T00:37:26.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.728+0000 7f8bf7678640 1 -- 172.21.15.37:0/1153106219 wait complete. 2024-04-26T00:37:26.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf7678640 10 monclient: build_initial_monmap 2024-04-26T00:37:26.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf7678640 10 monclient: monmap: 2024-04-26T00:37:26.737 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:37:26.737 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:37:26.737 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:37:26.737 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:37:26.737 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:37:26.738 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:37:26.738 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:37:26.738 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:37:26.738 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:37:26.738 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:26.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf7678640 1 Processor -- start 2024-04-26T00:37:26.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf7678640 1 -- start start 2024-04-26T00:37:26.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf7678640 10 monclient: init 2024-04-26T00:37:26.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf7678640 10 monclient: _reopen_session rank -1 2024-04-26T00:37:26.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf7678640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:37:26.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf7678640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8bf00a0300 0x7f8bf01ccd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:26.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf7678640 10 monclient(hunting): picked mon.noname-a con 0x7f8bf00a0300 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:37:26.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf7678640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8bf00a06e0 0x7f8bf01cd280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:26.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf7678640 10 monclient(hunting): picked mon.noname-c con 0x7f8bf00a06e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:37:26.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf7678640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bf01350e0 0x7f8bf01cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:26.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf7678640 10 monclient(hunting): picked mon.noname-b con 0x7f8bf01350e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:37:26.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf7678640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:26.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf7678640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf013e6f0 con 0x7f8bf00a0300 2024-04-26T00:37:26.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf7678640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:26.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf7678640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf013e870 con 0x7f8bf01350e0 2024-04-26T00:37:26.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf7678640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:26.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf7678640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8bf013e9f0 con 0x7f8bf00a06e0 2024-04-26T00:37:26.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf7678640 10 monclient(hunting): _renew_subs 2024-04-26T00:37:26.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf7678640 10 monclient(hunting): authenticate will time out at 2028.041154s 2024-04-26T00:37:26.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf4c13640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8bf00a06e0 0x7f8bf01cd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:26.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf5414640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8bf00a0300 0x7f8bf01ccd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:26.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf5414640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8bf00a0300 0x7f8bf01ccd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:45456/0 (socket says 172.21.15.37:45456) 2024-04-26T00:37:26.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf5414640 1 -- 172.21.15.37:0/3313177234 learned_addr learned my addr 172.21.15.37:0/3313177234 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:37:26.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf5414640 10 monclient(hunting): get_auth_request con 0x7f8bf00a0300 auth_method 0 2024-04-26T00:37:26.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf5414640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:26.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf5414640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:26.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf5414640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:26.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf4c13640 10 monclient(hunting): get_auth_request con 0x7f8bf00a06e0 auth_method 0 2024-04-26T00:37:26.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf4c13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:26.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf4c13640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:26.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf4c13640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:26.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf4c13640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:26.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf4c13640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:26.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf4c13640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:26.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf5414640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:26.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf5414640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:26.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf5414640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:26.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf5c15640 1 --2- 172.21.15.37:0/3313177234 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bf01350e0 0x7f8bf01cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:26.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf5c15640 10 monclient(hunting): get_auth_request con 0x7f8bf01350e0 auth_method 0 2024-04-26T00:37:26.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf5c15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:26.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf5c15640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:26.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf5c15640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:26.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf4c13640 10 monclient(hunting): handle_auth_done global_id 204185 payload 824 2024-04-26T00:37:26.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf4c13640 1 -- 172.21.15.37:0/3313177234 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bf01350e0 msgr2=0x7f8bf01cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:26.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf4c13640 1 --2- 172.21.15.37:0/3313177234 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bf01350e0 0x7f8bf01cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:26.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf4c13640 1 -- 172.21.15.37:0/3313177234 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8bf00a0300 msgr2=0x7f8bf01ccd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:26.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf4c13640 1 --2- 172.21.15.37:0/3313177234 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8bf00a0300 0x7f8bf01ccd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:26.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf4c13640 10 monclient: _finish_hunting 0 2024-04-26T00:37:26.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf4c13640 1 monclient: found mon.noname-c 2024-04-26T00:37:26.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf4c13640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:37:26.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf4c13640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:37:26.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf4c13640 1 -- 172.21.15.37:0/3313177234 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bf0138e90 con 0x7f8bf00a06e0 2024-04-26T00:37:26.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.732+0000 7f8bf5414640 1 --2- 172.21.15.37:0/3313177234 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8bf00a0300 0x7f8bf01ccd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:37:26.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8bf4c13640 1 -- 172.21.15.37:0/3313177234 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8bf00a06e0 msgr2=0x7f8bf01cd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:26.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8bf4c13640 1 --2- 172.21.15.37:0/3313177234 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8bf00a06e0 0x7f8bf01cd280 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8bf00a1290 tx=0x7f8be0019180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:26.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8be5ffb640 1 -- 172.21.15.37:0/3313177234 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be00b0070 con 0x7f8bf00a06e0 2024-04-26T00:37:26.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8be5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:26.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8be5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:37:26.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8be5ffb640 10 monclient: dump: 2024-04-26T00:37:26.745 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:26.745 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:26.745 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:26.745 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:26.745 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:26.745 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:26.745 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:26.745 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:26.745 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:26.745 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:26.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8be5ffb640 10 monclient: _finish_auth 0 2024-04-26T00:37:26.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8be5ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:37:26.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8be5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:37:26.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8be5ffb640 1 -- 172.21.15.37:0/3313177234 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8be00b1040 con 0x7f8bf00a06e0 2024-04-26T00:37:26.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8bf7678640 5 monclient: authenticate success, global_id 204185 2024-04-26T00:37:26.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8bf7678640 10 monclient: _renew_subs 2024-04-26T00:37:26.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8bf7678640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:37:26.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8bf7678640 1 -- 172.21.15.37:0/3313177234 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bf009f550 con 0x7f8bf00a06e0 2024-04-26T00:37:26.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8be5ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:37:26.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8be5ffb640 1 -- 172.21.15.37:0/3313177234 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be00aaa00 con 0x7f8bf00a06e0 2024-04-26T00:37:26.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8be5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:26.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8be5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:37:26.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8be5ffb640 10 monclient: dump: 2024-04-26T00:37:26.746 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:26.747 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:26.747 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:26.747 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:26.747 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:26.747 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:26.747 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:26.747 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:26.747 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:26.747 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:26.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8be77fe640 4 set_mon_vals no callback set 2024-04-26T00:37:26.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8bf7678640 10 monclient: _renew_subs 2024-04-26T00:37:26.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8bf7678640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:37:26.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8bf7678640 1 -- 172.21.15.37:0/3313177234 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bf013a5c0 con 0x7f8bf00a06e0 2024-04-26T00:37:26.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8bf7678640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:26.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8bf7678640 1 --2- 172.21.15.37:0/3313177234 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8bf012ea60 0x7f8bf012ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:26.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8bf7678640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:26.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8bf7678640 1 -- 172.21.15.37:0/3313177234 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf00a0cd0 con 0x7f8bf012ea60 2024-04-26T00:37:26.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8bf5414640 1 --2- 172.21.15.37:0/3313177234 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8bf012ea60 0x7f8bf012ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:26.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8bf5414640 10 monclient: get_auth_request con 0x7f8bf012ea60 auth_method 0 2024-04-26T00:37:26.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8bf5414640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:26.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8bf5414640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:26.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8bf5414640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:26.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8bf5414640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:26.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8bf5414640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:26.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8bf5414640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:26.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8be5ffb640 1 -- 172.21.15.37:0/3313177234 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f8be00aabe0 con 0x7f8bf00a06e0 2024-04-26T00:37:26.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8bf5414640 10 monclient(hunting): handle_auth_done global_id 144804 payload 824 2024-04-26T00:37:26.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8bf5414640 1 -- 172.21.15.37:0/3313177234 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8bf004f510 con 0x7f8bf012ea60 2024-04-26T00:37:26.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8be5ffb640 1 --2- 172.21.15.37:0/3313177234 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8bd4035ed0 0x7f8bd4038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:26.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8be5ffb640 1 -- 172.21.15.37:0/3313177234 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(686..686 src has 370..686) ==== 3899+0+0 (secure 0 0 0) 0x7f8be00e27c0 con 0x7f8bf00a06e0 2024-04-26T00:37:26.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8bf5c15640 1 --2- 172.21.15.37:0/3313177234 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8bd4035ed0 0x7f8bd4038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:26.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8bf5c15640 10 monclient: get_auth_request con 0x7f8bd4035ed0 auth_method 0 2024-04-26T00:37:26.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8bf5414640 1 -- 172.21.15.37:0/3313177234 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8bf012ea60 msgr2=0x7f8bf012ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:26.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8bf5414640 1 --2- 172.21.15.37:0/3313177234 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8bf012ea60 0x7f8bf012ee40 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f8bec009ef0 tx=0x7f8bec00f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:26.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8be5ffb640 1 -- 172.21.15.37:0/3313177234 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bec014860 con 0x7f8bf012ea60 2024-04-26T00:37:26.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8be5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:26.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8be5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:37:26.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.736+0000 7f8be5ffb640 10 monclient: dump: 2024-04-26T00:37:26.749 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:26.749 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:26.749 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:26.749 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:26.749 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:26.749 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:26.749 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:26.750 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:26.750 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:26.750 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:26.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.740+0000 7f8be5ffb640 1 -- 172.21.15.37:0/3313177234 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8bec014860 con 0x7f8bf012ea60 2024-04-26T00:37:26.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.740+0000 7f8be5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:26.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.740+0000 7f8be5ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:37:26.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.740+0000 7f8be5ffb640 1 -- 172.21.15.37:0/3313177234 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8bf012ea60 msgr2=0x7f8bf012ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:26.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.740+0000 7f8be5ffb640 1 --2- 172.21.15.37:0/3313177234 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8bf012ea60 0x7f8bf012ee40 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f8bec009ef0 tx=0x7f8bec00f610 comp rx=0 tx=0).stop 2024-04-26T00:37:26.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.740+0000 7f8be5ffb640 1 -- 172.21.15.37:0/3313177234 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8bf012ea60 msgr2=0x7f8bf012ee40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:37:26.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.740+0000 7f8be5ffb640 1 --2- 172.21.15.37:0/3313177234 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8bf012ea60 0x7f8bf012ee40 secure :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f8bec009ef0 tx=0x7f8bec00f610 comp rx=0 tx=0).stop 2024-04-26T00:37:26.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.740+0000 7f8bf5c15640 1 --2- 172.21.15.37:0/3313177234 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8bd4035ed0 0x7f8bd4038390 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f8bf0138d80 tx=0x7f8be800b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:26.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.744+0000 7f8bf7678640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:37:26.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.744+0000 7f8bf7678640 1 --2- 172.21.15.37:0/3313177234 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8bf012f380 0x7f8bf0133ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:26.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.744+0000 7f8bf7678640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:26.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.744+0000 7f8bf7678640 1 -- 172.21.15.37:0/3313177234 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf013a0e0 con 0x7f8bf012f380 2024-04-26T00:37:26.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.744+0000 7f8bf5414640 1 --2- 172.21.15.37:0/3313177234 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8bf012f380 0x7f8bf0133ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:26.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.744+0000 7f8bf5414640 10 monclient: get_auth_request con 0x7f8bf012f380 auth_method 0 2024-04-26T00:37:26.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.744+0000 7f8bf5414640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:26.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.744+0000 7f8bf5414640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:26.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.744+0000 7f8bf5414640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:26.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.744+0000 7f8bf5414640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:26.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.744+0000 7f8bf5414640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:26.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.744+0000 7f8bf5414640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:26.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.744+0000 7f8bf5414640 10 monclient(hunting): handle_auth_done global_id 144807 payload 824 2024-04-26T00:37:26.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.744+0000 7f8bf5414640 1 -- 172.21.15.37:0/3313177234 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8bf006ba50 con 0x7f8bf012f380 2024-04-26T00:37:26.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.748+0000 7f8bf5414640 1 -- 172.21.15.37:0/3313177234 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8bf012f380 msgr2=0x7f8bf0133ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:26.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.748+0000 7f8bf5414640 1 --2- 172.21.15.37:0/3313177234 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8bf012f380 0x7f8bf0133ee0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f8bec009ef0 tx=0x7f8bec0046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:26.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.748+0000 7f8be5ffb640 1 -- 172.21.15.37:0/3313177234 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bec004220 con 0x7f8bf012f380 2024-04-26T00:37:26.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.748+0000 7f8be5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:26.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.748+0000 7f8be5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:37:26.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.748+0000 7f8be5ffb640 10 monclient: dump: 2024-04-26T00:37:26.753 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:26.753 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:26.753 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:26.753 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:26.753 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:26.753 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:26.753 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:26.754 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:26.754 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:26.754 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:26.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.748+0000 7f8be5ffb640 1 -- 172.21.15.37:0/3313177234 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4890 (secure 0 0 0) 0x7f8bec004220 con 0x7f8bf012f380 2024-04-26T00:37:26.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.748+0000 7f8be5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:37:26.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.748+0000 7f8be5ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:37:26.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.748+0000 7f8be5ffb640 1 -- 172.21.15.37:0/3313177234 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8bf012f380 msgr2=0x7f8bf0133ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:26.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.748+0000 7f8be5ffb640 1 --2- 172.21.15.37:0/3313177234 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8bf012f380 0x7f8bf0133ee0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f8bec009ef0 tx=0x7f8bec0046d0 comp rx=0 tx=0).stop 2024-04-26T00:37:26.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.748+0000 7f8be5ffb640 1 -- 172.21.15.37:0/3313177234 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8bf012f380 msgr2=0x7f8bf0133ee0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:37:26.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.748+0000 7f8be5ffb640 1 --2- 172.21.15.37:0/3313177234 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8bf012f380 0x7f8bf0133ee0 secure :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f8bec009ef0 tx=0x7f8bec0046d0 comp rx=0 tx=0).stop 2024-04-26T00:37:26.754 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:37:26.755 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:37:26.755 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:37:26.755 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:37:26.755 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 132, 2024-04-26T00:37:26.755 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 333291, 2024-04-26T00:37:26.755 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:37:26.755 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:37:26.755 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:37:26.755 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:37:26.755 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:26.755 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:37:26.755 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:37:26.755 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:37:26.755 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:37:26.756 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:37:26.756 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:37:26.756 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:37:26.756 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:37:26.756 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:37:26.756 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:37:26.756 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:37:26.756 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:37:26.756 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:37:26.756 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:37:26.756 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:37:26.756 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:26.756 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:37:26.756 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:37:26.756 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:37:26.757 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:37:26.757 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:37:26.757 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:37:26.757 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:37:26.757 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:37:26.757 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:37:26.757 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:37:26.757 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:37:26.757 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:37:26.757 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:37:26.757 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:26.757 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:26.757 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:37:26.757 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:37:26.758 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:37:26.758 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:37:26.758 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:37:26.758 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:37:26.758 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:37:26.758 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:37:26.758 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:37:26.758 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:37:26.758 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:37:26.758 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:37:26.758 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:37:26.758 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:37:26.758 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:37:26.758 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:37:26.758 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:37:26.759 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:37:26.759 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:37:26.759 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:37:26.759 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:37:26.759 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:37:26.759 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:37:26.759 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:37:26.759 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:37:26.759 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:37:26.759 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:37:26.759 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:37:26.759 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:26.759 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:37:26.759 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:26.759 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:37:26.760 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:26.760 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:37:26.760 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:37:26.760 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:37:26.760 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:37:26.760 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:26.760 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:37:26.760 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:37:26.760 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:26.760 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:26.760 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:26.760 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:37:26.760 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:37:26.760 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:26.760 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:26.761 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:26.761 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:26.761 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:37:26.761 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:37:26.761 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:37:26.761 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:37:26.761 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:37:26.761 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:26.761 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:26.761 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:37:26.761 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:37:26.761 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:37:26.761 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:37:26.761 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:26.761 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:37:26.761 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:37:26.762 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:26.762 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:26.762 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:26.762 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:37:26.762 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:37:26.762 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:26.762 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:26.762 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:26.762 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:26.762 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:37:26.762 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:37:26.762 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:37:26.762 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:37:26.762 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:37:26.763 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:26.763 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:26.763 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:37:26.763 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:37:26.763 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:37:26.763 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:37:26.763 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:26.763 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:37:26.763 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:37:26.763 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:26.763 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:26.763 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:26.763 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:37:26.763 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:37:26.763 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:26.763 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:26.764 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:26.764 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:26.764 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:37:26.764 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:37:26.764 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:37:26.764 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:37:26.764 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:37:26.764 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:26.764 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:26.764 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:26.764 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:37:26.764 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:37:26.764 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:26.764 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:37:26.764 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:37:26.765 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:37:26.765 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:26.765 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:26.765 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:37:26.765 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:26.765 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:37:26.765 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:37:26.765 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:37:26.765 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:26.765 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:26.765 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:37:26.765 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:26.765 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:37:26.765 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:37:26.765 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:37:26.765 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:26.766 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:26.766 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:37:26.766 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:26.766 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:37:26.766 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:37:26.766 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:37:26.766 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:26.766 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:26.766 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:26.766 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:37:26.766 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:37:26.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.748+0000 7f8bf7678640 1 -- 172.21.15.37:0/3313177234 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8bd4035ed0 msgr2=0x7f8bd4038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:26.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.748+0000 7f8bf7678640 1 --2- 172.21.15.37:0/3313177234 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8bd4035ed0 0x7f8bd4038390 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f8bf0138d80 tx=0x7f8be800b040 comp rx=0 tx=0).stop 2024-04-26T00:37:26.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.748+0000 7f8bf7678640 10 monclient: shutdown 2024-04-26T00:37:26.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.748+0000 7f8bf7678640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:37:26.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.748+0000 7f8bf7678640 1 -- 172.21.15.37:0/3313177234 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8bf00a06e0 msgr2=0x7f8bf01cd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:26.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.748+0000 7f8bf7678640 1 --2- 172.21.15.37:0/3313177234 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8bf00a06e0 0x7f8bf01cd280 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8bf00a1290 tx=0x7f8be0019180 comp rx=0 tx=0).stop 2024-04-26T00:37:26.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.748+0000 7f8bf5c15640 1 -- 172.21.15.37:0/3313177234 reap_dead start 2024-04-26T00:37:26.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.748+0000 7f8bf5c15640 1 -- 172.21.15.37:0/3313177234 reap_dead start 2024-04-26T00:37:26.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.748+0000 7f8bf7678640 1 -- 172.21.15.37:0/3313177234 shutdown_connections 2024-04-26T00:37:26.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.748+0000 7f8bf7678640 1 -- 172.21.15.37:0/3313177234 >> 172.21.15.37:0/3313177234 conn(0x7f8bf00a1d50 msgr2=0x7f8bf0131b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:37:26.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.748+0000 7f8bf7678640 1 -- 172.21.15.37:0/3313177234 shutdown_connections 2024-04-26T00:37:26.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.748+0000 7f8bf7678640 1 -- 172.21.15.37:0/3313177234 wait complete. 2024-04-26T00:37:26.769 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:37:26.769 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:37:26.769 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:37:26.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f9253e1f640 10 monclient: get_monmap_and_config 2024-04-26T00:37:26.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f9253e1f640 10 monclient: build_initial_monmap 2024-04-26T00:37:26.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f9253e1f640 10 monclient: monmap: 2024-04-26T00:37:26.884 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:37:26.884 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:26.884 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:37:26.888677+0000 2024-04-26T00:37:26.884 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:37:26.888677+0000 2024-04-26T00:37:26.884 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:37:26.884 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:26.884 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:37:26.884 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:37:26.885 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:37:26.885 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:37:26.885 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:37:26.885 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:26.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f9253e1f640 1 Processor -- start 2024-04-26T00:37:26.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f9253e1f640 1 -- start start 2024-04-26T00:37:26.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f9253e1f640 10 monclient: init 2024-04-26T00:37:26.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f9253e1f640 10 monclient: _reopen_session rank -1 2024-04-26T00:37:26.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f9253e1f640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-04-26T00:37:26.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f9253e1f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f924c130eb0 0x7f924c09ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:26.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f9253e1f640 10 monclient(hunting): picked mon.noname-b con 0x7f924c130eb0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:37:26.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f9253e1f640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f924c131290 0x7f924c133b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:26.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f9253e1f640 10 monclient(hunting): picked mon.noname-c con 0x7f924c131290 addr v2:172.21.15.45:3300/0 2024-04-26T00:37:26.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f9253e1f640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f924c1340d0 0x7f924c134580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:26.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f9253e1f640 10 monclient(hunting): picked mon.noname-a con 0x7f924c1340d0 addr v2:172.21.15.37:3300/0 2024-04-26T00:37:26.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f9253e1f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:26.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f9253e1f640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f924c09df60 con 0x7f924c1340d0 2024-04-26T00:37:26.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f9253e1f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:26.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f9253e1f640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f924c09c190 con 0x7f924c131290 2024-04-26T00:37:26.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f9253e1f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:26.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f9253e1f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f924c0a11c0 con 0x7f924c130eb0 2024-04-26T00:37:26.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f9253e1f640 10 monclient(hunting): _renew_subs 2024-04-26T00:37:26.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f9253e1f640 10 monclient(hunting): authenticate will time out at 2028.194302s 2024-04-26T00:37:26.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f9251bbb640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f924c130eb0 0x7f924c09ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:26.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f92523bc640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f924c1340d0 0x7f924c134580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:26.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f9251bbb640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f924c130eb0 0x7f924c09ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:34520/0 (socket says 172.21.15.37:34520) 2024-04-26T00:37:26.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f92523bc640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f924c1340d0 0x7f924c134580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:45484/0 (socket says 172.21.15.37:45484) 2024-04-26T00:37:26.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f9251bbb640 1 -- 172.21.15.37:0/2875823100 learned_addr learned my addr 172.21.15.37:0/2875823100 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:37:26.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f9251bbb640 10 monclient(hunting): get_auth_request con 0x7f924c130eb0 auth_method 0 2024-04-26T00:37:26.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f9251bbb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:26.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f9251bbb640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:26.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f9251bbb640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:26.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f92523bc640 10 monclient(hunting): get_auth_request con 0x7f924c1340d0 auth_method 0 2024-04-26T00:37:26.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f92523bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:26.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f92523bc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:26.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f92523bc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:26.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f92523bc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:26.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f92523bc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:26.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f92523bc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:26.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f9251bbb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:26.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f9251bbb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:26.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f9251bbb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:26.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f92513ba640 1 --2- 172.21.15.37:0/2875823100 >> v2:172.21.15.45:3300/0 conn(0x7f924c131290 0x7f924c133b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:26.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f92513ba640 10 monclient(hunting): get_auth_request con 0x7f924c131290 auth_method 0 2024-04-26T00:37:26.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f92513ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:26.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f92513ba640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:26.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f92513ba640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:26.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f92523bc640 10 monclient(hunting): handle_auth_done global_id 144810 payload 274 2024-04-26T00:37:26.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f92523bc640 1 -- 172.21.15.37:0/2875823100 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f924c130eb0 msgr2=0x7f924c09ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:37:26.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f92523bc640 1 --2- 172.21.15.37:0/2875823100 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f924c130eb0 0x7f924c09ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:26.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f92523bc640 1 -- 172.21.15.37:0/2875823100 >> v2:172.21.15.45:3300/0 conn(0x7f924c131290 msgr2=0x7f924c133b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:37:26.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f92523bc640 1 --2- 172.21.15.37:0/2875823100 >> v2:172.21.15.45:3300/0 conn(0x7f924c131290 0x7f924c133b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:26.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f92523bc640 10 monclient: _finish_hunting 0 2024-04-26T00:37:26.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f92523bc640 1 monclient: found mon.noname-a 2024-04-26T00:37:26.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f92523bc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:37:26.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f92523bc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:37:26.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f92523bc640 1 -- 172.21.15.37:0/2875823100 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f924c09dc80 con 0x7f924c1340d0 2024-04-26T00:37:26.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.884+0000 7f9251bbb640 1 --2- 172.21.15.37:0/2875823100 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f924c130eb0 0x7f924c09ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:37:26.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.888+0000 7f92523bc640 1 -- 172.21.15.37:0/2875823100 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f924c1340d0 msgr2=0x7f924c134580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:26.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.888+0000 7f92523bc640 1 --2- 172.21.15.37:0/2875823100 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f924c1340d0 0x7f924c134580 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f924c06d330 tx=0x7f924401c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75cb855432108900 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:26.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.888+0000 7f9250bb9640 1 -- 172.21.15.37:0/2875823100 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92440b2070 con 0x7f924c1340d0 2024-04-26T00:37:26.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.888+0000 7f9250bb9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:26.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.888+0000 7f9250bb9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:37:26.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.888+0000 7f9250bb9640 10 monclient: dump: 2024-04-26T00:37:26.895 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:26.895 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:26.895 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:26.895 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:26.895 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:26.895 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:26.895 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:26.895 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:26.896 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:26.896 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:26.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.888+0000 7f9250bb9640 10 monclient: _finish_auth 0 2024-04-26T00:37:26.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.888+0000 7f9250bb9640 10 monclient: _check_auth_tickets 2024-04-26T00:37:26.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.888+0000 7f9250bb9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:37:26.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.888+0000 7f9250bb9640 1 -- 172.21.15.37:0/2875823100 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92440a5780 con 0x7f924c1340d0 2024-04-26T00:37:26.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.888+0000 7f9250bb9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:37:26.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.888+0000 7f9250bb9640 1 -- 172.21.15.37:0/2875823100 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92440a5b50 con 0x7f924c1340d0 2024-04-26T00:37:26.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.888+0000 7f9253e1f640 5 monclient: authenticate success, global_id 144810 2024-04-26T00:37:26.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.888+0000 7f9253e1f640 10 monclient: get_monmap_and_config success 2024-04-26T00:37:26.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.888+0000 7f9253e1f640 4 set_mon_vals no callback set 2024-04-26T00:37:26.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.888+0000 7f9250bb9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:26.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.888+0000 7f9250bb9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:37:26.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.888+0000 7f9250bb9640 10 monclient: dump: 2024-04-26T00:37:26.896 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:26.897 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:26.897 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:26.897 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:26.897 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:26.897 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:26.897 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:26.897 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:26.897 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:26.897 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:26.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.888+0000 7f9253e1f640 10 monclient: shutdown 2024-04-26T00:37:26.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.888+0000 7f9253e1f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:37:26.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.888+0000 7f9253e1f640 1 -- 172.21.15.37:0/2875823100 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f924c1340d0 msgr2=0x7f924c134580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:26.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.888+0000 7f9253e1f640 1 --2- 172.21.15.37:0/2875823100 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f924c1340d0 0x7f924c134580 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f924c06d330 tx=0x7f924401c7f0 comp rx=0 tx=0).stop 2024-04-26T00:37:26.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.888+0000 7f9253e1f640 1 -- 172.21.15.37:0/2875823100 shutdown_connections 2024-04-26T00:37:26.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.888+0000 7f9253e1f640 1 --2- 172.21.15.37:0/2875823100 >> v2:172.21.15.45:3300/0 conn(0x7f924c131290 0x7f924c133b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:26.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.888+0000 7f9253e1f640 1 --2- 172.21.15.37:0/2875823100 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f924c1340d0 0x7f924c134580 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:26.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.888+0000 7f9253e1f640 1 --2- 172.21.15.37:0/2875823100 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f924c130eb0 0x7f924c09ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:26.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.888+0000 7f9253e1f640 1 -- 172.21.15.37:0/2875823100 >> 172.21.15.37:0/2875823100 conn(0x7f924c0a1d50 msgr2=0x7f924c0a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:37:26.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.888+0000 7f9253e1f640 1 -- 172.21.15.37:0/2875823100 shutdown_connections 2024-04-26T00:37:26.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.888+0000 7f9253e1f640 1 -- 172.21.15.37:0/2875823100 wait complete. 2024-04-26T00:37:26.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f9253e1f640 10 monclient: build_initial_monmap 2024-04-26T00:37:26.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f9253e1f640 10 monclient: monmap: 2024-04-26T00:37:26.898 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:37:26.898 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:37:26.898 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:37:26.898 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:37:26.898 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:37:26.898 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:37:26.898 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:37:26.898 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:37:26.899 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:37:26.899 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:26.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f9253e1f640 1 Processor -- start 2024-04-26T00:37:26.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f9253e1f640 1 -- start start 2024-04-26T00:37:26.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f9253e1f640 10 monclient: init 2024-04-26T00:37:26.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f9253e1f640 10 monclient: _reopen_session rank -1 2024-04-26T00:37:26.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f9253e1f640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:37:26.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f9253e1f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f924c130eb0 0x7f924c13a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:26.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f9253e1f640 10 monclient(hunting): picked mon.noname-b con 0x7f924c130eb0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:37:26.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f9253e1f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f924c131290 0x7f924c1ced50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:26.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f9253e1f640 10 monclient(hunting): picked mon.noname-a con 0x7f924c131290 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:37:26.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f9253e1f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f924c1340d0 0x7f924c1cf290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:26.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f9253e1f640 10 monclient(hunting): picked mon.noname-c con 0x7f924c1340d0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:37:26.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f9253e1f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:26.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f9253e1f640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f924c09fc50 con 0x7f924c131290 2024-04-26T00:37:26.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f9253e1f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:26.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f9253e1f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f924c131880 con 0x7f924c130eb0 2024-04-26T00:37:26.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f9253e1f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:26.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f9253e1f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f924c09fdd0 con 0x7f924c1340d0 2024-04-26T00:37:26.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f9253e1f640 10 monclient(hunting): _renew_subs 2024-04-26T00:37:26.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f9253e1f640 10 monclient(hunting): authenticate will time out at 2028.201438s 2024-04-26T00:37:26.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f92513ba640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f924c131290 0x7f924c1ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:26.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f92523bc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f924c1340d0 0x7f924c1cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:26.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f92513ba640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f924c131290 0x7f924c1ced50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:45488/0 (socket says 172.21.15.37:45488) 2024-04-26T00:37:26.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f92523bc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f924c1340d0 0x7f924c1cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:34530/0 (socket says 172.21.15.37:34530) 2024-04-26T00:37:26.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f92513ba640 1 -- 172.21.15.37:0/1015641128 learned_addr learned my addr 172.21.15.37:0/1015641128 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:37:26.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f92513ba640 10 monclient(hunting): get_auth_request con 0x7f924c131290 auth_method 0 2024-04-26T00:37:26.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f92513ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:26.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f92513ba640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:26.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f92513ba640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:26.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f92523bc640 10 monclient(hunting): get_auth_request con 0x7f924c1340d0 auth_method 0 2024-04-26T00:37:26.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f92523bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:26.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f92523bc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:26.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f92523bc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:26.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f92513ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:26.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f92513ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:26.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f92513ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:26.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f92523bc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:26.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f92523bc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:26.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f92523bc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:26.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f92513ba640 10 monclient(hunting): handle_auth_done global_id 144813 payload 824 2024-04-26T00:37:26.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f9251bbb640 1 --2- 172.21.15.37:0/1015641128 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f924c130eb0 0x7f924c13a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:26.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f92513ba640 1 -- 172.21.15.37:0/1015641128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f924c1340d0 msgr2=0x7f924c1cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:26.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f92513ba640 1 --2- 172.21.15.37:0/1015641128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f924c1340d0 0x7f924c1cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:26.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f92513ba640 1 -- 172.21.15.37:0/1015641128 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f924c130eb0 msgr2=0x7f924c13a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:26.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f92513ba640 1 --2- 172.21.15.37:0/1015641128 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f924c130eb0 0x7f924c13a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:26.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f92513ba640 10 monclient: _finish_hunting 0 2024-04-26T00:37:26.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f92513ba640 1 monclient: found mon.noname-a 2024-04-26T00:37:26.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f92513ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:37:26.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f92513ba640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:37:26.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f92513ba640 1 -- 172.21.15.37:0/1015641128 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f924c0a0390 con 0x7f924c131290 2024-04-26T00:37:26.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f9251bbb640 10 monclient: get_auth_request con 0x7f924c130eb0 auth_method 0 2024-04-26T00:37:26.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f92523bc640 1 --2- 172.21.15.37:0/1015641128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f924c1340d0 0x7f924c1cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:37:26.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.892+0000 7f9251bbb640 1 --2- 172.21.15.37:0/1015641128 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f924c130eb0 0x7f924c13a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:37:26.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f92513ba640 1 -- 172.21.15.37:0/1015641128 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f924c131290 msgr2=0x7f924c1ced50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:26.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f92513ba640 1 --2- 172.21.15.37:0/1015641128 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f924c131290 0x7f924c1ced50 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f924800d800 tx=0x7f924800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:26.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f923e7fc640 1 -- 172.21.15.37:0/1015641128 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f924800b8d0 con 0x7f924c131290 2024-04-26T00:37:26.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f923e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:26.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f923e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:37:26.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f923e7fc640 10 monclient: dump: 2024-04-26T00:37:26.905 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:26.906 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:26.906 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:26.906 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:26.906 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:26.906 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:26.906 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:26.906 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:26.906 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:26.906 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:26.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f923e7fc640 10 monclient: _finish_auth 0 2024-04-26T00:37:26.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f923e7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:37:26.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f923e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:37:26.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f923e7fc640 1 -- 172.21.15.37:0/1015641128 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9248019070 con 0x7f924c131290 2024-04-26T00:37:26.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f923e7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:37:26.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f923e7fc640 1 -- 172.21.15.37:0/1015641128 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f924800bc70 con 0x7f924c131290 2024-04-26T00:37:26.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f923e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:26.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f923e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:37:26.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f923e7fc640 10 monclient: dump: 2024-04-26T00:37:26.907 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:26.907 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:26.907 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:26.907 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:26.907 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:26.907 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:26.907 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:26.907 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:26.907 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:26.907 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:26.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f923ffff640 4 set_mon_vals no callback set 2024-04-26T00:37:26.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f9253e1f640 5 monclient: authenticate success, global_id 144813 2024-04-26T00:37:26.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f9253e1f640 10 monclient: _renew_subs 2024-04-26T00:37:26.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f9253e1f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:37:26.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f9253e1f640 1 -- 172.21.15.37:0/1015641128 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f924c0a1470 con 0x7f924c131290 2024-04-26T00:37:26.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f9253e1f640 10 monclient: _renew_subs 2024-04-26T00:37:26.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f9253e1f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:37:26.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f9253e1f640 1 -- 172.21.15.37:0/1015641128 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f924c139840 con 0x7f924c131290 2024-04-26T00:37:26.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f9253e1f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:26.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f9253e1f640 1 --2- 172.21.15.37:0/1015641128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f924c1cf7d0 0x7f924c12da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:26.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f9253e1f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:26.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f9253e1f640 1 -- 172.21.15.37:0/1015641128 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f924c1cfbb0 con 0x7f924c1cf7d0 2024-04-26T00:37:26.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f9251bbb640 1 --2- 172.21.15.37:0/1015641128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f924c1cf7d0 0x7f924c12da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:26.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f9251bbb640 10 monclient: get_auth_request con 0x7f924c1cf7d0 auth_method 0 2024-04-26T00:37:26.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f9251bbb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:26.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f9251bbb640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:26.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f9251bbb640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:26.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f9251bbb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:26.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f9251bbb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:26.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f9251bbb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:26.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f923e7fc640 1 -- 172.21.15.37:0/1015641128 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f924801e460 con 0x7f924c131290 2024-04-26T00:37:26.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f9251bbb640 10 monclient(hunting): handle_auth_done global_id 204194 payload 824 2024-04-26T00:37:26.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f9251bbb640 1 -- 172.21.15.37:0/1015641128 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f924c04f510 con 0x7f924c1cf7d0 2024-04-26T00:37:26.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f923e7fc640 1 --2- 172.21.15.37:0/1015641128 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9230035ed0 0x7f9230038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:26.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f923e7fc640 1 -- 172.21.15.37:0/1015641128 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(686..686 src has 370..686) ==== 3899+0+0 (secure 0 0 0) 0x7f9248048e40 con 0x7f924c131290 2024-04-26T00:37:26.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f92523bc640 1 --2- 172.21.15.37:0/1015641128 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9230035ed0 0x7f9230038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:26.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.896+0000 7f92523bc640 10 monclient: get_auth_request con 0x7f9230035ed0 auth_method 0 2024-04-26T00:37:26.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.900+0000 7f9251bbb640 1 -- 172.21.15.37:0/1015641128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f924c1cf7d0 msgr2=0x7f924c12da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:26.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.900+0000 7f9251bbb640 1 --2- 172.21.15.37:0/1015641128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f924c1cf7d0 0x7f924c12da50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f92400107c0 tx=0x7f9240010c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:26.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.900+0000 7f923e7fc640 1 -- 172.21.15.37:0/1015641128 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f924000f040 con 0x7f924c1cf7d0 2024-04-26T00:37:26.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.900+0000 7f923e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:26.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.900+0000 7f923e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:37:26.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.900+0000 7f923e7fc640 10 monclient: dump: 2024-04-26T00:37:26.910 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:26.910 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:26.910 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:26.910 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:26.910 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:26.910 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:26.910 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:26.910 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:26.910 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:26.910 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:26.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.900+0000 7f923e7fc640 1 -- 172.21.15.37:0/1015641128 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9240010e70 con 0x7f924c1cf7d0 2024-04-26T00:37:26.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.900+0000 7f923e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:26.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.900+0000 7f923e7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:37:26.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.900+0000 7f923e7fc640 1 -- 172.21.15.37:0/1015641128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f924c1cf7d0 msgr2=0x7f924c12da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:26.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.900+0000 7f923e7fc640 1 --2- 172.21.15.37:0/1015641128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f924c1cf7d0 0x7f924c12da50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f92400107c0 tx=0x7f9240010c80 comp rx=0 tx=0).stop 2024-04-26T00:37:26.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.900+0000 7f923e7fc640 1 -- 172.21.15.37:0/1015641128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f924c1cf7d0 msgr2=0x7f924c12da50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:37:26.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.900+0000 7f923e7fc640 1 --2- 172.21.15.37:0/1015641128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f924c1cf7d0 0x7f924c12da50 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f92400107c0 tx=0x7f9240010c80 comp rx=0 tx=0).stop 2024-04-26T00:37:26.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.900+0000 7f92523bc640 1 --2- 172.21.15.37:0/1015641128 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9230035ed0 0x7f9230038390 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f92440ac000 tx=0x7f92440a7070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:26.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.904+0000 7f9253e1f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:37:26.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.904+0000 7f9253e1f640 1 --2- 172.21.15.37:0/1015641128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f924c12df90 0x7f924c12e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:26.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.904+0000 7f9253e1f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:26.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.904+0000 7f9253e1f640 1 -- 172.21.15.37:0/1015641128 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f924c12e8e0 con 0x7f924c12df90 2024-04-26T00:37:26.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.904+0000 7f9251bbb640 1 --2- 172.21.15.37:0/1015641128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f924c12df90 0x7f924c12e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:26.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.904+0000 7f9251bbb640 10 monclient: get_auth_request con 0x7f924c12df90 auth_method 0 2024-04-26T00:37:26.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.904+0000 7f9251bbb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:26.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.904+0000 7f9251bbb640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:26.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.904+0000 7f9251bbb640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:26.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.904+0000 7f9251bbb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:26.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.904+0000 7f9251bbb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:26.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.904+0000 7f9251bbb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:26.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.904+0000 7f9251bbb640 10 monclient(hunting): handle_auth_done global_id 204197 payload 824 2024-04-26T00:37:26.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.904+0000 7f9251bbb640 1 -- 172.21.15.37:0/1015641128 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f924c0a0f60 con 0x7f924c12df90 2024-04-26T00:37:26.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.908+0000 7f9251bbb640 1 -- 172.21.15.37:0/1015641128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f924c12df90 msgr2=0x7f924c12e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:26.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.908+0000 7f9251bbb640 1 --2- 172.21.15.37:0/1015641128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f924c12df90 0x7f924c12e3a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f92400107c0 tx=0x7f92400046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:26.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.908+0000 7f923e7fc640 1 -- 172.21.15.37:0/1015641128 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9240006660 con 0x7f924c12df90 2024-04-26T00:37:26.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.908+0000 7f923e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:26.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.908+0000 7f923e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:37:26.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.908+0000 7f923e7fc640 10 monclient: dump: 2024-04-26T00:37:26.914 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:26.914 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:26.914 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:26.914 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:26.914 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:26.914 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:26.914 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:26.914 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:26.914 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:26.915 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:26.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.908+0000 7f923e7fc640 1 -- 172.21.15.37:0/1015641128 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4547 (secure 0 0 0) 0x7f9240006660 con 0x7f924c12df90 2024-04-26T00:37:26.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.908+0000 7f923e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:37:26.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.908+0000 7f923e7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:37:26.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.908+0000 7f923e7fc640 1 -- 172.21.15.37:0/1015641128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f924c12df90 msgr2=0x7f924c12e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:26.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.908+0000 7f923e7fc640 1 --2- 172.21.15.37:0/1015641128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f924c12df90 0x7f924c12e3a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f92400107c0 tx=0x7f92400046d0 comp rx=0 tx=0).stop 2024-04-26T00:37:26.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.908+0000 7f923e7fc640 1 -- 172.21.15.37:0/1015641128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f924c12df90 msgr2=0x7f924c12e3a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:37:26.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.908+0000 7f923e7fc640 1 --2- 172.21.15.37:0/1015641128 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f924c12df90 0x7f924c12e3a0 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f92400107c0 tx=0x7f92400046d0 comp rx=0 tx=0).stop 2024-04-26T00:37:26.915 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:37:26.915 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:37:26.915 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:37:26.916 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:37:26.916 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 132, 2024-04-26T00:37:26.916 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 52515, 2024-04-26T00:37:26.916 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:37:26.916 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:37:26.916 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:37:26.916 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:37:26.916 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:26.916 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:37:26.916 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:37:26.916 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:37:26.916 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:37:26.916 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:37:26.916 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:37:26.916 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:37:26.917 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:37:26.917 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:37:26.917 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:37:26.917 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:37:26.917 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:37:26.917 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:37:26.917 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:37:26.917 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:37:26.917 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:26.917 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:37:26.917 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:37:26.917 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:37:26.917 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:37:26.917 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:37:26.917 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:37:26.918 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:37:26.918 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:37:26.918 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:37:26.918 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:37:26.918 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:37:26.918 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:37:26.918 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:37:26.918 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:26.918 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:26.918 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:37:26.918 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:37:26.918 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:37:26.918 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:37:26.918 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:37:26.919 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:37:26.919 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:37:26.919 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:37:26.919 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:37:26.919 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:37:26.919 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:37:26.919 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:37:26.919 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:37:26.919 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:37:26.919 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:37:26.919 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:37:26.919 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:37:26.919 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:37:26.919 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:37:26.919 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:37:26.920 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:37:26.920 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:37:26.920 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:37:26.920 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:37:26.920 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:37:26.920 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:37:26.920 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:37:26.920 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:37:26.920 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:26.920 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:37:26.920 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:26.920 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:37:26.920 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:26.920 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:37:26.920 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:37:26.921 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:37:26.921 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:37:26.921 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:26.921 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:37:26.921 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:37:26.921 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:26.921 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:26.921 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:26.921 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:37:26.921 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:37:26.921 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:26.921 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:26.921 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:26.921 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:26.922 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:37:26.922 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:37:26.922 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:37:26.922 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:37:26.922 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:37:26.922 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:26.922 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:26.922 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:37:26.922 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:37:26.922 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:37:26.922 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:37:26.922 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:26.922 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:37:26.922 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:37:26.922 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:26.923 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:26.923 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:26.923 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:37:26.923 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:37:26.923 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:26.923 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:26.923 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:26.923 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:26.923 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:37:26.923 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:37:26.923 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:37:26.923 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:37:26.923 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:37:26.923 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:26.923 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:26.924 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:37:26.924 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:37:26.924 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:37:26.924 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:37:26.924 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:26.924 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:37:26.924 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:37:26.924 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:26.924 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:26.924 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:26.924 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:37:26.924 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:37:26.924 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:26.924 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:26.924 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:26.925 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:26.925 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:37:26.925 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:37:26.925 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:37:26.925 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:37:26.925 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:37:26.925 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:26.925 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:26.925 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:26.925 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:37:26.925 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:37:26.925 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:26.925 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:37:26.925 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:37:26.925 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:37:26.926 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:26.926 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:26.926 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:37:26.926 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:26.926 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:37:26.926 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:37:26.926 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:37:26.926 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:26.926 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:26.926 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:26.926 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:37:26.926 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:37:26.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.908+0000 7f9253e1f640 1 -- 172.21.15.37:0/1015641128 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9230035ed0 msgr2=0x7f9230038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:26.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.908+0000 7f9253e1f640 1 --2- 172.21.15.37:0/1015641128 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9230035ed0 0x7f9230038390 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f92440ac000 tx=0x7f92440a7070 comp rx=0 tx=0).stop 2024-04-26T00:37:26.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.908+0000 7f9253e1f640 10 monclient: shutdown 2024-04-26T00:37:26.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.908+0000 7f9253e1f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:37:26.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.908+0000 7f9253e1f640 1 -- 172.21.15.37:0/1015641128 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f924c131290 msgr2=0x7f924c1ced50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:26.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.908+0000 7f9253e1f640 1 --2- 172.21.15.37:0/1015641128 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f924c131290 0x7f924c1ced50 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f924800d800 tx=0x7f924800dcc0 comp rx=0 tx=0).stop 2024-04-26T00:37:26.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.908+0000 7f92523bc640 1 -- 172.21.15.37:0/1015641128 reap_dead start 2024-04-26T00:37:26.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.908+0000 7f92523bc640 1 -- 172.21.15.37:0/1015641128 reap_dead start 2024-04-26T00:37:26.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.908+0000 7f9253e1f640 1 -- 172.21.15.37:0/1015641128 shutdown_connections 2024-04-26T00:37:26.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.908+0000 7f9253e1f640 1 -- 172.21.15.37:0/1015641128 >> 172.21.15.37:0/1015641128 conn(0x7f924c0a1d50 msgr2=0x7f924c071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:37:26.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.908+0000 7f9253e1f640 1 -- 172.21.15.37:0/1015641128 shutdown_connections 2024-04-26T00:37:26.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:26.908+0000 7f9253e1f640 1 -- 172.21.15.37:0/1015641128 wait complete. 2024-04-26T00:37:26.929 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:37:26.929 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:37:26.929 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:37:26.992 INFO:tasks.rados.rados.0.smithi045.stdout:3076: done (5 left) 2024-04-26T00:37:26.992 INFO:tasks.rados.rados.0.smithi045.stdout:3077: write oid 12 current snap is 329 2024-04-26T00:37:26.992 INFO:tasks.rados.rados.0.smithi045.stdout:3077: seq_num 1105 ranges {571735=421330,1562132=416708} 2024-04-26T00:37:27.003 INFO:tasks.rados.rados.0.smithi045.stdout:3077: writing smithi04526579-12 from 571735 to 993065 tid 1 2024-04-26T00:37:27.016 INFO:tasks.rados.rados.0.smithi045.stdout:3077: writing smithi04526579-12 from 1562132 to 1978840 tid 2 2024-04-26T00:37:27.016 INFO:tasks.rados.rados.0.smithi045.stdout:3078: read oid 39 snap -1 2024-04-26T00:37:27.016 INFO:tasks.rados.rados.0.smithi045.stdout:3078: expect (ObjNum 1093 snap 324 seq_num 1093) 2024-04-26T00:37:27.016 INFO:tasks.rados.rados.0.smithi045.stdout:3079: write (excl) oid 22 current snap is 329 2024-04-26T00:37:27.016 INFO:tasks.rados.rados.0.smithi045.stdout:3079: seq_num 1106 ranges {588542=496579,1620614=676719} 2024-04-26T00:37:27.029 INFO:tasks.rados.rados.0.smithi045.stdout:3079: writing smithi04526579-22 from 588542 to 1085121 tid 1 2024-04-26T00:37:27.043 INFO:tasks.rados.rados.0.smithi045.stdout:3079: writing smithi04526579-22 from 1620614 to 2297333 tid 2 2024-04-26T00:37:27.043 INFO:tasks.rados.rados.0.smithi045.stdout:3080: snap_create 2024-04-26T00:37:27.043 INFO:tasks.rados.rados.0.smithi045.stdout:3070: finishing copy_from to smithi04526579-20 2024-04-26T00:37:27.043 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 2213 (ObjNum 1065 snap 320 seq_num 1065) dirty exists 2024-04-26T00:37:27.044 INFO:tasks.rados.rados.0.smithi045.stdout:3072: finishing write tid 2 to smithi04526579-8 2024-04-26T00:37:27.044 INFO:tasks.rados.rados.0.smithi045.stdout:3072: finishing write tid 3 to smithi04526579-8 2024-04-26T00:37:27.044 INFO:tasks.rados.rados.0.smithi045.stdout:3072: finishing write tid 4 to smithi04526579-8 2024-04-26T00:37:27.044 INFO:tasks.rados.rados.0.smithi045.stdout:3072: oid 8 updating version 0 to 2196 2024-04-26T00:37:27.044 INFO:tasks.rados.rados.0.smithi045.stdout:3072: oid 8 version 2196 is already newer than 2194 2024-04-26T00:37:27.044 INFO:tasks.rados.rados.0.smithi045.stdout:3072: oid 8 version 2196 is already newer than 2195 2024-04-26T00:37:27.044 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 2196 (ObjNum 1103 snap 329 seq_num 1103) dirty exists 2024-04-26T00:37:27.044 INFO:tasks.rados.rados.0.smithi045.stdout:3072: left oid 8 (ObjNum 1103 snap 329 seq_num 1103) 2024-04-26T00:37:27.044 INFO:tasks.rados.rados.0.smithi045.stdout:3074: finishing rollback tid 0 to smithi04526579-19 2024-04-26T00:37:27.044 INFO:tasks.rados.rados.0.smithi045.stdout:3073: expect (ObjNum 1057 snap 318 seq_num 1057) 2024-04-26T00:37:27.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde674d9640 10 monclient: get_monmap_and_config 2024-04-26T00:37:27.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde674d9640 10 monclient: build_initial_monmap 2024-04-26T00:37:27.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde674d9640 10 monclient: monmap: 2024-04-26T00:37:27.051 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:37:27.051 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:27.051 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:37:27.055978+0000 2024-04-26T00:37:27.051 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:37:27.055978+0000 2024-04-26T00:37:27.051 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:37:27.051 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:27.052 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:37:27.052 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:37:27.052 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:37:27.052 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:37:27.052 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:37:27.052 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:27.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde674d9640 1 Processor -- start 2024-04-26T00:37:27.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde674d9640 1 -- start start 2024-04-26T00:37:27.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde674d9640 10 monclient: init 2024-04-26T00:37:27.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde674d9640 10 monclient: _reopen_session rank -1 2024-04-26T00:37:27.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde674d9640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-04-26T00:37:27.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde674d9640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde6009c780 0x7fde601350b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:27.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde674d9640 10 monclient(hunting): picked mon.noname-b con 0x7fde6009c780 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:37:27.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde674d9640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fde6009cb60 0x7fde600a1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:27.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde674d9640 10 monclient(hunting): picked mon.noname-a con 0x7fde6009cb60 addr v2:172.21.15.37:3300/0 2024-04-26T00:37:27.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde674d9640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fde600a20a0 0x7fde600a2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:27.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde674d9640 10 monclient(hunting): picked mon.noname-c con 0x7fde600a20a0 addr v2:172.21.15.45:3300/0 2024-04-26T00:37:27.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde674d9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:27.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde674d9640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fde60135b50 con 0x7fde6009cb60 2024-04-26T00:37:27.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde674d9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:27.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde674d9640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fde601355f0 con 0x7fde600a20a0 2024-04-26T00:37:27.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde674d9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:27.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde674d9640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fde6006b560 con 0x7fde6009c780 2024-04-26T00:37:27.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde674d9640 10 monclient(hunting): _renew_subs 2024-04-26T00:37:27.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde674d9640 10 monclient(hunting): authenticate will time out at 2028.361449s 2024-04-26T00:37:27.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde65275640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde6009c780 0x7fde601350b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:27.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde64a74640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fde6009cb60 0x7fde600a1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:27.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde65275640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde6009c780 0x7fde601350b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:34556/0 (socket says 172.21.15.37:34556) 2024-04-26T00:37:27.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde65275640 1 -- 172.21.15.37:0/1826488303 learned_addr learned my addr 172.21.15.37:0/1826488303 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:37:27.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde65275640 10 monclient(hunting): get_auth_request con 0x7fde6009c780 auth_method 0 2024-04-26T00:37:27.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde65275640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:27.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde65275640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:27.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde65275640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:27.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde64a74640 10 monclient(hunting): get_auth_request con 0x7fde6009cb60 auth_method 0 2024-04-26T00:37:27.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde64a74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:27.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde64a74640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:27.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde64a74640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:27.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde65275640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:27.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde65275640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:27.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde65275640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:27.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde64a74640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:27.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde64a74640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:27.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde64a74640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:27.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde65a76640 1 --2- 172.21.15.37:0/1826488303 >> v2:172.21.15.45:3300/0 conn(0x7fde600a20a0 0x7fde600a2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:27.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde65275640 10 monclient(hunting): handle_auth_done global_id 204200 payload 274 2024-04-26T00:37:27.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde65275640 1 -- 172.21.15.37:0/1826488303 >> v2:172.21.15.45:3300/0 conn(0x7fde600a20a0 msgr2=0x7fde600a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:37:27.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde65275640 1 --2- 172.21.15.37:0/1826488303 >> v2:172.21.15.45:3300/0 conn(0x7fde600a20a0 0x7fde600a2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:27.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde65275640 1 -- 172.21.15.37:0/1826488303 >> v2:172.21.15.37:3300/0 conn(0x7fde6009cb60 msgr2=0x7fde600a1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:37:27.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde65275640 1 --2- 172.21.15.37:0/1826488303 >> v2:172.21.15.37:3300/0 conn(0x7fde6009cb60 0x7fde600a1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:27.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde65275640 10 monclient: _finish_hunting 0 2024-04-26T00:37:27.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde65275640 1 monclient: found mon.noname-b 2024-04-26T00:37:27.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde65275640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:37:27.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde65275640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:37:27.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.052+0000 7fde65275640 1 -- 172.21.15.37:0/1826488303 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde6006b1d0 con 0x7fde6009c780 2024-04-26T00:37:27.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.056+0000 7fde65275640 1 -- 172.21.15.37:0/1826488303 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde6009c780 msgr2=0x7fde601350b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:27.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.056+0000 7fde65275640 1 --2- 172.21.15.37:0/1826488303 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde6009c780 0x7fde601350b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fde600a0310 tx=0x7fde5c01c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39b76bef0b413e96 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:27.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.056+0000 7fde57fff640 1 -- 172.21.15.37:0/1826488303 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5c0b0070 con 0x7fde6009c780 2024-04-26T00:37:27.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.056+0000 7fde57fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:27.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.056+0000 7fde57fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:37:27.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.056+0000 7fde57fff640 10 monclient: dump: 2024-04-26T00:37:27.066 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:27.066 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:27.066 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:27.066 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:27.066 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:27.066 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:27.066 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:27.066 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:27.066 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:27.066 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:27.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.056+0000 7fde57fff640 10 monclient: _finish_auth 0 2024-04-26T00:37:27.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.056+0000 7fde57fff640 10 monclient: _check_auth_tickets 2024-04-26T00:37:27.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.056+0000 7fde57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:37:27.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.056+0000 7fde57fff640 1 -- 172.21.15.37:0/1826488303 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde5c0ab600 con 0x7fde6009c780 2024-04-26T00:37:27.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.056+0000 7fde57fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:37:27.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.056+0000 7fde674d9640 5 monclient: authenticate success, global_id 204200 2024-04-26T00:37:27.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.056+0000 7fde57fff640 1 -- 172.21.15.37:0/1826488303 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5c0ab9d0 con 0x7fde6009c780 2024-04-26T00:37:27.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.056+0000 7fde57fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:27.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.056+0000 7fde57fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:37:27.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.056+0000 7fde57fff640 10 monclient: dump: 2024-04-26T00:37:27.067 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:27.067 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:27.067 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:27.067 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:27.067 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:27.067 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:27.067 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:27.067 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:27.067 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:27.068 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:27.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.056+0000 7fde674d9640 10 monclient: get_monmap_and_config success 2024-04-26T00:37:27.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.056+0000 7fde674d9640 4 set_mon_vals no callback set 2024-04-26T00:37:27.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.056+0000 7fde674d9640 10 monclient: shutdown 2024-04-26T00:37:27.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.056+0000 7fde674d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:37:27.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.056+0000 7fde674d9640 1 -- 172.21.15.37:0/1826488303 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde6009c780 msgr2=0x7fde601350b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:27.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.056+0000 7fde674d9640 1 --2- 172.21.15.37:0/1826488303 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde6009c780 0x7fde601350b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fde600a0310 tx=0x7fde5c01c3e0 comp rx=0 tx=0).stop 2024-04-26T00:37:27.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.056+0000 7fde674d9640 1 -- 172.21.15.37:0/1826488303 shutdown_connections 2024-04-26T00:37:27.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.056+0000 7fde674d9640 1 --2- 172.21.15.37:0/1826488303 >> v2:172.21.15.45:3300/0 conn(0x7fde600a20a0 0x7fde600a2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:27.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.056+0000 7fde674d9640 1 --2- 172.21.15.37:0/1826488303 >> v2:172.21.15.37:3300/0 conn(0x7fde6009cb60 0x7fde600a1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:27.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.056+0000 7fde674d9640 1 --2- 172.21.15.37:0/1826488303 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde6009c780 0x7fde601350b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:27.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.056+0000 7fde674d9640 1 -- 172.21.15.37:0/1826488303 >> 172.21.15.37:0/1826488303 conn(0x7fde600a2e20 msgr2=0x7fde600a3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:37:27.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.056+0000 7fde674d9640 1 -- 172.21.15.37:0/1826488303 shutdown_connections 2024-04-26T00:37:27.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.056+0000 7fde674d9640 1 -- 172.21.15.37:0/1826488303 wait complete. 2024-04-26T00:37:27.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.056+0000 7fde674d9640 10 monclient: build_initial_monmap 2024-04-26T00:37:27.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.056+0000 7fde674d9640 10 monclient: monmap: 2024-04-26T00:37:27.069 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:37:27.069 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:37:27.069 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:37:27.069 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:37:27.069 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:37:27.069 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:37:27.069 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:37:27.069 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:37:27.069 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:37:27.069 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:27.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.056+0000 7fde674d9640 1 Processor -- start 2024-04-26T00:37:27.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.056+0000 7fde674d9640 1 -- start start 2024-04-26T00:37:27.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.056+0000 7fde674d9640 10 monclient: init 2024-04-26T00:37:27.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.056+0000 7fde674d9640 10 monclient: _reopen_session rank -1 2024-04-26T00:37:27.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde674d9640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:37:27.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde674d9640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde6009c780 0x7fde601cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:27.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde674d9640 10 monclient(hunting): picked mon.noname-a con 0x7fde6009c780 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:37:27.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde674d9640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde6009cb60 0x7fde601ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:27.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde674d9640 10 monclient(hunting): picked mon.noname-c con 0x7fde6009cb60 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:37:27.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde674d9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde600a20a0 0x7fde601cd450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:27.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde674d9640 10 monclient(hunting): picked mon.noname-b con 0x7fde600a20a0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:37:27.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde674d9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:27.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde674d9640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fde6009d0c0 con 0x7fde6009c780 2024-04-26T00:37:27.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde674d9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:27.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde674d9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fde6009cf40 con 0x7fde600a20a0 2024-04-26T00:37:27.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde674d9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:27.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde674d9640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fde6013e740 con 0x7fde6009cb60 2024-04-26T00:37:27.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde674d9640 10 monclient(hunting): _renew_subs 2024-04-26T00:37:27.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde674d9640 10 monclient(hunting): authenticate will time out at 2028.368088s 2024-04-26T00:37:27.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde65275640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde6009c780 0x7fde601cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:27.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde64a74640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde6009cb60 0x7fde601ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:27.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde65275640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde6009c780 0x7fde601cc9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:45506/0 (socket says 172.21.15.37:45506) 2024-04-26T00:37:27.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde64a74640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde6009cb60 0x7fde601ccf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:34572/0 (socket says 172.21.15.37:34572) 2024-04-26T00:37:27.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde65275640 1 -- 172.21.15.37:0/2876972350 learned_addr learned my addr 172.21.15.37:0/2876972350 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:37:27.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde65275640 10 monclient(hunting): get_auth_request con 0x7fde6009c780 auth_method 0 2024-04-26T00:37:27.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde65275640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:27.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde65275640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:27.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde65275640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:27.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde64a74640 10 monclient(hunting): get_auth_request con 0x7fde6009cb60 auth_method 0 2024-04-26T00:37:27.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde64a74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:27.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde64a74640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:27.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde64a74640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:27.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde64a74640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:27.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde64a74640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:27.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde64a74640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:27.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde65275640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:27.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde65275640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:27.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde65275640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:27.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde65a76640 1 --2- 172.21.15.37:0/2876972350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde600a20a0 0x7fde601cd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:27.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde65a76640 10 monclient(hunting): get_auth_request con 0x7fde600a20a0 auth_method 0 2024-04-26T00:37:27.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde65a76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:27.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde65a76640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:27.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde65a76640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:27.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde64a74640 10 monclient(hunting): handle_auth_done global_id 204203 payload 824 2024-04-26T00:37:27.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde64a74640 1 -- 172.21.15.37:0/2876972350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde600a20a0 msgr2=0x7fde601cd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:27.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde64a74640 1 --2- 172.21.15.37:0/2876972350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde600a20a0 0x7fde601cd450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:27.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde64a74640 1 -- 172.21.15.37:0/2876972350 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde6009c780 msgr2=0x7fde601cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:27.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde64a74640 1 --2- 172.21.15.37:0/2876972350 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde6009c780 0x7fde601cc9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:27.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde64a74640 10 monclient: _finish_hunting 0 2024-04-26T00:37:27.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde64a74640 1 monclient: found mon.noname-c 2024-04-26T00:37:27.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde64a74640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:37:27.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde64a74640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:37:27.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde64a74640 1 -- 172.21.15.37:0/2876972350 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde600a0360 con 0x7fde6009cb60 2024-04-26T00:37:27.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde65275640 1 --2- 172.21.15.37:0/2876972350 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fde6009c780 0x7fde601cc9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:37:27.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde64a74640 1 -- 172.21.15.37:0/2876972350 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde6009cb60 msgr2=0x7fde601ccf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:27.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde64a74640 1 --2- 172.21.15.37:0/2876972350 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde6009cb60 0x7fde601ccf10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fde5000e870 tx=0x7fde5000ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:27.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde55ffb640 1 -- 172.21.15.37:0/2876972350 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde50017070 con 0x7fde6009cb60 2024-04-26T00:37:27.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde55ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:27.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde55ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:37:27.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde55ffb640 10 monclient: dump: 2024-04-26T00:37:27.074 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:27.074 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:27.074 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:27.074 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:27.074 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:27.074 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:27.074 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:27.074 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:27.075 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:27.075 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:27.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde55ffb640 10 monclient: _finish_auth 0 2024-04-26T00:37:27.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde55ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:37:27.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:37:27.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde55ffb640 1 -- 172.21.15.37:0/2876972350 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde5000caf0 con 0x7fde6009cb60 2024-04-26T00:37:27.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde674d9640 5 monclient: authenticate success, global_id 204203 2024-04-26T00:37:27.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde674d9640 10 monclient: _renew_subs 2024-04-26T00:37:27.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde674d9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:37:27.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde674d9640 1 -- 172.21.15.37:0/2876972350 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde6012e0a0 con 0x7fde6009cb60 2024-04-26T00:37:27.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde55ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:37:27.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde55ffb640 1 -- 172.21.15.37:0/2876972350 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde50013470 con 0x7fde6009cb60 2024-04-26T00:37:27.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde55ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:27.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde55ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:37:27.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde55ffb640 10 monclient: dump: 2024-04-26T00:37:27.075 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:27.076 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:27.076 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:27.076 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:27.076 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:27.076 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:27.076 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:27.076 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:27.076 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:27.076 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:27.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde577fe640 4 set_mon_vals no callback set 2024-04-26T00:37:27.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde674d9640 10 monclient: _renew_subs 2024-04-26T00:37:27.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde674d9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:37:27.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde674d9640 1 -- 172.21.15.37:0/2876972350 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fde6012c8a0 con 0x7fde6009cb60 2024-04-26T00:37:27.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde55ffb640 1 -- 172.21.15.37:0/2876972350 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fde5001a070 con 0x7fde6009cb60 2024-04-26T00:37:27.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde674d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:27.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde674d9640 1 --2- 172.21.15.37:0/2876972350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde6012e960 0x7fde60099040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:27.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde674d9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:27.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.060+0000 7fde674d9640 1 -- 172.21.15.37:0/2876972350 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fde6004f510 con 0x7fde6012e960 2024-04-26T00:37:27.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.064+0000 7fde55ffb640 1 --2- 172.21.15.37:0/2876972350 >> v2:172.21.15.45:6812/2379918623 conn(0x7fde44035ed0 0x7fde44038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:27.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.064+0000 7fde55ffb640 1 -- 172.21.15.37:0/2876972350 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(687..687 src has 370..687) ==== 3943+0+0 (secure 0 0 0) 0x7fde50048a90 con 0x7fde6009cb60 2024-04-26T00:37:27.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.064+0000 7fde65275640 1 --2- 172.21.15.37:0/2876972350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde6012e960 0x7fde60099040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:27.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.064+0000 7fde65275640 10 monclient: get_auth_request con 0x7fde6012e960 auth_method 0 2024-04-26T00:37:27.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.064+0000 7fde65275640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:27.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.064+0000 7fde65275640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:27.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.064+0000 7fde65275640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:27.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.064+0000 7fde65a76640 1 --2- 172.21.15.37:0/2876972350 >> v2:172.21.15.45:6812/2379918623 conn(0x7fde44035ed0 0x7fde44038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:27.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.064+0000 7fde65275640 1 -- 172.21.15.37:0/2876972350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde6012e960 msgr2=0x7fde60099040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:37:27.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.064+0000 7fde65275640 1 -- 172.21.15.37:0/2876972350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde6012e960 msgr2=0x7fde60099040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:37:27.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.064+0000 7fde65275640 1 --2- 172.21.15.37:0/2876972350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde6012e960 0x7fde60099040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:37:27.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.064+0000 7fde65275640 1 --2- 172.21.15.37:0/2876972350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde6012e960 0x7fde60099040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:37:27.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.064+0000 7fde65a76640 10 monclient: get_auth_request con 0x7fde44035ed0 auth_method 0 2024-04-26T00:37:27.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.064+0000 7fde65a76640 1 --2- 172.21.15.37:0/2876972350 >> v2:172.21.15.45:6812/2379918623 conn(0x7fde44035ed0 0x7fde44038390 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fde580074e0 tx=0x7fde58007470 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:27.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.268+0000 7fde65275640 1 --2- 172.21.15.37:0/2876972350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde6012e960 0x7fde60099040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:27.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.268+0000 7fde65275640 10 monclient: get_auth_request con 0x7fde6012e960 auth_method 0 2024-04-26T00:37:27.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.268+0000 7fde65275640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:27.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.268+0000 7fde65275640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:27.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.268+0000 7fde65275640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:37:27.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.268+0000 7fde65275640 1 -- 172.21.15.37:0/2876972350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde6012e960 msgr2=0x7fde60099040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:37:27.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.268+0000 7fde65275640 1 -- 172.21.15.37:0/2876972350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde6012e960 msgr2=0x7fde60099040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:37:27.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.268+0000 7fde65275640 1 --2- 172.21.15.37:0/2876972350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde6012e960 0x7fde60099040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:37:27.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.268+0000 7fde65275640 1 --2- 172.21.15.37:0/2876972350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde6012e960 0x7fde60099040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:37:27.294 INFO:tasks.rados.rados.0.smithi045.stdout:3074: finishing rollback tid 1 to smithi04526579-19 2024-04-26T00:37:27.294 INFO:tasks.rados.rados.0.smithi045.stdout:3074: finishing rollback tid 2 to smithi04526579-19 2024-04-26T00:37:27.294 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 2412 (ObjNum 1055 snap 318 seq_num 1055) dirty exists 2024-04-26T00:37:27.294 INFO:tasks.rados.rados.0.smithi045.stdout:3075: finishing write tid 1 to smithi04526579-34 2024-04-26T00:37:27.294 INFO:tasks.rados.rados.0.smithi045.stdout:3075: finishing write tid 2 to smithi04526579-34 2024-04-26T00:37:27.294 INFO:tasks.rados.rados.0.smithi045.stdout:3075: finishing write tid 3 to smithi04526579-34 2024-04-26T00:37:27.294 INFO:tasks.rados.rados.0.smithi045.stdout:3075: finishing write tid 4 to smithi04526579-34 2024-04-26T00:37:27.294 INFO:tasks.rados.rados.0.smithi045.stdout:3075: oid 34 updating version 0 to 2333 2024-04-26T00:37:27.295 INFO:tasks.rados.rados.0.smithi045.stdout:3075: oid 34 version 2333 is already newer than 2331 2024-04-26T00:37:27.295 INFO:tasks.rados.rados.0.smithi045.stdout:3075: oid 34 version 2333 is already newer than 2332 2024-04-26T00:37:27.295 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 2333 (ObjNum 1104 snap 329 seq_num 1104) dirty exists 2024-04-26T00:37:27.295 INFO:tasks.rados.rados.0.smithi045.stdout:3075: left oid 34 (ObjNum 1104 snap 329 seq_num 1104) 2024-04-26T00:37:27.295 INFO:tasks.rados.rados.0.smithi045.stdout:3077: finishing write tid 1 to smithi04526579-12 2024-04-26T00:37:27.295 INFO:tasks.rados.rados.0.smithi045.stdout:3077: finishing write tid 2 to smithi04526579-12 2024-04-26T00:37:27.295 INFO:tasks.rados.rados.0.smithi045.stdout:3077: finishing write tid 3 to smithi04526579-12 2024-04-26T00:37:27.295 INFO:tasks.rados.rados.0.smithi045.stdout:3077: finishing write tid 4 to smithi04526579-12 2024-04-26T00:37:27.295 INFO:tasks.rados.rados.0.smithi045.stdout:3077: oid 12 updating version 0 to 2214 2024-04-26T00:37:27.295 INFO:tasks.rados.rados.0.smithi045.stdout:3077: oid 12 updating version 2214 to 2216 2024-04-26T00:37:27.295 INFO:tasks.rados.rados.0.smithi045.stdout:3077: oid 12 version 2216 is already newer than 2215 2024-04-26T00:37:27.295 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 2216 (ObjNum 1105 snap 329 seq_num 1105) dirty exists 2024-04-26T00:37:27.295 INFO:tasks.rados.rados.0.smithi045.stdout:3077: left oid 12 (ObjNum 1105 snap 329 seq_num 1105) 2024-04-26T00:37:27.295 INFO:tasks.rados.rados.0.smithi045.stdout:3078: expect (ObjNum 1093 snap 324 seq_num 1093) 2024-04-26T00:37:27.372 INFO:tasks.rados.rados.0.smithi045.stdout:3079: finishing write tid 1 to smithi04526579-22 2024-04-26T00:37:27.372 INFO:tasks.rados.rados.0.smithi045.stdout:3079: finishing write tid 2 to smithi04526579-22 2024-04-26T00:37:27.372 INFO:tasks.rados.rados.0.smithi045.stdout:3079: finishing write tid 3 to smithi04526579-22 2024-04-26T00:37:27.372 INFO:tasks.rados.rados.0.smithi045.stdout:3079: finishing write tid 4 to smithi04526579-22 2024-04-26T00:37:27.372 INFO:tasks.rados.rados.0.smithi045.stdout:3079: oid 22 updating version 0 to 2411 2024-04-26T00:37:27.372 INFO:tasks.rados.rados.0.smithi045.stdout:3079: oid 22 updating version 2411 to 2412 2024-04-26T00:37:27.372 INFO:tasks.rados.rados.0.smithi045.stdout:3079: oid 22 updating version 2412 to 2413 2024-04-26T00:37:27.372 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 2413 (ObjNum 1106 snap 329 seq_num 1106) dirty exists 2024-04-26T00:37:27.372 INFO:tasks.rados.rados.0.smithi045.stdout:3079: left oid 22 (ObjNum 1106 snap 329 seq_num 1106) 2024-04-26T00:37:27.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.672+0000 7fde65275640 1 --2- 172.21.15.37:0/2876972350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde6012e960 0x7fde60099040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:27.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.672+0000 7fde65275640 10 monclient: get_auth_request con 0x7fde6012e960 auth_method 0 2024-04-26T00:37:27.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.672+0000 7fde65275640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:27.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.672+0000 7fde65275640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:27.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.672+0000 7fde65275640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:37:27.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.672+0000 7fde65275640 1 -- 172.21.15.37:0/2876972350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde6012e960 msgr2=0x7fde60099040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:37:27.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.672+0000 7fde65275640 1 -- 172.21.15.37:0/2876972350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde6012e960 msgr2=0x7fde60099040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:37:27.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.672+0000 7fde65275640 1 --2- 172.21.15.37:0/2876972350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde6012e960 0x7fde60099040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:37:27.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:27.672+0000 7fde65275640 1 --2- 172.21.15.37:0/2876972350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde6012e960 0x7fde60099040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:37:28.013 INFO:tasks.rados.rados.0.smithi045.stdout:3070: done (8 left) 2024-04-26T00:37:28.013 INFO:tasks.rados.rados.0.smithi045.stdout:3072: done (7 left) 2024-04-26T00:37:28.013 INFO:tasks.rados.rados.0.smithi045.stdout:3073: done (6 left) 2024-04-26T00:37:28.013 INFO:tasks.rados.rados.0.smithi045.stdout:3074: done (5 left) 2024-04-26T00:37:28.013 INFO:tasks.rados.rados.0.smithi045.stdout:3075: done (4 left) 2024-04-26T00:37:28.013 INFO:tasks.rados.rados.0.smithi045.stdout:3077: done (3 left) 2024-04-26T00:37:28.013 INFO:tasks.rados.rados.0.smithi045.stdout:3078: done (2 left) 2024-04-26T00:37:28.013 INFO:tasks.rados.rados.0.smithi045.stdout:3079: done (1 left) 2024-04-26T00:37:28.013 INFO:tasks.rados.rados.0.smithi045.stdout:3080: done (0 left) 2024-04-26T00:37:28.013 INFO:tasks.rados.rados.0.smithi045.stdout:3081: write (excl) oid 3 current snap is 330 2024-04-26T00:37:28.013 INFO:tasks.rados.rados.0.smithi045.stdout:3081: seq_num 1107 ranges {605349=571828,1679096=420377} 2024-04-26T00:37:28.026 INFO:tasks.rados.rados.0.smithi045.stdout:3081: writing smithi04526579-3 from 605349 to 1177177 tid 1 2024-04-26T00:37:28.036 INFO:tasks.rados.rados.0.smithi045.stdout:3081: writing smithi04526579-3 from 1679096 to 2099473 tid 2 2024-04-26T00:37:28.036 INFO:tasks.rados.rados.0.smithi045.stdout:3082: read oid 5 snap -1 2024-04-26T00:37:28.036 INFO:tasks.rados.rados.0.smithi045.stdout:3082: expect (ObjNum 955 snap 295 seq_num 955) 2024-04-26T00:37:28.036 INFO:tasks.rados.rados.0.smithi045.stdout:3083: write (excl) oid 31 current snap is 330 2024-04-26T00:37:28.036 INFO:tasks.rados.rados.0.smithi045.stdout:3083: seq_num 1108 ranges {622156=647077,1737578=680388} 2024-04-26T00:37:28.048 INFO:tasks.rados.rados.0.smithi045.stdout:3083: writing smithi04526579-31 from 622156 to 1269233 tid 1 2024-04-26T00:37:28.060 INFO:tasks.rados.rados.0.smithi045.stdout:3083: writing smithi04526579-31 from 1737578 to 2417966 tid 2 2024-04-26T00:37:28.060 INFO:tasks.rados.rados.0.smithi045.stdout:3084: read oid 16 snap -1 2024-04-26T00:37:28.060 INFO:tasks.rados.rados.0.smithi045.stdout:3084: expect (ObjNum 1102 snap 329 seq_num 1102) 2024-04-26T00:37:28.060 INFO:tasks.rados.rados.0.smithi045.stdout:3085: snap_create 2024-04-26T00:37:28.060 INFO:tasks.rados.rados.0.smithi045.stdout:3081: finishing write tid 1 to smithi04526579-3 2024-04-26T00:37:28.061 INFO:tasks.rados.rados.0.smithi045.stdout:3081: finishing write tid 2 to smithi04526579-3 2024-04-26T00:37:28.061 INFO:tasks.rados.rados.0.smithi045.stdout:3081: finishing write tid 3 to smithi04526579-3 2024-04-26T00:37:28.061 INFO:tasks.rados.rados.0.smithi045.stdout:3082: expect (ObjNum 955 snap 295 seq_num 955) 2024-04-26T00:37:28.151 INFO:tasks.rados.rados.0.smithi045.stdout:3081: finishing write tid 4 to smithi04526579-3 2024-04-26T00:37:28.152 INFO:tasks.rados.rados.0.smithi045.stdout:3081: oid 3 updating version 0 to 2335 2024-04-26T00:37:28.152 INFO:tasks.rados.rados.0.smithi045.stdout:3081: oid 3 updating version 2335 to 2336 2024-04-26T00:37:28.152 INFO:tasks.rados.rados.0.smithi045.stdout:3081: oid 3 version 2336 is already newer than 2334 2024-04-26T00:37:28.152 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 2336 (ObjNum 1107 snap 330 seq_num 1107) dirty exists 2024-04-26T00:37:28.152 INFO:tasks.rados.rados.0.smithi045.stdout:3081: left oid 3 (ObjNum 1107 snap 330 seq_num 1107) 2024-04-26T00:37:28.152 INFO:tasks.rados.rados.0.smithi045.stdout:3084: expect (ObjNum 1102 snap 329 seq_num 1102) 2024-04-26T00:37:28.269 INFO:tasks.rados.rados.0.smithi045.stdout:3083: finishing write tid 1 to smithi04526579-31 2024-04-26T00:37:28.270 INFO:tasks.rados.rados.0.smithi045.stdout:3083: finishing write tid 2 to smithi04526579-31 2024-04-26T00:37:28.270 INFO:tasks.rados.rados.0.smithi045.stdout:3083: finishing write tid 3 to smithi04526579-31 2024-04-26T00:37:28.270 INFO:tasks.rados.rados.0.smithi045.stdout:3083: finishing write tid 4 to smithi04526579-31 2024-04-26T00:37:28.270 INFO:tasks.rados.rados.0.smithi045.stdout:3083: oid 31 updating version 0 to 2046 2024-04-26T00:37:28.270 INFO:tasks.rados.rados.0.smithi045.stdout:3083: oid 31 version 2046 is already newer than 2045 2024-04-26T00:37:28.270 INFO:tasks.rados.rados.0.smithi045.stdout:3083: oid 31 version 2046 is already newer than 2044 2024-04-26T00:37:28.270 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 2046 (ObjNum 1108 snap 330 seq_num 1108) dirty exists 2024-04-26T00:37:28.270 INFO:tasks.rados.rados.0.smithi045.stdout:3083: left oid 31 (ObjNum 1108 snap 330 seq_num 1108) 2024-04-26T00:37:28.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.476+0000 7fde65275640 1 --2- 172.21.15.37:0/2876972350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde6012e960 0x7fde60099040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:28.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.476+0000 7fde65275640 10 monclient: get_auth_request con 0x7fde6012e960 auth_method 0 2024-04-26T00:37:28.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.476+0000 7fde65275640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:28.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.476+0000 7fde65275640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:28.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.476+0000 7fde65275640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:37:28.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.476+0000 7fde65275640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:28.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.476+0000 7fde65275640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:28.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.476+0000 7fde65275640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:28.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.476+0000 7fde65275640 10 monclient(hunting): handle_auth_done global_id 214099 payload 824 2024-04-26T00:37:28.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.476+0000 7fde65275640 1 -- 172.21.15.37:0/2876972350 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fde600008e0 con 0x7fde6012e960 2024-04-26T00:37:28.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.480+0000 7fde65275640 1 -- 172.21.15.37:0/2876972350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde6012e960 msgr2=0x7fde60099040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:28.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.480+0000 7fde65275640 1 --2- 172.21.15.37:0/2876972350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde6012e960 0x7fde60099040 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fde5c01aa90 tx=0x7fde5c019180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:28.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.480+0000 7fde55ffb640 1 -- 172.21.15.37:0/2876972350 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5c0b0040 con 0x7fde6012e960 2024-04-26T00:37:28.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.480+0000 7fde55ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:28.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.480+0000 7fde55ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:37:28.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.480+0000 7fde55ffb640 10 monclient: dump: 2024-04-26T00:37:28.480 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:28.480 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:28.480 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:28.480 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:28.480 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:28.480 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:28.480 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:28.480 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:28.481 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:28.481 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:28.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.480+0000 7fde55ffb640 1 -- 172.21.15.37:0/2876972350 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fde5c0b5410 con 0x7fde6012e960 2024-04-26T00:37:28.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.480+0000 7fde55ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:28.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.480+0000 7fde55ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:37:28.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.480+0000 7fde55ffb640 1 -- 172.21.15.37:0/2876972350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde6012e960 msgr2=0x7fde60099040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:28.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.480+0000 7fde55ffb640 1 --2- 172.21.15.37:0/2876972350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde6012e960 0x7fde60099040 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fde5c01aa90 tx=0x7fde5c019180 comp rx=0 tx=0).stop 2024-04-26T00:37:28.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.480+0000 7fde55ffb640 1 -- 172.21.15.37:0/2876972350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde6012e960 msgr2=0x7fde60099040 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:37:28.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.480+0000 7fde55ffb640 1 --2- 172.21.15.37:0/2876972350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde6012e960 0x7fde60099040 secure :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fde5c01aa90 tx=0x7fde5c019180 comp rx=0 tx=0).stop 2024-04-26T00:37:28.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.488+0000 7fde674d9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:37:28.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.488+0000 7fde674d9640 1 --2- 172.21.15.37:0/2876972350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde60099580 0x7fde60099990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:28.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.488+0000 7fde674d9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:28.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.488+0000 7fde674d9640 1 -- 172.21.15.37:0/2876972350 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fde60099ed0 con 0x7fde60099580 2024-04-26T00:37:28.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.488+0000 7fde65275640 1 --2- 172.21.15.37:0/2876972350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde60099580 0x7fde60099990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:28.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.488+0000 7fde65275640 10 monclient: get_auth_request con 0x7fde60099580 auth_method 0 2024-04-26T00:37:28.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.488+0000 7fde65275640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:28.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.488+0000 7fde65275640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:28.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.488+0000 7fde65275640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:28.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.488+0000 7fde65275640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:28.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.488+0000 7fde65275640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:28.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.488+0000 7fde65275640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:28.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.488+0000 7fde65275640 10 monclient(hunting): handle_auth_done global_id 214102 payload 824 2024-04-26T00:37:28.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.488+0000 7fde65275640 1 -- 172.21.15.37:0/2876972350 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fde600726a0 con 0x7fde60099580 2024-04-26T00:37:28.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.492+0000 7fde65275640 1 -- 172.21.15.37:0/2876972350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde60099580 msgr2=0x7fde60099990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:28.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.492+0000 7fde65275640 1 --2- 172.21.15.37:0/2876972350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde60099580 0x7fde60099990 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fde5c000c10 tx=0x7fde5c0a8000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:28.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.492+0000 7fde55ffb640 1 -- 172.21.15.37:0/2876972350 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5c0c9070 con 0x7fde60099580 2024-04-26T00:37:28.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.492+0000 7fde55ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:28.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.492+0000 7fde55ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:37:28.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.492+0000 7fde55ffb640 10 monclient: dump: 2024-04-26T00:37:28.493 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:28.493 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:28.493 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:28.494 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:28.494 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:28.494 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:28.494 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:28.494 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:28.494 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:28.494 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:28.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.492+0000 7fde55ffb640 1 -- 172.21.15.37:0/2876972350 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7fde5c003de0 con 0x7fde60099580 2024-04-26T00:37:28.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.492+0000 7fde55ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:37:28.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.492+0000 7fde55ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:37:28.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.492+0000 7fde55ffb640 1 -- 172.21.15.37:0/2876972350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde60099580 msgr2=0x7fde60099990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:28.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.492+0000 7fde55ffb640 1 --2- 172.21.15.37:0/2876972350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde60099580 0x7fde60099990 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fde5c000c10 tx=0x7fde5c0a8000 comp rx=0 tx=0).stop 2024-04-26T00:37:28.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.492+0000 7fde55ffb640 1 -- 172.21.15.37:0/2876972350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde60099580 msgr2=0x7fde60099990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:37:28.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.492+0000 7fde55ffb640 1 --2- 172.21.15.37:0/2876972350 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fde60099580 0x7fde60099990 secure :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fde5c000c10 tx=0x7fde5c0a8000 comp rx=0 tx=0).stop 2024-04-26T00:37:28.495 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:37:28.495 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:37:28.495 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:37:28.496 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:37:28.496 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 132, 2024-04-26T00:37:28.496 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 5599, 2024-04-26T00:37:28.496 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:37:28.496 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:37:28.496 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:37:28.496 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:37:28.496 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:28.496 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 2, 2024-04-26T00:37:28.496 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:37:28.496 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:37:28.496 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:37:28.496 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:37:28.496 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:37:28.497 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:37:28.497 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:37:28.497 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:37:28.497 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:37:28.497 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:37:28.497 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:37:28.497 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:37:28.497 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:37:28.497 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:37:28.497 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:28.497 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:37:28.497 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:37:28.497 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:37:28.497 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:37:28.497 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:37:28.498 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:37:28.498 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:37:28.498 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:37:28.498 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:37:28.498 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:37:28.498 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:37:28.498 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:37:28.498 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:37:28.498 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:28.498 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:28.498 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:37:28.498 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:37:28.498 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:37:28.498 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:37:28.498 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:37:28.499 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:37:28.499 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:37:28.499 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:37:28.499 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:37:28.499 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:37:28.499 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:37:28.499 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:37:28.499 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:37:28.499 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:37:28.499 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:37:28.499 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:37:28.499 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:37:28.499 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:37:28.499 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:37:28.499 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:37:28.500 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:37:28.500 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:37:28.500 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:37:28.500 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:37:28.500 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:37:28.500 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:37:28.500 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:37:28.500 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:37:28.500 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:28.500 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:37:28.500 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:28.500 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:37:28.500 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:28.500 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:37:28.500 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:37:28.501 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:37:28.501 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:37:28.501 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:28.501 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:37:28.501 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:37:28.501 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:28.501 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:28.501 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:28.501 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:37:28.501 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:37:28.501 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:28.501 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:28.501 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:28.501 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:28.501 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:37:28.502 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:37:28.502 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:37:28.502 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:37:28.502 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:37:28.502 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:28.502 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:28.502 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:37:28.502 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:37:28.502 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:37:28.502 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:37:28.502 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:28.502 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:37:28.502 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:37:28.503 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:28.503 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:28.503 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:28.503 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:37:28.503 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:37:28.503 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:28.503 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:28.503 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:28.503 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:28.503 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:37:28.503 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:37:28.503 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:37:28.503 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:37:28.503 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:37:28.503 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:28.504 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:28.504 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:37:28.504 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:37:28.504 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:37:28.504 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:37:28.504 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:28.504 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:37:28.504 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:37:28.504 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:28.504 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:28.504 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:28.504 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:37:28.504 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:37:28.504 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:28.505 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:28.505 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:28.505 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:28.505 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:37:28.505 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:37:28.505 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:37:28.505 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:37:28.505 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:37:28.505 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:28.505 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:28.505 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:28.505 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:37:28.505 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:37:28.505 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:28.505 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:37:28.506 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:37:28.506 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:37:28.506 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:28.506 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:28.506 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:37:28.506 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:28.506 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:37:28.506 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:37:28.506 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:37:28.506 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:28.506 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:28.506 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:28.506 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:37:28.506 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:37:28.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.492+0000 7fde674d9640 1 -- 172.21.15.37:0/2876972350 >> v2:172.21.15.45:6812/2379918623 conn(0x7fde44035ed0 msgr2=0x7fde44038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:28.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.492+0000 7fde674d9640 1 --2- 172.21.15.37:0/2876972350 >> v2:172.21.15.45:6812/2379918623 conn(0x7fde44035ed0 0x7fde44038390 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fde580074e0 tx=0x7fde58007470 comp rx=0 tx=0).stop 2024-04-26T00:37:28.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.492+0000 7fde674d9640 10 monclient: shutdown 2024-04-26T00:37:28.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.492+0000 7fde674d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:37:28.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.492+0000 7fde674d9640 1 -- 172.21.15.37:0/2876972350 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde6009cb60 msgr2=0x7fde601ccf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:28.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.492+0000 7fde674d9640 1 --2- 172.21.15.37:0/2876972350 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fde6009cb60 0x7fde601ccf10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fde5000e870 tx=0x7fde5000ed30 comp rx=0 tx=0).stop 2024-04-26T00:37:28.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.492+0000 7fde65a76640 1 -- 172.21.15.37:0/2876972350 reap_dead start 2024-04-26T00:37:28.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.492+0000 7fde65a76640 1 -- 172.21.15.37:0/2876972350 reap_dead start 2024-04-26T00:37:28.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.492+0000 7fde674d9640 1 -- 172.21.15.37:0/2876972350 shutdown_connections 2024-04-26T00:37:28.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.492+0000 7fde674d9640 1 -- 172.21.15.37:0/2876972350 >> 172.21.15.37:0/2876972350 conn(0x7fde600a2e20 msgr2=0x7fde601315e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:37:28.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.492+0000 7fde674d9640 1 -- 172.21.15.37:0/2876972350 shutdown_connections 2024-04-26T00:37:28.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.496+0000 7fde674d9640 1 -- 172.21.15.37:0/2876972350 wait complete. 2024-04-26T00:37:28.520 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:37:28.520 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-26T00:37:28.520 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-26T00:37:28.568 INFO:tasks.rados.rados.0.smithi045.stdout:3081: done (4 left) 2024-04-26T00:37:28.568 INFO:tasks.rados.rados.0.smithi045.stdout:3082: done (3 left) 2024-04-26T00:37:28.568 INFO:tasks.rados.rados.0.smithi045.stdout:3083: done (2 left) 2024-04-26T00:37:28.569 INFO:tasks.rados.rados.0.smithi045.stdout:3084: done (1 left) 2024-04-26T00:37:28.569 INFO:tasks.rados.rados.0.smithi045.stdout:3085: done (0 left) 2024-04-26T00:37:28.569 INFO:tasks.rados.rados.0.smithi045.stdout:3086: write (excl) oid 26 current snap is 331 2024-04-26T00:37:28.569 INFO:tasks.rados.rados.0.smithi045.stdout:3086: seq_num 1109 ranges {638963=722326,2079707=424046} 2024-04-26T00:37:28.583 INFO:tasks.rados.rados.0.smithi045.stdout:3086: writing smithi04526579-26 from 638963 to 1361289 tid 1 2024-04-26T00:37:28.597 INFO:tasks.rados.rados.0.smithi045.stdout:3086: writing smithi04526579-26 from 2079707 to 2503753 tid 2 2024-04-26T00:37:28.597 INFO:tasks.rados.rados.0.smithi045.stdout:3087: write (excl) oid 31 current snap is 331 2024-04-26T00:37:28.597 INFO:tasks.rados.rados.0.smithi045.stdout:3087: seq_num 1110 ranges {655770=513928,1854542=684057} 2024-04-26T00:37:28.606 INFO:tasks.rados.rados.0.smithi045.stdout:3087: writing smithi04526579-31 from 655770 to 1169698 tid 1 2024-04-26T00:37:28.614 INFO:tasks.rados.rados.0.smithi045.stdout:3087: writing smithi04526579-31 from 1854542 to 2538599 tid 2 2024-04-26T00:37:28.614 INFO:tasks.rados.rados.0.smithi045.stdout:3088: write (excl) oid 35 current snap is 331 2024-04-26T00:37:28.614 INFO:tasks.rados.rados.0.smithi045.stdout:3086: finishing write tid 1 to smithi04526579-26 2024-04-26T00:37:28.614 INFO:tasks.rados.rados.0.smithi045.stdout:3088: seq_num 1111 ranges {672577=589177,1913024=427715} 2024-04-26T00:37:28.621 INFO:tasks.rados.rados.0.smithi045.stdout:3088: writing smithi04526579-35 from 672577 to 1261754 tid 1 2024-04-26T00:37:28.631 INFO:tasks.rados.rados.0.smithi045.stdout:3088: writing smithi04526579-35 from 1913024 to 2340739 tid 2 2024-04-26T00:37:28.631 INFO:tasks.rados.rados.0.smithi045.stdout:3089: write oid 48 current snap is 331 2024-04-26T00:37:28.631 INFO:tasks.rados.rados.0.smithi045.stdout:3089: seq_num 1112 ranges {689384=664426,1971506=687726} 2024-04-26T00:37:28.639 INFO:tasks.rados.rados.0.smithi045.stdout:3089: writing smithi04526579-48 from 689384 to 1353810 tid 1 2024-04-26T00:37:28.648 INFO:tasks.rados.rados.0.smithi045.stdout:3089: writing smithi04526579-48 from 1971506 to 2659232 tid 2 2024-04-26T00:37:28.648 INFO:tasks.rados.rados.0.smithi045.stdout:3090: snap_create 2024-04-26T00:37:28.649 INFO:tasks.rados.rados.0.smithi045.stdout:3086: finishing write tid 2 to smithi04526579-26 2024-04-26T00:37:28.649 INFO:tasks.rados.rados.0.smithi045.stdout:3086: finishing write tid 3 to smithi04526579-26 2024-04-26T00:37:28.649 INFO:tasks.rados.rados.0.smithi045.stdout:3086: finishing write tid 4 to smithi04526579-26 2024-04-26T00:37:28.649 INFO:tasks.rados.rados.0.smithi045.stdout:3086: oid 26 updating version 0 to 2436 2024-04-26T00:37:28.649 INFO:tasks.rados.rados.0.smithi045.stdout:3086: oid 26 version 2436 is already newer than 2435 2024-04-26T00:37:28.649 INFO:tasks.rados.rados.0.smithi045.stdout:3086: oid 26 version 2436 is already newer than 2434 2024-04-26T00:37:28.649 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 2436 (ObjNum 1109 snap 331 seq_num 1109) dirty exists 2024-04-26T00:37:28.649 INFO:tasks.rados.rados.0.smithi045.stdout:3086: left oid 26 (ObjNum 1109 snap 331 seq_num 1109) 2024-04-26T00:37:28.649 INFO:tasks.rados.rados.0.smithi045.stdout:3087: finishing write tid 1 to smithi04526579-31 2024-04-26T00:37:28.649 INFO:tasks.rados.rados.0.smithi045.stdout:3087: finishing write tid 2 to smithi04526579-31 2024-04-26T00:37:28.649 INFO:tasks.rados.rados.0.smithi045.stdout:3087: finishing write tid 3 to smithi04526579-31 2024-04-26T00:37:28.649 INFO:tasks.rados.rados.0.smithi045.stdout:3087: finishing write tid 4 to smithi04526579-31 2024-04-26T00:37:28.649 INFO:tasks.rados.rados.0.smithi045.stdout:3087: oid 31 updating version 0 to 2047 2024-04-26T00:37:28.649 INFO:tasks.rados.rados.0.smithi045.stdout:3087: oid 31 updating version 2047 to 2049 2024-04-26T00:37:28.649 INFO:tasks.rados.rados.0.smithi045.stdout:3087: oid 31 version 2049 is already newer than 2048 2024-04-26T00:37:28.650 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 2049 (ObjNum 1110 snap 331 seq_num 1110) dirty exists 2024-04-26T00:37:28.650 INFO:tasks.rados.rados.0.smithi045.stdout:3087: left oid 31 (ObjNum 1110 snap 331 seq_num 1110) 2024-04-26T00:37:28.650 INFO:tasks.rados.rados.0.smithi045.stdout:3088: finishing write tid 1 to smithi04526579-35 2024-04-26T00:37:28.650 INFO:tasks.rados.rados.0.smithi045.stdout:3088: finishing write tid 2 to smithi04526579-35 2024-04-26T00:37:28.650 INFO:tasks.rados.rados.0.smithi045.stdout:3088: finishing write tid 3 to smithi04526579-35 2024-04-26T00:37:28.650 INFO:tasks.rados.rados.0.smithi045.stdout:3088: finishing write tid 4 to smithi04526579-35 2024-04-26T00:37:28.650 INFO:tasks.rados.rados.0.smithi045.stdout:3088: oid 35 updating version 0 to 2414 2024-04-26T00:37:28.650 INFO:tasks.rados.rados.0.smithi045.stdout:3088: oid 35 updating version 2414 to 2415 2024-04-26T00:37:28.650 INFO:tasks.rados.rados.0.smithi045.stdout:3088: oid 35 version 2415 is already newer than 2413 2024-04-26T00:37:28.651 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 2415 (ObjNum 1111 snap 331 seq_num 1111) dirty exists 2024-04-26T00:37:28.651 INFO:tasks.rados.rados.0.smithi045.stdout:3088: left oid 35 (ObjNum 1111 snap 331 seq_num 1111) 2024-04-26T00:37:28.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff56e5b2640 10 monclient: get_monmap_and_config 2024-04-26T00:37:28.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff56e5b2640 10 monclient: build_initial_monmap 2024-04-26T00:37:28.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff56e5b2640 10 monclient: monmap: 2024-04-26T00:37:28.657 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:37:28.657 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:28.657 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:37:28.661433+0000 2024-04-26T00:37:28.657 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:37:28.661433+0000 2024-04-26T00:37:28.657 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:37:28.657 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:28.657 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:37:28.657 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:37:28.657 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:37:28.657 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:37:28.657 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:37:28.658 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:28.659 INFO:tasks.rados.rados.0.smithi045.stdout:3089: finishing write tid 1 to smithi04526579-48 2024-04-26T00:37:28.660 INFO:tasks.rados.rados.0.smithi045.stdout:3089: finishing write tid 2 to smithi04526579-48 2024-04-26T00:37:28.660 INFO:tasks.rados.rados.0.smithi045.stdout:3089: finishing write tid 3 to smithi04526579-48 2024-04-26T00:37:28.660 INFO:tasks.rados.rados.0.smithi045.stdout:3089: finishing write tid 4 to smithi04526579-48 2024-04-26T00:37:28.660 INFO:tasks.rados.rados.0.smithi045.stdout:3089: oid 48 updating version 0 to 2339 2024-04-26T00:37:28.660 INFO:tasks.rados.rados.0.smithi045.stdout:3089: oid 48 version 2339 is already newer than 2337 2024-04-26T00:37:28.660 INFO:tasks.rados.rados.0.smithi045.stdout:3089: oid 48 version 2339 is already newer than 2338 2024-04-26T00:37:28.660 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 2339 (ObjNum 1112 snap 331 seq_num 1112) dirty exists 2024-04-26T00:37:28.660 INFO:tasks.rados.rados.0.smithi045.stdout:3089: left oid 48 (ObjNum 1112 snap 331 seq_num 1112) 2024-04-26T00:37:28.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff56e5b2640 1 Processor -- start 2024-04-26T00:37:28.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff56e5b2640 1 -- start start 2024-04-26T00:37:28.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff56e5b2640 10 monclient: init 2024-04-26T00:37:28.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff56e5b2640 10 monclient: _reopen_session rank -1 2024-04-26T00:37:28.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff56e5b2640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-04-26T00:37:28.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff56e5b2640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff5680a0260 0x7ff5680a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:28.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff56e5b2640 10 monclient(hunting): picked mon.noname-a con 0x7ff5680a0260 addr v2:172.21.15.37:3300/0 2024-04-26T00:37:28.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff56e5b2640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff568134b30 0x7ff568134f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:28.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff56e5b2640 10 monclient(hunting): picked mon.noname-c con 0x7ff568134b30 addr v2:172.21.15.45:3300/0 2024-04-26T00:37:28.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff56e5b2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff568135490 addr v1:172.21.15.45:6789/0 2024-04-26T00:37:28.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff56e5b2640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5680a1130 con 0x7ff568135490 2024-04-26T00:37:28.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff56e5b2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:28.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff56e5b2640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7ff56809ec10 con 0x7ff5680a0260 2024-04-26T00:37:28.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff56e5b2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:28.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff56e5b2640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7ff56809f0f0 con 0x7ff568134b30 2024-04-26T00:37:28.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff56e5b2640 10 monclient(hunting): _renew_subs 2024-04-26T00:37:28.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff56e5b2640 10 monclient(hunting): authenticate will time out at 2029.966946s 2024-04-26T00:37:28.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff567fff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff5680a0260 0x7ff5680a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:28.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff567fff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff5680a0260 0x7ff5680a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:45512/0 (socket says 172.21.15.37:45512) 2024-04-26T00:37:28.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff567fff640 1 -- 172.21.15.37:0/2914384070 learned_addr learned my addr 172.21.15.37:0/2914384070 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:37:28.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff567fff640 10 monclient(hunting): get_auth_request con 0x7ff5680a0260 auth_method 0 2024-04-26T00:37:28.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff567fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:28.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff567fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:28.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff567fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:28.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff567fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:28.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff567fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:28.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff567fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:28.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff55ffff640 1 --2- 172.21.15.37:0/2914384070 >> v2:172.21.15.45:3300/0 conn(0x7ff568134b30 0x7ff568134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:28.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff55ffff640 10 monclient(hunting): get_auth_request con 0x7ff568134b30 auth_method 0 2024-04-26T00:37:28.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff55ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:28.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff55ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:28.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff55ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:28.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff567fff640 10 monclient(hunting): handle_auth_done global_id 144822 payload 274 2024-04-26T00:37:28.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff567fff640 1 -- 172.21.15.37:0/2914384070 >> v2:172.21.15.45:3300/0 conn(0x7ff568134b30 msgr2=0x7ff568134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:37:28.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff567fff640 1 --2- 172.21.15.37:0/2914384070 >> v2:172.21.15.45:3300/0 conn(0x7ff568134b30 0x7ff568134f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:28.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff567fff640 1 -- 172.21.15.37:0/2914384070 >> v1:172.21.15.45:6789/0 conn(0x7ff568135490 legacy=0x7ff56813be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:37:28.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff567fff640 10 monclient: _finish_hunting 0 2024-04-26T00:37:28.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff567fff640 1 monclient: found mon.noname-a 2024-04-26T00:37:28.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff567fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:37:28.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff567fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:37:28.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.656+0000 7ff567fff640 1 -- 172.21.15.37:0/2914384070 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff56806cff0 con 0x7ff5680a0260 2024-04-26T00:37:28.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.660+0000 7ff567fff640 1 -- 172.21.15.37:0/2914384070 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff5680a0260 msgr2=0x7ff5680a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:28.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.660+0000 7ff567fff640 1 --2- 172.21.15.37:0/2914384070 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff5680a0260 0x7ff5680a0640 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7ff56809f690 tx=0x7ff55801c1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54bb4d2c56a436e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:28.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.660+0000 7ff5677fe640 1 -- 172.21.15.37:0/2914384070 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5580b0070 con 0x7ff5680a0260 2024-04-26T00:37:28.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.660+0000 7ff5677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:28.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.660+0000 7ff5677fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:37:28.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.660+0000 7ff5677fe640 10 monclient: dump: 2024-04-26T00:37:28.667 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:28.667 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:28.667 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:28.667 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:28.667 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:28.668 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:28.668 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:28.668 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:28.668 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:28.668 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:28.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.660+0000 7ff5677fe640 10 monclient: _finish_auth 0 2024-04-26T00:37:28.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.660+0000 7ff5677fe640 10 monclient: _check_auth_tickets 2024-04-26T00:37:28.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.660+0000 7ff5677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:37:28.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.660+0000 7ff56e5b2640 5 monclient: authenticate success, global_id 144822 2024-04-26T00:37:28.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.660+0000 7ff5677fe640 1 -- 172.21.15.37:0/2914384070 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5580ab600 con 0x7ff5680a0260 2024-04-26T00:37:28.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.660+0000 7ff56e5b2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:37:28.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.660+0000 7ff5677fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:37:28.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.660+0000 7ff56e5b2640 10 monclient: get_monmap_and_config success 2024-04-26T00:37:28.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.660+0000 7ff56e5b2640 4 set_mon_vals no callback set 2024-04-26T00:37:28.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.660+0000 7ff5677fe640 1 -- 172.21.15.37:0/2914384070 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5580ab9d0 con 0x7ff5680a0260 2024-04-26T00:37:28.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.660+0000 7ff5677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:28.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff5677fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:37:28.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff5677fe640 10 monclient: dump: 2024-04-26T00:37:28.669 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:28.669 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:28.669 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:28.669 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:28.669 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:28.669 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:28.669 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:28.669 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:28.669 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:28.669 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:28.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff56e5b2640 10 monclient: shutdown 2024-04-26T00:37:28.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff56e5b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:37:28.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff56e5b2640 1 -- 172.21.15.37:0/2914384070 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff5680a0260 msgr2=0x7ff5680a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:28.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff56e5b2640 1 --2- 172.21.15.37:0/2914384070 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff5680a0260 0x7ff5680a0640 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7ff56809f690 tx=0x7ff55801c1c0 comp rx=0 tx=0).stop 2024-04-26T00:37:28.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff56e5b2640 1 -- 172.21.15.37:0/2914384070 shutdown_connections 2024-04-26T00:37:28.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff56e5b2640 1 --2- 172.21.15.37:0/2914384070 >> v2:172.21.15.45:3300/0 conn(0x7ff568134b30 0x7ff568134f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:28.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff56e5b2640 1 --2- 172.21.15.37:0/2914384070 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff5680a0260 0x7ff5680a0640 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:28.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff56e5b2640 1 -- 172.21.15.37:0/2914384070 >> 172.21.15.37:0/2914384070 conn(0x7ff5680a1ad0 msgr2=0x7ff5680a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:37:28.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff56e5b2640 1 -- 172.21.15.37:0/2914384070 shutdown_connections 2024-04-26T00:37:28.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff56e5b2640 1 -- 172.21.15.37:0/2914384070 wait complete. 2024-04-26T00:37:28.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff56e5b2640 10 monclient: build_initial_monmap 2024-04-26T00:37:28.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff56e5b2640 10 monclient: monmap: 2024-04-26T00:37:28.670 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:37:28.670 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:37:28.670 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:37:28.670 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:37:28.671 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:37:28.671 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:37:28.671 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:37:28.671 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:37:28.671 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:37:28.671 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:28.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff56e5b2640 1 Processor -- start 2024-04-26T00:37:28.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff56e5b2640 1 -- start start 2024-04-26T00:37:28.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff56e5b2640 10 monclient: init 2024-04-26T00:37:28.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff56e5b2640 10 monclient: _reopen_session rank -1 2024-04-26T00:37:28.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff56e5b2640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:37:28.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff56e5b2640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff5680a0260 0x7ff5681315d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:28.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff56e5b2640 10 monclient(hunting): picked mon.noname-c con 0x7ff5680a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:37:28.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff56e5b2640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff568134b30 0x7ff568139550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:28.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff56e5b2640 10 monclient(hunting): picked mon.noname-a con 0x7ff568134b30 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:37:28.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff56e5b2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff568135490 0x7ff568139a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:28.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff56e5b2640 10 monclient(hunting): picked mon.noname-b con 0x7ff568135490 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:37:28.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff56e5b2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:28.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff56e5b2640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff56809f430 con 0x7ff568134b30 2024-04-26T00:37:28.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff56e5b2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:28.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff56e5b2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff56809f2b0 con 0x7ff568135490 2024-04-26T00:37:28.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff56e5b2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:28.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff56e5b2640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff56809fe90 con 0x7ff5680a0260 2024-04-26T00:37:28.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff56e5b2640 10 monclient(hunting): _renew_subs 2024-04-26T00:37:28.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff56e5b2640 10 monclient(hunting): authenticate will time out at 2029.974076s 2024-04-26T00:37:28.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff55ffff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff568134b30 0x7ff568139550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:28.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff567fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff5680a0260 0x7ff5681315d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:28.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff55ffff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff568134b30 0x7ff568139550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:45516/0 (socket says 172.21.15.37:45516) 2024-04-26T00:37:28.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff55ffff640 1 -- 172.21.15.37:0/4033709671 learned_addr learned my addr 172.21.15.37:0/4033709671 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:37:28.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff55ffff640 10 monclient(hunting): get_auth_request con 0x7ff568134b30 auth_method 0 2024-04-26T00:37:28.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff55ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:28.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff55ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:28.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff55ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:28.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff567fff640 10 monclient(hunting): get_auth_request con 0x7ff5680a0260 auth_method 0 2024-04-26T00:37:28.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff567fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:28.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff567fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:28.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff567fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:28.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff55ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:28.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff55ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:28.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff55ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:28.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff567fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:28.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff567fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:28.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff567fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:28.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff56cb4f640 1 --2- 172.21.15.37:0/4033709671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff568135490 0x7ff568139a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:28.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff567fff640 10 monclient(hunting): handle_auth_done global_id 204206 payload 824 2024-04-26T00:37:28.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff567fff640 1 -- 172.21.15.37:0/4033709671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff568135490 msgr2=0x7ff568139a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:28.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff567fff640 1 --2- 172.21.15.37:0/4033709671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff568135490 0x7ff568139a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:28.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff567fff640 1 -- 172.21.15.37:0/4033709671 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff568134b30 msgr2=0x7ff568139550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:28.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff567fff640 1 --2- 172.21.15.37:0/4033709671 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff568134b30 0x7ff568139550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:28.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff567fff640 10 monclient: _finish_hunting 0 2024-04-26T00:37:28.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff567fff640 1 monclient: found mon.noname-c 2024-04-26T00:37:28.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff567fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:37:28.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff567fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:37:28.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff567fff640 1 -- 172.21.15.37:0/4033709671 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff56813b1f0 con 0x7ff5680a0260 2024-04-26T00:37:28.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff56cb4f640 10 monclient: get_auth_request con 0x7ff568135490 auth_method 0 2024-04-26T00:37:28.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff56cb4f640 1 --2- 172.21.15.37:0/4033709671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff568135490 0x7ff568139a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:37:28.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.664+0000 7ff55ffff640 1 --2- 172.21.15.37:0/4033709671 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff568134b30 0x7ff568139550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:37:28.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.668+0000 7ff567fff640 1 -- 172.21.15.37:0/4033709671 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff5680a0260 msgr2=0x7ff5681315d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:28.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.668+0000 7ff567fff640 1 --2- 172.21.15.37:0/4033709671 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff5680a0260 0x7ff5681315d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff56809c120 tx=0x7ff558018f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:28.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.668+0000 7ff5657fa640 1 -- 172.21.15.37:0/4033709671 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5580b9070 con 0x7ff5680a0260 2024-04-26T00:37:28.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.668+0000 7ff5657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:28.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.668+0000 7ff5657fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:37:28.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.668+0000 7ff5657fa640 10 monclient: dump: 2024-04-26T00:37:28.679 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:28.679 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:28.681 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:28.681 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:28.681 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:28.681 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:28.681 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:28.681 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:28.681 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:28.681 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:28.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.668+0000 7ff5657fa640 10 monclient: _finish_auth 0 2024-04-26T00:37:28.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.668+0000 7ff5657fa640 10 monclient: _check_auth_tickets 2024-04-26T00:37:28.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.668+0000 7ff5657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:37:28.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.668+0000 7ff5657fa640 1 -- 172.21.15.37:0/4033709671 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5580b4420 con 0x7ff5680a0260 2024-04-26T00:37:28.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.668+0000 7ff5657fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:37:28.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.668+0000 7ff5657fa640 1 -- 172.21.15.37:0/4033709671 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5580b0040 con 0x7ff5680a0260 2024-04-26T00:37:28.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.668+0000 7ff56e5b2640 5 monclient: authenticate success, global_id 204206 2024-04-26T00:37:28.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.668+0000 7ff56e5b2640 10 monclient: _renew_subs 2024-04-26T00:37:28.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.668+0000 7ff56e5b2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:37:28.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.668+0000 7ff56e5b2640 1 -- 172.21.15.37:0/4033709671 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff56811bf60 con 0x7ff5680a0260 2024-04-26T00:37:28.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.668+0000 7ff5667fc640 4 set_mon_vals no callback set 2024-04-26T00:37:28.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.668+0000 7ff5657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:28.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.668+0000 7ff5657fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:37:28.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.668+0000 7ff5657fa640 10 monclient: dump: 2024-04-26T00:37:28.683 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:28.683 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:28.683 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:28.683 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:28.683 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:28.683 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:28.683 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:28.683 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:28.683 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:28.683 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:28.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.668+0000 7ff56e5b2640 10 monclient: _renew_subs 2024-04-26T00:37:28.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.668+0000 7ff56e5b2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:37:28.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.668+0000 7ff56e5b2640 1 -- 172.21.15.37:0/4033709671 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff568132df0 con 0x7ff5680a0260 2024-04-26T00:37:28.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.668+0000 7ff56e5b2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:28.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.668+0000 7ff56e5b2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:28.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.668+0000 7ff5657fa640 1 -- 172.21.15.37:0/4033709671 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff5580bc070 con 0x7ff5680a0260 2024-04-26T00:37:28.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.668+0000 7ff56e5b2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:37:28.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.668+0000 7ff56e5b2640 1 -- 172.21.15.37:0/4033709671 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff568066f40 con 0x7ff5680a0260 2024-04-26T00:37:28.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.668+0000 7ff5657fa640 1 --2- 172.21.15.37:0/4033709671 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff548035ed0 0x7ff548038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:28.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.668+0000 7ff5657fa640 1 -- 172.21.15.37:0/4033709671 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(689..689 src has 370..689) ==== 3927+0+0 (secure 0 0 0) 0x7ff5580e1ae0 con 0x7ff5680a0260 2024-04-26T00:37:28.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.668+0000 7ff55ffff640 1 --2- 172.21.15.37:0/4033709671 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff548035ed0 0x7ff548038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:28.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.668+0000 7ff55ffff640 10 monclient: get_auth_request con 0x7ff548035ed0 auth_method 0 2024-04-26T00:37:28.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.668+0000 7ff55ffff640 1 -- 172.21.15.37:0/4033709671 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff548035ed0 msgr2=0x7ff548038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:37:28.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.668+0000 7ff55ffff640 1 -- 172.21.15.37:0/4033709671 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff548035ed0 msgr2=0x7ff548038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:37:28.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.668+0000 7ff55ffff640 1 --2- 172.21.15.37:0/4033709671 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff548035ed0 0x7ff548038390 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7ff56813cb80 tx=0x7ff550008040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:37:28.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.668+0000 7ff55ffff640 1 --2- 172.21.15.37:0/4033709671 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff548035ed0 0x7ff548038390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:37:28.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.672+0000 7ff5657fa640 1 -- 172.21.15.37:0/4033709671 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff5680a1130 con 0x7ff5680a0260 2024-04-26T00:37:28.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.672+0000 7ff5657fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:28.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.672+0000 7ff5657fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:37:28.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.872+0000 7ff55ffff640 1 --2- 172.21.15.37:0/4033709671 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff548035ed0 0x7ff548038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:28.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.872+0000 7ff55ffff640 10 monclient: get_auth_request con 0x7ff548035ed0 auth_method 0 2024-04-26T00:37:28.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.872+0000 7ff55ffff640 1 --2- 172.21.15.37:0/4033709671 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff548035ed0 0x7ff548038390 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7ff56813cb80 tx=0x7ff550015000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:28.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.900+0000 7ff56e5b2640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-26T00:37:28.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.900+0000 7ff56e5b2640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-26T00:37:28.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.900+0000 7ff56e5b2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:37:28.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.900+0000 7ff56e5b2640 1 -- 172.21.15.37:0/4033709671 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7ff56813bad0 con 0x7ff5680a0260 2024-04-26T00:37:28.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.900+0000 7ff5657fa640 1 -- 172.21.15.37:0/4033709671 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7ff5580bc360 con 0x7ff5680a0260 2024-04-26T00:37:28.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.904+0000 7ff5657fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-26T00:37:28.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.904+0000 7ff5657fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:37:28.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.904+0000 7ff56e5b2640 1 -- 172.21.15.37:0/4033709671 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff548035ed0 msgr2=0x7ff548038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:28.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.904+0000 7ff56e5b2640 1 --2- 172.21.15.37:0/4033709671 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff548035ed0 0x7ff548038390 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7ff56813cb80 tx=0x7ff550015000 comp rx=0 tx=0).stop 2024-04-26T00:37:28.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.904+0000 7ff56e5b2640 10 monclient: shutdown 2024-04-26T00:37:28.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.904+0000 7ff56e5b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:37:28.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.904+0000 7ff56e5b2640 1 -- 172.21.15.37:0/4033709671 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff5680a0260 msgr2=0x7ff5681315d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:28.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.904+0000 7ff56e5b2640 1 --2- 172.21.15.37:0/4033709671 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff5680a0260 0x7ff5681315d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff56809c120 tx=0x7ff558018f60 comp rx=0 tx=0).stop 2024-04-26T00:37:28.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.904+0000 7ff56e5b2640 1 -- 172.21.15.37:0/4033709671 shutdown_connections 2024-04-26T00:37:28.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.904+0000 7ff56e5b2640 1 --2- 172.21.15.37:0/4033709671 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff548035ed0 0x7ff548038390 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:28.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.904+0000 7ff56e5b2640 1 --2- 172.21.15.37:0/4033709671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff568135490 0x7ff568139a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:28.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.904+0000 7ff56e5b2640 1 --2- 172.21.15.37:0/4033709671 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff568134b30 0x7ff568139550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:28.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.904+0000 7ff56e5b2640 1 --2- 172.21.15.37:0/4033709671 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff5680a0260 0x7ff5681315d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:28.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.904+0000 7ff56e5b2640 1 -- 172.21.15.37:0/4033709671 >> 172.21.15.37:0/4033709671 conn(0x7ff5680a1ad0 msgr2=0x7ff568133000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:37:28.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.904+0000 7ff56e5b2640 1 -- 172.21.15.37:0/4033709671 shutdown_connections 2024-04-26T00:37:28.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:28.904+0000 7ff56e5b2640 1 -- 172.21.15.37:0/4033709671 wait complete. 2024-04-26T00:37:28.922 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-26T00:37:28.992 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:28.992+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:28.992 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:28.992+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=11,kv=9,logm=54} crc {auth=3188830012,config=692619927,health=12383961,kv=377031510,logm=915112405}) 2024-04-26T00:37:28.992 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:28.992+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=23,config=2,health=12,kv=10,logm=53} crc {auth=338647095,config=692619927,health=3816966473,kv=3485254834,logm=3474279329}) 2024-04-26T00:37:28.993 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:28.992+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:28.993 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:28.992+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=11,kv=9,logm=54} crc {auth=3188830012,config=692619927,health=12383961,kv=377031510,logm=915112405}) 2024-04-26T00:37:28.993 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:28.992+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=25,config=2,health=12,kv=10,logm=51} crc {auth=2741810369,config=692619927,health=3816966473,kv=1034996044,logm=3156005300}) 2024-04-26T00:37:29.008 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.008+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.009 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.008+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2655077791}) 2024-04-26T00:37:29.009 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.008+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1734225572}) 2024-04-26T00:37:29.009 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.008+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.009 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.008+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2655077791}) 2024-04-26T00:37:29.009 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.008+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1162882161}) 2024-04-26T00:37:29.017 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.016+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.017 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.016+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3559261955}) 2024-04-26T00:37:29.017 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.016+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2936464163}) 2024-04-26T00:37:29.017 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.016+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.017 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.016+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3559261955}) 2024-04-26T00:37:29.017 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.016+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2247371056}) 2024-04-26T00:37:29.350 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.352+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.351 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.352+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2605084342}) 2024-04-26T00:37:29.351 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.352+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3825051773}) 2024-04-26T00:37:29.351 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.352+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.351 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.352+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2605084342}) 2024-04-26T00:37:29.351 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.352+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3828565797}) 2024-04-26T00:37:29.355 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.352+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.355 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.352+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2451448204}) 2024-04-26T00:37:29.355 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.352+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1552081183}) 2024-04-26T00:37:29.355 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.352+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.355 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.352+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2451448204}) 2024-04-26T00:37:29.355 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.356+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=521315215}) 2024-04-26T00:37:29.358 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.356+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.358 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.356+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1783664632}) 2024-04-26T00:37:29.359 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.356+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2768304285}) 2024-04-26T00:37:29.359 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.356+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.359 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.356+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1783664632}) 2024-04-26T00:37:29.359 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.356+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=498686709}) 2024-04-26T00:37:29.362 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.360+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.362 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.360+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4044992514}) 2024-04-26T00:37:29.362 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.360+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4225768177}) 2024-04-26T00:37:29.362 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.360+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.362 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.360+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4044992514}) 2024-04-26T00:37:29.362 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.360+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2060002224}) 2024-04-26T00:37:29.365 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.364+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.365 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.364+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3594884832}) 2024-04-26T00:37:29.365 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.364+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=521827044}) 2024-04-26T00:37:29.365 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.364+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.365 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.364+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3594884832}) 2024-04-26T00:37:29.366 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.364+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=639313354}) 2024-04-26T00:37:29.369 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.368+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.369 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.368+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4055120921}) 2024-04-26T00:37:29.369 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.368+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3672589390}) 2024-04-26T00:37:29.369 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.368+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.369 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.368+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4055120921}) 2024-04-26T00:37:29.369 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.368+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1229341287}) 2024-04-26T00:37:29.372 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.372+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.372 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.372+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1553090087}) 2024-04-26T00:37:29.372 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.372+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=866922810}) 2024-04-26T00:37:29.373 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.372+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.373 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.372+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1553090087}) 2024-04-26T00:37:29.373 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.372+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3245777728}) 2024-04-26T00:37:29.375 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.376+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.376 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.376+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1327412132}) 2024-04-26T00:37:29.376 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.376+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=745152865}) 2024-04-26T00:37:29.376 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.376+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.376 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.376+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1327412132}) 2024-04-26T00:37:29.376 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.376+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4077999950}) 2024-04-26T00:37:29.378 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.376+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.378 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.376+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3549711264}) 2024-04-26T00:37:29.378 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.376+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2423504583}) 2024-04-26T00:37:29.378 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.376+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.378 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.376+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3549711264}) 2024-04-26T00:37:29.378 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.376+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2183860874}) 2024-04-26T00:37:29.380 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.380+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.380 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.380+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2837220399}) 2024-04-26T00:37:29.380 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.380+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4086742912}) 2024-04-26T00:37:29.380 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.380+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.380 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.380+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2837220399}) 2024-04-26T00:37:29.381 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.380+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2710905080}) 2024-04-26T00:37:29.382 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.380+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.383 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.380+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2841776360}) 2024-04-26T00:37:29.383 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.380+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2254056073}) 2024-04-26T00:37:29.383 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.380+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.383 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.380+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2841776360}) 2024-04-26T00:37:29.383 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.380+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2241285967}) 2024-04-26T00:37:29.385 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.384+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.385 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.384+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=814749595}) 2024-04-26T00:37:29.385 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.384+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=275793277}) 2024-04-26T00:37:29.385 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.384+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.385 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.384+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=814749595}) 2024-04-26T00:37:29.385 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.384+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2294784131}) 2024-04-26T00:37:29.387 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.388+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.387 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.388+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3647349606}) 2024-04-26T00:37:29.387 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.388+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1517778481}) 2024-04-26T00:37:29.387 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.388+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.387 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.388+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3647349606}) 2024-04-26T00:37:29.388 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.388+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1543846321}) 2024-04-26T00:37:29.389 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.388+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.389 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.388+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2763294232}) 2024-04-26T00:37:29.390 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.388+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2525803828}) 2024-04-26T00:37:29.390 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.388+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.390 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.388+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2763294232}) 2024-04-26T00:37:29.390 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.388+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=539702487}) 2024-04-26T00:37:29.392 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.392+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.392 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.392+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4266601997}) 2024-04-26T00:37:29.392 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.392+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2130634345}) 2024-04-26T00:37:29.392 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.392+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.392 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.392+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4266601997}) 2024-04-26T00:37:29.392 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.392+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2404159004}) 2024-04-26T00:37:29.395 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.392+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.395 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.392+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3569422085}) 2024-04-26T00:37:29.395 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.392+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=497665863}) 2024-04-26T00:37:29.395 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.392+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.395 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.392+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3569422085}) 2024-04-26T00:37:29.395 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.392+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1668850991}) 2024-04-26T00:37:29.397 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.396+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.397 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.396+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4024422880}) 2024-04-26T00:37:29.397 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.396+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2608693178}) 2024-04-26T00:37:29.397 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.396+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.397 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.396+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4024422880}) 2024-04-26T00:37:29.397 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.396+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2628441270}) 2024-04-26T00:37:29.400 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.400+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.400 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.400+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2804271226}) 2024-04-26T00:37:29.400 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.400+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3871996785}) 2024-04-26T00:37:29.400 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.400+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.400 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.400+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2804271226}) 2024-04-26T00:37:29.400 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.400+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2854240713}) 2024-04-26T00:37:29.402 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.400+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.402 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.400+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3036870423}) 2024-04-26T00:37:29.402 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.400+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1569244432}) 2024-04-26T00:37:29.402 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.400+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.402 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.400+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3036870423}) 2024-04-26T00:37:29.402 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.400+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=467517844}) 2024-04-26T00:37:29.404 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.404+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.404 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.404+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=741709892}) 2024-04-26T00:37:29.405 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.404+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3109338895}) 2024-04-26T00:37:29.405 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.404+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.405 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.404+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=741709892}) 2024-04-26T00:37:29.405 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.404+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3344945713}) 2024-04-26T00:37:29.407 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.404+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.407 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.404+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3563322026}) 2024-04-26T00:37:29.407 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.404+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1405697983}) 2024-04-26T00:37:29.407 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.404+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.407 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.404+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3563322026}) 2024-04-26T00:37:29.407 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.404+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4290915050}) 2024-04-26T00:37:29.409 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.408+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.409 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.408+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=43482520}) 2024-04-26T00:37:29.410 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.408+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1496309791}) 2024-04-26T00:37:29.410 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.408+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.410 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.408+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=43482520}) 2024-04-26T00:37:29.410 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.408+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1305604424}) 2024-04-26T00:37:29.411 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.412+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.412 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.412+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=610431861}) 2024-04-26T00:37:29.412 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.412+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2132870881}) 2024-04-26T00:37:29.412 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.412+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.412 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.412+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=610431861}) 2024-04-26T00:37:29.412 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.412+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1780533592}) 2024-04-26T00:37:29.414 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.412+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.414 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.412+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2410631587}) 2024-04-26T00:37:29.414 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.412+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2407562399}) 2024-04-26T00:37:29.414 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.412+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.414 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.412+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2410631587}) 2024-04-26T00:37:29.414 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.412+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3264065131}) 2024-04-26T00:37:29.417 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.416+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.417 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.416+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1191996136}) 2024-04-26T00:37:29.417 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.416+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=215185010}) 2024-04-26T00:37:29.417 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.416+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.417 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.416+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1191996136}) 2024-04-26T00:37:29.417 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.416+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4187566339}) 2024-04-26T00:37:29.419 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.416+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.419 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.416+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=34931044}) 2024-04-26T00:37:29.419 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.416+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4260972138}) 2024-04-26T00:37:29.419 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.416+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.419 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.416+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=34931044}) 2024-04-26T00:37:29.419 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.416+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=71572958}) 2024-04-26T00:37:29.421 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.420+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.421 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.420+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1220138891}) 2024-04-26T00:37:29.421 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.420+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=570021738}) 2024-04-26T00:37:29.421 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.420+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.421 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.420+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1220138891}) 2024-04-26T00:37:29.421 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.420+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2231187498}) 2024-04-26T00:37:29.423 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.424+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.423 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.424+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3391535776}) 2024-04-26T00:37:29.423 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.424+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2932981063}) 2024-04-26T00:37:29.423 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.424+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.423 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.424+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3391535776}) 2024-04-26T00:37:29.424 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.424+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3612657425}) 2024-04-26T00:37:29.425 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.424+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.425 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.424+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3900684574}) 2024-04-26T00:37:29.425 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.424+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2345498463}) 2024-04-26T00:37:29.426 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.424+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.426 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.424+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3900684574}) 2024-04-26T00:37:29.426 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.424+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2033683332}) 2024-04-26T00:37:29.428 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.428+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.428 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.428+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4226235555}) 2024-04-26T00:37:29.428 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.428+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3736165144}) 2024-04-26T00:37:29.428 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.428+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.428 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.428+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4226235555}) 2024-04-26T00:37:29.428 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.428+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2789487683}) 2024-04-26T00:37:29.430 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.428+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.430 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.428+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3045048004}) 2024-04-26T00:37:29.430 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.428+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2227657718}) 2024-04-26T00:37:29.430 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.428+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.430 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.428+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3045048004}) 2024-04-26T00:37:29.430 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.428+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1096095790}) 2024-04-26T00:37:29.432 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.432+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.433 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.432+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2391988957}) 2024-04-26T00:37:29.433 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.432+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3026590543}) 2024-04-26T00:37:29.433 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.432+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.433 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.432+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2391988957}) 2024-04-26T00:37:29.433 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.432+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1122404865}) 2024-04-26T00:37:29.435 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.432+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.435 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.432+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3912351355}) 2024-04-26T00:37:29.435 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.436+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=703437303}) 2024-04-26T00:37:29.435 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.436+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.435 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.436+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3912351355}) 2024-04-26T00:37:29.435 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.436+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3648470862}) 2024-04-26T00:37:29.437 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.436+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.437 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.436+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3229433471}) 2024-04-26T00:37:29.437 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.436+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3893374871}) 2024-04-26T00:37:29.437 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.436+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.437 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.436+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3229433471}) 2024-04-26T00:37:29.438 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.436+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4232722426}) 2024-04-26T00:37:29.439 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.440+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.440 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.440+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2832578024}) 2024-04-26T00:37:29.440 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.440+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1800783872}) 2024-04-26T00:37:29.440 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.440+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.440 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.440+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2832578024}) 2024-04-26T00:37:29.440 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.440+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=600390554}) 2024-04-26T00:37:29.442 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.440+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.442 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.440+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3621251789}) 2024-04-26T00:37:29.442 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.440+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1310791112}) 2024-04-26T00:37:29.442 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.440+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.442 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.440+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3621251789}) 2024-04-26T00:37:29.442 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.440+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3114455040}) 2024-04-26T00:37:29.444 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.444+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.444 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.444+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4227467934}) 2024-04-26T00:37:29.444 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.444+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3115534840}) 2024-04-26T00:37:29.444 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.444+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.445 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.444+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4227467934}) 2024-04-26T00:37:29.445 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.444+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1018271815}) 2024-04-26T00:37:29.447 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.448+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.447 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.448+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=766310028}) 2024-04-26T00:37:29.447 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.448+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=838025481}) 2024-04-26T00:37:29.447 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.448+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.447 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.448+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=766310028}) 2024-04-26T00:37:29.447 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.448+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3893309703}) 2024-04-26T00:37:29.449 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.448+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.450 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.448+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4019545058}) 2024-04-26T00:37:29.450 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.448+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1847709924}) 2024-04-26T00:37:29.450 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.448+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.450 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.448+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4019545058}) 2024-04-26T00:37:29.450 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.448+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2107859561}) 2024-04-26T00:37:29.452 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.452+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.452 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.452+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=708450946}) 2024-04-26T00:37:29.452 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.452+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3685199168}) 2024-04-26T00:37:29.452 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.452+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.452 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.452+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=708450946}) 2024-04-26T00:37:29.452 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.452+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2602114848}) 2024-04-26T00:37:29.454 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.452+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.454 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.452+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3256316928}) 2024-04-26T00:37:29.454 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.452+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3933989876}) 2024-04-26T00:37:29.454 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.452+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.454 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.452+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3256316928}) 2024-04-26T00:37:29.454 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.452+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=379024310}) 2024-04-26T00:37:29.456 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.456+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.456 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.456+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3764969009}) 2024-04-26T00:37:29.456 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.456+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2544083761}) 2024-04-26T00:37:29.456 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.456+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.457 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.456+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3764969009}) 2024-04-26T00:37:29.457 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.456+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=275647147}) 2024-04-26T00:37:29.458 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.456+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.459 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.456+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3983735677}) 2024-04-26T00:37:29.459 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.456+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3568594229}) 2024-04-26T00:37:29.459 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.456+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.459 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.456+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3983735677}) 2024-04-26T00:37:29.459 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.456+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2476184753}) 2024-04-26T00:37:29.461 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.460+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.461 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.460+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1699183631}) 2024-04-26T00:37:29.461 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.460+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3237106363}) 2024-04-26T00:37:29.461 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.460+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.461 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.460+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1699183631}) 2024-04-26T00:37:29.461 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.460+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2680957891}) 2024-04-26T00:37:29.463 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.460+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.463 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.460+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2884682612}) 2024-04-26T00:37:29.463 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.460+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2172557463}) 2024-04-26T00:37:29.463 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.464+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.463 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.464+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2884682612}) 2024-04-26T00:37:29.463 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.464+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1118943415}) 2024-04-26T00:37:29.465 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.464+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.465 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.464+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=792969254}) 2024-04-26T00:37:29.465 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.464+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2489694224}) 2024-04-26T00:37:29.465 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.464+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.466 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.464+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=792969254}) 2024-04-26T00:37:29.466 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.464+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3215607421}) 2024-04-26T00:37:29.468 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.468+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.468 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.468+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3869098501}) 2024-04-26T00:37:29.468 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.468+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1549161535}) 2024-04-26T00:37:29.468 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.468+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.468 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.468+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3869098501}) 2024-04-26T00:37:29.468 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.468+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=38715670}) 2024-04-26T00:37:29.470 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.468+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.470 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.468+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1561100683}) 2024-04-26T00:37:29.470 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.468+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3558442372}) 2024-04-26T00:37:29.471 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.468+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.471 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.468+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1561100683}) 2024-04-26T00:37:29.471 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.468+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1673433583}) 2024-04-26T00:37:29.473 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.472+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.473 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.472+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4042812048}) 2024-04-26T00:37:29.473 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.472+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1169026634}) 2024-04-26T00:37:29.473 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.472+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.473 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.472+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4042812048}) 2024-04-26T00:37:29.473 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.472+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=807031958}) 2024-04-26T00:37:29.475 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.476+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.475 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.476+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=111087392}) 2024-04-26T00:37:29.475 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.476+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=946500496}) 2024-04-26T00:37:29.475 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.476+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.476 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.476+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=111087392}) 2024-04-26T00:37:29.476 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.476+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2981059771}) 2024-04-26T00:37:29.478 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.476+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.478 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.476+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2969486059}) 2024-04-26T00:37:29.478 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.476+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4037717057}) 2024-04-26T00:37:29.478 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.476+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.478 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.476+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2969486059}) 2024-04-26T00:37:29.478 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.476+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=784261673}) 2024-04-26T00:37:29.480 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.480+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.480 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.480+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=934790289}) 2024-04-26T00:37:29.481 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.480+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2725524860}) 2024-04-26T00:37:29.481 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.480+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.481 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.480+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=934790289}) 2024-04-26T00:37:29.481 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.480+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2373608461}) 2024-04-26T00:37:29.483 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.480+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.483 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.480+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3425977719}) 2024-04-26T00:37:29.483 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.484+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2992175813}) 2024-04-26T00:37:29.483 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.484+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.483 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.484+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3425977719}) 2024-04-26T00:37:29.483 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.484+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4070948628}) 2024-04-26T00:37:29.486 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.484+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.486 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.484+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3578067221}) 2024-04-26T00:37:29.486 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.484+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3462763166}) 2024-04-26T00:37:29.486 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.484+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.486 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.484+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3578067221}) 2024-04-26T00:37:29.486 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.484+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1773916751}) 2024-04-26T00:37:29.488 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.488+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.488 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.488+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4241657444}) 2024-04-26T00:37:29.488 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.488+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1616753789}) 2024-04-26T00:37:29.488 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.488+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.489 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.488+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4241657444}) 2024-04-26T00:37:29.489 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.488+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3421606462}) 2024-04-26T00:37:29.491 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.488+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.491 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.488+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2421317572}) 2024-04-26T00:37:29.491 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.488+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2542676836}) 2024-04-26T00:37:29.491 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.488+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.491 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.488+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2421317572}) 2024-04-26T00:37:29.491 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.488+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=120625719}) 2024-04-26T00:37:29.493 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.492+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.493 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.492+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1697341672}) 2024-04-26T00:37:29.493 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.492+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2933700540}) 2024-04-26T00:37:29.493 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.492+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.493 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.492+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1697341672}) 2024-04-26T00:37:29.493 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.492+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1622367071}) 2024-04-26T00:37:29.495 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.496+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.496 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.496+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3680396410}) 2024-04-26T00:37:29.496 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.496+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=227641626}) 2024-04-26T00:37:29.496 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.496+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.496 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.496+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3680396410}) 2024-04-26T00:37:29.496 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.496+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=855034765}) 2024-04-26T00:37:29.498 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.496+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.498 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.496+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3168886907}) 2024-04-26T00:37:29.498 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.496+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4052775109}) 2024-04-26T00:37:29.498 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.496+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.498 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.496+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3168886907}) 2024-04-26T00:37:29.498 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.496+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4045337789}) 2024-04-26T00:37:29.500 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.500+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.500 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.500+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3944492146}) 2024-04-26T00:37:29.500 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.500+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1792608121}) 2024-04-26T00:37:29.500 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.500+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.500 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.500+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3944492146}) 2024-04-26T00:37:29.501 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.500+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1139008379}) 2024-04-26T00:37:29.502 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.500+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.502 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.500+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4014693227}) 2024-04-26T00:37:29.502 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.500+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1670426786}) 2024-04-26T00:37:29.502 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.500+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.502 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.500+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4014693227}) 2024-04-26T00:37:29.505 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.500+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3084378800}) 2024-04-26T00:37:29.573 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.572+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.573 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.572+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4104480540}) 2024-04-26T00:37:29.573 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.572+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1505492439}) 2024-04-26T00:37:29.573 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.572+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.574 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.572+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4104480540}) 2024-04-26T00:37:29.574 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.572+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2879417471}) 2024-04-26T00:37:29.581 INFO:tasks.rados.rados.0.smithi045.stdout:3086: done (4 left) 2024-04-26T00:37:29.581 INFO:tasks.rados.rados.0.smithi045.stdout:3087: done (3 left) 2024-04-26T00:37:29.581 INFO:tasks.rados.rados.0.smithi045.stdout:3088: done (2 left) 2024-04-26T00:37:29.581 INFO:tasks.rados.rados.0.smithi045.stdout:3089: done (1 left) 2024-04-26T00:37:29.581 INFO:tasks.rados.rados.0.smithi045.stdout:3090: done (0 left) 2024-04-26T00:37:29.581 INFO:tasks.rados.rados.0.smithi045.stdout:3091: copy_from oid 26 from oid 37 current snap is 332 2024-04-26T00:37:29.581 INFO:tasks.rados.rados.0.smithi045.stdout:3092: copy_from oid 10 from oid 48 current snap is 332 2024-04-26T00:37:29.581 INFO:tasks.rados.rados.0.smithi045.stdout:3093: copy_from oid 21 from oid 30 current snap is 332 2024-04-26T00:37:29.581 INFO:tasks.rados.rados.0.smithi045.stdout:3094: snap_remove snap 327 2024-04-26T00:37:29.582 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.580+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.582 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.580+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2085878909}) 2024-04-26T00:37:29.582 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.580+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3734968503}) 2024-04-26T00:37:29.582 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.580+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.583 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.580+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2085878909}) 2024-04-26T00:37:29.583 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.580+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=311402259}) 2024-04-26T00:37:29.923 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:37:29.923 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:37:29.936 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.936+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.936 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.936+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=37,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=21} crc {logm=1570319404,mdsmap=1401145748,mgr=319753808,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=3350041192,mon_config_key=4061063117,monmap=3876866194,osd_metadata=839604482,osd_pg_creating=3824027401,osd_snap=3438422244}) 2024-04-26T00:37:29.936 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.936+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=36,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23} crc {logm=530285274,mdsmap=2840952586,mgr=139108616,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=775137130,mon_config_key=4061063117,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=3824027401,osd_snap=3695372818}) 2024-04-26T00:37:29.936 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.936+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:29.936 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.936+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=37,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=21} crc {logm=1570319404,mdsmap=1401145748,mgr=319753808,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=3350041192,mon_config_key=4061063117,monmap=3876866194,osd_metadata=839604482,osd_pg_creating=3824027401,osd_snap=3438422244}) 2024-04-26T00:37:29.936 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:29.936+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=37,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=19} crc {logm=2466532577,mdsmap=2840952586,mgr=4237326694,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=3350041192,mon_config_key=4061063117,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=3824027401,osd_snap=1254753454}) 2024-04-26T00:37:30.021 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:30.020+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:30.021 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:30.020+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=100} crc {osd_snap=4217562004}) 2024-04-26T00:37:30.021 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:30.020+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=100} crc {osd_snap=4265394425}) 2024-04-26T00:37:30.021 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:30.020+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:30.021 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:30.020+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=100} crc {osd_snap=4217562004}) 2024-04-26T00:37:30.021 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:30.020+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=100} crc {osd_snap=3344108747}) 2024-04-26T00:37:30.049 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:30.048+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:30.050 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:30.048+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=4,osdmap=96} crc {osd_snap=408893634,osdmap=639903835}) 2024-04-26T00:37:30.050 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:30.048+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=5,osdmap=95} crc {osd_snap=439998358,osdmap=3981966979}) 2024-04-26T00:37:30.050 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:30.048+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:30.050 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:30.048+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=4,osdmap=96} crc {osd_snap=408893634,osdmap=639903835}) 2024-04-26T00:37:30.050 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:30.048+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=4,osdmap=96} crc {osd_snap=408893634,osdmap=2335196307}) 2024-04-26T00:37:30.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.056+0000 7ff83d2d8640 10 monclient: get_monmap_and_config 2024-04-26T00:37:30.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.056+0000 7ff83d2d8640 10 monclient: build_initial_monmap 2024-04-26T00:37:30.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.056+0000 7ff83d2d8640 10 monclient: monmap: 2024-04-26T00:37:30.057 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:37:30.057 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:30.058 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:37:30.062218+0000 2024-04-26T00:37:30.058 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:37:30.062218+0000 2024-04-26T00:37:30.058 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:37:30.058 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:30.058 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:37:30.058 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:37:30.058 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:37:30.058 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:37:30.058 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:37:30.058 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:30.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.056+0000 7ff83d2d8640 1 Processor -- start 2024-04-26T00:37:30.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.056+0000 7ff83d2d8640 1 -- start start 2024-04-26T00:37:30.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.056+0000 7ff83d2d8640 10 monclient: init 2024-04-26T00:37:30.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.056+0000 7ff83d2d8640 10 monclient: _reopen_session rank -1 2024-04-26T00:37:30.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.056+0000 7ff83d2d8640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-04-26T00:37:30.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.056+0000 7ff83d2d8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff83809b460 0x7ff83809b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:30.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.056+0000 7ff83d2d8640 10 monclient(hunting): picked mon.noname-b con 0x7ff83809b460 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:37:30.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.056+0000 7ff83d2d8640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff83809cee0 0x7ff838133b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:30.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.056+0000 7ff83d2d8640 10 monclient(hunting): picked mon.noname-c con 0x7ff83809cee0 addr v2:172.21.15.45:3300/0 2024-04-26T00:37:30.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.056+0000 7ff83d2d8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff838134040 addr v1:172.21.15.37:6789/0 2024-04-26T00:37:30.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.056+0000 7ff83d2d8640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff83809bd80 con 0x7ff838134040 2024-04-26T00:37:30.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.056+0000 7ff83d2d8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:30.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.056+0000 7ff83d2d8640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7ff83809aec0 con 0x7ff83809cee0 2024-04-26T00:37:30.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.056+0000 7ff83d2d8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:30.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.056+0000 7ff83d2d8640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff83809c7e0 con 0x7ff83809b460 2024-04-26T00:37:30.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.056+0000 7ff83d2d8640 10 monclient(hunting): _renew_subs 2024-04-26T00:37:30.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.056+0000 7ff83d2d8640 10 monclient(hunting): authenticate will time out at 2031.367804s 2024-04-26T00:37:30.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.056+0000 7ff836ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff83809b460 0x7ff83809b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:30.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.056+0000 7ff836ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff83809b460 0x7ff83809b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:59054/0 (socket says 172.21.15.37:59054) 2024-04-26T00:37:30.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.056+0000 7ff836ffd640 1 -- 172.21.15.37:0/3123023900 learned_addr learned my addr 172.21.15.37:0/3123023900 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:37:30.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.056+0000 7ff836ffd640 10 monclient(hunting): get_auth_request con 0x7ff83809b460 auth_method 0 2024-04-26T00:37:30.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.056+0000 7ff836ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:30.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.056+0000 7ff836ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:30.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.056+0000 7ff836ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:30.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.060+0000 7ff836ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:30.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.060+0000 7ff836ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:30.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.060+0000 7ff836ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:30.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.060+0000 7ff8367fc640 1 --2- 172.21.15.37:0/3123023900 >> v2:172.21.15.45:3300/0 conn(0x7ff83809cee0 0x7ff838133b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:30.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.060+0000 7ff8367fc640 10 monclient(hunting): get_auth_request con 0x7ff83809cee0 auth_method 0 2024-04-26T00:37:30.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.060+0000 7ff8367fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:30.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.060+0000 7ff8367fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:30.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.060+0000 7ff8367fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:30.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.060+0000 7ff836ffd640 10 monclient(hunting): handle_auth_done global_id 204209 payload 274 2024-04-26T00:37:30.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.060+0000 7ff836ffd640 1 -- 172.21.15.37:0/3123023900 >> v2:172.21.15.45:3300/0 conn(0x7ff83809cee0 msgr2=0x7ff838133b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:37:30.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.060+0000 7ff836ffd640 1 --2- 172.21.15.37:0/3123023900 >> v2:172.21.15.45:3300/0 conn(0x7ff83809cee0 0x7ff838133b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:30.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.060+0000 7ff836ffd640 1 -- 172.21.15.37:0/3123023900 >> v1:172.21.15.37:6789/0 conn(0x7ff838134040 legacy=0x7ff83813ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:37:30.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.060+0000 7ff836ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:37:30.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.060+0000 7ff836ffd640 1 monclient: found mon.noname-b 2024-04-26T00:37:30.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.060+0000 7ff836ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:37:30.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.060+0000 7ff836ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:37:30.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.060+0000 7ff836ffd640 1 -- 172.21.15.37:0/3123023900 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff83809c500 con 0x7ff83809b460 2024-04-26T00:37:30.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff836ffd640 1 -- 172.21.15.37:0/3123023900 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff83809b460 msgr2=0x7ff83809b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:30.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff836ffd640 1 --2- 172.21.15.37:0/3123023900 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff83809b460 0x7ff83809b840 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff82800bb10 tx=0x7ff82801eb10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f079ea3be61e3226 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:30.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff835ffb640 1 -- 172.21.15.37:0/3123023900 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8280b4070 con 0x7ff83809b460 2024-04-26T00:37:30.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff835ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:30.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff835ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:37:30.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff835ffb640 10 monclient: dump: 2024-04-26T00:37:30.067 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:30.067 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:30.067 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.067 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.067 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:30.067 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:30.067 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:30.067 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:30.067 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:30.067 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:30.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff835ffb640 10 monclient: _finish_auth 0 2024-04-26T00:37:30.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff835ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:37:30.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff835ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:37:30.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff835ffb640 1 -- 172.21.15.37:0/3123023900 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8280a78a0 con 0x7ff83809b460 2024-04-26T00:37:30.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff835ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:37:30.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff83d2d8640 5 monclient: authenticate success, global_id 204209 2024-04-26T00:37:30.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff83d2d8640 10 monclient: get_monmap_and_config success 2024-04-26T00:37:30.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff83d2d8640 4 set_mon_vals no callback set 2024-04-26T00:37:30.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff835ffb640 1 -- 172.21.15.37:0/3123023900 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8280a7c70 con 0x7ff83809b460 2024-04-26T00:37:30.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff835ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:30.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff835ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:37:30.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff835ffb640 10 monclient: dump: 2024-04-26T00:37:30.068 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:30.068 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:30.068 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.068 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.068 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:30.069 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:30.069 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:30.069 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:30.069 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:30.069 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:30.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff83d2d8640 10 monclient: shutdown 2024-04-26T00:37:30.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff83d2d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:37:30.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff83d2d8640 1 -- 172.21.15.37:0/3123023900 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff83809b460 msgr2=0x7ff83809b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:30.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff83d2d8640 1 --2- 172.21.15.37:0/3123023900 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff83809b460 0x7ff83809b840 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff82800bb10 tx=0x7ff82801eb10 comp rx=0 tx=0).stop 2024-04-26T00:37:30.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff83d2d8640 1 -- 172.21.15.37:0/3123023900 shutdown_connections 2024-04-26T00:37:30.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff83d2d8640 1 --2- 172.21.15.37:0/3123023900 >> v2:172.21.15.45:3300/0 conn(0x7ff83809cee0 0x7ff838133b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:30.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff83d2d8640 1 --2- 172.21.15.37:0/3123023900 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff83809b460 0x7ff83809b840 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:30.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff83d2d8640 1 -- 172.21.15.37:0/3123023900 >> 172.21.15.37:0/3123023900 conn(0x7ff8380a2d60 msgr2=0x7ff8380a3140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:37:30.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff83d2d8640 1 -- 172.21.15.37:0/3123023900 shutdown_connections 2024-04-26T00:37:30.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff83d2d8640 1 -- 172.21.15.37:0/3123023900 wait complete. 2024-04-26T00:37:30.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff83d2d8640 10 monclient: build_initial_monmap 2024-04-26T00:37:30.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff83d2d8640 10 monclient: monmap: 2024-04-26T00:37:30.074 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:37:30.074 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:37:30.074 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:37:30.074 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:37:30.074 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:37:30.074 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:37:30.074 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:37:30.075 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:37:30.075 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:37:30.075 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:30.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff83d2d8640 1 Processor -- start 2024-04-26T00:37:30.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff83d2d8640 1 -- start start 2024-04-26T00:37:30.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff83d2d8640 10 monclient: init 2024-04-26T00:37:30.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff83d2d8640 10 monclient: _reopen_session rank -1 2024-04-26T00:37:30.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff83d2d8640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:37:30.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff83d2d8640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff83809b460 0x7ff8381cc370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:30.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff83d2d8640 10 monclient(hunting): picked mon.noname-a con 0x7ff83809b460 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:37:30.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff83d2d8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff83809cee0 0x7ff838133410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:30.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff83d2d8640 10 monclient(hunting): picked mon.noname-b con 0x7ff83809cee0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:37:30.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff83d2d8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff838134040 0x7ff838133950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:30.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff83d2d8640 10 monclient(hunting): picked mon.noname-c con 0x7ff838134040 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:37:30.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff83d2d8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:30.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff83d2d8640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff8380a2a10 con 0x7ff83809b460 2024-04-26T00:37:30.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff83d2d8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:30.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff83d2d8640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff8380a2890 con 0x7ff83809cee0 2024-04-26T00:37:30.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff83d2d8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:30.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff83d2d8640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff83806b9e0 con 0x7ff838134040 2024-04-26T00:37:30.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff83d2d8640 10 monclient(hunting): _renew_subs 2024-04-26T00:37:30.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff83d2d8640 10 monclient(hunting): authenticate will time out at 2031.375095s 2024-04-26T00:37:30.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff836ffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff83809b460 0x7ff8381cc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:30.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff8377fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff838134040 0x7ff838133950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:30.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff836ffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff83809b460 0x7ff8381cc370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:54234/0 (socket says 172.21.15.37:54234) 2024-04-26T00:37:30.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff836ffd640 1 -- 172.21.15.37:0/2442648088 learned_addr learned my addr 172.21.15.37:0/2442648088 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:37:30.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff836ffd640 10 monclient(hunting): get_auth_request con 0x7ff83809b460 auth_method 0 2024-04-26T00:37:30.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff836ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:30.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff836ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:30.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff836ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:30.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff8377fe640 10 monclient(hunting): get_auth_request con 0x7ff838134040 auth_method 0 2024-04-26T00:37:30.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff8377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:30.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff8377fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:30.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff8377fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:30.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff836ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:30.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff836ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:30.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff836ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:30.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff8377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:30.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff8377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:30.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff8377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:30.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff8367fc640 1 --2- 172.21.15.37:0/2442648088 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff83809cee0 0x7ff838133410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:30.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff836ffd640 10 monclient(hunting): handle_auth_done global_id 144828 payload 824 2024-04-26T00:37:30.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff836ffd640 1 -- 172.21.15.37:0/2442648088 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff838134040 msgr2=0x7ff838133950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:30.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff836ffd640 1 --2- 172.21.15.37:0/2442648088 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff838134040 0x7ff838133950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:30.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff836ffd640 1 -- 172.21.15.37:0/2442648088 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff83809cee0 msgr2=0x7ff838133410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:30.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff836ffd640 1 --2- 172.21.15.37:0/2442648088 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff83809cee0 0x7ff838133410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:30.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff836ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:37:30.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff836ffd640 1 monclient: found mon.noname-a 2024-04-26T00:37:30.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff836ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:37:30.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff836ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:37:30.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff836ffd640 1 -- 172.21.15.37:0/2442648088 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff838121de0 con 0x7ff83809b460 2024-04-26T00:37:30.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff8367fc640 10 monclient: get_auth_request con 0x7ff83809cee0 auth_method 0 2024-04-26T00:37:30.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.064+0000 7ff8367fc640 1 --2- 172.21.15.37:0/2442648088 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff83809cee0 0x7ff838133410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:37:30.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.068+0000 7ff836ffd640 1 -- 172.21.15.37:0/2442648088 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff83809b460 msgr2=0x7ff8381cc370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:30.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.068+0000 7ff836ffd640 1 --2- 172.21.15.37:0/2442648088 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff83809b460 0x7ff8381cc370 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ff82801c740 tx=0x7ff82801b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:30.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.068+0000 7ff81f7fe640 1 -- 172.21.15.37:0/2442648088 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8280bd070 con 0x7ff83809b460 2024-04-26T00:37:30.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.068+0000 7ff81f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:30.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.068+0000 7ff81f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:37:30.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.068+0000 7ff81f7fe640 10 monclient: dump: 2024-04-26T00:37:30.079 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:30.079 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:30.079 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.079 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.079 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:30.079 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:30.079 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:30.079 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:30.080 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:30.080 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:30.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.068+0000 7ff81f7fe640 10 monclient: _finish_auth 0 2024-04-26T00:37:30.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.068+0000 7ff81f7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:37:30.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.068+0000 7ff81f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:37:30.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.068+0000 7ff81f7fe640 1 -- 172.21.15.37:0/2442648088 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8280af850 con 0x7ff83809b460 2024-04-26T00:37:30.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.068+0000 7ff81f7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:37:30.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.068+0000 7ff81f7fe640 1 -- 172.21.15.37:0/2442648088 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8280b4040 con 0x7ff83809b460 2024-04-26T00:37:30.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.068+0000 7ff83d2d8640 5 monclient: authenticate success, global_id 144828 2024-04-26T00:37:30.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.068+0000 7ff81f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:30.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.068+0000 7ff81f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:37:30.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.068+0000 7ff81f7fe640 10 monclient: dump: 2024-04-26T00:37:30.080 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:30.080 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:30.080 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.080 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.081 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:30.081 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:30.081 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:30.081 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:30.081 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:30.081 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:30.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.068+0000 7ff83d2d8640 10 monclient: _renew_subs 2024-04-26T00:37:30.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.068+0000 7ff83d2d8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:37:30.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.068+0000 7ff834ff9640 4 set_mon_vals no callback set 2024-04-26T00:37:30.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.068+0000 7ff83d2d8640 1 -- 172.21.15.37:0/2442648088 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff83813b350 con 0x7ff83809b460 2024-04-26T00:37:30.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.068+0000 7ff83d2d8640 10 monclient: _renew_subs 2024-04-26T00:37:30.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.068+0000 7ff83d2d8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:37:30.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.068+0000 7ff83d2d8640 1 -- 172.21.15.37:0/2442648088 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff83809f510 con 0x7ff83809b460 2024-04-26T00:37:30.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.068+0000 7ff83d2d8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:30.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.068+0000 7ff83d2d8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:30.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.068+0000 7ff83d2d8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:37:30.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.068+0000 7ff83d2d8640 1 -- 172.21.15.37:0/2442648088 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff838066f40 con 0x7ff83809b460 2024-04-26T00:37:30.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.068+0000 7ff81f7fe640 1 -- 172.21.15.37:0/2442648088 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff8280c0070 con 0x7ff83809b460 2024-04-26T00:37:30.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.068+0000 7ff81f7fe640 1 --2- 172.21.15.37:0/2442648088 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff824035f00 0x7ff8240383c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:30.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.068+0000 7ff81f7fe640 1 -- 172.21.15.37:0/2442648088 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(690..690 src has 370..690) ==== 3899+0+0 (secure 0 0 0) 0x7ff8280e2530 con 0x7ff83809b460 2024-04-26T00:37:30.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.068+0000 7ff8367fc640 1 --2- 172.21.15.37:0/2442648088 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff824035f00 0x7ff8240383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:30.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.068+0000 7ff8367fc640 10 monclient: get_auth_request con 0x7ff824035f00 auth_method 0 2024-04-26T00:37:30.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.072+0000 7ff8367fc640 1 --2- 172.21.15.37:0/2442648088 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff824035f00 0x7ff8240383c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7ff8381d0950 tx=0x7ff82c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:30.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.072+0000 7ff81f7fe640 1 -- 172.21.15.37:0/2442648088 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff8280187c0 con 0x7ff83809b460 2024-04-26T00:37:30.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.072+0000 7ff81f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:30.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.072+0000 7ff81f7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:37:30.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.295+0000 7ff83d2d8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:37:30.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.295+0000 7ff83d2d8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:37:30.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.295+0000 7ff83d2d8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:37:30.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.295+0000 7ff83d2d8640 1 -- 172.21.15.37:0/2442648088 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff83813aa70 con 0x7ff83809b460 2024-04-26T00:37:30.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.295+0000 7ff81f7fe640 1 -- 172.21.15.37:0/2442648088 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7ff8280e22f0 con 0x7ff83809b460 2024-04-26T00:37:30.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.295+0000 7ff81f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:37:30.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.295+0000 7ff81f7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:37:30.298 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":132,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:37:30.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.299+0000 7ff83d2d8640 1 -- 172.21.15.37:0/2442648088 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff824035f00 msgr2=0x7ff8240383c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:30.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.299+0000 7ff83d2d8640 1 --2- 172.21.15.37:0/2442648088 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff824035f00 0x7ff8240383c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7ff8381d0950 tx=0x7ff82c008040 comp rx=0 tx=0).stop 2024-04-26T00:37:30.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.299+0000 7ff83d2d8640 10 monclient: shutdown 2024-04-26T00:37:30.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.299+0000 7ff83d2d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:37:30.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.299+0000 7ff83d2d8640 1 -- 172.21.15.37:0/2442648088 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff83809b460 msgr2=0x7ff8381cc370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:30.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.299+0000 7ff83d2d8640 1 --2- 172.21.15.37:0/2442648088 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff83809b460 0x7ff8381cc370 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ff82801c740 tx=0x7ff82801b8b0 comp rx=0 tx=0).stop 2024-04-26T00:37:30.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.299+0000 7ff83d2d8640 1 -- 172.21.15.37:0/2442648088 shutdown_connections 2024-04-26T00:37:30.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.299+0000 7ff83d2d8640 1 --2- 172.21.15.37:0/2442648088 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff824035f00 0x7ff8240383c0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:30.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.299+0000 7ff83d2d8640 1 --2- 172.21.15.37:0/2442648088 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff838134040 0x7ff838133950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:30.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.299+0000 7ff83d2d8640 1 --2- 172.21.15.37:0/2442648088 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff83809cee0 0x7ff838133410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:30.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.299+0000 7ff83d2d8640 1 --2- 172.21.15.37:0/2442648088 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff83809b460 0x7ff8381cc370 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:30.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.299+0000 7ff83d2d8640 1 -- 172.21.15.37:0/2442648088 >> 172.21.15.37:0/2442648088 conn(0x7ff8380a2d60 msgr2=0x7ff83809dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:37:30.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.299+0000 7ff83d2d8640 1 -- 172.21.15.37:0/2442648088 shutdown_connections 2024-04-26T00:37:30.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.299+0000 7ff83d2d8640 1 -- 172.21.15.37:0/2442648088 wait complete. 2024-04-26T00:37:30.319 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:37:30.319 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-26T00:37:30.319 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:37:30.319 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:37:30.342 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:30.339+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:30.342 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:30.339+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3005909169}) 2024-04-26T00:37:30.342 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:30.339+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3533578824}) 2024-04-26T00:37:30.342 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:30.339+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:30.342 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:30.339+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3005909169}) 2024-04-26T00:37:30.342 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:30.339+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3607725124}) 2024-04-26T00:37:30.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7fd8ea640 10 monclient: get_monmap_and_config 2024-04-26T00:37:30.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7fd8ea640 10 monclient: build_initial_monmap 2024-04-26T00:37:30.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7fd8ea640 10 monclient: monmap: 2024-04-26T00:37:30.460 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:37:30.460 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:30.460 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:37:30.464704+0000 2024-04-26T00:37:30.460 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:37:30.464704+0000 2024-04-26T00:37:30.461 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:37:30.461 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:30.461 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:37:30.461 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:37:30.461 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:37:30.461 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:37:30.461 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:37:30.461 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:30.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7fd8ea640 1 Processor -- start 2024-04-26T00:37:30.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7fd8ea640 1 -- start start 2024-04-26T00:37:30.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7fd8ea640 10 monclient: init 2024-04-26T00:37:30.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7fd8ea640 10 monclient: _reopen_session rank -1 2024-04-26T00:37:30.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7fd8ea640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-04-26T00:37:30.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7fd8ea640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb7f8130eb0 addr v1:172.21.15.45:6789/0 2024-04-26T00:37:30.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7fd8ea640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb7f809ef20 0x7fb7f8138eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:30.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7fd8ea640 10 monclient(hunting): picked mon.noname-b con 0x7fb7f809ef20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:37:30.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7fd8ea640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb7f81393f0 0x7fb7f813b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:30.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7fd8ea640 10 monclient(hunting): picked mon.noname-a con 0x7fb7f81393f0 addr v2:172.21.15.37:3300/0 2024-04-26T00:37:30.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7fd8ea640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7f809c190 con 0x7fb7f8130eb0 2024-04-26T00:37:30.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7fd8ea640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:30.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7fd8ea640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fb7f809df60 con 0x7fb7f81393f0 2024-04-26T00:37:30.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7fd8ea640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:30.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7fd8ea640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb7f80a11c0 con 0x7fb7f809ef20 2024-04-26T00:37:30.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7fd8ea640 10 monclient(hunting): _renew_subs 2024-04-26T00:37:30.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7fd8ea640 10 monclient(hunting): authenticate will time out at 2031.771136s 2024-04-26T00:37:30.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7effff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb7f809ef20 0x7fb7f8138eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:30.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7f77fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb7f81393f0 0x7fb7f813b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:30.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7f77fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb7f81393f0 0x7fb7f813b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:54246/0 (socket says 172.21.15.37:54246) 2024-04-26T00:37:30.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7effff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb7f809ef20 0x7fb7f8138eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:59068/0 (socket says 172.21.15.37:59068) 2024-04-26T00:37:30.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7f77fe640 1 -- 172.21.15.37:0/4082823493 learned_addr learned my addr 172.21.15.37:0/4082823493 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:37:30.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7f77fe640 10 monclient(hunting): get_auth_request con 0x7fb7f81393f0 auth_method 0 2024-04-26T00:37:30.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:30.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7f77fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:30.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:30.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7effff640 10 monclient(hunting): get_auth_request con 0x7fb7f809ef20 auth_method 0 2024-04-26T00:37:30.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:30.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7effff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:30.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7effff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:30.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:30.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:30.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:30.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:30.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:30.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:30.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7f77fe640 10 monclient(hunting): handle_auth_done global_id 144831 payload 274 2024-04-26T00:37:30.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7f77fe640 1 -- 172.21.15.37:0/4082823493 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb7f809ef20 msgr2=0x7fb7f8138eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:37:30.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7f77fe640 1 --2- 172.21.15.37:0/4082823493 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb7f809ef20 0x7fb7f8138eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:30.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7f77fe640 1 -- 172.21.15.37:0/4082823493 >> v1:172.21.15.45:6789/0 conn(0x7fb7f8130eb0 legacy=0x7fb7f8131290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:37:30.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7f77fe640 10 monclient: _finish_hunting 0 2024-04-26T00:37:30.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7f77fe640 1 monclient: found mon.noname-a 2024-04-26T00:37:30.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:37:30.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7f77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:37:30.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.459+0000 7fb7f77fe640 1 -- 172.21.15.37:0/4082823493 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7f809dc80 con 0x7fb7f81393f0 2024-04-26T00:37:30.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.463+0000 7fb7f77fe640 1 -- 172.21.15.37:0/4082823493 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb7f81393f0 msgr2=0x7fb7f813b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:30.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.463+0000 7fb7f77fe640 1 --2- 172.21.15.37:0/4082823493 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb7f81393f0 0x7fb7f813b7e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0009e50 tx=0x7fb7f001c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62482ed458d5b256 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:30.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7f67fc640 1 -- 172.21.15.37:0/4082823493 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7f00a56a0 con 0x7fb7f81393f0 2024-04-26T00:37:30.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7f67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:30.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7f67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:37:30.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7f67fc640 10 monclient: dump: 2024-04-26T00:37:30.469 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:30.469 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:30.469 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.469 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.469 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:30.469 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:30.470 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:30.470 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:30.470 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:30.470 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:30.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7f67fc640 10 monclient: _finish_auth 0 2024-04-26T00:37:30.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7f67fc640 10 monclient: _check_auth_tickets 2024-04-26T00:37:30.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:37:30.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7f67fc640 1 -- 172.21.15.37:0/4082823493 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7f00a5840 con 0x7fb7f81393f0 2024-04-26T00:37:30.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7f67fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:37:30.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7f67fc640 1 -- 172.21.15.37:0/4082823493 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7f00a5c10 con 0x7fb7f81393f0 2024-04-26T00:37:30.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7fd8ea640 5 monclient: authenticate success, global_id 144831 2024-04-26T00:37:30.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7fd8ea640 10 monclient: get_monmap_and_config success 2024-04-26T00:37:30.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7fd8ea640 4 set_mon_vals no callback set 2024-04-26T00:37:30.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7f67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:30.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7f67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:37:30.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7f67fc640 10 monclient: dump: 2024-04-26T00:37:30.471 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:30.471 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:30.471 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.471 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.471 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:30.471 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:30.471 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:30.471 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:30.471 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:30.472 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:30.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7fd8ea640 10 monclient: shutdown 2024-04-26T00:37:30.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7fd8ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:37:30.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7fd8ea640 1 -- 172.21.15.37:0/4082823493 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb7f81393f0 msgr2=0x7fb7f813b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:30.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7fd8ea640 1 --2- 172.21.15.37:0/4082823493 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb7f81393f0 0x7fb7f813b7e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0009e50 tx=0x7fb7f001c880 comp rx=0 tx=0).stop 2024-04-26T00:37:30.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7fd8ea640 1 -- 172.21.15.37:0/4082823493 shutdown_connections 2024-04-26T00:37:30.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7fd8ea640 1 --2- 172.21.15.37:0/4082823493 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb7f81393f0 0x7fb7f813b7e0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:30.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7fd8ea640 1 --2- 172.21.15.37:0/4082823493 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb7f809ef20 0x7fb7f8138eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:30.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7fd8ea640 1 -- 172.21.15.37:0/4082823493 >> 172.21.15.37:0/4082823493 conn(0x7fb7f80a1d50 msgr2=0x7fb7f80a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:37:30.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7fd8ea640 1 -- 172.21.15.37:0/4082823493 shutdown_connections 2024-04-26T00:37:30.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7fd8ea640 1 -- 172.21.15.37:0/4082823493 wait complete. 2024-04-26T00:37:30.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7fd8ea640 10 monclient: build_initial_monmap 2024-04-26T00:37:30.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7fd8ea640 10 monclient: monmap: 2024-04-26T00:37:30.473 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:37:30.473 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:37:30.473 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:37:30.473 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:37:30.473 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:37:30.473 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:37:30.473 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:37:30.473 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:37:30.473 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:37:30.473 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:30.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7fd8ea640 1 Processor -- start 2024-04-26T00:37:30.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7fd8ea640 1 -- start start 2024-04-26T00:37:30.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7fd8ea640 10 monclient: init 2024-04-26T00:37:30.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7fd8ea640 10 monclient: _reopen_session rank -1 2024-04-26T00:37:30.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7fd8ea640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:37:30.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7fd8ea640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb7f809ef20 0x7fb7f81cf500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:30.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7fd8ea640 10 monclient(hunting): picked mon.noname-a con 0x7fb7f809ef20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:37:30.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7fd8ea640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb7f8130eb0 0x7fb7f81cfa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:30.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7fd8ea640 10 monclient(hunting): picked mon.noname-c con 0x7fb7f8130eb0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:37:30.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7fd8ea640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7f81393f0 0x7fb7f81cff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:30.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7fd8ea640 10 monclient(hunting): picked mon.noname-b con 0x7fb7f81393f0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:37:30.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7fd8ea640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:30.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7fd8ea640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb7f809fdd0 con 0x7fb7f809ef20 2024-04-26T00:37:30.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7fd8ea640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:30.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7fd8ea640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb7f809fc50 con 0x7fb7f81393f0 2024-04-26T00:37:30.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7fd8ea640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:30.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7fd8ea640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb7f809ff50 con 0x7fb7f8130eb0 2024-04-26T00:37:30.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7fd8ea640 10 monclient(hunting): _renew_subs 2024-04-26T00:37:30.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7fd8ea640 10 monclient(hunting): authenticate will time out at 2031.778257s 2024-04-26T00:37:30.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7effff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb7f8130eb0 0x7fb7f81cfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:30.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7f6ffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb7f809ef20 0x7fb7f81cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:30.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7effff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb7f8130eb0 0x7fb7f81cfa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:59084/0 (socket says 172.21.15.37:59084) 2024-04-26T00:37:30.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7effff640 1 -- 172.21.15.37:0/3493741194 learned_addr learned my addr 172.21.15.37:0/3493741194 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:37:30.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7effff640 10 monclient(hunting): get_auth_request con 0x7fb7f8130eb0 auth_method 0 2024-04-26T00:37:30.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7f6ffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb7f809ef20 0x7fb7f81cf500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:54262/0 (socket says 172.21.15.37:54262) 2024-04-26T00:37:30.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:30.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7effff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:30.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7effff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:30.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7f6ffd640 10 monclient(hunting): get_auth_request con 0x7fb7f809ef20 auth_method 0 2024-04-26T00:37:30.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:30.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7f6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:30.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:30.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:30.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:30.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:30.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:30.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:30.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:30.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7f77fe640 1 --2- 172.21.15.37:0/3493741194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7f81393f0 0x7fb7f81cff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:30.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7effff640 10 monclient(hunting): handle_auth_done global_id 204218 payload 824 2024-04-26T00:37:30.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7effff640 1 -- 172.21.15.37:0/3493741194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7f81393f0 msgr2=0x7fb7f81cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:30.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7effff640 1 --2- 172.21.15.37:0/3493741194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7f81393f0 0x7fb7f81cff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:30.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7effff640 1 -- 172.21.15.37:0/3493741194 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb7f809ef20 msgr2=0x7fb7f81cf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:30.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7effff640 1 --2- 172.21.15.37:0/3493741194 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb7f809ef20 0x7fb7f81cf500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:30.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7effff640 10 monclient: _finish_hunting 0 2024-04-26T00:37:30.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7effff640 1 monclient: found mon.noname-c 2024-04-26T00:37:30.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:37:30.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7effff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:37:30.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7effff640 1 -- 172.21.15.37:0/3493741194 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7f80a1470 con 0x7fb7f8130eb0 2024-04-26T00:37:30.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7f77fe640 10 monclient: get_auth_request con 0x7fb7f81393f0 auth_method 0 2024-04-26T00:37:30.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.467+0000 7fb7f77fe640 1 --2- 172.21.15.37:0/3493741194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7f81393f0 0x7fb7f81cff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:37:30.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7effff640 1 -- 172.21.15.37:0/3493741194 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb7f8130eb0 msgr2=0x7fb7f81cfa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:30.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7effff640 1 --2- 172.21.15.37:0/3493741194 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb7f8130eb0 0x7fb7f81cfa40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800e8b0 tx=0x7fb7e800ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:30.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7f4ff9640 1 -- 172.21.15.37:0/3493741194 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7e8017070 con 0x7fb7f8130eb0 2024-04-26T00:37:30.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7f4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:30.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7f4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:37:30.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7f4ff9640 10 monclient: dump: 2024-04-26T00:37:30.482 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:30.482 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:30.482 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.482 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.483 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:30.483 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:30.483 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:30.483 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:30.483 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:30.483 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:30.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7f4ff9640 10 monclient: _finish_auth 0 2024-04-26T00:37:30.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7f4ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:37:30.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:37:30.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7f4ff9640 1 -- 172.21.15.37:0/3493741194 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7e800cb50 con 0x7fb7f8130eb0 2024-04-26T00:37:30.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7fd8ea640 5 monclient: authenticate success, global_id 204218 2024-04-26T00:37:30.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7fd8ea640 10 monclient: _renew_subs 2024-04-26T00:37:30.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7fd8ea640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:37:30.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7fd8ea640 1 -- 172.21.15.37:0/3493741194 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7f80a49a0 con 0x7fb7f8130eb0 2024-04-26T00:37:30.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7f4ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:37:30.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7f4ff9640 1 -- 172.21.15.37:0/3493741194 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7e8013470 con 0x7fb7f8130eb0 2024-04-26T00:37:30.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7f4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:30.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7f4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:37:30.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7f4ff9640 10 monclient: dump: 2024-04-26T00:37:30.484 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:30.484 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:30.484 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.484 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.484 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:30.484 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:30.484 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:30.484 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:30.484 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:30.484 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:30.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7f5ffb640 4 set_mon_vals no callback set 2024-04-26T00:37:30.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7fd8ea640 10 monclient: _renew_subs 2024-04-26T00:37:30.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7fd8ea640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:37:30.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7fd8ea640 1 -- 172.21.15.37:0/3493741194 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7f8137490 con 0x7fb7f8130eb0 2024-04-26T00:37:30.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7fd8ea640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:30.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7fd8ea640 1 --2- 172.21.15.37:0/3493741194 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb7f81333c0 0x7fb7f81337a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:30.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7fd8ea640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:30.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7fd8ea640 1 -- 172.21.15.37:0/3493741194 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb7f8136c50 con 0x7fb7f81333c0 2024-04-26T00:37:30.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7f4ff9640 1 -- 172.21.15.37:0/3493741194 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb7e801a070 con 0x7fb7f8130eb0 2024-04-26T00:37:30.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7f6ffd640 1 --2- 172.21.15.37:0/3493741194 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb7f81333c0 0x7fb7f81337a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:30.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7f6ffd640 10 monclient: get_auth_request con 0x7fb7f81333c0 auth_method 0 2024-04-26T00:37:30.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:30.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7f6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:30.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:30.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:30.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:30.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:30.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7f4ff9640 1 --2- 172.21.15.37:0/3493741194 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb7d8035ed0 0x7fb7d8038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:30.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7f4ff9640 1 -- 172.21.15.37:0/3493741194 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(690..690 src has 370..690) ==== 3899+0+0 (secure 0 0 0) 0x7fb7e8048b00 con 0x7fb7f8130eb0 2024-04-26T00:37:30.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7f6ffd640 10 monclient(hunting): handle_auth_done global_id 144837 payload 824 2024-04-26T00:37:30.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7f6ffd640 1 -- 172.21.15.37:0/3493741194 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb7f804f510 con 0x7fb7f81333c0 2024-04-26T00:37:30.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7f77fe640 1 --2- 172.21.15.37:0/3493741194 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb7d8035ed0 0x7fb7d8038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:30.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.471+0000 7fb7f77fe640 10 monclient: get_auth_request con 0x7fb7d8035ed0 auth_method 0 2024-04-26T00:37:30.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.475+0000 7fb7f6ffd640 1 -- 172.21.15.37:0/3493741194 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb7f81333c0 msgr2=0x7fb7f81337a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:30.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.475+0000 7fb7f6ffd640 1 --2- 172.21.15.37:0/3493741194 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb7f81333c0 0x7fb7f81337a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb7e000f770 tx=0x7fb7e000fc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:30.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.475+0000 7fb7f4ff9640 1 -- 172.21.15.37:0/3493741194 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7e000fe40 con 0x7fb7f81333c0 2024-04-26T00:37:30.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.475+0000 7fb7f4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:30.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.475+0000 7fb7f4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:37:30.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.475+0000 7fb7f4ff9640 10 monclient: dump: 2024-04-26T00:37:30.486 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:30.487 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:30.487 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.487 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.487 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:30.487 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:30.487 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:30.487 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:30.487 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:30.487 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:30.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.475+0000 7fb7f4ff9640 1 -- 172.21.15.37:0/3493741194 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb7e0019070 con 0x7fb7f81333c0 2024-04-26T00:37:30.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.475+0000 7fb7f4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:30.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.475+0000 7fb7f4ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:37:30.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.475+0000 7fb7f4ff9640 1 -- 172.21.15.37:0/3493741194 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb7f81333c0 msgr2=0x7fb7f81337a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:30.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.475+0000 7fb7f4ff9640 1 --2- 172.21.15.37:0/3493741194 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb7f81333c0 0x7fb7f81337a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb7e000f770 tx=0x7fb7e000fc30 comp rx=0 tx=0).stop 2024-04-26T00:37:30.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.475+0000 7fb7f4ff9640 1 -- 172.21.15.37:0/3493741194 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb7f81333c0 msgr2=0x7fb7f81337a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:37:30.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.475+0000 7fb7f4ff9640 1 --2- 172.21.15.37:0/3493741194 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb7f81333c0 0x7fb7f81337a0 secure :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb7e000f770 tx=0x7fb7e000fc30 comp rx=0 tx=0).stop 2024-04-26T00:37:30.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.475+0000 7fb7f77fe640 1 --2- 172.21.15.37:0/3493741194 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb7d8035ed0 0x7fb7d8038390 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fb7f00ab000 tx=0x7fb7f00a7070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:30.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.479+0000 7fb7fd8ea640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:37:30.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.479+0000 7fb7fd8ea640 1 --2- 172.21.15.37:0/3493741194 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb7f8133ce0 0x7fb7f813e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:30.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.479+0000 7fb7fd8ea640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:30.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.479+0000 7fb7fd8ea640 1 -- 172.21.15.37:0/3493741194 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb7f80a0f60 con 0x7fb7f8133ce0 2024-04-26T00:37:30.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.479+0000 7fb7f6ffd640 1 --2- 172.21.15.37:0/3493741194 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb7f8133ce0 0x7fb7f813e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:30.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.479+0000 7fb7f6ffd640 10 monclient: get_auth_request con 0x7fb7f8133ce0 auth_method 0 2024-04-26T00:37:30.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.479+0000 7fb7f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:30.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.479+0000 7fb7f6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:30.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.479+0000 7fb7f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:30.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.479+0000 7fb7f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:30.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.479+0000 7fb7f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:30.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.479+0000 7fb7f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:30.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.479+0000 7fb7f6ffd640 10 monclient(hunting): handle_auth_done global_id 144840 payload 824 2024-04-26T00:37:30.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.479+0000 7fb7f6ffd640 1 -- 172.21.15.37:0/3493741194 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb7f8135aa0 con 0x7fb7f8133ce0 2024-04-26T00:37:30.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.483+0000 7fb7f6ffd640 1 -- 172.21.15.37:0/3493741194 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb7f8133ce0 msgr2=0x7fb7f813e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:30.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.483+0000 7fb7f6ffd640 1 --2- 172.21.15.37:0/3493741194 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb7f8133ce0 0x7fb7f813e540 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb7e000f770 tx=0x7fb7e0003770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:30.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.483+0000 7fb7f4ff9640 1 -- 172.21.15.37:0/3493741194 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7e0006e00 con 0x7fb7f8133ce0 2024-04-26T00:37:30.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.483+0000 7fb7f4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:30.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.483+0000 7fb7f4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:37:30.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.483+0000 7fb7f4ff9640 10 monclient: dump: 2024-04-26T00:37:30.491 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:30.491 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:30.491 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.491 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.491 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:30.491 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:30.491 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:30.491 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:30.491 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:30.492 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:30.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.483+0000 7fb7f4ff9640 1 -- 172.21.15.37:0/3493741194 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4890 (secure 0 0 0) 0x7fb7e0006e00 con 0x7fb7f8133ce0 2024-04-26T00:37:30.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.483+0000 7fb7f4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:37:30.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.483+0000 7fb7f4ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:37:30.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.483+0000 7fb7f4ff9640 1 -- 172.21.15.37:0/3493741194 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb7f8133ce0 msgr2=0x7fb7f813e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:30.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.483+0000 7fb7f4ff9640 1 --2- 172.21.15.37:0/3493741194 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb7f8133ce0 0x7fb7f813e540 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb7e000f770 tx=0x7fb7e0003770 comp rx=0 tx=0).stop 2024-04-26T00:37:30.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.483+0000 7fb7f4ff9640 1 -- 172.21.15.37:0/3493741194 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb7f8133ce0 msgr2=0x7fb7f813e540 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:37:30.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.483+0000 7fb7f4ff9640 1 --2- 172.21.15.37:0/3493741194 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb7f8133ce0 0x7fb7f813e540 secure :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb7e000f770 tx=0x7fb7e0003770 comp rx=0 tx=0).stop 2024-04-26T00:37:30.492 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:37:30.492 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:37:30.493 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:37:30.493 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:37:30.493 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 132, 2024-04-26T00:37:30.493 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 337027, 2024-04-26T00:37:30.493 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:37:30.493 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:37:30.493 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:37:30.493 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:37:30.493 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:30.493 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 4, 2024-04-26T00:37:30.493 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:37:30.493 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:37:30.493 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:37:30.494 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:37:30.494 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:37:30.494 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:37:30.494 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:37:30.494 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:37:30.494 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:37:30.494 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:37:30.494 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:37:30.494 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:37:30.494 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:37:30.494 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:37:30.494 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:30.494 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:37:30.494 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:37:30.495 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:37:30.495 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:37:30.495 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:37:30.495 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:37:30.495 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:37:30.495 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:37:30.495 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:37:30.495 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:37:30.495 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:37:30.495 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:37:30.495 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:37:30.495 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:30.495 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:30.495 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:37:30.496 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:37:30.496 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:37:30.496 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:37:30.496 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:37:30.496 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:37:30.496 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:37:30.496 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:37:30.496 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:37:30.496 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:37:30.496 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:37:30.496 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:37:30.496 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:37:30.496 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:37:30.496 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:37:30.497 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:37:30.497 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:37:30.497 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:37:30.497 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:37:30.497 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:37:30.497 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:37:30.497 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:37:30.497 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:37:30.497 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:37:30.497 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:37:30.497 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:37:30.497 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:37:30.497 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:37:30.497 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:30.497 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:37:30.498 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:30.498 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:37:30.498 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:30.498 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:37:30.498 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:37:30.498 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:37:30.498 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:37:30.498 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:30.498 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:37:30.498 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:37:30.498 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:30.498 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:30.498 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:30.498 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:37:30.499 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:37:30.499 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:30.499 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:30.499 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:30.499 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:30.499 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:37:30.499 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:37:30.499 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:37:30.499 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:37:30.499 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:37:30.499 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:30.499 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:30.499 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:37:30.499 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:37:30.499 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:37:30.500 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:37:30.500 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:30.500 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:37:30.500 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:37:30.500 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:30.500 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:30.500 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:30.500 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:37:30.500 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:37:30.500 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:30.500 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:30.500 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:30.500 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:30.500 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:37:30.500 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:37:30.501 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:37:30.501 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:37:30.501 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:37:30.501 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:30.501 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:30.501 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:37:30.501 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:37:30.501 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:37:30.501 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:37:30.501 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:30.501 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:37:30.501 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:37:30.501 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:30.502 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:30.502 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:30.502 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:37:30.502 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:37:30.502 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:30.502 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:30.502 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:30.502 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:30.502 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:37:30.502 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:37:30.502 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:37:30.502 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:37:30.502 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:37:30.502 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:30.502 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:30.503 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:30.503 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:37:30.503 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:37:30.503 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:30.503 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:37:30.503 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:37:30.503 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:37:30.503 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:30.503 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:30.503 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:37:30.503 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:30.503 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:37:30.503 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:37:30.503 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:37:30.503 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:30.503 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:30.504 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:37:30.504 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:30.504 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:37:30.504 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:37:30.504 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:37:30.504 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:30.504 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:30.504 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:37:30.504 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:30.504 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:37:30.504 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:37:30.504 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:37:30.504 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:30.504 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:30.504 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:30.505 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:37:30.505 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:37:30.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.483+0000 7fb7fd8ea640 1 -- 172.21.15.37:0/3493741194 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb7d8035ed0 msgr2=0x7fb7d8038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:30.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.483+0000 7fb7fd8ea640 1 --2- 172.21.15.37:0/3493741194 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb7d8035ed0 0x7fb7d8038390 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fb7f00ab000 tx=0x7fb7f00a7070 comp rx=0 tx=0).stop 2024-04-26T00:37:30.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.483+0000 7fb7fd8ea640 10 monclient: shutdown 2024-04-26T00:37:30.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.483+0000 7fb7fd8ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:37:30.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.483+0000 7fb7fd8ea640 1 -- 172.21.15.37:0/3493741194 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb7f8130eb0 msgr2=0x7fb7f81cfa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:30.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.483+0000 7fb7fd8ea640 1 --2- 172.21.15.37:0/3493741194 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb7f8130eb0 0x7fb7f81cfa40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800e8b0 tx=0x7fb7e800ed70 comp rx=0 tx=0).stop 2024-04-26T00:37:30.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.483+0000 7fb7f77fe640 1 -- 172.21.15.37:0/3493741194 reap_dead start 2024-04-26T00:37:30.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.483+0000 7fb7f77fe640 1 -- 172.21.15.37:0/3493741194 reap_dead start 2024-04-26T00:37:30.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.483+0000 7fb7fd8ea640 1 -- 172.21.15.37:0/3493741194 shutdown_connections 2024-04-26T00:37:30.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.483+0000 7fb7fd8ea640 1 -- 172.21.15.37:0/3493741194 >> 172.21.15.37:0/3493741194 conn(0x7fb7f80a1d50 msgr2=0x7fb7f80726b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:37:30.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.483+0000 7fb7fd8ea640 1 -- 172.21.15.37:0/3493741194 shutdown_connections 2024-04-26T00:37:30.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.483+0000 7fb7fd8ea640 1 -- 172.21.15.37:0/3493741194 wait complete. 2024-04-26T00:37:30.507 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:37:30.507 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:37:30.507 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:37:30.577 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:30.575+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:30.577 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:30.575+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3795720834}) 2024-04-26T00:37:30.577 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:30.575+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2213439390}) 2024-04-26T00:37:30.578 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:30.575+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:30.578 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:30.575+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3795720834}) 2024-04-26T00:37:30.578 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:30.575+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1037607158}) 2024-04-26T00:37:30.584 INFO:tasks.rados.rados.0.smithi045.stdout:3094: done (3 left) 2024-04-26T00:37:30.584 INFO:tasks.rados.rados.0.smithi045.stdout:3095: read oid 22 snap -1 2024-04-26T00:37:30.584 INFO:tasks.rados.rados.0.smithi045.stdout:3095: expect (ObjNum 1106 snap 329 seq_num 1106) 2024-04-26T00:37:30.585 INFO:tasks.rados.rados.0.smithi045.stdout:3096: rollback oid 33 current snap is 332 2024-04-26T00:37:30.585 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 33 to 330 2024-04-26T00:37:30.585 INFO:tasks.rados.rados.0.smithi045.stdout:3092: finishing copy_from racing read to smithi04526579-10 2024-04-26T00:37:30.585 INFO:tasks.rados.rados.0.smithi045.stdout:3097: read oid 2 snap 329 2024-04-26T00:37:30.585 INFO:tasks.rados.rados.0.smithi045.stdout:3097: expect (ObjNum 1074 snap 322 seq_num 1074) 2024-04-26T00:37:30.585 INFO:tasks.rados.rados.0.smithi045.stdout:3098: rollback oid 11 current snap is 332 2024-04-26T00:37:30.585 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 11 to 331 2024-04-26T00:37:30.585 INFO:tasks.rados.rados.0.smithi045.stdout:3093: finishing copy_from racing read to smithi04526579-21 2024-04-26T00:37:30.585 INFO:tasks.rados.rados.0.smithi045.stdout:3092: finishing copy_from to smithi04526579-10 2024-04-26T00:37:30.586 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 2339 (ObjNum 1112 snap 331 seq_num 1112) dirty exists 2024-04-26T00:37:30.586 INFO:tasks.rados.rados.0.smithi045.stdout:3093: finishing copy_from to smithi04526579-21 2024-04-26T00:37:30.586 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 1192 (ObjNum 1087 snap 324 seq_num 1087) dirty exists 2024-04-26T00:37:30.586 INFO:tasks.rados.rados.0.smithi045.stdout:3091: finishing copy_from racing read to smithi04526579-26 2024-04-26T00:37:30.586 INFO:tasks.rados.rados.0.smithi045.stdout:3091: finishing copy_from to smithi04526579-26 2024-04-26T00:37:30.586 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 2420 (ObjNum 1051 snap 317 seq_num 1051) dirty exists 2024-04-26T00:37:30.586 INFO:tasks.rados.rados.0.smithi045.stdout:3091: done (6 left) 2024-04-26T00:37:30.586 INFO:tasks.rados.rados.0.smithi045.stdout:3092: done (5 left) 2024-04-26T00:37:30.586 INFO:tasks.rados.rados.0.smithi045.stdout:3093: done (4 left) 2024-04-26T00:37:30.586 INFO:tasks.rados.rados.0.smithi045.stdout:3099: read oid 29 snap -1 2024-04-26T00:37:30.586 INFO:tasks.rados.rados.0.smithi045.stdout:3099: expect (ObjNum 987 snap 302 seq_num 987) 2024-04-26T00:37:30.586 INFO:tasks.rados.rados.0.smithi045.stdout:3100: write (excl) oid 36 current snap is 332 2024-04-26T00:37:30.586 INFO:tasks.rados.rados.0.smithi045.stdout:3100: seq_num 1113 ranges {706191=739675,1913635=431384} 2024-04-26T00:37:30.587 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:30.583+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:30.587 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:30.583+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1422952940}) 2024-04-26T00:37:30.587 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:30.583+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=704944028}) 2024-04-26T00:37:30.587 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:30.583+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:30.587 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:30.583+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1422952940}) 2024-04-26T00:37:30.587 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:30.583+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2637480770}) 2024-04-26T00:37:30.594 INFO:tasks.rados.rados.0.smithi045.stdout:3100: writing smithi04526579-36 from 706191 to 1445866 tid 1 2024-04-26T00:37:30.600 INFO:tasks.rados.rados.0.smithi045.stdout:3100: writing smithi04526579-36 from 1913635 to 2345019 tid 2 2024-04-26T00:37:30.601 INFO:tasks.rados.rados.0.smithi045.stdout:3101: read oid 34 snap -1 2024-04-26T00:37:30.601 INFO:tasks.rados.rados.0.smithi045.stdout:3101: expect (ObjNum 1104 snap 329 seq_num 1104) 2024-04-26T00:37:30.601 INFO:tasks.rados.rados.0.smithi045.stdout:3102: read oid 46 snap -1 2024-04-26T00:37:30.601 INFO:tasks.rados.rados.0.smithi045.stdout:3102: expect deleted 2024-04-26T00:37:30.601 INFO:tasks.rados.rados.0.smithi045.stdout:3103: snap_remove snap 326 2024-04-26T00:37:30.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.635+0000 7f26fb141640 10 monclient: get_monmap_and_config 2024-04-26T00:37:30.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.635+0000 7f26fb141640 10 monclient: build_initial_monmap 2024-04-26T00:37:30.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.635+0000 7f26fb141640 10 monclient: monmap: 2024-04-26T00:37:30.639 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:37:30.639 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:30.639 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:37:30.642979+0000 2024-04-26T00:37:30.639 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:37:30.642979+0000 2024-04-26T00:37:30.639 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:37:30.639 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:30.639 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:37:30.639 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:37:30.639 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:37:30.639 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:37:30.639 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:37:30.639 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:30.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.635+0000 7f26fb141640 1 Processor -- start 2024-04-26T00:37:30.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.635+0000 7f26fb141640 1 -- start start 2024-04-26T00:37:30.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.635+0000 7f26fb141640 10 monclient: init 2024-04-26T00:37:30.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.639+0000 7f26fb141640 10 monclient: _reopen_session rank -1 2024-04-26T00:37:30.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.639+0000 7f26fb141640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-04-26T00:37:30.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.639+0000 7f26fb141640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f26f4136f20 0x7f26f409bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:30.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.639+0000 7f26fb141640 10 monclient(hunting): picked mon.noname-c con 0x7f26f4136f20 addr v2:172.21.15.45:3300/0 2024-04-26T00:37:30.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.639+0000 7f26fb141640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f26f4137300 addr v1:172.21.15.45:6789/0 2024-04-26T00:37:30.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.639+0000 7f26fb141640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f26f413c450 0x7f26f413e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:30.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.639+0000 7f26fb141640 10 monclient(hunting): picked mon.noname-a con 0x7f26f413c450 addr v2:172.21.15.37:3300/0 2024-04-26T00:37:30.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.639+0000 7f26fb141640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26f40a50f0 con 0x7f26f4137300 2024-04-26T00:37:30.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.639+0000 7f26fb141640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:30.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.639+0000 7f26fb141640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f26f409b330 con 0x7f26f413c450 2024-04-26T00:37:30.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.639+0000 7f26fb141640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:30.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.639+0000 7f26fb141640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f26f409b610 con 0x7f26f4136f20 2024-04-26T00:37:30.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.639+0000 7f26fb141640 10 monclient(hunting): _renew_subs 2024-04-26T00:37:30.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.639+0000 7f26fb141640 10 monclient(hunting): authenticate will time out at 2031.948313s 2024-04-26T00:37:30.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.639+0000 7f26f96de640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f26f413c450 0x7f26f413e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:30.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.639+0000 7f26f96de640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f26f413c450 0x7f26f413e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:54302/0 (socket says 172.21.15.37:54302) 2024-04-26T00:37:30.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.639+0000 7f26f96de640 1 -- 172.21.15.37:0/3466555575 learned_addr learned my addr 172.21.15.37:0/3466555575 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:37:30.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.639+0000 7f26f96de640 10 monclient(hunting): get_auth_request con 0x7f26f413c450 auth_method 0 2024-04-26T00:37:30.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.639+0000 7f26f96de640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:30.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.639+0000 7f26f96de640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:30.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.639+0000 7f26f96de640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:30.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.639+0000 7f26f96de640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:30.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.639+0000 7f26f96de640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:30.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.639+0000 7f26f96de640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:30.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.639+0000 7f26f8edd640 1 --2- 172.21.15.37:0/3466555575 >> v2:172.21.15.45:3300/0 conn(0x7f26f4136f20 0x7f26f409bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:30.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.639+0000 7f26f8edd640 10 monclient(hunting): get_auth_request con 0x7f26f4136f20 auth_method 0 2024-04-26T00:37:30.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.639+0000 7f26f8edd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:30.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.639+0000 7f26f8edd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:30.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.639+0000 7f26f8edd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:30.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.639+0000 7f26f96de640 10 monclient(hunting): handle_auth_done global_id 144843 payload 274 2024-04-26T00:37:30.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.639+0000 7f26f96de640 1 -- 172.21.15.37:0/3466555575 >> v2:172.21.15.45:3300/0 conn(0x7f26f4136f20 msgr2=0x7f26f409bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:37:30.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.639+0000 7f26f96de640 1 --2- 172.21.15.37:0/3466555575 >> v2:172.21.15.45:3300/0 conn(0x7f26f4136f20 0x7f26f409bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:30.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.639+0000 7f26f96de640 1 -- 172.21.15.37:0/3466555575 >> v1:172.21.15.45:6789/0 conn(0x7f26f4137300 legacy=0x7f26f41377b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:37:30.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.639+0000 7f26f96de640 10 monclient: _finish_hunting 0 2024-04-26T00:37:30.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.639+0000 7f26f96de640 1 monclient: found mon.noname-a 2024-04-26T00:37:30.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.639+0000 7f26f96de640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:37:30.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.639+0000 7f26f96de640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:37:30.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.639+0000 7f26f96de640 1 -- 172.21.15.37:0/3466555575 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26f409b8f0 con 0x7f26f413c450 2024-04-26T00:37:30.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26f96de640 1 -- 172.21.15.37:0/3466555575 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26f413c450 msgr2=0x7f26f413e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:30.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26f96de640 1 --2- 172.21.15.37:0/3466555575 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26f413c450 0x7f26f413e840 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f26f4068940 tx=0x7f26ec01c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43132b7e32cd57e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:30.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26eb7fe640 1 -- 172.21.15.37:0/3466555575 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ec0a57e0 con 0x7f26f413c450 2024-04-26T00:37:30.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:30.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26eb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:37:30.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26eb7fe640 10 monclient: dump: 2024-04-26T00:37:30.649 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:30.649 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:30.649 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.649 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.649 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:30.649 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:30.649 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:30.649 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:30.650 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:30.650 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:30.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26eb7fe640 10 monclient: _finish_auth 0 2024-04-26T00:37:30.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26eb7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:37:30.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:37:30.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26eb7fe640 1 -- 172.21.15.37:0/3466555575 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26ec0a5980 con 0x7f26f413c450 2024-04-26T00:37:30.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26eb7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:37:30.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26eb7fe640 1 -- 172.21.15.37:0/3466555575 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ec0a5d50 con 0x7f26f413c450 2024-04-26T00:37:30.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:30.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26eb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:37:30.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26eb7fe640 10 monclient: dump: 2024-04-26T00:37:30.650 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:30.650 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:30.650 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.651 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.651 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:30.651 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:30.651 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:30.651 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:30.651 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:30.651 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:30.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26fb141640 5 monclient: authenticate success, global_id 144843 2024-04-26T00:37:30.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26fb141640 10 monclient: get_monmap_and_config success 2024-04-26T00:37:30.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26fb141640 4 set_mon_vals no callback set 2024-04-26T00:37:30.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26fb141640 10 monclient: shutdown 2024-04-26T00:37:30.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26fb141640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:37:30.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26fb141640 1 -- 172.21.15.37:0/3466555575 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26f413c450 msgr2=0x7f26f413e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:30.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26fb141640 1 --2- 172.21.15.37:0/3466555575 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26f413c450 0x7f26f413e840 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f26f4068940 tx=0x7f26ec01c900 comp rx=0 tx=0).stop 2024-04-26T00:37:30.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26fb141640 1 -- 172.21.15.37:0/3466555575 shutdown_connections 2024-04-26T00:37:30.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26fb141640 1 --2- 172.21.15.37:0/3466555575 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26f413c450 0x7f26f413e840 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:30.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26fb141640 1 --2- 172.21.15.37:0/3466555575 >> v2:172.21.15.45:3300/0 conn(0x7f26f4136f20 0x7f26f409bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:30.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26fb141640 1 -- 172.21.15.37:0/3466555575 >> 172.21.15.37:0/3466555575 conn(0x7f26f40a61b0 msgr2=0x7f26f40a65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:37:30.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26fb141640 1 -- 172.21.15.37:0/3466555575 shutdown_connections 2024-04-26T00:37:30.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26fb141640 1 -- 172.21.15.37:0/3466555575 wait complete. 2024-04-26T00:37:30.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26fb141640 10 monclient: build_initial_monmap 2024-04-26T00:37:30.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26fb141640 10 monclient: monmap: 2024-04-26T00:37:30.652 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:37:30.652 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:37:30.652 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:37:30.652 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:37:30.652 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:37:30.652 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:37:30.652 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:37:30.653 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:37:30.653 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:37:30.653 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:30.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26fb141640 1 Processor -- start 2024-04-26T00:37:30.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26fb141640 1 -- start start 2024-04-26T00:37:30.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26fb141640 10 monclient: init 2024-04-26T00:37:30.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26fb141640 10 monclient: _reopen_session rank -1 2024-04-26T00:37:30.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26fb141640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:37:30.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26fb141640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26f4136f20 0x7f26f41ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:30.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26fb141640 10 monclient(hunting): picked mon.noname-c con 0x7f26f4136f20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:37:30.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26fb141640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26f4137300 0x7f26f4134f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:30.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26fb141640 10 monclient(hunting): picked mon.noname-a con 0x7f26f4137300 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:37:30.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26fb141640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26f413c450 0x7f26f4137f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:30.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26fb141640 10 monclient(hunting): picked mon.noname-b con 0x7f26f413c450 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:37:30.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26fb141640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:30.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26fb141640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f26f4141930 con 0x7f26f4137300 2024-04-26T00:37:30.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26fb141640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:30.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26fb141640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f26f41417b0 con 0x7f26f413c450 2024-04-26T00:37:30.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26fb141640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:30.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26fb141640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f26f4141ab0 con 0x7f26f4136f20 2024-04-26T00:37:30.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26fb141640 10 monclient(hunting): _renew_subs 2024-04-26T00:37:30.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26fb141640 10 monclient(hunting): authenticate will time out at 2031.955731s 2024-04-26T00:37:30.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26ebfff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26f4137300 0x7f26f4134f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:30.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26f8edd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26f4136f20 0x7f26f41ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:30.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26f8edd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26f4136f20 0x7f26f41ceec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:59092/0 (socket says 172.21.15.37:59092) 2024-04-26T00:37:30.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26ebfff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26f4137300 0x7f26f4134f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:54316/0 (socket says 172.21.15.37:54316) 2024-04-26T00:37:30.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26f8edd640 1 -- 172.21.15.37:0/3481568648 learned_addr learned my addr 172.21.15.37:0/3481568648 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:37:30.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26f8edd640 10 monclient(hunting): get_auth_request con 0x7f26f4136f20 auth_method 0 2024-04-26T00:37:30.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26f8edd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:30.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26f8edd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:30.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26f8edd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:30.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26ebfff640 10 monclient(hunting): get_auth_request con 0x7f26f4137300 auth_method 0 2024-04-26T00:37:30.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:30.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26ebfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:30.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.643+0000 7f26ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:30.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26f8edd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:30.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26f8edd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:30.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26f8edd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:30.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:30.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:30.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:30.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26f96de640 1 --2- 172.21.15.37:0/3481568648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26f413c450 0x7f26f4137f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:30.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26f96de640 10 monclient(hunting): get_auth_request con 0x7f26f413c450 auth_method 0 2024-04-26T00:37:30.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26f96de640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:30.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26f96de640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:30.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26f96de640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:30.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26ebfff640 10 monclient(hunting): handle_auth_done global_id 144846 payload 824 2024-04-26T00:37:30.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26ebfff640 1 -- 172.21.15.37:0/3481568648 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26f4136f20 msgr2=0x7f26f41ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:30.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26ebfff640 1 --2- 172.21.15.37:0/3481568648 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26f4136f20 0x7f26f41ceec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:30.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26ebfff640 1 -- 172.21.15.37:0/3481568648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26f413c450 msgr2=0x7f26f4137f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:30.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26ebfff640 1 --2- 172.21.15.37:0/3481568648 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f26f413c450 0x7f26f4137f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:30.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26ebfff640 10 monclient: _finish_hunting 0 2024-04-26T00:37:30.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26ebfff640 1 monclient: found mon.noname-a 2024-04-26T00:37:30.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:37:30.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26ebfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:37:30.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26ebfff640 1 -- 172.21.15.37:0/3481568648 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26f41399e0 con 0x7f26f4137300 2024-04-26T00:37:30.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26f8edd640 1 --2- 172.21.15.37:0/3481568648 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26f4136f20 0x7f26f41ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:37:30.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26ebfff640 1 -- 172.21.15.37:0/3481568648 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26f4137300 msgr2=0x7f26f4134f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:30.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26ebfff640 1 --2- 172.21.15.37:0/3481568648 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26f4137300 0x7f26f4134f60 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f26e400c3e0 tx=0x7f26e400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:30.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26e97fa640 1 -- 172.21.15.37:0/3481568648 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e40119a0 con 0x7f26f4137300 2024-04-26T00:37:30.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26e97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:30.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26e97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:37:30.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26e97fa640 10 monclient: dump: 2024-04-26T00:37:30.660 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:30.660 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:30.660 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.660 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.660 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:30.660 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:30.660 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:30.661 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:30.661 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:30.661 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:30.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26e97fa640 10 monclient: _finish_auth 0 2024-04-26T00:37:30.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26e97fa640 10 monclient: _check_auth_tickets 2024-04-26T00:37:30.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26e97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:37:30.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26e97fa640 1 -- 172.21.15.37:0/3481568648 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26e4017070 con 0x7f26f4137300 2024-04-26T00:37:30.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26e97fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:37:30.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26e97fa640 1 -- 172.21.15.37:0/3481568648 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e4011d80 con 0x7f26f4137300 2024-04-26T00:37:30.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26e97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:30.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26e97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:37:30.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26e97fa640 10 monclient: dump: 2024-04-26T00:37:30.661 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:30.661 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:30.662 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.662 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.662 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:30.662 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:30.662 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:30.662 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:30.662 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:30.662 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:30.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26eaffd640 4 set_mon_vals no callback set 2024-04-26T00:37:30.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26fb141640 5 monclient: authenticate success, global_id 144846 2024-04-26T00:37:30.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26fb141640 10 monclient: _renew_subs 2024-04-26T00:37:30.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26fb141640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:37:30.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26fb141640 1 -- 172.21.15.37:0/3481568648 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26f4135ea0 con 0x7f26f4137300 2024-04-26T00:37:30.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26fb141640 10 monclient: _renew_subs 2024-04-26T00:37:30.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26fb141640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:37:30.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.647+0000 7f26fb141640 1 -- 172.21.15.37:0/3481568648 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26f413b4a0 con 0x7f26f4137300 2024-04-26T00:37:30.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.651+0000 7f26fb141640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:30.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.651+0000 7f26fb141640 1 --2- 172.21.15.37:0/3481568648 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26f402c000 0x7f26f402c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:30.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.651+0000 7f26fb141640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:30.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.651+0000 7f26fb141640 1 -- 172.21.15.37:0/3481568648 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f26f413b930 con 0x7f26f402c000 2024-04-26T00:37:30.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.651+0000 7f26f8edd640 1 --2- 172.21.15.37:0/3481568648 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26f402c000 0x7f26f402c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:30.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.651+0000 7f26f8edd640 10 monclient: get_auth_request con 0x7f26f402c000 auth_method 0 2024-04-26T00:37:30.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.651+0000 7f26f8edd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:30.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.651+0000 7f26f8edd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:30.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.651+0000 7f26f8edd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:30.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.651+0000 7f26e97fa640 1 -- 172.21.15.37:0/3481568648 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f26e401c460 con 0x7f26f4137300 2024-04-26T00:37:30.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.651+0000 7f26f8edd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:30.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.651+0000 7f26f8edd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:30.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.651+0000 7f26f8edd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:30.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.651+0000 7f26f8edd640 10 monclient(hunting): handle_auth_done global_id 204224 payload 824 2024-04-26T00:37:30.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.651+0000 7f26f8edd640 1 -- 172.21.15.37:0/3481568648 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f26f404f510 con 0x7f26f402c000 2024-04-26T00:37:30.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.651+0000 7f26e97fa640 1 --2- 172.21.15.37:0/3481568648 >> v2:172.21.15.45:6812/2379918623 conn(0x7f26e0035f00 0x7f26e00383c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:30.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.651+0000 7f26e97fa640 1 -- 172.21.15.37:0/3481568648 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(691..691 src has 370..691) ==== 3927+0+0 (secure 0 0 0) 0x7f26e4048830 con 0x7f26f4137300 2024-04-26T00:37:30.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.651+0000 7f26f96de640 1 --2- 172.21.15.37:0/3481568648 >> v2:172.21.15.45:6812/2379918623 conn(0x7f26e0035f00 0x7f26e00383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:30.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.651+0000 7f26f96de640 10 monclient: get_auth_request con 0x7f26e0035f00 auth_method 0 2024-04-26T00:37:30.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.651+0000 7f26f8edd640 1 -- 172.21.15.37:0/3481568648 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26f402c000 msgr2=0x7f26f402c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:30.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.651+0000 7f26f8edd640 1 --2- 172.21.15.37:0/3481568648 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26f402c000 0x7f26f402c3e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f26f0010cb0 tx=0x7f26f000e5a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:30.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.651+0000 7f26e97fa640 1 -- 172.21.15.37:0/3481568648 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f0018070 con 0x7f26f402c000 2024-04-26T00:37:30.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.651+0000 7f26e97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:30.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.651+0000 7f26e97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:37:30.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.651+0000 7f26e97fa640 10 monclient: dump: 2024-04-26T00:37:30.664 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:30.664 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:30.664 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.665 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.665 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:30.665 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:30.665 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:30.665 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:30.665 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:30.665 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:30.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.651+0000 7f26e97fa640 1 -- 172.21.15.37:0/3481568648 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f26f0014640 con 0x7f26f402c000 2024-04-26T00:37:30.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.651+0000 7f26e97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:30.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.651+0000 7f26e97fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:37:30.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.651+0000 7f26e97fa640 1 -- 172.21.15.37:0/3481568648 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26f402c000 msgr2=0x7f26f402c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:30.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.651+0000 7f26e97fa640 1 --2- 172.21.15.37:0/3481568648 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26f402c000 0x7f26f402c3e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f26f0010cb0 tx=0x7f26f000e5a0 comp rx=0 tx=0).stop 2024-04-26T00:37:30.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.651+0000 7f26e97fa640 1 -- 172.21.15.37:0/3481568648 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26f402c000 msgr2=0x7f26f402c3e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:37:30.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.651+0000 7f26e97fa640 1 --2- 172.21.15.37:0/3481568648 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26f402c000 0x7f26f402c3e0 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f26f0010cb0 tx=0x7f26f000e5a0 comp rx=0 tx=0).stop 2024-04-26T00:37:30.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.655+0000 7f26f96de640 1 --2- 172.21.15.37:0/3481568648 >> v2:172.21.15.45:6812/2379918623 conn(0x7f26e0035f00 0x7f26e00383c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f26f4137ec0 tx=0x7f26ec0a7070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:30.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.659+0000 7f26fb141640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:37:30.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.659+0000 7f26fb141640 1 --2- 172.21.15.37:0/3481568648 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26f402c920 0x7f26f402cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:30.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.659+0000 7f26fb141640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:30.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.659+0000 7f26fb141640 1 -- 172.21.15.37:0/3481568648 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f26f40a3000 con 0x7f26f402c920 2024-04-26T00:37:30.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.659+0000 7f26f8edd640 1 --2- 172.21.15.37:0/3481568648 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26f402c920 0x7f26f402cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:30.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.659+0000 7f26f8edd640 10 monclient: get_auth_request con 0x7f26f402c920 auth_method 0 2024-04-26T00:37:30.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.659+0000 7f26f8edd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:30.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.659+0000 7f26f8edd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:30.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.659+0000 7f26f8edd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:30.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.659+0000 7f26f8edd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:30.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.659+0000 7f26f8edd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:30.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.659+0000 7f26f8edd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:30.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.659+0000 7f26f8edd640 10 monclient(hunting): handle_auth_done global_id 204227 payload 824 2024-04-26T00:37:30.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.659+0000 7f26f8edd640 1 -- 172.21.15.37:0/3481568648 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f26f413b6a0 con 0x7f26f402c920 2024-04-26T00:37:30.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.663+0000 7f26f8edd640 1 -- 172.21.15.37:0/3481568648 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26f402c920 msgr2=0x7f26f402cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:30.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.663+0000 7f26f8edd640 1 --2- 172.21.15.37:0/3481568648 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26f402c920 0x7f26f402cd30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f26f0010cb0 tx=0x7f26f00046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:30.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.663+0000 7f26e97fa640 1 -- 172.21.15.37:0/3481568648 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f001daf0 con 0x7f26f402c920 2024-04-26T00:37:30.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.663+0000 7f26e97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:30.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.663+0000 7f26e97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:37:30.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.663+0000 7f26e97fa640 10 monclient: dump: 2024-04-26T00:37:30.669 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:30.669 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:30.669 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.669 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.669 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:30.669 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:30.669 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:30.669 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:30.669 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:30.669 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:30.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.663+0000 7f26e97fa640 1 -- 172.21.15.37:0/3481568648 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4547 (secure 0 0 0) 0x7f26f00273f0 con 0x7f26f402c920 2024-04-26T00:37:30.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.663+0000 7f26e97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:37:30.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.663+0000 7f26e97fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:37:30.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.663+0000 7f26e97fa640 1 -- 172.21.15.37:0/3481568648 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26f402c920 msgr2=0x7f26f402cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:30.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.663+0000 7f26e97fa640 1 --2- 172.21.15.37:0/3481568648 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26f402c920 0x7f26f402cd30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f26f0010cb0 tx=0x7f26f00046d0 comp rx=0 tx=0).stop 2024-04-26T00:37:30.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.663+0000 7f26e97fa640 1 -- 172.21.15.37:0/3481568648 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26f402c920 msgr2=0x7f26f402cd30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:37:30.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.663+0000 7f26e97fa640 1 --2- 172.21.15.37:0/3481568648 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26f402c920 0x7f26f402cd30 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f26f0010cb0 tx=0x7f26f00046d0 comp rx=0 tx=0).stop 2024-04-26T00:37:30.670 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:37:30.670 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:37:30.670 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:37:30.670 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:37:30.670 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 132, 2024-04-26T00:37:30.670 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 56271, 2024-04-26T00:37:30.671 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:37:30.671 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:37:30.671 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:37:30.671 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:37:30.671 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:30.671 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 4, 2024-04-26T00:37:30.671 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:37:30.671 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:37:30.671 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:37:30.671 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:37:30.671 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:37:30.671 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:37:30.671 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:37:30.671 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:37:30.671 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:37:30.672 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:37:30.672 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:37:30.672 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:37:30.672 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:37:30.672 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:37:30.672 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:30.672 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:37:30.672 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:37:30.672 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:37:30.672 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:37:30.672 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:37:30.672 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:37:30.672 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:37:30.672 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:37:30.672 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:37:30.673 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:37:30.673 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:37:30.673 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:37:30.673 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:37:30.673 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:30.673 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:30.673 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:37:30.673 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:37:30.673 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:37:30.673 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:37:30.673 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:37:30.673 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:37:30.673 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:37:30.673 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:37:30.674 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:37:30.674 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:37:30.674 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:37:30.674 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:37:30.674 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:37:30.674 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:37:30.674 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:37:30.674 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:37:30.674 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:37:30.674 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:37:30.674 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:37:30.674 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:37:30.674 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:37:30.674 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:37:30.675 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:37:30.675 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:37:30.675 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:37:30.675 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:37:30.675 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:37:30.675 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:37:30.675 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:30.675 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:37:30.675 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:30.675 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:37:30.675 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:30.675 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:37:30.675 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:37:30.675 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:37:30.675 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:37:30.676 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:30.676 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:37:30.676 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:37:30.676 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:30.676 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:30.676 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:30.676 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:37:30.676 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:37:30.676 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:30.676 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:30.676 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:30.676 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:30.676 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:37:30.676 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:37:30.677 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:37:30.677 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:37:30.677 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:37:30.677 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:30.677 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:30.677 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:37:30.677 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:37:30.677 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:37:30.677 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:37:30.677 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:30.677 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:37:30.677 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:37:30.677 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:30.677 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:30.677 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:30.678 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:37:30.678 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:37:30.678 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:30.678 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:30.678 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:30.678 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:30.678 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:37:30.678 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:37:30.678 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:37:30.678 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:37:30.678 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:37:30.678 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:30.678 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:30.678 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:37:30.679 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:37:30.679 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:37:30.679 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:37:30.679 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:30.679 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:37:30.679 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:37:30.679 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:30.679 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:30.679 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:30.679 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:37:30.679 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:37:30.679 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:30.679 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:30.679 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:30.680 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:30.680 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:37:30.680 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:37:30.680 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:37:30.680 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:37:30.680 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:37:30.680 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:30.680 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:30.680 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:30.680 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:37:30.680 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:37:30.680 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:30.680 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:37:30.680 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:37:30.680 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:37:30.681 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:30.681 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:30.681 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:37:30.681 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:30.681 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:37:30.681 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:37:30.681 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:37:30.681 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:30.681 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:30.681 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:30.681 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:37:30.681 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:37:30.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.663+0000 7f26fb141640 1 -- 172.21.15.37:0/3481568648 >> v2:172.21.15.45:6812/2379918623 conn(0x7f26e0035f00 msgr2=0x7f26e00383c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:30.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.663+0000 7f26fb141640 1 --2- 172.21.15.37:0/3481568648 >> v2:172.21.15.45:6812/2379918623 conn(0x7f26e0035f00 0x7f26e00383c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f26f4137ec0 tx=0x7f26ec0a7070 comp rx=0 tx=0).stop 2024-04-26T00:37:30.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.663+0000 7f26fb141640 10 monclient: shutdown 2024-04-26T00:37:30.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.663+0000 7f26fb141640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:37:30.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.663+0000 7f26fb141640 1 -- 172.21.15.37:0/3481568648 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26f4137300 msgr2=0x7f26f4134f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:30.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.663+0000 7f26fb141640 1 --2- 172.21.15.37:0/3481568648 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26f4137300 0x7f26f4134f60 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f26e400c3e0 tx=0x7f26e400c8a0 comp rx=0 tx=0).stop 2024-04-26T00:37:30.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.663+0000 7f26f96de640 1 -- 172.21.15.37:0/3481568648 reap_dead start 2024-04-26T00:37:30.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.663+0000 7f26f96de640 1 -- 172.21.15.37:0/3481568648 reap_dead start 2024-04-26T00:37:30.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.663+0000 7f26fb141640 1 -- 172.21.15.37:0/3481568648 shutdown_connections 2024-04-26T00:37:30.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.663+0000 7f26fb141640 1 -- 172.21.15.37:0/3481568648 >> 172.21.15.37:0/3481568648 conn(0x7f26f40a61b0 msgr2=0x7f26f40a7de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:37:30.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.663+0000 7f26fb141640 1 -- 172.21.15.37:0/3481568648 shutdown_connections 2024-04-26T00:37:30.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.663+0000 7f26fb141640 1 -- 172.21.15.37:0/3481568648 wait complete. 2024-04-26T00:37:30.684 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:37:30.684 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:37:30.684 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:37:30.768 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:30.767+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:30.768 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:30.767+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=77} crc {osdmap=27624208}) 2024-04-26T00:37:30.768 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:30.767+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=78} crc {osdmap=1650617895}) 2024-04-26T00:37:30.768 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:30.767+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:37:30.768 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:30.767+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=77} crc {osdmap=27624208}) 2024-04-26T00:37:30.768 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:37:30.767+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=74} crc {osdmap=3549615301}) 2024-04-26T00:37:30.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.803+0000 7f6fabb0c640 10 monclient: get_monmap_and_config 2024-04-26T00:37:30.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.803+0000 7f6fabb0c640 10 monclient: build_initial_monmap 2024-04-26T00:37:30.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.803+0000 7f6fabb0c640 10 monclient: monmap: 2024-04-26T00:37:30.807 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:37:30.807 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:30.807 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:37:30.811777+0000 2024-04-26T00:37:30.807 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:37:30.811777+0000 2024-04-26T00:37:30.807 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:37:30.808 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:30.808 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:37:30.808 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:37:30.808 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:37:30.808 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:37:30.808 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:37:30.808 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:30.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.807+0000 7f6fabb0c640 1 Processor -- start 2024-04-26T00:37:30.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.807+0000 7f6fabb0c640 1 -- start start 2024-04-26T00:37:30.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.807+0000 7f6fabb0c640 10 monclient: init 2024-04-26T00:37:30.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.807+0000 7f6fabb0c640 10 monclient: _reopen_session rank -1 2024-04-26T00:37:30.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.807+0000 7f6fabb0c640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-04-26T00:37:30.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.807+0000 7f6fabb0c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6fa40726b0 0x7f6fa4072a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:30.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.807+0000 7f6fabb0c640 10 monclient(hunting): picked mon.noname-b con 0x7f6fa40726b0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:37:30.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.807+0000 7f6fabb0c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6fa40a0110 addr v1:172.21.15.45:6789/0 2024-04-26T00:37:30.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.807+0000 7f6fabb0c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6fa41374e0 addr v1:172.21.15.37:6789/0 2024-04-26T00:37:30.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.807+0000 7f6fabb0c640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fa40a1470 con 0x7f6fa41374e0 2024-04-26T00:37:30.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.807+0000 7f6fabb0c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fa40a0bd0 con 0x7f6fa40a0110 2024-04-26T00:37:30.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.807+0000 7f6fabb0c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:30.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.807+0000 7f6fabb0c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f6fa40a1bd0 con 0x7f6fa40726b0 2024-04-26T00:37:30.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.807+0000 7f6fabb0c640 10 monclient(hunting): _renew_subs 2024-04-26T00:37:30.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.807+0000 7f6fabb0c640 10 monclient(hunting): authenticate will time out at 2032.117644s 2024-04-26T00:37:30.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.807+0000 7f6faa0a9640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f6fa41374e0 0x7f6fa413f110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:37262/0 (socket says 172.21.15.37:37262) 2024-04-26T00:37:30.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.807+0000 7f6faa0a9640 1 -- 172.21.15.37:0/2463136581 learned_addr learned my addr 172.21.15.37:0/2463136581 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:37:30.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.807+0000 7f6fa98a8640 1 --2- 172.21.15.37:0/2463136581 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6fa40726b0 0x7f6fa4072a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:30.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.807+0000 7f6fa98a8640 10 monclient(hunting): get_auth_request con 0x7f6fa40726b0 auth_method 0 2024-04-26T00:37:30.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.807+0000 7f6fa98a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:30.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.807+0000 7f6fa98a8640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:30.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.807+0000 7f6fa98a8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:30.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.807+0000 7f6fa98a8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:30.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.807+0000 7f6fa98a8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:30.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.807+0000 7f6fa98a8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:30.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.807+0000 7f6fa98a8640 10 monclient(hunting): handle_auth_done global_id 204230 payload 274 2024-04-26T00:37:30.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.807+0000 7f6fa98a8640 1 -- 172.21.15.37:0/2463136581 >> v1:172.21.15.45:6789/0 conn(0x7f6fa40a0110 legacy=0x7f6fa4136dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:37:30.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.807+0000 7f6fa98a8640 1 -- 172.21.15.37:0/2463136581 >> v1:172.21.15.37:6789/0 conn(0x7f6fa41374e0 legacy=0x7f6fa413f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:37:30.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.807+0000 7f6fa98a8640 10 monclient: _finish_hunting 0 2024-04-26T00:37:30.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.807+0000 7f6fa98a8640 1 monclient: found mon.noname-b 2024-04-26T00:37:30.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.807+0000 7f6fa98a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:37:30.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.807+0000 7f6fa98a8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:37:30.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.807+0000 7f6fa98a8640 1 -- 172.21.15.37:0/2463136581 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fa40a1190 con 0x7f6fa40726b0 2024-04-26T00:37:30.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fa98a8640 1 -- 172.21.15.37:0/2463136581 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6fa40726b0 msgr2=0x7f6fa4072a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:30.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fa98a8640 1 --2- 172.21.15.37:0/2463136581 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6fa40726b0 0x7f6fa4072a90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6fa00097f0 tx=0x7f6fa001c140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1de34b86f35a4a55 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:30.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fa88a6640 1 -- 172.21.15.37:0/2463136581 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa00b1070 con 0x7f6fa40726b0 2024-04-26T00:37:30.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fa88a6640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:30.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fa88a6640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:37:30.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fa88a6640 10 monclient: dump: 2024-04-26T00:37:30.816 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:30.816 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:30.817 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.817 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.817 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:30.817 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:30.817 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:30.817 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:30.817 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:30.817 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:30.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fa88a6640 10 monclient: _finish_auth 0 2024-04-26T00:37:30.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fa88a6640 10 monclient: _check_auth_tickets 2024-04-26T00:37:30.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fa88a6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:37:30.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fa88a6640 1 -- 172.21.15.37:0/2463136581 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fa00ac440 con 0x7f6fa40726b0 2024-04-26T00:37:30.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fabb0c640 5 monclient: authenticate success, global_id 204230 2024-04-26T00:37:30.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fabb0c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:37:30.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fa88a6640 10 monclient: handle_config config(0 keys) 2024-04-26T00:37:30.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fabb0c640 10 monclient: get_monmap_and_config success 2024-04-26T00:37:30.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fa88a6640 1 -- 172.21.15.37:0/2463136581 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa00ac810 con 0x7f6fa40726b0 2024-04-26T00:37:30.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fabb0c640 4 set_mon_vals no callback set 2024-04-26T00:37:30.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fa88a6640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:30.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fa88a6640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:37:30.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fa88a6640 10 monclient: dump: 2024-04-26T00:37:30.818 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:30.818 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:30.818 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.818 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.818 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:30.819 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:30.819 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:30.819 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:30.819 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:30.819 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:30.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fabb0c640 10 monclient: shutdown 2024-04-26T00:37:30.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fabb0c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:37:30.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fabb0c640 1 -- 172.21.15.37:0/2463136581 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6fa40726b0 msgr2=0x7f6fa4072a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:30.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fabb0c640 1 --2- 172.21.15.37:0/2463136581 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6fa40726b0 0x7f6fa4072a90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6fa00097f0 tx=0x7f6fa001c140 comp rx=0 tx=0).stop 2024-04-26T00:37:30.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fabb0c640 1 -- 172.21.15.37:0/2463136581 shutdown_connections 2024-04-26T00:37:30.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fabb0c640 1 --2- 172.21.15.37:0/2463136581 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6fa40726b0 0x7f6fa4072a90 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:30.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fabb0c640 1 -- 172.21.15.37:0/2463136581 >> 172.21.15.37:0/2463136581 conn(0x7f6fa40a1d50 msgr2=0x7f6fa40a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:37:30.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fabb0c640 1 -- 172.21.15.37:0/2463136581 shutdown_connections 2024-04-26T00:37:30.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fabb0c640 1 -- 172.21.15.37:0/2463136581 wait complete. 2024-04-26T00:37:30.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fabb0c640 10 monclient: build_initial_monmap 2024-04-26T00:37:30.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fabb0c640 10 monclient: monmap: 2024-04-26T00:37:30.828 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:37:30.828 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:37:30.828 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:37:30.828 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:37:30.828 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:37:30.828 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:37:30.828 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:37:30.828 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:37:30.828 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:37:30.828 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:30.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fabb0c640 1 Processor -- start 2024-04-26T00:37:30.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fabb0c640 1 -- start start 2024-04-26T00:37:30.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fabb0c640 10 monclient: init 2024-04-26T00:37:30.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fabb0c640 10 monclient: _reopen_session rank -1 2024-04-26T00:37:30.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fabb0c640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:37:30.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fabb0c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6fa40a0110 0x7f6fa4131070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:30.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fabb0c640 10 monclient(hunting): picked mon.noname-a con 0x7f6fa40a0110 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:37:30.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fabb0c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fa41374e0 0x7f6fa41315b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:30.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fabb0c640 10 monclient(hunting): picked mon.noname-b con 0x7f6fa41374e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:37:30.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fabb0c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6fa413a990 0x7f6fa41cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:30.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fabb0c640 10 monclient(hunting): picked mon.noname-c con 0x7f6fa413a990 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:37:30.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fabb0c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:30.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fabb0c640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa41435e0 con 0x7f6fa40a0110 2024-04-26T00:37:30.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fabb0c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:30.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fabb0c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa4143460 con 0x7f6fa41374e0 2024-04-26T00:37:30.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fabb0c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:30.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fabb0c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f6fa4143760 con 0x7f6fa413a990 2024-04-26T00:37:30.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fabb0c640 10 monclient(hunting): _renew_subs 2024-04-26T00:37:30.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fabb0c640 10 monclient(hunting): authenticate will time out at 2032.126621s 2024-04-26T00:37:30.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6faa0a9640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6fa413a990 0x7f6fa41cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:30.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fa98a8640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6fa40a0110 0x7f6fa4131070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:30.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6faa0a9640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6fa413a990 0x7f6fa41cf8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:59132/0 (socket says 172.21.15.37:59132) 2024-04-26T00:37:30.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6faa0a9640 1 -- 172.21.15.37:0/1256748268 learned_addr learned my addr 172.21.15.37:0/1256748268 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:37:30.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6faa0a9640 10 monclient(hunting): get_auth_request con 0x7f6fa413a990 auth_method 0 2024-04-26T00:37:30.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6faa0a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:30.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6faa0a9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:30.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6faa0a9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:30.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fa98a8640 10 monclient(hunting): get_auth_request con 0x7f6fa40a0110 auth_method 0 2024-04-26T00:37:30.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fa98a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:30.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fa98a8640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:30.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fa98a8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:30.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6faa0a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:30.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6faa0a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:30.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6faa0a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:30.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fa98a8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:30.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fa98a8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:30.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fa98a8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:30.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fa90a7640 1 --2- 172.21.15.37:0/1256748268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fa41374e0 0x7f6fa41315b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:30.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fa90a7640 10 monclient(hunting): get_auth_request con 0x7f6fa41374e0 auth_method 0 2024-04-26T00:37:30.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fa90a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:30.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fa90a7640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:30.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fa90a7640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:30.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6faa0a9640 10 monclient(hunting): handle_auth_done global_id 204233 payload 824 2024-04-26T00:37:30.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6faa0a9640 1 -- 172.21.15.37:0/1256748268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fa41374e0 msgr2=0x7f6fa41315b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:30.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6faa0a9640 1 --2- 172.21.15.37:0/1256748268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fa41374e0 0x7f6fa41315b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:30.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6faa0a9640 1 -- 172.21.15.37:0/1256748268 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6fa40a0110 msgr2=0x7f6fa4131070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:30.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6faa0a9640 1 --2- 172.21.15.37:0/1256748268 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6fa40a0110 0x7f6fa4131070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:30.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6faa0a9640 10 monclient: _finish_hunting 0 2024-04-26T00:37:30.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6faa0a9640 1 monclient: found mon.noname-c 2024-04-26T00:37:30.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6faa0a9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:37:30.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6faa0a9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:37:30.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6faa0a9640 1 -- 172.21.15.37:0/1256748268 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fa413c5e0 con 0x7f6fa413a990 2024-04-26T00:37:30.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.815+0000 7f6fa98a8640 1 --2- 172.21.15.37:0/1256748268 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6fa40a0110 0x7f6fa4131070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:37:30.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6faa0a9640 1 -- 172.21.15.37:0/1256748268 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6fa413a990 msgr2=0x7f6fa41cf8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:30.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6faa0a9640 1 --2- 172.21.15.37:0/1256748268 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6fa413a990 0x7f6fa41cf8f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c00cb20 tx=0x7f6f9c00a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:30.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6f9a7fc640 1 -- 172.21.15.37:0/1256748268 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c00ae50 con 0x7f6fa413a990 2024-04-26T00:37:30.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6f9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:30.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6f9a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:37:30.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6f9a7fc640 10 monclient: dump: 2024-04-26T00:37:30.833 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:30.833 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:30.833 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.833 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.834 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:30.834 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:30.834 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:30.834 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:30.834 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:30.834 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:30.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6f9a7fc640 10 monclient: _finish_auth 0 2024-04-26T00:37:30.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6f9a7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:37:30.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6f9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:37:30.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6f9a7fc640 1 -- 172.21.15.37:0/1256748268 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f9c012400 con 0x7f6fa413a990 2024-04-26T00:37:30.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6fabb0c640 5 monclient: authenticate success, global_id 204233 2024-04-26T00:37:30.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6fabb0c640 10 monclient: _renew_subs 2024-04-26T00:37:30.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6fabb0c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:37:30.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6fabb0c640 1 -- 172.21.15.37:0/1256748268 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fa40a05d0 con 0x7f6fa413a990 2024-04-26T00:37:30.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6f9a7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:37:30.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6f9a7fc640 1 -- 172.21.15.37:0/1256748268 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c0127d0 con 0x7f6fa413a990 2024-04-26T00:37:30.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6f9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:30.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6f9a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:37:30.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6f9a7fc640 10 monclient: dump: 2024-04-26T00:37:30.835 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:30.835 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:30.835 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.835 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.835 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:30.835 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:30.835 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:30.835 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:30.835 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:30.836 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:30.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6f9b7fe640 4 set_mon_vals no callback set 2024-04-26T00:37:30.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6fabb0c640 10 monclient: _renew_subs 2024-04-26T00:37:30.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6fabb0c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:37:30.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6fabb0c640 1 -- 172.21.15.37:0/1256748268 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fa413e030 con 0x7f6fa413a990 2024-04-26T00:37:30.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6fabb0c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:30.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6fabb0c640 1 --2- 172.21.15.37:0/1256748268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fa41cfe30 0x7f6fa41d0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:30.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6fabb0c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:30.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6fabb0c640 1 -- 172.21.15.37:0/1256748268 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa41d0750 con 0x7f6fa41cfe30 2024-04-26T00:37:30.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6f9a7fc640 1 -- 172.21.15.37:0/1256748268 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f6f9c012970 con 0x7f6fa413a990 2024-04-26T00:37:30.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6fa98a8640 1 --2- 172.21.15.37:0/1256748268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fa41cfe30 0x7f6fa41d0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:30.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6f9a7fc640 1 --2- 172.21.15.37:0/1256748268 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6f88036070 0x7f6f88038530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:30.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6fa98a8640 10 monclient: get_auth_request con 0x7f6fa41cfe30 auth_method 0 2024-04-26T00:37:30.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6fa98a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:30.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6fa98a8640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:30.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6fa98a8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:30.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6f9a7fc640 1 -- 172.21.15.37:0/1256748268 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(691..691 src has 370..691) ==== 3927+0+0 (secure 0 0 0) 0x7f6f9c049110 con 0x7f6fa413a990 2024-04-26T00:37:30.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6fa98a8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:30.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6fa98a8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:30.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6fa98a8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:30.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6fa90a7640 1 --2- 172.21.15.37:0/1256748268 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6f88036070 0x7f6f88038530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:30.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6fa90a7640 10 monclient: get_auth_request con 0x7f6f88036070 auth_method 0 2024-04-26T00:37:30.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6fa98a8640 10 monclient(hunting): handle_auth_done global_id 214120 payload 824 2024-04-26T00:37:30.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.819+0000 7f6fa98a8640 1 -- 172.21.15.37:0/1256748268 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6fa404f510 con 0x7f6fa41cfe30 2024-04-26T00:37:30.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.823+0000 7f6fa98a8640 1 -- 172.21.15.37:0/1256748268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fa41cfe30 msgr2=0x7f6fa41d0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:30.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.823+0000 7f6fa98a8640 1 --2- 172.21.15.37:0/1256748268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fa41cfe30 0x7f6fa41d0210 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6fa406dbf0 tx=0x7f6fa0018ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:30.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.823+0000 7f6fa90a7640 1 --2- 172.21.15.37:0/1256748268 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6f88036070 0x7f6f88038530 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f6fa4136bf0 tx=0x7f6f94009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:30.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.823+0000 7f6f9a7fc640 1 -- 172.21.15.37:0/1256748268 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa00bb030 con 0x7f6fa41cfe30 2024-04-26T00:37:30.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.823+0000 7f6f9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:30.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.823+0000 7f6f9a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:37:30.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.823+0000 7f6f9a7fc640 10 monclient: dump: 2024-04-26T00:37:30.838 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:30.838 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:30.838 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.838 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.838 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:30.838 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:30.838 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:30.839 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:30.839 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:30.839 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:30.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.823+0000 7f6f9a7fc640 1 -- 172.21.15.37:0/1256748268 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6fa00b7740 con 0x7f6fa41cfe30 2024-04-26T00:37:30.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.823+0000 7f6f9a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:30.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.823+0000 7f6f9a7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:37:30.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.823+0000 7f6f9a7fc640 1 -- 172.21.15.37:0/1256748268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fa41cfe30 msgr2=0x7f6fa41d0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:30.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.823+0000 7f6f9a7fc640 1 --2- 172.21.15.37:0/1256748268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fa41cfe30 0x7f6fa41d0210 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6fa406dbf0 tx=0x7f6fa0018ee0 comp rx=0 tx=0).stop 2024-04-26T00:37:30.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.823+0000 7f6f9a7fc640 1 -- 172.21.15.37:0/1256748268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fa41cfe30 msgr2=0x7f6fa41d0210 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:37:30.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.823+0000 7f6f9a7fc640 1 --2- 172.21.15.37:0/1256748268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fa41cfe30 0x7f6fa41d0210 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6fa406dbf0 tx=0x7f6fa0018ee0 comp rx=0 tx=0).stop 2024-04-26T00:37:30.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.831+0000 7f6fabb0c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:37:30.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.831+0000 7f6fabb0c640 1 --2- 172.21.15.37:0/1256748268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fa4135190 0x7f6fa4135570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:30.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.831+0000 7f6fabb0c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:30.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.831+0000 7f6fabb0c640 1 -- 172.21.15.37:0/1256748268 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa406ba50 con 0x7f6fa4135190 2024-04-26T00:37:30.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.831+0000 7f6fa98a8640 1 --2- 172.21.15.37:0/1256748268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fa4135190 0x7f6fa4135570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:30.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.831+0000 7f6fa98a8640 10 monclient: get_auth_request con 0x7f6fa4135190 auth_method 0 2024-04-26T00:37:30.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.831+0000 7f6fa98a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:30.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.831+0000 7f6fa98a8640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:30.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.831+0000 7f6fa98a8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:30.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.831+0000 7f6fa98a8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:30.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.831+0000 7f6fa98a8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:30.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.831+0000 7f6fa98a8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:30.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.831+0000 7f6fa98a8640 10 monclient(hunting): handle_auth_done global_id 214123 payload 824 2024-04-26T00:37:30.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.831+0000 7f6fa98a8640 1 -- 172.21.15.37:0/1256748268 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6fa413f850 con 0x7f6fa4135190 2024-04-26T00:37:30.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.835+0000 7f6fa98a8640 1 -- 172.21.15.37:0/1256748268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fa4135190 msgr2=0x7f6fa4135570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:30.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.835+0000 7f6fa98a8640 1 --2- 172.21.15.37:0/1256748268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fa4135190 0x7f6fa4135570 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6fa001c9e0 tx=0x7f6fa00a6000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:30.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.835+0000 7f6f9a7fc640 1 -- 172.21.15.37:0/1256748268 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa00bb070 con 0x7f6fa4135190 2024-04-26T00:37:30.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.835+0000 7f6f9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:30.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.835+0000 7f6f9a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:37:30.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.835+0000 7f6f9a7fc640 10 monclient: dump: 2024-04-26T00:37:30.843 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:30.843 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:30.843 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.843 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:30.843 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:30.843 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:30.843 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:30.843 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:30.843 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:30.843 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:30.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.835+0000 7f6f9a7fc640 1 -- 172.21.15.37:0/1256748268 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7f6fa00b7e10 con 0x7f6fa4135190 2024-04-26T00:37:30.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.835+0000 7f6f9a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:37:30.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.835+0000 7f6f9a7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:37:30.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.835+0000 7f6f9a7fc640 1 -- 172.21.15.37:0/1256748268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fa4135190 msgr2=0x7f6fa4135570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:30.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.835+0000 7f6f9a7fc640 1 --2- 172.21.15.37:0/1256748268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fa4135190 0x7f6fa4135570 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6fa001c9e0 tx=0x7f6fa00a6000 comp rx=0 tx=0).stop 2024-04-26T00:37:30.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.835+0000 7f6f9a7fc640 1 -- 172.21.15.37:0/1256748268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fa4135190 msgr2=0x7f6fa4135570 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:37:30.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.835+0000 7f6f9a7fc640 1 --2- 172.21.15.37:0/1256748268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fa4135190 0x7f6fa4135570 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6fa001c9e0 tx=0x7f6fa00a6000 comp rx=0 tx=0).stop 2024-04-26T00:37:30.844 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:37:30.844 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:37:30.844 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:37:30.844 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:37:30.844 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 132, 2024-04-26T00:37:30.844 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 7943, 2024-04-26T00:37:30.845 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:37:30.845 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:37:30.845 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:37:30.845 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:37:30.845 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:30.845 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 4, 2024-04-26T00:37:30.845 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:37:30.845 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:37:30.845 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:37:30.845 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:37:30.845 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:37:30.845 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:37:30.845 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:37:30.846 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:37:30.846 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:37:30.846 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:37:30.846 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:37:30.846 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:37:30.846 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:37:30.846 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:37:30.846 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:30.846 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:37:30.847 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:37:30.847 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:37:30.847 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:37:30.847 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:37:30.847 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:37:30.847 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:37:30.847 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:37:30.847 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:37:30.847 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:37:30.847 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:37:30.847 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:37:30.847 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:37:30.847 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:30.847 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:30.847 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:37:30.848 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:37:30.848 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:37:30.848 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:37:30.848 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:37:30.848 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:37:30.848 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:37:30.848 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:37:30.848 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:37:30.848 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:37:30.848 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:37:30.848 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:37:30.848 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:37:30.848 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:37:30.848 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:37:30.849 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:37:30.849 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:37:30.849 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:37:30.849 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:37:30.849 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:37:30.849 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:37:30.849 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:37:30.849 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:37:30.849 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:37:30.850 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:37:30.850 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:37:30.850 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:37:30.850 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:37:30.850 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:30.850 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:37:30.850 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:30.850 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:37:30.850 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:30.851 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:37:30.851 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:37:30.851 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:37:30.852 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:37:30.852 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:30.852 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:37:30.852 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:37:30.852 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:30.852 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:30.852 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:30.852 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:37:30.852 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:37:30.852 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:30.852 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:30.852 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:30.853 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:30.853 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:37:30.853 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:37:30.853 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:37:30.853 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:37:30.853 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:37:30.853 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:30.853 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:30.853 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:37:30.853 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:37:30.853 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:37:30.853 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:37:30.854 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:30.854 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:37:30.854 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:37:30.854 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:30.854 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:30.854 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:30.854 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:37:30.854 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:37:30.854 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:30.854 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:30.854 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:30.854 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:30.854 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:37:30.854 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:37:30.855 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:37:30.855 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:37:30.855 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:37:30.855 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:30.855 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:30.855 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:37:30.855 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:37:30.855 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:37:30.855 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:37:30.855 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:30.855 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:37:30.855 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:37:30.855 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:30.855 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:30.855 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:30.856 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:37:30.856 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:37:30.856 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:30.856 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:30.856 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:30.856 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:30.856 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:37:30.856 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:37:30.856 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:37:30.856 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:37:30.856 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:37:30.856 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:30.856 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:30.856 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:30.856 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:37:30.857 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:37:30.857 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:30.857 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:37:30.857 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:37:30.857 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:37:30.857 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:30.857 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:30.857 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:37:30.857 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:30.857 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:37:30.857 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:37:30.857 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:37:30.857 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:30.857 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:30.858 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:30.858 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:37:30.858 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:37:30.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.835+0000 7f6fabb0c640 1 -- 172.21.15.37:0/1256748268 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6f88036070 msgr2=0x7f6f88038530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:30.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.835+0000 7f6fabb0c640 1 --2- 172.21.15.37:0/1256748268 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6f88036070 0x7f6f88038530 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f6fa4136bf0 tx=0x7f6f94009040 comp rx=0 tx=0).stop 2024-04-26T00:37:30.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.835+0000 7f6fabb0c640 10 monclient: shutdown 2024-04-26T00:37:30.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.835+0000 7f6fabb0c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:37:30.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.835+0000 7f6fabb0c640 1 -- 172.21.15.37:0/1256748268 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6fa413a990 msgr2=0x7f6fa41cf8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:30.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.835+0000 7f6fabb0c640 1 --2- 172.21.15.37:0/1256748268 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6fa413a990 0x7f6fa41cf8f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c00cb20 tx=0x7f6f9c00a060 comp rx=0 tx=0).stop 2024-04-26T00:37:30.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.835+0000 7f6faa0a9640 1 -- 172.21.15.37:0/1256748268 reap_dead start 2024-04-26T00:37:30.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.835+0000 7f6fabb0c640 1 -- 172.21.15.37:0/1256748268 shutdown_connections 2024-04-26T00:37:30.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.835+0000 7f6fabb0c640 1 --2- 172.21.15.37:0/1256748268 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6fa413a990 0x7f6fa41cf8f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:30.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.835+0000 7f6fabb0c640 1 -- 172.21.15.37:0/1256748268 >> 172.21.15.37:0/1256748268 conn(0x7f6fa40a1d50 msgr2=0x7f6fa409d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:37:30.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.835+0000 7f6fabb0c640 1 -- 172.21.15.37:0/1256748268 shutdown_connections 2024-04-26T00:37:30.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.835+0000 7f6fabb0c640 1 -- 172.21.15.37:0/1256748268 wait complete. 2024-04-26T00:37:30.861 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:37:30.861 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-04-26T00:37:30.861 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-26T00:37:30.862 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-04-26T00:37:30.862 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-04-26T00:37:30.862 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-26T00:37:30.862 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T00:37:30.878 INFO:tasks.ceph.mon.b:Stopped 2024-04-26T00:37:30.878 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-26T00:37:30.879 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:37:31.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.999+0000 7f3000c6b640 10 monclient: get_monmap_and_config 2024-04-26T00:37:31.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.999+0000 7f3000c6b640 10 monclient: build_initial_monmap 2024-04-26T00:37:31.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.999+0000 7f3000c6b640 10 monclient: monmap: 2024-04-26T00:37:31.002 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:37:31.002 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:31.002 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:37:31.006388+0000 2024-04-26T00:37:31.002 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:37:31.006388+0000 2024-04-26T00:37:31.002 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:37:31.002 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:31.002 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:37:31.002 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:37:31.002 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:37:31.003 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:37:31.003 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:37:31.003 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:31.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.999+0000 7f3000c6b640 1 Processor -- start 2024-04-26T00:37:31.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.999+0000 7f3000c6b640 1 -- start start 2024-04-26T00:37:31.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:30.999+0000 7f3000c6b640 10 monclient: init 2024-04-26T00:37:31.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.003+0000 7f3000c6b640 10 monclient: _reopen_session rank -1 2024-04-26T00:37:31.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.003+0000 7f3000c6b640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-04-26T00:37:31.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.003+0000 7f3000c6b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2ffc071fc0 addr v1:172.21.15.37:6789/0 2024-04-26T00:37:31.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.003+0000 7f3000c6b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ffc0723a0 0x7f2ffc0727e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:31.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.003+0000 7f3000c6b640 10 monclient(hunting): picked mon.noname-b con 0x7f2ffc0723a0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:37:31.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.003+0000 7f3000c6b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2ffc13c340 addr v1:172.21.15.45:6789/0 2024-04-26T00:37:31.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.003+0000 7f3000c6b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ffc073200 con 0x7f2ffc071fc0 2024-04-26T00:37:31.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.003+0000 7f3000c6b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ffc0a0c20 con 0x7f2ffc13c340 2024-04-26T00:37:31.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.003+0000 7f3000c6b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:31.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.003+0000 7f3000c6b640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2ffc0a0f00 con 0x7f2ffc0723a0 2024-04-26T00:37:31.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.003+0000 7f3000c6b640 10 monclient(hunting): _renew_subs 2024-04-26T00:37:31.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.003+0000 7f3000c6b640 10 monclient(hunting): authenticate will time out at 2032.312347s 2024-04-26T00:37:31.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.003+0000 7f2ff9d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ffc0723a0 0x7f2ffc0727e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:31.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.003+0000 7f2ffa59c640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f2ffc071fc0 0x7f2ffc137240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:37278/0 (socket says 172.21.15.37:37278) 2024-04-26T00:37:31.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.003+0000 7f2ffa59c640 1 -- 172.21.15.37:0/2943266974 learned_addr learned my addr 172.21.15.37:0/2943266974 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:37:31.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.003+0000 7f2ff9d9b640 10 monclient(hunting): get_auth_request con 0x7f2ffc0723a0 auth_method 0 2024-04-26T00:37:31.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.003+0000 7f2ff9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:31.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.003+0000 7f2ff9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:31.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.003+0000 7f2ff9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:31.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.003+0000 7f2ffad9d640 1 -- 172.21.15.37:0/2943266974 >> v1:172.21.15.45:6789/0 conn(0x7f2ffc13c340 legacy=0x7f2ffc13e730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:37:31.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.003+0000 7f2ff9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:31.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.003+0000 7f2ff9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:31.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.003+0000 7f2ff9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:31.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.003+0000 7f2ff9d9b640 10 monclient(hunting): handle_auth_done global_id 204236 payload 274 2024-04-26T00:37:31.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.003+0000 7f2ff9d9b640 1 -- 172.21.15.37:0/2943266974 >> v1:172.21.15.45:6789/0 conn(0x7f2ffc13c340 legacy=0x7f2ffc13e730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:37:31.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.003+0000 7f2ff9d9b640 1 -- 172.21.15.37:0/2943266974 >> v1:172.21.15.37:6789/0 conn(0x7f2ffc071fc0 legacy=0x7f2ffc137240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:37:31.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.003+0000 7f2ff9d9b640 10 monclient: _finish_hunting 0 2024-04-26T00:37:31.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.003+0000 7f2ff9d9b640 1 monclient: found mon.noname-b 2024-04-26T00:37:31.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.003+0000 7f2ff9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:37:31.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.003+0000 7f2ff9d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:37:31.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.003+0000 7f2ff9d9b640 1 -- 172.21.15.37:0/2943266974 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ffc137950 con 0x7f2ffc0723a0 2024-04-26T00:37:31.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.007+0000 7f2ff9d9b640 1 -- 172.21.15.37:0/2943266974 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ffc0723a0 msgr2=0x7f2ffc0727e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:31.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.007+0000 7f2ff9d9b640 1 --2- 172.21.15.37:0/2943266974 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ffc0723a0 0x7f2ffc0727e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2fec009dc0 tx=0x7f2fec01c7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c162965959725794 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:31.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.007+0000 7f2ff959a640 1 -- 172.21.15.37:0/2943266974 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fec0b2070 con 0x7f2ffc0723a0 2024-04-26T00:37:31.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.007+0000 7f2ff959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:31.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.007+0000 7f2ff959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:37:31.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.007+0000 7f2ff959a640 10 monclient: dump: 2024-04-26T00:37:31.009 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:31.010 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:31.010 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:31.010 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:31.010 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:31.010 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:31.010 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:31.010 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:31.010 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:31.010 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:31.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.007+0000 7f2ff959a640 10 monclient: _finish_auth 0 2024-04-26T00:37:31.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.007+0000 7f2ff959a640 10 monclient: _check_auth_tickets 2024-04-26T00:37:31.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.007+0000 7f2ff959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:37:31.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.007+0000 7f2ff959a640 1 -- 172.21.15.37:0/2943266974 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fec0a5840 con 0x7f2ffc0723a0 2024-04-26T00:37:31.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.007+0000 7f2ff959a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:37:31.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.007+0000 7f2ff959a640 1 -- 172.21.15.37:0/2943266974 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fec0a5c10 con 0x7f2ffc0723a0 2024-04-26T00:37:31.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.007+0000 7f2ff959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:31.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.007+0000 7f2ff959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:37:31.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.007+0000 7f2ff959a640 10 monclient: dump: 2024-04-26T00:37:31.011 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:31.011 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:31.011 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:31.011 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:31.011 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:31.011 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:31.011 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:31.011 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:31.011 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:31.011 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:31.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.007+0000 7f3000c6b640 5 monclient: authenticate success, global_id 204236 2024-04-26T00:37:31.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.007+0000 7f3000c6b640 10 monclient: get_monmap_and_config success 2024-04-26T00:37:31.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.007+0000 7f3000c6b640 4 set_mon_vals no callback set 2024-04-26T00:37:31.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.007+0000 7f3000c6b640 10 monclient: shutdown 2024-04-26T00:37:31.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.007+0000 7f3000c6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:37:31.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.007+0000 7f3000c6b640 1 -- 172.21.15.37:0/2943266974 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ffc0723a0 msgr2=0x7f2ffc0727e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:31.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.007+0000 7f3000c6b640 1 --2- 172.21.15.37:0/2943266974 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ffc0723a0 0x7f2ffc0727e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2fec009dc0 tx=0x7f2fec01c7d0 comp rx=0 tx=0).stop 2024-04-26T00:37:31.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.007+0000 7f3000c6b640 1 -- 172.21.15.37:0/2943266974 shutdown_connections 2024-04-26T00:37:31.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.007+0000 7f3000c6b640 1 --2- 172.21.15.37:0/2943266974 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ffc0723a0 0x7f2ffc0727e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:31.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.007+0000 7f3000c6b640 1 -- 172.21.15.37:0/2943266974 >> 172.21.15.37:0/2943266974 conn(0x7f2ffc0a1ad0 msgr2=0x7f2ffc0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:37:31.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.007+0000 7f3000c6b640 1 -- 172.21.15.37:0/2943266974 shutdown_connections 2024-04-26T00:37:31.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.007+0000 7f3000c6b640 1 -- 172.21.15.37:0/2943266974 wait complete. 2024-04-26T00:37:31.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f3000c6b640 10 monclient: build_initial_monmap 2024-04-26T00:37:31.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f3000c6b640 10 monclient: monmap: 2024-04-26T00:37:31.018 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:37:31.018 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:37:31.018 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:37:31.018 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:37:31.018 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:37:31.018 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:37:31.018 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:37:31.018 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:37:31.018 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:37:31.018 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:31.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f3000c6b640 1 Processor -- start 2024-04-26T00:37:31.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f3000c6b640 1 -- start start 2024-04-26T00:37:31.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f3000c6b640 10 monclient: init 2024-04-26T00:37:31.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f3000c6b640 10 monclient: _reopen_session rank -1 2024-04-26T00:37:31.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f3000c6b640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:37:31.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f3000c6b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ffc071fc0 0x7f2ffc1cf020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:31.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f3000c6b640 10 monclient(hunting): picked mon.noname-c con 0x7f2ffc071fc0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:37:31.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f3000c6b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ffc0723a0 0x7f2ffc1cf560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:31.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f3000c6b640 10 monclient(hunting): picked mon.noname-b con 0x7f2ffc0723a0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:37:31.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f3000c6b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ffc13c340 0x7f2ffc1cfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:31.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f3000c6b640 10 monclient(hunting): picked mon.noname-a con 0x7f2ffc13c340 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:37:31.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f3000c6b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:31.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f3000c6b640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f2ffc142a00 con 0x7f2ffc13c340 2024-04-26T00:37:31.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f3000c6b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:31.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f3000c6b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2ffc142880 con 0x7f2ffc0723a0 2024-04-26T00:37:31.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f3000c6b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:31.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f3000c6b640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2ffc142b80 con 0x7f2ffc071fc0 2024-04-26T00:37:31.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f3000c6b640 10 monclient(hunting): _renew_subs 2024-04-26T00:37:31.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f3000c6b640 10 monclient(hunting): authenticate will time out at 2032.321416s 2024-04-26T00:37:31.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f2ffa59c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ffc071fc0 0x7f2ffc1cf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:31.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f2ffa59c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ffc071fc0 0x7f2ffc1cf020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:59154/0 (socket says 172.21.15.37:59154) 2024-04-26T00:37:31.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f2ffa59c640 1 -- 172.21.15.37:0/4141324549 learned_addr learned my addr 172.21.15.37:0/4141324549 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:37:31.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f2ffa59c640 10 monclient(hunting): get_auth_request con 0x7f2ffc071fc0 auth_method 0 2024-04-26T00:37:31.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f2ffa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:31.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f2ffa59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:31.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f2ffa59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:31.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f2ffad9d640 1 --2- 172.21.15.37:0/4141324549 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ffc13c340 0x7f2ffc1cfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:31.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f2ff9d9b640 1 -- 172.21.15.37:0/4141324549 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ffc0723a0 msgr2=0x7f2ffc1cf560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:37:31.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f2ffa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:31.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f2ffa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:31.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f2ff9d9b640 1 --2- 172.21.15.37:0/4141324549 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ffc0723a0 0x7f2ffc1cf560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:37:31.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f2ffa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:31.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f2ffad9d640 10 monclient(hunting): get_auth_request con 0x7f2ffc13c340 auth_method 0 2024-04-26T00:37:31.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f2ffad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:31.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f2ffad9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:31.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f2ffad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:31.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f2ffad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:31.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f2ffad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:31.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f2ffad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:31.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f2ffa59c640 10 monclient(hunting): handle_auth_done global_id 204239 payload 824 2024-04-26T00:37:31.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f2ffa59c640 1 -- 172.21.15.37:0/4141324549 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ffc0723a0 msgr2=0x7f2ffc1cf560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:37:31.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f2ffa59c640 1 --2- 172.21.15.37:0/4141324549 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ffc0723a0 0x7f2ffc1cf560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:31.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f2ffa59c640 1 -- 172.21.15.37:0/4141324549 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ffc13c340 msgr2=0x7f2ffc1cfaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:31.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f2ffa59c640 1 --2- 172.21.15.37:0/4141324549 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ffc13c340 0x7f2ffc1cfaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:31.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f2ffa59c640 10 monclient: _finish_hunting 0 2024-04-26T00:37:31.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f2ffa59c640 1 monclient: found mon.noname-c 2024-04-26T00:37:31.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f2ffa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:37:31.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f2ffa59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:37:31.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.011+0000 7f2ffa59c640 1 -- 172.21.15.37:0/4141324549 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ffc13a240 con 0x7f2ffc071fc0 2024-04-26T00:37:31.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.015+0000 7f2ffa59c640 1 -- 172.21.15.37:0/4141324549 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ffc071fc0 msgr2=0x7f2ffc1cf020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:31.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.015+0000 7f2ffa59c640 1 --2- 172.21.15.37:0/4141324549 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ffc071fc0 0x7f2ffc1cf020 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2fe400c470 tx=0x7f2fe400c930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:31.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.015+0000 7f2ff2ffd640 1 -- 172.21.15.37:0/4141324549 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe4007070 con 0x7f2ffc071fc0 2024-04-26T00:37:31.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.015+0000 7f2ff2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:31.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.015+0000 7f2ff2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:37:31.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.015+0000 7f2ff2ffd640 10 monclient: dump: 2024-04-26T00:37:31.022 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:31.022 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:31.022 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:31.022 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:31.023 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:31.023 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:31.023 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:31.023 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:31.023 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:31.023 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:31.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.015+0000 7f2ff2ffd640 10 monclient: _finish_auth 0 2024-04-26T00:37:31.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.015+0000 7f2ff2ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:37:31.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.015+0000 7f2ff2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:37:31.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.015+0000 7f2ff2ffd640 1 -- 172.21.15.37:0/4141324549 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fe4007210 con 0x7f2ffc071fc0 2024-04-26T00:37:31.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.015+0000 7f2ff2ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:37:31.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.015+0000 7f2ff2ffd640 1 -- 172.21.15.37:0/4141324549 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe4007600 con 0x7f2ffc071fc0 2024-04-26T00:37:31.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.015+0000 7f3000c6b640 5 monclient: authenticate success, global_id 204239 2024-04-26T00:37:31.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.015+0000 7f3000c6b640 10 monclient: _renew_subs 2024-04-26T00:37:31.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.015+0000 7f3000c6b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:37:31.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.015+0000 7f2ff3fff640 4 set_mon_vals no callback set 2024-04-26T00:37:31.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.015+0000 7f3000c6b640 1 -- 172.21.15.37:0/4141324549 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ffc139350 con 0x7f2ffc071fc0 2024-04-26T00:37:31.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.015+0000 7f2ff2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:31.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.015+0000 7f2ff2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:37:31.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.015+0000 7f2ff2ffd640 10 monclient: dump: 2024-04-26T00:37:31.025 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:31.025 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:31.025 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:31.026 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:31.026 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:31.026 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:31.026 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:31.026 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:31.026 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:31.026 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:31.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.015+0000 7f3000c6b640 10 monclient: _renew_subs 2024-04-26T00:37:31.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.015+0000 7f3000c6b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:37:31.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.015+0000 7f3000c6b640 1 -- 172.21.15.37:0/4141324549 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ffc09efa0 con 0x7f2ffc071fc0 2024-04-26T00:37:31.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.015+0000 7f3000c6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:31.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.015+0000 7f3000c6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:31.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.015+0000 7f3000c6b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:37:31.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.015+0000 7f3000c6b640 1 -- 172.21.15.37:0/4141324549 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ffc066f40 con 0x7f2ffc071fc0 2024-04-26T00:37:31.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.015+0000 7f2ff2ffd640 1 -- 172.21.15.37:0/4141324549 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2fe401b400 con 0x7f2ffc071fc0 2024-04-26T00:37:31.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.015+0000 7f2ff2ffd640 1 --2- 172.21.15.37:0/4141324549 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2fdc035e30 0x7f2fdc0382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:31.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.015+0000 7f2ff2ffd640 1 -- 172.21.15.37:0/4141324549 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(691..691 src has 370..691) ==== 3927+0+0 (secure 0 0 0) 0x7f2fe40487f0 con 0x7f2ffc071fc0 2024-04-26T00:37:31.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.015+0000 7f2ff9d9b640 1 --2- 172.21.15.37:0/4141324549 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2fdc035e30 0x7f2fdc0382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:31.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.015+0000 7f2ff9d9b640 10 monclient: get_auth_request con 0x7f2fdc035e30 auth_method 0 2024-04-26T00:37:31.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.019+0000 7f2ff9d9b640 1 --2- 172.21.15.37:0/4141324549 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2fdc035e30 0x7f2fdc0382f0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f2fec0ac000 tx=0x7f2fec0a7070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:31.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.023+0000 7f2ff2ffd640 1 -- 172.21.15.37:0/4141324549 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f2ffc13a240 con 0x7f2ffc071fc0 2024-04-26T00:37:31.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.023+0000 7f2ff2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:31.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.023+0000 7f2ff2ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:37:31.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.243+0000 7f3000c6b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:37:31.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.243+0000 7f3000c6b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:37:31.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.243+0000 7f3000c6b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:37:31.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.243+0000 7f3000c6b640 1 -- 172.21.15.37:0/4141324549 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2ffc06b9a0 con 0x7f2ffc071fc0 2024-04-26T00:37:31.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.243+0000 7f2ff2ffd640 1 -- 172.21.15.37:0/4141324549 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f2fe4047540 con 0x7f2ffc071fc0 2024-04-26T00:37:31.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.243+0000 7f2ff2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:37:31.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.243+0000 7f2ff2ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:37:31.247 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":132,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:37:31.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.247+0000 7f3000c6b640 1 -- 172.21.15.37:0/4141324549 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2fdc035e30 msgr2=0x7f2fdc0382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:31.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.247+0000 7f3000c6b640 1 --2- 172.21.15.37:0/4141324549 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2fdc035e30 0x7f2fdc0382f0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f2fec0ac000 tx=0x7f2fec0a7070 comp rx=0 tx=0).stop 2024-04-26T00:37:31.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.247+0000 7f3000c6b640 10 monclient: shutdown 2024-04-26T00:37:31.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.247+0000 7f3000c6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:37:31.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.247+0000 7f3000c6b640 1 -- 172.21.15.37:0/4141324549 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ffc071fc0 msgr2=0x7f2ffc1cf020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:31.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.247+0000 7f3000c6b640 1 --2- 172.21.15.37:0/4141324549 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ffc071fc0 0x7f2ffc1cf020 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2fe400c470 tx=0x7f2fe400c930 comp rx=0 tx=0).stop 2024-04-26T00:37:31.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.247+0000 7f3000c6b640 1 -- 172.21.15.37:0/4141324549 shutdown_connections 2024-04-26T00:37:31.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.247+0000 7f3000c6b640 1 --2- 172.21.15.37:0/4141324549 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2fdc035e30 0x7f2fdc0382f0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:31.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.247+0000 7f3000c6b640 1 --2- 172.21.15.37:0/4141324549 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ffc13c340 0x7f2ffc1cfaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:31.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.247+0000 7f3000c6b640 1 --2- 172.21.15.37:0/4141324549 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ffc0723a0 0x7f2ffc1cf560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:31.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.247+0000 7f3000c6b640 1 --2- 172.21.15.37:0/4141324549 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2ffc071fc0 0x7f2ffc1cf020 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:31.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.247+0000 7f3000c6b640 1 -- 172.21.15.37:0/4141324549 >> 172.21.15.37:0/4141324549 conn(0x7f2ffc0a1ad0 msgr2=0x7f2ffc134e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:37:31.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.247+0000 7f3000c6b640 1 -- 172.21.15.37:0/4141324549 shutdown_connections 2024-04-26T00:37:31.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:31.247+0000 7f3000c6b640 1 -- 172.21.15.37:0/4141324549 wait complete. 2024-04-26T00:37:34.266 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:37:34.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.395+0000 7f1bd4209640 10 monclient: get_monmap_and_config 2024-04-26T00:37:34.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.395+0000 7f1bd4209640 10 monclient: build_initial_monmap 2024-04-26T00:37:34.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.395+0000 7f1bd4209640 10 monclient: monmap: 2024-04-26T00:37:34.398 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:37:34.398 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:34.398 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:37:34.401963+0000 2024-04-26T00:37:34.398 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:37:34.401963+0000 2024-04-26T00:37:34.398 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:37:34.398 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:34.398 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:37:34.398 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:37:34.398 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:37:34.398 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:37:34.398 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:37:34.398 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:34.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.395+0000 7f1bd4209640 1 Processor -- start 2024-04-26T00:37:34.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.395+0000 7f1bd4209640 1 -- start start 2024-04-26T00:37:34.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.395+0000 7f1bd4209640 10 monclient: init 2024-04-26T00:37:34.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.395+0000 7f1bd4209640 10 monclient: _reopen_session rank -1 2024-04-26T00:37:34.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.395+0000 7f1bd4209640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-04-26T00:37:34.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.395+0000 7f1bd4209640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1bcc136e10 addr v1:172.21.15.45:6789/0 2024-04-26T00:37:34.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.395+0000 7f1bd4209640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1bcc137900 0x7f1bcc09c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:34.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.395+0000 7f1bd4209640 10 monclient(hunting): picked mon.noname-c con 0x7f1bcc137900 addr v2:172.21.15.45:3300/0 2024-04-26T00:37:34.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.395+0000 7f1bd4209640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f1bcc13c340 0x7f1bcc13e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:34.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.395+0000 7f1bd4209640 10 monclient(hunting): picked mon.noname-a con 0x7f1bcc13c340 addr v2:172.21.15.37:3300/0 2024-04-26T00:37:34.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.395+0000 7f1bd4209640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bcc09b680 con 0x7f1bcc136e10 2024-04-26T00:37:34.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.395+0000 7f1bd4209640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:34.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.395+0000 7f1bd4209640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f1bcc09b8f0 con 0x7f1bcc13c340 2024-04-26T00:37:34.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.395+0000 7f1bd4209640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:34.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.395+0000 7f1bd4209640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f1bcc09bbd0 con 0x7f1bcc137900 2024-04-26T00:37:34.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.395+0000 7f1bd4209640 10 monclient(hunting): _renew_subs 2024-04-26T00:37:34.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.395+0000 7f1bd4209640 10 monclient(hunting): authenticate will time out at 2035.707782s 2024-04-26T00:37:34.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.395+0000 7f1bd17a4640 1 -- >> v2:172.21.15.45:3300/0 conn(0x7f1bcc137900 msgr2=0x7f1bcc09c5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:37:34.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.395+0000 7f1bd1fa5640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f1bcc136e10 legacy=0x7f1bcc1371f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:37:34.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.395+0000 7f1bd17a4640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1bcc137900 0x7f1bcc09c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:37:34.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.395+0000 7f1bd27a6640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f1bcc13c340 0x7f1bcc13e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:34.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.399+0000 7f1bd27a6640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f1bcc13c340 0x7f1bcc13e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:54330/0 (socket says 172.21.15.37:54330) 2024-04-26T00:37:34.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.399+0000 7f1bd27a6640 1 -- 172.21.15.37:0/2415099989 learned_addr learned my addr 172.21.15.37:0/2415099989 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:37:34.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.399+0000 7f1bd27a6640 10 monclient(hunting): get_auth_request con 0x7f1bcc13c340 auth_method 0 2024-04-26T00:37:34.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.399+0000 7f1bd27a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:34.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.399+0000 7f1bd27a6640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:34.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.399+0000 7f1bd27a6640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:34.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.399+0000 7f1bd27a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:34.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.399+0000 7f1bd27a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:34.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.399+0000 7f1bd27a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:34.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.399+0000 7f1bd27a6640 10 monclient(hunting): handle_auth_done global_id 144855 payload 274 2024-04-26T00:37:34.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.399+0000 7f1bd27a6640 1 -- 172.21.15.37:0/2415099989 >> v2:172.21.15.45:3300/0 conn(0x7f1bcc137900 msgr2=0x7f1bcc09c5b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:37:34.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.399+0000 7f1bd27a6640 1 --2- 172.21.15.37:0/2415099989 >> v2:172.21.15.45:3300/0 conn(0x7f1bcc137900 0x7f1bcc09c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:34.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.399+0000 7f1bd27a6640 1 -- 172.21.15.37:0/2415099989 >> v1:172.21.15.45:6789/0 conn(0x7f1bcc136e10 legacy=0x7f1bcc1371f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:37:34.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.399+0000 7f1bd27a6640 10 monclient: _finish_hunting 0 2024-04-26T00:37:34.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.399+0000 7f1bd27a6640 1 monclient: found mon.noname-a 2024-04-26T00:37:34.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.399+0000 7f1bd27a6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:37:34.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.399+0000 7f1bd27a6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:37:34.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.399+0000 7f1bd27a6640 1 -- 172.21.15.37:0/2415099989 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bcc0a5d90 con 0x7f1bcc13c340 2024-04-26T00:37:34.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd27a6640 1 -- 172.21.15.37:0/2415099989 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1bcc13c340 msgr2=0x7f1bcc13e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:34.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd27a6640 1 --2- 172.21.15.37:0/2415099989 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1bcc13c340 0x7f1bcc13e730 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc068e60 tx=0x7f1bc401c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd958963d28ab290 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:34.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd0fa3640 1 -- 172.21.15.37:0/2415099989 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc40a5550 con 0x7f1bcc13c340 2024-04-26T00:37:34.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd0fa3640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:34.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd0fa3640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:37:34.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd0fa3640 10 monclient: dump: 2024-04-26T00:37:34.406 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:34.406 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:34.406 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:34.406 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:34.406 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:34.406 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:34.406 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:34.406 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:34.406 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:34.406 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:34.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd0fa3640 10 monclient: _finish_auth 0 2024-04-26T00:37:34.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd0fa3640 10 monclient: _check_auth_tickets 2024-04-26T00:37:34.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd0fa3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:37:34.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd0fa3640 1 -- 172.21.15.37:0/2415099989 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bc40a56f0 con 0x7f1bcc13c340 2024-04-26T00:37:34.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd0fa3640 10 monclient: handle_config config(0 keys) 2024-04-26T00:37:34.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd0fa3640 1 -- 172.21.15.37:0/2415099989 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc40a5ac0 con 0x7f1bcc13c340 2024-04-26T00:37:34.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd0fa3640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:34.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd0fa3640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:37:34.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd0fa3640 10 monclient: dump: 2024-04-26T00:37:34.408 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:34.408 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:34.408 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:34.408 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:34.408 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:34.408 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:34.408 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:34.408 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:34.408 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:34.408 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:34.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd4209640 5 monclient: authenticate success, global_id 144855 2024-04-26T00:37:34.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd4209640 10 monclient: get_monmap_and_config success 2024-04-26T00:37:34.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd4209640 4 set_mon_vals no callback set 2024-04-26T00:37:34.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd4209640 10 monclient: shutdown 2024-04-26T00:37:34.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd4209640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:37:34.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd4209640 1 -- 172.21.15.37:0/2415099989 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1bcc13c340 msgr2=0x7f1bcc13e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:34.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd4209640 1 --2- 172.21.15.37:0/2415099989 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1bcc13c340 0x7f1bcc13e730 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc068e60 tx=0x7f1bc401c890 comp rx=0 tx=0).stop 2024-04-26T00:37:34.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd4209640 1 -- 172.21.15.37:0/2415099989 shutdown_connections 2024-04-26T00:37:34.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd4209640 1 --2- 172.21.15.37:0/2415099989 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1bcc13c340 0x7f1bcc13e730 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:34.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd4209640 1 --2- 172.21.15.37:0/2415099989 >> v2:172.21.15.45:3300/0 conn(0x7f1bcc137900 0x7f1bcc09c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:34.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd4209640 1 -- 172.21.15.37:0/2415099989 >> 172.21.15.37:0/2415099989 conn(0x7f1bcc0a64e0 msgr2=0x7f1bcc0a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:37:34.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd4209640 1 -- 172.21.15.37:0/2415099989 shutdown_connections 2024-04-26T00:37:34.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd4209640 1 -- 172.21.15.37:0/2415099989 wait complete. 2024-04-26T00:37:34.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd4209640 10 monclient: build_initial_monmap 2024-04-26T00:37:34.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd4209640 10 monclient: monmap: 2024-04-26T00:37:34.409 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:37:34.409 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:37:34.409 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:37:34.410 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:37:34.410 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:37:34.410 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:37:34.410 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:37:34.410 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:37:34.410 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:37:34.410 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:34.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd4209640 1 Processor -- start 2024-04-26T00:37:34.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd4209640 1 -- start start 2024-04-26T00:37:34.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd4209640 10 monclient: init 2024-04-26T00:37:34.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd4209640 10 monclient: _reopen_session rank -1 2024-04-26T00:37:34.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd4209640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:37:34.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd4209640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1bcc136e10 0x7f1bcc1cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:34.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd4209640 10 monclient(hunting): picked mon.noname-c con 0x7f1bcc136e10 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:37:34.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd4209640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1bcc137900 0x7f1bcc135d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:34.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd4209640 10 monclient(hunting): picked mon.noname-a con 0x7f1bcc137900 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:37:34.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd4209640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bcc13c340 0x7f1bcc138a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:34.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd4209640 10 monclient(hunting): picked mon.noname-b con 0x7f1bcc13c340 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:37:34.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd4209640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:34.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd4209640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f1bcc141820 con 0x7f1bcc137900 2024-04-26T00:37:34.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd4209640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:34.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd4209640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1bcc1416a0 con 0x7f1bcc13c340 2024-04-26T00:37:34.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd4209640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:34.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd4209640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f1bcc1419a0 con 0x7f1bcc136e10 2024-04-26T00:37:34.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd4209640 10 monclient(hunting): _renew_subs 2024-04-26T00:37:34.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd4209640 10 monclient(hunting): authenticate will time out at 2035.715181s 2024-04-26T00:37:34.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd1fa5640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1bcc136e10 0x7f1bcc1cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:34.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd17a4640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1bcc137900 0x7f1bcc135d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:34.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd27a6640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bcc13c340 msgr2=0x7f1bcc138a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:37:34.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd17a4640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1bcc137900 0x7f1bcc135d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:54338/0 (socket says 172.21.15.37:54338) 2024-04-26T00:37:34.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd17a4640 1 -- 172.21.15.37:0/3887708973 learned_addr learned my addr 172.21.15.37:0/3887708973 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:37:34.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd1fa5640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1bcc136e10 0x7f1bcc1cebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:59162/0 (socket says 172.21.15.37:59162) 2024-04-26T00:37:34.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd17a4640 10 monclient(hunting): get_auth_request con 0x7f1bcc137900 auth_method 0 2024-04-26T00:37:34.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd17a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:34.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd17a4640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:34.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd17a4640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:34.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd27a6640 1 --2- 172.21.15.37:0/3887708973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bcc13c340 0x7f1bcc138a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:37:34.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd1fa5640 10 monclient(hunting): get_auth_request con 0x7f1bcc136e10 auth_method 0 2024-04-26T00:37:34.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd1fa5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:34.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd1fa5640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:34.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd1fa5640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:34.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd17a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:34.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd17a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:34.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd17a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:34.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd1fa5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:34.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd1fa5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:34.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd1fa5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:34.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd17a4640 10 monclient(hunting): handle_auth_done global_id 144858 payload 824 2024-04-26T00:37:34.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd17a4640 1 -- 172.21.15.37:0/3887708973 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1bcc136e10 msgr2=0x7f1bcc1cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:34.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd17a4640 1 --2- 172.21.15.37:0/3887708973 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1bcc136e10 0x7f1bcc1cebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:34.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd17a4640 1 -- 172.21.15.37:0/3887708973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bcc13c340 msgr2=0x7f1bcc138a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:37:34.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd17a4640 1 --2- 172.21.15.37:0/3887708973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bcc13c340 0x7f1bcc138a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:34.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd17a4640 10 monclient: _finish_hunting 0 2024-04-26T00:37:34.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd17a4640 1 monclient: found mon.noname-a 2024-04-26T00:37:34.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd17a4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:37:34.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd17a4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:37:34.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd17a4640 1 -- 172.21.15.37:0/3887708973 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bcc120440 con 0x7f1bcc137900 2024-04-26T00:37:34.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.403+0000 7f1bd1fa5640 1 --2- 172.21.15.37:0/3887708973 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1bcc136e10 0x7f1bcc1cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:37:34.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.407+0000 7f1bd17a4640 1 -- 172.21.15.37:0/3887708973 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1bcc137900 msgr2=0x7f1bcc135d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:34.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.407+0000 7f1bd17a4640 1 --2- 172.21.15.37:0/3887708973 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1bcc137900 0x7f1bcc135d90 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc00b530 tx=0x7f1bbc00b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:34.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.407+0000 7f1bc27fc640 1 -- 172.21.15.37:0/3887708973 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc013020 con 0x7f1bcc137900 2024-04-26T00:37:34.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.407+0000 7f1bc27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:34.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.407+0000 7f1bc27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:37:34.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.407+0000 7f1bc27fc640 10 monclient: dump: 2024-04-26T00:37:34.418 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:34.418 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:34.418 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:34.419 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:34.419 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:34.419 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:34.419 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:34.419 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:34.419 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:34.419 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:34.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.407+0000 7f1bc27fc640 10 monclient: _finish_auth 0 2024-04-26T00:37:34.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.407+0000 7f1bc27fc640 10 monclient: _check_auth_tickets 2024-04-26T00:37:34.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.407+0000 7f1bc27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:37:34.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.407+0000 7f1bc27fc640 1 -- 172.21.15.37:0/3887708973 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bbc005070 con 0x7f1bcc137900 2024-04-26T00:37:34.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.407+0000 7f1bc27fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:37:34.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.407+0000 7f1bc27fc640 1 -- 172.21.15.37:0/3887708973 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc005460 con 0x7f1bcc137900 2024-04-26T00:37:34.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.407+0000 7f1bc27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:34.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.407+0000 7f1bc27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:37:34.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.407+0000 7f1bc27fc640 10 monclient: dump: 2024-04-26T00:37:34.420 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:34.420 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:34.420 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:34.420 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:34.420 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:34.420 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:34.420 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:34.420 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:34.420 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:34.420 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:34.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.407+0000 7f1bd4209640 5 monclient: authenticate success, global_id 144858 2024-04-26T00:37:34.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.407+0000 7f1bd4209640 10 monclient: _renew_subs 2024-04-26T00:37:34.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.407+0000 7f1bc3fff640 4 set_mon_vals no callback set 2024-04-26T00:37:34.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.407+0000 7f1bd4209640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:37:34.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.407+0000 7f1bd4209640 1 -- 172.21.15.37:0/3887708973 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bcc120160 con 0x7f1bcc137900 2024-04-26T00:37:34.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.407+0000 7f1bd4209640 10 monclient: _renew_subs 2024-04-26T00:37:34.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.407+0000 7f1bd4209640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:37:34.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.407+0000 7f1bd4209640 1 -- 172.21.15.37:0/3887708973 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bcc13b470 con 0x7f1bcc137900 2024-04-26T00:37:34.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.407+0000 7f1bd4209640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:34.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.407+0000 7f1bd4209640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:34.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.407+0000 7f1bd4209640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:37:34.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.407+0000 7f1bd4209640 1 -- 172.21.15.37:0/3887708973 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bcc075910 con 0x7f1bcc137900 2024-04-26T00:37:34.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.407+0000 7f1bc27fc640 1 -- 172.21.15.37:0/3887708973 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f1bbc016020 con 0x7f1bcc137900 2024-04-26T00:37:34.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.407+0000 7f1bc27fc640 1 --2- 172.21.15.37:0/3887708973 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1bb0035e30 0x7f1bb00382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:34.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.407+0000 7f1bc27fc640 1 -- 172.21.15.37:0/3887708973 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(691..691 src has 370..691) ==== 3927+0+0 (secure 0 0 0) 0x7f1bbc048700 con 0x7f1bcc137900 2024-04-26T00:37:34.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.411+0000 7f1bd1fa5640 1 --2- 172.21.15.37:0/3887708973 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1bb0035e30 0x7f1bb00382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:34.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.411+0000 7f1bd1fa5640 10 monclient: get_auth_request con 0x7f1bb0035e30 auth_method 0 2024-04-26T00:37:34.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.411+0000 7f1bd1fa5640 1 --2- 172.21.15.37:0/3887708973 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1bb0035e30 0x7f1bb00382f0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8003940 tx=0x7f1bc8003b50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:34.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.415+0000 7f1bc27fc640 1 -- 172.21.15.37:0/3887708973 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f1bbc01d020 con 0x7f1bcc137900 2024-04-26T00:37:34.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.415+0000 7f1bc27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:34.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.415+0000 7f1bc27fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:37:34.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.639+0000 7f1bd4209640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:37:34.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.639+0000 7f1bd4209640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:37:34.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.639+0000 7f1bd4209640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:37:34.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.639+0000 7f1bd4209640 1 -- 172.21.15.37:0/3887708973 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1bcc0a84a0 con 0x7f1bcc137900 2024-04-26T00:37:34.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.639+0000 7f1bc27fc640 1 -- 172.21.15.37:0/3887708973 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f1bbc018450 con 0x7f1bcc137900 2024-04-26T00:37:34.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.639+0000 7f1bc27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:37:34.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.639+0000 7f1bc27fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:37:34.641 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":132,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:37:34.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.639+0000 7f1bd4209640 1 -- 172.21.15.37:0/3887708973 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1bb0035e30 msgr2=0x7f1bb00382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:34.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.639+0000 7f1bd4209640 1 --2- 172.21.15.37:0/3887708973 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1bb0035e30 0x7f1bb00382f0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8003940 tx=0x7f1bc8003b50 comp rx=0 tx=0).stop 2024-04-26T00:37:34.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.639+0000 7f1bd4209640 10 monclient: shutdown 2024-04-26T00:37:34.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.639+0000 7f1bd4209640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:37:34.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.639+0000 7f1bd4209640 1 -- 172.21.15.37:0/3887708973 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1bcc137900 msgr2=0x7f1bcc135d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:34.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.639+0000 7f1bd4209640 1 --2- 172.21.15.37:0/3887708973 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1bcc137900 0x7f1bcc135d90 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc00b530 tx=0x7f1bbc00b9f0 comp rx=0 tx=0).stop 2024-04-26T00:37:34.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.639+0000 7f1bd4209640 1 -- 172.21.15.37:0/3887708973 shutdown_connections 2024-04-26T00:37:34.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.639+0000 7f1bd4209640 1 --2- 172.21.15.37:0/3887708973 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1bb0035e30 0x7f1bb00382f0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:34.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.639+0000 7f1bd4209640 1 --2- 172.21.15.37:0/3887708973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1bcc13c340 0x7f1bcc138a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:34.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.639+0000 7f1bd4209640 1 --2- 172.21.15.37:0/3887708973 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1bcc137900 0x7f1bcc135d90 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:34.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.639+0000 7f1bd4209640 1 --2- 172.21.15.37:0/3887708973 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1bcc136e10 0x7f1bcc1cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:34.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.643+0000 7f1bd4209640 1 -- 172.21.15.37:0/3887708973 >> 172.21.15.37:0/3887708973 conn(0x7f1bcc0a64e0 msgr2=0x7f1bcc13d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:37:34.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.643+0000 7f1bd4209640 1 -- 172.21.15.37:0/3887708973 shutdown_connections 2024-04-26T00:37:34.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:34.643+0000 7f1bd4209640 1 -- 172.21.15.37:0/3887708973 wait complete. 2024-04-26T00:37:37.661 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:37:37.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:37.791+0000 7f9e15db4640 10 monclient: get_monmap_and_config 2024-04-26T00:37:37.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:37.791+0000 7f9e15db4640 10 monclient: build_initial_monmap 2024-04-26T00:37:37.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:37.791+0000 7f9e15db4640 10 monclient: monmap: 2024-04-26T00:37:37.793 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:37:37.793 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:37.793 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:37:37.797504+0000 2024-04-26T00:37:37.793 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:37:37.797504+0000 2024-04-26T00:37:37.793 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:37:37.793 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:37.794 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:37:37.794 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:37:37.794 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:37:37.794 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:37:37.794 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:37:37.794 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:37.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:37.791+0000 7f9e15db4640 1 Processor -- start 2024-04-26T00:37:37.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:37.791+0000 7f9e15db4640 1 -- start start 2024-04-26T00:37:37.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:37.791+0000 7f9e15db4640 10 monclient: init 2024-04-26T00:37:37.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:37.791+0000 7f9e15db4640 10 monclient: _reopen_session rank -1 2024-04-26T00:37:37.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:37.791+0000 7f9e15db4640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-04-26T00:37:37.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:37.791+0000 7f9e15db4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9e10138330 addr v1:172.21.15.45:6789/0 2024-04-26T00:37:37.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:37.791+0000 7f9e15db4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9e1009de00 addr v1:172.21.15.37:6789/0 2024-04-26T00:37:37.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:37.791+0000 7f9e15db4640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9e10141bf0 0x7f9e10143fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:37.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:37.791+0000 7f9e15db4640 10 monclient(hunting): picked mon.noname-c con 0x7f9e10141bf0 addr v2:172.21.15.45:3300/0 2024-04-26T00:37:37.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:37.791+0000 7f9e15db4640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e1009cb80 con 0x7f9e1009de00 2024-04-26T00:37:37.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:37.791+0000 7f9e15db4640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e1009d140 con 0x7f9e10138330 2024-04-26T00:37:37.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:37.791+0000 7f9e15db4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:37.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:37.791+0000 7f9e15db4640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f9e1009d4b0 con 0x7f9e10141bf0 2024-04-26T00:37:37.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:37.791+0000 7f9e15db4640 10 monclient(hunting): _renew_subs 2024-04-26T00:37:37.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:37.791+0000 7f9e15db4640 10 monclient(hunting): authenticate will time out at 2039.103259s 2024-04-26T00:37:37.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:37.791+0000 7f9e0f7fe640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f9e10138330 legacy=0x7f9e10138710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:37:37.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:37.791+0000 7f9e0ffff640 1 -- >> v2:172.21.15.45:3300/0 conn(0x7f9e10141bf0 msgr2=0x7f9e10143fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:37:37.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:37.791+0000 7f9e0effd640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f9e1009de00 0x7f9e101404d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:37282/0 (socket says 172.21.15.37:37282) 2024-04-26T00:37:37.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:37.791+0000 7f9e0effd640 1 -- 172.21.15.37:0/913690994 learned_addr learned my addr 172.21.15.37:0/913690994 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:37:37.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:37.791+0000 7f9e0ffff640 1 --2- 172.21.15.37:0/913690994 >> v2:172.21.15.45:3300/0 conn(0x7f9e10141bf0 0x7f9e10143fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:37:37.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:37.795+0000 7f9e0effd640 1 -- 172.21.15.37:0/913690994 >> v1:172.21.15.37:6789/0 conn(0x7f9e1009de00 legacy=0x7f9e101404d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:37.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:37.995+0000 7f9e0f7fe640 1 -- 172.21.15.37:0/913690994 >> v1:172.21.15.45:6789/0 conn(0x7f9e10138330 legacy=0x7f9e10138710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:37:37.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:37.995+0000 7f9e0ffff640 1 -- 172.21.15.37:0/913690994 >> v2:172.21.15.45:3300/0 conn(0x7f9e10141bf0 msgr2=0x7f9e10143fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:37:37.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:37.995+0000 7f9e0ffff640 1 --2- 172.21.15.37:0/913690994 >> v2:172.21.15.45:3300/0 conn(0x7f9e10141bf0 0x7f9e10143fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:37:38.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:38.399+0000 7f9e0ffff640 1 -- 172.21.15.37:0/913690994 >> v2:172.21.15.45:3300/0 conn(0x7f9e10141bf0 msgr2=0x7f9e10143fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:37:38.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:38.399+0000 7f9e0ffff640 1 --2- 172.21.15.37:0/913690994 >> v2:172.21.15.45:3300/0 conn(0x7f9e10141bf0 0x7f9e10143fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:37:38.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:38.399+0000 7f9e0f7fe640 1 -- 172.21.15.37:0/913690994 >> v1:172.21.15.45:6789/0 conn(0x7f9e10138330 legacy=0x7f9e10138710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:37:39.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:39.203+0000 7f9e0f7fe640 1 -- 172.21.15.37:0/913690994 >> v1:172.21.15.45:6789/0 conn(0x7f9e10138330 legacy=0x7f9e10138710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:37:39.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:39.203+0000 7f9e0ffff640 1 -- 172.21.15.37:0/913690994 >> v2:172.21.15.45:3300/0 conn(0x7f9e10141bf0 msgr2=0x7f9e10143fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:37:39.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:39.203+0000 7f9e0ffff640 1 --2- 172.21.15.37:0/913690994 >> v2:172.21.15.45:3300/0 conn(0x7f9e10141bf0 0x7f9e10143fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:37:40.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.795+0000 7f9e0d7fa640 10 monclient(hunting): tick 2024-04-26T00:37:40.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.795+0000 7f9e0d7fa640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:37:40.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.795+0000 7f9e0d7fa640 1 monclient(hunting): continuing hunt 2024-04-26T00:37:40.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.795+0000 7f9e0d7fa640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:37:40.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.795+0000 7f9e0d7fa640 1 -- 172.21.15.37:0/913690994 >> v2:172.21.15.45:3300/0 conn(0x7f9e10141bf0 msgr2=0x7f9e10143fe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:37:40.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.795+0000 7f9e0d7fa640 1 --2- 172.21.15.37:0/913690994 >> v2:172.21.15.45:3300/0 conn(0x7f9e10141bf0 0x7f9e10143fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:40.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.795+0000 7f9e0d7fa640 1 -- 172.21.15.37:0/913690994 >> v1:172.21.15.45:6789/0 conn(0x7f9e10138330 legacy=0x7f9e10138710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:37:40.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.795+0000 7f9e0d7fa640 1 -- 172.21.15.37:0/913690994 >> v1:172.21.15.37:6789/0 conn(0x7f9e1009de00 legacy=0x7f9e101404d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:40.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.795+0000 7f9e0d7fa640 10 monclient: _add_conns ranks=[2,3] 2024-04-26T00:37:40.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.795+0000 7f9e0d7fa640 1 --2- 172.21.15.37:0/913690994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9df0003750 0x7f9df0005ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:40.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.795+0000 7f9e0d7fa640 10 monclient(hunting): picked mon.noname-b con 0x7f9df0003750 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:37:40.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.795+0000 7f9e0d7fa640 1 --2- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df00062d0 0x7f9df0008790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:40.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.795+0000 7f9e0d7fa640 10 monclient(hunting): picked mon.noname-a con 0x7f9df00062d0 addr v2:172.21.15.37:3300/0 2024-04-26T00:37:40.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.795+0000 7f9e0d7fa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:40.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.795+0000 7f9e0d7fa640 1 -- 172.21.15.37:0/913690994 --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f9e1009d4b0 con 0x7f9df00062d0 2024-04-26T00:37:40.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.795+0000 7f9e0d7fa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:40.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.795+0000 7f9e0d7fa640 1 -- 172.21.15.37:0/913690994 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9df0008e50 con 0x7f9df0003750 2024-04-26T00:37:40.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.795+0000 7f9e0d7fa640 10 monclient(hunting): _renew_subs 2024-04-26T00:37:40.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.795+0000 7f9e0f7fe640 1 --2- 172.21.15.37:0/913690994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9df0003750 0x7f9df0005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:40.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.795+0000 7f9e0effd640 1 --2- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df00062d0 0x7f9df0008790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:40.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.795+0000 7f9e0f7fe640 10 monclient(hunting): get_auth_request con 0x7f9df0003750 auth_method 0 2024-04-26T00:37:40.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.795+0000 7f9e0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:40.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.795+0000 7f9e0f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:40.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.795+0000 7f9e0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:40.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.795+0000 7f9e0effd640 10 monclient(hunting): get_auth_request con 0x7f9df00062d0 auth_method 0 2024-04-26T00:37:40.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.795+0000 7f9e0effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:40.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.795+0000 7f9e0effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:40.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.795+0000 7f9e0effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:40.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.795+0000 7f9e0f7fe640 1 -- 172.21.15.37:0/913690994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9df0003750 msgr2=0x7f9df0005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:37:40.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.795+0000 7f9e0f7fe640 1 -- 172.21.15.37:0/913690994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9df0003750 msgr2=0x7f9df0005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:37:40.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.795+0000 7f9e0f7fe640 1 --2- 172.21.15.37:0/913690994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9df0003750 0x7f9df0005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:37:40.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.795+0000 7f9e0effd640 1 -- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df00062d0 msgr2=0x7f9df0008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:37:40.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.795+0000 7f9e0effd640 1 -- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df00062d0 msgr2=0x7f9df0008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:37:40.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.795+0000 7f9e0effd640 1 --2- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df00062d0 0x7f9df0008790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:37:40.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.795+0000 7f9e0f7fe640 1 --2- 172.21.15.37:0/913690994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9df0003750 0x7f9df0005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:37:40.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.795+0000 7f9e0effd640 1 --2- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df00062d0 0x7f9df0008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:37:40.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.999+0000 7f9e0f7fe640 1 --2- 172.21.15.37:0/913690994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9df0003750 0x7f9df0005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:40.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.999+0000 7f9e0effd640 1 --2- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df00062d0 0x7f9df0008790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:40.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.999+0000 7f9e0f7fe640 10 monclient(hunting): get_auth_request con 0x7f9df0003750 auth_method 0 2024-04-26T00:37:40.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.999+0000 7f9e0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:40.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.999+0000 7f9e0f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:40.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.999+0000 7f9e0f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:37:40.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.999+0000 7f9e0effd640 10 monclient(hunting): get_auth_request con 0x7f9df00062d0 auth_method 0 2024-04-26T00:37:40.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.999+0000 7f9e0effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:40.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.999+0000 7f9e0effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:40.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.999+0000 7f9e0effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:37:40.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.999+0000 7f9e0f7fe640 1 -- 172.21.15.37:0/913690994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9df0003750 msgr2=0x7f9df0005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:37:40.997 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.999+0000 7f9e0f7fe640 1 -- 172.21.15.37:0/913690994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9df0003750 msgr2=0x7f9df0005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:37:40.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.999+0000 7f9e0f7fe640 1 --2- 172.21.15.37:0/913690994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9df0003750 0x7f9df0005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:37:40.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.999+0000 7f9e0effd640 1 -- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df00062d0 msgr2=0x7f9df0008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:37:40.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.999+0000 7f9e0effd640 1 -- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df00062d0 msgr2=0x7f9df0008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:37:40.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.999+0000 7f9e0effd640 1 --2- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df00062d0 0x7f9df0008790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:37:40.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.999+0000 7f9e0f7fe640 1 --2- 172.21.15.37:0/913690994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9df0003750 0x7f9df0005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:37:40.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:40.999+0000 7f9e0effd640 1 --2- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df00062d0 0x7f9df0008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:37:41.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:41.403+0000 7f9e0effd640 1 --2- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df00062d0 0x7f9df0008790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:41.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:41.403+0000 7f9e0f7fe640 1 --2- 172.21.15.37:0/913690994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9df0003750 0x7f9df0005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:41.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:41.403+0000 7f9e0effd640 10 monclient(hunting): get_auth_request con 0x7f9df00062d0 auth_method 0 2024-04-26T00:37:41.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:41.403+0000 7f9e0effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:41.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:41.403+0000 7f9e0effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:41.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:41.403+0000 7f9e0effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:37:41.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:41.403+0000 7f9e0f7fe640 10 monclient(hunting): get_auth_request con 0x7f9df0003750 auth_method 0 2024-04-26T00:37:41.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:41.403+0000 7f9e0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:41.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:41.403+0000 7f9e0f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:41.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:41.403+0000 7f9e0f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:37:41.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:41.403+0000 7f9e0effd640 1 -- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df00062d0 msgr2=0x7f9df0008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:37:41.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:41.403+0000 7f9e0effd640 1 -- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df00062d0 msgr2=0x7f9df0008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:37:41.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:41.403+0000 7f9e0effd640 1 --2- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df00062d0 0x7f9df0008790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:37:41.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:41.403+0000 7f9e0effd640 1 --2- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df00062d0 0x7f9df0008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:37:41.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:41.403+0000 7f9e0f7fe640 1 -- 172.21.15.37:0/913690994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9df0003750 msgr2=0x7f9df0005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:37:41.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:41.403+0000 7f9e0f7fe640 1 -- 172.21.15.37:0/913690994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9df0003750 msgr2=0x7f9df0005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:37:41.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:41.403+0000 7f9e0f7fe640 1 --2- 172.21.15.37:0/913690994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9df0003750 0x7f9df0005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:37:41.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:41.403+0000 7f9e0f7fe640 1 --2- 172.21.15.37:0/913690994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9df0003750 0x7f9df0005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:37:42.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:42.207+0000 7f9e0effd640 1 --2- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df00062d0 0x7f9df0008790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:42.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:42.207+0000 7f9e0effd640 10 monclient(hunting): get_auth_request con 0x7f9df00062d0 auth_method 0 2024-04-26T00:37:42.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:42.207+0000 7f9e0effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:42.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:42.207+0000 7f9e0effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:42.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:42.207+0000 7f9e0effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:37:42.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:42.207+0000 7f9e0f7fe640 1 --2- 172.21.15.37:0/913690994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9df0003750 0x7f9df0005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:42.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:42.207+0000 7f9e0f7fe640 10 monclient(hunting): get_auth_request con 0x7f9df0003750 auth_method 0 2024-04-26T00:37:42.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:42.207+0000 7f9e0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:42.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:42.207+0000 7f9e0f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:42.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:42.207+0000 7f9e0f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:37:42.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:42.207+0000 7f9e0effd640 1 -- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df00062d0 msgr2=0x7f9df0008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:37:42.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:42.207+0000 7f9e0effd640 1 -- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df00062d0 msgr2=0x7f9df0008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:37:42.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:42.207+0000 7f9e0effd640 1 --2- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df00062d0 0x7f9df0008790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:37:42.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:42.207+0000 7f9e0f7fe640 1 -- 172.21.15.37:0/913690994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9df0003750 msgr2=0x7f9df0005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:37:42.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:42.207+0000 7f9e0f7fe640 1 -- 172.21.15.37:0/913690994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9df0003750 msgr2=0x7f9df0005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:37:42.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:42.207+0000 7f9e0f7fe640 1 --2- 172.21.15.37:0/913690994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9df0003750 0x7f9df0005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:37:42.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:42.207+0000 7f9e0effd640 1 --2- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df00062d0 0x7f9df0008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:37:42.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:42.207+0000 7f9e0f7fe640 1 --2- 172.21.15.37:0/913690994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9df0003750 0x7f9df0005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:37:43.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:43.795+0000 7f9e0d7fa640 10 monclient(hunting): tick 2024-04-26T00:37:43.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:43.795+0000 7f9e0d7fa640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:37:43.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:43.795+0000 7f9e0d7fa640 1 monclient(hunting): continuing hunt 2024-04-26T00:37:43.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:43.795+0000 7f9e0d7fa640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:37:43.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:43.795+0000 7f9e0d7fa640 1 -- 172.21.15.37:0/913690994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9df0003750 msgr2=0x7f9df0005ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:37:43.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:43.795+0000 7f9e0d7fa640 1 --2- 172.21.15.37:0/913690994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9df0003750 0x7f9df0005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:43.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:43.795+0000 7f9e0d7fa640 1 -- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df00062d0 msgr2=0x7f9df0008790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:37:43.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:43.795+0000 7f9e0d7fa640 1 --2- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df00062d0 0x7f9df0008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:43.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:43.795+0000 7f9e0ffff640 1 -- 172.21.15.37:0/913690994 reap_dead start 2024-04-26T00:37:43.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:43.795+0000 7f9e0d7fa640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-04-26T00:37:43.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:43.795+0000 7f9e0d7fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9df0003b90 addr v1:172.21.15.45:6789/0 2024-04-26T00:37:43.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:43.795+0000 7f9e0d7fa640 1 --2- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df0007ea0 0x7f9df00082b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:43.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:43.795+0000 7f9e0d7fa640 10 monclient(hunting): picked mon.noname-a con 0x7f9df0007ea0 addr v2:172.21.15.37:3300/0 2024-04-26T00:37:43.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:43.795+0000 7f9e0d7fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9df0008850 addr v1:172.21.15.37:6789/0 2024-04-26T00:37:43.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:43.799+0000 7f9e0d7fa640 1 -- 172.21.15.37:0/913690994 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e100a12f0 con 0x7f9df0008850 2024-04-26T00:37:43.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:43.799+0000 7f9e0d7fa640 1 -- 172.21.15.37:0/913690994 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e1009d140 con 0x7f9df0003b90 2024-04-26T00:37:43.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:43.799+0000 7f9e0d7fa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:43.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:43.799+0000 7f9e0d7fa640 1 -- 172.21.15.37:0/913690994 --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f9e1009d4b0 con 0x7f9df0007ea0 2024-04-26T00:37:43.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:43.799+0000 7f9e0f7fe640 1 -- 172.21.15.37:0/913690994 >> v1:172.21.15.45:6789/0 conn(0x7f9df0003b90 legacy=0x7f9df0006780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:37:43.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:43.799+0000 7f9e0d7fa640 10 monclient(hunting): _renew_subs 2024-04-26T00:37:43.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:43.799+0000 7f9e0effd640 1 --2- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df0007ea0 0x7f9df00082b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:43.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:43.799+0000 7f9e0effd640 10 monclient(hunting): get_auth_request con 0x7f9df0007ea0 auth_method 0 2024-04-26T00:37:43.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:43.799+0000 7f9e0effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:43.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:43.799+0000 7f9e0effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:43.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:43.799+0000 7f9e0effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:43.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:43.799+0000 7f9e0effd640 1 -- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df0007ea0 msgr2=0x7f9df00082b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:37:43.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:43.799+0000 7f9e0effd640 1 -- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df0007ea0 msgr2=0x7f9df00082b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:37:43.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:43.799+0000 7f9e0effd640 1 --2- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df0007ea0 0x7f9df00082b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:37:43.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:43.799+0000 7f9e0effd640 1 --2- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df0007ea0 0x7f9df00082b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:37:43.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:43.799+0000 7f9e0ffff640 1 -- 172.21.15.37:0/913690994 >> v1:172.21.15.37:6789/0 conn(0x7f9df0008850 legacy=0x7f9df000d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:44.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:44.003+0000 7f9e0f7fe640 1 -- 172.21.15.37:0/913690994 >> v1:172.21.15.45:6789/0 conn(0x7f9df0003b90 legacy=0x7f9df0006780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:37:44.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:44.003+0000 7f9e0effd640 1 --2- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df0007ea0 0x7f9df00082b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:44.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:44.003+0000 7f9e0effd640 10 monclient(hunting): get_auth_request con 0x7f9df0007ea0 auth_method 0 2024-04-26T00:37:44.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:44.003+0000 7f9e0effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:44.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:44.003+0000 7f9e0effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:44.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:44.003+0000 7f9e0effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:37:44.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:44.003+0000 7f9e0effd640 1 -- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df0007ea0 msgr2=0x7f9df00082b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:37:44.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:44.003+0000 7f9e0effd640 1 -- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df0007ea0 msgr2=0x7f9df00082b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:37:44.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:44.003+0000 7f9e0effd640 1 --2- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df0007ea0 0x7f9df00082b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:37:44.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:44.003+0000 7f9e0effd640 1 --2- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df0007ea0 0x7f9df00082b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:37:44.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:44.407+0000 7f9e0f7fe640 1 -- 172.21.15.37:0/913690994 >> v1:172.21.15.45:6789/0 conn(0x7f9df0003b90 legacy=0x7f9df0006780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:37:44.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:44.407+0000 7f9e0effd640 1 --2- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df0007ea0 0x7f9df00082b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:44.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:44.407+0000 7f9e0effd640 10 monclient(hunting): get_auth_request con 0x7f9df0007ea0 auth_method 0 2024-04-26T00:37:44.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:44.407+0000 7f9e0effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:44.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:44.407+0000 7f9e0effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:44.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:44.407+0000 7f9e0effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:37:44.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:44.407+0000 7f9e0effd640 1 -- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df0007ea0 msgr2=0x7f9df00082b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:37:44.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:44.407+0000 7f9e0effd640 1 -- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df0007ea0 msgr2=0x7f9df00082b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:37:44.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:44.407+0000 7f9e0effd640 1 --2- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df0007ea0 0x7f9df00082b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:37:44.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:44.407+0000 7f9e0effd640 1 --2- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df0007ea0 0x7f9df00082b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:37:45.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.211+0000 7f9e0f7fe640 1 -- 172.21.15.37:0/913690994 >> v1:172.21.15.45:6789/0 conn(0x7f9df0003b90 legacy=0x7f9df0006780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:37:45.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.211+0000 7f9e0effd640 1 --2- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df0007ea0 0x7f9df00082b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:45.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.211+0000 7f9e0effd640 10 monclient(hunting): get_auth_request con 0x7f9df0007ea0 auth_method 0 2024-04-26T00:37:45.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.211+0000 7f9e0effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:45.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.211+0000 7f9e0effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:45.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.211+0000 7f9e0effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:37:45.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.211+0000 7f9e0effd640 1 -- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df0007ea0 msgr2=0x7f9df00082b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:37:45.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.211+0000 7f9e0effd640 1 -- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df0007ea0 msgr2=0x7f9df00082b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:37:45.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.211+0000 7f9e0effd640 1 --2- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df0007ea0 0x7f9df00082b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:37:45.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.211+0000 7f9e0effd640 1 --2- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df0007ea0 0x7f9df00082b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:37:45.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.651+0000 7f9e0e7fc640 1 -- 172.21.15.37:0/913690994 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1342360368 0 0) 0x7f9e100a12f0 con 0x7f9df0008850 2024-04-26T00:37:45.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.651+0000 7f9e0e7fc640 10 monclient(hunting): _negotiate 2024-04-26T00:37:45.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.651+0000 7f9e0e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:45.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.651+0000 7f9e0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:45.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.651+0000 7f9e0e7fc640 10 monclient(hunting): my global_id is 144861 2024-04-26T00:37:45.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.651+0000 7f9e0e7fc640 1 -- 172.21.15.37:0/913690994 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9dfc003530 con 0x7f9df0008850 2024-04-26T00:37:45.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0e7fc640 1 -- 172.21.15.37:0/913690994 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4139419551 0 0) 0x7f9dfc003530 con 0x7f9df0008850 2024-04-26T00:37:45.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0e7fc640 1 -- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df0007ea0 msgr2=0x7f9df00082b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:37:45.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0e7fc640 1 --2- 172.21.15.37:0/913690994 >> v2:172.21.15.37:3300/0 conn(0x7f9df0007ea0 0x7f9df00082b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:45.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0e7fc640 1 -- 172.21.15.37:0/913690994 >> v1:172.21.15.45:6789/0 conn(0x7f9df0003b90 legacy=0x7f9df0006780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:37:45.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0e7fc640 10 monclient: _finish_hunting 0 2024-04-26T00:37:45.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0e7fc640 1 monclient: found mon.noname-a-legacy 2024-04-26T00:37:45.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:37:45.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0e7fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.37:6789/0 2024-04-26T00:37:45.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0e7fc640 1 -- 172.21.15.37:0/913690994 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9df0010c20 con 0x7f9df0008850 2024-04-26T00:37:45.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0e7fc640 10 monclient: _finish_auth 0 2024-04-26T00:37:45.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0e7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:37:45.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:37:45.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0e7fc640 1 -- 172.21.15.37:0/913690994 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 866326386 0 0) 0x7f9e08004500 con 0x7f9df0008850 2024-04-26T00:37:45.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:45.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0e7fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-26T00:37:45.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0e7fc640 10 monclient: dump: 2024-04-26T00:37:45.657 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:45.657 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:45.657 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:45.657 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:45.657 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:45.657 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:45.657 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:45.657 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:45.657 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:45.657 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:45.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0e7fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] but i'm connected to v1:172.21.15.37:6789/0, reconnecting 2024-04-26T00:37:45.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0e7fc640 10 monclient: _reopen_session rank -1 2024-04-26T00:37:45.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0e7fc640 1 -- 172.21.15.37:0/913690994 >> v1:172.21.15.37:6789/0 conn(0x7f9df0008850 legacy=0x7f9df000d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:45.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0e7fc640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:37:45.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0e7fc640 1 --2- 172.21.15.37:0/913690994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9dfc0039c0 0x7f9dfc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:45.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0e7fc640 10 monclient(hunting): picked mon.c con 0x7f9dfc0039c0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:37:45.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0e7fc640 1 --2- 172.21.15.37:0/913690994 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dfc004810 0x7f9dfc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:45.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0e7fc640 10 monclient(hunting): picked mon.b con 0x7f9dfc004810 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:37:45.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0e7fc640 1 --2- 172.21.15.37:0/913690994 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9dfc005220 0x7f9dfc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:45.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0e7fc640 10 monclient(hunting): picked mon.a con 0x7f9dfc005220 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:37:45.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0e7fc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:45.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0e7fc640 1 -- 172.21.15.37:0/913690994 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9e1009d4b0 con 0x7f9dfc005220 2024-04-26T00:37:45.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0e7fc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:45.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0e7fc640 1 -- 172.21.15.37:0/913690994 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9dfc00c150 con 0x7f9dfc004810 2024-04-26T00:37:45.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0e7fc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:45.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0e7fc640 1 -- 172.21.15.37:0/913690994 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9dfc00c350 con 0x7f9dfc0039c0 2024-04-26T00:37:45.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0e7fc640 10 monclient(hunting): _renew_subs 2024-04-26T00:37:45.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0e7fc640 10 monclient(hunting): _finish_auth 0 2024-04-26T00:37:45.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0effd640 1 -- 172.21.15.37:0/913690994 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dfc004810 msgr2=0x7f9dfc004c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:37:45.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0effd640 1 --2- 172.21.15.37:0/913690994 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dfc004810 0x7f9dfc004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:37:45.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0ffff640 1 --2- 172.21.15.37:0/913690994 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9dfc005220 0x7f9dfc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:45.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0ffff640 10 monclient(hunting): get_auth_request con 0x7f9dfc005220 auth_method 0 2024-04-26T00:37:45.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:45.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:45.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:37:45.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0f7fe640 1 --2- 172.21.15.37:0/913690994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9dfc0039c0 0x7f9dfc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:45.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0f7fe640 10 monclient(hunting): get_auth_request con 0x7f9dfc0039c0 auth_method 0 2024-04-26T00:37:45.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:45.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:45.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:37:45.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:45.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:45.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:37:45.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:45.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:45.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:37:45.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0ffff640 10 monclient(hunting): handle_auth_done global_id 144861 payload 293 2024-04-26T00:37:45.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0ffff640 1 -- 172.21.15.37:0/913690994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9dfc0039c0 msgr2=0x7f9dfc003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:37:45.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0ffff640 1 --2- 172.21.15.37:0/913690994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9dfc0039c0 0x7f9dfc003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:45.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0ffff640 1 -- 172.21.15.37:0/913690994 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dfc004810 msgr2=0x7f9dfc004c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:37:45.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0ffff640 1 --2- 172.21.15.37:0/913690994 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9dfc004810 0x7f9dfc004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:45.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0ffff640 10 monclient: _finish_hunting 0 2024-04-26T00:37:45.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0ffff640 1 monclient: found mon.a 2024-04-26T00:37:45.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:37:45.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0ffff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:37:45.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0ffff640 1 -- 172.21.15.37:0/913690994 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dfc003530 con 0x7f9dfc005220 2024-04-26T00:37:45.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0ffff640 10 monclient: _finish_auth 0 2024-04-26T00:37:45.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0ffff640 10 monclient: _check_auth_tickets 2024-04-26T00:37:45.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e0ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:37:45.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e15db4640 5 monclient: authenticate success, global_id 144861 2024-04-26T00:37:45.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.655+0000 7f9e15db4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:37:45.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.663+0000 7f9e0ffff640 1 -- 172.21.15.37:0/913690994 reap_dead start 2024-04-26T00:37:45.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.663+0000 7f9e0ffff640 1 -- 172.21.15.37:0/913690994 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9dfc005220 msgr2=0x7f9dfc00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:45.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.663+0000 7f9e0ffff640 1 --2- 172.21.15.37:0/913690994 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9dfc005220 0x7f9dfc00b9d0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9e1006cfa0 tx=0x7f9e0801d4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fea180d03fa3ecb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:45.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.663+0000 7f9e0e7fc640 1 -- 172.21.15.37:0/913690994 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e080b1070 con 0x7f9dfc005220 2024-04-26T00:37:45.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.663+0000 7f9e0e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:45.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.663+0000 7f9e0e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:37:45.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.663+0000 7f9e0e7fc640 10 monclient: dump: 2024-04-26T00:37:45.663 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:45.663 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:45.663 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:45.663 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:45.663 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:45.664 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:45.664 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:45.664 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:45.664 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:45.664 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:45.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.663+0000 7f9e15db4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:37:45.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.663+0000 7f9e0e7fc640 1 -- 172.21.15.37:0/913690994 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e080ac640 con 0x7f9dfc005220 2024-04-26T00:37:45.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.663+0000 7f9e0e7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:37:45.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.663+0000 7f9e15db4640 10 monclient: get_monmap_and_config success 2024-04-26T00:37:45.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.663+0000 7f9e0e7fc640 1 -- 172.21.15.37:0/913690994 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e0800d350 con 0x7f9dfc005220 2024-04-26T00:37:45.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.663+0000 7f9e0e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:45.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.663+0000 7f9e15db4640 4 set_mon_vals no callback set 2024-04-26T00:37:45.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.663+0000 7f9e0e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:37:45.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.663+0000 7f9e0e7fc640 10 monclient: dump: 2024-04-26T00:37:45.664 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:45.665 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:45.665 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:45.665 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:45.665 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:45.665 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:45.665 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:45.665 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:45.665 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:45.665 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:45.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e15db4640 10 monclient: shutdown 2024-04-26T00:37:45.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e15db4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:37:45.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e15db4640 1 -- 172.21.15.37:0/913690994 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9dfc005220 msgr2=0x7f9dfc00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:45.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e15db4640 1 --2- 172.21.15.37:0/913690994 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9dfc005220 0x7f9dfc00b9d0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9e1006cfa0 tx=0x7f9e0801d4c0 comp rx=0 tx=0).stop 2024-04-26T00:37:45.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e15db4640 1 -- 172.21.15.37:0/913690994 shutdown_connections 2024-04-26T00:37:45.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e15db4640 1 --2- 172.21.15.37:0/913690994 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9dfc005220 0x7f9dfc00b9d0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:45.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e15db4640 1 -- 172.21.15.37:0/913690994 >> 172.21.15.37:0/913690994 conn(0x7f9e100a1ad0 msgr2=0x7f9e100a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:37:45.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e15db4640 1 -- 172.21.15.37:0/913690994 shutdown_connections 2024-04-26T00:37:45.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e15db4640 1 -- 172.21.15.37:0/913690994 wait complete. 2024-04-26T00:37:45.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e15db4640 10 monclient: build_initial_monmap 2024-04-26T00:37:45.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e15db4640 10 monclient: monmap: 2024-04-26T00:37:45.669 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:37:45.669 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:37:45.669 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:37:45.669 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:37:45.669 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:37:45.669 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:37:45.669 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:37:45.669 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:37:45.669 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:37:45.669 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:45.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e15db4640 1 Processor -- start 2024-04-26T00:37:45.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e15db4640 1 -- start start 2024-04-26T00:37:45.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e15db4640 10 monclient: init 2024-04-26T00:37:45.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e15db4640 10 monclient: _reopen_session rank -1 2024-04-26T00:37:45.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e15db4640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:37:45.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e15db4640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9dfc005220 0x7f9e10136700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:45.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e15db4640 10 monclient(hunting): picked mon.noname-c con 0x7f9dfc005220 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:37:45.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e15db4640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e10136c40 0x7f9e100707e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:45.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e15db4640 10 monclient(hunting): picked mon.noname-a con 0x7f9e10136c40 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:37:45.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e15db4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e101370c0 0x7f9e10070d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:45.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e15db4640 10 monclient(hunting): picked mon.noname-b con 0x7f9e101370c0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:37:45.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e15db4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:45.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e15db4640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9e1009fd60 con 0x7f9e10136c40 2024-04-26T00:37:45.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e15db4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:45.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e15db4640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9e1009fbe0 con 0x7f9e101370c0 2024-04-26T00:37:45.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e15db4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:45.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e15db4640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9e1009fee0 con 0x7f9dfc005220 2024-04-26T00:37:45.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e15db4640 10 monclient(hunting): _renew_subs 2024-04-26T00:37:45.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e15db4640 10 monclient(hunting): authenticate will time out at 2046.975567s 2024-04-26T00:37:45.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e0effd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e10136c40 0x7f9e100707e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:45.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e0f7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9dfc005220 0x7f9e10136700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:45.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e0effd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e10136c40 0x7f9e100707e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:49572/0 (socket says 172.21.15.37:49572) 2024-04-26T00:37:45.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e0ffff640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e101370c0 msgr2=0x7f9e10070d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:37:45.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e0effd640 1 -- 172.21.15.37:0/3556714840 learned_addr learned my addr 172.21.15.37:0/3556714840 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:37:45.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e0effd640 10 monclient(hunting): get_auth_request con 0x7f9e10136c40 auth_method 0 2024-04-26T00:37:45.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e0effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:45.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e0effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:45.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e0effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:45.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e0ffff640 1 --2- 172.21.15.37:0/3556714840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e101370c0 0x7f9e10070d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:37:45.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e0f7fe640 10 monclient(hunting): get_auth_request con 0x7f9dfc005220 auth_method 0 2024-04-26T00:37:45.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:45.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e0f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:45.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:45.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e0effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:45.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e0effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:45.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.667+0000 7f9e0effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:45.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.671+0000 7f9e0f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:45.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.671+0000 7f9e0f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:45.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.671+0000 7f9e0f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:45.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.671+0000 7f9e0effd640 10 monclient(hunting): handle_auth_done global_id 144864 payload 824 2024-04-26T00:37:45.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.671+0000 7f9e0effd640 1 -- 172.21.15.37:0/3556714840 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9dfc005220 msgr2=0x7f9e10136700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:45.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.671+0000 7f9e0effd640 1 --2- 172.21.15.37:0/3556714840 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9dfc005220 0x7f9e10136700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:45.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.671+0000 7f9e0effd640 1 -- 172.21.15.37:0/3556714840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e101370c0 msgr2=0x7f9e10070d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:37:45.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.671+0000 7f9e0effd640 1 --2- 172.21.15.37:0/3556714840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e101370c0 0x7f9e10070d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:45.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.671+0000 7f9e0effd640 10 monclient: _finish_hunting 0 2024-04-26T00:37:45.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.671+0000 7f9e0effd640 1 monclient: found mon.noname-a 2024-04-26T00:37:45.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.671+0000 7f9e0effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:37:45.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.671+0000 7f9e0effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:37:45.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.671+0000 7f9e0effd640 1 -- 172.21.15.37:0/3556714840 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e10125600 con 0x7f9e10136c40 2024-04-26T00:37:45.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.671+0000 7f9e0effd640 1 -- 172.21.15.37:0/3556714840 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e10136c40 msgr2=0x7f9e100707e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:45.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.671+0000 7f9e0effd640 1 --2- 172.21.15.37:0/3556714840 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e10136c40 0x7f9e100707e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f9df8007ba0 tx=0x7f9df8007820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:45.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.671+0000 7f9e07fff640 1 -- 172.21.15.37:0/3556714840 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df8001d10 con 0x7f9e10136c40 2024-04-26T00:37:45.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.671+0000 7f9e07fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:45.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.671+0000 7f9e07fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:37:45.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.671+0000 7f9e07fff640 10 monclient: dump: 2024-04-26T00:37:45.675 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:45.675 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:45.675 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:45.675 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:45.675 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:45.675 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:45.675 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:45.675 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:45.675 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:45.676 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:45.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.671+0000 7f9e07fff640 10 monclient: _finish_auth 0 2024-04-26T00:37:45.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.671+0000 7f9e07fff640 10 monclient: _check_auth_tickets 2024-04-26T00:37:45.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.671+0000 7f9e07fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:37:45.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.671+0000 7f9e07fff640 1 -- 172.21.15.37:0/3556714840 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9df8019870 con 0x7f9e10136c40 2024-04-26T00:37:45.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.671+0000 7f9e07fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:37:45.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.671+0000 7f9e07fff640 1 -- 172.21.15.37:0/3556714840 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df8019a60 con 0x7f9e10136c40 2024-04-26T00:37:45.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.671+0000 7f9e07fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:45.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.671+0000 7f9e07fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:37:45.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.671+0000 7f9e07fff640 10 monclient: dump: 2024-04-26T00:37:45.676 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:45.676 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:45.676 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:45.676 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:45.676 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:45.677 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:45.677 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:45.677 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:45.677 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:45.677 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:45.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.671+0000 7f9e0dffb640 4 set_mon_vals no callback set 2024-04-26T00:37:45.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.671+0000 7f9e15db4640 5 monclient: authenticate success, global_id 144864 2024-04-26T00:37:45.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.671+0000 7f9e15db4640 10 monclient: _renew_subs 2024-04-26T00:37:45.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.671+0000 7f9e15db4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:37:45.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.671+0000 7f9e15db4640 1 -- 172.21.15.37:0/3556714840 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e100a1590 con 0x7f9e10136c40 2024-04-26T00:37:45.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.671+0000 7f9e15db4640 10 monclient: _renew_subs 2024-04-26T00:37:45.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.671+0000 7f9e15db4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:37:45.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.671+0000 7f9e15db4640 1 -- 172.21.15.37:0/3556714840 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e10072430 con 0x7f9e10136c40 2024-04-26T00:37:45.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.675+0000 7f9e15db4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:45.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.675+0000 7f9e15db4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:45.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.675+0000 7f9e15db4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:37:45.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.675+0000 7f9e15db4640 1 -- 172.21.15.37:0/3556714840 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e10066f40 con 0x7f9e10136c40 2024-04-26T00:37:45.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.675+0000 7f9e07fff640 1 -- 172.21.15.37:0/3556714840 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9df8019c00 con 0x7f9e10136c40 2024-04-26T00:37:45.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.675+0000 7f9e07fff640 1 --2- 172.21.15.37:0/3556714840 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9df0036c50 0x7f9df0039110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:45.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.675+0000 7f9e07fff640 1 -- 172.21.15.37:0/3556714840 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(691..691 src has 370..691) ==== 3927+0+0 (secure 0 0 0) 0x7f9df804a370 con 0x7f9e10136c40 2024-04-26T00:37:45.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.675+0000 7f9e0f7fe640 1 --2- 172.21.15.37:0/3556714840 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9df0036c50 0x7f9df0039110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:45.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.675+0000 7f9e0f7fe640 10 monclient: get_auth_request con 0x7f9df0036c50 auth_method 0 2024-04-26T00:37:45.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.679+0000 7f9e0f7fe640 1 --2- 172.21.15.37:0/3556714840 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9df0036c50 0x7f9df0039110 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9e00005010 tx=0x7f9e0000d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:45.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.679+0000 7f9e07fff640 1 -- 172.21.15.37:0/3556714840 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9df802dca0 con 0x7f9e10136c40 2024-04-26T00:37:45.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.679+0000 7f9e07fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:45.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.679+0000 7f9e07fff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:37:45.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.903+0000 7f9e15db4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:37:45.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.903+0000 7f9e15db4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:37:45.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.903+0000 7f9e15db4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:37:45.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.903+0000 7f9e15db4640 1 -- 172.21.15.37:0/3556714840 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9e1006b9a0 con 0x7f9e10136c40 2024-04-26T00:37:45.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.903+0000 7f9e07fff640 1 -- 172.21.15.37:0/3556714840 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f9df804a120 con 0x7f9e10136c40 2024-04-26T00:37:45.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.903+0000 7f9e07fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:37:45.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.903+0000 7f9e07fff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:37:45.902 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":136,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:37:45.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.907+0000 7f9e15db4640 1 -- 172.21.15.37:0/3556714840 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9df0036c50 msgr2=0x7f9df0039110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:45.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.907+0000 7f9e15db4640 1 --2- 172.21.15.37:0/3556714840 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9df0036c50 0x7f9df0039110 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9e00005010 tx=0x7f9e0000d040 comp rx=0 tx=0).stop 2024-04-26T00:37:45.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.907+0000 7f9e15db4640 10 monclient: shutdown 2024-04-26T00:37:45.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.907+0000 7f9e15db4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:37:45.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.907+0000 7f9e15db4640 1 -- 172.21.15.37:0/3556714840 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e10136c40 msgr2=0x7f9e100707e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:45.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.907+0000 7f9e15db4640 1 --2- 172.21.15.37:0/3556714840 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e10136c40 0x7f9e100707e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f9df8007ba0 tx=0x7f9df8007820 comp rx=0 tx=0).stop 2024-04-26T00:37:45.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.907+0000 7f9e15db4640 1 -- 172.21.15.37:0/3556714840 shutdown_connections 2024-04-26T00:37:45.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.907+0000 7f9e15db4640 1 --2- 172.21.15.37:0/3556714840 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9df0036c50 0x7f9df0039110 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:45.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.907+0000 7f9e15db4640 1 --2- 172.21.15.37:0/3556714840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e101370c0 0x7f9e10070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:45.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.907+0000 7f9e15db4640 1 --2- 172.21.15.37:0/3556714840 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e10136c40 0x7f9e100707e0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:45.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.907+0000 7f9e15db4640 1 --2- 172.21.15.37:0/3556714840 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9dfc005220 0x7f9e10136700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:45.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.907+0000 7f9e15db4640 1 -- 172.21.15.37:0/3556714840 >> 172.21.15.37:0/3556714840 conn(0x7f9e100a1ad0 msgr2=0x7f9e101404d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:37:45.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.907+0000 7f9e15db4640 1 -- 172.21.15.37:0/3556714840 shutdown_connections 2024-04-26T00:37:45.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:45.907+0000 7f9e15db4640 1 -- 172.21.15.37:0/3556714840 wait complete. 2024-04-26T00:37:45.919 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-26T00:37:45.919 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-04-26T00:37:45.919 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:37:46.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d5dbc8640 10 monclient: get_monmap_and_config 2024-04-26T00:37:46.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d5dbc8640 10 monclient: build_initial_monmap 2024-04-26T00:37:46.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d5dbc8640 10 monclient: monmap: 2024-04-26T00:37:46.044 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:37:46.044 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:46.044 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:37:46.048033+0000 2024-04-26T00:37:46.044 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:37:46.048033+0000 2024-04-26T00:37:46.044 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:37:46.044 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:46.044 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:37:46.044 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:37:46.044 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:37:46.044 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:37:46.044 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:37:46.044 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:46.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d5dbc8640 1 Processor -- start 2024-04-26T00:37:46.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d5dbc8640 1 -- start start 2024-04-26T00:37:46.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d5dbc8640 10 monclient: init 2024-04-26T00:37:46.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d5dbc8640 10 monclient: _reopen_session rank -1 2024-04-26T00:37:46.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d5dbc8640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-04-26T00:37:46.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d5dbc8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9d580a3d90 addr v1:172.21.15.37:6789/0 2024-04-26T00:37:46.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d5dbc8640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9d580a4170 0x7f9d580a45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:46.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d5dbc8640 10 monclient(hunting): picked mon.noname-c con 0x7f9d580a4170 addr v2:172.21.15.45:3300/0 2024-04-26T00:37:46.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d5dbc8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d5813c3e0 0x7f9d5813e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:46.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d5dbc8640 10 monclient(hunting): picked mon.noname-b con 0x7f9d5813c3e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:37:46.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d5dbc8640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d580a5150 con 0x7f9d580a3d90 2024-04-26T00:37:46.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d5dbc8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:46.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d5dbc8640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f9d5809c650 con 0x7f9d580a4170 2024-04-26T00:37:46.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d5dbc8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:46.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d5dbc8640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9d5809b280 con 0x7f9d5813c3e0 2024-04-26T00:37:46.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d5dbc8640 10 monclient(hunting): _renew_subs 2024-04-26T00:37:46.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d5dbc8640 10 monclient(hunting): authenticate will time out at 2047.353625s 2024-04-26T00:37:46.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d56ffd640 1 -- >> v2:172.21.15.45:3300/0 conn(0x7f9d580a4170 msgr2=0x7f9d580a45e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:37:46.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d57fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d5813c3e0 0x7f9d5813e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:46.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d56ffd640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9d580a4170 0x7f9d580a45e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:37:46.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d577fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f9d580a3d90 0x7f9d5809bf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:57044/0 (socket says 172.21.15.37:57044) 2024-04-26T00:37:46.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d577fe640 1 -- 172.21.15.37:0/2644017850 learned_addr learned my addr 172.21.15.37:0/2644017850 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:37:46.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d57fff640 10 monclient(hunting): get_auth_request con 0x7f9d5813c3e0 auth_method 0 2024-04-26T00:37:46.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d57fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:46.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d57fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:46.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d57fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:46.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d57fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:46.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d57fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:46.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d57fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:46.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d57fff640 10 monclient(hunting): handle_auth_done global_id 204248 payload 274 2024-04-26T00:37:46.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d57fff640 1 -- 172.21.15.37:0/2644017850 >> v2:172.21.15.45:3300/0 conn(0x7f9d580a4170 msgr2=0x7f9d580a45e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:37:46.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d57fff640 1 --2- 172.21.15.37:0/2644017850 >> v2:172.21.15.45:3300/0 conn(0x7f9d580a4170 0x7f9d580a45e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:46.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d57fff640 1 -- 172.21.15.37:0/2644017850 >> v1:172.21.15.37:6789/0 conn(0x7f9d580a3d90 legacy=0x7f9d5809bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:37:46.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d57fff640 10 monclient: _finish_hunting 0 2024-04-26T00:37:46.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d57fff640 1 monclient: found mon.noname-b 2024-04-26T00:37:46.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:37:46.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d57fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:37:46.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.047+0000 7f9d57fff640 1 -- 172.21.15.37:0/2644017850 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d5809b560 con 0x7f9d5813c3e0 2024-04-26T00:37:46.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.051+0000 7f9d57fff640 1 -- 172.21.15.37:0/2644017850 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d5813c3e0 msgr2=0x7f9d5813e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:46.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.051+0000 7f9d57fff640 1 --2- 172.21.15.37:0/2644017850 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d5813c3e0 0x7f9d5813e7d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9d50009d90 tx=0x7f9d5001c810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d19f7d9d35650de4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:46.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.051+0000 7f9d567fc640 1 -- 172.21.15.37:0/2644017850 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d500b2070 con 0x7f9d5813c3e0 2024-04-26T00:37:46.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.051+0000 7f9d567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:46.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.051+0000 7f9d567fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:37:46.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.051+0000 7f9d567fc640 10 monclient: dump: 2024-04-26T00:37:46.051 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:46.051 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:46.051 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:46.051 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:46.051 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:46.052 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:46.052 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:46.052 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:46.052 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:46.052 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:46.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.051+0000 7f9d567fc640 10 monclient: _finish_auth 0 2024-04-26T00:37:46.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.051+0000 7f9d567fc640 10 monclient: _check_auth_tickets 2024-04-26T00:37:46.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.051+0000 7f9d567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:37:46.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.051+0000 7f9d567fc640 1 -- 172.21.15.37:0/2644017850 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d500a5880 con 0x7f9d5813c3e0 2024-04-26T00:37:46.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.051+0000 7f9d567fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:37:46.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.051+0000 7f9d567fc640 1 -- 172.21.15.37:0/2644017850 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d500a5c50 con 0x7f9d5813c3e0 2024-04-26T00:37:46.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.051+0000 7f9d5dbc8640 5 monclient: authenticate success, global_id 204248 2024-04-26T00:37:46.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.051+0000 7f9d5dbc8640 10 monclient: get_monmap_and_config success 2024-04-26T00:37:46.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.051+0000 7f9d5dbc8640 4 set_mon_vals no callback set 2024-04-26T00:37:46.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.051+0000 7f9d567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:46.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.051+0000 7f9d567fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:37:46.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.051+0000 7f9d567fc640 10 monclient: dump: 2024-04-26T00:37:46.053 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:46.053 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:46.053 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:46.053 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:46.053 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:46.053 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:46.053 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:46.053 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:46.053 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:46.053 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:46.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d5dbc8640 10 monclient: shutdown 2024-04-26T00:37:46.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d5dbc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:37:46.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d5dbc8640 1 -- 172.21.15.37:0/2644017850 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d5813c3e0 msgr2=0x7f9d5813e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:46.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d5dbc8640 1 --2- 172.21.15.37:0/2644017850 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d5813c3e0 0x7f9d5813e7d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9d50009d90 tx=0x7f9d5001c810 comp rx=0 tx=0).stop 2024-04-26T00:37:46.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d5dbc8640 1 -- 172.21.15.37:0/2644017850 shutdown_connections 2024-04-26T00:37:46.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d5dbc8640 1 --2- 172.21.15.37:0/2644017850 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d5813c3e0 0x7f9d5813e7d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:46.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d5dbc8640 1 --2- 172.21.15.37:0/2644017850 >> v2:172.21.15.45:3300/0 conn(0x7f9d580a4170 0x7f9d580a45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:46.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d5dbc8640 1 -- 172.21.15.37:0/2644017850 >> 172.21.15.37:0/2644017850 conn(0x7f9d580aa0a0 msgr2=0x7f9d580a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:37:46.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d5dbc8640 1 -- 172.21.15.37:0/2644017850 shutdown_connections 2024-04-26T00:37:46.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d5dbc8640 1 -- 172.21.15.37:0/2644017850 wait complete. 2024-04-26T00:37:46.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d5dbc8640 10 monclient: build_initial_monmap 2024-04-26T00:37:46.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d5dbc8640 10 monclient: monmap: 2024-04-26T00:37:46.060 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:37:46.060 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:37:46.060 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:37:46.060 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:37:46.060 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:37:46.060 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:37:46.060 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:37:46.060 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:37:46.060 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:37:46.060 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:46.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d5dbc8640 1 Processor -- start 2024-04-26T00:37:46.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d5dbc8640 1 -- start start 2024-04-26T00:37:46.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d5dbc8640 10 monclient: init 2024-04-26T00:37:46.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d5dbc8640 10 monclient: _reopen_session rank -1 2024-04-26T00:37:46.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d5dbc8640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:37:46.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d5dbc8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d580a3d90 0x7f9d581cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:46.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d5dbc8640 10 monclient(hunting): picked mon.noname-c con 0x7f9d580a3d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:37:46.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d5dbc8640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d580a4170 0x7f9d581cf490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:46.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d5dbc8640 10 monclient(hunting): picked mon.noname-a con 0x7f9d580a4170 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:37:46.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d5dbc8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d5813c3e0 0x7f9d58139ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:46.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d5dbc8640 10 monclient(hunting): picked mon.noname-b con 0x7f9d5813c3e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:37:46.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d5dbc8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:46.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d5dbc8640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9d581418c0 con 0x7f9d580a4170 2024-04-26T00:37:46.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d5dbc8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:46.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d5dbc8640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9d58141740 con 0x7f9d5813c3e0 2024-04-26T00:37:46.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d5dbc8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:46.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d5dbc8640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9d58141a40 con 0x7f9d580a3d90 2024-04-26T00:37:46.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d5dbc8640 10 monclient(hunting): _renew_subs 2024-04-26T00:37:46.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d5dbc8640 10 monclient(hunting): authenticate will time out at 2047.362149s 2024-04-26T00:37:46.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d577fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d580a3d90 0x7f9d581cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:46.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d56ffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d580a4170 0x7f9d581cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:46.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d57fff640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d5813c3e0 msgr2=0x7f9d58139ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:37:46.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d577fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d580a3d90 0x7f9d581cecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:46112/0 (socket says 172.21.15.37:46112) 2024-04-26T00:37:46.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d577fe640 1 -- 172.21.15.37:0/3994737327 learned_addr learned my addr 172.21.15.37:0/3994737327 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:37:46.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d56ffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d580a4170 0x7f9d581cf490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:49586/0 (socket says 172.21.15.37:49586) 2024-04-26T00:37:46.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d577fe640 10 monclient(hunting): get_auth_request con 0x7f9d580a3d90 auth_method 0 2024-04-26T00:37:46.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:46.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d577fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:46.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d577fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:46.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d57fff640 1 --2- 172.21.15.37:0/3994737327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d5813c3e0 0x7f9d58139ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:37:46.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d56ffd640 10 monclient(hunting): get_auth_request con 0x7f9d580a4170 auth_method 0 2024-04-26T00:37:46.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d56ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:46.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d56ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:46.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:46.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:46.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:46.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:46.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d56ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:46.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d56ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:46.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d56ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:46.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d577fe640 10 monclient(hunting): handle_auth_done global_id 204251 payload 824 2024-04-26T00:37:46.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d577fe640 1 -- 172.21.15.37:0/3994737327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d5813c3e0 msgr2=0x7f9d58139ba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:37:46.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d577fe640 1 --2- 172.21.15.37:0/3994737327 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d5813c3e0 0x7f9d58139ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:46.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d577fe640 1 -- 172.21.15.37:0/3994737327 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d580a4170 msgr2=0x7f9d581cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:46.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d577fe640 1 --2- 172.21.15.37:0/3994737327 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d580a4170 0x7f9d581cf490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:46.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d577fe640 10 monclient: _finish_hunting 0 2024-04-26T00:37:46.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d577fe640 1 monclient: found mon.noname-c 2024-04-26T00:37:46.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:37:46.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d577fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:37:46.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d577fe640 1 -- 172.21.15.37:0/3994737327 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d5813c7c0 con 0x7f9d580a3d90 2024-04-26T00:37:46.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.055+0000 7f9d56ffd640 1 --2- 172.21.15.37:0/3994737327 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d580a4170 0x7f9d581cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:37:46.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d577fe640 1 -- 172.21.15.37:0/3994737327 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d580a3d90 msgr2=0x7f9d581cecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:46.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d577fe640 1 --2- 172.21.15.37:0/3994737327 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d580a3d90 0x7f9d581cecc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9d4800c4a0 tx=0x7f9d4800c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:46.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d3ffff640 1 -- 172.21.15.37:0/3994737327 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d48017070 con 0x7f9d580a3d90 2024-04-26T00:37:46.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d3ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:46.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d3ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:37:46.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d3ffff640 10 monclient: dump: 2024-04-26T00:37:46.064 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:46.064 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:46.064 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:46.064 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:46.064 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:46.064 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:46.064 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:46.064 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:46.065 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:46.065 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:46.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d3ffff640 10 monclient: _finish_auth 0 2024-04-26T00:37:46.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d3ffff640 10 monclient: _check_auth_tickets 2024-04-26T00:37:46.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:37:46.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d5dbc8640 5 monclient: authenticate success, global_id 204251 2024-04-26T00:37:46.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d5dbc8640 10 monclient: _renew_subs 2024-04-26T00:37:46.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d3ffff640 1 -- 172.21.15.37:0/3994737327 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d48007070 con 0x7f9d580a3d90 2024-04-26T00:37:46.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d5dbc8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:37:46.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d5dbc8640 1 -- 172.21.15.37:0/3994737327 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d5809ada0 con 0x7f9d580a3d90 2024-04-26T00:37:46.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d3ffff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:37:46.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d3ffff640 1 -- 172.21.15.37:0/3994737327 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d48007460 con 0x7f9d580a3d90 2024-04-26T00:37:46.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d3ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:46.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d3ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:37:46.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d3ffff640 10 monclient: dump: 2024-04-26T00:37:46.066 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:46.066 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:46.066 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:46.066 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:46.066 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:46.066 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:46.066 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:46.066 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:46.066 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:46.066 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:46.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d5dbc8640 10 monclient: _renew_subs 2024-04-26T00:37:46.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d557fa640 4 set_mon_vals no callback set 2024-04-26T00:37:46.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d5dbc8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:37:46.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d5dbc8640 1 -- 172.21.15.37:0/3994737327 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d581212f0 con 0x7f9d580a3d90 2024-04-26T00:37:46.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d3ffff640 1 -- 172.21.15.37:0/3994737327 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9d4801a070 con 0x7f9d580a3d90 2024-04-26T00:37:46.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d5dbc8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:46.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d5dbc8640 1 --2- 172.21.15.37:0/3994737327 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d581cf9d0 0x7f9d5802c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:46.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d5dbc8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:46.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d5dbc8640 1 -- 172.21.15.37:0/3994737327 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9d5811fbe0 con 0x7f9d581cf9d0 2024-04-26T00:37:46.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d56ffd640 1 --2- 172.21.15.37:0/3994737327 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d581cf9d0 0x7f9d5802c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:46.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d3ffff640 1 --2- 172.21.15.37:0/3994737327 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9d44035eb0 0x7f9d44038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:46.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d56ffd640 10 monclient: get_auth_request con 0x7f9d581cf9d0 auth_method 0 2024-04-26T00:37:46.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d56ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:46.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d56ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:46.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:46.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d3ffff640 1 -- 172.21.15.37:0/3994737327 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(691..691 src has 370..691) ==== 3927+0+0 (secure 0 0 0) 0x7f9d48047f60 con 0x7f9d580a3d90 2024-04-26T00:37:46.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d56ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:46.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d56ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:46.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d56ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:46.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d56ffd640 10 monclient(hunting): handle_auth_done global_id 144870 payload 824 2024-04-26T00:37:46.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d56ffd640 1 -- 172.21.15.37:0/3994737327 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9d58120f70 con 0x7f9d581cf9d0 2024-04-26T00:37:46.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d57fff640 1 --2- 172.21.15.37:0/3994737327 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9d44035eb0 0x7f9d44038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:46.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.059+0000 7f9d57fff640 10 monclient: get_auth_request con 0x7f9d44035eb0 auth_method 0 2024-04-26T00:37:46.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.063+0000 7f9d56ffd640 1 -- 172.21.15.37:0/3994737327 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d581cf9d0 msgr2=0x7f9d5802c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:46.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.063+0000 7f9d56ffd640 1 --2- 172.21.15.37:0/3994737327 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d581cf9d0 0x7f9d5802c000 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f9d58137970 tx=0x7f9d4c005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:46.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.063+0000 7f9d3ffff640 1 -- 172.21.15.37:0/3994737327 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c016070 con 0x7f9d581cf9d0 2024-04-26T00:37:46.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.063+0000 7f9d3ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:46.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.063+0000 7f9d3ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:37:46.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.063+0000 7f9d3ffff640 10 monclient: dump: 2024-04-26T00:37:46.070 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:46.071 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:46.071 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:46.071 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:46.071 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:46.071 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:46.071 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:46.071 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:46.071 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:46.071 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:46.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.063+0000 7f9d3ffff640 1 -- 172.21.15.37:0/3994737327 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9d4c00ecd0 con 0x7f9d581cf9d0 2024-04-26T00:37:46.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.063+0000 7f9d3ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:46.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.063+0000 7f9d3ffff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:37:46.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.063+0000 7f9d3ffff640 1 -- 172.21.15.37:0/3994737327 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d581cf9d0 msgr2=0x7f9d5802c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:46.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.063+0000 7f9d3ffff640 1 --2- 172.21.15.37:0/3994737327 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d581cf9d0 0x7f9d5802c000 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f9d58137970 tx=0x7f9d4c005850 comp rx=0 tx=0).stop 2024-04-26T00:37:46.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.063+0000 7f9d3ffff640 1 -- 172.21.15.37:0/3994737327 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d581cf9d0 msgr2=0x7f9d5802c000 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:37:46.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.063+0000 7f9d3ffff640 1 --2- 172.21.15.37:0/3994737327 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d581cf9d0 0x7f9d5802c000 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:46.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.063+0000 7f9d57fff640 1 --2- 172.21.15.37:0/3994737327 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9d44035eb0 0x7f9d44038370 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f9d580a57e0 tx=0x7f9d500a7070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:46.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.067+0000 7f9d5dbc8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:37:46.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.067+0000 7f9d5dbc8640 1 --2- 172.21.15.37:0/3994737327 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d5802c540 0x7f9d5802c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:46.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.067+0000 7f9d5dbc8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:46.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.067+0000 7f9d5dbc8640 1 -- 172.21.15.37:0/3994737327 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9d5804f510 con 0x7f9d5802c540 2024-04-26T00:37:46.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.067+0000 7f9d56ffd640 1 --2- 172.21.15.37:0/3994737327 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d5802c540 0x7f9d5802c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:46.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.067+0000 7f9d56ffd640 10 monclient: get_auth_request con 0x7f9d5802c540 auth_method 0 2024-04-26T00:37:46.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.067+0000 7f9d56ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:46.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.067+0000 7f9d56ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:46.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.067+0000 7f9d56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:46.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.067+0000 7f9d56ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:46.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.067+0000 7f9d56ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:46.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.067+0000 7f9d56ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:46.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.067+0000 7f9d56ffd640 10 monclient(hunting): handle_auth_done global_id 144873 payload 824 2024-04-26T00:37:46.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.067+0000 7f9d56ffd640 1 -- 172.21.15.37:0/3994737327 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9d580a1ea0 con 0x7f9d5802c540 2024-04-26T00:37:46.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.071+0000 7f9d56ffd640 1 -- 172.21.15.37:0/3994737327 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d5802c540 msgr2=0x7f9d5802c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:46.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.071+0000 7f9d56ffd640 1 --2- 172.21.15.37:0/3994737327 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d5802c540 0x7f9d5802c950 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c018000 tx=0x7f9d4c0059e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:46.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.071+0000 7f9d3ffff640 1 -- 172.21.15.37:0/3994737327 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c016040 con 0x7f9d5802c540 2024-04-26T00:37:46.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.071+0000 7f9d3ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:46.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.071+0000 7f9d3ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:37:46.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.071+0000 7f9d3ffff640 10 monclient: dump: 2024-04-26T00:37:46.073 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:46.073 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:46.073 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:46.073 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:46.073 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:46.073 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:46.073 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:46.074 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:46.074 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:46.074 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:46.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.071+0000 7f9d3ffff640 1 -- 172.21.15.37:0/3994737327 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7f9d4c01b660 con 0x7f9d5802c540 2024-04-26T00:37:46.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.071+0000 7f9d3ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:37:46.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.071+0000 7f9d3ffff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:37:46.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.071+0000 7f9d3ffff640 1 -- 172.21.15.37:0/3994737327 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d5802c540 msgr2=0x7f9d5802c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:46.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.071+0000 7f9d3ffff640 1 --2- 172.21.15.37:0/3994737327 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d5802c540 0x7f9d5802c950 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c018000 tx=0x7f9d4c0059e0 comp rx=0 tx=0).stop 2024-04-26T00:37:46.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.071+0000 7f9d3ffff640 1 -- 172.21.15.37:0/3994737327 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d5802c540 msgr2=0x7f9d5802c950 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:37:46.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.071+0000 7f9d3ffff640 1 --2- 172.21.15.37:0/3994737327 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d5802c540 0x7f9d5802c950 secure :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c018000 tx=0x7f9d4c0059e0 comp rx=0 tx=0).stop 2024-04-26T00:37:46.074 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:37:46.075 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:37:46.075 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:37:46.075 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:37:46.075 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 136, 2024-04-26T00:37:46.075 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 352615, 2024-04-26T00:37:46.075 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:37:46.075 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:37:46.075 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:37:46.075 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:46.075 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:37:46.075 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:37:46.075 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:37:46.075 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:37:46.075 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:37:46.076 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:37:46.076 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:37:46.076 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:37:46.076 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:37:46.076 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:37:46.076 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:37:46.076 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:37:46.076 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:37:46.076 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:37:46.076 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:37:46.076 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:46.076 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:37:46.076 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:37:46.076 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:37:46.076 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:37:46.077 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:37:46.077 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:37:46.077 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:37:46.077 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:37:46.077 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:37:46.077 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:37:46.077 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:37:46.077 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:37:46.077 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:37:46.077 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:46.077 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:46.077 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:37:46.077 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:37:46.077 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:37:46.077 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:37:46.078 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:37:46.078 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:37:46.078 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:37:46.078 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:37:46.078 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:37:46.078 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:37:46.078 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:37:46.078 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:37:46.078 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:37:46.078 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:37:46.078 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:37:46.078 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:37:46.078 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:37:46.078 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:37:46.079 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:37:46.079 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:37:46.079 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:37:46.079 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:37:46.079 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:37:46.079 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:37:46.079 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:37:46.079 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:37:46.079 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:37:46.079 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:37:46.079 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:46.079 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:37:46.079 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:46.079 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:37:46.079 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:46.080 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:37:46.080 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:37:46.080 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:37:46.080 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:37:46.080 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:46.080 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:37:46.080 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:37:46.080 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:46.080 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:46.080 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:46.080 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:37:46.080 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:37:46.080 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:46.080 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:46.080 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:46.081 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:46.081 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:37:46.081 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:37:46.081 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:37:46.081 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:37:46.081 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:37:46.081 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:46.081 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:46.081 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:37:46.081 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:37:46.081 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:37:46.081 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:37:46.081 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:46.081 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:37:46.081 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:37:46.082 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:46.082 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:46.082 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:46.082 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:37:46.082 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:37:46.082 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:46.082 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:46.082 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:46.082 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:46.082 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:37:46.082 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:37:46.082 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:37:46.082 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:37:46.082 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:37:46.083 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:46.083 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:46.083 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:37:46.083 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:37:46.083 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:37:46.083 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:37:46.083 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:46.083 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:37:46.083 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:37:46.083 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:46.083 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:46.083 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:46.083 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:37:46.083 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:37:46.083 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:46.083 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:46.084 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:46.084 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:46.084 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:37:46.084 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:37:46.084 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:37:46.084 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:37:46.084 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:37:46.084 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:46.084 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:46.084 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:46.084 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:37:46.084 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:37:46.084 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:46.084 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:37:46.084 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:37:46.085 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:37:46.085 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:46.085 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:46.085 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:37:46.085 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:46.085 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:37:46.085 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:37:46.085 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:37:46.085 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:46.085 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:46.085 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:37:46.085 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:46.085 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:37:46.085 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:37:46.085 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:37:46.085 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:46.086 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:46.086 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:37:46.086 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:46.086 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:37:46.086 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:37:46.086 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:37:46.086 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:46.086 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:46.086 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:46.086 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:37:46.086 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:37:46.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.071+0000 7f9d5dbc8640 1 -- 172.21.15.37:0/3994737327 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9d44035eb0 msgr2=0x7f9d44038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:46.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.071+0000 7f9d5dbc8640 1 --2- 172.21.15.37:0/3994737327 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9d44035eb0 0x7f9d44038370 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f9d580a57e0 tx=0x7f9d500a7070 comp rx=0 tx=0).stop 2024-04-26T00:37:46.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.071+0000 7f9d5dbc8640 10 monclient: shutdown 2024-04-26T00:37:46.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.071+0000 7f9d5dbc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:37:46.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.071+0000 7f9d5dbc8640 1 -- 172.21.15.37:0/3994737327 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d580a3d90 msgr2=0x7f9d581cecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:46.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.071+0000 7f9d5dbc8640 1 --2- 172.21.15.37:0/3994737327 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d580a3d90 0x7f9d581cecc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9d4800c4a0 tx=0x7f9d4800c960 comp rx=0 tx=0).stop 2024-04-26T00:37:46.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.071+0000 7f9d57fff640 1 -- 172.21.15.37:0/3994737327 reap_dead start 2024-04-26T00:37:46.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.071+0000 7f9d57fff640 1 -- 172.21.15.37:0/3994737327 reap_dead start 2024-04-26T00:37:46.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.071+0000 7f9d5dbc8640 1 -- 172.21.15.37:0/3994737327 shutdown_connections 2024-04-26T00:37:46.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.071+0000 7f9d5dbc8640 1 -- 172.21.15.37:0/3994737327 >> 172.21.15.37:0/3994737327 conn(0x7f9d580aa0a0 msgr2=0x7f9d5813d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:37:46.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.071+0000 7f9d5dbc8640 1 -- 172.21.15.37:0/3994737327 shutdown_connections 2024-04-26T00:37:46.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.071+0000 7f9d5dbc8640 1 -- 172.21.15.37:0/3994737327 wait complete. 2024-04-26T00:37:46.089 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:37:46.089 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-04-26T00:37:46.089 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:37:46.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.207+0000 7f08998f8640 10 monclient: get_monmap_and_config 2024-04-26T00:37:46.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.207+0000 7f08998f8640 10 monclient: build_initial_monmap 2024-04-26T00:37:46.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.207+0000 7f08998f8640 10 monclient: monmap: 2024-04-26T00:37:46.204 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:37:46.204 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:46.204 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:37:46.208513+0000 2024-04-26T00:37:46.204 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:37:46.208513+0000 2024-04-26T00:37:46.204 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:37:46.204 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:46.204 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:37:46.205 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:37:46.205 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:37:46.205 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:37:46.205 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:37:46.205 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:46.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.207+0000 7f08998f8640 1 Processor -- start 2024-04-26T00:37:46.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.207+0000 7f08998f8640 1 -- start start 2024-04-26T00:37:46.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.207+0000 7f08998f8640 10 monclient: init 2024-04-26T00:37:46.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.207+0000 7f08998f8640 10 monclient: _reopen_session rank -1 2024-04-26T00:37:46.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.207+0000 7f08998f8640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-04-26T00:37:46.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.207+0000 7f08998f8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f089409e220 addr v1:172.21.15.37:6789/0 2024-04-26T00:37:46.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.207+0000 7f08998f8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f08940a14f0 0x7f08940a0360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:46.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.207+0000 7f08998f8640 10 monclient(hunting): picked mon.noname-b con 0x7f08940a14f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:37:46.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.207+0000 7f08998f8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f089409e600 addr v1:172.21.15.45:6789/0 2024-04-26T00:37:46.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.207+0000 7f08998f8640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08940a12b0 con 0x7f089409e220 2024-04-26T00:37:46.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.207+0000 7f08998f8640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f089409da20 con 0x7f089409e600 2024-04-26T00:37:46.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.207+0000 7f08998f8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:46.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.207+0000 7f08998f8640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f089409cd10 con 0x7f08940a14f0 2024-04-26T00:37:46.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.207+0000 7f08998f8640 10 monclient(hunting): _renew_subs 2024-04-26T00:37:46.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.207+0000 7f08998f8640 10 monclient(hunting): authenticate will time out at 2047.514398s 2024-04-26T00:37:46.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.207+0000 7f08937fe640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f089409e600 legacy=0x7f089413aa60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:37:46.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.207+0000 7f088a7fc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f08940a14f0 0x7f08940a0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:46.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.207+0000 7f0892ffd640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f089409e220 0x7f089409fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:57056/0 (socket says 172.21.15.37:57056) 2024-04-26T00:37:46.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.207+0000 7f0892ffd640 1 -- 172.21.15.37:0/1819252255 learned_addr learned my addr 172.21.15.37:0/1819252255 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:37:46.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.207+0000 7f088a7fc640 10 monclient(hunting): get_auth_request con 0x7f08940a14f0 auth_method 0 2024-04-26T00:37:46.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.207+0000 7f088a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:46.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.207+0000 7f088a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:46.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.207+0000 7f088a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:46.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.207+0000 7f088a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:46.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.207+0000 7f088a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:46.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.207+0000 7f088a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:46.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.207+0000 7f088a7fc640 10 monclient(hunting): handle_auth_done global_id 204254 payload 274 2024-04-26T00:37:46.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.207+0000 7f088a7fc640 1 -- 172.21.15.37:0/1819252255 >> v1:172.21.15.45:6789/0 conn(0x7f089409e600 legacy=0x7f089413aa60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:37:46.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.207+0000 7f088a7fc640 1 -- 172.21.15.37:0/1819252255 >> v1:172.21.15.37:6789/0 conn(0x7f089409e220 legacy=0x7f089409fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:37:46.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.207+0000 7f088a7fc640 10 monclient: _finish_hunting 0 2024-04-26T00:37:46.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.207+0000 7f088a7fc640 1 monclient: found mon.noname-b 2024-04-26T00:37:46.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.207+0000 7f088a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:37:46.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.207+0000 7f088a7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:37:46.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.207+0000 7f088a7fc640 1 -- 172.21.15.37:0/1819252255 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f089409dd00 con 0x7f08940a14f0 2024-04-26T00:37:46.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.211+0000 7f088a7fc640 1 -- 172.21.15.37:0/1819252255 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f08940a14f0 msgr2=0x7f08940a0360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:46.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.211+0000 7f088a7fc640 1 --2- 172.21.15.37:0/1819252255 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f08940a14f0 0x7f08940a0360 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f087c0097f0 tx=0x7f087c01c320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d41b73f75c4a961a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:46.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.211+0000 7f08927fc640 1 -- 172.21.15.37:0/1819252255 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087c0b0070 con 0x7f08940a14f0 2024-04-26T00:37:46.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.211+0000 7f08927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:46.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.211+0000 7f08927fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:37:46.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.211+0000 7f08927fc640 10 monclient: dump: 2024-04-26T00:37:46.211 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:46.211 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:46.211 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:46.211 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:46.211 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:46.211 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:46.211 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:46.211 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:46.211 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:46.211 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:46.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.211+0000 7f08927fc640 10 monclient: _finish_auth 0 2024-04-26T00:37:46.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.211+0000 7f08927fc640 10 monclient: _check_auth_tickets 2024-04-26T00:37:46.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.211+0000 7f08927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:37:46.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.211+0000 7f08927fc640 1 -- 172.21.15.37:0/1819252255 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08940a0ab0 con 0x7f08940a14f0 2024-04-26T00:37:46.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.211+0000 7f08927fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:37:46.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.211+0000 7f08927fc640 1 -- 172.21.15.37:0/1819252255 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087c0ab620 con 0x7f08940a14f0 2024-04-26T00:37:46.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.211+0000 7f08998f8640 5 monclient: authenticate success, global_id 204254 2024-04-26T00:37:46.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.211+0000 7f08998f8640 10 monclient: get_monmap_and_config success 2024-04-26T00:37:46.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.211+0000 7f08998f8640 4 set_mon_vals no callback set 2024-04-26T00:37:46.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.211+0000 7f08927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:46.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.211+0000 7f08927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:37:46.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.211+0000 7f08927fc640 10 monclient: dump: 2024-04-26T00:37:46.212 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:46.212 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:46.212 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:46.212 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:46.212 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:46.213 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:46.213 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:46.213 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:46.213 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:46.213 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:46.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.211+0000 7f08998f8640 10 monclient: shutdown 2024-04-26T00:37:46.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.211+0000 7f08998f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:37:46.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.211+0000 7f08998f8640 1 -- 172.21.15.37:0/1819252255 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f08940a14f0 msgr2=0x7f08940a0360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:46.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.211+0000 7f08998f8640 1 --2- 172.21.15.37:0/1819252255 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f08940a14f0 0x7f08940a0360 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f087c0097f0 tx=0x7f087c01c320 comp rx=0 tx=0).stop 2024-04-26T00:37:46.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.211+0000 7f08998f8640 1 -- 172.21.15.37:0/1819252255 shutdown_connections 2024-04-26T00:37:46.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.211+0000 7f08998f8640 1 --2- 172.21.15.37:0/1819252255 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f08940a14f0 0x7f08940a0360 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:46.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.211+0000 7f08998f8640 1 -- 172.21.15.37:0/1819252255 >> 172.21.15.37:0/1819252255 conn(0x7f08940a1d50 msgr2=0x7f08940a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:37:46.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f08998f8640 1 -- 172.21.15.37:0/1819252255 shutdown_connections 2024-04-26T00:37:46.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f08998f8640 1 -- 172.21.15.37:0/1819252255 wait complete. 2024-04-26T00:37:46.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f08998f8640 10 monclient: build_initial_monmap 2024-04-26T00:37:46.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f08998f8640 10 monclient: monmap: 2024-04-26T00:37:46.218 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:37:46.218 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:37:46.218 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:37:46.218 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:37:46.218 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:37:46.218 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:37:46.218 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:37:46.218 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:37:46.219 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:37:46.219 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:46.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f08998f8640 1 Processor -- start 2024-04-26T00:37:46.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f08998f8640 1 -- start start 2024-04-26T00:37:46.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f08998f8640 10 monclient: init 2024-04-26T00:37:46.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f08998f8640 10 monclient: _reopen_session rank -1 2024-04-26T00:37:46.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f08998f8640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:37:46.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f08998f8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f089409e220 0x7f089413aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:46.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f08998f8640 10 monclient(hunting): picked mon.noname-c con 0x7f089409e220 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:37:46.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f08998f8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f089409e600 0x7f0894135c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:46.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f08998f8640 10 monclient(hunting): picked mon.noname-b con 0x7f089409e600 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:37:46.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f08998f8640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f08940a14f0 0x7f089411b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:46.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f08998f8640 10 monclient(hunting): picked mon.noname-a con 0x7f08940a14f0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:37:46.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f08998f8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:46.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f08998f8640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f089413f840 con 0x7f08940a14f0 2024-04-26T00:37:46.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f08998f8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:46.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f08998f8640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f089413f6c0 con 0x7f089409e600 2024-04-26T00:37:46.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f08998f8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:46.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f08998f8640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f089413f9c0 con 0x7f089409e220 2024-04-26T00:37:46.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f08998f8640 10 monclient(hunting): _renew_subs 2024-04-26T00:37:46.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f08998f8640 10 monclient(hunting): authenticate will time out at 2047.521552s 2024-04-26T00:37:46.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f0892ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f089409e220 0x7f089413aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:46.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f088a7fc640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f089409e600 msgr2=0x7f0894135c70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:37:46.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f08937fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f08940a14f0 0x7f089411b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:46.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f0892ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f089409e220 0x7f089413aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:46132/0 (socket says 172.21.15.37:46132) 2024-04-26T00:37:46.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f0892ffd640 1 -- 172.21.15.37:0/4255576447 learned_addr learned my addr 172.21.15.37:0/4255576447 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:37:46.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f0892ffd640 10 monclient(hunting): get_auth_request con 0x7f089409e220 auth_method 0 2024-04-26T00:37:46.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f0892ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:46.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f0892ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:46.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f0892ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:46.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f088a7fc640 1 --2- 172.21.15.37:0/4255576447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f089409e600 0x7f0894135c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:37:46.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f08937fe640 10 monclient(hunting): get_auth_request con 0x7f08940a14f0 auth_method 0 2024-04-26T00:37:46.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f08937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:46.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f08937fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:46.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f08937fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:46.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f0892ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:46.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f0892ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:46.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f0892ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:46.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f08937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:46.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f08937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:46.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f08937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:46.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f0892ffd640 10 monclient(hunting): handle_auth_done global_id 204257 payload 824 2024-04-26T00:37:46.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f0892ffd640 1 -- 172.21.15.37:0/4255576447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f089409e600 msgr2=0x7f0894135c70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:37:46.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f0892ffd640 1 --2- 172.21.15.37:0/4255576447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f089409e600 0x7f0894135c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:46.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f0892ffd640 1 -- 172.21.15.37:0/4255576447 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f08940a14f0 msgr2=0x7f089411b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:46.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f0892ffd640 1 --2- 172.21.15.37:0/4255576447 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f08940a14f0 0x7f089411b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:46.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f0892ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:37:46.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f0892ffd640 1 monclient: found mon.noname-c 2024-04-26T00:37:46.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f0892ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:37:46.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f0892ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:37:46.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f0892ffd640 1 -- 172.21.15.37:0/4255576447 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f089411e300 con 0x7f089409e220 2024-04-26T00:37:46.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.215+0000 7f08937fe640 1 --2- 172.21.15.37:0/4255576447 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f08940a14f0 0x7f089411b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:37:46.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f0892ffd640 1 -- 172.21.15.37:0/4255576447 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f089409e220 msgr2=0x7f089413aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:46.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f0892ffd640 1 --2- 172.21.15.37:0/4255576447 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f089409e220 0x7f089413aed0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f088400c470 tx=0x7f088400c930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:46.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f0890ff9640 1 -- 172.21.15.37:0/4255576447 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0884007070 con 0x7f089409e220 2024-04-26T00:37:46.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f0890ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:46.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f0890ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:37:46.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f0890ff9640 10 monclient: dump: 2024-04-26T00:37:46.223 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:46.223 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:46.223 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:46.223 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:46.223 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:46.223 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:46.223 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:46.223 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:46.223 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:46.223 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:46.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f0890ff9640 10 monclient: _finish_auth 0 2024-04-26T00:37:46.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f0890ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:37:46.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f0890ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:37:46.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f08998f8640 5 monclient: authenticate success, global_id 204257 2024-04-26T00:37:46.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f0890ff9640 1 -- 172.21.15.37:0/4255576447 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0884007210 con 0x7f089409e220 2024-04-26T00:37:46.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f0890ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:37:46.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f08998f8640 10 monclient: _renew_subs 2024-04-26T00:37:46.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f0890ff9640 1 -- 172.21.15.37:0/4255576447 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0884007600 con 0x7f089409e220 2024-04-26T00:37:46.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f08998f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:37:46.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f08998f8640 1 -- 172.21.15.37:0/4255576447 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f089411e5e0 con 0x7f089409e220 2024-04-26T00:37:46.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f0890ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:46.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f0890ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:37:46.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f0890ff9640 10 monclient: dump: 2024-04-26T00:37:46.224 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:46.224 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:46.224 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:46.224 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:46.224 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:46.225 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:46.225 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:46.225 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:46.225 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:46.225 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:46.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f0891ffb640 4 set_mon_vals no callback set 2024-04-26T00:37:46.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f08998f8640 10 monclient: _renew_subs 2024-04-26T00:37:46.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f08998f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:37:46.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f08998f8640 1 -- 172.21.15.37:0/4255576447 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f08940a3f70 con 0x7f089409e220 2024-04-26T00:37:46.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f08998f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:46.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f08998f8640 1 --2- 172.21.15.37:0/4255576447 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f08941cd110 0x7f089411bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:46.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f08998f8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:46.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f08998f8640 1 -- 172.21.15.37:0/4255576447 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f08940a0ab0 con 0x7f08941cd110 2024-04-26T00:37:46.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f088a7fc640 1 --2- 172.21.15.37:0/4255576447 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f08941cd110 0x7f089411bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:46.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f088a7fc640 10 monclient: get_auth_request con 0x7f08941cd110 auth_method 0 2024-04-26T00:37:46.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f088a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:46.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f088a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:46.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f088a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:46.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f0890ff9640 1 -- 172.21.15.37:0/4255576447 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f088401b400 con 0x7f089409e220 2024-04-26T00:37:46.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f088a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:46.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f088a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:46.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f088a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:46.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f088a7fc640 10 monclient(hunting): handle_auth_done global_id 204260 payload 824 2024-04-26T00:37:46.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f088a7fc640 1 -- 172.21.15.37:0/4255576447 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f089411dee0 con 0x7f08941cd110 2024-04-26T00:37:46.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f0890ff9640 1 --2- 172.21.15.37:0/4255576447 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0874035eb0 0x7f0874038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:46.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f0890ff9640 1 -- 172.21.15.37:0/4255576447 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(691..691 src has 370..691) ==== 3927+0+0 (secure 0 0 0) 0x7f08840487b0 con 0x7f089409e220 2024-04-26T00:37:46.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f08937fe640 1 --2- 172.21.15.37:0/4255576447 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0874035eb0 0x7f0874038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:46.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.219+0000 7f08937fe640 10 monclient: get_auth_request con 0x7f0874035eb0 auth_method 0 2024-04-26T00:37:46.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.223+0000 7f088a7fc640 1 -- 172.21.15.37:0/4255576447 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f08941cd110 msgr2=0x7f089411bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:46.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.223+0000 7f088a7fc640 1 --2- 172.21.15.37:0/4255576447 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f08941cd110 0x7f089411bde0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f087c0ad000 tx=0x7f087c0190c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:46.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.223+0000 7f0890ff9640 1 -- 172.21.15.37:0/4255576447 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087c0bb070 con 0x7f08941cd110 2024-04-26T00:37:46.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.223+0000 7f0890ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:46.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.223+0000 7f0890ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:37:46.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.223+0000 7f0890ff9640 10 monclient: dump: 2024-04-26T00:37:46.229 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:46.229 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:46.229 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:46.229 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:46.230 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:46.230 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:46.230 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:46.230 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:46.230 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:46.230 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:46.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.223+0000 7f0890ff9640 1 -- 172.21.15.37:0/4255576447 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f087c0abd30 con 0x7f08941cd110 2024-04-26T00:37:46.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.223+0000 7f0890ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:37:46.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.223+0000 7f0890ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:37:46.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.223+0000 7f0890ff9640 1 -- 172.21.15.37:0/4255576447 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f08941cd110 msgr2=0x7f089411bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:46.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.223+0000 7f0890ff9640 1 --2- 172.21.15.37:0/4255576447 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f08941cd110 0x7f089411bde0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f087c0ad000 tx=0x7f087c0190c0 comp rx=0 tx=0).stop 2024-04-26T00:37:46.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.223+0000 7f0890ff9640 1 -- 172.21.15.37:0/4255576447 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f08941cd110 msgr2=0x7f089411bde0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:37:46.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.223+0000 7f0890ff9640 1 --2- 172.21.15.37:0/4255576447 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f08941cd110 0x7f089411bde0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:37:46.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.223+0000 7f08937fe640 1 --2- 172.21.15.37:0/4255576447 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0874035eb0 0x7f0874038370 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f088c003bd0 tx=0x7f088c00d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:46.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.227+0000 7f08998f8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:37:46.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.227+0000 7f08998f8640 1 --2- 172.21.15.37:0/4255576447 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f089411c320 0x7f0894136c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:37:46.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.227+0000 7f08998f8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:37:46.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.227+0000 7f08998f8640 1 -- 172.21.15.37:0/4255576447 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f089411c700 con 0x7f089411c320 2024-04-26T00:37:46.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.227+0000 7f088a7fc640 1 --2- 172.21.15.37:0/4255576447 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f089411c320 0x7f0894136c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:37:46.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.227+0000 7f088a7fc640 10 monclient: get_auth_request con 0x7f089411c320 auth_method 0 2024-04-26T00:37:46.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.227+0000 7f088a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:37:46.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.227+0000 7f088a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:37:46.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.227+0000 7f088a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:37:46.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.227+0000 7f088a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:37:46.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.227+0000 7f088a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:37:46.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.227+0000 7f088a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:37:46.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.227+0000 7f088a7fc640 10 monclient(hunting): handle_auth_done global_id 204263 payload 824 2024-04-26T00:37:46.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.227+0000 7f088a7fc640 1 -- 172.21.15.37:0/4255576447 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f089404f510 con 0x7f089411c320 2024-04-26T00:37:46.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.231+0000 7f088a7fc640 1 -- 172.21.15.37:0/4255576447 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f089411c320 msgr2=0x7f0894136c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:37:46.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.231+0000 7f088a7fc640 1 --2- 172.21.15.37:0/4255576447 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f089411c320 0x7f0894136c60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f087c009740 tx=0x7f087c0a8000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:37:46.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.231+0000 7f0890ff9640 1 -- 172.21.15.37:0/4255576447 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087c0b0040 con 0x7f089411c320 2024-04-26T00:37:46.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.231+0000 7f0890ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:37:46.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.231+0000 7f0890ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:37:46.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.231+0000 7f0890ff9640 10 monclient: dump: 2024-04-26T00:37:46.232 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:37:46.232 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:37:46.232 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:46.232 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:37:46.232 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:37:46.232 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:37:46.232 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:37:46.232 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:37:46.232 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:37:46.232 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:37:46.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.231+0000 7f0890ff9640 1 -- 172.21.15.37:0/4255576447 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4536 (secure 0 0 0) 0x7f087c003e30 con 0x7f089411c320 2024-04-26T00:37:46.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.231+0000 7f0890ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:37:46.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.231+0000 7f0890ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:37:46.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.231+0000 7f0890ff9640 1 -- 172.21.15.37:0/4255576447 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f089411c320 msgr2=0x7f0894136c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:46.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.231+0000 7f0890ff9640 1 --2- 172.21.15.37:0/4255576447 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f089411c320 0x7f0894136c60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f087c009740 tx=0x7f087c0a8000 comp rx=0 tx=0).stop 2024-04-26T00:37:46.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.231+0000 7f0890ff9640 1 -- 172.21.15.37:0/4255576447 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f089411c320 msgr2=0x7f0894136c60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:37:46.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.231+0000 7f0890ff9640 1 --2- 172.21.15.37:0/4255576447 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f089411c320 0x7f0894136c60 secure :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f087c009740 tx=0x7f087c0a8000 comp rx=0 tx=0).stop 2024-04-26T00:37:46.233 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:37:46.233 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:37:46.233 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:37:46.234 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:37:46.234 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 136, 2024-04-26T00:37:46.234 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 71839, 2024-04-26T00:37:46.234 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:37:46.234 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:37:46.234 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:37:46.234 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:46.234 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:37:46.234 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:37:46.234 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:37:46.234 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:37:46.234 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:37:46.234 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:37:46.234 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:37:46.235 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:37:46.235 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:37:46.235 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:37:46.235 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:37:46.235 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:37:46.235 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:37:46.235 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:37:46.235 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:37:46.235 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:46.235 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:37:46.235 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:37:46.235 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:37:46.235 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:37:46.235 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:37:46.236 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:37:46.236 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:37:46.236 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:37:46.236 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:37:46.236 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:37:46.236 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:37:46.236 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:37:46.236 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:37:46.236 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:46.236 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:46.236 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:37:46.236 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:37:46.236 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:37:46.236 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:37:46.236 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:37:46.236 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:37:46.237 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:37:46.237 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:37:46.237 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:37:46.237 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:37:46.237 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:37:46.237 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:37:46.237 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:37:46.237 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:37:46.237 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:37:46.237 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:37:46.237 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:37:46.237 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:37:46.237 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:37:46.238 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:37:46.238 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:37:46.238 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:37:46.238 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:37:46.238 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:37:46.238 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:37:46.238 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:37:46.238 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:37:46.238 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:37:46.238 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:46.238 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:37:46.238 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:46.238 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:37:46.238 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:46.239 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:37:46.239 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:37:46.239 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:37:46.239 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:37:46.239 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:46.239 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:37:46.239 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:37:46.239 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:46.239 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:46.239 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:46.239 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:37:46.239 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:37:46.239 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:46.239 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:46.239 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:46.240 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:46.240 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:37:46.240 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:37:46.240 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:37:46.240 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:37:46.240 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:37:46.240 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:46.240 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:46.240 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:37:46.240 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:37:46.240 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:37:46.240 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:37:46.240 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:46.240 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:37:46.240 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:37:46.241 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:46.241 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:46.241 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:46.241 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:37:46.241 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:37:46.241 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:46.241 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:46.241 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:46.241 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:46.241 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:37:46.241 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:37:46.241 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:37:46.241 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:37:46.241 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:37:46.242 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:46.242 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:46.242 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:37:46.242 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:37:46.242 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:37:46.242 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:37:46.242 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:46.242 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:37:46.242 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:37:46.242 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:46.242 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:46.242 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:46.242 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:37:46.242 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:37:46.243 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:37:46.243 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:46.243 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:46.243 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:46.243 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:37:46.243 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:37:46.243 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:37:46.243 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:37:46.243 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:37:46.243 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:46.243 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:46.243 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:46.243 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:37:46.243 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:37:46.243 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:46.243 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:37:46.244 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:37:46.244 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:37:46.244 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:46.244 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:37:46.244 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:37:46.244 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:37:46.244 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:37:46.244 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:37:46.244 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:37:46.244 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:37:46.244 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:37:46.244 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:37:46.244 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:37:46.244 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:37:46.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.231+0000 7f08998f8640 1 -- 172.21.15.37:0/4255576447 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0874035eb0 msgr2=0x7f0874038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:46.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.231+0000 7f08998f8640 1 --2- 172.21.15.37:0/4255576447 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0874035eb0 0x7f0874038370 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f088c003bd0 tx=0x7f088c00d040 comp rx=0 tx=0).stop 2024-04-26T00:37:46.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.231+0000 7f08998f8640 10 monclient: shutdown 2024-04-26T00:37:46.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.231+0000 7f08998f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:37:46.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.231+0000 7f08998f8640 1 -- 172.21.15.37:0/4255576447 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f089409e220 msgr2=0x7f089413aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:37:46.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.231+0000 7f08998f8640 1 --2- 172.21.15.37:0/4255576447 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f089409e220 0x7f089413aed0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f088400c470 tx=0x7f088400c930 comp rx=0 tx=0).stop 2024-04-26T00:37:46.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.231+0000 7f08937fe640 1 -- 172.21.15.37:0/4255576447 reap_dead start 2024-04-26T00:37:46.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.231+0000 7f08937fe640 1 -- 172.21.15.37:0/4255576447 reap_dead start 2024-04-26T00:37:46.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.231+0000 7f08998f8640 1 -- 172.21.15.37:0/4255576447 shutdown_connections 2024-04-26T00:37:46.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.231+0000 7f08998f8640 1 -- 172.21.15.37:0/4255576447 >> 172.21.15.37:0/4255576447 conn(0x7f08940a1d50 msgr2=0x7f089413e1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:37:46.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.231+0000 7f08998f8640 1 -- 172.21.15.37:0/4255576447 shutdown_connections 2024-04-26T00:37:46.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:37:46.231+0000 7f08998f8640 1 -- 172.21.15.37:0/4255576447 wait complete. 2024-04-26T00:37:46.247 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:37:46.247 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-04-26T00:37:46.639 INFO:tasks.rados.rados.0.smithi045.stdout:3103: done (8 left) 2024-04-26T00:37:46.639 INFO:tasks.rados.rados.0.smithi045.stdout:3104: delete oid 24 current snap is 332 2024-04-26T00:37:46.639 INFO:tasks.rados.rados.0.smithi045.stdout:3095: expect (ObjNum 1106 snap 329 seq_num 1106) 2024-04-26T00:37:46.781 INFO:tasks.rados.rados.0.smithi045.stdout:3098: finishing rollback tid 1 to smithi04526579-11 2024-04-26T00:37:46.781 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-04-26T00:37:46.781 INFO:tasks.rados.rados.0.smithi045.stdout:3096: finishing rollback tid 1 to smithi04526579-33 2024-04-26T00:37:46.781 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 0 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-04-26T00:37:46.781 INFO:tasks.rados.rados.0.smithi045.stdout:3099: expect (ObjNum 987 snap 302 seq_num 987) 2024-04-26T00:37:46.901 INFO:tasks.rados.rados.0.smithi045.stdout:3097: expect (ObjNum 1074 snap 322 seq_num 1074) 2024-04-26T00:37:47.034 INFO:tasks.rados.rados.0.smithi045.stdout:3100: finishing write tid 1 to smithi04526579-36 2024-04-26T00:37:47.034 INFO:tasks.rados.rados.0.smithi045.stdout:3100: finishing write tid 2 to smithi04526579-36 2024-04-26T00:37:47.034 INFO:tasks.rados.rados.0.smithi045.stdout:3100: finishing write tid 3 to smithi04526579-36 2024-04-26T00:37:47.034 INFO:tasks.rados.rados.0.smithi045.stdout:3100: finishing write tid 4 to smithi04526579-36 2024-04-26T00:37:47.034 INFO:tasks.rados.rados.0.smithi045.stdout:3100: oid 36 updating version 0 to 2414 2024-04-26T00:37:47.034 INFO:tasks.rados.rados.0.smithi045.stdout:3100: oid 36 updating version 2414 to 2415 2024-04-26T00:37:47.034 INFO:tasks.rados.rados.0.smithi045.stdout:3100: oid 36 updating version 2415 to 2416 2024-04-26T00:37:47.034 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 2416 (ObjNum 1113 snap 332 seq_num 1113) dirty exists 2024-04-26T00:37:47.035 INFO:tasks.rados.rados.0.smithi045.stdout:3100: left oid 36 (ObjNum 1113 snap 332 seq_num 1113) 2024-04-26T00:37:47.035 INFO:tasks.rados.rados.0.smithi045.stdout:3101: expect (ObjNum 1104 snap 329 seq_num 1104) 2024-04-26T00:37:47.191 INFO:tasks.rados.rados.0.smithi045.stdout:3095: done (8 left) 2024-04-26T00:37:47.192 INFO:tasks.rados.rados.0.smithi045.stdout:3096: done (7 left) 2024-04-26T00:37:47.192 INFO:tasks.rados.rados.0.smithi045.stdout:3097: done (6 left) 2024-04-26T00:37:47.192 INFO:tasks.rados.rados.0.smithi045.stdout:3098: done (5 left) 2024-04-26T00:37:47.192 INFO:tasks.rados.rados.0.smithi045.stdout:3099: done (4 left) 2024-04-26T00:37:47.192 INFO:tasks.rados.rados.0.smithi045.stdout:3100: done (3 left) 2024-04-26T00:37:47.192 INFO:tasks.rados.rados.0.smithi045.stdout:3101: done (2 left) 2024-04-26T00:37:47.192 INFO:tasks.rados.rados.0.smithi045.stdout:3102: done (1 left) 2024-04-26T00:37:47.192 INFO:tasks.rados.rados.0.smithi045.stdout:3104: done (0 left) 2024-04-26T00:37:47.192 INFO:tasks.rados.rados.0.smithi045.stdout:3105: snap_remove snap 331 2024-04-26T00:37:47.643 INFO:tasks.rados.rados.0.smithi045.stdout:3105: done (0 left) 2024-04-26T00:37:47.644 INFO:tasks.rados.rados.0.smithi045.stdout:3106: write oid 30 current snap is 332 2024-04-26T00:37:47.644 INFO:tasks.rados.rados.0.smithi045.stdout:3106: seq_num 1115 ranges {739805=490173,1630599=435053,2636284=103521} 2024-04-26T00:37:47.663 INFO:tasks.rados.rados.0.smithi045.stdout:3106: writing smithi04526579-30 from 739805 to 1229978 tid 1 2024-04-26T00:37:47.678 INFO:tasks.rados.rados.0.smithi045.stdout:3106: writing smithi04526579-30 from 1630599 to 2065652 tid 2 2024-04-26T00:37:47.690 INFO:tasks.rados.rados.0.smithi045.stdout:3106: writing smithi04526579-30 from 2636284 to 2739805 tid 3 2024-04-26T00:37:47.690 INFO:tasks.rados.rados.0.smithi045.stdout:3107: snap_create 2024-04-26T00:37:47.690 INFO:tasks.rados.rados.0.smithi045.stdout:3106: finishing write tid 1 to smithi04526579-30 2024-04-26T00:37:47.690 INFO:tasks.rados.rados.0.smithi045.stdout:3106: finishing write tid 2 to smithi04526579-30 2024-04-26T00:37:47.693 INFO:tasks.rados.rados.0.smithi045.stdout:3106: finishing write tid 3 to smithi04526579-30 2024-04-26T00:37:47.696 INFO:tasks.rados.rados.0.smithi045.stdout:3106: finishing write tid 4 to smithi04526579-30 2024-04-26T00:37:47.697 INFO:tasks.rados.rados.0.smithi045.stdout:3106: finishing write tid 5 to smithi04526579-30 2024-04-26T00:37:47.697 INFO:tasks.rados.rados.0.smithi045.stdout:3106: oid 30 updating version 0 to 1994 2024-04-26T00:37:47.697 INFO:tasks.rados.rados.0.smithi045.stdout:3106: oid 30 version 1994 is already newer than 1992 2024-04-26T00:37:47.697 INFO:tasks.rados.rados.0.smithi045.stdout:3106: oid 30 version 1994 is already newer than 1993 2024-04-26T00:37:47.697 INFO:tasks.rados.rados.0.smithi045.stdout:3106: oid 30 version 1994 is already newer than 1991 2024-04-26T00:37:47.697 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 1994 (ObjNum 1115 snap 332 seq_num 1115) dirty exists 2024-04-26T00:37:47.697 INFO:tasks.rados.rados.0.smithi045.stdout:3106: left oid 30 (ObjNum 1115 snap 332 seq_num 1115) 2024-04-26T00:37:48.574 INFO:tasks.rados.rados.0.smithi045.stdout:3106: done (1 left) 2024-04-26T00:37:48.574 INFO:tasks.rados.rados.0.smithi045.stdout:3107: done (0 left) 2024-04-26T00:37:48.574 INFO:tasks.rados.rados.0.smithi045.stdout:3108: snap_create 2024-04-26T00:37:49.578 INFO:tasks.rados.rados.0.smithi045.stdout:3108: done (0 left) 2024-04-26T00:37:49.578 INFO:tasks.rados.rados.0.smithi045.stdout:3109: read oid 35 snap 332 2024-04-26T00:37:49.578 INFO:tasks.rados.rados.0.smithi045.stdout:3109: expect (ObjNum 1111 snap 331 seq_num 1111) 2024-04-26T00:37:49.578 INFO:tasks.rados.rados.0.smithi045.stdout:3110: delete oid 2 current snap is 334 2024-04-26T00:37:49.592 INFO:tasks.rados.rados.0.smithi045.stdout:3110: done (1 left) 2024-04-26T00:37:49.592 INFO:tasks.rados.rados.0.smithi045.stdout:3111: snap_create 2024-04-26T00:37:49.597 INFO:tasks.rados.rados.0.smithi045.stdout:3109: expect (ObjNum 1111 snap 331 seq_num 1111) 2024-04-26T00:37:50.585 INFO:tasks.rados.rados.0.smithi045.stdout:3109: done (1 left) 2024-04-26T00:37:50.585 INFO:tasks.rados.rados.0.smithi045.stdout:3111: done (0 left) 2024-04-26T00:37:50.585 INFO:tasks.rados.rados.0.smithi045.stdout:3112: snap_create 2024-04-26T00:37:51.593 INFO:tasks.rados.rados.0.smithi045.stdout:3112: done (0 left) 2024-04-26T00:37:51.593 INFO:tasks.rados.rados.0.smithi045.stdout:3113: delete oid 25 current snap is 336 2024-04-26T00:37:51.603 INFO:tasks.rados.rados.0.smithi045.stdout:3113: done (0 left) 2024-04-26T00:37:51.603 INFO:tasks.rados.rados.0.smithi045.stdout:3114: snap_create 2024-04-26T00:37:52.597 INFO:tasks.rados.rados.0.smithi045.stdout:3114: done (0 left) 2024-04-26T00:37:52.597 INFO:tasks.rados.rados.0.smithi045.stdout:3115: copy_from oid 19 from oid 11 current snap is 337 2024-04-26T00:37:52.597 INFO:tasks.rados.rados.0.smithi045.stdout:3116: read oid 48 snap -1 2024-04-26T00:37:52.597 INFO:tasks.rados.rados.0.smithi045.stdout:3116: expect (ObjNum 1112 snap 331 seq_num 1112) 2024-04-26T00:37:52.597 INFO:tasks.rados.rados.0.smithi045.stdout:3117: rollback oid 43 current snap is 337 2024-04-26T00:37:52.597 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 43 to 329 2024-04-26T00:37:52.597 INFO:tasks.rados.rados.0.smithi045.stdout:3118: read oid 6 snap -1 2024-04-26T00:37:52.597 INFO:tasks.rados.rados.0.smithi045.stdout:3118: expect (ObjNum 1060 snap 318 seq_num 1060) 2024-04-26T00:37:52.598 INFO:tasks.rados.rados.0.smithi045.stdout:3119: rollback oid 32 current snap is 337 2024-04-26T00:37:52.598 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 32 to 332 2024-04-26T00:37:52.598 INFO:tasks.rados.rados.0.smithi045.stdout:3120: copy_from oid 20 from oid 39 current snap is 337 2024-04-26T00:37:52.598 INFO:tasks.rados.rados.0.smithi045.stdout:3121: write oid 36 current snap is 337 2024-04-26T00:37:52.598 INFO:tasks.rados.rados.0.smithi045.stdout:3121: seq_num 1118 ranges {790226=432273,1806045=582380} 2024-04-26T00:37:52.617 INFO:tasks.rados.rados.0.smithi045.stdout:3121: writing smithi04526579-36 from 790226 to 1222499 tid 1 2024-04-26T00:37:52.638 INFO:tasks.rados.rados.0.smithi045.stdout:3121: writing smithi04526579-36 from 1806045 to 2388425 tid 2 2024-04-26T00:37:52.638 INFO:tasks.rados.rados.0.smithi045.stdout:3122: write oid 35 current snap is 337 2024-04-26T00:37:52.638 INFO:tasks.rados.rados.0.smithi045.stdout:3122: seq_num 1119 ranges {407033=507522,1464527=442391,2345976=461057} 2024-04-26T00:37:52.651 INFO:tasks.rados.rados.0.smithi045.stdout:3122: writing smithi04526579-35 from 407033 to 914555 tid 1 2024-04-26T00:37:52.664 INFO:tasks.rados.rados.0.smithi045.stdout:3122: writing smithi04526579-35 from 1464527 to 1906918 tid 2 2024-04-26T00:37:52.674 INFO:tasks.rados.rados.0.smithi045.stdout:3122: writing smithi04526579-35 from 2345976 to 2807033 tid 3 2024-04-26T00:37:52.674 INFO:tasks.rados.rados.0.smithi045.stdout:3123: read oid 1 snap -1 2024-04-26T00:37:52.674 INFO:tasks.rados.rados.0.smithi045.stdout:3123: expect (ObjNum 1079 snap 322 seq_num 1079) 2024-04-26T00:37:52.674 INFO:tasks.rados.rados.0.smithi045.stdout:3124: read oid 2 snap -1 2024-04-26T00:37:52.674 INFO:tasks.rados.rados.0.smithi045.stdout:3124: expect deleted 2024-04-26T00:37:52.674 INFO:tasks.rados.rados.0.smithi045.stdout:3125: read oid 47 snap 334 2024-04-26T00:37:52.674 INFO:tasks.rados.rados.0.smithi045.stdout:3125: expect (ObjNum 1097 snap 326 seq_num 1097) 2024-04-26T00:37:52.674 INFO:tasks.rados.rados.0.smithi045.stdout:3126: read oid 21 snap -1 2024-04-26T00:37:52.674 INFO:tasks.rados.rados.0.smithi045.stdout:3126: expect (ObjNum 1087 snap 324 seq_num 1087) 2024-04-26T00:37:52.674 INFO:tasks.rados.rados.0.smithi045.stdout:3127: write (excl) oid 34 current snap is 337 2024-04-26T00:37:52.674 INFO:tasks.rados.rados.0.smithi045.stdout:3117: finishing rollback tid 1 to smithi04526579-43 2024-04-26T00:37:52.674 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 0 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-04-26T00:37:52.675 INFO:tasks.rados.rados.0.smithi045.stdout:3116: expect (ObjNum 1112 snap 331 seq_num 1112) 2024-04-26T00:37:52.855 INFO:tasks.rados.rados.0.smithi045.stdout:3118: expect (ObjNum 1060 snap 318 seq_num 1060) 2024-04-26T00:37:52.979 INFO:tasks.rados.rados.0.smithi045.stdout:3115: finishing copy_from to smithi04526579-19 2024-04-26T00:37:52.979 INFO:tasks.rados.rados.0.smithi045.stdout:3115: got expected ENOENT (src dne) 2024-04-26T00:37:52.979 INFO:tasks.rados.rados.0.smithi045.stdout:3115: finishing copy_from racing read to smithi04526579-19 2024-04-26T00:37:52.979 INFO:tasks.rados.rados.0.smithi045.stdout:3119: finishing rollback tid 0 to smithi04526579-32 2024-04-26T00:37:52.979 INFO:tasks.rados.rados.0.smithi045.stdout:3119: finishing rollback tid 1 to smithi04526579-32 2024-04-26T00:37:52.979 INFO:tasks.rados.rados.0.smithi045.stdout:3119: finishing rollback tid 2 to smithi04526579-32 2024-04-26T00:37:52.979 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 2418 (ObjNum 1101 snap 329 seq_num 1101) dirty exists 2024-04-26T00:37:52.979 INFO:tasks.rados.rados.0.smithi045.stdout:3121: finishing write tid 1 to smithi04526579-36 2024-04-26T00:37:52.979 INFO:tasks.rados.rados.0.smithi045.stdout:3120: finishing copy_from racing read to smithi04526579-20 2024-04-26T00:37:52.980 INFO:tasks.rados.rados.0.smithi045.stdout:3120: finishing copy_from to smithi04526579-20 2024-04-26T00:37:52.980 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 2368 (ObjNum 1093 snap 324 seq_num 1093) dirty exists 2024-04-26T00:37:52.980 INFO:tasks.rados.rados.0.smithi045.stdout:3121: finishing write tid 2 to smithi04526579-36 2024-04-26T00:37:52.980 INFO:tasks.rados.rados.0.smithi045.stdout:3121: finishing write tid 3 to smithi04526579-36 2024-04-26T00:37:52.980 INFO:tasks.rados.rados.0.smithi045.stdout:3121: finishing write tid 4 to smithi04526579-36 2024-04-26T00:37:52.980 INFO:tasks.rados.rados.0.smithi045.stdout:3121: oid 36 updating version 0 to 2417 2024-04-26T00:37:52.980 INFO:tasks.rados.rados.0.smithi045.stdout:3121: oid 36 updating version 2417 to 2419 2024-04-26T00:37:52.980 INFO:tasks.rados.rados.0.smithi045.stdout:3121: oid 36 version 2419 is already newer than 2418 2024-04-26T00:37:52.980 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 2419 (ObjNum 1118 snap 337 seq_num 1118) dirty exists 2024-04-26T00:37:52.980 INFO:tasks.rados.rados.0.smithi045.stdout:3121: left oid 36 (ObjNum 1118 snap 337 seq_num 1118) 2024-04-26T00:37:52.980 INFO:tasks.rados.rados.0.smithi045.stdout:3122: finishing write tid 1 to smithi04526579-35 2024-04-26T00:37:52.980 INFO:tasks.rados.rados.0.smithi045.stdout:3122: finishing write tid 2 to smithi04526579-35 2024-04-26T00:37:52.980 INFO:tasks.rados.rados.0.smithi045.stdout:3127: seq_num 1120 ranges {423840=582771,1523009=586049,2657046=166794} 2024-04-26T00:37:52.994 INFO:tasks.rados.rados.0.smithi045.stdout:3127: writing smithi04526579-34 from 423840 to 1006611 tid 1 2024-04-26T00:37:53.011 INFO:tasks.rados.rados.0.smithi045.stdout:3127: writing smithi04526579-34 from 1523009 to 2109058 tid 2 2024-04-26T00:37:53.023 INFO:tasks.rados.rados.0.smithi045.stdout:3127: writing smithi04526579-34 from 2657046 to 2823840 tid 3 2024-04-26T00:37:53.023 INFO:tasks.rados.rados.0.smithi045.stdout:3115: done (12 left) 2024-04-26T00:37:53.023 INFO:tasks.rados.rados.0.smithi045.stdout:3116: done (11 left) 2024-04-26T00:37:53.023 INFO:tasks.rados.rados.0.smithi045.stdout:3117: done (10 left) 2024-04-26T00:37:53.023 INFO:tasks.rados.rados.0.smithi045.stdout:3118: done (9 left) 2024-04-26T00:37:53.023 INFO:tasks.rados.rados.0.smithi045.stdout:3119: done (8 left) 2024-04-26T00:37:53.023 INFO:tasks.rados.rados.0.smithi045.stdout:3120: done (7 left) 2024-04-26T00:37:53.023 INFO:tasks.rados.rados.0.smithi045.stdout:3121: done (6 left) 2024-04-26T00:37:53.023 INFO:tasks.rados.rados.0.smithi045.stdout:3128: write (excl) oid 43 current snap is 337 2024-04-26T00:37:53.023 INFO:tasks.rados.rados.0.smithi045.stdout:3125: expect (ObjNum 1097 snap 326 seq_num 1097) 2024-04-26T00:37:53.185 INFO:tasks.rados.rados.0.smithi045.stdout:3122: finishing write tid 3 to smithi04526579-35 2024-04-26T00:37:53.185 INFO:tasks.rados.rados.0.smithi045.stdout:3123: expect (ObjNum 1079 snap 322 seq_num 1079) 2024-04-26T00:37:53.309 INFO:tasks.rados.rados.0.smithi045.stdout:3122: finishing write tid 4 to smithi04526579-35 2024-04-26T00:37:53.309 INFO:tasks.rados.rados.0.smithi045.stdout:3122: finishing write tid 5 to smithi04526579-35 2024-04-26T00:37:53.309 INFO:tasks.rados.rados.0.smithi045.stdout:3122: oid 35 updating version 0 to 2421 2024-04-26T00:37:53.309 INFO:tasks.rados.rados.0.smithi045.stdout:3122: oid 35 version 2421 is already newer than 2420 2024-04-26T00:37:53.309 INFO:tasks.rados.rados.0.smithi045.stdout:3122: oid 35 version 2421 is already newer than 2419 2024-04-26T00:37:53.309 INFO:tasks.rados.rados.0.smithi045.stdout:3122: oid 35 updating version 2421 to 2422 2024-04-26T00:37:53.309 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 2422 (ObjNum 1119 snap 337 seq_num 1119) dirty exists 2024-04-26T00:37:53.309 INFO:tasks.rados.rados.0.smithi045.stdout:3122: left oid 35 (ObjNum 1119 snap 337 seq_num 1119) 2024-04-26T00:37:53.309 INFO:tasks.rados.rados.0.smithi045.stdout:3126: expect (ObjNum 1087 snap 324 seq_num 1087) 2024-04-26T00:37:53.437 INFO:tasks.rados.rados.0.smithi045.stdout:3127: finishing write tid 1 to smithi04526579-34 2024-04-26T00:37:53.437 INFO:tasks.rados.rados.0.smithi045.stdout:3127: finishing write tid 2 to smithi04526579-34 2024-04-26T00:37:53.437 INFO:tasks.rados.rados.0.smithi045.stdout:3127: finishing write tid 3 to smithi04526579-34 2024-04-26T00:37:53.437 INFO:tasks.rados.rados.0.smithi045.stdout:3127: finishing write tid 4 to smithi04526579-34 2024-04-26T00:37:53.437 INFO:tasks.rados.rados.0.smithi045.stdout:3127: finishing write tid 5 to smithi04526579-34 2024-04-26T00:37:53.437 INFO:tasks.rados.rados.0.smithi045.stdout:3127: oid 34 updating version 0 to 2343 2024-04-26T00:37:53.437 INFO:tasks.rados.rados.0.smithi045.stdout:3127: oid 34 version 2343 is already newer than 2340 2024-04-26T00:37:53.438 INFO:tasks.rados.rados.0.smithi045.stdout:3127: oid 34 version 2343 is already newer than 2341 2024-04-26T00:37:53.438 INFO:tasks.rados.rados.0.smithi045.stdout:3127: oid 34 version 2343 is already newer than 2342 2024-04-26T00:37:53.438 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 2343 (ObjNum 1120 snap 337 seq_num 1120) dirty exists 2024-04-26T00:37:53.438 INFO:tasks.rados.rados.0.smithi045.stdout:3127: left oid 34 (ObjNum 1120 snap 337 seq_num 1120) 2024-04-26T00:37:53.438 INFO:tasks.rados.rados.0.smithi045.stdout:3128: seq_num 1121 ranges {440647=658020,1581491=446060,2800822=39825} 2024-04-26T00:37:53.454 INFO:tasks.rados.rados.0.smithi045.stdout:3128: writing smithi04526579-43 from 440647 to 1098667 tid 1 2024-04-26T00:37:53.470 INFO:tasks.rados.rados.0.smithi045.stdout:3128: writing smithi04526579-43 from 1581491 to 2027551 tid 2 2024-04-26T00:37:53.484 INFO:tasks.rados.rados.0.smithi045.stdout:3128: writing smithi04526579-43 from 2800822 to 2840647 tid 3 2024-04-26T00:37:53.484 INFO:tasks.rados.rados.0.smithi045.stdout:3122: done (6 left) 2024-04-26T00:37:53.484 INFO:tasks.rados.rados.0.smithi045.stdout:3123: done (5 left) 2024-04-26T00:37:53.484 INFO:tasks.rados.rados.0.smithi045.stdout:3124: done (4 left) 2024-04-26T00:37:53.484 INFO:tasks.rados.rados.0.smithi045.stdout:3125: done (3 left) 2024-04-26T00:37:53.484 INFO:tasks.rados.rados.0.smithi045.stdout:3126: done (2 left) 2024-04-26T00:37:53.484 INFO:tasks.rados.rados.0.smithi045.stdout:3127: done (1 left) 2024-04-26T00:37:53.484 INFO:tasks.rados.rados.0.smithi045.stdout:3129: snap_create 2024-04-26T00:37:53.484 INFO:tasks.rados.rados.0.smithi045.stdout:3128: finishing write tid 1 to smithi04526579-43 2024-04-26T00:37:53.485 INFO:tasks.rados.rados.0.smithi045.stdout:3128: finishing write tid 2 to smithi04526579-43 2024-04-26T00:37:53.488 INFO:tasks.rados.rados.0.smithi045.stdout:3128: finishing write tid 3 to smithi04526579-43 2024-04-26T00:37:53.488 INFO:tasks.rados.rados.0.smithi045.stdout:3128: finishing write tid 4 to smithi04526579-43 2024-04-26T00:37:53.489 INFO:tasks.rados.rados.0.smithi045.stdout:3128: finishing write tid 5 to smithi04526579-43 2024-04-26T00:37:53.489 INFO:tasks.rados.rados.0.smithi045.stdout:3128: oid 43 updating version 0 to 2341 2024-04-26T00:37:53.489 INFO:tasks.rados.rados.0.smithi045.stdout:3128: oid 43 updating version 2341 to 2342 2024-04-26T00:37:53.489 INFO:tasks.rados.rados.0.smithi045.stdout:3128: oid 43 updating version 2342 to 2343 2024-04-26T00:37:53.489 INFO:tasks.rados.rados.0.smithi045.stdout:3128: oid 43 version 2343 is already newer than 2340 2024-04-26T00:37:53.489 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 2343 (ObjNum 1121 snap 337 seq_num 1121) dirty exists 2024-04-26T00:37:53.489 INFO:tasks.rados.rados.0.smithi045.stdout:3128: left oid 43 (ObjNum 1121 snap 337 seq_num 1121) 2024-04-26T00:37:53.577 INFO:tasks.rados.rados.0.smithi045.stdout:3128: done (1 left) 2024-04-26T00:37:53.577 INFO:tasks.rados.rados.0.smithi045.stdout:3129: done (0 left) 2024-04-26T00:37:53.577 INFO:tasks.rados.rados.0.smithi045.stdout:3130: rollback oid 22 current snap is 338 2024-04-26T00:37:53.577 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 22 to 334 2024-04-26T00:37:53.577 INFO:tasks.rados.rados.0.smithi045.stdout:3131: read oid 4 snap -1 2024-04-26T00:37:53.577 INFO:tasks.rados.rados.0.smithi045.stdout:3131: expect (ObjNum 1080 snap 323 seq_num 1080) 2024-04-26T00:37:53.578 INFO:tasks.rados.rados.0.smithi045.stdout:3132: snap_create 2024-04-26T00:37:53.588 INFO:tasks.rados.rados.0.smithi045.stdout:3131: expect (ObjNum 1080 snap 323 seq_num 1080) 2024-04-26T00:37:53.720 INFO:tasks.rados.rados.0.smithi045.stdout:3130: finishing rollback tid 0 to smithi04526579-22 2024-04-26T00:37:53.720 INFO:tasks.rados.rados.0.smithi045.stdout:3130: finishing rollback tid 1 to smithi04526579-22 2024-04-26T00:37:53.720 INFO:tasks.rados.rados.0.smithi045.stdout:3130: finishing rollback tid 2 to smithi04526579-22 2024-04-26T00:37:53.720 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 2422 (ObjNum 1106 snap 329 seq_num 1106) dirty exists 2024-04-26T00:37:54.579 INFO:tasks.rados.rados.0.smithi045.stdout:3130: done (2 left) 2024-04-26T00:37:54.579 INFO:tasks.rados.rados.0.smithi045.stdout:3131: done (1 left) 2024-04-26T00:37:54.580 INFO:tasks.rados.rados.0.smithi045.stdout:3132: done (0 left) 2024-04-26T00:37:54.580 INFO:tasks.rados.rados.0.smithi045.stdout:3133: read oid 34 snap 335 2024-04-26T00:37:54.580 INFO:tasks.rados.rados.0.smithi045.stdout:3133: expect (ObjNum 1104 snap 329 seq_num 1104) 2024-04-26T00:37:54.580 INFO:tasks.rados.rados.0.smithi045.stdout:3134: rollback oid 35 current snap is 339 2024-04-26T00:37:54.580 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 35 to 336 2024-04-26T00:37:54.580 INFO:tasks.rados.rados.0.smithi045.stdout:3135: copy_from oid 43 from oid 8 current snap is 339 2024-04-26T00:37:54.581 INFO:tasks.rados.rados.0.smithi045.stdout:3136: write oid 32 current snap is 339 2024-04-26T00:37:54.581 INFO:tasks.rados.rados.0.smithi045.stdout:3136: seq_num 1122 ranges {457454=733269,1923620=589718} 2024-04-26T00:37:54.604 INFO:tasks.rados.rados.0.smithi045.stdout:3136: writing smithi04526579-32 from 457454 to 1190723 tid 1 2024-04-26T00:37:54.626 INFO:tasks.rados.rados.0.smithi045.stdout:3136: writing smithi04526579-32 from 1923620 to 2513338 tid 2 2024-04-26T00:37:54.626 INFO:tasks.rados.rados.0.smithi045.stdout:3137: read oid 15 snap -1 2024-04-26T00:37:54.627 INFO:tasks.rados.rados.0.smithi045.stdout:3137: expect deleted 2024-04-26T00:37:54.627 INFO:tasks.rados.rados.0.smithi045.stdout:3138: snap_create 2024-04-26T00:37:54.627 INFO:tasks.rados.rados.0.smithi045.stdout:3134: finishing rollback tid 0 to smithi04526579-35 2024-04-26T00:37:54.627 INFO:tasks.rados.rados.0.smithi045.stdout:3134: finishing rollback tid 1 to smithi04526579-35 2024-04-26T00:37:54.627 INFO:tasks.rados.rados.0.smithi045.stdout:3134: finishing rollback tid 2 to smithi04526579-35 2024-04-26T00:37:54.627 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 2425 (ObjNum 1111 snap 331 seq_num 1111) dirty exists 2024-04-26T00:37:54.628 INFO:tasks.rados.rados.0.smithi045.stdout:3133: expect (ObjNum 1104 snap 329 seq_num 1104) 2024-04-26T00:37:54.793 INFO:tasks.rados.rados.0.smithi045.stdout:3135: finishing copy_from racing read to smithi04526579-43 2024-04-26T00:37:54.794 INFO:tasks.rados.rados.0.smithi045.stdout:3136: finishing write tid 1 to smithi04526579-32 2024-04-26T00:37:54.794 INFO:tasks.rados.rados.0.smithi045.stdout:3135: finishing copy_from to smithi04526579-43 2024-04-26T00:37:54.794 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 2196 (ObjNum 1103 snap 329 seq_num 1103) dirty exists 2024-04-26T00:37:54.794 INFO:tasks.rados.rados.0.smithi045.stdout:3136: finishing write tid 2 to smithi04526579-32 2024-04-26T00:37:54.794 INFO:tasks.rados.rados.0.smithi045.stdout:3136: finishing write tid 3 to smithi04526579-32 2024-04-26T00:37:54.794 INFO:tasks.rados.rados.0.smithi045.stdout:3136: finishing write tid 4 to smithi04526579-32 2024-04-26T00:37:54.794 INFO:tasks.rados.rados.0.smithi045.stdout:3136: oid 32 updating version 0 to 2427 2024-04-26T00:37:54.794 INFO:tasks.rados.rados.0.smithi045.stdout:3136: oid 32 version 2427 is already newer than 2426 2024-04-26T00:37:54.794 INFO:tasks.rados.rados.0.smithi045.stdout:3136: oid 32 updating version 2427 to 2428 2024-04-26T00:37:54.794 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 2428 (ObjNum 1122 snap 339 seq_num 1122) dirty exists 2024-04-26T00:37:54.794 INFO:tasks.rados.rados.0.smithi045.stdout:3136: left oid 32 (ObjNum 1122 snap 339 seq_num 1122) 2024-04-26T00:37:55.582 INFO:tasks.rados.rados.0.smithi045.stdout:3133: done (5 left) 2024-04-26T00:37:55.582 INFO:tasks.rados.rados.0.smithi045.stdout:3134: done (4 left) 2024-04-26T00:37:55.582 INFO:tasks.rados.rados.0.smithi045.stdout:3135: done (3 left) 2024-04-26T00:37:55.582 INFO:tasks.rados.rados.0.smithi045.stdout:3136: done (2 left) 2024-04-26T00:37:55.582 INFO:tasks.rados.rados.0.smithi045.stdout:3137: done (1 left) 2024-04-26T00:37:55.583 INFO:tasks.rados.rados.0.smithi045.stdout:3138: done (0 left) 2024-04-26T00:37:55.583 INFO:tasks.rados.rados.0.smithi045.stdout:3139: snap_remove snap 339 2024-04-26T00:37:56.588 INFO:tasks.rados.rados.0.smithi045.stdout:3139: done (0 left) 2024-04-26T00:37:56.588 INFO:tasks.rados.rados.0.smithi045.stdout:3140: read oid 9 snap -1 2024-04-26T00:37:56.588 INFO:tasks.rados.rados.0.smithi045.stdout:3140: expect (ObjNum 1099 snap 326 seq_num 1099) 2024-04-26T00:37:56.588 INFO:tasks.rados.rados.0.smithi045.stdout:3141: write oid 41 current snap is 340 2024-04-26T00:37:56.588 INFO:tasks.rados.rados.0.smithi045.stdout:3141: seq_num 1123 ranges {474261=408518,1582102=449729,2739315=134946} 2024-04-26T00:37:56.602 INFO:tasks.rados.rados.0.smithi045.stdout:3141: writing smithi04526579-41 from 474261 to 882779 tid 1 2024-04-26T00:37:56.623 INFO:tasks.rados.rados.0.smithi045.stdout:3141: writing smithi04526579-41 from 1582102 to 2031831 tid 2 2024-04-26T00:37:56.637 INFO:tasks.rados.rados.0.smithi045.stdout:3141: writing smithi04526579-41 from 2739315 to 2874261 tid 3 2024-04-26T00:37:56.637 INFO:tasks.rados.rados.0.smithi045.stdout:3142: delete oid 7 current snap is 340 2024-04-26T00:37:56.638 INFO:tasks.rados.rados.0.smithi045.stdout:3140: expect (ObjNum 1099 snap 326 seq_num 1099) 2024-04-26T00:37:56.774 INFO:tasks.rados.rados.0.smithi045.stdout:3141: finishing write tid 1 to smithi04526579-41 2024-04-26T00:37:56.774 INFO:tasks.rados.rados.0.smithi045.stdout:3141: finishing write tid 2 to smithi04526579-41 2024-04-26T00:37:56.774 INFO:tasks.rados.rados.0.smithi045.stdout:3141: finishing write tid 3 to smithi04526579-41 2024-04-26T00:37:56.774 INFO:tasks.rados.rados.0.smithi045.stdout:3141: finishing write tid 4 to smithi04526579-41 2024-04-26T00:37:56.774 INFO:tasks.rados.rados.0.smithi045.stdout:3141: finishing write tid 5 to smithi04526579-41 2024-04-26T00:37:56.774 INFO:tasks.rados.rados.0.smithi045.stdout:3141: oid 41 updating version 0 to 2439 2024-04-26T00:37:56.778 INFO:tasks.rados.rados.0.smithi045.stdout:3141: oid 41 updating version 2439 to 2440 2024-04-26T00:37:56.778 INFO:tasks.rados.rados.0.smithi045.stdout:3141: oid 41 version 2440 is already newer than 2437 2024-04-26T00:37:56.778 INFO:tasks.rados.rados.0.smithi045.stdout:3141: oid 41 version 2440 is already newer than 2438 2024-04-26T00:37:56.778 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 2440 (ObjNum 1123 snap 340 seq_num 1123) dirty exists 2024-04-26T00:37:56.778 INFO:tasks.rados.rados.0.smithi045.stdout:3141: left oid 41 (ObjNum 1123 snap 340 seq_num 1123) 2024-04-26T00:37:56.779 INFO:tasks.rados.rados.0.smithi045.stdout:3140: done (2 left) 2024-04-26T00:37:56.779 INFO:tasks.rados.rados.0.smithi045.stdout:3141: done (1 left) 2024-04-26T00:37:56.779 INFO:tasks.rados.rados.0.smithi045.stdout:3142: done (0 left) 2024-04-26T00:37:56.779 INFO:tasks.rados.rados.0.smithi045.stdout:3143: delete oid 29 current snap is 340 2024-04-26T00:37:56.783 INFO:tasks.rados.rados.0.smithi045.stdout:3143: done (0 left) 2024-04-26T00:37:56.783 INFO:tasks.rados.rados.0.smithi045.stdout:3144: read oid 46 snap -1 2024-04-26T00:37:56.783 INFO:tasks.rados.rados.0.smithi045.stdout:3144: expect deleted 2024-04-26T00:37:56.783 INFO:tasks.rados.rados.0.smithi045.stdout:3145: read oid 30 snap -1 2024-04-26T00:37:56.783 INFO:tasks.rados.rados.0.smithi045.stdout:3145: expect (ObjNum 1115 snap 332 seq_num 1115) 2024-04-26T00:37:56.784 INFO:tasks.rados.rados.0.smithi045.stdout:3146: read oid 43 snap -1 2024-04-26T00:37:56.784 INFO:tasks.rados.rados.0.smithi045.stdout:3146: expect (ObjNum 1103 snap 329 seq_num 1103) 2024-04-26T00:37:56.784 INFO:tasks.rados.rados.0.smithi045.stdout:3147: write oid 31 current snap is 340 2024-04-26T00:37:56.784 INFO:tasks.rados.rados.0.smithi045.stdout:3147: seq_num 1126 ranges {524682=750618,1757548=597056} 2024-04-26T00:37:56.804 INFO:tasks.rados.rados.0.smithi045.stdout:3147: writing smithi04526579-31 from 524682 to 1275300 tid 1 2024-04-26T00:37:56.822 INFO:tasks.rados.rados.0.smithi045.stdout:3147: writing smithi04526579-31 from 1757548 to 2354604 tid 2 2024-04-26T00:37:56.822 INFO:tasks.rados.rados.0.smithi045.stdout:3148: write oid 44 current snap is 340 2024-04-26T00:37:56.822 INFO:tasks.rados.rados.0.smithi045.stdout:3148: seq_num 1127 ranges {541489=425867,1416030=740714,2732654=208835} 2024-04-26T00:37:56.837 INFO:tasks.rados.rados.0.smithi045.stdout:3148: writing smithi04526579-44 from 541489 to 967356 tid 1 2024-04-26T00:37:56.851 INFO:tasks.rados.rados.0.smithi045.stdout:3148: writing smithi04526579-44 from 1416030 to 2156744 tid 2 2024-04-26T00:37:56.859 INFO:tasks.rados.rados.0.smithi045.stdout:3148: writing smithi04526579-44 from 2732654 to 2941489 tid 3 2024-04-26T00:37:56.859 INFO:tasks.rados.rados.0.smithi045.stdout:3149: write (excl) oid 42 current snap is 340 2024-04-26T00:37:56.859 INFO:tasks.rados.rados.0.smithi045.stdout:3149: seq_num 1128 ranges {558296=501116,1474512=600725,2760077=198219} 2024-04-26T00:37:56.869 INFO:tasks.rados.rados.0.smithi045.stdout:3149: writing smithi04526579-42 from 558296 to 1059412 tid 1 2024-04-26T00:37:56.877 INFO:tasks.rados.rados.0.smithi045.stdout:3149: writing smithi04526579-42 from 1474512 to 2075237 tid 2 2024-04-26T00:37:56.883 INFO:tasks.rados.rados.0.smithi045.stdout:3149: writing smithi04526579-42 from 2760077 to 2958296 tid 3 2024-04-26T00:37:56.884 INFO:tasks.rados.rados.0.smithi045.stdout:3150: write oid 19 current snap is 340 2024-04-26T00:37:56.884 INFO:tasks.rados.rados.0.smithi045.stdout:3150: seq_num 1129 ranges {575103=576365,1932994=744383} 2024-04-26T00:37:56.892 INFO:tasks.rados.rados.0.smithi045.stdout:3150: writing smithi04526579-19 from 575103 to 1151468 tid 1 2024-04-26T00:37:56.902 INFO:tasks.rados.rados.0.smithi045.stdout:3150: writing smithi04526579-19 from 1932994 to 2677377 tid 2 2024-04-26T00:37:56.902 INFO:tasks.rados.rados.0.smithi045.stdout:3151: delete oid 37 current snap is 340 2024-04-26T00:37:56.903 INFO:tasks.rados.rados.0.smithi045.stdout:3146: expect (ObjNum 1103 snap 329 seq_num 1103) 2024-04-26T00:37:57.133 INFO:tasks.rados.rados.0.smithi045.stdout:3145: expect (ObjNum 1115 snap 332 seq_num 1115) 2024-04-26T00:37:57.295 INFO:tasks.rados.rados.0.smithi045.stdout:3147: finishing write tid 1 to smithi04526579-31 2024-04-26T00:37:57.295 INFO:tasks.rados.rados.0.smithi045.stdout:3147: finishing write tid 2 to smithi04526579-31 2024-04-26T00:37:57.295 INFO:tasks.rados.rados.0.smithi045.stdout:3147: finishing write tid 3 to smithi04526579-31 2024-04-26T00:37:57.295 INFO:tasks.rados.rados.0.smithi045.stdout:3147: finishing write tid 4 to smithi04526579-31 2024-04-26T00:37:57.295 INFO:tasks.rados.rados.0.smithi045.stdout:3147: oid 31 updating version 0 to 2053 2024-04-26T00:37:57.295 INFO:tasks.rados.rados.0.smithi045.stdout:3147: oid 31 version 2053 is already newer than 2052 2024-04-26T00:37:57.295 INFO:tasks.rados.rados.0.smithi045.stdout:3147: oid 31 version 2053 is already newer than 2051 2024-04-26T00:37:57.295 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 2053 (ObjNum 1126 snap 340 seq_num 1126) dirty exists 2024-04-26T00:37:57.296 INFO:tasks.rados.rados.0.smithi045.stdout:3147: left oid 31 (ObjNum 1126 snap 340 seq_num 1126) 2024-04-26T00:37:57.296 INFO:tasks.rados.rados.0.smithi045.stdout:3148: finishing write tid 1 to smithi04526579-44 2024-04-26T00:37:57.296 INFO:tasks.rados.rados.0.smithi045.stdout:3148: finishing write tid 2 to smithi04526579-44 2024-04-26T00:37:57.296 INFO:tasks.rados.rados.0.smithi045.stdout:3148: finishing write tid 3 to smithi04526579-44 2024-04-26T00:37:57.296 INFO:tasks.rados.rados.0.smithi045.stdout:3148: finishing write tid 4 to smithi04526579-44 2024-04-26T00:37:57.296 INFO:tasks.rados.rados.0.smithi045.stdout:3148: finishing write tid 5 to smithi04526579-44 2024-04-26T00:37:57.296 INFO:tasks.rados.rados.0.smithi045.stdout:3148: oid 44 updating version 0 to 2375 2024-04-26T00:37:57.296 INFO:tasks.rados.rados.0.smithi045.stdout:3148: oid 44 version 2375 is already newer than 2373 2024-04-26T00:37:57.296 INFO:tasks.rados.rados.0.smithi045.stdout:3148: oid 44 version 2375 is already newer than 2374 2024-04-26T00:37:57.296 INFO:tasks.rados.rados.0.smithi045.stdout:3148: oid 44 version 2375 is already newer than 2372 2024-04-26T00:37:57.296 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 2375 (ObjNum 1127 snap 340 seq_num 1127) dirty exists 2024-04-26T00:37:57.296 INFO:tasks.rados.rados.0.smithi045.stdout:3148: left oid 44 (ObjNum 1127 snap 340 seq_num 1127) 2024-04-26T00:37:57.297 INFO:tasks.rados.rados.0.smithi045.stdout:3149: finishing write tid 1 to smithi04526579-42 2024-04-26T00:37:57.297 INFO:tasks.rados.rados.0.smithi045.stdout:3149: finishing write tid 2 to smithi04526579-42 2024-04-26T00:37:57.297 INFO:tasks.rados.rados.0.smithi045.stdout:3149: finishing write tid 3 to smithi04526579-42 2024-04-26T00:37:57.297 INFO:tasks.rados.rados.0.smithi045.stdout:3149: finishing write tid 4 to smithi04526579-42 2024-04-26T00:37:57.297 INFO:tasks.rados.rados.0.smithi045.stdout:3149: finishing write tid 5 to smithi04526579-42 2024-04-26T00:37:57.297 INFO:tasks.rados.rados.0.smithi045.stdout:3149: oid 42 updating version 0 to 2430 2024-04-26T00:37:57.297 INFO:tasks.rados.rados.0.smithi045.stdout:3149: oid 42 updating version 2430 to 2432 2024-04-26T00:37:57.297 INFO:tasks.rados.rados.0.smithi045.stdout:3149: oid 42 version 2432 is already newer than 2429 2024-04-26T00:37:57.297 INFO:tasks.rados.rados.0.smithi045.stdout:3149: oid 42 version 2432 is already newer than 2431 2024-04-26T00:37:57.297 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 2432 (ObjNum 1128 snap 340 seq_num 1128) dirty exists 2024-04-26T00:37:57.297 INFO:tasks.rados.rados.0.smithi045.stdout:3149: left oid 42 (ObjNum 1128 snap 340 seq_num 1128) 2024-04-26T00:37:57.297 INFO:tasks.rados.rados.0.smithi045.stdout:3150: finishing write tid 1 to smithi04526579-19 2024-04-26T00:37:57.298 INFO:tasks.rados.rados.0.smithi045.stdout:3150: finishing write tid 2 to smithi04526579-19 2024-04-26T00:37:57.298 INFO:tasks.rados.rados.0.smithi045.stdout:3150: finishing write tid 3 to smithi04526579-19 2024-04-26T00:37:57.298 INFO:tasks.rados.rados.0.smithi045.stdout:3150: finishing write tid 4 to smithi04526579-19 2024-04-26T00:37:57.298 INFO:tasks.rados.rados.0.smithi045.stdout:3150: oid 19 updating version 0 to 2435 2024-04-26T00:37:57.298 INFO:tasks.rados.rados.0.smithi045.stdout:3150: oid 19 version 2435 is already newer than 2434 2024-04-26T00:37:57.298 INFO:tasks.rados.rados.0.smithi045.stdout:3150: oid 19 version 2435 is already newer than 2433 2024-04-26T00:37:57.298 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 2435 (ObjNum 1129 snap 340 seq_num 1129) dirty exists 2024-04-26T00:37:57.298 INFO:tasks.rados.rados.0.smithi045.stdout:3150: left oid 19 (ObjNum 1129 snap 340 seq_num 1129) 2024-04-26T00:37:57.298 INFO:tasks.rados.rados.0.smithi045.stdout:3144: done (7 left) 2024-04-26T00:37:57.298 INFO:tasks.rados.rados.0.smithi045.stdout:3145: done (6 left) 2024-04-26T00:37:57.298 INFO:tasks.rados.rados.0.smithi045.stdout:3146: done (5 left) 2024-04-26T00:37:57.299 INFO:tasks.rados.rados.0.smithi045.stdout:3147: done (4 left) 2024-04-26T00:37:57.299 INFO:tasks.rados.rados.0.smithi045.stdout:3148: done (3 left) 2024-04-26T00:37:57.299 INFO:tasks.rados.rados.0.smithi045.stdout:3149: done (2 left) 2024-04-26T00:37:57.299 INFO:tasks.rados.rados.0.smithi045.stdout:3150: done (1 left) 2024-04-26T00:37:57.299 INFO:tasks.rados.rados.0.smithi045.stdout:3151: done (0 left) 2024-04-26T00:37:57.299 INFO:tasks.rados.rados.0.smithi045.stdout:3152: rollback oid 22 current snap is 340 2024-04-26T00:37:57.299 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 22 to 337 2024-04-26T00:37:57.299 INFO:tasks.rados.rados.0.smithi045.stdout:3153: write oid 37 current snap is 340 2024-04-26T00:37:57.299 INFO:tasks.rados.rados.0.smithi045.stdout:3153: seq_num 1131 ranges {608717=726863,1933605=748052} 2024-04-26T00:37:57.316 INFO:tasks.rados.rados.0.smithi045.stdout:3153: writing smithi04526579-37 from 608717 to 1335580 tid 1 2024-04-26T00:37:57.338 INFO:tasks.rados.rados.0.smithi045.stdout:3153: writing smithi04526579-37 from 1933605 to 2681657 tid 2 2024-04-26T00:37:57.338 INFO:tasks.rados.rados.0.smithi045.stdout:3154: snap_remove snap 333 2024-04-26T00:37:57.601 INFO:tasks.rados.rados.0.smithi045.stdout:3154: done (2 left) 2024-04-26T00:37:57.602 INFO:tasks.rados.rados.0.smithi045.stdout:3155: snap_create 2024-04-26T00:37:57.602 INFO:tasks.rados.rados.0.smithi045.stdout:3152: finishing rollback tid 0 to smithi04526579-22 2024-04-26T00:37:57.602 INFO:tasks.rados.rados.0.smithi045.stdout:3152: finishing rollback tid 1 to smithi04526579-22 2024-04-26T00:37:57.602 INFO:tasks.rados.rados.0.smithi045.stdout:3152: finishing rollback tid 2 to smithi04526579-22 2024-04-26T00:37:57.602 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 2425 (ObjNum 1106 snap 329 seq_num 1106) dirty exists 2024-04-26T00:37:57.602 INFO:tasks.rados.rados.0.smithi045.stdout:3153: finishing write tid 1 to smithi04526579-37 2024-04-26T00:37:57.602 INFO:tasks.rados.rados.0.smithi045.stdout:3153: finishing write tid 2 to smithi04526579-37 2024-04-26T00:37:57.602 INFO:tasks.rados.rados.0.smithi045.stdout:3153: finishing write tid 3 to smithi04526579-37 2024-04-26T00:37:57.602 INFO:tasks.rados.rados.0.smithi045.stdout:3153: finishing write tid 4 to smithi04526579-37 2024-04-26T00:37:57.602 INFO:tasks.rados.rados.0.smithi045.stdout:3153: oid 37 updating version 0 to 2452 2024-04-26T00:37:57.602 INFO:tasks.rados.rados.0.smithi045.stdout:3153: oid 37 updating version 2452 to 2453 2024-04-26T00:37:57.603 INFO:tasks.rados.rados.0.smithi045.stdout:3153: oid 37 updating version 2453 to 2454 2024-04-26T00:37:57.603 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 2454 (ObjNum 1131 snap 340 seq_num 1131) dirty exists 2024-04-26T00:37:57.603 INFO:tasks.rados.rados.0.smithi045.stdout:3153: left oid 37 (ObjNum 1131 snap 340 seq_num 1131) 2024-04-26T00:37:58.578 INFO:tasks.rados.rados.0.smithi045.stdout:3152: done (2 left) 2024-04-26T00:37:58.578 INFO:tasks.rados.rados.0.smithi045.stdout:3153: done (1 left) 2024-04-26T00:37:58.578 INFO:tasks.rados.rados.0.smithi045.stdout:3155: done (0 left) 2024-04-26T00:37:58.578 INFO:tasks.rados.rados.0.smithi045.stdout:3156: snap_create 2024-04-26T00:37:59.580 INFO:tasks.rados.rados.0.smithi045.stdout:3156: done (0 left) 2024-04-26T00:37:59.580 INFO:tasks.rados.rados.0.smithi045.stdout:3157: write (excl) oid 47 current snap is 342 2024-04-26T00:37:59.581 INFO:tasks.rados.rados.0.smithi045.stdout:3157: seq_num 1132 ranges {625524=402112,1592087=608063,2869769=155755} 2024-04-26T00:37:59.597 INFO:tasks.rados.rados.0.smithi045.stdout:3157: writing smithi04526579-47 from 625524 to 1027636 tid 1 2024-04-26T00:37:59.616 INFO:tasks.rados.rados.0.smithi045.stdout:3157: writing smithi04526579-47 from 1592087 to 2200150 tid 2 2024-04-26T00:37:59.630 INFO:tasks.rados.rados.0.smithi045.stdout:3157: writing smithi04526579-47 from 2869769 to 3025524 tid 3 2024-04-26T00:37:59.630 INFO:tasks.rados.rados.0.smithi045.stdout:3158: rollback oid 37 current snap is 342 2024-04-26T00:37:59.630 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 37 to 330 2024-04-26T00:37:59.631 INFO:tasks.rados.rados.0.smithi045.stdout:3157: finishing write tid 1 to smithi04526579-47 2024-04-26T00:37:59.631 INFO:tasks.rados.rados.0.smithi045.stdout:3157: finishing write tid 2 to smithi04526579-47 2024-04-26T00:37:59.631 INFO:tasks.rados.rados.0.smithi045.stdout:3159: snap_create 2024-04-26T00:37:59.637 INFO:tasks.rados.rados.0.smithi045.stdout:3157: finishing write tid 3 to smithi04526579-47 2024-04-26T00:37:59.637 INFO:tasks.rados.rados.0.smithi045.stdout:3157: finishing write tid 4 to smithi04526579-47 2024-04-26T00:37:59.638 INFO:tasks.rados.rados.0.smithi045.stdout:3157: finishing write tid 5 to smithi04526579-47 2024-04-26T00:37:59.638 INFO:tasks.rados.rados.0.smithi045.stdout:3157: oid 47 updating version 0 to 2347 2024-04-26T00:37:59.638 INFO:tasks.rados.rados.0.smithi045.stdout:3157: oid 47 version 2347 is already newer than 2346 2024-04-26T00:37:59.638 INFO:tasks.rados.rados.0.smithi045.stdout:3157: oid 47 version 2347 is already newer than 2345 2024-04-26T00:37:59.638 INFO:tasks.rados.rados.0.smithi045.stdout:3157: oid 47 version 2347 is already newer than 2344 2024-04-26T00:37:59.638 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 2347 (ObjNum 1132 snap 342 seq_num 1132) dirty exists 2024-04-26T00:37:59.638 INFO:tasks.rados.rados.0.smithi045.stdout:3157: left oid 47 (ObjNum 1132 snap 342 seq_num 1132) 2024-04-26T00:37:59.646 INFO:tasks.rados.rados.0.smithi045.stdout:3158: finishing rollback tid 0 to smithi04526579-37 2024-04-26T00:37:59.646 INFO:tasks.rados.rados.0.smithi045.stdout:3158: finishing rollback tid 1 to smithi04526579-37 2024-04-26T00:37:59.646 INFO:tasks.rados.rados.0.smithi045.stdout:3158: finishing rollback tid 2 to smithi04526579-37 2024-04-26T00:37:59.647 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 2462 (ObjNum 1051 snap 317 seq_num 1051) dirty exists 2024-04-26T00:38:00.607 INFO:tasks.rados.rados.0.smithi045.stdout:3157: done (2 left) 2024-04-26T00:38:00.607 INFO:tasks.rados.rados.0.smithi045.stdout:3158: done (1 left) 2024-04-26T00:38:00.607 INFO:tasks.rados.rados.0.smithi045.stdout:3159: done (0 left) 2024-04-26T00:38:00.607 INFO:tasks.rados.rados.0.smithi045.stdout:3160: read oid 20 snap -1 2024-04-26T00:38:00.607 INFO:tasks.rados.rados.0.smithi045.stdout:3160: expect (ObjNum 1093 snap 324 seq_num 1093) 2024-04-26T00:38:00.607 INFO:tasks.rados.rados.0.smithi045.stdout:3161: read oid 30 snap 330 2024-04-26T00:38:00.607 INFO:tasks.rados.rados.0.smithi045.stdout:3161: expect (ObjNum 1087 snap 324 seq_num 1087) 2024-04-26T00:38:00.607 INFO:tasks.rados.rados.0.smithi045.stdout:3162: delete oid 23 current snap is 343 2024-04-26T00:38:00.619 INFO:tasks.rados.rados.0.smithi045.stdout:3161: expect (ObjNum 1087 snap 324 seq_num 1087) 2024-04-26T00:38:00.803 INFO:tasks.rados.rados.0.smithi045.stdout:3160: expect (ObjNum 1093 snap 324 seq_num 1093) 2024-04-26T00:38:00.881 INFO:tasks.rados.rados.0.smithi045.stdout:3160: done (2 left) 2024-04-26T00:38:00.881 INFO:tasks.rados.rados.0.smithi045.stdout:3161: done (1 left) 2024-04-26T00:38:00.881 INFO:tasks.rados.rados.0.smithi045.stdout:3162: done (0 left) 2024-04-26T00:38:00.882 INFO:tasks.rados.rados.0.smithi045.stdout:3163: delete oid 39 current snap is 343 2024-04-26T00:38:00.887 INFO:tasks.rados.rados.0.smithi045.stdout:3163: done (0 left) 2024-04-26T00:38:00.888 INFO:tasks.rados.rados.0.smithi045.stdout:3164: write (excl) oid 22 current snap is 343 2024-04-26T00:38:00.888 INFO:tasks.rados.rados.0.smithi045.stdout:3164: seq_num 1135 ranges {675945=744212,2167533=755390} 2024-04-26T00:38:00.911 INFO:tasks.rados.rados.0.smithi045.stdout:3164: writing smithi04526579-22 from 675945 to 1420157 tid 1 2024-04-26T00:38:00.936 INFO:tasks.rados.rados.0.smithi045.stdout:3164: writing smithi04526579-22 from 2167533 to 2922923 tid 2 2024-04-26T00:38:00.936 INFO:tasks.rados.rados.0.smithi045.stdout:3165: write oid 46 current snap is 343 2024-04-26T00:38:00.936 INFO:tasks.rados.rados.0.smithi045.stdout:3164: finishing write tid 1 to smithi04526579-22 2024-04-26T00:38:00.936 INFO:tasks.rados.rados.0.smithi045.stdout:3165: seq_num 1136 ranges {692752=419461,1826015=615401,2979461=113291} 2024-04-26T00:38:00.953 INFO:tasks.rados.rados.0.smithi045.stdout:3165: writing smithi04526579-46 from 692752 to 1112213 tid 1 2024-04-26T00:38:00.972 INFO:tasks.rados.rados.0.smithi045.stdout:3165: writing smithi04526579-46 from 1826015 to 2441416 tid 2 2024-04-26T00:38:00.980 INFO:tasks.rados.rados.0.smithi045.stdout:3165: writing smithi04526579-46 from 2979461 to 3092752 tid 3 2024-04-26T00:38:00.980 INFO:tasks.rados.rados.0.smithi045.stdout:3166: read oid 23 snap -1 2024-04-26T00:38:00.980 INFO:tasks.rados.rados.0.smithi045.stdout:3166: expect deleted 2024-04-26T00:38:00.980 INFO:tasks.rados.rados.0.smithi045.stdout:3167: write (excl) oid 13 current snap is 343 2024-04-26T00:38:00.981 INFO:tasks.rados.rados.0.smithi045.stdout:3167: seq_num 1137 ranges {709559=494710,1884497=759059} 2024-04-26T00:38:00.992 INFO:tasks.rados.rados.0.smithi045.stdout:3167: writing smithi04526579-13 from 709559 to 1204269 tid 1 2024-04-26T00:38:01.005 INFO:tasks.rados.rados.0.smithi045.stdout:3167: writing smithi04526579-13 from 1884497 to 2643556 tid 2 2024-04-26T00:38:01.005 INFO:tasks.rados.rados.0.smithi045.stdout:3168: snap_remove snap 334 2024-04-26T00:38:01.620 INFO:tasks.rados.rados.0.smithi045.stdout:3168: done (4 left) 2024-04-26T00:38:01.620 INFO:tasks.rados.rados.0.smithi045.stdout:3169: read oid 38 snap -1 2024-04-26T00:38:01.620 INFO:tasks.rados.rados.0.smithi045.stdout:3169: expect (ObjNum 1075 snap 322 seq_num 1075) 2024-04-26T00:38:01.620 INFO:tasks.rados.rados.0.smithi045.stdout:3170: read oid 2 snap 340 2024-04-26T00:38:01.620 INFO:tasks.rados.rados.0.smithi045.stdout:3170: expect deleted 2024-04-26T00:38:01.621 INFO:tasks.rados.rados.0.smithi045.stdout:3171: rollback oid 45 current snap is 343 2024-04-26T00:38:01.621 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 45 to 338 2024-04-26T00:38:01.621 INFO:tasks.rados.rados.0.smithi045.stdout:3164: finishing write tid 2 to smithi04526579-22 2024-04-26T00:38:01.621 INFO:tasks.rados.rados.0.smithi045.stdout:3164: finishing write tid 3 to smithi04526579-22 2024-04-26T00:38:01.621 INFO:tasks.rados.rados.0.smithi045.stdout:3164: finishing write tid 4 to smithi04526579-22 2024-04-26T00:38:01.621 INFO:tasks.rados.rados.0.smithi045.stdout:3164: oid 22 updating version 0 to 2426 2024-04-26T00:38:01.621 INFO:tasks.rados.rados.0.smithi045.stdout:3164: oid 22 updating version 2426 to 2428 2024-04-26T00:38:01.621 INFO:tasks.rados.rados.0.smithi045.stdout:3164: oid 22 version 2428 is already newer than 2427 2024-04-26T00:38:01.621 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 2428 (ObjNum 1135 snap 343 seq_num 1135) dirty exists 2024-04-26T00:38:01.621 INFO:tasks.rados.rados.0.smithi045.stdout:3164: left oid 22 (ObjNum 1135 snap 343 seq_num 1135) 2024-04-26T00:38:01.621 INFO:tasks.rados.rados.0.smithi045.stdout:3165: finishing write tid 1 to smithi04526579-46 2024-04-26T00:38:01.621 INFO:tasks.rados.rados.0.smithi045.stdout:3164: done (6 left) 2024-04-26T00:38:01.621 INFO:tasks.rados.rados.0.smithi045.stdout:3172: snap_create 2024-04-26T00:38:01.621 INFO:tasks.rados.rados.0.smithi045.stdout:3165: finishing write tid 2 to smithi04526579-46 2024-04-26T00:38:01.622 INFO:tasks.rados.rados.0.smithi045.stdout:3165: finishing write tid 3 to smithi04526579-46 2024-04-26T00:38:01.622 INFO:tasks.rados.rados.0.smithi045.stdout:3165: finishing write tid 4 to smithi04526579-46 2024-04-26T00:38:01.622 INFO:tasks.rados.rados.0.smithi045.stdout:3165: finishing write tid 5 to smithi04526579-46 2024-04-26T00:38:01.622 INFO:tasks.rados.rados.0.smithi045.stdout:3165: oid 46 updating version 0 to 2431 2024-04-26T00:38:01.622 INFO:tasks.rados.rados.0.smithi045.stdout:3165: oid 46 updating version 2431 to 2432 2024-04-26T00:38:01.622 INFO:tasks.rados.rados.0.smithi045.stdout:3165: oid 46 version 2432 is already newer than 2430 2024-04-26T00:38:01.622 INFO:tasks.rados.rados.0.smithi045.stdout:3165: oid 46 version 2432 is already newer than 2429 2024-04-26T00:38:01.622 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 2432 (ObjNum 1136 snap 343 seq_num 1136) dirty exists 2024-04-26T00:38:01.622 INFO:tasks.rados.rados.0.smithi045.stdout:3165: left oid 46 (ObjNum 1136 snap 343 seq_num 1136) 2024-04-26T00:38:01.622 INFO:tasks.rados.rados.0.smithi045.stdout:3167: finishing write tid 1 to smithi04526579-13 2024-04-26T00:38:01.622 INFO:tasks.rados.rados.0.smithi045.stdout:3167: finishing write tid 2 to smithi04526579-13 2024-04-26T00:38:01.622 INFO:tasks.rados.rados.0.smithi045.stdout:3167: finishing write tid 3 to smithi04526579-13 2024-04-26T00:38:01.622 INFO:tasks.rados.rados.0.smithi045.stdout:3167: finishing write tid 4 to smithi04526579-13 2024-04-26T00:38:01.622 INFO:tasks.rados.rados.0.smithi045.stdout:3167: oid 13 updating version 0 to 2198 2024-04-26T00:38:01.623 INFO:tasks.rados.rados.0.smithi045.stdout:3167: oid 13 updating version 2198 to 2199 2024-04-26T00:38:01.623 INFO:tasks.rados.rados.0.smithi045.stdout:3167: oid 13 updating version 2199 to 2200 2024-04-26T00:38:01.623 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 2200 (ObjNum 1137 snap 343 seq_num 1137) dirty exists 2024-04-26T00:38:01.623 INFO:tasks.rados.rados.0.smithi045.stdout:3167: left oid 13 (ObjNum 1137 snap 343 seq_num 1137) 2024-04-26T00:38:01.632 INFO:tasks.rados.rados.0.smithi045.stdout:3169: expect (ObjNum 1075 snap 322 seq_num 1075) 2024-04-26T00:38:01.787 INFO:tasks.rados.rados.0.smithi045.stdout:3171: finishing rollback tid 0 to smithi04526579-45 2024-04-26T00:38:01.787 INFO:tasks.rados.rados.0.smithi045.stdout:3171: finishing rollback tid 1 to smithi04526579-45 2024-04-26T00:38:01.787 INFO:tasks.rados.rados.0.smithi045.stdout:3171: finishing rollback tid 2 to smithi04526579-45 2024-04-26T00:38:01.787 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 2198 (ObjNum 1062 snap 318 seq_num 1062) dirty exists 2024-04-26T00:38:02.628 INFO:tasks.rados.rados.0.smithi045.stdout:3165: done (6 left) 2024-04-26T00:38:02.628 INFO:tasks.rados.rados.0.smithi045.stdout:3166: done (5 left) 2024-04-26T00:38:02.628 INFO:tasks.rados.rados.0.smithi045.stdout:3167: done (4 left) 2024-04-26T00:38:02.628 INFO:tasks.rados.rados.0.smithi045.stdout:3169: done (3 left) 2024-04-26T00:38:02.628 INFO:tasks.rados.rados.0.smithi045.stdout:3170: done (2 left) 2024-04-26T00:38:02.629 INFO:tasks.rados.rados.0.smithi045.stdout:3171: done (1 left) 2024-04-26T00:38:02.629 INFO:tasks.rados.rados.0.smithi045.stdout:3172: done (0 left) 2024-04-26T00:38:02.629 INFO:tasks.rados.rados.0.smithi045.stdout:3173: read oid 27 snap -1 2024-04-26T00:38:02.629 INFO:tasks.rados.rados.0.smithi045.stdout:3173: expect (ObjNum 1051 snap 317 seq_num 1051) 2024-04-26T00:38:02.629 INFO:tasks.rados.rados.0.smithi045.stdout:3174: copy_from oid 15 from oid 43 current snap is 344 2024-04-26T00:38:02.629 INFO:tasks.rados.rados.0.smithi045.stdout:3175: snap_remove snap 336 2024-04-26T00:38:03.582 INFO:tasks.rados.rados.0.smithi045.stdout:3175: done (2 left) 2024-04-26T00:38:03.582 INFO:tasks.rados.rados.0.smithi045.stdout:3176: rollback oid 20 current snap is 344 2024-04-26T00:38:03.582 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 20 to 337 2024-04-26T00:38:03.582 INFO:tasks.rados.rados.0.smithi045.stdout:3173: expect (ObjNum 1051 snap 317 seq_num 1051) 2024-04-26T00:38:03.817 INFO:tasks.rados.rados.0.smithi045.stdout:3174: finishing copy_from racing read to smithi04526579-15 2024-04-26T00:38:03.817 INFO:tasks.rados.rados.0.smithi045.stdout:3174: finishing copy_from to smithi04526579-15 2024-04-26T00:38:03.817 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 2196 (ObjNum 1103 snap 329 seq_num 1103) dirty exists 2024-04-26T00:38:03.817 INFO:tasks.rados.rados.0.smithi045.stdout:3173: done (2 left) 2024-04-26T00:38:03.817 INFO:tasks.rados.rados.0.smithi045.stdout:3174: done (1 left) 2024-04-26T00:38:03.817 INFO:tasks.rados.rados.0.smithi045.stdout:3177: read oid 38 snap -1 2024-04-26T00:38:03.817 INFO:tasks.rados.rados.0.smithi045.stdout:3177: expect (ObjNum 1075 snap 322 seq_num 1075) 2024-04-26T00:38:03.817 INFO:tasks.rados.rados.0.smithi045.stdout:3178: snap_remove snap 341 2024-04-26T00:38:04.636 INFO:tasks.rados.rados.0.smithi045.stdout:3178: done (2 left) 2024-04-26T00:38:04.636 INFO:tasks.rados.rados.0.smithi045.stdout:3179: delete oid 41 current snap is 344 2024-04-26T00:38:04.637 INFO:tasks.rados.rados.0.smithi045.stdout:3176: finishing rollback tid 0 to smithi04526579-20 2024-04-26T00:38:04.637 INFO:tasks.rados.rados.0.smithi045.stdout:3176: finishing rollback tid 1 to smithi04526579-20 2024-04-26T00:38:04.637 INFO:tasks.rados.rados.0.smithi045.stdout:3176: finishing rollback tid 2 to smithi04526579-20 2024-04-26T00:38:04.637 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 2379 (ObjNum 1093 snap 324 seq_num 1093) dirty exists 2024-04-26T00:38:04.637 INFO:tasks.rados.rados.0.smithi045.stdout:3177: expect (ObjNum 1075 snap 322 seq_num 1075) 2024-04-26T00:38:04.737 INFO:tasks.rados.rados.0.smithi045.stdout:3176: done (2 left) 2024-04-26T00:38:04.738 INFO:tasks.rados.rados.0.smithi045.stdout:3177: done (1 left) 2024-04-26T00:38:04.738 INFO:tasks.rados.rados.0.smithi045.stdout:3179: done (0 left) 2024-04-26T00:38:04.738 INFO:tasks.rados.rados.0.smithi045.stdout:3180: read oid 20 snap 335 2024-04-26T00:38:04.738 INFO:tasks.rados.rados.0.smithi045.stdout:3180: expect (ObjNum 1065 snap 320 seq_num 1065) 2024-04-26T00:38:04.738 INFO:tasks.rados.rados.0.smithi045.stdout:3181: snap_create 2024-04-26T00:38:04.756 INFO:tasks.rados.rados.0.smithi045.stdout:3180: expect (ObjNum 1065 snap 320 seq_num 1065) 2024-04-26T00:38:05.661 INFO:tasks.rados.rados.0.smithi045.stdout:3180: done (1 left) 2024-04-26T00:38:05.661 INFO:tasks.rados.rados.0.smithi045.stdout:3181: done (0 left) 2024-04-26T00:38:05.661 INFO:tasks.rados.rados.0.smithi045.stdout:3182: write oid 15 current snap is 345 2024-04-26T00:38:05.661 INFO:tasks.rados.rados.0.smithi045.stdout:3182: seq_num 1139 ranges {743173=645208,1885108=762728} 2024-04-26T00:38:05.683 INFO:tasks.rados.rados.0.smithi045.stdout:3182: writing smithi04526579-15 from 743173 to 1388381 tid 1 2024-04-26T00:38:05.705 INFO:tasks.rados.rados.0.smithi045.stdout:3182: writing smithi04526579-15 from 1885108 to 2647836 tid 2 2024-04-26T00:38:05.705 INFO:tasks.rados.rados.0.smithi045.stdout:3183: write oid 24 current snap is 345 2024-04-26T00:38:05.705 INFO:tasks.rados.rados.0.smithi045.stdout:3183: seq_num 1140 ranges {759980=720457,1943590=506386,2856447=303533} 2024-04-26T00:38:05.726 INFO:tasks.rados.rados.0.smithi045.stdout:3183: writing smithi04526579-24 from 759980 to 1480437 tid 1 2024-04-26T00:38:05.736 INFO:tasks.rados.rados.0.smithi045.stdout:3183: writing smithi04526579-24 from 1943590 to 2449976 tid 2 2024-04-26T00:38:05.743 INFO:tasks.rados.rados.0.smithi045.stdout:3183: writing smithi04526579-24 from 2856447 to 3159980 tid 3 2024-04-26T00:38:05.743 INFO:tasks.rados.rados.0.smithi045.stdout:3184: snap_create 2024-04-26T00:38:05.743 INFO:tasks.rados.rados.0.smithi045.stdout:3182: finishing write tid 1 to smithi04526579-15 2024-04-26T00:38:05.743 INFO:tasks.rados.rados.0.smithi045.stdout:3182: finishing write tid 2 to smithi04526579-15 2024-04-26T00:38:05.743 INFO:tasks.rados.rados.0.smithi045.stdout:3182: finishing write tid 3 to smithi04526579-15 2024-04-26T00:38:05.744 INFO:tasks.rados.rados.0.smithi045.stdout:3182: finishing write tid 4 to smithi04526579-15 2024-04-26T00:38:05.744 INFO:tasks.rados.rados.0.smithi045.stdout:3182: oid 15 updating version 0 to 2442 2024-04-26T00:38:05.744 INFO:tasks.rados.rados.0.smithi045.stdout:3182: oid 15 updating version 2442 to 2443 2024-04-26T00:38:05.744 INFO:tasks.rados.rados.0.smithi045.stdout:3182: oid 15 updating version 2443 to 2444 2024-04-26T00:38:05.744 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 2444 (ObjNum 1139 snap 345 seq_num 1139) dirty exists 2024-04-26T00:38:05.744 INFO:tasks.rados.rados.0.smithi045.stdout:3182: left oid 15 (ObjNum 1139 snap 345 seq_num 1139) 2024-04-26T00:38:05.744 INFO:tasks.rados.rados.0.smithi045.stdout:3183: finishing write tid 1 to smithi04526579-24 2024-04-26T00:38:05.752 INFO:tasks.rados.rados.0.smithi045.stdout:3183: finishing write tid 2 to smithi04526579-24 2024-04-26T00:38:05.752 INFO:tasks.rados.rados.0.smithi045.stdout:3183: finishing write tid 3 to smithi04526579-24 2024-04-26T00:38:05.753 INFO:tasks.rados.rados.0.smithi045.stdout:3183: finishing write tid 4 to smithi04526579-24 2024-04-26T00:38:05.754 INFO:tasks.rados.rados.0.smithi045.stdout:3183: finishing write tid 5 to smithi04526579-24 2024-04-26T00:38:05.754 INFO:tasks.rados.rados.0.smithi045.stdout:3183: oid 24 updating version 0 to 2474 2024-04-26T00:38:05.754 INFO:tasks.rados.rados.0.smithi045.stdout:3183: oid 24 updating version 2474 to 2475 2024-04-26T00:38:05.754 INFO:tasks.rados.rados.0.smithi045.stdout:3183: oid 24 updating version 2475 to 2476 2024-04-26T00:38:05.754 INFO:tasks.rados.rados.0.smithi045.stdout:3183: oid 24 version 2476 is already newer than 2473 2024-04-26T00:38:05.754 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 2476 (ObjNum 1140 snap 345 seq_num 1140) dirty exists 2024-04-26T00:38:05.754 INFO:tasks.rados.rados.0.smithi045.stdout:3183: left oid 24 (ObjNum 1140 snap 345 seq_num 1140) 2024-04-26T00:38:06.247 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-04-26T00:38:06.248 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-04-26T00:38:06.248 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-26T00:38:06.248 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-26T00:38:06.251 INFO:tasks.ceph.mon.b:Started 2024-04-26T00:38:06.251 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:38:06.251 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:38:06.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.383+0000 7f6189541640 10 monclient: get_monmap_and_config 2024-04-26T00:38:06.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.383+0000 7f6189541640 10 monclient: build_initial_monmap 2024-04-26T00:38:06.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.383+0000 7f6189541640 10 monclient: monmap: 2024-04-26T00:38:06.382 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:38:06.382 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:06.382 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:38:06.385738+0000 2024-04-26T00:38:06.382 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:38:06.385738+0000 2024-04-26T00:38:06.382 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:38:06.382 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:06.382 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:38:06.383 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:38:06.383 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:38:06.383 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:38:06.383 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:38:06.383 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:06.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.383+0000 7f6189541640 1 Processor -- start 2024-04-26T00:38:06.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.383+0000 7f6189541640 1 -- start start 2024-04-26T00:38:06.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.383+0000 7f6189541640 10 monclient: init 2024-04-26T00:38:06.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.383+0000 7f6189541640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:06.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.383+0000 7f6189541640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-04-26T00:38:06.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.383+0000 7f6189541640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6184136da0 0x7f618409bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:06.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.383+0000 7f6189541640 10 monclient(hunting): picked mon.noname-b con 0x7f6184136da0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:38:06.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.383+0000 7f6189541640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f618409c410 addr v1:172.21.15.45:6789/0 2024-04-26T00:38:06.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.383+0000 7f6189541640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6184137890 0x7f618413e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:06.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.383+0000 7f6189541640 10 monclient(hunting): picked mon.noname-a con 0x7f6184137890 addr v2:172.21.15.37:3300/0 2024-04-26T00:38:06.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.383+0000 7f6189541640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f618409b4f0 con 0x7f618409c410 2024-04-26T00:38:06.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.383+0000 7f6189541640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:06.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.383+0000 7f6189541640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f618409bb40 con 0x7f6184137890 2024-04-26T00:38:06.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.383+0000 7f6189541640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:06.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.383+0000 7f6189541640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f618409b210 con 0x7f6184136da0 2024-04-26T00:38:06.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.383+0000 7f6189541640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:06.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.383+0000 7f6189541640 10 monclient(hunting): authenticate will time out at 2067.691451s 2024-04-26T00:38:06.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.383+0000 7f6182ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6184136da0 0x7f618409bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:06.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.383+0000 7f61837fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6184137890 0x7f618413e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:06.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.383+0000 7f61837fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6184137890 0x7f618413e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:43686/0 (socket says 172.21.15.37:43686) 2024-04-26T00:38:06.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.383+0000 7f6182ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6184136da0 0x7f618409bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:59500/0 (socket says 172.21.15.37:59500) 2024-04-26T00:38:06.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.383+0000 7f61837fe640 1 -- 172.21.15.37:0/422533347 learned_addr learned my addr 172.21.15.37:0/422533347 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:38:06.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.383+0000 7f61837fe640 10 monclient(hunting): get_auth_request con 0x7f6184137890 auth_method 0 2024-04-26T00:38:06.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.383+0000 7f61837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:06.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.383+0000 7f61837fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:06.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.383+0000 7f61837fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:06.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.383+0000 7f6182ffd640 10 monclient(hunting): get_auth_request con 0x7f6184136da0 auth_method 0 2024-04-26T00:38:06.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.383+0000 7f6182ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:06.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.383+0000 7f6182ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:06.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.383+0000 7f6182ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:06.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.383+0000 7f61837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:06.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.383+0000 7f61837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:06.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.383+0000 7f61837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:06.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.383+0000 7f6182ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:06.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.383+0000 7f6182ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:06.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.383+0000 7f6182ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:06.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.387+0000 7f6182ffd640 10 monclient(hunting): handle_auth_done global_id 204266 payload 274 2024-04-26T00:38:06.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.387+0000 7f6182ffd640 1 -- 172.21.15.37:0/422533347 >> v2:172.21.15.37:3300/0 conn(0x7f6184137890 msgr2=0x7f618413e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:38:06.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.387+0000 7f6182ffd640 1 --2- 172.21.15.37:0/422533347 >> v2:172.21.15.37:3300/0 conn(0x7f6184137890 0x7f618413e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:06.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.387+0000 7f6182ffd640 1 -- 172.21.15.37:0/422533347 >> v1:172.21.15.45:6789/0 conn(0x7f618409c410 legacy=0x7f6184137180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:38:06.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.387+0000 7f6182ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:38:06.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.387+0000 7f6182ffd640 1 monclient: found mon.noname-b 2024-04-26T00:38:06.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.387+0000 7f6182ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:06.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.387+0000 7f6182ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:38:06.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.387+0000 7f6182ffd640 1 -- 172.21.15.37:0/422533347 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f618409b7d0 con 0x7f6184136da0 2024-04-26T00:38:06.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.387+0000 7f61837fe640 1 --2- 172.21.15.37:0/422533347 >> v2:172.21.15.37:3300/0 conn(0x7f6184137890 0x7f618413e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:38:06.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6182ffd640 1 -- 172.21.15.37:0/422533347 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6184136da0 msgr2=0x7f618409bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:06.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6182ffd640 1 --2- 172.21.15.37:0/422533347 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6184136da0 0x7f618409bed0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f616c0097c0 tx=0x7f616c01c320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd857ca19760c304 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:06.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6181ffb640 1 -- 172.21.15.37:0/422533347 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f616c0b0070 con 0x7f6184136da0 2024-04-26T00:38:06.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6181ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:06.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6181ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:38:06.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6181ffb640 10 monclient: dump: 2024-04-26T00:38:06.395 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:06.396 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:06.396 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:06.396 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:06.396 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:06.396 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:06.396 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:06.396 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:06.396 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:06.396 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:06.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6181ffb640 10 monclient: _finish_auth 0 2024-04-26T00:38:06.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6181ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:38:06.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6181ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:06.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6181ffb640 1 -- 172.21.15.37:0/422533347 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f616c0ab600 con 0x7f6184136da0 2024-04-26T00:38:06.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6181ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:38:06.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6181ffb640 1 -- 172.21.15.37:0/422533347 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f616c0ab9d0 con 0x7f6184136da0 2024-04-26T00:38:06.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6189541640 5 monclient: authenticate success, global_id 204266 2024-04-26T00:38:06.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6189541640 10 monclient: get_monmap_and_config success 2024-04-26T00:38:06.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6189541640 4 set_mon_vals no callback set 2024-04-26T00:38:06.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6181ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:06.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6181ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:38:06.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6181ffb640 10 monclient: dump: 2024-04-26T00:38:06.397 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:06.397 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:06.397 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:06.397 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:06.397 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:06.397 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:06.397 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:06.398 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:06.398 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:06.398 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:06.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6189541640 10 monclient: shutdown 2024-04-26T00:38:06.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6189541640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:38:06.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6189541640 1 -- 172.21.15.37:0/422533347 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6184136da0 msgr2=0x7f618409bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:06.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6189541640 1 --2- 172.21.15.37:0/422533347 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6184136da0 0x7f618409bed0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f616c0097c0 tx=0x7f616c01c320 comp rx=0 tx=0).stop 2024-04-26T00:38:06.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6189541640 1 -- 172.21.15.37:0/422533347 shutdown_connections 2024-04-26T00:38:06.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6189541640 1 --2- 172.21.15.37:0/422533347 >> v2:172.21.15.37:3300/0 conn(0x7f6184137890 0x7f618413e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:06.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6189541640 1 --2- 172.21.15.37:0/422533347 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6184136da0 0x7f618409bed0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:06.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6189541640 1 -- 172.21.15.37:0/422533347 >> 172.21.15.37:0/422533347 conn(0x7f61840a5f00 msgr2=0x7f61840a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:38:06.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6189541640 1 -- 172.21.15.37:0/422533347 shutdown_connections 2024-04-26T00:38:06.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6189541640 1 -- 172.21.15.37:0/422533347 wait complete. 2024-04-26T00:38:06.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6189541640 10 monclient: build_initial_monmap 2024-04-26T00:38:06.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6189541640 10 monclient: monmap: 2024-04-26T00:38:06.399 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:38:06.399 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:38:06.399 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:38:06.399 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:38:06.399 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:38:06.399 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:38:06.399 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:38:06.399 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:38:06.399 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:38:06.399 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:06.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6189541640 1 Processor -- start 2024-04-26T00:38:06.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6189541640 1 -- start start 2024-04-26T00:38:06.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6189541640 10 monclient: init 2024-04-26T00:38:06.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6189541640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:06.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6189541640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:38:06.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6189541640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f618409c410 0x7f61841cea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:06.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6189541640 10 monclient(hunting): picked mon.noname-b con 0x7f618409c410 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:38:06.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6189541640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6184136da0 0x7f61841cf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:06.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6189541640 10 monclient(hunting): picked mon.noname-c con 0x7f6184136da0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:38:06.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6189541640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6184137890 0x7f618413ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:06.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6189541640 10 monclient(hunting): picked mon.noname-a con 0x7f6184137890 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:38:06.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6189541640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:06.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6189541640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f6184141430 con 0x7f6184137890 2024-04-26T00:38:06.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6189541640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:06.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6189541640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f61841412b0 con 0x7f618409c410 2024-04-26T00:38:06.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6189541640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:06.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6189541640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f61841415b0 con 0x7f6184136da0 2024-04-26T00:38:06.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6189541640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:06.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6189541640 10 monclient(hunting): authenticate will time out at 2067.698573s 2024-04-26T00:38:06.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f61837fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6184137890 0x7f618413ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:06.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f61837fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6184137890 0x7f618413ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:43692/0 (socket says 172.21.15.37:43692) 2024-04-26T00:38:06.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f61837fe640 1 -- 172.21.15.37:0/2715383228 learned_addr learned my addr 172.21.15.37:0/2715383228 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:38:06.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f61837fe640 10 monclient(hunting): get_auth_request con 0x7f6184137890 auth_method 0 2024-04-26T00:38:06.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f61837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:06.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f61837fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:06.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f61837fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:06.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f61837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:06.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f61837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:06.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f61827fc640 1 --2- 172.21.15.37:0/2715383228 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6184136da0 0x7f61841cf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:06.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f61837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:06.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f61827fc640 10 monclient(hunting): get_auth_request con 0x7f6184136da0 auth_method 0 2024-04-26T00:38:06.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f61827fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:06.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f61827fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:06.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f61827fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:06.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6182ffd640 1 --2- 172.21.15.37:0/2715383228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f618409c410 0x7f61841cea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:06.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f61827fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:06.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f61827fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:06.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f61827fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:06.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6182ffd640 10 monclient(hunting): get_auth_request con 0x7f618409c410 auth_method 0 2024-04-26T00:38:06.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6182ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:06.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6182ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:06.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f6182ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:06.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f61837fe640 10 monclient(hunting): handle_auth_done global_id 144882 payload 824 2024-04-26T00:38:06.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f61837fe640 1 -- 172.21.15.37:0/2715383228 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6184136da0 msgr2=0x7f61841cf0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:06.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f61837fe640 1 --2- 172.21.15.37:0/2715383228 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6184136da0 0x7f61841cf0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:06.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f61837fe640 1 -- 172.21.15.37:0/2715383228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f618409c410 msgr2=0x7f61841cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:06.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f61837fe640 1 --2- 172.21.15.37:0/2715383228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f618409c410 0x7f61841cea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:06.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f61837fe640 10 monclient: _finish_hunting 0 2024-04-26T00:38:06.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f61837fe640 1 monclient: found mon.noname-a 2024-04-26T00:38:06.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f61837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:06.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f61837fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:38:06.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.391+0000 7f61837fe640 1 -- 172.21.15.37:0/2715383228 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f618413c8a0 con 0x7f6184137890 2024-04-26T00:38:06.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.395+0000 7f61837fe640 1 -- 172.21.15.37:0/2715383228 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6184137890 msgr2=0x7f618413ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:06.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.395+0000 7f61837fe640 1 --2- 172.21.15.37:0/2715383228 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6184137890 0x7f618413ac80 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f617c00efc0 tx=0x7f617c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:06.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.395+0000 7f617b7fe640 1 -- 172.21.15.37:0/2715383228 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617c00cdf0 con 0x7f6184137890 2024-04-26T00:38:06.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.395+0000 7f617b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:06.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.395+0000 7f617b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:38:06.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.395+0000 7f617b7fe640 10 monclient: dump: 2024-04-26T00:38:06.405 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:06.405 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:06.405 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:06.405 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:06.406 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:06.406 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:06.406 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:06.406 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:06.406 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:06.406 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:06.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.395+0000 7f617b7fe640 10 monclient: _finish_auth 0 2024-04-26T00:38:06.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.395+0000 7f617b7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:38:06.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.395+0000 7f617b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:06.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.395+0000 7f617b7fe640 1 -- 172.21.15.37:0/2715383228 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f617c0039a0 con 0x7f6184137890 2024-04-26T00:38:06.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.395+0000 7f6189541640 5 monclient: authenticate success, global_id 144882 2024-04-26T00:38:06.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.395+0000 7f6189541640 10 monclient: _renew_subs 2024-04-26T00:38:06.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.395+0000 7f6189541640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:38:06.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.395+0000 7f6189541640 1 -- 172.21.15.37:0/2715383228 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f618413d400 con 0x7f6184137890 2024-04-26T00:38:06.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.395+0000 7f617b7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:38:06.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.395+0000 7f617b7fe640 1 -- 172.21.15.37:0/2715383228 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617c003d70 con 0x7f6184137890 2024-04-26T00:38:06.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.395+0000 7f617b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:06.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.395+0000 7f617b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:38:06.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.395+0000 7f617b7fe640 10 monclient: dump: 2024-04-26T00:38:06.407 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:06.407 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:06.407 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:06.407 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:06.407 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:06.407 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:06.407 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:06.407 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:06.407 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:06.408 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:06.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.395+0000 7f61817fa640 4 set_mon_vals no callback set 2024-04-26T00:38:06.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.395+0000 7f6189541640 10 monclient: _renew_subs 2024-04-26T00:38:06.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.395+0000 7f6189541640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:38:06.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.395+0000 7f6189541640 1 -- 172.21.15.37:0/2715383228 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f618411ff10 con 0x7f6184137890 2024-04-26T00:38:06.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.395+0000 7f6189541640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:06.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.395+0000 7f6189541640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:06.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.395+0000 7f6189541640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:38:06.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.395+0000 7f6189541640 1 -- 172.21.15.37:0/2715383228 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6184066f40 con 0x7f6184137890 2024-04-26T00:38:06.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.395+0000 7f617b7fe640 1 -- 172.21.15.37:0/2715383228 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f617c01b030 con 0x7f6184137890 2024-04-26T00:38:06.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.395+0000 7f617b7fe640 1 --2- 172.21.15.37:0/2715383228 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6164035e80 0x7f6164038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:06.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.395+0000 7f617b7fe640 1 -- 172.21.15.37:0/2715383228 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(711..711 src has 370..711) ==== 3931+0+0 (secure 0 0 0) 0x7f617c048f10 con 0x7f6184137890 2024-04-26T00:38:06.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.395+0000 7f6182ffd640 1 --2- 172.21.15.37:0/2715383228 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6164035e80 0x7f6164038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:06.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.395+0000 7f6182ffd640 10 monclient: get_auth_request con 0x7f6164035e80 auth_method 0 2024-04-26T00:38:06.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.399+0000 7f6182ffd640 1 --2- 172.21.15.37:0/2715383228 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6164035e80 0x7f6164038340 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f616c0ab000 tx=0x7f616c0a4f90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:06.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.399+0000 7f617b7fe640 1 -- 172.21.15.37:0/2715383228 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f617c022020 con 0x7f6184137890 2024-04-26T00:38:06.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.399+0000 7f617b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:06.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.399+0000 7f617b7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:38:06.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.623+0000 7f6189541640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:38:06.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.623+0000 7f6189541640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:38:06.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.623+0000 7f6189541640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:38:06.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.623+0000 7f6189541640 1 -- 172.21.15.37:0/2715383228 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6184075d70 con 0x7f6184137890 2024-04-26T00:38:06.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.623+0000 7f617b7fe640 1 -- 172.21.15.37:0/2715383228 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7f617c010040 con 0x7f6184137890 2024-04-26T00:38:06.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.623+0000 7f617b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:38:06.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.623+0000 7f617b7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:38:06.622 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":136,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:38:06.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.623+0000 7f6189541640 1 -- 172.21.15.37:0/2715383228 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6164035e80 msgr2=0x7f6164038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:06.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.623+0000 7f6189541640 1 --2- 172.21.15.37:0/2715383228 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6164035e80 0x7f6164038340 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f616c0ab000 tx=0x7f616c0a4f90 comp rx=0 tx=0).stop 2024-04-26T00:38:06.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.623+0000 7f6189541640 10 monclient: shutdown 2024-04-26T00:38:06.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.623+0000 7f6189541640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:38:06.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.623+0000 7f6189541640 1 -- 172.21.15.37:0/2715383228 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6184137890 msgr2=0x7f618413ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:06.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.623+0000 7f6189541640 1 --2- 172.21.15.37:0/2715383228 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6184137890 0x7f618413ac80 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f617c00efc0 tx=0x7f617c00c490 comp rx=0 tx=0).stop 2024-04-26T00:38:06.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.623+0000 7f6189541640 1 -- 172.21.15.37:0/2715383228 shutdown_connections 2024-04-26T00:38:06.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.623+0000 7f6189541640 1 --2- 172.21.15.37:0/2715383228 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6164035e80 0x7f6164038340 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:06.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.623+0000 7f6189541640 1 --2- 172.21.15.37:0/2715383228 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6184137890 0x7f618413ac80 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:06.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.623+0000 7f6189541640 1 --2- 172.21.15.37:0/2715383228 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6184136da0 0x7f61841cf0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:06.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.623+0000 7f6189541640 1 --2- 172.21.15.37:0/2715383228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f618409c410 0x7f61841cea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:06.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.627+0000 7f6189541640 1 -- 172.21.15.37:0/2715383228 >> 172.21.15.37:0/2715383228 conn(0x7f61840a5f00 msgr2=0x7f61840a7200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:38:06.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.627+0000 7f6189541640 1 -- 172.21.15.37:0/2715383228 shutdown_connections 2024-04-26T00:38:06.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:06.627+0000 7f6189541640 1 -- 172.21.15.37:0/2715383228 wait complete. 2024-04-26T00:38:06.655 INFO:tasks.rados.rados.0.smithi045.stdout:3182: done (2 left) 2024-04-26T00:38:06.655 INFO:tasks.rados.rados.0.smithi045.stdout:3183: done (1 left) 2024-04-26T00:38:06.655 INFO:tasks.rados.rados.0.smithi045.stdout:3184: done (0 left) 2024-04-26T00:38:06.655 INFO:tasks.rados.rados.0.smithi045.stdout:3185: write (excl) oid 17 current snap is 346 2024-04-26T00:38:06.655 INFO:tasks.rados.rados.0.smithi045.stdout:3185: seq_num 1141 ranges {776787=512059,1718425=766397,3000223=176564} 2024-04-26T00:38:06.667 INFO:tasks.rados.rados.0.smithi045.stdout:3185: writing smithi04526579-17 from 776787 to 1288846 tid 1 2024-04-26T00:38:06.678 INFO:tasks.rados.rados.0.smithi045.stdout:3185: writing smithi04526579-17 from 1718425 to 2484822 tid 2 2024-04-26T00:38:06.684 INFO:tasks.rados.rados.0.smithi045.stdout:3185: writing smithi04526579-17 from 3000223 to 3176787 tid 3 2024-04-26T00:38:06.685 INFO:tasks.rados.rados.0.smithi045.stdout:3186: read oid 2 snap -1 2024-04-26T00:38:06.685 INFO:tasks.rados.rados.0.smithi045.stdout:3186: expect deleted 2024-04-26T00:38:06.685 INFO:tasks.rados.rados.0.smithi045.stdout:3187: rollback oid 21 current snap is 346 2024-04-26T00:38:06.685 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 21 to 345 2024-04-26T00:38:06.685 INFO:tasks.rados.rados.0.smithi045.stdout:3185: finishing write tid 1 to smithi04526579-17 2024-04-26T00:38:06.685 INFO:tasks.rados.rados.0.smithi045.stdout:3188: write oid 25 current snap is 346 2024-04-26T00:38:06.685 INFO:tasks.rados.rados.0.smithi045.stdout:3188: seq_num 1142 ranges {793594=587308,2060554=510055} 2024-04-26T00:38:06.697 INFO:tasks.rados.rados.0.smithi045.stdout:3188: writing smithi04526579-25 from 793594 to 1380902 tid 1 2024-04-26T00:38:06.708 INFO:tasks.rados.rados.0.smithi045.stdout:3188: writing smithi04526579-25 from 2060554 to 2570609 tid 2 2024-04-26T00:38:06.708 INFO:tasks.rados.rados.0.smithi045.stdout:3189: rollback oid 1 current snap is 346 2024-04-26T00:38:06.708 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 1 to 337 2024-04-26T00:38:06.708 INFO:tasks.rados.rados.0.smithi045.stdout:3187: finishing rollback tid 0 to smithi04526579-21 2024-04-26T00:38:06.708 INFO:tasks.rados.rados.0.smithi045.stdout:3187: finishing rollback tid 1 to smithi04526579-21 2024-04-26T00:38:06.708 INFO:tasks.rados.rados.0.smithi045.stdout:3187: finishing rollback tid 2 to smithi04526579-21 2024-04-26T00:38:06.708 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 2357 (ObjNum 1087 snap 324 seq_num 1087) dirty exists 2024-04-26T00:38:06.708 INFO:tasks.rados.rados.0.smithi045.stdout:3185: finishing write tid 2 to smithi04526579-17 2024-04-26T00:38:06.708 INFO:tasks.rados.rados.0.smithi045.stdout:3185: finishing write tid 3 to smithi04526579-17 2024-04-26T00:38:06.709 INFO:tasks.rados.rados.0.smithi045.stdout:3185: finishing write tid 4 to smithi04526579-17 2024-04-26T00:38:06.709 INFO:tasks.rados.rados.0.smithi045.stdout:3185: finishing write tid 5 to smithi04526579-17 2024-04-26T00:38:06.709 INFO:tasks.rados.rados.0.smithi045.stdout:3185: oid 17 updating version 0 to 2347 2024-04-26T00:38:06.709 INFO:tasks.rados.rados.0.smithi045.stdout:3185: oid 17 version 2347 is already newer than 2344 2024-04-26T00:38:06.709 INFO:tasks.rados.rados.0.smithi045.stdout:3185: oid 17 version 2347 is already newer than 2345 2024-04-26T00:38:06.709 INFO:tasks.rados.rados.0.smithi045.stdout:3185: oid 17 version 2347 is already newer than 2346 2024-04-26T00:38:06.709 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 2347 (ObjNum 1141 snap 346 seq_num 1141) dirty exists 2024-04-26T00:38:06.709 INFO:tasks.rados.rados.0.smithi045.stdout:3185: left oid 17 (ObjNum 1141 snap 346 seq_num 1141) 2024-04-26T00:38:06.709 INFO:tasks.rados.rados.0.smithi045.stdout:3188: finishing write tid 1 to smithi04526579-25 2024-04-26T00:38:06.709 INFO:tasks.rados.rados.0.smithi045.stdout:3185: done (4 left) 2024-04-26T00:38:06.709 INFO:tasks.rados.rados.0.smithi045.stdout:3186: done (3 left) 2024-04-26T00:38:06.709 INFO:tasks.rados.rados.0.smithi045.stdout:3187: done (2 left) 2024-04-26T00:38:06.709 INFO:tasks.rados.rados.0.smithi045.stdout:3190: read oid 49 snap 335 2024-04-26T00:38:06.709 INFO:tasks.rados.rados.0.smithi045.stdout:3190: expect (ObjNum 1085 snap 324 seq_num 1085) 2024-04-26T00:38:06.710 INFO:tasks.rados.rados.0.smithi045.stdout:3191: write (excl) oid 31 current snap is 346 2024-04-26T00:38:06.710 INFO:tasks.rados.rados.0.smithi045.stdout:3191: seq_num 1143 ranges {410401=662557,1719036=770066,2938716=271685} 2024-04-26T00:38:06.719 INFO:tasks.rados.rados.0.smithi045.stdout:3191: writing smithi04526579-31 from 410401 to 1072958 tid 1 2024-04-26T00:38:06.730 INFO:tasks.rados.rados.0.smithi045.stdout:3191: writing smithi04526579-31 from 1719036 to 2489102 tid 2 2024-04-26T00:38:06.736 INFO:tasks.rados.rados.0.smithi045.stdout:3191: writing smithi04526579-31 from 2938716 to 3210401 tid 3 2024-04-26T00:38:06.736 INFO:tasks.rados.rados.0.smithi045.stdout:3192: write oid 19 current snap is 346 2024-04-26T00:38:06.736 INFO:tasks.rados.rados.0.smithi045.stdout:3192: seq_num 1144 ranges {427208=737806,1777518=513724,2966139=261069} 2024-04-26T00:38:06.745 INFO:tasks.rados.rados.0.smithi045.stdout:3192: writing smithi04526579-19 from 427208 to 1165014 tid 1 2024-04-26T00:38:06.753 INFO:tasks.rados.rados.0.smithi045.stdout:3192: writing smithi04526579-19 from 1777518 to 2291242 tid 2 2024-04-26T00:38:06.759 INFO:tasks.rados.rados.0.smithi045.stdout:3192: writing smithi04526579-19 from 2966139 to 3227208 tid 3 2024-04-26T00:38:06.759 INFO:tasks.rados.rados.0.smithi045.stdout:3193: snap_create 2024-04-26T00:38:06.759 INFO:tasks.rados.rados.0.smithi045.stdout:3188: finishing write tid 2 to smithi04526579-25 2024-04-26T00:38:06.759 INFO:tasks.rados.rados.0.smithi045.stdout:3188: finishing write tid 3 to smithi04526579-25 2024-04-26T00:38:06.759 INFO:tasks.rados.rados.0.smithi045.stdout:3188: finishing write tid 4 to smithi04526579-25 2024-04-26T00:38:06.759 INFO:tasks.rados.rados.0.smithi045.stdout:3188: oid 25 updating version 0 to 2478 2024-04-26T00:38:06.759 INFO:tasks.rados.rados.0.smithi045.stdout:3188: oid 25 version 2478 is already newer than 2477 2024-04-26T00:38:06.760 INFO:tasks.rados.rados.0.smithi045.stdout:3188: oid 25 updating version 2478 to 2479 2024-04-26T00:38:06.760 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 2479 (ObjNum 1142 snap 346 seq_num 1142) dirty exists 2024-04-26T00:38:06.760 INFO:tasks.rados.rados.0.smithi045.stdout:3188: left oid 25 (ObjNum 1142 snap 346 seq_num 1142) 2024-04-26T00:38:06.760 INFO:tasks.rados.rados.0.smithi045.stdout:3190: expect (ObjNum 1085 snap 324 seq_num 1085) 2024-04-26T00:38:06.995 INFO:tasks.rados.rados.0.smithi045.stdout:3189: finishing rollback tid 0 to smithi04526579-1 2024-04-26T00:38:06.995 INFO:tasks.rados.rados.0.smithi045.stdout:3189: finishing rollback tid 1 to smithi04526579-1 2024-04-26T00:38:06.995 INFO:tasks.rados.rados.0.smithi045.stdout:3189: finishing rollback tid 2 to smithi04526579-1 2024-04-26T00:38:06.996 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 2447 (ObjNum 1079 snap 322 seq_num 1079) dirty exists 2024-04-26T00:38:06.996 INFO:tasks.rados.rados.0.smithi045.stdout:3191: finishing write tid 1 to smithi04526579-31 2024-04-26T00:38:06.996 INFO:tasks.rados.rados.0.smithi045.stdout:3191: finishing write tid 2 to smithi04526579-31 2024-04-26T00:38:06.996 INFO:tasks.rados.rados.0.smithi045.stdout:3191: finishing write tid 3 to smithi04526579-31 2024-04-26T00:38:06.996 INFO:tasks.rados.rados.0.smithi045.stdout:3191: finishing write tid 4 to smithi04526579-31 2024-04-26T00:38:06.996 INFO:tasks.rados.rados.0.smithi045.stdout:3191: finishing write tid 5 to smithi04526579-31 2024-04-26T00:38:06.996 INFO:tasks.rados.rados.0.smithi045.stdout:3191: oid 31 updating version 0 to 2055 2024-04-26T00:38:06.996 INFO:tasks.rados.rados.0.smithi045.stdout:3191: oid 31 updating version 2055 to 2057 2024-04-26T00:38:06.996 INFO:tasks.rados.rados.0.smithi045.stdout:3191: oid 31 version 2057 is already newer than 2056 2024-04-26T00:38:06.996 INFO:tasks.rados.rados.0.smithi045.stdout:3191: oid 31 version 2057 is already newer than 2054 2024-04-26T00:38:06.996 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 2057 (ObjNum 1143 snap 346 seq_num 1143) dirty exists 2024-04-26T00:38:06.996 INFO:tasks.rados.rados.0.smithi045.stdout:3191: left oid 31 (ObjNum 1143 snap 346 seq_num 1143) 2024-04-26T00:38:06.996 INFO:tasks.rados.rados.0.smithi045.stdout:3192: finishing write tid 1 to smithi04526579-19 2024-04-26T00:38:06.996 INFO:tasks.rados.rados.0.smithi045.stdout:3192: finishing write tid 2 to smithi04526579-19 2024-04-26T00:38:06.997 INFO:tasks.rados.rados.0.smithi045.stdout:3192: finishing write tid 3 to smithi04526579-19 2024-04-26T00:38:06.997 INFO:tasks.rados.rados.0.smithi045.stdout:3192: finishing write tid 4 to smithi04526579-19 2024-04-26T00:38:06.997 INFO:tasks.rados.rados.0.smithi045.stdout:3192: finishing write tid 5 to smithi04526579-19 2024-04-26T00:38:06.997 INFO:tasks.rados.rados.0.smithi045.stdout:3192: oid 19 updating version 0 to 2437 2024-04-26T00:38:06.997 INFO:tasks.rados.rados.0.smithi045.stdout:3192: oid 19 updating version 2437 to 2438 2024-04-26T00:38:06.997 INFO:tasks.rados.rados.0.smithi045.stdout:3192: oid 19 updating version 2438 to 2439 2024-04-26T00:38:06.997 INFO:tasks.rados.rados.0.smithi045.stdout:3192: oid 19 version 2439 is already newer than 2436 2024-04-26T00:38:06.997 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 2439 (ObjNum 1144 snap 346 seq_num 1144) dirty exists 2024-04-26T00:38:06.997 INFO:tasks.rados.rados.0.smithi045.stdout:3192: left oid 19 (ObjNum 1144 snap 346 seq_num 1144) 2024-04-26T00:38:07.671 INFO:tasks.rados.rados.0.smithi045.stdout:3188: done (5 left) 2024-04-26T00:38:07.671 INFO:tasks.rados.rados.0.smithi045.stdout:3189: done (4 left) 2024-04-26T00:38:07.672 INFO:tasks.rados.rados.0.smithi045.stdout:3190: done (3 left) 2024-04-26T00:38:07.672 INFO:tasks.rados.rados.0.smithi045.stdout:3191: done (2 left) 2024-04-26T00:38:07.672 INFO:tasks.rados.rados.0.smithi045.stdout:3192: done (1 left) 2024-04-26T00:38:07.672 INFO:tasks.rados.rados.0.smithi045.stdout:3193: done (0 left) 2024-04-26T00:38:07.672 INFO:tasks.rados.rados.0.smithi045.stdout:3194: snap_create 2024-04-26T00:38:08.587 INFO:tasks.rados.rados.0.smithi045.stdout:3194: done (0 left) 2024-04-26T00:38:08.587 INFO:tasks.rados.rados.0.smithi045.stdout:3195: read oid 26 snap 330 2024-04-26T00:38:08.587 INFO:tasks.rados.rados.0.smithi045.stdout:3195: expect deleted 2024-04-26T00:38:08.587 INFO:tasks.rados.rados.0.smithi045.stdout:3196: write (excl) oid 33 current snap is 348 2024-04-26T00:38:08.587 INFO:tasks.rados.rados.0.smithi045.stdout:3196: seq_num 1145 ranges {444015=413055,1436000=773735,2709915=534100} 2024-04-26T00:38:08.592 INFO:tasks.rados.rados.0.smithi045.stdout:3196: writing smithi04526579-33 from 444015 to 857070 tid 1 2024-04-26T00:38:08.601 INFO:tasks.rados.rados.0.smithi045.stdout:3196: writing smithi04526579-33 from 1436000 to 2209735 tid 2 2024-04-26T00:38:08.607 INFO:tasks.rados.rados.0.smithi045.stdout:3196: writing smithi04526579-33 from 2709915 to 3244015 tid 3 2024-04-26T00:38:08.607 INFO:tasks.rados.rados.0.smithi045.stdout:3197: delete oid 11 current snap is 348 2024-04-26T00:38:08.617 INFO:tasks.rados.rados.0.smithi045.stdout:3195: done (2 left) 2024-04-26T00:38:08.617 INFO:tasks.rados.rados.0.smithi045.stdout:3197: done (1 left) 2024-04-26T00:38:08.617 INFO:tasks.rados.rados.0.smithi045.stdout:3198: delete oid 49 current snap is 348 2024-04-26T00:38:08.622 INFO:tasks.rados.rados.0.smithi045.stdout:3196: finishing write tid 1 to smithi04526579-33 2024-04-26T00:38:08.623 INFO:tasks.rados.rados.0.smithi045.stdout:3196: finishing write tid 2 to smithi04526579-33 2024-04-26T00:38:08.625 INFO:tasks.rados.rados.0.smithi045.stdout:3196: finishing write tid 3 to smithi04526579-33 2024-04-26T00:38:08.628 INFO:tasks.rados.rados.0.smithi045.stdout:3196: finishing write tid 4 to smithi04526579-33 2024-04-26T00:38:08.628 INFO:tasks.rados.rados.0.smithi045.stdout:3196: finishing write tid 5 to smithi04526579-33 2024-04-26T00:38:08.628 INFO:tasks.rados.rados.0.smithi045.stdout:3196: oid 33 updating version 0 to 2480 2024-04-26T00:38:08.628 INFO:tasks.rados.rados.0.smithi045.stdout:3196: oid 33 updating version 2480 to 2483 2024-04-26T00:38:08.628 INFO:tasks.rados.rados.0.smithi045.stdout:3196: oid 33 version 2483 is already newer than 2482 2024-04-26T00:38:08.628 INFO:tasks.rados.rados.0.smithi045.stdout:3196: oid 33 version 2483 is already newer than 2481 2024-04-26T00:38:08.628 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 2483 (ObjNum 1145 snap 348 seq_num 1145) dirty exists 2024-04-26T00:38:08.628 INFO:tasks.rados.rados.0.smithi045.stdout:3196: left oid 33 (ObjNum 1145 snap 348 seq_num 1145) 2024-04-26T00:38:08.637 INFO:tasks.rados.rados.0.smithi045.stdout:3196: done (1 left) 2024-04-26T00:38:08.637 INFO:tasks.rados.rados.0.smithi045.stdout:3198: done (0 left) 2024-04-26T00:38:08.637 INFO:tasks.rados.rados.0.smithi045.stdout:3199: snap_remove snap 329 2024-04-26T00:38:09.641 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:38:09.685 INFO:tasks.rados.rados.0.smithi045.stdout:3199: done (0 left) 2024-04-26T00:38:09.685 INFO:tasks.rados.rados.0.smithi045.stdout:3200: read oid 9 snap -1 2024-04-26T00:38:09.685 INFO:tasks.rados.rados.0.smithi045.stdout:3200: expect (ObjNum 1099 snap 326 seq_num 1099) 2024-04-26T00:38:09.685 INFO:tasks.rados.rados.0.smithi045.stdout:3201: delete oid 19 current snap is 348 2024-04-26T00:38:09.703 INFO:tasks.rados.rados.0.smithi045.stdout:3200: expect (ObjNum 1099 snap 326 seq_num 1099) 2024-04-26T00:38:09.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.783+0000 7f769fe01640 10 monclient: get_monmap_and_config 2024-04-26T00:38:09.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.783+0000 7f769fe01640 10 monclient: build_initial_monmap 2024-04-26T00:38:09.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.783+0000 7f769fe01640 10 monclient: monmap: 2024-04-26T00:38:09.781 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:38:09.782 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:09.782 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:38:09.785433+0000 2024-04-26T00:38:09.782 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:38:09.785433+0000 2024-04-26T00:38:09.782 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:38:09.782 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:09.782 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:38:09.782 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:38:09.782 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:38:09.782 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:38:09.782 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:38:09.782 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:09.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.783+0000 7f769fe01640 1 Processor -- start 2024-04-26T00:38:09.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.783+0000 7f769fe01640 1 -- start start 2024-04-26T00:38:09.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.783+0000 7f769fe01640 10 monclient: init 2024-04-26T00:38:09.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.783+0000 7f769fe01640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:09.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.783+0000 7f769fe01640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-04-26T00:38:09.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.783+0000 7f769fe01640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f76980a0260 addr v1:172.21.15.45:6789/0 2024-04-26T00:38:09.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.783+0000 7f769fe01640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7698137e40 addr v1:172.21.15.37:6789/0 2024-04-26T00:38:09.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.783+0000 7f769fe01640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f769813b950 0x7f769813dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:09.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.783+0000 7f769fe01640 10 monclient(hunting): picked mon.noname-b con 0x7f769813b950 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:38:09.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.783+0000 7f769fe01640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76980a1130 con 0x7f7698137e40 2024-04-26T00:38:09.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.783+0000 7f769fe01640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f769806cff0 con 0x7f76980a0260 2024-04-26T00:38:09.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.783+0000 7f769fe01640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:09.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.783+0000 7f769fe01640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f76980a0d50 con 0x7f769813b950 2024-04-26T00:38:09.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.783+0000 7f769fe01640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:09.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.783+0000 7f769fe01640 10 monclient(hunting): authenticate will time out at 2071.091123s 2024-04-26T00:38:09.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.783+0000 7f769e39e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f769813b950 0x7f769813dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:09.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.783+0000 7f769d39c640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f7698137e40 0x7f769813a230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:51756/0 (socket says 172.21.15.37:51756) 2024-04-26T00:38:09.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.783+0000 7f769d39c640 1 -- 172.21.15.37:0/1780184497 learned_addr learned my addr 172.21.15.37:0/1780184497 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:38:09.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.783+0000 7f769e39e640 10 monclient(hunting): get_auth_request con 0x7f769813b950 auth_method 0 2024-04-26T00:38:09.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.783+0000 7f769e39e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:09.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.783+0000 7f769e39e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:09.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.783+0000 7f769e39e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:09.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.783+0000 7f769e39e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:09.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.783+0000 7f769e39e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:09.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.783+0000 7f769e39e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:09.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.783+0000 7f769e39e640 10 monclient(hunting): handle_auth_done global_id 204272 payload 274 2024-04-26T00:38:09.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.783+0000 7f769e39e640 1 -- 172.21.15.37:0/1780184497 >> v1:172.21.15.45:6789/0 conn(0x7f76980a0260 legacy=0x7f76980a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:38:09.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.783+0000 7f769e39e640 1 -- 172.21.15.37:0/1780184497 >> v1:172.21.15.37:6789/0 conn(0x7f7698137e40 legacy=0x7f769813a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:38:09.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.783+0000 7f769e39e640 10 monclient: _finish_hunting 0 2024-04-26T00:38:09.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.783+0000 7f769e39e640 1 monclient: found mon.noname-b 2024-04-26T00:38:09.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.783+0000 7f769e39e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:09.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.783+0000 7f769e39e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:38:09.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.783+0000 7f769e39e640 1 -- 172.21.15.37:0/1780184497 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f769809bc50 con 0x7f769813b950 2024-04-26T00:38:09.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.787+0000 7f769e39e640 1 -- 172.21.15.37:0/1780184497 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f769813b950 msgr2=0x7f769813dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:09.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.787+0000 7f769e39e640 1 --2- 172.21.15.37:0/1780184497 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f769813b950 0x7f769813dd40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7690009dc0 tx=0x7f769001c5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80bdf7166c5bfdf5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:09.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769cb9b640 1 -- 172.21.15.37:0/1780184497 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76900aa700 con 0x7f769813b950 2024-04-26T00:38:09.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769cb9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:09.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769cb9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:38:09.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769cb9b640 10 monclient: dump: 2024-04-26T00:38:09.792 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:09.792 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:09.792 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:09.792 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:09.792 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:09.792 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:09.792 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:09.792 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:09.792 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:09.792 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:09.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769cb9b640 10 monclient: _finish_auth 0 2024-04-26T00:38:09.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769cb9b640 10 monclient: _check_auth_tickets 2024-04-26T00:38:09.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769cb9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:09.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769cb9b640 1 -- 172.21.15.37:0/1780184497 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76900af070 con 0x7f769813b950 2024-04-26T00:38:09.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769cb9b640 10 monclient: handle_config config(0 keys) 2024-04-26T00:38:09.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769cb9b640 1 -- 172.21.15.37:0/1780184497 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76900aaac0 con 0x7f769813b950 2024-04-26T00:38:09.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769fe01640 5 monclient: authenticate success, global_id 204272 2024-04-26T00:38:09.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769fe01640 10 monclient: get_monmap_and_config success 2024-04-26T00:38:09.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769fe01640 4 set_mon_vals no callback set 2024-04-26T00:38:09.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769cb9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:09.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769cb9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:38:09.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769cb9b640 10 monclient: dump: 2024-04-26T00:38:09.793 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:09.793 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:09.793 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:09.793 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:09.794 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:09.794 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:09.794 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:09.794 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:09.794 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:09.794 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:09.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769fe01640 10 monclient: shutdown 2024-04-26T00:38:09.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769fe01640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:38:09.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769fe01640 1 -- 172.21.15.37:0/1780184497 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f769813b950 msgr2=0x7f769813dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:09.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769fe01640 1 --2- 172.21.15.37:0/1780184497 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f769813b950 0x7f769813dd40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7690009dc0 tx=0x7f769001c5f0 comp rx=0 tx=0).stop 2024-04-26T00:38:09.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769fe01640 1 -- 172.21.15.37:0/1780184497 shutdown_connections 2024-04-26T00:38:09.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769fe01640 1 --2- 172.21.15.37:0/1780184497 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f769813b950 0x7f769813dd40 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:09.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769fe01640 1 -- 172.21.15.37:0/1780184497 >> 172.21.15.37:0/1780184497 conn(0x7f76980a1ad0 msgr2=0x7f76980a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:38:09.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769fe01640 1 -- 172.21.15.37:0/1780184497 shutdown_connections 2024-04-26T00:38:09.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769fe01640 1 -- 172.21.15.37:0/1780184497 wait complete. 2024-04-26T00:38:09.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769fe01640 10 monclient: build_initial_monmap 2024-04-26T00:38:09.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769fe01640 10 monclient: monmap: 2024-04-26T00:38:09.795 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:38:09.795 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:38:09.795 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:38:09.795 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:38:09.795 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:38:09.795 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:38:09.795 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:38:09.795 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:38:09.796 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:38:09.796 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:09.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769fe01640 1 Processor -- start 2024-04-26T00:38:09.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769fe01640 1 -- start start 2024-04-26T00:38:09.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769fe01640 10 monclient: init 2024-04-26T00:38:09.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769fe01640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:09.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769fe01640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:38:09.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769fe01640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f76980a0260 0x7f7698134480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:09.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769fe01640 10 monclient(hunting): picked mon.noname-b con 0x7f76980a0260 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:38:09.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769fe01640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7698137e40 0x7f76981349c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:09.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769fe01640 10 monclient(hunting): picked mon.noname-a con 0x7f7698137e40 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:38:09.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769fe01640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f769813b950 0x7f7698134f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:09.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769fe01640 10 monclient(hunting): picked mon.noname-c con 0x7f769813b950 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:38:09.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769fe01640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:09.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769fe01640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f769809f0f0 con 0x7f7698137e40 2024-04-26T00:38:09.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769fe01640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:09.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769fe01640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f769809f4c0 con 0x7f76980a0260 2024-04-26T00:38:09.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769fe01640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:09.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769fe01640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f76980a00a0 con 0x7f769813b950 2024-04-26T00:38:09.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769fe01640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:09.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769fe01640 10 monclient(hunting): authenticate will time out at 2071.098431s 2024-04-26T00:38:09.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769e39e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f769813b950 0x7f7698134f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:09.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769d39c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7698137e40 0x7f76981349c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:09.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769e39e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f769813b950 0x7f7698134f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:52928/0 (socket says 172.21.15.37:52928) 2024-04-26T00:38:09.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769e39e640 1 -- 172.21.15.37:0/3482875442 learned_addr learned my addr 172.21.15.37:0/3482875442 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:38:09.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769e39e640 10 monclient(hunting): get_auth_request con 0x7f769813b950 auth_method 0 2024-04-26T00:38:09.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769d39c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7698137e40 0x7f76981349c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44574/0 (socket says 172.21.15.37:44574) 2024-04-26T00:38:09.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769e39e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:09.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769e39e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:09.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769e39e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:09.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769d39c640 10 monclient(hunting): get_auth_request con 0x7f7698137e40 auth_method 0 2024-04-26T00:38:09.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769d39c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:09.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769d39c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:09.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769d39c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:09.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769d39c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:09.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769d39c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:09.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769d39c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:09.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769e39e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:09.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769e39e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:09.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769db9d640 1 --2- 172.21.15.37:0/3482875442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f76980a0260 0x7f7698134480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:09.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769e39e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:09.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769db9d640 10 monclient(hunting): get_auth_request con 0x7f76980a0260 auth_method 0 2024-04-26T00:38:09.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769db9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:09.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769db9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:09.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769db9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:09.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769d39c640 10 monclient(hunting): handle_auth_done global_id 144885 payload 824 2024-04-26T00:38:09.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769db9d640 1 -- 172.21.15.37:0/3482875442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f76980a0260 msgr2=0x7f7698134480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:38:09.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769d39c640 1 -- 172.21.15.37:0/3482875442 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f769813b950 msgr2=0x7f7698134f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:09.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769db9d640 1 -- 172.21.15.37:0/3482875442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f76980a0260 msgr2=0x7f7698134480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:38:09.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769d39c640 1 --2- 172.21.15.37:0/3482875442 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f769813b950 0x7f7698134f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:09.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769db9d640 1 --2- 172.21.15.37:0/3482875442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f76980a0260 0x7f7698134480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:38:09.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769d39c640 1 -- 172.21.15.37:0/3482875442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f76980a0260 msgr2=0x7f7698134480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:09.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769db9d640 1 --2- 172.21.15.37:0/3482875442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f76980a0260 0x7f7698134480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:38:09.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769d39c640 1 --2- 172.21.15.37:0/3482875442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f76980a0260 0x7f7698134480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:09.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769d39c640 10 monclient: _finish_hunting 0 2024-04-26T00:38:09.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769d39c640 1 monclient: found mon.noname-a 2024-04-26T00:38:09.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769d39c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:09.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769d39c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:38:09.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769d39c640 1 -- 172.21.15.37:0/3482875442 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7698120620 con 0x7f7698137e40 2024-04-26T00:38:09.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.791+0000 7f769e39e640 1 --2- 172.21.15.37:0/3482875442 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f769813b950 0x7f7698134f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:38:09.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.795+0000 7f769d39c640 1 -- 172.21.15.37:0/3482875442 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7698137e40 msgr2=0x7f76981349c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:09.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.795+0000 7f769d39c640 1 --2- 172.21.15.37:0/3482875442 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7698137e40 0x7f76981349c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f768800c410 tx=0x7f768800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:09.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.795+0000 7f768e7fc640 1 -- 172.21.15.37:0/3482875442 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7688017070 con 0x7f7698137e40 2024-04-26T00:38:09.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.795+0000 7f768e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:09.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.795+0000 7f768e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:38:09.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.795+0000 7f768e7fc640 10 monclient: dump: 2024-04-26T00:38:09.802 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:09.803 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:09.803 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:09.803 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:09.803 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:09.803 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:09.803 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:09.803 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:09.803 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:09.803 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:09.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.795+0000 7f768e7fc640 10 monclient: _finish_auth 0 2024-04-26T00:38:09.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.795+0000 7f768e7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:38:09.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.795+0000 7f768e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:09.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.795+0000 7f768e7fc640 1 -- 172.21.15.37:0/3482875442 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76880070d0 con 0x7f7698137e40 2024-04-26T00:38:09.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.795+0000 7f768e7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:38:09.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.795+0000 7f768e7fc640 1 -- 172.21.15.37:0/3482875442 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76880074c0 con 0x7f7698137e40 2024-04-26T00:38:09.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.795+0000 7f769fe01640 5 monclient: authenticate success, global_id 144885 2024-04-26T00:38:09.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.795+0000 7f768e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:09.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.795+0000 7f768e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:38:09.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.795+0000 7f768e7fc640 10 monclient: dump: 2024-04-26T00:38:09.804 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:09.804 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:09.804 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:09.804 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:09.804 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:09.804 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:09.804 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:09.805 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:09.805 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:09.805 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:09.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.795+0000 7f768ffff640 4 set_mon_vals no callback set 2024-04-26T00:38:09.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.795+0000 7f769fe01640 10 monclient: _renew_subs 2024-04-26T00:38:09.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.795+0000 7f769fe01640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:38:09.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.795+0000 7f769fe01640 1 -- 172.21.15.37:0/3482875442 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f769811fd80 con 0x7f7698137e40 2024-04-26T00:38:09.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.795+0000 7f769fe01640 10 monclient: _renew_subs 2024-04-26T00:38:09.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.795+0000 7f769fe01640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:38:09.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.795+0000 7f769fe01640 1 -- 172.21.15.37:0/3482875442 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f769811fa30 con 0x7f7698137e40 2024-04-26T00:38:09.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.795+0000 7f769fe01640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:09.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.795+0000 7f769fe01640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:09.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.795+0000 7f769fe01640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:38:09.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.795+0000 7f769fe01640 1 -- 172.21.15.37:0/3482875442 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7698066f40 con 0x7f7698137e40 2024-04-26T00:38:09.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.795+0000 7f768e7fc640 1 -- 172.21.15.37:0/3482875442 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f768801a070 con 0x7f7698137e40 2024-04-26T00:38:09.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.795+0000 7f768e7fc640 1 --2- 172.21.15.37:0/3482875442 >> v2:172.21.15.45:6812/2379918623 conn(0x7f767c035e80 0x7f767c038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:09.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.795+0000 7f768e7fc640 1 -- 172.21.15.37:0/3482875442 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(715..715 src has 370..715) ==== 3927+0+0 (secure 0 0 0) 0x7f7688047f00 con 0x7f7698137e40 2024-04-26T00:38:09.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.795+0000 7f769db9d640 1 --2- 172.21.15.37:0/3482875442 >> v2:172.21.15.45:6812/2379918623 conn(0x7f767c035e80 0x7f767c038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:09.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.795+0000 7f769db9d640 10 monclient: get_auth_request con 0x7f767c035e80 auth_method 0 2024-04-26T00:38:09.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.799+0000 7f769db9d640 1 --2- 172.21.15.37:0/3482875442 >> v2:172.21.15.45:6812/2379918623 conn(0x7f767c035e80 0x7f767c038340 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f7694007fd0 tx=0x7f7694009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:09.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.799+0000 7f768e7fc640 1 -- 172.21.15.37:0/3482875442 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7688010300 con 0x7f7698137e40 2024-04-26T00:38:09.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.799+0000 7f768e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:09.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:09.799+0000 7f768e7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:38:09.809 INFO:tasks.rados.rados.0.smithi045.stdout:3200: done (1 left) 2024-04-26T00:38:09.809 INFO:tasks.rados.rados.0.smithi045.stdout:3201: done (0 left) 2024-04-26T00:38:09.809 INFO:tasks.rados.rados.0.smithi045.stdout:3202: read oid 28 snap 337 2024-04-26T00:38:09.809 INFO:tasks.rados.rados.0.smithi045.stdout:3202: expect (ObjNum 1041 snap 315 seq_num 1041) 2024-04-26T00:38:09.809 INFO:tasks.rados.rados.0.smithi045.stdout:3203: read oid 14 snap -1 2024-04-26T00:38:09.809 INFO:tasks.rados.rados.0.smithi045.stdout:3203: expect (ObjNum 1087 snap 324 seq_num 1087) 2024-04-26T00:38:09.809 INFO:tasks.rados.rados.0.smithi045.stdout:3204: read oid 27 snap -1 2024-04-26T00:38:09.809 INFO:tasks.rados.rados.0.smithi045.stdout:3204: expect (ObjNum 1051 snap 317 seq_num 1051) 2024-04-26T00:38:09.810 INFO:tasks.rados.rados.0.smithi045.stdout:3205: read oid 18 snap 346 2024-04-26T00:38:09.810 INFO:tasks.rados.rados.0.smithi045.stdout:3205: expect (ObjNum 1096 snap 325 seq_num 1096) 2024-04-26T00:38:09.810 INFO:tasks.rados.rados.0.smithi045.stdout:3206: read oid 17 snap -1 2024-04-26T00:38:09.810 INFO:tasks.rados.rados.0.smithi045.stdout:3206: expect (ObjNum 1141 snap 346 seq_num 1141) 2024-04-26T00:38:09.810 INFO:tasks.rados.rados.0.smithi045.stdout:3207: delete oid 43 current snap is 348 2024-04-26T00:38:09.810 INFO:tasks.rados.rados.0.smithi045.stdout:3202: expect (ObjNum 1041 snap 315 seq_num 1041) 2024-04-26T00:38:09.925 INFO:tasks.rados.rados.0.smithi045.stdout:3203: expect (ObjNum 1087 snap 324 seq_num 1087) 2024-04-26T00:38:10.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.027+0000 7f769fe01640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:38:10.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.027+0000 7f769fe01640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:38:10.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.027+0000 7f769fe01640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:38:10.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.027+0000 7f769fe01640 1 -- 172.21.15.37:0/3482875442 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f769806b9a0 con 0x7f7698137e40 2024-04-26T00:38:10.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.027+0000 7f768e7fc640 1 -- 172.21.15.37:0/3482875442 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f7688046c50 con 0x7f7698137e40 2024-04-26T00:38:10.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.027+0000 7f768e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:38:10.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.027+0000 7f768e7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:38:10.026 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":138,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:38:10.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.031+0000 7f769fe01640 1 -- 172.21.15.37:0/3482875442 >> v2:172.21.15.45:6812/2379918623 conn(0x7f767c035e80 msgr2=0x7f767c038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:10.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.031+0000 7f769fe01640 1 --2- 172.21.15.37:0/3482875442 >> v2:172.21.15.45:6812/2379918623 conn(0x7f767c035e80 0x7f767c038340 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f7694007fd0 tx=0x7f7694009040 comp rx=0 tx=0).stop 2024-04-26T00:38:10.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.031+0000 7f769fe01640 10 monclient: shutdown 2024-04-26T00:38:10.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.031+0000 7f769fe01640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:38:10.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.031+0000 7f769fe01640 1 -- 172.21.15.37:0/3482875442 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7698137e40 msgr2=0x7f76981349c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:10.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.031+0000 7f769fe01640 1 --2- 172.21.15.37:0/3482875442 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7698137e40 0x7f76981349c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f768800c410 tx=0x7f768800c8d0 comp rx=0 tx=0).stop 2024-04-26T00:38:10.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.031+0000 7f769fe01640 1 -- 172.21.15.37:0/3482875442 shutdown_connections 2024-04-26T00:38:10.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.031+0000 7f769fe01640 1 --2- 172.21.15.37:0/3482875442 >> v2:172.21.15.45:6812/2379918623 conn(0x7f767c035e80 0x7f767c038340 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:10.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.031+0000 7f769fe01640 1 --2- 172.21.15.37:0/3482875442 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f769813b950 0x7f7698134f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:10.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.031+0000 7f769fe01640 1 --2- 172.21.15.37:0/3482875442 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7698137e40 0x7f76981349c0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:10.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.031+0000 7f769fe01640 1 --2- 172.21.15.37:0/3482875442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f76980a0260 0x7f7698134480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:10.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.031+0000 7f769fe01640 1 -- 172.21.15.37:0/3482875442 >> 172.21.15.37:0/3482875442 conn(0x7f76980a1ad0 msgr2=0x7f7698136450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:38:10.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.031+0000 7f769fe01640 1 -- 172.21.15.37:0/3482875442 shutdown_connections 2024-04-26T00:38:10.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.031+0000 7f769fe01640 1 -- 172.21.15.37:0/3482875442 wait complete. 2024-04-26T00:38:10.045 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:38:10.045 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:38:10.045 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:38:10.058 INFO:tasks.rados.rados.0.smithi045.stdout:3206: expect (ObjNum 1141 snap 346 seq_num 1141) 2024-04-26T00:38:10.161 INFO:tasks.rados.rados.0.smithi045.stdout:3205: expect (ObjNum 1096 snap 325 seq_num 1096) 2024-04-26T00:38:10.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7afcb9640 10 monclient: get_monmap_and_config 2024-04-26T00:38:10.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7afcb9640 10 monclient: build_initial_monmap 2024-04-26T00:38:10.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7afcb9640 10 monclient: monmap: 2024-04-26T00:38:10.172 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:38:10.173 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:10.173 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:38:10.176075+0000 2024-04-26T00:38:10.173 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:38:10.176075+0000 2024-04-26T00:38:10.173 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:38:10.173 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:10.173 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:38:10.173 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:38:10.173 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:38:10.173 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:38:10.173 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:38:10.173 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:10.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7afcb9640 1 Processor -- start 2024-04-26T00:38:10.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7afcb9640 1 -- start start 2024-04-26T00:38:10.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7afcb9640 10 monclient: init 2024-04-26T00:38:10.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7afcb9640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:10.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7afcb9640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-04-26T00:38:10.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7afcb9640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe7a80a3d90 0x7fe7a809bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:10.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7afcb9640 10 monclient(hunting): picked mon.noname-b con 0x7fe7a80a3d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:38:10.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7afcb9640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe7a80a4170 0x7fe7a80a45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:10.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7afcb9640 10 monclient(hunting): picked mon.noname-c con 0x7fe7a80a4170 addr v2:172.21.15.45:3300/0 2024-04-26T00:38:10.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7afcb9640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fe7a8133b20 0x7fe7a8133f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:10.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7afcb9640 10 monclient(hunting): picked mon.noname-a con 0x7fe7a8133b20 addr v2:172.21.15.37:3300/0 2024-04-26T00:38:10.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7afcb9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:10.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7afcb9640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fe7a80a5150 con 0x7fe7a8133b20 2024-04-26T00:38:10.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7afcb9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:10.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7afcb9640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fe7a809b280 con 0x7fe7a80a4170 2024-04-26T00:38:10.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7afcb9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:10.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7afcb9640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe7a809b560 con 0x7fe7a80a3d90 2024-04-26T00:38:10.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7afcb9640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:10.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7afcb9640 10 monclient(hunting): authenticate will time out at 2071.481683s 2024-04-26T00:38:10.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7ada55640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe7a80a3d90 0x7fe7a809bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:10.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7ada55640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe7a80a3d90 0x7fe7a809bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:52932/0 (socket says 172.21.15.37:52932) 2024-04-26T00:38:10.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7ada55640 1 -- 172.21.15.37:0/3630195078 learned_addr learned my addr 172.21.15.37:0/3630195078 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:38:10.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7ada55640 10 monclient(hunting): get_auth_request con 0x7fe7a80a3d90 auth_method 0 2024-04-26T00:38:10.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7ada55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:10.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7ada55640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:10.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7ada55640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:10.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7ae256640 1 --2- 172.21.15.37:0/3630195078 >> v2:172.21.15.37:3300/0 conn(0x7fe7a8133b20 0x7fe7a8133f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:10.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7ae256640 10 monclient(hunting): get_auth_request con 0x7fe7a8133b20 auth_method 0 2024-04-26T00:38:10.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7ae256640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:10.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7ae256640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:10.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7ae256640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:10.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7ada55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:10.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7ada55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:10.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7ad254640 1 --2- 172.21.15.37:0/3630195078 >> v2:172.21.15.45:3300/0 conn(0x7fe7a80a4170 0x7fe7a80a45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:10.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7ada55640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:10.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7ae256640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:10.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7ae256640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:10.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7ae256640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:10.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7ad254640 10 monclient(hunting): get_auth_request con 0x7fe7a80a4170 auth_method 0 2024-04-26T00:38:10.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7ad254640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:10.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7ad254640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:10.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7ad254640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:10.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7ada55640 10 monclient(hunting): handle_auth_done global_id 204278 payload 274 2024-04-26T00:38:10.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7ada55640 1 -- 172.21.15.37:0/3630195078 >> v2:172.21.15.45:3300/0 conn(0x7fe7a80a4170 msgr2=0x7fe7a80a45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:38:10.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7ada55640 1 --2- 172.21.15.37:0/3630195078 >> v2:172.21.15.45:3300/0 conn(0x7fe7a80a4170 0x7fe7a80a45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:10.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7ada55640 1 -- 172.21.15.37:0/3630195078 >> v2:172.21.15.37:3300/0 conn(0x7fe7a8133b20 msgr2=0x7fe7a8133f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:38:10.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7ada55640 1 --2- 172.21.15.37:0/3630195078 >> v2:172.21.15.37:3300/0 conn(0x7fe7a8133b20 0x7fe7a8133f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:10.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7ada55640 10 monclient: _finish_hunting 0 2024-04-26T00:38:10.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7ada55640 1 monclient: found mon.noname-b 2024-04-26T00:38:10.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7ada55640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:10.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7ada55640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:38:10.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7ada55640 1 -- 172.21.15.37:0/3630195078 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7a80a4af0 con 0x7fe7a80a3d90 2024-04-26T00:38:10.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.175+0000 7fe7ae256640 1 --2- 172.21.15.37:0/3630195078 >> v2:172.21.15.37:3300/0 conn(0x7fe7a8133b20 0x7fe7a8133f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:38:10.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.179+0000 7fe7ada55640 1 -- 172.21.15.37:0/3630195078 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe7a80a3d90 msgr2=0x7fe7a809bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:10.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.179+0000 7fe7ada55640 1 --2- 172.21.15.37:0/3630195078 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe7a80a3d90 0x7fe7a809bf40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe7a80a6d60 tx=0x7fe7a001c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9507ce8c1332839 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:10.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.179+0000 7fe7aca53640 1 -- 172.21.15.37:0/3630195078 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a00b0070 con 0x7fe7a80a3d90 2024-04-26T00:38:10.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.179+0000 7fe7aca53640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:10.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.179+0000 7fe7aca53640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:38:10.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.179+0000 7fe7aca53640 10 monclient: dump: 2024-04-26T00:38:10.184 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:10.184 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:10.184 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:10.184 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:10.184 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:10.184 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:10.184 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:10.185 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:10.185 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:10.185 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:10.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.179+0000 7fe7aca53640 10 monclient: _finish_auth 0 2024-04-26T00:38:10.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.179+0000 7fe7aca53640 10 monclient: _check_auth_tickets 2024-04-26T00:38:10.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.179+0000 7fe7aca53640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:10.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.179+0000 7fe7aca53640 1 -- 172.21.15.37:0/3630195078 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7a00ab600 con 0x7fe7a80a3d90 2024-04-26T00:38:10.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.179+0000 7fe7aca53640 10 monclient: handle_config config(0 keys) 2024-04-26T00:38:10.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.179+0000 7fe7aca53640 1 -- 172.21.15.37:0/3630195078 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a00ab9d0 con 0x7fe7a80a3d90 2024-04-26T00:38:10.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.179+0000 7fe7afcb9640 5 monclient: authenticate success, global_id 204278 2024-04-26T00:38:10.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.179+0000 7fe7afcb9640 10 monclient: get_monmap_and_config success 2024-04-26T00:38:10.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.179+0000 7fe7afcb9640 4 set_mon_vals no callback set 2024-04-26T00:38:10.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.179+0000 7fe7aca53640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:10.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.179+0000 7fe7aca53640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:38:10.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.179+0000 7fe7aca53640 10 monclient: dump: 2024-04-26T00:38:10.186 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:10.186 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:10.186 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:10.186 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:10.186 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:10.186 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:10.186 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:10.186 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:10.186 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:10.186 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:10.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.179+0000 7fe7afcb9640 10 monclient: shutdown 2024-04-26T00:38:10.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.179+0000 7fe7afcb9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:38:10.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.179+0000 7fe7afcb9640 1 -- 172.21.15.37:0/3630195078 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe7a80a3d90 msgr2=0x7fe7a809bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:10.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.179+0000 7fe7afcb9640 1 --2- 172.21.15.37:0/3630195078 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe7a80a3d90 0x7fe7a809bf40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe7a80a6d60 tx=0x7fe7a001c3e0 comp rx=0 tx=0).stop 2024-04-26T00:38:10.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.179+0000 7fe7afcb9640 1 -- 172.21.15.37:0/3630195078 shutdown_connections 2024-04-26T00:38:10.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.179+0000 7fe7afcb9640 1 --2- 172.21.15.37:0/3630195078 >> v2:172.21.15.45:3300/0 conn(0x7fe7a80a4170 0x7fe7a80a45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:10.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.179+0000 7fe7afcb9640 1 --2- 172.21.15.37:0/3630195078 >> v2:172.21.15.37:3300/0 conn(0x7fe7a8133b20 0x7fe7a8133f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:10.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.179+0000 7fe7afcb9640 1 --2- 172.21.15.37:0/3630195078 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe7a80a3d90 0x7fe7a809bf40 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:10.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.179+0000 7fe7afcb9640 1 -- 172.21.15.37:0/3630195078 >> 172.21.15.37:0/3630195078 conn(0x7fe7a80aa0a0 msgr2=0x7fe7a80a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:38:10.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.179+0000 7fe7afcb9640 1 -- 172.21.15.37:0/3630195078 shutdown_connections 2024-04-26T00:38:10.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.179+0000 7fe7afcb9640 1 -- 172.21.15.37:0/3630195078 wait complete. 2024-04-26T00:38:10.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.179+0000 7fe7afcb9640 10 monclient: build_initial_monmap 2024-04-26T00:38:10.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.179+0000 7fe7afcb9640 10 monclient: monmap: 2024-04-26T00:38:10.187 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:38:10.187 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:38:10.187 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:38:10.187 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:38:10.187 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:38:10.187 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:38:10.188 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:38:10.188 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:38:10.188 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:38:10.188 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:10.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7afcb9640 1 Processor -- start 2024-04-26T00:38:10.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7afcb9640 1 -- start start 2024-04-26T00:38:10.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7afcb9640 10 monclient: init 2024-04-26T00:38:10.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7afcb9640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:10.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7afcb9640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:38:10.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7afcb9640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe7a80a3d90 0x7fe7a81cf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:10.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7afcb9640 10 monclient(hunting): picked mon.noname-c con 0x7fe7a80a3d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:38:10.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7afcb9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7a80a4170 0x7fe7a81cfc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:10.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7afcb9640 10 monclient(hunting): picked mon.noname-b con 0x7fe7a80a4170 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:38:10.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7afcb9640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7a8133b20 0x7fe7a81cbd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:10.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7afcb9640 10 monclient(hunting): picked mon.noname-a con 0x7fe7a8133b20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:38:10.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7afcb9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:10.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7afcb9640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a8140c10 con 0x7fe7a8133b20 2024-04-26T00:38:10.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7afcb9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:10.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7afcb9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a809c690 con 0x7fe7a80a4170 2024-04-26T00:38:10.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7afcb9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:10.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7afcb9640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe7a8140d90 con 0x7fe7a80a3d90 2024-04-26T00:38:10.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7afcb9640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:10.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7afcb9640 10 monclient(hunting): authenticate will time out at 2071.488863s 2024-04-26T00:38:10.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7ada55640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe7a80a3d90 0x7fe7a81cf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:10.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7ada55640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe7a80a3d90 0x7fe7a81cf6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:52934/0 (socket says 172.21.15.37:52934) 2024-04-26T00:38:10.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7ada55640 1 -- 172.21.15.37:0/3169148807 learned_addr learned my addr 172.21.15.37:0/3169148807 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:38:10.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7ae256640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7a8133b20 0x7fe7a81cbd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:10.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7ada55640 10 monclient(hunting): get_auth_request con 0x7fe7a80a3d90 auth_method 0 2024-04-26T00:38:10.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7ada55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:10.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7ada55640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:10.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7ada55640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:10.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7ae256640 10 monclient(hunting): get_auth_request con 0x7fe7a8133b20 auth_method 0 2024-04-26T00:38:10.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7ae256640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:10.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7ae256640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:10.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7ae256640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:10.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7ada55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:10.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7ada55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:10.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7ada55640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:10.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7ae256640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:10.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7ae256640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:10.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7ae256640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:10.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7ad254640 1 --2- 172.21.15.37:0/3169148807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7a80a4170 0x7fe7a81cfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:10.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7ad254640 10 monclient(hunting): get_auth_request con 0x7fe7a80a4170 auth_method 0 2024-04-26T00:38:10.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7ad254640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:10.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7ad254640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:10.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7ad254640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:10.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7ae256640 10 monclient(hunting): handle_auth_done global_id 144891 payload 824 2024-04-26T00:38:10.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7ae256640 1 -- 172.21.15.37:0/3169148807 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe7a80a3d90 msgr2=0x7fe7a81cf6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:10.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7ad254640 1 -- 172.21.15.37:0/3169148807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7a80a4170 msgr2=0x7fe7a81cfc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:38:10.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7ad254640 1 -- 172.21.15.37:0/3169148807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7a80a4170 msgr2=0x7fe7a81cfc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:38:10.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7ae256640 1 --2- 172.21.15.37:0/3169148807 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe7a80a3d90 0x7fe7a81cf6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:10.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7ad254640 1 --2- 172.21.15.37:0/3169148807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7a80a4170 0x7fe7a81cfc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:38:10.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7ae256640 1 -- 172.21.15.37:0/3169148807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7a80a4170 msgr2=0x7fe7a81cfc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:10.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7ad254640 1 --2- 172.21.15.37:0/3169148807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7a80a4170 0x7fe7a81cfc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:38:10.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7ae256640 1 --2- 172.21.15.37:0/3169148807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7a80a4170 0x7fe7a81cfc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:10.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7ae256640 10 monclient: _finish_hunting 0 2024-04-26T00:38:10.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7ae256640 1 monclient: found mon.noname-a 2024-04-26T00:38:10.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7ae256640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:10.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7ae256640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:38:10.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7ae256640 1 -- 172.21.15.37:0/3169148807 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7a813cb70 con 0x7fe7a8133b20 2024-04-26T00:38:10.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7ada55640 1 --2- 172.21.15.37:0/3169148807 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe7a80a3d90 0x7fe7a81cf6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:38:10.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7ae256640 1 -- 172.21.15.37:0/3169148807 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7a8133b20 msgr2=0x7fe7a81cbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:10.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7ae256640 1 --2- 172.21.15.37:0/3169148807 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7a8133b20 0x7fe7a81cbd80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fe7a400e930 tx=0x7fe7a400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:10.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe79a7fc640 1 -- 172.21.15.37:0/3169148807 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a4017070 con 0x7fe7a8133b20 2024-04-26T00:38:10.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe79a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:10.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe79a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:38:10.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe79a7fc640 10 monclient: dump: 2024-04-26T00:38:10.192 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:10.192 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:10.193 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:10.193 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:10.193 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:10.193 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:10.193 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:10.193 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:10.193 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:10.193 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:10.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe79a7fc640 10 monclient: _finish_auth 0 2024-04-26T00:38:10.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe79a7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:38:10.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe79a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:10.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe79a7fc640 1 -- 172.21.15.37:0/3169148807 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7a400cb50 con 0x7fe7a8133b20 2024-04-26T00:38:10.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7afcb9640 5 monclient: authenticate success, global_id 144891 2024-04-26T00:38:10.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7afcb9640 10 monclient: _renew_subs 2024-04-26T00:38:10.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7afcb9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:38:10.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe7afcb9640 1 -- 172.21.15.37:0/3169148807 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7a813ce50 con 0x7fe7a8133b20 2024-04-26T00:38:10.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe79a7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:38:10.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe79a7fc640 1 -- 172.21.15.37:0/3169148807 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a4004020 con 0x7fe7a8133b20 2024-04-26T00:38:10.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe79a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:10.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe79a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:38:10.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe79a7fc640 10 monclient: dump: 2024-04-26T00:38:10.194 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:10.194 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:10.194 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:10.194 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:10.194 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:10.194 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:10.194 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:10.195 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:10.195 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:10.195 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:10.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.183+0000 7fe79b7fe640 4 set_mon_vals no callback set 2024-04-26T00:38:10.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.187+0000 7fe7afcb9640 10 monclient: _renew_subs 2024-04-26T00:38:10.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.187+0000 7fe7afcb9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:38:10.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.187+0000 7fe7afcb9640 1 -- 172.21.15.37:0/3169148807 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7a813c890 con 0x7fe7a8133b20 2024-04-26T00:38:10.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.187+0000 7fe7afcb9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:10.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.187+0000 7fe7afcb9640 1 --2- 172.21.15.37:0/3169148807 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7a81cc2c0 0x7fe7a81cc6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:10.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.187+0000 7fe7afcb9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:10.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.187+0000 7fe7afcb9640 1 -- 172.21.15.37:0/3169148807 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a80a0920 con 0x7fe7a81cc2c0 2024-04-26T00:38:10.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.187+0000 7fe7ada55640 1 --2- 172.21.15.37:0/3169148807 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7a81cc2c0 0x7fe7a81cc6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:10.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.187+0000 7fe7ada55640 10 monclient: get_auth_request con 0x7fe7a81cc2c0 auth_method 0 2024-04-26T00:38:10.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.187+0000 7fe7ada55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:10.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.187+0000 7fe7ada55640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:10.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.187+0000 7fe7ada55640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:10.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.187+0000 7fe79a7fc640 1 -- 172.21.15.37:0/3169148807 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe7a4004200 con 0x7fe7a8133b20 2024-04-26T00:38:10.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.187+0000 7fe7ada55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:10.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.187+0000 7fe7ada55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:10.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.187+0000 7fe7ada55640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:10.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.187+0000 7fe7ada55640 10 monclient(hunting): handle_auth_done global_id 144894 payload 824 2024-04-26T00:38:10.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.187+0000 7fe7ada55640 1 -- 172.21.15.37:0/3169148807 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe7a81ccbe0 con 0x7fe7a81cc2c0 2024-04-26T00:38:10.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.187+0000 7fe79a7fc640 1 --2- 172.21.15.37:0/3169148807 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe78c035e80 0x7fe78c038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:10.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.187+0000 7fe79a7fc640 1 -- 172.21.15.37:0/3169148807 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(715..715 src has 370..715) ==== 3927+0+0 (secure 0 0 0) 0x7fe7a4048930 con 0x7fe7a8133b20 2024-04-26T00:38:10.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.187+0000 7fe7ad254640 1 --2- 172.21.15.37:0/3169148807 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe78c035e80 0x7fe78c038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:10.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.187+0000 7fe7ad254640 10 monclient: get_auth_request con 0x7fe78c035e80 auth_method 0 2024-04-26T00:38:10.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.187+0000 7fe7ada55640 1 -- 172.21.15.37:0/3169148807 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7a81cc2c0 msgr2=0x7fe7a81cc6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:10.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.187+0000 7fe7ada55640 1 --2- 172.21.15.37:0/3169148807 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7a81cc2c0 0x7fe7a81cc6a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fe7a001aa90 tx=0x7fe7a0019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:10.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.187+0000 7fe79a7fc640 1 -- 172.21.15.37:0/3169148807 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a00ba070 con 0x7fe7a81cc2c0 2024-04-26T00:38:10.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.187+0000 7fe79a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:10.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.187+0000 7fe79a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:38:10.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.187+0000 7fe79a7fc640 10 monclient: dump: 2024-04-26T00:38:10.201 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:10.201 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:10.201 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:10.201 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:10.201 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:10.201 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:10.201 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:10.201 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:10.201 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:10.201 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:10.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.191+0000 7fe79a7fc640 1 -- 172.21.15.37:0/3169148807 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe7a00b1040 con 0x7fe7a81cc2c0 2024-04-26T00:38:10.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.191+0000 7fe79a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:10.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.191+0000 7fe79a7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:38:10.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.191+0000 7fe79a7fc640 1 -- 172.21.15.37:0/3169148807 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7a81cc2c0 msgr2=0x7fe7a81cc6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:10.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.191+0000 7fe79a7fc640 1 --2- 172.21.15.37:0/3169148807 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7a81cc2c0 0x7fe7a81cc6a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fe7a001aa90 tx=0x7fe7a0019180 comp rx=0 tx=0).stop 2024-04-26T00:38:10.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.191+0000 7fe79a7fc640 1 -- 172.21.15.37:0/3169148807 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7a81cc2c0 msgr2=0x7fe7a81cc6a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:38:10.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.191+0000 7fe79a7fc640 1 --2- 172.21.15.37:0/3169148807 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7a81cc2c0 0x7fe7a81cc6a0 secure :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fe7a001aa90 tx=0x7fe7a0019180 comp rx=0 tx=0).stop 2024-04-26T00:38:10.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.191+0000 7fe7ad254640 1 --2- 172.21.15.37:0/3169148807 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe78c035e80 0x7fe78c038340 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fe7a813a710 tx=0x7fe79c00b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:10.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.195+0000 7fe7afcb9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:38:10.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.195+0000 7fe7afcb9640 1 --2- 172.21.15.37:0/3169148807 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7a8134b30 0x7fe7a8134f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:10.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.195+0000 7fe7afcb9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:10.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.195+0000 7fe7afcb9640 1 -- 172.21.15.37:0/3169148807 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a804f510 con 0x7fe7a8134b30 2024-04-26T00:38:10.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.195+0000 7fe7ada55640 1 --2- 172.21.15.37:0/3169148807 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7a8134b30 0x7fe7a8134f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:10.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.195+0000 7fe7ada55640 10 monclient: get_auth_request con 0x7fe7a8134b30 auth_method 0 2024-04-26T00:38:10.202 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.195+0000 7fe7ada55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:10.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.195+0000 7fe7ada55640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:10.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.195+0000 7fe7ada55640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:10.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.195+0000 7fe7ada55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:10.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.195+0000 7fe7ada55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:10.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.195+0000 7fe7ada55640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:10.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.195+0000 7fe7ada55640 10 monclient(hunting): handle_auth_done global_id 144897 payload 824 2024-04-26T00:38:10.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.195+0000 7fe7ada55640 1 -- 172.21.15.37:0/3169148807 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe7a813b440 con 0x7fe7a8134b30 2024-04-26T00:38:10.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.199+0000 7fe7ada55640 1 -- 172.21.15.37:0/3169148807 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7a8134b30 msgr2=0x7fe7a8134f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:10.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.199+0000 7fe7ada55640 1 --2- 172.21.15.37:0/3169148807 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7a8134b30 0x7fe7a8134f40 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fe7a813e3c0 tx=0x7fe7a0009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:10.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.199+0000 7fe79a7fc640 1 -- 172.21.15.37:0/3169148807 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a00b0040 con 0x7fe7a8134b30 2024-04-26T00:38:10.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.199+0000 7fe79a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:10.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.199+0000 7fe79a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:38:10.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.199+0000 7fe79a7fc640 10 monclient: dump: 2024-04-26T00:38:10.203 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:10.204 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:10.204 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:10.204 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:10.204 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:10.204 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:10.204 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:10.204 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:10.204 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:10.204 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:10.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.199+0000 7fe79a7fc640 1 -- 172.21.15.37:0/3169148807 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4890 (secure 0 0 0) 0x7fe7a00bfd60 con 0x7fe7a8134b30 2024-04-26T00:38:10.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.199+0000 7fe79a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:38:10.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.199+0000 7fe79a7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:38:10.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.199+0000 7fe79a7fc640 1 -- 172.21.15.37:0/3169148807 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7a8134b30 msgr2=0x7fe7a8134f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:10.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.199+0000 7fe79a7fc640 1 --2- 172.21.15.37:0/3169148807 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7a8134b30 0x7fe7a8134f40 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fe7a813e3c0 tx=0x7fe7a0009240 comp rx=0 tx=0).stop 2024-04-26T00:38:10.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.199+0000 7fe79a7fc640 1 -- 172.21.15.37:0/3169148807 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7a8134b30 msgr2=0x7fe7a8134f40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:38:10.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.199+0000 7fe79a7fc640 1 --2- 172.21.15.37:0/3169148807 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7a8134b30 0x7fe7a8134f40 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:10.205 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:38:10.205 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:38:10.205 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:38:10.205 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:38:10.205 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 138, 2024-04-26T00:38:10.205 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 376742, 2024-04-26T00:38:10.205 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:38:10.205 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:38:10.205 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:38:10.206 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:38:10.206 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:38:10.206 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:38:10.206 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:38:10.206 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:38:10.206 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:38:10.206 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:38:10.206 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:38:10.206 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:38:10.206 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:38:10.206 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:38:10.206 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:38:10.206 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:38:10.206 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:38:10.207 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:38:10.207 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:38:10.207 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:38:10.207 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:38:10.207 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:38:10.207 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:38:10.207 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:38:10.207 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:38:10.207 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:38:10.207 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:38:10.207 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:38:10.207 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:38:10.207 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:38:10.207 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:38:10.208 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:38:10.208 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:38:10.208 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:38:10.208 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:10.208 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:10.208 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:38:10.208 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:38:10.208 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:38:10.208 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:38:10.208 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:38:10.208 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:38:10.208 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:38:10.208 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:38:10.209 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:38:10.210 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:38:10.210 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:38:10.210 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:38:10.210 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:38:10.210 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:38:10.210 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:38:10.210 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:38:10.210 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:38:10.210 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:38:10.210 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:38:10.210 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:38:10.210 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:38:10.210 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:38:10.210 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:38:10.211 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:38:10.211 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:38:10.211 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:38:10.211 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:38:10.211 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:38:10.211 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:38:10.211 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:38:10.211 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:10.211 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:38:10.211 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:10.211 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:38:10.211 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:38:10.211 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:38:10.211 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:38:10.212 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:10.212 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:38:10.212 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:38:10.212 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:10.212 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:10.212 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:10.212 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:38:10.212 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:38:10.212 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:10.212 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:10.212 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:10.212 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:10.212 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:38:10.212 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:38:10.213 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:38:10.213 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:38:10.213 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:38:10.213 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:10.213 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:10.213 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:38:10.213 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:38:10.213 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:38:10.213 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:38:10.213 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:10.213 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:38:10.213 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:38:10.213 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:10.213 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:10.214 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:10.214 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:38:10.214 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:38:10.214 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:10.214 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:10.214 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:10.214 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:10.214 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:38:10.214 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:38:10.214 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:38:10.214 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:38:10.214 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:38:10.214 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:10.214 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:10.215 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:38:10.215 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:38:10.215 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:38:10.215 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:38:10.215 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:10.215 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:38:10.215 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:38:10.215 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:10.215 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:10.215 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:10.215 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:38:10.215 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:38:10.215 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:10.215 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:10.216 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:10.216 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:10.216 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:38:10.216 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:38:10.216 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:38:10.216 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:38:10.216 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:38:10.216 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:10.216 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:10.216 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:10.216 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:38:10.216 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:38:10.216 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:10.216 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:38:10.216 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:38:10.217 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:38:10.217 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:10.217 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:38:10.217 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:38:10.217 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:10.217 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:38:10.217 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:38:10.217 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:38:10.217 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:10.217 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:38:10.217 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:38:10.217 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:10.217 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:38:10.217 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:38:10.218 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 7 2024-04-26T00:38:10.218 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:10.218 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:38:10.218 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:38:10.218 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:10.218 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:38:10.218 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:38:10.218 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:38:10.218 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:10.218 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:10.218 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:10.218 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:38:10.218 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:38:10.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.199+0000 7fe7afcb9640 1 -- 172.21.15.37:0/3169148807 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe78c035e80 msgr2=0x7fe78c038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:10.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.199+0000 7fe7afcb9640 1 --2- 172.21.15.37:0/3169148807 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe78c035e80 0x7fe78c038340 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fe7a813a710 tx=0x7fe79c00b040 comp rx=0 tx=0).stop 2024-04-26T00:38:10.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.199+0000 7fe7afcb9640 10 monclient: shutdown 2024-04-26T00:38:10.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.199+0000 7fe7afcb9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:38:10.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.199+0000 7fe7afcb9640 1 -- 172.21.15.37:0/3169148807 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7a8133b20 msgr2=0x7fe7a81cbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:10.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.199+0000 7fe7afcb9640 1 --2- 172.21.15.37:0/3169148807 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7a8133b20 0x7fe7a81cbd80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fe7a400e930 tx=0x7fe7a400edf0 comp rx=0 tx=0).stop 2024-04-26T00:38:10.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.199+0000 7fe7ae256640 1 -- 172.21.15.37:0/3169148807 reap_dead start 2024-04-26T00:38:10.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.199+0000 7fe7ae256640 1 -- 172.21.15.37:0/3169148807 reap_dead start 2024-04-26T00:38:10.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.199+0000 7fe7afcb9640 1 -- 172.21.15.37:0/3169148807 shutdown_connections 2024-04-26T00:38:10.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.199+0000 7fe7afcb9640 1 -- 172.21.15.37:0/3169148807 >> 172.21.15.37:0/3169148807 conn(0x7fe7a80aa0a0 msgr2=0x7fe7a80a2400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:38:10.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.199+0000 7fe7afcb9640 1 -- 172.21.15.37:0/3169148807 shutdown_connections 2024-04-26T00:38:10.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.199+0000 7fe7afcb9640 1 -- 172.21.15.37:0/3169148807 wait complete. 2024-04-26T00:38:10.221 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:38:10.221 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:38:10.221 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:38:10.301 INFO:tasks.rados.rados.0.smithi045.stdout:3204: expect (ObjNum 1051 snap 317 seq_num 1051) 2024-04-26T00:38:10.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7aeb46d640 10 monclient: get_monmap_and_config 2024-04-26T00:38:10.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7aeb46d640 10 monclient: build_initial_monmap 2024-04-26T00:38:10.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7aeb46d640 10 monclient: monmap: 2024-04-26T00:38:10.340 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:38:10.340 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:10.340 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:38:10.344445+0000 2024-04-26T00:38:10.341 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:38:10.344445+0000 2024-04-26T00:38:10.341 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:38:10.341 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:10.341 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:38:10.341 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:38:10.341 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:38:10.341 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:38:10.341 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:38:10.341 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:10.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7aeb46d640 1 Processor -- start 2024-04-26T00:38:10.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7aeb46d640 1 -- start start 2024-04-26T00:38:10.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7aeb46d640 10 monclient: init 2024-04-26T00:38:10.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7aeb46d640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:10.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7aeb46d640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-04-26T00:38:10.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7aeb46d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7ae4136f20 addr v1:172.21.15.37:6789/0 2024-04-26T00:38:10.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7aeb46d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f7ae4137a10 0x7f7ae409bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:10.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7aeb46d640 10 monclient(hunting): picked mon.noname-a con 0x7f7ae4137a10 addr v2:172.21.15.37:3300/0 2024-04-26T00:38:10.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7aeb46d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7ae413c450 0x7f7ae413e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:10.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7aeb46d640 10 monclient(hunting): picked mon.noname-b con 0x7f7ae413c450 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:38:10.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7aeb46d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ae40a50f0 con 0x7f7ae4136f20 2024-04-26T00:38:10.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7aeb46d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:10.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7aeb46d640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f7ae409b330 con 0x7f7ae4137a10 2024-04-26T00:38:10.344 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7aeb46d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:10.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7aeb46d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f7ae409b610 con 0x7f7ae413c450 2024-04-26T00:38:10.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7aeb46d640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:10.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7aeb46d640 10 monclient(hunting): authenticate will time out at 2071.650630s 2024-04-26T00:38:10.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7ae8a08640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f7ae4137a10 0x7f7ae409bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:10.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7ae9a0a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7ae413c450 0x7f7ae413e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:10.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7ae8a08640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f7ae4137a10 0x7f7ae409bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44620/0 (socket says 172.21.15.37:44620) 2024-04-26T00:38:10.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7ae8a08640 1 -- 172.21.15.37:0/621879598 learned_addr learned my addr 172.21.15.37:0/621879598 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:38:10.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7ae8a08640 10 monclient(hunting): get_auth_request con 0x7f7ae4137a10 auth_method 0 2024-04-26T00:38:10.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7ae8a08640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:10.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7ae8a08640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:10.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7ae8a08640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:10.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7ae9209640 1 --1- 172.21.15.37:0/621879598 >> v1:172.21.15.37:6789/0 conn(0x7f7ae4136f20 0x7f7ae4137300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:51764/0 (socket says 172.21.15.37:51764) 2024-04-26T00:38:10.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7ae9a0a640 10 monclient(hunting): get_auth_request con 0x7f7ae413c450 auth_method 0 2024-04-26T00:38:10.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7ae9a0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:10.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7ae9a0a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:10.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7ae9a0a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:10.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7ae8a08640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:10.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7ae8a08640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:10.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7ae8a08640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:10.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7ae9a0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:10.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7ae9a0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:10.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7ae9a0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:10.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7ae9a0a640 10 monclient(hunting): handle_auth_done global_id 204284 payload 274 2024-04-26T00:38:10.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7ae9a0a640 1 -- 172.21.15.37:0/621879598 >> v2:172.21.15.37:3300/0 conn(0x7f7ae4137a10 msgr2=0x7f7ae409bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:38:10.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7ae9a0a640 1 --2- 172.21.15.37:0/621879598 >> v2:172.21.15.37:3300/0 conn(0x7f7ae4137a10 0x7f7ae409bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:10.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7ae9a0a640 1 -- 172.21.15.37:0/621879598 >> v1:172.21.15.37:6789/0 conn(0x7f7ae4136f20 legacy=0x7f7ae4137300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:38:10.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7ae9a0a640 10 monclient: _finish_hunting 0 2024-04-26T00:38:10.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7ae9a0a640 1 monclient: found mon.noname-b 2024-04-26T00:38:10.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7ae9a0a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:10.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7ae9a0a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:38:10.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7ae9a0a640 1 -- 172.21.15.37:0/621879598 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ae409b8f0 con 0x7f7ae413c450 2024-04-26T00:38:10.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.343+0000 7f7ae8a08640 1 --2- 172.21.15.37:0/621879598 >> v2:172.21.15.37:3300/0 conn(0x7f7ae4137a10 0x7f7ae409bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:38:10.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.347+0000 7f7ae9a0a640 1 -- 172.21.15.37:0/621879598 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7ae413c450 msgr2=0x7f7ae413e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:10.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.347+0000 7f7ae9a0a640 1 --2- 172.21.15.37:0/621879598 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7ae413c450 0x7f7ae413e840 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7adc009d90 tx=0x7f7adc01c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3a671693a28dc87 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:10.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.347+0000 7f7adbfff640 1 -- 172.21.15.37:0/621879598 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7adc0a6710 con 0x7f7ae413c450 2024-04-26T00:38:10.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.347+0000 7f7adbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:10.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.347+0000 7f7adbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:38:10.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.347+0000 7f7adbfff640 10 monclient: dump: 2024-04-26T00:38:10.349 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:10.349 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:10.349 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:10.349 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:10.349 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:10.349 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:10.349 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:10.349 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:10.350 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:10.350 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:10.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.347+0000 7f7adbfff640 10 monclient: _finish_auth 0 2024-04-26T00:38:10.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.347+0000 7f7adbfff640 10 monclient: _check_auth_tickets 2024-04-26T00:38:10.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.347+0000 7f7adbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:10.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.347+0000 7f7aeb46d640 5 monclient: authenticate success, global_id 204284 2024-04-26T00:38:10.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.347+0000 7f7aeb46d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:38:10.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.347+0000 7f7adbfff640 1 -- 172.21.15.37:0/621879598 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7adc0a68b0 con 0x7f7ae413c450 2024-04-26T00:38:10.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.347+0000 7f7adbfff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:38:10.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.347+0000 7f7aeb46d640 10 monclient: get_monmap_and_config success 2024-04-26T00:38:10.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.347+0000 7f7adbfff640 1 -- 172.21.15.37:0/621879598 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7adc0a6c80 con 0x7f7ae413c450 2024-04-26T00:38:10.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.347+0000 7f7aeb46d640 4 set_mon_vals no callback set 2024-04-26T00:38:10.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.347+0000 7f7adbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:10.350 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.347+0000 7f7adbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:38:10.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.347+0000 7f7adbfff640 10 monclient: dump: 2024-04-26T00:38:10.351 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:10.351 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:10.351 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:10.351 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:10.351 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:10.351 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:10.351 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:10.351 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:10.351 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:10.351 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:10.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7aeb46d640 10 monclient: shutdown 2024-04-26T00:38:10.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7aeb46d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:38:10.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7aeb46d640 1 -- 172.21.15.37:0/621879598 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7ae413c450 msgr2=0x7f7ae413e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:10.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7aeb46d640 1 --2- 172.21.15.37:0/621879598 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7ae413c450 0x7f7ae413e840 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7adc009d90 tx=0x7f7adc01c880 comp rx=0 tx=0).stop 2024-04-26T00:38:10.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7aeb46d640 1 -- 172.21.15.37:0/621879598 shutdown_connections 2024-04-26T00:38:10.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7aeb46d640 1 --2- 172.21.15.37:0/621879598 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7ae413c450 0x7f7ae413e840 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:10.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7aeb46d640 1 --2- 172.21.15.37:0/621879598 >> v2:172.21.15.37:3300/0 conn(0x7f7ae4137a10 0x7f7ae409bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:10.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7aeb46d640 1 -- 172.21.15.37:0/621879598 >> 172.21.15.37:0/621879598 conn(0x7f7ae40a61b0 msgr2=0x7f7ae40a65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:38:10.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7aeb46d640 1 -- 172.21.15.37:0/621879598 shutdown_connections 2024-04-26T00:38:10.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7aeb46d640 1 -- 172.21.15.37:0/621879598 wait complete. 2024-04-26T00:38:10.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7aeb46d640 10 monclient: build_initial_monmap 2024-04-26T00:38:10.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7aeb46d640 10 monclient: monmap: 2024-04-26T00:38:10.352 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:38:10.352 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:38:10.352 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:38:10.352 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:38:10.352 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:38:10.352 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:38:10.352 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:38:10.353 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:38:10.353 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:38:10.353 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:10.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7aeb46d640 1 Processor -- start 2024-04-26T00:38:10.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7aeb46d640 1 -- start start 2024-04-26T00:38:10.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7aeb46d640 10 monclient: init 2024-04-26T00:38:10.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7aeb46d640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:10.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7aeb46d640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:38:10.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7aeb46d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7ae4136f20 0x7f7ae413a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:10.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7aeb46d640 10 monclient(hunting): picked mon.noname-c con 0x7f7ae4136f20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:38:10.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7aeb46d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ae4137a10 0x7f7ae4138640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:10.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7aeb46d640 10 monclient(hunting): picked mon.noname-b con 0x7f7ae4137a10 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:38:10.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7aeb46d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7ae413c450 0x7f7ae4135240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:10.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7aeb46d640 10 monclient(hunting): picked mon.noname-a con 0x7f7ae413c450 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:38:10.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7aeb46d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:10.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7aeb46d640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae4141930 con 0x7f7ae413c450 2024-04-26T00:38:10.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7aeb46d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:10.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7aeb46d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae41417b0 con 0x7f7ae4137a10 2024-04-26T00:38:10.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7aeb46d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:10.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7aeb46d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f7ae4141ab0 con 0x7f7ae4136f20 2024-04-26T00:38:10.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7aeb46d640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:10.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7aeb46d640 10 monclient(hunting): authenticate will time out at 2071.657836s 2024-04-26T00:38:10.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7ae9209640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7ae4136f20 0x7f7ae413a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:10.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7ae9a0a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7ae413c450 0x7f7ae4135240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:10.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7ae9a0a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7ae413c450 0x7f7ae4135240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44622/0 (socket says 172.21.15.37:44622) 2024-04-26T00:38:10.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7ae9a0a640 1 -- 172.21.15.37:0/2604596689 learned_addr learned my addr 172.21.15.37:0/2604596689 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:38:10.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7ae9a0a640 10 monclient(hunting): get_auth_request con 0x7f7ae413c450 auth_method 0 2024-04-26T00:38:10.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7ae9a0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:10.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7ae9a0a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:10.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7ae9a0a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:10.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7ae9209640 10 monclient(hunting): get_auth_request con 0x7f7ae4136f20 auth_method 0 2024-04-26T00:38:10.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7ae9209640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:10.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7ae9209640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:10.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7ae9209640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:10.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7ae9a0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:10.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7ae9a0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:10.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7ae9a0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:10.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7ae9209640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:10.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7ae9209640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:10.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7ae9209640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:10.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7ae8a08640 1 --2- 172.21.15.37:0/2604596689 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ae4137a10 0x7f7ae4138640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:10.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7ae8a08640 10 monclient(hunting): get_auth_request con 0x7f7ae4137a10 auth_method 0 2024-04-26T00:38:10.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7ae8a08640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:10.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7ae8a08640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:10.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7ae8a08640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:10.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7ae9a0a640 10 monclient(hunting): handle_auth_done global_id 144903 payload 824 2024-04-26T00:38:10.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7ae9a0a640 1 -- 172.21.15.37:0/2604596689 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7ae4136f20 msgr2=0x7f7ae413a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:10.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7ae9a0a640 1 --2- 172.21.15.37:0/2604596689 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7ae4136f20 0x7f7ae413a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:10.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7ae9a0a640 1 -- 172.21.15.37:0/2604596689 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ae4137a10 msgr2=0x7f7ae4138640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:10.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7ae9a0a640 1 --2- 172.21.15.37:0/2604596689 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ae4137a10 0x7f7ae4138640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:10.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7ae9a0a640 10 monclient: _finish_hunting 0 2024-04-26T00:38:10.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7ae9a0a640 1 monclient: found mon.noname-a 2024-04-26T00:38:10.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7ae9a0a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:10.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7ae9a0a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:38:10.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.351+0000 7f7ae9a0a640 1 -- 172.21.15.37:0/2604596689 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ae413b6f0 con 0x7f7ae413c450 2024-04-26T00:38:10.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7ae9a0a640 1 -- 172.21.15.37:0/2604596689 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7ae413c450 msgr2=0x7f7ae4135240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:10.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7ae9a0a640 1 --2- 172.21.15.37:0/2604596689 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7ae413c450 0x7f7ae4135240 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f7adc01d120 tx=0x7f7adc019620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:10.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7ad9ffb640 1 -- 172.21.15.37:0/2604596689 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7adc0be070 con 0x7f7ae413c450 2024-04-26T00:38:10.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7ad9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:10.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7ad9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:38:10.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7ad9ffb640 10 monclient: dump: 2024-04-26T00:38:10.362 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:10.362 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:10.362 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:10.362 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:10.362 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:10.362 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:10.362 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:10.362 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:10.362 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:10.362 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:10.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7ad9ffb640 10 monclient: _finish_auth 0 2024-04-26T00:38:10.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7ad9ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:38:10.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7ad9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:10.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7aeb46d640 5 monclient: authenticate success, global_id 144903 2024-04-26T00:38:10.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7ad9ffb640 1 -- 172.21.15.37:0/2604596689 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7adc0b8840 con 0x7f7ae413c450 2024-04-26T00:38:10.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7aeb46d640 10 monclient: _renew_subs 2024-04-26T00:38:10.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7aeb46d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:38:10.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7aeb46d640 1 -- 172.21.15.37:0/2604596689 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ae413aea0 con 0x7f7ae413c450 2024-04-26T00:38:10.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7ad9ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:38:10.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7ad9ffb640 1 -- 172.21.15.37:0/2604596689 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7adc0b8bf0 con 0x7f7ae413c450 2024-04-26T00:38:10.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7ad9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:10.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7ad9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:38:10.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7ad9ffb640 10 monclient: dump: 2024-04-26T00:38:10.363 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:10.363 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:10.363 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:10.364 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:10.364 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:10.364 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:10.364 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:10.364 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:10.364 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:10.364 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:10.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7adb7fe640 4 set_mon_vals no callback set 2024-04-26T00:38:10.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7aeb46d640 10 monclient: _renew_subs 2024-04-26T00:38:10.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7aeb46d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:38:10.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7aeb46d640 1 -- 172.21.15.37:0/2604596689 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ae4139ca0 con 0x7f7ae413c450 2024-04-26T00:38:10.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7aeb46d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:10.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7aeb46d640 1 --2- 172.21.15.37:0/2604596689 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7ae4135780 0x7f7ae41cf750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:10.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7aeb46d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:10.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7aeb46d640 1 -- 172.21.15.37:0/2604596689 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f7ae404f510 con 0x7f7ae4135780 2024-04-26T00:38:10.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7ae9209640 1 --2- 172.21.15.37:0/2604596689 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7ae4135780 0x7f7ae41cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:10.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7ae9209640 10 monclient: get_auth_request con 0x7f7ae4135780 auth_method 0 2024-04-26T00:38:10.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7ae9209640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:10.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7ae9209640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:10.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7ae9209640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:10.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7ae9209640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:10.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7ae9209640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:10.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7ae9209640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:10.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7ae9209640 10 monclient(hunting): handle_auth_done global_id 204290 payload 824 2024-04-26T00:38:10.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7ae9209640 1 -- 172.21.15.37:0/2604596689 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7ae40a3910 con 0x7f7ae4135780 2024-04-26T00:38:10.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7ad9ffb640 1 -- 172.21.15.37:0/2604596689 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f7adc0c1070 con 0x7f7ae413c450 2024-04-26T00:38:10.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7ad9ffb640 1 --2- 172.21.15.37:0/2604596689 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7ac8035e80 0x7f7ac8038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:10.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7ad9ffb640 1 -- 172.21.15.37:0/2604596689 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(715..715 src has 370..715) ==== 3927+0+0 (secure 0 0 0) 0x7f7adc0e41c0 con 0x7f7ae413c450 2024-04-26T00:38:10.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7ae8a08640 1 --2- 172.21.15.37:0/2604596689 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7ac8035e80 0x7f7ac8038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:10.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.355+0000 7f7ae8a08640 10 monclient: get_auth_request con 0x7f7ac8035e80 auth_method 0 2024-04-26T00:38:10.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.359+0000 7f7ae9209640 1 -- 172.21.15.37:0/2604596689 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7ae4135780 msgr2=0x7f7ae41cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:10.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.359+0000 7f7ae9209640 1 --2- 172.21.15.37:0/2604596689 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7ae4135780 0x7f7ae41cf750 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7ae000e730 tx=0x7f7ae000ebf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:10.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.359+0000 7f7ad9ffb640 1 -- 172.21.15.37:0/2604596689 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae0004d60 con 0x7f7ae4135780 2024-04-26T00:38:10.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.359+0000 7f7ad9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:10.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.359+0000 7f7ad9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:38:10.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.359+0000 7f7ad9ffb640 10 monclient: dump: 2024-04-26T00:38:10.367 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:10.367 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:10.367 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:10.367 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:10.367 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:10.367 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:10.367 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:10.367 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:10.367 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:10.367 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:10.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.359+0000 7f7ad9ffb640 1 -- 172.21.15.37:0/2604596689 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7ae0019070 con 0x7f7ae4135780 2024-04-26T00:38:10.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.359+0000 7f7ad9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:10.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.359+0000 7f7ad9ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:38:10.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.359+0000 7f7ad9ffb640 1 -- 172.21.15.37:0/2604596689 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7ae4135780 msgr2=0x7f7ae41cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:10.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.359+0000 7f7ad9ffb640 1 --2- 172.21.15.37:0/2604596689 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7ae4135780 0x7f7ae41cf750 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7ae000e730 tx=0x7f7ae000ebf0 comp rx=0 tx=0).stop 2024-04-26T00:38:10.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.359+0000 7f7ad9ffb640 1 -- 172.21.15.37:0/2604596689 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7ae4135780 msgr2=0x7f7ae41cf750 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:38:10.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.359+0000 7f7ad9ffb640 1 --2- 172.21.15.37:0/2604596689 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7ae4135780 0x7f7ae41cf750 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:10.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.359+0000 7f7ae8a08640 1 --2- 172.21.15.37:0/2604596689 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7ac8035e80 0x7f7ac8038340 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7ae4139ff0 tx=0x7f7ad4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:10.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.363+0000 7f7aeb46d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:38:10.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.363+0000 7f7aeb46d640 1 --2- 172.21.15.37:0/2604596689 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7ae402c000 0x7f7ae402c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:10.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.363+0000 7f7aeb46d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:10.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.363+0000 7f7aeb46d640 1 -- 172.21.15.37:0/2604596689 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f7ae40a3000 con 0x7f7ae402c000 2024-04-26T00:38:10.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.363+0000 7f7ae9209640 1 --2- 172.21.15.37:0/2604596689 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7ae402c000 0x7f7ae402c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:10.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.363+0000 7f7ae9209640 10 monclient: get_auth_request con 0x7f7ae402c000 auth_method 0 2024-04-26T00:38:10.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.363+0000 7f7ae9209640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:10.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.363+0000 7f7ae9209640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:10.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.363+0000 7f7ae9209640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:10.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.363+0000 7f7ae9209640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:10.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.363+0000 7f7ae9209640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:10.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.363+0000 7f7ae9209640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:10.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.363+0000 7f7ae9209640 10 monclient(hunting): handle_auth_done global_id 204293 payload 824 2024-04-26T00:38:10.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.363+0000 7f7ae9209640 1 -- 172.21.15.37:0/2604596689 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7ae4075f10 con 0x7f7ae402c000 2024-04-26T00:38:10.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.367+0000 7f7ae9209640 1 -- 172.21.15.37:0/2604596689 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7ae402c000 msgr2=0x7f7ae402c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:10.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.367+0000 7f7ae9209640 1 --2- 172.21.15.37:0/2604596689 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7ae402c000 0x7f7ae402c410 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7ae000f010 tx=0x7f7ae0004940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:10.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.367+0000 7f7ad9ffb640 1 -- 172.21.15.37:0/2604596689 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae000c8b0 con 0x7f7ae402c000 2024-04-26T00:38:10.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.367+0000 7f7ad9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:10.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.367+0000 7f7ad9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:38:10.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.367+0000 7f7ad9ffb640 10 monclient: dump: 2024-04-26T00:38:10.372 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:10.372 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:10.372 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:10.372 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:10.372 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:10.372 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:10.372 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:10.372 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:10.372 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:10.372 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:10.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.367+0000 7f7ad9ffb640 1 -- 172.21.15.37:0/2604596689 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4547 (secure 0 0 0) 0x7f7ae000c8b0 con 0x7f7ae402c000 2024-04-26T00:38:10.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.367+0000 7f7ad9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:38:10.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.367+0000 7f7ad9ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:38:10.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.367+0000 7f7ad9ffb640 1 -- 172.21.15.37:0/2604596689 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7ae402c000 msgr2=0x7f7ae402c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:10.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.367+0000 7f7ad9ffb640 1 --2- 172.21.15.37:0/2604596689 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7ae402c000 0x7f7ae402c410 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7ae000f010 tx=0x7f7ae0004940 comp rx=0 tx=0).stop 2024-04-26T00:38:10.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.367+0000 7f7ad9ffb640 1 -- 172.21.15.37:0/2604596689 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7ae402c000 msgr2=0x7f7ae402c410 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:38:10.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.367+0000 7f7ad9ffb640 1 --2- 172.21.15.37:0/2604596689 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7ae402c000 0x7f7ae402c410 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7ae000f010 tx=0x7f7ae0004940 comp rx=0 tx=0).stop 2024-04-26T00:38:10.373 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:38:10.373 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:38:10.373 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:38:10.373 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:38:10.373 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 138, 2024-04-26T00:38:10.373 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 95974, 2024-04-26T00:38:10.373 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:38:10.374 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:38:10.374 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:38:10.374 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:38:10.374 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:38:10.374 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:38:10.374 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:38:10.374 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:38:10.374 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:38:10.374 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:38:10.374 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:38:10.374 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:38:10.374 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:38:10.374 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:38:10.374 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:38:10.374 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:38:10.375 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:38:10.375 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:38:10.375 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:38:10.375 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:38:10.375 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:38:10.375 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:38:10.375 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:38:10.375 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:38:10.375 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:38:10.375 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:38:10.375 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:38:10.375 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:38:10.375 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:38:10.375 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:38:10.376 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:38:10.376 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:38:10.376 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:38:10.376 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:38:10.376 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:10.376 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:10.376 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:38:10.376 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:38:10.376 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:38:10.376 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:38:10.376 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:38:10.376 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:38:10.376 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:38:10.376 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:38:10.376 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:38:10.377 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:38:10.377 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:38:10.377 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:38:10.377 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:38:10.377 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:38:10.377 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:38:10.377 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:38:10.377 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:38:10.377 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:38:10.377 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:38:10.377 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:38:10.377 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:38:10.377 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:38:10.377 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:38:10.377 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:38:10.378 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:38:10.378 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:38:10.378 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:38:10.378 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:38:10.378 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:38:10.378 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:38:10.378 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:10.378 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:38:10.378 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:10.378 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:38:10.378 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:38:10.378 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:38:10.378 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:38:10.378 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:10.379 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:38:10.379 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:38:10.379 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:10.379 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:10.379 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:10.379 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:38:10.379 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:38:10.379 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:10.379 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:10.379 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:10.379 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:10.379 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:38:10.379 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:38:10.379 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:38:10.379 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:38:10.380 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:38:10.380 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:10.380 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:10.380 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:38:10.380 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:38:10.380 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:38:10.380 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:38:10.380 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:10.380 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:38:10.380 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:38:10.380 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:10.380 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:10.380 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:10.380 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:38:10.380 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:38:10.381 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:10.381 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:10.381 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:10.381 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:10.381 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:38:10.381 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:38:10.381 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:38:10.381 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:38:10.381 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:38:10.381 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:10.381 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:10.381 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:38:10.381 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:38:10.381 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:38:10.382 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:38:10.382 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:10.382 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:38:10.382 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:38:10.382 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:10.382 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:10.382 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:10.382 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:38:10.382 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:38:10.382 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:10.382 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:10.382 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:10.382 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:10.382 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:38:10.382 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:38:10.383 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:38:10.383 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:38:10.383 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:38:10.383 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:10.383 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:10.383 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:10.383 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:38:10.383 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:38:10.383 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:10.383 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:38:10.383 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:38:10.383 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:38:10.383 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:10.383 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:38:10.384 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:38:10.384 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:10.384 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:38:10.384 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:38:10.384 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:38:10.384 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:10.384 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:10.384 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:10.384 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:38:10.384 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:38:10.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.367+0000 7f7aeb46d640 1 -- 172.21.15.37:0/2604596689 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7ac8035e80 msgr2=0x7f7ac8038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:10.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.367+0000 7f7aeb46d640 1 --2- 172.21.15.37:0/2604596689 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7ac8035e80 0x7f7ac8038340 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7ae4139ff0 tx=0x7f7ad4008040 comp rx=0 tx=0).stop 2024-04-26T00:38:10.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.367+0000 7f7aeb46d640 10 monclient: shutdown 2024-04-26T00:38:10.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.367+0000 7f7aeb46d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:38:10.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.367+0000 7f7aeb46d640 1 -- 172.21.15.37:0/2604596689 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7ae413c450 msgr2=0x7f7ae4135240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:10.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.367+0000 7f7aeb46d640 1 --2- 172.21.15.37:0/2604596689 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7ae413c450 0x7f7ae4135240 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f7adc01d120 tx=0x7f7adc019620 comp rx=0 tx=0).stop 2024-04-26T00:38:10.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.367+0000 7f7ae9a0a640 1 -- 172.21.15.37:0/2604596689 reap_dead start 2024-04-26T00:38:10.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.367+0000 7f7ae9a0a640 1 -- 172.21.15.37:0/2604596689 reap_dead start 2024-04-26T00:38:10.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.367+0000 7f7aeb46d640 1 -- 172.21.15.37:0/2604596689 shutdown_connections 2024-04-26T00:38:10.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.367+0000 7f7aeb46d640 1 -- 172.21.15.37:0/2604596689 >> 172.21.15.37:0/2604596689 conn(0x7f7ae40a61b0 msgr2=0x7f7ae40a81a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:38:10.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.367+0000 7f7aeb46d640 1 -- 172.21.15.37:0/2604596689 shutdown_connections 2024-04-26T00:38:10.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.367+0000 7f7aeb46d640 1 -- 172.21.15.37:0/2604596689 wait complete. 2024-04-26T00:38:10.387 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:38:10.387 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:38:10.387 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:38:10.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.503+0000 7ff05e486640 10 monclient: get_monmap_and_config 2024-04-26T00:38:10.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.503+0000 7ff05e486640 10 monclient: build_initial_monmap 2024-04-26T00:38:10.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.503+0000 7ff05e486640 10 monclient: monmap: 2024-04-26T00:38:10.503 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:38:10.503 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:10.503 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:38:10.506405+0000 2024-04-26T00:38:10.503 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:38:10.506405+0000 2024-04-26T00:38:10.503 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:38:10.503 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:10.503 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:38:10.503 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:38:10.503 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:38:10.503 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:38:10.503 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:38:10.503 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:10.503 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.503+0000 7ff05e486640 1 Processor -- start 2024-04-26T00:38:10.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.503+0000 7ff05e486640 1 -- start start 2024-04-26T00:38:10.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.503+0000 7ff05e486640 10 monclient: init 2024-04-26T00:38:10.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.503+0000 7ff05e486640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:10.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.503+0000 7ff05e486640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-04-26T00:38:10.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.503+0000 7ff05e486640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0580726b0 0x7ff058072a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:10.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.503+0000 7ff05e486640 10 monclient(hunting): picked mon.noname-b con 0x7ff0580726b0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:38:10.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.503+0000 7ff05e486640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff0580a0110 0x7ff058136dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:10.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.503+0000 7ff05e486640 10 monclient(hunting): picked mon.noname-c con 0x7ff0580a0110 addr v2:172.21.15.45:3300/0 2024-04-26T00:38:10.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.503+0000 7ff05e486640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff058137310 addr v1:172.21.15.45:6789/0 2024-04-26T00:38:10.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.503+0000 7ff05e486640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0580a1470 con 0x7ff058137310 2024-04-26T00:38:10.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.503+0000 7ff05e486640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:10.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.503+0000 7ff05e486640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7ff0580a1bd0 con 0x7ff0580a0110 2024-04-26T00:38:10.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.503+0000 7ff05e486640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:10.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.503+0000 7ff05e486640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff0580a0eb0 con 0x7ff0580726b0 2024-04-26T00:38:10.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.503+0000 7ff05e486640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:10.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.503+0000 7ff05e486640 10 monclient(hunting): authenticate will time out at 2071.811730s 2024-04-26T00:38:10.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.503+0000 7ff057fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0580726b0 0x7ff058072a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:10.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.503+0000 7ff057fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0580726b0 0x7ff058072a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:52966/0 (socket says 172.21.15.37:52966) 2024-04-26T00:38:10.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.503+0000 7ff057fff640 1 -- 172.21.15.37:0/3141739177 learned_addr learned my addr 172.21.15.37:0/3141739177 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:38:10.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.503+0000 7ff057fff640 10 monclient(hunting): get_auth_request con 0x7ff0580726b0 auth_method 0 2024-04-26T00:38:10.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.507+0000 7ff057fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:10.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.507+0000 7ff057fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:10.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.507+0000 7ff057fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:10.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.507+0000 7ff0577fe640 1 --2- 172.21.15.37:0/3141739177 >> v2:172.21.15.45:3300/0 conn(0x7ff0580a0110 0x7ff058136dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:10.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.507+0000 7ff0577fe640 10 monclient(hunting): get_auth_request con 0x7ff0580a0110 auth_method 0 2024-04-26T00:38:10.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.507+0000 7ff0577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:10.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.507+0000 7ff0577fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:10.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.507+0000 7ff0577fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:10.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.507+0000 7ff057fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:10.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.507+0000 7ff057fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:10.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.507+0000 7ff057fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:10.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.507+0000 7ff0577fe640 1 -- 172.21.15.37:0/3141739177 >> v2:172.21.15.45:3300/0 conn(0x7ff0580a0110 msgr2=0x7ff058136dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:38:10.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.507+0000 7ff0577fe640 1 -- 172.21.15.37:0/3141739177 >> v2:172.21.15.45:3300/0 conn(0x7ff0580a0110 msgr2=0x7ff058136dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:38:10.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.507+0000 7ff0577fe640 1 --2- 172.21.15.37:0/3141739177 >> v2:172.21.15.45:3300/0 conn(0x7ff0580a0110 0x7ff058136dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:38:10.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.507+0000 7ff0577fe640 1 --2- 172.21.15.37:0/3141739177 >> v2:172.21.15.45:3300/0 conn(0x7ff0580a0110 0x7ff058136dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:38:10.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.507+0000 7ff057fff640 10 monclient(hunting): handle_auth_done global_id 204296 payload 274 2024-04-26T00:38:10.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.507+0000 7ff057fff640 1 -- 172.21.15.37:0/3141739177 >> v2:172.21.15.45:3300/0 conn(0x7ff0580a0110 msgr2=0x7ff058136dd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:38:10.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.507+0000 7ff057fff640 1 --2- 172.21.15.37:0/3141739177 >> v2:172.21.15.45:3300/0 conn(0x7ff0580a0110 0x7ff058136dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:10.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.507+0000 7ff057fff640 1 -- 172.21.15.37:0/3141739177 >> v1:172.21.15.45:6789/0 conn(0x7ff058137310 legacy=0x7ff05813e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:38:10.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.507+0000 7ff057fff640 10 monclient: _finish_hunting 0 2024-04-26T00:38:10.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.507+0000 7ff057fff640 1 monclient: found mon.noname-b 2024-04-26T00:38:10.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.507+0000 7ff057fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:10.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.507+0000 7ff057fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:38:10.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.507+0000 7ff057fff640 1 -- 172.21.15.37:0/3141739177 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0580a0bd0 con 0x7ff0580726b0 2024-04-26T00:38:10.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff057fff640 1 -- 172.21.15.37:0/3141739177 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0580726b0 msgr2=0x7ff058072a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:10.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff057fff640 1 --2- 172.21.15.37:0/3141739177 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0580726b0 0x7ff058072a90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff048009870 tx=0x7ff04801c1d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63f6475f6bb7ed3d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:10.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff056ffd640 1 -- 172.21.15.37:0/3141739177 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0480b0070 con 0x7ff0580726b0 2024-04-26T00:38:10.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff056ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:10.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff056ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:38:10.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff056ffd640 10 monclient: dump: 2024-04-26T00:38:10.511 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:10.511 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:10.511 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:10.511 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:10.511 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:10.511 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:10.511 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:10.511 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:10.511 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:10.511 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:10.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff056ffd640 10 monclient: _finish_auth 0 2024-04-26T00:38:10.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff056ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:38:10.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff056ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:10.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff056ffd640 1 -- 172.21.15.37:0/3141739177 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0480ab400 con 0x7ff0580726b0 2024-04-26T00:38:10.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff056ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:38:10.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff056ffd640 1 -- 172.21.15.37:0/3141739177 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0480ab7d0 con 0x7ff0580726b0 2024-04-26T00:38:10.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff056ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:10.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff056ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:38:10.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff056ffd640 10 monclient: dump: 2024-04-26T00:38:10.512 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:10.512 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:10.512 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:10.512 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:10.512 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:10.512 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:10.512 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:10.512 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:10.512 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:10.512 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:10.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff05e486640 5 monclient: authenticate success, global_id 204296 2024-04-26T00:38:10.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff05e486640 10 monclient: get_monmap_and_config success 2024-04-26T00:38:10.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff05e486640 4 set_mon_vals no callback set 2024-04-26T00:38:10.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff05e486640 10 monclient: shutdown 2024-04-26T00:38:10.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff05e486640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:38:10.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff05e486640 1 -- 172.21.15.37:0/3141739177 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0580726b0 msgr2=0x7ff058072a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:10.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff05e486640 1 --2- 172.21.15.37:0/3141739177 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0580726b0 0x7ff058072a90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff048009870 tx=0x7ff04801c1d0 comp rx=0 tx=0).stop 2024-04-26T00:38:10.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff05e486640 1 -- 172.21.15.37:0/3141739177 shutdown_connections 2024-04-26T00:38:10.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff05e486640 1 --2- 172.21.15.37:0/3141739177 >> v2:172.21.15.45:3300/0 conn(0x7ff0580a0110 0x7ff058136dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:10.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff05e486640 1 --2- 172.21.15.37:0/3141739177 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0580726b0 0x7ff058072a90 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:10.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff05e486640 1 -- 172.21.15.37:0/3141739177 >> 172.21.15.37:0/3141739177 conn(0x7ff0580a1d50 msgr2=0x7ff0580a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:38:10.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff05e486640 1 -- 172.21.15.37:0/3141739177 shutdown_connections 2024-04-26T00:38:10.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff05e486640 1 -- 172.21.15.37:0/3141739177 wait complete. 2024-04-26T00:38:10.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff05e486640 10 monclient: build_initial_monmap 2024-04-26T00:38:10.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff05e486640 10 monclient: monmap: 2024-04-26T00:38:10.513 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:38:10.514 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:38:10.514 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:38:10.514 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:38:10.514 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:38:10.514 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:38:10.514 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:38:10.514 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:38:10.514 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:38:10.514 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:10.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff05e486640 1 Processor -- start 2024-04-26T00:38:10.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff05e486640 1 -- start start 2024-04-26T00:38:10.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff05e486640 10 monclient: init 2024-04-26T00:38:10.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff05e486640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:10.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff05e486640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:38:10.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff05e486640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0580726b0 0x7ff05813def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:10.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff05e486640 10 monclient(hunting): picked mon.noname-c con 0x7ff0580726b0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:38:10.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff05e486640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0580a0110 0x7ff05812fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:10.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff05e486640 10 monclient(hunting): picked mon.noname-a con 0x7ff0580a0110 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:38:10.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff05e486640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff058137310 0x7ff058130370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:10.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff05e486640 10 monclient(hunting): picked mon.noname-b con 0x7ff058137310 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:38:10.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff05e486640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:10.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff05e486640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff05809bc50 con 0x7ff0580a0110 2024-04-26T00:38:10.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff05e486640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:10.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff05e486640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff0580730e0 con 0x7ff058137310 2024-04-26T00:38:10.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff05e486640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:10.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff05e486640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff05809bdd0 con 0x7ff0580726b0 2024-04-26T00:38:10.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff05e486640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:10.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff05e486640 10 monclient(hunting): authenticate will time out at 2071.819492s 2024-04-26T00:38:10.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff057fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0580726b0 0x7ff05813def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:10.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff0577fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0580a0110 0x7ff05812fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:10.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff057fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0580726b0 0x7ff05813def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:52978/0 (socket says 172.21.15.37:52978) 2024-04-26T00:38:10.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff057fff640 1 -- 172.21.15.37:0/2147823750 learned_addr learned my addr 172.21.15.37:0/2147823750 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:38:10.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff057fff640 10 monclient(hunting): get_auth_request con 0x7ff0580726b0 auth_method 0 2024-04-26T00:38:10.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff057fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:10.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff057fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:10.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff057fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:10.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff0577fe640 10 monclient(hunting): get_auth_request con 0x7ff0580a0110 auth_method 0 2024-04-26T00:38:10.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff0577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:10.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff0577fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:10.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff0577fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:10.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff057fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:10.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff057fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:10.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff057fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:10.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff0577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:10.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff0577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:10.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff0577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:10.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff05ca23640 1 --2- 172.21.15.37:0/2147823750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff058137310 0x7ff058130370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:10.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff05ca23640 10 monclient(hunting): get_auth_request con 0x7ff058137310 auth_method 0 2024-04-26T00:38:10.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff05ca23640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:10.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff05ca23640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:10.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff05ca23640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:10.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.511+0000 7ff0577fe640 10 monclient(hunting): handle_auth_done global_id 144906 payload 824 2024-04-26T00:38:10.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff0577fe640 1 -- 172.21.15.37:0/2147823750 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0580726b0 msgr2=0x7ff05813def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:10.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff0577fe640 1 --2- 172.21.15.37:0/2147823750 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0580726b0 0x7ff05813def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:10.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff05ca23640 1 -- 172.21.15.37:0/2147823750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff058137310 msgr2=0x7ff058130370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:38:10.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff0577fe640 1 -- 172.21.15.37:0/2147823750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff058137310 msgr2=0x7ff058130370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:10.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff05ca23640 1 -- 172.21.15.37:0/2147823750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff058137310 msgr2=0x7ff058130370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:38:10.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff05ca23640 1 --2- 172.21.15.37:0/2147823750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff058137310 0x7ff058130370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:38:10.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff05ca23640 1 --2- 172.21.15.37:0/2147823750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff058137310 0x7ff058130370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:38:10.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff0577fe640 1 --2- 172.21.15.37:0/2147823750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff058137310 0x7ff058130370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:10.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff0577fe640 10 monclient: _finish_hunting 0 2024-04-26T00:38:10.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff0577fe640 1 monclient: found mon.noname-a 2024-04-26T00:38:10.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff0577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:10.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff0577fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:38:10.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff0577fe640 1 -- 172.21.15.37:0/2147823750 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff05813d490 con 0x7ff0580a0110 2024-04-26T00:38:10.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff057fff640 1 --2- 172.21.15.37:0/2147823750 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0580726b0 0x7ff05813def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:38:10.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff0577fe640 1 -- 172.21.15.37:0/2147823750 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0580a0110 msgr2=0x7ff05812fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:10.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff0577fe640 1 --2- 172.21.15.37:0/2147823750 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0580a0110 0x7ff05812fe30 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7ff04c00e7e0 tx=0x7ff04c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:10.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff054ff9640 1 -- 172.21.15.37:0/2147823750 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff04c017030 con 0x7ff0580a0110 2024-04-26T00:38:10.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff054ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:10.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff054ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:38:10.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff054ff9640 10 monclient: dump: 2024-04-26T00:38:10.523 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:10.523 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:10.523 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:10.524 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:10.524 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:10.524 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:10.524 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:10.524 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:10.524 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:10.524 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:10.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff054ff9640 10 monclient: _finish_auth 0 2024-04-26T00:38:10.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff054ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:38:10.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff054ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:10.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff054ff9640 1 -- 172.21.15.37:0/2147823750 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff04c004020 con 0x7ff0580a0110 2024-04-26T00:38:10.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff054ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:38:10.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff054ff9640 1 -- 172.21.15.37:0/2147823750 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff04c00f040 con 0x7ff0580a0110 2024-04-26T00:38:10.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff05e486640 5 monclient: authenticate success, global_id 144906 2024-04-26T00:38:10.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff05e486640 10 monclient: _renew_subs 2024-04-26T00:38:10.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff05e486640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:38:10.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff05e486640 1 -- 172.21.15.37:0/2147823750 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff05813bfe0 con 0x7ff0580a0110 2024-04-26T00:38:10.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff054ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:10.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff054ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:38:10.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff054ff9640 10 monclient: dump: 2024-04-26T00:38:10.525 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:10.525 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:10.525 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:10.525 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:10.525 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:10.525 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:10.525 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:10.525 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:10.525 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:10.525 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:10.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff055ffb640 4 set_mon_vals no callback set 2024-04-26T00:38:10.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff05e486640 10 monclient: _renew_subs 2024-04-26T00:38:10.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff05e486640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:38:10.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff05e486640 1 -- 172.21.15.37:0/2147823750 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff05813d1b0 con 0x7ff0580a0110 2024-04-26T00:38:10.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff05e486640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:10.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff05e486640 1 --2- 172.21.15.37:0/2147823750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff058139980 0x7ff058139d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:10.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff05e486640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:10.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff05e486640 1 -- 172.21.15.37:0/2147823750 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff05804f510 con 0x7ff058139980 2024-04-26T00:38:10.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff054ff9640 1 -- 172.21.15.37:0/2147823750 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff04c01a070 con 0x7ff0580a0110 2024-04-26T00:38:10.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff054ff9640 1 --2- 172.21.15.37:0/2147823750 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff044035ed0 0x7ff044038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:10.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff057fff640 1 --2- 172.21.15.37:0/2147823750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff058139980 0x7ff058139d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:10.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff054ff9640 1 -- 172.21.15.37:0/2147823750 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(715..715 src has 370..715) ==== 3927+0+0 (secure 0 0 0) 0x7ff04c048370 con 0x7ff0580a0110 2024-04-26T00:38:10.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff057fff640 10 monclient: get_auth_request con 0x7ff058139980 auth_method 0 2024-04-26T00:38:10.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff057fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:10.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff057fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:10.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.515+0000 7ff057fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:10.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.519+0000 7ff057fff640 1 -- 172.21.15.37:0/2147823750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff058139980 msgr2=0x7ff058139d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:38:10.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.519+0000 7ff057fff640 1 -- 172.21.15.37:0/2147823750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff058139980 msgr2=0x7ff058139d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:38:10.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.519+0000 7ff057fff640 1 --2- 172.21.15.37:0/2147823750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff058139980 0x7ff058139d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:38:10.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.519+0000 7ff057fff640 1 --2- 172.21.15.37:0/2147823750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff058139980 0x7ff058139d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:38:10.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.519+0000 7ff05ca23640 1 --2- 172.21.15.37:0/2147823750 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff044035ed0 0x7ff044038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:10.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.519+0000 7ff05ca23640 10 monclient: get_auth_request con 0x7ff044035ed0 auth_method 0 2024-04-26T00:38:10.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.519+0000 7ff05ca23640 1 --2- 172.21.15.37:0/2147823750 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff044035ed0 0x7ff044038390 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ff05813f3d0 tx=0x7ff050009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:10.528 INFO:tasks.rados.rados.0.smithi045.stdout:3202: done (5 left) 2024-04-26T00:38:10.528 INFO:tasks.rados.rados.0.smithi045.stdout:3203: done (4 left) 2024-04-26T00:38:10.528 INFO:tasks.rados.rados.0.smithi045.stdout:3204: done (3 left) 2024-04-26T00:38:10.528 INFO:tasks.rados.rados.0.smithi045.stdout:3205: done (2 left) 2024-04-26T00:38:10.528 INFO:tasks.rados.rados.0.smithi045.stdout:3206: done (1 left) 2024-04-26T00:38:10.528 INFO:tasks.rados.rados.0.smithi045.stdout:3207: done (0 left) 2024-04-26T00:38:10.528 INFO:tasks.rados.rados.0.smithi045.stdout:3208: write oid 34 current snap is 348 2024-04-26T00:38:10.528 INFO:tasks.rados.rados.0.smithi045.stdout:3208: seq_num 1150 ranges {528050=505653,1612057=524731,2614324=567403} 2024-04-26T00:38:10.532 INFO:tasks.rados.rados.0.smithi045.stdout:3208: writing smithi04526579-34 from 528050 to 1033703 tid 1 2024-04-26T00:38:10.553 INFO:tasks.rados.rados.0.smithi045.stdout:3208: writing smithi04526579-34 from 1612057 to 2136788 tid 2 2024-04-26T00:38:10.569 INFO:tasks.rados.rados.0.smithi045.stdout:3208: writing smithi04526579-34 from 2614324 to 3181727 tid 3 2024-04-26T00:38:10.569 INFO:tasks.rados.rados.0.smithi045.stdout:3209: read oid 45 snap 330 2024-04-26T00:38:10.569 INFO:tasks.rados.rados.0.smithi045.stdout:3209: expect (ObjNum 1062 snap 318 seq_num 1062) 2024-04-26T00:38:10.569 INFO:tasks.rados.rados.0.smithi045.stdout:3210: write (excl) oid 14 current snap is 348 2024-04-26T00:38:10.569 INFO:tasks.rados.rados.0.smithi045.stdout:3210: seq_num 1151 ranges {544857=580902,1670539=668389,3041747=303110} 2024-04-26T00:38:10.584 INFO:tasks.rados.rados.0.smithi045.stdout:3210: writing smithi04526579-14 from 544857 to 1125759 tid 1 2024-04-26T00:38:10.596 INFO:tasks.rados.rados.0.smithi045.stdout:3210: writing smithi04526579-14 from 1670539 to 2338928 tid 2 2024-04-26T00:38:10.605 INFO:tasks.rados.rados.0.smithi045.stdout:3210: writing smithi04526579-14 from 3041747 to 3344857 tid 3 2024-04-26T00:38:10.605 INFO:tasks.rados.rados.0.smithi045.stdout:3211: snap_create 2024-04-26T00:38:10.605 INFO:tasks.rados.rados.0.smithi045.stdout:3208: finishing write tid 1 to smithi04526579-34 2024-04-26T00:38:10.605 INFO:tasks.rados.rados.0.smithi045.stdout:3208: finishing write tid 2 to smithi04526579-34 2024-04-26T00:38:10.605 INFO:tasks.rados.rados.0.smithi045.stdout:3208: finishing write tid 3 to smithi04526579-34 2024-04-26T00:38:10.606 INFO:tasks.rados.rados.0.smithi045.stdout:3209: expect (ObjNum 1062 snap 318 seq_num 1062) 2024-04-26T00:38:10.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.723+0000 7ff057fff640 1 --2- 172.21.15.37:0/2147823750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff058139980 0x7ff058139d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:10.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.723+0000 7ff057fff640 10 monclient: get_auth_request con 0x7ff058139980 auth_method 0 2024-04-26T00:38:10.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.723+0000 7ff057fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:10.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.723+0000 7ff057fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:10.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.723+0000 7ff057fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:38:10.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.723+0000 7ff057fff640 1 -- 172.21.15.37:0/2147823750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff058139980 msgr2=0x7ff058139d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:38:10.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.723+0000 7ff057fff640 1 -- 172.21.15.37:0/2147823750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff058139980 msgr2=0x7ff058139d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:38:10.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.723+0000 7ff057fff640 1 --2- 172.21.15.37:0/2147823750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff058139980 0x7ff058139d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:38:10.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:10.723+0000 7ff057fff640 1 --2- 172.21.15.37:0/2147823750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff058139980 0x7ff058139d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:38:10.878 INFO:tasks.rados.rados.0.smithi045.stdout:3208: finishing write tid 4 to smithi04526579-34 2024-04-26T00:38:10.878 INFO:tasks.rados.rados.0.smithi045.stdout:3208: finishing write tid 5 to smithi04526579-34 2024-04-26T00:38:10.879 INFO:tasks.rados.rados.0.smithi045.stdout:3208: oid 34 updating version 0 to 2351 2024-04-26T00:38:10.879 INFO:tasks.rados.rados.0.smithi045.stdout:3208: oid 34 updating version 2351 to 2352 2024-04-26T00:38:10.879 INFO:tasks.rados.rados.0.smithi045.stdout:3208: oid 34 version 2352 is already newer than 2349 2024-04-26T00:38:10.879 INFO:tasks.rados.rados.0.smithi045.stdout:3208: oid 34 version 2352 is already newer than 2350 2024-04-26T00:38:10.879 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 2352 (ObjNum 1150 snap 348 seq_num 1150) dirty exists 2024-04-26T00:38:10.879 INFO:tasks.rados.rados.0.smithi045.stdout:3208: left oid 34 (ObjNum 1150 snap 348 seq_num 1150) 2024-04-26T00:38:10.879 INFO:tasks.rados.rados.0.smithi045.stdout:3210: finishing write tid 1 to smithi04526579-14 2024-04-26T00:38:10.879 INFO:tasks.rados.rados.0.smithi045.stdout:3210: finishing write tid 2 to smithi04526579-14 2024-04-26T00:38:10.879 INFO:tasks.rados.rados.0.smithi045.stdout:3210: finishing write tid 3 to smithi04526579-14 2024-04-26T00:38:10.879 INFO:tasks.rados.rados.0.smithi045.stdout:3210: finishing write tid 4 to smithi04526579-14 2024-04-26T00:38:10.879 INFO:tasks.rados.rados.0.smithi045.stdout:3210: finishing write tid 5 to smithi04526579-14 2024-04-26T00:38:10.879 INFO:tasks.rados.rados.0.smithi045.stdout:3210: oid 14 updating version 0 to 1198 2024-04-26T00:38:10.879 INFO:tasks.rados.rados.0.smithi045.stdout:3210: oid 14 version 1198 is already newer than 1196 2024-04-26T00:38:10.879 INFO:tasks.rados.rados.0.smithi045.stdout:3210: oid 14 version 1198 is already newer than 1197 2024-04-26T00:38:10.880 INFO:tasks.rados.rados.0.smithi045.stdout:3210: oid 14 updating version 1198 to 1199 2024-04-26T00:38:10.880 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 1199 (ObjNum 1151 snap 348 seq_num 1151) dirty exists 2024-04-26T00:38:10.880 INFO:tasks.rados.rados.0.smithi045.stdout:3210: left oid 14 (ObjNum 1151 snap 348 seq_num 1151) 2024-04-26T00:38:11.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.127+0000 7ff057fff640 1 --2- 172.21.15.37:0/2147823750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff058139980 0x7ff058139d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:11.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.127+0000 7ff057fff640 10 monclient: get_auth_request con 0x7ff058139980 auth_method 0 2024-04-26T00:38:11.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.127+0000 7ff057fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:11.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.127+0000 7ff057fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:11.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.127+0000 7ff057fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:38:11.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.127+0000 7ff057fff640 1 -- 172.21.15.37:0/2147823750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff058139980 msgr2=0x7ff058139d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:38:11.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.127+0000 7ff057fff640 1 -- 172.21.15.37:0/2147823750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff058139980 msgr2=0x7ff058139d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:38:11.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.127+0000 7ff057fff640 1 --2- 172.21.15.37:0/2147823750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff058139980 0x7ff058139d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:38:11.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.127+0000 7ff057fff640 1 --2- 172.21.15.37:0/2147823750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff058139980 0x7ff058139d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:38:11.452 INFO:tasks.rados.rados.0.smithi045.stdout:3208: done (3 left) 2024-04-26T00:38:11.452 INFO:tasks.rados.rados.0.smithi045.stdout:3209: done (2 left) 2024-04-26T00:38:11.452 INFO:tasks.rados.rados.0.smithi045.stdout:3210: done (1 left) 2024-04-26T00:38:11.453 INFO:tasks.rados.rados.0.smithi045.stdout:3211: done (0 left) 2024-04-26T00:38:11.453 INFO:tasks.rados.rados.0.smithi045.stdout:3212: snap_remove snap 348 2024-04-26T00:38:11.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.931+0000 7ff057fff640 1 --2- 172.21.15.37:0/2147823750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff058139980 0x7ff058139d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:11.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.931+0000 7ff057fff640 10 monclient: get_auth_request con 0x7ff058139980 auth_method 0 2024-04-26T00:38:11.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.931+0000 7ff057fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:11.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.931+0000 7ff057fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:11.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.931+0000 7ff057fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:38:11.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.931+0000 7ff057fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:11.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.931+0000 7ff057fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:11.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.931+0000 7ff057fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:11.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.931+0000 7ff057fff640 10 monclient(hunting): handle_auth_done global_id 224098 payload 824 2024-04-26T00:38:11.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.931+0000 7ff057fff640 1 -- 172.21.15.37:0/2147823750 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff058073280 con 0x7ff058139980 2024-04-26T00:38:11.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.935+0000 7ff057fff640 1 -- 172.21.15.37:0/2147823750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff058139980 msgr2=0x7ff058139d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:11.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.935+0000 7ff057fff640 1 --2- 172.21.15.37:0/2147823750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff058139980 0x7ff058139d60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff04801ca70 tx=0x7ff048018f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:11.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.935+0000 7ff054ff9640 1 -- 172.21.15.37:0/2147823750 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0480b0040 con 0x7ff058139980 2024-04-26T00:38:11.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.935+0000 7ff054ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:11.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.935+0000 7ff054ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:38:11.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.935+0000 7ff054ff9640 10 monclient: dump: 2024-04-26T00:38:11.933 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:11.933 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:11.933 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:11.933 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:11.933 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:11.933 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:11.933 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:11.933 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:11.933 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:11.933 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:11.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.935+0000 7ff054ff9640 1 -- 172.21.15.37:0/2147823750 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff0480b1040 con 0x7ff058139980 2024-04-26T00:38:11.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.935+0000 7ff054ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:11.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.935+0000 7ff054ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:38:11.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.935+0000 7ff054ff9640 1 -- 172.21.15.37:0/2147823750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff058139980 msgr2=0x7ff058139d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:11.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.935+0000 7ff054ff9640 1 --2- 172.21.15.37:0/2147823750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff058139980 0x7ff058139d60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff04801ca70 tx=0x7ff048018f70 comp rx=0 tx=0).stop 2024-04-26T00:38:11.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.935+0000 7ff054ff9640 1 -- 172.21.15.37:0/2147823750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff058139980 msgr2=0x7ff058139d60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:38:11.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.935+0000 7ff054ff9640 1 --2- 172.21.15.37:0/2147823750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff058139980 0x7ff058139d60 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff04801ca70 tx=0x7ff048018f70 comp rx=0 tx=0).stop 2024-04-26T00:38:11.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.943+0000 7ff05e486640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:38:11.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.943+0000 7ff05e486640 1 --2- 172.21.15.37:0/2147823750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff05813a2a0 0x7ff058133ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:11.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.943+0000 7ff05e486640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:11.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.943+0000 7ff05e486640 1 -- 172.21.15.37:0/2147823750 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff0581308b0 con 0x7ff05813a2a0 2024-04-26T00:38:11.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.943+0000 7ff057fff640 1 --2- 172.21.15.37:0/2147823750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff05813a2a0 0x7ff058133ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:11.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.943+0000 7ff057fff640 10 monclient: get_auth_request con 0x7ff05813a2a0 auth_method 0 2024-04-26T00:38:11.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.943+0000 7ff057fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:11.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.943+0000 7ff057fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:11.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.943+0000 7ff057fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:11.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.943+0000 7ff057fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:11.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.943+0000 7ff057fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:11.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.943+0000 7ff057fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:11.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.943+0000 7ff057fff640 10 monclient(hunting): handle_auth_done global_id 224101 payload 824 2024-04-26T00:38:11.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.943+0000 7ff057fff640 1 -- 172.21.15.37:0/2147823750 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff05806ba50 con 0x7ff05813a2a0 2024-04-26T00:38:11.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.947+0000 7ff057fff640 1 -- 172.21.15.37:0/2147823750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff05813a2a0 msgr2=0x7ff058133ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:11.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.947+0000 7ff057fff640 1 --2- 172.21.15.37:0/2147823750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff05813a2a0 0x7ff058133ee0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff0580a19d0 tx=0x7ff048009240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:11.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.947+0000 7ff054ff9640 1 -- 172.21.15.37:0/2147823750 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0480c5040 con 0x7ff05813a2a0 2024-04-26T00:38:11.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.947+0000 7ff054ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:11.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.947+0000 7ff054ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:38:11.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.947+0000 7ff054ff9640 10 monclient: dump: 2024-04-26T00:38:11.945 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:11.945 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:11.945 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:11.945 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:11.945 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:11.945 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:11.945 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:11.945 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:11.946 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:11.946 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:11.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.947+0000 7ff054ff9640 1 -- 172.21.15.37:0/2147823750 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7ff0480c6610 con 0x7ff05813a2a0 2024-04-26T00:38:11.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.947+0000 7ff054ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:38:11.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.947+0000 7ff054ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:38:11.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.947+0000 7ff054ff9640 1 -- 172.21.15.37:0/2147823750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff05813a2a0 msgr2=0x7ff058133ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:11.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.947+0000 7ff054ff9640 1 --2- 172.21.15.37:0/2147823750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff05813a2a0 0x7ff058133ee0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff0580a19d0 tx=0x7ff048009240 comp rx=0 tx=0).stop 2024-04-26T00:38:11.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.947+0000 7ff054ff9640 1 -- 172.21.15.37:0/2147823750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff05813a2a0 msgr2=0x7ff058133ee0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:38:11.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.947+0000 7ff054ff9640 1 --2- 172.21.15.37:0/2147823750 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff05813a2a0 0x7ff058133ee0 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff0580a19d0 tx=0x7ff048009240 comp rx=0 tx=0).stop 2024-04-26T00:38:11.947 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:38:11.947 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:38:11.947 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:38:11.947 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:38:11.947 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 138, 2024-04-26T00:38:11.947 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 5607, 2024-04-26T00:38:11.947 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:38:11.947 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:38:11.947 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:38:11.948 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:38:11.948 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:38:11.948 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 2, 2024-04-26T00:38:11.948 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:38:11.948 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:38:11.948 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:38:11.948 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:38:11.948 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:38:11.948 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:38:11.948 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:38:11.948 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:38:11.949 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:38:11.949 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:38:11.949 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:38:11.949 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:38:11.949 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:38:11.949 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:38:11.949 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:38:11.949 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:38:11.949 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:38:11.949 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:38:11.949 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:38:11.949 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:38:11.949 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:38:11.949 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:38:11.950 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:38:11.950 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:38:11.950 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:38:11.950 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:38:11.950 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:38:11.950 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:38:11.950 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:11.950 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:11.950 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:38:11.950 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:38:11.950 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:38:11.950 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:38:11.950 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:38:11.950 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:38:11.950 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:38:11.951 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:38:11.951 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:38:11.951 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:38:11.951 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:38:11.951 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:38:11.951 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:38:11.951 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:38:11.951 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:38:11.951 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:38:11.951 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:38:11.951 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:38:11.951 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:38:11.951 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:38:11.951 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:38:11.951 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:38:11.952 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:38:11.952 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:38:11.952 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:38:11.952 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:38:11.952 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:38:11.952 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:38:11.952 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:38:11.952 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:38:11.952 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:11.952 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:38:11.952 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:11.952 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:38:11.952 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:38:11.952 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:38:11.952 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:38:11.953 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:11.953 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:38:11.953 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:38:11.953 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:11.953 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:11.953 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:11.953 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:38:11.953 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:38:11.953 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:11.953 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:11.953 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:11.953 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:11.953 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:38:11.953 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:38:11.953 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:38:11.954 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:38:11.954 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:38:11.954 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:11.954 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:11.954 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:38:11.954 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:38:11.954 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:38:11.954 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:38:11.954 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:11.954 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:38:11.954 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:38:11.954 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:11.954 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:11.954 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:11.954 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:38:11.955 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:38:11.955 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:11.955 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:11.955 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:11.955 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:11.955 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:38:11.955 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:38:11.955 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:38:11.955 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:38:11.955 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:38:11.955 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:11.955 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:11.955 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:38:11.955 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:38:11.955 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:38:11.956 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:38:11.956 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:11.956 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:38:11.956 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:38:11.956 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:11.956 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:11.956 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:11.956 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:38:11.956 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:38:11.956 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:11.956 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:11.956 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:11.956 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:11.956 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:38:11.956 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:38:11.957 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:38:11.957 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:38:11.957 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:38:11.957 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:11.957 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:11.957 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:11.957 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:38:11.957 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:38:11.957 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:11.957 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:38:11.957 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:38:11.957 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:38:11.957 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:11.957 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:38:11.957 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:38:11.958 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:11.958 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:38:11.958 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:38:11.958 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:38:11.958 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:11.958 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:11.958 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:11.958 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:38:11.958 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:38:11.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.947+0000 7ff05e486640 1 -- 172.21.15.37:0/2147823750 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff044035ed0 msgr2=0x7ff044038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:11.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.947+0000 7ff05e486640 1 --2- 172.21.15.37:0/2147823750 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff044035ed0 0x7ff044038390 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ff05813f3d0 tx=0x7ff050009040 comp rx=0 tx=0).stop 2024-04-26T00:38:11.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.947+0000 7ff05e486640 10 monclient: shutdown 2024-04-26T00:38:11.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.947+0000 7ff05e486640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:38:11.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.947+0000 7ff05e486640 1 -- 172.21.15.37:0/2147823750 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0580a0110 msgr2=0x7ff05812fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:11.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.947+0000 7ff05e486640 1 --2- 172.21.15.37:0/2147823750 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0580a0110 0x7ff05812fe30 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7ff04c00e7e0 tx=0x7ff04c00eca0 comp rx=0 tx=0).stop 2024-04-26T00:38:11.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.947+0000 7ff05ca23640 1 -- 172.21.15.37:0/2147823750 reap_dead start 2024-04-26T00:38:11.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.947+0000 7ff05ca23640 1 -- 172.21.15.37:0/2147823750 reap_dead start 2024-04-26T00:38:11.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.947+0000 7ff05e486640 1 -- 172.21.15.37:0/2147823750 shutdown_connections 2024-04-26T00:38:11.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.947+0000 7ff05e486640 1 -- 172.21.15.37:0/2147823750 >> 172.21.15.37:0/2147823750 conn(0x7ff0580a1d50 msgr2=0x7ff0580a4ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:38:11.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.947+0000 7ff05e486640 1 -- 172.21.15.37:0/2147823750 shutdown_connections 2024-04-26T00:38:11.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:11.947+0000 7ff05e486640 1 -- 172.21.15.37:0/2147823750 wait complete. 2024-04-26T00:38:11.968 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:38:11.968 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-26T00:38:11.968 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-26T00:38:12.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.091+0000 7f711db8a640 10 monclient: get_monmap_and_config 2024-04-26T00:38:12.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.091+0000 7f711db8a640 10 monclient: build_initial_monmap 2024-04-26T00:38:12.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.091+0000 7f711db8a640 10 monclient: monmap: 2024-04-26T00:38:12.091 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:38:12.091 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:12.091 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:38:12.095116+0000 2024-04-26T00:38:12.091 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:38:12.095116+0000 2024-04-26T00:38:12.091 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:38:12.091 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:12.091 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:38:12.091 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:38:12.091 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:38:12.092 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:38:12.092 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:38:12.092 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:12.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.091+0000 7f711db8a640 1 Processor -- start 2024-04-26T00:38:12.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.095+0000 7f711db8a640 1 -- start start 2024-04-26T00:38:12.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.095+0000 7f711db8a640 10 monclient: init 2024-04-26T00:38:12.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.095+0000 7f711db8a640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:12.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.095+0000 7f711db8a640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-04-26T00:38:12.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.095+0000 7f711db8a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7118136eb0 addr v1:172.21.15.45:6789/0 2024-04-26T00:38:12.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.095+0000 7f711db8a640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f71181379a0 0x7f711813c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:12.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.095+0000 7f711db8a640 10 monclient(hunting): picked mon.noname-c con 0x7f71181379a0 addr v2:172.21.15.45:3300/0 2024-04-26T00:38:12.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.095+0000 7f711db8a640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f711813c990 0x7f711813ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:12.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.095+0000 7f711db8a640 10 monclient(hunting): picked mon.noname-a con 0x7f711813c990 addr v2:172.21.15.37:3300/0 2024-04-26T00:38:12.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.095+0000 7f711db8a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f711809b8c0 con 0x7f7118136eb0 2024-04-26T00:38:12.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.095+0000 7f711db8a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:12.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.095+0000 7f711db8a640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f711809c7d0 con 0x7f711813c990 2024-04-26T00:38:12.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.095+0000 7f711db8a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:12.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.095+0000 7f711db8a640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f711809b300 con 0x7f71181379a0 2024-04-26T00:38:12.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.095+0000 7f711db8a640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:12.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.095+0000 7f711db8a640 10 monclient(hunting): authenticate will time out at 2073.400744s 2024-04-26T00:38:12.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.095+0000 7f7117fff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f711813c990 0x7f711813ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:12.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.095+0000 7f7117fff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f711813c990 0x7f711813ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44638/0 (socket says 172.21.15.37:44638) 2024-04-26T00:38:12.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.095+0000 7f7117fff640 1 -- 172.21.15.37:0/3024915045 learned_addr learned my addr 172.21.15.37:0/3024915045 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:38:12.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.095+0000 7f7117fff640 10 monclient(hunting): get_auth_request con 0x7f711813c990 auth_method 0 2024-04-26T00:38:12.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.095+0000 7f7117fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:12.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.095+0000 7f7117fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:12.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.095+0000 7f7117fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:12.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.095+0000 7f7117fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:12.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.095+0000 7f7117fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:12.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.095+0000 7f7117fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:12.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.095+0000 7f7116ffd640 1 --2- 172.21.15.37:0/3024915045 >> v2:172.21.15.45:3300/0 conn(0x7f71181379a0 0x7f711813c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:12.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.095+0000 7f7116ffd640 10 monclient(hunting): get_auth_request con 0x7f71181379a0 auth_method 0 2024-04-26T00:38:12.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.095+0000 7f7116ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:12.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.095+0000 7f7116ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:12.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.095+0000 7f7116ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:12.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.095+0000 7f7117fff640 10 monclient(hunting): handle_auth_done global_id 144909 payload 274 2024-04-26T00:38:12.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.095+0000 7f7117fff640 1 -- 172.21.15.37:0/3024915045 >> v2:172.21.15.45:3300/0 conn(0x7f71181379a0 msgr2=0x7f711813c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:38:12.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.095+0000 7f7117fff640 1 --2- 172.21.15.37:0/3024915045 >> v2:172.21.15.45:3300/0 conn(0x7f71181379a0 0x7f711813c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:12.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.095+0000 7f7117fff640 1 -- 172.21.15.37:0/3024915045 >> v1:172.21.15.45:6789/0 conn(0x7f7118136eb0 legacy=0x7f7118137290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:38:12.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.095+0000 7f7117fff640 10 monclient: _finish_hunting 0 2024-04-26T00:38:12.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.095+0000 7f7117fff640 1 monclient: found mon.noname-a 2024-04-26T00:38:12.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.095+0000 7f7117fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:12.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.095+0000 7f7117fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:38:12.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.095+0000 7f7117fff640 1 -- 172.21.15.37:0/3024915045 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f711809b5e0 con 0x7f711813c990 2024-04-26T00:38:12.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.099+0000 7f7117fff640 1 -- 172.21.15.37:0/3024915045 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f711813c990 msgr2=0x7f711813ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:12.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.099+0000 7f7117fff640 1 --2- 172.21.15.37:0/3024915045 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f711813c990 0x7f711813ed80 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f71180a05c0 tx=0x7f711001c690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e892ff33f31921bb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:12.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.099+0000 7f71167fc640 1 -- 172.21.15.37:0/3024915045 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71100ac020 con 0x7f711813c990 2024-04-26T00:38:12.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.099+0000 7f71167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:12.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.099+0000 7f71167fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:38:12.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.099+0000 7f71167fc640 10 monclient: dump: 2024-04-26T00:38:12.098 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:12.098 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:12.098 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:12.098 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:12.098 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:12.098 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:12.098 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:12.098 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:12.098 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:12.098 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:12.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.099+0000 7f71167fc640 10 monclient: _finish_auth 0 2024-04-26T00:38:12.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.099+0000 7f71167fc640 10 monclient: _check_auth_tickets 2024-04-26T00:38:12.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.099+0000 7f71167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:12.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.099+0000 7f71167fc640 1 -- 172.21.15.37:0/3024915045 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71100a6470 con 0x7f711813c990 2024-04-26T00:38:12.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.099+0000 7f71167fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:38:12.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.099+0000 7f71167fc640 1 -- 172.21.15.37:0/3024915045 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71100a6840 con 0x7f711813c990 2024-04-26T00:38:12.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.099+0000 7f71167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:12.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.099+0000 7f71167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:38:12.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.099+0000 7f71167fc640 10 monclient: dump: 2024-04-26T00:38:12.099 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:12.099 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:12.099 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:12.099 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:12.099 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:12.099 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:12.099 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:12.099 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:12.099 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:12.100 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:12.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.099+0000 7f711db8a640 5 monclient: authenticate success, global_id 144909 2024-04-26T00:38:12.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.099+0000 7f711db8a640 10 monclient: get_monmap_and_config success 2024-04-26T00:38:12.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.099+0000 7f711db8a640 4 set_mon_vals no callback set 2024-04-26T00:38:12.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.099+0000 7f711db8a640 10 monclient: shutdown 2024-04-26T00:38:12.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.099+0000 7f711db8a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:38:12.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.099+0000 7f711db8a640 1 -- 172.21.15.37:0/3024915045 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f711813c990 msgr2=0x7f711813ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:12.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.099+0000 7f711db8a640 1 --2- 172.21.15.37:0/3024915045 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f711813c990 0x7f711813ed80 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f71180a05c0 tx=0x7f711001c690 comp rx=0 tx=0).stop 2024-04-26T00:38:12.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.099+0000 7f711db8a640 1 -- 172.21.15.37:0/3024915045 shutdown_connections 2024-04-26T00:38:12.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.099+0000 7f711db8a640 1 --2- 172.21.15.37:0/3024915045 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f711813c990 0x7f711813ed80 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:12.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.099+0000 7f711db8a640 1 --2- 172.21.15.37:0/3024915045 >> v2:172.21.15.45:3300/0 conn(0x7f71181379a0 0x7f711813c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:12.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.099+0000 7f711db8a640 1 -- 172.21.15.37:0/3024915045 >> 172.21.15.37:0/3024915045 conn(0x7f71180a6300 msgr2=0x7f71180a6710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:38:12.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.099+0000 7f711db8a640 1 -- 172.21.15.37:0/3024915045 shutdown_connections 2024-04-26T00:38:12.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.099+0000 7f711db8a640 1 -- 172.21.15.37:0/3024915045 wait complete. 2024-04-26T00:38:12.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.099+0000 7f711db8a640 10 monclient: build_initial_monmap 2024-04-26T00:38:12.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.099+0000 7f711db8a640 10 monclient: monmap: 2024-04-26T00:38:12.105 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:38:12.105 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:38:12.105 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:38:12.105 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:38:12.105 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:38:12.105 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:38:12.105 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:38:12.105 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:38:12.105 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:38:12.106 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:12.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.099+0000 7f711db8a640 1 Processor -- start 2024-04-26T00:38:12.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.099+0000 7f711db8a640 1 -- start start 2024-04-26T00:38:12.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.099+0000 7f711db8a640 10 monclient: init 2024-04-26T00:38:12.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f711db8a640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:12.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f711db8a640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:38:12.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f711db8a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7118136eb0 0x7f711813b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:12.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f711db8a640 10 monclient(hunting): picked mon.noname-c con 0x7f7118136eb0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:38:12.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f711db8a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71181379a0 0x7f7118133f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:12.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f711db8a640 10 monclient(hunting): picked mon.noname-b con 0x7f71181379a0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:38:12.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f711db8a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f711813c990 0x7f7118134490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:12.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f711db8a640 10 monclient(hunting): picked mon.noname-a con 0x7f711813c990 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:38:12.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f711db8a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:12.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f711db8a640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f711809abf0 con 0x7f711813c990 2024-04-26T00:38:12.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f711db8a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:12.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f711db8a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7118141ae0 con 0x7f71181379a0 2024-04-26T00:38:12.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f711db8a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:12.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f711db8a640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f711809ae10 con 0x7f7118136eb0 2024-04-26T00:38:12.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f711db8a640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:12.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f711db8a640 10 monclient(hunting): authenticate will time out at 2073.408491s 2024-04-26T00:38:12.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f7117fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f711813c990 0x7f7118134490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:12.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f71177fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7118136eb0 0x7f711813b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:12.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f7117fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f711813c990 0x7f7118134490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44654/0 (socket says 172.21.15.37:44654) 2024-04-26T00:38:12.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f7117fff640 1 -- 172.21.15.37:0/4033339127 learned_addr learned my addr 172.21.15.37:0/4033339127 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:38:12.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f7117fff640 10 monclient(hunting): get_auth_request con 0x7f711813c990 auth_method 0 2024-04-26T00:38:12.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f7117fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:12.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f7117fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:12.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f7117fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:12.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f71177fe640 10 monclient(hunting): get_auth_request con 0x7f7118136eb0 auth_method 0 2024-04-26T00:38:12.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f71177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:12.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f71177fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:12.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f71177fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:12.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f7117fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:12.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f7117fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:12.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f7117fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:12.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f71177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:12.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f71177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:12.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f71177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:12.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f7116ffd640 1 --2- 172.21.15.37:0/4033339127 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71181379a0 0x7f7118133f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:12.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f7116ffd640 10 monclient(hunting): get_auth_request con 0x7f71181379a0 auth_method 0 2024-04-26T00:38:12.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f7116ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:12.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f7116ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:12.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f7116ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:12.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f7117fff640 10 monclient(hunting): handle_auth_done global_id 144912 payload 824 2024-04-26T00:38:12.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f7117fff640 1 -- 172.21.15.37:0/4033339127 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7118136eb0 msgr2=0x7f711813b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:12.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f7117fff640 1 --2- 172.21.15.37:0/4033339127 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7118136eb0 0x7f711813b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:12.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f7117fff640 1 -- 172.21.15.37:0/4033339127 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71181379a0 msgr2=0x7f7118133f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:12.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f7117fff640 1 --2- 172.21.15.37:0/4033339127 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71181379a0 0x7f7118133f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:12.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f7117fff640 10 monclient: _finish_hunting 0 2024-04-26T00:38:12.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f7117fff640 1 monclient: found mon.noname-a 2024-04-26T00:38:12.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f7117fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:12.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f7117fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:38:12.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f7117fff640 1 -- 172.21.15.37:0/4033339127 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71181349d0 con 0x7f711813c990 2024-04-26T00:38:12.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f71177fe640 1 --2- 172.21.15.37:0/4033339127 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7118136eb0 0x7f711813b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:38:12.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f7117fff640 1 -- 172.21.15.37:0/4033339127 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f711813c990 msgr2=0x7f7118134490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:12.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f7117fff640 1 --2- 172.21.15.37:0/4033339127 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f711813c990 0x7f7118134490 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f711001a2c0 tx=0x7f7110019430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:12.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f70fffff640 1 -- 172.21.15.37:0/4033339127 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71100ac040 con 0x7f711813c990 2024-04-26T00:38:12.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f70fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:12.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f70fffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:38:12.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f70fffff640 10 monclient: dump: 2024-04-26T00:38:12.110 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:12.110 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:12.110 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:12.110 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:12.110 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:12.110 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:12.110 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:12.110 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:12.110 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:12.110 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:12.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f70fffff640 10 monclient: _finish_auth 0 2024-04-26T00:38:12.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f70fffff640 10 monclient: _check_auth_tickets 2024-04-26T00:38:12.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f70fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:12.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f70fffff640 1 -- 172.21.15.37:0/4033339127 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71100a7650 con 0x7f711813c990 2024-04-26T00:38:12.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f70fffff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:38:12.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f711db8a640 5 monclient: authenticate success, global_id 144912 2024-04-26T00:38:12.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f711db8a640 10 monclient: _renew_subs 2024-04-26T00:38:12.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f70fffff640 1 -- 172.21.15.37:0/4033339127 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71100a7a40 con 0x7f711813c990 2024-04-26T00:38:12.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f711db8a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:38:12.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f711db8a640 1 -- 172.21.15.37:0/4033339127 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f711813a8c0 con 0x7f711813c990 2024-04-26T00:38:12.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f71157fa640 4 set_mon_vals no callback set 2024-04-26T00:38:12.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f70fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:12.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f70fffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:38:12.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f70fffff640 10 monclient: dump: 2024-04-26T00:38:12.111 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:12.111 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:12.111 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:12.112 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:12.112 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:12.112 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:12.112 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:12.112 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:12.112 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:12.112 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:12.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f711db8a640 10 monclient: _renew_subs 2024-04-26T00:38:12.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f711db8a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:38:12.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.103+0000 7f711db8a640 1 -- 172.21.15.37:0/4033339127 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f711813bac0 con 0x7f711813c990 2024-04-26T00:38:12.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.107+0000 7f711db8a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:12.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.107+0000 7f711db8a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:12.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.107+0000 7f711db8a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:38:12.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.107+0000 7f711db8a640 1 -- 172.21.15.37:0/4033339127 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7118066f40 con 0x7f711813c990 2024-04-26T00:38:12.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.107+0000 7f70fffff640 1 -- 172.21.15.37:0/4033339127 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f71100af020 con 0x7f711813c990 2024-04-26T00:38:12.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.107+0000 7f70fffff640 1 --2- 172.21.15.37:0/4033339127 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7104035ed0 0x7f7104038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:12.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.107+0000 7f70fffff640 1 -- 172.21.15.37:0/4033339127 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(716..716 src has 370..716) ==== 3899+0+0 (secure 0 0 0) 0x7f71100e2fd0 con 0x7f711813c990 2024-04-26T00:38:12.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.107+0000 7f71177fe640 1 --2- 172.21.15.37:0/4033339127 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7104035ed0 0x7f7104038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:12.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.107+0000 7f71177fe640 10 monclient: get_auth_request con 0x7f7104035ed0 auth_method 0 2024-04-26T00:38:12.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.111+0000 7f71177fe640 1 --2- 172.21.15.37:0/4033339127 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7104035ed0 0x7f7104038390 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f7108007fd0 tx=0x7f7108007e40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:12.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.111+0000 7f70fffff640 1 -- 172.21.15.37:0/4033339127 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f711809b8c0 con 0x7f711813c990 2024-04-26T00:38:12.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.111+0000 7f70fffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:12.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.111+0000 7f70fffff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:38:12.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.335+0000 7f711db8a640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-26T00:38:12.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.335+0000 7f711db8a640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-26T00:38:12.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.335+0000 7f711db8a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:38:12.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.335+0000 7f711db8a640 1 -- 172.21.15.37:0/4033339127 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f711813af10 con 0x7f711813c990 2024-04-26T00:38:12.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.335+0000 7f70fffff640 1 -- 172.21.15.37:0/4033339127 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f71100c4440 con 0x7f711813c990 2024-04-26T00:38:12.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.335+0000 7f70fffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-26T00:38:12.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.335+0000 7f70fffff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:38:12.336 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.339+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:12.336 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.339+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=25,config=2,health=12,kv=9,logm=52} crc {auth=2037212684,config=692619927,health=3911963131,kv=3748211275,logm=3439642938}) 2024-04-26T00:38:12.336 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.339+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=24,config=2,health=11,kv=10,logm=53} crc {auth=350218817,config=692619927,health=2697108806,kv=1034996044,logm=1584380337}) 2024-04-26T00:38:12.336 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.339+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:12.337 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.339+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=25,config=2,health=12,kv=9,logm=52} crc {auth=2037212684,config=692619927,health=3911963131,kv=3748211275,logm=3439642938}) 2024-04-26T00:38:12.337 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.339+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=25,config=2,health=12,kv=10,logm=51} crc {auth=2273609603,config=692619927,health=3911963131,kv=1034996044,logm=2629802837}) 2024-04-26T00:38:12.339 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.339+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:12.339 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.339+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=109415380}) 2024-04-26T00:38:12.339 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.339+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3266431808}) 2024-04-26T00:38:12.340 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.339+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:12.340 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.339+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=109415380}) 2024-04-26T00:38:12.340 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.339+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4015415561}) 2024-04-26T00:38:12.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.343+0000 7f711db8a640 1 -- 172.21.15.37:0/4033339127 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7104035ed0 msgr2=0x7f7104038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:12.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.343+0000 7f711db8a640 1 --2- 172.21.15.37:0/4033339127 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7104035ed0 0x7f7104038390 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f7108007fd0 tx=0x7f7108007e40 comp rx=0 tx=0).stop 2024-04-26T00:38:12.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.343+0000 7f711db8a640 10 monclient: shutdown 2024-04-26T00:38:12.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.343+0000 7f711db8a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:38:12.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.343+0000 7f711db8a640 1 -- 172.21.15.37:0/4033339127 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f711813c990 msgr2=0x7f7118134490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:12.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.343+0000 7f711db8a640 1 --2- 172.21.15.37:0/4033339127 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f711813c990 0x7f7118134490 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f711001a2c0 tx=0x7f7110019430 comp rx=0 tx=0).stop 2024-04-26T00:38:12.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.343+0000 7f711db8a640 1 -- 172.21.15.37:0/4033339127 shutdown_connections 2024-04-26T00:38:12.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.343+0000 7f711db8a640 1 --2- 172.21.15.37:0/4033339127 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7104035ed0 0x7f7104038390 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:12.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.343+0000 7f711db8a640 1 --2- 172.21.15.37:0/4033339127 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f711813c990 0x7f7118134490 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:12.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.343+0000 7f711db8a640 1 --2- 172.21.15.37:0/4033339127 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71181379a0 0x7f7118133f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:12.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.343+0000 7f711db8a640 1 --2- 172.21.15.37:0/4033339127 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7118136eb0 0x7f711813b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:12.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.343+0000 7f711db8a640 1 -- 172.21.15.37:0/4033339127 >> 172.21.15.37:0/4033339127 conn(0x7f71180a6300 msgr2=0x7f7118135b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:38:12.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.343+0000 7f711db8a640 1 -- 172.21.15.37:0/4033339127 shutdown_connections 2024-04-26T00:38:12.342 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:12.343+0000 7f711db8a640 1 -- 172.21.15.37:0/4033339127 wait complete. 2024-04-26T00:38:12.343 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.343+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:12.343 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.343+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1180864983}) 2024-04-26T00:38:12.343 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.343+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=845847109}) 2024-04-26T00:38:12.343 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.343+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:12.343 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.343+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1180864983}) 2024-04-26T00:38:12.343 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.343+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=862395389}) 2024-04-26T00:38:12.344 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.347+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:12.344 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.347+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=245752103}) 2024-04-26T00:38:12.344 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.347+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2888285350}) 2024-04-26T00:38:12.344 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.347+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:12.344 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.347+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=245752103}) 2024-04-26T00:38:12.344 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.347+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2329524649}) 2024-04-26T00:38:12.347 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.347+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:12.347 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.347+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2398797796}) 2024-04-26T00:38:12.347 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.347+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4015954937}) 2024-04-26T00:38:12.347 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.347+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:12.347 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.347+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2398797796}) 2024-04-26T00:38:12.347 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.347+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2112009170}) 2024-04-26T00:38:12.349 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.351+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:12.349 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.351+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3886377466}) 2024-04-26T00:38:12.349 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.351+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3205737639}) 2024-04-26T00:38:12.349 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.351+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:12.349 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.351+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3886377466}) 2024-04-26T00:38:12.350 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.351+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3185569343}) 2024-04-26T00:38:12.351 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.351+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:12.351 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.351+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1157875589}) 2024-04-26T00:38:12.351 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.351+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=623231392}) 2024-04-26T00:38:12.351 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.351+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:12.351 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.351+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1157875589}) 2024-04-26T00:38:12.351 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.351+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4284468469}) 2024-04-26T00:38:12.359 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-26T00:38:12.448 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.451+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:12.448 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.451+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=580150627}) 2024-04-26T00:38:12.448 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.451+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3636871539}) 2024-04-26T00:38:12.449 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.451+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:12.449 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.451+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=580150627}) 2024-04-26T00:38:12.449 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.451+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2828160240}) 2024-04-26T00:38:12.454 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.455+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:12.454 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.455+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=459131618}) 2024-04-26T00:38:12.454 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.455+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1385200398}) 2024-04-26T00:38:12.454 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.455+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:12.454 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.455+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=459131618}) 2024-04-26T00:38:12.454 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.455+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1907264078}) 2024-04-26T00:38:12.465 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.467+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:12.465 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.467+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=856070236}) 2024-04-26T00:38:12.465 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.467+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=216972530}) 2024-04-26T00:38:12.465 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.467+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:12.465 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.467+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=856070236}) 2024-04-26T00:38:12.465 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.467+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4053056074}) 2024-04-26T00:38:12.468 INFO:tasks.rados.rados.0.smithi045.stdout:3212: done (0 left) 2024-04-26T00:38:12.468 INFO:tasks.rados.rados.0.smithi045.stdout:3213: read oid 27 snap 335 2024-04-26T00:38:12.468 INFO:tasks.rados.rados.0.smithi045.stdout:3213: expect (ObjNum 1051 snap 317 seq_num 1051) 2024-04-26T00:38:12.468 INFO:tasks.rados.rados.0.smithi045.stdout:3214: copy_from oid 36 from oid 22 current snap is 349 2024-04-26T00:38:12.468 INFO:tasks.rados.rados.0.smithi045.stdout:3215: read oid 24 snap -1 2024-04-26T00:38:12.468 INFO:tasks.rados.rados.0.smithi045.stdout:3215: expect (ObjNum 1140 snap 345 seq_num 1140) 2024-04-26T00:38:12.468 INFO:tasks.rados.rados.0.smithi045.stdout:3216: copy_from oid 12 from oid 48 current snap is 349 2024-04-26T00:38:12.469 INFO:tasks.rados.rados.0.smithi045.stdout:3217: snap_remove snap 343 2024-04-26T00:38:12.883 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.883+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:12.884 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.883+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1362765662}) 2024-04-26T00:38:12.884 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.883+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3693276397}) 2024-04-26T00:38:12.884 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.883+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:12.884 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.883+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1362765662}) 2024-04-26T00:38:12.884 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.883+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=62279854}) 2024-04-26T00:38:12.885 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.887+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:12.886 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.887+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3222963514}) 2024-04-26T00:38:12.886 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.887+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3352970264}) 2024-04-26T00:38:12.886 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.887+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:12.886 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.887+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3222963514}) 2024-04-26T00:38:12.886 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:12.887+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1651063568}) 2024-04-26T00:38:13.001 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.003+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.002 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.003+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4046316787}) 2024-04-26T00:38:13.002 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.003+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1884862697}) 2024-04-26T00:38:13.002 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.003+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.002 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.003+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4046316787}) 2024-04-26T00:38:13.002 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.003+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1648438186}) 2024-04-26T00:38:13.219 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.219+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.220 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.219+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3671122654}) 2024-04-26T00:38:13.220 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.219+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2542349865}) 2024-04-26T00:38:13.220 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.219+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.220 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.219+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3671122654}) 2024-04-26T00:38:13.220 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.219+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=637721939}) 2024-04-26T00:38:13.222 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.223+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.222 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.223+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2437135304}) 2024-04-26T00:38:13.222 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.223+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4120317243}) 2024-04-26T00:38:13.222 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.223+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.222 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.223+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2437135304}) 2024-04-26T00:38:13.223 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.223+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=926164338}) 2024-04-26T00:38:13.224 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.227+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.224 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.227+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3738429462}) 2024-04-26T00:38:13.225 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.227+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=977484844}) 2024-04-26T00:38:13.225 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.227+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.225 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.227+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3738429462}) 2024-04-26T00:38:13.225 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.227+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2039705350}) 2024-04-26T00:38:13.227 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.227+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.227 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.227+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2638013761}) 2024-04-26T00:38:13.227 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.227+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=871159287}) 2024-04-26T00:38:13.227 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.227+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.227 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.227+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2638013761}) 2024-04-26T00:38:13.227 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.227+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=9519560}) 2024-04-26T00:38:13.229 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.231+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.229 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.231+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2288429512}) 2024-04-26T00:38:13.229 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.231+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1735418808}) 2024-04-26T00:38:13.230 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.231+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.230 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.231+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2288429512}) 2024-04-26T00:38:13.230 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.231+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3782592541}) 2024-04-26T00:38:13.232 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.235+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.232 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.235+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1299199101}) 2024-04-26T00:38:13.233 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.235+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1276175277}) 2024-04-26T00:38:13.233 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.235+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.233 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.235+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1299199101}) 2024-04-26T00:38:13.233 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.235+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=259853405}) 2024-04-26T00:38:13.235 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.235+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.235 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.235+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3450856293}) 2024-04-26T00:38:13.235 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.235+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4173568812}) 2024-04-26T00:38:13.235 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.235+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.235 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.235+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3450856293}) 2024-04-26T00:38:13.235 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.235+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1135164512}) 2024-04-26T00:38:13.238 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.239+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.238 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.239+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1106240071}) 2024-04-26T00:38:13.238 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.239+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4042573856}) 2024-04-26T00:38:13.238 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.239+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.238 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.239+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1106240071}) 2024-04-26T00:38:13.239 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.239+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=643991819}) 2024-04-26T00:38:13.240 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.243+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.240 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.243+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4193131430}) 2024-04-26T00:38:13.240 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.243+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2666538735}) 2024-04-26T00:38:13.240 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.243+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.240 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.243+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4193131430}) 2024-04-26T00:38:13.240 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.243+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1632435787}) 2024-04-26T00:38:13.242 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.243+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.243 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.243+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3718683300}) 2024-04-26T00:38:13.243 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.243+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=457956754}) 2024-04-26T00:38:13.243 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.243+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.243 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.243+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3718683300}) 2024-04-26T00:38:13.243 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.243+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=635135477}) 2024-04-26T00:38:13.245 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.247+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.245 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.247+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2777825487}) 2024-04-26T00:38:13.245 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.247+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4232884840}) 2024-04-26T00:38:13.245 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.247+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.245 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.247+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2777825487}) 2024-04-26T00:38:13.245 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.247+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2433414751}) 2024-04-26T00:38:13.249 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.247+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.249 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.251+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2246381999}) 2024-04-26T00:38:13.249 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.251+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=505343646}) 2024-04-26T00:38:13.249 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.251+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.249 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.251+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2246381999}) 2024-04-26T00:38:13.249 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.251+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=77812724}) 2024-04-26T00:38:13.251 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.251+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.251 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.251+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3963892465}) 2024-04-26T00:38:13.251 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.251+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2003211545}) 2024-04-26T00:38:13.251 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.251+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.251 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.251+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3963892465}) 2024-04-26T00:38:13.251 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.251+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1167511860}) 2024-04-26T00:38:13.253 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.255+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.253 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.255+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=397470977}) 2024-04-26T00:38:13.253 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.255+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3439646498}) 2024-04-26T00:38:13.253 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.255+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.253 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.255+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=397470977}) 2024-04-26T00:38:13.254 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.255+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1843710018}) 2024-04-26T00:38:13.255 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.255+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.255 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.255+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1532758740}) 2024-04-26T00:38:13.255 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.255+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2759161346}) 2024-04-26T00:38:13.255 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.255+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.256 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.255+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1532758740}) 2024-04-26T00:38:13.256 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.255+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2141300673}) 2024-04-26T00:38:13.258 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.259+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.258 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.259+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2831658538}) 2024-04-26T00:38:13.259 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.259+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=104547949}) 2024-04-26T00:38:13.259 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.259+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.259 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.259+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2831658538}) 2024-04-26T00:38:13.259 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.259+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=941685536}) 2024-04-26T00:38:13.260 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.263+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.261 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.263+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1117630934}) 2024-04-26T00:38:13.261 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.263+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=830464310}) 2024-04-26T00:38:13.261 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.263+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.261 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.263+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1117630934}) 2024-04-26T00:38:13.261 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.263+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2548321096}) 2024-04-26T00:38:13.263 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.263+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.263 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.263+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=284308209}) 2024-04-26T00:38:13.263 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.263+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2609029013}) 2024-04-26T00:38:13.263 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.263+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.264 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.263+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=284308209}) 2024-04-26T00:38:13.264 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.263+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3354272961}) 2024-04-26T00:38:13.266 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.267+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.266 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.267+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3554003631}) 2024-04-26T00:38:13.266 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.267+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4037123587}) 2024-04-26T00:38:13.266 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.267+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.266 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.267+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3554003631}) 2024-04-26T00:38:13.266 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.267+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=380043579}) 2024-04-26T00:38:13.268 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.271+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.268 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.271+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1040467164}) 2024-04-26T00:38:13.268 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.271+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3931638679}) 2024-04-26T00:38:13.268 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.271+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.268 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.271+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1040467164}) 2024-04-26T00:38:13.268 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.271+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=801600062}) 2024-04-26T00:38:13.271 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.271+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.271 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.271+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=883321197}) 2024-04-26T00:38:13.271 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.271+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2006197916}) 2024-04-26T00:38:13.271 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.271+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.271 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.271+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=883321197}) 2024-04-26T00:38:13.271 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.271+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=32760144}) 2024-04-26T00:38:13.273 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.275+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.274 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.275+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2337212547}) 2024-04-26T00:38:13.274 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.275+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1835583530}) 2024-04-26T00:38:13.274 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.275+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.274 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.275+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2337212547}) 2024-04-26T00:38:13.274 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.275+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=294961645}) 2024-04-26T00:38:13.276 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.275+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.276 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.279+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2790393297}) 2024-04-26T00:38:13.276 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.279+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=39882230}) 2024-04-26T00:38:13.276 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.279+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.276 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.279+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2790393297}) 2024-04-26T00:38:13.276 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.279+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1202425182}) 2024-04-26T00:38:13.278 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.279+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.278 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.279+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4169660917}) 2024-04-26T00:38:13.278 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.279+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3567249145}) 2024-04-26T00:38:13.278 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.279+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.279 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.279+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4169660917}) 2024-04-26T00:38:13.279 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.279+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4029106369}) 2024-04-26T00:38:13.291 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.291+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.292 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.291+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3703001275}) 2024-04-26T00:38:13.292 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.291+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2382504908}) 2024-04-26T00:38:13.292 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.291+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.292 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.291+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3703001275}) 2024-04-26T00:38:13.292 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.291+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=533096922}) 2024-04-26T00:38:13.294 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.295+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.294 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.295+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=757345048}) 2024-04-26T00:38:13.294 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.295+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4110120487}) 2024-04-26T00:38:13.294 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.295+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.294 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.295+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=757345048}) 2024-04-26T00:38:13.294 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.295+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1459776969}) 2024-04-26T00:38:13.297 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.299+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.297 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.299+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3947276811}) 2024-04-26T00:38:13.297 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.299+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=340032536}) 2024-04-26T00:38:13.297 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.299+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.297 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.299+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3947276811}) 2024-04-26T00:38:13.297 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.299+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=997961884}) 2024-04-26T00:38:13.299 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.299+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.299 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.299+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1412274208}) 2024-04-26T00:38:13.299 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.299+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=599715917}) 2024-04-26T00:38:13.299 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.299+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.300 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.299+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1412274208}) 2024-04-26T00:38:13.300 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.299+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2664906998}) 2024-04-26T00:38:13.302 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.303+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.302 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.303+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2256966042}) 2024-04-26T00:38:13.302 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.303+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=384694914}) 2024-04-26T00:38:13.302 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.303+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.302 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.303+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2256966042}) 2024-04-26T00:38:13.302 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.303+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=822601378}) 2024-04-26T00:38:13.304 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.307+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.304 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.307+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1834658087}) 2024-04-26T00:38:13.304 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.307+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3816465016}) 2024-04-26T00:38:13.305 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.307+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.305 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.307+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1834658087}) 2024-04-26T00:38:13.305 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.307+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2924490386}) 2024-04-26T00:38:13.307 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.307+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.307 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.307+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=651993629}) 2024-04-26T00:38:13.307 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.307+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1788266313}) 2024-04-26T00:38:13.307 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.307+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.307 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.307+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=651993629}) 2024-04-26T00:38:13.307 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.307+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1103446281}) 2024-04-26T00:38:13.309 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.311+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.309 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.311+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=918311001}) 2024-04-26T00:38:13.310 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.311+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2919118514}) 2024-04-26T00:38:13.310 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.311+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.310 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.311+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=918311001}) 2024-04-26T00:38:13.310 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.311+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1923859012}) 2024-04-26T00:38:13.312 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.315+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.312 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.315+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=609755371}) 2024-04-26T00:38:13.312 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.315+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1089917207}) 2024-04-26T00:38:13.312 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.315+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.312 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.315+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=609755371}) 2024-04-26T00:38:13.312 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.315+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1880135870}) 2024-04-26T00:38:13.314 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.315+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.314 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.315+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=921954499}) 2024-04-26T00:38:13.315 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.315+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3476463719}) 2024-04-26T00:38:13.315 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.315+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.315 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.315+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=921954499}) 2024-04-26T00:38:13.315 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.315+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1517534401}) 2024-04-26T00:38:13.317 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.319+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.317 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.319+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1989743144}) 2024-04-26T00:38:13.317 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.319+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3175275281}) 2024-04-26T00:38:13.317 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.319+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.317 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.319+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1989743144}) 2024-04-26T00:38:13.317 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.319+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1375441583}) 2024-04-26T00:38:13.319 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.319+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.319 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.319+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3789344063}) 2024-04-26T00:38:13.319 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.319+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3760475124}) 2024-04-26T00:38:13.320 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.319+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.320 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.319+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3789344063}) 2024-04-26T00:38:13.320 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.319+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1978266068}) 2024-04-26T00:38:13.321 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.323+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.322 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.323+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2088423044}) 2024-04-26T00:38:13.322 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.323+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3739876886}) 2024-04-26T00:38:13.322 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.323+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.322 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.323+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2088423044}) 2024-04-26T00:38:13.322 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.323+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3274636450}) 2024-04-26T00:38:13.324 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.327+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.324 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.327+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2940397957}) 2024-04-26T00:38:13.324 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.327+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1130717651}) 2024-04-26T00:38:13.324 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.327+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.324 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.327+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2940397957}) 2024-04-26T00:38:13.324 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.327+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=881407967}) 2024-04-26T00:38:13.326 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.327+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.326 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.327+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1426940246}) 2024-04-26T00:38:13.327 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.327+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=198509073}) 2024-04-26T00:38:13.327 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.327+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.327 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.327+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1426940246}) 2024-04-26T00:38:13.327 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.327+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2786565383}) 2024-04-26T00:38:13.329 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.331+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.329 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.331+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1273999564}) 2024-04-26T00:38:13.329 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.331+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2649231743}) 2024-04-26T00:38:13.329 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.331+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.329 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.331+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1273999564}) 2024-04-26T00:38:13.329 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.331+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1305276048}) 2024-04-26T00:38:13.331 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.331+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.331 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.331+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=302973306}) 2024-04-26T00:38:13.331 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.331+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2617958811}) 2024-04-26T00:38:13.331 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.331+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.331 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.331+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=302973306}) 2024-04-26T00:38:13.331 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.331+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1896817658}) 2024-04-26T00:38:13.333 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.335+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.333 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.335+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1886780045}) 2024-04-26T00:38:13.333 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.335+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3802772979}) 2024-04-26T00:38:13.333 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.335+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.334 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.335+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1886780045}) 2024-04-26T00:38:13.334 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.335+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2498716551}) 2024-04-26T00:38:13.336 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.335+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.336 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.335+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2772914211}) 2024-04-26T00:38:13.336 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.335+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1729859313}) 2024-04-26T00:38:13.336 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.339+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.336 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.339+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2772914211}) 2024-04-26T00:38:13.336 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.339+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3901580436}) 2024-04-26T00:38:13.338 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.339+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.338 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.339+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=960043217}) 2024-04-26T00:38:13.338 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.339+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4052844962}) 2024-04-26T00:38:13.338 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.339+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.338 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.339+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=960043217}) 2024-04-26T00:38:13.338 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.339+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2003368182}) 2024-04-26T00:38:13.340 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.343+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.340 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.343+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1883107710}) 2024-04-26T00:38:13.340 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.343+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=799510330}) 2024-04-26T00:38:13.340 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.343+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.340 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.343+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1883107710}) 2024-04-26T00:38:13.340 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.343+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4069112528}) 2024-04-26T00:38:13.342 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.343+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.343 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.343+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3118623966}) 2024-04-26T00:38:13.343 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.343+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1722465791}) 2024-04-26T00:38:13.343 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.343+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.343 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.343+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3118623966}) 2024-04-26T00:38:13.343 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.343+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3767938304}) 2024-04-26T00:38:13.345 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.347+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.345 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.347+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2354052400}) 2024-04-26T00:38:13.345 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.347+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4108897060}) 2024-04-26T00:38:13.345 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.347+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.345 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.347+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2354052400}) 2024-04-26T00:38:13.345 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.347+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2619330526}) 2024-04-26T00:38:13.347 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.347+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.347 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.347+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3699068599}) 2024-04-26T00:38:13.347 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.347+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2655001023}) 2024-04-26T00:38:13.347 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.347+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.347 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.347+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3699068599}) 2024-04-26T00:38:13.347 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.347+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=192498328}) 2024-04-26T00:38:13.349 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.351+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.349 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.351+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3699920793}) 2024-04-26T00:38:13.349 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.351+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1849736415}) 2024-04-26T00:38:13.349 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.351+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.349 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.351+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3699920793}) 2024-04-26T00:38:13.349 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.351+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=232904307}) 2024-04-26T00:38:13.351 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.351+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.351 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.351+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3768062370}) 2024-04-26T00:38:13.351 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.351+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3585815884}) 2024-04-26T00:38:13.352 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.351+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.352 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.351+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3768062370}) 2024-04-26T00:38:13.352 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.351+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=523217471}) 2024-04-26T00:38:13.353 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.355+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.353 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.355+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3191617673}) 2024-04-26T00:38:13.354 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.355+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1521410030}) 2024-04-26T00:38:13.354 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.355+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.354 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.355+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3191617673}) 2024-04-26T00:38:13.354 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.355+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3614263927}) 2024-04-26T00:38:13.356 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.359+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.357 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.359+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1183952785}) 2024-04-26T00:38:13.357 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.359+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=750169729}) 2024-04-26T00:38:13.358 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.359+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.358 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.359+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1183952785}) 2024-04-26T00:38:13.358 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.359+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3909437291}) 2024-04-26T00:38:13.358 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.359+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.358 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.359+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=77129714}) 2024-04-26T00:38:13.358 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.359+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1903340549}) 2024-04-26T00:38:13.359 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.359+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.359 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.359+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=77129714}) 2024-04-26T00:38:13.359 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.359+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2542974818}) 2024-04-26T00:38:13.361 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:38:13.361 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:38:13.362 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.363+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.362 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.363+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=728206516}) 2024-04-26T00:38:13.362 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.363+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=632006330}) 2024-04-26T00:38:13.362 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.363+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.362 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.363+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=728206516}) 2024-04-26T00:38:13.362 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.363+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1392495965}) 2024-04-26T00:38:13.363 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.363+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.363 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.363+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4064808627}) 2024-04-26T00:38:13.363 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.363+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2063691479}) 2024-04-26T00:38:13.363 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.363+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.364 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.363+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4064808627}) 2024-04-26T00:38:13.365 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.363+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4135779050}) 2024-04-26T00:38:13.366 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.367+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.366 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.367+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4282712268}) 2024-04-26T00:38:13.366 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.367+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4036808423}) 2024-04-26T00:38:13.366 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.367+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.366 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.367+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4282712268}) 2024-04-26T00:38:13.366 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.367+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1154917453}) 2024-04-26T00:38:13.369 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.371+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.369 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.371+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=860110640}) 2024-04-26T00:38:13.369 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.371+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=944092064}) 2024-04-26T00:38:13.369 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.371+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.369 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.371+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=860110640}) 2024-04-26T00:38:13.369 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.371+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=814720693}) 2024-04-26T00:38:13.371 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.371+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.372 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.371+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1113446399}) 2024-04-26T00:38:13.372 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.371+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2980798842}) 2024-04-26T00:38:13.372 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.371+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.372 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.371+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1113446399}) 2024-04-26T00:38:13.372 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.371+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2327867831}) 2024-04-26T00:38:13.375 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.375+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.375 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.375+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=68,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=9,monmap=1} crc {logm=660141510,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=1822068625,mon_config_key=2768526065,monmap=3249110452}) 2024-04-26T00:38:13.375 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.375+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=71,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=8} crc {logm=1498839460,mdsmap=2840952586,mgr=3713829143,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=4218702391,mon_config_key=1303549602}) 2024-04-26T00:38:13.375 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.375+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.375 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.375+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=68,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=9,monmap=1} crc {logm=660141510,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=1822068625,mon_config_key=2768526065,monmap=3249110452}) 2024-04-26T00:38:13.375 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.375+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=72,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=8} crc {logm=1637229735,mdsmap=2840952586,mgr=2196320830,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=3741098765,mon_config_key=348320670}) 2024-04-26T00:38:13.377 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.379+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.377 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.379+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=91} crc {monmap=3140514009,osd_metadata=3306121363,osd_pg_creating=420960034,osd_snap=3119648556}) 2024-04-26T00:38:13.377 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.379+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=91} crc {monmap=3140514009,osd_metadata=3306121363,osd_pg_creating=420960034,osd_snap=136908140}) 2024-04-26T00:38:13.377 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.379+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.377 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.379+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=91} crc {monmap=3140514009,osd_metadata=3306121363,osd_pg_creating=420960034,osd_snap=3119648556}) 2024-04-26T00:38:13.377 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.379+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {monmap=1,osd_metadata=6,osd_snap=93} crc {monmap=1228700967,osd_metadata=3306121363,osd_snap=1555867726}) 2024-04-26T00:38:13.379 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.379+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.379 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.379+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=43,osdmap=57} crc {osd_snap=2701544370,osdmap=2373139291}) 2024-04-26T00:38:13.380 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.379+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=42,osdmap=58} crc {osd_snap=901592437,osdmap=2038081329}) 2024-04-26T00:38:13.380 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.379+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.380 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.379+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=43,osdmap=57} crc {osd_snap=2701544370,osdmap=2373139291}) 2024-04-26T00:38:13.380 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.379+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=42,osdmap=58} crc {osd_snap=479972809,osdmap=3151750189}) 2024-04-26T00:38:13.382 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.383+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.382 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.383+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=267740841}) 2024-04-26T00:38:13.382 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.383+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=599967615}) 2024-04-26T00:38:13.383 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.383+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.383 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.383+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=267740841}) 2024-04-26T00:38:13.383 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.383+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1206300403}) 2024-04-26T00:38:13.385 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.387+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.386 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.387+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4064887562}) 2024-04-26T00:38:13.386 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.387+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=838107014}) 2024-04-26T00:38:13.386 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.387+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.386 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.387+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4064887562}) 2024-04-26T00:38:13.386 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.387+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3389117325}) 2024-04-26T00:38:13.387 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.387+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.388 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.387+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2875699740}) 2024-04-26T00:38:13.388 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.387+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3701214343}) 2024-04-26T00:38:13.388 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.387+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.388 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.387+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2875699740}) 2024-04-26T00:38:13.388 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.387+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3344344193}) 2024-04-26T00:38:13.392 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.391+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.392 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.391+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=523019882}) 2024-04-26T00:38:13.392 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.391+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2304378316}) 2024-04-26T00:38:13.393 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.391+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.393 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.391+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=523019882}) 2024-04-26T00:38:13.393 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.395+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3877482537}) 2024-04-26T00:38:13.404 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.407+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.404 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.407+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=54} crc {osdmap=2508780743}) 2024-04-26T00:38:13.405 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.407+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=57} crc {osdmap=4092584631}) 2024-04-26T00:38:13.405 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.407+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:38:13.405 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.407+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=54} crc {osdmap=2508780743}) 2024-04-26T00:38:13.405 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:38:13.407+0000 7f760209f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=53} crc {osdmap=2397590886}) 2024-04-26T00:38:13.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.503+0000 7facc82fa640 10 monclient: get_monmap_and_config 2024-04-26T00:38:13.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.503+0000 7facc82fa640 10 monclient: build_initial_monmap 2024-04-26T00:38:13.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.503+0000 7facc82fa640 10 monclient: monmap: 2024-04-26T00:38:13.502 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:38:13.503 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:13.503 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:38:13.505813+0000 2024-04-26T00:38:13.503 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:38:13.505813+0000 2024-04-26T00:38:13.503 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:38:13.503 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:13.503 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:38:13.503 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:38:13.503 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:38:13.503 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:38:13.504 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:38:13.504 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:13.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.503+0000 7facc82fa640 1 Processor -- start 2024-04-26T00:38:13.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.503+0000 7facc82fa640 1 -- start start 2024-04-26T00:38:13.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.503+0000 7facc82fa640 10 monclient: init 2024-04-26T00:38:13.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.503+0000 7facc82fa640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:13.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.503+0000 7facc82fa640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-04-26T00:38:13.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.503+0000 7facc82fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7facc0136e10 addr v1:172.21.15.37:6789/0 2024-04-26T00:38:13.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.503+0000 7facc82fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7facc0137900 addr v1:172.21.15.45:6789/0 2024-04-26T00:38:13.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.503+0000 7facc82fa640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7facc009c5b0 0x7facc013fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:13.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.503+0000 7facc82fa640 10 monclient(hunting): picked mon.noname-c con 0x7facc009c5b0 addr v2:172.21.15.45:3300/0 2024-04-26T00:38:13.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.503+0000 7facc82fa640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facc009b680 con 0x7facc0136e10 2024-04-26T00:38:13.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.503+0000 7facc82fa640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facc009bbd0 con 0x7facc0137900 2024-04-26T00:38:13.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.503+0000 7facc82fa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:13.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.503+0000 7facc82fa640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7facc009b8f0 con 0x7facc009c5b0 2024-04-26T00:38:13.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.503+0000 7facc82fa640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:13.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.503+0000 7facc82fa640 10 monclient(hunting): authenticate will time out at 2074.811726s 2024-04-26T00:38:13.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.503+0000 7facc6096640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7facc0136e10 0x7facc01371f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:51776/0 (socket says 172.21.15.37:51776) 2024-04-26T00:38:13.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.503+0000 7facc6096640 1 -- 172.21.15.37:0/860279727 learned_addr learned my addr 172.21.15.37:0/860279727 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:38:13.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.507+0000 7facc6897640 1 --2- 172.21.15.37:0/860279727 >> v2:172.21.15.45:3300/0 conn(0x7facc009c5b0 0x7facc013fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:13.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.507+0000 7facc6897640 10 monclient(hunting): get_auth_request con 0x7facc009c5b0 auth_method 0 2024-04-26T00:38:13.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.507+0000 7facc6897640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:13.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.507+0000 7facc6897640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:13.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.507+0000 7facc6897640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:13.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.507+0000 7facc6897640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:13.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.507+0000 7facc6897640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:13.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.507+0000 7facc6897640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:13.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.507+0000 7facc6096640 1 -- 172.21.15.37:0/860279727 >> v1:172.21.15.37:6789/0 conn(0x7facc0136e10 legacy=0x7facc01371f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:13.508 INFO:tasks.rados.rados.0.smithi045.stdout:3217: done (4 left) 2024-04-26T00:38:13.508 INFO:tasks.rados.rados.0.smithi045.stdout:3218: read oid 18 snap -1 2024-04-26T00:38:13.508 INFO:tasks.rados.rados.0.smithi045.stdout:3218: expect (ObjNum 1096 snap 325 seq_num 1096) 2024-04-26T00:38:13.508 INFO:tasks.rados.rados.0.smithi045.stdout:3219: snap_create 2024-04-26T00:38:13.508 INFO:tasks.rados.rados.0.smithi045.stdout:3215: expect (ObjNum 1140 snap 345 seq_num 1140) 2024-04-26T00:38:13.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.507+0000 7facc5895640 1 -- 172.21.15.37:0/860279727 >> v1:172.21.15.45:6789/0 conn(0x7facc0137900 legacy=0x7facc013c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:13.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.507+0000 7facc6897640 10 monclient(hunting): handle_auth_done global_id 224110 payload 274 2024-04-26T00:38:13.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.507+0000 7facc6897640 1 -- 172.21.15.37:0/860279727 >> v1:172.21.15.45:6789/0 conn(0x7facc0137900 legacy=0x7facc013c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:13.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.507+0000 7facc6897640 1 -- 172.21.15.37:0/860279727 >> v1:172.21.15.37:6789/0 conn(0x7facc0136e10 legacy=0x7facc01371f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:13.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.507+0000 7facc6897640 10 monclient: _finish_hunting 0 2024-04-26T00:38:13.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.507+0000 7facc6897640 1 monclient: found mon.noname-c 2024-04-26T00:38:13.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.507+0000 7facc6897640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:13.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.507+0000 7facc6897640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:38:13.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.507+0000 7facc6897640 1 -- 172.21.15.37:0/860279727 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facc00a5d90 con 0x7facc009c5b0 2024-04-26T00:38:13.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.511+0000 7facc6897640 1 -- 172.21.15.37:0/860279727 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facc009c5b0 msgr2=0x7facc013fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:13.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.511+0000 7facc6897640 1 --2- 172.21.15.37:0/860279727 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facc009c5b0 0x7facc013fa70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7facb8009e90 tx=0x7facb801c930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9820a58d2e29aafe server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:13.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.515+0000 7facc5094640 1 -- 172.21.15.37:0/860279727 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb80b2070 con 0x7facc009c5b0 2024-04-26T00:38:13.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.515+0000 7facc5094640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:13.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.515+0000 7facc5094640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:38:13.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.515+0000 7facc5094640 10 monclient: dump: 2024-04-26T00:38:13.513 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:13.513 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:13.514 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:13.514 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:13.514 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:13.514 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:13.514 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:13.514 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:13.514 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:13.514 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:13.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.515+0000 7facc5094640 10 monclient: _finish_auth 0 2024-04-26T00:38:13.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.515+0000 7facc5094640 10 monclient: _check_auth_tickets 2024-04-26T00:38:13.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.515+0000 7facc5094640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:13.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.515+0000 7facc5094640 1 -- 172.21.15.37:0/860279727 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facb80a59e0 con 0x7facc009c5b0 2024-04-26T00:38:13.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.515+0000 7facc5094640 10 monclient: handle_config config(0 keys) 2024-04-26T00:38:13.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.515+0000 7facc5094640 1 -- 172.21.15.37:0/860279727 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb80a5db0 con 0x7facc009c5b0 2024-04-26T00:38:13.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.515+0000 7facc5094640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:13.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.515+0000 7facc5094640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:38:13.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.515+0000 7facc5094640 10 monclient: dump: 2024-04-26T00:38:13.515 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:13.515 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:13.515 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:13.515 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:13.515 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:13.515 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:13.515 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:13.516 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:13.516 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:13.516 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:13.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.515+0000 7facc82fa640 5 monclient: authenticate success, global_id 224110 2024-04-26T00:38:13.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.515+0000 7facc82fa640 10 monclient: get_monmap_and_config success 2024-04-26T00:38:13.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.515+0000 7facc82fa640 4 set_mon_vals no callback set 2024-04-26T00:38:13.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.515+0000 7facc82fa640 10 monclient: shutdown 2024-04-26T00:38:13.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.515+0000 7facc82fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:38:13.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.515+0000 7facc82fa640 1 -- 172.21.15.37:0/860279727 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facc009c5b0 msgr2=0x7facc013fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:13.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.515+0000 7facc82fa640 1 --2- 172.21.15.37:0/860279727 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facc009c5b0 0x7facc013fa70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7facb8009e90 tx=0x7facb801c930 comp rx=0 tx=0).stop 2024-04-26T00:38:13.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.515+0000 7facc82fa640 1 -- 172.21.15.37:0/860279727 shutdown_connections 2024-04-26T00:38:13.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.515+0000 7facc82fa640 1 --2- 172.21.15.37:0/860279727 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facc009c5b0 0x7facc013fa70 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:13.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.515+0000 7facc82fa640 1 -- 172.21.15.37:0/860279727 >> 172.21.15.37:0/860279727 conn(0x7facc00a64e0 msgr2=0x7facc00a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:38:13.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.515+0000 7facc82fa640 1 -- 172.21.15.37:0/860279727 shutdown_connections 2024-04-26T00:38:13.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.515+0000 7facc82fa640 1 -- 172.21.15.37:0/860279727 wait complete. 2024-04-26T00:38:13.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc82fa640 10 monclient: build_initial_monmap 2024-04-26T00:38:13.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc82fa640 10 monclient: monmap: 2024-04-26T00:38:13.518 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:38:13.518 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:38:13.518 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:38:13.518 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:38:13.518 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:38:13.518 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:38:13.518 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:38:13.519 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:38:13.519 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:38:13.519 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:13.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc82fa640 1 Processor -- start 2024-04-26T00:38:13.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc82fa640 1 -- start start 2024-04-26T00:38:13.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc82fa640 10 monclient: init 2024-04-26T00:38:13.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc82fa640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:13.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc82fa640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:38:13.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc82fa640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7facc009c5b0 0x7facc006e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:13.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc82fa640 10 monclient(hunting): picked mon.noname-c con 0x7facc009c5b0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:38:13.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc82fa640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facc0136e10 0x7facc013bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:13.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc82fa640 10 monclient(hunting): picked mon.noname-b con 0x7facc0136e10 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:38:13.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc82fa640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7facc0137900 0x7facc013c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:13.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc82fa640 10 monclient(hunting): picked mon.noname-a con 0x7facc0137900 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:38:13.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc82fa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:13.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc82fa640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7facc0142b60 con 0x7facc0137900 2024-04-26T00:38:13.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc82fa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:13.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc82fa640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7facc01429e0 con 0x7facc0136e10 2024-04-26T00:38:13.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc82fa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:13.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc82fa640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7facc0142ce0 con 0x7facc009c5b0 2024-04-26T00:38:13.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc82fa640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:13.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc82fa640 10 monclient(hunting): authenticate will time out at 2074.825934s 2024-04-26T00:38:13.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc6096640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7facc009c5b0 0x7facc006e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:13.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc6096640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7facc009c5b0 0x7facc006e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:52986/0 (socket says 172.21.15.37:52986) 2024-04-26T00:38:13.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc6096640 1 -- 172.21.15.37:0/1774067298 learned_addr learned my addr 172.21.15.37:0/1774067298 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:38:13.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc6096640 10 monclient(hunting): get_auth_request con 0x7facc009c5b0 auth_method 0 2024-04-26T00:38:13.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc6096640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:13.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc6096640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:13.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc6096640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:13.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc6897640 1 --2- 172.21.15.37:0/1774067298 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7facc0137900 0x7facc013c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:13.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc6897640 10 monclient(hunting): get_auth_request con 0x7facc0137900 auth_method 0 2024-04-26T00:38:13.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc6897640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:13.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc6897640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:13.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc6897640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:13.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc6096640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:13.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc6096640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:13.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc6096640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:13.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc6897640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:13.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc6897640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:13.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc5895640 1 --2- 172.21.15.37:0/1774067298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facc0136e10 0x7facc013bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:13.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc6897640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:13.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc5895640 10 monclient(hunting): get_auth_request con 0x7facc0136e10 auth_method 0 2024-04-26T00:38:13.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc5895640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:13.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc5895640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:13.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc5895640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:13.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc6096640 10 monclient(hunting): handle_auth_done global_id 204305 payload 824 2024-04-26T00:38:13.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc6096640 1 -- 172.21.15.37:0/1774067298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facc0136e10 msgr2=0x7facc013bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:13.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc6096640 1 --2- 172.21.15.37:0/1774067298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facc0136e10 0x7facc013bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:13.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc6096640 1 -- 172.21.15.37:0/1774067298 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7facc0137900 msgr2=0x7facc013c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:13.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc6096640 1 --2- 172.21.15.37:0/1774067298 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7facc0137900 0x7facc013c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:13.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc6096640 10 monclient: _finish_hunting 0 2024-04-26T00:38:13.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc6096640 1 monclient: found mon.noname-c 2024-04-26T00:38:13.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc6096640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:13.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc6096640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:38:13.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc6096640 1 -- 172.21.15.37:0/1774067298 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facc006ebd0 con 0x7facc009c5b0 2024-04-26T00:38:13.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.519+0000 7facc6897640 1 --2- 172.21.15.37:0/1774067298 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7facc0137900 0x7facc013c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:38:13.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.523+0000 7facc6096640 1 -- 172.21.15.37:0/1774067298 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7facc009c5b0 msgr2=0x7facc006e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:13.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.523+0000 7facc6096640 1 --2- 172.21.15.37:0/1774067298 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7facc009c5b0 0x7facc006e690 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7facbc00c460 tx=0x7facbc00c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:13.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.523+0000 7facb6ffd640 1 -- 172.21.15.37:0/1774067298 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facbc017070 con 0x7facc009c5b0 2024-04-26T00:38:13.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.523+0000 7facb6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:13.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.523+0000 7facb6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:38:13.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.523+0000 7facb6ffd640 10 monclient: dump: 2024-04-26T00:38:13.528 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:13.528 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:13.528 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:13.528 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:13.528 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:13.529 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:13.529 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:13.529 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:13.529 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:13.529 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:13.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.523+0000 7facb6ffd640 10 monclient: _finish_auth 0 2024-04-26T00:38:13.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.523+0000 7facb6ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:38:13.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.523+0000 7facb6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:13.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.523+0000 7facb6ffd640 1 -- 172.21.15.37:0/1774067298 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facbc007070 con 0x7facc009c5b0 2024-04-26T00:38:13.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.523+0000 7facc82fa640 5 monclient: authenticate success, global_id 204305 2024-04-26T00:38:13.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.523+0000 7facc82fa640 10 monclient: _renew_subs 2024-04-26T00:38:13.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.523+0000 7facc82fa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:13.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.523+0000 7facc82fa640 1 -- 172.21.15.37:0/1774067298 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facc009aca0 con 0x7facc009c5b0 2024-04-26T00:38:13.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.523+0000 7facb6ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:38:13.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.523+0000 7facb6ffd640 1 -- 172.21.15.37:0/1774067298 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facbc007440 con 0x7facc009c5b0 2024-04-26T00:38:13.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.523+0000 7facb6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:13.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.523+0000 7facb6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:38:13.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.523+0000 7facb6ffd640 10 monclient: dump: 2024-04-26T00:38:13.530 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:13.530 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:13.530 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:13.530 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:13.530 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:13.530 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:13.530 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:13.531 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:13.531 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:13.531 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:13.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.523+0000 7facb7fff640 4 set_mon_vals no callback set 2024-04-26T00:38:13.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.523+0000 7facc82fa640 10 monclient: _renew_subs 2024-04-26T00:38:13.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.523+0000 7facc82fa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:13.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.523+0000 7facc82fa640 1 -- 172.21.15.37:0/1774067298 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7facc013fa60 con 0x7facc009c5b0 2024-04-26T00:38:13.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.523+0000 7facc82fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:13.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.523+0000 7facc82fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:13.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.523+0000 7facc82fa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:13.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.523+0000 7facb6ffd640 1 -- 172.21.15.37:0/1774067298 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7facbc01a070 con 0x7facc009c5b0 2024-04-26T00:38:13.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.523+0000 7facc82fa640 1 -- 172.21.15.37:0/1774067298 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facc0075910 con 0x7facc009c5b0 2024-04-26T00:38:13.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.523+0000 7facb6ffd640 1 --2- 172.21.15.37:0/1774067298 >> v2:172.21.15.45:6812/2379918623 conn(0x7faca4035e80 0x7faca4038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:13.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.523+0000 7facb6ffd640 1 -- 172.21.15.37:0/1774067298 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(718..718 src has 370..718) ==== 3959+0+0 (secure 0 0 0) 0x7facbc047e20 con 0x7facc009c5b0 2024-04-26T00:38:13.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.523+0000 7facc5895640 1 --2- 172.21.15.37:0/1774067298 >> v2:172.21.15.45:6812/2379918623 conn(0x7faca4035e80 0x7faca4038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:13.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.523+0000 7facc5895640 10 monclient: get_auth_request con 0x7faca4035e80 auth_method 0 2024-04-26T00:38:13.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.527+0000 7facc5895640 1 --2- 172.21.15.37:0/1774067298 >> v2:172.21.15.45:6812/2379918623 conn(0x7faca4035e80 0x7faca4038340 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7facc013d6f0 tx=0x7facb0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:13.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.527+0000 7facb6ffd640 1 -- 172.21.15.37:0/1774067298 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7facc006ebd0 con 0x7facc009c5b0 2024-04-26T00:38:13.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.527+0000 7facb6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:13.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.527+0000 7facb6ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:38:13.645 INFO:tasks.rados.rados.0.smithi045.stdout:3216: finishing copy_from racing read to smithi04526579-12 2024-04-26T00:38:13.646 INFO:tasks.rados.rados.0.smithi045.stdout:3216: finishing copy_from to smithi04526579-12 2024-04-26T00:38:13.646 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 2339 (ObjNum 1112 snap 331 seq_num 1112) dirty exists 2024-04-26T00:38:13.646 INFO:tasks.rados.rados.0.smithi045.stdout:3213: expect (ObjNum 1051 snap 317 seq_num 1051) 2024-04-26T00:38:13.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.755+0000 7facc82fa640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:38:13.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.755+0000 7facc82fa640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:38:13.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.755+0000 7facc82fa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:13.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.755+0000 7facc82fa640 1 -- 172.21.15.37:0/1774067298 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7facc00a8f50 con 0x7facc009c5b0 2024-04-26T00:38:13.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.759+0000 7facb6ffd640 1 -- 172.21.15.37:0/1774067298 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7facbc01a360 con 0x7facc009c5b0 2024-04-26T00:38:13.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.759+0000 7facb6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:38:13.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.759+0000 7facb6ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:38:13.757 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":138,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:38:13.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.759+0000 7facc82fa640 1 -- 172.21.15.37:0/1774067298 >> v2:172.21.15.45:6812/2379918623 conn(0x7faca4035e80 msgr2=0x7faca4038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:13.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.759+0000 7facc82fa640 1 --2- 172.21.15.37:0/1774067298 >> v2:172.21.15.45:6812/2379918623 conn(0x7faca4035e80 0x7faca4038340 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7facc013d6f0 tx=0x7facb0009040 comp rx=0 tx=0).stop 2024-04-26T00:38:13.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.759+0000 7facc82fa640 10 monclient: shutdown 2024-04-26T00:38:13.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.759+0000 7facc82fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:38:13.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.759+0000 7facc82fa640 1 -- 172.21.15.37:0/1774067298 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7facc009c5b0 msgr2=0x7facc006e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:13.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.759+0000 7facc82fa640 1 --2- 172.21.15.37:0/1774067298 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7facc009c5b0 0x7facc006e690 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7facbc00c460 tx=0x7facbc00c920 comp rx=0 tx=0).stop 2024-04-26T00:38:13.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.759+0000 7facc82fa640 1 -- 172.21.15.37:0/1774067298 shutdown_connections 2024-04-26T00:38:13.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.759+0000 7facc82fa640 1 --2- 172.21.15.37:0/1774067298 >> v2:172.21.15.45:6812/2379918623 conn(0x7faca4035e80 0x7faca4038340 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:13.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.759+0000 7facc82fa640 1 --2- 172.21.15.37:0/1774067298 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7facc0137900 0x7facc013c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:13.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.759+0000 7facc82fa640 1 --2- 172.21.15.37:0/1774067298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7facc0136e10 0x7facc013bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:13.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.759+0000 7facc82fa640 1 --2- 172.21.15.37:0/1774067298 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7facc009c5b0 0x7facc006e690 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:13.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.759+0000 7facc82fa640 1 -- 172.21.15.37:0/1774067298 >> 172.21.15.37:0/1774067298 conn(0x7facc00a64e0 msgr2=0x7facc00a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:38:13.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.759+0000 7facc82fa640 1 -- 172.21.15.37:0/1774067298 shutdown_connections 2024-04-26T00:38:13.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.759+0000 7facc82fa640 1 -- 172.21.15.37:0/1774067298 wait complete. 2024-04-26T00:38:13.784 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:38:13.784 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-26T00:38:13.784 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:38:13.784 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:38:13.864 INFO:tasks.rados.rados.0.smithi045.stdout:3214: finishing copy_from racing read to smithi04526579-36 2024-04-26T00:38:13.865 INFO:tasks.rados.rados.0.smithi045.stdout:3214: finishing copy_from to smithi04526579-36 2024-04-26T00:38:13.865 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 2428 (ObjNum 1135 snap 343 seq_num 1135) dirty exists 2024-04-26T00:38:13.865 INFO:tasks.rados.rados.0.smithi045.stdout:3218: expect (ObjNum 1096 snap 325 seq_num 1096) 2024-04-26T00:38:13.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfef59e640 10 monclient: get_monmap_and_config 2024-04-26T00:38:13.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfef59e640 10 monclient: build_initial_monmap 2024-04-26T00:38:13.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfef59e640 10 monclient: monmap: 2024-04-26T00:38:13.945 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:38:13.945 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:13.946 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:38:13.948663+0000 2024-04-26T00:38:13.946 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:38:13.948663+0000 2024-04-26T00:38:13.946 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:38:13.946 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:13.946 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:38:13.947 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:38:13.947 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:38:13.947 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:38:13.947 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:38:13.947 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:13.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfef59e640 1 Processor -- start 2024-04-26T00:38:13.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfef59e640 1 -- start start 2024-04-26T00:38:13.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfef59e640 10 monclient: init 2024-04-26T00:38:13.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfef59e640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:13.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfef59e640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-04-26T00:38:13.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfef59e640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fbfe80a3d90 0x7fbfe809bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:13.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfef59e640 10 monclient(hunting): picked mon.noname-a con 0x7fbfe80a3d90 addr v2:172.21.15.37:3300/0 2024-04-26T00:38:13.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfef59e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbfe80a4170 0x7fbfe80a45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:13.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfef59e640 10 monclient(hunting): picked mon.noname-b con 0x7fbfe80a4170 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:38:13.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfef59e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbfe8133b20 addr v1:172.21.15.37:6789/0 2024-04-26T00:38:13.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfef59e640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfe80a5150 con 0x7fbfe8133b20 2024-04-26T00:38:13.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfef59e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:13.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfef59e640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fbfe809b280 con 0x7fbfe80a3d90 2024-04-26T00:38:13.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfef59e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:13.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfef59e640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fbfe809b560 con 0x7fbfe80a4170 2024-04-26T00:38:13.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfef59e640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:13.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfef59e640 10 monclient(hunting): authenticate will time out at 2075.255068s 2024-04-26T00:38:13.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfed33a640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fbfe80a3d90 0x7fbfe809bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:13.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfecb39640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbfe80a4170 0x7fbfe80a45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:13.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfed33a640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fbfe80a3d90 0x7fbfe809bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44674/0 (socket says 172.21.15.37:44674) 2024-04-26T00:38:13.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfed33a640 1 -- 172.21.15.37:0/1797776949 learned_addr learned my addr 172.21.15.37:0/1797776949 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:38:13.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfed33a640 10 monclient(hunting): get_auth_request con 0x7fbfe80a3d90 auth_method 0 2024-04-26T00:38:13.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfed33a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:13.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfed33a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:13.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfed33a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:13.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfedb3b640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fbfe8133b20 0x7fbfe8133f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:51786/0 (socket says 172.21.15.37:51786) 2024-04-26T00:38:13.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfecb39640 10 monclient(hunting): get_auth_request con 0x7fbfe80a4170 auth_method 0 2024-04-26T00:38:13.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfecb39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:13.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfecb39640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:13.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfecb39640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:13.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfecb39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:13.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfecb39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:13.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfecb39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:13.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfed33a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:13.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfed33a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:13.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfed33a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:13.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfed33a640 10 monclient(hunting): handle_auth_done global_id 144918 payload 274 2024-04-26T00:38:13.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfed33a640 1 -- 172.21.15.37:0/1797776949 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbfe80a4170 msgr2=0x7fbfe80a45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:38:13.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfed33a640 1 --2- 172.21.15.37:0/1797776949 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbfe80a4170 0x7fbfe80a45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:13.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfed33a640 1 -- 172.21.15.37:0/1797776949 >> v1:172.21.15.37:6789/0 conn(0x7fbfe8133b20 legacy=0x7fbfe8133f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:38:13.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfed33a640 10 monclient: _finish_hunting 0 2024-04-26T00:38:13.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfed33a640 1 monclient: found mon.noname-a 2024-04-26T00:38:13.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfed33a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:13.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfed33a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:38:13.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfed33a640 1 -- 172.21.15.37:0/1797776949 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfe80a4af0 con 0x7fbfe80a3d90 2024-04-26T00:38:13.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.947+0000 7fbfecb39640 1 --2- 172.21.15.37:0/1797776949 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbfe80a4170 0x7fbfe80a45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:38:13.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.951+0000 7fbfed33a640 1 -- 172.21.15.37:0/1797776949 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbfe80a3d90 msgr2=0x7fbfe809bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:13.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.951+0000 7fbfed33a640 1 --2- 172.21.15.37:0/1797776949 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbfe80a3d90 0x7fbfe809bf40 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fbfe4009870 tx=0x7fbfe401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a1c0d08c6c9772a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:13.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfdffff640 1 -- 172.21.15.37:0/1797776949 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe40b0070 con 0x7fbfe80a3d90 2024-04-26T00:38:13.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfdffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:13.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfdffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:38:13.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfdffff640 10 monclient: dump: 2024-04-26T00:38:13.962 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:13.962 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:13.962 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:13.962 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:13.962 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:13.963 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:13.963 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:13.963 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:13.963 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:13.963 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:13.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfdffff640 10 monclient: _finish_auth 0 2024-04-26T00:38:13.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfdffff640 10 monclient: _check_auth_tickets 2024-04-26T00:38:13.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfdffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:13.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfdffff640 1 -- 172.21.15.37:0/1797776949 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfe40ab600 con 0x7fbfe80a3d90 2024-04-26T00:38:13.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfef59e640 5 monclient: authenticate success, global_id 144918 2024-04-26T00:38:13.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfef59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:38:13.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfdffff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:38:13.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfef59e640 10 monclient: get_monmap_and_config success 2024-04-26T00:38:13.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfdffff640 1 -- 172.21.15.37:0/1797776949 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe40ab9d0 con 0x7fbfe80a3d90 2024-04-26T00:38:13.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfef59e640 4 set_mon_vals no callback set 2024-04-26T00:38:13.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfdffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:13.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfdffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:38:13.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfdffff640 10 monclient: dump: 2024-04-26T00:38:13.964 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:13.964 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:13.964 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:13.964 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:13.964 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:13.964 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:13.964 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:13.964 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:13.964 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:13.964 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:13.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfef59e640 10 monclient: shutdown 2024-04-26T00:38:13.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfef59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:38:13.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfef59e640 1 -- 172.21.15.37:0/1797776949 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbfe80a3d90 msgr2=0x7fbfe809bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:13.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfef59e640 1 --2- 172.21.15.37:0/1797776949 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbfe80a3d90 0x7fbfe809bf40 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fbfe4009870 tx=0x7fbfe401c3e0 comp rx=0 tx=0).stop 2024-04-26T00:38:13.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfef59e640 1 -- 172.21.15.37:0/1797776949 shutdown_connections 2024-04-26T00:38:13.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfef59e640 1 --2- 172.21.15.37:0/1797776949 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbfe80a4170 0x7fbfe80a45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:13.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfef59e640 1 --2- 172.21.15.37:0/1797776949 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbfe80a3d90 0x7fbfe809bf40 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:13.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfef59e640 1 -- 172.21.15.37:0/1797776949 >> 172.21.15.37:0/1797776949 conn(0x7fbfe80aa0a0 msgr2=0x7fbfe80a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:38:13.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfef59e640 1 -- 172.21.15.37:0/1797776949 shutdown_connections 2024-04-26T00:38:13.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfef59e640 1 -- 172.21.15.37:0/1797776949 wait complete. 2024-04-26T00:38:13.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfef59e640 10 monclient: build_initial_monmap 2024-04-26T00:38:13.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfef59e640 10 monclient: monmap: 2024-04-26T00:38:13.966 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:38:13.966 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:38:13.966 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:38:13.966 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:38:13.966 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:38:13.966 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:38:13.966 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:38:13.966 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:38:13.966 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:38:13.966 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:13.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfef59e640 1 Processor -- start 2024-04-26T00:38:13.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfef59e640 1 -- start start 2024-04-26T00:38:13.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfef59e640 10 monclient: init 2024-04-26T00:38:13.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfef59e640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:13.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfef59e640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:38:13.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfef59e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbfe80a3d90 0x7fbfe8132d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:13.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfef59e640 10 monclient(hunting): picked mon.noname-c con 0x7fbfe80a3d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:38:13.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfef59e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfe80a4170 0x7fbfe81332b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:13.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfef59e640 10 monclient(hunting): picked mon.noname-b con 0x7fbfe80a4170 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:38:13.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfef59e640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbfe8133b20 0x7fbfe8138d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:13.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfef59e640 10 monclient(hunting): picked mon.noname-a con 0x7fbfe8133b20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:38:13.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfef59e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:13.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfef59e640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe8141e30 con 0x7fbfe8133b20 2024-04-26T00:38:13.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfef59e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:13.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfef59e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe809c690 con 0x7fbfe80a4170 2024-04-26T00:38:13.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfef59e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:13.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfef59e640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fbfe8141fb0 con 0x7fbfe80a3d90 2024-04-26T00:38:13.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfef59e640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:13.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfef59e640 10 monclient(hunting): authenticate will time out at 2075.262568s 2024-04-26T00:38:13.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfed33a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbfe80a3d90 0x7fbfe8132d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:13.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfed33a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbfe80a3d90 0x7fbfe8132d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:53006/0 (socket says 172.21.15.37:53006) 2024-04-26T00:38:13.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfed33a640 1 -- 172.21.15.37:0/2588736373 learned_addr learned my addr 172.21.15.37:0/2588736373 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:38:13.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfed33a640 10 monclient(hunting): get_auth_request con 0x7fbfe80a3d90 auth_method 0 2024-04-26T00:38:13.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfedb3b640 1 --2- 172.21.15.37:0/2588736373 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbfe8133b20 0x7fbfe8138d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:13.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfed33a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:13.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfed33a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:13.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfed33a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:13.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfedb3b640 10 monclient(hunting): get_auth_request con 0x7fbfe8133b20 auth_method 0 2024-04-26T00:38:13.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfedb3b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:13.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfedb3b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:13.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfedb3b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:13.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfed33a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:13.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfed33a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:13.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfed33a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:13.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfedb3b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:13.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfedb3b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:13.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfedb3b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:13.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfecb39640 1 --2- 172.21.15.37:0/2588736373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfe80a4170 0x7fbfe81332b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:13.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfecb39640 10 monclient(hunting): get_auth_request con 0x7fbfe80a4170 auth_method 0 2024-04-26T00:38:13.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfecb39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:13.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfecb39640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:13.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfecb39640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:13.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfedb3b640 10 monclient(hunting): handle_auth_done global_id 144921 payload 824 2024-04-26T00:38:13.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfedb3b640 1 -- 172.21.15.37:0/2588736373 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbfe80a3d90 msgr2=0x7fbfe8132d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:13.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfedb3b640 1 --2- 172.21.15.37:0/2588736373 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbfe80a3d90 0x7fbfe8132d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:13.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfedb3b640 1 -- 172.21.15.37:0/2588736373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfe80a4170 msgr2=0x7fbfe81332b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:13.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfedb3b640 1 --2- 172.21.15.37:0/2588736373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfe80a4170 0x7fbfe81332b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:13.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfedb3b640 10 monclient: _finish_hunting 0 2024-04-26T00:38:13.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfedb3b640 1 monclient: found mon.noname-a 2024-04-26T00:38:13.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfedb3b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:13.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfedb3b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:38:13.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfedb3b640 1 -- 172.21.15.37:0/2588736373 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfe809ada0 con 0x7fbfe8133b20 2024-04-26T00:38:13.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfed33a640 1 --2- 172.21.15.37:0/2588736373 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbfe80a3d90 0x7fbfe8132d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:38:13.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.955+0000 7fbfecb39640 1 --2- 172.21.15.37:0/2588736373 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfe80a4170 0x7fbfe81332b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:38:13.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfedb3b640 1 -- 172.21.15.37:0/2588736373 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbfe8133b20 msgr2=0x7fbfe8138d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:13.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfedb3b640 1 --2- 172.21.15.37:0/2588736373 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbfe8133b20 0x7fbfe8138d80 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fbfe000c780 tx=0x7fbfe000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:13.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfddffb640 1 -- 172.21.15.37:0/2588736373 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe0017070 con 0x7fbfe8133b20 2024-04-26T00:38:13.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:13.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfddffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:38:13.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfddffb640 10 monclient: dump: 2024-04-26T00:38:13.979 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:13.979 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:13.979 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:13.979 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:13.979 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:13.979 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:13.979 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:13.979 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:13.979 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:13.979 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:13.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfddffb640 10 monclient: _finish_auth 0 2024-04-26T00:38:13.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfddffb640 10 monclient: _check_auth_tickets 2024-04-26T00:38:13.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:13.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfddffb640 1 -- 172.21.15.37:0/2588736373 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfe000a840 con 0x7fbfe8133b20 2024-04-26T00:38:13.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfef59e640 5 monclient: authenticate success, global_id 144921 2024-04-26T00:38:13.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfef59e640 10 monclient: _renew_subs 2024-04-26T00:38:13.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfef59e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:38:13.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfef59e640 1 -- 172.21.15.37:0/2588736373 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfe80a0350 con 0x7fbfe8133b20 2024-04-26T00:38:13.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfddffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:38:13.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfddffb640 1 -- 172.21.15.37:0/2588736373 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe000ac30 con 0x7fbfe8133b20 2024-04-26T00:38:13.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:13.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:38:13.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfddffb640 10 monclient: dump: 2024-04-26T00:38:13.980 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:13.980 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:13.981 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:13.981 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:13.981 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:13.981 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:13.981 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:13.981 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:13.981 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:13.981 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:13.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfdeffd640 4 set_mon_vals no callback set 2024-04-26T00:38:13.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfef59e640 10 monclient: _renew_subs 2024-04-26T00:38:13.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfef59e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:38:13.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfef59e640 1 -- 172.21.15.37:0/2588736373 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfe813d600 con 0x7fbfe8133b20 2024-04-26T00:38:13.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfef59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:13.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfef59e640 1 --2- 172.21.15.37:0/2588736373 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbfe81392c0 0x7fbfe81396a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:13.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfef59e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:13.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfef59e640 1 -- 172.21.15.37:0/2588736373 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe804f510 con 0x7fbfe81392c0 2024-04-26T00:38:13.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfed33a640 1 --2- 172.21.15.37:0/2588736373 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbfe81392c0 0x7fbfe81396a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:13.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfed33a640 10 monclient: get_auth_request con 0x7fbfe81392c0 auth_method 0 2024-04-26T00:38:13.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfed33a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:13.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfed33a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:13.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfed33a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:13.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfed33a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:13.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfed33a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:13.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfddffb640 1 -- 172.21.15.37:0/2588736373 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fbfe001a070 con 0x7fbfe8133b20 2024-04-26T00:38:13.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfed33a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:13.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfed33a640 10 monclient(hunting): handle_auth_done global_id 144924 payload 824 2024-04-26T00:38:13.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfed33a640 1 -- 172.21.15.37:0/2588736373 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbfe80008e0 con 0x7fbfe81392c0 2024-04-26T00:38:13.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfddffb640 1 --2- 172.21.15.37:0/2588736373 >> v2:172.21.15.45:6812/2379918623 conn(0x7fbfcc035eb0 0x7fbfcc038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:13.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfddffb640 1 -- 172.21.15.37:0/2588736373 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(719..719 src has 370..719) ==== 3943+0+0 (secure 0 0 0) 0x7fbfe00480b0 con 0x7fbfe8133b20 2024-04-26T00:38:13.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfecb39640 1 --2- 172.21.15.37:0/2588736373 >> v2:172.21.15.45:6812/2379918623 conn(0x7fbfcc035eb0 0x7fbfcc038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:13.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.959+0000 7fbfecb39640 10 monclient: get_auth_request con 0x7fbfcc035eb0 auth_method 0 2024-04-26T00:38:13.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.963+0000 7fbfed33a640 1 -- 172.21.15.37:0/2588736373 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbfe81392c0 msgr2=0x7fbfe81396a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:13.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.963+0000 7fbfed33a640 1 --2- 172.21.15.37:0/2588736373 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbfe81392c0 0x7fbfe81396a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fbfe401cc80 tx=0x7fbfe401ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:13.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.963+0000 7fbfddffb640 1 -- 172.21.15.37:0/2588736373 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe40bb020 con 0x7fbfe81392c0 2024-04-26T00:38:13.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.963+0000 7fbfddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:13.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.963+0000 7fbfddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:38:13.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.963+0000 7fbfddffb640 10 monclient: dump: 2024-04-26T00:38:13.983 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:13.983 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:13.983 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:13.983 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:13.983 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:13.984 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:13.984 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:13.984 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:13.984 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:13.984 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:13.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.963+0000 7fbfddffb640 1 -- 172.21.15.37:0/2588736373 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbfe40b6670 con 0x7fbfe81392c0 2024-04-26T00:38:13.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.963+0000 7fbfddffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:13.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.963+0000 7fbfddffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:38:13.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.963+0000 7fbfddffb640 1 -- 172.21.15.37:0/2588736373 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbfe81392c0 msgr2=0x7fbfe81396a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:13.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.963+0000 7fbfddffb640 1 --2- 172.21.15.37:0/2588736373 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbfe81392c0 0x7fbfe81396a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fbfe401cc80 tx=0x7fbfe401ac20 comp rx=0 tx=0).stop 2024-04-26T00:38:13.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.963+0000 7fbfddffb640 1 -- 172.21.15.37:0/2588736373 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbfe81392c0 msgr2=0x7fbfe81396a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:38:13.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.963+0000 7fbfddffb640 1 --2- 172.21.15.37:0/2588736373 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbfe81392c0 0x7fbfe81396a0 secure :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fbfe401cc80 tx=0x7fbfe401ac20 comp rx=0 tx=0).stop 2024-04-26T00:38:13.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.963+0000 7fbfecb39640 1 --2- 172.21.15.37:0/2588736373 >> v2:172.21.15.45:6812/2379918623 conn(0x7fbfcc035eb0 0x7fbfcc038370 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fbfe80a84f0 tx=0x7fbfd8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:13.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.967+0000 7fbfef59e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:38:13.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.967+0000 7fbfef59e640 1 --2- 172.21.15.37:0/2588736373 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbfe8139be0 0x7fbfe8134b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:13.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.967+0000 7fbfef59e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:13.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.967+0000 7fbfef59e640 1 -- 172.21.15.37:0/2588736373 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe81337f0 con 0x7fbfe8139be0 2024-04-26T00:38:13.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.967+0000 7fbfed33a640 1 --2- 172.21.15.37:0/2588736373 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbfe8139be0 0x7fbfe8134b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:13.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.967+0000 7fbfed33a640 10 monclient: get_auth_request con 0x7fbfe8139be0 auth_method 0 2024-04-26T00:38:13.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.967+0000 7fbfed33a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:13.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.967+0000 7fbfed33a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:13.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.967+0000 7fbfed33a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:13.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.971+0000 7fbfed33a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:13.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.971+0000 7fbfed33a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:13.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.971+0000 7fbfed33a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:13.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.971+0000 7fbfed33a640 10 monclient(hunting): handle_auth_done global_id 144927 payload 824 2024-04-26T00:38:13.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.971+0000 7fbfed33a640 1 -- 172.21.15.37:0/2588736373 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbfe813b090 con 0x7fbfe8139be0 2024-04-26T00:38:13.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.971+0000 7fbfed33a640 1 -- 172.21.15.37:0/2588736373 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbfe8139be0 msgr2=0x7fbfe8134b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:13.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.971+0000 7fbfed33a640 1 --2- 172.21.15.37:0/2588736373 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbfe8139be0 0x7fbfe8134b30 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fbfe4000c10 tx=0x7fbfe4019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:13.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.971+0000 7fbfddffb640 1 -- 172.21.15.37:0/2588736373 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe40b0040 con 0x7fbfe8139be0 2024-04-26T00:38:13.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.971+0000 7fbfddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:13.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.971+0000 7fbfddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:38:13.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.971+0000 7fbfddffb640 10 monclient: dump: 2024-04-26T00:38:13.986 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:13.986 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:13.986 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:13.986 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:13.986 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:13.986 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:13.986 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:13.986 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:13.986 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:13.986 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:13.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.971+0000 7fbfddffb640 1 -- 172.21.15.37:0/2588736373 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4890 (secure 0 0 0) 0x7fbfe40c2c70 con 0x7fbfe8139be0 2024-04-26T00:38:13.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.971+0000 7fbfddffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:38:13.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.971+0000 7fbfddffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:38:13.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.971+0000 7fbfddffb640 1 -- 172.21.15.37:0/2588736373 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbfe8139be0 msgr2=0x7fbfe8134b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:13.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.971+0000 7fbfddffb640 1 --2- 172.21.15.37:0/2588736373 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbfe8139be0 0x7fbfe8134b30 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fbfe4000c10 tx=0x7fbfe4019180 comp rx=0 tx=0).stop 2024-04-26T00:38:13.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.971+0000 7fbfddffb640 1 -- 172.21.15.37:0/2588736373 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbfe8139be0 msgr2=0x7fbfe8134b30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:38:13.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.971+0000 7fbfddffb640 1 --2- 172.21.15.37:0/2588736373 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbfe8139be0 0x7fbfe8134b30 secure :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fbfe4000c10 tx=0x7fbfe4019180 comp rx=0 tx=0).stop 2024-04-26T00:38:13.987 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:38:13.987 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:38:13.987 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:38:13.988 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:38:13.988 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 138, 2024-04-26T00:38:13.988 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 380514, 2024-04-26T00:38:13.988 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:38:13.988 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:38:13.988 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:38:13.988 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:38:13.988 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:38:13.988 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 4, 2024-04-26T00:38:13.988 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:38:13.988 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:38:13.988 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:38:13.988 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:38:13.989 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:38:13.989 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:38:13.989 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:38:13.989 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:38:13.989 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:38:13.989 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:38:13.989 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:38:13.989 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:38:13.989 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:38:13.989 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:38:13.989 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:38:13.989 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:38:13.989 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:38:13.989 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:38:13.989 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:38:13.990 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:38:13.990 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:38:13.990 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:38:13.990 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:38:13.990 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:38:13.990 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:38:13.990 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:38:13.990 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:38:13.990 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:38:13.990 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:13.990 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:13.990 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:38:13.990 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:38:13.990 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:38:13.991 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:38:13.991 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:38:13.991 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:38:13.991 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:38:13.991 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:38:13.991 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:38:13.991 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:38:13.991 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:38:13.991 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:38:13.991 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:38:13.991 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:38:13.991 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:38:13.991 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:38:13.991 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:38:13.991 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:38:13.992 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:38:13.992 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:38:13.992 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:38:13.992 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:38:13.992 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:38:13.992 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:38:13.992 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:38:13.992 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:38:13.992 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:38:13.992 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:38:13.992 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:38:13.992 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:38:13.992 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:13.992 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:38:13.993 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:13.993 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:38:13.993 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:38:13.993 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:38:13.993 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:38:13.993 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:13.993 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:38:13.993 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:38:13.993 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:13.993 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:13.993 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:13.993 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:38:13.993 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:38:13.993 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:13.994 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:13.994 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:13.994 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:13.994 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:38:13.994 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:38:13.994 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:38:13.994 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:38:13.994 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:38:13.994 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:13.994 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:13.994 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:38:13.994 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:38:13.994 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:38:13.994 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:38:13.995 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:13.995 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:38:13.995 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:38:13.995 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:13.995 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:13.995 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:13.995 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:38:13.995 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:38:13.995 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:13.995 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:13.995 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:13.995 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:13.995 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:38:13.995 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:38:13.995 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:38:13.996 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:38:13.996 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:38:13.996 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:13.996 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:13.996 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:38:13.996 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:38:13.996 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:38:13.996 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:38:13.996 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:13.996 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:38:13.996 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:38:13.996 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:13.996 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:13.996 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:13.996 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:38:13.997 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:38:13.997 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:13.997 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:13.997 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:13.997 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:13.997 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:38:13.997 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:38:13.997 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:38:13.997 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:38:13.997 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:38:13.997 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:13.997 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:13.997 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:13.997 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:38:13.998 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:38:13.998 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:13.998 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:38:13.998 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:38:13.998 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:38:13.998 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:13.998 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:38:13.998 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:38:13.998 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:13.998 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:38:13.998 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:38:13.998 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:38:13.998 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:13.998 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:38:13.998 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:38:13.999 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:13.999 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:38:13.999 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:38:13.999 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 7 2024-04-26T00:38:13.999 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:13.999 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:38:13.999 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:38:13.999 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:13.999 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:38:13.999 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:38:13.999 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:38:13.999 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:13.999 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:13.999 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:14.000 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:38:14.000 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:38:14.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.971+0000 7fbfef59e640 1 -- 172.21.15.37:0/2588736373 >> v2:172.21.15.45:6812/2379918623 conn(0x7fbfcc035eb0 msgr2=0x7fbfcc038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:14.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.971+0000 7fbfef59e640 1 --2- 172.21.15.37:0/2588736373 >> v2:172.21.15.45:6812/2379918623 conn(0x7fbfcc035eb0 0x7fbfcc038370 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fbfe80a84f0 tx=0x7fbfd8008040 comp rx=0 tx=0).stop 2024-04-26T00:38:14.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.971+0000 7fbfef59e640 10 monclient: shutdown 2024-04-26T00:38:14.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.971+0000 7fbfef59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:38:14.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.971+0000 7fbfef59e640 1 -- 172.21.15.37:0/2588736373 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbfe8133b20 msgr2=0x7fbfe8138d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:14.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.971+0000 7fbfef59e640 1 --2- 172.21.15.37:0/2588736373 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbfe8133b20 0x7fbfe8138d80 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fbfe000c780 tx=0x7fbfe000cc40 comp rx=0 tx=0).stop 2024-04-26T00:38:14.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.971+0000 7fbfedb3b640 1 -- 172.21.15.37:0/2588736373 reap_dead start 2024-04-26T00:38:14.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.971+0000 7fbfedb3b640 1 -- 172.21.15.37:0/2588736373 reap_dead start 2024-04-26T00:38:14.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.971+0000 7fbfef59e640 1 -- 172.21.15.37:0/2588736373 shutdown_connections 2024-04-26T00:38:14.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.971+0000 7fbfef59e640 1 -- 172.21.15.37:0/2588736373 >> 172.21.15.37:0/2588736373 conn(0x7fbfe80aa0a0 msgr2=0x7fbfe80a2080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:38:14.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.971+0000 7fbfef59e640 1 -- 172.21.15.37:0/2588736373 shutdown_connections 2024-04-26T00:38:14.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:13.971+0000 7fbfef59e640 1 -- 172.21.15.37:0/2588736373 wait complete. 2024-04-26T00:38:14.002 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:38:14.002 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:38:14.002 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:38:14.005 INFO:tasks.rados.rados.0.smithi045.stdout:3213: done (5 left) 2024-04-26T00:38:14.005 INFO:tasks.rados.rados.0.smithi045.stdout:3214: done (4 left) 2024-04-26T00:38:14.005 INFO:tasks.rados.rados.0.smithi045.stdout:3215: done (3 left) 2024-04-26T00:38:14.005 INFO:tasks.rados.rados.0.smithi045.stdout:3216: done (2 left) 2024-04-26T00:38:14.005 INFO:tasks.rados.rados.0.smithi045.stdout:3218: done (1 left) 2024-04-26T00:38:14.005 INFO:tasks.rados.rados.0.smithi045.stdout:3219: done (0 left) 2024-04-26T00:38:14.006 INFO:tasks.rados.rados.0.smithi045.stdout:3220: rollback oid 21 current snap is 350 2024-04-26T00:38:14.006 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 21 to 346 2024-04-26T00:38:14.006 INFO:tasks.rados.rados.0.smithi045.stdout:3221: write (excl) oid 37 current snap is 350 2024-04-26T00:38:14.006 INFO:tasks.rados.rados.0.smithi045.stdout:3221: seq_num 1152 ranges {561664=656151,1729021=528400,2669170=692494} 2024-04-26T00:38:14.025 INFO:tasks.rados.rados.0.smithi045.stdout:3221: writing smithi04526579-37 from 561664 to 1217815 tid 1 2024-04-26T00:38:14.041 INFO:tasks.rados.rados.0.smithi045.stdout:3221: writing smithi04526579-37 from 1729021 to 2257421 tid 2 2024-04-26T00:38:14.058 INFO:tasks.rados.rados.0.smithi045.stdout:3221: writing smithi04526579-37 from 2669170 to 3361664 tid 3 2024-04-26T00:38:14.058 INFO:tasks.rados.rados.0.smithi045.stdout:3222: write oid 29 current snap is 350 2024-04-26T00:38:14.058 INFO:tasks.rados.rados.0.smithi045.stdout:3222: seq_num 1153 ranges {578471=731400,1787503=672058,3096593=281878} 2024-04-26T00:38:14.072 INFO:tasks.rados.rados.0.smithi045.stdout:3222: writing smithi04526579-29 from 578471 to 1309871 tid 1 2024-04-26T00:38:14.083 INFO:tasks.rados.rados.0.smithi045.stdout:3222: writing smithi04526579-29 from 1787503 to 2459561 tid 2 2024-04-26T00:38:14.091 INFO:tasks.rados.rados.0.smithi045.stdout:3222: writing smithi04526579-29 from 3096593 to 3378471 tid 3 2024-04-26T00:38:14.092 INFO:tasks.rados.rados.0.smithi045.stdout:3223: snap_create 2024-04-26T00:38:14.092 INFO:tasks.rados.rados.0.smithi045.stdout:3220: finishing rollback tid 0 to smithi04526579-21 2024-04-26T00:38:14.092 INFO:tasks.rados.rados.0.smithi045.stdout:3220: finishing rollback tid 1 to smithi04526579-21 2024-04-26T00:38:14.092 INFO:tasks.rados.rados.0.smithi045.stdout:3220: finishing rollback tid 2 to smithi04526579-21 2024-04-26T00:38:14.092 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 2360 (ObjNum 1087 snap 324 seq_num 1087) dirty exists 2024-04-26T00:38:14.092 INFO:tasks.rados.rados.0.smithi045.stdout:3221: finishing write tid 1 to smithi04526579-37 2024-04-26T00:38:14.093 INFO:tasks.rados.rados.0.smithi045.stdout:3221: finishing write tid 2 to smithi04526579-37 2024-04-26T00:38:14.093 INFO:tasks.rados.rados.0.smithi045.stdout:3221: finishing write tid 3 to smithi04526579-37 2024-04-26T00:38:14.093 INFO:tasks.rados.rados.0.smithi045.stdout:3221: finishing write tid 4 to smithi04526579-37 2024-04-26T00:38:14.093 INFO:tasks.rados.rados.0.smithi045.stdout:3221: finishing write tid 5 to smithi04526579-37 2024-04-26T00:38:14.093 INFO:tasks.rados.rados.0.smithi045.stdout:3221: oid 37 updating version 0 to 2490 2024-04-26T00:38:14.093 INFO:tasks.rados.rados.0.smithi045.stdout:3221: oid 37 updating version 2490 to 2492 2024-04-26T00:38:14.093 INFO:tasks.rados.rados.0.smithi045.stdout:3221: oid 37 version 2492 is already newer than 2491 2024-04-26T00:38:14.093 INFO:tasks.rados.rados.0.smithi045.stdout:3221: oid 37 version 2492 is already newer than 2489 2024-04-26T00:38:14.093 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 2492 (ObjNum 1152 snap 350 seq_num 1152) dirty exists 2024-04-26T00:38:14.093 INFO:tasks.rados.rados.0.smithi045.stdout:3221: left oid 37 (ObjNum 1152 snap 350 seq_num 1152) 2024-04-26T00:38:14.093 INFO:tasks.rados.rados.0.smithi045.stdout:3222: finishing write tid 1 to smithi04526579-29 2024-04-26T00:38:14.093 INFO:tasks.rados.rados.0.smithi045.stdout:3222: finishing write tid 2 to smithi04526579-29 2024-04-26T00:38:14.095 INFO:tasks.rados.rados.0.smithi045.stdout:3222: finishing write tid 3 to smithi04526579-29 2024-04-26T00:38:14.096 INFO:tasks.rados.rados.0.smithi045.stdout:3222: finishing write tid 4 to smithi04526579-29 2024-04-26T00:38:14.097 INFO:tasks.rados.rados.0.smithi045.stdout:3222: finishing write tid 5 to smithi04526579-29 2024-04-26T00:38:14.097 INFO:tasks.rados.rados.0.smithi045.stdout:3222: oid 29 updating version 0 to 2201 2024-04-26T00:38:14.097 INFO:tasks.rados.rados.0.smithi045.stdout:3222: oid 29 updating version 2201 to 2203 2024-04-26T00:38:14.097 INFO:tasks.rados.rados.0.smithi045.stdout:3222: oid 29 version 2203 is already newer than 2202 2024-04-26T00:38:14.097 INFO:tasks.rados.rados.0.smithi045.stdout:3222: oid 29 updating version 2203 to 2204 2024-04-26T00:38:14.097 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 2204 (ObjNum 1153 snap 350 seq_num 1153) dirty exists 2024-04-26T00:38:14.098 INFO:tasks.rados.rados.0.smithi045.stdout:3222: left oid 29 (ObjNum 1153 snap 350 seq_num 1153) 2024-04-26T00:38:14.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a1cdf8640 10 monclient: get_monmap_and_config 2024-04-26T00:38:14.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a1cdf8640 10 monclient: build_initial_monmap 2024-04-26T00:38:14.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a1cdf8640 10 monclient: monmap: 2024-04-26T00:38:14.124 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:38:14.124 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:14.124 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:38:14.128404+0000 2024-04-26T00:38:14.125 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:38:14.128404+0000 2024-04-26T00:38:14.125 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:38:14.125 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:14.125 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:38:14.125 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:38:14.125 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:38:14.125 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:38:14.125 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:38:14.125 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:14.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a1cdf8640 1 Processor -- start 2024-04-26T00:38:14.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a1cdf8640 1 -- start start 2024-04-26T00:38:14.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a1cdf8640 10 monclient: init 2024-04-26T00:38:14.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a1cdf8640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:14.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a1cdf8640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-04-26T00:38:14.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a1cdf8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0a180a0300 0x7f0a1809fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:14.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a1cdf8640 10 monclient(hunting): picked mon.noname-b con 0x7f0a180a0300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:38:14.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a1cdf8640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0a180a06e0 0x7f0a18134ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:14.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a1cdf8640 10 monclient(hunting): picked mon.noname-c con 0x7f0a180a06e0 addr v2:172.21.15.45:3300/0 2024-04-26T00:38:14.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a1cdf8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0a181350e0 addr v1:172.21.15.45:6789/0 2024-04-26T00:38:14.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a1cdf8640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a1809c190 con 0x7f0a181350e0 2024-04-26T00:38:14.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a1cdf8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:14.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a1cdf8640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f0a180a13f0 con 0x7f0a180a06e0 2024-04-26T00:38:14.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a1cdf8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:14.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a1cdf8640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0a180a1110 con 0x7f0a180a0300 2024-04-26T00:38:14.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a1cdf8640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:14.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a1cdf8640 10 monclient(hunting): authenticate will time out at 2075.434094s 2024-04-26T00:38:14.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a1659c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0a180a0300 0x7f0a1809fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:14.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a1659c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0a180a0300 0x7f0a1809fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:53010/0 (socket says 172.21.15.37:53010) 2024-04-26T00:38:14.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a1659c640 1 -- 172.21.15.37:0/2572228450 learned_addr learned my addr 172.21.15.37:0/2572228450 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:38:14.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a1659c640 10 monclient(hunting): get_auth_request con 0x7f0a180a0300 auth_method 0 2024-04-26T00:38:14.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a1659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:14.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a1659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:14.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a1659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:14.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a1659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:14.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a1659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:14.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a1659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:14.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a15d9b640 1 --2- 172.21.15.37:0/2572228450 >> v2:172.21.15.45:3300/0 conn(0x7f0a180a06e0 0x7f0a18134ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:14.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a15d9b640 10 monclient(hunting): get_auth_request con 0x7f0a180a06e0 auth_method 0 2024-04-26T00:38:14.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a15d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:14.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a15d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:14.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a15d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:14.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a1659c640 10 monclient(hunting): handle_auth_done global_id 204314 payload 274 2024-04-26T00:38:14.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a1659c640 1 -- 172.21.15.37:0/2572228450 >> v2:172.21.15.45:3300/0 conn(0x7f0a180a06e0 msgr2=0x7f0a18134ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:38:14.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a1659c640 1 --2- 172.21.15.37:0/2572228450 >> v2:172.21.15.45:3300/0 conn(0x7f0a180a06e0 0x7f0a18134ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:14.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a1659c640 1 -- 172.21.15.37:0/2572228450 >> v1:172.21.15.45:6789/0 conn(0x7f0a181350e0 legacy=0x7f0a1813bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:38:14.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a1659c640 10 monclient: _finish_hunting 0 2024-04-26T00:38:14.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a1659c640 1 monclient: found mon.noname-b 2024-04-26T00:38:14.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a1659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:14.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a1659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:38:14.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.127+0000 7f0a1659c640 1 -- 172.21.15.37:0/2572228450 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a180a16d0 con 0x7f0a180a0300 2024-04-26T00:38:14.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.131+0000 7f0a1659c640 1 -- 172.21.15.37:0/2572228450 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0a180a0300 msgr2=0x7f0a1809fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:14.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.131+0000 7f0a1659c640 1 --2- 172.21.15.37:0/2572228450 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0a180a0300 0x7f0a1809fc50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0a08009870 tx=0x7f0a0801c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f7406a4f6f0bc9c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:14.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.131+0000 7f0a1559a640 1 -- 172.21.15.37:0/2572228450 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a080b0070 con 0x7f0a180a0300 2024-04-26T00:38:14.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.131+0000 7f0a1559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:14.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.131+0000 7f0a1559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:38:14.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.131+0000 7f0a1559a640 10 monclient: dump: 2024-04-26T00:38:14.133 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:14.133 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:14.133 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:14.133 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:14.133 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:14.133 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:14.133 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:14.133 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:14.133 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:14.133 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:14.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.131+0000 7f0a1559a640 10 monclient: _finish_auth 0 2024-04-26T00:38:14.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.131+0000 7f0a1559a640 10 monclient: _check_auth_tickets 2024-04-26T00:38:14.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.131+0000 7f0a1559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:14.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.131+0000 7f0a1559a640 1 -- 172.21.15.37:0/2572228450 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a080ab600 con 0x7f0a180a0300 2024-04-26T00:38:14.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.131+0000 7f0a1559a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:38:14.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.131+0000 7f0a1559a640 1 -- 172.21.15.37:0/2572228450 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a080ab9d0 con 0x7f0a180a0300 2024-04-26T00:38:14.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.131+0000 7f0a1cdf8640 5 monclient: authenticate success, global_id 204314 2024-04-26T00:38:14.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.131+0000 7f0a1cdf8640 10 monclient: get_monmap_and_config success 2024-04-26T00:38:14.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.131+0000 7f0a1cdf8640 4 set_mon_vals no callback set 2024-04-26T00:38:14.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.131+0000 7f0a1559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:14.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.131+0000 7f0a1559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:38:14.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.131+0000 7f0a1559a640 10 monclient: dump: 2024-04-26T00:38:14.134 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:14.134 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:14.134 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:14.134 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:14.134 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:14.135 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:14.135 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:14.135 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:14.135 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:14.135 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:14.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.131+0000 7f0a1cdf8640 10 monclient: shutdown 2024-04-26T00:38:14.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.131+0000 7f0a1cdf8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:38:14.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.131+0000 7f0a1cdf8640 1 -- 172.21.15.37:0/2572228450 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0a180a0300 msgr2=0x7f0a1809fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:14.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.131+0000 7f0a1cdf8640 1 --2- 172.21.15.37:0/2572228450 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0a180a0300 0x7f0a1809fc50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0a08009870 tx=0x7f0a0801c410 comp rx=0 tx=0).stop 2024-04-26T00:38:14.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.131+0000 7f0a1cdf8640 1 -- 172.21.15.37:0/2572228450 shutdown_connections 2024-04-26T00:38:14.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.131+0000 7f0a1cdf8640 1 --2- 172.21.15.37:0/2572228450 >> v2:172.21.15.45:3300/0 conn(0x7f0a180a06e0 0x7f0a18134ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:14.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.131+0000 7f0a1cdf8640 1 --2- 172.21.15.37:0/2572228450 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0a180a0300 0x7f0a1809fc50 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:14.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.131+0000 7f0a1cdf8640 1 -- 172.21.15.37:0/2572228450 >> 172.21.15.37:0/2572228450 conn(0x7f0a180a1d50 msgr2=0x7f0a180a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:38:14.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a1cdf8640 1 -- 172.21.15.37:0/2572228450 shutdown_connections 2024-04-26T00:38:14.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a1cdf8640 1 -- 172.21.15.37:0/2572228450 wait complete. 2024-04-26T00:38:14.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a1cdf8640 10 monclient: build_initial_monmap 2024-04-26T00:38:14.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a1cdf8640 10 monclient: monmap: 2024-04-26T00:38:14.136 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:38:14.136 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:38:14.136 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:38:14.136 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:38:14.136 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:38:14.136 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:38:14.136 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:38:14.136 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:38:14.136 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:38:14.137 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:14.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a1cdf8640 1 Processor -- start 2024-04-26T00:38:14.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a1cdf8640 1 -- start start 2024-04-26T00:38:14.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a1cdf8640 10 monclient: init 2024-04-26T00:38:14.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a1cdf8640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:14.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a1cdf8640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:38:14.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a1cdf8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a180a0300 0x7f0a181cd420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:14.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a1cdf8640 10 monclient(hunting): picked mon.noname-b con 0x7f0a180a0300 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:38:14.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a1cdf8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0a180a06e0 0x7f0a1813c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:14.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a1cdf8640 10 monclient(hunting): picked mon.noname-c con 0x7f0a180a06e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:38:14.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a1cdf8640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a181350e0 0x7f0a1813c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:14.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a1cdf8640 10 monclient(hunting): picked mon.noname-a con 0x7f0a181350e0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:38:14.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a1cdf8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:14.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a1cdf8640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f0a180a0c40 con 0x7f0a181350e0 2024-04-26T00:38:14.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a1cdf8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:14.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a1cdf8640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0a180a0ac0 con 0x7f0a180a0300 2024-04-26T00:38:14.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a1cdf8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:14.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a1cdf8640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0a1813fe10 con 0x7f0a180a06e0 2024-04-26T00:38:14.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a1cdf8640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:14.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a1cdf8640 10 monclient(hunting): authenticate will time out at 2075.441517s 2024-04-26T00:38:14.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a16d9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a181350e0 0x7f0a1813c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:14.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a16d9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a181350e0 0x7f0a1813c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44692/0 (socket says 172.21.15.37:44692) 2024-04-26T00:38:14.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a16d9d640 1 -- 172.21.15.37:0/2599035630 learned_addr learned my addr 172.21.15.37:0/2599035630 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:38:14.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a16d9d640 10 monclient(hunting): get_auth_request con 0x7f0a181350e0 auth_method 0 2024-04-26T00:38:14.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a16d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:14.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a16d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:14.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a16d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:14.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a15d9b640 1 --2- 172.21.15.37:0/2599035630 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0a180a06e0 0x7f0a1813c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:14.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a15d9b640 10 monclient(hunting): get_auth_request con 0x7f0a180a06e0 auth_method 0 2024-04-26T00:38:14.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a15d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:14.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a15d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:14.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a15d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:14.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a16d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:14.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a16d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:14.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a1659c640 1 --2- 172.21.15.37:0/2599035630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a180a0300 0x7f0a181cd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:14.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a16d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:14.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a16d9d640 1 -- 172.21.15.37:0/2599035630 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a181350e0 msgr2=0x7f0a1813c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:38:14.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a1659c640 10 monclient(hunting): get_auth_request con 0x7f0a180a0300 auth_method 0 2024-04-26T00:38:14.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a16d9d640 1 -- 172.21.15.37:0/2599035630 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a181350e0 msgr2=0x7f0a1813c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:38:14.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a1659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:14.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a1659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:14.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a1659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:14.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a16d9d640 1 --2- 172.21.15.37:0/2599035630 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a181350e0 0x7f0a1813c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:38:14.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a15d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:14.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a15d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:14.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a16d9d640 1 --2- 172.21.15.37:0/2599035630 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a181350e0 0x7f0a1813c7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:38:14.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a15d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:14.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a1659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:14.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a1659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:14.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a1659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:14.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a15d9b640 10 monclient(hunting): handle_auth_done global_id 204317 payload 824 2024-04-26T00:38:14.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a15d9b640 1 -- 172.21.15.37:0/2599035630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a180a0300 msgr2=0x7f0a181cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:14.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a15d9b640 1 --2- 172.21.15.37:0/2599035630 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0a180a0300 0x7f0a181cd420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:14.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a15d9b640 1 -- 172.21.15.37:0/2599035630 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a181350e0 msgr2=0x7f0a1813c7f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:38:14.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a15d9b640 1 --2- 172.21.15.37:0/2599035630 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a181350e0 0x7f0a1813c7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:14.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a15d9b640 10 monclient: _finish_hunting 0 2024-04-26T00:38:14.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a15d9b640 1 monclient: found mon.noname-c 2024-04-26T00:38:14.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a15d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:14.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a15d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:38:14.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.135+0000 7f0a15d9b640 1 -- 172.21.15.37:0/2599035630 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a18139b40 con 0x7f0a180a06e0 2024-04-26T00:38:14.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a15d9b640 1 -- 172.21.15.37:0/2599035630 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0a180a06e0 msgr2=0x7f0a1813c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:14.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a15d9b640 1 --2- 172.21.15.37:0/2599035630 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0a180a06e0 0x7f0a1813c2b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c00d5c0 tx=0x7f0a0c00da80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:14.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a06ffd640 1 -- 172.21.15.37:0/2599035630 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a0c012800 con 0x7f0a180a06e0 2024-04-26T00:38:14.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:14.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a06ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:38:14.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a06ffd640 10 monclient: dump: 2024-04-26T00:38:14.149 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:14.149 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:14.149 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:14.150 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:14.150 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:14.150 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:14.150 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:14.150 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:14.150 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:14.150 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:14.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a06ffd640 10 monclient: _finish_auth 0 2024-04-26T00:38:14.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a06ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:38:14.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:14.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a06ffd640 1 -- 172.21.15.37:0/2599035630 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a0c019070 con 0x7f0a180a06e0 2024-04-26T00:38:14.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a06ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:38:14.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a06ffd640 1 -- 172.21.15.37:0/2599035630 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a0c012be0 con 0x7f0a180a06e0 2024-04-26T00:38:14.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:14.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a06ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:38:14.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a06ffd640 10 monclient: dump: 2024-04-26T00:38:14.151 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:14.151 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:14.151 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:14.151 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:14.151 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:14.151 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:14.151 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:14.151 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:14.151 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:14.151 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:14.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a1cdf8640 5 monclient: authenticate success, global_id 204317 2024-04-26T00:38:14.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a1cdf8640 10 monclient: _renew_subs 2024-04-26T00:38:14.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a1cdf8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:14.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a1cdf8640 1 -- 172.21.15.37:0/2599035630 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a1809f550 con 0x7f0a180a06e0 2024-04-26T00:38:14.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a14d99640 4 set_mon_vals no callback set 2024-04-26T00:38:14.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a1cdf8640 10 monclient: _renew_subs 2024-04-26T00:38:14.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a1cdf8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:14.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a1cdf8640 1 -- 172.21.15.37:0/2599035630 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a18137850 con 0x7f0a180a06e0 2024-04-26T00:38:14.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a1cdf8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:14.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a1cdf8640 1 --2- 172.21.15.37:0/2599035630 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0a18135c40 0x7f0a18136020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:14.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a1cdf8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:14.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a1cdf8640 1 -- 172.21.15.37:0/2599035630 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0a18068f20 con 0x7f0a18135c40 2024-04-26T00:38:14.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a1659c640 1 --2- 172.21.15.37:0/2599035630 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0a18135c40 0x7f0a18136020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:14.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a1659c640 10 monclient: get_auth_request con 0x7f0a18135c40 auth_method 0 2024-04-26T00:38:14.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a1659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:14.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a1659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:14.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a1659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:14.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a1659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:14.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a1659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:14.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a06ffd640 1 -- 172.21.15.37:0/2599035630 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f0a0c01e460 con 0x7f0a180a06e0 2024-04-26T00:38:14.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a1659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:14.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a1659c640 10 monclient(hunting): handle_auth_done global_id 204320 payload 824 2024-04-26T00:38:14.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a1659c640 1 -- 172.21.15.37:0/2599035630 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0a1804f510 con 0x7f0a18135c40 2024-04-26T00:38:14.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a06ffd640 1 --2- 172.21.15.37:0/2599035630 >> v2:172.21.15.45:6812/2379918623 conn(0x7f09f8035e80 0x7f09f8038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:14.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a06ffd640 1 -- 172.21.15.37:0/2599035630 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(719..719 src has 370..719) ==== 3943+0+0 (secure 0 0 0) 0x7f0a0c048f60 con 0x7f0a180a06e0 2024-04-26T00:38:14.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a16d9d640 1 --2- 172.21.15.37:0/2599035630 >> v2:172.21.15.45:6812/2379918623 conn(0x7f09f8035e80 0x7f09f8038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:14.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.139+0000 7f0a16d9d640 10 monclient: get_auth_request con 0x7f09f8035e80 auth_method 0 2024-04-26T00:38:14.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.143+0000 7f0a1659c640 1 -- 172.21.15.37:0/2599035630 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0a18135c40 msgr2=0x7f0a18136020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:14.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.143+0000 7f0a1659c640 1 --2- 172.21.15.37:0/2599035630 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0a18135c40 0x7f0a18136020 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0a080ad000 tx=0x7f0a0801acb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:14.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.143+0000 7f0a06ffd640 1 -- 172.21.15.37:0/2599035630 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a080b0040 con 0x7f0a18135c40 2024-04-26T00:38:14.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.143+0000 7f0a06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:14.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.143+0000 7f0a06ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:38:14.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.143+0000 7f0a06ffd640 10 monclient: dump: 2024-04-26T00:38:14.154 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:14.154 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:14.154 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:14.154 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:14.154 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:14.154 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:14.154 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:14.154 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:14.154 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:14.155 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:14.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.143+0000 7f0a06ffd640 1 -- 172.21.15.37:0/2599035630 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0a080b6670 con 0x7f0a18135c40 2024-04-26T00:38:14.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.143+0000 7f0a06ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:14.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.143+0000 7f0a06ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:38:14.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.143+0000 7f0a06ffd640 1 -- 172.21.15.37:0/2599035630 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0a18135c40 msgr2=0x7f0a18136020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:14.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.143+0000 7f0a06ffd640 1 --2- 172.21.15.37:0/2599035630 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0a18135c40 0x7f0a18136020 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0a080ad000 tx=0x7f0a0801acb0 comp rx=0 tx=0).stop 2024-04-26T00:38:14.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.143+0000 7f0a06ffd640 1 -- 172.21.15.37:0/2599035630 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0a18135c40 msgr2=0x7f0a18136020 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:38:14.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.143+0000 7f0a06ffd640 1 --2- 172.21.15.37:0/2599035630 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0a18135c40 0x7f0a18136020 secure :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0a080ad000 tx=0x7f0a0801acb0 comp rx=0 tx=0).stop 2024-04-26T00:38:14.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.143+0000 7f0a16d9d640 1 --2- 172.21.15.37:0/2599035630 >> v2:172.21.15.45:6812/2379918623 conn(0x7f09f8035e80 0x7f09f8038340 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f0a18138ad0 tx=0x7f0a10003df0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:14.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.147+0000 7f0a1cdf8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:38:14.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.147+0000 7f0a1cdf8640 1 --2- 172.21.15.37:0/2599035630 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0a18136560 0x7f0a18070050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:14.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.147+0000 7f0a1cdf8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:14.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.147+0000 7f0a1cdf8640 1 -- 172.21.15.37:0/2599035630 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0a181cd960 con 0x7f0a18136560 2024-04-26T00:38:14.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.147+0000 7f0a1659c640 1 --2- 172.21.15.37:0/2599035630 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0a18136560 0x7f0a18070050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:14.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.147+0000 7f0a1659c640 10 monclient: get_auth_request con 0x7f0a18136560 auth_method 0 2024-04-26T00:38:14.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.147+0000 7f0a1659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:14.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.147+0000 7f0a1659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:14.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.147+0000 7f0a1659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:14.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.147+0000 7f0a1659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:14.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.147+0000 7f0a1659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:14.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.147+0000 7f0a1659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:14.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.147+0000 7f0a1659c640 10 monclient(hunting): handle_auth_done global_id 204323 payload 824 2024-04-26T00:38:14.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.147+0000 7f0a1659c640 1 -- 172.21.15.37:0/2599035630 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0a18138130 con 0x7f0a18136560 2024-04-26T00:38:14.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.151+0000 7f0a1659c640 1 -- 172.21.15.37:0/2599035630 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0a18136560 msgr2=0x7f0a18070050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:14.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.151+0000 7f0a1659c640 1 --2- 172.21.15.37:0/2599035630 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0a18136560 0x7f0a18070050 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0a0801ab20 tx=0x7f0a08019210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:14.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.151+0000 7f0a06ffd640 1 -- 172.21.15.37:0/2599035630 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a080b0050 con 0x7f0a18136560 2024-04-26T00:38:14.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.151+0000 7f0a06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:14.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.151+0000 7f0a06ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:38:14.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.151+0000 7f0a06ffd640 10 monclient: dump: 2024-04-26T00:38:14.159 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:14.159 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:14.159 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:14.159 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:14.159 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:14.159 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:14.159 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:14.159 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:14.159 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:14.160 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:14.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.151+0000 7f0a06ffd640 1 -- 172.21.15.37:0/2599035630 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4547 (secure 0 0 0) 0x7f0a080c5070 con 0x7f0a18136560 2024-04-26T00:38:14.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.151+0000 7f0a06ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:38:14.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.151+0000 7f0a06ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:38:14.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.151+0000 7f0a06ffd640 1 -- 172.21.15.37:0/2599035630 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0a18136560 msgr2=0x7f0a18070050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:14.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.151+0000 7f0a06ffd640 1 --2- 172.21.15.37:0/2599035630 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0a18136560 0x7f0a18070050 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0a0801ab20 tx=0x7f0a08019210 comp rx=0 tx=0).stop 2024-04-26T00:38:14.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.151+0000 7f0a06ffd640 1 -- 172.21.15.37:0/2599035630 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0a18136560 msgr2=0x7f0a18070050 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:38:14.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.151+0000 7f0a06ffd640 1 --2- 172.21.15.37:0/2599035630 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0a18136560 0x7f0a18070050 secure :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0a0801ab20 tx=0x7f0a08019210 comp rx=0 tx=0).stop 2024-04-26T00:38:14.160 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:38:14.161 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:38:14.161 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:38:14.161 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:38:14.161 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 138, 2024-04-26T00:38:14.161 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 99758, 2024-04-26T00:38:14.161 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:38:14.161 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:38:14.161 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:38:14.161 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:38:14.161 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:38:14.161 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 4, 2024-04-26T00:38:14.162 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:38:14.162 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:38:14.162 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:38:14.162 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:38:14.162 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:38:14.162 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:38:14.162 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:38:14.162 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:38:14.162 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:38:14.162 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:38:14.162 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:38:14.162 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:38:14.163 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:38:14.163 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:38:14.163 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:38:14.163 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:38:14.163 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:38:14.163 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:38:14.163 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:38:14.163 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:38:14.163 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:38:14.163 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:38:14.163 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:38:14.163 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:38:14.163 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:38:14.164 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:38:14.164 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:38:14.164 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:38:14.164 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:14.164 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:14.164 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:38:14.164 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:38:14.164 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:38:14.164 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:38:14.164 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:38:14.164 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:38:14.164 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:38:14.164 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:38:14.164 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:38:14.164 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:38:14.165 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:38:14.165 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:38:14.165 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:38:14.165 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:38:14.165 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:38:14.165 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:38:14.165 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:38:14.165 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:38:14.165 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:38:14.165 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:38:14.165 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:38:14.165 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:38:14.165 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:38:14.166 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:38:14.166 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:38:14.166 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:38:14.166 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:38:14.166 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:38:14.166 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:38:14.166 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:38:14.166 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:14.166 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:38:14.167 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:14.167 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:38:14.167 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:38:14.167 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:38:14.167 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:38:14.167 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:14.168 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:38:14.168 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:38:14.168 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:14.168 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:14.168 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:14.168 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:38:14.168 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:38:14.168 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:14.168 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:14.168 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:14.169 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:14.169 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:38:14.169 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:38:14.169 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:38:14.169 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:38:14.169 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:38:14.169 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:14.169 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:14.169 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:38:14.169 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:38:14.169 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:38:14.169 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:38:14.169 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:14.169 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:38:14.169 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:38:14.170 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:14.170 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:14.170 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:14.170 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:38:14.170 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:38:14.170 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:14.170 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:14.170 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:14.170 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:14.170 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:38:14.170 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:38:14.171 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:38:14.171 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:38:14.171 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:38:14.171 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:14.171 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:14.171 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:38:14.171 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:38:14.172 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:38:14.172 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:38:14.172 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:14.172 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:38:14.172 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:38:14.172 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:14.172 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:14.172 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:14.172 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:38:14.172 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:38:14.172 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:14.173 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:14.173 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:14.173 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:14.173 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:38:14.173 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:38:14.173 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:38:14.173 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:38:14.173 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:38:14.173 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:14.173 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:14.174 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:14.174 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:38:14.174 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:38:14.174 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:14.174 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:38:14.175 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:38:14.175 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:38:14.175 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:14.175 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:38:14.175 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:38:14.175 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:14.176 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:38:14.176 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:38:14.176 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:38:14.178 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:14.178 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:14.178 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:14.178 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:38:14.178 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:38:14.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.151+0000 7f0a1cdf8640 1 -- 172.21.15.37:0/2599035630 >> v2:172.21.15.45:6812/2379918623 conn(0x7f09f8035e80 msgr2=0x7f09f8038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:14.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.151+0000 7f0a1cdf8640 1 --2- 172.21.15.37:0/2599035630 >> v2:172.21.15.45:6812/2379918623 conn(0x7f09f8035e80 0x7f09f8038340 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f0a18138ad0 tx=0x7f0a10003df0 comp rx=0 tx=0).stop 2024-04-26T00:38:14.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.151+0000 7f0a1cdf8640 10 monclient: shutdown 2024-04-26T00:38:14.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.151+0000 7f0a1cdf8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:38:14.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.151+0000 7f0a1cdf8640 1 -- 172.21.15.37:0/2599035630 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0a180a06e0 msgr2=0x7f0a1813c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:14.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.151+0000 7f0a1cdf8640 1 --2- 172.21.15.37:0/2599035630 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0a180a06e0 0x7f0a1813c2b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c00d5c0 tx=0x7f0a0c00da80 comp rx=0 tx=0).stop 2024-04-26T00:38:14.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.151+0000 7f0a16d9d640 1 -- 172.21.15.37:0/2599035630 reap_dead start 2024-04-26T00:38:14.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.151+0000 7f0a1cdf8640 1 -- 172.21.15.37:0/2599035630 shutdown_connections 2024-04-26T00:38:14.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.151+0000 7f0a1cdf8640 1 --2- 172.21.15.37:0/2599035630 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0a180a06e0 0x7f0a1813c2b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:14.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.151+0000 7f0a1cdf8640 1 -- 172.21.15.37:0/2599035630 >> 172.21.15.37:0/2599035630 conn(0x7f0a180a1d50 msgr2=0x7f0a180a3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:38:14.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.151+0000 7f0a1cdf8640 1 -- 172.21.15.37:0/2599035630 shutdown_connections 2024-04-26T00:38:14.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.151+0000 7f0a1cdf8640 1 -- 172.21.15.37:0/2599035630 wait complete. 2024-04-26T00:38:14.183 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:38:14.183 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:38:14.184 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:38:14.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.303+0000 7fa272ee7640 10 monclient: get_monmap_and_config 2024-04-26T00:38:14.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.303+0000 7fa272ee7640 10 monclient: build_initial_monmap 2024-04-26T00:38:14.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.303+0000 7fa272ee7640 10 monclient: monmap: 2024-04-26T00:38:14.301 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:38:14.301 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:14.301 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:38:14.305345+0000 2024-04-26T00:38:14.302 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:38:14.305345+0000 2024-04-26T00:38:14.302 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:38:14.302 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:14.302 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:38:14.302 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:38:14.302 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:38:14.302 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:38:14.302 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:38:14.302 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:14.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.303+0000 7fa272ee7640 1 Processor -- start 2024-04-26T00:38:14.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.303+0000 7fa272ee7640 1 -- start start 2024-04-26T00:38:14.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.303+0000 7fa272ee7640 10 monclient: init 2024-04-26T00:38:14.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.303+0000 7fa272ee7640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:14.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.303+0000 7fa272ee7640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-04-26T00:38:14.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.303+0000 7fa272ee7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa26c0a3d90 addr v1:172.21.15.45:6789/0 2024-04-26T00:38:14.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.303+0000 7fa272ee7640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa26c0a4170 0x7fa26c0a45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:14.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.303+0000 7fa272ee7640 10 monclient(hunting): picked mon.noname-b con 0x7fa26c0a4170 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:38:14.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.303+0000 7fa272ee7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa26c13c3e0 addr v1:172.21.15.37:6789/0 2024-04-26T00:38:14.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.303+0000 7fa272ee7640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa26c0a5150 con 0x7fa26c13c3e0 2024-04-26T00:38:14.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.303+0000 7fa272ee7640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa26c09b280 con 0x7fa26c0a3d90 2024-04-26T00:38:14.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.303+0000 7fa272ee7640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:14.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.303+0000 7fa272ee7640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa26c09c650 con 0x7fa26c0a4170 2024-04-26T00:38:14.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.303+0000 7fa272ee7640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:14.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.303+0000 7fa272ee7640 10 monclient(hunting): authenticate will time out at 2075.611727s 2024-04-26T00:38:14.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.303+0000 7fa26bfff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa26c0a4170 0x7fa26c0a45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:14.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.303+0000 7fa26bfff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa26c0a4170 0x7fa26c0a45b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:53030/0 (socket says 172.21.15.37:53030) 2024-04-26T00:38:14.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.303+0000 7fa271484640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fa26c13c3e0 0x7fa26c13e7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:51798/0 (socket says 172.21.15.37:51798) 2024-04-26T00:38:14.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.303+0000 7fa26bfff640 1 -- 172.21.15.37:0/1711736614 learned_addr learned my addr 172.21.15.37:0/1711736614 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:38:14.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.303+0000 7fa26bfff640 10 monclient(hunting): get_auth_request con 0x7fa26c0a4170 auth_method 0 2024-04-26T00:38:14.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.303+0000 7fa26bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:14.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.303+0000 7fa26bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:14.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.303+0000 7fa26bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:14.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.307+0000 7fa26bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:14.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.307+0000 7fa26bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:14.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.307+0000 7fa26bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:14.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.307+0000 7fa26bfff640 10 monclient(hunting): handle_auth_done global_id 204326 payload 274 2024-04-26T00:38:14.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.307+0000 7fa26bfff640 1 -- 172.21.15.37:0/1711736614 >> v1:172.21.15.45:6789/0 conn(0x7fa26c0a3d90 legacy=0x7fa26c09bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:38:14.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.307+0000 7fa26bfff640 1 -- 172.21.15.37:0/1711736614 >> v1:172.21.15.37:6789/0 conn(0x7fa26c13c3e0 legacy=0x7fa26c13e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:38:14.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.307+0000 7fa26bfff640 10 monclient: _finish_hunting 0 2024-04-26T00:38:14.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.307+0000 7fa26bfff640 1 monclient: found mon.noname-b 2024-04-26T00:38:14.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.307+0000 7fa26bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:14.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.307+0000 7fa26bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:38:14.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.307+0000 7fa26bfff640 1 -- 172.21.15.37:0/1711736614 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa26c09b560 con 0x7fa26c0a4170 2024-04-26T00:38:14.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa26bfff640 1 -- 172.21.15.37:0/1711736614 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa26c0a4170 msgr2=0x7fa26c0a45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:14.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa26bfff640 1 --2- 172.21.15.37:0/1711736614 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa26c0a4170 0x7fa26c0a45b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa2580098a0 tx=0x7fa25801c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b798d538475e8a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:14.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa26b7fe640 1 -- 172.21.15.37:0/1711736614 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2580b0070 con 0x7fa26c0a4170 2024-04-26T00:38:14.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa26b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:14.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa26b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:38:14.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa26b7fe640 10 monclient: dump: 2024-04-26T00:38:14.313 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:14.313 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:14.313 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:14.313 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:14.313 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:14.314 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:14.314 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:14.314 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:14.314 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:14.314 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:14.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa26b7fe640 10 monclient: _finish_auth 0 2024-04-26T00:38:14.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa26b7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:38:14.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa26b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:14.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa272ee7640 5 monclient: authenticate success, global_id 204326 2024-04-26T00:38:14.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa26b7fe640 1 -- 172.21.15.37:0/1711736614 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2580ab600 con 0x7fa26c0a4170 2024-04-26T00:38:14.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa272ee7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:38:14.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa26b7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:38:14.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa272ee7640 10 monclient: get_monmap_and_config success 2024-04-26T00:38:14.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa26b7fe640 1 -- 172.21.15.37:0/1711736614 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2580ab9d0 con 0x7fa26c0a4170 2024-04-26T00:38:14.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa272ee7640 4 set_mon_vals no callback set 2024-04-26T00:38:14.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa26b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:14.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa26b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:38:14.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa26b7fe640 10 monclient: dump: 2024-04-26T00:38:14.315 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:14.315 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:14.315 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:14.315 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:14.315 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:14.315 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:14.315 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:14.315 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:14.315 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:14.315 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:14.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa272ee7640 10 monclient: shutdown 2024-04-26T00:38:14.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa272ee7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:38:14.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa272ee7640 1 -- 172.21.15.37:0/1711736614 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa26c0a4170 msgr2=0x7fa26c0a45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:14.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa272ee7640 1 --2- 172.21.15.37:0/1711736614 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa26c0a4170 0x7fa26c0a45b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa2580098a0 tx=0x7fa25801c3c0 comp rx=0 tx=0).stop 2024-04-26T00:38:14.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa272ee7640 1 -- 172.21.15.37:0/1711736614 shutdown_connections 2024-04-26T00:38:14.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa272ee7640 1 --2- 172.21.15.37:0/1711736614 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa26c0a4170 0x7fa26c0a45b0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:14.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa272ee7640 1 -- 172.21.15.37:0/1711736614 >> 172.21.15.37:0/1711736614 conn(0x7fa26c0aa0a0 msgr2=0x7fa26c0a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:38:14.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa272ee7640 1 -- 172.21.15.37:0/1711736614 shutdown_connections 2024-04-26T00:38:14.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa272ee7640 1 -- 172.21.15.37:0/1711736614 wait complete. 2024-04-26T00:38:14.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa272ee7640 10 monclient: build_initial_monmap 2024-04-26T00:38:14.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa272ee7640 10 monclient: monmap: 2024-04-26T00:38:14.316 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:38:14.316 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:38:14.316 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:38:14.316 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:38:14.316 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:38:14.316 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:38:14.317 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:38:14.317 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:38:14.317 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:38:14.317 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:14.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa272ee7640 1 Processor -- start 2024-04-26T00:38:14.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa272ee7640 1 -- start start 2024-04-26T00:38:14.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa272ee7640 10 monclient: init 2024-04-26T00:38:14.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa272ee7640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:14.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa272ee7640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:38:14.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa272ee7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa26c0a3d90 0x7fa26c1cf0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:14.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa272ee7640 10 monclient(hunting): picked mon.noname-b con 0x7fa26c0a3d90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:38:14.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa272ee7640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa26c0a4170 0x7fa26c136f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:14.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa272ee7640 10 monclient(hunting): picked mon.noname-a con 0x7fa26c0a4170 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:38:14.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa272ee7640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa26c13c3e0 0x7fa26c1391a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:14.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa272ee7640 10 monclient(hunting): picked mon.noname-c con 0x7fa26c13c3e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:38:14.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa272ee7640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:14.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa272ee7640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa26c142aa0 con 0x7fa26c0a4170 2024-04-26T00:38:14.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa272ee7640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:14.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa272ee7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa26c142920 con 0x7fa26c0a3d90 2024-04-26T00:38:14.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa272ee7640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:14.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa272ee7640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa26c142c20 con 0x7fa26c13c3e0 2024-04-26T00:38:14.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa272ee7640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:14.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa272ee7640 10 monclient(hunting): authenticate will time out at 2075.618836s 2024-04-26T00:38:14.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa26bfff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa26c0a4170 0x7fa26c136f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:14.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa271484640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa26c13c3e0 0x7fa26c1391a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:14.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa26bfff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa26c0a4170 0x7fa26c136f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:44708/0 (socket says 172.21.15.37:44708) 2024-04-26T00:38:14.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa26bfff640 1 -- 172.21.15.37:0/2578855764 learned_addr learned my addr 172.21.15.37:0/2578855764 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:38:14.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa26bfff640 10 monclient(hunting): get_auth_request con 0x7fa26c0a4170 auth_method 0 2024-04-26T00:38:14.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa26bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:14.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa26bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:14.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa26bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:14.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa271484640 10 monclient(hunting): get_auth_request con 0x7fa26c13c3e0 auth_method 0 2024-04-26T00:38:14.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa271484640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:14.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa271484640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:14.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa271484640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:14.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa26bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:14.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa26bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:14.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa26bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:14.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa271484640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:14.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa271484640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:14.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa271484640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:14.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa270c83640 1 --2- 172.21.15.37:0/2578855764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa26c0a3d90 0x7fa26c1cf0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:14.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa270c83640 10 monclient(hunting): get_auth_request con 0x7fa26c0a3d90 auth_method 0 2024-04-26T00:38:14.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa270c83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:14.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa270c83640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:14.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa270c83640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:14.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa26bfff640 10 monclient(hunting): handle_auth_done global_id 144933 payload 824 2024-04-26T00:38:14.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa26bfff640 1 -- 172.21.15.37:0/2578855764 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa26c13c3e0 msgr2=0x7fa26c1391a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:14.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa26bfff640 1 --2- 172.21.15.37:0/2578855764 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa26c13c3e0 0x7fa26c1391a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:14.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa26bfff640 1 -- 172.21.15.37:0/2578855764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa26c0a3d90 msgr2=0x7fa26c1cf0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:14.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa26bfff640 1 --2- 172.21.15.37:0/2578855764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa26c0a3d90 0x7fa26c1cf0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:14.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa26bfff640 10 monclient: _finish_hunting 0 2024-04-26T00:38:14.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa26bfff640 1 monclient: found mon.noname-a 2024-04-26T00:38:14.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa26bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:14.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa26bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:38:14.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa26bfff640 1 -- 172.21.15.37:0/2578855764 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa26c0a0800 con 0x7fa26c0a4170 2024-04-26T00:38:14.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.311+0000 7fa271484640 1 --2- 172.21.15.37:0/2578855764 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa26c13c3e0 0x7fa26c1391a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:38:14.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa26bfff640 1 -- 172.21.15.37:0/2578855764 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa26c0a4170 msgr2=0x7fa26c136f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:14.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa26bfff640 1 --2- 172.21.15.37:0/2578855764 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa26c0a4170 0x7fa26c136f40 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fa25801aa70 tx=0x7fa25801ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:14.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa2697fa640 1 -- 172.21.15.37:0/2578855764 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2580ba020 con 0x7fa26c0a4170 2024-04-26T00:38:14.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa2697fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:14.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa2697fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:38:14.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa2697fa640 10 monclient: dump: 2024-04-26T00:38:14.325 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:14.325 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:14.325 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:14.325 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:14.325 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:14.325 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:14.325 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:14.325 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:14.325 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:14.325 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:14.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa2697fa640 10 monclient: _finish_auth 0 2024-04-26T00:38:14.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa2697fa640 10 monclient: _check_auth_tickets 2024-04-26T00:38:14.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa2697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:14.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa272ee7640 5 monclient: authenticate success, global_id 144933 2024-04-26T00:38:14.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa2697fa640 1 -- 172.21.15.37:0/2578855764 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2580b6630 con 0x7fa26c0a4170 2024-04-26T00:38:14.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa272ee7640 10 monclient: _renew_subs 2024-04-26T00:38:14.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa272ee7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:38:14.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa272ee7640 1 -- 172.21.15.37:0/2578855764 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa26c1205e0 con 0x7fa26c0a4170 2024-04-26T00:38:14.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa2697fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:38:14.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa2697fa640 1 -- 172.21.15.37:0/2578855764 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2580b0040 con 0x7fa26c0a4170 2024-04-26T00:38:14.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa2697fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:14.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa2697fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:38:14.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa2697fa640 10 monclient: dump: 2024-04-26T00:38:14.327 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:14.327 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:14.327 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:14.327 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:14.327 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:14.327 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:14.327 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:14.327 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:14.327 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:14.327 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:14.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa272ee7640 10 monclient: _renew_subs 2024-04-26T00:38:14.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa272ee7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:38:14.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa272ee7640 1 -- 172.21.15.37:0/2578855764 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa26c13b840 con 0x7fa26c0a4170 2024-04-26T00:38:14.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa26a7fc640 4 set_mon_vals no callback set 2024-04-26T00:38:14.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa272ee7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:14.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa272ee7640 1 --2- 172.21.15.37:0/2578855764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa26c1396e0 0x7fa26c139ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:14.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa272ee7640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:14.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa272ee7640 1 -- 172.21.15.37:0/2578855764 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa26c13a000 con 0x7fa26c1396e0 2024-04-26T00:38:14.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa2697fa640 1 -- 172.21.15.37:0/2578855764 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa2580bd020 con 0x7fa26c0a4170 2024-04-26T00:38:14.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa270c83640 1 --2- 172.21.15.37:0/2578855764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa26c1396e0 0x7fa26c139ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:14.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa270c83640 10 monclient: get_auth_request con 0x7fa26c1396e0 auth_method 0 2024-04-26T00:38:14.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa270c83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:14.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa270c83640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:14.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa270c83640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:14.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa2697fa640 1 --2- 172.21.15.37:0/2578855764 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa250035e80 0x7fa250038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:14.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa2697fa640 1 -- 172.21.15.37:0/2578855764 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(719..719 src has 370..719) ==== 3943+0+0 (secure 0 0 0) 0x7fa2580e2ba0 con 0x7fa26c0a4170 2024-04-26T00:38:14.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa270c83640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:14.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa270c83640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:14.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa270c83640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:14.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa271484640 1 --2- 172.21.15.37:0/2578855764 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa250035e80 0x7fa250038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:14.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa271484640 10 monclient: get_auth_request con 0x7fa250035e80 auth_method 0 2024-04-26T00:38:14.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa270c83640 10 monclient(hunting): handle_auth_done global_id 224128 payload 824 2024-04-26T00:38:14.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.315+0000 7fa270c83640 1 -- 172.21.15.37:0/2578855764 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa26c04f510 con 0x7fa26c1396e0 2024-04-26T00:38:14.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.319+0000 7fa270c83640 1 -- 172.21.15.37:0/2578855764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa26c1396e0 msgr2=0x7fa26c139ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:14.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.319+0000 7fa270c83640 1 --2- 172.21.15.37:0/2578855764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa26c1396e0 0x7fa26c139ac0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa26000e4d0 tx=0x7fa26000e990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:14.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.319+0000 7fa271484640 1 --2- 172.21.15.37:0/2578855764 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa250035e80 0x7fa250038340 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fa26c13bff0 tx=0x7fa264004f40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:14.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.319+0000 7fa2697fa640 1 -- 172.21.15.37:0/2578855764 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa260007450 con 0x7fa26c1396e0 2024-04-26T00:38:14.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.319+0000 7fa2697fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:14.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.319+0000 7fa2697fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:38:14.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.319+0000 7fa2697fa640 10 monclient: dump: 2024-04-26T00:38:14.330 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:14.330 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:14.330 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:14.330 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:14.330 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:14.330 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:14.330 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:14.330 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:14.330 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:14.330 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:14.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.319+0000 7fa2697fa640 1 -- 172.21.15.37:0/2578855764 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa260007450 con 0x7fa26c1396e0 2024-04-26T00:38:14.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.319+0000 7fa2697fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:14.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.319+0000 7fa2697fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:38:14.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.319+0000 7fa2697fa640 1 -- 172.21.15.37:0/2578855764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa26c1396e0 msgr2=0x7fa26c139ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:14.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.319+0000 7fa2697fa640 1 --2- 172.21.15.37:0/2578855764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa26c1396e0 0x7fa26c139ac0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa26000e4d0 tx=0x7fa26000e990 comp rx=0 tx=0).stop 2024-04-26T00:38:14.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.319+0000 7fa2697fa640 1 -- 172.21.15.37:0/2578855764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa26c1396e0 msgr2=0x7fa26c139ac0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:38:14.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.319+0000 7fa2697fa640 1 --2- 172.21.15.37:0/2578855764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa26c1396e0 0x7fa26c139ac0 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa26000e4d0 tx=0x7fa26000e990 comp rx=0 tx=0).stop 2024-04-26T00:38:14.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.327+0000 7fa272ee7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:38:14.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.327+0000 7fa272ee7640 1 --2- 172.21.15.37:0/2578855764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa26c09d0b0 0x7fa26c09d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:14.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.327+0000 7fa272ee7640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:14.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.327+0000 7fa272ee7640 1 -- 172.21.15.37:0/2578855764 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa26c11fa70 con 0x7fa26c09d0b0 2024-04-26T00:38:14.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.327+0000 7fa270c83640 1 --2- 172.21.15.37:0/2578855764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa26c09d0b0 0x7fa26c09d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:14.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.327+0000 7fa270c83640 10 monclient: get_auth_request con 0x7fa26c09d0b0 auth_method 0 2024-04-26T00:38:14.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.327+0000 7fa270c83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:14.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.327+0000 7fa270c83640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:14.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.327+0000 7fa270c83640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:14.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.327+0000 7fa270c83640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:14.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.327+0000 7fa270c83640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:14.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.327+0000 7fa270c83640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:14.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.327+0000 7fa270c83640 10 monclient(hunting): handle_auth_done global_id 224131 payload 824 2024-04-26T00:38:14.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.327+0000 7fa270c83640 1 -- 172.21.15.37:0/2578855764 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa26c13ace0 con 0x7fa26c09d0b0 2024-04-26T00:38:14.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.331+0000 7fa270c83640 1 -- 172.21.15.37:0/2578855764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa26c09d0b0 msgr2=0x7fa26c09d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:14.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.331+0000 7fa270c83640 1 --2- 172.21.15.37:0/2578855764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa26c09d0b0 0x7fa26c09d4c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa26000e4d0 tx=0x7fa260003770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:14.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.331+0000 7fa2697fa640 1 -- 172.21.15.37:0/2578855764 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa26001e960 con 0x7fa26c09d0b0 2024-04-26T00:38:14.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.331+0000 7fa2697fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:14.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.331+0000 7fa2697fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:38:14.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.331+0000 7fa2697fa640 10 monclient: dump: 2024-04-26T00:38:14.334 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:14.334 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:14.334 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:14.334 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:14.335 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:14.335 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:14.335 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:14.335 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:14.335 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:14.335 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:14.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.331+0000 7fa2697fa640 1 -- 172.21.15.37:0/2578855764 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7fa260006e20 con 0x7fa26c09d0b0 2024-04-26T00:38:14.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.331+0000 7fa2697fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:38:14.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.331+0000 7fa2697fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:38:14.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.331+0000 7fa2697fa640 1 -- 172.21.15.37:0/2578855764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa26c09d0b0 msgr2=0x7fa26c09d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:14.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.331+0000 7fa2697fa640 1 --2- 172.21.15.37:0/2578855764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa26c09d0b0 0x7fa26c09d4c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa26000e4d0 tx=0x7fa260003770 comp rx=0 tx=0).stop 2024-04-26T00:38:14.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.331+0000 7fa2697fa640 1 -- 172.21.15.37:0/2578855764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa26c09d0b0 msgr2=0x7fa26c09d4c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:38:14.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.331+0000 7fa2697fa640 1 --2- 172.21.15.37:0/2578855764 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa26c09d0b0 0x7fa26c09d4c0 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa26000e4d0 tx=0x7fa260003770 comp rx=0 tx=0).stop 2024-04-26T00:38:14.336 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:38:14.336 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:38:14.336 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:38:14.336 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:38:14.336 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 138, 2024-04-26T00:38:14.336 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 7991, 2024-04-26T00:38:14.336 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:38:14.336 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:38:14.336 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:38:14.336 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:38:14.336 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:38:14.336 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 4, 2024-04-26T00:38:14.337 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:38:14.337 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:38:14.337 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:38:14.337 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:38:14.337 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:38:14.337 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:38:14.337 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:38:14.337 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:38:14.337 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:38:14.337 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:38:14.337 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:38:14.337 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:38:14.337 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:38:14.337 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:38:14.337 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:38:14.338 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:38:14.338 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:38:14.338 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:38:14.338 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:38:14.338 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:38:14.338 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:38:14.338 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:38:14.338 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:38:14.338 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:38:14.338 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:38:14.338 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:38:14.338 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:38:14.338 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:38:14.338 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:14.338 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:14.339 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:38:14.339 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:38:14.339 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:38:14.339 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:38:14.339 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:38:14.339 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:38:14.339 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:38:14.339 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:38:14.339 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:38:14.339 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:38:14.339 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:38:14.339 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:38:14.339 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:38:14.339 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:38:14.339 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:38:14.340 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:38:14.340 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:38:14.340 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:38:14.340 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:38:14.340 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:38:14.340 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:38:14.340 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:38:14.340 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:38:14.340 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:38:14.340 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:38:14.340 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:38:14.340 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:38:14.340 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:38:14.341 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:38:14.341 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:38:14.341 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:14.341 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:38:14.341 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:14.341 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:38:14.341 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:38:14.341 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:38:14.341 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:38:14.341 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:14.341 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:38:14.342 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:38:14.342 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:14.342 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:14.342 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:14.342 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:38:14.342 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:38:14.342 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:14.342 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:14.342 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:14.342 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:14.342 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:38:14.342 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:38:14.342 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:38:14.343 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:38:14.343 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:38:14.343 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:14.343 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:14.343 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:38:14.343 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:38:14.343 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:38:14.343 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:38:14.343 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:14.343 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:38:14.343 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:38:14.343 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:14.343 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:14.343 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:14.344 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:38:14.344 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:38:14.344 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:14.344 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:14.344 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:14.344 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:14.344 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:38:14.344 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:38:14.344 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:38:14.344 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:38:14.344 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:38:14.344 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:14.344 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:14.344 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:38:14.344 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:38:14.344 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:38:14.345 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:38:14.345 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:14.345 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:38:14.345 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:38:14.345 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:14.345 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:14.345 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:14.345 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:38:14.345 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:38:14.345 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:14.345 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:14.345 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:14.345 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:14.345 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:38:14.346 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:38:14.346 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:38:14.346 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:38:14.346 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:38:14.346 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:14.346 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:14.346 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:14.346 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:38:14.346 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:38:14.346 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:14.346 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:38:14.346 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:38:14.346 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:38:14.346 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:14.346 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:38:14.347 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:38:14.347 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:14.347 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:38:14.347 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:38:14.347 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:38:14.347 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:14.347 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:14.347 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:14.347 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:38:14.347 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:38:14.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.331+0000 7fa272ee7640 1 -- 172.21.15.37:0/2578855764 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa250035e80 msgr2=0x7fa250038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:14.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.331+0000 7fa272ee7640 1 --2- 172.21.15.37:0/2578855764 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa250035e80 0x7fa250038340 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fa26c13bff0 tx=0x7fa264004f40 comp rx=0 tx=0).stop 2024-04-26T00:38:14.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.331+0000 7fa272ee7640 10 monclient: shutdown 2024-04-26T00:38:14.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.331+0000 7fa272ee7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:38:14.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.331+0000 7fa272ee7640 1 -- 172.21.15.37:0/2578855764 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa26c0a4170 msgr2=0x7fa26c136f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:14.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.331+0000 7fa272ee7640 1 --2- 172.21.15.37:0/2578855764 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa26c0a4170 0x7fa26c136f40 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fa25801aa70 tx=0x7fa25801ac00 comp rx=0 tx=0).stop 2024-04-26T00:38:14.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.331+0000 7fa271484640 1 -- 172.21.15.37:0/2578855764 reap_dead start 2024-04-26T00:38:14.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.331+0000 7fa272ee7640 1 -- 172.21.15.37:0/2578855764 shutdown_connections 2024-04-26T00:38:14.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.331+0000 7fa272ee7640 1 --2- 172.21.15.37:0/2578855764 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa26c0a4170 0x7fa26c136f40 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:14.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.331+0000 7fa272ee7640 1 -- 172.21.15.37:0/2578855764 >> 172.21.15.37:0/2578855764 conn(0x7fa26c0aa0a0 msgr2=0x7fa26c0a12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:38:14.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.331+0000 7fa272ee7640 1 -- 172.21.15.37:0/2578855764 shutdown_connections 2024-04-26T00:38:14.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.331+0000 7fa272ee7640 1 -- 172.21.15.37:0/2578855764 wait complete. 2024-04-26T00:38:14.350 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:38:14.350 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-04-26T00:38:14.350 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-26T00:38:14.350 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-04-26T00:38:14.350 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-04-26T00:38:14.350 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-26T00:38:14.350 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T00:38:14.367 INFO:tasks.ceph.mon.a:Stopped 2024-04-26T00:38:14.367 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-26T00:38:14.368 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:38:14.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61e0012640 10 monclient: get_monmap_and_config 2024-04-26T00:38:14.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61e0012640 10 monclient: build_initial_monmap 2024-04-26T00:38:14.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61e0012640 10 monclient: monmap: 2024-04-26T00:38:14.525 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:38:14.525 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:14.525 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:38:14.529366+0000 2024-04-26T00:38:14.526 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:38:14.529366+0000 2024-04-26T00:38:14.526 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:38:14.526 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:14.526 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:38:14.526 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:38:14.526 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:38:14.526 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:38:14.526 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:38:14.526 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:14.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61e0012640 1 Processor -- start 2024-04-26T00:38:14.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61e0012640 1 -- start start 2024-04-26T00:38:14.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61e0012640 10 monclient: init 2024-04-26T00:38:14.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61e0012640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:14.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61e0012640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-04-26T00:38:14.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61e0012640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f61d8133b20 0x7f61d809e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:14.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61e0012640 10 monclient(hunting): picked mon.noname-a con 0x7f61d8133b20 addr v2:172.21.15.37:3300/0 2024-04-26T00:38:14.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61e0012640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61d8133f00 0x7f61d8134350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:14.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61e0012640 10 monclient(hunting): picked mon.noname-b con 0x7f61d8133f00 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:38:14.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61e0012640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f61d8130770 addr v1:172.21.15.45:6789/0 2024-04-26T00:38:14.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61e0012640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61d809fe20 con 0x7f61d8130770 2024-04-26T00:38:14.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61e0012640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:14.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61e0012640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f61d8072d00 con 0x7f61d8133b20 2024-04-26T00:38:14.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61e0012640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:14.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61e0012640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f61d809ba60 con 0x7f61d8133f00 2024-04-26T00:38:14.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61e0012640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:14.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61e0012640 10 monclient(hunting): authenticate will time out at 2075.835272s 2024-04-26T00:38:14.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61dddae640 1 -- >> v2:172.21.15.37:3300/0 conn(0x7f61d8133b20 msgr2=0x7f61d809e9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:38:14.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61dddae640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f61d8133b20 0x7f61d809e9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:38:14.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61dd5ad640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61d8133f00 0x7f61d8134350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:14.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61dd5ad640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61d8133f00 0x7f61d8134350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:53074/0 (socket says 172.21.15.37:53074) 2024-04-26T00:38:14.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61dd5ad640 1 -- 172.21.15.37:0/1647276221 learned_addr learned my addr 172.21.15.37:0/1647276221 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:38:14.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61dd5ad640 10 monclient(hunting): get_auth_request con 0x7f61d8133f00 auth_method 0 2024-04-26T00:38:14.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61dd5ad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:14.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61dd5ad640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:14.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61dd5ad640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:14.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61dd5ad640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:14.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61dd5ad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:14.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61dd5ad640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:14.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61dd5ad640 10 monclient(hunting): handle_auth_done global_id 204332 payload 274 2024-04-26T00:38:14.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61dd5ad640 1 -- 172.21.15.37:0/1647276221 >> v2:172.21.15.37:3300/0 conn(0x7f61d8133b20 msgr2=0x7f61d809e9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:38:14.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61dd5ad640 1 --2- 172.21.15.37:0/1647276221 >> v2:172.21.15.37:3300/0 conn(0x7f61d8133b20 0x7f61d809e9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:14.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61dd5ad640 1 -- 172.21.15.37:0/1647276221 >> v1:172.21.15.45:6789/0 conn(0x7f61d8130770 legacy=0x7f61d8130be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:38:14.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61dd5ad640 10 monclient: _finish_hunting 0 2024-04-26T00:38:14.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61dd5ad640 1 monclient: found mon.noname-b 2024-04-26T00:38:14.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61dd5ad640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:14.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61dd5ad640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:38:14.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.527+0000 7f61dd5ad640 1 -- 172.21.15.37:0/1647276221 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61d809dd10 con 0x7f61d8133f00 2024-04-26T00:38:14.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61dd5ad640 1 -- 172.21.15.37:0/1647276221 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61d8133f00 msgr2=0x7f61d8134350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:14.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61dd5ad640 1 --2- 172.21.15.37:0/1647276221 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61d8133f00 0x7f61d8134350 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f61c800bb10 tx=0x7f61c801e8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9abdc9dfd92c6505 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:14.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61dcdac640 1 -- 172.21.15.37:0/1647276221 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c80b4070 con 0x7f61d8133f00 2024-04-26T00:38:14.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61dcdac640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:14.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61dcdac640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:38:14.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61dcdac640 10 monclient: dump: 2024-04-26T00:38:14.533 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:14.533 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:14.533 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:14.533 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:14.533 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:14.533 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:14.534 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:14.534 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:14.534 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:14.534 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:14.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61dcdac640 10 monclient: _finish_auth 0 2024-04-26T00:38:14.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61dcdac640 10 monclient: _check_auth_tickets 2024-04-26T00:38:14.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61dcdac640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:14.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61dcdac640 1 -- 172.21.15.37:0/1647276221 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61c80a78c0 con 0x7f61d8133f00 2024-04-26T00:38:14.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61dcdac640 10 monclient: handle_config config(0 keys) 2024-04-26T00:38:14.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61dcdac640 1 -- 172.21.15.37:0/1647276221 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c80a7c90 con 0x7f61d8133f00 2024-04-26T00:38:14.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61e0012640 5 monclient: authenticate success, global_id 204332 2024-04-26T00:38:14.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61e0012640 10 monclient: get_monmap_and_config success 2024-04-26T00:38:14.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61e0012640 4 set_mon_vals no callback set 2024-04-26T00:38:14.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61dcdac640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:14.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61dcdac640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:38:14.535 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61dcdac640 10 monclient: dump: 2024-04-26T00:38:14.535 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:14.535 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:14.535 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:14.535 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:14.535 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:14.535 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:14.536 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:14.536 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:14.536 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:14.536 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:14.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61e0012640 10 monclient: shutdown 2024-04-26T00:38:14.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61e0012640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:38:14.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61e0012640 1 -- 172.21.15.37:0/1647276221 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61d8133f00 msgr2=0x7f61d8134350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:14.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61e0012640 1 --2- 172.21.15.37:0/1647276221 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61d8133f00 0x7f61d8134350 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f61c800bb10 tx=0x7f61c801e8c0 comp rx=0 tx=0).stop 2024-04-26T00:38:14.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61e0012640 1 -- 172.21.15.37:0/1647276221 shutdown_connections 2024-04-26T00:38:14.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61e0012640 1 --2- 172.21.15.37:0/1647276221 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61d8133f00 0x7f61d8134350 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:14.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61e0012640 1 --2- 172.21.15.37:0/1647276221 >> v2:172.21.15.37:3300/0 conn(0x7f61d8133b20 0x7f61d809e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:14.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61e0012640 1 -- 172.21.15.37:0/1647276221 >> 172.21.15.37:0/1647276221 conn(0x7f61d80a1ad0 msgr2=0x7f61d80a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:38:14.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61e0012640 1 -- 172.21.15.37:0/1647276221 shutdown_connections 2024-04-26T00:38:14.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61e0012640 1 -- 172.21.15.37:0/1647276221 wait complete. 2024-04-26T00:38:14.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61e0012640 10 monclient: build_initial_monmap 2024-04-26T00:38:14.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61e0012640 10 monclient: monmap: 2024-04-26T00:38:14.541 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:38:14.542 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:38:14.542 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:38:14.542 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:38:14.542 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:38:14.542 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:38:14.542 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:38:14.542 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:38:14.542 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:38:14.542 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:14.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61e0012640 1 Processor -- start 2024-04-26T00:38:14.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61e0012640 1 -- start start 2024-04-26T00:38:14.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61e0012640 10 monclient: init 2024-04-26T00:38:14.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61e0012640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:14.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61e0012640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:38:14.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61e0012640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61d8130770 0x7f61d813aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:14.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61e0012640 10 monclient(hunting): picked mon.noname-c con 0x7f61d8130770 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:38:14.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61e0012640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f61d8133b20 0x7f61d812f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:14.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61e0012640 10 monclient(hunting): picked mon.noname-a con 0x7f61d8133b20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:38:14.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61e0012640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61d8133f00 0x7f61d811b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:14.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61e0012640 10 monclient(hunting): picked mon.noname-b con 0x7f61d8133f00 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:38:14.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61e0012640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:14.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61e0012640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f61d80a1750 con 0x7f61d8133b20 2024-04-26T00:38:14.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61e0012640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:14.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61e0012640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f61d80a15d0 con 0x7f61d8133f00 2024-04-26T00:38:14.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61e0012640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:14.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61e0012640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f61d80a0a70 con 0x7f61d8130770 2024-04-26T00:38:14.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61dd5ad640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f61d8133b20 msgr2=0x7f61d812f7a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:38:14.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61e0012640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:14.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61e0012640 10 monclient(hunting): authenticate will time out at 2075.842877s 2024-04-26T00:38:14.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61dd5ad640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f61d8133b20 0x7f61d812f7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:38:14.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61dddae640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61d8130770 0x7f61d813aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:14.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61dddae640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61d8130770 0x7f61d813aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:53080/0 (socket says 172.21.15.37:53080) 2024-04-26T00:38:14.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61dddae640 1 -- 172.21.15.37:0/2961040496 learned_addr learned my addr 172.21.15.37:0/2961040496 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:38:14.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61dddae640 10 monclient(hunting): get_auth_request con 0x7f61d8130770 auth_method 0 2024-04-26T00:38:14.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61dddae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:14.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61dddae640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:14.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61dddae640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:14.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61dddae640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:14.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61dddae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:14.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61dddae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:14.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61de5af640 1 --2- 172.21.15.37:0/2961040496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61d8133f00 0x7f61d811b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:14.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61dddae640 10 monclient(hunting): handle_auth_done global_id 204335 payload 824 2024-04-26T00:38:14.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61dddae640 1 -- 172.21.15.37:0/2961040496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61d8133f00 msgr2=0x7f61d811b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:14.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61dddae640 1 --2- 172.21.15.37:0/2961040496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61d8133f00 0x7f61d811b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:14.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61dddae640 1 -- 172.21.15.37:0/2961040496 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f61d8133b20 msgr2=0x7f61d812f7a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:38:14.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61dddae640 1 --2- 172.21.15.37:0/2961040496 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f61d8133b20 0x7f61d812f7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:14.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61dddae640 10 monclient: _finish_hunting 0 2024-04-26T00:38:14.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61dddae640 1 monclient: found mon.noname-c 2024-04-26T00:38:14.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61dddae640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:14.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61dddae640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:38:14.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61dddae640 1 -- 172.21.15.37:0/2961040496 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61d813b630 con 0x7f61d8130770 2024-04-26T00:38:14.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61de5af640 10 monclient: get_auth_request con 0x7f61d8133f00 auth_method 0 2024-04-26T00:38:14.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.535+0000 7f61de5af640 1 --2- 172.21.15.37:0/2961040496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61d8133f00 0x7f61d811b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:38:14.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.539+0000 7f61dddae640 1 -- 172.21.15.37:0/2961040496 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61d8130770 msgr2=0x7f61d813aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:14.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.539+0000 7f61dddae640 1 --2- 172.21.15.37:0/2961040496 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61d8130770 0x7f61d813aa40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f61d400b580 tx=0x7f61d400ba40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:14.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.539+0000 7f61ce7fc640 1 -- 172.21.15.37:0/2961040496 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61d4016070 con 0x7f61d8130770 2024-04-26T00:38:14.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.539+0000 7f61ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:14.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.539+0000 7f61ce7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:38:14.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.539+0000 7f61ce7fc640 10 monclient: dump: 2024-04-26T00:38:14.546 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:14.546 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:14.547 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:14.547 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:14.547 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:14.547 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:14.547 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:14.547 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:14.547 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:14.547 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:14.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.539+0000 7f61ce7fc640 10 monclient: _finish_auth 0 2024-04-26T00:38:14.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.539+0000 7f61ce7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:38:14.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.539+0000 7f61ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:14.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.539+0000 7f61ce7fc640 1 -- 172.21.15.37:0/2961040496 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61d40050d0 con 0x7f61d8130770 2024-04-26T00:38:14.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.539+0000 7f61ce7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:38:14.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.539+0000 7f61ce7fc640 1 -- 172.21.15.37:0/2961040496 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61d40054c0 con 0x7f61d8130770 2024-04-26T00:38:14.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.539+0000 7f61ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:14.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.539+0000 7f61ce7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:38:14.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.539+0000 7f61ce7fc640 10 monclient: dump: 2024-04-26T00:38:14.549 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:14.549 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:14.549 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:14.549 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:14.549 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:14.549 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:14.549 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:14.549 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:14.549 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:14.549 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:14.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.539+0000 7f61e0012640 5 monclient: authenticate success, global_id 204335 2024-04-26T00:38:14.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.539+0000 7f61e0012640 10 monclient: _renew_subs 2024-04-26T00:38:14.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.539+0000 7f61cffff640 4 set_mon_vals no callback set 2024-04-26T00:38:14.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.539+0000 7f61e0012640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:14.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.539+0000 7f61e0012640 1 -- 172.21.15.37:0/2961040496 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61d811d8d0 con 0x7f61d8130770 2024-04-26T00:38:14.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.539+0000 7f61e0012640 10 monclient: _renew_subs 2024-04-26T00:38:14.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.539+0000 7f61e0012640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:14.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.539+0000 7f61e0012640 1 -- 172.21.15.37:0/2961040496 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f61d81302b0 con 0x7f61d8130770 2024-04-26T00:38:14.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.539+0000 7f61e0012640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:14.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.539+0000 7f61e0012640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:14.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.539+0000 7f61e0012640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:14.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.539+0000 7f61e0012640 1 -- 172.21.15.37:0/2961040496 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61d8066f40 con 0x7f61d8130770 2024-04-26T00:38:14.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.539+0000 7f61ce7fc640 1 -- 172.21.15.37:0/2961040496 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f61d40056a0 con 0x7f61d8130770 2024-04-26T00:38:14.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.539+0000 7f61ce7fc640 1 --2- 172.21.15.37:0/2961040496 >> v2:172.21.15.45:6812/2379918623 conn(0x7f61bc035e30 0x7f61bc0382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:14.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.539+0000 7f61ce7fc640 1 -- 172.21.15.37:0/2961040496 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(719..719 src has 370..719) ==== 3943+0+0 (secure 0 0 0) 0x7f61d4047d90 con 0x7f61d8130770 2024-04-26T00:38:14.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.539+0000 7f61dd5ad640 1 --2- 172.21.15.37:0/2961040496 >> v2:172.21.15.45:6812/2379918623 conn(0x7f61bc035e30 0x7f61bc0382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:14.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.539+0000 7f61dd5ad640 10 monclient: get_auth_request con 0x7f61bc035e30 auth_method 0 2024-04-26T00:38:14.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.543+0000 7f61dd5ad640 1 --2- 172.21.15.37:0/2961040496 >> v2:172.21.15.45:6812/2379918623 conn(0x7f61bc035e30 0x7f61bc0382f0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f61c80ae000 tx=0x7f61c8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:14.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.547+0000 7f61ce7fc640 1 -- 172.21.15.37:0/2961040496 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f61d400fcd0 con 0x7f61d8130770 2024-04-26T00:38:14.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.547+0000 7f61ce7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:14.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.547+0000 7f61ce7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:38:14.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.763+0000 7f61e0012640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:38:14.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.763+0000 7f61e0012640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:38:14.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.763+0000 7f61e0012640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:14.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.763+0000 7f61e0012640 1 -- 172.21.15.37:0/2961040496 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f61d811ddf0 con 0x7f61d8130770 2024-04-26T00:38:14.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.763+0000 7f61ce7fc640 1 -- 172.21.15.37:0/2961040496 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f61d400f7b0 con 0x7f61d8130770 2024-04-26T00:38:14.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.763+0000 7f61ce7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:38:14.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.763+0000 7f61ce7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:38:14.763 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":138,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:38:14.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.767+0000 7f61e0012640 1 -- 172.21.15.37:0/2961040496 >> v2:172.21.15.45:6812/2379918623 conn(0x7f61bc035e30 msgr2=0x7f61bc0382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:14.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.767+0000 7f61e0012640 1 --2- 172.21.15.37:0/2961040496 >> v2:172.21.15.45:6812/2379918623 conn(0x7f61bc035e30 0x7f61bc0382f0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f61c80ae000 tx=0x7f61c8008040 comp rx=0 tx=0).stop 2024-04-26T00:38:14.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.767+0000 7f61e0012640 10 monclient: shutdown 2024-04-26T00:38:14.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.767+0000 7f61e0012640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:38:14.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.767+0000 7f61e0012640 1 -- 172.21.15.37:0/2961040496 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61d8130770 msgr2=0x7f61d813aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:14.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.767+0000 7f61e0012640 1 --2- 172.21.15.37:0/2961040496 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61d8130770 0x7f61d813aa40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f61d400b580 tx=0x7f61d400ba40 comp rx=0 tx=0).stop 2024-04-26T00:38:14.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.767+0000 7f61e0012640 1 -- 172.21.15.37:0/2961040496 shutdown_connections 2024-04-26T00:38:14.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.767+0000 7f61e0012640 1 --2- 172.21.15.37:0/2961040496 >> v2:172.21.15.45:6812/2379918623 conn(0x7f61bc035e30 0x7f61bc0382f0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:14.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.767+0000 7f61e0012640 1 --2- 172.21.15.37:0/2961040496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61d8133f00 0x7f61d811b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:14.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.767+0000 7f61e0012640 1 --2- 172.21.15.37:0/2961040496 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f61d8133b20 0x7f61d812f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:14.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.767+0000 7f61e0012640 1 --2- 172.21.15.37:0/2961040496 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61d8130770 0x7f61d813aa40 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:14.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.767+0000 7f61e0012640 1 -- 172.21.15.37:0/2961040496 >> 172.21.15.37:0/2961040496 conn(0x7f61d80a1ad0 msgr2=0x7f61d809c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:38:14.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.767+0000 7f61e0012640 1 -- 172.21.15.37:0/2961040496 shutdown_connections 2024-04-26T00:38:14.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:14.767+0000 7f61e0012640 1 -- 172.21.15.37:0/2961040496 wait complete. 2024-04-26T00:38:17.781 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:38:17.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c3a3a8640 10 monclient: get_monmap_and_config 2024-04-26T00:38:17.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c3a3a8640 10 monclient: build_initial_monmap 2024-04-26T00:38:17.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c3a3a8640 10 monclient: monmap: 2024-04-26T00:38:17.914 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:38:17.914 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:17.914 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:38:17.917602+0000 2024-04-26T00:38:17.914 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:38:17.917602+0000 2024-04-26T00:38:17.914 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:38:17.914 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:17.914 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:38:17.914 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:38:17.914 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:38:17.914 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:38:17.914 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:38:17.914 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:17.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c3a3a8640 1 Processor -- start 2024-04-26T00:38:17.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c3a3a8640 1 -- start start 2024-04-26T00:38:17.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c3a3a8640 10 monclient: init 2024-04-26T00:38:17.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c3a3a8640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:17.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c3a3a8640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-04-26T00:38:17.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c3a3a8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9c340a0260 addr v1:172.21.15.37:6789/0 2024-04-26T00:38:17.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c3a3a8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9c34137e40 0x7f9c3413a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:17.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c3a3a8640 10 monclient(hunting): picked mon.noname-b con 0x7f9c34137e40 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:38:17.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c3a3a8640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9c3413a770 0x7f9c3413cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:17.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c3a3a8640 10 monclient(hunting): picked mon.noname-a con 0x7f9c3413a770 addr v2:172.21.15.37:3300/0 2024-04-26T00:38:17.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c3a3a8640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c340a1130 con 0x7f9c340a0260 2024-04-26T00:38:17.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c3a3a8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:17.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c3a3a8640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f9c340a0d50 con 0x7f9c3413a770 2024-04-26T00:38:17.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c3a3a8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:17.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c3a3a8640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9c3409ec10 con 0x7f9c34137e40 2024-04-26T00:38:17.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c3a3a8640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:17.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c3a3a8640 10 monclient(hunting): authenticate will time out at 2079.223269s 2024-04-26T00:38:17.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c33fff640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f9c340a0260 legacy=0x7f9c340a0640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:38:17.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c38945640 1 -- >> v2:172.21.15.37:3300/0 conn(0x7f9c3413a770 msgr2=0x7f9c3413cb60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:38:17.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c38945640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9c3413a770 0x7f9c3413cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:38:17.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c337fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9c34137e40 0x7f9c3413a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:17.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c337fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9c34137e40 0x7f9c3413a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:53096/0 (socket says 172.21.15.37:53096) 2024-04-26T00:38:17.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c337fe640 1 -- 172.21.15.37:0/2855977494 learned_addr learned my addr 172.21.15.37:0/2855977494 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:38:17.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c337fe640 10 monclient(hunting): get_auth_request con 0x7f9c34137e40 auth_method 0 2024-04-26T00:38:17.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:17.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c337fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:17.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c337fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:17.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:17.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:17.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:17.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c337fe640 10 monclient(hunting): handle_auth_done global_id 204338 payload 274 2024-04-26T00:38:17.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c337fe640 1 -- 172.21.15.37:0/2855977494 >> v2:172.21.15.37:3300/0 conn(0x7f9c3413a770 msgr2=0x7f9c3413cb60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:38:17.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c337fe640 1 --2- 172.21.15.37:0/2855977494 >> v2:172.21.15.37:3300/0 conn(0x7f9c3413a770 0x7f9c3413cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:17.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c337fe640 1 -- 172.21.15.37:0/2855977494 >> v1:172.21.15.37:6789/0 conn(0x7f9c340a0260 legacy=0x7f9c340a0640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:38:17.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c337fe640 10 monclient: _finish_hunting 0 2024-04-26T00:38:17.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c337fe640 1 monclient: found mon.noname-b 2024-04-26T00:38:17.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:17.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c337fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:38:17.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.915+0000 7f9c337fe640 1 -- 172.21.15.37:0/2855977494 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c3406cff0 con 0x7f9c34137e40 2024-04-26T00:38:17.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c337fe640 1 -- 172.21.15.37:0/2855977494 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9c34137e40 msgr2=0x7f9c3413a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:17.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c337fe640 1 --2- 172.21.15.37:0/2855977494 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9c34137e40 0x7f9c3413a230 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9c24009870 tx=0x7f9c2401c220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1f5b2842b61e9f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:17.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c32ffd640 1 -- 172.21.15.37:0/2855977494 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c240b0070 con 0x7f9c34137e40 2024-04-26T00:38:17.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:17.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c32ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:38:17.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c32ffd640 10 monclient: dump: 2024-04-26T00:38:17.925 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:17.925 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:17.925 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:17.925 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:17.926 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:17.926 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:17.926 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:17.926 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:17.926 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:17.926 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:17.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c32ffd640 10 monclient: _finish_auth 0 2024-04-26T00:38:17.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c32ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:38:17.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:17.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c32ffd640 1 -- 172.21.15.37:0/2855977494 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c240ab600 con 0x7f9c34137e40 2024-04-26T00:38:17.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c32ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:38:17.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c32ffd640 1 -- 172.21.15.37:0/2855977494 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c240ab9d0 con 0x7f9c34137e40 2024-04-26T00:38:17.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c3a3a8640 5 monclient: authenticate success, global_id 204338 2024-04-26T00:38:17.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c3a3a8640 10 monclient: get_monmap_and_config success 2024-04-26T00:38:17.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c3a3a8640 4 set_mon_vals no callback set 2024-04-26T00:38:17.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:17.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c32ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:38:17.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c32ffd640 10 monclient: dump: 2024-04-26T00:38:17.927 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:17.927 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:17.927 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:17.927 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:17.927 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:17.927 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:17.927 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:17.927 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:17.927 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:17.927 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:17.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c3a3a8640 10 monclient: shutdown 2024-04-26T00:38:17.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c3a3a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:38:17.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c3a3a8640 1 -- 172.21.15.37:0/2855977494 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9c34137e40 msgr2=0x7f9c3413a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:17.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c3a3a8640 1 --2- 172.21.15.37:0/2855977494 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9c34137e40 0x7f9c3413a230 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9c24009870 tx=0x7f9c2401c220 comp rx=0 tx=0).stop 2024-04-26T00:38:17.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c3a3a8640 1 -- 172.21.15.37:0/2855977494 shutdown_connections 2024-04-26T00:38:17.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c3a3a8640 1 --2- 172.21.15.37:0/2855977494 >> v2:172.21.15.37:3300/0 conn(0x7f9c3413a770 0x7f9c3413cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:17.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c3a3a8640 1 --2- 172.21.15.37:0/2855977494 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9c34137e40 0x7f9c3413a230 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:17.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c3a3a8640 1 -- 172.21.15.37:0/2855977494 >> 172.21.15.37:0/2855977494 conn(0x7f9c340a1ad0 msgr2=0x7f9c340a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:38:17.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c3a3a8640 1 -- 172.21.15.37:0/2855977494 shutdown_connections 2024-04-26T00:38:17.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c3a3a8640 1 -- 172.21.15.37:0/2855977494 wait complete. 2024-04-26T00:38:17.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c3a3a8640 10 monclient: build_initial_monmap 2024-04-26T00:38:17.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c3a3a8640 10 monclient: monmap: 2024-04-26T00:38:17.928 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:38:17.928 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:38:17.929 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:38:17.929 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:38:17.929 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:38:17.929 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:38:17.929 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:38:17.929 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:38:17.929 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:38:17.929 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:17.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c3a3a8640 1 Processor -- start 2024-04-26T00:38:17.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c3a3a8640 1 -- start start 2024-04-26T00:38:17.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c3a3a8640 10 monclient: init 2024-04-26T00:38:17.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c3a3a8640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:17.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c3a3a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:38:17.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c3a3a8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c340a0260 0x7f9c341ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:17.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c3a3a8640 10 monclient(hunting): picked mon.noname-b con 0x7f9c340a0260 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:38:17.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c3a3a8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9c34137e40 0x7f9c34134e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:17.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c3a3a8640 10 monclient(hunting): picked mon.noname-c con 0x7f9c34137e40 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:38:17.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c3a3a8640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c3413a770 0x7f9c3411ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:17.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c3a3a8640 10 monclient(hunting): picked mon.noname-a con 0x7f9c3413a770 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:38:17.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c3a3a8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:17.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c3a3a8640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9c3409f4c0 con 0x7f9c3413a770 2024-04-26T00:38:17.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c3a3a8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:17.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c3a3a8640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9c3409f0f0 con 0x7f9c340a0260 2024-04-26T00:38:17.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c3a3a8640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:17.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c3a3a8640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9c3413f8c0 con 0x7f9c34137e40 2024-04-26T00:38:17.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c3a3a8640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:17.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c3a3a8640 10 monclient(hunting): authenticate will time out at 2079.230449s 2024-04-26T00:38:17.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c38945640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c3413a770 msgr2=0x7f9c3411ba40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:38:17.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c38945640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c3413a770 0x7f9c3411ba40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:38:17.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c337fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9c34137e40 0x7f9c34134e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:17.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c337fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9c34137e40 0x7f9c34134e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:53098/0 (socket says 172.21.15.37:53098) 2024-04-26T00:38:17.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c337fe640 1 -- 172.21.15.37:0/2628466366 learned_addr learned my addr 172.21.15.37:0/2628466366 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:38:17.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c337fe640 10 monclient(hunting): get_auth_request con 0x7f9c34137e40 auth_method 0 2024-04-26T00:38:17.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:17.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c337fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:17.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c337fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:17.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:17.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:17.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:17.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c337fe640 10 monclient(hunting): handle_auth_done global_id 204341 payload 824 2024-04-26T00:38:17.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c33fff640 1 --2- 172.21.15.37:0/2628466366 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c340a0260 0x7f9c341ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:17.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c337fe640 1 -- 172.21.15.37:0/2628466366 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c340a0260 msgr2=0x7f9c341ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:17.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c337fe640 1 --2- 172.21.15.37:0/2628466366 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c340a0260 0x7f9c341ccd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:17.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c337fe640 1 -- 172.21.15.37:0/2628466366 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c3413a770 msgr2=0x7f9c3411ba40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:38:17.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c337fe640 1 --2- 172.21.15.37:0/2628466366 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c3413a770 0x7f9c3411ba40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:17.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c337fe640 10 monclient: _finish_hunting 0 2024-04-26T00:38:17.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c337fe640 1 monclient: found mon.noname-c 2024-04-26T00:38:17.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:17.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c337fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:38:17.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.923+0000 7f9c337fe640 1 -- 172.21.15.37:0/2628466366 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c340729d0 con 0x7f9c34137e40 2024-04-26T00:38:17.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.927+0000 7f9c337fe640 1 -- 172.21.15.37:0/2628466366 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9c34137e40 msgr2=0x7f9c34134e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:17.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.927+0000 7f9c337fe640 1 --2- 172.21.15.37:0/2628466366 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9c34137e40 0x7f9c34134e90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9c2401a8d0 tx=0x7f9c2401aa60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:17.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.927+0000 7f9c30ff9640 1 -- 172.21.15.37:0/2628466366 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c240ba020 con 0x7f9c34137e40 2024-04-26T00:38:17.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.927+0000 7f9c30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:17.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.927+0000 7f9c30ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:38:17.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.927+0000 7f9c30ff9640 10 monclient: dump: 2024-04-26T00:38:17.935 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:17.935 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:17.935 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:17.935 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:17.935 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:17.935 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:17.935 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:17.935 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:17.935 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:17.936 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:17.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.927+0000 7f9c30ff9640 10 monclient: _finish_auth 0 2024-04-26T00:38:17.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.927+0000 7f9c30ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:38:17.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.927+0000 7f9c30ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:17.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.927+0000 7f9c3a3a8640 5 monclient: authenticate success, global_id 204341 2024-04-26T00:38:17.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.927+0000 7f9c30ff9640 1 -- 172.21.15.37:0/2628466366 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c240b6830 con 0x7f9c34137e40 2024-04-26T00:38:17.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.927+0000 7f9c30ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:38:17.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.927+0000 7f9c3a3a8640 10 monclient: _renew_subs 2024-04-26T00:38:17.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.927+0000 7f9c30ff9640 1 -- 172.21.15.37:0/2628466366 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c240b0040 con 0x7f9c34137e40 2024-04-26T00:38:17.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.927+0000 7f9c3a3a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:17.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.927+0000 7f9c3a3a8640 1 -- 172.21.15.37:0/2628466366 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c34136170 con 0x7f9c34137e40 2024-04-26T00:38:17.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.927+0000 7f9c30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:17.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.927+0000 7f9c30ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:38:17.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.927+0000 7f9c30ff9640 10 monclient: dump: 2024-04-26T00:38:17.937 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:17.937 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:17.937 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:17.937 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:17.937 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:17.937 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:17.937 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:17.937 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:17.937 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:17.937 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:17.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.927+0000 7f9c327fc640 4 set_mon_vals no callback set 2024-04-26T00:38:17.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.927+0000 7f9c3a3a8640 10 monclient: _renew_subs 2024-04-26T00:38:17.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.927+0000 7f9c3a3a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:17.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.927+0000 7f9c3a3a8640 1 -- 172.21.15.37:0/2628466366 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c341cdb90 con 0x7f9c34137e40 2024-04-26T00:38:17.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.927+0000 7f9c30ff9640 1 -- 172.21.15.37:0/2628466366 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9c240bd020 con 0x7f9c34137e40 2024-04-26T00:38:17.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.927+0000 7f9c3a3a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:17.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.927+0000 7f9c3a3a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:17.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.927+0000 7f9c3a3a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:17.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.927+0000 7f9c3a3a8640 1 -- 172.21.15.37:0/2628466366 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c34066f40 con 0x7f9c34137e40 2024-04-26T00:38:17.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.927+0000 7f9c30ff9640 1 --2- 172.21.15.37:0/2628466366 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9c20035e30 0x7f9c200382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:17.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.927+0000 7f9c30ff9640 1 -- 172.21.15.37:0/2628466366 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(719..719 src has 370..719) ==== 3943+0+0 (secure 0 0 0) 0x7f9c240e2ef0 con 0x7f9c34137e40 2024-04-26T00:38:17.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.927+0000 7f9c33fff640 1 --2- 172.21.15.37:0/2628466366 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9c20035e30 0x7f9c200382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:17.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.927+0000 7f9c33fff640 10 monclient: get_auth_request con 0x7f9c20035e30 auth_method 0 2024-04-26T00:38:17.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.931+0000 7f9c33fff640 1 --2- 172.21.15.37:0/2628466366 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9c20035e30 0x7f9c200382f0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f9c28003990 tx=0x7f9c28008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:17.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.931+0000 7f9c30ff9640 1 -- 172.21.15.37:0/2628466366 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9c240c4030 con 0x7f9c34137e40 2024-04-26T00:38:17.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.931+0000 7f9c30ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:17.938 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:17.931+0000 7f9c30ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:38:18.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:18.151+0000 7f9c3a3a8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:38:18.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:18.151+0000 7f9c3a3a8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:38:18.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:18.151+0000 7f9c3a3a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:18.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:18.151+0000 7f9c3a3a8640 1 -- 172.21.15.37:0/2628466366 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9c3411d550 con 0x7f9c34137e40 2024-04-26T00:38:18.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:18.151+0000 7f9c30ff9640 1 -- 172.21.15.37:0/2628466366 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f9c34136170 con 0x7f9c34137e40 2024-04-26T00:38:18.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:18.151+0000 7f9c30ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:38:18.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:18.151+0000 7f9c30ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:38:18.150 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":138,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:38:18.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:18.155+0000 7f9c3a3a8640 1 -- 172.21.15.37:0/2628466366 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9c20035e30 msgr2=0x7f9c200382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:18.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:18.155+0000 7f9c3a3a8640 1 --2- 172.21.15.37:0/2628466366 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9c20035e30 0x7f9c200382f0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f9c28003990 tx=0x7f9c28008040 comp rx=0 tx=0).stop 2024-04-26T00:38:18.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:18.155+0000 7f9c3a3a8640 10 monclient: shutdown 2024-04-26T00:38:18.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:18.155+0000 7f9c3a3a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:38:18.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:18.155+0000 7f9c3a3a8640 1 -- 172.21.15.37:0/2628466366 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9c34137e40 msgr2=0x7f9c34134e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:18.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:18.155+0000 7f9c3a3a8640 1 --2- 172.21.15.37:0/2628466366 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9c34137e40 0x7f9c34134e90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9c2401a8d0 tx=0x7f9c2401aa60 comp rx=0 tx=0).stop 2024-04-26T00:38:18.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:18.155+0000 7f9c3a3a8640 1 -- 172.21.15.37:0/2628466366 shutdown_connections 2024-04-26T00:38:18.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:18.155+0000 7f9c3a3a8640 1 --2- 172.21.15.37:0/2628466366 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9c20035e30 0x7f9c200382f0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:18.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:18.155+0000 7f9c3a3a8640 1 --2- 172.21.15.37:0/2628466366 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c3413a770 0x7f9c3411ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:18.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:18.155+0000 7f9c3a3a8640 1 --2- 172.21.15.37:0/2628466366 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9c34137e40 0x7f9c34134e90 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:18.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:18.155+0000 7f9c3a3a8640 1 --2- 172.21.15.37:0/2628466366 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9c340a0260 0x7f9c341ccd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:18.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:18.155+0000 7f9c3a3a8640 1 -- 172.21.15.37:0/2628466366 >> 172.21.15.37:0/2628466366 conn(0x7f9c340a1ad0 msgr2=0x7f9c340a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:38:18.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:18.155+0000 7f9c3a3a8640 1 -- 172.21.15.37:0/2628466366 shutdown_connections 2024-04-26T00:38:18.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:18.155+0000 7f9c3a3a8640 1 -- 172.21.15.37:0/2628466366 wait complete. 2024-04-26T00:38:21.167 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:38:21.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.299+0000 7fd831282640 10 monclient: get_monmap_and_config 2024-04-26T00:38:21.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.299+0000 7fd831282640 10 monclient: build_initial_monmap 2024-04-26T00:38:21.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.299+0000 7fd831282640 10 monclient: monmap: 2024-04-26T00:38:21.298 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:38:21.298 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:21.298 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:38:21.301949+0000 2024-04-26T00:38:21.298 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:38:21.301949+0000 2024-04-26T00:38:21.299 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:38:21.299 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:21.299 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:38:21.299 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:38:21.299 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:38:21.299 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:38:21.299 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:38:21.299 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:21.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.299+0000 7fd831282640 1 Processor -- start 2024-04-26T00:38:21.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.299+0000 7fd831282640 1 -- start start 2024-04-26T00:38:21.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.299+0000 7fd831282640 10 monclient: init 2024-04-26T00:38:21.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.299+0000 7fd831282640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:21.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.299+0000 7fd831282640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-04-26T00:38:21.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.299+0000 7fd831282640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd82c0a0260 addr v1:172.21.15.45:6789/0 2024-04-26T00:38:21.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.299+0000 7fd831282640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd82c137e40 0x7fd82c13a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:21.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.299+0000 7fd831282640 10 monclient(hunting): picked mon.noname-a con 0x7fd82c137e40 addr v2:172.21.15.37:3300/0 2024-04-26T00:38:21.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.299+0000 7fd831282640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd82c13a770 0x7fd82c13cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:21.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.299+0000 7fd831282640 10 monclient(hunting): picked mon.noname-b con 0x7fd82c13a770 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:38:21.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.299+0000 7fd831282640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd82c0a1130 con 0x7fd82c0a0260 2024-04-26T00:38:21.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.299+0000 7fd831282640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:21.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.299+0000 7fd831282640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fd82c0a0d50 con 0x7fd82c137e40 2024-04-26T00:38:21.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.299+0000 7fd831282640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:21.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.299+0000 7fd831282640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd82c09ec10 con 0x7fd82c13a770 2024-04-26T00:38:21.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.299+0000 7fd831282640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:21.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.299+0000 7fd831282640 10 monclient(hunting): authenticate will time out at 2082.607674s 2024-04-26T00:38:21.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.299+0000 7fd82a7fc640 1 -- >> v2:172.21.15.37:3300/0 conn(0x7fd82c137e40 msgr2=0x7fd82c13a230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:38:21.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.299+0000 7fd82a7fc640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd82c137e40 0x7fd82c13a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:38:21.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.299+0000 7fd82b7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd82c13a770 0x7fd82c13cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:21.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.299+0000 7fd82b7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd82c13a770 0x7fd82c13cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:56322/0 (socket says 172.21.15.37:56322) 2024-04-26T00:38:21.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.299+0000 7fd82b7fe640 1 -- 172.21.15.37:0/3755748222 learned_addr learned my addr 172.21.15.37:0/3755748222 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:38:21.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.299+0000 7fd82b7fe640 10 monclient(hunting): get_auth_request con 0x7fd82c13a770 auth_method 0 2024-04-26T00:38:21.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.299+0000 7fd82b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:21.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.299+0000 7fd82b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:21.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.299+0000 7fd82b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:21.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.303+0000 7fd82b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:21.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.303+0000 7fd82b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:21.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.303+0000 7fd82b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:21.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.303+0000 7fd82b7fe640 10 monclient(hunting): handle_auth_done global_id 204344 payload 274 2024-04-26T00:38:21.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.303+0000 7fd82b7fe640 1 -- 172.21.15.37:0/3755748222 >> v2:172.21.15.37:3300/0 conn(0x7fd82c137e40 msgr2=0x7fd82c13a230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:38:21.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.303+0000 7fd82b7fe640 1 --2- 172.21.15.37:0/3755748222 >> v2:172.21.15.37:3300/0 conn(0x7fd82c137e40 0x7fd82c13a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:21.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.303+0000 7fd82b7fe640 1 -- 172.21.15.37:0/3755748222 >> v1:172.21.15.45:6789/0 conn(0x7fd82c0a0260 legacy=0x7fd82c0a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:38:21.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.303+0000 7fd82b7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:38:21.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.303+0000 7fd82b7fe640 1 monclient: found mon.noname-b 2024-04-26T00:38:21.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.303+0000 7fd82b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:21.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.303+0000 7fd82b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:38:21.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.303+0000 7fd82b7fe640 1 -- 172.21.15.37:0/3755748222 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd82c06cff0 con 0x7fd82c13a770 2024-04-26T00:38:21.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd82b7fe640 1 -- 172.21.15.37:0/3755748222 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd82c13a770 msgr2=0x7fd82c13cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:21.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd82b7fe640 1 --2- 172.21.15.37:0/3755748222 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd82c13a770 0x7fd82c13cb60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd824009870 tx=0x7fd82401c220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ea50bebb0d04bd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:21.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd829ffb640 1 -- 172.21.15.37:0/3755748222 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8240b0070 con 0x7fd82c13a770 2024-04-26T00:38:21.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd829ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:21.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd829ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:38:21.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd829ffb640 10 monclient: dump: 2024-04-26T00:38:21.306 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:21.306 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:21.306 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:21.306 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:21.306 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:21.307 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:21.307 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:21.307 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:21.307 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:21.307 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:21.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd829ffb640 10 monclient: _finish_auth 0 2024-04-26T00:38:21.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd829ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:38:21.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd829ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:21.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd829ffb640 1 -- 172.21.15.37:0/3755748222 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8240ab600 con 0x7fd82c13a770 2024-04-26T00:38:21.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd831282640 5 monclient: authenticate success, global_id 204344 2024-04-26T00:38:21.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd831282640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:38:21.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd829ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:38:21.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd829ffb640 1 -- 172.21.15.37:0/3755748222 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8240ab9d0 con 0x7fd82c13a770 2024-04-26T00:38:21.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd831282640 10 monclient: get_monmap_and_config success 2024-04-26T00:38:21.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd831282640 4 set_mon_vals no callback set 2024-04-26T00:38:21.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd829ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:21.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd829ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:38:21.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd829ffb640 10 monclient: dump: 2024-04-26T00:38:21.308 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:21.308 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:21.308 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:21.308 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:21.308 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:21.308 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:21.308 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:21.308 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:21.308 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:21.308 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:21.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd831282640 10 monclient: shutdown 2024-04-26T00:38:21.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd831282640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:38:21.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd831282640 1 -- 172.21.15.37:0/3755748222 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd82c13a770 msgr2=0x7fd82c13cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:21.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd831282640 1 --2- 172.21.15.37:0/3755748222 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd82c13a770 0x7fd82c13cb60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd824009870 tx=0x7fd82401c220 comp rx=0 tx=0).stop 2024-04-26T00:38:21.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd831282640 1 -- 172.21.15.37:0/3755748222 shutdown_connections 2024-04-26T00:38:21.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd831282640 1 --2- 172.21.15.37:0/3755748222 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd82c13a770 0x7fd82c13cb60 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:21.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd831282640 1 --2- 172.21.15.37:0/3755748222 >> v2:172.21.15.37:3300/0 conn(0x7fd82c137e40 0x7fd82c13a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:21.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd831282640 1 -- 172.21.15.37:0/3755748222 >> 172.21.15.37:0/3755748222 conn(0x7fd82c0a1ad0 msgr2=0x7fd82c0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:38:21.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd831282640 1 -- 172.21.15.37:0/3755748222 shutdown_connections 2024-04-26T00:38:21.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd831282640 1 -- 172.21.15.37:0/3755748222 wait complete. 2024-04-26T00:38:21.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd831282640 10 monclient: build_initial_monmap 2024-04-26T00:38:21.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd831282640 10 monclient: monmap: 2024-04-26T00:38:21.309 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:38:21.309 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:38:21.309 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:38:21.309 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:38:21.310 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:38:21.310 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:38:21.310 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:38:21.310 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:38:21.310 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:38:21.310 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:21.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd831282640 1 Processor -- start 2024-04-26T00:38:21.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd831282640 1 -- start start 2024-04-26T00:38:21.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd831282640 10 monclient: init 2024-04-26T00:38:21.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd831282640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:21.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd831282640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:38:21.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd831282640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd82c0a0260 0x7fd82c1cc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:21.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd831282640 10 monclient(hunting): picked mon.noname-b con 0x7fd82c0a0260 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:38:21.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd831282640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd82c137e40 0x7fd82c134bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:21.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd831282640 10 monclient(hunting): picked mon.noname-a con 0x7fd82c137e40 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:38:21.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd831282640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd82c13a770 0x7fd82c11b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:21.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd831282640 10 monclient(hunting): picked mon.noname-c con 0x7fd82c13a770 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:38:21.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd831282640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:21.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd831282640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd82c09f4c0 con 0x7fd82c137e40 2024-04-26T00:38:21.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd831282640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:21.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd831282640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd82c09f0f0 con 0x7fd82c0a0260 2024-04-26T00:38:21.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd831282640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:21.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd831282640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd82c13f8c0 con 0x7fd82c13a770 2024-04-26T00:38:21.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd831282640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:21.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd831282640 10 monclient(hunting): authenticate will time out at 2082.614823s 2024-04-26T00:38:21.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd82a7fc640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd82c137e40 msgr2=0x7fd82c134bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:38:21.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd82a7fc640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd82c137e40 0x7fd82c134bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:38:21.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd82b7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd82c13a770 0x7fd82c11b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:21.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd82b7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd82c13a770 0x7fd82c11b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:56326/0 (socket says 172.21.15.37:56326) 2024-04-26T00:38:21.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd82b7fe640 1 -- 172.21.15.37:0/1103914354 learned_addr learned my addr 172.21.15.37:0/1103914354 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:38:21.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd82b7fe640 10 monclient(hunting): get_auth_request con 0x7fd82c13a770 auth_method 0 2024-04-26T00:38:21.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd82b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:21.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd82b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:21.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd82b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:21.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd82b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:21.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd82b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:21.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd82b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:21.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd82affd640 1 --2- 172.21.15.37:0/1103914354 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd82c0a0260 0x7fd82c1cc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:21.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd82b7fe640 10 monclient(hunting): handle_auth_done global_id 204347 payload 824 2024-04-26T00:38:21.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd82b7fe640 1 -- 172.21.15.37:0/1103914354 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd82c0a0260 msgr2=0x7fd82c1cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:21.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd82b7fe640 1 --2- 172.21.15.37:0/1103914354 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd82c0a0260 0x7fd82c1cc9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:21.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd82b7fe640 1 -- 172.21.15.37:0/1103914354 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd82c137e40 msgr2=0x7fd82c134bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:38:21.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd82b7fe640 1 --2- 172.21.15.37:0/1103914354 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd82c137e40 0x7fd82c134bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:21.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd82b7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:38:21.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd82b7fe640 1 monclient: found mon.noname-c 2024-04-26T00:38:21.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd82b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:21.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd82b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:38:21.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd82b7fe640 1 -- 172.21.15.37:0/1103914354 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd82c11f570 con 0x7fd82c13a770 2024-04-26T00:38:21.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd82affd640 10 monclient: get_auth_request con 0x7fd82c0a0260 auth_method 0 2024-04-26T00:38:21.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.307+0000 7fd82affd640 1 --2- 172.21.15.37:0/1103914354 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd82c0a0260 0x7fd82c1cc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:38:21.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.311+0000 7fd82b7fe640 1 -- 172.21.15.37:0/1103914354 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd82c13a770 msgr2=0x7fd82c11b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:21.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.311+0000 7fd82b7fe640 1 --2- 172.21.15.37:0/1103914354 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd82c13a770 0x7fd82c11b620 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd82401a8d0 tx=0x7fd82401aa60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:21.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.311+0000 7fd82b7fe640 1 --2- 172.21.15.37:0/1103914354 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd82c13a770 0x7fd82c11b620 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd82401a8d0 tx=0x7fd82401aa60 comp rx=0 tx=0).queue_received will delay after 1.71409e+09 on 0x7fd8240b6630 config(0 keys) 2024-04-26T00:38:21.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.311+0000 7fd8237fe640 1 -- 172.21.15.37:0/1103914354 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8240ba020 con 0x7fd82c13a770 2024-04-26T00:38:21.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.311+0000 7fd8237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:21.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.311+0000 7fd8237fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:38:21.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.311+0000 7fd8237fe640 10 monclient: dump: 2024-04-26T00:38:21.318 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:21.318 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:21.318 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:21.318 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:21.318 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:21.318 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:21.318 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:21.318 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:21.318 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:21.318 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:21.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.311+0000 7fd8237fe640 10 monclient: _finish_auth 0 2024-04-26T00:38:21.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.311+0000 7fd8237fe640 10 monclient: _check_auth_tickets 2024-04-26T00:38:21.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.311+0000 7fd8237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:21.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.311+0000 7fd8237fe640 1 -- 172.21.15.37:0/1103914354 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8240b6630 con 0x7fd82c13a770 2024-04-26T00:38:21.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.311+0000 7fd831282640 5 monclient: authenticate success, global_id 204347 2024-04-26T00:38:21.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.311+0000 7fd831282640 10 monclient: _renew_subs 2024-04-26T00:38:21.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.311+0000 7fd831282640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:21.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.311+0000 7fd831282640 1 -- 172.21.15.37:0/1103914354 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd82c0729d0 con 0x7fd82c13a770 2024-04-26T00:38:21.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.311+0000 7fd8237fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:38:21.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.311+0000 7fd831282640 10 monclient: _renew_subs 2024-04-26T00:38:21.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.311+0000 7fd8297fa640 4 set_mon_vals no callback set 2024-04-26T00:38:21.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.311+0000 7fd831282640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:21.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.311+0000 7fd831282640 1 -- 172.21.15.37:0/1103914354 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd82c11d280 con 0x7fd82c13a770 2024-04-26T00:38:21.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.311+0000 7fd831282640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:21.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.311+0000 7fd831282640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:21.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.311+0000 7fd831282640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:21.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.311+0000 7fd831282640 1 -- 172.21.15.37:0/1103914354 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd82c066f40 con 0x7fd82c13a770 2024-04-26T00:38:21.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.311+0000 7fd8237fe640 1 -- 172.21.15.37:0/1103914354 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8240b0040 con 0x7fd82c13a770 2024-04-26T00:38:21.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.311+0000 7fd8237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:21.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.311+0000 7fd8237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:38:21.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.311+0000 7fd8237fe640 10 monclient: dump: 2024-04-26T00:38:21.320 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:21.320 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:21.320 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:21.320 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:21.320 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:21.320 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:21.320 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:21.320 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:21.321 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:21.321 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:21.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.311+0000 7fd8237fe640 1 -- 172.21.15.37:0/1103914354 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd8240bd020 con 0x7fd82c13a770 2024-04-26T00:38:21.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.311+0000 7fd8237fe640 1 --2- 172.21.15.37:0/1103914354 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd80c035e30 0x7fd80c0382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:21.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.311+0000 7fd82affd640 1 --2- 172.21.15.37:0/1103914354 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd80c035e30 0x7fd80c0382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:21.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.311+0000 7fd82affd640 10 monclient: get_auth_request con 0x7fd80c035e30 auth_method 0 2024-04-26T00:38:21.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.315+0000 7fd82affd640 1 --2- 172.21.15.37:0/1103914354 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd80c035e30 0x7fd80c0382f0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fd814003e40 tx=0x7fd814009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:21.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.315+0000 7fd8237fe640 1 -- 172.21.15.37:0/1103914354 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(719..719 src has 370..719) ==== 3943+0+0 (secure 0 0 0) 0x7fd8240e2ec0 con 0x7fd82c13a770 2024-04-26T00:38:21.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.935+0000 7fd8237fe640 1 -- 172.21.15.37:0/1103914354 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fd8240c4030 con 0x7fd82c13a770 2024-04-26T00:38:21.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.935+0000 7fd8237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:21.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:21.935+0000 7fd8237fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:38:22.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:22.163+0000 7fd831282640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:38:22.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:22.163+0000 7fd831282640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:38:22.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:22.163+0000 7fd831282640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:22.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:22.163+0000 7fd831282640 1 -- 172.21.15.37:0/1103914354 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd82c11f2f0 con 0x7fd82c13a770 2024-04-26T00:38:22.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:22.163+0000 7fd8237fe640 1 -- 172.21.15.37:0/1103914354 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1558 (secure 0 0 0) 0x7fd82c0a1130 con 0x7fd82c13a770 2024-04-26T00:38:22.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:22.163+0000 7fd8237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:38:22.163 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:22.163+0000 7fd8237fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:38:22.163 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":138,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:38:22.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:22.167+0000 7fd831282640 1 -- 172.21.15.37:0/1103914354 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd80c035e30 msgr2=0x7fd80c0382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:22.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:22.167+0000 7fd831282640 1 --2- 172.21.15.37:0/1103914354 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd80c035e30 0x7fd80c0382f0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fd814003e40 tx=0x7fd814009040 comp rx=0 tx=0).stop 2024-04-26T00:38:22.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:22.167+0000 7fd831282640 10 monclient: shutdown 2024-04-26T00:38:22.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:22.167+0000 7fd831282640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:38:22.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:22.167+0000 7fd831282640 1 -- 172.21.15.37:0/1103914354 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd82c13a770 msgr2=0x7fd82c11b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:22.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:22.167+0000 7fd831282640 1 --2- 172.21.15.37:0/1103914354 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd82c13a770 0x7fd82c11b620 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd82401a8d0 tx=0x7fd82401aa60 comp rx=0 tx=0).stop 2024-04-26T00:38:22.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:22.167+0000 7fd831282640 1 -- 172.21.15.37:0/1103914354 shutdown_connections 2024-04-26T00:38:22.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:22.167+0000 7fd831282640 1 --2- 172.21.15.37:0/1103914354 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd80c035e30 0x7fd80c0382f0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:22.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:22.167+0000 7fd831282640 1 --2- 172.21.15.37:0/1103914354 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd82c13a770 0x7fd82c11b620 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:22.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:22.167+0000 7fd831282640 1 --2- 172.21.15.37:0/1103914354 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd82c137e40 0x7fd82c134bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:22.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:22.167+0000 7fd831282640 1 --2- 172.21.15.37:0/1103914354 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd82c0a0260 0x7fd82c1cc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:22.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:22.167+0000 7fd831282640 1 -- 172.21.15.37:0/1103914354 >> 172.21.15.37:0/1103914354 conn(0x7fd82c0a1ad0 msgr2=0x7fd82c1375d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:38:22.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:22.167+0000 7fd831282640 1 -- 172.21.15.37:0/1103914354 shutdown_connections 2024-04-26T00:38:22.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:22.167+0000 7fd831282640 1 -- 172.21.15.37:0/1103914354 wait complete. 2024-04-26T00:38:25.182 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:38:25.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.315+0000 7eff6159c640 10 monclient: get_monmap_and_config 2024-04-26T00:38:25.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.315+0000 7eff6159c640 10 monclient: build_initial_monmap 2024-04-26T00:38:25.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.315+0000 7eff6159c640 10 monclient: monmap: 2024-04-26T00:38:25.314 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:38:25.314 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:25.314 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:38:25.317640+0000 2024-04-26T00:38:25.314 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:38:25.317640+0000 2024-04-26T00:38:25.314 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:38:25.314 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:25.314 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:38:25.314 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:38:25.315 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:38:25.315 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:38:25.315 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:38:25.315 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:25.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.315+0000 7eff6159c640 1 Processor -- start 2024-04-26T00:38:25.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.315+0000 7eff6159c640 1 -- start start 2024-04-26T00:38:25.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.315+0000 7eff6159c640 10 monclient: init 2024-04-26T00:38:25.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.315+0000 7eff6159c640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:25.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.315+0000 7eff6159c640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-04-26T00:38:25.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.315+0000 7eff6159c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7eff5c0a0260 addr v1:172.21.15.37:6789/0 2024-04-26T00:38:25.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.315+0000 7eff6159c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7eff5c137e40 0x7eff5c13a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:25.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.315+0000 7eff6159c640 10 monclient(hunting): picked mon.noname-c con 0x7eff5c137e40 addr v2:172.21.15.45:3300/0 2024-04-26T00:38:25.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.315+0000 7eff6159c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7eff5c13a770 0x7eff5c13cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:25.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.315+0000 7eff6159c640 10 monclient(hunting): picked mon.noname-a con 0x7eff5c13a770 addr v2:172.21.15.37:3300/0 2024-04-26T00:38:25.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.315+0000 7eff6159c640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff5c0a1130 con 0x7eff5c0a0260 2024-04-26T00:38:25.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.315+0000 7eff6159c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:25.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.315+0000 7eff6159c640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7eff5c0a0d50 con 0x7eff5c13a770 2024-04-26T00:38:25.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.315+0000 7eff6159c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:25.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.315+0000 7eff6159c640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7eff5c09ec10 con 0x7eff5c137e40 2024-04-26T00:38:25.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.315+0000 7eff6159c640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:25.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.315+0000 7eff6159c640 10 monclient(hunting): authenticate will time out at 2086.623284s 2024-04-26T00:38:25.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.315+0000 7eff5affd640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7eff5c0a0260 legacy=0x7eff5c0a0640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:38:25.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.315+0000 7eff5b7fe640 1 -- >> v2:172.21.15.37:3300/0 conn(0x7eff5c13a770 msgr2=0x7eff5c13cb60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:38:25.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.315+0000 7eff5b7fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7eff5c13a770 0x7eff5c13cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:38:25.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.315+0000 7eff5a7fc640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7eff5c137e40 0x7eff5c13a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:25.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.315+0000 7eff5a7fc640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7eff5c137e40 0x7eff5c13a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.37:51516/0 (socket says 172.21.15.37:51516) 2024-04-26T00:38:25.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.315+0000 7eff5a7fc640 1 -- 172.21.15.37:0/2393012523 learned_addr learned my addr 172.21.15.37:0/2393012523 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:38:25.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.315+0000 7eff5a7fc640 10 monclient(hunting): get_auth_request con 0x7eff5c137e40 auth_method 0 2024-04-26T00:38:25.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.315+0000 7eff5a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:25.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.315+0000 7eff5a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:25.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.315+0000 7eff5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:25.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.315+0000 7eff5a7fc640 1 -- 172.21.15.37:0/2393012523 >> v2:172.21.15.45:3300/0 conn(0x7eff5c137e40 msgr2=0x7eff5c13a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:38:25.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.315+0000 7eff5a7fc640 1 -- 172.21.15.37:0/2393012523 >> v2:172.21.15.45:3300/0 conn(0x7eff5c137e40 msgr2=0x7eff5c13a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:38:25.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.315+0000 7eff5a7fc640 1 --2- 172.21.15.37:0/2393012523 >> v2:172.21.15.45:3300/0 conn(0x7eff5c137e40 0x7eff5c13a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:38:25.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.319+0000 7eff5a7fc640 1 --2- 172.21.15.37:0/2393012523 >> v2:172.21.15.45:3300/0 conn(0x7eff5c137e40 0x7eff5c13a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:38:25.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.519+0000 7eff5affd640 1 -- 172.21.15.37:0/2393012523 >> v1:172.21.15.37:6789/0 conn(0x7eff5c0a0260 legacy=0x7eff5c0a0640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:38:25.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.519+0000 7eff5b7fe640 1 -- 172.21.15.37:0/2393012523 >> v2:172.21.15.37:3300/0 conn(0x7eff5c13a770 msgr2=0x7eff5c13cb60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:38:25.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.519+0000 7eff5b7fe640 1 --2- 172.21.15.37:0/2393012523 >> v2:172.21.15.37:3300/0 conn(0x7eff5c13a770 0x7eff5c13cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:38:25.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.523+0000 7eff5a7fc640 1 --2- 172.21.15.37:0/2393012523 >> v2:172.21.15.45:3300/0 conn(0x7eff5c137e40 0x7eff5c13a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:25.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.523+0000 7eff5a7fc640 10 monclient(hunting): get_auth_request con 0x7eff5c137e40 auth_method 0 2024-04-26T00:38:25.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.523+0000 7eff5a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:25.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.523+0000 7eff5a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:25.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.523+0000 7eff5a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:38:25.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.523+0000 7eff5a7fc640 1 -- 172.21.15.37:0/2393012523 >> v2:172.21.15.45:3300/0 conn(0x7eff5c137e40 msgr2=0x7eff5c13a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:38:25.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.523+0000 7eff5a7fc640 1 -- 172.21.15.37:0/2393012523 >> v2:172.21.15.45:3300/0 conn(0x7eff5c137e40 msgr2=0x7eff5c13a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:38:25.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.523+0000 7eff5a7fc640 1 --2- 172.21.15.37:0/2393012523 >> v2:172.21.15.45:3300/0 conn(0x7eff5c137e40 0x7eff5c13a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:38:25.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.523+0000 7eff5a7fc640 1 --2- 172.21.15.37:0/2393012523 >> v2:172.21.15.45:3300/0 conn(0x7eff5c137e40 0x7eff5c13a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:38:25.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.923+0000 7eff5affd640 1 -- 172.21.15.37:0/2393012523 >> v1:172.21.15.37:6789/0 conn(0x7eff5c0a0260 legacy=0x7eff5c0a0640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:38:25.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.923+0000 7eff5b7fe640 1 -- 172.21.15.37:0/2393012523 >> v2:172.21.15.37:3300/0 conn(0x7eff5c13a770 msgr2=0x7eff5c13cb60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:38:25.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.923+0000 7eff5b7fe640 1 --2- 172.21.15.37:0/2393012523 >> v2:172.21.15.37:3300/0 conn(0x7eff5c13a770 0x7eff5c13cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:38:25.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.927+0000 7eff5a7fc640 1 --2- 172.21.15.37:0/2393012523 >> v2:172.21.15.45:3300/0 conn(0x7eff5c137e40 0x7eff5c13a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:25.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.927+0000 7eff5a7fc640 10 monclient(hunting): get_auth_request con 0x7eff5c137e40 auth_method 0 2024-04-26T00:38:25.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.927+0000 7eff5a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:25.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.927+0000 7eff5a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:25.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.927+0000 7eff5a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:38:25.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.927+0000 7eff5a7fc640 1 -- 172.21.15.37:0/2393012523 >> v2:172.21.15.45:3300/0 conn(0x7eff5c137e40 msgr2=0x7eff5c13a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:38:25.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.927+0000 7eff5a7fc640 1 -- 172.21.15.37:0/2393012523 >> v2:172.21.15.45:3300/0 conn(0x7eff5c137e40 msgr2=0x7eff5c13a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:38:25.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.927+0000 7eff5a7fc640 1 --2- 172.21.15.37:0/2393012523 >> v2:172.21.15.45:3300/0 conn(0x7eff5c137e40 0x7eff5c13a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:38:25.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:25.927+0000 7eff5a7fc640 1 --2- 172.21.15.37:0/2393012523 >> v2:172.21.15.45:3300/0 conn(0x7eff5c137e40 0x7eff5c13a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:38:26.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:26.727+0000 7eff5b7fe640 1 -- 172.21.15.37:0/2393012523 >> v2:172.21.15.37:3300/0 conn(0x7eff5c13a770 msgr2=0x7eff5c13cb60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:38:26.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:26.727+0000 7eff5affd640 1 -- 172.21.15.37:0/2393012523 >> v1:172.21.15.37:6789/0 conn(0x7eff5c0a0260 legacy=0x7eff5c0a0640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:38:26.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:26.727+0000 7eff5b7fe640 1 --2- 172.21.15.37:0/2393012523 >> v2:172.21.15.37:3300/0 conn(0x7eff5c13a770 0x7eff5c13cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:38:26.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:26.731+0000 7eff5a7fc640 1 --2- 172.21.15.37:0/2393012523 >> v2:172.21.15.45:3300/0 conn(0x7eff5c137e40 0x7eff5c13a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:26.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:26.731+0000 7eff5a7fc640 10 monclient(hunting): get_auth_request con 0x7eff5c137e40 auth_method 0 2024-04-26T00:38:26.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:26.731+0000 7eff5a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:26.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:26.731+0000 7eff5a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:26.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:26.731+0000 7eff5a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:38:26.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:26.731+0000 7eff5a7fc640 1 -- 172.21.15.37:0/2393012523 >> v2:172.21.15.45:3300/0 conn(0x7eff5c137e40 msgr2=0x7eff5c13a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:38:26.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:26.731+0000 7eff5a7fc640 1 -- 172.21.15.37:0/2393012523 >> v2:172.21.15.45:3300/0 conn(0x7eff5c137e40 msgr2=0x7eff5c13a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:38:26.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:26.731+0000 7eff5a7fc640 1 --2- 172.21.15.37:0/2393012523 >> v2:172.21.15.45:3300/0 conn(0x7eff5c137e40 0x7eff5c13a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:38:26.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:26.731+0000 7eff5a7fc640 1 --2- 172.21.15.37:0/2393012523 >> v2:172.21.15.45:3300/0 conn(0x7eff5c137e40 0x7eff5c13a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:38:28.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.315+0000 7eff58ff9640 10 monclient(hunting): tick 2024-04-26T00:38:28.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.315+0000 7eff58ff9640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:38:28.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.315+0000 7eff58ff9640 1 monclient(hunting): continuing hunt 2024-04-26T00:38:28.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.315+0000 7eff58ff9640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:38:28.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.315+0000 7eff58ff9640 1 -- 172.21.15.37:0/2393012523 >> v2:172.21.15.45:3300/0 conn(0x7eff5c137e40 msgr2=0x7eff5c13a230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:38:28.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.315+0000 7eff58ff9640 1 --2- 172.21.15.37:0/2393012523 >> v2:172.21.15.45:3300/0 conn(0x7eff5c137e40 0x7eff5c13a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:28.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.315+0000 7eff58ff9640 1 -- 172.21.15.37:0/2393012523 >> v2:172.21.15.37:3300/0 conn(0x7eff5c13a770 msgr2=0x7eff5c13cb60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:38:28.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.315+0000 7eff58ff9640 1 --2- 172.21.15.37:0/2393012523 >> v2:172.21.15.37:3300/0 conn(0x7eff5c13a770 0x7eff5c13cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:28.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.315+0000 7eff58ff9640 1 -- 172.21.15.37:0/2393012523 >> v1:172.21.15.37:6789/0 conn(0x7eff5c0a0260 legacy=0x7eff5c0a0640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:38:28.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.315+0000 7eff58ff9640 10 monclient: _add_conns ranks=[2,1] 2024-04-26T00:38:28.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.315+0000 7eff58ff9640 1 --2- 172.21.15.37:0/2393012523 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7eff48003680 0x7eff48005ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:28.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.315+0000 7eff58ff9640 10 monclient(hunting): picked mon.noname-b con 0x7eff48003680 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:38:28.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.315+0000 7eff58ff9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7eff480061b0 addr v1:172.21.15.45:6789/0 2024-04-26T00:38:28.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.315+0000 7eff58ff9640 1 -- 172.21.15.37:0/2393012523 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff5c06cff0 con 0x7eff480061b0 2024-04-26T00:38:28.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.315+0000 7eff58ff9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:28.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.315+0000 7eff58ff9640 1 -- 172.21.15.37:0/2393012523 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7eff5c0a0d50 con 0x7eff48003680 2024-04-26T00:38:28.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.315+0000 7eff58ff9640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:28.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.315+0000 7eff5affd640 1 --2- 172.21.15.37:0/2393012523 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7eff48003680 0x7eff48005ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:28.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.319+0000 7eff5affd640 10 monclient(hunting): get_auth_request con 0x7eff48003680 auth_method 0 2024-04-26T00:38:28.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.319+0000 7eff5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:28.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.319+0000 7eff5affd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:28.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.319+0000 7eff5affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:28.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.319+0000 7eff5affd640 1 -- 172.21.15.37:0/2393012523 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7eff48003680 msgr2=0x7eff48005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:38:28.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.319+0000 7eff5affd640 1 -- 172.21.15.37:0/2393012523 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7eff48003680 msgr2=0x7eff48005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:38:28.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.319+0000 7eff5affd640 1 --2- 172.21.15.37:0/2393012523 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7eff48003680 0x7eff48005ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:38:28.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.319+0000 7eff5affd640 1 --2- 172.21.15.37:0/2393012523 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7eff48003680 0x7eff48005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:38:28.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.319+0000 7eff5a7fc640 1 -- 172.21.15.37:0/2393012523 >> v1:172.21.15.45:6789/0 conn(0x7eff480061b0 legacy=0x7eff48008670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:28.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.523+0000 7eff5affd640 1 --2- 172.21.15.37:0/2393012523 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7eff48003680 0x7eff48005ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:28.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.523+0000 7eff5affd640 10 monclient(hunting): get_auth_request con 0x7eff48003680 auth_method 0 2024-04-26T00:38:28.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.523+0000 7eff5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:28.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.523+0000 7eff5affd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:28.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.523+0000 7eff5affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:38:28.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.523+0000 7eff5affd640 1 -- 172.21.15.37:0/2393012523 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7eff48003680 msgr2=0x7eff48005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:38:28.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.523+0000 7eff5affd640 1 -- 172.21.15.37:0/2393012523 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7eff48003680 msgr2=0x7eff48005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:38:28.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.523+0000 7eff5affd640 1 --2- 172.21.15.37:0/2393012523 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7eff48003680 0x7eff48005ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:38:28.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.523+0000 7eff5affd640 1 --2- 172.21.15.37:0/2393012523 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7eff48003680 0x7eff48005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:38:28.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.819+0000 7eff59ffb640 1 -- 172.21.15.37:0/2393012523 <== mon.1 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3404552718 0 0) 0x7eff5c06cff0 con 0x7eff480061b0 2024-04-26T00:38:28.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.819+0000 7eff59ffb640 10 monclient(hunting): _negotiate 2024-04-26T00:38:28.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.819+0000 7eff59ffb640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:28.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.819+0000 7eff59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:28.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.819+0000 7eff59ffb640 10 monclient(hunting): my global_id is 224134 2024-04-26T00:38:28.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.819+0000 7eff59ffb640 1 -- 172.21.15.37:0/2393012523 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff44003530 con 0x7eff480061b0 2024-04-26T00:38:28.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff59ffb640 1 -- 172.21.15.37:0/2393012523 <== mon.1 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2690697486 0 0) 0x7eff44003530 con 0x7eff480061b0 2024-04-26T00:38:28.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff59ffb640 1 -- 172.21.15.37:0/2393012523 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7eff48003680 msgr2=0x7eff48005ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:38:28.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff59ffb640 1 --2- 172.21.15.37:0/2393012523 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7eff48003680 0x7eff48005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:28.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff59ffb640 10 monclient: _finish_hunting 0 2024-04-26T00:38:28.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff59ffb640 1 monclient: found mon.noname-c-legacy 2024-04-26T00:38:28.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff59ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:28.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff59ffb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.45:6789/0 2024-04-26T00:38:28.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff59ffb640 1 -- 172.21.15.37:0/2393012523 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff4800af70 con 0x7eff480061b0 2024-04-26T00:38:28.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff59ffb640 10 monclient: _finish_auth 0 2024-04-26T00:38:28.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff59ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:38:28.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff59ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:28.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff59ffb640 1 -- 172.21.15.37:0/2393012523 <== mon.1 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 866326386 0 0) 0x7eff5000f040 con 0x7eff480061b0 2024-04-26T00:38:28.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff59ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:28.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff59ffb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-04-26T00:38:28.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff59ffb640 10 monclient: dump: 2024-04-26T00:38:28.823 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:28.823 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:28.823 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:28.823 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:28.823 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:28.823 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:28.823 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:28.824 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:28.824 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:28.824 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:28.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff59ffb640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] but i'm connected to v1:172.21.15.45:6789/0, reconnecting 2024-04-26T00:38:28.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff59ffb640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:28.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff59ffb640 1 -- 172.21.15.37:0/2393012523 >> v1:172.21.15.45:6789/0 conn(0x7eff480061b0 legacy=0x7eff48008670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:28.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff5b7fe640 1 -- 172.21.15.37:0/2393012523 reap_dead start 2024-04-26T00:38:28.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff59ffb640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:38:28.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff59ffb640 1 --2- 172.21.15.37:0/2393012523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff440039c0 0x7eff44003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:28.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff59ffb640 10 monclient(hunting): picked mon.b con 0x7eff440039c0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:38:28.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff59ffb640 1 --2- 172.21.15.37:0/2393012523 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7eff44004810 0x7eff44004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:28.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff59ffb640 10 monclient(hunting): picked mon.a con 0x7eff44004810 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:38:28.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff59ffb640 1 --2- 172.21.15.37:0/2393012523 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7eff44005220 0x7eff4400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:28.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff59ffb640 10 monclient(hunting): picked mon.c con 0x7eff44005220 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:38:28.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff59ffb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:28.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff59ffb640 1 -- 172.21.15.37:0/2393012523 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7eff5c0a0d50 con 0x7eff44004810 2024-04-26T00:38:28.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff59ffb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:28.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff59ffb640 1 -- 172.21.15.37:0/2393012523 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7eff4400c150 con 0x7eff440039c0 2024-04-26T00:38:28.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff59ffb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:28.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff59ffb640 1 -- 172.21.15.37:0/2393012523 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7eff4400c350 con 0x7eff44005220 2024-04-26T00:38:28.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff59ffb640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:28.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff59ffb640 10 monclient(hunting): _finish_auth 0 2024-04-26T00:38:28.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff5a7fc640 1 -- 172.21.15.37:0/2393012523 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7eff44004810 msgr2=0x7eff44004c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:38:28.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff5a7fc640 1 --2- 172.21.15.37:0/2393012523 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7eff44004810 0x7eff44004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:38:28.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff5affd640 1 --2- 172.21.15.37:0/2393012523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff440039c0 0x7eff44003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:28.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff5affd640 10 monclient(hunting): get_auth_request con 0x7eff440039c0 auth_method 0 2024-04-26T00:38:28.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:28.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff5affd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:28.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff5affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:38:28.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff5affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:28.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff5affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:28.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff5affd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:38:28.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff5affd640 10 monclient(hunting): handle_auth_done global_id 224134 payload 293 2024-04-26T00:38:28.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff5affd640 1 -- 172.21.15.37:0/2393012523 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7eff44005220 msgr2=0x7eff4400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:38:28.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff5affd640 1 --2- 172.21.15.37:0/2393012523 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7eff44005220 0x7eff4400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:28.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff5affd640 1 -- 172.21.15.37:0/2393012523 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7eff44004810 msgr2=0x7eff44004c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:38:28.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff5affd640 1 --2- 172.21.15.37:0/2393012523 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7eff44004810 0x7eff44004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:28.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff5affd640 10 monclient: _finish_hunting 0 2024-04-26T00:38:28.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff5affd640 1 monclient: found mon.b 2024-04-26T00:38:28.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff5affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:28.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff5affd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.45:3300/0 2024-04-26T00:38:28.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff5affd640 1 -- 172.21.15.37:0/2393012523 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff44003530 con 0x7eff440039c0 2024-04-26T00:38:28.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff5affd640 10 monclient: _finish_auth 0 2024-04-26T00:38:28.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff5affd640 10 monclient: _check_auth_tickets 2024-04-26T00:38:28.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:28.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff6159c640 5 monclient: authenticate success, global_id 224134 2024-04-26T00:38:28.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.823+0000 7eff6159c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:38:28.829 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.831+0000 7eff5affd640 1 -- 172.21.15.37:0/2393012523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff440039c0 msgr2=0x7eff44003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:28.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.831+0000 7eff5affd640 1 --2- 172.21.15.37:0/2393012523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff440039c0 0x7eff44003dc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7eff4c003990 tx=0x7eff4c01d150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8212875e94a2cdc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:28.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.831+0000 7eff59ffb640 1 -- 172.21.15.37:0/2393012523 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff4c0ab070 con 0x7eff440039c0 2024-04-26T00:38:28.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.831+0000 7eff59ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:28.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.831+0000 7eff59ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:38:28.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.831+0000 7eff59ffb640 10 monclient: dump: 2024-04-26T00:38:28.833 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:28.833 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:28.833 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:28.833 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:28.833 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:28.833 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:28.833 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:28.834 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:28.834 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:28.834 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:28.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.831+0000 7eff59ffb640 1 -- 172.21.15.37:0/2393012523 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff4c0a5e40 con 0x7eff440039c0 2024-04-26T00:38:28.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.831+0000 7eff6159c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:38:28.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.831+0000 7eff59ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:38:28.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.831+0000 7eff6159c640 10 monclient: get_monmap_and_config success 2024-04-26T00:38:28.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.831+0000 7eff59ffb640 1 -- 172.21.15.37:0/2393012523 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff4c009b50 con 0x7eff440039c0 2024-04-26T00:38:28.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.831+0000 7eff6159c640 4 set_mon_vals no callback set 2024-04-26T00:38:28.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.831+0000 7eff59ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:28.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.831+0000 7eff59ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:38:28.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.831+0000 7eff59ffb640 10 monclient: dump: 2024-04-26T00:38:28.835 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:28.836 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:28.836 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:28.836 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:28.836 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:28.836 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:28.836 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:28.836 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:28.836 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:28.836 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:28.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.835+0000 7eff6159c640 10 monclient: shutdown 2024-04-26T00:38:28.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.835+0000 7eff6159c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:38:28.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.835+0000 7eff6159c640 1 -- 172.21.15.37:0/2393012523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff440039c0 msgr2=0x7eff44003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:28.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.835+0000 7eff6159c640 1 --2- 172.21.15.37:0/2393012523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff440039c0 0x7eff44003dc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7eff4c003990 tx=0x7eff4c01d150 comp rx=0 tx=0).stop 2024-04-26T00:38:28.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.835+0000 7eff6159c640 1 -- 172.21.15.37:0/2393012523 shutdown_connections 2024-04-26T00:38:28.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.835+0000 7eff6159c640 1 --2- 172.21.15.37:0/2393012523 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7eff44005220 0x7eff4400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:28.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.835+0000 7eff6159c640 1 --2- 172.21.15.37:0/2393012523 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7eff44004810 0x7eff44004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:28.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.835+0000 7eff6159c640 1 --2- 172.21.15.37:0/2393012523 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff440039c0 0x7eff44003dc0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:28.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.835+0000 7eff6159c640 1 -- 172.21.15.37:0/2393012523 >> 172.21.15.37:0/2393012523 conn(0x7eff5c0a1ad0 msgr2=0x7eff5c0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:38:28.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.835+0000 7eff6159c640 1 -- 172.21.15.37:0/2393012523 shutdown_connections 2024-04-26T00:38:28.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.835+0000 7eff6159c640 1 -- 172.21.15.37:0/2393012523 wait complete. 2024-04-26T00:38:28.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.835+0000 7eff6159c640 10 monclient: build_initial_monmap 2024-04-26T00:38:28.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.835+0000 7eff6159c640 10 monclient: monmap: 2024-04-26T00:38:28.844 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:38:28.844 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:38:28.844 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:38:28.844 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:38:28.844 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:38:28.844 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:38:28.844 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:38:28.844 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:38:28.844 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:38:28.844 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:28.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.835+0000 7eff6159c640 1 Processor -- start 2024-04-26T00:38:28.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.835+0000 7eff6159c640 1 -- start start 2024-04-26T00:38:28.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.835+0000 7eff6159c640 10 monclient: init 2024-04-26T00:38:28.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff6159c640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:28.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff6159c640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:38:28.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff6159c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7eff440039c0 0x7eff5c1cebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:28.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff6159c640 10 monclient(hunting): picked mon.noname-c con 0x7eff440039c0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:38:28.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff6159c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7eff44004810 0x7eff5c1cf0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:28.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff6159c640 10 monclient(hunting): picked mon.noname-a con 0x7eff44004810 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:38:28.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff6159c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff44005220 0x7eff5c0709c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:28.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff6159c640 10 monclient(hunting): picked mon.noname-b con 0x7eff44005220 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:38:28.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff6159c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:28.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff6159c640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7eff5c09f0d0 con 0x7eff44004810 2024-04-26T00:38:28.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff6159c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:28.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff6159c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7eff5c09f4c0 con 0x7eff44005220 2024-04-26T00:38:28.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff6159c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:28.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff5a7fc640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7eff44004810 msgr2=0x7eff5c1cf0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:38:28.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff6159c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7eff5c09fe90 con 0x7eff440039c0 2024-04-26T00:38:28.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff5a7fc640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7eff44004810 0x7eff5c1cf0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:38:28.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff6159c640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:28.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff6159c640 10 monclient(hunting): authenticate will time out at 2090.144775s 2024-04-26T00:38:28.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff5affd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7eff440039c0 0x7eff5c1cebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:28.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff5affd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7eff440039c0 0x7eff5c1cebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:56366/0 (socket says 172.21.15.37:56366) 2024-04-26T00:38:28.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff5affd640 1 -- 172.21.15.37:0/314618857 learned_addr learned my addr 172.21.15.37:0/314618857 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:38:28.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff5affd640 10 monclient(hunting): get_auth_request con 0x7eff440039c0 auth_method 0 2024-04-26T00:38:28.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:28.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff5affd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:28.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff5affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:28.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff5affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:28.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff5affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:28.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff5affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:28.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff5b7fe640 1 --2- 172.21.15.37:0/314618857 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff44005220 0x7eff5c0709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:28.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff5b7fe640 10 monclient(hunting): get_auth_request con 0x7eff44005220 auth_method 0 2024-04-26T00:38:28.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:28.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff5b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:28.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:28.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff5affd640 10 monclient(hunting): handle_auth_done global_id 204350 payload 824 2024-04-26T00:38:28.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff5affd640 1 -- 172.21.15.37:0/314618857 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff44005220 msgr2=0x7eff5c0709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:28.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff5affd640 1 --2- 172.21.15.37:0/314618857 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff44005220 0x7eff5c0709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:28.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff5affd640 1 -- 172.21.15.37:0/314618857 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7eff44004810 msgr2=0x7eff5c1cf0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:38:28.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff5affd640 1 --2- 172.21.15.37:0/314618857 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7eff44004810 0x7eff5c1cf0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:28.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff5affd640 10 monclient: _finish_hunting 0 2024-04-26T00:38:28.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff5affd640 1 monclient: found mon.noname-c 2024-04-26T00:38:28.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff5affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:28.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff5affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:38:28.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff5affd640 1 -- 172.21.15.37:0/314618857 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff5c121ce0 con 0x7eff440039c0 2024-04-26T00:38:28.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff5affd640 1 -- 172.21.15.37:0/314618857 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7eff440039c0 msgr2=0x7eff5c1cebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:28.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff5affd640 1 --2- 172.21.15.37:0/314618857 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7eff440039c0 0x7eff5c1cebb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7eff4c000eb0 tx=0x7eff44005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:28.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff437fe640 1 -- 172.21.15.37:0/314618857 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff4c0ab040 con 0x7eff440039c0 2024-04-26T00:38:28.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.839+0000 7eff437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:28.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.843+0000 7eff437fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:38:28.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.843+0000 7eff437fe640 10 monclient: dump: 2024-04-26T00:38:28.849 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:28.849 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:28.849 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:28.849 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:28.849 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:28.849 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:28.849 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:28.849 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:28.849 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:28.849 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:28.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.843+0000 7eff437fe640 10 monclient: _finish_auth 0 2024-04-26T00:38:28.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.843+0000 7eff437fe640 10 monclient: _check_auth_tickets 2024-04-26T00:38:28.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.843+0000 7eff437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:28.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.843+0000 7eff437fe640 1 -- 172.21.15.37:0/314618857 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff4c0a5c40 con 0x7eff440039c0 2024-04-26T00:38:28.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.843+0000 7eff6159c640 5 monclient: authenticate success, global_id 204350 2024-04-26T00:38:28.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.843+0000 7eff6159c640 10 monclient: _renew_subs 2024-04-26T00:38:28.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.843+0000 7eff6159c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:28.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.843+0000 7eff6159c640 1 -- 172.21.15.37:0/314618857 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff5c121fc0 con 0x7eff440039c0 2024-04-26T00:38:28.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.843+0000 7eff437fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:38:28.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.843+0000 7eff437fe640 1 -- 172.21.15.37:0/314618857 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff4c009b50 con 0x7eff440039c0 2024-04-26T00:38:28.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.843+0000 7eff437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:28.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.843+0000 7eff437fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:38:28.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.843+0000 7eff437fe640 10 monclient: dump: 2024-04-26T00:38:28.850 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:28.850 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:28.850 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:28.851 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:28.851 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:28.851 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:28.851 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:28.851 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:28.851 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:28.851 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:28.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.843+0000 7eff6159c640 10 monclient: _renew_subs 2024-04-26T00:38:28.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.843+0000 7eff597fa640 4 set_mon_vals no callback set 2024-04-26T00:38:28.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.843+0000 7eff6159c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:28.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.843+0000 7eff6159c640 1 -- 172.21.15.37:0/314618857 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7eff5c122c80 con 0x7eff440039c0 2024-04-26T00:38:28.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.843+0000 7eff6159c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:28.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.843+0000 7eff6159c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:28.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.843+0000 7eff6159c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:28.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.843+0000 7eff6159c640 1 -- 172.21.15.37:0/314618857 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff5c066f40 con 0x7eff440039c0 2024-04-26T00:38:28.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.843+0000 7eff437fe640 1 -- 172.21.15.37:0/314618857 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7eff4c0c1020 con 0x7eff440039c0 2024-04-26T00:38:28.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.843+0000 7eff437fe640 1 --2- 172.21.15.37:0/314618857 >> v2:172.21.15.45:6812/2379918623 conn(0x7eff480368a0 0x7eff48038d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:28.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.843+0000 7eff437fe640 1 -- 172.21.15.37:0/314618857 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(719..719 src has 370..719) ==== 3943+0+0 (secure 0 0 0) 0x7eff4c0e34a0 con 0x7eff440039c0 2024-04-26T00:38:28.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.843+0000 7eff5a7fc640 1 --2- 172.21.15.37:0/314618857 >> v2:172.21.15.45:6812/2379918623 conn(0x7eff480368a0 0x7eff48038d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:28.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.843+0000 7eff5a7fc640 10 monclient: get_auth_request con 0x7eff480368a0 auth_method 0 2024-04-26T00:38:28.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.847+0000 7eff5a7fc640 1 --2- 172.21.15.37:0/314618857 >> v2:172.21.15.45:6812/2379918623 conn(0x7eff480368a0 0x7eff48038d60 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7eff50006e10 tx=0x7eff50008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:28.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.851+0000 7eff437fe640 1 -- 172.21.15.37:0/314618857 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7eff4c017160 con 0x7eff440039c0 2024-04-26T00:38:28.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.851+0000 7eff437fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:28.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:28.851+0000 7eff437fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:38:28.888 INFO:tasks.rados.rados.0.smithi045.stdout:3220: done (3 left) 2024-04-26T00:38:28.888 INFO:tasks.rados.rados.0.smithi045.stdout:3221: done (2 left) 2024-04-26T00:38:28.889 INFO:tasks.rados.rados.0.smithi045.stdout:3222: done (1 left) 2024-04-26T00:38:28.889 INFO:tasks.rados.rados.0.smithi045.stdout:3223: done (0 left) 2024-04-26T00:38:28.889 INFO:tasks.rados.rados.0.smithi045.stdout:3224: copy_from oid 17 from oid 27 current snap is 351 2024-04-26T00:38:28.889 INFO:tasks.rados.rados.0.smithi045.stdout:3225: snap_create 2024-04-26T00:38:28.915 INFO:tasks.rados.rados.0.smithi045.stdout:3224: finishing copy_from racing read to smithi04526579-17 2024-04-26T00:38:28.920 INFO:tasks.rados.rados.0.smithi045.stdout:3224: finishing copy_from to smithi04526579-17 2024-04-26T00:38:28.920 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 2409 (ObjNum 1051 snap 317 seq_num 1051) dirty exists 2024-04-26T00:38:29.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.079+0000 7eff6159c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:38:29.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.079+0000 7eff6159c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:38:29.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.079+0000 7eff6159c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:29.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.079+0000 7eff6159c640 1 -- 172.21.15.37:0/314618857 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7eff5c122360 con 0x7eff440039c0 2024-04-26T00:38:29.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.083+0000 7eff437fe640 1 -- 172.21.15.37:0/314618857 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7eff4c0c2370 con 0x7eff440039c0 2024-04-26T00:38:29.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.083+0000 7eff437fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:38:29.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.083+0000 7eff437fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:38:29.081 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":140,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:38:29.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.083+0000 7eff6159c640 1 -- 172.21.15.37:0/314618857 >> v2:172.21.15.45:6812/2379918623 conn(0x7eff480368a0 msgr2=0x7eff48038d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:29.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.083+0000 7eff6159c640 1 --2- 172.21.15.37:0/314618857 >> v2:172.21.15.45:6812/2379918623 conn(0x7eff480368a0 0x7eff48038d60 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7eff50006e10 tx=0x7eff50008040 comp rx=0 tx=0).stop 2024-04-26T00:38:29.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.083+0000 7eff6159c640 10 monclient: shutdown 2024-04-26T00:38:29.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.083+0000 7eff6159c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:38:29.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.083+0000 7eff6159c640 1 -- 172.21.15.37:0/314618857 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7eff440039c0 msgr2=0x7eff5c1cebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:29.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.083+0000 7eff6159c640 1 --2- 172.21.15.37:0/314618857 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7eff440039c0 0x7eff5c1cebb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7eff4c000eb0 tx=0x7eff44005840 comp rx=0 tx=0).stop 2024-04-26T00:38:29.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.083+0000 7eff6159c640 1 -- 172.21.15.37:0/314618857 shutdown_connections 2024-04-26T00:38:29.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.083+0000 7eff6159c640 1 --2- 172.21.15.37:0/314618857 >> v2:172.21.15.45:6812/2379918623 conn(0x7eff480368a0 0x7eff48038d60 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:29.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.083+0000 7eff6159c640 1 --2- 172.21.15.37:0/314618857 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7eff44005220 0x7eff5c0709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:29.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.083+0000 7eff6159c640 1 --2- 172.21.15.37:0/314618857 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7eff44004810 0x7eff5c1cf0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:29.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.083+0000 7eff6159c640 1 --2- 172.21.15.37:0/314618857 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7eff440039c0 0x7eff5c1cebb0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:29.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.083+0000 7eff6159c640 1 -- 172.21.15.37:0/314618857 >> 172.21.15.37:0/314618857 conn(0x7eff5c0a1ad0 msgr2=0x7eff5c132140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:38:29.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.083+0000 7eff6159c640 1 -- 172.21.15.37:0/314618857 shutdown_connections 2024-04-26T00:38:29.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.083+0000 7eff6159c640 1 -- 172.21.15.37:0/314618857 wait complete. 2024-04-26T00:38:29.100 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-26T00:38:29.101 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-04-26T00:38:29.101 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:38:29.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.223+0000 7f219e003640 10 monclient: get_monmap_and_config 2024-04-26T00:38:29.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.223+0000 7f219e003640 10 monclient: build_initial_monmap 2024-04-26T00:38:29.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.223+0000 7f219e003640 10 monclient: monmap: 2024-04-26T00:38:29.221 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:38:29.221 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:29.221 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:38:29.224743+0000 2024-04-26T00:38:29.221 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:38:29.224743+0000 2024-04-26T00:38:29.221 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:38:29.221 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:29.221 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:38:29.221 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:38:29.221 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:38:29.222 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:38:29.222 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:38:29.222 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:29.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.223+0000 7f219e003640 1 Processor -- start 2024-04-26T00:38:29.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.223+0000 7f219e003640 1 -- start start 2024-04-26T00:38:29.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.223+0000 7f219e003640 10 monclient: init 2024-04-26T00:38:29.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.223+0000 7f219e003640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:29.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.223+0000 7f219e003640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-04-26T00:38:29.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.223+0000 7f219e003640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f2198136f20 0x7f219809bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:29.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.223+0000 7f219e003640 10 monclient(hunting): picked mon.noname-a con 0x7f2198136f20 addr v2:172.21.15.37:3300/0 2024-04-26T00:38:29.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.223+0000 7f219e003640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2198137300 addr v1:172.21.15.37:6789/0 2024-04-26T00:38:29.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.223+0000 7f219e003640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f219813c450 addr v1:172.21.15.45:6789/0 2024-04-26T00:38:29.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.223+0000 7f219e003640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21980a50f0 con 0x7f2198137300 2024-04-26T00:38:29.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.223+0000 7f219e003640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f219809b330 con 0x7f219813c450 2024-04-26T00:38:29.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.223+0000 7f219e003640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:29.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.223+0000 7f219e003640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f219809b610 con 0x7f2198136f20 2024-04-26T00:38:29.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.223+0000 7f219e003640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:29.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.223+0000 7f219e003640 10 monclient(hunting): authenticate will time out at 2090.530500s 2024-04-26T00:38:29.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.223+0000 7f2196ffd640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f2198137300 legacy=0x7f21981377b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:38:29.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.223+0000 7f21977fe640 1 -- >> v2:172.21.15.37:3300/0 conn(0x7f2198136f20 msgr2=0x7f219809bff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:38:29.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.223+0000 7f21977fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f2198136f20 0x7f219809bff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:38:29.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.223+0000 7f2197fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f219813c450 0x7f219813e840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.37:45642/0 (socket says 172.21.15.37:45642) 2024-04-26T00:38:29.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.223+0000 7f2197fff640 1 -- 172.21.15.37:0/2280172836 learned_addr learned my addr 172.21.15.37:0/2280172836 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:38:29.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f2197fff640 1 -- 172.21.15.37:0/2280172836 >> v1:172.21.15.45:6789/0 conn(0x7f219813c450 legacy=0x7f219813e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:29.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21967fc640 1 -- 172.21.15.37:0/2280172836 <== mon.1 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3824800858 0 0) 0x7f219809b330 con 0x7f219813c450 2024-04-26T00:38:29.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21967fc640 10 monclient(hunting): _negotiate 2024-04-26T00:38:29.225 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21967fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:29.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21967fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:29.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21967fc640 10 monclient(hunting): my global_id is 224140 2024-04-26T00:38:29.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21967fc640 1 -- 172.21.15.37:0/2280172836 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2180003530 con 0x7f219813c450 2024-04-26T00:38:29.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21967fc640 1 -- 172.21.15.37:0/2280172836 <== mon.1 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4203366052 0 0) 0x7f2180003530 con 0x7f219813c450 2024-04-26T00:38:29.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21967fc640 1 -- 172.21.15.37:0/2280172836 >> v2:172.21.15.37:3300/0 conn(0x7f2198136f20 msgr2=0x7f219809bff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:38:29.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21967fc640 1 --2- 172.21.15.37:0/2280172836 >> v2:172.21.15.37:3300/0 conn(0x7f2198136f20 0x7f219809bff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:29.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21967fc640 1 -- 172.21.15.37:0/2280172836 >> v1:172.21.15.37:6789/0 conn(0x7f2198137300 legacy=0x7f21981377b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:38:29.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21967fc640 10 monclient: _finish_hunting 0 2024-04-26T00:38:29.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21967fc640 1 monclient: found mon.noname-c-legacy 2024-04-26T00:38:29.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:29.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21967fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.45:6789/0 2024-04-26T00:38:29.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21967fc640 1 -- 172.21.15.37:0/2280172836 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f219809b8f0 con 0x7f219813c450 2024-04-26T00:38:29.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21967fc640 10 monclient: _finish_auth 0 2024-04-26T00:38:29.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21967fc640 10 monclient: _check_auth_tickets 2024-04-26T00:38:29.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:29.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21967fc640 1 -- 172.21.15.37:0/2280172836 <== mon.1 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 866326386 0 0) 0x7f2188001930 con 0x7f219813c450 2024-04-26T00:38:29.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:29.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21967fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-04-26T00:38:29.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21967fc640 10 monclient: dump: 2024-04-26T00:38:29.229 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:29.229 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:29.229 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:29.229 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:29.229 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:29.229 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:29.229 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:29.229 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:29.230 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:29.230 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:29.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21967fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] but i'm connected to v1:172.21.15.45:6789/0, reconnecting 2024-04-26T00:38:29.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21967fc640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:29.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21967fc640 1 -- 172.21.15.37:0/2280172836 >> v1:172.21.15.45:6789/0 conn(0x7f219813c450 legacy=0x7f219813e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:29.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21967fc640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:38:29.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21967fc640 1 --2- 172.21.15.37:0/2280172836 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f21800039c0 0x7f2180003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:29.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21967fc640 10 monclient(hunting): picked mon.c con 0x7f21800039c0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:38:29.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21967fc640 1 --2- 172.21.15.37:0/2280172836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2180004810 0x7f2180004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:29.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21967fc640 10 monclient(hunting): picked mon.b con 0x7f2180004810 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:38:29.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21967fc640 1 --2- 172.21.15.37:0/2280172836 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f21800052f0 0x7f218000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:29.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21967fc640 10 monclient(hunting): picked mon.a con 0x7f21800052f0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:38:29.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21967fc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:29.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21967fc640 1 -- 172.21.15.37:0/2280172836 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f219809b610 con 0x7f21800052f0 2024-04-26T00:38:29.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21967fc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:29.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21967fc640 1 -- 172.21.15.37:0/2280172836 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f218000c240 con 0x7f2180004810 2024-04-26T00:38:29.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21967fc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:29.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21967fc640 1 -- 172.21.15.37:0/2280172836 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f218000c440 con 0x7f21800039c0 2024-04-26T00:38:29.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21967fc640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:29.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21967fc640 10 monclient(hunting): _finish_auth 0 2024-04-26T00:38:29.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f2197fff640 1 -- 172.21.15.37:0/2280172836 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f21800052f0 msgr2=0x7f218000b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:38:29.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f2197fff640 1 --2- 172.21.15.37:0/2280172836 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f21800052f0 0x7f218000b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:38:29.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21977fe640 1 --2- 172.21.15.37:0/2280172836 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f21800039c0 0x7f2180003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:29.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21977fe640 10 monclient(hunting): get_auth_request con 0x7f21800039c0 auth_method 0 2024-04-26T00:38:29.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:29.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21977fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:29.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21977fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:38:29.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:29.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:29.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21977fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:38:29.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f2196ffd640 1 --2- 172.21.15.37:0/2280172836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2180004810 0x7f2180004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:29.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f2196ffd640 10 monclient(hunting): get_auth_request con 0x7f2180004810 auth_method 0 2024-04-26T00:38:29.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f2196ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:29.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f2196ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:29.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f2196ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:38:29.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21977fe640 10 monclient(hunting): handle_auth_done global_id 224140 payload 293 2024-04-26T00:38:29.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21977fe640 1 -- 172.21.15.37:0/2280172836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2180004810 msgr2=0x7f2180004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:38:29.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21977fe640 1 --2- 172.21.15.37:0/2280172836 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2180004810 0x7f2180004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:29.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21977fe640 1 -- 172.21.15.37:0/2280172836 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f21800052f0 msgr2=0x7f218000b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:38:29.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21977fe640 1 --2- 172.21.15.37:0/2280172836 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f21800052f0 0x7f218000b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:29.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21977fe640 10 monclient: _finish_hunting 0 2024-04-26T00:38:29.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21977fe640 1 monclient: found mon.c 2024-04-26T00:38:29.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:29.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f2197fff640 1 -- 172.21.15.37:0/2280172836 reap_dead start 2024-04-26T00:38:29.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21977fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:29.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21977fe640 1 -- 172.21.15.37:0/2280172836 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2180003530 con 0x7f21800039c0 2024-04-26T00:38:29.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21977fe640 10 monclient: _finish_auth 0 2024-04-26T00:38:29.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21977fe640 10 monclient: _check_auth_tickets 2024-04-26T00:38:29.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f21977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:29.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f219e003640 5 monclient: authenticate success, global_id 224140 2024-04-26T00:38:29.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.227+0000 7f219e003640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:38:29.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.231+0000 7f21977fe640 1 -- 172.21.15.37:0/2280172836 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f21800039c0 msgr2=0x7f2180003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:29.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.231+0000 7f21977fe640 1 --2- 172.21.15.37:0/2280172836 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f21800039c0 0x7f2180003dc0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2190003d10 tx=0x7f219001cab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8834665a2fba801 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:29.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.231+0000 7f21967fc640 1 -- 172.21.15.37:0/2280172836 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2190009a30 con 0x7f21800039c0 2024-04-26T00:38:29.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.231+0000 7f21967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:29.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.231+0000 7f21967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:38:29.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.231+0000 7f21967fc640 10 monclient: dump: 2024-04-26T00:38:29.239 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:29.239 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:29.239 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:29.239 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:29.239 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:29.239 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:29.239 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:29.240 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:29.240 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:29.240 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:29.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.231+0000 7f219e003640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:38:29.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.231+0000 7f21967fc640 1 -- 172.21.15.37:0/2280172836 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21900a9070 con 0x7f21800039c0 2024-04-26T00:38:29.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.231+0000 7f21967fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:38:29.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.231+0000 7f219e003640 10 monclient: get_monmap_and_config success 2024-04-26T00:38:29.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.231+0000 7f219e003640 4 set_mon_vals no callback set 2024-04-26T00:38:29.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.231+0000 7f21967fc640 1 -- 172.21.15.37:0/2280172836 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21900a59f0 con 0x7f21800039c0 2024-04-26T00:38:29.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.231+0000 7f21967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:29.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.231+0000 7f21967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:38:29.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.231+0000 7f21967fc640 10 monclient: dump: 2024-04-26T00:38:29.240 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:29.240 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:29.240 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:29.241 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:29.241 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:29.241 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:29.241 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:29.241 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:29.241 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:29.241 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:29.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f219e003640 10 monclient: shutdown 2024-04-26T00:38:29.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f219e003640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:38:29.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f219e003640 1 -- 172.21.15.37:0/2280172836 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f21800039c0 msgr2=0x7f2180003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:29.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f219e003640 1 --2- 172.21.15.37:0/2280172836 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f21800039c0 0x7f2180003dc0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2190003d10 tx=0x7f219001cab0 comp rx=0 tx=0).stop 2024-04-26T00:38:29.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f219e003640 1 -- 172.21.15.37:0/2280172836 shutdown_connections 2024-04-26T00:38:29.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f219e003640 1 --2- 172.21.15.37:0/2280172836 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f21800039c0 0x7f2180003dc0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:29.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f219e003640 1 -- 172.21.15.37:0/2280172836 >> 172.21.15.37:0/2280172836 conn(0x7f21980a61b0 msgr2=0x7f21980a65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:38:29.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f219e003640 1 -- 172.21.15.37:0/2280172836 shutdown_connections 2024-04-26T00:38:29.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f219e003640 1 -- 172.21.15.37:0/2280172836 wait complete. 2024-04-26T00:38:29.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f219e003640 10 monclient: build_initial_monmap 2024-04-26T00:38:29.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f219e003640 10 monclient: monmap: 2024-04-26T00:38:29.242 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:38:29.242 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:38:29.242 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:38:29.242 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:38:29.242 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:38:29.242 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:38:29.242 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:38:29.242 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:38:29.242 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:38:29.242 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:29.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f219e003640 1 Processor -- start 2024-04-26T00:38:29.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f219e003640 1 -- start start 2024-04-26T00:38:29.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f219e003640 10 monclient: init 2024-04-26T00:38:29.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f219e003640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:29.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f219e003640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:38:29.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f219e003640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21800039c0 0x7f2198131fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:29.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f219e003640 10 monclient(hunting): picked mon.noname-b con 0x7f21800039c0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:38:29.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f219e003640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f219813b850 0x7f21981324e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:29.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f219e003640 10 monclient(hunting): picked mon.noname-a con 0x7f219813b850 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:38:29.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f219e003640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2198137ef0 0x7f2198138350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:29.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f219e003640 10 monclient(hunting): picked mon.noname-c con 0x7f2198137ef0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:38:29.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f219e003640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:29.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f219e003640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f219809ac60 con 0x7f219813b850 2024-04-26T00:38:29.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f219e003640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:29.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f219e003640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f219809aae0 con 0x7f21800039c0 2024-04-26T00:38:29.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f219e003640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:29.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f219e003640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f219809bff0 con 0x7f2198137ef0 2024-04-26T00:38:29.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f219e003640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:29.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f219e003640 10 monclient(hunting): authenticate will time out at 2090.542069s 2024-04-26T00:38:29.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f2196ffd640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f219813b850 msgr2=0x7f21981324e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:38:29.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f2196ffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f219813b850 0x7f21981324e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:38:29.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f2197fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2198137ef0 0x7f2198138350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:29.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f2197fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2198137ef0 0x7f2198138350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:56382/0 (socket says 172.21.15.37:56382) 2024-04-26T00:38:29.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f2197fff640 1 -- 172.21.15.37:0/154753742 learned_addr learned my addr 172.21.15.37:0/154753742 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:38:29.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f2197fff640 10 monclient(hunting): get_auth_request con 0x7f2198137ef0 auth_method 0 2024-04-26T00:38:29.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f2197fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:29.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f2197fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:29.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f2197fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:29.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f2197fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:29.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f2197fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:29.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f2197fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:29.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f21977fe640 1 --2- 172.21.15.37:0/154753742 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21800039c0 0x7f2198131fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:29.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f21977fe640 10 monclient(hunting): get_auth_request con 0x7f21800039c0 auth_method 0 2024-04-26T00:38:29.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f21977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:29.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f21977fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:29.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f21977fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:29.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f2197fff640 10 monclient(hunting): handle_auth_done global_id 204353 payload 824 2024-04-26T00:38:29.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f2197fff640 1 -- 172.21.15.37:0/154753742 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21800039c0 msgr2=0x7f2198131fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:29.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f2197fff640 1 --2- 172.21.15.37:0/154753742 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21800039c0 0x7f2198131fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:29.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f2197fff640 1 -- 172.21.15.37:0/154753742 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f219813b850 msgr2=0x7f21981324e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:38:29.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f2197fff640 1 --2- 172.21.15.37:0/154753742 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f219813b850 0x7f21981324e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:29.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f2197fff640 10 monclient: _finish_hunting 0 2024-04-26T00:38:29.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f2197fff640 1 monclient: found mon.noname-c 2024-04-26T00:38:29.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f2197fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:29.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f2197fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:38:29.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.235+0000 7f2197fff640 1 -- 172.21.15.37:0/154753742 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21980a02d0 con 0x7f2198137ef0 2024-04-26T00:38:29.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f2197fff640 1 -- 172.21.15.37:0/154753742 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2198137ef0 msgr2=0x7f2198138350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:29.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f2197fff640 1 --2- 172.21.15.37:0/154753742 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2198137ef0 0x7f2198138350 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2198073860 tx=0x7f218800f4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:29.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f217ffff640 1 -- 172.21.15.37:0/154753742 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2188007af0 con 0x7f2198137ef0 2024-04-26T00:38:29.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f217ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:29.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f217ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:38:29.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f217ffff640 10 monclient: dump: 2024-04-26T00:38:29.246 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:29.246 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:29.246 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:29.246 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:29.246 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:29.247 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:29.247 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:29.247 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:29.247 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:29.247 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:29.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f217ffff640 10 monclient: _finish_auth 0 2024-04-26T00:38:29.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f217ffff640 10 monclient: _check_auth_tickets 2024-04-26T00:38:29.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f217ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:29.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f217ffff640 1 -- 172.21.15.37:0/154753742 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2188007c90 con 0x7f2198137ef0 2024-04-26T00:38:29.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f217ffff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:38:29.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f217ffff640 1 -- 172.21.15.37:0/154753742 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f218800a350 con 0x7f2198137ef0 2024-04-26T00:38:29.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f219e003640 5 monclient: authenticate success, global_id 204353 2024-04-26T00:38:29.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f219e003640 10 monclient: _renew_subs 2024-04-26T00:38:29.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f219e003640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:29.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f219e003640 1 -- 172.21.15.37:0/154753742 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f219813bd00 con 0x7f2198137ef0 2024-04-26T00:38:29.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f217ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:29.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f217ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:38:29.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f2195ffb640 4 set_mon_vals no callback set 2024-04-26T00:38:29.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f217ffff640 10 monclient: dump: 2024-04-26T00:38:29.248 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:29.248 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:29.248 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:29.248 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:29.248 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:29.248 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:29.248 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:29.248 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:29.248 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:29.249 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:29.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f219e003640 10 monclient: _renew_subs 2024-04-26T00:38:29.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f219e003640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:29.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f219e003640 1 -- 172.21.15.37:0/154753742 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2198132a20 con 0x7f2198137ef0 2024-04-26T00:38:29.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f219e003640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:29.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f219e003640 1 --2- 172.21.15.37:0/154753742 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2198138890 0x7f21980a28f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:29.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f219e003640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:29.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f219e003640 1 -- 172.21.15.37:0/154753742 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f21981363f0 con 0x7f2198138890 2024-04-26T00:38:29.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f21977fe640 1 --2- 172.21.15.37:0/154753742 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2198138890 0x7f21980a28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:29.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f21977fe640 10 monclient: get_auth_request con 0x7f2198138890 auth_method 0 2024-04-26T00:38:29.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f21977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:29.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f21977fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:29.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f21977fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:29.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f217ffff640 1 -- 172.21.15.37:0/154753742 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2188015030 con 0x7f2198137ef0 2024-04-26T00:38:29.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f21977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:29.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f21977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:29.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f21977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:29.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f217ffff640 1 --2- 172.21.15.37:0/154753742 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2184035eb0 0x7f2184038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:29.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f21977fe640 10 monclient(hunting): handle_auth_done global_id 204356 payload 824 2024-04-26T00:38:29.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f217ffff640 1 -- 172.21.15.37:0/154753742 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(720..720 src has 370..720) ==== 3915+0+0 (secure 0 0 0) 0x7f218804a800 con 0x7f2198137ef0 2024-04-26T00:38:29.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f21977fe640 1 -- 172.21.15.37:0/154753742 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f219804f510 con 0x7f2198138890 2024-04-26T00:38:29.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f2196ffd640 1 --2- 172.21.15.37:0/154753742 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2184035eb0 0x7f2184038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:29.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.239+0000 7f2196ffd640 10 monclient: get_auth_request con 0x7f2184035eb0 auth_method 0 2024-04-26T00:38:29.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.243+0000 7f21977fe640 1 -- 172.21.15.37:0/154753742 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2198138890 msgr2=0x7f21980a28f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:29.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.243+0000 7f21977fe640 1 --2- 172.21.15.37:0/154753742 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2198138890 0x7f21980a28f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f21900ad000 tx=0x7f2180005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:29.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.243+0000 7f217ffff640 1 -- 172.21.15.37:0/154753742 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21900ae880 con 0x7f2198138890 2024-04-26T00:38:29.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.243+0000 7f217ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:29.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.243+0000 7f217ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:38:29.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.243+0000 7f217ffff640 10 monclient: dump: 2024-04-26T00:38:29.253 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:29.253 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:29.253 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:29.253 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:29.254 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:29.254 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:29.254 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:29.254 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:29.254 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:29.254 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:29.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.243+0000 7f217ffff640 1 -- 172.21.15.37:0/154753742 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f21900a9050 con 0x7f2198138890 2024-04-26T00:38:29.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.243+0000 7f217ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:29.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.243+0000 7f217ffff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:38:29.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.243+0000 7f217ffff640 1 -- 172.21.15.37:0/154753742 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2198138890 msgr2=0x7f21980a28f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:29.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.243+0000 7f217ffff640 1 --2- 172.21.15.37:0/154753742 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2198138890 0x7f21980a28f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f21900ad000 tx=0x7f2180005840 comp rx=0 tx=0).stop 2024-04-26T00:38:29.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.243+0000 7f217ffff640 1 -- 172.21.15.37:0/154753742 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2198138890 msgr2=0x7f21980a28f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:38:29.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.243+0000 7f217ffff640 1 --2- 172.21.15.37:0/154753742 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2198138890 0x7f21980a28f0 secure :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f21900ad000 tx=0x7f2180005840 comp rx=0 tx=0).stop 2024-04-26T00:38:29.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.243+0000 7f2196ffd640 1 --2- 172.21.15.37:0/154753742 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2184035eb0 0x7f2184038370 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f219809fed0 tx=0x7f218c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:29.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.247+0000 7f219e003640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:38:29.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.247+0000 7f219e003640 1 --2- 172.21.15.37:0/154753742 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f21980a2e30 0x7f21980a3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:29.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.247+0000 7f219e003640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:29.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.247+0000 7f219e003640 1 -- 172.21.15.37:0/154753742 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2198138c70 con 0x7f21980a2e30 2024-04-26T00:38:29.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.247+0000 7f21977fe640 1 --2- 172.21.15.37:0/154753742 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f21980a2e30 0x7f21980a3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:29.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.247+0000 7f21977fe640 10 monclient: get_auth_request con 0x7f21980a2e30 auth_method 0 2024-04-26T00:38:29.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.247+0000 7f21977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:29.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.247+0000 7f21977fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:29.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.247+0000 7f21977fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:29.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.247+0000 7f21977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:29.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.247+0000 7f21977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:29.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.247+0000 7f21977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:29.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.247+0000 7f21977fe640 10 monclient(hunting): handle_auth_done global_id 204359 payload 824 2024-04-26T00:38:29.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.247+0000 7f21977fe640 1 -- 172.21.15.37:0/154753742 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2198075f10 con 0x7f21980a2e30 2024-04-26T00:38:29.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.251+0000 7f21977fe640 1 -- 172.21.15.37:0/154753742 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f21980a2e30 msgr2=0x7f21980a3210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:29.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.251+0000 7f21977fe640 1 --2- 172.21.15.37:0/154753742 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f21980a2e30 0x7f21980a3210 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f219001caf0 tx=0x7f2190009520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:29.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.251+0000 7f217ffff640 1 -- 172.21.15.37:0/154753742 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21900bdc70 con 0x7f21980a2e30 2024-04-26T00:38:29.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.251+0000 7f217ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:29.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.251+0000 7f217ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:38:29.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.251+0000 7f217ffff640 10 monclient: dump: 2024-04-26T00:38:29.256 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:29.256 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:29.256 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:29.256 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:29.256 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:29.256 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:29.256 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:29.256 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:29.256 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:29.257 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:29.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.251+0000 7f217ffff640 1 -- 172.21.15.37:0/154753742 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4709 (secure 0 0 0) 0x7f21900b4070 con 0x7f21980a2e30 2024-04-26T00:38:29.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.251+0000 7f217ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:38:29.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.251+0000 7f217ffff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:38:29.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.251+0000 7f217ffff640 1 -- 172.21.15.37:0/154753742 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f21980a2e30 msgr2=0x7f21980a3210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:29.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.251+0000 7f217ffff640 1 --2- 172.21.15.37:0/154753742 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f21980a2e30 0x7f21980a3210 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f219001caf0 tx=0x7f2190009520 comp rx=0 tx=0).stop 2024-04-26T00:38:29.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.251+0000 7f217ffff640 1 -- 172.21.15.37:0/154753742 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f21980a2e30 msgr2=0x7f21980a3210 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:38:29.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.251+0000 7f217ffff640 1 --2- 172.21.15.37:0/154753742 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f21980a2e30 0x7f21980a3210 secure :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f219001caf0 tx=0x7f2190009520 comp rx=0 tx=0).stop 2024-04-26T00:38:29.257 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:38:29.257 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:38:29.257 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:38:29.258 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:38:29.258 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 140, 2024-04-26T00:38:29.258 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 114858, 2024-04-26T00:38:29.258 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:38:29.258 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:38:29.258 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:38:29.258 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:38:29.258 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:38:29.258 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:38:29.258 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:38:29.258 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:38:29.258 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:38:29.258 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:38:29.259 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:38:29.259 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:38:29.259 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:38:29.259 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:38:29.259 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:38:29.259 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:38:29.259 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:38:29.259 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:38:29.259 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:38:29.259 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:38:29.259 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:38:29.259 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:38:29.259 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:38:29.259 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:38:29.260 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:38:29.260 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:38:29.260 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:38:29.260 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:38:29.260 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:38:29.260 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:38:29.260 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:38:29.260 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:38:29.260 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:38:29.260 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:29.260 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:29.260 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:38:29.260 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:38:29.261 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:38:29.261 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:38:29.261 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:38:29.261 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:38:29.261 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:38:29.261 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:38:29.261 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:38:29.261 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:38:29.261 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:38:29.261 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:38:29.262 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:38:29.262 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:38:29.262 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:38:29.262 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:38:29.262 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:38:29.262 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:38:29.262 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:38:29.262 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:38:29.262 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:38:29.262 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:38:29.262 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:38:29.262 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:38:29.262 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:38:29.263 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:38:29.263 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:38:29.263 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:38:29.263 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:38:29.263 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:38:29.263 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:29.264 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:38:29.264 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:29.264 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:38:29.264 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:38:29.264 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:38:29.264 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:38:29.264 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:29.264 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:38:29.264 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:38:29.265 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:29.265 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:29.265 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:29.265 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:38:29.265 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:38:29.265 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:29.265 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:29.265 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:29.265 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:29.265 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:38:29.265 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:38:29.265 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:38:29.265 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:38:29.265 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:38:29.265 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:29.266 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:29.266 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:38:29.266 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:38:29.266 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:38:29.266 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:38:29.266 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:29.266 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:38:29.266 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:38:29.266 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:29.266 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:29.266 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:29.266 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:38:29.266 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:38:29.267 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:29.267 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:29.267 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:29.267 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:29.267 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:38:29.267 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:38:29.267 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:38:29.267 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:38:29.267 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:38:29.267 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:29.267 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:29.267 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:38:29.267 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:38:29.267 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:38:29.268 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:38:29.268 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:29.268 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:38:29.268 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:38:29.268 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:29.268 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:29.268 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:29.268 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:38:29.268 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:38:29.268 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:29.268 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:29.268 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:29.268 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:29.268 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:38:29.268 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:38:29.269 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:38:29.269 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:38:29.269 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:38:29.269 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:29.269 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:29.269 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:29.269 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:38:29.269 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:38:29.269 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:29.269 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:38:29.269 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:38:29.269 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:38:29.269 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:29.269 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:38:29.269 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:38:29.270 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:29.270 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:38:29.270 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:38:29.270 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:38:29.270 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:29.270 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:38:29.270 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:38:29.270 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:29.270 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:38:29.270 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:38:29.270 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:38:29.270 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:29.270 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:29.271 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:29.271 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:38:29.271 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:38:29.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.251+0000 7f219e003640 1 -- 172.21.15.37:0/154753742 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2184035eb0 msgr2=0x7f2184038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:29.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.251+0000 7f219e003640 1 --2- 172.21.15.37:0/154753742 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2184035eb0 0x7f2184038370 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f219809fed0 tx=0x7f218c008040 comp rx=0 tx=0).stop 2024-04-26T00:38:29.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.251+0000 7f219e003640 10 monclient: shutdown 2024-04-26T00:38:29.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.251+0000 7f219e003640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:38:29.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.251+0000 7f219e003640 1 -- 172.21.15.37:0/154753742 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2198137ef0 msgr2=0x7f2198138350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:29.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.251+0000 7f219e003640 1 --2- 172.21.15.37:0/154753742 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2198137ef0 0x7f2198138350 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2198073860 tx=0x7f218800f4b0 comp rx=0 tx=0).stop 2024-04-26T00:38:29.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.251+0000 7f2197fff640 1 -- 172.21.15.37:0/154753742 reap_dead start 2024-04-26T00:38:29.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.251+0000 7f219e003640 1 -- 172.21.15.37:0/154753742 shutdown_connections 2024-04-26T00:38:29.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.251+0000 7f219e003640 1 --2- 172.21.15.37:0/154753742 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2198137ef0 0x7f2198138350 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:29.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.251+0000 7f219e003640 1 -- 172.21.15.37:0/154753742 >> 172.21.15.37:0/154753742 conn(0x7f21980a61b0 msgr2=0x7f21980a7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:38:29.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.251+0000 7f219e003640 1 -- 172.21.15.37:0/154753742 shutdown_connections 2024-04-26T00:38:29.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.251+0000 7f219e003640 1 -- 172.21.15.37:0/154753742 wait complete. 2024-04-26T00:38:29.274 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:38:29.274 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-04-26T00:38:29.274 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:38:29.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb95e1a640 10 monclient: get_monmap_and_config 2024-04-26T00:38:29.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb95e1a640 10 monclient: build_initial_monmap 2024-04-26T00:38:29.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb95e1a640 10 monclient: monmap: 2024-04-26T00:38:29.389 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:38:29.389 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:29.389 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:38:29.392678+0000 2024-04-26T00:38:29.389 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:38:29.392678+0000 2024-04-26T00:38:29.389 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:38:29.389 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:29.389 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:38:29.389 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:38:29.390 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:38:29.390 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:38:29.390 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:38:29.390 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:29.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb95e1a640 1 Processor -- start 2024-04-26T00:38:29.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb95e1a640 1 -- start start 2024-04-26T00:38:29.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb95e1a640 10 monclient: init 2024-04-26T00:38:29.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb95e1a640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:29.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb95e1a640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-04-26T00:38:29.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb95e1a640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fcb900726b0 0x7fcb90072a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:29.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb95e1a640 10 monclient(hunting): picked mon.noname-a con 0x7fcb900726b0 addr v2:172.21.15.37:3300/0 2024-04-26T00:38:29.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb95e1a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb900a0110 0x7fcb90136dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:29.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb95e1a640 10 monclient(hunting): picked mon.noname-b con 0x7fcb900a0110 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:38:29.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb95e1a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcb90137310 addr v1:172.21.15.45:6789/0 2024-04-26T00:38:29.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb95e1a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb900a1470 con 0x7fcb90137310 2024-04-26T00:38:29.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb95e1a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:29.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb95e1a640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fcb900a1bd0 con 0x7fcb900726b0 2024-04-26T00:38:29.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb8f7fe640 1 -- >> v2:172.21.15.37:3300/0 conn(0x7fcb900726b0 msgr2=0x7fcb90072a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:38:29.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb95e1a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:29.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb95e1a640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fcb900a0eb0 con 0x7fcb900a0110 2024-04-26T00:38:29.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb95e1a640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:29.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb95e1a640 10 monclient(hunting): authenticate will time out at 2090.698584s 2024-04-26T00:38:29.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb8f7fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fcb900726b0 0x7fcb90072a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:38:29.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb8effd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb900a0110 0x7fcb90136dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:29.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb8effd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb900a0110 0x7fcb90136dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:56402/0 (socket says 172.21.15.37:56402) 2024-04-26T00:38:29.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb8effd640 1 -- 172.21.15.37:0/3371015208 learned_addr learned my addr 172.21.15.37:0/3371015208 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:38:29.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb8effd640 10 monclient(hunting): get_auth_request con 0x7fcb900a0110 auth_method 0 2024-04-26T00:38:29.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb8effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:29.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb8effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:29.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb8effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:29.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb8effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:29.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb8effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:29.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb8effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:29.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb8effd640 10 monclient(hunting): handle_auth_done global_id 204362 payload 274 2024-04-26T00:38:29.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb8effd640 1 -- 172.21.15.37:0/3371015208 >> v2:172.21.15.37:3300/0 conn(0x7fcb900726b0 msgr2=0x7fcb90072a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:38:29.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb8effd640 1 --2- 172.21.15.37:0/3371015208 >> v2:172.21.15.37:3300/0 conn(0x7fcb900726b0 0x7fcb90072a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:29.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb8effd640 1 -- 172.21.15.37:0/3371015208 >> v1:172.21.15.45:6789/0 conn(0x7fcb90137310 legacy=0x7fcb9013e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:38:29.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb8effd640 10 monclient: _finish_hunting 0 2024-04-26T00:38:29.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb8effd640 1 monclient: found mon.noname-b 2024-04-26T00:38:29.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb8effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:29.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb8effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:38:29.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.391+0000 7fcb8effd640 1 -- 172.21.15.37:0/3371015208 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb900a0bd0 con 0x7fcb900a0110 2024-04-26T00:38:29.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.395+0000 7fcb8effd640 1 -- 172.21.15.37:0/3371015208 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb900a0110 msgr2=0x7fcb90136dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:29.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.395+0000 7fcb8effd640 1 --2- 172.21.15.37:0/3371015208 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb900a0110 0x7fcb90136dd0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fcb78009e50 tx=0x7fcb7801c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52b67f4dcbe711fd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:29.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.395+0000 7fcb8e7fc640 1 -- 172.21.15.37:0/3371015208 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb780a5520 con 0x7fcb900a0110 2024-04-26T00:38:29.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.395+0000 7fcb8e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:29.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.395+0000 7fcb8e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:38:29.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.395+0000 7fcb8e7fc640 10 monclient: dump: 2024-04-26T00:38:29.397 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:29.397 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:29.397 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:29.397 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:29.397 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:29.397 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:29.397 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:29.398 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:29.398 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:29.398 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:29.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.395+0000 7fcb8e7fc640 10 monclient: _finish_auth 0 2024-04-26T00:38:29.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.395+0000 7fcb8e7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:38:29.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.395+0000 7fcb8e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:29.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.395+0000 7fcb95e1a640 5 monclient: authenticate success, global_id 204362 2024-04-26T00:38:29.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.395+0000 7fcb8e7fc640 1 -- 172.21.15.37:0/3371015208 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb780a56c0 con 0x7fcb900a0110 2024-04-26T00:38:29.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.395+0000 7fcb95e1a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:38:29.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.395+0000 7fcb8e7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:38:29.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.395+0000 7fcb95e1a640 10 monclient: get_monmap_and_config success 2024-04-26T00:38:29.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.395+0000 7fcb95e1a640 4 set_mon_vals no callback set 2024-04-26T00:38:29.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.395+0000 7fcb8e7fc640 1 -- 172.21.15.37:0/3371015208 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb780a5a90 con 0x7fcb900a0110 2024-04-26T00:38:29.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.395+0000 7fcb8e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:29.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.395+0000 7fcb8e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:38:29.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.395+0000 7fcb8e7fc640 10 monclient: dump: 2024-04-26T00:38:29.399 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:29.399 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:29.399 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:29.399 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:29.399 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:29.399 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:29.399 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:29.399 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:29.399 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:29.399 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:29.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb95e1a640 10 monclient: shutdown 2024-04-26T00:38:29.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb95e1a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:38:29.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb95e1a640 1 -- 172.21.15.37:0/3371015208 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb900a0110 msgr2=0x7fcb90136dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:29.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb95e1a640 1 --2- 172.21.15.37:0/3371015208 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb900a0110 0x7fcb90136dd0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fcb78009e50 tx=0x7fcb7801c6c0 comp rx=0 tx=0).stop 2024-04-26T00:38:29.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb95e1a640 1 -- 172.21.15.37:0/3371015208 shutdown_connections 2024-04-26T00:38:29.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb95e1a640 1 --2- 172.21.15.37:0/3371015208 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb900a0110 0x7fcb90136dd0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:29.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb95e1a640 1 --2- 172.21.15.37:0/3371015208 >> v2:172.21.15.37:3300/0 conn(0x7fcb900726b0 0x7fcb90072a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:29.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb95e1a640 1 -- 172.21.15.37:0/3371015208 >> 172.21.15.37:0/3371015208 conn(0x7fcb900a1d50 msgr2=0x7fcb900a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:38:29.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb95e1a640 1 -- 172.21.15.37:0/3371015208 shutdown_connections 2024-04-26T00:38:29.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb95e1a640 1 -- 172.21.15.37:0/3371015208 wait complete. 2024-04-26T00:38:29.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb95e1a640 10 monclient: build_initial_monmap 2024-04-26T00:38:29.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb95e1a640 10 monclient: monmap: 2024-04-26T00:38:29.400 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:38:29.400 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:38:29.400 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:38:29.401 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:38:29.401 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:38:29.401 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:38:29.401 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:38:29.401 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:38:29.401 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:38:29.401 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:29.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb95e1a640 1 Processor -- start 2024-04-26T00:38:29.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb95e1a640 1 -- start start 2024-04-26T00:38:29.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb95e1a640 10 monclient: init 2024-04-26T00:38:29.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb95e1a640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:29.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb95e1a640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:38:29.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb95e1a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcb900726b0 0x7fcb901cf090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:29.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb95e1a640 10 monclient(hunting): picked mon.noname-a con 0x7fcb900726b0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:38:29.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb95e1a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb900a0110 0x7fcb90131900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:29.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb95e1a640 10 monclient(hunting): picked mon.noname-c con 0x7fcb900a0110 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:38:29.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb95e1a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb90137310 0x7fcb9013ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:29.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb95e1a640 10 monclient(hunting): picked mon.noname-b con 0x7fcb90137310 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:38:29.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb95e1a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:29.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb95e1a640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fcb9009bc50 con 0x7fcb900726b0 2024-04-26T00:38:29.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb95e1a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:29.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb95e1a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcb900730e0 con 0x7fcb90137310 2024-04-26T00:38:29.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb95e1a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:29.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb95e1a640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fcb9009bdd0 con 0x7fcb900a0110 2024-04-26T00:38:29.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb95e1a640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:29.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb95e1a640 10 monclient(hunting): authenticate will time out at 2090.705681s 2024-04-26T00:38:29.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb8f7fe640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcb900726b0 msgr2=0x7fcb901cf090 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:38:29.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb8f7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcb900726b0 0x7fcb901cf090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:38:29.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb8effd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb900a0110 0x7fcb90131900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:29.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb8effd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb900a0110 0x7fcb90131900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:56404/0 (socket says 172.21.15.37:56404) 2024-04-26T00:38:29.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb8effd640 1 -- 172.21.15.37:0/736105139 learned_addr learned my addr 172.21.15.37:0/736105139 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:38:29.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb8effd640 10 monclient(hunting): get_auth_request con 0x7fcb900a0110 auth_method 0 2024-04-26T00:38:29.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb8effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:29.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb8effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:29.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb8effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:29.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb8effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:29.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb8effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:29.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb8effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:29.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb8effd640 10 monclient(hunting): handle_auth_done global_id 204365 payload 824 2024-04-26T00:38:29.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb8effd640 1 -- 172.21.15.37:0/736105139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb90137310 msgr2=0x7fcb9013ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:29.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb8ffff640 1 --2- 172.21.15.37:0/736105139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb90137310 0x7fcb9013ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:29.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb8effd640 1 --2- 172.21.15.37:0/736105139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb90137310 0x7fcb9013ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:29.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb8effd640 1 -- 172.21.15.37:0/736105139 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcb900726b0 msgr2=0x7fcb901cf090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:38:29.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb8effd640 1 --2- 172.21.15.37:0/736105139 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcb900726b0 0x7fcb901cf090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:29.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb8effd640 10 monclient: _finish_hunting 0 2024-04-26T00:38:29.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb8effd640 1 monclient: found mon.noname-c 2024-04-26T00:38:29.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb8effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:29.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb8effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:38:29.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.399+0000 7fcb8effd640 1 -- 172.21.15.37:0/736105139 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb9013e550 con 0x7fcb900a0110 2024-04-26T00:38:29.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb8effd640 1 -- 172.21.15.37:0/736105139 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb900a0110 msgr2=0x7fcb90131900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:29.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb8effd640 1 --2- 172.21.15.37:0/736105139 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb900a0110 0x7fcb90131900 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fcb780af040 tx=0x7fcb78019460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:29.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb87fff640 1 -- 172.21.15.37:0/736105139 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb780bc020 con 0x7fcb900a0110 2024-04-26T00:38:29.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb87fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:29.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb87fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:38:29.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb87fff640 10 monclient: dump: 2024-04-26T00:38:29.409 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:29.409 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:29.409 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:29.409 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:29.409 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:29.409 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:29.409 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:29.409 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:29.409 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:29.409 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:29.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb87fff640 10 monclient: _finish_auth 0 2024-04-26T00:38:29.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb87fff640 10 monclient: _check_auth_tickets 2024-04-26T00:38:29.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:29.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb95e1a640 5 monclient: authenticate success, global_id 204365 2024-04-26T00:38:29.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb95e1a640 10 monclient: _renew_subs 2024-04-26T00:38:29.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb95e1a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:29.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb87fff640 1 -- 172.21.15.37:0/736105139 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb780b8880 con 0x7fcb900a0110 2024-04-26T00:38:29.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb95e1a640 1 -- 172.21.15.37:0/736105139 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb9013d6b0 con 0x7fcb900a0110 2024-04-26T00:38:29.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb87fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:38:29.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb87fff640 1 -- 172.21.15.37:0/736105139 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb780b8c50 con 0x7fcb900a0110 2024-04-26T00:38:29.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb87fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:29.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb87fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:38:29.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb87fff640 10 monclient: dump: 2024-04-26T00:38:29.410 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:29.411 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:29.411 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:29.411 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:29.411 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:29.411 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:29.411 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:29.411 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:29.411 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:29.411 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:29.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb95e1a640 10 monclient: _renew_subs 2024-04-26T00:38:29.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb95e1a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:29.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb95e1a640 1 -- 172.21.15.37:0/736105139 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb9013e210 con 0x7fcb900a0110 2024-04-26T00:38:29.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb8dffb640 4 set_mon_vals no callback set 2024-04-26T00:38:29.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb87fff640 1 -- 172.21.15.37:0/736105139 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fcb780bf020 con 0x7fcb900a0110 2024-04-26T00:38:29.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb87fff640 1 --2- 172.21.15.37:0/736105139 >> v2:172.21.15.45:6812/2379918623 conn(0x7fcb70035e80 0x7fcb70038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:29.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb87fff640 1 -- 172.21.15.37:0/736105139 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(720..720 src has 370..720) ==== 3915+0+0 (secure 0 0 0) 0x7fcb780e3ae0 con 0x7fcb900a0110 2024-04-26T00:38:29.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb95e1a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:29.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb95e1a640 1 --2- 172.21.15.37:0/736105139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9013bf50 0x7fcb9013c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:29.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb95e1a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:29.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb95e1a640 1 -- 172.21.15.37:0/736105139 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcb900a05d0 con 0x7fcb9013bf50 2024-04-26T00:38:29.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb8f7fe640 1 --2- 172.21.15.37:0/736105139 >> v2:172.21.15.45:6812/2379918623 conn(0x7fcb70035e80 0x7fcb70038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:29.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb8f7fe640 10 monclient: get_auth_request con 0x7fcb70035e80 auth_method 0 2024-04-26T00:38:29.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb8ffff640 1 --2- 172.21.15.37:0/736105139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9013bf50 0x7fcb9013c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:29.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb8ffff640 10 monclient: get_auth_request con 0x7fcb9013bf50 auth_method 0 2024-04-26T00:38:29.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:29.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb8ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:29.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:29.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb8ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:29.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb8ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:29.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb8ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:29.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb8ffff640 10 monclient(hunting): handle_auth_done global_id 224146 payload 824 2024-04-26T00:38:29.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.403+0000 7fcb8ffff640 1 -- 172.21.15.37:0/736105139 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcb9013c870 con 0x7fcb9013bf50 2024-04-26T00:38:29.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.407+0000 7fcb8f7fe640 1 --2- 172.21.15.37:0/736105139 >> v2:172.21.15.45:6812/2379918623 conn(0x7fcb70035e80 0x7fcb70038340 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fcb80006fd0 tx=0x7fcb80006d40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:29.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.407+0000 7fcb8ffff640 1 -- 172.21.15.37:0/736105139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9013bf50 msgr2=0x7fcb9013c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:29.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.407+0000 7fcb8ffff640 1 --2- 172.21.15.37:0/736105139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9013bf50 0x7fcb9013c330 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcb8800d380 tx=0x7fcb8800d840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:29.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.407+0000 7fcb87fff640 1 -- 172.21.15.37:0/736105139 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb88016020 con 0x7fcb9013bf50 2024-04-26T00:38:29.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.407+0000 7fcb87fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:29.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.407+0000 7fcb87fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:38:29.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.407+0000 7fcb87fff640 10 monclient: dump: 2024-04-26T00:38:29.414 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:29.414 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:29.414 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:29.414 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:29.414 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:29.414 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:29.414 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:29.414 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:29.415 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:29.415 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:29.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.411+0000 7fcb87fff640 1 -- 172.21.15.37:0/736105139 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcb88012a10 con 0x7fcb9013bf50 2024-04-26T00:38:29.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.411+0000 7fcb87fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:29.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.411+0000 7fcb87fff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:38:29.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.411+0000 7fcb87fff640 1 -- 172.21.15.37:0/736105139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9013bf50 msgr2=0x7fcb9013c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:29.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.411+0000 7fcb87fff640 1 --2- 172.21.15.37:0/736105139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9013bf50 0x7fcb9013c330 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcb8800d380 tx=0x7fcb8800d840 comp rx=0 tx=0).stop 2024-04-26T00:38:29.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.411+0000 7fcb87fff640 1 -- 172.21.15.37:0/736105139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9013bf50 msgr2=0x7fcb9013c330 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:38:29.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.411+0000 7fcb87fff640 1 --2- 172.21.15.37:0/736105139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb9013bf50 0x7fcb9013c330 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcb8800d380 tx=0x7fcb8800d840 comp rx=0 tx=0).stop 2024-04-26T00:38:29.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.415+0000 7fcb95e1a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:38:29.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.415+0000 7fcb95e1a640 1 --2- 172.21.15.37:0/736105139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb90137de0 0x7fcb901381f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:29.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.415+0000 7fcb95e1a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:29.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.415+0000 7fcb95e1a640 1 -- 172.21.15.37:0/736105139 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcb9004f510 con 0x7fcb90137de0 2024-04-26T00:38:29.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.415+0000 7fcb8ffff640 1 --2- 172.21.15.37:0/736105139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb90137de0 0x7fcb901381f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:29.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.415+0000 7fcb8ffff640 10 monclient: get_auth_request con 0x7fcb90137de0 auth_method 0 2024-04-26T00:38:29.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.415+0000 7fcb8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:29.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.415+0000 7fcb8ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:29.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.415+0000 7fcb8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:29.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.415+0000 7fcb8ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:29.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.415+0000 7fcb8ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:29.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.415+0000 7fcb8ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:29.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.419+0000 7fcb8ffff640 10 monclient(hunting): handle_auth_done global_id 224149 payload 824 2024-04-26T00:38:29.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.419+0000 7fcb8ffff640 1 -- 172.21.15.37:0/736105139 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcb900a3aa0 con 0x7fcb90137de0 2024-04-26T00:38:29.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.419+0000 7fcb8ffff640 1 -- 172.21.15.37:0/736105139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb90137de0 msgr2=0x7fcb901381f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:29.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.419+0000 7fcb8ffff640 1 --2- 172.21.15.37:0/736105139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb90137de0 0x7fcb901381f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcb8800d380 tx=0x7fcb88003770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:29.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.419+0000 7fcb87fff640 1 -- 172.21.15.37:0/736105139 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb88016040 con 0x7fcb90137de0 2024-04-26T00:38:29.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.419+0000 7fcb87fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:29.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.419+0000 7fcb87fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:38:29.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.419+0000 7fcb87fff640 10 monclient: dump: 2024-04-26T00:38:29.419 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:29.419 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:29.419 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:29.419 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:29.419 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:29.419 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:29.419 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:29.419 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:29.419 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:29.419 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:29.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.419+0000 7fcb8ffff640 1 -- 172.21.15.37:0/736105139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb90137de0 msgr2=0x7fcb901381f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:38:29.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.419+0000 7fcb8ffff640 1 -- 172.21.15.37:0/736105139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb90137de0 msgr2=0x7fcb901381f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:38:29.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.419+0000 7fcb8ffff640 1 --2- 172.21.15.37:0/736105139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb90137de0 0x7fcb901381f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcb8800d380 tx=0x7fcb88003770 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:38:29.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.419+0000 7fcb8ffff640 1 --2- 172.21.15.37:0/736105139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb90137de0 0x7fcb901381f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcb8800d380 tx=0x7fcb88003770 comp rx=0 tx=0).stop 2024-04-26T00:38:29.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.419+0000 7fcb87fff640 1 -- 172.21.15.37:0/736105139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb90137de0 msgr2=0x7fcb901381f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:38:29.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.419+0000 7fcb87fff640 1 --2- 172.21.15.37:0/736105139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb90137de0 0x7fcb901381f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:29.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.419+0000 7fcb87fff640 1 --2- 172.21.15.37:0/736105139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb7003bd80 0x7fcb7003e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:29.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.419+0000 7fcb87fff640 1 -- 172.21.15.37:0/736105139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb90137de0 msgr2=0x7fcb901381f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:38:29.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.419+0000 7fcb87fff640 1 --2- 172.21.15.37:0/736105139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb90137de0 0x7fcb901381f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:29.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.419+0000 7fcb87fff640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:29.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.419+0000 7fcb87fff640 1 -- 172.21.15.37:0/736105139 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcb7003e780 con 0x7fcb7003bd80 2024-04-26T00:38:29.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.419+0000 7fcb8ffff640 1 --2- 172.21.15.37:0/736105139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb7003bd80 0x7fcb7003e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:29.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.419+0000 7fcb8ffff640 10 monclient: get_auth_request con 0x7fcb7003bd80 auth_method 0 2024-04-26T00:38:29.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.423+0000 7fcb8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:29.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.423+0000 7fcb8ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:29.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.423+0000 7fcb8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:29.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.423+0000 7fcb8ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:29.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.423+0000 7fcb8ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:29.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.423+0000 7fcb8ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:29.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.423+0000 7fcb8ffff640 10 monclient(hunting): handle_auth_done global_id 224152 payload 824 2024-04-26T00:38:29.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.423+0000 7fcb8ffff640 1 -- 172.21.15.37:0/736105139 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcb88012a10 con 0x7fcb7003bd80 2024-04-26T00:38:29.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.423+0000 7fcb8ffff640 1 -- 172.21.15.37:0/736105139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb7003bd80 msgr2=0x7fcb7003e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:29.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.423+0000 7fcb8ffff640 1 --2- 172.21.15.37:0/736105139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb7003bd80 0x7fcb7003e170 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcb8800d880 tx=0x7fcb88009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:29.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.427+0000 7fcb87fff640 1 -- 172.21.15.37:0/736105139 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb88034030 con 0x7fcb7003bd80 2024-04-26T00:38:29.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.427+0000 7fcb87fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:29.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.427+0000 7fcb87fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:38:29.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.427+0000 7fcb87fff640 10 monclient: dump: 2024-04-26T00:38:29.427 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:29.427 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:29.427 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:29.427 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:29.428 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:29.428 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:29.428 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:29.428 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:29.428 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:29.428 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:29.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.427+0000 7fcb87fff640 1 -- 172.21.15.37:0/736105139 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7fcb88025c00 con 0x7fcb7003bd80 2024-04-26T00:38:29.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.427+0000 7fcb87fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:38:29.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.427+0000 7fcb87fff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:38:29.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.427+0000 7fcb87fff640 1 -- 172.21.15.37:0/736105139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb7003bd80 msgr2=0x7fcb7003e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:29.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.427+0000 7fcb87fff640 1 --2- 172.21.15.37:0/736105139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb7003bd80 0x7fcb7003e170 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcb8800d880 tx=0x7fcb88009040 comp rx=0 tx=0).stop 2024-04-26T00:38:29.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.427+0000 7fcb87fff640 1 -- 172.21.15.37:0/736105139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb7003bd80 msgr2=0x7fcb7003e170 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:38:29.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.427+0000 7fcb87fff640 1 --2- 172.21.15.37:0/736105139 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb7003bd80 0x7fcb7003e170 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcb8800d880 tx=0x7fcb88009040 comp rx=0 tx=0).stop 2024-04-26T00:38:29.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.427+0000 7fcb8ffff640 1 -- 172.21.15.37:0/736105139 reap_dead start 2024-04-26T00:38:29.429 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:38:29.429 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:38:29.429 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:38:29.429 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:38:29.430 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 140, 2024-04-26T00:38:29.430 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 23087, 2024-04-26T00:38:29.430 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:38:29.430 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:38:29.430 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:38:29.430 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:38:29.430 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:38:29.431 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:38:29.431 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:38:29.431 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:38:29.431 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:38:29.431 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:38:29.431 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:38:29.431 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:38:29.431 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:38:29.431 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:38:29.431 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:38:29.431 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:38:29.432 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:38:29.432 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:38:29.432 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:38:29.432 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:38:29.432 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:38:29.432 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:38:29.432 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:38:29.432 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:38:29.432 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:38:29.432 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:38:29.432 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:38:29.432 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:38:29.432 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:38:29.433 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:38:29.433 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:38:29.433 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:38:29.433 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:38:29.433 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:29.433 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:29.433 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:38:29.433 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:38:29.433 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:38:29.433 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:38:29.433 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:38:29.433 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:38:29.434 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:38:29.434 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:38:29.434 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:38:29.434 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:38:29.434 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:38:29.434 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:38:29.434 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:38:29.434 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:38:29.434 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:38:29.434 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:38:29.434 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:38:29.434 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:38:29.435 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:38:29.435 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:38:29.435 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:38:29.435 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:38:29.435 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:38:29.435 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:38:29.435 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:38:29.435 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:38:29.435 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:38:29.435 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:38:29.435 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:38:29.435 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:38:29.436 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:29.436 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:38:29.436 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:29.436 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:38:29.436 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:38:29.436 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:38:29.436 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:38:29.436 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:29.436 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:38:29.437 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:38:29.437 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:29.437 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:29.437 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:29.437 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:38:29.437 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:38:29.437 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:29.437 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:29.437 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:29.438 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:29.438 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:38:29.438 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:38:29.438 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:38:29.438 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:38:29.438 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:38:29.438 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:29.438 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:29.438 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:38:29.438 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:38:29.438 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:38:29.438 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:38:29.438 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:29.439 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:38:29.439 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:38:29.439 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:29.439 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:29.439 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:29.439 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:38:29.439 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:38:29.439 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:29.439 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:29.439 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:29.440 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:29.440 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:38:29.440 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:38:29.440 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:38:29.440 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:38:29.440 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:38:29.440 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:29.440 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:29.440 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:38:29.440 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:38:29.441 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:38:29.441 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:38:29.441 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:29.441 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:38:29.441 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:38:29.441 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:29.441 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:29.441 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:29.441 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:38:29.441 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:38:29.441 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:38:29.442 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:29.442 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:29.442 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:29.442 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:38:29.442 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:38:29.442 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:38:29.442 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:38:29.442 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:38:29.442 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:29.442 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:29.442 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:29.443 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:38:29.443 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:38:29.443 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:29.443 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:38:29.443 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:38:29.443 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:38:29.443 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:29.443 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:38:29.443 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:38:29.443 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:29.443 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:38:29.443 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:38:29.444 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:38:29.444 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:29.444 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:38:29.444 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:38:29.444 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:29.444 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:38:29.444 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:38:29.444 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:38:29.444 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:29.444 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:38:29.444 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:38:29.445 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:38:29.445 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:38:29.445 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:38:29.445 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:38:29.445 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:38:29.445 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:38:29.445 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:38:29.445 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:38:29.445 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:38:29.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.427+0000 7fcb95e1a640 1 -- 172.21.15.37:0/736105139 >> v2:172.21.15.45:6812/2379918623 conn(0x7fcb70035e80 msgr2=0x7fcb70038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:29.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.427+0000 7fcb95e1a640 1 --2- 172.21.15.37:0/736105139 >> v2:172.21.15.45:6812/2379918623 conn(0x7fcb70035e80 0x7fcb70038340 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fcb80006fd0 tx=0x7fcb80006d40 comp rx=0 tx=0).stop 2024-04-26T00:38:29.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.427+0000 7fcb95e1a640 10 monclient: shutdown 2024-04-26T00:38:29.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.427+0000 7fcb95e1a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:38:29.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.427+0000 7fcb95e1a640 1 -- 172.21.15.37:0/736105139 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb900a0110 msgr2=0x7fcb90131900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:29.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.427+0000 7fcb95e1a640 1 --2- 172.21.15.37:0/736105139 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb900a0110 0x7fcb90131900 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fcb780af040 tx=0x7fcb78019460 comp rx=0 tx=0).stop 2024-04-26T00:38:29.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.427+0000 7fcb95e1a640 1 -- 172.21.15.37:0/736105139 shutdown_connections 2024-04-26T00:38:29.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.427+0000 7fcb95e1a640 1 --2- 172.21.15.37:0/736105139 >> v2:172.21.15.45:6812/2379918623 conn(0x7fcb70035e80 0x7fcb70038340 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:29.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.427+0000 7fcb95e1a640 1 --2- 172.21.15.37:0/736105139 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcb900a0110 0x7fcb90131900 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:29.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.427+0000 7fcb95e1a640 1 -- 172.21.15.37:0/736105139 >> 172.21.15.37:0/736105139 conn(0x7fcb900a1d50 msgr2=0x7fcb900a4ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:38:29.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.427+0000 7fcb95e1a640 1 -- 172.21.15.37:0/736105139 shutdown_connections 2024-04-26T00:38:29.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:29.427+0000 7fcb95e1a640 1 -- 172.21.15.37:0/736105139 wait complete. 2024-04-26T00:38:29.448 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:38:29.448 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-04-26T00:38:29.449 INFO:tasks.rados.rados.0.smithi045.stdout:3224: done (1 left) 2024-04-26T00:38:29.449 INFO:tasks.rados.rados.0.smithi045.stdout:3225: done (0 left) 2024-04-26T00:38:29.449 INFO:tasks.rados.rados.0.smithi045.stdout:3226: snap_create 2024-04-26T00:38:30.777 INFO:tasks.rados.rados.0.smithi045.stdout:3226: done (0 left) 2024-04-26T00:38:30.777 INFO:tasks.rados.rados.0.smithi045.stdout:3227: read oid 19 snap 335 2024-04-26T00:38:30.777 INFO:tasks.rados.rados.0.smithi045.stdout:3227: expect (ObjNum 1055 snap 318 seq_num 1055) 2024-04-26T00:38:30.777 INFO:tasks.rados.rados.0.smithi045.stdout:3228: copy_from oid 41 from oid 45 current snap is 353 2024-04-26T00:38:30.777 INFO:tasks.rados.rados.0.smithi045.stdout:3229: read oid 2 snap -1 2024-04-26T00:38:30.777 INFO:tasks.rados.rados.0.smithi045.stdout:3229: expect deleted 2024-04-26T00:38:30.778 INFO:tasks.rados.rados.0.smithi045.stdout:3230: snap_create 2024-04-26T00:38:30.808 INFO:tasks.rados.rados.0.smithi045.stdout:3227: expect (ObjNum 1055 snap 318 seq_num 1055) 2024-04-26T00:38:30.986 INFO:tasks.rados.rados.0.smithi045.stdout:3228: finishing copy_from racing read to smithi04526579-41 2024-04-26T00:38:30.986 INFO:tasks.rados.rados.0.smithi045.stdout:3228: finishing copy_from to smithi04526579-41 2024-04-26T00:38:30.986 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 2198 (ObjNum 1062 snap 318 seq_num 1062) dirty exists 2024-04-26T00:38:31.779 INFO:tasks.rados.rados.0.smithi045.stdout:3227: done (3 left) 2024-04-26T00:38:31.779 INFO:tasks.rados.rados.0.smithi045.stdout:3228: done (2 left) 2024-04-26T00:38:31.779 INFO:tasks.rados.rados.0.smithi045.stdout:3229: done (1 left) 2024-04-26T00:38:31.779 INFO:tasks.rados.rados.0.smithi045.stdout:3230: done (0 left) 2024-04-26T00:38:31.779 INFO:tasks.rados.rados.0.smithi045.stdout:3231: snap_create 2024-04-26T00:38:32.782 INFO:tasks.rados.rados.0.smithi045.stdout:3231: done (0 left) 2024-04-26T00:38:32.782 INFO:tasks.rados.rados.0.smithi045.stdout:3232: write oid 14 current snap is 355 2024-04-26T00:38:32.782 INFO:tasks.rados.rados.0.smithi045.stdout:3232: seq_num 1154 ranges {595278=406649,1729632=532069,3007663=387615} 2024-04-26T00:38:32.798 INFO:tasks.rados.rados.0.smithi045.stdout:3232: writing smithi04526579-14 from 595278 to 1001927 tid 1 2024-04-26T00:38:32.819 INFO:tasks.rados.rados.0.smithi045.stdout:3232: writing smithi04526579-14 from 1729632 to 2261701 tid 2 2024-04-26T00:38:32.837 INFO:tasks.rados.rados.0.smithi045.stdout:3232: writing smithi04526579-14 from 3007663 to 3395278 tid 3 2024-04-26T00:38:32.837 INFO:tasks.rados.rados.0.smithi045.stdout:3233: copy_from oid 11 from oid 15 current snap is 355 2024-04-26T00:38:32.837 INFO:tasks.rados.rados.0.smithi045.stdout:3232: finishing write tid 1 to smithi04526579-14 2024-04-26T00:38:32.837 INFO:tasks.rados.rados.0.smithi045.stdout:3232: finishing write tid 2 to smithi04526579-14 2024-04-26T00:38:32.837 INFO:tasks.rados.rados.0.smithi045.stdout:3234: write oid 30 current snap is 355 2024-04-26T00:38:32.837 INFO:tasks.rados.rados.0.smithi045.stdout:3234: seq_num 1155 ranges {612085=481898,1788114=675727,3035086=376999} 2024-04-26T00:38:32.850 INFO:tasks.rados.rados.0.smithi045.stdout:3234: writing smithi04526579-30 from 612085 to 1093983 tid 1 2024-04-26T00:38:32.864 INFO:tasks.rados.rados.0.smithi045.stdout:3234: writing smithi04526579-30 from 1788114 to 2463841 tid 2 2024-04-26T00:38:32.873 INFO:tasks.rados.rados.0.smithi045.stdout:3234: writing smithi04526579-30 from 3035086 to 3412085 tid 3 2024-04-26T00:38:32.873 INFO:tasks.rados.rados.0.smithi045.stdout:3235: snap_create 2024-04-26T00:38:32.873 INFO:tasks.rados.rados.0.smithi045.stdout:3232: finishing write tid 3 to smithi04526579-14 2024-04-26T00:38:32.873 INFO:tasks.rados.rados.0.smithi045.stdout:3232: finishing write tid 4 to smithi04526579-14 2024-04-26T00:38:32.873 INFO:tasks.rados.rados.0.smithi045.stdout:3232: finishing write tid 5 to smithi04526579-14 2024-04-26T00:38:32.873 INFO:tasks.rados.rados.0.smithi045.stdout:3232: oid 14 updating version 0 to 1201 2024-04-26T00:38:32.873 INFO:tasks.rados.rados.0.smithi045.stdout:3232: oid 14 updating version 1201 to 1202 2024-04-26T00:38:32.873 INFO:tasks.rados.rados.0.smithi045.stdout:3232: oid 14 updating version 1202 to 1203 2024-04-26T00:38:32.873 INFO:tasks.rados.rados.0.smithi045.stdout:3232: oid 14 version 1203 is already newer than 1200 2024-04-26T00:38:32.873 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 1203 (ObjNum 1154 snap 355 seq_num 1154) dirty exists 2024-04-26T00:38:32.873 INFO:tasks.rados.rados.0.smithi045.stdout:3232: left oid 14 (ObjNum 1154 snap 355 seq_num 1154) 2024-04-26T00:38:32.873 INFO:tasks.rados.rados.0.smithi045.stdout:3233: finishing copy_from racing read to smithi04526579-11 2024-04-26T00:38:32.874 INFO:tasks.rados.rados.0.smithi045.stdout:3233: finishing copy_from to smithi04526579-11 2024-04-26T00:38:32.874 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 2444 (ObjNum 1139 snap 345 seq_num 1139) dirty exists 2024-04-26T00:38:32.874 INFO:tasks.rados.rados.0.smithi045.stdout:3234: finishing write tid 1 to smithi04526579-30 2024-04-26T00:38:32.874 INFO:tasks.rados.rados.0.smithi045.stdout:3234: finishing write tid 2 to smithi04526579-30 2024-04-26T00:38:32.877 INFO:tasks.rados.rados.0.smithi045.stdout:3234: finishing write tid 3 to smithi04526579-30 2024-04-26T00:38:32.878 INFO:tasks.rados.rados.0.smithi045.stdout:3234: finishing write tid 4 to smithi04526579-30 2024-04-26T00:38:32.878 INFO:tasks.rados.rados.0.smithi045.stdout:3234: finishing write tid 5 to smithi04526579-30 2024-04-26T00:38:32.878 INFO:tasks.rados.rados.0.smithi045.stdout:3234: oid 30 updating version 0 to 1995 2024-04-26T00:38:32.879 INFO:tasks.rados.rados.0.smithi045.stdout:3234: oid 30 updating version 1995 to 1998 2024-04-26T00:38:32.879 INFO:tasks.rados.rados.0.smithi045.stdout:3234: oid 30 version 1998 is already newer than 1996 2024-04-26T00:38:32.879 INFO:tasks.rados.rados.0.smithi045.stdout:3234: oid 30 version 1998 is already newer than 1997 2024-04-26T00:38:32.879 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 1998 (ObjNum 1155 snap 355 seq_num 1155) dirty exists 2024-04-26T00:38:32.879 INFO:tasks.rados.rados.0.smithi045.stdout:3234: left oid 30 (ObjNum 1155 snap 355 seq_num 1155) 2024-04-26T00:38:33.797 INFO:tasks.rados.rados.0.smithi045.stdout:3232: done (3 left) 2024-04-26T00:38:33.797 INFO:tasks.rados.rados.0.smithi045.stdout:3233: done (2 left) 2024-04-26T00:38:33.797 INFO:tasks.rados.rados.0.smithi045.stdout:3234: done (1 left) 2024-04-26T00:38:33.797 INFO:tasks.rados.rados.0.smithi045.stdout:3235: done (0 left) 2024-04-26T00:38:33.797 INFO:tasks.rados.rados.0.smithi045.stdout:3236: delete oid 1 current snap is 356 2024-04-26T00:38:33.804 INFO:tasks.rados.rados.0.smithi045.stdout:3236: done (0 left) 2024-04-26T00:38:33.804 INFO:tasks.rados.rados.0.smithi045.stdout:3237: snap_create 2024-04-26T00:38:34.428 INFO:tasks.rados.rados.0.smithi045.stdout:3237: done (0 left) 2024-04-26T00:38:34.428 INFO:tasks.rados.rados.0.smithi045.stdout:3238: snap_remove snap 354 2024-04-26T00:38:35.441 INFO:tasks.rados.rados.0.smithi045.stdout:3238: done (0 left) 2024-04-26T00:38:35.441 INFO:tasks.rados.rados.0.smithi045.stdout:3239: write (excl) oid 7 current snap is 357 2024-04-26T00:38:35.441 INFO:tasks.rados.rados.0.smithi045.stdout:3239: seq_num 1157 ranges {645699=748749,2021431=679396,3206285=239414} 2024-04-26T00:38:35.464 INFO:tasks.rados.rados.0.smithi045.stdout:3239: writing smithi04526579-7 from 645699 to 1394448 tid 1 2024-04-26T00:38:35.485 INFO:tasks.rados.rados.0.smithi045.stdout:3239: writing smithi04526579-7 from 2021431 to 2700827 tid 2 2024-04-26T00:38:35.496 INFO:tasks.rados.rados.0.smithi045.stdout:3239: writing smithi04526579-7 from 3206285 to 3445699 tid 3 2024-04-26T00:38:35.497 INFO:tasks.rados.rados.0.smithi045.stdout:3240: read oid 21 snap -1 2024-04-26T00:38:35.497 INFO:tasks.rados.rados.0.smithi045.stdout:3240: expect (ObjNum 1087 snap 324 seq_num 1087) 2024-04-26T00:38:35.497 INFO:tasks.rados.rados.0.smithi045.stdout:3241: delete oid 28 current snap is 357 2024-04-26T00:38:35.497 INFO:tasks.rados.rados.0.smithi045.stdout:3239: finishing write tid 1 to smithi04526579-7 2024-04-26T00:38:35.497 INFO:tasks.rados.rados.0.smithi045.stdout:3239: finishing write tid 2 to smithi04526579-7 2024-04-26T00:38:35.503 INFO:tasks.rados.rados.0.smithi045.stdout:3241: done (2 left) 2024-04-26T00:38:35.503 INFO:tasks.rados.rados.0.smithi045.stdout:3242: snap_remove snap 353 2024-04-26T00:38:36.448 INFO:tasks.rados.rados.0.smithi045.stdout:3242: done (2 left) 2024-04-26T00:38:36.448 INFO:tasks.rados.rados.0.smithi045.stdout:3243: snap_create 2024-04-26T00:38:36.449 INFO:tasks.rados.rados.0.smithi045.stdout:3239: finishing write tid 3 to smithi04526579-7 2024-04-26T00:38:36.449 INFO:tasks.rados.rados.0.smithi045.stdout:3240: expect (ObjNum 1087 snap 324 seq_num 1087) 2024-04-26T00:38:36.614 INFO:tasks.rados.rados.0.smithi045.stdout:3239: finishing write tid 4 to smithi04526579-7 2024-04-26T00:38:36.614 INFO:tasks.rados.rados.0.smithi045.stdout:3239: finishing write tid 5 to smithi04526579-7 2024-04-26T00:38:36.615 INFO:tasks.rados.rados.0.smithi045.stdout:3239: oid 7 updating version 0 to 2495 2024-04-26T00:38:36.615 INFO:tasks.rados.rados.0.smithi045.stdout:3239: oid 7 version 2495 is already newer than 2493 2024-04-26T00:38:36.615 INFO:tasks.rados.rados.0.smithi045.stdout:3239: oid 7 updating version 2495 to 2496 2024-04-26T00:38:36.615 INFO:tasks.rados.rados.0.smithi045.stdout:3239: oid 7 version 2496 is already newer than 2494 2024-04-26T00:38:36.615 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 2496 (ObjNum 1157 snap 357 seq_num 1157) dirty exists 2024-04-26T00:38:36.615 INFO:tasks.rados.rados.0.smithi045.stdout:3239: left oid 7 (ObjNum 1157 snap 357 seq_num 1157) 2024-04-26T00:38:37.448 INFO:tasks.rados.rados.0.smithi045.stdout:3239: done (2 left) 2024-04-26T00:38:37.449 INFO:tasks.rados.rados.0.smithi045.stdout:3240: done (1 left) 2024-04-26T00:38:37.449 INFO:tasks.rados.rados.0.smithi045.stdout:3243: done (0 left) 2024-04-26T00:38:37.449 INFO:tasks.rados.rados.0.smithi045.stdout:3244: snap_create 2024-04-26T00:38:38.586 INFO:tasks.rados.rados.0.smithi045.stdout:3244: done (0 left) 2024-04-26T00:38:38.586 INFO:tasks.rados.rados.0.smithi045.stdout:3245: delete oid 2 current snap is 359 2024-04-26T00:38:38.593 INFO:tasks.rados.rados.0.smithi045.stdout:3245: done (0 left) 2024-04-26T00:38:38.594 INFO:tasks.rados.rados.0.smithi045.stdout:3246: snap_remove snap 342 2024-04-26T00:38:39.593 INFO:tasks.rados.rados.0.smithi045.stdout:3246: done (0 left) 2024-04-26T00:38:39.593 INFO:tasks.rados.rados.0.smithi045.stdout:3247: copy_from oid 26 from oid 14 current snap is 359 2024-04-26T00:38:39.594 INFO:tasks.rados.rados.0.smithi045.stdout:3248: rollback oid 44 current snap is 359 2024-04-26T00:38:39.594 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 44 to 352 2024-04-26T00:38:39.594 INFO:tasks.rados.rados.0.smithi045.stdout:3249: copy_from oid 3 from oid 1 current snap is 359 2024-04-26T00:38:39.594 INFO:tasks.rados.rados.0.smithi045.stdout:3250: write oid 15 current snap is 359 2024-04-26T00:38:39.594 INFO:tasks.rados.rados.0.smithi045.stdout:3250: seq_num 1160 ranges {696120=574496,1680524=543076,2888554=512829} 2024-04-26T00:38:39.616 INFO:tasks.rados.rados.0.smithi045.stdout:3250: writing smithi04526579-15 from 696120 to 1270616 tid 1 2024-04-26T00:38:39.634 INFO:tasks.rados.rados.0.smithi045.stdout:3250: writing smithi04526579-15 from 1680524 to 2223600 tid 2 2024-04-26T00:38:39.652 INFO:tasks.rados.rados.0.smithi045.stdout:3250: writing smithi04526579-15 from 2888554 to 3401383 tid 3 2024-04-26T00:38:39.652 INFO:tasks.rados.rados.0.smithi045.stdout:3251: read oid 30 snap 340 2024-04-26T00:38:39.652 INFO:tasks.rados.rados.0.smithi045.stdout:3251: expect (ObjNum 1115 snap 332 seq_num 1115) 2024-04-26T00:38:39.652 INFO:tasks.rados.rados.0.smithi045.stdout:3252: rollback oid 6 current snap is 359 2024-04-26T00:38:39.652 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 6 to 356 2024-04-26T00:38:39.652 INFO:tasks.rados.rados.0.smithi045.stdout:3249: finishing copy_from to smithi04526579-3 2024-04-26T00:38:39.652 INFO:tasks.rados.rados.0.smithi045.stdout:3249: got expected ENOENT (src dne) 2024-04-26T00:38:39.653 INFO:tasks.rados.rados.0.smithi045.stdout:3249: finishing copy_from racing read to smithi04526579-3 2024-04-26T00:38:39.653 INFO:tasks.rados.rados.0.smithi045.stdout:3248: finishing rollback tid 0 to smithi04526579-44 2024-04-26T00:38:39.653 INFO:tasks.rados.rados.0.smithi045.stdout:3248: finishing rollback tid 1 to smithi04526579-44 2024-04-26T00:38:39.653 INFO:tasks.rados.rados.0.smithi045.stdout:3248: finishing rollback tid 2 to smithi04526579-44 2024-04-26T00:38:39.653 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 2382 (ObjNum 1127 snap 340 seq_num 1127) dirty exists 2024-04-26T00:38:39.653 INFO:tasks.rados.rados.0.smithi045.stdout:3247: finishing copy_from racing read to smithi04526579-26 2024-04-26T00:38:39.653 INFO:tasks.rados.rados.0.smithi045.stdout:3250: finishing write tid 1 to smithi04526579-15 2024-04-26T00:38:39.653 INFO:tasks.rados.rados.0.smithi045.stdout:3247: finishing copy_from to smithi04526579-26 2024-04-26T00:38:39.653 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 1203 (ObjNum 1154 snap 355 seq_num 1154) dirty exists 2024-04-26T00:38:39.653 INFO:tasks.rados.rados.0.smithi045.stdout:3250: finishing write tid 2 to smithi04526579-15 2024-04-26T00:38:39.653 INFO:tasks.rados.rados.0.smithi045.stdout:3247: done (5 left) 2024-04-26T00:38:39.653 INFO:tasks.rados.rados.0.smithi045.stdout:3248: done (4 left) 2024-04-26T00:38:39.653 INFO:tasks.rados.rados.0.smithi045.stdout:3249: done (3 left) 2024-04-26T00:38:39.653 INFO:tasks.rados.rados.0.smithi045.stdout:3253: rollback oid 9 current snap is 359 2024-04-26T00:38:39.653 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 9 to 346 2024-04-26T00:38:39.654 INFO:tasks.rados.rados.0.smithi045.stdout:3254: read oid 11 snap 340 2024-04-26T00:38:39.654 INFO:tasks.rados.rados.0.smithi045.stdout:3254: expect deleted 2024-04-26T00:38:39.654 INFO:tasks.rados.rados.0.smithi045.stdout:3255: delete oid 25 current snap is 359 2024-04-26T00:38:39.658 INFO:tasks.rados.rados.0.smithi045.stdout:3250: finishing write tid 3 to smithi04526579-15 2024-04-26T00:38:39.658 INFO:tasks.rados.rados.0.smithi045.stdout:3250: finishing write tid 4 to smithi04526579-15 2024-04-26T00:38:39.659 INFO:tasks.rados.rados.0.smithi045.stdout:3250: finishing write tid 5 to smithi04526579-15 2024-04-26T00:38:39.659 INFO:tasks.rados.rados.0.smithi045.stdout:3250: oid 15 updating version 0 to 2452 2024-04-26T00:38:39.659 INFO:tasks.rados.rados.0.smithi045.stdout:3250: oid 15 version 2452 is already newer than 2451 2024-04-26T00:38:39.659 INFO:tasks.rados.rados.0.smithi045.stdout:3250: oid 15 updating version 2452 to 2453 2024-04-26T00:38:39.659 INFO:tasks.rados.rados.0.smithi045.stdout:3250: oid 15 version 2453 is already newer than 2450 2024-04-26T00:38:39.659 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 2453 (ObjNum 1160 snap 359 seq_num 1160) dirty exists 2024-04-26T00:38:39.659 INFO:tasks.rados.rados.0.smithi045.stdout:3250: left oid 15 (ObjNum 1160 snap 359 seq_num 1160) 2024-04-26T00:38:39.661 INFO:tasks.rados.rados.0.smithi045.stdout:3250: done (5 left) 2024-04-26T00:38:39.661 INFO:tasks.rados.rados.0.smithi045.stdout:3254: done (4 left) 2024-04-26T00:38:39.661 INFO:tasks.rados.rados.0.smithi045.stdout:3255: done (3 left) 2024-04-26T00:38:39.661 INFO:tasks.rados.rados.0.smithi045.stdout:3256: rollback oid 36 current snap is 359 2024-04-26T00:38:39.661 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 36 to 345 2024-04-26T00:38:39.661 INFO:tasks.rados.rados.0.smithi045.stdout:3257: rollback oid 4 current snap is 359 2024-04-26T00:38:39.661 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 4 to 340 2024-04-26T00:38:39.662 INFO:tasks.rados.rados.0.smithi045.stdout:3258: write (excl) oid 49 current snap is 359 2024-04-26T00:38:39.662 INFO:tasks.rados.rados.0.smithi045.stdout:3258: seq_num 1162 ranges {729734=724994,2197488=430392,3227047=302687} 2024-04-26T00:38:39.680 INFO:tasks.rados.rados.0.smithi045.stdout:3258: writing smithi04526579-49 from 729734 to 1454728 tid 1 2024-04-26T00:38:39.695 INFO:tasks.rados.rados.0.smithi045.stdout:3258: writing smithi04526579-49 from 2197488 to 2627880 tid 2 2024-04-26T00:38:39.705 INFO:tasks.rados.rados.0.smithi045.stdout:3258: writing smithi04526579-49 from 3227047 to 3529734 tid 3 2024-04-26T00:38:39.705 INFO:tasks.rados.rados.0.smithi045.stdout:3259: snap_remove snap 357 2024-04-26T00:38:40.596 INFO:tasks.rados.rados.0.smithi045.stdout:3259: done (6 left) 2024-04-26T00:38:40.597 INFO:tasks.rados.rados.0.smithi045.stdout:3260: write oid 38 current snap is 359 2024-04-26T00:38:40.597 INFO:tasks.rados.rados.0.smithi045.stdout:3253: finishing rollback tid 0 to smithi04526579-9 2024-04-26T00:38:40.597 INFO:tasks.rados.rados.0.smithi045.stdout:3260: seq_num 1163 ranges {746541=400243,1739617=690403,3138117=408424} 2024-04-26T00:38:40.611 INFO:tasks.rados.rados.0.smithi045.stdout:3260: writing smithi04526579-38 from 746541 to 1146784 tid 1 2024-04-26T00:38:40.625 INFO:tasks.rados.rados.0.smithi045.stdout:3260: writing smithi04526579-38 from 1739617 to 2430020 tid 2 2024-04-26T00:38:40.637 INFO:tasks.rados.rados.0.smithi045.stdout:3260: writing smithi04526579-38 from 3138117 to 3546541 tid 3 2024-04-26T00:38:40.638 INFO:tasks.rados.rados.0.smithi045.stdout:3261: rollback oid 29 current snap is 359 2024-04-26T00:38:40.638 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 29 to 332 2024-04-26T00:38:40.638 INFO:tasks.rados.rados.0.smithi045.stdout:3251: expect (ObjNum 1115 snap 332 seq_num 1115) 2024-04-26T00:38:40.845 INFO:tasks.rados.rados.0.smithi045.stdout:3253: finishing rollback tid 1 to smithi04526579-9 2024-04-26T00:38:40.846 INFO:tasks.rados.rados.0.smithi045.stdout:3253: finishing rollback tid 2 to smithi04526579-9 2024-04-26T00:38:40.846 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 2060 (ObjNum 1099 snap 326 seq_num 1099) dirty exists 2024-04-26T00:38:40.846 INFO:tasks.rados.rados.0.smithi045.stdout:3252: finishing rollback tid 0 to smithi04526579-6 2024-04-26T00:38:40.846 INFO:tasks.rados.rados.0.smithi045.stdout:3251: done (7 left) 2024-04-26T00:38:40.846 INFO:tasks.rados.rados.0.smithi045.stdout:3253: done (6 left) 2024-04-26T00:38:40.846 INFO:tasks.rados.rados.0.smithi045.stdout:3262: write (excl) oid 3 current snap is 359 2024-04-26T00:38:40.846 INFO:tasks.rados.rados.0.smithi045.stdout:3262: seq_num 1164 ranges {763348=591845,1914452=434061,2881893=674270} 2024-04-26T00:38:40.853 INFO:tasks.rados.rados.0.smithi045.stdout:3262: writing smithi04526579-3 from 763348 to 1355193 tid 1 2024-04-26T00:38:40.864 INFO:tasks.rados.rados.0.smithi045.stdout:3262: writing smithi04526579-3 from 1914452 to 2348513 tid 2 2024-04-26T00:38:40.882 INFO:tasks.rados.rados.0.smithi045.stdout:3262: writing smithi04526579-3 from 2881893 to 3556163 tid 3 2024-04-26T00:38:40.882 INFO:tasks.rados.rados.0.smithi045.stdout:3263: write oid 37 current snap is 359 2024-04-26T00:38:40.883 INFO:tasks.rados.rados.0.smithi045.stdout:3252: finishing rollback tid 1 to smithi04526579-6 2024-04-26T00:38:40.883 INFO:tasks.rados.rados.0.smithi045.stdout:3252: finishing rollback tid 2 to smithi04526579-6 2024-04-26T00:38:40.883 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 2001 (ObjNum 1060 snap 318 seq_num 1060) dirty exists 2024-04-26T00:38:40.883 INFO:tasks.rados.rados.0.smithi045.stdout:3256: finishing rollback tid 0 to smithi04526579-36 2024-04-26T00:38:40.883 INFO:tasks.rados.rados.0.smithi045.stdout:3256: finishing rollback tid 1 to smithi04526579-36 2024-04-26T00:38:40.883 INFO:tasks.rados.rados.0.smithi045.stdout:3256: finishing rollback tid 2 to smithi04526579-36 2024-04-26T00:38:40.883 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 2435 (ObjNum 1118 snap 337 seq_num 1118) dirty exists 2024-04-26T00:38:40.883 INFO:tasks.rados.rados.0.smithi045.stdout:3257: finishing rollback tid 0 to smithi04526579-4 2024-04-26T00:38:40.883 INFO:tasks.rados.rados.0.smithi045.stdout:3257: finishing rollback tid 1 to smithi04526579-4 2024-04-26T00:38:40.883 INFO:tasks.rados.rados.0.smithi045.stdout:3257: finishing rollback tid 2 to smithi04526579-4 2024-04-26T00:38:40.883 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 2063 (ObjNum 1080 snap 323 seq_num 1080) dirty exists 2024-04-26T00:38:40.883 INFO:tasks.rados.rados.0.smithi045.stdout:3258: finishing write tid 1 to smithi04526579-49 2024-04-26T00:38:40.883 INFO:tasks.rados.rados.0.smithi045.stdout:3263: seq_num 1165 ranges {780155=667094,1972934=694072,3309316=270839} 2024-04-26T00:38:40.893 INFO:tasks.rados.rados.0.smithi045.stdout:3263: writing smithi04526579-37 from 780155 to 1447249 tid 1 2024-04-26T00:38:40.907 INFO:tasks.rados.rados.0.smithi045.stdout:3263: writing smithi04526579-37 from 1972934 to 2667006 tid 2 2024-04-26T00:38:40.916 INFO:tasks.rados.rados.0.smithi045.stdout:3263: writing smithi04526579-37 from 3309316 to 3580155 tid 3 2024-04-26T00:38:40.916 INFO:tasks.rados.rados.0.smithi045.stdout:3252: done (7 left) 2024-04-26T00:38:40.916 INFO:tasks.rados.rados.0.smithi045.stdout:3256: done (6 left) 2024-04-26T00:38:40.916 INFO:tasks.rados.rados.0.smithi045.stdout:3257: done (5 left) 2024-04-26T00:38:40.916 INFO:tasks.rados.rados.0.smithi045.stdout:3264: copy_from oid 34 from oid 12 current snap is 359 2024-04-26T00:38:40.916 INFO:tasks.rados.rados.0.smithi045.stdout:3258: finishing write tid 2 to smithi04526579-49 2024-04-26T00:38:40.916 INFO:tasks.rados.rados.0.smithi045.stdout:3258: finishing write tid 3 to smithi04526579-49 2024-04-26T00:38:40.916 INFO:tasks.rados.rados.0.smithi045.stdout:3258: finishing write tid 4 to smithi04526579-49 2024-04-26T00:38:40.917 INFO:tasks.rados.rados.0.smithi045.stdout:3258: finishing write tid 5 to smithi04526579-49 2024-04-26T00:38:40.917 INFO:tasks.rados.rados.0.smithi045.stdout:3258: oid 49 updating version 0 to 2354 2024-04-26T00:38:40.917 INFO:tasks.rados.rados.0.smithi045.stdout:3258: oid 49 updating version 2354 to 2356 2024-04-26T00:38:40.917 INFO:tasks.rados.rados.0.smithi045.stdout:3258: oid 49 version 2356 is already newer than 2355 2024-04-26T00:38:40.917 INFO:tasks.rados.rados.0.smithi045.stdout:3258: oid 49 version 2356 is already newer than 2353 2024-04-26T00:38:40.917 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 2356 (ObjNum 1162 snap 359 seq_num 1162) dirty exists 2024-04-26T00:38:40.917 INFO:tasks.rados.rados.0.smithi045.stdout:3258: left oid 49 (ObjNum 1162 snap 359 seq_num 1162) 2024-04-26T00:38:40.917 INFO:tasks.rados.rados.0.smithi045.stdout:3258: done (5 left) 2024-04-26T00:38:40.917 INFO:tasks.rados.rados.0.smithi045.stdout:3265: snap_remove snap 340 2024-04-26T00:38:41.612 INFO:tasks.rados.rados.0.smithi045.stdout:3265: done (5 left) 2024-04-26T00:38:41.612 INFO:tasks.rados.rados.0.smithi045.stdout:3266: snap_create 2024-04-26T00:38:41.612 INFO:tasks.rados.rados.0.smithi045.stdout:3260: finishing write tid 1 to smithi04526579-38 2024-04-26T00:38:41.612 INFO:tasks.rados.rados.0.smithi045.stdout:3260: finishing write tid 2 to smithi04526579-38 2024-04-26T00:38:41.612 INFO:tasks.rados.rados.0.smithi045.stdout:3260: finishing write tid 3 to smithi04526579-38 2024-04-26T00:38:41.612 INFO:tasks.rados.rados.0.smithi045.stdout:3260: finishing write tid 4 to smithi04526579-38 2024-04-26T00:38:41.612 INFO:tasks.rados.rados.0.smithi045.stdout:3260: finishing write tid 5 to smithi04526579-38 2024-04-26T00:38:41.613 INFO:tasks.rados.rados.0.smithi045.stdout:3260: oid 38 updating version 0 to 2386 2024-04-26T00:38:41.613 INFO:tasks.rados.rados.0.smithi045.stdout:3260: oid 38 version 2386 is already newer than 2383 2024-04-26T00:38:41.613 INFO:tasks.rados.rados.0.smithi045.stdout:3260: oid 38 version 2386 is already newer than 2385 2024-04-26T00:38:41.613 INFO:tasks.rados.rados.0.smithi045.stdout:3260: oid 38 version 2386 is already newer than 2384 2024-04-26T00:38:41.613 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 2386 (ObjNum 1163 snap 359 seq_num 1163) dirty exists 2024-04-26T00:38:41.613 INFO:tasks.rados.rados.0.smithi045.stdout:3260: left oid 38 (ObjNum 1163 snap 359 seq_num 1163) 2024-04-26T00:38:41.613 INFO:tasks.rados.rados.0.smithi045.stdout:3261: finishing rollback tid 0 to smithi04526579-29 2024-04-26T00:38:41.613 INFO:tasks.rados.rados.0.smithi045.stdout:3261: finishing rollback tid 1 to smithi04526579-29 2024-04-26T00:38:41.613 INFO:tasks.rados.rados.0.smithi045.stdout:3261: finishing rollback tid 2 to smithi04526579-29 2024-04-26T00:38:41.613 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 2207 (ObjNum 987 snap 302 seq_num 987) dirty exists 2024-04-26T00:38:41.613 INFO:tasks.rados.rados.0.smithi045.stdout:3262: finishing write tid 1 to smithi04526579-3 2024-04-26T00:38:41.613 INFO:tasks.rados.rados.0.smithi045.stdout:3262: finishing write tid 2 to smithi04526579-3 2024-04-26T00:38:41.613 INFO:tasks.rados.rados.0.smithi045.stdout:3262: finishing write tid 3 to smithi04526579-3 2024-04-26T00:38:41.613 INFO:tasks.rados.rados.0.smithi045.stdout:3262: finishing write tid 4 to smithi04526579-3 2024-04-26T00:38:41.613 INFO:tasks.rados.rados.0.smithi045.stdout:3262: finishing write tid 5 to smithi04526579-3 2024-04-26T00:38:41.614 INFO:tasks.rados.rados.0.smithi045.stdout:3262: oid 3 updating version 0 to 2360 2024-04-26T00:38:41.614 INFO:tasks.rados.rados.0.smithi045.stdout:3262: oid 3 version 2360 is already newer than 2357 2024-04-26T00:38:41.614 INFO:tasks.rados.rados.0.smithi045.stdout:3262: oid 3 version 2360 is already newer than 2359 2024-04-26T00:38:41.614 INFO:tasks.rados.rados.0.smithi045.stdout:3262: oid 3 version 2360 is already newer than 2358 2024-04-26T00:38:41.614 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 2360 (ObjNum 1164 snap 359 seq_num 1164) dirty exists 2024-04-26T00:38:41.614 INFO:tasks.rados.rados.0.smithi045.stdout:3262: left oid 3 (ObjNum 1164 snap 359 seq_num 1164) 2024-04-26T00:38:41.614 INFO:tasks.rados.rados.0.smithi045.stdout:3263: finishing write tid 1 to smithi04526579-37 2024-04-26T00:38:41.614 INFO:tasks.rados.rados.0.smithi045.stdout:3263: finishing write tid 2 to smithi04526579-37 2024-04-26T00:38:41.614 INFO:tasks.rados.rados.0.smithi045.stdout:3263: finishing write tid 3 to smithi04526579-37 2024-04-26T00:38:41.614 INFO:tasks.rados.rados.0.smithi045.stdout:3263: finishing write tid 4 to smithi04526579-37 2024-04-26T00:38:41.614 INFO:tasks.rados.rados.0.smithi045.stdout:3263: finishing write tid 5 to smithi04526579-37 2024-04-26T00:38:41.614 INFO:tasks.rados.rados.0.smithi045.stdout:3263: oid 37 updating version 0 to 2504 2024-04-26T00:38:41.614 INFO:tasks.rados.rados.0.smithi045.stdout:3263: oid 37 updating version 2504 to 2505 2024-04-26T00:38:41.614 INFO:tasks.rados.rados.0.smithi045.stdout:3263: oid 37 updating version 2505 to 2507 2024-04-26T00:38:41.615 INFO:tasks.rados.rados.0.smithi045.stdout:3263: oid 37 version 2507 is already newer than 2506 2024-04-26T00:38:41.615 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 2507 (ObjNum 1165 snap 359 seq_num 1165) dirty exists 2024-04-26T00:38:41.615 INFO:tasks.rados.rados.0.smithi045.stdout:3263: left oid 37 (ObjNum 1165 snap 359 seq_num 1165) 2024-04-26T00:38:41.615 INFO:tasks.rados.rados.0.smithi045.stdout:3264: finishing copy_from racing read to smithi04526579-34 2024-04-26T00:38:41.615 INFO:tasks.rados.rados.0.smithi045.stdout:3264: finishing copy_from to smithi04526579-34 2024-04-26T00:38:41.615 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 2339 (ObjNum 1112 snap 331 seq_num 1112) dirty exists 2024-04-26T00:38:42.614 INFO:tasks.rados.rados.0.smithi045.stdout:3260: done (5 left) 2024-04-26T00:38:42.614 INFO:tasks.rados.rados.0.smithi045.stdout:3261: done (4 left) 2024-04-26T00:38:42.614 INFO:tasks.rados.rados.0.smithi045.stdout:3262: done (3 left) 2024-04-26T00:38:42.614 INFO:tasks.rados.rados.0.smithi045.stdout:3263: done (2 left) 2024-04-26T00:38:42.614 INFO:tasks.rados.rados.0.smithi045.stdout:3264: done (1 left) 2024-04-26T00:38:42.614 INFO:tasks.rados.rados.0.smithi045.stdout:3266: done (0 left) 2024-04-26T00:38:42.614 INFO:tasks.rados.rados.0.smithi045.stdout:3267: write (excl) oid 7 current snap is 360 2024-04-26T00:38:42.615 INFO:tasks.rados.rados.0.smithi045.stdout:3267: seq_num 1166 ranges {796962=742343,2031416=437730,2936739=413167} 2024-04-26T00:38:42.632 INFO:tasks.rados.rados.0.smithi045.stdout:3267: writing smithi04526579-7 from 796962 to 1539305 tid 1 2024-04-26T00:38:42.643 INFO:tasks.rados.rados.0.smithi045.stdout:3267: writing smithi04526579-7 from 2031416 to 2469146 tid 2 2024-04-26T00:38:42.653 INFO:tasks.rados.rados.0.smithi045.stdout:3267: writing smithi04526579-7 from 2936739 to 3349906 tid 3 2024-04-26T00:38:42.654 INFO:tasks.rados.rados.0.smithi045.stdout:3268: snap_remove snap 350 2024-04-26T00:38:43.632 INFO:tasks.rados.rados.0.smithi045.stdout:3268: done (1 left) 2024-04-26T00:38:43.633 INFO:tasks.rados.rados.0.smithi045.stdout:3269: rollback oid 30 current snap is 360 2024-04-26T00:38:43.633 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 30 to 356 2024-04-26T00:38:43.633 INFO:tasks.rados.rados.0.smithi045.stdout:3267: finishing write tid 1 to smithi04526579-7 2024-04-26T00:38:43.633 INFO:tasks.rados.rados.0.smithi045.stdout:3267: finishing write tid 2 to smithi04526579-7 2024-04-26T00:38:43.633 INFO:tasks.rados.rados.0.smithi045.stdout:3267: finishing write tid 3 to smithi04526579-7 2024-04-26T00:38:43.634 INFO:tasks.rados.rados.0.smithi045.stdout:3267: finishing write tid 4 to smithi04526579-7 2024-04-26T00:38:43.634 INFO:tasks.rados.rados.0.smithi045.stdout:3267: finishing write tid 5 to smithi04526579-7 2024-04-26T00:38:43.634 INFO:tasks.rados.rados.0.smithi045.stdout:3267: oid 7 updating version 0 to 2511 2024-04-26T00:38:43.634 INFO:tasks.rados.rados.0.smithi045.stdout:3267: oid 7 updating version 2511 to 2512 2024-04-26T00:38:43.634 INFO:tasks.rados.rados.0.smithi045.stdout:3267: oid 7 updating version 2512 to 2514 2024-04-26T00:38:43.634 INFO:tasks.rados.rados.0.smithi045.stdout:3267: oid 7 version 2514 is already newer than 2513 2024-04-26T00:38:43.635 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 2514 (ObjNum 1166 snap 360 seq_num 1166) dirty exists 2024-04-26T00:38:43.635 INFO:tasks.rados.rados.0.smithi045.stdout:3267: left oid 7 (ObjNum 1166 snap 360 seq_num 1166) 2024-04-26T00:38:43.635 INFO:tasks.rados.rados.0.smithi045.stdout:3267: done (1 left) 2024-04-26T00:38:43.635 INFO:tasks.rados.rados.0.smithi045.stdout:3270: write oid 4 current snap is 360 2024-04-26T00:38:43.635 INFO:tasks.rados.rados.0.smithi045.stdout:3270: seq_num 1167 ranges {413769=417592,1573545=697741,2847809=624439} 2024-04-26T00:38:43.643 INFO:tasks.rados.rados.0.smithi045.stdout:3270: writing smithi04526579-4 from 413769 to 831361 tid 1 2024-04-26T00:38:43.664 INFO:tasks.rados.rados.0.smithi045.stdout:3270: writing smithi04526579-4 from 1573545 to 2271286 tid 2 2024-04-26T00:38:43.678 INFO:tasks.rados.rados.0.smithi045.stdout:3270: writing smithi04526579-4 from 2847809 to 3472248 tid 3 2024-04-26T00:38:43.678 INFO:tasks.rados.rados.0.smithi045.stdout:3271: write (excl) oid 19 current snap is 360 2024-04-26T00:38:43.678 INFO:tasks.rados.rados.0.smithi045.stdout:3271: seq_num 1168 ranges {430576=492841,1632027=441399,2475232=435711,3365397=265179} 2024-04-26T00:38:43.688 INFO:tasks.rados.rados.0.smithi045.stdout:3271: writing smithi04526579-19 from 430576 to 923417 tid 1 2024-04-26T00:38:43.699 INFO:tasks.rados.rados.0.smithi045.stdout:3271: writing smithi04526579-19 from 1632027 to 2073426 tid 2 2024-04-26T00:38:43.706 INFO:tasks.rados.rados.0.smithi045.stdout:3271: writing smithi04526579-19 from 2475232 to 2910943 tid 3 2024-04-26T00:38:43.712 INFO:tasks.rados.rados.0.smithi045.stdout:3271: writing smithi04526579-19 from 3365397 to 3630576 tid 4 2024-04-26T00:38:43.713 INFO:tasks.rados.rados.0.smithi045.stdout:3272: read oid 23 snap -1 2024-04-26T00:38:43.713 INFO:tasks.rados.rados.0.smithi045.stdout:3272: expect deleted 2024-04-26T00:38:43.713 INFO:tasks.rados.rados.0.smithi045.stdout:3273: snap_create 2024-04-26T00:38:43.713 INFO:tasks.rados.rados.0.smithi045.stdout:3269: finishing rollback tid 0 to smithi04526579-30 2024-04-26T00:38:43.713 INFO:tasks.rados.rados.0.smithi045.stdout:3269: finishing rollback tid 1 to smithi04526579-30 2024-04-26T00:38:43.713 INFO:tasks.rados.rados.0.smithi045.stdout:3269: finishing rollback tid 2 to smithi04526579-30 2024-04-26T00:38:43.713 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 2004 (ObjNum 1155 snap 355 seq_num 1155) dirty exists 2024-04-26T00:38:43.713 INFO:tasks.rados.rados.0.smithi045.stdout:3270: finishing write tid 1 to smithi04526579-4 2024-04-26T00:38:43.713 INFO:tasks.rados.rados.0.smithi045.stdout:3270: finishing write tid 2 to smithi04526579-4 2024-04-26T00:38:43.713 INFO:tasks.rados.rados.0.smithi045.stdout:3270: finishing write tid 3 to smithi04526579-4 2024-04-26T00:38:43.713 INFO:tasks.rados.rados.0.smithi045.stdout:3270: finishing write tid 4 to smithi04526579-4 2024-04-26T00:38:43.713 INFO:tasks.rados.rados.0.smithi045.stdout:3270: finishing write tid 5 to smithi04526579-4 2024-04-26T00:38:43.713 INFO:tasks.rados.rados.0.smithi045.stdout:3270: oid 4 updating version 0 to 2065 2024-04-26T00:38:43.714 INFO:tasks.rados.rados.0.smithi045.stdout:3270: oid 4 version 2065 is already newer than 2064 2024-04-26T00:38:43.714 INFO:tasks.rados.rados.0.smithi045.stdout:3270: oid 4 updating version 2065 to 2067 2024-04-26T00:38:43.714 INFO:tasks.rados.rados.0.smithi045.stdout:3270: oid 4 version 2067 is already newer than 2066 2024-04-26T00:38:43.714 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 2067 (ObjNum 1167 snap 360 seq_num 1167) dirty exists 2024-04-26T00:38:43.714 INFO:tasks.rados.rados.0.smithi045.stdout:3270: left oid 4 (ObjNum 1167 snap 360 seq_num 1167) 2024-04-26T00:38:43.714 INFO:tasks.rados.rados.0.smithi045.stdout:3271: finishing write tid 1 to smithi04526579-19 2024-04-26T00:38:43.714 INFO:tasks.rados.rados.0.smithi045.stdout:3271: finishing write tid 2 to smithi04526579-19 2024-04-26T00:38:43.714 INFO:tasks.rados.rados.0.smithi045.stdout:3271: finishing write tid 3 to smithi04526579-19 2024-04-26T00:38:43.720 INFO:tasks.rados.rados.0.smithi045.stdout:3271: finishing write tid 4 to smithi04526579-19 2024-04-26T00:38:43.720 INFO:tasks.rados.rados.0.smithi045.stdout:3271: finishing write tid 5 to smithi04526579-19 2024-04-26T00:38:43.721 INFO:tasks.rados.rados.0.smithi045.stdout:3271: finishing write tid 6 to smithi04526579-19 2024-04-26T00:38:43.721 INFO:tasks.rados.rados.0.smithi045.stdout:3271: oid 19 updating version 0 to 2442 2024-04-26T00:38:43.721 INFO:tasks.rados.rados.0.smithi045.stdout:3271: oid 19 updating version 2442 to 2445 2024-04-26T00:38:43.721 INFO:tasks.rados.rados.0.smithi045.stdout:3271: oid 19 version 2445 is already newer than 2443 2024-04-26T00:38:43.721 INFO:tasks.rados.rados.0.smithi045.stdout:3271: oid 19 version 2445 is already newer than 2444 2024-04-26T00:38:43.721 INFO:tasks.rados.rados.0.smithi045.stdout:3271: oid 19 version 2445 is already newer than 2441 2024-04-26T00:38:43.721 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 2445 (ObjNum 1168 snap 360 seq_num 1168) dirty exists 2024-04-26T00:38:43.722 INFO:tasks.rados.rados.0.smithi045.stdout:3271: left oid 19 (ObjNum 1168 snap 360 seq_num 1168) 2024-04-26T00:38:44.444 INFO:tasks.rados.rados.0.smithi045.stdout:3269: done (4 left) 2024-04-26T00:38:44.444 INFO:tasks.rados.rados.0.smithi045.stdout:3270: done (3 left) 2024-04-26T00:38:44.444 INFO:tasks.rados.rados.0.smithi045.stdout:3271: done (2 left) 2024-04-26T00:38:44.444 INFO:tasks.rados.rados.0.smithi045.stdout:3272: done (1 left) 2024-04-26T00:38:44.444 INFO:tasks.rados.rados.0.smithi045.stdout:3273: done (0 left) 2024-04-26T00:38:44.445 INFO:tasks.rados.rados.0.smithi045.stdout:3274: snap_create 2024-04-26T00:38:45.453 INFO:tasks.rados.rados.0.smithi045.stdout:3274: done (0 left) 2024-04-26T00:38:45.453 INFO:tasks.rados.rados.0.smithi045.stdout:3275: write oid 31 current snap is 362 2024-04-26T00:38:45.453 INFO:tasks.rados.rados.0.smithi045.stdout:3275: seq_num 1169 ranges {447383=568090,1690509=701410,2902655=646983} 2024-04-26T00:38:45.465 INFO:tasks.rados.rados.0.smithi045.stdout:3275: writing smithi04526579-31 from 447383 to 1015473 tid 1 2024-04-26T00:38:45.481 INFO:tasks.rados.rados.0.smithi045.stdout:3275: writing smithi04526579-31 from 1690509 to 2391919 tid 2 2024-04-26T00:38:45.494 INFO:tasks.rados.rados.0.smithi045.stdout:3275: writing smithi04526579-31 from 2902655 to 3549638 tid 3 2024-04-26T00:38:45.494 INFO:tasks.rados.rados.0.smithi045.stdout:3276: snap_remove snap 335 2024-04-26T00:38:46.461 INFO:tasks.rados.rados.0.smithi045.stdout:3276: done (1 left) 2024-04-26T00:38:46.461 INFO:tasks.rados.rados.0.smithi045.stdout:3277: snap_remove snap 358 2024-04-26T00:38:47.470 INFO:tasks.rados.rados.0.smithi045.stdout:3277: done (1 left) 2024-04-26T00:38:47.470 INFO:tasks.rados.rados.0.smithi045.stdout:3278: rollback oid 35 current snap is 362 2024-04-26T00:38:47.470 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 35 to 344 2024-04-26T00:38:47.470 INFO:tasks.rados.rados.0.smithi045.stdout:3275: finishing write tid 1 to smithi04526579-31 2024-04-26T00:38:47.470 INFO:tasks.rados.rados.0.smithi045.stdout:3275: finishing write tid 2 to smithi04526579-31 2024-04-26T00:38:47.470 INFO:tasks.rados.rados.0.smithi045.stdout:3275: finishing write tid 3 to smithi04526579-31 2024-04-26T00:38:47.470 INFO:tasks.rados.rados.0.smithi045.stdout:3275: finishing write tid 4 to smithi04526579-31 2024-04-26T00:38:47.470 INFO:tasks.rados.rados.0.smithi045.stdout:3275: finishing write tid 5 to smithi04526579-31 2024-04-26T00:38:47.470 INFO:tasks.rados.rados.0.smithi045.stdout:3275: oid 31 updating version 0 to 2070 2024-04-26T00:38:47.471 INFO:tasks.rados.rados.0.smithi045.stdout:3275: oid 31 version 2070 is already newer than 2068 2024-04-26T00:38:47.471 INFO:tasks.rados.rados.0.smithi045.stdout:3275: oid 31 updating version 2070 to 2071 2024-04-26T00:38:47.471 INFO:tasks.rados.rados.0.smithi045.stdout:3275: oid 31 version 2071 is already newer than 2069 2024-04-26T00:38:47.471 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 2071 (ObjNum 1169 snap 362 seq_num 1169) dirty exists 2024-04-26T00:38:47.471 INFO:tasks.rados.rados.0.smithi045.stdout:3275: left oid 31 (ObjNum 1169 snap 362 seq_num 1169) 2024-04-26T00:38:47.474 INFO:tasks.rados.rados.0.smithi045.stdout:3275: done (1 left) 2024-04-26T00:38:47.474 INFO:tasks.rados.rados.0.smithi045.stdout:3279: snap_remove snap 356 2024-04-26T00:38:48.475 INFO:tasks.rados.rados.0.smithi045.stdout:3279: done (1 left) 2024-04-26T00:38:48.475 INFO:tasks.rados.rados.0.smithi045.stdout:3280: read oid 1 snap -1 2024-04-26T00:38:48.475 INFO:tasks.rados.rados.0.smithi045.stdout:3280: expect deleted 2024-04-26T00:38:48.475 INFO:tasks.rados.rados.0.smithi045.stdout:3281: write (excl) oid 9 current snap is 362 2024-04-26T00:38:48.475 INFO:tasks.rados.rados.0.smithi045.stdout:3281: seq_num 1170 ranges {464190=643339,1748991=445068,2930078=574608} 2024-04-26T00:38:48.497 INFO:tasks.rados.rados.0.smithi045.stdout:3281: writing smithi04526579-9 from 464190 to 1107529 tid 1 2024-04-26T00:38:48.516 INFO:tasks.rados.rados.0.smithi045.stdout:3281: writing smithi04526579-9 from 1748991 to 2194059 tid 2 2024-04-26T00:38:48.535 INFO:tasks.rados.rados.0.smithi045.stdout:3281: writing smithi04526579-9 from 2930078 to 3504686 tid 3 2024-04-26T00:38:48.535 INFO:tasks.rados.rados.0.smithi045.stdout:3282: read oid 47 snap 338 2024-04-26T00:38:48.535 INFO:tasks.rados.rados.0.smithi045.stdout:3282: expect (ObjNum 1097 snap 326 seq_num 1097) 2024-04-26T00:38:48.535 INFO:tasks.rados.rados.0.smithi045.stdout:3283: snap_remove snap 359 2024-04-26T00:38:49.446 INFO:tasks.rados.rados.0.smithi045.stdout:3283: done (4 left) 2024-04-26T00:38:49.447 INFO:tasks.rados.rados.0.smithi045.stdout:3284: read oid 30 snap 345 2024-04-26T00:38:49.447 INFO:tasks.rados.rados.0.smithi045.stdout:3284: expect (ObjNum 1115 snap 332 seq_num 1115) 2024-04-26T00:38:49.447 INFO:tasks.rados.rados.0.smithi045.stdout:3285: delete oid 26 current snap is 362 2024-04-26T00:38:49.447 INFO:tasks.rados.rados.0.smithi045.stdout:3278: finishing rollback tid 0 to smithi04526579-35 2024-04-26T00:38:49.447 INFO:tasks.rados.rados.0.smithi045.stdout:3278: finishing rollback tid 1 to smithi04526579-35 2024-04-26T00:38:49.447 INFO:tasks.rados.rados.0.smithi045.stdout:3278: finishing rollback tid 2 to smithi04526579-35 2024-04-26T00:38:49.448 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 2448 (ObjNum 1111 snap 331 seq_num 1111) dirty exists 2024-04-26T00:38:49.448 INFO:tasks.rados.rados.0.smithi045.stdout:3281: finishing write tid 1 to smithi04526579-9 2024-04-26T00:38:49.448 INFO:tasks.rados.rados.0.smithi045.stdout:3281: finishing write tid 2 to smithi04526579-9 2024-04-26T00:38:49.448 INFO:tasks.rados.rados.0.smithi045.stdout:3281: finishing write tid 3 to smithi04526579-9 2024-04-26T00:38:49.448 INFO:tasks.rados.rados.0.smithi045.stdout:3281: finishing write tid 4 to smithi04526579-9 2024-04-26T00:38:49.448 INFO:tasks.rados.rados.0.smithi045.stdout:3281: finishing write tid 5 to smithi04526579-9 2024-04-26T00:38:49.448 INFO:tasks.rados.rados.0.smithi045.stdout:3281: oid 9 updating version 0 to 2075 2024-04-26T00:38:49.448 INFO:tasks.rados.rados.0.smithi045.stdout:3281: oid 9 version 2075 is already newer than 2072 2024-04-26T00:38:49.448 INFO:tasks.rados.rados.0.smithi045.stdout:3281: oid 9 version 2075 is already newer than 2074 2024-04-26T00:38:49.448 INFO:tasks.rados.rados.0.smithi045.stdout:3281: oid 9 version 2075 is already newer than 2073 2024-04-26T00:38:49.448 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 2075 (ObjNum 1170 snap 362 seq_num 1170) dirty exists 2024-04-26T00:38:49.448 INFO:tasks.rados.rados.0.smithi045.stdout:3281: left oid 9 (ObjNum 1170 snap 362 seq_num 1170) 2024-04-26T00:38:49.448 INFO:tasks.rados.rados.0.smithi045.stdout:3282: expect (ObjNum 1097 snap 326 seq_num 1097) 2024-04-26T00:38:49.450 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-04-26T00:38:49.450 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-04-26T00:38:49.450 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-26T00:38:49.450 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-26T00:38:49.453 INFO:tasks.ceph.mon.a:Started 2024-04-26T00:38:49.453 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:38:49.454 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:38:49.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.579+0000 7f68cf45d640 10 monclient: get_monmap_and_config 2024-04-26T00:38:49.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.579+0000 7f68cf45d640 10 monclient: build_initial_monmap 2024-04-26T00:38:49.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.579+0000 7f68cf45d640 10 monclient: monmap: 2024-04-26T00:38:49.586 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:38:49.586 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:49.586 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:38:49.583799+0000 2024-04-26T00:38:49.587 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:38:49.583799+0000 2024-04-26T00:38:49.587 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:38:49.587 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:49.587 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:38:49.587 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:38:49.588 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:38:49.588 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:38:49.588 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:38:49.588 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:49.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cf45d640 1 Processor -- start 2024-04-26T00:38:49.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cf45d640 1 -- start start 2024-04-26T00:38:49.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cf45d640 10 monclient: init 2024-04-26T00:38:49.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cf45d640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:49.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cf45d640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-04-26T00:38:49.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cf45d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68c8136da0 0x7f68c809bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:49.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cf45d640 10 monclient(hunting): picked mon.noname-b con 0x7f68c8136da0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:38:49.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cf45d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f68c809c410 0x7f68c8137180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:49.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cf45d640 10 monclient(hunting): picked mon.noname-c con 0x7f68c809c410 addr v2:172.21.15.45:3300/0 2024-04-26T00:38:49.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cf45d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f68c81376c0 addr v1:172.21.15.37:6789/0 2024-04-26T00:38:49.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cf45d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68c809b4f0 con 0x7f68c81376c0 2024-04-26T00:38:49.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cf45d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:49.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cf45d640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f68c809b210 con 0x7f68c809c410 2024-04-26T00:38:49.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cf45d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:49.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cf45d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f68c809b7d0 con 0x7f68c8136da0 2024-04-26T00:38:49.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cf45d640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:49.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cf45d640 10 monclient(hunting): authenticate will time out at 2110.889232s 2024-04-26T00:38:49.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cd1f9640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68c8136da0 0x7f68c809bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:49.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cd1f9640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68c8136da0 0x7f68c809bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:39644/0 (socket says 172.21.15.37:39644) 2024-04-26T00:38:49.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cd9fa640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f68c81376c0 0x7f68c8133b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:54134/0 (socket says 172.21.15.37:54134) 2024-04-26T00:38:49.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cd1f9640 1 -- 172.21.15.37:0/328465015 learned_addr learned my addr 172.21.15.37:0/328465015 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:38:49.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cd1f9640 10 monclient(hunting): get_auth_request con 0x7f68c8136da0 auth_method 0 2024-04-26T00:38:49.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cd1f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:49.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cd1f9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:49.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cd1f9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:49.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cd1f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:49.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cd1f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:49.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cc9f8640 1 --2- 172.21.15.37:0/328465015 >> v2:172.21.15.45:3300/0 conn(0x7f68c809c410 0x7f68c8137180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:49.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cd1f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:49.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cc9f8640 10 monclient(hunting): get_auth_request con 0x7f68c809c410 auth_method 0 2024-04-26T00:38:49.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cc9f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:49.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cc9f8640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:49.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cc9f8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:49.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cd1f9640 10 monclient(hunting): handle_auth_done global_id 204368 payload 274 2024-04-26T00:38:49.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cd1f9640 1 -- 172.21.15.37:0/328465015 >> v2:172.21.15.45:3300/0 conn(0x7f68c809c410 msgr2=0x7f68c8137180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:38:49.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cd1f9640 1 --2- 172.21.15.37:0/328465015 >> v2:172.21.15.45:3300/0 conn(0x7f68c809c410 0x7f68c8137180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:49.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cd1f9640 1 -- 172.21.15.37:0/328465015 >> v1:172.21.15.37:6789/0 conn(0x7f68c81376c0 legacy=0x7f68c8133b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:38:49.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cd1f9640 10 monclient: _finish_hunting 0 2024-04-26T00:38:49.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cd1f9640 1 monclient: found mon.noname-b 2024-04-26T00:38:49.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cd1f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:49.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cd1f9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:38:49.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.583+0000 7f68cd1f9640 1 -- 172.21.15.37:0/328465015 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68c8137b30 con 0x7f68c8136da0 2024-04-26T00:38:49.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.587+0000 7f68cd1f9640 1 -- 172.21.15.37:0/328465015 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68c8136da0 msgr2=0x7f68c809bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:49.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.587+0000 7f68cd1f9640 1 --2- 172.21.15.37:0/328465015 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68c8136da0 0x7f68c809bed0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f68c4009870 tx=0x7f68c401c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56fc8e61e1deae27 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:49.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.587+0000 7f68bffff640 1 -- 172.21.15.37:0/328465015 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c40b0070 con 0x7f68c8136da0 2024-04-26T00:38:49.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.587+0000 7f68bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:49.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.587+0000 7f68bffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:38:49.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.587+0000 7f68bffff640 10 monclient: dump: 2024-04-26T00:38:49.607 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:49.607 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:49.607 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:49.607 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:49.607 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:49.607 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:49.607 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:49.608 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:49.608 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:49.608 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:49.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.587+0000 7f68bffff640 10 monclient: _finish_auth 0 2024-04-26T00:38:49.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.587+0000 7f68bffff640 10 monclient: _check_auth_tickets 2024-04-26T00:38:49.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.587+0000 7f68bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:49.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.587+0000 7f68bffff640 1 -- 172.21.15.37:0/328465015 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68c40ab600 con 0x7f68c8136da0 2024-04-26T00:38:49.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.587+0000 7f68bffff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:38:49.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.587+0000 7f68bffff640 1 -- 172.21.15.37:0/328465015 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c40ab9d0 con 0x7f68c8136da0 2024-04-26T00:38:49.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.587+0000 7f68cf45d640 5 monclient: authenticate success, global_id 204368 2024-04-26T00:38:49.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.587+0000 7f68cf45d640 10 monclient: get_monmap_and_config success 2024-04-26T00:38:49.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.587+0000 7f68cf45d640 4 set_mon_vals no callback set 2024-04-26T00:38:49.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.587+0000 7f68bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:49.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.587+0000 7f68bffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:38:49.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.587+0000 7f68bffff640 10 monclient: dump: 2024-04-26T00:38:49.610 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:49.610 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:49.610 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:49.610 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:49.610 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:49.610 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:49.611 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:49.611 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:49.611 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:49.611 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:49.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.587+0000 7f68cf45d640 10 monclient: shutdown 2024-04-26T00:38:49.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.587+0000 7f68cf45d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:38:49.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.587+0000 7f68cf45d640 1 -- 172.21.15.37:0/328465015 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68c8136da0 msgr2=0x7f68c809bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:49.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.587+0000 7f68cf45d640 1 --2- 172.21.15.37:0/328465015 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68c8136da0 0x7f68c809bed0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f68c4009870 tx=0x7f68c401c3e0 comp rx=0 tx=0).stop 2024-04-26T00:38:49.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.587+0000 7f68cf45d640 1 -- 172.21.15.37:0/328465015 shutdown_connections 2024-04-26T00:38:49.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.587+0000 7f68cf45d640 1 --2- 172.21.15.37:0/328465015 >> v2:172.21.15.45:3300/0 conn(0x7f68c809c410 0x7f68c8137180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:49.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.587+0000 7f68cf45d640 1 --2- 172.21.15.37:0/328465015 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68c8136da0 0x7f68c809bed0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:49.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.587+0000 7f68cf45d640 1 -- 172.21.15.37:0/328465015 >> 172.21.15.37:0/328465015 conn(0x7f68c80a5f00 msgr2=0x7f68c80a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:38:49.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.587+0000 7f68cf45d640 1 -- 172.21.15.37:0/328465015 shutdown_connections 2024-04-26T00:38:49.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.587+0000 7f68cf45d640 1 -- 172.21.15.37:0/328465015 wait complete. 2024-04-26T00:38:49.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.587+0000 7f68cf45d640 10 monclient: build_initial_monmap 2024-04-26T00:38:49.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.587+0000 7f68cf45d640 10 monclient: monmap: 2024-04-26T00:38:49.614 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:38:49.614 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:38:49.614 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:38:49.614 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:38:49.615 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:38:49.615 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:38:49.615 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:38:49.615 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:38:49.615 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:38:49.616 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:49.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.587+0000 7f68cf45d640 1 Processor -- start 2024-04-26T00:38:49.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.587+0000 7f68cf45d640 1 -- start start 2024-04-26T00:38:49.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.587+0000 7f68cf45d640 10 monclient: init 2024-04-26T00:38:49.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cf45d640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:49.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cf45d640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:38:49.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cf45d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68c809c410 0x7f68c81cf640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:49.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cf45d640 10 monclient(hunting): picked mon.noname-c con 0x7f68c809c410 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:38:49.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cf45d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68c8136da0 0x7f68c813de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:49.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cf45d640 10 monclient(hunting): picked mon.noname-b con 0x7f68c8136da0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:38:49.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cf45d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f68c81376c0 0x7f68c813e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:49.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cf45d640 10 monclient(hunting): picked mon.noname-a con 0x7f68c81376c0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:38:49.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cf45d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:49.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cf45d640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f68c8141ea0 con 0x7f68c81376c0 2024-04-26T00:38:49.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cf45d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:49.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cf45d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f68c8141d20 con 0x7f68c8136da0 2024-04-26T00:38:49.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cf45d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:49.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cf45d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f68c8142020 con 0x7f68c809c410 2024-04-26T00:38:49.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cf45d640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:49.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cf45d640 10 monclient(hunting): authenticate will time out at 2110.896191s 2024-04-26T00:38:49.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cd1f9640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68c809c410 0x7f68c81cf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:49.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cd9fa640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f68c81376c0 0x7f68c813e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:49.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cd1f9640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68c809c410 0x7f68c81cf640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:39658/0 (socket says 172.21.15.37:39658) 2024-04-26T00:38:49.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cd9fa640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f68c81376c0 0x7f68c813e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:33718/0 (socket says 172.21.15.37:33718) 2024-04-26T00:38:49.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cd1f9640 1 -- 172.21.15.37:0/4198844843 learned_addr learned my addr 172.21.15.37:0/4198844843 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:38:49.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cd1f9640 10 monclient(hunting): get_auth_request con 0x7f68c809c410 auth_method 0 2024-04-26T00:38:49.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cd1f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:49.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cd1f9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:49.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cd1f9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:49.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cd9fa640 10 monclient(hunting): get_auth_request con 0x7f68c81376c0 auth_method 0 2024-04-26T00:38:49.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cd9fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:49.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cd9fa640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:49.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cd9fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:49.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cd1f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:49.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cd1f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:49.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cc9f8640 1 --2- 172.21.15.37:0/4198844843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68c8136da0 0x7f68c813de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:49.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cd1f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:49.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cd9fa640 1 -- 172.21.15.37:0/4198844843 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f68c81376c0 msgr2=0x7f68c813e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:38:49.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cd9fa640 1 -- 172.21.15.37:0/4198844843 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f68c81376c0 msgr2=0x7f68c813e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:38:49.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cd9fa640 1 --2- 172.21.15.37:0/4198844843 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f68c81376c0 0x7f68c813e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:38:49.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cd9fa640 1 --2- 172.21.15.37:0/4198844843 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f68c81376c0 0x7f68c813e3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:38:49.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cc9f8640 10 monclient(hunting): get_auth_request con 0x7f68c8136da0 auth_method 0 2024-04-26T00:38:49.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cc9f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:49.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cc9f8640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:49.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cc9f8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:49.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cd1f9640 10 monclient(hunting): handle_auth_done global_id 204371 payload 824 2024-04-26T00:38:49.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cd1f9640 1 -- 172.21.15.37:0/4198844843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68c8136da0 msgr2=0x7f68c813de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:49.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cd1f9640 1 --2- 172.21.15.37:0/4198844843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68c8136da0 0x7f68c813de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:49.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cd1f9640 1 -- 172.21.15.37:0/4198844843 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f68c81376c0 msgr2=0x7f68c813e3c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:38:49.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cd1f9640 1 --2- 172.21.15.37:0/4198844843 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f68c81376c0 0x7f68c813e3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:49.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cd1f9640 10 monclient: _finish_hunting 0 2024-04-26T00:38:49.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cd1f9640 1 monclient: found mon.noname-c 2024-04-26T00:38:49.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cd1f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:49.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cd1f9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:38:49.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cd1f9640 1 -- 172.21.15.37:0/4198844843 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68c8138630 con 0x7f68c809c410 2024-04-26T00:38:49.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cd1f9640 1 -- 172.21.15.37:0/4198844843 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68c809c410 msgr2=0x7f68c81cf640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:49.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cd1f9640 1 --2- 172.21.15.37:0/4198844843 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68c809c410 0x7f68c81cf640 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f68c40ae040 tx=0x7f68c4019180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:49.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68bdffb640 1 -- 172.21.15.37:0/4198844843 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c40b0040 con 0x7f68c809c410 2024-04-26T00:38:49.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68bdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:49.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68bdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:38:49.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68bdffb640 10 monclient: dump: 2024-04-26T00:38:49.628 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:49.628 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:49.628 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:49.628 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:49.628 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:49.628 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:49.629 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:49.629 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:49.629 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:49.629 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:49.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68bdffb640 10 monclient: _finish_auth 0 2024-04-26T00:38:49.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68bdffb640 10 monclient: _check_auth_tickets 2024-04-26T00:38:49.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:49.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cf45d640 5 monclient: authenticate success, global_id 204371 2024-04-26T00:38:49.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68bdffb640 1 -- 172.21.15.37:0/4198844843 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68c40b9070 con 0x7f68c809c410 2024-04-26T00:38:49.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68bdffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:38:49.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68bdffb640 1 -- 172.21.15.37:0/4198844843 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c40b39f0 con 0x7f68c809c410 2024-04-26T00:38:49.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cf45d640 10 monclient: _renew_subs 2024-04-26T00:38:49.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cf45d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:49.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cf45d640 1 -- 172.21.15.37:0/4198844843 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68c8138910 con 0x7f68c809c410 2024-04-26T00:38:49.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68bf7fe640 4 set_mon_vals no callback set 2024-04-26T00:38:49.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68bdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:49.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68bdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:38:49.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68bdffb640 10 monclient: dump: 2024-04-26T00:38:49.631 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:49.631 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:49.632 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:49.632 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:49.632 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:49.632 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:49.632 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:49.633 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:49.633 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:49.633 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:49.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cf45d640 10 monclient: _renew_subs 2024-04-26T00:38:49.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cf45d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:49.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cf45d640 1 -- 172.21.15.37:0/4198844843 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f68c8137db0 con 0x7f68c809c410 2024-04-26T00:38:49.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cf45d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:49.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cf45d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:49.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cf45d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:49.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68cf45d640 1 -- 172.21.15.37:0/4198844843 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68c8066f40 con 0x7f68c809c410 2024-04-26T00:38:49.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.591+0000 7f68bdffb640 1 -- 172.21.15.37:0/4198844843 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f68c40b3bd0 con 0x7f68c809c410 2024-04-26T00:38:49.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.595+0000 7f68bdffb640 1 --2- 172.21.15.37:0/4198844843 >> v2:172.21.15.45:6812/2379918623 conn(0x7f68ac035e80 0x7f68ac038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:49.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.595+0000 7f68bdffb640 1 -- 172.21.15.37:0/4198844843 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(741..741 src has 705..741) ==== 4019+0+0 (secure 0 0 0) 0x7f68c40e29e0 con 0x7f68c809c410 2024-04-26T00:38:49.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.595+0000 7f68cc9f8640 1 --2- 172.21.15.37:0/4198844843 >> v2:172.21.15.45:6812/2379918623 conn(0x7f68ac035e80 0x7f68ac038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:49.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.595+0000 7f68cc9f8640 10 monclient: get_auth_request con 0x7f68ac035e80 auth_method 0 2024-04-26T00:38:49.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.595+0000 7f68cc9f8640 1 --2- 172.21.15.37:0/4198844843 >> v2:172.21.15.45:6812/2379918623 conn(0x7f68ac035e80 0x7f68ac038340 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f68c8131fb0 tx=0x7f68b8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:49.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.599+0000 7f68bdffb640 1 -- 172.21.15.37:0/4198844843 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f68c40160f0 con 0x7f68c809c410 2024-04-26T00:38:49.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.599+0000 7f68bdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:49.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.599+0000 7f68bdffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:38:49.663 INFO:tasks.rados.rados.0.smithi045.stdout:3284: expect (ObjNum 1115 snap 332 seq_num 1115) 2024-04-26T00:38:49.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.823+0000 7f68cf45d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:38:49.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.823+0000 7f68cf45d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:38:49.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.823+0000 7f68cf45d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:49.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.823+0000 7f68cf45d640 1 -- 172.21.15.37:0/4198844843 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f68c81325e0 con 0x7f68c809c410 2024-04-26T00:38:49.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.823+0000 7f68bdffb640 1 -- 172.21.15.37:0/4198844843 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7f68c40aa3a0 con 0x7f68c809c410 2024-04-26T00:38:49.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.823+0000 7f68bdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:38:49.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.823+0000 7f68bdffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:38:49.824 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":140,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:38:49.824 INFO:tasks.rados.rados.0.smithi045.stdout:3278: done (5 left) 2024-04-26T00:38:49.825 INFO:tasks.rados.rados.0.smithi045.stdout:3280: done (4 left) 2024-04-26T00:38:49.825 INFO:tasks.rados.rados.0.smithi045.stdout:3281: done (3 left) 2024-04-26T00:38:49.825 INFO:tasks.rados.rados.0.smithi045.stdout:3282: done (2 left) 2024-04-26T00:38:49.825 INFO:tasks.rados.rados.0.smithi045.stdout:3284: done (1 left) 2024-04-26T00:38:49.825 INFO:tasks.rados.rados.0.smithi045.stdout:3285: done (0 left) 2024-04-26T00:38:49.825 INFO:tasks.rados.rados.0.smithi045.stdout:3286: snap_remove snap 345 2024-04-26T00:38:49.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.827+0000 7f68cf45d640 1 -- 172.21.15.37:0/4198844843 >> v2:172.21.15.45:6812/2379918623 conn(0x7f68ac035e80 msgr2=0x7f68ac038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:49.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.827+0000 7f68cf45d640 1 --2- 172.21.15.37:0/4198844843 >> v2:172.21.15.45:6812/2379918623 conn(0x7f68ac035e80 0x7f68ac038340 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f68c8131fb0 tx=0x7f68b8008040 comp rx=0 tx=0).stop 2024-04-26T00:38:49.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.827+0000 7f68cf45d640 10 monclient: shutdown 2024-04-26T00:38:49.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.827+0000 7f68cf45d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:38:49.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.827+0000 7f68cf45d640 1 -- 172.21.15.37:0/4198844843 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68c809c410 msgr2=0x7f68c81cf640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:49.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.827+0000 7f68cf45d640 1 --2- 172.21.15.37:0/4198844843 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68c809c410 0x7f68c81cf640 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f68c40ae040 tx=0x7f68c4019180 comp rx=0 tx=0).stop 2024-04-26T00:38:49.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.827+0000 7f68cf45d640 1 -- 172.21.15.37:0/4198844843 shutdown_connections 2024-04-26T00:38:49.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.827+0000 7f68cf45d640 1 --2- 172.21.15.37:0/4198844843 >> v2:172.21.15.45:6812/2379918623 conn(0x7f68ac035e80 0x7f68ac038340 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:49.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.827+0000 7f68cf45d640 1 --2- 172.21.15.37:0/4198844843 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f68c81376c0 0x7f68c813e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:49.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.827+0000 7f68cf45d640 1 --2- 172.21.15.37:0/4198844843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68c8136da0 0x7f68c813de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:49.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.827+0000 7f68cf45d640 1 --2- 172.21.15.37:0/4198844843 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68c809c410 0x7f68c81cf640 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:49.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.827+0000 7f68cf45d640 1 -- 172.21.15.37:0/4198844843 >> 172.21.15.37:0/4198844843 conn(0x7f68c80a5f00 msgr2=0x7f68c80a2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:38:49.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.827+0000 7f68cf45d640 1 -- 172.21.15.37:0/4198844843 shutdown_connections 2024-04-26T00:38:49.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:49.827+0000 7f68cf45d640 1 -- 172.21.15.37:0/4198844843 wait complete. 2024-04-26T00:38:50.492 INFO:tasks.rados.rados.0.smithi045.stdout:3286: done (0 left) 2024-04-26T00:38:50.492 INFO:tasks.rados.rados.0.smithi045.stdout:3287: read oid 28 snap -1 2024-04-26T00:38:50.492 INFO:tasks.rados.rados.0.smithi045.stdout:3287: expect deleted 2024-04-26T00:38:50.492 INFO:tasks.rados.rados.0.smithi045.stdout:3288: write (excl) oid 41 current snap is 362 2024-04-26T00:38:50.492 INFO:tasks.rados.rados.0.smithi045.stdout:3288: seq_num 1172 ranges {497804=510190,1465955=448737,2584924=597152} 2024-04-26T00:38:50.508 INFO:tasks.rados.rados.0.smithi045.stdout:3288: writing smithi04526579-41 from 497804 to 1007994 tid 1 2024-04-26T00:38:50.524 INFO:tasks.rados.rados.0.smithi045.stdout:3288: writing smithi04526579-41 from 1465955 to 1914692 tid 2 2024-04-26T00:38:50.541 INFO:tasks.rados.rados.0.smithi045.stdout:3288: writing smithi04526579-41 from 2584924 to 3182076 tid 3 2024-04-26T00:38:50.541 INFO:tasks.rados.rados.0.smithi045.stdout:3289: write (excl) oid 2 current snap is 362 2024-04-26T00:38:50.542 INFO:tasks.rados.rados.0.smithi045.stdout:3289: seq_num 1173 ranges {514611=585439,1524437=592395,2895994=408424} 2024-04-26T00:38:50.553 INFO:tasks.rados.rados.0.smithi045.stdout:3289: writing smithi04526579-2 from 514611 to 1100050 tid 1 2024-04-26T00:38:50.563 INFO:tasks.rados.rados.0.smithi045.stdout:3289: writing smithi04526579-2 from 1524437 to 2116832 tid 2 2024-04-26T00:38:50.573 INFO:tasks.rados.rados.0.smithi045.stdout:3289: writing smithi04526579-2 from 2895994 to 3304418 tid 3 2024-04-26T00:38:50.573 INFO:tasks.rados.rados.0.smithi045.stdout:3290: snap_create 2024-04-26T00:38:50.573 INFO:tasks.rados.rados.0.smithi045.stdout:3288: finishing write tid 1 to smithi04526579-41 2024-04-26T00:38:50.573 INFO:tasks.rados.rados.0.smithi045.stdout:3288: finishing write tid 2 to smithi04526579-41 2024-04-26T00:38:50.573 INFO:tasks.rados.rados.0.smithi045.stdout:3288: finishing write tid 3 to smithi04526579-41 2024-04-26T00:38:50.573 INFO:tasks.rados.rados.0.smithi045.stdout:3288: finishing write tid 4 to smithi04526579-41 2024-04-26T00:38:50.573 INFO:tasks.rados.rados.0.smithi045.stdout:3288: finishing write tid 5 to smithi04526579-41 2024-04-26T00:38:50.573 INFO:tasks.rados.rados.0.smithi045.stdout:3288: oid 41 updating version 0 to 2458 2024-04-26T00:38:50.574 INFO:tasks.rados.rados.0.smithi045.stdout:3288: oid 41 version 2458 is already newer than 2457 2024-04-26T00:38:50.574 INFO:tasks.rados.rados.0.smithi045.stdout:3288: oid 41 version 2458 is already newer than 2456 2024-04-26T00:38:50.574 INFO:tasks.rados.rados.0.smithi045.stdout:3288: oid 41 version 2458 is already newer than 2455 2024-04-26T00:38:50.574 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 2458 (ObjNum 1172 snap 362 seq_num 1172) dirty exists 2024-04-26T00:38:50.574 INFO:tasks.rados.rados.0.smithi045.stdout:3288: left oid 41 (ObjNum 1172 snap 362 seq_num 1172) 2024-04-26T00:38:50.574 INFO:tasks.rados.rados.0.smithi045.stdout:3289: finishing write tid 1 to smithi04526579-2 2024-04-26T00:38:50.574 INFO:tasks.rados.rados.0.smithi045.stdout:3289: finishing write tid 2 to smithi04526579-2 2024-04-26T00:38:50.581 INFO:tasks.rados.rados.0.smithi045.stdout:3289: finishing write tid 3 to smithi04526579-2 2024-04-26T00:38:50.581 INFO:tasks.rados.rados.0.smithi045.stdout:3289: finishing write tid 4 to smithi04526579-2 2024-04-26T00:38:50.582 INFO:tasks.rados.rados.0.smithi045.stdout:3289: finishing write tid 5 to smithi04526579-2 2024-04-26T00:38:50.582 INFO:tasks.rados.rados.0.smithi045.stdout:3289: oid 2 updating version 0 to 2078 2024-04-26T00:38:50.582 INFO:tasks.rados.rados.0.smithi045.stdout:3289: oid 2 version 2078 is already newer than 2076 2024-04-26T00:38:50.582 INFO:tasks.rados.rados.0.smithi045.stdout:3289: oid 2 version 2078 is already newer than 2077 2024-04-26T00:38:50.582 INFO:tasks.rados.rados.0.smithi045.stdout:3289: oid 2 updating version 2078 to 2079 2024-04-26T00:38:50.582 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 2079 (ObjNum 1173 snap 362 seq_num 1173) dirty exists 2024-04-26T00:38:50.582 INFO:tasks.rados.rados.0.smithi045.stdout:3289: left oid 2 (ObjNum 1173 snap 362 seq_num 1173) 2024-04-26T00:38:51.833 INFO:tasks.rados.rados.0.smithi045.stdout:3287: done (3 left) 2024-04-26T00:38:51.833 INFO:tasks.rados.rados.0.smithi045.stdout:3288: done (2 left) 2024-04-26T00:38:51.833 INFO:tasks.rados.rados.0.smithi045.stdout:3289: done (1 left) 2024-04-26T00:38:51.833 INFO:tasks.rados.rados.0.smithi045.stdout:3290: done (0 left) 2024-04-26T00:38:51.833 INFO:tasks.rados.rados.0.smithi045.stdout:3291: read oid 38 snap -1 2024-04-26T00:38:51.833 INFO:tasks.rados.rados.0.smithi045.stdout:3291: expect (ObjNum 1163 snap 359 seq_num 1163) 2024-04-26T00:38:51.834 INFO:tasks.rados.rados.0.smithi045.stdout:3292: delete oid 28 current snap is 363 2024-04-26T00:38:51.840 INFO:tasks.rados.rados.0.smithi045.stdout:3292: done (1 left) 2024-04-26T00:38:51.840 INFO:tasks.rados.rados.0.smithi045.stdout:3293: snap_create 2024-04-26T00:38:51.857 INFO:tasks.rados.rados.0.smithi045.stdout:3291: expect (ObjNum 1163 snap 359 seq_num 1163) 2024-04-26T00:38:52.835 INFO:tasks.rados.rados.0.smithi045.stdout:3291: done (1 left) 2024-04-26T00:38:52.835 INFO:tasks.rados.rados.0.smithi045.stdout:3293: done (0 left) 2024-04-26T00:38:52.835 INFO:tasks.rados.rados.0.smithi045.stdout:3294: delete oid 27 current snap is 364 2024-04-26T00:38:52.844 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:38:52.850 INFO:tasks.rados.rados.0.smithi045.stdout:3294: done (0 left) 2024-04-26T00:38:52.850 INFO:tasks.rados.rados.0.smithi045.stdout:3295: read oid 41 snap -1 2024-04-26T00:38:52.850 INFO:tasks.rados.rados.0.smithi045.stdout:3295: expect (ObjNum 1172 snap 362 seq_num 1172) 2024-04-26T00:38:52.850 INFO:tasks.rados.rados.0.smithi045.stdout:3296: delete oid 14 current snap is 364 2024-04-26T00:38:52.866 INFO:tasks.rados.rados.0.smithi045.stdout:3296: done (1 left) 2024-04-26T00:38:52.866 INFO:tasks.rados.rados.0.smithi045.stdout:3297: read oid 7 snap -1 2024-04-26T00:38:52.866 INFO:tasks.rados.rados.0.smithi045.stdout:3297: expect (ObjNum 1166 snap 360 seq_num 1166) 2024-04-26T00:38:52.866 INFO:tasks.rados.rados.0.smithi045.stdout:3298: read oid 26 snap 338 2024-04-26T00:38:52.866 INFO:tasks.rados.rados.0.smithi045.stdout:3298: expect (ObjNum 1051 snap 317 seq_num 1051) 2024-04-26T00:38:52.867 INFO:tasks.rados.rados.0.smithi045.stdout:3299: read oid 29 snap -1 2024-04-26T00:38:52.867 INFO:tasks.rados.rados.0.smithi045.stdout:3299: expect (ObjNum 987 snap 302 seq_num 987) 2024-04-26T00:38:52.867 INFO:tasks.rados.rados.0.smithi045.stdout:3300: snap_create 2024-04-26T00:38:52.867 INFO:tasks.rados.rados.0.smithi045.stdout:3295: expect (ObjNum 1172 snap 362 seq_num 1172) 2024-04-26T00:38:52.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.970+0000 7f79091b3640 10 monclient: get_monmap_and_config 2024-04-26T00:38:52.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.970+0000 7f79091b3640 10 monclient: build_initial_monmap 2024-04-26T00:38:52.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.970+0000 7f79091b3640 10 monclient: monmap: 2024-04-26T00:38:52.971 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:38:52.971 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:52.972 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:38:52.975117+0000 2024-04-26T00:38:52.972 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:38:52.975117+0000 2024-04-26T00:38:52.972 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:38:52.972 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:52.972 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:38:52.972 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:38:52.972 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:38:52.972 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:38:52.972 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:38:52.972 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:52.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f79091b3640 1 Processor -- start 2024-04-26T00:38:52.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f79091b3640 1 -- start start 2024-04-26T00:38:52.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f79091b3640 10 monclient: init 2024-04-26T00:38:52.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f79091b3640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:52.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f79091b3640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-04-26T00:38:52.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f79091b3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f79040a0260 0x7f79040a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:52.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f79091b3640 10 monclient(hunting): picked mon.noname-b con 0x7f79040a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:38:52.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f79091b3640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f7904134b30 0x7f7904134f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:52.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f79091b3640 10 monclient(hunting): picked mon.noname-a con 0x7f7904134b30 addr v2:172.21.15.37:3300/0 2024-04-26T00:38:52.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f79091b3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7904135490 addr v1:172.21.15.37:6789/0 2024-04-26T00:38:52.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f79091b3640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79040a1130 con 0x7f7904135490 2024-04-26T00:38:52.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f79091b3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:52.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f79091b3640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f790409ec10 con 0x7f7904134b30 2024-04-26T00:38:52.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f79091b3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:52.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f79091b3640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f790409f0f0 con 0x7f79040a0260 2024-04-26T00:38:52.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f79091b3640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:52.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f79091b3640 10 monclient(hunting): authenticate will time out at 2114.280948s 2024-04-26T00:38:52.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f7902d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f79040a0260 0x7f79040a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:52.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f790259c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f7904134b30 0x7f7904134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:52.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f7902d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f79040a0260 0x7f79040a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:40076/0 (socket says 172.21.15.37:40076) 2024-04-26T00:38:52.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f7902d9d640 1 -- 172.21.15.37:0/2698347706 learned_addr learned my addr 172.21.15.37:0/2698347706 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:38:52.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f7902d9d640 10 monclient(hunting): get_auth_request con 0x7f79040a0260 auth_method 0 2024-04-26T00:38:52.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f7902d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:52.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f7902d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:52.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f7902d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:52.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f790259c640 10 monclient(hunting): get_auth_request con 0x7f7904134b30 auth_method 0 2024-04-26T00:38:52.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f790259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:52.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f790259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:52.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f790259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:52.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f790259c640 1 -- 172.21.15.37:0/2698347706 >> v2:172.21.15.37:3300/0 conn(0x7f7904134b30 msgr2=0x7f7904134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:38:52.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f7902d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:52.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f7902d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:52.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f790259c640 1 -- 172.21.15.37:0/2698347706 >> v2:172.21.15.37:3300/0 conn(0x7f7904134b30 msgr2=0x7f7904134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:38:52.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f790259c640 1 --2- 172.21.15.37:0/2698347706 >> v2:172.21.15.37:3300/0 conn(0x7f7904134b30 0x7f7904134f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:38:52.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f7902d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:52.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f790259c640 1 --2- 172.21.15.37:0/2698347706 >> v2:172.21.15.37:3300/0 conn(0x7f7904134b30 0x7f7904134f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:38:52.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f7902d9d640 10 monclient(hunting): handle_auth_done global_id 204374 payload 274 2024-04-26T00:38:52.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f7902d9d640 1 -- 172.21.15.37:0/2698347706 >> v2:172.21.15.37:3300/0 conn(0x7f7904134b30 msgr2=0x7f7904134f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:38:52.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f7902d9d640 1 --2- 172.21.15.37:0/2698347706 >> v2:172.21.15.37:3300/0 conn(0x7f7904134b30 0x7f7904134f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:52.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f7902d9d640 1 -- 172.21.15.37:0/2698347706 >> v1:172.21.15.37:6789/0 conn(0x7f7904135490 legacy=0x7f790413be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:38:52.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f7902d9d640 10 monclient: _finish_hunting 0 2024-04-26T00:38:52.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f7902d9d640 1 monclient: found mon.noname-b 2024-04-26T00:38:52.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f7902d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:52.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f7902d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:38:52.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.974+0000 7f7902d9d640 1 -- 172.21.15.37:0/2698347706 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f790406cff0 con 0x7f79040a0260 2024-04-26T00:38:52.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.978+0000 7f7902d9d640 1 -- 172.21.15.37:0/2698347706 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f79040a0260 msgr2=0x7f79040a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:52.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.978+0000 7f7902d9d640 1 --2- 172.21.15.37:0/2698347706 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f79040a0260 0x7f79040a0640 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f78f4009870 tx=0x7f78f401c200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ac376b1a23e7718 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:52.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.978+0000 7f7901d9b640 1 -- 172.21.15.37:0/2698347706 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f40b0070 con 0x7f79040a0260 2024-04-26T00:38:52.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.978+0000 7f7901d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:52.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.978+0000 7f7901d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:38:52.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.978+0000 7f7901d9b640 10 monclient: dump: 2024-04-26T00:38:52.979 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:52.979 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:52.979 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:52.980 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:52.980 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:52.980 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:52.980 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:52.980 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:52.980 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:52.980 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:52.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.978+0000 7f7901d9b640 10 monclient: _finish_auth 0 2024-04-26T00:38:52.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.978+0000 7f7901d9b640 10 monclient: _check_auth_tickets 2024-04-26T00:38:52.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.978+0000 7f7901d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:52.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.978+0000 7f7901d9b640 1 -- 172.21.15.37:0/2698347706 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78f40ab600 con 0x7f79040a0260 2024-04-26T00:38:52.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.978+0000 7f7901d9b640 10 monclient: handle_config config(0 keys) 2024-04-26T00:38:52.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.978+0000 7f7901d9b640 1 -- 172.21.15.37:0/2698347706 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f40ab9d0 con 0x7f79040a0260 2024-04-26T00:38:52.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.978+0000 7f79091b3640 5 monclient: authenticate success, global_id 204374 2024-04-26T00:38:52.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.978+0000 7f79091b3640 10 monclient: get_monmap_and_config success 2024-04-26T00:38:52.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.978+0000 7f79091b3640 4 set_mon_vals no callback set 2024-04-26T00:38:52.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.978+0000 7f7901d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:52.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.978+0000 7f7901d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:38:52.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.978+0000 7f7901d9b640 10 monclient: dump: 2024-04-26T00:38:52.981 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:52.981 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:52.981 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:52.981 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:52.981 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:52.981 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:52.981 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:52.981 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:52.981 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:52.981 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:52.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.978+0000 7f79091b3640 10 monclient: shutdown 2024-04-26T00:38:52.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.978+0000 7f79091b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:38:52.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.978+0000 7f79091b3640 1 -- 172.21.15.37:0/2698347706 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f79040a0260 msgr2=0x7f79040a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:52.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.978+0000 7f79091b3640 1 --2- 172.21.15.37:0/2698347706 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f79040a0260 0x7f79040a0640 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f78f4009870 tx=0x7f78f401c200 comp rx=0 tx=0).stop 2024-04-26T00:38:52.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.978+0000 7f79091b3640 1 -- 172.21.15.37:0/2698347706 shutdown_connections 2024-04-26T00:38:52.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.978+0000 7f79091b3640 1 --2- 172.21.15.37:0/2698347706 >> v2:172.21.15.37:3300/0 conn(0x7f7904134b30 0x7f7904134f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:52.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.978+0000 7f79091b3640 1 --2- 172.21.15.37:0/2698347706 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f79040a0260 0x7f79040a0640 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:52.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.978+0000 7f79091b3640 1 -- 172.21.15.37:0/2698347706 >> 172.21.15.37:0/2698347706 conn(0x7f79040a1ad0 msgr2=0x7f79040a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:38:52.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.978+0000 7f79091b3640 1 -- 172.21.15.37:0/2698347706 shutdown_connections 2024-04-26T00:38:52.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.978+0000 7f79091b3640 1 -- 172.21.15.37:0/2698347706 wait complete. 2024-04-26T00:38:52.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.978+0000 7f79091b3640 10 monclient: build_initial_monmap 2024-04-26T00:38:52.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.978+0000 7f79091b3640 10 monclient: monmap: 2024-04-26T00:38:52.987 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:38:52.987 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:38:52.987 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:38:52.987 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:38:52.987 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:38:52.987 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:38:52.987 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:38:52.987 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:38:52.987 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:38:52.987 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:52.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f79091b3640 1 Processor -- start 2024-04-26T00:38:52.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f79091b3640 1 -- start start 2024-04-26T00:38:52.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f79091b3640 10 monclient: init 2024-04-26T00:38:52.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f79091b3640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:52.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f79091b3640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:38:52.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f79091b3640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f79040a0260 0x7f790413c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:52.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f79091b3640 10 monclient(hunting): picked mon.noname-a con 0x7f79040a0260 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:38:52.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f79091b3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7904134b30 0x7f790409c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:52.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f79091b3640 10 monclient(hunting): picked mon.noname-c con 0x7f7904134b30 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:38:52.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f79091b3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7904135490 0x7f790409ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:52.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f79091b3640 10 monclient(hunting): picked mon.noname-b con 0x7f7904135490 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:38:52.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f79091b3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:52.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f79091b3640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f790409f430 con 0x7f79040a0260 2024-04-26T00:38:52.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f79091b3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:52.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f79091b3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f790409f2b0 con 0x7f7904135490 2024-04-26T00:38:52.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f79091b3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:52.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f79091b3640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f790409fe90 con 0x7f7904134b30 2024-04-26T00:38:52.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f79091b3640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:52.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f79091b3640 10 monclient(hunting): authenticate will time out at 2114.288636s 2024-04-26T00:38:52.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f7902d9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f79040a0260 0x7f790413c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:52.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f7902d9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f79040a0260 0x7f790413c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:47868/0 (socket says 172.21.15.37:47868) 2024-04-26T00:38:52.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f7902d9d640 1 -- 172.21.15.37:0/1067560384 learned_addr learned my addr 172.21.15.37:0/1067560384 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:38:52.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f7902d9d640 10 monclient(hunting): get_auth_request con 0x7f79040a0260 auth_method 0 2024-04-26T00:38:52.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f7902d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:52.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f7902d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:52.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f7902d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:52.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f790259c640 1 --2- 172.21.15.37:0/1067560384 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7904134b30 0x7f790409c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:52.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f790259c640 10 monclient(hunting): get_auth_request con 0x7f7904134b30 auth_method 0 2024-04-26T00:38:52.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f790259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:52.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f790259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:52.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f790259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:52.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f7902d9d640 1 -- 172.21.15.37:0/1067560384 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f79040a0260 msgr2=0x7f790413c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:38:52.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f7902d9d640 1 -- 172.21.15.37:0/1067560384 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f79040a0260 msgr2=0x7f790413c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:38:52.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f7902d9d640 1 --2- 172.21.15.37:0/1067560384 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f79040a0260 0x7f790413c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:38:52.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f790259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:52.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f790259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:52.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f7902d9d640 1 --2- 172.21.15.37:0/1067560384 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f79040a0260 0x7f790413c180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:38:52.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f790259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:52.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f790359e640 1 --2- 172.21.15.37:0/1067560384 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7904135490 0x7f790409ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:52.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f790259c640 10 monclient(hunting): handle_auth_done global_id 204377 payload 824 2024-04-26T00:38:52.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f790259c640 1 -- 172.21.15.37:0/1067560384 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7904135490 msgr2=0x7f790409ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:52.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f790259c640 1 --2- 172.21.15.37:0/1067560384 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7904135490 0x7f790409ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:52.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f790259c640 1 -- 172.21.15.37:0/1067560384 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f79040a0260 msgr2=0x7f790413c180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:38:52.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f790259c640 1 --2- 172.21.15.37:0/1067560384 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f79040a0260 0x7f790413c180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:52.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f790259c640 10 monclient: _finish_hunting 0 2024-04-26T00:38:52.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f790259c640 1 monclient: found mon.noname-c 2024-04-26T00:38:52.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f790259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:52.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f790259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:38:52.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f790259c640 1 -- 172.21.15.37:0/1067560384 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f790413a230 con 0x7f7904134b30 2024-04-26T00:38:52.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f790359e640 10 monclient: get_auth_request con 0x7f7904135490 auth_method 0 2024-04-26T00:38:52.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f790359e640 1 --2- 172.21.15.37:0/1067560384 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7904135490 0x7f790409ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:38:52.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f790259c640 1 -- 172.21.15.37:0/1067560384 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7904134b30 msgr2=0x7f790409c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:52.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f790259c640 1 --2- 172.21.15.37:0/1067560384 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7904134b30 0x7f790409c500 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f78f800e7e0 tx=0x7f78f800eca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:52.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f78eb7fe640 1 -- 172.21.15.37:0/1067560384 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f8017030 con 0x7f7904134b30 2024-04-26T00:38:52.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f78eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:52.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f78eb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:38:52.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f78eb7fe640 10 monclient: dump: 2024-04-26T00:38:52.991 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:52.991 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:52.991 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:52.991 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:52.991 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:52.991 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:52.991 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:52.992 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:52.992 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:52.992 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:52.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f78eb7fe640 10 monclient: _finish_auth 0 2024-04-26T00:38:52.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f78eb7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:38:52.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f78eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:52.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f78eb7fe640 1 -- 172.21.15.37:0/1067560384 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78f8004020 con 0x7f7904134b30 2024-04-26T00:38:52.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f78eb7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:38:52.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f78eb7fe640 1 -- 172.21.15.37:0/1067560384 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f800f040 con 0x7f7904134b30 2024-04-26T00:38:52.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f78eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:52.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f78eb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:38:52.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f78eb7fe640 10 monclient: dump: 2024-04-26T00:38:52.992 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:52.992 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:52.992 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:52.992 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:52.992 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:52.993 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:52.993 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:52.993 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:52.993 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:52.993 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:52.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f79091b3640 5 monclient: authenticate success, global_id 204377 2024-04-26T00:38:52.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f79091b3640 10 monclient: _renew_subs 2024-04-26T00:38:52.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f79091b3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:52.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f79091b3640 1 -- 172.21.15.37:0/1067560384 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f790409e1a0 con 0x7f7904134b30 2024-04-26T00:38:52.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f790159a640 4 set_mon_vals no callback set 2024-04-26T00:38:52.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f79091b3640 10 monclient: _renew_subs 2024-04-26T00:38:52.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f79091b3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:52.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.982+0000 7f79091b3640 1 -- 172.21.15.37:0/1067560384 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f790413ba80 con 0x7f7904134b30 2024-04-26T00:38:52.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.986+0000 7f79091b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:52.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.986+0000 7f79091b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:52.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.986+0000 7f79091b3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:52.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.986+0000 7f79091b3640 1 -- 172.21.15.37:0/1067560384 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f790409f8d0 con 0x7f7904134b30 2024-04-26T00:38:52.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.986+0000 7f78eb7fe640 1 -- 172.21.15.37:0/1067560384 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f78f801a070 con 0x7f7904134b30 2024-04-26T00:38:52.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.986+0000 7f78eb7fe640 1 --2- 172.21.15.37:0/1067560384 >> v2:172.21.15.45:6812/2379918623 conn(0x7f78f0035f00 0x7f78f00383c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:52.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.986+0000 7f78eb7fe640 1 -- 172.21.15.37:0/1067560384 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(744..744 src has 705..744) ==== 3947+0+0 (secure 0 0 0) 0x7f78f8048430 con 0x7f7904134b30 2024-04-26T00:38:52.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.986+0000 7f7902d9d640 1 --2- 172.21.15.37:0/1067560384 >> v2:172.21.15.45:6812/2379918623 conn(0x7f78f0035f00 0x7f78f00383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:52.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.986+0000 7f7902d9d640 10 monclient: get_auth_request con 0x7f78f0035f00 auth_method 0 2024-04-26T00:38:52.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.990+0000 7f7902d9d640 1 --2- 172.21.15.37:0/1067560384 >> v2:172.21.15.45:6812/2379918623 conn(0x7f78f0035f00 0x7f78f00383c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f78f4000c10 tx=0x7f78f40a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:52.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.990+0000 7f78eb7fe640 1 -- 172.21.15.37:0/1067560384 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f78f801ae40 con 0x7f7904134b30 2024-04-26T00:38:52.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.990+0000 7f78eb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:52.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:52.990+0000 7f78eb7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:38:53.129 INFO:tasks.rados.rados.0.smithi045.stdout:3297: expect (ObjNum 1166 snap 360 seq_num 1166) 2024-04-26T00:38:53.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:53.210+0000 7f79091b3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:38:53.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:53.210+0000 7f79091b3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:38:53.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:53.210+0000 7f79091b3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:53.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:53.210+0000 7f79091b3640 1 -- 172.21.15.37:0/1067560384 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7904066f40 con 0x7f7904134b30 2024-04-26T00:38:53.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:53.210+0000 7f78eb7fe640 1 -- 172.21.15.37:0/1067560384 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7f78f801c810 con 0x7f7904134b30 2024-04-26T00:38:53.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:53.210+0000 7f78eb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:38:53.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:53.210+0000 7f78eb7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:38:53.211 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":140,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:38:53.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:53.214+0000 7f79091b3640 1 -- 172.21.15.37:0/1067560384 >> v2:172.21.15.45:6812/2379918623 conn(0x7f78f0035f00 msgr2=0x7f78f00383c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:53.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:53.214+0000 7f79091b3640 1 --2- 172.21.15.37:0/1067560384 >> v2:172.21.15.45:6812/2379918623 conn(0x7f78f0035f00 0x7f78f00383c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f78f4000c10 tx=0x7f78f40a6000 comp rx=0 tx=0).stop 2024-04-26T00:38:53.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:53.214+0000 7f79091b3640 10 monclient: shutdown 2024-04-26T00:38:53.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:53.214+0000 7f79091b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:38:53.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:53.214+0000 7f79091b3640 1 -- 172.21.15.37:0/1067560384 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7904134b30 msgr2=0x7f790409c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:53.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:53.214+0000 7f79091b3640 1 --2- 172.21.15.37:0/1067560384 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7904134b30 0x7f790409c500 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f78f800e7e0 tx=0x7f78f800eca0 comp rx=0 tx=0).stop 2024-04-26T00:38:53.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:53.214+0000 7f79091b3640 1 -- 172.21.15.37:0/1067560384 shutdown_connections 2024-04-26T00:38:53.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:53.214+0000 7f79091b3640 1 --2- 172.21.15.37:0/1067560384 >> v2:172.21.15.45:6812/2379918623 conn(0x7f78f0035f00 0x7f78f00383c0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:53.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:53.214+0000 7f79091b3640 1 --2- 172.21.15.37:0/1067560384 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7904135490 0x7f790409ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:53.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:53.214+0000 7f79091b3640 1 --2- 172.21.15.37:0/1067560384 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7904134b30 0x7f790409c500 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:53.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:53.214+0000 7f79091b3640 1 --2- 172.21.15.37:0/1067560384 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f79040a0260 0x7f790413c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:53.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:53.214+0000 7f79091b3640 1 -- 172.21.15.37:0/1067560384 >> 172.21.15.37:0/1067560384 conn(0x7f79040a1ad0 msgr2=0x7f79040a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:38:53.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:53.214+0000 7f79091b3640 1 -- 172.21.15.37:0/1067560384 shutdown_connections 2024-04-26T00:38:53.214 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:53.214+0000 7f79091b3640 1 -- 172.21.15.37:0/1067560384 wait complete. 2024-04-26T00:38:53.294 INFO:tasks.rados.rados.0.smithi045.stdout:3298: expect (ObjNum 1051 snap 317 seq_num 1051) 2024-04-26T00:38:53.507 INFO:tasks.rados.rados.0.smithi045.stdout:3299: expect (ObjNum 987 snap 302 seq_num 987) 2024-04-26T00:38:53.858 INFO:tasks.rados.rados.0.smithi045.stdout:3295: done (4 left) 2024-04-26T00:38:53.858 INFO:tasks.rados.rados.0.smithi045.stdout:3297: done (3 left) 2024-04-26T00:38:53.858 INFO:tasks.rados.rados.0.smithi045.stdout:3298: done (2 left) 2024-04-26T00:38:53.858 INFO:tasks.rados.rados.0.smithi045.stdout:3299: done (1 left) 2024-04-26T00:38:53.858 INFO:tasks.rados.rados.0.smithi045.stdout:3300: done (0 left) 2024-04-26T00:38:53.858 INFO:tasks.rados.rados.0.smithi045.stdout:3301: copy_from oid 41 from oid 48 current snap is 365 2024-04-26T00:38:53.858 INFO:tasks.rados.rados.0.smithi045.stdout:3302: write oid 49 current snap is 365 2024-04-26T00:38:53.859 INFO:tasks.rados.rados.0.smithi045.stdout:3302: seq_num 1177 ranges {581839=486435,1642012=599733,3005686=569865} 2024-04-26T00:38:53.876 INFO:tasks.rados.rados.0.smithi045.stdout:3302: writing smithi04526579-49 from 581839 to 1068274 tid 1 2024-04-26T00:38:53.897 INFO:tasks.rados.rados.0.smithi045.stdout:3302: writing smithi04526579-49 from 1642012 to 2241745 tid 2 2024-04-26T00:38:53.917 INFO:tasks.rados.rados.0.smithi045.stdout:3302: writing smithi04526579-49 from 3005686 to 3575551 tid 3 2024-04-26T00:38:53.917 INFO:tasks.rados.rados.0.smithi045.stdout:3303: snap_remove snap 338 2024-04-26T00:38:54.447 INFO:tasks.rados.rados.0.smithi045.stdout:3303: done (2 left) 2024-04-26T00:38:54.447 INFO:tasks.rados.rados.0.smithi045.stdout:3304: snap_create 2024-04-26T00:38:54.447 INFO:tasks.rados.rados.0.smithi045.stdout:3301: finishing copy_from racing read to smithi04526579-41 2024-04-26T00:38:54.447 INFO:tasks.rados.rados.0.smithi045.stdout:3302: finishing write tid 1 to smithi04526579-49 2024-04-26T00:38:54.447 INFO:tasks.rados.rados.0.smithi045.stdout:3301: finishing copy_from to smithi04526579-41 2024-04-26T00:38:54.447 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 2339 (ObjNum 1112 snap 331 seq_num 1112) dirty exists 2024-04-26T00:38:54.447 INFO:tasks.rados.rados.0.smithi045.stdout:3302: finishing write tid 2 to smithi04526579-49 2024-04-26T00:38:54.447 INFO:tasks.rados.rados.0.smithi045.stdout:3302: finishing write tid 3 to smithi04526579-49 2024-04-26T00:38:54.448 INFO:tasks.rados.rados.0.smithi045.stdout:3302: finishing write tid 4 to smithi04526579-49 2024-04-26T00:38:54.448 INFO:tasks.rados.rados.0.smithi045.stdout:3302: finishing write tid 5 to smithi04526579-49 2024-04-26T00:38:54.448 INFO:tasks.rados.rados.0.smithi045.stdout:3302: oid 49 updating version 0 to 2363 2024-04-26T00:38:54.448 INFO:tasks.rados.rados.0.smithi045.stdout:3302: oid 49 version 2363 is already newer than 2361 2024-04-26T00:38:54.448 INFO:tasks.rados.rados.0.smithi045.stdout:3302: oid 49 version 2363 is already newer than 2362 2024-04-26T00:38:54.448 INFO:tasks.rados.rados.0.smithi045.stdout:3302: oid 49 updating version 2363 to 2364 2024-04-26T00:38:54.448 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 2364 (ObjNum 1177 snap 365 seq_num 1177) dirty exists 2024-04-26T00:38:54.448 INFO:tasks.rados.rados.0.smithi045.stdout:3302: left oid 49 (ObjNum 1177 snap 365 seq_num 1177) 2024-04-26T00:38:55.458 INFO:tasks.rados.rados.0.smithi045.stdout:3301: done (2 left) 2024-04-26T00:38:55.459 INFO:tasks.rados.rados.0.smithi045.stdout:3302: done (1 left) 2024-04-26T00:38:55.459 INFO:tasks.rados.rados.0.smithi045.stdout:3304: done (0 left) 2024-04-26T00:38:55.459 INFO:tasks.rados.rados.0.smithi045.stdout:3305: copy_from oid 2 from oid 24 current snap is 366 2024-04-26T00:38:55.459 INFO:tasks.rados.rados.0.smithi045.stdout:3306: write oid 40 current snap is 366 2024-04-26T00:38:55.459 INFO:tasks.rados.rados.0.smithi045.stdout:3306: seq_num 1178 ranges {598646=561684,1700494=459744,2633109=781137} 2024-04-26T00:38:55.477 INFO:tasks.rados.rados.0.smithi045.stdout:3306: writing smithi04526579-40 from 598646 to 1160330 tid 1 2024-04-26T00:38:55.494 INFO:tasks.rados.rados.0.smithi045.stdout:3306: writing smithi04526579-40 from 1700494 to 2160238 tid 2 2024-04-26T00:38:55.513 INFO:tasks.rados.rados.0.smithi045.stdout:3306: writing smithi04526579-40 from 2633109 to 3414246 tid 3 2024-04-26T00:38:55.513 INFO:tasks.rados.rados.0.smithi045.stdout:3307: read oid 3 snap -1 2024-04-26T00:38:55.514 INFO:tasks.rados.rados.0.smithi045.stdout:3307: expect (ObjNum 1164 snap 359 seq_num 1164) 2024-04-26T00:38:55.514 INFO:tasks.rados.rados.0.smithi045.stdout:3308: snap_create 2024-04-26T00:38:55.514 INFO:tasks.rados.rados.0.smithi045.stdout:3305: finishing copy_from racing read to smithi04526579-2 2024-04-26T00:38:55.514 INFO:tasks.rados.rados.0.smithi045.stdout:3305: finishing copy_from to smithi04526579-2 2024-04-26T00:38:55.514 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 2476 (ObjNum 1140 snap 345 seq_num 1140) dirty exists 2024-04-26T00:38:55.514 INFO:tasks.rados.rados.0.smithi045.stdout:3306: finishing write tid 1 to smithi04526579-40 2024-04-26T00:38:55.514 INFO:tasks.rados.rados.0.smithi045.stdout:3306: finishing write tid 2 to smithi04526579-40 2024-04-26T00:38:55.521 INFO:tasks.rados.rados.0.smithi045.stdout:3306: finishing write tid 3 to smithi04526579-40 2024-04-26T00:38:55.521 INFO:tasks.rados.rados.0.smithi045.stdout:3306: finishing write tid 4 to smithi04526579-40 2024-04-26T00:38:55.522 INFO:tasks.rados.rados.0.smithi045.stdout:3306: finishing write tid 5 to smithi04526579-40 2024-04-26T00:38:55.522 INFO:tasks.rados.rados.0.smithi045.stdout:3306: oid 40 updating version 0 to 2340 2024-04-26T00:38:55.522 INFO:tasks.rados.rados.0.smithi045.stdout:3306: oid 40 updating version 2340 to 2343 2024-04-26T00:38:55.522 INFO:tasks.rados.rados.0.smithi045.stdout:3306: oid 40 version 2343 is already newer than 2342 2024-04-26T00:38:55.522 INFO:tasks.rados.rados.0.smithi045.stdout:3306: oid 40 version 2343 is already newer than 2341 2024-04-26T00:38:55.522 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 2343 (ObjNum 1178 snap 366 seq_num 1178) dirty exists 2024-04-26T00:38:55.522 INFO:tasks.rados.rados.0.smithi045.stdout:3306: left oid 40 (ObjNum 1178 snap 366 seq_num 1178) 2024-04-26T00:38:55.535 INFO:tasks.rados.rados.0.smithi045.stdout:3307: expect (ObjNum 1164 snap 359 seq_num 1164) 2024-04-26T00:38:56.229 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:38:56.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.362+0000 7fed4d52e640 10 monclient: get_monmap_and_config 2024-04-26T00:38:56.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.362+0000 7fed4d52e640 10 monclient: build_initial_monmap 2024-04-26T00:38:56.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.362+0000 7fed4d52e640 10 monclient: monmap: 2024-04-26T00:38:56.362 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:38:56.362 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:56.362 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:38:56.365771+0000 2024-04-26T00:38:56.362 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:38:56.365771+0000 2024-04-26T00:38:56.363 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:38:56.363 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:56.363 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:38:56.363 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:38:56.363 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:38:56.363 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:38:56.363 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:38:56.363 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:56.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.362+0000 7fed4d52e640 1 Processor -- start 2024-04-26T00:38:56.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.362+0000 7fed4d52e640 1 -- start start 2024-04-26T00:38:56.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.362+0000 7fed4d52e640 10 monclient: init 2024-04-26T00:38:56.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.362+0000 7fed4d52e640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:56.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.362+0000 7fed4d52e640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-04-26T00:38:56.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.362+0000 7fed4d52e640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fed48136eb0 0x7fed48137290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:56.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.362+0000 7fed4d52e640 10 monclient(hunting): picked mon.noname-a con 0x7fed48136eb0 addr v2:172.21.15.37:3300/0 2024-04-26T00:38:56.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.362+0000 7fed4d52e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fed481377d0 addr v1:172.21.15.37:6789/0 2024-04-26T00:38:56.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.362+0000 7fed4d52e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fed481342a0 0x7fed4813f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:56.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.362+0000 7fed4d52e640 10 monclient(hunting): picked mon.noname-c con 0x7fed481342a0 addr v2:172.21.15.45:3300/0 2024-04-26T00:38:56.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.362+0000 7fed4d52e640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed4809b8c0 con 0x7fed481377d0 2024-04-26T00:38:56.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.362+0000 7fed4d52e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:56.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.362+0000 7fed4d52e640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fed4809c7d0 con 0x7fed48136eb0 2024-04-26T00:38:56.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.362+0000 7fed4d52e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:56.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.362+0000 7fed4d52e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fed4809b300 con 0x7fed481342a0 2024-04-26T00:38:56.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.362+0000 7fed4d52e640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:56.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.362+0000 7fed4d52e640 10 monclient(hunting): authenticate will time out at 2117.671523s 2024-04-26T00:38:56.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.362+0000 7fed46ffd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fed48136eb0 0x7fed48137290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:56.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.362+0000 7fed46ffd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fed48136eb0 0x7fed48137290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:47870/0 (socket says 172.21.15.37:47870) 2024-04-26T00:38:56.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.362+0000 7fed467fc640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fed481377d0 0x7fed48133b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:43614/0 (socket says 172.21.15.37:43614) 2024-04-26T00:38:56.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.362+0000 7fed46ffd640 1 -- 172.21.15.37:0/3514416617 learned_addr learned my addr 172.21.15.37:0/3514416617 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:38:56.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.362+0000 7fed46ffd640 10 monclient(hunting): get_auth_request con 0x7fed48136eb0 auth_method 0 2024-04-26T00:38:56.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.362+0000 7fed46ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:56.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.362+0000 7fed46ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:56.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.362+0000 7fed46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:56.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.362+0000 7fed46ffd640 1 -- 172.21.15.37:0/3514416617 >> v2:172.21.15.37:3300/0 conn(0x7fed48136eb0 msgr2=0x7fed48137290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:38:56.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.362+0000 7fed46ffd640 1 -- 172.21.15.37:0/3514416617 >> v2:172.21.15.37:3300/0 conn(0x7fed48136eb0 msgr2=0x7fed48137290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:38:56.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.362+0000 7fed46ffd640 1 --2- 172.21.15.37:0/3514416617 >> v2:172.21.15.37:3300/0 conn(0x7fed48136eb0 0x7fed48137290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:38:56.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.362+0000 7fed46ffd640 1 --2- 172.21.15.37:0/3514416617 >> v2:172.21.15.37:3300/0 conn(0x7fed48136eb0 0x7fed48137290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:38:56.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.362+0000 7fed477fe640 1 --2- 172.21.15.37:0/3514416617 >> v2:172.21.15.45:3300/0 conn(0x7fed481342a0 0x7fed4813f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:56.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.366+0000 7fed477fe640 10 monclient(hunting): get_auth_request con 0x7fed481342a0 auth_method 0 2024-04-26T00:38:56.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.366+0000 7fed477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:56.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.366+0000 7fed477fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:56.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.366+0000 7fed477fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:56.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.366+0000 7fed477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:56.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.366+0000 7fed477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:56.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.366+0000 7fed477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:56.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.366+0000 7fed477fe640 10 monclient(hunting): handle_auth_done global_id 224161 payload 274 2024-04-26T00:38:56.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.366+0000 7fed477fe640 1 -- 172.21.15.37:0/3514416617 >> v2:172.21.15.37:3300/0 conn(0x7fed48136eb0 msgr2=0x7fed48137290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:38:56.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.366+0000 7fed477fe640 1 --2- 172.21.15.37:0/3514416617 >> v2:172.21.15.37:3300/0 conn(0x7fed48136eb0 0x7fed48137290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:56.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.366+0000 7fed477fe640 1 -- 172.21.15.37:0/3514416617 >> v1:172.21.15.37:6789/0 conn(0x7fed481377d0 legacy=0x7fed48133b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:38:56.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.366+0000 7fed477fe640 10 monclient: _finish_hunting 0 2024-04-26T00:38:56.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.366+0000 7fed477fe640 1 monclient: found mon.noname-c 2024-04-26T00:38:56.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.366+0000 7fed477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:56.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.366+0000 7fed477fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:38:56.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.366+0000 7fed477fe640 1 -- 172.21.15.37:0/3514416617 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed4809b5e0 con 0x7fed481342a0 2024-04-26T00:38:56.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.370+0000 7fed477fe640 1 -- 172.21.15.37:0/3514416617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed481342a0 msgr2=0x7fed4813f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:56.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.370+0000 7fed477fe640 1 --2- 172.21.15.37:0/3514416617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed481342a0 0x7fed4813f470 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fed4806b940 tx=0x7fed4001c120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9676264373bdffe4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:56.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.370+0000 7fed45ffb640 1 -- 172.21.15.37:0/3514416617 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed400b0070 con 0x7fed481342a0 2024-04-26T00:38:56.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.370+0000 7fed45ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:56.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.370+0000 7fed45ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:38:56.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.370+0000 7fed45ffb640 10 monclient: dump: 2024-04-26T00:38:56.371 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:56.371 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:56.371 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:56.371 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:56.372 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:56.372 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:56.372 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:56.372 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:56.372 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:56.372 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:56.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.370+0000 7fed45ffb640 10 monclient: _finish_auth 0 2024-04-26T00:38:56.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.370+0000 7fed45ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:38:56.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.370+0000 7fed45ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:56.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.370+0000 7fed45ffb640 1 -- 172.21.15.37:0/3514416617 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed400ab600 con 0x7fed481342a0 2024-04-26T00:38:56.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.370+0000 7fed4d52e640 5 monclient: authenticate success, global_id 224161 2024-04-26T00:38:56.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.370+0000 7fed4d52e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:38:56.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.370+0000 7fed45ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:38:56.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.370+0000 7fed45ffb640 1 -- 172.21.15.37:0/3514416617 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed400ab9d0 con 0x7fed481342a0 2024-04-26T00:38:56.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.370+0000 7fed45ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:56.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.370+0000 7fed45ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:38:56.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.370+0000 7fed45ffb640 10 monclient: dump: 2024-04-26T00:38:56.373 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:56.373 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:56.373 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:56.373 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:56.373 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:56.373 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:56.373 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:56.374 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:56.374 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:56.374 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:56.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.370+0000 7fed4d52e640 10 monclient: get_monmap_and_config success 2024-04-26T00:38:56.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.370+0000 7fed4d52e640 4 set_mon_vals no callback set 2024-04-26T00:38:56.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.370+0000 7fed4d52e640 10 monclient: shutdown 2024-04-26T00:38:56.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.370+0000 7fed4d52e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:38:56.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.370+0000 7fed4d52e640 1 -- 172.21.15.37:0/3514416617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed481342a0 msgr2=0x7fed4813f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:56.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.370+0000 7fed4d52e640 1 --2- 172.21.15.37:0/3514416617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed481342a0 0x7fed4813f470 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fed4806b940 tx=0x7fed4001c120 comp rx=0 tx=0).stop 2024-04-26T00:38:56.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.370+0000 7fed4d52e640 1 -- 172.21.15.37:0/3514416617 shutdown_connections 2024-04-26T00:38:56.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.370+0000 7fed4d52e640 1 --2- 172.21.15.37:0/3514416617 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed481342a0 0x7fed4813f470 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:56.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.370+0000 7fed4d52e640 1 --2- 172.21.15.37:0/3514416617 >> v2:172.21.15.37:3300/0 conn(0x7fed48136eb0 0x7fed48137290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:56.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.370+0000 7fed4d52e640 1 -- 172.21.15.37:0/3514416617 >> 172.21.15.37:0/3514416617 conn(0x7fed480a6300 msgr2=0x7fed480a6710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:38:56.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.370+0000 7fed4d52e640 1 -- 172.21.15.37:0/3514416617 shutdown_connections 2024-04-26T00:38:56.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.370+0000 7fed4d52e640 1 -- 172.21.15.37:0/3514416617 wait complete. 2024-04-26T00:38:56.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.370+0000 7fed4d52e640 10 monclient: build_initial_monmap 2024-04-26T00:38:56.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.370+0000 7fed4d52e640 10 monclient: monmap: 2024-04-26T00:38:56.379 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:38:56.379 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:38:56.379 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:38:56.379 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:38:56.379 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:38:56.379 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:38:56.379 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:38:56.379 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:38:56.379 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:38:56.379 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:56.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.370+0000 7fed4d52e640 1 Processor -- start 2024-04-26T00:38:56.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.370+0000 7fed4d52e640 1 -- start start 2024-04-26T00:38:56.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.370+0000 7fed4d52e640 10 monclient: init 2024-04-26T00:38:56.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed4d52e640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:56.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed4d52e640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:38:56.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed4d52e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed481342a0 0x7fed481cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:56.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed4d52e640 10 monclient(hunting): picked mon.noname-c con 0x7fed481342a0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:38:56.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed4d52e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed48136eb0 0x7fed4813df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:56.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed4d52e640 10 monclient(hunting): picked mon.noname-b con 0x7fed48136eb0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:38:56.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed4d52e640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed481377d0 0x7fed4813e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:56.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed4d52e640 10 monclient(hunting): picked mon.noname-a con 0x7fed481377d0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:38:56.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed4d52e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:56.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed4d52e640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fed4809a9e0 con 0x7fed481377d0 2024-04-26T00:38:56.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed4d52e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:56.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed4d52e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fed4809ab60 con 0x7fed48136eb0 2024-04-26T00:38:56.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed4d52e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:56.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed4d52e640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fed481421d0 con 0x7fed481342a0 2024-04-26T00:38:56.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed4d52e640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:56.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed4d52e640 10 monclient(hunting): authenticate will time out at 2117.680597s 2024-04-26T00:38:56.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed46ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed481342a0 0x7fed481cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:56.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed477fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed481377d0 0x7fed4813e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:56.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed477fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed481377d0 0x7fed4813e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:47872/0 (socket says 172.21.15.37:47872) 2024-04-26T00:38:56.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed477fe640 1 -- 172.21.15.37:0/4185548122 learned_addr learned my addr 172.21.15.37:0/4185548122 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:38:56.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed477fe640 10 monclient(hunting): get_auth_request con 0x7fed481377d0 auth_method 0 2024-04-26T00:38:56.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:56.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed477fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:56.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed477fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:56.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed46ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed481342a0 0x7fed481cf4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:40114/0 (socket says 172.21.15.37:40114) 2024-04-26T00:38:56.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed46ffd640 10 monclient(hunting): get_auth_request con 0x7fed481342a0 auth_method 0 2024-04-26T00:38:56.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed46ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:56.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed46ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:56.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:56.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed477fe640 1 -- 172.21.15.37:0/4185548122 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed481377d0 msgr2=0x7fed4813e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:38:56.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed477fe640 1 -- 172.21.15.37:0/4185548122 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed481377d0 msgr2=0x7fed4813e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:38:56.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed477fe640 1 --2- 172.21.15.37:0/4185548122 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed481377d0 0x7fed4813e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:38:56.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed46ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:56.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed46ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:56.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed477fe640 1 --2- 172.21.15.37:0/4185548122 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed481377d0 0x7fed4813e4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:38:56.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed46ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:56.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed467fc640 1 --2- 172.21.15.37:0/4185548122 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed48136eb0 0x7fed4813df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:56.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed467fc640 10 monclient(hunting): get_auth_request con 0x7fed48136eb0 auth_method 0 2024-04-26T00:38:56.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:56.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed467fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:56.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed467fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:56.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed46ffd640 10 monclient(hunting): handle_auth_done global_id 204380 payload 824 2024-04-26T00:38:56.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed46ffd640 1 -- 172.21.15.37:0/4185548122 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed48136eb0 msgr2=0x7fed4813df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:56.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed46ffd640 1 --2- 172.21.15.37:0/4185548122 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed48136eb0 0x7fed4813df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:56.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed46ffd640 1 -- 172.21.15.37:0/4185548122 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed481377d0 msgr2=0x7fed4813e4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:38:56.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed46ffd640 1 --2- 172.21.15.37:0/4185548122 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed481377d0 0x7fed4813e4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:56.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed46ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:38:56.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed46ffd640 1 monclient: found mon.noname-c 2024-04-26T00:38:56.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed46ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:56.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed46ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:38:56.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed46ffd640 1 -- 172.21.15.37:0/4185548122 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed48133a10 con 0x7fed481342a0 2024-04-26T00:38:56.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed46ffd640 1 -- 172.21.15.37:0/4185548122 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed481342a0 msgr2=0x7fed481cf4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:56.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed46ffd640 1 --2- 172.21.15.37:0/4185548122 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed481342a0 0x7fed481cf4d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fed3800e590 tx=0x7fed3800ea50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:56.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed3f7fe640 1 -- 172.21.15.37:0/4185548122 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed38017070 con 0x7fed481342a0 2024-04-26T00:38:56.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed3f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:56.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed3f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:38:56.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed3f7fe640 10 monclient: dump: 2024-04-26T00:38:56.383 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:56.384 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:56.384 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:56.384 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:56.384 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:56.384 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:56.384 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:56.384 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:56.384 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:56.384 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:56.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed3f7fe640 10 monclient: _finish_auth 0 2024-04-26T00:38:56.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed3f7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:38:56.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:56.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed4d52e640 5 monclient: authenticate success, global_id 204380 2024-04-26T00:38:56.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed3f7fe640 1 -- 172.21.15.37:0/4185548122 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed38004220 con 0x7fed481342a0 2024-04-26T00:38:56.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed3f7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:38:56.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed4d52e640 10 monclient: _renew_subs 2024-04-26T00:38:56.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed3f7fe640 1 -- 172.21.15.37:0/4185548122 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3800f040 con 0x7fed481342a0 2024-04-26T00:38:56.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed4d52e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:56.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed4d52e640 1 -- 172.21.15.37:0/4185548122 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed4813efa0 con 0x7fed481342a0 2024-04-26T00:38:56.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed457fa640 4 set_mon_vals no callback set 2024-04-26T00:38:56.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed3f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:56.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed3f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:38:56.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed3f7fe640 10 monclient: dump: 2024-04-26T00:38:56.385 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:56.385 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:56.385 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:56.385 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:56.385 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:56.386 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:56.386 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:56.386 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:56.386 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:56.386 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:56.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed4d52e640 10 monclient: _renew_subs 2024-04-26T00:38:56.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed4d52e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:56.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.374+0000 7fed4d52e640 1 -- 172.21.15.37:0/4185548122 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fed4813e9e0 con 0x7fed481342a0 2024-04-26T00:38:56.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.378+0000 7fed4d52e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:56.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.378+0000 7fed4d52e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:56.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.378+0000 7fed4d52e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:56.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.378+0000 7fed3f7fe640 1 -- 172.21.15.37:0/4185548122 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fed38013620 con 0x7fed481342a0 2024-04-26T00:38:56.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.378+0000 7fed4d52e640 1 -- 172.21.15.37:0/4185548122 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed48066f40 con 0x7fed481342a0 2024-04-26T00:38:56.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.378+0000 7fed3f7fe640 1 --2- 172.21.15.37:0/4185548122 >> v2:172.21.15.45:6812/2379918623 conn(0x7fed28035e80 0x7fed28038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:56.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.378+0000 7fed3f7fe640 1 -- 172.21.15.37:0/4185548122 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(747..747 src has 705..747) ==== 3915+0+0 (secure 0 0 0) 0x7fed38048540 con 0x7fed481342a0 2024-04-26T00:38:56.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.378+0000 7fed467fc640 1 --2- 172.21.15.37:0/4185548122 >> v2:172.21.15.45:6812/2379918623 conn(0x7fed28035e80 0x7fed28038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:56.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.378+0000 7fed467fc640 10 monclient: get_auth_request con 0x7fed28035e80 auth_method 0 2024-04-26T00:38:56.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.382+0000 7fed467fc640 1 --2- 172.21.15.37:0/4185548122 >> v2:172.21.15.45:6812/2379918623 conn(0x7fed28035e80 0x7fed28038340 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fed30007fd0 tx=0x7fed30009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:56.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.382+0000 7fed3f7fe640 1 -- 172.21.15.37:0/4185548122 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fed380472a0 con 0x7fed481342a0 2024-04-26T00:38:56.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.382+0000 7fed3f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:56.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.382+0000 7fed3f7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:38:56.469 INFO:tasks.rados.rados.0.smithi045.stdout:3305: done (3 left) 2024-04-26T00:38:56.469 INFO:tasks.rados.rados.0.smithi045.stdout:3306: done (2 left) 2024-04-26T00:38:56.470 INFO:tasks.rados.rados.0.smithi045.stdout:3307: done (1 left) 2024-04-26T00:38:56.470 INFO:tasks.rados.rados.0.smithi045.stdout:3308: done (0 left) 2024-04-26T00:38:56.470 INFO:tasks.rados.rados.0.smithi045.stdout:3309: rollback oid 47 current snap is 367 2024-04-26T00:38:56.470 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 47 to 362 2024-04-26T00:38:56.470 INFO:tasks.rados.rados.0.smithi045.stdout:3310: write oid 19 current snap is 367 2024-04-26T00:38:56.470 INFO:tasks.rados.rados.0.smithi045.stdout:3310: seq_num 1179 ranges {615453=753286,2158976=603402,3460532=354921} 2024-04-26T00:38:56.488 INFO:tasks.rados.rados.0.smithi045.stdout:3310: writing smithi04526579-19 from 615453 to 1368739 tid 1 2024-04-26T00:38:56.507 INFO:tasks.rados.rados.0.smithi045.stdout:3310: writing smithi04526579-19 from 2158976 to 2762378 tid 2 2024-04-26T00:38:56.520 INFO:tasks.rados.rados.0.smithi045.stdout:3310: writing smithi04526579-19 from 3460532 to 3815453 tid 3 2024-04-26T00:38:56.520 INFO:tasks.rados.rados.0.smithi045.stdout:3311: snap_create 2024-04-26T00:38:56.521 INFO:tasks.rados.rados.0.smithi045.stdout:3309: finishing rollback tid 0 to smithi04526579-47 2024-04-26T00:38:56.521 INFO:tasks.rados.rados.0.smithi045.stdout:3309: finishing rollback tid 1 to smithi04526579-47 2024-04-26T00:38:56.521 INFO:tasks.rados.rados.0.smithi045.stdout:3309: finishing rollback tid 2 to smithi04526579-47 2024-04-26T00:38:56.521 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 2367 (ObjNum 1132 snap 342 seq_num 1132) dirty exists 2024-04-26T00:38:56.521 INFO:tasks.rados.rados.0.smithi045.stdout:3310: finishing write tid 1 to smithi04526579-19 2024-04-26T00:38:56.521 INFO:tasks.rados.rados.0.smithi045.stdout:3310: finishing write tid 2 to smithi04526579-19 2024-04-26T00:38:56.527 INFO:tasks.rados.rados.0.smithi045.stdout:3310: finishing write tid 3 to smithi04526579-19 2024-04-26T00:38:56.529 INFO:tasks.rados.rados.0.smithi045.stdout:3310: finishing write tid 4 to smithi04526579-19 2024-04-26T00:38:56.529 INFO:tasks.rados.rados.0.smithi045.stdout:3310: finishing write tid 5 to smithi04526579-19 2024-04-26T00:38:56.529 INFO:tasks.rados.rados.0.smithi045.stdout:3310: oid 19 updating version 0 to 2452 2024-04-26T00:38:56.529 INFO:tasks.rados.rados.0.smithi045.stdout:3310: oid 19 version 2452 is already newer than 2451 2024-04-26T00:38:56.529 INFO:tasks.rados.rados.0.smithi045.stdout:3310: oid 19 version 2452 is already newer than 2450 2024-04-26T00:38:56.529 INFO:tasks.rados.rados.0.smithi045.stdout:3310: oid 19 updating version 2452 to 2453 2024-04-26T00:38:56.529 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 2453 (ObjNum 1179 snap 367 seq_num 1179) dirty exists 2024-04-26T00:38:56.529 INFO:tasks.rados.rados.0.smithi045.stdout:3310: left oid 19 (ObjNum 1179 snap 367 seq_num 1179) 2024-04-26T00:38:56.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.614+0000 7fed4d52e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:38:56.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.614+0000 7fed4d52e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:38:56.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.614+0000 7fed4d52e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:56.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.614+0000 7fed4d52e640 1 -- 172.21.15.37:0/4185548122 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fed48139960 con 0x7fed481342a0 2024-04-26T00:38:56.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.618+0000 7fed3f7fe640 1 -- 172.21.15.37:0/4185548122 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7fed3804a6d0 con 0x7fed481342a0 2024-04-26T00:38:56.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.618+0000 7fed3f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:38:56.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.618+0000 7fed3f7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:38:56.616 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":140,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":27,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:38:56.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.618+0000 7fed4d52e640 1 -- 172.21.15.37:0/4185548122 >> v2:172.21.15.45:6812/2379918623 conn(0x7fed28035e80 msgr2=0x7fed28038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:56.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.618+0000 7fed4d52e640 1 --2- 172.21.15.37:0/4185548122 >> v2:172.21.15.45:6812/2379918623 conn(0x7fed28035e80 0x7fed28038340 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fed30007fd0 tx=0x7fed30009040 comp rx=0 tx=0).stop 2024-04-26T00:38:56.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.618+0000 7fed4d52e640 10 monclient: shutdown 2024-04-26T00:38:56.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.618+0000 7fed4d52e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:38:56.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.618+0000 7fed4d52e640 1 -- 172.21.15.37:0/4185548122 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed481342a0 msgr2=0x7fed481cf4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:56.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.618+0000 7fed4d52e640 1 --2- 172.21.15.37:0/4185548122 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed481342a0 0x7fed481cf4d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fed3800e590 tx=0x7fed3800ea50 comp rx=0 tx=0).stop 2024-04-26T00:38:56.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.618+0000 7fed4d52e640 1 -- 172.21.15.37:0/4185548122 shutdown_connections 2024-04-26T00:38:56.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.618+0000 7fed4d52e640 1 --2- 172.21.15.37:0/4185548122 >> v2:172.21.15.45:6812/2379918623 conn(0x7fed28035e80 0x7fed28038340 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:56.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.618+0000 7fed4d52e640 1 --2- 172.21.15.37:0/4185548122 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed481377d0 0x7fed4813e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:56.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.618+0000 7fed4d52e640 1 --2- 172.21.15.37:0/4185548122 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed48136eb0 0x7fed4813df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:56.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.618+0000 7fed4d52e640 1 --2- 172.21.15.37:0/4185548122 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed481342a0 0x7fed481cf4d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:56.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.618+0000 7fed4d52e640 1 -- 172.21.15.37:0/4185548122 >> 172.21.15.37:0/4185548122 conn(0x7fed480a6300 msgr2=0x7fed48134ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:38:56.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.618+0000 7fed4d52e640 1 -- 172.21.15.37:0/4185548122 shutdown_connections 2024-04-26T00:38:56.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:56.618+0000 7fed4d52e640 1 -- 172.21.15.37:0/4185548122 wait complete. 2024-04-26T00:38:57.472 INFO:tasks.rados.rados.0.smithi045.stdout:3309: done (2 left) 2024-04-26T00:38:57.472 INFO:tasks.rados.rados.0.smithi045.stdout:3310: done (1 left) 2024-04-26T00:38:57.472 INFO:tasks.rados.rados.0.smithi045.stdout:3311: done (0 left) 2024-04-26T00:38:57.472 INFO:tasks.rados.rados.0.smithi045.stdout:3312: write oid 2 current snap is 368 2024-04-26T00:38:57.472 INFO:tasks.rados.rados.0.smithi045.stdout:3312: seq_num 1180 ranges {632260=428535,1817458=463413,2687955=520034,3709324=122936} 2024-04-26T00:38:57.484 INFO:tasks.rados.rados.0.smithi045.stdout:3312: writing smithi04526579-2 from 632260 to 1060795 tid 1 2024-04-26T00:38:57.500 INFO:tasks.rados.rados.0.smithi045.stdout:3312: writing smithi04526579-2 from 1817458 to 2280871 tid 2 2024-04-26T00:38:57.511 INFO:tasks.rados.rados.0.smithi045.stdout:3312: writing smithi04526579-2 from 2687955 to 3207989 tid 3 2024-04-26T00:38:57.519 INFO:tasks.rados.rados.0.smithi045.stdout:3312: writing smithi04526579-2 from 3709324 to 3832260 tid 4 2024-04-26T00:38:57.520 INFO:tasks.rados.rados.0.smithi045.stdout:3313: delete oid 4 current snap is 368 2024-04-26T00:38:57.520 INFO:tasks.rados.rados.0.smithi045.stdout:3312: finishing write tid 1 to smithi04526579-2 2024-04-26T00:38:57.520 INFO:tasks.rados.rados.0.smithi045.stdout:3312: finishing write tid 2 to smithi04526579-2 2024-04-26T00:38:57.520 INFO:tasks.rados.rados.0.smithi045.stdout:3312: finishing write tid 3 to smithi04526579-2 2024-04-26T00:38:57.533 INFO:tasks.rados.rados.0.smithi045.stdout:3312: finishing write tid 4 to smithi04526579-2 2024-04-26T00:38:57.533 INFO:tasks.rados.rados.0.smithi045.stdout:3312: finishing write tid 5 to smithi04526579-2 2024-04-26T00:38:57.534 INFO:tasks.rados.rados.0.smithi045.stdout:3312: finishing write tid 6 to smithi04526579-2 2024-04-26T00:38:57.534 INFO:tasks.rados.rados.0.smithi045.stdout:3312: oid 2 updating version 0 to 2478 2024-04-26T00:38:57.534 INFO:tasks.rados.rados.0.smithi045.stdout:3312: oid 2 updating version 2478 to 2481 2024-04-26T00:38:57.534 INFO:tasks.rados.rados.0.smithi045.stdout:3312: oid 2 version 2481 is already newer than 2480 2024-04-26T00:38:57.534 INFO:tasks.rados.rados.0.smithi045.stdout:3312: oid 2 version 2481 is already newer than 2477 2024-04-26T00:38:57.534 INFO:tasks.rados.rados.0.smithi045.stdout:3312: oid 2 version 2481 is already newer than 2479 2024-04-26T00:38:57.534 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 2481 (ObjNum 1180 snap 368 seq_num 1180) dirty exists 2024-04-26T00:38:57.534 INFO:tasks.rados.rados.0.smithi045.stdout:3312: left oid 2 (ObjNum 1180 snap 368 seq_num 1180) 2024-04-26T00:38:57.535 INFO:tasks.rados.rados.0.smithi045.stdout:3312: done (1 left) 2024-04-26T00:38:57.535 INFO:tasks.rados.rados.0.smithi045.stdout:3313: done (0 left) 2024-04-26T00:38:57.535 INFO:tasks.rados.rados.0.smithi045.stdout:3314: delete oid 23 current snap is 368 2024-04-26T00:38:57.544 INFO:tasks.rados.rados.0.smithi045.stdout:3314: done (0 left) 2024-04-26T00:38:57.544 INFO:tasks.rados.rados.0.smithi045.stdout:3315: read oid 32 snap 364 2024-04-26T00:38:57.544 INFO:tasks.rados.rados.0.smithi045.stdout:3315: expect (ObjNum 1122 snap 339 seq_num 1122) 2024-04-26T00:38:57.544 INFO:tasks.rados.rados.0.smithi045.stdout:3316: snap_remove snap 363 2024-04-26T00:38:58.486 INFO:tasks.rados.rados.0.smithi045.stdout:3316: done (1 left) 2024-04-26T00:38:58.487 INFO:tasks.rados.rados.0.smithi045.stdout:3317: snap_remove snap 365 2024-04-26T00:38:59.451 INFO:tasks.rados.rados.0.smithi045.stdout:3317: done (1 left) 2024-04-26T00:38:59.452 INFO:tasks.rados.rados.0.smithi045.stdout:3318: copy_from oid 19 from oid 33 current snap is 368 2024-04-26T00:38:59.452 INFO:tasks.rados.rados.0.smithi045.stdout:3315: expect (ObjNum 1122 snap 339 seq_num 1122) 2024-04-26T00:38:59.635 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:38:59.688 INFO:tasks.rados.rados.0.smithi045.stdout:3318: finishing copy_from racing read to smithi04526579-19 2024-04-26T00:38:59.689 INFO:tasks.rados.rados.0.smithi045.stdout:3318: finishing copy_from to smithi04526579-19 2024-04-26T00:38:59.689 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 2483 (ObjNum 1145 snap 348 seq_num 1145) dirty exists 2024-04-26T00:38:59.689 INFO:tasks.rados.rados.0.smithi045.stdout:3315: done (1 left) 2024-04-26T00:38:59.689 INFO:tasks.rados.rados.0.smithi045.stdout:3318: done (0 left) 2024-04-26T00:38:59.689 INFO:tasks.rados.rados.0.smithi045.stdout:3319: write (excl) oid 39 current snap is 368 2024-04-26T00:38:59.689 INFO:tasks.rados.rados.0.smithi045.stdout:3319: seq_num 1183 ranges {682681=654282,1876551=610740,3053871=470203} 2024-04-26T00:38:59.710 INFO:tasks.rados.rados.0.smithi045.stdout:3319: writing smithi04526579-39 from 682681 to 1336963 tid 1 2024-04-26T00:38:59.729 INFO:tasks.rados.rados.0.smithi045.stdout:3319: writing smithi04526579-39 from 1876551 to 2487291 tid 2 2024-04-26T00:38:59.746 INFO:tasks.rados.rados.0.smithi045.stdout:3319: writing smithi04526579-39 from 3053871 to 3524074 tid 3 2024-04-26T00:38:59.746 INFO:tasks.rados.rados.0.smithi045.stdout:3320: copy_from oid 50 from oid 41 current snap is 368 2024-04-26T00:38:59.746 INFO:tasks.rados.rados.0.smithi045.stdout:3319: finishing write tid 1 to smithi04526579-39 2024-04-26T00:38:59.746 INFO:tasks.rados.rados.0.smithi045.stdout:3319: finishing write tid 2 to smithi04526579-39 2024-04-26T00:38:59.746 INFO:tasks.rados.rados.0.smithi045.stdout:3321: delete oid 26 current snap is 368 2024-04-26T00:38:59.752 INFO:tasks.rados.rados.0.smithi045.stdout:3321: done (2 left) 2024-04-26T00:38:59.753 INFO:tasks.rados.rados.0.smithi045.stdout:3322: read oid 5 snap -1 2024-04-26T00:38:59.753 INFO:tasks.rados.rados.0.smithi045.stdout:3322: expect (ObjNum 955 snap 295 seq_num 955) 2024-04-26T00:38:59.753 INFO:tasks.rados.rados.0.smithi045.stdout:3323: read oid 30 snap -1 2024-04-26T00:38:59.753 INFO:tasks.rados.rados.0.smithi045.stdout:3323: expect (ObjNum 1155 snap 355 seq_num 1155) 2024-04-26T00:38:59.753 INFO:tasks.rados.rados.0.smithi045.stdout:3324: write (excl) oid 46 current snap is 368 2024-04-26T00:38:59.753 INFO:tasks.rados.rados.0.smithi045.stdout:3324: seq_num 1185 ranges {716295=404780,1593515=614409,2708717=492747,3640519=275776} 2024-04-26T00:38:59.767 INFO:tasks.rados.rados.0.smithi045.stdout:3324: writing smithi04526579-46 from 716295 to 1121075 tid 1 2024-04-26T00:38:59.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d75252640 10 monclient: get_monmap_and_config 2024-04-26T00:38:59.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d75252640 10 monclient: build_initial_monmap 2024-04-26T00:38:59.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d75252640 10 monclient: monmap: 2024-04-26T00:38:59.774 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:38:59.774 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:59.774 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:38:59.777374+0000 2024-04-26T00:38:59.775 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:38:59.777374+0000 2024-04-26T00:38:59.775 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:38:59.775 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:59.775 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:38:59.775 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:38:59.775 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:38:59.775 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:38:59.775 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:38:59.775 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:59.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d75252640 1 Processor -- start 2024-04-26T00:38:59.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d75252640 1 -- start start 2024-04-26T00:38:59.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d75252640 10 monclient: init 2024-04-26T00:38:59.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d75252640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:59.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d75252640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-04-26T00:38:59.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d75252640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3d70136e10 0x7f3d7009c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:59.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d75252640 10 monclient(hunting): picked mon.noname-b con 0x7f3d70136e10 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:38:59.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d75252640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3d701371f0 0x7f3d701375d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:59.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d75252640 10 monclient(hunting): picked mon.noname-c con 0x7f3d701371f0 addr v2:172.21.15.45:3300/0 2024-04-26T00:38:59.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d75252640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f3d70135d90 0x7f3d70136240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:59.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d75252640 10 monclient(hunting): picked mon.noname-a con 0x7f3d70135d90 addr v2:172.21.15.37:3300/0 2024-04-26T00:38:59.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d75252640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:59.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d75252640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f3d7009b8f0 con 0x7f3d70135d90 2024-04-26T00:38:59.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d75252640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:59.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d75252640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f3d7009b680 con 0x7f3d701371f0 2024-04-26T00:38:59.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d75252640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:59.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d75252640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f3d7009bbd0 con 0x7f3d70136e10 2024-04-26T00:38:59.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d75252640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:59.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d75252640 10 monclient(hunting): authenticate will time out at 2121.083334s 2024-04-26T00:38:59.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d6ed9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3d70136e10 0x7f3d7009c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:59.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d6ed9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3d70136e10 0x7f3d7009c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:35554/0 (socket says 172.21.15.37:35554) 2024-04-26T00:38:59.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d6ed9d640 1 -- 172.21.15.37:0/947004726 learned_addr learned my addr 172.21.15.37:0/947004726 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:38:59.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d6ed9d640 10 monclient(hunting): get_auth_request con 0x7f3d70136e10 auth_method 0 2024-04-26T00:38:59.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d6ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:59.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d6ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:59.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d6ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:59.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d6f59e640 1 --2- 172.21.15.37:0/947004726 >> v2:172.21.15.37:3300/0 conn(0x7f3d70135d90 0x7f3d70136240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:59.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d6f59e640 10 monclient(hunting): get_auth_request con 0x7f3d70135d90 auth_method 0 2024-04-26T00:38:59.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d6f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:59.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d6f59e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:59.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d6f59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:59.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d6ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:59.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d6ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:59.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d6ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:59.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d6659c640 1 --2- 172.21.15.37:0/947004726 >> v2:172.21.15.45:3300/0 conn(0x7f3d701371f0 0x7f3d701375d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:59.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d6f59e640 1 -- 172.21.15.37:0/947004726 >> v2:172.21.15.37:3300/0 conn(0x7f3d70135d90 msgr2=0x7f3d70136240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:38:59.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d6f59e640 1 -- 172.21.15.37:0/947004726 >> v2:172.21.15.37:3300/0 conn(0x7f3d70135d90 msgr2=0x7f3d70136240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:38:59.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d6f59e640 1 --2- 172.21.15.37:0/947004726 >> v2:172.21.15.37:3300/0 conn(0x7f3d70135d90 0x7f3d70136240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:38:59.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d6659c640 10 monclient(hunting): get_auth_request con 0x7f3d701371f0 auth_method 0 2024-04-26T00:38:59.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d6659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:59.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d6659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:59.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d6659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:59.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d6f59e640 1 --2- 172.21.15.37:0/947004726 >> v2:172.21.15.37:3300/0 conn(0x7f3d70135d90 0x7f3d70136240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:38:59.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d6ed9d640 10 monclient(hunting): handle_auth_done global_id 204383 payload 274 2024-04-26T00:38:59.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d6ed9d640 1 -- 172.21.15.37:0/947004726 >> v2:172.21.15.45:3300/0 conn(0x7f3d701371f0 msgr2=0x7f3d701375d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:38:59.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d6ed9d640 1 --2- 172.21.15.37:0/947004726 >> v2:172.21.15.45:3300/0 conn(0x7f3d701371f0 0x7f3d701375d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:59.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d6ed9d640 1 -- 172.21.15.37:0/947004726 >> v2:172.21.15.37:3300/0 conn(0x7f3d70135d90 msgr2=0x7f3d70136240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:38:59.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d6ed9d640 1 --2- 172.21.15.37:0/947004726 >> v2:172.21.15.37:3300/0 conn(0x7f3d70135d90 0x7f3d70136240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:59.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d6ed9d640 10 monclient: _finish_hunting 0 2024-04-26T00:38:59.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d6ed9d640 1 monclient: found mon.noname-b 2024-04-26T00:38:59.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d6ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:59.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d6ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:38:59.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.774+0000 7f3d6ed9d640 1 -- 172.21.15.37:0/947004726 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d70137b10 con 0x7f3d70136e10 2024-04-26T00:38:59.783 INFO:tasks.rados.rados.0.smithi045.stdout:3324: writing smithi04526579-46 from 1593515 to 2207924 tid 2 2024-04-26T00:38:59.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6ed9d640 1 -- 172.21.15.37:0/947004726 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3d70136e10 msgr2=0x7f3d7009c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:59.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6ed9d640 1 --2- 172.21.15.37:0/947004726 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3d70136e10 0x7f3d7009c5b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3d70075d80 tx=0x7f3d6001c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18899e8983e88b11 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:59.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6e59c640 1 -- 172.21.15.37:0/947004726 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d600b1070 con 0x7f3d70136e10 2024-04-26T00:38:59.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6e59c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:59.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6e59c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:38:59.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6e59c640 10 monclient: dump: 2024-04-26T00:38:59.787 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:59.787 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:59.787 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:59.787 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:59.787 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:59.788 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:59.788 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:59.788 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:59.788 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:59.788 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:59.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6e59c640 10 monclient: _finish_auth 0 2024-04-26T00:38:59.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6e59c640 10 monclient: _check_auth_tickets 2024-04-26T00:38:59.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:59.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6e59c640 1 -- 172.21.15.37:0/947004726 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d600ac620 con 0x7f3d70136e10 2024-04-26T00:38:59.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d75252640 5 monclient: authenticate success, global_id 204383 2024-04-26T00:38:59.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d75252640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:38:59.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6e59c640 10 monclient: handle_config config(0 keys) 2024-04-26T00:38:59.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d75252640 10 monclient: get_monmap_and_config success 2024-04-26T00:38:59.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d75252640 4 set_mon_vals no callback set 2024-04-26T00:38:59.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6e59c640 1 -- 172.21.15.37:0/947004726 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d600ac9f0 con 0x7f3d70136e10 2024-04-26T00:38:59.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6e59c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:59.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6e59c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:38:59.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6e59c640 10 monclient: dump: 2024-04-26T00:38:59.789 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:59.789 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:59.789 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:59.789 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:59.789 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:59.789 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:59.789 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:59.789 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:59.789 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:59.789 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:59.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d75252640 10 monclient: shutdown 2024-04-26T00:38:59.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d75252640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:38:59.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d75252640 1 -- 172.21.15.37:0/947004726 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3d70136e10 msgr2=0x7f3d7009c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:59.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d75252640 1 --2- 172.21.15.37:0/947004726 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3d70136e10 0x7f3d7009c5b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3d70075d80 tx=0x7f3d6001c3e0 comp rx=0 tx=0).stop 2024-04-26T00:38:59.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d75252640 1 -- 172.21.15.37:0/947004726 shutdown_connections 2024-04-26T00:38:59.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d75252640 1 --2- 172.21.15.37:0/947004726 >> v2:172.21.15.45:3300/0 conn(0x7f3d701371f0 0x7f3d701375d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:59.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d75252640 1 --2- 172.21.15.37:0/947004726 >> v2:172.21.15.37:3300/0 conn(0x7f3d70135d90 0x7f3d70136240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:59.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d75252640 1 --2- 172.21.15.37:0/947004726 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3d70136e10 0x7f3d7009c5b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:59.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d75252640 1 -- 172.21.15.37:0/947004726 >> 172.21.15.37:0/947004726 conn(0x7f3d700a64e0 msgr2=0x7f3d700a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:38:59.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d75252640 1 -- 172.21.15.37:0/947004726 shutdown_connections 2024-04-26T00:38:59.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d75252640 1 -- 172.21.15.37:0/947004726 wait complete. 2024-04-26T00:38:59.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d75252640 10 monclient: build_initial_monmap 2024-04-26T00:38:59.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d75252640 10 monclient: monmap: 2024-04-26T00:38:59.791 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:38:59.791 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:38:59.791 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:38:59.791 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:38:59.791 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:38:59.791 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:38:59.791 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:38:59.791 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:38:59.791 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:38:59.791 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:59.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d75252640 1 Processor -- start 2024-04-26T00:38:59.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d75252640 1 -- start start 2024-04-26T00:38:59.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d75252640 10 monclient: init 2024-04-26T00:38:59.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d75252640 10 monclient: _reopen_session rank -1 2024-04-26T00:38:59.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d75252640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:38:59.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d75252640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3d70135d90 0x7f3d701cf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:59.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d75252640 10 monclient(hunting): picked mon.noname-c con 0x7f3d70135d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:38:59.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d75252640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3d70136e10 0x7f3d701cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:59.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d75252640 10 monclient(hunting): picked mon.noname-a con 0x7f3d70136e10 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:38:59.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d75252640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d701371f0 0x7f3d7013ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:59.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d75252640 10 monclient(hunting): picked mon.noname-b con 0x7f3d701371f0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:38:59.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d75252640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:59.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d75252640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f3d70140cf0 con 0x7f3d70136e10 2024-04-26T00:38:59.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d75252640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:59.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d75252640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3d70140b70 con 0x7f3d701371f0 2024-04-26T00:38:59.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d75252640 10 monclient(hunting): start opening mon connection 2024-04-26T00:38:59.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d75252640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f3d70140e70 con 0x7f3d70135d90 2024-04-26T00:38:59.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d75252640 10 monclient(hunting): _renew_subs 2024-04-26T00:38:59.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d75252640 10 monclient(hunting): authenticate will time out at 2121.090769s 2024-04-26T00:38:59.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6659c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3d70136e10 0x7f3d701cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:59.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6ed9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3d70135d90 0x7f3d701cf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:59.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6ed9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3d70135d90 0x7f3d701cf1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:35560/0 (socket says 172.21.15.37:35560) 2024-04-26T00:38:59.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6ed9d640 1 -- 172.21.15.37:0/1302034967 learned_addr learned my addr 172.21.15.37:0/1302034967 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:38:59.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6659c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3d70136e10 0x7f3d701cf8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51904/0 (socket says 172.21.15.37:51904) 2024-04-26T00:38:59.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6ed9d640 10 monclient(hunting): get_auth_request con 0x7f3d70135d90 auth_method 0 2024-04-26T00:38:59.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:59.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:59.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:59.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6659c640 10 monclient(hunting): get_auth_request con 0x7f3d70136e10 auth_method 0 2024-04-26T00:38:59.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:59.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:59.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:59.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:38:59.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:38:59.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:38:59.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6659c640 1 -- 172.21.15.37:0/1302034967 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3d70136e10 msgr2=0x7f3d701cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:38:59.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6659c640 1 -- 172.21.15.37:0/1302034967 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3d70136e10 msgr2=0x7f3d701cf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:38:59.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6659c640 1 --2- 172.21.15.37:0/1302034967 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3d70136e10 0x7f3d701cf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:38:59.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6659c640 1 --2- 172.21.15.37:0/1302034967 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3d70136e10 0x7f3d701cf8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:38:59.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6f59e640 1 --2- 172.21.15.37:0/1302034967 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d701371f0 0x7f3d7013ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:59.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6f59e640 10 monclient(hunting): get_auth_request con 0x7f3d701371f0 auth_method 0 2024-04-26T00:38:59.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:38:59.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6f59e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:38:59.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6f59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:38:59.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6ed9d640 10 monclient(hunting): handle_auth_done global_id 204386 payload 824 2024-04-26T00:38:59.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6ed9d640 1 -- 172.21.15.37:0/1302034967 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d701371f0 msgr2=0x7f3d7013ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:38:59.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6ed9d640 1 --2- 172.21.15.37:0/1302034967 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d701371f0 0x7f3d7013ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:59.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6ed9d640 1 -- 172.21.15.37:0/1302034967 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3d70136e10 msgr2=0x7f3d701cf8d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:38:59.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6ed9d640 1 --2- 172.21.15.37:0/1302034967 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3d70136e10 0x7f3d701cf8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:38:59.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6ed9d640 10 monclient: _finish_hunting 0 2024-04-26T00:38:59.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6ed9d640 1 monclient: found mon.noname-c 2024-04-26T00:38:59.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:38:59.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:38:59.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.782+0000 7f3d6ed9d640 1 -- 172.21.15.37:0/1302034967 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d7013c910 con 0x7f3d70135d90 2024-04-26T00:38:59.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.786+0000 7f3d6ed9d640 1 -- 172.21.15.37:0/1302034967 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3d70135d90 msgr2=0x7f3d701cf1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:38:59.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.786+0000 7f3d6ed9d640 1 --2- 172.21.15.37:0/1302034967 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3d70135d90 0x7f3d701cf1a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3d6001cc80 tx=0x7f3d6001ac20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:59.796 INFO:tasks.rados.rados.0.smithi045.stdout:3324: writing smithi04526579-46 from 2708717 to 3201464 tid 3 2024-04-26T00:38:59.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.786+0000 7f3d67fff640 1 -- 172.21.15.37:0/1302034967 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d600b1040 con 0x7f3d70135d90 2024-04-26T00:38:59.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.786+0000 7f3d67fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:59.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.786+0000 7f3d67fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:38:59.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.786+0000 7f3d67fff640 10 monclient: dump: 2024-04-26T00:38:59.798 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:59.798 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:59.798 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:59.798 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:59.798 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:59.798 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:59.799 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:59.799 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:59.799 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:59.799 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:59.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.786+0000 7f3d67fff640 10 monclient: _finish_auth 0 2024-04-26T00:38:59.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.786+0000 7f3d67fff640 10 monclient: _check_auth_tickets 2024-04-26T00:38:59.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.786+0000 7f3d67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:38:59.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.786+0000 7f3d67fff640 1 -- 172.21.15.37:0/1302034967 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d600bc070 con 0x7f3d70135d90 2024-04-26T00:38:59.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.786+0000 7f3d67fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:38:59.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.786+0000 7f3d67fff640 1 -- 172.21.15.37:0/1302034967 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d600b7690 con 0x7f3d70135d90 2024-04-26T00:38:59.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.786+0000 7f3d67fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:38:59.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.786+0000 7f3d67fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:38:59.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.786+0000 7f3d67fff640 10 monclient: dump: 2024-04-26T00:38:59.800 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:38:59.800 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:38:59.800 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:59.800 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:38:59.800 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:38:59.800 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:38:59.800 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:38:59.800 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:38:59.800 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:38:59.801 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:38:59.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.786+0000 7f3d75252640 5 monclient: authenticate success, global_id 204386 2024-04-26T00:38:59.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.786+0000 7f3d6dd9b640 4 set_mon_vals no callback set 2024-04-26T00:38:59.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.786+0000 7f3d75252640 10 monclient: _renew_subs 2024-04-26T00:38:59.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.786+0000 7f3d75252640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:59.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.786+0000 7f3d75252640 1 -- 172.21.15.37:0/1302034967 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d701208a0 con 0x7f3d70135d90 2024-04-26T00:38:59.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.786+0000 7f3d75252640 10 monclient: _renew_subs 2024-04-26T00:38:59.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.786+0000 7f3d75252640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:59.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.786+0000 7f3d75252640 1 -- 172.21.15.37:0/1302034967 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d7011fc30 con 0x7f3d70135d90 2024-04-26T00:38:59.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.790+0000 7f3d67fff640 1 -- 172.21.15.37:0/1302034967 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f3d600b7830 con 0x7f3d70135d90 2024-04-26T00:38:59.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.790+0000 7f3d75252640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:59.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.790+0000 7f3d75252640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:59.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.790+0000 7f3d75252640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:38:59.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.790+0000 7f3d75252640 1 -- 172.21.15.37:0/1302034967 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d70066f40 con 0x7f3d70135d90 2024-04-26T00:38:59.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.790+0000 7f3d67fff640 1 --2- 172.21.15.37:0/1302034967 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3d50035ed0 0x7f3d50038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:38:59.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.790+0000 7f3d67fff640 1 -- 172.21.15.37:0/1302034967 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(751..751 src has 705..751) ==== 3987+0+0 (secure 0 0 0) 0x7f3d600e2cb0 con 0x7f3d70135d90 2024-04-26T00:38:59.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.790+0000 7f3d6659c640 1 --2- 172.21.15.37:0/1302034967 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3d50035ed0 0x7f3d50038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:38:59.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.790+0000 7f3d6659c640 10 monclient: get_auth_request con 0x7f3d50035ed0 auth_method 0 2024-04-26T00:38:59.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.794+0000 7f3d6659c640 1 --2- 172.21.15.37:0/1302034967 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3d50035ed0 0x7f3d50038390 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f3d7013dfc0 tx=0x7f3d58008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:38:59.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.794+0000 7f3d67fff640 1 -- 172.21.15.37:0/1302034967 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f3d701208a0 con 0x7f3d70135d90 2024-04-26T00:38:59.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.794+0000 7f3d67fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:38:59.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:38:59.794+0000 7f3d67fff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:38:59.803 INFO:tasks.rados.rados.0.smithi045.stdout:3324: writing smithi04526579-46 from 3640519 to 3916295 tid 4 2024-04-26T00:38:59.803 INFO:tasks.rados.rados.0.smithi045.stdout:3325: snap_create 2024-04-26T00:38:59.803 INFO:tasks.rados.rados.0.smithi045.stdout:3319: finishing write tid 3 to smithi04526579-39 2024-04-26T00:38:59.803 INFO:tasks.rados.rados.0.smithi045.stdout:3319: finishing write tid 4 to smithi04526579-39 2024-04-26T00:38:59.803 INFO:tasks.rados.rados.0.smithi045.stdout:3319: finishing write tid 5 to smithi04526579-39 2024-04-26T00:38:59.803 INFO:tasks.rados.rados.0.smithi045.stdout:3319: oid 39 updating version 0 to 2388 2024-04-26T00:38:59.803 INFO:tasks.rados.rados.0.smithi045.stdout:3319: oid 39 updating version 2388 to 2389 2024-04-26T00:38:59.803 INFO:tasks.rados.rados.0.smithi045.stdout:3319: oid 39 updating version 2389 to 2390 2024-04-26T00:38:59.803 INFO:tasks.rados.rados.0.smithi045.stdout:3319: oid 39 version 2390 is already newer than 2387 2024-04-26T00:38:59.803 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 2390 (ObjNum 1183 snap 368 seq_num 1183) dirty exists 2024-04-26T00:38:59.803 INFO:tasks.rados.rados.0.smithi045.stdout:3319: left oid 39 (ObjNum 1183 snap 368 seq_num 1183) 2024-04-26T00:38:59.804 INFO:tasks.rados.rados.0.smithi045.stdout:3322: expect (ObjNum 955 snap 295 seq_num 955) 2024-04-26T00:38:59.896 INFO:tasks.rados.rados.0.smithi045.stdout:3323: expect (ObjNum 1155 snap 355 seq_num 1155) 2024-04-26T00:39:00.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:00.014+0000 7f3d75252640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:39:00.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:00.014+0000 7f3d75252640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:39:00.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:00.014+0000 7f3d75252640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:00.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:00.014+0000 7f3d75252640 1 -- 172.21.15.37:0/1302034967 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3d70120450 con 0x7f3d70135d90 2024-04-26T00:39:00.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:00.018+0000 7f3d67fff640 1 -- 172.21.15.37:0/1302034967 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7f3d600160f0 con 0x7f3d70135d90 2024-04-26T00:39:00.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:00.018+0000 7f3d67fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:39:00.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:00.018+0000 7f3d67fff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:39:00.017 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":140,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":31,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:39:00.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:00.018+0000 7f3d75252640 1 -- 172.21.15.37:0/1302034967 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3d50035ed0 msgr2=0x7f3d50038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:00.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:00.018+0000 7f3d75252640 1 --2- 172.21.15.37:0/1302034967 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3d50035ed0 0x7f3d50038390 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f3d7013dfc0 tx=0x7f3d58008040 comp rx=0 tx=0).stop 2024-04-26T00:39:00.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:00.018+0000 7f3d75252640 10 monclient: shutdown 2024-04-26T00:39:00.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:00.018+0000 7f3d75252640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:00.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:00.018+0000 7f3d75252640 1 -- 172.21.15.37:0/1302034967 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3d70135d90 msgr2=0x7f3d701cf1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:00.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:00.018+0000 7f3d75252640 1 --2- 172.21.15.37:0/1302034967 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3d70135d90 0x7f3d701cf1a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3d6001cc80 tx=0x7f3d6001ac20 comp rx=0 tx=0).stop 2024-04-26T00:39:00.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:00.018+0000 7f3d75252640 1 -- 172.21.15.37:0/1302034967 shutdown_connections 2024-04-26T00:39:00.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:00.018+0000 7f3d75252640 1 --2- 172.21.15.37:0/1302034967 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3d50035ed0 0x7f3d50038390 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:00.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:00.018+0000 7f3d75252640 1 --2- 172.21.15.37:0/1302034967 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d701371f0 0x7f3d7013ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:00.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:00.018+0000 7f3d75252640 1 --2- 172.21.15.37:0/1302034967 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3d70136e10 0x7f3d701cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:00.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:00.018+0000 7f3d75252640 1 --2- 172.21.15.37:0/1302034967 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3d70135d90 0x7f3d701cf1a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:00.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:00.018+0000 7f3d75252640 1 -- 172.21.15.37:0/1302034967 >> 172.21.15.37:0/1302034967 conn(0x7f3d700a64e0 msgr2=0x7f3d700a88d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:00.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:00.018+0000 7f3d75252640 1 -- 172.21.15.37:0/1302034967 shutdown_connections 2024-04-26T00:39:00.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:00.018+0000 7f3d75252640 1 -- 172.21.15.37:0/1302034967 wait complete. 2024-04-26T00:39:00.125 INFO:tasks.rados.rados.0.smithi045.stdout:3320: finishing copy_from racing read to smithi04526579-50 2024-04-26T00:39:00.125 INFO:tasks.rados.rados.0.smithi045.stdout:3320: finishing copy_from to smithi04526579-50 2024-04-26T00:39:00.125 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 2339 (ObjNum 1112 snap 331 seq_num 1112) dirty exists 2024-04-26T00:39:00.125 INFO:tasks.rados.rados.0.smithi045.stdout:3324: finishing write tid 1 to smithi04526579-46 2024-04-26T00:39:00.125 INFO:tasks.rados.rados.0.smithi045.stdout:3324: finishing write tid 2 to smithi04526579-46 2024-04-26T00:39:00.125 INFO:tasks.rados.rados.0.smithi045.stdout:3324: finishing write tid 3 to smithi04526579-46 2024-04-26T00:39:00.126 INFO:tasks.rados.rados.0.smithi045.stdout:3324: finishing write tid 4 to smithi04526579-46 2024-04-26T00:39:00.126 INFO:tasks.rados.rados.0.smithi045.stdout:3324: finishing write tid 5 to smithi04526579-46 2024-04-26T00:39:00.126 INFO:tasks.rados.rados.0.smithi045.stdout:3324: finishing write tid 6 to smithi04526579-46 2024-04-26T00:39:00.126 INFO:tasks.rados.rados.0.smithi045.stdout:3324: oid 46 updating version 0 to 2440 2024-04-26T00:39:00.126 INFO:tasks.rados.rados.0.smithi045.stdout:3324: oid 46 version 2440 is already newer than 2436 2024-04-26T00:39:00.126 INFO:tasks.rados.rados.0.smithi045.stdout:3324: oid 46 version 2440 is already newer than 2439 2024-04-26T00:39:00.126 INFO:tasks.rados.rados.0.smithi045.stdout:3324: oid 46 version 2440 is already newer than 2438 2024-04-26T00:39:00.126 INFO:tasks.rados.rados.0.smithi045.stdout:3324: oid 46 version 2440 is already newer than 2437 2024-04-26T00:39:00.126 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 2440 (ObjNum 1185 snap 368 seq_num 1185) dirty exists 2024-04-26T00:39:00.126 INFO:tasks.rados.rados.0.smithi045.stdout:3324: left oid 46 (ObjNum 1185 snap 368 seq_num 1185) 2024-04-26T00:39:00.492 INFO:tasks.rados.rados.0.smithi045.stdout:3319: done (5 left) 2024-04-26T00:39:00.492 INFO:tasks.rados.rados.0.smithi045.stdout:3320: done (4 left) 2024-04-26T00:39:00.492 INFO:tasks.rados.rados.0.smithi045.stdout:3322: done (3 left) 2024-04-26T00:39:00.492 INFO:tasks.rados.rados.0.smithi045.stdout:3323: done (2 left) 2024-04-26T00:39:00.493 INFO:tasks.rados.rados.0.smithi045.stdout:3324: done (1 left) 2024-04-26T00:39:00.493 INFO:tasks.rados.rados.0.smithi045.stdout:3325: done (0 left) 2024-04-26T00:39:00.493 INFO:tasks.rados.rados.0.smithi045.stdout:3326: write (excl) oid 4 current snap is 369 2024-04-26T00:39:00.493 INFO:tasks.rados.rados.0.smithi045.stdout:3326: seq_num 1186 ranges {733102=596382,1768350=758067,3136140=704019} 2024-04-26T00:39:00.504 INFO:tasks.rados.rados.0.smithi045.stdout:3326: writing smithi04526579-4 from 733102 to 1329484 tid 1 2024-04-26T00:39:00.516 INFO:tasks.rados.rados.0.smithi045.stdout:3326: writing smithi04526579-4 from 1768350 to 2526417 tid 2 2024-04-26T00:39:00.529 INFO:tasks.rados.rados.0.smithi045.stdout:3326: writing smithi04526579-4 from 3136140 to 3840159 tid 3 2024-04-26T00:39:00.529 INFO:tasks.rados.rados.0.smithi045.stdout:3327: read oid 17 snap -1 2024-04-26T00:39:00.529 INFO:tasks.rados.rados.0.smithi045.stdout:3327: expect (ObjNum 1051 snap 317 seq_num 1051) 2024-04-26T00:39:00.529 INFO:tasks.rados.rados.0.smithi045.stdout:3328: write (excl) oid 7 current snap is 369 2024-04-26T00:39:00.529 INFO:tasks.rados.rados.0.smithi045.stdout:3326: finishing write tid 1 to smithi04526579-4 2024-04-26T00:39:00.529 INFO:tasks.rados.rados.0.smithi045.stdout:3326: finishing write tid 2 to smithi04526579-4 2024-04-26T00:39:00.529 INFO:tasks.rados.rados.0.smithi045.stdout:3328: seq_num 1187 ranges {749909=671631,2110479=618078,3163563=515291} 2024-04-26T00:39:00.544 INFO:tasks.rados.rados.0.smithi045.stdout:3328: writing smithi04526579-7 from 749909 to 1421540 tid 1 2024-04-26T00:39:00.558 INFO:tasks.rados.rados.0.smithi045.stdout:3328: writing smithi04526579-7 from 2110479 to 2728557 tid 2 2024-04-26T00:39:00.566 INFO:tasks.rados.rados.0.smithi045.stdout:3328: writing smithi04526579-7 from 3163563 to 3678854 tid 3 2024-04-26T00:39:00.567 INFO:tasks.rados.rados.0.smithi045.stdout:3329: copy_from oid 19 from oid 10 current snap is 369 2024-04-26T00:39:00.567 INFO:tasks.rados.rados.0.smithi045.stdout:3327: expect (ObjNum 1051 snap 317 seq_num 1051) 2024-04-26T00:39:00.844 INFO:tasks.rados.rados.0.smithi045.stdout:3326: finishing write tid 3 to smithi04526579-4 2024-04-26T00:39:00.844 INFO:tasks.rados.rados.0.smithi045.stdout:3326: finishing write tid 4 to smithi04526579-4 2024-04-26T00:39:00.845 INFO:tasks.rados.rados.0.smithi045.stdout:3326: finishing write tid 5 to smithi04526579-4 2024-04-26T00:39:00.845 INFO:tasks.rados.rados.0.smithi045.stdout:3326: oid 4 updating version 0 to 2485 2024-04-26T00:39:00.845 INFO:tasks.rados.rados.0.smithi045.stdout:3326: oid 4 updating version 2485 to 2486 2024-04-26T00:39:00.845 INFO:tasks.rados.rados.0.smithi045.stdout:3326: oid 4 version 2486 is already newer than 2484 2024-04-26T00:39:00.845 INFO:tasks.rados.rados.0.smithi045.stdout:3326: oid 4 version 2486 is already newer than 2483 2024-04-26T00:39:00.845 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 2486 (ObjNum 1186 snap 369 seq_num 1186) dirty exists 2024-04-26T00:39:00.845 INFO:tasks.rados.rados.0.smithi045.stdout:3326: left oid 4 (ObjNum 1186 snap 369 seq_num 1186) 2024-04-26T00:39:00.845 INFO:tasks.rados.rados.0.smithi045.stdout:3328: finishing write tid 1 to smithi04526579-7 2024-04-26T00:39:00.845 INFO:tasks.rados.rados.0.smithi045.stdout:3328: finishing write tid 2 to smithi04526579-7 2024-04-26T00:39:00.845 INFO:tasks.rados.rados.0.smithi045.stdout:3328: finishing write tid 3 to smithi04526579-7 2024-04-26T00:39:00.845 INFO:tasks.rados.rados.0.smithi045.stdout:3328: finishing write tid 4 to smithi04526579-7 2024-04-26T00:39:00.845 INFO:tasks.rados.rados.0.smithi045.stdout:3328: finishing write tid 5 to smithi04526579-7 2024-04-26T00:39:00.845 INFO:tasks.rados.rados.0.smithi045.stdout:3328: oid 7 updating version 0 to 2545 2024-04-26T00:39:00.846 INFO:tasks.rados.rados.0.smithi045.stdout:3328: oid 7 version 2545 is already newer than 2544 2024-04-26T00:39:00.846 INFO:tasks.rados.rados.0.smithi045.stdout:3328: oid 7 version 2545 is already newer than 2543 2024-04-26T00:39:00.846 INFO:tasks.rados.rados.0.smithi045.stdout:3328: oid 7 version 2545 is already newer than 2542 2024-04-26T00:39:00.846 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 2545 (ObjNum 1187 snap 369 seq_num 1187) dirty exists 2024-04-26T00:39:00.846 INFO:tasks.rados.rados.0.smithi045.stdout:3328: left oid 7 (ObjNum 1187 snap 369 seq_num 1187) 2024-04-26T00:39:00.846 INFO:tasks.rados.rados.0.smithi045.stdout:3329: finishing copy_from racing read to smithi04526579-19 2024-04-26T00:39:00.846 INFO:tasks.rados.rados.0.smithi045.stdout:3329: finishing copy_from to smithi04526579-19 2024-04-26T00:39:00.846 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 2339 (ObjNum 1112 snap 331 seq_num 1112) dirty exists 2024-04-26T00:39:00.846 INFO:tasks.rados.rados.0.smithi045.stdout:3326: done (3 left) 2024-04-26T00:39:00.846 INFO:tasks.rados.rados.0.smithi045.stdout:3327: done (2 left) 2024-04-26T00:39:00.846 INFO:tasks.rados.rados.0.smithi045.stdout:3328: done (1 left) 2024-04-26T00:39:00.846 INFO:tasks.rados.rados.0.smithi045.stdout:3329: done (0 left) 2024-04-26T00:39:00.847 INFO:tasks.rados.rados.0.smithi045.stdout:3330: snap_remove snap 349 2024-04-26T00:39:01.504 INFO:tasks.rados.rados.0.smithi045.stdout:3330: done (0 left) 2024-04-26T00:39:01.504 INFO:tasks.rados.rados.0.smithi045.stdout:3331: copy_from oid 34 from oid 15 current snap is 369 2024-04-26T00:39:01.504 INFO:tasks.rados.rados.0.smithi045.stdout:3332: read oid 42 snap -1 2024-04-26T00:39:01.505 INFO:tasks.rados.rados.0.smithi045.stdout:3332: expect (ObjNum 1128 snap 340 seq_num 1128) 2024-04-26T00:39:01.505 INFO:tasks.rados.rados.0.smithi045.stdout:3333: snap_remove snap 367 2024-04-26T00:39:02.689 INFO:tasks.rados.rados.0.smithi045.stdout:3333: done (2 left) 2024-04-26T00:39:02.689 INFO:tasks.rados.rados.0.smithi045.stdout:3334: write oid 39 current snap is 369 2024-04-26T00:39:02.689 INFO:tasks.rados.rados.0.smithi045.stdout:3332: expect (ObjNum 1128 snap 340 seq_num 1128) 2024-04-26T00:39:02.823 INFO:tasks.rados.rados.0.smithi045.stdout:3331: finishing copy_from racing read to smithi04526579-34 2024-04-26T00:39:02.823 INFO:tasks.rados.rados.0.smithi045.stdout:3331: finishing copy_from to smithi04526579-34 2024-04-26T00:39:02.823 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 2453 (ObjNum 1160 snap 359 seq_num 1160) dirty exists 2024-04-26T00:39:02.823 INFO:tasks.rados.rados.0.smithi045.stdout:3334: seq_num 1188 ranges {766716=746880,2168961=761736,3474633=442916} 2024-04-26T00:39:02.847 INFO:tasks.rados.rados.0.smithi045.stdout:3334: writing smithi04526579-39 from 766716 to 1513596 tid 1 2024-04-26T00:39:02.870 INFO:tasks.rados.rados.0.smithi045.stdout:3334: writing smithi04526579-39 from 2168961 to 2930697 tid 2 2024-04-26T00:39:02.884 INFO:tasks.rados.rados.0.smithi045.stdout:3334: writing smithi04526579-39 from 3474633 to 3917549 tid 3 2024-04-26T00:39:02.884 INFO:tasks.rados.rados.0.smithi045.stdout:3331: done (2 left) 2024-04-26T00:39:02.884 INFO:tasks.rados.rados.0.smithi045.stdout:3332: done (1 left) 2024-04-26T00:39:02.884 INFO:tasks.rados.rados.0.smithi045.stdout:3335: rollback oid 8 current snap is 369 2024-04-26T00:39:02.884 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 8 to 361 2024-04-26T00:39:02.885 INFO:tasks.rados.rados.0.smithi045.stdout:3334: finishing write tid 1 to smithi04526579-39 2024-04-26T00:39:02.885 INFO:tasks.rados.rados.0.smithi045.stdout:3334: finishing write tid 2 to smithi04526579-39 2024-04-26T00:39:02.885 INFO:tasks.rados.rados.0.smithi045.stdout:3336: write oid 41 current snap is 369 2024-04-26T00:39:02.885 INFO:tasks.rados.rados.0.smithi045.stdout:3336: seq_num 1189 ranges {783523=422129,1827443=621747,3218409=654188} 2024-04-26T00:39:02.897 INFO:tasks.rados.rados.0.smithi045.stdout:3336: writing smithi04526579-41 from 783523 to 1205652 tid 1 2024-04-26T00:39:02.909 INFO:tasks.rados.rados.0.smithi045.stdout:3336: writing smithi04526579-41 from 1827443 to 2449190 tid 2 2024-04-26T00:39:02.922 INFO:tasks.rados.rados.0.smithi045.stdout:3336: writing smithi04526579-41 from 3218409 to 3872597 tid 3 2024-04-26T00:39:02.922 INFO:tasks.rados.rados.0.smithi045.stdout:3337: read oid 23 snap -1 2024-04-26T00:39:02.922 INFO:tasks.rados.rados.0.smithi045.stdout:3337: expect deleted 2024-04-26T00:39:02.922 INFO:tasks.rados.rados.0.smithi045.stdout:3338: copy_from oid 15 from oid 16 current snap is 369 2024-04-26T00:39:02.922 INFO:tasks.rados.rados.0.smithi045.stdout:3334: finishing write tid 3 to smithi04526579-39 2024-04-26T00:39:02.922 INFO:tasks.rados.rados.0.smithi045.stdout:3334: finishing write tid 4 to smithi04526579-39 2024-04-26T00:39:02.922 INFO:tasks.rados.rados.0.smithi045.stdout:3334: finishing write tid 5 to smithi04526579-39 2024-04-26T00:39:02.922 INFO:tasks.rados.rados.0.smithi045.stdout:3334: oid 39 updating version 0 to 2392 2024-04-26T00:39:02.922 INFO:tasks.rados.rados.0.smithi045.stdout:3334: oid 39 updating version 2392 to 2394 2024-04-26T00:39:02.922 INFO:tasks.rados.rados.0.smithi045.stdout:3334: oid 39 version 2394 is already newer than 2391 2024-04-26T00:39:02.923 INFO:tasks.rados.rados.0.smithi045.stdout:3334: oid 39 version 2394 is already newer than 2393 2024-04-26T00:39:02.923 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 2394 (ObjNum 1188 snap 369 seq_num 1188) dirty exists 2024-04-26T00:39:02.923 INFO:tasks.rados.rados.0.smithi045.stdout:3334: left oid 39 (ObjNum 1188 snap 369 seq_num 1188) 2024-04-26T00:39:02.923 INFO:tasks.rados.rados.0.smithi045.stdout:3335: finishing rollback tid 0 to smithi04526579-8 2024-04-26T00:39:02.923 INFO:tasks.rados.rados.0.smithi045.stdout:3335: finishing rollback tid 1 to smithi04526579-8 2024-04-26T00:39:02.923 INFO:tasks.rados.rados.0.smithi045.stdout:3335: finishing rollback tid 2 to smithi04526579-8 2024-04-26T00:39:02.923 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 2210 (ObjNum 1103 snap 329 seq_num 1103) dirty exists 2024-04-26T00:39:02.923 INFO:tasks.rados.rados.0.smithi045.stdout:3336: finishing write tid 1 to smithi04526579-41 2024-04-26T00:39:02.923 INFO:tasks.rados.rados.0.smithi045.stdout:3334: done (4 left) 2024-04-26T00:39:02.923 INFO:tasks.rados.rados.0.smithi045.stdout:3335: done (3 left) 2024-04-26T00:39:02.923 INFO:tasks.rados.rados.0.smithi045.stdout:3339: snap_create 2024-04-26T00:39:02.923 INFO:tasks.rados.rados.0.smithi045.stdout:3336: finishing write tid 2 to smithi04526579-41 2024-04-26T00:39:02.928 INFO:tasks.rados.rados.0.smithi045.stdout:3336: finishing write tid 3 to smithi04526579-41 2024-04-26T00:39:02.928 INFO:tasks.rados.rados.0.smithi045.stdout:3336: finishing write tid 4 to smithi04526579-41 2024-04-26T00:39:02.929 INFO:tasks.rados.rados.0.smithi045.stdout:3336: finishing write tid 5 to smithi04526579-41 2024-04-26T00:39:02.929 INFO:tasks.rados.rados.0.smithi045.stdout:3336: oid 41 updating version 0 to 2459 2024-04-26T00:39:02.929 INFO:tasks.rados.rados.0.smithi045.stdout:3336: oid 41 updating version 2459 to 2460 2024-04-26T00:39:02.929 INFO:tasks.rados.rados.0.smithi045.stdout:3336: oid 41 updating version 2460 to 2462 2024-04-26T00:39:02.929 INFO:tasks.rados.rados.0.smithi045.stdout:3336: oid 41 version 2462 is already newer than 2461 2024-04-26T00:39:02.929 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 2462 (ObjNum 1189 snap 369 seq_num 1189) dirty exists 2024-04-26T00:39:02.929 INFO:tasks.rados.rados.0.smithi045.stdout:3336: left oid 41 (ObjNum 1189 snap 369 seq_num 1189) 2024-04-26T00:39:02.951 INFO:tasks.rados.rados.0.smithi045.stdout:3338: finishing copy_from racing read to smithi04526579-15 2024-04-26T00:39:02.952 INFO:tasks.rados.rados.0.smithi045.stdout:3338: finishing copy_from to smithi04526579-15 2024-04-26T00:39:02.952 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 1195 (ObjNum 1102 snap 329 seq_num 1102) dirty exists 2024-04-26T00:39:03.036 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:39:03.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f606e540640 10 monclient: get_monmap_and_config 2024-04-26T00:39:03.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f606e540640 10 monclient: build_initial_monmap 2024-04-26T00:39:03.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f606e540640 10 monclient: monmap: 2024-04-26T00:39:03.170 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:03.170 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:03.170 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:39:03.173049+0000 2024-04-26T00:39:03.170 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:39:03.173049+0000 2024-04-26T00:39:03.170 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:03.170 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:03.170 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:39:03.170 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:39:03.170 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:39:03.170 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:39:03.170 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:39:03.171 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:03.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f606e540640 1 Processor -- start 2024-04-26T00:39:03.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f606e540640 1 -- start start 2024-04-26T00:39:03.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f606e540640 10 monclient: init 2024-04-26T00:39:03.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f606e540640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:03.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f606e540640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-04-26T00:39:03.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f606e540640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6068133b20 0x7f606809e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:03.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f606e540640 10 monclient(hunting): picked mon.noname-c con 0x7f6068133b20 addr v2:172.21.15.45:3300/0 2024-04-26T00:39:03.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f606e540640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6068133f00 addr v1:172.21.15.45:6789/0 2024-04-26T00:39:03.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f606e540640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6068139050 0x7f606813b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:03.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f606e540640 10 monclient(hunting): picked mon.noname-a con 0x7f6068139050 addr v2:172.21.15.37:3300/0 2024-04-26T00:39:03.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f606e540640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f606809fe20 con 0x7f6068133f00 2024-04-26T00:39:03.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f606e540640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:03.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f606e540640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f6068072d00 con 0x7f6068139050 2024-04-26T00:39:03.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f606e540640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:03.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f606e540640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f606809ba60 con 0x7f6068133b20 2024-04-26T00:39:03.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f606e540640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:03.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f606e540640 10 monclient(hunting): authenticate will time out at 2124.478889s 2024-04-26T00:39:03.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f606cadd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6068139050 0x7f606813b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:03.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f606cadd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6068139050 0x7f606813b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51926/0 (socket says 172.21.15.37:51926) 2024-04-26T00:39:03.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f606cadd640 1 -- 172.21.15.37:0/3340890584 learned_addr learned my addr 172.21.15.37:0/3340890584 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:03.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f606cadd640 10 monclient(hunting): get_auth_request con 0x7f6068139050 auth_method 0 2024-04-26T00:39:03.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f606cadd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:03.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f606cadd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:03.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f606cadd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:03.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f606cadd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:03.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f606cadd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:03.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f606cadd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:03.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f6067fff640 1 --2- 172.21.15.37:0/3340890584 >> v2:172.21.15.45:3300/0 conn(0x7f6068133b20 0x7f606809e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:03.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f6067fff640 10 monclient(hunting): get_auth_request con 0x7f6068133b20 auth_method 0 2024-04-26T00:39:03.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f6067fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:03.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f6067fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:03.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f6067fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:03.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f606cadd640 10 monclient(hunting): handle_auth_done global_id 234099 payload 274 2024-04-26T00:39:03.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f606cadd640 1 -- 172.21.15.37:0/3340890584 >> v2:172.21.15.45:3300/0 conn(0x7f6068133b20 msgr2=0x7f606809e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:03.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f606cadd640 1 --2- 172.21.15.37:0/3340890584 >> v2:172.21.15.45:3300/0 conn(0x7f6068133b20 0x7f606809e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:03.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f606cadd640 1 -- 172.21.15.37:0/3340890584 >> v1:172.21.15.45:6789/0 conn(0x7f6068133f00 legacy=0x7f6068134350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:03.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f606cadd640 10 monclient: _finish_hunting 0 2024-04-26T00:39:03.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f606cadd640 1 monclient: found mon.noname-a 2024-04-26T00:39:03.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f606cadd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:03.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f606cadd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:39:03.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.170+0000 7f606cadd640 1 -- 172.21.15.37:0/3340890584 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f606809dd10 con 0x7f6068139050 2024-04-26T00:39:03.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.174+0000 7f606cadd640 1 -- 172.21.15.37:0/3340890584 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6068139050 msgr2=0x7f606813b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:03.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.174+0000 7f606cadd640 1 --2- 172.21.15.37:0/3340890584 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6068139050 0x7f606813b440 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f606809be70 tx=0x7f606001e860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52edfdad419c7851 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:03.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.174+0000 7f6066ffd640 1 -- 172.21.15.37:0/3340890584 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60600b4070 con 0x7f6068139050 2024-04-26T00:39:03.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.174+0000 7f6066ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:03.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.174+0000 7f6066ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:39:03.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.174+0000 7f6066ffd640 10 monclient: dump: 2024-04-26T00:39:03.178 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:03.178 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:03.178 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.178 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.178 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:03.178 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:03.178 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:03.178 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:03.178 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:03.178 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:03.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.174+0000 7f6066ffd640 10 monclient: _finish_auth 0 2024-04-26T00:39:03.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.174+0000 7f6066ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:39:03.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.174+0000 7f6066ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:03.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.174+0000 7f6066ffd640 1 -- 172.21.15.37:0/3340890584 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60600a7740 con 0x7f6068139050 2024-04-26T00:39:03.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.174+0000 7f6066ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:03.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.174+0000 7f6066ffd640 1 -- 172.21.15.37:0/3340890584 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60600a7b10 con 0x7f6068139050 2024-04-26T00:39:03.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.174+0000 7f606e540640 5 monclient: authenticate success, global_id 234099 2024-04-26T00:39:03.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.174+0000 7f606e540640 10 monclient: get_monmap_and_config success 2024-04-26T00:39:03.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.174+0000 7f606e540640 4 set_mon_vals no callback set 2024-04-26T00:39:03.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.174+0000 7f6066ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:03.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.174+0000 7f6066ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:39:03.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.174+0000 7f6066ffd640 10 monclient: dump: 2024-04-26T00:39:03.179 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:03.179 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:03.179 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.179 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.180 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:03.180 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:03.180 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:03.180 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:03.180 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:03.180 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:03.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f606e540640 10 monclient: shutdown 2024-04-26T00:39:03.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f606e540640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:03.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f606e540640 1 -- 172.21.15.37:0/3340890584 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6068139050 msgr2=0x7f606813b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:03.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f606e540640 1 --2- 172.21.15.37:0/3340890584 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6068139050 0x7f606813b440 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f606809be70 tx=0x7f606001e860 comp rx=0 tx=0).stop 2024-04-26T00:39:03.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f606e540640 1 -- 172.21.15.37:0/3340890584 shutdown_connections 2024-04-26T00:39:03.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f606e540640 1 --2- 172.21.15.37:0/3340890584 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6068139050 0x7f606813b440 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:03.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f606e540640 1 --2- 172.21.15.37:0/3340890584 >> v2:172.21.15.45:3300/0 conn(0x7f6068133b20 0x7f606809e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:03.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f606e540640 1 -- 172.21.15.37:0/3340890584 >> 172.21.15.37:0/3340890584 conn(0x7f60680a1ad0 msgr2=0x7f60680a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:03.180 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f606e540640 1 -- 172.21.15.37:0/3340890584 shutdown_connections 2024-04-26T00:39:03.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f606e540640 1 -- 172.21.15.37:0/3340890584 wait complete. 2024-04-26T00:39:03.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f606e540640 10 monclient: build_initial_monmap 2024-04-26T00:39:03.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f606e540640 10 monclient: monmap: 2024-04-26T00:39:03.181 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:03.181 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:39:03.181 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:39:03.181 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:39:03.181 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:03.181 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:39:03.181 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:39:03.181 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:39:03.181 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:39:03.181 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:03.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f606e540640 1 Processor -- start 2024-04-26T00:39:03.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f606e540640 1 -- start start 2024-04-26T00:39:03.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f606e540640 10 monclient: init 2024-04-26T00:39:03.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f606e540640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:03.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f606e540640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:39:03.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f606e540640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6068133b20 0x7f60681ca800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:03.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f606e540640 10 monclient(hunting): picked mon.noname-c con 0x7f6068133b20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:03.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f606e540640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6068133f00 0x7f6068130870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:03.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f606e540640 10 monclient(hunting): picked mon.noname-b con 0x7f6068133f00 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:39:03.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f606e540640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6068139050 0x7f606806e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:03.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f606e540640 10 monclient(hunting): picked mon.noname-a con 0x7f6068139050 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:39:03.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f606e540640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:03.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f606e540640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f60680a0bf0 con 0x7f6068139050 2024-04-26T00:39:03.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f606e540640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:03.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f606e540640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f60680a0a70 con 0x7f6068133f00 2024-04-26T00:39:03.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f606e540640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:03.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f606e540640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f60680a0d70 con 0x7f6068133b20 2024-04-26T00:39:03.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f606e540640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:03.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f606e540640 10 monclient(hunting): authenticate will time out at 2124.486435s 2024-04-26T00:39:03.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f6067fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6068133b20 0x7f60681ca800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:03.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f606cadd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6068139050 0x7f606806e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:03.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f6067fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6068133b20 0x7f60681ca800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:35594/0 (socket says 172.21.15.37:35594) 2024-04-26T00:39:03.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f6067fff640 1 -- 172.21.15.37:0/3978074883 learned_addr learned my addr 172.21.15.37:0/3978074883 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:03.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f6067fff640 10 monclient(hunting): get_auth_request con 0x7f6068133b20 auth_method 0 2024-04-26T00:39:03.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f6067fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:03.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f6067fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:03.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f6067fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:03.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f606cadd640 10 monclient(hunting): get_auth_request con 0x7f6068139050 auth_method 0 2024-04-26T00:39:03.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f606cadd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:03.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f606cadd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:03.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f606cadd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:03.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f6067fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:03.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f6067fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:03.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f6067fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:03.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f606cadd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:03.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f606cadd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:03.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f606cadd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:03.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f60677fe640 1 --2- 172.21.15.37:0/3978074883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6068133f00 0x7f6068130870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:03.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f60677fe640 10 monclient(hunting): get_auth_request con 0x7f6068133f00 auth_method 0 2024-04-26T00:39:03.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f60677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:03.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f60677fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:03.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f60677fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:03.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f6067fff640 10 monclient(hunting): handle_auth_done global_id 204389 payload 824 2024-04-26T00:39:03.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f6067fff640 1 -- 172.21.15.37:0/3978074883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6068133f00 msgr2=0x7f6068130870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:03.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f6067fff640 1 --2- 172.21.15.37:0/3978074883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6068133f00 0x7f6068130870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:03.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f6067fff640 1 -- 172.21.15.37:0/3978074883 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6068139050 msgr2=0x7f606806e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:03.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f6067fff640 1 --2- 172.21.15.37:0/3978074883 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6068139050 0x7f606806e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:03.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f6067fff640 10 monclient: _finish_hunting 0 2024-04-26T00:39:03.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f6067fff640 1 monclient: found mon.noname-c 2024-04-26T00:39:03.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f6067fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:03.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f6067fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:39:03.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f6067fff640 1 -- 172.21.15.37:0/3978074883 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60680a0000 con 0x7f6068133b20 2024-04-26T00:39:03.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.178+0000 7f606cadd640 1 --2- 172.21.15.37:0/3978074883 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6068139050 0x7f606806e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:39:03.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.182+0000 7f6067fff640 1 -- 172.21.15.37:0/3978074883 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6068133b20 msgr2=0x7f60681ca800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:03.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.182+0000 7f6067fff640 1 --2- 172.21.15.37:0/3978074883 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6068133b20 0x7f60681ca800 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6058007e60 tx=0x7f605800e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:03.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.182+0000 7f6064ff9640 1 -- 172.21.15.37:0/3978074883 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6058013790 con 0x7f6068133b20 2024-04-26T00:39:03.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.182+0000 7f6064ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:03.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.182+0000 7f6064ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:39:03.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.182+0000 7f6064ff9640 10 monclient: dump: 2024-04-26T00:39:03.190 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:03.190 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:03.190 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.190 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.191 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:03.191 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:03.191 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:03.191 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:03.191 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:03.191 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:03.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.182+0000 7f6064ff9640 10 monclient: _finish_auth 0 2024-04-26T00:39:03.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.182+0000 7f6064ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:39:03.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.182+0000 7f6064ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:03.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.182+0000 7f6064ff9640 1 -- 172.21.15.37:0/3978074883 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6058019070 con 0x7f6068133b20 2024-04-26T00:39:03.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.182+0000 7f6064ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:03.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.182+0000 7f6064ff9640 1 -- 172.21.15.37:0/3978074883 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6058013b70 con 0x7f6068133b20 2024-04-26T00:39:03.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.182+0000 7f606e540640 5 monclient: authenticate success, global_id 204389 2024-04-26T00:39:03.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.182+0000 7f606e540640 10 monclient: _renew_subs 2024-04-26T00:39:03.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.182+0000 7f606e540640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:03.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.182+0000 7f606e540640 1 -- 172.21.15.37:0/3978074883 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60680704c0 con 0x7f6068133b20 2024-04-26T00:39:03.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.182+0000 7f60667fc640 4 set_mon_vals no callback set 2024-04-26T00:39:03.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.182+0000 7f6064ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:03.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.182+0000 7f6064ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:03.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.182+0000 7f6064ff9640 10 monclient: dump: 2024-04-26T00:39:03.192 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:03.192 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:03.192 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.192 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.192 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:03.192 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:03.192 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:03.192 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:03.192 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:03.192 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:03.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.182+0000 7f606e540640 10 monclient: _renew_subs 2024-04-26T00:39:03.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.182+0000 7f606e540640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:03.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.182+0000 7f606e540640 1 -- 172.21.15.37:0/3978074883 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f60681cb680 con 0x7f6068133b20 2024-04-26T00:39:03.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.182+0000 7f606e540640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:03.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.182+0000 7f606e540640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:03.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.182+0000 7f606e540640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:03.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.182+0000 7f606e540640 1 -- 172.21.15.37:0/3978074883 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6068066f40 con 0x7f6068133b20 2024-04-26T00:39:03.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.182+0000 7f6064ff9640 1 -- 172.21.15.37:0/3978074883 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f6058013d10 con 0x7f6068133b20 2024-04-26T00:39:03.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.182+0000 7f6064ff9640 1 --2- 172.21.15.37:0/3978074883 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6048035eb0 0x7f6048038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:03.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.182+0000 7f6064ff9640 1 -- 172.21.15.37:0/3978074883 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(754..754 src has 705..754) ==== 3975+0+0 (secure 0 0 0) 0x7f6058049370 con 0x7f6068133b20 2024-04-26T00:39:03.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.182+0000 7f60677fe640 1 --2- 172.21.15.37:0/3978074883 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6048035eb0 0x7f6048038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:03.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.182+0000 7f60677fe640 10 monclient: get_auth_request con 0x7f6048035eb0 auth_method 0 2024-04-26T00:39:03.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.186+0000 7f60677fe640 1 --2- 172.21.15.37:0/3978074883 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6048035eb0 0x7f6048038370 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f60681315e0 tx=0x7f6050009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:03.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.186+0000 7f6064ff9640 1 -- 172.21.15.37:0/3978074883 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f605801e840 con 0x7f6068133b20 2024-04-26T00:39:03.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.186+0000 7f6064ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:03.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.186+0000 7f6064ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:39:03.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.410+0000 7f606e540640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:39:03.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.410+0000 7f606e540640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:39:03.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.410+0000 7f606e540640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:03.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.410+0000 7f606e540640 1 -- 172.21.15.37:0/3978074883 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f606806b9a0 con 0x7f6068133b20 2024-04-26T00:39:03.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.410+0000 7f6064ff9640 1 -- 172.21.15.37:0/3978074883 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f60580490f0 con 0x7f6068133b20 2024-04-26T00:39:03.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.410+0000 7f6064ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:39:03.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.410+0000 7f6064ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:39:03.411 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":144,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:39:03.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.414+0000 7f606e540640 1 -- 172.21.15.37:0/3978074883 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6048035eb0 msgr2=0x7f6048038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:03.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.414+0000 7f606e540640 1 --2- 172.21.15.37:0/3978074883 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6048035eb0 0x7f6048038370 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f60681315e0 tx=0x7f6050009040 comp rx=0 tx=0).stop 2024-04-26T00:39:03.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.414+0000 7f606e540640 10 monclient: shutdown 2024-04-26T00:39:03.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.414+0000 7f606e540640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:03.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.414+0000 7f606e540640 1 -- 172.21.15.37:0/3978074883 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6068133b20 msgr2=0x7f60681ca800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:03.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.414+0000 7f606e540640 1 --2- 172.21.15.37:0/3978074883 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6068133b20 0x7f60681ca800 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6058007e60 tx=0x7f605800e4b0 comp rx=0 tx=0).stop 2024-04-26T00:39:03.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.414+0000 7f606e540640 1 -- 172.21.15.37:0/3978074883 shutdown_connections 2024-04-26T00:39:03.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.414+0000 7f606e540640 1 --2- 172.21.15.37:0/3978074883 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6048035eb0 0x7f6048038370 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:03.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.414+0000 7f606e540640 1 --2- 172.21.15.37:0/3978074883 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6068139050 0x7f606806e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:03.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.414+0000 7f606e540640 1 --2- 172.21.15.37:0/3978074883 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6068133f00 0x7f6068130870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:03.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.414+0000 7f606e540640 1 --2- 172.21.15.37:0/3978074883 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6068133b20 0x7f60681ca800 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:03.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.414+0000 7f606e540640 1 -- 172.21.15.37:0/3978074883 >> 172.21.15.37:0/3978074883 conn(0x7f60680a1ad0 msgr2=0x7f60680a34e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:03.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.414+0000 7f606e540640 1 -- 172.21.15.37:0/3978074883 shutdown_connections 2024-04-26T00:39:03.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.414+0000 7f606e540640 1 -- 172.21.15.37:0/3978074883 wait complete. 2024-04-26T00:39:03.429 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:39:03.429 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:39:03.430 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:39:03.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb70acec640 10 monclient: get_monmap_and_config 2024-04-26T00:39:03.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb70acec640 10 monclient: build_initial_monmap 2024-04-26T00:39:03.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb70acec640 10 monclient: monmap: 2024-04-26T00:39:03.557 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:03.557 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:03.557 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:39:03.560108+0000 2024-04-26T00:39:03.557 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:39:03.560108+0000 2024-04-26T00:39:03.557 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:03.557 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:03.557 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:39:03.557 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:39:03.557 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:39:03.557 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:39:03.557 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:39:03.557 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:03.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb70acec640 1 Processor -- start 2024-04-26T00:39:03.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb70acec640 1 -- start start 2024-04-26T00:39:03.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb70acec640 10 monclient: init 2024-04-26T00:39:03.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb70acec640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:03.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb70acec640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-04-26T00:39:03.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb70acec640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb704136f20 addr v1:172.21.15.45:6789/0 2024-04-26T00:39:03.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb70acec640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb704137a10 0x7fb70409bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:03.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb70acec640 10 monclient(hunting): picked mon.noname-b con 0x7fb704137a10 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:03.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb70acec640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fb70413c450 0x7fb70413e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:03.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb70acec640 10 monclient(hunting): picked mon.noname-c con 0x7fb70413c450 addr v2:172.21.15.45:3300/0 2024-04-26T00:39:03.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb70acec640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7040a50f0 con 0x7fb704136f20 2024-04-26T00:39:03.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb70acec640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:03.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb70acec640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fb70409b330 con 0x7fb70413c450 2024-04-26T00:39:03.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb70acec640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:03.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb70acec640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb70409b610 con 0x7fb704137a10 2024-04-26T00:39:03.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb70acec640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:03.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb70acec640 10 monclient(hunting): authenticate will time out at 2124.865776s 2024-04-26T00:39:03.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb703fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb704137a10 0x7fb70409bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:03.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb703fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb704137a10 0x7fb70409bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:35610/0 (socket says 172.21.15.37:35610) 2024-04-26T00:39:03.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb703fff640 1 -- 172.21.15.37:0/3031897918 learned_addr learned my addr 172.21.15.37:0/3031897918 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:03.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb703fff640 10 monclient(hunting): get_auth_request con 0x7fb704137a10 auth_method 0 2024-04-26T00:39:03.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb703fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:03.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb703fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:03.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb703fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:03.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb703fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:03.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb703fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:03.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb703fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:03.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb709289640 1 --2- 172.21.15.37:0/3031897918 >> v2:172.21.15.45:3300/0 conn(0x7fb70413c450 0x7fb70413e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:03.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb703fff640 10 monclient(hunting): handle_auth_done global_id 204392 payload 274 2024-04-26T00:39:03.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb703fff640 1 -- 172.21.15.37:0/3031897918 >> v2:172.21.15.45:3300/0 conn(0x7fb70413c450 msgr2=0x7fb70413e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:03.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb703fff640 1 --2- 172.21.15.37:0/3031897918 >> v2:172.21.15.45:3300/0 conn(0x7fb70413c450 0x7fb70413e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:03.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb703fff640 1 -- 172.21.15.37:0/3031897918 >> v1:172.21.15.45:6789/0 conn(0x7fb704136f20 legacy=0x7fb704137300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:03.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb703fff640 10 monclient: _finish_hunting 0 2024-04-26T00:39:03.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb703fff640 1 monclient: found mon.noname-b 2024-04-26T00:39:03.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb703fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:03.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb703fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:39:03.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb703fff640 1 -- 172.21.15.37:0/3031897918 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb70409b8f0 con 0x7fb704137a10 2024-04-26T00:39:03.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb709289640 10 monclient: get_auth_request con 0x7fb70413c450 auth_method 0 2024-04-26T00:39:03.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.558+0000 7fb709289640 1 --2- 172.21.15.37:0/3031897918 >> v2:172.21.15.45:3300/0 conn(0x7fb70413c450 0x7fb70413e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:39:03.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.562+0000 7fb703fff640 1 -- 172.21.15.37:0/3031897918 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb704137a10 msgr2=0x7fb70409bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:03.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.562+0000 7fb703fff640 1 --2- 172.21.15.37:0/3031897918 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb704137a10 0x7fb70409bff0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb704137e90 tx=0x7fb6ec01c430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22223b5962476ce server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:03.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.562+0000 7fb7037fe640 1 -- 172.21.15.37:0/3031897918 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec0b0070 con 0x7fb704137a10 2024-04-26T00:39:03.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.562+0000 7fb7037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:03.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.562+0000 7fb7037fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:39:03.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.562+0000 7fb7037fe640 10 monclient: dump: 2024-04-26T00:39:03.568 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:03.568 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:03.568 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.568 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.568 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:03.568 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:03.568 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:03.568 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:03.568 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:03.568 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:03.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.562+0000 7fb7037fe640 10 monclient: _finish_auth 0 2024-04-26T00:39:03.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.562+0000 7fb7037fe640 10 monclient: _check_auth_tickets 2024-04-26T00:39:03.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.562+0000 7fb7037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:03.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.562+0000 7fb7037fe640 1 -- 172.21.15.37:0/3031897918 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6ec0ab600 con 0x7fb704137a10 2024-04-26T00:39:03.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.562+0000 7fb7037fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:03.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.562+0000 7fb7037fe640 1 -- 172.21.15.37:0/3031897918 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec0ab9d0 con 0x7fb704137a10 2024-04-26T00:39:03.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.562+0000 7fb70acec640 5 monclient: authenticate success, global_id 204392 2024-04-26T00:39:03.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.562+0000 7fb70acec640 10 monclient: get_monmap_and_config success 2024-04-26T00:39:03.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.562+0000 7fb70acec640 4 set_mon_vals no callback set 2024-04-26T00:39:03.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.562+0000 7fb7037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:03.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.562+0000 7fb7037fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:03.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.562+0000 7fb7037fe640 10 monclient: dump: 2024-04-26T00:39:03.569 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:03.569 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:03.569 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.569 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.570 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:03.570 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:03.570 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:03.570 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:03.570 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:03.570 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:03.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.562+0000 7fb70acec640 10 monclient: shutdown 2024-04-26T00:39:03.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.562+0000 7fb70acec640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:03.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.562+0000 7fb70acec640 1 -- 172.21.15.37:0/3031897918 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb704137a10 msgr2=0x7fb70409bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:03.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.562+0000 7fb70acec640 1 --2- 172.21.15.37:0/3031897918 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb704137a10 0x7fb70409bff0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb704137e90 tx=0x7fb6ec01c430 comp rx=0 tx=0).stop 2024-04-26T00:39:03.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.562+0000 7fb70acec640 1 -- 172.21.15.37:0/3031897918 shutdown_connections 2024-04-26T00:39:03.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.562+0000 7fb70acec640 1 --2- 172.21.15.37:0/3031897918 >> v2:172.21.15.45:3300/0 conn(0x7fb70413c450 0x7fb70413e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:03.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.562+0000 7fb70acec640 1 --2- 172.21.15.37:0/3031897918 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb704137a10 0x7fb70409bff0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:03.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.562+0000 7fb70acec640 1 -- 172.21.15.37:0/3031897918 >> 172.21.15.37:0/3031897918 conn(0x7fb7040a61b0 msgr2=0x7fb7040a65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:03.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.562+0000 7fb70acec640 1 -- 172.21.15.37:0/3031897918 shutdown_connections 2024-04-26T00:39:03.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.562+0000 7fb70acec640 1 -- 172.21.15.37:0/3031897918 wait complete. 2024-04-26T00:39:03.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb70acec640 10 monclient: build_initial_monmap 2024-04-26T00:39:03.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb70acec640 10 monclient: monmap: 2024-04-26T00:39:03.571 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:03.571 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:39:03.571 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:39:03.571 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:39:03.571 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:03.571 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:39:03.571 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:39:03.571 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:39:03.571 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:39:03.571 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:03.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb70acec640 1 Processor -- start 2024-04-26T00:39:03.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb70acec640 1 -- start start 2024-04-26T00:39:03.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb70acec640 10 monclient: init 2024-04-26T00:39:03.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb70acec640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:03.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb70acec640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:39:03.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb70acec640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb704136f20 0x7fb704138600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:03.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb70acec640 10 monclient(hunting): picked mon.noname-a con 0x7fb704136f20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:39:03.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb70acec640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb704137a10 0x7fb70413a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:03.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb70acec640 10 monclient(hunting): picked mon.noname-c con 0x7fb704137a10 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:03.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb70acec640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb70413c450 0x7fb7041351a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:03.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb70acec640 10 monclient(hunting): picked mon.noname-b con 0x7fb70413c450 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:39:03.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb70acec640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:03.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb70acec640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb704141930 con 0x7fb704136f20 2024-04-26T00:39:03.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb70acec640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:03.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb70acec640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb7041417b0 con 0x7fb70413c450 2024-04-26T00:39:03.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb70acec640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:03.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb70acec640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb704141ab0 con 0x7fb704137a10 2024-04-26T00:39:03.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb70acec640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:03.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb70acec640 10 monclient(hunting): authenticate will time out at 2124.873124s 2024-04-26T00:39:03.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb708a88640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb704136f20 0x7fb704138600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:03.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb703fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb704137a10 0x7fb70413a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:03.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb708a88640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb704136f20 0x7fb704138600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51944/0 (socket says 172.21.15.37:51944) 2024-04-26T00:39:03.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb703fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb704137a10 0x7fb70413a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:35622/0 (socket says 172.21.15.37:35622) 2024-04-26T00:39:03.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb708a88640 1 -- 172.21.15.37:0/1319340777 learned_addr learned my addr 172.21.15.37:0/1319340777 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:03.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb708a88640 10 monclient(hunting): get_auth_request con 0x7fb704136f20 auth_method 0 2024-04-26T00:39:03.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb708a88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:03.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb708a88640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:03.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb708a88640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:03.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb703fff640 10 monclient(hunting): get_auth_request con 0x7fb704137a10 auth_method 0 2024-04-26T00:39:03.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb703fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:03.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb703fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:03.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb703fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:03.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb703fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:03.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb703fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:03.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb709289640 1 --2- 172.21.15.37:0/1319340777 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb70413c450 0x7fb7041351a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:03.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb703fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:03.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb708a88640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:03.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb708a88640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:03.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb708a88640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:03.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb709289640 10 monclient(hunting): get_auth_request con 0x7fb70413c450 auth_method 0 2024-04-26T00:39:03.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb709289640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:03.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb709289640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:03.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb709289640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:03.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb709289640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:03.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb709289640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:03.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb709289640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:03.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb703fff640 10 monclient(hunting): handle_auth_done global_id 204395 payload 824 2024-04-26T00:39:03.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb703fff640 1 -- 172.21.15.37:0/1319340777 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb70413c450 msgr2=0x7fb7041351a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:03.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb703fff640 1 --2- 172.21.15.37:0/1319340777 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb70413c450 0x7fb7041351a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:03.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb703fff640 1 -- 172.21.15.37:0/1319340777 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb704136f20 msgr2=0x7fb704138600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:03.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb703fff640 1 --2- 172.21.15.37:0/1319340777 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb704136f20 0x7fb704138600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:03.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb703fff640 10 monclient: _finish_hunting 0 2024-04-26T00:39:03.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb703fff640 1 monclient: found mon.noname-c 2024-04-26T00:39:03.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb703fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:03.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb703fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:39:03.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb703fff640 1 -- 172.21.15.37:0/1319340777 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb70413b710 con 0x7fb704137a10 2024-04-26T00:39:03.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.566+0000 7fb708a88640 1 --2- 172.21.15.37:0/1319340777 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb704136f20 0x7fb704138600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:39:03.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb703fff640 1 -- 172.21.15.37:0/1319340777 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb704137a10 msgr2=0x7fb70413a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:03.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb703fff640 1 --2- 172.21.15.37:0/1319340777 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb704137a10 0x7fb70413a740 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb70409b790 tx=0x7fb6ec0191d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:03.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb7017fa640 1 -- 172.21.15.37:0/1319340777 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec0b0070 con 0x7fb704137a10 2024-04-26T00:39:03.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb7017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:03.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb7017fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:39:03.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb7017fa640 10 monclient: dump: 2024-04-26T00:39:03.579 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:03.579 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:03.579 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.579 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.579 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:03.580 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:03.580 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:03.580 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:03.580 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:03.580 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:03.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb7017fa640 10 monclient: _finish_auth 0 2024-04-26T00:39:03.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb7017fa640 10 monclient: _check_auth_tickets 2024-04-26T00:39:03.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb7017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:03.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb7017fa640 1 -- 172.21.15.37:0/1319340777 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6ec0b1040 con 0x7fb704137a10 2024-04-26T00:39:03.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb70acec640 5 monclient: authenticate success, global_id 204395 2024-04-26T00:39:03.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb70acec640 10 monclient: _renew_subs 2024-04-26T00:39:03.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb70acec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:03.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb70acec640 1 -- 172.21.15.37:0/1319340777 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb70413b980 con 0x7fb704137a10 2024-04-26T00:39:03.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb7017fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:03.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb7017fa640 1 -- 172.21.15.37:0/1319340777 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec0aa9e0 con 0x7fb704137a10 2024-04-26T00:39:03.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb7017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:03.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb7017fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:03.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb7017fa640 10 monclient: dump: 2024-04-26T00:39:03.581 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:03.581 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:03.581 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.581 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.581 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:03.581 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:03.581 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:03.581 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:03.581 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:03.581 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:03.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb70acec640 10 monclient: _renew_subs 2024-04-26T00:39:03.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb702ffd640 4 set_mon_vals no callback set 2024-04-26T00:39:03.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb70acec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:03.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb70acec640 1 -- 172.21.15.37:0/1319340777 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb70411fb30 con 0x7fb704137a10 2024-04-26T00:39:03.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb70acec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:03.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb70acec640 1 --2- 172.21.15.37:0/1319340777 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb7041356e0 0x7fb7041cf700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:03.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb70acec640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:03.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb70acec640 1 -- 172.21.15.37:0/1319340777 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb70404f510 con 0x7fb7041356e0 2024-04-26T00:39:03.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb708a88640 1 --2- 172.21.15.37:0/1319340777 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb7041356e0 0x7fb7041cf700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:03.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb708a88640 10 monclient: get_auth_request con 0x7fb7041356e0 auth_method 0 2024-04-26T00:39:03.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb708a88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:03.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb708a88640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:03.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb708a88640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:03.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb708a88640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:03.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb708a88640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:03.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb708a88640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:03.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb7017fa640 1 -- 172.21.15.37:0/1319340777 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb6ec0aab80 con 0x7fb704137a10 2024-04-26T00:39:03.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb708a88640 10 monclient(hunting): handle_auth_done global_id 234108 payload 824 2024-04-26T00:39:03.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb708a88640 1 -- 172.21.15.37:0/1319340777 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb7040a3910 con 0x7fb7041356e0 2024-04-26T00:39:03.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb7017fa640 1 --2- 172.21.15.37:0/1319340777 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb6e4035ed0 0x7fb6e4038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:03.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb7017fa640 1 -- 172.21.15.37:0/1319340777 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(754..754 src has 705..754) ==== 3975+0+0 (secure 0 0 0) 0x7fb6ec0e2910 con 0x7fb704137a10 2024-04-26T00:39:03.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb709289640 1 --2- 172.21.15.37:0/1319340777 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb6e4035ed0 0x7fb6e4038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:03.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb709289640 10 monclient: get_auth_request con 0x7fb6e4035ed0 auth_method 0 2024-04-26T00:39:03.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb708a88640 1 -- 172.21.15.37:0/1319340777 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb7041356e0 msgr2=0x7fb7041cf700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:03.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb708a88640 1 --2- 172.21.15.37:0/1319340777 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb7041356e0 0x7fb7041cf700 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb6f400fb10 tx=0x7fb6f40045a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:03.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb7017fa640 1 -- 172.21.15.37:0/1319340777 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f4018030 con 0x7fb7041356e0 2024-04-26T00:39:03.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb7017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:03.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb7017fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:39:03.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.570+0000 7fb7017fa640 10 monclient: dump: 2024-04-26T00:39:03.584 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:03.584 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:03.584 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.584 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.584 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:03.584 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:03.584 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:03.584 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:03.584 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:03.584 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:03.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.574+0000 7fb7017fa640 1 -- 172.21.15.37:0/1319340777 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb6f400de50 con 0x7fb7041356e0 2024-04-26T00:39:03.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.574+0000 7fb7017fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:03.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.574+0000 7fb7017fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:39:03.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.574+0000 7fb7017fa640 1 -- 172.21.15.37:0/1319340777 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb7041356e0 msgr2=0x7fb7041cf700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:03.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.574+0000 7fb7017fa640 1 --2- 172.21.15.37:0/1319340777 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb7041356e0 0x7fb7041cf700 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb6f400fb10 tx=0x7fb6f40045a0 comp rx=0 tx=0).stop 2024-04-26T00:39:03.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.574+0000 7fb7017fa640 1 -- 172.21.15.37:0/1319340777 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb7041356e0 msgr2=0x7fb7041cf700 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:39:03.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.574+0000 7fb7017fa640 1 --2- 172.21.15.37:0/1319340777 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb7041356e0 0x7fb7041cf700 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb6f400fb10 tx=0x7fb6f40045a0 comp rx=0 tx=0).stop 2024-04-26T00:39:03.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.574+0000 7fb709289640 1 --2- 172.21.15.37:0/1319340777 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb6e4035ed0 0x7fb6e4038390 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fb704139dd0 tx=0x7fb6fc007780 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:03.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.578+0000 7fb70acec640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:39:03.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.578+0000 7fb70acec640 1 --2- 172.21.15.37:0/1319340777 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb70402c000 0x7fb70402c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:03.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.578+0000 7fb70acec640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:03.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.578+0000 7fb70acec640 1 -- 172.21.15.37:0/1319340777 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb7040a3000 con 0x7fb70402c000 2024-04-26T00:39:03.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.578+0000 7fb708a88640 1 --2- 172.21.15.37:0/1319340777 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb70402c000 0x7fb70402c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:03.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.578+0000 7fb708a88640 10 monclient: get_auth_request con 0x7fb70402c000 auth_method 0 2024-04-26T00:39:03.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.578+0000 7fb708a88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:03.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.578+0000 7fb708a88640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:03.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.578+0000 7fb708a88640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:03.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.578+0000 7fb708a88640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:03.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.578+0000 7fb708a88640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:03.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.578+0000 7fb708a88640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:03.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.578+0000 7fb708a88640 10 monclient(hunting): handle_auth_done global_id 234111 payload 824 2024-04-26T00:39:03.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.578+0000 7fb708a88640 1 -- 172.21.15.37:0/1319340777 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb70413ac80 con 0x7fb70402c000 2024-04-26T00:39:03.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.582+0000 7fb708a88640 1 -- 172.21.15.37:0/1319340777 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb70402c000 msgr2=0x7fb70402c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:03.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.582+0000 7fb708a88640 1 --2- 172.21.15.37:0/1319340777 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb70402c000 0x7fb70402c410 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb6f4018000 tx=0x7fb6f4007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:03.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.582+0000 7fb7017fa640 1 -- 172.21.15.37:0/1319340777 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f4004b60 con 0x7fb70402c000 2024-04-26T00:39:03.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.582+0000 7fb7017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:03.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.582+0000 7fb7017fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:39:03.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.582+0000 7fb7017fa640 10 monclient: dump: 2024-04-26T00:39:03.587 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:03.587 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:03.587 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.588 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.588 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:03.588 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:03.588 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:03.588 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:03.588 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:03.588 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:03.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.582+0000 7fb7017fa640 1 -- 172.21.15.37:0/1319340777 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4549 (secure 0 0 0) 0x7fb6f4004b60 con 0x7fb70402c000 2024-04-26T00:39:03.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.582+0000 7fb7017fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:39:03.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.582+0000 7fb7017fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:39:03.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.582+0000 7fb7017fa640 1 -- 172.21.15.37:0/1319340777 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb70402c000 msgr2=0x7fb70402c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:03.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.582+0000 7fb7017fa640 1 --2- 172.21.15.37:0/1319340777 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb70402c000 0x7fb70402c410 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb6f4018000 tx=0x7fb6f4007f90 comp rx=0 tx=0).stop 2024-04-26T00:39:03.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.582+0000 7fb7017fa640 1 -- 172.21.15.37:0/1319340777 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb70402c000 msgr2=0x7fb70402c410 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:39:03.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.582+0000 7fb7017fa640 1 --2- 172.21.15.37:0/1319340777 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb70402c000 0x7fb70402c410 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb6f4018000 tx=0x7fb6f4007f90 comp rx=0 tx=0).stop 2024-04-26T00:39:03.589 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:39:03.589 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:39:03.589 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:39:03.589 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:39:03.589 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 144, 2024-04-26T00:39:03.589 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 14043, 2024-04-26T00:39:03.589 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:39:03.589 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:39:03.589 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:39:03.589 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:39:03.589 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:03.590 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:39:03.590 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:39:03.590 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:39:03.590 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:39:03.590 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:03.590 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:03.590 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:03.590 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:03.590 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:03.590 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:03.590 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:03.590 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:03.590 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:03.591 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:03.591 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:03.591 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:03.591 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:39:03.591 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:39:03.591 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:03.591 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:03.591 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:03.591 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:03.591 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:03.591 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:03.591 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:03.591 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:03.591 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:03.592 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:03.592 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:03.592 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:03.592 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:03.592 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:39:03.592 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:39:03.592 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:39:03.592 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:39:03.592 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:39:03.592 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:39:03.592 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:39:03.592 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:39:03.592 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:39:03.592 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:39:03.592 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:39:03.593 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:39:03.593 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:39:03.593 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:39:03.593 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:39:03.593 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:39:03.593 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:39:03.593 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:03.593 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:03.593 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:03.593 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:03.593 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:03.593 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:03.593 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:03.593 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:03.593 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:03.593 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:03.594 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:03.594 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:03.594 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:39:03.594 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:03.594 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:39:03.594 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:03.594 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:39:03.594 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:39:03.594 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:03.594 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:03.594 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:03.594 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:03.594 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:39:03.595 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:03.595 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:03.595 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:03.595 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:03.595 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:39:03.595 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:03.595 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:03.595 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:03.595 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:03.595 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:39:03.595 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:39:03.595 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:03.595 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:03.595 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:03.595 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:03.596 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:03.596 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:39:03.596 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:39:03.596 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:03.596 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:03.596 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:03.596 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:03.596 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:39:03.596 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:03.596 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:03.596 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:03.596 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:03.596 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:39:03.596 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:03.596 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:03.597 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:03.597 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:03.597 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:39:03.597 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:39:03.597 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:03.597 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:03.597 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:03.597 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:03.597 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:03.597 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:39:03.597 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:39:03.597 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:03.597 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:03.597 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:03.598 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:03.598 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:39:03.598 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:03.598 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:03.598 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:03.598 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:03.598 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:39:03.598 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:03.598 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:03.598 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:03.598 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:03.598 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:39:03.598 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:39:03.598 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:03.599 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:03.599 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:03.599 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:03.599 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:03.599 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:03.599 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:39:03.599 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:39:03.599 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:03.599 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:03.599 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:03.599 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:39:03.599 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:03.599 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:03.599 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:39:03.600 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:03.600 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:03.600 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:03.600 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:39:03.600 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:03.600 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:03.600 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:03.600 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:39:03.600 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:39:03.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.582+0000 7fb70acec640 1 -- 172.21.15.37:0/1319340777 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb6e4035ed0 msgr2=0x7fb6e4038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:03.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.582+0000 7fb70acec640 1 --2- 172.21.15.37:0/1319340777 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb6e4035ed0 0x7fb6e4038390 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fb704139dd0 tx=0x7fb6fc007780 comp rx=0 tx=0).stop 2024-04-26T00:39:03.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.582+0000 7fb70acec640 10 monclient: shutdown 2024-04-26T00:39:03.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.582+0000 7fb70acec640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:03.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.582+0000 7fb70acec640 1 -- 172.21.15.37:0/1319340777 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb704137a10 msgr2=0x7fb70413a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:03.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.582+0000 7fb70acec640 1 --2- 172.21.15.37:0/1319340777 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb704137a10 0x7fb70413a740 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb70409b790 tx=0x7fb6ec0191d0 comp rx=0 tx=0).stop 2024-04-26T00:39:03.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.582+0000 7fb709289640 1 -- 172.21.15.37:0/1319340777 reap_dead start 2024-04-26T00:39:03.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.582+0000 7fb709289640 1 -- 172.21.15.37:0/1319340777 reap_dead start 2024-04-26T00:39:03.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.582+0000 7fb70acec640 1 -- 172.21.15.37:0/1319340777 shutdown_connections 2024-04-26T00:39:03.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.582+0000 7fb70acec640 1 -- 172.21.15.37:0/1319340777 >> 172.21.15.37:0/1319340777 conn(0x7fb7040a61b0 msgr2=0x7fb7040a81a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:03.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.582+0000 7fb70acec640 1 -- 172.21.15.37:0/1319340777 shutdown_connections 2024-04-26T00:39:03.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.582+0000 7fb70acec640 1 -- 172.21.15.37:0/1319340777 wait complete. 2024-04-26T00:39:03.603 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:39:03.603 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:39:03.603 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:39:03.696 INFO:tasks.rados.rados.0.smithi045.stdout:3336: done (3 left) 2024-04-26T00:39:03.697 INFO:tasks.rados.rados.0.smithi045.stdout:3337: done (2 left) 2024-04-26T00:39:03.697 INFO:tasks.rados.rados.0.smithi045.stdout:3338: done (1 left) 2024-04-26T00:39:03.697 INFO:tasks.rados.rados.0.smithi045.stdout:3339: done (0 left) 2024-04-26T00:39:03.697 INFO:tasks.rados.rados.0.smithi045.stdout:3340: read oid 7 snap -1 2024-04-26T00:39:03.697 INFO:tasks.rados.rados.0.smithi045.stdout:3340: expect (ObjNum 1187 snap 369 seq_num 1187) 2024-04-26T00:39:03.697 INFO:tasks.rados.rados.0.smithi045.stdout:3341: write oid 32 current snap is 370 2024-04-26T00:39:03.697 INFO:tasks.rados.rados.0.smithi045.stdout:3341: seq_num 1190 ranges {400330=497378,1485925=514405} 2024-04-26T00:39:03.704 INFO:tasks.rados.rados.0.smithi045.stdout:3341: writing smithi04526579-32 from 400330 to 897708 tid 1 2024-04-26T00:39:03.714 INFO:tasks.rados.rados.0.smithi045.stdout:3341: writing smithi04526579-32 from 1485925 to 2000330 tid 2 2024-04-26T00:39:03.714 INFO:tasks.rados.rados.0.smithi045.stdout:3342: read oid 25 snap -1 2024-04-26T00:39:03.714 INFO:tasks.rados.rados.0.smithi045.stdout:3342: expect deleted 2024-04-26T00:39:03.714 INFO:tasks.rados.rados.0.smithi045.stdout:3343: write (excl) oid 9 current snap is 370 2024-04-26T00:39:03.714 INFO:tasks.rados.rados.0.smithi045.stdout:3343: seq_num 1191 ranges {417137=572627,1428054=589083} 2024-04-26T00:39:03.722 INFO:tasks.rados.rados.0.smithi045.stdout:3343: writing smithi04526579-9 from 417137 to 989764 tid 1 2024-04-26T00:39:03.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.722+0000 7ffbbc60c640 10 monclient: get_monmap_and_config 2024-04-26T00:39:03.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.722+0000 7ffbbc60c640 10 monclient: build_initial_monmap 2024-04-26T00:39:03.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.722+0000 7ffbbc60c640 10 monclient: monmap: 2024-04-26T00:39:03.724 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:03.724 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:03.724 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:39:03.727152+0000 2024-04-26T00:39:03.724 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:39:03.727152+0000 2024-04-26T00:39:03.724 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:03.724 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:03.724 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:39:03.724 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:39:03.724 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:39:03.724 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:39:03.724 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:39:03.724 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:03.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbbc60c640 1 Processor -- start 2024-04-26T00:39:03.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbbc60c640 1 -- start start 2024-04-26T00:39:03.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbbc60c640 10 monclient: init 2024-04-26T00:39:03.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbbc60c640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:03.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbbc60c640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-04-26T00:39:03.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbbc60c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb4134c30 0x7ffbb4135010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:03.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbbc60c640 10 monclient(hunting): picked mon.noname-b con 0x7ffbb4134c30 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:03.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbbc60c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffbb40726c0 addr v1:172.21.15.45:6789/0 2024-04-26T00:39:03.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbbc60c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ffbb40a1d00 0x7ffbb4135550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:03.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbbc60c640 10 monclient(hunting): picked mon.noname-c con 0x7ffbb40a1d00 addr v2:172.21.15.45:3300/0 2024-04-26T00:39:03.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbbc60c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbb40a37a0 con 0x7ffbb40726c0 2024-04-26T00:39:03.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbbc60c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:03.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbbc60c640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7ffbb40707a0 con 0x7ffbb40a1d00 2024-04-26T00:39:03.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbbc60c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:03.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbbc60c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ffbb4135a90 con 0x7ffbb4134c30 2024-04-26T00:39:03.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbbc60c640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:03.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbbc60c640 10 monclient(hunting): authenticate will time out at 2125.033369s 2024-04-26T00:39:03.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbba3a8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb4134c30 0x7ffbb4135010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:03.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbba3a8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb4134c30 0x7ffbb4135010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:35634/0 (socket says 172.21.15.37:35634) 2024-04-26T00:39:03.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbba3a8640 1 -- 172.21.15.37:0/3488736471 learned_addr learned my addr 172.21.15.37:0/3488736471 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:03.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbba3a8640 10 monclient(hunting): get_auth_request con 0x7ffbb4134c30 auth_method 0 2024-04-26T00:39:03.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbba3a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:03.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbba3a8640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:03.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbba3a8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:03.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbba3a8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:03.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbba3a8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:03.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbba3a8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:03.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbbaba9640 1 --2- 172.21.15.37:0/3488736471 >> v2:172.21.15.45:3300/0 conn(0x7ffbb40a1d00 0x7ffbb4135550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:03.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbbaba9640 10 monclient(hunting): get_auth_request con 0x7ffbb40a1d00 auth_method 0 2024-04-26T00:39:03.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbbaba9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:03.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbbaba9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:03.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbbaba9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:03.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbba3a8640 10 monclient(hunting): handle_auth_done global_id 204398 payload 274 2024-04-26T00:39:03.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbba3a8640 1 -- 172.21.15.37:0/3488736471 >> v2:172.21.15.45:3300/0 conn(0x7ffbb40a1d00 msgr2=0x7ffbb4135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:03.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbba3a8640 1 --2- 172.21.15.37:0/3488736471 >> v2:172.21.15.45:3300/0 conn(0x7ffbb40a1d00 0x7ffbb4135550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:03.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbba3a8640 1 -- 172.21.15.37:0/3488736471 >> v1:172.21.15.45:6789/0 conn(0x7ffbb40726c0 legacy=0x7ffbb4131a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:03.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbba3a8640 10 monclient: _finish_hunting 0 2024-04-26T00:39:03.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbba3a8640 1 monclient: found mon.noname-b 2024-04-26T00:39:03.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbba3a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:03.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbba3a8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:39:03.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.726+0000 7ffbba3a8640 1 -- 172.21.15.37:0/3488736471 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbb4071420 con 0x7ffbb4134c30 2024-04-26T00:39:03.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.730+0000 7ffbba3a8640 1 -- 172.21.15.37:0/3488736471 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb4134c30 msgr2=0x7ffbb4135010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:03.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.730+0000 7ffbba3a8640 1 --2- 172.21.15.37:0/3488736471 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb4134c30 0x7ffbb4135010 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ffba4009870 tx=0x7ffba401c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=653706cc6a640b08 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:03.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.730+0000 7ffbb93a6640 1 -- 172.21.15.37:0/3488736471 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba40b0070 con 0x7ffbb4134c30 2024-04-26T00:39:03.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.730+0000 7ffbb93a6640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:03.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.730+0000 7ffbb93a6640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:39:03.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.730+0000 7ffbb93a6640 10 monclient: dump: 2024-04-26T00:39:03.731 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:03.731 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:03.731 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.732 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.732 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:03.732 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:03.732 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:03.732 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:03.732 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:03.732 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:03.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.730+0000 7ffbb93a6640 10 monclient: _finish_auth 0 2024-04-26T00:39:03.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.730+0000 7ffbb93a6640 10 monclient: _check_auth_tickets 2024-04-26T00:39:03.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.730+0000 7ffbb93a6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:03.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.730+0000 7ffbb93a6640 1 -- 172.21.15.37:0/3488736471 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffba40ab600 con 0x7ffbb4134c30 2024-04-26T00:39:03.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.730+0000 7ffbb93a6640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:03.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.730+0000 7ffbb93a6640 1 -- 172.21.15.37:0/3488736471 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba40ab9d0 con 0x7ffbb4134c30 2024-04-26T00:39:03.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.730+0000 7ffbbc60c640 5 monclient: authenticate success, global_id 204398 2024-04-26T00:39:03.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.730+0000 7ffbbc60c640 10 monclient: get_monmap_and_config success 2024-04-26T00:39:03.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.730+0000 7ffbbc60c640 4 set_mon_vals no callback set 2024-04-26T00:39:03.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.730+0000 7ffbb93a6640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:03.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.730+0000 7ffbb93a6640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:03.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.730+0000 7ffbb93a6640 10 monclient: dump: 2024-04-26T00:39:03.733 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:03.733 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:03.733 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.733 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.733 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:03.733 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:03.733 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:03.733 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:03.733 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:03.734 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:03.734 INFO:tasks.rados.rados.0.smithi045.stdout:3343: writing smithi04526579-9 from 1428054 to 2017137 tid 2 2024-04-26T00:39:03.734 INFO:tasks.rados.rados.0.smithi045.stdout:3344: write oid 4 current snap is 370 2024-04-26T00:39:03.735 INFO:tasks.rados.rados.0.smithi045.stdout:3344: seq_num 1192 ranges {433944=647876,1486536=547408} 2024-04-26T00:39:03.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.730+0000 7ffbbc60c640 10 monclient: shutdown 2024-04-26T00:39:03.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.730+0000 7ffbbc60c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:03.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.730+0000 7ffbbc60c640 1 -- 172.21.15.37:0/3488736471 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb4134c30 msgr2=0x7ffbb4135010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:03.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.730+0000 7ffbbc60c640 1 --2- 172.21.15.37:0/3488736471 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb4134c30 0x7ffbb4135010 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ffba4009870 tx=0x7ffba401c3e0 comp rx=0 tx=0).stop 2024-04-26T00:39:03.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.730+0000 7ffbbc60c640 1 -- 172.21.15.37:0/3488736471 shutdown_connections 2024-04-26T00:39:03.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.730+0000 7ffbbc60c640 1 --2- 172.21.15.37:0/3488736471 >> v2:172.21.15.45:3300/0 conn(0x7ffbb40a1d00 0x7ffbb4135550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:03.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.730+0000 7ffbbc60c640 1 --2- 172.21.15.37:0/3488736471 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb4134c30 0x7ffbb4135010 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:03.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.730+0000 7ffbbc60c640 1 -- 172.21.15.37:0/3488736471 >> 172.21.15.37:0/3488736471 conn(0x7ffbb40a7300 msgr2=0x7ffbb40a76e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:03.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.730+0000 7ffbbc60c640 1 -- 172.21.15.37:0/3488736471 shutdown_connections 2024-04-26T00:39:03.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.730+0000 7ffbbc60c640 1 -- 172.21.15.37:0/3488736471 wait complete. 2024-04-26T00:39:03.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbbc60c640 10 monclient: build_initial_monmap 2024-04-26T00:39:03.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbbc60c640 10 monclient: monmap: 2024-04-26T00:39:03.740 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:03.740 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:39:03.740 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:39:03.740 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:39:03.740 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:03.741 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:39:03.741 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:39:03.741 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:39:03.741 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:39:03.741 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:03.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbbc60c640 1 Processor -- start 2024-04-26T00:39:03.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbbc60c640 1 -- start start 2024-04-26T00:39:03.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbbc60c640 10 monclient: init 2024-04-26T00:39:03.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbbc60c640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:03.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbbc60c640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:39:03.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbbc60c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffbb40726c0 0x7ffbb40a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:03.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbbc60c640 10 monclient(hunting): picked mon.noname-b con 0x7ffbb40726c0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:39:03.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbbc60c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb40a1d00 0x7ffbb412fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:03.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbbc60c640 10 monclient(hunting): picked mon.noname-c con 0x7ffbb40a1d00 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:03.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbbc60c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffbb4134c30 0x7ffbb4130370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:03.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbbc60c640 10 monclient(hunting): picked mon.noname-a con 0x7ffbb4134c30 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:39:03.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbbc60c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:03.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbbc60c640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb413fde0 con 0x7ffbb4134c30 2024-04-26T00:39:03.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbbc60c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:03.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbbc60c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb413fc60 con 0x7ffbb40726c0 2024-04-26T00:39:03.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbbc60c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:03.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbbc60c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ffbb413ff60 con 0x7ffbb40a1d00 2024-04-26T00:39:03.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbbc60c640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:03.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbbc60c640 10 monclient(hunting): authenticate will time out at 2125.041788s 2024-04-26T00:39:03.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbb9ba7640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb40a1d00 0x7ffbb412fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:03.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbb9ba7640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb40a1d00 0x7ffbb412fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:35636/0 (socket says 172.21.15.37:35636) 2024-04-26T00:39:03.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbb9ba7640 1 -- 172.21.15.37:0/2483037277 learned_addr learned my addr 172.21.15.37:0/2483037277 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:03.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbb9ba7640 10 monclient(hunting): get_auth_request con 0x7ffbb40a1d00 auth_method 0 2024-04-26T00:39:03.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbb9ba7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:03.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbb9ba7640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:03.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbb9ba7640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:03.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbbaba9640 1 --2- 172.21.15.37:0/2483037277 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffbb4134c30 0x7ffbb4130370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:03.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbba3a8640 1 --2- 172.21.15.37:0/2483037277 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffbb40726c0 0x7ffbb40a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:03.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbbaba9640 10 monclient(hunting): get_auth_request con 0x7ffbb4134c30 auth_method 0 2024-04-26T00:39:03.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbbaba9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:03.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbbaba9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:03.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbbaba9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:03.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbba3a8640 10 monclient(hunting): get_auth_request con 0x7ffbb40726c0 auth_method 0 2024-04-26T00:39:03.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbba3a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:03.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbba3a8640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:03.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbba3a8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:03.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbb9ba7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:03.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbb9ba7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:03.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbb9ba7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:03.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbbaba9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:03.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbbaba9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:03.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbbaba9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:03.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbb9ba7640 10 monclient(hunting): handle_auth_done global_id 204401 payload 824 2024-04-26T00:39:03.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbb9ba7640 1 -- 172.21.15.37:0/2483037277 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffbb40726c0 msgr2=0x7ffbb40a0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:03.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbb9ba7640 1 --2- 172.21.15.37:0/2483037277 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffbb40726c0 0x7ffbb40a0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:03.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbb9ba7640 1 -- 172.21.15.37:0/2483037277 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffbb4134c30 msgr2=0x7ffbb4130370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:03.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbb9ba7640 1 --2- 172.21.15.37:0/2483037277 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffbb4134c30 0x7ffbb4130370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:03.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbb9ba7640 10 monclient: _finish_hunting 0 2024-04-26T00:39:03.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbb9ba7640 1 monclient: found mon.noname-c 2024-04-26T00:39:03.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbb9ba7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:03.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbb9ba7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:39:03.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.734+0000 7ffbb9ba7640 1 -- 172.21.15.37:0/2483037277 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbb409de40 con 0x7ffbb40a1d00 2024-04-26T00:39:03.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbb9ba7640 1 -- 172.21.15.37:0/2483037277 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb40a1d00 msgr2=0x7ffbb412fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:03.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbb9ba7640 1 --2- 172.21.15.37:0/2483037277 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb40a1d00 0x7ffbb412fe30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ffbb000c480 tx=0x7ffbb000c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:03.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbaaffd640 1 -- 172.21.15.37:0/2483037277 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb0012a40 con 0x7ffbb40a1d00 2024-04-26T00:39:03.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbaaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:03.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbaaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:39:03.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbaaffd640 10 monclient: dump: 2024-04-26T00:39:03.745 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:03.745 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:03.745 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.745 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.745 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:03.745 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:03.745 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:03.745 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:03.745 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:03.746 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:03.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbaaffd640 10 monclient: _finish_auth 0 2024-04-26T00:39:03.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbaaffd640 10 monclient: _check_auth_tickets 2024-04-26T00:39:03.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbaaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:03.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbaaffd640 1 -- 172.21.15.37:0/2483037277 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbb0017070 con 0x7ffbb40a1d00 2024-04-26T00:39:03.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbbc60c640 5 monclient: authenticate success, global_id 204401 2024-04-26T00:39:03.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbbc60c640 10 monclient: _renew_subs 2024-04-26T00:39:03.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbbc60c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:03.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbbc60c640 1 -- 172.21.15.37:0/2483037277 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbb41308b0 con 0x7ffbb40a1d00 2024-04-26T00:39:03.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbaaffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:03.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbaaffd640 1 -- 172.21.15.37:0/2483037277 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb0012e20 con 0x7ffbb40a1d00 2024-04-26T00:39:03.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbaaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:03.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbaaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:03.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbaaffd640 10 monclient: dump: 2024-04-26T00:39:03.747 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:03.747 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:03.747 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.747 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.747 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:03.747 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:03.747 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:03.747 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:03.747 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:03.747 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:03.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbbc60c640 10 monclient: _renew_subs 2024-04-26T00:39:03.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbbc60c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:03.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbbc60c640 1 -- 172.21.15.37:0/2483037277 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbb40a0430 con 0x7ffbb40a1d00 2024-04-26T00:39:03.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbb8ba5640 4 set_mon_vals no callback set 2024-04-26T00:39:03.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbaaffd640 1 -- 172.21.15.37:0/2483037277 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ffbb001c460 con 0x7ffbb40a1d00 2024-04-26T00:39:03.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbbc60c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:03.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbbc60c640 1 --2- 172.21.15.37:0/2483037277 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb41cf3f0 0x7ffbb41cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:03.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbbc60c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:03.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbbc60c640 1 -- 172.21.15.37:0/2483037277 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ffbb404f510 con 0x7ffbb41cf3f0 2024-04-26T00:39:03.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbba3a8640 1 --2- 172.21.15.37:0/2483037277 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb41cf3f0 0x7ffbb41cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:03.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbaaffd640 1 --2- 172.21.15.37:0/2483037277 >> v2:172.21.15.45:6812/2379918623 conn(0x7ffb98035e80 0x7ffb98038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:03.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbba3a8640 10 monclient: get_auth_request con 0x7ffbb41cf3f0 auth_method 0 2024-04-26T00:39:03.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbba3a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:03.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbba3a8640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:03.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbba3a8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:03.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbaaffd640 1 -- 172.21.15.37:0/2483037277 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(755..755 src has 705..755) ==== 3947+0+0 (secure 0 0 0) 0x7ffbb0048900 con 0x7ffbb40a1d00 2024-04-26T00:39:03.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbba3a8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:03.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbba3a8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:03.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbba3a8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:03.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbba3a8640 10 monclient(hunting): handle_auth_done global_id 204404 payload 824 2024-04-26T00:39:03.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbba3a8640 1 -- 172.21.15.37:0/2483037277 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ffbb40008e0 con 0x7ffbb41cf3f0 2024-04-26T00:39:03.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbbaba9640 1 --2- 172.21.15.37:0/2483037277 >> v2:172.21.15.45:6812/2379918623 conn(0x7ffb98035e80 0x7ffb98038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:03.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.738+0000 7ffbbaba9640 10 monclient: get_auth_request con 0x7ffb98035e80 auth_method 0 2024-04-26T00:39:03.750 INFO:tasks.rados.rados.0.smithi045.stdout:3344: writing smithi04526579-4 from 433944 to 1081820 tid 1 2024-04-26T00:39:03.750 INFO:tasks.rados.rados.0.smithi045.stdout:3344: writing smithi04526579-4 from 1486536 to 2033944 tid 2 2024-04-26T00:39:03.750 INFO:tasks.rados.rados.0.smithi045.stdout:3345: copy_from oid 18 from oid 43 current snap is 370 2024-04-26T00:39:03.750 INFO:tasks.rados.rados.0.smithi045.stdout:3340: expect (ObjNum 1187 snap 369 seq_num 1187) 2024-04-26T00:39:03.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.742+0000 7ffbba3a8640 1 -- 172.21.15.37:0/2483037277 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb41cf3f0 msgr2=0x7ffbb41cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:03.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.742+0000 7ffbba3a8640 1 --2- 172.21.15.37:0/2483037277 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb41cf3f0 0x7ffbb41cf7d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ffbb406edb0 tx=0x7ffba4019180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:03.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.742+0000 7ffbaaffd640 1 -- 172.21.15.37:0/2483037277 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba40b0040 con 0x7ffbb41cf3f0 2024-04-26T00:39:03.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.742+0000 7ffbaaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:03.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.742+0000 7ffbaaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:03.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.742+0000 7ffbaaffd640 10 monclient: dump: 2024-04-26T00:39:03.752 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:03.752 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:03.752 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.752 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.752 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:03.752 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:03.752 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:03.753 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:03.753 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:03.753 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:03.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.742+0000 7ffbaaffd640 1 -- 172.21.15.37:0/2483037277 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ffba40b5680 con 0x7ffbb41cf3f0 2024-04-26T00:39:03.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.742+0000 7ffbaaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:03.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.742+0000 7ffbaaffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:39:03.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.742+0000 7ffbaaffd640 1 -- 172.21.15.37:0/2483037277 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb41cf3f0 msgr2=0x7ffbb41cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:03.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.742+0000 7ffbaaffd640 1 --2- 172.21.15.37:0/2483037277 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb41cf3f0 0x7ffbb41cf7d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ffbb406edb0 tx=0x7ffba4019180 comp rx=0 tx=0).stop 2024-04-26T00:39:03.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.742+0000 7ffbaaffd640 1 -- 172.21.15.37:0/2483037277 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb41cf3f0 msgr2=0x7ffbb41cf7d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:39:03.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.742+0000 7ffbaaffd640 1 --2- 172.21.15.37:0/2483037277 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb41cf3f0 0x7ffbb41cf7d0 secure :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ffbb406edb0 tx=0x7ffba4019180 comp rx=0 tx=0).stop 2024-04-26T00:39:03.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.742+0000 7ffbbaba9640 1 --2- 172.21.15.37:0/2483037277 >> v2:172.21.15.45:6812/2379918623 conn(0x7ffb98035e80 0x7ffb98038340 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4070c50 tx=0x7ffbac009820 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:03.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.750+0000 7ffbbc60c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:39:03.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.750+0000 7ffbbc60c640 1 --2- 172.21.15.37:0/2483037277 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb41cfd10 0x7ffbb413a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:03.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.750+0000 7ffbbc60c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:03.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.750+0000 7ffbbc60c640 1 -- 172.21.15.37:0/2483037277 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ffbb409fb90 con 0x7ffbb41cfd10 2024-04-26T00:39:03.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.750+0000 7ffbba3a8640 1 --2- 172.21.15.37:0/2483037277 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb41cfd10 0x7ffbb413a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:03.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.750+0000 7ffbba3a8640 10 monclient: get_auth_request con 0x7ffbb41cfd10 auth_method 0 2024-04-26T00:39:03.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.750+0000 7ffbba3a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:03.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.750+0000 7ffbba3a8640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:03.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.750+0000 7ffbba3a8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:03.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.750+0000 7ffbba3a8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:03.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.750+0000 7ffbba3a8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:03.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.750+0000 7ffbba3a8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:03.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.750+0000 7ffbba3a8640 10 monclient(hunting): handle_auth_done global_id 204407 payload 824 2024-04-26T00:39:03.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.750+0000 7ffbba3a8640 1 -- 172.21.15.37:0/2483037277 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ffbb41d0120 con 0x7ffbb41cfd10 2024-04-26T00:39:03.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.754+0000 7ffbba3a8640 1 -- 172.21.15.37:0/2483037277 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb41cfd10 msgr2=0x7ffbb413a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:03.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.754+0000 7ffbba3a8640 1 --2- 172.21.15.37:0/2483037277 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb41cfd10 0x7ffbb413a050 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ffba401aa90 tx=0x7ffba40a8000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:03.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.754+0000 7ffbaaffd640 1 -- 172.21.15.37:0/2483037277 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba40b0070 con 0x7ffbb41cfd10 2024-04-26T00:39:03.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.754+0000 7ffbaaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:03.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.754+0000 7ffbaaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:03.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.754+0000 7ffbaaffd640 10 monclient: dump: 2024-04-26T00:39:03.755 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:03.756 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:03.756 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.756 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.756 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:03.756 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:03.756 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:03.756 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:03.756 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:03.756 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:03.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.754+0000 7ffbaaffd640 1 -- 172.21.15.37:0/2483037277 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4718 (secure 0 0 0) 0x7ffba40bfa10 con 0x7ffbb41cfd10 2024-04-26T00:39:03.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.754+0000 7ffbaaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:39:03.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.754+0000 7ffbaaffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:39:03.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.754+0000 7ffbaaffd640 1 -- 172.21.15.37:0/2483037277 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb41cfd10 msgr2=0x7ffbb413a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:03.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.754+0000 7ffbaaffd640 1 --2- 172.21.15.37:0/2483037277 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb41cfd10 0x7ffbb413a050 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ffba401aa90 tx=0x7ffba40a8000 comp rx=0 tx=0).stop 2024-04-26T00:39:03.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.754+0000 7ffbaaffd640 1 -- 172.21.15.37:0/2483037277 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb41cfd10 msgr2=0x7ffbb413a050 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:39:03.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.754+0000 7ffbaaffd640 1 --2- 172.21.15.37:0/2483037277 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb41cfd10 0x7ffbb413a050 secure :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ffba401aa90 tx=0x7ffba40a8000 comp rx=0 tx=0).stop 2024-04-26T00:39:03.757 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:39:03.757 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:39:03.757 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:39:03.757 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:39:03.757 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 144, 2024-04-26T00:39:03.757 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 149362, 2024-04-26T00:39:03.757 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:39:03.757 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:39:03.757 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:39:03.758 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:39:03.758 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:03.758 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 1, 2024-04-26T00:39:03.758 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:39:03.758 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:39:03.758 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:39:03.758 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:03.758 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:03.758 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:03.758 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:03.758 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:03.758 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:03.758 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:03.759 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:03.759 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:03.759 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:03.759 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:03.759 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:03.759 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:39:03.759 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:39:03.759 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:03.759 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:03.759 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:03.759 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:03.759 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:03.759 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:03.759 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:03.760 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:03.760 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:03.760 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:03.760 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:03.760 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:03.760 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:03.760 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:39:03.760 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:39:03.760 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:39:03.760 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:39:03.760 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:39:03.760 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:39:03.760 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:39:03.760 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:39:03.761 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:39:03.761 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:39:03.761 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:39:03.761 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:39:03.761 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:39:03.761 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:39:03.761 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:39:03.761 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:39:03.761 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:39:03.761 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:03.761 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:03.761 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:03.761 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:03.761 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:03.761 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:03.762 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:03.762 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:03.762 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:03.762 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:03.762 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:03.762 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:03.762 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:39:03.762 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:03.762 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:39:03.762 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:03.762 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:39:03.762 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:39:03.762 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:03.763 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:03.763 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:03.763 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:03.763 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:39:03.763 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:03.763 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:03.763 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:03.763 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:03.763 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:39:03.763 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:03.763 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:03.763 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:03.763 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:03.763 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:39:03.764 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:39:03.764 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:03.764 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:03.764 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:03.764 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:03.764 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:03.764 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:39:03.764 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:39:03.764 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:03.764 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:03.764 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:03.764 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:03.764 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:39:03.764 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:03.764 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:03.765 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:03.765 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:03.765 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:39:03.765 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:03.765 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:03.765 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:03.765 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:03.765 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:39:03.765 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:39:03.765 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:03.765 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:03.765 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:03.765 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:03.765 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:03.765 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:39:03.766 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:39:03.766 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:03.766 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:03.766 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:03.766 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:03.766 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:39:03.766 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:03.766 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:03.766 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:03.766 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:03.766 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:39:03.766 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:03.766 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:03.767 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:03.767 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:03.767 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:39:03.767 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:39:03.767 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:03.767 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:03.767 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:03.767 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:03.767 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:03.767 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:03.767 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:39:03.767 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:39:03.767 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:03.769 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:03.769 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:03.769 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:39:03.769 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:03.769 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:03.769 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:39:03.769 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:03.769 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:03.769 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:03.770 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:39:03.770 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:03.770 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:03.770 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:39:03.770 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:03.770 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:03.770 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:03.770 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:39:03.770 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:03.770 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:03.770 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:03.770 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:39:03.770 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:39:03.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.754+0000 7ffbbc60c640 1 -- 172.21.15.37:0/2483037277 >> v2:172.21.15.45:6812/2379918623 conn(0x7ffb98035e80 msgr2=0x7ffb98038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:03.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.754+0000 7ffbbc60c640 1 --2- 172.21.15.37:0/2483037277 >> v2:172.21.15.45:6812/2379918623 conn(0x7ffb98035e80 0x7ffb98038340 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4070c50 tx=0x7ffbac009820 comp rx=0 tx=0).stop 2024-04-26T00:39:03.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.754+0000 7ffbbc60c640 10 monclient: shutdown 2024-04-26T00:39:03.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.754+0000 7ffbbc60c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:03.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.754+0000 7ffbbc60c640 1 -- 172.21.15.37:0/2483037277 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb40a1d00 msgr2=0x7ffbb412fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:03.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.754+0000 7ffbbc60c640 1 --2- 172.21.15.37:0/2483037277 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbb40a1d00 0x7ffbb412fe30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ffbb000c480 tx=0x7ffbb000c940 comp rx=0 tx=0).stop 2024-04-26T00:39:03.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.754+0000 7ffbbaba9640 1 -- 172.21.15.37:0/2483037277 reap_dead start 2024-04-26T00:39:03.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.754+0000 7ffbbaba9640 1 -- 172.21.15.37:0/2483037277 reap_dead start 2024-04-26T00:39:03.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.754+0000 7ffbbc60c640 1 -- 172.21.15.37:0/2483037277 shutdown_connections 2024-04-26T00:39:03.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.754+0000 7ffbbc60c640 1 -- 172.21.15.37:0/2483037277 >> 172.21.15.37:0/2483037277 conn(0x7ffbb40a7300 msgr2=0x7ffbb4132c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:03.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.754+0000 7ffbbc60c640 1 -- 172.21.15.37:0/2483037277 shutdown_connections 2024-04-26T00:39:03.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.754+0000 7ffbbc60c640 1 -- 172.21.15.37:0/2483037277 wait complete. 2024-04-26T00:39:03.773 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:39:03.773 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:39:03.774 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:39:03.885 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.886+0000 7fb075635640 10 monclient: get_monmap_and_config 2024-04-26T00:39:03.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.886+0000 7fb075635640 10 monclient: build_initial_monmap 2024-04-26T00:39:03.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.886+0000 7fb075635640 10 monclient: monmap: 2024-04-26T00:39:03.886 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:03.886 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:03.886 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:39:03.889180+0000 2024-04-26T00:39:03.886 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:39:03.889180+0000 2024-04-26T00:39:03.886 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:03.886 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:03.886 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:39:03.886 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:39:03.886 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:39:03.886 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:39:03.887 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:39:03.887 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:03.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.886+0000 7fb075635640 1 Processor -- start 2024-04-26T00:39:03.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.886+0000 7fb075635640 1 -- start start 2024-04-26T00:39:03.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.886+0000 7fb075635640 10 monclient: init 2024-04-26T00:39:03.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.886+0000 7fb075635640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:03.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.886+0000 7fb075635640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-04-26T00:39:03.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.886+0000 7fb075635640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb0700a3d90 addr v1:172.21.15.37:6789/0 2024-04-26T00:39:03.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.886+0000 7fb075635640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb0700a4170 0x7fb0700a45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:03.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.886+0000 7fb075635640 10 monclient(hunting): picked mon.noname-b con 0x7fb0700a4170 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:03.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.886+0000 7fb075635640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb07013c3e0 addr v1:172.21.15.45:6789/0 2024-04-26T00:39:03.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.886+0000 7fb075635640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0700a5150 con 0x7fb0700a3d90 2024-04-26T00:39:03.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.886+0000 7fb075635640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb07009b280 con 0x7fb07013c3e0 2024-04-26T00:39:03.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.886+0000 7fb075635640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:03.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.886+0000 7fb075635640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb07009c650 con 0x7fb0700a4170 2024-04-26T00:39:03.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.886+0000 7fb075635640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:03.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.886+0000 7fb075635640 10 monclient(hunting): authenticate will time out at 2125.194942s 2024-04-26T00:39:03.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.886+0000 7fb06e7fc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb0700a4170 0x7fb0700a45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:03.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.886+0000 7fb06e7fc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb0700a4170 0x7fb0700a45b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:35670/0 (socket says 172.21.15.37:35670) 2024-04-26T00:39:03.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.886+0000 7fb06e7fc640 1 -- 172.21.15.37:0/3879810281 learned_addr learned my addr 172.21.15.37:0/3879810281 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:03.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.886+0000 7fb06e7fc640 10 monclient(hunting): get_auth_request con 0x7fb0700a4170 auth_method 0 2024-04-26T00:39:03.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.886+0000 7fb06e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:03.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.886+0000 7fb06e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:03.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.886+0000 7fb06e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:03.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.886+0000 7fb06e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:03.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.886+0000 7fb06e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:03.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.886+0000 7fb06e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:03.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.886+0000 7fb06e7fc640 10 monclient(hunting): handle_auth_done global_id 204410 payload 274 2024-04-26T00:39:03.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.886+0000 7fb06e7fc640 1 -- 172.21.15.37:0/3879810281 >> v1:172.21.15.45:6789/0 conn(0x7fb07013c3e0 legacy=0x7fb07013e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:03.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.886+0000 7fb06e7fc640 1 -- 172.21.15.37:0/3879810281 >> v1:172.21.15.37:6789/0 conn(0x7fb0700a3d90 legacy=0x7fb07009bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:03.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.886+0000 7fb06e7fc640 10 monclient: _finish_hunting 0 2024-04-26T00:39:03.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.886+0000 7fb06e7fc640 1 monclient: found mon.noname-b 2024-04-26T00:39:03.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.886+0000 7fb06e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:03.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.886+0000 7fb06e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:39:03.891 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.886+0000 7fb06e7fc640 1 -- 172.21.15.37:0/3879810281 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb07009b560 con 0x7fb0700a4170 2024-04-26T00:39:03.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.890+0000 7fb06e7fc640 1 -- 172.21.15.37:0/3879810281 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb0700a4170 msgr2=0x7fb0700a45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:03.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.890+0000 7fb06e7fc640 1 --2- 172.21.15.37:0/3879810281 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb0700a4170 0x7fb0700a45b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb0580098a0 tx=0x7fb05801c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=156444327de1e5b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:03.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.890+0000 7fb06dffb640 1 -- 172.21.15.37:0/3879810281 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0580b0070 con 0x7fb0700a4170 2024-04-26T00:39:03.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.890+0000 7fb06dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:03.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.890+0000 7fb06dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:39:03.893 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.890+0000 7fb06dffb640 10 monclient: dump: 2024-04-26T00:39:03.893 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:03.893 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:03.894 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.894 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.894 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:03.894 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:03.894 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:03.894 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:03.894 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:03.894 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:03.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.890+0000 7fb06dffb640 10 monclient: _finish_auth 0 2024-04-26T00:39:03.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.890+0000 7fb06dffb640 10 monclient: _check_auth_tickets 2024-04-26T00:39:03.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.890+0000 7fb06dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:03.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.890+0000 7fb06dffb640 1 -- 172.21.15.37:0/3879810281 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0580ab600 con 0x7fb0700a4170 2024-04-26T00:39:03.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.890+0000 7fb06dffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:03.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.890+0000 7fb06dffb640 1 -- 172.21.15.37:0/3879810281 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0580ab9d0 con 0x7fb0700a4170 2024-04-26T00:39:03.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.890+0000 7fb075635640 5 monclient: authenticate success, global_id 204410 2024-04-26T00:39:03.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.890+0000 7fb075635640 10 monclient: get_monmap_and_config success 2024-04-26T00:39:03.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.890+0000 7fb075635640 4 set_mon_vals no callback set 2024-04-26T00:39:03.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.890+0000 7fb06dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:03.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.890+0000 7fb06dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:03.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.890+0000 7fb06dffb640 10 monclient: dump: 2024-04-26T00:39:03.895 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:03.895 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:03.895 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.896 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.896 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:03.896 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:03.896 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:03.896 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:03.896 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:03.896 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:03.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb075635640 10 monclient: shutdown 2024-04-26T00:39:03.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb075635640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:03.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb075635640 1 -- 172.21.15.37:0/3879810281 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb0700a4170 msgr2=0x7fb0700a45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:03.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb075635640 1 --2- 172.21.15.37:0/3879810281 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb0700a4170 0x7fb0700a45b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb0580098a0 tx=0x7fb05801c3c0 comp rx=0 tx=0).stop 2024-04-26T00:39:03.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb075635640 1 -- 172.21.15.37:0/3879810281 shutdown_connections 2024-04-26T00:39:03.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb075635640 1 --2- 172.21.15.37:0/3879810281 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb0700a4170 0x7fb0700a45b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:03.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb075635640 1 -- 172.21.15.37:0/3879810281 >> 172.21.15.37:0/3879810281 conn(0x7fb0700aa0a0 msgr2=0x7fb0700a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:03.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb075635640 1 -- 172.21.15.37:0/3879810281 shutdown_connections 2024-04-26T00:39:03.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb075635640 1 -- 172.21.15.37:0/3879810281 wait complete. 2024-04-26T00:39:03.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb075635640 10 monclient: build_initial_monmap 2024-04-26T00:39:03.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb075635640 10 monclient: monmap: 2024-04-26T00:39:03.902 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:03.902 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:39:03.902 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:39:03.902 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:39:03.902 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:03.902 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:39:03.902 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:39:03.902 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:39:03.902 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:39:03.903 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:03.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb075635640 1 Processor -- start 2024-04-26T00:39:03.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb075635640 1 -- start start 2024-04-26T00:39:03.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb075635640 10 monclient: init 2024-04-26T00:39:03.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb075635640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:03.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb075635640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:39:03.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb075635640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0700a3d90 0x7fb070136a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:03.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb075635640 10 monclient(hunting): picked mon.noname-b con 0x7fb0700a3d90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:39:03.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb075635640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb0700a4170 0x7fb070136fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:03.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb075635640 10 monclient(hunting): picked mon.noname-c con 0x7fb0700a4170 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:03.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb075635640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb07013c3e0 0x7fb070138bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:03.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb075635640 10 monclient(hunting): picked mon.noname-a con 0x7fb07013c3e0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:39:03.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb075635640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:03.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb075635640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb070142aa0 con 0x7fb07013c3e0 2024-04-26T00:39:03.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb075635640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:03.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb075635640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb070142920 con 0x7fb0700a3d90 2024-04-26T00:39:03.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb075635640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:03.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb075635640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb070142c20 con 0x7fb0700a4170 2024-04-26T00:39:03.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb075635640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:03.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb075635640 10 monclient(hunting): authenticate will time out at 2125.201927s 2024-04-26T00:39:03.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb06e7fc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb0700a4170 0x7fb070136fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:03.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb06f7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb07013c3e0 0x7fb070138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:03.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb06e7fc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb0700a4170 0x7fb070136fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:35678/0 (socket says 172.21.15.37:35678) 2024-04-26T00:39:03.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb06e7fc640 1 -- 172.21.15.37:0/207291377 learned_addr learned my addr 172.21.15.37:0/207291377 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:03.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb06e7fc640 10 monclient(hunting): get_auth_request con 0x7fb0700a4170 auth_method 0 2024-04-26T00:39:03.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb06e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:03.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb06e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:03.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb06e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:03.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb06f7fe640 10 monclient(hunting): get_auth_request con 0x7fb07013c3e0 auth_method 0 2024-04-26T00:39:03.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb06f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:03.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb06f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:03.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb06f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:03.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb06e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:03.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb06e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:03.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb06e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:03.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb06effd640 1 --2- 172.21.15.37:0/207291377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0700a3d90 0x7fb070136a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:03.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb06f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:03.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb06f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:03.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb06f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:03.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb06effd640 10 monclient(hunting): get_auth_request con 0x7fb0700a3d90 auth_method 0 2024-04-26T00:39:03.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb06effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:03.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb06effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:03.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb06effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:03.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb06effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:03.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb06effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:03.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb06effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:03.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb06e7fc640 10 monclient(hunting): handle_auth_done global_id 204413 payload 824 2024-04-26T00:39:03.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb06e7fc640 1 -- 172.21.15.37:0/207291377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0700a3d90 msgr2=0x7fb070136a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:03.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb06e7fc640 1 --2- 172.21.15.37:0/207291377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0700a3d90 0x7fb070136a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:03.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb06e7fc640 1 -- 172.21.15.37:0/207291377 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb07013c3e0 msgr2=0x7fb070138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:03.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb06e7fc640 1 --2- 172.21.15.37:0/207291377 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb07013c3e0 0x7fb070138bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:03.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb06e7fc640 10 monclient: _finish_hunting 0 2024-04-26T00:39:03.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb06e7fc640 1 monclient: found mon.noname-c 2024-04-26T00:39:03.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb06e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:03.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb06e7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:39:03.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb06e7fc640 1 -- 172.21.15.37:0/207291377 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb07009ada0 con 0x7fb0700a4170 2024-04-26T00:39:03.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.894+0000 7fb06f7fe640 1 --2- 172.21.15.37:0/207291377 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb07013c3e0 0x7fb070138bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:39:03.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb06e7fc640 1 -- 172.21.15.37:0/207291377 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb0700a4170 msgr2=0x7fb070136fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:03.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb06e7fc640 1 --2- 172.21.15.37:0/207291377 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb0700a4170 0x7fb070136fb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb0580098a0 tx=0x7fb058019160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:03.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb0677fe640 1 -- 172.21.15.37:0/207291377 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0580b0050 con 0x7fb0700a4170 2024-04-26T00:39:03.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb0677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:03.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb0677fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:39:03.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb0677fe640 10 monclient: dump: 2024-04-26T00:39:03.907 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:03.907 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:03.907 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.907 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.907 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:03.908 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:03.908 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:03.908 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:03.908 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:03.908 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:03.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb0677fe640 10 monclient: _finish_auth 0 2024-04-26T00:39:03.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb0677fe640 10 monclient: _check_auth_tickets 2024-04-26T00:39:03.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb0677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:03.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb0677fe640 1 -- 172.21.15.37:0/207291377 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0580ba070 con 0x7fb0700a4170 2024-04-26T00:39:03.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb0677fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:03.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb075635640 5 monclient: authenticate success, global_id 204413 2024-04-26T00:39:03.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb0677fe640 1 -- 172.21.15.37:0/207291377 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0580b4a00 con 0x7fb0700a4170 2024-04-26T00:39:03.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb075635640 10 monclient: _renew_subs 2024-04-26T00:39:03.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb075635640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:03.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb075635640 1 -- 172.21.15.37:0/207291377 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb07013a860 con 0x7fb0700a4170 2024-04-26T00:39:03.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb06d7fa640 4 set_mon_vals no callback set 2024-04-26T00:39:03.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb0677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:03.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb0677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:03.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb0677fe640 10 monclient: dump: 2024-04-26T00:39:03.909 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:03.909 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:03.909 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.909 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.909 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:03.909 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:03.909 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:03.909 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:03.909 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:03.909 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:03.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb075635640 10 monclient: _renew_subs 2024-04-26T00:39:03.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb075635640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:03.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb075635640 1 -- 172.21.15.37:0/207291377 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb07011fd50 con 0x7fb0700a4170 2024-04-26T00:39:03.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb075635640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:03.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb075635640 1 --2- 172.21.15.37:0/207291377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb070139100 0x7fb0701394e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:03.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb075635640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:03.910 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb075635640 1 -- 172.21.15.37:0/207291377 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb0700a0790 con 0x7fb070139100 2024-04-26T00:39:03.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb0677fe640 1 -- 172.21.15.37:0/207291377 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb0580b4be0 con 0x7fb0700a4170 2024-04-26T00:39:03.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb06effd640 1 --2- 172.21.15.37:0/207291377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb070139100 0x7fb0701394e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:03.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb0677fe640 1 --2- 172.21.15.37:0/207291377 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb050035ed0 0x7fb050038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:03.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb06effd640 10 monclient: get_auth_request con 0x7fb070139100 auth_method 0 2024-04-26T00:39:03.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb06effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:03.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb06effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:03.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb06effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:03.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb0677fe640 1 -- 172.21.15.37:0/207291377 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(755..755 src has 705..755) ==== 3947+0+0 (secure 0 0 0) 0x7fb0580e2a60 con 0x7fb0700a4170 2024-04-26T00:39:03.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb06effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:03.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb06effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:03.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb06effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:03.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb06f7fe640 1 --2- 172.21.15.37:0/207291377 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb050035ed0 0x7fb050038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:03.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb06effd640 10 monclient(hunting): handle_auth_done global_id 224191 payload 824 2024-04-26T00:39:03.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb06effd640 1 -- 172.21.15.37:0/207291377 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb07004f510 con 0x7fb070139100 2024-04-26T00:39:03.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.898+0000 7fb06f7fe640 10 monclient: get_auth_request con 0x7fb050035ed0 auth_method 0 2024-04-26T00:39:03.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.902+0000 7fb06effd640 1 -- 172.21.15.37:0/207291377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb070139100 msgr2=0x7fb0701394e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:03.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.902+0000 7fb06effd640 1 --2- 172.21.15.37:0/207291377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb070139100 0x7fb0701394e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb06800fae0 tx=0x7fb068004810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:03.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.902+0000 7fb0677fe640 1 -- 172.21.15.37:0/207291377 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb06800dd00 con 0x7fb070139100 2024-04-26T00:39:03.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.902+0000 7fb0677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:03.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.902+0000 7fb0677fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:39:03.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.902+0000 7fb0677fe640 10 monclient: dump: 2024-04-26T00:39:03.912 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:03.912 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:03.912 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.912 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.912 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:03.912 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:03.913 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:03.913 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:03.913 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:03.913 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:03.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.902+0000 7fb06f7fe640 1 --2- 172.21.15.37:0/207291377 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb050035ed0 0x7fb050038390 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fb060007fd0 tx=0x7fb060007dc0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:03.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.902+0000 7fb0677fe640 1 -- 172.21.15.37:0/207291377 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb06800dd00 con 0x7fb070139100 2024-04-26T00:39:03.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.902+0000 7fb0677fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:03.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.902+0000 7fb0677fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:39:03.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.902+0000 7fb0677fe640 1 -- 172.21.15.37:0/207291377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb070139100 msgr2=0x7fb0701394e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:03.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.902+0000 7fb0677fe640 1 --2- 172.21.15.37:0/207291377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb070139100 0x7fb0701394e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb06800fae0 tx=0x7fb068004810 comp rx=0 tx=0).stop 2024-04-26T00:39:03.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.902+0000 7fb0677fe640 1 -- 172.21.15.37:0/207291377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb070139100 msgr2=0x7fb0701394e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:39:03.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.902+0000 7fb0677fe640 1 --2- 172.21.15.37:0/207291377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb070139100 0x7fb0701394e0 secure :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb06800fae0 tx=0x7fb068004810 comp rx=0 tx=0).stop 2024-04-26T00:39:03.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.910+0000 7fb075635640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:39:03.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.910+0000 7fb075635640 1 --2- 172.21.15.37:0/207291377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb070139a20 0x7fb07013c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:03.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.910+0000 7fb075635640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:03.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.910+0000 7fb075635640 1 -- 172.21.15.37:0/207291377 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb070075dc0 con 0x7fb070139a20 2024-04-26T00:39:03.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.910+0000 7fb06effd640 1 --2- 172.21.15.37:0/207291377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb070139a20 0x7fb07013c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:03.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.910+0000 7fb06effd640 10 monclient: get_auth_request con 0x7fb070139a20 auth_method 0 2024-04-26T00:39:03.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.910+0000 7fb06effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:03.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.910+0000 7fb06effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:03.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.910+0000 7fb06effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:03.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.910+0000 7fb06effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:03.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.910+0000 7fb06effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:03.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.910+0000 7fb06effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:03.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.910+0000 7fb06effd640 10 monclient(hunting): handle_auth_done global_id 224194 payload 824 2024-04-26T00:39:03.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.910+0000 7fb06effd640 1 -- 172.21.15.37:0/207291377 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb07011f9e0 con 0x7fb070139a20 2024-04-26T00:39:03.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.910+0000 7fb06effd640 1 -- 172.21.15.37:0/207291377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb070139a20 msgr2=0x7fb07013c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:03.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.910+0000 7fb06effd640 1 --2- 172.21.15.37:0/207291377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb070139a20 0x7fb07013c7c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb06800fae0 tx=0x7fb068003af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:03.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.910+0000 7fb0677fe640 1 -- 172.21.15.37:0/207291377 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb068014470 con 0x7fb070139a20 2024-04-26T00:39:03.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.910+0000 7fb0677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:03.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.910+0000 7fb0677fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:39:03.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.910+0000 7fb0677fe640 10 monclient: dump: 2024-04-26T00:39:03.916 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:03.916 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:03.916 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.916 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:03.916 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:03.917 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:03.917 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:03.917 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:03.917 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:03.917 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:03.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.914+0000 7fb0677fe640 1 -- 172.21.15.37:0/207291377 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7fb068014470 con 0x7fb070139a20 2024-04-26T00:39:03.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.914+0000 7fb0677fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:39:03.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.914+0000 7fb0677fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:39:03.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.914+0000 7fb0677fe640 1 -- 172.21.15.37:0/207291377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb070139a20 msgr2=0x7fb07013c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:03.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.914+0000 7fb0677fe640 1 --2- 172.21.15.37:0/207291377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb070139a20 0x7fb07013c7c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb06800fae0 tx=0x7fb068003af0 comp rx=0 tx=0).stop 2024-04-26T00:39:03.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.914+0000 7fb0677fe640 1 -- 172.21.15.37:0/207291377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb070139a20 msgr2=0x7fb07013c7c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:39:03.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.914+0000 7fb0677fe640 1 --2- 172.21.15.37:0/207291377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb070139a20 0x7fb07013c7c0 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb06800fae0 tx=0x7fb068003af0 comp rx=0 tx=0).stop 2024-04-26T00:39:03.918 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:39:03.918 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:39:03.918 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:39:03.918 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:39:03.918 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 144, 2024-04-26T00:39:03.918 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 57575, 2024-04-26T00:39:03.918 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:39:03.918 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:39:03.918 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:39:03.918 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:39:03.918 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:03.918 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 1, 2024-04-26T00:39:03.918 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:39:03.918 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:39:03.919 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:39:03.919 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:03.919 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:03.919 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:03.919 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:03.919 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:03.919 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:03.919 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:03.919 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:03.919 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:03.919 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:03.919 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:03.919 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:03.919 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:39:03.919 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:39:03.920 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:03.920 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:03.920 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:03.920 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:03.920 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:03.920 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:03.920 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:03.920 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:03.920 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:03.920 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:03.920 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:03.920 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:03.921 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:03.921 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:39:03.921 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:39:03.921 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:39:03.921 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:39:03.921 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:39:03.921 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:39:03.922 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:39:03.922 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:39:03.922 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:39:03.922 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:39:03.922 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:39:03.922 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:39:03.922 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:39:03.922 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:39:03.923 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:39:03.923 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:39:03.923 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:39:03.923 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:03.923 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:03.924 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:03.924 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:03.924 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:03.924 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:03.924 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:03.924 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:03.924 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:03.925 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:03.925 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:03.925 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:03.925 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:39:03.925 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:03.925 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:39:03.925 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:03.926 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:39:03.926 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:39:03.926 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:03.926 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:03.926 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:03.926 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:03.926 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:39:03.927 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:03.927 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:03.927 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:03.927 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:03.927 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:39:03.927 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:03.927 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:03.927 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:03.928 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:03.928 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:39:03.928 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:39:03.928 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:03.928 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:03.928 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:03.929 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:03.929 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:03.929 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:39:03.929 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:39:03.929 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:03.929 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:03.930 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:03.930 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:03.930 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:39:03.930 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:03.930 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:03.930 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:03.930 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:03.930 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:39:03.931 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:03.931 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:03.931 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:03.931 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:03.931 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:39:03.931 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:39:03.931 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:03.931 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:03.931 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:03.931 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:03.931 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:03.931 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:39:03.932 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:39:03.932 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:03.932 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:03.932 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:03.932 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:03.932 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:39:03.932 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:03.932 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:03.932 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:03.932 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:03.932 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:39:03.932 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:03.932 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:03.932 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:03.933 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:03.933 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:39:03.933 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:39:03.933 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:03.933 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:03.933 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:03.933 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:03.933 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:03.933 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:03.933 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:39:03.933 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:39:03.933 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:03.933 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:03.933 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:03.933 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:39:03.934 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:03.934 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:03.934 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:39:03.934 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:03.934 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:03.934 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:03.934 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:39:03.934 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:03.934 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:03.934 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:39:03.934 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:03.934 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:03.934 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:03.934 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:39:03.935 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:03.935 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:03.935 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:39:03.935 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:03.935 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:03.935 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:03.935 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:39:03.935 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:03.935 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:03.935 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:03.935 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:39:03.935 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:39:03.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.914+0000 7fb075635640 1 -- 172.21.15.37:0/207291377 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb050035ed0 msgr2=0x7fb050038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:03.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.914+0000 7fb075635640 1 --2- 172.21.15.37:0/207291377 >> v2:172.21.15.45:6812/2379918623 conn(0x7fb050035ed0 0x7fb050038390 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fb060007fd0 tx=0x7fb060007dc0 comp rx=0 tx=0).stop 2024-04-26T00:39:03.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.914+0000 7fb075635640 10 monclient: shutdown 2024-04-26T00:39:03.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.914+0000 7fb075635640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:03.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.914+0000 7fb075635640 1 -- 172.21.15.37:0/207291377 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb0700a4170 msgr2=0x7fb070136fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:03.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.914+0000 7fb075635640 1 --2- 172.21.15.37:0/207291377 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb0700a4170 0x7fb070136fb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb0580098a0 tx=0x7fb058019160 comp rx=0 tx=0).stop 2024-04-26T00:39:03.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.914+0000 7fb06f7fe640 1 -- 172.21.15.37:0/207291377 reap_dead start 2024-04-26T00:39:03.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.914+0000 7fb06f7fe640 1 -- 172.21.15.37:0/207291377 reap_dead start 2024-04-26T00:39:03.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.914+0000 7fb075635640 1 -- 172.21.15.37:0/207291377 shutdown_connections 2024-04-26T00:39:03.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.914+0000 7fb075635640 1 -- 172.21.15.37:0/207291377 >> 172.21.15.37:0/207291377 conn(0x7fb0700aa0a0 msgr2=0x7fb0700a2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:03.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.914+0000 7fb075635640 1 -- 172.21.15.37:0/207291377 shutdown_connections 2024-04-26T00:39:03.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:03.914+0000 7fb075635640 1 -- 172.21.15.37:0/207291377 wait complete. 2024-04-26T00:39:03.938 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:39:03.938 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-26T00:39:03.938 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-26T00:39:04.025 INFO:tasks.rados.rados.0.smithi045.stdout:3341: finishing write tid 1 to smithi04526579-32 2024-04-26T00:39:04.025 INFO:tasks.rados.rados.0.smithi045.stdout:3341: finishing write tid 2 to smithi04526579-32 2024-04-26T00:39:04.026 INFO:tasks.rados.rados.0.smithi045.stdout:3341: finishing write tid 3 to smithi04526579-32 2024-04-26T00:39:04.026 INFO:tasks.rados.rados.0.smithi045.stdout:3341: finishing write tid 4 to smithi04526579-32 2024-04-26T00:39:04.026 INFO:tasks.rados.rados.0.smithi045.stdout:3341: oid 32 updating version 0 to 2486 2024-04-26T00:39:04.026 INFO:tasks.rados.rados.0.smithi045.stdout:3341: oid 32 version 2486 is already newer than 2485 2024-04-26T00:39:04.026 INFO:tasks.rados.rados.0.smithi045.stdout:3341: oid 32 version 2486 is already newer than 2484 2024-04-26T00:39:04.026 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 2486 (ObjNum 1190 snap 370 seq_num 1190) dirty exists 2024-04-26T00:39:04.026 INFO:tasks.rados.rados.0.smithi045.stdout:3341: left oid 32 (ObjNum 1190 snap 370 seq_num 1190) 2024-04-26T00:39:04.026 INFO:tasks.rados.rados.0.smithi045.stdout:3343: finishing write tid 1 to smithi04526579-9 2024-04-26T00:39:04.026 INFO:tasks.rados.rados.0.smithi045.stdout:3343: finishing write tid 2 to smithi04526579-9 2024-04-26T00:39:04.026 INFO:tasks.rados.rados.0.smithi045.stdout:3343: finishing write tid 3 to smithi04526579-9 2024-04-26T00:39:04.026 INFO:tasks.rados.rados.0.smithi045.stdout:3343: finishing write tid 4 to smithi04526579-9 2024-04-26T00:39:04.026 INFO:tasks.rados.rados.0.smithi045.stdout:3343: oid 9 updating version 0 to 2487 2024-04-26T00:39:04.027 INFO:tasks.rados.rados.0.smithi045.stdout:3343: oid 9 updating version 2487 to 2489 2024-04-26T00:39:04.027 INFO:tasks.rados.rados.0.smithi045.stdout:3343: oid 9 version 2489 is already newer than 2488 2024-04-26T00:39:04.027 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 2489 (ObjNum 1191 snap 370 seq_num 1191) dirty exists 2024-04-26T00:39:04.027 INFO:tasks.rados.rados.0.smithi045.stdout:3343: left oid 9 (ObjNum 1191 snap 370 seq_num 1191) 2024-04-26T00:39:04.027 INFO:tasks.rados.rados.0.smithi045.stdout:3344: finishing write tid 1 to smithi04526579-4 2024-04-26T00:39:04.027 INFO:tasks.rados.rados.0.smithi045.stdout:3344: finishing write tid 2 to smithi04526579-4 2024-04-26T00:39:04.027 INFO:tasks.rados.rados.0.smithi045.stdout:3345: finishing copy_from to smithi04526579-18 2024-04-26T00:39:04.027 INFO:tasks.rados.rados.0.smithi045.stdout:3345: got expected ENOENT (src dne) 2024-04-26T00:39:04.027 INFO:tasks.rados.rados.0.smithi045.stdout:3344: finishing write tid 3 to smithi04526579-4 2024-04-26T00:39:04.027 INFO:tasks.rados.rados.0.smithi045.stdout:3345: finishing copy_from racing read to smithi04526579-18 2024-04-26T00:39:04.027 INFO:tasks.rados.rados.0.smithi045.stdout:3344: finishing write tid 4 to smithi04526579-4 2024-04-26T00:39:04.027 INFO:tasks.rados.rados.0.smithi045.stdout:3344: oid 4 updating version 0 to 2491 2024-04-26T00:39:04.027 INFO:tasks.rados.rados.0.smithi045.stdout:3344: oid 4 version 2491 is already newer than 2490 2024-04-26T00:39:04.027 INFO:tasks.rados.rados.0.smithi045.stdout:3344: oid 4 updating version 2491 to 2492 2024-04-26T00:39:04.027 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 2492 (ObjNum 1192 snap 370 seq_num 1192) dirty exists 2024-04-26T00:39:04.028 INFO:tasks.rados.rados.0.smithi045.stdout:3344: left oid 4 (ObjNum 1192 snap 370 seq_num 1192) 2024-04-26T00:39:04.028 INFO:tasks.rados.rados.0.smithi045.stdout:3340: done (5 left) 2024-04-26T00:39:04.028 INFO:tasks.rados.rados.0.smithi045.stdout:3341: done (4 left) 2024-04-26T00:39:04.028 INFO:tasks.rados.rados.0.smithi045.stdout:3342: done (3 left) 2024-04-26T00:39:04.028 INFO:tasks.rados.rados.0.smithi045.stdout:3343: done (2 left) 2024-04-26T00:39:04.028 INFO:tasks.rados.rados.0.smithi045.stdout:3344: done (1 left) 2024-04-26T00:39:04.028 INFO:tasks.rados.rados.0.smithi045.stdout:3345: done (0 left) 2024-04-26T00:39:04.028 INFO:tasks.rados.rados.0.smithi045.stdout:3346: copy_from oid 47 from oid 44 current snap is 370 2024-04-26T00:39:04.028 INFO:tasks.rados.rados.0.smithi045.stdout:3347: snap_create 2024-04-26T00:39:04.050 INFO:tasks.rados.rados.0.smithi045.stdout:3346: finishing copy_from racing read to smithi04526579-47 2024-04-26T00:39:04.051 INFO:tasks.rados.rados.0.smithi045.stdout:3346: finishing copy_from to smithi04526579-47 2024-04-26T00:39:04.051 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 2382 (ObjNum 1127 snap 340 seq_num 1127) dirty exists 2024-04-26T00:39:04.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d2cb9d640 10 monclient: get_monmap_and_config 2024-04-26T00:39:04.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d2cb9d640 10 monclient: build_initial_monmap 2024-04-26T00:39:04.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d2cb9d640 10 monclient: monmap: 2024-04-26T00:39:04.053 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:04.053 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:04.053 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:39:04.056814+0000 2024-04-26T00:39:04.053 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:39:04.056814+0000 2024-04-26T00:39:04.053 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:04.053 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:04.053 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:39:04.054 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:39:04.054 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:39:04.054 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:39:04.054 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:39:04.054 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:04.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d2cb9d640 1 Processor -- start 2024-04-26T00:39:04.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d2cb9d640 1 -- start start 2024-04-26T00:39:04.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d2cb9d640 10 monclient: init 2024-04-26T00:39:04.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d2cb9d640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:04.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d2cb9d640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-04-26T00:39:04.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d2cb9d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9d280a0260 0x7f9d280a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:04.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d2cb9d640 10 monclient(hunting): picked mon.noname-a con 0x7f9d280a0260 addr v2:172.21.15.37:3300/0 2024-04-26T00:39:04.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d2cb9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d28134b30 0x7f9d28134f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:04.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d2cb9d640 10 monclient(hunting): picked mon.noname-b con 0x7f9d28134b30 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:04.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d2cb9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9d28135490 addr v1:172.21.15.37:6789/0 2024-04-26T00:39:04.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d2cb9d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d280a1130 con 0x7f9d28135490 2024-04-26T00:39:04.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d2cb9d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:04.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d2cb9d640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f9d2809ec10 con 0x7f9d280a0260 2024-04-26T00:39:04.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d2cb9d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:04.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d2cb9d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9d2809f0f0 con 0x7f9d28134b30 2024-04-26T00:39:04.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d2cb9d640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:04.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d2cb9d640 10 monclient(hunting): authenticate will time out at 2125.362406s 2024-04-26T00:39:04.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d26d9d640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f9d28135490 0x7f9d2813be60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:52850/0 (socket says 172.21.15.37:52850) 2024-04-26T00:39:04.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d26d9d640 1 -- 172.21.15.37:0/1968667074 learned_addr learned my addr 172.21.15.37:0/1968667074 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:39:04.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d2659c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9d280a0260 0x7f9d280a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:04.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d2659c640 10 monclient(hunting): get_auth_request con 0x7f9d280a0260 auth_method 0 2024-04-26T00:39:04.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d2659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:04.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d2659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:04.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d2659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:04.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d25d9b640 1 --2- 172.21.15.37:0/1968667074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d28134b30 0x7f9d28134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:04.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d25d9b640 10 monclient(hunting): get_auth_request con 0x7f9d28134b30 auth_method 0 2024-04-26T00:39:04.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d25d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:04.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d25d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:04.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d25d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:04.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d2659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:04.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d2659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:04.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d2659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:04.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d25d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:04.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d25d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:04.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d25d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:04.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d25d9b640 10 monclient(hunting): handle_auth_done global_id 204416 payload 274 2024-04-26T00:39:04.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d25d9b640 1 -- 172.21.15.37:0/1968667074 >> v2:172.21.15.37:3300/0 conn(0x7f9d280a0260 msgr2=0x7f9d280a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:04.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d25d9b640 1 --2- 172.21.15.37:0/1968667074 >> v2:172.21.15.37:3300/0 conn(0x7f9d280a0260 0x7f9d280a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:04.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d25d9b640 1 -- 172.21.15.37:0/1968667074 >> v1:172.21.15.37:6789/0 conn(0x7f9d28135490 legacy=0x7f9d2813be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:04.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d25d9b640 10 monclient: _finish_hunting 0 2024-04-26T00:39:04.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d25d9b640 1 monclient: found mon.noname-b 2024-04-26T00:39:04.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d25d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:04.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d25d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:39:04.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d25d9b640 1 -- 172.21.15.37:0/1968667074 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d2806cff0 con 0x7f9d28134b30 2024-04-26T00:39:04.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.054+0000 7f9d2659c640 1 --2- 172.21.15.37:0/1968667074 >> v2:172.21.15.37:3300/0 conn(0x7f9d280a0260 0x7f9d280a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:39:04.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.058+0000 7f9d25d9b640 1 -- 172.21.15.37:0/1968667074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d28134b30 msgr2=0x7f9d28134f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:04.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.058+0000 7f9d25d9b640 1 --2- 172.21.15.37:0/1968667074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d28134b30 0x7f9d28134f50 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9d180097c0 tx=0x7f9d1801c140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7822a6a875881f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:04.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.058+0000 7f9d2559a640 1 -- 172.21.15.37:0/1968667074 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d180b1070 con 0x7f9d28134b30 2024-04-26T00:39:04.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.058+0000 7f9d2559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:04.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.058+0000 7f9d2559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:39:04.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.058+0000 7f9d2559a640 10 monclient: dump: 2024-04-26T00:39:04.070 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:04.070 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:04.070 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:04.070 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:04.070 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:04.070 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:04.070 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:04.070 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:04.070 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:04.070 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:04.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.058+0000 7f9d2559a640 10 monclient: _finish_auth 0 2024-04-26T00:39:04.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.058+0000 7f9d2559a640 10 monclient: _check_auth_tickets 2024-04-26T00:39:04.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.058+0000 7f9d2559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:04.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.058+0000 7f9d2559a640 1 -- 172.21.15.37:0/1968667074 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d180ac440 con 0x7f9d28134b30 2024-04-26T00:39:04.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.058+0000 7f9d2559a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:04.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.058+0000 7f9d2559a640 1 -- 172.21.15.37:0/1968667074 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d180ac810 con 0x7f9d28134b30 2024-04-26T00:39:04.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.058+0000 7f9d2559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:04.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.058+0000 7f9d2559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:04.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.058+0000 7f9d2559a640 10 monclient: dump: 2024-04-26T00:39:04.071 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:04.071 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:04.071 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:04.071 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:04.071 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:04.071 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:04.071 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:04.071 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:04.071 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:04.072 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:04.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.058+0000 7f9d2cb9d640 5 monclient: authenticate success, global_id 204416 2024-04-26T00:39:04.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.058+0000 7f9d2cb9d640 10 monclient: get_monmap_and_config success 2024-04-26T00:39:04.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.058+0000 7f9d2cb9d640 4 set_mon_vals no callback set 2024-04-26T00:39:04.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2cb9d640 10 monclient: shutdown 2024-04-26T00:39:04.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2cb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:04.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2cb9d640 1 -- 172.21.15.37:0/1968667074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d28134b30 msgr2=0x7f9d28134f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:04.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2cb9d640 1 --2- 172.21.15.37:0/1968667074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d28134b30 0x7f9d28134f50 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9d180097c0 tx=0x7f9d1801c140 comp rx=0 tx=0).stop 2024-04-26T00:39:04.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2cb9d640 1 -- 172.21.15.37:0/1968667074 shutdown_connections 2024-04-26T00:39:04.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2cb9d640 1 --2- 172.21.15.37:0/1968667074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d28134b30 0x7f9d28134f50 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:04.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2cb9d640 1 --2- 172.21.15.37:0/1968667074 >> v2:172.21.15.37:3300/0 conn(0x7f9d280a0260 0x7f9d280a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:04.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2cb9d640 1 -- 172.21.15.37:0/1968667074 >> 172.21.15.37:0/1968667074 conn(0x7f9d280a1ad0 msgr2=0x7f9d280a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:04.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2cb9d640 1 -- 172.21.15.37:0/1968667074 shutdown_connections 2024-04-26T00:39:04.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2cb9d640 1 -- 172.21.15.37:0/1968667074 wait complete. 2024-04-26T00:39:04.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2cb9d640 10 monclient: build_initial_monmap 2024-04-26T00:39:04.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2cb9d640 10 monclient: monmap: 2024-04-26T00:39:04.073 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:04.073 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:39:04.073 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:39:04.073 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:39:04.073 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:04.073 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:39:04.073 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:39:04.073 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:39:04.073 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:39:04.073 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:04.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2cb9d640 1 Processor -- start 2024-04-26T00:39:04.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2cb9d640 1 -- start start 2024-04-26T00:39:04.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2cb9d640 10 monclient: init 2024-04-26T00:39:04.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2cb9d640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:04.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2cb9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:39:04.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2cb9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d280a0260 0x7f9d28130ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:04.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2cb9d640 10 monclient(hunting): picked mon.noname-c con 0x7f9d280a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:04.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2cb9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d28134b30 0x7f9d28131420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:04.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2cb9d640 10 monclient(hunting): picked mon.noname-a con 0x7f9d28134b30 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:39:04.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2cb9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d28135490 0x7f9d281396e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:04.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2cb9d640 10 monclient(hunting): picked mon.noname-b con 0x7f9d28135490 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:39:04.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2cb9d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:04.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2cb9d640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9d2809f430 con 0x7f9d28134b30 2024-04-26T00:39:04.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2cb9d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:04.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2cb9d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9d2809f2b0 con 0x7f9d28135490 2024-04-26T00:39:04.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2cb9d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:04.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2cb9d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9d2809fe90 con 0x7f9d280a0260 2024-04-26T00:39:04.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2cb9d640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:04.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2cb9d640 10 monclient(hunting): authenticate will time out at 2125.370501s 2024-04-26T00:39:04.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2659c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d280a0260 0x7f9d28130ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:04.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d25d9b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d28134b30 0x7f9d28131420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:04.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2659c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d280a0260 0x7f9d28130ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:35686/0 (socket says 172.21.15.37:35686) 2024-04-26T00:39:04.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2659c640 1 -- 172.21.15.37:0/929570035 learned_addr learned my addr 172.21.15.37:0/929570035 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:04.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2659c640 10 monclient(hunting): get_auth_request con 0x7f9d280a0260 auth_method 0 2024-04-26T00:39:04.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:04.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:04.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:04.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d25d9b640 10 monclient(hunting): get_auth_request con 0x7f9d28134b30 auth_method 0 2024-04-26T00:39:04.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d25d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:04.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d25d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:04.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d25d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:04.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:04.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:04.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d26d9d640 1 --2- 172.21.15.37:0/929570035 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d28135490 0x7f9d281396e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:04.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:04.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d25d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:04.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d25d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:04.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d25d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:04.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d26d9d640 10 monclient(hunting): get_auth_request con 0x7f9d28135490 auth_method 0 2024-04-26T00:39:04.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d26d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:04.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d26d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:04.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d26d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:04.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2659c640 10 monclient(hunting): handle_auth_done global_id 204419 payload 824 2024-04-26T00:39:04.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2659c640 1 -- 172.21.15.37:0/929570035 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d28135490 msgr2=0x7f9d281396e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:04.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2659c640 1 --2- 172.21.15.37:0/929570035 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d28135490 0x7f9d281396e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:04.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2659c640 1 -- 172.21.15.37:0/929570035 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d28134b30 msgr2=0x7f9d28131420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:04.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2659c640 1 --2- 172.21.15.37:0/929570035 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d28134b30 0x7f9d28131420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:04.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2659c640 10 monclient: _finish_hunting 0 2024-04-26T00:39:04.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2659c640 1 monclient: found mon.noname-c 2024-04-26T00:39:04.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:04.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:39:04.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d2659c640 1 -- 172.21.15.37:0/929570035 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d2813b730 con 0x7f9d280a0260 2024-04-26T00:39:04.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d26d9d640 1 --2- 172.21.15.37:0/929570035 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d28135490 0x7f9d281396e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:39:04.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.062+0000 7f9d25d9b640 1 --2- 172.21.15.37:0/929570035 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d28134b30 0x7f9d28131420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:39:04.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.066+0000 7f9d2659c640 1 -- 172.21.15.37:0/929570035 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d280a0260 msgr2=0x7f9d28130ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:04.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.066+0000 7f9d2659c640 1 --2- 172.21.15.37:0/929570035 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d280a0260 0x7f9d28130ee0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c00ebe0 tx=0x7f9d1c00c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:04.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.066+0000 7f9d16ffd640 1 -- 172.21.15.37:0/929570035 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d1c017070 con 0x7f9d280a0260 2024-04-26T00:39:04.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.066+0000 7f9d16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:04.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.066+0000 7f9d16ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:39:04.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.066+0000 7f9d16ffd640 10 monclient: dump: 2024-04-26T00:39:04.077 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:04.077 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:04.078 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:04.078 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:04.078 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:04.078 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:04.078 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:04.078 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:04.078 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:04.078 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:04.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.066+0000 7f9d16ffd640 10 monclient: _finish_auth 0 2024-04-26T00:39:04.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.066+0000 7f9d16ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:39:04.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.066+0000 7f9d16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:04.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.066+0000 7f9d16ffd640 1 -- 172.21.15.37:0/929570035 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d1c00ce40 con 0x7f9d280a0260 2024-04-26T00:39:04.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.066+0000 7f9d2cb9d640 5 monclient: authenticate success, global_id 204419 2024-04-26T00:39:04.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.066+0000 7f9d16ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:04.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.066+0000 7f9d16ffd640 1 -- 172.21.15.37:0/929570035 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d1c013690 con 0x7f9d280a0260 2024-04-26T00:39:04.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.066+0000 7f9d16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:04.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.066+0000 7f9d16ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:04.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.066+0000 7f9d16ffd640 10 monclient: dump: 2024-04-26T00:39:04.079 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:04.079 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:04.079 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:04.079 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:04.079 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:04.079 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:04.079 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:04.079 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:04.079 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:04.079 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:04.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.066+0000 7f9d24d99640 4 set_mon_vals no callback set 2024-04-26T00:39:04.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.066+0000 7f9d2cb9d640 10 monclient: _renew_subs 2024-04-26T00:39:04.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.066+0000 7f9d2cb9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:04.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.066+0000 7f9d2cb9d640 1 -- 172.21.15.37:0/929570035 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d280729d0 con 0x7f9d280a0260 2024-04-26T00:39:04.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.066+0000 7f9d2cb9d640 10 monclient: _renew_subs 2024-04-26T00:39:04.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.066+0000 7f9d2cb9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:04.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.066+0000 7f9d2cb9d640 1 -- 172.21.15.37:0/929570035 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d2811bc80 con 0x7f9d280a0260 2024-04-26T00:39:04.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.066+0000 7f9d2659c640 1 --2- 172.21.15.37:0/929570035 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d280a0260 0x7f9d28130ee0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c00ebe0 tx=0x7f9d1c00c520 comp rx=0 tx=0).queue_received will delay after 1.71409e+09 on 0x7f9d1c048eb0 osd_map(755..755 src has 705..755) 2024-04-26T00:39:04.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.066+0000 7f9d16ffd640 1 -- 172.21.15.37:0/929570035 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9d1c01a070 con 0x7f9d280a0260 2024-04-26T00:39:04.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.066+0000 7f9d2cb9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:04.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.066+0000 7f9d2cb9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:04.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.066+0000 7f9d2cb9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:04.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.066+0000 7f9d2cb9d640 1 -- 172.21.15.37:0/929570035 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d28066f40 con 0x7f9d280a0260 2024-04-26T00:39:04.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.066+0000 7f9d16ffd640 1 --2- 172.21.15.37:0/929570035 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9d08035ee0 0x7f9d080383a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:04.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.066+0000 7f9d25d9b640 1 --2- 172.21.15.37:0/929570035 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9d08035ee0 0x7f9d080383a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:04.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.066+0000 7f9d25d9b640 10 monclient: get_auth_request con 0x7f9d08035ee0 auth_method 0 2024-04-26T00:39:04.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.070+0000 7f9d25d9b640 1 --2- 172.21.15.37:0/929570035 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9d08035ee0 0x7f9d080383a0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f9d180097c0 tx=0x7f9d18008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:04.081 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.070+0000 7f9d16ffd640 1 -- 172.21.15.37:0/929570035 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(755..755 src has 705..755) ==== 3947+0+0 (secure 0 0 0) 0x7f9d1c048eb0 con 0x7f9d280a0260 2024-04-26T00:39:04.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.894+0000 7f9d16ffd640 1 -- 172.21.15.37:0/929570035 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9d1c047bf0 con 0x7f9d280a0260 2024-04-26T00:39:04.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.894+0000 7f9d16ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:04.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:04.894+0000 7f9d16ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:39:05.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:05.118+0000 7f9d2cb9d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-26T00:39:05.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:05.118+0000 7f9d2cb9d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-26T00:39:05.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:05.118+0000 7f9d2cb9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:05.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:05.118+0000 7f9d2cb9d640 1 -- 172.21.15.37:0/929570035 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f9d2813bf80 con 0x7f9d280a0260 2024-04-26T00:39:05.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:05.122+0000 7f9d16ffd640 1 -- 172.21.15.37:0/929570035 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f9d1c024510 con 0x7f9d280a0260 2024-04-26T00:39:05.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:05.122+0000 7f9d16ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-26T00:39:05.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:05.122+0000 7f9d16ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:39:05.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:05.126+0000 7f9d2cb9d640 1 -- 172.21.15.37:0/929570035 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9d08035ee0 msgr2=0x7f9d080383a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:05.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:05.126+0000 7f9d2cb9d640 1 --2- 172.21.15.37:0/929570035 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9d08035ee0 0x7f9d080383a0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f9d180097c0 tx=0x7f9d18008040 comp rx=0 tx=0).stop 2024-04-26T00:39:05.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:05.126+0000 7f9d2cb9d640 10 monclient: shutdown 2024-04-26T00:39:05.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:05.126+0000 7f9d2cb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:05.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:05.126+0000 7f9d2cb9d640 1 -- 172.21.15.37:0/929570035 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d280a0260 msgr2=0x7f9d28130ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:05.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:05.126+0000 7f9d2cb9d640 1 --2- 172.21.15.37:0/929570035 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d280a0260 0x7f9d28130ee0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c00ebe0 tx=0x7f9d1c00c520 comp rx=0 tx=0).stop 2024-04-26T00:39:05.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:05.126+0000 7f9d2cb9d640 1 -- 172.21.15.37:0/929570035 shutdown_connections 2024-04-26T00:39:05.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:05.126+0000 7f9d2cb9d640 1 --2- 172.21.15.37:0/929570035 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9d08035ee0 0x7f9d080383a0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:05.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:05.126+0000 7f9d2cb9d640 1 --2- 172.21.15.37:0/929570035 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d28135490 0x7f9d281396e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:05.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:05.126+0000 7f9d2cb9d640 1 --2- 172.21.15.37:0/929570035 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d28134b30 0x7f9d28131420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:05.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:05.126+0000 7f9d2cb9d640 1 --2- 172.21.15.37:0/929570035 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d280a0260 0x7f9d28130ee0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:05.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:05.126+0000 7f9d2cb9d640 1 -- 172.21.15.37:0/929570035 >> 172.21.15.37:0/929570035 conn(0x7f9d280a1ad0 msgr2=0x7f9d28071cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:05.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:05.126+0000 7f9d2cb9d640 1 -- 172.21.15.37:0/929570035 shutdown_connections 2024-04-26T00:39:05.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:05.126+0000 7f9d2cb9d640 1 -- 172.21.15.37:0/929570035 wait complete. 2024-04-26T00:39:05.129 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.130+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:05.129 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.130+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=12,kv=10,logm=48} crc {auth=1754543563,config=692619927,health=832402908,kv=1034996044,logm=2796780877}) 2024-04-26T00:39:05.129 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.130+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=10,kv=10,logm=52} crc {auth=3751383839,config=692619927,health=1387123127,kv=1034996044,logm=4018204216}) 2024-04-26T00:39:05.129 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.130+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:05.129 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.130+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=12,kv=10,logm=48} crc {auth=1754543563,config=692619927,health=832402908,kv=1034996044,logm=2796780877}) 2024-04-26T00:39:05.129 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.130+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=11,kv=9,logm=52} crc {auth=3173364986,config=692619927,health=2393717183,kv=2885089190,logm=1217652268}) 2024-04-26T00:39:05.130 INFO:tasks.rados.rados.0.smithi045.stdout:3346: done (1 left) 2024-04-26T00:39:05.130 INFO:tasks.rados.rados.0.smithi045.stdout:3347: done (0 left) 2024-04-26T00:39:05.130 INFO:tasks.rados.rados.0.smithi045.stdout:3348: rollback oid 2 current snap is 371 2024-04-26T00:39:05.130 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 2 to 337 2024-04-26T00:39:05.130 INFO:tasks.rados.rados.0.smithi045.stdout:3349: write oid 48 current snap is 371 2024-04-26T00:39:05.130 INFO:tasks.rados.rados.0.smithi045.stdout:3349: seq_num 1193 ranges {450751=723125,1945018=105733} 2024-04-26T00:39:05.138 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.138+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:05.138 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.138+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2602765002}) 2024-04-26T00:39:05.138 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.138+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=197400530}) 2024-04-26T00:39:05.138 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.138+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:05.139 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.138+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2602765002}) 2024-04-26T00:39:05.139 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.138+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=526048976}) 2024-04-26T00:39:05.144 INFO:tasks.rados.rados.0.smithi045.stdout:3349: writing smithi04526579-48 from 450751 to 1173876 tid 1 2024-04-26T00:39:05.155 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-26T00:39:05.156 INFO:tasks.rados.rados.0.smithi045.stdout:3349: writing smithi04526579-48 from 1945018 to 2050751 tid 2 2024-04-26T00:39:05.156 INFO:tasks.rados.rados.0.smithi045.stdout:3350: read oid 38 snap -1 2024-04-26T00:39:05.156 INFO:tasks.rados.rados.0.smithi045.stdout:3350: expect (ObjNum 1163 snap 359 seq_num 1163) 2024-04-26T00:39:05.156 INFO:tasks.rados.rados.0.smithi045.stdout:3351: read oid 36 snap -1 2024-04-26T00:39:05.156 INFO:tasks.rados.rados.0.smithi045.stdout:3351: expect (ObjNum 1118 snap 337 seq_num 1118) 2024-04-26T00:39:05.156 INFO:tasks.rados.rados.0.smithi045.stdout:3352: write (excl) oid 50 current snap is 371 2024-04-26T00:39:05.156 INFO:tasks.rados.rados.0.smithi045.stdout:3352: seq_num 1194 ranges {467558=514727,1719853=347705} 2024-04-26T00:39:05.167 INFO:tasks.rados.rados.0.smithi045.stdout:3352: writing smithi04526579-50 from 467558 to 982285 tid 1 2024-04-26T00:39:05.181 INFO:tasks.rados.rados.0.smithi045.stdout:3352: writing smithi04526579-50 from 1719853 to 2067558 tid 2 2024-04-26T00:39:05.182 INFO:tasks.rados.rados.0.smithi045.stdout:3353: copy_from oid 19 from oid 49 current snap is 371 2024-04-26T00:39:05.182 INFO:tasks.rados.rados.0.smithi045.stdout:3348: finishing rollback tid 0 to smithi04526579-2 2024-04-26T00:39:05.182 INFO:tasks.rados.rados.0.smithi045.stdout:3348: finishing rollback tid 1 to smithi04526579-2 2024-04-26T00:39:05.182 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 2494 (ObjNum 0 snap 0 seq_num 304) dirty dne 2024-04-26T00:39:05.182 INFO:tasks.rados.rados.0.smithi045.stdout:3351: expect (ObjNum 1118 snap 337 seq_num 1118) 2024-04-26T00:39:05.187 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.186+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:05.187 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.186+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1241812223}) 2024-04-26T00:39:05.187 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.186+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=962521439}) 2024-04-26T00:39:05.187 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.186+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:05.187 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.186+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1241812223}) 2024-04-26T00:39:05.187 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.186+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2691293405}) 2024-04-26T00:39:05.343 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.342+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:05.343 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.342+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4274942216}) 2024-04-26T00:39:05.343 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.342+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1648909490}) 2024-04-26T00:39:05.343 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.342+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:05.343 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.342+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4274942216}) 2024-04-26T00:39:05.343 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.342+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2096378480}) 2024-04-26T00:39:05.345 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.346+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:05.345 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.346+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=403604677}) 2024-04-26T00:39:05.345 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.346+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=115113002}) 2024-04-26T00:39:05.345 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.346+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:05.346 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.346+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=403604677}) 2024-04-26T00:39:05.346 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.346+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2195919034}) 2024-04-26T00:39:05.347 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.346+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:05.348 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.346+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4241871561}) 2024-04-26T00:39:05.348 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.346+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=640869323}) 2024-04-26T00:39:05.348 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.346+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:05.348 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.346+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4241871561}) 2024-04-26T00:39:05.348 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.346+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3460159040}) 2024-04-26T00:39:05.349 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.350+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:05.349 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.350+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3676291424}) 2024-04-26T00:39:05.350 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.350+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3024011720}) 2024-04-26T00:39:05.350 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.350+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:05.350 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.350+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3676291424}) 2024-04-26T00:39:05.350 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.350+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3204446218}) 2024-04-26T00:39:05.351 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.350+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:05.351 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.350+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1158976090}) 2024-04-26T00:39:05.351 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.350+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2201259806}) 2024-04-26T00:39:05.352 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.350+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:05.352 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.350+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1158976090}) 2024-04-26T00:39:05.352 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.350+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2796693867}) 2024-04-26T00:39:05.353 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.354+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:05.353 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.354+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3029469548}) 2024-04-26T00:39:05.354 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.354+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2840760040}) 2024-04-26T00:39:05.354 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.354+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:05.354 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.354+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3029469548}) 2024-04-26T00:39:05.354 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.354+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1300078905}) 2024-04-26T00:39:05.355 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.354+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:05.356 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.354+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3295619600}) 2024-04-26T00:39:05.356 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.354+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2073944734}) 2024-04-26T00:39:05.356 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.354+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:05.356 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.354+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3295619600}) 2024-04-26T00:39:05.356 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.354+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3271009458}) 2024-04-26T00:39:05.358 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.358+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:05.358 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.358+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=41028107}) 2024-04-26T00:39:05.358 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.358+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1770703174}) 2024-04-26T00:39:05.358 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.358+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:05.358 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.358+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=41028107}) 2024-04-26T00:39:05.358 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.358+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1169032211}) 2024-04-26T00:39:05.359 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.358+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:05.359 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.358+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2941325949}) 2024-04-26T00:39:05.360 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.358+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3447099512}) 2024-04-26T00:39:05.360 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.358+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:05.360 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.358+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2941325949}) 2024-04-26T00:39:05.360 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.358+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=77050306}) 2024-04-26T00:39:05.361 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.362+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:05.361 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.362+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2271351994}) 2024-04-26T00:39:05.362 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.362+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=190500292}) 2024-04-26T00:39:05.362 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.362+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:05.362 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.362+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2271351994}) 2024-04-26T00:39:05.362 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.362+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3845473556}) 2024-04-26T00:39:05.364 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.362+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:05.364 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.362+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=523116667}) 2024-04-26T00:39:05.364 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.362+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3184492382}) 2024-04-26T00:39:05.364 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.362+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:05.364 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.362+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=523116667}) 2024-04-26T00:39:05.364 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.362+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2952196389}) 2024-04-26T00:39:05.365 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.366+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:05.366 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.366+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2458527373}) 2024-04-26T00:39:05.366 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.366+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=418395755}) 2024-04-26T00:39:05.366 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.366+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:05.366 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.366+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2458527373}) 2024-04-26T00:39:05.366 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.366+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=857094545}) 2024-04-26T00:39:05.367 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.366+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:05.368 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.366+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=383093018}) 2024-04-26T00:39:05.368 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.366+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3344762903}) 2024-04-26T00:39:05.368 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.366+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:05.368 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.366+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=383093018}) 2024-04-26T00:39:05.368 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.366+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2933848531}) 2024-04-26T00:39:05.393 INFO:tasks.rados.rados.0.smithi045.stdout:3349: finishing write tid 1 to smithi04526579-48 2024-04-26T00:39:05.394 INFO:tasks.rados.rados.0.smithi045.stdout:3350: expect (ObjNum 1163 snap 359 seq_num 1163) 2024-04-26T00:39:05.475 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.474+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:05.475 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.474+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2271755558}) 2024-04-26T00:39:05.475 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.474+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1782806737}) 2024-04-26T00:39:05.475 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.474+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:05.475 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.474+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2271755558}) 2024-04-26T00:39:05.476 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.474+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4074091811}) 2024-04-26T00:39:05.553 INFO:tasks.rados.rados.0.smithi045.stdout:3349: finishing write tid 2 to smithi04526579-48 2024-04-26T00:39:05.553 INFO:tasks.rados.rados.0.smithi045.stdout:3349: finishing write tid 3 to smithi04526579-48 2024-04-26T00:39:05.553 INFO:tasks.rados.rados.0.smithi045.stdout:3349: finishing write tid 4 to smithi04526579-48 2024-04-26T00:39:05.553 INFO:tasks.rados.rados.0.smithi045.stdout:3349: oid 48 updating version 0 to 2454 2024-04-26T00:39:05.553 INFO:tasks.rados.rados.0.smithi045.stdout:3349: oid 48 updating version 2454 to 2455 2024-04-26T00:39:05.554 INFO:tasks.rados.rados.0.smithi045.stdout:3349: oid 48 updating version 2455 to 2456 2024-04-26T00:39:05.554 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 2456 (ObjNum 1193 snap 371 seq_num 1193) dirty exists 2024-04-26T00:39:05.554 INFO:tasks.rados.rados.0.smithi045.stdout:3349: left oid 48 (ObjNum 1193 snap 371 seq_num 1193) 2024-04-26T00:39:05.554 INFO:tasks.rados.rados.0.smithi045.stdout:3352: finishing write tid 1 to smithi04526579-50 2024-04-26T00:39:05.554 INFO:tasks.rados.rados.0.smithi045.stdout:3352: finishing write tid 2 to smithi04526579-50 2024-04-26T00:39:05.554 INFO:tasks.rados.rados.0.smithi045.stdout:3352: finishing write tid 3 to smithi04526579-50 2024-04-26T00:39:05.554 INFO:tasks.rados.rados.0.smithi045.stdout:3352: finishing write tid 4 to smithi04526579-50 2024-04-26T00:39:05.554 INFO:tasks.rados.rados.0.smithi045.stdout:3352: oid 50 updating version 0 to 2411 2024-04-26T00:39:05.554 INFO:tasks.rados.rados.0.smithi045.stdout:3352: oid 50 updating version 2411 to 2412 2024-04-26T00:39:05.554 INFO:tasks.rados.rados.0.smithi045.stdout:3352: oid 50 version 2412 is already newer than 2410 2024-04-26T00:39:05.554 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 2412 (ObjNum 1194 snap 371 seq_num 1194) dirty exists 2024-04-26T00:39:05.554 INFO:tasks.rados.rados.0.smithi045.stdout:3352: left oid 50 (ObjNum 1194 snap 371 seq_num 1194) 2024-04-26T00:39:05.555 INFO:tasks.rados.rados.0.smithi045.stdout:3353: finishing copy_from racing read to smithi04526579-19 2024-04-26T00:39:05.555 INFO:tasks.rados.rados.0.smithi045.stdout:3353: finishing copy_from to smithi04526579-19 2024-04-26T00:39:05.555 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 2364 (ObjNum 1177 snap 365 seq_num 1177) dirty exists 2024-04-26T00:39:05.555 INFO:tasks.rados.rados.0.smithi045.stdout:3348: done (5 left) 2024-04-26T00:39:05.555 INFO:tasks.rados.rados.0.smithi045.stdout:3349: done (4 left) 2024-04-26T00:39:05.555 INFO:tasks.rados.rados.0.smithi045.stdout:3350: done (3 left) 2024-04-26T00:39:05.555 INFO:tasks.rados.rados.0.smithi045.stdout:3351: done (2 left) 2024-04-26T00:39:05.555 INFO:tasks.rados.rados.0.smithi045.stdout:3352: done (1 left) 2024-04-26T00:39:05.555 INFO:tasks.rados.rados.0.smithi045.stdout:3353: done (0 left) 2024-04-26T00:39:05.555 INFO:tasks.rados.rados.0.smithi045.stdout:3354: copy_from oid 49 from oid 46 current snap is 371 2024-04-26T00:39:05.555 INFO:tasks.rados.rados.0.smithi045.stdout:3355: snap_create 2024-04-26T00:39:05.556 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.554+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:05.556 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.554+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3979901678}) 2024-04-26T00:39:05.556 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.554+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3054210162}) 2024-04-26T00:39:05.556 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.554+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:05.556 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.554+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3979901678}) 2024-04-26T00:39:05.556 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:05.554+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3355580461}) 2024-04-26T00:39:05.592 INFO:tasks.rados.rados.0.smithi045.stdout:3354: finishing copy_from racing read to smithi04526579-49 2024-04-26T00:39:05.592 INFO:tasks.rados.rados.0.smithi045.stdout:3354: finishing copy_from to smithi04526579-49 2024-04-26T00:39:05.592 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 2440 (ObjNum 1185 snap 368 seq_num 1185) dirty exists 2024-04-26T00:39:06.136 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.134+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.136 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.134+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1718394094}) 2024-04-26T00:39:06.136 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.134+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4236252660}) 2024-04-26T00:39:06.136 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.134+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.136 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.134+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1718394094}) 2024-04-26T00:39:06.136 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.134+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3792311075}) 2024-04-26T00:39:06.149 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.150+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.150 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.150+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2289829331}) 2024-04-26T00:39:06.150 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.150+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2313739419}) 2024-04-26T00:39:06.150 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.150+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.150 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.150+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2289829331}) 2024-04-26T00:39:06.150 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.150+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3030855726}) 2024-04-26T00:39:06.156 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:39:06.156 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:39:06.160 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.158+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.160 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.158+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1941183734}) 2024-04-26T00:39:06.160 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.158+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1978867816}) 2024-04-26T00:39:06.160 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.158+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.160 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.158+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1941183734}) 2024-04-26T00:39:06.160 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.158+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3221661959}) 2024-04-26T00:39:06.161 INFO:tasks.rados.rados.0.smithi045.stdout:3354: done (1 left) 2024-04-26T00:39:06.161 INFO:tasks.rados.rados.0.smithi045.stdout:3355: done (0 left) 2024-04-26T00:39:06.162 INFO:tasks.rados.rados.0.smithi045.stdout:3356: copy_from oid 25 from oid 38 current snap is 372 2024-04-26T00:39:06.162 INFO:tasks.rados.rados.0.smithi045.stdout:3357: snap_remove snap 366 2024-04-26T00:39:06.162 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.162+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.163 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.162+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3880142895}) 2024-04-26T00:39:06.163 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.162+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2127927484}) 2024-04-26T00:39:06.163 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.162+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.163 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.162+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3880142895}) 2024-04-26T00:39:06.163 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.162+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3897707350}) 2024-04-26T00:39:06.165 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.166+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.165 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.166+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3247380283}) 2024-04-26T00:39:06.165 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.166+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1204588727}) 2024-04-26T00:39:06.165 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.166+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.165 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.166+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3247380283}) 2024-04-26T00:39:06.165 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.166+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2138705297}) 2024-04-26T00:39:06.170 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.170+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.170 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.170+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3806020907}) 2024-04-26T00:39:06.170 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.170+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2584252030}) 2024-04-26T00:39:06.170 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.170+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.170 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.170+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3806020907}) 2024-04-26T00:39:06.170 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.170+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3399999351}) 2024-04-26T00:39:06.174 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.174+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.174 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.174+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3318113606}) 2024-04-26T00:39:06.174 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.174+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1041243572}) 2024-04-26T00:39:06.174 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.174+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.174 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.174+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3318113606}) 2024-04-26T00:39:06.174 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.174+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3272469906}) 2024-04-26T00:39:06.177 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.178+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.177 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.178+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2575377158}) 2024-04-26T00:39:06.177 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.178+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2781088142}) 2024-04-26T00:39:06.177 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.178+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.177 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.178+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2575377158}) 2024-04-26T00:39:06.177 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.178+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3504414370}) 2024-04-26T00:39:06.184 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.182+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.184 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.182+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=903817726}) 2024-04-26T00:39:06.184 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.182+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=377890079}) 2024-04-26T00:39:06.184 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.182+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.184 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.182+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=903817726}) 2024-04-26T00:39:06.184 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.182+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2406868791}) 2024-04-26T00:39:06.192 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.194+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.192 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.194+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1151522817}) 2024-04-26T00:39:06.193 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.194+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1323151150}) 2024-04-26T00:39:06.193 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.194+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.193 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.194+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1151522817}) 2024-04-26T00:39:06.193 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.194+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3928847687}) 2024-04-26T00:39:06.200 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.202+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.200 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.202+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3208975910}) 2024-04-26T00:39:06.200 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.202+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=345590054}) 2024-04-26T00:39:06.200 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.202+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.201 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.202+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3208975910}) 2024-04-26T00:39:06.201 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.202+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2359034010}) 2024-04-26T00:39:06.203 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.202+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.203 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.202+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1220416736}) 2024-04-26T00:39:06.203 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.202+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2936370458}) 2024-04-26T00:39:06.203 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.202+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.203 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.202+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1220416736}) 2024-04-26T00:39:06.203 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.202+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=839382894}) 2024-04-26T00:39:06.205 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.206+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.205 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.206+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2917400476}) 2024-04-26T00:39:06.205 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.206+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1515034923}) 2024-04-26T00:39:06.205 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.206+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.205 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.206+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2917400476}) 2024-04-26T00:39:06.205 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.206+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=778513611}) 2024-04-26T00:39:06.206 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.206+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.206 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.206+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2246546909}) 2024-04-26T00:39:06.206 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.206+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2495175067}) 2024-04-26T00:39:06.206 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.206+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.206 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.206+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2246546909}) 2024-04-26T00:39:06.207 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.206+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1487136035}) 2024-04-26T00:39:06.208 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.206+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.208 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.206+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2643920990}) 2024-04-26T00:39:06.208 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.206+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1381128749}) 2024-04-26T00:39:06.208 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.210+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.208 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.210+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2643920990}) 2024-04-26T00:39:06.208 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.210+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3397146121}) 2024-04-26T00:39:06.210 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.210+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.210 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.210+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2380290970}) 2024-04-26T00:39:06.210 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.210+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3379711215}) 2024-04-26T00:39:06.210 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.210+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.210 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.210+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2380290970}) 2024-04-26T00:39:06.210 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.210+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=522939916}) 2024-04-26T00:39:06.211 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.210+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.212 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.210+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=784547198}) 2024-04-26T00:39:06.212 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.210+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2205318994}) 2024-04-26T00:39:06.212 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.210+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.212 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.210+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=784547198}) 2024-04-26T00:39:06.212 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.210+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=826133097}) 2024-04-26T00:39:06.213 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.214+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.213 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.214+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1810236939}) 2024-04-26T00:39:06.214 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.214+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4234782155}) 2024-04-26T00:39:06.214 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.214+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.214 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.214+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1810236939}) 2024-04-26T00:39:06.214 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.214+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=737557465}) 2024-04-26T00:39:06.215 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.214+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.215 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.214+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2764559244}) 2024-04-26T00:39:06.215 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.214+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3397778273}) 2024-04-26T00:39:06.216 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.214+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.216 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.214+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2764559244}) 2024-04-26T00:39:06.216 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.214+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=88459731}) 2024-04-26T00:39:06.217 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.218+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.217 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.218+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2023936686}) 2024-04-26T00:39:06.217 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.218+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1130946000}) 2024-04-26T00:39:06.217 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.218+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.217 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.218+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2023936686}) 2024-04-26T00:39:06.217 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.218+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2738280751}) 2024-04-26T00:39:06.219 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.218+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.219 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.218+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1528981006}) 2024-04-26T00:39:06.219 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.218+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=847121506}) 2024-04-26T00:39:06.219 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.218+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.219 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.218+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1528981006}) 2024-04-26T00:39:06.219 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.218+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=341427917}) 2024-04-26T00:39:06.221 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.222+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.221 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.222+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3933138531}) 2024-04-26T00:39:06.221 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.222+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=241988846}) 2024-04-26T00:39:06.221 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.222+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.222 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.222+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3933138531}) 2024-04-26T00:39:06.222 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.222+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1174491903}) 2024-04-26T00:39:06.222 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.222+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.222 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.222+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3701272674}) 2024-04-26T00:39:06.223 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.222+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=875407913}) 2024-04-26T00:39:06.223 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.222+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.223 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.222+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3701272674}) 2024-04-26T00:39:06.223 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.222+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3724201204}) 2024-04-26T00:39:06.224 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.226+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.225 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.226+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1568810679}) 2024-04-26T00:39:06.225 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.226+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=266654407}) 2024-04-26T00:39:06.225 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.226+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.225 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.226+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1568810679}) 2024-04-26T00:39:06.225 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.226+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=643401194}) 2024-04-26T00:39:06.227 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.226+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.227 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.226+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2859098099}) 2024-04-26T00:39:06.227 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.226+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1915395189}) 2024-04-26T00:39:06.227 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.226+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.227 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.226+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2859098099}) 2024-04-26T00:39:06.227 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.226+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2756512294}) 2024-04-26T00:39:06.228 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.230+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.228 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.230+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2355070263}) 2024-04-26T00:39:06.228 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.230+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3833120998}) 2024-04-26T00:39:06.229 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.230+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.229 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.230+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2355070263}) 2024-04-26T00:39:06.229 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.230+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1483540992}) 2024-04-26T00:39:06.230 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.230+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.230 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.230+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=667063553}) 2024-04-26T00:39:06.230 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.230+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2280074606}) 2024-04-26T00:39:06.230 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.230+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.230 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.230+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=667063553}) 2024-04-26T00:39:06.230 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.230+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3890365820}) 2024-04-26T00:39:06.232 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.230+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.232 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.230+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=317222065}) 2024-04-26T00:39:06.232 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.230+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2415592102}) 2024-04-26T00:39:06.232 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.230+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.232 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.230+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=317222065}) 2024-04-26T00:39:06.232 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.230+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2015571154}) 2024-04-26T00:39:06.234 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.234+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.234 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.234+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3554277454}) 2024-04-26T00:39:06.234 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.234+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=689782845}) 2024-04-26T00:39:06.234 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.234+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.234 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.234+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3554277454}) 2024-04-26T00:39:06.234 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.234+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3605854339}) 2024-04-26T00:39:06.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.302+0000 7f8580dce640 10 monclient: get_monmap_and_config 2024-04-26T00:39:06.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.302+0000 7f8580dce640 10 monclient: build_initial_monmap 2024-04-26T00:39:06.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.302+0000 7f8580dce640 10 monclient: monmap: 2024-04-26T00:39:06.302 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:06.302 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:06.302 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:39:06.305160+0000 2024-04-26T00:39:06.302 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:39:06.305160+0000 2024-04-26T00:39:06.302 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:06.302 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:06.302 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:39:06.302 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:39:06.302 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:39:06.302 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:39:06.302 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:39:06.302 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:06.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.302+0000 7f8580dce640 1 Processor -- start 2024-04-26T00:39:06.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.302+0000 7f8580dce640 1 -- start start 2024-04-26T00:39:06.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.302+0000 7f8580dce640 10 monclient: init 2024-04-26T00:39:06.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.302+0000 7f8580dce640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:06.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.302+0000 7f8580dce640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-04-26T00:39:06.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.302+0000 7f8580dce640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f857c136e10 0x7f857c09c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:06.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.302+0000 7f8580dce640 10 monclient(hunting): picked mon.noname-b con 0x7f857c136e10 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:06.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.302+0000 7f8580dce640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f857c1371f0 0x7f857c1375d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:06.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.302+0000 7f8580dce640 10 monclient(hunting): picked mon.noname-a con 0x7f857c1371f0 addr v2:172.21.15.37:3300/0 2024-04-26T00:39:06.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.302+0000 7f8580dce640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f857c135d90 addr v1:172.21.15.37:6789/0 2024-04-26T00:39:06.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.302+0000 7f8580dce640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f857c09b680 con 0x7f857c135d90 2024-04-26T00:39:06.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.302+0000 7f8580dce640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:06.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.302+0000 7f8580dce640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f857c09b8f0 con 0x7f857c1371f0 2024-04-26T00:39:06.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.302+0000 7f8580dce640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:06.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.302+0000 7f8580dce640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f857c09bbd0 con 0x7f857c136e10 2024-04-26T00:39:06.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.302+0000 7f8580dce640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:06.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.302+0000 7f8580dce640 10 monclient(hunting): authenticate will time out at 2127.611619s 2024-04-26T00:39:06.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.302+0000 7f8573fff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f857c1371f0 0x7f857c1375d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:06.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.302+0000 7f857a59c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f857c136e10 0x7f857c09c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:06.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.302+0000 7f857ad9d640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f857c135d90 0x7f857c136240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:52852/0 (socket says 172.21.15.37:52852) 2024-04-26T00:39:06.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.302+0000 7f857ad9d640 1 -- 172.21.15.37:0/2272685857 learned_addr learned my addr 172.21.15.37:0/2272685857 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:39:06.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.302+0000 7f8573fff640 10 monclient(hunting): get_auth_request con 0x7f857c1371f0 auth_method 0 2024-04-26T00:39:06.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.302+0000 7f8573fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:06.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.302+0000 7f8573fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:06.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.302+0000 7f8573fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:06.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.302+0000 7f857a59c640 10 monclient(hunting): get_auth_request con 0x7f857c136e10 auth_method 0 2024-04-26T00:39:06.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.302+0000 7f857a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:06.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.302+0000 7f857a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:06.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.302+0000 7f857a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:06.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.302+0000 7f8573fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:06.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.302+0000 7f8573fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:06.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.306+0000 7f8573fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:06.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.306+0000 7f857a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:06.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.306+0000 7f857a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:06.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.306+0000 7f857a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:06.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.306+0000 7f8573fff640 10 monclient(hunting): handle_auth_done global_id 234126 payload 274 2024-04-26T00:39:06.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.306+0000 7f8573fff640 1 -- 172.21.15.37:0/2272685857 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f857c136e10 msgr2=0x7f857c09c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:06.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.306+0000 7f8573fff640 1 --2- 172.21.15.37:0/2272685857 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f857c136e10 0x7f857c09c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:06.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.306+0000 7f8573fff640 1 -- 172.21.15.37:0/2272685857 >> v1:172.21.15.37:6789/0 conn(0x7f857c135d90 legacy=0x7f857c136240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:06.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.306+0000 7f8573fff640 10 monclient: _finish_hunting 0 2024-04-26T00:39:06.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.306+0000 7f8573fff640 1 monclient: found mon.noname-a 2024-04-26T00:39:06.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.306+0000 7f8573fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:06.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.306+0000 7f8573fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:39:06.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.306+0000 7f8573fff640 1 -- 172.21.15.37:0/2272685857 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f857c137b10 con 0x7f857c1371f0 2024-04-26T00:39:06.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.306+0000 7f857a59c640 1 --2- 172.21.15.37:0/2272685857 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f857c136e10 0x7f857c09c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:39:06.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8573fff640 1 -- 172.21.15.37:0/2272685857 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f857c1371f0 msgr2=0x7f857c1375d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:06.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8573fff640 1 --2- 172.21.15.37:0/2272685857 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f857c1371f0 0x7f857c1375d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f85640097c0 tx=0x7f856401c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80f4b5f07fd8f8d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:06.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8579d9b640 1 -- 172.21.15.37:0/2272685857 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85640b0070 con 0x7f857c1371f0 2024-04-26T00:39:06.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8579d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:06.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8579d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:39:06.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8579d9b640 10 monclient: dump: 2024-04-26T00:39:06.309 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:06.309 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:06.309 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:06.309 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:06.309 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:06.309 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:06.309 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:06.309 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:06.309 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:06.310 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:06.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8579d9b640 10 monclient: _finish_auth 0 2024-04-26T00:39:06.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8579d9b640 10 monclient: _check_auth_tickets 2024-04-26T00:39:06.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8579d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:06.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8579d9b640 1 -- 172.21.15.37:0/2272685857 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85640ab600 con 0x7f857c1371f0 2024-04-26T00:39:06.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8579d9b640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:06.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8579d9b640 1 -- 172.21.15.37:0/2272685857 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85640ab9d0 con 0x7f857c1371f0 2024-04-26T00:39:06.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8580dce640 5 monclient: authenticate success, global_id 234126 2024-04-26T00:39:06.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8580dce640 10 monclient: get_monmap_and_config success 2024-04-26T00:39:06.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8580dce640 4 set_mon_vals no callback set 2024-04-26T00:39:06.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8579d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:06.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8579d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:39:06.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8579d9b640 10 monclient: dump: 2024-04-26T00:39:06.310 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:06.310 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:06.311 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:06.311 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:06.311 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:06.311 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:06.311 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:06.311 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:06.311 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:06.311 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:06.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8580dce640 10 monclient: shutdown 2024-04-26T00:39:06.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8580dce640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:06.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8580dce640 1 -- 172.21.15.37:0/2272685857 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f857c1371f0 msgr2=0x7f857c1375d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:06.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8580dce640 1 --2- 172.21.15.37:0/2272685857 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f857c1371f0 0x7f857c1375d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f85640097c0 tx=0x7f856401c320 comp rx=0 tx=0).stop 2024-04-26T00:39:06.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8580dce640 1 -- 172.21.15.37:0/2272685857 shutdown_connections 2024-04-26T00:39:06.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8580dce640 1 --2- 172.21.15.37:0/2272685857 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f857c1371f0 0x7f857c1375d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:06.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8580dce640 1 --2- 172.21.15.37:0/2272685857 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f857c136e10 0x7f857c09c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:06.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8580dce640 1 -- 172.21.15.37:0/2272685857 >> 172.21.15.37:0/2272685857 conn(0x7f857c0a64e0 msgr2=0x7f857c0a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:06.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8580dce640 1 -- 172.21.15.37:0/2272685857 shutdown_connections 2024-04-26T00:39:06.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8580dce640 1 -- 172.21.15.37:0/2272685857 wait complete. 2024-04-26T00:39:06.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8580dce640 10 monclient: build_initial_monmap 2024-04-26T00:39:06.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8580dce640 10 monclient: monmap: 2024-04-26T00:39:06.314 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:06.314 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:39:06.314 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:39:06.314 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:39:06.314 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:06.315 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:39:06.315 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:39:06.315 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:39:06.315 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:39:06.315 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:06.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8580dce640 1 Processor -- start 2024-04-26T00:39:06.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8580dce640 1 -- start start 2024-04-26T00:39:06.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8580dce640 10 monclient: init 2024-04-26T00:39:06.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8580dce640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:06.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8580dce640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:39:06.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8580dce640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f857c135d90 0x7f857c1cf2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:06.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8580dce640 10 monclient(hunting): picked mon.noname-c con 0x7f857c135d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:06.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8580dce640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f857c136e10 0x7f857c134d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:06.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8580dce640 10 monclient(hunting): picked mon.noname-b con 0x7f857c136e10 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:39:06.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8580dce640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f857c1371f0 0x7f857c13aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:06.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8580dce640 10 monclient(hunting): picked mon.noname-a con 0x7f857c1371f0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:39:06.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8580dce640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:06.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8580dce640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f857c141f10 con 0x7f857c1371f0 2024-04-26T00:39:06.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8580dce640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:06.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8580dce640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f857c141d90 con 0x7f857c136e10 2024-04-26T00:39:06.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8580dce640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:06.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8580dce640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f857c142090 con 0x7f857c135d90 2024-04-26T00:39:06.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8580dce640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:06.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8580dce640 10 monclient(hunting): authenticate will time out at 2127.618973s 2024-04-26T00:39:06.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f857a59c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f857c135d90 0x7f857c1cf2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:06.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f857a59c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f857c135d90 0x7f857c1cf2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:35726/0 (socket says 172.21.15.37:35726) 2024-04-26T00:39:06.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f857a59c640 1 -- 172.21.15.37:0/142495540 learned_addr learned my addr 172.21.15.37:0/142495540 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:06.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f857ad9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f857c1371f0 0x7f857c13aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:06.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f857a59c640 10 monclient(hunting): get_auth_request con 0x7f857c135d90 auth_method 0 2024-04-26T00:39:06.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f857a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:06.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f857a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:06.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f857a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:06.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f857ad9d640 10 monclient(hunting): get_auth_request con 0x7f857c1371f0 auth_method 0 2024-04-26T00:39:06.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f857ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:06.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f857ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:06.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f857ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:06.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f857a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:06.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f857a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:06.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f857a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:06.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f857ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:06.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f857ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:06.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f857ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:06.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f857a59c640 10 monclient(hunting): handle_auth_done global_id 204425 payload 824 2024-04-26T00:39:06.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f8573fff640 1 --2- 172.21.15.37:0/142495540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f857c136e10 0x7f857c134d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:06.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f857a59c640 1 -- 172.21.15.37:0/142495540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f857c136e10 msgr2=0x7f857c134d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:06.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f857a59c640 1 --2- 172.21.15.37:0/142495540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f857c136e10 0x7f857c134d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:06.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f857a59c640 1 -- 172.21.15.37:0/142495540 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f857c1371f0 msgr2=0x7f857c13aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:06.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f857a59c640 1 --2- 172.21.15.37:0/142495540 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f857c1371f0 0x7f857c13aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:06.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f857a59c640 10 monclient: _finish_hunting 0 2024-04-26T00:39:06.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f857a59c640 1 monclient: found mon.noname-c 2024-04-26T00:39:06.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f857a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:06.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f857a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:39:06.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f857a59c640 1 -- 172.21.15.37:0/142495540 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f857c13e940 con 0x7f857c135d90 2024-04-26T00:39:06.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.310+0000 7f857ad9d640 1 --2- 172.21.15.37:0/142495540 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f857c1371f0 0x7f857c13aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:39:06.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.314+0000 7f857a59c640 1 -- 172.21.15.37:0/142495540 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f857c135d90 msgr2=0x7f857c1cf2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:06.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.314+0000 7f857a59c640 1 --2- 172.21.15.37:0/142495540 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f857c135d90 0x7f857c1cf2e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f856c00ebe0 tx=0x7f856c00c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:06.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.314+0000 7f8572ffd640 1 -- 172.21.15.37:0/142495540 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f856c017070 con 0x7f857c135d90 2024-04-26T00:39:06.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.314+0000 7f8572ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:06.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.314+0000 7f8572ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:39:06.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.314+0000 7f8572ffd640 10 monclient: dump: 2024-04-26T00:39:06.320 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:06.320 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:06.320 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:06.320 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:06.320 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:06.320 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:06.321 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:06.321 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:06.321 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:06.321 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:06.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.314+0000 7f8572ffd640 10 monclient: _finish_auth 0 2024-04-26T00:39:06.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.314+0000 7f8572ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:39:06.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.314+0000 7f8572ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:06.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.314+0000 7f8580dce640 5 monclient: authenticate success, global_id 204425 2024-04-26T00:39:06.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.314+0000 7f8572ffd640 1 -- 172.21.15.37:0/142495540 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f856c00ce40 con 0x7f857c135d90 2024-04-26T00:39:06.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.314+0000 7f8572ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:06.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.314+0000 7f8572ffd640 1 -- 172.21.15.37:0/142495540 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f856c013690 con 0x7f857c135d90 2024-04-26T00:39:06.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.314+0000 7f8580dce640 10 monclient: _renew_subs 2024-04-26T00:39:06.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.314+0000 7f8580dce640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:06.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.314+0000 7f8580dce640 1 -- 172.21.15.37:0/142495540 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f857c09aca0 con 0x7f857c135d90 2024-04-26T00:39:06.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.314+0000 7f857959a640 4 set_mon_vals no callback set 2024-04-26T00:39:06.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.314+0000 7f8572ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:06.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.314+0000 7f8572ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:06.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.314+0000 7f8572ffd640 10 monclient: dump: 2024-04-26T00:39:06.322 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:06.322 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:06.322 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:06.322 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:06.322 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:06.322 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:06.322 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:06.322 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:06.322 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:06.322 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:06.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.314+0000 7f8580dce640 10 monclient: _renew_subs 2024-04-26T00:39:06.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.314+0000 7f8580dce640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:06.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.314+0000 7f8580dce640 1 -- 172.21.15.37:0/142495540 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f857c13e660 con 0x7f857c135d90 2024-04-26T00:39:06.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.314+0000 7f8580dce640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:06.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.314+0000 7f8580dce640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:06.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.314+0000 7f8580dce640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:06.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.314+0000 7f8580dce640 1 -- 172.21.15.37:0/142495540 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f857c075910 con 0x7f857c135d90 2024-04-26T00:39:06.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.314+0000 7f8572ffd640 1 -- 172.21.15.37:0/142495540 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f856c01a070 con 0x7f857c135d90 2024-04-26T00:39:06.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.314+0000 7f8572ffd640 1 --2- 172.21.15.37:0/142495540 >> v2:172.21.15.45:6812/2379918623 conn(0x7f855c035e80 0x7f855c038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:06.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.314+0000 7f8572ffd640 1 -- 172.21.15.37:0/142495540 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(757..757 src has 705..757) ==== 3915+0+0 (secure 0 0 0) 0x7f856c048b30 con 0x7f857c135d90 2024-04-26T00:39:06.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.314+0000 7f8573fff640 1 --2- 172.21.15.37:0/142495540 >> v2:172.21.15.45:6812/2379918623 conn(0x7f855c035e80 0x7f855c038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:06.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.314+0000 7f8573fff640 10 monclient: get_auth_request con 0x7f855c035e80 auth_method 0 2024-04-26T00:39:06.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.318+0000 7f8573fff640 1 --2- 172.21.15.37:0/142495540 >> v2:172.21.15.45:6812/2379918623 conn(0x7f855c035e80 0x7f855c038340 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f857c09d490 tx=0x7f8564008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:06.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.322+0000 7f8572ffd640 1 -- 172.21.15.37:0/142495540 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f856c011510 con 0x7f857c135d90 2024-04-26T00:39:06.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.322+0000 7f8572ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:06.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.322+0000 7f8572ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:39:06.450 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.450+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.451 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.450+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=964351714}) 2024-04-26T00:39:06.451 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.450+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2677265658}) 2024-04-26T00:39:06.451 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.450+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.451 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.450+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=964351714}) 2024-04-26T00:39:06.451 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.450+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=348551454}) 2024-04-26T00:39:06.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.542+0000 7f8580dce640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:39:06.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.542+0000 7f8580dce640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:39:06.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.542+0000 7f8580dce640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:06.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.542+0000 7f8580dce640 1 -- 172.21.15.37:0/142495540 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f857c13ce90 con 0x7f857c135d90 2024-04-26T00:39:06.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.546+0000 7f8572ffd640 1 -- 172.21.15.37:0/142495540 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f856c013d50 con 0x7f857c135d90 2024-04-26T00:39:06.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.546+0000 7f8572ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:39:06.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.546+0000 7f8572ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:39:06.544 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":144,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:39:06.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.546+0000 7f8580dce640 1 -- 172.21.15.37:0/142495540 >> v2:172.21.15.45:6812/2379918623 conn(0x7f855c035e80 msgr2=0x7f855c038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:06.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.546+0000 7f8580dce640 1 --2- 172.21.15.37:0/142495540 >> v2:172.21.15.45:6812/2379918623 conn(0x7f855c035e80 0x7f855c038340 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f857c09d490 tx=0x7f8564008040 comp rx=0 tx=0).stop 2024-04-26T00:39:06.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.546+0000 7f8580dce640 10 monclient: shutdown 2024-04-26T00:39:06.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.546+0000 7f8580dce640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:06.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.546+0000 7f8580dce640 1 -- 172.21.15.37:0/142495540 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f857c135d90 msgr2=0x7f857c1cf2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:06.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.546+0000 7f8580dce640 1 --2- 172.21.15.37:0/142495540 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f857c135d90 0x7f857c1cf2e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f856c00ebe0 tx=0x7f856c00c520 comp rx=0 tx=0).stop 2024-04-26T00:39:06.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.546+0000 7f8580dce640 1 -- 172.21.15.37:0/142495540 shutdown_connections 2024-04-26T00:39:06.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.546+0000 7f8580dce640 1 --2- 172.21.15.37:0/142495540 >> v2:172.21.15.45:6812/2379918623 conn(0x7f855c035e80 0x7f855c038340 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:06.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.546+0000 7f8580dce640 1 --2- 172.21.15.37:0/142495540 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f857c1371f0 0x7f857c13aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:06.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.546+0000 7f8580dce640 1 --2- 172.21.15.37:0/142495540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f857c136e10 0x7f857c134d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:06.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.546+0000 7f8580dce640 1 --2- 172.21.15.37:0/142495540 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f857c135d90 0x7f857c1cf2e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:06.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.546+0000 7f8580dce640 1 -- 172.21.15.37:0/142495540 >> 172.21.15.37:0/142495540 conn(0x7f857c0a64e0 msgr2=0x7f857c0a86b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:06.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.546+0000 7f8580dce640 1 -- 172.21.15.37:0/142495540 shutdown_connections 2024-04-26T00:39:06.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.546+0000 7f8580dce640 1 -- 172.21.15.37:0/142495540 wait complete. 2024-04-26T00:39:06.563 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:39:06.563 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-26T00:39:06.563 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:39:06.563 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:39:06.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.734+0000 7f28991d2640 10 monclient: get_monmap_and_config 2024-04-26T00:39:06.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.734+0000 7f28991d2640 10 monclient: build_initial_monmap 2024-04-26T00:39:06.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.734+0000 7f28991d2640 10 monclient: monmap: 2024-04-26T00:39:06.733 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:06.733 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:06.733 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:39:06.736558+0000 2024-04-26T00:39:06.733 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:39:06.736558+0000 2024-04-26T00:39:06.733 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:06.734 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:06.734 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:39:06.734 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:39:06.734 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:39:06.734 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:39:06.734 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:39:06.734 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:06.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.734+0000 7f28991d2640 1 Processor -- start 2024-04-26T00:39:06.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.734+0000 7f28991d2640 1 -- start start 2024-04-26T00:39:06.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.734+0000 7f28991d2640 10 monclient: init 2024-04-26T00:39:06.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.734+0000 7f28991d2640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:06.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.734+0000 7f28991d2640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-04-26T00:39:06.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.734+0000 7f28991d2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f28940a2c80 addr v1:172.21.15.45:6789/0 2024-04-26T00:39:06.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.734+0000 7f28991d2640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f28940a1170 0x7f289413be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:06.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.734+0000 7f28991d2640 10 monclient(hunting): picked mon.noname-c con 0x7f28940a1170 addr v2:172.21.15.45:3300/0 2024-04-26T00:39:06.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.734+0000 7f28991d2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f289413c390 addr v1:172.21.15.37:6789/0 2024-04-26T00:39:06.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.734+0000 7f28991d2640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f289409e770 con 0x7f289413c390 2024-04-26T00:39:06.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.734+0000 7f28991d2640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f289409ea10 con 0x7f28940a2c80 2024-04-26T00:39:06.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.734+0000 7f28991d2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:06.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.734+0000 7f28991d2640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f289409f040 con 0x7f28940a1170 2024-04-26T00:39:06.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.734+0000 7f28991d2640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:06.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.734+0000 7f28991d2640 10 monclient(hunting): authenticate will time out at 2128.043051s 2024-04-26T00:39:06.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.734+0000 7f289359e640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f289413c390 0x7f289413e780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:52856/0 (socket says 172.21.15.37:52856) 2024-04-26T00:39:06.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.734+0000 7f289359e640 1 -- 172.21.15.37:0/4055884299 learned_addr learned my addr 172.21.15.37:0/4055884299 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:39:06.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.734+0000 7f288a59c640 1 --2- 172.21.15.37:0/4055884299 >> v2:172.21.15.45:3300/0 conn(0x7f28940a1170 0x7f289413be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:06.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.734+0000 7f288a59c640 10 monclient(hunting): get_auth_request con 0x7f28940a1170 auth_method 0 2024-04-26T00:39:06.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.734+0000 7f288a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:06.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.734+0000 7f288a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:06.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.734+0000 7f288a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:06.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.734+0000 7f288a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:06.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.734+0000 7f288a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:06.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.734+0000 7f288a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:06.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.738+0000 7f288a59c640 10 monclient(hunting): handle_auth_done global_id 224200 payload 274 2024-04-26T00:39:06.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.738+0000 7f288a59c640 1 -- 172.21.15.37:0/4055884299 >> v1:172.21.15.45:6789/0 conn(0x7f28940a2c80 legacy=0x7f289409f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:06.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.738+0000 7f288a59c640 1 -- 172.21.15.37:0/4055884299 >> v1:172.21.15.37:6789/0 conn(0x7f289413c390 legacy=0x7f289413e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:06.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.738+0000 7f288a59c640 10 monclient: _finish_hunting 0 2024-04-26T00:39:06.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.738+0000 7f288a59c640 1 monclient: found mon.noname-c 2024-04-26T00:39:06.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.738+0000 7f288a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:06.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.738+0000 7f288a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:39:06.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.738+0000 7f288a59c640 1 -- 172.21.15.37:0/4055884299 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28940a15a0 con 0x7f28940a1170 2024-04-26T00:39:06.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.742+0000 7f288a59c640 1 -- 172.21.15.37:0/4055884299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28940a1170 msgr2=0x7f289413be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:06.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.742+0000 7f288a59c640 1 --2- 172.21.15.37:0/4055884299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28940a1170 0x7f289413be50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f287c0097c0 tx=0x7f287c01c2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=869c8168ff7878f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:06.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.742+0000 7f289259c640 1 -- 172.21.15.37:0/4055884299 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f287c0b0070 con 0x7f28940a1170 2024-04-26T00:39:06.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.742+0000 7f289259c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:06.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.742+0000 7f289259c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:39:06.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.742+0000 7f289259c640 10 monclient: dump: 2024-04-26T00:39:06.743 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:06.743 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:06.743 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:06.743 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:06.743 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:06.743 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:06.743 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:06.743 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:06.743 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:06.743 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:06.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.742+0000 7f289259c640 10 monclient: _finish_auth 0 2024-04-26T00:39:06.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.742+0000 7f289259c640 10 monclient: _check_auth_tickets 2024-04-26T00:39:06.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.742+0000 7f289259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:06.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.742+0000 7f289259c640 1 -- 172.21.15.37:0/4055884299 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f287c0a97f0 con 0x7f28940a1170 2024-04-26T00:39:06.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.742+0000 7f289259c640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:06.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.742+0000 7f289259c640 1 -- 172.21.15.37:0/4055884299 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f287c0a9bc0 con 0x7f28940a1170 2024-04-26T00:39:06.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.742+0000 7f289259c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:06.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.742+0000 7f289259c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:39:06.744 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.742+0000 7f289259c640 10 monclient: dump: 2024-04-26T00:39:06.744 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:06.744 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:06.744 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:06.744 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:06.744 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:06.744 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:06.744 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:06.744 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:06.745 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:06.745 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:06.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.742+0000 7f28991d2640 5 monclient: authenticate success, global_id 224200 2024-04-26T00:39:06.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.742+0000 7f28991d2640 10 monclient: get_monmap_and_config success 2024-04-26T00:39:06.745 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.742+0000 7f28991d2640 4 set_mon_vals no callback set 2024-04-26T00:39:06.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.742+0000 7f28991d2640 10 monclient: shutdown 2024-04-26T00:39:06.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.742+0000 7f28991d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:06.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.742+0000 7f28991d2640 1 -- 172.21.15.37:0/4055884299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28940a1170 msgr2=0x7f289413be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:06.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.742+0000 7f28991d2640 1 --2- 172.21.15.37:0/4055884299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28940a1170 0x7f289413be50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f287c0097c0 tx=0x7f287c01c2e0 comp rx=0 tx=0).stop 2024-04-26T00:39:06.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.742+0000 7f28991d2640 1 -- 172.21.15.37:0/4055884299 shutdown_connections 2024-04-26T00:39:06.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.742+0000 7f28991d2640 1 --2- 172.21.15.37:0/4055884299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28940a1170 0x7f289413be50 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:06.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.742+0000 7f28991d2640 1 -- 172.21.15.37:0/4055884299 >> 172.21.15.37:0/4055884299 conn(0x7f28940a7de0 msgr2=0x7f28940a3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:06.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.742+0000 7f28991d2640 1 -- 172.21.15.37:0/4055884299 shutdown_connections 2024-04-26T00:39:06.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.742+0000 7f28991d2640 1 -- 172.21.15.37:0/4055884299 wait complete. 2024-04-26T00:39:06.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.742+0000 7f28991d2640 10 monclient: build_initial_monmap 2024-04-26T00:39:06.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.742+0000 7f28991d2640 10 monclient: monmap: 2024-04-26T00:39:06.749 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:06.749 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:39:06.749 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:39:06.749 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:39:06.749 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:06.750 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:39:06.750 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:39:06.750 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:39:06.750 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:39:06.750 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:06.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f28991d2640 1 Processor -- start 2024-04-26T00:39:06.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f28991d2640 1 -- start start 2024-04-26T00:39:06.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f28991d2640 10 monclient: init 2024-04-26T00:39:06.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f28991d2640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:06.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f28991d2640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:39:06.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f28991d2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28940a1170 0x7f28941cece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:06.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f28991d2640 10 monclient(hunting): picked mon.noname-b con 0x7f28940a1170 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:39:06.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f28991d2640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28940a2c80 0x7f289413b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:06.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f28991d2640 10 monclient(hunting): picked mon.noname-a con 0x7f28940a2c80 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:39:06.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f28991d2640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f289413c390 0x7f289413b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:06.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f28991d2640 10 monclient(hunting): picked mon.noname-c con 0x7f289413c390 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:06.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f28991d2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:06.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f28991d2640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f28941428d0 con 0x7f28940a2c80 2024-04-26T00:39:06.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f28991d2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:06.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f28991d2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f28940a3060 con 0x7f28940a1170 2024-04-26T00:39:06.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f28991d2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:06.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f28991d2640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2894142a50 con 0x7f289413c390 2024-04-26T00:39:06.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f28991d2640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:06.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f28991d2640 10 monclient(hunting): authenticate will time out at 2128.053126s 2024-04-26T00:39:06.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f288a59c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28940a2c80 0x7f289413b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:06.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f288a59c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28940a2c80 0x7f289413b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52028/0 (socket says 172.21.15.37:52028) 2024-04-26T00:39:06.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f288a59c640 1 -- 172.21.15.37:0/2103427248 learned_addr learned my addr 172.21.15.37:0/2103427248 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:06.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f288a59c640 10 monclient(hunting): get_auth_request con 0x7f28940a2c80 auth_method 0 2024-04-26T00:39:06.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f288a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:06.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f288a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:06.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f288a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:06.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f289359e640 1 --2- 172.21.15.37:0/2103427248 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f289413c390 0x7f289413b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:06.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f289359e640 10 monclient(hunting): get_auth_request con 0x7f289413c390 auth_method 0 2024-04-26T00:39:06.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f289359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:06.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f289359e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:06.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f289359e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:06.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f288a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:06.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f288a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:06.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f288a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:06.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f289359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:06.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f289359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:06.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f289359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:06.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f2892d9d640 1 --2- 172.21.15.37:0/2103427248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28940a1170 0x7f28941cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:06.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f2892d9d640 10 monclient(hunting): get_auth_request con 0x7f28940a1170 auth_method 0 2024-04-26T00:39:06.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f2892d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:06.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f2892d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:06.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f2892d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:06.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f288a59c640 10 monclient(hunting): handle_auth_done global_id 234132 payload 824 2024-04-26T00:39:06.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f288a59c640 1 -- 172.21.15.37:0/2103427248 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f289413c390 msgr2=0x7f289413b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:06.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f288a59c640 1 --2- 172.21.15.37:0/2103427248 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f289413c390 0x7f289413b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:06.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f288a59c640 1 -- 172.21.15.37:0/2103427248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28940a1170 msgr2=0x7f28941cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:06.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f288a59c640 1 --2- 172.21.15.37:0/2103427248 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28940a1170 0x7f28941cece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:06.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f288a59c640 10 monclient: _finish_hunting 0 2024-04-26T00:39:06.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f288a59c640 1 monclient: found mon.noname-a 2024-04-26T00:39:06.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f288a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:06.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f288a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:39:06.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f288a59c640 1 -- 172.21.15.37:0/2103427248 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28940725f0 con 0x7f28940a2c80 2024-04-26T00:39:06.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.746+0000 7f289359e640 1 --2- 172.21.15.37:0/2103427248 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f289413c390 0x7f289413b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:39:06.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f288a59c640 1 -- 172.21.15.37:0/2103427248 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28940a2c80 msgr2=0x7f289413b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:06.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f288a59c640 1 --2- 172.21.15.37:0/2103427248 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28940a2c80 0x7f289413b260 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f287c01a990 tx=0x7f287c019080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:06.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f288bfff640 1 -- 172.21.15.37:0/2103427248 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f287c0b9070 con 0x7f28940a2c80 2024-04-26T00:39:06.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f288bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:06.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f288bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:39:06.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f288bfff640 10 monclient: dump: 2024-04-26T00:39:06.754 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:06.754 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:06.754 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:06.755 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:06.755 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:06.755 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:06.755 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:06.755 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:06.755 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:06.755 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:06.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f288bfff640 10 monclient: _finish_auth 0 2024-04-26T00:39:06.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f288bfff640 10 monclient: _check_auth_tickets 2024-04-26T00:39:06.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f288bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:06.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f28991d2640 5 monclient: authenticate success, global_id 234132 2024-04-26T00:39:06.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f28991d2640 10 monclient: _renew_subs 2024-04-26T00:39:06.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f28991d2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:39:06.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f28991d2640 1 -- 172.21.15.37:0/2103427248 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28941385a0 con 0x7f28940a2c80 2024-04-26T00:39:06.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f288bfff640 1 -- 172.21.15.37:0/2103427248 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f287c0b5aa0 con 0x7f28940a2c80 2024-04-26T00:39:06.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f288bfff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:06.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f288bfff640 1 -- 172.21.15.37:0/2103427248 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f287c0b0040 con 0x7f28940a2c80 2024-04-26T00:39:06.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f288bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:06.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f288bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:39:06.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f288bfff640 10 monclient: dump: 2024-04-26T00:39:06.756 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:06.756 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:06.756 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:06.756 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:06.756 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:06.756 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:06.756 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:06.757 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:06.757 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:06.757 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:06.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f2891d9b640 4 set_mon_vals no callback set 2024-04-26T00:39:06.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f28991d2640 10 monclient: _renew_subs 2024-04-26T00:39:06.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f28991d2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:39:06.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f28991d2640 1 -- 172.21.15.37:0/2103427248 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2894139490 con 0x7f28940a2c80 2024-04-26T00:39:06.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f28991d2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:06.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f28991d2640 1 --2- 172.21.15.37:0/2103427248 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f289413bd60 0x7f2894070950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:06.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f28991d2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:06.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f28991d2640 1 -- 172.21.15.37:0/2103427248 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f289404f510 con 0x7f289413bd60 2024-04-26T00:39:06.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f2892d9d640 1 --2- 172.21.15.37:0/2103427248 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f289413bd60 0x7f2894070950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:06.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f2892d9d640 10 monclient: get_auth_request con 0x7f289413bd60 auth_method 0 2024-04-26T00:39:06.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f2892d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:06.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f2892d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:06.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f288bfff640 1 -- 172.21.15.37:0/2103427248 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f287c0bc070 con 0x7f28940a2c80 2024-04-26T00:39:06.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f2892d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:06.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f2892d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:06.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f2892d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:06.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f2892d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:06.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f288bfff640 1 --2- 172.21.15.37:0/2103427248 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2874035e80 0x7f2874038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:06.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f288bfff640 1 -- 172.21.15.37:0/2103427248 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(757..757 src has 705..757) ==== 3915+0+0 (secure 0 0 0) 0x7f287c0e3150 con 0x7f28940a2c80 2024-04-26T00:39:06.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f2892d9d640 10 monclient(hunting): handle_auth_done global_id 234135 payload 824 2024-04-26T00:39:06.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f2892d9d640 1 -- 172.21.15.37:0/2103427248 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f28940008e0 con 0x7f289413bd60 2024-04-26T00:39:06.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f289359e640 1 --2- 172.21.15.37:0/2103427248 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2874035e80 0x7f2874038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:06.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f289359e640 10 monclient: get_auth_request con 0x7f2874035e80 auth_method 0 2024-04-26T00:39:06.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f2892d9d640 1 -- 172.21.15.37:0/2103427248 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f289413bd60 msgr2=0x7f2894070950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:06.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f2892d9d640 1 --2- 172.21.15.37:0/2103427248 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f289413bd60 0x7f2894070950 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f288400e450 tx=0x7f288400e910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:06.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f288bfff640 1 -- 172.21.15.37:0/2103427248 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2884007450 con 0x7f289413bd60 2024-04-26T00:39:06.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f288bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:06.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f288bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:39:06.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.750+0000 7f288bfff640 10 monclient: dump: 2024-04-26T00:39:06.762 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:06.762 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:06.762 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:06.762 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:06.763 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:06.763 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:06.763 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:06.763 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:06.763 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:06.763 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:06.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.754+0000 7f288bfff640 1 -- 172.21.15.37:0/2103427248 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2884007450 con 0x7f289413bd60 2024-04-26T00:39:06.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.754+0000 7f288bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:06.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.754+0000 7f288bfff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:39:06.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.754+0000 7f288bfff640 1 -- 172.21.15.37:0/2103427248 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f289413bd60 msgr2=0x7f2894070950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:06.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.754+0000 7f288bfff640 1 --2- 172.21.15.37:0/2103427248 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f289413bd60 0x7f2894070950 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f288400e450 tx=0x7f288400e910 comp rx=0 tx=0).stop 2024-04-26T00:39:06.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.754+0000 7f288bfff640 1 -- 172.21.15.37:0/2103427248 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f289413bd60 msgr2=0x7f2894070950 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:39:06.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.754+0000 7f288bfff640 1 --2- 172.21.15.37:0/2103427248 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f289413bd60 0x7f2894070950 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f288400e450 tx=0x7f288400e910 comp rx=0 tx=0).stop 2024-04-26T00:39:06.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.754+0000 7f289359e640 1 --2- 172.21.15.37:0/2103427248 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2874035e80 0x7f2874038340 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f288c005c10 tx=0x7f288c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:06.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.758+0000 7f28991d2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:39:06.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.758+0000 7f28991d2640 1 --2- 172.21.15.37:0/2103427248 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2894070e90 0x7f28940712a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:06.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.758+0000 7f28991d2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:06.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.758+0000 7f28991d2640 1 -- 172.21.15.37:0/2103427248 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f28940717e0 con 0x7f2894070e90 2024-04-26T00:39:06.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.758+0000 7f2892d9d640 1 --2- 172.21.15.37:0/2103427248 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2894070e90 0x7f28940712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:06.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.758+0000 7f2892d9d640 10 monclient: get_auth_request con 0x7f2894070e90 auth_method 0 2024-04-26T00:39:06.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.758+0000 7f2892d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:06.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.758+0000 7f2892d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:06.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.758+0000 7f2892d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:06.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.758+0000 7f2892d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:06.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.758+0000 7f2892d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:06.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.758+0000 7f2892d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:06.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.758+0000 7f2892d9d640 10 monclient(hunting): handle_auth_done global_id 234138 payload 824 2024-04-26T00:39:06.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.758+0000 7f2892d9d640 1 -- 172.21.15.37:0/2103427248 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2894139820 con 0x7f2894070e90 2024-04-26T00:39:06.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.762+0000 7f2892d9d640 1 -- 172.21.15.37:0/2103427248 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2894070e90 msgr2=0x7f28940712a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:06.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.762+0000 7f2892d9d640 1 --2- 172.21.15.37:0/2103427248 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2894070e90 0x7f28940712a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f288400e450 tx=0x7f2884003770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:06.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.762+0000 7f288bfff640 1 -- 172.21.15.37:0/2103427248 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2884006e60 con 0x7f2894070e90 2024-04-26T00:39:06.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.762+0000 7f288bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:06.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.762+0000 7f288bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:39:06.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.762+0000 7f288bfff640 10 monclient: dump: 2024-04-26T00:39:06.766 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:06.766 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:06.766 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:06.766 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:06.766 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:06.766 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:06.766 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:06.766 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:06.766 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:06.766 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:06.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.762+0000 7f288bfff640 1 -- 172.21.15.37:0/2103427248 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4549 (secure 0 0 0) 0x7f2884013420 con 0x7f2894070e90 2024-04-26T00:39:06.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.762+0000 7f288bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:39:06.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.762+0000 7f288bfff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:39:06.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.762+0000 7f288bfff640 1 -- 172.21.15.37:0/2103427248 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2894070e90 msgr2=0x7f28940712a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:06.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.762+0000 7f288bfff640 1 --2- 172.21.15.37:0/2103427248 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2894070e90 0x7f28940712a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f288400e450 tx=0x7f2884003770 comp rx=0 tx=0).stop 2024-04-26T00:39:06.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.762+0000 7f288bfff640 1 -- 172.21.15.37:0/2103427248 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2894070e90 msgr2=0x7f28940712a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:39:06.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.762+0000 7f288bfff640 1 --2- 172.21.15.37:0/2103427248 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2894070e90 0x7f28940712a0 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f288400e450 tx=0x7f2884003770 comp rx=0 tx=0).stop 2024-04-26T00:39:06.767 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:39:06.767 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:39:06.767 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:39:06.767 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:39:06.767 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 144, 2024-04-26T00:39:06.767 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 17223, 2024-04-26T00:39:06.767 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:39:06.767 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:39:06.768 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:39:06.768 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:39:06.768 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:06.768 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 4, 2024-04-26T00:39:06.768 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:39:06.768 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:39:06.768 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:39:06.768 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:06.768 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:06.768 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:06.768 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:06.768 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:06.768 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:06.768 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:06.769 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:06.769 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:06.769 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:06.769 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:06.769 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:06.769 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:39:06.769 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:39:06.769 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:06.769 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:06.769 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:06.769 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:06.769 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:06.769 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:06.769 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:06.769 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:06.769 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:06.770 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:06.770 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:06.770 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:06.770 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:06.770 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:39:06.770 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:39:06.770 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:39:06.770 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:39:06.770 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:39:06.770 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:39:06.770 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:39:06.770 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:39:06.770 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:39:06.771 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:39:06.771 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:39:06.771 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:39:06.771 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:39:06.771 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:39:06.771 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:39:06.771 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:39:06.771 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:39:06.771 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:06.771 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:06.771 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:06.771 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:06.771 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:06.771 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:06.771 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:06.771 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:06.772 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:06.772 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:06.772 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:06.772 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:06.772 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:39:06.772 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:06.772 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:39:06.772 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:06.772 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:39:06.772 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:39:06.772 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:06.772 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:06.772 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:06.772 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:06.772 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:39:06.773 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:06.773 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:06.773 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:06.773 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:06.773 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:39:06.773 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:06.773 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:06.773 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:06.773 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:06.773 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:39:06.773 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:39:06.773 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:06.773 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:06.773 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:06.773 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:06.774 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:06.774 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:39:06.774 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:39:06.774 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:06.774 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:06.774 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:06.774 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:06.774 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:39:06.774 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:06.774 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:06.774 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:06.774 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:06.774 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:39:06.774 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:06.775 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:06.775 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:06.775 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:06.775 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:39:06.775 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:39:06.775 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:06.775 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:06.775 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:06.775 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:06.775 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:06.775 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:39:06.775 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:39:06.775 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:06.775 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:06.775 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:06.775 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:06.776 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:39:06.776 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:06.776 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:06.776 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:06.776 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:06.776 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:39:06.776 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:06.776 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:06.776 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:06.776 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:06.776 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:39:06.776 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:39:06.776 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:06.776 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:06.776 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:06.777 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:06.777 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:06.777 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:06.777 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:39:06.777 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:39:06.777 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:06.777 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:06.777 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:06.777 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:39:06.777 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:06.777 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:06.777 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:39:06.777 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:06.777 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:06.777 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:06.778 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:39:06.778 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:06.778 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:06.778 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:06.778 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:39:06.778 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:39:06.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.762+0000 7f28991d2640 1 -- 172.21.15.37:0/2103427248 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2874035e80 msgr2=0x7f2874038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:06.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.762+0000 7f28991d2640 1 --2- 172.21.15.37:0/2103427248 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2874035e80 0x7f2874038340 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f288c005c10 tx=0x7f288c009040 comp rx=0 tx=0).stop 2024-04-26T00:39:06.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.762+0000 7f28991d2640 10 monclient: shutdown 2024-04-26T00:39:06.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.762+0000 7f28991d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:06.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.762+0000 7f28991d2640 1 -- 172.21.15.37:0/2103427248 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28940a2c80 msgr2=0x7f289413b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:06.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.762+0000 7f28991d2640 1 --2- 172.21.15.37:0/2103427248 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28940a2c80 0x7f289413b260 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f287c01a990 tx=0x7f287c019080 comp rx=0 tx=0).stop 2024-04-26T00:39:06.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.762+0000 7f289359e640 1 -- 172.21.15.37:0/2103427248 reap_dead start 2024-04-26T00:39:06.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.762+0000 7f289359e640 1 -- 172.21.15.37:0/2103427248 reap_dead start 2024-04-26T00:39:06.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.762+0000 7f28991d2640 1 -- 172.21.15.37:0/2103427248 shutdown_connections 2024-04-26T00:39:06.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.762+0000 7f28991d2640 1 -- 172.21.15.37:0/2103427248 >> 172.21.15.37:0/2103427248 conn(0x7f28940a7de0 msgr2=0x7f289413e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:06.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.762+0000 7f28991d2640 1 -- 172.21.15.37:0/2103427248 shutdown_connections 2024-04-26T00:39:06.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.762+0000 7f28991d2640 1 -- 172.21.15.37:0/2103427248 wait complete. 2024-04-26T00:39:06.781 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:39:06.781 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:39:06.781 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:39:06.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.898+0000 7ffb9bf6a640 10 monclient: get_monmap_and_config 2024-04-26T00:39:06.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.898+0000 7ffb9bf6a640 10 monclient: build_initial_monmap 2024-04-26T00:39:06.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.898+0000 7ffb9bf6a640 10 monclient: monmap: 2024-04-26T00:39:06.897 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:06.897 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:06.897 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:39:06.900311+0000 2024-04-26T00:39:06.897 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:39:06.900311+0000 2024-04-26T00:39:06.897 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:06.897 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:06.897 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:39:06.897 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:39:06.897 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:39:06.897 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:39:06.897 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:39:06.898 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:06.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.898+0000 7ffb9bf6a640 1 Processor -- start 2024-04-26T00:39:06.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.898+0000 7ffb9bf6a640 1 -- start start 2024-04-26T00:39:06.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.898+0000 7ffb9bf6a640 10 monclient: init 2024-04-26T00:39:06.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.898+0000 7ffb9bf6a640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:06.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.898+0000 7ffb9bf6a640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-04-26T00:39:06.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.898+0000 7ffb9bf6a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffb940a3d90 addr v1:172.21.15.45:6789/0 2024-04-26T00:39:06.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.898+0000 7ffb9bf6a640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ffb940a4170 0x7ffb940a45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:06.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.898+0000 7ffb9bf6a640 10 monclient(hunting): picked mon.noname-a con 0x7ffb940a4170 addr v2:172.21.15.37:3300/0 2024-04-26T00:39:06.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.898+0000 7ffb9bf6a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffb9413c3e0 addr v1:172.21.15.37:6789/0 2024-04-26T00:39:06.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.898+0000 7ffb9bf6a640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb940a5150 con 0x7ffb9413c3e0 2024-04-26T00:39:06.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.898+0000 7ffb9bf6a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb9409b280 con 0x7ffb940a3d90 2024-04-26T00:39:06.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.898+0000 7ffb9bf6a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:06.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.898+0000 7ffb9bf6a640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7ffb9409c650 con 0x7ffb940a4170 2024-04-26T00:39:06.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.898+0000 7ffb9bf6a640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:06.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.898+0000 7ffb9bf6a640 10 monclient(hunting): authenticate will time out at 2128.205924s 2024-04-26T00:39:06.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.898+0000 7ffb99505640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ffb940a4170 0x7ffb940a45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:06.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.898+0000 7ffb9a507640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7ffb9413c3e0 0x7ffb9413e7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:52858/0 (socket says 172.21.15.37:52858) 2024-04-26T00:39:06.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.898+0000 7ffb9a507640 1 -- 172.21.15.37:0/3162082749 learned_addr learned my addr 172.21.15.37:0/3162082749 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:39:06.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.898+0000 7ffb99505640 10 monclient(hunting): get_auth_request con 0x7ffb940a4170 auth_method 0 2024-04-26T00:39:06.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.898+0000 7ffb99505640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:06.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.898+0000 7ffb99505640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:06.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.898+0000 7ffb99505640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:06.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.898+0000 7ffb99505640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:06.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.898+0000 7ffb99505640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:06.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.898+0000 7ffb99505640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:06.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.898+0000 7ffb99505640 10 monclient(hunting): handle_auth_done global_id 234141 payload 274 2024-04-26T00:39:06.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.898+0000 7ffb99505640 1 -- 172.21.15.37:0/3162082749 >> v1:172.21.15.45:6789/0 conn(0x7ffb940a3d90 legacy=0x7ffb9409bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:06.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.898+0000 7ffb99505640 1 -- 172.21.15.37:0/3162082749 >> v1:172.21.15.37:6789/0 conn(0x7ffb9413c3e0 legacy=0x7ffb9413e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:06.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.898+0000 7ffb99505640 10 monclient: _finish_hunting 0 2024-04-26T00:39:06.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.898+0000 7ffb99505640 1 monclient: found mon.noname-a 2024-04-26T00:39:06.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.898+0000 7ffb99505640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:06.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.898+0000 7ffb99505640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:39:06.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.898+0000 7ffb99505640 1 -- 172.21.15.37:0/3162082749 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb9409b560 con 0x7ffb940a4170 2024-04-26T00:39:06.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.902+0000 7ffb99505640 1 -- 172.21.15.37:0/3162082749 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffb940a4170 msgr2=0x7ffb940a45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:06.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.902+0000 7ffb99505640 1 --2- 172.21.15.37:0/3162082749 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffb940a4170 0x7ffb940a45e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ffb840097c0 tx=0x7ffb8401c2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5b677f8e625e08f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:06.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.902+0000 7ffb98d04640 1 -- 172.21.15.37:0/3162082749 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb840b0070 con 0x7ffb940a4170 2024-04-26T00:39:06.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.902+0000 7ffb98d04640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:06.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.902+0000 7ffb98d04640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:39:06.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.902+0000 7ffb98d04640 10 monclient: dump: 2024-04-26T00:39:06.903 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:06.904 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:06.904 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:06.904 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:06.904 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:06.904 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:06.904 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:06.904 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:06.904 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:06.904 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:06.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.902+0000 7ffb98d04640 10 monclient: _finish_auth 0 2024-04-26T00:39:06.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.902+0000 7ffb98d04640 10 monclient: _check_auth_tickets 2024-04-26T00:39:06.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.902+0000 7ffb98d04640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:06.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.902+0000 7ffb98d04640 1 -- 172.21.15.37:0/3162082749 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb840ab600 con 0x7ffb940a4170 2024-04-26T00:39:06.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.902+0000 7ffb98d04640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:06.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.902+0000 7ffb98d04640 1 -- 172.21.15.37:0/3162082749 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb840ab9d0 con 0x7ffb940a4170 2024-04-26T00:39:06.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.902+0000 7ffb9bf6a640 5 monclient: authenticate success, global_id 234141 2024-04-26T00:39:06.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.902+0000 7ffb9bf6a640 10 monclient: get_monmap_and_config success 2024-04-26T00:39:06.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.902+0000 7ffb9bf6a640 4 set_mon_vals no callback set 2024-04-26T00:39:06.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.902+0000 7ffb98d04640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:06.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.902+0000 7ffb98d04640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:39:06.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.902+0000 7ffb98d04640 10 monclient: dump: 2024-04-26T00:39:06.905 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:06.905 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:06.905 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:06.905 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:06.905 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:06.906 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:06.906 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:06.906 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:06.906 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:06.906 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:06.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.902+0000 7ffb9bf6a640 10 monclient: shutdown 2024-04-26T00:39:06.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.902+0000 7ffb9bf6a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:06.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.902+0000 7ffb9bf6a640 1 -- 172.21.15.37:0/3162082749 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffb940a4170 msgr2=0x7ffb940a45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:06.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.902+0000 7ffb9bf6a640 1 --2- 172.21.15.37:0/3162082749 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffb940a4170 0x7ffb940a45e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ffb840097c0 tx=0x7ffb8401c2e0 comp rx=0 tx=0).stop 2024-04-26T00:39:06.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.902+0000 7ffb9bf6a640 1 -- 172.21.15.37:0/3162082749 shutdown_connections 2024-04-26T00:39:06.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.902+0000 7ffb9bf6a640 1 --2- 172.21.15.37:0/3162082749 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffb940a4170 0x7ffb940a45e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:06.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.902+0000 7ffb9bf6a640 1 -- 172.21.15.37:0/3162082749 >> 172.21.15.37:0/3162082749 conn(0x7ffb940aa0a0 msgr2=0x7ffb940a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:06.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.902+0000 7ffb9bf6a640 1 -- 172.21.15.37:0/3162082749 shutdown_connections 2024-04-26T00:39:06.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.902+0000 7ffb9bf6a640 1 -- 172.21.15.37:0/3162082749 wait complete. 2024-04-26T00:39:06.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb9bf6a640 10 monclient: build_initial_monmap 2024-04-26T00:39:06.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb9bf6a640 10 monclient: monmap: 2024-04-26T00:39:06.912 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:06.912 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:39:06.912 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:39:06.912 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:39:06.912 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:06.912 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:39:06.912 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:39:06.912 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:39:06.912 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:39:06.912 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:06.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb9bf6a640 1 Processor -- start 2024-04-26T00:39:06.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb9bf6a640 1 -- start start 2024-04-26T00:39:06.912 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb9bf6a640 10 monclient: init 2024-04-26T00:39:06.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb9bf6a640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:06.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb9bf6a640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:39:06.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb9bf6a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffb940a3d90 0x7ffb941cedf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:06.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb9bf6a640 10 monclient(hunting): picked mon.noname-c con 0x7ffb940a3d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:06.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb9bf6a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb940a4170 0x7ffb941cf5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:06.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb9bf6a640 10 monclient(hunting): picked mon.noname-b con 0x7ffb940a4170 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:39:06.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb9bf6a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffb9413c3e0 0x7ffb94138d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:06.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb9bf6a640 10 monclient(hunting): picked mon.noname-a con 0x7ffb9413c3e0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:39:06.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb9bf6a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:06.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb9bf6a640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ffb94142aa0 con 0x7ffb9413c3e0 2024-04-26T00:39:06.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb9bf6a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:06.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb9bf6a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ffb94142920 con 0x7ffb940a4170 2024-04-26T00:39:06.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb9bf6a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:06.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb9bf6a640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ffb94142c20 con 0x7ffb940a3d90 2024-04-26T00:39:06.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb9bf6a640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:06.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb9bf6a640 10 monclient(hunting): authenticate will time out at 2128.213363s 2024-04-26T00:39:06.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb99d06640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffb940a3d90 0x7ffb941cedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:06.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb99d06640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffb940a3d90 0x7ffb941cedf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:35742/0 (socket says 172.21.15.37:35742) 2024-04-26T00:39:06.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb99d06640 1 -- 172.21.15.37:0/115626626 learned_addr learned my addr 172.21.15.37:0/115626626 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:06.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb99d06640 10 monclient(hunting): get_auth_request con 0x7ffb940a3d90 auth_method 0 2024-04-26T00:39:06.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb99d06640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:06.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb99d06640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:06.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb99d06640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:06.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb9a507640 1 --2- 172.21.15.37:0/115626626 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffb9413c3e0 0x7ffb94138d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:06.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb9a507640 10 monclient(hunting): get_auth_request con 0x7ffb9413c3e0 auth_method 0 2024-04-26T00:39:06.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb9a507640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:06.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb9a507640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:06.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb9a507640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:06.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb99d06640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:06.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb99d06640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:06.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb99d06640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:06.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb99505640 1 --2- 172.21.15.37:0/115626626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb940a4170 0x7ffb941cf5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:06.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb9a507640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:06.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb9a507640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:06.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb9a507640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:06.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb99505640 10 monclient(hunting): get_auth_request con 0x7ffb940a4170 auth_method 0 2024-04-26T00:39:06.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb99505640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:06.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb99505640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:06.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb99505640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:06.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb99505640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:06.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb99505640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:06.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb99505640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:06.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb99d06640 10 monclient(hunting): handle_auth_done global_id 204431 payload 824 2024-04-26T00:39:06.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb99d06640 1 -- 172.21.15.37:0/115626626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb940a4170 msgr2=0x7ffb941cf5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:06.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb99d06640 1 --2- 172.21.15.37:0/115626626 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb940a4170 0x7ffb941cf5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:06.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb99d06640 1 -- 172.21.15.37:0/115626626 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffb9413c3e0 msgr2=0x7ffb94138d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:06.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb99d06640 1 --2- 172.21.15.37:0/115626626 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffb9413c3e0 0x7ffb94138d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:06.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb99d06640 10 monclient: _finish_hunting 0 2024-04-26T00:39:06.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb99d06640 1 monclient: found mon.noname-c 2024-04-26T00:39:06.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb99d06640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:06.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb99d06640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:39:06.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb99d06640 1 -- 172.21.15.37:0/115626626 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb9413a9d0 con 0x7ffb940a3d90 2024-04-26T00:39:06.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.906+0000 7ffb9a507640 1 --2- 172.21.15.37:0/115626626 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffb9413c3e0 0x7ffb94138d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:39:06.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb99d06640 1 -- 172.21.15.37:0/115626626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffb940a3d90 msgr2=0x7ffb941cedf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:06.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb99d06640 1 --2- 172.21.15.37:0/115626626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffb940a3d90 0x7ffb941cedf0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ffb9000c450 tx=0x7ffb9000c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:06.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb8a7fc640 1 -- 172.21.15.37:0/115626626 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb900119a0 con 0x7ffb940a3d90 2024-04-26T00:39:06.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb8a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:06.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb8a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:39:06.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb8a7fc640 10 monclient: dump: 2024-04-26T00:39:06.917 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:06.917 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:06.917 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:06.917 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:06.917 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:06.917 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:06.917 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:06.917 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:06.918 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:06.918 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:06.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb8a7fc640 10 monclient: _finish_auth 0 2024-04-26T00:39:06.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb8a7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:39:06.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb8a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:06.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb8a7fc640 1 -- 172.21.15.37:0/115626626 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb90017070 con 0x7ffb940a3d90 2024-04-26T00:39:06.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb8a7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:06.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb8a7fc640 1 -- 172.21.15.37:0/115626626 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb90011da0 con 0x7ffb940a3d90 2024-04-26T00:39:06.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb8a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:06.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb8a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:06.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb8a7fc640 10 monclient: dump: 2024-04-26T00:39:06.918 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:06.918 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:06.918 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:06.918 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:06.919 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:06.919 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:06.919 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:06.919 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:06.919 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:06.919 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:06.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb9bf6a640 5 monclient: authenticate success, global_id 204431 2024-04-26T00:39:06.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb9bf6a640 10 monclient: _renew_subs 2024-04-26T00:39:06.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb9bf6a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:06.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb8bfff640 4 set_mon_vals no callback set 2024-04-26T00:39:06.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb9bf6a640 1 -- 172.21.15.37:0/115626626 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb94120460 con 0x7ffb940a3d90 2024-04-26T00:39:06.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb9bf6a640 10 monclient: _renew_subs 2024-04-26T00:39:06.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb9bf6a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:06.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb9bf6a640 1 -- 172.21.15.37:0/115626626 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb9411f9e0 con 0x7ffb940a3d90 2024-04-26T00:39:06.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb9bf6a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:06.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb9bf6a640 1 --2- 172.21.15.37:0/115626626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffb941cfb00 0x7ffb941392c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:06.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb9bf6a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:06.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb9bf6a640 1 -- 172.21.15.37:0/115626626 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ffb940a0890 con 0x7ffb941cfb00 2024-04-26T00:39:06.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb99505640 1 --2- 172.21.15.37:0/115626626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffb941cfb00 0x7ffb941392c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:06.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb8a7fc640 1 -- 172.21.15.37:0/115626626 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ffb9001c460 con 0x7ffb940a3d90 2024-04-26T00:39:06.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb99505640 10 monclient: get_auth_request con 0x7ffb941cfb00 auth_method 0 2024-04-26T00:39:06.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb99505640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:06.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb99505640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:06.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb99505640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:06.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb99505640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:06.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb99505640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:06.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb99505640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:06.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb8a7fc640 1 --2- 172.21.15.37:0/115626626 >> v2:172.21.15.45:6812/2379918623 conn(0x7ffb78035ed0 0x7ffb78038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:06.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb8a7fc640 1 -- 172.21.15.37:0/115626626 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(757..757 src has 705..757) ==== 3915+0+0 (secure 0 0 0) 0x7ffb900488d0 con 0x7ffb940a3d90 2024-04-26T00:39:06.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb99505640 10 monclient(hunting): handle_auth_done global_id 204434 payload 824 2024-04-26T00:39:06.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb99505640 1 -- 172.21.15.37:0/115626626 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ffb9409ada0 con 0x7ffb941cfb00 2024-04-26T00:39:06.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb9a507640 1 --2- 172.21.15.37:0/115626626 >> v2:172.21.15.45:6812/2379918623 conn(0x7ffb78035ed0 0x7ffb78038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:06.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.910+0000 7ffb9a507640 10 monclient: get_auth_request con 0x7ffb78035ed0 auth_method 0 2024-04-26T00:39:06.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.914+0000 7ffb99505640 1 -- 172.21.15.37:0/115626626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffb941cfb00 msgr2=0x7ffb941392c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:06.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.914+0000 7ffb99505640 1 --2- 172.21.15.37:0/115626626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffb941cfb00 0x7ffb941392c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ffb840ad000 tx=0x7ffb84019080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:06.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.914+0000 7ffb8a7fc640 1 -- 172.21.15.37:0/115626626 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb840bc070 con 0x7ffb941cfb00 2024-04-26T00:39:06.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.914+0000 7ffb8a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:06.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.914+0000 7ffb8a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:06.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.914+0000 7ffb8a7fc640 10 monclient: dump: 2024-04-26T00:39:06.924 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:06.924 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:06.924 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:06.924 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:06.924 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:06.924 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:06.924 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:06.924 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:06.924 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:06.924 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:06.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.914+0000 7ffb8a7fc640 1 -- 172.21.15.37:0/115626626 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ffb840b1040 con 0x7ffb941cfb00 2024-04-26T00:39:06.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.914+0000 7ffb8a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:06.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.914+0000 7ffb8a7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:39:06.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.914+0000 7ffb8a7fc640 1 -- 172.21.15.37:0/115626626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffb941cfb00 msgr2=0x7ffb941392c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:06.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.914+0000 7ffb8a7fc640 1 --2- 172.21.15.37:0/115626626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffb941cfb00 0x7ffb941392c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ffb840ad000 tx=0x7ffb84019080 comp rx=0 tx=0).stop 2024-04-26T00:39:06.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.914+0000 7ffb8a7fc640 1 -- 172.21.15.37:0/115626626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffb941cfb00 msgr2=0x7ffb941392c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:39:06.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.914+0000 7ffb8a7fc640 1 --2- 172.21.15.37:0/115626626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffb941cfb00 0x7ffb941392c0 secure :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ffb840ad000 tx=0x7ffb84019080 comp rx=0 tx=0).stop 2024-04-26T00:39:06.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.914+0000 7ffb9a507640 1 --2- 172.21.15.37:0/115626626 >> v2:172.21.15.45:6812/2379918623 conn(0x7ffb78035ed0 0x7ffb78038390 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c005c10 tx=0x7ffb8c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:06.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.918+0000 7ffb9bf6a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:39:06.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.918+0000 7ffb9bf6a640 1 --2- 172.21.15.37:0/115626626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffb94139800 0x7ffb9409d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:06.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.918+0000 7ffb9bf6a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:06.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.918+0000 7ffb9bf6a640 1 -- 172.21.15.37:0/115626626 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ffb94139be0 con 0x7ffb94139800 2024-04-26T00:39:06.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.918+0000 7ffb99505640 1 --2- 172.21.15.37:0/115626626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffb94139800 0x7ffb9409d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:06.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.918+0000 7ffb99505640 10 monclient: get_auth_request con 0x7ffb94139800 auth_method 0 2024-04-26T00:39:06.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.918+0000 7ffb99505640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:06.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.918+0000 7ffb99505640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:06.925 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.918+0000 7ffb99505640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:06.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.918+0000 7ffb99505640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:06.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.918+0000 7ffb99505640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:06.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.918+0000 7ffb99505640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:06.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.918+0000 7ffb99505640 10 monclient(hunting): handle_auth_done global_id 204437 payload 824 2024-04-26T00:39:06.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.918+0000 7ffb99505640 1 -- 172.21.15.37:0/115626626 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ffb94075dc0 con 0x7ffb94139800 2024-04-26T00:39:06.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.922+0000 7ffb99505640 1 -- 172.21.15.37:0/115626626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffb94139800 msgr2=0x7ffb9409d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:06.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.922+0000 7ffb99505640 1 --2- 172.21.15.37:0/115626626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffb94139800 0x7ffb9409d070 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ffb840097c0 tx=0x7ffb84009190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:06.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.922+0000 7ffb8a7fc640 1 -- 172.21.15.37:0/115626626 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb840b0040 con 0x7ffb94139800 2024-04-26T00:39:06.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.922+0000 7ffb8a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:06.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.922+0000 7ffb8a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:06.926 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.922+0000 7ffb8a7fc640 10 monclient: dump: 2024-04-26T00:39:06.926 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:06.927 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:06.927 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:06.927 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:06.927 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:06.927 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:06.927 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:06.927 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:06.927 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:06.927 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:06.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.922+0000 7ffb8a7fc640 1 -- 172.21.15.37:0/115626626 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4718 (secure 0 0 0) 0x7ffb84003620 con 0x7ffb94139800 2024-04-26T00:39:06.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.922+0000 7ffb8a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:39:06.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.922+0000 7ffb8a7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:39:06.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.922+0000 7ffb8a7fc640 1 -- 172.21.15.37:0/115626626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffb94139800 msgr2=0x7ffb9409d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:06.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.922+0000 7ffb8a7fc640 1 --2- 172.21.15.37:0/115626626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffb94139800 0x7ffb9409d070 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ffb840097c0 tx=0x7ffb84009190 comp rx=0 tx=0).stop 2024-04-26T00:39:06.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.922+0000 7ffb8a7fc640 1 -- 172.21.15.37:0/115626626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffb94139800 msgr2=0x7ffb9409d070 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:39:06.927 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.922+0000 7ffb8a7fc640 1 --2- 172.21.15.37:0/115626626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffb94139800 0x7ffb9409d070 secure :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ffb840097c0 tx=0x7ffb84009190 comp rx=0 tx=0).stop 2024-04-26T00:39:06.928 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:39:06.928 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:39:06.928 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:39:06.928 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:39:06.928 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 144, 2024-04-26T00:39:06.928 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 152530, 2024-04-26T00:39:06.928 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:39:06.928 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:39:06.928 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:39:06.928 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:39:06.928 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:06.928 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 4, 2024-04-26T00:39:06.929 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:39:06.929 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:39:06.929 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:39:06.929 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:06.929 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:06.929 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:06.929 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:06.929 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:06.929 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:06.929 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:06.929 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:06.929 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:06.929 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:06.930 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:06.930 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:06.930 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:39:06.930 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:39:06.930 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:06.930 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:06.930 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:06.930 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:06.930 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:06.930 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:06.930 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:06.931 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:06.931 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:06.931 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:06.931 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:06.931 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:06.931 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:06.931 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:39:06.931 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:39:06.931 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:39:06.931 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:39:06.931 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:39:06.931 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:39:06.931 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:39:06.931 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:39:06.931 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:39:06.932 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:39:06.932 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:39:06.932 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:39:06.932 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:39:06.932 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:39:06.932 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:39:06.932 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:39:06.932 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:39:06.932 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:06.932 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:06.932 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:06.932 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:06.932 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:06.932 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:06.932 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:06.933 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:06.933 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:06.933 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:06.933 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:06.933 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:06.933 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:39:06.933 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:06.933 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:39:06.933 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:06.933 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:39:06.933 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:39:06.933 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:06.933 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:06.934 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:06.934 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:06.934 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:39:06.934 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:06.934 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:06.934 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:06.934 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:06.934 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:39:06.934 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:06.935 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:06.935 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:06.935 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:06.935 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:39:06.935 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:39:06.935 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:06.935 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:06.935 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:06.935 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:06.935 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:06.935 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:39:06.935 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:39:06.935 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:06.935 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:06.936 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:06.936 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:06.936 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:39:06.936 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:06.936 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:06.936 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:06.936 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:06.936 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:39:06.937 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:06.937 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:06.937 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:06.937 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:06.937 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:39:06.937 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:39:06.937 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:06.937 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:06.937 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:06.937 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:06.937 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:06.937 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:39:06.937 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:39:06.937 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:06.938 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:06.938 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:06.938 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:06.938 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:39:06.938 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:06.938 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:06.938 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:06.938 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:06.938 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:39:06.938 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:06.938 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:06.938 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:06.939 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:06.939 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:39:06.939 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:39:06.939 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:06.939 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:06.939 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:06.939 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:06.939 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:06.939 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:06.939 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:39:06.939 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:39:06.939 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:06.939 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:06.939 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:06.939 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:39:06.939 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:06.940 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:06.940 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:39:06.940 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:06.940 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:06.940 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:06.940 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:39:06.940 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:06.940 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:06.940 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:39:06.940 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:06.940 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:06.940 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:06.940 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:39:06.940 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:06.940 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:06.941 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:06.941 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:39:06.941 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:39:06.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.922+0000 7ffb9bf6a640 1 -- 172.21.15.37:0/115626626 >> v2:172.21.15.45:6812/2379918623 conn(0x7ffb78035ed0 msgr2=0x7ffb78038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:06.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.922+0000 7ffb9bf6a640 1 --2- 172.21.15.37:0/115626626 >> v2:172.21.15.45:6812/2379918623 conn(0x7ffb78035ed0 0x7ffb78038390 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c005c10 tx=0x7ffb8c009040 comp rx=0 tx=0).stop 2024-04-26T00:39:06.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.922+0000 7ffb9bf6a640 10 monclient: shutdown 2024-04-26T00:39:06.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.922+0000 7ffb9bf6a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:06.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.922+0000 7ffb9bf6a640 1 -- 172.21.15.37:0/115626626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffb940a3d90 msgr2=0x7ffb941cedf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:06.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.922+0000 7ffb9bf6a640 1 --2- 172.21.15.37:0/115626626 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffb940a3d90 0x7ffb941cedf0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ffb9000c450 tx=0x7ffb9000c910 comp rx=0 tx=0).stop 2024-04-26T00:39:06.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.922+0000 7ffb9a507640 1 -- 172.21.15.37:0/115626626 reap_dead start 2024-04-26T00:39:06.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.922+0000 7ffb9a507640 1 -- 172.21.15.37:0/115626626 reap_dead start 2024-04-26T00:39:06.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.922+0000 7ffb9bf6a640 1 -- 172.21.15.37:0/115626626 shutdown_connections 2024-04-26T00:39:06.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.922+0000 7ffb9bf6a640 1 -- 172.21.15.37:0/115626626 >> 172.21.15.37:0/115626626 conn(0x7ffb940aa0a0 msgr2=0x7ffb940a2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:06.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.922+0000 7ffb9bf6a640 1 -- 172.21.15.37:0/115626626 shutdown_connections 2024-04-26T00:39:06.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:06.922+0000 7ffb9bf6a640 1 -- 172.21.15.37:0/115626626 wait complete. 2024-04-26T00:39:06.943 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.934+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.943 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.934+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3462530586}) 2024-04-26T00:39:06.943 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.934+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1844048847}) 2024-04-26T00:39:06.943 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.934+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.943 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.934+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3462530586}) 2024-04-26T00:39:06.943 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.934+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4008289632}) 2024-04-26T00:39:06.943 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.938+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.943 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.938+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4203590315}) 2024-04-26T00:39:06.944 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.938+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3056938984}) 2024-04-26T00:39:06.944 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.938+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.944 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.938+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4203590315}) 2024-04-26T00:39:06.944 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.938+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=63299305}) 2024-04-26T00:39:06.945 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:39:06.945 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:39:06.945 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:39:06.948 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.938+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.948 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.938+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2330476553}) 2024-04-26T00:39:06.948 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.938+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3844554198}) 2024-04-26T00:39:06.948 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.938+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.948 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.938+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2330476553}) 2024-04-26T00:39:06.949 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.938+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=450596099}) 2024-04-26T00:39:06.949 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.942+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.949 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.942+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=889163284}) 2024-04-26T00:39:06.949 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.942+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3694387794}) 2024-04-26T00:39:06.949 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.942+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.949 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.942+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=889163284}) 2024-04-26T00:39:06.949 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.942+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1051461370}) 2024-04-26T00:39:06.949 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.942+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.949 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.942+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3598863834}) 2024-04-26T00:39:06.949 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.942+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=209247475}) 2024-04-26T00:39:06.949 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.942+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.950 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.942+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3598863834}) 2024-04-26T00:39:06.950 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.942+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3686802007}) 2024-04-26T00:39:06.950 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.946+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.950 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.946+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=442570810}) 2024-04-26T00:39:06.950 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.946+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=124904175}) 2024-04-26T00:39:06.950 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.946+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.950 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.946+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=442570810}) 2024-04-26T00:39:06.950 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.946+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=518063223}) 2024-04-26T00:39:06.950 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.946+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.950 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.946+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3210786462}) 2024-04-26T00:39:06.950 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.946+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=24256452}) 2024-04-26T00:39:06.950 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.946+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.950 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.946+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3210786462}) 2024-04-26T00:39:06.950 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.946+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2896189064}) 2024-04-26T00:39:06.950 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.946+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.951 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.946+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=959052203}) 2024-04-26T00:39:06.951 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.946+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2175891039}) 2024-04-26T00:39:06.951 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.946+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.951 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.946+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=959052203}) 2024-04-26T00:39:06.951 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.946+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=385111756}) 2024-04-26T00:39:06.952 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.950+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.952 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.950+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3692696111}) 2024-04-26T00:39:06.953 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.950+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=101772516}) 2024-04-26T00:39:06.953 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.950+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.953 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.950+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3692696111}) 2024-04-26T00:39:06.953 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.950+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1640980112}) 2024-04-26T00:39:06.953 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.950+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.953 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.950+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=12514503}) 2024-04-26T00:39:06.953 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.950+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2984625242}) 2024-04-26T00:39:06.953 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.950+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.953 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.950+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=12514503}) 2024-04-26T00:39:06.953 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.950+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3364333955}) 2024-04-26T00:39:06.953 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.954+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.953 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.954+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4188186980}) 2024-04-26T00:39:06.953 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.954+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=285204345}) 2024-04-26T00:39:06.953 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.954+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.953 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.954+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4188186980}) 2024-04-26T00:39:06.954 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.954+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3511751202}) 2024-04-26T00:39:06.954 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.954+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.954 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.954+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3798318365}) 2024-04-26T00:39:06.954 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.954+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1888536648}) 2024-04-26T00:39:06.955 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.954+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.955 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.954+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3798318365}) 2024-04-26T00:39:06.955 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.954+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3641048138}) 2024-04-26T00:39:06.956 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.954+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.956 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.954+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2025249529}) 2024-04-26T00:39:06.956 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.954+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1872131007}) 2024-04-26T00:39:06.956 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.954+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.956 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.954+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2025249529}) 2024-04-26T00:39:06.956 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.954+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3729643329}) 2024-04-26T00:39:06.957 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.958+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.957 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.958+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=43618908}) 2024-04-26T00:39:06.957 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.958+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2550566032}) 2024-04-26T00:39:06.958 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.958+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.958 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.958+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=43618908}) 2024-04-26T00:39:06.958 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.958+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3706486497}) 2024-04-26T00:39:06.959 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.958+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.959 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.958+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4076965365}) 2024-04-26T00:39:06.959 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.958+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3984822148}) 2024-04-26T00:39:06.959 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.958+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.959 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.958+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4076965365}) 2024-04-26T00:39:06.959 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.958+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3664588152}) 2024-04-26T00:39:06.961 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.962+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.961 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.962+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3420108210}) 2024-04-26T00:39:06.961 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.962+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2239761071}) 2024-04-26T00:39:06.961 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.962+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.961 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.962+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3420108210}) 2024-04-26T00:39:06.961 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.962+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2218275062}) 2024-04-26T00:39:06.962 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.962+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.962 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.962+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4209004163}) 2024-04-26T00:39:06.962 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.962+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3145570772}) 2024-04-26T00:39:06.963 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.962+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.963 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.962+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4209004163}) 2024-04-26T00:39:06.963 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.962+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3463691082}) 2024-04-26T00:39:06.964 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.962+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.964 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.962+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2824225575}) 2024-04-26T00:39:06.964 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.962+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4162098857}) 2024-04-26T00:39:06.964 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.962+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.964 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.966+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2824225575}) 2024-04-26T00:39:06.964 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.966+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3731586741}) 2024-04-26T00:39:06.966 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.966+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.966 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.966+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1037019328}) 2024-04-26T00:39:06.966 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.966+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2202606004}) 2024-04-26T00:39:06.966 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.966+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.966 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.966+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1037019328}) 2024-04-26T00:39:06.966 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.966+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3177142839}) 2024-04-26T00:39:06.967 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.966+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.967 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.966+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1081353400}) 2024-04-26T00:39:06.967 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.966+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2230541477}) 2024-04-26T00:39:06.968 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.966+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.968 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.966+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1081353400}) 2024-04-26T00:39:06.968 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.966+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=591125225}) 2024-04-26T00:39:06.969 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.970+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.969 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.970+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=731285526}) 2024-04-26T00:39:06.969 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.970+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1641505472}) 2024-04-26T00:39:06.969 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.970+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.969 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.970+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=731285526}) 2024-04-26T00:39:06.969 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.970+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2387847650}) 2024-04-26T00:39:06.970 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.970+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.970 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.970+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2721760879}) 2024-04-26T00:39:06.971 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.970+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2316839194}) 2024-04-26T00:39:06.971 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.970+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.971 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.970+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2721760879}) 2024-04-26T00:39:06.971 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.970+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=636562536}) 2024-04-26T00:39:06.972 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.974+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.972 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.974+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2703271748}) 2024-04-26T00:39:06.972 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.974+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1212647804}) 2024-04-26T00:39:06.972 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.974+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.972 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.974+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2703271748}) 2024-04-26T00:39:06.973 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.974+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4007160243}) 2024-04-26T00:39:06.974 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.974+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.974 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.974+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3012966069}) 2024-04-26T00:39:06.974 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.974+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4245855437}) 2024-04-26T00:39:06.974 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.974+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.974 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.974+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3012966069}) 2024-04-26T00:39:06.974 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.974+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=497971543}) 2024-04-26T00:39:06.975 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.974+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.976 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.974+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1093013116}) 2024-04-26T00:39:06.976 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.974+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3814234887}) 2024-04-26T00:39:06.976 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.974+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.976 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.974+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1093013116}) 2024-04-26T00:39:06.976 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.974+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1405160545}) 2024-04-26T00:39:06.977 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.978+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.977 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.978+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4211772136}) 2024-04-26T00:39:06.977 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.978+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=463690599}) 2024-04-26T00:39:06.977 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.978+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.977 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.978+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4211772136}) 2024-04-26T00:39:06.977 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.978+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1267891287}) 2024-04-26T00:39:06.979 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.978+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.979 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.978+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2399043922}) 2024-04-26T00:39:06.979 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.978+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1838517511}) 2024-04-26T00:39:06.979 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.978+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.979 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.978+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2399043922}) 2024-04-26T00:39:06.979 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.978+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=605864387}) 2024-04-26T00:39:06.980 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.982+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.981 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.982+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4011307198}) 2024-04-26T00:39:06.981 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.982+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=806890874}) 2024-04-26T00:39:06.981 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.982+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.981 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.982+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4011307198}) 2024-04-26T00:39:06.981 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.982+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1894599333}) 2024-04-26T00:39:06.982 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.982+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.982 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.982+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=515468995}) 2024-04-26T00:39:06.983 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.982+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=230046207}) 2024-04-26T00:39:06.983 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.982+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.983 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.982+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=515468995}) 2024-04-26T00:39:06.983 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.982+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2359669789}) 2024-04-26T00:39:06.985 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.986+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.985 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.986+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3221390649}) 2024-04-26T00:39:06.985 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.986+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3408931855}) 2024-04-26T00:39:06.985 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.986+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.985 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.986+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3221390649}) 2024-04-26T00:39:06.985 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.986+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3829594077}) 2024-04-26T00:39:06.987 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.986+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.987 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.986+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=42,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=16} crc {logm=2313796125,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=3423934630,mon_config_key=3247883965,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=3453183674,osd_snap=2926742791}) 2024-04-26T00:39:06.988 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.986+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=45,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=10,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=15} crc {logm=4001006035,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=1150763500,mon_config_key=752439442,monmap=3876866194,osd_metadata=1088026102,osd_pg_creating=3453183674,osd_snap=168410693}) 2024-04-26T00:39:06.988 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.986+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.988 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.986+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=42,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=16} crc {logm=2313796125,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=3423934630,mon_config_key=3247883965,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=3453183674,osd_snap=2926742791}) 2024-04-26T00:39:06.988 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.986+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=42,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=16} crc {logm=3250205960,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=3423934630,mon_config_key=686164493,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=3453183674,osd_snap=2815030651}) 2024-04-26T00:39:06.990 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.990+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.990 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.990+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=100} crc {osd_snap=2027102091}) 2024-04-26T00:39:06.990 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.990+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=100} crc {osd_snap=3040102606}) 2024-04-26T00:39:06.990 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.990+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.990 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.990+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=100} crc {osd_snap=2027102091}) 2024-04-26T00:39:06.991 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.990+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=100} crc {osd_snap=3598659815}) 2024-04-26T00:39:06.992 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.990+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.992 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.990+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=31,osdmap=69} crc {osd_snap=3580596754,osdmap=205523444}) 2024-04-26T00:39:06.993 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.990+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=29,osdmap=71} crc {osd_snap=2679331713,osdmap=3881234079}) 2024-04-26T00:39:06.993 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.990+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:06.993 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.990+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=31,osdmap=69} crc {osd_snap=3580596754,osdmap=205523444}) 2024-04-26T00:39:06.993 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.990+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=31,osdmap=69} crc {osd_snap=4275937605,osdmap=4108787625}) 2024-04-26T00:39:07.000 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.998+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:07.001 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.998+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=21} crc {osdmap=3546939821}) 2024-04-26T00:39:07.001 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.998+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=24} crc {osdmap=8586067}) 2024-04-26T00:39:07.001 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.998+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:07.001 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.998+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=21} crc {osdmap=3546939821}) 2024-04-26T00:39:07.001 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:39:06.998+0000 7f679330f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=24} crc {osdmap=103087309}) 2024-04-26T00:39:07.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.070+0000 7f37abdb2640 10 monclient: get_monmap_and_config 2024-04-26T00:39:07.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.070+0000 7f37abdb2640 10 monclient: build_initial_monmap 2024-04-26T00:39:07.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.070+0000 7f37abdb2640 10 monclient: monmap: 2024-04-26T00:39:07.069 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:07.069 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:07.069 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:39:07.072392+0000 2024-04-26T00:39:07.069 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:39:07.072392+0000 2024-04-26T00:39:07.069 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:07.070 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:07.070 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:39:07.070 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:39:07.070 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:39:07.070 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:39:07.070 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:39:07.070 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:07.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.070+0000 7f37abdb2640 1 Processor -- start 2024-04-26T00:39:07.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.070+0000 7f37abdb2640 1 -- start start 2024-04-26T00:39:07.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.070+0000 7f37abdb2640 10 monclient: init 2024-04-26T00:39:07.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.070+0000 7f37abdb2640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:07.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.070+0000 7f37abdb2640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-04-26T00:39:07.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.070+0000 7f37abdb2640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37a40726b0 0x7f37a4072a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:07.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.070+0000 7f37abdb2640 10 monclient(hunting): picked mon.noname-b con 0x7f37a40726b0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:07.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.070+0000 7f37abdb2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f37a40a0110 addr v1:172.21.15.37:6789/0 2024-04-26T00:39:07.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.070+0000 7f37abdb2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f37a41374e0 addr v1:172.21.15.45:6789/0 2024-04-26T00:39:07.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.070+0000 7f37abdb2640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37a40a1470 con 0x7f37a40a0110 2024-04-26T00:39:07.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.070+0000 7f37abdb2640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37a40a0bd0 con 0x7f37a41374e0 2024-04-26T00:39:07.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.070+0000 7f37abdb2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:07.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.070+0000 7f37abdb2640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f37a40a1bd0 con 0x7f37a40726b0 2024-04-26T00:39:07.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.070+0000 7f37abdb2640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:07.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.070+0000 7f37abdb2640 10 monclient(hunting): authenticate will time out at 2128.378082s 2024-04-26T00:39:07.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.070+0000 7f37a9b4e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37a40726b0 0x7f37a4072a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:07.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.070+0000 7f37a934d640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f37a40a0110 0x7f37a4136dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:52870/0 (socket says 172.21.15.37:52870) 2024-04-26T00:39:07.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.070+0000 7f37a934d640 1 -- 172.21.15.37:0/2835080528 learned_addr learned my addr 172.21.15.37:0/2835080528 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:39:07.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.070+0000 7f37a9b4e640 10 monclient(hunting): get_auth_request con 0x7f37a40726b0 auth_method 0 2024-04-26T00:39:07.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.070+0000 7f37a9b4e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:07.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.070+0000 7f37a9b4e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:07.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.070+0000 7f37a9b4e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:07.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.070+0000 7f37a9b4e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:07.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.070+0000 7f37a9b4e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:07.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.070+0000 7f37a9b4e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:07.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.070+0000 7f37a9b4e640 10 monclient(hunting): handle_auth_done global_id 204440 payload 274 2024-04-26T00:39:07.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.070+0000 7f37a9b4e640 1 -- 172.21.15.37:0/2835080528 >> v1:172.21.15.45:6789/0 conn(0x7f37a41374e0 legacy=0x7f37a413f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:07.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.070+0000 7f37a9b4e640 1 -- 172.21.15.37:0/2835080528 >> v1:172.21.15.37:6789/0 conn(0x7f37a40a0110 legacy=0x7f37a4136dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:07.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.070+0000 7f37a9b4e640 10 monclient: _finish_hunting 0 2024-04-26T00:39:07.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.070+0000 7f37a9b4e640 1 monclient: found mon.noname-b 2024-04-26T00:39:07.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.070+0000 7f37a9b4e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:07.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.070+0000 7f37a9b4e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:39:07.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.070+0000 7f37a9b4e640 1 -- 172.21.15.37:0/2835080528 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37a40a1190 con 0x7f37a40726b0 2024-04-26T00:39:07.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.074+0000 7f37a9b4e640 1 -- 172.21.15.37:0/2835080528 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37a40726b0 msgr2=0x7f37a4072a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:07.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.074+0000 7f37a9b4e640 1 --2- 172.21.15.37:0/2835080528 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37a40726b0 0x7f37a4072a90 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f37940097f0 tx=0x7f379401c140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e29076b16a829cb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:07.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.074+0000 7f37a8b4c640 1 -- 172.21.15.37:0/2835080528 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37940b1070 con 0x7f37a40726b0 2024-04-26T00:39:07.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.074+0000 7f37a8b4c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:07.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.074+0000 7f37a8b4c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:39:07.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.074+0000 7f37a8b4c640 10 monclient: dump: 2024-04-26T00:39:07.076 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:07.077 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:07.077 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:07.077 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:07.077 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:07.077 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:07.077 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:07.077 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:07.077 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:07.077 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:07.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.074+0000 7f37a8b4c640 10 monclient: _finish_auth 0 2024-04-26T00:39:07.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.074+0000 7f37a8b4c640 10 monclient: _check_auth_tickets 2024-04-26T00:39:07.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.074+0000 7f37a8b4c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:07.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.074+0000 7f37a8b4c640 1 -- 172.21.15.37:0/2835080528 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37940ac440 con 0x7f37a40726b0 2024-04-26T00:39:07.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.074+0000 7f37abdb2640 5 monclient: authenticate success, global_id 204440 2024-04-26T00:39:07.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.074+0000 7f37abdb2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:39:07.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.074+0000 7f37a8b4c640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:07.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.074+0000 7f37abdb2640 10 monclient: get_monmap_and_config success 2024-04-26T00:39:07.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.074+0000 7f37a8b4c640 1 -- 172.21.15.37:0/2835080528 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37940ac810 con 0x7f37a40726b0 2024-04-26T00:39:07.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.074+0000 7f37a8b4c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:07.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.074+0000 7f37abdb2640 4 set_mon_vals no callback set 2024-04-26T00:39:07.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.074+0000 7f37a8b4c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:07.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.074+0000 7f37a8b4c640 10 monclient: dump: 2024-04-26T00:39:07.078 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:07.078 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:07.078 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:07.078 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:07.078 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:07.078 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:07.078 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:07.079 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:07.079 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:07.079 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:07.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37abdb2640 10 monclient: shutdown 2024-04-26T00:39:07.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37abdb2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:07.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37abdb2640 1 -- 172.21.15.37:0/2835080528 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37a40726b0 msgr2=0x7f37a4072a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:07.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37abdb2640 1 --2- 172.21.15.37:0/2835080528 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37a40726b0 0x7f37a4072a90 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f37940097f0 tx=0x7f379401c140 comp rx=0 tx=0).stop 2024-04-26T00:39:07.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37abdb2640 1 -- 172.21.15.37:0/2835080528 shutdown_connections 2024-04-26T00:39:07.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37abdb2640 1 --2- 172.21.15.37:0/2835080528 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37a40726b0 0x7f37a4072a90 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:07.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37abdb2640 1 -- 172.21.15.37:0/2835080528 >> 172.21.15.37:0/2835080528 conn(0x7f37a40a1d50 msgr2=0x7f37a40a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:07.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37abdb2640 1 -- 172.21.15.37:0/2835080528 shutdown_connections 2024-04-26T00:39:07.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37abdb2640 1 -- 172.21.15.37:0/2835080528 wait complete. 2024-04-26T00:39:07.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37abdb2640 10 monclient: build_initial_monmap 2024-04-26T00:39:07.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37abdb2640 10 monclient: monmap: 2024-04-26T00:39:07.088 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:07.088 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:39:07.088 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:39:07.088 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:39:07.088 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:07.088 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:39:07.088 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:39:07.088 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:39:07.089 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:39:07.089 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:07.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37abdb2640 1 Processor -- start 2024-04-26T00:39:07.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37abdb2640 1 -- start start 2024-04-26T00:39:07.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37abdb2640 10 monclient: init 2024-04-26T00:39:07.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37abdb2640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:07.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37abdb2640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:39:07.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37abdb2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37a40726b0 0x7f37a41cf7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:07.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37abdb2640 10 monclient(hunting): picked mon.noname-b con 0x7f37a40726b0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:39:07.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37abdb2640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37a40a0110 0x7f37a41cfd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:07.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37abdb2640 10 monclient(hunting): picked mon.noname-c con 0x7f37a40a0110 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:07.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37abdb2640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f37a41374e0 0x7f37a41d0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:07.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37abdb2640 10 monclient(hunting): picked mon.noname-a con 0x7f37a41374e0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:39:07.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37abdb2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:07.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37abdb2640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f37a41435e0 con 0x7f37a41374e0 2024-04-26T00:39:07.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37abdb2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:07.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37abdb2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f37a4143460 con 0x7f37a40726b0 2024-04-26T00:39:07.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37abdb2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:07.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37abdb2640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f37a4143760 con 0x7f37a40a0110 2024-04-26T00:39:07.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37abdb2640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:07.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37abdb2640 10 monclient(hunting): authenticate will time out at 2128.385706s 2024-04-26T00:39:07.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37a934d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37a40a0110 0x7f37a41cfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:07.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37aa34f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f37a41374e0 0x7f37a41d0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:07.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37a934d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37a40a0110 0x7f37a41cfd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:35776/0 (socket says 172.21.15.37:35776) 2024-04-26T00:39:07.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37a934d640 1 -- 172.21.15.37:0/493557053 learned_addr learned my addr 172.21.15.37:0/493557053 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:07.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37a934d640 10 monclient(hunting): get_auth_request con 0x7f37a40a0110 auth_method 0 2024-04-26T00:39:07.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37a934d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:07.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37a934d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:07.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37a934d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:07.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37aa34f640 10 monclient(hunting): get_auth_request con 0x7f37a41374e0 auth_method 0 2024-04-26T00:39:07.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37aa34f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:07.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37aa34f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:07.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37aa34f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:07.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37a934d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:07.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37a934d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:07.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37a934d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:07.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37aa34f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:07.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37aa34f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:07.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37aa34f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:07.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37a9b4e640 1 --2- 172.21.15.37:0/493557053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37a40726b0 0x7f37a41cf7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:07.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37a9b4e640 10 monclient(hunting): get_auth_request con 0x7f37a40726b0 auth_method 0 2024-04-26T00:39:07.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37a9b4e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:07.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37a9b4e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:07.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37a9b4e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:07.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37a934d640 10 monclient(hunting): handle_auth_done global_id 204443 payload 824 2024-04-26T00:39:07.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37a934d640 1 -- 172.21.15.37:0/493557053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37a40726b0 msgr2=0x7f37a41cf7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:07.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37a934d640 1 --2- 172.21.15.37:0/493557053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37a40726b0 0x7f37a41cf7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:07.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37a934d640 1 -- 172.21.15.37:0/493557053 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f37a41374e0 msgr2=0x7f37a41d0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:07.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37a934d640 1 --2- 172.21.15.37:0/493557053 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f37a41374e0 0x7f37a41d0270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:07.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37a934d640 10 monclient: _finish_hunting 0 2024-04-26T00:39:07.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37a934d640 1 monclient: found mon.noname-c 2024-04-26T00:39:07.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37a934d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:07.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37a934d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:39:07.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37a934d640 1 -- 172.21.15.37:0/493557053 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37a413c890 con 0x7f37a40a0110 2024-04-26T00:39:07.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.078+0000 7f37aa34f640 1 --2- 172.21.15.37:0/493557053 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f37a41374e0 0x7f37a41d0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:39:07.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a934d640 1 -- 172.21.15.37:0/493557053 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37a40a0110 msgr2=0x7f37a41cfd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:07.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a934d640 1 --2- 172.21.15.37:0/493557053 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37a40a0110 0x7f37a41cfd30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f37a000c470 tx=0x7f37a000c930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:07.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f379a7fc640 1 -- 172.21.15.37:0/493557053 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37a0007070 con 0x7f37a40a0110 2024-04-26T00:39:07.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f379a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:07.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f379a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:39:07.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f379a7fc640 10 monclient: dump: 2024-04-26T00:39:07.093 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:07.093 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:07.093 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:07.093 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:07.093 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:07.093 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:07.093 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:07.094 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:07.094 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:07.094 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:07.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f379a7fc640 10 monclient: _finish_auth 0 2024-04-26T00:39:07.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f379a7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:39:07.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f379a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:07.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f379a7fc640 1 -- 172.21.15.37:0/493557053 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37a0007210 con 0x7f37a40a0110 2024-04-26T00:39:07.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37abdb2640 5 monclient: authenticate success, global_id 204443 2024-04-26T00:39:07.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37abdb2640 10 monclient: _renew_subs 2024-04-26T00:39:07.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37abdb2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:07.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37abdb2640 1 -- 172.21.15.37:0/493557053 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37a413e350 con 0x7f37a40a0110 2024-04-26T00:39:07.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f379a7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:07.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f379a7fc640 1 -- 172.21.15.37:0/493557053 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37a0007640 con 0x7f37a40a0110 2024-04-26T00:39:07.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f379a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:07.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f379a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:07.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f379a7fc640 10 monclient: dump: 2024-04-26T00:39:07.095 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:07.095 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:07.095 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:07.095 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:07.095 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:07.095 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:07.095 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:07.095 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:07.095 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:07.095 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:07.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f379bfff640 4 set_mon_vals no callback set 2024-04-26T00:39:07.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37abdb2640 10 monclient: _renew_subs 2024-04-26T00:39:07.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37abdb2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:07.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37abdb2640 1 -- 172.21.15.37:0/493557053 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f37a413fb70 con 0x7f37a40a0110 2024-04-26T00:39:07.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a934d640 1 -- 172.21.15.37:0/493557053 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37a40a0110 msgr2=0x7f37a41cfd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:39:07.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a934d640 1 -- 172.21.15.37:0/493557053 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37a40a0110 msgr2=0x7f37a41cfd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:39:07.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a934d640 1 --2- 172.21.15.37:0/493557053 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37a40a0110 0x7f37a41cfd30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f37a000c470 tx=0x7f37a000c930 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:39:07.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a934d640 1 --2- 172.21.15.37:0/493557053 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37a40a0110 0x7f37a41cfd30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f37a000c470 tx=0x7f37a000c930 comp rx=0 tx=0).stop 2024-04-26T00:39:07.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f379a7fc640 10 monclient: ms_handle_reset current mon [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:07.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f379a7fc640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:07.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f379a7fc640 1 -- 172.21.15.37:0/493557053 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37a40a0110 msgr2=0x7f37a41cfd30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:39:07.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f379a7fc640 1 --2- 172.21.15.37:0/493557053 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37a40a0110 0x7f37a41cfd30 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:07.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f379a7fc640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:39:07.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f379a7fc640 1 --2- 172.21.15.37:0/493557053 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3788003870 0x7f3788005c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:07.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f379a7fc640 10 monclient(hunting): picked mon.c con 0x7f3788003870 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:07.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f379a7fc640 1 --2- 172.21.15.37:0/493557053 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f37a41374e0 0x7f3788008720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:07.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f379a7fc640 10 monclient(hunting): picked mon.a con 0x7f37a41374e0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:39:07.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f379a7fc640 1 --2- 172.21.15.37:0/493557053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37a40726b0 0x7f378800b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:07.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f379a7fc640 10 monclient(hunting): picked mon.b con 0x7f37a40726b0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:39:07.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f379a7fc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:07.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f379a7fc640 1 -- 172.21.15.37:0/493557053 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f378800bbb0 con 0x7f37a41374e0 2024-04-26T00:39:07.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f379a7fc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:07.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f379a7fc640 1 -- 172.21.15.37:0/493557053 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f378800bd30 con 0x7f37a40726b0 2024-04-26T00:39:07.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f379a7fc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:07.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f379a7fc640 1 -- 172.21.15.37:0/493557053 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f378800bee0 con 0x7f3788003870 2024-04-26T00:39:07.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f379a7fc640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:07.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a9b4e640 1 --2- 172.21.15.37:0/493557053 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3788003870 0x7f3788005c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:07.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a9b4e640 10 monclient(hunting): get_auth_request con 0x7f3788003870 auth_method 0 2024-04-26T00:39:07.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a9b4e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:07.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a9b4e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:07.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a9b4e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:39:07.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a934d640 1 --2- 172.21.15.37:0/493557053 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f37a41374e0 0x7f3788008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:07.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a934d640 10 monclient(hunting): get_auth_request con 0x7f37a41374e0 auth_method 0 2024-04-26T00:39:07.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a934d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:07.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a934d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:07.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a934d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:39:07.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a9b4e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:07.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a9b4e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:07.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37abdb2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:07.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a9b4e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:39:07.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37abdb2640 1 --2- 172.21.15.37:0/493557053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37a41cffb0 0x7f37a4130b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:07.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37abdb2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:07.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37abdb2640 1 -- 172.21.15.37:0/493557053 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f37a413adb0 con 0x7f37a41cffb0 2024-04-26T00:39:07.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a934d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:07.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a934d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:07.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a934d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:39:07.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37aa34f640 1 --2- 172.21.15.37:0/493557053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37a40726b0 0x7f378800b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:07.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37aa34f640 10 monclient(hunting): get_auth_request con 0x7f37a40726b0 auth_method 0 2024-04-26T00:39:07.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37aa34f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:07.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37aa34f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:07.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37aa34f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:39:07.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a9b4e640 10 monclient(hunting): handle_auth_done global_id 204443 payload 293 2024-04-26T00:39:07.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a9b4e640 1 -- 172.21.15.37:0/493557053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37a40726b0 msgr2=0x7f378800b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:07.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a9b4e640 1 --2- 172.21.15.37:0/493557053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37a40726b0 0x7f378800b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:07.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a9b4e640 1 -- 172.21.15.37:0/493557053 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f37a41374e0 msgr2=0x7f3788008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:07.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a9b4e640 1 --2- 172.21.15.37:0/493557053 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f37a41374e0 0x7f3788008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:07.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a9b4e640 10 monclient: _finish_hunting 0 2024-04-26T00:39:07.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a9b4e640 1 monclient: found mon.c 2024-04-26T00:39:07.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a9b4e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:07.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a9b4e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:07.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a9b4e640 1 -- 172.21.15.37:0/493557053 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7f378800c210 con 0x7f3788003870 2024-04-26T00:39:07.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a9b4e640 10 monclient: _finish_auth 0 2024-04-26T00:39:07.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a9b4e640 10 monclient: _check_auth_tickets 2024-04-26T00:39:07.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a9b4e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:07.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a9b4e640 1 --2- 172.21.15.37:0/493557053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37a41cffb0 0x7f37a4130b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:07.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a9b4e640 10 monclient: get_auth_request con 0x7f37a41cffb0 auth_method 0 2024-04-26T00:39:07.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a9b4e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:07.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a9b4e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:07.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a9b4e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:07.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a9b4e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:07.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a9b4e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:07.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a9b4e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:07.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a9b4e640 10 monclient(hunting): handle_auth_done global_id 224212 payload 824 2024-04-26T00:39:07.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.082+0000 7f37a9b4e640 1 -- 172.21.15.37:0/493557053 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f37a40a05d0 con 0x7f37a41cffb0 2024-04-26T00:39:07.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.086+0000 7f37a9b4e640 1 -- 172.21.15.37:0/493557053 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3788003870 msgr2=0x7f3788005c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:07.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.086+0000 7f37a9b4e640 1 --2- 172.21.15.37:0/493557053 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3788003870 0x7f3788005c60 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f37940ab000 tx=0x7f37a0004a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:07.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.086+0000 7f379a7fc640 1 -- 172.21.15.37:0/493557053 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37940b1040 con 0x7f3788003870 2024-04-26T00:39:07.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.086+0000 7f379a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:07.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.086+0000 7f379a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:07.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.086+0000 7f379a7fc640 10 monclient: dump: 2024-04-26T00:39:07.101 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:07.101 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:07.101 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:07.101 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:07.101 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:07.101 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:07.101 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:07.101 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:07.101 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:07.101 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:07.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.086+0000 7f379a7fc640 1 -- 172.21.15.37:0/493557053 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37940c2440 con 0x7f3788003870 2024-04-26T00:39:07.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.086+0000 7f379a7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:07.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.086+0000 7f379b7fe640 4 set_mon_vals no callback set 2024-04-26T00:39:07.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.086+0000 7f379a7fc640 1 -- 172.21.15.37:0/493557053 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f37940b8420 con 0x7f3788003870 2024-04-26T00:39:07.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.086+0000 7f379a7fc640 1 --2- 172.21.15.37:0/493557053 >> v2:172.21.15.45:6812/2379918623 conn(0x7f37a40a0110 0x7f3788040660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:07.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.086+0000 7f379a7fc640 1 -- 172.21.15.37:0/493557053 <== mon.2 v2:172.21.15.37:3301/0 4 ==== osd_map(757..757 src has 705..757) ==== 3915+0+0 (secure 0 0 0) 0x7f37940e28f0 con 0x7f3788003870 2024-04-26T00:39:07.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.086+0000 7f37a9b4e640 1 -- 172.21.15.37:0/493557053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37a41cffb0 msgr2=0x7f37a4130b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:07.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.086+0000 7f37a9b4e640 1 --2- 172.21.15.37:0/493557053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37a41cffb0 0x7f37a4130b80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f379401c180 tx=0x7f37940046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:07.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.086+0000 7f37a934d640 1 --2- 172.21.15.37:0/493557053 >> v2:172.21.15.45:6812/2379918623 conn(0x7f37a40a0110 0x7f3788040660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:07.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.086+0000 7f37a934d640 10 monclient: get_auth_request con 0x7f37a40a0110 auth_method 0 2024-04-26T00:39:07.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.086+0000 7f379a7fc640 1 -- 172.21.15.37:0/493557053 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37940aa6c0 con 0x7f37a41cffb0 2024-04-26T00:39:07.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.086+0000 7f379a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:07.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.086+0000 7f379a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:39:07.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.086+0000 7f379a7fc640 10 monclient: dump: 2024-04-26T00:39:07.103 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:07.103 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:07.103 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:07.103 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:07.103 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:07.103 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:07.103 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:07.103 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:07.103 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:07.103 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:07.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.086+0000 7f379a7fc640 1 -- 172.21.15.37:0/493557053 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f37940c1800 con 0x7f37a41cffb0 2024-04-26T00:39:07.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.086+0000 7f379a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:07.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.086+0000 7f379a7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:39:07.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.086+0000 7f379a7fc640 1 -- 172.21.15.37:0/493557053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37a41cffb0 msgr2=0x7f37a4130b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:07.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.086+0000 7f379a7fc640 1 --2- 172.21.15.37:0/493557053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37a41cffb0 0x7f37a4130b80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f379401c180 tx=0x7f37940046d0 comp rx=0 tx=0).stop 2024-04-26T00:39:07.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.086+0000 7f379a7fc640 1 -- 172.21.15.37:0/493557053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37a41cffb0 msgr2=0x7f37a4130b80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:39:07.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.086+0000 7f379a7fc640 1 --2- 172.21.15.37:0/493557053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37a41cffb0 0x7f37a4130b80 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:07.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.090+0000 7f37a934d640 1 --2- 172.21.15.37:0/493557053 >> v2:172.21.15.45:6812/2379918623 conn(0x7f37a40a0110 0x7f3788040660 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f37a409eb10 tx=0x7f37a002a040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:07.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.094+0000 7f37abdb2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:39:07.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.094+0000 7f37abdb2640 1 --2- 172.21.15.37:0/493557053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37a41d0390 0x7f37a41310c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:07.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.094+0000 7f37abdb2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:07.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.094+0000 7f37abdb2640 1 -- 172.21.15.37:0/493557053 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f37a413e9a0 con 0x7f37a41d0390 2024-04-26T00:39:07.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.094+0000 7f37aa34f640 1 --2- 172.21.15.37:0/493557053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37a41d0390 0x7f37a41310c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:07.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.094+0000 7f37aa34f640 10 monclient: get_auth_request con 0x7f37a41d0390 auth_method 0 2024-04-26T00:39:07.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.094+0000 7f37aa34f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:07.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.094+0000 7f37aa34f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:07.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.094+0000 7f37aa34f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:07.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.094+0000 7f37aa34f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:07.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.094+0000 7f37aa34f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:07.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.094+0000 7f37aa34f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:07.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.094+0000 7f37aa34f640 10 monclient(hunting): handle_auth_done global_id 224215 payload 824 2024-04-26T00:39:07.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.094+0000 7f37aa34f640 1 -- 172.21.15.37:0/493557053 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f37a4073240 con 0x7f37a41d0390 2024-04-26T00:39:07.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.098+0000 7f37aa34f640 1 -- 172.21.15.37:0/493557053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37a41d0390 msgr2=0x7f37a41310c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:07.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.098+0000 7f37aa34f640 1 --2- 172.21.15.37:0/493557053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37a41d0390 0x7f37a41310c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f37a413d380 tx=0x7f379c00f5a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:07.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.098+0000 7f379a7fc640 1 -- 172.21.15.37:0/493557053 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f379c01b030 con 0x7f37a41d0390 2024-04-26T00:39:07.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.098+0000 7f379a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:07.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.098+0000 7f379a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:39:07.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.098+0000 7f379a7fc640 10 monclient: dump: 2024-04-26T00:39:07.107 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:07.107 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:07.107 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:07.108 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:07.108 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:07.108 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:07.108 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:07.108 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:07.108 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:07.108 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:07.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.098+0000 7f379a7fc640 1 -- 172.21.15.37:0/493557053 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f379c011e80 con 0x7f37a41d0390 2024-04-26T00:39:07.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.098+0000 7f379a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:39:07.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.098+0000 7f379a7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:39:07.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.098+0000 7f379a7fc640 1 -- 172.21.15.37:0/493557053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37a41d0390 msgr2=0x7f37a41310c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:07.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.098+0000 7f379a7fc640 1 --2- 172.21.15.37:0/493557053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37a41d0390 0x7f37a41310c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f37a413d380 tx=0x7f379c00f5a0 comp rx=0 tx=0).stop 2024-04-26T00:39:07.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.098+0000 7f379a7fc640 1 -- 172.21.15.37:0/493557053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37a41d0390 msgr2=0x7f37a41310c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:39:07.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.098+0000 7f379a7fc640 1 --2- 172.21.15.37:0/493557053 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f37a41d0390 0x7f37a41310c0 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f37a413d380 tx=0x7f379c00f5a0 comp rx=0 tx=0).stop 2024-04-26T00:39:07.109 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:39:07.109 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:39:07.109 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:39:07.109 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:39:07.109 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 144, 2024-04-26T00:39:07.109 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 60759, 2024-04-26T00:39:07.109 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:39:07.109 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:39:07.109 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:39:07.109 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:39:07.109 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:07.109 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 4, 2024-04-26T00:39:07.110 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:39:07.110 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:39:07.110 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:39:07.110 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:07.110 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:07.110 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:07.110 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:07.110 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:07.110 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:07.110 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:07.110 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:07.110 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:07.110 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:07.110 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:07.111 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:07.111 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:39:07.111 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:39:07.111 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:07.111 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:07.111 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:07.111 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:07.111 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:07.111 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:07.112 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:07.112 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:07.112 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:07.112 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:07.112 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:07.112 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:07.112 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:07.112 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:39:07.112 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:39:07.112 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:39:07.112 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:39:07.112 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:39:07.112 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:39:07.112 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:39:07.112 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:39:07.113 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:39:07.113 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:39:07.113 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:39:07.113 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:39:07.113 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:39:07.113 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:39:07.113 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:39:07.113 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:39:07.113 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:39:07.113 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:07.113 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:07.113 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:07.113 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:07.113 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:07.113 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:07.114 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:07.114 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:07.114 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:07.114 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:07.114 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:07.114 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:07.114 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:39:07.114 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:07.114 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:39:07.114 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:07.114 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:39:07.114 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:39:07.114 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:07.114 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:07.115 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:07.115 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:07.115 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:39:07.115 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:07.115 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:07.115 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:07.115 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:07.115 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:39:07.115 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:07.115 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:07.115 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:07.115 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:07.116 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:39:07.116 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:39:07.116 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:07.116 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:07.116 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:07.116 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:07.116 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:07.116 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:39:07.116 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:39:07.116 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:07.116 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:07.116 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:07.116 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:07.116 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:39:07.116 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:07.117 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:07.117 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:07.117 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:07.117 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:39:07.117 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:07.117 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:07.117 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:07.117 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:07.117 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:39:07.117 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:39:07.117 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:07.117 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:07.117 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:07.118 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:07.118 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:07.118 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:39:07.118 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:39:07.118 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:07.118 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:07.118 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:07.118 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:07.118 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:39:07.118 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:07.118 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:07.118 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:07.118 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:07.118 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:39:07.119 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:07.119 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:07.119 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:07.119 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:07.119 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:39:07.119 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:39:07.119 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:07.119 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:07.119 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:07.119 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:07.119 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:07.119 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:07.119 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:39:07.119 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:39:07.120 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:07.120 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:07.120 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:07.120 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:39:07.120 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:07.120 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:07.120 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:39:07.120 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:07.120 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:07.120 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:07.120 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:39:07.121 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:07.121 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:07.121 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:39:07.121 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:07.121 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:07.121 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:07.121 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:39:07.121 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:07.121 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:07.121 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:39:07.122 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:07.122 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:07.122 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:07.122 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:39:07.122 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:07.122 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:07.122 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:07.122 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:39:07.122 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:39:07.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.098+0000 7f37abdb2640 1 -- 172.21.15.37:0/493557053 >> v2:172.21.15.45:6812/2379918623 conn(0x7f37a40a0110 msgr2=0x7f3788040660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:07.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.098+0000 7f37abdb2640 1 --2- 172.21.15.37:0/493557053 >> v2:172.21.15.45:6812/2379918623 conn(0x7f37a40a0110 0x7f3788040660 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f37a409eb10 tx=0x7f37a002a040 comp rx=0 tx=0).stop 2024-04-26T00:39:07.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.098+0000 7f37abdb2640 10 monclient: shutdown 2024-04-26T00:39:07.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.098+0000 7f37abdb2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:07.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.098+0000 7f37abdb2640 1 -- 172.21.15.37:0/493557053 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3788003870 msgr2=0x7f3788005c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:07.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.098+0000 7f37abdb2640 1 --2- 172.21.15.37:0/493557053 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3788003870 0x7f3788005c60 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f37940ab000 tx=0x7f37a0004a20 comp rx=0 tx=0).stop 2024-04-26T00:39:07.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.098+0000 7f37aa34f640 1 -- 172.21.15.37:0/493557053 reap_dead start 2024-04-26T00:39:07.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.098+0000 7f37aa34f640 1 -- 172.21.15.37:0/493557053 reap_dead start 2024-04-26T00:39:07.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.098+0000 7f37abdb2640 1 -- 172.21.15.37:0/493557053 shutdown_connections 2024-04-26T00:39:07.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.098+0000 7f37abdb2640 1 -- 172.21.15.37:0/493557053 >> 172.21.15.37:0/493557053 conn(0x7f37a40a1d50 msgr2=0x7f37a40a3dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:07.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.098+0000 7f37abdb2640 1 -- 172.21.15.37:0/493557053 shutdown_connections 2024-04-26T00:39:07.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.098+0000 7f37abdb2640 1 -- 172.21.15.37:0/493557053 wait complete. 2024-04-26T00:39:07.125 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:39:07.125 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-04-26T00:39:07.125 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-26T00:39:07.125 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-04-26T00:39:07.125 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-04-26T00:39:07.125 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-26T00:39:07.125 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T00:39:07.179 INFO:tasks.ceph.mon.a:Stopped 2024-04-26T00:39:07.179 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-26T00:39:07.180 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:39:07.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.342+0000 7f164b647640 10 monclient: get_monmap_and_config 2024-04-26T00:39:07.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.342+0000 7f164b647640 10 monclient: build_initial_monmap 2024-04-26T00:39:07.343 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.342+0000 7f164b647640 10 monclient: monmap: 2024-04-26T00:39:07.343 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:07.343 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:07.343 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:39:07.346342+0000 2024-04-26T00:39:07.343 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:39:07.346342+0000 2024-04-26T00:39:07.343 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:07.343 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:07.343 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:39:07.344 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:39:07.344 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:39:07.344 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:39:07.344 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:39:07.344 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:07.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.342+0000 7f164b647640 1 Processor -- start 2024-04-26T00:39:07.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.342+0000 7f164b647640 1 -- start start 2024-04-26T00:39:07.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.342+0000 7f164b647640 10 monclient: init 2024-04-26T00:39:07.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.346+0000 7f164b647640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:07.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.346+0000 7f164b647640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-04-26T00:39:07.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.346+0000 7f164b647640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f1644133b20 0x7f164409e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:07.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.346+0000 7f164b647640 10 monclient(hunting): picked mon.noname-c con 0x7f1644133b20 addr v2:172.21.15.45:3300/0 2024-04-26T00:39:07.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.346+0000 7f164b647640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1644133f00 0x7f1644134350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:07.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.346+0000 7f164b647640 10 monclient(hunting): picked mon.noname-b con 0x7f1644133f00 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:07.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.346+0000 7f164b647640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1644130770 addr v1:172.21.15.37:6789/0 2024-04-26T00:39:07.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.346+0000 7f164b647640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f164409fe20 con 0x7f1644130770 2024-04-26T00:39:07.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.346+0000 7f164b647640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:07.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.346+0000 7f164b647640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f1644072d00 con 0x7f1644133b20 2024-04-26T00:39:07.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.346+0000 7f164b647640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:07.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.346+0000 7f164b647640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f164409ba60 con 0x7f1644133f00 2024-04-26T00:39:07.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.346+0000 7f164b647640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:07.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.346+0000 7f164b647640 10 monclient(hunting): authenticate will time out at 2128.652809s 2024-04-26T00:39:07.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.346+0000 7f1649be4640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f1644130770 legacy=0x7f1644130be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:39:07.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.346+0000 7f1648be2640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1644133f00 0x7f1644134350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:07.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.346+0000 7f1648be2640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1644133f00 0x7f1644134350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:35784/0 (socket says 172.21.15.37:35784) 2024-04-26T00:39:07.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.346+0000 7f1648be2640 1 -- 172.21.15.37:0/654085996 learned_addr learned my addr 172.21.15.37:0/654085996 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:07.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.346+0000 7f1648be2640 10 monclient(hunting): get_auth_request con 0x7f1644133f00 auth_method 0 2024-04-26T00:39:07.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.346+0000 7f1648be2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:07.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.346+0000 7f1648be2640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:07.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.346+0000 7f1648be2640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:07.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.346+0000 7f1648be2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:07.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.346+0000 7f1648be2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:07.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.346+0000 7f1648be2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:07.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.346+0000 7f16493e3640 1 --2- 172.21.15.37:0/654085996 >> v2:172.21.15.45:3300/0 conn(0x7f1644133b20 0x7f164409e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:07.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.346+0000 7f16493e3640 10 monclient(hunting): get_auth_request con 0x7f1644133b20 auth_method 0 2024-04-26T00:39:07.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.346+0000 7f16493e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:07.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.346+0000 7f16493e3640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:07.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.346+0000 7f16493e3640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:07.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.346+0000 7f1648be2640 10 monclient(hunting): handle_auth_done global_id 204446 payload 274 2024-04-26T00:39:07.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.346+0000 7f1648be2640 1 -- 172.21.15.37:0/654085996 >> v2:172.21.15.45:3300/0 conn(0x7f1644133b20 msgr2=0x7f164409e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:07.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.346+0000 7f1648be2640 1 --2- 172.21.15.37:0/654085996 >> v2:172.21.15.45:3300/0 conn(0x7f1644133b20 0x7f164409e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:07.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.346+0000 7f1648be2640 1 -- 172.21.15.37:0/654085996 >> v1:172.21.15.37:6789/0 conn(0x7f1644130770 legacy=0x7f1644130be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:39:07.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.346+0000 7f1648be2640 10 monclient: _finish_hunting 0 2024-04-26T00:39:07.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.346+0000 7f1648be2640 1 monclient: found mon.noname-b 2024-04-26T00:39:07.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.346+0000 7f1648be2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:07.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.346+0000 7f1648be2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:39:07.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.346+0000 7f1648be2640 1 -- 172.21.15.37:0/654085996 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f164409dd10 con 0x7f1644133f00 2024-04-26T00:39:07.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.350+0000 7f1648be2640 1 -- 172.21.15.37:0/654085996 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1644133f00 msgr2=0x7f1644134350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:07.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.350+0000 7f1648be2640 1 --2- 172.21.15.37:0/654085996 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1644133f00 0x7f1644134350 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1640009870 tx=0x7f164001c1d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5feaa7754d86f173 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:07.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.350+0000 7f163bfff640 1 -- 172.21.15.37:0/654085996 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16400b0070 con 0x7f1644133f00 2024-04-26T00:39:07.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.350+0000 7f163bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:07.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.350+0000 7f163bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:39:07.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.350+0000 7f163bfff640 10 monclient: dump: 2024-04-26T00:39:07.352 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:07.352 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:07.352 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:07.352 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:07.352 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:07.352 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:07.352 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:07.353 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:07.353 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:07.353 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:07.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.350+0000 7f163bfff640 10 monclient: _finish_auth 0 2024-04-26T00:39:07.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.350+0000 7f163bfff640 10 monclient: _check_auth_tickets 2024-04-26T00:39:07.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.350+0000 7f163bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:07.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.350+0000 7f163bfff640 1 -- 172.21.15.37:0/654085996 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16400ab600 con 0x7f1644133f00 2024-04-26T00:39:07.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.350+0000 7f163bfff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:07.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.350+0000 7f163bfff640 1 -- 172.21.15.37:0/654085996 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16400ab9d0 con 0x7f1644133f00 2024-04-26T00:39:07.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.350+0000 7f164b647640 5 monclient: authenticate success, global_id 204446 2024-04-26T00:39:07.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.350+0000 7f164b647640 10 monclient: get_monmap_and_config success 2024-04-26T00:39:07.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.350+0000 7f164b647640 4 set_mon_vals no callback set 2024-04-26T00:39:07.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.350+0000 7f163bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:07.353 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.350+0000 7f163bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:07.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.350+0000 7f163bfff640 10 monclient: dump: 2024-04-26T00:39:07.354 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:07.354 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:07.354 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:07.354 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:07.354 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:07.354 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:07.354 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:07.354 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:07.354 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:07.354 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:07.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.350+0000 7f164b647640 10 monclient: shutdown 2024-04-26T00:39:07.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.350+0000 7f164b647640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:07.354 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.350+0000 7f164b647640 1 -- 172.21.15.37:0/654085996 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1644133f00 msgr2=0x7f1644134350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:07.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.350+0000 7f164b647640 1 --2- 172.21.15.37:0/654085996 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1644133f00 0x7f1644134350 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1640009870 tx=0x7f164001c1d0 comp rx=0 tx=0).stop 2024-04-26T00:39:07.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.350+0000 7f164b647640 1 -- 172.21.15.37:0/654085996 shutdown_connections 2024-04-26T00:39:07.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.350+0000 7f164b647640 1 --2- 172.21.15.37:0/654085996 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1644133f00 0x7f1644134350 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:07.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.350+0000 7f164b647640 1 --2- 172.21.15.37:0/654085996 >> v2:172.21.15.45:3300/0 conn(0x7f1644133b20 0x7f164409e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:07.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.350+0000 7f164b647640 1 -- 172.21.15.37:0/654085996 >> 172.21.15.37:0/654085996 conn(0x7f16440a1ad0 msgr2=0x7f16440a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:07.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.350+0000 7f164b647640 1 -- 172.21.15.37:0/654085996 shutdown_connections 2024-04-26T00:39:07.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.350+0000 7f164b647640 1 -- 172.21.15.37:0/654085996 wait complete. 2024-04-26T00:39:07.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.350+0000 7f164b647640 10 monclient: build_initial_monmap 2024-04-26T00:39:07.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.350+0000 7f164b647640 10 monclient: monmap: 2024-04-26T00:39:07.355 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:07.355 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:39:07.355 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:39:07.355 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:39:07.356 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:07.356 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:39:07.356 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:39:07.356 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:39:07.356 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:39:07.356 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:07.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.350+0000 7f164b647640 1 Processor -- start 2024-04-26T00:39:07.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.350+0000 7f164b647640 1 -- start start 2024-04-26T00:39:07.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.350+0000 7f164b647640 10 monclient: init 2024-04-26T00:39:07.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f164b647640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:07.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f164b647640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:39:07.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f164b647640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1644130770 0x7f16441cacd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:07.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f164b647640 10 monclient(hunting): picked mon.noname-c con 0x7f1644130770 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:07.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f164b647640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1644133b20 0x7f164413b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:07.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f164b647640 10 monclient(hunting): picked mon.noname-a con 0x7f1644133b20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:39:07.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f164b647640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1644133f00 0x7f164413b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:07.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f164b647640 10 monclient(hunting): picked mon.noname-b con 0x7f1644133f00 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:39:07.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f164b647640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:07.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f164b647640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f16440a1750 con 0x7f1644133b20 2024-04-26T00:39:07.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f164b647640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:07.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f164b647640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f16440a15d0 con 0x7f1644133f00 2024-04-26T00:39:07.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f164b647640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:07.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f164b647640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f16440a0260 con 0x7f1644130770 2024-04-26T00:39:07.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f164b647640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:07.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f164b647640 10 monclient(hunting): authenticate will time out at 2128.660253s 2024-04-26T00:39:07.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f1648be2640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1644133b20 msgr2=0x7f164413b030 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:39:07.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f1648be2640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1644133b20 0x7f164413b030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:39:07.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f16493e3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1644130770 0x7f16441cacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:07.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f16493e3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1644130770 0x7f16441cacd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:35796/0 (socket says 172.21.15.37:35796) 2024-04-26T00:39:07.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f16493e3640 1 -- 172.21.15.37:0/991282669 learned_addr learned my addr 172.21.15.37:0/991282669 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:07.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f16493e3640 10 monclient(hunting): get_auth_request con 0x7f1644130770 auth_method 0 2024-04-26T00:39:07.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f16493e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:07.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f16493e3640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:07.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f16493e3640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:07.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f16493e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:07.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f16493e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:07.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f16493e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:07.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f1649be4640 1 --2- 172.21.15.37:0/991282669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1644133f00 0x7f164413b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:07.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f16493e3640 10 monclient(hunting): handle_auth_done global_id 204449 payload 824 2024-04-26T00:39:07.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f16493e3640 1 -- 172.21.15.37:0/991282669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1644133f00 msgr2=0x7f164413b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:07.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f16493e3640 1 --2- 172.21.15.37:0/991282669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1644133f00 0x7f164413b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:07.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f16493e3640 1 -- 172.21.15.37:0/991282669 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1644133b20 msgr2=0x7f164413b030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:39:07.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f16493e3640 1 --2- 172.21.15.37:0/991282669 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1644133b20 0x7f164413b030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:07.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f16493e3640 10 monclient: _finish_hunting 0 2024-04-26T00:39:07.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f16493e3640 1 monclient: found mon.noname-c 2024-04-26T00:39:07.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f16493e3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:07.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f16493e3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:39:07.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f16493e3640 1 -- 172.21.15.37:0/991282669 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16440a0000 con 0x7f1644130770 2024-04-26T00:39:07.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f1649be4640 10 monclient: get_auth_request con 0x7f1644133f00 auth_method 0 2024-04-26T00:39:07.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f1649be4640 1 --2- 172.21.15.37:0/991282669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1644133f00 0x7f164413b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:39:07.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f16493e3640 1 -- 172.21.15.37:0/991282669 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1644130770 msgr2=0x7f16441cacd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:07.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f16493e3640 1 --2- 172.21.15.37:0/991282669 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1644130770 0x7f16441cacd0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f163400d830 tx=0x7f163400dcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:07.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f1639ffb640 1 -- 172.21.15.37:0/991282669 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f163400b8d0 con 0x7f1644130770 2024-04-26T00:39:07.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f1639ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:07.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f1639ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:39:07.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f1639ffb640 10 monclient: dump: 2024-04-26T00:39:07.364 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:07.364 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:07.364 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:07.364 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:07.364 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:07.364 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:07.364 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:07.364 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:07.364 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:07.364 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:07.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f1639ffb640 10 monclient: _finish_auth 0 2024-04-26T00:39:07.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f1639ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:39:07.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f1639ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:07.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f1639ffb640 1 -- 172.21.15.37:0/991282669 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1634019070 con 0x7f1644130770 2024-04-26T00:39:07.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f164b647640 5 monclient: authenticate success, global_id 204449 2024-04-26T00:39:07.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f164b647640 10 monclient: _renew_subs 2024-04-26T00:39:07.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f164b647640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:07.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f164b647640 1 -- 172.21.15.37:0/991282669 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16440a4940 con 0x7f1644130770 2024-04-26T00:39:07.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f1639ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:07.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f1639ffb640 1 -- 172.21.15.37:0/991282669 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f163400bc70 con 0x7f1644130770 2024-04-26T00:39:07.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f1639ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:07.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f1639ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:07.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f1639ffb640 10 monclient: dump: 2024-04-26T00:39:07.365 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:07.365 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:07.366 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:07.366 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:07.366 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:07.366 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:07.366 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:07.366 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:07.366 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:07.366 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:07.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f164b647640 10 monclient: _renew_subs 2024-04-26T00:39:07.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f164b647640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:07.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f164b647640 1 -- 172.21.15.37:0/991282669 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f16441cb210 con 0x7f1644130770 2024-04-26T00:39:07.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.354+0000 7f163affd640 4 set_mon_vals no callback set 2024-04-26T00:39:07.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.358+0000 7f1639ffb640 1 -- 172.21.15.37:0/991282669 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f163401e460 con 0x7f1644130770 2024-04-26T00:39:07.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.358+0000 7f164b647640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:07.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.358+0000 7f164b647640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:07.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.358+0000 7f164b647640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:07.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.358+0000 7f164b647640 1 -- 172.21.15.37:0/991282669 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1644066f40 con 0x7f1644130770 2024-04-26T00:39:07.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.358+0000 7f1639ffb640 1 --2- 172.21.15.37:0/991282669 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1628035e80 0x7f1628038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:07.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.358+0000 7f1639ffb640 1 -- 172.21.15.37:0/991282669 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(757..757 src has 705..757) ==== 3915+0+0 (secure 0 0 0) 0x7f1634048ff0 con 0x7f1644130770 2024-04-26T00:39:07.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.358+0000 7f1648be2640 1 --2- 172.21.15.37:0/991282669 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1628035e80 0x7f1628038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:07.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.358+0000 7f1648be2640 10 monclient: get_auth_request con 0x7f1628035e80 auth_method 0 2024-04-26T00:39:07.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.358+0000 7f1648be2640 1 --2- 172.21.15.37:0/991282669 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1628035e80 0x7f1628038340 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f164406db40 tx=0x7f16400a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:07.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.362+0000 7f1639ffb640 1 -- 172.21.15.37:0/991282669 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f163400be10 con 0x7f1644130770 2024-04-26T00:39:07.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.362+0000 7f1639ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:07.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.362+0000 7f1639ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:39:07.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.582+0000 7f164b647640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:39:07.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.582+0000 7f164b647640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:39:07.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.582+0000 7f164b647640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:07.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.582+0000 7f164b647640 1 -- 172.21.15.37:0/991282669 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1644134ef0 con 0x7f1644130770 2024-04-26T00:39:07.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.582+0000 7f1639ffb640 1 -- 172.21.15.37:0/991282669 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f1634047d50 con 0x7f1644130770 2024-04-26T00:39:07.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.582+0000 7f1639ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:39:07.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.582+0000 7f1639ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:39:07.584 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":144,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:39:07.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.586+0000 7f164b647640 1 -- 172.21.15.37:0/991282669 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1628035e80 msgr2=0x7f1628038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:07.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.586+0000 7f164b647640 1 --2- 172.21.15.37:0/991282669 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1628035e80 0x7f1628038340 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f164406db40 tx=0x7f16400a6000 comp rx=0 tx=0).stop 2024-04-26T00:39:07.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.586+0000 7f164b647640 10 monclient: shutdown 2024-04-26T00:39:07.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.586+0000 7f164b647640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:07.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.586+0000 7f164b647640 1 -- 172.21.15.37:0/991282669 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1644130770 msgr2=0x7f16441cacd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:07.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.586+0000 7f164b647640 1 --2- 172.21.15.37:0/991282669 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1644130770 0x7f16441cacd0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f163400d830 tx=0x7f163400dcf0 comp rx=0 tx=0).stop 2024-04-26T00:39:07.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.586+0000 7f164b647640 1 -- 172.21.15.37:0/991282669 shutdown_connections 2024-04-26T00:39:07.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.586+0000 7f164b647640 1 --2- 172.21.15.37:0/991282669 >> v2:172.21.15.45:6812/2379918623 conn(0x7f1628035e80 0x7f1628038340 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:07.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.586+0000 7f164b647640 1 --2- 172.21.15.37:0/991282669 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1644133f00 0x7f164413b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:07.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.586+0000 7f164b647640 1 --2- 172.21.15.37:0/991282669 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1644133b20 0x7f164413b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:07.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.586+0000 7f164b647640 1 --2- 172.21.15.37:0/991282669 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1644130770 0x7f16441cacd0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:07.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.586+0000 7f164b647640 1 -- 172.21.15.37:0/991282669 >> 172.21.15.37:0/991282669 conn(0x7f16440a1ad0 msgr2=0x7f164409c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:07.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.586+0000 7f164b647640 1 -- 172.21.15.37:0/991282669 shutdown_connections 2024-04-26T00:39:07.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:07.586+0000 7f164b647640 1 -- 172.21.15.37:0/991282669 wait complete. 2024-04-26T00:39:10.603 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:39:10.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.734+0000 7f890516e640 10 monclient: get_monmap_and_config 2024-04-26T00:39:10.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.734+0000 7f890516e640 10 monclient: build_initial_monmap 2024-04-26T00:39:10.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.734+0000 7f890516e640 10 monclient: monmap: 2024-04-26T00:39:10.734 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:10.734 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:10.734 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:39:10.737165+0000 2024-04-26T00:39:10.734 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:39:10.737165+0000 2024-04-26T00:39:10.734 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:10.734 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:10.734 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:39:10.734 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:39:10.734 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:39:10.734 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:39:10.734 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:39:10.734 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:10.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.734+0000 7f890516e640 1 Processor -- start 2024-04-26T00:39:10.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.734+0000 7f890516e640 1 -- start start 2024-04-26T00:39:10.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.734+0000 7f890516e640 10 monclient: init 2024-04-26T00:39:10.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.734+0000 7f890516e640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:10.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.734+0000 7f890516e640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-04-26T00:39:10.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.734+0000 7f890516e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f89000a0260 addr v1:172.21.15.37:6789/0 2024-04-26T00:39:10.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.734+0000 7f890516e640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8900137e40 0x7f890013a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:10.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.734+0000 7f890516e640 10 monclient(hunting): picked mon.noname-c con 0x7f8900137e40 addr v2:172.21.15.45:3300/0 2024-04-26T00:39:10.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.734+0000 7f890516e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f890013a770 addr v1:172.21.15.45:6789/0 2024-04-26T00:39:10.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.734+0000 7f890516e640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89000a1130 con 0x7f89000a0260 2024-04-26T00:39:10.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.734+0000 7f890516e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f890006cff0 con 0x7f890013a770 2024-04-26T00:39:10.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.734+0000 7f890516e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:10.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.734+0000 7f890516e640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f89000a0d50 con 0x7f8900137e40 2024-04-26T00:39:10.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.734+0000 7f890516e640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:10.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.734+0000 7f890516e640 10 monclient(hunting): authenticate will time out at 2132.042820s 2024-04-26T00:39:10.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.734+0000 7f88fed9d640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f89000a0260 legacy=0x7f89000a0640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:39:10.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.734+0000 7f88fe59c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8900137e40 0x7f890013a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:10.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.734+0000 7f88ff59e640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f890013a770 0x7f890013cb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.37:55848/0 (socket says 172.21.15.37:55848) 2024-04-26T00:39:10.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.734+0000 7f88ff59e640 1 -- 172.21.15.37:0/3327473526 learned_addr learned my addr 172.21.15.37:0/3327473526 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:39:10.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.734+0000 7f88fe59c640 10 monclient(hunting): get_auth_request con 0x7f8900137e40 auth_method 0 2024-04-26T00:39:10.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.734+0000 7f88fe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:10.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.734+0000 7f88fe59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:10.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.734+0000 7f88fe59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:10.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.734+0000 7f88fe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:10.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.734+0000 7f88fe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:10.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.734+0000 7f88fe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:10.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.734+0000 7f88fe59c640 10 monclient(hunting): handle_auth_done global_id 224221 payload 274 2024-04-26T00:39:10.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.734+0000 7f88fe59c640 1 -- 172.21.15.37:0/3327473526 >> v1:172.21.15.45:6789/0 conn(0x7f890013a770 legacy=0x7f890013cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:10.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.734+0000 7f88fe59c640 1 -- 172.21.15.37:0/3327473526 >> v1:172.21.15.37:6789/0 conn(0x7f89000a0260 legacy=0x7f89000a0640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:39:10.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.734+0000 7f88fe59c640 10 monclient: _finish_hunting 0 2024-04-26T00:39:10.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.734+0000 7f88fe59c640 1 monclient: found mon.noname-c 2024-04-26T00:39:10.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.734+0000 7f88fe59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:10.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.734+0000 7f88fe59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:39:10.739 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.734+0000 7f88fe59c640 1 -- 172.21.15.37:0/3327473526 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f890009bc50 con 0x7f8900137e40 2024-04-26T00:39:10.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f88fe59c640 1 -- 172.21.15.37:0/3327473526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8900137e40 msgr2=0x7f890013a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:10.740 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f88fe59c640 1 --2- 172.21.15.37:0/3327473526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8900137e40 0x7f890013a230 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f88f00097c0 tx=0x7f88f001c100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6857a47c0282b31e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:10.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f88fdd9b640 1 -- 172.21.15.37:0/3327473526 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f00b1070 con 0x7f8900137e40 2024-04-26T00:39:10.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f88fdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:10.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f88fdd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:39:10.741 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f88fdd9b640 10 monclient: dump: 2024-04-26T00:39:10.741 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:10.741 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:10.742 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:10.742 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:10.742 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:10.742 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:10.742 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:10.742 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:10.742 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:10.742 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:10.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f88fdd9b640 10 monclient: _finish_auth 0 2024-04-26T00:39:10.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f88fdd9b640 10 monclient: _check_auth_tickets 2024-04-26T00:39:10.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f88fdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:10.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f88fdd9b640 1 -- 172.21.15.37:0/3327473526 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88f00ac440 con 0x7f8900137e40 2024-04-26T00:39:10.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f88fdd9b640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:10.742 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f88fdd9b640 1 -- 172.21.15.37:0/3327473526 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f00ac810 con 0x7f8900137e40 2024-04-26T00:39:10.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f88fdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:10.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f88fdd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:39:10.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f88fdd9b640 10 monclient: dump: 2024-04-26T00:39:10.743 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:10.743 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:10.743 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:10.743 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:10.743 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:10.743 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:10.743 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:10.743 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:10.743 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:10.743 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:10.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f890516e640 5 monclient: authenticate success, global_id 224221 2024-04-26T00:39:10.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f890516e640 10 monclient: get_monmap_and_config success 2024-04-26T00:39:10.743 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f890516e640 4 set_mon_vals no callback set 2024-04-26T00:39:10.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f890516e640 10 monclient: shutdown 2024-04-26T00:39:10.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f890516e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:10.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f890516e640 1 -- 172.21.15.37:0/3327473526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8900137e40 msgr2=0x7f890013a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:10.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f890516e640 1 --2- 172.21.15.37:0/3327473526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8900137e40 0x7f890013a230 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f88f00097c0 tx=0x7f88f001c100 comp rx=0 tx=0).stop 2024-04-26T00:39:10.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f890516e640 1 -- 172.21.15.37:0/3327473526 shutdown_connections 2024-04-26T00:39:10.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f890516e640 1 --2- 172.21.15.37:0/3327473526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8900137e40 0x7f890013a230 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:10.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f890516e640 1 -- 172.21.15.37:0/3327473526 >> 172.21.15.37:0/3327473526 conn(0x7f89000a1ad0 msgr2=0x7f89000a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:10.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f890516e640 1 -- 172.21.15.37:0/3327473526 shutdown_connections 2024-04-26T00:39:10.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f890516e640 1 -- 172.21.15.37:0/3327473526 wait complete. 2024-04-26T00:39:10.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f890516e640 10 monclient: build_initial_monmap 2024-04-26T00:39:10.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f890516e640 10 monclient: monmap: 2024-04-26T00:39:10.748 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:10.748 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:39:10.748 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:39:10.749 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:39:10.749 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:10.749 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:39:10.749 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:39:10.749 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:39:10.749 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:39:10.749 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:10.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f890516e640 1 Processor -- start 2024-04-26T00:39:10.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f890516e640 1 -- start start 2024-04-26T00:39:10.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f890516e640 10 monclient: init 2024-04-26T00:39:10.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f890516e640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:10.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f890516e640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:39:10.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f890516e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89000a0260 0x7f8900070dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:10.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f890516e640 10 monclient(hunting): picked mon.noname-b con 0x7f89000a0260 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:39:10.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f890516e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8900137e40 0x7f8900134910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:10.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f890516e640 10 monclient(hunting): picked mon.noname-c con 0x7f8900137e40 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:10.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f890516e640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f890013a770 0x7f890011b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:10.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f890516e640 10 monclient(hunting): picked mon.noname-a con 0x7f890013a770 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:39:10.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f890516e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:10.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f890516e640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f890009f0f0 con 0x7f890013a770 2024-04-26T00:39:10.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f890516e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:10.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f890516e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f890009f4c0 con 0x7f89000a0260 2024-04-26T00:39:10.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f890516e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:10.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f890516e640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f89000a00a0 con 0x7f8900137e40 2024-04-26T00:39:10.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f890516e640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:10.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f890516e640 10 monclient(hunting): authenticate will time out at 2132.051745s 2024-04-26T00:39:10.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f88ff59e640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f890013a770 msgr2=0x7f890011b620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:39:10.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f88ff59e640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f890013a770 0x7f890011b620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:39:10.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f88fe59c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8900137e40 0x7f8900134910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:10.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f88fe59c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8900137e40 0x7f8900134910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:44958/0 (socket says 172.21.15.37:44958) 2024-04-26T00:39:10.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f88fe59c640 1 -- 172.21.15.37:0/3621198569 learned_addr learned my addr 172.21.15.37:0/3621198569 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:10.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f88fe59c640 10 monclient(hunting): get_auth_request con 0x7f8900137e40 auth_method 0 2024-04-26T00:39:10.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f88fe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:10.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f88fe59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:10.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.742+0000 7f88fe59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:10.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f88fe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:10.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f88fe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:10.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f88fe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:10.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f88fed9d640 1 --2- 172.21.15.37:0/3621198569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89000a0260 0x7f8900070dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:10.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f88fed9d640 10 monclient(hunting): get_auth_request con 0x7f89000a0260 auth_method 0 2024-04-26T00:39:10.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f88fed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:10.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f88fed9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:10.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f88fed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:10.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f88fe59c640 10 monclient(hunting): handle_auth_done global_id 204452 payload 824 2024-04-26T00:39:10.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f88fe59c640 1 -- 172.21.15.37:0/3621198569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89000a0260 msgr2=0x7f8900070dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:10.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f88fe59c640 1 --2- 172.21.15.37:0/3621198569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89000a0260 0x7f8900070dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:10.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f88fe59c640 1 -- 172.21.15.37:0/3621198569 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f890013a770 msgr2=0x7f890011b620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:39:10.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f88fe59c640 1 --2- 172.21.15.37:0/3621198569 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f890013a770 0x7f890011b620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:10.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f88fe59c640 10 monclient: _finish_hunting 0 2024-04-26T00:39:10.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f88fe59c640 1 monclient: found mon.noname-c 2024-04-26T00:39:10.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f88fe59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:10.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f88fe59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:39:10.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f88fe59c640 1 -- 172.21.15.37:0/3621198569 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f890011e2b0 con 0x7f8900137e40 2024-04-26T00:39:10.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f88fed9d640 1 --2- 172.21.15.37:0/3621198569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89000a0260 0x7f8900070dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:39:10.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f88fe59c640 1 -- 172.21.15.37:0/3621198569 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8900137e40 msgr2=0x7f8900134910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:10.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f88fe59c640 1 --2- 172.21.15.37:0/3621198569 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8900137e40 0x7f8900134910 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f88f001a7b0 tx=0x7f88f0018ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:10.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f88f77fe640 1 -- 172.21.15.37:0/3621198569 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f00ba020 con 0x7f8900137e40 2024-04-26T00:39:10.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f88f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:10.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f88f77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:39:10.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f88f77fe640 10 monclient: dump: 2024-04-26T00:39:10.753 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:10.753 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:10.753 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:10.753 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:10.753 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:10.753 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:10.753 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:10.753 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:10.753 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:10.753 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:10.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f88f77fe640 10 monclient: _finish_auth 0 2024-04-26T00:39:10.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f88f77fe640 10 monclient: _check_auth_tickets 2024-04-26T00:39:10.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f88f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:10.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f88f77fe640 1 -- 172.21.15.37:0/3621198569 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88f00ace50 con 0x7f8900137e40 2024-04-26T00:39:10.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f88f77fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:10.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f88f77fe640 1 -- 172.21.15.37:0/3621198569 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f00b1040 con 0x7f8900137e40 2024-04-26T00:39:10.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f88f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:10.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f88f77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:10.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f88f77fe640 10 monclient: dump: 2024-04-26T00:39:10.754 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:10.754 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:10.754 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:10.754 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:10.754 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:10.754 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:10.754 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:10.754 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:10.755 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:10.755 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:10.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f88fd59a640 4 set_mon_vals no callback set 2024-04-26T00:39:10.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f890516e640 5 monclient: authenticate success, global_id 204452 2024-04-26T00:39:10.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f890516e640 10 monclient: _renew_subs 2024-04-26T00:39:10.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f890516e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:10.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f890516e640 1 -- 172.21.15.37:0/3621198569 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8900135ed0 con 0x7f8900137e40 2024-04-26T00:39:10.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f890516e640 10 monclient: _renew_subs 2024-04-26T00:39:10.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f890516e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:10.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.746+0000 7f890516e640 1 -- 172.21.15.37:0/3621198569 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8900071a10 con 0x7f8900137e40 2024-04-26T00:39:10.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.750+0000 7f890516e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:10.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.750+0000 7f890516e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:10.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.750+0000 7f890516e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:10.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.750+0000 7f890516e640 1 -- 172.21.15.37:0/3621198569 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8900066f40 con 0x7f8900137e40 2024-04-26T00:39:10.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.750+0000 7f88f77fe640 1 -- 172.21.15.37:0/3621198569 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f88f00bd020 con 0x7f8900137e40 2024-04-26T00:39:10.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.750+0000 7f88f77fe640 1 --2- 172.21.15.37:0/3621198569 >> v2:172.21.15.45:6812/2379918623 conn(0x7f88e0035e80 0x7f88e0038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:10.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.750+0000 7f88f77fe640 1 -- 172.21.15.37:0/3621198569 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(757..757 src has 705..757) ==== 3915+0+0 (secure 0 0 0) 0x7f88f00e2b10 con 0x7f8900137e40 2024-04-26T00:39:10.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.750+0000 7f88fed9d640 1 --2- 172.21.15.37:0/3621198569 >> v2:172.21.15.45:6812/2379918623 conn(0x7f88e0035e80 0x7f88e0038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:10.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.750+0000 7f88fed9d640 10 monclient: get_auth_request con 0x7f88e0035e80 auth_method 0 2024-04-26T00:39:10.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.750+0000 7f88fed9d640 1 --2- 172.21.15.37:0/3621198569 >> v2:172.21.15.45:6812/2379918623 conn(0x7f88e0035e80 0x7f88e0038340 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f88e8006fd0 tx=0x7f88e8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:10.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.750+0000 7f88f77fe640 1 -- 172.21.15.37:0/3621198569 DispatchQueue::entry inject delay of 0.002000 2024-04-26T00:39:10.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.754+0000 7f88f77fe640 1 -- 172.21.15.37:0/3621198569 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f88f00c4030 con 0x7f8900137e40 2024-04-26T00:39:10.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.754+0000 7f88f77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:10.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.754+0000 7f88f77fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:39:10.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.974+0000 7f890516e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:39:10.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.974+0000 7f890516e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:39:10.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.974+0000 7f890516e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:10.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.974+0000 7f890516e640 1 -- 172.21.15.37:0/3621198569 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f890006b9a0 con 0x7f8900137e40 2024-04-26T00:39:10.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.978+0000 7f88f77fe640 1 -- 172.21.15.37:0/3621198569 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f89000a1130 con 0x7f8900137e40 2024-04-26T00:39:10.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.978+0000 7f88f77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:39:10.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.978+0000 7f88f77fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:39:10.977 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":144,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:39:10.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.978+0000 7f890516e640 1 -- 172.21.15.37:0/3621198569 >> v2:172.21.15.45:6812/2379918623 conn(0x7f88e0035e80 msgr2=0x7f88e0038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:10.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.978+0000 7f890516e640 1 --2- 172.21.15.37:0/3621198569 >> v2:172.21.15.45:6812/2379918623 conn(0x7f88e0035e80 0x7f88e0038340 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f88e8006fd0 tx=0x7f88e8008040 comp rx=0 tx=0).stop 2024-04-26T00:39:10.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.978+0000 7f890516e640 10 monclient: shutdown 2024-04-26T00:39:10.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.978+0000 7f890516e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:10.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.978+0000 7f890516e640 1 -- 172.21.15.37:0/3621198569 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8900137e40 msgr2=0x7f8900134910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:10.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.978+0000 7f890516e640 1 --2- 172.21.15.37:0/3621198569 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8900137e40 0x7f8900134910 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f88f001a7b0 tx=0x7f88f0018ea0 comp rx=0 tx=0).stop 2024-04-26T00:39:10.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.978+0000 7f890516e640 1 -- 172.21.15.37:0/3621198569 shutdown_connections 2024-04-26T00:39:10.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.978+0000 7f890516e640 1 --2- 172.21.15.37:0/3621198569 >> v2:172.21.15.45:6812/2379918623 conn(0x7f88e0035e80 0x7f88e0038340 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:10.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.978+0000 7f890516e640 1 --2- 172.21.15.37:0/3621198569 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f890013a770 0x7f890011b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:10.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.978+0000 7f890516e640 1 --2- 172.21.15.37:0/3621198569 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8900137e40 0x7f8900134910 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:10.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.978+0000 7f890516e640 1 --2- 172.21.15.37:0/3621198569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89000a0260 0x7f8900070dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:10.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.978+0000 7f890516e640 1 -- 172.21.15.37:0/3621198569 >> 172.21.15.37:0/3621198569 conn(0x7f89000a1ad0 msgr2=0x7f89000a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:10.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.978+0000 7f890516e640 1 -- 172.21.15.37:0/3621198569 shutdown_connections 2024-04-26T00:39:10.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:10.978+0000 7f890516e640 1 -- 172.21.15.37:0/3621198569 wait complete. 2024-04-26T00:39:13.998 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:39:14.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:14.130+0000 7f029664c640 10 monclient: get_monmap_and_config 2024-04-26T00:39:14.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:14.130+0000 7f029664c640 10 monclient: build_initial_monmap 2024-04-26T00:39:14.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:14.130+0000 7f029664c640 10 monclient: monmap: 2024-04-26T00:39:14.130 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:14.131 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:14.131 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:39:14.133855+0000 2024-04-26T00:39:14.131 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:39:14.133855+0000 2024-04-26T00:39:14.131 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:14.131 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:14.131 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:39:14.131 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:39:14.131 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:39:14.131 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:39:14.131 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:39:14.131 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:14.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:14.130+0000 7f029664c640 1 Processor -- start 2024-04-26T00:39:14.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:14.130+0000 7f029664c640 1 -- start start 2024-04-26T00:39:14.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:14.130+0000 7f029664c640 10 monclient: init 2024-04-26T00:39:14.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:14.130+0000 7f029664c640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:14.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:14.130+0000 7f029664c640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-04-26T00:39:14.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:14.130+0000 7f029664c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f02900a0260 0x7f02900a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:14.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:14.130+0000 7f029664c640 10 monclient(hunting): picked mon.noname-a con 0x7f02900a0260 addr v2:172.21.15.37:3300/0 2024-04-26T00:39:14.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:14.130+0000 7f029664c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0290134b30 addr v1:172.21.15.37:6789/0 2024-04-26T00:39:14.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:14.130+0000 7f029664c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0290135660 addr v1:172.21.15.45:6789/0 2024-04-26T00:39:14.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:14.130+0000 7f029664c640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02900a1130 con 0x7f0290134b30 2024-04-26T00:39:14.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:14.130+0000 7f029664c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f029006cff0 con 0x7f0290135660 2024-04-26T00:39:14.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:14.130+0000 7f029664c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:14.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:14.130+0000 7f029664c640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f029009ec10 con 0x7f02900a0260 2024-04-26T00:39:14.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:14.130+0000 7f029664c640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:14.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:14.130+0000 7f029664c640 10 monclient(hunting): authenticate will time out at 2135.439548s 2024-04-26T00:39:14.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:14.130+0000 7f028f7fe640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f0290134b30 legacy=0x7f0290134f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:39:14.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:14.130+0000 7f028ffff640 1 -- >> v2:172.21.15.37:3300/0 conn(0x7f02900a0260 msgr2=0x7f02900a0640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:39:14.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:14.130+0000 7f028ffff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f02900a0260 0x7f02900a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:39:14.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:14.130+0000 7f0294be9640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f0290135660 0x7f029013ce70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.37:55852/0 (socket says 172.21.15.37:55852) 2024-04-26T00:39:14.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:14.130+0000 7f0294be9640 1 -- 172.21.15.37:0/1813287437 learned_addr learned my addr 172.21.15.37:0/1813287437 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:39:14.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:14.134+0000 7f0294be9640 1 -- 172.21.15.37:0/1813287437 >> v1:172.21.15.45:6789/0 conn(0x7f0290135660 legacy=0x7f029013ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:14.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:14.334+0000 7f028f7fe640 1 -- 172.21.15.37:0/1813287437 >> v1:172.21.15.37:6789/0 conn(0x7f0290134b30 legacy=0x7f0290134f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:39:14.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:14.334+0000 7f028ffff640 1 -- 172.21.15.37:0/1813287437 >> v2:172.21.15.37:3300/0 conn(0x7f02900a0260 msgr2=0x7f02900a0640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:39:14.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:14.334+0000 7f028ffff640 1 --2- 172.21.15.37:0/1813287437 >> v2:172.21.15.37:3300/0 conn(0x7f02900a0260 0x7f02900a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:39:14.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:14.738+0000 7f028f7fe640 1 -- 172.21.15.37:0/1813287437 >> v1:172.21.15.37:6789/0 conn(0x7f0290134b30 legacy=0x7f0290134f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:39:14.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:14.738+0000 7f028ffff640 1 -- 172.21.15.37:0/1813287437 >> v2:172.21.15.37:3300/0 conn(0x7f02900a0260 msgr2=0x7f02900a0640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:39:14.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:14.738+0000 7f028ffff640 1 --2- 172.21.15.37:0/1813287437 >> v2:172.21.15.37:3300/0 conn(0x7f02900a0260 0x7f02900a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:39:15.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:15.542+0000 7f028ffff640 1 -- 172.21.15.37:0/1813287437 >> v2:172.21.15.37:3300/0 conn(0x7f02900a0260 msgr2=0x7f02900a0640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:39:15.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:15.542+0000 7f028f7fe640 1 -- 172.21.15.37:0/1813287437 >> v1:172.21.15.37:6789/0 conn(0x7f0290134b30 legacy=0x7f0290134f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:39:15.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:15.542+0000 7f028ffff640 1 --2- 172.21.15.37:0/1813287437 >> v2:172.21.15.37:3300/0 conn(0x7f02900a0260 0x7f02900a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:39:17.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028dffb640 10 monclient(hunting): tick 2024-04-26T00:39:17.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028dffb640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:39:17.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028dffb640 1 monclient(hunting): continuing hunt 2024-04-26T00:39:17.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028dffb640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:39:17.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028dffb640 1 -- 172.21.15.37:0/1813287437 >> v2:172.21.15.37:3300/0 conn(0x7f02900a0260 msgr2=0x7f02900a0640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:39:17.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028dffb640 1 --2- 172.21.15.37:0/1813287437 >> v2:172.21.15.37:3300/0 conn(0x7f02900a0260 0x7f02900a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:17.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028dffb640 1 -- 172.21.15.37:0/1813287437 >> v1:172.21.15.45:6789/0 conn(0x7f0290135660 legacy=0x7f029013ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:17.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028dffb640 1 -- 172.21.15.37:0/1813287437 >> v1:172.21.15.37:6789/0 conn(0x7f0290134b30 legacy=0x7f0290134f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:39:17.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028dffb640 10 monclient: _add_conns ranks=[2,4] 2024-04-26T00:39:17.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028dffb640 1 --2- 172.21.15.37:0/1813287437 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0270003750 0x7f0270005ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:17.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028dffb640 10 monclient(hunting): picked mon.noname-b con 0x7f0270003750 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:17.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028dffb640 1 --2- 172.21.15.37:0/1813287437 >> v2:172.21.15.45:3300/0 conn(0x7f02700062d0 0x7f0270008770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:17.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028dffb640 10 monclient(hunting): picked mon.noname-c con 0x7f02700062d0 addr v2:172.21.15.45:3300/0 2024-04-26T00:39:17.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028dffb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:17.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028dffb640 1 -- 172.21.15.37:0/1813287437 --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f0270008e00 con 0x7f02700062d0 2024-04-26T00:39:17.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028dffb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:17.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028dffb640 1 -- 172.21.15.37:0/1813287437 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0270008fb0 con 0x7f0270003750 2024-04-26T00:39:17.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028dffb640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:17.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028ffff640 1 --2- 172.21.15.37:0/1813287437 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0270003750 0x7f0270005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:17.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028ffff640 10 monclient(hunting): get_auth_request con 0x7f0270003750 auth_method 0 2024-04-26T00:39:17.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:17.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:17.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:17.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028f7fe640 1 --2- 172.21.15.37:0/1813287437 >> v2:172.21.15.45:3300/0 conn(0x7f02700062d0 0x7f0270008770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:17.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028f7fe640 10 monclient(hunting): get_auth_request con 0x7f02700062d0 auth_method 0 2024-04-26T00:39:17.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:17.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:17.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:17.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:17.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:17.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:17.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:17.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:17.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:17.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028ffff640 10 monclient(hunting): handle_auth_done global_id 204455 payload 274 2024-04-26T00:39:17.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028ffff640 1 -- 172.21.15.37:0/1813287437 >> v2:172.21.15.45:3300/0 conn(0x7f02700062d0 msgr2=0x7f0270008770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:17.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028ffff640 1 --2- 172.21.15.37:0/1813287437 >> v2:172.21.15.45:3300/0 conn(0x7f02700062d0 0x7f0270008770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:17.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028ffff640 10 monclient: _finish_hunting 0 2024-04-26T00:39:17.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028ffff640 1 monclient: found mon.noname-b 2024-04-26T00:39:17.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:17.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:39:17.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.134+0000 7f028ffff640 1 -- 172.21.15.37:0/1813287437 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f029009bc50 con 0x7f0270003750 2024-04-26T00:39:17.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.142+0000 7f028ffff640 1 -- 172.21.15.37:0/1813287437 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0270003750 msgr2=0x7f0270005ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:17.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.142+0000 7f028ffff640 1 --2- 172.21.15.37:0/1813287437 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0270003750 0x7f0270005ba0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f02800038c0 tx=0x7f028001cb40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d06637fe8c404e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:17.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.142+0000 7f028effd640 1 -- 172.21.15.37:0/1813287437 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0280003c80 con 0x7f0270003750 2024-04-26T00:39:17.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.142+0000 7f028effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:17.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.142+0000 7f028effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:39:17.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.142+0000 7f028effd640 10 monclient: dump: 2024-04-26T00:39:17.143 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:17.143 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:17.143 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:17.143 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:17.143 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:17.143 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:17.143 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:17.143 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:17.144 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:17.144 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:17.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.142+0000 7f028effd640 10 monclient: _finish_auth 0 2024-04-26T00:39:17.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.142+0000 7f028effd640 10 monclient: _check_auth_tickets 2024-04-26T00:39:17.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.142+0000 7f028effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:17.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.142+0000 7f028effd640 1 -- 172.21.15.37:0/1813287437 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02800a9030 con 0x7f0270003750 2024-04-26T00:39:17.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.142+0000 7f028effd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:17.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.142+0000 7f028effd640 1 -- 172.21.15.37:0/1813287437 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02800a5a50 con 0x7f0270003750 2024-04-26T00:39:17.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.142+0000 7f029664c640 5 monclient: authenticate success, global_id 204455 2024-04-26T00:39:17.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.142+0000 7f029664c640 10 monclient: get_monmap_and_config success 2024-04-26T00:39:17.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.142+0000 7f029664c640 4 set_mon_vals no callback set 2024-04-26T00:39:17.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.142+0000 7f028effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:17.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.142+0000 7f028effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:17.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.142+0000 7f028effd640 10 monclient: dump: 2024-04-26T00:39:17.145 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:17.145 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:17.145 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:17.145 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:17.145 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:17.145 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:17.145 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:17.145 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:17.145 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:17.145 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:17.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.142+0000 7f029664c640 10 monclient: shutdown 2024-04-26T00:39:17.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.142+0000 7f029664c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:17.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.142+0000 7f029664c640 1 -- 172.21.15.37:0/1813287437 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0270003750 msgr2=0x7f0270005ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:17.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.142+0000 7f029664c640 1 --2- 172.21.15.37:0/1813287437 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0270003750 0x7f0270005ba0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f02800038c0 tx=0x7f028001cb40 comp rx=0 tx=0).stop 2024-04-26T00:39:17.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.142+0000 7f0294be9640 1 -- 172.21.15.37:0/1813287437 reap_dead start 2024-04-26T00:39:17.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.142+0000 7f029664c640 1 -- 172.21.15.37:0/1813287437 shutdown_connections 2024-04-26T00:39:17.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.142+0000 7f029664c640 1 -- 172.21.15.37:0/1813287437 >> 172.21.15.37:0/1813287437 conn(0x7f02900a1ad0 msgr2=0x7f02900a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:17.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.142+0000 7f029664c640 1 -- 172.21.15.37:0/1813287437 shutdown_connections 2024-04-26T00:39:17.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.142+0000 7f029664c640 1 -- 172.21.15.37:0/1813287437 wait complete. 2024-04-26T00:39:17.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f029664c640 10 monclient: build_initial_monmap 2024-04-26T00:39:17.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f029664c640 10 monclient: monmap: 2024-04-26T00:39:17.149 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:17.149 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:39:17.149 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:39:17.149 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:39:17.149 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:17.149 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:39:17.149 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:39:17.149 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:39:17.150 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:39:17.150 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:17.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f029664c640 1 Processor -- start 2024-04-26T00:39:17.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f029664c640 1 -- start start 2024-04-26T00:39:17.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f029664c640 10 monclient: init 2024-04-26T00:39:17.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f029664c640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:17.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f029664c640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:39:17.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f029664c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f029013d310 0x7f0290138d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:17.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f029664c640 10 monclient(hunting): picked mon.noname-a con 0x7f029013d310 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:39:17.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f029664c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0290139280 0x7f0290139680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:17.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f029664c640 10 monclient(hunting): picked mon.noname-c con 0x7f0290139280 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:17.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f029664c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02901cef20 0x7f02901cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:17.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f029664c640 10 monclient(hunting): picked mon.noname-b con 0x7f02901cef20 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:39:17.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f029664c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:17.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f029664c640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f029009f0d0 con 0x7f029013d310 2024-04-26T00:39:17.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f029664c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:17.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f029664c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f029009f4c0 con 0x7f02901cef20 2024-04-26T00:39:17.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f029664c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:17.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f029664c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f029009c4d0 con 0x7f0290139280 2024-04-26T00:39:17.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f029664c640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:17.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f029664c640 10 monclient(hunting): authenticate will time out at 2138.454057s 2024-04-26T00:39:17.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f028ffff640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f029013d310 msgr2=0x7f0290138d40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:39:17.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f028ffff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f029013d310 0x7f0290138d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:39:17.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f028f7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0290139280 0x7f0290139680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:17.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f028f7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0290139280 0x7f0290139680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:44984/0 (socket says 172.21.15.37:44984) 2024-04-26T00:39:17.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f028f7fe640 1 -- 172.21.15.37:0/1717503839 learned_addr learned my addr 172.21.15.37:0/1717503839 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:17.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f028f7fe640 10 monclient(hunting): get_auth_request con 0x7f0290139280 auth_method 0 2024-04-26T00:39:17.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f028f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:17.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f028f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:17.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f028f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:17.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f028f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:17.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f028f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:17.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f028f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:17.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f0294be9640 1 --2- 172.21.15.37:0/1717503839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02901cef20 0x7f02901cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:17.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f0294be9640 10 monclient(hunting): get_auth_request con 0x7f02901cef20 auth_method 0 2024-04-26T00:39:17.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f0294be9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:17.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f0294be9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:17.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f0294be9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:17.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f028f7fe640 10 monclient(hunting): handle_auth_done global_id 204458 payload 824 2024-04-26T00:39:17.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f028f7fe640 1 -- 172.21.15.37:0/1717503839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02901cef20 msgr2=0x7f02901cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:17.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f028f7fe640 1 --2- 172.21.15.37:0/1717503839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02901cef20 0x7f02901cf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:17.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f028f7fe640 1 -- 172.21.15.37:0/1717503839 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f029013d310 msgr2=0x7f0290138d40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:39:17.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f028f7fe640 1 --2- 172.21.15.37:0/1717503839 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f029013d310 0x7f0290138d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:17.152 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f028f7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:39:17.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f028f7fe640 1 monclient: found mon.noname-c 2024-04-26T00:39:17.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f028f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:17.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f028f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:39:17.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f028f7fe640 1 -- 172.21.15.37:0/1717503839 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f029013b400 con 0x7f0290139280 2024-04-26T00:39:17.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.146+0000 7f0294be9640 1 --2- 172.21.15.37:0/1717503839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02901cef20 0x7f02901cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:39:17.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.150+0000 7f028f7fe640 1 -- 172.21.15.37:0/1717503839 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0290139280 msgr2=0x7f0290139680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:17.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.150+0000 7f028f7fe640 1 --2- 172.21.15.37:0/1717503839 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0290139280 0x7f0290139680 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f027800be00 tx=0x7f0278007660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:17.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.150+0000 7f028cff9640 1 -- 172.21.15.37:0/1717503839 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f027801a070 con 0x7f0290139280 2024-04-26T00:39:17.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.150+0000 7f028cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:17.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.150+0000 7f028cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:39:17.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.150+0000 7f028cff9640 10 monclient: dump: 2024-04-26T00:39:17.155 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:17.155 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:17.156 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:17.156 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:17.156 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:17.156 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:17.156 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:17.156 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:17.156 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:17.156 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:17.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.150+0000 7f028cff9640 10 monclient: _finish_auth 0 2024-04-26T00:39:17.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.150+0000 7f028cff9640 10 monclient: _check_auth_tickets 2024-04-26T00:39:17.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.150+0000 7f028cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:17.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.150+0000 7f028cff9640 1 -- 172.21.15.37:0/1717503839 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02780054e0 con 0x7f0290139280 2024-04-26T00:39:17.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.150+0000 7f028cff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:17.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.150+0000 7f028cff9640 1 -- 172.21.15.37:0/1717503839 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02780058b0 con 0x7f0290139280 2024-04-26T00:39:17.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.150+0000 7f029664c640 5 monclient: authenticate success, global_id 204458 2024-04-26T00:39:17.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.150+0000 7f029664c640 10 monclient: _renew_subs 2024-04-26T00:39:17.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.150+0000 7f029664c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:17.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.150+0000 7f029664c640 1 -- 172.21.15.37:0/1717503839 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f029013c2f0 con 0x7f0290139280 2024-04-26T00:39:17.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.150+0000 7f028e7fc640 4 set_mon_vals no callback set 2024-04-26T00:39:17.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.150+0000 7f028cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:17.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.150+0000 7f028cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:17.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.150+0000 7f028cff9640 10 monclient: dump: 2024-04-26T00:39:17.157 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:17.157 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:17.157 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:17.157 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:17.157 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:17.157 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:17.158 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:17.158 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:17.158 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:17.158 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:17.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.150+0000 7f029664c640 10 monclient: _renew_subs 2024-04-26T00:39:17.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.150+0000 7f029664c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:17.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.150+0000 7f029664c640 1 -- 172.21.15.37:0/1717503839 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f029013da20 con 0x7f0290139280 2024-04-26T00:39:17.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.150+0000 7f029664c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:17.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.150+0000 7f029664c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:17.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.150+0000 7f029664c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:17.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.150+0000 7f028f7fe640 1 --2- 172.21.15.37:0/1717503839 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0290139280 0x7f0290139680 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f027800be00 tx=0x7f0278007660 comp rx=0 tx=0).queue_received will delay after 1.71409e+09 on 0x7f0278049440 osd_map(757..757 src has 705..757) 2024-04-26T00:39:17.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.150+0000 7f029664c640 1 -- 172.21.15.37:0/1717503839 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0290066f40 con 0x7f0290139280 2024-04-26T00:39:17.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.150+0000 7f028cff9640 1 -- 172.21.15.37:0/1717503839 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f02780143e0 con 0x7f0290139280 2024-04-26T00:39:17.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.150+0000 7f028cff9640 1 --2- 172.21.15.37:0/1717503839 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0270036e70 0x7f0270039330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:17.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.150+0000 7f028ffff640 1 --2- 172.21.15.37:0/1717503839 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0270036e70 0x7f0270039330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:17.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.150+0000 7f028ffff640 10 monclient: get_auth_request con 0x7f0270036e70 auth_method 0 2024-04-26T00:39:17.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.154+0000 7f028ffff640 1 --2- 172.21.15.37:0/1717503839 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0270036e70 0x7f0270039330 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f02700062a0 tx=0x7f02800a7070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:17.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.158+0000 7f028cff9640 1 -- 172.21.15.37:0/1717503839 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(757..757 src has 705..757) ==== 3915+0+0 (secure 0 0 0) 0x7f0278049440 con 0x7f0290139280 2024-04-26T00:39:17.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.978+0000 7f028cff9640 1 -- 172.21.15.37:0/1717503839 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f0278005a50 con 0x7f0290139280 2024-04-26T00:39:17.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.978+0000 7f028cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:17.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:17.978+0000 7f028cff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:39:18.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:18.226+0000 7f029664c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:39:18.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:18.226+0000 7f029664c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:39:18.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:18.226+0000 7f029664c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:18.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:18.226+0000 7f029664c640 1 -- 172.21.15.37:0/1717503839 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0290071660 con 0x7f0290139280 2024-04-26T00:39:20.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:20.146+0000 7f02877fe640 10 monclient: tick 2024-04-26T00:39:20.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:20.146+0000 7f02877fe640 10 monclient: _check_auth_tickets 2024-04-26T00:39:20.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:20.146+0000 7f02877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:20.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:20.146+0000 7f02877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:21.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:21.146+0000 7f02877fe640 10 monclient: tick 2024-04-26T00:39:21.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:21.146+0000 7f02877fe640 10 monclient: _check_auth_tickets 2024-04-26T00:39:21.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:21.146+0000 7f02877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:22.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.146+0000 7f02877fe640 10 monclient: tick 2024-04-26T00:39:22.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.146+0000 7f02877fe640 10 monclient: _check_auth_tickets 2024-04-26T00:39:22.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.146+0000 7f02877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:22.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.178+0000 7f028cff9640 1 -- 172.21.15.37:0/1717503839 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f02780481a0 con 0x7f0290139280 2024-04-26T00:39:22.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.178+0000 7f028cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:39:22.179 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.178+0000 7f028cff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:39:22.179 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":146,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:39:22.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.182+0000 7f029664c640 1 -- 172.21.15.37:0/1717503839 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0270036e70 msgr2=0x7f0270039330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:22.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.182+0000 7f029664c640 1 --2- 172.21.15.37:0/1717503839 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0270036e70 0x7f0270039330 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f02700062a0 tx=0x7f02800a7070 comp rx=0 tx=0).stop 2024-04-26T00:39:22.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.182+0000 7f029664c640 10 monclient: shutdown 2024-04-26T00:39:22.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.182+0000 7f029664c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:22.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.182+0000 7f029664c640 1 -- 172.21.15.37:0/1717503839 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0290139280 msgr2=0x7f0290139680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:22.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.182+0000 7f029664c640 1 --2- 172.21.15.37:0/1717503839 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0290139280 0x7f0290139680 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f027800be00 tx=0x7f0278007660 comp rx=0 tx=0).stop 2024-04-26T00:39:22.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.182+0000 7f029664c640 1 -- 172.21.15.37:0/1717503839 shutdown_connections 2024-04-26T00:39:22.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.182+0000 7f029664c640 1 --2- 172.21.15.37:0/1717503839 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0270036e70 0x7f0270039330 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:22.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.182+0000 7f029664c640 1 --2- 172.21.15.37:0/1717503839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f02901cef20 0x7f02901cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:22.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.182+0000 7f029664c640 1 --2- 172.21.15.37:0/1717503839 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0290139280 0x7f0290139680 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:22.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.182+0000 7f029664c640 1 --2- 172.21.15.37:0/1717503839 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f029013d310 0x7f0290138d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:22.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.182+0000 7f029664c640 1 -- 172.21.15.37:0/1717503839 >> 172.21.15.37:0/1717503839 conn(0x7f02900a1ad0 msgr2=0x7f0290134420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:22.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.182+0000 7f029664c640 1 -- 172.21.15.37:0/1717503839 shutdown_connections 2024-04-26T00:39:22.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.182+0000 7f029664c640 1 -- 172.21.15.37:0/1717503839 wait complete. 2024-04-26T00:39:22.207 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-26T00:39:22.207 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-04-26T00:39:22.207 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:39:22.216 INFO:tasks.rados.rados.0.smithi045.stdout:3357: done (1 left) 2024-04-26T00:39:22.216 INFO:tasks.rados.rados.0.smithi045.stdout:3358: read oid 44 snap -1 2024-04-26T00:39:22.216 INFO:tasks.rados.rados.0.smithi045.stdout:3358: expect (ObjNum 1127 snap 340 seq_num 1127) 2024-04-26T00:39:22.216 INFO:tasks.rados.rados.0.smithi045.stdout:3359: write (excl) oid 1 current snap is 372 2024-04-26T00:39:22.216 INFO:tasks.rados.rados.0.smithi045.stdout:3359: seq_num 1195 ranges {484365=589976,1661982=422383} 2024-04-26T00:39:22.222 INFO:tasks.rados.rados.0.smithi045.stdout:3359: writing smithi04526579-1 from 484365 to 1074341 tid 1 2024-04-26T00:39:22.229 INFO:tasks.rados.rados.0.smithi045.stdout:3359: writing smithi04526579-1 from 1661982 to 2084365 tid 2 2024-04-26T00:39:22.229 INFO:tasks.rados.rados.0.smithi045.stdout:3360: read oid 24 snap -1 2024-04-26T00:39:22.229 INFO:tasks.rados.rados.0.smithi045.stdout:3360: expect (ObjNum 1140 snap 345 seq_num 1140) 2024-04-26T00:39:22.229 INFO:tasks.rados.rados.0.smithi045.stdout:3361: delete oid 46 current snap is 372 2024-04-26T00:39:22.230 INFO:tasks.rados.rados.0.smithi045.stdout:3356: finishing copy_from racing read to smithi04526579-25 2024-04-26T00:39:22.230 INFO:tasks.rados.rados.0.smithi045.stdout:3356: finishing copy_from to smithi04526579-25 2024-04-26T00:39:22.230 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 2386 (ObjNum 1163 snap 359 seq_num 1163) dirty exists 2024-04-26T00:39:22.232 INFO:tasks.rados.rados.0.smithi045.stdout:3358: expect (ObjNum 1127 snap 340 seq_num 1127) 2024-04-26T00:39:22.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f927b90d640 10 monclient: get_monmap_and_config 2024-04-26T00:39:22.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f927b90d640 10 monclient: build_initial_monmap 2024-04-26T00:39:22.341 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f927b90d640 10 monclient: monmap: 2024-04-26T00:39:22.341 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:22.342 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:22.342 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:39:22.344592+0000 2024-04-26T00:39:22.342 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:39:22.344592+0000 2024-04-26T00:39:22.342 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:22.342 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:22.342 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:39:22.342 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:39:22.342 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:39:22.342 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:39:22.342 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:39:22.342 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:22.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f927b90d640 1 Processor -- start 2024-04-26T00:39:22.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f927b90d640 1 -- start start 2024-04-26T00:39:22.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f927b90d640 10 monclient: init 2024-04-26T00:39:22.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f927b90d640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:22.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f927b90d640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-04-26T00:39:22.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f927b90d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f92740a3d90 0x7f927409bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:22.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f927b90d640 10 monclient(hunting): picked mon.noname-b con 0x7f92740a3d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:22.345 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f927b90d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f92740a4170 0x7f92740a45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:22.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f927b90d640 10 monclient(hunting): picked mon.noname-a con 0x7f92740a4170 addr v2:172.21.15.37:3300/0 2024-04-26T00:39:22.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f927b90d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9274133b20 0x7f9274133f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:22.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f927b90d640 10 monclient(hunting): picked mon.noname-c con 0x7f9274133b20 addr v2:172.21.15.45:3300/0 2024-04-26T00:39:22.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f927b90d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:22.346 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f927b90d640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f92740a5150 con 0x7f92740a4170 2024-04-26T00:39:22.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f927b90d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:22.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f927b90d640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f927409b280 con 0x7f9274133b20 2024-04-26T00:39:22.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f927b90d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:22.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f927b90d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f927409b560 con 0x7f92740a3d90 2024-04-26T00:39:22.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f927b90d640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:22.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f927b90d640 10 monclient(hunting): authenticate will time out at 2143.650385s 2024-04-26T00:39:22.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f9278ea8640 1 -- >> v2:172.21.15.37:3300/0 conn(0x7f92740a4170 msgr2=0x7f92740a45b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:39:22.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f9278ea8640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f92740a4170 0x7f92740a45b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:39:22.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f92796a9640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f92740a3d90 0x7f927409bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:22.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f92796a9640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f92740a3d90 0x7f927409bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:35396/0 (socket says 172.21.15.37:35396) 2024-04-26T00:39:22.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f92796a9640 1 -- 172.21.15.37:0/4018370859 learned_addr learned my addr 172.21.15.37:0/4018370859 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:22.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f92796a9640 10 monclient(hunting): get_auth_request con 0x7f92740a3d90 auth_method 0 2024-04-26T00:39:22.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f92796a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:22.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f92796a9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:22.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f92796a9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:22.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f9279eaa640 1 --2- 172.21.15.37:0/4018370859 >> v2:172.21.15.45:3300/0 conn(0x7f9274133b20 0x7f9274133f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:22.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f92796a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:22.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f92796a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:22.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f92796a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:22.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f9279eaa640 10 monclient(hunting): get_auth_request con 0x7f9274133b20 auth_method 0 2024-04-26T00:39:22.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f9279eaa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:22.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f9279eaa640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:22.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f9279eaa640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:22.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f92796a9640 10 monclient(hunting): handle_auth_done global_id 204461 payload 274 2024-04-26T00:39:22.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f92796a9640 1 -- 172.21.15.37:0/4018370859 >> v2:172.21.15.45:3300/0 conn(0x7f9274133b20 msgr2=0x7f9274133f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:22.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f92796a9640 1 --2- 172.21.15.37:0/4018370859 >> v2:172.21.15.45:3300/0 conn(0x7f9274133b20 0x7f9274133f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:22.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f92796a9640 1 -- 172.21.15.37:0/4018370859 >> v2:172.21.15.37:3300/0 conn(0x7f92740a4170 msgr2=0x7f92740a45b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:39:22.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f92796a9640 1 --2- 172.21.15.37:0/4018370859 >> v2:172.21.15.37:3300/0 conn(0x7f92740a4170 0x7f92740a45b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:22.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f92796a9640 10 monclient: _finish_hunting 0 2024-04-26T00:39:22.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f92796a9640 1 monclient: found mon.noname-b 2024-04-26T00:39:22.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f92796a9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:22.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f92796a9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:39:22.349 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.342+0000 7f92796a9640 1 -- 172.21.15.37:0/4018370859 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92740a4af0 con 0x7f92740a3d90 2024-04-26T00:39:22.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.346+0000 7f92796a9640 1 -- 172.21.15.37:0/4018370859 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f92740a3d90 msgr2=0x7f927409bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:22.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.346+0000 7f92796a9640 1 --2- 172.21.15.37:0/4018370859 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f92740a3d90 0x7f927409bf40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9274076100 tx=0x7f926c01c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c86eee2333f01d33 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:22.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.346+0000 7f9273fff640 1 -- 172.21.15.37:0/4018370859 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f926c0a5660 con 0x7f92740a3d90 2024-04-26T00:39:22.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.346+0000 7f9273fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:22.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.346+0000 7f9273fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:39:22.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.346+0000 7f9273fff640 10 monclient: dump: 2024-04-26T00:39:22.355 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:22.355 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:22.355 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:22.355 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:22.356 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:22.356 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:22.356 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:22.356 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:22.356 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:22.356 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:22.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.346+0000 7f9273fff640 10 monclient: _finish_auth 0 2024-04-26T00:39:22.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.346+0000 7f9273fff640 10 monclient: _check_auth_tickets 2024-04-26T00:39:22.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.346+0000 7f9273fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:22.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.346+0000 7f9273fff640 1 -- 172.21.15.37:0/4018370859 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f926c0a5800 con 0x7f92740a3d90 2024-04-26T00:39:22.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.346+0000 7f927b90d640 5 monclient: authenticate success, global_id 204461 2024-04-26T00:39:22.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.346+0000 7f927b90d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:39:22.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.346+0000 7f9273fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:22.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.346+0000 7f927b90d640 10 monclient: get_monmap_and_config success 2024-04-26T00:39:22.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.346+0000 7f927b90d640 4 set_mon_vals no callback set 2024-04-26T00:39:22.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.346+0000 7f9273fff640 1 -- 172.21.15.37:0/4018370859 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f926c0a5bd0 con 0x7f92740a3d90 2024-04-26T00:39:22.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.346+0000 7f9273fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:22.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.346+0000 7f9273fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:22.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.346+0000 7f9273fff640 10 monclient: dump: 2024-04-26T00:39:22.357 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:22.357 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:22.357 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:22.357 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:22.358 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:22.358 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:22.358 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:22.358 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:22.358 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:22.358 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:22.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.346+0000 7f927b90d640 10 monclient: shutdown 2024-04-26T00:39:22.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.346+0000 7f927b90d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:22.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.346+0000 7f927b90d640 1 -- 172.21.15.37:0/4018370859 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f92740a3d90 msgr2=0x7f927409bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:22.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.346+0000 7f927b90d640 1 --2- 172.21.15.37:0/4018370859 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f92740a3d90 0x7f927409bf40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9274076100 tx=0x7f926c01c8d0 comp rx=0 tx=0).stop 2024-04-26T00:39:22.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.346+0000 7f927b90d640 1 -- 172.21.15.37:0/4018370859 shutdown_connections 2024-04-26T00:39:22.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.346+0000 7f927b90d640 1 --2- 172.21.15.37:0/4018370859 >> v2:172.21.15.45:3300/0 conn(0x7f9274133b20 0x7f9274133f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:22.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.346+0000 7f927b90d640 1 --2- 172.21.15.37:0/4018370859 >> v2:172.21.15.37:3300/0 conn(0x7f92740a4170 0x7f92740a45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:22.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.346+0000 7f927b90d640 1 --2- 172.21.15.37:0/4018370859 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f92740a3d90 0x7f927409bf40 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:22.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.346+0000 7f927b90d640 1 -- 172.21.15.37:0/4018370859 >> 172.21.15.37:0/4018370859 conn(0x7f92740aa0a0 msgr2=0x7f92740a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:22.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.346+0000 7f927b90d640 1 -- 172.21.15.37:0/4018370859 shutdown_connections 2024-04-26T00:39:22.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f927b90d640 1 -- 172.21.15.37:0/4018370859 wait complete. 2024-04-26T00:39:22.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f927b90d640 10 monclient: build_initial_monmap 2024-04-26T00:39:22.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f927b90d640 10 monclient: monmap: 2024-04-26T00:39:22.359 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:22.359 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:39:22.359 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:39:22.359 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:39:22.359 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:22.359 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:39:22.360 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:39:22.360 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:39:22.360 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:39:22.360 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:22.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f927b90d640 1 Processor -- start 2024-04-26T00:39:22.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f927b90d640 1 -- start start 2024-04-26T00:39:22.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f927b90d640 10 monclient: init 2024-04-26T00:39:22.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f927b90d640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:22.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f927b90d640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:39:22.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f927b90d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f92740a3d90 0x7f927413d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:22.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f927b90d640 10 monclient(hunting): picked mon.noname-a con 0x7f92740a3d90 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:39:22.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f927b90d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92740a4170 0x7f927413d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:22.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f927b90d640 10 monclient(hunting): picked mon.noname-b con 0x7f92740a4170 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:39:22.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f927b90d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9274133b20 0x7f92741cf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:22.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f927b90d640 10 monclient(hunting): picked mon.noname-c con 0x7f9274133b20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:22.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f927b90d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:22.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f927b90d640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9274140c10 con 0x7f92740a3d90 2024-04-26T00:39:22.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f927b90d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:22.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f927b90d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f927409c690 con 0x7f92740a4170 2024-04-26T00:39:22.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f927b90d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:22.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f927b90d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9274140d90 con 0x7f9274133b20 2024-04-26T00:39:22.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f927b90d640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:22.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f927b90d640 10 monclient(hunting): authenticate will time out at 2143.657581s 2024-04-26T00:39:22.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f92796a9640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f92740a3d90 msgr2=0x7f927413d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:39:22.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f92796a9640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f92740a3d90 0x7f927413d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:39:22.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f9279eaa640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9274133b20 0x7f92741cf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:22.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f9279eaa640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9274133b20 0x7f92741cf0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:35404/0 (socket says 172.21.15.37:35404) 2024-04-26T00:39:22.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f9279eaa640 1 -- 172.21.15.37:0/2748237058 learned_addr learned my addr 172.21.15.37:0/2748237058 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:22.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f9279eaa640 10 monclient(hunting): get_auth_request con 0x7f9274133b20 auth_method 0 2024-04-26T00:39:22.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f9279eaa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:22.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f9279eaa640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:22.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f9279eaa640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:22.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f9279eaa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:22.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f9279eaa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:22.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f9279eaa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:22.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f9278ea8640 1 --2- 172.21.15.37:0/2748237058 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92740a4170 0x7f927413d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:22.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f9278ea8640 10 monclient(hunting): get_auth_request con 0x7f92740a4170 auth_method 0 2024-04-26T00:39:22.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f9278ea8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:22.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f9278ea8640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:22.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f9278ea8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:22.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f9279eaa640 10 monclient(hunting): handle_auth_done global_id 204464 payload 824 2024-04-26T00:39:22.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f9279eaa640 1 -- 172.21.15.37:0/2748237058 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92740a4170 msgr2=0x7f927413d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:22.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f9279eaa640 1 --2- 172.21.15.37:0/2748237058 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92740a4170 0x7f927413d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:22.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f9279eaa640 1 -- 172.21.15.37:0/2748237058 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f92740a3d90 msgr2=0x7f927413d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:39:22.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f9279eaa640 1 --2- 172.21.15.37:0/2748237058 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f92740a3d90 0x7f927413d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:22.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f9279eaa640 10 monclient: _finish_hunting 0 2024-04-26T00:39:22.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f9279eaa640 1 monclient: found mon.noname-c 2024-04-26T00:39:22.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f9279eaa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:22.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f9279eaa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:39:22.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.350+0000 7f9279eaa640 1 -- 172.21.15.37:0/2748237058 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f927413aa00 con 0x7f9274133b20 2024-04-26T00:39:22.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f9279eaa640 1 -- 172.21.15.37:0/2748237058 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9274133b20 msgr2=0x7f92741cf0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:22.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f9279eaa640 1 --2- 172.21.15.37:0/2748237058 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9274133b20 0x7f92741cf0c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f926400d830 tx=0x7f926400dcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:22.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f9279eaa640 1 --2- 172.21.15.37:0/2748237058 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9274133b20 0x7f92741cf0c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f926400d830 tx=0x7f926400dcf0 comp rx=0 tx=0).queue_received will delay after 1.71409e+09 on 0x7f926400b8d0 mon_map magic: 0 2024-04-26T00:39:22.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f9271ffb640 1 -- 172.21.15.37:0/2748237058 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f926400b8d0 con 0x7f9274133b20 2024-04-26T00:39:22.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f9271ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:22.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f9271ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:39:22.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f9271ffb640 10 monclient: dump: 2024-04-26T00:39:22.364 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:22.364 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:22.364 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:22.364 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:22.364 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:22.364 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:22.364 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:22.364 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:22.364 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:22.364 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:22.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f9271ffb640 10 monclient: _finish_auth 0 2024-04-26T00:39:22.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f9271ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:39:22.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f9271ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:22.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f9271ffb640 1 -- 172.21.15.37:0/2748237058 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9264019070 con 0x7f9274133b20 2024-04-26T00:39:22.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f9271ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:22.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f927b90d640 5 monclient: authenticate success, global_id 204464 2024-04-26T00:39:22.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f927b90d640 10 monclient: _renew_subs 2024-04-26T00:39:22.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f927b90d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:22.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f927b90d640 1 -- 172.21.15.37:0/2748237058 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f927413ace0 con 0x7f9274133b20 2024-04-26T00:39:22.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f9272ffd640 4 set_mon_vals no callback set 2024-04-26T00:39:22.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f927b90d640 10 monclient: _renew_subs 2024-04-26T00:39:22.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f927b90d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:22.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f927b90d640 1 -- 172.21.15.37:0/2748237058 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f92740a0ab0 con 0x7f9274133b20 2024-04-26T00:39:22.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f927b90d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:22.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f927b90d640 1 --2- 172.21.15.37:0/2748237058 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9274138380 0x7f92741cf600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:22.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f927b90d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:22.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f927b90d640 1 -- 172.21.15.37:0/2748237058 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9274138760 con 0x7f9274138380 2024-04-26T00:39:22.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f92796a9640 1 --2- 172.21.15.37:0/2748237058 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9274138380 0x7f92741cf600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:22.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f92796a9640 10 monclient: get_auth_request con 0x7f9274138380 auth_method 0 2024-04-26T00:39:22.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f92796a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:22.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f92796a9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:22.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f92796a9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:22.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f92796a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:22.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f92796a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:22.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f92796a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:22.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f9271ffb640 1 -- 172.21.15.37:0/2748237058 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f926400bcb0 con 0x7f9274133b20 2024-04-26T00:39:22.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f9271ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:22.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f9271ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:22.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f9271ffb640 10 monclient: dump: 2024-04-26T00:39:22.366 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:22.366 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:22.366 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:22.366 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:22.366 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:22.367 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:22.367 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:22.367 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:22.367 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:22.367 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:22.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f92796a9640 10 monclient(hunting): handle_auth_done global_id 204467 payload 824 2024-04-26T00:39:22.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f9271ffb640 1 -- 172.21.15.37:0/2748237058 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f926401e460 con 0x7f9274133b20 2024-04-26T00:39:22.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f92796a9640 1 -- 172.21.15.37:0/2748237058 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f927404f510 con 0x7f9274138380 2024-04-26T00:39:22.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f9271ffb640 1 --2- 172.21.15.37:0/2748237058 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9260035e30 0x7f92600382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:22.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f9278ea8640 1 --2- 172.21.15.37:0/2748237058 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9260035e30 0x7f92600382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:22.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.354+0000 7f9278ea8640 10 monclient: get_auth_request con 0x7f9260035e30 auth_method 0 2024-04-26T00:39:22.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.358+0000 7f92796a9640 1 -- 172.21.15.37:0/2748237058 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9274138380 msgr2=0x7f92741cf600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:22.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.358+0000 7f92796a9640 1 --2- 172.21.15.37:0/2748237058 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9274138380 0x7f92741cf600 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f92740a6d60 tx=0x7f926c019670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:22.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.358+0000 7f9271ffb640 1 -- 172.21.15.37:0/2748237058 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f926c0bc020 con 0x7f9274138380 2024-04-26T00:39:22.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.358+0000 7f9271ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:22.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.358+0000 7f9271ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:22.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.358+0000 7f9271ffb640 10 monclient: dump: 2024-04-26T00:39:22.370 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:22.370 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:22.370 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:22.370 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:22.371 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:22.371 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:22.371 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:22.371 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:22.371 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:22.371 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:22.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.358+0000 7f9271ffb640 1 -- 172.21.15.37:0/2748237058 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f926c0b7450 con 0x7f9274138380 2024-04-26T00:39:22.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.358+0000 7f9271ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:22.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.358+0000 7f9271ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:39:22.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.358+0000 7f9271ffb640 1 -- 172.21.15.37:0/2748237058 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9274138380 msgr2=0x7f92741cf600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:22.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.358+0000 7f9271ffb640 1 --2- 172.21.15.37:0/2748237058 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9274138380 0x7f92741cf600 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f92740a6d60 tx=0x7f926c019670 comp rx=0 tx=0).stop 2024-04-26T00:39:22.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.358+0000 7f9271ffb640 1 -- 172.21.15.37:0/2748237058 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9274138380 msgr2=0x7f92741cf600 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:39:22.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.358+0000 7f9271ffb640 1 --2- 172.21.15.37:0/2748237058 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9274138380 0x7f92741cf600 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:22.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.358+0000 7f9278ea8640 1 --2- 172.21.15.37:0/2748237058 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9260035e30 0x7f92600382f0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f9268006fd0 tx=0x7f9268008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:22.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.362+0000 7f927b90d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:39:22.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.362+0000 7f927b90d640 1 --2- 172.21.15.37:0/2748237058 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f92741cfb40 0x7f9274131d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:22.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.362+0000 7f927b90d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:22.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.362+0000 7f927b90d640 1 -- 172.21.15.37:0/2748237058 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f92741cff20 con 0x7f92741cfb40 2024-04-26T00:39:22.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.362+0000 7f92796a9640 1 --2- 172.21.15.37:0/2748237058 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f92741cfb40 0x7f9274131d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:22.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.362+0000 7f92796a9640 10 monclient: get_auth_request con 0x7f92741cfb40 auth_method 0 2024-04-26T00:39:22.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.362+0000 7f92796a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:22.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.362+0000 7f92796a9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:22.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.362+0000 7f92796a9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:22.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.362+0000 7f92796a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:22.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.362+0000 7f92796a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:22.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.362+0000 7f92796a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:22.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.362+0000 7f92796a9640 10 monclient(hunting): handle_auth_done global_id 204470 payload 824 2024-04-26T00:39:22.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.362+0000 7f92796a9640 1 -- 172.21.15.37:0/2748237058 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9274075dc0 con 0x7f92741cfb40 2024-04-26T00:39:22.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.366+0000 7f92796a9640 1 -- 172.21.15.37:0/2748237058 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f92741cfb40 msgr2=0x7f9274131d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:22.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.366+0000 7f92796a9640 1 --2- 172.21.15.37:0/2748237058 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f92741cfb40 0x7f9274131d10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f926c01af80 tx=0x7f926c009810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:22.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.366+0000 7f9271ffb640 1 -- 172.21.15.37:0/2748237058 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f926c0bc040 con 0x7f92741cfb40 2024-04-26T00:39:22.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.366+0000 7f9271ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:22.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.366+0000 7f9271ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:22.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.366+0000 7f9271ffb640 10 monclient: dump: 2024-04-26T00:39:22.373 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:22.373 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:22.373 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:22.373 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:22.374 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:22.374 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:22.374 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:22.374 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:22.374 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:22.374 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:22.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.366+0000 7f9271ffb640 1 -- 172.21.15.37:0/2748237058 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4709 (secure 0 0 0) 0x7f926c0b7d90 con 0x7f92741cfb40 2024-04-26T00:39:22.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.366+0000 7f9271ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:39:22.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.366+0000 7f9271ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:39:22.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.366+0000 7f9271ffb640 1 -- 172.21.15.37:0/2748237058 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f92741cfb40 msgr2=0x7f9274131d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:22.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.366+0000 7f9271ffb640 1 --2- 172.21.15.37:0/2748237058 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f92741cfb40 0x7f9274131d10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f926c01af80 tx=0x7f926c009810 comp rx=0 tx=0).stop 2024-04-26T00:39:22.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.366+0000 7f9271ffb640 1 -- 172.21.15.37:0/2748237058 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f92741cfb40 msgr2=0x7f9274131d10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:39:22.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.366+0000 7f9271ffb640 1 --2- 172.21.15.37:0/2748237058 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f92741cfb40 0x7f9274131d10 secure :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f926c01af80 tx=0x7f926c009810 comp rx=0 tx=0).stop 2024-04-26T00:39:22.375 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:39:22.375 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:39:22.375 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:39:22.375 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:39:22.375 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 146, 2024-04-26T00:39:22.375 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 167973, 2024-04-26T00:39:22.375 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:39:22.375 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:39:22.375 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:39:22.375 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:22.375 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:39:22.375 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:39:22.376 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:39:22.376 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:39:22.376 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:22.376 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:22.376 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:22.376 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:22.376 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:22.376 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:22.376 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:22.376 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:22.376 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:22.376 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:22.376 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:22.376 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:22.376 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:39:22.377 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:39:22.377 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:22.377 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:22.377 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:22.377 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:22.377 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:22.377 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:22.377 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:22.377 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:22.377 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:22.378 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:22.378 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:22.378 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:22.378 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:22.378 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:39:22.378 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:39:22.378 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:39:22.378 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:39:22.378 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:39:22.378 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:39:22.378 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:39:22.378 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:39:22.378 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:39:22.378 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:39:22.379 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:39:22.379 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:39:22.379 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:39:22.379 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:39:22.379 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:39:22.379 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:39:22.379 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:39:22.379 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:22.379 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:22.379 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:22.379 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:22.379 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:22.379 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:22.379 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:22.379 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:22.380 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:22.380 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:22.380 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:22.380 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:22.380 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:39:22.380 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:22.380 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:39:22.380 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:22.380 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:39:22.380 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:39:22.380 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:22.380 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:22.380 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:22.380 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:22.381 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:39:22.381 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:22.381 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:22.381 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:22.381 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:22.381 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:39:22.381 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:22.381 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:22.381 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:22.381 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:22.381 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:39:22.381 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:39:22.381 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:22.381 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:22.381 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:22.382 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:22.382 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:22.382 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:39:22.382 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:39:22.382 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:22.382 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:22.382 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:22.382 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:22.382 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:39:22.382 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:22.382 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:22.382 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:22.382 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:22.382 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:39:22.383 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:22.383 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:22.383 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:22.383 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:22.383 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:39:22.383 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:39:22.383 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:22.383 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:22.383 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:22.383 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:22.383 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:22.383 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:39:22.383 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:39:22.383 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:22.383 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:22.384 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:22.384 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:22.384 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:39:22.384 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:22.384 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:22.384 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:22.384 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:22.384 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:39:22.384 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:22.384 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:22.384 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:22.384 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:22.384 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:39:22.384 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:39:22.384 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:22.385 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:22.385 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:22.385 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:22.385 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:22.385 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:22.385 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:39:22.385 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:39:22.385 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:22.385 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:22.385 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:22.385 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:39:22.385 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:22.385 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:22.385 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:39:22.385 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:22.386 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:22.386 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:22.386 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:39:22.386 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:22.386 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:22.386 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:39:22.386 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:22.386 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:22.386 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:22.386 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:39:22.386 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:22.386 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:22.386 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:22.386 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:39:22.387 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:39:22.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.366+0000 7f927b90d640 1 -- 172.21.15.37:0/2748237058 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9260035e30 msgr2=0x7f92600382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:22.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.366+0000 7f927b90d640 1 --2- 172.21.15.37:0/2748237058 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9260035e30 0x7f92600382f0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f9268006fd0 tx=0x7f9268008040 comp rx=0 tx=0).stop 2024-04-26T00:39:22.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.366+0000 7f927b90d640 10 monclient: shutdown 2024-04-26T00:39:22.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.366+0000 7f927b90d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:22.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.366+0000 7f927b90d640 1 -- 172.21.15.37:0/2748237058 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9274133b20 msgr2=0x7f92741cf0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:22.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.366+0000 7f927b90d640 1 --2- 172.21.15.37:0/2748237058 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9274133b20 0x7f92741cf0c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f926400d830 tx=0x7f926400dcf0 comp rx=0 tx=0).stop 2024-04-26T00:39:22.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.366+0000 7f9279eaa640 1 -- 172.21.15.37:0/2748237058 reap_dead start 2024-04-26T00:39:22.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.366+0000 7f9279eaa640 1 -- 172.21.15.37:0/2748237058 reap_dead start 2024-04-26T00:39:22.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.366+0000 7f9271ffb640 1 -- 172.21.15.37:0/2748237058 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(758..758 src has 705..758) ==== 3943+0+0 (unknown 0 0 0) 0x7f9264048ff0 con 0x7f9274133b20 2024-04-26T00:39:22.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.366+0000 7f927b90d640 1 -- 172.21.15.37:0/2748237058 shutdown_connections 2024-04-26T00:39:22.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.366+0000 7f927b90d640 1 -- 172.21.15.37:0/2748237058 >> 172.21.15.37:0/2748237058 conn(0x7f92740aa0a0 msgr2=0x7f92740a2400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:22.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.366+0000 7f927b90d640 1 -- 172.21.15.37:0/2748237058 shutdown_connections 2024-04-26T00:39:22.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.366+0000 7f927b90d640 1 -- 172.21.15.37:0/2748237058 wait complete. 2024-04-26T00:39:22.389 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:39:22.389 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-04-26T00:39:22.389 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:39:22.424 INFO:tasks.rados.rados.0.smithi045.stdout:3359: finishing write tid 1 to smithi04526579-1 2024-04-26T00:39:22.424 INFO:tasks.rados.rados.0.smithi045.stdout:3359: finishing write tid 2 to smithi04526579-1 2024-04-26T00:39:22.424 INFO:tasks.rados.rados.0.smithi045.stdout:3356: done (4 left) 2024-04-26T00:39:22.424 INFO:tasks.rados.rados.0.smithi045.stdout:3358: done (3 left) 2024-04-26T00:39:22.424 INFO:tasks.rados.rados.0.smithi045.stdout:3361: done (2 left) 2024-04-26T00:39:22.424 INFO:tasks.rados.rados.0.smithi045.stdout:3362: write oid 36 current snap is 372 2024-04-26T00:39:22.424 INFO:tasks.rados.rados.0.smithi045.stdout:3359: finishing write tid 3 to smithi04526579-1 2024-04-26T00:39:22.425 INFO:tasks.rados.rados.0.smithi045.stdout:3359: finishing write tid 4 to smithi04526579-1 2024-04-26T00:39:22.425 INFO:tasks.rados.rados.0.smithi045.stdout:3359: oid 1 updating version 0 to 2463 2024-04-26T00:39:22.425 INFO:tasks.rados.rados.0.smithi045.stdout:3359: oid 1 updating version 2463 to 2464 2024-04-26T00:39:22.425 INFO:tasks.rados.rados.0.smithi045.stdout:3359: oid 1 updating version 2464 to 2465 2024-04-26T00:39:22.425 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 2465 (ObjNum 1195 snap 372 seq_num 1195) dirty exists 2024-04-26T00:39:22.425 INFO:tasks.rados.rados.0.smithi045.stdout:3359: left oid 1 (ObjNum 1195 snap 372 seq_num 1195) 2024-04-26T00:39:22.425 INFO:tasks.rados.rados.0.smithi045.stdout:3360: expect (ObjNum 1140 snap 345 seq_num 1140) 2024-04-26T00:39:22.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f2414bf9640 10 monclient: get_monmap_and_config 2024-04-26T00:39:22.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f2414bf9640 10 monclient: build_initial_monmap 2024-04-26T00:39:22.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f2414bf9640 10 monclient: monmap: 2024-04-26T00:39:22.505 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:22.505 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:22.505 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:39:22.508190+0000 2024-04-26T00:39:22.505 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:39:22.508190+0000 2024-04-26T00:39:22.505 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:22.506 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:22.506 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:39:22.506 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:39:22.506 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:39:22.506 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:39:22.506 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:39:22.506 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:22.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f2414bf9640 1 Processor -- start 2024-04-26T00:39:22.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f2414bf9640 1 -- start start 2024-04-26T00:39:22.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f2414bf9640 10 monclient: init 2024-04-26T00:39:22.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f2414bf9640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:22.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f2414bf9640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-04-26T00:39:22.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f2414bf9640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f24100a0300 0x7f241009fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:22.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f2414bf9640 10 monclient(hunting): picked mon.noname-c con 0x7f24100a0300 addr v2:172.21.15.45:3300/0 2024-04-26T00:39:22.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f2414bf9640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f24100a06e0 0x7f2410134ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:22.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f2414bf9640 10 monclient(hunting): picked mon.noname-a con 0x7f24100a06e0 addr v2:172.21.15.37:3300/0 2024-04-26T00:39:22.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f2414bf9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f24101350e0 addr v1:172.21.15.37:6789/0 2024-04-26T00:39:22.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f2414bf9640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f241009c190 con 0x7f24101350e0 2024-04-26T00:39:22.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f2414bf9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:22.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f2414bf9640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f24100a13f0 con 0x7f24100a06e0 2024-04-26T00:39:22.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f2414bf9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:22.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f2414bf9640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f24100a1110 con 0x7f24100a0300 2024-04-26T00:39:22.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f2414bf9640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:22.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f2414bf9640 10 monclient(hunting): authenticate will time out at 2143.813874s 2024-04-26T00:39:22.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f240ed9d640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f24101350e0 legacy=0x7f241013bed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:39:22.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f240dd9b640 1 -- >> v2:172.21.15.37:3300/0 conn(0x7f24100a06e0 msgr2=0x7f2410134ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:39:22.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f240dd9b640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f24100a06e0 0x7f2410134ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:39:22.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f240e59c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f24100a0300 0x7f241009fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:22.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f240e59c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f24100a0300 0x7f241009fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.37:47328/0 (socket says 172.21.15.37:47328) 2024-04-26T00:39:22.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f240e59c640 1 -- 172.21.15.37:0/3635894810 learned_addr learned my addr 172.21.15.37:0/3635894810 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:22.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f240e59c640 10 monclient(hunting): get_auth_request con 0x7f24100a0300 auth_method 0 2024-04-26T00:39:22.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f240e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:22.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f240e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:22.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f240e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:22.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f240e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:22.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f240e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:22.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f240e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:22.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f240e59c640 10 monclient(hunting): handle_auth_done global_id 224239 payload 274 2024-04-26T00:39:22.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f240e59c640 1 -- 172.21.15.37:0/3635894810 >> v2:172.21.15.37:3300/0 conn(0x7f24100a06e0 msgr2=0x7f2410134ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:39:22.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f240e59c640 1 --2- 172.21.15.37:0/3635894810 >> v2:172.21.15.37:3300/0 conn(0x7f24100a06e0 0x7f2410134ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:22.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f240e59c640 1 -- 172.21.15.37:0/3635894810 >> v1:172.21.15.37:6789/0 conn(0x7f24101350e0 legacy=0x7f241013bed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:39:22.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f240e59c640 10 monclient: _finish_hunting 0 2024-04-26T00:39:22.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f240e59c640 1 monclient: found mon.noname-c 2024-04-26T00:39:22.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f240e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:22.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f240e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:39:22.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.506+0000 7f240e59c640 1 -- 172.21.15.37:0/3635894810 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24100a16d0 con 0x7f24100a0300 2024-04-26T00:39:22.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.510+0000 7f240e59c640 1 -- 172.21.15.37:0/3635894810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24100a0300 msgr2=0x7f241009fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:22.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.510+0000 7f240e59c640 1 --2- 172.21.15.37:0/3635894810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24100a0300 0x7f241009fc50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f24100764d0 tx=0x7f240801c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bda94605f1c61f78 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:22.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240d59a640 1 -- 172.21.15.37:0/3635894810 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24080a5610 con 0x7f24100a0300 2024-04-26T00:39:22.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:22.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:39:22.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240d59a640 10 monclient: dump: 2024-04-26T00:39:22.514 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:22.514 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:22.514 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:22.514 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:22.514 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:22.515 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:22.515 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:22.515 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:22.515 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:22.515 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:22.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240d59a640 10 monclient: _finish_auth 0 2024-04-26T00:39:22.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240d59a640 10 monclient: _check_auth_tickets 2024-04-26T00:39:22.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:22.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f2414bf9640 5 monclient: authenticate success, global_id 224239 2024-04-26T00:39:22.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f2414bf9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:39:22.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240d59a640 1 -- 172.21.15.37:0/3635894810 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24080a57b0 con 0x7f24100a0300 2024-04-26T00:39:22.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240d59a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:22.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f2414bf9640 10 monclient: get_monmap_and_config success 2024-04-26T00:39:22.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f2414bf9640 4 set_mon_vals no callback set 2024-04-26T00:39:22.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240d59a640 1 -- 172.21.15.37:0/3635894810 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24080a5b80 con 0x7f24100a0300 2024-04-26T00:39:22.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:22.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:39:22.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240d59a640 10 monclient: dump: 2024-04-26T00:39:22.516 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:22.516 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:22.516 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:22.516 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:22.516 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:22.516 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:22.516 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:22.516 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:22.516 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:22.516 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:22.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f2414bf9640 10 monclient: shutdown 2024-04-26T00:39:22.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f2414bf9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:22.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f2414bf9640 1 -- 172.21.15.37:0/3635894810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24100a0300 msgr2=0x7f241009fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:22.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f2414bf9640 1 --2- 172.21.15.37:0/3635894810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24100a0300 0x7f241009fc50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f24100764d0 tx=0x7f240801c8c0 comp rx=0 tx=0).stop 2024-04-26T00:39:22.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f2414bf9640 1 -- 172.21.15.37:0/3635894810 shutdown_connections 2024-04-26T00:39:22.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f2414bf9640 1 --2- 172.21.15.37:0/3635894810 >> v2:172.21.15.37:3300/0 conn(0x7f24100a06e0 0x7f2410134ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:22.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f2414bf9640 1 --2- 172.21.15.37:0/3635894810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24100a0300 0x7f241009fc50 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:22.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f2414bf9640 1 -- 172.21.15.37:0/3635894810 >> 172.21.15.37:0/3635894810 conn(0x7f24100a1d50 msgr2=0x7f24100a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:22.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f2414bf9640 1 -- 172.21.15.37:0/3635894810 shutdown_connections 2024-04-26T00:39:22.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f2414bf9640 1 -- 172.21.15.37:0/3635894810 wait complete. 2024-04-26T00:39:22.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f2414bf9640 10 monclient: build_initial_monmap 2024-04-26T00:39:22.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f2414bf9640 10 monclient: monmap: 2024-04-26T00:39:22.517 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:22.517 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:39:22.518 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:39:22.518 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:39:22.518 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:22.518 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:39:22.518 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:39:22.518 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:39:22.518 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:39:22.518 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:22.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f2414bf9640 1 Processor -- start 2024-04-26T00:39:22.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f2414bf9640 1 -- start start 2024-04-26T00:39:22.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f2414bf9640 10 monclient: init 2024-04-26T00:39:22.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f2414bf9640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:22.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f2414bf9640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:39:22.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f2414bf9640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f24100a0300 0x7f241013bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:22.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f2414bf9640 10 monclient(hunting): picked mon.noname-a con 0x7f24100a0300 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:39:22.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f2414bf9640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f24100a06e0 0x7f241012ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:22.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f2414bf9640 10 monclient(hunting): picked mon.noname-c con 0x7f24100a06e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:22.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f2414bf9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24101350e0 0x7f24101304c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:22.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f2414bf9640 10 monclient(hunting): picked mon.noname-b con 0x7f24101350e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:39:22.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f2414bf9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:22.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f2414bf9640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f24100a0c40 con 0x7f24100a0300 2024-04-26T00:39:22.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f2414bf9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:22.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f2414bf9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f24100a0ac0 con 0x7f24101350e0 2024-04-26T00:39:22.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f2414bf9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:22.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f2414bf9640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f241013fe10 con 0x7f24100a06e0 2024-04-26T00:39:22.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f2414bf9640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:22.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f2414bf9640 10 monclient(hunting): authenticate will time out at 2143.822329s 2024-04-26T00:39:22.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240e59c640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f24100a0300 msgr2=0x7f241013bfe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:39:22.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240e59c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f24100a0300 0x7f241013bfe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:39:22.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240dd9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f24100a06e0 0x7f241012ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:22.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240dd9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f24100a06e0 0x7f241012ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:35452/0 (socket says 172.21.15.37:35452) 2024-04-26T00:39:22.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240dd9b640 1 -- 172.21.15.37:0/3862946435 learned_addr learned my addr 172.21.15.37:0/3862946435 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:22.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240dd9b640 10 monclient(hunting): get_auth_request con 0x7f24100a06e0 auth_method 0 2024-04-26T00:39:22.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:22.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:22.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:22.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:22.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:22.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:22.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240ed9d640 1 --2- 172.21.15.37:0/3862946435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24101350e0 0x7f24101304c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:22.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240ed9d640 10 monclient(hunting): get_auth_request con 0x7f24101350e0 auth_method 0 2024-04-26T00:39:22.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:22.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:22.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:22.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240dd9b640 10 monclient(hunting): handle_auth_done global_id 204473 payload 824 2024-04-26T00:39:22.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240dd9b640 1 -- 172.21.15.37:0/3862946435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24101350e0 msgr2=0x7f24101304c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:22.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240dd9b640 1 --2- 172.21.15.37:0/3862946435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24101350e0 0x7f24101304c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:22.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240dd9b640 1 -- 172.21.15.37:0/3862946435 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f24100a0300 msgr2=0x7f241013bfe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:39:22.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240dd9b640 1 --2- 172.21.15.37:0/3862946435 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f24100a0300 0x7f241013bfe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:22.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240dd9b640 10 monclient: _finish_hunting 0 2024-04-26T00:39:22.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240dd9b640 1 monclient: found mon.noname-c 2024-04-26T00:39:22.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:22.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:39:22.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240dd9b640 1 -- 172.21.15.37:0/3862946435 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f241011c090 con 0x7f24100a06e0 2024-04-26T00:39:22.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.514+0000 7f240ed9d640 1 --2- 172.21.15.37:0/3862946435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24101350e0 0x7f24101304c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:39:22.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f240dd9b640 1 -- 172.21.15.37:0/3862946435 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f24100a06e0 msgr2=0x7f241012ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:22.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f240dd9b640 1 --2- 172.21.15.37:0/3862946435 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f24100a06e0 0x7f241012ff80 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f23f800b580 tx=0x7f23f800ba40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:22.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f2406ffd640 1 -- 172.21.15.37:0/3862946435 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23f8016070 con 0x7f24100a06e0 2024-04-26T00:39:22.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f2406ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:22.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f2406ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:39:22.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f2406ffd640 10 monclient: dump: 2024-04-26T00:39:22.528 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:22.528 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:22.528 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:22.528 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:22.528 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:22.528 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:22.528 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:22.528 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:22.528 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:22.528 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:22.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f2406ffd640 10 monclient: _finish_auth 0 2024-04-26T00:39:22.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f2406ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:39:22.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f2406ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:22.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f2414bf9640 5 monclient: authenticate success, global_id 204473 2024-04-26T00:39:22.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f2414bf9640 10 monclient: _renew_subs 2024-04-26T00:39:22.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f2406ffd640 1 -- 172.21.15.37:0/3862946435 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23f80050d0 con 0x7f24100a06e0 2024-04-26T00:39:22.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f2414bf9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:22.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f2414bf9640 1 -- 172.21.15.37:0/3862946435 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f241011c370 con 0x7f24100a06e0 2024-04-26T00:39:22.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f2406ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:22.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f2406ffd640 1 -- 172.21.15.37:0/3862946435 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23f80054c0 con 0x7f24100a06e0 2024-04-26T00:39:22.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f2406ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:22.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f2406ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:22.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f2406ffd640 10 monclient: dump: 2024-04-26T00:39:22.529 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:22.530 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:22.530 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:22.530 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:22.530 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:22.530 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:22.530 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:22.530 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:22.530 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:22.530 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:22.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f240cd99640 4 set_mon_vals no callback set 2024-04-26T00:39:22.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f2414bf9640 10 monclient: _renew_subs 2024-04-26T00:39:22.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f2414bf9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:22.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f2414bf9640 1 -- 172.21.15.37:0/3862946435 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f24101cc560 con 0x7f24100a06e0 2024-04-26T00:39:22.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f2414bf9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:22.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f2414bf9640 1 --2- 172.21.15.37:0/3862946435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f241009e260 0x7f24101cf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:22.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f2414bf9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:22.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f2414bf9640 1 -- 172.21.15.37:0/3862946435 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f241011c890 con 0x7f241009e260 2024-04-26T00:39:22.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f2406ffd640 1 -- 172.21.15.37:0/3862946435 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f23f8019070 con 0x7f24100a06e0 2024-04-26T00:39:22.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f240e59c640 1 --2- 172.21.15.37:0/3862946435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f241009e260 0x7f24101cf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:22.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f2406ffd640 1 --2- 172.21.15.37:0/3862946435 >> v2:172.21.15.45:6812/2379918623 conn(0x7f23f0035e80 0x7f23f0038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:22.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f240e59c640 10 monclient: get_auth_request con 0x7f241009e260 auth_method 0 2024-04-26T00:39:22.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f240e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:22.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f240e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:22.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f240e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:22.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f2406ffd640 1 -- 172.21.15.37:0/3862946435 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(758..758 src has 705..758) ==== 3943+0+0 (secure 0 0 0) 0x7f23f8047e80 con 0x7f24100a06e0 2024-04-26T00:39:22.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f240e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:22.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f240e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:22.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f240e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:22.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f240ed9d640 1 --2- 172.21.15.37:0/3862946435 >> v2:172.21.15.45:6812/2379918623 conn(0x7f23f0035e80 0x7f23f0038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:22.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f240ed9d640 10 monclient: get_auth_request con 0x7f23f0035e80 auth_method 0 2024-04-26T00:39:22.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f240e59c640 10 monclient(hunting): handle_auth_done global_id 224245 payload 824 2024-04-26T00:39:22.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.518+0000 7f240e59c640 1 -- 172.21.15.37:0/3862946435 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f241004f510 con 0x7f241009e260 2024-04-26T00:39:22.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.522+0000 7f240ed9d640 1 --2- 172.21.15.37:0/3862946435 >> v2:172.21.15.45:6812/2379918623 conn(0x7f23f0035e80 0x7f23f0038340 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f2400006fd0 tx=0x7f2400006ea0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:22.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.522+0000 7f240e59c640 1 -- 172.21.15.37:0/3862946435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f241009e260 msgr2=0x7f24101cf5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:22.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.522+0000 7f240e59c640 1 --2- 172.21.15.37:0/3862946435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f241009e260 0x7f24101cf5b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f241006de30 tx=0x7f24080196c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:22.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.522+0000 7f2406ffd640 1 -- 172.21.15.37:0/3862946435 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24080bd030 con 0x7f241009e260 2024-04-26T00:39:22.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.522+0000 7f2406ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:22.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.522+0000 7f2406ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:39:22.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.522+0000 7f2406ffd640 10 monclient: dump: 2024-04-26T00:39:22.532 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:22.532 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:22.533 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:22.533 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:22.533 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:22.533 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:22.533 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:22.533 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:22.533 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:22.533 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:22.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.522+0000 7f2406ffd640 1 -- 172.21.15.37:0/3862946435 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f24080b9470 con 0x7f241009e260 2024-04-26T00:39:22.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.522+0000 7f2406ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:22.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.522+0000 7f2406ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:39:22.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.522+0000 7f2406ffd640 1 -- 172.21.15.37:0/3862946435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f241009e260 msgr2=0x7f24101cf5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:22.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.522+0000 7f2406ffd640 1 --2- 172.21.15.37:0/3862946435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f241009e260 0x7f24101cf5b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f241006de30 tx=0x7f24080196c0 comp rx=0 tx=0).stop 2024-04-26T00:39:22.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.522+0000 7f2406ffd640 1 -- 172.21.15.37:0/3862946435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f241009e260 msgr2=0x7f24101cf5b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:39:22.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.522+0000 7f2406ffd640 1 --2- 172.21.15.37:0/3862946435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f241009e260 0x7f24101cf5b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:22.534 INFO:tasks.rados.rados.0.smithi045.stdout:3362: seq_num 1197 ranges {517979=740474,1778946=339033} 2024-04-26T00:39:22.534 INFO:tasks.rados.rados.0.smithi045.stdout:3362: writing smithi04526579-36 from 517979 to 1258453 tid 1 2024-04-26T00:39:22.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.530+0000 7f2414bf9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:39:22.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.530+0000 7f2414bf9640 1 --2- 172.21.15.37:0/3862946435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24101cfaf0 0x7f24101cff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:22.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.530+0000 7f2414bf9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:22.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.530+0000 7f2414bf9640 1 -- 172.21.15.37:0/3862946435 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f24101d0440 con 0x7f24101cfaf0 2024-04-26T00:39:22.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.530+0000 7f240e59c640 1 --2- 172.21.15.37:0/3862946435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24101cfaf0 0x7f24101cff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:22.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.530+0000 7f240e59c640 10 monclient: get_auth_request con 0x7f24101cfaf0 auth_method 0 2024-04-26T00:39:22.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.530+0000 7f240e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:22.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.530+0000 7f240e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:22.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.530+0000 7f240e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:22.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.530+0000 7f240e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:22.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.530+0000 7f240e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:22.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.530+0000 7f240e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:22.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.530+0000 7f240e59c640 10 monclient(hunting): handle_auth_done global_id 224248 payload 824 2024-04-26T00:39:22.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.530+0000 7f240e59c640 1 -- 172.21.15.37:0/3862946435 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f241009eb40 con 0x7f24101cfaf0 2024-04-26T00:39:22.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.534+0000 7f240e59c640 1 -- 172.21.15.37:0/3862946435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24101cfaf0 msgr2=0x7f24101cff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:22.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.534+0000 7f240e59c640 1 --2- 172.21.15.37:0/3862946435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24101cfaf0 0x7f24101cff00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f240801afd0 tx=0x7f2408009810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:22.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.534+0000 7f2406ffd640 1 -- 172.21.15.37:0/3862946435 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24080c6070 con 0x7f24101cfaf0 2024-04-26T00:39:22.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.534+0000 7f2406ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:22.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.534+0000 7f2406ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:39:22.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.534+0000 7f2406ffd640 10 monclient: dump: 2024-04-26T00:39:22.537 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:22.537 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:22.537 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:22.537 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:22.537 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:22.537 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:22.538 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:22.538 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:22.538 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:22.538 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:22.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.534+0000 7f2406ffd640 1 -- 172.21.15.37:0/3862946435 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f24080b7650 con 0x7f24101cfaf0 2024-04-26T00:39:22.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.534+0000 7f2406ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:39:22.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.534+0000 7f2406ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:39:22.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.534+0000 7f2406ffd640 1 -- 172.21.15.37:0/3862946435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24101cfaf0 msgr2=0x7f24101cff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:22.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.534+0000 7f2406ffd640 1 --2- 172.21.15.37:0/3862946435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24101cfaf0 0x7f24101cff00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f240801afd0 tx=0x7f2408009810 comp rx=0 tx=0).stop 2024-04-26T00:39:22.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.534+0000 7f2406ffd640 1 -- 172.21.15.37:0/3862946435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24101cfaf0 msgr2=0x7f24101cff00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:39:22.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.534+0000 7f2406ffd640 1 --2- 172.21.15.37:0/3862946435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f24101cfaf0 0x7f24101cff00 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:22.539 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:39:22.539 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:39:22.539 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:39:22.539 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:39:22.539 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 146, 2024-04-26T00:39:22.539 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 76198, 2024-04-26T00:39:22.539 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:39:22.539 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:39:22.539 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:39:22.539 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:22.539 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:39:22.539 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:39:22.539 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:39:22.540 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:39:22.540 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:22.540 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:22.540 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:22.540 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:22.540 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:22.540 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:22.540 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:22.540 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:22.540 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:22.540 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:22.540 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:22.540 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:22.540 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:39:22.540 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:39:22.541 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:22.541 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:22.541 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:22.541 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:22.541 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:22.541 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:22.541 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:22.541 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:22.541 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:22.541 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:22.541 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:22.541 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:22.541 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:22.541 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:39:22.541 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:39:22.542 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:39:22.542 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:39:22.542 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:39:22.542 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:39:22.542 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:39:22.542 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:39:22.542 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:39:22.542 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:39:22.542 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:39:22.542 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:39:22.542 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:39:22.542 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:39:22.542 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:39:22.543 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:39:22.543 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:39:22.543 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:22.543 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:22.543 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:22.543 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:22.543 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:22.543 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:22.543 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:22.543 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:22.543 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:22.543 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:22.543 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:22.543 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:22.544 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:39:22.544 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:22.544 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:39:22.544 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:22.544 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:39:22.544 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:39:22.544 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:22.544 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:22.544 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:22.544 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:22.544 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:39:22.544 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:22.544 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:22.544 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:22.544 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:22.545 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:39:22.545 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:22.545 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:22.545 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:22.545 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:22.545 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:39:22.545 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:39:22.545 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:22.545 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:22.545 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:22.545 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:22.545 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:22.545 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:39:22.545 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:39:22.545 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:22.545 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:22.546 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:22.546 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:22.546 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:39:22.546 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:22.546 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:22.546 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:22.546 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:22.546 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:39:22.546 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:22.546 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:22.546 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:22.546 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:22.546 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:39:22.546 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:39:22.547 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:22.547 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:22.547 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:22.547 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:22.547 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:22.547 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:39:22.547 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:39:22.547 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:22.547 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:22.547 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:22.547 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:22.547 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:39:22.547 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:22.547 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:22.547 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:22.548 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:22.548 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:39:22.548 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:22.548 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:22.548 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:22.548 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:22.548 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:39:22.548 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:39:22.548 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:22.548 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:22.548 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:22.548 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:22.548 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:22.548 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:22.549 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:39:22.549 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:39:22.549 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:22.549 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:22.549 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:22.549 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:39:22.549 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:22.549 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:22.549 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:39:22.549 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:22.549 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:22.549 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:22.550 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:39:22.550 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:22.550 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:22.550 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:39:22.550 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:22.550 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:22.550 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:22.550 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:39:22.550 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:22.550 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:22.550 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:39:22.550 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:22.550 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:22.551 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:22.551 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:39:22.551 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:22.551 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:22.551 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:22.551 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:39:22.551 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:39:22.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.534+0000 7f2414bf9640 1 -- 172.21.15.37:0/3862946435 >> v2:172.21.15.45:6812/2379918623 conn(0x7f23f0035e80 msgr2=0x7f23f0038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:22.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.534+0000 7f2414bf9640 1 --2- 172.21.15.37:0/3862946435 >> v2:172.21.15.45:6812/2379918623 conn(0x7f23f0035e80 0x7f23f0038340 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f2400006fd0 tx=0x7f2400006ea0 comp rx=0 tx=0).stop 2024-04-26T00:39:22.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.534+0000 7f2414bf9640 10 monclient: shutdown 2024-04-26T00:39:22.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.534+0000 7f2414bf9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:22.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.534+0000 7f2414bf9640 1 -- 172.21.15.37:0/3862946435 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f24100a06e0 msgr2=0x7f241012ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:22.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.534+0000 7f2414bf9640 1 --2- 172.21.15.37:0/3862946435 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f24100a06e0 0x7f241012ff80 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f23f800b580 tx=0x7f23f800ba40 comp rx=0 tx=0).stop 2024-04-26T00:39:22.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.534+0000 7f240ed9d640 1 -- 172.21.15.37:0/3862946435 reap_dead start 2024-04-26T00:39:22.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.534+0000 7f2414bf9640 1 -- 172.21.15.37:0/3862946435 shutdown_connections 2024-04-26T00:39:22.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.534+0000 7f2414bf9640 1 --2- 172.21.15.37:0/3862946435 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f24100a06e0 0x7f241012ff80 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:22.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.534+0000 7f2414bf9640 1 -- 172.21.15.37:0/3862946435 >> 172.21.15.37:0/3862946435 conn(0x7f24100a1d50 msgr2=0x7f24100a3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:22.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.534+0000 7f2414bf9640 1 -- 172.21.15.37:0/3862946435 shutdown_connections 2024-04-26T00:39:22.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:22.534+0000 7f2414bf9640 1 -- 172.21.15.37:0/3862946435 wait complete. 2024-04-26T00:39:22.553 INFO:tasks.rados.rados.0.smithi045.stdout:3362: writing smithi04526579-36 from 1778946 to 2117979 tid 2 2024-04-26T00:39:22.553 INFO:tasks.rados.rados.0.smithi045.stdout:3359: done (2 left) 2024-04-26T00:39:22.553 INFO:tasks.rados.rados.0.smithi045.stdout:3360: done (1 left) 2024-04-26T00:39:22.553 INFO:tasks.rados.rados.0.smithi045.stdout:3363: snap_remove snap 370 2024-04-26T00:39:22.555 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:39:22.555 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-04-26T00:39:23.228 INFO:tasks.rados.rados.0.smithi045.stdout:3363: done (1 left) 2024-04-26T00:39:23.228 INFO:tasks.rados.rados.0.smithi045.stdout:3364: read oid 6 snap -1 2024-04-26T00:39:23.228 INFO:tasks.rados.rados.0.smithi045.stdout:3364: expect (ObjNum 1060 snap 318 seq_num 1060) 2024-04-26T00:39:23.228 INFO:tasks.rados.rados.0.smithi045.stdout:3365: delete oid 1 current snap is 372 2024-04-26T00:39:23.228 INFO:tasks.rados.rados.0.smithi045.stdout:3362: finishing write tid 1 to smithi04526579-36 2024-04-26T00:39:23.228 INFO:tasks.rados.rados.0.smithi045.stdout:3362: finishing write tid 2 to smithi04526579-36 2024-04-26T00:39:23.228 INFO:tasks.rados.rados.0.smithi045.stdout:3362: finishing write tid 3 to smithi04526579-36 2024-04-26T00:39:23.228 INFO:tasks.rados.rados.0.smithi045.stdout:3362: finishing write tid 4 to smithi04526579-36 2024-04-26T00:39:23.228 INFO:tasks.rados.rados.0.smithi045.stdout:3362: oid 36 updating version 0 to 2443 2024-04-26T00:39:23.228 INFO:tasks.rados.rados.0.smithi045.stdout:3362: oid 36 version 2443 is already newer than 2442 2024-04-26T00:39:23.228 INFO:tasks.rados.rados.0.smithi045.stdout:3362: oid 36 updating version 2443 to 2444 2024-04-26T00:39:23.229 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 2444 (ObjNum 1197 snap 372 seq_num 1197) dirty exists 2024-04-26T00:39:23.229 INFO:tasks.rados.rados.0.smithi045.stdout:3362: left oid 36 (ObjNum 1197 snap 372 seq_num 1197) 2024-04-26T00:39:23.235 INFO:tasks.rados.rados.0.smithi045.stdout:3362: done (2 left) 2024-04-26T00:39:23.235 INFO:tasks.rados.rados.0.smithi045.stdout:3365: done (1 left) 2024-04-26T00:39:23.235 INFO:tasks.rados.rados.0.smithi045.stdout:3366: snap_remove snap 364 2024-04-26T00:39:24.234 INFO:tasks.rados.rados.0.smithi045.stdout:3366: done (1 left) 2024-04-26T00:39:24.234 INFO:tasks.rados.rados.0.smithi045.stdout:3367: write (excl) oid 35 current snap is 372 2024-04-26T00:39:24.234 INFO:tasks.rados.rados.0.smithi045.stdout:3367: seq_num 1199 ranges {551593=490972,1779557=372036} 2024-04-26T00:39:24.251 INFO:tasks.rados.rados.0.smithi045.stdout:3367: writing smithi04526579-35 from 551593 to 1042565 tid 1 2024-04-26T00:39:24.269 INFO:tasks.rados.rados.0.smithi045.stdout:3367: writing smithi04526579-35 from 1779557 to 2151593 tid 2 2024-04-26T00:39:24.270 INFO:tasks.rados.rados.0.smithi045.stdout:3368: write (excl) oid 3 current snap is 372 2024-04-26T00:39:24.270 INFO:tasks.rados.rados.0.smithi045.stdout:3368: seq_num 1200 ranges {568400=566221,1838039=330361} 2024-04-26T00:39:24.289 INFO:tasks.rados.rados.0.smithi045.stdout:3368: writing smithi04526579-3 from 568400 to 1134621 tid 1 2024-04-26T00:39:24.302 INFO:tasks.rados.rados.0.smithi045.stdout:3368: writing smithi04526579-3 from 1838039 to 2168400 tid 2 2024-04-26T00:39:24.302 INFO:tasks.rados.rados.0.smithi045.stdout:3369: write (excl) oid 19 current snap is 372 2024-04-26T00:39:24.302 INFO:tasks.rados.rados.0.smithi045.stdout:3369: seq_num 1201 ranges {585207=641470,1896521=288686} 2024-04-26T00:39:24.315 INFO:tasks.rados.rados.0.smithi045.stdout:3369: writing smithi04526579-19 from 585207 to 1226677 tid 1 2024-04-26T00:39:24.323 INFO:tasks.rados.rados.0.smithi045.stdout:3369: writing smithi04526579-19 from 1896521 to 2185207 tid 2 2024-04-26T00:39:24.324 INFO:tasks.rados.rados.0.smithi045.stdout:3370: read oid 15 snap 352 2024-04-26T00:39:24.324 INFO:tasks.rados.rados.0.smithi045.stdout:3370: expect (ObjNum 1139 snap 345 seq_num 1139) 2024-04-26T00:39:24.324 INFO:tasks.rados.rados.0.smithi045.stdout:3371: write oid 29 current snap is 372 2024-04-26T00:39:24.324 INFO:tasks.rados.rados.0.smithi045.stdout:3364: expect (ObjNum 1060 snap 318 seq_num 1060) 2024-04-26T00:39:24.506 INFO:tasks.rados.rados.0.smithi045.stdout:3367: finishing write tid 1 to smithi04526579-35 2024-04-26T00:39:24.507 INFO:tasks.rados.rados.0.smithi045.stdout:3367: finishing write tid 2 to smithi04526579-35 2024-04-26T00:39:24.507 INFO:tasks.rados.rados.0.smithi045.stdout:3368: finishing write tid 1 to smithi04526579-3 2024-04-26T00:39:24.507 INFO:tasks.rados.rados.0.smithi045.stdout:3367: finishing write tid 3 to smithi04526579-35 2024-04-26T00:39:24.507 INFO:tasks.rados.rados.0.smithi045.stdout:3367: finishing write tid 4 to smithi04526579-35 2024-04-26T00:39:24.507 INFO:tasks.rados.rados.0.smithi045.stdout:3367: oid 35 updating version 0 to 2489 2024-04-26T00:39:24.507 INFO:tasks.rados.rados.0.smithi045.stdout:3367: oid 35 version 2489 is already newer than 2487 2024-04-26T00:39:24.507 INFO:tasks.rados.rados.0.smithi045.stdout:3367: oid 35 version 2489 is already newer than 2488 2024-04-26T00:39:24.507 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 2489 (ObjNum 1199 snap 372 seq_num 1199) dirty exists 2024-04-26T00:39:24.507 INFO:tasks.rados.rados.0.smithi045.stdout:3367: left oid 35 (ObjNum 1199 snap 372 seq_num 1199) 2024-04-26T00:39:24.507 INFO:tasks.rados.rados.0.smithi045.stdout:3368: finishing write tid 2 to smithi04526579-3 2024-04-26T00:39:24.507 INFO:tasks.rados.rados.0.smithi045.stdout:3368: finishing write tid 3 to smithi04526579-3 2024-04-26T00:39:24.507 INFO:tasks.rados.rados.0.smithi045.stdout:3368: finishing write tid 4 to smithi04526579-3 2024-04-26T00:39:24.507 INFO:tasks.rados.rados.0.smithi045.stdout:3368: oid 3 updating version 0 to 2457 2024-04-26T00:39:24.507 INFO:tasks.rados.rados.0.smithi045.stdout:3368: oid 3 updating version 2457 to 2458 2024-04-26T00:39:24.508 INFO:tasks.rados.rados.0.smithi045.stdout:3368: oid 3 updating version 2458 to 2459 2024-04-26T00:39:24.508 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 2459 (ObjNum 1200 snap 372 seq_num 1200) dirty exists 2024-04-26T00:39:24.508 INFO:tasks.rados.rados.0.smithi045.stdout:3368: left oid 3 (ObjNum 1200 snap 372 seq_num 1200) 2024-04-26T00:39:24.508 INFO:tasks.rados.rados.0.smithi045.stdout:3369: finishing write tid 1 to smithi04526579-19 2024-04-26T00:39:24.508 INFO:tasks.rados.rados.0.smithi045.stdout:3370: expect (ObjNum 1139 snap 345 seq_num 1139) 2024-04-26T00:39:24.706 INFO:tasks.rados.rados.0.smithi045.stdout:3369: finishing write tid 2 to smithi04526579-19 2024-04-26T00:39:24.706 INFO:tasks.rados.rados.0.smithi045.stdout:3369: finishing write tid 3 to smithi04526579-19 2024-04-26T00:39:24.706 INFO:tasks.rados.rados.0.smithi045.stdout:3369: finishing write tid 4 to smithi04526579-19 2024-04-26T00:39:24.706 INFO:tasks.rados.rados.0.smithi045.stdout:3369: oid 19 updating version 0 to 2491 2024-04-26T00:39:24.706 INFO:tasks.rados.rados.0.smithi045.stdout:3369: oid 19 updating version 2491 to 2492 2024-04-26T00:39:24.706 INFO:tasks.rados.rados.0.smithi045.stdout:3369: oid 19 version 2492 is already newer than 2490 2024-04-26T00:39:24.706 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 2492 (ObjNum 1201 snap 372 seq_num 1201) dirty exists 2024-04-26T00:39:24.706 INFO:tasks.rados.rados.0.smithi045.stdout:3369: left oid 19 (ObjNum 1201 snap 372 seq_num 1201) 2024-04-26T00:39:24.706 INFO:tasks.rados.rados.0.smithi045.stdout:3371: seq_num 1202 ranges {602014=433072,1671356=530658} 2024-04-26T00:39:24.717 INFO:tasks.rados.rados.0.smithi045.stdout:3371: writing smithi04526579-29 from 602014 to 1035086 tid 1 2024-04-26T00:39:24.731 INFO:tasks.rados.rados.0.smithi045.stdout:3371: writing smithi04526579-29 from 1671356 to 2202014 tid 2 2024-04-26T00:39:24.731 INFO:tasks.rados.rados.0.smithi045.stdout:3364: done (5 left) 2024-04-26T00:39:24.731 INFO:tasks.rados.rados.0.smithi045.stdout:3367: done (4 left) 2024-04-26T00:39:24.731 INFO:tasks.rados.rados.0.smithi045.stdout:3368: done (3 left) 2024-04-26T00:39:24.731 INFO:tasks.rados.rados.0.smithi045.stdout:3369: done (2 left) 2024-04-26T00:39:24.731 INFO:tasks.rados.rados.0.smithi045.stdout:3370: done (1 left) 2024-04-26T00:39:24.731 INFO:tasks.rados.rados.0.smithi045.stdout:3372: copy_from oid 18 from oid 1 current snap is 372 2024-04-26T00:39:24.731 INFO:tasks.rados.rados.0.smithi045.stdout:3371: finishing write tid 1 to smithi04526579-29 2024-04-26T00:39:24.731 INFO:tasks.rados.rados.0.smithi045.stdout:3373: read oid 23 snap -1 2024-04-26T00:39:24.731 INFO:tasks.rados.rados.0.smithi045.stdout:3373: expect deleted 2024-04-26T00:39:24.731 INFO:tasks.rados.rados.0.smithi045.stdout:3374: snap_remove snap 352 2024-04-26T00:39:25.472 INFO:tasks.rados.rados.0.smithi045.stdout:3374: done (3 left) 2024-04-26T00:39:25.473 INFO:tasks.rados.rados.0.smithi045.stdout:3375: snap_remove snap 371 2024-04-26T00:39:26.480 INFO:tasks.rados.rados.0.smithi045.stdout:3375: done (3 left) 2024-04-26T00:39:26.481 INFO:tasks.rados.rados.0.smithi045.stdout:3376: write oid 42 current snap is 372 2024-04-26T00:39:26.481 INFO:tasks.rados.rados.0.smithi045.stdout:3376: seq_num 1203 ranges {618821=508321,1613485=531077} 2024-04-26T00:39:26.498 INFO:tasks.rados.rados.0.smithi045.stdout:3376: writing smithi04526579-42 from 618821 to 1127142 tid 1 2024-04-26T00:39:26.515 INFO:tasks.rados.rados.0.smithi045.stdout:3376: writing smithi04526579-42 from 1613485 to 2144562 tid 2 2024-04-26T00:39:26.515 INFO:tasks.rados.rados.0.smithi045.stdout:3377: snap_create 2024-04-26T00:39:26.515 INFO:tasks.rados.rados.0.smithi045.stdout:3372: finishing copy_from to smithi04526579-18 2024-04-26T00:39:26.515 INFO:tasks.rados.rados.0.smithi045.stdout:3372: got expected ENOENT (src dne) 2024-04-26T00:39:26.515 INFO:tasks.rados.rados.0.smithi045.stdout:3372: finishing copy_from racing read to smithi04526579-18 2024-04-26T00:39:26.516 INFO:tasks.rados.rados.0.smithi045.stdout:3371: finishing write tid 2 to smithi04526579-29 2024-04-26T00:39:26.516 INFO:tasks.rados.rados.0.smithi045.stdout:3371: finishing write tid 3 to smithi04526579-29 2024-04-26T00:39:26.516 INFO:tasks.rados.rados.0.smithi045.stdout:3371: finishing write tid 4 to smithi04526579-29 2024-04-26T00:39:26.516 INFO:tasks.rados.rados.0.smithi045.stdout:3371: oid 29 updating version 0 to 2212 2024-04-26T00:39:26.516 INFO:tasks.rados.rados.0.smithi045.stdout:3371: oid 29 updating version 2212 to 2213 2024-04-26T00:39:26.516 INFO:tasks.rados.rados.0.smithi045.stdout:3371: oid 29 version 2213 is already newer than 2211 2024-04-26T00:39:26.516 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 2213 (ObjNum 1202 snap 372 seq_num 1202) dirty exists 2024-04-26T00:39:26.516 INFO:tasks.rados.rados.0.smithi045.stdout:3371: left oid 29 (ObjNum 1202 snap 372 seq_num 1202) 2024-04-26T00:39:26.522 INFO:tasks.rados.rados.0.smithi045.stdout:3376: finishing write tid 1 to smithi04526579-42 2024-04-26T00:39:26.534 INFO:tasks.rados.rados.0.smithi045.stdout:3376: finishing write tid 2 to smithi04526579-42 2024-04-26T00:39:26.534 INFO:tasks.rados.rados.0.smithi045.stdout:3376: finishing write tid 3 to smithi04526579-42 2024-04-26T00:39:26.534 INFO:tasks.rados.rados.0.smithi045.stdout:3376: finishing write tid 4 to smithi04526579-42 2024-04-26T00:39:26.535 INFO:tasks.rados.rados.0.smithi045.stdout:3376: oid 42 updating version 0 to 2495 2024-04-26T00:39:26.535 INFO:tasks.rados.rados.0.smithi045.stdout:3376: oid 42 version 2495 is already newer than 2493 2024-04-26T00:39:26.535 INFO:tasks.rados.rados.0.smithi045.stdout:3376: oid 42 version 2495 is already newer than 2494 2024-04-26T00:39:26.535 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 2495 (ObjNum 1203 snap 372 seq_num 1203) dirty exists 2024-04-26T00:39:26.535 INFO:tasks.rados.rados.0.smithi045.stdout:3376: left oid 42 (ObjNum 1203 snap 372 seq_num 1203) 2024-04-26T00:39:27.487 INFO:tasks.rados.rados.0.smithi045.stdout:3371: done (4 left) 2024-04-26T00:39:27.488 INFO:tasks.rados.rados.0.smithi045.stdout:3372: done (3 left) 2024-04-26T00:39:27.488 INFO:tasks.rados.rados.0.smithi045.stdout:3373: done (2 left) 2024-04-26T00:39:27.488 INFO:tasks.rados.rados.0.smithi045.stdout:3376: done (1 left) 2024-04-26T00:39:27.488 INFO:tasks.rados.rados.0.smithi045.stdout:3377: done (0 left) 2024-04-26T00:39:27.488 INFO:tasks.rados.rados.0.smithi045.stdout:3378: write (excl) oid 25 current snap is 373 2024-04-26T00:39:27.488 INFO:tasks.rados.rados.0.smithi045.stdout:3378: seq_num 1204 ranges {635628=583570,1671967=563661} 2024-04-26T00:39:27.507 INFO:tasks.rados.rados.0.smithi045.stdout:3378: writing smithi04526579-25 from 635628 to 1219198 tid 1 2024-04-26T00:39:27.524 INFO:tasks.rados.rados.0.smithi045.stdout:3378: writing smithi04526579-25 from 1671967 to 2235628 tid 2 2024-04-26T00:39:27.524 INFO:tasks.rados.rados.0.smithi045.stdout:3379: write (excl) oid 19 current snap is 373 2024-04-26T00:39:27.524 INFO:tasks.rados.rados.0.smithi045.stdout:3379: seq_num 1205 ranges {652435=658819,1730449=521986} 2024-04-26T00:39:27.544 INFO:tasks.rados.rados.0.smithi045.stdout:3379: writing smithi04526579-19 from 652435 to 1311254 tid 1 2024-04-26T00:39:27.556 INFO:tasks.rados.rados.0.smithi045.stdout:3379: writing smithi04526579-19 from 1730449 to 2252435 tid 2 2024-04-26T00:39:27.556 INFO:tasks.rados.rados.0.smithi045.stdout:3380: write (excl) oid 17 current snap is 373 2024-04-26T00:39:27.556 INFO:tasks.rados.rados.0.smithi045.stdout:3380: seq_num 1206 ranges {669242=734068,2188931=80311} 2024-04-26T00:39:27.569 INFO:tasks.rados.rados.0.smithi045.stdout:3380: writing smithi04526579-17 from 669242 to 1403310 tid 1 2024-04-26T00:39:27.576 INFO:tasks.rados.rados.0.smithi045.stdout:3380: writing smithi04526579-17 from 2188931 to 2269242 tid 2 2024-04-26T00:39:27.576 INFO:tasks.rados.rados.0.smithi045.stdout:3381: snap_remove snap 369 2024-04-26T00:39:28.497 INFO:tasks.rados.rados.0.smithi045.stdout:3381: done (3 left) 2024-04-26T00:39:28.497 INFO:tasks.rados.rados.0.smithi045.stdout:3382: snap_remove snap 372 2024-04-26T00:39:29.458 INFO:tasks.rados.rados.0.smithi045.stdout:3382: done (3 left) 2024-04-26T00:39:29.458 INFO:tasks.rados.rados.0.smithi045.stdout:3383: snap_remove snap 355 2024-04-26T00:39:30.474 INFO:tasks.rados.rados.0.smithi045.stdout:3378: finishing write tid 1 to smithi04526579-25 2024-04-26T00:39:30.474 INFO:tasks.rados.rados.0.smithi045.stdout:3378: finishing write tid 2 to smithi04526579-25 2024-04-26T00:39:30.475 INFO:tasks.rados.rados.0.smithi045.stdout:3378: finishing write tid 3 to smithi04526579-25 2024-04-26T00:39:30.475 INFO:tasks.rados.rados.0.smithi045.stdout:3378: finishing write tid 4 to smithi04526579-25 2024-04-26T00:39:30.475 INFO:tasks.rados.rados.0.smithi045.stdout:3378: oid 25 updating version 0 to 2563 2024-04-26T00:39:30.475 INFO:tasks.rados.rados.0.smithi045.stdout:3378: oid 25 updating version 2563 to 2564 2024-04-26T00:39:30.475 INFO:tasks.rados.rados.0.smithi045.stdout:3378: oid 25 version 2564 is already newer than 2562 2024-04-26T00:39:30.475 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 2564 (ObjNum 1204 snap 373 seq_num 1204) dirty exists 2024-04-26T00:39:30.475 INFO:tasks.rados.rados.0.smithi045.stdout:3378: left oid 25 (ObjNum 1204 snap 373 seq_num 1204) 2024-04-26T00:39:30.475 INFO:tasks.rados.rados.0.smithi045.stdout:3379: finishing write tid 1 to smithi04526579-19 2024-04-26T00:39:30.475 INFO:tasks.rados.rados.0.smithi045.stdout:3379: finishing write tid 2 to smithi04526579-19 2024-04-26T00:39:30.475 INFO:tasks.rados.rados.0.smithi045.stdout:3379: finishing write tid 3 to smithi04526579-19 2024-04-26T00:39:30.475 INFO:tasks.rados.rados.0.smithi045.stdout:3379: finishing write tid 4 to smithi04526579-19 2024-04-26T00:39:30.475 INFO:tasks.rados.rados.0.smithi045.stdout:3379: oid 19 updating version 0 to 2498 2024-04-26T00:39:30.475 INFO:tasks.rados.rados.0.smithi045.stdout:3379: oid 19 version 2498 is already newer than 2497 2024-04-26T00:39:30.475 INFO:tasks.rados.rados.0.smithi045.stdout:3379: oid 19 version 2498 is already newer than 2496 2024-04-26T00:39:30.475 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 2498 (ObjNum 1205 snap 373 seq_num 1205) dirty exists 2024-04-26T00:39:30.476 INFO:tasks.rados.rados.0.smithi045.stdout:3379: left oid 19 (ObjNum 1205 snap 373 seq_num 1205) 2024-04-26T00:39:30.476 INFO:tasks.rados.rados.0.smithi045.stdout:3380: finishing write tid 1 to smithi04526579-17 2024-04-26T00:39:30.476 INFO:tasks.rados.rados.0.smithi045.stdout:3380: finishing write tid 2 to smithi04526579-17 2024-04-26T00:39:30.476 INFO:tasks.rados.rados.0.smithi045.stdout:3380: finishing write tid 3 to smithi04526579-17 2024-04-26T00:39:30.476 INFO:tasks.rados.rados.0.smithi045.stdout:3380: finishing write tid 4 to smithi04526579-17 2024-04-26T00:39:30.476 INFO:tasks.rados.rados.0.smithi045.stdout:3380: oid 17 updating version 0 to 2413 2024-04-26T00:39:30.476 INFO:tasks.rados.rados.0.smithi045.stdout:3380: oid 17 updating version 2413 to 2415 2024-04-26T00:39:30.476 INFO:tasks.rados.rados.0.smithi045.stdout:3380: oid 17 version 2415 is already newer than 2414 2024-04-26T00:39:30.476 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 2415 (ObjNum 1206 snap 373 seq_num 1206) dirty exists 2024-04-26T00:39:30.476 INFO:tasks.rados.rados.0.smithi045.stdout:3380: left oid 17 (ObjNum 1206 snap 373 seq_num 1206) 2024-04-26T00:39:30.476 INFO:tasks.rados.rados.0.smithi045.stdout:3378: done (3 left) 2024-04-26T00:39:30.476 INFO:tasks.rados.rados.0.smithi045.stdout:3379: done (2 left) 2024-04-26T00:39:30.476 INFO:tasks.rados.rados.0.smithi045.stdout:3380: done (1 left) 2024-04-26T00:39:30.476 INFO:tasks.rados.rados.0.smithi045.stdout:3383: done (0 left) 2024-04-26T00:39:30.476 INFO:tasks.rados.rados.0.smithi045.stdout:3384: read oid 45 snap -1 2024-04-26T00:39:30.476 INFO:tasks.rados.rados.0.smithi045.stdout:3384: expect (ObjNum 1062 snap 318 seq_num 1062) 2024-04-26T00:39:30.477 INFO:tasks.rados.rados.0.smithi045.stdout:3385: read oid 5 snap -1 2024-04-26T00:39:30.477 INFO:tasks.rados.rados.0.smithi045.stdout:3385: expect (ObjNum 955 snap 295 seq_num 955) 2024-04-26T00:39:30.478 INFO:tasks.rados.rados.0.smithi045.stdout:3386: read oid 26 snap -1 2024-04-26T00:39:30.478 INFO:tasks.rados.rados.0.smithi045.stdout:3386: expect deleted 2024-04-26T00:39:30.479 INFO:tasks.rados.rados.0.smithi045.stdout:3387: read oid 29 snap -1 2024-04-26T00:39:30.479 INFO:tasks.rados.rados.0.smithi045.stdout:3387: expect (ObjNum 1202 snap 372 seq_num 1202) 2024-04-26T00:39:30.479 INFO:tasks.rados.rados.0.smithi045.stdout:3388: rollback oid 7 current snap is 373 2024-04-26T00:39:30.480 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 7 to 347 2024-04-26T00:39:30.480 INFO:tasks.rados.rados.0.smithi045.stdout:3389: write (excl) oid 50 current snap is 373 2024-04-26T00:39:30.480 INFO:tasks.rados.rados.0.smithi045.stdout:3389: seq_num 1207 ranges {686049=409317,1847413=438636} 2024-04-26T00:39:30.494 INFO:tasks.rados.rados.0.smithi045.stdout:3389: writing smithi04526579-50 from 686049 to 1095366 tid 1 2024-04-26T00:39:30.510 INFO:tasks.rados.rados.0.smithi045.stdout:3389: writing smithi04526579-50 from 1847413 to 2286049 tid 2 2024-04-26T00:39:30.510 INFO:tasks.rados.rados.0.smithi045.stdout:3390: rollback oid 49 current snap is 373 2024-04-26T00:39:30.510 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 49 to 361 2024-04-26T00:39:30.510 INFO:tasks.rados.rados.0.smithi045.stdout:3384: expect (ObjNum 1062 snap 318 seq_num 1062) 2024-04-26T00:39:30.769 INFO:tasks.rados.rados.0.smithi045.stdout:3385: expect (ObjNum 955 snap 295 seq_num 955) 2024-04-26T00:39:30.860 INFO:tasks.rados.rados.0.smithi045.stdout:3387: expect (ObjNum 1202 snap 372 seq_num 1202) 2024-04-26T00:39:30.973 INFO:tasks.rados.rados.0.smithi045.stdout:3389: finishing write tid 1 to smithi04526579-50 2024-04-26T00:39:30.973 INFO:tasks.rados.rados.0.smithi045.stdout:3388: finishing rollback tid 0 to smithi04526579-7 2024-04-26T00:39:30.973 INFO:tasks.rados.rados.0.smithi045.stdout:3388: finishing rollback tid 1 to smithi04526579-7 2024-04-26T00:39:30.973 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 2568 (ObjNum 145 snap 0 seq_num 2242206320) dirty dne 2024-04-26T00:39:30.973 INFO:tasks.rados.rados.0.smithi045.stdout:3389: finishing write tid 2 to smithi04526579-50 2024-04-26T00:39:30.973 INFO:tasks.rados.rados.0.smithi045.stdout:3384: done (6 left) 2024-04-26T00:39:30.973 INFO:tasks.rados.rados.0.smithi045.stdout:3385: done (5 left) 2024-04-26T00:39:30.973 INFO:tasks.rados.rados.0.smithi045.stdout:3386: done (4 left) 2024-04-26T00:39:30.973 INFO:tasks.rados.rados.0.smithi045.stdout:3387: done (3 left) 2024-04-26T00:39:30.973 INFO:tasks.rados.rados.0.smithi045.stdout:3388: done (2 left) 2024-04-26T00:39:30.974 INFO:tasks.rados.rados.0.smithi045.stdout:3391: snap_create 2024-04-26T00:39:30.974 INFO:tasks.rados.rados.0.smithi045.stdout:3389: finishing write tid 3 to smithi04526579-50 2024-04-26T00:39:30.974 INFO:tasks.rados.rados.0.smithi045.stdout:3389: finishing write tid 4 to smithi04526579-50 2024-04-26T00:39:30.974 INFO:tasks.rados.rados.0.smithi045.stdout:3389: oid 50 updating version 0 to 2417 2024-04-26T00:39:30.974 INFO:tasks.rados.rados.0.smithi045.stdout:3389: oid 50 version 2417 is already newer than 2416 2024-04-26T00:39:30.974 INFO:tasks.rados.rados.0.smithi045.stdout:3389: oid 50 updating version 2417 to 2418 2024-04-26T00:39:30.974 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 2418 (ObjNum 1207 snap 373 seq_num 1207) dirty exists 2024-04-26T00:39:30.974 INFO:tasks.rados.rados.0.smithi045.stdout:3389: left oid 50 (ObjNum 1207 snap 373 seq_num 1207) 2024-04-26T00:39:30.974 INFO:tasks.rados.rados.0.smithi045.stdout:3390: finishing rollback tid 0 to smithi04526579-49 2024-04-26T00:39:30.974 INFO:tasks.rados.rados.0.smithi045.stdout:3390: finishing rollback tid 1 to smithi04526579-49 2024-04-26T00:39:30.974 INFO:tasks.rados.rados.0.smithi045.stdout:3390: finishing rollback tid 2 to smithi04526579-49 2024-04-26T00:39:30.974 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 2462 (ObjNum 1162 snap 359 seq_num 1162) dirty exists 2024-04-26T00:39:31.513 INFO:tasks.rados.rados.0.smithi045.stdout:3389: done (2 left) 2024-04-26T00:39:31.513 INFO:tasks.rados.rados.0.smithi045.stdout:3390: done (1 left) 2024-04-26T00:39:31.513 INFO:tasks.rados.rados.0.smithi045.stdout:3391: done (0 left) 2024-04-26T00:39:31.513 INFO:tasks.rados.rados.0.smithi045.stdout:3392: rollback oid 6 current snap is 374 2024-04-26T00:39:31.513 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 6 to 368 2024-04-26T00:39:31.513 INFO:tasks.rados.rados.0.smithi045.stdout:3393: rollback oid 23 current snap is 374 2024-04-26T00:39:31.513 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 23 to 368 2024-04-26T00:39:31.513 INFO:tasks.rados.rados.0.smithi045.stdout:3394: write (excl) oid 2 current snap is 374 2024-04-26T00:39:31.513 INFO:tasks.rados.rados.0.smithi045.stdout:3394: seq_num 1208 ranges {702856=484566,1789542=513314} 2024-04-26T00:39:31.532 INFO:tasks.rados.rados.0.smithi045.stdout:3394: writing smithi04526579-2 from 702856 to 1187422 tid 1 2024-04-26T00:39:31.551 INFO:tasks.rados.rados.0.smithi045.stdout:3394: writing smithi04526579-2 from 1789542 to 2302856 tid 2 2024-04-26T00:39:31.551 INFO:tasks.rados.rados.0.smithi045.stdout:3395: snap_remove snap 344 2024-04-26T00:39:32.522 INFO:tasks.rados.rados.0.smithi045.stdout:3395: done (3 left) 2024-04-26T00:39:32.522 INFO:tasks.rados.rados.0.smithi045.stdout:3396: delete oid 42 current snap is 374 2024-04-26T00:39:32.522 INFO:tasks.rados.rados.0.smithi045.stdout:3393: finishing rollback tid 1 to smithi04526579-23 2024-04-26T00:39:32.522 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 2217 (ObjNum 929 snap 0 seq_num 2242351888) dirty dne 2024-04-26T00:39:32.522 INFO:tasks.rados.rados.0.smithi045.stdout:3392: finishing rollback tid 0 to smithi04526579-6 2024-04-26T00:39:32.522 INFO:tasks.rados.rados.0.smithi045.stdout:3392: finishing rollback tid 1 to smithi04526579-6 2024-04-26T00:39:32.522 INFO:tasks.rados.rados.0.smithi045.stdout:3392: finishing rollback tid 2 to smithi04526579-6 2024-04-26T00:39:32.522 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 2007 (ObjNum 1060 snap 318 seq_num 1060) dirty exists 2024-04-26T00:39:32.522 INFO:tasks.rados.rados.0.smithi045.stdout:3394: finishing write tid 1 to smithi04526579-2 2024-04-26T00:39:32.522 INFO:tasks.rados.rados.0.smithi045.stdout:3394: finishing write tid 2 to smithi04526579-2 2024-04-26T00:39:32.522 INFO:tasks.rados.rados.0.smithi045.stdout:3394: finishing write tid 3 to smithi04526579-2 2024-04-26T00:39:32.522 INFO:tasks.rados.rados.0.smithi045.stdout:3394: finishing write tid 4 to smithi04526579-2 2024-04-26T00:39:32.523 INFO:tasks.rados.rados.0.smithi045.stdout:3394: oid 2 updating version 0 to 2495 2024-04-26T00:39:32.523 INFO:tasks.rados.rados.0.smithi045.stdout:3394: oid 2 updating version 2495 to 2496 2024-04-26T00:39:32.523 INFO:tasks.rados.rados.0.smithi045.stdout:3394: oid 2 updating version 2496 to 2497 2024-04-26T00:39:32.523 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 2497 (ObjNum 1208 snap 374 seq_num 1208) dirty exists 2024-04-26T00:39:32.523 INFO:tasks.rados.rados.0.smithi045.stdout:3394: left oid 2 (ObjNum 1208 snap 374 seq_num 1208) 2024-04-26T00:39:32.553 INFO:tasks.rados.rados.0.smithi045.stdout:3392: done (3 left) 2024-04-26T00:39:32.553 INFO:tasks.rados.rados.0.smithi045.stdout:3393: done (2 left) 2024-04-26T00:39:32.553 INFO:tasks.rados.rados.0.smithi045.stdout:3394: done (1 left) 2024-04-26T00:39:32.553 INFO:tasks.rados.rados.0.smithi045.stdout:3396: done (0 left) 2024-04-26T00:39:32.553 INFO:tasks.rados.rados.0.smithi045.stdout:3397: delete oid 39 current snap is 374 2024-04-26T00:39:32.564 INFO:tasks.rados.rados.0.smithi045.stdout:3397: done (0 left) 2024-04-26T00:39:32.564 INFO:tasks.rados.rados.0.smithi045.stdout:3398: snap_create 2024-04-26T00:39:33.616 INFO:tasks.rados.rados.0.smithi045.stdout:3398: done (0 left) 2024-04-26T00:39:33.616 INFO:tasks.rados.rados.0.smithi045.stdout:3399: write oid 24 current snap is 375 2024-04-26T00:39:33.616 INFO:tasks.rados.rados.0.smithi045.stdout:3399: seq_num 1211 ranges {753277=426666,1681341=545753} 2024-04-26T00:39:33.635 INFO:tasks.rados.rados.0.smithi045.stdout:3399: writing smithi04526579-24 from 753277 to 1179943 tid 1 2024-04-26T00:39:33.652 INFO:tasks.rados.rados.0.smithi045.stdout:3399: writing smithi04526579-24 from 1681341 to 2227094 tid 2 2024-04-26T00:39:33.652 INFO:tasks.rados.rados.0.smithi045.stdout:3400: snap_remove snap 368 2024-04-26T00:39:34.535 INFO:tasks.rados.rados.0.smithi045.stdout:3400: done (1 left) 2024-04-26T00:39:34.535 INFO:tasks.rados.rados.0.smithi045.stdout:3401: write (excl) oid 42 current snap is 375 2024-04-26T00:39:34.535 INFO:tasks.rados.rados.0.smithi045.stdout:3399: finishing write tid 1 to smithi04526579-24 2024-04-26T00:39:34.535 INFO:tasks.rados.rados.0.smithi045.stdout:3399: finishing write tid 2 to smithi04526579-24 2024-04-26T00:39:34.535 INFO:tasks.rados.rados.0.smithi045.stdout:3399: finishing write tid 3 to smithi04526579-24 2024-04-26T00:39:34.535 INFO:tasks.rados.rados.0.smithi045.stdout:3401: seq_num 1212 ranges {770084=501915,2023470=346614} 2024-04-26T00:39:34.555 INFO:tasks.rados.rados.0.smithi045.stdout:3401: writing smithi04526579-42 from 770084 to 1271999 tid 1 2024-04-26T00:39:34.574 INFO:tasks.rados.rados.0.smithi045.stdout:3401: writing smithi04526579-42 from 2023470 to 2370084 tid 2 2024-04-26T00:39:34.574 INFO:tasks.rados.rados.0.smithi045.stdout:3402: copy_from oid 32 from oid 28 current snap is 375 2024-04-26T00:39:34.574 INFO:tasks.rados.rados.0.smithi045.stdout:3399: finishing write tid 4 to smithi04526579-24 2024-04-26T00:39:34.574 INFO:tasks.rados.rados.0.smithi045.stdout:3399: oid 24 updating version 0 to 2578 2024-04-26T00:39:34.574 INFO:tasks.rados.rados.0.smithi045.stdout:3399: oid 24 version 2578 is already newer than 2576 2024-04-26T00:39:34.575 INFO:tasks.rados.rados.0.smithi045.stdout:3399: oid 24 version 2578 is already newer than 2577 2024-04-26T00:39:34.575 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 2578 (ObjNum 1211 snap 375 seq_num 1211) dirty exists 2024-04-26T00:39:34.575 INFO:tasks.rados.rados.0.smithi045.stdout:3399: left oid 24 (ObjNum 1211 snap 375 seq_num 1211) 2024-04-26T00:39:34.575 INFO:tasks.rados.rados.0.smithi045.stdout:3401: finishing write tid 1 to smithi04526579-42 2024-04-26T00:39:34.575 INFO:tasks.rados.rados.0.smithi045.stdout:3399: done (2 left) 2024-04-26T00:39:34.575 INFO:tasks.rados.rados.0.smithi045.stdout:3403: snap_create 2024-04-26T00:39:34.583 INFO:tasks.rados.rados.0.smithi045.stdout:3401: finishing write tid 2 to smithi04526579-42 2024-04-26T00:39:34.583 INFO:tasks.rados.rados.0.smithi045.stdout:3401: finishing write tid 3 to smithi04526579-42 2024-04-26T00:39:34.584 INFO:tasks.rados.rados.0.smithi045.stdout:3401: finishing write tid 4 to smithi04526579-42 2024-04-26T00:39:34.584 INFO:tasks.rados.rados.0.smithi045.stdout:3401: oid 42 updating version 0 to 2502 2024-04-26T00:39:34.584 INFO:tasks.rados.rados.0.smithi045.stdout:3401: oid 42 version 2502 is already newer than 2501 2024-04-26T00:39:34.584 INFO:tasks.rados.rados.0.smithi045.stdout:3401: oid 42 version 2502 is already newer than 2500 2024-04-26T00:39:34.584 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 2502 (ObjNum 1212 snap 375 seq_num 1212) dirty exists 2024-04-26T00:39:34.584 INFO:tasks.rados.rados.0.smithi045.stdout:3401: left oid 42 (ObjNum 1212 snap 375 seq_num 1212) 2024-04-26T00:39:34.585 INFO:tasks.rados.rados.0.smithi045.stdout:3402: finishing copy_from to smithi04526579-32 2024-04-26T00:39:34.585 INFO:tasks.rados.rados.0.smithi045.stdout:3402: got expected ENOENT (src dne) 2024-04-26T00:39:34.585 INFO:tasks.rados.rados.0.smithi045.stdout:3402: finishing copy_from racing read to smithi04526579-32 2024-04-26T00:39:35.470 INFO:tasks.rados.rados.0.smithi045.stdout:3401: done (2 left) 2024-04-26T00:39:35.471 INFO:tasks.rados.rados.0.smithi045.stdout:3402: done (1 left) 2024-04-26T00:39:35.471 INFO:tasks.rados.rados.0.smithi045.stdout:3403: done (0 left) 2024-04-26T00:39:35.471 INFO:tasks.rados.rados.0.smithi045.stdout:3404: snap_remove snap 330 2024-04-26T00:39:36.479 INFO:tasks.rados.rados.0.smithi045.stdout:3404: done (0 left) 2024-04-26T00:39:36.479 INFO:tasks.rados.rados.0.smithi045.stdout:3405: snap_create 2024-04-26T00:39:37.481 INFO:tasks.rados.rados.0.smithi045.stdout:3405: done (0 left) 2024-04-26T00:39:37.481 INFO:tasks.rados.rados.0.smithi045.stdout:3406: read oid 40 snap -1 2024-04-26T00:39:37.481 INFO:tasks.rados.rados.0.smithi045.stdout:3406: expect (ObjNum 1178 snap 366 seq_num 1178) 2024-04-26T00:39:37.481 INFO:tasks.rados.rados.0.smithi045.stdout:3407: snap_create 2024-04-26T00:39:37.496 INFO:tasks.rados.rados.0.smithi045.stdout:3406: expect (ObjNum 1178 snap 366 seq_num 1178) 2024-04-26T00:39:38.497 INFO:tasks.rados.rados.0.smithi045.stdout:3406: done (1 left) 2024-04-26T00:39:38.497 INFO:tasks.rados.rados.0.smithi045.stdout:3407: done (0 left) 2024-04-26T00:39:38.497 INFO:tasks.rados.rados.0.smithi045.stdout:3408: delete oid 37 current snap is 378 2024-04-26T00:39:38.510 INFO:tasks.rados.rados.0.smithi045.stdout:3408: done (0 left) 2024-04-26T00:39:38.511 INFO:tasks.rados.rados.0.smithi045.stdout:3409: read oid 19 snap -1 2024-04-26T00:39:38.511 INFO:tasks.rados.rados.0.smithi045.stdout:3409: expect (ObjNum 1205 snap 373 seq_num 1205) 2024-04-26T00:39:38.511 INFO:tasks.rados.rados.0.smithi045.stdout:3410: rollback oid 25 current snap is 378 2024-04-26T00:39:38.511 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 25 to 361 2024-04-26T00:39:38.511 INFO:tasks.rados.rados.0.smithi045.stdout:3411: write (excl) oid 43 current snap is 378 2024-04-26T00:39:38.511 INFO:tasks.rados.rados.0.smithi045.stdout:3411: seq_num 1214 ranges {403698=652413,1740434=663264} 2024-04-26T00:39:38.530 INFO:tasks.rados.rados.0.smithi045.stdout:3411: writing smithi04526579-43 from 403698 to 1056111 tid 1 2024-04-26T00:39:38.553 INFO:tasks.rados.rados.0.smithi045.stdout:3411: writing smithi04526579-43 from 1740434 to 2403698 tid 2 2024-04-26T00:39:38.553 INFO:tasks.rados.rados.0.smithi045.stdout:3412: copy_from oid 31 from oid 40 current snap is 378 2024-04-26T00:39:38.554 INFO:tasks.rados.rados.0.smithi045.stdout:3410: finishing rollback tid 0 to smithi04526579-25 2024-04-26T00:39:38.554 INFO:tasks.rados.rados.0.smithi045.stdout:3410: finishing rollback tid 1 to smithi04526579-25 2024-04-26T00:39:38.554 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 2591 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-04-26T00:39:38.554 INFO:tasks.rados.rados.0.smithi045.stdout:3409: expect (ObjNum 1205 snap 373 seq_num 1205) 2024-04-26T00:39:38.689 INFO:tasks.rados.rados.0.smithi045.stdout:3411: finishing write tid 1 to smithi04526579-43 2024-04-26T00:39:38.689 INFO:tasks.rados.rados.0.smithi045.stdout:3409: done (3 left) 2024-04-26T00:39:38.689 INFO:tasks.rados.rados.0.smithi045.stdout:3410: done (2 left) 2024-04-26T00:39:38.689 INFO:tasks.rados.rados.0.smithi045.stdout:3413: copy_from oid 46 from oid 1 current snap is 378 2024-04-26T00:39:38.689 INFO:tasks.rados.rados.0.smithi045.stdout:3411: finishing write tid 2 to smithi04526579-43 2024-04-26T00:39:38.689 INFO:tasks.rados.rados.0.smithi045.stdout:3411: finishing write tid 3 to smithi04526579-43 2024-04-26T00:39:38.689 INFO:tasks.rados.rados.0.smithi045.stdout:3411: finishing write tid 4 to smithi04526579-43 2024-04-26T00:39:38.689 INFO:tasks.rados.rados.0.smithi045.stdout:3411: oid 43 updating version 0 to 2419 2024-04-26T00:39:38.689 INFO:tasks.rados.rados.0.smithi045.stdout:3411: oid 43 updating version 2419 to 2420 2024-04-26T00:39:38.689 INFO:tasks.rados.rados.0.smithi045.stdout:3411: oid 43 updating version 2420 to 2421 2024-04-26T00:39:38.690 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 2421 (ObjNum 1214 snap 378 seq_num 1214) dirty exists 2024-04-26T00:39:38.690 INFO:tasks.rados.rados.0.smithi045.stdout:3411: left oid 43 (ObjNum 1214 snap 378 seq_num 1214) 2024-04-26T00:39:38.690 INFO:tasks.rados.rados.0.smithi045.stdout:3412: finishing copy_from racing read to smithi04526579-31 2024-04-26T00:39:38.690 INFO:tasks.rados.rados.0.smithi045.stdout:3412: finishing copy_from to smithi04526579-31 2024-04-26T00:39:38.690 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 2343 (ObjNum 1178 snap 366 seq_num 1178) dirty exists 2024-04-26T00:39:38.690 INFO:tasks.rados.rados.0.smithi045.stdout:3411: done (2 left) 2024-04-26T00:39:38.690 INFO:tasks.rados.rados.0.smithi045.stdout:3412: done (1 left) 2024-04-26T00:39:38.690 INFO:tasks.rados.rados.0.smithi045.stdout:3414: rollback oid 41 current snap is 378 2024-04-26T00:39:38.690 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 41 to 376 2024-04-26T00:39:38.690 INFO:tasks.rados.rados.0.smithi045.stdout:3415: rollback oid 15 current snap is 378 2024-04-26T00:39:38.690 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 15 to 346 2024-04-26T00:39:38.690 INFO:tasks.rados.rados.0.smithi045.stdout:3416: write (excl) oid 2 current snap is 378 2024-04-26T00:39:38.690 INFO:tasks.rados.rados.0.smithi045.stdout:3416: seq_num 1215 ranges {420505=727662,1798916=553091} 2024-04-26T00:39:38.695 INFO:tasks.rados.rados.0.smithi045.stdout:3416: writing smithi04526579-2 from 420505 to 1148167 tid 1 2024-04-26T00:39:38.702 INFO:tasks.rados.rados.0.smithi045.stdout:3416: writing smithi04526579-2 from 1798916 to 2352007 tid 2 2024-04-26T00:39:38.702 INFO:tasks.rados.rados.0.smithi045.stdout:3417: read oid 11 snap 360 2024-04-26T00:39:38.702 INFO:tasks.rados.rados.0.smithi045.stdout:3417: expect (ObjNum 1139 snap 345 seq_num 1139) 2024-04-26T00:39:38.702 INFO:tasks.rados.rados.0.smithi045.stdout:3418: write oid 13 current snap is 378 2024-04-26T00:39:38.703 INFO:tasks.rados.rados.0.smithi045.stdout:3418: seq_num 1216 ranges {437312=402911,1341045=696749} 2024-04-26T00:39:38.707 INFO:tasks.rados.rados.0.smithi045.stdout:3418: writing smithi04526579-13 from 437312 to 840223 tid 1 2024-04-26T00:39:38.717 INFO:tasks.rados.rados.0.smithi045.stdout:3418: writing smithi04526579-13 from 1341045 to 2037794 tid 2 2024-04-26T00:39:38.717 INFO:tasks.rados.rados.0.smithi045.stdout:3419: read oid 9 snap -1 2024-04-26T00:39:38.717 INFO:tasks.rados.rados.0.smithi045.stdout:3419: expect (ObjNum 1191 snap 370 seq_num 1191) 2024-04-26T00:39:38.717 INFO:tasks.rados.rados.0.smithi045.stdout:3414: finishing rollback tid 0 to smithi04526579-41 2024-04-26T00:39:38.717 INFO:tasks.rados.rados.0.smithi045.stdout:3414: finishing rollback tid 1 to smithi04526579-41 2024-04-26T00:39:38.717 INFO:tasks.rados.rados.0.smithi045.stdout:3413: finishing copy_from to smithi04526579-46 2024-04-26T00:39:38.717 INFO:tasks.rados.rados.0.smithi045.stdout:3413: got expected ENOENT (src dne) 2024-04-26T00:39:38.717 INFO:tasks.rados.rados.0.smithi045.stdout:3413: finishing copy_from racing read to smithi04526579-46 2024-04-26T00:39:38.717 INFO:tasks.rados.rados.0.smithi045.stdout:3414: finishing rollback tid 2 to smithi04526579-41 2024-04-26T00:39:38.717 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 2469 (ObjNum 1189 snap 369 seq_num 1189) dirty exists 2024-04-26T00:39:38.717 INFO:tasks.rados.rados.0.smithi045.stdout:3413: done (6 left) 2024-04-26T00:39:38.717 INFO:tasks.rados.rados.0.smithi045.stdout:3414: done (5 left) 2024-04-26T00:39:38.717 INFO:tasks.rados.rados.0.smithi045.stdout:3420: write oid 1 current snap is 378 2024-04-26T00:39:38.718 INFO:tasks.rados.rados.0.smithi045.stdout:3420: seq_num 1217 ranges {454119=594513,1515880=556760} 2024-04-26T00:39:38.723 INFO:tasks.rados.rados.0.smithi045.stdout:3420: writing smithi04526579-1 from 454119 to 1048632 tid 1 2024-04-26T00:39:38.730 INFO:tasks.rados.rados.0.smithi045.stdout:3420: writing smithi04526579-1 from 1515880 to 2072640 tid 2 2024-04-26T00:39:38.730 INFO:tasks.rados.rados.0.smithi045.stdout:3421: snap_create 2024-04-26T00:39:38.731 INFO:tasks.rados.rados.0.smithi045.stdout:3417: expect (ObjNum 1139 snap 345 seq_num 1139) 2024-04-26T00:39:38.931 INFO:tasks.rados.rados.0.smithi045.stdout:3416: finishing write tid 1 to smithi04526579-2 2024-04-26T00:39:38.931 INFO:tasks.rados.rados.0.smithi045.stdout:3416: finishing write tid 2 to smithi04526579-2 2024-04-26T00:39:38.931 INFO:tasks.rados.rados.0.smithi045.stdout:3416: finishing write tid 3 to smithi04526579-2 2024-04-26T00:39:38.931 INFO:tasks.rados.rados.0.smithi045.stdout:3416: finishing write tid 4 to smithi04526579-2 2024-04-26T00:39:38.931 INFO:tasks.rados.rados.0.smithi045.stdout:3416: oid 2 updating version 0 to 2499 2024-04-26T00:39:38.932 INFO:tasks.rados.rados.0.smithi045.stdout:3416: oid 2 version 2499 is already newer than 2498 2024-04-26T00:39:38.932 INFO:tasks.rados.rados.0.smithi045.stdout:3416: oid 2 updating version 2499 to 2500 2024-04-26T00:39:38.932 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 2500 (ObjNum 1215 snap 378 seq_num 1215) dirty exists 2024-04-26T00:39:38.932 INFO:tasks.rados.rados.0.smithi045.stdout:3416: left oid 2 (ObjNum 1215 snap 378 seq_num 1215) 2024-04-26T00:39:38.932 INFO:tasks.rados.rados.0.smithi045.stdout:3418: finishing write tid 1 to smithi04526579-13 2024-04-26T00:39:38.932 INFO:tasks.rados.rados.0.smithi045.stdout:3415: finishing rollback tid 0 to smithi04526579-15 2024-04-26T00:39:38.932 INFO:tasks.rados.rados.0.smithi045.stdout:3415: finishing rollback tid 1 to smithi04526579-15 2024-04-26T00:39:38.932 INFO:tasks.rados.rados.0.smithi045.stdout:3415: finishing rollback tid 2 to smithi04526579-15 2024-04-26T00:39:38.932 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 2472 (ObjNum 1139 snap 345 seq_num 1139) dirty exists 2024-04-26T00:39:38.932 INFO:tasks.rados.rados.0.smithi045.stdout:3419: expect (ObjNum 1191 snap 370 seq_num 1191) 2024-04-26T00:39:39.029 INFO:tasks.rados.rados.0.smithi045.stdout:3418: finishing write tid 2 to smithi04526579-13 2024-04-26T00:39:39.030 INFO:tasks.rados.rados.0.smithi045.stdout:3418: finishing write tid 3 to smithi04526579-13 2024-04-26T00:39:39.030 INFO:tasks.rados.rados.0.smithi045.stdout:3418: finishing write tid 4 to smithi04526579-13 2024-04-26T00:39:39.030 INFO:tasks.rados.rados.0.smithi045.stdout:3418: oid 13 updating version 0 to 2214 2024-04-26T00:39:39.030 INFO:tasks.rados.rados.0.smithi045.stdout:3418: oid 13 updating version 2214 to 2215 2024-04-26T00:39:39.030 INFO:tasks.rados.rados.0.smithi045.stdout:3418: oid 13 updating version 2215 to 2216 2024-04-26T00:39:39.030 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 2216 (ObjNum 1216 snap 378 seq_num 1216) dirty exists 2024-04-26T00:39:39.030 INFO:tasks.rados.rados.0.smithi045.stdout:3418: left oid 13 (ObjNum 1216 snap 378 seq_num 1216) 2024-04-26T00:39:39.030 INFO:tasks.rados.rados.0.smithi045.stdout:3420: finishing write tid 1 to smithi04526579-1 2024-04-26T00:39:39.030 INFO:tasks.rados.rados.0.smithi045.stdout:3420: finishing write tid 2 to smithi04526579-1 2024-04-26T00:39:39.030 INFO:tasks.rados.rados.0.smithi045.stdout:3420: finishing write tid 3 to smithi04526579-1 2024-04-26T00:39:39.030 INFO:tasks.rados.rados.0.smithi045.stdout:3420: finishing write tid 4 to smithi04526579-1 2024-04-26T00:39:39.030 INFO:tasks.rados.rados.0.smithi045.stdout:3420: oid 1 updating version 0 to 2473 2024-04-26T00:39:39.031 INFO:tasks.rados.rados.0.smithi045.stdout:3420: oid 1 updating version 2473 to 2475 2024-04-26T00:39:39.031 INFO:tasks.rados.rados.0.smithi045.stdout:3420: oid 1 version 2475 is already newer than 2474 2024-04-26T00:39:39.031 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 2475 (ObjNum 1217 snap 378 seq_num 1217) dirty exists 2024-04-26T00:39:39.031 INFO:tasks.rados.rados.0.smithi045.stdout:3420: left oid 1 (ObjNum 1217 snap 378 seq_num 1217) 2024-04-26T00:39:39.461 INFO:tasks.rados.rados.0.smithi045.stdout:3415: done (6 left) 2024-04-26T00:39:39.461 INFO:tasks.rados.rados.0.smithi045.stdout:3416: done (5 left) 2024-04-26T00:39:39.461 INFO:tasks.rados.rados.0.smithi045.stdout:3417: done (4 left) 2024-04-26T00:39:39.462 INFO:tasks.rados.rados.0.smithi045.stdout:3418: done (3 left) 2024-04-26T00:39:39.462 INFO:tasks.rados.rados.0.smithi045.stdout:3419: done (2 left) 2024-04-26T00:39:39.462 INFO:tasks.rados.rados.0.smithi045.stdout:3420: done (1 left) 2024-04-26T00:39:39.462 INFO:tasks.rados.rados.0.smithi045.stdout:3421: done (0 left) 2024-04-26T00:39:39.462 INFO:tasks.rados.rados.0.smithi045.stdout:3422: read oid 37 snap -1 2024-04-26T00:39:39.462 INFO:tasks.rados.rados.0.smithi045.stdout:3422: expect deleted 2024-04-26T00:39:39.462 INFO:tasks.rados.rados.0.smithi045.stdout:3423: rollback oid 38 current snap is 379 2024-04-26T00:39:39.462 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 38 to 332 2024-04-26T00:39:39.462 INFO:tasks.rados.rados.0.smithi045.stdout:3424: snap_remove snap 347 2024-04-26T00:39:40.475 INFO:tasks.rados.rados.0.smithi045.stdout:3424: done (2 left) 2024-04-26T00:39:40.475 INFO:tasks.rados.rados.0.smithi045.stdout:3425: snap_create 2024-04-26T00:39:40.475 INFO:tasks.rados.rados.0.smithi045.stdout:3423: finishing rollback tid 0 to smithi04526579-38 2024-04-26T00:39:40.476 INFO:tasks.rados.rados.0.smithi045.stdout:3423: finishing rollback tid 1 to smithi04526579-38 2024-04-26T00:39:40.476 INFO:tasks.rados.rados.0.smithi045.stdout:3423: finishing rollback tid 2 to smithi04526579-38 2024-04-26T00:39:40.476 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 2398 (ObjNum 1075 snap 322 seq_num 1075) dirty exists 2024-04-26T00:39:41.478 INFO:tasks.rados.rados.0.smithi045.stdout:3422: done (2 left) 2024-04-26T00:39:41.478 INFO:tasks.rados.rados.0.smithi045.stdout:3423: done (1 left) 2024-04-26T00:39:41.478 INFO:tasks.rados.rados.0.smithi045.stdout:3425: done (0 left) 2024-04-26T00:39:41.479 INFO:tasks.rados.rados.0.smithi045.stdout:3426: snap_create 2024-04-26T00:39:42.555 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-04-26T00:39:42.556 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-04-26T00:39:42.556 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-26T00:39:42.556 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-26T00:39:42.559 INFO:tasks.ceph.mon.a:Started 2024-04-26T00:39:42.559 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:39:42.559 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:39:42.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.678+0000 7efe69b35640 10 monclient: get_monmap_and_config 2024-04-26T00:39:42.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.678+0000 7efe69b35640 10 monclient: build_initial_monmap 2024-04-26T00:39:42.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.678+0000 7efe69b35640 10 monclient: monmap: 2024-04-26T00:39:42.680 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:42.680 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:42.680 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:39:42.682998+0000 2024-04-26T00:39:42.680 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:39:42.682998+0000 2024-04-26T00:39:42.680 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:42.680 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:42.680 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:39:42.680 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:39:42.680 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:39:42.681 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:39:42.681 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:39:42.681 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:42.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.678+0000 7efe69b35640 1 Processor -- start 2024-04-26T00:39:42.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.678+0000 7efe69b35640 1 -- start start 2024-04-26T00:39:42.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.678+0000 7efe69b35640 10 monclient: init 2024-04-26T00:39:42.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe69b35640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:42.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe69b35640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-04-26T00:39:42.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe69b35640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7efe640a04e0 0x7efe640a13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:42.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe69b35640 10 monclient(hunting): picked mon.noname-a con 0x7efe640a04e0 addr v2:172.21.15.37:3300/0 2024-04-26T00:39:42.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe69b35640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efe640a08c0 addr v1:172.21.15.45:6789/0 2024-04-26T00:39:42.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe69b35640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efe6406c9f0 0x7efe6413ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:42.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe69b35640 10 monclient(hunting): picked mon.noname-b con 0x7efe6406c9f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:42.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe69b35640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe6409f2f0 con 0x7efe640a08c0 2024-04-26T00:39:42.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe69b35640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:42.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe69b35640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7efe6409e910 con 0x7efe640a04e0 2024-04-26T00:39:42.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe69b35640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:42.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe69b35640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7efe640a3730 con 0x7efe6406c9f0 2024-04-26T00:39:42.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe69b35640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:42.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe69b35640 10 monclient(hunting): authenticate will time out at 2163.988333s 2024-04-26T00:39:42.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe637fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7efe640a04e0 0x7efe640a13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:42.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe637fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7efe640a04e0 0x7efe640a13d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:41662/0 (socket says 172.21.15.37:41662) 2024-04-26T00:39:42.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe637fe640 1 -- 172.21.15.37:0/2328653865 learned_addr learned my addr 172.21.15.37:0/2328653865 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:42.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe637fe640 10 monclient(hunting): get_auth_request con 0x7efe640a04e0 auth_method 0 2024-04-26T00:39:42.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:42.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe637fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:42.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe637fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:42.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe63fff640 1 --2- 172.21.15.37:0/2328653865 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efe6406c9f0 0x7efe6413ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:42.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe63fff640 10 monclient(hunting): get_auth_request con 0x7efe6406c9f0 auth_method 0 2024-04-26T00:39:42.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:42.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe63fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:42.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe63fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:42.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe637fe640 1 -- 172.21.15.37:0/2328653865 >> v2:172.21.15.37:3300/0 conn(0x7efe640a04e0 msgr2=0x7efe640a13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:39:42.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe637fe640 1 -- 172.21.15.37:0/2328653865 >> v2:172.21.15.37:3300/0 conn(0x7efe640a04e0 msgr2=0x7efe640a13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:39:42.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe637fe640 1 --2- 172.21.15.37:0/2328653865 >> v2:172.21.15.37:3300/0 conn(0x7efe640a04e0 0x7efe640a13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:39:42.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe637fe640 1 --2- 172.21.15.37:0/2328653865 >> v2:172.21.15.37:3300/0 conn(0x7efe640a04e0 0x7efe640a13d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:39:42.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe63fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:42.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe63fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:42.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe63fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:42.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe63fff640 10 monclient(hunting): handle_auth_done global_id 204476 payload 274 2024-04-26T00:39:42.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe63fff640 1 -- 172.21.15.37:0/2328653865 >> v2:172.21.15.37:3300/0 conn(0x7efe640a04e0 msgr2=0x7efe640a13d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:39:42.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe63fff640 1 --2- 172.21.15.37:0/2328653865 >> v2:172.21.15.37:3300/0 conn(0x7efe640a04e0 0x7efe640a13d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:42.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe63fff640 1 -- 172.21.15.37:0/2328653865 >> v1:172.21.15.45:6789/0 conn(0x7efe640a08c0 legacy=0x7efe6406c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:42.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe63fff640 10 monclient: _finish_hunting 0 2024-04-26T00:39:42.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe63fff640 1 monclient: found mon.noname-b 2024-04-26T00:39:42.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe63fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:42.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe63fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:39:42.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.682+0000 7efe63fff640 1 -- 172.21.15.37:0/2328653865 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe640a34f0 con 0x7efe6406c9f0 2024-04-26T00:39:42.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe63fff640 1 -- 172.21.15.37:0/2328653865 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efe6406c9f0 msgr2=0x7efe6413ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:42.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe63fff640 1 --2- 172.21.15.37:0/2328653865 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efe6406c9f0 0x7efe6413ee70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7efe5c009d90 tx=0x7efe5c01c610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13da76160793ba60 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:42.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe627fc640 1 -- 172.21.15.37:0/2328653865 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe5c0a6040 con 0x7efe6406c9f0 2024-04-26T00:39:42.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:42.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe627fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:39:42.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe627fc640 10 monclient: dump: 2024-04-26T00:39:42.698 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:42.698 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:42.698 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:42.698 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:42.698 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:42.699 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:42.699 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:42.699 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:42.699 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:42.699 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:42.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe627fc640 10 monclient: _finish_auth 0 2024-04-26T00:39:42.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe627fc640 10 monclient: _check_auth_tickets 2024-04-26T00:39:42.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:42.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe627fc640 1 -- 172.21.15.37:0/2328653865 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe5c0af070 con 0x7efe6406c9f0 2024-04-26T00:39:42.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe627fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:42.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe627fc640 1 -- 172.21.15.37:0/2328653865 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe5c0aa860 con 0x7efe6406c9f0 2024-04-26T00:39:42.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe69b35640 5 monclient: authenticate success, global_id 204476 2024-04-26T00:39:42.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe69b35640 10 monclient: get_monmap_and_config success 2024-04-26T00:39:42.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe69b35640 4 set_mon_vals no callback set 2024-04-26T00:39:42.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:42.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:42.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe627fc640 10 monclient: dump: 2024-04-26T00:39:42.700 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:42.700 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:42.700 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:42.700 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:42.700 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:42.700 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:42.700 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:42.700 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:42.700 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:42.700 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:42.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe69b35640 10 monclient: shutdown 2024-04-26T00:39:42.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe69b35640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:42.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe69b35640 1 -- 172.21.15.37:0/2328653865 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efe6406c9f0 msgr2=0x7efe6413ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:42.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe69b35640 1 --2- 172.21.15.37:0/2328653865 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efe6406c9f0 0x7efe6413ee70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7efe5c009d90 tx=0x7efe5c01c610 comp rx=0 tx=0).stop 2024-04-26T00:39:42.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe69b35640 1 -- 172.21.15.37:0/2328653865 shutdown_connections 2024-04-26T00:39:42.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe69b35640 1 --2- 172.21.15.37:0/2328653865 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efe6406c9f0 0x7efe6413ee70 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:42.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe69b35640 1 --2- 172.21.15.37:0/2328653865 >> v2:172.21.15.37:3300/0 conn(0x7efe640a04e0 0x7efe640a13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:42.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe69b35640 1 -- 172.21.15.37:0/2328653865 >> 172.21.15.37:0/2328653865 conn(0x7efe640a4170 msgr2=0x7efe640a4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:42.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe69b35640 1 -- 172.21.15.37:0/2328653865 shutdown_connections 2024-04-26T00:39:42.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe69b35640 1 -- 172.21.15.37:0/2328653865 wait complete. 2024-04-26T00:39:42.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe69b35640 10 monclient: build_initial_monmap 2024-04-26T00:39:42.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe69b35640 10 monclient: monmap: 2024-04-26T00:39:42.701 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:42.702 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:39:42.702 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:39:42.702 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:39:42.702 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:42.702 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:39:42.702 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:39:42.702 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:39:42.702 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:39:42.702 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:42.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe69b35640 1 Processor -- start 2024-04-26T00:39:42.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe69b35640 1 -- start start 2024-04-26T00:39:42.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe69b35640 10 monclient: init 2024-04-26T00:39:42.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe69b35640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:42.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe69b35640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:39:42.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe69b35640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe6406c9f0 0x7efe641cf030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:42.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe69b35640 10 monclient(hunting): picked mon.noname-b con 0x7efe6406c9f0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:39:42.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe69b35640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efe640a04e0 0x7efe6413ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:42.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe69b35640 10 monclient(hunting): picked mon.noname-c con 0x7efe640a04e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:42.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe69b35640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efe640a08c0 0x7efe64121cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:42.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe69b35640 10 monclient(hunting): picked mon.noname-a con 0x7efe640a08c0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:39:42.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe69b35640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:42.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe69b35640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7efe64141d50 con 0x7efe640a08c0 2024-04-26T00:39:42.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe69b35640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:42.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe69b35640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7efe64141bd0 con 0x7efe6406c9f0 2024-04-26T00:39:42.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe69b35640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:42.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe69b35640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7efe64141ed0 con 0x7efe640a04e0 2024-04-26T00:39:42.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe69b35640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:42.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe69b35640 10 monclient(hunting): authenticate will time out at 2163.995284s 2024-04-26T00:39:42.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe63fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efe640a08c0 0x7efe64121cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:42.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe62ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efe640a04e0 0x7efe6413ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:42.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe63fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efe640a08c0 0x7efe64121cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:41670/0 (socket says 172.21.15.37:41670) 2024-04-26T00:39:42.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe62ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efe640a04e0 0x7efe6413ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:38588/0 (socket says 172.21.15.37:38588) 2024-04-26T00:39:42.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe63fff640 1 -- 172.21.15.37:0/2980666911 learned_addr learned my addr 172.21.15.37:0/2980666911 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:42.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe63fff640 10 monclient(hunting): get_auth_request con 0x7efe640a08c0 auth_method 0 2024-04-26T00:39:42.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:42.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe63fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:42.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe63fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:42.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe62ffd640 10 monclient(hunting): get_auth_request con 0x7efe640a04e0 auth_method 0 2024-04-26T00:39:42.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe62ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:42.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe62ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:42.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:42.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe63fff640 1 -- 172.21.15.37:0/2980666911 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efe640a08c0 msgr2=0x7efe64121cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:39:42.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe62ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:42.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe63fff640 1 -- 172.21.15.37:0/2980666911 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efe640a08c0 msgr2=0x7efe64121cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:39:42.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe62ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:42.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe63fff640 1 --2- 172.21.15.37:0/2980666911 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efe640a08c0 0x7efe64121cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:39:42.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe62ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:42.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe63fff640 1 --2- 172.21.15.37:0/2980666911 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efe640a08c0 0x7efe64121cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:39:42.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe637fe640 1 --2- 172.21.15.37:0/2980666911 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe6406c9f0 0x7efe641cf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:42.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe637fe640 10 monclient(hunting): get_auth_request con 0x7efe6406c9f0 auth_method 0 2024-04-26T00:39:42.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:42.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe637fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:42.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe637fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:42.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe62ffd640 10 monclient(hunting): handle_auth_done global_id 204479 payload 824 2024-04-26T00:39:42.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe62ffd640 1 -- 172.21.15.37:0/2980666911 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe6406c9f0 msgr2=0x7efe641cf030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:42.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe62ffd640 1 --2- 172.21.15.37:0/2980666911 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe6406c9f0 0x7efe641cf030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:42.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe62ffd640 1 -- 172.21.15.37:0/2980666911 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efe640a08c0 msgr2=0x7efe64121cd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:39:42.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe62ffd640 1 --2- 172.21.15.37:0/2980666911 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efe640a08c0 0x7efe64121cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:42.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe62ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:39:42.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe62ffd640 1 monclient: found mon.noname-c 2024-04-26T00:39:42.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe62ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:42.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe62ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:39:42.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.686+0000 7efe62ffd640 1 -- 172.21.15.37:0/2980666911 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe64123d80 con 0x7efe640a04e0 2024-04-26T00:39:42.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.690+0000 7efe62ffd640 1 -- 172.21.15.37:0/2980666911 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efe640a04e0 msgr2=0x7efe6413ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:42.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.690+0000 7efe62ffd640 1 --2- 172.21.15.37:0/2980666911 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efe640a04e0 0x7efe6413ae00 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7efe5400c410 tx=0x7efe5400c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:42.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.690+0000 7efe5bfff640 1 -- 172.21.15.37:0/2980666911 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe54017070 con 0x7efe640a04e0 2024-04-26T00:39:42.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.690+0000 7efe5bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:42.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.690+0000 7efe5bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:39:42.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.690+0000 7efe5bfff640 10 monclient: dump: 2024-04-26T00:39:42.706 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:42.706 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:42.706 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:42.707 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:42.707 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:42.707 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:42.707 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:42.707 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:42.707 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:42.707 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:42.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.690+0000 7efe5bfff640 10 monclient: _finish_auth 0 2024-04-26T00:39:42.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.690+0000 7efe5bfff640 10 monclient: _check_auth_tickets 2024-04-26T00:39:42.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.690+0000 7efe5bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:42.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.690+0000 7efe69b35640 5 monclient: authenticate success, global_id 204479 2024-04-26T00:39:42.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.690+0000 7efe5bfff640 1 -- 172.21.15.37:0/2980666911 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe540069e0 con 0x7efe640a04e0 2024-04-26T00:39:42.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.690+0000 7efe69b35640 10 monclient: _renew_subs 2024-04-26T00:39:42.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.690+0000 7efe69b35640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:42.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.690+0000 7efe69b35640 1 -- 172.21.15.37:0/2980666911 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe6409ddd0 con 0x7efe640a04e0 2024-04-26T00:39:42.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.690+0000 7efe5bfff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:42.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.690+0000 7efe5bfff640 1 -- 172.21.15.37:0/2980666911 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe54006dd0 con 0x7efe640a04e0 2024-04-26T00:39:42.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.690+0000 7efe5bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:42.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.690+0000 7efe5bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:42.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.690+0000 7efe5bfff640 10 monclient: dump: 2024-04-26T00:39:42.708 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:42.708 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:42.708 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:42.708 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:42.709 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:42.709 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:42.709 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:42.709 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:42.709 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:42.709 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:42.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.690+0000 7efe61ffb640 4 set_mon_vals no callback set 2024-04-26T00:39:42.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.690+0000 7efe69b35640 10 monclient: _renew_subs 2024-04-26T00:39:42.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.690+0000 7efe69b35640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:42.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.690+0000 7efe69b35640 1 -- 172.21.15.37:0/2980666911 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efe64124ef0 con 0x7efe640a04e0 2024-04-26T00:39:42.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.690+0000 7efe5bfff640 1 -- 172.21.15.37:0/2980666911 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7efe5401a070 con 0x7efe640a04e0 2024-04-26T00:39:42.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.690+0000 7efe69b35640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:42.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.690+0000 7efe69b35640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:42.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.690+0000 7efe69b35640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:42.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.690+0000 7efe69b35640 1 -- 172.21.15.37:0/2980666911 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe64066f40 con 0x7efe640a04e0 2024-04-26T00:39:42.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.690+0000 7efe5bfff640 1 --2- 172.21.15.37:0/2980666911 >> v2:172.21.15.45:6812/2379918623 conn(0x7efe44035eb0 0x7efe44038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:42.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.690+0000 7efe5bfff640 1 -- 172.21.15.37:0/2980666911 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(779..779 src has 705..779) ==== 3915+0+0 (secure 0 0 0) 0x7efe54047e80 con 0x7efe640a04e0 2024-04-26T00:39:42.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.690+0000 7efe637fe640 1 --2- 172.21.15.37:0/2980666911 >> v2:172.21.15.45:6812/2379918623 conn(0x7efe44035eb0 0x7efe44038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:42.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.690+0000 7efe637fe640 10 monclient: get_auth_request con 0x7efe44035eb0 auth_method 0 2024-04-26T00:39:42.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.694+0000 7efe637fe640 1 --2- 172.21.15.37:0/2980666911 >> v2:172.21.15.45:6812/2379918623 conn(0x7efe44035eb0 0x7efe44038370 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7efe64123820 tx=0x7efe4c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:42.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.698+0000 7efe5bfff640 1 -- 172.21.15.37:0/2980666911 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7efe54046be0 con 0x7efe640a04e0 2024-04-26T00:39:42.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.698+0000 7efe5bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:42.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.698+0000 7efe5bfff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:39:42.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.918+0000 7efe69b35640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:39:42.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.918+0000 7efe69b35640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:39:42.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.918+0000 7efe69b35640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:42.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.918+0000 7efe69b35640 1 -- 172.21.15.37:0/2980666911 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7efe6413b340 con 0x7efe640a04e0 2024-04-26T00:39:42.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.922+0000 7efe5bfff640 1 -- 172.21.15.37:0/2980666911 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7efe5401a970 con 0x7efe640a04e0 2024-04-26T00:39:42.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.922+0000 7efe5bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:39:42.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.922+0000 7efe5bfff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:39:42.921 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":146,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:39:42.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.922+0000 7efe69b35640 1 -- 172.21.15.37:0/2980666911 >> v2:172.21.15.45:6812/2379918623 conn(0x7efe44035eb0 msgr2=0x7efe44038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:42.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.922+0000 7efe69b35640 1 --2- 172.21.15.37:0/2980666911 >> v2:172.21.15.45:6812/2379918623 conn(0x7efe44035eb0 0x7efe44038370 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7efe64123820 tx=0x7efe4c008040 comp rx=0 tx=0).stop 2024-04-26T00:39:42.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.922+0000 7efe69b35640 10 monclient: shutdown 2024-04-26T00:39:42.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.922+0000 7efe69b35640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:42.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.922+0000 7efe69b35640 1 -- 172.21.15.37:0/2980666911 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efe640a04e0 msgr2=0x7efe6413ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:42.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.922+0000 7efe69b35640 1 --2- 172.21.15.37:0/2980666911 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efe640a04e0 0x7efe6413ae00 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7efe5400c410 tx=0x7efe5400c8d0 comp rx=0 tx=0).stop 2024-04-26T00:39:42.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.922+0000 7efe69b35640 1 -- 172.21.15.37:0/2980666911 shutdown_connections 2024-04-26T00:39:42.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.922+0000 7efe69b35640 1 --2- 172.21.15.37:0/2980666911 >> v2:172.21.15.45:6812/2379918623 conn(0x7efe44035eb0 0x7efe44038370 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:42.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.922+0000 7efe69b35640 1 --2- 172.21.15.37:0/2980666911 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efe640a08c0 0x7efe64121cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:42.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.922+0000 7efe69b35640 1 --2- 172.21.15.37:0/2980666911 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efe640a04e0 0x7efe6413ae00 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:42.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.922+0000 7efe69b35640 1 --2- 172.21.15.37:0/2980666911 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe6406c9f0 0x7efe641cf030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:42.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.922+0000 7efe69b35640 1 -- 172.21.15.37:0/2980666911 >> 172.21.15.37:0/2980666911 conn(0x7efe640a4170 msgr2=0x7efe640a6590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:42.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.922+0000 7efe69b35640 1 -- 172.21.15.37:0/2980666911 shutdown_connections 2024-04-26T00:39:42.924 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:42.922+0000 7efe69b35640 1 -- 172.21.15.37:0/2980666911 wait complete. 2024-04-26T00:39:43.016 INFO:tasks.rados.rados.0.smithi045.stdout:3426: done (0 left) 2024-04-26T00:39:43.016 INFO:tasks.rados.rados.0.smithi045.stdout:3427: read oid 33 snap -1 2024-04-26T00:39:43.016 INFO:tasks.rados.rados.0.smithi045.stdout:3427: expect (ObjNum 1145 snap 348 seq_num 1145) 2024-04-26T00:39:43.016 INFO:tasks.rados.rados.0.smithi045.stdout:3428: snap_create 2024-04-26T00:39:43.024 INFO:tasks.rados.rados.0.smithi045.stdout:3427: expect (ObjNum 1145 snap 348 seq_num 1145) 2024-04-26T00:39:44.441 INFO:tasks.rados.rados.0.smithi045.stdout:3427: done (1 left) 2024-04-26T00:39:44.441 INFO:tasks.rados.rados.0.smithi045.stdout:3428: done (0 left) 2024-04-26T00:39:44.442 INFO:tasks.rados.rados.0.smithi045.stdout:3429: delete oid 34 current snap is 382 2024-04-26T00:39:44.449 INFO:tasks.rados.rados.0.smithi045.stdout:3429: done (0 left) 2024-04-26T00:39:44.449 INFO:tasks.rados.rados.0.smithi045.stdout:3430: snap_remove snap 360 2024-04-26T00:39:44.463 INFO:tasks.rados.rados.0.smithi045.stdout:3430: done (0 left) 2024-04-26T00:39:44.463 INFO:tasks.rados.rados.0.smithi045.stdout:3431: delete oid 32 current snap is 382 2024-04-26T00:39:44.497 INFO:tasks.rados.rados.0.smithi045.stdout:3431: done (0 left) 2024-04-26T00:39:44.497 INFO:tasks.rados.rados.0.smithi045.stdout:3432: read oid 30 snap -1 2024-04-26T00:39:44.497 INFO:tasks.rados.rados.0.smithi045.stdout:3432: expect (ObjNum 1155 snap 355 seq_num 1155) 2024-04-26T00:39:44.497 INFO:tasks.rados.rados.0.smithi045.stdout:3433: write (excl) oid 3 current snap is 382 2024-04-26T00:39:44.497 INFO:tasks.rados.rados.0.smithi045.stdout:3433: seq_num 1220 ranges {504540=420260,1574973=704087} 2024-04-26T00:39:44.514 INFO:tasks.rados.rados.0.smithi045.stdout:3433: writing smithi04526579-3 from 504540 to 924800 tid 1 2024-04-26T00:39:44.537 INFO:tasks.rados.rados.0.smithi045.stdout:3433: writing smithi04526579-3 from 1574973 to 2279060 tid 2 2024-04-26T00:39:44.537 INFO:tasks.rados.rados.0.smithi045.stdout:3434: read oid 49 snap -1 2024-04-26T00:39:44.537 INFO:tasks.rados.rados.0.smithi045.stdout:3434: expect (ObjNum 1162 snap 359 seq_num 1162) 2024-04-26T00:39:44.537 INFO:tasks.rados.rados.0.smithi045.stdout:3435: write (excl) oid 8 current snap is 382 2024-04-26T00:39:44.537 INFO:tasks.rados.rados.0.smithi045.stdout:3435: seq_num 1221 ranges {521347=495509,1633455=564098} 2024-04-26T00:39:44.551 INFO:tasks.rados.rados.0.smithi045.stdout:3435: writing smithi04526579-8 from 521347 to 1016856 tid 1 2024-04-26T00:39:44.566 INFO:tasks.rados.rados.0.smithi045.stdout:3435: writing smithi04526579-8 from 1633455 to 2197553 tid 2 2024-04-26T00:39:44.566 INFO:tasks.rados.rados.0.smithi045.stdout:3436: read oid 41 snap -1 2024-04-26T00:39:44.566 INFO:tasks.rados.rados.0.smithi045.stdout:3436: expect (ObjNum 1189 snap 369 seq_num 1189) 2024-04-26T00:39:44.566 INFO:tasks.rados.rados.0.smithi045.stdout:3437: read oid 37 snap 380 2024-04-26T00:39:44.566 INFO:tasks.rados.rados.0.smithi045.stdout:3437: expect deleted 2024-04-26T00:39:44.566 INFO:tasks.rados.rados.0.smithi045.stdout:3438: read oid 13 snap -1 2024-04-26T00:39:44.566 INFO:tasks.rados.rados.0.smithi045.stdout:3438: expect (ObjNum 1216 snap 378 seq_num 1216) 2024-04-26T00:39:44.566 INFO:tasks.rados.rados.0.smithi045.stdout:3439: snap_remove snap 337 2024-04-26T00:39:45.522 INFO:tasks.rados.rados.0.smithi045.stdout:3439: done (7 left) 2024-04-26T00:39:45.522 INFO:tasks.rados.rados.0.smithi045.stdout:3440: copy_from oid 32 from oid 11 current snap is 382 2024-04-26T00:39:45.522 INFO:tasks.rados.rados.0.smithi045.stdout:3432: expect (ObjNum 1155 snap 355 seq_num 1155) 2024-04-26T00:39:45.755 INFO:tasks.rados.rados.0.smithi045.stdout:3433: finishing write tid 1 to smithi04526579-3 2024-04-26T00:39:45.755 INFO:tasks.rados.rados.0.smithi045.stdout:3433: finishing write tid 2 to smithi04526579-3 2024-04-26T00:39:45.755 INFO:tasks.rados.rados.0.smithi045.stdout:3433: finishing write tid 3 to smithi04526579-3 2024-04-26T00:39:45.755 INFO:tasks.rados.rados.0.smithi045.stdout:3433: finishing write tid 4 to smithi04526579-3 2024-04-26T00:39:45.755 INFO:tasks.rados.rados.0.smithi045.stdout:3433: oid 3 updating version 0 to 2466 2024-04-26T00:39:45.755 INFO:tasks.rados.rados.0.smithi045.stdout:3433: oid 3 version 2466 is already newer than 2465 2024-04-26T00:39:45.755 INFO:tasks.rados.rados.0.smithi045.stdout:3433: oid 3 version 2466 is already newer than 2464 2024-04-26T00:39:45.755 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 2466 (ObjNum 1220 snap 382 seq_num 1220) dirty exists 2024-04-26T00:39:45.755 INFO:tasks.rados.rados.0.smithi045.stdout:3433: left oid 3 (ObjNum 1220 snap 382 seq_num 1220) 2024-04-26T00:39:45.755 INFO:tasks.rados.rados.0.smithi045.stdout:3435: finishing write tid 1 to smithi04526579-8 2024-04-26T00:39:45.755 INFO:tasks.rados.rados.0.smithi045.stdout:3434: expect (ObjNum 1162 snap 359 seq_num 1162) 2024-04-26T00:39:45.868 INFO:tasks.rados.rados.0.smithi045.stdout:3436: expect (ObjNum 1189 snap 369 seq_num 1189) 2024-04-26T00:39:45.940 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:39:46.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e46b5640 10 monclient: get_monmap_and_config 2024-04-26T00:39:46.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e46b5640 10 monclient: build_initial_monmap 2024-04-26T00:39:46.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e46b5640 10 monclient: monmap: 2024-04-26T00:39:46.070 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:46.071 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:46.071 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:39:46.073526+0000 2024-04-26T00:39:46.071 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:39:46.073526+0000 2024-04-26T00:39:46.071 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:46.071 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:46.071 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:39:46.071 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:39:46.071 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:39:46.071 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:39:46.071 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:39:46.071 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:46.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e46b5640 1 Processor -- start 2024-04-26T00:39:46.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e46b5640 1 -- start start 2024-04-26T00:39:46.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e46b5640 10 monclient: init 2024-04-26T00:39:46.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e46b5640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:46.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e46b5640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-04-26T00:39:46.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e46b5640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60dc0a0260 0x7f60dc0a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:46.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e46b5640 10 monclient(hunting): picked mon.noname-b con 0x7f60dc0a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:46.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e46b5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f60dc134b30 addr v1:172.21.15.45:6789/0 2024-04-26T00:39:46.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e46b5640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f60dc135660 0x7f60dc13ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:46.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e46b5640 10 monclient(hunting): picked mon.noname-a con 0x7f60dc135660 addr v2:172.21.15.37:3300/0 2024-04-26T00:39:46.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e46b5640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60dc0a1130 con 0x7f60dc134b30 2024-04-26T00:39:46.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e46b5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:46.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e46b5640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f60dc09ec10 con 0x7f60dc135660 2024-04-26T00:39:46.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e46b5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:46.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e46b5640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f60dc09f0f0 con 0x7f60dc0a0260 2024-04-26T00:39:46.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e46b5640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:46.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e46b5640 10 monclient(hunting): authenticate will time out at 2167.379248s 2024-04-26T00:39:46.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e2451640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60dc0a0260 0x7f60dc0a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:46.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e2451640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60dc0a0260 0x7f60dc0a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:38616/0 (socket says 172.21.15.37:38616) 2024-04-26T00:39:46.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e2451640 1 -- 172.21.15.37:0/2822146354 learned_addr learned my addr 172.21.15.37:0/2822146354 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:46.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e2451640 10 monclient(hunting): get_auth_request con 0x7f60dc0a0260 auth_method 0 2024-04-26T00:39:46.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e2451640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:46.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e2451640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:46.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e2451640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:46.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e2c52640 1 --2- 172.21.15.37:0/2822146354 >> v2:172.21.15.37:3300/0 conn(0x7f60dc135660 0x7f60dc13ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:46.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e2451640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:46.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e2451640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:46.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e2451640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:46.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e2c52640 10 monclient(hunting): get_auth_request con 0x7f60dc135660 auth_method 0 2024-04-26T00:39:46.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e2c52640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:46.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e2c52640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:46.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e2c52640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:46.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e2c52640 1 -- 172.21.15.37:0/2822146354 >> v2:172.21.15.37:3300/0 conn(0x7f60dc135660 msgr2=0x7f60dc13ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:39:46.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e2c52640 1 -- 172.21.15.37:0/2822146354 >> v2:172.21.15.37:3300/0 conn(0x7f60dc135660 msgr2=0x7f60dc13ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:39:46.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e2c52640 1 --2- 172.21.15.37:0/2822146354 >> v2:172.21.15.37:3300/0 conn(0x7f60dc135660 0x7f60dc13ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:39:46.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e2451640 10 monclient(hunting): handle_auth_done global_id 204482 payload 274 2024-04-26T00:39:46.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e2451640 1 -- 172.21.15.37:0/2822146354 >> v2:172.21.15.37:3300/0 conn(0x7f60dc135660 msgr2=0x7f60dc13ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:46.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e2c52640 1 --2- 172.21.15.37:0/2822146354 >> v2:172.21.15.37:3300/0 conn(0x7f60dc135660 0x7f60dc13ce70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:39:46.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e2451640 1 --2- 172.21.15.37:0/2822146354 >> v2:172.21.15.37:3300/0 conn(0x7f60dc135660 0x7f60dc13ce70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:46.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e2451640 1 -- 172.21.15.37:0/2822146354 >> v1:172.21.15.45:6789/0 conn(0x7f60dc134b30 legacy=0x7f60dc134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:46.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e2451640 10 monclient: _finish_hunting 0 2024-04-26T00:39:46.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e2451640 1 monclient: found mon.noname-b 2024-04-26T00:39:46.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e2451640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:46.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e2451640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:39:46.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.070+0000 7f60e2451640 1 -- 172.21.15.37:0/2822146354 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60dc06cff0 con 0x7f60dc0a0260 2024-04-26T00:39:46.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e2451640 1 -- 172.21.15.37:0/2822146354 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60dc0a0260 msgr2=0x7f60dc0a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:46.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e2451640 1 --2- 172.21.15.37:0/2822146354 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60dc0a0260 0x7f60dc0a0640 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f60d8009870 tx=0x7f60d801c200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=881e74636e5ee66e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:46.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e144f640 1 -- 172.21.15.37:0/2822146354 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60d80b0070 con 0x7f60dc0a0260 2024-04-26T00:39:46.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e144f640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:46.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e144f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:39:46.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e144f640 10 monclient: dump: 2024-04-26T00:39:46.084 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:46.084 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:46.084 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:46.084 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:46.084 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:46.084 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:46.084 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:46.084 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:46.085 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:46.085 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:46.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e144f640 10 monclient: _finish_auth 0 2024-04-26T00:39:46.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e144f640 10 monclient: _check_auth_tickets 2024-04-26T00:39:46.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e144f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:46.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e144f640 1 -- 172.21.15.37:0/2822146354 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60d80ab600 con 0x7f60dc0a0260 2024-04-26T00:39:46.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e46b5640 5 monclient: authenticate success, global_id 204482 2024-04-26T00:39:46.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e46b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:39:46.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e144f640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:46.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e144f640 1 -- 172.21.15.37:0/2822146354 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60d80ab9d0 con 0x7f60dc0a0260 2024-04-26T00:39:46.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e144f640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:46.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e144f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:46.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e144f640 10 monclient: dump: 2024-04-26T00:39:46.085 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:46.086 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:46.086 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:46.086 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:46.086 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:46.086 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:46.086 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:46.086 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:46.086 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:46.086 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:46.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e46b5640 10 monclient: get_monmap_and_config success 2024-04-26T00:39:46.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e46b5640 4 set_mon_vals no callback set 2024-04-26T00:39:46.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e46b5640 10 monclient: shutdown 2024-04-26T00:39:46.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e46b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:46.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e46b5640 1 -- 172.21.15.37:0/2822146354 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60dc0a0260 msgr2=0x7f60dc0a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:46.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e46b5640 1 --2- 172.21.15.37:0/2822146354 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60dc0a0260 0x7f60dc0a0640 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f60d8009870 tx=0x7f60d801c200 comp rx=0 tx=0).stop 2024-04-26T00:39:46.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e46b5640 1 -- 172.21.15.37:0/2822146354 shutdown_connections 2024-04-26T00:39:46.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e46b5640 1 --2- 172.21.15.37:0/2822146354 >> v2:172.21.15.37:3300/0 conn(0x7f60dc135660 0x7f60dc13ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:46.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e46b5640 1 --2- 172.21.15.37:0/2822146354 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60dc0a0260 0x7f60dc0a0640 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:46.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e46b5640 1 -- 172.21.15.37:0/2822146354 >> 172.21.15.37:0/2822146354 conn(0x7f60dc0a1ad0 msgr2=0x7f60dc0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:46.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e46b5640 1 -- 172.21.15.37:0/2822146354 shutdown_connections 2024-04-26T00:39:46.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e46b5640 1 -- 172.21.15.37:0/2822146354 wait complete. 2024-04-26T00:39:46.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e46b5640 10 monclient: build_initial_monmap 2024-04-26T00:39:46.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e46b5640 10 monclient: monmap: 2024-04-26T00:39:46.087 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:46.087 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:39:46.087 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:39:46.087 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:39:46.087 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:46.088 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:39:46.088 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:39:46.088 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:39:46.088 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:39:46.088 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:46.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e46b5640 1 Processor -- start 2024-04-26T00:39:46.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e46b5640 1 -- start start 2024-04-26T00:39:46.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e46b5640 10 monclient: init 2024-04-26T00:39:46.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e46b5640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:46.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e46b5640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:39:46.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e46b5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60dc0a0260 0x7f60dc11b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:46.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e46b5640 10 monclient(hunting): picked mon.noname-b con 0x7f60dc0a0260 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:39:46.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e46b5640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60dc134b30 0x7f60dc131c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:46.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e46b5640 10 monclient(hunting): picked mon.noname-a con 0x7f60dc134b30 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:39:46.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e46b5640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60dc135660 0x7f60dc1ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:46.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e46b5640 10 monclient(hunting): picked mon.noname-c con 0x7f60dc135660 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:46.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e46b5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:46.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e46b5640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f60dc0a0010 con 0x7f60dc134b30 2024-04-26T00:39:46.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e46b5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:46.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e46b5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f60dc09fe90 con 0x7f60dc0a0260 2024-04-26T00:39:46.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e46b5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:46.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e46b5640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f60dc13fbd0 con 0x7f60dc135660 2024-04-26T00:39:46.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e46b5640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:46.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e46b5640 10 monclient(hunting): authenticate will time out at 2167.386745s 2024-04-26T00:39:46.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e2c52640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60dc135660 0x7f60dc1ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:46.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e1c50640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60dc134b30 0x7f60dc131c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:46.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e2c52640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60dc135660 0x7f60dc1ccd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:38618/0 (socket says 172.21.15.37:38618) 2024-04-26T00:39:46.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e1c50640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60dc134b30 0x7f60dc131c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:41688/0 (socket says 172.21.15.37:41688) 2024-04-26T00:39:46.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e2c52640 1 -- 172.21.15.37:0/1292564399 learned_addr learned my addr 172.21.15.37:0/1292564399 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:46.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e2c52640 10 monclient(hunting): get_auth_request con 0x7f60dc135660 auth_method 0 2024-04-26T00:39:46.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e2c52640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:46.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e2c52640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:46.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e2c52640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:46.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e1c50640 10 monclient(hunting): get_auth_request con 0x7f60dc134b30 auth_method 0 2024-04-26T00:39:46.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e1c50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:46.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e1c50640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:46.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e1c50640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:46.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e2c52640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:46.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e2c52640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:46.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e2451640 1 --2- 172.21.15.37:0/1292564399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60dc0a0260 0x7f60dc11b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:46.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e1c50640 1 -- 172.21.15.37:0/1292564399 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60dc134b30 msgr2=0x7f60dc131c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:39:46.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e1c50640 1 -- 172.21.15.37:0/1292564399 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60dc134b30 msgr2=0x7f60dc131c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:39:46.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e1c50640 1 --2- 172.21.15.37:0/1292564399 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60dc134b30 0x7f60dc131c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:39:46.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e2c52640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:46.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e2451640 10 monclient(hunting): get_auth_request con 0x7f60dc0a0260 auth_method 0 2024-04-26T00:39:46.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e2451640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:46.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e2451640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:46.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e2451640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:46.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e1c50640 1 --2- 172.21.15.37:0/1292564399 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60dc134b30 0x7f60dc131c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:39:46.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e2451640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:46.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e2451640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:46.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e2451640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:46.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e2c52640 10 monclient(hunting): handle_auth_done global_id 204485 payload 824 2024-04-26T00:39:46.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e2c52640 1 -- 172.21.15.37:0/1292564399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60dc0a0260 msgr2=0x7f60dc11b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:46.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e2c52640 1 --2- 172.21.15.37:0/1292564399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60dc0a0260 0x7f60dc11b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:46.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e2c52640 1 -- 172.21.15.37:0/1292564399 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60dc134b30 msgr2=0x7f60dc131c20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:39:46.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e2c52640 1 --2- 172.21.15.37:0/1292564399 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60dc134b30 0x7f60dc131c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:46.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e2c52640 10 monclient: _finish_hunting 0 2024-04-26T00:39:46.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e2c52640 1 monclient: found mon.noname-c 2024-04-26T00:39:46.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e2c52640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:46.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e2c52640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:39:46.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.078+0000 7f60e2c52640 1 -- 172.21.15.37:0/1292564399 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60dc11c2d0 con 0x7f60dc135660 2024-04-26T00:39:46.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.082+0000 7f60e2c52640 1 -- 172.21.15.37:0/1292564399 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60dc135660 msgr2=0x7f60dc1ccd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:46.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.082+0000 7f60e2c52640 1 --2- 172.21.15.37:0/1292564399 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60dc135660 0x7f60dc1ccd60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f60d4007e30 tx=0x7f60d400e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:46.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.082+0000 7f60d2ffd640 1 -- 172.21.15.37:0/1292564399 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60d4014620 con 0x7f60dc135660 2024-04-26T00:39:46.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.082+0000 7f60d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:46.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.082+0000 7f60d2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:39:46.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.082+0000 7f60d2ffd640 10 monclient: dump: 2024-04-26T00:39:46.094 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:46.094 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:46.094 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:46.094 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:46.094 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:46.094 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:46.094 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:46.094 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:46.094 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:46.094 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:46.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.082+0000 7f60d2ffd640 10 monclient: _finish_auth 0 2024-04-26T00:39:46.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.082+0000 7f60d2ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:39:46.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.082+0000 7f60d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:46.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.082+0000 7f60d2ffd640 1 -- 172.21.15.37:0/1292564399 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60d4019070 con 0x7f60dc135660 2024-04-26T00:39:46.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.082+0000 7f60d2ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:46.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.082+0000 7f60d2ffd640 1 -- 172.21.15.37:0/1292564399 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60d4014a00 con 0x7f60dc135660 2024-04-26T00:39:46.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.082+0000 7f60d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:46.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.082+0000 7f60d2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:46.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.082+0000 7f60d2ffd640 10 monclient: dump: 2024-04-26T00:39:46.095 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:46.095 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:46.095 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:46.095 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:46.095 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:46.095 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:46.095 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:46.096 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:46.096 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:46.096 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:46.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.082+0000 7f60e46b5640 5 monclient: authenticate success, global_id 204485 2024-04-26T00:39:46.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.082+0000 7f60e46b5640 10 monclient: _renew_subs 2024-04-26T00:39:46.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.082+0000 7f60e0c4e640 4 set_mon_vals no callback set 2024-04-26T00:39:46.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.082+0000 7f60e46b5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:46.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.082+0000 7f60e46b5640 1 -- 172.21.15.37:0/1292564399 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60dc134800 con 0x7f60dc135660 2024-04-26T00:39:46.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.082+0000 7f60e46b5640 10 monclient: _renew_subs 2024-04-26T00:39:46.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.082+0000 7f60e46b5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:46.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.082+0000 7f60e46b5640 1 -- 172.21.15.37:0/1292564399 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f60dc11bd10 con 0x7f60dc135660 2024-04-26T00:39:46.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.082+0000 7f60e46b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:46.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.082+0000 7f60e46b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:46.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.082+0000 7f60e46b5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:46.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.082+0000 7f60e46b5640 1 -- 172.21.15.37:0/1292564399 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60dc066f40 con 0x7f60dc135660 2024-04-26T00:39:46.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.082+0000 7f60d2ffd640 1 -- 172.21.15.37:0/1292564399 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f60d4014be0 con 0x7f60dc135660 2024-04-26T00:39:46.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.082+0000 7f60d2ffd640 1 --2- 172.21.15.37:0/1292564399 >> v2:172.21.15.45:6812/2379918623 conn(0x7f60c0035e80 0x7f60c0038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:46.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.082+0000 7f60d2ffd640 1 -- 172.21.15.37:0/1292564399 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(782..782 src has 705..782) ==== 3959+0+0 (secure 0 0 0) 0x7f60d4049160 con 0x7f60dc135660 2024-04-26T00:39:46.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.082+0000 7f60e2451640 1 --2- 172.21.15.37:0/1292564399 >> v2:172.21.15.45:6812/2379918623 conn(0x7f60c0035e80 0x7f60c0038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:46.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.082+0000 7f60e2451640 10 monclient: get_auth_request con 0x7f60c0035e80 auth_method 0 2024-04-26T00:39:46.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.086+0000 7f60e2451640 1 --2- 172.21.15.37:0/1292564399 >> v2:172.21.15.45:6812/2379918623 conn(0x7f60c0035e80 0x7f60c0038340 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f60dc06b640 tx=0x7f60d8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:46.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.086+0000 7f60d2ffd640 1 -- 172.21.15.37:0/1292564399 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f60d401d890 con 0x7f60dc135660 2024-04-26T00:39:46.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.086+0000 7f60d2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:46.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.086+0000 7f60d2ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:39:46.122 INFO:tasks.rados.rados.0.smithi045.stdout:3435: finishing write tid 2 to smithi04526579-8 2024-04-26T00:39:46.122 INFO:tasks.rados.rados.0.smithi045.stdout:3435: finishing write tid 3 to smithi04526579-8 2024-04-26T00:39:46.123 INFO:tasks.rados.rados.0.smithi045.stdout:3435: finishing write tid 4 to smithi04526579-8 2024-04-26T00:39:46.123 INFO:tasks.rados.rados.0.smithi045.stdout:3435: oid 8 updating version 0 to 2218 2024-04-26T00:39:46.123 INFO:tasks.rados.rados.0.smithi045.stdout:3435: oid 8 version 2218 is already newer than 2217 2024-04-26T00:39:46.123 INFO:tasks.rados.rados.0.smithi045.stdout:3435: oid 8 updating version 2218 to 2219 2024-04-26T00:39:46.123 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 2219 (ObjNum 1221 snap 382 seq_num 1221) dirty exists 2024-04-26T00:39:46.123 INFO:tasks.rados.rados.0.smithi045.stdout:3435: left oid 8 (ObjNum 1221 snap 382 seq_num 1221) 2024-04-26T00:39:46.123 INFO:tasks.rados.rados.0.smithi045.stdout:3438: expect (ObjNum 1216 snap 378 seq_num 1216) 2024-04-26T00:39:46.251 INFO:tasks.rados.rados.0.smithi045.stdout:3440: finishing copy_from racing read to smithi04526579-32 2024-04-26T00:39:46.251 INFO:tasks.rados.rados.0.smithi045.stdout:3440: finishing copy_from to smithi04526579-32 2024-04-26T00:39:46.251 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 2444 (ObjNum 1139 snap 345 seq_num 1139) dirty exists 2024-04-26T00:39:46.251 INFO:tasks.rados.rados.0.smithi045.stdout:3432: done (7 left) 2024-04-26T00:39:46.251 INFO:tasks.rados.rados.0.smithi045.stdout:3433: done (6 left) 2024-04-26T00:39:46.251 INFO:tasks.rados.rados.0.smithi045.stdout:3434: done (5 left) 2024-04-26T00:39:46.251 INFO:tasks.rados.rados.0.smithi045.stdout:3435: done (4 left) 2024-04-26T00:39:46.251 INFO:tasks.rados.rados.0.smithi045.stdout:3436: done (3 left) 2024-04-26T00:39:46.251 INFO:tasks.rados.rados.0.smithi045.stdout:3437: done (2 left) 2024-04-26T00:39:46.251 INFO:tasks.rados.rados.0.smithi045.stdout:3438: done (1 left) 2024-04-26T00:39:46.251 INFO:tasks.rados.rados.0.smithi045.stdout:3440: done (0 left) 2024-04-26T00:39:46.251 INFO:tasks.rados.rados.0.smithi045.stdout:3441: delete oid 46 current snap is 382 2024-04-26T00:39:46.255 INFO:tasks.rados.rados.0.smithi045.stdout:3441: done (0 left) 2024-04-26T00:39:46.255 INFO:tasks.rados.rados.0.smithi045.stdout:3442: delete oid 31 current snap is 382 2024-04-26T00:39:46.265 INFO:tasks.rados.rados.0.smithi045.stdout:3442: done (0 left) 2024-04-26T00:39:46.266 INFO:tasks.rados.rados.0.smithi045.stdout:3443: read oid 44 snap 373 2024-04-26T00:39:46.266 INFO:tasks.rados.rados.0.smithi045.stdout:3443: expect (ObjNum 1127 snap 340 seq_num 1127) 2024-04-26T00:39:46.266 INFO:tasks.rados.rados.0.smithi045.stdout:3444: write oid 48 current snap is 382 2024-04-26T00:39:46.266 INFO:tasks.rados.rados.0.smithi045.stdout:3444: seq_num 1224 ranges {571768=437609,1808901=711425} 2024-04-26T00:39:46.282 INFO:tasks.rados.rados.0.smithi045.stdout:3444: writing smithi04526579-48 from 571768 to 1009377 tid 1 2024-04-26T00:39:46.307 INFO:tasks.rados.rados.0.smithi045.stdout:3444: writing smithi04526579-48 from 1808901 to 2520326 tid 2 2024-04-26T00:39:46.307 INFO:tasks.rados.rados.0.smithi045.stdout:3445: rollback oid 34 current snap is 382 2024-04-26T00:39:46.308 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 34 to 375 2024-04-26T00:39:46.308 INFO:tasks.rados.rados.0.smithi045.stdout:3443: expect (ObjNum 1127 snap 340 seq_num 1127) 2024-04-26T00:39:46.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.318+0000 7f60e46b5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:39:46.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.318+0000 7f60e46b5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:39:46.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.318+0000 7f60e46b5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:46.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.318+0000 7f60e46b5640 1 -- 172.21.15.37:0/1292564399 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f60dc13b970 con 0x7f60dc135660 2024-04-26T00:39:46.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.322+0000 7f60d2ffd640 1 -- 172.21.15.37:0/1292564399 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7f60d401b310 con 0x7f60dc135660 2024-04-26T00:39:46.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.322+0000 7f60d2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:39:46.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.322+0000 7f60d2ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:39:46.321 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":146,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:39:46.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.322+0000 7f60e46b5640 1 -- 172.21.15.37:0/1292564399 >> v2:172.21.15.45:6812/2379918623 conn(0x7f60c0035e80 msgr2=0x7f60c0038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:46.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.322+0000 7f60e46b5640 1 --2- 172.21.15.37:0/1292564399 >> v2:172.21.15.45:6812/2379918623 conn(0x7f60c0035e80 0x7f60c0038340 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f60dc06b640 tx=0x7f60d8008040 comp rx=0 tx=0).stop 2024-04-26T00:39:46.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.322+0000 7f60e46b5640 10 monclient: shutdown 2024-04-26T00:39:46.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.322+0000 7f60e46b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:46.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.322+0000 7f60e46b5640 1 -- 172.21.15.37:0/1292564399 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60dc135660 msgr2=0x7f60dc1ccd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:46.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.322+0000 7f60e46b5640 1 --2- 172.21.15.37:0/1292564399 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60dc135660 0x7f60dc1ccd60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f60d4007e30 tx=0x7f60d400e4b0 comp rx=0 tx=0).stop 2024-04-26T00:39:46.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.322+0000 7f60e46b5640 1 -- 172.21.15.37:0/1292564399 shutdown_connections 2024-04-26T00:39:46.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.322+0000 7f60e46b5640 1 --2- 172.21.15.37:0/1292564399 >> v2:172.21.15.45:6812/2379918623 conn(0x7f60c0035e80 0x7f60c0038340 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:46.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.322+0000 7f60e46b5640 1 --2- 172.21.15.37:0/1292564399 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60dc135660 0x7f60dc1ccd60 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:46.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.322+0000 7f60e46b5640 1 --2- 172.21.15.37:0/1292564399 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60dc134b30 0x7f60dc131c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:46.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.322+0000 7f60e46b5640 1 --2- 172.21.15.37:0/1292564399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60dc0a0260 0x7f60dc11b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:46.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.322+0000 7f60e46b5640 1 -- 172.21.15.37:0/1292564399 >> 172.21.15.37:0/1292564399 conn(0x7f60dc0a1ad0 msgr2=0x7f60dc0a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:46.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.322+0000 7f60e46b5640 1 -- 172.21.15.37:0/1292564399 shutdown_connections 2024-04-26T00:39:46.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:46.322+0000 7f60e46b5640 1 -- 172.21.15.37:0/1292564399 wait complete. 2024-04-26T00:39:46.496 INFO:tasks.rados.rados.0.smithi045.stdout:3444: finishing write tid 1 to smithi04526579-48 2024-04-26T00:39:46.496 INFO:tasks.rados.rados.0.smithi045.stdout:3444: finishing write tid 2 to smithi04526579-48 2024-04-26T00:39:46.496 INFO:tasks.rados.rados.0.smithi045.stdout:3444: finishing write tid 3 to smithi04526579-48 2024-04-26T00:39:46.496 INFO:tasks.rados.rados.0.smithi045.stdout:3444: finishing write tid 4 to smithi04526579-48 2024-04-26T00:39:46.496 INFO:tasks.rados.rados.0.smithi045.stdout:3444: oid 48 updating version 0 to 2467 2024-04-26T00:39:46.496 INFO:tasks.rados.rados.0.smithi045.stdout:3444: oid 48 updating version 2467 to 2468 2024-04-26T00:39:46.496 INFO:tasks.rados.rados.0.smithi045.stdout:3444: oid 48 updating version 2468 to 2469 2024-04-26T00:39:46.496 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 2469 (ObjNum 1224 snap 382 seq_num 1224) dirty exists 2024-04-26T00:39:46.496 INFO:tasks.rados.rados.0.smithi045.stdout:3444: left oid 48 (ObjNum 1224 snap 382 seq_num 1224) 2024-04-26T00:39:46.496 INFO:tasks.rados.rados.0.smithi045.stdout:3445: finishing rollback tid 1 to smithi04526579-34 2024-04-26T00:39:46.497 INFO:tasks.rados.rados.0.smithi045.stdout:3445: finishing rollback tid 2 to smithi04526579-34 2024-04-26T00:39:46.497 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 2471 (ObjNum 1160 snap 359 seq_num 1160) dirty exists 2024-04-26T00:39:46.497 INFO:tasks.rados.rados.0.smithi045.stdout:3443: done (2 left) 2024-04-26T00:39:46.497 INFO:tasks.rados.rados.0.smithi045.stdout:3444: done (1 left) 2024-04-26T00:39:46.497 INFO:tasks.rados.rados.0.smithi045.stdout:3445: done (0 left) 2024-04-26T00:39:46.497 INFO:tasks.rados.rados.0.smithi045.stdout:3446: rollback oid 50 current snap is 382 2024-04-26T00:39:46.497 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 50 to 375 2024-04-26T00:39:46.497 INFO:tasks.rados.rados.0.smithi045.stdout:3447: read oid 39 snap -1 2024-04-26T00:39:46.497 INFO:tasks.rados.rados.0.smithi045.stdout:3447: expect deleted 2024-04-26T00:39:46.497 INFO:tasks.rados.rados.0.smithi045.stdout:3448: read oid 26 snap 376 2024-04-26T00:39:46.497 INFO:tasks.rados.rados.0.smithi045.stdout:3448: expect deleted 2024-04-26T00:39:46.498 INFO:tasks.rados.rados.0.smithi045.stdout:3449: snap_remove snap 379 2024-04-26T00:39:46.526 INFO:tasks.rados.rados.0.smithi045.stdout:3449: done (3 left) 2024-04-26T00:39:46.526 INFO:tasks.rados.rados.0.smithi045.stdout:3450: write oid 6 current snap is 382 2024-04-26T00:39:46.526 INFO:tasks.rados.rados.0.smithi045.stdout:3450: seq_num 1225 ranges {588575=512858,1867383=455083} 2024-04-26T00:39:46.545 INFO:tasks.rados.rados.0.smithi045.stdout:3450: writing smithi04526579-6 from 588575 to 1101433 tid 1 2024-04-26T00:39:46.565 INFO:tasks.rados.rados.0.smithi045.stdout:3450: writing smithi04526579-6 from 1867383 to 2322466 tid 2 2024-04-26T00:39:46.565 INFO:tasks.rados.rados.0.smithi045.stdout:3451: write oid 46 current snap is 382 2024-04-26T00:39:46.565 INFO:tasks.rados.rados.0.smithi045.stdout:3451: seq_num 1226 ranges {605382=588107,1925865=679517} 2024-04-26T00:39:46.583 INFO:tasks.rados.rados.0.smithi045.stdout:3451: writing smithi04526579-46 from 605382 to 1193489 tid 1 2024-04-26T00:39:46.600 INFO:tasks.rados.rados.0.smithi045.stdout:3451: writing smithi04526579-46 from 1925865 to 2605382 tid 2 2024-04-26T00:39:46.600 INFO:tasks.rados.rados.0.smithi045.stdout:3452: read oid 11 snap 376 2024-04-26T00:39:46.600 INFO:tasks.rados.rados.0.smithi045.stdout:3452: expect (ObjNum 1139 snap 345 seq_num 1139) 2024-04-26T00:39:46.600 INFO:tasks.rados.rados.0.smithi045.stdout:3453: read oid 14 snap -1 2024-04-26T00:39:46.600 INFO:tasks.rados.rados.0.smithi045.stdout:3453: expect deleted 2024-04-26T00:39:46.601 INFO:tasks.rados.rados.0.smithi045.stdout:3454: read oid 20 snap -1 2024-04-26T00:39:46.601 INFO:tasks.rados.rados.0.smithi045.stdout:3454: expect (ObjNum 1093 snap 324 seq_num 1093) 2024-04-26T00:39:46.601 INFO:tasks.rados.rados.0.smithi045.stdout:3455: delete oid 45 current snap is 382 2024-04-26T00:39:46.601 INFO:tasks.rados.rados.0.smithi045.stdout:3446: finishing rollback tid 0 to smithi04526579-50 2024-04-26T00:39:46.601 INFO:tasks.rados.rados.0.smithi045.stdout:3446: finishing rollback tid 1 to smithi04526579-50 2024-04-26T00:39:46.601 INFO:tasks.rados.rados.0.smithi045.stdout:3446: finishing rollback tid 2 to smithi04526579-50 2024-04-26T00:39:46.601 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 2424 (ObjNum 1207 snap 373 seq_num 1207) dirty exists 2024-04-26T00:39:46.601 INFO:tasks.rados.rados.0.smithi045.stdout:3450: finishing write tid 1 to smithi04526579-6 2024-04-26T00:39:46.601 INFO:tasks.rados.rados.0.smithi045.stdout:3450: finishing write tid 2 to smithi04526579-6 2024-04-26T00:39:46.601 INFO:tasks.rados.rados.0.smithi045.stdout:3450: finishing write tid 3 to smithi04526579-6 2024-04-26T00:39:46.601 INFO:tasks.rados.rados.0.smithi045.stdout:3450: finishing write tid 4 to smithi04526579-6 2024-04-26T00:39:46.601 INFO:tasks.rados.rados.0.smithi045.stdout:3450: oid 6 updating version 0 to 2009 2024-04-26T00:39:46.601 INFO:tasks.rados.rados.0.smithi045.stdout:3450: oid 6 updating version 2009 to 2010 2024-04-26T00:39:46.602 INFO:tasks.rados.rados.0.smithi045.stdout:3450: oid 6 version 2010 is already newer than 2008 2024-04-26T00:39:46.602 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 2010 (ObjNum 1225 snap 382 seq_num 1225) dirty exists 2024-04-26T00:39:46.602 INFO:tasks.rados.rados.0.smithi045.stdout:3450: left oid 6 (ObjNum 1225 snap 382 seq_num 1225) 2024-04-26T00:39:46.602 INFO:tasks.rados.rados.0.smithi045.stdout:3451: finishing write tid 1 to smithi04526579-46 2024-04-26T00:39:46.606 INFO:tasks.rados.rados.0.smithi045.stdout:3452: expect (ObjNum 1139 snap 345 seq_num 1139) 2024-04-26T00:39:46.838 INFO:tasks.rados.rados.0.smithi045.stdout:3451: finishing write tid 2 to smithi04526579-46 2024-04-26T00:39:46.838 INFO:tasks.rados.rados.0.smithi045.stdout:3451: finishing write tid 3 to smithi04526579-46 2024-04-26T00:39:46.838 INFO:tasks.rados.rados.0.smithi045.stdout:3451: finishing write tid 4 to smithi04526579-46 2024-04-26T00:39:46.838 INFO:tasks.rados.rados.0.smithi045.stdout:3451: oid 46 updating version 0 to 2446 2024-04-26T00:39:46.838 INFO:tasks.rados.rados.0.smithi045.stdout:3451: oid 46 version 2446 is already newer than 2445 2024-04-26T00:39:46.838 INFO:tasks.rados.rados.0.smithi045.stdout:3451: oid 46 updating version 2446 to 2447 2024-04-26T00:39:46.838 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 2447 (ObjNum 1226 snap 382 seq_num 1226) dirty exists 2024-04-26T00:39:46.838 INFO:tasks.rados.rados.0.smithi045.stdout:3451: left oid 46 (ObjNum 1226 snap 382 seq_num 1226) 2024-04-26T00:39:46.838 INFO:tasks.rados.rados.0.smithi045.stdout:3454: expect (ObjNum 1093 snap 324 seq_num 1093) 2024-04-26T00:39:46.913 INFO:tasks.rados.rados.0.smithi045.stdout:3446: done (8 left) 2024-04-26T00:39:46.913 INFO:tasks.rados.rados.0.smithi045.stdout:3447: done (7 left) 2024-04-26T00:39:46.913 INFO:tasks.rados.rados.0.smithi045.stdout:3448: done (6 left) 2024-04-26T00:39:46.914 INFO:tasks.rados.rados.0.smithi045.stdout:3450: done (5 left) 2024-04-26T00:39:46.914 INFO:tasks.rados.rados.0.smithi045.stdout:3451: done (4 left) 2024-04-26T00:39:46.914 INFO:tasks.rados.rados.0.smithi045.stdout:3452: done (3 left) 2024-04-26T00:39:46.914 INFO:tasks.rados.rados.0.smithi045.stdout:3453: done (2 left) 2024-04-26T00:39:46.914 INFO:tasks.rados.rados.0.smithi045.stdout:3454: done (1 left) 2024-04-26T00:39:46.914 INFO:tasks.rados.rados.0.smithi045.stdout:3455: done (0 left) 2024-04-26T00:39:46.914 INFO:tasks.rados.rados.0.smithi045.stdout:3456: delete oid 30 current snap is 382 2024-04-26T00:39:46.918 INFO:tasks.rados.rados.0.smithi045.stdout:3456: done (0 left) 2024-04-26T00:39:46.918 INFO:tasks.rados.rados.0.smithi045.stdout:3457: copy_from oid 22 from oid 11 current snap is 382 2024-04-26T00:39:46.918 INFO:tasks.rados.rados.0.smithi045.stdout:3458: rollback oid 1 current snap is 382 2024-04-26T00:39:46.918 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 1 to 351 2024-04-26T00:39:46.919 INFO:tasks.rados.rados.0.smithi045.stdout:3459: rollback oid 41 current snap is 382 2024-04-26T00:39:46.919 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 41 to 380 2024-04-26T00:39:46.919 INFO:tasks.rados.rados.0.smithi045.stdout:3460: delete oid 9 current snap is 382 2024-04-26T00:39:46.925 INFO:tasks.rados.rados.0.smithi045.stdout:3460: done (3 left) 2024-04-26T00:39:46.925 INFO:tasks.rados.rados.0.smithi045.stdout:3461: delete oid 33 current snap is 382 2024-04-26T00:39:46.936 INFO:tasks.rados.rados.0.smithi045.stdout:3461: done (3 left) 2024-04-26T00:39:46.936 INFO:tasks.rados.rados.0.smithi045.stdout:3462: delete oid 20 current snap is 382 2024-04-26T00:39:46.937 INFO:tasks.rados.rados.0.smithi045.stdout:3458: finishing rollback tid 0 to smithi04526579-1 2024-04-26T00:39:46.937 INFO:tasks.rados.rados.0.smithi045.stdout:3458: finishing rollback tid 1 to smithi04526579-1 2024-04-26T00:39:46.942 INFO:tasks.rados.rados.0.smithi045.stdout:3462: done (3 left) 2024-04-26T00:39:46.942 INFO:tasks.rados.rados.0.smithi045.stdout:3463: snap_create 2024-04-26T00:39:46.943 INFO:tasks.rados.rados.0.smithi045.stdout:3457: finishing copy_from racing read to smithi04526579-22 2024-04-26T00:39:46.944 INFO:tasks.rados.rados.0.smithi045.stdout:3458: finishing rollback tid 2 to smithi04526579-1 2024-04-26T00:39:46.945 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 2478 (ObjNum 1079 snap 322 seq_num 1079) dirty exists 2024-04-26T00:39:46.945 INFO:tasks.rados.rados.0.smithi045.stdout:3459: finishing rollback tid 0 to smithi04526579-41 2024-04-26T00:39:46.945 INFO:tasks.rados.rados.0.smithi045.stdout:3459: finishing rollback tid 1 to smithi04526579-41 2024-04-26T00:39:46.946 INFO:tasks.rados.rados.0.smithi045.stdout:3459: finishing rollback tid 2 to smithi04526579-41 2024-04-26T00:39:46.946 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 2481 (ObjNum 1189 snap 369 seq_num 1189) dirty exists 2024-04-26T00:39:46.948 INFO:tasks.rados.rados.0.smithi045.stdout:3457: finishing copy_from to smithi04526579-22 2024-04-26T00:39:46.948 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 2444 (ObjNum 1139 snap 345 seq_num 1139) dirty exists 2024-04-26T00:39:48.009 INFO:tasks.rados.rados.0.smithi045.stdout:3457: done (3 left) 2024-04-26T00:39:48.009 INFO:tasks.rados.rados.0.smithi045.stdout:3458: done (2 left) 2024-04-26T00:39:48.009 INFO:tasks.rados.rados.0.smithi045.stdout:3459: done (1 left) 2024-04-26T00:39:48.010 INFO:tasks.rados.rados.0.smithi045.stdout:3463: done (0 left) 2024-04-26T00:39:48.010 INFO:tasks.rados.rados.0.smithi045.stdout:3464: rollback oid 10 current snap is 383 2024-04-26T00:39:48.010 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 10 to 362 2024-04-26T00:39:48.010 INFO:tasks.rados.rados.0.smithi045.stdout:3465: write oid 8 current snap is 383 2024-04-26T00:39:48.010 INFO:tasks.rados.rados.0.smithi045.stdout:3465: seq_num 1232 ranges {706224=755954,2160404=545820} 2024-04-26T00:39:48.033 INFO:tasks.rados.rados.0.smithi045.stdout:3465: writing smithi04526579-8 from 706224 to 1462178 tid 1 2024-04-26T00:39:48.054 INFO:tasks.rados.rados.0.smithi045.stdout:3465: writing smithi04526579-8 from 2160404 to 2706224 tid 2 2024-04-26T00:39:48.054 INFO:tasks.rados.rados.0.smithi045.stdout:3466: write (excl) oid 5 current snap is 383 2024-04-26T00:39:48.054 INFO:tasks.rados.rados.0.smithi045.stdout:3464: finishing rollback tid 0 to smithi04526579-10 2024-04-26T00:39:48.054 INFO:tasks.rados.rados.0.smithi045.stdout:3464: finishing rollback tid 1 to smithi04526579-10 2024-04-26T00:39:48.054 INFO:tasks.rados.rados.0.smithi045.stdout:3464: finishing rollback tid 2 to smithi04526579-10 2024-04-26T00:39:48.054 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 2427 (ObjNum 1112 snap 331 seq_num 1112) dirty exists 2024-04-26T00:39:48.054 INFO:tasks.rados.rados.0.smithi045.stdout:3465: finishing write tid 1 to smithi04526579-8 2024-04-26T00:39:48.054 INFO:tasks.rados.rados.0.smithi045.stdout:3466: seq_num 1233 ranges {723031=431203,1818886=469759} 2024-04-26T00:39:48.073 INFO:tasks.rados.rados.0.smithi045.stdout:3466: writing smithi04526579-5 from 723031 to 1154234 tid 1 2024-04-26T00:39:48.090 INFO:tasks.rados.rados.0.smithi045.stdout:3466: writing smithi04526579-5 from 1818886 to 2288645 tid 2 2024-04-26T00:39:48.090 INFO:tasks.rados.rados.0.smithi045.stdout:3464: done (2 left) 2024-04-26T00:39:48.090 INFO:tasks.rados.rados.0.smithi045.stdout:3467: delete oid 3 current snap is 383 2024-04-26T00:39:48.090 INFO:tasks.rados.rados.0.smithi045.stdout:3465: finishing write tid 2 to smithi04526579-8 2024-04-26T00:39:48.090 INFO:tasks.rados.rados.0.smithi045.stdout:3465: finishing write tid 3 to smithi04526579-8 2024-04-26T00:39:48.090 INFO:tasks.rados.rados.0.smithi045.stdout:3465: finishing write tid 4 to smithi04526579-8 2024-04-26T00:39:48.090 INFO:tasks.rados.rados.0.smithi045.stdout:3465: oid 8 updating version 0 to 2220 2024-04-26T00:39:48.090 INFO:tasks.rados.rados.0.smithi045.stdout:3465: oid 8 updating version 2220 to 2222 2024-04-26T00:39:48.090 INFO:tasks.rados.rados.0.smithi045.stdout:3465: oid 8 version 2222 is already newer than 2221 2024-04-26T00:39:48.090 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 2222 (ObjNum 1232 snap 383 seq_num 1232) dirty exists 2024-04-26T00:39:48.090 INFO:tasks.rados.rados.0.smithi045.stdout:3465: left oid 8 (ObjNum 1232 snap 383 seq_num 1232) 2024-04-26T00:39:48.091 INFO:tasks.rados.rados.0.smithi045.stdout:3466: finishing write tid 1 to smithi04526579-5 2024-04-26T00:39:48.100 INFO:tasks.rados.rados.0.smithi045.stdout:3465: done (2 left) 2024-04-26T00:39:48.100 INFO:tasks.rados.rados.0.smithi045.stdout:3467: done (1 left) 2024-04-26T00:39:48.100 INFO:tasks.rados.rados.0.smithi045.stdout:3468: snap_create 2024-04-26T00:39:48.100 INFO:tasks.rados.rados.0.smithi045.stdout:3466: finishing write tid 2 to smithi04526579-5 2024-04-26T00:39:48.101 INFO:tasks.rados.rados.0.smithi045.stdout:3466: finishing write tid 3 to smithi04526579-5 2024-04-26T00:39:48.103 INFO:tasks.rados.rados.0.smithi045.stdout:3466: finishing write tid 4 to smithi04526579-5 2024-04-26T00:39:48.103 INFO:tasks.rados.rados.0.smithi045.stdout:3466: oid 5 updating version 0 to 2611 2024-04-26T00:39:48.103 INFO:tasks.rados.rados.0.smithi045.stdout:3466: oid 5 updating version 2611 to 2612 2024-04-26T00:39:48.103 INFO:tasks.rados.rados.0.smithi045.stdout:3466: oid 5 updating version 2612 to 2613 2024-04-26T00:39:48.103 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 2613 (ObjNum 1233 snap 383 seq_num 1233) dirty exists 2024-04-26T00:39:48.103 INFO:tasks.rados.rados.0.smithi045.stdout:3466: left oid 5 (ObjNum 1233 snap 383 seq_num 1233) 2024-04-26T00:39:48.547 INFO:tasks.rados.rados.0.smithi045.stdout:3466: done (1 left) 2024-04-26T00:39:48.547 INFO:tasks.rados.rados.0.smithi045.stdout:3468: done (0 left) 2024-04-26T00:39:48.547 INFO:tasks.rados.rados.0.smithi045.stdout:3469: read oid 31 snap -1 2024-04-26T00:39:48.547 INFO:tasks.rados.rados.0.smithi045.stdout:3469: expect deleted 2024-04-26T00:39:48.547 INFO:tasks.rados.rados.0.smithi045.stdout:3470: write oid 39 current snap is 384 2024-04-26T00:39:48.547 INFO:tasks.rados.rados.0.smithi045.stdout:3470: seq_num 1235 ranges {756645=581701,1935850=473428} 2024-04-26T00:39:48.566 INFO:tasks.rados.rados.0.smithi045.stdout:3470: writing smithi04526579-39 from 756645 to 1338346 tid 1 2024-04-26T00:39:48.581 INFO:tasks.rados.rados.0.smithi045.stdout:3470: writing smithi04526579-39 from 1935850 to 2409278 tid 2 2024-04-26T00:39:48.581 INFO:tasks.rados.rados.0.smithi045.stdout:3471: snap_remove snap 377 2024-04-26T00:39:49.342 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:39:49.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.474+0000 7fc6925e3640 10 monclient: get_monmap_and_config 2024-04-26T00:39:49.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.474+0000 7fc6925e3640 10 monclient: build_initial_monmap 2024-04-26T00:39:49.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.474+0000 7fc6925e3640 10 monclient: monmap: 2024-04-26T00:39:49.476 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:49.476 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:49.476 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:39:49.478450+0000 2024-04-26T00:39:49.476 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:39:49.478450+0000 2024-04-26T00:39:49.476 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:49.476 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:49.476 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:39:49.476 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:39:49.476 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:39:49.476 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:39:49.476 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:39:49.476 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:49.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.478+0000 7fc6925e3640 1 Processor -- start 2024-04-26T00:39:49.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.478+0000 7fc6925e3640 1 -- start start 2024-04-26T00:39:49.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.478+0000 7fc6925e3640 10 monclient: init 2024-04-26T00:39:49.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.478+0000 7fc6925e3640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:49.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.478+0000 7fc6925e3640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-04-26T00:39:49.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.478+0000 7fc6925e3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc68c136e10 0x7fc68c09c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:49.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.478+0000 7fc6925e3640 10 monclient(hunting): picked mon.noname-b con 0x7fc68c136e10 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:49.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.478+0000 7fc6925e3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc68c1371f0 addr v1:172.21.15.37:6789/0 2024-04-26T00:39:49.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.478+0000 7fc6925e3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc68c13c340 addr v1:172.21.15.45:6789/0 2024-04-26T00:39:49.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.478+0000 7fc6925e3640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc68c09b680 con 0x7fc68c1371f0 2024-04-26T00:39:49.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.478+0000 7fc6925e3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc68c09bbd0 con 0x7fc68c13c340 2024-04-26T00:39:49.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.478+0000 7fc6925e3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:49.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.478+0000 7fc6925e3640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc68c09b8f0 con 0x7fc68c136e10 2024-04-26T00:39:49.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.478+0000 7fc6925e3640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:49.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.478+0000 7fc6925e3640 10 monclient(hunting): authenticate will time out at 2170.785194s 2024-04-26T00:39:49.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.478+0000 7fc68bfff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc68c136e10 0x7fc68c09c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:49.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.478+0000 7fc6837fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fc68c1371f0 0x7fc68c1375d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:38212/0 (socket says 172.21.15.37:38212) 2024-04-26T00:39:49.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.478+0000 7fc6837fe640 1 -- 172.21.15.37:0/3596793734 learned_addr learned my addr 172.21.15.37:0/3596793734 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:39:49.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.478+0000 7fc68bfff640 10 monclient(hunting): get_auth_request con 0x7fc68c136e10 auth_method 0 2024-04-26T00:39:49.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.478+0000 7fc68bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:49.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.478+0000 7fc68bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:49.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.478+0000 7fc68bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:49.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.478+0000 7fc68bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:49.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.478+0000 7fc68bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:49.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.478+0000 7fc68bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:49.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.478+0000 7fc68bfff640 10 monclient(hunting): handle_auth_done global_id 204488 payload 274 2024-04-26T00:39:49.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.478+0000 7fc68bfff640 1 -- 172.21.15.37:0/3596793734 >> v1:172.21.15.45:6789/0 conn(0x7fc68c13c340 legacy=0x7fc68c13e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:49.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.478+0000 7fc68bfff640 1 -- 172.21.15.37:0/3596793734 >> v1:172.21.15.37:6789/0 conn(0x7fc68c1371f0 legacy=0x7fc68c1375d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:49.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.478+0000 7fc68bfff640 10 monclient: _finish_hunting 0 2024-04-26T00:39:49.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.478+0000 7fc68bfff640 1 monclient: found mon.noname-b 2024-04-26T00:39:49.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.478+0000 7fc68bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:49.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.478+0000 7fc68bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:39:49.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.478+0000 7fc68bfff640 1 -- 172.21.15.37:0/3596793734 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc68c0a5d90 con 0x7fc68c136e10 2024-04-26T00:39:49.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.482+0000 7fc68bfff640 1 -- 172.21.15.37:0/3596793734 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc68c136e10 msgr2=0x7fc68c09c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:49.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.482+0000 7fc68bfff640 1 --2- 172.21.15.37:0/3596793734 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc68c136e10 0x7fc68c09c5b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc67c009820 tx=0x7fc67c01c300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61e2b3944d6e516e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:49.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.482+0000 7fc68b7fe640 1 -- 172.21.15.37:0/3596793734 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc67c0b0070 con 0x7fc68c136e10 2024-04-26T00:39:49.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.482+0000 7fc68b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:49.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.482+0000 7fc68b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:39:49.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.482+0000 7fc68b7fe640 10 monclient: dump: 2024-04-26T00:39:49.483 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:49.483 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:49.483 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:49.483 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:49.483 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:49.483 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:49.483 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:49.483 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:49.483 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:49.484 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:49.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.482+0000 7fc68b7fe640 10 monclient: _finish_auth 0 2024-04-26T00:39:49.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.482+0000 7fc68b7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:39:49.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.482+0000 7fc68b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:49.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.482+0000 7fc68b7fe640 1 -- 172.21.15.37:0/3596793734 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc67c0ab600 con 0x7fc68c136e10 2024-04-26T00:39:49.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.482+0000 7fc68b7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:49.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.482+0000 7fc68b7fe640 1 -- 172.21.15.37:0/3596793734 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc67c0ab9d0 con 0x7fc68c136e10 2024-04-26T00:39:49.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.482+0000 7fc6925e3640 5 monclient: authenticate success, global_id 204488 2024-04-26T00:39:49.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.482+0000 7fc6925e3640 10 monclient: get_monmap_and_config success 2024-04-26T00:39:49.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.482+0000 7fc6925e3640 4 set_mon_vals no callback set 2024-04-26T00:39:49.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.482+0000 7fc68b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:49.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.482+0000 7fc68b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:49.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.482+0000 7fc68b7fe640 10 monclient: dump: 2024-04-26T00:39:49.484 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:49.485 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:49.485 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:49.485 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:49.485 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:49.485 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:49.485 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:49.485 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:49.485 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:49.485 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:49.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.482+0000 7fc6925e3640 10 monclient: shutdown 2024-04-26T00:39:49.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.482+0000 7fc6925e3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:49.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.482+0000 7fc6925e3640 1 -- 172.21.15.37:0/3596793734 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc68c136e10 msgr2=0x7fc68c09c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:49.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.482+0000 7fc6925e3640 1 --2- 172.21.15.37:0/3596793734 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc68c136e10 0x7fc68c09c5b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc67c009820 tx=0x7fc67c01c300 comp rx=0 tx=0).stop 2024-04-26T00:39:49.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.482+0000 7fc6925e3640 1 -- 172.21.15.37:0/3596793734 shutdown_connections 2024-04-26T00:39:49.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.482+0000 7fc6925e3640 1 --2- 172.21.15.37:0/3596793734 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc68c136e10 0x7fc68c09c5b0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:49.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.482+0000 7fc6925e3640 1 -- 172.21.15.37:0/3596793734 >> 172.21.15.37:0/3596793734 conn(0x7fc68c0a64e0 msgr2=0x7fc68c0a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:49.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.482+0000 7fc6925e3640 1 -- 172.21.15.37:0/3596793734 shutdown_connections 2024-04-26T00:39:49.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.482+0000 7fc6925e3640 1 -- 172.21.15.37:0/3596793734 wait complete. 2024-04-26T00:39:49.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.482+0000 7fc6925e3640 10 monclient: build_initial_monmap 2024-04-26T00:39:49.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.482+0000 7fc6925e3640 10 monclient: monmap: 2024-04-26T00:39:49.489 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:49.489 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:39:49.489 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:39:49.489 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:39:49.489 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:49.490 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:39:49.490 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:39:49.490 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:39:49.490 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:39:49.490 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:49.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.482+0000 7fc6925e3640 1 Processor -- start 2024-04-26T00:39:49.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.482+0000 7fc6925e3640 1 -- start start 2024-04-26T00:39:49.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.482+0000 7fc6925e3640 10 monclient: init 2024-04-26T00:39:49.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc6925e3640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:49.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc6925e3640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:39:49.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc6925e3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc68c136e10 0x7fc68c135730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:49.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc6925e3640 10 monclient(hunting): picked mon.noname-c con 0x7fc68c136e10 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:49.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc6925e3640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc68c1371f0 0x7fc68c135c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:49.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc6925e3640 10 monclient(hunting): picked mon.noname-a con 0x7fc68c1371f0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:39:49.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc6925e3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc68c13c340 0x7fc68c137e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:49.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc6925e3640 10 monclient(hunting): picked mon.noname-b con 0x7fc68c13c340 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:39:49.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc6925e3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:49.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc6925e3640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fc68c142a00 con 0x7fc68c1371f0 2024-04-26T00:39:49.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc6925e3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:49.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc6925e3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc68c142880 con 0x7fc68c13c340 2024-04-26T00:39:49.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc6925e3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:49.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc6925e3640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc68c142b80 con 0x7fc68c136e10 2024-04-26T00:39:49.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc6925e3640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:49.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc6925e3640 10 monclient(hunting): authenticate will time out at 2170.792256s 2024-04-26T00:39:49.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc68bfff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc68c136e10 0x7fc68c135730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:49.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc6837fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc68c1371f0 0x7fc68c135c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:49.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc68bfff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc68c136e10 0x7fc68c135730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:38650/0 (socket says 172.21.15.37:38650) 2024-04-26T00:39:49.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc68bfff640 1 -- 172.21.15.37:0/3585593599 learned_addr learned my addr 172.21.15.37:0/3585593599 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:49.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc68bfff640 10 monclient(hunting): get_auth_request con 0x7fc68c136e10 auth_method 0 2024-04-26T00:39:49.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc68bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:49.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc68bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:49.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc68bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:49.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc6837fe640 10 monclient(hunting): get_auth_request con 0x7fc68c1371f0 auth_method 0 2024-04-26T00:39:49.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc6837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:49.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc6837fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:49.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc6837fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:49.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc68bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:49.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc68bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:49.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc68bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:49.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc6837fe640 1 -- 172.21.15.37:0/3585593599 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc68c1371f0 msgr2=0x7fc68c135c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:39:49.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc6837fe640 1 -- 172.21.15.37:0/3585593599 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc68c1371f0 msgr2=0x7fc68c135c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:39:49.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc6837fe640 1 --2- 172.21.15.37:0/3585593599 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc68c1371f0 0x7fc68c135c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:39:49.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc6837fe640 1 --2- 172.21.15.37:0/3585593599 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc68c1371f0 0x7fc68c135c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:39:49.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc690b80640 1 --2- 172.21.15.37:0/3585593599 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc68c13c340 0x7fc68c137e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:49.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc68bfff640 10 monclient(hunting): handle_auth_done global_id 204491 payload 824 2024-04-26T00:39:49.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc68bfff640 1 -- 172.21.15.37:0/3585593599 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc68c13c340 msgr2=0x7fc68c137e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:49.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc68bfff640 1 --2- 172.21.15.37:0/3585593599 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc68c13c340 0x7fc68c137e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:49.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc68bfff640 1 -- 172.21.15.37:0/3585593599 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc68c1371f0 msgr2=0x7fc68c135c70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:39:49.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc68bfff640 1 --2- 172.21.15.37:0/3585593599 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc68c1371f0 0x7fc68c135c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:49.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc68bfff640 10 monclient: _finish_hunting 0 2024-04-26T00:39:49.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc68bfff640 1 monclient: found mon.noname-c 2024-04-26T00:39:49.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc68bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:49.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc68bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:39:49.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc68bfff640 1 -- 172.21.15.37:0/3585593599 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc68c13b630 con 0x7fc68c136e10 2024-04-26T00:39:49.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc690b80640 10 monclient: get_auth_request con 0x7fc68c13c340 auth_method 0 2024-04-26T00:39:49.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc690b80640 1 --2- 172.21.15.37:0/3585593599 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc68c13c340 0x7fc68c137e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:39:49.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc68bfff640 1 -- 172.21.15.37:0/3585593599 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc68c136e10 msgr2=0x7fc68c135730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:49.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc68bfff640 1 --2- 172.21.15.37:0/3585593599 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc68c136e10 0x7fc68c135730 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc67c0ae040 tx=0x7fc67c01ab40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:49.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.486+0000 7fc6897fa640 1 -- 172.21.15.37:0/3585593599 DispatchQueue::entry inject delay of 0.002000 2024-04-26T00:39:49.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.490+0000 7fc6897fa640 1 -- 172.21.15.37:0/3585593599 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc67c0b0050 con 0x7fc68c136e10 2024-04-26T00:39:49.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.490+0000 7fc6897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:49.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.490+0000 7fc6897fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:39:49.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.490+0000 7fc6897fa640 10 monclient: dump: 2024-04-26T00:39:49.496 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:49.497 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:49.497 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:49.497 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:49.497 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:49.497 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:49.497 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:49.497 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:49.497 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:49.497 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:49.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.490+0000 7fc6897fa640 10 monclient: _finish_auth 0 2024-04-26T00:39:49.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.490+0000 7fc6897fa640 10 monclient: _check_auth_tickets 2024-04-26T00:39:49.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.490+0000 7fc6897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:49.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.490+0000 7fc6897fa640 1 -- 172.21.15.37:0/3585593599 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc67c0bc070 con 0x7fc68c136e10 2024-04-26T00:39:49.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.490+0000 7fc6897fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:49.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.490+0000 7fc6897fa640 1 -- 172.21.15.37:0/3585593599 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc67c0b5c20 con 0x7fc68c136e10 2024-04-26T00:39:49.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.490+0000 7fc6925e3640 5 monclient: authenticate success, global_id 204491 2024-04-26T00:39:49.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.490+0000 7fc6925e3640 10 monclient: _renew_subs 2024-04-26T00:39:49.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.490+0000 7fc6925e3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:49.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.490+0000 7fc6925e3640 1 -- 172.21.15.37:0/3585593599 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc68c13a110 con 0x7fc68c136e10 2024-04-26T00:39:49.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.490+0000 7fc68affd640 4 set_mon_vals no callback set 2024-04-26T00:39:49.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.490+0000 7fc6897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:49.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.490+0000 7fc6897fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:49.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.490+0000 7fc6897fa640 10 monclient: dump: 2024-04-26T00:39:49.498 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:49.498 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:49.498 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:49.498 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:49.498 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:49.499 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:49.499 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:49.499 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:49.499 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:49.499 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:49.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.490+0000 7fc6925e3640 10 monclient: _renew_subs 2024-04-26T00:39:49.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.490+0000 7fc6925e3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:49.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.490+0000 7fc6925e3640 1 -- 172.21.15.37:0/3585593599 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc68c13b350 con 0x7fc68c136e10 2024-04-26T00:39:49.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.490+0000 7fc6897fa640 1 -- 172.21.15.37:0/3585593599 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc67c0b7450 con 0x7fc68c136e10 2024-04-26T00:39:49.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.490+0000 7fc6925e3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:49.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.490+0000 7fc6925e3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:49.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.490+0000 7fc6925e3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:49.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.490+0000 7fc6925e3640 1 -- 172.21.15.37:0/3585593599 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc68c075910 con 0x7fc68c136e10 2024-04-26T00:39:49.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.490+0000 7fc6897fa640 1 --2- 172.21.15.37:0/3585593599 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc66c035f00 0x7fc66c0383c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:49.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.490+0000 7fc6897fa640 1 -- 172.21.15.37:0/3585593599 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(785..785 src has 705..785) ==== 3947+0+0 (secure 0 0 0) 0x7fc67c0e3000 con 0x7fc68c136e10 2024-04-26T00:39:49.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.490+0000 7fc6837fe640 1 --2- 172.21.15.37:0/3585593599 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc66c035f00 0x7fc66c0383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:49.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.490+0000 7fc6837fe640 10 monclient: get_auth_request con 0x7fc66c035f00 auth_method 0 2024-04-26T00:39:49.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.494+0000 7fc6837fe640 1 --2- 172.21.15.37:0/3585593599 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc66c035f00 0x7fc66c0383c0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fc68c136d00 tx=0x7fc674009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:49.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.494+0000 7fc6897fa640 1 -- 172.21.15.37:0/3585593599 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc67c0c5020 con 0x7fc68c136e10 2024-04-26T00:39:49.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.494+0000 7fc6897fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:49.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.494+0000 7fc6897fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:39:49.611 INFO:tasks.rados.rados.0.smithi045.stdout:3471: done (2 left) 2024-04-26T00:39:49.611 INFO:tasks.rados.rados.0.smithi045.stdout:3472: write oid 37 current snap is 384 2024-04-26T00:39:49.611 INFO:tasks.rados.rados.0.smithi045.stdout:3472: seq_num 1236 ranges {773452=656950,1877979=617086} 2024-04-26T00:39:49.631 INFO:tasks.rados.rados.0.smithi045.stdout:3472: writing smithi04526579-37 from 773452 to 1430402 tid 1 2024-04-26T00:39:49.647 INFO:tasks.rados.rados.0.smithi045.stdout:3472: writing smithi04526579-37 from 1877979 to 2495065 tid 2 2024-04-26T00:39:49.647 INFO:tasks.rados.rados.0.smithi045.stdout:3473: write oid 13 current snap is 384 2024-04-26T00:39:49.647 INFO:tasks.rados.rados.0.smithi045.stdout:3473: seq_num 1237 ranges {790259=732199,1936461=477097} 2024-04-26T00:39:49.666 INFO:tasks.rados.rados.0.smithi045.stdout:3473: writing smithi04526579-13 from 790259 to 1522458 tid 1 2024-04-26T00:39:49.677 INFO:tasks.rados.rados.0.smithi045.stdout:3473: writing smithi04526579-13 from 1936461 to 2413558 tid 2 2024-04-26T00:39:49.677 INFO:tasks.rados.rados.0.smithi045.stdout:3474: write (excl) oid 25 current snap is 384 2024-04-26T00:39:49.678 INFO:tasks.rados.rados.0.smithi045.stdout:3474: seq_num 1238 ranges {407066=407448,1594943=620755} 2024-04-26T00:39:49.685 INFO:tasks.rados.rados.0.smithi045.stdout:3474: writing smithi04526579-25 from 407066 to 814514 tid 1 2024-04-26T00:39:49.697 INFO:tasks.rados.rados.0.smithi045.stdout:3474: writing smithi04526579-25 from 1594943 to 2215698 tid 2 2024-04-26T00:39:49.697 INFO:tasks.rados.rados.0.smithi045.stdout:3475: write oid 48 current snap is 384 2024-04-26T00:39:49.697 INFO:tasks.rados.rados.0.smithi045.stdout:3475: seq_num 1239 ranges {423873=482697,1653425=480766,2691441=132432} 2024-04-26T00:39:49.703 INFO:tasks.rados.rados.0.smithi045.stdout:3475: writing smithi04526579-48 from 423873 to 906570 tid 1 2024-04-26T00:39:49.712 INFO:tasks.rados.rados.0.smithi045.stdout:3475: writing smithi04526579-48 from 1653425 to 2134191 tid 2 2024-04-26T00:39:49.718 INFO:tasks.rados.rados.0.smithi045.stdout:3475: writing smithi04526579-48 from 2691441 to 2823873 tid 3 2024-04-26T00:39:49.718 INFO:tasks.rados.rados.0.smithi045.stdout:3476: snap_create 2024-04-26T00:39:49.719 INFO:tasks.rados.rados.0.smithi045.stdout:3470: finishing write tid 1 to smithi04526579-39 2024-04-26T00:39:49.719 INFO:tasks.rados.rados.0.smithi045.stdout:3470: finishing write tid 2 to smithi04526579-39 2024-04-26T00:39:49.719 INFO:tasks.rados.rados.0.smithi045.stdout:3470: finishing write tid 3 to smithi04526579-39 2024-04-26T00:39:49.719 INFO:tasks.rados.rados.0.smithi045.stdout:3470: finishing write tid 4 to smithi04526579-39 2024-04-26T00:39:49.719 INFO:tasks.rados.rados.0.smithi045.stdout:3470: oid 39 updating version 0 to 2401 2024-04-26T00:39:49.719 INFO:tasks.rados.rados.0.smithi045.stdout:3470: oid 39 updating version 2401 to 2402 2024-04-26T00:39:49.719 INFO:tasks.rados.rados.0.smithi045.stdout:3470: oid 39 version 2402 is already newer than 2400 2024-04-26T00:39:49.719 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 2402 (ObjNum 1235 snap 384 seq_num 1235) dirty exists 2024-04-26T00:39:49.719 INFO:tasks.rados.rados.0.smithi045.stdout:3470: left oid 39 (ObjNum 1235 snap 384 seq_num 1235) 2024-04-26T00:39:49.719 INFO:tasks.rados.rados.0.smithi045.stdout:3472: finishing write tid 1 to smithi04526579-37 2024-04-26T00:39:49.720 INFO:tasks.rados.rados.0.smithi045.stdout:3472: finishing write tid 2 to smithi04526579-37 2024-04-26T00:39:49.720 INFO:tasks.rados.rados.0.smithi045.stdout:3472: finishing write tid 3 to smithi04526579-37 2024-04-26T00:39:49.720 INFO:tasks.rados.rados.0.smithi045.stdout:3472: finishing write tid 4 to smithi04526579-37 2024-04-26T00:39:49.720 INFO:tasks.rados.rados.0.smithi045.stdout:3472: oid 37 updating version 0 to 2623 2024-04-26T00:39:49.720 INFO:tasks.rados.rados.0.smithi045.stdout:3472: oid 37 version 2623 is already newer than 2622 2024-04-26T00:39:49.720 INFO:tasks.rados.rados.0.smithi045.stdout:3472: oid 37 updating version 2623 to 2624 2024-04-26T00:39:49.720 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 2624 (ObjNum 1236 snap 384 seq_num 1236) dirty exists 2024-04-26T00:39:49.720 INFO:tasks.rados.rados.0.smithi045.stdout:3472: left oid 37 (ObjNum 1236 snap 384 seq_num 1236) 2024-04-26T00:39:49.720 INFO:tasks.rados.rados.0.smithi045.stdout:3473: finishing write tid 1 to smithi04526579-13 2024-04-26T00:39:49.720 INFO:tasks.rados.rados.0.smithi045.stdout:3473: finishing write tid 2 to smithi04526579-13 2024-04-26T00:39:49.720 INFO:tasks.rados.rados.0.smithi045.stdout:3473: finishing write tid 3 to smithi04526579-13 2024-04-26T00:39:49.720 INFO:tasks.rados.rados.0.smithi045.stdout:3473: finishing write tid 4 to smithi04526579-13 2024-04-26T00:39:49.720 INFO:tasks.rados.rados.0.smithi045.stdout:3473: oid 13 updating version 0 to 2224 2024-04-26T00:39:49.721 INFO:tasks.rados.rados.0.smithi045.stdout:3473: oid 13 updating version 2224 to 2225 2024-04-26T00:39:49.721 INFO:tasks.rados.rados.0.smithi045.stdout:3473: oid 13 version 2225 is already newer than 2223 2024-04-26T00:39:49.721 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 2225 (ObjNum 1237 snap 384 seq_num 1237) dirty exists 2024-04-26T00:39:49.721 INFO:tasks.rados.rados.0.smithi045.stdout:3473: left oid 13 (ObjNum 1237 snap 384 seq_num 1237) 2024-04-26T00:39:49.721 INFO:tasks.rados.rados.0.smithi045.stdout:3474: finishing write tid 1 to smithi04526579-25 2024-04-26T00:39:49.721 INFO:tasks.rados.rados.0.smithi045.stdout:3474: finishing write tid 2 to smithi04526579-25 2024-04-26T00:39:49.721 INFO:tasks.rados.rados.0.smithi045.stdout:3474: finishing write tid 3 to smithi04526579-25 2024-04-26T00:39:49.721 INFO:tasks.rados.rados.0.smithi045.stdout:3474: finishing write tid 4 to smithi04526579-25 2024-04-26T00:39:49.721 INFO:tasks.rados.rados.0.smithi045.stdout:3474: oid 25 updating version 0 to 2625 2024-04-26T00:39:49.721 INFO:tasks.rados.rados.0.smithi045.stdout:3474: oid 25 updating version 2625 to 2626 2024-04-26T00:39:49.721 INFO:tasks.rados.rados.0.smithi045.stdout:3474: oid 25 updating version 2626 to 2627 2024-04-26T00:39:49.721 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 2627 (ObjNum 1238 snap 384 seq_num 1238) dirty exists 2024-04-26T00:39:49.722 INFO:tasks.rados.rados.0.smithi045.stdout:3474: left oid 25 (ObjNum 1238 snap 384 seq_num 1238) 2024-04-26T00:39:49.724 INFO:tasks.rados.rados.0.smithi045.stdout:3475: finishing write tid 1 to smithi04526579-48 2024-04-26T00:39:49.724 INFO:tasks.rados.rados.0.smithi045.stdout:3475: finishing write tid 2 to smithi04526579-48 2024-04-26T00:39:49.724 INFO:tasks.rados.rados.0.smithi045.stdout:3475: finishing write tid 3 to smithi04526579-48 2024-04-26T00:39:49.725 INFO:tasks.rados.rados.0.smithi045.stdout:3475: finishing write tid 4 to smithi04526579-48 2024-04-26T00:39:49.725 INFO:tasks.rados.rados.0.smithi045.stdout:3475: finishing write tid 5 to smithi04526579-48 2024-04-26T00:39:49.725 INFO:tasks.rados.rados.0.smithi045.stdout:3475: oid 48 updating version 0 to 2475 2024-04-26T00:39:49.725 INFO:tasks.rados.rados.0.smithi045.stdout:3475: oid 48 version 2475 is already newer than 2474 2024-04-26T00:39:49.726 INFO:tasks.rados.rados.0.smithi045.stdout:3475: oid 48 version 2475 is already newer than 2473 2024-04-26T00:39:49.726 INFO:tasks.rados.rados.0.smithi045.stdout:3475: oid 48 updating version 2475 to 2476 2024-04-26T00:39:49.726 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 2476 (ObjNum 1239 snap 384 seq_num 1239) dirty exists 2024-04-26T00:39:49.726 INFO:tasks.rados.rados.0.smithi045.stdout:3475: left oid 48 (ObjNum 1239 snap 384 seq_num 1239) 2024-04-26T00:39:49.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.754+0000 7fc6925e3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:39:49.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.754+0000 7fc6925e3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:39:49.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.754+0000 7fc6925e3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:49.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.754+0000 7fc6925e3640 1 -- 172.21.15.37:0/3585593599 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc68c066f40 con 0x7fc68c136e10 2024-04-26T00:39:49.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.754+0000 7fc6897fa640 1 -- 172.21.15.37:0/3585593599 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fc67c016010 con 0x7fc68c136e10 2024-04-26T00:39:49.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.754+0000 7fc6897fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:39:49.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.754+0000 7fc6897fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:39:49.754 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":148,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:39:49.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.754+0000 7fc6925e3640 1 -- 172.21.15.37:0/3585593599 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc66c035f00 msgr2=0x7fc66c0383c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:49.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.754+0000 7fc6925e3640 1 --2- 172.21.15.37:0/3585593599 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc66c035f00 0x7fc66c0383c0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fc68c136d00 tx=0x7fc674009040 comp rx=0 tx=0).stop 2024-04-26T00:39:49.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.754+0000 7fc6925e3640 10 monclient: shutdown 2024-04-26T00:39:49.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.754+0000 7fc6925e3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:49.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.754+0000 7fc6925e3640 1 -- 172.21.15.37:0/3585593599 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc68c136e10 msgr2=0x7fc68c135730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:49.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.754+0000 7fc6925e3640 1 --2- 172.21.15.37:0/3585593599 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc68c136e10 0x7fc68c135730 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc67c0ae040 tx=0x7fc67c01ab40 comp rx=0 tx=0).stop 2024-04-26T00:39:49.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.754+0000 7fc6925e3640 1 -- 172.21.15.37:0/3585593599 shutdown_connections 2024-04-26T00:39:49.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.754+0000 7fc6925e3640 1 --2- 172.21.15.37:0/3585593599 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc66c035f00 0x7fc66c0383c0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:49.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.754+0000 7fc6925e3640 1 --2- 172.21.15.37:0/3585593599 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc68c13c340 0x7fc68c137e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:49.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.754+0000 7fc6925e3640 1 --2- 172.21.15.37:0/3585593599 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc68c1371f0 0x7fc68c135c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:49.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.754+0000 7fc6925e3640 1 --2- 172.21.15.37:0/3585593599 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc68c136e10 0x7fc68c135730 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:49.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.758+0000 7fc6925e3640 1 -- 172.21.15.37:0/3585593599 >> 172.21.15.37:0/3585593599 conn(0x7fc68c0a64e0 msgr2=0x7fc68c0a7490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:49.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.758+0000 7fc6925e3640 1 -- 172.21.15.37:0/3585593599 shutdown_connections 2024-04-26T00:39:49.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.758+0000 7fc6925e3640 1 -- 172.21.15.37:0/3585593599 wait complete. 2024-04-26T00:39:49.773 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:39:49.773 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:39:49.774 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:39:49.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.898+0000 7f93d7ec9640 10 monclient: get_monmap_and_config 2024-04-26T00:39:49.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.898+0000 7f93d7ec9640 10 monclient: build_initial_monmap 2024-04-26T00:39:49.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.898+0000 7f93d7ec9640 10 monclient: monmap: 2024-04-26T00:39:49.899 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:49.899 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:49.899 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:39:49.901605+0000 2024-04-26T00:39:49.899 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:39:49.901605+0000 2024-04-26T00:39:49.899 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:49.899 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:49.899 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:39:49.899 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:39:49.899 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:39:49.899 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:39:49.899 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:39:49.900 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:49.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.898+0000 7f93d7ec9640 1 Processor -- start 2024-04-26T00:39:49.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.898+0000 7f93d7ec9640 1 -- start start 2024-04-26T00:39:49.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.898+0000 7f93d7ec9640 10 monclient: init 2024-04-26T00:39:49.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.898+0000 7f93d7ec9640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:49.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.898+0000 7f93d7ec9640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-04-26T00:39:49.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.898+0000 7f93d7ec9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f93d00726b0 addr v1:172.21.15.45:6789/0 2024-04-26T00:39:49.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.898+0000 7f93d7ec9640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f93d00a0110 0x7f93d013c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:49.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.898+0000 7f93d7ec9640 10 monclient(hunting): picked mon.noname-a con 0x7f93d00a0110 addr v2:172.21.15.37:3300/0 2024-04-26T00:39:49.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.898+0000 7f93d7ec9640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f93d013c630 0x7f93d013ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:49.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.898+0000 7f93d7ec9640 10 monclient(hunting): picked mon.noname-c con 0x7f93d013c630 addr v2:172.21.15.45:3300/0 2024-04-26T00:39:49.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.898+0000 7f93d7ec9640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93d00a1470 con 0x7f93d00726b0 2024-04-26T00:39:49.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.898+0000 7f93d7ec9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:49.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.898+0000 7f93d7ec9640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f93d00a1bd0 con 0x7f93d00a0110 2024-04-26T00:39:49.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.898+0000 7f93d7ec9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:49.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.898+0000 7f93d7ec9640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f93d00a0eb0 con 0x7f93d013c630 2024-04-26T00:39:49.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.898+0000 7f93d7ec9640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:49.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.898+0000 7f93d7ec9640 10 monclient(hunting): authenticate will time out at 2171.207288s 2024-04-26T00:39:49.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.898+0000 7f93d5464640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f93d00a0110 0x7f93d013c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:49.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.898+0000 7f93d5464640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f93d00a0110 0x7f93d013c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:45272/0 (socket says 172.21.15.37:45272) 2024-04-26T00:39:49.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.898+0000 7f93d5464640 1 -- 172.21.15.37:0/2289704206 learned_addr learned my addr 172.21.15.37:0/2289704206 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:49.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.898+0000 7f93d5464640 10 monclient(hunting): get_auth_request con 0x7f93d00a0110 auth_method 0 2024-04-26T00:39:49.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.898+0000 7f93d5464640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:49.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.898+0000 7f93d5464640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:49.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.898+0000 7f93d5464640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:49.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.898+0000 7f93d5464640 1 -- 172.21.15.37:0/2289704206 >> v2:172.21.15.37:3300/0 conn(0x7f93d00a0110 msgr2=0x7f93d013c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:39:49.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.898+0000 7f93d5464640 1 -- 172.21.15.37:0/2289704206 >> v2:172.21.15.37:3300/0 conn(0x7f93d00a0110 msgr2=0x7f93d013c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:39:49.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.898+0000 7f93d5464640 1 --2- 172.21.15.37:0/2289704206 >> v2:172.21.15.37:3300/0 conn(0x7f93d00a0110 0x7f93d013c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:39:49.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.898+0000 7f93d5464640 1 --2- 172.21.15.37:0/2289704206 >> v2:172.21.15.37:3300/0 conn(0x7f93d00a0110 0x7f93d013c0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:39:49.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.898+0000 7f93d6466640 1 --2- 172.21.15.37:0/2289704206 >> v2:172.21.15.45:3300/0 conn(0x7f93d013c630 0x7f93d013ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:49.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.898+0000 7f93d6466640 10 monclient(hunting): get_auth_request con 0x7f93d013c630 auth_method 0 2024-04-26T00:39:49.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.898+0000 7f93d6466640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:49.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.898+0000 7f93d6466640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:49.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.898+0000 7f93d6466640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:49.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.898+0000 7f93d6466640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:49.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.898+0000 7f93d6466640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:49.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.898+0000 7f93d6466640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:49.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.902+0000 7f93d6466640 10 monclient(hunting): handle_auth_done global_id 224257 payload 274 2024-04-26T00:39:49.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.902+0000 7f93d6466640 1 -- 172.21.15.37:0/2289704206 >> v2:172.21.15.37:3300/0 conn(0x7f93d00a0110 msgr2=0x7f93d013c0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:39:49.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.902+0000 7f93d6466640 1 --2- 172.21.15.37:0/2289704206 >> v2:172.21.15.37:3300/0 conn(0x7f93d00a0110 0x7f93d013c0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:49.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.902+0000 7f93d6466640 1 -- 172.21.15.37:0/2289704206 >> v1:172.21.15.45:6789/0 conn(0x7f93d00726b0 legacy=0x7f93d0072a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:49.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.902+0000 7f93d6466640 10 monclient: _finish_hunting 0 2024-04-26T00:39:49.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.902+0000 7f93d6466640 1 monclient: found mon.noname-c 2024-04-26T00:39:49.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.902+0000 7f93d6466640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:49.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.902+0000 7f93d6466640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:39:49.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.902+0000 7f93d6466640 1 -- 172.21.15.37:0/2289704206 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93d00a0bd0 con 0x7f93d013c630 2024-04-26T00:39:49.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d6466640 1 -- 172.21.15.37:0/2289704206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93d013c630 msgr2=0x7f93d013ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:49.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d6466640 1 --2- 172.21.15.37:0/2289704206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93d013c630 0x7f93d013ea20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f93d00a2da0 tx=0x7f93c801c5a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51db35c255e16c49 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:49.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d4c63640 1 -- 172.21.15.37:0/2289704206 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c80b0600 con 0x7f93d013c630 2024-04-26T00:39:49.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d4c63640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:49.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d4c63640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:39:49.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d4c63640 10 monclient: dump: 2024-04-26T00:39:49.906 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:49.906 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:49.906 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:49.906 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:49.906 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:49.907 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:49.907 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:49.907 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:49.907 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:49.907 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:49.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d4c63640 10 monclient: _finish_auth 0 2024-04-26T00:39:49.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d4c63640 10 monclient: _check_auth_tickets 2024-04-26T00:39:49.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d4c63640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:49.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d4c63640 1 -- 172.21.15.37:0/2289704206 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93c80b07a0 con 0x7f93d013c630 2024-04-26T00:39:49.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d7ec9640 5 monclient: authenticate success, global_id 224257 2024-04-26T00:39:49.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d7ec9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:39:49.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d4c63640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:49.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d4c63640 1 -- 172.21.15.37:0/2289704206 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c80b0b70 con 0x7f93d013c630 2024-04-26T00:39:49.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d7ec9640 10 monclient: get_monmap_and_config success 2024-04-26T00:39:49.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d7ec9640 4 set_mon_vals no callback set 2024-04-26T00:39:49.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d4c63640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:49.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d4c63640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:39:49.908 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d4c63640 10 monclient: dump: 2024-04-26T00:39:49.908 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:49.908 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:49.908 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:49.908 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:49.908 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:49.908 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:49.908 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:49.908 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:49.908 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:49.908 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:49.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d7ec9640 10 monclient: shutdown 2024-04-26T00:39:49.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d7ec9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:49.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d7ec9640 1 -- 172.21.15.37:0/2289704206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93d013c630 msgr2=0x7f93d013ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:49.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d7ec9640 1 --2- 172.21.15.37:0/2289704206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93d013c630 0x7f93d013ea20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f93d00a2da0 tx=0x7f93c801c5a0 comp rx=0 tx=0).stop 2024-04-26T00:39:49.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d7ec9640 1 -- 172.21.15.37:0/2289704206 shutdown_connections 2024-04-26T00:39:49.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d7ec9640 1 --2- 172.21.15.37:0/2289704206 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93d013c630 0x7f93d013ea20 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:49.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d7ec9640 1 --2- 172.21.15.37:0/2289704206 >> v2:172.21.15.37:3300/0 conn(0x7f93d00a0110 0x7f93d013c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:49.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d7ec9640 1 -- 172.21.15.37:0/2289704206 >> 172.21.15.37:0/2289704206 conn(0x7f93d00a1d50 msgr2=0x7f93d00a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:49.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d7ec9640 1 -- 172.21.15.37:0/2289704206 shutdown_connections 2024-04-26T00:39:49.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d7ec9640 1 -- 172.21.15.37:0/2289704206 wait complete. 2024-04-26T00:39:49.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d7ec9640 10 monclient: build_initial_monmap 2024-04-26T00:39:49.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d7ec9640 10 monclient: monmap: 2024-04-26T00:39:49.915 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:49.915 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:39:49.915 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:39:49.915 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:39:49.915 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:49.915 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:39:49.915 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:39:49.915 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:39:49.915 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:39:49.915 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:49.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d7ec9640 1 Processor -- start 2024-04-26T00:39:49.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d7ec9640 1 -- start start 2024-04-26T00:39:49.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d7ec9640 10 monclient: init 2024-04-26T00:39:49.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d7ec9640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:49.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d7ec9640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:39:49.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d7ec9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93d00726b0 0x7f93d01cee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:49.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d7ec9640 10 monclient(hunting): picked mon.noname-b con 0x7f93d00726b0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:39:49.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d7ec9640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d00a0110 0x7f93d0136c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:49.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d7ec9640 10 monclient(hunting): picked mon.noname-a con 0x7f93d00a0110 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:39:49.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d7ec9640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f93d013c630 0x7f93d011d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:49.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d7ec9640 10 monclient(hunting): picked mon.noname-c con 0x7f93d013c630 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:49.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d7ec9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:49.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d7ec9640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f93d0141900 con 0x7f93d00a0110 2024-04-26T00:39:49.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d7ec9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:49.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d7ec9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f93d0141780 con 0x7f93d00726b0 2024-04-26T00:39:49.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d7ec9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:49.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d7ec9640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f93d0141a80 con 0x7f93d013c630 2024-04-26T00:39:49.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d7ec9640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:49.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d7ec9640 10 monclient(hunting): authenticate will time out at 2171.215766s 2024-04-26T00:39:49.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d5464640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d00a0110 0x7f93d0136c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:49.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d6466640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f93d013c630 0x7f93d011d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:49.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d5464640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d00a0110 0x7f93d0136c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:45276/0 (socket says 172.21.15.37:45276) 2024-04-26T00:39:49.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d5464640 1 -- 172.21.15.37:0/2231529222 learned_addr learned my addr 172.21.15.37:0/2231529222 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:49.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d5464640 10 monclient(hunting): get_auth_request con 0x7f93d00a0110 auth_method 0 2024-04-26T00:39:49.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d5464640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:49.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d5464640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:49.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d5464640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:49.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d6466640 10 monclient(hunting): get_auth_request con 0x7f93d013c630 auth_method 0 2024-04-26T00:39:49.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d6466640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:49.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d6466640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:49.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.906+0000 7f93d6466640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:49.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93d6466640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:49.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93d6466640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:49.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93d5464640 1 -- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d00a0110 msgr2=0x7f93d0136c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:39:49.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93d5464640 1 -- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d00a0110 msgr2=0x7f93d0136c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:39:49.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93d5464640 1 --2- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d00a0110 0x7f93d0136c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:39:49.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93d6466640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:49.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93d5464640 1 --2- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d00a0110 0x7f93d0136c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:39:49.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93d5c65640 1 --2- 172.21.15.37:0/2231529222 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93d00726b0 0x7f93d01cee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:49.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93d5c65640 10 monclient(hunting): get_auth_request con 0x7f93d00726b0 auth_method 0 2024-04-26T00:39:49.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93d5c65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:49.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93d5c65640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:49.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93d5c65640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:49.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93d6466640 10 monclient(hunting): handle_auth_done global_id 204494 payload 824 2024-04-26T00:39:49.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93d6466640 1 -- 172.21.15.37:0/2231529222 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93d00726b0 msgr2=0x7f93d01cee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:49.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93d6466640 1 --2- 172.21.15.37:0/2231529222 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93d00726b0 0x7f93d01cee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:49.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93d6466640 1 -- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d00a0110 msgr2=0x7f93d0136c50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:39:49.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93d6466640 1 --2- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d00a0110 0x7f93d0136c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:49.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93d6466640 10 monclient: _finish_hunting 0 2024-04-26T00:39:49.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93d6466640 1 monclient: found mon.noname-c 2024-04-26T00:39:49.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93d6466640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:49.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93d6466640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:39:49.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93d6466640 1 -- 172.21.15.37:0/2231529222 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93d0121300 con 0x7f93d013c630 2024-04-26T00:39:49.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93d6466640 1 -- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f93d013c630 msgr2=0x7f93d011d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:49.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93d6466640 1 --2- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f93d013c630 0x7f93d011d650 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f93d00764d0 tx=0x7f93c8019340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:49.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93c67fc640 1 -- 172.21.15.37:0/2231529222 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c80a8810 con 0x7f93d013c630 2024-04-26T00:39:49.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:49.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93c67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:39:49.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93c67fc640 10 monclient: dump: 2024-04-26T00:39:49.919 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:49.919 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:49.919 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:49.919 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:49.919 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:49.919 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:49.919 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:49.920 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:49.920 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:49.920 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:49.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93c67fc640 10 monclient: _finish_auth 0 2024-04-26T00:39:49.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93c67fc640 10 monclient: _check_auth_tickets 2024-04-26T00:39:49.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:49.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93c67fc640 1 -- 172.21.15.37:0/2231529222 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93c80a89b0 con 0x7f93d013c630 2024-04-26T00:39:49.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93c67fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:49.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93c67fc640 1 -- 172.21.15.37:0/2231529222 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c80a8d80 con 0x7f93d013c630 2024-04-26T00:39:49.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:49.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93c67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:49.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93c67fc640 10 monclient: dump: 2024-04-26T00:39:49.920 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:49.920 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:49.920 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:49.921 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:49.921 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:49.921 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:49.921 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:49.921 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:49.921 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:49.921 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:49.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93c7fff640 4 set_mon_vals no callback set 2024-04-26T00:39:49.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93d7ec9640 5 monclient: authenticate success, global_id 204494 2024-04-26T00:39:49.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93d7ec9640 10 monclient: _renew_subs 2024-04-26T00:39:49.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93d7ec9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:49.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93d7ec9640 1 -- 172.21.15.37:0/2231529222 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93d01215e0 con 0x7f93d013c630 2024-04-26T00:39:49.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93d7ec9640 10 monclient: _renew_subs 2024-04-26T00:39:49.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93d7ec9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:49.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93d7ec9640 1 -- 172.21.15.37:0/2231529222 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f93d011f2f0 con 0x7f93d013c630 2024-04-26T00:39:49.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93d7ec9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:49.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93d7ec9640 1 --2- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d0137190 0x7f93d011db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:49.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93d7ec9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:49.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.910+0000 7f93d7ec9640 1 -- 172.21.15.37:0/2231529222 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f93d0137c50 con 0x7f93d0137190 2024-04-26T00:39:49.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.914+0000 7f93d5c65640 1 --2- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d0137190 0x7f93d011db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:49.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.914+0000 7f93d5c65640 10 monclient: get_auth_request con 0x7f93d0137190 auth_method 0 2024-04-26T00:39:49.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.914+0000 7f93d5c65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:49.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.914+0000 7f93d5c65640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:49.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.914+0000 7f93d5c65640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:49.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.914+0000 7f93d5c65640 1 -- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d0137190 msgr2=0x7f93d011db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:39:49.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.914+0000 7f93d5c65640 1 -- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d0137190 msgr2=0x7f93d011db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:39:49.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.914+0000 7f93d5c65640 1 --2- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d0137190 0x7f93d011db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:39:49.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.914+0000 7f93d5c65640 1 --2- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d0137190 0x7f93d011db90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:39:49.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.914+0000 7f93c67fc640 1 -- 172.21.15.37:0/2231529222 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f93c80c2020 con 0x7f93d013c630 2024-04-26T00:39:49.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.914+0000 7f93c67fc640 1 --2- 172.21.15.37:0/2231529222 >> v2:172.21.15.45:6812/2379918623 conn(0x7f93b4035e80 0x7f93b4038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:49.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.914+0000 7f93c67fc640 1 -- 172.21.15.37:0/2231529222 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(786..786 src has 705..786) ==== 4003+0+0 (secure 0 0 0) 0x7f93c80e3d40 con 0x7f93d013c630 2024-04-26T00:39:49.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.914+0000 7f93d5464640 1 --2- 172.21.15.37:0/2231529222 >> v2:172.21.15.45:6812/2379918623 conn(0x7f93b4035e80 0x7f93b4038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:49.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.914+0000 7f93d5464640 10 monclient: get_auth_request con 0x7f93b4035e80 auth_method 0 2024-04-26T00:39:49.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:49.914+0000 7f93d5464640 1 --2- 172.21.15.37:0/2231529222 >> v2:172.21.15.45:6812/2379918623 conn(0x7f93b4035e80 0x7f93b4038340 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f93d011eef0 tx=0x7f93c0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:50.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:50.118+0000 7f93d5c65640 1 --2- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d0137190 0x7f93d011db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:50.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:50.118+0000 7f93d5c65640 10 monclient: get_auth_request con 0x7f93d0137190 auth_method 0 2024-04-26T00:39:50.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:50.118+0000 7f93d5c65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:50.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:50.118+0000 7f93d5c65640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:50.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:50.118+0000 7f93d5c65640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:39:50.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:50.118+0000 7f93d5c65640 1 -- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d0137190 msgr2=0x7f93d011db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:39:50.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:50.118+0000 7f93d5c65640 1 -- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d0137190 msgr2=0x7f93d011db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:39:50.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:50.118+0000 7f93d5c65640 1 --2- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d0137190 0x7f93d011db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:39:50.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:50.118+0000 7f93d5c65640 1 --2- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d0137190 0x7f93d011db90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:39:50.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:50.522+0000 7f93d5c65640 1 --2- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d0137190 0x7f93d011db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:50.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:50.522+0000 7f93d5c65640 10 monclient: get_auth_request con 0x7f93d0137190 auth_method 0 2024-04-26T00:39:50.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:50.522+0000 7f93d5c65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:50.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:50.522+0000 7f93d5c65640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:50.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:50.522+0000 7f93d5c65640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:39:50.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:50.522+0000 7f93d5c65640 1 -- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d0137190 msgr2=0x7f93d011db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:39:50.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:50.522+0000 7f93d5c65640 1 -- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d0137190 msgr2=0x7f93d011db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:39:50.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:50.522+0000 7f93d5c65640 1 --2- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d0137190 0x7f93d011db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:39:50.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:50.522+0000 7f93d5c65640 1 --2- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d0137190 0x7f93d011db90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:39:50.823 INFO:tasks.rados.rados.0.smithi045.stdout:3469: done (6 left) 2024-04-26T00:39:50.823 INFO:tasks.rados.rados.0.smithi045.stdout:3470: done (5 left) 2024-04-26T00:39:50.823 INFO:tasks.rados.rados.0.smithi045.stdout:3472: done (4 left) 2024-04-26T00:39:50.823 INFO:tasks.rados.rados.0.smithi045.stdout:3473: done (3 left) 2024-04-26T00:39:50.824 INFO:tasks.rados.rados.0.smithi045.stdout:3474: done (2 left) 2024-04-26T00:39:50.824 INFO:tasks.rados.rados.0.smithi045.stdout:3475: done (1 left) 2024-04-26T00:39:50.824 INFO:tasks.rados.rados.0.smithi045.stdout:3476: done (0 left) 2024-04-26T00:39:50.824 INFO:tasks.rados.rados.0.smithi045.stdout:3477: delete oid 32 current snap is 385 2024-04-26T00:39:50.837 INFO:tasks.rados.rados.0.smithi045.stdout:3477: done (0 left) 2024-04-26T00:39:50.837 INFO:tasks.rados.rados.0.smithi045.stdout:3478: read oid 43 snap -1 2024-04-26T00:39:50.837 INFO:tasks.rados.rados.0.smithi045.stdout:3478: expect (ObjNum 1214 snap 378 seq_num 1214) 2024-04-26T00:39:50.837 INFO:tasks.rados.rados.0.smithi045.stdout:3479: write (excl) oid 7 current snap is 385 2024-04-26T00:39:50.837 INFO:tasks.rados.rados.0.smithi045.stdout:3479: seq_num 1241 ranges {457487=749548,1770389=484435,2746287=111200} 2024-04-26T00:39:50.847 INFO:tasks.rados.rados.0.smithi045.stdout:3479: writing smithi04526579-7 from 457487 to 1207035 tid 1 2024-04-26T00:39:50.855 INFO:tasks.rados.rados.0.smithi045.stdout:3479: writing smithi04526579-7 from 1770389 to 2254824 tid 2 2024-04-26T00:39:50.861 INFO:tasks.rados.rados.0.smithi045.stdout:3479: writing smithi04526579-7 from 2746287 to 2857487 tid 3 2024-04-26T00:39:50.861 INFO:tasks.rados.rados.0.smithi045.stdout:3480: copy_from oid 1 from oid 31 current snap is 385 2024-04-26T00:39:50.861 INFO:tasks.rados.rados.0.smithi045.stdout:3478: expect (ObjNum 1214 snap 378 seq_num 1214) 2024-04-26T00:39:50.992 INFO:tasks.rados.rados.0.smithi045.stdout:3479: finishing write tid 1 to smithi04526579-7 2024-04-26T00:39:50.992 INFO:tasks.rados.rados.0.smithi045.stdout:3480: finishing copy_from to smithi04526579-1 2024-04-26T00:39:50.992 INFO:tasks.rados.rados.0.smithi045.stdout:3480: got expected ENOENT (src dne) 2024-04-26T00:39:50.992 INFO:tasks.rados.rados.0.smithi045.stdout:3480: finishing copy_from racing read to smithi04526579-1 2024-04-26T00:39:50.992 INFO:tasks.rados.rados.0.smithi045.stdout:3479: finishing write tid 2 to smithi04526579-7 2024-04-26T00:39:50.992 INFO:tasks.rados.rados.0.smithi045.stdout:3479: finishing write tid 3 to smithi04526579-7 2024-04-26T00:39:50.992 INFO:tasks.rados.rados.0.smithi045.stdout:3479: finishing write tid 4 to smithi04526579-7 2024-04-26T00:39:50.992 INFO:tasks.rados.rados.0.smithi045.stdout:3479: finishing write tid 5 to smithi04526579-7 2024-04-26T00:39:50.992 INFO:tasks.rados.rados.0.smithi045.stdout:3479: oid 7 updating version 0 to 2630 2024-04-26T00:39:50.992 INFO:tasks.rados.rados.0.smithi045.stdout:3479: oid 7 version 2630 is already newer than 2629 2024-04-26T00:39:50.992 INFO:tasks.rados.rados.0.smithi045.stdout:3479: oid 7 version 2630 is already newer than 2628 2024-04-26T00:39:50.992 INFO:tasks.rados.rados.0.smithi045.stdout:3479: oid 7 updating version 2630 to 2631 2024-04-26T00:39:50.993 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 7 v 2631 (ObjNum 1241 snap 385 seq_num 1241) dirty exists 2024-04-26T00:39:50.993 INFO:tasks.rados.rados.0.smithi045.stdout:3479: left oid 7 (ObjNum 1241 snap 385 seq_num 1241) 2024-04-26T00:39:50.993 INFO:tasks.rados.rados.0.smithi045.stdout:3478: done (2 left) 2024-04-26T00:39:50.993 INFO:tasks.rados.rados.0.smithi045.stdout:3479: done (1 left) 2024-04-26T00:39:50.993 INFO:tasks.rados.rados.0.smithi045.stdout:3480: done (0 left) 2024-04-26T00:39:50.993 INFO:tasks.rados.rados.0.smithi045.stdout:3481: rollback oid 16 current snap is 385 2024-04-26T00:39:50.993 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 16 to 332 2024-04-26T00:39:50.993 INFO:tasks.rados.rados.0.smithi045.stdout:3482: snap_create 2024-04-26T00:39:51.005 INFO:tasks.rados.rados.0.smithi045.stdout:3481: finishing rollback tid 0 to smithi04526579-16 2024-04-26T00:39:51.009 INFO:tasks.rados.rados.0.smithi045.stdout:3481: finishing rollback tid 1 to smithi04526579-16 2024-04-26T00:39:51.009 INFO:tasks.rados.rados.0.smithi045.stdout:3481: finishing rollback tid 2 to smithi04526579-16 2024-04-26T00:39:51.009 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 1207 (ObjNum 1102 snap 329 seq_num 1102) dirty exists 2024-04-26T00:39:51.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:51.326+0000 7f93d5c65640 1 --2- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d0137190 0x7f93d011db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:51.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:51.326+0000 7f93d5c65640 10 monclient: get_auth_request con 0x7f93d0137190 auth_method 0 2024-04-26T00:39:51.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:51.326+0000 7f93d5c65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:51.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:51.326+0000 7f93d5c65640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:51.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:51.326+0000 7f93d5c65640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:39:51.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:51.326+0000 7f93d5c65640 1 -- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d0137190 msgr2=0x7f93d011db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:39:51.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:51.326+0000 7f93d5c65640 1 -- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d0137190 msgr2=0x7f93d011db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:39:51.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:51.326+0000 7f93d5c65640 1 --2- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d0137190 0x7f93d011db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:39:51.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:51.326+0000 7f93d5c65640 1 --2- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d0137190 0x7f93d011db90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:39:51.825 INFO:tasks.rados.rados.0.smithi045.stdout:3481: done (1 left) 2024-04-26T00:39:51.825 INFO:tasks.rados.rados.0.smithi045.stdout:3482: done (0 left) 2024-04-26T00:39:51.825 INFO:tasks.rados.rados.0.smithi045.stdout:3483: snap_create 2024-04-26T00:39:52.834 INFO:tasks.rados.rados.0.smithi045.stdout:3483: done (0 left) 2024-04-26T00:39:52.834 INFO:tasks.rados.rados.0.smithi045.stdout:3484: read oid 24 snap -1 2024-04-26T00:39:52.834 INFO:tasks.rados.rados.0.smithi045.stdout:3484: expect (ObjNum 1211 snap 375 seq_num 1211) 2024-04-26T00:39:52.835 INFO:tasks.rados.rados.0.smithi045.stdout:3485: delete oid 43 current snap is 387 2024-04-26T00:39:52.840 INFO:tasks.rados.rados.0.smithi045.stdout:3485: done (1 left) 2024-04-26T00:39:52.841 INFO:tasks.rados.rados.0.smithi045.stdout:3486: snap_remove snap 382 2024-04-26T00:39:52.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.906+0000 7f93c57fa640 10 monclient: tick 2024-04-26T00:39:52.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.906+0000 7f93c57fa640 10 monclient: _check_auth_tickets 2024-04-26T00:39:52.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.906+0000 7f93c57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:52.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.906+0000 7f93c57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:52.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.930+0000 7f93d5c65640 1 --2- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d0137190 0x7f93d011db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:52.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.930+0000 7f93d5c65640 10 monclient: get_auth_request con 0x7f93d0137190 auth_method 0 2024-04-26T00:39:52.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.930+0000 7f93d5c65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:52.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.930+0000 7f93d5c65640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:52.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.930+0000 7f93d5c65640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:39:52.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.930+0000 7f93d5c65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:52.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.930+0000 7f93d5c65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:52.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.930+0000 7f93d5c65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:52.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.930+0000 7f93d5c65640 10 monclient(hunting): handle_auth_done global_id 244098 payload 824 2024-04-26T00:39:52.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.930+0000 7f93d5c65640 1 -- 172.21.15.37:0/2231529222 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f93d004f510 con 0x7f93d0137190 2024-04-26T00:39:52.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.930+0000 7f93d5c65640 1 -- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d0137190 msgr2=0x7f93d011db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:52.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.930+0000 7f93d5c65640 1 --2- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d0137190 0x7f93d011db90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f93cc015580 tx=0x7f93cc015a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:52.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.930+0000 7f93c67fc640 1 -- 172.21.15.37:0/2231529222 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93cc01b070 con 0x7f93d0137190 2024-04-26T00:39:52.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.930+0000 7f93c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:52.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.930+0000 7f93c67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:39:52.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.930+0000 7f93c67fc640 10 monclient: dump: 2024-04-26T00:39:52.932 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:52.932 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:52.932 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:52.932 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:52.932 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:52.932 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:52.932 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:52.932 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:52.932 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:52.932 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:52.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.930+0000 7f93c67fc640 1 -- 172.21.15.37:0/2231529222 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f93cc013970 con 0x7f93d0137190 2024-04-26T00:39:52.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.930+0000 7f93c67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:52.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.930+0000 7f93c67fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:39:52.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.930+0000 7f93c67fc640 1 -- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d0137190 msgr2=0x7f93d011db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:52.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.930+0000 7f93c67fc640 1 --2- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d0137190 0x7f93d011db90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f93cc015580 tx=0x7f93cc015a40 comp rx=0 tx=0).stop 2024-04-26T00:39:52.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.930+0000 7f93c67fc640 1 -- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d0137190 msgr2=0x7f93d011db90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:39:52.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.930+0000 7f93c67fc640 1 --2- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d0137190 0x7f93d011db90 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f93cc015580 tx=0x7f93cc015a40 comp rx=0 tx=0).stop 2024-04-26T00:39:52.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.938+0000 7f93d7ec9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:39:52.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.938+0000 7f93d7ec9640 1 --2- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d0120070 0x7f93d011e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:52.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.938+0000 7f93d7ec9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:52.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.938+0000 7f93d7ec9640 1 -- 172.21.15.37:0/2231529222 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f93d006ba50 con 0x7f93d0120070 2024-04-26T00:39:52.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.942+0000 7f93d5c65640 1 --2- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d0120070 0x7f93d011e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:52.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.942+0000 7f93d5c65640 10 monclient: get_auth_request con 0x7f93d0120070 auth_method 0 2024-04-26T00:39:52.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.942+0000 7f93d5c65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:52.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.942+0000 7f93d5c65640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:52.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.942+0000 7f93d5c65640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:52.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.942+0000 7f93d5c65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:52.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.942+0000 7f93d5c65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:52.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.942+0000 7f93d5c65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:52.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.942+0000 7f93d5c65640 10 monclient(hunting): handle_auth_done global_id 244101 payload 824 2024-04-26T00:39:52.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.942+0000 7f93d5c65640 1 -- 172.21.15.37:0/2231529222 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f93d0120d40 con 0x7f93d0120070 2024-04-26T00:39:52.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.942+0000 7f93d5c65640 1 -- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d0120070 msgr2=0x7f93d011e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:52.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.942+0000 7f93d5c65640 1 --2- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d0120070 0x7f93d011e0d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f93cc01d000 tx=0x7f93cc003770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:52.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.942+0000 7f93c67fc640 1 -- 172.21.15.37:0/2231529222 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93cc007640 con 0x7f93d0120070 2024-04-26T00:39:52.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.942+0000 7f93c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:52.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.942+0000 7f93c67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:39:52.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.942+0000 7f93c67fc640 10 monclient: dump: 2024-04-26T00:39:52.945 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:52.945 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:52.945 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:52.945 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:52.945 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:52.945 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:52.945 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:52.945 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:52.945 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:52.945 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:52.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.946+0000 7f93c67fc640 1 -- 172.21.15.37:0/2231529222 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4547 (secure 0 0 0) 0x7f93cc006e60 con 0x7f93d0120070 2024-04-26T00:39:52.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.946+0000 7f93c67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:39:52.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.946+0000 7f93c67fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:39:52.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.946+0000 7f93c67fc640 1 -- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d0120070 msgr2=0x7f93d011e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:52.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.946+0000 7f93c67fc640 1 --2- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d0120070 0x7f93d011e0d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f93cc01d000 tx=0x7f93cc003770 comp rx=0 tx=0).stop 2024-04-26T00:39:52.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.946+0000 7f93c67fc640 1 -- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d0120070 msgr2=0x7f93d011e0d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:39:52.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.946+0000 7f93c67fc640 1 --2- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93d0120070 0x7f93d011e0d0 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f93cc01d000 tx=0x7f93cc003770 comp rx=0 tx=0).stop 2024-04-26T00:39:52.947 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:39:52.947 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:39:52.947 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:39:52.947 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:39:52.947 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 148, 2024-04-26T00:39:52.947 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 10299, 2024-04-26T00:39:52.947 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:39:52.947 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:39:52.947 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:39:52.947 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:39:52.947 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:52.947 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 3, 2024-04-26T00:39:52.948 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:39:52.948 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:39:52.948 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:39:52.948 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:52.948 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:52.948 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:52.948 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:52.948 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:52.948 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:52.948 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:52.948 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:52.948 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:52.948 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:52.948 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:52.949 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:52.949 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:39:52.949 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:39:52.949 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:52.949 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:52.949 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:52.949 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:52.949 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:52.949 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:52.949 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:52.949 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:52.949 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:52.949 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:52.949 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:52.949 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:52.950 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:52.950 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:39:52.950 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:39:52.950 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:39:52.950 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:39:52.950 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:39:52.950 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:39:52.950 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:39:52.950 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:39:52.950 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:39:52.950 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:39:52.950 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:39:52.950 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:39:52.950 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:39:52.951 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:39:52.951 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:39:52.951 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:39:52.951 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:39:52.951 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:52.951 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:52.951 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:52.951 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:52.951 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:52.951 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:52.951 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:52.951 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:52.951 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:52.951 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:52.952 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:52.952 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:52.952 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:39:52.952 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:52.952 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:39:52.952 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:52.952 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:39:52.952 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:39:52.952 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:52.952 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:52.952 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:52.952 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:52.952 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:39:52.952 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:52.952 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:52.953 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:52.953 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:52.953 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:39:52.953 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:52.953 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:52.953 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:52.953 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:52.953 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:39:52.953 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:39:52.953 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:52.953 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:52.953 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:52.953 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:52.953 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:52.953 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:39:52.954 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:39:52.954 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:52.954 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:52.954 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:52.954 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:52.954 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:39:52.954 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:52.954 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:52.954 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:52.954 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:52.954 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:39:52.954 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:52.954 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:52.954 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:52.955 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:52.955 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:39:52.955 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:39:52.955 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:52.955 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:52.955 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:52.955 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:52.955 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:52.955 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:39:52.955 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:39:52.955 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:52.955 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:52.955 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:52.956 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:52.956 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:39:52.956 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:52.956 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:52.956 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:52.956 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:52.956 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:39:52.956 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:52.956 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:52.956 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:52.956 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:52.956 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:39:52.956 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:39:52.956 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:52.957 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:52.957 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:52.957 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:52.957 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:52.957 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:52.957 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:39:52.957 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:39:52.957 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:52.957 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:52.957 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:52.957 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:39:52.957 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:52.957 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:52.957 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:39:52.957 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:52.958 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:52.958 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:52.958 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:39:52.958 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:52.958 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:52.958 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:52.958 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:39:52.958 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:39:52.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.946+0000 7f93d7ec9640 1 -- 172.21.15.37:0/2231529222 >> v2:172.21.15.45:6812/2379918623 conn(0x7f93b4035e80 msgr2=0x7f93b4038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:52.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.946+0000 7f93d7ec9640 1 --2- 172.21.15.37:0/2231529222 >> v2:172.21.15.45:6812/2379918623 conn(0x7f93b4035e80 0x7f93b4038340 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f93d011eef0 tx=0x7f93c0008040 comp rx=0 tx=0).stop 2024-04-26T00:39:52.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.946+0000 7f93d7ec9640 10 monclient: shutdown 2024-04-26T00:39:52.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.946+0000 7f93d7ec9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:52.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.946+0000 7f93d7ec9640 1 -- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f93d013c630 msgr2=0x7f93d011d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:52.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.946+0000 7f93d6466640 1 -- 172.21.15.37:0/2231529222 reap_dead start 2024-04-26T00:39:52.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.946+0000 7f93d7ec9640 1 --2- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f93d013c630 0x7f93d011d650 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f93d00764d0 tx=0x7f93c8019340 comp rx=0 tx=0).stop 2024-04-26T00:39:52.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.946+0000 7f93d7ec9640 1 -- 172.21.15.37:0/2231529222 shutdown_connections 2024-04-26T00:39:52.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.946+0000 7f93d7ec9640 1 --2- 172.21.15.37:0/2231529222 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f93d013c630 0x7f93d011d650 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:52.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.946+0000 7f93d7ec9640 1 -- 172.21.15.37:0/2231529222 >> 172.21.15.37:0/2231529222 conn(0x7f93d00a1d50 msgr2=0x7f93d00a48b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:52.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.946+0000 7f93d7ec9640 1 -- 172.21.15.37:0/2231529222 shutdown_connections 2024-04-26T00:39:52.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:52.946+0000 7f93d7ec9640 1 -- 172.21.15.37:0/2231529222 wait complete. 2024-04-26T00:39:52.973 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:39:52.973 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:39:52.973 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:39:53.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9aa288f640 10 monclient: get_monmap_and_config 2024-04-26T00:39:53.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9aa288f640 10 monclient: build_initial_monmap 2024-04-26T00:39:53.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9aa288f640 10 monclient: monmap: 2024-04-26T00:39:53.105 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:53.106 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:53.106 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:39:53.108289+0000 2024-04-26T00:39:53.106 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:39:53.108289+0000 2024-04-26T00:39:53.106 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:53.106 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:53.106 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:39:53.106 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:39:53.106 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:39:53.106 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:39:53.106 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:39:53.106 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:53.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9aa288f640 1 Processor -- start 2024-04-26T00:39:53.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9aa288f640 1 -- start start 2024-04-26T00:39:53.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9aa288f640 10 monclient: init 2024-04-26T00:39:53.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9aa288f640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:53.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9aa288f640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-04-26T00:39:53.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9aa288f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9a9c09e220 addr v1:172.21.15.37:6789/0 2024-04-26T00:39:53.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9aa288f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a9c0a14f0 0x7f9a9c0a0360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:53.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9aa288f640 10 monclient(hunting): picked mon.noname-b con 0x7f9a9c0a14f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:53.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9aa288f640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9a9c09e600 0x7f9a9c13aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:53.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9aa288f640 10 monclient(hunting): picked mon.noname-c con 0x7f9a9c09e600 addr v2:172.21.15.45:3300/0 2024-04-26T00:39:53.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9aa288f640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a9c0a12b0 con 0x7f9a9c09e220 2024-04-26T00:39:53.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9aa288f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:53.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9aa288f640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f9a9c09cd10 con 0x7f9a9c09e600 2024-04-26T00:39:53.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9aa288f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:53.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9aa288f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9a9c09da20 con 0x7f9a9c0a14f0 2024-04-26T00:39:53.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9aa288f640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:53.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9aa288f640 10 monclient(hunting): authenticate will time out at 2174.414162s 2024-04-26T00:39:53.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9a9bfff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f9a9c09e220 0x7f9a9c09fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:43266/0 (socket says 172.21.15.37:43266) 2024-04-26T00:39:53.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9a9bfff640 1 -- 172.21.15.37:0/1392090229 learned_addr learned my addr 172.21.15.37:0/1392090229 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:39:53.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9a9b7fe640 1 --2- 172.21.15.37:0/1392090229 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a9c0a14f0 0x7f9a9c0a0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:53.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9a9b7fe640 10 monclient(hunting): get_auth_request con 0x7f9a9c0a14f0 auth_method 0 2024-04-26T00:39:53.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9a9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:53.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9a9b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:53.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9a9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:53.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9aa0e2c640 1 --2- 172.21.15.37:0/1392090229 >> v2:172.21.15.45:3300/0 conn(0x7f9a9c09e600 0x7f9a9c13aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:53.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9a9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:53.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9a9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:53.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9a9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:53.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9aa0e2c640 10 monclient(hunting): get_auth_request con 0x7f9a9c09e600 auth_method 0 2024-04-26T00:39:53.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9aa0e2c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:53.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9aa0e2c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:53.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9aa0e2c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:53.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9aa0e2c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:53.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9aa0e2c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:53.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9aa0e2c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:53.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9a9b7fe640 10 monclient(hunting): handle_auth_done global_id 204497 payload 274 2024-04-26T00:39:53.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9a9b7fe640 1 -- 172.21.15.37:0/1392090229 >> v2:172.21.15.45:3300/0 conn(0x7f9a9c09e600 msgr2=0x7f9a9c13aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:53.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9a9b7fe640 1 --2- 172.21.15.37:0/1392090229 >> v2:172.21.15.45:3300/0 conn(0x7f9a9c09e600 0x7f9a9c13aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:53.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9a9b7fe640 1 -- 172.21.15.37:0/1392090229 >> v1:172.21.15.37:6789/0 conn(0x7f9a9c09e220 legacy=0x7f9a9c09fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:53.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9a9b7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:39:53.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9a9b7fe640 1 monclient: found mon.noname-b 2024-04-26T00:39:53.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9a9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:53.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9a9b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:39:53.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.106+0000 7f9a9b7fe640 1 -- 172.21.15.37:0/1392090229 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a9c09dd00 con 0x7f9a9c0a14f0 2024-04-26T00:39:53.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.110+0000 7f9a9b7fe640 1 -- 172.21.15.37:0/1392090229 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a9c0a14f0 msgr2=0x7f9a9c0a0360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:53.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.110+0000 7f9a9b7fe640 1 --2- 172.21.15.37:0/1392090229 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a9c0a14f0 0x7f9a9c0a0360 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9a900097c0 tx=0x7f9a9001c320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=942643376777380d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:53.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.110+0000 7f9a9affd640 1 -- 172.21.15.37:0/1392090229 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a900b0070 con 0x7f9a9c0a14f0 2024-04-26T00:39:53.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.110+0000 7f9a9affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:53.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.110+0000 7f9a9affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:39:53.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.110+0000 7f9a9affd640 10 monclient: dump: 2024-04-26T00:39:53.114 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:53.114 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:53.114 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:53.114 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:53.115 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:53.115 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:53.115 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:53.115 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:53.115 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:53.115 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:53.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.110+0000 7f9a9affd640 10 monclient: _finish_auth 0 2024-04-26T00:39:53.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.110+0000 7f9a9affd640 10 monclient: _check_auth_tickets 2024-04-26T00:39:53.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.110+0000 7f9a9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:53.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.110+0000 7f9a9affd640 1 -- 172.21.15.37:0/1392090229 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a9c0a0ab0 con 0x7f9a9c0a14f0 2024-04-26T00:39:53.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.110+0000 7f9a9affd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:53.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.110+0000 7f9a9affd640 1 -- 172.21.15.37:0/1392090229 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a900ab820 con 0x7f9a9c0a14f0 2024-04-26T00:39:53.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.110+0000 7f9aa288f640 5 monclient: authenticate success, global_id 204497 2024-04-26T00:39:53.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.110+0000 7f9aa288f640 10 monclient: get_monmap_and_config success 2024-04-26T00:39:53.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.110+0000 7f9aa288f640 4 set_mon_vals no callback set 2024-04-26T00:39:53.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.110+0000 7f9a9affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:53.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.110+0000 7f9a9affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:53.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.110+0000 7f9a9affd640 10 monclient: dump: 2024-04-26T00:39:53.116 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:53.116 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:53.116 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:53.116 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:53.116 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:53.116 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:53.116 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:53.116 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:53.117 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:53.117 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:53.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9aa288f640 10 monclient: shutdown 2024-04-26T00:39:53.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9aa288f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:53.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9aa288f640 1 -- 172.21.15.37:0/1392090229 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a9c0a14f0 msgr2=0x7f9a9c0a0360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:53.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9aa288f640 1 --2- 172.21.15.37:0/1392090229 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a9c0a14f0 0x7f9a9c0a0360 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9a900097c0 tx=0x7f9a9001c320 comp rx=0 tx=0).stop 2024-04-26T00:39:53.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9aa288f640 1 -- 172.21.15.37:0/1392090229 shutdown_connections 2024-04-26T00:39:53.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9aa288f640 1 --2- 172.21.15.37:0/1392090229 >> v2:172.21.15.45:3300/0 conn(0x7f9a9c09e600 0x7f9a9c13aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:53.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9aa288f640 1 --2- 172.21.15.37:0/1392090229 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a9c0a14f0 0x7f9a9c0a0360 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:53.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9aa288f640 1 -- 172.21.15.37:0/1392090229 >> 172.21.15.37:0/1392090229 conn(0x7f9a9c0a1d50 msgr2=0x7f9a9c0a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:53.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9aa288f640 1 -- 172.21.15.37:0/1392090229 shutdown_connections 2024-04-26T00:39:53.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9aa288f640 1 -- 172.21.15.37:0/1392090229 wait complete. 2024-04-26T00:39:53.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9aa288f640 10 monclient: build_initial_monmap 2024-04-26T00:39:53.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9aa288f640 10 monclient: monmap: 2024-04-26T00:39:53.117 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:53.118 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:39:53.118 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:39:53.118 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:39:53.118 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:53.118 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:39:53.118 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:39:53.118 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:39:53.118 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:39:53.118 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:53.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9aa288f640 1 Processor -- start 2024-04-26T00:39:53.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9aa288f640 1 -- start start 2024-04-26T00:39:53.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9aa288f640 10 monclient: init 2024-04-26T00:39:53.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9aa288f640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:53.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9aa288f640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:39:53.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9aa288f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9a9c09e220 0x7f9a9c11b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:53.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9aa288f640 10 monclient(hunting): picked mon.noname-a con 0x7f9a9c09e220 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:39:53.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9aa288f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a9c09e600 0x7f9a9c11bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:53.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9aa288f640 10 monclient(hunting): picked mon.noname-c con 0x7f9a9c09e600 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:53.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9aa288f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a9c0a14f0 0x7f9a9c11c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:53.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9aa288f640 10 monclient(hunting): picked mon.noname-b con 0x7f9a9c0a14f0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:39:53.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9aa288f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:53.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9aa288f640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9c13f790 con 0x7f9a9c09e220 2024-04-26T00:39:53.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9aa288f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:53.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9aa288f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9c13f610 con 0x7f9a9c0a14f0 2024-04-26T00:39:53.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9aa288f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:53.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9aa288f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9a9c13f910 con 0x7f9a9c09e600 2024-04-26T00:39:53.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9aa288f640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:53.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9aa288f640 10 monclient(hunting): authenticate will time out at 2174.422705s 2024-04-26T00:39:53.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9a9b7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a9c09e600 0x7f9a9c11bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:53.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9a9bfff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9a9c09e220 0x7f9a9c11b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:53.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9a9b7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a9c09e600 0x7f9a9c11bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:37964/0 (socket says 172.21.15.37:37964) 2024-04-26T00:39:53.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9a9bfff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9a9c09e220 0x7f9a9c11b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:45338/0 (socket says 172.21.15.37:45338) 2024-04-26T00:39:53.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9a9b7fe640 1 -- 172.21.15.37:0/1067067325 learned_addr learned my addr 172.21.15.37:0/1067067325 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:53.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9a9b7fe640 10 monclient(hunting): get_auth_request con 0x7f9a9c09e600 auth_method 0 2024-04-26T00:39:53.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9a9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:53.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9a9b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:53.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9a9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:53.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9a9bfff640 10 monclient(hunting): get_auth_request con 0x7f9a9c09e220 auth_method 0 2024-04-26T00:39:53.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9a9bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:53.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9a9bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:53.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9a9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:53.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9a9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:53.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9a9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:53.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9a9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:53.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9a9bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:53.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9a9bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:53.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9a9bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:53.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9aa0e2c640 1 --2- 172.21.15.37:0/1067067325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a9c0a14f0 0x7f9a9c11c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:53.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9aa0e2c640 10 monclient(hunting): get_auth_request con 0x7f9a9c0a14f0 auth_method 0 2024-04-26T00:39:53.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9aa0e2c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:53.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9aa0e2c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:53.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9aa0e2c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:53.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9a9b7fe640 10 monclient(hunting): handle_auth_done global_id 204500 payload 824 2024-04-26T00:39:53.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9a9b7fe640 1 -- 172.21.15.37:0/1067067325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a9c0a14f0 msgr2=0x7f9a9c11c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:53.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9a9b7fe640 1 --2- 172.21.15.37:0/1067067325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9a9c0a14f0 0x7f9a9c11c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:53.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9a9b7fe640 1 -- 172.21.15.37:0/1067067325 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9a9c09e220 msgr2=0x7f9a9c11b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:53.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9a9b7fe640 1 --2- 172.21.15.37:0/1067067325 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9a9c09e220 0x7f9a9c11b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:53.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9a9b7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:39:53.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9a9b7fe640 1 monclient: found mon.noname-c 2024-04-26T00:39:53.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9a9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:53.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9a9b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:39:53.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.114+0000 7f9a9b7fe640 1 -- 172.21.15.37:0/1067067325 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a9c072990 con 0x7f9a9c09e600 2024-04-26T00:39:53.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9a9b7fe640 1 -- 172.21.15.37:0/1067067325 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a9c09e600 msgr2=0x7f9a9c11bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:53.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9a9b7fe640 1 --2- 172.21.15.37:0/1067067325 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a9c09e600 0x7f9a9c11bde0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9a900097c0 tx=0x7f9a900190c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:53.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9a98ff9640 1 -- 172.21.15.37:0/1067067325 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a900b9020 con 0x7f9a9c09e600 2024-04-26T00:39:53.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9a98ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:53.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9a98ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:39:53.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9a98ff9640 10 monclient: dump: 2024-04-26T00:39:53.128 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:53.128 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:53.128 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:53.128 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:53.128 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:53.128 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:53.128 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:53.128 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:53.128 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:53.128 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:53.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9a98ff9640 10 monclient: _finish_auth 0 2024-04-26T00:39:53.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9a98ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:39:53.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9a98ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:53.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9a98ff9640 1 -- 172.21.15.37:0/1067067325 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a900b5410 con 0x7f9a9c09e600 2024-04-26T00:39:53.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9a98ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:53.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9a98ff9640 1 -- 172.21.15.37:0/1067067325 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a900b0040 con 0x7f9a9c09e600 2024-04-26T00:39:53.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9a98ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:53.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9a98ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:53.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9a98ff9640 10 monclient: dump: 2024-04-26T00:39:53.129 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:53.129 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:53.129 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:53.129 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:53.129 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:53.129 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:53.129 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:53.129 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:53.129 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:53.129 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:53.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9aa288f640 5 monclient: authenticate success, global_id 204500 2024-04-26T00:39:53.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9aa288f640 10 monclient: _renew_subs 2024-04-26T00:39:53.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9aa288f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:53.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9aa288f640 1 -- 172.21.15.37:0/1067067325 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a9c11e220 con 0x7f9a9c09e600 2024-04-26T00:39:53.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9a99ffb640 4 set_mon_vals no callback set 2024-04-26T00:39:53.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9aa288f640 10 monclient: _renew_subs 2024-04-26T00:39:53.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9aa288f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:53.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9aa288f640 1 -- 172.21.15.37:0/1067067325 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a9c09c970 con 0x7f9a9c09e600 2024-04-26T00:39:53.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9aa288f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:53.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9aa288f640 1 --2- 172.21.15.37:0/1067067325 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a9c11cf90 0x7f9a9c133c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:53.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9aa288f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:53.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9aa288f640 1 -- 172.21.15.37:0/1067067325 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9a9c0a0ab0 con 0x7f9a9c11cf90 2024-04-26T00:39:53.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9a98ff9640 1 -- 172.21.15.37:0/1067067325 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9a900bc020 con 0x7f9a9c09e600 2024-04-26T00:39:53.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9a9bfff640 1 --2- 172.21.15.37:0/1067067325 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a9c11cf90 0x7f9a9c133c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:53.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9a9bfff640 10 monclient: get_auth_request con 0x7f9a9c11cf90 auth_method 0 2024-04-26T00:39:53.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9a9bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:53.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9a9bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:53.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9a9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:53.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9a9bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:53.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9a9bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:53.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9a9bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:53.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9a98ff9640 1 --2- 172.21.15.37:0/1067067325 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9a88035ed0 0x7f9a88038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:53.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9a9bfff640 10 monclient(hunting): handle_auth_done global_id 204503 payload 824 2024-04-26T00:39:53.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9a9bfff640 1 -- 172.21.15.37:0/1067067325 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9a9c04f510 con 0x7f9a9c11cf90 2024-04-26T00:39:53.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9a98ff9640 1 -- 172.21.15.37:0/1067067325 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(789..789 src has 705..789) ==== 3931+0+0 (secure 0 0 0) 0x7f9a900e27f0 con 0x7f9a9c09e600 2024-04-26T00:39:53.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9aa0e2c640 1 --2- 172.21.15.37:0/1067067325 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9a88035ed0 0x7f9a88038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:53.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.118+0000 7f9aa0e2c640 10 monclient: get_auth_request con 0x7f9a88035ed0 auth_method 0 2024-04-26T00:39:53.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.122+0000 7f9a9bfff640 1 -- 172.21.15.37:0/1067067325 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a9c11cf90 msgr2=0x7f9a9c133c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:53.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.122+0000 7f9a9bfff640 1 --2- 172.21.15.37:0/1067067325 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a9c11cf90 0x7f9a9c133c30 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c007980 tx=0x7f9a8c00f5a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:53.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.122+0000 7f9a98ff9640 1 -- 172.21.15.37:0/1067067325 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c017030 con 0x7f9a9c11cf90 2024-04-26T00:39:53.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.122+0000 7f9a98ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:53.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.122+0000 7f9a98ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:53.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.122+0000 7f9a98ff9640 10 monclient: dump: 2024-04-26T00:39:53.134 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:53.134 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:53.134 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:53.134 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:53.134 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:53.134 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:53.134 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:53.134 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:53.134 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:53.134 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:53.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.122+0000 7f9a98ff9640 1 -- 172.21.15.37:0/1067067325 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9a8c006020 con 0x7f9a9c11cf90 2024-04-26T00:39:53.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.122+0000 7f9a98ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:53.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.122+0000 7f9a98ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:39:53.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.122+0000 7f9a98ff9640 1 -- 172.21.15.37:0/1067067325 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a9c11cf90 msgr2=0x7f9a9c133c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:53.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.122+0000 7f9a98ff9640 1 --2- 172.21.15.37:0/1067067325 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a9c11cf90 0x7f9a9c133c30 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c007980 tx=0x7f9a8c00f5a0 comp rx=0 tx=0).stop 2024-04-26T00:39:53.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.122+0000 7f9a98ff9640 1 -- 172.21.15.37:0/1067067325 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a9c11cf90 msgr2=0x7f9a9c133c30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:39:53.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.122+0000 7f9a98ff9640 1 --2- 172.21.15.37:0/1067067325 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a9c11cf90 0x7f9a9c133c30 secure :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c007980 tx=0x7f9a8c00f5a0 comp rx=0 tx=0).stop 2024-04-26T00:39:53.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.122+0000 7f9aa0e2c640 1 --2- 172.21.15.37:0/1067067325 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9a88035ed0 0x7f9a88038390 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c071290 tx=0x7f9a94008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:53.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.130+0000 7f9aa288f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:39:53.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.130+0000 7f9aa288f640 1 --2- 172.21.15.37:0/1067067325 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a9c02c000 0x7f9a9c02c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:53.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.130+0000 7f9aa288f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:53.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.130+0000 7f9aa288f640 1 -- 172.21.15.37:0/1067067325 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9a9c1396a0 con 0x7f9a9c02c000 2024-04-26T00:39:53.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.130+0000 7f9a9bfff640 1 --2- 172.21.15.37:0/1067067325 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a9c02c000 0x7f9a9c02c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:53.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.130+0000 7f9a9bfff640 10 monclient: get_auth_request con 0x7f9a9c02c000 auth_method 0 2024-04-26T00:39:53.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.130+0000 7f9a9bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:53.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.130+0000 7f9a9bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:53.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.130+0000 7f9a9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:53.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.130+0000 7f9a9bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:53.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.130+0000 7f9a9bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:53.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.130+0000 7f9a9bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:53.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.130+0000 7f9a9bfff640 10 monclient(hunting): handle_auth_done global_id 204506 payload 824 2024-04-26T00:39:53.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.130+0000 7f9a9bfff640 1 -- 172.21.15.37:0/1067067325 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9a9c070b70 con 0x7f9a9c02c000 2024-04-26T00:39:53.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.130+0000 7f9a9bfff640 1 -- 172.21.15.37:0/1067067325 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a9c02c000 msgr2=0x7f9a9c02c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:53.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.130+0000 7f9a9bfff640 1 --2- 172.21.15.37:0/1067067325 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a9c02c000 0x7f9a9c02c410 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c017000 tx=0x7f9a8c004cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:53.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.130+0000 7f9a98ff9640 1 -- 172.21.15.37:0/1067067325 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c01d240 con 0x7f9a9c02c000 2024-04-26T00:39:53.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.130+0000 7f9a98ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:53.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.130+0000 7f9a98ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:53.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.130+0000 7f9a98ff9640 10 monclient: dump: 2024-04-26T00:39:53.139 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:53.140 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:53.140 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:53.140 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:53.141 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:53.141 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:53.141 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:53.141 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:53.141 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:53.141 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:53.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.130+0000 7f9a98ff9640 1 -- 172.21.15.37:0/1067067325 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7f9a8c01d240 con 0x7f9a9c02c000 2024-04-26T00:39:53.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.130+0000 7f9a98ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:39:53.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.130+0000 7f9a98ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:39:53.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.130+0000 7f9a98ff9640 1 -- 172.21.15.37:0/1067067325 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a9c02c000 msgr2=0x7f9a9c02c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:53.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.130+0000 7f9a98ff9640 1 --2- 172.21.15.37:0/1067067325 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a9c02c000 0x7f9a9c02c410 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c017000 tx=0x7f9a8c004cd0 comp rx=0 tx=0).stop 2024-04-26T00:39:53.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.130+0000 7f9a98ff9640 1 -- 172.21.15.37:0/1067067325 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a9c02c000 msgr2=0x7f9a9c02c410 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:39:53.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.130+0000 7f9a98ff9640 1 --2- 172.21.15.37:0/1067067325 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a9c02c000 0x7f9a9c02c410 secure :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c017000 tx=0x7f9a8c004cd0 comp rx=0 tx=0).stop 2024-04-26T00:39:53.142 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:39:53.142 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:39:53.142 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:39:53.142 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:39:53.142 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 148, 2024-04-26T00:39:53.143 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 198737, 2024-04-26T00:39:53.143 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:39:53.143 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:39:53.143 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:39:53.143 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:39:53.143 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:53.143 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 4, 2024-04-26T00:39:53.143 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:39:53.143 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:39:53.143 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:39:53.143 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:53.143 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:53.143 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:53.143 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:53.144 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:53.144 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:53.144 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:53.144 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:53.144 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:53.144 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:53.144 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:53.144 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:53.144 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:39:53.144 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:39:53.144 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:53.144 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:53.144 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:53.145 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:53.145 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:53.145 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:53.145 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:53.145 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:53.145 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:53.145 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:53.145 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:53.145 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:53.145 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:53.145 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:39:53.145 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:39:53.145 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:39:53.145 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:39:53.145 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:39:53.146 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:39:53.146 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:39:53.146 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:39:53.146 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:39:53.146 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:39:53.146 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:39:53.146 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:39:53.146 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:39:53.146 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:39:53.146 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:39:53.146 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:39:53.146 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:39:53.146 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:53.146 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:53.147 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:53.147 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:53.147 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:53.147 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:53.147 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:53.147 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:53.147 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:53.147 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:53.147 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:53.147 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:53.147 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:39:53.147 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:53.147 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:39:53.147 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:53.148 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:39:53.148 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:39:53.148 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:53.148 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:53.148 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:53.148 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:53.148 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:39:53.148 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:53.148 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:53.148 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:53.148 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:53.148 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:39:53.148 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:53.148 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:53.149 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:53.149 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:53.149 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:39:53.149 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:39:53.149 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:53.149 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:53.149 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:53.149 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:53.149 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:53.149 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:39:53.149 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:39:53.149 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:53.149 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:53.149 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:53.149 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:53.150 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:39:53.150 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:53.150 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:53.150 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:53.150 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:53.150 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:39:53.150 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:53.150 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:53.150 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:53.150 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:53.150 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:39:53.150 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:39:53.150 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:53.150 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:53.151 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:53.151 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:53.151 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:53.151 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:39:53.151 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:39:53.151 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:53.151 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:53.151 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:53.151 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:53.151 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:39:53.151 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:53.151 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:53.151 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:53.151 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:53.152 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:39:53.152 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:53.152 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:53.152 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:53.152 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:53.152 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:39:53.152 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:39:53.152 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:53.152 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:53.152 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:53.152 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:53.152 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:53.152 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:53.152 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:39:53.152 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:39:53.153 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:53.153 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:53.153 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:53.153 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:39:53.153 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:53.153 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:53.153 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:39:53.153 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:53.153 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:53.153 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:53.153 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:39:53.153 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:53.153 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:53.153 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:39:53.154 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:53.154 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:53.154 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:53.154 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:39:53.154 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:53.154 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:53.154 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:53.154 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:39:53.154 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:39:53.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.130+0000 7f9aa288f640 1 -- 172.21.15.37:0/1067067325 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9a88035ed0 msgr2=0x7f9a88038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:53.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.130+0000 7f9aa288f640 1 --2- 172.21.15.37:0/1067067325 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9a88035ed0 0x7f9a88038390 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c071290 tx=0x7f9a94008040 comp rx=0 tx=0).stop 2024-04-26T00:39:53.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.130+0000 7f9aa288f640 10 monclient: shutdown 2024-04-26T00:39:53.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.130+0000 7f9aa288f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:53.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.130+0000 7f9aa288f640 1 -- 172.21.15.37:0/1067067325 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a9c09e600 msgr2=0x7f9a9c11bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:53.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.130+0000 7f9aa288f640 1 --2- 172.21.15.37:0/1067067325 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a9c09e600 0x7f9a9c11bde0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9a900097c0 tx=0x7f9a900190c0 comp rx=0 tx=0).stop 2024-04-26T00:39:53.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.130+0000 7f9aa0e2c640 1 -- 172.21.15.37:0/1067067325 reap_dead start 2024-04-26T00:39:53.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.130+0000 7f9aa0e2c640 1 -- 172.21.15.37:0/1067067325 reap_dead start 2024-04-26T00:39:53.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.134+0000 7f9aa288f640 1 -- 172.21.15.37:0/1067067325 shutdown_connections 2024-04-26T00:39:53.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.134+0000 7f9aa288f640 1 -- 172.21.15.37:0/1067067325 >> 172.21.15.37:0/1067067325 conn(0x7f9a9c0a1d50 msgr2=0x7f9a9c0a3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:53.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.134+0000 7f9aa288f640 1 -- 172.21.15.37:0/1067067325 shutdown_connections 2024-04-26T00:39:53.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.134+0000 7f9aa288f640 1 -- 172.21.15.37:0/1067067325 wait complete. 2024-04-26T00:39:53.157 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:39:53.157 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:39:53.157 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:39:53.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78c0aaa640 10 monclient: get_monmap_and_config 2024-04-26T00:39:53.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78c0aaa640 10 monclient: build_initial_monmap 2024-04-26T00:39:53.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78c0aaa640 10 monclient: monmap: 2024-04-26T00:39:53.278 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:53.278 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:53.278 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:39:53.280217+0000 2024-04-26T00:39:53.278 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:39:53.280217+0000 2024-04-26T00:39:53.278 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:53.279 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:53.279 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:39:53.279 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:39:53.279 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:39:53.279 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:39:53.279 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:39:53.279 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:53.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78c0aaa640 1 Processor -- start 2024-04-26T00:39:53.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78c0aaa640 1 -- start start 2024-04-26T00:39:53.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78c0aaa640 10 monclient: init 2024-04-26T00:39:53.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78c0aaa640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:53.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78c0aaa640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-04-26T00:39:53.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78c0aaa640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78bc0a3d90 0x7f78bc09bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:53.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78c0aaa640 10 monclient(hunting): picked mon.noname-b con 0x7f78bc0a3d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:53.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78c0aaa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f78bc0a4170 addr v1:172.21.15.45:6789/0 2024-04-26T00:39:53.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78c0aaa640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f78bc13c3e0 0x7f78bc13e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:53.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78c0aaa640 10 monclient(hunting): picked mon.noname-c con 0x7f78bc13c3e0 addr v2:172.21.15.45:3300/0 2024-04-26T00:39:53.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78c0aaa640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78bc0a5150 con 0x7f78bc0a4170 2024-04-26T00:39:53.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78c0aaa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:53.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78c0aaa640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f78bc09b280 con 0x7f78bc13c3e0 2024-04-26T00:39:53.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78c0aaa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:53.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78c0aaa640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f78bc09b560 con 0x7f78bc0a3d90 2024-04-26T00:39:53.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78c0aaa640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:53.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78c0aaa640 10 monclient(hunting): authenticate will time out at 2174.585990s 2024-04-26T00:39:53.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78ba59c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78bc0a3d90 0x7f78bc09bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:53.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78ba59c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78bc0a3d90 0x7f78bc09bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:37988/0 (socket says 172.21.15.37:37988) 2024-04-26T00:39:53.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78ba59c640 1 -- 172.21.15.37:0/4007050788 learned_addr learned my addr 172.21.15.37:0/4007050788 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:53.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78ba59c640 10 monclient(hunting): get_auth_request con 0x7f78bc0a3d90 auth_method 0 2024-04-26T00:39:53.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78ba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:53.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78ba59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:53.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78ba59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:53.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78ba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:53.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78ba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:53.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78ba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:53.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78bad9d640 1 --2- 172.21.15.37:0/4007050788 >> v2:172.21.15.45:3300/0 conn(0x7f78bc13c3e0 0x7f78bc13e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:53.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78bad9d640 10 monclient(hunting): get_auth_request con 0x7f78bc13c3e0 auth_method 0 2024-04-26T00:39:53.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78bad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:53.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78bad9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:53.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78bad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:53.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78ba59c640 10 monclient(hunting): handle_auth_done global_id 204509 payload 274 2024-04-26T00:39:53.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78ba59c640 1 -- 172.21.15.37:0/4007050788 >> v2:172.21.15.45:3300/0 conn(0x7f78bc13c3e0 msgr2=0x7f78bc13e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:53.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78ba59c640 1 --2- 172.21.15.37:0/4007050788 >> v2:172.21.15.45:3300/0 conn(0x7f78bc13c3e0 0x7f78bc13e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:53.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78ba59c640 1 -- 172.21.15.37:0/4007050788 >> v1:172.21.15.45:6789/0 conn(0x7f78bc0a4170 legacy=0x7f78bc0a45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:53.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78ba59c640 10 monclient: _finish_hunting 0 2024-04-26T00:39:53.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78ba59c640 1 monclient: found mon.noname-b 2024-04-26T00:39:53.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78ba59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:53.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78ba59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:39:53.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.278+0000 7f78ba59c640 1 -- 172.21.15.37:0/4007050788 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78bc06dc60 con 0x7f78bc0a3d90 2024-04-26T00:39:53.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.282+0000 7f78ba59c640 1 -- 172.21.15.37:0/4007050788 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78bc0a3d90 msgr2=0x7f78bc09bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:53.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.282+0000 7f78ba59c640 1 --2- 172.21.15.37:0/4007050788 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78bc0a3d90 0x7f78bc09bf40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f78ac009870 tx=0x7f78ac01c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=faadab1dd4c7a3f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:53.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.282+0000 7f78b959a640 1 -- 172.21.15.37:0/4007050788 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78ac0b0070 con 0x7f78bc0a3d90 2024-04-26T00:39:53.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.282+0000 7f78b959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:53.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.282+0000 7f78b959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:39:53.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.282+0000 7f78b959a640 10 monclient: dump: 2024-04-26T00:39:53.295 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:53.295 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:53.295 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:53.295 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:53.295 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:53.295 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:53.295 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:53.295 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:53.295 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:53.295 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:53.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.282+0000 7f78b959a640 10 monclient: _finish_auth 0 2024-04-26T00:39:53.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.282+0000 7f78b959a640 10 monclient: _check_auth_tickets 2024-04-26T00:39:53.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.282+0000 7f78b959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:53.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.282+0000 7f78c0aaa640 5 monclient: authenticate success, global_id 204509 2024-04-26T00:39:53.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.282+0000 7f78b959a640 1 -- 172.21.15.37:0/4007050788 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78ac0ab600 con 0x7f78bc0a3d90 2024-04-26T00:39:53.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.282+0000 7f78c0aaa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:39:53.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.282+0000 7f78b959a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:53.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.282+0000 7f78c0aaa640 10 monclient: get_monmap_and_config success 2024-04-26T00:39:53.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.282+0000 7f78c0aaa640 4 set_mon_vals no callback set 2024-04-26T00:39:53.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.282+0000 7f78b959a640 1 -- 172.21.15.37:0/4007050788 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78ac0ab9d0 con 0x7f78bc0a3d90 2024-04-26T00:39:53.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.282+0000 7f78b959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:53.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.282+0000 7f78b959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:53.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.282+0000 7f78b959a640 10 monclient: dump: 2024-04-26T00:39:53.297 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:53.297 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:53.297 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:53.297 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:53.297 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:53.297 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:53.297 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:53.297 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:53.297 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:53.298 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:53.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.282+0000 7f78c0aaa640 10 monclient: shutdown 2024-04-26T00:39:53.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.282+0000 7f78c0aaa640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:53.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.282+0000 7f78c0aaa640 1 -- 172.21.15.37:0/4007050788 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78bc0a3d90 msgr2=0x7f78bc09bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:53.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.282+0000 7f78c0aaa640 1 --2- 172.21.15.37:0/4007050788 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78bc0a3d90 0x7f78bc09bf40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f78ac009870 tx=0x7f78ac01c3e0 comp rx=0 tx=0).stop 2024-04-26T00:39:53.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78c0aaa640 1 -- 172.21.15.37:0/4007050788 shutdown_connections 2024-04-26T00:39:53.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78c0aaa640 1 --2- 172.21.15.37:0/4007050788 >> v2:172.21.15.45:3300/0 conn(0x7f78bc13c3e0 0x7f78bc13e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:53.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78c0aaa640 1 --2- 172.21.15.37:0/4007050788 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78bc0a3d90 0x7f78bc09bf40 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:53.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78c0aaa640 1 -- 172.21.15.37:0/4007050788 >> 172.21.15.37:0/4007050788 conn(0x7f78bc0aa0a0 msgr2=0x7f78bc0a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:53.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78c0aaa640 1 -- 172.21.15.37:0/4007050788 shutdown_connections 2024-04-26T00:39:53.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78c0aaa640 1 -- 172.21.15.37:0/4007050788 wait complete. 2024-04-26T00:39:53.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78c0aaa640 10 monclient: build_initial_monmap 2024-04-26T00:39:53.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78c0aaa640 10 monclient: monmap: 2024-04-26T00:39:53.298 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:53.299 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:39:53.299 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:39:53.299 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:39:53.299 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:53.299 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:39:53.299 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:39:53.299 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:39:53.299 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:39:53.299 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:53.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78c0aaa640 1 Processor -- start 2024-04-26T00:39:53.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78c0aaa640 1 -- start start 2024-04-26T00:39:53.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78c0aaa640 10 monclient: init 2024-04-26T00:39:53.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78c0aaa640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:53.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78c0aaa640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:39:53.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78c0aaa640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78bc0a3d90 0x7f78bc120430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:53.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78c0aaa640 10 monclient(hunting): picked mon.noname-c con 0x7f78bc0a3d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:53.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78c0aaa640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78bc0a4170 0x7f78bc13a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:53.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78c0aaa640 10 monclient(hunting): picked mon.noname-b con 0x7f78bc0a4170 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:39:53.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78c0aaa640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78bc13c3e0 0x7f78bc137010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:53.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78c0aaa640 10 monclient(hunting): picked mon.noname-a con 0x7f78bc13c3e0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:39:53.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78c0aaa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:53.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78c0aaa640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f78bc141740 con 0x7f78bc13c3e0 2024-04-26T00:39:53.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78c0aaa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:53.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78c0aaa640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f78bc09c690 con 0x7f78bc0a4170 2024-04-26T00:39:53.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78c0aaa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:53.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78c0aaa640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f78bc1418c0 con 0x7f78bc0a3d90 2024-04-26T00:39:53.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78c0aaa640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:53.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78c0aaa640 10 monclient(hunting): authenticate will time out at 2174.593634s 2024-04-26T00:39:53.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78ba59c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78bc0a3d90 0x7f78bc120430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:53.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78ba59c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78bc0a3d90 0x7f78bc120430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:37998/0 (socket says 172.21.15.37:37998) 2024-04-26T00:39:53.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78bad9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78bc13c3e0 0x7f78bc137010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:53.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78ba59c640 1 -- 172.21.15.37:0/873903960 learned_addr learned my addr 172.21.15.37:0/873903960 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:53.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78ba59c640 10 monclient(hunting): get_auth_request con 0x7f78bc0a3d90 auth_method 0 2024-04-26T00:39:53.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78ba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:53.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78ba59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:53.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78ba59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:53.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78bad9d640 10 monclient(hunting): get_auth_request con 0x7f78bc13c3e0 auth_method 0 2024-04-26T00:39:53.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78bad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:53.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78bad9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:53.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78bad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:53.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78ba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:53.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78ba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:53.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78ba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:53.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78bad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:53.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78bad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:53.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78bad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:53.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78b9d9b640 1 --2- 172.21.15.37:0/873903960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78bc0a4170 0x7f78bc13a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:53.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78b9d9b640 10 monclient(hunting): get_auth_request con 0x7f78bc0a4170 auth_method 0 2024-04-26T00:39:53.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78b9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:53.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78b9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:53.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78b9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:53.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78bad9d640 10 monclient(hunting): handle_auth_done global_id 244107 payload 824 2024-04-26T00:39:53.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78bad9d640 1 -- 172.21.15.37:0/873903960 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78bc0a3d90 msgr2=0x7f78bc120430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:53.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78bad9d640 1 --2- 172.21.15.37:0/873903960 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78bc0a3d90 0x7f78bc120430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:53.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78bad9d640 1 -- 172.21.15.37:0/873903960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78bc0a4170 msgr2=0x7f78bc13a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:53.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78bad9d640 1 --2- 172.21.15.37:0/873903960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78bc0a4170 0x7f78bc13a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:53.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78bad9d640 10 monclient: _finish_hunting 0 2024-04-26T00:39:53.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78bad9d640 1 monclient: found mon.noname-a 2024-04-26T00:39:53.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78bad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:53.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78bad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:39:53.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78bad9d640 1 -- 172.21.15.37:0/873903960 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78bc13cd10 con 0x7f78bc13c3e0 2024-04-26T00:39:53.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.286+0000 7f78ba59c640 1 --2- 172.21.15.37:0/873903960 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78bc0a3d90 0x7f78bc120430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:39:53.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78bad9d640 1 -- 172.21.15.37:0/873903960 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78bc13c3e0 msgr2=0x7f78bc137010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:53.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78bad9d640 1 --2- 172.21.15.37:0/873903960 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78bc13c3e0 0x7f78bc137010 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f78b4007d90 tx=0x7f78b400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:53.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78aaffd640 1 -- 172.21.15.37:0/873903960 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b40039a0 con 0x7f78bc13c3e0 2024-04-26T00:39:53.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:53.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78aaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:39:53.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78aaffd640 10 monclient: dump: 2024-04-26T00:39:53.304 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:53.304 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:53.304 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:53.304 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:53.304 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:53.304 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:53.305 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:53.305 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:53.305 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:53.305 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:53.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78aaffd640 10 monclient: _finish_auth 0 2024-04-26T00:39:53.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78aaffd640 10 monclient: _check_auth_tickets 2024-04-26T00:39:53.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:53.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78aaffd640 1 -- 172.21.15.37:0/873903960 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78b4003b60 con 0x7f78bc13c3e0 2024-04-26T00:39:53.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78aaffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:53.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78aaffd640 1 -- 172.21.15.37:0/873903960 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b4004020 con 0x7f78bc13c3e0 2024-04-26T00:39:53.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:53.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78aaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:39:53.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78aaffd640 10 monclient: dump: 2024-04-26T00:39:53.305 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:53.305 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:53.306 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:53.306 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:53.306 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:53.306 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:53.306 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:53.306 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:53.306 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:53.306 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:53.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78c0aaa640 5 monclient: authenticate success, global_id 244107 2024-04-26T00:39:53.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78b8d99640 4 set_mon_vals no callback set 2024-04-26T00:39:53.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78c0aaa640 10 monclient: _renew_subs 2024-04-26T00:39:53.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78c0aaa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:39:53.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78c0aaa640 1 -- 172.21.15.37:0/873903960 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78bc09ada0 con 0x7f78bc13c3e0 2024-04-26T00:39:53.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78c0aaa640 10 monclient: _renew_subs 2024-04-26T00:39:53.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78c0aaa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:39:53.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78c0aaa640 1 -- 172.21.15.37:0/873903960 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78bc11ff90 con 0x7f78bc13c3e0 2024-04-26T00:39:53.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78c0aaa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:53.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78c0aaa640 1 --2- 172.21.15.37:0/873903960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78bc137550 0x7f78bc1cf580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:53.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78c0aaa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:53.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78c0aaa640 1 -- 172.21.15.37:0/873903960 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f78bc13b700 con 0x7f78bc137550 2024-04-26T00:39:53.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78aaffd640 1 -- 172.21.15.37:0/873903960 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f78b401b020 con 0x7f78bc13c3e0 2024-04-26T00:39:53.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78aaffd640 1 --2- 172.21.15.37:0/873903960 >> v2:172.21.15.45:6812/2379918623 conn(0x7f789c035ed0 0x7f789c038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:53.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78aaffd640 1 -- 172.21.15.37:0/873903960 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(789..789 src has 705..789) ==== 3931+0+0 (secure 0 0 0) 0x7f78b4048c50 con 0x7f78bc13c3e0 2024-04-26T00:39:53.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78ba59c640 1 --2- 172.21.15.37:0/873903960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78bc137550 0x7f78bc1cf580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:53.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78ba59c640 10 monclient: get_auth_request con 0x7f78bc137550 auth_method 0 2024-04-26T00:39:53.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78ba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:53.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78ba59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:53.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78ba59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:53.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78b9d9b640 1 --2- 172.21.15.37:0/873903960 >> v2:172.21.15.45:6812/2379918623 conn(0x7f789c035ed0 0x7f789c038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:53.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78ba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:53.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78ba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:53.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78ba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:53.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78b9d9b640 10 monclient: get_auth_request con 0x7f789c035ed0 auth_method 0 2024-04-26T00:39:53.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78ba59c640 10 monclient(hunting): handle_auth_done global_id 224275 payload 824 2024-04-26T00:39:53.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.290+0000 7f78ba59c640 1 -- 172.21.15.37:0/873903960 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f78bc04f510 con 0x7f78bc137550 2024-04-26T00:39:53.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.294+0000 7f78ba59c640 1 -- 172.21.15.37:0/873903960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78bc137550 msgr2=0x7f78bc1cf580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:53.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.294+0000 7f78ba59c640 1 --2- 172.21.15.37:0/873903960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78bc137550 0x7f78bc1cf580 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f78ac01cc80 tx=0x7f78ac019180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:53.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.294+0000 7f78b9d9b640 1 --2- 172.21.15.37:0/873903960 >> v2:172.21.15.45:6812/2379918623 conn(0x7f789c035ed0 0x7f789c038390 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f78bc137db0 tx=0x7f78b0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:53.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.294+0000 7f78aaffd640 1 -- 172.21.15.37:0/873903960 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78ac0ba070 con 0x7f78bc137550 2024-04-26T00:39:53.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.294+0000 7f78aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:53.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.294+0000 7f78aaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:39:53.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.294+0000 7f78aaffd640 10 monclient: dump: 2024-04-26T00:39:53.311 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:53.311 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:53.312 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:53.312 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:53.312 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:53.312 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:53.312 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:53.312 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:53.312 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:53.312 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:53.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.298+0000 7f78aaffd640 1 -- 172.21.15.37:0/873903960 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f78ac0b1040 con 0x7f78bc137550 2024-04-26T00:39:53.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.298+0000 7f78aaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:53.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.298+0000 7f78aaffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:39:53.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.298+0000 7f78aaffd640 1 -- 172.21.15.37:0/873903960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78bc137550 msgr2=0x7f78bc1cf580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:53.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.298+0000 7f78aaffd640 1 --2- 172.21.15.37:0/873903960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78bc137550 0x7f78bc1cf580 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f78ac01cc80 tx=0x7f78ac019180 comp rx=0 tx=0).stop 2024-04-26T00:39:53.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.298+0000 7f78aaffd640 1 -- 172.21.15.37:0/873903960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78bc137550 msgr2=0x7f78bc1cf580 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:39:53.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.298+0000 7f78aaffd640 1 --2- 172.21.15.37:0/873903960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78bc137550 0x7f78bc1cf580 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f78ac01cc80 tx=0x7f78ac019180 comp rx=0 tx=0).stop 2024-04-26T00:39:53.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.302+0000 7f78c0aaa640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:39:53.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.302+0000 7f78c0aaa640 1 --2- 172.21.15.37:0/873903960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78bc02c000 0x7f78bc02c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:53.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.302+0000 7f78c0aaa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:53.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.302+0000 7f78c0aaa640 1 -- 172.21.15.37:0/873903960 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f78bc0a1ea0 con 0x7f78bc02c000 2024-04-26T00:39:53.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.302+0000 7f78ba59c640 1 --2- 172.21.15.37:0/873903960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78bc02c000 0x7f78bc02c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:53.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.302+0000 7f78ba59c640 10 monclient: get_auth_request con 0x7f78bc02c000 auth_method 0 2024-04-26T00:39:53.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.302+0000 7f78ba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:53.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.302+0000 7f78ba59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:53.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.302+0000 7f78ba59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:53.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.302+0000 7f78ba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:53.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.302+0000 7f78ba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:53.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.302+0000 7f78ba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:53.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.302+0000 7f78ba59c640 10 monclient(hunting): handle_auth_done global_id 224278 payload 824 2024-04-26T00:39:53.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.302+0000 7f78ba59c640 1 -- 172.21.15.37:0/873903960 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f78bc075dc0 con 0x7f78bc02c000 2024-04-26T00:39:53.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.306+0000 7f78ba59c640 1 -- 172.21.15.37:0/873903960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78bc02c000 msgr2=0x7f78bc02c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:53.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.306+0000 7f78ba59c640 1 --2- 172.21.15.37:0/873903960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78bc02c000 0x7f78bc02c410 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f78bc13b880 tx=0x7f78ac0a8000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:53.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.306+0000 7f78aaffd640 1 -- 172.21.15.37:0/873903960 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78ac0b0040 con 0x7f78bc02c000 2024-04-26T00:39:53.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.306+0000 7f78aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:53.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.306+0000 7f78aaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:39:53.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.306+0000 7f78aaffd640 10 monclient: dump: 2024-04-26T00:39:53.314 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:53.314 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:53.314 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:53.314 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:53.315 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:53.315 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:53.315 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:53.315 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:53.315 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:53.315 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:53.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.306+0000 7f78aaffd640 1 -- 172.21.15.37:0/873903960 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4888 (secure 0 0 0) 0x7f78ac0aa3a0 con 0x7f78bc02c000 2024-04-26T00:39:53.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.306+0000 7f78aaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:39:53.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.306+0000 7f78aaffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:39:53.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.306+0000 7f78aaffd640 1 -- 172.21.15.37:0/873903960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78bc02c000 msgr2=0x7f78bc02c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:53.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.306+0000 7f78aaffd640 1 --2- 172.21.15.37:0/873903960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78bc02c000 0x7f78bc02c410 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f78bc13b880 tx=0x7f78ac0a8000 comp rx=0 tx=0).stop 2024-04-26T00:39:53.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.306+0000 7f78aaffd640 1 -- 172.21.15.37:0/873903960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78bc02c000 msgr2=0x7f78bc02c410 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:39:53.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.306+0000 7f78aaffd640 1 --2- 172.21.15.37:0/873903960 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f78bc02c000 0x7f78bc02c410 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f78bc13b880 tx=0x7f78ac0a8000 comp rx=0 tx=0).stop 2024-04-26T00:39:53.316 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:39:53.316 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:39:53.316 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:39:53.316 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:39:53.317 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 148, 2024-04-26T00:39:53.317 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 106970, 2024-04-26T00:39:53.317 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:39:53.317 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:39:53.317 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:39:53.317 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:39:53.317 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:53.317 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 4, 2024-04-26T00:39:53.317 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:39:53.318 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:39:53.318 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:39:53.318 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:53.318 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:53.318 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:53.318 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:53.318 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:53.318 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:53.318 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:53.318 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:53.318 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:53.318 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:53.319 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:53.319 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:53.319 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:39:53.319 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:39:53.319 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:53.319 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:53.319 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:53.319 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:53.319 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:53.319 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:53.319 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:53.319 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:53.319 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:53.320 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:53.320 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:53.320 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:53.320 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:53.320 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:39:53.320 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:39:53.320 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:39:53.320 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:39:53.320 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:39:53.320 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:39:53.320 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:39:53.320 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:39:53.320 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:39:53.320 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:39:53.321 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:39:53.321 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:39:53.321 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:39:53.321 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:39:53.321 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:39:53.321 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:39:53.321 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:39:53.321 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:53.321 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:53.321 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:53.321 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:53.321 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:53.321 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:53.321 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:53.321 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:53.322 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:53.322 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:53.322 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:53.322 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:53.322 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:39:53.322 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:53.322 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:39:53.322 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:53.322 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:39:53.322 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:39:53.322 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:53.322 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:53.322 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:53.322 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:53.323 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:39:53.323 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:53.323 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:53.323 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:53.323 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:53.323 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:39:53.323 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:53.323 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:53.323 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:53.323 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:53.323 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:39:53.323 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:39:53.323 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:53.323 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:53.324 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:53.324 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:53.324 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:53.324 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:39:53.324 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:39:53.324 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:53.324 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:53.324 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:53.324 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:53.324 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:39:53.324 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:53.324 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:53.324 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:53.324 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:53.325 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:39:53.325 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:53.325 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:53.325 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:53.325 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:53.325 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:39:53.325 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:39:53.325 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:53.325 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:53.325 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:53.325 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:53.325 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:53.325 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:39:53.326 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:39:53.326 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:53.326 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:53.326 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:53.326 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:53.326 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:39:53.326 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:53.326 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:53.326 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:53.326 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:53.326 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:39:53.326 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:53.326 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:53.327 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:53.327 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:53.327 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:39:53.327 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:39:53.327 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:53.327 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:53.327 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:53.327 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:53.327 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:53.327 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:53.327 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:39:53.328 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:39:53.328 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:53.328 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:53.328 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:53.328 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:39:53.328 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:53.328 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:53.328 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:39:53.328 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:53.328 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:53.328 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:53.328 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:39:53.328 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:53.328 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:53.329 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:39:53.329 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:53.329 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:53.329 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:53.329 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:39:53.329 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:53.329 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:53.329 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:39:53.329 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:53.329 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:53.329 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:53.329 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:39:53.329 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:53.330 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:53.330 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:53.330 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:39:53.330 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:39:53.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.306+0000 7f78c0aaa640 1 -- 172.21.15.37:0/873903960 >> v2:172.21.15.45:6812/2379918623 conn(0x7f789c035ed0 msgr2=0x7f789c038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:53.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.306+0000 7f78c0aaa640 1 --2- 172.21.15.37:0/873903960 >> v2:172.21.15.45:6812/2379918623 conn(0x7f789c035ed0 0x7f789c038390 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f78bc137db0 tx=0x7f78b0009040 comp rx=0 tx=0).stop 2024-04-26T00:39:53.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.306+0000 7f78c0aaa640 10 monclient: shutdown 2024-04-26T00:39:53.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.306+0000 7f78c0aaa640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:53.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.306+0000 7f78c0aaa640 1 -- 172.21.15.37:0/873903960 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78bc13c3e0 msgr2=0x7f78bc137010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:53.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.306+0000 7f78c0aaa640 1 --2- 172.21.15.37:0/873903960 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78bc13c3e0 0x7f78bc137010 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f78b4007d90 tx=0x7f78b400e4b0 comp rx=0 tx=0).stop 2024-04-26T00:39:53.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.306+0000 7f78bad9d640 1 -- 172.21.15.37:0/873903960 reap_dead start 2024-04-26T00:39:53.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.306+0000 7f78c0aaa640 1 -- 172.21.15.37:0/873903960 shutdown_connections 2024-04-26T00:39:53.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.306+0000 7f78c0aaa640 1 --2- 172.21.15.37:0/873903960 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78bc13c3e0 0x7f78bc137010 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:53.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.306+0000 7f78c0aaa640 1 -- 172.21.15.37:0/873903960 >> 172.21.15.37:0/873903960 conn(0x7f78bc0aa0a0 msgr2=0x7f78bc09f080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:53.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.306+0000 7f78c0aaa640 1 -- 172.21.15.37:0/873903960 shutdown_connections 2024-04-26T00:39:53.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.306+0000 7f78c0aaa640 1 -- 172.21.15.37:0/873903960 wait complete. 2024-04-26T00:39:53.332 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:39:53.332 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-26T00:39:53.332 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-26T00:39:53.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.446+0000 7f9930690640 10 monclient: get_monmap_and_config 2024-04-26T00:39:53.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.446+0000 7f9930690640 10 monclient: build_initial_monmap 2024-04-26T00:39:53.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.446+0000 7f9930690640 10 monclient: monmap: 2024-04-26T00:39:53.447 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:53.447 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:53.447 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:39:53.449845+0000 2024-04-26T00:39:53.447 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:39:53.449845+0000 2024-04-26T00:39:53.447 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:53.447 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:53.447 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:39:53.447 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:39:53.447 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:39:53.447 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:39:53.448 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:39:53.448 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:53.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.446+0000 7f9930690640 1 Processor -- start 2024-04-26T00:39:53.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.446+0000 7f9930690640 1 -- start start 2024-04-26T00:39:53.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.446+0000 7f9930690640 10 monclient: init 2024-04-26T00:39:53.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.446+0000 7f9930690640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:53.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.446+0000 7f9930690640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-04-26T00:39:53.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.446+0000 7f9930690640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9928133b20 addr v1:172.21.15.37:6789/0 2024-04-26T00:39:53.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.446+0000 7f9930690640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9928133f00 addr v1:172.21.15.45:6789/0 2024-04-26T00:39:53.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.446+0000 7f9930690640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f992813a060 0x7f992813c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:53.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.446+0000 7f9930690640 10 monclient(hunting): picked mon.noname-b con 0x7f992813a060 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:53.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.446+0000 7f9930690640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f992809fe20 con 0x7f9928133b20 2024-04-26T00:39:53.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.446+0000 7f9930690640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9928072d00 con 0x7f9928133f00 2024-04-26T00:39:53.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.446+0000 7f9930690640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:53.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.446+0000 7f9930690640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f992809e620 con 0x7f992813a060 2024-04-26T00:39:53.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.446+0000 7f9930690640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:53.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.446+0000 7f9930690640 10 monclient(hunting): authenticate will time out at 2174.755982s 2024-04-26T00:39:53.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.450+0000 7f992e42c640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f9928133b20 0x7f992809e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:43274/0 (socket says 172.21.15.37:43274) 2024-04-26T00:39:53.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.450+0000 7f992e42c640 1 -- 172.21.15.37:0/2452740975 learned_addr learned my addr 172.21.15.37:0/2452740975 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:39:53.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.450+0000 7f992ec2d640 1 --2- 172.21.15.37:0/2452740975 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f992813a060 0x7f992813c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:53.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.450+0000 7f992ec2d640 10 monclient(hunting): get_auth_request con 0x7f992813a060 auth_method 0 2024-04-26T00:39:53.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.450+0000 7f992ec2d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:53.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.450+0000 7f992ec2d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:53.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.450+0000 7f992ec2d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:53.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.450+0000 7f992ec2d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:53.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.450+0000 7f992ec2d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:53.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.450+0000 7f992ec2d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:53.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.450+0000 7f992ec2d640 10 monclient(hunting): handle_auth_done global_id 204515 payload 274 2024-04-26T00:39:53.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.450+0000 7f992ec2d640 1 -- 172.21.15.37:0/2452740975 >> v1:172.21.15.45:6789/0 conn(0x7f9928133f00 legacy=0x7f9928134350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:53.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.450+0000 7f992ec2d640 1 -- 172.21.15.37:0/2452740975 >> v1:172.21.15.37:6789/0 conn(0x7f9928133b20 legacy=0x7f992809e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:53.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.450+0000 7f992ec2d640 10 monclient: _finish_hunting 0 2024-04-26T00:39:53.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.450+0000 7f992ec2d640 1 monclient: found mon.noname-b 2024-04-26T00:39:53.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.450+0000 7f992ec2d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:53.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.450+0000 7f992ec2d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:39:53.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.450+0000 7f992ec2d640 1 -- 172.21.15.37:0/2452740975 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f992809ba60 con 0x7f992813a060 2024-04-26T00:39:53.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.454+0000 7f992ec2d640 1 -- 172.21.15.37:0/2452740975 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f992813a060 msgr2=0x7f992813c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:53.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.454+0000 7f992ec2d640 1 --2- 172.21.15.37:0/2452740975 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f992813a060 0x7f992813c450 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f992809bc30 tx=0x7f992001c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6b2a318191b251e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:53.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.454+0000 7f992d42a640 1 -- 172.21.15.37:0/2452740975 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99200a67c0 con 0x7f992813a060 2024-04-26T00:39:53.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.454+0000 7f992d42a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:53.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.454+0000 7f992d42a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:39:53.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.454+0000 7f992d42a640 10 monclient: dump: 2024-04-26T00:39:53.456 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:53.456 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:53.456 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:53.456 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:53.456 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:53.456 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:53.456 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:53.456 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:53.456 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:53.456 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:53.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.454+0000 7f992d42a640 10 monclient: _finish_auth 0 2024-04-26T00:39:53.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.454+0000 7f992d42a640 10 monclient: _check_auth_tickets 2024-04-26T00:39:53.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.454+0000 7f992d42a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:53.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.454+0000 7f992d42a640 1 -- 172.21.15.37:0/2452740975 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99200a6960 con 0x7f992813a060 2024-04-26T00:39:53.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.454+0000 7f9930690640 5 monclient: authenticate success, global_id 204515 2024-04-26T00:39:53.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.454+0000 7f9930690640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:39:53.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.454+0000 7f992d42a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:53.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.454+0000 7f992d42a640 1 -- 172.21.15.37:0/2452740975 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99200a6d30 con 0x7f992813a060 2024-04-26T00:39:53.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.454+0000 7f9930690640 10 monclient: get_monmap_and_config success 2024-04-26T00:39:53.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.454+0000 7f9930690640 4 set_mon_vals no callback set 2024-04-26T00:39:53.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.454+0000 7f992d42a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:53.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.454+0000 7f992d42a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:53.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.454+0000 7f992d42a640 10 monclient: dump: 2024-04-26T00:39:53.457 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:53.457 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:53.457 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:53.457 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:53.458 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:53.458 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:53.458 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:53.458 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:53.458 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:53.458 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:53.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.454+0000 7f9930690640 10 monclient: shutdown 2024-04-26T00:39:53.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.454+0000 7f9930690640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:53.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.454+0000 7f9930690640 1 -- 172.21.15.37:0/2452740975 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f992813a060 msgr2=0x7f992813c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:53.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.454+0000 7f9930690640 1 --2- 172.21.15.37:0/2452740975 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f992813a060 0x7f992813c450 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f992809bc30 tx=0x7f992001c650 comp rx=0 tx=0).stop 2024-04-26T00:39:53.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.454+0000 7f9930690640 1 -- 172.21.15.37:0/2452740975 shutdown_connections 2024-04-26T00:39:53.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.454+0000 7f9930690640 1 --2- 172.21.15.37:0/2452740975 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f992813a060 0x7f992813c450 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:53.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.454+0000 7f9930690640 1 -- 172.21.15.37:0/2452740975 >> 172.21.15.37:0/2452740975 conn(0x7f99280a1ad0 msgr2=0x7f99280a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:53.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.454+0000 7f9930690640 1 -- 172.21.15.37:0/2452740975 shutdown_connections 2024-04-26T00:39:53.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.454+0000 7f9930690640 1 -- 172.21.15.37:0/2452740975 wait complete. 2024-04-26T00:39:53.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.454+0000 7f9930690640 10 monclient: build_initial_monmap 2024-04-26T00:39:53.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.454+0000 7f9930690640 10 monclient: monmap: 2024-04-26T00:39:53.463 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:53.463 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:39:53.463 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:39:53.463 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:39:53.463 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:53.463 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:39:53.463 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:39:53.463 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:39:53.463 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:39:53.463 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:53.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.454+0000 7f9930690640 1 Processor -- start 2024-04-26T00:39:53.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.454+0000 7f9930690640 1 -- start start 2024-04-26T00:39:53.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.454+0000 7f9930690640 10 monclient: init 2024-04-26T00:39:53.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f9930690640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:53.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f9930690640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:39:53.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f9930690640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9928133b20 0x7f9928138120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:53.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f9930690640 10 monclient(hunting): picked mon.noname-c con 0x7f9928133b20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:53.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f9930690640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9928133f00 0x7f9928138660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:53.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f9930690640 10 monclient(hunting): picked mon.noname-a con 0x7f9928133f00 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:39:53.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f9930690640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f992813a060 0x7f9928138ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:53.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f9930690640 10 monclient(hunting): picked mon.noname-b con 0x7f992813a060 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:39:53.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f9930690640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:53.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f9930690640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f99280a05f0 con 0x7f9928133f00 2024-04-26T00:39:53.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f9930690640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:53.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f9930690640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f99280a0470 con 0x7f992813a060 2024-04-26T00:39:53.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f9930690640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:53.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f9930690640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f99280a0770 con 0x7f9928133b20 2024-04-26T00:39:53.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f9930690640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:53.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f9930690640 10 monclient(hunting): authenticate will time out at 2174.764465s 2024-04-26T00:39:53.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f992e42c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9928133b20 0x7f9928138120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:53.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f992dc2b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9928133f00 0x7f9928138660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:53.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f992e42c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9928133b20 0x7f9928138120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:38018/0 (socket says 172.21.15.37:38018) 2024-04-26T00:39:53.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f992e42c640 1 -- 172.21.15.37:0/2501011631 learned_addr learned my addr 172.21.15.37:0/2501011631 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:53.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f992e42c640 10 monclient(hunting): get_auth_request con 0x7f9928133b20 auth_method 0 2024-04-26T00:39:53.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f992e42c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:53.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f992e42c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:53.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f992e42c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:53.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f992dc2b640 10 monclient(hunting): get_auth_request con 0x7f9928133f00 auth_method 0 2024-04-26T00:39:53.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f992dc2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:53.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f992dc2b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:53.465 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f992dc2b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:53.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f992e42c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:53.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f992e42c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:53.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f992e42c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:53.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f992dc2b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:53.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f992dc2b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:53.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f992dc2b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:53.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f992e42c640 10 monclient(hunting): handle_auth_done global_id 204518 payload 824 2024-04-26T00:39:53.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f992ec2d640 1 --2- 172.21.15.37:0/2501011631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f992813a060 0x7f9928138ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:53.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f992e42c640 1 -- 172.21.15.37:0/2501011631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f992813a060 msgr2=0x7f9928138ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:53.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f992e42c640 1 --2- 172.21.15.37:0/2501011631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f992813a060 0x7f9928138ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:53.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f992e42c640 1 -- 172.21.15.37:0/2501011631 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9928133f00 msgr2=0x7f9928138660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:53.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f992e42c640 1 --2- 172.21.15.37:0/2501011631 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9928133f00 0x7f9928138660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:53.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f992e42c640 10 monclient: _finish_hunting 0 2024-04-26T00:39:53.466 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f992e42c640 1 monclient: found mon.noname-c 2024-04-26T00:39:53.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f992e42c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:53.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f992e42c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:39:53.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f992e42c640 1 -- 172.21.15.37:0/2501011631 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f992806f0d0 con 0x7f9928133b20 2024-04-26T00:39:53.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f992dc2b640 1 --2- 172.21.15.37:0/2501011631 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9928133f00 0x7f9928138660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:39:53.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f992e42c640 1 -- 172.21.15.37:0/2501011631 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9928133b20 msgr2=0x7f9928138120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:53.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f992e42c640 1 --2- 172.21.15.37:0/2501011631 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9928133b20 0x7f9928138120 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f991800c470 tx=0x7f991800c930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:53.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f991effd640 1 -- 172.21.15.37:0/2501011631 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9918007070 con 0x7f9928133b20 2024-04-26T00:39:53.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f991effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:53.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f991effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:39:53.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f991effd640 10 monclient: dump: 2024-04-26T00:39:53.467 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:53.467 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:53.467 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:53.468 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:53.468 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:53.468 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:53.468 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:53.468 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:53.468 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:53.468 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:53.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f991effd640 10 monclient: _finish_auth 0 2024-04-26T00:39:53.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f991effd640 10 monclient: _check_auth_tickets 2024-04-26T00:39:53.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f991effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:53.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f991effd640 1 -- 172.21.15.37:0/2501011631 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9918007210 con 0x7f9928133b20 2024-04-26T00:39:53.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f9930690640 5 monclient: authenticate success, global_id 204518 2024-04-26T00:39:53.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f9930690640 10 monclient: _renew_subs 2024-04-26T00:39:53.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f9930690640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:53.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f9930690640 1 -- 172.21.15.37:0/2501011631 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f992806e1f0 con 0x7f9928133b20 2024-04-26T00:39:53.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f991effd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:53.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f991effd640 1 -- 172.21.15.37:0/2501011631 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9918007600 con 0x7f9928133b20 2024-04-26T00:39:53.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f991effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:53.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f991effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:53.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f991effd640 10 monclient: dump: 2024-04-26T00:39:53.469 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:53.469 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:53.469 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:53.469 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:53.469 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:53.470 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:53.470 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:53.470 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:53.470 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:53.470 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:53.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f9930690640 10 monclient: _renew_subs 2024-04-26T00:39:53.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f9930690640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:53.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f991ffff640 4 set_mon_vals no callback set 2024-04-26T00:39:53.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.458+0000 7f9930690640 1 -- 172.21.15.37:0/2501011631 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f992806e900 con 0x7f9928133b20 2024-04-26T00:39:53.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.462+0000 7f991effd640 1 -- 172.21.15.37:0/2501011631 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f991801b400 con 0x7f9928133b20 2024-04-26T00:39:53.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.462+0000 7f9930690640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:53.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.462+0000 7f9930690640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:53.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.462+0000 7f9930690640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:53.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.462+0000 7f9930690640 1 -- 172.21.15.37:0/2501011631 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9928066f40 con 0x7f9928133b20 2024-04-26T00:39:53.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.462+0000 7f991effd640 1 --2- 172.21.15.37:0/2501011631 >> v2:172.21.15.45:6812/2379918623 conn(0x7f990c035e80 0x7f990c038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:53.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.462+0000 7f991effd640 1 -- 172.21.15.37:0/2501011631 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(789..789 src has 705..789) ==== 3931+0+0 (secure 0 0 0) 0x7f9918048870 con 0x7f9928133b20 2024-04-26T00:39:53.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.462+0000 7f992dc2b640 1 --2- 172.21.15.37:0/2501011631 >> v2:172.21.15.45:6812/2379918623 conn(0x7f990c035e80 0x7f990c038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:53.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.462+0000 7f992dc2b640 10 monclient: get_auth_request con 0x7f990c035e80 auth_method 0 2024-04-26T00:39:53.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.462+0000 7f992dc2b640 1 --2- 172.21.15.37:0/2501011631 >> v2:172.21.15.45:6812/2379918623 conn(0x7f990c035e80 0x7f990c038340 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f99281398f0 tx=0x7f9924007e40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:53.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.466+0000 7f991effd640 1 -- 172.21.15.37:0/2501011631 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f99180475c0 con 0x7f9928133b20 2024-04-26T00:39:53.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.466+0000 7f991effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:53.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.466+0000 7f991effd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:39:53.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.690+0000 7f9930690640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-26T00:39:53.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.690+0000 7f9930690640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-26T00:39:53.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.690+0000 7f9930690640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:53.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.690+0000 7f9930690640 1 -- 172.21.15.37:0/2501011631 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f992806b9a0 con 0x7f9928133b20 2024-04-26T00:39:53.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.690+0000 7f991effd640 1 -- 172.21.15.37:0/2501011631 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f9918012400 con 0x7f9928133b20 2024-04-26T00:39:53.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.690+0000 7f991effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-26T00:39:53.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.690+0000 7f991effd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:39:53.693 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.690+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:53.693 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.690+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=29,config=2,health=11,kv=8,logm=50} crc {auth=269856665,config=692619927,health=1140756147,kv=439360147,logm=1035661301}) 2024-04-26T00:39:53.693 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.690+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=12,kv=9,logm=49} crc {auth=3396046467,config=692619927,health=222745339,kv=1034996044,logm=2716992813}) 2024-04-26T00:39:53.693 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.690+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:53.693 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.690+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=29,config=2,health=11,kv=8,logm=50} crc {auth=269856665,config=692619927,health=1140756147,kv=439360147,logm=1035661301}) 2024-04-26T00:39:53.693 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.690+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=29,config=2,health=12,kv=9,logm=48} crc {auth=269856665,config=692619927,health=1497466813,kv=2885089190,logm=3711066070}) 2024-04-26T00:39:53.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.694+0000 7f9930690640 1 -- 172.21.15.37:0/2501011631 >> v2:172.21.15.45:6812/2379918623 conn(0x7f990c035e80 msgr2=0x7f990c038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:53.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.694+0000 7f9930690640 1 --2- 172.21.15.37:0/2501011631 >> v2:172.21.15.45:6812/2379918623 conn(0x7f990c035e80 0x7f990c038340 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f99281398f0 tx=0x7f9924007e40 comp rx=0 tx=0).stop 2024-04-26T00:39:53.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.694+0000 7f9930690640 10 monclient: shutdown 2024-04-26T00:39:53.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.694+0000 7f9930690640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:53.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.694+0000 7f9930690640 1 -- 172.21.15.37:0/2501011631 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9928133b20 msgr2=0x7f9928138120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:53.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.694+0000 7f9930690640 1 --2- 172.21.15.37:0/2501011631 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9928133b20 0x7f9928138120 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f991800c470 tx=0x7f991800c930 comp rx=0 tx=0).stop 2024-04-26T00:39:53.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.694+0000 7f9930690640 1 -- 172.21.15.37:0/2501011631 shutdown_connections 2024-04-26T00:39:53.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.694+0000 7f9930690640 1 --2- 172.21.15.37:0/2501011631 >> v2:172.21.15.45:6812/2379918623 conn(0x7f990c035e80 0x7f990c038340 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:53.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.694+0000 7f9930690640 1 --2- 172.21.15.37:0/2501011631 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f992813a060 0x7f9928138ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:53.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.694+0000 7f9930690640 1 --2- 172.21.15.37:0/2501011631 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9928133f00 0x7f9928138660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:53.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.694+0000 7f9930690640 1 --2- 172.21.15.37:0/2501011631 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9928133b20 0x7f9928138120 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:53.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.694+0000 7f9930690640 1 -- 172.21.15.37:0/2501011631 >> 172.21.15.37:0/2501011631 conn(0x7f99280a1ad0 msgr2=0x7f992809c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:53.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.694+0000 7f9930690640 1 -- 172.21.15.37:0/2501011631 shutdown_connections 2024-04-26T00:39:53.696 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.694+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:53.696 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.694+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1700243208}) 2024-04-26T00:39:53.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:53.694+0000 7f9930690640 1 -- 172.21.15.37:0/2501011631 wait complete. 2024-04-26T00:39:53.697 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.694+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3714941513}) 2024-04-26T00:39:53.697 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.694+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:53.697 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.694+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1700243208}) 2024-04-26T00:39:53.697 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.694+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1285882248}) 2024-04-26T00:39:53.698 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.698+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:53.698 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.698+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1245234563}) 2024-04-26T00:39:53.698 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.698+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3480576886}) 2024-04-26T00:39:53.698 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.698+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:53.698 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.698+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1245234563}) 2024-04-26T00:39:53.698 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.698+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3401536351}) 2024-04-26T00:39:53.701 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.702+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:53.701 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.702+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3059703004}) 2024-04-26T00:39:53.701 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.702+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1790095913}) 2024-04-26T00:39:53.701 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.702+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:53.701 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.702+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3059703004}) 2024-04-26T00:39:53.701 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.702+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3887660036}) 2024-04-26T00:39:53.704 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.702+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:53.704 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.702+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3973861150}) 2024-04-26T00:39:53.704 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.702+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4218387413}) 2024-04-26T00:39:53.704 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.702+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:53.704 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.702+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3973861150}) 2024-04-26T00:39:53.704 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.702+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2055968958}) 2024-04-26T00:39:53.707 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.706+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:53.707 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.706+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1074605294}) 2024-04-26T00:39:53.707 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.706+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=271484959}) 2024-04-26T00:39:53.707 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.706+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:53.707 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.706+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1074605294}) 2024-04-26T00:39:53.707 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.706+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2435106754}) 2024-04-26T00:39:53.709 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.710+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:53.709 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.710+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3045038385}) 2024-04-26T00:39:53.709 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.710+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1310259218}) 2024-04-26T00:39:53.709 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.710+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:53.709 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.710+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3045038385}) 2024-04-26T00:39:53.709 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.710+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3431210016}) 2024-04-26T00:39:53.711 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.710+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:53.712 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.710+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2743462534}) 2024-04-26T00:39:53.712 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.710+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3454770972}) 2024-04-26T00:39:53.712 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.710+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:53.712 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.710+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2743462534}) 2024-04-26T00:39:53.712 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.710+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1119408587}) 2024-04-26T00:39:53.714 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-26T00:39:53.715 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.714+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:53.715 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.714+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2149852532}) 2024-04-26T00:39:53.715 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.714+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1566646448}) 2024-04-26T00:39:53.715 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.714+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:53.716 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.714+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2149852532}) 2024-04-26T00:39:53.716 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.714+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1565084658}) 2024-04-26T00:39:53.717 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.714+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:53.717 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.714+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=294451683}) 2024-04-26T00:39:53.717 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.714+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2679294913}) 2024-04-26T00:39:53.717 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.714+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:53.718 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.714+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=294451683}) 2024-04-26T00:39:53.718 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.714+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3807775548}) 2024-04-26T00:39:53.829 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.826+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:53.829 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.826+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=270721780}) 2024-04-26T00:39:53.829 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.826+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=120878125}) 2024-04-26T00:39:53.829 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.826+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:53.829 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.826+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=270721780}) 2024-04-26T00:39:53.829 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.826+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1464243577}) 2024-04-26T00:39:53.835 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.834+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:53.835 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.834+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2654378208}) 2024-04-26T00:39:53.835 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.834+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=637227238}) 2024-04-26T00:39:53.835 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.834+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:53.835 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.834+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2654378208}) 2024-04-26T00:39:53.835 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.834+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=793007171}) 2024-04-26T00:39:53.854 INFO:tasks.rados.rados.0.smithi045.stdout:3486: done (1 left) 2024-04-26T00:39:53.854 INFO:tasks.rados.rados.0.smithi045.stdout:3487: read oid 47 snap -1 2024-04-26T00:39:53.854 INFO:tasks.rados.rados.0.smithi045.stdout:3487: expect (ObjNum 1127 snap 340 seq_num 1127) 2024-04-26T00:39:53.854 INFO:tasks.rados.rados.0.smithi045.stdout:3488: delete oid 21 current snap is 387 2024-04-26T00:39:53.854 INFO:tasks.rados.rados.0.smithi045.stdout:3484: expect (ObjNum 1211 snap 375 seq_num 1211) 2024-04-26T00:39:53.863 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.850+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:53.863 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.850+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2788508463}) 2024-04-26T00:39:53.863 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.850+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3762761835}) 2024-04-26T00:39:53.863 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.850+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:53.864 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.850+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2788508463}) 2024-04-26T00:39:53.864 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:53.850+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1651518908}) 2024-04-26T00:39:54.011 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.010+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.012 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.010+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2671882662}) 2024-04-26T00:39:54.012 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.010+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3022434234}) 2024-04-26T00:39:54.012 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.010+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.012 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.010+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2671882662}) 2024-04-26T00:39:54.012 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.010+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1921111629}) 2024-04-26T00:39:54.015 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.014+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.015 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.014+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3714700327}) 2024-04-26T00:39:54.015 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.014+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1196874047}) 2024-04-26T00:39:54.016 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.014+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.016 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.014+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3714700327}) 2024-04-26T00:39:54.016 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.014+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=984672314}) 2024-04-26T00:39:54.022 INFO:tasks.rados.rados.0.smithi045.stdout:3487: expect (ObjNum 1127 snap 340 seq_num 1127) 2024-04-26T00:39:54.177 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.178+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.177 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.178+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2761749458}) 2024-04-26T00:39:54.177 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.178+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=927725435}) 2024-04-26T00:39:54.177 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.178+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.177 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.178+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2761749458}) 2024-04-26T00:39:54.177 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.178+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=400763724}) 2024-04-26T00:39:54.179 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.178+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.179 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.178+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1805997057}) 2024-04-26T00:39:54.179 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.178+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1337930446}) 2024-04-26T00:39:54.179 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.178+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.179 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.178+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1805997057}) 2024-04-26T00:39:54.179 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.178+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3682853740}) 2024-04-26T00:39:54.182 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.182+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.182 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.182+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4094259690}) 2024-04-26T00:39:54.182 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.182+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=213014171}) 2024-04-26T00:39:54.182 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.182+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.182 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.182+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4094259690}) 2024-04-26T00:39:54.182 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.182+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1031825389}) 2024-04-26T00:39:54.184 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.182+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.185 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.182+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=137184463}) 2024-04-26T00:39:54.185 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.182+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2096480193}) 2024-04-26T00:39:54.185 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.182+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.185 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.182+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=137184463}) 2024-04-26T00:39:54.185 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.182+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=719088872}) 2024-04-26T00:39:54.187 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.186+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.187 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.186+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1971976359}) 2024-04-26T00:39:54.188 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.186+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=87904766}) 2024-04-26T00:39:54.188 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.186+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.188 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.186+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1971976359}) 2024-04-26T00:39:54.188 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.186+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3202156250}) 2024-04-26T00:39:54.190 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.190+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.190 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.190+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3566848328}) 2024-04-26T00:39:54.190 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.190+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3550075323}) 2024-04-26T00:39:54.190 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.190+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.190 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.190+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3566848328}) 2024-04-26T00:39:54.190 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.190+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3110829910}) 2024-04-26T00:39:54.192 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.190+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.192 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.190+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=584793621}) 2024-04-26T00:39:54.193 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.190+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=679419328}) 2024-04-26T00:39:54.193 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.190+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.193 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.190+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=584793621}) 2024-04-26T00:39:54.193 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.190+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3233757643}) 2024-04-26T00:39:54.195 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.194+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.195 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.194+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1249125211}) 2024-04-26T00:39:54.195 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.194+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=72451630}) 2024-04-26T00:39:54.195 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.194+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.195 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.194+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1249125211}) 2024-04-26T00:39:54.195 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.194+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1934169747}) 2024-04-26T00:39:54.198 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.198+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.198 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.198+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2970301553}) 2024-04-26T00:39:54.198 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.198+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=50707229}) 2024-04-26T00:39:54.198 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.198+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.198 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.198+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2970301553}) 2024-04-26T00:39:54.198 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.198+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3721734339}) 2024-04-26T00:39:54.200 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.198+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.200 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.198+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1407060383}) 2024-04-26T00:39:54.200 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.198+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4207779521}) 2024-04-26T00:39:54.200 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.198+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.200 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.198+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1407060383}) 2024-04-26T00:39:54.200 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.198+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4291400877}) 2024-04-26T00:39:54.202 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.202+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.203 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.202+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4285498835}) 2024-04-26T00:39:54.203 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.202+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3565739940}) 2024-04-26T00:39:54.203 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.202+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.203 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.202+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4285498835}) 2024-04-26T00:39:54.203 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.202+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3522379576}) 2024-04-26T00:39:54.205 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.206+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.205 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.206+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=737793088}) 2024-04-26T00:39:54.205 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.206+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1460543766}) 2024-04-26T00:39:54.205 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.206+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.205 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.206+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=737793088}) 2024-04-26T00:39:54.205 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.206+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2954162185}) 2024-04-26T00:39:54.208 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.206+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.208 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.206+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1915319298}) 2024-04-26T00:39:54.208 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.206+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4167030725}) 2024-04-26T00:39:54.208 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.206+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.208 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.206+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1915319298}) 2024-04-26T00:39:54.208 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.206+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=939659380}) 2024-04-26T00:39:54.210 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.210+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.210 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.210+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=141368731}) 2024-04-26T00:39:54.210 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.210+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3365299005}) 2024-04-26T00:39:54.210 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.210+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.210 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.210+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=141368731}) 2024-04-26T00:39:54.210 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.210+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1405181460}) 2024-04-26T00:39:54.211 INFO:tasks.rados.rados.0.smithi045.stdout:3484: done (2 left) 2024-04-26T00:39:54.211 INFO:tasks.rados.rados.0.smithi045.stdout:3487: done (1 left) 2024-04-26T00:39:54.211 INFO:tasks.rados.rados.0.smithi045.stdout:3488: done (0 left) 2024-04-26T00:39:54.211 INFO:tasks.rados.rados.0.smithi045.stdout:3489: copy_from oid 37 from oid 38 current snap is 387 2024-04-26T00:39:54.211 INFO:tasks.rados.rados.0.smithi045.stdout:3490: delete oid 35 current snap is 387 2024-04-26T00:39:54.213 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.214+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.213 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.214+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1864906245}) 2024-04-26T00:39:54.213 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.214+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=52083732}) 2024-04-26T00:39:54.213 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.214+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.213 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.214+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1864906245}) 2024-04-26T00:39:54.214 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.214+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2756379194}) 2024-04-26T00:39:54.216 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.214+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.216 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.214+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1231556872}) 2024-04-26T00:39:54.217 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.214+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=420696271}) 2024-04-26T00:39:54.217 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.214+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.217 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.214+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1231556872}) 2024-04-26T00:39:54.217 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.214+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=945187863}) 2024-04-26T00:39:54.220 INFO:tasks.rados.rados.0.smithi045.stdout:3490: done (1 left) 2024-04-26T00:39:54.220 INFO:tasks.rados.rados.0.smithi045.stdout:3491: copy_from oid 5 from oid 16 current snap is 387 2024-04-26T00:39:54.220 INFO:tasks.rados.rados.0.smithi045.stdout:3492: read oid 36 snap -1 2024-04-26T00:39:54.220 INFO:tasks.rados.rados.0.smithi045.stdout:3492: expect (ObjNum 1197 snap 372 seq_num 1197) 2024-04-26T00:39:54.220 INFO:tasks.rados.rados.0.smithi045.stdout:3493: snap_remove snap 374 2024-04-26T00:39:54.221 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.218+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.221 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.218+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1971170356}) 2024-04-26T00:39:54.221 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.218+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1971144857}) 2024-04-26T00:39:54.221 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.218+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.221 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.222+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1971170356}) 2024-04-26T00:39:54.221 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.222+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1630034338}) 2024-04-26T00:39:54.225 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.222+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.225 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.222+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=141583152}) 2024-04-26T00:39:54.225 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.222+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=312169665}) 2024-04-26T00:39:54.225 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.222+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.225 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.222+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=141583152}) 2024-04-26T00:39:54.225 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.222+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1178971128}) 2024-04-26T00:39:54.227 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.226+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.227 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.226+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3509802560}) 2024-04-26T00:39:54.228 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.226+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2721921251}) 2024-04-26T00:39:54.228 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.226+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.228 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.226+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3509802560}) 2024-04-26T00:39:54.228 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.226+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3754236171}) 2024-04-26T00:39:54.231 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.230+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.231 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.230+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2357830927}) 2024-04-26T00:39:54.231 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.230+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4060120412}) 2024-04-26T00:39:54.231 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.230+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.231 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.230+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2357830927}) 2024-04-26T00:39:54.231 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.230+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2536205306}) 2024-04-26T00:39:54.235 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.234+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.235 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.234+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=946002030}) 2024-04-26T00:39:54.235 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.234+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3629922149}) 2024-04-26T00:39:54.235 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.234+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.235 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.234+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=946002030}) 2024-04-26T00:39:54.235 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.234+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2836088921}) 2024-04-26T00:39:54.238 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.238+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.238 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.238+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1237214815}) 2024-04-26T00:39:54.238 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.238+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=785007444}) 2024-04-26T00:39:54.238 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.238+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.238 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.238+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1237214815}) 2024-04-26T00:39:54.238 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.238+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3327808712}) 2024-04-26T00:39:54.461 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.458+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.461 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.458+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1903500145}) 2024-04-26T00:39:54.461 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.458+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2083302601}) 2024-04-26T00:39:54.461 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.458+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.461 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.458+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1903500145}) 2024-04-26T00:39:54.461 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.458+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1493404471}) 2024-04-26T00:39:54.476 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.470+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.476 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.470+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2064763328}) 2024-04-26T00:39:54.476 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.470+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2565330053}) 2024-04-26T00:39:54.476 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.470+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.476 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.470+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2064763328}) 2024-04-26T00:39:54.477 INFO:tasks.rados.rados.0.smithi045.stdout:3493: done (3 left) 2024-04-26T00:39:54.477 INFO:tasks.rados.rados.0.smithi045.stdout:3494: delete oid 15 current snap is 387 2024-04-26T00:39:54.477 INFO:tasks.rados.rados.0.smithi045.stdout:3492: expect (ObjNum 1197 snap 372 seq_num 1197) 2024-04-26T00:39:54.477 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.470+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3640285070}) 2024-04-26T00:39:54.506 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.506+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.506 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.506+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2743760411}) 2024-04-26T00:39:54.506 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.506+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3556113409}) 2024-04-26T00:39:54.506 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.506+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.506 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.506+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2743760411}) 2024-04-26T00:39:54.506 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.506+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4076029309}) 2024-04-26T00:39:54.661 INFO:tasks.rados.rados.0.smithi045.stdout:3489: finishing copy_from racing read to smithi04526579-37 2024-04-26T00:39:54.661 INFO:tasks.rados.rados.0.smithi045.stdout:3489: finishing copy_from to smithi04526579-37 2024-04-26T00:39:54.661 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 2398 (ObjNum 1075 snap 322 seq_num 1075) dirty exists 2024-04-26T00:39:54.661 INFO:tasks.rados.rados.0.smithi045.stdout:3491: finishing copy_from racing read to smithi04526579-5 2024-04-26T00:39:54.661 INFO:tasks.rados.rados.0.smithi045.stdout:3491: finishing copy_from to smithi04526579-5 2024-04-26T00:39:54.661 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 1207 (ObjNum 1102 snap 329 seq_num 1102) dirty exists 2024-04-26T00:39:54.661 INFO:tasks.rados.rados.0.smithi045.stdout:3489: done (3 left) 2024-04-26T00:39:54.662 INFO:tasks.rados.rados.0.smithi045.stdout:3491: done (2 left) 2024-04-26T00:39:54.662 INFO:tasks.rados.rados.0.smithi045.stdout:3492: done (1 left) 2024-04-26T00:39:54.662 INFO:tasks.rados.rados.0.smithi045.stdout:3494: done (0 left) 2024-04-26T00:39:54.662 INFO:tasks.rados.rados.0.smithi045.stdout:3495: snap_remove snap 380 2024-04-26T00:39:54.714 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:39:54.715 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:39:54.829 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.830+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.830 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.830+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1943441226}) 2024-04-26T00:39:54.830 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.830+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2827572659}) 2024-04-26T00:39:54.830 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.830+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.830 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.830+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1943441226}) 2024-04-26T00:39:54.830 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.830+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2656749402}) 2024-04-26T00:39:54.831 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.830+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.831 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.830+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1890917732}) 2024-04-26T00:39:54.832 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.830+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2378167412}) 2024-04-26T00:39:54.832 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.830+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.832 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.830+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1890917732}) 2024-04-26T00:39:54.832 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.830+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1494246348}) 2024-04-26T00:39:54.833 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.834+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.833 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.834+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4013972233}) 2024-04-26T00:39:54.833 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.834+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1581596221}) 2024-04-26T00:39:54.834 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.834+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.834 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.834+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4013972233}) 2024-04-26T00:39:54.834 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.834+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2358603524}) 2024-04-26T00:39:54.835 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.834+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.835 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.834+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1103300577}) 2024-04-26T00:39:54.836 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.834+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2397762361}) 2024-04-26T00:39:54.836 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.834+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.836 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.834+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1103300577}) 2024-04-26T00:39:54.836 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.834+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=864549361}) 2024-04-26T00:39:54.837 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.838+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.838 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.838+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=334961551}) 2024-04-26T00:39:54.838 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.838+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3645082499}) 2024-04-26T00:39:54.838 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.838+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.838 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.838+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=334961551}) 2024-04-26T00:39:54.838 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.838+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1593949934}) 2024-04-26T00:39:54.839 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.838+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.840 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.838+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2259916324}) 2024-04-26T00:39:54.840 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.838+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=849805505}) 2024-04-26T00:39:54.840 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.838+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.840 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.838+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2259916324}) 2024-04-26T00:39:54.840 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.838+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3793986928}) 2024-04-26T00:39:54.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e34a7640 10 monclient: get_monmap_and_config 2024-04-26T00:39:54.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e34a7640 10 monclient: build_initial_monmap 2024-04-26T00:39:54.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e34a7640 10 monclient: monmap: 2024-04-26T00:39:54.841 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:54.841 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:54.841 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:39:54.844118+0000 2024-04-26T00:39:54.841 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:39:54.844118+0000 2024-04-26T00:39:54.841 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:54.841 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:54.841 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:39:54.841 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:39:54.841 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:39:54.841 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:39:54.841 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:39:54.842 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:54.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e34a7640 1 Processor -- start 2024-04-26T00:39:54.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e34a7640 1 -- start start 2024-04-26T00:39:54.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e34a7640 10 monclient: init 2024-04-26T00:39:54.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e34a7640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:54.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e34a7640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-04-26T00:39:54.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e34a7640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f73dc136e10 0x7f73dc09c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:54.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e34a7640 10 monclient(hunting): picked mon.noname-a con 0x7f73dc136e10 addr v2:172.21.15.37:3300/0 2024-04-26T00:39:54.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e34a7640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f73dc1371f0 0x7f73dc137600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:54.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e34a7640 10 monclient(hunting): picked mon.noname-c con 0x7f73dc1371f0 addr v2:172.21.15.45:3300/0 2024-04-26T00:39:54.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e34a7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f73dc135d90 addr v1:172.21.15.45:6789/0 2024-04-26T00:39:54.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e34a7640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73dc09b680 con 0x7f73dc135d90 2024-04-26T00:39:54.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e34a7640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:54.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e34a7640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f73dc09b8f0 con 0x7f73dc136e10 2024-04-26T00:39:54.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e34a7640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:54.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e34a7640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f73dc09bbd0 con 0x7f73dc1371f0 2024-04-26T00:39:54.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e34a7640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:54.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e34a7640 10 monclient(hunting): authenticate will time out at 2176.150102s 2024-04-26T00:39:54.844 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.845 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=887168760}) 2024-04-26T00:39:54.845 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1047754725}) 2024-04-26T00:39:54.845 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.845 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=887168760}) 2024-04-26T00:39:54.845 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1430525601}) 2024-04-26T00:39:54.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e1243640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f73dc136e10 0x7f73dc09c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:54.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e1243640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f73dc136e10 0x7f73dc09c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:45372/0 (socket says 172.21.15.37:45372) 2024-04-26T00:39:54.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e1243640 1 -- 172.21.15.37:0/969416664 learned_addr learned my addr 172.21.15.37:0/969416664 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:54.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e1243640 10 monclient(hunting): get_auth_request con 0x7f73dc136e10 auth_method 0 2024-04-26T00:39:54.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e1243640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:54.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e1243640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:54.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e1243640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:54.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e0a42640 1 --2- 172.21.15.37:0/969416664 >> v2:172.21.15.45:3300/0 conn(0x7f73dc1371f0 0x7f73dc137600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:54.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e0a42640 10 monclient(hunting): get_auth_request con 0x7f73dc1371f0 auth_method 0 2024-04-26T00:39:54.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e0a42640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:54.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e0a42640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:54.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e0a42640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:54.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e1243640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:54.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e1243640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:54.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e1243640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:54.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e0a42640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:54.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e0a42640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:54.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e0a42640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:54.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e1243640 10 monclient(hunting): handle_auth_done global_id 244113 payload 274 2024-04-26T00:39:54.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e1243640 1 -- 172.21.15.37:0/969416664 >> v2:172.21.15.45:3300/0 conn(0x7f73dc1371f0 msgr2=0x7f73dc137600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:54.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e1243640 1 --2- 172.21.15.37:0/969416664 >> v2:172.21.15.45:3300/0 conn(0x7f73dc1371f0 0x7f73dc137600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:54.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e1243640 1 -- 172.21.15.37:0/969416664 >> v1:172.21.15.45:6789/0 conn(0x7f73dc135d90 legacy=0x7f73dc136220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:54.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e1243640 10 monclient: _finish_hunting 0 2024-04-26T00:39:54.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e1243640 1 monclient: found mon.noname-a 2024-04-26T00:39:54.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e1243640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:54.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e1243640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:39:54.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.842+0000 7f73e1243640 1 -- 172.21.15.37:0/969416664 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73dc137b40 con 0x7f73dc136e10 2024-04-26T00:39:54.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.846+0000 7f73e1243640 1 -- 172.21.15.37:0/969416664 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f73dc136e10 msgr2=0x7f73dc09c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:54.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.846+0000 7f73e1243640 1 --2- 172.21.15.37:0/969416664 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f73dc136e10 0x7f73dc09c5b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f73dc068e60 tx=0x7f73d001c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a5d4acbfc3eb1b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:54.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.850+0000 7f73cffff640 1 -- 172.21.15.37:0/969416664 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d00a5750 con 0x7f73dc136e10 2024-04-26T00:39:54.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.850+0000 7f73cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:54.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.850+0000 7f73cffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:39:54.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.850+0000 7f73cffff640 10 monclient: dump: 2024-04-26T00:39:54.852 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:54.852 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:54.852 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:54.852 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:54.852 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:54.852 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:54.852 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:54.852 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:54.852 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:54.852 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:54.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.850+0000 7f73cffff640 10 monclient: _finish_auth 0 2024-04-26T00:39:54.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.850+0000 7f73cffff640 10 monclient: _check_auth_tickets 2024-04-26T00:39:54.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.850+0000 7f73cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:54.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.850+0000 7f73e34a7640 5 monclient: authenticate success, global_id 244113 2024-04-26T00:39:54.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.850+0000 7f73cffff640 1 -- 172.21.15.37:0/969416664 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73d00a58f0 con 0x7f73dc136e10 2024-04-26T00:39:54.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.850+0000 7f73e34a7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:39:54.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.850+0000 7f73cffff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:54.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.850+0000 7f73e34a7640 10 monclient: get_monmap_and_config success 2024-04-26T00:39:54.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.850+0000 7f73e34a7640 4 set_mon_vals no callback set 2024-04-26T00:39:54.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.850+0000 7f73cffff640 1 -- 172.21.15.37:0/969416664 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d00a5cc0 con 0x7f73dc136e10 2024-04-26T00:39:54.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.850+0000 7f73cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:54.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.850+0000 7f73cffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:39:54.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.850+0000 7f73cffff640 10 monclient: dump: 2024-04-26T00:39:54.853 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:54.853 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:54.853 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:54.853 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:54.853 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:54.853 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:54.853 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:54.853 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:54.854 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:54.854 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:54.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.850+0000 7f73e34a7640 10 monclient: shutdown 2024-04-26T00:39:54.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.850+0000 7f73e34a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:54.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.850+0000 7f73e34a7640 1 -- 172.21.15.37:0/969416664 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f73dc136e10 msgr2=0x7f73dc09c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:54.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.850+0000 7f73e34a7640 1 --2- 172.21.15.37:0/969416664 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f73dc136e10 0x7f73dc09c5b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f73dc068e60 tx=0x7f73d001c870 comp rx=0 tx=0).stop 2024-04-26T00:39:54.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.850+0000 7f73e34a7640 1 -- 172.21.15.37:0/969416664 shutdown_connections 2024-04-26T00:39:54.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.850+0000 7f73e34a7640 1 --2- 172.21.15.37:0/969416664 >> v2:172.21.15.45:3300/0 conn(0x7f73dc1371f0 0x7f73dc137600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:54.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.850+0000 7f73e34a7640 1 --2- 172.21.15.37:0/969416664 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f73dc136e10 0x7f73dc09c5b0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:54.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.850+0000 7f73e34a7640 1 -- 172.21.15.37:0/969416664 >> 172.21.15.37:0/969416664 conn(0x7f73dc0a64e0 msgr2=0x7f73dc0a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:54.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.850+0000 7f73e34a7640 1 -- 172.21.15.37:0/969416664 shutdown_connections 2024-04-26T00:39:54.855 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.850+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.855 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.850+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3473676007}) 2024-04-26T00:39:54.855 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.850+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2233373157}) 2024-04-26T00:39:54.855 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.850+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.855 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.850+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3473676007}) 2024-04-26T00:39:54.855 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.850+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3903757681}) 2024-04-26T00:39:54.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.850+0000 7f73e34a7640 1 -- 172.21.15.37:0/969416664 wait complete. 2024-04-26T00:39:54.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.850+0000 7f73e34a7640 10 monclient: build_initial_monmap 2024-04-26T00:39:54.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.850+0000 7f73e34a7640 10 monclient: monmap: 2024-04-26T00:39:54.859 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:54.859 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:39:54.859 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:39:54.859 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:39:54.859 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:54.860 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:39:54.860 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:39:54.860 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:39:54.860 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:39:54.860 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:54.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e34a7640 1 Processor -- start 2024-04-26T00:39:54.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e34a7640 1 -- start start 2024-04-26T00:39:54.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e34a7640 10 monclient: init 2024-04-26T00:39:54.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e34a7640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:54.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e34a7640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:39:54.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e34a7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73dc135d90 0x7f73dc13d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:54.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e34a7640 10 monclient(hunting): picked mon.noname-b con 0x7f73dc135d90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:39:54.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e34a7640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f73dc136e10 0x7f73dc13b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:54.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e34a7640 10 monclient(hunting): picked mon.noname-a con 0x7f73dc136e10 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:39:54.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e34a7640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73dc1371f0 0x7f73dc134fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:54.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e34a7640 10 monclient(hunting): picked mon.noname-c con 0x7f73dc1371f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:54.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e34a7640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:54.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e34a7640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f73dc141f10 con 0x7f73dc136e10 2024-04-26T00:39:54.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e34a7640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:54.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e34a7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f73dc141d90 con 0x7f73dc135d90 2024-04-26T00:39:54.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e34a7640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:54.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e34a7640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f73dc142090 con 0x7f73dc1371f0 2024-04-26T00:39:54.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e34a7640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:54.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e34a7640 10 monclient(hunting): authenticate will time out at 2176.160870s 2024-04-26T00:39:54.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e0a42640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f73dc136e10 0x7f73dc13b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:54.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e1a44640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73dc1371f0 0x7f73dc134fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:54.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e0a42640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f73dc136e10 0x7f73dc13b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:45380/0 (socket says 172.21.15.37:45380) 2024-04-26T00:39:54.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e0a42640 1 -- 172.21.15.37:0/3025534825 learned_addr learned my addr 172.21.15.37:0/3025534825 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:54.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e0a42640 10 monclient(hunting): get_auth_request con 0x7f73dc136e10 auth_method 0 2024-04-26T00:39:54.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e0a42640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:54.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e0a42640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:54.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e0a42640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:54.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e1a44640 10 monclient(hunting): get_auth_request con 0x7f73dc1371f0 auth_method 0 2024-04-26T00:39:54.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e1a44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:54.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e1a44640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:54.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e1a44640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:54.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e0a42640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:54.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e0a42640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:54.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e0a42640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:54.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e1a44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:54.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e1a44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:54.862 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e1a44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:54.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e1243640 1 --2- 172.21.15.37:0/3025534825 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73dc135d90 0x7f73dc13d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:54.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e1243640 10 monclient(hunting): get_auth_request con 0x7f73dc135d90 auth_method 0 2024-04-26T00:39:54.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e1243640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:54.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e1243640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:54.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e1243640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:54.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e0a42640 10 monclient(hunting): handle_auth_done global_id 244116 payload 824 2024-04-26T00:39:54.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e0a42640 1 -- 172.21.15.37:0/3025534825 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73dc1371f0 msgr2=0x7f73dc134fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:54.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e0a42640 1 --2- 172.21.15.37:0/3025534825 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73dc1371f0 0x7f73dc134fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:54.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e0a42640 1 -- 172.21.15.37:0/3025534825 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73dc135d90 msgr2=0x7f73dc13d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:54.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e0a42640 1 --2- 172.21.15.37:0/3025534825 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73dc135d90 0x7f73dc13d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:54.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e0a42640 10 monclient: _finish_hunting 0 2024-04-26T00:39:54.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e0a42640 1 monclient: found mon.noname-a 2024-04-26T00:39:54.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e0a42640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:54.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e0a42640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:39:54.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e0a42640 1 -- 172.21.15.37:0/3025534825 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73dc0a7fd0 con 0x7f73dc136e10 2024-04-26T00:39:54.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e1a44640 1 --2- 172.21.15.37:0/3025534825 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73dc1371f0 0x7f73dc134fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:39:54.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e1243640 1 --2- 172.21.15.37:0/3025534825 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73dc135d90 0x7f73dc13d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:39:54.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e0a42640 1 -- 172.21.15.37:0/3025534825 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f73dc136e10 msgr2=0x7f73dc13b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:54.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e0a42640 1 --2- 172.21.15.37:0/3025534825 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f73dc136e10 0x7f73dc13b100 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f73d800e930 tx=0x7f73d800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:54.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73cdffb640 1 -- 172.21.15.37:0/3025534825 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d8017070 con 0x7f73dc136e10 2024-04-26T00:39:54.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:54.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73cdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:39:54.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73cdffb640 10 monclient: dump: 2024-04-26T00:39:54.864 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:54.864 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:54.865 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:54.865 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:54.865 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:54.865 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:54.865 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:54.865 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:54.865 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:54.865 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:54.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73cdffb640 10 monclient: _finish_auth 0 2024-04-26T00:39:54.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73cdffb640 10 monclient: _check_auth_tickets 2024-04-26T00:39:54.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:54.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73cdffb640 1 -- 172.21.15.37:0/3025534825 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73d800cc10 con 0x7f73dc136e10 2024-04-26T00:39:54.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73cdffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:54.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73cdffb640 1 -- 172.21.15.37:0/3025534825 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d8013470 con 0x7f73dc136e10 2024-04-26T00:39:54.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:54.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73cdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:39:54.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73cdffb640 10 monclient: dump: 2024-04-26T00:39:54.866 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:54.866 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:54.866 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:54.866 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:54.866 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:54.866 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:54.866 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:54.866 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:54.866 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:54.866 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:54.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e34a7640 5 monclient: authenticate success, global_id 244116 2024-04-26T00:39:54.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e34a7640 10 monclient: _renew_subs 2024-04-26T00:39:54.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73cf7fe640 4 set_mon_vals no callback set 2024-04-26T00:39:54.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.854+0000 7f73e34a7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:39:54.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.858+0000 7f73e34a7640 1 -- 172.21.15.37:0/3025534825 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73dc13c8e0 con 0x7f73dc136e10 2024-04-26T00:39:54.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.858+0000 7f73e34a7640 10 monclient: _renew_subs 2024-04-26T00:39:54.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.858+0000 7f73e34a7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:39:54.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.858+0000 7f73e34a7640 1 -- 172.21.15.37:0/3025534825 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73dc13d1c0 con 0x7f73dc136e10 2024-04-26T00:39:54.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.858+0000 7f73cdffb640 1 -- 172.21.15.37:0/3025534825 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f73d801a070 con 0x7f73dc136e10 2024-04-26T00:39:54.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.858+0000 7f73e34a7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:54.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.858+0000 7f73e34a7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:54.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.858+0000 7f73e34a7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:39:54.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.858+0000 7f73e34a7640 1 -- 172.21.15.37:0/3025534825 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73dc0a7cc0 con 0x7f73dc136e10 2024-04-26T00:39:54.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.858+0000 7f73cdffb640 1 --2- 172.21.15.37:0/3025534825 >> v2:172.21.15.45:6812/2379918623 conn(0x7f73c0035e80 0x7f73c0038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:54.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.858+0000 7f73cdffb640 1 -- 172.21.15.37:0/3025534825 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(791..791 src has 705..791) ==== 4003+0+0 (secure 0 0 0) 0x7f73d8048c80 con 0x7f73dc136e10 2024-04-26T00:39:54.868 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.858+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.868 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.858+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1714274478}) 2024-04-26T00:39:54.868 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.858+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1199583726}) 2024-04-26T00:39:54.868 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.858+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.868 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.858+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1714274478}) 2024-04-26T00:39:54.869 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.858+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1953898432}) 2024-04-26T00:39:54.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.858+0000 7f73e1243640 1 --2- 172.21.15.37:0/3025534825 >> v2:172.21.15.45:6812/2379918623 conn(0x7f73c0035e80 0x7f73c0038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:54.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.858+0000 7f73e1243640 10 monclient: get_auth_request con 0x7f73c0035e80 auth_method 0 2024-04-26T00:39:54.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.862+0000 7f73e1243640 1 --2- 172.21.15.37:0/3025534825 >> v2:172.21.15.45:6812/2379918623 conn(0x7f73c0035e80 0x7f73c0038340 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f73d00ab000 tx=0x7f73d00a7070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:54.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.862+0000 7f73cdffb640 1 -- 172.21.15.37:0/3025534825 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f73d8047980 con 0x7f73dc136e10 2024-04-26T00:39:54.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.862+0000 7f73cdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:54.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:54.862+0000 7f73cdffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:39:54.872 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.862+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.872 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.862+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=559633286}) 2024-04-26T00:39:54.872 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.862+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4266800278}) 2024-04-26T00:39:54.872 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.862+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.872 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.862+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=559633286}) 2024-04-26T00:39:54.872 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.862+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1240413119}) 2024-04-26T00:39:54.872 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.866+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.872 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.866+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1353195756}) 2024-04-26T00:39:54.872 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.866+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=818959251}) 2024-04-26T00:39:54.872 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.866+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.873 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.866+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1353195756}) 2024-04-26T00:39:54.873 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.866+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=841551087}) 2024-04-26T00:39:54.873 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.866+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.873 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.866+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1573803225}) 2024-04-26T00:39:54.873 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.866+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1760300665}) 2024-04-26T00:39:54.873 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.866+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.873 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.866+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1573803225}) 2024-04-26T00:39:54.873 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.866+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1319237648}) 2024-04-26T00:39:54.873 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.870+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.873 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.870+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2403374315}) 2024-04-26T00:39:54.873 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.870+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=351010084}) 2024-04-26T00:39:54.873 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.870+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.873 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.870+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2403374315}) 2024-04-26T00:39:54.873 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.870+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=617990360}) 2024-04-26T00:39:54.874 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.870+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.874 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.870+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3037013961}) 2024-04-26T00:39:54.874 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.870+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3239517359}) 2024-04-26T00:39:54.874 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.870+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.874 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.870+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3037013961}) 2024-04-26T00:39:54.874 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.870+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4101685010}) 2024-04-26T00:39:54.875 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.874+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.875 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.874+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3058939953}) 2024-04-26T00:39:54.875 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.874+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2749402278}) 2024-04-26T00:39:54.875 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.874+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.875 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.874+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3058939953}) 2024-04-26T00:39:54.875 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.874+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3493119329}) 2024-04-26T00:39:54.876 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.874+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.876 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.874+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4050096401}) 2024-04-26T00:39:54.876 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.874+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1605830546}) 2024-04-26T00:39:54.876 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.874+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.876 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.874+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4050096401}) 2024-04-26T00:39:54.876 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.874+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=723106860}) 2024-04-26T00:39:54.877 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.878+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.877 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.878+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3348333689}) 2024-04-26T00:39:54.877 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.878+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4209351197}) 2024-04-26T00:39:54.877 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.878+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.877 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.878+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3348333689}) 2024-04-26T00:39:54.877 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.878+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1871690169}) 2024-04-26T00:39:54.879 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.878+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.879 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.878+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2333443999}) 2024-04-26T00:39:54.879 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.878+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1066082975}) 2024-04-26T00:39:54.879 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.878+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.879 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.878+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2333443999}) 2024-04-26T00:39:54.879 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.878+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2584118447}) 2024-04-26T00:39:54.881 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.882+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.881 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.882+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2282971424}) 2024-04-26T00:39:54.881 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.882+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1250102226}) 2024-04-26T00:39:54.881 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.882+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.881 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.882+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2282971424}) 2024-04-26T00:39:54.881 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.882+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3428353423}) 2024-04-26T00:39:54.883 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.882+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.883 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.882+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1559793162}) 2024-04-26T00:39:54.883 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.882+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=522404777}) 2024-04-26T00:39:54.883 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.882+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.883 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.882+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1559793162}) 2024-04-26T00:39:54.883 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.882+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4197893488}) 2024-04-26T00:39:54.885 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.886+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.885 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.886+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1167452227}) 2024-04-26T00:39:54.885 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.886+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1901984945}) 2024-04-26T00:39:54.885 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.886+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.885 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.886+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1167452227}) 2024-04-26T00:39:54.885 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.886+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4242000664}) 2024-04-26T00:39:54.887 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.886+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.887 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.886+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1794087850}) 2024-04-26T00:39:54.887 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.886+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2945161666}) 2024-04-26T00:39:54.887 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.886+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.887 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.886+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1794087850}) 2024-04-26T00:39:54.887 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.886+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3805405651}) 2024-04-26T00:39:54.889 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.886+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.889 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.886+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2246365479}) 2024-04-26T00:39:54.889 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.890+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2060131113}) 2024-04-26T00:39:54.889 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.890+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.889 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.890+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2246365479}) 2024-04-26T00:39:54.889 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.890+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3459417112}) 2024-04-26T00:39:54.890 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.890+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.891 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.890+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2223603747}) 2024-04-26T00:39:54.891 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.890+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2778492873}) 2024-04-26T00:39:54.891 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.890+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.891 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.890+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2223603747}) 2024-04-26T00:39:54.891 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.890+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=930586692}) 2024-04-26T00:39:54.893 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.894+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.893 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.894+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4285308359}) 2024-04-26T00:39:54.893 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.894+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4251279540}) 2024-04-26T00:39:54.893 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.894+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.893 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.894+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4285308359}) 2024-04-26T00:39:54.893 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.894+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3285101684}) 2024-04-26T00:39:54.895 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.894+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.895 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.894+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2402250095}) 2024-04-26T00:39:54.895 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.894+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3450068567}) 2024-04-26T00:39:54.895 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.894+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.895 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.894+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2402250095}) 2024-04-26T00:39:54.895 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.894+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2184936428}) 2024-04-26T00:39:54.897 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.898+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.897 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.898+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=693393429}) 2024-04-26T00:39:54.897 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.898+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=445138744}) 2024-04-26T00:39:54.897 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.898+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.897 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.898+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=693393429}) 2024-04-26T00:39:54.897 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.898+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=142690763}) 2024-04-26T00:39:54.899 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.898+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.899 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.898+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3290375146}) 2024-04-26T00:39:54.899 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.898+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1239708545}) 2024-04-26T00:39:54.899 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.898+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.899 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.898+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3290375146}) 2024-04-26T00:39:54.899 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.898+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1344526935}) 2024-04-26T00:39:54.901 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.902+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.901 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.902+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1200733048}) 2024-04-26T00:39:54.901 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.902+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3139009510}) 2024-04-26T00:39:54.901 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.902+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.901 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.902+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1200733048}) 2024-04-26T00:39:54.902 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.902+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1162399136}) 2024-04-26T00:39:54.903 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.902+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.903 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.902+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2670798597}) 2024-04-26T00:39:54.903 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.902+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3636568989}) 2024-04-26T00:39:54.903 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.902+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.903 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.902+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2670798597}) 2024-04-26T00:39:54.904 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.902+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2991569373}) 2024-04-26T00:39:54.906 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.906+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.906 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.906+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4254124734}) 2024-04-26T00:39:54.906 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.906+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3805888265}) 2024-04-26T00:39:54.906 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.906+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.906 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.906+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4254124734}) 2024-04-26T00:39:54.906 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.906+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2994878499}) 2024-04-26T00:39:54.908 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.906+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.908 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.906+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=585129744}) 2024-04-26T00:39:54.908 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.906+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1266890584}) 2024-04-26T00:39:54.908 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.906+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.908 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.906+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=585129744}) 2024-04-26T00:39:54.908 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.906+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3984678889}) 2024-04-26T00:39:54.910 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.910+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.910 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.910+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3583315394}) 2024-04-26T00:39:54.910 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.910+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2924032788}) 2024-04-26T00:39:54.910 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.910+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.910 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.910+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3583315394}) 2024-04-26T00:39:54.910 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.910+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3374809043}) 2024-04-26T00:39:54.912 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.910+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.912 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.910+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2787427360}) 2024-04-26T00:39:54.912 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.910+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3994838148}) 2024-04-26T00:39:54.912 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.910+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.912 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.910+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2787427360}) 2024-04-26T00:39:54.913 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.910+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2317177283}) 2024-04-26T00:39:54.915 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.914+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.915 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.914+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1300885582}) 2024-04-26T00:39:54.915 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.914+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=190209067}) 2024-04-26T00:39:54.915 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.914+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.915 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.914+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1300885582}) 2024-04-26T00:39:54.915 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.914+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2391520048}) 2024-04-26T00:39:54.917 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.918+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.917 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.918+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3941019642}) 2024-04-26T00:39:54.917 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.918+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=558893667}) 2024-04-26T00:39:54.917 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.918+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.917 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.918+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3941019642}) 2024-04-26T00:39:54.917 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.918+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2476635045}) 2024-04-26T00:39:54.919 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.918+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.919 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.918+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1722051119}) 2024-04-26T00:39:54.919 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.918+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2453646370}) 2024-04-26T00:39:54.919 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.918+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.919 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.918+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1722051119}) 2024-04-26T00:39:54.919 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.918+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2823799071}) 2024-04-26T00:39:54.921 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.922+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.921 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.922+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3796204379}) 2024-04-26T00:39:54.921 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.922+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1034667706}) 2024-04-26T00:39:54.921 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.922+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.921 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.922+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3796204379}) 2024-04-26T00:39:54.921 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.922+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3293240196}) 2024-04-26T00:39:54.923 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.922+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.923 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.922+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3162558814}) 2024-04-26T00:39:54.923 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.922+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=44121380}) 2024-04-26T00:39:54.923 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.922+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.923 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.922+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3162558814}) 2024-04-26T00:39:54.923 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.922+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3270819930}) 2024-04-26T00:39:54.925 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.926+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.925 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.926+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3185850802}) 2024-04-26T00:39:54.925 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.926+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4209482253}) 2024-04-26T00:39:54.925 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.926+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.925 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.926+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3185850802}) 2024-04-26T00:39:54.925 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.926+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1337080191}) 2024-04-26T00:39:54.927 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.926+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.927 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.926+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=969676814}) 2024-04-26T00:39:54.927 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.926+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2984190670}) 2024-04-26T00:39:54.927 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.926+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.927 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.926+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=969676814}) 2024-04-26T00:39:54.927 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.926+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4053794892}) 2024-04-26T00:39:54.929 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.930+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.929 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.930+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3600915572}) 2024-04-26T00:39:54.929 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.930+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1594772382}) 2024-04-26T00:39:54.929 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.930+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.929 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.930+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3600915572}) 2024-04-26T00:39:54.929 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.930+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3619866234}) 2024-04-26T00:39:54.931 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.930+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.931 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.930+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3313223410}) 2024-04-26T00:39:54.931 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.930+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3980066231}) 2024-04-26T00:39:54.931 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.930+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.931 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.930+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3313223410}) 2024-04-26T00:39:54.931 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.930+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1083590676}) 2024-04-26T00:39:54.932 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.930+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.932 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.930+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1589861886}) 2024-04-26T00:39:54.932 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.930+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=226732563}) 2024-04-26T00:39:54.933 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.930+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.933 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.930+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1589861886}) 2024-04-26T00:39:54.933 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.930+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3454880050}) 2024-04-26T00:39:54.934 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.934+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.934 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.934+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1550911306}) 2024-04-26T00:39:54.934 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.934+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2084542851}) 2024-04-26T00:39:54.934 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.934+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.934 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.934+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1550911306}) 2024-04-26T00:39:54.934 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.934+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=737346999}) 2024-04-26T00:39:54.936 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.934+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.936 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.934+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=68,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=9} crc {logm=634243443,mdsmap=2840952586,mgr=1214999089,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=3625468472,mon_config_key=2418587132}) 2024-04-26T00:39:54.936 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.934+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=71,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5} crc {logm=3966604877,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=3625468472,mon_config_key=3159346785}) 2024-04-26T00:39:54.936 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.934+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.936 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.934+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=68,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=9} crc {logm=634243443,mdsmap=2840952586,mgr=1214999089,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=3625468472,mon_config_key=2418587132}) 2024-04-26T00:39:54.936 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.934+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=68,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=9} crc {logm=2602515055,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=369122857,mon_config_key=2418587132}) 2024-04-26T00:39:54.938 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.938+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.938 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.938+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=2,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=89} crc {mon_config_key=2908311886,monmap=4194790699,osd_metadata=3306121363,osd_pg_creating=241526288,osd_snap=1675072307}) 2024-04-26T00:39:54.938 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.938+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=2,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=88} crc {mon_config_key=2908311886,monmap=3876866194,osd_metadata=2845665730,osd_pg_creating=241526288,osd_snap=2716800869}) 2024-04-26T00:39:54.938 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.938+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.938 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.938+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=2,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=89} crc {mon_config_key=2908311886,monmap=4194790699,osd_metadata=3306121363,osd_pg_creating=241526288,osd_snap=1675072307}) 2024-04-26T00:39:54.939 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.938+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {monmap=3,osd_metadata=6,osd_snap=91} crc {monmap=3876866194,osd_metadata=3306121363,osd_snap=3783474005}) 2024-04-26T00:39:54.940 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.938+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.940 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.938+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=54,osdmap=46} crc {osd_snap=3717379803,osdmap=1989830141}) 2024-04-26T00:39:54.940 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.938+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=58,osdmap=42} crc {osd_snap=1455175081,osdmap=927863403}) 2024-04-26T00:39:54.940 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.938+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.941 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.938+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=54,osdmap=46} crc {osd_snap=3717379803,osdmap=1989830141}) 2024-04-26T00:39:54.941 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.938+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=56,osdmap=44} crc {osd_snap=2617772955,osdmap=2761618273}) 2024-04-26T00:39:54.952 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.950+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.952 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.950+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=94} crc {osdmap=1584366797}) 2024-04-26T00:39:54.952 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.950+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=89} crc {osdmap=2302565229}) 2024-04-26T00:39:54.952 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.950+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:39:54.952 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.950+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=94} crc {osdmap=1584366797}) 2024-04-26T00:39:54.952 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:39:54.950+0000 7f73a0e3b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=97} crc {osdmap=3271915010}) 2024-04-26T00:39:55.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.094+0000 7f73e34a7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:39:55.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.094+0000 7f73e34a7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:39:55.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.094+0000 7f73e34a7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:39:55.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.094+0000 7f73e34a7640 1 -- 172.21.15.37:0/3025534825 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f73dc075ee0 con 0x7f73dc136e10 2024-04-26T00:39:55.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.094+0000 7f73cdffb640 1 -- 172.21.15.37:0/3025534825 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f73d8013dc0 con 0x7f73dc136e10 2024-04-26T00:39:55.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.094+0000 7f73cdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:39:55.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.094+0000 7f73cdffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:39:55.096 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":148,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:39:55.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.098+0000 7f73e34a7640 1 -- 172.21.15.37:0/3025534825 >> v2:172.21.15.45:6812/2379918623 conn(0x7f73c0035e80 msgr2=0x7f73c0038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:55.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.098+0000 7f73e34a7640 1 --2- 172.21.15.37:0/3025534825 >> v2:172.21.15.45:6812/2379918623 conn(0x7f73c0035e80 0x7f73c0038340 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f73d00ab000 tx=0x7f73d00a7070 comp rx=0 tx=0).stop 2024-04-26T00:39:55.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.098+0000 7f73e34a7640 10 monclient: shutdown 2024-04-26T00:39:55.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.098+0000 7f73e34a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:55.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.098+0000 7f73e34a7640 1 -- 172.21.15.37:0/3025534825 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f73dc136e10 msgr2=0x7f73dc13b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:55.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.098+0000 7f73e34a7640 1 --2- 172.21.15.37:0/3025534825 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f73dc136e10 0x7f73dc13b100 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f73d800e930 tx=0x7f73d800edf0 comp rx=0 tx=0).stop 2024-04-26T00:39:55.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.098+0000 7f73e34a7640 1 -- 172.21.15.37:0/3025534825 shutdown_connections 2024-04-26T00:39:55.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.098+0000 7f73e34a7640 1 --2- 172.21.15.37:0/3025534825 >> v2:172.21.15.45:6812/2379918623 conn(0x7f73c0035e80 0x7f73c0038340 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:55.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.098+0000 7f73e34a7640 1 --2- 172.21.15.37:0/3025534825 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73dc1371f0 0x7f73dc134fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:55.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.098+0000 7f73e34a7640 1 --2- 172.21.15.37:0/3025534825 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f73dc136e10 0x7f73dc13b100 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:55.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.098+0000 7f73e34a7640 1 --2- 172.21.15.37:0/3025534825 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f73dc135d90 0x7f73dc13d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:55.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.098+0000 7f73e34a7640 1 -- 172.21.15.37:0/3025534825 >> 172.21.15.37:0/3025534825 conn(0x7f73dc0a64e0 msgr2=0x7f73dc0a4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:55.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.098+0000 7f73e34a7640 1 -- 172.21.15.37:0/3025534825 shutdown_connections 2024-04-26T00:39:55.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.098+0000 7f73e34a7640 1 -- 172.21.15.37:0/3025534825 wait complete. 2024-04-26T00:39:55.114 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:39:55.114 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-26T00:39:55.114 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:39:55.114 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:39:55.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1c32b640 10 monclient: get_monmap_and_config 2024-04-26T00:39:55.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1c32b640 10 monclient: build_initial_monmap 2024-04-26T00:39:55.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1c32b640 10 monclient: monmap: 2024-04-26T00:39:55.241 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:55.241 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:55.241 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:39:55.244092+0000 2024-04-26T00:39:55.241 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:39:55.244092+0000 2024-04-26T00:39:55.241 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:55.241 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:55.242 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:39:55.242 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:39:55.242 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:39:55.242 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:39:55.242 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:39:55.242 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:55.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1c32b640 1 Processor -- start 2024-04-26T00:39:55.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1c32b640 1 -- start start 2024-04-26T00:39:55.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1c32b640 10 monclient: init 2024-04-26T00:39:55.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1c32b640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:55.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1c32b640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-04-26T00:39:55.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1c32b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7c140a3d90 0x7f7c1409bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:55.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1c32b640 10 monclient(hunting): picked mon.noname-c con 0x7f7c140a3d90 addr v2:172.21.15.45:3300/0 2024-04-26T00:39:55.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1c32b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7c140a4170 addr v1:172.21.15.45:6789/0 2024-04-26T00:39:55.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1c32b640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f7c1413c3e0 0x7f7c1413e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:55.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1c32b640 10 monclient(hunting): picked mon.noname-a con 0x7f7c1413c3e0 addr v2:172.21.15.37:3300/0 2024-04-26T00:39:55.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1c32b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c140a5150 con 0x7f7c140a4170 2024-04-26T00:39:55.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1c32b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:55.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1c32b640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f7c1409b280 con 0x7f7c1413c3e0 2024-04-26T00:39:55.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1c32b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:55.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1c32b640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f7c1409b560 con 0x7f7c140a3d90 2024-04-26T00:39:55.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1c32b640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:55.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1c32b640 10 monclient(hunting): authenticate will time out at 2176.549919s 2024-04-26T00:39:55.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1a8c8640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f7c1413c3e0 0x7f7c1413e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:55.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1a8c8640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f7c1413c3e0 0x7f7c1413e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:45392/0 (socket says 172.21.15.37:45392) 2024-04-26T00:39:55.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1a8c8640 1 -- 172.21.15.37:0/148558200 learned_addr learned my addr 172.21.15.37:0/148558200 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:55.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1a8c8640 10 monclient(hunting): get_auth_request con 0x7f7c1413c3e0 auth_method 0 2024-04-26T00:39:55.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1a8c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:55.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1a8c8640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:55.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1a8c8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:55.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1a0c7640 1 --2- 172.21.15.37:0/148558200 >> v2:172.21.15.45:3300/0 conn(0x7f7c140a3d90 0x7f7c1409bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:55.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1a8c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:55.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1a8c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:55.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1a8c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:55.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1a0c7640 10 monclient(hunting): get_auth_request con 0x7f7c140a3d90 auth_method 0 2024-04-26T00:39:55.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1a0c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:55.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1a0c7640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:55.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1a0c7640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:55.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1a0c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:55.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1a0c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:55.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1a0c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:55.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1a8c8640 10 monclient(hunting): handle_auth_done global_id 244119 payload 274 2024-04-26T00:39:55.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1a8c8640 1 -- 172.21.15.37:0/148558200 >> v2:172.21.15.45:3300/0 conn(0x7f7c140a3d90 msgr2=0x7f7c1409bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:55.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1a8c8640 1 --2- 172.21.15.37:0/148558200 >> v2:172.21.15.45:3300/0 conn(0x7f7c140a3d90 0x7f7c1409bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:55.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1a8c8640 1 -- 172.21.15.37:0/148558200 >> v1:172.21.15.45:6789/0 conn(0x7f7c140a4170 legacy=0x7f7c140a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:55.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1a8c8640 10 monclient: _finish_hunting 0 2024-04-26T00:39:55.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1a8c8640 1 monclient: found mon.noname-a 2024-04-26T00:39:55.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1a8c8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:55.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1a8c8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:39:55.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.242+0000 7f7c1a8c8640 1 -- 172.21.15.37:0/148558200 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c1406dc60 con 0x7f7c1413c3e0 2024-04-26T00:39:55.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.246+0000 7f7c1a8c8640 1 -- 172.21.15.37:0/148558200 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c1413c3e0 msgr2=0x7f7c1413e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:55.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.246+0000 7f7c1a8c8640 1 --2- 172.21.15.37:0/148558200 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c1413c3e0 0x7f7c1413e7d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7c140a57e0 tx=0x7f7c0c01c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b8816b978f8d232 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:55.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c190c5640 1 -- 172.21.15.37:0/148558200 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c0b0070 con 0x7f7c1413c3e0 2024-04-26T00:39:55.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c190c5640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:55.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c190c5640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:39:55.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c190c5640 10 monclient: dump: 2024-04-26T00:39:55.250 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:55.250 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:55.250 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.250 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.250 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:55.251 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:55.251 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:55.251 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:55.251 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:55.251 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:55.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c190c5640 10 monclient: _finish_auth 0 2024-04-26T00:39:55.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c190c5640 10 monclient: _check_auth_tickets 2024-04-26T00:39:55.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c190c5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:55.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c190c5640 1 -- 172.21.15.37:0/148558200 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c0c0ab600 con 0x7f7c1413c3e0 2024-04-26T00:39:55.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c190c5640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:55.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c190c5640 1 -- 172.21.15.37:0/148558200 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c0ab9d0 con 0x7f7c1413c3e0 2024-04-26T00:39:55.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1c32b640 5 monclient: authenticate success, global_id 244119 2024-04-26T00:39:55.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1c32b640 10 monclient: get_monmap_and_config success 2024-04-26T00:39:55.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1c32b640 4 set_mon_vals no callback set 2024-04-26T00:39:55.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c190c5640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:55.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c190c5640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:39:55.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c190c5640 10 monclient: dump: 2024-04-26T00:39:55.252 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:55.252 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:55.252 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.252 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.252 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:55.252 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:55.252 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:55.252 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:55.252 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:55.252 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:55.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1c32b640 10 monclient: shutdown 2024-04-26T00:39:55.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1c32b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:55.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1c32b640 1 -- 172.21.15.37:0/148558200 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c1413c3e0 msgr2=0x7f7c1413e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:55.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1c32b640 1 --2- 172.21.15.37:0/148558200 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c1413c3e0 0x7f7c1413e7d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7c140a57e0 tx=0x7f7c0c01c3a0 comp rx=0 tx=0).stop 2024-04-26T00:39:55.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1c32b640 1 -- 172.21.15.37:0/148558200 shutdown_connections 2024-04-26T00:39:55.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1c32b640 1 --2- 172.21.15.37:0/148558200 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c1413c3e0 0x7f7c1413e7d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:55.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1c32b640 1 --2- 172.21.15.37:0/148558200 >> v2:172.21.15.45:3300/0 conn(0x7f7c140a3d90 0x7f7c1409bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:55.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1c32b640 1 -- 172.21.15.37:0/148558200 >> 172.21.15.37:0/148558200 conn(0x7f7c140aa0a0 msgr2=0x7f7c140a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:55.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1c32b640 1 -- 172.21.15.37:0/148558200 shutdown_connections 2024-04-26T00:39:55.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1c32b640 1 -- 172.21.15.37:0/148558200 wait complete. 2024-04-26T00:39:55.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1c32b640 10 monclient: build_initial_monmap 2024-04-26T00:39:55.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1c32b640 10 monclient: monmap: 2024-04-26T00:39:55.253 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:55.253 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:39:55.253 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:39:55.253 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:39:55.254 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:55.254 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:39:55.254 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:39:55.254 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:39:55.254 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:39:55.254 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:55.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1c32b640 1 Processor -- start 2024-04-26T00:39:55.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1c32b640 1 -- start start 2024-04-26T00:39:55.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1c32b640 10 monclient: init 2024-04-26T00:39:55.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1c32b640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:55.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1c32b640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:39:55.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1c32b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c140a3d90 0x7f7c141cf190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:55.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1c32b640 10 monclient(hunting): picked mon.noname-b con 0x7f7c140a3d90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:39:55.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1c32b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c140a4170 0x7f7c14137160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:55.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1c32b640 10 monclient(hunting): picked mon.noname-a con 0x7f7c140a4170 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:39:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1c32b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c1413c3e0 0x7f7c141376a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1c32b640 10 monclient(hunting): picked mon.noname-c con 0x7f7c1413c3e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1c32b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1c32b640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f7c141418c0 con 0x7f7c140a4170 2024-04-26T00:39:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1c32b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1c32b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7c14141740 con 0x7f7c140a3d90 2024-04-26T00:39:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1c32b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1c32b640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f7c14141a40 con 0x7f7c1413c3e0 2024-04-26T00:39:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1c32b640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1c32b640 10 monclient(hunting): authenticate will time out at 2176.558238s 2024-04-26T00:39:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1a8c8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c1413c3e0 0x7f7c141376a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1a8c8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c1413c3e0 0x7f7c141376a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:38026/0 (socket says 172.21.15.37:38026) 2024-04-26T00:39:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1a8c8640 1 -- 172.21.15.37:0/1761170378 learned_addr learned my addr 172.21.15.37:0/1761170378 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:55.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1a8c8640 10 monclient(hunting): get_auth_request con 0x7f7c1413c3e0 auth_method 0 2024-04-26T00:39:55.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1a8c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:55.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1a8c8640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:55.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1a8c8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:55.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c198c6640 1 --2- 172.21.15.37:0/1761170378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c140a4170 0x7f7c14137160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:55.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1a8c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:55.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1a8c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:55.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1a8c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:55.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c198c6640 10 monclient(hunting): get_auth_request con 0x7f7c140a4170 auth_method 0 2024-04-26T00:39:55.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c198c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:55.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c198c6640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:55.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c198c6640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:55.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1a0c7640 1 --2- 172.21.15.37:0/1761170378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c140a3d90 0x7f7c141cf190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:55.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1a8c8640 10 monclient(hunting): handle_auth_done global_id 204524 payload 824 2024-04-26T00:39:55.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1a8c8640 1 -- 172.21.15.37:0/1761170378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c140a3d90 msgr2=0x7f7c141cf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:55.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1a8c8640 1 --2- 172.21.15.37:0/1761170378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c140a3d90 0x7f7c141cf190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:55.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1a8c8640 1 -- 172.21.15.37:0/1761170378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c140a4170 msgr2=0x7f7c14137160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:55.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1a8c8640 1 --2- 172.21.15.37:0/1761170378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c140a4170 0x7f7c14137160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:55.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1a8c8640 10 monclient: _finish_hunting 0 2024-04-26T00:39:55.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1a8c8640 1 monclient: found mon.noname-c 2024-04-26T00:39:55.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1a8c8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:55.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1a8c8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:39:55.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c1a8c8640 1 -- 172.21.15.37:0/1761170378 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c14133c10 con 0x7f7c1413c3e0 2024-04-26T00:39:55.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.250+0000 7f7c198c6640 1 --2- 172.21.15.37:0/1761170378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c140a4170 0x7f7c14137160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:39:55.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c1a8c8640 1 -- 172.21.15.37:0/1761170378 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c1413c3e0 msgr2=0x7f7c141376a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:55.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c1a8c8640 1 --2- 172.21.15.37:0/1761170378 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c1413c3e0 0x7f7c141376a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c018fb0 tx=0x7f7c0c019140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:55.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c06ffd640 1 -- 172.21.15.37:0/1761170378 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c0b0070 con 0x7f7c1413c3e0 2024-04-26T00:39:55.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:55.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c06ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:39:55.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c06ffd640 10 monclient: dump: 2024-04-26T00:39:55.262 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:55.262 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:55.262 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.262 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.262 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:55.262 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:55.262 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:55.262 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:55.262 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:55.262 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:55.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c06ffd640 10 monclient: _finish_auth 0 2024-04-26T00:39:55.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c06ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:39:55.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:55.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c1c32b640 5 monclient: authenticate success, global_id 204524 2024-04-26T00:39:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c06ffd640 1 -- 172.21.15.37:0/1761170378 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c0c0b1040 con 0x7f7c1413c3e0 2024-04-26T00:39:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c06ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c06ffd640 1 -- 172.21.15.37:0/1761170378 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c0abe60 con 0x7f7c1413c3e0 2024-04-26T00:39:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c06ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c06ffd640 10 monclient: dump: 2024-04-26T00:39:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:55.263 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:55.264 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c188c4640 4 set_mon_vals no callback set 2024-04-26T00:39:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c1c32b640 10 monclient: _renew_subs 2024-04-26T00:39:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c1c32b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c1c32b640 1 -- 172.21.15.37:0/1761170378 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c1409ada0 con 0x7f7c1413c3e0 2024-04-26T00:39:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c1c32b640 10 monclient: _renew_subs 2024-04-26T00:39:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c1c32b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c1c32b640 1 -- 172.21.15.37:0/1761170378 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c14133930 con 0x7f7c1413c3e0 2024-04-26T00:39:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c1c32b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c1c32b640 1 --2- 172.21.15.37:0/1761170378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c1413a030 0x7f7c1413a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c1c32b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c1c32b640 1 -- 172.21.15.37:0/1761170378 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f7c141322b0 con 0x7f7c1413a030 2024-04-26T00:39:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c1a0c7640 1 --2- 172.21.15.37:0/1761170378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c1413a030 0x7f7c1413a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:55.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c1a0c7640 10 monclient: get_auth_request con 0x7f7c1413a030 auth_method 0 2024-04-26T00:39:55.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c1a0c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:55.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c1a0c7640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:55.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c1a0c7640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:55.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c06ffd640 1 -- 172.21.15.37:0/1761170378 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f7c0c0aaa40 con 0x7f7c1413c3e0 2024-04-26T00:39:55.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c1a0c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:55.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c1a0c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:55.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c1a0c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:55.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c06ffd640 1 --2- 172.21.15.37:0/1761170378 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7bf8035e80 0x7f7bf8038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:55.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c06ffd640 1 -- 172.21.15.37:0/1761170378 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(791..791 src has 705..791) ==== 4003+0+0 (secure 0 0 0) 0x7f7c0c0e28b0 con 0x7f7c1413c3e0 2024-04-26T00:39:55.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c1a0c7640 10 monclient(hunting): handle_auth_done global_id 244125 payload 824 2024-04-26T00:39:55.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c1a0c7640 1 -- 172.21.15.37:0/1761170378 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7c1404f510 con 0x7f7c1413a030 2024-04-26T00:39:55.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c198c6640 1 --2- 172.21.15.37:0/1761170378 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7bf8035e80 0x7f7bf8038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:55.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.254+0000 7f7c198c6640 10 monclient: get_auth_request con 0x7f7bf8035e80 auth_method 0 2024-04-26T00:39:55.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.258+0000 7f7c1a0c7640 1 -- 172.21.15.37:0/1761170378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c1413a030 msgr2=0x7f7c1413a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:55.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.258+0000 7f7c1a0c7640 1 --2- 172.21.15.37:0/1761170378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c1413a030 0x7f7c1413a410 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7c080107c0 tx=0x7f7c08010c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:55.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.258+0000 7f7c06ffd640 1 -- 172.21.15.37:0/1761170378 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0800f040 con 0x7f7c1413a030 2024-04-26T00:39:55.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.258+0000 7f7c06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:55.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.258+0000 7f7c06ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:39:55.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.258+0000 7f7c06ffd640 10 monclient: dump: 2024-04-26T00:39:55.266 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:55.266 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:55.266 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.266 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.266 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:55.266 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:55.266 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:55.266 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:55.266 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:55.266 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:55.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.258+0000 7f7c06ffd640 1 -- 172.21.15.37:0/1761170378 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7c08010e70 con 0x7f7c1413a030 2024-04-26T00:39:55.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.258+0000 7f7c06ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:55.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.258+0000 7f7c06ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:39:55.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.258+0000 7f7c06ffd640 1 -- 172.21.15.37:0/1761170378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c1413a030 msgr2=0x7f7c1413a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:55.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.258+0000 7f7c06ffd640 1 --2- 172.21.15.37:0/1761170378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c1413a030 0x7f7c1413a410 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7c080107c0 tx=0x7f7c08010c80 comp rx=0 tx=0).stop 2024-04-26T00:39:55.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.258+0000 7f7c06ffd640 1 -- 172.21.15.37:0/1761170378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c1413a030 msgr2=0x7f7c1413a410 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:39:55.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.258+0000 7f7c06ffd640 1 --2- 172.21.15.37:0/1761170378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c1413a030 0x7f7c1413a410 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:55.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.258+0000 7f7c198c6640 1 --2- 172.21.15.37:0/1761170378 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7bf8035e80 0x7f7bf8038340 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f7c1413b830 tx=0x7f7c10009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:55.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.262+0000 7f7c1c32b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:39:55.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.262+0000 7f7c1c32b640 1 --2- 172.21.15.37:0/1761170378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c1413a950 0x7f7c1409e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:55.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.262+0000 7f7c1c32b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:55.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.262+0000 7f7c1c32b640 1 -- 172.21.15.37:0/1761170378 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f7c141cf6d0 con 0x7f7c1413a950 2024-04-26T00:39:55.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.262+0000 7f7c1a0c7640 1 --2- 172.21.15.37:0/1761170378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c1413a950 0x7f7c1409e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:55.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.262+0000 7f7c1a0c7640 10 monclient: get_auth_request con 0x7f7c1413a950 auth_method 0 2024-04-26T00:39:55.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.262+0000 7f7c1a0c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:55.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.262+0000 7f7c1a0c7640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:55.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.262+0000 7f7c1a0c7640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.266+0000 7f7c1a0c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.266+0000 7f7c1a0c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.266+0000 7f7c1a0c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.266+0000 7f7c1a0c7640 10 monclient(hunting): handle_auth_done global_id 244128 payload 824 2024-04-26T00:39:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.266+0000 7f7c1a0c7640 1 -- 172.21.15.37:0/1761170378 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7c14075dc0 con 0x7f7c1413a950 2024-04-26T00:39:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.266+0000 7f7c1a0c7640 1 -- 172.21.15.37:0/1761170378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c1413a950 msgr2=0x7f7c1409e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.266+0000 7f7c1a0c7640 1 --2- 172.21.15.37:0/1761170378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c1413a950 0x7f7c1409e180 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7c080107c0 tx=0x7f7c080046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.266+0000 7f7c06ffd640 1 -- 172.21.15.37:0/1761170378 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c08021980 con 0x7f7c1413a950 2024-04-26T00:39:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.266+0000 7f7c06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.266+0000 7f7c06ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:39:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.266+0000 7f7c06ffd640 10 monclient: dump: 2024-04-26T00:39:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:55.270 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:55.271 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.266+0000 7f7c06ffd640 1 -- 172.21.15.37:0/1761170378 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4547 (secure 0 0 0) 0x7f7c08025070 con 0x7f7c1413a950 2024-04-26T00:39:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.266+0000 7f7c06ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:39:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.266+0000 7f7c06ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:39:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.266+0000 7f7c06ffd640 1 -- 172.21.15.37:0/1761170378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c1413a950 msgr2=0x7f7c1409e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.266+0000 7f7c06ffd640 1 --2- 172.21.15.37:0/1761170378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c1413a950 0x7f7c1409e180 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7c080107c0 tx=0x7f7c080046d0 comp rx=0 tx=0).stop 2024-04-26T00:39:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.266+0000 7f7c06ffd640 1 -- 172.21.15.37:0/1761170378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c1413a950 msgr2=0x7f7c1409e180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:39:55.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.266+0000 7f7c06ffd640 1 --2- 172.21.15.37:0/1761170378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c1413a950 0x7f7c1409e180 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7c080107c0 tx=0x7f7c080046d0 comp rx=0 tx=0).stop 2024-04-26T00:39:55.272 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:39:55.272 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:39:55.272 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:39:55.272 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:39:55.272 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 148, 2024-04-26T00:39:55.272 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 12623, 2024-04-26T00:39:55.272 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:39:55.272 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:39:55.272 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:39:55.272 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:39:55.272 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:55.273 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 6, 2024-04-26T00:39:55.273 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:39:55.273 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:39:55.273 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:39:55.273 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:55.273 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:55.273 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:55.273 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:55.273 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:55.273 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:55.273 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:55.273 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:55.273 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:55.274 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:55.274 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:55.274 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:55.274 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:39:55.274 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:39:55.274 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:55.274 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:55.274 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:55.274 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:55.274 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:55.274 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:55.274 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:55.274 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:55.274 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:55.275 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:55.275 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:55.275 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:55.275 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:55.275 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:39:55.275 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:39:55.275 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:39:55.275 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:39:55.275 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:39:55.275 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:39:55.275 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:39:55.276 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:39:55.276 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:39:55.276 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:39:55.276 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:39:55.276 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:39:55.276 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:39:55.276 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:39:55.276 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:39:55.276 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:39:55.276 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:39:55.276 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:55.276 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:55.276 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:55.276 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:55.276 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:55.277 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:55.277 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:55.277 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:55.277 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:55.277 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:55.277 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:55.277 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:55.277 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:39:55.277 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:55.277 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:39:55.277 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:55.277 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:39:55.277 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:39:55.277 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:55.278 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:55.278 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:55.278 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:55.278 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:39:55.278 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:55.278 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:55.278 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:55.278 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:55.278 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:39:55.278 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:55.278 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:55.278 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:55.278 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:55.278 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:39:55.278 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:39:55.279 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:55.279 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:55.279 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:55.279 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:55.279 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:55.279 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:39:55.279 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:39:55.279 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:55.279 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:55.279 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:55.279 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:55.279 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:39:55.279 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:55.279 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:55.279 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:55.280 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:55.280 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:39:55.280 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:55.280 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:55.280 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:55.280 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:55.280 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:39:55.280 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:39:55.280 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:55.280 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:55.280 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:55.280 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:55.280 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:55.280 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:39:55.280 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:39:55.281 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:55.281 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:55.281 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:55.281 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:55.281 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:39:55.281 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:55.281 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:55.281 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:55.281 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:55.281 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:39:55.281 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:55.281 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:55.281 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:55.281 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:55.282 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:39:55.282 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:39:55.282 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:55.282 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:55.282 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:55.282 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:55.282 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:55.282 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:55.282 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:39:55.282 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:39:55.282 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:55.282 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:55.282 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:55.282 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:39:55.282 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:55.283 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:55.283 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:39:55.283 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:55.283 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:55.283 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:55.283 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:39:55.283 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:55.283 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:55.283 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:55.283 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:39:55.283 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:39:55.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.266+0000 7f7c1c32b640 1 -- 172.21.15.37:0/1761170378 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7bf8035e80 msgr2=0x7f7bf8038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:55.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.266+0000 7f7c1c32b640 1 --2- 172.21.15.37:0/1761170378 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7bf8035e80 0x7f7bf8038340 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f7c1413b830 tx=0x7f7c10009040 comp rx=0 tx=0).stop 2024-04-26T00:39:55.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.266+0000 7f7c1c32b640 10 monclient: shutdown 2024-04-26T00:39:55.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.266+0000 7f7c1c32b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:55.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.266+0000 7f7c1c32b640 1 -- 172.21.15.37:0/1761170378 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c1413c3e0 msgr2=0x7f7c141376a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:55.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.266+0000 7f7c1c32b640 1 --2- 172.21.15.37:0/1761170378 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c1413c3e0 0x7f7c141376a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c018fb0 tx=0x7f7c0c019140 comp rx=0 tx=0).stop 2024-04-26T00:39:55.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.266+0000 7f7c1a8c8640 1 -- 172.21.15.37:0/1761170378 reap_dead start 2024-04-26T00:39:55.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.266+0000 7f7c1a8c8640 1 -- 172.21.15.37:0/1761170378 reap_dead start 2024-04-26T00:39:55.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.266+0000 7f7c1c32b640 1 -- 172.21.15.37:0/1761170378 shutdown_connections 2024-04-26T00:39:55.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.266+0000 7f7c1c32b640 1 -- 172.21.15.37:0/1761170378 >> 172.21.15.37:0/1761170378 conn(0x7f7c140aa0a0 msgr2=0x7f7c140a8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:55.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.266+0000 7f7c1c32b640 1 -- 172.21.15.37:0/1761170378 shutdown_connections 2024-04-26T00:39:55.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.266+0000 7f7c1c32b640 1 -- 172.21.15.37:0/1761170378 wait complete. 2024-04-26T00:39:55.286 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:39:55.286 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:39:55.286 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:39:55.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.398+0000 7fc0400ae640 10 monclient: get_monmap_and_config 2024-04-26T00:39:55.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.398+0000 7fc0400ae640 10 monclient: build_initial_monmap 2024-04-26T00:39:55.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.398+0000 7fc0400ae640 10 monclient: monmap: 2024-04-26T00:39:55.401 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:55.401 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:55.401 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:39:55.403543+0000 2024-04-26T00:39:55.401 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:39:55.403543+0000 2024-04-26T00:39:55.401 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:55.401 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:55.401 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:39:55.401 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:39:55.401 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:39:55.401 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:39:55.401 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:39:55.401 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:55.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.402+0000 7fc0400ae640 1 Processor -- start 2024-04-26T00:39:55.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.402+0000 7fc0400ae640 1 -- start start 2024-04-26T00:39:55.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.402+0000 7fc0400ae640 10 monclient: init 2024-04-26T00:39:55.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.402+0000 7fc0400ae640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:55.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.402+0000 7fc0400ae640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-04-26T00:39:55.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.402+0000 7fc0400ae640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc038130eb0 addr v1:172.21.15.45:6789/0 2024-04-26T00:39:55.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.402+0000 7fc0400ae640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc03809ef20 addr v1:172.21.15.37:6789/0 2024-04-26T00:39:55.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.402+0000 7fc0400ae640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc03813a5d0 0x7fc03813c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:55.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.402+0000 7fc0400ae640 10 monclient(hunting): picked mon.noname-a con 0x7fc03813a5d0 addr v2:172.21.15.37:3300/0 2024-04-26T00:39:55.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.402+0000 7fc0400ae640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc03809c190 con 0x7fc03809ef20 2024-04-26T00:39:55.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.402+0000 7fc0400ae640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0380a11c0 con 0x7fc038130eb0 2024-04-26T00:39:55.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.402+0000 7fc0400ae640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:55.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.402+0000 7fc0400ae640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fc03809df60 con 0x7fc03813a5d0 2024-04-26T00:39:55.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.402+0000 7fc0400ae640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:55.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.402+0000 7fc0400ae640 10 monclient(hunting): authenticate will time out at 2176.709230s 2024-04-26T00:39:55.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.402+0000 7fc03e64b640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc03813a5d0 0x7fc03813c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:55.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.402+0000 7fc03e64b640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc03813a5d0 0x7fc03813c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:45426/0 (socket says 172.21.15.37:45426) 2024-04-26T00:39:55.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.402+0000 7fc03d649640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fc03809ef20 0x7fc038138eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:43290/0 (socket says 172.21.15.37:43290) 2024-04-26T00:39:55.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.402+0000 7fc03e64b640 1 -- 172.21.15.37:0/2789747860 learned_addr learned my addr 172.21.15.37:0/2789747860 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:55.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.402+0000 7fc03e64b640 10 monclient(hunting): get_auth_request con 0x7fc03813a5d0 auth_method 0 2024-04-26T00:39:55.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.402+0000 7fc03e64b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:55.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.402+0000 7fc03e64b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:55.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.402+0000 7fc03e64b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:55.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.402+0000 7fc03e64b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:55.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.402+0000 7fc03e64b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:55.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.402+0000 7fc03e64b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:55.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.402+0000 7fc03e64b640 10 monclient(hunting): handle_auth_done global_id 244131 payload 274 2024-04-26T00:39:55.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.402+0000 7fc03e64b640 1 -- 172.21.15.37:0/2789747860 >> v1:172.21.15.45:6789/0 conn(0x7fc038130eb0 legacy=0x7fc038131290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:55.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.402+0000 7fc03e64b640 1 -- 172.21.15.37:0/2789747860 >> v1:172.21.15.37:6789/0 conn(0x7fc03809ef20 legacy=0x7fc038138eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:55.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.402+0000 7fc03e64b640 10 monclient: _finish_hunting 0 2024-04-26T00:39:55.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.402+0000 7fc03e64b640 1 monclient: found mon.noname-a 2024-04-26T00:39:55.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.402+0000 7fc03e64b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:55.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.402+0000 7fc03e64b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:39:55.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.402+0000 7fc03e64b640 1 -- 172.21.15.37:0/2789747860 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc03809dc80 con 0x7fc03813a5d0 2024-04-26T00:39:55.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.406+0000 7fc03e64b640 1 -- 172.21.15.37:0/2789747860 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc03813a5d0 msgr2=0x7fc03813c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:55.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.406+0000 7fc03e64b640 1 --2- 172.21.15.37:0/2789747860 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc03813a5d0 0x7fc03813c9c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc030009e50 tx=0x7fc03001c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee90bd0d8698553f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:55.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.406+0000 7fc03ce48640 1 -- 172.21.15.37:0/2789747860 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0300a5720 con 0x7fc03813a5d0 2024-04-26T00:39:55.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.406+0000 7fc03ce48640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:55.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.406+0000 7fc03ce48640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:39:55.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.406+0000 7fc03ce48640 10 monclient: dump: 2024-04-26T00:39:55.408 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:55.408 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:55.408 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.408 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.408 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:55.409 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:55.409 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:55.409 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:55.409 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:55.409 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:55.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.406+0000 7fc03ce48640 10 monclient: _finish_auth 0 2024-04-26T00:39:55.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.406+0000 7fc03ce48640 10 monclient: _check_auth_tickets 2024-04-26T00:39:55.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.406+0000 7fc03ce48640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:55.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.406+0000 7fc03ce48640 1 -- 172.21.15.37:0/2789747860 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0300a58c0 con 0x7fc03813a5d0 2024-04-26T00:39:55.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.406+0000 7fc03ce48640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:55.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.406+0000 7fc03ce48640 1 -- 172.21.15.37:0/2789747860 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0300a5c90 con 0x7fc03813a5d0 2024-04-26T00:39:55.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.406+0000 7fc0400ae640 5 monclient: authenticate success, global_id 244131 2024-04-26T00:39:55.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.406+0000 7fc0400ae640 10 monclient: get_monmap_and_config success 2024-04-26T00:39:55.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.406+0000 7fc0400ae640 4 set_mon_vals no callback set 2024-04-26T00:39:55.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.406+0000 7fc03ce48640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:55.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.406+0000 7fc03ce48640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:39:55.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.406+0000 7fc03ce48640 10 monclient: dump: 2024-04-26T00:39:55.410 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:55.410 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:55.411 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.411 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.411 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:55.411 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:55.411 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:55.411 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:55.411 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:55.411 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:55.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.406+0000 7fc0400ae640 10 monclient: shutdown 2024-04-26T00:39:55.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.406+0000 7fc0400ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:55.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.406+0000 7fc0400ae640 1 -- 172.21.15.37:0/2789747860 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc03813a5d0 msgr2=0x7fc03813c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:55.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.406+0000 7fc0400ae640 1 --2- 172.21.15.37:0/2789747860 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc03813a5d0 0x7fc03813c9c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc030009e50 tx=0x7fc03001c840 comp rx=0 tx=0).stop 2024-04-26T00:39:55.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.406+0000 7fc0400ae640 1 -- 172.21.15.37:0/2789747860 shutdown_connections 2024-04-26T00:39:55.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.406+0000 7fc0400ae640 1 --2- 172.21.15.37:0/2789747860 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc03813a5d0 0x7fc03813c9c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:55.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.406+0000 7fc0400ae640 1 -- 172.21.15.37:0/2789747860 >> 172.21.15.37:0/2789747860 conn(0x7fc0380a1d50 msgr2=0x7fc0380a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:55.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.406+0000 7fc0400ae640 1 -- 172.21.15.37:0/2789747860 shutdown_connections 2024-04-26T00:39:55.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.406+0000 7fc0400ae640 1 -- 172.21.15.37:0/2789747860 wait complete. 2024-04-26T00:39:55.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.406+0000 7fc0400ae640 10 monclient: build_initial_monmap 2024-04-26T00:39:55.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.406+0000 7fc0400ae640 10 monclient: monmap: 2024-04-26T00:39:55.418 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:55.418 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:39:55.418 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:39:55.418 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:39:55.419 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:55.419 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:39:55.419 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:39:55.419 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:39:55.419 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:39:55.419 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:55.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.406+0000 7fc0400ae640 1 Processor -- start 2024-04-26T00:39:55.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.406+0000 7fc0400ae640 1 -- start start 2024-04-26T00:39:55.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.406+0000 7fc0400ae640 10 monclient: init 2024-04-26T00:39:55.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc0400ae640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:55.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc0400ae640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:39:55.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc0400ae640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc03809ef20 0x7fc0380724b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:55.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc0400ae640 10 monclient(hunting): picked mon.noname-b con 0x7fc03809ef20 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:39:55.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc0400ae640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc038130eb0 0x7fc038131a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:55.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc0400ae640 10 monclient(hunting): picked mon.noname-a con 0x7fc038130eb0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:39:55.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc0400ae640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc03813a5d0 0x7fc038131f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:55.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc0400ae640 10 monclient(hunting): picked mon.noname-c con 0x7fc03813a5d0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:55.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc0400ae640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:55.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc0400ae640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fc03809fdd0 con 0x7fc038130eb0 2024-04-26T00:39:55.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc0400ae640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:55.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc0400ae640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc03809fc50 con 0x7fc03809ef20 2024-04-26T00:39:55.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc0400ae640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:55.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc0400ae640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc03809ff50 con 0x7fc03813a5d0 2024-04-26T00:39:55.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc0400ae640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:55.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc0400ae640 10 monclient(hunting): authenticate will time out at 2176.716297s 2024-04-26T00:39:55.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc03d649640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc038130eb0 0x7fc038131a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:55.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc03e64b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc03813a5d0 0x7fc038131f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:55.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc03d649640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc038130eb0 0x7fc038131a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:45438/0 (socket says 172.21.15.37:45438) 2024-04-26T00:39:55.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc03e64b640 10 monclient(hunting): get_auth_request con 0x7fc03813a5d0 auth_method 0 2024-04-26T00:39:55.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc03d649640 1 -- 172.21.15.37:0/2334301429 learned_addr learned my addr 172.21.15.37:0/2334301429 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:55.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc03e64b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:55.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc03e64b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:55.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc03e64b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:55.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc03d649640 10 monclient(hunting): get_auth_request con 0x7fc038130eb0 auth_method 0 2024-04-26T00:39:55.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc03d649640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:55.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc03d649640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:55.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc03d649640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:55.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc03e64b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:55.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc03e64b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:55.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc03e64b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:55.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc03d649640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:55.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc03d649640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:55.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc03d649640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:55.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc03de4a640 1 --2- 172.21.15.37:0/2334301429 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc03809ef20 0x7fc0380724b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:55.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc03de4a640 10 monclient(hunting): get_auth_request con 0x7fc03809ef20 auth_method 0 2024-04-26T00:39:55.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc03de4a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:55.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc03de4a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:55.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc03de4a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:55.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc03e64b640 10 monclient(hunting): handle_auth_done global_id 204527 payload 824 2024-04-26T00:39:55.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc03e64b640 1 -- 172.21.15.37:0/2334301429 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc03809ef20 msgr2=0x7fc0380724b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:55.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc03e64b640 1 --2- 172.21.15.37:0/2334301429 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc03809ef20 0x7fc0380724b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:55.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc03e64b640 1 -- 172.21.15.37:0/2334301429 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc038130eb0 msgr2=0x7fc038131a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:55.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc03e64b640 1 --2- 172.21.15.37:0/2334301429 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc038130eb0 0x7fc038131a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:55.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc03e64b640 10 monclient: _finish_hunting 0 2024-04-26T00:39:55.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc03e64b640 1 monclient: found mon.noname-c 2024-04-26T00:39:55.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc03e64b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:55.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc03e64b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:39:55.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc03e64b640 1 -- 172.21.15.37:0/2334301429 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0380a1470 con 0x7fc03813a5d0 2024-04-26T00:39:55.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc03d649640 1 --2- 172.21.15.37:0/2334301429 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc038130eb0 0x7fc038131a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:39:55.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc03e64b640 1 -- 172.21.15.37:0/2334301429 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc03813a5d0 msgr2=0x7fc038131f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:55.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc03e64b640 1 --2- 172.21.15.37:0/2334301429 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc03813a5d0 0x7fc038131f40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc0300ab000 tx=0x7fc0300195e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:55.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc02e7fc640 1 -- 172.21.15.37:0/2334301429 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0300bc070 con 0x7fc03813a5d0 2024-04-26T00:39:55.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc02e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:55.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc02e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:39:55.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc02e7fc640 10 monclient: dump: 2024-04-26T00:39:55.423 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:55.423 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:55.423 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.423 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.423 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:55.423 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:55.424 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:55.424 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:55.424 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:55.424 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:55.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc02e7fc640 10 monclient: _finish_auth 0 2024-04-26T00:39:55.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc02e7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:39:55.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc02e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:55.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc02e7fc640 1 -- 172.21.15.37:0/2334301429 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0300b8880 con 0x7fc03813a5d0 2024-04-26T00:39:55.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc0400ae640 5 monclient: authenticate success, global_id 204527 2024-04-26T00:39:55.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc0400ae640 10 monclient: _renew_subs 2024-04-26T00:39:55.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc0400ae640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:55.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc0400ae640 1 -- 172.21.15.37:0/2334301429 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc03812f0d0 con 0x7fc03813a5d0 2024-04-26T00:39:55.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc02e7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:55.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc02e7fc640 1 -- 172.21.15.37:0/2334301429 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0300b8c50 con 0x7fc03813a5d0 2024-04-26T00:39:55.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc02e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:55.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc02e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:55.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc02e7fc640 10 monclient: dump: 2024-04-26T00:39:55.425 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:55.425 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:55.425 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.425 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.425 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:55.425 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:55.425 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:55.425 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:55.425 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:55.425 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:55.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc02ffff640 4 set_mon_vals no callback set 2024-04-26T00:39:55.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc0400ae640 10 monclient: _renew_subs 2024-04-26T00:39:55.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc0400ae640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:39:55.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc0400ae640 1 -- 172.21.15.37:0/2334301429 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc038132d60 con 0x7fc03813a5d0 2024-04-26T00:39:55.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc0400ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:55.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.410+0000 7fc0400ae640 1 --2- 172.21.15.37:0/2334301429 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc038132480 0x7fc038134d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:55.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.414+0000 7fc0400ae640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:55.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.414+0000 7fc0400ae640 1 -- 172.21.15.37:0/2334301429 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc038132860 con 0x7fc038132480 2024-04-26T00:39:55.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.414+0000 7fc03de4a640 1 --2- 172.21.15.37:0/2334301429 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc038132480 0x7fc038134d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:55.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.414+0000 7fc03de4a640 10 monclient: get_auth_request con 0x7fc038132480 auth_method 0 2024-04-26T00:39:55.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.414+0000 7fc03de4a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:55.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.414+0000 7fc03de4a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:55.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.414+0000 7fc03de4a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:55.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.414+0000 7fc02e7fc640 1 -- 172.21.15.37:0/2334301429 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc0300c03f0 con 0x7fc03813a5d0 2024-04-26T00:39:55.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.414+0000 7fc03de4a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:55.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.414+0000 7fc03de4a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:55.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.414+0000 7fc03de4a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:55.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.414+0000 7fc02e7fc640 1 --2- 172.21.15.37:0/2334301429 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc01c035e80 0x7fc01c038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:55.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.414+0000 7fc03de4a640 10 monclient(hunting): handle_auth_done global_id 204530 payload 824 2024-04-26T00:39:55.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.414+0000 7fc02e7fc640 1 -- 172.21.15.37:0/2334301429 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(791..791 src has 705..791) ==== 4003+0+0 (secure 0 0 0) 0x7fc0300e3540 con 0x7fc03813a5d0 2024-04-26T00:39:55.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.414+0000 7fc03de4a640 1 -- 172.21.15.37:0/2334301429 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc03804f510 con 0x7fc038132480 2024-04-26T00:39:55.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.414+0000 7fc03d649640 1 --2- 172.21.15.37:0/2334301429 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc01c035e80 0x7fc01c038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:55.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.414+0000 7fc03d649640 10 monclient: get_auth_request con 0x7fc01c035e80 auth_method 0 2024-04-26T00:39:55.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.414+0000 7fc03de4a640 1 -- 172.21.15.37:0/2334301429 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc038132480 msgr2=0x7fc038134d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:55.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.414+0000 7fc03de4a640 1 --2- 172.21.15.37:0/2334301429 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc038132480 0x7fc038134d40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc02800e450 tx=0x7fc02800e910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:55.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.414+0000 7fc02e7fc640 1 -- 172.21.15.37:0/2334301429 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc028007450 con 0x7fc038132480 2024-04-26T00:39:55.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.414+0000 7fc02e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:55.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.414+0000 7fc02e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:55.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.414+0000 7fc02e7fc640 10 monclient: dump: 2024-04-26T00:39:55.427 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:55.427 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:55.427 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.428 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.428 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:55.428 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:55.428 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:55.428 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:55.428 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:55.428 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:55.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.414+0000 7fc02e7fc640 1 -- 172.21.15.37:0/2334301429 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc028007450 con 0x7fc038132480 2024-04-26T00:39:55.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.414+0000 7fc02e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:55.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.414+0000 7fc02e7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:39:55.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.414+0000 7fc02e7fc640 1 -- 172.21.15.37:0/2334301429 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc038132480 msgr2=0x7fc038134d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:55.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.414+0000 7fc02e7fc640 1 --2- 172.21.15.37:0/2334301429 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc038132480 0x7fc038134d40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc02800e450 tx=0x7fc02800e910 comp rx=0 tx=0).stop 2024-04-26T00:39:55.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.414+0000 7fc02e7fc640 1 -- 172.21.15.37:0/2334301429 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc038132480 msgr2=0x7fc038134d40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:39:55.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.414+0000 7fc02e7fc640 1 --2- 172.21.15.37:0/2334301429 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc038132480 0x7fc038134d40 secure :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc02800e450 tx=0x7fc02800e910 comp rx=0 tx=0).stop 2024-04-26T00:39:55.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.414+0000 7fc03d649640 1 --2- 172.21.15.37:0/2334301429 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc01c035e80 0x7fc01c038340 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fc038072ca0 tx=0x7fc034007f30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:55.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.422+0000 7fc0400ae640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:39:55.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.422+0000 7fc0400ae640 1 --2- 172.21.15.37:0/2334301429 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc03806dc40 0x7fc03806e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:55.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.422+0000 7fc0400ae640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:55.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.422+0000 7fc0400ae640 1 -- 172.21.15.37:0/2334301429 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc038134820 con 0x7fc03806dc40 2024-04-26T00:39:55.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.422+0000 7fc03de4a640 1 --2- 172.21.15.37:0/2334301429 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc03806dc40 0x7fc03806e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:55.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.422+0000 7fc03de4a640 10 monclient: get_auth_request con 0x7fc03806dc40 auth_method 0 2024-04-26T00:39:55.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.422+0000 7fc03de4a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:55.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.422+0000 7fc03de4a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:55.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.422+0000 7fc03de4a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:55.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.422+0000 7fc03de4a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:55.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.422+0000 7fc03de4a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:55.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.422+0000 7fc03de4a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:55.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.422+0000 7fc03de4a640 10 monclient(hunting): handle_auth_done global_id 204533 payload 824 2024-04-26T00:39:55.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.422+0000 7fc03de4a640 1 -- 172.21.15.37:0/2334301429 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc038068e90 con 0x7fc03806dc40 2024-04-26T00:39:55.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.426+0000 7fc03de4a640 1 -- 172.21.15.37:0/2334301429 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc03806dc40 msgr2=0x7fc03806e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:55.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.426+0000 7fc03de4a640 1 --2- 172.21.15.37:0/2334301429 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc03806dc40 0x7fc03806e020 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc02800e450 tx=0x7fc028003770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:55.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.426+0000 7fc02e7fc640 1 -- 172.21.15.37:0/2334301429 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc02801eb70 con 0x7fc03806dc40 2024-04-26T00:39:55.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.426+0000 7fc02e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:55.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.426+0000 7fc02e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:39:55.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.426+0000 7fc02e7fc640 10 monclient: dump: 2024-04-26T00:39:55.431 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:55.432 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:55.432 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.432 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.432 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:55.432 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:55.432 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:55.432 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:55.432 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:55.432 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:55.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.426+0000 7fc02e7fc640 1 -- 172.21.15.37:0/2334301429 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7fc02801eb70 con 0x7fc03806dc40 2024-04-26T00:39:55.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.426+0000 7fc02e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:39:55.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.426+0000 7fc02e7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:39:55.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.426+0000 7fc02e7fc640 1 -- 172.21.15.37:0/2334301429 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc03806dc40 msgr2=0x7fc03806e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:55.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.426+0000 7fc02e7fc640 1 --2- 172.21.15.37:0/2334301429 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc03806dc40 0x7fc03806e020 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc02800e450 tx=0x7fc028003770 comp rx=0 tx=0).stop 2024-04-26T00:39:55.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.426+0000 7fc02e7fc640 1 -- 172.21.15.37:0/2334301429 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc03806dc40 msgr2=0x7fc03806e020 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:39:55.433 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.426+0000 7fc02e7fc640 1 --2- 172.21.15.37:0/2334301429 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc03806dc40 0x7fc03806e020 secure :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc02800e450 tx=0x7fc028003770 comp rx=0 tx=0).stop 2024-04-26T00:39:55.433 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:39:55.433 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:39:55.433 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:39:55.433 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:39:55.433 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 148, 2024-04-26T00:39:55.433 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 201033, 2024-04-26T00:39:55.433 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:39:55.433 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:39:55.434 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:39:55.434 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:39:55.434 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:55.434 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 6, 2024-04-26T00:39:55.434 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:39:55.434 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:39:55.434 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:39:55.434 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:55.434 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:55.435 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:55.435 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:55.435 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:55.435 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:55.435 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:55.435 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:55.435 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:55.435 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:55.435 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:55.435 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:55.436 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:39:55.436 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:39:55.436 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:55.436 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:55.436 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:55.436 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:55.436 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:55.436 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:55.436 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:55.436 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:55.436 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:55.436 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:55.436 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:55.436 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:55.437 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:55.437 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:39:55.437 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:39:55.437 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:39:55.437 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:39:55.437 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:39:55.437 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:39:55.437 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:39:55.437 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:39:55.437 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:39:55.437 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:39:55.437 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:39:55.437 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:39:55.438 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:39:55.438 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:39:55.438 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:39:55.438 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:39:55.438 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:39:55.438 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:55.438 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:55.438 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:55.438 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:55.439 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:55.439 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:55.439 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:55.439 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:55.439 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:55.439 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:55.439 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:55.439 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:55.439 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:39:55.439 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:55.439 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:39:55.439 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:55.439 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:39:55.440 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:39:55.440 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:55.440 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:55.440 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:55.440 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:55.440 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:39:55.440 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:55.440 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:55.440 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:55.440 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:55.440 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:39:55.440 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:55.440 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:55.440 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:55.440 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:55.441 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:39:55.441 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:39:55.441 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:55.441 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:55.441 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:55.441 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:55.441 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:55.441 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:39:55.441 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:39:55.441 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:55.441 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:55.441 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:55.441 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:55.441 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:39:55.441 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:55.442 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:55.442 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:55.442 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:55.442 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:39:55.442 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:55.442 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:55.442 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:55.442 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:55.442 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:39:55.442 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:39:55.442 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:55.442 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:55.442 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:55.443 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:55.443 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:55.443 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:39:55.443 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:39:55.443 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:55.443 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:55.443 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:55.443 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:55.443 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:39:55.443 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:55.443 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:55.444 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:55.444 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:55.444 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:39:55.444 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:55.444 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:55.444 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:55.444 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:55.444 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:39:55.444 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:39:55.444 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:55.444 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:55.444 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:55.444 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:55.445 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:55.445 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:55.445 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:39:55.445 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:39:55.445 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:55.445 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:55.445 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:55.445 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:39:55.445 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:55.445 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:55.445 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:39:55.445 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:55.446 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:55.446 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:55.446 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:39:55.446 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:55.446 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:55.446 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:39:55.446 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:55.446 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:55.446 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:55.446 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:39:55.446 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:55.446 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:55.446 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:55.447 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:39:55.447 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:39:55.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.426+0000 7fc0400ae640 1 -- 172.21.15.37:0/2334301429 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc01c035e80 msgr2=0x7fc01c038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:55.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.426+0000 7fc0400ae640 1 --2- 172.21.15.37:0/2334301429 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc01c035e80 0x7fc01c038340 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fc038072ca0 tx=0x7fc034007f30 comp rx=0 tx=0).stop 2024-04-26T00:39:55.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.426+0000 7fc0400ae640 10 monclient: shutdown 2024-04-26T00:39:55.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.426+0000 7fc0400ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:55.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.426+0000 7fc0400ae640 1 -- 172.21.15.37:0/2334301429 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc03813a5d0 msgr2=0x7fc038131f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:55.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.426+0000 7fc0400ae640 1 --2- 172.21.15.37:0/2334301429 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc03813a5d0 0x7fc038131f40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc0300ab000 tx=0x7fc0300195e0 comp rx=0 tx=0).stop 2024-04-26T00:39:55.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.426+0000 7fc03e64b640 1 -- 172.21.15.37:0/2334301429 reap_dead start 2024-04-26T00:39:55.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.426+0000 7fc0400ae640 1 -- 172.21.15.37:0/2334301429 shutdown_connections 2024-04-26T00:39:55.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.426+0000 7fc0400ae640 1 --2- 172.21.15.37:0/2334301429 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc03813a5d0 0x7fc038131f40 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:55.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.426+0000 7fc0400ae640 1 -- 172.21.15.37:0/2334301429 >> 172.21.15.37:0/2334301429 conn(0x7fc0380a1d50 msgr2=0x7fc0380a3680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:55.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.426+0000 7fc0400ae640 1 -- 172.21.15.37:0/2334301429 shutdown_connections 2024-04-26T00:39:55.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.426+0000 7fc0400ae640 1 -- 172.21.15.37:0/2334301429 wait complete. 2024-04-26T00:39:55.449 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:39:55.449 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:39:55.449 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:39:55.495 INFO:tasks.rados.rados.0.smithi045.stdout:3495: done (0 left) 2024-04-26T00:39:55.495 INFO:tasks.rados.rados.0.smithi045.stdout:3496: read oid 33 snap -1 2024-04-26T00:39:55.495 INFO:tasks.rados.rados.0.smithi045.stdout:3496: expect deleted 2024-04-26T00:39:55.496 INFO:tasks.rados.rados.0.smithi045.stdout:3497: read oid 50 snap -1 2024-04-26T00:39:55.496 INFO:tasks.rados.rados.0.smithi045.stdout:3497: expect (ObjNum 1207 snap 373 seq_num 1207) 2024-04-26T00:39:55.496 INFO:tasks.rados.rados.0.smithi045.stdout:3498: snap_create 2024-04-26T00:39:55.506 INFO:tasks.rados.rados.0.smithi045.stdout:3497: expect (ObjNum 1207 snap 373 seq_num 1207) 2024-04-26T00:39:55.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.570+0000 7f1911225640 10 monclient: get_monmap_and_config 2024-04-26T00:39:55.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.570+0000 7f1911225640 10 monclient: build_initial_monmap 2024-04-26T00:39:55.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.570+0000 7f1911225640 10 monclient: monmap: 2024-04-26T00:39:55.574 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:55.574 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:55.574 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:39:55.575576+0000 2024-04-26T00:39:55.574 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:39:55.575576+0000 2024-04-26T00:39:55.575 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:55.575 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:55.575 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:39:55.575 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:39:55.575 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:39:55.575 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:39:55.575 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:39:55.576 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:55.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.574+0000 7f1911225640 1 Processor -- start 2024-04-26T00:39:55.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.574+0000 7f1911225640 1 -- start start 2024-04-26T00:39:55.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.574+0000 7f1911225640 10 monclient: init 2024-04-26T00:39:55.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.574+0000 7f1911225640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:55.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.574+0000 7f1911225640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-04-26T00:39:55.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.574+0000 7f1911225640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f190c0a3d90 addr v1:172.21.15.37:6789/0 2024-04-26T00:39:55.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.574+0000 7f1911225640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f190c0a4170 0x7f190c0a45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:55.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.574+0000 7f1911225640 10 monclient(hunting): picked mon.noname-c con 0x7f190c0a4170 addr v2:172.21.15.45:3300/0 2024-04-26T00:39:55.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.574+0000 7f1911225640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f190c13c3e0 addr v1:172.21.15.45:6789/0 2024-04-26T00:39:55.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.574+0000 7f1911225640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f190c0a5150 con 0x7f190c0a3d90 2024-04-26T00:39:55.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.574+0000 7f1911225640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f190c09b280 con 0x7f190c13c3e0 2024-04-26T00:39:55.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.574+0000 7f1911225640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:55.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.574+0000 7f1911225640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f190c09c650 con 0x7f190c0a4170 2024-04-26T00:39:55.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.574+0000 7f1911225640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:55.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.574+0000 7f1911225640 10 monclient(hunting): authenticate will time out at 2176.881216s 2024-04-26T00:39:55.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.574+0000 7f190ad9d640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f190c0a3d90 0x7f190c09bf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:43294/0 (socket says 172.21.15.37:43294) 2024-04-26T00:39:55.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.574+0000 7f190ad9d640 1 -- 172.21.15.37:0/1967058981 learned_addr learned my addr 172.21.15.37:0/1967058981 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:39:55.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.574+0000 7f190a59c640 1 --2- 172.21.15.37:0/1967058981 >> v2:172.21.15.45:3300/0 conn(0x7f190c0a4170 0x7f190c0a45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:55.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.574+0000 7f190a59c640 10 monclient(hunting): get_auth_request con 0x7f190c0a4170 auth_method 0 2024-04-26T00:39:55.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.574+0000 7f190a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:55.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.574+0000 7f190a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:55.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.574+0000 7f190a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:55.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.574+0000 7f190a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:55.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.574+0000 7f190a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:55.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.574+0000 7f190a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:55.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.574+0000 7f190a59c640 10 monclient(hunting): handle_auth_done global_id 224293 payload 274 2024-04-26T00:39:55.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.574+0000 7f190a59c640 1 -- 172.21.15.37:0/1967058981 >> v1:172.21.15.45:6789/0 conn(0x7f190c13c3e0 legacy=0x7f190c13e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:55.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.574+0000 7f190a59c640 1 -- 172.21.15.37:0/1967058981 >> v1:172.21.15.37:6789/0 conn(0x7f190c0a3d90 legacy=0x7f190c09bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:55.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.574+0000 7f190a59c640 10 monclient: _finish_hunting 0 2024-04-26T00:39:55.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.574+0000 7f190a59c640 1 monclient: found mon.noname-c 2024-04-26T00:39:55.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.574+0000 7f190a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:55.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.574+0000 7f190a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:39:55.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.574+0000 7f190a59c640 1 -- 172.21.15.37:0/1967058981 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f190c09b560 con 0x7f190c0a4170 2024-04-26T00:39:55.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.578+0000 7f190a59c640 1 -- 172.21.15.37:0/1967058981 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f190c0a4170 msgr2=0x7f190c0a45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:55.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.578+0000 7f190a59c640 1 --2- 172.21.15.37:0/1967058981 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f190c0a4170 0x7f190c0a45e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f19000097c0 tx=0x7f190001c2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=269cf36979ee03e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:55.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1909d9b640 1 -- 172.21.15.37:0/1967058981 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19000b0070 con 0x7f190c0a4170 2024-04-26T00:39:55.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1909d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:55.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1909d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:39:55.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1909d9b640 10 monclient: dump: 2024-04-26T00:39:55.601 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:55.602 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:55.602 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.602 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.602 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:55.602 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:55.602 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:55.603 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:55.603 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:55.603 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:55.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1909d9b640 10 monclient: _finish_auth 0 2024-04-26T00:39:55.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1909d9b640 10 monclient: _check_auth_tickets 2024-04-26T00:39:55.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1909d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:55.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1909d9b640 1 -- 172.21.15.37:0/1967058981 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19000ab600 con 0x7f190c0a4170 2024-04-26T00:39:55.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1909d9b640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:55.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1909d9b640 1 -- 172.21.15.37:0/1967058981 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19000ab9d0 con 0x7f190c0a4170 2024-04-26T00:39:55.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1909d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:55.604 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1909d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:39:55.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1909d9b640 10 monclient: dump: 2024-04-26T00:39:55.605 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:55.605 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:55.605 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.605 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.605 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:55.606 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:55.606 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:55.606 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:55.606 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:55.606 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:55.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1911225640 5 monclient: authenticate success, global_id 224293 2024-04-26T00:39:55.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1911225640 10 monclient: get_monmap_and_config success 2024-04-26T00:39:55.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1911225640 4 set_mon_vals no callback set 2024-04-26T00:39:55.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1911225640 10 monclient: shutdown 2024-04-26T00:39:55.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1911225640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:55.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1911225640 1 -- 172.21.15.37:0/1967058981 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f190c0a4170 msgr2=0x7f190c0a45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:55.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1911225640 1 --2- 172.21.15.37:0/1967058981 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f190c0a4170 0x7f190c0a45e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f19000097c0 tx=0x7f190001c2e0 comp rx=0 tx=0).stop 2024-04-26T00:39:55.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1911225640 1 -- 172.21.15.37:0/1967058981 shutdown_connections 2024-04-26T00:39:55.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1911225640 1 --2- 172.21.15.37:0/1967058981 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f190c0a4170 0x7f190c0a45e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:55.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1911225640 1 -- 172.21.15.37:0/1967058981 >> 172.21.15.37:0/1967058981 conn(0x7f190c0aa0a0 msgr2=0x7f190c0a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:55.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1911225640 1 -- 172.21.15.37:0/1967058981 shutdown_connections 2024-04-26T00:39:55.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1911225640 1 -- 172.21.15.37:0/1967058981 wait complete. 2024-04-26T00:39:55.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1911225640 10 monclient: build_initial_monmap 2024-04-26T00:39:55.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1911225640 10 monclient: monmap: 2024-04-26T00:39:55.609 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:55.609 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:39:55.609 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:39:55.610 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:39:55.610 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:55.610 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:39:55.610 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:39:55.610 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:39:55.610 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:39:55.611 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:55.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1911225640 1 Processor -- start 2024-04-26T00:39:55.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1911225640 1 -- start start 2024-04-26T00:39:55.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1911225640 10 monclient: init 2024-04-26T00:39:55.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1911225640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:55.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1911225640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:39:55.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1911225640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f190c0a3d90 0x7f190c1cee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:55.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1911225640 10 monclient(hunting): picked mon.noname-b con 0x7f190c0a3d90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:39:55.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1911225640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f190c0a4170 0x7f190c136bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:55.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1911225640 10 monclient(hunting): picked mon.noname-a con 0x7f190c0a4170 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:39:55.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1911225640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f190c13c3e0 0x7f190c138c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:55.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1911225640 10 monclient(hunting): picked mon.noname-c con 0x7f190c13c3e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:55.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1911225640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:55.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1911225640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f190c142aa0 con 0x7f190c0a4170 2024-04-26T00:39:55.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1911225640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:55.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1911225640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f190c142920 con 0x7f190c0a3d90 2024-04-26T00:39:55.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1911225640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:55.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1911225640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f190c142c20 con 0x7f190c13c3e0 2024-04-26T00:39:55.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1911225640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:55.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f1911225640 10 monclient(hunting): authenticate will time out at 2176.891149s 2024-04-26T00:39:55.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f190a59c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f190c0a4170 0x7f190c136bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:55.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f190a59c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f190c0a4170 0x7f190c136bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:45440/0 (socket says 172.21.15.37:45440) 2024-04-26T00:39:55.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f190a59c640 1 -- 172.21.15.37:0/4160885380 learned_addr learned my addr 172.21.15.37:0/4160885380 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:55.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f190a59c640 10 monclient(hunting): get_auth_request con 0x7f190c0a4170 auth_method 0 2024-04-26T00:39:55.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f190a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:55.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f190a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:55.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f190a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:55.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f190b59e640 1 --2- 172.21.15.37:0/4160885380 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f190c13c3e0 0x7f190c138c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:55.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f190b59e640 10 monclient(hunting): get_auth_request con 0x7f190c13c3e0 auth_method 0 2024-04-26T00:39:55.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f190b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:55.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f190b59e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:55.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f190b59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:55.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f190a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:55.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f190a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:55.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f190a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:55.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f190ad9d640 1 --2- 172.21.15.37:0/4160885380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f190c0a3d90 0x7f190c1cee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:55.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f190ad9d640 10 monclient(hunting): get_auth_request con 0x7f190c0a3d90 auth_method 0 2024-04-26T00:39:55.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f190ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:55.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f190ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:55.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f190ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:55.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f190b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:55.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f190b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:55.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f190b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:55.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f190a59c640 10 monclient(hunting): handle_auth_done global_id 244137 payload 824 2024-04-26T00:39:55.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f190a59c640 1 -- 172.21.15.37:0/4160885380 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f190c13c3e0 msgr2=0x7f190c138c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:55.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f190a59c640 1 --2- 172.21.15.37:0/4160885380 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f190c13c3e0 0x7f190c138c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:55.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f190a59c640 1 -- 172.21.15.37:0/4160885380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f190c0a3d90 msgr2=0x7f190c1cee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:55.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f190a59c640 1 --2- 172.21.15.37:0/4160885380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f190c0a3d90 0x7f190c1cee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:55.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f190a59c640 10 monclient: _finish_hunting 0 2024-04-26T00:39:55.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f190a59c640 1 monclient: found mon.noname-a 2024-04-26T00:39:55.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f190a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:55.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f190a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:39:55.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f190a59c640 1 -- 172.21.15.37:0/4160885380 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f190c120260 con 0x7f190c0a4170 2024-04-26T00:39:55.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.582+0000 7f190ad9d640 1 --2- 172.21.15.37:0/4160885380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f190c0a3d90 0x7f190c1cee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:39:55.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.586+0000 7f190a59c640 1 -- 172.21.15.37:0/4160885380 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f190c0a4170 msgr2=0x7f190c136bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:55.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.586+0000 7f190a59c640 1 --2- 172.21.15.37:0/4160885380 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f190c0a4170 0x7f190c136bc0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1900018ef0 tx=0x7f1900019080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:55.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.586+0000 7f18f37fe640 1 -- 172.21.15.37:0/4160885380 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19000b9070 con 0x7f190c0a4170 2024-04-26T00:39:55.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.586+0000 7f18f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:55.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.586+0000 7f18f37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:39:55.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.586+0000 7f18f37fe640 10 monclient: dump: 2024-04-26T00:39:55.621 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:55.621 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:55.621 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.621 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.621 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:55.622 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:55.622 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:55.622 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:55.622 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:55.622 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:55.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.586+0000 7f18f37fe640 10 monclient: _finish_auth 0 2024-04-26T00:39:55.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.586+0000 7f18f37fe640 10 monclient: _check_auth_tickets 2024-04-26T00:39:55.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.586+0000 7f18f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:55.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.586+0000 7f18f37fe640 1 -- 172.21.15.37:0/4160885380 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19000b5890 con 0x7f190c0a4170 2024-04-26T00:39:55.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.586+0000 7f1911225640 5 monclient: authenticate success, global_id 244137 2024-04-26T00:39:55.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.586+0000 7f1911225640 10 monclient: _renew_subs 2024-04-26T00:39:55.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.586+0000 7f1911225640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:39:55.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.586+0000 7f1911225640 1 -- 172.21.15.37:0/4160885380 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f190c1cfd50 con 0x7f190c0a4170 2024-04-26T00:39:55.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.586+0000 7f18f37fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:55.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.586+0000 7f18f37fe640 1 -- 172.21.15.37:0/4160885380 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19000b0040 con 0x7f190c0a4170 2024-04-26T00:39:55.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.586+0000 7f18f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:55.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.586+0000 7f18f37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:39:55.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.586+0000 7f18f37fe640 10 monclient: dump: 2024-04-26T00:39:55.623 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:55.623 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:55.623 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.623 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.623 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:55.624 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:55.624 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:55.624 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:55.624 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:55.624 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:55.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.586+0000 7f1911225640 10 monclient: _renew_subs 2024-04-26T00:39:55.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.586+0000 7f1911225640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:39:55.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.586+0000 7f1911225640 1 -- 172.21.15.37:0/4160885380 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f190c0a0800 con 0x7f190c0a4170 2024-04-26T00:39:55.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.586+0000 7f1908d99640 4 set_mon_vals no callback set 2024-04-26T00:39:55.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.586+0000 7f1911225640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:55.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.586+0000 7f1911225640 1 --2- 172.21.15.37:0/4160885380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f190c139150 0x7f190c139530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:55.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.586+0000 7f1911225640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:55.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.586+0000 7f1911225640 1 -- 172.21.15.37:0/4160885380 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f190c139a70 con 0x7f190c139150 2024-04-26T00:39:55.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.586+0000 7f18f37fe640 1 -- 172.21.15.37:0/4160885380 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f19000bc070 con 0x7f190c0a4170 2024-04-26T00:39:55.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.586+0000 7f18f37fe640 1 --2- 172.21.15.37:0/4160885380 >> v2:172.21.15.45:6812/2379918623 conn(0x7f18f8035eb0 0x7f18f8038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:55.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.586+0000 7f190ad9d640 1 --2- 172.21.15.37:0/4160885380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f190c139150 0x7f190c139530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:55.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.586+0000 7f18f37fe640 1 -- 172.21.15.37:0/4160885380 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(792..792 src has 705..792) ==== 3959+0+0 (secure 0 0 0) 0x7f19000e2c30 con 0x7f190c0a4170 2024-04-26T00:39:55.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.586+0000 7f190ad9d640 10 monclient: get_auth_request con 0x7f190c139150 auth_method 0 2024-04-26T00:39:55.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.586+0000 7f190ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:55.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.586+0000 7f190ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:55.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.586+0000 7f190ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:55.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.586+0000 7f190b59e640 1 --2- 172.21.15.37:0/4160885380 >> v2:172.21.15.45:6812/2379918623 conn(0x7f18f8035eb0 0x7f18f8038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:55.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.586+0000 7f190ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:55.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.586+0000 7f190ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:55.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.586+0000 7f190ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:55.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.586+0000 7f190b59e640 10 monclient: get_auth_request con 0x7f18f8035eb0 auth_method 0 2024-04-26T00:39:55.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.590+0000 7f190ad9d640 10 monclient(hunting): handle_auth_done global_id 224299 payload 824 2024-04-26T00:39:55.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.590+0000 7f190ad9d640 1 -- 172.21.15.37:0/4160885380 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f190c04f510 con 0x7f190c139150 2024-04-26T00:39:55.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.590+0000 7f190b59e640 1 --2- 172.21.15.37:0/4160885380 >> v2:172.21.15.45:6812/2379918623 conn(0x7f18f8035eb0 0x7f18f8038370 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f18fc007fd0 tx=0x7f18fc007dc0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:55.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.590+0000 7f190ad9d640 1 -- 172.21.15.37:0/4160885380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f190c139150 msgr2=0x7f190c139530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:55.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.590+0000 7f190ad9d640 1 --2- 172.21.15.37:0/4160885380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f190c139150 0x7f190c139530 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f190400ec30 tx=0x7f190400c4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:55.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.590+0000 7f18f37fe640 1 -- 172.21.15.37:0/4160885380 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f190400ce40 con 0x7f190c139150 2024-04-26T00:39:55.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.590+0000 7f18f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:55.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.590+0000 7f18f37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:39:55.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.590+0000 7f18f37fe640 10 monclient: dump: 2024-04-26T00:39:55.626 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:55.626 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:55.626 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.626 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.626 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:55.626 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:55.626 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:55.626 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:55.626 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:55.626 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:55.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.590+0000 7f18f37fe640 1 -- 172.21.15.37:0/4160885380 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1904018070 con 0x7f190c139150 2024-04-26T00:39:55.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.590+0000 7f18f37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:55.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.590+0000 7f18f37fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:39:55.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.590+0000 7f18f37fe640 1 -- 172.21.15.37:0/4160885380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f190c139150 msgr2=0x7f190c139530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:55.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.590+0000 7f18f37fe640 1 --2- 172.21.15.37:0/4160885380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f190c139150 0x7f190c139530 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f190400ec30 tx=0x7f190400c4b0 comp rx=0 tx=0).stop 2024-04-26T00:39:55.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.590+0000 7f18f37fe640 1 -- 172.21.15.37:0/4160885380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f190c139150 msgr2=0x7f190c139530 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:39:55.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.590+0000 7f18f37fe640 1 --2- 172.21.15.37:0/4160885380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f190c139150 0x7f190c139530 secure :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f190400ec30 tx=0x7f190400c4b0 comp rx=0 tx=0).stop 2024-04-26T00:39:55.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.598+0000 7f1911225640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:39:55.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.598+0000 7f1911225640 1 --2- 172.21.15.37:0/4160885380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f190c09d070 0x7f190c09d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:55.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.598+0000 7f1911225640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:55.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.598+0000 7f1911225640 1 -- 172.21.15.37:0/4160885380 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f190c11f9e0 con 0x7f190c09d070 2024-04-26T00:39:55.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.598+0000 7f190ad9d640 1 --2- 172.21.15.37:0/4160885380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f190c09d070 0x7f190c09d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:55.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.598+0000 7f190ad9d640 10 monclient: get_auth_request con 0x7f190c09d070 auth_method 0 2024-04-26T00:39:55.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.598+0000 7f190ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:55.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.598+0000 7f190ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:55.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.598+0000 7f190ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:55.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.598+0000 7f190ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:55.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.598+0000 7f190ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:55.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.598+0000 7f190ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:55.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.598+0000 7f190ad9d640 10 monclient(hunting): handle_auth_done global_id 224302 payload 824 2024-04-26T00:39:55.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.598+0000 7f190ad9d640 1 -- 172.21.15.37:0/4160885380 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f190c075dc0 con 0x7f190c09d070 2024-04-26T00:39:55.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.602+0000 7f190ad9d640 1 -- 172.21.15.37:0/4160885380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f190c09d070 msgr2=0x7f190c09d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:55.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.602+0000 7f190ad9d640 1 --2- 172.21.15.37:0/4160885380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f190c09d070 0x7f190c09d450 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1904017000 tx=0x7f1904004cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:55.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.602+0000 7f18f37fe640 1 -- 172.21.15.37:0/4160885380 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19040074e0 con 0x7f190c09d070 2024-04-26T00:39:55.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.602+0000 7f18f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:55.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.602+0000 7f18f37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:39:55.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.602+0000 7f18f37fe640 10 monclient: dump: 2024-04-26T00:39:55.629 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:55.629 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:55.629 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.629 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.629 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:55.629 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:55.629 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:55.630 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:55.630 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:55.630 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:55.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.602+0000 7f18f37fe640 1 -- 172.21.15.37:0/4160885380 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4888 (secure 0 0 0) 0x7f19040074e0 con 0x7f190c09d070 2024-04-26T00:39:55.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.602+0000 7f18f37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:39:55.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.602+0000 7f18f37fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:39:55.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.602+0000 7f18f37fe640 1 -- 172.21.15.37:0/4160885380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f190c09d070 msgr2=0x7f190c09d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:55.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.602+0000 7f18f37fe640 1 --2- 172.21.15.37:0/4160885380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f190c09d070 0x7f190c09d450 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1904017000 tx=0x7f1904004cd0 comp rx=0 tx=0).stop 2024-04-26T00:39:55.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.602+0000 7f18f37fe640 1 -- 172.21.15.37:0/4160885380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f190c09d070 msgr2=0x7f190c09d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:39:55.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.602+0000 7f18f37fe640 1 --2- 172.21.15.37:0/4160885380 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f190c09d070 0x7f190c09d450 secure :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1904017000 tx=0x7f1904004cd0 comp rx=0 tx=0).stop 2024-04-26T00:39:55.630 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:39:55.631 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:39:55.631 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:39:55.631 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:39:55.631 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 148, 2024-04-26T00:39:55.631 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 109266, 2024-04-26T00:39:55.631 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:39:55.631 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:39:55.631 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:39:55.631 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:39:55.631 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:55.631 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 6, 2024-04-26T00:39:55.631 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:39:55.631 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:39:55.631 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:39:55.631 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:55.632 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:55.632 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:55.632 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:55.632 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:55.632 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:55.632 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:55.632 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:55.632 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:55.632 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:55.632 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:55.632 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:55.632 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:39:55.632 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:39:55.632 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:55.633 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:55.633 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:55.633 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:55.633 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:55.633 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:55.633 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:55.633 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:55.633 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:55.633 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:55.633 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:55.633 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:55.633 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:55.633 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:39:55.633 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:39:55.633 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:39:55.634 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:39:55.634 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:39:55.634 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:39:55.634 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:39:55.634 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:39:55.634 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:39:55.634 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:39:55.634 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:39:55.634 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:39:55.634 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:39:55.634 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:39:55.634 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:39:55.634 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:39:55.634 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:39:55.635 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:39:55.635 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:39:55.635 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:39:55.635 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:39:55.635 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:39:55.635 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:39:55.635 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:39:55.635 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:39:55.635 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:39:55.635 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:39:55.635 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:39:55.635 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:55.635 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:39:55.635 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:55.635 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:39:55.636 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:55.636 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:39:55.636 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:39:55.636 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:55.636 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:55.636 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:55.636 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:55.636 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:39:55.636 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:55.636 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:55.636 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:55.636 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:55.636 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:39:55.636 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:55.636 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:55.636 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:55.637 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:55.637 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:39:55.637 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:39:55.637 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:55.637 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:55.637 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:55.637 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:55.637 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:55.637 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:39:55.637 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:39:55.637 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:55.637 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:55.637 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:55.637 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:55.637 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:39:55.638 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:55.638 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:55.638 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:55.638 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:55.638 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:39:55.638 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:55.638 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:55.638 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:55.638 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:55.638 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:39:55.638 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:39:55.638 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:55.638 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:55.638 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:55.639 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:55.639 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:55.639 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:39:55.639 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:39:55.639 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:39:55.639 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:39:55.639 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:55.639 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:39:55.639 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:39:55.639 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:55.639 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:55.639 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:55.639 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:39:55.639 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:39:55.639 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:39:55.640 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:55.640 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:55.640 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:55.640 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:39:55.640 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:39:55.640 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:39:55.640 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:39:55.640 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:39:55.640 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:55.640 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:55.640 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:55.640 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:39:55.640 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:39:55.640 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:55.640 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:55.641 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:55.641 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:39:55.641 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:55.641 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:55.641 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:39:55.641 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:55.641 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:55.641 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:55.641 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:39:55.641 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:55.641 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:55.641 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:39:55.641 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:55.641 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:55.641 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:55.642 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:39:55.642 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:55.642 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:39:55.642 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:39:55.642 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:39:55.642 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:39:55.642 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:39:55.642 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:39:55.642 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:39:55.642 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:39:55.642 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:39:55.642 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:39:55.642 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:39:55.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.602+0000 7f1911225640 1 -- 172.21.15.37:0/4160885380 >> v2:172.21.15.45:6812/2379918623 conn(0x7f18f8035eb0 msgr2=0x7f18f8038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:55.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.602+0000 7f1911225640 1 --2- 172.21.15.37:0/4160885380 >> v2:172.21.15.45:6812/2379918623 conn(0x7f18f8035eb0 0x7f18f8038370 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f18fc007fd0 tx=0x7f18fc007dc0 comp rx=0 tx=0).stop 2024-04-26T00:39:55.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.602+0000 7f1911225640 10 monclient: shutdown 2024-04-26T00:39:55.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.602+0000 7f1911225640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:55.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.602+0000 7f1911225640 1 -- 172.21.15.37:0/4160885380 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f190c0a4170 msgr2=0x7f190c136bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:55.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.602+0000 7f1911225640 1 --2- 172.21.15.37:0/4160885380 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f190c0a4170 0x7f190c136bc0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1900018ef0 tx=0x7f1900019080 comp rx=0 tx=0).stop 2024-04-26T00:39:55.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.602+0000 7f190b59e640 1 -- 172.21.15.37:0/4160885380 reap_dead start 2024-04-26T00:39:55.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.602+0000 7f1911225640 1 -- 172.21.15.37:0/4160885380 shutdown_connections 2024-04-26T00:39:55.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.602+0000 7f1911225640 1 --2- 172.21.15.37:0/4160885380 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f190c0a4170 0x7f190c136bc0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:55.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.602+0000 7f1911225640 1 -- 172.21.15.37:0/4160885380 >> 172.21.15.37:0/4160885380 conn(0x7f190c0aa0a0 msgr2=0x7f190c0a22d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:55.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.602+0000 7f1911225640 1 -- 172.21.15.37:0/4160885380 shutdown_connections 2024-04-26T00:39:55.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.602+0000 7f1911225640 1 -- 172.21.15.37:0/4160885380 wait complete. 2024-04-26T00:39:55.645 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:39:55.645 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-04-26T00:39:55.645 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-26T00:39:55.645 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-04-26T00:39:55.645 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-04-26T00:39:55.645 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-26T00:39:55.645 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T00:39:55.661 INFO:tasks.ceph.mon.c:Stopped 2024-04-26T00:39:55.661 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-26T00:39:55.661 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:39:55.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ef35f640 10 monclient: get_monmap_and_config 2024-04-26T00:39:55.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ef35f640 10 monclient: build_initial_monmap 2024-04-26T00:39:55.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ef35f640 10 monclient: monmap: 2024-04-26T00:39:55.830 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:55.830 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:55.830 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:39:55.833033+0000 2024-04-26T00:39:55.830 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:39:55.833033+0000 2024-04-26T00:39:55.830 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:55.831 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:55.831 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:39:55.831 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:39:55.831 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:39:55.831 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:39:55.831 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:39:55.831 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:55.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ef35f640 1 Processor -- start 2024-04-26T00:39:55.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ef35f640 1 -- start start 2024-04-26T00:39:55.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ef35f640 10 monclient: init 2024-04-26T00:39:55.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ef35f640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:55.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ef35f640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-04-26T00:39:55.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ef35f640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc5e80a0260 0x7fc5e8072e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:55.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ef35f640 10 monclient(hunting): picked mon.noname-c con 0x7fc5e80a0260 addr v2:172.21.15.45:3300/0 2024-04-26T00:39:55.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ef35f640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc5e80a0640 0x7fc5e80a0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:55.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ef35f640 10 monclient(hunting): picked mon.noname-a con 0x7fc5e80a0640 addr v2:172.21.15.37:3300/0 2024-04-26T00:39:55.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ef35f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc5e81337a0 addr v1:172.21.15.45:6789/0 2024-04-26T00:39:55.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ef35f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5e80a12e0 con 0x7fc5e81337a0 2024-04-26T00:39:55.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ef35f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:55.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ef35f640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fc5e806d050 con 0x7fc5e80a0640 2024-04-26T00:39:55.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ef35f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:55.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ef35f640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc5e809dcb0 con 0x7fc5e80a0260 2024-04-26T00:39:55.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ef35f640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:55.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ef35f640 10 monclient(hunting): authenticate will time out at 2177.138874s 2024-04-26T00:39:55.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ec8fa640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc5e80a0640 0x7fc5e80a0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:55.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ec8fa640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc5e80a0640 0x7fc5e80a0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:45492/0 (socket says 172.21.15.37:45492) 2024-04-26T00:39:55.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ec8fa640 1 -- 172.21.15.37:0/3031340616 learned_addr learned my addr 172.21.15.37:0/3031340616 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:55.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ec8fa640 10 monclient(hunting): get_auth_request con 0x7fc5e80a0640 auth_method 0 2024-04-26T00:39:55.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ec8fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:55.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ec8fa640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:55.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ec8fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:55.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ed0fb640 1 --2- 172.21.15.37:0/3031340616 >> v2:172.21.15.45:3300/0 conn(0x7fc5e80a0260 0x7fc5e8072e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:55.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ec8fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:55.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ec8fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:55.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ec8fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:55.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ed0fb640 10 monclient(hunting): get_auth_request con 0x7fc5e80a0260 auth_method 0 2024-04-26T00:39:55.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ed0fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:55.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ed0fb640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:55.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ed0fb640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:55.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ec8fa640 10 monclient(hunting): handle_auth_done global_id 244140 payload 274 2024-04-26T00:39:55.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ec8fa640 1 -- 172.21.15.37:0/3031340616 >> v2:172.21.15.45:3300/0 conn(0x7fc5e80a0260 msgr2=0x7fc5e8072e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:55.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ec8fa640 1 --2- 172.21.15.37:0/3031340616 >> v2:172.21.15.45:3300/0 conn(0x7fc5e80a0260 0x7fc5e8072e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:55.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ec8fa640 1 -- 172.21.15.37:0/3031340616 >> v1:172.21.15.45:6789/0 conn(0x7fc5e81337a0 legacy=0x7fc5e8133bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:55.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ec8fa640 10 monclient: _finish_hunting 0 2024-04-26T00:39:55.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ec8fa640 1 monclient: found mon.noname-a 2024-04-26T00:39:55.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ec8fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:55.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ec8fa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:39:55.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.830+0000 7fc5ec8fa640 1 -- 172.21.15.37:0/3031340616 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5e80a16f0 con 0x7fc5e80a0640 2024-04-26T00:39:55.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ec8fa640 1 -- 172.21.15.37:0/3031340616 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5e80a0640 msgr2=0x7fc5e80a0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:55.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ec8fa640 1 --2- 172.21.15.37:0/3031340616 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5e80a0640 0x7fc5e80a0a50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc5e80a2ad0 tx=0x7fc5e401c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3d2d9f58081cd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:55.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5dffff640 1 -- 172.21.15.37:0/3031340616 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5e40a5540 con 0x7fc5e80a0640 2024-04-26T00:39:55.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5dffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:55.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5dffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:39:55.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5dffff640 10 monclient: dump: 2024-04-26T00:39:55.838 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:55.838 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:55.838 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.838 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.838 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:55.838 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:55.839 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:55.839 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:55.839 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:55.839 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:55.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5dffff640 10 monclient: _finish_auth 0 2024-04-26T00:39:55.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5dffff640 10 monclient: _check_auth_tickets 2024-04-26T00:39:55.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:55.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5dffff640 1 -- 172.21.15.37:0/3031340616 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5e40a56e0 con 0x7fc5e80a0640 2024-04-26T00:39:55.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5dffff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:55.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5dffff640 1 -- 172.21.15.37:0/3031340616 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5e40a5ab0 con 0x7fc5e80a0640 2024-04-26T00:39:55.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ef35f640 5 monclient: authenticate success, global_id 244140 2024-04-26T00:39:55.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ef35f640 10 monclient: get_monmap_and_config success 2024-04-26T00:39:55.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ef35f640 4 set_mon_vals no callback set 2024-04-26T00:39:55.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5dffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:55.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5dffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:39:55.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5dffff640 10 monclient: dump: 2024-04-26T00:39:55.840 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:55.840 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:55.840 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.840 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.840 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:55.840 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:55.840 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:55.840 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:55.840 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:55.840 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:55.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ef35f640 10 monclient: shutdown 2024-04-26T00:39:55.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ef35f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:55.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ef35f640 1 -- 172.21.15.37:0/3031340616 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5e80a0640 msgr2=0x7fc5e80a0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:55.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ef35f640 1 --2- 172.21.15.37:0/3031340616 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5e80a0640 0x7fc5e80a0a50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc5e80a2ad0 tx=0x7fc5e401c900 comp rx=0 tx=0).stop 2024-04-26T00:39:55.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ef35f640 1 -- 172.21.15.37:0/3031340616 shutdown_connections 2024-04-26T00:39:55.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ef35f640 1 --2- 172.21.15.37:0/3031340616 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5e80a0640 0x7fc5e80a0a50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:55.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ef35f640 1 --2- 172.21.15.37:0/3031340616 >> v2:172.21.15.45:3300/0 conn(0x7fc5e80a0260 0x7fc5e8072e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:55.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ef35f640 1 -- 172.21.15.37:0/3031340616 >> 172.21.15.37:0/3031340616 conn(0x7fc5e80a1ad0 msgr2=0x7fc5e80a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:55.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ef35f640 1 -- 172.21.15.37:0/3031340616 shutdown_connections 2024-04-26T00:39:55.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ef35f640 1 -- 172.21.15.37:0/3031340616 wait complete. 2024-04-26T00:39:55.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ef35f640 10 monclient: build_initial_monmap 2024-04-26T00:39:55.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ef35f640 10 monclient: monmap: 2024-04-26T00:39:55.845 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:55.845 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:39:55.845 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:39:55.845 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:39:55.845 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:55.845 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:39:55.845 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:39:55.845 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:39:55.845 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:39:55.845 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:55.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ef35f640 1 Processor -- start 2024-04-26T00:39:55.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ef35f640 1 -- start start 2024-04-26T00:39:55.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ef35f640 10 monclient: init 2024-04-26T00:39:55.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ef35f640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:55.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ef35f640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:39:55.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ef35f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc5e80a0260 0x7fc5e81cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:55.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ef35f640 10 monclient(hunting): picked mon.noname-c con 0x7fc5e80a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:55.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ef35f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5e80a0640 0x7fc5e8132b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:55.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ef35f640 10 monclient(hunting): picked mon.noname-b con 0x7fc5e80a0640 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:39:55.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ef35f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5e81337a0 0x7fc5e81330a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:55.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ef35f640 10 monclient(hunting): picked mon.noname-a con 0x7fc5e81337a0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:39:55.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ed0fb640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc5e80a0260 msgr2=0x7fc5e81cf3c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:39:55.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ef35f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:55.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ef35f640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fc5e813feb0 con 0x7fc5e81337a0 2024-04-26T00:39:55.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ef35f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:55.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ef35f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc5e813fd30 con 0x7fc5e80a0640 2024-04-26T00:39:55.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ef35f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:55.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ef35f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc5e8140030 con 0x7fc5e80a0260 2024-04-26T00:39:55.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ed0fb640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc5e80a0260 0x7fc5e81cf3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:39:55.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ef35f640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:55.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ef35f640 10 monclient(hunting): authenticate will time out at 2177.147305s 2024-04-26T00:39:55.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ed8fc640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5e81337a0 0x7fc5e81330a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:55.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ed8fc640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5e81337a0 0x7fc5e81330a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:45502/0 (socket says 172.21.15.37:45502) 2024-04-26T00:39:55.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ed8fc640 1 -- 172.21.15.37:0/1649925923 learned_addr learned my addr 172.21.15.37:0/1649925923 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:55.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ed8fc640 10 monclient(hunting): get_auth_request con 0x7fc5e81337a0 auth_method 0 2024-04-26T00:39:55.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ed8fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:55.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ed8fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:55.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ed8fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:55.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ed8fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:55.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ed8fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:55.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ed8fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:55.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ec8fa640 1 --2- 172.21.15.37:0/1649925923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5e80a0640 0x7fc5e8132b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:55.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ec8fa640 10 monclient(hunting): get_auth_request con 0x7fc5e80a0640 auth_method 0 2024-04-26T00:39:55.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ec8fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:55.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ec8fa640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:55.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ec8fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:55.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ed8fc640 10 monclient(hunting): handle_auth_done global_id 244143 payload 824 2024-04-26T00:39:55.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ed8fc640 1 -- 172.21.15.37:0/1649925923 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc5e80a0260 msgr2=0x7fc5e81cf3c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:39:55.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ed8fc640 1 --2- 172.21.15.37:0/1649925923 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc5e80a0260 0x7fc5e81cf3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:55.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ed8fc640 1 -- 172.21.15.37:0/1649925923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5e80a0640 msgr2=0x7fc5e8132b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:55.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ed8fc640 1 --2- 172.21.15.37:0/1649925923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5e80a0640 0x7fc5e8132b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:55.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ed8fc640 10 monclient: _finish_hunting 0 2024-04-26T00:39:55.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ed8fc640 1 monclient: found mon.noname-a 2024-04-26T00:39:55.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ed8fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:55.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ed8fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:39:55.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.838+0000 7fc5ed8fc640 1 -- 172.21.15.37:0/1649925923 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5e81cf900 con 0x7fc5e81337a0 2024-04-26T00:39:55.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.842+0000 7fc5ed8fc640 1 -- 172.21.15.37:0/1649925923 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5e81337a0 msgr2=0x7fc5e81330a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:55.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.842+0000 7fc5ed8fc640 1 --2- 172.21.15.37:0/1649925923 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5e81337a0 0x7fc5e81330a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc5d800c3c0 tx=0x7fc5d800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:55.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.842+0000 7fc5ddffb640 1 -- 172.21.15.37:0/1649925923 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5d80119a0 con 0x7fc5e81337a0 2024-04-26T00:39:55.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.842+0000 7fc5ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:55.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.842+0000 7fc5ddffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:39:55.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.842+0000 7fc5ddffb640 10 monclient: dump: 2024-04-26T00:39:55.849 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:55.849 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:55.849 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.849 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.849 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:55.849 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:55.849 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:55.849 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:55.849 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:55.849 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:55.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.842+0000 7fc5ddffb640 10 monclient: _finish_auth 0 2024-04-26T00:39:55.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.842+0000 7fc5ddffb640 10 monclient: _check_auth_tickets 2024-04-26T00:39:55.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.842+0000 7fc5ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:55.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.842+0000 7fc5ddffb640 1 -- 172.21.15.37:0/1649925923 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5d8017070 con 0x7fc5e81337a0 2024-04-26T00:39:55.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.842+0000 7fc5ddffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:55.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.842+0000 7fc5ddffb640 1 -- 172.21.15.37:0/1649925923 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5d8011d80 con 0x7fc5e81337a0 2024-04-26T00:39:55.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.842+0000 7fc5ef35f640 5 monclient: authenticate success, global_id 244143 2024-04-26T00:39:55.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.842+0000 7fc5ef35f640 10 monclient: _renew_subs 2024-04-26T00:39:55.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.842+0000 7fc5ef35f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:39:55.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.842+0000 7fc5ef35f640 1 -- 172.21.15.37:0/1649925923 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5e813ab50 con 0x7fc5e81337a0 2024-04-26T00:39:55.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.842+0000 7fc5df7fe640 4 set_mon_vals no callback set 2024-04-26T00:39:55.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.842+0000 7fc5ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:55.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.842+0000 7fc5ddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:39:55.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.842+0000 7fc5ddffb640 10 monclient: dump: 2024-04-26T00:39:55.850 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:55.851 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:55.851 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.851 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:55.851 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:55.851 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:55.851 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:55.851 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:55.851 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:55.851 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:55.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.842+0000 7fc5ef35f640 10 monclient: _renew_subs 2024-04-26T00:39:55.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.842+0000 7fc5ef35f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:39:55.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.842+0000 7fc5ef35f640 1 -- 172.21.15.37:0/1649925923 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5e8139c60 con 0x7fc5e81337a0 2024-04-26T00:39:55.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.842+0000 7fc5ddffb640 1 -- 172.21.15.37:0/1649925923 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc5d801c460 con 0x7fc5e81337a0 2024-04-26T00:39:55.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.842+0000 7fc5ef35f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:55.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.842+0000 7fc5ef35f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:55.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.842+0000 7fc5ef35f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:39:55.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.842+0000 7fc5ddffb640 1 --2- 172.21.15.37:0/1649925923 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc5d4035e30 0x7fc5d40382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:55.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.842+0000 7fc5ef35f640 1 -- 172.21.15.37:0/1649925923 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5e8066f40 con 0x7fc5e81337a0 2024-04-26T00:39:55.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.842+0000 7fc5ddffb640 1 -- 172.21.15.37:0/1649925923 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(792..792 src has 705..792) ==== 3959+0+0 (secure 0 0 0) 0x7fc5d80488e0 con 0x7fc5e81337a0 2024-04-26T00:39:55.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.842+0000 7fc5ed0fb640 1 --2- 172.21.15.37:0/1649925923 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc5d4035e30 0x7fc5d40382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:55.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.846+0000 7fc5ed0fb640 10 monclient: get_auth_request con 0x7fc5d4035e30 auth_method 0 2024-04-26T00:39:55.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.846+0000 7fc5ed0fb640 1 --2- 172.21.15.37:0/1649925923 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc5d4035e30 0x7fc5d40382f0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0006e90 tx=0x7fc5e0006e20 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:55.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.850+0000 7fc5ddffb640 1 -- 172.21.15.37:0/1649925923 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc5d8037e20 con 0x7fc5e81337a0 2024-04-26T00:39:55.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.850+0000 7fc5ddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:55.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:55.850+0000 7fc5ddffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:39:56.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:56.074+0000 7fc5ef35f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:39:56.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:56.074+0000 7fc5ef35f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:39:56.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:56.074+0000 7fc5ef35f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:39:56.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:56.074+0000 7fc5ef35f640 1 -- 172.21.15.37:0/1649925923 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc5e806b9a0 con 0x7fc5e81337a0 2024-04-26T00:39:56.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:56.074+0000 7fc5ddffb640 1 -- 172.21.15.37:0/1649925923 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fc5d8047610 con 0x7fc5e81337a0 2024-04-26T00:39:56.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:56.074+0000 7fc5ddffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:39:56.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:56.074+0000 7fc5ddffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:39:56.076 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":148,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:39:56.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:56.078+0000 7fc5ef35f640 1 -- 172.21.15.37:0/1649925923 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc5d4035e30 msgr2=0x7fc5d40382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:56.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:56.078+0000 7fc5ef35f640 1 --2- 172.21.15.37:0/1649925923 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc5d4035e30 0x7fc5d40382f0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0006e90 tx=0x7fc5e0006e20 comp rx=0 tx=0).stop 2024-04-26T00:39:56.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:56.078+0000 7fc5ef35f640 10 monclient: shutdown 2024-04-26T00:39:56.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:56.078+0000 7fc5ef35f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:56.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:56.078+0000 7fc5ef35f640 1 -- 172.21.15.37:0/1649925923 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5e81337a0 msgr2=0x7fc5e81330a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:56.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:56.078+0000 7fc5ef35f640 1 --2- 172.21.15.37:0/1649925923 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5e81337a0 0x7fc5e81330a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc5d800c3c0 tx=0x7fc5d800c880 comp rx=0 tx=0).stop 2024-04-26T00:39:56.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:56.078+0000 7fc5ef35f640 1 -- 172.21.15.37:0/1649925923 shutdown_connections 2024-04-26T00:39:56.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:56.078+0000 7fc5ef35f640 1 --2- 172.21.15.37:0/1649925923 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc5d4035e30 0x7fc5d40382f0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:56.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:56.078+0000 7fc5ef35f640 1 --2- 172.21.15.37:0/1649925923 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5e81337a0 0x7fc5e81330a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:56.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:56.078+0000 7fc5ef35f640 1 --2- 172.21.15.37:0/1649925923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc5e80a0640 0x7fc5e8132b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:56.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:56.078+0000 7fc5ef35f640 1 --2- 172.21.15.37:0/1649925923 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc5e80a0260 0x7fc5e81cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:56.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:56.078+0000 7fc5ef35f640 1 -- 172.21.15.37:0/1649925923 >> 172.21.15.37:0/1649925923 conn(0x7fc5e80a1ad0 msgr2=0x7fc5e80a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:56.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:56.078+0000 7fc5ef35f640 1 -- 172.21.15.37:0/1649925923 shutdown_connections 2024-04-26T00:39:56.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:56.078+0000 7fc5ef35f640 1 -- 172.21.15.37:0/1649925923 wait complete. 2024-04-26T00:39:59.096 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:39:59.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.226+0000 7f84d9f96640 10 monclient: get_monmap_and_config 2024-04-26T00:39:59.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.226+0000 7f84d9f96640 10 monclient: build_initial_monmap 2024-04-26T00:39:59.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.226+0000 7f84d9f96640 10 monclient: monmap: 2024-04-26T00:39:59.227 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:59.227 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:59.227 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:39:59.229781+0000 2024-04-26T00:39:59.227 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:39:59.229781+0000 2024-04-26T00:39:59.227 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:59.227 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:59.227 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:39:59.227 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:39:59.227 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:39:59.228 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:39:59.228 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:39:59.228 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:59.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.226+0000 7f84d9f96640 1 Processor -- start 2024-04-26T00:39:59.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.226+0000 7f84d9f96640 1 -- start start 2024-04-26T00:39:59.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.226+0000 7f84d9f96640 10 monclient: init 2024-04-26T00:39:59.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.226+0000 7f84d9f96640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:59.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.226+0000 7f84d9f96640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-04-26T00:39:59.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.226+0000 7f84d9f96640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f84d40a0260 0x7f84d40a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:59.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.226+0000 7f84d9f96640 10 monclient(hunting): picked mon.noname-c con 0x7f84d40a0260 addr v2:172.21.15.45:3300/0 2024-04-26T00:39:59.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.226+0000 7f84d9f96640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f84d4134b30 0x7f84d4134f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:59.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.226+0000 7f84d9f96640 10 monclient(hunting): picked mon.noname-a con 0x7f84d4134b30 addr v2:172.21.15.37:3300/0 2024-04-26T00:39:59.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.226+0000 7f84d9f96640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f84d4135490 addr v1:172.21.15.45:6789/0 2024-04-26T00:39:59.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.226+0000 7f84d9f96640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84d40a1130 con 0x7f84d4135490 2024-04-26T00:39:59.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.226+0000 7f84d9f96640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:59.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.226+0000 7f84d9f96640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f84d409ec10 con 0x7f84d4134b30 2024-04-26T00:39:59.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.226+0000 7f84d9f96640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:59.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.226+0000 7f84d9f96640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f84d409f0f0 con 0x7f84d40a0260 2024-04-26T00:39:59.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.226+0000 7f84d9f96640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:59.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.226+0000 7f84d9f96640 10 monclient(hunting): authenticate will time out at 2180.535994s 2024-04-26T00:39:59.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.230+0000 7f84d2ffd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f84d4134b30 0x7f84d4134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:59.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.230+0000 7f84d2ffd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f84d4134b30 0x7f84d4134f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:45516/0 (socket says 172.21.15.37:45516) 2024-04-26T00:39:59.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.230+0000 7f84d2ffd640 1 -- 172.21.15.37:0/2801413767 learned_addr learned my addr 172.21.15.37:0/2801413767 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:59.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.230+0000 7f84d2ffd640 10 monclient(hunting): get_auth_request con 0x7f84d4134b30 auth_method 0 2024-04-26T00:39:59.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.230+0000 7f84d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:59.231 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.230+0000 7f84d2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:59.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.230+0000 7f84d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:59.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.230+0000 7f84d37fe640 1 --2- 172.21.15.37:0/2801413767 >> v2:172.21.15.45:3300/0 conn(0x7f84d40a0260 0x7f84d40a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:59.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.230+0000 7f84d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:59.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.230+0000 7f84d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:59.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.230+0000 7f84d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:59.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.230+0000 7f84d37fe640 10 monclient(hunting): get_auth_request con 0x7f84d40a0260 auth_method 0 2024-04-26T00:39:59.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.230+0000 7f84d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:59.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.230+0000 7f84d37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:59.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.230+0000 7f84d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:59.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.230+0000 7f84d2ffd640 10 monclient(hunting): handle_auth_done global_id 244146 payload 274 2024-04-26T00:39:59.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.230+0000 7f84d2ffd640 1 -- 172.21.15.37:0/2801413767 >> v2:172.21.15.45:3300/0 conn(0x7f84d40a0260 msgr2=0x7f84d40a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:59.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.230+0000 7f84d2ffd640 1 --2- 172.21.15.37:0/2801413767 >> v2:172.21.15.45:3300/0 conn(0x7f84d40a0260 0x7f84d40a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:59.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.230+0000 7f84d2ffd640 1 -- 172.21.15.37:0/2801413767 >> v1:172.21.15.45:6789/0 conn(0x7f84d4135490 legacy=0x7f84d413be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:39:59.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.230+0000 7f84d2ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:39:59.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.230+0000 7f84d2ffd640 1 monclient: found mon.noname-a 2024-04-26T00:39:59.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.230+0000 7f84d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:59.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.230+0000 7f84d2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:39:59.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.230+0000 7f84d2ffd640 1 -- 172.21.15.37:0/2801413767 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84d406cff0 con 0x7f84d4134b30 2024-04-26T00:39:59.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.230+0000 7f84d37fe640 1 --2- 172.21.15.37:0/2801413767 >> v2:172.21.15.45:3300/0 conn(0x7f84d40a0260 0x7f84d40a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:39:59.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.234+0000 7f84d2ffd640 1 -- 172.21.15.37:0/2801413767 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84d4134b30 msgr2=0x7f84d4134f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:59.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.234+0000 7f84d2ffd640 1 --2- 172.21.15.37:0/2801413767 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84d4134b30 0x7f84d4134f50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f84d40a2ad0 tx=0x7f84c801c1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af61604e067081a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:59.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.234+0000 7f84d27fc640 1 -- 172.21.15.37:0/2801413767 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c80b0070 con 0x7f84d4134b30 2024-04-26T00:39:59.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.234+0000 7f84d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:59.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.234+0000 7f84d27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:39:59.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.234+0000 7f84d27fc640 10 monclient: dump: 2024-04-26T00:39:59.236 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:59.236 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:59.236 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:59.236 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:59.236 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:59.236 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:59.236 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:59.236 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:59.236 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:59.236 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:59.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.234+0000 7f84d27fc640 10 monclient: _finish_auth 0 2024-04-26T00:39:59.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.234+0000 7f84d27fc640 10 monclient: _check_auth_tickets 2024-04-26T00:39:59.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.234+0000 7f84d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:59.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.234+0000 7f84d27fc640 1 -- 172.21.15.37:0/2801413767 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84c80ab600 con 0x7f84d4134b30 2024-04-26T00:39:59.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.234+0000 7f84d27fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:59.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.234+0000 7f84d27fc640 1 -- 172.21.15.37:0/2801413767 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c80ab9d0 con 0x7f84d4134b30 2024-04-26T00:39:59.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.234+0000 7f84d9f96640 5 monclient: authenticate success, global_id 244146 2024-04-26T00:39:59.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.234+0000 7f84d9f96640 10 monclient: get_monmap_and_config success 2024-04-26T00:39:59.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.234+0000 7f84d9f96640 4 set_mon_vals no callback set 2024-04-26T00:39:59.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.234+0000 7f84d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:59.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.234+0000 7f84d27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:39:59.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.234+0000 7f84d27fc640 10 monclient: dump: 2024-04-26T00:39:59.237 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:59.237 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:59.237 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:59.237 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:59.237 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:59.238 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:59.238 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:59.238 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:59.238 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:59.238 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:59.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.234+0000 7f84d9f96640 10 monclient: shutdown 2024-04-26T00:39:59.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.234+0000 7f84d9f96640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:59.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.234+0000 7f84d9f96640 1 -- 172.21.15.37:0/2801413767 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84d4134b30 msgr2=0x7f84d4134f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:59.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.234+0000 7f84d9f96640 1 --2- 172.21.15.37:0/2801413767 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84d4134b30 0x7f84d4134f50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f84d40a2ad0 tx=0x7f84c801c1c0 comp rx=0 tx=0).stop 2024-04-26T00:39:59.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.234+0000 7f84d9f96640 1 -- 172.21.15.37:0/2801413767 shutdown_connections 2024-04-26T00:39:59.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.234+0000 7f84d9f96640 1 --2- 172.21.15.37:0/2801413767 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84d4134b30 0x7f84d4134f50 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:59.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.234+0000 7f84d9f96640 1 --2- 172.21.15.37:0/2801413767 >> v2:172.21.15.45:3300/0 conn(0x7f84d40a0260 0x7f84d40a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:59.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.234+0000 7f84d9f96640 1 -- 172.21.15.37:0/2801413767 >> 172.21.15.37:0/2801413767 conn(0x7f84d40a1ad0 msgr2=0x7f84d40a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:59.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.234+0000 7f84d9f96640 1 -- 172.21.15.37:0/2801413767 shutdown_connections 2024-04-26T00:39:59.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.234+0000 7f84d9f96640 1 -- 172.21.15.37:0/2801413767 wait complete. 2024-04-26T00:39:59.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.234+0000 7f84d9f96640 10 monclient: build_initial_monmap 2024-04-26T00:39:59.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.234+0000 7f84d9f96640 10 monclient: monmap: 2024-04-26T00:39:59.239 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:39:59.239 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:39:59.239 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:39:59.239 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:39:59.239 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:39:59.239 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:39:59.239 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:39:59.239 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:39:59.239 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:39:59.239 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:59.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.234+0000 7f84d9f96640 1 Processor -- start 2024-04-26T00:39:59.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.234+0000 7f84d9f96640 1 -- start start 2024-04-26T00:39:59.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.234+0000 7f84d9f96640 10 monclient: init 2024-04-26T00:39:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d9f96640 10 monclient: _reopen_session rank -1 2024-04-26T00:39:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d9f96640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:39:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d9f96640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f84d40a0260 0x7f84d41ccc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d9f96640 10 monclient(hunting): picked mon.noname-c con 0x7f84d40a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:39:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d9f96640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84d4134b30 0x7f84d4131910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d9f96640 10 monclient(hunting): picked mon.noname-a con 0x7f84d4134b30 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:39:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d9f96640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84d4135490 0x7f84d4139940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d9f96640 10 monclient(hunting): picked mon.noname-b con 0x7f84d4135490 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:39:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d9f96640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d9f96640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f84d409f430 con 0x7f84d4134b30 2024-04-26T00:39:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d9f96640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d9f96640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f84d409f2b0 con 0x7f84d4135490 2024-04-26T00:39:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d9f96640 10 monclient(hunting): start opening mon connection 2024-04-26T00:39:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d9f96640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f84d409fe90 con 0x7f84d40a0260 2024-04-26T00:39:59.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d9f96640 10 monclient(hunting): _renew_subs 2024-04-26T00:39:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d9f96640 10 monclient(hunting): authenticate will time out at 2180.544292s 2024-04-26T00:39:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d37fe640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f84d40a0260 msgr2=0x7f84d41ccc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:39:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d37fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f84d40a0260 0x7f84d41ccc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:39:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d2ffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84d4134b30 0x7f84d4131910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d2ffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84d4134b30 0x7f84d4131910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:45524/0 (socket says 172.21.15.37:45524) 2024-04-26T00:39:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d2ffd640 1 -- 172.21.15.37:0/1608080371 learned_addr learned my addr 172.21.15.37:0/1608080371 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:39:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d2ffd640 10 monclient(hunting): get_auth_request con 0x7f84d4134b30 auth_method 0 2024-04-26T00:39:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:39:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:39:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:39:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:39:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:39:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:39:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d3fff640 1 --2- 172.21.15.37:0/1608080371 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84d4135490 0x7f84d4139940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d2ffd640 10 monclient(hunting): handle_auth_done global_id 244149 payload 824 2024-04-26T00:39:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d2ffd640 1 -- 172.21.15.37:0/1608080371 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f84d40a0260 msgr2=0x7f84d41ccc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:39:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d2ffd640 1 --2- 172.21.15.37:0/1608080371 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f84d40a0260 0x7f84d41ccc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d2ffd640 1 -- 172.21.15.37:0/1608080371 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84d4135490 msgr2=0x7f84d4139940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d2ffd640 1 --2- 172.21.15.37:0/1608080371 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84d4135490 0x7f84d4139940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d2ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:39:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d2ffd640 1 monclient: found mon.noname-a 2024-04-26T00:39:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:39:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:39:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d2ffd640 1 -- 172.21.15.37:0/1608080371 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84d40729d0 con 0x7f84d4134b30 2024-04-26T00:39:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d2ffd640 1 -- 172.21.15.37:0/1608080371 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84d4134b30 msgr2=0x7f84d4131910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:39:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d2ffd640 1 --2- 172.21.15.37:0/1608080371 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84d4134b30 0x7f84d4131910 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f84c801a870 tx=0x7f84c8018f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84bbfff640 1 -- 172.21.15.37:0/1608080371 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c80ba070 con 0x7f84d4134b30 2024-04-26T00:39:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84bbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:39:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84bbfff640 10 monclient: dump: 2024-04-26T00:39:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:59.247 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84bbfff640 10 monclient: _finish_auth 0 2024-04-26T00:39:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84bbfff640 10 monclient: _check_auth_tickets 2024-04-26T00:39:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:39:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84bbfff640 1 -- 172.21.15.37:0/1608080371 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84c80b1040 con 0x7f84d4134b30 2024-04-26T00:39:59.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d9f96640 5 monclient: authenticate success, global_id 244149 2024-04-26T00:39:59.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d9f96640 10 monclient: _renew_subs 2024-04-26T00:39:59.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d9f96640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:39:59.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d9f96640 1 -- 172.21.15.37:0/1608080371 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84d4134870 con 0x7f84d4134b30 2024-04-26T00:39:59.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84bbfff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:39:59.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84bbfff640 1 -- 172.21.15.37:0/1608080371 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c80b0070 con 0x7f84d4134b30 2024-04-26T00:39:59.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:39:59.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84bbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:39:59.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84bbfff640 10 monclient: dump: 2024-04-26T00:39:59.248 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:39:59.248 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:39:59.248 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:59.248 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:39:59.249 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:39:59.249 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:39:59.249 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:39:59.249 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:39:59.249 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:39:59.249 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:39:59.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d9f96640 10 monclient: _renew_subs 2024-04-26T00:39:59.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d9f96640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:39:59.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d1ffb640 4 set_mon_vals no callback set 2024-04-26T00:39:59.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.238+0000 7f84d9f96640 1 -- 172.21.15.37:0/1608080371 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84d411c2a0 con 0x7f84d4134b30 2024-04-26T00:39:59.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.242+0000 7f84d9f96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:59.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.242+0000 7f84d9f96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:59.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.242+0000 7f84d9f96640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:39:59.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.242+0000 7f84d9f96640 1 -- 172.21.15.37:0/1608080371 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84d4066f40 con 0x7f84d4134b30 2024-04-26T00:39:59.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.242+0000 7f84bbfff640 1 -- 172.21.15.37:0/1608080371 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f84c80bd070 con 0x7f84d4134b30 2024-04-26T00:39:59.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.242+0000 7f84bbfff640 1 --2- 172.21.15.37:0/1608080371 >> v2:172.21.15.45:6812/2379918623 conn(0x7f84c0035e30 0x7f84c00382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:39:59.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.242+0000 7f84bbfff640 1 -- 172.21.15.37:0/1608080371 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(792..792 src has 705..792) ==== 3959+0+0 (secure 0 0 0) 0x7f84c80e1c30 con 0x7f84d4134b30 2024-04-26T00:39:59.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.242+0000 7f84d37fe640 1 --2- 172.21.15.37:0/1608080371 >> v2:172.21.15.45:6812/2379918623 conn(0x7f84c0035e30 0x7f84c00382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:39:59.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.242+0000 7f84d37fe640 10 monclient: get_auth_request con 0x7f84c0035e30 auth_method 0 2024-04-26T00:39:59.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.246+0000 7f84d37fe640 1 --2- 172.21.15.37:0/1608080371 >> v2:172.21.15.45:6812/2379918623 conn(0x7f84c0035e30 0x7f84c00382f0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f84c40096e0 tx=0x7f84c4009a20 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:39:59.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.246+0000 7f84bbfff640 1 -- 172.21.15.37:0/1608080371 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f84d40a1130 con 0x7f84d4134b30 2024-04-26T00:39:59.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.246+0000 7f84bbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:39:59.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.246+0000 7f84bbfff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:39:59.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.470+0000 7f84d9f96640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:39:59.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.470+0000 7f84d9f96640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:39:59.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.470+0000 7f84d9f96640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:39:59.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.470+0000 7f84d9f96640 1 -- 172.21.15.37:0/1608080371 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f84d41cdaf0 con 0x7f84d4134b30 2024-04-26T00:39:59.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.474+0000 7f84bbfff640 1 -- 172.21.15.37:0/1608080371 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1558 (secure 0 0 0) 0x7f84c80b4360 con 0x7f84d4134b30 2024-04-26T00:39:59.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.474+0000 7f84bbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:39:59.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.474+0000 7f84bbfff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:39:59.473 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":148,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:39:59.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.474+0000 7f84d9f96640 1 -- 172.21.15.37:0/1608080371 >> v2:172.21.15.45:6812/2379918623 conn(0x7f84c0035e30 msgr2=0x7f84c00382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:59.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.474+0000 7f84d9f96640 1 --2- 172.21.15.37:0/1608080371 >> v2:172.21.15.45:6812/2379918623 conn(0x7f84c0035e30 0x7f84c00382f0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f84c40096e0 tx=0x7f84c4009a20 comp rx=0 tx=0).stop 2024-04-26T00:39:59.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.474+0000 7f84d9f96640 10 monclient: shutdown 2024-04-26T00:39:59.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.474+0000 7f84d9f96640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:39:59.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.474+0000 7f84d9f96640 1 -- 172.21.15.37:0/1608080371 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84d4134b30 msgr2=0x7f84d4131910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:39:59.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.474+0000 7f84d9f96640 1 --2- 172.21.15.37:0/1608080371 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84d4134b30 0x7f84d4131910 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f84c801a870 tx=0x7f84c8018f60 comp rx=0 tx=0).stop 2024-04-26T00:39:59.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.474+0000 7f84d9f96640 1 -- 172.21.15.37:0/1608080371 shutdown_connections 2024-04-26T00:39:59.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.474+0000 7f84d9f96640 1 --2- 172.21.15.37:0/1608080371 >> v2:172.21.15.45:6812/2379918623 conn(0x7f84c0035e30 0x7f84c00382f0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:59.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.474+0000 7f84d9f96640 1 --2- 172.21.15.37:0/1608080371 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84d4135490 0x7f84d4139940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:59.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.474+0000 7f84d9f96640 1 --2- 172.21.15.37:0/1608080371 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84d4134b30 0x7f84d4131910 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:59.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.474+0000 7f84d9f96640 1 --2- 172.21.15.37:0/1608080371 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f84d40a0260 0x7f84d41ccc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:39:59.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.474+0000 7f84d9f96640 1 -- 172.21.15.37:0/1608080371 >> 172.21.15.37:0/1608080371 conn(0x7f84d40a1ad0 msgr2=0x7f84d40a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:39:59.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.474+0000 7f84d9f96640 1 -- 172.21.15.37:0/1608080371 shutdown_connections 2024-04-26T00:39:59.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:39:59.474+0000 7f84d9f96640 1 -- 172.21.15.37:0/1608080371 wait complete. 2024-04-26T00:40:02.494 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:40:02.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.626+0000 7f21e4ec0640 10 monclient: get_monmap_and_config 2024-04-26T00:40:02.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.626+0000 7f21e4ec0640 10 monclient: build_initial_monmap 2024-04-26T00:40:02.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.626+0000 7f21e4ec0640 10 monclient: monmap: 2024-04-26T00:40:02.627 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:40:02.627 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:02.627 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:40:02.629583+0000 2024-04-26T00:40:02.627 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:40:02.629583+0000 2024-04-26T00:40:02.627 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:40:02.627 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:02.627 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:40:02.627 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:40:02.627 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:40:02.627 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:40:02.627 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:40:02.628 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:02.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.626+0000 7f21e4ec0640 1 Processor -- start 2024-04-26T00:40:02.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.626+0000 7f21e4ec0640 1 -- start start 2024-04-26T00:40:02.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.626+0000 7f21e4ec0640 10 monclient: init 2024-04-26T00:40:02.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.626+0000 7f21e4ec0640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:02.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.626+0000 7f21e4ec0640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-04-26T00:40:02.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.626+0000 7f21e4ec0640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f21e00a0260 0x7f21e00a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:02.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.626+0000 7f21e4ec0640 10 monclient(hunting): picked mon.noname-c con 0x7f21e00a0260 addr v2:172.21.15.45:3300/0 2024-04-26T00:40:02.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.626+0000 7f21e4ec0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f21e0134b30 addr v1:172.21.15.45:6789/0 2024-04-26T00:40:02.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.626+0000 7f21e4ec0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f21e0135660 addr v1:172.21.15.37:6789/0 2024-04-26T00:40:02.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.626+0000 7f21e4ec0640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21e00a1130 con 0x7f21e0135660 2024-04-26T00:40:02.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.626+0000 7f21e4ec0640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21e006cff0 con 0x7f21e0134b30 2024-04-26T00:40:02.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.626+0000 7f21e4ec0640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:02.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.626+0000 7f21e4ec0640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f21e009ec10 con 0x7f21e00a0260 2024-04-26T00:40:02.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.626+0000 7f21e4ec0640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:02.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.626+0000 7f21e4ec0640 10 monclient(hunting): authenticate will time out at 2183.935383s 2024-04-26T00:40:02.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.626+0000 7f21ded9d640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f21e0135660 0x7f21e013ce70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:43426/0 (socket says 172.21.15.37:43426) 2024-04-26T00:40:02.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.626+0000 7f21ded9d640 1 -- 172.21.15.37:0/2924938832 learned_addr learned my addr 172.21.15.37:0/2924938832 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:40:02.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.626+0000 7f21de59c640 1 --2- 172.21.15.37:0/2924938832 >> v2:172.21.15.45:3300/0 conn(0x7f21e00a0260 0x7f21e00a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:02.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.626+0000 7f21de59c640 10 monclient(hunting): get_auth_request con 0x7f21e00a0260 auth_method 0 2024-04-26T00:40:02.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.626+0000 7f21de59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:02.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.626+0000 7f21de59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:02.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.626+0000 7f21de59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:02.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.626+0000 7f21de59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:02.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.626+0000 7f21de59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:02.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.626+0000 7f21de59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:02.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.630+0000 7f21de59c640 10 monclient(hunting): handle_auth_done global_id 224314 payload 274 2024-04-26T00:40:02.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.630+0000 7f21de59c640 1 -- 172.21.15.37:0/2924938832 >> v1:172.21.15.45:6789/0 conn(0x7f21e0134b30 legacy=0x7f21e0134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:40:02.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.630+0000 7f21de59c640 1 -- 172.21.15.37:0/2924938832 >> v1:172.21.15.37:6789/0 conn(0x7f21e0135660 legacy=0x7f21e013ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:40:02.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.630+0000 7f21de59c640 10 monclient: _finish_hunting 0 2024-04-26T00:40:02.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.630+0000 7f21de59c640 1 monclient: found mon.noname-c 2024-04-26T00:40:02.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.630+0000 7f21de59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:02.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.630+0000 7f21de59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:40:02.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.630+0000 7f21de59c640 1 -- 172.21.15.37:0/2924938832 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21e009bc50 con 0x7f21e00a0260 2024-04-26T00:40:02.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.634+0000 7f21de59c640 1 -- 172.21.15.37:0/2924938832 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21e00a0260 msgr2=0x7f21e00a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:02.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.634+0000 7f21de59c640 1 --2- 172.21.15.37:0/2924938832 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21e00a0260 0x7f21e00a0640 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f21d00097c0 tx=0x7f21d001c100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b20682b4cad7116 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:02.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.634+0000 7f21dd59a640 1 -- 172.21.15.37:0/2924938832 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21d00b1070 con 0x7f21e00a0260 2024-04-26T00:40:02.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.634+0000 7f21dd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:02.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.634+0000 7f21dd59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:40:02.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.634+0000 7f21dd59a640 10 monclient: dump: 2024-04-26T00:40:02.634 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:02.635 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:02.635 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:02.635 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:02.635 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:02.635 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:02.635 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:02.635 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:02.635 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:02.635 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:02.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.634+0000 7f21dd59a640 10 monclient: _finish_auth 0 2024-04-26T00:40:02.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.634+0000 7f21dd59a640 10 monclient: _check_auth_tickets 2024-04-26T00:40:02.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.634+0000 7f21dd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:02.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.634+0000 7f21dd59a640 1 -- 172.21.15.37:0/2924938832 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21d00ac440 con 0x7f21e00a0260 2024-04-26T00:40:02.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.634+0000 7f21dd59a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:40:02.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.634+0000 7f21dd59a640 1 -- 172.21.15.37:0/2924938832 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21d00ac810 con 0x7f21e00a0260 2024-04-26T00:40:02.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.634+0000 7f21dd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:02.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.634+0000 7f21dd59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:40:02.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.634+0000 7f21dd59a640 10 monclient: dump: 2024-04-26T00:40:02.636 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:02.636 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:02.636 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:02.636 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:02.636 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:02.636 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:02.637 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:02.637 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:02.637 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:02.637 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:02.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.634+0000 7f21e4ec0640 5 monclient: authenticate success, global_id 224314 2024-04-26T00:40:02.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.634+0000 7f21e4ec0640 10 monclient: get_monmap_and_config success 2024-04-26T00:40:02.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.634+0000 7f21e4ec0640 4 set_mon_vals no callback set 2024-04-26T00:40:02.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.634+0000 7f21e4ec0640 10 monclient: shutdown 2024-04-26T00:40:02.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.634+0000 7f21e4ec0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:40:02.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.634+0000 7f21e4ec0640 1 -- 172.21.15.37:0/2924938832 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21e00a0260 msgr2=0x7f21e00a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:02.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.634+0000 7f21e4ec0640 1 --2- 172.21.15.37:0/2924938832 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21e00a0260 0x7f21e00a0640 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f21d00097c0 tx=0x7f21d001c100 comp rx=0 tx=0).stop 2024-04-26T00:40:02.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.634+0000 7f21e4ec0640 1 -- 172.21.15.37:0/2924938832 shutdown_connections 2024-04-26T00:40:02.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.634+0000 7f21e4ec0640 1 --2- 172.21.15.37:0/2924938832 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21e00a0260 0x7f21e00a0640 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:02.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.634+0000 7f21e4ec0640 1 -- 172.21.15.37:0/2924938832 >> 172.21.15.37:0/2924938832 conn(0x7f21e00a1ad0 msgr2=0x7f21e00a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:40:02.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.634+0000 7f21e4ec0640 1 -- 172.21.15.37:0/2924938832 shutdown_connections 2024-04-26T00:40:02.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.634+0000 7f21e4ec0640 1 -- 172.21.15.37:0/2924938832 wait complete. 2024-04-26T00:40:02.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.634+0000 7f21e4ec0640 10 monclient: build_initial_monmap 2024-04-26T00:40:02.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.634+0000 7f21e4ec0640 10 monclient: monmap: 2024-04-26T00:40:02.643 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:40:02.643 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:40:02.643 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:40:02.643 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:40:02.643 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:40:02.643 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:40:02.643 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:40:02.643 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:40:02.643 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:40:02.643 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:02.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.634+0000 7f21e4ec0640 1 Processor -- start 2024-04-26T00:40:02.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.634+0000 7f21e4ec0640 1 -- start start 2024-04-26T00:40:02.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.634+0000 7f21e4ec0640 10 monclient: init 2024-04-26T00:40:02.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21e4ec0640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:02.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21e4ec0640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:40:02.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21e4ec0640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f21e00a0260 0x7f21e01cef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:02.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21e4ec0640 10 monclient(hunting): picked mon.noname-a con 0x7f21e00a0260 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:40:02.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21e4ec0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21e0134b30 0x7f21e0131a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:02.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21e4ec0640 10 monclient(hunting): picked mon.noname-b con 0x7f21e0134b30 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:40:02.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21e4ec0640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f21e0135660 0x7f21e013c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:02.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21e4ec0640 10 monclient(hunting): picked mon.noname-c con 0x7f21e0135660 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:40:02.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21e4ec0640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:02.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21e4ec0640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f21e009f4c0 con 0x7f21e00a0260 2024-04-26T00:40:02.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21e4ec0640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:02.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21e4ec0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f21e009f0f0 con 0x7f21e0134b30 2024-04-26T00:40:02.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21e4ec0640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:02.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21e4ec0640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f21e0140db0 con 0x7f21e0135660 2024-04-26T00:40:02.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21e4ec0640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:02.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21e4ec0640 10 monclient(hunting): authenticate will time out at 2183.944163s 2024-04-26T00:40:02.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21ded9d640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f21e0135660 msgr2=0x7f21e013c070 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:40:02.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21de59c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f21e00a0260 0x7f21e01cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:02.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21ded9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f21e0135660 0x7f21e013c070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:40:02.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21de59c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f21e00a0260 0x7f21e01cef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40066/0 (socket says 172.21.15.37:40066) 2024-04-26T00:40:02.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21de59c640 1 -- 172.21.15.37:0/2764473312 learned_addr learned my addr 172.21.15.37:0/2764473312 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:40:02.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21de59c640 10 monclient(hunting): get_auth_request con 0x7f21e00a0260 auth_method 0 2024-04-26T00:40:02.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21de59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:02.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21de59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:02.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21de59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:02.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21de59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:02.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21de59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:02.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21de59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:02.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21ddd9b640 1 --2- 172.21.15.37:0/2764473312 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21e0134b30 0x7f21e0131a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:02.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21ddd9b640 10 monclient(hunting): get_auth_request con 0x7f21e0134b30 auth_method 0 2024-04-26T00:40:02.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21ddd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:02.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21ddd9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:02.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21ddd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:02.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21de59c640 10 monclient(hunting): handle_auth_done global_id 244152 payload 824 2024-04-26T00:40:02.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21de59c640 1 -- 172.21.15.37:0/2764473312 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f21e0135660 msgr2=0x7f21e013c070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:40:02.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21de59c640 1 --2- 172.21.15.37:0/2764473312 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f21e0135660 0x7f21e013c070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:02.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21de59c640 1 -- 172.21.15.37:0/2764473312 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21e0134b30 msgr2=0x7f21e0131a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:02.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21de59c640 1 --2- 172.21.15.37:0/2764473312 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21e0134b30 0x7f21e0131a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:02.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21de59c640 10 monclient: _finish_hunting 0 2024-04-26T00:40:02.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21de59c640 1 monclient: found mon.noname-a 2024-04-26T00:40:02.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21de59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:02.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21de59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:40:02.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21de59c640 1 -- 172.21.15.37:0/2764473312 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21e013dc70 con 0x7f21e00a0260 2024-04-26T00:40:02.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21de59c640 1 -- 172.21.15.37:0/2764473312 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f21e00a0260 msgr2=0x7f21e01cef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:02.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21de59c640 1 --2- 172.21.15.37:0/2764473312 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f21e00a0260 0x7f21e01cef50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f21d001a7b0 tx=0x7f21d0018ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:02.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21d6ffd640 1 -- 172.21.15.37:0/2764473312 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21d00b9020 con 0x7f21e00a0260 2024-04-26T00:40:02.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:02.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21d6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:40:02.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21d6ffd640 10 monclient: dump: 2024-04-26T00:40:02.647 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:02.647 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:02.647 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:02.647 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:02.647 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:02.648 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:02.648 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:02.648 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:02.648 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:02.648 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:02.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21d6ffd640 10 monclient: _finish_auth 0 2024-04-26T00:40:02.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21d6ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:40:02.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:02.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21d6ffd640 1 -- 172.21.15.37:0/2764473312 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21d00b5610 con 0x7f21e00a0260 2024-04-26T00:40:02.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21e4ec0640 5 monclient: authenticate success, global_id 244152 2024-04-26T00:40:02.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21e4ec0640 10 monclient: _renew_subs 2024-04-26T00:40:02.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21e4ec0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:40:02.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21e4ec0640 1 -- 172.21.15.37:0/2764473312 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21e01cf490 con 0x7f21e00a0260 2024-04-26T00:40:02.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21d6ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:40:02.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21d6ffd640 1 -- 172.21.15.37:0/2764473312 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21d00b1040 con 0x7f21e00a0260 2024-04-26T00:40:02.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:02.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21d6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:40:02.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21d6ffd640 10 monclient: dump: 2024-04-26T00:40:02.649 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:02.649 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:02.649 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:02.650 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:02.650 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:02.650 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:02.650 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:02.650 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:02.650 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:02.650 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:02.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21e4ec0640 10 monclient: _renew_subs 2024-04-26T00:40:02.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21e4ec0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:40:02.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21e4ec0640 1 -- 172.21.15.37:0/2764473312 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21e0134700 con 0x7f21e00a0260 2024-04-26T00:40:02.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.638+0000 7f21dcd99640 4 set_mon_vals no callback set 2024-04-26T00:40:02.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.642+0000 7f21d6ffd640 1 -- 172.21.15.37:0/2764473312 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f21d00bc020 con 0x7f21e00a0260 2024-04-26T00:40:02.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.642+0000 7f21e4ec0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:02.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.642+0000 7f21d6ffd640 1 --2- 172.21.15.37:0/2764473312 >> v2:172.21.15.45:6812/2379918623 conn(0x7f21c0035e30 0x7f21c00382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:02.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.642+0000 7f21e4ec0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:02.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.642+0000 7f21e4ec0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:40:02.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.642+0000 7f21e4ec0640 1 -- 172.21.15.37:0/2764473312 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21e0066f40 con 0x7f21e00a0260 2024-04-26T00:40:02.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.642+0000 7f21d6ffd640 1 -- 172.21.15.37:0/2764473312 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(792..792 src has 705..792) ==== 3959+0+0 (secure 0 0 0) 0x7f21d00e26a0 con 0x7f21e00a0260 2024-04-26T00:40:02.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.642+0000 7f21ddd9b640 1 --2- 172.21.15.37:0/2764473312 >> v2:172.21.15.45:6812/2379918623 conn(0x7f21c0035e30 0x7f21c00382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:02.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.642+0000 7f21ddd9b640 10 monclient: get_auth_request con 0x7f21c0035e30 auth_method 0 2024-04-26T00:40:02.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.642+0000 7f21ddd9b640 1 --2- 172.21.15.37:0/2764473312 >> v2:172.21.15.45:6812/2379918623 conn(0x7f21c0035e30 0x7f21c00382f0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f21e013d910 tx=0x7f21c8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:02.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.646+0000 7f21d6ffd640 1 -- 172.21.15.37:0/2764473312 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f21d00c3070 con 0x7f21e00a0260 2024-04-26T00:40:02.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.646+0000 7f21d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:02.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.646+0000 7f21d6ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:40:02.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.862+0000 7f21e4ec0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:40:02.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.862+0000 7f21e4ec0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:40:02.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.862+0000 7f21e4ec0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:40:02.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.862+0000 7f21e4ec0640 1 -- 172.21.15.37:0/2764473312 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f21e011cbd0 con 0x7f21e00a0260 2024-04-26T00:40:02.863 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.862+0000 7f21d6ffd640 1 -- 172.21.15.37:0/2764473312 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1558 (secure 0 0 0) 0x7f21e00a1130 con 0x7f21e00a0260 2024-04-26T00:40:02.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.862+0000 7f21d6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:40:02.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.862+0000 7f21d6ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:40:02.864 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":148,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:40:02.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.866+0000 7f21e4ec0640 1 -- 172.21.15.37:0/2764473312 >> v2:172.21.15.45:6812/2379918623 conn(0x7f21c0035e30 msgr2=0x7f21c00382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:02.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.866+0000 7f21e4ec0640 1 --2- 172.21.15.37:0/2764473312 >> v2:172.21.15.45:6812/2379918623 conn(0x7f21c0035e30 0x7f21c00382f0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f21e013d910 tx=0x7f21c8009040 comp rx=0 tx=0).stop 2024-04-26T00:40:02.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.866+0000 7f21e4ec0640 10 monclient: shutdown 2024-04-26T00:40:02.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.866+0000 7f21e4ec0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:40:02.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.866+0000 7f21e4ec0640 1 -- 172.21.15.37:0/2764473312 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f21e00a0260 msgr2=0x7f21e01cef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:02.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.866+0000 7f21e4ec0640 1 --2- 172.21.15.37:0/2764473312 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f21e00a0260 0x7f21e01cef50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f21d001a7b0 tx=0x7f21d0018ea0 comp rx=0 tx=0).stop 2024-04-26T00:40:02.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.866+0000 7f21e4ec0640 1 -- 172.21.15.37:0/2764473312 shutdown_connections 2024-04-26T00:40:02.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.866+0000 7f21e4ec0640 1 --2- 172.21.15.37:0/2764473312 >> v2:172.21.15.45:6812/2379918623 conn(0x7f21c0035e30 0x7f21c00382f0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:02.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.866+0000 7f21e4ec0640 1 --2- 172.21.15.37:0/2764473312 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f21e0135660 0x7f21e013c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:02.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.866+0000 7f21e4ec0640 1 --2- 172.21.15.37:0/2764473312 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f21e0134b30 0x7f21e0131a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:02.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.866+0000 7f21e4ec0640 1 --2- 172.21.15.37:0/2764473312 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f21e00a0260 0x7f21e01cef50 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:02.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.866+0000 7f21e4ec0640 1 -- 172.21.15.37:0/2764473312 >> 172.21.15.37:0/2764473312 conn(0x7f21e00a1ad0 msgr2=0x7f21e00a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:40:02.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.866+0000 7f21e4ec0640 1 -- 172.21.15.37:0/2764473312 shutdown_connections 2024-04-26T00:40:02.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:02.866+0000 7f21e4ec0640 1 -- 172.21.15.37:0/2764473312 wait complete. 2024-04-26T00:40:05.882 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:40:06.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.014+0000 7f2dc2918640 10 monclient: get_monmap_and_config 2024-04-26T00:40:06.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.014+0000 7f2dc2918640 10 monclient: build_initial_monmap 2024-04-26T00:40:06.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.014+0000 7f2dc2918640 10 monclient: monmap: 2024-04-26T00:40:06.015 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:40:06.015 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:06.015 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:40:06.017791+0000 2024-04-26T00:40:06.015 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:40:06.017791+0000 2024-04-26T00:40:06.015 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:40:06.015 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:06.015 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:40:06.015 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:40:06.015 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:40:06.016 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:40:06.016 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:40:06.016 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:06.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.014+0000 7f2dc2918640 1 Processor -- start 2024-04-26T00:40:06.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.014+0000 7f2dc2918640 1 -- start start 2024-04-26T00:40:06.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.014+0000 7f2dc2918640 10 monclient: init 2024-04-26T00:40:06.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.014+0000 7f2dc2918640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:06.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.014+0000 7f2dc2918640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-04-26T00:40:06.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.014+0000 7f2dc2918640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2dbc0a0260 addr v1:172.21.15.37:6789/0 2024-04-26T00:40:06.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.014+0000 7f2dc2918640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2dbc137e40 0x7f2dbc13a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:06.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.014+0000 7f2dc2918640 10 monclient(hunting): picked mon.noname-b con 0x7f2dbc137e40 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:40:06.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.014+0000 7f2dc2918640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f2dbc13a770 0x7f2dbc13cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:06.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.014+0000 7f2dc2918640 10 monclient(hunting): picked mon.noname-a con 0x7f2dbc13a770 addr v2:172.21.15.37:3300/0 2024-04-26T00:40:06.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.014+0000 7f2dc2918640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2dbc0a1130 con 0x7f2dbc0a0260 2024-04-26T00:40:06.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.014+0000 7f2dbb7fe640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2dbc137e40 msgr2=0x7f2dbc13a230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:40:06.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.014+0000 7f2dbb7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2dbc137e40 0x7f2dbc13a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:40:06.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.014+0000 7f2dc2918640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:06.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.014+0000 7f2dc2918640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f2dbc0a0d50 con 0x7f2dbc13a770 2024-04-26T00:40:06.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.014+0000 7f2dc2918640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:06.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.014+0000 7f2dc2918640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2dbc09ec10 con 0x7f2dbc137e40 2024-04-26T00:40:06.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.014+0000 7f2dc2918640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:06.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.014+0000 7f2dc2918640 10 monclient(hunting): authenticate will time out at 2187.323742s 2024-04-26T00:40:06.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.014+0000 7f2dbbfff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f2dbc0a0260 0x7f2dbc0a0640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:43434/0 (socket says 172.21.15.37:43434) 2024-04-26T00:40:06.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.014+0000 7f2dbbfff640 1 -- 172.21.15.37:0/2958359868 learned_addr learned my addr 172.21.15.37:0/2958359868 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:40:06.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.014+0000 7f2dc0eb5640 1 --2- 172.21.15.37:0/2958359868 >> v2:172.21.15.37:3300/0 conn(0x7f2dbc13a770 0x7f2dbc13cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:06.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.014+0000 7f2dc0eb5640 10 monclient(hunting): get_auth_request con 0x7f2dbc13a770 auth_method 0 2024-04-26T00:40:06.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.014+0000 7f2dc0eb5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:06.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.014+0000 7f2dc0eb5640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:06.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.014+0000 7f2dc0eb5640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:06.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.018+0000 7f2dc0eb5640 1 -- 172.21.15.37:0/2958359868 >> v2:172.21.15.37:3300/0 conn(0x7f2dbc13a770 msgr2=0x7f2dbc13cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:40:06.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.018+0000 7f2dc0eb5640 1 -- 172.21.15.37:0/2958359868 >> v2:172.21.15.37:3300/0 conn(0x7f2dbc13a770 msgr2=0x7f2dbc13cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:40:06.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.018+0000 7f2dc0eb5640 1 --2- 172.21.15.37:0/2958359868 >> v2:172.21.15.37:3300/0 conn(0x7f2dbc13a770 0x7f2dbc13cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:40:06.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.018+0000 7f2dc0eb5640 1 --2- 172.21.15.37:0/2958359868 >> v2:172.21.15.37:3300/0 conn(0x7f2dbc13a770 0x7f2dbc13cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:40:06.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.018+0000 7f2dbbfff640 1 -- 172.21.15.37:0/2958359868 >> v1:172.21.15.37:6789/0 conn(0x7f2dbc0a0260 legacy=0x7f2dbc0a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:06.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.218+0000 7f2dbb7fe640 1 -- 172.21.15.37:0/2958359868 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2dbc137e40 msgr2=0x7f2dbc13a230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:40:06.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.218+0000 7f2dbb7fe640 1 --2- 172.21.15.37:0/2958359868 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2dbc137e40 0x7f2dbc13a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:40:06.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.222+0000 7f2dc0eb5640 1 --2- 172.21.15.37:0/2958359868 >> v2:172.21.15.37:3300/0 conn(0x7f2dbc13a770 0x7f2dbc13cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:06.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.222+0000 7f2dc0eb5640 10 monclient(hunting): get_auth_request con 0x7f2dbc13a770 auth_method 0 2024-04-26T00:40:06.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.222+0000 7f2dc0eb5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:06.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.222+0000 7f2dc0eb5640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:06.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.222+0000 7f2dc0eb5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:40:06.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.222+0000 7f2dc0eb5640 1 -- 172.21.15.37:0/2958359868 >> v2:172.21.15.37:3300/0 conn(0x7f2dbc13a770 msgr2=0x7f2dbc13cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:40:06.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.222+0000 7f2dc0eb5640 1 -- 172.21.15.37:0/2958359868 >> v2:172.21.15.37:3300/0 conn(0x7f2dbc13a770 msgr2=0x7f2dbc13cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:40:06.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.222+0000 7f2dc0eb5640 1 --2- 172.21.15.37:0/2958359868 >> v2:172.21.15.37:3300/0 conn(0x7f2dbc13a770 0x7f2dbc13cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:40:06.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.222+0000 7f2dc0eb5640 1 --2- 172.21.15.37:0/2958359868 >> v2:172.21.15.37:3300/0 conn(0x7f2dbc13a770 0x7f2dbc13cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:40:06.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.622+0000 7f2dbb7fe640 1 -- 172.21.15.37:0/2958359868 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2dbc137e40 msgr2=0x7f2dbc13a230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:40:06.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.622+0000 7f2dbb7fe640 1 --2- 172.21.15.37:0/2958359868 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2dbc137e40 0x7f2dbc13a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:40:06.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.626+0000 7f2dc0eb5640 1 --2- 172.21.15.37:0/2958359868 >> v2:172.21.15.37:3300/0 conn(0x7f2dbc13a770 0x7f2dbc13cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:06.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.626+0000 7f2dc0eb5640 10 monclient(hunting): get_auth_request con 0x7f2dbc13a770 auth_method 0 2024-04-26T00:40:06.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.626+0000 7f2dc0eb5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:06.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.626+0000 7f2dc0eb5640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:06.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.626+0000 7f2dc0eb5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:40:06.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.626+0000 7f2dc0eb5640 1 -- 172.21.15.37:0/2958359868 >> v2:172.21.15.37:3300/0 conn(0x7f2dbc13a770 msgr2=0x7f2dbc13cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:40:06.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.626+0000 7f2dc0eb5640 1 -- 172.21.15.37:0/2958359868 >> v2:172.21.15.37:3300/0 conn(0x7f2dbc13a770 msgr2=0x7f2dbc13cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:40:06.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.626+0000 7f2dc0eb5640 1 --2- 172.21.15.37:0/2958359868 >> v2:172.21.15.37:3300/0 conn(0x7f2dbc13a770 0x7f2dbc13cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:40:06.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:06.626+0000 7f2dc0eb5640 1 --2- 172.21.15.37:0/2958359868 >> v2:172.21.15.37:3300/0 conn(0x7f2dbc13a770 0x7f2dbc13cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:40:07.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:07.426+0000 7f2dbb7fe640 1 -- 172.21.15.37:0/2958359868 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2dbc137e40 msgr2=0x7f2dbc13a230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:40:07.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:07.426+0000 7f2dbb7fe640 1 --2- 172.21.15.37:0/2958359868 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2dbc137e40 0x7f2dbc13a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:40:07.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:07.430+0000 7f2dc0eb5640 1 --2- 172.21.15.37:0/2958359868 >> v2:172.21.15.37:3300/0 conn(0x7f2dbc13a770 0x7f2dbc13cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:07.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:07.430+0000 7f2dc0eb5640 10 monclient(hunting): get_auth_request con 0x7f2dbc13a770 auth_method 0 2024-04-26T00:40:07.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:07.430+0000 7f2dc0eb5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:07.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:07.430+0000 7f2dc0eb5640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:07.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:07.430+0000 7f2dc0eb5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:40:07.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:07.430+0000 7f2dc0eb5640 1 -- 172.21.15.37:0/2958359868 >> v2:172.21.15.37:3300/0 conn(0x7f2dbc13a770 msgr2=0x7f2dbc13cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:40:07.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:07.430+0000 7f2dc0eb5640 1 -- 172.21.15.37:0/2958359868 >> v2:172.21.15.37:3300/0 conn(0x7f2dbc13a770 msgr2=0x7f2dbc13cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:40:07.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:07.430+0000 7f2dc0eb5640 1 --2- 172.21.15.37:0/2958359868 >> v2:172.21.15.37:3300/0 conn(0x7f2dbc13a770 0x7f2dbc13cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:40:07.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:07.430+0000 7f2dc0eb5640 1 --2- 172.21.15.37:0/2958359868 >> v2:172.21.15.37:3300/0 conn(0x7f2dbc13a770 0x7f2dbc13cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:40:09.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.014+0000 7f2db9ffb640 10 monclient(hunting): tick 2024-04-26T00:40:09.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.014+0000 7f2db9ffb640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:40:09.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.014+0000 7f2db9ffb640 1 monclient(hunting): continuing hunt 2024-04-26T00:40:09.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.014+0000 7f2db9ffb640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:40:09.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.014+0000 7f2db9ffb640 1 -- 172.21.15.37:0/2958359868 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2dbc137e40 msgr2=0x7f2dbc13a230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:40:09.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.014+0000 7f2db9ffb640 1 --2- 172.21.15.37:0/2958359868 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2dbc137e40 0x7f2dbc13a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:09.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.014+0000 7f2db9ffb640 1 -- 172.21.15.37:0/2958359868 >> v2:172.21.15.37:3300/0 conn(0x7f2dbc13a770 msgr2=0x7f2dbc13cb60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:40:09.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.014+0000 7f2db9ffb640 1 --2- 172.21.15.37:0/2958359868 >> v2:172.21.15.37:3300/0 conn(0x7f2dbc13a770 0x7f2dbc13cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:09.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.014+0000 7f2db9ffb640 1 -- 172.21.15.37:0/2958359868 >> v1:172.21.15.37:6789/0 conn(0x7f2dbc0a0260 legacy=0x7f2dbc0a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:09.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.014+0000 7f2db9ffb640 10 monclient: _add_conns ranks=[4,1] 2024-04-26T00:40:09.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.014+0000 7f2db9ffb640 1 --2- 172.21.15.37:0/2958359868 >> v2:172.21.15.45:3300/0 conn(0x7f2da8003770 0x7f2da8005bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:09.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.014+0000 7f2db9ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f2da8003770 addr v2:172.21.15.45:3300/0 2024-04-26T00:40:09.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.014+0000 7f2db9ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2da80062c0 addr v1:172.21.15.45:6789/0 2024-04-26T00:40:09.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.014+0000 7f2db9ffb640 1 -- 172.21.15.37:0/2958359868 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2dbc06cff0 con 0x7f2da80062c0 2024-04-26T00:40:09.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.014+0000 7f2db9ffb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:09.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.014+0000 7f2db9ffb640 1 -- 172.21.15.37:0/2958359868 --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f2dbc0a0d50 con 0x7f2da8003770 2024-04-26T00:40:09.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.014+0000 7f2db9ffb640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:09.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.018+0000 7f2dbbfff640 1 --2- 172.21.15.37:0/2958359868 >> v2:172.21.15.45:3300/0 conn(0x7f2da8003770 0x7f2da8005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:09.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.018+0000 7f2dbbfff640 10 monclient(hunting): get_auth_request con 0x7f2da8003770 auth_method 0 2024-04-26T00:40:09.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.018+0000 7f2dbbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:09.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.018+0000 7f2dbbfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:09.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.018+0000 7f2dbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:09.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.018+0000 7f2dbbfff640 1 -- 172.21.15.37:0/2958359868 >> v2:172.21.15.45:3300/0 conn(0x7f2da8003770 msgr2=0x7f2da8005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:40:09.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.018+0000 7f2dbbfff640 1 -- 172.21.15.37:0/2958359868 >> v2:172.21.15.45:3300/0 conn(0x7f2da8003770 msgr2=0x7f2da8005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:40:09.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.018+0000 7f2dbbfff640 1 --2- 172.21.15.37:0/2958359868 >> v2:172.21.15.45:3300/0 conn(0x7f2da8003770 0x7f2da8005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:40:09.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.018+0000 7f2dbbfff640 1 --2- 172.21.15.37:0/2958359868 >> v2:172.21.15.45:3300/0 conn(0x7f2da8003770 0x7f2da8005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:40:09.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.018+0000 7f2dbb7fe640 1 -- 172.21.15.37:0/2958359868 >> v1:172.21.15.45:6789/0 conn(0x7f2da80062c0 legacy=0x7f2da8008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:09.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.222+0000 7f2dbbfff640 1 --2- 172.21.15.37:0/2958359868 >> v2:172.21.15.45:3300/0 conn(0x7f2da8003770 0x7f2da8005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:09.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.222+0000 7f2dbbfff640 10 monclient(hunting): get_auth_request con 0x7f2da8003770 auth_method 0 2024-04-26T00:40:09.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.222+0000 7f2dbbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:09.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.222+0000 7f2dbbfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:09.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.222+0000 7f2dbbfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:40:09.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.222+0000 7f2dbbfff640 1 -- 172.21.15.37:0/2958359868 >> v2:172.21.15.45:3300/0 conn(0x7f2da8003770 msgr2=0x7f2da8005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:40:09.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.222+0000 7f2dbbfff640 1 -- 172.21.15.37:0/2958359868 >> v2:172.21.15.45:3300/0 conn(0x7f2da8003770 msgr2=0x7f2da8005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:40:09.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.222+0000 7f2dbbfff640 1 --2- 172.21.15.37:0/2958359868 >> v2:172.21.15.45:3300/0 conn(0x7f2da8003770 0x7f2da8005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:40:09.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.222+0000 7f2dbbfff640 1 --2- 172.21.15.37:0/2958359868 >> v2:172.21.15.45:3300/0 conn(0x7f2da8003770 0x7f2da8005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:40:09.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.626+0000 7f2dbbfff640 1 --2- 172.21.15.37:0/2958359868 >> v2:172.21.15.45:3300/0 conn(0x7f2da8003770 0x7f2da8005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:09.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.626+0000 7f2dbbfff640 10 monclient(hunting): get_auth_request con 0x7f2da8003770 auth_method 0 2024-04-26T00:40:09.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.626+0000 7f2dbbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:09.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.626+0000 7f2dbbfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:09.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.626+0000 7f2dbbfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:40:09.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.626+0000 7f2dbbfff640 1 -- 172.21.15.37:0/2958359868 >> v2:172.21.15.45:3300/0 conn(0x7f2da8003770 msgr2=0x7f2da8005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:40:09.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.626+0000 7f2dbbfff640 1 -- 172.21.15.37:0/2958359868 >> v2:172.21.15.45:3300/0 conn(0x7f2da8003770 msgr2=0x7f2da8005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:40:09.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.626+0000 7f2dbbfff640 1 --2- 172.21.15.37:0/2958359868 >> v2:172.21.15.45:3300/0 conn(0x7f2da8003770 0x7f2da8005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:40:09.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:09.626+0000 7f2dbbfff640 1 --2- 172.21.15.37:0/2958359868 >> v2:172.21.15.45:3300/0 conn(0x7f2da8003770 0x7f2da8005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:40:10.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.430+0000 7f2dbbfff640 1 --2- 172.21.15.37:0/2958359868 >> v2:172.21.15.45:3300/0 conn(0x7f2da8003770 0x7f2da8005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:10.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.430+0000 7f2dbbfff640 10 monclient(hunting): get_auth_request con 0x7f2da8003770 auth_method 0 2024-04-26T00:40:10.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.430+0000 7f2dbbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:10.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.430+0000 7f2dbbfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:10.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.430+0000 7f2dbbfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:40:10.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.430+0000 7f2dbbfff640 1 -- 172.21.15.37:0/2958359868 >> v2:172.21.15.45:3300/0 conn(0x7f2da8003770 msgr2=0x7f2da8005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:40:10.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.430+0000 7f2dbbfff640 1 -- 172.21.15.37:0/2958359868 >> v2:172.21.15.45:3300/0 conn(0x7f2da8003770 msgr2=0x7f2da8005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:40:10.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.430+0000 7f2dbbfff640 1 --2- 172.21.15.37:0/2958359868 >> v2:172.21.15.45:3300/0 conn(0x7f2da8003770 0x7f2da8005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:40:10.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.430+0000 7f2dbbfff640 1 --2- 172.21.15.37:0/2958359868 >> v2:172.21.15.45:3300/0 conn(0x7f2da8003770 0x7f2da8005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:40:10.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.534+0000 7f2dbaffd640 1 -- 172.21.15.37:0/2958359868 <== mon.1 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4233941337 0 0) 0x7f2dbc06cff0 con 0x7f2da80062c0 2024-04-26T00:40:10.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.534+0000 7f2dbaffd640 10 monclient(hunting): _negotiate 2024-04-26T00:40:10.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.534+0000 7f2dbaffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:10.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.534+0000 7f2dbaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:10.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.534+0000 7f2dbaffd640 10 monclient(hunting): my global_id is 224320 2024-04-26T00:40:10.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.534+0000 7f2dbaffd640 1 -- 172.21.15.37:0/2958359868 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2da4003530 con 0x7f2da80062c0 2024-04-26T00:40:10.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbaffd640 1 -- 172.21.15.37:0/2958359868 <== mon.1 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 442357272 0 0) 0x7f2da4003530 con 0x7f2da80062c0 2024-04-26T00:40:10.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbaffd640 1 -- 172.21.15.37:0/2958359868 >> v2:172.21.15.45:3300/0 conn(0x7f2da8003770 msgr2=0x7f2da8005bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:40:10.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbaffd640 1 --2- 172.21.15.37:0/2958359868 >> v2:172.21.15.45:3300/0 conn(0x7f2da8003770 0x7f2da8005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:10.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbaffd640 10 monclient: _finish_hunting 0 2024-04-26T00:40:10.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbaffd640 1 monclient: found mon.noname-c-legacy 2024-04-26T00:40:10.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:10.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbaffd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.45:6789/0 2024-04-26T00:40:10.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbaffd640 1 -- 172.21.15.37:0/2958359868 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2da800b070 con 0x7f2da80062c0 2024-04-26T00:40:10.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbaffd640 10 monclient: _finish_auth 0 2024-04-26T00:40:10.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbaffd640 10 monclient: _check_auth_tickets 2024-04-26T00:40:10.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:10.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbaffd640 1 -- 172.21.15.37:0/2958359868 <== mon.1 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 866326386 0 0) 0x7f2db00045d0 con 0x7f2da80062c0 2024-04-26T00:40:10.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:10.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbaffd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-04-26T00:40:10.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbaffd640 10 monclient: dump: 2024-04-26T00:40:10.541 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:10.541 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:10.541 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:10.541 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:10.541 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:10.541 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:10.541 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:10.541 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:10.541 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:10.542 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:10.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbaffd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] but i'm connected to v1:172.21.15.45:6789/0, reconnecting 2024-04-26T00:40:10.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbaffd640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:10.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbaffd640 1 -- 172.21.15.37:0/2958359868 >> v1:172.21.15.45:6789/0 conn(0x7f2da80062c0 legacy=0x7f2da8008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:10.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dc0eb5640 1 -- 172.21.15.37:0/2958359868 reap_dead start 2024-04-26T00:40:10.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbaffd640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:40:10.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbaffd640 1 --2- 172.21.15.37:0/2958359868 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2da40039c0 0x7f2da4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:10.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbaffd640 10 monclient(hunting): picked mon.c con 0x7f2da40039c0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:40:10.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbaffd640 1 --2- 172.21.15.37:0/2958359868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2da4004810 0x7f2da4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:10.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbaffd640 10 monclient(hunting): picked mon.b con 0x7f2da4004810 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:40:10.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbbfff640 1 -- 172.21.15.37:0/2958359868 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2da40039c0 msgr2=0x7f2da4003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:40:10.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbaffd640 1 --2- 172.21.15.37:0/2958359868 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2da4005220 0x7f2da400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:10.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbbfff640 1 --2- 172.21.15.37:0/2958359868 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2da40039c0 0x7f2da4003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:40:10.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbaffd640 10 monclient(hunting): picked mon.a con 0x7f2da4005220 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:40:10.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbaffd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:10.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbaffd640 1 -- 172.21.15.37:0/2958359868 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f2dbc0a0d50 con 0x7f2da4005220 2024-04-26T00:40:10.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbaffd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:10.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbaffd640 1 -- 172.21.15.37:0/2958359868 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2da400c150 con 0x7f2da4004810 2024-04-26T00:40:10.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbaffd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:10.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbaffd640 1 -- 172.21.15.37:0/2958359868 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2da400c350 con 0x7f2da40039c0 2024-04-26T00:40:10.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbaffd640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:10.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbaffd640 10 monclient(hunting): _finish_auth 0 2024-04-26T00:40:10.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbb7fe640 1 --2- 172.21.15.37:0/2958359868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2da4004810 0x7f2da4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:10.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbb7fe640 10 monclient(hunting): get_auth_request con 0x7f2da4004810 auth_method 0 2024-04-26T00:40:10.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:10.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:10.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:40:10.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:10.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:10.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:40:10.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbb7fe640 10 monclient(hunting): handle_auth_done global_id 224320 payload 293 2024-04-26T00:40:10.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbb7fe640 1 -- 172.21.15.37:0/2958359868 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2da40039c0 msgr2=0x7f2da4003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:40:10.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbb7fe640 1 --2- 172.21.15.37:0/2958359868 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2da40039c0 0x7f2da4003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:10.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbb7fe640 1 -- 172.21.15.37:0/2958359868 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2da4005220 msgr2=0x7f2da400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:40:10.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbb7fe640 1 --2- 172.21.15.37:0/2958359868 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2da4005220 0x7f2da400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:10.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbb7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:40:10.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbb7fe640 1 monclient: found mon.b 2024-04-26T00:40:10.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:10.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbb7fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.45:3300/0 2024-04-26T00:40:10.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbb7fe640 1 -- 172.21.15.37:0/2958359868 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2da4003530 con 0x7f2da4004810 2024-04-26T00:40:10.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbb7fe640 10 monclient: _finish_auth 0 2024-04-26T00:40:10.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbb7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:40:10.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dbb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:10.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dc2918640 5 monclient: authenticate success, global_id 224320 2024-04-26T00:40:10.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.538+0000 7f2dc2918640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:40:10.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.546+0000 7f2dbb7fe640 1 -- 172.21.15.37:0/2958359868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2da4004810 msgr2=0x7f2da4004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:10.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.546+0000 7f2dbb7fe640 1 --2- 172.21.15.37:0/2958359868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2da4004810 0x7f2da4004c90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2db00047f0 tx=0x7f2db001f670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6b9ddf42e51b2ec server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:10.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.546+0000 7f2dbaffd640 1 -- 172.21.15.37:0/2958359868 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2db000c200 con 0x7f2da4004810 2024-04-26T00:40:10.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.546+0000 7f2dbaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:10.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.546+0000 7f2dbaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:40:10.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.546+0000 7f2dbaffd640 10 monclient: dump: 2024-04-26T00:40:10.548 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:10.548 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:10.548 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:10.549 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:10.549 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:10.549 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:10.549 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:10.549 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:10.549 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:10.549 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:10.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.546+0000 7f2dbaffd640 1 -- 172.21.15.37:0/2958359868 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2db0004be0 con 0x7f2da4004810 2024-04-26T00:40:10.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.546+0000 7f2dc2918640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:40:10.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.546+0000 7f2dbaffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:40:10.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.546+0000 7f2dc2918640 10 monclient: get_monmap_and_config success 2024-04-26T00:40:10.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.546+0000 7f2dbaffd640 1 -- 172.21.15.37:0/2958359868 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2db00060d0 con 0x7f2da4004810 2024-04-26T00:40:10.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.546+0000 7f2dbaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:10.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.546+0000 7f2dc2918640 4 set_mon_vals no callback set 2024-04-26T00:40:10.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.546+0000 7f2dbaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:40:10.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.546+0000 7f2dbaffd640 10 monclient: dump: 2024-04-26T00:40:10.550 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:10.550 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:10.550 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:10.550 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:10.550 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:10.550 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:10.550 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:10.550 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:10.550 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:10.550 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:10.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.550+0000 7f2dc2918640 10 monclient: shutdown 2024-04-26T00:40:10.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.550+0000 7f2dc2918640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:40:10.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.550+0000 7f2dc2918640 1 -- 172.21.15.37:0/2958359868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2da4004810 msgr2=0x7f2da4004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:10.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.550+0000 7f2dc2918640 1 --2- 172.21.15.37:0/2958359868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2da4004810 0x7f2da4004c90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2db00047f0 tx=0x7f2db001f670 comp rx=0 tx=0).stop 2024-04-26T00:40:10.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.550+0000 7f2dc2918640 1 -- 172.21.15.37:0/2958359868 shutdown_connections 2024-04-26T00:40:10.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.550+0000 7f2dc2918640 1 --2- 172.21.15.37:0/2958359868 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2da4005220 0x7f2da400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:10.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.550+0000 7f2dc2918640 1 --2- 172.21.15.37:0/2958359868 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2da4004810 0x7f2da4004c90 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:10.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.550+0000 7f2dc2918640 1 --2- 172.21.15.37:0/2958359868 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2da40039c0 0x7f2da4003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:10.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.550+0000 7f2dc2918640 1 -- 172.21.15.37:0/2958359868 >> 172.21.15.37:0/2958359868 conn(0x7f2dbc0a1ad0 msgr2=0x7f2dbc0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:40:10.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.550+0000 7f2dc2918640 1 -- 172.21.15.37:0/2958359868 shutdown_connections 2024-04-26T00:40:10.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.550+0000 7f2dc2918640 1 -- 172.21.15.37:0/2958359868 wait complete. 2024-04-26T00:40:10.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.550+0000 7f2dc2918640 10 monclient: build_initial_monmap 2024-04-26T00:40:10.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.550+0000 7f2dc2918640 10 monclient: monmap: 2024-04-26T00:40:10.553 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:40:10.553 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:40:10.553 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:40:10.553 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:40:10.553 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:40:10.553 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:40:10.553 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:40:10.553 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:40:10.553 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:40:10.553 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:10.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dc2918640 1 Processor -- start 2024-04-26T00:40:10.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dc2918640 1 -- start start 2024-04-26T00:40:10.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dc2918640 10 monclient: init 2024-04-26T00:40:10.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dc2918640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:10.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dc2918640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:40:10.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dc2918640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2da40039c0 0x7f2dbc134bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:10.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dc2918640 10 monclient(hunting): picked mon.noname-a con 0x7f2da40039c0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:40:10.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dc2918640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2da4004810 0x7f2dbc135110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:10.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dc2918640 10 monclient(hunting): picked mon.noname-c con 0x7f2da4004810 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:40:10.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dc2918640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2da4005220 0x7f2dbc09c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:10.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dc2918640 10 monclient(hunting): picked mon.noname-b con 0x7f2da4005220 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:40:10.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dc2918640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:10.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dc2918640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f2dbc09f2b0 con 0x7f2da40039c0 2024-04-26T00:40:10.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dbb7fe640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2da4004810 msgr2=0x7f2dbc135110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:40:10.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dc2918640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:10.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dc2918640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2dbc09f0f0 con 0x7f2da4005220 2024-04-26T00:40:10.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dc2918640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:10.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dbb7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2da4004810 0x7f2dbc135110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:40:10.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dc2918640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2dbc09f430 con 0x7f2da4004810 2024-04-26T00:40:10.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dc2918640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:10.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dbbfff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2da40039c0 0x7f2dbc134bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:10.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dc2918640 10 monclient(hunting): authenticate will time out at 2191.862101s 2024-04-26T00:40:10.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dbbfff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2da40039c0 0x7f2dbc134bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:57752/0 (socket says 172.21.15.37:57752) 2024-04-26T00:40:10.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dbbfff640 1 -- 172.21.15.37:0/2066265821 learned_addr learned my addr 172.21.15.37:0/2066265821 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:40:10.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dbbfff640 10 monclient(hunting): get_auth_request con 0x7f2da40039c0 auth_method 0 2024-04-26T00:40:10.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dbbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:10.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dbbfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:10.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:10.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dbbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:10.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dbbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:10.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dbbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:10.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dc0eb5640 1 --2- 172.21.15.37:0/2066265821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2da4005220 0x7f2dbc09c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:10.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dc0eb5640 10 monclient(hunting): get_auth_request con 0x7f2da4005220 auth_method 0 2024-04-26T00:40:10.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dc0eb5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:10.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dc0eb5640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:10.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dc0eb5640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:10.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dbbfff640 10 monclient(hunting): handle_auth_done global_id 244155 payload 824 2024-04-26T00:40:10.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dbbfff640 1 -- 172.21.15.37:0/2066265821 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2da4004810 msgr2=0x7f2dbc135110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:40:10.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dbbfff640 1 --2- 172.21.15.37:0/2066265821 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2da4004810 0x7f2dbc135110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:10.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dbbfff640 1 -- 172.21.15.37:0/2066265821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2da4005220 msgr2=0x7f2dbc09c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:10.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dbbfff640 1 --2- 172.21.15.37:0/2066265821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2da4005220 0x7f2dbc09c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:10.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dbbfff640 10 monclient: _finish_hunting 0 2024-04-26T00:40:10.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dbbfff640 1 monclient: found mon.noname-a 2024-04-26T00:40:10.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dbbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:10.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dbbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:40:10.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.554+0000 7f2dbbfff640 1 -- 172.21.15.37:0/2066265821 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dbc09b4f0 con 0x7f2da40039c0 2024-04-26T00:40:10.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.558+0000 7f2dbbfff640 1 -- 172.21.15.37:0/2066265821 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2da40039c0 msgr2=0x7f2dbc134bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:10.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.558+0000 7f2dbbfff640 1 --2- 172.21.15.37:0/2066265821 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2da40039c0 0x7f2dbc134bd0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2dac007440 tx=0x7f2dac0015b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:10.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.558+0000 7f2db8ff9640 1 -- 172.21.15.37:0/2066265821 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dac004c10 con 0x7f2da40039c0 2024-04-26T00:40:10.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.558+0000 7f2db8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:10.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.558+0000 7f2db8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:40:10.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.558+0000 7f2db8ff9640 10 monclient: dump: 2024-04-26T00:40:10.562 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:10.566 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:10.566 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:10.566 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:10.566 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:10.566 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:10.566 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:10.566 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:10.566 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:10.566 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:10.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.558+0000 7f2db8ff9640 10 monclient: _finish_auth 0 2024-04-26T00:40:10.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.558+0000 7f2db8ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:40:10.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.558+0000 7f2db8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:10.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.558+0000 7f2db8ff9640 1 -- 172.21.15.37:0/2066265821 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dac004dd0 con 0x7f2da40039c0 2024-04-26T00:40:10.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.558+0000 7f2db8ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:40:10.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.558+0000 7f2db8ff9640 1 -- 172.21.15.37:0/2066265821 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dac005290 con 0x7f2da40039c0 2024-04-26T00:40:10.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.558+0000 7f2dc2918640 5 monclient: authenticate success, global_id 244155 2024-04-26T00:40:10.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.558+0000 7f2db8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:10.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.558+0000 7f2db8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:40:10.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.558+0000 7f2db8ff9640 10 monclient: dump: 2024-04-26T00:40:10.567 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:10.567 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:10.567 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:10.567 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:10.567 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:10.567 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:10.567 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:10.568 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:10.568 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:10.568 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:10.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.558+0000 7f2dc2918640 10 monclient: _renew_subs 2024-04-26T00:40:10.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.558+0000 7f2dc2918640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:40:10.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.558+0000 7f2dc2918640 1 -- 172.21.15.37:0/2066265821 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dbc122d80 con 0x7f2da40039c0 2024-04-26T00:40:10.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.558+0000 7f2dba7fc640 4 set_mon_vals no callback set 2024-04-26T00:40:10.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.558+0000 7f2dc2918640 10 monclient: _renew_subs 2024-04-26T00:40:10.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.558+0000 7f2dc2918640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:40:10.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.558+0000 7f2dc2918640 1 -- 172.21.15.37:0/2066265821 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dbc0729d0 con 0x7f2da40039c0 2024-04-26T00:40:10.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.558+0000 7f2dc2918640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:10.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.558+0000 7f2dc2918640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:10.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.558+0000 7f2dc2918640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:40:10.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.558+0000 7f2dc2918640 1 -- 172.21.15.37:0/2066265821 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dbc066f40 con 0x7f2da40039c0 2024-04-26T00:40:10.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.558+0000 7f2db8ff9640 1 -- 172.21.15.37:0/2066265821 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2dac005430 con 0x7f2da40039c0 2024-04-26T00:40:10.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.558+0000 7f2db8ff9640 1 --2- 172.21.15.37:0/2066265821 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2da80368b0 0x7f2da8038d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:10.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.558+0000 7f2db8ff9640 1 -- 172.21.15.37:0/2066265821 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(792..792 src has 705..792) ==== 3959+0+0 (secure 0 0 0) 0x7f2dac049640 con 0x7f2da40039c0 2024-04-26T00:40:10.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.558+0000 7f2dbb7fe640 1 --2- 172.21.15.37:0/2066265821 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2da80368b0 0x7f2da8038d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:10.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.558+0000 7f2dbb7fe640 10 monclient: get_auth_request con 0x7f2da80368b0 auth_method 0 2024-04-26T00:40:10.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.562+0000 7f2dbb7fe640 1 --2- 172.21.15.37:0/2066265821 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2da80368b0 0x7f2da8038d70 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f2db00ac000 tx=0x7f2db00aa000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:10.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.566+0000 7f2db8ff9640 1 -- 172.21.15.37:0/2066265821 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f2dbc09b4f0 con 0x7f2da40039c0 2024-04-26T00:40:10.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.566+0000 7f2db8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:10.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.566+0000 7f2db8ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:40:10.716 INFO:tasks.rados.rados.0.smithi045.stdout:3496: done (2 left) 2024-04-26T00:40:10.717 INFO:tasks.rados.rados.0.smithi045.stdout:3497: done (1 left) 2024-04-26T00:40:10.717 INFO:tasks.rados.rados.0.smithi045.stdout:3498: done (0 left) 2024-04-26T00:40:10.717 INFO:tasks.rados.rados.0.smithi045.stdout:3499: write oid 15 current snap is 388 2024-04-26T00:40:10.717 INFO:tasks.rados.rados.0.smithi045.stdout:3499: seq_num 1246 ranges {541522=725793,1946446=635431} 2024-04-26T00:40:10.737 INFO:tasks.rados.rados.0.smithi045.stdout:3499: writing smithi04526579-15 from 541522 to 1267315 tid 1 2024-04-26T00:40:10.759 INFO:tasks.rados.rados.0.smithi045.stdout:3499: writing smithi04526579-15 from 1946446 to 2581877 tid 2 2024-04-26T00:40:10.759 INFO:tasks.rados.rados.0.smithi045.stdout:3500: read oid 45 snap 362 2024-04-26T00:40:10.759 INFO:tasks.rados.rados.0.smithi045.stdout:3500: expect (ObjNum 1062 snap 318 seq_num 1062) 2024-04-26T00:40:10.759 INFO:tasks.rados.rados.0.smithi045.stdout:3501: read oid 38 snap -1 2024-04-26T00:40:10.760 INFO:tasks.rados.rados.0.smithi045.stdout:3501: expect (ObjNum 1075 snap 322 seq_num 1075) 2024-04-26T00:40:10.760 INFO:tasks.rados.rados.0.smithi045.stdout:3502: write (excl) oid 29 current snap is 388 2024-04-26T00:40:10.760 INFO:tasks.rados.rados.0.smithi045.stdout:3499: finishing write tid 1 to smithi04526579-15 2024-04-26T00:40:10.760 INFO:tasks.rados.rados.0.smithi045.stdout:3502: seq_num 1247 ranges {558329=517395,1721281=779089} 2024-04-26T00:40:10.775 INFO:tasks.rados.rados.0.smithi045.stdout:3502: writing smithi04526579-29 from 558329 to 1075724 tid 1 2024-04-26T00:40:10.791 INFO:tasks.rados.rados.0.smithi045.stdout:3502: writing smithi04526579-29 from 1721281 to 2500370 tid 2 2024-04-26T00:40:10.791 INFO:tasks.rados.rados.0.smithi045.stdout:3503: read oid 27 snap 373 2024-04-26T00:40:10.791 INFO:tasks.rados.rados.0.smithi045.stdout:3503: expect deleted 2024-04-26T00:40:10.791 INFO:tasks.rados.rados.0.smithi045.stdout:3504: snap_remove snap 375 2024-04-26T00:40:10.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.794+0000 7f2dc2918640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:40:10.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.794+0000 7f2dc2918640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:40:10.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.794+0000 7f2dc2918640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:40:10.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.794+0000 7f2dc2918640 1 -- 172.21.15.37:0/2066265821 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2dbc121930 con 0x7f2da40039c0 2024-04-26T00:40:10.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.794+0000 7f2db8ff9640 1 -- 172.21.15.37:0/2066265821 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f2dbc0a1130 con 0x7f2da40039c0 2024-04-26T00:40:10.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.794+0000 7f2db8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:40:10.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.794+0000 7f2db8ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:40:10.797 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":150,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:40:10.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.798+0000 7f2dc2918640 1 -- 172.21.15.37:0/2066265821 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2da80368b0 msgr2=0x7f2da8038d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:10.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.798+0000 7f2dc2918640 1 --2- 172.21.15.37:0/2066265821 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2da80368b0 0x7f2da8038d70 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f2db00ac000 tx=0x7f2db00aa000 comp rx=0 tx=0).stop 2024-04-26T00:40:10.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.798+0000 7f2dc2918640 10 monclient: shutdown 2024-04-26T00:40:10.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.798+0000 7f2dc2918640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:40:10.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.798+0000 7f2dc2918640 1 -- 172.21.15.37:0/2066265821 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2da40039c0 msgr2=0x7f2dbc134bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:10.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.798+0000 7f2dc2918640 1 --2- 172.21.15.37:0/2066265821 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2da40039c0 0x7f2dbc134bd0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2dac007440 tx=0x7f2dac0015b0 comp rx=0 tx=0).stop 2024-04-26T00:40:10.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.798+0000 7f2dc2918640 1 -- 172.21.15.37:0/2066265821 shutdown_connections 2024-04-26T00:40:10.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.798+0000 7f2dc2918640 1 --2- 172.21.15.37:0/2066265821 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2da80368b0 0x7f2da8038d70 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:10.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.798+0000 7f2dc2918640 1 --2- 172.21.15.37:0/2066265821 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2da4005220 0x7f2dbc09c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:10.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.798+0000 7f2dc2918640 1 --2- 172.21.15.37:0/2066265821 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2da4004810 0x7f2dbc135110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:10.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.798+0000 7f2dc2918640 1 --2- 172.21.15.37:0/2066265821 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2da40039c0 0x7f2dbc134bd0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:10.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.798+0000 7f2dc2918640 1 -- 172.21.15.37:0/2066265821 >> 172.21.15.37:0/2066265821 conn(0x7f2dbc0a1ad0 msgr2=0x7f2dbc139a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:40:10.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.798+0000 7f2dc2918640 1 -- 172.21.15.37:0/2066265821 shutdown_connections 2024-04-26T00:40:10.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.798+0000 7f2dc2918640 1 -- 172.21.15.37:0/2066265821 wait complete. 2024-04-26T00:40:10.817 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-26T00:40:10.817 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-04-26T00:40:10.817 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:40:10.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.934+0000 7fa7dc119640 10 monclient: get_monmap_and_config 2024-04-26T00:40:10.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.934+0000 7fa7dc119640 10 monclient: build_initial_monmap 2024-04-26T00:40:10.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.934+0000 7fa7dc119640 10 monclient: monmap: 2024-04-26T00:40:10.933 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:40:10.933 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:10.933 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:40:10.936047+0000 2024-04-26T00:40:10.933 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:40:10.936047+0000 2024-04-26T00:40:10.933 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:40:10.933 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:10.934 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:40:10.934 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:40:10.934 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:40:10.934 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:40:10.934 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:40:10.934 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:10.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.934+0000 7fa7dc119640 1 Processor -- start 2024-04-26T00:40:10.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.934+0000 7fa7dc119640 1 -- start start 2024-04-26T00:40:10.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.934+0000 7fa7dc119640 10 monclient: init 2024-04-26T00:40:10.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.934+0000 7fa7dc119640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:10.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.934+0000 7fa7dc119640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-04-26T00:40:10.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.934+0000 7fa7dc119640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7d40a3d90 0x7fa7d409bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:10.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.934+0000 7fa7dc119640 10 monclient(hunting): picked mon.noname-b con 0x7fa7d40a3d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:40:10.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.934+0000 7fa7dc119640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa7d40a4170 addr v1:172.21.15.45:6789/0 2024-04-26T00:40:10.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.934+0000 7fa7dc119640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa7d413c3e0 addr v1:172.21.15.37:6789/0 2024-04-26T00:40:10.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.934+0000 7fa7dc119640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7d40a5150 con 0x7fa7d413c3e0 2024-04-26T00:40:10.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.934+0000 7fa7dc119640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7d409b280 con 0x7fa7d40a4170 2024-04-26T00:40:10.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.934+0000 7fa7dc119640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:10.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.934+0000 7fa7dc119640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa7d409b560 con 0x7fa7d40a3d90 2024-04-26T00:40:10.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.934+0000 7fa7dc119640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:10.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.934+0000 7fa7dc119640 10 monclient(hunting): authenticate will time out at 2192.241731s 2024-04-26T00:40:10.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.934+0000 7fa7d9eb5640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7d40a3d90 msgr2=0x7fa7d409bf40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:40:10.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.934+0000 7fa7d9eb5640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7d40a3d90 0x7fa7d409bf40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:40:10.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.934+0000 7fa7da6b6640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fa7d413c3e0 0x7fa7d413e7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:50388/0 (socket says 172.21.15.37:50388) 2024-04-26T00:40:10.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.934+0000 7fa7da6b6640 1 -- 172.21.15.37:0/343766546 learned_addr learned my addr 172.21.15.37:0/343766546 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:40:10.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7da6b6640 1 -- 172.21.15.37:0/343766546 >> v1:172.21.15.37:6789/0 conn(0x7fa7d413c3e0 legacy=0x7fa7d413e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:10.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d8eb3640 1 -- 172.21.15.37:0/343766546 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 504156905 0 0) 0x7fa7d40a5150 con 0x7fa7d413c3e0 2024-04-26T00:40:10.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d8eb3640 10 monclient(hunting): _negotiate 2024-04-26T00:40:10.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d8eb3640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:10.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d8eb3640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:10.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d8eb3640 10 monclient(hunting): my global_id is 244158 2024-04-26T00:40:10.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d8eb3640 1 -- 172.21.15.37:0/343766546 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7bc003530 con 0x7fa7d413c3e0 2024-04-26T00:40:10.939 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d96b4640 1 -- 172.21.15.37:0/343766546 >> v1:172.21.15.45:6789/0 conn(0x7fa7d40a4170 legacy=0x7fa7d40a45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:10.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d8eb3640 1 -- 172.21.15.37:0/343766546 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1271688881 0 0) 0x7fa7bc003530 con 0x7fa7d413c3e0 2024-04-26T00:40:10.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d8eb3640 1 -- 172.21.15.37:0/343766546 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7d40a3d90 msgr2=0x7fa7d409bf40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:40:10.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d8eb3640 1 --2- 172.21.15.37:0/343766546 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7d40a3d90 0x7fa7d409bf40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:10.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d8eb3640 1 -- 172.21.15.37:0/343766546 >> v1:172.21.15.45:6789/0 conn(0x7fa7d40a4170 legacy=0x7fa7d40a45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:10.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d8eb3640 10 monclient: _finish_hunting 0 2024-04-26T00:40:10.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d8eb3640 1 monclient: found mon.noname-a-legacy 2024-04-26T00:40:10.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d8eb3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:10.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d8eb3640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.37:6789/0 2024-04-26T00:40:10.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d8eb3640 1 -- 172.21.15.37:0/343766546 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7d406dc60 con 0x7fa7d413c3e0 2024-04-26T00:40:10.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d8eb3640 10 monclient: _finish_auth 0 2024-04-26T00:40:10.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d8eb3640 10 monclient: _check_auth_tickets 2024-04-26T00:40:10.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d8eb3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:10.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d8eb3640 1 -- 172.21.15.37:0/343766546 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 866326386 0 0) 0x7fa7c4001930 con 0x7fa7d413c3e0 2024-04-26T00:40:10.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d8eb3640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:10.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d8eb3640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-26T00:40:10.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d8eb3640 10 monclient: dump: 2024-04-26T00:40:10.941 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:10.941 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:10.941 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:10.941 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:10.941 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:10.941 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:10.941 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:10.941 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:10.941 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:10.941 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:10.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d8eb3640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] but i'm connected to v1:172.21.15.37:6789/0, reconnecting 2024-04-26T00:40:10.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d8eb3640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:10.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d8eb3640 1 -- 172.21.15.37:0/343766546 >> v1:172.21.15.37:6789/0 conn(0x7fa7d413c3e0 legacy=0x7fa7d413e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:10.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d8eb3640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:40:10.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d8eb3640 1 --2- 172.21.15.37:0/343766546 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7bc0039f0 0x7fa7bc003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:10.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d8eb3640 10 monclient(hunting): picked mon.b con 0x7fa7bc0039f0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:40:10.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d8eb3640 1 --2- 172.21.15.37:0/343766546 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7d40a3d90 0x7fa7bc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:10.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d8eb3640 10 monclient(hunting): picked mon.c con 0x7fa7d40a3d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:40:10.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d8eb3640 1 --2- 172.21.15.37:0/343766546 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7bc004f50 0x7fa7bc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:10.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d8eb3640 10 monclient(hunting): picked mon.a con 0x7fa7bc004f50 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:40:10.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d8eb3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:10.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d8eb3640 1 -- 172.21.15.37:0/343766546 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d409b560 con 0x7fa7bc004f50 2024-04-26T00:40:10.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d8eb3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:10.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d8eb3640 1 -- 172.21.15.37:0/343766546 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa7bc005650 con 0x7fa7bc0039f0 2024-04-26T00:40:10.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d8eb3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:10.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d8eb3640 1 -- 172.21.15.37:0/343766546 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa7bc00bf40 con 0x7fa7d40a3d90 2024-04-26T00:40:10.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d8eb3640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:10.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d8eb3640 10 monclient(hunting): _finish_auth 0 2024-04-26T00:40:10.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d96b4640 1 -- 172.21.15.37:0/343766546 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7d40a3d90 msgr2=0x7fa7bc004810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:40:10.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d96b4640 1 --2- 172.21.15.37:0/343766546 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7d40a3d90 0x7fa7bc004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:40:10.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7da6b6640 1 --2- 172.21.15.37:0/343766546 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7bc004f50 0x7fa7bc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:10.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7da6b6640 10 monclient(hunting): get_auth_request con 0x7fa7bc004f50 auth_method 0 2024-04-26T00:40:10.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7da6b6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:10.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7da6b6640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:10.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7da6b6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:40:10.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7da6b6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:10.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7da6b6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:10.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d9eb5640 1 --2- 172.21.15.37:0/343766546 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7bc0039f0 0x7fa7bc003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:10.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7da6b6640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:40:10.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d9eb5640 10 monclient(hunting): get_auth_request con 0x7fa7bc0039f0 auth_method 0 2024-04-26T00:40:10.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d9eb5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:10.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d9eb5640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:10.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d9eb5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:40:10.943 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7da6b6640 10 monclient(hunting): handle_auth_done global_id 244158 payload 293 2024-04-26T00:40:10.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7da6b6640 1 -- 172.21.15.37:0/343766546 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7d40a3d90 msgr2=0x7fa7bc004810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:40:10.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7da6b6640 1 --2- 172.21.15.37:0/343766546 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7d40a3d90 0x7fa7bc004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:10.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7da6b6640 1 -- 172.21.15.37:0/343766546 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7bc0039f0 msgr2=0x7fa7bc003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:40:10.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7da6b6640 1 --2- 172.21.15.37:0/343766546 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7bc0039f0 0x7fa7bc003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:10.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7da6b6640 10 monclient: _finish_hunting 0 2024-04-26T00:40:10.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7da6b6640 1 monclient: found mon.a 2024-04-26T00:40:10.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7da6b6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:10.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7da6b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:40:10.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7da6b6640 1 -- 172.21.15.37:0/343766546 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7bc003530 con 0x7fa7bc004f50 2024-04-26T00:40:10.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7da6b6640 10 monclient: _finish_auth 0 2024-04-26T00:40:10.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7da6b6640 10 monclient: _check_auth_tickets 2024-04-26T00:40:10.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7da6b6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:10.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7dc119640 5 monclient: authenticate success, global_id 244158 2024-04-26T00:40:10.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7dc119640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:40:10.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.938+0000 7fa7d9eb5640 1 --2- 172.21.15.37:0/343766546 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7bc0039f0 0x7fa7bc003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:40:10.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7da6b6640 1 -- 172.21.15.37:0/343766546 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7bc004f50 msgr2=0x7fa7bc00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:10.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7da6b6640 1 --2- 172.21.15.37:0/343766546 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7bc004f50 0x7fa7bc00b9d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa7c4001b50 tx=0x7fa7c401cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7af941ab18a98476 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:10.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7d8eb3640 1 -- 172.21.15.37:0/343766546 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c400c0e0 con 0x7fa7bc004f50 2024-04-26T00:40:10.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7d8eb3640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:10.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7d8eb3640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:40:10.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7d8eb3640 10 monclient: dump: 2024-04-26T00:40:10.951 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:10.951 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:10.951 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:10.951 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:10.951 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:10.951 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:10.951 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:10.951 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:10.951 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:10.951 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:10.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7d8eb3640 1 -- 172.21.15.37:0/343766546 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7c40a9070 con 0x7fa7bc004f50 2024-04-26T00:40:10.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7d8eb3640 10 monclient: handle_config config(0 keys) 2024-04-26T00:40:10.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7d8eb3640 1 -- 172.21.15.37:0/343766546 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c4005d20 con 0x7fa7bc004f50 2024-04-26T00:40:10.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7dc119640 10 monclient: get_monmap_and_config success 2024-04-26T00:40:10.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7dc119640 4 set_mon_vals no callback set 2024-04-26T00:40:10.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7d8eb3640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:10.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7d8eb3640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:40:10.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7d8eb3640 10 monclient: dump: 2024-04-26T00:40:10.952 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:10.952 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:10.952 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:10.952 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:10.952 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:10.952 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:10.952 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:10.952 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:10.952 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:10.952 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:10.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7dc119640 10 monclient: shutdown 2024-04-26T00:40:10.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7dc119640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:40:10.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7dc119640 1 -- 172.21.15.37:0/343766546 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7bc004f50 msgr2=0x7fa7bc00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:10.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7dc119640 1 --2- 172.21.15.37:0/343766546 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7bc004f50 0x7fa7bc00b9d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa7c4001b50 tx=0x7fa7c401cc70 comp rx=0 tx=0).stop 2024-04-26T00:40:10.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7da6b6640 1 -- 172.21.15.37:0/343766546 reap_dead start 2024-04-26T00:40:10.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7dc119640 1 -- 172.21.15.37:0/343766546 shutdown_connections 2024-04-26T00:40:10.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7dc119640 1 -- 172.21.15.37:0/343766546 >> 172.21.15.37:0/343766546 conn(0x7fa7d40aa0a0 msgr2=0x7fa7d40a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:40:10.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7dc119640 1 -- 172.21.15.37:0/343766546 shutdown_connections 2024-04-26T00:40:10.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7dc119640 1 -- 172.21.15.37:0/343766546 wait complete. 2024-04-26T00:40:10.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7dc119640 10 monclient: build_initial_monmap 2024-04-26T00:40:10.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7dc119640 10 monclient: monmap: 2024-04-26T00:40:10.953 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:40:10.953 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:40:10.953 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:40:10.953 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:40:10.953 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:40:10.954 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:40:10.954 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:40:10.954 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:40:10.954 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:40:10.954 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:10.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7dc119640 1 Processor -- start 2024-04-26T00:40:10.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7dc119640 1 -- start start 2024-04-26T00:40:10.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7dc119640 10 monclient: init 2024-04-26T00:40:10.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7dc119640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:10.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7dc119640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:40:10.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7dc119640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7d40a2bd0 0x7fa7d41373f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:10.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7dc119640 10 monclient(hunting): picked mon.noname-b con 0x7fa7d40a2bd0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:40:10.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7dc119640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7d4134970 0x7fa7d4137930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:10.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7dc119640 10 monclient(hunting): picked mon.noname-c con 0x7fa7d4134970 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:40:10.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7dc119640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7d4132d20 0x7fa7d41331d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:10.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7dc119640 10 monclient(hunting): picked mon.noname-a con 0x7fa7d4132d20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:40:10.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7dc119640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:10.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7dc119640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d4140280 con 0x7fa7d4132d20 2024-04-26T00:40:10.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7dc119640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:10.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7dc119640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d4140100 con 0x7fa7d40a2bd0 2024-04-26T00:40:10.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7dc119640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:10.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7dc119640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa7d4140400 con 0x7fa7d4134970 2024-04-26T00:40:10.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7dc119640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:10.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.942+0000 7fa7dc119640 10 monclient(hunting): authenticate will time out at 2192.251995s 2024-04-26T00:40:10.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7d96b4640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7d4134970 msgr2=0x7fa7d4137930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:40:10.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7d96b4640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7d4134970 0x7fa7d4137930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:40:10.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7da6b6640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7d4132d20 0x7fa7d41331d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:10.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7da6b6640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7d4132d20 0x7fa7d41331d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:57762/0 (socket says 172.21.15.37:57762) 2024-04-26T00:40:10.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7da6b6640 1 -- 172.21.15.37:0/2516269306 learned_addr learned my addr 172.21.15.37:0/2516269306 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:40:10.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7da6b6640 10 monclient(hunting): get_auth_request con 0x7fa7d4132d20 auth_method 0 2024-04-26T00:40:10.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7da6b6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:10.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7da6b6640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:10.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7da6b6640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:10.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7da6b6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:10.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7da6b6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:10.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7da6b6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:10.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7d9eb5640 1 --2- 172.21.15.37:0/2516269306 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7d40a2bd0 0x7fa7d41373f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:10.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7da6b6640 10 monclient(hunting): handle_auth_done global_id 244161 payload 824 2024-04-26T00:40:10.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7da6b6640 1 -- 172.21.15.37:0/2516269306 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7d4134970 msgr2=0x7fa7d4137930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:40:10.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7da6b6640 1 --2- 172.21.15.37:0/2516269306 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7d4134970 0x7fa7d4137930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:10.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7da6b6640 1 -- 172.21.15.37:0/2516269306 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7d40a2bd0 msgr2=0x7fa7d41373f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:10.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7da6b6640 1 --2- 172.21.15.37:0/2516269306 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7d40a2bd0 0x7fa7d41373f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:10.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7da6b6640 10 monclient: _finish_hunting 0 2024-04-26T00:40:10.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7da6b6640 1 monclient: found mon.noname-a 2024-04-26T00:40:10.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7da6b6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:10.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7da6b6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:40:10.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7da6b6640 1 -- 172.21.15.37:0/2516269306 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7d41cf0d0 con 0x7fa7d4132d20 2024-04-26T00:40:10.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7d9eb5640 10 monclient: get_auth_request con 0x7fa7d40a2bd0 auth_method 0 2024-04-26T00:40:10.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7d9eb5640 1 --2- 172.21.15.37:0/2516269306 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7d40a2bd0 0x7fa7d41373f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:40:10.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7da6b6640 1 -- 172.21.15.37:0/2516269306 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7d4132d20 msgr2=0x7fa7d41331d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:10.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7da6b6640 1 --2- 172.21.15.37:0/2516269306 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7d4132d20 0x7fa7d41331d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa7c401d510 tx=0x7fa7bc005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:10.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7d27fc640 1 -- 172.21.15.37:0/2516269306 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c4001730 con 0x7fa7d4132d20 2024-04-26T00:40:10.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:10.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7d27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:40:10.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7d27fc640 10 monclient: dump: 2024-04-26T00:40:10.957 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:10.957 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:10.957 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:10.957 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:10.958 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:10.958 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:10.958 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:10.958 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:10.958 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:10.958 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:10.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7d27fc640 10 monclient: _finish_auth 0 2024-04-26T00:40:10.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7d27fc640 10 monclient: _check_auth_tickets 2024-04-26T00:40:10.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:10.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7d27fc640 1 -- 172.21.15.37:0/2516269306 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7c40018d0 con 0x7fa7d4132d20 2024-04-26T00:40:10.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7d27fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:40:10.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7d27fc640 1 -- 172.21.15.37:0/2516269306 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c40a5aa0 con 0x7fa7d4132d20 2024-04-26T00:40:10.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:10.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7d27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:40:10.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7d27fc640 10 monclient: dump: 2024-04-26T00:40:10.959 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:10.959 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:10.959 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:10.959 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:10.959 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:10.959 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:10.959 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:10.959 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:10.959 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:10.959 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:10.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7d3fff640 4 set_mon_vals no callback set 2024-04-26T00:40:10.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7dc119640 5 monclient: authenticate success, global_id 244161 2024-04-26T00:40:10.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7dc119640 10 monclient: _renew_subs 2024-04-26T00:40:10.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7dc119640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:40:10.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7dc119640 1 -- 172.21.15.37:0/2516269306 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7d41cfb10 con 0x7fa7d4132d20 2024-04-26T00:40:10.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7dc119640 10 monclient: _renew_subs 2024-04-26T00:40:10.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7dc119640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:40:10.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.946+0000 7fa7dc119640 1 -- 172.21.15.37:0/2516269306 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7d409ada0 con 0x7fa7d4132d20 2024-04-26T00:40:10.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.950+0000 7fa7dc119640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:10.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.950+0000 7fa7dc119640 1 --2- 172.21.15.37:0/2516269306 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7d402d180 0x7fa7d4133710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:10.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.950+0000 7fa7dc119640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:10.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.950+0000 7fa7dc119640 1 -- 172.21.15.37:0/2516269306 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d40a7820 con 0x7fa7d402d180 2024-04-26T00:40:10.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.950+0000 7fa7d9eb5640 1 --2- 172.21.15.37:0/2516269306 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7d402d180 0x7fa7d4133710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:10.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.950+0000 7fa7d9eb5640 10 monclient: get_auth_request con 0x7fa7d402d180 auth_method 0 2024-04-26T00:40:10.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.950+0000 7fa7d9eb5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:10.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.950+0000 7fa7d9eb5640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:10.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.950+0000 7fa7d9eb5640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:10.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.950+0000 7fa7d27fc640 1 -- 172.21.15.37:0/2516269306 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa7c40b6030 con 0x7fa7d4132d20 2024-04-26T00:40:10.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.950+0000 7fa7d9eb5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:10.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.950+0000 7fa7d9eb5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:10.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.950+0000 7fa7d9eb5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:10.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.950+0000 7fa7d9eb5640 10 monclient(hunting): handle_auth_done global_id 244164 payload 824 2024-04-26T00:40:10.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.950+0000 7fa7d27fc640 1 --2- 172.21.15.37:0/2516269306 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa7c0035e30 0x7fa7c00382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:10.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.950+0000 7fa7d9eb5640 1 -- 172.21.15.37:0/2516269306 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa7d402d560 con 0x7fa7d402d180 2024-04-26T00:40:10.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.950+0000 7fa7d27fc640 1 -- 172.21.15.37:0/2516269306 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(793..793 src has 705..793) ==== 3931+0+0 (secure 0 0 0) 0x7fa7c40e1b70 con 0x7fa7d4132d20 2024-04-26T00:40:10.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.950+0000 7fa7d96b4640 1 --2- 172.21.15.37:0/2516269306 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa7c0035e30 0x7fa7c00382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:10.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.950+0000 7fa7d96b4640 10 monclient: get_auth_request con 0x7fa7c0035e30 auth_method 0 2024-04-26T00:40:10.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.950+0000 7fa7d9eb5640 1 -- 172.21.15.37:0/2516269306 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7d402d180 msgr2=0x7fa7d4133710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:10.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.950+0000 7fa7d9eb5640 1 --2- 172.21.15.37:0/2516269306 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7d402d180 0x7fa7d4133710 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc00e930 tx=0x7fa7cc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:10.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.950+0000 7fa7d27fc640 1 -- 172.21.15.37:0/2516269306 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7cc017640 con 0x7fa7d402d180 2024-04-26T00:40:10.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.950+0000 7fa7d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:10.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.950+0000 7fa7d27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:40:10.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.950+0000 7fa7d27fc640 10 monclient: dump: 2024-04-26T00:40:10.963 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:10.963 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:10.963 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:10.964 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:10.964 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:10.964 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:10.964 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:10.964 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:10.964 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:10.964 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:10.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.950+0000 7fa7d27fc640 1 -- 172.21.15.37:0/2516269306 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa7cc017640 con 0x7fa7d402d180 2024-04-26T00:40:10.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.950+0000 7fa7d27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:10.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.950+0000 7fa7d27fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:40:10.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.950+0000 7fa7d27fc640 1 -- 172.21.15.37:0/2516269306 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7d402d180 msgr2=0x7fa7d4133710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:10.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.950+0000 7fa7d27fc640 1 --2- 172.21.15.37:0/2516269306 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7d402d180 0x7fa7d4133710 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc00e930 tx=0x7fa7cc00edf0 comp rx=0 tx=0).stop 2024-04-26T00:40:10.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.950+0000 7fa7d27fc640 1 -- 172.21.15.37:0/2516269306 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7d402d180 msgr2=0x7fa7d4133710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:40:10.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.950+0000 7fa7d27fc640 1 --2- 172.21.15.37:0/2516269306 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7d402d180 0x7fa7d4133710 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc00e930 tx=0x7fa7cc00edf0 comp rx=0 tx=0).stop 2024-04-26T00:40:10.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.954+0000 7fa7d96b4640 1 --2- 172.21.15.37:0/2516269306 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa7c0035e30 0x7fa7c00382f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fa7d40a3460 tx=0x7fa7c8007a00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:10.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.958+0000 7fa7dc119640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:40:10.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.958+0000 7fa7dc119640 1 --2- 172.21.15.37:0/2516269306 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7d40623f0 0x7fa7d4062800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:10.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.958+0000 7fa7dc119640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:10.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.958+0000 7fa7dc119640 1 -- 172.21.15.37:0/2516269306 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d404f510 con 0x7fa7d40623f0 2024-04-26T00:40:10.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.958+0000 7fa7d9eb5640 1 --2- 172.21.15.37:0/2516269306 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7d40623f0 0x7fa7d4062800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:10.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.958+0000 7fa7d9eb5640 10 monclient: get_auth_request con 0x7fa7d40623f0 auth_method 0 2024-04-26T00:40:10.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.958+0000 7fa7d9eb5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:10.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.958+0000 7fa7d9eb5640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:10.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.958+0000 7fa7d9eb5640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:10.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.958+0000 7fa7d9eb5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:10.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.958+0000 7fa7d9eb5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:10.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.958+0000 7fa7d9eb5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:10.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.958+0000 7fa7d9eb5640 10 monclient(hunting): handle_auth_done global_id 244167 payload 824 2024-04-26T00:40:10.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.958+0000 7fa7d9eb5640 1 -- 172.21.15.37:0/2516269306 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa7d409e780 con 0x7fa7d40623f0 2024-04-26T00:40:10.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.962+0000 7fa7d9eb5640 1 -- 172.21.15.37:0/2516269306 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7d40623f0 msgr2=0x7fa7d4062800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:10.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.962+0000 7fa7d9eb5640 1 --2- 172.21.15.37:0/2516269306 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7d40623f0 0x7fa7d4062800 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc017000 tx=0x7fa7cc005da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:10.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.962+0000 7fa7d27fc640 1 -- 172.21.15.37:0/2516269306 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7cc01d820 con 0x7fa7d40623f0 2024-04-26T00:40:10.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.962+0000 7fa7d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:10.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.962+0000 7fa7d27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:40:10.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.962+0000 7fa7d27fc640 10 monclient: dump: 2024-04-26T00:40:10.966 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:10.966 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:10.966 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:10.966 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:10.966 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:10.966 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:10.966 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:10.967 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:10.967 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:10.967 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:10.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.962+0000 7fa7d27fc640 1 -- 172.21.15.37:0/2516269306 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4708 (secure 0 0 0) 0x7fa7cc01d820 con 0x7fa7d40623f0 2024-04-26T00:40:10.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.962+0000 7fa7d27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:40:10.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.962+0000 7fa7d27fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:40:10.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.962+0000 7fa7d27fc640 1 -- 172.21.15.37:0/2516269306 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7d40623f0 msgr2=0x7fa7d4062800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:10.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.962+0000 7fa7d27fc640 1 --2- 172.21.15.37:0/2516269306 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7d40623f0 0x7fa7d4062800 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc017000 tx=0x7fa7cc005da0 comp rx=0 tx=0).stop 2024-04-26T00:40:10.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.962+0000 7fa7d27fc640 1 -- 172.21.15.37:0/2516269306 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7d40623f0 msgr2=0x7fa7d4062800 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:40:10.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.962+0000 7fa7d27fc640 1 --2- 172.21.15.37:0/2516269306 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7d40623f0 0x7fa7d4062800 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc017000 tx=0x7fa7cc005da0 comp rx=0 tx=0).stop 2024-04-26T00:40:10.967 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:40:10.967 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:40:10.968 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:40:10.968 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:40:10.968 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 150, 2024-04-26T00:40:10.968 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 28319, 2024-04-26T00:40:10.968 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:40:10.968 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:40:10.968 INFO:teuthology.orchestra.run.smithi037.stdout: 1 2024-04-26T00:40:10.968 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:10.968 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:40:10.968 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:40:10.968 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:40:10.968 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:40:10.968 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:40:10.968 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:40:10.969 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:40:10.969 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:40:10.969 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:40:10.969 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:40:10.969 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:40:10.969 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:40:10.969 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:40:10.969 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:40:10.969 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:40:10.969 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:10.969 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:40:10.969 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:40:10.969 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:40:10.969 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:40:10.969 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:40:10.970 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:40:10.970 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:40:10.970 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:40:10.970 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:40:10.970 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:40:10.970 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:40:10.970 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:40:10.970 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:40:10.970 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:10.970 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:10.970 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:40:10.970 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:40:10.970 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:40:10.970 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:40:10.971 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:40:10.971 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:40:10.973 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:40:10.973 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:40:10.973 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:40:10.973 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:40:10.974 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:40:10.974 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:40:10.974 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:40:10.974 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:40:10.974 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:40:10.974 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:40:10.974 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:40:10.974 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:40:10.974 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:40:10.974 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:40:10.974 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:40:10.974 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:40:10.974 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:40:10.974 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:40:10.975 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:40:10.975 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:40:10.975 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:40:10.975 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:40:10.975 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:10.975 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:40:10.975 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:10.975 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:40:10.975 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:10.975 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:40:10.975 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:40:10.975 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:40:10.975 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:40:10.975 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:10.975 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:40:10.976 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:40:10.976 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:10.976 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:10.976 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:10.976 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:40:10.976 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:40:10.976 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:10.976 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:10.976 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:10.976 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:10.976 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:40:10.976 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:40:10.976 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:40:10.976 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:40:10.977 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:40:10.977 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:10.977 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:10.977 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:40:10.977 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:40:10.977 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:40:10.977 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:40:10.977 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:10.977 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:40:10.977 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:40:10.977 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:10.977 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:10.977 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:10.977 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:40:10.977 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:40:10.978 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:10.978 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:10.978 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:10.978 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:10.978 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:40:10.978 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:40:10.978 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:40:10.978 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:40:10.978 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:40:10.978 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:10.978 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:10.978 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:40:10.978 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:40:10.978 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:40:10.979 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:40:10.979 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:10.979 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:40:10.979 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:40:10.979 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:10.979 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:10.979 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:10.979 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:40:10.979 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:40:10.979 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:10.979 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:10.979 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:10.979 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:10.979 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:40:10.980 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:40:10.980 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:40:10.980 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:40:10.980 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:40:10.980 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:10.980 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:10.980 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:10.980 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:40:10.980 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:40:10.980 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:10.980 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:40:10.980 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:40:10.980 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:40:10.980 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:10.980 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:10.981 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:40:10.981 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:10.981 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:40:10.981 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:40:10.981 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:40:10.981 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:10.981 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:10.981 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:40:10.981 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:10.981 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:40:10.981 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:40:10.981 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:40:10.981 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:10.981 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:10.981 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:10.982 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:40:10.982 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:40:10.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.962+0000 7fa7dc119640 1 -- 172.21.15.37:0/2516269306 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa7c0035e30 msgr2=0x7fa7c00382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:10.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.962+0000 7fa7dc119640 1 --2- 172.21.15.37:0/2516269306 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa7c0035e30 0x7fa7c00382f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fa7d40a3460 tx=0x7fa7c8007a00 comp rx=0 tx=0).stop 2024-04-26T00:40:10.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.962+0000 7fa7dc119640 10 monclient: shutdown 2024-04-26T00:40:10.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.962+0000 7fa7dc119640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:40:10.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.962+0000 7fa7dc119640 1 -- 172.21.15.37:0/2516269306 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7d4132d20 msgr2=0x7fa7d41331d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:10.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.962+0000 7fa7dc119640 1 --2- 172.21.15.37:0/2516269306 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7d4132d20 0x7fa7d41331d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa7c401d510 tx=0x7fa7bc005840 comp rx=0 tx=0).stop 2024-04-26T00:40:10.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.962+0000 7fa7da6b6640 1 -- 172.21.15.37:0/2516269306 reap_dead start 2024-04-26T00:40:10.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.962+0000 7fa7da6b6640 1 -- 172.21.15.37:0/2516269306 reap_dead start 2024-04-26T00:40:10.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.962+0000 7fa7dc119640 1 -- 172.21.15.37:0/2516269306 shutdown_connections 2024-04-26T00:40:10.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.962+0000 7fa7dc119640 1 -- 172.21.15.37:0/2516269306 >> 172.21.15.37:0/2516269306 conn(0x7fa7d40aa0a0 msgr2=0x7fa7d413e7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:40:10.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.962+0000 7fa7dc119640 1 -- 172.21.15.37:0/2516269306 shutdown_connections 2024-04-26T00:40:10.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:10.962+0000 7fa7dc119640 1 -- 172.21.15.37:0/2516269306 wait complete. 2024-04-26T00:40:10.984 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:40:10.984 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-04-26T00:40:10.984 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:40:11.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.098+0000 7fab8f306640 10 monclient: get_monmap_and_config 2024-04-26T00:40:11.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.098+0000 7fab8f306640 10 monclient: build_initial_monmap 2024-04-26T00:40:11.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.098+0000 7fab8f306640 10 monclient: monmap: 2024-04-26T00:40:11.100 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:40:11.100 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:11.100 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:40:11.102314+0000 2024-04-26T00:40:11.100 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:40:11.102314+0000 2024-04-26T00:40:11.100 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:40:11.100 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:11.100 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:40:11.100 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:40:11.100 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:40:11.100 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:40:11.100 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:40:11.100 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:11.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.098+0000 7fab8f306640 1 Processor -- start 2024-04-26T00:40:11.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.098+0000 7fab8f306640 1 -- start start 2024-04-26T00:40:11.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.098+0000 7fab8f306640 10 monclient: init 2024-04-26T00:40:11.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.098+0000 7fab8f306640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:11.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.098+0000 7fab8f306640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-04-26T00:40:11.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.098+0000 7fab8f306640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fab880a0300 addr v1:172.21.15.45:6789/0 2024-04-26T00:40:11.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.098+0000 7fab8f306640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fab8809fc50 addr v1:172.21.15.37:6789/0 2024-04-26T00:40:11.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.098+0000 7fab8f306640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fab8813b5e0 0x7fab8813d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:11.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.098+0000 7fab8f306640 10 monclient(hunting): picked mon.noname-c con 0x7fab8813b5e0 addr v2:172.21.15.45:3300/0 2024-04-26T00:40:11.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.098+0000 7fab8f306640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab8809c190 con 0x7fab8809fc50 2024-04-26T00:40:11.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.098+0000 7fab8f306640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab880a1110 con 0x7fab880a0300 2024-04-26T00:40:11.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.098+0000 7fab8f306640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:11.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.098+0000 7fab8f306640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fab880a16d0 con 0x7fab8813b5e0 2024-04-26T00:40:11.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.098+0000 7fab8f306640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:11.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.098+0000 7fab8f306640 10 monclient(hunting): authenticate will time out at 2192.407924s 2024-04-26T00:40:11.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.102+0000 7fab8c8a1640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fab8809fc50 0x7fab88139ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:50402/0 (socket says 172.21.15.37:50402) 2024-04-26T00:40:11.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.102+0000 7fab8c8a1640 1 -- 172.21.15.37:0/3285567511 learned_addr learned my addr 172.21.15.37:0/3285567511 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:40:11.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.102+0000 7fab8d8a3640 1 --2- 172.21.15.37:0/3285567511 >> v2:172.21.15.45:3300/0 conn(0x7fab8813b5e0 0x7fab8813d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:11.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.102+0000 7fab8d8a3640 10 monclient(hunting): get_auth_request con 0x7fab8813b5e0 auth_method 0 2024-04-26T00:40:11.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.102+0000 7fab8d8a3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:11.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.102+0000 7fab8d8a3640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:11.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.102+0000 7fab8d8a3640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:11.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.102+0000 7fab8d8a3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:11.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.102+0000 7fab8d8a3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:11.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.102+0000 7fab8d8a3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:11.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.102+0000 7fab8d8a3640 10 monclient(hunting): handle_auth_done global_id 224329 payload 274 2024-04-26T00:40:11.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.102+0000 7fab8d8a3640 1 -- 172.21.15.37:0/3285567511 >> v1:172.21.15.45:6789/0 conn(0x7fab880a0300 legacy=0x7fab880a06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:40:11.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.102+0000 7fab8d8a3640 1 -- 172.21.15.37:0/3285567511 >> v1:172.21.15.37:6789/0 conn(0x7fab8809fc50 legacy=0x7fab88139ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:40:11.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.102+0000 7fab8d8a3640 10 monclient: _finish_hunting 0 2024-04-26T00:40:11.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.102+0000 7fab8d8a3640 1 monclient: found mon.noname-c 2024-04-26T00:40:11.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.102+0000 7fab8d8a3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:11.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.102+0000 7fab8d8a3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:40:11.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.102+0000 7fab8d8a3640 1 -- 172.21.15.37:0/3285567511 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab8809f2a0 con 0x7fab8813b5e0 2024-04-26T00:40:11.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.106+0000 7fab8d8a3640 1 -- 172.21.15.37:0/3285567511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab8813b5e0 msgr2=0x7fab8813d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:11.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.106+0000 7fab8d8a3640 1 --2- 172.21.15.37:0/3285567511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab8813b5e0 0x7fab8813d9d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fab8809f4d0 tx=0x7fab8001c2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a664e825bc1d2bb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:11.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.106+0000 7fab7ffff640 1 -- 172.21.15.37:0/3285567511 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab800b0070 con 0x7fab8813b5e0 2024-04-26T00:40:11.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.106+0000 7fab7ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:11.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.106+0000 7fab7ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:40:11.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.106+0000 7fab7ffff640 10 monclient: dump: 2024-04-26T00:40:11.108 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:11.108 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:11.109 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:11.109 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:11.109 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:11.109 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:11.109 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:11.109 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:11.109 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:11.109 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:11.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.106+0000 7fab7ffff640 10 monclient: _finish_auth 0 2024-04-26T00:40:11.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.106+0000 7fab7ffff640 10 monclient: _check_auth_tickets 2024-04-26T00:40:11.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.106+0000 7fab7ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:11.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.106+0000 7fab7ffff640 1 -- 172.21.15.37:0/3285567511 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab800ab600 con 0x7fab8813b5e0 2024-04-26T00:40:11.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.106+0000 7fab7ffff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:40:11.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.106+0000 7fab7ffff640 1 -- 172.21.15.37:0/3285567511 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab800ab9d0 con 0x7fab8813b5e0 2024-04-26T00:40:11.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.106+0000 7fab7ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:11.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.106+0000 7fab7ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:40:11.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.106+0000 7fab7ffff640 10 monclient: dump: 2024-04-26T00:40:11.110 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:11.110 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:11.110 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:11.110 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:11.110 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:11.110 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:11.110 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:11.110 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:11.110 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:11.110 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:11.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.106+0000 7fab8f306640 5 monclient: authenticate success, global_id 224329 2024-04-26T00:40:11.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.106+0000 7fab8f306640 10 monclient: get_monmap_and_config success 2024-04-26T00:40:11.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.106+0000 7fab8f306640 4 set_mon_vals no callback set 2024-04-26T00:40:11.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.106+0000 7fab8f306640 10 monclient: shutdown 2024-04-26T00:40:11.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.106+0000 7fab8f306640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:40:11.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.106+0000 7fab8f306640 1 -- 172.21.15.37:0/3285567511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab8813b5e0 msgr2=0x7fab8813d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:11.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.106+0000 7fab8f306640 1 --2- 172.21.15.37:0/3285567511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab8813b5e0 0x7fab8813d9d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fab8809f4d0 tx=0x7fab8001c2e0 comp rx=0 tx=0).stop 2024-04-26T00:40:11.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.106+0000 7fab8f306640 1 -- 172.21.15.37:0/3285567511 shutdown_connections 2024-04-26T00:40:11.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.106+0000 7fab8f306640 1 --2- 172.21.15.37:0/3285567511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab8813b5e0 0x7fab8813d9d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:11.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.106+0000 7fab8f306640 1 -- 172.21.15.37:0/3285567511 >> 172.21.15.37:0/3285567511 conn(0x7fab880a1d50 msgr2=0x7fab880a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:40:11.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.106+0000 7fab8f306640 1 -- 172.21.15.37:0/3285567511 shutdown_connections 2024-04-26T00:40:11.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.106+0000 7fab8f306640 1 -- 172.21.15.37:0/3285567511 wait complete. 2024-04-26T00:40:11.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.106+0000 7fab8f306640 10 monclient: build_initial_monmap 2024-04-26T00:40:11.111 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.106+0000 7fab8f306640 10 monclient: monmap: 2024-04-26T00:40:11.111 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:40:11.111 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:40:11.111 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:40:11.111 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:40:11.111 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:40:11.111 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:40:11.112 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:40:11.112 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:40:11.112 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:40:11.112 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:11.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8f306640 1 Processor -- start 2024-04-26T00:40:11.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8f306640 1 -- start start 2024-04-26T00:40:11.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8f306640 10 monclient: init 2024-04-26T00:40:11.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8f306640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:11.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8f306640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:40:11.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8f306640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab8809fc50 0x7fab881397d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:11.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8f306640 10 monclient(hunting): picked mon.noname-b con 0x7fab8809fc50 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:40:11.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8f306640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab880a0300 0x7fab88134a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:11.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8f306640 10 monclient(hunting): picked mon.noname-a con 0x7fab880a0300 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:40:11.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8f306640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab8813b5e0 0x7fab8811b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:11.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8f306640 10 monclient(hunting): picked mon.noname-c con 0x7fab8813b5e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:40:11.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8f306640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:11.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8f306640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fab88140ac0 con 0x7fab880a0300 2024-04-26T00:40:11.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8f306640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:11.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8f306640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fab88140940 con 0x7fab8809fc50 2024-04-26T00:40:11.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8f306640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:11.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8f306640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fab88140c40 con 0x7fab8813b5e0 2024-04-26T00:40:11.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8f306640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:11.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8f306640 10 monclient(hunting): authenticate will time out at 2192.416658s 2024-04-26T00:40:11.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8d8a3640 1 -- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab8813b5e0 msgr2=0x7fab8811b8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3301/0 2024-04-26T00:40:11.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8d8a3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab8813b5e0 0x7fab8811b8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:40:11.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8c8a1640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab880a0300 0x7fab88134a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:11.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8c8a1640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab880a0300 0x7fab88134a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:57794/0 (socket says 172.21.15.37:57794) 2024-04-26T00:40:11.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8c8a1640 1 -- 172.21.15.37:0/3646096099 learned_addr learned my addr 172.21.15.37:0/3646096099 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:40:11.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8c8a1640 10 monclient(hunting): get_auth_request con 0x7fab880a0300 auth_method 0 2024-04-26T00:40:11.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8c8a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:11.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8c8a1640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:11.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8c8a1640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:11.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8c8a1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:11.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8c8a1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:11.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8c8a1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:11.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8d0a2640 1 --2- 172.21.15.37:0/3646096099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab8809fc50 0x7fab881397d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:11.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8c8a1640 10 monclient(hunting): handle_auth_done global_id 244170 payload 824 2024-04-26T00:40:11.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8c8a1640 1 -- 172.21.15.37:0/3646096099 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab8813b5e0 msgr2=0x7fab8811b8a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:40:11.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8c8a1640 1 --2- 172.21.15.37:0/3646096099 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab8813b5e0 0x7fab8811b8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:11.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8c8a1640 1 -- 172.21.15.37:0/3646096099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab8809fc50 msgr2=0x7fab881397d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:11.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8c8a1640 1 --2- 172.21.15.37:0/3646096099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab8809fc50 0x7fab881397d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:11.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8c8a1640 10 monclient: _finish_hunting 0 2024-04-26T00:40:11.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8c8a1640 1 monclient: found mon.noname-a 2024-04-26T00:40:11.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8c8a1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:11.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8c8a1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:40:11.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8c8a1640 1 -- 172.21.15.37:0/3646096099 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab8813a460 con 0x7fab880a0300 2024-04-26T00:40:11.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8d0a2640 10 monclient: get_auth_request con 0x7fab8809fc50 auth_method 0 2024-04-26T00:40:11.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8d0a2640 1 --2- 172.21.15.37:0/3646096099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab8809fc50 0x7fab881397d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:40:11.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8c8a1640 1 -- 172.21.15.37:0/3646096099 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab880a0300 msgr2=0x7fab88134a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:11.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8c8a1640 1 --2- 172.21.15.37:0/3646096099 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab880a0300 0x7fab88134a20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fab7800c4a0 tx=0x7fab7800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:11.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab7dffb640 1 -- 172.21.15.37:0/3646096099 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab78017070 con 0x7fab880a0300 2024-04-26T00:40:11.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab7dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:11.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab7dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:40:11.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab7dffb640 10 monclient: dump: 2024-04-26T00:40:11.121 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:11.121 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:11.121 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:11.121 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:11.121 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:11.121 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:11.121 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:11.121 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:11.121 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:11.122 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:11.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab7dffb640 10 monclient: _finish_auth 0 2024-04-26T00:40:11.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab7dffb640 10 monclient: _check_auth_tickets 2024-04-26T00:40:11.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab7dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:11.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab7dffb640 1 -- 172.21.15.37:0/3646096099 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab78007070 con 0x7fab880a0300 2024-04-26T00:40:11.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8f306640 5 monclient: authenticate success, global_id 244170 2024-04-26T00:40:11.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8f306640 10 monclient: _renew_subs 2024-04-26T00:40:11.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8f306640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:40:11.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8f306640 1 -- 172.21.15.37:0/3646096099 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab8809f550 con 0x7fab880a0300 2024-04-26T00:40:11.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab7dffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:40:11.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab7dffb640 1 -- 172.21.15.37:0/3646096099 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab78007460 con 0x7fab880a0300 2024-04-26T00:40:11.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab7dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:11.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab7dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:40:11.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab7dffb640 10 monclient: dump: 2024-04-26T00:40:11.123 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:11.123 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:11.123 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:11.123 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:11.123 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:11.123 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:11.123 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:11.123 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:11.123 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:11.123 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:11.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8f306640 10 monclient: _renew_subs 2024-04-26T00:40:11.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8f306640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:40:11.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab7effd640 4 set_mon_vals no callback set 2024-04-26T00:40:11.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.110+0000 7fab8f306640 1 -- 172.21.15.37:0/3646096099 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab8811db00 con 0x7fab880a0300 2024-04-26T00:40:11.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.114+0000 7fab8f306640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:11.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.114+0000 7fab8f306640 1 --2- 172.21.15.37:0/3646096099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab881c5d10 0x7fab8811bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:11.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.114+0000 7fab8f306640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:11.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.114+0000 7fab8f306640 1 -- 172.21.15.37:0/3646096099 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fab8804f510 con 0x7fab881c5d10 2024-04-26T00:40:11.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.114+0000 7fab7dffb640 1 -- 172.21.15.37:0/3646096099 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fab7801a070 con 0x7fab880a0300 2024-04-26T00:40:11.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.114+0000 7fab7dffb640 1 --2- 172.21.15.37:0/3646096099 >> v2:172.21.15.45:6812/2379918623 conn(0x7fab6c035e30 0x7fab6c0382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:11.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.114+0000 7fab7dffb640 1 -- 172.21.15.37:0/3646096099 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(793..793 src has 705..793) ==== 3931+0+0 (secure 0 0 0) 0x7fab78048050 con 0x7fab880a0300 2024-04-26T00:40:11.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.114+0000 7fab8d0a2640 1 --2- 172.21.15.37:0/3646096099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab881c5d10 0x7fab8811bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:11.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.114+0000 7fab8d0a2640 10 monclient: get_auth_request con 0x7fab881c5d10 auth_method 0 2024-04-26T00:40:11.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.114+0000 7fab8d0a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:11.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.114+0000 7fab8d0a2640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:11.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.114+0000 7fab8d0a2640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:11.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.114+0000 7fab8d0a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:11.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.114+0000 7fab8d0a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:11.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.114+0000 7fab8d8a3640 1 --2- 172.21.15.37:0/3646096099 >> v2:172.21.15.45:6812/2379918623 conn(0x7fab6c035e30 0x7fab6c0382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:11.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.114+0000 7fab8d0a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:11.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.114+0000 7fab8d8a3640 10 monclient: get_auth_request con 0x7fab6c035e30 auth_method 0 2024-04-26T00:40:11.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.114+0000 7fab8d0a2640 10 monclient(hunting): handle_auth_done global_id 224332 payload 824 2024-04-26T00:40:11.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.114+0000 7fab8d0a2640 1 -- 172.21.15.37:0/3646096099 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fab8809eb60 con 0x7fab881c5d10 2024-04-26T00:40:11.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.114+0000 7fab8d0a2640 1 -- 172.21.15.37:0/3646096099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab881c5d10 msgr2=0x7fab8811bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:11.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.114+0000 7fab8d0a2640 1 --2- 172.21.15.37:0/3646096099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab881c5d10 0x7fab8811bde0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fab8400d860 tx=0x7fab8400dd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:11.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.114+0000 7fab8d8a3640 1 --2- 172.21.15.37:0/3646096099 >> v2:172.21.15.45:6812/2379918623 conn(0x7fab6c035e30 0x7fab6c0382f0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fab800ab000 tx=0x7fab80008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:11.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.118+0000 7fab7dffb640 1 -- 172.21.15.37:0/3646096099 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab84006d60 con 0x7fab881c5d10 2024-04-26T00:40:11.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.118+0000 7fab7dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:11.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.118+0000 7fab7dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:40:11.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.118+0000 7fab7dffb640 10 monclient: dump: 2024-04-26T00:40:11.125 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:11.125 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:11.125 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:11.125 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:11.126 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:11.126 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:11.126 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:11.126 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:11.126 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:11.126 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:11.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.118+0000 7fab7dffb640 1 -- 172.21.15.37:0/3646096099 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fab84006d60 con 0x7fab881c5d10 2024-04-26T00:40:11.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.118+0000 7fab7dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:11.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.118+0000 7fab7dffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:40:11.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.118+0000 7fab7dffb640 1 -- 172.21.15.37:0/3646096099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab881c5d10 msgr2=0x7fab8811bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:11.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.118+0000 7fab7dffb640 1 --2- 172.21.15.37:0/3646096099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab881c5d10 0x7fab8811bde0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fab8400d860 tx=0x7fab8400dd20 comp rx=0 tx=0).stop 2024-04-26T00:40:11.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.118+0000 7fab7dffb640 1 -- 172.21.15.37:0/3646096099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab881c5d10 msgr2=0x7fab8811bde0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:40:11.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.118+0000 7fab7dffb640 1 --2- 172.21.15.37:0/3646096099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab881c5d10 0x7fab8811bde0 secure :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fab8400d860 tx=0x7fab8400dd20 comp rx=0 tx=0).stop 2024-04-26T00:40:11.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.122+0000 7fab8f306640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:40:11.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.122+0000 7fab8f306640 1 --2- 172.21.15.37:0/3646096099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab8811c320 0x7fab88135c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:11.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.122+0000 7fab8f306640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:11.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.122+0000 7fab8f306640 1 -- 172.21.15.37:0/3646096099 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fab8811c730 con 0x7fab8811c320 2024-04-26T00:40:11.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.122+0000 7fab8d0a2640 1 --2- 172.21.15.37:0/3646096099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab8811c320 0x7fab88135c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:11.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.122+0000 7fab8d0a2640 10 monclient: get_auth_request con 0x7fab8811c320 auth_method 0 2024-04-26T00:40:11.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.122+0000 7fab8d0a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:11.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.122+0000 7fab8d0a2640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:11.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.122+0000 7fab8d0a2640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:11.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.122+0000 7fab8d0a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:11.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.122+0000 7fab8d0a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:11.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.122+0000 7fab8d0a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:11.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.122+0000 7fab8d0a2640 10 monclient(hunting): handle_auth_done global_id 224335 payload 824 2024-04-26T00:40:11.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.122+0000 7fab8d0a2640 1 -- 172.21.15.37:0/3646096099 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fab8813ac00 con 0x7fab8811c320 2024-04-26T00:40:11.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.126+0000 7fab8d0a2640 1 -- 172.21.15.37:0/3646096099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab8811c320 msgr2=0x7fab88135c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:11.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.126+0000 7fab8d0a2640 1 --2- 172.21.15.37:0/3646096099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab8811c320 0x7fab88135c70 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fab84017000 tx=0x7fab84003af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:11.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.126+0000 7fab7dffb640 1 -- 172.21.15.37:0/3646096099 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab8401a3f0 con 0x7fab8811c320 2024-04-26T00:40:11.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.126+0000 7fab7dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:11.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.126+0000 7fab7dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:40:11.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.126+0000 7fab7dffb640 10 monclient: dump: 2024-04-26T00:40:11.130 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:11.130 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:11.130 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:11.130 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:11.130 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:11.130 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:11.130 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:11.130 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:11.130 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:11.130 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:11.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.126+0000 7fab7dffb640 1 -- 172.21.15.37:0/3646096099 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4877 (secure 0 0 0) 0x7fab840243f0 con 0x7fab8811c320 2024-04-26T00:40:11.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.126+0000 7fab7dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:40:11.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.126+0000 7fab7dffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:40:11.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.126+0000 7fab7dffb640 1 -- 172.21.15.37:0/3646096099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab8811c320 msgr2=0x7fab88135c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:11.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.126+0000 7fab7dffb640 1 --2- 172.21.15.37:0/3646096099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab8811c320 0x7fab88135c70 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fab84017000 tx=0x7fab84003af0 comp rx=0 tx=0).stop 2024-04-26T00:40:11.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.126+0000 7fab7dffb640 1 -- 172.21.15.37:0/3646096099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab8811c320 msgr2=0x7fab88135c70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:40:11.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.126+0000 7fab7dffb640 1 --2- 172.21.15.37:0/3646096099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab8811c320 0x7fab88135c70 secure :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fab84017000 tx=0x7fab84003af0 comp rx=0 tx=0).stop 2024-04-26T00:40:11.131 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:40:11.131 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:40:11.131 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:40:11.131 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:40:11.131 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 150, 2024-04-26T00:40:11.131 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 124789, 2024-04-26T00:40:11.131 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:40:11.132 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:40:11.132 INFO:teuthology.orchestra.run.smithi037.stdout: 1 2024-04-26T00:40:11.132 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:11.132 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:40:11.132 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:40:11.132 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:40:11.132 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:40:11.132 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:40:11.132 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:40:11.132 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:40:11.132 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:40:11.132 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:40:11.132 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:40:11.132 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:40:11.133 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:40:11.133 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:40:11.133 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:40:11.133 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:40:11.133 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:11.133 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:40:11.133 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:40:11.133 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:40:11.133 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:40:11.133 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:40:11.133 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:40:11.133 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:40:11.133 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:40:11.133 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:40:11.133 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:40:11.134 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:40:11.134 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:40:11.134 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:40:11.134 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:11.134 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:11.134 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:40:11.134 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:40:11.134 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:40:11.134 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:40:11.134 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:40:11.134 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:40:11.134 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:40:11.134 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:40:11.134 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:40:11.134 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:40:11.135 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:40:11.135 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:40:11.135 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:40:11.135 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:40:11.135 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:40:11.135 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:40:11.135 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:40:11.135 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:40:11.135 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:40:11.135 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:40:11.135 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:40:11.135 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:40:11.135 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:40:11.135 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:40:11.136 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:40:11.136 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:40:11.136 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:40:11.136 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:40:11.136 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:11.136 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:40:11.136 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:11.136 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:40:11.136 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:11.136 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:40:11.136 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:40:11.136 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:40:11.136 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:40:11.136 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:11.137 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:40:11.137 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:40:11.137 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:11.137 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:11.137 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:11.137 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:40:11.137 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:40:11.137 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:11.137 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:11.137 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:11.137 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:11.137 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:40:11.137 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:40:11.137 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:40:11.137 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:40:11.137 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:40:11.138 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:11.138 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:11.138 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:40:11.138 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:40:11.138 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:40:11.138 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:40:11.138 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:11.138 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:40:11.138 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:40:11.138 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:11.138 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:11.138 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:11.139 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:40:11.139 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:40:11.139 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:11.139 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:11.139 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:11.139 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:11.139 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:40:11.139 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:40:11.139 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:40:11.139 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:40:11.139 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:40:11.139 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:11.139 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:11.139 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:40:11.139 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:40:11.140 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:40:11.140 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:40:11.140 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:11.140 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:40:11.140 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:40:11.140 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:11.140 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:11.140 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:11.140 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:40:11.140 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:40:11.140 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:11.140 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:11.140 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:11.140 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:11.141 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:40:11.141 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:40:11.141 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:40:11.141 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:40:11.141 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:40:11.141 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:11.141 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:11.141 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:11.141 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:40:11.141 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:40:11.141 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:11.141 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:40:11.141 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:40:11.141 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:40:11.141 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:11.141 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:11.142 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:40:11.142 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:11.142 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:40:11.142 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:40:11.142 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:40:11.142 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:11.142 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:11.142 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:40:11.142 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:11.142 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:40:11.142 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:40:11.142 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:40:11.142 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:11.142 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:11.142 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:40:11.143 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:11.143 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:40:11.143 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:40:11.143 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:40:11.143 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:11.143 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:11.143 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:11.143 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:40:11.143 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:40:11.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.126+0000 7fab8f306640 1 -- 172.21.15.37:0/3646096099 >> v2:172.21.15.45:6812/2379918623 conn(0x7fab6c035e30 msgr2=0x7fab6c0382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:11.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.126+0000 7fab8f306640 1 --2- 172.21.15.37:0/3646096099 >> v2:172.21.15.45:6812/2379918623 conn(0x7fab6c035e30 0x7fab6c0382f0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fab800ab000 tx=0x7fab80008040 comp rx=0 tx=0).stop 2024-04-26T00:40:11.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.126+0000 7fab8f306640 10 monclient: shutdown 2024-04-26T00:40:11.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.126+0000 7fab8f306640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:40:11.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.126+0000 7fab8f306640 1 -- 172.21.15.37:0/3646096099 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab880a0300 msgr2=0x7fab88134a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:11.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.126+0000 7fab8f306640 1 --2- 172.21.15.37:0/3646096099 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab880a0300 0x7fab88134a20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fab7800c4a0 tx=0x7fab7800c960 comp rx=0 tx=0).stop 2024-04-26T00:40:11.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.126+0000 7fab8d8a3640 1 -- 172.21.15.37:0/3646096099 reap_dead start 2024-04-26T00:40:11.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.126+0000 7fab8d8a3640 1 -- 172.21.15.37:0/3646096099 reap_dead start 2024-04-26T00:40:11.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.126+0000 7fab8f306640 1 -- 172.21.15.37:0/3646096099 shutdown_connections 2024-04-26T00:40:11.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.126+0000 7fab8f306640 1 -- 172.21.15.37:0/3646096099 >> 172.21.15.37:0/3646096099 conn(0x7fab880a1d50 msgr2=0x7fab8813aed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:40:11.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.130+0000 7fab8f306640 1 -- 172.21.15.37:0/3646096099 shutdown_connections 2024-04-26T00:40:11.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:11.130+0000 7fab8f306640 1 -- 172.21.15.37:0/3646096099 wait complete. 2024-04-26T00:40:11.145 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:40:11.145 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-04-26T00:40:11.723 INFO:tasks.rados.rados.0.smithi045.stdout:3504: done (5 left) 2024-04-26T00:40:11.723 INFO:tasks.rados.rados.0.smithi045.stdout:3505: read oid 19 snap -1 2024-04-26T00:40:11.723 INFO:tasks.rados.rados.0.smithi045.stdout:3505: expect (ObjNum 1205 snap 373 seq_num 1205) 2024-04-26T00:40:11.723 INFO:tasks.rados.rados.0.smithi045.stdout:3506: read oid 31 snap -1 2024-04-26T00:40:11.723 INFO:tasks.rados.rados.0.smithi045.stdout:3506: expect deleted 2024-04-26T00:40:11.723 INFO:tasks.rados.rados.0.smithi045.stdout:3507: read oid 12 snap -1 2024-04-26T00:40:11.723 INFO:tasks.rados.rados.0.smithi045.stdout:3507: expect (ObjNum 1112 snap 331 seq_num 1112) 2024-04-26T00:40:11.723 INFO:tasks.rados.rados.0.smithi045.stdout:3508: read oid 11 snap -1 2024-04-26T00:40:11.724 INFO:tasks.rados.rados.0.smithi045.stdout:3508: expect (ObjNum 1139 snap 345 seq_num 1139) 2024-04-26T00:40:11.724 INFO:tasks.rados.rados.0.smithi045.stdout:3509: delete oid 39 current snap is 388 2024-04-26T00:40:11.724 INFO:tasks.rados.rados.0.smithi045.stdout:3499: finishing write tid 2 to smithi04526579-15 2024-04-26T00:40:11.724 INFO:tasks.rados.rados.0.smithi045.stdout:3499: finishing write tid 3 to smithi04526579-15 2024-04-26T00:40:11.724 INFO:tasks.rados.rados.0.smithi045.stdout:3499: finishing write tid 4 to smithi04526579-15 2024-04-26T00:40:11.725 INFO:tasks.rados.rados.0.smithi045.stdout:3499: oid 15 updating version 0 to 2485 2024-04-26T00:40:11.725 INFO:tasks.rados.rados.0.smithi045.stdout:3499: oid 15 version 2485 is already newer than 2484 2024-04-26T00:40:11.725 INFO:tasks.rados.rados.0.smithi045.stdout:3499: oid 15 version 2485 is already newer than 2483 2024-04-26T00:40:11.725 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 2485 (ObjNum 1246 snap 388 seq_num 1246) dirty exists 2024-04-26T00:40:11.725 INFO:tasks.rados.rados.0.smithi045.stdout:3499: left oid 15 (ObjNum 1246 snap 388 seq_num 1246) 2024-04-26T00:40:11.725 INFO:tasks.rados.rados.0.smithi045.stdout:3500: expect (ObjNum 1062 snap 318 seq_num 1062) 2024-04-26T00:40:12.002 INFO:tasks.rados.rados.0.smithi045.stdout:3501: expect (ObjNum 1075 snap 322 seq_num 1075) 2024-04-26T00:40:12.101 INFO:tasks.rados.rados.0.smithi045.stdout:3502: finishing write tid 1 to smithi04526579-29 2024-04-26T00:40:12.101 INFO:tasks.rados.rados.0.smithi045.stdout:3502: finishing write tid 2 to smithi04526579-29 2024-04-26T00:40:12.101 INFO:tasks.rados.rados.0.smithi045.stdout:3502: finishing write tid 3 to smithi04526579-29 2024-04-26T00:40:12.101 INFO:tasks.rados.rados.0.smithi045.stdout:3502: finishing write tid 4 to smithi04526579-29 2024-04-26T00:40:12.102 INFO:tasks.rados.rados.0.smithi045.stdout:3502: oid 29 updating version 0 to 2227 2024-04-26T00:40:12.102 INFO:tasks.rados.rados.0.smithi045.stdout:3502: oid 29 updating version 2227 to 2228 2024-04-26T00:40:12.102 INFO:tasks.rados.rados.0.smithi045.stdout:3502: oid 29 version 2228 is already newer than 2226 2024-04-26T00:40:12.102 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 2228 (ObjNum 1247 snap 388 seq_num 1247) dirty exists 2024-04-26T00:40:12.102 INFO:tasks.rados.rados.0.smithi045.stdout:3502: left oid 29 (ObjNum 1247 snap 388 seq_num 1247) 2024-04-26T00:40:12.102 INFO:tasks.rados.rados.0.smithi045.stdout:3508: expect (ObjNum 1139 snap 345 seq_num 1139) 2024-04-26T00:40:12.300 INFO:tasks.rados.rados.0.smithi045.stdout:3507: expect (ObjNum 1112 snap 331 seq_num 1112) 2024-04-26T00:40:12.458 INFO:tasks.rados.rados.0.smithi045.stdout:3505: expect (ObjNum 1205 snap 373 seq_num 1205) 2024-04-26T00:40:12.588 INFO:tasks.rados.rados.0.smithi045.stdout:3499: done (9 left) 2024-04-26T00:40:12.588 INFO:tasks.rados.rados.0.smithi045.stdout:3500: done (8 left) 2024-04-26T00:40:12.588 INFO:tasks.rados.rados.0.smithi045.stdout:3501: done (7 left) 2024-04-26T00:40:12.588 INFO:tasks.rados.rados.0.smithi045.stdout:3502: done (6 left) 2024-04-26T00:40:12.589 INFO:tasks.rados.rados.0.smithi045.stdout:3503: done (5 left) 2024-04-26T00:40:12.589 INFO:tasks.rados.rados.0.smithi045.stdout:3505: done (4 left) 2024-04-26T00:40:12.589 INFO:tasks.rados.rados.0.smithi045.stdout:3506: done (3 left) 2024-04-26T00:40:12.589 INFO:tasks.rados.rados.0.smithi045.stdout:3507: done (2 left) 2024-04-26T00:40:12.589 INFO:tasks.rados.rados.0.smithi045.stdout:3508: done (1 left) 2024-04-26T00:40:12.589 INFO:tasks.rados.rados.0.smithi045.stdout:3509: done (0 left) 2024-04-26T00:40:12.589 INFO:tasks.rados.rados.0.smithi045.stdout:3510: rollback oid 46 current snap is 388 2024-04-26T00:40:12.589 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 46 to 361 2024-04-26T00:40:12.589 INFO:tasks.rados.rados.0.smithi045.stdout:3511: read oid 28 snap -1 2024-04-26T00:40:12.589 INFO:tasks.rados.rados.0.smithi045.stdout:3511: expect deleted 2024-04-26T00:40:12.589 INFO:tasks.rados.rados.0.smithi045.stdout:3512: rollback oid 41 current snap is 388 2024-04-26T00:40:12.589 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 41 to 386 2024-04-26T00:40:12.590 INFO:tasks.rados.rados.0.smithi045.stdout:3513: read oid 44 snap -1 2024-04-26T00:40:12.590 INFO:tasks.rados.rados.0.smithi045.stdout:3513: expect (ObjNum 1127 snap 340 seq_num 1127) 2024-04-26T00:40:12.590 INFO:tasks.rados.rados.0.smithi045.stdout:3514: read oid 50 snap -1 2024-04-26T00:40:12.590 INFO:tasks.rados.rados.0.smithi045.stdout:3514: expect (ObjNum 1207 snap 373 seq_num 1207) 2024-04-26T00:40:12.590 INFO:tasks.rados.rados.0.smithi045.stdout:3515: delete oid 9 current snap is 388 2024-04-26T00:40:12.596 INFO:tasks.rados.rados.0.smithi045.stdout:3511: done (5 left) 2024-04-26T00:40:12.596 INFO:tasks.rados.rados.0.smithi045.stdout:3515: done (4 left) 2024-04-26T00:40:12.596 INFO:tasks.rados.rados.0.smithi045.stdout:3516: write (excl) oid 2 current snap is 388 2024-04-26T00:40:12.596 INFO:tasks.rados.rados.0.smithi045.stdout:3516: seq_num 1250 ranges {608750=743142,1780374=642769} 2024-04-26T00:40:12.606 INFO:tasks.rados.rados.0.smithi045.stdout:3516: writing smithi04526579-2 from 608750 to 1351892 tid 1 2024-04-26T00:40:12.622 INFO:tasks.rados.rados.0.smithi045.stdout:3516: writing smithi04526579-2 from 1780374 to 2423143 tid 2 2024-04-26T00:40:12.622 INFO:tasks.rados.rados.0.smithi045.stdout:3517: rollback oid 49 current snap is 388 2024-04-26T00:40:12.622 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 49 to 362 2024-04-26T00:40:12.622 INFO:tasks.rados.rados.0.smithi045.stdout:3514: expect (ObjNum 1207 snap 373 seq_num 1207) 2024-04-26T00:40:12.803 INFO:tasks.rados.rados.0.smithi045.stdout:3512: finishing rollback tid 0 to smithi04526579-41 2024-04-26T00:40:12.803 INFO:tasks.rados.rados.0.smithi045.stdout:3512: finishing rollback tid 1 to smithi04526579-41 2024-04-26T00:40:12.803 INFO:tasks.rados.rados.0.smithi045.stdout:3512: finishing rollback tid 2 to smithi04526579-41 2024-04-26T00:40:12.803 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 2488 (ObjNum 1189 snap 369 seq_num 1189) dirty exists 2024-04-26T00:40:12.803 INFO:tasks.rados.rados.0.smithi045.stdout:3510: finishing rollback tid 0 to smithi04526579-46 2024-04-26T00:40:12.803 INFO:tasks.rados.rados.0.smithi045.stdout:3510: finishing rollback tid 1 to smithi04526579-46 2024-04-26T00:40:12.803 INFO:tasks.rados.rados.0.smithi045.stdout:3510: finishing rollback tid 2 to smithi04526579-46 2024-04-26T00:40:12.803 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 2450 (ObjNum 1136 snap 343 seq_num 1136) dirty exists 2024-04-26T00:40:12.804 INFO:tasks.rados.rados.0.smithi045.stdout:3513: expect (ObjNum 1127 snap 340 seq_num 1127) 2024-04-26T00:40:12.985 INFO:tasks.rados.rados.0.smithi045.stdout:3516: finishing write tid 1 to smithi04526579-2 2024-04-26T00:40:12.985 INFO:tasks.rados.rados.0.smithi045.stdout:3516: finishing write tid 2 to smithi04526579-2 2024-04-26T00:40:12.986 INFO:tasks.rados.rados.0.smithi045.stdout:3516: finishing write tid 3 to smithi04526579-2 2024-04-26T00:40:12.986 INFO:tasks.rados.rados.0.smithi045.stdout:3516: finishing write tid 4 to smithi04526579-2 2024-04-26T00:40:12.986 INFO:tasks.rados.rados.0.smithi045.stdout:3516: oid 2 updating version 0 to 2505 2024-04-26T00:40:12.986 INFO:tasks.rados.rados.0.smithi045.stdout:3516: oid 2 version 2505 is already newer than 2504 2024-04-26T00:40:12.986 INFO:tasks.rados.rados.0.smithi045.stdout:3516: oid 2 version 2505 is already newer than 2503 2024-04-26T00:40:12.986 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 2505 (ObjNum 1250 snap 388 seq_num 1250) dirty exists 2024-04-26T00:40:12.986 INFO:tasks.rados.rados.0.smithi045.stdout:3516: left oid 2 (ObjNum 1250 snap 388 seq_num 1250) 2024-04-26T00:40:12.986 INFO:tasks.rados.rados.0.smithi045.stdout:3517: finishing rollback tid 0 to smithi04526579-49 2024-04-26T00:40:12.986 INFO:tasks.rados.rados.0.smithi045.stdout:3517: finishing rollback tid 1 to smithi04526579-49 2024-04-26T00:40:12.986 INFO:tasks.rados.rados.0.smithi045.stdout:3517: finishing rollback tid 2 to smithi04526579-49 2024-04-26T00:40:12.986 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 2479 (ObjNum 1162 snap 359 seq_num 1162) dirty exists 2024-04-26T00:40:12.986 INFO:tasks.rados.rados.0.smithi045.stdout:3510: done (5 left) 2024-04-26T00:40:12.986 INFO:tasks.rados.rados.0.smithi045.stdout:3512: done (4 left) 2024-04-26T00:40:12.987 INFO:tasks.rados.rados.0.smithi045.stdout:3513: done (3 left) 2024-04-26T00:40:12.987 INFO:tasks.rados.rados.0.smithi045.stdout:3514: done (2 left) 2024-04-26T00:40:12.987 INFO:tasks.rados.rados.0.smithi045.stdout:3516: done (1 left) 2024-04-26T00:40:12.987 INFO:tasks.rados.rados.0.smithi045.stdout:3517: done (0 left) 2024-04-26T00:40:12.987 INFO:tasks.rados.rados.0.smithi045.stdout:3518: delete oid 21 current snap is 388 2024-04-26T00:40:12.989 INFO:tasks.rados.rados.0.smithi045.stdout:3518: done (0 left) 2024-04-26T00:40:12.989 INFO:tasks.rados.rados.0.smithi045.stdout:3519: snap_create 2024-04-26T00:40:13.734 INFO:tasks.rados.rados.0.smithi045.stdout:3519: done (0 left) 2024-04-26T00:40:13.734 INFO:tasks.rados.rados.0.smithi045.stdout:3520: copy_from oid 4 from oid 23 current snap is 389 2024-04-26T00:40:13.734 INFO:tasks.rados.rados.0.smithi045.stdout:3521: snap_remove snap 362 2024-04-26T00:40:14.762 INFO:tasks.rados.rados.0.smithi045.stdout:3521: done (1 left) 2024-04-26T00:40:14.762 INFO:tasks.rados.rados.0.smithi045.stdout:3522: read oid 32 snap -1 2024-04-26T00:40:14.762 INFO:tasks.rados.rados.0.smithi045.stdout:3522: expect deleted 2024-04-26T00:40:14.762 INFO:tasks.rados.rados.0.smithi045.stdout:3523: write (excl) oid 30 current snap is 389 2024-04-26T00:40:14.762 INFO:tasks.rados.rados.0.smithi045.stdout:3520: finishing copy_from to smithi04526579-4 2024-04-26T00:40:14.762 INFO:tasks.rados.rados.0.smithi045.stdout:3520: got expected ENOENT (src dne) 2024-04-26T00:40:14.762 INFO:tasks.rados.rados.0.smithi045.stdout:3520: finishing copy_from racing read to smithi04526579-4 2024-04-26T00:40:14.762 INFO:tasks.rados.rados.0.smithi045.stdout:3523: seq_num 1252 ranges {642364=493640,1897338=646438} 2024-04-26T00:40:14.769 INFO:tasks.rados.rados.0.smithi045.stdout:3523: writing smithi04526579-30 from 642364 to 1136004 tid 1 2024-04-26T00:40:14.779 INFO:tasks.rados.rados.0.smithi045.stdout:3523: writing smithi04526579-30 from 1897338 to 2543776 tid 2 2024-04-26T00:40:14.779 INFO:tasks.rados.rados.0.smithi045.stdout:3520: done (2 left) 2024-04-26T00:40:14.779 INFO:tasks.rados.rados.0.smithi045.stdout:3524: snap_create 2024-04-26T00:40:14.786 INFO:tasks.rados.rados.0.smithi045.stdout:3523: finishing write tid 1 to smithi04526579-30 2024-04-26T00:40:14.789 INFO:tasks.rados.rados.0.smithi045.stdout:3523: finishing write tid 2 to smithi04526579-30 2024-04-26T00:40:14.790 INFO:tasks.rados.rados.0.smithi045.stdout:3523: finishing write tid 3 to smithi04526579-30 2024-04-26T00:40:14.790 INFO:tasks.rados.rados.0.smithi045.stdout:3523: finishing write tid 4 to smithi04526579-30 2024-04-26T00:40:14.790 INFO:tasks.rados.rados.0.smithi045.stdout:3523: oid 30 updating version 0 to 2014 2024-04-26T00:40:14.790 INFO:tasks.rados.rados.0.smithi045.stdout:3523: oid 30 version 2014 is already newer than 2012 2024-04-26T00:40:14.790 INFO:tasks.rados.rados.0.smithi045.stdout:3523: oid 30 version 2014 is already newer than 2013 2024-04-26T00:40:14.791 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 2014 (ObjNum 1252 snap 389 seq_num 1252) dirty exists 2024-04-26T00:40:14.791 INFO:tasks.rados.rados.0.smithi045.stdout:3523: left oid 30 (ObjNum 1252 snap 389 seq_num 1252) 2024-04-26T00:40:15.769 INFO:tasks.rados.rados.0.smithi045.stdout:3522: done (2 left) 2024-04-26T00:40:15.769 INFO:tasks.rados.rados.0.smithi045.stdout:3523: done (1 left) 2024-04-26T00:40:15.769 INFO:tasks.rados.rados.0.smithi045.stdout:3524: done (0 left) 2024-04-26T00:40:15.770 INFO:tasks.rados.rados.0.smithi045.stdout:3525: read oid 9 snap 361 2024-04-26T00:40:15.770 INFO:tasks.rados.rados.0.smithi045.stdout:3525: expect (ObjNum 1099 snap 326 seq_num 1099) 2024-04-26T00:40:15.770 INFO:tasks.rados.rados.0.smithi045.stdout:3526: rollback oid 22 current snap is 390 2024-04-26T00:40:15.770 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 22 to 361 2024-04-26T00:40:15.770 INFO:tasks.rados.rados.0.smithi045.stdout:3527: copy_from oid 3 from oid 15 current snap is 390 2024-04-26T00:40:15.771 INFO:tasks.rados.rados.0.smithi045.stdout:3528: read oid 48 snap -1 2024-04-26T00:40:15.771 INFO:tasks.rados.rados.0.smithi045.stdout:3528: expect (ObjNum 1239 snap 384 seq_num 1239) 2024-04-26T00:40:15.771 INFO:tasks.rados.rados.0.smithi045.stdout:3529: read oid 23 snap 383 2024-04-26T00:40:15.771 INFO:tasks.rados.rados.0.smithi045.stdout:3529: expect deleted 2024-04-26T00:40:15.771 INFO:tasks.rados.rados.0.smithi045.stdout:3530: read oid 26 snap -1 2024-04-26T00:40:15.771 INFO:tasks.rados.rados.0.smithi045.stdout:3530: expect deleted 2024-04-26T00:40:15.771 INFO:tasks.rados.rados.0.smithi045.stdout:3531: snap_create 2024-04-26T00:40:15.783 INFO:tasks.rados.rados.0.smithi045.stdout:3525: expect (ObjNum 1099 snap 326 seq_num 1099) 2024-04-26T00:40:15.916 INFO:tasks.rados.rados.0.smithi045.stdout:3526: finishing rollback tid 0 to smithi04526579-22 2024-04-26T00:40:15.917 INFO:tasks.rados.rados.0.smithi045.stdout:3526: finishing rollback tid 1 to smithi04526579-22 2024-04-26T00:40:15.917 INFO:tasks.rados.rados.0.smithi045.stdout:3526: finishing rollback tid 2 to smithi04526579-22 2024-04-26T00:40:15.917 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 2453 (ObjNum 1135 snap 343 seq_num 1135) dirty exists 2024-04-26T00:40:15.917 INFO:tasks.rados.rados.0.smithi045.stdout:3528: expect (ObjNum 1239 snap 384 seq_num 1239) 2024-04-26T00:40:16.159 INFO:tasks.rados.rados.0.smithi045.stdout:3527: finishing copy_from racing read to smithi04526579-3 2024-04-26T00:40:16.159 INFO:tasks.rados.rados.0.smithi045.stdout:3527: finishing copy_from to smithi04526579-3 2024-04-26T00:40:16.159 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 2485 (ObjNum 1246 snap 388 seq_num 1246) dirty exists 2024-04-26T00:40:16.775 INFO:tasks.rados.rados.0.smithi045.stdout:3525: done (6 left) 2024-04-26T00:40:16.775 INFO:tasks.rados.rados.0.smithi045.stdout:3526: done (5 left) 2024-04-26T00:40:16.775 INFO:tasks.rados.rados.0.smithi045.stdout:3527: done (4 left) 2024-04-26T00:40:16.775 INFO:tasks.rados.rados.0.smithi045.stdout:3528: done (3 left) 2024-04-26T00:40:16.775 INFO:tasks.rados.rados.0.smithi045.stdout:3529: done (2 left) 2024-04-26T00:40:16.775 INFO:tasks.rados.rados.0.smithi045.stdout:3530: done (1 left) 2024-04-26T00:40:16.775 INFO:tasks.rados.rados.0.smithi045.stdout:3531: done (0 left) 2024-04-26T00:40:16.775 INFO:tasks.rados.rados.0.smithi045.stdout:3532: read oid 34 snap 361 2024-04-26T00:40:16.775 INFO:tasks.rados.rados.0.smithi045.stdout:3532: expect (ObjNum 1112 snap 331 seq_num 1112) 2024-04-26T00:40:16.775 INFO:tasks.rados.rados.0.smithi045.stdout:3533: write (excl) oid 46 current snap is 391 2024-04-26T00:40:16.775 INFO:tasks.rados.rados.0.smithi045.stdout:3533: seq_num 1253 ranges {659171=568889,1839467=790096} 2024-04-26T00:40:16.794 INFO:tasks.rados.rados.0.smithi045.stdout:3533: writing smithi04526579-46 from 659171 to 1228060 tid 1 2024-04-26T00:40:16.819 INFO:tasks.rados.rados.0.smithi045.stdout:3533: writing smithi04526579-46 from 1839467 to 2629563 tid 2 2024-04-26T00:40:16.819 INFO:tasks.rados.rados.0.smithi045.stdout:3534: copy_from oid 21 from oid 1 current snap is 391 2024-04-26T00:40:16.819 INFO:tasks.rados.rados.0.smithi045.stdout:3532: expect (ObjNum 1112 snap 331 seq_num 1112) 2024-04-26T00:40:16.997 INFO:tasks.rados.rados.0.smithi045.stdout:3533: finishing write tid 1 to smithi04526579-46 2024-04-26T00:40:16.998 INFO:tasks.rados.rados.0.smithi045.stdout:3533: finishing write tid 2 to smithi04526579-46 2024-04-26T00:40:16.998 INFO:tasks.rados.rados.0.smithi045.stdout:3533: finishing write tid 3 to smithi04526579-46 2024-04-26T00:40:16.998 INFO:tasks.rados.rados.0.smithi045.stdout:3533: finishing write tid 4 to smithi04526579-46 2024-04-26T00:40:16.998 INFO:tasks.rados.rados.0.smithi045.stdout:3533: oid 46 updating version 0 to 2455 2024-04-26T00:40:16.998 INFO:tasks.rados.rados.0.smithi045.stdout:3533: oid 46 version 2455 is already newer than 2454 2024-04-26T00:40:16.998 INFO:tasks.rados.rados.0.smithi045.stdout:3533: oid 46 updating version 2455 to 2456 2024-04-26T00:40:16.998 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 2456 (ObjNum 1253 snap 391 seq_num 1253) dirty exists 2024-04-26T00:40:16.998 INFO:tasks.rados.rados.0.smithi045.stdout:3533: left oid 46 (ObjNum 1253 snap 391 seq_num 1253) 2024-04-26T00:40:16.998 INFO:tasks.rados.rados.0.smithi045.stdout:3534: finishing copy_from racing read to smithi04526579-21 2024-04-26T00:40:16.998 INFO:tasks.rados.rados.0.smithi045.stdout:3534: finishing copy_from to smithi04526579-21 2024-04-26T00:40:16.998 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 2478 (ObjNum 1079 snap 322 seq_num 1079) dirty exists 2024-04-26T00:40:16.999 INFO:tasks.rados.rados.0.smithi045.stdout:3532: done (2 left) 2024-04-26T00:40:16.999 INFO:tasks.rados.rados.0.smithi045.stdout:3533: done (1 left) 2024-04-26T00:40:16.999 INFO:tasks.rados.rados.0.smithi045.stdout:3534: done (0 left) 2024-04-26T00:40:16.999 INFO:tasks.rados.rados.0.smithi045.stdout:3535: write (excl) oid 19 current snap is 391 2024-04-26T00:40:16.999 INFO:tasks.rados.rados.0.smithi045.stdout:3535: seq_num 1254 ranges {675978=644138,1897949=650107} 2024-04-26T00:40:17.018 INFO:tasks.rados.rados.0.smithi045.stdout:3535: writing smithi04526579-19 from 675978 to 1320116 tid 1 2024-04-26T00:40:17.040 INFO:tasks.rados.rados.0.smithi045.stdout:3535: writing smithi04526579-19 from 1897949 to 2548056 tid 2 2024-04-26T00:40:17.040 INFO:tasks.rados.rados.0.smithi045.stdout:3536: read oid 34 snap -1 2024-04-26T00:40:17.040 INFO:tasks.rados.rados.0.smithi045.stdout:3536: expect (ObjNum 1160 snap 359 seq_num 1160) 2024-04-26T00:40:17.040 INFO:tasks.rados.rados.0.smithi045.stdout:3537: read oid 49 snap 381 2024-04-26T00:40:17.040 INFO:tasks.rados.rados.0.smithi045.stdout:3537: expect (ObjNum 1162 snap 359 seq_num 1162) 2024-04-26T00:40:17.040 INFO:tasks.rados.rados.0.smithi045.stdout:3538: rollback oid 16 current snap is 391 2024-04-26T00:40:17.040 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 16 to 390 2024-04-26T00:40:17.040 INFO:tasks.rados.rados.0.smithi045.stdout:3535: finishing write tid 1 to smithi04526579-19 2024-04-26T00:40:17.040 INFO:tasks.rados.rados.0.smithi045.stdout:3539: delete oid 15 current snap is 391 2024-04-26T00:40:17.046 INFO:tasks.rados.rados.0.smithi045.stdout:3539: done (4 left) 2024-04-26T00:40:17.046 INFO:tasks.rados.rados.0.smithi045.stdout:3540: write oid 9 current snap is 391 2024-04-26T00:40:17.046 INFO:tasks.rados.rados.0.smithi045.stdout:3540: seq_num 1256 ranges {709592=510989,1731266=653776,3041279=68313} 2024-04-26T00:40:17.069 INFO:tasks.rados.rados.0.smithi045.stdout:3540: writing smithi04526579-9 from 709592 to 1220581 tid 1 2024-04-26T00:40:17.087 INFO:tasks.rados.rados.0.smithi045.stdout:3540: writing smithi04526579-9 from 1731266 to 2385042 tid 2 2024-04-26T00:40:17.096 INFO:tasks.rados.rados.0.smithi045.stdout:3540: writing smithi04526579-9 from 3041279 to 3109592 tid 3 2024-04-26T00:40:17.096 INFO:tasks.rados.rados.0.smithi045.stdout:3541: read oid 48 snap -1 2024-04-26T00:40:17.096 INFO:tasks.rados.rados.0.smithi045.stdout:3541: expect (ObjNum 1239 snap 384 seq_num 1239) 2024-04-26T00:40:17.096 INFO:tasks.rados.rados.0.smithi045.stdout:3542: snap_create 2024-04-26T00:40:17.096 INFO:tasks.rados.rados.0.smithi045.stdout:3535: finishing write tid 2 to smithi04526579-19 2024-04-26T00:40:17.097 INFO:tasks.rados.rados.0.smithi045.stdout:3535: finishing write tid 3 to smithi04526579-19 2024-04-26T00:40:17.097 INFO:tasks.rados.rados.0.smithi045.stdout:3535: finishing write tid 4 to smithi04526579-19 2024-04-26T00:40:17.097 INFO:tasks.rados.rados.0.smithi045.stdout:3535: oid 19 updating version 0 to 2508 2024-04-26T00:40:17.097 INFO:tasks.rados.rados.0.smithi045.stdout:3535: oid 19 version 2508 is already newer than 2507 2024-04-26T00:40:17.097 INFO:tasks.rados.rados.0.smithi045.stdout:3535: oid 19 version 2508 is already newer than 2506 2024-04-26T00:40:17.097 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 2508 (ObjNum 1254 snap 391 seq_num 1254) dirty exists 2024-04-26T00:40:17.097 INFO:tasks.rados.rados.0.smithi045.stdout:3535: left oid 19 (ObjNum 1254 snap 391 seq_num 1254) 2024-04-26T00:40:17.097 INFO:tasks.rados.rados.0.smithi045.stdout:3538: finishing rollback tid 0 to smithi04526579-16 2024-04-26T00:40:17.097 INFO:tasks.rados.rados.0.smithi045.stdout:3538: finishing rollback tid 1 to smithi04526579-16 2024-04-26T00:40:17.097 INFO:tasks.rados.rados.0.smithi045.stdout:3536: expect (ObjNum 1160 snap 359 seq_num 1160) 2024-04-26T00:40:17.371 INFO:tasks.rados.rados.0.smithi045.stdout:3538: finishing rollback tid 2 to smithi04526579-16 2024-04-26T00:40:17.372 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 1210 (ObjNum 1102 snap 329 seq_num 1102) dirty exists 2024-04-26T00:40:17.372 INFO:tasks.rados.rados.0.smithi045.stdout:3537: expect (ObjNum 1162 snap 359 seq_num 1162) 2024-04-26T00:40:17.488 INFO:tasks.rados.rados.0.smithi045.stdout:3540: finishing write tid 1 to smithi04526579-9 2024-04-26T00:40:17.488 INFO:tasks.rados.rados.0.smithi045.stdout:3540: finishing write tid 2 to smithi04526579-9 2024-04-26T00:40:17.488 INFO:tasks.rados.rados.0.smithi045.stdout:3540: finishing write tid 3 to smithi04526579-9 2024-04-26T00:40:17.488 INFO:tasks.rados.rados.0.smithi045.stdout:3540: finishing write tid 4 to smithi04526579-9 2024-04-26T00:40:17.488 INFO:tasks.rados.rados.0.smithi045.stdout:3540: finishing write tid 5 to smithi04526579-9 2024-04-26T00:40:17.488 INFO:tasks.rados.rados.0.smithi045.stdout:3540: oid 9 updating version 0 to 2506 2024-04-26T00:40:17.488 INFO:tasks.rados.rados.0.smithi045.stdout:3540: oid 9 updating version 2506 to 2509 2024-04-26T00:40:17.488 INFO:tasks.rados.rados.0.smithi045.stdout:3540: oid 9 version 2509 is already newer than 2507 2024-04-26T00:40:17.488 INFO:tasks.rados.rados.0.smithi045.stdout:3540: oid 9 version 2509 is already newer than 2508 2024-04-26T00:40:17.488 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 2509 (ObjNum 1256 snap 391 seq_num 1256) dirty exists 2024-04-26T00:40:17.488 INFO:tasks.rados.rados.0.smithi045.stdout:3540: left oid 9 (ObjNum 1256 snap 391 seq_num 1256) 2024-04-26T00:40:17.488 INFO:tasks.rados.rados.0.smithi045.stdout:3541: expect (ObjNum 1239 snap 384 seq_num 1239) 2024-04-26T00:40:17.735 INFO:tasks.rados.rados.0.smithi045.stdout:3535: done (6 left) 2024-04-26T00:40:17.735 INFO:tasks.rados.rados.0.smithi045.stdout:3536: done (5 left) 2024-04-26T00:40:17.736 INFO:tasks.rados.rados.0.smithi045.stdout:3537: done (4 left) 2024-04-26T00:40:17.736 INFO:tasks.rados.rados.0.smithi045.stdout:3538: done (3 left) 2024-04-26T00:40:17.736 INFO:tasks.rados.rados.0.smithi045.stdout:3540: done (2 left) 2024-04-26T00:40:17.736 INFO:tasks.rados.rados.0.smithi045.stdout:3541: done (1 left) 2024-04-26T00:40:17.736 INFO:tasks.rados.rados.0.smithi045.stdout:3542: done (0 left) 2024-04-26T00:40:17.736 INFO:tasks.rados.rados.0.smithi045.stdout:3543: snap_remove snap 386 2024-04-26T00:40:18.673 INFO:tasks.rados.rados.0.smithi045.stdout:3543: done (0 left) 2024-04-26T00:40:18.673 INFO:tasks.rados.rados.0.smithi045.stdout:3544: write (excl) oid 9 current snap is 392 2024-04-26T00:40:18.673 INFO:tasks.rados.rados.0.smithi045.stdout:3544: seq_num 1257 ranges {726399=586238,2073395=797434} 2024-04-26T00:40:18.696 INFO:tasks.rados.rados.0.smithi045.stdout:3544: writing smithi04526579-9 from 726399 to 1312637 tid 1 2024-04-26T00:40:18.721 INFO:tasks.rados.rados.0.smithi045.stdout:3544: writing smithi04526579-9 from 2073395 to 2870829 tid 2 2024-04-26T00:40:18.721 INFO:tasks.rados.rados.0.smithi045.stdout:3545: rollback oid 6 current snap is 392 2024-04-26T00:40:18.722 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 6 to 381 2024-04-26T00:40:18.722 INFO:tasks.rados.rados.0.smithi045.stdout:3544: finishing write tid 1 to smithi04526579-9 2024-04-26T00:40:18.722 INFO:tasks.rados.rados.0.smithi045.stdout:3546: delete oid 44 current snap is 392 2024-04-26T00:40:18.733 INFO:tasks.rados.rados.0.smithi045.stdout:3546: done (2 left) 2024-04-26T00:40:18.733 INFO:tasks.rados.rados.0.smithi045.stdout:3547: read oid 24 snap -1 2024-04-26T00:40:18.733 INFO:tasks.rados.rados.0.smithi045.stdout:3547: expect (ObjNum 1211 snap 375 seq_num 1211) 2024-04-26T00:40:18.733 INFO:tasks.rados.rados.0.smithi045.stdout:3548: delete oid 7 current snap is 392 2024-04-26T00:40:18.733 INFO:tasks.rados.rados.0.smithi045.stdout:3545: finishing rollback tid 0 to smithi04526579-6 2024-04-26T00:40:18.735 INFO:tasks.rados.rados.0.smithi045.stdout:3544: finishing write tid 2 to smithi04526579-9 2024-04-26T00:40:18.735 INFO:tasks.rados.rados.0.smithi045.stdout:3545: finishing rollback tid 1 to smithi04526579-6 2024-04-26T00:40:18.735 INFO:tasks.rados.rados.0.smithi045.stdout:3545: finishing rollback tid 2 to smithi04526579-6 2024-04-26T00:40:18.736 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 2017 (ObjNum 1060 snap 318 seq_num 1060) dirty exists 2024-04-26T00:40:18.736 INFO:tasks.rados.rados.0.smithi045.stdout:3544: finishing write tid 3 to smithi04526579-9 2024-04-26T00:40:18.739 INFO:tasks.rados.rados.0.smithi045.stdout:3544: finishing write tid 4 to smithi04526579-9 2024-04-26T00:40:18.740 INFO:tasks.rados.rados.0.smithi045.stdout:3544: oid 9 updating version 0 to 2512 2024-04-26T00:40:18.740 INFO:tasks.rados.rados.0.smithi045.stdout:3544: oid 9 version 2512 is already newer than 2511 2024-04-26T00:40:18.740 INFO:tasks.rados.rados.0.smithi045.stdout:3544: oid 9 version 2512 is already newer than 2510 2024-04-26T00:40:18.740 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 2512 (ObjNum 1257 snap 392 seq_num 1257) dirty exists 2024-04-26T00:40:18.740 INFO:tasks.rados.rados.0.smithi045.stdout:3544: left oid 9 (ObjNum 1257 snap 392 seq_num 1257) 2024-04-26T00:40:18.742 INFO:tasks.rados.rados.0.smithi045.stdout:3547: expect (ObjNum 1211 snap 375 seq_num 1211) 2024-04-26T00:40:18.910 INFO:tasks.rados.rados.0.smithi045.stdout:3544: done (3 left) 2024-04-26T00:40:18.910 INFO:tasks.rados.rados.0.smithi045.stdout:3545: done (2 left) 2024-04-26T00:40:18.911 INFO:tasks.rados.rados.0.smithi045.stdout:3547: done (1 left) 2024-04-26T00:40:18.911 INFO:tasks.rados.rados.0.smithi045.stdout:3548: done (0 left) 2024-04-26T00:40:18.911 INFO:tasks.rados.rados.0.smithi045.stdout:3549: read oid 38 snap -1 2024-04-26T00:40:18.911 INFO:tasks.rados.rados.0.smithi045.stdout:3549: expect (ObjNum 1075 snap 322 seq_num 1075) 2024-04-26T00:40:18.911 INFO:tasks.rados.rados.0.smithi045.stdout:3550: snap_create 2024-04-26T00:40:18.919 INFO:tasks.rados.rados.0.smithi045.stdout:3549: expect (ObjNum 1075 snap 322 seq_num 1075) 2024-04-26T00:40:19.793 INFO:tasks.rados.rados.0.smithi045.stdout:3549: done (1 left) 2024-04-26T00:40:19.793 INFO:tasks.rados.rados.0.smithi045.stdout:3550: done (0 left) 2024-04-26T00:40:19.793 INFO:tasks.rados.rados.0.smithi045.stdout:3551: write oid 21 current snap is 393 2024-04-26T00:40:19.793 INFO:tasks.rados.rados.0.smithi045.stdout:3551: seq_num 1260 ranges {776820=411985,1848841=544761,2918265=258555} 2024-04-26T00:40:19.812 INFO:tasks.rados.rados.0.smithi045.stdout:3551: writing smithi04526579-21 from 776820 to 1188805 tid 1 2024-04-26T00:40:19.831 INFO:tasks.rados.rados.0.smithi045.stdout:3551: writing smithi04526579-21 from 1848841 to 2393602 tid 2 2024-04-26T00:40:19.844 INFO:tasks.rados.rados.0.smithi045.stdout:3551: writing smithi04526579-21 from 2918265 to 3176820 tid 3 2024-04-26T00:40:19.845 INFO:tasks.rados.rados.0.smithi045.stdout:3552: copy_from oid 23 from oid 39 current snap is 393 2024-04-26T00:40:19.845 INFO:tasks.rados.rados.0.smithi045.stdout:3551: finishing write tid 1 to smithi04526579-21 2024-04-26T00:40:19.845 INFO:tasks.rados.rados.0.smithi045.stdout:3551: finishing write tid 2 to smithi04526579-21 2024-04-26T00:40:19.845 INFO:tasks.rados.rados.0.smithi045.stdout:3553: snap_remove snap 376 2024-04-26T00:40:20.799 INFO:tasks.rados.rados.0.smithi045.stdout:3553: done (2 left) 2024-04-26T00:40:20.799 INFO:tasks.rados.rados.0.smithi045.stdout:3554: read oid 8 snap -1 2024-04-26T00:40:20.799 INFO:tasks.rados.rados.0.smithi045.stdout:3554: expect (ObjNum 1232 snap 383 seq_num 1232) 2024-04-26T00:40:20.799 INFO:tasks.rados.rados.0.smithi045.stdout:3555: delete oid 5 current snap is 393 2024-04-26T00:40:20.800 INFO:tasks.rados.rados.0.smithi045.stdout:3552: finishing copy_from to smithi04526579-23 2024-04-26T00:40:20.800 INFO:tasks.rados.rados.0.smithi045.stdout:3552: got expected ENOENT (src dne) 2024-04-26T00:40:20.800 INFO:tasks.rados.rados.0.smithi045.stdout:3552: finishing copy_from racing read to smithi04526579-23 2024-04-26T00:40:20.800 INFO:tasks.rados.rados.0.smithi045.stdout:3551: finishing write tid 3 to smithi04526579-21 2024-04-26T00:40:20.800 INFO:tasks.rados.rados.0.smithi045.stdout:3551: finishing write tid 4 to smithi04526579-21 2024-04-26T00:40:20.800 INFO:tasks.rados.rados.0.smithi045.stdout:3551: finishing write tid 5 to smithi04526579-21 2024-04-26T00:40:20.800 INFO:tasks.rados.rados.0.smithi045.stdout:3551: oid 21 updating version 0 to 2481 2024-04-26T00:40:20.800 INFO:tasks.rados.rados.0.smithi045.stdout:3551: oid 21 updating version 2481 to 2482 2024-04-26T00:40:20.800 INFO:tasks.rados.rados.0.smithi045.stdout:3551: oid 21 version 2482 is already newer than 2479 2024-04-26T00:40:20.800 INFO:tasks.rados.rados.0.smithi045.stdout:3551: oid 21 version 2482 is already newer than 2480 2024-04-26T00:40:20.800 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 2482 (ObjNum 1260 snap 393 seq_num 1260) dirty exists 2024-04-26T00:40:20.801 INFO:tasks.rados.rados.0.smithi045.stdout:3551: left oid 21 (ObjNum 1260 snap 393 seq_num 1260) 2024-04-26T00:40:20.824 INFO:tasks.rados.rados.0.smithi045.stdout:3551: done (3 left) 2024-04-26T00:40:20.824 INFO:tasks.rados.rados.0.smithi045.stdout:3552: done (2 left) 2024-04-26T00:40:20.824 INFO:tasks.rados.rados.0.smithi045.stdout:3555: done (1 left) 2024-04-26T00:40:20.824 INFO:tasks.rados.rados.0.smithi045.stdout:3556: read oid 2 snap 378 2024-04-26T00:40:20.825 INFO:tasks.rados.rados.0.smithi045.stdout:3556: expect (ObjNum 1215 snap 378 seq_num 1215) 2024-04-26T00:40:20.825 INFO:tasks.rados.rados.0.smithi045.stdout:3557: delete oid 1 current snap is 393 2024-04-26T00:40:20.829 INFO:tasks.rados.rados.0.smithi045.stdout:3554: expect (ObjNum 1232 snap 383 seq_num 1232) 2024-04-26T00:40:20.998 INFO:tasks.rados.rados.0.smithi045.stdout:3556: expect (ObjNum 1215 snap 378 seq_num 1215) 2024-04-26T00:40:21.114 INFO:tasks.rados.rados.0.smithi045.stdout:3554: done (2 left) 2024-04-26T00:40:21.114 INFO:tasks.rados.rados.0.smithi045.stdout:3556: done (1 left) 2024-04-26T00:40:21.114 INFO:tasks.rados.rados.0.smithi045.stdout:3557: done (0 left) 2024-04-26T00:40:21.114 INFO:tasks.rados.rados.0.smithi045.stdout:3558: snap_create 2024-04-26T00:40:21.815 INFO:tasks.rados.rados.0.smithi045.stdout:3558: done (0 left) 2024-04-26T00:40:21.815 INFO:tasks.rados.rados.0.smithi045.stdout:3559: write (excl) oid 15 current snap is 394 2024-04-26T00:40:21.815 INFO:tasks.rados.rados.0.smithi045.stdout:3559: seq_num 1263 ranges {427241=754085,1624287=408441,2600534=549964} 2024-04-26T00:40:21.833 INFO:tasks.rados.rados.0.smithi045.stdout:3559: writing smithi04526579-15 from 427241 to 1181326 tid 1 2024-04-26T00:40:21.847 INFO:tasks.rados.rados.0.smithi045.stdout:3559: writing smithi04526579-15 from 1624287 to 2032728 tid 2 2024-04-26T00:40:21.866 INFO:tasks.rados.rados.0.smithi045.stdout:3559: writing smithi04526579-15 from 2600534 to 3150498 tid 3 2024-04-26T00:40:21.866 INFO:tasks.rados.rados.0.smithi045.stdout:3560: rollback oid 18 current snap is 394 2024-04-26T00:40:21.866 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 18 to 373 2024-04-26T00:40:21.866 INFO:tasks.rados.rados.0.smithi045.stdout:3559: finishing write tid 1 to smithi04526579-15 2024-04-26T00:40:21.866 INFO:tasks.rados.rados.0.smithi045.stdout:3559: finishing write tid 2 to smithi04526579-15 2024-04-26T00:40:21.866 INFO:tasks.rados.rados.0.smithi045.stdout:3561: read oid 10 snap -1 2024-04-26T00:40:21.866 INFO:tasks.rados.rados.0.smithi045.stdout:3561: expect (ObjNum 1112 snap 331 seq_num 1112) 2024-04-26T00:40:21.866 INFO:tasks.rados.rados.0.smithi045.stdout:3562: write oid 50 current snap is 394 2024-04-26T00:40:21.867 INFO:tasks.rados.rados.0.smithi045.stdout:3562: seq_num 1264 ranges {444048=429334,1282769=552099,2627957=616091} 2024-04-26T00:40:21.877 INFO:tasks.rados.rados.0.smithi045.stdout:3562: writing smithi04526579-50 from 444048 to 873382 tid 1 2024-04-26T00:40:21.888 INFO:tasks.rados.rados.0.smithi045.stdout:3562: writing smithi04526579-50 from 1282769 to 1834868 tid 2 2024-04-26T00:40:21.901 INFO:tasks.rados.rados.0.smithi045.stdout:3562: writing smithi04526579-50 from 2627957 to 3244048 tid 3 2024-04-26T00:40:21.901 INFO:tasks.rados.rados.0.smithi045.stdout:3563: read oid 20 snap -1 2024-04-26T00:40:21.901 INFO:tasks.rados.rados.0.smithi045.stdout:3563: expect deleted 2024-04-26T00:40:21.901 INFO:tasks.rados.rados.0.smithi045.stdout:3564: copy_from oid 19 from oid 48 current snap is 394 2024-04-26T00:40:21.902 INFO:tasks.rados.rados.0.smithi045.stdout:3559: finishing write tid 3 to smithi04526579-15 2024-04-26T00:40:21.902 INFO:tasks.rados.rados.0.smithi045.stdout:3561: expect (ObjNum 1112 snap 331 seq_num 1112) 2024-04-26T00:40:22.082 INFO:tasks.rados.rados.0.smithi045.stdout:3559: finishing write tid 4 to smithi04526579-15 2024-04-26T00:40:22.082 INFO:tasks.rados.rados.0.smithi045.stdout:3559: finishing write tid 5 to smithi04526579-15 2024-04-26T00:40:22.082 INFO:tasks.rados.rados.0.smithi045.stdout:3559: oid 15 updating version 0 to 2491 2024-04-26T00:40:22.082 INFO:tasks.rados.rados.0.smithi045.stdout:3559: oid 15 updating version 2491 to 2494 2024-04-26T00:40:22.082 INFO:tasks.rados.rados.0.smithi045.stdout:3559: oid 15 version 2494 is already newer than 2493 2024-04-26T00:40:22.082 INFO:tasks.rados.rados.0.smithi045.stdout:3559: oid 15 version 2494 is already newer than 2492 2024-04-26T00:40:22.083 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 2494 (ObjNum 1263 snap 394 seq_num 1263) dirty exists 2024-04-26T00:40:22.083 INFO:tasks.rados.rados.0.smithi045.stdout:3559: left oid 15 (ObjNum 1263 snap 394 seq_num 1263) 2024-04-26T00:40:22.083 INFO:tasks.rados.rados.0.smithi045.stdout:3560: finishing rollback tid 0 to smithi04526579-18 2024-04-26T00:40:22.083 INFO:tasks.rados.rados.0.smithi045.stdout:3560: finishing rollback tid 1 to smithi04526579-18 2024-04-26T00:40:22.083 INFO:tasks.rados.rados.0.smithi045.stdout:3560: finishing rollback tid 2 to smithi04526579-18 2024-04-26T00:40:22.083 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 2689 (ObjNum 1096 snap 325 seq_num 1096) dirty exists 2024-04-26T00:40:22.083 INFO:tasks.rados.rados.0.smithi045.stdout:3562: finishing write tid 1 to smithi04526579-50 2024-04-26T00:40:22.083 INFO:tasks.rados.rados.0.smithi045.stdout:3562: finishing write tid 2 to smithi04526579-50 2024-04-26T00:40:22.083 INFO:tasks.rados.rados.0.smithi045.stdout:3562: finishing write tid 3 to smithi04526579-50 2024-04-26T00:40:22.083 INFO:tasks.rados.rados.0.smithi045.stdout:3562: finishing write tid 4 to smithi04526579-50 2024-04-26T00:40:22.083 INFO:tasks.rados.rados.0.smithi045.stdout:3562: finishing write tid 5 to smithi04526579-50 2024-04-26T00:40:22.083 INFO:tasks.rados.rados.0.smithi045.stdout:3562: oid 50 updating version 0 to 2429 2024-04-26T00:40:22.083 INFO:tasks.rados.rados.0.smithi045.stdout:3562: oid 50 updating version 2429 to 2432 2024-04-26T00:40:22.083 INFO:tasks.rados.rados.0.smithi045.stdout:3562: oid 50 version 2432 is already newer than 2430 2024-04-26T00:40:22.084 INFO:tasks.rados.rados.0.smithi045.stdout:3562: oid 50 version 2432 is already newer than 2431 2024-04-26T00:40:22.084 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 2432 (ObjNum 1264 snap 394 seq_num 1264) dirty exists 2024-04-26T00:40:22.084 INFO:tasks.rados.rados.0.smithi045.stdout:3562: left oid 50 (ObjNum 1264 snap 394 seq_num 1264) 2024-04-26T00:40:22.084 INFO:tasks.rados.rados.0.smithi045.stdout:3564: finishing copy_from racing read to smithi04526579-19 2024-04-26T00:40:22.084 INFO:tasks.rados.rados.0.smithi045.stdout:3564: finishing copy_from to smithi04526579-19 2024-04-26T00:40:22.084 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 2476 (ObjNum 1239 snap 384 seq_num 1239) dirty exists 2024-04-26T00:40:22.084 INFO:tasks.rados.rados.0.smithi045.stdout:3559: done (5 left) 2024-04-26T00:40:22.084 INFO:tasks.rados.rados.0.smithi045.stdout:3560: done (4 left) 2024-04-26T00:40:22.084 INFO:tasks.rados.rados.0.smithi045.stdout:3561: done (3 left) 2024-04-26T00:40:22.084 INFO:tasks.rados.rados.0.smithi045.stdout:3562: done (2 left) 2024-04-26T00:40:22.084 INFO:tasks.rados.rados.0.smithi045.stdout:3563: done (1 left) 2024-04-26T00:40:22.084 INFO:tasks.rados.rados.0.smithi045.stdout:3564: done (0 left) 2024-04-26T00:40:22.084 INFO:tasks.rados.rados.0.smithi045.stdout:3565: delete oid 32 current snap is 394 2024-04-26T00:40:22.088 INFO:tasks.rados.rados.0.smithi045.stdout:3565: done (0 left) 2024-04-26T00:40:22.088 INFO:tasks.rados.rados.0.smithi045.stdout:3566: read oid 11 snap -1 2024-04-26T00:40:22.088 INFO:tasks.rados.rados.0.smithi045.stdout:3566: expect (ObjNum 1139 snap 345 seq_num 1139) 2024-04-26T00:40:22.088 INFO:tasks.rados.rados.0.smithi045.stdout:3567: delete oid 46 current snap is 394 2024-04-26T00:40:22.095 INFO:tasks.rados.rados.0.smithi045.stdout:3566: expect (ObjNum 1139 snap 345 seq_num 1139) 2024-04-26T00:40:22.297 INFO:tasks.rados.rados.0.smithi045.stdout:3566: done (1 left) 2024-04-26T00:40:22.297 INFO:tasks.rados.rados.0.smithi045.stdout:3567: done (0 left) 2024-04-26T00:40:22.297 INFO:tasks.rados.rados.0.smithi045.stdout:3568: read oid 15 snap -1 2024-04-26T00:40:22.297 INFO:tasks.rados.rados.0.smithi045.stdout:3568: expect (ObjNum 1263 snap 394 seq_num 1263) 2024-04-26T00:40:22.297 INFO:tasks.rados.rados.0.smithi045.stdout:3569: snap_remove snap 361 2024-04-26T00:40:22.667 INFO:tasks.rados.rados.0.smithi045.stdout:3569: done (1 left) 2024-04-26T00:40:22.667 INFO:tasks.rados.rados.0.smithi045.stdout:3570: read oid 39 snap -1 2024-04-26T00:40:22.668 INFO:tasks.rados.rados.0.smithi045.stdout:3570: expect deleted 2024-04-26T00:40:22.668 INFO:tasks.rados.rados.0.smithi045.stdout:3571: delete oid 29 current snap is 394 2024-04-26T00:40:22.668 INFO:tasks.rados.rados.0.smithi045.stdout:3568: expect (ObjNum 1263 snap 394 seq_num 1263) 2024-04-26T00:40:22.791 INFO:tasks.rados.rados.0.smithi045.stdout:3568: done (2 left) 2024-04-26T00:40:22.791 INFO:tasks.rados.rados.0.smithi045.stdout:3570: done (1 left) 2024-04-26T00:40:22.791 INFO:tasks.rados.rados.0.smithi045.stdout:3571: done (0 left) 2024-04-26T00:40:22.791 INFO:tasks.rados.rados.0.smithi045.stdout:3572: delete oid 20 current snap is 394 2024-04-26T00:40:22.796 INFO:tasks.rados.rados.0.smithi045.stdout:3572: done (0 left) 2024-04-26T00:40:22.796 INFO:tasks.rados.rados.0.smithi045.stdout:3573: write oid 1 current snap is 394 2024-04-26T00:40:22.796 INFO:tasks.rados.rados.0.smithi045.stdout:3573: seq_num 1269 ranges {528083=405579,1342473=419448,2248719=733949} 2024-04-26T00:40:22.801 INFO:tasks.rados.rados.0.smithi045.stdout:3573: writing smithi04526579-1 from 528083 to 933662 tid 1 2024-04-26T00:40:22.808 INFO:tasks.rados.rados.0.smithi045.stdout:3573: writing smithi04526579-1 from 1342473 to 1761921 tid 2 2024-04-26T00:40:22.819 INFO:tasks.rados.rados.0.smithi045.stdout:3573: writing smithi04526579-1 from 2248719 to 2982668 tid 3 2024-04-26T00:40:22.819 INFO:tasks.rados.rados.0.smithi045.stdout:3574: write oid 15 current snap is 394 2024-04-26T00:40:22.819 INFO:tasks.rados.rados.0.smithi045.stdout:3574: seq_num 1270 ranges {544890=597181,1917308=563106,3076142=268748} 2024-04-26T00:40:22.829 INFO:tasks.rados.rados.0.smithi045.stdout:3574: writing smithi04526579-15 from 544890 to 1142071 tid 1 2024-04-26T00:40:22.840 INFO:tasks.rados.rados.0.smithi045.stdout:3574: writing smithi04526579-15 from 1917308 to 2480414 tid 2 2024-04-26T00:40:22.849 INFO:tasks.rados.rados.0.smithi045.stdout:3574: writing smithi04526579-15 from 3076142 to 3344890 tid 3 2024-04-26T00:40:22.849 INFO:tasks.rados.rados.0.smithi045.stdout:3575: snap_create 2024-04-26T00:40:22.849 INFO:tasks.rados.rados.0.smithi045.stdout:3573: finishing write tid 1 to smithi04526579-1 2024-04-26T00:40:22.849 INFO:tasks.rados.rados.0.smithi045.stdout:3573: finishing write tid 2 to smithi04526579-1 2024-04-26T00:40:22.849 INFO:tasks.rados.rados.0.smithi045.stdout:3573: finishing write tid 3 to smithi04526579-1 2024-04-26T00:40:22.849 INFO:tasks.rados.rados.0.smithi045.stdout:3573: finishing write tid 4 to smithi04526579-1 2024-04-26T00:40:22.849 INFO:tasks.rados.rados.0.smithi045.stdout:3573: finishing write tid 5 to smithi04526579-1 2024-04-26T00:40:22.849 INFO:tasks.rados.rados.0.smithi045.stdout:3573: oid 1 updating version 0 to 2498 2024-04-26T00:40:22.849 INFO:tasks.rados.rados.0.smithi045.stdout:3573: oid 1 version 2498 is already newer than 2495 2024-04-26T00:40:22.849 INFO:tasks.rados.rados.0.smithi045.stdout:3573: oid 1 version 2498 is already newer than 2496 2024-04-26T00:40:22.849 INFO:tasks.rados.rados.0.smithi045.stdout:3573: oid 1 version 2498 is already newer than 2497 2024-04-26T00:40:22.849 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 2498 (ObjNum 1269 snap 394 seq_num 1269) dirty exists 2024-04-26T00:40:22.849 INFO:tasks.rados.rados.0.smithi045.stdout:3573: left oid 1 (ObjNum 1269 snap 394 seq_num 1269) 2024-04-26T00:40:22.850 INFO:tasks.rados.rados.0.smithi045.stdout:3574: finishing write tid 1 to smithi04526579-15 2024-04-26T00:40:22.850 INFO:tasks.rados.rados.0.smithi045.stdout:3574: finishing write tid 2 to smithi04526579-15 2024-04-26T00:40:22.855 INFO:tasks.rados.rados.0.smithi045.stdout:3574: finishing write tid 3 to smithi04526579-15 2024-04-26T00:40:22.855 INFO:tasks.rados.rados.0.smithi045.stdout:3574: finishing write tid 4 to smithi04526579-15 2024-04-26T00:40:22.856 INFO:tasks.rados.rados.0.smithi045.stdout:3574: finishing write tid 5 to smithi04526579-15 2024-04-26T00:40:22.856 INFO:tasks.rados.rados.0.smithi045.stdout:3574: oid 15 updating version 0 to 2499 2024-04-26T00:40:22.856 INFO:tasks.rados.rados.0.smithi045.stdout:3574: oid 15 updating version 2499 to 2500 2024-04-26T00:40:22.856 INFO:tasks.rados.rados.0.smithi045.stdout:3574: oid 15 updating version 2500 to 2502 2024-04-26T00:40:22.856 INFO:tasks.rados.rados.0.smithi045.stdout:3574: oid 15 version 2502 is already newer than 2501 2024-04-26T00:40:22.856 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 2502 (ObjNum 1270 snap 394 seq_num 1270) dirty exists 2024-04-26T00:40:22.856 INFO:tasks.rados.rados.0.smithi045.stdout:3574: left oid 15 (ObjNum 1270 snap 394 seq_num 1270) 2024-04-26T00:40:23.823 INFO:tasks.rados.rados.0.smithi045.stdout:3573: done (2 left) 2024-04-26T00:40:23.823 INFO:tasks.rados.rados.0.smithi045.stdout:3574: done (1 left) 2024-04-26T00:40:23.823 INFO:tasks.rados.rados.0.smithi045.stdout:3575: done (0 left) 2024-04-26T00:40:23.823 INFO:tasks.rados.rados.0.smithi045.stdout:3576: write oid 30 current snap is 395 2024-04-26T00:40:23.823 INFO:tasks.rados.rados.0.smithi045.stdout:3576: seq_num 1271 ranges {561697=672430,1975790=706764,3103565=258132} 2024-04-26T00:40:23.835 INFO:tasks.rados.rados.0.smithi045.stdout:3576: writing smithi04526579-30 from 561697 to 1234127 tid 1 2024-04-26T00:40:23.847 INFO:tasks.rados.rados.0.smithi045.stdout:3576: writing smithi04526579-30 from 1975790 to 2682554 tid 2 2024-04-26T00:40:23.853 INFO:tasks.rados.rados.0.smithi045.stdout:3576: writing smithi04526579-30 from 3103565 to 3361697 tid 3 2024-04-26T00:40:23.853 INFO:tasks.rados.rados.0.smithi045.stdout:3577: snap_create 2024-04-26T00:40:23.854 INFO:tasks.rados.rados.0.smithi045.stdout:3576: finishing write tid 1 to smithi04526579-30 2024-04-26T00:40:23.859 INFO:tasks.rados.rados.0.smithi045.stdout:3576: finishing write tid 2 to smithi04526579-30 2024-04-26T00:40:23.860 INFO:tasks.rados.rados.0.smithi045.stdout:3576: finishing write tid 3 to smithi04526579-30 2024-04-26T00:40:23.860 INFO:tasks.rados.rados.0.smithi045.stdout:3576: finishing write tid 4 to smithi04526579-30 2024-04-26T00:40:23.861 INFO:tasks.rados.rados.0.smithi045.stdout:3576: finishing write tid 5 to smithi04526579-30 2024-04-26T00:40:23.861 INFO:tasks.rados.rados.0.smithi045.stdout:3576: oid 30 updating version 0 to 2021 2024-04-26T00:40:23.861 INFO:tasks.rados.rados.0.smithi045.stdout:3576: oid 30 version 2021 is already newer than 2019 2024-04-26T00:40:23.861 INFO:tasks.rados.rados.0.smithi045.stdout:3576: oid 30 version 2021 is already newer than 2020 2024-04-26T00:40:23.861 INFO:tasks.rados.rados.0.smithi045.stdout:3576: oid 30 version 2021 is already newer than 2018 2024-04-26T00:40:23.861 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 2021 (ObjNum 1271 snap 395 seq_num 1271) dirty exists 2024-04-26T00:40:23.861 INFO:tasks.rados.rados.0.smithi045.stdout:3576: left oid 30 (ObjNum 1271 snap 395 seq_num 1271) 2024-04-26T00:40:24.826 INFO:tasks.rados.rados.0.smithi045.stdout:3576: done (1 left) 2024-04-26T00:40:24.826 INFO:tasks.rados.rados.0.smithi045.stdout:3577: done (0 left) 2024-04-26T00:40:24.826 INFO:tasks.rados.rados.0.smithi045.stdout:3578: copy_from oid 17 from oid 25 current snap is 396 2024-04-26T00:40:24.826 INFO:tasks.rados.rados.0.smithi045.stdout:3579: snap_create 2024-04-26T00:40:24.850 INFO:tasks.rados.rados.0.smithi045.stdout:3578: finishing copy_from racing read to smithi04526579-17 2024-04-26T00:40:24.856 INFO:tasks.rados.rados.0.smithi045.stdout:3578: finishing copy_from to smithi04526579-17 2024-04-26T00:40:24.856 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 2627 (ObjNum 1238 snap 384 seq_num 1238) dirty exists 2024-04-26T00:40:26.587 INFO:tasks.rados.rados.0.smithi045.stdout:3578: done (1 left) 2024-04-26T00:40:26.587 INFO:tasks.rados.rados.0.smithi045.stdout:3579: done (0 left) 2024-04-26T00:40:26.587 INFO:tasks.rados.rados.0.smithi045.stdout:3580: read oid 35 snap -1 2024-04-26T00:40:26.587 INFO:tasks.rados.rados.0.smithi045.stdout:3580: expect deleted 2024-04-26T00:40:26.587 INFO:tasks.rados.rados.0.smithi045.stdout:3581: snap_create 2024-04-26T00:40:26.850 INFO:tasks.rados.rados.0.smithi045.stdout:3580: done (1 left) 2024-04-26T00:40:26.850 INFO:tasks.rados.rados.0.smithi045.stdout:3581: done (0 left) 2024-04-26T00:40:26.850 INFO:tasks.rados.rados.0.smithi045.stdout:3582: snap_remove snap 391 2024-04-26T00:40:27.675 INFO:tasks.rados.rados.0.smithi045.stdout:3582: done (0 left) 2024-04-26T00:40:27.675 INFO:tasks.rados.rados.0.smithi045.stdout:3583: write oid 6 current snap is 398 2024-04-26T00:40:27.675 INFO:tasks.rados.rados.0.smithi045.stdout:3583: seq_num 1272 ranges {578504=747679,2034272=566775,3130988=247516} 2024-04-26T00:40:27.698 INFO:tasks.rados.rados.0.smithi045.stdout:3583: writing smithi04526579-6 from 578504 to 1326183 tid 1 2024-04-26T00:40:27.719 INFO:tasks.rados.rados.0.smithi045.stdout:3583: writing smithi04526579-6 from 2034272 to 2601047 tid 2 2024-04-26T00:40:27.730 INFO:tasks.rados.rados.0.smithi045.stdout:3583: writing smithi04526579-6 from 3130988 to 3378504 tid 3 2024-04-26T00:40:27.730 INFO:tasks.rados.rados.0.smithi045.stdout:3584: snap_remove snap 387 2024-04-26T00:40:28.687 INFO:tasks.rados.rados.0.smithi045.stdout:3584: done (1 left) 2024-04-26T00:40:28.687 INFO:tasks.rados.rados.0.smithi045.stdout:3585: rollback oid 4 current snap is 398 2024-04-26T00:40:28.687 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 4 to 332 2024-04-26T00:40:28.687 INFO:tasks.rados.rados.0.smithi045.stdout:3583: finishing write tid 1 to smithi04526579-6 2024-04-26T00:40:28.688 INFO:tasks.rados.rados.0.smithi045.stdout:3583: finishing write tid 2 to smithi04526579-6 2024-04-26T00:40:28.688 INFO:tasks.rados.rados.0.smithi045.stdout:3583: finishing write tid 3 to smithi04526579-6 2024-04-26T00:40:28.688 INFO:tasks.rados.rados.0.smithi045.stdout:3583: finishing write tid 4 to smithi04526579-6 2024-04-26T00:40:28.688 INFO:tasks.rados.rados.0.smithi045.stdout:3583: finishing write tid 5 to smithi04526579-6 2024-04-26T00:40:28.688 INFO:tasks.rados.rados.0.smithi045.stdout:3583: oid 6 updating version 0 to 2023 2024-04-26T00:40:28.688 INFO:tasks.rados.rados.0.smithi045.stdout:3583: oid 6 updating version 2023 to 2024 2024-04-26T00:40:28.688 INFO:tasks.rados.rados.0.smithi045.stdout:3583: oid 6 updating version 2024 to 2025 2024-04-26T00:40:28.688 INFO:tasks.rados.rados.0.smithi045.stdout:3583: oid 6 version 2025 is already newer than 2022 2024-04-26T00:40:28.688 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 2025 (ObjNum 1272 snap 398 seq_num 1272) dirty exists 2024-04-26T00:40:28.688 INFO:tasks.rados.rados.0.smithi045.stdout:3583: left oid 6 (ObjNum 1272 snap 398 seq_num 1272) 2024-04-26T00:40:28.688 INFO:tasks.rados.rados.0.smithi045.stdout:3583: done (1 left) 2024-04-26T00:40:28.688 INFO:tasks.rados.rados.0.smithi045.stdout:3586: delete oid 13 current snap is 398 2024-04-26T00:40:28.700 INFO:tasks.rados.rados.0.smithi045.stdout:3586: done (1 left) 2024-04-26T00:40:28.700 INFO:tasks.rados.rados.0.smithi045.stdout:3587: read oid 31 snap 351 2024-04-26T00:40:28.700 INFO:tasks.rados.rados.0.smithi045.stdout:3587: expect (ObjNum 1143 snap 346 seq_num 1143) 2024-04-26T00:40:28.700 INFO:tasks.rados.rados.0.smithi045.stdout:3588: write (excl) oid 28 current snap is 398 2024-04-26T00:40:28.700 INFO:tasks.rados.rados.0.smithi045.stdout:3588: seq_num 1274 ranges {612118=498177,1634883=570444,2669481=706662} 2024-04-26T00:40:28.719 INFO:tasks.rados.rados.0.smithi045.stdout:3588: writing smithi04526579-28 from 612118 to 1110295 tid 1 2024-04-26T00:40:28.737 INFO:tasks.rados.rados.0.smithi045.stdout:3588: writing smithi04526579-28 from 1634883 to 2205327 tid 2 2024-04-26T00:40:28.755 INFO:tasks.rados.rados.0.smithi045.stdout:3588: writing smithi04526579-28 from 2669481 to 3376143 tid 3 2024-04-26T00:40:28.755 INFO:tasks.rados.rados.0.smithi045.stdout:3589: rollback oid 3 current snap is 398 2024-04-26T00:40:28.756 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 3 to 332 2024-04-26T00:40:28.756 INFO:tasks.rados.rados.0.smithi045.stdout:3585: finishing rollback tid 0 to smithi04526579-4 2024-04-26T00:40:28.756 INFO:tasks.rados.rados.0.smithi045.stdout:3585: finishing rollback tid 1 to smithi04526579-4 2024-04-26T00:40:28.756 INFO:tasks.rados.rados.0.smithi045.stdout:3585: finishing rollback tid 2 to smithi04526579-4 2024-04-26T00:40:28.756 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 2515 (ObjNum 1080 snap 323 seq_num 1080) dirty exists 2024-04-26T00:40:28.756 INFO:tasks.rados.rados.0.smithi045.stdout:3588: finishing write tid 1 to smithi04526579-28 2024-04-26T00:40:28.756 INFO:tasks.rados.rados.0.smithi045.stdout:3587: expect (ObjNum 1143 snap 346 seq_num 1143) 2024-04-26T00:40:29.076 INFO:tasks.rados.rados.0.smithi045.stdout:3588: finishing write tid 2 to smithi04526579-28 2024-04-26T00:40:29.076 INFO:tasks.rados.rados.0.smithi045.stdout:3588: finishing write tid 3 to smithi04526579-28 2024-04-26T00:40:29.076 INFO:tasks.rados.rados.0.smithi045.stdout:3588: finishing write tid 4 to smithi04526579-28 2024-04-26T00:40:29.076 INFO:tasks.rados.rados.0.smithi045.stdout:3588: finishing write tid 5 to smithi04526579-28 2024-04-26T00:40:29.076 INFO:tasks.rados.rados.0.smithi045.stdout:3588: oid 28 updating version 0 to 2503 2024-04-26T00:40:29.076 INFO:tasks.rados.rados.0.smithi045.stdout:3588: oid 28 updating version 2503 to 2505 2024-04-26T00:40:29.076 INFO:tasks.rados.rados.0.smithi045.stdout:3588: oid 28 version 2505 is already newer than 2504 2024-04-26T00:40:29.076 INFO:tasks.rados.rados.0.smithi045.stdout:3588: oid 28 updating version 2505 to 2506 2024-04-26T00:40:29.076 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 2506 (ObjNum 1274 snap 398 seq_num 1274) dirty exists 2024-04-26T00:40:29.077 INFO:tasks.rados.rados.0.smithi045.stdout:3588: left oid 28 (ObjNum 1274 snap 398 seq_num 1274) 2024-04-26T00:40:29.077 INFO:tasks.rados.rados.0.smithi045.stdout:3589: finishing rollback tid 0 to smithi04526579-3 2024-04-26T00:40:29.077 INFO:tasks.rados.rados.0.smithi045.stdout:3589: finishing rollback tid 1 to smithi04526579-3 2024-04-26T00:40:29.077 INFO:tasks.rados.rados.0.smithi045.stdout:3589: finishing rollback tid 2 to smithi04526579-3 2024-04-26T00:40:29.077 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 2488 (ObjNum 1107 snap 330 seq_num 1107) dirty exists 2024-04-26T00:40:29.077 INFO:tasks.rados.rados.0.smithi045.stdout:3585: done (3 left) 2024-04-26T00:40:29.077 INFO:tasks.rados.rados.0.smithi045.stdout:3587: done (2 left) 2024-04-26T00:40:29.077 INFO:tasks.rados.rados.0.smithi045.stdout:3588: done (1 left) 2024-04-26T00:40:29.077 INFO:tasks.rados.rados.0.smithi045.stdout:3589: done (0 left) 2024-04-26T00:40:29.077 INFO:tasks.rados.rados.0.smithi045.stdout:3590: read oid 27 snap 378 2024-04-26T00:40:29.077 INFO:tasks.rados.rados.0.smithi045.stdout:3590: expect deleted 2024-04-26T00:40:29.077 INFO:tasks.rados.rados.0.smithi045.stdout:3591: delete oid 39 current snap is 398 2024-04-26T00:40:29.081 INFO:tasks.rados.rados.0.smithi045.stdout:3590: done (1 left) 2024-04-26T00:40:29.082 INFO:tasks.rados.rados.0.smithi045.stdout:3591: done (0 left) 2024-04-26T00:40:29.082 INFO:tasks.rados.rados.0.smithi045.stdout:3592: rollback oid 10 current snap is 398 2024-04-26T00:40:29.082 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 10 to 394 2024-04-26T00:40:29.082 INFO:tasks.rados.rados.0.smithi045.stdout:3593: copy_from oid 6 from oid 22 current snap is 398 2024-04-26T00:40:29.082 INFO:tasks.rados.rados.0.smithi045.stdout:3594: delete oid 30 current snap is 398 2024-04-26T00:40:29.090 INFO:tasks.rados.rados.0.smithi045.stdout:3592: finishing rollback tid 0 to smithi04526579-10 2024-04-26T00:40:29.091 INFO:tasks.rados.rados.0.smithi045.stdout:3592: finishing rollback tid 1 to smithi04526579-10 2024-04-26T00:40:29.091 INFO:tasks.rados.rados.0.smithi045.stdout:3592: finishing rollback tid 2 to smithi04526579-10 2024-04-26T00:40:29.091 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 2630 (ObjNum 1112 snap 331 seq_num 1112) dirty exists 2024-04-26T00:40:29.100 INFO:tasks.rados.rados.0.smithi045.stdout:3593: finishing copy_from racing read to smithi04526579-6 2024-04-26T00:40:29.101 INFO:tasks.rados.rados.0.smithi045.stdout:3592: done (2 left) 2024-04-26T00:40:29.101 INFO:tasks.rados.rados.0.smithi045.stdout:3594: done (1 left) 2024-04-26T00:40:29.101 INFO:tasks.rados.rados.0.smithi045.stdout:3595: copy_from oid 33 from oid 30 current snap is 398 2024-04-26T00:40:29.101 INFO:tasks.rados.rados.0.smithi045.stdout:3596: write oid 34 current snap is 398 2024-04-26T00:40:29.101 INFO:tasks.rados.rados.0.smithi045.stdout:3596: seq_num 1277 ranges {662539=723924,2093976=717771,3435397=27142} 2024-04-26T00:40:29.124 INFO:tasks.rados.rados.0.smithi045.stdout:3596: writing smithi04526579-34 from 662539 to 1386463 tid 1 2024-04-26T00:40:29.147 INFO:tasks.rados.rados.0.smithi045.stdout:3596: writing smithi04526579-34 from 2093976 to 2811747 tid 2 2024-04-26T00:40:29.158 INFO:tasks.rados.rados.0.smithi045.stdout:3596: writing smithi04526579-34 from 3435397 to 3462539 tid 3 2024-04-26T00:40:29.158 INFO:tasks.rados.rados.0.smithi045.stdout:3597: read oid 49 snap -1 2024-04-26T00:40:29.158 INFO:tasks.rados.rados.0.smithi045.stdout:3597: expect (ObjNum 1162 snap 359 seq_num 1162) 2024-04-26T00:40:29.158 INFO:tasks.rados.rados.0.smithi045.stdout:3598: read oid 41 snap -1 2024-04-26T00:40:29.158 INFO:tasks.rados.rados.0.smithi045.stdout:3598: expect (ObjNum 1189 snap 369 seq_num 1189) 2024-04-26T00:40:29.158 INFO:tasks.rados.rados.0.smithi045.stdout:3599: read oid 47 snap -1 2024-04-26T00:40:29.158 INFO:tasks.rados.rados.0.smithi045.stdout:3599: expect (ObjNum 1127 snap 340 seq_num 1127) 2024-04-26T00:40:29.158 INFO:tasks.rados.rados.0.smithi045.stdout:3600: rollback oid 46 current snap is 398 2024-04-26T00:40:29.158 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 46 to 396 2024-04-26T00:40:29.158 INFO:tasks.rados.rados.0.smithi045.stdout:3595: finishing copy_from to smithi04526579-33 2024-04-26T00:40:29.158 INFO:tasks.rados.rados.0.smithi045.stdout:3595: got expected ENOENT (src dne) 2024-04-26T00:40:29.158 INFO:tasks.rados.rados.0.smithi045.stdout:3595: finishing copy_from racing read to smithi04526579-33 2024-04-26T00:40:29.159 INFO:tasks.rados.rados.0.smithi045.stdout:3593: finishing copy_from to smithi04526579-6 2024-04-26T00:40:29.159 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 2453 (ObjNum 1135 snap 343 seq_num 1135) dirty exists 2024-04-26T00:40:29.159 INFO:tasks.rados.rados.0.smithi045.stdout:3596: finishing write tid 1 to smithi04526579-34 2024-04-26T00:40:29.159 INFO:tasks.rados.rados.0.smithi045.stdout:3596: finishing write tid 2 to smithi04526579-34 2024-04-26T00:40:29.159 INFO:tasks.rados.rados.0.smithi045.stdout:3593: done (6 left) 2024-04-26T00:40:29.159 INFO:tasks.rados.rados.0.smithi045.stdout:3595: done (5 left) 2024-04-26T00:40:29.159 INFO:tasks.rados.rados.0.smithi045.stdout:3601: write oid 12 current snap is 398 2024-04-26T00:40:29.159 INFO:tasks.rados.rados.0.smithi045.stdout:3601: seq_num 1278 ranges {679346=515526,1868811=577782,3179173=300173} 2024-04-26T00:40:29.171 INFO:tasks.rados.rados.0.smithi045.stdout:3601: writing smithi04526579-12 from 679346 to 1194872 tid 1 2024-04-26T00:40:29.185 INFO:tasks.rados.rados.0.smithi045.stdout:3601: writing smithi04526579-12 from 1868811 to 2446593 tid 2 2024-04-26T00:40:29.194 INFO:tasks.rados.rados.0.smithi045.stdout:3601: writing smithi04526579-12 from 3179173 to 3479346 tid 3 2024-04-26T00:40:29.194 INFO:tasks.rados.rados.0.smithi045.stdout:3602: snap_remove snap 351 2024-04-26T00:40:29.870 INFO:tasks.rados.rados.0.smithi045.stdout:3602: done (6 left) 2024-04-26T00:40:29.870 INFO:tasks.rados.rados.0.smithi045.stdout:3603: read oid 27 snap 397 2024-04-26T00:40:29.870 INFO:tasks.rados.rados.0.smithi045.stdout:3603: expect deleted 2024-04-26T00:40:29.870 INFO:tasks.rados.rados.0.smithi045.stdout:3604: write oid 28 current snap is 398 2024-04-26T00:40:29.870 INFO:tasks.rados.rados.0.smithi045.stdout:3604: seq_num 1279 ranges {696153=590775,1927293=721440,3206596=289557} 2024-04-26T00:40:29.880 INFO:tasks.rados.rados.0.smithi045.stdout:3604: writing smithi04526579-28 from 696153 to 1286928 tid 1 2024-04-26T00:40:29.892 INFO:tasks.rados.rados.0.smithi045.stdout:3604: writing smithi04526579-28 from 1927293 to 2648733 tid 2 2024-04-26T00:40:29.899 INFO:tasks.rados.rados.0.smithi045.stdout:3604: writing smithi04526579-28 from 3206596 to 3496153 tid 3 2024-04-26T00:40:29.899 INFO:tasks.rados.rados.0.smithi045.stdout:3605: delete oid 9 current snap is 398 2024-04-26T00:40:29.899 INFO:tasks.rados.rados.0.smithi045.stdout:3600: finishing rollback tid 1 to smithi04526579-46 2024-04-26T00:40:29.899 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 2457 (ObjNum 145 snap 0 seq_num 2242339536) dirty dne 2024-04-26T00:40:29.899 INFO:tasks.rados.rados.0.smithi045.stdout:3596: finishing write tid 3 to smithi04526579-34 2024-04-26T00:40:29.900 INFO:tasks.rados.rados.0.smithi045.stdout:3596: finishing write tid 4 to smithi04526579-34 2024-04-26T00:40:29.900 INFO:tasks.rados.rados.0.smithi045.stdout:3596: finishing write tid 5 to smithi04526579-34 2024-04-26T00:40:29.900 INFO:tasks.rados.rados.0.smithi045.stdout:3596: oid 34 updating version 0 to 2489 2024-04-26T00:40:29.900 INFO:tasks.rados.rados.0.smithi045.stdout:3596: oid 34 updating version 2489 to 2490 2024-04-26T00:40:29.900 INFO:tasks.rados.rados.0.smithi045.stdout:3596: oid 34 updating version 2490 to 2492 2024-04-26T00:40:29.900 INFO:tasks.rados.rados.0.smithi045.stdout:3596: oid 34 version 2492 is already newer than 2491 2024-04-26T00:40:29.900 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 2492 (ObjNum 1277 snap 398 seq_num 1277) dirty exists 2024-04-26T00:40:29.900 INFO:tasks.rados.rados.0.smithi045.stdout:3596: left oid 34 (ObjNum 1277 snap 398 seq_num 1277) 2024-04-26T00:40:29.900 INFO:tasks.rados.rados.0.smithi045.stdout:3598: expect (ObjNum 1189 snap 369 seq_num 1189) 2024-04-26T00:40:30.219 INFO:tasks.rados.rados.0.smithi045.stdout:3597: expect (ObjNum 1162 snap 359 seq_num 1162) 2024-04-26T00:40:30.335 INFO:tasks.rados.rados.0.smithi045.stdout:3599: expect (ObjNum 1127 snap 340 seq_num 1127) 2024-04-26T00:40:30.526 INFO:tasks.rados.rados.0.smithi045.stdout:3601: finishing write tid 1 to smithi04526579-12 2024-04-26T00:40:30.526 INFO:tasks.rados.rados.0.smithi045.stdout:3601: finishing write tid 2 to smithi04526579-12 2024-04-26T00:40:30.526 INFO:tasks.rados.rados.0.smithi045.stdout:3601: finishing write tid 3 to smithi04526579-12 2024-04-26T00:40:30.526 INFO:tasks.rados.rados.0.smithi045.stdout:3601: finishing write tid 4 to smithi04526579-12 2024-04-26T00:40:30.526 INFO:tasks.rados.rados.0.smithi045.stdout:3601: finishing write tid 5 to smithi04526579-12 2024-04-26T00:40:30.527 INFO:tasks.rados.rados.0.smithi045.stdout:3601: oid 12 updating version 0 to 2346 2024-04-26T00:40:30.527 INFO:tasks.rados.rados.0.smithi045.stdout:3601: oid 12 version 2346 is already newer than 2345 2024-04-26T00:40:30.527 INFO:tasks.rados.rados.0.smithi045.stdout:3601: oid 12 updating version 2346 to 2347 2024-04-26T00:40:30.527 INFO:tasks.rados.rados.0.smithi045.stdout:3601: oid 12 version 2347 is already newer than 2344 2024-04-26T00:40:30.527 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 2347 (ObjNum 1278 snap 398 seq_num 1278) dirty exists 2024-04-26T00:40:30.527 INFO:tasks.rados.rados.0.smithi045.stdout:3601: left oid 12 (ObjNum 1278 snap 398 seq_num 1278) 2024-04-26T00:40:30.527 INFO:tasks.rados.rados.0.smithi045.stdout:3604: finishing write tid 1 to smithi04526579-28 2024-04-26T00:40:30.527 INFO:tasks.rados.rados.0.smithi045.stdout:3604: finishing write tid 2 to smithi04526579-28 2024-04-26T00:40:30.527 INFO:tasks.rados.rados.0.smithi045.stdout:3596: done (8 left) 2024-04-26T00:40:30.527 INFO:tasks.rados.rados.0.smithi045.stdout:3597: done (7 left) 2024-04-26T00:40:30.527 INFO:tasks.rados.rados.0.smithi045.stdout:3598: done (6 left) 2024-04-26T00:40:30.527 INFO:tasks.rados.rados.0.smithi045.stdout:3599: done (5 left) 2024-04-26T00:40:30.527 INFO:tasks.rados.rados.0.smithi045.stdout:3600: done (4 left) 2024-04-26T00:40:30.527 INFO:tasks.rados.rados.0.smithi045.stdout:3601: done (3 left) 2024-04-26T00:40:30.528 INFO:tasks.rados.rados.0.smithi045.stdout:3603: done (2 left) 2024-04-26T00:40:30.528 INFO:tasks.rados.rados.0.smithi045.stdout:3605: done (1 left) 2024-04-26T00:40:30.528 INFO:tasks.rados.rados.0.smithi045.stdout:3606: copy_from oid 10 from oid 29 current snap is 398 2024-04-26T00:40:30.528 INFO:tasks.rados.rados.0.smithi045.stdout:3604: finishing write tid 3 to smithi04526579-28 2024-04-26T00:40:30.528 INFO:tasks.rados.rados.0.smithi045.stdout:3604: finishing write tid 4 to smithi04526579-28 2024-04-26T00:40:30.528 INFO:tasks.rados.rados.0.smithi045.stdout:3604: finishing write tid 5 to smithi04526579-28 2024-04-26T00:40:30.528 INFO:tasks.rados.rados.0.smithi045.stdout:3604: oid 28 updating version 0 to 2507 2024-04-26T00:40:30.528 INFO:tasks.rados.rados.0.smithi045.stdout:3604: oid 28 updating version 2507 to 2508 2024-04-26T00:40:30.528 INFO:tasks.rados.rados.0.smithi045.stdout:3604: oid 28 updating version 2508 to 2509 2024-04-26T00:40:30.528 INFO:tasks.rados.rados.0.smithi045.stdout:3604: oid 28 updating version 2509 to 2510 2024-04-26T00:40:30.528 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 2510 (ObjNum 1279 snap 398 seq_num 1279) dirty exists 2024-04-26T00:40:30.528 INFO:tasks.rados.rados.0.smithi045.stdout:3604: left oid 28 (ObjNum 1279 snap 398 seq_num 1279) 2024-04-26T00:40:30.528 INFO:tasks.rados.rados.0.smithi045.stdout:3604: done (1 left) 2024-04-26T00:40:30.528 INFO:tasks.rados.rados.0.smithi045.stdout:3607: read oid 13 snap -1 2024-04-26T00:40:30.528 INFO:tasks.rados.rados.0.smithi045.stdout:3607: expect deleted 2024-04-26T00:40:30.528 INFO:tasks.rados.rados.0.smithi045.stdout:3608: write (excl) oid 16 current snap is 398 2024-04-26T00:40:30.528 INFO:tasks.rados.rados.0.smithi045.stdout:3608: seq_num 1281 ranges {729767=741273,1927904=725109,3145089=384678} 2024-04-26T00:40:30.547 INFO:tasks.rados.rados.0.smithi045.stdout:3608: writing smithi04526579-16 from 729767 to 1471040 tid 1 2024-04-26T00:40:30.565 INFO:tasks.rados.rados.0.smithi045.stdout:3608: writing smithi04526579-16 from 1927904 to 2653013 tid 2 2024-04-26T00:40:30.579 INFO:tasks.rados.rados.0.smithi045.stdout:3608: writing smithi04526579-16 from 3145089 to 3529767 tid 3 2024-04-26T00:40:30.579 INFO:tasks.rados.rados.0.smithi045.stdout:3609: read oid 31 snap -1 2024-04-26T00:40:30.579 INFO:tasks.rados.rados.0.smithi045.stdout:3609: expect deleted 2024-04-26T00:40:30.579 INFO:tasks.rados.rados.0.smithi045.stdout:3610: write (excl) oid 4 current snap is 398 2024-04-26T00:40:30.579 INFO:tasks.rados.rados.0.smithi045.stdout:3610: seq_num 1282 ranges {746574=416522,1586386=468767,2772512=629544} 2024-04-26T00:40:30.593 INFO:tasks.rados.rados.0.smithi045.stdout:3610: writing smithi04526579-4 from 746574 to 1163096 tid 1 2024-04-26T00:40:30.602 INFO:tasks.rados.rados.0.smithi045.stdout:3610: writing smithi04526579-4 from 1586386 to 2055153 tid 2 2024-04-26T00:40:30.614 INFO:tasks.rados.rados.0.smithi045.stdout:3610: writing smithi04526579-4 from 2772512 to 3402056 tid 3 2024-04-26T00:40:30.614 INFO:tasks.rados.rados.0.smithi045.stdout:3611: write oid 8 current snap is 398 2024-04-26T00:40:30.614 INFO:tasks.rados.rados.0.smithi045.stdout:3611: seq_num 1283 ranges {763381=491771,2044868=728778,3199935=363446} 2024-04-26T00:40:30.623 INFO:tasks.rados.rados.0.smithi045.stdout:3611: writing smithi04526579-8 from 763381 to 1255152 tid 1 2024-04-26T00:40:30.634 INFO:tasks.rados.rados.0.smithi045.stdout:3611: writing smithi04526579-8 from 2044868 to 2773646 tid 2 2024-04-26T00:40:30.639 INFO:tasks.rados.rados.0.smithi045.stdout:3611: writing smithi04526579-8 from 3199935 to 3563381 tid 3 2024-04-26T00:40:30.640 INFO:tasks.rados.rados.0.smithi045.stdout:3612: snap_remove snap 383 2024-04-26T00:40:30.883 INFO:tasks.rados.rados.0.smithi045.stdout:3612: done (6 left) 2024-04-26T00:40:30.883 INFO:tasks.rados.rados.0.smithi045.stdout:3613: write (excl) oid 49 current snap is 398 2024-04-26T00:40:30.883 INFO:tasks.rados.rados.0.smithi045.stdout:3613: seq_num 1284 ranges {780188=567020,2103350=472436,3227358=352830} 2024-04-26T00:40:30.892 INFO:tasks.rados.rados.0.smithi045.stdout:3613: writing smithi04526579-49 from 780188 to 1347208 tid 1 2024-04-26T00:40:30.900 INFO:tasks.rados.rados.0.smithi045.stdout:3613: writing smithi04526579-49 from 2103350 to 2575786 tid 2 2024-04-26T00:40:30.907 INFO:tasks.rados.rados.0.smithi045.stdout:3613: writing smithi04526579-49 from 3227358 to 3580188 tid 3 2024-04-26T00:40:30.907 INFO:tasks.rados.rados.0.smithi045.stdout:3614: snap_remove snap 381 2024-04-26T00:40:31.146 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-04-26T00:40:31.146 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-04-26T00:40:31.146 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-26T00:40:31.146 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-26T00:40:31.149 INFO:tasks.ceph.mon.c:Started 2024-04-26T00:40:31.149 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:40:31.150 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:40:31.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e22ed2640 10 monclient: get_monmap_and_config 2024-04-26T00:40:31.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e22ed2640 10 monclient: build_initial_monmap 2024-04-26T00:40:31.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e22ed2640 10 monclient: monmap: 2024-04-26T00:40:31.275 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:40:31.275 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:31.275 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:40:31.277724+0000 2024-04-26T00:40:31.275 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:40:31.277724+0000 2024-04-26T00:40:31.275 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:40:31.275 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:31.275 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:40:31.275 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:40:31.276 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:40:31.276 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:40:31.276 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:40:31.276 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:31.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e22ed2640 1 Processor -- start 2024-04-26T00:40:31.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e22ed2640 1 -- start start 2024-04-26T00:40:31.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e22ed2640 10 monclient: init 2024-04-26T00:40:31.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e22ed2640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:31.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e22ed2640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-04-26T00:40:31.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e22ed2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3e1c136da0 addr v1:172.21.15.37:6789/0 2024-04-26T00:40:31.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e22ed2640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3e1c137180 0x7f3e1c1375c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:31.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e22ed2640 10 monclient(hunting): picked mon.noname-b con 0x7f3e1c137180 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:40:31.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e22ed2640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f3e1c13c2d0 0x7f3e1c13e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:31.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e22ed2640 10 monclient(hunting): picked mon.noname-a con 0x7f3e1c13c2d0 addr v2:172.21.15.37:3300/0 2024-04-26T00:40:31.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e22ed2640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e1c09b4f0 con 0x7f3e1c136da0 2024-04-26T00:40:31.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e22ed2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:31.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e22ed2640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f3e1c09b210 con 0x7f3e1c13c2d0 2024-04-26T00:40:31.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e22ed2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:31.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e22ed2640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f3e1c09b7d0 con 0x7f3e1c137180 2024-04-26T00:40:31.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e22ed2640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:31.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e22ed2640 10 monclient(hunting): authenticate will time out at 2212.582997s 2024-04-26T00:40:31.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e2146f640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f3e1c13c2d0 0x7f3e1c13e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:31.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e2146f640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f3e1c13c2d0 0x7f3e1c13e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:60842/0 (socket says 172.21.15.37:60842) 2024-04-26T00:40:31.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e2146f640 1 -- 172.21.15.37:0/2335577738 learned_addr learned my addr 172.21.15.37:0/2335577738 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:40:31.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e1bfff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3e1c137180 0x7f3e1c1375c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:31.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e20c6e640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f3e1c136da0 0x7f3e1c09bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:36244/0 (socket says 172.21.15.37:36244) 2024-04-26T00:40:31.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e2146f640 10 monclient(hunting): get_auth_request con 0x7f3e1c13c2d0 auth_method 0 2024-04-26T00:40:31.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e2146f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:31.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e2146f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:31.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e2146f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:31.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e1bfff640 10 monclient(hunting): get_auth_request con 0x7f3e1c137180 auth_method 0 2024-04-26T00:40:31.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:31.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e1bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:31.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:31.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e2146f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:31.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e2146f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:31.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e2146f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:31.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e1bfff640 1 -- 172.21.15.37:0/2335577738 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3e1c137180 msgr2=0x7f3e1c1375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:40:31.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e1bfff640 1 -- 172.21.15.37:0/2335577738 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3e1c137180 msgr2=0x7f3e1c1375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:40:31.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e1bfff640 1 --2- 172.21.15.37:0/2335577738 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3e1c137180 0x7f3e1c1375c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:40:31.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e1bfff640 1 --2- 172.21.15.37:0/2335577738 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3e1c137180 0x7f3e1c1375c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:40:31.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e2146f640 10 monclient(hunting): handle_auth_done global_id 244173 payload 274 2024-04-26T00:40:31.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e2146f640 1 -- 172.21.15.37:0/2335577738 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3e1c137180 msgr2=0x7f3e1c1375c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:40:31.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e2146f640 1 --2- 172.21.15.37:0/2335577738 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3e1c137180 0x7f3e1c1375c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:31.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e2146f640 1 -- 172.21.15.37:0/2335577738 >> v1:172.21.15.37:6789/0 conn(0x7f3e1c136da0 legacy=0x7f3e1c09bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:40:31.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e2146f640 10 monclient: _finish_hunting 0 2024-04-26T00:40:31.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e2146f640 1 monclient: found mon.noname-a 2024-04-26T00:40:31.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e2146f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:31.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e2146f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:40:31.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.273+0000 7f3e2146f640 1 -- 172.21.15.37:0/2335577738 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e1c137b00 con 0x7f3e1c13c2d0 2024-04-26T00:40:31.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.277+0000 7f3e2146f640 1 -- 172.21.15.37:0/2335577738 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e1c13c2d0 msgr2=0x7f3e1c13e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:31.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.277+0000 7f3e2146f640 1 --2- 172.21.15.37:0/2335577738 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e1c13c2d0 0x7f3e1c13e6c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3e14009e50 tx=0x7f3e1401c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd09a00a91b136a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:31.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.277+0000 7f3e1b7fe640 1 -- 172.21.15.37:0/2335577738 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e140a5690 con 0x7f3e1c13c2d0 2024-04-26T00:40:31.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.277+0000 7f3e1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:31.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.277+0000 7f3e1b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:40:31.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.277+0000 7f3e1b7fe640 10 monclient: dump: 2024-04-26T00:40:31.285 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:31.285 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:31.285 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:31.285 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:31.285 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:31.285 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:31.285 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:31.285 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:31.285 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:31.286 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:31.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.277+0000 7f3e1b7fe640 10 monclient: _finish_auth 0 2024-04-26T00:40:31.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.277+0000 7f3e1b7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:40:31.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.277+0000 7f3e1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:31.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.277+0000 7f3e1b7fe640 1 -- 172.21.15.37:0/2335577738 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e140a5830 con 0x7f3e1c13c2d0 2024-04-26T00:40:31.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.277+0000 7f3e22ed2640 5 monclient: authenticate success, global_id 244173 2024-04-26T00:40:31.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.277+0000 7f3e22ed2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:40:31.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.277+0000 7f3e1b7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:40:31.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.277+0000 7f3e22ed2640 10 monclient: get_monmap_and_config success 2024-04-26T00:40:31.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.277+0000 7f3e1b7fe640 1 -- 172.21.15.37:0/2335577738 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e140a5c00 con 0x7f3e1c13c2d0 2024-04-26T00:40:31.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.277+0000 7f3e1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:31.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.277+0000 7f3e22ed2640 4 set_mon_vals no callback set 2024-04-26T00:40:31.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.277+0000 7f3e1b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:40:31.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.277+0000 7f3e1b7fe640 10 monclient: dump: 2024-04-26T00:40:31.287 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:31.287 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:31.287 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:31.287 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:31.287 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:31.287 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:31.287 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:31.287 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:31.287 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:31.287 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:31.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e22ed2640 10 monclient: shutdown 2024-04-26T00:40:31.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e22ed2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:40:31.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e22ed2640 1 -- 172.21.15.37:0/2335577738 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e1c13c2d0 msgr2=0x7f3e1c13e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:31.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e22ed2640 1 --2- 172.21.15.37:0/2335577738 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e1c13c2d0 0x7f3e1c13e6c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3e14009e50 tx=0x7f3e1401c7f0 comp rx=0 tx=0).stop 2024-04-26T00:40:31.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e22ed2640 1 -- 172.21.15.37:0/2335577738 shutdown_connections 2024-04-26T00:40:31.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e22ed2640 1 --2- 172.21.15.37:0/2335577738 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e1c13c2d0 0x7f3e1c13e6c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:31.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e22ed2640 1 --2- 172.21.15.37:0/2335577738 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3e1c137180 0x7f3e1c1375c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:31.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e22ed2640 1 -- 172.21.15.37:0/2335577738 >> 172.21.15.37:0/2335577738 conn(0x7f3e1c0a5f00 msgr2=0x7f3e1c0a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:40:31.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e22ed2640 1 -- 172.21.15.37:0/2335577738 shutdown_connections 2024-04-26T00:40:31.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e22ed2640 1 -- 172.21.15.37:0/2335577738 wait complete. 2024-04-26T00:40:31.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e22ed2640 10 monclient: build_initial_monmap 2024-04-26T00:40:31.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e22ed2640 10 monclient: monmap: 2024-04-26T00:40:31.288 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:40:31.288 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:40:31.288 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:40:31.288 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:40:31.288 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:40:31.288 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:40:31.288 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:40:31.288 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:40:31.288 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:40:31.288 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:31.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e22ed2640 1 Processor -- start 2024-04-26T00:40:31.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e22ed2640 1 -- start start 2024-04-26T00:40:31.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e22ed2640 10 monclient: init 2024-04-26T00:40:31.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e22ed2640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:31.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e22ed2640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:40:31.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e22ed2640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e1c136da0 0x7f3e1c1cee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:31.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e22ed2640 10 monclient(hunting): picked mon.noname-a con 0x7f3e1c136da0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:40:31.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e22ed2640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3e1c137180 0x7f3e1c13ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:31.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e22ed2640 10 monclient(hunting): picked mon.noname-c con 0x7f3e1c137180 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:40:31.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e22ed2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e1c13c2d0 0x7f3e1c13b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:31.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e22ed2640 10 monclient(hunting): picked mon.noname-b con 0x7f3e1c13c2d0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:40:31.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e22ed2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:31.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e22ed2640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1c1417b0 con 0x7f3e1c136da0 2024-04-26T00:40:31.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e22ed2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:31.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e22ed2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1c141630 con 0x7f3e1c13c2d0 2024-04-26T00:40:31.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e22ed2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:31.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e22ed2640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f3e1c141930 con 0x7f3e1c137180 2024-04-26T00:40:31.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e22ed2640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:31.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e22ed2640 10 monclient(hunting): authenticate will time out at 2212.589962s 2024-04-26T00:40:31.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e20c6e640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e1c136da0 0x7f3e1c1cee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:31.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e1bfff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3e1c137180 0x7f3e1c13ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:31.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e20c6e640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e1c136da0 0x7f3e1c1cee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:60846/0 (socket says 172.21.15.37:60846) 2024-04-26T00:40:31.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e20c6e640 1 -- 172.21.15.37:0/4222427308 learned_addr learned my addr 172.21.15.37:0/4222427308 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:40:31.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e1bfff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3e1c137180 0x7f3e1c13ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:52102/0 (socket says 172.21.15.37:52102) 2024-04-26T00:40:31.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e20c6e640 10 monclient(hunting): get_auth_request con 0x7f3e1c136da0 auth_method 0 2024-04-26T00:40:31.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e20c6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:31.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e20c6e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:31.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e20c6e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:31.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e1bfff640 10 monclient(hunting): get_auth_request con 0x7f3e1c137180 auth_method 0 2024-04-26T00:40:31.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:31.290 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e1bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:31.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:31.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e20c6e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:31.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e20c6e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:31.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e20c6e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:31.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e1bfff640 1 -- 172.21.15.37:0/4222427308 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3e1c137180 msgr2=0x7f3e1c13ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:40:31.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e1bfff640 1 -- 172.21.15.37:0/4222427308 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3e1c137180 msgr2=0x7f3e1c13ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:40:31.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e1bfff640 1 --2- 172.21.15.37:0/4222427308 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3e1c137180 0x7f3e1c13ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:40:31.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e1bfff640 1 --2- 172.21.15.37:0/4222427308 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3e1c137180 0x7f3e1c13ae70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:40:31.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e2146f640 1 --2- 172.21.15.37:0/4222427308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e1c13c2d0 0x7f3e1c13b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:31.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e20c6e640 10 monclient(hunting): handle_auth_done global_id 244176 payload 824 2024-04-26T00:40:31.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e20c6e640 1 -- 172.21.15.37:0/4222427308 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3e1c137180 msgr2=0x7f3e1c13ae70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:40:31.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e20c6e640 1 --2- 172.21.15.37:0/4222427308 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3e1c137180 0x7f3e1c13ae70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:31.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e20c6e640 1 -- 172.21.15.37:0/4222427308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e1c13c2d0 msgr2=0x7f3e1c13b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:31.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e20c6e640 1 --2- 172.21.15.37:0/4222427308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e1c13c2d0 0x7f3e1c13b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:31.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e20c6e640 10 monclient: _finish_hunting 0 2024-04-26T00:40:31.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e20c6e640 1 monclient: found mon.noname-a 2024-04-26T00:40:31.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e20c6e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:31.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e20c6e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:40:31.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e20c6e640 1 -- 172.21.15.37:0/4222427308 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e1c0a85d0 con 0x7f3e1c136da0 2024-04-26T00:40:31.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e2146f640 10 monclient: get_auth_request con 0x7f3e1c13c2d0 auth_method 0 2024-04-26T00:40:31.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.281+0000 7f3e2146f640 1 --2- 172.21.15.37:0/4222427308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e1c13c2d0 0x7f3e1c13b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:40:31.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.285+0000 7f3e20c6e640 1 -- 172.21.15.37:0/4222427308 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e1c136da0 msgr2=0x7f3e1c1cee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:31.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.285+0000 7f3e20c6e640 1 --2- 172.21.15.37:0/4222427308 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e1c136da0 0x7f3e1c1cee90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3e1000c460 tx=0x7f3e1000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:31.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.285+0000 7f3e197fa640 1 -- 172.21.15.37:0/4222427308 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e10017070 con 0x7f3e1c136da0 2024-04-26T00:40:31.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.285+0000 7f3e197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:31.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.285+0000 7f3e197fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:40:31.294 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.285+0000 7f3e197fa640 10 monclient: dump: 2024-04-26T00:40:31.294 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:31.294 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:31.294 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:31.294 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:31.294 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:31.295 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:31.295 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:31.295 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:31.295 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:31.295 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:31.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.285+0000 7f3e197fa640 10 monclient: _finish_auth 0 2024-04-26T00:40:31.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.285+0000 7f3e197fa640 10 monclient: _check_auth_tickets 2024-04-26T00:40:31.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.285+0000 7f3e197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:31.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.285+0000 7f3e197fa640 1 -- 172.21.15.37:0/4222427308 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e10005820 con 0x7f3e1c136da0 2024-04-26T00:40:31.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.285+0000 7f3e197fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:40:31.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.285+0000 7f3e197fa640 1 -- 172.21.15.37:0/4222427308 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e10005c10 con 0x7f3e1c136da0 2024-04-26T00:40:31.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.285+0000 7f3e197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:31.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.285+0000 7f3e197fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:40:31.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.285+0000 7f3e197fa640 10 monclient: dump: 2024-04-26T00:40:31.295 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:31.296 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:31.296 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:31.296 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:31.296 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:31.296 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:31.296 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:31.296 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:31.296 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:31.296 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:31.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.285+0000 7f3e22ed2640 5 monclient: authenticate success, global_id 244176 2024-04-26T00:40:31.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.285+0000 7f3e22ed2640 10 monclient: _renew_subs 2024-04-26T00:40:31.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.285+0000 7f3e1affd640 4 set_mon_vals no callback set 2024-04-26T00:40:31.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.285+0000 7f3e22ed2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:40:31.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.285+0000 7f3e22ed2640 1 -- 172.21.15.37:0/4222427308 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e1c139940 con 0x7f3e1c136da0 2024-04-26T00:40:31.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.285+0000 7f3e22ed2640 10 monclient: _renew_subs 2024-04-26T00:40:31.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.285+0000 7f3e22ed2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:40:31.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.285+0000 7f3e22ed2640 1 -- 172.21.15.37:0/4222427308 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e1c1390a0 con 0x7f3e1c136da0 2024-04-26T00:40:31.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.285+0000 7f3e22ed2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:31.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.285+0000 7f3e22ed2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:31.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.285+0000 7f3e22ed2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:40:31.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.285+0000 7f3e22ed2640 1 -- 172.21.15.37:0/4222427308 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e1c066f40 con 0x7f3e1c136da0 2024-04-26T00:40:31.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.285+0000 7f3e197fa640 1 -- 172.21.15.37:0/4222427308 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f3e1001a070 con 0x7f3e1c136da0 2024-04-26T00:40:31.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.285+0000 7f3e197fa640 1 --2- 172.21.15.37:0/4222427308 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3e00035ed0 0x7f3e00038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:31.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.285+0000 7f3e197fa640 1 -- 172.21.15.37:0/4222427308 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(813..813 src has 705..813) ==== 3975+0+0 (secure 0 0 0) 0x7f3e10047f60 con 0x7f3e1c136da0 2024-04-26T00:40:31.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.285+0000 7f3e1bfff640 1 --2- 172.21.15.37:0/4222427308 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3e00035ed0 0x7f3e00038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:31.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.285+0000 7f3e1bfff640 10 monclient: get_auth_request con 0x7f3e00035ed0 auth_method 0 2024-04-26T00:40:31.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.289+0000 7f3e1bfff640 1 --2- 172.21.15.37:0/4222427308 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3e00035ed0 0x7f3e00038390 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f3e08006aa0 tx=0x7f3e08006a30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:31.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.289+0000 7f3e197fa640 1 -- 172.21.15.37:0/4222427308 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f3e1c0a85d0 con 0x7f3e1c136da0 2024-04-26T00:40:31.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.289+0000 7f3e197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:31.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.289+0000 7f3e197fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:40:31.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.517+0000 7f3e22ed2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:40:31.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.517+0000 7f3e22ed2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:40:31.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.517+0000 7f3e22ed2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:40:31.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.517+0000 7f3e22ed2640 1 -- 172.21.15.37:0/4222427308 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3e1c1381b0 con 0x7f3e1c136da0 2024-04-26T00:40:31.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.517+0000 7f3e197fa640 1 -- 172.21.15.37:0/4222427308 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7f3e10010300 con 0x7f3e1c136da0 2024-04-26T00:40:31.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.517+0000 7f3e197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:40:31.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.517+0000 7f3e197fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:40:31.519 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":150,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:40:31.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.521+0000 7f3e22ed2640 1 -- 172.21.15.37:0/4222427308 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3e00035ed0 msgr2=0x7f3e00038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:31.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.521+0000 7f3e22ed2640 1 --2- 172.21.15.37:0/4222427308 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3e00035ed0 0x7f3e00038390 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f3e08006aa0 tx=0x7f3e08006a30 comp rx=0 tx=0).stop 2024-04-26T00:40:31.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.521+0000 7f3e22ed2640 10 monclient: shutdown 2024-04-26T00:40:31.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.521+0000 7f3e22ed2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:40:31.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.521+0000 7f3e22ed2640 1 -- 172.21.15.37:0/4222427308 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e1c136da0 msgr2=0x7f3e1c1cee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:31.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.521+0000 7f3e22ed2640 1 --2- 172.21.15.37:0/4222427308 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e1c136da0 0x7f3e1c1cee90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3e1000c460 tx=0x7f3e1000c920 comp rx=0 tx=0).stop 2024-04-26T00:40:31.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.521+0000 7f3e22ed2640 1 -- 172.21.15.37:0/4222427308 shutdown_connections 2024-04-26T00:40:31.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.521+0000 7f3e22ed2640 1 --2- 172.21.15.37:0/4222427308 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3e00035ed0 0x7f3e00038390 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:31.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.521+0000 7f3e22ed2640 1 --2- 172.21.15.37:0/4222427308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e1c13c2d0 0x7f3e1c13b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:31.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.521+0000 7f3e22ed2640 1 --2- 172.21.15.37:0/4222427308 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3e1c137180 0x7f3e1c13ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:31.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.521+0000 7f3e22ed2640 1 --2- 172.21.15.37:0/4222427308 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3e1c136da0 0x7f3e1c1cee90 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:31.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.521+0000 7f3e22ed2640 1 -- 172.21.15.37:0/4222427308 >> 172.21.15.37:0/4222427308 conn(0x7f3e1c0a5f00 msgr2=0x7f3e1c13d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:40:31.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.521+0000 7f3e22ed2640 1 -- 172.21.15.37:0/4222427308 shutdown_connections 2024-04-26T00:40:31.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:31.521+0000 7f3e22ed2640 1 -- 172.21.15.37:0/4222427308 wait complete. 2024-04-26T00:40:31.888 INFO:tasks.rados.rados.0.smithi045.stdout:3614: done (7 left) 2024-04-26T00:40:31.888 INFO:tasks.rados.rados.0.smithi045.stdout:3615: write (excl) oid 9 current snap is 398 2024-04-26T00:40:31.888 INFO:tasks.rados.rados.0.smithi045.stdout:3615: seq_num 1285 ranges {796995=758622,2161832=732447} 2024-04-26T00:40:31.899 INFO:tasks.rados.rados.0.smithi045.stdout:3615: writing smithi04526579-9 from 796995 to 1555617 tid 1 2024-04-26T00:40:31.909 INFO:tasks.rados.rados.0.smithi045.stdout:3615: writing smithi04526579-9 from 2161832 to 2894279 tid 2 2024-04-26T00:40:31.909 INFO:tasks.rados.rados.0.smithi045.stdout:3616: snap_remove snap 373 2024-04-26T00:40:32.674 INFO:tasks.rados.rados.0.smithi045.stdout:3616: done (8 left) 2024-04-26T00:40:32.674 INFO:tasks.rados.rados.0.smithi045.stdout:3617: delete oid 38 current snap is 398 2024-04-26T00:40:32.675 INFO:tasks.rados.rados.0.smithi045.stdout:3606: finishing copy_from to smithi04526579-10 2024-04-26T00:40:32.675 INFO:tasks.rados.rados.0.smithi045.stdout:3606: got expected ENOENT (src dne) 2024-04-26T00:40:32.675 INFO:tasks.rados.rados.0.smithi045.stdout:3606: finishing copy_from racing read to smithi04526579-10 2024-04-26T00:40:32.675 INFO:tasks.rados.rados.0.smithi045.stdout:3608: finishing write tid 1 to smithi04526579-16 2024-04-26T00:40:32.675 INFO:tasks.rados.rados.0.smithi045.stdout:3608: finishing write tid 2 to smithi04526579-16 2024-04-26T00:40:32.675 INFO:tasks.rados.rados.0.smithi045.stdout:3608: finishing write tid 3 to smithi04526579-16 2024-04-26T00:40:32.675 INFO:tasks.rados.rados.0.smithi045.stdout:3608: finishing write tid 4 to smithi04526579-16 2024-04-26T00:40:32.675 INFO:tasks.rados.rados.0.smithi045.stdout:3608: finishing write tid 5 to smithi04526579-16 2024-04-26T00:40:32.675 INFO:tasks.rados.rados.0.smithi045.stdout:3608: oid 16 updating version 0 to 1211 2024-04-26T00:40:32.675 INFO:tasks.rados.rados.0.smithi045.stdout:3608: oid 16 updating version 1211 to 1214 2024-04-26T00:40:32.675 INFO:tasks.rados.rados.0.smithi045.stdout:3608: oid 16 version 1214 is already newer than 1213 2024-04-26T00:40:32.675 INFO:tasks.rados.rados.0.smithi045.stdout:3608: oid 16 version 1214 is already newer than 1212 2024-04-26T00:40:32.675 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 1214 (ObjNum 1281 snap 398 seq_num 1281) dirty exists 2024-04-26T00:40:32.675 INFO:tasks.rados.rados.0.smithi045.stdout:3608: left oid 16 (ObjNum 1281 snap 398 seq_num 1281) 2024-04-26T00:40:32.675 INFO:tasks.rados.rados.0.smithi045.stdout:3610: finishing write tid 1 to smithi04526579-4 2024-04-26T00:40:32.676 INFO:tasks.rados.rados.0.smithi045.stdout:3610: finishing write tid 2 to smithi04526579-4 2024-04-26T00:40:32.676 INFO:tasks.rados.rados.0.smithi045.stdout:3610: finishing write tid 3 to smithi04526579-4 2024-04-26T00:40:32.676 INFO:tasks.rados.rados.0.smithi045.stdout:3610: finishing write tid 4 to smithi04526579-4 2024-04-26T00:40:32.676 INFO:tasks.rados.rados.0.smithi045.stdout:3610: finishing write tid 5 to smithi04526579-4 2024-04-26T00:40:32.676 INFO:tasks.rados.rados.0.smithi045.stdout:3610: oid 4 updating version 0 to 2518 2024-04-26T00:40:32.676 INFO:tasks.rados.rados.0.smithi045.stdout:3610: oid 4 updating version 2518 to 2520 2024-04-26T00:40:32.676 INFO:tasks.rados.rados.0.smithi045.stdout:3610: oid 4 version 2520 is already newer than 2517 2024-04-26T00:40:32.676 INFO:tasks.rados.rados.0.smithi045.stdout:3610: oid 4 version 2520 is already newer than 2519 2024-04-26T00:40:32.676 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 2520 (ObjNum 1282 snap 398 seq_num 1282) dirty exists 2024-04-26T00:40:32.676 INFO:tasks.rados.rados.0.smithi045.stdout:3610: left oid 4 (ObjNum 1282 snap 398 seq_num 1282) 2024-04-26T00:40:32.676 INFO:tasks.rados.rados.0.smithi045.stdout:3611: finishing write tid 1 to smithi04526579-8 2024-04-26T00:40:32.676 INFO:tasks.rados.rados.0.smithi045.stdout:3611: finishing write tid 2 to smithi04526579-8 2024-04-26T00:40:32.676 INFO:tasks.rados.rados.0.smithi045.stdout:3611: finishing write tid 3 to smithi04526579-8 2024-04-26T00:40:32.676 INFO:tasks.rados.rados.0.smithi045.stdout:3611: finishing write tid 4 to smithi04526579-8 2024-04-26T00:40:32.676 INFO:tasks.rados.rados.0.smithi045.stdout:3611: finishing write tid 5 to smithi04526579-8 2024-04-26T00:40:32.677 INFO:tasks.rados.rados.0.smithi045.stdout:3611: oid 8 updating version 0 to 2233 2024-04-26T00:40:32.677 INFO:tasks.rados.rados.0.smithi045.stdout:3611: oid 8 updating version 2233 to 2234 2024-04-26T00:40:32.677 INFO:tasks.rados.rados.0.smithi045.stdout:3611: oid 8 version 2234 is already newer than 2231 2024-04-26T00:40:32.677 INFO:tasks.rados.rados.0.smithi045.stdout:3611: oid 8 version 2234 is already newer than 2232 2024-04-26T00:40:32.677 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 2234 (ObjNum 1283 snap 398 seq_num 1283) dirty exists 2024-04-26T00:40:32.677 INFO:tasks.rados.rados.0.smithi045.stdout:3611: left oid 8 (ObjNum 1283 snap 398 seq_num 1283) 2024-04-26T00:40:32.677 INFO:tasks.rados.rados.0.smithi045.stdout:3613: finishing write tid 1 to smithi04526579-49 2024-04-26T00:40:32.677 INFO:tasks.rados.rados.0.smithi045.stdout:3613: finishing write tid 2 to smithi04526579-49 2024-04-26T00:40:32.677 INFO:tasks.rados.rados.0.smithi045.stdout:3613: finishing write tid 3 to smithi04526579-49 2024-04-26T00:40:32.677 INFO:tasks.rados.rados.0.smithi045.stdout:3613: finishing write tid 4 to smithi04526579-49 2024-04-26T00:40:32.677 INFO:tasks.rados.rados.0.smithi045.stdout:3613: finishing write tid 5 to smithi04526579-49 2024-04-26T00:40:32.677 INFO:tasks.rados.rados.0.smithi045.stdout:3613: oid 49 updating version 0 to 2494 2024-04-26T00:40:32.677 INFO:tasks.rados.rados.0.smithi045.stdout:3613: oid 49 version 2494 is already newer than 2493 2024-04-26T00:40:32.677 INFO:tasks.rados.rados.0.smithi045.stdout:3613: oid 49 updating version 2494 to 2496 2024-04-26T00:40:32.677 INFO:tasks.rados.rados.0.smithi045.stdout:3613: oid 49 version 2496 is already newer than 2495 2024-04-26T00:40:32.678 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 2496 (ObjNum 1284 snap 398 seq_num 1284) dirty exists 2024-04-26T00:40:32.678 INFO:tasks.rados.rados.0.smithi045.stdout:3613: left oid 49 (ObjNum 1284 snap 398 seq_num 1284) 2024-04-26T00:40:32.678 INFO:tasks.rados.rados.0.smithi045.stdout:3615: finishing write tid 1 to smithi04526579-9 2024-04-26T00:40:32.678 INFO:tasks.rados.rados.0.smithi045.stdout:3615: finishing write tid 2 to smithi04526579-9 2024-04-26T00:40:32.678 INFO:tasks.rados.rados.0.smithi045.stdout:3615: finishing write tid 3 to smithi04526579-9 2024-04-26T00:40:32.678 INFO:tasks.rados.rados.0.smithi045.stdout:3615: finishing write tid 4 to smithi04526579-9 2024-04-26T00:40:32.678 INFO:tasks.rados.rados.0.smithi045.stdout:3615: oid 9 updating version 0 to 2522 2024-04-26T00:40:32.678 INFO:tasks.rados.rados.0.smithi045.stdout:3615: oid 9 version 2522 is already newer than 2521 2024-04-26T00:40:32.678 INFO:tasks.rados.rados.0.smithi045.stdout:3615: oid 9 updating version 2522 to 2523 2024-04-26T00:40:32.678 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 2523 (ObjNum 1285 snap 398 seq_num 1285) dirty exists 2024-04-26T00:40:32.678 INFO:tasks.rados.rados.0.smithi045.stdout:3615: left oid 9 (ObjNum 1285 snap 398 seq_num 1285) 2024-04-26T00:40:32.707 INFO:tasks.rados.rados.0.smithi045.stdout:3606: done (8 left) 2024-04-26T00:40:32.707 INFO:tasks.rados.rados.0.smithi045.stdout:3607: done (7 left) 2024-04-26T00:40:32.707 INFO:tasks.rados.rados.0.smithi045.stdout:3608: done (6 left) 2024-04-26T00:40:32.707 INFO:tasks.rados.rados.0.smithi045.stdout:3609: done (5 left) 2024-04-26T00:40:32.707 INFO:tasks.rados.rados.0.smithi045.stdout:3610: done (4 left) 2024-04-26T00:40:32.708 INFO:tasks.rados.rados.0.smithi045.stdout:3611: done (3 left) 2024-04-26T00:40:32.708 INFO:tasks.rados.rados.0.smithi045.stdout:3613: done (2 left) 2024-04-26T00:40:32.708 INFO:tasks.rados.rados.0.smithi045.stdout:3615: done (1 left) 2024-04-26T00:40:32.708 INFO:tasks.rados.rados.0.smithi045.stdout:3617: done (0 left) 2024-04-26T00:40:32.708 INFO:tasks.rados.rados.0.smithi045.stdout:3618: rollback oid 1 current snap is 398 2024-04-26T00:40:32.708 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 1 to 388 2024-04-26T00:40:32.708 INFO:tasks.rados.rados.0.smithi045.stdout:3619: snap_create 2024-04-26T00:40:32.718 INFO:tasks.rados.rados.0.smithi045.stdout:3618: finishing rollback tid 0 to smithi04526579-1 2024-04-26T00:40:32.718 INFO:tasks.rados.rados.0.smithi045.stdout:3618: finishing rollback tid 1 to smithi04526579-1 2024-04-26T00:40:32.719 INFO:tasks.rados.rados.0.smithi045.stdout:3618: finishing rollback tid 2 to smithi04526579-1 2024-04-26T00:40:32.719 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 2513 (ObjNum 1079 snap 322 seq_num 1079) dirty exists 2024-04-26T00:40:34.156 INFO:tasks.rados.rados.0.smithi045.stdout:3618: done (1 left) 2024-04-26T00:40:34.157 INFO:tasks.rados.rados.0.smithi045.stdout:3619: done (0 left) 2024-04-26T00:40:34.157 INFO:tasks.rados.rados.0.smithi045.stdout:3620: delete oid 25 current snap is 399 2024-04-26T00:40:34.165 INFO:tasks.rados.rados.0.smithi045.stdout:3620: done (0 left) 2024-04-26T00:40:34.165 INFO:tasks.rados.rados.0.smithi045.stdout:3621: read oid 1 snap -1 2024-04-26T00:40:34.165 INFO:tasks.rados.rados.0.smithi045.stdout:3621: expect (ObjNum 1079 snap 322 seq_num 1079) 2024-04-26T00:40:34.165 INFO:tasks.rados.rados.0.smithi045.stdout:3622: snap_create 2024-04-26T00:40:34.178 INFO:tasks.rados.rados.0.smithi045.stdout:3621: expect (ObjNum 1079 snap 322 seq_num 1079) 2024-04-26T00:40:34.540 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:40:34.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.669+0000 7f6b2dd00640 10 monclient: get_monmap_and_config 2024-04-26T00:40:34.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.669+0000 7f6b2dd00640 10 monclient: build_initial_monmap 2024-04-26T00:40:34.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.669+0000 7f6b2dd00640 10 monclient: monmap: 2024-04-26T00:40:34.671 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:40:34.671 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:34.671 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:40:34.673428+0000 2024-04-26T00:40:34.671 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:40:34.673428+0000 2024-04-26T00:40:34.671 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:40:34.671 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:34.671 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:40:34.671 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:40:34.671 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:40:34.671 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:40:34.672 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:40:34.672 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:34.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.669+0000 7f6b2dd00640 1 Processor -- start 2024-04-26T00:40:34.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.669+0000 7f6b2dd00640 1 -- start start 2024-04-26T00:40:34.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.669+0000 7f6b2dd00640 10 monclient: init 2024-04-26T00:40:34.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.669+0000 7f6b2dd00640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:34.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.669+0000 7f6b2dd00640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-04-26T00:40:34.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.669+0000 7f6b2dd00640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6b28136da0 addr v1:172.21.15.37:6789/0 2024-04-26T00:40:34.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.669+0000 7f6b2dd00640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6b28137180 addr v1:172.21.15.45:6789/0 2024-04-26T00:40:34.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.669+0000 7f6b2dd00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6b2813d2e0 0x7f6b2813f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:34.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.669+0000 7f6b2dd00640 10 monclient(hunting): picked mon.noname-b con 0x7f6b2813d2e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:40:34.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.669+0000 7f6b2dd00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b2809b4f0 con 0x7f6b28136da0 2024-04-26T00:40:34.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.669+0000 7f6b2dd00640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b2809b7d0 con 0x7f6b28137180 2024-04-26T00:40:34.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.669+0000 7f6b2dd00640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:34.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.669+0000 7f6b2dd00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f6b2809bb40 con 0x7f6b2813d2e0 2024-04-26T00:40:34.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.669+0000 7f6b2dd00640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:34.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.669+0000 7f6b2dd00640 10 monclient(hunting): authenticate will time out at 2215.979069s 2024-04-26T00:40:34.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.669+0000 7f6b277fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f6b28136da0 0x7f6b2809bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:36252/0 (socket says 172.21.15.37:36252) 2024-04-26T00:40:34.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.669+0000 7f6b277fe640 1 -- 172.21.15.37:0/2251996684 learned_addr learned my addr 172.21.15.37:0/2251996684 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:40:34.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.669+0000 7f6b27fff640 1 --2- 172.21.15.37:0/2251996684 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6b2813d2e0 0x7f6b2813f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:34.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.669+0000 7f6b27fff640 10 monclient(hunting): get_auth_request con 0x7f6b2813d2e0 auth_method 0 2024-04-26T00:40:34.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.669+0000 7f6b27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:34.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.669+0000 7f6b27fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:34.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.669+0000 7f6b27fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:34.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.669+0000 7f6b27fff640 1 -- 172.21.15.37:0/2251996684 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6b2813d2e0 msgr2=0x7f6b2813f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:40:34.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.669+0000 7f6b27fff640 1 -- 172.21.15.37:0/2251996684 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6b2813d2e0 msgr2=0x7f6b2813f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:40:34.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.669+0000 7f6b27fff640 1 --2- 172.21.15.37:0/2251996684 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6b2813d2e0 0x7f6b2813f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:40:34.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.669+0000 7f6b27fff640 1 --2- 172.21.15.37:0/2251996684 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6b2813d2e0 0x7f6b2813f6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:40:34.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b277fe640 1 -- 172.21.15.37:0/2251996684 >> v1:172.21.15.37:6789/0 conn(0x7f6b28136da0 legacy=0x7f6b2809bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:34.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b267fc640 1 -- 172.21.15.37:0/2251996684 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2044577932 0 0) 0x7f6b2809b4f0 con 0x7f6b28136da0 2024-04-26T00:40:34.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b267fc640 10 monclient(hunting): _negotiate 2024-04-26T00:40:34.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b267fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:34.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b267fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:34.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b267fc640 10 monclient(hunting): my global_id is 244179 2024-04-26T00:40:34.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b267fc640 1 -- 172.21.15.37:0/2251996684 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b10003530 con 0x7f6b28136da0 2024-04-26T00:40:34.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b26ffd640 1 -- 172.21.15.37:0/2251996684 >> v1:172.21.15.45:6789/0 conn(0x7f6b28137180 legacy=0x7f6b281375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:34.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b267fc640 1 -- 172.21.15.37:0/2251996684 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 706861908 0 0) 0x7f6b10003530 con 0x7f6b28136da0 2024-04-26T00:40:34.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b267fc640 1 -- 172.21.15.37:0/2251996684 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6b2813d2e0 msgr2=0x7f6b2813f6d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:40:34.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b267fc640 1 --2- 172.21.15.37:0/2251996684 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6b2813d2e0 0x7f6b2813f6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:34.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b267fc640 1 -- 172.21.15.37:0/2251996684 >> v1:172.21.15.45:6789/0 conn(0x7f6b28137180 legacy=0x7f6b281375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:34.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b267fc640 10 monclient: _finish_hunting 0 2024-04-26T00:40:34.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b267fc640 1 monclient: found mon.noname-a-legacy 2024-04-26T00:40:34.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:34.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b267fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.37:6789/0 2024-04-26T00:40:34.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b267fc640 1 -- 172.21.15.37:0/2251996684 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b2809c5e0 con 0x7f6b28136da0 2024-04-26T00:40:34.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b267fc640 10 monclient: _finish_auth 0 2024-04-26T00:40:34.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b267fc640 10 monclient: _check_auth_tickets 2024-04-26T00:40:34.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:34.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b267fc640 1 -- 172.21.15.37:0/2251996684 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 866326386 0 0) 0x7f6b18001930 con 0x7f6b28136da0 2024-04-26T00:40:34.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:34.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b267fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-26T00:40:34.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b267fc640 10 monclient: dump: 2024-04-26T00:40:34.680 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:34.680 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:34.680 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:34.680 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:34.680 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:34.680 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:34.680 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:34.680 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:34.680 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:34.680 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:34.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b267fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] but i'm connected to v1:172.21.15.37:6789/0, reconnecting 2024-04-26T00:40:34.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b267fc640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:34.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b267fc640 1 -- 172.21.15.37:0/2251996684 >> v1:172.21.15.37:6789/0 conn(0x7f6b28136da0 legacy=0x7f6b2809bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:34.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b267fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:40:34.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b267fc640 1 --2- 172.21.15.37:0/2251996684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b10003a70 0x7f6b10003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:34.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b267fc640 10 monclient(hunting): picked mon.b con 0x7f6b10003a70 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:40:34.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b267fc640 1 --2- 172.21.15.37:0/2251996684 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6b10004810 0x7f6b10004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:34.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b267fc640 10 monclient(hunting): picked mon.a con 0x7f6b10004810 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:40:34.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b267fc640 1 --2- 172.21.15.37:0/2251996684 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6b2813d2e0 0x7f6b1000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:34.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b267fc640 10 monclient(hunting): picked mon.c con 0x7f6b2813d2e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:40:34.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b267fc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:34.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b267fc640 1 -- 172.21.15.37:0/2251996684 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f6b2809bb40 con 0x7f6b10004810 2024-04-26T00:40:34.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b267fc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:34.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b267fc640 1 -- 172.21.15.37:0/2251996684 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6b10005660 con 0x7f6b10003a70 2024-04-26T00:40:34.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b267fc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:34.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b267fc640 1 -- 172.21.15.37:0/2251996684 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f6b1000bf60 con 0x7f6b2813d2e0 2024-04-26T00:40:34.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b267fc640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:34.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b267fc640 10 monclient(hunting): _finish_auth 0 2024-04-26T00:40:34.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b26ffd640 1 --2- 172.21.15.37:0/2251996684 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6b10004810 0x7f6b10004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:34.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b27fff640 1 --2- 172.21.15.37:0/2251996684 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6b2813d2e0 0x7f6b1000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:34.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b27fff640 10 monclient(hunting): get_auth_request con 0x7f6b2813d2e0 auth_method 0 2024-04-26T00:40:34.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:34.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b27fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:34.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b27fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:40:34.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b26ffd640 10 monclient(hunting): get_auth_request con 0x7f6b10004810 auth_method 0 2024-04-26T00:40:34.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b26ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:34.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b26ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:34.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b26ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:40:34.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b27fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:34.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b27fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:34.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b27fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:40:34.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b26ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:34.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b26ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:34.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b26ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:40:34.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b277fe640 1 --2- 172.21.15.37:0/2251996684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b10003a70 0x7f6b10003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:34.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b277fe640 10 monclient(hunting): get_auth_request con 0x7f6b10003a70 auth_method 0 2024-04-26T00:40:34.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:34.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b277fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:34.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b277fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:40:34.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b27fff640 10 monclient(hunting): handle_auth_done global_id 244179 payload 293 2024-04-26T00:40:34.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b27fff640 1 -- 172.21.15.37:0/2251996684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b10003a70 msgr2=0x7f6b10003e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:40:34.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b27fff640 1 --2- 172.21.15.37:0/2251996684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b10003a70 0x7f6b10003e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:34.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b27fff640 1 -- 172.21.15.37:0/2251996684 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6b10004810 msgr2=0x7f6b10004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:40:34.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b27fff640 1 --2- 172.21.15.37:0/2251996684 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6b10004810 0x7f6b10004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:34.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b27fff640 10 monclient: _finish_hunting 0 2024-04-26T00:40:34.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b27fff640 1 monclient: found mon.c 2024-04-26T00:40:34.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b27fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:34.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b27fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:40:34.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b27fff640 1 -- 172.21.15.37:0/2251996684 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b10003530 con 0x7f6b2813d2e0 2024-04-26T00:40:34.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b27fff640 10 monclient: _finish_auth 0 2024-04-26T00:40:34.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b27fff640 10 monclient: _check_auth_tickets 2024-04-26T00:40:34.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:34.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b2dd00640 5 monclient: authenticate success, global_id 244179 2024-04-26T00:40:34.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.673+0000 7f6b2dd00640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:40:34.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.677+0000 7f6b27fff640 1 -- 172.21.15.37:0/2251996684 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6b2813d2e0 msgr2=0x7f6b1000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:34.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.677+0000 7f6b27fff640 1 --2- 172.21.15.37:0/2251996684 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6b2813d2e0 0x7f6b1000b9d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6b20009fd0 tx=0x7f6b2001cde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e41b68a6767d2fbf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:34.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.677+0000 7f6b267fc640 1 -- 172.21.15.37:0/2251996684 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b200b2070 con 0x7f6b2813d2e0 2024-04-26T00:40:34.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.677+0000 7f6b267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:34.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.677+0000 7f6b267fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:40:34.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.677+0000 7f6b267fc640 10 monclient: dump: 2024-04-26T00:40:34.691 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:34.691 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:34.691 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:34.691 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:34.691 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:34.691 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:34.691 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:34.691 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:34.691 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:34.691 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:34.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.677+0000 7f6b2dd00640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:40:34.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.677+0000 7f6b267fc640 1 -- 172.21.15.37:0/2251996684 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b200a5ac0 con 0x7f6b2813d2e0 2024-04-26T00:40:34.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.677+0000 7f6b267fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:40:34.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.677+0000 7f6b2dd00640 10 monclient: get_monmap_and_config success 2024-04-26T00:40:34.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.677+0000 7f6b267fc640 1 -- 172.21.15.37:0/2251996684 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b20009bd0 con 0x7f6b2813d2e0 2024-04-26T00:40:34.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.677+0000 7f6b267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:34.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.677+0000 7f6b2dd00640 4 set_mon_vals no callback set 2024-04-26T00:40:34.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.677+0000 7f6b267fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:40:34.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.677+0000 7f6b267fc640 10 monclient: dump: 2024-04-26T00:40:34.692 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:34.692 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:34.692 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:34.692 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:34.692 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:34.692 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:34.692 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:34.692 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:34.692 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:34.693 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:34.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.677+0000 7f6b2dd00640 10 monclient: shutdown 2024-04-26T00:40:34.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.677+0000 7f6b2dd00640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:40:34.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.677+0000 7f6b2dd00640 1 -- 172.21.15.37:0/2251996684 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6b2813d2e0 msgr2=0x7f6b1000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:34.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.677+0000 7f6b2dd00640 1 --2- 172.21.15.37:0/2251996684 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6b2813d2e0 0x7f6b1000b9d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6b20009fd0 tx=0x7f6b2001cde0 comp rx=0 tx=0).stop 2024-04-26T00:40:34.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.677+0000 7f6b27fff640 1 -- 172.21.15.37:0/2251996684 reap_dead start 2024-04-26T00:40:34.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.677+0000 7f6b2dd00640 1 -- 172.21.15.37:0/2251996684 shutdown_connections 2024-04-26T00:40:34.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.677+0000 7f6b2dd00640 1 -- 172.21.15.37:0/2251996684 >> 172.21.15.37:0/2251996684 conn(0x7f6b280a5f00 msgr2=0x7f6b280a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:40:34.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.677+0000 7f6b2dd00640 1 -- 172.21.15.37:0/2251996684 shutdown_connections 2024-04-26T00:40:34.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.677+0000 7f6b2dd00640 1 -- 172.21.15.37:0/2251996684 wait complete. 2024-04-26T00:40:34.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b2dd00640 10 monclient: build_initial_monmap 2024-04-26T00:40:34.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b2dd00640 10 monclient: monmap: 2024-04-26T00:40:34.693 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:40:34.693 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:40:34.693 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:40:34.694 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:40:34.694 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:40:34.694 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:40:34.694 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:40:34.694 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:40:34.694 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:40:34.694 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:34.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b2dd00640 1 Processor -- start 2024-04-26T00:40:34.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b2dd00640 1 -- start start 2024-04-26T00:40:34.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b2dd00640 10 monclient: init 2024-04-26T00:40:34.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b2dd00640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:34.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b2dd00640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:40:34.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b2dd00640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b28132f40 0x7f6b28133320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:34.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b2dd00640 10 monclient(hunting): picked mon.noname-b con 0x7f6b28132f40 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:40:34.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b2dd00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6b2813a5b0 0x7f6b281ce9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:34.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b2dd00640 10 monclient(hunting): picked mon.noname-c con 0x7f6b2813a5b0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:40:34.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b2dd00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6b281cef20 0x7f6b281cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:34.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b2dd00640 10 monclient(hunting): picked mon.noname-a con 0x7f6b281cef20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:40:34.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b2dd00640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:34.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b2dd00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f6b28140270 con 0x7f6b281cef20 2024-04-26T00:40:34.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b2dd00640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:34.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b2dd00640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6b281400f0 con 0x7f6b28132f40 2024-04-26T00:40:34.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b2dd00640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:34.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b2dd00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f6b281403f0 con 0x7f6b2813a5b0 2024-04-26T00:40:34.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b2dd00640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:34.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b2dd00640 10 monclient(hunting): authenticate will time out at 2215.989479s 2024-04-26T00:40:34.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b26ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6b2813a5b0 0x7f6b281ce9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:34.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b27fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6b281cef20 0x7f6b281cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:34.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b26ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6b2813a5b0 0x7f6b281ce9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:52126/0 (socket says 172.21.15.37:52126) 2024-04-26T00:40:34.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b26ffd640 1 -- 172.21.15.37:0/3001632634 learned_addr learned my addr 172.21.15.37:0/3001632634 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:40:34.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b27fff640 10 monclient(hunting): get_auth_request con 0x7f6b281cef20 auth_method 0 2024-04-26T00:40:34.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:34.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b27fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:34.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b27fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:34.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b26ffd640 10 monclient(hunting): get_auth_request con 0x7f6b2813a5b0 auth_method 0 2024-04-26T00:40:34.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b26ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:34.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b26ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:34.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:34.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b27fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:34.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b27fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:34.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b27fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:34.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b277fe640 1 --2- 172.21.15.37:0/3001632634 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b28132f40 0x7f6b28133320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:34.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b277fe640 10 monclient(hunting): get_auth_request con 0x7f6b28132f40 auth_method 0 2024-04-26T00:40:34.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:34.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b277fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:34.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b277fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:34.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b26ffd640 1 -- 172.21.15.37:0/3001632634 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6b2813a5b0 msgr2=0x7f6b281ce9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:40:34.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b26ffd640 1 -- 172.21.15.37:0/3001632634 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6b2813a5b0 msgr2=0x7f6b281ce9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:40:34.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b26ffd640 1 --2- 172.21.15.37:0/3001632634 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6b2813a5b0 0x7f6b281ce9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:40:34.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b26ffd640 1 --2- 172.21.15.37:0/3001632634 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6b2813a5b0 0x7f6b281ce9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:40:34.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:34.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:34.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:34.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b27fff640 10 monclient(hunting): handle_auth_done global_id 244182 payload 824 2024-04-26T00:40:34.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b27fff640 1 -- 172.21.15.37:0/3001632634 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6b2813a5b0 msgr2=0x7f6b281ce9e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:40:34.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b27fff640 1 --2- 172.21.15.37:0/3001632634 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6b2813a5b0 0x7f6b281ce9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:34.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b27fff640 1 -- 172.21.15.37:0/3001632634 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b28132f40 msgr2=0x7f6b28133320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:34.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b27fff640 1 --2- 172.21.15.37:0/3001632634 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b28132f40 0x7f6b28133320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:34.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b27fff640 10 monclient: _finish_hunting 0 2024-04-26T00:40:34.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b27fff640 1 monclient: found mon.noname-a 2024-04-26T00:40:34.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b27fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:34.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b27fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:40:34.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.681+0000 7f6b27fff640 1 -- 172.21.15.37:0/3001632634 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b280a83a0 con 0x7f6b281cef20 2024-04-26T00:40:34.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.685+0000 7f6b27fff640 1 -- 172.21.15.37:0/3001632634 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6b281cef20 msgr2=0x7f6b281cf3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:34.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.685+0000 7f6b27fff640 1 --2- 172.21.15.37:0/3001632634 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6b281cef20 0x7f6b281cf3a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6b200ac000 tx=0x7f6b20019b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:34.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.685+0000 7f6b0ffff640 1 -- 172.21.15.37:0/3001632634 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b200b2040 con 0x7f6b281cef20 2024-04-26T00:40:34.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.685+0000 7f6b0ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:34.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.685+0000 7f6b0ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:40:34.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.685+0000 7f6b0ffff640 10 monclient: dump: 2024-04-26T00:40:34.698 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:34.698 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:34.698 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:34.698 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:34.699 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:34.699 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:34.699 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:34.699 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:34.699 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:34.699 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:34.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.685+0000 7f6b0ffff640 10 monclient: _finish_auth 0 2024-04-26T00:40:34.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.685+0000 7f6b0ffff640 10 monclient: _check_auth_tickets 2024-04-26T00:40:34.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.685+0000 7f6b0ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:34.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.685+0000 7f6b0ffff640 1 -- 172.21.15.37:0/3001632634 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b200bd070 con 0x7f6b281cef20 2024-04-26T00:40:34.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.685+0000 7f6b0ffff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:40:34.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.685+0000 7f6b0ffff640 1 -- 172.21.15.37:0/3001632634 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b20009bd0 con 0x7f6b281cef20 2024-04-26T00:40:34.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.685+0000 7f6b2dd00640 5 monclient: authenticate success, global_id 244182 2024-04-26T00:40:34.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.685+0000 7f6b2dd00640 10 monclient: _renew_subs 2024-04-26T00:40:34.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.685+0000 7f6b2dd00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:40:34.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.685+0000 7f6b2dd00640 1 -- 172.21.15.37:0/3001632634 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b2813b510 con 0x7f6b281cef20 2024-04-26T00:40:34.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.685+0000 7f6b0ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:34.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.685+0000 7f6b25ffb640 4 set_mon_vals no callback set 2024-04-26T00:40:34.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.685+0000 7f6b0ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:40:34.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.685+0000 7f6b0ffff640 10 monclient: dump: 2024-04-26T00:40:34.700 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:34.700 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:34.700 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:34.700 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:34.700 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:34.700 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:34.700 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:34.700 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:34.700 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:34.700 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:34.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.685+0000 7f6b2dd00640 10 monclient: _renew_subs 2024-04-26T00:40:34.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.685+0000 7f6b2dd00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:40:34.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.685+0000 7f6b2dd00640 1 -- 172.21.15.37:0/3001632634 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b28137fd0 con 0x7f6b281cef20 2024-04-26T00:40:34.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.685+0000 7f6b2dd00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:34.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.685+0000 7f6b2dd00640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:34.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.685+0000 7f6b2dd00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:40:34.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.685+0000 7f6b0ffff640 1 -- 172.21.15.37:0/3001632634 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f6b200c2460 con 0x7f6b281cef20 2024-04-26T00:40:34.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.685+0000 7f6b2dd00640 1 -- 172.21.15.37:0/3001632634 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b28066f40 con 0x7f6b281cef20 2024-04-26T00:40:34.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.685+0000 7f6b0ffff640 1 --2- 172.21.15.37:0/3001632634 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6b14035e80 0x7f6b14038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:34.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.685+0000 7f6b0ffff640 1 -- 172.21.15.37:0/3001632634 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(816..816 src has 705..816) ==== 3947+0+0 (secure 0 0 0) 0x7f6b200e3630 con 0x7f6b281cef20 2024-04-26T00:40:34.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.685+0000 7f6b277fe640 1 --2- 172.21.15.37:0/3001632634 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6b14035e80 0x7f6b14038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:34.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.685+0000 7f6b277fe640 10 monclient: get_auth_request con 0x7f6b14035e80 auth_method 0 2024-04-26T00:40:34.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.689+0000 7f6b277fe640 1 --2- 172.21.15.37:0/3001632634 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6b14035e80 0x7f6b14038340 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f6b18001aa0 tx=0x7f6b18007040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:34.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.689+0000 7f6b0ffff640 1 -- 172.21.15.37:0/3001632634 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6b2813b510 con 0x7f6b281cef20 2024-04-26T00:40:34.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.689+0000 7f6b0ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:34.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.689+0000 7f6b0ffff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:40:34.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.913+0000 7f6b2dd00640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:40:34.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.913+0000 7f6b2dd00640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:40:34.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.913+0000 7f6b2dd00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:40:34.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.913+0000 7f6b2dd00640 1 -- 172.21.15.37:0/3001632634 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6b28075d70 con 0x7f6b281cef20 2024-04-26T00:40:34.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.917+0000 7f6b0ffff640 1 -- 172.21.15.37:0/3001632634 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7f6b280a83a0 con 0x7f6b281cef20 2024-04-26T00:40:34.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.917+0000 7f6b0ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:40:34.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.917+0000 7f6b0ffff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:40:34.918 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":150,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:40:34.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.917+0000 7f6b2dd00640 1 -- 172.21.15.37:0/3001632634 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6b14035e80 msgr2=0x7f6b14038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:34.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.917+0000 7f6b2dd00640 1 --2- 172.21.15.37:0/3001632634 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6b14035e80 0x7f6b14038340 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f6b18001aa0 tx=0x7f6b18007040 comp rx=0 tx=0).stop 2024-04-26T00:40:34.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.917+0000 7f6b2dd00640 10 monclient: shutdown 2024-04-26T00:40:34.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.917+0000 7f6b2dd00640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:40:34.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.917+0000 7f6b2dd00640 1 -- 172.21.15.37:0/3001632634 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6b281cef20 msgr2=0x7f6b281cf3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:34.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.917+0000 7f6b2dd00640 1 --2- 172.21.15.37:0/3001632634 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6b281cef20 0x7f6b281cf3a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6b200ac000 tx=0x7f6b20019b80 comp rx=0 tx=0).stop 2024-04-26T00:40:34.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.917+0000 7f6b2dd00640 1 -- 172.21.15.37:0/3001632634 shutdown_connections 2024-04-26T00:40:34.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.917+0000 7f6b2dd00640 1 --2- 172.21.15.37:0/3001632634 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6b14035e80 0x7f6b14038340 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:34.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.917+0000 7f6b2dd00640 1 --2- 172.21.15.37:0/3001632634 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6b281cef20 0x7f6b281cf3a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:34.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.917+0000 7f6b2dd00640 1 --2- 172.21.15.37:0/3001632634 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6b2813a5b0 0x7f6b281ce9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:34.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.917+0000 7f6b2dd00640 1 --2- 172.21.15.37:0/3001632634 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b28132f40 0x7f6b28133320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:34.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.917+0000 7f6b2dd00640 1 -- 172.21.15.37:0/3001632634 >> 172.21.15.37:0/3001632634 conn(0x7f6b280a5f00 msgr2=0x7f6b280a7200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:40:34.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.917+0000 7f6b2dd00640 1 -- 172.21.15.37:0/3001632634 shutdown_connections 2024-04-26T00:40:34.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:34.917+0000 7f6b2dd00640 1 -- 172.21.15.37:0/3001632634 wait complete. 2024-04-26T00:40:35.160 INFO:tasks.rados.rados.0.smithi045.stdout:3621: done (1 left) 2024-04-26T00:40:35.160 INFO:tasks.rados.rados.0.smithi045.stdout:3622: done (0 left) 2024-04-26T00:40:35.160 INFO:tasks.rados.rados.0.smithi045.stdout:3623: delete oid 35 current snap is 400 2024-04-26T00:40:35.173 INFO:tasks.rados.rados.0.smithi045.stdout:3623: done (0 left) 2024-04-26T00:40:35.174 INFO:tasks.rados.rados.0.smithi045.stdout:3624: rollback oid 26 current snap is 400 2024-04-26T00:40:35.174 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 26 to 388 2024-04-26T00:40:35.174 INFO:tasks.rados.rados.0.smithi045.stdout:3625: copy_from oid 44 from oid 46 current snap is 400 2024-04-26T00:40:35.174 INFO:tasks.rados.rados.0.smithi045.stdout:3626: read oid 14 snap -1 2024-04-26T00:40:35.174 INFO:tasks.rados.rados.0.smithi045.stdout:3626: expect deleted 2024-04-26T00:40:35.174 INFO:tasks.rados.rados.0.smithi045.stdout:3627: rollback oid 40 current snap is 400 2024-04-26T00:40:35.174 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 40 to 385 2024-04-26T00:40:35.174 INFO:tasks.rados.rados.0.smithi045.stdout:3628: read oid 50 snap -1 2024-04-26T00:40:35.174 INFO:tasks.rados.rados.0.smithi045.stdout:3628: expect (ObjNum 1264 snap 394 seq_num 1264) 2024-04-26T00:40:35.174 INFO:tasks.rados.rados.0.smithi045.stdout:3629: snap_create 2024-04-26T00:40:35.175 INFO:tasks.rados.rados.0.smithi045.stdout:3625: finishing copy_from to smithi04526579-44 2024-04-26T00:40:35.175 INFO:tasks.rados.rados.0.smithi045.stdout:3625: got expected ENOENT (src dne) 2024-04-26T00:40:35.175 INFO:tasks.rados.rados.0.smithi045.stdout:3625: finishing copy_from racing read to smithi04526579-44 2024-04-26T00:40:35.188 INFO:tasks.rados.rados.0.smithi045.stdout:3628: expect (ObjNum 1264 snap 394 seq_num 1264) 2024-04-26T00:40:35.429 INFO:tasks.rados.rados.0.smithi045.stdout:3624: finishing rollback tid 1 to smithi04526579-26 2024-04-26T00:40:35.429 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 2454 (ObjNum 145 snap 0 seq_num 907320672) dirty dne 2024-04-26T00:40:35.429 INFO:tasks.rados.rados.0.smithi045.stdout:3627: finishing rollback tid 0 to smithi04526579-40 2024-04-26T00:40:35.429 INFO:tasks.rados.rados.0.smithi045.stdout:3627: finishing rollback tid 1 to smithi04526579-40 2024-04-26T00:40:35.429 INFO:tasks.rados.rados.0.smithi045.stdout:3627: finishing rollback tid 2 to smithi04526579-40 2024-04-26T00:40:35.429 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 2350 (ObjNum 1178 snap 366 seq_num 1178) dirty exists 2024-04-26T00:40:36.234 INFO:tasks.rados.rados.0.smithi045.stdout:3624: done (5 left) 2024-04-26T00:40:36.234 INFO:tasks.rados.rados.0.smithi045.stdout:3625: done (4 left) 2024-04-26T00:40:36.235 INFO:tasks.rados.rados.0.smithi045.stdout:3626: done (3 left) 2024-04-26T00:40:36.235 INFO:tasks.rados.rados.0.smithi045.stdout:3627: done (2 left) 2024-04-26T00:40:36.235 INFO:tasks.rados.rados.0.smithi045.stdout:3628: done (1 left) 2024-04-26T00:40:36.235 INFO:tasks.rados.rados.0.smithi045.stdout:3629: done (0 left) 2024-04-26T00:40:36.235 INFO:tasks.rados.rados.0.smithi045.stdout:3630: write (excl) oid 23 current snap is 401 2024-04-26T00:40:36.235 INFO:tasks.rados.rados.0.smithi045.stdout:3630: seq_num 1289 ranges {464223=659618,1879407=739785,3248120=416103} 2024-04-26T00:40:36.253 INFO:tasks.rados.rados.0.smithi045.stdout:3630: writing smithi04526579-23 from 464223 to 1123841 tid 1 2024-04-26T00:40:36.278 INFO:tasks.rados.rados.0.smithi045.stdout:3630: writing smithi04526579-23 from 1879407 to 2619192 tid 2 2024-04-26T00:40:36.295 INFO:tasks.rados.rados.0.smithi045.stdout:3630: writing smithi04526579-23 from 3248120 to 3664223 tid 3 2024-04-26T00:40:36.295 INFO:tasks.rados.rados.0.smithi045.stdout:3631: rollback oid 38 current snap is 401 2024-04-26T00:40:36.295 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 38 to 398 2024-04-26T00:40:36.295 INFO:tasks.rados.rados.0.smithi045.stdout:3630: finishing write tid 1 to smithi04526579-23 2024-04-26T00:40:36.295 INFO:tasks.rados.rados.0.smithi045.stdout:3630: finishing write tid 2 to smithi04526579-23 2024-04-26T00:40:36.295 INFO:tasks.rados.rados.0.smithi045.stdout:3632: read oid 5 snap -1 2024-04-26T00:40:36.295 INFO:tasks.rados.rados.0.smithi045.stdout:3632: expect deleted 2024-04-26T00:40:36.295 INFO:tasks.rados.rados.0.smithi045.stdout:3633: write oid 12 current snap is 401 2024-04-26T00:40:36.295 INFO:tasks.rados.rados.0.smithi045.stdout:3633: seq_num 1290 ranges {481030=734867,1937889=483443,2875543=552426} 2024-04-26T00:40:36.309 INFO:tasks.rados.rados.0.smithi045.stdout:3633: writing smithi04526579-12 from 481030 to 1215897 tid 1 2024-04-26T00:40:36.321 INFO:tasks.rados.rados.0.smithi045.stdout:3633: writing smithi04526579-12 from 1937889 to 2421332 tid 2 2024-04-26T00:40:36.329 INFO:tasks.rados.rados.0.smithi045.stdout:3633: writing smithi04526579-12 from 2875543 to 3427969 tid 3 2024-04-26T00:40:36.329 INFO:tasks.rados.rados.0.smithi045.stdout:3634: write oid 22 current snap is 401 2024-04-26T00:40:36.330 INFO:tasks.rados.rados.0.smithi045.stdout:3634: seq_num 1291 ranges {497837=410116,1596371=743454,2902966=763698} 2024-04-26T00:40:36.338 INFO:tasks.rados.rados.0.smithi045.stdout:3634: writing smithi04526579-22 from 497837 to 907953 tid 1 2024-04-26T00:40:36.346 INFO:tasks.rados.rados.0.smithi045.stdout:3634: writing smithi04526579-22 from 1596371 to 2339825 tid 2 2024-04-26T00:40:36.355 INFO:tasks.rados.rados.0.smithi045.stdout:3634: writing smithi04526579-22 from 2902966 to 3666664 tid 3 2024-04-26T00:40:36.355 INFO:tasks.rados.rados.0.smithi045.stdout:3635: delete oid 31 current snap is 401 2024-04-26T00:40:36.355 INFO:tasks.rados.rados.0.smithi045.stdout:3630: finishing write tid 3 to smithi04526579-23 2024-04-26T00:40:36.355 INFO:tasks.rados.rados.0.smithi045.stdout:3630: finishing write tid 4 to smithi04526579-23 2024-04-26T00:40:36.355 INFO:tasks.rados.rados.0.smithi045.stdout:3630: finishing write tid 5 to smithi04526579-23 2024-04-26T00:40:36.355 INFO:tasks.rados.rados.0.smithi045.stdout:3630: oid 23 updating version 0 to 2354 2024-04-26T00:40:36.356 INFO:tasks.rados.rados.0.smithi045.stdout:3630: oid 23 version 2354 is already newer than 2352 2024-04-26T00:40:36.356 INFO:tasks.rados.rados.0.smithi045.stdout:3630: oid 23 version 2354 is already newer than 2353 2024-04-26T00:40:36.356 INFO:tasks.rados.rados.0.smithi045.stdout:3630: oid 23 version 2354 is already newer than 2351 2024-04-26T00:40:36.356 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 2354 (ObjNum 1289 snap 401 seq_num 1289) dirty exists 2024-04-26T00:40:36.356 INFO:tasks.rados.rados.0.smithi045.stdout:3630: left oid 23 (ObjNum 1289 snap 401 seq_num 1289) 2024-04-26T00:40:36.356 INFO:tasks.rados.rados.0.smithi045.stdout:3631: finishing rollback tid 1 to smithi04526579-38 2024-04-26T00:40:36.356 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 2405 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-04-26T00:40:36.356 INFO:tasks.rados.rados.0.smithi045.stdout:3633: finishing write tid 1 to smithi04526579-12 2024-04-26T00:40:36.356 INFO:tasks.rados.rados.0.smithi045.stdout:3633: finishing write tid 2 to smithi04526579-12 2024-04-26T00:40:36.356 INFO:tasks.rados.rados.0.smithi045.stdout:3633: finishing write tid 3 to smithi04526579-12 2024-04-26T00:40:36.356 INFO:tasks.rados.rados.0.smithi045.stdout:3633: finishing write tid 4 to smithi04526579-12 2024-04-26T00:40:36.356 INFO:tasks.rados.rados.0.smithi045.stdout:3633: finishing write tid 5 to smithi04526579-12 2024-04-26T00:40:36.356 INFO:tasks.rados.rados.0.smithi045.stdout:3633: oid 12 updating version 0 to 2356 2024-04-26T00:40:36.356 INFO:tasks.rados.rados.0.smithi045.stdout:3633: oid 12 updating version 2356 to 2358 2024-04-26T00:40:36.356 INFO:tasks.rados.rados.0.smithi045.stdout:3633: oid 12 version 2358 is already newer than 2355 2024-04-26T00:40:36.357 INFO:tasks.rados.rados.0.smithi045.stdout:3633: oid 12 version 2358 is already newer than 2357 2024-04-26T00:40:36.357 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 2358 (ObjNum 1290 snap 401 seq_num 1290) dirty exists 2024-04-26T00:40:36.357 INFO:tasks.rados.rados.0.smithi045.stdout:3633: left oid 12 (ObjNum 1290 snap 401 seq_num 1290) 2024-04-26T00:40:36.357 INFO:tasks.rados.rados.0.smithi045.stdout:3634: finishing write tid 1 to smithi04526579-22 2024-04-26T00:40:36.359 INFO:tasks.rados.rados.0.smithi045.stdout:3634: finishing write tid 2 to smithi04526579-22 2024-04-26T00:40:36.362 INFO:tasks.rados.rados.0.smithi045.stdout:3630: done (5 left) 2024-04-26T00:40:36.362 INFO:tasks.rados.rados.0.smithi045.stdout:3631: done (4 left) 2024-04-26T00:40:36.362 INFO:tasks.rados.rados.0.smithi045.stdout:3632: done (3 left) 2024-04-26T00:40:36.362 INFO:tasks.rados.rados.0.smithi045.stdout:3633: done (2 left) 2024-04-26T00:40:36.363 INFO:tasks.rados.rados.0.smithi045.stdout:3635: done (1 left) 2024-04-26T00:40:36.363 INFO:tasks.rados.rados.0.smithi045.stdout:3636: snap_create 2024-04-26T00:40:36.363 INFO:tasks.rados.rados.0.smithi045.stdout:3634: finishing write tid 3 to smithi04526579-22 2024-04-26T00:40:36.363 INFO:tasks.rados.rados.0.smithi045.stdout:3634: finishing write tid 4 to smithi04526579-22 2024-04-26T00:40:36.363 INFO:tasks.rados.rados.0.smithi045.stdout:3634: finishing write tid 5 to smithi04526579-22 2024-04-26T00:40:36.363 INFO:tasks.rados.rados.0.smithi045.stdout:3634: oid 22 updating version 0 to 2460 2024-04-26T00:40:36.363 INFO:tasks.rados.rados.0.smithi045.stdout:3634: oid 22 version 2460 is already newer than 2458 2024-04-26T00:40:36.363 INFO:tasks.rados.rados.0.smithi045.stdout:3634: oid 22 updating version 2460 to 2461 2024-04-26T00:40:36.363 INFO:tasks.rados.rados.0.smithi045.stdout:3634: oid 22 version 2461 is already newer than 2459 2024-04-26T00:40:36.364 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 2461 (ObjNum 1291 snap 401 seq_num 1291) dirty exists 2024-04-26T00:40:36.364 INFO:tasks.rados.rados.0.smithi045.stdout:3634: left oid 22 (ObjNum 1291 snap 401 seq_num 1291) 2024-04-26T00:40:37.249 INFO:tasks.rados.rados.0.smithi045.stdout:3634: done (1 left) 2024-04-26T00:40:37.249 INFO:tasks.rados.rados.0.smithi045.stdout:3636: done (0 left) 2024-04-26T00:40:37.250 INFO:tasks.rados.rados.0.smithi045.stdout:3637: write (excl) oid 31 current snap is 402 2024-04-26T00:40:37.250 INFO:tasks.rados.rados.0.smithi045.stdout:3637: seq_num 1293 ranges {531451=676967,1713335=747123,2957812=502595} 2024-04-26T00:40:37.256 INFO:tasks.rados.rados.0.smithi045.stdout:3637: writing smithi04526579-31 from 531451 to 1208418 tid 1 2024-04-26T00:40:37.264 INFO:tasks.rados.rados.0.smithi045.stdout:3637: writing smithi04526579-31 from 1713335 to 2460458 tid 2 2024-04-26T00:40:37.270 INFO:tasks.rados.rados.0.smithi045.stdout:3637: writing smithi04526579-31 from 2957812 to 3460407 tid 3 2024-04-26T00:40:37.270 INFO:tasks.rados.rados.0.smithi045.stdout:3638: write (excl) oid 6 current snap is 402 2024-04-26T00:40:37.270 INFO:tasks.rados.rados.0.smithi045.stdout:3638: seq_num 1294 ranges {548258=752216,1771817=490781,2985235=713867} 2024-04-26T00:40:37.277 INFO:tasks.rados.rados.0.smithi045.stdout:3638: writing smithi04526579-6 from 548258 to 1300474 tid 1 2024-04-26T00:40:37.284 INFO:tasks.rados.rados.0.smithi045.stdout:3638: writing smithi04526579-6 from 1771817 to 2262598 tid 2 2024-04-26T00:40:37.293 INFO:tasks.rados.rados.0.smithi045.stdout:3638: writing smithi04526579-6 from 2985235 to 3699102 tid 3 2024-04-26T00:40:37.294 INFO:tasks.rados.rados.0.smithi045.stdout:3639: write (excl) oid 48 current snap is 402 2024-04-26T00:40:37.294 INFO:tasks.rados.rados.0.smithi045.stdout:3639: seq_num 1295 ranges {565065=427465,1430299=634439,2496305=525139,3472104=292961} 2024-04-26T00:40:37.300 INFO:tasks.rados.rados.0.smithi045.stdout:3639: writing smithi04526579-48 from 565065 to 992530 tid 1 2024-04-26T00:40:37.307 INFO:tasks.rados.rados.0.smithi045.stdout:3639: writing smithi04526579-48 from 1430299 to 2064738 tid 2 2024-04-26T00:40:37.313 INFO:tasks.rados.rados.0.smithi045.stdout:3639: writing smithi04526579-48 from 2496305 to 3021444 tid 3 2024-04-26T00:40:37.318 INFO:tasks.rados.rados.0.smithi045.stdout:3639: writing smithi04526579-48 from 3472104 to 3765065 tid 4 2024-04-26T00:40:37.318 INFO:tasks.rados.rados.0.smithi045.stdout:3640: snap_remove snap 388 2024-04-26T00:40:37.318 INFO:tasks.rados.rados.0.smithi045.stdout:3637: finishing write tid 1 to smithi04526579-31 2024-04-26T00:40:37.937 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:40:38.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.065+0000 7fe0ce31d640 10 monclient: get_monmap_and_config 2024-04-26T00:40:38.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.065+0000 7fe0ce31d640 10 monclient: build_initial_monmap 2024-04-26T00:40:38.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.065+0000 7fe0ce31d640 10 monclient: monmap: 2024-04-26T00:40:38.067 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:40:38.067 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:38.067 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:40:38.069789+0000 2024-04-26T00:40:38.067 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:40:38.069789+0000 2024-04-26T00:40:38.067 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:40:38.068 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:38.068 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:40:38.068 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:40:38.068 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:40:38.068 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:40:38.068 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:40:38.068 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:38.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.065+0000 7fe0ce31d640 1 Processor -- start 2024-04-26T00:40:38.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.065+0000 7fe0ce31d640 1 -- start start 2024-04-26T00:40:38.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.065+0000 7fe0ce31d640 10 monclient: init 2024-04-26T00:40:38.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.065+0000 7fe0ce31d640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:38.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.065+0000 7fe0ce31d640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-04-26T00:40:38.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.065+0000 7fe0ce31d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fe0c8136e10 0x7fe0c809c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:38.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.065+0000 7fe0ce31d640 10 monclient(hunting): picked mon.noname-a con 0x7fe0c8136e10 addr v2:172.21.15.37:3300/0 2024-04-26T00:40:38.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.065+0000 7fe0ce31d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe0c81371f0 0x7fe0c8137600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:38.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.065+0000 7fe0ce31d640 10 monclient(hunting): picked mon.noname-c con 0x7fe0c81371f0 addr v2:172.21.15.45:3300/0 2024-04-26T00:40:38.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.065+0000 7fe0ce31d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe0c8135d90 addr v1:172.21.15.45:6789/0 2024-04-26T00:40:38.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.065+0000 7fe0ce31d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0c809b680 con 0x7fe0c8135d90 2024-04-26T00:40:38.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.065+0000 7fe0ce31d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:38.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.065+0000 7fe0ce31d640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fe0c809b8f0 con 0x7fe0c8136e10 2024-04-26T00:40:38.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.065+0000 7fe0ce31d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:38.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.065+0000 7fe0ce31d640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fe0c809bbd0 con 0x7fe0c81371f0 2024-04-26T00:40:38.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.065+0000 7fe0ce31d640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:38.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.065+0000 7fe0ce31d640 10 monclient(hunting): authenticate will time out at 2219.375582s 2024-04-26T00:40:38.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.065+0000 7fe0c7fff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fe0c8136e10 0x7fe0c809c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:38.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.065+0000 7fe0c7fff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fe0c8136e10 0x7fe0c809c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:60916/0 (socket says 172.21.15.37:60916) 2024-04-26T00:40:38.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.065+0000 7fe0c7fff640 1 -- 172.21.15.37:0/2038707483 learned_addr learned my addr 172.21.15.37:0/2038707483 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:40:38.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.065+0000 7fe0c7fff640 10 monclient(hunting): get_auth_request con 0x7fe0c8136e10 auth_method 0 2024-04-26T00:40:38.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.065+0000 7fe0c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:38.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.065+0000 7fe0c7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:38.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.065+0000 7fe0c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:38.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.069+0000 7fe0c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:38.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.069+0000 7fe0c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:38.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.069+0000 7fe0c77fe640 1 --2- 172.21.15.37:0/2038707483 >> v2:172.21.15.45:3300/0 conn(0x7fe0c81371f0 0x7fe0c8137600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:38.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.069+0000 7fe0c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:38.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.069+0000 7fe0c77fe640 10 monclient(hunting): get_auth_request con 0x7fe0c81371f0 auth_method 0 2024-04-26T00:40:38.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.069+0000 7fe0c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:38.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.069+0000 7fe0c77fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:38.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.069+0000 7fe0c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:38.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.069+0000 7fe0c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:38.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.069+0000 7fe0c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:38.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.069+0000 7fe0c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:38.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.069+0000 7fe0c7fff640 10 monclient(hunting): handle_auth_done global_id 244185 payload 274 2024-04-26T00:40:38.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.069+0000 7fe0c7fff640 1 -- 172.21.15.37:0/2038707483 >> v2:172.21.15.45:3300/0 conn(0x7fe0c81371f0 msgr2=0x7fe0c8137600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:40:38.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.069+0000 7fe0c7fff640 1 --2- 172.21.15.37:0/2038707483 >> v2:172.21.15.45:3300/0 conn(0x7fe0c81371f0 0x7fe0c8137600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:38.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.069+0000 7fe0c7fff640 1 -- 172.21.15.37:0/2038707483 >> v1:172.21.15.45:6789/0 conn(0x7fe0c8135d90 legacy=0x7fe0c8136220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:40:38.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.069+0000 7fe0c7fff640 10 monclient: _finish_hunting 0 2024-04-26T00:40:38.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.069+0000 7fe0c7fff640 1 monclient: found mon.noname-a 2024-04-26T00:40:38.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.069+0000 7fe0c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:38.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.069+0000 7fe0c7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:40:38.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.069+0000 7fe0c7fff640 1 -- 172.21.15.37:0/2038707483 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0c8137b40 con 0x7fe0c8136e10 2024-04-26T00:40:38.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.073+0000 7fe0c7fff640 1 -- 172.21.15.37:0/2038707483 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0c8136e10 msgr2=0x7fe0c809c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:38.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.073+0000 7fe0c7fff640 1 --2- 172.21.15.37:0/2038707483 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0c8136e10 0x7fe0c809c5b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe0c8068e60 tx=0x7fe0bc01c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d5f87201fc37326 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:38.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.073+0000 7fe0c6ffd640 1 -- 172.21.15.37:0/2038707483 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0bc0b1070 con 0x7fe0c8136e10 2024-04-26T00:40:38.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.073+0000 7fe0c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:38.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.073+0000 7fe0c6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:40:38.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.073+0000 7fe0c6ffd640 10 monclient: dump: 2024-04-26T00:40:38.077 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:38.077 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:38.077 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.077 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.077 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:38.077 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:38.077 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:38.077 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:38.077 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:38.077 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:38.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.073+0000 7fe0c6ffd640 10 monclient: _finish_auth 0 2024-04-26T00:40:38.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.073+0000 7fe0c6ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:40:38.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.073+0000 7fe0c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:38.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.073+0000 7fe0c6ffd640 1 -- 172.21.15.37:0/2038707483 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0bc0ac620 con 0x7fe0c8136e10 2024-04-26T00:40:38.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.073+0000 7fe0ce31d640 5 monclient: authenticate success, global_id 244185 2024-04-26T00:40:38.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.073+0000 7fe0ce31d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:40:38.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.073+0000 7fe0c6ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:40:38.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.073+0000 7fe0ce31d640 10 monclient: get_monmap_and_config success 2024-04-26T00:40:38.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.073+0000 7fe0ce31d640 4 set_mon_vals no callback set 2024-04-26T00:40:38.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.073+0000 7fe0c6ffd640 1 -- 172.21.15.37:0/2038707483 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0bc0ac9f0 con 0x7fe0c8136e10 2024-04-26T00:40:38.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.073+0000 7fe0c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:38.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.073+0000 7fe0c6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:40:38.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.073+0000 7fe0c6ffd640 10 monclient: dump: 2024-04-26T00:40:38.078 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:38.078 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:38.078 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.078 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.079 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:38.079 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:38.079 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:38.079 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:38.079 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:38.079 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:38.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.073+0000 7fe0ce31d640 10 monclient: shutdown 2024-04-26T00:40:38.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.073+0000 7fe0ce31d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:40:38.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.073+0000 7fe0ce31d640 1 -- 172.21.15.37:0/2038707483 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0c8136e10 msgr2=0x7fe0c809c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:38.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.073+0000 7fe0ce31d640 1 --2- 172.21.15.37:0/2038707483 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0c8136e10 0x7fe0c809c5b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe0c8068e60 tx=0x7fe0bc01c3a0 comp rx=0 tx=0).stop 2024-04-26T00:40:38.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.073+0000 7fe0ce31d640 1 -- 172.21.15.37:0/2038707483 shutdown_connections 2024-04-26T00:40:38.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.073+0000 7fe0ce31d640 1 --2- 172.21.15.37:0/2038707483 >> v2:172.21.15.45:3300/0 conn(0x7fe0c81371f0 0x7fe0c8137600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:38.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.073+0000 7fe0ce31d640 1 --2- 172.21.15.37:0/2038707483 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0c8136e10 0x7fe0c809c5b0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:38.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.073+0000 7fe0ce31d640 1 -- 172.21.15.37:0/2038707483 >> 172.21.15.37:0/2038707483 conn(0x7fe0c80a64e0 msgr2=0x7fe0c80a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:40:38.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.073+0000 7fe0ce31d640 1 -- 172.21.15.37:0/2038707483 shutdown_connections 2024-04-26T00:40:38.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.073+0000 7fe0ce31d640 1 -- 172.21.15.37:0/2038707483 wait complete. 2024-04-26T00:40:38.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.073+0000 7fe0ce31d640 10 monclient: build_initial_monmap 2024-04-26T00:40:38.084 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.073+0000 7fe0ce31d640 10 monclient: monmap: 2024-04-26T00:40:38.085 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:40:38.085 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:40:38.085 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:40:38.085 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:40:38.085 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:40:38.085 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:40:38.085 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:40:38.085 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:40:38.086 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:40:38.086 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:38.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0ce31d640 1 Processor -- start 2024-04-26T00:40:38.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0ce31d640 1 -- start start 2024-04-26T00:40:38.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0ce31d640 10 monclient: init 2024-04-26T00:40:38.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0ce31d640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:38.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0ce31d640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:40:38.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0ce31d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0c8135d90 0x7fe0c813b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:38.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0ce31d640 10 monclient(hunting): picked mon.noname-b con 0x7fe0c8135d90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:40:38.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0ce31d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe0c8136e10 0x7fe0c813d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:38.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0ce31d640 10 monclient(hunting): picked mon.noname-c con 0x7fe0c8136e10 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:40:38.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0ce31d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0c81371f0 0x7fe0c81350f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:38.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0ce31d640 10 monclient(hunting): picked mon.noname-a con 0x7fe0c81371f0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:40:38.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0ce31d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:38.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0ce31d640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c8141f10 con 0x7fe0c81371f0 2024-04-26T00:40:38.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0ce31d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:38.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0ce31d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c8141d90 con 0x7fe0c8135d90 2024-04-26T00:40:38.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0ce31d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:38.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0ce31d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe0c8142090 con 0x7fe0c8136e10 2024-04-26T00:40:38.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0ce31d640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:38.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0ce31d640 10 monclient(hunting): authenticate will time out at 2219.384902s 2024-04-26T00:40:38.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0cc8ba640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0c81371f0 0x7fe0c81350f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:38.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0cc8ba640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0c81371f0 0x7fe0c81350f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:60922/0 (socket says 172.21.15.37:60922) 2024-04-26T00:40:38.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0cc8ba640 1 -- 172.21.15.37:0/2667135354 learned_addr learned my addr 172.21.15.37:0/2667135354 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:40:38.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0cc8ba640 10 monclient(hunting): get_auth_request con 0x7fe0c81371f0 auth_method 0 2024-04-26T00:40:38.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0cc8ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:38.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0cc8ba640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:38.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0cc8ba640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:38.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0c77fe640 1 --2- 172.21.15.37:0/2667135354 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe0c8136e10 0x7fe0c813d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:38.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0c77fe640 10 monclient(hunting): get_auth_request con 0x7fe0c8136e10 auth_method 0 2024-04-26T00:40:38.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:38.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0c77fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:38.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:38.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0cc8ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:38.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0cc8ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:38.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0cc8ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:38.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0c7fff640 1 --2- 172.21.15.37:0/2667135354 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0c8135d90 0x7fe0c813b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:38.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:38.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:38.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:38.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0c7fff640 10 monclient(hunting): get_auth_request con 0x7fe0c8135d90 auth_method 0 2024-04-26T00:40:38.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:38.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0c7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:38.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:38.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0cc8ba640 10 monclient(hunting): handle_auth_done global_id 244188 payload 824 2024-04-26T00:40:38.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0cc8ba640 1 -- 172.21.15.37:0/2667135354 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe0c8136e10 msgr2=0x7fe0c813d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:38.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0cc8ba640 1 --2- 172.21.15.37:0/2667135354 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe0c8136e10 0x7fe0c813d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:38.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0cc8ba640 1 -- 172.21.15.37:0/2667135354 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0c8135d90 msgr2=0x7fe0c813b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:38.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0cc8ba640 1 --2- 172.21.15.37:0/2667135354 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0c8135d90 0x7fe0c813b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:38.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0cc8ba640 10 monclient: _finish_hunting 0 2024-04-26T00:40:38.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0cc8ba640 1 monclient: found mon.noname-a 2024-04-26T00:40:38.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0cc8ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:38.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0cc8ba640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:40:38.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0cc8ba640 1 -- 172.21.15.37:0/2667135354 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0c813ecc0 con 0x7fe0c81371f0 2024-04-26T00:40:38.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0cc8ba640 1 -- 172.21.15.37:0/2667135354 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0c81371f0 msgr2=0x7fe0c81350f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:38.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0cc8ba640 1 --2- 172.21.15.37:0/2667135354 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0c81371f0 0x7fe0c81350f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe0c000c780 tx=0x7fe0c000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:38.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0c4ff9640 1 -- 172.21.15.37:0/2667135354 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0c000ac60 con 0x7fe0c81371f0 2024-04-26T00:40:38.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0c4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:38.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0c4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:40:38.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0c4ff9640 10 monclient: dump: 2024-04-26T00:40:38.091 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:38.091 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:38.091 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.092 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.092 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:38.092 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:38.092 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:38.092 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:38.092 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:38.092 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:38.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0c4ff9640 10 monclient: _finish_auth 0 2024-04-26T00:40:38.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0c4ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:40:38.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0c4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:38.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0ce31d640 5 monclient: authenticate success, global_id 244188 2024-04-26T00:40:38.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0c4ff9640 1 -- 172.21.15.37:0/2667135354 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0c000ae00 con 0x7fe0c81371f0 2024-04-26T00:40:38.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.077+0000 7fe0c4ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:40:38.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.081+0000 7fe0ce31d640 10 monclient: _renew_subs 2024-04-26T00:40:38.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.081+0000 7fe0c4ff9640 1 -- 172.21.15.37:0/2667135354 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0c0004bb0 con 0x7fe0c81371f0 2024-04-26T00:40:38.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.081+0000 7fe0ce31d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:40:38.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.081+0000 7fe0ce31d640 1 -- 172.21.15.37:0/2667135354 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0c80a7de0 con 0x7fe0c81371f0 2024-04-26T00:40:38.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.081+0000 7fe0c4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:38.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.081+0000 7fe0c67fc640 4 set_mon_vals no callback set 2024-04-26T00:40:38.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.081+0000 7fe0c4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:40:38.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.081+0000 7fe0c4ff9640 10 monclient: dump: 2024-04-26T00:40:38.093 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:38.093 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:38.093 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.093 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.093 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:38.093 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:38.093 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:38.093 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:38.094 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:38.094 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:38.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.081+0000 7fe0ce31d640 10 monclient: _renew_subs 2024-04-26T00:40:38.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.081+0000 7fe0ce31d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:40:38.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.081+0000 7fe0ce31d640 1 -- 172.21.15.37:0/2667135354 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0c813c1a0 con 0x7fe0c81371f0 2024-04-26T00:40:38.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.081+0000 7fe0c4ff9640 1 -- 172.21.15.37:0/2667135354 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe0c0005030 con 0x7fe0c81371f0 2024-04-26T00:40:38.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.081+0000 7fe0ce31d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:38.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.081+0000 7fe0ce31d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:38.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.081+0000 7fe0ce31d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:40:38.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.081+0000 7fe0ce31d640 1 -- 172.21.15.37:0/2667135354 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0c8075910 con 0x7fe0c81371f0 2024-04-26T00:40:38.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.081+0000 7fe0c4ff9640 1 --2- 172.21.15.37:0/2667135354 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe0b4035f00 0x7fe0b40383c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:38.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.081+0000 7fe0c4ff9640 1 -- 172.21.15.37:0/2667135354 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(819..819 src has 705..819) ==== 3947+0+0 (secure 0 0 0) 0x7fe0c0048930 con 0x7fe0c81371f0 2024-04-26T00:40:38.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.081+0000 7fe0c7fff640 1 --2- 172.21.15.37:0/2667135354 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe0b4035f00 0x7fe0b40383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:38.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.081+0000 7fe0c7fff640 10 monclient: get_auth_request con 0x7fe0b4035f00 auth_method 0 2024-04-26T00:40:38.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.085+0000 7fe0c7fff640 1 --2- 172.21.15.37:0/2667135354 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe0b4035f00 0x7fe0b40383c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc0ac000 tx=0x7fe0bc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:38.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.085+0000 7fe0c4ff9640 1 -- 172.21.15.37:0/2667135354 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe0c0020020 con 0x7fe0c81371f0 2024-04-26T00:40:38.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.085+0000 7fe0c4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:38.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.085+0000 7fe0c4ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:40:38.268 INFO:tasks.rados.rados.0.smithi045.stdout:3640: done (3 left) 2024-04-26T00:40:38.268 INFO:tasks.rados.rados.0.smithi045.stdout:3641: snap_create 2024-04-26T00:40:38.268 INFO:tasks.rados.rados.0.smithi045.stdout:3637: finishing write tid 2 to smithi04526579-31 2024-04-26T00:40:38.268 INFO:tasks.rados.rados.0.smithi045.stdout:3637: finishing write tid 3 to smithi04526579-31 2024-04-26T00:40:38.268 INFO:tasks.rados.rados.0.smithi045.stdout:3637: finishing write tid 4 to smithi04526579-31 2024-04-26T00:40:38.268 INFO:tasks.rados.rados.0.smithi045.stdout:3637: finishing write tid 5 to smithi04526579-31 2024-04-26T00:40:38.268 INFO:tasks.rados.rados.0.smithi045.stdout:3637: oid 31 updating version 0 to 2525 2024-04-26T00:40:38.268 INFO:tasks.rados.rados.0.smithi045.stdout:3637: oid 31 updating version 2525 to 2527 2024-04-26T00:40:38.268 INFO:tasks.rados.rados.0.smithi045.stdout:3637: oid 31 version 2527 is already newer than 2524 2024-04-26T00:40:38.268 INFO:tasks.rados.rados.0.smithi045.stdout:3637: oid 31 version 2527 is already newer than 2526 2024-04-26T00:40:38.268 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 2527 (ObjNum 1293 snap 402 seq_num 1293) dirty exists 2024-04-26T00:40:38.268 INFO:tasks.rados.rados.0.smithi045.stdout:3637: left oid 31 (ObjNum 1293 snap 402 seq_num 1293) 2024-04-26T00:40:38.268 INFO:tasks.rados.rados.0.smithi045.stdout:3638: finishing write tid 1 to smithi04526579-6 2024-04-26T00:40:38.269 INFO:tasks.rados.rados.0.smithi045.stdout:3638: finishing write tid 2 to smithi04526579-6 2024-04-26T00:40:38.269 INFO:tasks.rados.rados.0.smithi045.stdout:3638: finishing write tid 3 to smithi04526579-6 2024-04-26T00:40:38.269 INFO:tasks.rados.rados.0.smithi045.stdout:3638: finishing write tid 4 to smithi04526579-6 2024-04-26T00:40:38.269 INFO:tasks.rados.rados.0.smithi045.stdout:3638: finishing write tid 5 to smithi04526579-6 2024-04-26T00:40:38.269 INFO:tasks.rados.rados.0.smithi045.stdout:3638: oid 6 updating version 0 to 2457 2024-04-26T00:40:38.269 INFO:tasks.rados.rados.0.smithi045.stdout:3638: oid 6 version 2457 is already newer than 2455 2024-04-26T00:40:38.269 INFO:tasks.rados.rados.0.smithi045.stdout:3638: oid 6 version 2457 is already newer than 2454 2024-04-26T00:40:38.269 INFO:tasks.rados.rados.0.smithi045.stdout:3638: oid 6 version 2457 is already newer than 2456 2024-04-26T00:40:38.269 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 2457 (ObjNum 1294 snap 402 seq_num 1294) dirty exists 2024-04-26T00:40:38.269 INFO:tasks.rados.rados.0.smithi045.stdout:3638: left oid 6 (ObjNum 1294 snap 402 seq_num 1294) 2024-04-26T00:40:38.269 INFO:tasks.rados.rados.0.smithi045.stdout:3639: finishing write tid 1 to smithi04526579-48 2024-04-26T00:40:38.269 INFO:tasks.rados.rados.0.smithi045.stdout:3639: finishing write tid 2 to smithi04526579-48 2024-04-26T00:40:38.269 INFO:tasks.rados.rados.0.smithi045.stdout:3639: finishing write tid 3 to smithi04526579-48 2024-04-26T00:40:38.269 INFO:tasks.rados.rados.0.smithi045.stdout:3639: finishing write tid 4 to smithi04526579-48 2024-04-26T00:40:38.269 INFO:tasks.rados.rados.0.smithi045.stdout:3639: finishing write tid 5 to smithi04526579-48 2024-04-26T00:40:38.269 INFO:tasks.rados.rados.0.smithi045.stdout:3639: finishing write tid 6 to smithi04526579-48 2024-04-26T00:40:38.270 INFO:tasks.rados.rados.0.smithi045.stdout:3639: oid 48 updating version 0 to 2497 2024-04-26T00:40:38.270 INFO:tasks.rados.rados.0.smithi045.stdout:3639: oid 48 updating version 2497 to 2499 2024-04-26T00:40:38.270 INFO:tasks.rados.rados.0.smithi045.stdout:3639: oid 48 updating version 2499 to 2501 2024-04-26T00:40:38.270 INFO:tasks.rados.rados.0.smithi045.stdout:3639: oid 48 version 2501 is already newer than 2498 2024-04-26T00:40:38.270 INFO:tasks.rados.rados.0.smithi045.stdout:3639: oid 48 version 2501 is already newer than 2500 2024-04-26T00:40:38.270 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 2501 (ObjNum 1295 snap 402 seq_num 1295) dirty exists 2024-04-26T00:40:38.270 INFO:tasks.rados.rados.0.smithi045.stdout:3639: left oid 48 (ObjNum 1295 snap 402 seq_num 1295) 2024-04-26T00:40:38.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.321+0000 7fe0ce31d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:40:38.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.321+0000 7fe0ce31d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:40:38.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.321+0000 7fe0ce31d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:40:38.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.321+0000 7fe0ce31d640 1 -- 172.21.15.37:0/2667135354 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe0c8066f40 con 0x7fe0c81371f0 2024-04-26T00:40:38.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.325+0000 7fe0c4ff9640 1 -- 172.21.15.37:0/2667135354 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fe0c0047670 con 0x7fe0c81371f0 2024-04-26T00:40:38.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.325+0000 7fe0c4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:40:38.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.325+0000 7fe0c4ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:40:38.326 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":152,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:40:38.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.325+0000 7fe0ce31d640 1 -- 172.21.15.37:0/2667135354 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe0b4035f00 msgr2=0x7fe0b40383c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:38.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.325+0000 7fe0ce31d640 1 --2- 172.21.15.37:0/2667135354 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe0b4035f00 0x7fe0b40383c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc0ac000 tx=0x7fe0bc008040 comp rx=0 tx=0).stop 2024-04-26T00:40:38.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.325+0000 7fe0ce31d640 10 monclient: shutdown 2024-04-26T00:40:38.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.325+0000 7fe0ce31d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:40:38.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.325+0000 7fe0ce31d640 1 -- 172.21.15.37:0/2667135354 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0c81371f0 msgr2=0x7fe0c81350f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:38.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.325+0000 7fe0ce31d640 1 --2- 172.21.15.37:0/2667135354 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0c81371f0 0x7fe0c81350f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe0c000c780 tx=0x7fe0c000cc40 comp rx=0 tx=0).stop 2024-04-26T00:40:38.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.325+0000 7fe0ce31d640 1 -- 172.21.15.37:0/2667135354 shutdown_connections 2024-04-26T00:40:38.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.325+0000 7fe0ce31d640 1 --2- 172.21.15.37:0/2667135354 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe0b4035f00 0x7fe0b40383c0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:38.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.325+0000 7fe0ce31d640 1 --2- 172.21.15.37:0/2667135354 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0c81371f0 0x7fe0c81350f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:38.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.325+0000 7fe0ce31d640 1 --2- 172.21.15.37:0/2667135354 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe0c8136e10 0x7fe0c813d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:38.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.325+0000 7fe0ce31d640 1 --2- 172.21.15.37:0/2667135354 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0c8135d90 0x7fe0c813b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:38.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.325+0000 7fe0ce31d640 1 -- 172.21.15.37:0/2667135354 >> 172.21.15.37:0/2667135354 conn(0x7fe0c80a64e0 msgr2=0x7fe0c80a4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:40:38.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.325+0000 7fe0ce31d640 1 -- 172.21.15.37:0/2667135354 shutdown_connections 2024-04-26T00:40:38.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.325+0000 7fe0ce31d640 1 -- 172.21.15.37:0/2667135354 wait complete. 2024-04-26T00:40:38.343 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:40:38.344 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:40:38.344 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:40:38.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.461+0000 7f443912c640 10 monclient: get_monmap_and_config 2024-04-26T00:40:38.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.461+0000 7f443912c640 10 monclient: build_initial_monmap 2024-04-26T00:40:38.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.461+0000 7f443912c640 10 monclient: monmap: 2024-04-26T00:40:38.464 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:40:38.464 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:38.464 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:40:38.466743+0000 2024-04-26T00:40:38.464 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:40:38.466743+0000 2024-04-26T00:40:38.464 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:40:38.464 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:38.465 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:40:38.465 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:40:38.465 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:40:38.465 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:40:38.465 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:40:38.465 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:38.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f443912c640 1 Processor -- start 2024-04-26T00:40:38.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f443912c640 1 -- start start 2024-04-26T00:40:38.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f443912c640 10 monclient: init 2024-04-26T00:40:38.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f443912c640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:38.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f443912c640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-04-26T00:40:38.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f443912c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f44340a0300 0x7f443409fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:38.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f443912c640 10 monclient(hunting): picked mon.noname-b con 0x7f44340a0300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:40:38.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f443912c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f44340a06e0 addr v1:172.21.15.37:6789/0 2024-04-26T00:40:38.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f443912c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f44341352b0 0x7f443413cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:38.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f443912c640 10 monclient(hunting): picked mon.noname-a con 0x7f44341352b0 addr v2:172.21.15.37:3300/0 2024-04-26T00:40:38.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f443912c640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f443409c190 con 0x7f44340a06e0 2024-04-26T00:40:38.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f443912c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:38.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f443912c640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f44340a13f0 con 0x7f44341352b0 2024-04-26T00:40:38.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f443912c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:38.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f443912c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f44340a1110 con 0x7f44340a0300 2024-04-26T00:40:38.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f443912c640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:38.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f443912c640 10 monclient(hunting): authenticate will time out at 2219.772854s 2024-04-26T00:40:38.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f4432d9d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f44340a0300 0x7f443409fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:38.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f443359e640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f44341352b0 0x7f443413cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:38.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f442bfff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f44340a06e0 0x7f4434134ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:36266/0 (socket says 172.21.15.37:36266) 2024-04-26T00:40:38.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f442bfff640 1 -- 172.21.15.37:0/3839616806 learned_addr learned my addr 172.21.15.37:0/3839616806 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:40:38.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f4432d9d640 10 monclient(hunting): get_auth_request con 0x7f44340a0300 auth_method 0 2024-04-26T00:40:38.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f443359e640 1 --2- 172.21.15.37:0/3839616806 >> v2:172.21.15.37:3300/0 conn(0x7f44341352b0 0x7f443413cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:60928/0 (socket says 172.21.15.37:60928) 2024-04-26T00:40:38.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f4432d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:38.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f4432d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:38.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f4432d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:38.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f443359e640 10 monclient(hunting): get_auth_request con 0x7f44341352b0 auth_method 0 2024-04-26T00:40:38.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f443359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:38.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f443359e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:38.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f443359e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:38.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f4432d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:38.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f4432d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:38.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f4432d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:38.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f443359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:38.469 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f443359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:38.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f443359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:38.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f4432d9d640 10 monclient(hunting): handle_auth_done global_id 254102 payload 274 2024-04-26T00:40:38.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f4432d9d640 1 -- 172.21.15.37:0/3839616806 >> v2:172.21.15.37:3300/0 conn(0x7f44341352b0 msgr2=0x7f443413cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:40:38.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f4432d9d640 1 --2- 172.21.15.37:0/3839616806 >> v2:172.21.15.37:3300/0 conn(0x7f44341352b0 0x7f443413cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:38.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f4432d9d640 1 -- 172.21.15.37:0/3839616806 >> v1:172.21.15.37:6789/0 conn(0x7f44340a06e0 legacy=0x7f4434134ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:40:38.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f4432d9d640 10 monclient: _finish_hunting 0 2024-04-26T00:40:38.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f4432d9d640 1 monclient: found mon.noname-b 2024-04-26T00:40:38.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f4432d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:38.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f4432d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:40:38.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f4432d9d640 1 -- 172.21.15.37:0/3839616806 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44340a16d0 con 0x7f44340a0300 2024-04-26T00:40:38.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.465+0000 7f443359e640 1 --2- 172.21.15.37:0/3839616806 >> v2:172.21.15.37:3300/0 conn(0x7f44341352b0 0x7f443413cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:40:38.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f4432d9d640 1 -- 172.21.15.37:0/3839616806 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f44340a0300 msgr2=0x7f443409fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:38.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f4432d9d640 1 --2- 172.21.15.37:0/3839616806 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f44340a0300 0x7f443409fc50 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f44240097c0 tx=0x7f442401c320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=825a55d8680ca9b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:38.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443259c640 1 -- 172.21.15.37:0/3839616806 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44240b0070 con 0x7f44340a0300 2024-04-26T00:40:38.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443259c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:38.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443259c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:40:38.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443259c640 10 monclient: dump: 2024-04-26T00:40:38.474 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:38.475 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:38.475 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.475 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.475 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:38.475 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:38.475 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:38.475 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:38.475 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:38.475 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:38.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443259c640 10 monclient: _finish_auth 0 2024-04-26T00:40:38.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443259c640 10 monclient: _check_auth_tickets 2024-04-26T00:40:38.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:38.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443259c640 1 -- 172.21.15.37:0/3839616806 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44240ab600 con 0x7f44340a0300 2024-04-26T00:40:38.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443912c640 5 monclient: authenticate success, global_id 254102 2024-04-26T00:40:38.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443912c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:40:38.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443259c640 10 monclient: handle_config config(0 keys) 2024-04-26T00:40:38.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443259c640 1 -- 172.21.15.37:0/3839616806 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44240ab9d0 con 0x7f44340a0300 2024-04-26T00:40:38.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443912c640 10 monclient: get_monmap_and_config success 2024-04-26T00:40:38.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443912c640 4 set_mon_vals no callback set 2024-04-26T00:40:38.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443259c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:38.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443259c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:40:38.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443259c640 10 monclient: dump: 2024-04-26T00:40:38.476 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:38.476 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:38.476 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.476 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.476 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:38.476 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:38.476 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:38.477 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:38.477 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:38.477 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:38.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443912c640 10 monclient: shutdown 2024-04-26T00:40:38.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443912c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:40:38.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443912c640 1 -- 172.21.15.37:0/3839616806 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f44340a0300 msgr2=0x7f443409fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:38.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443912c640 1 --2- 172.21.15.37:0/3839616806 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f44340a0300 0x7f443409fc50 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f44240097c0 tx=0x7f442401c320 comp rx=0 tx=0).stop 2024-04-26T00:40:38.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443912c640 1 -- 172.21.15.37:0/3839616806 shutdown_connections 2024-04-26T00:40:38.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443912c640 1 --2- 172.21.15.37:0/3839616806 >> v2:172.21.15.37:3300/0 conn(0x7f44341352b0 0x7f443413cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:38.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443912c640 1 --2- 172.21.15.37:0/3839616806 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f44340a0300 0x7f443409fc50 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:38.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443912c640 1 -- 172.21.15.37:0/3839616806 >> 172.21.15.37:0/3839616806 conn(0x7f44340a1d50 msgr2=0x7f44340a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:40:38.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443912c640 1 -- 172.21.15.37:0/3839616806 shutdown_connections 2024-04-26T00:40:38.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443912c640 1 -- 172.21.15.37:0/3839616806 wait complete. 2024-04-26T00:40:38.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443912c640 10 monclient: build_initial_monmap 2024-04-26T00:40:38.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443912c640 10 monclient: monmap: 2024-04-26T00:40:38.479 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:40:38.479 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:40:38.479 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:40:38.479 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:40:38.479 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:40:38.479 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:40:38.479 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:40:38.479 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:40:38.479 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:40:38.479 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:38.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443912c640 1 Processor -- start 2024-04-26T00:40:38.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443912c640 1 -- start start 2024-04-26T00:40:38.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443912c640 10 monclient: init 2024-04-26T00:40:38.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443912c640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:38.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443912c640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:40:38.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443912c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f44340a0300 0x7f44341cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:38.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443912c640 10 monclient(hunting): picked mon.noname-a con 0x7f44340a0300 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:40:38.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443912c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44340a06e0 0x7f44341cf910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:38.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443912c640 10 monclient(hunting): picked mon.noname-b con 0x7f44340a06e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:40:38.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443912c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f44341352b0 0x7f44341cfe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:38.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443912c640 10 monclient(hunting): picked mon.noname-c con 0x7f44341352b0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:40:38.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443912c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:38.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443912c640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f443413fc40 con 0x7f44340a0300 2024-04-26T00:40:38.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443912c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:38.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443912c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f443413fdc0 con 0x7f44340a06e0 2024-04-26T00:40:38.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443912c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:38.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443912c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f443413ff40 con 0x7f44341352b0 2024-04-26T00:40:38.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443912c640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:38.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443912c640 10 monclient(hunting): authenticate will time out at 2219.779866s 2024-04-26T00:40:38.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f4432d9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f44340a0300 0x7f44341cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:38.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.469+0000 7f443359e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f44341352b0 0x7f44341cfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:38.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f443359e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f44341352b0 0x7f44341cfe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:52184/0 (socket says 172.21.15.37:52184) 2024-04-26T00:40:38.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f4432d9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f44340a0300 0x7f44341cf3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:60930/0 (socket says 172.21.15.37:60930) 2024-04-26T00:40:38.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f443359e640 1 -- 172.21.15.37:0/3727667672 learned_addr learned my addr 172.21.15.37:0/3727667672 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:40:38.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f443359e640 10 monclient(hunting): get_auth_request con 0x7f44341352b0 auth_method 0 2024-04-26T00:40:38.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f443359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:38.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f443359e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:38.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f443359e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:38.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f4432d9d640 10 monclient(hunting): get_auth_request con 0x7f44340a0300 auth_method 0 2024-04-26T00:40:38.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f4432d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:38.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f4432d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:38.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f4432d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:38.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f4432d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:38.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f4432d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:38.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f4432d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:38.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f443359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:38.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f443359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:38.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f443359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:38.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f4432d9d640 10 monclient(hunting): handle_auth_done global_id 244194 payload 824 2024-04-26T00:40:38.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f442bfff640 1 --2- 172.21.15.37:0/3727667672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44340a06e0 0x7f44341cf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:38.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f4432d9d640 1 -- 172.21.15.37:0/3727667672 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f44341352b0 msgr2=0x7f44341cfe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:38.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f4432d9d640 1 --2- 172.21.15.37:0/3727667672 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f44341352b0 0x7f44341cfe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:38.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f4432d9d640 1 -- 172.21.15.37:0/3727667672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44340a06e0 msgr2=0x7f44341cf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:38.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f4432d9d640 1 --2- 172.21.15.37:0/3727667672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44340a06e0 0x7f44341cf910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:38.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f4432d9d640 10 monclient: _finish_hunting 0 2024-04-26T00:40:38.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f4432d9d640 1 monclient: found mon.noname-a 2024-04-26T00:40:38.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f4432d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:38.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f4432d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:40:38.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f4432d9d640 1 -- 172.21.15.37:0/3727667672 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f443413b8d0 con 0x7f44340a0300 2024-04-26T00:40:38.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f442bfff640 10 monclient: get_auth_request con 0x7f44340a06e0 auth_method 0 2024-04-26T00:40:38.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f442bfff640 1 --2- 172.21.15.37:0/3727667672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44340a06e0 0x7f44341cf910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:40:38.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f4432d9d640 1 -- 172.21.15.37:0/3727667672 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f44340a0300 msgr2=0x7f44341cf3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:38.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f4432d9d640 1 --2- 172.21.15.37:0/3727667672 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f44340a0300 0x7f44341cf3d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f44240ab000 tx=0x7f442401ab60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:38.493 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f442b7fe640 1 -- 172.21.15.37:0/3727667672 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44240b0050 con 0x7f44340a0300 2024-04-26T00:40:38.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f442b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:38.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f442b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:40:38.494 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f442b7fe640 10 monclient: dump: 2024-04-26T00:40:38.494 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:38.494 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:38.494 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.494 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.494 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:38.494 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:38.494 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:38.494 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:38.494 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:38.494 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:38.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f442b7fe640 10 monclient: _finish_auth 0 2024-04-26T00:40:38.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f442b7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:40:38.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f442b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:38.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f443912c640 5 monclient: authenticate success, global_id 244194 2024-04-26T00:40:38.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f443912c640 10 monclient: _renew_subs 2024-04-26T00:40:38.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f442b7fe640 1 -- 172.21.15.37:0/3727667672 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44240bc070 con 0x7f44340a0300 2024-04-26T00:40:38.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f443912c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:40:38.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f443912c640 1 -- 172.21.15.37:0/3727667672 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f443413bbb0 con 0x7f44340a0300 2024-04-26T00:40:38.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f442b7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:40:38.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f442b7fe640 1 -- 172.21.15.37:0/3727667672 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44240b5a60 con 0x7f44340a0300 2024-04-26T00:40:38.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f442b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:38.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f442b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:40:38.495 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f442b7fe640 10 monclient: dump: 2024-04-26T00:40:38.495 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:38.495 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:38.496 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.496 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.496 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:38.496 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:38.496 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:38.496 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:38.496 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:38.496 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:38.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f4431d9b640 4 set_mon_vals no callback set 2024-04-26T00:40:38.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f443912c640 10 monclient: _renew_subs 2024-04-26T00:40:38.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f443912c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:40:38.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f443912c640 1 -- 172.21.15.37:0/3727667672 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f443409c750 con 0x7f44340a0300 2024-04-26T00:40:38.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f443912c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:38.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f443912c640 1 --2- 172.21.15.37:0/3727667672 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f443413b370 0x7f443412d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:38.496 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f443912c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:38.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f443912c640 1 -- 172.21.15.37:0/3727667672 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f44340a0cd0 con 0x7f443413b370 2024-04-26T00:40:38.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f442bfff640 1 --2- 172.21.15.37:0/3727667672 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f443413b370 0x7f443412d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:38.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f442bfff640 10 monclient: get_auth_request con 0x7f443413b370 auth_method 0 2024-04-26T00:40:38.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f442bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:38.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f442bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:38.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f442bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:38.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f442bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:38.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f442bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:38.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f442bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:38.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f442b7fe640 1 -- 172.21.15.37:0/3727667672 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f44240b5c40 con 0x7f44340a0300 2024-04-26T00:40:38.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f442bfff640 10 monclient(hunting): handle_auth_done global_id 244197 payload 824 2024-04-26T00:40:38.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f442bfff640 1 -- 172.21.15.37:0/3727667672 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f443404f510 con 0x7f443413b370 2024-04-26T00:40:38.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.473+0000 7f442b7fe640 1 --2- 172.21.15.37:0/3727667672 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4414035e80 0x7f4414038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:38.497 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.477+0000 7f442b7fe640 1 -- 172.21.15.37:0/3727667672 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(820..820 src has 705..820) ==== 3991+0+0 (secure 0 0 0) 0x7f44240e3130 con 0x7f44340a0300 2024-04-26T00:40:38.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.477+0000 7f443359e640 1 --2- 172.21.15.37:0/3727667672 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4414035e80 0x7f4414038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:38.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.477+0000 7f443359e640 10 monclient: get_auth_request con 0x7f4414035e80 auth_method 0 2024-04-26T00:40:38.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.477+0000 7f442bfff640 1 -- 172.21.15.37:0/3727667672 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f443413b370 msgr2=0x7f443412d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:38.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.477+0000 7f442bfff640 1 --2- 172.21.15.37:0/3727667672 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f443413b370 0x7f443412d9c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f443413ce80 tx=0x7f441c00d950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:38.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.477+0000 7f442b7fe640 1 -- 172.21.15.37:0/3727667672 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441c007070 con 0x7f443413b370 2024-04-26T00:40:38.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.477+0000 7f442b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:38.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.477+0000 7f442b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:40:38.498 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.477+0000 7f442b7fe640 10 monclient: dump: 2024-04-26T00:40:38.498 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:38.498 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:38.498 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.498 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.498 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:38.498 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:38.498 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:38.499 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:38.499 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:38.499 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:38.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.477+0000 7f442b7fe640 1 -- 172.21.15.37:0/3727667672 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f441c017070 con 0x7f443413b370 2024-04-26T00:40:38.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.477+0000 7f442b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:38.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.477+0000 7f442b7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:40:38.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.477+0000 7f442b7fe640 1 -- 172.21.15.37:0/3727667672 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f443413b370 msgr2=0x7f443412d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:38.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.477+0000 7f442b7fe640 1 --2- 172.21.15.37:0/3727667672 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f443413b370 0x7f443412d9c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f443413ce80 tx=0x7f441c00d950 comp rx=0 tx=0).stop 2024-04-26T00:40:38.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.477+0000 7f442b7fe640 1 -- 172.21.15.37:0/3727667672 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f443413b370 msgr2=0x7f443412d9c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:40:38.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.477+0000 7f442b7fe640 1 --2- 172.21.15.37:0/3727667672 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f443413b370 0x7f443412d9c0 secure :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f443413ce80 tx=0x7f441c00d950 comp rx=0 tx=0).stop 2024-04-26T00:40:38.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.477+0000 7f443359e640 1 --2- 172.21.15.37:0/3727667672 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4414035e80 0x7f4414038340 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f442c009bd0 tx=0x7f442c00b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:38.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.485+0000 7f443912c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:40:38.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.485+0000 7f443912c640 1 --2- 172.21.15.37:0/3727667672 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f443412df00 0x7f443412e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:38.499 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.485+0000 7f443912c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:38.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.485+0000 7f443912c640 1 -- 172.21.15.37:0/3727667672 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f443412e850 con 0x7f443412df00 2024-04-26T00:40:38.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.485+0000 7f442bfff640 1 --2- 172.21.15.37:0/3727667672 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f443412df00 0x7f443412e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:38.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.485+0000 7f442bfff640 10 monclient: get_auth_request con 0x7f443412df00 auth_method 0 2024-04-26T00:40:38.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.485+0000 7f442bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:38.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.485+0000 7f442bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:38.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.485+0000 7f442bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:38.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.485+0000 7f442bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:38.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.485+0000 7f442bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:38.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.485+0000 7f442bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:38.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.485+0000 7f442bfff640 10 monclient(hunting): handle_auth_done global_id 244200 payload 824 2024-04-26T00:40:38.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.485+0000 7f442bfff640 1 -- 172.21.15.37:0/3727667672 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f443409eb60 con 0x7f443412df00 2024-04-26T00:40:38.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.485+0000 7f442bfff640 1 -- 172.21.15.37:0/3727667672 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f443412df00 msgr2=0x7f443412e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:38.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.485+0000 7f442bfff640 1 --2- 172.21.15.37:0/3727667672 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f443412df00 0x7f443412e310 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f443413ce80 tx=0x7f441c004950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:38.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.485+0000 7f442b7fe640 1 -- 172.21.15.37:0/3727667672 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441c007710 con 0x7f443412df00 2024-04-26T00:40:38.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.485+0000 7f442b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:38.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.485+0000 7f442b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:40:38.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.485+0000 7f442b7fe640 10 monclient: dump: 2024-04-26T00:40:38.501 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:38.501 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:38.501 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.501 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.501 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:38.501 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:38.501 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:38.501 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:38.501 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:38.502 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:38.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.485+0000 7f442b7fe640 1 -- 172.21.15.37:0/3727667672 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4719 (secure 0 0 0) 0x7f441c007710 con 0x7f443412df00 2024-04-26T00:40:38.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.485+0000 7f442b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:40:38.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.485+0000 7f442b7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:40:38.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.485+0000 7f442b7fe640 1 -- 172.21.15.37:0/3727667672 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f443412df00 msgr2=0x7f443412e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:38.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.485+0000 7f442b7fe640 1 --2- 172.21.15.37:0/3727667672 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f443412df00 0x7f443412e310 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f443413ce80 tx=0x7f441c004950 comp rx=0 tx=0).stop 2024-04-26T00:40:38.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.485+0000 7f442b7fe640 1 -- 172.21.15.37:0/3727667672 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f443412df00 msgr2=0x7f443412e310 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:40:38.502 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.485+0000 7f442b7fe640 1 --2- 172.21.15.37:0/3727667672 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f443412df00 0x7f443412e310 secure :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f443413ce80 tx=0x7f441c004950 comp rx=0 tx=0).stop 2024-04-26T00:40:38.502 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:40:38.502 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:40:38.503 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:40:38.503 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:40:38.503 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 152, 2024-04-26T00:40:38.503 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 55843, 2024-04-26T00:40:38.503 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:40:38.503 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:40:38.503 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:40:38.503 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:40:38.503 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:38.503 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 2, 2024-04-26T00:40:38.503 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:40:38.503 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:40:38.503 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:40:38.503 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:40:38.503 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:40:38.504 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:40:38.504 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:40:38.504 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:40:38.504 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:40:38.504 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:40:38.504 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:40:38.504 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:40:38.504 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:40:38.504 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:40:38.504 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:38.504 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:40:38.504 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:40:38.504 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:40:38.504 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:40:38.504 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:40:38.505 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:40:38.505 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:40:38.505 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:40:38.505 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:40:38.505 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:40:38.505 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:40:38.505 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:40:38.505 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:40:38.505 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:38.506 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:38.506 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:40:38.506 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:40:38.506 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:40:38.506 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:40:38.507 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:40:38.507 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:40:38.507 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:40:38.507 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:40:38.507 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:40:38.507 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:40:38.507 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:40:38.507 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:40:38.507 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:40:38.507 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:40:38.507 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:40:38.508 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:40:38.508 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:40:38.508 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:40:38.508 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:40:38.508 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:40:38.508 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:40:38.508 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:40:38.508 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:40:38.508 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:40:38.508 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:40:38.508 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:40:38.509 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:40:38.509 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:40:38.509 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:38.509 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:40:38.509 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:38.509 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:40:38.509 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:38.509 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:40:38.509 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:40:38.509 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:40:38.509 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:40:38.509 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:38.510 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:40:38.510 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:40:38.510 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:38.510 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:38.510 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:38.510 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:40:38.510 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:40:38.510 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:38.510 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:38.510 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:38.510 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:38.511 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:40:38.511 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:40:38.511 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:40:38.511 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:40:38.511 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:40:38.511 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:38.511 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:38.511 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:40:38.511 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:40:38.511 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:40:38.511 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:40:38.512 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:38.512 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:40:38.512 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:40:38.512 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:38.512 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:38.512 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:38.512 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:40:38.512 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:40:38.512 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:38.512 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:38.512 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:38.512 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:38.513 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:40:38.513 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:40:38.513 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:40:38.513 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:40:38.513 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:40:38.513 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:38.513 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:38.513 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:40:38.513 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:40:38.513 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:40:38.513 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:40:38.513 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:38.514 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:40:38.514 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:40:38.514 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:38.514 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:38.514 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:38.514 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:40:38.514 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:40:38.514 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:38.514 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:38.514 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:38.514 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:38.515 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:40:38.515 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:40:38.515 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:40:38.515 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:40:38.515 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:40:38.515 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:38.515 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:38.515 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:38.515 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:40:38.515 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:40:38.515 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:38.515 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:40:38.515 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:40:38.515 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:40:38.516 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:38.516 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:38.516 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:40:38.516 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:38.516 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:40:38.516 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:40:38.516 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:40:38.516 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:38.516 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:38.516 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:40:38.516 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:38.516 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:40:38.516 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:40:38.516 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:40:38.517 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:38.517 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:38.517 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:38.517 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:40:38.517 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:40:38.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.485+0000 7f443912c640 1 -- 172.21.15.37:0/3727667672 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4414035e80 msgr2=0x7f4414038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:38.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.485+0000 7f443912c640 1 --2- 172.21.15.37:0/3727667672 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4414035e80 0x7f4414038340 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f442c009bd0 tx=0x7f442c00b040 comp rx=0 tx=0).stop 2024-04-26T00:40:38.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.485+0000 7f443912c640 10 monclient: shutdown 2024-04-26T00:40:38.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.485+0000 7f443912c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:40:38.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.485+0000 7f443912c640 1 -- 172.21.15.37:0/3727667672 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f44340a0300 msgr2=0x7f44341cf3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:38.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.485+0000 7f443912c640 1 --2- 172.21.15.37:0/3727667672 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f44340a0300 0x7f44341cf3d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f44240ab000 tx=0x7f442401ab60 comp rx=0 tx=0).stop 2024-04-26T00:40:38.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.485+0000 7f443359e640 1 -- 172.21.15.37:0/3727667672 reap_dead start 2024-04-26T00:40:38.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.485+0000 7f443359e640 1 -- 172.21.15.37:0/3727667672 reap_dead start 2024-04-26T00:40:38.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.485+0000 7f443912c640 1 -- 172.21.15.37:0/3727667672 shutdown_connections 2024-04-26T00:40:38.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.485+0000 7f443912c640 1 -- 172.21.15.37:0/3727667672 >> 172.21.15.37:0/3727667672 conn(0x7f44340a1d50 msgr2=0x7f4434071130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:40:38.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.485+0000 7f443912c640 1 -- 172.21.15.37:0/3727667672 shutdown_connections 2024-04-26T00:40:38.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.485+0000 7f443912c640 1 -- 172.21.15.37:0/3727667672 wait complete. 2024-04-26T00:40:38.520 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:40:38.520 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:40:38.520 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:40:38.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484f3dd640 10 monclient: get_monmap_and_config 2024-04-26T00:40:38.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484f3dd640 10 monclient: build_initial_monmap 2024-04-26T00:40:38.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484f3dd640 10 monclient: monmap: 2024-04-26T00:40:38.627 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:40:38.627 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:38.628 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:40:38.629935+0000 2024-04-26T00:40:38.628 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:40:38.629935+0000 2024-04-26T00:40:38.628 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:40:38.628 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:38.628 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:40:38.628 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:40:38.628 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:40:38.628 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:40:38.629 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:40:38.629 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:38.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484f3dd640 1 Processor -- start 2024-04-26T00:40:38.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484f3dd640 1 -- start start 2024-04-26T00:40:38.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484f3dd640 10 monclient: init 2024-04-26T00:40:38.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484f3dd640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:38.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484f3dd640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-04-26T00:40:38.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484f3dd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f484809e220 0x7f484809fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:38.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484f3dd640 10 monclient(hunting): picked mon.noname-b con 0x7f484809e220 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:40:38.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484f3dd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f48480a14f0 addr v1:172.21.15.45:6789/0 2024-04-26T00:40:38.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484f3dd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f484809e600 0x7f484813aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:38.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484f3dd640 10 monclient(hunting): picked mon.noname-a con 0x7f484809e600 addr v2:172.21.15.37:3300/0 2024-04-26T00:40:38.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484f3dd640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48480a12b0 con 0x7f48480a14f0 2024-04-26T00:40:38.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484f3dd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:38.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484f3dd640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f484809cd10 con 0x7f484809e600 2024-04-26T00:40:38.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484f3dd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:38.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484f3dd640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f484809da20 con 0x7f484809e220 2024-04-26T00:40:38.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484f3dd640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:38.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484f3dd640 10 monclient(hunting): authenticate will time out at 2219.935523s 2024-04-26T00:40:38.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484d179640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f484809e220 0x7f484809fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:38.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484d179640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f484809e220 0x7f484809fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:52194/0 (socket says 172.21.15.37:52194) 2024-04-26T00:40:38.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484d179640 1 -- 172.21.15.37:0/116955106 learned_addr learned my addr 172.21.15.37:0/116955106 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:40:38.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484d179640 10 monclient(hunting): get_auth_request con 0x7f484809e220 auth_method 0 2024-04-26T00:40:38.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484d179640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:38.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484d179640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:38.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484d179640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:38.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484d97a640 1 --2- 172.21.15.37:0/116955106 >> v2:172.21.15.37:3300/0 conn(0x7f484809e600 0x7f484813aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:38.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484d97a640 10 monclient(hunting): get_auth_request con 0x7f484809e600 auth_method 0 2024-04-26T00:40:38.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484d97a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:38.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484d97a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:38.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484d97a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:38.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484d179640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:38.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484d179640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:38.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484d179640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:38.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484d97a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:38.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484d97a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:38.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484d97a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:38.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484d179640 10 monclient(hunting): handle_auth_done global_id 254108 payload 274 2024-04-26T00:40:38.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484d179640 1 -- 172.21.15.37:0/116955106 >> v2:172.21.15.37:3300/0 conn(0x7f484809e600 msgr2=0x7f484813aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:40:38.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484d179640 1 --2- 172.21.15.37:0/116955106 >> v2:172.21.15.37:3300/0 conn(0x7f484809e600 0x7f484813aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:38.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484d179640 1 -- 172.21.15.37:0/116955106 >> v1:172.21.15.45:6789/0 conn(0x7f48480a14f0 legacy=0x7f48480a0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:40:38.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484d179640 10 monclient: _finish_hunting 0 2024-04-26T00:40:38.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484d179640 1 monclient: found mon.noname-b 2024-04-26T00:40:38.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.625+0000 7f484d179640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:38.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.629+0000 7f484d179640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:40:38.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.629+0000 7f484d179640 1 -- 172.21.15.37:0/116955106 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f484809dd00 con 0x7f484809e220 2024-04-26T00:40:38.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.629+0000 7f484d97a640 1 --2- 172.21.15.37:0/116955106 >> v2:172.21.15.37:3300/0 conn(0x7f484809e600 0x7f484813aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:40:38.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484d179640 1 -- 172.21.15.37:0/116955106 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f484809e220 msgr2=0x7f484809fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:38.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484d179640 1 --2- 172.21.15.37:0/116955106 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f484809e220 0x7f484809fc50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4844009870 tx=0x7f484401c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee0bc2f6542e4511 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:38.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f483ffff640 1 -- 172.21.15.37:0/116955106 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48440b0070 con 0x7f484809e220 2024-04-26T00:40:38.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f483ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:38.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f483ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:40:38.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f483ffff640 10 monclient: dump: 2024-04-26T00:40:38.645 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:38.645 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:38.645 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.645 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.645 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:38.645 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:38.645 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:38.645 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:38.645 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:38.646 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:38.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f483ffff640 10 monclient: _finish_auth 0 2024-04-26T00:40:38.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f483ffff640 10 monclient: _check_auth_tickets 2024-04-26T00:40:38.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f483ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:38.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484f3dd640 5 monclient: authenticate success, global_id 254108 2024-04-26T00:40:38.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f483ffff640 1 -- 172.21.15.37:0/116955106 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48480a0ab0 con 0x7f484809e220 2024-04-26T00:40:38.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f483ffff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:40:38.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484f3dd640 10 monclient: get_monmap_and_config success 2024-04-26T00:40:38.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484f3dd640 4 set_mon_vals no callback set 2024-04-26T00:40:38.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f483ffff640 1 -- 172.21.15.37:0/116955106 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48440ab820 con 0x7f484809e220 2024-04-26T00:40:38.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f483ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:38.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f483ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:40:38.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f483ffff640 10 monclient: dump: 2024-04-26T00:40:38.646 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:38.647 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:38.647 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.647 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.647 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:38.647 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:38.647 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:38.647 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:38.647 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:38.647 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:38.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484f3dd640 10 monclient: shutdown 2024-04-26T00:40:38.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484f3dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:40:38.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484f3dd640 1 -- 172.21.15.37:0/116955106 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f484809e220 msgr2=0x7f484809fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:38.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484f3dd640 1 --2- 172.21.15.37:0/116955106 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f484809e220 0x7f484809fc50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4844009870 tx=0x7f484401c3e0 comp rx=0 tx=0).stop 2024-04-26T00:40:38.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484f3dd640 1 -- 172.21.15.37:0/116955106 shutdown_connections 2024-04-26T00:40:38.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484f3dd640 1 --2- 172.21.15.37:0/116955106 >> v2:172.21.15.37:3300/0 conn(0x7f484809e600 0x7f484813aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:38.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484f3dd640 1 --2- 172.21.15.37:0/116955106 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f484809e220 0x7f484809fc50 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:38.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484f3dd640 1 -- 172.21.15.37:0/116955106 >> 172.21.15.37:0/116955106 conn(0x7f48480a1d50 msgr2=0x7f48480a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:40:38.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484f3dd640 1 -- 172.21.15.37:0/116955106 shutdown_connections 2024-04-26T00:40:38.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484f3dd640 1 -- 172.21.15.37:0/116955106 wait complete. 2024-04-26T00:40:38.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484f3dd640 10 monclient: build_initial_monmap 2024-04-26T00:40:38.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484f3dd640 10 monclient: monmap: 2024-04-26T00:40:38.648 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:40:38.648 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:40:38.648 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:40:38.648 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:40:38.648 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:40:38.648 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:40:38.648 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:40:38.648 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:40:38.648 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:40:38.649 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:38.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484f3dd640 1 Processor -- start 2024-04-26T00:40:38.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484f3dd640 1 -- start start 2024-04-26T00:40:38.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484f3dd640 10 monclient: init 2024-04-26T00:40:38.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484f3dd640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:38.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484f3dd640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:40:38.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484f3dd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f484809e220 0x7f48481cd490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:38.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484f3dd640 10 monclient(hunting): picked mon.noname-c con 0x7f484809e220 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:40:38.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484f3dd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f484809e600 0x7f484811f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:38.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484f3dd640 10 monclient(hunting): picked mon.noname-b con 0x7f484809e600 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:40:38.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484f3dd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f48480a14f0 0x7f484811f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:38.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484f3dd640 10 monclient(hunting): picked mon.noname-a con 0x7f48480a14f0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:40:38.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484f3dd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:38.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484f3dd640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f484813b450 con 0x7f48480a14f0 2024-04-26T00:40:38.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484f3dd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:38.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484f3dd640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f484813b2d0 con 0x7f484809e600 2024-04-26T00:40:38.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484f3dd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:38.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484f3dd640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f484813b5d0 con 0x7f484809e220 2024-04-26T00:40:38.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484f3dd640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:38.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484f3dd640 10 monclient(hunting): authenticate will time out at 2219.942138s 2024-04-26T00:40:38.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484d179640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f484809e220 0x7f48481cd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:38.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484d179640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f484809e220 0x7f48481cd490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:52202/0 (socket says 172.21.15.37:52202) 2024-04-26T00:40:38.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484d179640 1 -- 172.21.15.37:0/2560015649 learned_addr learned my addr 172.21.15.37:0/2560015649 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:40:38.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484d179640 10 monclient(hunting): get_auth_request con 0x7f484809e220 auth_method 0 2024-04-26T00:40:38.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484d179640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:38.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484d179640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:38.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484d179640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:38.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484d97a640 1 --2- 172.21.15.37:0/2560015649 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f48480a14f0 0x7f484811f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:38.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484d97a640 10 monclient(hunting): get_auth_request con 0x7f48480a14f0 auth_method 0 2024-04-26T00:40:38.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484d97a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:38.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484d97a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:38.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484d97a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:38.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484d179640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:38.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484d179640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:38.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484d179640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:38.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484d97a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:38.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484d97a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:38.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484d97a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:38.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484d179640 10 monclient(hunting): handle_auth_done global_id 254111 payload 824 2024-04-26T00:40:38.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484d179640 1 -- 172.21.15.37:0/2560015649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f484809e600 msgr2=0x7f484811f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:38.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484d179640 1 --2- 172.21.15.37:0/2560015649 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f484809e600 0x7f484811f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:38.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484d179640 1 -- 172.21.15.37:0/2560015649 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f48480a14f0 msgr2=0x7f484811f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:38.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484d179640 1 --2- 172.21.15.37:0/2560015649 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f48480a14f0 0x7f484811f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:38.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484d179640 10 monclient: _finish_hunting 0 2024-04-26T00:40:38.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484d179640 1 monclient: found mon.noname-c 2024-04-26T00:40:38.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484d179640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:38.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484d179640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:40:38.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.633+0000 7f484d179640 1 -- 172.21.15.37:0/2560015649 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4848122e20 con 0x7f484809e220 2024-04-26T00:40:38.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f484d179640 1 -- 172.21.15.37:0/2560015649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f484809e220 msgr2=0x7f48481cd490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:38.653 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f484d179640 1 --2- 172.21.15.37:0/2560015649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f484809e220 0x7f48481cd490 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f48440ae040 tx=0x7f4844019180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:38.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f483dffb640 1 -- 172.21.15.37:0/2560015649 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48440b9070 con 0x7f484809e220 2024-04-26T00:40:38.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f483dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:38.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f483dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:40:38.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f483dffb640 10 monclient: dump: 2024-04-26T00:40:38.654 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:38.654 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:38.654 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.654 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.654 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:38.654 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:38.654 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:38.654 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:38.655 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:38.655 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:38.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f483dffb640 10 monclient: _finish_auth 0 2024-04-26T00:40:38.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f483dffb640 10 monclient: _check_auth_tickets 2024-04-26T00:40:38.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f483dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:38.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f483dffb640 1 -- 172.21.15.37:0/2560015649 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48440b4420 con 0x7f484809e220 2024-04-26T00:40:38.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f483dffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:40:38.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f483dffb640 1 -- 172.21.15.37:0/2560015649 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48440b0040 con 0x7f484809e220 2024-04-26T00:40:38.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f484f3dd640 5 monclient: authenticate success, global_id 254111 2024-04-26T00:40:38.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f484f3dd640 10 monclient: _renew_subs 2024-04-26T00:40:38.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f484f3dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:40:38.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f484f3dd640 1 -- 172.21.15.37:0/2560015649 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f484809c970 con 0x7f484809e220 2024-04-26T00:40:38.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f483effd640 4 set_mon_vals no callback set 2024-04-26T00:40:38.655 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f483dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:38.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f483dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:40:38.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f483dffb640 10 monclient: dump: 2024-04-26T00:40:38.656 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:38.656 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:38.656 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.656 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.656 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:38.656 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:38.656 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:38.656 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:38.656 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:38.656 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:38.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f484f3dd640 10 monclient: _renew_subs 2024-04-26T00:40:38.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f484f3dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:40:38.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f484f3dd640 1 -- 172.21.15.37:0/2560015649 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4848123100 con 0x7f484809e220 2024-04-26T00:40:38.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f484f3dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:38.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f484f3dd640 1 --2- 172.21.15.37:0/2560015649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48481cd9d0 0x7f484811fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:38.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f484f3dd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:38.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f484f3dd640 1 -- 172.21.15.37:0/2560015649 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f48480a0ab0 con 0x7f48481cd9d0 2024-04-26T00:40:38.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f484c978640 1 --2- 172.21.15.37:0/2560015649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48481cd9d0 0x7f484811fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:38.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f484c978640 10 monclient: get_auth_request con 0x7f48481cd9d0 auth_method 0 2024-04-26T00:40:38.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f484c978640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:38.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f484c978640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:38.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f484c978640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:38.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f483dffb640 1 -- 172.21.15.37:0/2560015649 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f48440bc070 con 0x7f484809e220 2024-04-26T00:40:38.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f484c978640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:38.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f484c978640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:38.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f484c978640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:38.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f484c978640 10 monclient(hunting): handle_auth_done global_id 254114 payload 824 2024-04-26T00:40:38.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f484c978640 1 -- 172.21.15.37:0/2560015649 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f484804f510 con 0x7f48481cd9d0 2024-04-26T00:40:38.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f483dffb640 1 --2- 172.21.15.37:0/2560015649 >> v2:172.21.15.45:6812/2379918623 conn(0x7f482c035e80 0x7f482c038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:38.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f483dffb640 1 -- 172.21.15.37:0/2560015649 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(820..820 src has 705..820) ==== 3991+0+0 (secure 0 0 0) 0x7f48440e1bd0 con 0x7f484809e220 2024-04-26T00:40:38.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f484d97a640 1 --2- 172.21.15.37:0/2560015649 >> v2:172.21.15.45:6812/2379918623 conn(0x7f482c035e80 0x7f482c038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:38.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.637+0000 7f484d97a640 10 monclient: get_auth_request con 0x7f482c035e80 auth_method 0 2024-04-26T00:40:38.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.641+0000 7f484c978640 1 -- 172.21.15.37:0/2560015649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48481cd9d0 msgr2=0x7f484811fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:38.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.641+0000 7f484c978640 1 --2- 172.21.15.37:0/2560015649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48481cd9d0 0x7f484811fd10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f483800c310 tx=0x7f483800c7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:38.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.641+0000 7f483dffb640 1 -- 172.21.15.37:0/2560015649 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4838018070 con 0x7f48481cd9d0 2024-04-26T00:40:38.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.641+0000 7f483dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:38.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.641+0000 7f483dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:40:38.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.641+0000 7f483dffb640 10 monclient: dump: 2024-04-26T00:40:38.658 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:38.658 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:38.658 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.658 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.659 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:38.659 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:38.659 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:38.659 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:38.659 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:38.659 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:38.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.641+0000 7f483dffb640 1 -- 172.21.15.37:0/2560015649 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4838004800 con 0x7f48481cd9d0 2024-04-26T00:40:38.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.641+0000 7f483dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:38.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.641+0000 7f483dffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:40:38.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.641+0000 7f483dffb640 1 -- 172.21.15.37:0/2560015649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48481cd9d0 msgr2=0x7f484811fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:38.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.641+0000 7f483dffb640 1 --2- 172.21.15.37:0/2560015649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48481cd9d0 0x7f484811fd10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f483800c310 tx=0x7f483800c7d0 comp rx=0 tx=0).stop 2024-04-26T00:40:38.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.641+0000 7f483dffb640 1 -- 172.21.15.37:0/2560015649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48481cd9d0 msgr2=0x7f484811fd10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:40:38.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.641+0000 7f483dffb640 1 --2- 172.21.15.37:0/2560015649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f48481cd9d0 0x7f484811fd10 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f483800c310 tx=0x7f483800c7d0 comp rx=0 tx=0).stop 2024-04-26T00:40:38.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.641+0000 7f484d97a640 1 --2- 172.21.15.37:0/2560015649 >> v2:172.21.15.45:6812/2379918623 conn(0x7f482c035e80 0x7f482c038340 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f4840009fd0 tx=0x7f484000b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:38.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.645+0000 7f484f3dd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:40:38.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.645+0000 7f484f3dd640 1 --2- 172.21.15.37:0/2560015649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4848120250 0x7f484811b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:38.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.645+0000 7f484f3dd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:38.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.645+0000 7f484f3dd640 1 -- 172.21.15.37:0/2560015649 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4848120660 con 0x7f4848120250 2024-04-26T00:40:38.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.645+0000 7f484c978640 1 --2- 172.21.15.37:0/2560015649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4848120250 0x7f484811b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:38.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.645+0000 7f484c978640 10 monclient: get_auth_request con 0x7f4848120250 auth_method 0 2024-04-26T00:40:38.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.645+0000 7f484c978640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:38.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.645+0000 7f484c978640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:38.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.645+0000 7f484c978640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:38.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.645+0000 7f484c978640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:38.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.645+0000 7f484c978640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:38.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.645+0000 7f484c978640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:38.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.645+0000 7f484c978640 10 monclient(hunting): handle_auth_done global_id 254117 payload 824 2024-04-26T00:40:38.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.645+0000 7f484c978640 1 -- 172.21.15.37:0/2560015649 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f48481216f0 con 0x7f4848120250 2024-04-26T00:40:38.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.649+0000 7f484c978640 1 -- 172.21.15.37:0/2560015649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4848120250 msgr2=0x7f484811b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:38.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.649+0000 7f484c978640 1 --2- 172.21.15.37:0/2560015649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4848120250 0x7f484811b8a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4838015000 tx=0x7f4838003770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:38.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.649+0000 7f483dffb640 1 -- 172.21.15.37:0/2560015649 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4838018030 con 0x7f4848120250 2024-04-26T00:40:38.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.649+0000 7f483dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:38.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.649+0000 7f483dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:40:38.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.649+0000 7f483dffb640 10 monclient: dump: 2024-04-26T00:40:38.663 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:38.663 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:38.663 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.663 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.663 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:38.663 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:38.663 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:38.663 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:38.663 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:38.663 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:38.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.649+0000 7f483dffb640 1 -- 172.21.15.37:0/2560015649 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7f483801a830 con 0x7f4848120250 2024-04-26T00:40:38.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.649+0000 7f483dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:40:38.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.649+0000 7f483dffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:40:38.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.649+0000 7f483dffb640 1 -- 172.21.15.37:0/2560015649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4848120250 msgr2=0x7f484811b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:38.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.649+0000 7f483dffb640 1 --2- 172.21.15.37:0/2560015649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4848120250 0x7f484811b8a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4838015000 tx=0x7f4838003770 comp rx=0 tx=0).stop 2024-04-26T00:40:38.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.649+0000 7f483dffb640 1 -- 172.21.15.37:0/2560015649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4848120250 msgr2=0x7f484811b8a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:40:38.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.649+0000 7f483dffb640 1 --2- 172.21.15.37:0/2560015649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4848120250 0x7f484811b8a0 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4838015000 tx=0x7f4838003770 comp rx=0 tx=0).stop 2024-04-26T00:40:38.664 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:40:38.664 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:40:38.664 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:40:38.664 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:40:38.664 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 152, 2024-04-26T00:40:38.664 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 7419, 2024-04-26T00:40:38.665 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:40:38.665 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:40:38.665 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:40:38.665 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:40:38.665 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:38.665 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 2, 2024-04-26T00:40:38.665 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:40:38.665 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:40:38.665 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:40:38.665 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:40:38.665 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:40:38.665 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:40:38.665 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:40:38.665 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:40:38.665 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:40:38.666 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:40:38.666 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:40:38.666 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:40:38.666 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:40:38.666 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:40:38.666 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:38.666 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:40:38.666 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:40:38.666 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:40:38.666 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:40:38.666 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:40:38.666 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:40:38.666 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:40:38.666 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:40:38.667 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:40:38.667 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:40:38.667 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:40:38.667 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:40:38.667 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:40:38.667 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:38.667 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:38.667 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:40:38.667 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:40:38.667 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:40:38.667 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:40:38.667 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:40:38.667 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:40:38.667 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:40:38.668 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:40:38.668 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:40:38.668 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:40:38.668 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:40:38.668 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:40:38.668 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:40:38.668 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:40:38.668 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:40:38.668 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:40:38.668 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:40:38.668 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:40:38.668 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:40:38.668 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:40:38.668 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:40:38.668 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:40:38.669 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:40:38.669 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:40:38.669 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:40:38.669 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:40:38.669 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:40:38.669 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:40:38.669 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:38.669 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:40:38.669 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:38.669 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:40:38.669 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:38.669 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:40:38.669 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:40:38.669 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:40:38.670 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:40:38.670 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:38.670 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:40:38.670 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:40:38.670 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:38.670 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:38.670 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:38.670 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:40:38.670 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:40:38.670 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:38.670 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:38.670 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:38.670 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:38.670 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:40:38.670 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:40:38.671 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:40:38.671 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:40:38.671 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:40:38.671 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:38.671 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:38.671 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:40:38.671 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:40:38.671 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:40:38.671 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:40:38.671 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:38.671 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:40:38.671 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:40:38.671 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:38.671 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:38.671 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:38.672 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:40:38.672 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:40:38.672 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:38.672 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:38.672 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:38.672 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:38.672 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:40:38.672 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:40:38.672 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:40:38.672 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:40:38.672 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:40:38.672 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:38.672 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:38.672 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:40:38.673 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:40:38.673 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:40:38.673 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:40:38.673 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:38.673 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:40:38.673 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:40:38.673 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:38.673 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:38.673 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:38.673 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:40:38.673 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:40:38.673 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:38.673 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:38.673 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:38.674 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:38.674 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:40:38.674 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:40:38.674 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:40:38.674 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:40:38.674 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:40:38.674 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:38.674 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:38.674 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:38.674 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:40:38.674 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:40:38.674 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:38.674 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:40:38.674 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:40:38.675 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:40:38.675 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:38.675 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:38.675 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:40:38.675 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:38.675 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:40:38.675 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:40:38.675 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:40:38.675 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:38.675 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:38.675 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:38.675 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:40:38.675 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:40:38.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.649+0000 7f484f3dd640 1 -- 172.21.15.37:0/2560015649 >> v2:172.21.15.45:6812/2379918623 conn(0x7f482c035e80 msgr2=0x7f482c038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:38.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.649+0000 7f484f3dd640 1 --2- 172.21.15.37:0/2560015649 >> v2:172.21.15.45:6812/2379918623 conn(0x7f482c035e80 0x7f482c038340 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f4840009fd0 tx=0x7f484000b040 comp rx=0 tx=0).stop 2024-04-26T00:40:38.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.649+0000 7f484f3dd640 10 monclient: shutdown 2024-04-26T00:40:38.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.649+0000 7f484f3dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:40:38.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.649+0000 7f484f3dd640 1 -- 172.21.15.37:0/2560015649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f484809e220 msgr2=0x7f48481cd490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:38.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.649+0000 7f484f3dd640 1 --2- 172.21.15.37:0/2560015649 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f484809e220 0x7f48481cd490 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f48440ae040 tx=0x7f4844019180 comp rx=0 tx=0).stop 2024-04-26T00:40:38.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.649+0000 7f484d97a640 1 -- 172.21.15.37:0/2560015649 reap_dead start 2024-04-26T00:40:38.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.649+0000 7f484d97a640 1 -- 172.21.15.37:0/2560015649 reap_dead start 2024-04-26T00:40:38.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.649+0000 7f484f3dd640 1 -- 172.21.15.37:0/2560015649 shutdown_connections 2024-04-26T00:40:38.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.649+0000 7f484f3dd640 1 -- 172.21.15.37:0/2560015649 >> 172.21.15.37:0/2560015649 conn(0x7f48480a1d50 msgr2=0x7f48480a4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:40:38.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.649+0000 7f484f3dd640 1 -- 172.21.15.37:0/2560015649 shutdown_connections 2024-04-26T00:40:38.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.649+0000 7f484f3dd640 1 -- 172.21.15.37:0/2560015649 wait complete. 2024-04-26T00:40:38.678 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:40:38.678 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:40:38.678 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:40:38.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.789+0000 7f5666467640 10 monclient: get_monmap_and_config 2024-04-26T00:40:38.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.789+0000 7f5666467640 10 monclient: build_initial_monmap 2024-04-26T00:40:38.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.789+0000 7f5666467640 10 monclient: monmap: 2024-04-26T00:40:38.792 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:40:38.792 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:38.792 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:40:38.794741+0000 2024-04-26T00:40:38.793 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:40:38.794741+0000 2024-04-26T00:40:38.793 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:40:38.793 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:38.793 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:40:38.793 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:40:38.793 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:40:38.793 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:40:38.793 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:40:38.793 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:38.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f5666467640 1 Processor -- start 2024-04-26T00:40:38.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f5666467640 1 -- start start 2024-04-26T00:40:38.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f5666467640 10 monclient: init 2024-04-26T00:40:38.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f5666467640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:38.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f5666467640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-04-26T00:40:38.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f5666467640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f56600a38e0 0x7f56600a1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:38.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f5666467640 10 monclient(hunting): picked mon.noname-c con 0x7f56600a38e0 addr v2:172.21.15.45:3300/0 2024-04-26T00:40:38.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f5666467640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f56600a2240 addr v1:172.21.15.37:6789/0 2024-04-26T00:40:38.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f5666467640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f56600a2650 0x7f5660137b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:38.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f5666467640 10 monclient(hunting): picked mon.noname-a con 0x7f56600a2650 addr v2:172.21.15.37:3300/0 2024-04-26T00:40:38.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f5666467640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f566009a9d0 con 0x7f56600a2240 2024-04-26T00:40:38.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f5666467640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:38.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f5666467640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f566009bc80 con 0x7f56600a2650 2024-04-26T00:40:38.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f5666467640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:38.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f5666467640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f56600a3cc0 con 0x7f56600a38e0 2024-04-26T00:40:38.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f5666467640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:38.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f5666467640 10 monclient(hunting): authenticate will time out at 2220.100951s 2024-04-26T00:40:38.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f5664a04640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f56600a2650 0x7f5660137b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:38.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f565f7fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f56600a2240 0x7f5660137470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:36278/0 (socket says 172.21.15.37:36278) 2024-04-26T00:40:38.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f565f7fe640 1 -- 172.21.15.37:0/389370749 learned_addr learned my addr 172.21.15.37:0/389370749 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:40:38.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f5664a04640 10 monclient(hunting): get_auth_request con 0x7f56600a2650 auth_method 0 2024-04-26T00:40:38.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f5664a04640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:38.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f5664a04640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:38.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f5664a04640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:38.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f5664a04640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:38.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f5664a04640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:38.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f5664a04640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:38.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f565ffff640 1 --2- 172.21.15.37:0/389370749 >> v2:172.21.15.45:3300/0 conn(0x7f56600a38e0 0x7f56600a1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:38.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f565ffff640 10 monclient(hunting): get_auth_request con 0x7f56600a38e0 auth_method 0 2024-04-26T00:40:38.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f565ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:38.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f565ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:38.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f565ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:38.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f5664a04640 10 monclient(hunting): handle_auth_done global_id 244209 payload 274 2024-04-26T00:40:38.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f5664a04640 1 -- 172.21.15.37:0/389370749 >> v2:172.21.15.45:3300/0 conn(0x7f56600a38e0 msgr2=0x7f56600a1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:40:38.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f5664a04640 1 --2- 172.21.15.37:0/389370749 >> v2:172.21.15.45:3300/0 conn(0x7f56600a38e0 0x7f56600a1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:38.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f5664a04640 1 -- 172.21.15.37:0/389370749 >> v1:172.21.15.37:6789/0 conn(0x7f56600a2240 legacy=0x7f5660137470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:40:38.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f5664a04640 10 monclient: _finish_hunting 0 2024-04-26T00:40:38.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f5664a04640 1 monclient: found mon.noname-a 2024-04-26T00:40:38.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f5664a04640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:38.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f5664a04640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:40:38.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.793+0000 7f5664a04640 1 -- 172.21.15.37:0/389370749 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f566009acb0 con 0x7f56600a2650 2024-04-26T00:40:38.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.797+0000 7f5664a04640 1 -- 172.21.15.37:0/389370749 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56600a2650 msgr2=0x7f5660137b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:38.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.797+0000 7f5664a04640 1 --2- 172.21.15.37:0/389370749 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56600a2650 0x7f5660137b80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f566006f120 tx=0x7f565801c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d19fce485ec421f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:38.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.797+0000 7f565effd640 1 -- 172.21.15.37:0/389370749 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56580b2070 con 0x7f56600a2650 2024-04-26T00:40:38.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.797+0000 7f565effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:38.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.797+0000 7f565effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:40:38.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.797+0000 7f565effd640 10 monclient: dump: 2024-04-26T00:40:38.801 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:38.801 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:38.801 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.801 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.801 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:38.801 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:38.801 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:38.802 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:38.802 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:38.802 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:38.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.797+0000 7f565effd640 10 monclient: _finish_auth 0 2024-04-26T00:40:38.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.797+0000 7f565effd640 10 monclient: _check_auth_tickets 2024-04-26T00:40:38.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.797+0000 7f565effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:38.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.797+0000 7f565effd640 1 -- 172.21.15.37:0/389370749 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56580a5800 con 0x7f56600a2650 2024-04-26T00:40:38.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.797+0000 7f565effd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:40:38.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.797+0000 7f565effd640 1 -- 172.21.15.37:0/389370749 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56580a5bd0 con 0x7f56600a2650 2024-04-26T00:40:38.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.797+0000 7f5666467640 5 monclient: authenticate success, global_id 244209 2024-04-26T00:40:38.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.797+0000 7f5666467640 10 monclient: get_monmap_and_config success 2024-04-26T00:40:38.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.797+0000 7f5666467640 4 set_mon_vals no callback set 2024-04-26T00:40:38.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.797+0000 7f565effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:38.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.797+0000 7f565effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:40:38.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.797+0000 7f565effd640 10 monclient: dump: 2024-04-26T00:40:38.803 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:38.803 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:38.803 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.803 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.803 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:38.803 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:38.803 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:38.803 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:38.803 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:38.803 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:38.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.797+0000 7f5666467640 10 monclient: shutdown 2024-04-26T00:40:38.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.797+0000 7f5666467640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:40:38.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.797+0000 7f5666467640 1 -- 172.21.15.37:0/389370749 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56600a2650 msgr2=0x7f5660137b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:38.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.797+0000 7f5666467640 1 --2- 172.21.15.37:0/389370749 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56600a2650 0x7f5660137b80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f566006f120 tx=0x7f565801c780 comp rx=0 tx=0).stop 2024-04-26T00:40:38.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.797+0000 7f5666467640 1 -- 172.21.15.37:0/389370749 shutdown_connections 2024-04-26T00:40:38.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.797+0000 7f5666467640 1 --2- 172.21.15.37:0/389370749 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56600a2650 0x7f5660137b80 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:38.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.797+0000 7f5666467640 1 --2- 172.21.15.37:0/389370749 >> v2:172.21.15.45:3300/0 conn(0x7f56600a38e0 0x7f56600a1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:38.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.797+0000 7f5666467640 1 -- 172.21.15.37:0/389370749 >> 172.21.15.37:0/389370749 conn(0x7f56600a2df0 msgr2=0x7f56600a31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:40:38.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.797+0000 7f5666467640 1 -- 172.21.15.37:0/389370749 shutdown_connections 2024-04-26T00:40:38.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.797+0000 7f5666467640 1 -- 172.21.15.37:0/389370749 wait complete. 2024-04-26T00:40:38.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.797+0000 7f5666467640 10 monclient: build_initial_monmap 2024-04-26T00:40:38.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.797+0000 7f5666467640 10 monclient: monmap: 2024-04-26T00:40:38.804 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:40:38.804 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:40:38.804 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:40:38.804 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:40:38.805 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:40:38.805 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:40:38.805 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:40:38.805 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:40:38.805 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:40:38.805 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:38.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f5666467640 1 Processor -- start 2024-04-26T00:40:38.810 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f5666467640 1 -- start start 2024-04-26T00:40:38.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f5666467640 10 monclient: init 2024-04-26T00:40:38.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f5666467640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:38.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f5666467640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:40:38.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f5666467640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56600a2240 0x7f5660136f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:38.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f5666467640 10 monclient(hunting): picked mon.noname-c con 0x7f56600a2240 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:40:38.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f5666467640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56600a2650 0x7f56601359c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:38.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f5666467640 10 monclient(hunting): picked mon.noname-a con 0x7f56600a2650 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:40:38.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f5666467640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56600a38e0 0x7f566013e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:38.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f5666467640 10 monclient(hunting): picked mon.noname-b con 0x7f56600a38e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:40:38.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f5666467640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:38.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f5666467640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f566009a360 con 0x7f56600a2650 2024-04-26T00:40:38.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f5666467640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:38.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f5666467640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f56600a2a80 con 0x7f56600a38e0 2024-04-26T00:40:38.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f5666467640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:38.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f5666467640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f56601387c0 con 0x7f56600a2240 2024-04-26T00:40:38.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f5666467640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:38.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f5666467640 10 monclient(hunting): authenticate will time out at 2220.108757s 2024-04-26T00:40:38.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565f7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56600a2650 0x7f56601359c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:38.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565ffff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56600a2240 0x7f5660136f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:38.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565f7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56600a2650 0x7f56601359c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:60984/0 (socket says 172.21.15.37:60984) 2024-04-26T00:40:38.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565ffff640 10 monclient(hunting): get_auth_request con 0x7f56600a2240 auth_method 0 2024-04-26T00:40:38.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565f7fe640 1 -- 172.21.15.37:0/1493388467 learned_addr learned my addr 172.21.15.37:0/1493388467 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:40:38.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:38.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:38.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:38.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565f7fe640 10 monclient(hunting): get_auth_request con 0x7f56600a2650 auth_method 0 2024-04-26T00:40:38.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:38.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:38.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:38.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:38.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:38.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:38.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:38.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:38.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:38.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f5664a04640 1 --2- 172.21.15.37:0/1493388467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56600a38e0 0x7f566013e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:38.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f5664a04640 10 monclient(hunting): get_auth_request con 0x7f56600a38e0 auth_method 0 2024-04-26T00:40:38.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f5664a04640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:38.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f5664a04640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:38.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f5664a04640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:38.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565f7fe640 10 monclient(hunting): handle_auth_done global_id 244212 payload 824 2024-04-26T00:40:38.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565f7fe640 1 -- 172.21.15.37:0/1493388467 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56600a2240 msgr2=0x7f5660136f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:38.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565f7fe640 1 --2- 172.21.15.37:0/1493388467 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56600a2240 0x7f5660136f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:38.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565f7fe640 1 -- 172.21.15.37:0/1493388467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56600a38e0 msgr2=0x7f566013e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:38.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565f7fe640 1 --2- 172.21.15.37:0/1493388467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56600a38e0 0x7f566013e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:38.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565f7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:40:38.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565f7fe640 1 monclient: found mon.noname-a 2024-04-26T00:40:38.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:38.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:40:38.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565f7fe640 1 -- 172.21.15.37:0/1493388467 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5660135f00 con 0x7f56600a2650 2024-04-26T00:40:38.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565f7fe640 1 -- 172.21.15.37:0/1493388467 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56600a2650 msgr2=0x7f56601359c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:38.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565f7fe640 1 --2- 172.21.15.37:0/1493388467 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56600a2650 0x7f56601359c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f565400b430 tx=0x7f565400b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:38.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565cff9640 1 -- 172.21.15.37:0/1493388467 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5654015020 con 0x7f56600a2650 2024-04-26T00:40:38.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:38.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:40:38.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565cff9640 10 monclient: dump: 2024-04-26T00:40:38.815 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:38.815 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:38.815 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.815 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.815 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:38.815 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:38.815 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:38.815 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:38.815 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:38.815 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:38.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565cff9640 10 monclient: _finish_auth 0 2024-04-26T00:40:38.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565cff9640 10 monclient: _check_auth_tickets 2024-04-26T00:40:38.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:38.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565cff9640 1 -- 172.21.15.37:0/1493388467 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5654011a60 con 0x7f56600a2650 2024-04-26T00:40:38.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f5666467640 5 monclient: authenticate success, global_id 244212 2024-04-26T00:40:38.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f5666467640 10 monclient: _renew_subs 2024-04-26T00:40:38.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f5666467640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:40:38.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f5666467640 1 -- 172.21.15.37:0/1493388467 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f566013e7a0 con 0x7f56600a2650 2024-04-26T00:40:38.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565cff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:40:38.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565cff9640 1 -- 172.21.15.37:0/1493388467 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5654011e50 con 0x7f56600a2650 2024-04-26T00:40:38.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:38.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:40:38.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565cff9640 10 monclient: dump: 2024-04-26T00:40:38.816 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:38.816 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:38.816 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.817 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.817 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:38.817 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:38.817 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:38.817 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:38.817 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:38.817 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:38.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f565e7fc640 4 set_mon_vals no callback set 2024-04-26T00:40:38.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f5666467640 10 monclient: _renew_subs 2024-04-26T00:40:38.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f5666467640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:40:38.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.801+0000 7f5666467640 1 -- 172.21.15.37:0/1493388467 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f566013ec90 con 0x7f56600a2650 2024-04-26T00:40:38.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.805+0000 7f5666467640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:38.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.805+0000 7f5666467640 1 --2- 172.21.15.37:0/1493388467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56601ced60 0x7f56601cf140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:38.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.805+0000 7f5666467640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:38.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.805+0000 7f5666467640 1 -- 172.21.15.37:0/1493388467 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f566004f510 con 0x7f56601ced60 2024-04-26T00:40:38.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.805+0000 7f565cff9640 1 -- 172.21.15.37:0/1493388467 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f5654018020 con 0x7f56600a2650 2024-04-26T00:40:38.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.805+0000 7f565ffff640 1 --2- 172.21.15.37:0/1493388467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56601ced60 0x7f56601cf140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:38.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.805+0000 7f565cff9640 1 --2- 172.21.15.37:0/1493388467 >> v2:172.21.15.45:6812/2379918623 conn(0x7f564c036070 0x7f564c038530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:38.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.805+0000 7f565ffff640 10 monclient: get_auth_request con 0x7f56601ced60 auth_method 0 2024-04-26T00:40:38.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.805+0000 7f565ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:38.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.805+0000 7f565ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:38.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.805+0000 7f565ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:38.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.805+0000 7f565cff9640 1 -- 172.21.15.37:0/1493388467 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(820..820 src has 705..820) ==== 3991+0+0 (secure 0 0 0) 0x7f56540486f0 con 0x7f56600a2650 2024-04-26T00:40:38.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.805+0000 7f565ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:38.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.805+0000 7f565ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:38.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.805+0000 7f565ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:38.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.805+0000 7f5664a04640 1 --2- 172.21.15.37:0/1493388467 >> v2:172.21.15.45:6812/2379918623 conn(0x7f564c036070 0x7f564c038530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:38.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.805+0000 7f5664a04640 10 monclient: get_auth_request con 0x7f564c036070 auth_method 0 2024-04-26T00:40:38.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.805+0000 7f565ffff640 10 monclient(hunting): handle_auth_done global_id 224356 payload 824 2024-04-26T00:40:38.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.805+0000 7f565ffff640 1 -- 172.21.15.37:0/1493388467 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f566009a4e0 con 0x7f56601ced60 2024-04-26T00:40:38.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.809+0000 7f565ffff640 1 -- 172.21.15.37:0/1493388467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56601ced60 msgr2=0x7f56601cf140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:38.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.809+0000 7f565ffff640 1 --2- 172.21.15.37:0/1493388467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56601ced60 0x7f56601cf140 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5650010820 tx=0x7f5650010ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:38.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.809+0000 7f5664a04640 1 --2- 172.21.15.37:0/1493388467 >> v2:172.21.15.45:6812/2379918623 conn(0x7f564c036070 0x7f564c038530 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f566006e6c0 tx=0x7f56580a7070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:38.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.809+0000 7f565cff9640 1 -- 172.21.15.37:0/1493388467 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5650018020 con 0x7f56601ced60 2024-04-26T00:40:38.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.809+0000 7f565cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:38.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.809+0000 7f565cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:40:38.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.809+0000 7f565cff9640 10 monclient: dump: 2024-04-26T00:40:38.819 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:38.819 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:38.819 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.819 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.819 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:38.819 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:38.819 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:38.820 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:38.820 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:38.820 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:38.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.809+0000 7f565cff9640 1 -- 172.21.15.37:0/1493388467 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5650004450 con 0x7f56601ced60 2024-04-26T00:40:38.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.809+0000 7f565cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:38.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.809+0000 7f565cff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:40:38.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.809+0000 7f565cff9640 1 -- 172.21.15.37:0/1493388467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56601ced60 msgr2=0x7f56601cf140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:38.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.809+0000 7f565cff9640 1 --2- 172.21.15.37:0/1493388467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56601ced60 0x7f56601cf140 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5650010820 tx=0x7f5650010ce0 comp rx=0 tx=0).stop 2024-04-26T00:40:38.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.809+0000 7f565cff9640 1 -- 172.21.15.37:0/1493388467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56601ced60 msgr2=0x7f56601cf140 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:40:38.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.809+0000 7f565cff9640 1 --2- 172.21.15.37:0/1493388467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56601ced60 0x7f56601cf140 secure :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5650010820 tx=0x7f5650010ce0 comp rx=0 tx=0).stop 2024-04-26T00:40:38.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.813+0000 7f5666467640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:40:38.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.813+0000 7f5666467640 1 --2- 172.21.15.37:0/1493388467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56601cf680 0x7f5660071510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:38.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.813+0000 7f5666467640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:38.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.813+0000 7f5666467640 1 -- 172.21.15.37:0/1493388467 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5660136130 con 0x7f56601cf680 2024-04-26T00:40:38.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.813+0000 7f565ffff640 1 --2- 172.21.15.37:0/1493388467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56601cf680 0x7f5660071510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:38.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.813+0000 7f565ffff640 10 monclient: get_auth_request con 0x7f56601cf680 auth_method 0 2024-04-26T00:40:38.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.813+0000 7f565ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:38.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.813+0000 7f565ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:38.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.813+0000 7f565ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:38.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.813+0000 7f565ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:38.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.813+0000 7f565ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:38.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.813+0000 7f565ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:38.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.817+0000 7f565ffff640 10 monclient(hunting): handle_auth_done global_id 224359 payload 824 2024-04-26T00:40:38.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.817+0000 7f565ffff640 1 -- 172.21.15.37:0/1493388467 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f566006caf0 con 0x7f56601cf680 2024-04-26T00:40:38.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.817+0000 7f565ffff640 1 -- 172.21.15.37:0/1493388467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56601cf680 msgr2=0x7f5660071510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:38.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.817+0000 7f565ffff640 1 --2- 172.21.15.37:0/1493388467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56601cf680 0x7f5660071510 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5650010820 tx=0x7f56500046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:38.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.817+0000 7f565cff9640 1 -- 172.21.15.37:0/1493388467 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5650018030 con 0x7f56601cf680 2024-04-26T00:40:38.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.817+0000 7f565cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:38.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.817+0000 7f565cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:40:38.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.817+0000 7f565cff9640 10 monclient: dump: 2024-04-26T00:40:38.823 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:38.824 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:38.824 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.824 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.824 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:38.824 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:38.824 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:38.824 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:38.824 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:38.824 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:38.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.817+0000 7f565cff9640 1 -- 172.21.15.37:0/1493388467 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4888 (secure 0 0 0) 0x7f565000ecb0 con 0x7f56601cf680 2024-04-26T00:40:38.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.817+0000 7f565cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:40:38.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.817+0000 7f565cff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:40:38.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.817+0000 7f565cff9640 1 -- 172.21.15.37:0/1493388467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56601cf680 msgr2=0x7f5660071510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:38.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.817+0000 7f565cff9640 1 --2- 172.21.15.37:0/1493388467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56601cf680 0x7f5660071510 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5650010820 tx=0x7f56500046d0 comp rx=0 tx=0).stop 2024-04-26T00:40:38.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.817+0000 7f565cff9640 1 -- 172.21.15.37:0/1493388467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56601cf680 msgr2=0x7f5660071510 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:40:38.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.817+0000 7f565cff9640 1 --2- 172.21.15.37:0/1493388467 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56601cf680 0x7f5660071510 secure :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5650010820 tx=0x7f56500046d0 comp rx=0 tx=0).stop 2024-04-26T00:40:38.825 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:40:38.825 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:40:38.825 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:40:38.825 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:40:38.825 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 152, 2024-04-26T00:40:38.825 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 152481, 2024-04-26T00:40:38.825 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:40:38.826 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:40:38.826 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:40:38.826 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:40:38.826 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:38.826 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 2, 2024-04-26T00:40:38.826 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:40:38.826 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:40:38.826 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:40:38.826 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:40:38.826 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:40:38.826 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:40:38.826 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:40:38.826 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:40:38.827 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:40:38.827 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:40:38.827 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:40:38.827 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:40:38.827 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:40:38.827 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:40:38.827 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:38.827 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:40:38.827 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:40:38.827 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:40:38.827 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:40:38.827 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:40:38.827 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:40:38.828 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:40:38.828 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:40:38.828 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:40:38.828 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:40:38.828 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:40:38.828 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:40:38.828 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:40:38.828 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:38.828 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:38.828 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:40:38.828 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:40:38.828 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:40:38.828 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:40:38.829 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:40:38.829 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:40:38.829 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:40:38.829 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:40:38.829 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:40:38.829 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:40:38.829 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:40:38.829 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:40:38.829 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:40:38.829 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:40:38.829 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:40:38.829 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:40:38.829 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:40:38.829 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:40:38.829 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:40:38.830 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:40:38.830 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:40:38.830 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:40:38.830 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:40:38.830 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:40:38.830 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:40:38.830 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:40:38.830 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:40:38.830 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:40:38.830 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:38.830 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:40:38.830 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:38.830 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:40:38.831 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:38.831 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:40:38.831 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:40:38.831 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:40:38.831 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:40:38.831 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:38.831 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:40:38.831 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:40:38.831 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:38.831 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:38.831 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:38.831 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:40:38.831 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:40:38.832 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:38.832 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:38.832 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:38.832 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:38.832 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:40:38.832 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:40:38.832 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:40:38.832 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:40:38.832 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:40:38.832 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:38.832 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:38.832 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:40:38.832 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:40:38.832 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:40:38.833 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:40:38.833 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:38.833 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:40:38.833 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:40:38.833 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:38.833 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:38.833 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:38.833 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:40:38.833 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:40:38.833 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:38.833 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:38.833 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:38.833 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:38.833 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:40:38.833 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:40:38.833 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:40:38.834 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:40:38.834 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:40:38.834 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:38.834 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:38.834 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:40:38.834 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:40:38.834 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:40:38.834 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:40:38.834 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:38.834 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:40:38.834 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:40:38.834 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:38.834 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:38.834 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:38.834 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:40:38.835 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:40:38.835 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:38.835 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:38.835 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:38.835 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:38.835 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:40:38.835 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:40:38.835 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:40:38.835 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:40:38.835 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:40:38.835 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:38.835 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:38.835 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:38.835 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:40:38.835 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:40:38.835 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:38.836 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:40:38.836 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:40:38.836 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:40:38.836 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:38.836 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:38.836 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:40:38.836 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:38.836 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:40:38.836 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:40:38.836 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:40:38.836 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:38.836 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:38.836 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:40:38.836 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:38.837 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:40:38.837 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:40:38.837 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:40:38.837 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:38.837 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:38.837 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:40:38.837 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:38.837 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:40:38.837 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:40:38.837 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:40:38.837 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:38.837 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:38.837 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:38.837 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:40:38.837 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:40:38.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.821+0000 7f5666467640 1 -- 172.21.15.37:0/1493388467 >> v2:172.21.15.45:6812/2379918623 conn(0x7f564c036070 msgr2=0x7f564c038530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:38.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.821+0000 7f5666467640 1 --2- 172.21.15.37:0/1493388467 >> v2:172.21.15.45:6812/2379918623 conn(0x7f564c036070 0x7f564c038530 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f566006e6c0 tx=0x7f56580a7070 comp rx=0 tx=0).stop 2024-04-26T00:40:38.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.821+0000 7f5666467640 10 monclient: shutdown 2024-04-26T00:40:38.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.821+0000 7f5666467640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:40:38.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.821+0000 7f5666467640 1 -- 172.21.15.37:0/1493388467 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56600a2650 msgr2=0x7f56601359c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:38.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.821+0000 7f5666467640 1 --2- 172.21.15.37:0/1493388467 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56600a2650 0x7f56601359c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f565400b430 tx=0x7f565400b8f0 comp rx=0 tx=0).stop 2024-04-26T00:40:38.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.821+0000 7f5664a04640 1 -- 172.21.15.37:0/1493388467 reap_dead start 2024-04-26T00:40:38.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.821+0000 7f5664a04640 1 -- 172.21.15.37:0/1493388467 reap_dead start 2024-04-26T00:40:38.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.821+0000 7f5666467640 1 -- 172.21.15.37:0/1493388467 shutdown_connections 2024-04-26T00:40:38.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.821+0000 7f5666467640 1 -- 172.21.15.37:0/1493388467 >> 172.21.15.37:0/1493388467 conn(0x7f56600a2df0 msgr2=0x7f566009f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:40:38.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.821+0000 7f5666467640 1 -- 172.21.15.37:0/1493388467 shutdown_connections 2024-04-26T00:40:38.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.821+0000 7f5666467640 1 -- 172.21.15.37:0/1493388467 wait complete. 2024-04-26T00:40:38.840 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:40:38.840 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-26T00:40:38.840 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-26T00:40:38.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.949+0000 7fc6540fa640 10 monclient: get_monmap_and_config 2024-04-26T00:40:38.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.949+0000 7fc6540fa640 10 monclient: build_initial_monmap 2024-04-26T00:40:38.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.949+0000 7fc6540fa640 10 monclient: monmap: 2024-04-26T00:40:38.952 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:40:38.952 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:38.952 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:40:38.954995+0000 2024-04-26T00:40:38.953 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:40:38.954995+0000 2024-04-26T00:40:38.953 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:40:38.953 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:38.953 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:40:38.953 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:40:38.953 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:40:38.953 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:40:38.953 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:40:38.953 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:38.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.953+0000 7fc6540fa640 1 Processor -- start 2024-04-26T00:40:38.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.953+0000 7fc6540fa640 1 -- start start 2024-04-26T00:40:38.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.953+0000 7fc6540fa640 10 monclient: init 2024-04-26T00:40:38.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.953+0000 7fc6540fa640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:38.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.953+0000 7fc6540fa640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-04-26T00:40:38.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.953+0000 7fc6540fa640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc64c0a0260 0x7fc64c0a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:38.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.953+0000 7fc6540fa640 10 monclient(hunting): picked mon.noname-b con 0x7fc64c0a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:40:38.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.953+0000 7fc6540fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc64c134b30 addr v1:172.21.15.45:6789/0 2024-04-26T00:40:38.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.953+0000 7fc6540fa640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc64c135660 0x7fc64c13ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:38.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.953+0000 7fc6540fa640 10 monclient(hunting): picked mon.noname-c con 0x7fc64c135660 addr v2:172.21.15.45:3300/0 2024-04-26T00:40:38.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.953+0000 7fc6540fa640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc64c0a1130 con 0x7fc64c134b30 2024-04-26T00:40:38.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.953+0000 7fc6540fa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:38.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.953+0000 7fc6540fa640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc64c09ec10 con 0x7fc64c135660 2024-04-26T00:40:38.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.953+0000 7fc6540fa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:38.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.953+0000 7fc6540fa640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc64c09f0f0 con 0x7fc64c0a0260 2024-04-26T00:40:38.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.953+0000 7fc6540fa640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:38.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.953+0000 7fc6540fa640 10 monclient(hunting): authenticate will time out at 2220.261160s 2024-04-26T00:40:38.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.953+0000 7fc651e96640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc64c0a0260 0x7fc64c0a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:38.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.953+0000 7fc651e96640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc64c0a0260 0x7fc64c0a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:52236/0 (socket says 172.21.15.37:52236) 2024-04-26T00:40:38.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.953+0000 7fc651e96640 1 -- 172.21.15.37:0/2893273492 learned_addr learned my addr 172.21.15.37:0/2893273492 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:40:38.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.953+0000 7fc651e96640 10 monclient(hunting): get_auth_request con 0x7fc64c0a0260 auth_method 0 2024-04-26T00:40:38.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.953+0000 7fc651e96640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:38.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.953+0000 7fc651e96640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:38.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.953+0000 7fc651e96640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:38.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.953+0000 7fc651e96640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:38.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.953+0000 7fc651e96640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:38.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.953+0000 7fc651e96640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:38.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.953+0000 7fc652697640 1 --2- 172.21.15.37:0/2893273492 >> v2:172.21.15.45:3300/0 conn(0x7fc64c135660 0x7fc64c13ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:38.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.953+0000 7fc651e96640 10 monclient(hunting): handle_auth_done global_id 254123 payload 274 2024-04-26T00:40:38.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.953+0000 7fc651e96640 1 -- 172.21.15.37:0/2893273492 >> v2:172.21.15.45:3300/0 conn(0x7fc64c135660 msgr2=0x7fc64c13ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:40:38.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.953+0000 7fc651e96640 1 --2- 172.21.15.37:0/2893273492 >> v2:172.21.15.45:3300/0 conn(0x7fc64c135660 0x7fc64c13ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:38.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.953+0000 7fc651e96640 1 -- 172.21.15.37:0/2893273492 >> v1:172.21.15.45:6789/0 conn(0x7fc64c134b30 legacy=0x7fc64c134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:40:38.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.953+0000 7fc651e96640 10 monclient: _finish_hunting 0 2024-04-26T00:40:38.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.953+0000 7fc651e96640 1 monclient: found mon.noname-b 2024-04-26T00:40:38.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.953+0000 7fc651e96640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:38.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.953+0000 7fc651e96640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:40:38.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.953+0000 7fc651e96640 1 -- 172.21.15.37:0/2893273492 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc64c06cff0 con 0x7fc64c0a0260 2024-04-26T00:40:38.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.957+0000 7fc651e96640 1 -- 172.21.15.37:0/2893273492 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc64c0a0260 msgr2=0x7fc64c0a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:38.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.957+0000 7fc651e96640 1 --2- 172.21.15.37:0/2893273492 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc64c0a0260 0x7fc64c0a0640 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc63c009870 tx=0x7fc63c01c200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b428d8837a9ca680 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:38.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.957+0000 7fc650e94640 1 -- 172.21.15.37:0/2893273492 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc63c0b0070 con 0x7fc64c0a0260 2024-04-26T00:40:38.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.957+0000 7fc650e94640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:38.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.957+0000 7fc650e94640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:40:38.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.957+0000 7fc650e94640 10 monclient: dump: 2024-04-26T00:40:38.959 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:38.959 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:38.959 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.960 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.960 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:38.960 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:38.960 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:38.960 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:38.960 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:38.960 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:38.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.957+0000 7fc650e94640 10 monclient: _finish_auth 0 2024-04-26T00:40:38.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.957+0000 7fc650e94640 10 monclient: _check_auth_tickets 2024-04-26T00:40:38.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.957+0000 7fc650e94640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:38.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.957+0000 7fc650e94640 1 -- 172.21.15.37:0/2893273492 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc63c0ab600 con 0x7fc64c0a0260 2024-04-26T00:40:38.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.957+0000 7fc650e94640 10 monclient: handle_config config(0 keys) 2024-04-26T00:40:38.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.957+0000 7fc650e94640 1 -- 172.21.15.37:0/2893273492 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc63c0ab9d0 con 0x7fc64c0a0260 2024-04-26T00:40:38.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.957+0000 7fc6540fa640 5 monclient: authenticate success, global_id 254123 2024-04-26T00:40:38.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.957+0000 7fc6540fa640 10 monclient: get_monmap_and_config success 2024-04-26T00:40:38.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.957+0000 7fc6540fa640 4 set_mon_vals no callback set 2024-04-26T00:40:38.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.957+0000 7fc650e94640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:38.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.957+0000 7fc650e94640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:40:38.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.957+0000 7fc650e94640 10 monclient: dump: 2024-04-26T00:40:38.961 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:38.961 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:38.961 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.961 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.961 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:38.961 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:38.961 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:38.961 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:38.961 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:38.961 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:38.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.957+0000 7fc6540fa640 10 monclient: shutdown 2024-04-26T00:40:38.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.957+0000 7fc6540fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:40:38.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.957+0000 7fc6540fa640 1 -- 172.21.15.37:0/2893273492 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc64c0a0260 msgr2=0x7fc64c0a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:38.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.957+0000 7fc6540fa640 1 --2- 172.21.15.37:0/2893273492 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc64c0a0260 0x7fc64c0a0640 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc63c009870 tx=0x7fc63c01c200 comp rx=0 tx=0).stop 2024-04-26T00:40:38.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.957+0000 7fc6540fa640 1 -- 172.21.15.37:0/2893273492 shutdown_connections 2024-04-26T00:40:38.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.957+0000 7fc6540fa640 1 --2- 172.21.15.37:0/2893273492 >> v2:172.21.15.45:3300/0 conn(0x7fc64c135660 0x7fc64c13ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:38.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.957+0000 7fc6540fa640 1 --2- 172.21.15.37:0/2893273492 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc64c0a0260 0x7fc64c0a0640 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:38.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.957+0000 7fc6540fa640 1 -- 172.21.15.37:0/2893273492 >> 172.21.15.37:0/2893273492 conn(0x7fc64c0a1ad0 msgr2=0x7fc64c0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:40:38.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.957+0000 7fc6540fa640 1 -- 172.21.15.37:0/2893273492 shutdown_connections 2024-04-26T00:40:38.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.957+0000 7fc6540fa640 1 -- 172.21.15.37:0/2893273492 wait complete. 2024-04-26T00:40:38.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.957+0000 7fc6540fa640 10 monclient: build_initial_monmap 2024-04-26T00:40:38.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.957+0000 7fc6540fa640 10 monclient: monmap: 2024-04-26T00:40:38.967 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:40:38.967 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:40:38.967 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:40:38.967 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:40:38.967 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:40:38.967 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:40:38.967 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:40:38.967 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:40:38.967 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:40:38.967 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:38.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.957+0000 7fc6540fa640 1 Processor -- start 2024-04-26T00:40:38.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.957+0000 7fc6540fa640 1 -- start start 2024-04-26T00:40:38.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.957+0000 7fc6540fa640 10 monclient: init 2024-04-26T00:40:38.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc6540fa640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:38.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc6540fa640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:40:38.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc6540fa640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc64c0a0260 0x7fc64c11cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:38.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc6540fa640 10 monclient(hunting): picked mon.noname-c con 0x7fc64c0a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:40:38.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc6540fa640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc64c134b30 0x7fc64c11b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:38.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc6540fa640 10 monclient(hunting): picked mon.noname-a con 0x7fc64c134b30 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:40:38.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc6540fa640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc64c135660 0x7fc64c131a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:38.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc6540fa640 10 monclient(hunting): picked mon.noname-b con 0x7fc64c135660 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:40:38.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc6540fa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:38.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc6540fa640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fc64c0a0010 con 0x7fc64c134b30 2024-04-26T00:40:38.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc6540fa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:38.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc6540fa640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc64c09fe90 con 0x7fc64c135660 2024-04-26T00:40:38.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc6540fa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:38.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc6540fa640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc64c13fbd0 con 0x7fc64c0a0260 2024-04-26T00:40:38.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc6540fa640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:38.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc6540fa640 10 monclient(hunting): authenticate will time out at 2220.268304s 2024-04-26T00:40:38.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc651695640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc64c134b30 0x7fc64c11b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:38.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc651e96640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc64c0a0260 0x7fc64c11cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:38.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc651e96640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc64c0a0260 0x7fc64c11cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:52242/0 (socket says 172.21.15.37:52242) 2024-04-26T00:40:38.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc651e96640 1 -- 172.21.15.37:0/2325783869 learned_addr learned my addr 172.21.15.37:0/2325783869 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:40:38.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc651695640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc64c134b30 0x7fc64c11b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:60990/0 (socket says 172.21.15.37:60990) 2024-04-26T00:40:38.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc651e96640 10 monclient(hunting): get_auth_request con 0x7fc64c0a0260 auth_method 0 2024-04-26T00:40:38.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc651e96640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:38.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc651e96640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:38.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc651e96640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:38.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc651695640 10 monclient(hunting): get_auth_request con 0x7fc64c134b30 auth_method 0 2024-04-26T00:40:38.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc651695640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:38.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc651695640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:38.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc651695640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:38.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc651e96640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:38.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc651e96640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:38.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc651e96640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:38.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc651695640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:38.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc651695640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:38.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc651695640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:38.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc652697640 1 --2- 172.21.15.37:0/2325783869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc64c135660 0x7fc64c131a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:38.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc652697640 10 monclient(hunting): get_auth_request con 0x7fc64c135660 auth_method 0 2024-04-26T00:40:38.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc652697640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:38.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc652697640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:38.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc652697640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:38.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc651e96640 10 monclient(hunting): handle_auth_done global_id 254126 payload 824 2024-04-26T00:40:38.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc651e96640 1 -- 172.21.15.37:0/2325783869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc64c135660 msgr2=0x7fc64c131a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:38.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc651e96640 1 --2- 172.21.15.37:0/2325783869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc64c135660 0x7fc64c131a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:38.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc651e96640 1 -- 172.21.15.37:0/2325783869 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc64c134b30 msgr2=0x7fc64c11b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:38.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc651e96640 1 --2- 172.21.15.37:0/2325783869 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc64c134b30 0x7fc64c11b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:38.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc651e96640 10 monclient: _finish_hunting 0 2024-04-26T00:40:38.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc651e96640 1 monclient: found mon.noname-c 2024-04-26T00:40:38.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc651e96640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:38.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc651e96640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:40:38.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc651e96640 1 -- 172.21.15.37:0/2325783869 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc64c071ba0 con 0x7fc64c0a0260 2024-04-26T00:40:38.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc651695640 1 --2- 172.21.15.37:0/2325783869 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc64c134b30 0x7fc64c11b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:40:38.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc651e96640 1 -- 172.21.15.37:0/2325783869 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc64c0a0260 msgr2=0x7fc64c11cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:38.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc651e96640 1 --2- 172.21.15.37:0/2325783869 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc64c0a0260 0x7fc64c11cc40 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc63c019e30 tx=0x7fc63c018fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:38.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc6427fc640 1 -- 172.21.15.37:0/2325783869 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc63c0ba070 con 0x7fc64c0a0260 2024-04-26T00:40:38.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc6427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:38.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc6427fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:40:38.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc6427fc640 10 monclient: dump: 2024-04-26T00:40:38.971 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:38.971 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:38.971 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.972 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.972 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:38.972 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:38.972 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:38.972 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:38.972 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:38.972 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:38.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc6427fc640 10 monclient: _finish_auth 0 2024-04-26T00:40:38.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc6427fc640 10 monclient: _check_auth_tickets 2024-04-26T00:40:38.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc6427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:38.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc6540fa640 5 monclient: authenticate success, global_id 254126 2024-04-26T00:40:38.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc6540fa640 10 monclient: _renew_subs 2024-04-26T00:40:38.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc6427fc640 1 -- 172.21.15.37:0/2325783869 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc63c0b1040 con 0x7fc64c0a0260 2024-04-26T00:40:38.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc6540fa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:40:38.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc6540fa640 1 -- 172.21.15.37:0/2325783869 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc64c11e0f0 con 0x7fc64c0a0260 2024-04-26T00:40:38.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc6427fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:40:38.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc6427fc640 1 -- 172.21.15.37:0/2325783869 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc63c0b0070 con 0x7fc64c0a0260 2024-04-26T00:40:38.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc6427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:38.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc6427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:40:38.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc6427fc640 10 monclient: dump: 2024-04-26T00:40:38.973 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:38.973 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:38.973 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.973 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:38.973 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:38.973 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:38.973 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:38.973 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:38.973 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:38.973 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:38.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc6437fe640 4 set_mon_vals no callback set 2024-04-26T00:40:38.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc6540fa640 10 monclient: _renew_subs 2024-04-26T00:40:38.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc6540fa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:40:38.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.961+0000 7fc6540fa640 1 -- 172.21.15.37:0/2325783869 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc64c131f60 con 0x7fc64c0a0260 2024-04-26T00:40:38.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.965+0000 7fc6540fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:38.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.965+0000 7fc6540fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:38.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.965+0000 7fc6427fc640 1 -- 172.21.15.37:0/2325783869 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc63c0bd070 con 0x7fc64c0a0260 2024-04-26T00:40:38.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.965+0000 7fc6540fa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:40:38.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.965+0000 7fc6540fa640 1 -- 172.21.15.37:0/2325783869 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc64c066f40 con 0x7fc64c0a0260 2024-04-26T00:40:38.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.965+0000 7fc6427fc640 1 --2- 172.21.15.37:0/2325783869 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc630035eb0 0x7fc630038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:38.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.965+0000 7fc6427fc640 1 -- 172.21.15.37:0/2325783869 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(820..820 src has 705..820) ==== 3991+0+0 (secure 0 0 0) 0x7fc63c0e1c10 con 0x7fc64c0a0260 2024-04-26T00:40:38.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.965+0000 7fc651695640 1 --2- 172.21.15.37:0/2325783869 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc630035eb0 0x7fc630038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:38.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.965+0000 7fc651695640 10 monclient: get_auth_request con 0x7fc630035eb0 auth_method 0 2024-04-26T00:40:38.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.965+0000 7fc651695640 1 --2- 172.21.15.37:0/2325783869 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc630035eb0 0x7fc630038370 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fc64c071680 tx=0x7fc648007e40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:38.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.969+0000 7fc6427fc640 1 -- 172.21.15.37:0/2325783869 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc64c0a1130 con 0x7fc64c0a0260 2024-04-26T00:40:38.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.969+0000 7fc6427fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:38.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:38.969+0000 7fc6427fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:40:39.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:39.193+0000 7fc6540fa640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-26T00:40:39.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:39.193+0000 7fc6540fa640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-26T00:40:39.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:39.193+0000 7fc6540fa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:40:39.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:39.193+0000 7fc6540fa640 1 -- 172.21.15.37:0/2325783869 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fc64c11dd00 con 0x7fc64c0a0260 2024-04-26T00:40:39.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:39.197+0000 7fc6427fc640 1 -- 172.21.15.37:0/2325783869 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fc63c0bd360 con 0x7fc64c0a0260 2024-04-26T00:40:39.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:39.197+0000 7fc6427fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-26T00:40:39.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:39.197+0000 7fc6427fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:40:39.198 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.197+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.198 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.197+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=11,kv=9,logm=54} crc {auth=642774140,config=692619927,health=1140756147,kv=3748211275,logm=3495413055}) 2024-04-26T00:40:39.198 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.197+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=11,kv=9,logm=52} crc {auth=676521973,config=692619928,health=3158893089,kv=789508982,logm=2827934648}) 2024-04-26T00:40:39.198 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.197+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.198 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.197+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=11,kv=9,logm=54} crc {auth=642774140,config=692619927,health=1140756147,kv=3748211275,logm=3495413055}) 2024-04-26T00:40:39.198 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.197+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=24,config=2,health=12,kv=9,logm=53} crc {auth=3566143647,config=692619927,health=1497466813,kv=3913313004,logm=2362115929}) 2024-04-26T00:40:39.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:39.197+0000 7fc6540fa640 1 -- 172.21.15.37:0/2325783869 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc630035eb0 msgr2=0x7fc630038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:39.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:39.197+0000 7fc6540fa640 1 --2- 172.21.15.37:0/2325783869 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc630035eb0 0x7fc630038370 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fc64c071680 tx=0x7fc648007e40 comp rx=0 tx=0).stop 2024-04-26T00:40:39.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:39.197+0000 7fc6540fa640 10 monclient: shutdown 2024-04-26T00:40:39.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:39.197+0000 7fc6540fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:40:39.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:39.197+0000 7fc6540fa640 1 -- 172.21.15.37:0/2325783869 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc64c0a0260 msgr2=0x7fc64c11cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:39.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:39.197+0000 7fc6540fa640 1 --2- 172.21.15.37:0/2325783869 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc64c0a0260 0x7fc64c11cc40 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc63c019e30 tx=0x7fc63c018fa0 comp rx=0 tx=0).stop 2024-04-26T00:40:39.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:39.197+0000 7fc6540fa640 1 -- 172.21.15.37:0/2325783869 shutdown_connections 2024-04-26T00:40:39.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:39.197+0000 7fc6540fa640 1 --2- 172.21.15.37:0/2325783869 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc630035eb0 0x7fc630038370 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:39.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:39.197+0000 7fc6540fa640 1 --2- 172.21.15.37:0/2325783869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc64c135660 0x7fc64c131a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:39.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:39.197+0000 7fc6540fa640 1 --2- 172.21.15.37:0/2325783869 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc64c134b30 0x7fc64c11b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:39.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:39.197+0000 7fc6540fa640 1 --2- 172.21.15.37:0/2325783869 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc64c0a0260 0x7fc64c11cc40 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:39.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:39.197+0000 7fc6540fa640 1 -- 172.21.15.37:0/2325783869 >> 172.21.15.37:0/2325783869 conn(0x7fc64c0a1ad0 msgr2=0x7fc64c133aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:40:39.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:39.197+0000 7fc6540fa640 1 -- 172.21.15.37:0/2325783869 shutdown_connections 2024-04-26T00:40:39.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:39.197+0000 7fc6540fa640 1 -- 172.21.15.37:0/2325783869 wait complete. 2024-04-26T00:40:39.201 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.197+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.201 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.197+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=697199216}) 2024-04-26T00:40:39.202 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.197+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3284299196}) 2024-04-26T00:40:39.202 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.197+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.202 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.197+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=697199216}) 2024-04-26T00:40:39.202 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.201+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1067279591}) 2024-04-26T00:40:39.207 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.201+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.207 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.201+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1327972728}) 2024-04-26T00:40:39.207 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.201+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4194231728}) 2024-04-26T00:40:39.207 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.201+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.207 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.201+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1327972728}) 2024-04-26T00:40:39.207 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.201+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=51615727}) 2024-04-26T00:40:39.207 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.205+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.207 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.205+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2837191872}) 2024-04-26T00:40:39.207 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.205+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1738256795}) 2024-04-26T00:40:39.207 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.205+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.207 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.205+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2837191872}) 2024-04-26T00:40:39.208 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.205+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2406243322}) 2024-04-26T00:40:39.211 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.209+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.211 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.209+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3726764835}) 2024-04-26T00:40:39.211 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.209+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3005797943}) 2024-04-26T00:40:39.212 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.209+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.212 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.209+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3726764835}) 2024-04-26T00:40:39.212 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.209+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=510772661}) 2024-04-26T00:40:39.213 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.209+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.213 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.209+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=583654208}) 2024-04-26T00:40:39.213 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.209+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2711230065}) 2024-04-26T00:40:39.213 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.209+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.213 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.209+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=583654208}) 2024-04-26T00:40:39.213 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.209+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=971012303}) 2024-04-26T00:40:39.216 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.213+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.216 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.213+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=833082407}) 2024-04-26T00:40:39.216 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.213+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=367500909}) 2024-04-26T00:40:39.216 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.213+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.216 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.213+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=833082407}) 2024-04-26T00:40:39.216 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.213+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2279711794}) 2024-04-26T00:40:39.217 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-26T00:40:39.226 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.225+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.226 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.225+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3983994600}) 2024-04-26T00:40:39.226 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.225+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2557449831}) 2024-04-26T00:40:39.226 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.225+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.226 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.225+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3983994600}) 2024-04-26T00:40:39.226 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.225+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1341755661}) 2024-04-26T00:40:39.228 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.225+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.228 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.225+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3550844642}) 2024-04-26T00:40:39.228 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.225+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1123645570}) 2024-04-26T00:40:39.229 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.225+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.229 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.225+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3550844642}) 2024-04-26T00:40:39.229 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.225+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4186740023}) 2024-04-26T00:40:39.231 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.229+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.231 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.229+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=230006617}) 2024-04-26T00:40:39.231 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.229+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1626291241}) 2024-04-26T00:40:39.231 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.229+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.231 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.229+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=230006617}) 2024-04-26T00:40:39.231 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.229+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2614229326}) 2024-04-26T00:40:39.233 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.233+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.233 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.233+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1907179442}) 2024-04-26T00:40:39.233 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.233+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1663379530}) 2024-04-26T00:40:39.234 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.233+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.234 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.233+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1907179442}) 2024-04-26T00:40:39.234 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.233+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=467947809}) 2024-04-26T00:40:39.236 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.233+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.236 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.233+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2213535398}) 2024-04-26T00:40:39.236 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.233+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1359616143}) 2024-04-26T00:40:39.236 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.233+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.236 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.233+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2213535398}) 2024-04-26T00:40:39.236 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.233+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2586847868}) 2024-04-26T00:40:39.238 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.237+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.238 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.237+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4175880888}) 2024-04-26T00:40:39.238 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.237+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1055482778}) 2024-04-26T00:40:39.238 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.237+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.238 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.237+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4175880888}) 2024-04-26T00:40:39.239 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.237+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2662694326}) 2024-04-26T00:40:39.241 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.237+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.241 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.237+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=448695144}) 2024-04-26T00:40:39.241 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.237+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4122628211}) 2024-04-26T00:40:39.241 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.237+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.241 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.237+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=448695144}) 2024-04-26T00:40:39.241 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.237+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=931071534}) 2024-04-26T00:40:39.243 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.241+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.243 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.241+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3604197758}) 2024-04-26T00:40:39.243 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.241+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2662378119}) 2024-04-26T00:40:39.243 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.241+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.243 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.241+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3604197758}) 2024-04-26T00:40:39.244 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.241+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1950924982}) 2024-04-26T00:40:39.245 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.245+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.245 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.245+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=969857594}) 2024-04-26T00:40:39.246 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.245+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2180243970}) 2024-04-26T00:40:39.246 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.245+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.246 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.245+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=969857594}) 2024-04-26T00:40:39.246 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.245+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4063643389}) 2024-04-26T00:40:39.248 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.245+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.248 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.245+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4139806654}) 2024-04-26T00:40:39.249 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.245+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1716294949}) 2024-04-26T00:40:39.249 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.245+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.249 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.245+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4139806654}) 2024-04-26T00:40:39.249 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.245+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2765133228}) 2024-04-26T00:40:39.259 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.257+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.259 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.257+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=110084769}) 2024-04-26T00:40:39.259 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.257+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1936588754}) 2024-04-26T00:40:39.259 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.257+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.259 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.257+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=110084769}) 2024-04-26T00:40:39.259 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.257+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=601639691}) 2024-04-26T00:40:39.271 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.269+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.271 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.269+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3289881620}) 2024-04-26T00:40:39.271 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.269+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=32614465}) 2024-04-26T00:40:39.271 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.269+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.271 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.269+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3289881620}) 2024-04-26T00:40:39.271 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.269+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1718941703}) 2024-04-26T00:40:39.277 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.277+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.277 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.277+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3156460583}) 2024-04-26T00:40:39.277 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.277+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2530817738}) 2024-04-26T00:40:39.277 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.277+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.277 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.277+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3156460583}) 2024-04-26T00:40:39.277 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.277+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3513894990}) 2024-04-26T00:40:39.279 INFO:tasks.rados.rados.0.smithi045.stdout:3637: done (3 left) 2024-04-26T00:40:39.279 INFO:tasks.rados.rados.0.smithi045.stdout:3638: done (2 left) 2024-04-26T00:40:39.279 INFO:tasks.rados.rados.0.smithi045.stdout:3639: done (1 left) 2024-04-26T00:40:39.279 INFO:tasks.rados.rados.0.smithi045.stdout:3641: done (0 left) 2024-04-26T00:40:39.279 INFO:tasks.rados.rados.0.smithi045.stdout:3642: snap_remove snap 395 2024-04-26T00:40:39.280 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.277+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.280 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.277+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1757638321}) 2024-04-26T00:40:39.280 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.277+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2250566801}) 2024-04-26T00:40:39.280 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.277+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.280 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.277+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1757638321}) 2024-04-26T00:40:39.280 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.277+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3977486329}) 2024-04-26T00:40:39.303 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.301+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.303 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.301+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=977855762}) 2024-04-26T00:40:39.303 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.301+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2805912304}) 2024-04-26T00:40:39.303 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.301+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.303 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.301+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=977855762}) 2024-04-26T00:40:39.303 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.301+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3812506382}) 2024-04-26T00:40:39.306 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.305+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.306 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.305+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4161537294}) 2024-04-26T00:40:39.306 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.305+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2359266981}) 2024-04-26T00:40:39.307 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.305+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.307 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.305+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4161537294}) 2024-04-26T00:40:39.307 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.305+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1907278293}) 2024-04-26T00:40:39.358 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.357+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.358 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.357+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1738049971}) 2024-04-26T00:40:39.358 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.357+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3428857004}) 2024-04-26T00:40:39.358 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.357+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.358 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.357+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1738049971}) 2024-04-26T00:40:39.358 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.357+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3370088931}) 2024-04-26T00:40:39.503 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.501+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.503 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.501+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3804543198}) 2024-04-26T00:40:39.504 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.501+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3499069534}) 2024-04-26T00:40:39.504 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.501+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.504 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.501+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3804543198}) 2024-04-26T00:40:39.504 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.501+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=7908214}) 2024-04-26T00:40:39.549 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.549+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.549 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.549+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3312238054}) 2024-04-26T00:40:39.549 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.549+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2837783688}) 2024-04-26T00:40:39.549 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.549+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.550 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.549+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3312238054}) 2024-04-26T00:40:39.550 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.549+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=969958872}) 2024-04-26T00:40:39.551 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.549+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.551 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.549+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=804207461}) 2024-04-26T00:40:39.551 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.549+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=307269072}) 2024-04-26T00:40:39.552 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.549+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.552 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.549+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=804207461}) 2024-04-26T00:40:39.552 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.549+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1323569173}) 2024-04-26T00:40:39.554 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.553+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.554 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.553+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1202923044}) 2024-04-26T00:40:39.554 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.553+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1759875763}) 2024-04-26T00:40:39.554 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.553+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.554 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.553+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1202923044}) 2024-04-26T00:40:39.554 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.553+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1905972042}) 2024-04-26T00:40:39.782 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.781+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.783 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.781+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3099577590}) 2024-04-26T00:40:39.783 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.781+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3521692251}) 2024-04-26T00:40:39.783 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.781+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.783 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.781+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3099577590}) 2024-04-26T00:40:39.783 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.781+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3173480240}) 2024-04-26T00:40:39.792 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.789+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.792 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.789+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1593727637}) 2024-04-26T00:40:39.793 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.789+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=771919431}) 2024-04-26T00:40:39.793 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.789+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.793 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.789+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1593727637}) 2024-04-26T00:40:39.793 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.789+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1351633995}) 2024-04-26T00:40:39.794 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.793+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.795 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.793+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4190818043}) 2024-04-26T00:40:39.795 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.793+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4084268236}) 2024-04-26T00:40:39.795 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.793+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.795 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.793+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4190818043}) 2024-04-26T00:40:39.795 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.793+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2546866642}) 2024-04-26T00:40:39.796 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.793+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.797 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.793+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2945172852}) 2024-04-26T00:40:39.797 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.793+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=318173802}) 2024-04-26T00:40:39.797 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.793+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.797 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.793+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2945172852}) 2024-04-26T00:40:39.797 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.793+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3001818384}) 2024-04-26T00:40:39.799 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.797+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.799 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.797+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1754212734}) 2024-04-26T00:40:39.799 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.797+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=406524557}) 2024-04-26T00:40:39.799 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.797+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.799 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.797+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1754212734}) 2024-04-26T00:40:39.799 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.797+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4166581637}) 2024-04-26T00:40:39.801 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.797+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.801 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.797+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4044994199}) 2024-04-26T00:40:39.801 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.801+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=685754622}) 2024-04-26T00:40:39.801 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.801+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.801 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.801+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4044994199}) 2024-04-26T00:40:39.801 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.801+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3958898667}) 2024-04-26T00:40:39.803 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.801+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.803 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.801+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1073828256}) 2024-04-26T00:40:39.803 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.801+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4055858460}) 2024-04-26T00:40:39.803 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.801+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.803 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.801+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1073828256}) 2024-04-26T00:40:39.804 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.801+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4273605884}) 2024-04-26T00:40:39.805 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.805+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.805 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.805+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1952200517}) 2024-04-26T00:40:39.805 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.805+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2396968614}) 2024-04-26T00:40:39.805 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.805+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.806 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.805+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1952200517}) 2024-04-26T00:40:39.806 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.805+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2952002853}) 2024-04-26T00:40:39.850 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.849+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.850 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.849+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2649854074}) 2024-04-26T00:40:39.850 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.849+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=683393370}) 2024-04-26T00:40:39.850 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.849+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.850 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.849+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2649854074}) 2024-04-26T00:40:39.850 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.849+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3816088697}) 2024-04-26T00:40:39.852 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.849+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.852 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.849+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2606051016}) 2024-04-26T00:40:39.852 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.849+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1234675420}) 2024-04-26T00:40:39.852 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.849+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.852 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.849+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2606051016}) 2024-04-26T00:40:39.852 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.849+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4016554790}) 2024-04-26T00:40:39.854 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.853+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.854 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.853+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3716292166}) 2024-04-26T00:40:39.854 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.853+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3663100140}) 2024-04-26T00:40:39.855 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.853+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.855 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.853+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3716292166}) 2024-04-26T00:40:39.855 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.853+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4110087865}) 2024-04-26T00:40:39.857 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.853+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.857 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.853+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3454277529}) 2024-04-26T00:40:39.857 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.853+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3748067913}) 2024-04-26T00:40:39.857 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.853+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.857 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.853+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3454277529}) 2024-04-26T00:40:39.857 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.853+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1950265018}) 2024-04-26T00:40:39.859 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.857+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.859 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.857+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1836857569}) 2024-04-26T00:40:39.859 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.857+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=676551229}) 2024-04-26T00:40:39.859 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.857+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.859 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.857+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1836857569}) 2024-04-26T00:40:39.859 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.857+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2380242826}) 2024-04-26T00:40:39.861 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.857+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.861 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.857+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=79761802}) 2024-04-26T00:40:39.861 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.857+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4044065794}) 2024-04-26T00:40:39.861 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.861+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.861 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.861+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=79761802}) 2024-04-26T00:40:39.861 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.861+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=660674101}) 2024-04-26T00:40:39.863 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.861+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.863 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.861+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1018387958}) 2024-04-26T00:40:39.863 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.861+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2790137649}) 2024-04-26T00:40:39.863 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.861+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.864 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.861+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1018387958}) 2024-04-26T00:40:39.864 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.861+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2753565854}) 2024-04-26T00:40:39.865 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.865+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.865 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.865+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3977921390}) 2024-04-26T00:40:39.866 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.865+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1418631270}) 2024-04-26T00:40:39.866 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.865+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.866 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.865+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3977921390}) 2024-04-26T00:40:39.866 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.865+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=530678855}) 2024-04-26T00:40:39.868 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.865+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.868 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.865+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1250241930}) 2024-04-26T00:40:39.868 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.865+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=679473475}) 2024-04-26T00:40:39.868 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.865+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.868 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.865+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1250241930}) 2024-04-26T00:40:39.868 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.865+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3132667936}) 2024-04-26T00:40:39.870 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.869+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.870 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.869+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3665965978}) 2024-04-26T00:40:39.870 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.869+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1392437687}) 2024-04-26T00:40:39.870 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.869+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.870 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.869+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3665965978}) 2024-04-26T00:40:39.870 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.869+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2918848290}) 2024-04-26T00:40:39.872 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.869+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.872 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.869+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2630145942}) 2024-04-26T00:40:39.872 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.869+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3622284528}) 2024-04-26T00:40:39.872 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.869+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.872 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.869+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2630145942}) 2024-04-26T00:40:39.873 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.869+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=848471842}) 2024-04-26T00:40:39.874 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.873+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.874 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.873+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3101065360}) 2024-04-26T00:40:39.875 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.873+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3099368641}) 2024-04-26T00:40:39.875 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.873+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.875 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.873+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3101065360}) 2024-04-26T00:40:39.875 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.873+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3355430678}) 2024-04-26T00:40:39.876 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.873+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.877 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.873+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4174714261}) 2024-04-26T00:40:39.877 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.873+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=996542609}) 2024-04-26T00:40:39.877 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.873+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.877 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.873+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4174714261}) 2024-04-26T00:40:39.877 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.873+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1551503067}) 2024-04-26T00:40:39.879 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.877+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.879 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.877+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1736694467}) 2024-04-26T00:40:39.879 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.877+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3199586929}) 2024-04-26T00:40:39.879 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.877+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.879 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.877+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1736694467}) 2024-04-26T00:40:39.879 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.877+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1416163664}) 2024-04-26T00:40:39.881 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.881+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.881 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.881+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1305414540}) 2024-04-26T00:40:39.881 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.881+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1929909091}) 2024-04-26T00:40:39.881 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.881+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.881 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.881+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1305414540}) 2024-04-26T00:40:39.882 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.881+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4267988084}) 2024-04-26T00:40:39.883 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.881+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.883 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.881+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2350514272}) 2024-04-26T00:40:39.883 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.881+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3287269153}) 2024-04-26T00:40:39.883 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.881+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.883 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.881+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2350514272}) 2024-04-26T00:40:39.884 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.881+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2432934595}) 2024-04-26T00:40:39.885 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.885+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.886 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.885+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3931627688}) 2024-04-26T00:40:39.886 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.885+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2182128711}) 2024-04-26T00:40:39.886 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.885+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.886 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.885+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3931627688}) 2024-04-26T00:40:39.886 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.885+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1858731176}) 2024-04-26T00:40:39.888 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.885+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.888 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.885+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1470147229}) 2024-04-26T00:40:39.888 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.885+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3956877872}) 2024-04-26T00:40:39.888 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.885+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.888 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.885+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1470147229}) 2024-04-26T00:40:39.888 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.885+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1542272271}) 2024-04-26T00:40:39.890 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.889+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.890 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.889+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2428438112}) 2024-04-26T00:40:39.891 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.889+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=183911953}) 2024-04-26T00:40:39.891 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.889+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.891 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.889+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2428438112}) 2024-04-26T00:40:39.891 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.889+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=48859144}) 2024-04-26T00:40:39.892 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.889+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.892 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.889+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2745965179}) 2024-04-26T00:40:39.892 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.889+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=418453546}) 2024-04-26T00:40:39.892 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.889+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.892 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.889+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2745965179}) 2024-04-26T00:40:39.893 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.889+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2503883638}) 2024-04-26T00:40:39.895 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.893+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.895 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.893+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1594925167}) 2024-04-26T00:40:39.895 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.893+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2115374762}) 2024-04-26T00:40:39.895 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.893+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.895 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.893+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1594925167}) 2024-04-26T00:40:39.895 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.893+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=120267212}) 2024-04-26T00:40:39.897 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.897+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.897 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.897+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2564133246}) 2024-04-26T00:40:39.897 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.897+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1171429034}) 2024-04-26T00:40:39.897 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.897+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.897 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.897+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2564133246}) 2024-04-26T00:40:39.897 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.897+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2032518566}) 2024-04-26T00:40:39.899 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.897+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.899 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.897+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=121979777}) 2024-04-26T00:40:39.900 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.897+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=989539859}) 2024-04-26T00:40:39.900 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.897+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.900 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.897+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=121979777}) 2024-04-26T00:40:39.900 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.897+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=917189693}) 2024-04-26T00:40:39.902 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.901+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.902 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.901+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1183665367}) 2024-04-26T00:40:39.902 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.901+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3918077882}) 2024-04-26T00:40:39.902 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.901+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.902 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.901+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1183665367}) 2024-04-26T00:40:39.902 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.901+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3996048062}) 2024-04-26T00:40:39.904 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.901+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.904 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.901+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4131388457}) 2024-04-26T00:40:39.904 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.901+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2706825589}) 2024-04-26T00:40:39.904 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.901+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.904 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.901+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4131388457}) 2024-04-26T00:40:39.904 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.901+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1714794973}) 2024-04-26T00:40:39.906 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.905+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.906 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.905+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=387680600}) 2024-04-26T00:40:39.906 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.905+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1412900135}) 2024-04-26T00:40:39.907 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.905+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.907 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.905+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=387680600}) 2024-04-26T00:40:39.907 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.905+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1101027097}) 2024-04-26T00:40:39.909 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.905+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.909 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.905+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=919374559}) 2024-04-26T00:40:39.909 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.905+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1878791324}) 2024-04-26T00:40:39.909 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.905+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.909 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.909+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=919374559}) 2024-04-26T00:40:39.909 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.909+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1620994796}) 2024-04-26T00:40:39.911 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.909+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.911 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.909+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=547748645}) 2024-04-26T00:40:39.911 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.909+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=72038327}) 2024-04-26T00:40:39.912 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.909+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.912 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.909+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=547748645}) 2024-04-26T00:40:39.912 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.909+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2703282400}) 2024-04-26T00:40:39.914 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.913+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.914 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.913+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1480559797}) 2024-04-26T00:40:39.914 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.913+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1626842936}) 2024-04-26T00:40:39.915 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.913+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.915 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.913+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1480559797}) 2024-04-26T00:40:39.915 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.913+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1467279519}) 2024-04-26T00:40:39.917 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.913+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.917 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.913+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=262644871}) 2024-04-26T00:40:39.917 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.913+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1719827058}) 2024-04-26T00:40:39.917 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.913+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.917 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.913+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=262644871}) 2024-04-26T00:40:39.917 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.913+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=686567086}) 2024-04-26T00:40:39.919 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.917+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.919 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.917+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2137407184}) 2024-04-26T00:40:39.920 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.917+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1909163634}) 2024-04-26T00:40:39.920 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.917+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.920 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.917+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2137407184}) 2024-04-26T00:40:39.920 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.917+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4248073928}) 2024-04-26T00:40:39.922 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.921+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.922 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.921+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3445198479}) 2024-04-26T00:40:39.922 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.921+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3864783082}) 2024-04-26T00:40:39.922 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.921+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:39.922 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.921+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3445198479}) 2024-04-26T00:40:39.922 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:39.921+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=51070124}) 2024-04-26T00:40:40.218 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:40:40.219 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:40:40.266 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.265+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:40.267 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.265+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=493804708}) 2024-04-26T00:40:40.267 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.265+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2463542780}) 2024-04-26T00:40:40.267 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.265+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:40.267 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.265+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=493804708}) 2024-04-26T00:40:40.267 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.265+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=605807986}) 2024-04-26T00:40:40.291 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.289+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:40.291 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.289+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=737468469}) 2024-04-26T00:40:40.291 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.289+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4029116263}) 2024-04-26T00:40:40.291 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.289+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:40.291 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.289+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=737468469}) 2024-04-26T00:40:40.291 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.289+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=692508710}) 2024-04-26T00:40:40.297 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.293+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:40.297 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.293+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=440081209}) 2024-04-26T00:40:40.297 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.293+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=455749309}) 2024-04-26T00:40:40.297 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.293+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:40.297 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.293+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=440081209}) 2024-04-26T00:40:40.297 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.293+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2335145249}) 2024-04-26T00:40:40.304 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.301+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:40.304 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.301+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2884037178}) 2024-04-26T00:40:40.304 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.301+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=86962407}) 2024-04-26T00:40:40.304 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.301+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:40.304 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.301+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2884037178}) 2024-04-26T00:40:40.304 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.301+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2293600088}) 2024-04-26T00:40:40.305 INFO:tasks.rados.rados.0.smithi045.stdout:3642: done (0 left) 2024-04-26T00:40:40.305 INFO:tasks.rados.rados.0.smithi045.stdout:3643: delete oid 3 current snap is 403 2024-04-26T00:40:40.316 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.313+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:40.317 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.313+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2818693956}) 2024-04-26T00:40:40.317 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.313+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4247601414}) 2024-04-26T00:40:40.317 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.313+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:40.317 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.313+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2818693956}) 2024-04-26T00:40:40.317 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.313+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4063869775}) 2024-04-26T00:40:40.330 INFO:tasks.rados.rados.0.smithi045.stdout:3643: done (0 left) 2024-04-26T00:40:40.330 INFO:tasks.rados.rados.0.smithi045.stdout:3644: copy_from oid 40 from oid 19 current snap is 403 2024-04-26T00:40:40.330 INFO:tasks.rados.rados.0.smithi045.stdout:3645: snap_remove snap 402 2024-04-26T00:40:40.331 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.329+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:40.331 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.329+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3875837788}) 2024-04-26T00:40:40.332 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.329+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3216401169}) 2024-04-26T00:40:40.332 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.329+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:40.332 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.329+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3875837788}) 2024-04-26T00:40:40.332 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.329+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3737668085}) 2024-04-26T00:40:40.336 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.333+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:40.336 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.333+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1777543255}) 2024-04-26T00:40:40.336 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.333+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1806073086}) 2024-04-26T00:40:40.336 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.333+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:40.336 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.333+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1777543255}) 2024-04-26T00:40:40.336 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.333+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4225675128}) 2024-04-26T00:40:40.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.361+0000 7f8a3b574640 10 monclient: get_monmap_and_config 2024-04-26T00:40:40.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.361+0000 7f8a3b574640 10 monclient: build_initial_monmap 2024-04-26T00:40:40.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.361+0000 7f8a3b574640 10 monclient: monmap: 2024-04-26T00:40:40.365 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:40:40.365 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:40.365 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:40:40.367432+0000 2024-04-26T00:40:40.365 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:40:40.367432+0000 2024-04-26T00:40:40.365 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:40:40.365 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:40.365 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:40:40.366 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:40:40.366 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:40:40.366 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:40:40.366 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:40:40.366 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:40.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a3b574640 1 Processor -- start 2024-04-26T00:40:40.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a3b574640 1 -- start start 2024-04-26T00:40:40.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a3b574640 10 monclient: init 2024-04-26T00:40:40.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a3b574640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:40.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a3b574640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-04-26T00:40:40.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a3b574640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8a341351c0 0x7f8a34134c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:40.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a3b574640 10 monclient(hunting): picked mon.noname-c con 0x7f8a341351c0 addr v2:172.21.15.45:3300/0 2024-04-26T00:40:40.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a3b574640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a341355a0 0x7f8a340a1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:40.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a3b574640 10 monclient(hunting): picked mon.noname-b con 0x7f8a341355a0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:40:40.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a3b574640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f8a340a21b0 0x7f8a340a2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:40.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a3b574640 10 monclient(hunting): picked mon.noname-a con 0x7f8a340a21b0 addr v2:172.21.15.37:3300/0 2024-04-26T00:40:40.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a3b574640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:40.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a3b574640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f8a3406d310 con 0x7f8a340a21b0 2024-04-26T00:40:40.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a3b574640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:40.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a3b574640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f8a3406b1d0 con 0x7f8a341351c0 2024-04-26T00:40:40.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a3b574640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:40.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a3b574640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8a3409d8e0 con 0x7f8a341355a0 2024-04-26T00:40:40.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a3b574640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:40.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a3b574640 10 monclient(hunting): authenticate will time out at 2221.673176s 2024-04-26T00:40:40.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a39b11640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f8a340a21b0 0x7f8a340a2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:40.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a38b0f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a341355a0 0x7f8a340a1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:40.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a39b11640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f8a340a21b0 0x7f8a340a2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52092/0 (socket says 172.21.15.37:52092) 2024-04-26T00:40:40.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a38b0f640 10 monclient(hunting): get_auth_request con 0x7f8a341355a0 auth_method 0 2024-04-26T00:40:40.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a39b11640 1 -- 172.21.15.37:0/1700779896 learned_addr learned my addr 172.21.15.37:0/1700779896 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:40:40.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a38b0f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:40.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a38b0f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:40.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a38b0f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:40.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a39b11640 10 monclient(hunting): get_auth_request con 0x7f8a340a21b0 auth_method 0 2024-04-26T00:40:40.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a39b11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:40.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a39b11640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:40.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a39b11640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:40.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a39b11640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:40.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a39b11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:40.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a39b11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:40.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a39310640 1 --2- 172.21.15.37:0/1700779896 >> v2:172.21.15.45:3300/0 conn(0x7f8a341351c0 0x7f8a34134c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:40.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a38b0f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:40.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a38b0f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:40.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a38b0f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:40.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a39310640 10 monclient(hunting): get_auth_request con 0x7f8a341351c0 auth_method 0 2024-04-26T00:40:40.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a39310640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:40.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a39310640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:40.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a39310640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:40.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a39b11640 10 monclient(hunting): handle_auth_done global_id 244218 payload 274 2024-04-26T00:40:40.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a39b11640 1 -- 172.21.15.37:0/1700779896 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a341355a0 msgr2=0x7f8a340a1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:40:40.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a39b11640 1 --2- 172.21.15.37:0/1700779896 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a341355a0 0x7f8a340a1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:40.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a39b11640 1 -- 172.21.15.37:0/1700779896 >> v2:172.21.15.45:3300/0 conn(0x7f8a341351c0 msgr2=0x7f8a34134c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:40:40.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a39b11640 1 --2- 172.21.15.37:0/1700779896 >> v2:172.21.15.45:3300/0 conn(0x7f8a341351c0 0x7f8a34134c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:40.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a39b11640 10 monclient: _finish_hunting 0 2024-04-26T00:40:40.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a39b11640 1 monclient: found mon.noname-a 2024-04-26T00:40:40.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a39b11640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:40.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a39b11640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:40:40.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a39b11640 1 -- 172.21.15.37:0/1700779896 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a3406bb80 con 0x7f8a340a21b0 2024-04-26T00:40:40.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.365+0000 7f8a38b0f640 1 --2- 172.21.15.37:0/1700779896 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a341355a0 0x7f8a340a1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:40:40.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.369+0000 7f8a39b11640 1 -- 172.21.15.37:0/1700779896 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a340a21b0 msgr2=0x7f8a340a2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:40.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.369+0000 7f8a39b11640 1 --2- 172.21.15.37:0/1700779896 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a340a21b0 0x7f8a340a2600 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8a3409d320 tx=0x7f8a2c01c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f257280da6b4471e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:40.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.369+0000 7f8a2bfff640 1 -- 172.21.15.37:0/1700779896 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c0a5640 con 0x7f8a340a21b0 2024-04-26T00:40:40.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.369+0000 7f8a2bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:40.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.369+0000 7f8a2bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:40:40.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.369+0000 7f8a2bfff640 10 monclient: dump: 2024-04-26T00:40:40.376 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:40.376 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:40.376 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:40.376 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:40.376 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:40.377 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:40.377 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:40.377 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:40.377 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:40.377 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:40.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.369+0000 7f8a2bfff640 10 monclient: _finish_auth 0 2024-04-26T00:40:40.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.369+0000 7f8a2bfff640 10 monclient: _check_auth_tickets 2024-04-26T00:40:40.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.369+0000 7f8a2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:40.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.369+0000 7f8a2bfff640 1 -- 172.21.15.37:0/1700779896 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a2c0a57e0 con 0x7f8a340a21b0 2024-04-26T00:40:40.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.369+0000 7f8a2bfff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:40:40.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.369+0000 7f8a2bfff640 1 -- 172.21.15.37:0/1700779896 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c0a5bb0 con 0x7f8a340a21b0 2024-04-26T00:40:40.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.369+0000 7f8a3b574640 5 monclient: authenticate success, global_id 244218 2024-04-26T00:40:40.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.369+0000 7f8a3b574640 10 monclient: get_monmap_and_config success 2024-04-26T00:40:40.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.369+0000 7f8a3b574640 4 set_mon_vals no callback set 2024-04-26T00:40:40.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.369+0000 7f8a2bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:40.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.369+0000 7f8a2bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:40:40.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.369+0000 7f8a2bfff640 10 monclient: dump: 2024-04-26T00:40:40.378 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:40.378 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:40.378 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:40.378 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:40.378 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:40.378 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:40.378 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:40.379 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:40.379 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:40.379 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:40.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.369+0000 7f8a3b574640 10 monclient: shutdown 2024-04-26T00:40:40.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.369+0000 7f8a3b574640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:40:40.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.369+0000 7f8a3b574640 1 -- 172.21.15.37:0/1700779896 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a340a21b0 msgr2=0x7f8a340a2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:40.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.369+0000 7f8a3b574640 1 --2- 172.21.15.37:0/1700779896 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a340a21b0 0x7f8a340a2600 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8a3409d320 tx=0x7f8a2c01c8b0 comp rx=0 tx=0).stop 2024-04-26T00:40:40.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.369+0000 7f8a3b574640 1 -- 172.21.15.37:0/1700779896 shutdown_connections 2024-04-26T00:40:40.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.369+0000 7f8a3b574640 1 --2- 172.21.15.37:0/1700779896 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a340a21b0 0x7f8a340a2600 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:40.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.369+0000 7f8a3b574640 1 --2- 172.21.15.37:0/1700779896 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a341355a0 0x7f8a340a1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:40.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.369+0000 7f8a3b574640 1 --2- 172.21.15.37:0/1700779896 >> v2:172.21.15.45:3300/0 conn(0x7f8a341351c0 0x7f8a34134c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:40.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.369+0000 7f8a3b574640 1 -- 172.21.15.37:0/1700779896 >> 172.21.15.37:0/1700779896 conn(0x7f8a340a2d60 msgr2=0x7f8a340a3140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:40:40.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.369+0000 7f8a3b574640 1 -- 172.21.15.37:0/1700779896 shutdown_connections 2024-04-26T00:40:40.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.369+0000 7f8a3b574640 1 -- 172.21.15.37:0/1700779896 wait complete. 2024-04-26T00:40:40.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.369+0000 7f8a3b574640 10 monclient: build_initial_monmap 2024-04-26T00:40:40.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.369+0000 7f8a3b574640 10 monclient: monmap: 2024-04-26T00:40:40.380 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:40:40.380 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:40:40.380 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:40:40.380 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:40:40.380 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:40:40.380 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:40:40.380 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:40:40.380 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:40:40.380 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:40:40.380 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:40.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a3b574640 1 Processor -- start 2024-04-26T00:40:40.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a3b574640 1 -- start start 2024-04-26T00:40:40.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a3b574640 10 monclient: init 2024-04-26T00:40:40.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a3b574640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:40.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a3b574640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:40:40.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a3b574640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a340a21b0 0x7f8a341ced90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:40.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a3b574640 10 monclient(hunting): picked mon.noname-a con 0x7f8a340a21b0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:40:40.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a3b574640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a341351c0 0x7f8a34133ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:40.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a3b574640 10 monclient(hunting): picked mon.noname-b con 0x7f8a341351c0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:40:40.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a3b574640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a341355a0 0x7f8a3411d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:40.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a3b574640 10 monclient(hunting): picked mon.noname-c con 0x7f8a341355a0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:40:40.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a3b574640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:40.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a3b574640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3413eaf0 con 0x7f8a340a21b0 2024-04-26T00:40:40.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a3b574640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:40.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a3b574640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3413e970 con 0x7f8a341351c0 2024-04-26T00:40:40.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a3b574640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:40.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a3b574640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8a3413ec70 con 0x7f8a341355a0 2024-04-26T00:40:40.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a3b574640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:40.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a3b574640 10 monclient(hunting): authenticate will time out at 2221.680633s 2024-04-26T00:40:40.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a39b11640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a341355a0 0x7f8a3411d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:40.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a39310640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a340a21b0 0x7f8a341ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:40.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a39b11640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a341355a0 0x7f8a3411d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:44024/0 (socket says 172.21.15.37:44024) 2024-04-26T00:40:40.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a39b11640 1 -- 172.21.15.37:0/2170414155 learned_addr learned my addr 172.21.15.37:0/2170414155 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:40:40.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a39b11640 10 monclient(hunting): get_auth_request con 0x7f8a341355a0 auth_method 0 2024-04-26T00:40:40.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a39b11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:40.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a39b11640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:40.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a39b11640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:40.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a39310640 10 monclient(hunting): get_auth_request con 0x7f8a340a21b0 auth_method 0 2024-04-26T00:40:40.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a39310640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:40.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a39310640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:40.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a39310640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:40.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a39b11640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:40.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a39b11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:40.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a39b11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:40.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a39310640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:40.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a39310640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:40.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a39310640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:40.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a38b0f640 1 --2- 172.21.15.37:0/2170414155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a341351c0 0x7f8a34133ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:40.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a38b0f640 10 monclient(hunting): get_auth_request con 0x7f8a341351c0 auth_method 0 2024-04-26T00:40:40.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a38b0f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:40.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a38b0f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:40.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a38b0f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:40.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a39310640 10 monclient(hunting): handle_auth_done global_id 244221 payload 824 2024-04-26T00:40:40.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a39310640 1 -- 172.21.15.37:0/2170414155 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a341355a0 msgr2=0x7f8a3411d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:40.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a39310640 1 --2- 172.21.15.37:0/2170414155 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a341355a0 0x7f8a3411d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:40.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a39310640 1 -- 172.21.15.37:0/2170414155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a341351c0 msgr2=0x7f8a34133ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:40.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a39310640 1 --2- 172.21.15.37:0/2170414155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a341351c0 0x7f8a34133ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:40.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a39310640 10 monclient: _finish_hunting 0 2024-04-26T00:40:40.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a39310640 1 monclient: found mon.noname-a 2024-04-26T00:40:40.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a39310640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:40.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a39310640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:40:40.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a39310640 1 -- 172.21.15.37:0/2170414155 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a340a5b20 con 0x7f8a340a21b0 2024-04-26T00:40:40.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a39b11640 1 --2- 172.21.15.37:0/2170414155 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a341355a0 0x7f8a3411d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:40:40.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a39310640 1 -- 172.21.15.37:0/2170414155 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a340a21b0 msgr2=0x7f8a341ced90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:40.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a39310640 1 --2- 172.21.15.37:0/2170414155 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a340a21b0 0x7f8a341ced90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8a3000d5f0 tx=0x7f8a3000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:40.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a29ffb640 1 -- 172.21.15.37:0/2170414155 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a30018020 con 0x7f8a340a21b0 2024-04-26T00:40:40.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a29ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:40.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a29ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:40:40.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a29ffb640 10 monclient: dump: 2024-04-26T00:40:40.386 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:40.386 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:40.386 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:40.386 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:40.386 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:40.387 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:40.387 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:40.387 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:40.387 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:40.387 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:40.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a29ffb640 10 monclient: _finish_auth 0 2024-04-26T00:40:40.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a29ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:40:40.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a29ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:40.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a29ffb640 1 -- 172.21.15.37:0/2170414155 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a30014830 con 0x7f8a340a21b0 2024-04-26T00:40:40.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a29ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:40:40.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a29ffb640 1 -- 172.21.15.37:0/2170414155 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a30014c00 con 0x7f8a340a21b0 2024-04-26T00:40:40.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a29ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:40.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a29ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:40:40.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a29ffb640 10 monclient: dump: 2024-04-26T00:40:40.388 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:40.388 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:40.388 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:40.388 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:40.388 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:40.388 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:40.388 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:40.388 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:40.388 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:40.388 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:40.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a3b574640 5 monclient: authenticate success, global_id 244221 2024-04-26T00:40:40.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a2b7fe640 4 set_mon_vals no callback set 2024-04-26T00:40:40.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a3b574640 10 monclient: _renew_subs 2024-04-26T00:40:40.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a3b574640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:40:40.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.373+0000 7f8a3b574640 1 -- 172.21.15.37:0/2170414155 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a341211a0 con 0x7f8a340a21b0 2024-04-26T00:40:40.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.377+0000 7f8a3b574640 10 monclient: _renew_subs 2024-04-26T00:40:40.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.377+0000 7f8a3b574640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:40:40.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.377+0000 7f8a3b574640 1 -- 172.21.15.37:0/2170414155 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a341cf2d0 con 0x7f8a340a21b0 2024-04-26T00:40:40.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.377+0000 7f8a39310640 1 --2- 172.21.15.37:0/2170414155 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a340a21b0 0x7f8a341ced90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8a3000d5f0 tx=0x7f8a3000dab0 comp rx=0 tx=0).queue_received will delay after 1.71409e+09 on 0x7f8a30048f80 osd_map(822..822 src has 705..822) 2024-04-26T00:40:40.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.377+0000 7f8a29ffb640 1 -- 172.21.15.37:0/2170414155 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f8a3001b020 con 0x7f8a340a21b0 2024-04-26T00:40:40.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.377+0000 7f8a3b574640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:40.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.377+0000 7f8a3b574640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:40.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.377+0000 7f8a3b574640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:40:40.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.377+0000 7f8a3b574640 1 -- 172.21.15.37:0/2170414155 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a34072f50 con 0x7f8a340a21b0 2024-04-26T00:40:40.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.377+0000 7f8a29ffb640 1 --2- 172.21.15.37:0/2170414155 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8a18035e80 0x7f8a18038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:40.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.377+0000 7f8a38b0f640 1 --2- 172.21.15.37:0/2170414155 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8a18035e80 0x7f8a18038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:40.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.377+0000 7f8a38b0f640 10 monclient: get_auth_request con 0x7f8a18035e80 auth_method 0 2024-04-26T00:40:40.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.381+0000 7f8a38b0f640 1 --2- 172.21.15.37:0/2170414155 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8a18035e80 0x7f8a18038340 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f8a341213b0 tx=0x7f8a24008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:40.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:40.381+0000 7f8a29ffb640 1 -- 172.21.15.37:0/2170414155 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(822..822 src has 705..822) ==== 4007+0+0 (secure 0 0 0) 0x7f8a30048f80 con 0x7f8a340a21b0 2024-04-26T00:40:40.601 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.601+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:40.601 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.601+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1686052984}) 2024-04-26T00:40:40.602 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.601+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=404288713}) 2024-04-26T00:40:40.602 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.601+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:40.602 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.601+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1686052984}) 2024-04-26T00:40:40.602 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:40.601+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2643040153}) 2024-04-26T00:40:41.106 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.101+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:41.106 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.101+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1543605449}) 2024-04-26T00:40:41.106 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.105+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=51776454}) 2024-04-26T00:40:41.106 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.105+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:41.106 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.105+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1543605449}) 2024-04-26T00:40:41.107 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.105+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2439217667}) 2024-04-26T00:40:41.118 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.117+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:41.118 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.117+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2859239851}) 2024-04-26T00:40:41.119 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.117+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=263914905}) 2024-04-26T00:40:41.119 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.117+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:41.119 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.117+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2859239851}) 2024-04-26T00:40:41.119 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.117+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3404597306}) 2024-04-26T00:40:41.121 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.117+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:41.121 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.117+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3846730051}) 2024-04-26T00:40:41.121 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.117+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2221928863}) 2024-04-26T00:40:41.121 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.117+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:41.121 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.117+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3846730051}) 2024-04-26T00:40:41.122 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.117+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2061608979}) 2024-04-26T00:40:41.123 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.121+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:41.123 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.121+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1667946105}) 2024-04-26T00:40:41.123 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.121+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=894835296}) 2024-04-26T00:40:41.123 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.121+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:41.124 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.121+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1667946105}) 2024-04-26T00:40:41.124 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.121+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2601548785}) 2024-04-26T00:40:41.126 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.121+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:41.126 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.121+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3773172202}) 2024-04-26T00:40:41.126 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.121+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2633486311}) 2024-04-26T00:40:41.126 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.121+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:41.126 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.121+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3773172202}) 2024-04-26T00:40:41.127 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.121+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=575636571}) 2024-04-26T00:40:41.127 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.125+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:41.127 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.125+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1804257936}) 2024-04-26T00:40:41.128 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.125+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3280683840}) 2024-04-26T00:40:41.128 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.125+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:41.128 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.125+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1804257936}) 2024-04-26T00:40:41.128 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.125+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1163588276}) 2024-04-26T00:40:41.129 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.129+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:41.129 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.129+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3340911598}) 2024-04-26T00:40:41.130 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.129+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2184378145}) 2024-04-26T00:40:41.130 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.129+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:41.130 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.129+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3340911598}) 2024-04-26T00:40:41.130 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.129+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=610440265}) 2024-04-26T00:40:41.132 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.129+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:41.132 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.129+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2032130151}) 2024-04-26T00:40:41.132 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.129+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2062314263}) 2024-04-26T00:40:41.132 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.129+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:41.132 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.129+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2032130151}) 2024-04-26T00:40:41.132 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.129+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2827587182}) 2024-04-26T00:40:41.134 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.133+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:41.134 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.133+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=250077821}) 2024-04-26T00:40:41.134 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.133+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2749273370}) 2024-04-26T00:40:41.134 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.133+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:41.134 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.133+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=250077821}) 2024-04-26T00:40:41.135 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.133+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=666241767}) 2024-04-26T00:40:41.137 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.133+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:41.137 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.133+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=160926461}) 2024-04-26T00:40:41.137 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.133+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=882031886}) 2024-04-26T00:40:41.137 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.133+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:41.137 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.133+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=160926461}) 2024-04-26T00:40:41.137 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.133+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4212007068}) 2024-04-26T00:40:41.138 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.137+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:41.138 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.137+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=327083562}) 2024-04-26T00:40:41.138 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.137+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1290142437}) 2024-04-26T00:40:41.138 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.137+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:41.139 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.137+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=327083562}) 2024-04-26T00:40:41.139 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.137+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3792642113}) 2024-04-26T00:40:41.140 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.137+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:41.140 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.137+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=994928710}) 2024-04-26T00:40:41.140 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.137+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2061207075}) 2024-04-26T00:40:41.140 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.137+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:41.140 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.137+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=994928710}) 2024-04-26T00:40:41.140 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.137+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1698061244}) 2024-04-26T00:40:41.141 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.141+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:41.141 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.141+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4248149797}) 2024-04-26T00:40:41.142 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.141+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2736023686}) 2024-04-26T00:40:41.142 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.141+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:41.142 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.141+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4248149797}) 2024-04-26T00:40:41.142 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.141+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3454886983}) 2024-04-26T00:40:41.143 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.141+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:41.143 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.141+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1333871306}) 2024-04-26T00:40:41.143 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.141+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1897627833}) 2024-04-26T00:40:41.143 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.141+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:41.143 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.141+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1333871306}) 2024-04-26T00:40:41.143 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.141+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2232267223}) 2024-04-26T00:40:41.146 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.145+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:41.146 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.145+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=33,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=23} crc {logm=895002539,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=1878003279,mon_config_key=4061063117,monmap=491605398,osd_metadata=124267790,osd_pg_creating=2858142071,osd_snap=3277615088}) 2024-04-26T00:40:41.146 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.145+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=30,mdsmap=2,mgr=6,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29} crc {logm=2413806973,mdsmap=1401145748,mgr=693576549,mgr_metadata=3686404438,mgrstat=1878003279,mon_config_key=4061063117,monmap=3876866194,osd_metadata=3306121364,osd_pg_creating=2858142071,osd_snap=2083421460}) 2024-04-26T00:40:41.146 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.145+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:41.146 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.145+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=33,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=23} crc {logm=895002539,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=1878003279,mon_config_key=4061063117,monmap=491605398,osd_metadata=124267790,osd_pg_creating=2858142071,osd_snap=3277615088}) 2024-04-26T00:40:41.146 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.145+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=33,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=8,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=26} crc {logm=1139090819,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=1878003279,mon_config_key=1139944024,monmap=3876866194,osd_metadata=2387245979,osd_pg_creating=2858142071,osd_snap=2068058148}) 2024-04-26T00:40:41.148 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.145+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:41.148 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.145+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=100} crc {osd_snap=2350608074}) 2024-04-26T00:40:41.148 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.145+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=100} crc {osd_snap=2331507553}) 2024-04-26T00:40:41.148 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.145+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:41.149 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.145+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=100} crc {osd_snap=2350608074}) 2024-04-26T00:40:41.149 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.145+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=100} crc {osd_snap=1622971053}) 2024-04-26T00:40:41.151 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.149+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:41.151 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.149+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=33,osdmap=67} crc {osd_snap=2358130857,osdmap=3480470905}) 2024-04-26T00:40:41.151 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.149+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=32,osdmap=68} crc {osd_snap=2970524806,osdmap=2284855851}) 2024-04-26T00:40:41.151 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.149+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:41.151 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.149+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=33,osdmap=67} crc {osd_snap=2358130857,osdmap=3480470905}) 2024-04-26T00:40:41.151 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.149+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=35,osdmap=65} crc {osd_snap=1240829855,osdmap=2325987539}) 2024-04-26T00:40:41.154 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.153+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:41.155 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.153+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1148528723}) 2024-04-26T00:40:41.155 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.153+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2352649702}) 2024-04-26T00:40:41.155 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.153+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:41.155 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.153+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1148528723}) 2024-04-26T00:40:41.155 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.153+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2159173419}) 2024-04-26T00:40:41.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.205+0000 7f8a29ffb640 1 -- 172.21.15.37:0/2170414155 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f8a30022020 con 0x7f8a340a21b0 2024-04-26T00:40:41.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.205+0000 7f8a29ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:41.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.205+0000 7f8a29ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:40:41.274 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.273+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:41.274 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.273+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=19} crc {osdmap=985437907}) 2024-04-26T00:40:41.274 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.273+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=16} crc {osdmap=440963444}) 2024-04-26T00:40:41.274 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.273+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:40:41.274 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.273+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=19} crc {osdmap=985437907}) 2024-04-26T00:40:41.274 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T00:40:41.273+0000 7fe4defdb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=18} crc {osdmap=2305545922}) 2024-04-26T00:40:41.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.441+0000 7f8a3b574640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:40:41.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.441+0000 7f8a3b574640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:40:41.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.441+0000 7f8a3b574640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:40:41.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.441+0000 7f8a3b574640 1 -- 172.21.15.37:0/2170414155 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8a34066f40 con 0x7f8a340a21b0 2024-04-26T00:40:41.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.441+0000 7f8a29ffb640 1 -- 172.21.15.37:0/2170414155 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f8a3001c3b0 con 0x7f8a340a21b0 2024-04-26T00:40:41.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.441+0000 7f8a29ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:40:41.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.441+0000 7f8a29ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:40:41.445 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":152,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:40:41.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.445+0000 7f8a3b574640 1 -- 172.21.15.37:0/2170414155 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8a18035e80 msgr2=0x7f8a18038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:41.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.445+0000 7f8a3b574640 1 --2- 172.21.15.37:0/2170414155 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8a18035e80 0x7f8a18038340 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f8a341213b0 tx=0x7f8a24008040 comp rx=0 tx=0).stop 2024-04-26T00:40:41.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.445+0000 7f8a3b574640 10 monclient: shutdown 2024-04-26T00:40:41.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.445+0000 7f8a3b574640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:40:41.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.445+0000 7f8a3b574640 1 -- 172.21.15.37:0/2170414155 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a340a21b0 msgr2=0x7f8a341ced90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:41.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.445+0000 7f8a3b574640 1 --2- 172.21.15.37:0/2170414155 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a340a21b0 0x7f8a341ced90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8a3000d5f0 tx=0x7f8a3000dab0 comp rx=0 tx=0).stop 2024-04-26T00:40:41.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.445+0000 7f8a3b574640 1 -- 172.21.15.37:0/2170414155 shutdown_connections 2024-04-26T00:40:41.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.445+0000 7f8a3b574640 1 --2- 172.21.15.37:0/2170414155 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8a18035e80 0x7f8a18038340 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:41.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.445+0000 7f8a3b574640 1 --2- 172.21.15.37:0/2170414155 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8a341355a0 0x7f8a3411d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:41.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.445+0000 7f8a3b574640 1 --2- 172.21.15.37:0/2170414155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a341351c0 0x7f8a34133ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:41.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.445+0000 7f8a3b574640 1 --2- 172.21.15.37:0/2170414155 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a340a21b0 0x7f8a341ced90 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:41.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.445+0000 7f8a3b574640 1 -- 172.21.15.37:0/2170414155 >> 172.21.15.37:0/2170414155 conn(0x7f8a340a2d60 msgr2=0x7f8a340a52c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:40:41.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.445+0000 7f8a3b574640 1 -- 172.21.15.37:0/2170414155 shutdown_connections 2024-04-26T00:40:41.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.445+0000 7f8a3b574640 1 -- 172.21.15.37:0/2170414155 wait complete. 2024-04-26T00:40:41.464 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:40:41.464 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-26T00:40:41.464 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:40:41.464 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:40:41.530 INFO:tasks.rados.rados.0.smithi045.stdout:3645: done (1 left) 2024-04-26T00:40:41.531 INFO:tasks.rados.rados.0.smithi045.stdout:3646: write (excl) oid 30 current snap is 403 2024-04-26T00:40:41.531 INFO:tasks.rados.rados.0.smithi045.stdout:3646: seq_num 1297 ranges {598679=577963,1947263=638108,3351151=447528} 2024-04-26T00:40:41.540 INFO:tasks.rados.rados.0.smithi045.stdout:3646: writing smithi04526579-30 from 598679 to 1176642 tid 1 2024-04-26T00:40:41.552 INFO:tasks.rados.rados.0.smithi045.stdout:3646: writing smithi04526579-30 from 1947263 to 2585371 tid 2 2024-04-26T00:40:41.563 INFO:tasks.rados.rados.0.smithi045.stdout:3646: writing smithi04526579-30 from 3351151 to 3798679 tid 3 2024-04-26T00:40:41.563 INFO:tasks.rados.rados.0.smithi045.stdout:3647: write (excl) oid 27 current snap is 403 2024-04-26T00:40:41.563 INFO:tasks.rados.rados.0.smithi045.stdout:3644: finishing copy_from racing read to smithi04526579-40 2024-04-26T00:40:41.563 INFO:tasks.rados.rados.0.smithi045.stdout:3644: finishing copy_from to smithi04526579-40 2024-04-26T00:40:41.563 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 2476 (ObjNum 1239 snap 384 seq_num 1239) dirty exists 2024-04-26T00:40:41.563 INFO:tasks.rados.rados.0.smithi045.stdout:3646: finishing write tid 1 to smithi04526579-30 2024-04-26T00:40:41.563 INFO:tasks.rados.rados.0.smithi045.stdout:3646: finishing write tid 2 to smithi04526579-30 2024-04-26T00:40:41.563 INFO:tasks.rados.rados.0.smithi045.stdout:3647: seq_num 1298 ranges {615486=653212,1889392=498119,2978574=475308} 2024-04-26T00:40:41.574 INFO:tasks.rados.rados.0.smithi045.stdout:3647: writing smithi04526579-27 from 615486 to 1268698 tid 1 2024-04-26T00:40:41.585 INFO:tasks.rados.rados.0.smithi045.stdout:3647: writing smithi04526579-27 from 1889392 to 2387511 tid 2 2024-04-26T00:40:41.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.589+0000 7f0fca12c640 10 monclient: get_monmap_and_config 2024-04-26T00:40:41.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.589+0000 7f0fca12c640 10 monclient: build_initial_monmap 2024-04-26T00:40:41.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.589+0000 7f0fca12c640 10 monclient: monmap: 2024-04-26T00:40:41.593 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:40:41.593 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:41.593 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:40:41.595358+0000 2024-04-26T00:40:41.593 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:40:41.595358+0000 2024-04-26T00:40:41.593 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:40:41.593 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:41.594 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:40:41.594 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:40:41.594 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:40:41.594 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:40:41.594 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:40:41.594 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:41.596 INFO:tasks.rados.rados.0.smithi045.stdout:3647: writing smithi04526579-27 from 2978574 to 3453882 tid 3 2024-04-26T00:40:41.596 INFO:tasks.rados.rados.0.smithi045.stdout:3644: done (2 left) 2024-04-26T00:40:41.596 INFO:tasks.rados.rados.0.smithi045.stdout:3648: write (excl) oid 41 current snap is 403 2024-04-26T00:40:41.596 INFO:tasks.rados.rados.0.smithi045.stdout:3646: finishing write tid 3 to smithi04526579-30 2024-04-26T00:40:41.596 INFO:tasks.rados.rados.0.smithi045.stdout:3648: seq_num 1299 ranges {632293=728461,1947874=641777,3005997=686580} 2024-04-26T00:40:41.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.593+0000 7f0fca12c640 1 Processor -- start 2024-04-26T00:40:41.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.593+0000 7f0fca12c640 1 -- start start 2024-04-26T00:40:41.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.593+0000 7f0fca12c640 10 monclient: init 2024-04-26T00:40:41.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.593+0000 7f0fca12c640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:41.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.593+0000 7f0fca12c640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-04-26T00:40:41.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.593+0000 7f0fca12c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0fc40a2fe0 addr v1:172.21.15.37:6789/0 2024-04-26T00:40:41.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.593+0000 7f0fca12c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0fc4072570 0x7f0fc4072980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:41.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.593+0000 7f0fca12c640 10 monclient(hunting): picked mon.noname-c con 0x7f0fc4072570 addr v2:172.21.15.45:3300/0 2024-04-26T00:40:41.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.593+0000 7f0fca12c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0fc413bfd0 addr v1:172.21.15.45:6789/0 2024-04-26T00:40:41.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.593+0000 7f0fca12c640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fc409f010 con 0x7f0fc40a2fe0 2024-04-26T00:40:41.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.593+0000 7f0fca12c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fc406e780 con 0x7f0fc413bfd0 2024-04-26T00:40:41.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.593+0000 7f0fca12c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:41.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.593+0000 7f0fca12c640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f0fc406ea60 con 0x7f0fc4072570 2024-04-26T00:40:41.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.593+0000 7f0fca12c640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:41.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.593+0000 7f0fca12c640 10 monclient(hunting): authenticate will time out at 2222.901435s 2024-04-26T00:40:41.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.593+0000 7f0fc37fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f0fc40a2fe0 0x7f0fc4133930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:48768/0 (socket says 172.21.15.37:48768) 2024-04-26T00:40:41.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.593+0000 7f0fc37fe640 1 -- 172.21.15.37:0/1995493191 learned_addr learned my addr 172.21.15.37:0/1995493191 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:40:41.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.593+0000 7f0fc2ffd640 1 --2- 172.21.15.37:0/1995493191 >> v2:172.21.15.45:3300/0 conn(0x7f0fc4072570 0x7f0fc4072980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:41.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.593+0000 7f0fc2ffd640 10 monclient(hunting): get_auth_request con 0x7f0fc4072570 auth_method 0 2024-04-26T00:40:41.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.593+0000 7f0fc2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:41.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.593+0000 7f0fc2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:41.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.593+0000 7f0fc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:41.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.593+0000 7f0fc2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:41.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.593+0000 7f0fc2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:41.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.593+0000 7f0fc2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:41.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.593+0000 7f0fc2ffd640 10 monclient(hunting): handle_auth_done global_id 224371 payload 274 2024-04-26T00:40:41.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.593+0000 7f0fc2ffd640 1 -- 172.21.15.37:0/1995493191 >> v1:172.21.15.45:6789/0 conn(0x7f0fc413bfd0 legacy=0x7f0fc413e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:40:41.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.593+0000 7f0fc2ffd640 1 -- 172.21.15.37:0/1995493191 >> v1:172.21.15.37:6789/0 conn(0x7f0fc40a2fe0 legacy=0x7f0fc4133930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:40:41.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.593+0000 7f0fc2ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:40:41.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.593+0000 7f0fc2ffd640 1 monclient: found mon.noname-c 2024-04-26T00:40:41.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.593+0000 7f0fc2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:41.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.593+0000 7f0fc2ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:40:41.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.593+0000 7f0fc2ffd640 1 -- 172.21.15.37:0/1995493191 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fc406ed40 con 0x7f0fc4072570 2024-04-26T00:40:41.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.597+0000 7f0fc2ffd640 1 -- 172.21.15.37:0/1995493191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fc4072570 msgr2=0x7f0fc4072980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:41.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.597+0000 7f0fc2ffd640 1 --2- 172.21.15.37:0/1995493191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fc4072570 0x7f0fc4072980 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0fac0097c0 tx=0x7f0fac01c2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42a3a117febd23e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:41.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc27fc640 1 -- 172.21.15.37:0/1995493191 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fac0b0070 con 0x7f0fc4072570 2024-04-26T00:40:41.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:41.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:40:41.603 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc27fc640 10 monclient: dump: 2024-04-26T00:40:41.604 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:41.604 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:41.604 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:41.604 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:41.604 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:41.604 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:41.604 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:41.604 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:41.605 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:41.605 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:41.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc27fc640 10 monclient: _finish_auth 0 2024-04-26T00:40:41.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc27fc640 10 monclient: _check_auth_tickets 2024-04-26T00:40:41.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:41.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fca12c640 5 monclient: authenticate success, global_id 224371 2024-04-26T00:40:41.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fca12c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:40:41.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc27fc640 1 -- 172.21.15.37:0/1995493191 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fac0ab600 con 0x7f0fc4072570 2024-04-26T00:40:41.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc27fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:40:41.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc27fc640 1 -- 172.21.15.37:0/1995493191 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fac0ab9d0 con 0x7f0fc4072570 2024-04-26T00:40:41.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fca12c640 10 monclient: get_monmap_and_config success 2024-04-26T00:40:41.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fca12c640 4 set_mon_vals no callback set 2024-04-26T00:40:41.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:41.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:40:41.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc27fc640 10 monclient: dump: 2024-04-26T00:40:41.606 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:41.606 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:41.606 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:41.606 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:41.606 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:41.606 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:41.606 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:41.606 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:41.607 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:41.607 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:41.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fca12c640 10 monclient: shutdown 2024-04-26T00:40:41.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fca12c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:40:41.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fca12c640 1 -- 172.21.15.37:0/1995493191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fc4072570 msgr2=0x7f0fc4072980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:41.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fca12c640 1 --2- 172.21.15.37:0/1995493191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fc4072570 0x7f0fc4072980 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0fac0097c0 tx=0x7f0fac01c2e0 comp rx=0 tx=0).stop 2024-04-26T00:40:41.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fca12c640 1 -- 172.21.15.37:0/1995493191 shutdown_connections 2024-04-26T00:40:41.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fca12c640 1 --2- 172.21.15.37:0/1995493191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fc4072570 0x7f0fc4072980 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:41.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fca12c640 1 -- 172.21.15.37:0/1995493191 >> 172.21.15.37:0/1995493191 conn(0x7f0fc40a24c0 msgr2=0x7f0fc40a28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:40:41.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fca12c640 1 -- 172.21.15.37:0/1995493191 shutdown_connections 2024-04-26T00:40:41.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fca12c640 1 -- 172.21.15.37:0/1995493191 wait complete. 2024-04-26T00:40:41.608 INFO:tasks.rados.rados.0.smithi045.stdout:3648: writing smithi04526579-41 from 632293 to 1360754 tid 1 2024-04-26T00:40:41.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fca12c640 10 monclient: build_initial_monmap 2024-04-26T00:40:41.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fca12c640 10 monclient: monmap: 2024-04-26T00:40:41.615 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:40:41.615 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:40:41.615 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:40:41.615 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:40:41.616 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:40:41.616 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:40:41.616 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:40:41.616 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:40:41.616 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:40:41.616 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:41.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fca12c640 1 Processor -- start 2024-04-26T00:40:41.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fca12c640 1 -- start start 2024-04-26T00:40:41.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fca12c640 10 monclient: init 2024-04-26T00:40:41.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fca12c640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:41.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fca12c640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:40:41.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fca12c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0fc4072570 0x7f0fc4130800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:41.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fca12c640 10 monclient(hunting): picked mon.noname-a con 0x7f0fc4072570 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:40:41.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fca12c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fc40a2fe0 0x7f0fc4135590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:41.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fca12c640 10 monclient(hunting): picked mon.noname-b con 0x7f0fc40a2fe0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:40:41.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fca12c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0fc413bfd0 0x7f0fc4135ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:41.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fca12c640 10 monclient(hunting): picked mon.noname-c con 0x7f0fc413bfd0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:40:41.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fca12c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:41.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fca12c640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc4142aa0 con 0x7f0fc4072570 2024-04-26T00:40:41.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fca12c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:41.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fca12c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc4142920 con 0x7f0fc40a2fe0 2024-04-26T00:40:41.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fca12c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:41.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fca12c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0fc4142c20 con 0x7f0fc413bfd0 2024-04-26T00:40:41.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fca12c640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:41.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fca12c640 10 monclient(hunting): authenticate will time out at 2222.910888s 2024-04-26T00:40:41.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc37fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0fc4072570 0x7f0fc4130800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:41.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc3fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0fc413bfd0 0x7f0fc4135ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:41.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc37fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0fc4072570 0x7f0fc4130800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52126/0 (socket says 172.21.15.37:52126) 2024-04-26T00:40:41.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc37fe640 1 -- 172.21.15.37:0/2826500305 learned_addr learned my addr 172.21.15.37:0/2826500305 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:40:41.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc37fe640 10 monclient(hunting): get_auth_request con 0x7f0fc4072570 auth_method 0 2024-04-26T00:40:41.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:41.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:41.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:41.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc3fff640 10 monclient(hunting): get_auth_request con 0x7f0fc413bfd0 auth_method 0 2024-04-26T00:40:41.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:41.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:41.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:41.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:41.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:41.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:41.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:41.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:41.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc2ffd640 1 --2- 172.21.15.37:0/2826500305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fc40a2fe0 0x7f0fc4135590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:41.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:41.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc2ffd640 10 monclient(hunting): get_auth_request con 0x7f0fc40a2fe0 auth_method 0 2024-04-26T00:40:41.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:41.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:41.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:41.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc3fff640 10 monclient(hunting): handle_auth_done global_id 254135 payload 824 2024-04-26T00:40:41.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc3fff640 1 -- 172.21.15.37:0/2826500305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fc40a2fe0 msgr2=0x7f0fc4135590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:41.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc3fff640 1 --2- 172.21.15.37:0/2826500305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fc40a2fe0 0x7f0fc4135590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:41.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc3fff640 1 -- 172.21.15.37:0/2826500305 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0fc4072570 msgr2=0x7f0fc4130800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:41.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc3fff640 1 --2- 172.21.15.37:0/2826500305 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0fc4072570 0x7f0fc4130800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:41.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc3fff640 10 monclient: _finish_hunting 0 2024-04-26T00:40:41.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc3fff640 1 monclient: found mon.noname-c 2024-04-26T00:40:41.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:41.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:40:41.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc3fff640 1 -- 172.21.15.37:0/2826500305 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fc4139480 con 0x7f0fc413bfd0 2024-04-26T00:40:41.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc2ffd640 1 --2- 172.21.15.37:0/2826500305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0fc40a2fe0 0x7f0fc4135590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:40:41.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.601+0000 7f0fc37fe640 1 --2- 172.21.15.37:0/2826500305 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0fc4072570 0x7f0fc4130800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:40:41.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fc3fff640 1 -- 172.21.15.37:0/2826500305 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0fc413bfd0 msgr2=0x7f0fc4135ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:41.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fc3fff640 1 --2- 172.21.15.37:0/2826500305 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0fc413bfd0 0x7f0fc4135ad0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc00c780 tx=0x7f0fbc00cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:41.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fbbfff640 1 -- 172.21.15.37:0/2826500305 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc016020 con 0x7f0fc413bfd0 2024-04-26T00:40:41.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fbbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:41.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fbbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:40:41.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fbbfff640 10 monclient: dump: 2024-04-26T00:40:41.621 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:41.621 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:41.621 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:41.621 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:41.621 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:41.621 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:41.621 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:41.621 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:41.621 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:41.621 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:41.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fbbfff640 10 monclient: _finish_auth 0 2024-04-26T00:40:41.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fbbfff640 10 monclient: _check_auth_tickets 2024-04-26T00:40:41.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fbbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:41.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fca12c640 5 monclient: authenticate success, global_id 254135 2024-04-26T00:40:41.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fca12c640 10 monclient: _renew_subs 2024-04-26T00:40:41.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fbbfff640 1 -- 172.21.15.37:0/2826500305 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fbc00ac60 con 0x7f0fc413bfd0 2024-04-26T00:40:41.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fca12c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:40:41.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fca12c640 1 -- 172.21.15.37:0/2826500305 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fc4137190 con 0x7f0fc413bfd0 2024-04-26T00:40:41.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fbbfff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:40:41.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fbbfff640 1 -- 172.21.15.37:0/2826500305 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc012660 con 0x7f0fc413bfd0 2024-04-26T00:40:41.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fbbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:41.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fbbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:40:41.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fc1ffb640 4 set_mon_vals no callback set 2024-04-26T00:40:41.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fbbfff640 10 monclient: dump: 2024-04-26T00:40:41.623 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:41.623 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:41.623 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:41.623 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:41.623 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:41.623 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:41.623 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:41.623 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:41.623 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:41.623 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:41.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fca12c640 10 monclient: _renew_subs 2024-04-26T00:40:41.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fca12c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:40:41.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fca12c640 1 -- 172.21.15.37:0/2826500305 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fc4139e80 con 0x7f0fc413bfd0 2024-04-26T00:40:41.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fca12c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:41.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fca12c640 1 --2- 172.21.15.37:0/2826500305 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0fc4130d40 0x7f0fc4136010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:41.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fca12c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:41.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fca12c640 1 -- 172.21.15.37:0/2826500305 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc409fd30 con 0x7f0fc4130d40 2024-04-26T00:40:41.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fc37fe640 1 --2- 172.21.15.37:0/2826500305 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0fc4130d40 0x7f0fc4136010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:41.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fc37fe640 10 monclient: get_auth_request con 0x7f0fc4130d40 auth_method 0 2024-04-26T00:40:41.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fc37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:41.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fc37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:41.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:41.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fc37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:41.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fc37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:41.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fc37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:41.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fbbfff640 1 -- 172.21.15.37:0/2826500305 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f0fbc019020 con 0x7f0fc413bfd0 2024-04-26T00:40:41.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fc37fe640 10 monclient(hunting): handle_auth_done global_id 244227 payload 824 2024-04-26T00:40:41.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fc37fe640 1 -- 172.21.15.37:0/2826500305 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0fc404f510 con 0x7f0fc4130d40 2024-04-26T00:40:41.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fbbfff640 1 --2- 172.21.15.37:0/2826500305 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0fa4035f00 0x7f0fa40383c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:41.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fbbfff640 1 -- 172.21.15.37:0/2826500305 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(823..823 src has 705..823) ==== 4023+0+0 (secure 0 0 0) 0x7f0fbc048950 con 0x7f0fc413bfd0 2024-04-26T00:40:41.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fc2ffd640 1 --2- 172.21.15.37:0/2826500305 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0fa4035f00 0x7f0fa40383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:41.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.605+0000 7f0fc2ffd640 10 monclient: get_auth_request con 0x7f0fa4035f00 auth_method 0 2024-04-26T00:40:41.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.609+0000 7f0fc37fe640 1 -- 172.21.15.37:0/2826500305 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0fc4130d40 msgr2=0x7f0fc4136010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:41.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.609+0000 7f0fc37fe640 1 --2- 172.21.15.37:0/2826500305 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0fc4130d40 0x7f0fc4136010 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0fb40072a0 tx=0x7f0fb4007760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:41.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.609+0000 7f0fbbfff640 1 -- 172.21.15.37:0/2826500305 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb400fde0 con 0x7f0fc4130d40 2024-04-26T00:40:41.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.609+0000 7f0fbbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:41.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.609+0000 7f0fbbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:40:41.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.609+0000 7f0fbbfff640 10 monclient: dump: 2024-04-26T00:40:41.626 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:41.626 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:41.626 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:41.626 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:41.626 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:41.627 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:41.627 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:41.627 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:41.627 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:41.627 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:41.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.609+0000 7f0fbbfff640 1 -- 172.21.15.37:0/2826500305 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0fb400fde0 con 0x7f0fc4130d40 2024-04-26T00:40:41.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.609+0000 7f0fbbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:41.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.609+0000 7f0fbbfff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:40:41.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.609+0000 7f0fbbfff640 1 -- 172.21.15.37:0/2826500305 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0fc4130d40 msgr2=0x7f0fc4136010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:41.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.609+0000 7f0fbbfff640 1 --2- 172.21.15.37:0/2826500305 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0fc4130d40 0x7f0fc4136010 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0fb40072a0 tx=0x7f0fb4007760 comp rx=0 tx=0).stop 2024-04-26T00:40:41.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.609+0000 7f0fbbfff640 1 -- 172.21.15.37:0/2826500305 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0fc4130d40 msgr2=0x7f0fc4136010 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:40:41.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.609+0000 7f0fbbfff640 1 --2- 172.21.15.37:0/2826500305 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0fc4130d40 0x7f0fc4136010 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0fb40072a0 tx=0x7f0fb4007760 comp rx=0 tx=0).stop 2024-04-26T00:40:41.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.609+0000 7f0fc2ffd640 1 --2- 172.21.15.37:0/2826500305 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0fa4035f00 0x7f0fa40383c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f0fac0ad000 tx=0x7f0fac0a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:41.630 INFO:tasks.rados.rados.0.smithi045.stdout:3648: writing smithi04526579-41 from 1947874 to 2589651 tid 2 2024-04-26T00:40:41.630 INFO:tasks.rados.rados.0.smithi045.stdout:3648: writing smithi04526579-41 from 3005997 to 3692577 tid 3 2024-04-26T00:40:41.630 INFO:tasks.rados.rados.0.smithi045.stdout:3649: read oid 47 snap -1 2024-04-26T00:40:41.630 INFO:tasks.rados.rados.0.smithi045.stdout:3649: expect (ObjNum 1127 snap 340 seq_num 1127) 2024-04-26T00:40:41.630 INFO:tasks.rados.rados.0.smithi045.stdout:3650: copy_from oid 44 from oid 37 current snap is 403 2024-04-26T00:40:41.630 INFO:tasks.rados.rados.0.smithi045.stdout:3651: delete oid 15 current snap is 403 2024-04-26T00:40:41.630 INFO:tasks.rados.rados.0.smithi045.stdout:3646: finishing write tid 4 to smithi04526579-30 2024-04-26T00:40:41.630 INFO:tasks.rados.rados.0.smithi045.stdout:3646: finishing write tid 5 to smithi04526579-30 2024-04-26T00:40:41.630 INFO:tasks.rados.rados.0.smithi045.stdout:3646: oid 30 updating version 0 to 2461 2024-04-26T00:40:41.630 INFO:tasks.rados.rados.0.smithi045.stdout:3646: oid 30 version 2461 is already newer than 2459 2024-04-26T00:40:41.631 INFO:tasks.rados.rados.0.smithi045.stdout:3646: oid 30 version 2461 is already newer than 2460 2024-04-26T00:40:41.631 INFO:tasks.rados.rados.0.smithi045.stdout:3646: oid 30 version 2461 is already newer than 2458 2024-04-26T00:40:41.631 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 2461 (ObjNum 1297 snap 403 seq_num 1297) dirty exists 2024-04-26T00:40:41.631 INFO:tasks.rados.rados.0.smithi045.stdout:3646: left oid 30 (ObjNum 1297 snap 403 seq_num 1297) 2024-04-26T00:40:41.631 INFO:tasks.rados.rados.0.smithi045.stdout:3647: finishing write tid 1 to smithi04526579-27 2024-04-26T00:40:41.631 INFO:tasks.rados.rados.0.smithi045.stdout:3647: finishing write tid 2 to smithi04526579-27 2024-04-26T00:40:41.631 INFO:tasks.rados.rados.0.smithi045.stdout:3647: finishing write tid 3 to smithi04526579-27 2024-04-26T00:40:41.631 INFO:tasks.rados.rados.0.smithi045.stdout:3647: finishing write tid 4 to smithi04526579-27 2024-04-26T00:40:41.631 INFO:tasks.rados.rados.0.smithi045.stdout:3647: finishing write tid 5 to smithi04526579-27 2024-04-26T00:40:41.631 INFO:tasks.rados.rados.0.smithi045.stdout:3647: oid 27 updating version 0 to 2512 2024-04-26T00:40:41.631 INFO:tasks.rados.rados.0.smithi045.stdout:3647: oid 27 version 2512 is already newer than 2509 2024-04-26T00:40:41.631 INFO:tasks.rados.rados.0.smithi045.stdout:3647: oid 27 version 2512 is already newer than 2511 2024-04-26T00:40:41.631 INFO:tasks.rados.rados.0.smithi045.stdout:3647: oid 27 version 2512 is already newer than 2510 2024-04-26T00:40:41.631 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 2512 (ObjNum 1298 snap 403 seq_num 1298) dirty exists 2024-04-26T00:40:41.632 INFO:tasks.rados.rados.0.smithi045.stdout:3647: left oid 27 (ObjNum 1298 snap 403 seq_num 1298) 2024-04-26T00:40:41.632 INFO:tasks.rados.rados.0.smithi045.stdout:3648: finishing write tid 1 to smithi04526579-41 2024-04-26T00:40:41.632 INFO:tasks.rados.rados.0.smithi045.stdout:3648: finishing write tid 2 to smithi04526579-41 2024-04-26T00:40:41.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.617+0000 7f0fca12c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:40:41.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.617+0000 7f0fca12c640 1 --2- 172.21.15.37:0/2826500305 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0fc4136550 0x7f0fc413c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:41.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.617+0000 7f0fca12c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:41.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.617+0000 7f0fca12c640 1 -- 172.21.15.37:0/2826500305 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc4136960 con 0x7f0fc4136550 2024-04-26T00:40:41.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.617+0000 7f0fc37fe640 1 --2- 172.21.15.37:0/2826500305 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0fc4136550 0x7f0fc413c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:41.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.617+0000 7f0fc37fe640 10 monclient: get_auth_request con 0x7f0fc4136550 auth_method 0 2024-04-26T00:40:41.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.617+0000 7f0fc37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:41.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.617+0000 7f0fc37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:41.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.617+0000 7f0fc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:41.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.617+0000 7f0fc37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:41.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.617+0000 7f0fc37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:41.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.617+0000 7f0fc37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:41.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.617+0000 7f0fc37fe640 10 monclient(hunting): handle_auth_done global_id 244230 payload 824 2024-04-26T00:40:41.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.617+0000 7f0fc37fe640 1 -- 172.21.15.37:0/2826500305 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0fc4133260 con 0x7f0fc4136550 2024-04-26T00:40:41.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.617+0000 7f0fc37fe640 1 -- 172.21.15.37:0/2826500305 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0fc4136550 msgr2=0x7f0fc413c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:41.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.617+0000 7f0fc37fe640 1 --2- 172.21.15.37:0/2826500305 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0fc4136550 0x7f0fc413c3b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4017000 tx=0x7f0fb4004950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:41.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.617+0000 7f0fbbfff640 1 -- 172.21.15.37:0/2826500305 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb401ee40 con 0x7f0fc4136550 2024-04-26T00:40:41.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.617+0000 7f0fbbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:41.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.617+0000 7f0fbbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:40:41.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.617+0000 7f0fbbfff640 10 monclient: dump: 2024-04-26T00:40:41.634 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:41.634 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:41.634 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:41.634 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:41.634 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:41.635 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:41.635 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:41.635 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:41.635 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:41.635 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:41.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.617+0000 7f0fbbfff640 1 -- 172.21.15.37:0/2826500305 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4719 (secure 0 0 0) 0x7f0fb4006240 con 0x7f0fc4136550 2024-04-26T00:40:41.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.617+0000 7f0fbbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:40:41.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.617+0000 7f0fbbfff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:40:41.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.617+0000 7f0fbbfff640 1 -- 172.21.15.37:0/2826500305 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0fc4136550 msgr2=0x7f0fc413c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:41.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.617+0000 7f0fbbfff640 1 --2- 172.21.15.37:0/2826500305 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0fc4136550 0x7f0fc413c3b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4017000 tx=0x7f0fb4004950 comp rx=0 tx=0).stop 2024-04-26T00:40:41.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.617+0000 7f0fbbfff640 1 -- 172.21.15.37:0/2826500305 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0fc4136550 msgr2=0x7f0fc413c3b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:40:41.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.617+0000 7f0fbbfff640 1 --2- 172.21.15.37:0/2826500305 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0fc4136550 0x7f0fc413c3b0 secure :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4017000 tx=0x7f0fb4004950 comp rx=0 tx=0).stop 2024-04-26T00:40:41.636 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:40:41.636 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:40:41.636 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:40:41.636 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:40:41.636 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 152, 2024-04-26T00:40:41.636 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 58975, 2024-04-26T00:40:41.636 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:40:41.636 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:40:41.636 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:40:41.636 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:40:41.636 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:41.636 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 5, 2024-04-26T00:40:41.637 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:40:41.637 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:40:41.637 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:40:41.637 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:40:41.637 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:40:41.637 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:40:41.637 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:40:41.637 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:40:41.637 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:40:41.637 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:40:41.637 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:40:41.637 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:40:41.637 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:40:41.637 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:40:41.638 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:41.638 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:40:41.638 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:40:41.638 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:40:41.638 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:40:41.638 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:40:41.638 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:40:41.638 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:40:41.638 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:40:41.638 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:40:41.638 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:40:41.638 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:40:41.638 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:40:41.639 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:40:41.639 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:41.639 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:41.639 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:40:41.639 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:40:41.639 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:40:41.639 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:40:41.639 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:40:41.639 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:40:41.639 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:40:41.639 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:40:41.639 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:40:41.639 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:40:41.639 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:40:41.640 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:40:41.640 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:40:41.640 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:40:41.640 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:40:41.640 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:40:41.640 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:40:41.640 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:40:41.640 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:40:41.640 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:40:41.640 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:40:41.640 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:40:41.640 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:40:41.640 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:40:41.640 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:40:41.640 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:40:41.641 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:40:41.641 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:40:41.641 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:41.641 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:40:41.641 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:41.641 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:40:41.641 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:41.641 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:40:41.641 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:40:41.641 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:40:41.641 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:40:41.641 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:41.641 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:40:41.641 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:40:41.642 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:41.642 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:41.642 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:41.642 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:40:41.642 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:40:41.642 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:41.642 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:41.642 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:41.642 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:41.642 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:40:41.642 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:40:41.642 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:40:41.642 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:40:41.643 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:40:41.643 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:41.643 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:41.643 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:40:41.643 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:40:41.643 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:40:41.643 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:40:41.643 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:41.643 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:40:41.643 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:40:41.643 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:41.644 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:41.644 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:41.644 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:40:41.644 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:40:41.644 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:41.644 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:41.644 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:41.644 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:41.644 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:40:41.644 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:40:41.644 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:40:41.644 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:40:41.644 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:40:41.644 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:41.645 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:41.645 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:40:41.645 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:40:41.645 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:40:41.645 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:40:41.645 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:41.645 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:40:41.645 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:40:41.645 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:41.645 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:41.645 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:41.645 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:40:41.645 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:40:41.645 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:41.646 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:41.646 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:41.646 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:41.646 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:40:41.646 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:40:41.646 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:40:41.646 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:40:41.646 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:40:41.646 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:41.646 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:41.646 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:41.646 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:40:41.646 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:40:41.646 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:41.647 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:40:41.647 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:40:41.647 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:40:41.647 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:41.647 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:41.647 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:40:41.647 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:41.647 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:40:41.647 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:40:41.647 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:40:41.647 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:41.647 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:41.647 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:40:41.647 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:41.647 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:40:41.648 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:40:41.648 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:40:41.648 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:41.648 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:41.648 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:41.648 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:40:41.648 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:40:41.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.617+0000 7f0fca12c640 1 -- 172.21.15.37:0/2826500305 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0fa4035f00 msgr2=0x7f0fa40383c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:41.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.617+0000 7f0fca12c640 1 --2- 172.21.15.37:0/2826500305 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0fa4035f00 0x7f0fa40383c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f0fac0ad000 tx=0x7f0fac0a6000 comp rx=0 tx=0).stop 2024-04-26T00:40:41.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.617+0000 7f0fca12c640 10 monclient: shutdown 2024-04-26T00:40:41.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.617+0000 7f0fca12c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:40:41.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.617+0000 7f0fca12c640 1 -- 172.21.15.37:0/2826500305 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0fc413bfd0 msgr2=0x7f0fc4135ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:41.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.617+0000 7f0fca12c640 1 --2- 172.21.15.37:0/2826500305 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0fc413bfd0 0x7f0fc4135ad0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc00c780 tx=0x7f0fbc00cc40 comp rx=0 tx=0).stop 2024-04-26T00:40:41.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.617+0000 7f0fc3fff640 1 -- 172.21.15.37:0/2826500305 reap_dead start 2024-04-26T00:40:41.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.617+0000 7f0fc3fff640 1 -- 172.21.15.37:0/2826500305 reap_dead start 2024-04-26T00:40:41.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.617+0000 7f0fca12c640 1 -- 172.21.15.37:0/2826500305 shutdown_connections 2024-04-26T00:40:41.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.617+0000 7f0fca12c640 1 -- 172.21.15.37:0/2826500305 >> 172.21.15.37:0/2826500305 conn(0x7f0fc40a24c0 msgr2=0x7f0fc40a0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:40:41.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.617+0000 7f0fca12c640 1 -- 172.21.15.37:0/2826500305 shutdown_connections 2024-04-26T00:40:41.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.617+0000 7f0fca12c640 1 -- 172.21.15.37:0/2826500305 wait complete. 2024-04-26T00:40:41.651 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:40:41.651 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:40:41.651 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:40:41.652 INFO:tasks.rados.rados.0.smithi045.stdout:3650: finishing copy_from racing read to smithi04526579-44 2024-04-26T00:40:41.652 INFO:tasks.rados.rados.0.smithi045.stdout:3650: finishing copy_from to smithi04526579-44 2024-04-26T00:40:41.652 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 2398 (ObjNum 1075 snap 322 seq_num 1075) dirty exists 2024-04-26T00:40:41.652 INFO:tasks.rados.rados.0.smithi045.stdout:3648: finishing write tid 3 to smithi04526579-41 2024-04-26T00:40:41.653 INFO:tasks.rados.rados.0.smithi045.stdout:3648: finishing write tid 4 to smithi04526579-41 2024-04-26T00:40:41.653 INFO:tasks.rados.rados.0.smithi045.stdout:3648: finishing write tid 5 to smithi04526579-41 2024-04-26T00:40:41.653 INFO:tasks.rados.rados.0.smithi045.stdout:3648: oid 41 updating version 0 to 2516 2024-04-26T00:40:41.653 INFO:tasks.rados.rados.0.smithi045.stdout:3648: oid 41 version 2516 is already newer than 2514 2024-04-26T00:40:41.653 INFO:tasks.rados.rados.0.smithi045.stdout:3648: oid 41 version 2516 is already newer than 2515 2024-04-26T00:40:41.653 INFO:tasks.rados.rados.0.smithi045.stdout:3648: oid 41 updating version 2516 to 2517 2024-04-26T00:40:41.653 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 2517 (ObjNum 1299 snap 403 seq_num 1299) dirty exists 2024-04-26T00:40:41.653 INFO:tasks.rados.rados.0.smithi045.stdout:3648: left oid 41 (ObjNum 1299 snap 403 seq_num 1299) 2024-04-26T00:40:41.653 INFO:tasks.rados.rados.0.smithi045.stdout:3649: expect (ObjNum 1127 snap 340 seq_num 1127) 2024-04-26T00:40:41.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f423419b640 10 monclient: get_monmap_and_config 2024-04-26T00:40:41.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f423419b640 10 monclient: build_initial_monmap 2024-04-26T00:40:41.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f423419b640 10 monclient: monmap: 2024-04-26T00:40:41.774 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:40:41.774 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:41.774 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:40:41.776191+0000 2024-04-26T00:40:41.774 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:40:41.776191+0000 2024-04-26T00:40:41.774 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:40:41.774 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:41.774 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:40:41.774 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:40:41.774 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:40:41.774 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:40:41.774 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:40:41.774 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:41.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f423419b640 1 Processor -- start 2024-04-26T00:40:41.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f423419b640 1 -- start start 2024-04-26T00:40:41.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f423419b640 10 monclient: init 2024-04-26T00:40:41.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f423419b640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:41.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f423419b640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-04-26T00:40:41.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f423419b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f422c072180 0x7f422c072560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:41.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f423419b640 10 monclient(hunting): picked mon.noname-c con 0x7f422c072180 addr v2:172.21.15.45:3300/0 2024-04-26T00:40:41.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f423419b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f422c071670 0x7f422c0a13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:41.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f423419b640 10 monclient(hunting): picked mon.noname-b con 0x7f422c071670 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:40:41.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f423419b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f422c071a80 addr v1:172.21.15.37:6789/0 2024-04-26T00:40:41.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f423419b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f422c070580 con 0x7f422c071a80 2024-04-26T00:40:41.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f423419b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:41.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f423419b640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f422c0702a0 con 0x7f422c072180 2024-04-26T00:40:41.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f423419b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:41.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f423419b640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f422c070860 con 0x7f422c071670 2024-04-26T00:40:41.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f423419b640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:41.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f423419b640 10 monclient(hunting): authenticate will time out at 2223.081947s 2024-04-26T00:40:41.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f4232738640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f422c071a80 0x7f422c0a1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:48770/0 (socket says 172.21.15.37:48770) 2024-04-26T00:40:41.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f4231736640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f422c071670 0x7f422c0a13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:41.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f4232738640 1 -- 172.21.15.37:0/3596046723 learned_addr learned my addr 172.21.15.37:0/3596046723 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:40:41.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f4231736640 10 monclient(hunting): get_auth_request con 0x7f422c071670 auth_method 0 2024-04-26T00:40:41.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f4231736640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:41.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f4231736640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:41.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f4231736640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:41.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f4231736640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:41.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f4231736640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:41.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f4231f37640 1 --2- 172.21.15.37:0/3596046723 >> v2:172.21.15.45:3300/0 conn(0x7f422c072180 0x7f422c072560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:41.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f4231736640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:41.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f4231f37640 10 monclient(hunting): get_auth_request con 0x7f422c072180 auth_method 0 2024-04-26T00:40:41.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f4231f37640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:41.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f4231f37640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:41.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f4231f37640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:41.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f4231736640 10 monclient(hunting): handle_auth_done global_id 254138 payload 274 2024-04-26T00:40:41.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f4231736640 1 -- 172.21.15.37:0/3596046723 >> v2:172.21.15.45:3300/0 conn(0x7f422c072180 msgr2=0x7f422c072560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:40:41.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f4231736640 1 --2- 172.21.15.37:0/3596046723 >> v2:172.21.15.45:3300/0 conn(0x7f422c072180 0x7f422c072560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:41.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f4231736640 1 -- 172.21.15.37:0/3596046723 >> v1:172.21.15.37:6789/0 conn(0x7f422c071a80 legacy=0x7f422c0a1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:40:41.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f4231736640 10 monclient: _finish_hunting 0 2024-04-26T00:40:41.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f4231736640 1 monclient: found mon.noname-b 2024-04-26T00:40:41.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f4231736640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:41.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f4231736640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:40:41.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.773+0000 7f4231736640 1 -- 172.21.15.37:0/3596046723 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f422c072aa0 con 0x7f422c071670 2024-04-26T00:40:41.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.777+0000 7f4231736640 1 -- 172.21.15.37:0/3596046723 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f422c071670 msgr2=0x7f422c0a13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:41.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.777+0000 7f4231736640 1 --2- 172.21.15.37:0/3596046723 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f422c071670 0x7f422c0a13d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f421c0097c0 tx=0x7f421c01c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cc8e89ee11d5c9c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:41.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.777+0000 7f4230f35640 1 -- 172.21.15.37:0/3596046723 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f421c0b0070 con 0x7f422c071670 2024-04-26T00:40:41.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.777+0000 7f4230f35640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:41.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.777+0000 7f4230f35640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:40:41.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.777+0000 7f4230f35640 10 monclient: dump: 2024-04-26T00:40:41.782 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:41.782 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:41.782 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:41.782 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:41.782 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:41.782 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:41.782 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:41.782 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:41.782 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:41.783 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:41.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.777+0000 7f4230f35640 10 monclient: _finish_auth 0 2024-04-26T00:40:41.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.777+0000 7f4230f35640 10 monclient: _check_auth_tickets 2024-04-26T00:40:41.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.777+0000 7f4230f35640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:41.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.777+0000 7f4230f35640 1 -- 172.21.15.37:0/3596046723 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f421c0ab600 con 0x7f422c071670 2024-04-26T00:40:41.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.777+0000 7f423419b640 5 monclient: authenticate success, global_id 254138 2024-04-26T00:40:41.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.777+0000 7f423419b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:40:41.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.777+0000 7f4230f35640 10 monclient: handle_config config(0 keys) 2024-04-26T00:40:41.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.777+0000 7f4230f35640 1 -- 172.21.15.37:0/3596046723 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f421c0ab9d0 con 0x7f422c071670 2024-04-26T00:40:41.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.777+0000 7f423419b640 10 monclient: get_monmap_and_config success 2024-04-26T00:40:41.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.777+0000 7f423419b640 4 set_mon_vals no callback set 2024-04-26T00:40:41.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.777+0000 7f4230f35640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:41.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.777+0000 7f4230f35640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:40:41.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.777+0000 7f4230f35640 10 monclient: dump: 2024-04-26T00:40:41.784 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:41.784 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:41.784 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:41.784 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:41.784 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:41.784 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:41.784 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:41.784 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:41.784 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:41.784 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:41.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.777+0000 7f423419b640 10 monclient: shutdown 2024-04-26T00:40:41.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.777+0000 7f423419b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:40:41.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.777+0000 7f423419b640 1 -- 172.21.15.37:0/3596046723 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f422c071670 msgr2=0x7f422c0a13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:41.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.777+0000 7f423419b640 1 --2- 172.21.15.37:0/3596046723 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f422c071670 0x7f422c0a13d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f421c0097c0 tx=0x7f421c01c440 comp rx=0 tx=0).stop 2024-04-26T00:40:41.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.777+0000 7f423419b640 1 -- 172.21.15.37:0/3596046723 shutdown_connections 2024-04-26T00:40:41.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.777+0000 7f423419b640 1 --2- 172.21.15.37:0/3596046723 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f422c071670 0x7f422c0a13d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:41.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.777+0000 7f423419b640 1 --2- 172.21.15.37:0/3596046723 >> v2:172.21.15.45:3300/0 conn(0x7f422c072180 0x7f422c072560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:41.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.777+0000 7f423419b640 1 -- 172.21.15.37:0/3596046723 >> 172.21.15.37:0/3596046723 conn(0x7f422c0a24c0 msgr2=0x7f422c0a28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:40:41.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.777+0000 7f423419b640 1 -- 172.21.15.37:0/3596046723 shutdown_connections 2024-04-26T00:40:41.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.777+0000 7f423419b640 1 -- 172.21.15.37:0/3596046723 wait complete. 2024-04-26T00:40:41.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f423419b640 10 monclient: build_initial_monmap 2024-04-26T00:40:41.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f423419b640 10 monclient: monmap: 2024-04-26T00:40:41.785 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:40:41.785 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:40:41.785 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:40:41.785 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:40:41.785 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:40:41.785 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:40:41.785 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:40:41.785 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:40:41.786 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:40:41.786 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:41.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f423419b640 1 Processor -- start 2024-04-26T00:40:41.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f423419b640 1 -- start start 2024-04-26T00:40:41.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f423419b640 10 monclient: init 2024-04-26T00:40:41.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f423419b640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:41.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f423419b640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:40:41.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f423419b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f422c071670 0x7f422c1caee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:41.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f423419b640 10 monclient(hunting): picked mon.noname-b con 0x7f422c071670 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:40:41.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f423419b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f422c071a80 0x7f422c1cb420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:41.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f423419b640 10 monclient(hunting): picked mon.noname-a con 0x7f422c071a80 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:40:41.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f423419b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f422c072180 0x7f422c1c7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:41.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f423419b640 10 monclient(hunting): picked mon.noname-c con 0x7f422c072180 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:40:41.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f423419b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:41.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f423419b640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f422c13b8b0 con 0x7f422c071a80 2024-04-26T00:40:41.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f423419b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:41.791 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f423419b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f422c13b730 con 0x7f422c071670 2024-04-26T00:40:41.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f423419b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:41.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f423419b640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f422c13ba30 con 0x7f422c072180 2024-04-26T00:40:41.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f423419b640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:41.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f423419b640 10 monclient(hunting): authenticate will time out at 2223.089896s 2024-04-26T00:40:41.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f4232738640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f422c072180 0x7f422c1c7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:41.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f4231736640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f422c071a80 0x7f422c1cb420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:41.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f4231736640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f422c071a80 0x7f422c1cb420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52146/0 (socket says 172.21.15.37:52146) 2024-04-26T00:40:41.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f4232738640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f422c072180 0x7f422c1c7430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:44062/0 (socket says 172.21.15.37:44062) 2024-04-26T00:40:41.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f4231736640 1 -- 172.21.15.37:0/439708414 learned_addr learned my addr 172.21.15.37:0/439708414 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:40:41.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f4231736640 10 monclient(hunting): get_auth_request con 0x7f422c071a80 auth_method 0 2024-04-26T00:40:41.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f4231736640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:41.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f4231736640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:41.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f4231736640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:41.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f4232738640 10 monclient(hunting): get_auth_request con 0x7f422c072180 auth_method 0 2024-04-26T00:40:41.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f4232738640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:41.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f4232738640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:41.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f4232738640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:41.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f4231f37640 1 --2- 172.21.15.37:0/439708414 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f422c071670 0x7f422c1caee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:41.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f4231736640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:41.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f4231736640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:41.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f4231736640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:41.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f4231f37640 10 monclient(hunting): get_auth_request con 0x7f422c071670 auth_method 0 2024-04-26T00:40:41.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f4231f37640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:41.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f4231f37640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:41.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f4231f37640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:41.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f4232738640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:41.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f4232738640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:41.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f4232738640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:41.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f4231f37640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:41.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f4231f37640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:41.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f4231f37640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:41.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f4231736640 10 monclient(hunting): handle_auth_done global_id 244233 payload 824 2024-04-26T00:40:41.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f4231736640 1 -- 172.21.15.37:0/439708414 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f422c072180 msgr2=0x7f422c1c7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:41.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f4231736640 1 --2- 172.21.15.37:0/439708414 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f422c072180 0x7f422c1c7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:41.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f4231736640 1 -- 172.21.15.37:0/439708414 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f422c071670 msgr2=0x7f422c1caee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:41.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f4231736640 1 --2- 172.21.15.37:0/439708414 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f422c071670 0x7f422c1caee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:41.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f4231736640 10 monclient: _finish_hunting 0 2024-04-26T00:40:41.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f4231736640 1 monclient: found mon.noname-a 2024-04-26T00:40:41.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f4231736640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:41.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f4231736640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:40:41.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.781+0000 7f4231736640 1 -- 172.21.15.37:0/439708414 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f422c072d80 con 0x7f422c071a80 2024-04-26T00:40:41.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f4231736640 1 -- 172.21.15.37:0/439708414 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f422c071a80 msgr2=0x7f422c1cb420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:41.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f4231736640 1 --2- 172.21.15.37:0/439708414 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f422c071a80 0x7f422c1cb420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f421c0ae040 tx=0x7f421c0191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:41.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f42227fc640 1 -- 172.21.15.37:0/439708414 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f421c0b8030 con 0x7f422c071a80 2024-04-26T00:40:41.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f42227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:41.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f42227fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:40:41.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f42227fc640 10 monclient: dump: 2024-04-26T00:40:41.796 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:41.796 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:41.796 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:41.796 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:41.796 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:41.796 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:41.796 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:41.796 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:41.796 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:41.796 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:41.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f42227fc640 10 monclient: _finish_auth 0 2024-04-26T00:40:41.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f42227fc640 10 monclient: _check_auth_tickets 2024-04-26T00:40:41.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f42227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:41.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f42227fc640 1 -- 172.21.15.37:0/439708414 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f421c0ab400 con 0x7f422c071a80 2024-04-26T00:40:41.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f42227fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:40:41.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f42227fc640 1 -- 172.21.15.37:0/439708414 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f421c0b0040 con 0x7f422c071a80 2024-04-26T00:40:41.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f42227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:41.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f42227fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:40:41.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f42227fc640 10 monclient: dump: 2024-04-26T00:40:41.797 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:41.797 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:41.797 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:41.797 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:41.797 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:41.798 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:41.798 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:41.798 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:41.798 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:41.798 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:41.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f423419b640 5 monclient: authenticate success, global_id 244233 2024-04-26T00:40:41.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f4223fff640 4 set_mon_vals no callback set 2024-04-26T00:40:41.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f423419b640 10 monclient: _renew_subs 2024-04-26T00:40:41.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f423419b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:40:41.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f423419b640 1 -- 172.21.15.37:0/439708414 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f422c13a1a0 con 0x7f422c071a80 2024-04-26T00:40:41.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f423419b640 10 monclient: _renew_subs 2024-04-26T00:40:41.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f423419b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:40:41.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f423419b640 1 -- 172.21.15.37:0/439708414 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f422c1ca990 con 0x7f422c071a80 2024-04-26T00:40:41.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f423419b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:41.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f423419b640 1 --2- 172.21.15.37:0/439708414 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f422c139be0 0x7f422c1c7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:41.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f423419b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:41.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f423419b640 1 -- 172.21.15.37:0/439708414 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f422c04f510 con 0x7f422c139be0 2024-04-26T00:40:41.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f4231f37640 1 --2- 172.21.15.37:0/439708414 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f422c139be0 0x7f422c1c7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:41.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f42227fc640 1 -- 172.21.15.37:0/439708414 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f421c0bb070 con 0x7f422c071a80 2024-04-26T00:40:41.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f4231f37640 10 monclient: get_auth_request con 0x7f422c139be0 auth_method 0 2024-04-26T00:40:41.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f4231f37640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:41.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f4231f37640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:41.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f4231f37640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:41.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f4231f37640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:41.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f4231f37640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:41.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f4231f37640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:41.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f42227fc640 1 --2- 172.21.15.37:0/439708414 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4210035f30 0x7f42100383f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:41.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f42227fc640 1 -- 172.21.15.37:0/439708414 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(823..823 src has 705..823) ==== 4023+0+0 (secure 0 0 0) 0x7f421c0e24b0 con 0x7f422c071a80 2024-04-26T00:40:41.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f4231f37640 10 monclient(hunting): handle_auth_done global_id 254144 payload 824 2024-04-26T00:40:41.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f4231f37640 1 -- 172.21.15.37:0/439708414 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f422c138a70 con 0x7f422c139be0 2024-04-26T00:40:41.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f4232738640 1 --2- 172.21.15.37:0/439708414 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4210035f30 0x7f42100383f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:41.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.785+0000 7f4232738640 10 monclient: get_auth_request con 0x7f4210035f30 auth_method 0 2024-04-26T00:40:41.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.789+0000 7f4231f37640 1 -- 172.21.15.37:0/439708414 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f422c139be0 msgr2=0x7f422c1c7970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:41.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.789+0000 7f4231f37640 1 --2- 172.21.15.37:0/439708414 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f422c139be0 0x7f422c1c7970 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4228010820 tx=0x7f4228010ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:41.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.789+0000 7f42227fc640 1 -- 172.21.15.37:0/439708414 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4228018020 con 0x7f422c139be0 2024-04-26T00:40:41.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.789+0000 7f42227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:41.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.789+0000 7f42227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:40:41.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.789+0000 7f42227fc640 10 monclient: dump: 2024-04-26T00:40:41.800 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:41.800 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:41.800 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:41.800 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:41.801 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:41.801 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:41.801 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:41.801 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:41.801 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:41.801 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:41.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.789+0000 7f42227fc640 1 -- 172.21.15.37:0/439708414 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4228004450 con 0x7f422c139be0 2024-04-26T00:40:41.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.789+0000 7f42227fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:41.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.789+0000 7f42227fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:40:41.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.789+0000 7f42227fc640 1 -- 172.21.15.37:0/439708414 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f422c139be0 msgr2=0x7f422c1c7970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:41.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.789+0000 7f42227fc640 1 --2- 172.21.15.37:0/439708414 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f422c139be0 0x7f422c1c7970 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4228010820 tx=0x7f4228010ce0 comp rx=0 tx=0).stop 2024-04-26T00:40:41.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.789+0000 7f42227fc640 1 -- 172.21.15.37:0/439708414 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f422c139be0 msgr2=0x7f422c1c7970 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:40:41.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.789+0000 7f42227fc640 1 --2- 172.21.15.37:0/439708414 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f422c139be0 0x7f422c1c7970 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4228010820 tx=0x7f4228010ce0 comp rx=0 tx=0).stop 2024-04-26T00:40:41.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.789+0000 7f4232738640 1 --2- 172.21.15.37:0/439708414 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4210035f30 0x7f42100383f0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f42240055b0 tx=0x7f4224005540 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:41.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.793+0000 7f423419b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:40:41.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.793+0000 7f423419b640 1 --2- 172.21.15.37:0/439708414 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f422c1c7eb0 0x7f422c09ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:41.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.793+0000 7f423419b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:41.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.793+0000 7f423419b640 1 -- 172.21.15.37:0/439708414 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f422c1c8290 con 0x7f422c1c7eb0 2024-04-26T00:40:41.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.793+0000 7f4231f37640 1 --2- 172.21.15.37:0/439708414 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f422c1c7eb0 0x7f422c09ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:41.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.793+0000 7f4231f37640 10 monclient: get_auth_request con 0x7f422c1c7eb0 auth_method 0 2024-04-26T00:40:41.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.793+0000 7f4231f37640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:41.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.793+0000 7f4231f37640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:41.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.793+0000 7f4231f37640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:41.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.793+0000 7f4231f37640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:41.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.793+0000 7f4231f37640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:41.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.793+0000 7f4231f37640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:41.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.793+0000 7f4231f37640 10 monclient(hunting): handle_auth_done global_id 254147 payload 824 2024-04-26T00:40:41.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.793+0000 7f4231f37640 1 -- 172.21.15.37:0/439708414 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f422c136310 con 0x7f422c1c7eb0 2024-04-26T00:40:41.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.797+0000 7f4231f37640 1 -- 172.21.15.37:0/439708414 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f422c1c7eb0 msgr2=0x7f422c09ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:41.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.797+0000 7f4231f37640 1 --2- 172.21.15.37:0/439708414 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f422c1c7eb0 0x7f422c09ab90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4228010820 tx=0x7f42280046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:41.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.797+0000 7f42227fc640 1 -- 172.21.15.37:0/439708414 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4228018030 con 0x7f422c1c7eb0 2024-04-26T00:40:41.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.797+0000 7f42227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:41.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.797+0000 7f42227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:40:41.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.797+0000 7f42227fc640 10 monclient: dump: 2024-04-26T00:40:41.805 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:41.805 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:41.805 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:41.805 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:41.805 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:41.805 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:41.805 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:41.805 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:41.805 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:41.805 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:41.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.797+0000 7f42227fc640 1 -- 172.21.15.37:0/439708414 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4547 (secure 0 0 0) 0x7f422800ecb0 con 0x7f422c1c7eb0 2024-04-26T00:40:41.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.797+0000 7f42227fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:40:41.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.797+0000 7f42227fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:40:41.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.797+0000 7f42227fc640 1 -- 172.21.15.37:0/439708414 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f422c1c7eb0 msgr2=0x7f422c09ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:41.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.797+0000 7f42227fc640 1 --2- 172.21.15.37:0/439708414 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f422c1c7eb0 0x7f422c09ab90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4228010820 tx=0x7f42280046d0 comp rx=0 tx=0).stop 2024-04-26T00:40:41.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.797+0000 7f42227fc640 1 -- 172.21.15.37:0/439708414 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f422c1c7eb0 msgr2=0x7f422c09ab90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:40:41.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.797+0000 7f42227fc640 1 --2- 172.21.15.37:0/439708414 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f422c1c7eb0 0x7f422c09ab90 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:41.806 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:40:41.806 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:40:41.806 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:40:41.806 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:40:41.806 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 152, 2024-04-26T00:40:41.806 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 10567, 2024-04-26T00:40:41.806 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:40:41.806 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:40:41.806 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:40:41.807 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:40:41.807 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:41.807 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 5, 2024-04-26T00:40:41.807 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:40:41.807 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:40:41.807 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:40:41.807 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:40:41.807 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:40:41.807 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:40:41.808 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:40:41.808 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:40:41.808 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:40:41.808 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:40:41.808 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:40:41.808 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:40:41.808 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:40:41.808 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:40:41.808 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:41.808 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:40:41.808 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:40:41.808 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:40:41.808 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:40:41.808 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:40:41.808 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:40:41.808 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:40:41.809 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:40:41.809 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:40:41.809 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:40:41.809 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:40:41.809 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:40:41.809 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:40:41.809 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:41.809 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:41.809 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:40:41.809 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:40:41.809 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:40:41.809 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:40:41.809 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:40:41.809 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:40:41.810 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:40:41.810 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:40:41.810 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:40:41.810 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:40:41.810 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:40:41.810 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:40:41.810 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:40:41.810 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:40:41.810 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:40:41.810 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:40:41.810 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:40:41.810 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:40:41.810 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:40:41.810 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:40:41.811 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:40:41.811 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:40:41.811 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:40:41.811 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:40:41.811 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:40:41.811 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:40:41.811 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:40:41.811 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:40:41.811 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:41.811 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:40:41.811 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:41.811 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:40:41.811 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:41.811 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:40:41.811 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:40:41.811 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:40:41.812 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:40:41.812 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:41.812 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:40:41.812 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:40:41.812 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:41.812 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:41.812 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:41.812 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:40:41.812 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:40:41.812 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:41.812 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:41.812 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:41.812 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:41.812 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:40:41.812 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:40:41.813 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:40:41.813 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:40:41.813 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:40:41.813 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:41.813 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:41.813 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:40:41.813 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:40:41.813 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:40:41.813 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:40:41.813 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:41.813 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:40:41.813 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:40:41.813 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:41.813 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:41.813 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:41.814 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:40:41.814 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:40:41.814 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:41.814 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:41.814 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:41.814 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:41.814 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:40:41.814 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:40:41.814 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:40:41.814 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:40:41.814 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:40:41.814 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:41.814 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:41.814 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:40:41.814 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:40:41.815 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:40:41.815 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:40:41.815 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:41.815 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:40:41.815 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:40:41.815 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:41.815 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:41.815 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:41.815 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:40:41.815 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:40:41.815 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:41.815 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:41.815 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:41.815 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:41.815 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:40:41.816 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:40:41.816 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:40:41.816 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:40:41.816 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:40:41.816 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:41.816 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:41.816 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:41.816 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:40:41.816 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:40:41.816 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:41.816 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:40:41.816 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:40:41.816 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:40:41.816 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:41.816 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:41.816 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:40:41.817 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:41.817 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:40:41.817 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:40:41.817 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:40:41.817 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:41.817 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:41.817 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:41.817 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:40:41.817 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:40:41.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.797+0000 7f423419b640 1 -- 172.21.15.37:0/439708414 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4210035f30 msgr2=0x7f42100383f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:41.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.797+0000 7f423419b640 1 --2- 172.21.15.37:0/439708414 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4210035f30 0x7f42100383f0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f42240055b0 tx=0x7f4224005540 comp rx=0 tx=0).stop 2024-04-26T00:40:41.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.797+0000 7f423419b640 10 monclient: shutdown 2024-04-26T00:40:41.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.797+0000 7f423419b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:40:41.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.797+0000 7f423419b640 1 -- 172.21.15.37:0/439708414 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f422c071a80 msgr2=0x7f422c1cb420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:41.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.797+0000 7f423419b640 1 --2- 172.21.15.37:0/439708414 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f422c071a80 0x7f422c1cb420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f421c0ae040 tx=0x7f421c0191e0 comp rx=0 tx=0).stop 2024-04-26T00:40:41.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.797+0000 7f4232738640 1 -- 172.21.15.37:0/439708414 reap_dead start 2024-04-26T00:40:41.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.797+0000 7f4232738640 1 -- 172.21.15.37:0/439708414 reap_dead start 2024-04-26T00:40:41.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.797+0000 7f423419b640 1 -- 172.21.15.37:0/439708414 shutdown_connections 2024-04-26T00:40:41.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.797+0000 7f423419b640 1 -- 172.21.15.37:0/439708414 >> 172.21.15.37:0/439708414 conn(0x7f422c0a24c0 msgr2=0x7f422c135440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:40:41.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.797+0000 7f423419b640 1 -- 172.21.15.37:0/439708414 shutdown_connections 2024-04-26T00:40:41.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.797+0000 7f423419b640 1 -- 172.21.15.37:0/439708414 wait complete. 2024-04-26T00:40:41.820 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:40:41.820 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:40:41.820 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:40:41.888 INFO:tasks.rados.rados.0.smithi045.stdout:3646: done (5 left) 2024-04-26T00:40:41.889 INFO:tasks.rados.rados.0.smithi045.stdout:3647: done (4 left) 2024-04-26T00:40:41.889 INFO:tasks.rados.rados.0.smithi045.stdout:3648: done (3 left) 2024-04-26T00:40:41.889 INFO:tasks.rados.rados.0.smithi045.stdout:3649: done (2 left) 2024-04-26T00:40:41.889 INFO:tasks.rados.rados.0.smithi045.stdout:3650: done (1 left) 2024-04-26T00:40:41.889 INFO:tasks.rados.rados.0.smithi045.stdout:3651: done (0 left) 2024-04-26T00:40:41.889 INFO:tasks.rados.rados.0.smithi045.stdout:3652: delete oid 6 current snap is 403 2024-04-26T00:40:41.895 INFO:tasks.rados.rados.0.smithi045.stdout:3652: done (0 left) 2024-04-26T00:40:41.895 INFO:tasks.rados.rados.0.smithi045.stdout:3653: write oid 33 current snap is 403 2024-04-26T00:40:41.895 INFO:tasks.rados.rados.0.smithi045.stdout:3653: seq_num 1302 ranges {682714=670561,2123320=505457,3088266=636749} 2024-04-26T00:40:41.909 INFO:tasks.rados.rados.0.smithi045.stdout:3653: writing smithi04526579-33 from 682714 to 1353275 tid 1 2024-04-26T00:40:41.923 INFO:tasks.rados.rados.0.smithi045.stdout:3653: writing smithi04526579-33 from 2123320 to 2628777 tid 2 2024-04-26T00:40:41.934 INFO:tasks.rados.rados.0.smithi045.stdout:3653: writing smithi04526579-33 from 3088266 to 3725015 tid 3 2024-04-26T00:40:41.934 INFO:tasks.rados.rados.0.smithi045.stdout:3654: snap_create 2024-04-26T00:40:41.934 INFO:tasks.rados.rados.0.smithi045.stdout:3653: finishing write tid 1 to smithi04526579-33 2024-04-26T00:40:41.934 INFO:tasks.rados.rados.0.smithi045.stdout:3653: finishing write tid 2 to smithi04526579-33 2024-04-26T00:40:41.941 INFO:tasks.rados.rados.0.smithi045.stdout:3653: finishing write tid 3 to smithi04526579-33 2024-04-26T00:40:41.942 INFO:tasks.rados.rados.0.smithi045.stdout:3653: finishing write tid 4 to smithi04526579-33 2024-04-26T00:40:41.942 INFO:tasks.rados.rados.0.smithi045.stdout:3653: finishing write tid 5 to smithi04526579-33 2024-04-26T00:40:41.942 INFO:tasks.rados.rados.0.smithi045.stdout:3653: oid 33 updating version 0 to 2762 2024-04-26T00:40:41.943 INFO:tasks.rados.rados.0.smithi045.stdout:3653: oid 33 updating version 2762 to 2763 2024-04-26T00:40:41.943 INFO:tasks.rados.rados.0.smithi045.stdout:3653: oid 33 version 2763 is already newer than 2760 2024-04-26T00:40:41.943 INFO:tasks.rados.rados.0.smithi045.stdout:3653: oid 33 version 2763 is already newer than 2761 2024-04-26T00:40:41.943 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 2763 (ObjNum 1302 snap 403 seq_num 1302) dirty exists 2024-04-26T00:40:41.943 INFO:tasks.rados.rados.0.smithi045.stdout:3653: left oid 33 (ObjNum 1302 snap 403 seq_num 1302) 2024-04-26T00:40:41.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.945+0000 7f6a533ab640 10 monclient: get_monmap_and_config 2024-04-26T00:40:41.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.945+0000 7f6a533ab640 10 monclient: build_initial_monmap 2024-04-26T00:40:41.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.945+0000 7f6a533ab640 10 monclient: monmap: 2024-04-26T00:40:41.947 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:40:41.947 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:41.947 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:40:41.949858+0000 2024-04-26T00:40:41.947 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:40:41.949858+0000 2024-04-26T00:40:41.947 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:40:41.947 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:41.947 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:40:41.947 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:40:41.948 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:40:41.948 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:40:41.948 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:40:41.948 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:41.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.945+0000 7f6a533ab640 1 Processor -- start 2024-04-26T00:40:41.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.945+0000 7f6a533ab640 1 -- start start 2024-04-26T00:40:41.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.945+0000 7f6a533ab640 10 monclient: init 2024-04-26T00:40:41.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a533ab640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:41.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a533ab640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-04-26T00:40:41.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a533ab640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a4c09e220 0x7f6a4c09fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:41.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a533ab640 10 monclient(hunting): picked mon.noname-b con 0x7f6a4c09e220 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:40:41.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a533ab640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6a4c0a14f0 addr v1:172.21.15.45:6789/0 2024-04-26T00:40:41.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a533ab640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6a4c09e600 0x7f6a4c13aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:41.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a533ab640 10 monclient(hunting): picked mon.noname-a con 0x7f6a4c09e600 addr v2:172.21.15.37:3300/0 2024-04-26T00:40:41.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a533ab640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a4c0a12b0 con 0x7f6a4c0a14f0 2024-04-26T00:40:41.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a533ab640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:41.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a533ab640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f6a4c09cd10 con 0x7f6a4c09e600 2024-04-26T00:40:41.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a533ab640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:41.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a533ab640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f6a4c09da20 con 0x7f6a4c09e220 2024-04-26T00:40:41.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a533ab640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:41.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a533ab640 10 monclient(hunting): authenticate will time out at 2223.256266s 2024-04-26T00:40:41.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a51147640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a4c09e220 0x7f6a4c09fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:41.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a51147640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a4c09e220 0x7f6a4c09fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:44094/0 (socket says 172.21.15.37:44094) 2024-04-26T00:40:41.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a51948640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6a4c09e600 0x7f6a4c13aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:41.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a51147640 1 -- 172.21.15.37:0/191997170 learned_addr learned my addr 172.21.15.37:0/191997170 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:40:41.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a51147640 10 monclient(hunting): get_auth_request con 0x7f6a4c09e220 auth_method 0 2024-04-26T00:40:41.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a51147640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:41.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a51147640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:41.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a51147640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:41.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a51948640 10 monclient(hunting): get_auth_request con 0x7f6a4c09e600 auth_method 0 2024-04-26T00:40:41.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a51948640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:41.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a51948640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:41.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a51948640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:41.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a51147640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:41.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a51147640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:41.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a51147640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:41.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a51948640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:41.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a51948640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:41.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a51948640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:41.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a51948640 10 monclient(hunting): handle_auth_done global_id 244236 payload 274 2024-04-26T00:40:41.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a51948640 1 -- 172.21.15.37:0/191997170 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a4c09e220 msgr2=0x7f6a4c09fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:40:41.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a51948640 1 --2- 172.21.15.37:0/191997170 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a4c09e220 0x7f6a4c09fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:41.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a51948640 1 -- 172.21.15.37:0/191997170 >> v1:172.21.15.45:6789/0 conn(0x7f6a4c0a14f0 legacy=0x7f6a4c0a0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:40:41.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a51948640 10 monclient: _finish_hunting 0 2024-04-26T00:40:41.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a51948640 1 monclient: found mon.noname-a 2024-04-26T00:40:41.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a51948640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:41.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a51948640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:40:41.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a51948640 1 -- 172.21.15.37:0/191997170 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a4c09dd00 con 0x7f6a4c09e600 2024-04-26T00:40:41.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.949+0000 7f6a51147640 1 --2- 172.21.15.37:0/191997170 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a4c09e220 0x7f6a4c09fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:40:41.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a51948640 1 -- 172.21.15.37:0/191997170 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a4c09e600 msgr2=0x7f6a4c13aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:41.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a51948640 1 --2- 172.21.15.37:0/191997170 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a4c09e600 0x7f6a4c13aa60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6a44009d90 tx=0x7f6a4401c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=637f11e0c8e2bab2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:41.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a43fff640 1 -- 172.21.15.37:0/191997170 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a440a6710 con 0x7f6a4c09e600 2024-04-26T00:40:41.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a43fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:41.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a43fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:40:41.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a43fff640 10 monclient: dump: 2024-04-26T00:40:41.956 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:41.956 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:41.956 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:41.956 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:41.956 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:41.956 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:41.956 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:41.956 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:41.956 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:41.956 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:41.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a43fff640 10 monclient: _finish_auth 0 2024-04-26T00:40:41.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a43fff640 10 monclient: _check_auth_tickets 2024-04-26T00:40:41.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:41.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a533ab640 5 monclient: authenticate success, global_id 244236 2024-04-26T00:40:41.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a533ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:40:41.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a43fff640 1 -- 172.21.15.37:0/191997170 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a4c0a0ab0 con 0x7f6a4c09e600 2024-04-26T00:40:41.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a43fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:40:41.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a43fff640 1 -- 172.21.15.37:0/191997170 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a440a6ad0 con 0x7f6a4c09e600 2024-04-26T00:40:41.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a43fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:41.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a43fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:40:41.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a43fff640 10 monclient: dump: 2024-04-26T00:40:41.957 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:41.957 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:41.957 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:41.957 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:41.957 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:41.957 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:41.958 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:41.958 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:41.958 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:41.958 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:41.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a533ab640 10 monclient: get_monmap_and_config success 2024-04-26T00:40:41.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a533ab640 4 set_mon_vals no callback set 2024-04-26T00:40:41.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a533ab640 10 monclient: shutdown 2024-04-26T00:40:41.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a533ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:40:41.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a533ab640 1 -- 172.21.15.37:0/191997170 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a4c09e600 msgr2=0x7f6a4c13aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:41.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a533ab640 1 --2- 172.21.15.37:0/191997170 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a4c09e600 0x7f6a4c13aa60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6a44009d90 tx=0x7f6a4401c7f0 comp rx=0 tx=0).stop 2024-04-26T00:40:41.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a533ab640 1 -- 172.21.15.37:0/191997170 shutdown_connections 2024-04-26T00:40:41.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a533ab640 1 --2- 172.21.15.37:0/191997170 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a4c09e600 0x7f6a4c13aa60 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:41.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a533ab640 1 --2- 172.21.15.37:0/191997170 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a4c09e220 0x7f6a4c09fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:41.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a533ab640 1 -- 172.21.15.37:0/191997170 >> 172.21.15.37:0/191997170 conn(0x7f6a4c0a1d50 msgr2=0x7f6a4c0a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:40:41.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a533ab640 1 -- 172.21.15.37:0/191997170 shutdown_connections 2024-04-26T00:40:41.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a533ab640 1 -- 172.21.15.37:0/191997170 wait complete. 2024-04-26T00:40:41.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a533ab640 10 monclient: build_initial_monmap 2024-04-26T00:40:41.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a533ab640 10 monclient: monmap: 2024-04-26T00:40:41.964 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:40:41.964 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:40:41.964 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:40:41.965 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:40:41.965 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:40:41.965 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:40:41.965 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:40:41.965 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:40:41.965 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:40:41.965 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:41.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a533ab640 1 Processor -- start 2024-04-26T00:40:41.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a533ab640 1 -- start start 2024-04-26T00:40:41.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a533ab640 10 monclient: init 2024-04-26T00:40:41.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a533ab640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:41.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a533ab640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:40:41.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a533ab640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a4c09e220 0x7f6a4c06dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:41.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a533ab640 10 monclient(hunting): picked mon.noname-c con 0x7f6a4c09e220 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:40:41.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a533ab640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a4c09e600 0x7f6a4c06e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:41.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a533ab640 10 monclient(hunting): picked mon.noname-a con 0x7f6a4c09e600 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:40:41.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a533ab640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a4c0a14f0 0x7f6a4c06e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:41.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a533ab640 10 monclient(hunting): picked mon.noname-b con 0x7f6a4c0a14f0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:40:41.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a533ab640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:41.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a533ab640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f6a4c13b440 con 0x7f6a4c09e600 2024-04-26T00:40:41.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a533ab640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:41.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.953+0000 7f6a533ab640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6a4c13b2c0 con 0x7f6a4c0a14f0 2024-04-26T00:40:41.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a533ab640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:41.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a533ab640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f6a4c13b5c0 con 0x7f6a4c09e220 2024-04-26T00:40:41.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a533ab640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:41.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a533ab640 10 monclient(hunting): authenticate will time out at 2223.264016s 2024-04-26T00:40:41.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a51147640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a4c09e220 0x7f6a4c06dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:41.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a50946640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a4c09e600 0x7f6a4c06e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:41.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a51147640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a4c09e220 0x7f6a4c06dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:44102/0 (socket says 172.21.15.37:44102) 2024-04-26T00:40:41.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a50946640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a4c09e600 0x7f6a4c06e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:52158/0 (socket says 172.21.15.37:52158) 2024-04-26T00:40:41.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a51147640 1 -- 172.21.15.37:0/1546956942 learned_addr learned my addr 172.21.15.37:0/1546956942 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:40:41.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a51147640 10 monclient(hunting): get_auth_request con 0x7f6a4c09e220 auth_method 0 2024-04-26T00:40:41.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a51147640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:41.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a51147640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:41.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a51147640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:41.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a50946640 10 monclient(hunting): get_auth_request con 0x7f6a4c09e600 auth_method 0 2024-04-26T00:40:41.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a50946640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:41.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a50946640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:41.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a50946640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:41.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a51948640 1 --2- 172.21.15.37:0/1546956942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a4c0a14f0 0x7f6a4c06e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:41.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a51147640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:41.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a51147640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:41.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a51147640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:41.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a51948640 10 monclient(hunting): get_auth_request con 0x7f6a4c0a14f0 auth_method 0 2024-04-26T00:40:41.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a51948640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:41.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a51948640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:41.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a51948640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:41.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a50946640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:41.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a50946640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:41.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a50946640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:41.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a51147640 10 monclient(hunting): handle_auth_done global_id 254153 payload 824 2024-04-26T00:40:41.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a51147640 1 -- 172.21.15.37:0/1546956942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a4c0a14f0 msgr2=0x7f6a4c06e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:41.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a51147640 1 --2- 172.21.15.37:0/1546956942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a4c0a14f0 0x7f6a4c06e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:41.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a51147640 1 -- 172.21.15.37:0/1546956942 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a4c09e600 msgr2=0x7f6a4c06e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:41.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a51147640 1 --2- 172.21.15.37:0/1546956942 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a4c09e600 0x7f6a4c06e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:41.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a51147640 10 monclient: _finish_hunting 0 2024-04-26T00:40:41.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a51147640 1 monclient: found mon.noname-c 2024-04-26T00:40:41.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a51147640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:41.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a51147640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:40:41.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a51147640 1 -- 172.21.15.37:0/1546956942 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a4c06f840 con 0x7f6a4c09e220 2024-04-26T00:40:41.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a51948640 1 --2- 172.21.15.37:0/1546956942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a4c0a14f0 0x7f6a4c06e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:40:41.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a51147640 1 -- 172.21.15.37:0/1546956942 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a4c09e220 msgr2=0x7f6a4c06dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:41.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a51147640 1 --2- 172.21.15.37:0/1546956942 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a4c09e220 0x7f6a4c06dc40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6a4800e9d0 tx=0x7f6a4800ee90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:41.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a41ffb640 1 -- 172.21.15.37:0/1546956942 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a48017070 con 0x7f6a4c09e220 2024-04-26T00:40:41.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a41ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:41.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a41ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:40:41.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a41ffb640 10 monclient: dump: 2024-04-26T00:40:41.969 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:41.969 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:41.969 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:41.969 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:41.969 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:41.969 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:41.969 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:41.970 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:41.970 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:41.970 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:41.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a41ffb640 10 monclient: _finish_auth 0 2024-04-26T00:40:41.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a41ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:40:41.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a41ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:41.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a41ffb640 1 -- 172.21.15.37:0/1546956942 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a4800cc70 con 0x7f6a4c09e220 2024-04-26T00:40:41.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a41ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:40:41.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a41ffb640 1 -- 172.21.15.37:0/1546956942 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a48013670 con 0x7f6a4c09e220 2024-04-26T00:40:41.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a41ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:41.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a41ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:40:41.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a41ffb640 10 monclient: dump: 2024-04-26T00:40:41.970 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:41.970 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:41.970 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:41.971 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:41.971 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:41.971 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:41.971 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:41.971 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:41.971 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:41.971 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:41.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a437fe640 4 set_mon_vals no callback set 2024-04-26T00:40:41.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a533ab640 5 monclient: authenticate success, global_id 254153 2024-04-26T00:40:41.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a533ab640 10 monclient: _renew_subs 2024-04-26T00:40:41.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a533ab640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:40:41.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a533ab640 1 -- 172.21.15.37:0/1546956942 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a4c09c970 con 0x7f6a4c09e220 2024-04-26T00:40:41.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a533ab640 10 monclient: _renew_subs 2024-04-26T00:40:41.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a533ab640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:40:41.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.957+0000 7f6a533ab640 1 -- 172.21.15.37:0/1546956942 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a4c071250 con 0x7f6a4c09e220 2024-04-26T00:40:41.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.961+0000 7f6a533ab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:41.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.961+0000 7f6a533ab640 1 --2- 172.21.15.37:0/1546956942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a4c071670 0x7f6a4c1cd060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:41.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.961+0000 7f6a533ab640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:41.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.961+0000 7f6a533ab640 1 -- 172.21.15.37:0/1546956942 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6a4c06fac0 con 0x7f6a4c071670 2024-04-26T00:40:41.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.961+0000 7f6a41ffb640 1 -- 172.21.15.37:0/1546956942 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f6a4801a070 con 0x7f6a4c09e220 2024-04-26T00:40:41.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.961+0000 7f6a41ffb640 1 --2- 172.21.15.37:0/1546956942 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6a30035ed0 0x7f6a30038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:41.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.961+0000 7f6a41ffb640 1 -- 172.21.15.37:0/1546956942 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(823..823 src has 705..823) ==== 4023+0+0 (secure 0 0 0) 0x7f6a48048ae0 con 0x7f6a4c09e220 2024-04-26T00:40:41.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.961+0000 7f6a50946640 1 --2- 172.21.15.37:0/1546956942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a4c071670 0x7f6a4c1cd060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:41.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.961+0000 7f6a50946640 10 monclient: get_auth_request con 0x7f6a4c071670 auth_method 0 2024-04-26T00:40:41.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.961+0000 7f6a50946640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:41.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.961+0000 7f6a50946640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:41.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.961+0000 7f6a50946640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:41.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.961+0000 7f6a50946640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:41.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.961+0000 7f6a50946640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:41.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.961+0000 7f6a50946640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:41.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.961+0000 7f6a51948640 1 --2- 172.21.15.37:0/1546956942 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6a30035ed0 0x7f6a30038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:41.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.961+0000 7f6a51948640 10 monclient: get_auth_request con 0x7f6a30035ed0 auth_method 0 2024-04-26T00:40:41.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.961+0000 7f6a50946640 10 monclient(hunting): handle_auth_done global_id 224386 payload 824 2024-04-26T00:40:41.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.961+0000 7f6a50946640 1 -- 172.21.15.37:0/1546956942 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6a4c0a0ab0 con 0x7f6a4c071670 2024-04-26T00:40:41.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.965+0000 7f6a50946640 1 -- 172.21.15.37:0/1546956942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a4c071670 msgr2=0x7f6a4c1cd060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:41.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.965+0000 7f6a50946640 1 --2- 172.21.15.37:0/1546956942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a4c071670 0x7f6a4c1cd060 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c13a060 tx=0x7f6a3c00fc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:41.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.965+0000 7f6a51948640 1 --2- 172.21.15.37:0/1546956942 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6a30035ed0 0x7f6a30038390 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c076500 tx=0x7f6a440a7000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:41.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.965+0000 7f6a41ffb640 1 -- 172.21.15.37:0/1546956942 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3c003b80 con 0x7f6a4c071670 2024-04-26T00:40:41.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.965+0000 7f6a41ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:41.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.965+0000 7f6a41ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:40:41.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.965+0000 7f6a41ffb640 10 monclient: dump: 2024-04-26T00:40:41.975 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:41.975 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:41.975 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:41.975 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:41.975 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:41.975 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:41.975 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:41.976 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:41.976 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:41.976 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:41.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.965+0000 7f6a41ffb640 1 -- 172.21.15.37:0/1546956942 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6a3c019070 con 0x7f6a4c071670 2024-04-26T00:40:41.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.965+0000 7f6a41ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:41.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.965+0000 7f6a41ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:40:41.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.965+0000 7f6a41ffb640 1 -- 172.21.15.37:0/1546956942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a4c071670 msgr2=0x7f6a4c1cd060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:41.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.965+0000 7f6a41ffb640 1 --2- 172.21.15.37:0/1546956942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a4c071670 0x7f6a4c1cd060 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c13a060 tx=0x7f6a3c00fc00 comp rx=0 tx=0).stop 2024-04-26T00:40:41.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.965+0000 7f6a41ffb640 1 -- 172.21.15.37:0/1546956942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a4c071670 msgr2=0x7f6a4c1cd060 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:40:41.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.965+0000 7f6a41ffb640 1 --2- 172.21.15.37:0/1546956942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a4c071670 0x7f6a4c1cd060 secure :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c13a060 tx=0x7f6a3c00fc00 comp rx=0 tx=0).stop 2024-04-26T00:40:41.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.969+0000 7f6a533ab640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:40:41.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.969+0000 7f6a533ab640 1 --2- 172.21.15.37:0/1546956942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a4c1cd5a0 0x7f6a4c1cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:41.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.969+0000 7f6a533ab640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:41.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.969+0000 7f6a533ab640 1 -- 172.21.15.37:0/1546956942 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6a4c1cdef0 con 0x7f6a4c1cd5a0 2024-04-26T00:40:41.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.969+0000 7f6a50946640 1 --2- 172.21.15.37:0/1546956942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a4c1cd5a0 0x7f6a4c1cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:41.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.969+0000 7f6a50946640 10 monclient: get_auth_request con 0x7f6a4c1cd5a0 auth_method 0 2024-04-26T00:40:41.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.969+0000 7f6a50946640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:41.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.969+0000 7f6a50946640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:41.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.969+0000 7f6a50946640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:41.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.969+0000 7f6a50946640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:41.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.969+0000 7f6a50946640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:41.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.969+0000 7f6a50946640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:41.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.973+0000 7f6a50946640 10 monclient(hunting): handle_auth_done global_id 224389 payload 824 2024-04-26T00:40:41.977 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.973+0000 7f6a50946640 1 -- 172.21.15.37:0/1546956942 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6a4c04f510 con 0x7f6a4c1cd5a0 2024-04-26T00:40:41.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.973+0000 7f6a50946640 1 -- 172.21.15.37:0/1546956942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a4c1cd5a0 msgr2=0x7f6a4c1cd9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:41.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.973+0000 7f6a50946640 1 --2- 172.21.15.37:0/1546956942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a4c1cd5a0 0x7f6a4c1cd9b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c018000 tx=0x7f6a3c003770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:41.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.973+0000 7f6a41ffb640 1 -- 172.21.15.37:0/1546956942 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3c006e00 con 0x7f6a4c1cd5a0 2024-04-26T00:40:41.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.973+0000 7f6a41ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:41.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.973+0000 7f6a41ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:40:41.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.973+0000 7f6a41ffb640 10 monclient: dump: 2024-04-26T00:40:41.978 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:41.978 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:41.978 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:41.979 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:41.979 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:41.979 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:41.979 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:41.979 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:41.979 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:41.979 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:41.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.973+0000 7f6a41ffb640 1 -- 172.21.15.37:0/1546956942 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4888 (secure 0 0 0) 0x7f6a3c006e00 con 0x7f6a4c1cd5a0 2024-04-26T00:40:41.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.973+0000 7f6a41ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:40:41.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.973+0000 7f6a41ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:40:41.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.973+0000 7f6a41ffb640 1 -- 172.21.15.37:0/1546956942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a4c1cd5a0 msgr2=0x7f6a4c1cd9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:41.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.973+0000 7f6a41ffb640 1 --2- 172.21.15.37:0/1546956942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a4c1cd5a0 0x7f6a4c1cd9b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c018000 tx=0x7f6a3c003770 comp rx=0 tx=0).stop 2024-04-26T00:40:41.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.973+0000 7f6a41ffb640 1 -- 172.21.15.37:0/1546956942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a4c1cd5a0 msgr2=0x7f6a4c1cd9b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:40:41.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.973+0000 7f6a41ffb640 1 --2- 172.21.15.37:0/1546956942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a4c1cd5a0 0x7f6a4c1cd9b0 secure :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c018000 tx=0x7f6a3c003770 comp rx=0 tx=0).stop 2024-04-26T00:40:41.980 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:40:41.980 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:40:41.980 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:40:41.980 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:40:41.980 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 152, 2024-04-26T00:40:41.980 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 155637, 2024-04-26T00:40:41.980 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:40:41.980 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:40:41.980 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:40:41.980 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:40:41.980 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:41.980 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 5, 2024-04-26T00:40:41.981 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:40:41.981 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:40:41.981 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:40:41.981 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:40:41.981 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:40:41.981 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:40:41.981 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:40:41.981 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:40:41.981 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:40:41.981 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:40:41.981 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:40:41.981 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:40:41.981 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:40:41.981 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:40:41.981 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:41.982 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:40:41.982 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:40:41.982 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:40:41.982 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:40:41.982 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:40:41.982 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:40:41.982 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:40:41.982 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:40:41.982 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:40:41.982 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:40:41.982 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:40:41.982 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:40:41.982 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:40:41.982 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:41.982 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:41.983 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:40:41.983 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:40:41.983 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:40:41.983 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:40:41.983 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:40:41.983 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:40:41.983 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:40:41.983 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:40:41.983 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:40:41.983 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:40:41.983 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:40:41.983 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:40:41.983 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:40:41.983 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:40:41.983 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:40:41.983 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:40:41.984 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:40:41.984 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:40:41.984 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:40:41.984 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:40:41.984 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:40:41.984 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:40:41.984 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:40:41.984 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:40:41.984 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:40:41.984 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:40:41.984 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:40:41.984 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:40:41.984 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:41.984 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:40:41.985 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:41.985 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:40:41.985 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:41.985 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:40:41.985 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:40:41.985 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:40:41.985 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:40:41.985 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:41.985 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:40:41.985 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:40:41.985 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:41.985 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:41.985 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:41.986 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:40:41.987 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:40:41.987 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:41.987 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:41.987 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:41.987 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:41.987 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:40:41.987 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:40:41.988 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:40:41.988 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:40:41.988 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:40:41.988 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:41.988 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:41.988 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:40:41.988 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:40:41.988 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:40:41.988 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:40:41.988 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:41.988 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:40:41.988 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:40:41.988 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:41.988 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:41.989 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:41.989 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:40:41.989 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:40:41.989 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:41.989 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:41.989 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:41.989 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:41.989 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:40:41.989 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:40:41.989 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:40:41.989 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:40:41.989 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:40:41.989 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:41.989 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:41.990 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:40:41.990 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:40:41.990 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:40:41.990 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:40:41.990 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:41.990 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:40:41.990 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:40:41.990 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:41.990 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:41.990 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:41.990 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:40:41.990 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:40:41.990 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:41.990 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:41.991 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:41.991 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:41.991 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:40:41.991 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:40:41.991 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:40:41.991 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:40:41.991 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:40:41.991 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:41.991 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:41.991 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:41.991 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:40:41.991 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:40:41.991 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:41.991 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:40:41.991 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:40:41.992 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:40:41.992 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:41.992 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:41.992 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:40:41.992 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:41.992 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:40:41.992 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:40:41.992 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:40:41.992 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:41.992 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:41.992 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:40:41.992 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:41.992 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:40:41.992 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:40:41.992 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:40:41.993 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:41.993 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:41.993 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:40:41.993 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:41.993 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:40:41.993 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:40:41.993 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:40:41.993 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:41.993 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:41.993 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:41.993 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:40:41.993 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:40:41.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.977+0000 7f6a533ab640 1 -- 172.21.15.37:0/1546956942 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6a30035ed0 msgr2=0x7f6a30038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:41.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.977+0000 7f6a533ab640 1 --2- 172.21.15.37:0/1546956942 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6a30035ed0 0x7f6a30038390 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c076500 tx=0x7f6a440a7000 comp rx=0 tx=0).stop 2024-04-26T00:40:41.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.977+0000 7f6a533ab640 10 monclient: shutdown 2024-04-26T00:40:41.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.977+0000 7f6a533ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:40:41.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.977+0000 7f6a533ab640 1 -- 172.21.15.37:0/1546956942 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a4c09e220 msgr2=0x7f6a4c06dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:41.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.977+0000 7f6a533ab640 1 --2- 172.21.15.37:0/1546956942 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a4c09e220 0x7f6a4c06dc40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6a4800e9d0 tx=0x7f6a4800ee90 comp rx=0 tx=0).stop 2024-04-26T00:40:41.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.977+0000 7f6a51948640 1 -- 172.21.15.37:0/1546956942 reap_dead start 2024-04-26T00:40:41.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.977+0000 7f6a533ab640 1 -- 172.21.15.37:0/1546956942 shutdown_connections 2024-04-26T00:40:41.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.977+0000 7f6a533ab640 1 --2- 172.21.15.37:0/1546956942 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a4c09e220 0x7f6a4c06dc40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:41.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.977+0000 7f6a533ab640 1 -- 172.21.15.37:0/1546956942 >> 172.21.15.37:0/1546956942 conn(0x7f6a4c0a1d50 msgr2=0x7f6a4c13a620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:40:41.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.977+0000 7f6a533ab640 1 -- 172.21.15.37:0/1546956942 shutdown_connections 2024-04-26T00:40:41.996 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:41.977+0000 7f6a533ab640 1 -- 172.21.15.37:0/1546956942 wait complete. 2024-04-26T00:40:41.996 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:40:41.996 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-04-26T00:40:41.996 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-26T00:40:41.996 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-04-26T00:40:41.996 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-04-26T00:40:41.997 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-26T00:40:41.997 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T00:40:42.009 INFO:tasks.ceph.mon.a:Stopped 2024-04-26T00:40:42.009 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-26T00:40:42.009 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:40:42.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.165+0000 7f6df40e2640 10 monclient: get_monmap_and_config 2024-04-26T00:40:42.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.165+0000 7f6df40e2640 10 monclient: build_initial_monmap 2024-04-26T00:40:42.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.165+0000 7f6df40e2640 10 monclient: monmap: 2024-04-26T00:40:42.167 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:40:42.167 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:42.167 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:40:42.169341+0000 2024-04-26T00:40:42.167 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:40:42.169341+0000 2024-04-26T00:40:42.167 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:40:42.167 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:42.167 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:40:42.167 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:40:42.167 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:40:42.167 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:40:42.168 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:40:42.168 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:42.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.165+0000 7f6df40e2640 1 Processor -- start 2024-04-26T00:40:42.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.165+0000 7f6df40e2640 1 -- start start 2024-04-26T00:40:42.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.165+0000 7f6df40e2640 10 monclient: init 2024-04-26T00:40:42.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.165+0000 7f6df40e2640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:42.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.165+0000 7f6df40e2640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-04-26T00:40:42.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.165+0000 7f6df40e2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6dec0a0260 addr v1:172.21.15.37:6789/0 2024-04-26T00:40:42.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.165+0000 7f6df40e2640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6dec137e40 0x7f6dec13a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:42.170 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.165+0000 7f6df40e2640 10 monclient(hunting): picked mon.noname-c con 0x7f6dec137e40 addr v2:172.21.15.45:3300/0 2024-04-26T00:40:42.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.165+0000 7f6df40e2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6dec13a770 addr v1:172.21.15.45:6789/0 2024-04-26T00:40:42.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.165+0000 7f6df40e2640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6dec0a1130 con 0x7f6dec0a0260 2024-04-26T00:40:42.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.165+0000 7f6df40e2640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6dec06cff0 con 0x7f6dec13a770 2024-04-26T00:40:42.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.165+0000 7f6df40e2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:42.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.165+0000 7f6df40e2640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f6dec0a0d50 con 0x7f6dec137e40 2024-04-26T00:40:42.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.165+0000 7f6df40e2640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:42.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.165+0000 7f6df40e2640 10 monclient(hunting): authenticate will time out at 2223.475194s 2024-04-26T00:40:42.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.165+0000 7f6df1e7e640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f6dec0a0260 legacy=0x7f6dec0a0640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:40:42.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.165+0000 7f6df167d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6dec137e40 0x7f6dec13a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:42.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.165+0000 7f6df267f640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f6dec13a770 0x7f6dec13cb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.37:36082/0 (socket says 172.21.15.37:36082) 2024-04-26T00:40:42.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.165+0000 7f6df267f640 1 -- 172.21.15.37:0/3303802778 learned_addr learned my addr 172.21.15.37:0/3303802778 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:40:42.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.165+0000 7f6df167d640 10 monclient(hunting): get_auth_request con 0x7f6dec137e40 auth_method 0 2024-04-26T00:40:42.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.165+0000 7f6df167d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:42.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.165+0000 7f6df167d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:42.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.165+0000 7f6df167d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:42.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.165+0000 7f6df167d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:42.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.165+0000 7f6df167d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:42.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.165+0000 7f6df167d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:42.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.169+0000 7f6df167d640 10 monclient(hunting): handle_auth_done global_id 224392 payload 274 2024-04-26T00:40:42.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.169+0000 7f6df167d640 1 -- 172.21.15.37:0/3303802778 >> v1:172.21.15.45:6789/0 conn(0x7f6dec13a770 legacy=0x7f6dec13cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:40:42.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.169+0000 7f6df167d640 1 -- 172.21.15.37:0/3303802778 >> v1:172.21.15.37:6789/0 conn(0x7f6dec0a0260 legacy=0x7f6dec0a0640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:40:42.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.169+0000 7f6df167d640 10 monclient: _finish_hunting 0 2024-04-26T00:40:42.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.169+0000 7f6df167d640 1 monclient: found mon.noname-c 2024-04-26T00:40:42.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.169+0000 7f6df167d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:42.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.169+0000 7f6df167d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:40:42.172 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.169+0000 7f6df167d640 1 -- 172.21.15.37:0/3303802778 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dec09bc50 con 0x7f6dec137e40 2024-04-26T00:40:42.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df167d640 1 -- 172.21.15.37:0/3303802778 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6dec137e40 msgr2=0x7f6dec13a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:42.173 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df167d640 1 --2- 172.21.15.37:0/3303802778 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6dec137e40 0x7f6dec13a230 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6de00097c0 tx=0x7f6de001c100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eebc0e1d3d98b266 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:42.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df0e7c640 1 -- 172.21.15.37:0/3303802778 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de00b1070 con 0x7f6dec137e40 2024-04-26T00:40:42.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df0e7c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:42.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df0e7c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:40:42.174 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df0e7c640 10 monclient: dump: 2024-04-26T00:40:42.174 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:42.175 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:42.175 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:42.175 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:42.175 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:42.175 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:42.175 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:42.175 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:42.175 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:42.175 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:42.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df0e7c640 10 monclient: _finish_auth 0 2024-04-26T00:40:42.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df0e7c640 10 monclient: _check_auth_tickets 2024-04-26T00:40:42.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df0e7c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:42.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df0e7c640 1 -- 172.21.15.37:0/3303802778 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6de00ac440 con 0x7f6dec137e40 2024-04-26T00:40:42.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df0e7c640 10 monclient: handle_config config(0 keys) 2024-04-26T00:40:42.175 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df0e7c640 1 -- 172.21.15.37:0/3303802778 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de00ac810 con 0x7f6dec137e40 2024-04-26T00:40:42.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df0e7c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:42.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df0e7c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:40:42.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df0e7c640 10 monclient: dump: 2024-04-26T00:40:42.176 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:42.176 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:42.176 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:42.176 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:42.176 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:42.176 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:42.176 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:42.177 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:42.177 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:42.177 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:42.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df40e2640 5 monclient: authenticate success, global_id 224392 2024-04-26T00:40:42.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df40e2640 10 monclient: get_monmap_and_config success 2024-04-26T00:40:42.177 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df40e2640 4 set_mon_vals no callback set 2024-04-26T00:40:42.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df40e2640 10 monclient: shutdown 2024-04-26T00:40:42.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df40e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:40:42.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df40e2640 1 -- 172.21.15.37:0/3303802778 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6dec137e40 msgr2=0x7f6dec13a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:42.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df40e2640 1 --2- 172.21.15.37:0/3303802778 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6dec137e40 0x7f6dec13a230 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6de00097c0 tx=0x7f6de001c100 comp rx=0 tx=0).stop 2024-04-26T00:40:42.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df40e2640 1 -- 172.21.15.37:0/3303802778 shutdown_connections 2024-04-26T00:40:42.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df40e2640 1 --2- 172.21.15.37:0/3303802778 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6dec137e40 0x7f6dec13a230 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:42.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df40e2640 1 -- 172.21.15.37:0/3303802778 >> 172.21.15.37:0/3303802778 conn(0x7f6dec0a1ad0 msgr2=0x7f6dec0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:40:42.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df40e2640 1 -- 172.21.15.37:0/3303802778 shutdown_connections 2024-04-26T00:40:42.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df40e2640 1 -- 172.21.15.37:0/3303802778 wait complete. 2024-04-26T00:40:42.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df40e2640 10 monclient: build_initial_monmap 2024-04-26T00:40:42.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df40e2640 10 monclient: monmap: 2024-04-26T00:40:42.183 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:40:42.183 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:40:42.183 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:40:42.183 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:40:42.183 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:40:42.183 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:40:42.183 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:40:42.183 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:40:42.183 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:40:42.183 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:42.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df40e2640 1 Processor -- start 2024-04-26T00:40:42.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df40e2640 1 -- start start 2024-04-26T00:40:42.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df40e2640 10 monclient: init 2024-04-26T00:40:42.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df40e2640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:42.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df40e2640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:40:42.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df40e2640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6dec0a0260 0x7f6dec1cee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:42.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df40e2640 10 monclient(hunting): picked mon.noname-c con 0x7f6dec0a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:40:42.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df40e2640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6dec137e40 0x7f6dec134b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:42.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df40e2640 10 monclient(hunting): picked mon.noname-a con 0x7f6dec137e40 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:40:42.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df40e2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6dec13a770 0x7f6dec135060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:42.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df40e2640 10 monclient(hunting): picked mon.noname-b con 0x7f6dec13a770 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:40:42.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df40e2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:42.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df40e2640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f6dec09f0f0 con 0x7f6dec137e40 2024-04-26T00:40:42.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df40e2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:42.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df40e2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6dec09f4c0 con 0x7f6dec13a770 2024-04-26T00:40:42.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df40e2640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:42.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df40e2640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f6dec0a00a0 con 0x7f6dec0a0260 2024-04-26T00:40:42.184 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df40e2640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:42.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df40e2640 10 monclient(hunting): authenticate will time out at 2223.483504s 2024-04-26T00:40:42.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df167d640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6dec137e40 msgr2=0x7f6dec134b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:40:42.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df167d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6dec137e40 0x7f6dec134b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:40:42.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df1e7e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6dec0a0260 0x7f6dec1cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:42.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df1e7e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6dec0a0260 0x7f6dec1cee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:44118/0 (socket says 172.21.15.37:44118) 2024-04-26T00:40:42.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df1e7e640 1 -- 172.21.15.37:0/577494754 learned_addr learned my addr 172.21.15.37:0/577494754 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:40:42.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df1e7e640 10 monclient(hunting): get_auth_request con 0x7f6dec0a0260 auth_method 0 2024-04-26T00:40:42.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df1e7e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:42.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df1e7e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:42.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df1e7e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:42.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df1e7e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:42.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df1e7e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:42.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df1e7e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:42.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df267f640 1 --2- 172.21.15.37:0/577494754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6dec13a770 0x7f6dec135060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:42.185 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df267f640 10 monclient(hunting): get_auth_request con 0x7f6dec13a770 auth_method 0 2024-04-26T00:40:42.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df267f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:42.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df267f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:42.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.173+0000 7f6df267f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:42.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.177+0000 7f6df1e7e640 10 monclient(hunting): handle_auth_done global_id 254156 payload 824 2024-04-26T00:40:42.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.177+0000 7f6df1e7e640 1 -- 172.21.15.37:0/577494754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6dec13a770 msgr2=0x7f6dec135060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:42.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.177+0000 7f6df1e7e640 1 --2- 172.21.15.37:0/577494754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6dec13a770 0x7f6dec135060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:42.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.177+0000 7f6df1e7e640 1 -- 172.21.15.37:0/577494754 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6dec137e40 msgr2=0x7f6dec134b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:40:42.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.177+0000 7f6df1e7e640 1 --2- 172.21.15.37:0/577494754 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6dec137e40 0x7f6dec134b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:42.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.177+0000 7f6df1e7e640 10 monclient: _finish_hunting 0 2024-04-26T00:40:42.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.177+0000 7f6df1e7e640 1 monclient: found mon.noname-c 2024-04-26T00:40:42.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.177+0000 7f6df1e7e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:42.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.177+0000 7f6df1e7e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:40:42.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.177+0000 7f6df1e7e640 1 -- 172.21.15.37:0/577494754 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dec0729d0 con 0x7f6dec0a0260 2024-04-26T00:40:42.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.177+0000 7f6df267f640 1 --2- 172.21.15.37:0/577494754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6dec13a770 0x7f6dec135060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:40:42.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.177+0000 7f6df1e7e640 1 -- 172.21.15.37:0/577494754 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6dec0a0260 msgr2=0x7f6dec1cee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:42.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.177+0000 7f6df1e7e640 1 --2- 172.21.15.37:0/577494754 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6dec0a0260 0x7f6dec1cee70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6de400b9e0 tx=0x7f6de400bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:42.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.177+0000 7f6df1e7e640 1 --2- 172.21.15.37:0/577494754 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6dec0a0260 0x7f6dec1cee70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6de400b9e0 tx=0x7f6de400bea0 comp rx=0 tx=0).queue_received will delay after 1.71409e+09 on 0x7f6de400cdb0 mon_map magic: 0 2024-04-26T00:40:42.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.177+0000 7f6dea7fc640 1 -- 172.21.15.37:0/577494754 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de400cdb0 con 0x7f6dec0a0260 2024-04-26T00:40:42.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.177+0000 7f6dea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:42.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.177+0000 7f6dea7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:40:42.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.177+0000 7f6dea7fc640 10 monclient: dump: 2024-04-26T00:40:42.187 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:42.187 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:42.187 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:42.187 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:42.187 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:42.187 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:42.187 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:42.187 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:42.188 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:42.188 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:42.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.177+0000 7f6dea7fc640 10 monclient: _finish_auth 0 2024-04-26T00:40:42.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.177+0000 7f6dea7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:40:42.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.177+0000 7f6dea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:42.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.177+0000 7f6dea7fc640 1 -- 172.21.15.37:0/577494754 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6de4010070 con 0x7f6dec0a0260 2024-04-26T00:40:42.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.177+0000 7f6dea7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:40:42.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.177+0000 7f6debfff640 4 set_mon_vals no callback set 2024-04-26T00:40:42.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.177+0000 7f6df40e2640 5 monclient: authenticate success, global_id 254156 2024-04-26T00:40:42.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.177+0000 7f6df40e2640 10 monclient: _renew_subs 2024-04-26T00:40:42.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.177+0000 7f6df40e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:40:42.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.177+0000 7f6df40e2640 1 -- 172.21.15.37:0/577494754 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dec135ed0 con 0x7f6dec0a0260 2024-04-26T00:40:42.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.177+0000 7f6df40e2640 10 monclient: _renew_subs 2024-04-26T00:40:42.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.177+0000 7f6df40e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:40:42.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.177+0000 7f6df40e2640 1 -- 172.21.15.37:0/577494754 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dec1204e0 con 0x7f6dec0a0260 2024-04-26T00:40:42.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.181+0000 7f6df40e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:42.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.181+0000 7f6df40e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:42.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.181+0000 7f6df40e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:40:42.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.181+0000 7f6df40e2640 1 -- 172.21.15.37:0/577494754 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dec066f40 con 0x7f6dec0a0260 2024-04-26T00:40:42.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.181+0000 7f6dea7fc640 1 -- 172.21.15.37:0/577494754 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de40156b0 con 0x7f6dec0a0260 2024-04-26T00:40:42.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.181+0000 7f6dea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:42.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.181+0000 7f6dea7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:40:42.189 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.181+0000 7f6dea7fc640 10 monclient: dump: 2024-04-26T00:40:42.189 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:42.189 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:42.189 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:42.189 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:42.189 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:42.189 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:42.189 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:42.190 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:42.190 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:42.190 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:42.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.181+0000 7f6dea7fc640 1 -- 172.21.15.37:0/577494754 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f6de401e3f0 con 0x7f6dec0a0260 2024-04-26T00:40:42.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.181+0000 7f6dea7fc640 1 --2- 172.21.15.37:0/577494754 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6dd8035e30 0x7f6dd80382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:42.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.181+0000 7f6df167d640 1 --2- 172.21.15.37:0/577494754 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6dd8035e30 0x7f6dd80382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:42.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.181+0000 7f6df167d640 10 monclient: get_auth_request con 0x7f6dd8035e30 auth_method 0 2024-04-26T00:40:42.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.181+0000 7f6dea7fc640 1 -- 172.21.15.37:0/577494754 DispatchQueue::entry inject delay of 0.002000 2024-04-26T00:40:42.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.181+0000 7f6df167d640 1 --2- 172.21.15.37:0/577494754 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6dd8035e30 0x7f6dd80382f0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f6de00ab000 tx=0x7f6de00a6070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:42.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.185+0000 7f6dea7fc640 1 -- 172.21.15.37:0/577494754 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(823..823 src has 705..823) ==== 4023+0+0 (secure 0 0 0) 0x7f6de4048ec0 con 0x7f6dec0a0260 2024-04-26T00:40:42.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.801+0000 7f6dea7fc640 1 -- 172.21.15.37:0/577494754 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6de4047bb0 con 0x7f6dec0a0260 2024-04-26T00:40:42.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.801+0000 7f6dea7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:42.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:42.801+0000 7f6dea7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:40:43.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:43.033+0000 7f6df40e2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:40:43.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:43.033+0000 7f6df40e2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:40:43.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:43.033+0000 7f6df40e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:40:43.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:43.033+0000 7f6df40e2640 1 -- 172.21.15.37:0/577494754 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6dec0715e0 con 0x7f6dec0a0260 2024-04-26T00:40:43.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:43.033+0000 7f6dea7fc640 1 -- 172.21.15.37:0/577494754 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f6de401d3a0 con 0x7f6dec0a0260 2024-04-26T00:40:43.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:43.033+0000 7f6dea7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:40:43.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:43.033+0000 7f6dea7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:40:43.035 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":152,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:40:43.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:43.033+0000 7f6df40e2640 1 -- 172.21.15.37:0/577494754 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6dd8035e30 msgr2=0x7f6dd80382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:43.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:43.033+0000 7f6df40e2640 1 --2- 172.21.15.37:0/577494754 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6dd8035e30 0x7f6dd80382f0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f6de00ab000 tx=0x7f6de00a6070 comp rx=0 tx=0).stop 2024-04-26T00:40:43.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:43.033+0000 7f6df40e2640 10 monclient: shutdown 2024-04-26T00:40:43.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:43.033+0000 7f6df40e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:40:43.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:43.033+0000 7f6df40e2640 1 -- 172.21.15.37:0/577494754 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6dec0a0260 msgr2=0x7f6dec1cee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:43.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:43.033+0000 7f6df40e2640 1 --2- 172.21.15.37:0/577494754 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6dec0a0260 0x7f6dec1cee70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6de400b9e0 tx=0x7f6de400bea0 comp rx=0 tx=0).stop 2024-04-26T00:40:43.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:43.037+0000 7f6df40e2640 1 -- 172.21.15.37:0/577494754 shutdown_connections 2024-04-26T00:40:43.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:43.037+0000 7f6df40e2640 1 --2- 172.21.15.37:0/577494754 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6dd8035e30 0x7f6dd80382f0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:43.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:43.037+0000 7f6df40e2640 1 --2- 172.21.15.37:0/577494754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6dec13a770 0x7f6dec135060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:43.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:43.037+0000 7f6df40e2640 1 --2- 172.21.15.37:0/577494754 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6dec137e40 0x7f6dec134b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:43.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:43.037+0000 7f6df40e2640 1 --2- 172.21.15.37:0/577494754 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6dec0a0260 0x7f6dec1cee70 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:43.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:43.037+0000 7f6df40e2640 1 -- 172.21.15.37:0/577494754 >> 172.21.15.37:0/577494754 conn(0x7f6dec0a1ad0 msgr2=0x7f6dec0a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:40:43.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:43.037+0000 7f6df40e2640 1 -- 172.21.15.37:0/577494754 shutdown_connections 2024-04-26T00:40:43.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:43.037+0000 7f6df40e2640 1 -- 172.21.15.37:0/577494754 wait complete. 2024-04-26T00:40:46.056 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:40:46.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71fe091640 10 monclient: get_monmap_and_config 2024-04-26T00:40:46.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71fe091640 10 monclient: build_initial_monmap 2024-04-26T00:40:46.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71fe091640 10 monclient: monmap: 2024-04-26T00:40:46.187 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:40:46.187 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:46.187 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:40:46.189538+0000 2024-04-26T00:40:46.187 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:40:46.189538+0000 2024-04-26T00:40:46.187 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:40:46.187 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:46.187 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:40:46.188 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:40:46.188 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:40:46.188 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:40:46.188 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:40:46.188 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:46.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71fe091640 1 Processor -- start 2024-04-26T00:40:46.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71fe091640 1 -- start start 2024-04-26T00:40:46.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71fe091640 10 monclient: init 2024-04-26T00:40:46.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71fe091640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:46.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71fe091640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-04-26T00:40:46.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71fe091640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f71f80a0260 0x7f71f80a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:46.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71fe091640 10 monclient(hunting): picked mon.noname-b con 0x7f71f80a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:40:46.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71fe091640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f71f8134b30 0x7f71f8134f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:46.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71fe091640 10 monclient(hunting): picked mon.noname-c con 0x7f71f8134b30 addr v2:172.21.15.45:3300/0 2024-04-26T00:40:46.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71fe091640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f71f8135490 addr v1:172.21.15.45:6789/0 2024-04-26T00:40:46.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71fe091640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71f80a1130 con 0x7f71f8135490 2024-04-26T00:40:46.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71fe091640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:46.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71fe091640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f71f809ec10 con 0x7f71f8134b30 2024-04-26T00:40:46.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71fe091640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:46.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71fe091640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f71f809f0f0 con 0x7f71f80a0260 2024-04-26T00:40:46.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71fe091640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:46.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71fe091640 10 monclient(hunting): authenticate will time out at 2227.495263s 2024-04-26T00:40:46.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71f77fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f71f80a0260 0x7f71f80a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:46.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71f77fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f71f80a0260 0x7f71f80a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:44132/0 (socket says 172.21.15.37:44132) 2024-04-26T00:40:46.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71f77fe640 1 -- 172.21.15.37:0/288108025 learned_addr learned my addr 172.21.15.37:0/288108025 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:40:46.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71f77fe640 10 monclient(hunting): get_auth_request con 0x7f71f80a0260 auth_method 0 2024-04-26T00:40:46.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:46.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71f77fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:46.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:46.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:46.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:46.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:46.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71f6ffd640 1 --2- 172.21.15.37:0/288108025 >> v2:172.21.15.45:3300/0 conn(0x7f71f8134b30 0x7f71f8134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:46.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71f77fe640 10 monclient(hunting): handle_auth_done global_id 254159 payload 274 2024-04-26T00:40:46.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71f77fe640 1 -- 172.21.15.37:0/288108025 >> v2:172.21.15.45:3300/0 conn(0x7f71f8134b30 msgr2=0x7f71f8134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:40:46.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71f77fe640 1 --2- 172.21.15.37:0/288108025 >> v2:172.21.15.45:3300/0 conn(0x7f71f8134b30 0x7f71f8134f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:46.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71f77fe640 1 -- 172.21.15.37:0/288108025 >> v1:172.21.15.45:6789/0 conn(0x7f71f8135490 legacy=0x7f71f813be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:40:46.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71f77fe640 10 monclient: _finish_hunting 0 2024-04-26T00:40:46.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71f77fe640 1 monclient: found mon.noname-b 2024-04-26T00:40:46.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:46.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71f77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:40:46.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71f77fe640 1 -- 172.21.15.37:0/288108025 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71f806cff0 con 0x7f71f80a0260 2024-04-26T00:40:46.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71f6ffd640 10 monclient: get_auth_request con 0x7f71f8134b30 auth_method 0 2024-04-26T00:40:46.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.185+0000 7f71f6ffd640 1 --2- 172.21.15.37:0/288108025 >> v2:172.21.15.45:3300/0 conn(0x7f71f8134b30 0x7f71f8134f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:40:46.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.193+0000 7f71f77fe640 1 -- 172.21.15.37:0/288108025 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f71f80a0260 msgr2=0x7f71f80a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:46.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.193+0000 7f71f77fe640 1 --2- 172.21.15.37:0/288108025 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f71f80a0260 0x7f71f80a0640 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f71e8009e50 tx=0x7f71e801c6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ad89292b59d0bb0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:46.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.193+0000 7f71f67fc640 1 -- 172.21.15.37:0/288108025 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71e80a9460 con 0x7f71f80a0260 2024-04-26T00:40:46.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.193+0000 7f71f67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:46.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.193+0000 7f71f67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:40:46.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.193+0000 7f71f67fc640 10 monclient: dump: 2024-04-26T00:40:46.196 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:46.196 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:46.196 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:46.196 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:46.196 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:46.196 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:46.196 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:46.197 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:46.197 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:46.197 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:46.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.193+0000 7f71f67fc640 10 monclient: _finish_auth 0 2024-04-26T00:40:46.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.193+0000 7f71f67fc640 10 monclient: _check_auth_tickets 2024-04-26T00:40:46.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.193+0000 7f71f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:46.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.193+0000 7f71f67fc640 1 -- 172.21.15.37:0/288108025 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71e80ae070 con 0x7f71f80a0260 2024-04-26T00:40:46.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.193+0000 7f71fe091640 5 monclient: authenticate success, global_id 254159 2024-04-26T00:40:46.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.193+0000 7f71fe091640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:40:46.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.193+0000 7f71f67fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:40:46.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.193+0000 7f71fe091640 10 monclient: get_monmap_and_config success 2024-04-26T00:40:46.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.193+0000 7f71fe091640 4 set_mon_vals no callback set 2024-04-26T00:40:46.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.193+0000 7f71f67fc640 1 -- 172.21.15.37:0/288108025 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71e80a9820 con 0x7f71f80a0260 2024-04-26T00:40:46.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.193+0000 7f71f67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:46.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.193+0000 7f71f67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:40:46.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.193+0000 7f71f67fc640 10 monclient: dump: 2024-04-26T00:40:46.198 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:46.198 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:46.198 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:46.198 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:46.198 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:46.198 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:46.198 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:46.198 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:46.198 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:46.198 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:46.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.193+0000 7f71fe091640 10 monclient: shutdown 2024-04-26T00:40:46.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.193+0000 7f71fe091640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:40:46.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.193+0000 7f71fe091640 1 -- 172.21.15.37:0/288108025 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f71f80a0260 msgr2=0x7f71f80a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:46.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.193+0000 7f71fe091640 1 --2- 172.21.15.37:0/288108025 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f71f80a0260 0x7f71f80a0640 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f71e8009e50 tx=0x7f71e801c6d0 comp rx=0 tx=0).stop 2024-04-26T00:40:46.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.193+0000 7f71fe091640 1 -- 172.21.15.37:0/288108025 shutdown_connections 2024-04-26T00:40:46.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.193+0000 7f71fe091640 1 --2- 172.21.15.37:0/288108025 >> v2:172.21.15.45:3300/0 conn(0x7f71f8134b30 0x7f71f8134f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:46.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.193+0000 7f71fe091640 1 --2- 172.21.15.37:0/288108025 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f71f80a0260 0x7f71f80a0640 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:46.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.193+0000 7f71fe091640 1 -- 172.21.15.37:0/288108025 >> 172.21.15.37:0/288108025 conn(0x7f71f80a1ad0 msgr2=0x7f71f80a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:40:46.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.193+0000 7f71fe091640 1 -- 172.21.15.37:0/288108025 shutdown_connections 2024-04-26T00:40:46.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.193+0000 7f71fe091640 1 -- 172.21.15.37:0/288108025 wait complete. 2024-04-26T00:40:46.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.193+0000 7f71fe091640 10 monclient: build_initial_monmap 2024-04-26T00:40:46.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.193+0000 7f71fe091640 10 monclient: monmap: 2024-04-26T00:40:46.204 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:40:46.204 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:40:46.204 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:40:46.204 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:40:46.204 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:40:46.204 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:40:46.204 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:40:46.204 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:40:46.204 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:40:46.204 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:46.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71fe091640 1 Processor -- start 2024-04-26T00:40:46.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71fe091640 1 -- start start 2024-04-26T00:40:46.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71fe091640 10 monclient: init 2024-04-26T00:40:46.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71fe091640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:46.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71fe091640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:40:46.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71fe091640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f71f80a0260 0x7f71f813c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:46.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71fe091640 10 monclient(hunting): picked mon.noname-c con 0x7f71f80a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:40:46.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71fe091640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f71f8134b30 0x7f71f809c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:46.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71fe091640 10 monclient(hunting): picked mon.noname-a con 0x7f71f8134b30 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:40:46.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71fe091640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71f8135490 0x7f71f809ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:46.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71fe091640 10 monclient(hunting): picked mon.noname-b con 0x7f71f8135490 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:40:46.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71fe091640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:46.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71fe091640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f71f809f430 con 0x7f71f8134b30 2024-04-26T00:40:46.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71fe091640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:46.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71fe091640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f71f809f2b0 con 0x7f71f8135490 2024-04-26T00:40:46.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71fe091640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:46.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71fe091640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f71f809fe90 con 0x7f71f80a0260 2024-04-26T00:40:46.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71fe091640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:46.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71fe091640 10 monclient(hunting): authenticate will time out at 2227.504853s 2024-04-26T00:40:46.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71f6ffd640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f71f8134b30 msgr2=0x7f71f809c500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:40:46.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71f6ffd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f71f8134b30 0x7f71f809c500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:40:46.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71f77fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f71f80a0260 0x7f71f813c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:46.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71f77fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f71f80a0260 0x7f71f813c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:44136/0 (socket says 172.21.15.37:44136) 2024-04-26T00:40:46.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71f77fe640 1 -- 172.21.15.37:0/2794937869 learned_addr learned my addr 172.21.15.37:0/2794937869 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:40:46.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71f77fe640 10 monclient(hunting): get_auth_request con 0x7f71f80a0260 auth_method 0 2024-04-26T00:40:46.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:46.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71f77fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:46.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:46.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:46.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:46.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:46.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71f7fff640 1 --2- 172.21.15.37:0/2794937869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71f8135490 0x7f71f809ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:46.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71f7fff640 10 monclient(hunting): get_auth_request con 0x7f71f8135490 auth_method 0 2024-04-26T00:40:46.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:46.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71f7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:46.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:46.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71f77fe640 10 monclient(hunting): handle_auth_done global_id 254162 payload 824 2024-04-26T00:40:46.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71f77fe640 1 -- 172.21.15.37:0/2794937869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71f8135490 msgr2=0x7f71f809ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:46.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71f77fe640 1 --2- 172.21.15.37:0/2794937869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71f8135490 0x7f71f809ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:46.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71f77fe640 1 -- 172.21.15.37:0/2794937869 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f71f8134b30 msgr2=0x7f71f809c500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:40:46.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71f77fe640 1 --2- 172.21.15.37:0/2794937869 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f71f8134b30 0x7f71f809c500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:46.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71f77fe640 10 monclient: _finish_hunting 0 2024-04-26T00:40:46.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71f77fe640 1 monclient: found mon.noname-c 2024-04-26T00:40:46.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:46.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71f77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:40:46.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71f77fe640 1 -- 172.21.15.37:0/2794937869 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71f809e0e0 con 0x7f71f80a0260 2024-04-26T00:40:46.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71f77fe640 1 -- 172.21.15.37:0/2794937869 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f71f80a0260 msgr2=0x7f71f813c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:46.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71f77fe640 1 --2- 172.21.15.37:0/2794937869 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f71f80a0260 0x7f71f813c180 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f71e801cf70 tx=0x7f71e8019470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:46.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71effff640 1 -- 172.21.15.37:0/2794937869 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71e80b8070 con 0x7f71f80a0260 2024-04-26T00:40:46.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71effff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:46.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71effff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:40:46.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71effff640 10 monclient: dump: 2024-04-26T00:40:46.208 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:46.208 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:46.208 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:46.208 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:46.208 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:46.208 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:46.208 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:46.208 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:46.208 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:46.208 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:46.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71effff640 10 monclient: _finish_auth 0 2024-04-26T00:40:46.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71effff640 10 monclient: _check_auth_tickets 2024-04-26T00:40:46.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:46.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71fe091640 5 monclient: authenticate success, global_id 254162 2024-04-26T00:40:46.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71effff640 1 -- 172.21.15.37:0/2794937869 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71e80a9cd0 con 0x7f71f80a0260 2024-04-26T00:40:46.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.197+0000 7f71fe091640 10 monclient: _renew_subs 2024-04-26T00:40:46.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.201+0000 7f71fe091640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:40:46.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.201+0000 7f71fe091640 1 -- 172.21.15.37:0/2794937869 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71f813aee0 con 0x7f71f80a0260 2024-04-26T00:40:46.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.201+0000 7f71effff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:40:46.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.201+0000 7f71effff640 1 -- 172.21.15.37:0/2794937869 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71e80bc650 con 0x7f71f80a0260 2024-04-26T00:40:46.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.201+0000 7f71effff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:46.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.201+0000 7f71effff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:40:46.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.201+0000 7f71effff640 10 monclient: dump: 2024-04-26T00:40:46.209 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:46.209 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:46.210 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:46.210 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:46.210 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:46.210 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:46.210 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:46.210 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:46.210 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:46.210 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:46.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.201+0000 7f71f5ffb640 4 set_mon_vals no callback set 2024-04-26T00:40:46.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.201+0000 7f71fe091640 10 monclient: _renew_subs 2024-04-26T00:40:46.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.201+0000 7f71fe091640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:40:46.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.201+0000 7f71fe091640 1 -- 172.21.15.37:0/2794937869 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f71f8139ff0 con 0x7f71f80a0260 2024-04-26T00:40:46.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.201+0000 7f71fe091640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:46.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.201+0000 7f71fe091640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:46.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.201+0000 7f71fe091640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:40:46.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.201+0000 7f71effff640 1 -- 172.21.15.37:0/2794937869 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f71e80c5430 con 0x7f71f80a0260 2024-04-26T00:40:46.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.201+0000 7f71fe091640 1 -- 172.21.15.37:0/2794937869 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71f8066f40 con 0x7f71f80a0260 2024-04-26T00:40:46.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.201+0000 7f71effff640 1 --2- 172.21.15.37:0/2794937869 >> v2:172.21.15.45:6812/2379918623 conn(0x7f71d8035eb0 0x7f71d8038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:46.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.201+0000 7f71effff640 1 -- 172.21.15.37:0/2794937869 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(823..823 src has 705..823) ==== 4023+0+0 (secure 0 0 0) 0x7f71e80e3090 con 0x7f71f80a0260 2024-04-26T00:40:46.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.201+0000 7f71f6ffd640 1 --2- 172.21.15.37:0/2794937869 >> v2:172.21.15.45:6812/2379918623 conn(0x7f71d8035eb0 0x7f71d8038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:46.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.201+0000 7f71f6ffd640 10 monclient: get_auth_request con 0x7f71d8035eb0 auth_method 0 2024-04-26T00:40:46.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.205+0000 7f71f6ffd640 1 --2- 172.21.15.37:0/2794937869 >> v2:172.21.15.45:6812/2379918623 conn(0x7f71d8035eb0 0x7f71d8038370 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f71e0006fd0 tx=0x7f71e0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:46.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.205+0000 7f71f6ffd640 1 -- 172.21.15.37:0/2794937869 >> v2:172.21.15.45:6812/2379918623 conn(0x7f71d8035eb0 msgr2=0x7f71d8038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:40:46.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.205+0000 7f71f6ffd640 1 -- 172.21.15.37:0/2794937869 >> v2:172.21.15.45:6812/2379918623 conn(0x7f71d8035eb0 msgr2=0x7f71d8038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:40:46.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.205+0000 7f71f6ffd640 1 --2- 172.21.15.37:0/2794937869 >> v2:172.21.15.45:6812/2379918623 conn(0x7f71d8035eb0 0x7f71d8038370 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f71e0006fd0 tx=0x7f71e0008040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:40:46.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.205+0000 7f71f6ffd640 1 --2- 172.21.15.37:0/2794937869 >> v2:172.21.15.45:6812/2379918623 conn(0x7f71d8035eb0 0x7f71d8038370 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f71e0006fd0 tx=0x7f71e0008040 comp rx=0 tx=0).stop 2024-04-26T00:40:46.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.205+0000 7f71effff640 1 -- 172.21.15.37:0/2794937869 >> v2:172.21.15.45:6812/2379918623 conn(0x7f71d8035eb0 msgr2=0x7f71d8038370 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:40:46.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.205+0000 7f71effff640 1 --2- 172.21.15.37:0/2794937869 >> v2:172.21.15.45:6812/2379918623 conn(0x7f71d8035eb0 0x7f71d8038370 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:46.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.205+0000 7f71effff640 1 -- 172.21.15.37:0/2794937869 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f71f80a1130 con 0x7f71f80a0260 2024-04-26T00:40:46.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.205+0000 7f71effff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:46.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.205+0000 7f71effff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:40:46.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.421+0000 7f71fe091640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:40:46.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.421+0000 7f71fe091640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:40:46.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.421+0000 7f71fe091640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:40:46.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.421+0000 7f71fe091640 1 -- 172.21.15.37:0/2794937869 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f71f81337c0 con 0x7f71f80a0260 2024-04-26T00:40:46.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.425+0000 7f71effff640 1 -- 172.21.15.37:0/2794937869 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1558 (secure 0 0 0) 0x7f71e80e1d80 con 0x7f71f80a0260 2024-04-26T00:40:46.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.425+0000 7f71effff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:40:46.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.425+0000 7f71effff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:40:46.426 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":152,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:40:46.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.425+0000 7f71fe091640 10 monclient: shutdown 2024-04-26T00:40:46.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.425+0000 7f71fe091640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:40:46.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.425+0000 7f71fe091640 1 -- 172.21.15.37:0/2794937869 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f71f80a0260 msgr2=0x7f71f813c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:46.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.425+0000 7f71fe091640 1 --2- 172.21.15.37:0/2794937869 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f71f80a0260 0x7f71f813c180 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f71e801cf70 tx=0x7f71e8019470 comp rx=0 tx=0).stop 2024-04-26T00:40:46.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.425+0000 7f71fe091640 1 -- 172.21.15.37:0/2794937869 shutdown_connections 2024-04-26T00:40:46.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.425+0000 7f71fe091640 1 --2- 172.21.15.37:0/2794937869 >> v2:172.21.15.45:6812/2379918623 conn(0x7f71d8035eb0 0x7f71d8038370 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:46.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.425+0000 7f71fe091640 1 --2- 172.21.15.37:0/2794937869 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71f8135490 0x7f71f809ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:46.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.425+0000 7f71fe091640 1 --2- 172.21.15.37:0/2794937869 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f71f8134b30 0x7f71f809c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:46.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.425+0000 7f71fe091640 1 --2- 172.21.15.37:0/2794937869 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f71f80a0260 0x7f71f813c180 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:46.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.425+0000 7f71fe091640 1 -- 172.21.15.37:0/2794937869 >> 172.21.15.37:0/2794937869 conn(0x7f71f80a1ad0 msgr2=0x7f71f80a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:40:46.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.425+0000 7f71fe091640 1 -- 172.21.15.37:0/2794937869 shutdown_connections 2024-04-26T00:40:46.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:46.425+0000 7f71fe091640 1 -- 172.21.15.37:0/2794937869 wait complete. 2024-04-26T00:40:49.443 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:40:49.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:49.573+0000 7f9e3d11c640 10 monclient: get_monmap_and_config 2024-04-26T00:40:49.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:49.573+0000 7f9e3d11c640 10 monclient: build_initial_monmap 2024-04-26T00:40:49.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:49.573+0000 7f9e3d11c640 10 monclient: monmap: 2024-04-26T00:40:49.575 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:40:49.575 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:49.575 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:40:49.577370+0000 2024-04-26T00:40:49.575 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:40:49.577370+0000 2024-04-26T00:40:49.575 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:40:49.575 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:49.575 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:40:49.575 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:40:49.575 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:40:49.575 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:40:49.575 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:40:49.575 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:49.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:49.573+0000 7f9e3d11c640 1 Processor -- start 2024-04-26T00:40:49.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:49.573+0000 7f9e3d11c640 1 -- start start 2024-04-26T00:40:49.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:49.573+0000 7f9e3d11c640 10 monclient: init 2024-04-26T00:40:49.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:49.573+0000 7f9e3d11c640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:49.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:49.573+0000 7f9e3d11c640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-04-26T00:40:49.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:49.573+0000 7f9e3d11c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9e38136da0 addr v1:172.21.15.37:6789/0 2024-04-26T00:40:49.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:49.573+0000 7f9e3d11c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9e38137180 0x7f9e381375c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:49.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:49.573+0000 7f9e3d11c640 10 monclient(hunting): picked mon.noname-a con 0x7f9e38137180 addr v2:172.21.15.37:3300/0 2024-04-26T00:40:49.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:49.573+0000 7f9e3d11c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9e3813c2d0 addr v1:172.21.15.45:6789/0 2024-04-26T00:40:49.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:49.573+0000 7f9e3d11c640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e3809b4f0 con 0x7f9e38136da0 2024-04-26T00:40:49.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:49.573+0000 7f9e3d11c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e3809b7d0 con 0x7f9e3813c2d0 2024-04-26T00:40:49.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:49.573+0000 7f9e3d11c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:49.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:49.573+0000 7f9e3d11c640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f9e3809b210 con 0x7f9e38137180 2024-04-26T00:40:49.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:49.573+0000 7f9e3d11c640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:49.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:49.573+0000 7f9e3d11c640 10 monclient(hunting): authenticate will time out at 2230.883776s 2024-04-26T00:40:49.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:49.573+0000 7f9e2ffff640 1 -- >> v2:172.21.15.37:3300/0 conn(0x7f9e38137180 msgr2=0x7f9e381375c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:40:49.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:49.573+0000 7f9e36d9d640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f9e38136da0 legacy=0x7f9e3809bed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:40:49.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:49.573+0000 7f9e2ffff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9e38137180 0x7f9e381375c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:40:49.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:49.577+0000 7f9e3759e640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f9e3813c2d0 0x7f9e3813e6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.37:36086/0 (socket says 172.21.15.37:36086) 2024-04-26T00:40:49.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:49.577+0000 7f9e3759e640 1 -- 172.21.15.37:0/264269494 learned_addr learned my addr 172.21.15.37:0/264269494 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:40:49.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:49.577+0000 7f9e3759e640 1 -- 172.21.15.37:0/264269494 >> v1:172.21.15.45:6789/0 conn(0x7f9e3813c2d0 legacy=0x7f9e3813e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:49.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:49.777+0000 7f9e2ffff640 1 -- 172.21.15.37:0/264269494 >> v2:172.21.15.37:3300/0 conn(0x7f9e38137180 msgr2=0x7f9e381375c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:40:49.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:49.777+0000 7f9e36d9d640 1 -- 172.21.15.37:0/264269494 >> v1:172.21.15.37:6789/0 conn(0x7f9e38136da0 legacy=0x7f9e3809bed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:40:49.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:49.777+0000 7f9e2ffff640 1 --2- 172.21.15.37:0/264269494 >> v2:172.21.15.37:3300/0 conn(0x7f9e38137180 0x7f9e381375c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:40:50.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:50.181+0000 7f9e2ffff640 1 -- 172.21.15.37:0/264269494 >> v2:172.21.15.37:3300/0 conn(0x7f9e38137180 msgr2=0x7f9e381375c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:40:50.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:50.181+0000 7f9e36d9d640 1 -- 172.21.15.37:0/264269494 >> v1:172.21.15.37:6789/0 conn(0x7f9e38136da0 legacy=0x7f9e3809bed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:40:50.181 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:50.181+0000 7f9e2ffff640 1 --2- 172.21.15.37:0/264269494 >> v2:172.21.15.37:3300/0 conn(0x7f9e38137180 0x7f9e381375c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:40:50.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:50.985+0000 7f9e2ffff640 1 -- 172.21.15.37:0/264269494 >> v2:172.21.15.37:3300/0 conn(0x7f9e38137180 msgr2=0x7f9e381375c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:40:50.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:50.985+0000 7f9e36d9d640 1 -- 172.21.15.37:0/264269494 >> v1:172.21.15.37:6789/0 conn(0x7f9e38136da0 legacy=0x7f9e3809bed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:40:50.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:50.985+0000 7f9e2ffff640 1 --2- 172.21.15.37:0/264269494 >> v2:172.21.15.37:3300/0 conn(0x7f9e38137180 0x7f9e381375c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:40:52.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.573+0000 7f9e3559a640 10 monclient(hunting): tick 2024-04-26T00:40:52.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.573+0000 7f9e3559a640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:40:52.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.573+0000 7f9e3559a640 1 monclient(hunting): continuing hunt 2024-04-26T00:40:52.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.573+0000 7f9e3559a640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:40:52.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.573+0000 7f9e3559a640 1 -- 172.21.15.37:0/264269494 >> v2:172.21.15.37:3300/0 conn(0x7f9e38137180 msgr2=0x7f9e381375c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:40:52.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.573+0000 7f9e3559a640 1 --2- 172.21.15.37:0/264269494 >> v2:172.21.15.37:3300/0 conn(0x7f9e38137180 0x7f9e381375c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:52.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.573+0000 7f9e3559a640 1 -- 172.21.15.37:0/264269494 >> v1:172.21.15.45:6789/0 conn(0x7f9e3813c2d0 legacy=0x7f9e3813e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:52.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.573+0000 7f9e3559a640 1 -- 172.21.15.37:0/264269494 >> v1:172.21.15.37:6789/0 conn(0x7f9e38136da0 legacy=0x7f9e3809bed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:40:52.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.573+0000 7f9e3559a640 10 monclient: _add_conns ranks=[2,4] 2024-04-26T00:40:52.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.573+0000 7f9e3559a640 1 --2- 172.21.15.37:0/264269494 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e18003750 0x7f9e18005ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:52.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.577+0000 7f9e3559a640 10 monclient(hunting): picked mon.noname-b con 0x7f9e18003750 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:40:52.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.577+0000 7f9e3559a640 1 --2- 172.21.15.37:0/264269494 >> v2:172.21.15.45:3300/0 conn(0x7f9e180062d0 0x7f9e18008790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:52.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.577+0000 7f9e3559a640 10 monclient(hunting): picked mon.noname-c con 0x7f9e180062d0 addr v2:172.21.15.45:3300/0 2024-04-26T00:40:52.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.577+0000 7f9e3559a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:52.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.577+0000 7f9e3559a640 1 -- 172.21.15.37:0/264269494 --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f9e3809b210 con 0x7f9e180062d0 2024-04-26T00:40:52.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.577+0000 7f9e3559a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:52.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.577+0000 7f9e3559a640 1 -- 172.21.15.37:0/264269494 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9e18008e30 con 0x7f9e18003750 2024-04-26T00:40:52.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.577+0000 7f9e3559a640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:52.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.577+0000 7f9e36d9d640 1 --2- 172.21.15.37:0/264269494 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e18003750 0x7f9e18005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:52.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.577+0000 7f9e36d9d640 10 monclient(hunting): get_auth_request con 0x7f9e18003750 auth_method 0 2024-04-26T00:40:52.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.577+0000 7f9e36d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:52.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.577+0000 7f9e36d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:52.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.577+0000 7f9e36d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:52.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.577+0000 7f9e2ffff640 1 --2- 172.21.15.37:0/264269494 >> v2:172.21.15.45:3300/0 conn(0x7f9e180062d0 0x7f9e18008790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:52.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.577+0000 7f9e2ffff640 10 monclient(hunting): get_auth_request con 0x7f9e180062d0 auth_method 0 2024-04-26T00:40:52.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.577+0000 7f9e2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:52.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.577+0000 7f9e2ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:52.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.577+0000 7f9e2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:52.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.577+0000 7f9e36d9d640 1 -- 172.21.15.37:0/264269494 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e18003750 msgr2=0x7f9e18005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:40:52.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.577+0000 7f9e36d9d640 1 -- 172.21.15.37:0/264269494 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e18003750 msgr2=0x7f9e18005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:40:52.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.577+0000 7f9e36d9d640 1 --2- 172.21.15.37:0/264269494 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e18003750 0x7f9e18005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:40:52.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.577+0000 7f9e36d9d640 1 --2- 172.21.15.37:0/264269494 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e18003750 0x7f9e18005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:40:52.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.577+0000 7f9e2ffff640 1 -- 172.21.15.37:0/264269494 >> v2:172.21.15.45:3300/0 conn(0x7f9e180062d0 msgr2=0x7f9e18008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:40:52.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.577+0000 7f9e2ffff640 1 -- 172.21.15.37:0/264269494 >> v2:172.21.15.45:3300/0 conn(0x7f9e180062d0 msgr2=0x7f9e18008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:40:52.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.577+0000 7f9e2ffff640 1 --2- 172.21.15.37:0/264269494 >> v2:172.21.15.45:3300/0 conn(0x7f9e180062d0 0x7f9e18008790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:40:52.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.577+0000 7f9e2ffff640 1 --2- 172.21.15.37:0/264269494 >> v2:172.21.15.45:3300/0 conn(0x7f9e180062d0 0x7f9e18008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:40:52.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.781+0000 7f9e36d9d640 1 --2- 172.21.15.37:0/264269494 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e18003750 0x7f9e18005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:52.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.781+0000 7f9e36d9d640 10 monclient(hunting): get_auth_request con 0x7f9e18003750 auth_method 0 2024-04-26T00:40:52.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.781+0000 7f9e36d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:52.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.781+0000 7f9e36d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:52.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.781+0000 7f9e36d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:40:52.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.781+0000 7f9e2ffff640 1 --2- 172.21.15.37:0/264269494 >> v2:172.21.15.45:3300/0 conn(0x7f9e180062d0 0x7f9e18008790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:52.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.781+0000 7f9e2ffff640 10 monclient(hunting): get_auth_request con 0x7f9e180062d0 auth_method 0 2024-04-26T00:40:52.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.781+0000 7f9e2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:52.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.781+0000 7f9e2ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:52.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.781+0000 7f9e2ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:40:52.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.781+0000 7f9e36d9d640 1 -- 172.21.15.37:0/264269494 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e18003750 msgr2=0x7f9e18005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:40:52.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.781+0000 7f9e36d9d640 1 -- 172.21.15.37:0/264269494 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e18003750 msgr2=0x7f9e18005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:40:52.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.781+0000 7f9e36d9d640 1 --2- 172.21.15.37:0/264269494 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e18003750 0x7f9e18005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:40:52.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.781+0000 7f9e2ffff640 1 -- 172.21.15.37:0/264269494 >> v2:172.21.15.45:3300/0 conn(0x7f9e180062d0 msgr2=0x7f9e18008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:40:52.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.781+0000 7f9e2ffff640 1 -- 172.21.15.37:0/264269494 >> v2:172.21.15.45:3300/0 conn(0x7f9e180062d0 msgr2=0x7f9e18008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:40:52.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.781+0000 7f9e36d9d640 1 --2- 172.21.15.37:0/264269494 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e18003750 0x7f9e18005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:40:52.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.781+0000 7f9e2ffff640 1 --2- 172.21.15.37:0/264269494 >> v2:172.21.15.45:3300/0 conn(0x7f9e180062d0 0x7f9e18008790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:40:52.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:52.781+0000 7f9e2ffff640 1 --2- 172.21.15.37:0/264269494 >> v2:172.21.15.45:3300/0 conn(0x7f9e180062d0 0x7f9e18008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:40:53.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:53.185+0000 7f9e36d9d640 1 --2- 172.21.15.37:0/264269494 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e18003750 0x7f9e18005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:53.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:53.185+0000 7f9e36d9d640 10 monclient(hunting): get_auth_request con 0x7f9e18003750 auth_method 0 2024-04-26T00:40:53.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:53.185+0000 7f9e36d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:53.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:53.185+0000 7f9e36d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:53.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:53.185+0000 7f9e36d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:40:53.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:53.185+0000 7f9e2ffff640 1 --2- 172.21.15.37:0/264269494 >> v2:172.21.15.45:3300/0 conn(0x7f9e180062d0 0x7f9e18008790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:53.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:53.185+0000 7f9e2ffff640 10 monclient(hunting): get_auth_request con 0x7f9e180062d0 auth_method 0 2024-04-26T00:40:53.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:53.185+0000 7f9e2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:53.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:53.185+0000 7f9e2ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:53.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:53.185+0000 7f9e2ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:40:53.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:53.185+0000 7f9e36d9d640 1 -- 172.21.15.37:0/264269494 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e18003750 msgr2=0x7f9e18005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:40:53.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:53.185+0000 7f9e36d9d640 1 -- 172.21.15.37:0/264269494 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e18003750 msgr2=0x7f9e18005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:40:53.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:53.185+0000 7f9e36d9d640 1 --2- 172.21.15.37:0/264269494 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e18003750 0x7f9e18005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:40:53.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:53.185+0000 7f9e2ffff640 1 -- 172.21.15.37:0/264269494 >> v2:172.21.15.45:3300/0 conn(0x7f9e180062d0 msgr2=0x7f9e18008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:40:53.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:53.185+0000 7f9e2ffff640 1 -- 172.21.15.37:0/264269494 >> v2:172.21.15.45:3300/0 conn(0x7f9e180062d0 msgr2=0x7f9e18008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:40:53.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:53.185+0000 7f9e2ffff640 1 --2- 172.21.15.37:0/264269494 >> v2:172.21.15.45:3300/0 conn(0x7f9e180062d0 0x7f9e18008790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:40:53.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:53.185+0000 7f9e2ffff640 1 --2- 172.21.15.37:0/264269494 >> v2:172.21.15.45:3300/0 conn(0x7f9e180062d0 0x7f9e18008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:40:53.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:53.185+0000 7f9e36d9d640 1 --2- 172.21.15.37:0/264269494 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e18003750 0x7f9e18005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:40:53.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:53.989+0000 7f9e36d9d640 1 --2- 172.21.15.37:0/264269494 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e18003750 0x7f9e18005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:53.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:53.989+0000 7f9e36d9d640 10 monclient(hunting): get_auth_request con 0x7f9e18003750 auth_method 0 2024-04-26T00:40:53.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:53.989+0000 7f9e36d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:53.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:53.989+0000 7f9e36d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:53.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:53.989+0000 7f9e36d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:40:53.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:53.989+0000 7f9e2ffff640 1 --2- 172.21.15.37:0/264269494 >> v2:172.21.15.45:3300/0 conn(0x7f9e180062d0 0x7f9e18008790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:53.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:53.989+0000 7f9e2ffff640 10 monclient(hunting): get_auth_request con 0x7f9e180062d0 auth_method 0 2024-04-26T00:40:53.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:53.989+0000 7f9e2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:53.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:53.989+0000 7f9e2ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:53.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:53.989+0000 7f9e2ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:40:53.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:53.989+0000 7f9e36d9d640 1 -- 172.21.15.37:0/264269494 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e18003750 msgr2=0x7f9e18005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:40:53.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:53.989+0000 7f9e36d9d640 1 -- 172.21.15.37:0/264269494 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e18003750 msgr2=0x7f9e18005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:40:53.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:53.989+0000 7f9e36d9d640 1 --2- 172.21.15.37:0/264269494 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e18003750 0x7f9e18005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:40:53.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:53.989+0000 7f9e2ffff640 1 -- 172.21.15.37:0/264269494 >> v2:172.21.15.45:3300/0 conn(0x7f9e180062d0 msgr2=0x7f9e18008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:40:53.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:53.989+0000 7f9e2ffff640 1 -- 172.21.15.37:0/264269494 >> v2:172.21.15.45:3300/0 conn(0x7f9e180062d0 msgr2=0x7f9e18008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:40:53.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:53.989+0000 7f9e2ffff640 1 --2- 172.21.15.37:0/264269494 >> v2:172.21.15.45:3300/0 conn(0x7f9e180062d0 0x7f9e18008790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:40:53.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:53.989+0000 7f9e2ffff640 1 --2- 172.21.15.37:0/264269494 >> v2:172.21.15.45:3300/0 conn(0x7f9e180062d0 0x7f9e18008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:40:53.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:53.989+0000 7f9e36d9d640 1 --2- 172.21.15.37:0/264269494 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e18003750 0x7f9e18005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:40:55.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.577+0000 7f9e3559a640 10 monclient(hunting): tick 2024-04-26T00:40:55.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.577+0000 7f9e3559a640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:40:55.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.577+0000 7f9e3559a640 1 monclient(hunting): continuing hunt 2024-04-26T00:40:55.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.577+0000 7f9e3559a640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:40:55.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.577+0000 7f9e3559a640 1 -- 172.21.15.37:0/264269494 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e18003750 msgr2=0x7f9e18005ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:40:55.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.577+0000 7f9e3559a640 1 --2- 172.21.15.37:0/264269494 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e18003750 0x7f9e18005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:55.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.577+0000 7f9e3559a640 1 -- 172.21.15.37:0/264269494 >> v2:172.21.15.45:3300/0 conn(0x7f9e180062d0 msgr2=0x7f9e18008790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:40:55.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.577+0000 7f9e3559a640 1 --2- 172.21.15.37:0/264269494 >> v2:172.21.15.45:3300/0 conn(0x7f9e180062d0 0x7f9e18008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:55.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.577+0000 7f9e3559a640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-04-26T00:40:55.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.577+0000 7f9e3759e640 1 -- 172.21.15.37:0/264269494 reap_dead start 2024-04-26T00:40:55.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.577+0000 7f9e3559a640 1 --2- 172.21.15.37:0/264269494 >> v2:172.21.15.45:3300/0 conn(0x7f9e18009230 0x7f9e1800b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:55.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.577+0000 7f9e3559a640 10 monclient(hunting): picked mon.noname-c con 0x7f9e18009230 addr v2:172.21.15.45:3300/0 2024-04-26T00:40:55.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.577+0000 7f9e3559a640 1 --2- 172.21.15.37:0/264269494 >> v2:172.21.15.37:3300/0 conn(0x7f9e38137180 0x7f9e18008750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:55.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.577+0000 7f9e3559a640 10 monclient(hunting): picked mon.noname-a con 0x7f9e38137180 addr v2:172.21.15.37:3300/0 2024-04-26T00:40:55.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.577+0000 7f9e3559a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9e38136da0 addr v1:172.21.15.37:6789/0 2024-04-26T00:40:55.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.577+0000 7f9e3559a640 1 -- 172.21.15.37:0/264269494 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e38137b00 con 0x7f9e38136da0 2024-04-26T00:40:55.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.577+0000 7f9e3559a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:55.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.577+0000 7f9e3559a640 1 -- 172.21.15.37:0/264269494 --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f9e3809b210 con 0x7f9e38137180 2024-04-26T00:40:55.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.577+0000 7f9e3559a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:55.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.577+0000 7f9e3559a640 1 -- 172.21.15.37:0/264269494 --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f9e18008e30 con 0x7f9e18009230 2024-04-26T00:40:55.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.577+0000 7f9e3559a640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:55.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.577+0000 7f9e2ffff640 1 -- 172.21.15.37:0/264269494 >> v2:172.21.15.37:3300/0 conn(0x7f9e38137180 msgr2=0x7f9e18008750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:40:55.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.577+0000 7f9e2ffff640 1 --2- 172.21.15.37:0/264269494 >> v2:172.21.15.37:3300/0 conn(0x7f9e38137180 0x7f9e18008750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:40:55.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.577+0000 7f9e3759e640 1 -- 172.21.15.37:0/264269494 >> v1:172.21.15.37:6789/0 conn(0x7f9e38136da0 legacy=0x7f9e1800dd30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:40:55.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.577+0000 7f9e36d9d640 1 --2- 172.21.15.37:0/264269494 >> v2:172.21.15.45:3300/0 conn(0x7f9e18009230 0x7f9e1800b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:55.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.577+0000 7f9e36d9d640 10 monclient(hunting): get_auth_request con 0x7f9e18009230 auth_method 0 2024-04-26T00:40:55.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.577+0000 7f9e36d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:55.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.577+0000 7f9e36d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:55.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.577+0000 7f9e36d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:55.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.577+0000 7f9e36d9d640 1 -- 172.21.15.37:0/264269494 >> v2:172.21.15.45:3300/0 conn(0x7f9e18009230 msgr2=0x7f9e1800b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:40:55.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.577+0000 7f9e36d9d640 1 -- 172.21.15.37:0/264269494 >> v2:172.21.15.45:3300/0 conn(0x7f9e18009230 msgr2=0x7f9e1800b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:40:55.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.577+0000 7f9e36d9d640 1 --2- 172.21.15.37:0/264269494 >> v2:172.21.15.45:3300/0 conn(0x7f9e18009230 0x7f9e1800b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:40:55.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.577+0000 7f9e36d9d640 1 --2- 172.21.15.37:0/264269494 >> v2:172.21.15.45:3300/0 conn(0x7f9e18009230 0x7f9e1800b680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:40:55.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.781+0000 7f9e3759e640 1 -- 172.21.15.37:0/264269494 >> v1:172.21.15.37:6789/0 conn(0x7f9e38136da0 legacy=0x7f9e1800dd30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:40:55.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.781+0000 7f9e2ffff640 1 -- 172.21.15.37:0/264269494 >> v2:172.21.15.37:3300/0 conn(0x7f9e38137180 msgr2=0x7f9e18008750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:40:55.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.781+0000 7f9e2ffff640 1 --2- 172.21.15.37:0/264269494 >> v2:172.21.15.37:3300/0 conn(0x7f9e38137180 0x7f9e18008750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:40:55.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.781+0000 7f9e36d9d640 1 --2- 172.21.15.37:0/264269494 >> v2:172.21.15.45:3300/0 conn(0x7f9e18009230 0x7f9e1800b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:55.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.781+0000 7f9e36d9d640 10 monclient(hunting): get_auth_request con 0x7f9e18009230 auth_method 0 2024-04-26T00:40:55.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.781+0000 7f9e36d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:55.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.781+0000 7f9e36d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:55.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.781+0000 7f9e36d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:40:55.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.781+0000 7f9e36d9d640 1 -- 172.21.15.37:0/264269494 >> v2:172.21.15.45:3300/0 conn(0x7f9e18009230 msgr2=0x7f9e1800b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:40:55.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.781+0000 7f9e36d9d640 1 -- 172.21.15.37:0/264269494 >> v2:172.21.15.45:3300/0 conn(0x7f9e18009230 msgr2=0x7f9e1800b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:40:55.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.781+0000 7f9e36d9d640 1 --2- 172.21.15.37:0/264269494 >> v2:172.21.15.45:3300/0 conn(0x7f9e18009230 0x7f9e1800b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:40:55.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:55.781+0000 7f9e36d9d640 1 --2- 172.21.15.37:0/264269494 >> v2:172.21.15.45:3300/0 conn(0x7f9e18009230 0x7f9e1800b680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:40:56.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.185+0000 7f9e2ffff640 1 -- 172.21.15.37:0/264269494 >> v2:172.21.15.37:3300/0 conn(0x7f9e38137180 msgr2=0x7f9e18008750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:40:56.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.185+0000 7f9e3759e640 1 -- 172.21.15.37:0/264269494 >> v1:172.21.15.37:6789/0 conn(0x7f9e38136da0 legacy=0x7f9e1800dd30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:40:56.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.185+0000 7f9e2ffff640 1 --2- 172.21.15.37:0/264269494 >> v2:172.21.15.37:3300/0 conn(0x7f9e38137180 0x7f9e18008750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:40:56.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.185+0000 7f9e36d9d640 1 --2- 172.21.15.37:0/264269494 >> v2:172.21.15.45:3300/0 conn(0x7f9e18009230 0x7f9e1800b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:56.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.185+0000 7f9e36d9d640 10 monclient(hunting): get_auth_request con 0x7f9e18009230 auth_method 0 2024-04-26T00:40:56.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.185+0000 7f9e36d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:56.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.185+0000 7f9e36d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:56.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.185+0000 7f9e36d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:40:56.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.185+0000 7f9e36d9d640 1 -- 172.21.15.37:0/264269494 >> v2:172.21.15.45:3300/0 conn(0x7f9e18009230 msgr2=0x7f9e1800b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:40:56.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.185+0000 7f9e36d9d640 1 -- 172.21.15.37:0/264269494 >> v2:172.21.15.45:3300/0 conn(0x7f9e18009230 msgr2=0x7f9e1800b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:40:56.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.185+0000 7f9e36d9d640 1 --2- 172.21.15.37:0/264269494 >> v2:172.21.15.45:3300/0 conn(0x7f9e18009230 0x7f9e1800b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:40:56.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.185+0000 7f9e36d9d640 1 --2- 172.21.15.37:0/264269494 >> v2:172.21.15.45:3300/0 conn(0x7f9e18009230 0x7f9e1800b680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:40:56.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.989+0000 7f9e3759e640 1 -- 172.21.15.37:0/264269494 >> v1:172.21.15.37:6789/0 conn(0x7f9e38136da0 legacy=0x7f9e1800dd30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:40:56.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.989+0000 7f9e2ffff640 1 -- 172.21.15.37:0/264269494 >> v2:172.21.15.37:3300/0 conn(0x7f9e38137180 msgr2=0x7f9e18008750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:40:56.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.989+0000 7f9e2ffff640 1 --2- 172.21.15.37:0/264269494 >> v2:172.21.15.37:3300/0 conn(0x7f9e38137180 0x7f9e18008750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:40:56.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.989+0000 7f9e36d9d640 1 --2- 172.21.15.37:0/264269494 >> v2:172.21.15.45:3300/0 conn(0x7f9e18009230 0x7f9e1800b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:56.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.989+0000 7f9e36d9d640 10 monclient(hunting): get_auth_request con 0x7f9e18009230 auth_method 0 2024-04-26T00:40:56.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.989+0000 7f9e36d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:56.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.989+0000 7f9e36d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:56.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.989+0000 7f9e36d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:40:56.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.989+0000 7f9e36d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:56.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.989+0000 7f9e36d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:56.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.989+0000 7f9e36d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:56.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.989+0000 7f9e36d9d640 10 monclient(hunting): handle_auth_done global_id 224401 payload 274 2024-04-26T00:40:56.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.989+0000 7f9e36d9d640 1 -- 172.21.15.37:0/264269494 >> v2:172.21.15.37:3300/0 conn(0x7f9e38137180 msgr2=0x7f9e18008750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:40:56.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.989+0000 7f9e36d9d640 1 --2- 172.21.15.37:0/264269494 >> v2:172.21.15.37:3300/0 conn(0x7f9e38137180 0x7f9e18008750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:56.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.989+0000 7f9e36d9d640 1 -- 172.21.15.37:0/264269494 >> v1:172.21.15.37:6789/0 conn(0x7f9e38136da0 legacy=0x7f9e1800dd30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:40:56.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.989+0000 7f9e36d9d640 10 monclient: _finish_hunting 0 2024-04-26T00:40:56.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.989+0000 7f9e36d9d640 1 monclient: found mon.noname-c 2024-04-26T00:40:56.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.989+0000 7f9e36d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:56.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.989+0000 7f9e36d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:40:56.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.989+0000 7f9e36d9d640 1 -- 172.21.15.37:0/264269494 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e3809b4f0 con 0x7f9e18009230 2024-04-26T00:40:56.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.997+0000 7f9e36d9d640 1 -- 172.21.15.37:0/264269494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e18009230 msgr2=0x7f9e1800b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:56.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.997+0000 7f9e36d9d640 1 --2- 172.21.15.37:0/264269494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e18009230 0x7f9e1800b680 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9e3809bc00 tx=0x7f9e2801d670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b8da3faf1dc3c10 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:56.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.997+0000 7f9e3659c640 1 -- 172.21.15.37:0/264269494 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e28003a50 con 0x7f9e18009230 2024-04-26T00:40:56.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.997+0000 7f9e3659c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:56.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.997+0000 7f9e3659c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:40:56.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.997+0000 7f9e3659c640 10 monclient: dump: 2024-04-26T00:40:56.999 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:57.000 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:57.000 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:57.000 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:57.000 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:57.000 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:57.000 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:57.000 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:57.000 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:57.000 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:57.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.997+0000 7f9e3659c640 10 monclient: _finish_auth 0 2024-04-26T00:40:57.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.997+0000 7f9e3659c640 10 monclient: _check_auth_tickets 2024-04-26T00:40:57.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.997+0000 7f9e3659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:57.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.997+0000 7f9e3659c640 1 -- 172.21.15.37:0/264269494 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e280b0070 con 0x7f9e18009230 2024-04-26T00:40:57.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.997+0000 7f9e3659c640 10 monclient: handle_config config(0 keys) 2024-04-26T00:40:57.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.997+0000 7f9e3659c640 1 -- 172.21.15.37:0/264269494 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e28003e10 con 0x7f9e18009230 2024-04-26T00:40:57.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.997+0000 7f9e3659c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:57.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.997+0000 7f9e3659c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:40:57.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.997+0000 7f9e3659c640 10 monclient: dump: 2024-04-26T00:40:57.001 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:57.001 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:57.001 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:57.001 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:57.001 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:57.001 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:57.001 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:57.001 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:57.001 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:57.001 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:57.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.997+0000 7f9e3d11c640 5 monclient: authenticate success, global_id 224401 2024-04-26T00:40:57.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.997+0000 7f9e3d11c640 10 monclient: get_monmap_and_config success 2024-04-26T00:40:57.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.997+0000 7f9e3d11c640 4 set_mon_vals no callback set 2024-04-26T00:40:57.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.997+0000 7f9e3d11c640 10 monclient: shutdown 2024-04-26T00:40:57.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.997+0000 7f9e3d11c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:40:57.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.997+0000 7f9e3d11c640 1 -- 172.21.15.37:0/264269494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e18009230 msgr2=0x7f9e1800b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:57.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.997+0000 7f9e3d11c640 1 --2- 172.21.15.37:0/264269494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e18009230 0x7f9e1800b680 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9e3809bc00 tx=0x7f9e2801d670 comp rx=0 tx=0).stop 2024-04-26T00:40:57.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.997+0000 7f9e3d11c640 1 -- 172.21.15.37:0/264269494 shutdown_connections 2024-04-26T00:40:57.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.997+0000 7f9e3d11c640 1 --2- 172.21.15.37:0/264269494 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e18009230 0x7f9e1800b680 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:57.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.997+0000 7f9e3d11c640 1 --2- 172.21.15.37:0/264269494 >> v2:172.21.15.37:3300/0 conn(0x7f9e38137180 0x7f9e18008750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:57.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.997+0000 7f9e3d11c640 1 -- 172.21.15.37:0/264269494 >> 172.21.15.37:0/264269494 conn(0x7f9e380a5f00 msgr2=0x7f9e380a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:40:57.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.997+0000 7f9e3d11c640 1 -- 172.21.15.37:0/264269494 shutdown_connections 2024-04-26T00:40:57.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:56.997+0000 7f9e3d11c640 1 -- 172.21.15.37:0/264269494 wait complete. 2024-04-26T00:40:57.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.001+0000 7f9e3d11c640 10 monclient: build_initial_monmap 2024-04-26T00:40:57.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.001+0000 7f9e3d11c640 10 monclient: monmap: 2024-04-26T00:40:57.004 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:40:57.004 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:40:57.004 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:40:57.004 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:40:57.004 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:40:57.004 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:40:57.004 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:40:57.004 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:40:57.004 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:40:57.004 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:57.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.001+0000 7f9e3d11c640 1 Processor -- start 2024-04-26T00:40:57.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.001+0000 7f9e3d11c640 1 -- start start 2024-04-26T00:40:57.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.001+0000 7f9e3d11c640 10 monclient: init 2024-04-26T00:40:57.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.001+0000 7f9e3d11c640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:57.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.001+0000 7f9e3d11c640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:40:57.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.001+0000 7f9e3d11c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e18009230 0x7f9e381cf700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:57.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.001+0000 7f9e3d11c640 10 monclient(hunting): picked mon.noname-a con 0x7f9e18009230 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:40:57.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.001+0000 7f9e3d11c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e38136da0 0x7f9e381cfc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:57.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.001+0000 7f9e3d11c640 10 monclient(hunting): picked mon.noname-b con 0x7f9e38136da0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:40:57.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.001+0000 7f9e3d11c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e38137180 0x7f9e3809ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:57.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.001+0000 7f9e3d11c640 10 monclient(hunting): picked mon.noname-c con 0x7f9e38137180 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:40:57.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.001+0000 7f9e36d9d640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e18009230 msgr2=0x7f9e381cf700 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:40:57.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.001+0000 7f9e3d11c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:57.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.001+0000 7f9e3d11c640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9e38133f20 con 0x7f9e18009230 2024-04-26T00:40:57.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.001+0000 7f9e36d9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e18009230 0x7f9e381cf700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:40:57.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.001+0000 7f9e3d11c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:57.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.001+0000 7f9e3d11c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9e38133da0 con 0x7f9e38136da0 2024-04-26T00:40:57.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.001+0000 7f9e3d11c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:57.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.001+0000 7f9e3d11c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9e381340a0 con 0x7f9e38137180 2024-04-26T00:40:57.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.001+0000 7f9e3d11c640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:57.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.001+0000 7f9e3d11c640 10 monclient(hunting): authenticate will time out at 2238.311675s 2024-04-26T00:40:57.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.001+0000 7f9e3759e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e38137180 0x7f9e3809ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:57.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.001+0000 7f9e3759e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e38137180 0x7f9e3809ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:58598/0 (socket says 172.21.15.37:58598) 2024-04-26T00:40:57.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.001+0000 7f9e3759e640 1 -- 172.21.15.37:0/2920020503 learned_addr learned my addr 172.21.15.37:0/2920020503 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:40:57.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.001+0000 7f9e3759e640 10 monclient(hunting): get_auth_request con 0x7f9e38137180 auth_method 0 2024-04-26T00:40:57.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.001+0000 7f9e3759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:57.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.001+0000 7f9e3759e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:57.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.001+0000 7f9e3759e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:57.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e2ffff640 1 --2- 172.21.15.37:0/2920020503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e38136da0 0x7f9e381cfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:57.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e2ffff640 10 monclient(hunting): get_auth_request con 0x7f9e38136da0 auth_method 0 2024-04-26T00:40:57.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:57.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e2ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:57.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:57.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e3759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:57.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e3759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:57.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e3759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:57.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:57.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:57.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:57.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e3759e640 10 monclient(hunting): handle_auth_done global_id 254165 payload 824 2024-04-26T00:40:57.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e3759e640 1 -- 172.21.15.37:0/2920020503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e38136da0 msgr2=0x7f9e381cfc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:57.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e3759e640 1 --2- 172.21.15.37:0/2920020503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e38136da0 0x7f9e381cfc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:57.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e3759e640 1 -- 172.21.15.37:0/2920020503 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e18009230 msgr2=0x7f9e381cf700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:40:57.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e3759e640 1 --2- 172.21.15.37:0/2920020503 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e18009230 0x7f9e381cf700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:57.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e3759e640 10 monclient: _finish_hunting 0 2024-04-26T00:40:57.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e3759e640 1 monclient: found mon.noname-c 2024-04-26T00:40:57.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e3759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:57.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e3759e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:40:57.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e3759e640 1 -- 172.21.15.37:0/2920020503 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e38121f90 con 0x7f9e38137180 2024-04-26T00:40:57.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e3759e640 1 -- 172.21.15.37:0/2920020503 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e38137180 msgr2=0x7f9e3809ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:57.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e3759e640 1 --2- 172.21.15.37:0/2920020503 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e38137180 0x7f9e3809ea00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9e380a5400 tx=0x7f9e3000a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:57.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e2f7fe640 1 -- 172.21.15.37:0/2920020503 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e30017070 con 0x7f9e38137180 2024-04-26T00:40:57.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e2f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:57.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e2f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:40:57.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e2f7fe640 10 monclient: dump: 2024-04-26T00:40:57.013 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:57.013 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:57.013 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:57.013 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:57.013 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:57.013 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:57.013 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:57.013 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:57.013 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:57.013 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:57.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e2f7fe640 10 monclient: _finish_auth 0 2024-04-26T00:40:57.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e2f7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:40:57.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:57.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e2f7fe640 1 -- 172.21.15.37:0/2920020503 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e3000ad10 con 0x7f9e38137180 2024-04-26T00:40:57.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e2f7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:40:57.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e2f7fe640 1 -- 172.21.15.37:0/2920020503 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e30012620 con 0x7f9e38137180 2024-04-26T00:40:57.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e2f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:57.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e2f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:40:57.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e2f7fe640 10 monclient: dump: 2024-04-26T00:40:57.014 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:57.014 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:57.014 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:57.014 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:57.014 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:57.015 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:57.015 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:57.015 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:57.015 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:57.015 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:57.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e3d11c640 5 monclient: authenticate success, global_id 254165 2024-04-26T00:40:57.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e3d11c640 10 monclient: _renew_subs 2024-04-26T00:40:57.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e3d11c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:40:57.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e35d9b640 4 set_mon_vals no callback set 2024-04-26T00:40:57.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e3d11c640 1 -- 172.21.15.37:0/2920020503 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e3809aca0 con 0x7f9e38137180 2024-04-26T00:40:57.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e3d11c640 10 monclient: _renew_subs 2024-04-26T00:40:57.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e3d11c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:40:57.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.005+0000 7f9e3d11c640 1 -- 172.21.15.37:0/2920020503 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e38120310 con 0x7f9e38137180 2024-04-26T00:40:57.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.009+0000 7f9e2f7fe640 1 -- 172.21.15.37:0/2920020503 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9e3001a070 con 0x7f9e38137180 2024-04-26T00:40:57.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.009+0000 7f9e3d11c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:57.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.009+0000 7f9e3d11c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:57.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.009+0000 7f9e3d11c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:40:57.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.009+0000 7f9e3d11c640 1 -- 172.21.15.37:0/2920020503 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e38066f40 con 0x7f9e38137180 2024-04-26T00:40:57.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.009+0000 7f9e2f7fe640 1 --2- 172.21.15.37:0/2920020503 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9e18037160 0x7f9e18039620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:57.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.009+0000 7f9e2f7fe640 1 -- 172.21.15.37:0/2920020503 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(823..823 src has 705..823) ==== 4023+0+0 (secure 0 0 0) 0x7f9e30048730 con 0x7f9e38137180 2024-04-26T00:40:57.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.009+0000 7f9e36d9d640 1 --2- 172.21.15.37:0/2920020503 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9e18037160 0x7f9e18039620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:57.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.009+0000 7f9e36d9d640 10 monclient: get_auth_request con 0x7f9e18037160 auth_method 0 2024-04-26T00:40:57.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.013+0000 7f9e36d9d640 1 --2- 172.21.15.37:0/2920020503 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9e18037160 0x7f9e18039620 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f9e1800bc90 tx=0x7f9e28010040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:57.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.013+0000 7f9e2f7fe640 1 -- 172.21.15.37:0/2920020503 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9e30047420 con 0x7f9e38137180 2024-04-26T00:40:57.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.013+0000 7f9e2f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:57.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.013+0000 7f9e2f7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:40:57.262 INFO:tasks.rados.rados.0.smithi045.stdout:3653: done (1 left) 2024-04-26T00:40:57.262 INFO:tasks.rados.rados.0.smithi045.stdout:3654: done (0 left) 2024-04-26T00:40:57.262 INFO:tasks.rados.rados.0.smithi045.stdout:3655: delete oid 23 current snap is 404 2024-04-26T00:40:57.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.265+0000 7f9e3d11c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:40:57.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.265+0000 7f9e3d11c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:40:57.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.265+0000 7f9e3d11c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:40:57.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.265+0000 7f9e3d11c640 1 -- 172.21.15.37:0/2920020503 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9e38121a90 con 0x7f9e38137180 2024-04-26T00:40:57.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.265+0000 7f9e2f7fe640 1 -- 172.21.15.37:0/2920020503 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f9e30007a50 con 0x7f9e38137180 2024-04-26T00:40:57.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.265+0000 7f9e2f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:40:57.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.265+0000 7f9e2f7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:40:57.268 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":154,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:40:57.269 INFO:tasks.rados.rados.0.smithi045.stdout:3655: done (0 left) 2024-04-26T00:40:57.269 INFO:tasks.rados.rados.0.smithi045.stdout:3656: read oid 40 snap -1 2024-04-26T00:40:57.269 INFO:tasks.rados.rados.0.smithi045.stdout:3656: expect (ObjNum 1239 snap 384 seq_num 1239) 2024-04-26T00:40:57.269 INFO:tasks.rados.rados.0.smithi045.stdout:3657: delete oid 48 current snap is 404 2024-04-26T00:40:57.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.269+0000 7f9e3d11c640 1 -- 172.21.15.37:0/2920020503 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9e18037160 msgr2=0x7f9e18039620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:57.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.269+0000 7f9e3d11c640 1 --2- 172.21.15.37:0/2920020503 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9e18037160 0x7f9e18039620 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f9e1800bc90 tx=0x7f9e28010040 comp rx=0 tx=0).stop 2024-04-26T00:40:57.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.269+0000 7f9e3d11c640 10 monclient: shutdown 2024-04-26T00:40:57.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.269+0000 7f9e3d11c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:40:57.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.269+0000 7f9e3d11c640 1 -- 172.21.15.37:0/2920020503 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e38137180 msgr2=0x7f9e3809ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:57.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.269+0000 7f9e3d11c640 1 --2- 172.21.15.37:0/2920020503 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e38137180 0x7f9e3809ea00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9e380a5400 tx=0x7f9e3000a430 comp rx=0 tx=0).stop 2024-04-26T00:40:57.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.269+0000 7f9e3d11c640 1 -- 172.21.15.37:0/2920020503 shutdown_connections 2024-04-26T00:40:57.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.269+0000 7f9e3d11c640 1 --2- 172.21.15.37:0/2920020503 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9e18037160 0x7f9e18039620 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:57.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.269+0000 7f9e3d11c640 1 --2- 172.21.15.37:0/2920020503 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9e38137180 0x7f9e3809ea00 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:57.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.269+0000 7f9e3d11c640 1 --2- 172.21.15.37:0/2920020503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e38136da0 0x7f9e381cfc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:57.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.269+0000 7f9e3d11c640 1 --2- 172.21.15.37:0/2920020503 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9e18009230 0x7f9e381cf700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:57.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.269+0000 7f9e3d11c640 1 -- 172.21.15.37:0/2920020503 >> 172.21.15.37:0/2920020503 conn(0x7f9e380a5f00 msgr2=0x7f9e38133370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:40:57.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.269+0000 7f9e3d11c640 1 -- 172.21.15.37:0/2920020503 shutdown_connections 2024-04-26T00:40:57.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.269+0000 7f9e3d11c640 1 -- 172.21.15.37:0/2920020503 wait complete. 2024-04-26T00:40:57.278 INFO:tasks.rados.rados.0.smithi045.stdout:3656: expect (ObjNum 1239 snap 384 seq_num 1239) 2024-04-26T00:40:57.288 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-26T00:40:57.288 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-04-26T00:40:57.289 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:40:57.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f418236c640 10 monclient: get_monmap_and_config 2024-04-26T00:40:57.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f418236c640 10 monclient: build_initial_monmap 2024-04-26T00:40:57.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f418236c640 10 monclient: monmap: 2024-04-26T00:40:57.414 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:40:57.414 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:57.415 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:40:57.416790+0000 2024-04-26T00:40:57.415 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:40:57.416790+0000 2024-04-26T00:40:57.415 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:40:57.415 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:57.415 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:40:57.415 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:40:57.415 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:40:57.415 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:40:57.415 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:40:57.415 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:57.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f418236c640 1 Processor -- start 2024-04-26T00:40:57.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f418236c640 1 -- start start 2024-04-26T00:40:57.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f418236c640 10 monclient: init 2024-04-26T00:40:57.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f418236c640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:57.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f418236c640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-04-26T00:40:57.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f418236c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f417c0a3d90 0x7f417c09bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:57.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f418236c640 10 monclient(hunting): picked mon.noname-a con 0x7f417c0a3d90 addr v2:172.21.15.37:3300/0 2024-04-26T00:40:57.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f418236c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f417c0a4170 0x7f417c0a45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:57.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f418236c640 10 monclient(hunting): picked mon.noname-c con 0x7f417c0a4170 addr v2:172.21.15.45:3300/0 2024-04-26T00:40:57.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f418236c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f417c133b20 0x7f417c133f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:57.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f418236c640 10 monclient(hunting): picked mon.noname-b con 0x7f417c133b20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:40:57.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f418236c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:57.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f418236c640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f417c0a5150 con 0x7f417c0a3d90 2024-04-26T00:40:57.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f418236c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:57.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f418236c640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f417c09b280 con 0x7f417c0a4170 2024-04-26T00:40:57.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f418236c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:57.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f418236c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f417c09b560 con 0x7f417c133b20 2024-04-26T00:40:57.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f418236c640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:57.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f418236c640 10 monclient(hunting): authenticate will time out at 2238.722515s 2024-04-26T00:40:57.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f417bfff640 1 -- >> v2:172.21.15.37:3300/0 conn(0x7f417c0a3d90 msgr2=0x7f417c09bf40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:40:57.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f417bfff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f417c0a3d90 0x7f417c09bf40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:40:57.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f4180909640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f417c133b20 0x7f417c133f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:57.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f4180909640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f417c133b20 0x7f417c133f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:58602/0 (socket says 172.21.15.37:58602) 2024-04-26T00:40:57.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f4180909640 1 -- 172.21.15.37:0/4207825737 learned_addr learned my addr 172.21.15.37:0/4207825737 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:40:57.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f4180909640 10 monclient(hunting): get_auth_request con 0x7f417c133b20 auth_method 0 2024-04-26T00:40:57.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f4180909640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:57.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f4180909640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:57.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f4180909640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:57.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f4180909640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:57.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f4180909640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:57.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f4180909640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:57.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f417b7fe640 1 --2- 172.21.15.37:0/4207825737 >> v2:172.21.15.45:3300/0 conn(0x7f417c0a4170 0x7f417c0a45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:57.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f417b7fe640 10 monclient(hunting): get_auth_request con 0x7f417c0a4170 auth_method 0 2024-04-26T00:40:57.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f417b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:57.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f417b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:57.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f417b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:57.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f4180909640 10 monclient(hunting): handle_auth_done global_id 254168 payload 274 2024-04-26T00:40:57.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f4180909640 1 -- 172.21.15.37:0/4207825737 >> v2:172.21.15.45:3300/0 conn(0x7f417c0a4170 msgr2=0x7f417c0a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:40:57.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f4180909640 1 --2- 172.21.15.37:0/4207825737 >> v2:172.21.15.45:3300/0 conn(0x7f417c0a4170 0x7f417c0a45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:57.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f4180909640 1 -- 172.21.15.37:0/4207825737 >> v2:172.21.15.37:3300/0 conn(0x7f417c0a3d90 msgr2=0x7f417c09bf40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:40:57.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f4180909640 1 --2- 172.21.15.37:0/4207825737 >> v2:172.21.15.37:3300/0 conn(0x7f417c0a3d90 0x7f417c09bf40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:57.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f4180909640 10 monclient: _finish_hunting 0 2024-04-26T00:40:57.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f4180909640 1 monclient: found mon.noname-b 2024-04-26T00:40:57.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f4180909640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:57.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f4180909640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:40:57.421 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.413+0000 7f4180909640 1 -- 172.21.15.37:0/4207825737 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f417c0a4b20 con 0x7f417c133b20 2024-04-26T00:40:57.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.417+0000 7f4180909640 1 -- 172.21.15.37:0/4207825737 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f417c133b20 msgr2=0x7f417c133f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:57.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.417+0000 7f4180909640 1 --2- 172.21.15.37:0/4207825737 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f417c133b20 0x7f417c133f90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f417c09d140 tx=0x7f417401c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b939e54c2b1538cd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:57.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.417+0000 7f417affd640 1 -- 172.21.15.37:0/4207825737 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41740b0070 con 0x7f417c133b20 2024-04-26T00:40:57.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.417+0000 7f417affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:57.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.417+0000 7f417affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:40:57.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.417+0000 7f417affd640 10 monclient: dump: 2024-04-26T00:40:57.425 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:57.425 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:57.425 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:57.425 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:57.425 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:57.425 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:57.425 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:57.425 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:57.425 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:57.425 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:57.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.417+0000 7f417affd640 10 monclient: _finish_auth 0 2024-04-26T00:40:57.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.417+0000 7f417affd640 10 monclient: _check_auth_tickets 2024-04-26T00:40:57.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.417+0000 7f417affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:57.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.417+0000 7f417affd640 1 -- 172.21.15.37:0/4207825737 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41740ab600 con 0x7f417c133b20 2024-04-26T00:40:57.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.417+0000 7f417affd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:40:57.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.417+0000 7f418236c640 5 monclient: authenticate success, global_id 254168 2024-04-26T00:40:57.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.417+0000 7f417affd640 1 -- 172.21.15.37:0/4207825737 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41740ab9d0 con 0x7f417c133b20 2024-04-26T00:40:57.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.417+0000 7f418236c640 10 monclient: get_monmap_and_config success 2024-04-26T00:40:57.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.417+0000 7f418236c640 4 set_mon_vals no callback set 2024-04-26T00:40:57.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.417+0000 7f417affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:57.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.417+0000 7f417affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:40:57.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.417+0000 7f417affd640 10 monclient: dump: 2024-04-26T00:40:57.426 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:57.426 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:57.426 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:57.426 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:57.427 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:57.427 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:57.427 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:57.427 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:57.427 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:57.427 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:57.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.417+0000 7f418236c640 10 monclient: shutdown 2024-04-26T00:40:57.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.417+0000 7f418236c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:40:57.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.417+0000 7f418236c640 1 -- 172.21.15.37:0/4207825737 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f417c133b20 msgr2=0x7f417c133f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:57.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.417+0000 7f418236c640 1 --2- 172.21.15.37:0/4207825737 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f417c133b20 0x7f417c133f90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f417c09d140 tx=0x7f417401c400 comp rx=0 tx=0).stop 2024-04-26T00:40:57.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f418236c640 1 -- 172.21.15.37:0/4207825737 shutdown_connections 2024-04-26T00:40:57.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f418236c640 1 --2- 172.21.15.37:0/4207825737 >> v2:172.21.15.45:3300/0 conn(0x7f417c0a4170 0x7f417c0a45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:57.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f418236c640 1 --2- 172.21.15.37:0/4207825737 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f417c133b20 0x7f417c133f90 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:57.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f418236c640 1 --2- 172.21.15.37:0/4207825737 >> v2:172.21.15.37:3300/0 conn(0x7f417c0a3d90 0x7f417c09bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:57.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f418236c640 1 -- 172.21.15.37:0/4207825737 >> 172.21.15.37:0/4207825737 conn(0x7f417c0aa0a0 msgr2=0x7f417c0a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:40:57.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f418236c640 1 -- 172.21.15.37:0/4207825737 shutdown_connections 2024-04-26T00:40:57.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f418236c640 1 -- 172.21.15.37:0/4207825737 wait complete. 2024-04-26T00:40:57.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f418236c640 10 monclient: build_initial_monmap 2024-04-26T00:40:57.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f418236c640 10 monclient: monmap: 2024-04-26T00:40:57.428 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:40:57.428 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:40:57.428 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:40:57.428 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:40:57.428 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:40:57.428 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:40:57.428 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:40:57.428 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:40:57.428 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:40:57.428 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:57.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f418236c640 1 Processor -- start 2024-04-26T00:40:57.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f418236c640 1 -- start start 2024-04-26T00:40:57.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f418236c640 10 monclient: init 2024-04-26T00:40:57.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f418236c640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:57.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f418236c640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:40:57.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f418236c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f417c0a3d90 0x7f417c132e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:57.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f418236c640 10 monclient(hunting): picked mon.noname-c con 0x7f417c0a3d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:40:57.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f418236c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f417c0a4170 0x7f417c1333a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:57.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f418236c640 10 monclient(hunting): picked mon.noname-b con 0x7f417c0a4170 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:40:57.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f418236c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f417c133b20 0x7f417c1cf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:57.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f418236c640 10 monclient(hunting): picked mon.noname-a con 0x7f417c133b20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:40:57.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f418236c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:57.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f418236c640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f417c140c10 con 0x7f417c133b20 2024-04-26T00:40:57.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f418236c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:57.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f418236c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f417c09c690 con 0x7f417c0a4170 2024-04-26T00:40:57.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f418236c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:57.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f418236c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f417c140d90 con 0x7f417c0a3d90 2024-04-26T00:40:57.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f418236c640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:57.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f418236c640 10 monclient(hunting): authenticate will time out at 2238.729601s 2024-04-26T00:40:57.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f4180909640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f417c133b20 msgr2=0x7f417c1cf1a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:40:57.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f4180909640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f417c133b20 0x7f417c1cf1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:40:57.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f417bfff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f417c0a3d90 0x7f417c132e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:57.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f417bfff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f417c0a3d90 0x7f417c132e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:58614/0 (socket says 172.21.15.37:58614) 2024-04-26T00:40:57.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f417bfff640 1 -- 172.21.15.37:0/43181774 learned_addr learned my addr 172.21.15.37:0/43181774 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:40:57.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f417bfff640 10 monclient(hunting): get_auth_request con 0x7f417c0a3d90 auth_method 0 2024-04-26T00:40:57.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f417bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:57.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f417bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:57.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f417bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:57.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f417bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:57.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f417bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:57.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f417bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:57.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f417b7fe640 1 --2- 172.21.15.37:0/43181774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f417c0a4170 0x7f417c1333a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:57.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f417b7fe640 10 monclient(hunting): get_auth_request con 0x7f417c0a4170 auth_method 0 2024-04-26T00:40:57.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f417b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:57.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f417b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:57.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f417b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:57.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f417bfff640 10 monclient(hunting): handle_auth_done global_id 254171 payload 824 2024-04-26T00:40:57.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f417bfff640 1 -- 172.21.15.37:0/43181774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f417c0a4170 msgr2=0x7f417c1333a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:57.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f417bfff640 1 --2- 172.21.15.37:0/43181774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f417c0a4170 0x7f417c1333a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:57.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f417bfff640 1 -- 172.21.15.37:0/43181774 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f417c133b20 msgr2=0x7f417c1cf1a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:40:57.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f417bfff640 1 --2- 172.21.15.37:0/43181774 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f417c133b20 0x7f417c1cf1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:57.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f417bfff640 10 monclient: _finish_hunting 0 2024-04-26T00:40:57.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f417bfff640 1 monclient: found mon.noname-c 2024-04-26T00:40:57.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f417bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:57.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f417bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:40:57.432 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.421+0000 7f417bfff640 1 -- 172.21.15.37:0/43181774 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f417c0a29d0 con 0x7f417c0a3d90 2024-04-26T00:40:57.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f417bfff640 1 -- 172.21.15.37:0/43181774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f417c0a3d90 msgr2=0x7f417c132e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:57.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f417bfff640 1 --2- 172.21.15.37:0/43181774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f417c0a3d90 0x7f417c132e60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f416400b550 tx=0x7f416400ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:57.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f4178ff9640 1 -- 172.21.15.37:0/43181774 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41640050f0 con 0x7f417c0a3d90 2024-04-26T00:40:57.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f4178ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:57.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f4178ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:40:57.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f4178ff9640 10 monclient: dump: 2024-04-26T00:40:57.435 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:57.435 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:57.435 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:57.435 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:57.435 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:57.435 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:57.435 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:57.435 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:57.435 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:57.436 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:57.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f4178ff9640 10 monclient: _finish_auth 0 2024-04-26T00:40:57.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f4178ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:40:57.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f4178ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:57.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f4178ff9640 1 -- 172.21.15.37:0/43181774 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4164015070 con 0x7f417c0a3d90 2024-04-26T00:40:57.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f418236c640 5 monclient: authenticate success, global_id 254171 2024-04-26T00:40:57.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f418236c640 10 monclient: _renew_subs 2024-04-26T00:40:57.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f418236c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:40:57.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f418236c640 1 -- 172.21.15.37:0/43181774 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f417c13cce0 con 0x7f417c0a3d90 2024-04-26T00:40:57.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f4178ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:40:57.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f4178ff9640 1 -- 172.21.15.37:0/43181774 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41640054d0 con 0x7f417c0a3d90 2024-04-26T00:40:57.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f4178ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:57.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f4178ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:40:57.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f4178ff9640 10 monclient: dump: 2024-04-26T00:40:57.436 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:57.437 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:57.437 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:57.437 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:57.437 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:57.437 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:57.437 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:57.437 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:57.437 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:57.437 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:57.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f417a7fc640 4 set_mon_vals no callback set 2024-04-26T00:40:57.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f418236c640 10 monclient: _renew_subs 2024-04-26T00:40:57.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f418236c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:40:57.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f418236c640 1 -- 172.21.15.37:0/43181774 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f417c0a0790 con 0x7f417c0a3d90 2024-04-26T00:40:57.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f418236c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:57.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f418236c640 1 --2- 172.21.15.37:0/43181774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f417c1cf6e0 0x7f417c1cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:57.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f418236c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:57.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f418236c640 1 -- 172.21.15.37:0/43181774 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f417c13b380 con 0x7f417c1cf6e0 2024-04-26T00:40:57.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f417b7fe640 1 --2- 172.21.15.37:0/43181774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f417c1cf6e0 0x7f417c1cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:57.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f417b7fe640 10 monclient: get_auth_request con 0x7f417c1cf6e0 auth_method 0 2024-04-26T00:40:57.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f417b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:57.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f417b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:57.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f417b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:57.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f4178ff9640 1 -- 172.21.15.37:0/43181774 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f41640056b0 con 0x7f417c0a3d90 2024-04-26T00:40:57.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f417b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:57.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f417b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:57.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f417b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:57.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f4178ff9640 1 --2- 172.21.15.37:0/43181774 >> v2:172.21.15.45:6812/2379918623 conn(0x7f415c035e60 0x7f415c038320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:57.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f4178ff9640 1 -- 172.21.15.37:0/43181774 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(824..824 src has 705..824) ==== 3995+0+0 (secure 0 0 0) 0x7f4164048a00 con 0x7f417c0a3d90 2024-04-26T00:40:57.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f417b7fe640 10 monclient(hunting): handle_auth_done global_id 254174 payload 824 2024-04-26T00:40:57.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f417b7fe640 1 -- 172.21.15.37:0/43181774 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f417c1d0000 con 0x7f417c1cf6e0 2024-04-26T00:40:57.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f4180909640 1 --2- 172.21.15.37:0/43181774 >> v2:172.21.15.45:6812/2379918623 conn(0x7f415c035e60 0x7f415c038320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:57.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f4180909640 10 monclient: get_auth_request con 0x7f415c035e60 auth_method 0 2024-04-26T00:40:57.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f417b7fe640 1 -- 172.21.15.37:0/43181774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f417c1cf6e0 msgr2=0x7f417c1cfac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:57.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.425+0000 7f417b7fe640 1 --2- 172.21.15.37:0/43181774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f417c1cf6e0 0x7f417c1cfac0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f417c13a5f0 tx=0x7f416c007700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:57.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.429+0000 7f4178ff9640 1 -- 172.21.15.37:0/43181774 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416c013780 con 0x7f417c1cf6e0 2024-04-26T00:40:57.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.429+0000 7f4178ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:57.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.429+0000 7f4178ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:40:57.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.429+0000 7f4178ff9640 10 monclient: dump: 2024-04-26T00:40:57.439 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:57.439 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:57.439 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:57.439 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:57.439 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:57.439 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:57.440 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:57.440 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:57.440 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:57.440 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:57.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.429+0000 7f4178ff9640 1 -- 172.21.15.37:0/43181774 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f416c019070 con 0x7f417c1cf6e0 2024-04-26T00:40:57.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.429+0000 7f4178ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:57.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.429+0000 7f4178ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:40:57.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.429+0000 7f4178ff9640 1 -- 172.21.15.37:0/43181774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f417c1cf6e0 msgr2=0x7f417c1cfac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:57.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.429+0000 7f4178ff9640 1 --2- 172.21.15.37:0/43181774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f417c1cf6e0 0x7f417c1cfac0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f417c13a5f0 tx=0x7f416c007700 comp rx=0 tx=0).stop 2024-04-26T00:40:57.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.429+0000 7f4178ff9640 1 -- 172.21.15.37:0/43181774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f417c1cf6e0 msgr2=0x7f417c1cfac0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:40:57.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.429+0000 7f4178ff9640 1 --2- 172.21.15.37:0/43181774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f417c1cf6e0 0x7f417c1cfac0 secure :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f417c13a5f0 tx=0x7f416c007700 comp rx=0 tx=0).stop 2024-04-26T00:40:57.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.429+0000 7f4180909640 1 --2- 172.21.15.37:0/43181774 >> v2:172.21.15.45:6812/2379918623 conn(0x7f415c035e60 0x7f415c038320 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f417c0a57e0 tx=0x7f4174008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:57.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.433+0000 7f418236c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:40:57.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.433+0000 7f418236c640 1 --2- 172.21.15.37:0/43181774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f417c134b30 0x7f417c134f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:57.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.433+0000 7f418236c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:57.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.433+0000 7f418236c640 1 -- 172.21.15.37:0/43181774 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f417c04f510 con 0x7f417c134b30 2024-04-26T00:40:57.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.433+0000 7f417b7fe640 1 --2- 172.21.15.37:0/43181774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f417c134b30 0x7f417c134f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:57.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.433+0000 7f417b7fe640 10 monclient: get_auth_request con 0x7f417c134b30 auth_method 0 2024-04-26T00:40:57.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.433+0000 7f417b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:57.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.433+0000 7f417b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:57.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.433+0000 7f417b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:57.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.433+0000 7f417b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:57.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.433+0000 7f417b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:57.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.433+0000 7f417b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:57.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.433+0000 7f417b7fe640 10 monclient(hunting): handle_auth_done global_id 254177 payload 824 2024-04-26T00:40:57.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.433+0000 7f417b7fe640 1 -- 172.21.15.37:0/43181774 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f417c13a9f0 con 0x7f417c134b30 2024-04-26T00:40:57.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.437+0000 7f417b7fe640 1 -- 172.21.15.37:0/43181774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f417c134b30 msgr2=0x7f417c134f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:57.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.437+0000 7f417b7fe640 1 --2- 172.21.15.37:0/43181774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f417c134b30 0x7f417c134f40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f416c018000 tx=0x7f416c004a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:57.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.437+0000 7f4178ff9640 1 -- 172.21.15.37:0/43181774 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416c01e230 con 0x7f417c134b30 2024-04-26T00:40:57.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.437+0000 7f4178ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:57.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.437+0000 7f4178ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:40:57.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.437+0000 7f4178ff9640 10 monclient: dump: 2024-04-26T00:40:57.443 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:57.444 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:57.444 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:57.444 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:57.444 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:57.444 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:57.444 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:57.444 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:57.444 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:57.444 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:57.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.437+0000 7f4178ff9640 1 -- 172.21.15.37:0/43181774 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4708 (secure 0 0 0) 0x7f416c01e230 con 0x7f417c134b30 2024-04-26T00:40:57.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.437+0000 7f4178ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:40:57.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.437+0000 7f4178ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:40:57.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.437+0000 7f4178ff9640 1 -- 172.21.15.37:0/43181774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f417c134b30 msgr2=0x7f417c134f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:57.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.437+0000 7f4178ff9640 1 --2- 172.21.15.37:0/43181774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f417c134b30 0x7f417c134f40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f416c018000 tx=0x7f416c004a50 comp rx=0 tx=0).stop 2024-04-26T00:40:57.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.437+0000 7f4178ff9640 1 -- 172.21.15.37:0/43181774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f417c134b30 msgr2=0x7f417c134f40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:40:57.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.437+0000 7f4178ff9640 1 --2- 172.21.15.37:0/43181774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f417c134b30 0x7f417c134f40 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f416c018000 tx=0x7f416c004a50 comp rx=0 tx=0).stop 2024-04-26T00:40:57.445 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:40:57.445 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:40:57.445 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:40:57.445 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:40:57.445 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 154, 2024-04-26T00:40:57.445 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 26207, 2024-04-26T00:40:57.445 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:40:57.445 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:40:57.445 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:40:57.445 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:57.446 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:40:57.446 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:40:57.446 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:40:57.446 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:40:57.446 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:40:57.446 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:40:57.446 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:40:57.446 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:40:57.446 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:40:57.446 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:40:57.446 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:40:57.446 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:40:57.446 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:40:57.446 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:40:57.446 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:40:57.447 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:57.447 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:40:57.447 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:40:57.447 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:40:57.447 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:40:57.447 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:40:57.447 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:40:57.447 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:40:57.447 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:40:57.447 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:40:57.447 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:40:57.447 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:40:57.447 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:40:57.447 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:40:57.447 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:57.448 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:57.448 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:40:57.448 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:40:57.448 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:40:57.448 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:40:57.448 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:40:57.448 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:40:57.448 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:40:57.448 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:40:57.448 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:40:57.448 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:40:57.448 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:40:57.448 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:40:57.448 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:40:57.448 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:40:57.449 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:40:57.449 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:40:57.449 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:40:57.449 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:40:57.449 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:40:57.449 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:40:57.449 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:40:57.449 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:40:57.449 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:40:57.449 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:40:57.449 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:40:57.449 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:40:57.449 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:40:57.449 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:40:57.449 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:57.450 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:40:57.450 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:57.450 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:40:57.450 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:57.450 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:40:57.450 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:40:57.450 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:40:57.450 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:40:57.450 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:57.450 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:40:57.450 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:40:57.450 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:57.450 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:57.450 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:57.451 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:40:57.451 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:40:57.451 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:57.451 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:57.451 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:57.451 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:57.451 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:40:57.451 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:40:57.451 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:40:57.451 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:40:57.451 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:40:57.451 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:57.451 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:57.451 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:40:57.451 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:40:57.451 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:40:57.452 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:40:57.452 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:57.452 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:40:57.452 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:40:57.452 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:57.452 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:57.452 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:57.452 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:40:57.452 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:40:57.452 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:57.452 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:57.452 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:57.452 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:57.452 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:40:57.452 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:40:57.453 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:40:57.453 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:40:57.453 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:40:57.453 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:57.453 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:57.453 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:40:57.453 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:40:57.453 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:40:57.453 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:40:57.453 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:57.453 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:40:57.453 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:40:57.453 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:57.453 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:57.453 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:57.454 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:40:57.454 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:40:57.454 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:57.454 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:57.454 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:57.454 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:57.454 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:40:57.454 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:40:57.454 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:40:57.454 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:40:57.454 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:40:57.454 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:57.454 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:57.454 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:57.454 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:40:57.455 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:40:57.455 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:57.455 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:40:57.455 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:40:57.455 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:40:57.455 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:57.455 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:57.455 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:40:57.455 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:57.455 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:40:57.455 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:40:57.455 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:40:57.455 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:57.455 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:57.455 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:40:57.455 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:57.456 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:40:57.456 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:40:57.456 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:40:57.456 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:57.456 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:57.456 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:57.456 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:40:57.456 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:40:57.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.437+0000 7f418236c640 1 -- 172.21.15.37:0/43181774 >> v2:172.21.15.45:6812/2379918623 conn(0x7f415c035e60 msgr2=0x7f415c038320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:57.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.437+0000 7f418236c640 1 --2- 172.21.15.37:0/43181774 >> v2:172.21.15.45:6812/2379918623 conn(0x7f415c035e60 0x7f415c038320 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f417c0a57e0 tx=0x7f4174008040 comp rx=0 tx=0).stop 2024-04-26T00:40:57.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.437+0000 7f418236c640 10 monclient: shutdown 2024-04-26T00:40:57.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.437+0000 7f418236c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:40:57.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.437+0000 7f418236c640 1 -- 172.21.15.37:0/43181774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f417c0a3d90 msgr2=0x7f417c132e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:57.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.437+0000 7f418236c640 1 --2- 172.21.15.37:0/43181774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f417c0a3d90 0x7f417c132e60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f416400b550 tx=0x7f416400ba10 comp rx=0 tx=0).stop 2024-04-26T00:40:57.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.437+0000 7f4180909640 1 -- 172.21.15.37:0/43181774 reap_dead start 2024-04-26T00:40:57.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.437+0000 7f418236c640 1 -- 172.21.15.37:0/43181774 shutdown_connections 2024-04-26T00:40:57.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.437+0000 7f418236c640 1 --2- 172.21.15.37:0/43181774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f417c0a3d90 0x7f417c132e60 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:57.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.437+0000 7f418236c640 1 -- 172.21.15.37:0/43181774 >> 172.21.15.37:0/43181774 conn(0x7f417c0aa0a0 msgr2=0x7f417c0a22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:40:57.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.437+0000 7f418236c640 1 -- 172.21.15.37:0/43181774 shutdown_connections 2024-04-26T00:40:57.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.437+0000 7f418236c640 1 -- 172.21.15.37:0/43181774 wait complete. 2024-04-26T00:40:57.458 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:40:57.458 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-04-26T00:40:57.459 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:40:57.538 INFO:tasks.rados.rados.0.smithi045.stdout:3656: done (1 left) 2024-04-26T00:40:57.538 INFO:tasks.rados.rados.0.smithi045.stdout:3657: done (0 left) 2024-04-26T00:40:57.539 INFO:tasks.rados.rados.0.smithi045.stdout:3658: copy_from oid 36 from oid 45 current snap is 404 2024-04-26T00:40:57.539 INFO:tasks.rados.rados.0.smithi045.stdout:3659: delete oid 34 current snap is 404 2024-04-26T00:40:57.539 INFO:tasks.rados.rados.0.smithi045.stdout:3658: finishing copy_from to smithi04526579-36 2024-04-26T00:40:57.539 INFO:tasks.rados.rados.0.smithi045.stdout:3658: got expected ENOENT (src dne) 2024-04-26T00:40:57.540 INFO:tasks.rados.rados.0.smithi045.stdout:3658: finishing copy_from racing read to smithi04526579-36 2024-04-26T00:40:57.544 INFO:tasks.rados.rados.0.smithi045.stdout:3658: done (1 left) 2024-04-26T00:40:57.544 INFO:tasks.rados.rados.0.smithi045.stdout:3659: done (0 left) 2024-04-26T00:40:57.545 INFO:tasks.rados.rados.0.smithi045.stdout:3660: write (excl) oid 14 current snap is 404 2024-04-26T00:40:57.545 INFO:tasks.rados.rados.0.smithi045.stdout:3660: seq_num 1306 ranges {749942=571557,1840895=796442,3365252=584690} 2024-04-26T00:40:57.552 INFO:tasks.rados.rados.0.smithi045.stdout:3660: writing smithi04526579-14 from 749942 to 1321499 tid 1 2024-04-26T00:40:57.560 INFO:tasks.rados.rados.0.smithi045.stdout:3660: writing smithi04526579-14 from 1840895 to 2637337 tid 2 2024-04-26T00:40:57.567 INFO:tasks.rados.rados.0.smithi045.stdout:3660: writing smithi04526579-14 from 3365252 to 3949942 tid 3 2024-04-26T00:40:57.567 INFO:tasks.rados.rados.0.smithi045.stdout:3661: write (excl) oid 39 current snap is 404 2024-04-26T00:40:57.567 INFO:tasks.rados.rados.0.smithi045.stdout:3661: seq_num 1307 ranges {766749=646806,1899377=656453,3109028=609462} 2024-04-26T00:40:57.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.573+0000 7f8653a93640 10 monclient: get_monmap_and_config 2024-04-26T00:40:57.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.573+0000 7f8653a93640 10 monclient: build_initial_monmap 2024-04-26T00:40:57.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.573+0000 7f8653a93640 10 monclient: monmap: 2024-04-26T00:40:57.575 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:40:57.575 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:57.575 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:40:57.577467+0000 2024-04-26T00:40:57.575 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:40:57.577467+0000 2024-04-26T00:40:57.575 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:40:57.575 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:57.575 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:40:57.575 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:40:57.575 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:40:57.575 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:40:57.576 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:40:57.576 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:57.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.573+0000 7f8653a93640 1 Processor -- start 2024-04-26T00:40:57.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.573+0000 7f8653a93640 1 -- start start 2024-04-26T00:40:57.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.573+0000 7f8653a93640 10 monclient: init 2024-04-26T00:40:57.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.573+0000 7f8653a93640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:57.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.573+0000 7f8653a93640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-04-26T00:40:57.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.573+0000 7f8653a93640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f864c0a3d90 0x7f864c09bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:57.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.573+0000 7f8653a93640 10 monclient(hunting): picked mon.noname-b con 0x7f864c0a3d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:40:57.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.573+0000 7f8653a93640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f864c0a4170 addr v1:172.21.15.37:6789/0 2024-04-26T00:40:57.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.573+0000 7f8653a93640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f864c13c3e0 0x7f864c13e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:57.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.573+0000 7f8653a93640 10 monclient(hunting): picked mon.noname-c con 0x7f864c13c3e0 addr v2:172.21.15.45:3300/0 2024-04-26T00:40:57.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.573+0000 7f8653a93640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f864c0a5150 con 0x7f864c0a4170 2024-04-26T00:40:57.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.573+0000 7f8653a93640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:57.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.573+0000 7f8653a93640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f864c09b280 con 0x7f864c13c3e0 2024-04-26T00:40:57.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.573+0000 7f8653a93640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:57.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.573+0000 7f8653a93640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f864c09b560 con 0x7f864c0a3d90 2024-04-26T00:40:57.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.573+0000 7f8653a93640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:57.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.573+0000 7f8653a93640 10 monclient(hunting): authenticate will time out at 2238.883832s 2024-04-26T00:40:57.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.573+0000 7f865102e640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f864c0a4170 legacy=0x7f864c0a45b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:40:57.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.577+0000 7f865182f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f864c0a3d90 0x7f864c09bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:57.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.577+0000 7f865182f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f864c0a3d90 0x7f864c09bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:58634/0 (socket says 172.21.15.37:58634) 2024-04-26T00:40:57.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.577+0000 7f865182f640 1 -- 172.21.15.37:0/1209842499 learned_addr learned my addr 172.21.15.37:0/1209842499 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:40:57.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.577+0000 7f865182f640 10 monclient(hunting): get_auth_request con 0x7f864c0a3d90 auth_method 0 2024-04-26T00:40:57.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.577+0000 7f865182f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:57.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.577+0000 7f865182f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:57.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.577+0000 7f865182f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:57.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.577+0000 7f865182f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:57.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.577+0000 7f865182f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:57.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.577+0000 7f8652030640 1 --2- 172.21.15.37:0/1209842499 >> v2:172.21.15.45:3300/0 conn(0x7f864c13c3e0 0x7f864c13e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:57.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.577+0000 7f865182f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:57.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.577+0000 7f8652030640 10 monclient(hunting): get_auth_request con 0x7f864c13c3e0 auth_method 0 2024-04-26T00:40:57.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.577+0000 7f8652030640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:57.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.577+0000 7f8652030640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:57.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.577+0000 7f8652030640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:57.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.577+0000 7f8652030640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:57.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.577+0000 7f8652030640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:57.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.577+0000 7f8652030640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:57.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.577+0000 7f865182f640 10 monclient(hunting): handle_auth_done global_id 254180 payload 274 2024-04-26T00:40:57.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.577+0000 7f865182f640 1 -- 172.21.15.37:0/1209842499 >> v2:172.21.15.45:3300/0 conn(0x7f864c13c3e0 msgr2=0x7f864c13e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:40:57.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.577+0000 7f865182f640 1 --2- 172.21.15.37:0/1209842499 >> v2:172.21.15.45:3300/0 conn(0x7f864c13c3e0 0x7f864c13e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:57.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.577+0000 7f865182f640 1 -- 172.21.15.37:0/1209842499 >> v1:172.21.15.37:6789/0 conn(0x7f864c0a4170 legacy=0x7f864c0a45b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:40:57.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.577+0000 7f865182f640 10 monclient: _finish_hunting 0 2024-04-26T00:40:57.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.577+0000 7f865182f640 1 monclient: found mon.noname-b 2024-04-26T00:40:57.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.577+0000 7f865182f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:57.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.577+0000 7f865182f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:40:57.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.577+0000 7f865182f640 1 -- 172.21.15.37:0/1209842499 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f864c06dc60 con 0x7f864c0a3d90 2024-04-26T00:40:57.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f865182f640 1 -- 172.21.15.37:0/1209842499 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f864c0a3d90 msgr2=0x7f864c09bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:57.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f865182f640 1 --2- 172.21.15.37:0/1209842499 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f864c0a3d90 0x7f864c09bf40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8648009870 tx=0x7f864801c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d636c064f04ab86a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:57.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f865082d640 1 -- 172.21.15.37:0/1209842499 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86480b0070 con 0x7f864c0a3d90 2024-04-26T00:40:57.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f865082d640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:57.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f865082d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:40:57.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f865082d640 10 monclient: dump: 2024-04-26T00:40:57.583 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:57.583 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:57.583 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:57.583 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:57.583 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:57.583 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:57.583 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:57.583 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:57.583 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:57.583 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:57.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f865082d640 10 monclient: _finish_auth 0 2024-04-26T00:40:57.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f865082d640 10 monclient: _check_auth_tickets 2024-04-26T00:40:57.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f865082d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:57.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f865082d640 1 -- 172.21.15.37:0/1209842499 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86480ab600 con 0x7f864c0a3d90 2024-04-26T00:40:57.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f865082d640 10 monclient: handle_config config(0 keys) 2024-04-26T00:40:57.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f865082d640 1 -- 172.21.15.37:0/1209842499 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86480ab9d0 con 0x7f864c0a3d90 2024-04-26T00:40:57.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8653a93640 5 monclient: authenticate success, global_id 254180 2024-04-26T00:40:57.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8653a93640 10 monclient: get_monmap_and_config success 2024-04-26T00:40:57.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8653a93640 4 set_mon_vals no callback set 2024-04-26T00:40:57.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f865082d640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:57.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f865082d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:40:57.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f865082d640 10 monclient: dump: 2024-04-26T00:40:57.584 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:57.584 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:57.584 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:57.584 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:57.584 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:57.584 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:57.584 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:57.585 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:57.585 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:57.585 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:57.585 INFO:tasks.rados.rados.0.smithi045.stdout:3661: writing smithi04526579-39 from 766749 to 1413555 tid 1 2024-04-26T00:40:57.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8653a93640 10 monclient: shutdown 2024-04-26T00:40:57.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8653a93640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:40:57.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8653a93640 1 -- 172.21.15.37:0/1209842499 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f864c0a3d90 msgr2=0x7f864c09bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:57.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8653a93640 1 --2- 172.21.15.37:0/1209842499 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f864c0a3d90 0x7f864c09bf40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8648009870 tx=0x7f864801c3e0 comp rx=0 tx=0).stop 2024-04-26T00:40:57.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8653a93640 1 -- 172.21.15.37:0/1209842499 shutdown_connections 2024-04-26T00:40:57.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8653a93640 1 --2- 172.21.15.37:0/1209842499 >> v2:172.21.15.45:3300/0 conn(0x7f864c13c3e0 0x7f864c13e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:57.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8653a93640 1 --2- 172.21.15.37:0/1209842499 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f864c0a3d90 0x7f864c09bf40 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:57.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8653a93640 1 -- 172.21.15.37:0/1209842499 >> 172.21.15.37:0/1209842499 conn(0x7f864c0aa0a0 msgr2=0x7f864c0a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:40:57.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8653a93640 1 -- 172.21.15.37:0/1209842499 shutdown_connections 2024-04-26T00:40:57.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8653a93640 1 -- 172.21.15.37:0/1209842499 wait complete. 2024-04-26T00:40:57.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8653a93640 10 monclient: build_initial_monmap 2024-04-26T00:40:57.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8653a93640 10 monclient: monmap: 2024-04-26T00:40:57.590 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:40:57.590 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:40:57.590 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:40:57.590 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:40:57.590 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:40:57.590 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:40:57.590 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:40:57.590 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:40:57.590 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:40:57.590 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:57.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8653a93640 1 Processor -- start 2024-04-26T00:40:57.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8653a93640 1 -- start start 2024-04-26T00:40:57.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8653a93640 10 monclient: init 2024-04-26T00:40:57.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8653a93640 10 monclient: _reopen_session rank -1 2024-04-26T00:40:57.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8653a93640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:40:57.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8653a93640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f864c0a3d90 0x7f864c1cec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:57.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8653a93640 10 monclient(hunting): picked mon.noname-b con 0x7f864c0a3d90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:40:57.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8653a93640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f864c0a4170 0x7f864c13aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:57.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8653a93640 10 monclient(hunting): picked mon.noname-a con 0x7f864c0a4170 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:40:57.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8653a93640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f864c13c3e0 0x7f864c121990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:57.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8653a93640 10 monclient(hunting): picked mon.noname-c con 0x7f864c13c3e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:40:57.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8653a93640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:57.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8653a93640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f864c141740 con 0x7f864c0a4170 2024-04-26T00:40:57.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8653a93640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:57.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8653a93640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f864c09c690 con 0x7f864c0a3d90 2024-04-26T00:40:57.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8653a93640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:57.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8653a93640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f864c1418c0 con 0x7f864c13c3e0 2024-04-26T00:40:57.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8653a93640 10 monclient(hunting): _renew_subs 2024-04-26T00:40:57.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8653a93640 10 monclient(hunting): authenticate will time out at 2238.891655s 2024-04-26T00:40:57.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f865102e640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f864c0a4170 msgr2=0x7f864c13aec0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:40:57.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f865102e640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f864c0a4170 0x7f864c13aec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:40:57.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8652030640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f864c13c3e0 0x7f864c121990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:57.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8652030640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f864c13c3e0 0x7f864c121990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:58648/0 (socket says 172.21.15.37:58648) 2024-04-26T00:40:57.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8652030640 1 -- 172.21.15.37:0/3138958096 learned_addr learned my addr 172.21.15.37:0/3138958096 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:40:57.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8652030640 10 monclient(hunting): get_auth_request con 0x7f864c13c3e0 auth_method 0 2024-04-26T00:40:57.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8652030640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:57.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8652030640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:57.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8652030640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:57.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8652030640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:57.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8652030640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:57.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.581+0000 7f8652030640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:57.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.585+0000 7f865182f640 1 --2- 172.21.15.37:0/3138958096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f864c0a3d90 0x7f864c1cec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:57.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.585+0000 7f865182f640 10 monclient(hunting): get_auth_request con 0x7f864c0a3d90 auth_method 0 2024-04-26T00:40:57.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.585+0000 7f865182f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:57.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.585+0000 7f865182f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:57.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.585+0000 7f865182f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:57.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.585+0000 7f8652030640 10 monclient(hunting): handle_auth_done global_id 254183 payload 824 2024-04-26T00:40:57.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.585+0000 7f8652030640 1 -- 172.21.15.37:0/3138958096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f864c0a3d90 msgr2=0x7f864c1cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:57.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.585+0000 7f8652030640 1 --2- 172.21.15.37:0/3138958096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f864c0a3d90 0x7f864c1cec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:57.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.585+0000 7f8652030640 1 -- 172.21.15.37:0/3138958096 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f864c0a4170 msgr2=0x7f864c13aec0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:40:57.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.585+0000 7f8652030640 1 --2- 172.21.15.37:0/3138958096 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f864c0a4170 0x7f864c13aec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:40:57.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.585+0000 7f8652030640 10 monclient: _finish_hunting 0 2024-04-26T00:40:57.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.585+0000 7f8652030640 1 monclient: found mon.noname-c 2024-04-26T00:40:57.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.585+0000 7f8652030640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:40:57.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.585+0000 7f8652030640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:40:57.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.585+0000 7f8652030640 1 -- 172.21.15.37:0/3138958096 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f864c13cc70 con 0x7f864c13c3e0 2024-04-26T00:40:57.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.585+0000 7f8652030640 1 -- 172.21.15.37:0/3138958096 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f864c13c3e0 msgr2=0x7f864c121990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:57.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.585+0000 7f8652030640 1 --2- 172.21.15.37:0/3138958096 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f864c13c3e0 0x7f864c121990 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f864400efc0 tx=0x7f864400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:57.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.585+0000 7f86427fc640 1 -- 172.21.15.37:0/3138958096 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8644018020 con 0x7f864c13c3e0 2024-04-26T00:40:57.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.585+0000 7f86427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:57.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.585+0000 7f86427fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:40:57.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.585+0000 7f86427fc640 10 monclient: dump: 2024-04-26T00:40:57.594 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:57.594 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:57.594 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:57.594 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:57.594 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:57.594 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:57.595 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:57.595 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:57.595 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:57.595 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:57.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.585+0000 7f86427fc640 10 monclient: _finish_auth 0 2024-04-26T00:40:57.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.585+0000 7f86427fc640 10 monclient: _check_auth_tickets 2024-04-26T00:40:57.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.585+0000 7f86427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:40:57.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.585+0000 7f86427fc640 1 -- 172.21.15.37:0/3138958096 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8644014680 con 0x7f864c13c3e0 2024-04-26T00:40:57.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.585+0000 7f86427fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:40:57.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.585+0000 7f86427fc640 1 -- 172.21.15.37:0/3138958096 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8644014a70 con 0x7f864c13c3e0 2024-04-26T00:40:57.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.585+0000 7f86427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:57.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.585+0000 7f86427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:40:57.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.585+0000 7f86427fc640 10 monclient: dump: 2024-04-26T00:40:57.595 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:57.595 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:57.596 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:57.596 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:57.596 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:57.596 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:57.596 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:57.596 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:57.596 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:57.596 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:57.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.585+0000 7f8643fff640 4 set_mon_vals no callback set 2024-04-26T00:40:57.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.585+0000 7f8653a93640 5 monclient: authenticate success, global_id 254183 2024-04-26T00:40:57.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.585+0000 7f8653a93640 10 monclient: _renew_subs 2024-04-26T00:40:57.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.585+0000 7f8653a93640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:40:57.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.585+0000 7f8653a93640 1 -- 172.21.15.37:0/3138958096 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f864c09ada0 con 0x7f864c13c3e0 2024-04-26T00:40:57.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.585+0000 7f8653a93640 10 monclient: _renew_subs 2024-04-26T00:40:57.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.585+0000 7f8653a93640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:40:57.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.585+0000 7f8653a93640 1 -- 172.21.15.37:0/3138958096 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f864c125570 con 0x7f864c13c3e0 2024-04-26T00:40:57.597 INFO:tasks.rados.rados.0.smithi045.stdout:3661: writing smithi04526579-39 from 1899377 to 2555830 tid 2 2024-04-26T00:40:57.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.589+0000 7f8653a93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:57.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.589+0000 7f8653a93640 1 --2- 172.21.15.37:0/3138958096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f864c13b400 0x7f864c121ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:57.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.589+0000 7f8653a93640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:57.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.589+0000 7f8653a93640 1 -- 172.21.15.37:0/3138958096 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f864c1251f0 con 0x7f864c13b400 2024-04-26T00:40:57.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.589+0000 7f86427fc640 1 -- 172.21.15.37:0/3138958096 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f864401b020 con 0x7f864c13c3e0 2024-04-26T00:40:57.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.589+0000 7f86427fc640 1 --2- 172.21.15.37:0/3138958096 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8630035eb0 0x7f8630038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:57.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.589+0000 7f86427fc640 1 -- 172.21.15.37:0/3138958096 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(824..824 src has 705..824) ==== 3995+0+0 (secure 0 0 0) 0x7f8644049690 con 0x7f864c13c3e0 2024-04-26T00:40:57.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.589+0000 7f865182f640 1 --2- 172.21.15.37:0/3138958096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f864c13b400 0x7f864c121ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:57.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.589+0000 7f865182f640 10 monclient: get_auth_request con 0x7f864c13b400 auth_method 0 2024-04-26T00:40:57.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.589+0000 7f865182f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:57.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.589+0000 7f865182f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:57.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.589+0000 7f865182f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:57.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.589+0000 7f865182f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:57.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.589+0000 7f865182f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:57.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.589+0000 7f865182f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:57.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.589+0000 7f865102e640 1 --2- 172.21.15.37:0/3138958096 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8630035eb0 0x7f8630038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:57.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.589+0000 7f865102e640 10 monclient: get_auth_request con 0x7f8630035eb0 auth_method 0 2024-04-26T00:40:57.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.589+0000 7f865182f640 10 monclient(hunting): handle_auth_done global_id 224419 payload 824 2024-04-26T00:40:57.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.589+0000 7f865182f640 1 -- 172.21.15.37:0/3138958096 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f864c04f510 con 0x7f864c13b400 2024-04-26T00:40:57.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.589+0000 7f865182f640 1 -- 172.21.15.37:0/3138958096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f864c13b400 msgr2=0x7f864c121ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:57.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.589+0000 7f865182f640 1 --2- 172.21.15.37:0/3138958096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f864c13b400 0x7f864c121ed0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f864c0a6d60 tx=0x7f864801ac20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:57.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.593+0000 7f865102e640 1 --2- 172.21.15.37:0/3138958096 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8630035eb0 0x7f8630038370 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f863c006fd0 tx=0x7f863c006d40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:57.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.593+0000 7f86427fc640 1 -- 172.21.15.37:0/3138958096 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86480bb020 con 0x7f864c13b400 2024-04-26T00:40:57.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.593+0000 7f86427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:57.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.593+0000 7f86427fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:40:57.601 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.593+0000 7f86427fc640 10 monclient: dump: 2024-04-26T00:40:57.601 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:57.601 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:57.601 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:57.601 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:57.601 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:57.601 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:57.602 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:57.602 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:57.602 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:57.602 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:57.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.593+0000 7f86427fc640 1 -- 172.21.15.37:0/3138958096 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f86480b1040 con 0x7f864c13b400 2024-04-26T00:40:57.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.593+0000 7f86427fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:40:57.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.593+0000 7f86427fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:40:57.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.593+0000 7f86427fc640 1 -- 172.21.15.37:0/3138958096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f864c13b400 msgr2=0x7f864c121ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:57.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.593+0000 7f86427fc640 1 --2- 172.21.15.37:0/3138958096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f864c13b400 0x7f864c121ed0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f864c0a6d60 tx=0x7f864801ac20 comp rx=0 tx=0).stop 2024-04-26T00:40:57.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.593+0000 7f86427fc640 1 -- 172.21.15.37:0/3138958096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f864c13b400 msgr2=0x7f864c121ed0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:40:57.602 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.593+0000 7f86427fc640 1 --2- 172.21.15.37:0/3138958096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f864c13b400 0x7f864c121ed0 secure :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f864c0a6d60 tx=0x7f864801ac20 comp rx=0 tx=0).stop 2024-04-26T00:40:57.603 INFO:tasks.rados.rados.0.smithi045.stdout:3661: writing smithi04526579-39 from 3109028 to 3718490 tid 3 2024-04-26T00:40:57.604 INFO:tasks.rados.rados.0.smithi045.stdout:3662: rollback oid 20 current snap is 404 2024-04-26T00:40:57.604 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 20 to 393 2024-04-26T00:40:57.604 INFO:tasks.rados.rados.0.smithi045.stdout:3660: finishing write tid 1 to smithi04526579-14 2024-04-26T00:40:57.604 INFO:tasks.rados.rados.0.smithi045.stdout:3663: write oid 33 current snap is 404 2024-04-26T00:40:57.604 INFO:tasks.rados.rados.0.smithi045.stdout:3663: seq_num 1308 ranges {783556=438408,1674212=400111,2736451=420734,3882270=101286} 2024-04-26T00:40:57.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.597+0000 7f8653a93640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:40:57.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.597+0000 7f8653a93640 1 --2- 172.21.15.37:0/3138958096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f864c122410 0x7f864c0a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:40:57.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.597+0000 7f8653a93640 10 monclient(hunting): start opening mon connection 2024-04-26T00:40:57.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.597+0000 7f8653a93640 1 -- 172.21.15.37:0/3138958096 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f864c122820 con 0x7f864c122410 2024-04-26T00:40:57.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.601+0000 7f865182f640 1 --2- 172.21.15.37:0/3138958096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f864c122410 0x7f864c0a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:40:57.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.601+0000 7f865182f640 10 monclient: get_auth_request con 0x7f864c122410 auth_method 0 2024-04-26T00:40:57.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.601+0000 7f865182f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:40:57.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.601+0000 7f865182f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:40:57.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.601+0000 7f865182f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:40:57.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.601+0000 7f865182f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:40:57.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.601+0000 7f865182f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:40:57.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.601+0000 7f865182f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:40:57.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.601+0000 7f865182f640 10 monclient(hunting): handle_auth_done global_id 224422 payload 824 2024-04-26T00:40:57.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.601+0000 7f865182f640 1 -- 172.21.15.37:0/3138958096 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f864c0008e0 con 0x7f864c122410 2024-04-26T00:40:57.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.601+0000 7f865182f640 1 -- 172.21.15.37:0/3138958096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f864c122410 msgr2=0x7f864c0a17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:40:57.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.601+0000 7f865182f640 1 --2- 172.21.15.37:0/3138958096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f864c122410 0x7f864c0a17a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f864801cc80 tx=0x7f8648019180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:40:57.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.601+0000 7f86427fc640 1 -- 172.21.15.37:0/3138958096 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86480bb070 con 0x7f864c122410 2024-04-26T00:40:57.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.601+0000 7f86427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:40:57.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.601+0000 7f86427fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:40:57.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.601+0000 7f86427fc640 10 monclient: dump: 2024-04-26T00:40:57.606 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:40:57.606 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:40:57.606 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:57.606 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:40:57.606 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:40:57.607 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:40:57.607 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:40:57.607 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:40:57.607 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:40:57.607 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:40:57.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.605+0000 7f86427fc640 1 -- 172.21.15.37:0/3138958096 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4877 (secure 0 0 0) 0x7f86480b7ca0 con 0x7f864c122410 2024-04-26T00:40:57.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.605+0000 7f86427fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:40:57.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.605+0000 7f86427fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:40:57.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.605+0000 7f86427fc640 1 -- 172.21.15.37:0/3138958096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f864c122410 msgr2=0x7f864c0a17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:57.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.605+0000 7f86427fc640 1 --2- 172.21.15.37:0/3138958096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f864c122410 0x7f864c0a17a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f864801cc80 tx=0x7f8648019180 comp rx=0 tx=0).stop 2024-04-26T00:40:57.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.605+0000 7f86427fc640 1 -- 172.21.15.37:0/3138958096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f864c122410 msgr2=0x7f864c0a17a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:40:57.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.605+0000 7f86427fc640 1 --2- 172.21.15.37:0/3138958096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f864c122410 0x7f864c0a17a0 secure :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f864801cc80 tx=0x7f8648019180 comp rx=0 tx=0).stop 2024-04-26T00:40:57.608 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:40:57.608 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:40:57.608 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:40:57.608 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:40:57.608 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 154, 2024-04-26T00:40:57.608 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 171264, 2024-04-26T00:40:57.609 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:40:57.609 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:40:57.609 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:40:57.609 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:57.609 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 1, 2024-04-26T00:40:57.609 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:40:57.609 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:40:57.609 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:40:57.609 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:40:57.609 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:40:57.609 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:40:57.609 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:40:57.609 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:40:57.609 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:40:57.609 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:40:57.610 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:40:57.610 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:40:57.610 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:40:57.610 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:40:57.610 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:57.610 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:40:57.610 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:40:57.610 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:40:57.610 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:40:57.610 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:40:57.610 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:40:57.610 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:40:57.610 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:40:57.611 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:40:57.611 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:40:57.611 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:40:57.611 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:40:57.611 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:40:57.611 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:57.611 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:57.611 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:40:57.611 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:40:57.611 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:40:57.611 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:40:57.611 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:40:57.611 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:40:57.611 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:40:57.611 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:40:57.612 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:40:57.612 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:40:57.612 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:40:57.612 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:40:57.612 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:40:57.612 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:40:57.612 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:40:57.612 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:40:57.612 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:40:57.612 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:40:57.612 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:40:57.612 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:40:57.612 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:40:57.612 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:40:57.612 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:40:57.613 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:40:57.613 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:40:57.613 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:40:57.613 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:40:57.613 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:40:57.613 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:57.613 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:40:57.613 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:57.613 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:40:57.613 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:57.613 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:40:57.613 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:40:57.613 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:40:57.613 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:40:57.613 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:57.614 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:40:57.614 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:40:57.614 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:57.614 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:57.614 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:57.614 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:40:57.614 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:40:57.614 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:57.614 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:57.614 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:57.614 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:57.614 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:40:57.614 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:40:57.614 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:40:57.614 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:40:57.615 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:40:57.615 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:57.615 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:57.615 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:40:57.615 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:40:57.615 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:40:57.615 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:40:57.615 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:57.615 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:40:57.615 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:40:57.615 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:57.615 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:57.615 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:57.615 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:40:57.615 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:40:57.616 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:57.616 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:57.616 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:57.616 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:57.616 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:40:57.616 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:40:57.616 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:40:57.616 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:40:57.616 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:40:57.616 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:57.616 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:57.616 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:40:57.616 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:40:57.616 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:40:57.617 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:40:57.617 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:57.617 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:40:57.617 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:40:57.617 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:57.617 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:57.617 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:57.617 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:40:57.617 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:40:57.617 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:40:57.617 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:57.617 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:57.617 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:57.617 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:40:57.617 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:40:57.618 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:40:57.618 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:40:57.618 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:40:57.618 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:57.618 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:57.618 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:57.618 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:40:57.618 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:40:57.618 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:57.618 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:40:57.618 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:40:57.618 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:40:57.618 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:57.618 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:57.618 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:40:57.619 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:57.619 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:40:57.619 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:40:57.619 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:40:57.619 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:57.619 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:57.619 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:40:57.619 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:57.619 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:40:57.619 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:40:57.619 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:40:57.619 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:57.619 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:40:57.619 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:40:57.619 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:40:57.620 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:40:57.620 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:40:57.620 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:40:57.620 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:40:57.620 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:40:57.620 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:40:57.620 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:40:57.620 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:40:57.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.605+0000 7f8653a93640 1 -- 172.21.15.37:0/3138958096 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8630035eb0 msgr2=0x7f8630038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:57.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.605+0000 7f8653a93640 1 --2- 172.21.15.37:0/3138958096 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8630035eb0 0x7f8630038370 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f863c006fd0 tx=0x7f863c006d40 comp rx=0 tx=0).stop 2024-04-26T00:40:57.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.605+0000 7f8653a93640 10 monclient: shutdown 2024-04-26T00:40:57.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.605+0000 7f8653a93640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:40:57.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.605+0000 7f8653a93640 1 -- 172.21.15.37:0/3138958096 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f864c13c3e0 msgr2=0x7f864c121990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:40:57.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.605+0000 7f8653a93640 1 --2- 172.21.15.37:0/3138958096 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f864c13c3e0 0x7f864c121990 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f864400efc0 tx=0x7f864400c490 comp rx=0 tx=0).stop 2024-04-26T00:40:57.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.605+0000 7f8652030640 1 -- 172.21.15.37:0/3138958096 reap_dead start 2024-04-26T00:40:57.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.605+0000 7f8652030640 1 -- 172.21.15.37:0/3138958096 reap_dead start 2024-04-26T00:40:57.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.605+0000 7f8653a93640 1 -- 172.21.15.37:0/3138958096 shutdown_connections 2024-04-26T00:40:57.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.605+0000 7f8653a93640 1 -- 172.21.15.37:0/3138958096 >> 172.21.15.37:0/3138958096 conn(0x7f864c0aa0a0 msgr2=0x7f864c0a8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:40:57.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.605+0000 7f8653a93640 1 -- 172.21.15.37:0/3138958096 shutdown_connections 2024-04-26T00:40:57.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:40:57.605+0000 7f8653a93640 1 -- 172.21.15.37:0/3138958096 wait complete. 2024-04-26T00:40:57.622 INFO:tasks.rados.rados.0.smithi045.stdout:3663: writing smithi04526579-33 from 783556 to 1221964 tid 1 2024-04-26T00:40:57.622 INFO:tasks.rados.rados.0.smithi045.stdout:3663: writing smithi04526579-33 from 1674212 to 2074323 tid 2 2024-04-26T00:40:57.626 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:40:57.626 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-04-26T00:40:57.626 INFO:tasks.rados.rados.0.smithi045.stdout:3663: writing smithi04526579-33 from 2736451 to 3157185 tid 3 2024-04-26T00:40:57.632 INFO:tasks.rados.rados.0.smithi045.stdout:3663: writing smithi04526579-33 from 3882270 to 3983556 tid 4 2024-04-26T00:40:57.632 INFO:tasks.rados.rados.0.smithi045.stdout:3664: snap_create 2024-04-26T00:40:57.632 INFO:tasks.rados.rados.0.smithi045.stdout:3660: finishing write tid 2 to smithi04526579-14 2024-04-26T00:40:57.632 INFO:tasks.rados.rados.0.smithi045.stdout:3660: finishing write tid 3 to smithi04526579-14 2024-04-26T00:40:57.632 INFO:tasks.rados.rados.0.smithi045.stdout:3660: finishing write tid 4 to smithi04526579-14 2024-04-26T00:40:57.633 INFO:tasks.rados.rados.0.smithi045.stdout:3660: finishing write tid 5 to smithi04526579-14 2024-04-26T00:40:57.633 INFO:tasks.rados.rados.0.smithi045.stdout:3660: oid 14 updating version 0 to 1216 2024-04-26T00:40:57.633 INFO:tasks.rados.rados.0.smithi045.stdout:3660: oid 14 updating version 1216 to 1218 2024-04-26T00:40:57.633 INFO:tasks.rados.rados.0.smithi045.stdout:3660: oid 14 version 1218 is already newer than 1215 2024-04-26T00:40:57.633 INFO:tasks.rados.rados.0.smithi045.stdout:3660: oid 14 version 1218 is already newer than 1217 2024-04-26T00:40:57.633 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 14 v 1218 (ObjNum 1306 snap 404 seq_num 1306) dirty exists 2024-04-26T00:40:57.633 INFO:tasks.rados.rados.0.smithi045.stdout:3660: left oid 14 (ObjNum 1306 snap 404 seq_num 1306) 2024-04-26T00:40:57.633 INFO:tasks.rados.rados.0.smithi045.stdout:3661: finishing write tid 1 to smithi04526579-39 2024-04-26T00:40:57.633 INFO:tasks.rados.rados.0.smithi045.stdout:3661: finishing write tid 2 to smithi04526579-39 2024-04-26T00:40:57.633 INFO:tasks.rados.rados.0.smithi045.stdout:3661: finishing write tid 3 to smithi04526579-39 2024-04-26T00:40:57.633 INFO:tasks.rados.rados.0.smithi045.stdout:3661: finishing write tid 4 to smithi04526579-39 2024-04-26T00:40:57.633 INFO:tasks.rados.rados.0.smithi045.stdout:3661: finishing write tid 5 to smithi04526579-39 2024-04-26T00:40:57.633 INFO:tasks.rados.rados.0.smithi045.stdout:3661: oid 39 updating version 0 to 2407 2024-04-26T00:40:57.633 INFO:tasks.rados.rados.0.smithi045.stdout:3661: oid 39 version 2407 is already newer than 2406 2024-04-26T00:40:57.634 INFO:tasks.rados.rados.0.smithi045.stdout:3661: oid 39 updating version 2407 to 2409 2024-04-26T00:40:57.634 INFO:tasks.rados.rados.0.smithi045.stdout:3661: oid 39 version 2409 is already newer than 2408 2024-04-26T00:40:57.634 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 2409 (ObjNum 1307 snap 404 seq_num 1307) dirty exists 2024-04-26T00:40:57.634 INFO:tasks.rados.rados.0.smithi045.stdout:3661: left oid 39 (ObjNum 1307 snap 404 seq_num 1307) 2024-04-26T00:40:57.634 INFO:tasks.rados.rados.0.smithi045.stdout:3662: finishing rollback tid 1 to smithi04526579-20 2024-04-26T00:40:57.634 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 2399 (ObjNum 129 snap 0 seq_num 2242410088) dirty dne 2024-04-26T00:40:57.634 INFO:tasks.rados.rados.0.smithi045.stdout:3663: finishing write tid 1 to smithi04526579-33 2024-04-26T00:40:57.634 INFO:tasks.rados.rados.0.smithi045.stdout:3663: finishing write tid 2 to smithi04526579-33 2024-04-26T00:40:57.634 INFO:tasks.rados.rados.0.smithi045.stdout:3663: finishing write tid 3 to smithi04526579-33 2024-04-26T00:40:57.635 INFO:tasks.rados.rados.0.smithi045.stdout:3663: finishing write tid 4 to smithi04526579-33 2024-04-26T00:40:57.636 INFO:tasks.rados.rados.0.smithi045.stdout:3663: finishing write tid 5 to smithi04526579-33 2024-04-26T00:40:57.636 INFO:tasks.rados.rados.0.smithi045.stdout:3663: finishing write tid 6 to smithi04526579-33 2024-04-26T00:40:57.636 INFO:tasks.rados.rados.0.smithi045.stdout:3663: oid 33 updating version 0 to 2767 2024-04-26T00:40:57.636 INFO:tasks.rados.rados.0.smithi045.stdout:3663: oid 33 version 2767 is already newer than 2765 2024-04-26T00:40:57.637 INFO:tasks.rados.rados.0.smithi045.stdout:3663: oid 33 updating version 2767 to 2769 2024-04-26T00:40:57.637 INFO:tasks.rados.rados.0.smithi045.stdout:3663: oid 33 version 2769 is already newer than 2766 2024-04-26T00:40:57.637 INFO:tasks.rados.rados.0.smithi045.stdout:3663: oid 33 version 2769 is already newer than 2768 2024-04-26T00:40:57.637 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 2769 (ObjNum 1308 snap 404 seq_num 1308) dirty exists 2024-04-26T00:40:57.637 INFO:tasks.rados.rados.0.smithi045.stdout:3663: left oid 33 (ObjNum 1308 snap 404 seq_num 1308) 2024-04-26T00:40:58.594 INFO:tasks.rados.rados.0.smithi045.stdout:3660: done (4 left) 2024-04-26T00:40:58.594 INFO:tasks.rados.rados.0.smithi045.stdout:3661: done (3 left) 2024-04-26T00:40:58.594 INFO:tasks.rados.rados.0.smithi045.stdout:3662: done (2 left) 2024-04-26T00:40:58.594 INFO:tasks.rados.rados.0.smithi045.stdout:3663: done (1 left) 2024-04-26T00:40:58.594 INFO:tasks.rados.rados.0.smithi045.stdout:3664: done (0 left) 2024-04-26T00:40:58.594 INFO:tasks.rados.rados.0.smithi045.stdout:3665: read oid 49 snap -1 2024-04-26T00:40:58.594 INFO:tasks.rados.rados.0.smithi045.stdout:3665: expect (ObjNum 1284 snap 398 seq_num 1284) 2024-04-26T00:40:58.594 INFO:tasks.rados.rados.0.smithi045.stdout:3666: snap_create 2024-04-26T00:40:58.616 INFO:tasks.rados.rados.0.smithi045.stdout:3665: expect (ObjNum 1284 snap 398 seq_num 1284) 2024-04-26T00:40:59.597 INFO:tasks.rados.rados.0.smithi045.stdout:3665: done (1 left) 2024-04-26T00:40:59.597 INFO:tasks.rados.rados.0.smithi045.stdout:3666: done (0 left) 2024-04-26T00:40:59.597 INFO:tasks.rados.rados.0.smithi045.stdout:3667: write (excl) oid 40 current snap is 406 2024-04-26T00:40:59.597 INFO:tasks.rados.rados.0.smithi045.stdout:3667: seq_num 1309 ranges {400363=513657,1332694=660122} 2024-04-26T00:40:59.604 INFO:tasks.rados.rados.0.smithi045.stdout:3667: writing smithi04526579-40 from 400363 to 914020 tid 1 2024-04-26T00:40:59.615 INFO:tasks.rados.rados.0.smithi045.stdout:3667: writing smithi04526579-40 from 1332694 to 1992816 tid 2 2024-04-26T00:40:59.615 INFO:tasks.rados.rados.0.smithi045.stdout:3667: finishing write tid 1 to smithi04526579-40 2024-04-26T00:40:59.616 INFO:tasks.rados.rados.0.smithi045.stdout:3668: read oid 47 snap 405 2024-04-26T00:40:59.616 INFO:tasks.rados.rados.0.smithi045.stdout:3668: expect (ObjNum 1127 snap 340 seq_num 1127) 2024-04-26T00:40:59.616 INFO:tasks.rados.rados.0.smithi045.stdout:3669: copy_from oid 16 from oid 49 current snap is 406 2024-04-26T00:40:59.616 INFO:tasks.rados.rados.0.smithi045.stdout:3670: read oid 26 snap 384 2024-04-26T00:40:59.616 INFO:tasks.rados.rados.0.smithi045.stdout:3670: expect deleted 2024-04-26T00:40:59.616 INFO:tasks.rados.rados.0.smithi045.stdout:3671: delete oid 29 current snap is 406 2024-04-26T00:40:59.621 INFO:tasks.rados.rados.0.smithi045.stdout:3670: done (4 left) 2024-04-26T00:40:59.621 INFO:tasks.rados.rados.0.smithi045.stdout:3671: done (3 left) 2024-04-26T00:40:59.621 INFO:tasks.rados.rados.0.smithi045.stdout:3672: snap_remove snap 396 2024-04-26T00:41:00.606 INFO:tasks.rados.rados.0.smithi045.stdout:3672: done (3 left) 2024-04-26T00:41:00.606 INFO:tasks.rados.rados.0.smithi045.stdout:3673: rollback oid 9 current snap is 406 2024-04-26T00:41:00.606 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 9 to 405 2024-04-26T00:41:00.606 INFO:tasks.rados.rados.0.smithi045.stdout:3668: expect (ObjNum 1127 snap 340 seq_num 1127) 2024-04-26T00:41:00.803 INFO:tasks.rados.rados.0.smithi045.stdout:3667: finishing write tid 2 to smithi04526579-40 2024-04-26T00:41:00.804 INFO:tasks.rados.rados.0.smithi045.stdout:3667: finishing write tid 3 to smithi04526579-40 2024-04-26T00:41:00.804 INFO:tasks.rados.rados.0.smithi045.stdout:3667: finishing write tid 4 to smithi04526579-40 2024-04-26T00:41:00.804 INFO:tasks.rados.rados.0.smithi045.stdout:3667: oid 40 updating version 0 to 2478 2024-04-26T00:41:00.804 INFO:tasks.rados.rados.0.smithi045.stdout:3667: oid 40 updating version 2478 to 2480 2024-04-26T00:41:00.804 INFO:tasks.rados.rados.0.smithi045.stdout:3667: oid 40 version 2480 is already newer than 2479 2024-04-26T00:41:00.804 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 2480 (ObjNum 1309 snap 406 seq_num 1309) dirty exists 2024-04-26T00:41:00.804 INFO:tasks.rados.rados.0.smithi045.stdout:3667: left oid 40 (ObjNum 1309 snap 406 seq_num 1309) 2024-04-26T00:41:00.804 INFO:tasks.rados.rados.0.smithi045.stdout:3669: finishing copy_from racing read to smithi04526579-16 2024-04-26T00:41:00.804 INFO:tasks.rados.rados.0.smithi045.stdout:3669: finishing copy_from to smithi04526579-16 2024-04-26T00:41:00.804 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 2496 (ObjNum 1284 snap 398 seq_num 1284) dirty exists 2024-04-26T00:41:00.804 INFO:tasks.rados.rados.0.smithi045.stdout:3667: done (3 left) 2024-04-26T00:41:00.804 INFO:tasks.rados.rados.0.smithi045.stdout:3668: done (2 left) 2024-04-26T00:41:00.804 INFO:tasks.rados.rados.0.smithi045.stdout:3669: done (1 left) 2024-04-26T00:41:00.804 INFO:tasks.rados.rados.0.smithi045.stdout:3674: snap_create 2024-04-26T00:41:00.805 INFO:tasks.rados.rados.0.smithi045.stdout:3673: finishing rollback tid 0 to smithi04526579-9 2024-04-26T00:41:00.805 INFO:tasks.rados.rados.0.smithi045.stdout:3673: finishing rollback tid 1 to smithi04526579-9 2024-04-26T00:41:00.805 INFO:tasks.rados.rados.0.smithi045.stdout:3673: finishing rollback tid 2 to smithi04526579-9 2024-04-26T00:41:00.805 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 2530 (ObjNum 1285 snap 398 seq_num 1285) dirty exists 2024-04-26T00:41:01.250 INFO:tasks.rados.rados.0.smithi045.stdout:3673: done (1 left) 2024-04-26T00:41:01.250 INFO:tasks.rados.rados.0.smithi045.stdout:3674: done (0 left) 2024-04-26T00:41:01.250 INFO:tasks.rados.rados.0.smithi045.stdout:3675: write oid 24 current snap is 407 2024-04-26T00:41:01.250 INFO:tasks.rados.rados.0.smithi045.stdout:3675: seq_num 1311 ranges {433977=664155,1733305=300672} 2024-04-26T00:41:01.267 INFO:tasks.rados.rados.0.smithi045.stdout:3675: writing smithi04526579-24 from 433977 to 1098132 tid 1 2024-04-26T00:41:01.283 INFO:tasks.rados.rados.0.smithi045.stdout:3675: writing smithi04526579-24 from 1733305 to 2033977 tid 2 2024-04-26T00:41:01.283 INFO:tasks.rados.rados.0.smithi045.stdout:3676: rollback oid 50 current snap is 407 2024-04-26T00:41:01.283 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 50 to 389 2024-04-26T00:41:01.283 INFO:tasks.rados.rados.0.smithi045.stdout:3677: delete oid 19 current snap is 407 2024-04-26T00:41:01.284 INFO:tasks.rados.rados.0.smithi045.stdout:3675: finishing write tid 1 to smithi04526579-24 2024-04-26T00:41:01.298 INFO:tasks.rados.rados.0.smithi045.stdout:3675: finishing write tid 2 to smithi04526579-24 2024-04-26T00:41:01.298 INFO:tasks.rados.rados.0.smithi045.stdout:3675: finishing write tid 3 to smithi04526579-24 2024-04-26T00:41:01.299 INFO:tasks.rados.rados.0.smithi045.stdout:3677: done (2 left) 2024-04-26T00:41:01.299 INFO:tasks.rados.rados.0.smithi045.stdout:3678: read oid 44 snap -1 2024-04-26T00:41:01.299 INFO:tasks.rados.rados.0.smithi045.stdout:3678: expect (ObjNum 1075 snap 322 seq_num 1075) 2024-04-26T00:41:01.300 INFO:tasks.rados.rados.0.smithi045.stdout:3679: write (excl) oid 21 current snap is 407 2024-04-26T00:41:01.300 INFO:tasks.rados.rados.0.smithi045.stdout:3675: finishing write tid 4 to smithi04526579-24 2024-04-26T00:41:01.300 INFO:tasks.rados.rados.0.smithi045.stdout:3675: oid 24 updating version 0 to 2773 2024-04-26T00:41:01.300 INFO:tasks.rados.rados.0.smithi045.stdout:3675: oid 24 updating version 2773 to 2774 2024-04-26T00:41:01.300 INFO:tasks.rados.rados.0.smithi045.stdout:3675: oid 24 updating version 2774 to 2775 2024-04-26T00:41:01.300 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 2775 (ObjNum 1311 snap 407 seq_num 1311) dirty exists 2024-04-26T00:41:01.300 INFO:tasks.rados.rados.0.smithi045.stdout:3675: left oid 24 (ObjNum 1311 snap 407 seq_num 1311) 2024-04-26T00:41:01.300 INFO:tasks.rados.rados.0.smithi045.stdout:3679: seq_num 1313 ranges {467591=414653,1450269=617322} 2024-04-26T00:41:01.313 INFO:tasks.rados.rados.0.smithi045.stdout:3679: writing smithi04526579-21 from 467591 to 882244 tid 1 2024-04-26T00:41:01.332 INFO:tasks.rados.rados.0.smithi045.stdout:3679: writing smithi04526579-21 from 1450269 to 2067591 tid 2 2024-04-26T00:41:01.332 INFO:tasks.rados.rados.0.smithi045.stdout:3675: done (3 left) 2024-04-26T00:41:01.333 INFO:tasks.rados.rados.0.smithi045.stdout:3680: copy_from oid 29 from oid 46 current snap is 407 2024-04-26T00:41:01.333 INFO:tasks.rados.rados.0.smithi045.stdout:3676: finishing rollback tid 0 to smithi04526579-50 2024-04-26T00:41:01.333 INFO:tasks.rados.rados.0.smithi045.stdout:3676: finishing rollback tid 1 to smithi04526579-50 2024-04-26T00:41:01.333 INFO:tasks.rados.rados.0.smithi045.stdout:3676: finishing rollback tid 2 to smithi04526579-50 2024-04-26T00:41:01.333 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 2633 (ObjNum 1207 snap 373 seq_num 1207) dirty exists 2024-04-26T00:41:01.333 INFO:tasks.rados.rados.0.smithi045.stdout:3678: expect (ObjNum 1075 snap 322 seq_num 1075) 2024-04-26T00:41:01.470 INFO:tasks.rados.rados.0.smithi045.stdout:3679: finishing write tid 1 to smithi04526579-21 2024-04-26T00:41:01.470 INFO:tasks.rados.rados.0.smithi045.stdout:3680: finishing copy_from to smithi04526579-29 2024-04-26T00:41:01.471 INFO:tasks.rados.rados.0.smithi045.stdout:3680: got expected ENOENT (src dne) 2024-04-26T00:41:01.471 INFO:tasks.rados.rados.0.smithi045.stdout:3680: finishing copy_from racing read to smithi04526579-29 2024-04-26T00:41:01.471 INFO:tasks.rados.rados.0.smithi045.stdout:3679: finishing write tid 2 to smithi04526579-21 2024-04-26T00:41:01.471 INFO:tasks.rados.rados.0.smithi045.stdout:3679: finishing write tid 3 to smithi04526579-21 2024-04-26T00:41:01.471 INFO:tasks.rados.rados.0.smithi045.stdout:3679: finishing write tid 4 to smithi04526579-21 2024-04-26T00:41:01.471 INFO:tasks.rados.rados.0.smithi045.stdout:3679: oid 21 updating version 0 to 2485 2024-04-26T00:41:01.471 INFO:tasks.rados.rados.0.smithi045.stdout:3679: oid 21 version 2485 is already newer than 2483 2024-04-26T00:41:01.471 INFO:tasks.rados.rados.0.smithi045.stdout:3679: oid 21 version 2485 is already newer than 2484 2024-04-26T00:41:01.471 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 2485 (ObjNum 1313 snap 407 seq_num 1313) dirty exists 2024-04-26T00:41:01.471 INFO:tasks.rados.rados.0.smithi045.stdout:3679: left oid 21 (ObjNum 1313 snap 407 seq_num 1313) 2024-04-26T00:41:01.471 INFO:tasks.rados.rados.0.smithi045.stdout:3676: done (3 left) 2024-04-26T00:41:01.471 INFO:tasks.rados.rados.0.smithi045.stdout:3678: done (2 left) 2024-04-26T00:41:01.471 INFO:tasks.rados.rados.0.smithi045.stdout:3679: done (1 left) 2024-04-26T00:41:01.472 INFO:tasks.rados.rados.0.smithi045.stdout:3680: done (0 left) 2024-04-26T00:41:01.472 INFO:tasks.rados.rados.0.smithi045.stdout:3681: read oid 16 snap -1 2024-04-26T00:41:01.472 INFO:tasks.rados.rados.0.smithi045.stdout:3681: expect (ObjNum 1284 snap 398 seq_num 1284) 2024-04-26T00:41:01.472 INFO:tasks.rados.rados.0.smithi045.stdout:3682: write oid 5 current snap is 407 2024-04-26T00:41:01.472 INFO:tasks.rados.rados.0.smithi045.stdout:3682: seq_num 1314 ranges {484398=489902,1392398=411118} 2024-04-26T00:41:01.485 INFO:tasks.rados.rados.0.smithi045.stdout:3682: writing smithi04526579-5 from 484398 to 974300 tid 1 2024-04-26T00:41:01.497 INFO:tasks.rados.rados.0.smithi045.stdout:3682: writing smithi04526579-5 from 1392398 to 1803516 tid 2 2024-04-26T00:41:01.497 INFO:tasks.rados.rados.0.smithi045.stdout:3683: snap_remove snap 389 2024-04-26T00:41:03.049 INFO:tasks.rados.rados.0.smithi045.stdout:3683: done (2 left) 2024-04-26T00:41:03.049 INFO:tasks.rados.rados.0.smithi045.stdout:3684: write (excl) oid 26 current snap is 407 2024-04-26T00:41:03.049 INFO:tasks.rados.rados.0.smithi045.stdout:3684: seq_num 1315 ranges {501205=565151,1850880=250325} 2024-04-26T00:41:03.068 INFO:tasks.rados.rados.0.smithi045.stdout:3684: writing smithi04526579-26 from 501205 to 1066356 tid 1 2024-04-26T00:41:03.086 INFO:tasks.rados.rados.0.smithi045.stdout:3684: writing smithi04526579-26 from 1850880 to 2101205 tid 2 2024-04-26T00:41:03.086 INFO:tasks.rados.rados.0.smithi045.stdout:3685: snap_remove snap 392 2024-04-26T00:41:04.061 INFO:tasks.rados.rados.0.smithi045.stdout:3685: done (3 left) 2024-04-26T00:41:04.062 INFO:tasks.rados.rados.0.smithi045.stdout:3686: read oid 36 snap 397 2024-04-26T00:41:04.062 INFO:tasks.rados.rados.0.smithi045.stdout:3686: expect (ObjNum 1197 snap 372 seq_num 1197) 2024-04-26T00:41:04.062 INFO:tasks.rados.rados.0.smithi045.stdout:3687: read oid 6 snap -1 2024-04-26T00:41:04.062 INFO:tasks.rados.rados.0.smithi045.stdout:3687: expect deleted 2024-04-26T00:41:04.062 INFO:tasks.rados.rados.0.smithi045.stdout:3688: snap_remove snap 398 2024-04-26T00:41:05.061 INFO:tasks.rados.rados.0.smithi045.stdout:3681: expect (ObjNum 1284 snap 398 seq_num 1284) 2024-04-26T00:41:05.259 INFO:tasks.rados.rados.0.smithi045.stdout:3682: finishing write tid 1 to smithi04526579-5 2024-04-26T00:41:05.259 INFO:tasks.rados.rados.0.smithi045.stdout:3682: finishing write tid 2 to smithi04526579-5 2024-04-26T00:41:05.259 INFO:tasks.rados.rados.0.smithi045.stdout:3682: finishing write tid 3 to smithi04526579-5 2024-04-26T00:41:05.259 INFO:tasks.rados.rados.0.smithi045.stdout:3682: finishing write tid 4 to smithi04526579-5 2024-04-26T00:41:05.259 INFO:tasks.rados.rados.0.smithi045.stdout:3682: oid 5 updating version 0 to 2776 2024-04-26T00:41:05.259 INFO:tasks.rados.rados.0.smithi045.stdout:3682: oid 5 updating version 2776 to 2778 2024-04-26T00:41:05.259 INFO:tasks.rados.rados.0.smithi045.stdout:3682: oid 5 version 2778 is already newer than 2777 2024-04-26T00:41:05.259 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 2778 (ObjNum 1314 snap 407 seq_num 1314) dirty exists 2024-04-26T00:41:05.259 INFO:tasks.rados.rados.0.smithi045.stdout:3682: left oid 5 (ObjNum 1314 snap 407 seq_num 1314) 2024-04-26T00:41:05.259 INFO:tasks.rados.rados.0.smithi045.stdout:3684: finishing write tid 1 to smithi04526579-26 2024-04-26T00:41:05.260 INFO:tasks.rados.rados.0.smithi045.stdout:3684: finishing write tid 2 to smithi04526579-26 2024-04-26T00:41:05.260 INFO:tasks.rados.rados.0.smithi045.stdout:3684: finishing write tid 3 to smithi04526579-26 2024-04-26T00:41:05.260 INFO:tasks.rados.rados.0.smithi045.stdout:3684: finishing write tid 4 to smithi04526579-26 2024-04-26T00:41:05.260 INFO:tasks.rados.rados.0.smithi045.stdout:3684: oid 26 updating version 0 to 2520 2024-04-26T00:41:05.260 INFO:tasks.rados.rados.0.smithi045.stdout:3684: oid 26 version 2520 is already newer than 2519 2024-04-26T00:41:05.260 INFO:tasks.rados.rados.0.smithi045.stdout:3684: oid 26 updating version 2520 to 2521 2024-04-26T00:41:05.260 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 2521 (ObjNum 1315 snap 407 seq_num 1315) dirty exists 2024-04-26T00:41:05.260 INFO:tasks.rados.rados.0.smithi045.stdout:3684: left oid 26 (ObjNum 1315 snap 407 seq_num 1315) 2024-04-26T00:41:05.260 INFO:tasks.rados.rados.0.smithi045.stdout:3686: expect (ObjNum 1197 snap 372 seq_num 1197) 2024-04-26T00:41:05.383 INFO:tasks.rados.rados.0.smithi045.stdout:3681: done (5 left) 2024-04-26T00:41:05.383 INFO:tasks.rados.rados.0.smithi045.stdout:3682: done (4 left) 2024-04-26T00:41:05.383 INFO:tasks.rados.rados.0.smithi045.stdout:3684: done (3 left) 2024-04-26T00:41:05.384 INFO:tasks.rados.rados.0.smithi045.stdout:3686: done (2 left) 2024-04-26T00:41:05.384 INFO:tasks.rados.rados.0.smithi045.stdout:3687: done (1 left) 2024-04-26T00:41:05.384 INFO:tasks.rados.rados.0.smithi045.stdout:3688: done (0 left) 2024-04-26T00:41:05.384 INFO:tasks.rados.rados.0.smithi045.stdout:3689: rollback oid 40 current snap is 407 2024-04-26T00:41:05.384 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 40 to 400 2024-04-26T00:41:05.384 INFO:tasks.rados.rados.0.smithi045.stdout:3690: delete oid 39 current snap is 407 2024-04-26T00:41:05.392 INFO:tasks.rados.rados.0.smithi045.stdout:3689: finishing rollback tid 0 to smithi04526579-40 2024-04-26T00:41:05.393 INFO:tasks.rados.rados.0.smithi045.stdout:3689: finishing rollback tid 1 to smithi04526579-40 2024-04-26T00:41:05.393 INFO:tasks.rados.rados.0.smithi045.stdout:3689: finishing rollback tid 2 to smithi04526579-40 2024-04-26T00:41:05.393 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 2483 (ObjNum 1178 snap 366 seq_num 1178) dirty exists 2024-04-26T00:41:05.394 INFO:tasks.rados.rados.0.smithi045.stdout:3689: done (1 left) 2024-04-26T00:41:05.394 INFO:tasks.rados.rados.0.smithi045.stdout:3690: done (0 left) 2024-04-26T00:41:05.394 INFO:tasks.rados.rados.0.smithi045.stdout:3691: write oid 28 current snap is 407 2024-04-26T00:41:05.394 INFO:tasks.rados.rados.0.smithi045.stdout:3691: seq_num 1317 ranges {534819=432002,1684197=450622} 2024-04-26T00:41:05.399 INFO:tasks.rados.rados.0.smithi045.stdout:3691: writing smithi04526579-28 from 534819 to 966821 tid 1 2024-04-26T00:41:05.405 INFO:tasks.rados.rados.0.smithi045.stdout:3691: writing smithi04526579-28 from 1684197 to 2134819 tid 2 2024-04-26T00:41:05.406 INFO:tasks.rados.rados.0.smithi045.stdout:3692: delete oid 27 current snap is 407 2024-04-26T00:41:05.411 INFO:tasks.rados.rados.0.smithi045.stdout:3691: finishing write tid 1 to smithi04526579-28 2024-04-26T00:41:05.415 INFO:tasks.rados.rados.0.smithi045.stdout:3691: finishing write tid 2 to smithi04526579-28 2024-04-26T00:41:05.415 INFO:tasks.rados.rados.0.smithi045.stdout:3692: done (1 left) 2024-04-26T00:41:05.415 INFO:tasks.rados.rados.0.smithi045.stdout:3693: write (excl) oid 8 current snap is 407 2024-04-26T00:41:05.415 INFO:tasks.rados.rados.0.smithi045.stdout:3693: seq_num 1319 ranges {568433=582500,1684808=483625} 2024-04-26T00:41:05.421 INFO:tasks.rados.rados.0.smithi045.stdout:3693: writing smithi04526579-8 from 568433 to 1150933 tid 1 2024-04-26T00:41:05.428 INFO:tasks.rados.rados.0.smithi045.stdout:3693: writing smithi04526579-8 from 1684808 to 2168433 tid 2 2024-04-26T00:41:05.428 INFO:tasks.rados.rados.0.smithi045.stdout:3694: rollback oid 6 current snap is 407 2024-04-26T00:41:05.429 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 6 to 397 2024-04-26T00:41:05.429 INFO:tasks.rados.rados.0.smithi045.stdout:3691: finishing write tid 3 to smithi04526579-28 2024-04-26T00:41:05.429 INFO:tasks.rados.rados.0.smithi045.stdout:3691: finishing write tid 4 to smithi04526579-28 2024-04-26T00:41:05.429 INFO:tasks.rados.rados.0.smithi045.stdout:3691: oid 28 updating version 0 to 2524 2024-04-26T00:41:05.429 INFO:tasks.rados.rados.0.smithi045.stdout:3691: oid 28 version 2524 is already newer than 2522 2024-04-26T00:41:05.429 INFO:tasks.rados.rados.0.smithi045.stdout:3691: oid 28 version 2524 is already newer than 2523 2024-04-26T00:41:05.429 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 2524 (ObjNum 1317 snap 407 seq_num 1317) dirty exists 2024-04-26T00:41:05.429 INFO:tasks.rados.rados.0.smithi045.stdout:3691: left oid 28 (ObjNum 1317 snap 407 seq_num 1317) 2024-04-26T00:41:05.429 INFO:tasks.rados.rados.0.smithi045.stdout:3691: done (2 left) 2024-04-26T00:41:05.429 INFO:tasks.rados.rados.0.smithi045.stdout:3695: read oid 23 snap 399 2024-04-26T00:41:05.429 INFO:tasks.rados.rados.0.smithi045.stdout:3695: expect deleted 2024-04-26T00:41:05.429 INFO:tasks.rados.rados.0.smithi045.stdout:3696: copy_from oid 50 from oid 17 current snap is 407 2024-04-26T00:41:05.429 INFO:tasks.rados.rados.0.smithi045.stdout:3697: delete oid 9 current snap is 407 2024-04-26T00:41:05.432 INFO:tasks.rados.rados.0.smithi045.stdout:3694: finishing rollback tid 1 to smithi04526579-6 2024-04-26T00:41:05.433 INFO:tasks.rados.rados.0.smithi045.stdout:3694: finishing rollback tid 2 to smithi04526579-6 2024-04-26T00:41:05.433 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 2464 (ObjNum 1060 snap 318 seq_num 1060) dirty exists 2024-04-26T00:41:05.438 INFO:tasks.rados.rados.0.smithi045.stdout:3696: finishing copy_from racing read to smithi04526579-50 2024-04-26T00:41:05.438 INFO:tasks.rados.rados.0.smithi045.stdout:3694: done (4 left) 2024-04-26T00:41:05.438 INFO:tasks.rados.rados.0.smithi045.stdout:3695: done (3 left) 2024-04-26T00:41:05.438 INFO:tasks.rados.rados.0.smithi045.stdout:3697: done (2 left) 2024-04-26T00:41:05.438 INFO:tasks.rados.rados.0.smithi045.stdout:3698: snap_remove snap 390 2024-04-26T00:41:06.076 INFO:tasks.rados.rados.0.smithi045.stdout:3698: done (2 left) 2024-04-26T00:41:06.076 INFO:tasks.rados.rados.0.smithi045.stdout:3699: write (excl) oid 41 current snap is 407 2024-04-26T00:41:06.076 INFO:tasks.rados.rados.0.smithi045.stdout:3693: finishing write tid 1 to smithi04526579-8 2024-04-26T00:41:06.076 INFO:tasks.rados.rados.0.smithi045.stdout:3693: finishing write tid 2 to smithi04526579-8 2024-04-26T00:41:06.076 INFO:tasks.rados.rados.0.smithi045.stdout:3693: finishing write tid 3 to smithi04526579-8 2024-04-26T00:41:06.076 INFO:tasks.rados.rados.0.smithi045.stdout:3693: finishing write tid 4 to smithi04526579-8 2024-04-26T00:41:06.076 INFO:tasks.rados.rados.0.smithi045.stdout:3693: oid 8 updating version 0 to 2237 2024-04-26T00:41:06.076 INFO:tasks.rados.rados.0.smithi045.stdout:3693: oid 8 version 2237 is already newer than 2235 2024-04-26T00:41:06.076 INFO:tasks.rados.rados.0.smithi045.stdout:3693: oid 8 version 2237 is already newer than 2236 2024-04-26T00:41:06.077 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 2237 (ObjNum 1319 snap 407 seq_num 1319) dirty exists 2024-04-26T00:41:06.077 INFO:tasks.rados.rados.0.smithi045.stdout:3693: left oid 8 (ObjNum 1319 snap 407 seq_num 1319) 2024-04-26T00:41:06.077 INFO:tasks.rados.rados.0.smithi045.stdout:3696: finishing copy_from to smithi04526579-50 2024-04-26T00:41:06.077 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 2627 (ObjNum 1238 snap 384 seq_num 1238) dirty exists 2024-04-26T00:41:06.077 INFO:tasks.rados.rados.0.smithi045.stdout:3699: seq_num 1321 ranges {602047=732998,1801772=400275} 2024-04-26T00:41:06.100 INFO:tasks.rados.rados.0.smithi045.stdout:3699: writing smithi04526579-41 from 602047 to 1335045 tid 1 2024-04-26T00:41:06.116 INFO:tasks.rados.rados.0.smithi045.stdout:3699: writing smithi04526579-41 from 1801772 to 2202047 tid 2 2024-04-26T00:41:06.116 INFO:tasks.rados.rados.0.smithi045.stdout:3693: done (2 left) 2024-04-26T00:41:06.116 INFO:tasks.rados.rados.0.smithi045.stdout:3696: done (1 left) 2024-04-26T00:41:06.116 INFO:tasks.rados.rados.0.smithi045.stdout:3700: rollback oid 46 current snap is 407 2024-04-26T00:41:06.116 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 46 to 332 2024-04-26T00:41:06.117 INFO:tasks.rados.rados.0.smithi045.stdout:3701: read oid 21 snap -1 2024-04-26T00:41:06.117 INFO:tasks.rados.rados.0.smithi045.stdout:3701: expect (ObjNum 1313 snap 407 seq_num 1313) 2024-04-26T00:41:06.117 INFO:tasks.rados.rados.0.smithi045.stdout:3702: rollback oid 47 current snap is 407 2024-04-26T00:41:06.117 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 47 to 393 2024-04-26T00:41:06.117 INFO:tasks.rados.rados.0.smithi045.stdout:3703: snap_remove snap 394 2024-04-26T00:41:06.253 INFO:tasks.rados.rados.0.smithi045.stdout:3703: done (4 left) 2024-04-26T00:41:06.254 INFO:tasks.rados.rados.0.smithi045.stdout:3704: snap_create 2024-04-26T00:41:06.254 INFO:tasks.rados.rados.0.smithi045.stdout:3699: finishing write tid 1 to smithi04526579-41 2024-04-26T00:41:06.254 INFO:tasks.rados.rados.0.smithi045.stdout:3700: finishing rollback tid 1 to smithi04526579-46 2024-04-26T00:41:06.254 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 2457 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-04-26T00:41:06.254 INFO:tasks.rados.rados.0.smithi045.stdout:3701: expect (ObjNum 1313 snap 407 seq_num 1313) 2024-04-26T00:41:06.380 INFO:tasks.rados.rados.0.smithi045.stdout:3702: finishing rollback tid 0 to smithi04526579-47 2024-04-26T00:41:06.381 INFO:tasks.rados.rados.0.smithi045.stdout:3699: finishing write tid 2 to smithi04526579-41 2024-04-26T00:41:06.381 INFO:tasks.rados.rados.0.smithi045.stdout:3699: finishing write tid 3 to smithi04526579-41 2024-04-26T00:41:06.381 INFO:tasks.rados.rados.0.smithi045.stdout:3699: finishing write tid 4 to smithi04526579-41 2024-04-26T00:41:06.381 INFO:tasks.rados.rados.0.smithi045.stdout:3699: oid 41 updating version 0 to 2525 2024-04-26T00:41:06.381 INFO:tasks.rados.rados.0.smithi045.stdout:3699: oid 41 updating version 2525 to 2526 2024-04-26T00:41:06.381 INFO:tasks.rados.rados.0.smithi045.stdout:3699: oid 41 updating version 2526 to 2527 2024-04-26T00:41:06.381 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 41 v 2527 (ObjNum 1321 snap 407 seq_num 1321) dirty exists 2024-04-26T00:41:06.381 INFO:tasks.rados.rados.0.smithi045.stdout:3699: left oid 41 (ObjNum 1321 snap 407 seq_num 1321) 2024-04-26T00:41:06.381 INFO:tasks.rados.rados.0.smithi045.stdout:3702: finishing rollback tid 1 to smithi04526579-47 2024-04-26T00:41:06.381 INFO:tasks.rados.rados.0.smithi045.stdout:3702: finishing rollback tid 2 to smithi04526579-47 2024-04-26T00:41:06.381 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 2507 (ObjNum 1127 snap 340 seq_num 1127) dirty exists 2024-04-26T00:41:07.266 INFO:tasks.rados.rados.0.smithi045.stdout:3699: done (4 left) 2024-04-26T00:41:07.266 INFO:tasks.rados.rados.0.smithi045.stdout:3700: done (3 left) 2024-04-26T00:41:07.267 INFO:tasks.rados.rados.0.smithi045.stdout:3701: done (2 left) 2024-04-26T00:41:07.267 INFO:tasks.rados.rados.0.smithi045.stdout:3702: done (1 left) 2024-04-26T00:41:07.267 INFO:tasks.rados.rados.0.smithi045.stdout:3704: done (0 left) 2024-04-26T00:41:07.267 INFO:tasks.rados.rados.0.smithi045.stdout:3705: rollback oid 19 current snap is 408 2024-04-26T00:41:07.267 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 19 to 406 2024-04-26T00:41:07.267 INFO:tasks.rados.rados.0.smithi045.stdout:3706: write oid 49 current snap is 408 2024-04-26T00:41:07.267 INFO:tasks.rados.rados.0.smithi045.stdout:3706: seq_num 1322 ranges {618854=408247,1743901=425794} 2024-04-26T00:41:07.283 INFO:tasks.rados.rados.0.smithi045.stdout:3706: writing smithi04526579-49 from 618854 to 1027101 tid 1 2024-04-26T00:41:07.302 INFO:tasks.rados.rados.0.smithi045.stdout:3706: writing smithi04526579-49 from 1743901 to 2169695 tid 2 2024-04-26T00:41:07.302 INFO:tasks.rados.rados.0.smithi045.stdout:3707: delete oid 48 current snap is 408 2024-04-26T00:41:07.303 INFO:tasks.rados.rados.0.smithi045.stdout:3705: finishing rollback tid 1 to smithi04526579-19 2024-04-26T00:41:07.303 INFO:tasks.rados.rados.0.smithi045.stdout:3705: finishing rollback tid 2 to smithi04526579-19 2024-04-26T00:41:07.303 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 2516 (ObjNum 1239 snap 384 seq_num 1239) dirty exists 2024-04-26T00:41:07.303 INFO:tasks.rados.rados.0.smithi045.stdout:3706: finishing write tid 1 to smithi04526579-49 2024-04-26T00:41:07.313 INFO:tasks.rados.rados.0.smithi045.stdout:3706: finishing write tid 2 to smithi04526579-49 2024-04-26T00:41:07.313 INFO:tasks.rados.rados.0.smithi045.stdout:3706: finishing write tid 3 to smithi04526579-49 2024-04-26T00:41:07.314 INFO:tasks.rados.rados.0.smithi045.stdout:3706: finishing write tid 4 to smithi04526579-49 2024-04-26T00:41:07.314 INFO:tasks.rados.rados.0.smithi045.stdout:3706: oid 49 updating version 0 to 2508 2024-04-26T00:41:07.314 INFO:tasks.rados.rados.0.smithi045.stdout:3706: oid 49 updating version 2508 to 2509 2024-04-26T00:41:07.314 INFO:tasks.rados.rados.0.smithi045.stdout:3706: oid 49 updating version 2509 to 2510 2024-04-26T00:41:07.315 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 2510 (ObjNum 1322 snap 408 seq_num 1322) dirty exists 2024-04-26T00:41:07.315 INFO:tasks.rados.rados.0.smithi045.stdout:3706: left oid 49 (ObjNum 1322 snap 408 seq_num 1322) 2024-04-26T00:41:07.316 INFO:tasks.rados.rados.0.smithi045.stdout:3705: done (2 left) 2024-04-26T00:41:07.316 INFO:tasks.rados.rados.0.smithi045.stdout:3706: done (1 left) 2024-04-26T00:41:07.316 INFO:tasks.rados.rados.0.smithi045.stdout:3707: done (0 left) 2024-04-26T00:41:07.316 INFO:tasks.rados.rados.0.smithi045.stdout:3708: rollback oid 49 current snap is 408 2024-04-26T00:41:07.316 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 49 to 403 2024-04-26T00:41:07.316 INFO:tasks.rados.rados.0.smithi045.stdout:3709: snap_create 2024-04-26T00:41:07.328 INFO:tasks.rados.rados.0.smithi045.stdout:3708: finishing rollback tid 0 to smithi04526579-49 2024-04-26T00:41:07.328 INFO:tasks.rados.rados.0.smithi045.stdout:3708: finishing rollback tid 1 to smithi04526579-49 2024-04-26T00:41:07.329 INFO:tasks.rados.rados.0.smithi045.stdout:3708: finishing rollback tid 2 to smithi04526579-49 2024-04-26T00:41:07.329 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 2513 (ObjNum 1284 snap 398 seq_num 1284) dirty exists 2024-04-26T00:41:08.273 INFO:tasks.rados.rados.0.smithi045.stdout:3708: done (1 left) 2024-04-26T00:41:08.273 INFO:tasks.rados.rados.0.smithi045.stdout:3709: done (0 left) 2024-04-26T00:41:08.273 INFO:tasks.rados.rados.0.smithi045.stdout:3710: read oid 38 snap -1 2024-04-26T00:41:08.273 INFO:tasks.rados.rados.0.smithi045.stdout:3710: expect deleted 2024-04-26T00:41:08.273 INFO:tasks.rados.rados.0.smithi045.stdout:3711: delete oid 36 current snap is 409 2024-04-26T00:41:08.285 INFO:tasks.rados.rados.0.smithi045.stdout:3710: done (1 left) 2024-04-26T00:41:08.285 INFO:tasks.rados.rados.0.smithi045.stdout:3711: done (0 left) 2024-04-26T00:41:08.285 INFO:tasks.rados.rados.0.smithi045.stdout:3712: snap_create 2024-04-26T00:41:09.323 INFO:tasks.rados.rados.0.smithi045.stdout:3712: done (0 left) 2024-04-26T00:41:09.323 INFO:tasks.rados.rados.0.smithi045.stdout:3713: write oid 30 current snap is 410 2024-04-26T00:41:09.323 INFO:tasks.rados.rados.0.smithi045.stdout:3713: seq_num 1325 ranges {669275=750347,2035700=233575} 2024-04-26T00:41:09.346 INFO:tasks.rados.rados.0.smithi045.stdout:3713: writing smithi04526579-30 from 669275 to 1419622 tid 1 2024-04-26T00:41:09.361 INFO:tasks.rados.rados.0.smithi045.stdout:3713: writing smithi04526579-30 from 2035700 to 2269275 tid 2 2024-04-26T00:41:09.361 INFO:tasks.rados.rados.0.smithi045.stdout:3714: snap_remove snap 397 2024-04-26T00:41:10.337 INFO:tasks.rados.rados.0.smithi045.stdout:3714: done (1 left) 2024-04-26T00:41:10.337 INFO:tasks.rados.rados.0.smithi045.stdout:3715: snap_remove snap 400 2024-04-26T00:41:11.341 INFO:tasks.rados.rados.0.smithi045.stdout:3715: done (1 left) 2024-04-26T00:41:11.342 INFO:tasks.rados.rados.0.smithi045.stdout:3716: write oid 11 current snap is 410 2024-04-26T00:41:11.342 INFO:tasks.rados.rados.0.smithi045.stdout:3716: seq_num 1326 ranges {686082=425596,1577829=433132} 2024-04-26T00:41:11.361 INFO:tasks.rados.rados.0.smithi045.stdout:3716: writing smithi04526579-11 from 686082 to 1111678 tid 1 2024-04-26T00:41:11.377 INFO:tasks.rados.rados.0.smithi045.stdout:3716: writing smithi04526579-11 from 1577829 to 2010961 tid 2 2024-04-26T00:41:11.377 INFO:tasks.rados.rados.0.smithi045.stdout:3717: write oid 25 current snap is 410 2024-04-26T00:41:11.377 INFO:tasks.rados.rados.0.smithi045.stdout:3717: seq_num 1327 ranges {702889=500845,1636311=576790} 2024-04-26T00:41:11.396 INFO:tasks.rados.rados.0.smithi045.stdout:3717: writing smithi04526579-25 from 702889 to 1203734 tid 1 2024-04-26T00:41:11.409 INFO:tasks.rados.rados.0.smithi045.stdout:3717: writing smithi04526579-25 from 1636311 to 2213101 tid 2 2024-04-26T00:41:11.409 INFO:tasks.rados.rados.0.smithi045.stdout:3718: read oid 41 snap -1 2024-04-26T00:41:11.409 INFO:tasks.rados.rados.0.smithi045.stdout:3718: expect (ObjNum 1321 snap 407 seq_num 1321) 2024-04-26T00:41:11.409 INFO:tasks.rados.rados.0.smithi045.stdout:3719: rollback oid 43 current snap is 410 2024-04-26T00:41:11.409 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 43 to 406 2024-04-26T00:41:11.409 INFO:tasks.rados.rados.0.smithi045.stdout:3713: finishing write tid 1 to smithi04526579-30 2024-04-26T00:41:11.410 INFO:tasks.rados.rados.0.smithi045.stdout:3713: finishing write tid 2 to smithi04526579-30 2024-04-26T00:41:11.410 INFO:tasks.rados.rados.0.smithi045.stdout:3713: finishing write tid 3 to smithi04526579-30 2024-04-26T00:41:11.410 INFO:tasks.rados.rados.0.smithi045.stdout:3713: finishing write tid 4 to smithi04526579-30 2024-04-26T00:41:11.410 INFO:tasks.rados.rados.0.smithi045.stdout:3713: oid 30 updating version 0 to 2465 2024-04-26T00:41:11.410 INFO:tasks.rados.rados.0.smithi045.stdout:3713: oid 30 updating version 2465 to 2466 2024-04-26T00:41:11.410 INFO:tasks.rados.rados.0.smithi045.stdout:3713: oid 30 updating version 2466 to 2467 2024-04-26T00:41:11.410 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 2467 (ObjNum 1325 snap 410 seq_num 1325) dirty exists 2024-04-26T00:41:11.410 INFO:tasks.rados.rados.0.smithi045.stdout:3713: left oid 30 (ObjNum 1325 snap 410 seq_num 1325) 2024-04-26T00:41:11.410 INFO:tasks.rados.rados.0.smithi045.stdout:3716: finishing write tid 1 to smithi04526579-11 2024-04-26T00:41:11.410 INFO:tasks.rados.rados.0.smithi045.stdout:3716: finishing write tid 2 to smithi04526579-11 2024-04-26T00:41:11.410 INFO:tasks.rados.rados.0.smithi045.stdout:3713: done (4 left) 2024-04-26T00:41:11.410 INFO:tasks.rados.rados.0.smithi045.stdout:3720: delete oid 17 current snap is 410 2024-04-26T00:41:11.410 INFO:tasks.rados.rados.0.smithi045.stdout:3716: finishing write tid 3 to smithi04526579-11 2024-04-26T00:41:11.410 INFO:tasks.rados.rados.0.smithi045.stdout:3716: finishing write tid 4 to smithi04526579-11 2024-04-26T00:41:11.411 INFO:tasks.rados.rados.0.smithi045.stdout:3716: oid 11 updating version 0 to 2447 2024-04-26T00:41:11.411 INFO:tasks.rados.rados.0.smithi045.stdout:3716: oid 11 version 2447 is already newer than 2446 2024-04-26T00:41:11.411 INFO:tasks.rados.rados.0.smithi045.stdout:3716: oid 11 updating version 2447 to 2448 2024-04-26T00:41:11.411 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 2448 (ObjNum 1326 snap 410 seq_num 1326) dirty exists 2024-04-26T00:41:11.411 INFO:tasks.rados.rados.0.smithi045.stdout:3716: left oid 11 (ObjNum 1326 snap 410 seq_num 1326) 2024-04-26T00:41:11.411 INFO:tasks.rados.rados.0.smithi045.stdout:3717: finishing write tid 1 to smithi04526579-25 2024-04-26T00:41:11.417 INFO:tasks.rados.rados.0.smithi045.stdout:3718: expect (ObjNum 1321 snap 407 seq_num 1321) 2024-04-26T00:41:11.616 INFO:tasks.rados.rados.0.smithi045.stdout:3719: finishing rollback tid 1 to smithi04526579-43 2024-04-26T00:41:11.616 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 2428 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-04-26T00:41:11.616 INFO:tasks.rados.rados.0.smithi045.stdout:3717: finishing write tid 2 to smithi04526579-25 2024-04-26T00:41:11.616 INFO:tasks.rados.rados.0.smithi045.stdout:3717: finishing write tid 3 to smithi04526579-25 2024-04-26T00:41:11.617 INFO:tasks.rados.rados.0.smithi045.stdout:3717: finishing write tid 4 to smithi04526579-25 2024-04-26T00:41:11.617 INFO:tasks.rados.rados.0.smithi045.stdout:3717: oid 25 updating version 0 to 2821 2024-04-26T00:41:11.617 INFO:tasks.rados.rados.0.smithi045.stdout:3717: oid 25 updating version 2821 to 2822 2024-04-26T00:41:11.617 INFO:tasks.rados.rados.0.smithi045.stdout:3717: oid 25 updating version 2822 to 2823 2024-04-26T00:41:11.617 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 2823 (ObjNum 1327 snap 410 seq_num 1327) dirty exists 2024-04-26T00:41:11.617 INFO:tasks.rados.rados.0.smithi045.stdout:3717: left oid 25 (ObjNum 1327 snap 410 seq_num 1327) 2024-04-26T00:41:11.617 INFO:tasks.rados.rados.0.smithi045.stdout:3716: done (4 left) 2024-04-26T00:41:11.617 INFO:tasks.rados.rados.0.smithi045.stdout:3717: done (3 left) 2024-04-26T00:41:11.617 INFO:tasks.rados.rados.0.smithi045.stdout:3718: done (2 left) 2024-04-26T00:41:11.617 INFO:tasks.rados.rados.0.smithi045.stdout:3719: done (1 left) 2024-04-26T00:41:11.617 INFO:tasks.rados.rados.0.smithi045.stdout:3720: done (0 left) 2024-04-26T00:41:11.617 INFO:tasks.rados.rados.0.smithi045.stdout:3721: delete oid 11 current snap is 410 2024-04-26T00:41:11.621 INFO:tasks.rados.rados.0.smithi045.stdout:3721: done (0 left) 2024-04-26T00:41:11.621 INFO:tasks.rados.rados.0.smithi045.stdout:3722: delete oid 13 current snap is 410 2024-04-26T00:41:11.626 INFO:tasks.rados.rados.0.smithi045.stdout:3722: done (0 left) 2024-04-26T00:41:11.626 INFO:tasks.rados.rados.0.smithi045.stdout:3723: snap_remove snap 406 2024-04-26T00:41:12.356 INFO:tasks.rados.rados.0.smithi045.stdout:3723: done (0 left) 2024-04-26T00:41:12.356 INFO:tasks.rados.rados.0.smithi045.stdout:3724: snap_create 2024-04-26T00:41:13.360 INFO:tasks.rados.rados.0.smithi045.stdout:3724: done (0 left) 2024-04-26T00:41:13.360 INFO:tasks.rados.rados.0.smithi045.stdout:3725: copy_from oid 20 from oid 44 current snap is 411 2024-04-26T00:41:13.360 INFO:tasks.rados.rados.0.smithi045.stdout:3726: rollback oid 29 current snap is 411 2024-04-26T00:41:13.361 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 29 to 407 2024-04-26T00:41:13.361 INFO:tasks.rados.rados.0.smithi045.stdout:3727: delete oid 14 current snap is 411 2024-04-26T00:41:13.366 INFO:tasks.rados.rados.0.smithi045.stdout:3726: finishing rollback tid 1 to smithi04526579-29 2024-04-26T00:41:13.366 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 2229 (ObjNum 545 snap 0 seq_num 2242529040) dirty dne 2024-04-26T00:41:13.378 INFO:tasks.rados.rados.0.smithi045.stdout:3726: done (2 left) 2024-04-26T00:41:13.378 INFO:tasks.rados.rados.0.smithi045.stdout:3727: done (1 left) 2024-04-26T00:41:13.378 INFO:tasks.rados.rados.0.smithi045.stdout:3728: copy_from oid 19 from oid 28 current snap is 411 2024-04-26T00:41:13.378 INFO:tasks.rados.rados.0.smithi045.stdout:3729: write (excl) oid 31 current snap is 411 2024-04-26T00:41:13.378 INFO:tasks.rados.rados.0.smithi045.stdout:3729: seq_num 1332 ranges {786924=593443,1928721=458203} 2024-04-26T00:41:13.400 INFO:tasks.rados.rados.0.smithi045.stdout:3729: writing smithi04526579-31 from 786924 to 1380367 tid 1 2024-04-26T00:41:13.417 INFO:tasks.rados.rados.0.smithi045.stdout:3729: writing smithi04526579-31 from 1928721 to 2386924 tid 2 2024-04-26T00:41:13.417 INFO:tasks.rados.rados.0.smithi045.stdout:3730: write (excl) oid 6 current snap is 411 2024-04-26T00:41:13.417 INFO:tasks.rados.rados.0.smithi045.stdout:3725: finishing copy_from racing read to smithi04526579-20 2024-04-26T00:41:13.417 INFO:tasks.rados.rados.0.smithi045.stdout:3725: finishing copy_from to smithi04526579-20 2024-04-26T00:41:13.417 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 2398 (ObjNum 1075 snap 322 seq_num 1075) dirty exists 2024-04-26T00:41:13.417 INFO:tasks.rados.rados.0.smithi045.stdout:3728: finishing copy_from racing read to smithi04526579-19 2024-04-26T00:41:13.418 INFO:tasks.rados.rados.0.smithi045.stdout:3730: seq_num 1333 ranges {403731=668692,1587203=587797} 2024-04-26T00:41:13.433 INFO:tasks.rados.rados.0.smithi045.stdout:3730: writing smithi04526579-6 from 403731 to 1072423 tid 1 2024-04-26T00:41:13.450 INFO:tasks.rados.rados.0.smithi045.stdout:3730: writing smithi04526579-6 from 1587203 to 2175000 tid 2 2024-04-26T00:41:13.450 INFO:tasks.rados.rados.0.smithi045.stdout:3725: done (3 left) 2024-04-26T00:41:13.450 INFO:tasks.rados.rados.0.smithi045.stdout:3731: copy_from oid 11 from oid 34 current snap is 411 2024-04-26T00:41:13.451 INFO:tasks.rados.rados.0.smithi045.stdout:3728: finishing copy_from to smithi04526579-19 2024-04-26T00:41:13.451 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 2524 (ObjNum 1317 snap 407 seq_num 1317) dirty exists 2024-04-26T00:41:13.451 INFO:tasks.rados.rados.0.smithi045.stdout:3729: finishing write tid 1 to smithi04526579-31 2024-04-26T00:41:13.451 INFO:tasks.rados.rados.0.smithi045.stdout:3729: finishing write tid 2 to smithi04526579-31 2024-04-26T00:41:13.451 INFO:tasks.rados.rados.0.smithi045.stdout:3729: finishing write tid 3 to smithi04526579-31 2024-04-26T00:41:13.451 INFO:tasks.rados.rados.0.smithi045.stdout:3729: finishing write tid 4 to smithi04526579-31 2024-04-26T00:41:13.451 INFO:tasks.rados.rados.0.smithi045.stdout:3729: oid 31 updating version 0 to 2532 2024-04-26T00:41:13.451 INFO:tasks.rados.rados.0.smithi045.stdout:3729: oid 31 updating version 2532 to 2533 2024-04-26T00:41:13.451 INFO:tasks.rados.rados.0.smithi045.stdout:3729: oid 31 updating version 2533 to 2534 2024-04-26T00:41:13.451 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 2534 (ObjNum 1332 snap 411 seq_num 1332) dirty exists 2024-04-26T00:41:13.451 INFO:tasks.rados.rados.0.smithi045.stdout:3729: left oid 31 (ObjNum 1332 snap 411 seq_num 1332) 2024-04-26T00:41:13.451 INFO:tasks.rados.rados.0.smithi045.stdout:3730: finishing write tid 1 to smithi04526579-6 2024-04-26T00:41:13.451 INFO:tasks.rados.rados.0.smithi045.stdout:3728: done (3 left) 2024-04-26T00:41:13.451 INFO:tasks.rados.rados.0.smithi045.stdout:3729: done (2 left) 2024-04-26T00:41:13.451 INFO:tasks.rados.rados.0.smithi045.stdout:3732: rollback oid 29 current snap is 411 2024-04-26T00:41:13.452 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 29 to 409 2024-04-26T00:41:13.452 INFO:tasks.rados.rados.0.smithi045.stdout:3733: read oid 4 snap -1 2024-04-26T00:41:13.452 INFO:tasks.rados.rados.0.smithi045.stdout:3733: expect (ObjNum 1282 snap 398 seq_num 1282) 2024-04-26T00:41:13.452 INFO:tasks.rados.rados.0.smithi045.stdout:3734: read oid 37 snap -1 2024-04-26T00:41:13.452 INFO:tasks.rados.rados.0.smithi045.stdout:3734: expect (ObjNum 1075 snap 322 seq_num 1075) 2024-04-26T00:41:13.452 INFO:tasks.rados.rados.0.smithi045.stdout:3735: delete oid 49 current snap is 411 2024-04-26T00:41:13.458 INFO:tasks.rados.rados.0.smithi045.stdout:3730: finishing write tid 2 to smithi04526579-6 2024-04-26T00:41:13.458 INFO:tasks.rados.rados.0.smithi045.stdout:3731: finishing copy_from to smithi04526579-11 2024-04-26T00:41:13.458 INFO:tasks.rados.rados.0.smithi045.stdout:3731: got expected ENOENT (src dne) 2024-04-26T00:41:13.458 INFO:tasks.rados.rados.0.smithi045.stdout:3731: finishing copy_from racing read to smithi04526579-11 2024-04-26T00:41:13.459 INFO:tasks.rados.rados.0.smithi045.stdout:3732: finishing rollback tid 1 to smithi04526579-29 2024-04-26T00:41:13.459 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 2229 (ObjNum 513 snap 0 seq_num 2242613144) dirty dne 2024-04-26T00:41:13.460 INFO:tasks.rados.rados.0.smithi045.stdout:3730: finishing write tid 3 to smithi04526579-6 2024-04-26T00:41:13.461 INFO:tasks.rados.rados.0.smithi045.stdout:3730: finishing write tid 4 to smithi04526579-6 2024-04-26T00:41:13.461 INFO:tasks.rados.rados.0.smithi045.stdout:3730: oid 6 updating version 0 to 2470 2024-04-26T00:41:13.461 INFO:tasks.rados.rados.0.smithi045.stdout:3730: oid 6 version 2470 is already newer than 2469 2024-04-26T00:41:13.461 INFO:tasks.rados.rados.0.smithi045.stdout:3730: oid 6 version 2470 is already newer than 2468 2024-04-26T00:41:13.461 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 2470 (ObjNum 1333 snap 411 seq_num 1333) dirty exists 2024-04-26T00:41:13.461 INFO:tasks.rados.rados.0.smithi045.stdout:3730: left oid 6 (ObjNum 1333 snap 411 seq_num 1333) 2024-04-26T00:41:13.462 INFO:tasks.rados.rados.0.smithi045.stdout:3730: done (5 left) 2024-04-26T00:41:13.462 INFO:tasks.rados.rados.0.smithi045.stdout:3731: done (4 left) 2024-04-26T00:41:13.462 INFO:tasks.rados.rados.0.smithi045.stdout:3732: done (3 left) 2024-04-26T00:41:13.462 INFO:tasks.rados.rados.0.smithi045.stdout:3735: done (2 left) 2024-04-26T00:41:13.462 INFO:tasks.rados.rados.0.smithi045.stdout:3736: snap_create 2024-04-26T00:41:13.462 INFO:tasks.rados.rados.0.smithi045.stdout:3734: expect (ObjNum 1075 snap 322 seq_num 1075) 2024-04-26T00:41:13.607 INFO:tasks.rados.rados.0.smithi045.stdout:3733: expect (ObjNum 1282 snap 398 seq_num 1282) 2024-04-26T00:41:14.374 INFO:tasks.rados.rados.0.smithi045.stdout:3733: done (2 left) 2024-04-26T00:41:14.374 INFO:tasks.rados.rados.0.smithi045.stdout:3734: done (1 left) 2024-04-26T00:41:14.374 INFO:tasks.rados.rados.0.smithi045.stdout:3736: done (0 left) 2024-04-26T00:41:14.374 INFO:tasks.rados.rados.0.smithi045.stdout:3737: rollback oid 9 current snap is 412 2024-04-26T00:41:14.374 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 9 to 346 2024-04-26T00:41:14.374 INFO:tasks.rados.rados.0.smithi045.stdout:3738: write oid 45 current snap is 412 2024-04-26T00:41:14.374 INFO:tasks.rados.rados.0.smithi045.stdout:3738: seq_num 1335 ranges {437345=419190,1587814=591466} 2024-04-26T00:41:14.383 INFO:tasks.rados.rados.0.smithi045.stdout:3738: writing smithi04526579-45 from 437345 to 856535 tid 1 2024-04-26T00:41:14.391 INFO:tasks.rados.rados.0.smithi045.stdout:3738: writing smithi04526579-45 from 1587814 to 2179280 tid 2 2024-04-26T00:41:14.391 INFO:tasks.rados.rados.0.smithi045.stdout:3739: snap_create 2024-04-26T00:41:14.391 INFO:tasks.rados.rados.0.smithi045.stdout:3737: finishing rollback tid 1 to smithi04526579-9 2024-04-26T00:41:14.391 INFO:tasks.rados.rados.0.smithi045.stdout:3737: finishing rollback tid 2 to smithi04526579-9 2024-04-26T00:41:14.391 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 2536 (ObjNum 1099 snap 326 seq_num 1099) dirty exists 2024-04-26T00:41:14.391 INFO:tasks.rados.rados.0.smithi045.stdout:3738: finishing write tid 1 to smithi04526579-45 2024-04-26T00:41:14.401 INFO:tasks.rados.rados.0.smithi045.stdout:3738: finishing write tid 2 to smithi04526579-45 2024-04-26T00:41:14.401 INFO:tasks.rados.rados.0.smithi045.stdout:3738: finishing write tid 3 to smithi04526579-45 2024-04-26T00:41:14.401 INFO:tasks.rados.rados.0.smithi045.stdout:3738: finishing write tid 4 to smithi04526579-45 2024-04-26T00:41:14.401 INFO:tasks.rados.rados.0.smithi045.stdout:3738: oid 45 updating version 0 to 2451 2024-04-26T00:41:14.401 INFO:tasks.rados.rados.0.smithi045.stdout:3738: oid 45 updating version 2451 to 2452 2024-04-26T00:41:14.402 INFO:tasks.rados.rados.0.smithi045.stdout:3738: oid 45 version 2452 is already newer than 2450 2024-04-26T00:41:14.402 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 2452 (ObjNum 1335 snap 412 seq_num 1335) dirty exists 2024-04-26T00:41:14.402 INFO:tasks.rados.rados.0.smithi045.stdout:3738: left oid 45 (ObjNum 1335 snap 412 seq_num 1335) 2024-04-26T00:41:15.377 INFO:tasks.rados.rados.0.smithi045.stdout:3737: done (2 left) 2024-04-26T00:41:15.377 INFO:tasks.rados.rados.0.smithi045.stdout:3738: done (1 left) 2024-04-26T00:41:15.378 INFO:tasks.rados.rados.0.smithi045.stdout:3739: done (0 left) 2024-04-26T00:41:15.378 INFO:tasks.rados.rados.0.smithi045.stdout:3740: read oid 18 snap -1 2024-04-26T00:41:15.378 INFO:tasks.rados.rados.0.smithi045.stdout:3740: expect (ObjNum 1096 snap 325 seq_num 1096) 2024-04-26T00:41:15.378 INFO:tasks.rados.rados.0.smithi045.stdout:3741: rollback oid 39 current snap is 413 2024-04-26T00:41:15.378 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 39 to 409 2024-04-26T00:41:15.378 INFO:tasks.rados.rados.0.smithi045.stdout:3742: delete oid 50 current snap is 413 2024-04-26T00:41:15.385 INFO:tasks.rados.rados.0.smithi045.stdout:3742: done (2 left) 2024-04-26T00:41:15.385 INFO:tasks.rados.rados.0.smithi045.stdout:3743: copy_from oid 3 from oid 17 current snap is 413 2024-04-26T00:41:15.385 INFO:tasks.rados.rados.0.smithi045.stdout:3744: read oid 41 snap 407 2024-04-26T00:41:15.385 INFO:tasks.rados.rados.0.smithi045.stdout:3744: expect (ObjNum 1321 snap 407 seq_num 1321) 2024-04-26T00:41:15.386 INFO:tasks.rados.rados.0.smithi045.stdout:3745: rollback oid 28 current snap is 413 2024-04-26T00:41:15.386 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 28 to 403 2024-04-26T00:41:15.386 INFO:tasks.rados.rados.0.smithi045.stdout:3746: copy_from oid 9 from oid 8 current snap is 413 2024-04-26T00:41:15.386 INFO:tasks.rados.rados.0.smithi045.stdout:3747: delete oid 45 current snap is 413 2024-04-26T00:41:15.387 INFO:tasks.rados.rados.0.smithi045.stdout:3743: finishing copy_from to smithi04526579-3 2024-04-26T00:41:15.387 INFO:tasks.rados.rados.0.smithi045.stdout:3743: got expected ENOENT (src dne) 2024-04-26T00:41:15.388 INFO:tasks.rados.rados.0.smithi045.stdout:3743: finishing copy_from racing read to smithi04526579-3 2024-04-26T00:41:15.388 INFO:tasks.rados.rados.0.smithi045.stdout:3741: finishing rollback tid 1 to smithi04526579-39 2024-04-26T00:41:15.388 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 39 v 2410 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-04-26T00:41:15.394 INFO:tasks.rados.rados.0.smithi045.stdout:3744: expect (ObjNum 1321 snap 407 seq_num 1321) 2024-04-26T00:41:15.595 INFO:tasks.rados.rados.0.smithi045.stdout:3740: expect (ObjNum 1096 snap 325 seq_num 1096) 2024-04-26T00:41:15.735 INFO:tasks.rados.rados.0.smithi045.stdout:3745: finishing rollback tid 0 to smithi04526579-28 2024-04-26T00:41:15.735 INFO:tasks.rados.rados.0.smithi045.stdout:3745: finishing rollback tid 1 to smithi04526579-28 2024-04-26T00:41:15.735 INFO:tasks.rados.rados.0.smithi045.stdout:3746: finishing copy_from racing read to smithi04526579-9 2024-04-26T00:41:15.735 INFO:tasks.rados.rados.0.smithi045.stdout:3745: finishing rollback tid 2 to smithi04526579-28 2024-04-26T00:41:15.735 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 2530 (ObjNum 1279 snap 398 seq_num 1279) dirty exists 2024-04-26T00:41:15.735 INFO:tasks.rados.rados.0.smithi045.stdout:3746: finishing copy_from to smithi04526579-9 2024-04-26T00:41:15.735 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 2237 (ObjNum 1319 snap 407 seq_num 1319) dirty exists 2024-04-26T00:41:15.735 INFO:tasks.rados.rados.0.smithi045.stdout:3740: done (6 left) 2024-04-26T00:41:15.736 INFO:tasks.rados.rados.0.smithi045.stdout:3741: done (5 left) 2024-04-26T00:41:15.736 INFO:tasks.rados.rados.0.smithi045.stdout:3743: done (4 left) 2024-04-26T00:41:15.736 INFO:tasks.rados.rados.0.smithi045.stdout:3744: done (3 left) 2024-04-26T00:41:15.736 INFO:tasks.rados.rados.0.smithi045.stdout:3745: done (2 left) 2024-04-26T00:41:15.736 INFO:tasks.rados.rados.0.smithi045.stdout:3746: done (1 left) 2024-04-26T00:41:15.736 INFO:tasks.rados.rados.0.smithi045.stdout:3747: done (0 left) 2024-04-26T00:41:15.736 INFO:tasks.rados.rados.0.smithi045.stdout:3748: snap_create 2024-04-26T00:41:16.255 INFO:tasks.rados.rados.0.smithi045.stdout:3748: done (0 left) 2024-04-26T00:41:16.255 INFO:tasks.rados.rados.0.smithi045.stdout:3749: snap_remove snap 408 2024-04-26T00:41:17.265 INFO:tasks.rados.rados.0.smithi045.stdout:3749: done (0 left) 2024-04-26T00:41:17.265 INFO:tasks.rados.rados.0.smithi045.stdout:3750: rollback oid 43 current snap is 414 2024-04-26T00:41:17.266 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 43 to 401 2024-04-26T00:41:17.266 INFO:tasks.rados.rados.0.smithi045.stdout:3751: rollback oid 24 current snap is 414 2024-04-26T00:41:17.266 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 24 to 393 2024-04-26T00:41:17.266 INFO:tasks.rados.rados.0.smithi045.stdout:3752: delete oid 40 current snap is 414 2024-04-26T00:41:17.278 INFO:tasks.rados.rados.0.smithi045.stdout:3750: finishing rollback tid 1 to smithi04526579-43 2024-04-26T00:41:17.278 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 43 v 2428 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-04-26T00:41:17.281 INFO:tasks.rados.rados.0.smithi045.stdout:3750: done (2 left) 2024-04-26T00:41:17.281 INFO:tasks.rados.rados.0.smithi045.stdout:3752: done (1 left) 2024-04-26T00:41:17.281 INFO:tasks.rados.rados.0.smithi045.stdout:3753: snap_create 2024-04-26T00:41:17.288 INFO:tasks.rados.rados.0.smithi045.stdout:3751: finishing rollback tid 0 to smithi04526579-24 2024-04-26T00:41:17.291 INFO:tasks.rados.rados.0.smithi045.stdout:3751: finishing rollback tid 1 to smithi04526579-24 2024-04-26T00:41:17.291 INFO:tasks.rados.rados.0.smithi045.stdout:3751: finishing rollback tid 2 to smithi04526579-24 2024-04-26T00:41:17.291 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 24 v 2829 (ObjNum 1211 snap 375 seq_num 1211) dirty exists 2024-04-26T00:41:17.626 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-04-26T00:41:17.627 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-04-26T00:41:17.627 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-26T00:41:17.627 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-26T00:41:17.630 INFO:tasks.ceph.mon.a:Started 2024-04-26T00:41:17.630 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:41:17.630 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:41:17.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.753+0000 7f859fe09640 10 monclient: get_monmap_and_config 2024-04-26T00:41:17.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.753+0000 7f859fe09640 10 monclient: build_initial_monmap 2024-04-26T00:41:17.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.753+0000 7f859fe09640 10 monclient: monmap: 2024-04-26T00:41:17.756 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:41:17.756 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:17.756 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:41:17.758197+0000 2024-04-26T00:41:17.756 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:41:17.758197+0000 2024-04-26T00:41:17.756 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:41:17.756 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:17.756 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:41:17.757 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:41:17.757 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:41:17.757 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:41:17.757 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:41:17.757 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:17.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.753+0000 7f859fe09640 1 Processor -- start 2024-04-26T00:41:17.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.753+0000 7f859fe09640 1 -- start start 2024-04-26T00:41:17.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.753+0000 7f859fe09640 10 monclient: init 2024-04-26T00:41:17.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.753+0000 7f859fe09640 10 monclient: _reopen_session rank -1 2024-04-26T00:41:17.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.753+0000 7f859fe09640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-04-26T00:41:17.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.753+0000 7f859fe09640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f85980a2c10 addr v1:172.21.15.37:6789/0 2024-04-26T00:41:17.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.753+0000 7f859fe09640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f859809ddd0 0x7f8598139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:17.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.753+0000 7f859fe09640 10 monclient(hunting): picked mon.noname-c con 0x7f859809ddd0 addr v2:172.21.15.45:3300/0 2024-04-26T00:41:17.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.753+0000 7f859fe09640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8598135800 addr v1:172.21.15.45:6789/0 2024-04-26T00:41:17.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.753+0000 7f859fe09640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85980a1670 con 0x7f85980a2c10 2024-04-26T00:41:17.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.753+0000 7f859fe09640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85980a3880 con 0x7f8598135800 2024-04-26T00:41:17.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.753+0000 7f859fe09640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:17.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.753+0000 7f859fe09640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f85980a3640 con 0x7f859809ddd0 2024-04-26T00:41:17.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.753+0000 7f859fe09640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:17.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.753+0000 7f859fe09640 10 monclient(hunting): authenticate will time out at 2259.063818s 2024-04-26T00:41:17.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.753+0000 7f859dba5640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f85980a2c10 0x7f85981350f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:51272/0 (socket says 172.21.15.37:51272) 2024-04-26T00:41:17.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.757+0000 7f859dba5640 1 -- 172.21.15.37:0/1147027480 learned_addr learned my addr 172.21.15.37:0/1147027480 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:41:17.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.757+0000 7f859d3a4640 1 --2- 172.21.15.37:0/1147027480 >> v2:172.21.15.45:3300/0 conn(0x7f859809ddd0 0x7f8598139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:17.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.757+0000 7f859d3a4640 10 monclient(hunting): get_auth_request con 0x7f859809ddd0 auth_method 0 2024-04-26T00:41:17.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.757+0000 7f859d3a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:17.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.757+0000 7f859d3a4640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:17.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.757+0000 7f859d3a4640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:17.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.757+0000 7f859d3a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:17.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.757+0000 7f859d3a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:17.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.757+0000 7f859d3a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:17.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.757+0000 7f859dba5640 1 -- 172.21.15.37:0/1147027480 >> v1:172.21.15.37:6789/0 conn(0x7f85980a2c10 legacy=0x7f85981350f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:17.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.757+0000 7f859e3a6640 1 -- 172.21.15.37:0/1147027480 >> v1:172.21.15.45:6789/0 conn(0x7f8598135800 legacy=0x7f859813c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:17.761 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.757+0000 7f859d3a4640 10 monclient(hunting): handle_auth_done global_id 224425 payload 274 2024-04-26T00:41:17.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.757+0000 7f859d3a4640 1 -- 172.21.15.37:0/1147027480 >> v1:172.21.15.45:6789/0 conn(0x7f8598135800 legacy=0x7f859813c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:17.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.757+0000 7f859d3a4640 1 -- 172.21.15.37:0/1147027480 >> v1:172.21.15.37:6789/0 conn(0x7f85980a2c10 legacy=0x7f85981350f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:17.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.757+0000 7f859d3a4640 10 monclient: _finish_hunting 0 2024-04-26T00:41:17.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.757+0000 7f859d3a4640 1 monclient: found mon.noname-c 2024-04-26T00:41:17.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.757+0000 7f859d3a4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:17.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.757+0000 7f859d3a4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:41:17.762 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.757+0000 7f859d3a4640 1 -- 172.21.15.37:0/1147027480 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85980a07b0 con 0x7f859809ddd0 2024-04-26T00:41:17.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.761+0000 7f859d3a4640 1 -- 172.21.15.37:0/1147027480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f859809ddd0 msgr2=0x7f8598139f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:17.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.761+0000 7f859d3a4640 1 --2- 172.21.15.37:0/1147027480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f859809ddd0 0x7f8598139f60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f85880098c0 tx=0x7f858801c460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24f85b8a46a2bd1c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:17.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.761+0000 7f859cba3640 1 -- 172.21.15.37:0/1147027480 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85880b0070 con 0x7f859809ddd0 2024-04-26T00:41:17.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.761+0000 7f859cba3640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:17.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.761+0000 7f859cba3640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:41:17.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.761+0000 7f859cba3640 10 monclient: dump: 2024-04-26T00:41:17.766 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:17.766 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:17.766 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:17.766 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:17.766 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:17.767 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:17.767 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:17.767 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:17.767 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:17.767 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:17.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.761+0000 7f859cba3640 10 monclient: _finish_auth 0 2024-04-26T00:41:17.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.761+0000 7f859cba3640 10 monclient: _check_auth_tickets 2024-04-26T00:41:17.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.761+0000 7f859cba3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:41:17.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.761+0000 7f859cba3640 1 -- 172.21.15.37:0/1147027480 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85880ab600 con 0x7f859809ddd0 2024-04-26T00:41:17.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.761+0000 7f859fe09640 5 monclient: authenticate success, global_id 224425 2024-04-26T00:41:17.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.761+0000 7f859fe09640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:41:17.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.761+0000 7f859cba3640 10 monclient: handle_config config(0 keys) 2024-04-26T00:41:17.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.761+0000 7f859fe09640 10 monclient: get_monmap_and_config success 2024-04-26T00:41:17.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859fe09640 4 set_mon_vals no callback set 2024-04-26T00:41:17.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859cba3640 1 -- 172.21.15.37:0/1147027480 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85880ab9d0 con 0x7f859809ddd0 2024-04-26T00:41:17.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859cba3640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:17.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859cba3640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:41:17.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859cba3640 10 monclient: dump: 2024-04-26T00:41:17.768 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:17.768 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:17.768 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:17.768 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:17.768 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:17.768 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:17.768 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:17.768 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:17.768 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:17.768 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:17.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859fe09640 10 monclient: shutdown 2024-04-26T00:41:17.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859fe09640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:41:17.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859fe09640 1 -- 172.21.15.37:0/1147027480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f859809ddd0 msgr2=0x7f8598139f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:17.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859fe09640 1 --2- 172.21.15.37:0/1147027480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f859809ddd0 0x7f8598139f60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f85880098c0 tx=0x7f858801c460 comp rx=0 tx=0).stop 2024-04-26T00:41:17.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859fe09640 1 -- 172.21.15.37:0/1147027480 shutdown_connections 2024-04-26T00:41:17.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859fe09640 1 --2- 172.21.15.37:0/1147027480 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f859809ddd0 0x7f8598139f60 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:17.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859fe09640 1 -- 172.21.15.37:0/1147027480 >> 172.21.15.37:0/1147027480 conn(0x7f85980a7c40 msgr2=0x7f85980a3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:41:17.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859fe09640 1 -- 172.21.15.37:0/1147027480 shutdown_connections 2024-04-26T00:41:17.769 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859fe09640 1 -- 172.21.15.37:0/1147027480 wait complete. 2024-04-26T00:41:17.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859fe09640 10 monclient: build_initial_monmap 2024-04-26T00:41:17.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859fe09640 10 monclient: monmap: 2024-04-26T00:41:17.774 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:41:17.774 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:41:17.774 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:41:17.774 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:41:17.774 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:41:17.774 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:41:17.774 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:41:17.774 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:41:17.774 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:41:17.774 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:17.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859fe09640 1 Processor -- start 2024-04-26T00:41:17.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859fe09640 1 -- start start 2024-04-26T00:41:17.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859fe09640 10 monclient: init 2024-04-26T00:41:17.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859fe09640 10 monclient: _reopen_session rank -1 2024-04-26T00:41:17.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859fe09640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:41:17.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859fe09640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f859809ddd0 0x7f85981cef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:17.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859fe09640 10 monclient(hunting): picked mon.noname-a con 0x7f859809ddd0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:41:17.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859fe09640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85980a2c10 0x7f85981cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:17.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859fe09640 10 monclient(hunting): picked mon.noname-c con 0x7f85980a2c10 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:41:17.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859fe09640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8598135800 0x7f85981cfa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:17.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859fe09640 10 monclient(hunting): picked mon.noname-b con 0x7f8598135800 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:41:17.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859fe09640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:17.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859fe09640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f8598140b90 con 0x7f859809ddd0 2024-04-26T00:41:17.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859fe09640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:17.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859fe09640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8598140a10 con 0x7f8598135800 2024-04-26T00:41:17.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859fe09640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:17.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859fe09640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8598140d10 con 0x7f85980a2c10 2024-04-26T00:41:17.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859fe09640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:17.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859fe09640 10 monclient(hunting): authenticate will time out at 2259.074171s 2024-04-26T00:41:17.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859dba5640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f859809ddd0 0x7f85981cef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:17.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859d3a4640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85980a2c10 0x7f85981cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:17.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859d3a4640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85980a2c10 0x7f85981cf4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:55872/0 (socket says 172.21.15.37:55872) 2024-04-26T00:41:17.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859d3a4640 1 -- 172.21.15.37:0/2272735848 learned_addr learned my addr 172.21.15.37:0/2272735848 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:41:17.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859dba5640 10 monclient(hunting): get_auth_request con 0x7f859809ddd0 auth_method 0 2024-04-26T00:41:17.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859dba5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:17.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859dba5640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:17.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859dba5640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:17.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859d3a4640 10 monclient(hunting): get_auth_request con 0x7f85980a2c10 auth_method 0 2024-04-26T00:41:17.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859d3a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:17.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859d3a4640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:17.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859d3a4640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:17.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859dba5640 1 -- 172.21.15.37:0/2272735848 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f859809ddd0 msgr2=0x7f85981cef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:41:17.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859dba5640 1 -- 172.21.15.37:0/2272735848 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f859809ddd0 msgr2=0x7f85981cef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:41:17.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859d3a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:17.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859d3a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:17.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859dba5640 1 --2- 172.21.15.37:0/2272735848 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f859809ddd0 0x7f85981cef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:41:17.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859d3a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:17.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859dba5640 1 --2- 172.21.15.37:0/2272735848 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f859809ddd0 0x7f85981cef90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:41:17.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859e3a6640 1 --2- 172.21.15.37:0/2272735848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8598135800 0x7f85981cfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:17.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859e3a6640 10 monclient(hunting): get_auth_request con 0x7f8598135800 auth_method 0 2024-04-26T00:41:17.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859e3a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:17.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859e3a6640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:17.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859e3a6640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:17.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859d3a4640 10 monclient(hunting): handle_auth_done global_id 254186 payload 824 2024-04-26T00:41:17.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859d3a4640 1 -- 172.21.15.37:0/2272735848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8598135800 msgr2=0x7f85981cfa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:17.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859d3a4640 1 --2- 172.21.15.37:0/2272735848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8598135800 0x7f85981cfa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:17.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859d3a4640 1 -- 172.21.15.37:0/2272735848 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f859809ddd0 msgr2=0x7f85981cef90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:41:17.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859d3a4640 1 --2- 172.21.15.37:0/2272735848 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f859809ddd0 0x7f85981cef90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:17.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859d3a4640 10 monclient: _finish_hunting 0 2024-04-26T00:41:17.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859d3a4640 1 monclient: found mon.noname-c 2024-04-26T00:41:17.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859d3a4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:17.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859d3a4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:41:17.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859d3a4640 1 -- 172.21.15.37:0/2272735848 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f859811e460 con 0x7f85980a2c10 2024-04-26T00:41:17.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.765+0000 7f859e3a6640 1 --2- 172.21.15.37:0/2272735848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8598135800 0x7f85981cfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:41:17.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.769+0000 7f859d3a4640 1 -- 172.21.15.37:0/2272735848 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85980a2c10 msgr2=0x7f85981cf4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:17.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.769+0000 7f859d3a4640 1 --2- 172.21.15.37:0/2272735848 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85980a2c10 0x7f85981cf4d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f85880ae040 tx=0x7f8588019200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:17.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.769+0000 7f858e7fc640 1 -- 172.21.15.37:0/2272735848 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85880b0040 con 0x7f85980a2c10 2024-04-26T00:41:17.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.769+0000 7f858e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:17.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.769+0000 7f858e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:41:17.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.769+0000 7f858e7fc640 10 monclient: dump: 2024-04-26T00:41:17.779 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:17.779 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:17.779 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:17.779 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:17.779 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:17.779 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:17.779 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:17.779 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:17.779 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:17.779 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:17.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.769+0000 7f858e7fc640 10 monclient: _finish_auth 0 2024-04-26T00:41:17.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.769+0000 7f858e7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:41:17.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.769+0000 7f858e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:41:17.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.769+0000 7f858e7fc640 1 -- 172.21.15.37:0/2272735848 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85880b9070 con 0x7f85980a2c10 2024-04-26T00:41:17.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.769+0000 7f859fe09640 5 monclient: authenticate success, global_id 254186 2024-04-26T00:41:17.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.769+0000 7f859fe09640 10 monclient: _renew_subs 2024-04-26T00:41:17.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.769+0000 7f859fe09640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:41:17.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.769+0000 7f859fe09640 1 -- 172.21.15.37:0/2272735848 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f859813c930 con 0x7f85980a2c10 2024-04-26T00:41:17.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.769+0000 7f858e7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:41:17.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.769+0000 7f858e7fc640 1 -- 172.21.15.37:0/2272735848 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85880aac40 con 0x7f85980a2c10 2024-04-26T00:41:17.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.769+0000 7f858e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:17.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.769+0000 7f858e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:41:17.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.769+0000 7f858e7fc640 10 monclient: dump: 2024-04-26T00:41:17.780 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:17.780 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:17.780 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:17.781 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:17.781 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:17.781 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:17.781 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:17.781 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:17.781 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:17.781 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:17.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.769+0000 7f859fe09640 10 monclient: _renew_subs 2024-04-26T00:41:17.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.769+0000 7f859fe09640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:41:17.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.769+0000 7f859fe09640 1 -- 172.21.15.37:0/2272735848 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f859811d850 con 0x7f85980a2c10 2024-04-26T00:41:17.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.769+0000 7f858ffff640 4 set_mon_vals no callback set 2024-04-26T00:41:17.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.769+0000 7f859fe09640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:17.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.769+0000 7f859fe09640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:17.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.769+0000 7f859fe09640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:41:17.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.769+0000 7f859d3a4640 1 --2- 172.21.15.37:0/2272735848 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85980a2c10 0x7f85981cf4d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f85880ae040 tx=0x7f8588019200 comp rx=0 tx=0).queue_received will delay after 1.71409e+09 on 0x7f85880be460 mgrmap(e 5) 2024-04-26T00:41:17.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.769+0000 7f859fe09640 1 -- 172.21.15.37:0/2272735848 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8598066f40 con 0x7f85980a2c10 2024-04-26T00:41:17.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.769+0000 7f858e7fc640 1 -- 172.21.15.37:0/2272735848 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f85880be460 con 0x7f85980a2c10 2024-04-26T00:41:17.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.769+0000 7f858e7fc640 1 --2- 172.21.15.37:0/2272735848 >> v2:172.21.15.45:6812/2379918623 conn(0x7f857c035e80 0x7f857c038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:17.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.769+0000 7f859dba5640 1 --2- 172.21.15.37:0/2272735848 >> v2:172.21.15.45:6812/2379918623 conn(0x7f857c035e80 0x7f857c038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:17.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.769+0000 7f859dba5640 10 monclient: get_auth_request con 0x7f857c035e80 auth_method 0 2024-04-26T00:41:17.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.773+0000 7f859dba5640 1 --2- 172.21.15.37:0/2272735848 >> v2:172.21.15.45:6812/2379918623 conn(0x7f857c035e80 0x7f857c038340 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f8594005890 tx=0x7f8594009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:17.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:17.773+0000 7f858e7fc640 1 -- 172.21.15.37:0/2272735848 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(844..844 src has 705..844) ==== 3975+0+0 (secure 0 0 0) 0x7f85880e2c00 con 0x7f85980a2c10 2024-04-26T00:41:18.273 INFO:tasks.rados.rados.0.smithi045.stdout:3751: done (1 left) 2024-04-26T00:41:18.273 INFO:tasks.rados.rados.0.smithi045.stdout:3753: done (0 left) 2024-04-26T00:41:18.274 INFO:tasks.rados.rados.0.smithi045.stdout:3754: read oid 22 snap 412 2024-04-26T00:41:18.274 INFO:tasks.rados.rados.0.smithi045.stdout:3754: expect (ObjNum 1291 snap 401 seq_num 1291) 2024-04-26T00:41:18.274 INFO:tasks.rados.rados.0.smithi045.stdout:3755: delete oid 27 current snap is 415 2024-04-26T00:41:18.284 INFO:tasks.rados.rados.0.smithi045.stdout:3755: done (1 left) 2024-04-26T00:41:18.284 INFO:tasks.rados.rados.0.smithi045.stdout:3756: write (excl) oid 29 current snap is 415 2024-04-26T00:41:18.284 INFO:tasks.rados.rados.0.smithi045.stdout:3756: seq_num 1340 ranges {521380=511788,1480224=742462} 2024-04-26T00:41:18.301 INFO:tasks.rados.rados.0.smithi045.stdout:3756: writing smithi04526579-29 from 521380 to 1033168 tid 1 2024-04-26T00:41:18.322 INFO:tasks.rados.rados.0.smithi045.stdout:3756: writing smithi04526579-29 from 1480224 to 2222686 tid 2 2024-04-26T00:41:18.322 INFO:tasks.rados.rados.0.smithi045.stdout:3757: read oid 12 snap -1 2024-04-26T00:41:18.322 INFO:tasks.rados.rados.0.smithi045.stdout:3757: expect (ObjNum 1290 snap 401 seq_num 1290) 2024-04-26T00:41:18.322 INFO:tasks.rados.rados.0.smithi045.stdout:3758: write (excl) oid 13 current snap is 415 2024-04-26T00:41:18.322 INFO:tasks.rados.rados.0.smithi045.stdout:3758: seq_num 1341 ranges {538187=587037,1538706=602473} 2024-04-26T00:41:18.341 INFO:tasks.rados.rados.0.smithi045.stdout:3758: writing smithi04526579-13 from 538187 to 1125224 tid 1 2024-04-26T00:41:18.354 INFO:tasks.rados.rados.0.smithi045.stdout:3758: writing smithi04526579-13 from 1538706 to 2141179 tid 2 2024-04-26T00:41:18.354 INFO:tasks.rados.rados.0.smithi045.stdout:3759: write oid 1 current snap is 415 2024-04-26T00:41:18.354 INFO:tasks.rados.rados.0.smithi045.stdout:3759: seq_num 1342 ranges {554994=662286,1997188=557806} 2024-04-26T00:41:18.366 INFO:tasks.rados.rados.0.smithi045.stdout:3759: writing smithi04526579-1 from 554994 to 1217280 tid 1 2024-04-26T00:41:18.378 INFO:tasks.rados.rados.0.smithi045.stdout:3759: writing smithi04526579-1 from 1997188 to 2554994 tid 2 2024-04-26T00:41:18.378 INFO:tasks.rados.rados.0.smithi045.stdout:3760: snap_create 2024-04-26T00:41:18.378 INFO:tasks.rados.rados.0.smithi045.stdout:3754: expect (ObjNum 1291 snap 401 seq_num 1291) 2024-04-26T00:41:18.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:18.597+0000 7f858e7fc640 1 -- 172.21.15.37:0/2272735848 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f8588016170 con 0x7f85980a2c10 2024-04-26T00:41:18.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:18.597+0000 7f858e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:18.600 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:18.597+0000 7f858e7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:41:18.603 INFO:tasks.rados.rados.0.smithi045.stdout:3756: finishing write tid 1 to smithi04526579-29 2024-04-26T00:41:18.603 INFO:tasks.rados.rados.0.smithi045.stdout:3756: finishing write tid 2 to smithi04526579-29 2024-04-26T00:41:18.603 INFO:tasks.rados.rados.0.smithi045.stdout:3756: finishing write tid 3 to smithi04526579-29 2024-04-26T00:41:18.603 INFO:tasks.rados.rados.0.smithi045.stdout:3756: finishing write tid 4 to smithi04526579-29 2024-04-26T00:41:18.603 INFO:tasks.rados.rados.0.smithi045.stdout:3756: oid 29 updating version 0 to 2240 2024-04-26T00:41:18.603 INFO:tasks.rados.rados.0.smithi045.stdout:3756: oid 29 version 2240 is already newer than 2239 2024-04-26T00:41:18.603 INFO:tasks.rados.rados.0.smithi045.stdout:3756: oid 29 version 2240 is already newer than 2238 2024-04-26T00:41:18.603 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 2240 (ObjNum 1340 snap 415 seq_num 1340) dirty exists 2024-04-26T00:41:18.603 INFO:tasks.rados.rados.0.smithi045.stdout:3756: left oid 29 (ObjNum 1340 snap 415 seq_num 1340) 2024-04-26T00:41:18.604 INFO:tasks.rados.rados.0.smithi045.stdout:3757: expect (ObjNum 1290 snap 401 seq_num 1290) 2024-04-26T00:41:18.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:18.829+0000 7f859fe09640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:41:18.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:18.829+0000 7f859fe09640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:41:18.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:18.829+0000 7f859fe09640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:41:18.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:18.829+0000 7f859fe09640 1 -- 172.21.15.37:0/2272735848 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f859811df30 con 0x7f85980a2c10 2024-04-26T00:41:18.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:18.829+0000 7f858e7fc640 1 -- 172.21.15.37:0/2272735848 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7f85880bce00 con 0x7f85980a2c10 2024-04-26T00:41:18.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:18.829+0000 7f858e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:41:18.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:18.829+0000 7f858e7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:41:18.832 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":154,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":22,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:41:18.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:18.833+0000 7f859fe09640 1 -- 172.21.15.37:0/2272735848 >> v2:172.21.15.45:6812/2379918623 conn(0x7f857c035e80 msgr2=0x7f857c038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:18.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:18.833+0000 7f859fe09640 1 --2- 172.21.15.37:0/2272735848 >> v2:172.21.15.45:6812/2379918623 conn(0x7f857c035e80 0x7f857c038340 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f8594005890 tx=0x7f8594009040 comp rx=0 tx=0).stop 2024-04-26T00:41:18.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:18.833+0000 7f859fe09640 10 monclient: shutdown 2024-04-26T00:41:18.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:18.833+0000 7f859fe09640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:41:18.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:18.833+0000 7f859fe09640 1 -- 172.21.15.37:0/2272735848 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85980a2c10 msgr2=0x7f85981cf4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:18.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:18.833+0000 7f859fe09640 1 --2- 172.21.15.37:0/2272735848 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85980a2c10 0x7f85981cf4d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f85880ae040 tx=0x7f8588019200 comp rx=0 tx=0).stop 2024-04-26T00:41:18.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:18.833+0000 7f859fe09640 1 -- 172.21.15.37:0/2272735848 shutdown_connections 2024-04-26T00:41:18.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:18.833+0000 7f859fe09640 1 --2- 172.21.15.37:0/2272735848 >> v2:172.21.15.45:6812/2379918623 conn(0x7f857c035e80 0x7f857c038340 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:18.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:18.833+0000 7f859fe09640 1 --2- 172.21.15.37:0/2272735848 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8598135800 0x7f85981cfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:18.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:18.833+0000 7f859fe09640 1 --2- 172.21.15.37:0/2272735848 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f85980a2c10 0x7f85981cf4d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:18.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:18.833+0000 7f859fe09640 1 --2- 172.21.15.37:0/2272735848 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f859809ddd0 0x7f85981cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:18.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:18.833+0000 7f859fe09640 1 -- 172.21.15.37:0/2272735848 >> 172.21.15.37:0/2272735848 conn(0x7f85980a7c40 msgr2=0x7f85981344d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:41:18.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:18.833+0000 7f859fe09640 1 -- 172.21.15.37:0/2272735848 shutdown_connections 2024-04-26T00:41:18.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:18.833+0000 7f859fe09640 1 -- 172.21.15.37:0/2272735848 wait complete. 2024-04-26T00:41:18.892 INFO:tasks.rados.rados.0.smithi045.stdout:3758: finishing write tid 1 to smithi04526579-13 2024-04-26T00:41:18.892 INFO:tasks.rados.rados.0.smithi045.stdout:3758: finishing write tid 2 to smithi04526579-13 2024-04-26T00:41:18.892 INFO:tasks.rados.rados.0.smithi045.stdout:3758: finishing write tid 3 to smithi04526579-13 2024-04-26T00:41:18.892 INFO:tasks.rados.rados.0.smithi045.stdout:3758: finishing write tid 4 to smithi04526579-13 2024-04-26T00:41:18.892 INFO:tasks.rados.rados.0.smithi045.stdout:3758: oid 13 updating version 0 to 2243 2024-04-26T00:41:18.892 INFO:tasks.rados.rados.0.smithi045.stdout:3758: oid 13 version 2243 is already newer than 2242 2024-04-26T00:41:18.892 INFO:tasks.rados.rados.0.smithi045.stdout:3758: oid 13 version 2243 is already newer than 2241 2024-04-26T00:41:18.892 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 2243 (ObjNum 1341 snap 415 seq_num 1341) dirty exists 2024-04-26T00:41:18.892 INFO:tasks.rados.rados.0.smithi045.stdout:3758: left oid 13 (ObjNum 1341 snap 415 seq_num 1341) 2024-04-26T00:41:18.892 INFO:tasks.rados.rados.0.smithi045.stdout:3759: finishing write tid 1 to smithi04526579-1 2024-04-26T00:41:18.892 INFO:tasks.rados.rados.0.smithi045.stdout:3759: finishing write tid 2 to smithi04526579-1 2024-04-26T00:41:18.892 INFO:tasks.rados.rados.0.smithi045.stdout:3759: finishing write tid 3 to smithi04526579-1 2024-04-26T00:41:18.892 INFO:tasks.rados.rados.0.smithi045.stdout:3759: finishing write tid 4 to smithi04526579-1 2024-04-26T00:41:18.893 INFO:tasks.rados.rados.0.smithi045.stdout:3759: oid 1 updating version 0 to 2532 2024-04-26T00:41:18.893 INFO:tasks.rados.rados.0.smithi045.stdout:3759: oid 1 version 2532 is already newer than 2531 2024-04-26T00:41:18.893 INFO:tasks.rados.rados.0.smithi045.stdout:3759: oid 1 updating version 2532 to 2533 2024-04-26T00:41:18.893 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 2533 (ObjNum 1342 snap 415 seq_num 1342) dirty exists 2024-04-26T00:41:18.893 INFO:tasks.rados.rados.0.smithi045.stdout:3759: left oid 1 (ObjNum 1342 snap 415 seq_num 1342) 2024-04-26T00:41:19.277 INFO:tasks.rados.rados.0.smithi045.stdout:3754: done (5 left) 2024-04-26T00:41:19.277 INFO:tasks.rados.rados.0.smithi045.stdout:3756: done (4 left) 2024-04-26T00:41:19.277 INFO:tasks.rados.rados.0.smithi045.stdout:3757: done (3 left) 2024-04-26T00:41:19.277 INFO:tasks.rados.rados.0.smithi045.stdout:3758: done (2 left) 2024-04-26T00:41:19.277 INFO:tasks.rados.rados.0.smithi045.stdout:3759: done (1 left) 2024-04-26T00:41:19.278 INFO:tasks.rados.rados.0.smithi045.stdout:3760: done (0 left) 2024-04-26T00:41:19.278 INFO:tasks.rados.rados.0.smithi045.stdout:3761: delete oid 27 current snap is 416 2024-04-26T00:41:19.284 INFO:tasks.rados.rados.0.smithi045.stdout:3761: done (0 left) 2024-04-26T00:41:19.284 INFO:tasks.rados.rados.0.smithi045.stdout:3762: snap_remove snap 378 2024-04-26T00:41:20.308 INFO:tasks.rados.rados.0.smithi045.stdout:3762: done (0 left) 2024-04-26T00:41:20.308 INFO:tasks.rados.rados.0.smithi045.stdout:3763: delete oid 11 current snap is 416 2024-04-26T00:41:20.318 INFO:tasks.rados.rados.0.smithi045.stdout:3763: done (0 left) 2024-04-26T00:41:20.319 INFO:tasks.rados.rados.0.smithi045.stdout:3764: write (excl) oid 35 current snap is 416 2024-04-26T00:41:20.319 INFO:tasks.rados.rados.0.smithi045.stdout:3764: seq_num 1345 ranges {605415=488033,1656281=493458} 2024-04-26T00:41:20.339 INFO:tasks.rados.rados.0.smithi045.stdout:3764: writing smithi04526579-35 from 605415 to 1093448 tid 1 2024-04-26T00:41:20.359 INFO:tasks.rados.rados.0.smithi045.stdout:3764: writing smithi04526579-35 from 1656281 to 2149739 tid 2 2024-04-26T00:41:20.359 INFO:tasks.rados.rados.0.smithi045.stdout:3765: write (excl) oid 22 current snap is 416 2024-04-26T00:41:20.359 INFO:tasks.rados.rados.0.smithi045.stdout:3764: finishing write tid 1 to smithi04526579-35 2024-04-26T00:41:20.359 INFO:tasks.rados.rados.0.smithi045.stdout:3765: seq_num 1346 ranges {622222=679635,1831116=753469} 2024-04-26T00:41:20.375 INFO:tasks.rados.rados.0.smithi045.stdout:3765: writing smithi04526579-22 from 622222 to 1301857 tid 1 2024-04-26T00:41:20.393 INFO:tasks.rados.rados.0.smithi045.stdout:3765: writing smithi04526579-22 from 1831116 to 2584585 tid 2 2024-04-26T00:41:20.393 INFO:tasks.rados.rados.0.smithi045.stdout:3766: read oid 47 snap -1 2024-04-26T00:41:20.393 INFO:tasks.rados.rados.0.smithi045.stdout:3766: expect (ObjNum 1127 snap 340 seq_num 1127) 2024-04-26T00:41:20.393 INFO:tasks.rados.rados.0.smithi045.stdout:3767: write (excl) oid 2 current snap is 416 2024-04-26T00:41:20.393 INFO:tasks.rados.rados.0.smithi045.stdout:3767: seq_num 1347 ranges {639029=754884,2173245=465784} 2024-04-26T00:41:20.411 INFO:tasks.rados.rados.0.smithi045.stdout:3767: writing smithi04526579-2 from 639029 to 1393913 tid 1 2024-04-26T00:41:20.422 INFO:tasks.rados.rados.0.smithi045.stdout:3767: writing smithi04526579-2 from 2173245 to 2639029 tid 2 2024-04-26T00:41:20.422 INFO:tasks.rados.rados.0.smithi045.stdout:3768: snap_remove snap 411 2024-04-26T00:41:21.852 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:41:21.966 INFO:tasks.rados.rados.0.smithi045.stdout:3768: done (4 left) 2024-04-26T00:41:21.966 INFO:tasks.rados.rados.0.smithi045.stdout:3769: snap_create 2024-04-26T00:41:21.966 INFO:tasks.rados.rados.0.smithi045.stdout:3764: finishing write tid 2 to smithi04526579-35 2024-04-26T00:41:21.966 INFO:tasks.rados.rados.0.smithi045.stdout:3764: finishing write tid 3 to smithi04526579-35 2024-04-26T00:41:21.966 INFO:tasks.rados.rados.0.smithi045.stdout:3764: finishing write tid 4 to smithi04526579-35 2024-04-26T00:41:21.966 INFO:tasks.rados.rados.0.smithi045.stdout:3764: oid 35 updating version 0 to 2525 2024-04-26T00:41:21.966 INFO:tasks.rados.rados.0.smithi045.stdout:3764: oid 35 updating version 2525 to 2526 2024-04-26T00:41:21.966 INFO:tasks.rados.rados.0.smithi045.stdout:3764: oid 35 updating version 2526 to 2527 2024-04-26T00:41:21.966 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 2527 (ObjNum 1345 snap 416 seq_num 1345) dirty exists 2024-04-26T00:41:21.966 INFO:tasks.rados.rados.0.smithi045.stdout:3764: left oid 35 (ObjNum 1345 snap 416 seq_num 1345) 2024-04-26T00:41:21.967 INFO:tasks.rados.rados.0.smithi045.stdout:3765: finishing write tid 1 to smithi04526579-22 2024-04-26T00:41:21.967 INFO:tasks.rados.rados.0.smithi045.stdout:3765: finishing write tid 2 to smithi04526579-22 2024-04-26T00:41:21.967 INFO:tasks.rados.rados.0.smithi045.stdout:3765: finishing write tid 3 to smithi04526579-22 2024-04-26T00:41:21.967 INFO:tasks.rados.rados.0.smithi045.stdout:3765: finishing write tid 4 to smithi04526579-22 2024-04-26T00:41:21.967 INFO:tasks.rados.rados.0.smithi045.stdout:3765: oid 22 updating version 0 to 2463 2024-04-26T00:41:21.967 INFO:tasks.rados.rados.0.smithi045.stdout:3765: oid 22 updating version 2463 to 2464 2024-04-26T00:41:21.967 INFO:tasks.rados.rados.0.smithi045.stdout:3765: oid 22 updating version 2464 to 2465 2024-04-26T00:41:21.967 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 2465 (ObjNum 1346 snap 416 seq_num 1346) dirty exists 2024-04-26T00:41:21.967 INFO:tasks.rados.rados.0.smithi045.stdout:3765: left oid 22 (ObjNum 1346 snap 416 seq_num 1346) 2024-04-26T00:41:21.967 INFO:tasks.rados.rados.0.smithi045.stdout:3766: expect (ObjNum 1127 snap 340 seq_num 1127) 2024-04-26T00:41:22.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:21.997+0000 7f65e603b640 10 monclient: get_monmap_and_config 2024-04-26T00:41:22.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:21.997+0000 7f65e603b640 10 monclient: build_initial_monmap 2024-04-26T00:41:22.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:21.997+0000 7f65e603b640 10 monclient: monmap: 2024-04-26T00:41:22.000 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:41:22.000 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:22.001 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:41:22.002530+0000 2024-04-26T00:41:22.001 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:41:22.002530+0000 2024-04-26T00:41:22.001 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:41:22.001 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:22.001 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:41:22.001 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:41:22.001 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:41:22.001 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:41:22.001 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:41:22.001 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:22.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:21.997+0000 7f65e603b640 1 Processor -- start 2024-04-26T00:41:22.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:21.997+0000 7f65e603b640 1 -- start start 2024-04-26T00:41:22.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:21.997+0000 7f65e603b640 10 monclient: init 2024-04-26T00:41:22.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65e603b640 10 monclient: _reopen_session rank -1 2024-04-26T00:41:22.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65e603b640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-04-26T00:41:22.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65e603b640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f65e0136da0 0x7f65e009bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:22.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65e603b640 10 monclient(hunting): picked mon.noname-a con 0x7f65e0136da0 addr v2:172.21.15.37:3300/0 2024-04-26T00:41:22.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65e603b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f65e009c410 addr v1:172.21.15.37:6789/0 2024-04-26T00:41:22.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65e603b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f65e0137890 addr v1:172.21.15.45:6789/0 2024-04-26T00:41:22.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65e603b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65e009b4f0 con 0x7f65e009c410 2024-04-26T00:41:22.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65e603b640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65e009b7d0 con 0x7f65e0137890 2024-04-26T00:41:22.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65e603b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:22.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65e603b640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f65e009bb40 con 0x7f65e0136da0 2024-04-26T00:41:22.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65e603b640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:22.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65e603b640 10 monclient(hunting): authenticate will time out at 2263.308259s 2024-04-26T00:41:22.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65df7fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f65e0136da0 0x7f65e009bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:22.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65deffd640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f65e009c410 0x7f65e0137180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:40142/0 (socket says 172.21.15.37:40142) 2024-04-26T00:41:22.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65deffd640 1 -- 172.21.15.37:0/3974758123 learned_addr learned my addr 172.21.15.37:0/3974758123 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:41:22.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65df7fe640 10 monclient(hunting): get_auth_request con 0x7f65e0136da0 auth_method 0 2024-04-26T00:41:22.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:22.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65df7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:22.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:22.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65df7fe640 1 -- 172.21.15.37:0/3974758123 >> v2:172.21.15.37:3300/0 conn(0x7f65e0136da0 msgr2=0x7f65e009bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:41:22.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65df7fe640 1 -- 172.21.15.37:0/3974758123 >> v2:172.21.15.37:3300/0 conn(0x7f65e0136da0 msgr2=0x7f65e009bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:41:22.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65df7fe640 1 --2- 172.21.15.37:0/3974758123 >> v2:172.21.15.37:3300/0 conn(0x7f65e0136da0 0x7f65e009bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:41:22.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65df7fe640 1 --2- 172.21.15.37:0/3974758123 >> v2:172.21.15.37:3300/0 conn(0x7f65e0136da0 0x7f65e009bed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:41:22.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65deffd640 1 -- 172.21.15.37:0/3974758123 >> v1:172.21.15.37:6789/0 conn(0x7f65e009c410 legacy=0x7f65e0137180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:22.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65dffff640 1 -- 172.21.15.37:0/3974758123 >> v1:172.21.15.45:6789/0 conn(0x7f65e0137890 legacy=0x7f65e013e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:22.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65de7fc640 1 -- 172.21.15.37:0/3974758123 <== mon.1 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2841966300 0 0) 0x7f65e009b7d0 con 0x7f65e0137890 2024-04-26T00:41:22.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65de7fc640 10 monclient(hunting): _negotiate 2024-04-26T00:41:22.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65de7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:22.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:22.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65de7fc640 10 monclient(hunting): my global_id is 224431 2024-04-26T00:41:22.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65de7fc640 1 -- 172.21.15.37:0/3974758123 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65cc003530 con 0x7f65e0137890 2024-04-26T00:41:22.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65de7fc640 1 -- 172.21.15.37:0/3974758123 <== mon.1 v1:172.21.15.45:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3813300741 0 0) 0x7f65cc003530 con 0x7f65e0137890 2024-04-26T00:41:22.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65de7fc640 1 -- 172.21.15.37:0/3974758123 >> v2:172.21.15.37:3300/0 conn(0x7f65e0136da0 msgr2=0x7f65e009bed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:41:22.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65de7fc640 1 --2- 172.21.15.37:0/3974758123 >> v2:172.21.15.37:3300/0 conn(0x7f65e0136da0 0x7f65e009bed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:22.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65de7fc640 1 -- 172.21.15.37:0/3974758123 >> v1:172.21.15.37:6789/0 conn(0x7f65e009c410 legacy=0x7f65e0137180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:22.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65de7fc640 10 monclient: _finish_hunting 0 2024-04-26T00:41:22.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65de7fc640 1 monclient: found mon.noname-c-legacy 2024-04-26T00:41:22.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65de7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:22.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65de7fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.45:6789/0 2024-04-26T00:41:22.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65de7fc640 1 -- 172.21.15.37:0/3974758123 --> v1:172.21.15.45:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65e006dbb0 con 0x7f65e0137890 2024-04-26T00:41:22.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65de7fc640 10 monclient: _finish_auth 0 2024-04-26T00:41:22.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65de7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:41:22.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:41:22.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65de7fc640 1 -- 172.21.15.37:0/3974758123 <== mon.1 v1:172.21.15.45:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 866326386 0 0) 0x7f65d8001dc0 con 0x7f65e0137890 2024-04-26T00:41:22.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65de7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:22.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65de7fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-04-26T00:41:22.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65de7fc640 10 monclient: dump: 2024-04-26T00:41:22.009 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:22.009 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:22.009 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:22.009 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:22.010 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:22.010 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:22.010 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:22.010 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:22.010 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:22.010 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:22.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65de7fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] but i'm connected to v1:172.21.15.45:6789/0, reconnecting 2024-04-26T00:41:22.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65de7fc640 10 monclient: _reopen_session rank -1 2024-04-26T00:41:22.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65de7fc640 1 -- 172.21.15.37:0/3974758123 >> v1:172.21.15.45:6789/0 conn(0x7f65e0137890 legacy=0x7f65e013e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:22.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65de7fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:41:22.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65de7fc640 1 --2- 172.21.15.37:0/3974758123 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65cc0039c0 0x7f65cc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:22.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65de7fc640 10 monclient(hunting): picked mon.b con 0x7f65cc0039c0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:41:22.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65de7fc640 1 --2- 172.21.15.37:0/3974758123 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65cc004810 0x7f65cc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:22.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65de7fc640 10 monclient(hunting): picked mon.a con 0x7f65cc004810 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:41:22.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65de7fc640 1 --2- 172.21.15.37:0/3974758123 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f65cc0053d0 0x7f65cc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:22.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65de7fc640 10 monclient(hunting): picked mon.c con 0x7f65cc0053d0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:41:22.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65de7fc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:22.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65de7fc640 1 -- 172.21.15.37:0/3974758123 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f65e009bb40 con 0x7f65cc004810 2024-04-26T00:41:22.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65de7fc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:22.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65de7fc640 1 -- 172.21.15.37:0/3974758123 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f65cc00c260 con 0x7f65cc0039c0 2024-04-26T00:41:22.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65de7fc640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:22.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65de7fc640 1 -- 172.21.15.37:0/3974758123 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f65cc00c460 con 0x7f65cc0053d0 2024-04-26T00:41:22.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65de7fc640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:22.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.001+0000 7f65de7fc640 10 monclient(hunting): _finish_auth 0 2024-04-26T00:41:22.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.005+0000 7f65deffd640 1 --2- 172.21.15.37:0/3974758123 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65cc004810 0x7f65cc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:22.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.005+0000 7f65dffff640 1 --2- 172.21.15.37:0/3974758123 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f65cc0053d0 0x7f65cc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:22.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.005+0000 7f65deffd640 10 monclient(hunting): get_auth_request con 0x7f65cc004810 auth_method 0 2024-04-26T00:41:22.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.005+0000 7f65deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:22.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.005+0000 7f65deffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:22.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.005+0000 7f65deffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:41:22.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.005+0000 7f65dffff640 10 monclient(hunting): get_auth_request con 0x7f65cc0053d0 auth_method 0 2024-04-26T00:41:22.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.005+0000 7f65dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:22.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.005+0000 7f65dffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:22.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.005+0000 7f65dffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:41:22.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.005+0000 7f65df7fe640 1 --2- 172.21.15.37:0/3974758123 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65cc0039c0 0x7f65cc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:22.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.005+0000 7f65dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:22.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.005+0000 7f65dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:22.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.005+0000 7f65dffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:41:22.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.005+0000 7f65deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:22.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.005+0000 7f65deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:22.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.005+0000 7f65deffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:41:22.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.005+0000 7f65df7fe640 10 monclient(hunting): get_auth_request con 0x7f65cc0039c0 auth_method 0 2024-04-26T00:41:22.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.005+0000 7f65df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:22.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.005+0000 7f65df7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:22.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.005+0000 7f65df7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:41:22.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.005+0000 7f65dffff640 10 monclient(hunting): handle_auth_done global_id 224431 payload 293 2024-04-26T00:41:22.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.005+0000 7f65dffff640 1 -- 172.21.15.37:0/3974758123 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65cc0039c0 msgr2=0x7f65cc003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:41:22.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.005+0000 7f65dffff640 1 --2- 172.21.15.37:0/3974758123 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65cc0039c0 0x7f65cc003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:22.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.005+0000 7f65dffff640 1 -- 172.21.15.37:0/3974758123 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65cc004810 msgr2=0x7f65cc004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:41:22.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.005+0000 7f65dffff640 1 --2- 172.21.15.37:0/3974758123 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65cc004810 0x7f65cc004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:22.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.005+0000 7f65dffff640 10 monclient: _finish_hunting 0 2024-04-26T00:41:22.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.005+0000 7f65dffff640 1 monclient: found mon.c 2024-04-26T00:41:22.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.005+0000 7f65dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:22.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.005+0000 7f65dffff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:41:22.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.005+0000 7f65dffff640 1 -- 172.21.15.37:0/3974758123 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65cc003530 con 0x7f65cc0053d0 2024-04-26T00:41:22.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.005+0000 7f65dffff640 10 monclient: _finish_auth 0 2024-04-26T00:41:22.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.005+0000 7f65dffff640 10 monclient: _check_auth_tickets 2024-04-26T00:41:22.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.005+0000 7f65dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:41:22.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.005+0000 7f65e603b640 5 monclient: authenticate success, global_id 224431 2024-04-26T00:41:22.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.005+0000 7f65e603b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:41:22.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.005+0000 7f65deffd640 1 --2- 172.21.15.37:0/3974758123 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65cc004810 0x7f65cc004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:41:22.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.005+0000 7f65dffff640 1 -- 172.21.15.37:0/3974758123 reap_dead start 2024-04-26T00:41:22.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65dffff640 1 -- 172.21.15.37:0/3974758123 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f65cc0053d0 msgr2=0x7f65cc00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:22.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65dffff640 1 --2- 172.21.15.37:0/3974758123 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f65cc0053d0 0x7f65cc00b9d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f65d8001870 tx=0x7f65d801d0d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fec45f2d217ba31e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:22.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65de7fc640 1 -- 172.21.15.37:0/3974758123 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d0001730 con 0x7f65cc0053d0 2024-04-26T00:41:22.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65de7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:22.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65de7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:41:22.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65de7fc640 10 monclient: dump: 2024-04-26T00:41:22.019 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:22.019 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:22.019 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:22.019 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:22.020 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:22.020 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:22.020 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:22.020 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:22.020 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:22.020 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:22.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65de7fc640 1 -- 172.21.15.37:0/3974758123 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65d80a9070 con 0x7f65cc0053d0 2024-04-26T00:41:22.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65de7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:41:22.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65de7fc640 1 -- 172.21.15.37:0/3974758123 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d800a370 con 0x7f65cc0053d0 2024-04-26T00:41:22.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65e603b640 10 monclient: get_monmap_and_config success 2024-04-26T00:41:22.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65e603b640 4 set_mon_vals no callback set 2024-04-26T00:41:22.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65de7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:22.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65de7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:41:22.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65de7fc640 10 monclient: dump: 2024-04-26T00:41:22.020 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:22.021 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:22.021 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:22.021 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:22.021 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:22.021 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:22.021 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:22.021 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:22.021 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:22.021 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:22.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65e603b640 10 monclient: shutdown 2024-04-26T00:41:22.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65e603b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:41:22.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65e603b640 1 -- 172.21.15.37:0/3974758123 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f65cc0053d0 msgr2=0x7f65cc00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:22.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65e603b640 1 --2- 172.21.15.37:0/3974758123 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f65cc0053d0 0x7f65cc00b9d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f65d8001870 tx=0x7f65d801d0d0 comp rx=0 tx=0).stop 2024-04-26T00:41:22.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65e603b640 1 -- 172.21.15.37:0/3974758123 shutdown_connections 2024-04-26T00:41:22.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65e603b640 1 --2- 172.21.15.37:0/3974758123 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f65cc0053d0 0x7f65cc00b9d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:22.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65e603b640 1 -- 172.21.15.37:0/3974758123 >> 172.21.15.37:0/3974758123 conn(0x7f65e00a5f00 msgr2=0x7f65e00a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:41:22.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65e603b640 1 -- 172.21.15.37:0/3974758123 shutdown_connections 2024-04-26T00:41:22.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65e603b640 1 -- 172.21.15.37:0/3974758123 wait complete. 2024-04-26T00:41:22.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65e603b640 10 monclient: build_initial_monmap 2024-04-26T00:41:22.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65e603b640 10 monclient: monmap: 2024-04-26T00:41:22.022 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:41:22.022 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:41:22.022 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:41:22.022 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:41:22.022 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:41:22.022 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:41:22.022 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:41:22.022 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:41:22.023 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:41:22.023 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:22.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65e603b640 1 Processor -- start 2024-04-26T00:41:22.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65e603b640 1 -- start start 2024-04-26T00:41:22.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65e603b640 10 monclient: init 2024-04-26T00:41:22.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65e603b640 10 monclient: _reopen_session rank -1 2024-04-26T00:41:22.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65e603b640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:41:22.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65e603b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65cc0053d0 0x7f65e01ce880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:22.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65e603b640 10 monclient(hunting): picked mon.noname-a con 0x7f65cc0053d0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:41:22.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65e603b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f65e01cf050 0x7f65e013aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:22.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65e603b640 10 monclient(hunting): picked mon.noname-c con 0x7f65e01cf050 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:41:22.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65e603b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65e01cf490 0x7f65e013b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:22.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65e603b640 10 monclient(hunting): picked mon.noname-b con 0x7f65e01cf490 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:41:22.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65e603b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:22.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65e603b640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f65e009c260 con 0x7f65cc0053d0 2024-04-26T00:41:22.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65e603b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:22.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65e603b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f65e009c0e0 con 0x7f65e01cf490 2024-04-26T00:41:22.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65e603b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:22.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65e603b640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f65e0137180 con 0x7f65e01cf050 2024-04-26T00:41:22.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65e603b640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:22.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65e603b640 10 monclient(hunting): authenticate will time out at 2263.318819s 2024-04-26T00:41:22.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65df7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65cc0053d0 0x7f65e01ce880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:22.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65df7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65cc0053d0 0x7f65e01ce880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:37554/0 (socket says 172.21.15.37:37554) 2024-04-26T00:41:22.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65df7fe640 1 -- 172.21.15.37:0/2478881662 learned_addr learned my addr 172.21.15.37:0/2478881662 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:41:22.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65df7fe640 10 monclient(hunting): get_auth_request con 0x7f65cc0053d0 auth_method 0 2024-04-26T00:41:22.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:22.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65df7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:22.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:22.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65deffd640 1 --2- 172.21.15.37:0/2478881662 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f65e01cf050 0x7f65e013aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:22.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65deffd640 10 monclient(hunting): get_auth_request con 0x7f65e01cf050 auth_method 0 2024-04-26T00:41:22.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:22.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65deffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:22.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65deffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:22.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65df7fe640 1 -- 172.21.15.37:0/2478881662 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65cc0053d0 msgr2=0x7f65e01ce880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:41:22.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65df7fe640 1 -- 172.21.15.37:0/2478881662 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65cc0053d0 msgr2=0x7f65e01ce880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:41:22.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65df7fe640 1 --2- 172.21.15.37:0/2478881662 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65cc0053d0 0x7f65e01ce880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:41:22.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65df7fe640 1 --2- 172.21.15.37:0/2478881662 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65cc0053d0 0x7f65e01ce880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:41:22.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:22.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:22.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:22.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65dffff640 1 --2- 172.21.15.37:0/2478881662 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65e01cf490 0x7f65e013b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:22.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65dffff640 10 monclient(hunting): get_auth_request con 0x7f65e01cf490 auth_method 0 2024-04-26T00:41:22.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:22.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65dffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:22.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65dffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:22.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65deffd640 10 monclient(hunting): handle_auth_done global_id 254189 payload 824 2024-04-26T00:41:22.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65deffd640 1 -- 172.21.15.37:0/2478881662 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65e01cf490 msgr2=0x7f65e013b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:22.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65deffd640 1 --2- 172.21.15.37:0/2478881662 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65e01cf490 0x7f65e013b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:22.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65deffd640 1 -- 172.21.15.37:0/2478881662 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65cc0053d0 msgr2=0x7f65e01ce880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:41:22.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65deffd640 1 --2- 172.21.15.37:0/2478881662 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65cc0053d0 0x7f65e01ce880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:22.026 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65deffd640 10 monclient: _finish_hunting 0 2024-04-26T00:41:22.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65deffd640 1 monclient: found mon.noname-c 2024-04-26T00:41:22.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:22.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65deffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:41:22.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65deffd640 1 -- 172.21.15.37:0/2478881662 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65e009aca0 con 0x7f65e01cf050 2024-04-26T00:41:22.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.009+0000 7f65dffff640 1 --2- 172.21.15.37:0/2478881662 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65e01cf490 0x7f65e013b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:41:22.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.013+0000 7f65deffd640 1 -- 172.21.15.37:0/2478881662 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f65e01cf050 msgr2=0x7f65e013aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:22.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.013+0000 7f65deffd640 1 --2- 172.21.15.37:0/2478881662 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f65e01cf050 0x7f65e013aec0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f65d0007610 tx=0x7f65d00019e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:22.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.013+0000 7f65d7fff640 1 -- 172.21.15.37:0/2478881662 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d0011790 con 0x7f65e01cf050 2024-04-26T00:41:22.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.013+0000 7f65d7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:22.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.013+0000 7f65d7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:41:22.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.013+0000 7f65d7fff640 10 monclient: dump: 2024-04-26T00:41:22.028 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:22.028 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:22.028 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:22.028 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:22.028 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:22.029 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:22.029 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:22.029 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:22.029 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:22.029 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:22.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.013+0000 7f65d7fff640 10 monclient: _finish_auth 0 2024-04-26T00:41:22.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.013+0000 7f65d7fff640 10 monclient: _check_auth_tickets 2024-04-26T00:41:22.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.013+0000 7f65d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:41:22.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.013+0000 7f65d7fff640 1 -- 172.21.15.37:0/2478881662 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65d0018070 con 0x7f65e01cf050 2024-04-26T00:41:22.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.013+0000 7f65d7fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:41:22.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.013+0000 7f65d7fff640 1 -- 172.21.15.37:0/2478881662 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d0011b30 con 0x7f65e01cf050 2024-04-26T00:41:22.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.013+0000 7f65e603b640 5 monclient: authenticate success, global_id 254189 2024-04-26T00:41:22.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.013+0000 7f65e603b640 10 monclient: _renew_subs 2024-04-26T00:41:22.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.013+0000 7f65e603b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:41:22.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.013+0000 7f65e603b640 1 -- 172.21.15.37:0/2478881662 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65e013cae0 con 0x7f65e01cf050 2024-04-26T00:41:22.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.013+0000 7f65dd7fa640 4 set_mon_vals no callback set 2024-04-26T00:41:22.030 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.013+0000 7f65d7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:22.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.013+0000 7f65d7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:41:22.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.013+0000 7f65d7fff640 10 monclient: dump: 2024-04-26T00:41:22.031 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:22.031 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:22.031 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:22.031 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:22.032 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:22.032 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:22.032 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:22.032 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:22.032 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:22.032 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:22.032 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.013+0000 7f65e603b640 10 monclient: _renew_subs 2024-04-26T00:41:22.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.013+0000 7f65e603b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:41:22.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.013+0000 7f65e603b640 1 -- 172.21.15.37:0/2478881662 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f65e013d380 con 0x7f65e01cf050 2024-04-26T00:41:22.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.013+0000 7f65e603b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:22.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.013+0000 7f65e603b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:22.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.013+0000 7f65e603b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:41:22.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.013+0000 7f65d7fff640 1 -- 172.21.15.37:0/2478881662 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f65d0011cd0 con 0x7f65e01cf050 2024-04-26T00:41:22.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.013+0000 7f65e603b640 1 -- 172.21.15.37:0/2478881662 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65e0066f40 con 0x7f65e01cf050 2024-04-26T00:41:22.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.013+0000 7f65d7fff640 1 --2- 172.21.15.37:0/2478881662 >> v2:172.21.15.45:6812/2379918623 conn(0x7f65c0035ed0 0x7f65c0038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:22.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.013+0000 7f65d7fff640 1 -- 172.21.15.37:0/2478881662 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(848..848 src has 705..848) ==== 4007+0+0 (secure 0 0 0) 0x7f65d0048b90 con 0x7f65e01cf050 2024-04-26T00:41:22.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.013+0000 7f65df7fe640 1 --2- 172.21.15.37:0/2478881662 >> v2:172.21.15.45:6812/2379918623 conn(0x7f65c0035ed0 0x7f65c0038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:22.034 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.013+0000 7f65df7fe640 10 monclient: get_auth_request con 0x7f65c0035ed0 auth_method 0 2024-04-26T00:41:22.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.017+0000 7f65df7fe640 1 --2- 172.21.15.37:0/2478881662 >> v2:172.21.15.45:6812/2379918623 conn(0x7f65c0035ed0 0x7f65c0038390 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f65c8006d50 tx=0x7f65c800e490 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:22.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.017+0000 7f65d7fff640 1 -- 172.21.15.37:0/2478881662 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f65d0047890 con 0x7f65e01cf050 2024-04-26T00:41:22.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.017+0000 7f65d7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:22.035 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.017+0000 7f65d7fff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:41:22.190 INFO:tasks.rados.rados.0.smithi045.stdout:3767: finishing write tid 1 to smithi04526579-2 2024-04-26T00:41:22.190 INFO:tasks.rados.rados.0.smithi045.stdout:3767: finishing write tid 2 to smithi04526579-2 2024-04-26T00:41:22.190 INFO:tasks.rados.rados.0.smithi045.stdout:3767: finishing write tid 3 to smithi04526579-2 2024-04-26T00:41:22.190 INFO:tasks.rados.rados.0.smithi045.stdout:3767: finishing write tid 4 to smithi04526579-2 2024-04-26T00:41:22.190 INFO:tasks.rados.rados.0.smithi045.stdout:3767: oid 2 updating version 0 to 2538 2024-04-26T00:41:22.190 INFO:tasks.rados.rados.0.smithi045.stdout:3767: oid 2 updating version 2538 to 2539 2024-04-26T00:41:22.190 INFO:tasks.rados.rados.0.smithi045.stdout:3767: oid 2 version 2539 is already newer than 2537 2024-04-26T00:41:22.190 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 2539 (ObjNum 1347 snap 416 seq_num 1347) dirty exists 2024-04-26T00:41:22.191 INFO:tasks.rados.rados.0.smithi045.stdout:3767: left oid 2 (ObjNum 1347 snap 416 seq_num 1347) 2024-04-26T00:41:22.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.241+0000 7f65e603b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:41:22.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.241+0000 7f65e603b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:41:22.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.241+0000 7f65e603b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:41:22.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.241+0000 7f65e603b640 1 -- 172.21.15.37:0/2478881662 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f65e0075d70 con 0x7f65e01cf050 2024-04-26T00:41:22.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.241+0000 7f65d7fff640 1 -- 172.21.15.37:0/2478881662 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7f65d001d890 con 0x7f65e01cf050 2024-04-26T00:41:22.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.241+0000 7f65d7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:41:22.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.241+0000 7f65d7fff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:41:22.245 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":154,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":25,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:41:22.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.245+0000 7f65e603b640 1 -- 172.21.15.37:0/2478881662 >> v2:172.21.15.45:6812/2379918623 conn(0x7f65c0035ed0 msgr2=0x7f65c0038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:22.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.245+0000 7f65e603b640 1 --2- 172.21.15.37:0/2478881662 >> v2:172.21.15.45:6812/2379918623 conn(0x7f65c0035ed0 0x7f65c0038390 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f65c8006d50 tx=0x7f65c800e490 comp rx=0 tx=0).stop 2024-04-26T00:41:22.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.245+0000 7f65e603b640 10 monclient: shutdown 2024-04-26T00:41:22.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.245+0000 7f65e603b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:41:22.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.245+0000 7f65e603b640 1 -- 172.21.15.37:0/2478881662 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f65e01cf050 msgr2=0x7f65e013aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:22.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.245+0000 7f65e603b640 1 --2- 172.21.15.37:0/2478881662 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f65e01cf050 0x7f65e013aec0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f65d0007610 tx=0x7f65d00019e0 comp rx=0 tx=0).stop 2024-04-26T00:41:22.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.245+0000 7f65e603b640 1 -- 172.21.15.37:0/2478881662 shutdown_connections 2024-04-26T00:41:22.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.245+0000 7f65e603b640 1 --2- 172.21.15.37:0/2478881662 >> v2:172.21.15.45:6812/2379918623 conn(0x7f65c0035ed0 0x7f65c0038390 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:22.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.245+0000 7f65e603b640 1 --2- 172.21.15.37:0/2478881662 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65e01cf490 0x7f65e013b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:22.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.245+0000 7f65e603b640 1 --2- 172.21.15.37:0/2478881662 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f65e01cf050 0x7f65e013aec0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:22.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.245+0000 7f65e603b640 1 --2- 172.21.15.37:0/2478881662 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65cc0053d0 0x7f65e01ce880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:22.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.245+0000 7f65e603b640 1 -- 172.21.15.37:0/2478881662 >> 172.21.15.37:0/2478881662 conn(0x7f65e00a5f00 msgr2=0x7f65e013f010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:41:22.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.245+0000 7f65e603b640 1 -- 172.21.15.37:0/2478881662 shutdown_connections 2024-04-26T00:41:22.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:22.245+0000 7f65e603b640 1 -- 172.21.15.37:0/2478881662 wait complete. 2024-04-26T00:41:22.968 INFO:tasks.rados.rados.0.smithi045.stdout:3764: done (4 left) 2024-04-26T00:41:22.969 INFO:tasks.rados.rados.0.smithi045.stdout:3765: done (3 left) 2024-04-26T00:41:22.969 INFO:tasks.rados.rados.0.smithi045.stdout:3766: done (2 left) 2024-04-26T00:41:22.969 INFO:tasks.rados.rados.0.smithi045.stdout:3767: done (1 left) 2024-04-26T00:41:22.969 INFO:tasks.rados.rados.0.smithi045.stdout:3769: done (0 left) 2024-04-26T00:41:22.969 INFO:tasks.rados.rados.0.smithi045.stdout:3770: snap_remove snap 415 2024-04-26T00:41:23.983 INFO:tasks.rados.rados.0.smithi045.stdout:3770: done (0 left) 2024-04-26T00:41:23.983 INFO:tasks.rados.rados.0.smithi045.stdout:3771: read oid 2 snap 409 2024-04-26T00:41:23.983 INFO:tasks.rados.rados.0.smithi045.stdout:3771: expect (ObjNum 1250 snap 388 seq_num 1250) 2024-04-26T00:41:23.983 INFO:tasks.rados.rados.0.smithi045.stdout:3772: read oid 50 snap -1 2024-04-26T00:41:23.983 INFO:tasks.rados.rados.0.smithi045.stdout:3772: expect deleted 2024-04-26T00:41:23.984 INFO:tasks.rados.rados.0.smithi045.stdout:3773: snap_remove snap 399 2024-04-26T00:41:24.988 INFO:tasks.rados.rados.0.smithi045.stdout:3773: done (2 left) 2024-04-26T00:41:24.988 INFO:tasks.rados.rados.0.smithi045.stdout:3774: read oid 17 snap -1 2024-04-26T00:41:24.988 INFO:tasks.rados.rados.0.smithi045.stdout:3774: expect deleted 2024-04-26T00:41:24.988 INFO:tasks.rados.rados.0.smithi045.stdout:3775: read oid 7 snap -1 2024-04-26T00:41:24.988 INFO:tasks.rados.rados.0.smithi045.stdout:3775: expect deleted 2024-04-26T00:41:24.988 INFO:tasks.rados.rados.0.smithi045.stdout:3776: copy_from oid 33 from oid 23 current snap is 417 2024-04-26T00:41:24.988 INFO:tasks.rados.rados.0.smithi045.stdout:3771: expect (ObjNum 1250 snap 388 seq_num 1250) 2024-04-26T00:41:25.185 INFO:tasks.rados.rados.0.smithi045.stdout:3776: finishing copy_from to smithi04526579-33 2024-04-26T00:41:25.186 INFO:tasks.rados.rados.0.smithi045.stdout:3776: got expected ENOENT (src dne) 2024-04-26T00:41:25.186 INFO:tasks.rados.rados.0.smithi045.stdout:3776: finishing copy_from racing read to smithi04526579-33 2024-04-26T00:41:25.186 INFO:tasks.rados.rados.0.smithi045.stdout:3771: done (4 left) 2024-04-26T00:41:25.186 INFO:tasks.rados.rados.0.smithi045.stdout:3772: done (3 left) 2024-04-26T00:41:25.186 INFO:tasks.rados.rados.0.smithi045.stdout:3774: done (2 left) 2024-04-26T00:41:25.186 INFO:tasks.rados.rados.0.smithi045.stdout:3775: done (1 left) 2024-04-26T00:41:25.186 INFO:tasks.rados.rados.0.smithi045.stdout:3776: done (0 left) 2024-04-26T00:41:25.186 INFO:tasks.rados.rados.0.smithi045.stdout:3777: read oid 7 snap -1 2024-04-26T00:41:25.186 INFO:tasks.rados.rados.0.smithi045.stdout:3777: expect deleted 2024-04-26T00:41:25.186 INFO:tasks.rados.rados.0.smithi045.stdout:3778: write oid 3 current snap is 417 2024-04-26T00:41:25.186 INFO:tasks.rados.rados.0.smithi045.stdout:3778: seq_num 1348 ranges {655836=430133,1831727=757138} 2024-04-26T00:41:25.203 INFO:tasks.rados.rados.0.smithi045.stdout:3778: writing smithi04526579-3 from 655836 to 1085969 tid 1 2024-04-26T00:41:25.229 INFO:tasks.rados.rados.0.smithi045.stdout:3778: writing smithi04526579-3 from 1831727 to 2588865 tid 2 2024-04-26T00:41:25.229 INFO:tasks.rados.rados.0.smithi045.stdout:3779: rollback oid 22 current snap is 417 2024-04-26T00:41:25.229 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 22 to 346 2024-04-26T00:41:25.229 INFO:tasks.rados.rados.0.smithi045.stdout:3778: finishing write tid 1 to smithi04526579-3 2024-04-26T00:41:25.230 INFO:tasks.rados.rados.0.smithi045.stdout:3777: done (2 left) 2024-04-26T00:41:25.230 INFO:tasks.rados.rados.0.smithi045.stdout:3780: copy_from oid 16 from oid 6 current snap is 417 2024-04-26T00:41:25.230 INFO:tasks.rados.rados.0.smithi045.stdout:3781: snap_create 2024-04-26T00:41:25.242 INFO:tasks.rados.rados.0.smithi045.stdout:3779: finishing rollback tid 0 to smithi04526579-22 2024-04-26T00:41:25.242 INFO:tasks.rados.rados.0.smithi045.stdout:3778: finishing write tid 2 to smithi04526579-3 2024-04-26T00:41:25.242 INFO:tasks.rados.rados.0.smithi045.stdout:3778: finishing write tid 3 to smithi04526579-3 2024-04-26T00:41:25.243 INFO:tasks.rados.rados.0.smithi045.stdout:3779: finishing rollback tid 1 to smithi04526579-22 2024-04-26T00:41:25.243 INFO:tasks.rados.rados.0.smithi045.stdout:3779: finishing rollback tid 2 to smithi04526579-22 2024-04-26T00:41:25.243 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 2468 (ObjNum 1135 snap 343 seq_num 1135) dirty exists 2024-04-26T00:41:25.243 INFO:tasks.rados.rados.0.smithi045.stdout:3778: finishing write tid 4 to smithi04526579-3 2024-04-26T00:41:25.243 INFO:tasks.rados.rados.0.smithi045.stdout:3778: oid 3 updating version 0 to 2517 2024-04-26T00:41:25.243 INFO:tasks.rados.rados.0.smithi045.stdout:3778: oid 3 version 2517 is already newer than 2515 2024-04-26T00:41:25.243 INFO:tasks.rados.rados.0.smithi045.stdout:3778: oid 3 version 2517 is already newer than 2516 2024-04-26T00:41:25.243 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 2517 (ObjNum 1348 snap 417 seq_num 1348) dirty exists 2024-04-26T00:41:25.243 INFO:tasks.rados.rados.0.smithi045.stdout:3778: left oid 3 (ObjNum 1348 snap 417 seq_num 1348) 2024-04-26T00:41:25.257 INFO:tasks.rados.rados.0.smithi045.stdout:3780: finishing copy_from racing read to smithi04526579-16 2024-04-26T00:41:25.259 INFO:tasks.rados.rados.0.smithi045.stdout:3780: finishing copy_from to smithi04526579-16 2024-04-26T00:41:25.259 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 2470 (ObjNum 1333 snap 411 seq_num 1333) dirty exists 2024-04-26T00:41:25.264 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:41:25.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fdaa17cb640 10 monclient: get_monmap_and_config 2024-04-26T00:41:25.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fdaa17cb640 10 monclient: build_initial_monmap 2024-04-26T00:41:25.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fdaa17cb640 10 monclient: monmap: 2024-04-26T00:41:25.392 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:41:25.392 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:25.392 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:41:25.393713+0000 2024-04-26T00:41:25.392 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:41:25.393713+0000 2024-04-26T00:41:25.392 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:41:25.392 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:25.392 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:41:25.392 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:41:25.392 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:41:25.392 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:41:25.392 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:41:25.392 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:25.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fdaa17cb640 1 Processor -- start 2024-04-26T00:41:25.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fdaa17cb640 1 -- start start 2024-04-26T00:41:25.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fdaa17cb640 10 monclient: init 2024-04-26T00:41:25.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fdaa17cb640 10 monclient: _reopen_session rank -1 2024-04-26T00:41:25.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fdaa17cb640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-04-26T00:41:25.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fdaa17cb640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fda9c0a0260 0x7fda9c0a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:25.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fdaa17cb640 10 monclient(hunting): picked mon.noname-b con 0x7fda9c0a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:41:25.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fdaa17cb640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fda9c134b30 0x7fda9c134f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:25.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fdaa17cb640 10 monclient(hunting): picked mon.noname-c con 0x7fda9c134b30 addr v2:172.21.15.45:3300/0 2024-04-26T00:41:25.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fdaa17cb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fda9c135490 addr v1:172.21.15.45:6789/0 2024-04-26T00:41:25.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fdaa17cb640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda9c0a1130 con 0x7fda9c135490 2024-04-26T00:41:25.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fdaa17cb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:25.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fdaa17cb640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fda9c09ec10 con 0x7fda9c134b30 2024-04-26T00:41:25.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fdaa17cb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:25.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fdaa17cb640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fda9c09f0f0 con 0x7fda9c0a0260 2024-04-26T00:41:25.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fdaa17cb640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:25.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fdaa17cb640 10 monclient(hunting): authenticate will time out at 2266.699354s 2024-04-26T00:41:25.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fda9affd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fda9c0a0260 0x7fda9c0a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:25.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fda9affd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fda9c0a0260 0x7fda9c0a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:40636/0 (socket says 172.21.15.37:40636) 2024-04-26T00:41:25.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fda9affd640 1 -- 172.21.15.37:0/3121712115 learned_addr learned my addr 172.21.15.37:0/3121712115 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:41:25.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fda9affd640 10 monclient(hunting): get_auth_request con 0x7fda9c0a0260 auth_method 0 2024-04-26T00:41:25.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fda9affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:25.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fda9affd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:25.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fda9affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:25.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fda9affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:25.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fda9affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:25.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fda9affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:25.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fda9a7fc640 1 --2- 172.21.15.37:0/3121712115 >> v2:172.21.15.45:3300/0 conn(0x7fda9c134b30 0x7fda9c134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:25.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fda9a7fc640 10 monclient(hunting): get_auth_request con 0x7fda9c134b30 auth_method 0 2024-04-26T00:41:25.396 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fda9a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:25.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fda9a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:25.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fda9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:25.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fda9affd640 10 monclient(hunting): handle_auth_done global_id 254192 payload 274 2024-04-26T00:41:25.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fda9affd640 1 -- 172.21.15.37:0/3121712115 >> v2:172.21.15.45:3300/0 conn(0x7fda9c134b30 msgr2=0x7fda9c134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:41:25.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fda9affd640 1 --2- 172.21.15.37:0/3121712115 >> v2:172.21.15.45:3300/0 conn(0x7fda9c134b30 0x7fda9c134f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:25.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fda9affd640 1 -- 172.21.15.37:0/3121712115 >> v1:172.21.15.45:6789/0 conn(0x7fda9c135490 legacy=0x7fda9c13be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:41:25.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fda9affd640 10 monclient: _finish_hunting 0 2024-04-26T00:41:25.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fda9affd640 1 monclient: found mon.noname-b 2024-04-26T00:41:25.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fda9affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:25.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fda9affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:41:25.397 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.389+0000 7fda9affd640 1 -- 172.21.15.37:0/3121712115 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda9c06cff0 con 0x7fda9c0a0260 2024-04-26T00:41:25.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda9affd640 1 -- 172.21.15.37:0/3121712115 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fda9c0a0260 msgr2=0x7fda9c0a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:25.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda9affd640 1 --2- 172.21.15.37:0/3121712115 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fda9c0a0260 0x7fda9c0a0640 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fda84009870 tx=0x7fda8401c200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cec5275ca82976e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:25.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda99ffb640 1 -- 172.21.15.37:0/3121712115 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda840b0070 con 0x7fda9c0a0260 2024-04-26T00:41:25.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda99ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:25.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda99ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:41:25.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda99ffb640 10 monclient: dump: 2024-04-26T00:41:25.400 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:25.400 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:25.400 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:25.400 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:25.400 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:25.400 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:25.400 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:25.400 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:25.400 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:25.400 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:25.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda99ffb640 10 monclient: _finish_auth 0 2024-04-26T00:41:25.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda99ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:41:25.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda99ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:41:25.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda99ffb640 1 -- 172.21.15.37:0/3121712115 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda840ab600 con 0x7fda9c0a0260 2024-04-26T00:41:25.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fdaa17cb640 5 monclient: authenticate success, global_id 254192 2024-04-26T00:41:25.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fdaa17cb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:41:25.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda99ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:41:25.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda99ffb640 1 -- 172.21.15.37:0/3121712115 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda840ab9d0 con 0x7fda9c0a0260 2024-04-26T00:41:25.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda99ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:25.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda99ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:41:25.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda99ffb640 10 monclient: dump: 2024-04-26T00:41:25.401 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:25.401 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:25.401 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:25.401 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:25.401 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:25.402 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:25.402 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:25.402 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:25.402 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:25.402 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:25.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fdaa17cb640 10 monclient: get_monmap_and_config success 2024-04-26T00:41:25.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fdaa17cb640 4 set_mon_vals no callback set 2024-04-26T00:41:25.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fdaa17cb640 10 monclient: shutdown 2024-04-26T00:41:25.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fdaa17cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:41:25.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fdaa17cb640 1 -- 172.21.15.37:0/3121712115 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fda9c0a0260 msgr2=0x7fda9c0a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:25.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fdaa17cb640 1 --2- 172.21.15.37:0/3121712115 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fda9c0a0260 0x7fda9c0a0640 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fda84009870 tx=0x7fda8401c200 comp rx=0 tx=0).stop 2024-04-26T00:41:25.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fdaa17cb640 1 -- 172.21.15.37:0/3121712115 shutdown_connections 2024-04-26T00:41:25.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fdaa17cb640 1 --2- 172.21.15.37:0/3121712115 >> v2:172.21.15.45:3300/0 conn(0x7fda9c134b30 0x7fda9c134f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:25.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fdaa17cb640 1 --2- 172.21.15.37:0/3121712115 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fda9c0a0260 0x7fda9c0a0640 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:25.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fdaa17cb640 1 -- 172.21.15.37:0/3121712115 >> 172.21.15.37:0/3121712115 conn(0x7fda9c0a1ad0 msgr2=0x7fda9c0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:41:25.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fdaa17cb640 1 -- 172.21.15.37:0/3121712115 shutdown_connections 2024-04-26T00:41:25.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fdaa17cb640 1 -- 172.21.15.37:0/3121712115 wait complete. 2024-04-26T00:41:25.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fdaa17cb640 10 monclient: build_initial_monmap 2024-04-26T00:41:25.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fdaa17cb640 10 monclient: monmap: 2024-04-26T00:41:25.403 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:41:25.403 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:41:25.403 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:41:25.403 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:41:25.403 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:41:25.403 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:41:25.403 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:41:25.404 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:41:25.404 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:41:25.404 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:25.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fdaa17cb640 1 Processor -- start 2024-04-26T00:41:25.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fdaa17cb640 1 -- start start 2024-04-26T00:41:25.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fdaa17cb640 10 monclient: init 2024-04-26T00:41:25.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fdaa17cb640 10 monclient: _reopen_session rank -1 2024-04-26T00:41:25.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fdaa17cb640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:41:25.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fdaa17cb640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fda9c0a0260 0x7fda9c131d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:25.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fdaa17cb640 10 monclient(hunting): picked mon.noname-c con 0x7fda9c0a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:41:25.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fdaa17cb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda9c134b30 0x7fda9c132290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:25.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fdaa17cb640 10 monclient(hunting): picked mon.noname-b con 0x7fda9c134b30 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:41:25.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fdaa17cb640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fda9c135490 0x7fda9c11b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:25.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fdaa17cb640 10 monclient(hunting): picked mon.noname-a con 0x7fda9c135490 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:41:25.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fdaa17cb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:25.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fdaa17cb640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fda9c09f430 con 0x7fda9c135490 2024-04-26T00:41:25.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fdaa17cb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:25.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fdaa17cb640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fda9c09f2b0 con 0x7fda9c134b30 2024-04-26T00:41:25.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fdaa17cb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:25.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fdaa17cb640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fda9c09fe90 con 0x7fda9c0a0260 2024-04-26T00:41:25.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fdaa17cb640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:25.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fdaa17cb640 10 monclient(hunting): authenticate will time out at 2266.706658s 2024-04-26T00:41:25.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda9affd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fda9c0a0260 0x7fda9c131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:25.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda9b7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fda9c135490 0x7fda9c11b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:25.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda9b7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fda9c135490 0x7fda9c11b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:37566/0 (socket says 172.21.15.37:37566) 2024-04-26T00:41:25.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda9b7fe640 1 -- 172.21.15.37:0/928340715 learned_addr learned my addr 172.21.15.37:0/928340715 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:41:25.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda9b7fe640 10 monclient(hunting): get_auth_request con 0x7fda9c135490 auth_method 0 2024-04-26T00:41:25.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:25.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda9b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:25.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:25.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda9affd640 10 monclient(hunting): get_auth_request con 0x7fda9c0a0260 auth_method 0 2024-04-26T00:41:25.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda9affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:25.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda9affd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:25.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda9affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:25.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda9b7fe640 1 -- 172.21.15.37:0/928340715 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fda9c135490 msgr2=0x7fda9c11b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:41:25.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda9b7fe640 1 -- 172.21.15.37:0/928340715 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fda9c135490 msgr2=0x7fda9c11b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:41:25.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda9b7fe640 1 --2- 172.21.15.37:0/928340715 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fda9c135490 0x7fda9c11b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:41:25.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda9b7fe640 1 --2- 172.21.15.37:0/928340715 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fda9c135490 0x7fda9c11b620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:41:25.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda9a7fc640 1 --2- 172.21.15.37:0/928340715 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda9c134b30 0x7fda9c132290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:25.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda9affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:25.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda9affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:25.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda9affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:25.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda9a7fc640 10 monclient(hunting): get_auth_request con 0x7fda9c134b30 auth_method 0 2024-04-26T00:41:25.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda9a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:25.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda9a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:25.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:25.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda9a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:25.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda9a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:25.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda9a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:25.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda9affd640 10 monclient(hunting): handle_auth_done global_id 254195 payload 824 2024-04-26T00:41:25.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda9affd640 1 -- 172.21.15.37:0/928340715 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda9c134b30 msgr2=0x7fda9c132290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:25.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda9affd640 1 --2- 172.21.15.37:0/928340715 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda9c134b30 0x7fda9c132290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:25.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda9affd640 1 -- 172.21.15.37:0/928340715 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fda9c135490 msgr2=0x7fda9c11b620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:41:25.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda9affd640 1 --2- 172.21.15.37:0/928340715 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fda9c135490 0x7fda9c11b620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:25.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda9affd640 10 monclient: _finish_hunting 0 2024-04-26T00:41:25.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda9affd640 1 monclient: found mon.noname-c 2024-04-26T00:41:25.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda9affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:25.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda9affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:41:25.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.397+0000 7fda9affd640 1 -- 172.21.15.37:0/928340715 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda9c132bf0 con 0x7fda9c0a0260 2024-04-26T00:41:25.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.401+0000 7fda9affd640 1 -- 172.21.15.37:0/928340715 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fda9c0a0260 msgr2=0x7fda9c131d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:25.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.401+0000 7fda9affd640 1 --2- 172.21.15.37:0/928340715 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fda9c0a0260 0x7fda9c131d50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fda840ad000 tx=0x7fda84018fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:25.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.401+0000 7fda937fe640 1 -- 172.21.15.37:0/928340715 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda840ba070 con 0x7fda9c0a0260 2024-04-26T00:41:25.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.401+0000 7fda937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:25.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.401+0000 7fda937fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:41:25.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.401+0000 7fda937fe640 10 monclient: dump: 2024-04-26T00:41:25.412 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:25.412 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:25.413 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:25.413 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:25.413 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:25.413 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:25.413 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:25.413 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:25.413 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:25.413 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:25.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.401+0000 7fda937fe640 10 monclient: _finish_auth 0 2024-04-26T00:41:25.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.401+0000 7fda937fe640 10 monclient: _check_auth_tickets 2024-04-26T00:41:25.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.401+0000 7fda937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:41:25.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.401+0000 7fda937fe640 1 -- 172.21.15.37:0/928340715 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda840b1040 con 0x7fda9c0a0260 2024-04-26T00:41:25.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.401+0000 7fdaa17cb640 5 monclient: authenticate success, global_id 254195 2024-04-26T00:41:25.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.401+0000 7fda937fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:41:25.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.401+0000 7fda937fe640 1 -- 172.21.15.37:0/928340715 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda840b0070 con 0x7fda9c0a0260 2024-04-26T00:41:25.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.401+0000 7fdaa17cb640 10 monclient: _renew_subs 2024-04-26T00:41:25.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.401+0000 7fdaa17cb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:41:25.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.401+0000 7fdaa17cb640 1 -- 172.21.15.37:0/928340715 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda9c071ee0 con 0x7fda9c0a0260 2024-04-26T00:41:25.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.401+0000 7fda937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:25.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.401+0000 7fda937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:41:25.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.401+0000 7fda98ff9640 4 set_mon_vals no callback set 2024-04-26T00:41:25.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.401+0000 7fda937fe640 10 monclient: dump: 2024-04-26T00:41:25.415 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:25.415 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:25.415 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:25.415 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:25.415 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:25.415 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:25.415 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:25.415 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:25.416 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:25.416 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:25.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.401+0000 7fdaa17cb640 10 monclient: _renew_subs 2024-04-26T00:41:25.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.401+0000 7fdaa17cb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:41:25.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.401+0000 7fdaa17cb640 1 -- 172.21.15.37:0/928340715 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fda9c11de60 con 0x7fda9c0a0260 2024-04-26T00:41:25.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.401+0000 7fdaa17cb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:25.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.401+0000 7fdaa17cb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:25.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.401+0000 7fdaa17cb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:41:25.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.401+0000 7fdaa17cb640 1 -- 172.21.15.37:0/928340715 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda9c09f8d0 con 0x7fda9c0a0260 2024-04-26T00:41:25.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.401+0000 7fda937fe640 1 -- 172.21.15.37:0/928340715 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fda840bd070 con 0x7fda9c0a0260 2024-04-26T00:41:25.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.401+0000 7fda937fe640 1 --2- 172.21.15.37:0/928340715 >> v2:172.21.15.45:6812/2379918623 conn(0x7fda7c035f00 0x7fda7c0383c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:25.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.401+0000 7fda937fe640 1 -- 172.21.15.37:0/928340715 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(851..851 src has 705..851) ==== 4023+0+0 (secure 0 0 0) 0x7fda840e1c60 con 0x7fda9c0a0260 2024-04-26T00:41:25.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.401+0000 7fda9a7fc640 1 --2- 172.21.15.37:0/928340715 >> v2:172.21.15.45:6812/2379918623 conn(0x7fda7c035f00 0x7fda7c0383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:25.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.401+0000 7fda9a7fc640 10 monclient: get_auth_request con 0x7fda7c035f00 auth_method 0 2024-04-26T00:41:25.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.405+0000 7fda9a7fc640 1 --2- 172.21.15.37:0/928340715 >> v2:172.21.15.45:6812/2379918623 conn(0x7fda7c035f00 0x7fda7c0383c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fda9c11d700 tx=0x7fda940098d0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:25.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.405+0000 7fda937fe640 1 -- 172.21.15.37:0/928340715 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fda9c0a1130 con 0x7fda9c0a0260 2024-04-26T00:41:25.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.405+0000 7fda937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:25.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.405+0000 7fda937fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:41:25.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.629+0000 7fdaa17cb640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:41:25.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.629+0000 7fdaa17cb640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:41:25.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.629+0000 7fdaa17cb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:41:25.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.629+0000 7fdaa17cb640 1 -- 172.21.15.37:0/928340715 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fda9c066f40 con 0x7fda9c0a0260 2024-04-26T00:41:25.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.629+0000 7fda937fe640 1 -- 172.21.15.37:0/928340715 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7fda840b4360 con 0x7fda9c0a0260 2024-04-26T00:41:25.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.629+0000 7fda937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:41:25.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.629+0000 7fda937fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:41:25.633 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":154,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":29,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:41:25.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.633+0000 7fdaa17cb640 1 -- 172.21.15.37:0/928340715 >> v2:172.21.15.45:6812/2379918623 conn(0x7fda7c035f00 msgr2=0x7fda7c0383c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:25.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.633+0000 7fdaa17cb640 1 --2- 172.21.15.37:0/928340715 >> v2:172.21.15.45:6812/2379918623 conn(0x7fda7c035f00 0x7fda7c0383c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fda9c11d700 tx=0x7fda940098d0 comp rx=0 tx=0).stop 2024-04-26T00:41:25.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.633+0000 7fdaa17cb640 10 monclient: shutdown 2024-04-26T00:41:25.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.633+0000 7fdaa17cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:41:25.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.633+0000 7fdaa17cb640 1 -- 172.21.15.37:0/928340715 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fda9c0a0260 msgr2=0x7fda9c131d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:25.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.633+0000 7fdaa17cb640 1 --2- 172.21.15.37:0/928340715 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fda9c0a0260 0x7fda9c131d50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fda840ad000 tx=0x7fda84018fa0 comp rx=0 tx=0).stop 2024-04-26T00:41:25.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.633+0000 7fdaa17cb640 1 -- 172.21.15.37:0/928340715 shutdown_connections 2024-04-26T00:41:25.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.633+0000 7fdaa17cb640 1 --2- 172.21.15.37:0/928340715 >> v2:172.21.15.45:6812/2379918623 conn(0x7fda7c035f00 0x7fda7c0383c0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:25.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.633+0000 7fdaa17cb640 1 --2- 172.21.15.37:0/928340715 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fda9c135490 0x7fda9c11b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:25.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.633+0000 7fdaa17cb640 1 --2- 172.21.15.37:0/928340715 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda9c134b30 0x7fda9c132290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:25.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.633+0000 7fdaa17cb640 1 --2- 172.21.15.37:0/928340715 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fda9c0a0260 0x7fda9c131d50 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:25.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.633+0000 7fdaa17cb640 1 -- 172.21.15.37:0/928340715 >> 172.21.15.37:0/928340715 conn(0x7fda9c0a1ad0 msgr2=0x7fda9c0a3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:41:25.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.633+0000 7fdaa17cb640 1 -- 172.21.15.37:0/928340715 shutdown_connections 2024-04-26T00:41:25.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:25.633+0000 7fdaa17cb640 1 -- 172.21.15.37:0/928340715 wait complete. 2024-04-26T00:41:26.005 INFO:tasks.rados.rados.0.smithi045.stdout:3778: done (3 left) 2024-04-26T00:41:26.006 INFO:tasks.rados.rados.0.smithi045.stdout:3779: done (2 left) 2024-04-26T00:41:26.006 INFO:tasks.rados.rados.0.smithi045.stdout:3780: done (1 left) 2024-04-26T00:41:26.006 INFO:tasks.rados.rados.0.smithi045.stdout:3781: done (0 left) 2024-04-26T00:41:26.006 INFO:tasks.rados.rados.0.smithi045.stdout:3782: write oid 19 current snap is 418 2024-04-26T00:41:26.006 INFO:tasks.rados.rados.0.smithi045.stdout:3782: seq_num 1349 ranges {672643=505382,1890209=500796} 2024-04-26T00:41:26.024 INFO:tasks.rados.rados.0.smithi045.stdout:3782: writing smithi04526579-19 from 672643 to 1178025 tid 1 2024-04-26T00:41:26.044 INFO:tasks.rados.rados.0.smithi045.stdout:3782: writing smithi04526579-19 from 1890209 to 2391005 tid 2 2024-04-26T00:41:26.045 INFO:tasks.rados.rados.0.smithi045.stdout:3783: snap_create 2024-04-26T00:41:26.045 INFO:tasks.rados.rados.0.smithi045.stdout:3782: finishing write tid 1 to smithi04526579-19 2024-04-26T00:41:26.055 INFO:tasks.rados.rados.0.smithi045.stdout:3782: finishing write tid 2 to smithi04526579-19 2024-04-26T00:41:26.055 INFO:tasks.rados.rados.0.smithi045.stdout:3782: finishing write tid 3 to smithi04526579-19 2024-04-26T00:41:26.055 INFO:tasks.rados.rados.0.smithi045.stdout:3782: finishing write tid 4 to smithi04526579-19 2024-04-26T00:41:26.055 INFO:tasks.rados.rados.0.smithi045.stdout:3782: oid 19 updating version 0 to 2529 2024-04-26T00:41:26.055 INFO:tasks.rados.rados.0.smithi045.stdout:3782: oid 19 updating version 2529 to 2530 2024-04-26T00:41:26.056 INFO:tasks.rados.rados.0.smithi045.stdout:3782: oid 19 version 2530 is already newer than 2528 2024-04-26T00:41:26.056 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 2530 (ObjNum 1349 snap 418 seq_num 1349) dirty exists 2024-04-26T00:41:26.056 INFO:tasks.rados.rados.0.smithi045.stdout:3782: left oid 19 (ObjNum 1349 snap 418 seq_num 1349) 2024-04-26T00:41:26.266 INFO:tasks.rados.rados.0.smithi045.stdout:3782: done (1 left) 2024-04-26T00:41:26.266 INFO:tasks.rados.rados.0.smithi045.stdout:3783: done (0 left) 2024-04-26T00:41:26.266 INFO:tasks.rados.rados.0.smithi045.stdout:3784: write (excl) oid 48 current snap is 419 2024-04-26T00:41:26.266 INFO:tasks.rados.rados.0.smithi045.stdout:3784: seq_num 1350 ranges {689450=580631,1948691=740759} 2024-04-26T00:41:26.285 INFO:tasks.rados.rados.0.smithi045.stdout:3784: writing smithi04526579-48 from 689450 to 1270081 tid 1 2024-04-26T00:41:26.310 INFO:tasks.rados.rados.0.smithi045.stdout:3784: writing smithi04526579-48 from 1948691 to 2689450 tid 2 2024-04-26T00:41:26.310 INFO:tasks.rados.rados.0.smithi045.stdout:3785: copy_from oid 36 from oid 45 current snap is 419 2024-04-26T00:41:26.311 INFO:tasks.rados.rados.0.smithi045.stdout:3784: finishing write tid 1 to smithi04526579-48 2024-04-26T00:41:26.311 INFO:tasks.rados.rados.0.smithi045.stdout:3786: copy_from oid 19 from oid 2 current snap is 419 2024-04-26T00:41:26.311 INFO:tasks.rados.rados.0.smithi045.stdout:3787: write oid 33 current snap is 419 2024-04-26T00:41:26.311 INFO:tasks.rados.rados.0.smithi045.stdout:3787: seq_num 1351 ranges {706257=655880,1890820=504465} 2024-04-26T00:41:26.327 INFO:tasks.rados.rados.0.smithi045.stdout:3787: writing smithi04526579-33 from 706257 to 1362137 tid 1 2024-04-26T00:41:26.338 INFO:tasks.rados.rados.0.smithi045.stdout:3787: writing smithi04526579-33 from 1890820 to 2395285 tid 2 2024-04-26T00:41:26.339 INFO:tasks.rados.rados.0.smithi045.stdout:3788: write oid 5 current snap is 419 2024-04-26T00:41:26.339 INFO:tasks.rados.rados.0.smithi045.stdout:3788: seq_num 1352 ranges {723064=731129,1949302=764476} 2024-04-26T00:41:26.351 INFO:tasks.rados.rados.0.smithi045.stdout:3788: writing smithi04526579-5 from 723064 to 1454193 tid 1 2024-04-26T00:41:26.361 INFO:tasks.rados.rados.0.smithi045.stdout:3788: writing smithi04526579-5 from 1949302 to 2713778 tid 2 2024-04-26T00:41:26.361 INFO:tasks.rados.rados.0.smithi045.stdout:3789: read oid 43 snap -1 2024-04-26T00:41:26.362 INFO:tasks.rados.rados.0.smithi045.stdout:3789: expect deleted 2024-04-26T00:41:26.362 INFO:tasks.rados.rados.0.smithi045.stdout:3790: write (excl) oid 35 current snap is 419 2024-04-26T00:41:26.362 INFO:tasks.rados.rados.0.smithi045.stdout:3790: seq_num 1353 ranges {739871=406378,1607784=508134} 2024-04-26T00:41:26.370 INFO:tasks.rados.rados.0.smithi045.stdout:3790: writing smithi04526579-35 from 739871 to 1146249 tid 1 2024-04-26T00:41:26.377 INFO:tasks.rados.rados.0.smithi045.stdout:3790: writing smithi04526579-35 from 1607784 to 2115918 tid 2 2024-04-26T00:41:26.377 INFO:tasks.rados.rados.0.smithi045.stdout:3791: write oid 38 current snap is 419 2024-04-26T00:41:26.377 INFO:tasks.rados.rados.0.smithi045.stdout:3791: seq_num 1354 ranges {756678=597980,1782619=768145} 2024-04-26T00:41:26.385 INFO:tasks.rados.rados.0.smithi045.stdout:3791: writing smithi04526579-38 from 756678 to 1354658 tid 1 2024-04-26T00:41:26.393 INFO:tasks.rados.rados.0.smithi045.stdout:3791: writing smithi04526579-38 from 1782619 to 2550764 tid 2 2024-04-26T00:41:26.393 INFO:tasks.rados.rados.0.smithi045.stdout:3792: rollback oid 10 current snap is 419 2024-04-26T00:41:26.393 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 10 to 393 2024-04-26T00:41:26.393 INFO:tasks.rados.rados.0.smithi045.stdout:3785: finishing copy_from to smithi04526579-36 2024-04-26T00:41:26.393 INFO:tasks.rados.rados.0.smithi045.stdout:3785: got expected ENOENT (src dne) 2024-04-26T00:41:26.393 INFO:tasks.rados.rados.0.smithi045.stdout:3785: finishing copy_from racing read to smithi04526579-36 2024-04-26T00:41:26.393 INFO:tasks.rados.rados.0.smithi045.stdout:3784: finishing write tid 2 to smithi04526579-48 2024-04-26T00:41:26.394 INFO:tasks.rados.rados.0.smithi045.stdout:3784: finishing write tid 3 to smithi04526579-48 2024-04-26T00:41:26.394 INFO:tasks.rados.rados.0.smithi045.stdout:3784: finishing write tid 4 to smithi04526579-48 2024-04-26T00:41:26.394 INFO:tasks.rados.rados.0.smithi045.stdout:3784: oid 48 updating version 0 to 2519 2024-04-26T00:41:26.394 INFO:tasks.rados.rados.0.smithi045.stdout:3784: oid 48 version 2519 is already newer than 2518 2024-04-26T00:41:26.394 INFO:tasks.rados.rados.0.smithi045.stdout:3784: oid 48 updating version 2519 to 2520 2024-04-26T00:41:26.394 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 2520 (ObjNum 1350 snap 419 seq_num 1350) dirty exists 2024-04-26T00:41:26.394 INFO:tasks.rados.rados.0.smithi045.stdout:3784: left oid 48 (ObjNum 1350 snap 419 seq_num 1350) 2024-04-26T00:41:26.394 INFO:tasks.rados.rados.0.smithi045.stdout:3786: finishing copy_from racing read to smithi04526579-19 2024-04-26T00:41:26.394 INFO:tasks.rados.rados.0.smithi045.stdout:3784: done (8 left) 2024-04-26T00:41:26.394 INFO:tasks.rados.rados.0.smithi045.stdout:3785: done (7 left) 2024-04-26T00:41:26.394 INFO:tasks.rados.rados.0.smithi045.stdout:3793: snap_remove snap 385 2024-04-26T00:41:27.273 INFO:tasks.rados.rados.0.smithi045.stdout:3793: done (7 left) 2024-04-26T00:41:27.273 INFO:tasks.rados.rados.0.smithi045.stdout:3794: read oid 7 snap -1 2024-04-26T00:41:27.273 INFO:tasks.rados.rados.0.smithi045.stdout:3794: expect deleted 2024-04-26T00:41:27.273 INFO:tasks.rados.rados.0.smithi045.stdout:3795: write oid 3 current snap is 419 2024-04-26T00:41:27.273 INFO:tasks.rados.rados.0.smithi045.stdout:3795: seq_num 1355 ranges {773485=673229,2124748=511803} 2024-04-26T00:41:27.282 INFO:tasks.rados.rados.0.smithi045.stdout:3795: writing smithi04526579-3 from 773485 to 1446714 tid 1 2024-04-26T00:41:27.291 INFO:tasks.rados.rados.0.smithi045.stdout:3795: writing smithi04526579-3 from 2124748 to 2636551 tid 2 2024-04-26T00:41:27.291 INFO:tasks.rados.rados.0.smithi045.stdout:3796: read oid 23 snap -1 2024-04-26T00:41:27.292 INFO:tasks.rados.rados.0.smithi045.stdout:3796: expect deleted 2024-04-26T00:41:27.293 INFO:tasks.rados.rados.0.smithi045.stdout:3797: snap_create 2024-04-26T00:41:27.293 INFO:tasks.rados.rados.0.smithi045.stdout:3786: finishing copy_from to smithi04526579-19 2024-04-26T00:41:27.293 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 19 v 2539 (ObjNum 1347 snap 416 seq_num 1347) dirty exists 2024-04-26T00:41:27.293 INFO:tasks.rados.rados.0.smithi045.stdout:3787: finishing write tid 1 to smithi04526579-33 2024-04-26T00:41:27.293 INFO:tasks.rados.rados.0.smithi045.stdout:3787: finishing write tid 2 to smithi04526579-33 2024-04-26T00:41:27.293 INFO:tasks.rados.rados.0.smithi045.stdout:3787: finishing write tid 3 to smithi04526579-33 2024-04-26T00:41:27.293 INFO:tasks.rados.rados.0.smithi045.stdout:3787: finishing write tid 4 to smithi04526579-33 2024-04-26T00:41:27.293 INFO:tasks.rados.rados.0.smithi045.stdout:3787: oid 33 updating version 0 to 2844 2024-04-26T00:41:27.293 INFO:tasks.rados.rados.0.smithi045.stdout:3787: oid 33 updating version 2844 to 2845 2024-04-26T00:41:27.293 INFO:tasks.rados.rados.0.smithi045.stdout:3787: oid 33 version 2845 is already newer than 2843 2024-04-26T00:41:27.293 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 2845 (ObjNum 1351 snap 419 seq_num 1351) dirty exists 2024-04-26T00:41:27.293 INFO:tasks.rados.rados.0.smithi045.stdout:3787: left oid 33 (ObjNum 1351 snap 419 seq_num 1351) 2024-04-26T00:41:27.293 INFO:tasks.rados.rados.0.smithi045.stdout:3788: finishing write tid 1 to smithi04526579-5 2024-04-26T00:41:27.294 INFO:tasks.rados.rados.0.smithi045.stdout:3788: finishing write tid 2 to smithi04526579-5 2024-04-26T00:41:27.294 INFO:tasks.rados.rados.0.smithi045.stdout:3788: finishing write tid 3 to smithi04526579-5 2024-04-26T00:41:27.294 INFO:tasks.rados.rados.0.smithi045.stdout:3788: finishing write tid 4 to smithi04526579-5 2024-04-26T00:41:27.294 INFO:tasks.rados.rados.0.smithi045.stdout:3788: oid 5 updating version 0 to 2849 2024-04-26T00:41:27.294 INFO:tasks.rados.rados.0.smithi045.stdout:3788: oid 5 version 2849 is already newer than 2848 2024-04-26T00:41:27.294 INFO:tasks.rados.rados.0.smithi045.stdout:3788: oid 5 version 2849 is already newer than 2847 2024-04-26T00:41:27.294 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 2849 (ObjNum 1352 snap 419 seq_num 1352) dirty exists 2024-04-26T00:41:27.294 INFO:tasks.rados.rados.0.smithi045.stdout:3788: left oid 5 (ObjNum 1352 snap 419 seq_num 1352) 2024-04-26T00:41:27.294 INFO:tasks.rados.rados.0.smithi045.stdout:3790: finishing write tid 1 to smithi04526579-35 2024-04-26T00:41:27.294 INFO:tasks.rados.rados.0.smithi045.stdout:3790: finishing write tid 2 to smithi04526579-35 2024-04-26T00:41:27.294 INFO:tasks.rados.rados.0.smithi045.stdout:3790: finishing write tid 3 to smithi04526579-35 2024-04-26T00:41:27.294 INFO:tasks.rados.rados.0.smithi045.stdout:3790: finishing write tid 4 to smithi04526579-35 2024-04-26T00:41:27.294 INFO:tasks.rados.rados.0.smithi045.stdout:3790: oid 35 updating version 0 to 2540 2024-04-26T00:41:27.294 INFO:tasks.rados.rados.0.smithi045.stdout:3790: oid 35 updating version 2540 to 2542 2024-04-26T00:41:27.295 INFO:tasks.rados.rados.0.smithi045.stdout:3790: oid 35 version 2542 is already newer than 2541 2024-04-26T00:41:27.295 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 2542 (ObjNum 1353 snap 419 seq_num 1353) dirty exists 2024-04-26T00:41:27.295 INFO:tasks.rados.rados.0.smithi045.stdout:3790: left oid 35 (ObjNum 1353 snap 419 seq_num 1353) 2024-04-26T00:41:27.295 INFO:tasks.rados.rados.0.smithi045.stdout:3791: finishing write tid 1 to smithi04526579-38 2024-04-26T00:41:27.295 INFO:tasks.rados.rados.0.smithi045.stdout:3791: finishing write tid 2 to smithi04526579-38 2024-04-26T00:41:27.295 INFO:tasks.rados.rados.0.smithi045.stdout:3791: finishing write tid 3 to smithi04526579-38 2024-04-26T00:41:27.295 INFO:tasks.rados.rados.0.smithi045.stdout:3791: finishing write tid 4 to smithi04526579-38 2024-04-26T00:41:27.295 INFO:tasks.rados.rados.0.smithi045.stdout:3791: oid 38 updating version 0 to 2411 2024-04-26T00:41:27.295 INFO:tasks.rados.rados.0.smithi045.stdout:3791: oid 38 updating version 2411 to 2412 2024-04-26T00:41:27.295 INFO:tasks.rados.rados.0.smithi045.stdout:3791: oid 38 updating version 2412 to 2413 2024-04-26T00:41:27.295 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 38 v 2413 (ObjNum 1354 snap 419 seq_num 1354) dirty exists 2024-04-26T00:41:27.295 INFO:tasks.rados.rados.0.smithi045.stdout:3791: left oid 38 (ObjNum 1354 snap 419 seq_num 1354) 2024-04-26T00:41:27.295 INFO:tasks.rados.rados.0.smithi045.stdout:3792: finishing rollback tid 0 to smithi04526579-10 2024-04-26T00:41:27.295 INFO:tasks.rados.rados.0.smithi045.stdout:3792: finishing rollback tid 1 to smithi04526579-10 2024-04-26T00:41:27.296 INFO:tasks.rados.rados.0.smithi045.stdout:3792: finishing rollback tid 2 to smithi04526579-10 2024-04-26T00:41:27.296 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 2638 (ObjNum 1112 snap 331 seq_num 1112) dirty exists 2024-04-26T00:41:27.310 INFO:tasks.rados.rados.0.smithi045.stdout:3795: finishing write tid 1 to smithi04526579-3 2024-04-26T00:41:27.314 INFO:tasks.rados.rados.0.smithi045.stdout:3795: finishing write tid 2 to smithi04526579-3 2024-04-26T00:41:27.319 INFO:tasks.rados.rados.0.smithi045.stdout:3795: finishing write tid 3 to smithi04526579-3 2024-04-26T00:41:27.319 INFO:tasks.rados.rados.0.smithi045.stdout:3795: finishing write tid 4 to smithi04526579-3 2024-04-26T00:41:27.319 INFO:tasks.rados.rados.0.smithi045.stdout:3795: oid 3 updating version 0 to 2523 2024-04-26T00:41:27.320 INFO:tasks.rados.rados.0.smithi045.stdout:3795: oid 3 version 2523 is already newer than 2522 2024-04-26T00:41:27.320 INFO:tasks.rados.rados.0.smithi045.stdout:3795: oid 3 version 2523 is already newer than 2521 2024-04-26T00:41:27.320 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 2523 (ObjNum 1355 snap 419 seq_num 1355) dirty exists 2024-04-26T00:41:27.320 INFO:tasks.rados.rados.0.smithi045.stdout:3795: left oid 3 (ObjNum 1355 snap 419 seq_num 1355) 2024-04-26T00:41:28.281 INFO:tasks.rados.rados.0.smithi045.stdout:3786: done (10 left) 2024-04-26T00:41:28.281 INFO:tasks.rados.rados.0.smithi045.stdout:3787: done (9 left) 2024-04-26T00:41:28.281 INFO:tasks.rados.rados.0.smithi045.stdout:3788: done (8 left) 2024-04-26T00:41:28.281 INFO:tasks.rados.rados.0.smithi045.stdout:3789: done (7 left) 2024-04-26T00:41:28.281 INFO:tasks.rados.rados.0.smithi045.stdout:3790: done (6 left) 2024-04-26T00:41:28.281 INFO:tasks.rados.rados.0.smithi045.stdout:3791: done (5 left) 2024-04-26T00:41:28.281 INFO:tasks.rados.rados.0.smithi045.stdout:3792: done (4 left) 2024-04-26T00:41:28.281 INFO:tasks.rados.rados.0.smithi045.stdout:3794: done (3 left) 2024-04-26T00:41:28.281 INFO:tasks.rados.rados.0.smithi045.stdout:3795: done (2 left) 2024-04-26T00:41:28.281 INFO:tasks.rados.rados.0.smithi045.stdout:3796: done (1 left) 2024-04-26T00:41:28.281 INFO:tasks.rados.rados.0.smithi045.stdout:3797: done (0 left) 2024-04-26T00:41:28.282 INFO:tasks.rados.rados.0.smithi045.stdout:3798: write (excl) oid 49 current snap is 420 2024-04-26T00:41:28.282 INFO:tasks.rados.rados.0.smithi045.stdout:3798: seq_num 1356 ranges {790292=748478,2183230=607062} 2024-04-26T00:41:28.294 INFO:tasks.rados.rados.0.smithi045.stdout:3798: writing smithi04526579-49 from 790292 to 1538770 tid 1 2024-04-26T00:41:28.304 INFO:tasks.rados.rados.0.smithi045.stdout:3798: writing smithi04526579-49 from 2183230 to 2790292 tid 2 2024-04-26T00:41:28.304 INFO:tasks.rados.rados.0.smithi045.stdout:3799: read oid 38 snap -1 2024-04-26T00:41:28.304 INFO:tasks.rados.rados.0.smithi045.stdout:3799: expect (ObjNum 1354 snap 419 seq_num 1354) 2024-04-26T00:41:28.304 INFO:tasks.rados.rados.0.smithi045.stdout:3800: read oid 45 snap 413 2024-04-26T00:41:28.304 INFO:tasks.rados.rados.0.smithi045.stdout:3800: expect deleted 2024-04-26T00:41:28.305 INFO:tasks.rados.rados.0.smithi045.stdout:3801: write oid 4 current snap is 420 2024-04-26T00:41:28.305 INFO:tasks.rados.rados.0.smithi045.stdout:3801: seq_num 1357 ranges {407099=423727,1441712=515472,2582060=225039} 2024-04-26T00:41:28.310 INFO:tasks.rados.rados.0.smithi045.stdout:3801: writing smithi04526579-4 from 407099 to 830826 tid 1 2024-04-26T00:41:28.319 INFO:tasks.rados.rados.0.smithi045.stdout:3801: writing smithi04526579-4 from 1441712 to 1957184 tid 2 2024-04-26T00:41:28.326 INFO:tasks.rados.rados.0.smithi045.stdout:3801: writing smithi04526579-4 from 2582060 to 2807099 tid 3 2024-04-26T00:41:28.326 INFO:tasks.rados.rados.0.smithi045.stdout:3802: read oid 10 snap -1 2024-04-26T00:41:28.326 INFO:tasks.rados.rados.0.smithi045.stdout:3802: expect (ObjNum 1112 snap 331 seq_num 1112) 2024-04-26T00:41:28.326 INFO:tasks.rados.rados.0.smithi045.stdout:3803: copy_from oid 48 from oid 25 current snap is 420 2024-04-26T00:41:28.326 INFO:tasks.rados.rados.0.smithi045.stdout:3798: finishing write tid 1 to smithi04526579-49 2024-04-26T00:41:28.326 INFO:tasks.rados.rados.0.smithi045.stdout:3798: finishing write tid 2 to smithi04526579-49 2024-04-26T00:41:28.326 INFO:tasks.rados.rados.0.smithi045.stdout:3798: finishing write tid 3 to smithi04526579-49 2024-04-26T00:41:28.327 INFO:tasks.rados.rados.0.smithi045.stdout:3800: done (5 left) 2024-04-26T00:41:28.327 INFO:tasks.rados.rados.0.smithi045.stdout:3804: delete oid 39 current snap is 420 2024-04-26T00:41:28.327 INFO:tasks.rados.rados.0.smithi045.stdout:3798: finishing write tid 4 to smithi04526579-49 2024-04-26T00:41:28.327 INFO:tasks.rados.rados.0.smithi045.stdout:3798: oid 49 updating version 0 to 2525 2024-04-26T00:41:28.327 INFO:tasks.rados.rados.0.smithi045.stdout:3798: oid 49 updating version 2525 to 2526 2024-04-26T00:41:28.327 INFO:tasks.rados.rados.0.smithi045.stdout:3798: oid 49 version 2526 is already newer than 2524 2024-04-26T00:41:28.327 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 2526 (ObjNum 1356 snap 420 seq_num 1356) dirty exists 2024-04-26T00:41:28.327 INFO:tasks.rados.rados.0.smithi045.stdout:3798: left oid 49 (ObjNum 1356 snap 420 seq_num 1356) 2024-04-26T00:41:28.327 INFO:tasks.rados.rados.0.smithi045.stdout:3799: expect (ObjNum 1354 snap 419 seq_num 1354) 2024-04-26T00:41:28.457 INFO:tasks.rados.rados.0.smithi045.stdout:3801: finishing write tid 1 to smithi04526579-4 2024-04-26T00:41:28.457 INFO:tasks.rados.rados.0.smithi045.stdout:3801: finishing write tid 2 to smithi04526579-4 2024-04-26T00:41:28.457 INFO:tasks.rados.rados.0.smithi045.stdout:3801: finishing write tid 3 to smithi04526579-4 2024-04-26T00:41:28.457 INFO:tasks.rados.rados.0.smithi045.stdout:3798: done (5 left) 2024-04-26T00:41:28.458 INFO:tasks.rados.rados.0.smithi045.stdout:3799: done (4 left) 2024-04-26T00:41:28.458 INFO:tasks.rados.rados.0.smithi045.stdout:3804: done (3 left) 2024-04-26T00:41:28.458 INFO:tasks.rados.rados.0.smithi045.stdout:3805: delete oid 20 current snap is 420 2024-04-26T00:41:28.458 INFO:tasks.rados.rados.0.smithi045.stdout:3802: expect (ObjNum 1112 snap 331 seq_num 1112) 2024-04-26T00:41:28.615 INFO:tasks.rados.rados.0.smithi045.stdout:3801: finishing write tid 4 to smithi04526579-4 2024-04-26T00:41:28.615 INFO:tasks.rados.rados.0.smithi045.stdout:3801: finishing write tid 5 to smithi04526579-4 2024-04-26T00:41:28.615 INFO:tasks.rados.rados.0.smithi045.stdout:3801: oid 4 updating version 0 to 2542 2024-04-26T00:41:28.615 INFO:tasks.rados.rados.0.smithi045.stdout:3801: oid 4 version 2542 is already newer than 2541 2024-04-26T00:41:28.615 INFO:tasks.rados.rados.0.smithi045.stdout:3801: oid 4 version 2542 is already newer than 2540 2024-04-26T00:41:28.615 INFO:tasks.rados.rados.0.smithi045.stdout:3801: oid 4 updating version 2542 to 2543 2024-04-26T00:41:28.615 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 2543 (ObjNum 1357 snap 420 seq_num 1357) dirty exists 2024-04-26T00:41:28.615 INFO:tasks.rados.rados.0.smithi045.stdout:3801: left oid 4 (ObjNum 1357 snap 420 seq_num 1357) 2024-04-26T00:41:28.615 INFO:tasks.rados.rados.0.smithi045.stdout:3803: finishing copy_from racing read to smithi04526579-48 2024-04-26T00:41:28.615 INFO:tasks.rados.rados.0.smithi045.stdout:3803: finishing copy_from to smithi04526579-48 2024-04-26T00:41:28.615 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 2823 (ObjNum 1327 snap 410 seq_num 1327) dirty exists 2024-04-26T00:41:28.616 INFO:tasks.rados.rados.0.smithi045.stdout:3801: done (3 left) 2024-04-26T00:41:28.616 INFO:tasks.rados.rados.0.smithi045.stdout:3802: done (2 left) 2024-04-26T00:41:28.616 INFO:tasks.rados.rados.0.smithi045.stdout:3803: done (1 left) 2024-04-26T00:41:28.616 INFO:tasks.rados.rados.0.smithi045.stdout:3805: done (0 left) 2024-04-26T00:41:28.616 INFO:tasks.rados.rados.0.smithi045.stdout:3806: write (excl) oid 21 current snap is 420 2024-04-26T00:41:28.616 INFO:tasks.rados.rados.0.smithi045.stdout:3806: seq_num 1360 ranges {457520=649474,1900805=662799} 2024-04-26T00:41:28.621 INFO:tasks.rados.rados.0.smithi045.stdout:3806: writing smithi04526579-21 from 457520 to 1106994 tid 1 2024-04-26T00:41:28.629 INFO:tasks.rados.rados.0.smithi045.stdout:3806: writing smithi04526579-21 from 1900805 to 2563604 tid 2 2024-04-26T00:41:28.629 INFO:tasks.rados.rados.0.smithi045.stdout:3807: snap_create 2024-04-26T00:41:28.637 INFO:tasks.rados.rados.0.smithi045.stdout:3806: finishing write tid 1 to smithi04526579-21 2024-04-26T00:41:28.637 INFO:tasks.rados.rados.0.smithi045.stdout:3806: finishing write tid 2 to smithi04526579-21 2024-04-26T00:41:28.638 INFO:tasks.rados.rados.0.smithi045.stdout:3806: finishing write tid 3 to smithi04526579-21 2024-04-26T00:41:28.639 INFO:tasks.rados.rados.0.smithi045.stdout:3806: finishing write tid 4 to smithi04526579-21 2024-04-26T00:41:28.639 INFO:tasks.rados.rados.0.smithi045.stdout:3806: oid 21 updating version 0 to 2487 2024-04-26T00:41:28.639 INFO:tasks.rados.rados.0.smithi045.stdout:3806: oid 21 updating version 2487 to 2488 2024-04-26T00:41:28.639 INFO:tasks.rados.rados.0.smithi045.stdout:3806: oid 21 version 2488 is already newer than 2486 2024-04-26T00:41:28.639 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 2488 (ObjNum 1360 snap 420 seq_num 1360) dirty exists 2024-04-26T00:41:28.639 INFO:tasks.rados.rados.0.smithi045.stdout:3806: left oid 21 (ObjNum 1360 snap 420 seq_num 1360) 2024-04-26T00:41:28.654 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:41:28.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.781+0000 7f399c85f640 10 monclient: get_monmap_and_config 2024-04-26T00:41:28.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.781+0000 7f399c85f640 10 monclient: build_initial_monmap 2024-04-26T00:41:28.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.781+0000 7f399c85f640 10 monclient: monmap: 2024-04-26T00:41:28.786 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:41:28.786 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:28.786 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:41:28.787680+0000 2024-04-26T00:41:28.786 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:41:28.787680+0000 2024-04-26T00:41:28.786 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:41:28.786 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:28.786 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:41:28.786 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:41:28.786 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:41:28.786 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:41:28.786 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:41:28.786 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:28.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.785+0000 7f399c85f640 1 Processor -- start 2024-04-26T00:41:28.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.785+0000 7f399c85f640 1 -- start start 2024-04-26T00:41:28.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.785+0000 7f399c85f640 10 monclient: init 2024-04-26T00:41:28.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.785+0000 7f399c85f640 10 monclient: _reopen_session rank -1 2024-04-26T00:41:28.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.785+0000 7f399c85f640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-04-26T00:41:28.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.785+0000 7f399c85f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f39980724d0 addr v1:172.21.15.37:6789/0 2024-04-26T00:41:28.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.785+0000 7f399c85f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3998133990 addr v1:172.21.15.45:6789/0 2024-04-26T00:41:28.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.785+0000 7f399c85f640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3998133da0 0x7f399813f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:28.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.785+0000 7f399c85f640 10 monclient(hunting): picked mon.noname-c con 0x7f3998133da0 addr v2:172.21.15.45:3300/0 2024-04-26T00:41:28.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.785+0000 7f399c85f640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f399806e9e0 con 0x7f39980724d0 2024-04-26T00:41:28.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.785+0000 7f399c85f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f399806efa0 con 0x7f3998133990 2024-04-26T00:41:28.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.785+0000 7f399c85f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:28.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.785+0000 7f399c85f640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f3998072fc0 con 0x7f3998133da0 2024-04-26T00:41:28.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.785+0000 7f399c85f640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:28.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.785+0000 7f399c85f640 10 monclient(hunting): authenticate will time out at 2270.093704s 2024-04-26T00:41:28.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.785+0000 7f399659c640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f39980724d0 0x7f39980728b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:40144/0 (socket says 172.21.15.37:40144) 2024-04-26T00:41:28.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.785+0000 7f399659c640 1 -- 172.21.15.37:0/2196430930 learned_addr learned my addr 172.21.15.37:0/2196430930 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:41:28.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.785+0000 7f3996d9d640 1 --2- 172.21.15.37:0/2196430930 >> v2:172.21.15.45:3300/0 conn(0x7f3998133da0 0x7f399813f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:28.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.785+0000 7f3996d9d640 10 monclient(hunting): get_auth_request con 0x7f3998133da0 auth_method 0 2024-04-26T00:41:28.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.785+0000 7f3996d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:28.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.785+0000 7f3996d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:28.789 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.785+0000 7f3996d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:28.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.785+0000 7f3996d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:28.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.785+0000 7f3996d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:28.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.785+0000 7f3996d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:28.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.785+0000 7f3996d9d640 10 monclient(hunting): handle_auth_done global_id 224443 payload 274 2024-04-26T00:41:28.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.785+0000 7f3996d9d640 1 -- 172.21.15.37:0/2196430930 >> v1:172.21.15.45:6789/0 conn(0x7f3998133990 legacy=0x7f399813c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:41:28.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.785+0000 7f3996d9d640 1 -- 172.21.15.37:0/2196430930 >> v1:172.21.15.37:6789/0 conn(0x7f39980724d0 legacy=0x7f39980728b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:41:28.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.785+0000 7f3996d9d640 10 monclient: _finish_hunting 0 2024-04-26T00:41:28.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.785+0000 7f3996d9d640 1 monclient: found mon.noname-c 2024-04-26T00:41:28.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.785+0000 7f3996d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:28.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.785+0000 7f3996d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:41:28.790 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.785+0000 7f3996d9d640 1 -- 172.21.15.37:0/2196430930 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f399809f9e0 con 0x7f3998133da0 2024-04-26T00:41:28.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.789+0000 7f3996d9d640 1 -- 172.21.15.37:0/2196430930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3998133da0 msgr2=0x7f399813f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:28.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.789+0000 7f3996d9d640 1 --2- 172.21.15.37:0/2196430930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3998133da0 0x7f399813f7c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f39880097c0 tx=0x7f398801c2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8371b5fd2f3886f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:28.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.789+0000 7f399559a640 1 -- 172.21.15.37:0/2196430930 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39880b0070 con 0x7f3998133da0 2024-04-26T00:41:28.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.789+0000 7f399559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:28.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.789+0000 7f399559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:41:28.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.789+0000 7f399559a640 10 monclient: dump: 2024-04-26T00:41:28.794 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:28.794 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:28.794 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:28.794 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:28.794 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:28.794 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:28.794 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:28.794 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:28.795 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:28.795 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:28.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.789+0000 7f399559a640 10 monclient: _finish_auth 0 2024-04-26T00:41:28.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.789+0000 7f399559a640 10 monclient: _check_auth_tickets 2024-04-26T00:41:28.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.789+0000 7f399559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:41:28.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.789+0000 7f399559a640 1 -- 172.21.15.37:0/2196430930 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39880ab600 con 0x7f3998133da0 2024-04-26T00:41:28.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.789+0000 7f399559a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:41:28.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.789+0000 7f399c85f640 5 monclient: authenticate success, global_id 224443 2024-04-26T00:41:28.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.789+0000 7f399c85f640 10 monclient: get_monmap_and_config success 2024-04-26T00:41:28.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.789+0000 7f399c85f640 4 set_mon_vals no callback set 2024-04-26T00:41:28.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.789+0000 7f399559a640 1 -- 172.21.15.37:0/2196430930 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39880ab9d0 con 0x7f3998133da0 2024-04-26T00:41:28.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.789+0000 7f399559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:28.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.789+0000 7f399559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:41:28.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.789+0000 7f399559a640 10 monclient: dump: 2024-04-26T00:41:28.796 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:28.796 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:28.796 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:28.796 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:28.796 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:28.796 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:28.796 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:28.796 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:28.796 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:28.796 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:28.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.789+0000 7f399c85f640 10 monclient: shutdown 2024-04-26T00:41:28.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.789+0000 7f399c85f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:41:28.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.789+0000 7f399c85f640 1 -- 172.21.15.37:0/2196430930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3998133da0 msgr2=0x7f399813f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:28.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.789+0000 7f399c85f640 1 --2- 172.21.15.37:0/2196430930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3998133da0 0x7f399813f7c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f39880097c0 tx=0x7f398801c2e0 comp rx=0 tx=0).stop 2024-04-26T00:41:28.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.789+0000 7f399c85f640 1 -- 172.21.15.37:0/2196430930 shutdown_connections 2024-04-26T00:41:28.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.789+0000 7f399c85f640 1 --2- 172.21.15.37:0/2196430930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3998133da0 0x7f399813f7c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:28.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.789+0000 7f399c85f640 1 -- 172.21.15.37:0/2196430930 >> 172.21.15.37:0/2196430930 conn(0x7f39980a24c0 msgr2=0x7f39980a28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:41:28.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.789+0000 7f399c85f640 1 -- 172.21.15.37:0/2196430930 shutdown_connections 2024-04-26T00:41:28.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.789+0000 7f399c85f640 1 -- 172.21.15.37:0/2196430930 wait complete. 2024-04-26T00:41:28.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399c85f640 10 monclient: build_initial_monmap 2024-04-26T00:41:28.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399c85f640 10 monclient: monmap: 2024-04-26T00:41:28.797 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:41:28.797 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:41:28.797 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:41:28.797 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:41:28.797 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:41:28.797 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:41:28.798 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:41:28.798 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:41:28.798 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:41:28.798 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:28.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399c85f640 1 Processor -- start 2024-04-26T00:41:28.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399c85f640 1 -- start start 2024-04-26T00:41:28.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399c85f640 10 monclient: init 2024-04-26T00:41:28.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399c85f640 10 monclient: _reopen_session rank -1 2024-04-26T00:41:28.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399c85f640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:41:28.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399c85f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f39980724d0 0x7f3998130700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:28.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399c85f640 10 monclient(hunting): picked mon.noname-c con 0x7f39980724d0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:41:28.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399c85f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3998133990 0x7f3998130c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:28.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399c85f640 10 monclient(hunting): picked mon.noname-a con 0x7f3998133990 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:41:28.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399c85f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3998133da0 0x7f39981387f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:28.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399c85f640 10 monclient(hunting): picked mon.noname-b con 0x7f3998133da0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:41:28.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399c85f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:28.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399c85f640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f3998142ff0 con 0x7f3998133990 2024-04-26T00:41:28.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399c85f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:28.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399c85f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3998142e70 con 0x7f3998133da0 2024-04-26T00:41:28.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399c85f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:28.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399c85f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f3998143170 con 0x7f39980724d0 2024-04-26T00:41:28.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399c85f640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:28.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399c85f640 10 monclient(hunting): authenticate will time out at 2270.101369s 2024-04-26T00:41:28.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399659c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f39980724d0 0x7f3998130700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:28.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f3995d9b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3998133990 0x7f3998130c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:28.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399659c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f39980724d0 0x7f3998130700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:40640/0 (socket says 172.21.15.37:40640) 2024-04-26T00:41:28.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f3995d9b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3998133990 0x7f3998130c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:37580/0 (socket says 172.21.15.37:37580) 2024-04-26T00:41:28.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399659c640 1 -- 172.21.15.37:0/28713954 learned_addr learned my addr 172.21.15.37:0/28713954 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:41:28.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399659c640 10 monclient(hunting): get_auth_request con 0x7f39980724d0 auth_method 0 2024-04-26T00:41:28.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:28.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:28.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:28.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f3995d9b640 10 monclient(hunting): get_auth_request con 0x7f3998133990 auth_method 0 2024-04-26T00:41:28.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f3995d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:28.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f3995d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:28.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f3995d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:28.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:28.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:28.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:28.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f3995d9b640 1 -- 172.21.15.37:0/28713954 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3998133990 msgr2=0x7f3998130c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:41:28.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f3995d9b640 1 -- 172.21.15.37:0/28713954 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3998133990 msgr2=0x7f3998130c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:41:28.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f3995d9b640 1 --2- 172.21.15.37:0/28713954 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3998133990 0x7f3998130c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:41:28.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f3995d9b640 1 --2- 172.21.15.37:0/28713954 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3998133990 0x7f3998130c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:41:28.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f3996d9d640 1 --2- 172.21.15.37:0/28713954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3998133da0 0x7f39981387f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:28.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399659c640 10 monclient(hunting): handle_auth_done global_id 254198 payload 824 2024-04-26T00:41:28.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399659c640 1 -- 172.21.15.37:0/28713954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3998133da0 msgr2=0x7f39981387f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:28.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399659c640 1 --2- 172.21.15.37:0/28713954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3998133da0 0x7f39981387f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:28.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399659c640 1 -- 172.21.15.37:0/28713954 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3998133990 msgr2=0x7f3998130c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:41:28.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399659c640 1 --2- 172.21.15.37:0/28713954 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3998133990 0x7f3998130c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:28.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399659c640 10 monclient: _finish_hunting 0 2024-04-26T00:41:28.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399659c640 1 monclient: found mon.noname-c 2024-04-26T00:41:28.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:28.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:41:28.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399659c640 1 -- 172.21.15.37:0/28713954 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f399813c4a0 con 0x7f39980724d0 2024-04-26T00:41:28.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399659c640 1 -- 172.21.15.37:0/28713954 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f39980724d0 msgr2=0x7f3998130700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:41:28.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399659c640 1 -- 172.21.15.37:0/28713954 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f39980724d0 msgr2=0x7f3998130700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:41:28.803 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399659c640 1 --2- 172.21.15.37:0/28713954 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f39980724d0 0x7f3998130700 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f399000c470 tx=0x7f399000c930 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:41:28.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.793+0000 7f399659c640 1 --2- 172.21.15.37:0/28713954 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f39980724d0 0x7f3998130700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:41:29.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f399659c640 1 --2- 172.21.15.37:0/28713954 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f39980724d0 0x7f3998130700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:29.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f399659c640 10 monclient: get_auth_request con 0x7f39980724d0 auth_method 0 2024-04-26T00:41:29.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f399659c640 0 --2- 172.21.15.37:0/28713954 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f39980724d0 0x7f3998130700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-04-26T00:41:29.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f399659c640 1 --2- 172.21.15.37:0/28713954 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f39980724d0 0x7f3998130700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:29.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f398effd640 10 monclient: ms_handle_reset current mon [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:41:29.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f398effd640 10 monclient: _reopen_session rank -1 2024-04-26T00:41:29.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f398effd640 1 -- 172.21.15.37:0/28713954 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f39980724d0 msgr2=0x7f3998130700 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:41:29.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f398effd640 1 --2- 172.21.15.37:0/28713954 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f39980724d0 0x7f3998130700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:29.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f398effd640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:41:29.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f398effd640 1 --2- 172.21.15.37:0/28713954 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3998133990 0x7f39780054e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:29.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f398effd640 10 monclient(hunting): picked mon.noname-a con 0x7f3998133990 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:41:29.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f398effd640 1 --2- 172.21.15.37:0/28713954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3998133da0 0x7f3978007fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:29.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f398effd640 10 monclient(hunting): picked mon.noname-b con 0x7f3998133da0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:41:29.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f398effd640 1 --2- 172.21.15.37:0/28713954 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3978008a10 0x7f397800aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:29.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f398effd640 10 monclient(hunting): picked mon.noname-c con 0x7f3978008a10 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:41:29.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f398effd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:29.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f398effd640 1 -- 172.21.15.37:0/28713954 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f397800b9a0 con 0x7f3998133990 2024-04-26T00:41:29.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f398effd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:29.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f398effd640 1 -- 172.21.15.37:0/28713954 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f397800bb20 con 0x7f3998133da0 2024-04-26T00:41:29.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f398effd640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:29.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f398effd640 1 -- 172.21.15.37:0/28713954 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f397800bd20 con 0x7f3978008a10 2024-04-26T00:41:29.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f398effd640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:29.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f399659c640 1 --2- 172.21.15.37:0/28713954 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3998133990 0x7f39780054e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:29.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f3996d9d640 1 --2- 172.21.15.37:0/28713954 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3978008a10 0x7f397800aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:29.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f399659c640 10 monclient(hunting): get_auth_request con 0x7f3998133990 auth_method 0 2024-04-26T00:41:29.002 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f399659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:29.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f399659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:29.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f399659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:41:29.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f3996d9d640 10 monclient(hunting): get_auth_request con 0x7f3978008a10 auth_method 0 2024-04-26T00:41:29.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f3996d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:29.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f3996d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:29.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f3996d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:41:29.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f3995d9b640 1 --2- 172.21.15.37:0/28713954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3998133da0 0x7f3978007fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:29.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f399659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:29.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f399659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:29.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f399659c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:41:29.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f3995d9b640 10 monclient(hunting): get_auth_request con 0x7f3998133da0 auth_method 0 2024-04-26T00:41:29.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f3995d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:29.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f3995d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:29.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f3995d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:41:29.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f3996d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:29.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f3996d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:29.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f3996d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:41:29.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f3995d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:29.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f3995d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:29.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f3995d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:41:29.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f399659c640 10 monclient(hunting): handle_auth_done global_id 254198 payload 293 2024-04-26T00:41:29.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f399659c640 1 -- 172.21.15.37:0/28713954 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3978008a10 msgr2=0x7f397800aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:29.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f399659c640 1 --2- 172.21.15.37:0/28713954 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3978008a10 0x7f397800aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:29.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f399659c640 1 -- 172.21.15.37:0/28713954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3998133da0 msgr2=0x7f3978007fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:29.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f399659c640 1 --2- 172.21.15.37:0/28713954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3998133da0 0x7f3978007fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:29.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f399659c640 10 monclient: _finish_hunting 0 2024-04-26T00:41:29.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f399659c640 1 monclient: found mon.noname-a 2024-04-26T00:41:29.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f399659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:29.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f399659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:41:29.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:28.997+0000 7f399659c640 1 -- 172.21.15.37:0/28713954 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f397800bfe0 con 0x7f3998133990 2024-04-26T00:41:29.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:29.001+0000 7f399659c640 1 -- 172.21.15.37:0/28713954 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3998133990 msgr2=0x7f39780054e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:29.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:29.001+0000 7f399659c640 1 --2- 172.21.15.37:0/28713954 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3998133990 0x7f39780054e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f399813a510 tx=0x7f3990004a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:29.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:29.017+0000 7f398effd640 1 -- 172.21.15.37:0/28713954 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3990005cd0 con 0x7f3998133990 2024-04-26T00:41:29.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:29.017+0000 7f398effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:29.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:29.017+0000 7f398effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:41:29.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:29.017+0000 7f398effd640 10 monclient: dump: 2024-04-26T00:41:29.019 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:29.019 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:29.019 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:29.020 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:29.020 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:29.020 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:29.020 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:29.020 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:29.020 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:29.020 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:29.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:29.017+0000 7f398effd640 10 monclient: _finish_auth 0 2024-04-26T00:41:29.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:29.017+0000 7f398effd640 10 monclient: _check_auth_tickets 2024-04-26T00:41:29.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:29.017+0000 7f398effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:41:29.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:29.017+0000 7f398effd640 1 -- 172.21.15.37:0/28713954 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3990019070 con 0x7f3998133990 2024-04-26T00:41:29.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:29.017+0000 7f398effd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:41:29.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:29.017+0000 7f398effd640 1 -- 172.21.15.37:0/28713954 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39900076a0 con 0x7f3998133990 2024-04-26T00:41:29.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:29.017+0000 7f398effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:29.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:29.017+0000 7f398effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:41:29.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:29.017+0000 7f398effd640 10 monclient: dump: 2024-04-26T00:41:29.021 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:29.021 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:29.021 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:29.021 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:29.021 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:29.021 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:29.021 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:29.021 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:29.021 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:29.021 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:29.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:29.017+0000 7f399c85f640 5 monclient: authenticate success, global_id 254198 2024-04-26T00:41:29.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:29.017+0000 7f399c85f640 10 monclient: _renew_subs 2024-04-26T00:41:29.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:29.017+0000 7f3994d99640 4 set_mon_vals no callback set 2024-04-26T00:41:29.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:29.017+0000 7f399c85f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:41:29.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:29.017+0000 7f399c85f640 1 -- 172.21.15.37:0/28713954 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f399813aff0 con 0x7f3998133990 2024-04-26T00:41:29.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:29.017+0000 7f399c85f640 10 monclient: _renew_subs 2024-04-26T00:41:29.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:29.017+0000 7f399c85f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:41:29.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:29.017+0000 7f399c85f640 1 -- 172.21.15.37:0/28713954 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f399813ad70 con 0x7f3998133990 2024-04-26T00:41:29.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:29.017+0000 7f399c85f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:29.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:29.017+0000 7f399c85f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:29.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:29.017+0000 7f399c85f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:41:29.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:29.017+0000 7f399c85f640 1 -- 172.21.15.37:0/28713954 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3998066f40 con 0x7f3998133990 2024-04-26T00:41:29.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:29.017+0000 7f398effd640 1 -- 172.21.15.37:0/28713954 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f399001e470 con 0x7f3998133990 2024-04-26T00:41:29.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:29.017+0000 7f398effd640 1 --2- 172.21.15.37:0/28713954 >> v2:172.21.15.45:6812/2379918623 conn(0x7f39980724d0 0x7f39780405f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:29.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:29.017+0000 7f398effd640 1 -- 172.21.15.37:0/28713954 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(849..849 src has 705..849) ==== 3979+0+0 (secure 0 0 0) 0x7f39900499d0 con 0x7f3998133990 2024-04-26T00:41:29.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:29.021+0000 7f3995d9b640 1 --2- 172.21.15.37:0/28713954 >> v2:172.21.15.45:6812/2379918623 conn(0x7f39980724d0 0x7f39780405f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:29.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:29.021+0000 7f3995d9b640 10 monclient: get_auth_request con 0x7f39980724d0 auth_method 0 2024-04-26T00:41:29.025 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:29.021+0000 7f3995d9b640 1 --2- 172.21.15.37:0/28713954 >> v2:172.21.15.45:6812/2379918623 conn(0x7f39980724d0 0x7f39780405f0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f399813c0f0 tx=0x7f3980009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:29.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:29.025+0000 7f398effd640 1 -- 172.21.15.37:0/28713954 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f3990005af0 con 0x7f3998133990 2024-04-26T00:41:29.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:29.025+0000 7f398effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:29.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:29.025+0000 7f398effd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:41:29.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:29.265+0000 7f399c85f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:41:29.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:29.265+0000 7f399c85f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:41:29.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:29.265+0000 7f399c85f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:41:29.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:29.265+0000 7f399c85f640 1 -- 172.21.15.37:0/28713954 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f399806c1e0 con 0x7f3998133990 2024-04-26T00:41:29.287 INFO:tasks.rados.rados.0.smithi045.stdout:3806: done (1 left) 2024-04-26T00:41:29.287 INFO:tasks.rados.rados.0.smithi045.stdout:3807: done (0 left) 2024-04-26T00:41:29.287 INFO:tasks.rados.rados.0.smithi045.stdout:3808: delete oid 30 current snap is 421 2024-04-26T00:41:29.298 INFO:tasks.rados.rados.0.smithi045.stdout:3808: done (0 left) 2024-04-26T00:41:29.298 INFO:tasks.rados.rados.0.smithi045.stdout:3809: snap_remove snap 332 2024-04-26T00:41:30.296 INFO:tasks.rados.rados.0.smithi045.stdout:3809: done (0 left) 2024-04-26T00:41:30.297 INFO:tasks.rados.rados.0.smithi045.stdout:3810: write oid 40 current snap is 421 2024-04-26T00:41:30.297 INFO:tasks.rados.rados.0.smithi045.stdout:3810: seq_num 1362 ranges {491134=516325,1734122=666468} 2024-04-26T00:41:30.313 INFO:tasks.rados.rados.0.smithi045.stdout:3810: writing smithi04526579-40 from 491134 to 1007459 tid 1 2024-04-26T00:41:30.339 INFO:tasks.rados.rados.0.smithi045.stdout:3810: writing smithi04526579-40 from 1734122 to 2400590 tid 2 2024-04-26T00:41:30.339 INFO:tasks.rados.rados.0.smithi045.stdout:3811: write oid 18 current snap is 421 2024-04-26T00:41:30.339 INFO:tasks.rados.rados.0.smithi045.stdout:3811: seq_num 1363 ranges {507941=591574,1676251=526479,2746598=161343} 2024-04-26T00:41:30.356 INFO:tasks.rados.rados.0.smithi045.stdout:3811: writing smithi04526579-18 from 507941 to 1099515 tid 1 2024-04-26T00:41:30.369 INFO:tasks.rados.rados.0.smithi045.stdout:3811: writing smithi04526579-18 from 1676251 to 2202730 tid 2 2024-04-26T00:41:30.376 INFO:tasks.rados.rados.0.smithi045.stdout:3811: writing smithi04526579-18 from 2746598 to 2907941 tid 3 2024-04-26T00:41:30.376 INFO:tasks.rados.rados.0.smithi045.stdout:3812: delete oid 47 current snap is 421 2024-04-26T00:41:30.376 INFO:tasks.rados.rados.0.smithi045.stdout:3810: finishing write tid 1 to smithi04526579-40 2024-04-26T00:41:30.376 INFO:tasks.rados.rados.0.smithi045.stdout:3810: finishing write tid 2 to smithi04526579-40 2024-04-26T00:41:30.376 INFO:tasks.rados.rados.0.smithi045.stdout:3810: finishing write tid 3 to smithi04526579-40 2024-04-26T00:41:30.376 INFO:tasks.rados.rados.0.smithi045.stdout:3810: finishing write tid 4 to smithi04526579-40 2024-04-26T00:41:30.376 INFO:tasks.rados.rados.0.smithi045.stdout:3810: oid 40 updating version 0 to 2486 2024-04-26T00:41:30.377 INFO:tasks.rados.rados.0.smithi045.stdout:3810: oid 40 updating version 2486 to 2487 2024-04-26T00:41:30.377 INFO:tasks.rados.rados.0.smithi045.stdout:3810: oid 40 version 2487 is already newer than 2485 2024-04-26T00:41:30.377 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 2487 (ObjNum 1362 snap 421 seq_num 1362) dirty exists 2024-04-26T00:41:30.377 INFO:tasks.rados.rados.0.smithi045.stdout:3810: left oid 40 (ObjNum 1362 snap 421 seq_num 1362) 2024-04-26T00:41:30.377 INFO:tasks.rados.rados.0.smithi045.stdout:3811: finishing write tid 1 to smithi04526579-18 2024-04-26T00:41:30.377 INFO:tasks.rados.rados.0.smithi045.stdout:3811: finishing write tid 2 to smithi04526579-18 2024-04-26T00:41:30.384 INFO:tasks.rados.rados.0.smithi045.stdout:3811: finishing write tid 3 to smithi04526579-18 2024-04-26T00:41:30.384 INFO:tasks.rados.rados.0.smithi045.stdout:3811: finishing write tid 4 to smithi04526579-18 2024-04-26T00:41:30.384 INFO:tasks.rados.rados.0.smithi045.stdout:3811: finishing write tid 5 to smithi04526579-18 2024-04-26T00:41:30.384 INFO:tasks.rados.rados.0.smithi045.stdout:3811: oid 18 updating version 0 to 2872 2024-04-26T00:41:30.385 INFO:tasks.rados.rados.0.smithi045.stdout:3811: oid 18 version 2872 is already newer than 2871 2024-04-26T00:41:30.385 INFO:tasks.rados.rados.0.smithi045.stdout:3811: oid 18 updating version 2872 to 2874 2024-04-26T00:41:30.385 INFO:tasks.rados.rados.0.smithi045.stdout:3811: oid 18 version 2874 is already newer than 2873 2024-04-26T00:41:30.385 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 2874 (ObjNum 1363 snap 421 seq_num 1363) dirty exists 2024-04-26T00:41:30.385 INFO:tasks.rados.rados.0.smithi045.stdout:3811: left oid 18 (ObjNum 1363 snap 421 seq_num 1363) 2024-04-26T00:41:30.385 INFO:tasks.rados.rados.0.smithi045.stdout:3810: done (2 left) 2024-04-26T00:41:30.386 INFO:tasks.rados.rados.0.smithi045.stdout:3811: done (1 left) 2024-04-26T00:41:30.386 INFO:tasks.rados.rados.0.smithi045.stdout:3812: done (0 left) 2024-04-26T00:41:30.386 INFO:tasks.rados.rados.0.smithi045.stdout:3813: copy_from oid 42 from oid 18 current snap is 421 2024-04-26T00:41:30.386 INFO:tasks.rados.rados.0.smithi045.stdout:3814: write oid 11 current snap is 421 2024-04-26T00:41:30.386 INFO:tasks.rados.rados.0.smithi045.stdout:3814: seq_num 1365 ranges {541555=742072,1793215=530148,2801444=140111} 2024-04-26T00:41:30.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.385+0000 7f398effd640 1 -- 172.21.15.37:0/28713954 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f399001ee20 con 0x7f3998133990 2024-04-26T00:41:30.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.385+0000 7f398effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:41:30.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.385+0000 7f398effd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:41:30.388 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":156,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:41:30.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.389+0000 7f399c85f640 1 -- 172.21.15.37:0/28713954 >> v2:172.21.15.45:6812/2379918623 conn(0x7f39980724d0 msgr2=0x7f39780405f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:30.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.389+0000 7f399c85f640 1 --2- 172.21.15.37:0/28713954 >> v2:172.21.15.45:6812/2379918623 conn(0x7f39980724d0 0x7f39780405f0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f399813c0f0 tx=0x7f3980009040 comp rx=0 tx=0).stop 2024-04-26T00:41:30.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.389+0000 7f399c85f640 10 monclient: shutdown 2024-04-26T00:41:30.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.389+0000 7f399c85f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:41:30.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.389+0000 7f399c85f640 1 -- 172.21.15.37:0/28713954 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3998133990 msgr2=0x7f39780054e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:30.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.389+0000 7f399c85f640 1 --2- 172.21.15.37:0/28713954 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3998133990 0x7f39780054e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f399813a510 tx=0x7f3990004a20 comp rx=0 tx=0).stop 2024-04-26T00:41:30.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.389+0000 7f399c85f640 1 -- 172.21.15.37:0/28713954 shutdown_connections 2024-04-26T00:41:30.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.389+0000 7f399c85f640 1 --2- 172.21.15.37:0/28713954 >> v2:172.21.15.45:6812/2379918623 conn(0x7f39980724d0 0x7f39780405f0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:30.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.389+0000 7f399c85f640 1 --2- 172.21.15.37:0/28713954 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3978008a10 0x7f397800aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:30.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.389+0000 7f399c85f640 1 --2- 172.21.15.37:0/28713954 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3998133da0 0x7f3978007fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:30.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.389+0000 7f399c85f640 1 --2- 172.21.15.37:0/28713954 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3998133990 0x7f39780054e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:30.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.389+0000 7f399c85f640 1 -- 172.21.15.37:0/28713954 >> 172.21.15.37:0/28713954 conn(0x7f39980a24c0 msgr2=0x7f39981320e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:41:30.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.389+0000 7f399c85f640 1 -- 172.21.15.37:0/28713954 shutdown_connections 2024-04-26T00:41:30.394 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.389+0000 7f399c85f640 1 -- 172.21.15.37:0/28713954 wait complete. 2024-04-26T00:41:30.400 INFO:tasks.rados.rados.0.smithi045.stdout:3814: writing smithi04526579-11 from 541555 to 1283627 tid 1 2024-04-26T00:41:30.411 INFO:tasks.rados.rados.0.smithi045.stdout:3814: writing smithi04526579-11 from 1793215 to 2323363 tid 2 2024-04-26T00:41:30.418 INFO:tasks.rados.rados.0.smithi045.stdout:3814: writing smithi04526579-11 from 2801444 to 2941555 tid 3 2024-04-26T00:41:30.419 INFO:tasks.rados.rados.0.smithi045.stdout:3815: delete oid 24 current snap is 421 2024-04-26T00:41:30.419 INFO:tasks.rados.rados.0.smithi045.stdout:3813: finishing copy_from racing read to smithi04526579-42 2024-04-26T00:41:30.419 INFO:tasks.rados.rados.0.smithi045.stdout:3814: finishing write tid 1 to smithi04526579-11 2024-04-26T00:41:30.419 INFO:tasks.rados.rados.0.smithi045.stdout:3813: finishing copy_from to smithi04526579-42 2024-04-26T00:41:30.419 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 2874 (ObjNum 1363 snap 421 seq_num 1363) dirty exists 2024-04-26T00:41:30.419 INFO:tasks.rados.rados.0.smithi045.stdout:3814: finishing write tid 2 to smithi04526579-11 2024-04-26T00:41:30.420 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:41:30.420 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:41:30.421 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:41:30.423 INFO:tasks.rados.rados.0.smithi045.stdout:3814: finishing write tid 3 to smithi04526579-11 2024-04-26T00:41:30.423 INFO:tasks.rados.rados.0.smithi045.stdout:3814: finishing write tid 4 to smithi04526579-11 2024-04-26T00:41:30.423 INFO:tasks.rados.rados.0.smithi045.stdout:3814: finishing write tid 5 to smithi04526579-11 2024-04-26T00:41:30.423 INFO:tasks.rados.rados.0.smithi045.stdout:3814: oid 11 updating version 0 to 2457 2024-04-26T00:41:30.423 INFO:tasks.rados.rados.0.smithi045.stdout:3814: oid 11 version 2457 is already newer than 2456 2024-04-26T00:41:30.423 INFO:tasks.rados.rados.0.smithi045.stdout:3814: oid 11 version 2457 is already newer than 2454 2024-04-26T00:41:30.423 INFO:tasks.rados.rados.0.smithi045.stdout:3814: oid 11 version 2457 is already newer than 2455 2024-04-26T00:41:30.423 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 2457 (ObjNum 1365 snap 421 seq_num 1365) dirty exists 2024-04-26T00:41:30.423 INFO:tasks.rados.rados.0.smithi045.stdout:3814: left oid 11 (ObjNum 1365 snap 421 seq_num 1365) 2024-04-26T00:41:30.423 INFO:tasks.rados.rados.0.smithi045.stdout:3813: done (2 left) 2024-04-26T00:41:30.424 INFO:tasks.rados.rados.0.smithi045.stdout:3814: done (1 left) 2024-04-26T00:41:30.424 INFO:tasks.rados.rados.0.smithi045.stdout:3815: done (0 left) 2024-04-26T00:41:30.424 INFO:tasks.rados.rados.0.smithi045.stdout:3816: snap_create 2024-04-26T00:41:30.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41b680c640 10 monclient: get_monmap_and_config 2024-04-26T00:41:30.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41b680c640 10 monclient: build_initial_monmap 2024-04-26T00:41:30.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41b680c640 10 monclient: monmap: 2024-04-26T00:41:30.550 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:41:30.550 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:30.550 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:41:30.552152+0000 2024-04-26T00:41:30.550 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:41:30.552152+0000 2024-04-26T00:41:30.550 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:41:30.550 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:30.550 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:41:30.550 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:41:30.550 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:41:30.551 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:41:30.551 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:41:30.551 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:30.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41b680c640 1 Processor -- start 2024-04-26T00:41:30.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41b680c640 1 -- start start 2024-04-26T00:41:30.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41b680c640 10 monclient: init 2024-04-26T00:41:30.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41b680c640 10 monclient: _reopen_session rank -1 2024-04-26T00:41:30.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41b680c640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-04-26T00:41:30.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41b680c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f41b00a3d90 addr v1:172.21.15.45:6789/0 2024-04-26T00:41:30.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41b680c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f41b00a4170 0x7f41b00a45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:30.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41b680c640 10 monclient(hunting): picked mon.noname-a con 0x7f41b00a4170 addr v2:172.21.15.37:3300/0 2024-04-26T00:41:30.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41b680c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41b013c3e0 0x7f41b013e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:30.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41b680c640 10 monclient(hunting): picked mon.noname-b con 0x7f41b013c3e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:41:30.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41b680c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41b00a5150 con 0x7f41b00a3d90 2024-04-26T00:41:30.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41b680c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:30.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41b680c640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f41b009c650 con 0x7f41b00a4170 2024-04-26T00:41:30.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41b680c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:30.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41b680c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f41b009b280 con 0x7f41b013c3e0 2024-04-26T00:41:30.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41b680c640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:30.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41b680c640 10 monclient(hunting): authenticate will time out at 2271.857866s 2024-04-26T00:41:30.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41b4da9640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41b013c3e0 0x7f41b013e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:30.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41af7fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f41b00a4170 0x7f41b00a45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:30.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41af7fe640 1 -- >> v2:172.21.15.37:3300/0 conn(0x7f41b00a4170 msgr2=0x7f41b00a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:41:30.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41af7fe640 1 -- >> v2:172.21.15.37:3300/0 conn(0x7f41b00a4170 msgr2=0x7f41b00a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:41:30.554 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41af7fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f41b00a4170 0x7f41b00a45e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:41:30.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41b4da9640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41b013c3e0 0x7f41b013e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:45186/0 (socket says 172.21.15.37:45186) 2024-04-26T00:41:30.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41b4da9640 1 -- 172.21.15.37:0/1692768612 learned_addr learned my addr 172.21.15.37:0/1692768612 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:41:30.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41b4da9640 10 monclient(hunting): get_auth_request con 0x7f41b013c3e0 auth_method 0 2024-04-26T00:41:30.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41b4da9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:30.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41b4da9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:30.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41b4da9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:30.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41af7fe640 1 --2- 172.21.15.37:0/1692768612 >> v2:172.21.15.37:3300/0 conn(0x7f41b00a4170 0x7f41b00a45e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:41:30.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41b4da9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:30.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41b4da9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:30.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41b4da9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:30.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41b4da9640 10 monclient(hunting): handle_auth_done global_id 254201 payload 274 2024-04-26T00:41:30.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41b4da9640 1 -- 172.21.15.37:0/1692768612 >> v2:172.21.15.37:3300/0 conn(0x7f41b00a4170 msgr2=0x7f41b00a45e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:41:30.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41b4da9640 1 --2- 172.21.15.37:0/1692768612 >> v2:172.21.15.37:3300/0 conn(0x7f41b00a4170 0x7f41b00a45e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:30.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41b4da9640 1 -- 172.21.15.37:0/1692768612 >> v1:172.21.15.45:6789/0 conn(0x7f41b00a3d90 legacy=0x7f41b009bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:41:30.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41b4da9640 10 monclient: _finish_hunting 0 2024-04-26T00:41:30.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41b4da9640 1 monclient: found mon.noname-b 2024-04-26T00:41:30.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41b4da9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:30.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41b4da9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:41:30.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.549+0000 7f41b4da9640 1 -- 172.21.15.37:0/1692768612 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41b009b560 con 0x7f41b013c3e0 2024-04-26T00:41:30.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.553+0000 7f41b4da9640 1 -- 172.21.15.37:0/1692768612 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41b013c3e0 msgr2=0x7f41b013e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:30.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.553+0000 7f41b4da9640 1 --2- 172.21.15.37:0/1692768612 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41b013c3e0 0x7f41b013e7d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f41a8009870 tx=0x7f41a801c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6984739cc7743537 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:30.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.553+0000 7f41aeffd640 1 -- 172.21.15.37:0/1692768612 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a80b0070 con 0x7f41b013c3e0 2024-04-26T00:41:30.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.553+0000 7f41aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:30.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.553+0000 7f41aeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:41:30.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.553+0000 7f41aeffd640 10 monclient: dump: 2024-04-26T00:41:30.565 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:30.565 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:30.565 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:30.565 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:30.565 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:30.565 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:30.565 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:30.565 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:30.566 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:30.566 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:30.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.553+0000 7f41aeffd640 10 monclient: _finish_auth 0 2024-04-26T00:41:30.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.553+0000 7f41aeffd640 10 monclient: _check_auth_tickets 2024-04-26T00:41:30.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.553+0000 7f41aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:41:30.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.553+0000 7f41aeffd640 1 -- 172.21.15.37:0/1692768612 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41a80ab600 con 0x7f41b013c3e0 2024-04-26T00:41:30.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.553+0000 7f41b680c640 5 monclient: authenticate success, global_id 254201 2024-04-26T00:41:30.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.553+0000 7f41b680c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:41:30.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.553+0000 7f41aeffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:41:30.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.553+0000 7f41b680c640 10 monclient: get_monmap_and_config success 2024-04-26T00:41:30.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.553+0000 7f41b680c640 4 set_mon_vals no callback set 2024-04-26T00:41:30.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.553+0000 7f41aeffd640 1 -- 172.21.15.37:0/1692768612 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a80ab9d0 con 0x7f41b013c3e0 2024-04-26T00:41:30.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.553+0000 7f41aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:30.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.553+0000 7f41aeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:41:30.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.553+0000 7f41aeffd640 10 monclient: dump: 2024-04-26T00:41:30.567 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:30.567 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:30.567 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:30.567 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:30.567 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:30.567 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:30.567 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:30.567 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:30.567 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:30.567 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:30.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.553+0000 7f41b680c640 10 monclient: shutdown 2024-04-26T00:41:30.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.553+0000 7f41b680c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:41:30.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.553+0000 7f41b680c640 1 -- 172.21.15.37:0/1692768612 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41b013c3e0 msgr2=0x7f41b013e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:30.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.553+0000 7f41b680c640 1 --2- 172.21.15.37:0/1692768612 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41b013c3e0 0x7f41b013e7d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f41a8009870 tx=0x7f41a801c3e0 comp rx=0 tx=0).stop 2024-04-26T00:41:30.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.553+0000 7f41b680c640 1 -- 172.21.15.37:0/1692768612 shutdown_connections 2024-04-26T00:41:30.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.553+0000 7f41b680c640 1 --2- 172.21.15.37:0/1692768612 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41b013c3e0 0x7f41b013e7d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:30.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.553+0000 7f41b680c640 1 --2- 172.21.15.37:0/1692768612 >> v2:172.21.15.37:3300/0 conn(0x7f41b00a4170 0x7f41b00a45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:30.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.553+0000 7f41b680c640 1 -- 172.21.15.37:0/1692768612 >> 172.21.15.37:0/1692768612 conn(0x7f41b00aa0a0 msgr2=0x7f41b00a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:41:30.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.553+0000 7f41b680c640 1 -- 172.21.15.37:0/1692768612 shutdown_connections 2024-04-26T00:41:30.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.553+0000 7f41b680c640 1 -- 172.21.15.37:0/1692768612 wait complete. 2024-04-26T00:41:30.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b680c640 10 monclient: build_initial_monmap 2024-04-26T00:41:30.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b680c640 10 monclient: monmap: 2024-04-26T00:41:30.568 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:41:30.568 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:41:30.568 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:41:30.568 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:41:30.568 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:41:30.568 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:41:30.568 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:41:30.568 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:41:30.568 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:41:30.569 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:30.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b680c640 1 Processor -- start 2024-04-26T00:41:30.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b680c640 1 -- start start 2024-04-26T00:41:30.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b680c640 10 monclient: init 2024-04-26T00:41:30.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b680c640 10 monclient: _reopen_session rank -1 2024-04-26T00:41:30.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b680c640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:41:30.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b680c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41b00a3d90 0x7f41b01cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:30.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b680c640 10 monclient(hunting): picked mon.noname-b con 0x7f41b00a3d90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:41:30.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b680c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b00a4170 0x7f41b01cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:30.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b680c640 10 monclient(hunting): picked mon.noname-a con 0x7f41b00a4170 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:41:30.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b680c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41b013c3e0 0x7f41b0139df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:30.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b680c640 10 monclient(hunting): picked mon.noname-c con 0x7f41b013c3e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:41:30.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b680c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:30.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b680c640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f41b01418c0 con 0x7f41b00a4170 2024-04-26T00:41:30.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b680c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:30.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b680c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f41b0141740 con 0x7f41b00a3d90 2024-04-26T00:41:30.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b680c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:30.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b680c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f41b0141a40 con 0x7f41b013c3e0 2024-04-26T00:41:30.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b680c640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:30.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b680c640 10 monclient(hunting): authenticate will time out at 2271.865140s 2024-04-26T00:41:30.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41af7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b00a4170 0x7f41b01cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:30.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41af7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b00a4170 0x7f41b01cf4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:45844/0 (socket says 172.21.15.37:45844) 2024-04-26T00:41:30.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b4da9640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41b013c3e0 0x7f41b0139df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:30.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41af7fe640 1 -- 172.21.15.37:0/3858316744 learned_addr learned my addr 172.21.15.37:0/3858316744 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:41:30.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41af7fe640 10 monclient(hunting): get_auth_request con 0x7f41b00a4170 auth_method 0 2024-04-26T00:41:30.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:30.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41af7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:30.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:30.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b4da9640 10 monclient(hunting): get_auth_request con 0x7f41b013c3e0 auth_method 0 2024-04-26T00:41:30.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b4da9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:30.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b4da9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:30.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b4da9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:30.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b4da9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:30.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b4da9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:30.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41af7fe640 1 -- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b00a4170 msgr2=0x7f41b01cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:41:30.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41af7fe640 1 -- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b00a4170 msgr2=0x7f41b01cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:41:30.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41af7fe640 1 --2- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b00a4170 0x7f41b01cf4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:41:30.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b4da9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:30.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41af7fe640 1 --2- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b00a4170 0x7f41b01cf4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:41:30.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41affff640 1 --2- 172.21.15.37:0/3858316744 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41b00a3d90 0x7f41b01cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:30.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b4da9640 10 monclient(hunting): handle_auth_done global_id 254204 payload 824 2024-04-26T00:41:30.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b4da9640 1 -- 172.21.15.37:0/3858316744 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41b00a3d90 msgr2=0x7f41b01cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:30.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b4da9640 1 --2- 172.21.15.37:0/3858316744 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41b00a3d90 0x7f41b01cecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:30.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b4da9640 1 -- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b00a4170 msgr2=0x7f41b01cf4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:41:30.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b4da9640 1 --2- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b00a4170 0x7f41b01cf4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:30.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b4da9640 10 monclient: _finish_hunting 0 2024-04-26T00:41:30.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b4da9640 1 monclient: found mon.noname-c 2024-04-26T00:41:30.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b4da9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:30.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b4da9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:41:30.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b4da9640 1 -- 172.21.15.37:0/3858316744 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41b013ba90 con 0x7f41b013c3e0 2024-04-26T00:41:30.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41affff640 10 monclient: get_auth_request con 0x7f41b00a3d90 auth_method 0 2024-04-26T00:41:30.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41affff640 1 --2- 172.21.15.37:0/3858316744 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41b00a3d90 0x7f41b01cecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:41:30.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b4da9640 1 -- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41b013c3e0 msgr2=0x7f41b0139df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:30.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.557+0000 7f41b4da9640 1 --2- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41b013c3e0 0x7f41b0139df0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f41a80ab000 tx=0x7f41a801ac20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:30.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.561+0000 7f41acff9640 1 -- 172.21.15.37:0/3858316744 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a80ba030 con 0x7f41b013c3e0 2024-04-26T00:41:30.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.561+0000 7f41acff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:30.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.561+0000 7f41acff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:41:30.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.561+0000 7f41acff9640 10 monclient: dump: 2024-04-26T00:41:30.572 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:30.572 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:30.572 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:30.572 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:30.573 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:30.573 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:30.573 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:30.573 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:30.573 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:30.573 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:30.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.561+0000 7f41acff9640 10 monclient: _finish_auth 0 2024-04-26T00:41:30.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.561+0000 7f41acff9640 10 monclient: _check_auth_tickets 2024-04-26T00:41:30.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.561+0000 7f41acff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:41:30.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.561+0000 7f41acff9640 1 -- 172.21.15.37:0/3858316744 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41a80ab400 con 0x7f41b013c3e0 2024-04-26T00:41:30.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.561+0000 7f41acff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:41:30.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.561+0000 7f41acff9640 1 -- 172.21.15.37:0/3858316744 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a80b0040 con 0x7f41b013c3e0 2024-04-26T00:41:30.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.561+0000 7f41acff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:30.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.561+0000 7f41acff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:41:30.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.561+0000 7f41acff9640 10 monclient: dump: 2024-04-26T00:41:30.573 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:30.574 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:30.574 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:30.574 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:30.574 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:30.574 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:30.574 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:30.574 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:30.574 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:30.574 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:30.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.561+0000 7f41ae7fc640 4 set_mon_vals no callback set 2024-04-26T00:41:30.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.561+0000 7f41b680c640 5 monclient: authenticate success, global_id 254204 2024-04-26T00:41:30.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.561+0000 7f41b680c640 10 monclient: _renew_subs 2024-04-26T00:41:30.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.561+0000 7f41b680c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:41:30.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.561+0000 7f41b680c640 1 -- 172.21.15.37:0/3858316744 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41b0120f30 con 0x7f41b013c3e0 2024-04-26T00:41:30.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.561+0000 7f41b680c640 10 monclient: _renew_subs 2024-04-26T00:41:30.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.561+0000 7f41b680c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:41:30.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.561+0000 7f41b680c640 1 -- 172.21.15.37:0/3858316744 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f41b0120040 con 0x7f41b013c3e0 2024-04-26T00:41:30.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.561+0000 7f41b680c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:30.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.561+0000 7f41b680c640 1 --2- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b0120380 0x7f41b013a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:30.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.561+0000 7f41b680c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:30.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.561+0000 7f41b680c640 1 -- 172.21.15.37:0/3858316744 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f41b009ada0 con 0x7f41b0120380 2024-04-26T00:41:30.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.561+0000 7f41affff640 1 --2- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b0120380 0x7f41b013a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:30.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.561+0000 7f41affff640 10 monclient: get_auth_request con 0x7f41b0120380 auth_method 0 2024-04-26T00:41:30.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.561+0000 7f41affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:30.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.561+0000 7f41affff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:30.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.561+0000 7f41affff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:30.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.561+0000 7f41acff9640 1 -- 172.21.15.37:0/3858316744 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f41a80bd070 con 0x7f41b013c3e0 2024-04-26T00:41:30.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.561+0000 7f41affff640 1 -- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b0120380 msgr2=0x7f41b013a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:41:30.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.561+0000 7f41affff640 1 -- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b0120380 msgr2=0x7f41b013a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:41:30.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.561+0000 7f41affff640 1 --2- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b0120380 0x7f41b013a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:41:30.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.561+0000 7f41affff640 1 --2- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b0120380 0x7f41b013a330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:41:30.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.561+0000 7f41acff9640 1 --2- 172.21.15.37:0/3858316744 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4190035ed0 0x7f4190038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:30.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.561+0000 7f41acff9640 1 -- 172.21.15.37:0/3858316744 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(857..857 src has 705..857) ==== 3943+0+0 (secure 0 0 0) 0x7f41a80e2a80 con 0x7f41b013c3e0 2024-04-26T00:41:30.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.561+0000 7f41af7fe640 1 --2- 172.21.15.37:0/3858316744 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4190035ed0 0x7f4190038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:30.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.561+0000 7f41af7fe640 10 monclient: get_auth_request con 0x7f4190035ed0 auth_method 0 2024-04-26T00:41:30.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.565+0000 7f41af7fe640 1 --2- 172.21.15.37:0/3858316744 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4190035ed0 0x7f4190038390 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f41b013b460 tx=0x7f4198008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:30.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.765+0000 7f41affff640 1 --2- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b0120380 0x7f41b013a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:30.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.765+0000 7f41affff640 10 monclient: get_auth_request con 0x7f41b0120380 auth_method 0 2024-04-26T00:41:30.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.765+0000 7f41affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:30.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.765+0000 7f41affff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:30.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.765+0000 7f41affff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:41:30.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.765+0000 7f41affff640 1 -- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b0120380 msgr2=0x7f41b013a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:41:30.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.765+0000 7f41affff640 1 -- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b0120380 msgr2=0x7f41b013a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:41:30.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.765+0000 7f41affff640 1 --2- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b0120380 0x7f41b013a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:41:30.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:30.765+0000 7f41affff640 1 --2- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b0120380 0x7f41b013a330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:41:31.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:31.169+0000 7f41affff640 1 --2- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b0120380 0x7f41b013a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:31.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:31.169+0000 7f41affff640 10 monclient: get_auth_request con 0x7f41b0120380 auth_method 0 2024-04-26T00:41:31.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:31.169+0000 7f41affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:31.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:31.169+0000 7f41affff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:31.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:31.169+0000 7f41affff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:41:31.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:31.169+0000 7f41affff640 1 -- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b0120380 msgr2=0x7f41b013a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:41:31.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:31.169+0000 7f41affff640 1 -- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b0120380 msgr2=0x7f41b013a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:41:31.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:31.169+0000 7f41affff640 1 --2- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b0120380 0x7f41b013a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:41:31.171 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:31.169+0000 7f41affff640 1 --2- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b0120380 0x7f41b013a330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:41:31.433 INFO:tasks.rados.rados.0.smithi045.stdout:3816: done (0 left) 2024-04-26T00:41:31.433 INFO:tasks.rados.rados.0.smithi045.stdout:3817: snap_remove snap 407 2024-04-26T00:41:31.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:31.973+0000 7f41affff640 1 --2- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b0120380 0x7f41b013a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:31.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:31.973+0000 7f41affff640 10 monclient: get_auth_request con 0x7f41b0120380 auth_method 0 2024-04-26T00:41:31.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:31.973+0000 7f41affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:31.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:31.973+0000 7f41affff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:31.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:31.973+0000 7f41affff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:41:31.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:31.973+0000 7f41affff640 1 -- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b0120380 msgr2=0x7f41b013a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:41:31.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:31.973+0000 7f41affff640 1 -- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b0120380 msgr2=0x7f41b013a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:41:31.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:31.973+0000 7f41affff640 1 --2- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b0120380 0x7f41b013a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:41:31.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:31.973+0000 7f41affff640 1 --2- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b0120380 0x7f41b013a330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:41:32.439 INFO:tasks.rados.rados.0.smithi045.stdout:3817: done (0 left) 2024-04-26T00:41:32.439 INFO:tasks.rados.rados.0.smithi045.stdout:3818: read oid 34 snap -1 2024-04-26T00:41:32.439 INFO:tasks.rados.rados.0.smithi045.stdout:3818: expect deleted 2024-04-26T00:41:32.439 INFO:tasks.rados.rados.0.smithi045.stdout:3819: write (excl) oid 6 current snap is 422 2024-04-26T00:41:32.439 INFO:tasks.rados.rados.0.smithi045.stdout:3819: seq_num 1367 ranges {575169=492570,1793826=417464,2623584=351585} 2024-04-26T00:41:32.456 INFO:tasks.rados.rados.0.smithi045.stdout:3819: writing smithi04526579-6 from 575169 to 1067739 tid 1 2024-04-26T00:41:32.474 INFO:tasks.rados.rados.0.smithi045.stdout:3819: writing smithi04526579-6 from 1793826 to 2211290 tid 2 2024-04-26T00:41:32.487 INFO:tasks.rados.rados.0.smithi045.stdout:3819: writing smithi04526579-6 from 2623584 to 2975169 tid 3 2024-04-26T00:41:32.487 INFO:tasks.rados.rados.0.smithi045.stdout:3820: read oid 16 snap -1 2024-04-26T00:41:32.487 INFO:tasks.rados.rados.0.smithi045.stdout:3820: expect (ObjNum 1333 snap 411 seq_num 1333) 2024-04-26T00:41:32.487 INFO:tasks.rados.rados.0.smithi045.stdout:3821: delete oid 33 current snap is 422 2024-04-26T00:41:32.487 INFO:tasks.rados.rados.0.smithi045.stdout:3819: finishing write tid 1 to smithi04526579-6 2024-04-26T00:41:32.487 INFO:tasks.rados.rados.0.smithi045.stdout:3819: finishing write tid 2 to smithi04526579-6 2024-04-26T00:41:32.493 INFO:tasks.rados.rados.0.smithi045.stdout:3819: finishing write tid 3 to smithi04526579-6 2024-04-26T00:41:32.493 INFO:tasks.rados.rados.0.smithi045.stdout:3819: finishing write tid 4 to smithi04526579-6 2024-04-26T00:41:32.494 INFO:tasks.rados.rados.0.smithi045.stdout:3819: finishing write tid 5 to smithi04526579-6 2024-04-26T00:41:32.494 INFO:tasks.rados.rados.0.smithi045.stdout:3819: oid 6 updating version 0 to 2473 2024-04-26T00:41:32.494 INFO:tasks.rados.rados.0.smithi045.stdout:3819: oid 6 version 2473 is already newer than 2472 2024-04-26T00:41:32.494 INFO:tasks.rados.rados.0.smithi045.stdout:3819: oid 6 updating version 2473 to 2474 2024-04-26T00:41:32.494 INFO:tasks.rados.rados.0.smithi045.stdout:3819: oid 6 updating version 2474 to 2475 2024-04-26T00:41:32.494 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 6 v 2475 (ObjNum 1367 snap 422 seq_num 1367) dirty exists 2024-04-26T00:41:32.494 INFO:tasks.rados.rados.0.smithi045.stdout:3819: left oid 6 (ObjNum 1367 snap 422 seq_num 1367) 2024-04-26T00:41:32.497 INFO:tasks.rados.rados.0.smithi045.stdout:3818: done (3 left) 2024-04-26T00:41:32.497 INFO:tasks.rados.rados.0.smithi045.stdout:3819: done (2 left) 2024-04-26T00:41:32.497 INFO:tasks.rados.rados.0.smithi045.stdout:3821: done (1 left) 2024-04-26T00:41:32.498 INFO:tasks.rados.rados.0.smithi045.stdout:3822: read oid 26 snap -1 2024-04-26T00:41:32.498 INFO:tasks.rados.rados.0.smithi045.stdout:3822: expect (ObjNum 1315 snap 407 seq_num 1315) 2024-04-26T00:41:32.498 INFO:tasks.rados.rados.0.smithi045.stdout:3823: delete oid 40 current snap is 422 2024-04-26T00:41:32.503 INFO:tasks.rados.rados.0.smithi045.stdout:3820: expect (ObjNum 1333 snap 411 seq_num 1333) 2024-04-26T00:41:32.662 INFO:tasks.rados.rados.0.smithi045.stdout:3822: expect (ObjNum 1315 snap 407 seq_num 1315) 2024-04-26T00:41:32.734 INFO:tasks.rados.rados.0.smithi045.stdout:3820: done (2 left) 2024-04-26T00:41:32.734 INFO:tasks.rados.rados.0.smithi045.stdout:3822: done (1 left) 2024-04-26T00:41:32.734 INFO:tasks.rados.rados.0.smithi045.stdout:3823: done (0 left) 2024-04-26T00:41:32.734 INFO:tasks.rados.rados.0.smithi045.stdout:3824: rollback oid 32 current snap is 422 2024-04-26T00:41:32.734 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 32 to 410 2024-04-26T00:41:32.734 INFO:tasks.rados.rados.0.smithi045.stdout:3825: read oid 40 snap -1 2024-04-26T00:41:32.734 INFO:tasks.rados.rados.0.smithi045.stdout:3825: expect deleted 2024-04-26T00:41:32.734 INFO:tasks.rados.rados.0.smithi045.stdout:3826: copy_from oid 25 from oid 13 current snap is 422 2024-04-26T00:41:32.734 INFO:tasks.rados.rados.0.smithi045.stdout:3827: read oid 33 snap -1 2024-04-26T00:41:32.734 INFO:tasks.rados.rados.0.smithi045.stdout:3827: expect deleted 2024-04-26T00:41:32.735 INFO:tasks.rados.rados.0.smithi045.stdout:3828: snap_create 2024-04-26T00:41:32.743 INFO:tasks.rados.rados.0.smithi045.stdout:3824: finishing rollback tid 1 to smithi04526579-32 2024-04-26T00:41:32.743 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 2504 (ObjNum 113 snap 0 seq_num 907320640) dirty dne 2024-04-26T00:41:32.757 INFO:tasks.rados.rados.0.smithi045.stdout:3826: finishing copy_from racing read to smithi04526579-25 2024-04-26T00:41:32.758 INFO:tasks.rados.rados.0.smithi045.stdout:3826: finishing copy_from to smithi04526579-25 2024-04-26T00:41:32.758 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 25 v 2243 (ObjNum 1341 snap 415 seq_num 1341) dirty exists 2024-04-26T00:41:33.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.557+0000 7f41a77fe640 10 monclient: tick 2024-04-26T00:41:33.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.557+0000 7f41a77fe640 10 monclient: _check_auth_tickets 2024-04-26T00:41:33.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.557+0000 7f41a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:41:33.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.557+0000 7f41a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:33.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.577+0000 7f41affff640 1 --2- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b0120380 0x7f41b013a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:33.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.577+0000 7f41affff640 10 monclient: get_auth_request con 0x7f41b0120380 auth_method 0 2024-04-26T00:41:33.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.577+0000 7f41affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:33.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.577+0000 7f41affff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:33.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.577+0000 7f41affff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:41:33.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.577+0000 7f41affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:33.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.577+0000 7f41affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:33.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.577+0000 7f41affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:33.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.577+0000 7f41affff640 10 monclient(hunting): handle_auth_done global_id 264099 payload 824 2024-04-26T00:41:33.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.577+0000 7f41affff640 1 -- 172.21.15.37:0/3858316744 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f41b013c930 con 0x7f41b0120380 2024-04-26T00:41:33.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.577+0000 7f41affff640 1 -- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b0120380 msgr2=0x7f41b013a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:33.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.577+0000 7f41affff640 1 --2- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b0120380 0x7f41b013a330 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f41a0015a50 tx=0x7f41a0015f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:33.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.577+0000 7f41acff9640 1 -- 172.21.15.37:0/3858316744 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a0013ea0 con 0x7f41b0120380 2024-04-26T00:41:33.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.577+0000 7f41acff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:33.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.577+0000 7f41acff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:41:33.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.577+0000 7f41acff9640 10 monclient: dump: 2024-04-26T00:41:33.582 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:33.582 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:33.582 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:33.582 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:33.582 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:33.582 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:33.582 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:33.582 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:33.582 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:33.582 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:33.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.581+0000 7f41acff9640 1 -- 172.21.15.37:0/3858316744 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f41a0006b60 con 0x7f41b0120380 2024-04-26T00:41:33.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.581+0000 7f41acff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:33.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.581+0000 7f41acff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:41:33.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.581+0000 7f41acff9640 1 -- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b0120380 msgr2=0x7f41b013a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:33.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.581+0000 7f41acff9640 1 --2- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b0120380 0x7f41b013a330 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f41a0015a50 tx=0x7f41a0015f10 comp rx=0 tx=0).stop 2024-04-26T00:41:33.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.581+0000 7f41acff9640 1 -- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b0120380 msgr2=0x7f41b013a330 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:41:33.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.581+0000 7f41acff9640 1 --2- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b0120380 0x7f41b013a330 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f41a0015a50 tx=0x7f41a0015f10 comp rx=0 tx=0).stop 2024-04-26T00:41:33.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.589+0000 7f41b680c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:41:33.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.589+0000 7f41b680c640 1 --2- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b013a870 0x7f41b009e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:33.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.589+0000 7f41b680c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:33.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.589+0000 7f41b680c640 1 -- 172.21.15.37:0/3858316744 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f41b013ac80 con 0x7f41b013a870 2024-04-26T00:41:33.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.589+0000 7f41affff640 1 --2- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b013a870 0x7f41b009e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:33.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.589+0000 7f41affff640 10 monclient: get_auth_request con 0x7f41b013a870 auth_method 0 2024-04-26T00:41:33.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.589+0000 7f41affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:33.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.589+0000 7f41affff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:33.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.589+0000 7f41affff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:33.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.589+0000 7f41affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:33.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.589+0000 7f41affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:33.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.589+0000 7f41affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:33.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.589+0000 7f41affff640 10 monclient(hunting): handle_auth_done global_id 264102 payload 824 2024-04-26T00:41:33.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.589+0000 7f41affff640 1 -- 172.21.15.37:0/3858316744 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f41b004f510 con 0x7f41b013a870 2024-04-26T00:41:33.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.589+0000 7f41affff640 1 -- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b013a870 msgr2=0x7f41b009e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:33.593 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.589+0000 7f41affff640 1 --2- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b013a870 0x7f41b009e080 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f41a0015a50 tx=0x7f41a0003a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:33.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.593+0000 7f41acff9640 1 -- 172.21.15.37:0/3858316744 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a0017840 con 0x7f41b013a870 2024-04-26T00:41:33.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.593+0000 7f41acff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:33.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.593+0000 7f41acff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:41:33.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.593+0000 7f41acff9640 10 monclient: dump: 2024-04-26T00:41:33.594 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:33.594 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:33.594 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:33.594 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:33.594 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:33.594 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:33.594 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:33.594 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:33.594 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:33.595 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:33.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.593+0000 7f41acff9640 1 -- 172.21.15.37:0/3858316744 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4547 (secure 0 0 0) 0x7f41a0007a00 con 0x7f41b013a870 2024-04-26T00:41:33.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.593+0000 7f41acff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:41:33.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.593+0000 7f41acff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:41:33.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.593+0000 7f41acff9640 1 -- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b013a870 msgr2=0x7f41b009e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:33.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.593+0000 7f41acff9640 1 --2- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b013a870 0x7f41b009e080 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f41a0015a50 tx=0x7f41a0003a80 comp rx=0 tx=0).stop 2024-04-26T00:41:33.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.593+0000 7f41acff9640 1 -- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b013a870 msgr2=0x7f41b009e080 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:41:33.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.593+0000 7f41acff9640 1 --2- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41b013a870 0x7f41b009e080 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f41a0015a50 tx=0x7f41a0003a80 comp rx=0 tx=0).stop 2024-04-26T00:41:33.596 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:41:33.596 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:41:33.596 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:41:33.596 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:41:33.596 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 156, 2024-04-26T00:41:33.596 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 15883, 2024-04-26T00:41:33.596 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:41:33.596 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:41:33.596 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:41:33.596 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:41:33.596 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:33.596 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 3, 2024-04-26T00:41:33.597 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:41:33.597 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:41:33.597 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:41:33.597 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:41:33.597 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:41:33.597 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:41:33.597 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:41:33.597 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:41:33.597 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:41:33.597 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:41:33.597 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:41:33.597 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:41:33.597 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:41:33.597 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:41:33.598 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:33.598 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:41:33.598 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:41:33.598 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:41:33.598 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:41:33.598 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:41:33.598 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:41:33.598 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:41:33.598 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:41:33.598 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:41:33.598 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:41:33.598 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:41:33.598 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:41:33.598 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:41:33.599 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:33.599 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:33.599 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:41:33.599 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:41:33.599 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:41:33.599 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:41:33.599 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:41:33.599 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:41:33.599 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:41:33.599 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:41:33.599 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:41:33.599 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:41:33.599 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:41:33.599 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:41:33.599 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:41:33.599 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:41:33.599 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:41:33.600 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:41:33.600 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:41:33.600 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:41:33.600 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:41:33.600 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:41:33.600 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:41:33.600 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:41:33.600 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:41:33.600 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:41:33.600 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:41:33.600 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:41:33.600 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:41:33.600 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:41:33.601 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:33.601 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:41:33.601 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:33.601 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:41:33.601 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:33.601 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:41:33.601 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:41:33.601 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:41:33.601 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:41:33.601 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:33.601 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:41:33.601 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:41:33.601 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:33.601 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:33.601 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:33.602 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:41:33.602 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:41:33.602 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:33.602 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:33.602 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:33.602 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:33.602 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:41:33.602 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:41:33.602 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:41:33.602 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:41:33.602 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:41:33.602 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:33.602 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:33.603 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:41:33.603 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:41:33.603 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:41:33.603 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:41:33.603 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:33.603 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:41:33.603 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:41:33.603 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:33.603 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:33.603 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:33.603 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:41:33.603 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:41:33.603 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:33.603 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:33.603 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:33.604 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:33.604 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:41:33.604 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:41:33.604 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:41:33.604 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:41:33.604 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:41:33.604 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:33.604 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:33.604 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:41:33.604 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:41:33.604 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:41:33.604 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:41:33.604 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:33.605 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:41:33.605 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:41:33.605 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:33.605 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:33.605 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:33.605 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:41:33.605 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:41:33.605 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:33.605 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:33.605 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:33.605 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:33.605 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:41:33.605 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:41:33.606 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:41:33.606 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:41:33.606 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:41:33.606 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:33.606 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:33.606 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:33.606 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:41:33.606 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:41:33.606 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:33.606 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:41:33.606 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:41:33.606 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:41:33.606 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:33.606 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:33.607 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:41:33.607 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:33.607 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:41:33.607 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:41:33.607 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:41:33.607 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:33.607 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:33.607 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:33.607 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:41:33.607 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:41:33.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.593+0000 7f41b680c640 1 -- 172.21.15.37:0/3858316744 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4190035ed0 msgr2=0x7f4190038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:33.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.593+0000 7f41b680c640 1 --2- 172.21.15.37:0/3858316744 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4190035ed0 0x7f4190038390 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f41b013b460 tx=0x7f4198008040 comp rx=0 tx=0).stop 2024-04-26T00:41:33.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.593+0000 7f41b680c640 10 monclient: shutdown 2024-04-26T00:41:33.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.593+0000 7f41b680c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:41:33.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.593+0000 7f41b680c640 1 -- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41b013c3e0 msgr2=0x7f41b0139df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:33.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.593+0000 7f41b680c640 1 --2- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41b013c3e0 0x7f41b0139df0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f41a80ab000 tx=0x7f41a801ac20 comp rx=0 tx=0).stop 2024-04-26T00:41:33.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.593+0000 7f41b4da9640 1 -- 172.21.15.37:0/3858316744 reap_dead start 2024-04-26T00:41:33.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.593+0000 7f41b680c640 1 -- 172.21.15.37:0/3858316744 shutdown_connections 2024-04-26T00:41:33.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.593+0000 7f41b680c640 1 --2- 172.21.15.37:0/3858316744 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41b013c3e0 0x7f41b0139df0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:33.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.593+0000 7f41b680c640 1 -- 172.21.15.37:0/3858316744 >> 172.21.15.37:0/3858316744 conn(0x7f41b00aa0a0 msgr2=0x7f41b00a8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:41:33.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.593+0000 7f41b680c640 1 -- 172.21.15.37:0/3858316744 shutdown_connections 2024-04-26T00:41:33.609 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.593+0000 7f41b680c640 1 -- 172.21.15.37:0/3858316744 wait complete. 2024-04-26T00:41:33.619 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:41:33.619 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:41:33.620 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:41:33.731 INFO:tasks.rados.rados.0.smithi045.stdout:3824: done (4 left) 2024-04-26T00:41:33.731 INFO:tasks.rados.rados.0.smithi045.stdout:3825: done (3 left) 2024-04-26T00:41:33.731 INFO:tasks.rados.rados.0.smithi045.stdout:3826: done (2 left) 2024-04-26T00:41:33.731 INFO:tasks.rados.rados.0.smithi045.stdout:3827: done (1 left) 2024-04-26T00:41:33.731 INFO:tasks.rados.rados.0.smithi045.stdout:3828: done (0 left) 2024-04-26T00:41:33.731 INFO:tasks.rados.rados.0.smithi045.stdout:3829: read oid 16 snap -1 2024-04-26T00:41:33.731 INFO:tasks.rados.rados.0.smithi045.stdout:3829: expect (ObjNum 1333 snap 411 seq_num 1333) 2024-04-26T00:41:33.731 INFO:tasks.rados.rados.0.smithi045.stdout:3830: delete oid 17 current snap is 423 2024-04-26T00:41:33.738 INFO:tasks.rados.rados.0.smithi045.stdout:3830: done (1 left) 2024-04-26T00:41:33.738 INFO:tasks.rados.rados.0.smithi045.stdout:3831: write oid 50 current snap is 423 2024-04-26T00:41:33.738 INFO:tasks.rados.rados.0.smithi045.stdout:3831: seq_num 1371 ranges {642397=509919,1627754=424802,2733276=309121} 2024-04-26T00:41:33.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.753+0000 7f68ae0ac640 10 monclient: get_monmap_and_config 2024-04-26T00:41:33.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.753+0000 7f68ae0ac640 10 monclient: build_initial_monmap 2024-04-26T00:41:33.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.753+0000 7f68ae0ac640 10 monclient: monmap: 2024-04-26T00:41:33.754 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:41:33.754 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:33.754 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:41:33.756306+0000 2024-04-26T00:41:33.754 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:41:33.756306+0000 2024-04-26T00:41:33.754 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:41:33.755 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:33.755 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:41:33.755 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:41:33.755 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:41:33.755 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:41:33.755 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:41:33.755 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:33.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.753+0000 7f68ae0ac640 1 Processor -- start 2024-04-26T00:41:33.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.753+0000 7f68ae0ac640 1 -- start start 2024-04-26T00:41:33.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.753+0000 7f68ae0ac640 10 monclient: init 2024-04-26T00:41:33.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.753+0000 7f68ae0ac640 10 monclient: _reopen_session rank -1 2024-04-26T00:41:33.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.753+0000 7f68ae0ac640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-04-26T00:41:33.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.753+0000 7f68ae0ac640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f68a80a0300 addr v1:172.21.15.37:6789/0 2024-04-26T00:41:33.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.753+0000 7f68ae0ac640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f68a809fc50 addr v1:172.21.15.45:6789/0 2024-04-26T00:41:33.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.753+0000 7f68ae0ac640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68a813b5e0 0x7f68a813d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:33.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.753+0000 7f68ae0ac640 10 monclient(hunting): picked mon.noname-b con 0x7f68a813b5e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:41:33.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.753+0000 7f68ae0ac640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68a809c190 con 0x7f68a80a0300 2024-04-26T00:41:33.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.753+0000 7f68ae0ac640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68a80a1110 con 0x7f68a809fc50 2024-04-26T00:41:33.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.753+0000 7f68ae0ac640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:33.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.753+0000 7f68ae0ac640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f68a80a16d0 con 0x7f68a813b5e0 2024-04-26T00:41:33.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.753+0000 7f68ae0ac640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:33.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.753+0000 7f68ae0ac640 10 monclient(hunting): authenticate will time out at 2275.062716s 2024-04-26T00:41:33.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.753+0000 7f68a77fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f68a80a0300 0x7f68a80a06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:44384/0 (socket says 172.21.15.37:44384) 2024-04-26T00:41:33.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.753+0000 7f68a77fe640 1 -- 172.21.15.37:0/3492629516 learned_addr learned my addr 172.21.15.37:0/3492629516 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:41:33.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.753+0000 7f68a7fff640 1 --2- 172.21.15.37:0/3492629516 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68a813b5e0 0x7f68a813d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:33.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.753+0000 7f68a7fff640 10 monclient(hunting): get_auth_request con 0x7f68a813b5e0 auth_method 0 2024-04-26T00:41:33.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.753+0000 7f68a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:33.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.753+0000 7f68a7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:33.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.753+0000 7f68a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:33.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.753+0000 7f68a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:33.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.753+0000 7f68a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:33.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.753+0000 7f68a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:33.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.753+0000 7f68a7fff640 10 monclient(hunting): handle_auth_done global_id 254207 payload 274 2024-04-26T00:41:33.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.753+0000 7f68a7fff640 1 -- 172.21.15.37:0/3492629516 >> v1:172.21.15.45:6789/0 conn(0x7f68a809fc50 legacy=0x7f68a8139ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:41:33.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.753+0000 7f68a7fff640 1 -- 172.21.15.37:0/3492629516 >> v1:172.21.15.37:6789/0 conn(0x7f68a80a0300 legacy=0x7f68a80a06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:41:33.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.753+0000 7f68a7fff640 10 monclient: _finish_hunting 0 2024-04-26T00:41:33.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.753+0000 7f68a7fff640 1 monclient: found mon.noname-b 2024-04-26T00:41:33.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.753+0000 7f68a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:33.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.753+0000 7f68a7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:41:33.760 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.753+0000 7f68a7fff640 1 -- 172.21.15.37:0/3492629516 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68a809f2a0 con 0x7f68a813b5e0 2024-04-26T00:41:33.761 INFO:tasks.rados.rados.0.smithi045.stdout:3831: writing smithi04526579-50 from 642397 to 1152316 tid 1 2024-04-26T00:41:33.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.757+0000 7f68a7fff640 1 -- 172.21.15.37:0/3492629516 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68a813b5e0 msgr2=0x7f68a813d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:33.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.757+0000 7f68a7fff640 1 --2- 172.21.15.37:0/3492629516 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68a813b5e0 0x7f68a813d9d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f68a0009dc0 tx=0x7f68a001c710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9570006582cc4ce server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:33.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.757+0000 7f68a6ffd640 1 -- 172.21.15.37:0/3492629516 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a00b2070 con 0x7f68a813b5e0 2024-04-26T00:41:33.763 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.757+0000 7f68a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:33.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.757+0000 7f68a6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:41:33.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.757+0000 7f68a6ffd640 10 monclient: dump: 2024-04-26T00:41:33.764 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:33.764 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:33.764 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:33.764 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:33.764 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:33.764 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:33.764 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:33.764 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:33.764 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:33.764 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:33.764 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.757+0000 7f68a6ffd640 10 monclient: _finish_auth 0 2024-04-26T00:41:33.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.757+0000 7f68a6ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:41:33.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.757+0000 7f68a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:41:33.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.757+0000 7f68a6ffd640 1 -- 172.21.15.37:0/3492629516 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68a00a5890 con 0x7f68a813b5e0 2024-04-26T00:41:33.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.757+0000 7f68a6ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:41:33.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.757+0000 7f68a6ffd640 1 -- 172.21.15.37:0/3492629516 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a00a5c60 con 0x7f68a813b5e0 2024-04-26T00:41:33.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.757+0000 7f68a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:33.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.757+0000 7f68a6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:41:33.765 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.757+0000 7f68a6ffd640 10 monclient: dump: 2024-04-26T00:41:33.765 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:33.765 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:33.765 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:33.765 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:33.765 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:33.765 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:33.765 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:33.766 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:33.766 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:33.766 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:33.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.757+0000 7f68ae0ac640 5 monclient: authenticate success, global_id 254207 2024-04-26T00:41:33.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.757+0000 7f68ae0ac640 10 monclient: get_monmap_and_config success 2024-04-26T00:41:33.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.757+0000 7f68ae0ac640 4 set_mon_vals no callback set 2024-04-26T00:41:33.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68ae0ac640 10 monclient: shutdown 2024-04-26T00:41:33.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68ae0ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:41:33.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68ae0ac640 1 -- 172.21.15.37:0/3492629516 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68a813b5e0 msgr2=0x7f68a813d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:33.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68ae0ac640 1 --2- 172.21.15.37:0/3492629516 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68a813b5e0 0x7f68a813d9d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f68a0009dc0 tx=0x7f68a001c710 comp rx=0 tx=0).stop 2024-04-26T00:41:33.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68ae0ac640 1 -- 172.21.15.37:0/3492629516 shutdown_connections 2024-04-26T00:41:33.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68ae0ac640 1 --2- 172.21.15.37:0/3492629516 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68a813b5e0 0x7f68a813d9d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:33.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68ae0ac640 1 -- 172.21.15.37:0/3492629516 >> 172.21.15.37:0/3492629516 conn(0x7f68a80a1d50 msgr2=0x7f68a80a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:41:33.766 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68ae0ac640 1 -- 172.21.15.37:0/3492629516 shutdown_connections 2024-04-26T00:41:33.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68ae0ac640 1 -- 172.21.15.37:0/3492629516 wait complete. 2024-04-26T00:41:33.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68ae0ac640 10 monclient: build_initial_monmap 2024-04-26T00:41:33.767 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68ae0ac640 10 monclient: monmap: 2024-04-26T00:41:33.767 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:41:33.767 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:41:33.767 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:41:33.767 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:41:33.767 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:41:33.767 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:41:33.767 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:41:33.767 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:41:33.768 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:41:33.768 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:33.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68ae0ac640 1 Processor -- start 2024-04-26T00:41:33.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68ae0ac640 1 -- start start 2024-04-26T00:41:33.768 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68ae0ac640 10 monclient: init 2024-04-26T00:41:33.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68ae0ac640 10 monclient: _reopen_session rank -1 2024-04-26T00:41:33.772 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68ae0ac640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:41:33.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68ae0ac640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68a809fc50 0x7f68a8139980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:33.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68ae0ac640 10 monclient(hunting): picked mon.noname-c con 0x7f68a809fc50 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:41:33.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68ae0ac640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f68a80a0300 0x7f68a8134ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:33.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68ae0ac640 10 monclient(hunting): picked mon.noname-a con 0x7f68a80a0300 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:41:33.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68ae0ac640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68a813b5e0 0x7f68a81cf090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:33.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68ae0ac640 10 monclient(hunting): picked mon.noname-b con 0x7f68a813b5e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:41:33.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68ae0ac640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:33.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68ae0ac640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f68a8140ac0 con 0x7f68a80a0300 2024-04-26T00:41:33.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68ae0ac640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:33.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68ae0ac640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f68a8140940 con 0x7f68a813b5e0 2024-04-26T00:41:33.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68ae0ac640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:33.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68ae0ac640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f68a8140c40 con 0x7f68a809fc50 2024-04-26T00:41:33.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68ae0ac640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:33.773 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68ae0ac640 10 monclient(hunting): authenticate will time out at 2275.070333s 2024-04-26T00:41:33.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68a77fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68a809fc50 0x7f68a8139980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:33.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f689ffff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f68a80a0300 0x7f68a8134ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:33.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68a77fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68a809fc50 0x7f68a8139980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:45208/0 (socket says 172.21.15.37:45208) 2024-04-26T00:41:33.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68a77fe640 1 -- 172.21.15.37:0/1185439150 learned_addr learned my addr 172.21.15.37:0/1185439150 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:41:33.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68a77fe640 10 monclient(hunting): get_auth_request con 0x7f68a809fc50 auth_method 0 2024-04-26T00:41:33.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:33.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68a77fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:33.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:33.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f689ffff640 10 monclient(hunting): get_auth_request con 0x7f68a80a0300 auth_method 0 2024-04-26T00:41:33.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f689ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:33.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f689ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:33.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f689ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:33.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68a77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:33.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68a77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:33.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68a77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:33.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f689ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:33.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f689ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:33.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f689ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:33.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68a7fff640 1 --2- 172.21.15.37:0/1185439150 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68a813b5e0 0x7f68a81cf090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:33.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68a7fff640 10 monclient(hunting): get_auth_request con 0x7f68a813b5e0 auth_method 0 2024-04-26T00:41:33.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:33.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68a7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:33.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:33.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68a77fe640 10 monclient(hunting): handle_auth_done global_id 254210 payload 824 2024-04-26T00:41:33.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68a77fe640 1 -- 172.21.15.37:0/1185439150 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68a813b5e0 msgr2=0x7f68a81cf090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:33.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68a77fe640 1 --2- 172.21.15.37:0/1185439150 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68a813b5e0 0x7f68a81cf090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:33.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68a77fe640 1 -- 172.21.15.37:0/1185439150 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f68a80a0300 msgr2=0x7f68a8134ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:33.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68a77fe640 1 --2- 172.21.15.37:0/1185439150 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f68a80a0300 0x7f68a8134ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:33.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68a77fe640 10 monclient: _finish_hunting 0 2024-04-26T00:41:33.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68a77fe640 1 monclient: found mon.noname-c 2024-04-26T00:41:33.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:33.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68a77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:41:33.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.761+0000 7f68a77fe640 1 -- 172.21.15.37:0/1185439150 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68a813ad50 con 0x7f68a809fc50 2024-04-26T00:41:33.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f68a77fe640 1 -- 172.21.15.37:0/1185439150 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68a809fc50 msgr2=0x7f68a8139980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:33.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f68a77fe640 1 --2- 172.21.15.37:0/1185439150 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68a809fc50 0x7f68a8139980 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f689800c470 tx=0x7f689800c930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:33.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f68a4ff9640 1 -- 172.21.15.37:0/1185439150 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6898007070 con 0x7f68a809fc50 2024-04-26T00:41:33.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f68a4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:33.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f68a4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:41:33.776 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f68a4ff9640 10 monclient: dump: 2024-04-26T00:41:33.776 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:33.776 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:33.776 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:33.777 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:33.777 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:33.777 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:33.777 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:33.777 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:33.777 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:33.777 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:33.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f68a4ff9640 10 monclient: _finish_auth 0 2024-04-26T00:41:33.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f68a4ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:41:33.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f68a4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:41:33.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f68a4ff9640 1 -- 172.21.15.37:0/1185439150 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6898007210 con 0x7f68a809fc50 2024-04-26T00:41:33.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f68ae0ac640 5 monclient: authenticate success, global_id 254210 2024-04-26T00:41:33.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f68ae0ac640 10 monclient: _renew_subs 2024-04-26T00:41:33.777 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f68ae0ac640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:41:33.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f68ae0ac640 1 -- 172.21.15.37:0/1185439150 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68a813a610 con 0x7f68a809fc50 2024-04-26T00:41:33.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f68a4ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:41:33.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f68a4ff9640 1 -- 172.21.15.37:0/1185439150 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6898007600 con 0x7f68a809fc50 2024-04-26T00:41:33.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f68a4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:33.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f68a4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:41:33.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f68a4ff9640 10 monclient: dump: 2024-04-26T00:41:33.778 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:33.778 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:33.778 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:33.778 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:33.778 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:33.778 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:33.778 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:33.778 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:33.779 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:33.779 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:33.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f68a5ffb640 4 set_mon_vals no callback set 2024-04-26T00:41:33.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f68ae0ac640 10 monclient: _renew_subs 2024-04-26T00:41:33.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f68ae0ac640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:41:33.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f68ae0ac640 1 -- 172.21.15.37:0/1185439150 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f68a811fde0 con 0x7f68a809fc50 2024-04-26T00:41:33.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f68ae0ac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:33.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f68ae0ac640 1 --2- 172.21.15.37:0/1185439150 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68a8135220 0x7f68a81cf5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:33.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f68ae0ac640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:33.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f68ae0ac640 1 -- 172.21.15.37:0/1185439150 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f68a8120270 con 0x7f68a8135220 2024-04-26T00:41:33.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f68a4ff9640 1 -- 172.21.15.37:0/1185439150 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f689801b400 con 0x7f68a809fc50 2024-04-26T00:41:33.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f689ffff640 1 --2- 172.21.15.37:0/1185439150 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68a8135220 0x7f68a81cf5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:33.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f689ffff640 10 monclient: get_auth_request con 0x7f68a8135220 auth_method 0 2024-04-26T00:41:33.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f689ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:33.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f689ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:33.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f689ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:33.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f689ffff640 1 -- 172.21.15.37:0/1185439150 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68a8135220 msgr2=0x7f68a81cf5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:41:33.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f689ffff640 1 -- 172.21.15.37:0/1185439150 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68a8135220 msgr2=0x7f68a81cf5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:41:33.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f689ffff640 1 --2- 172.21.15.37:0/1185439150 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68a8135220 0x7f68a81cf5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:41:33.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f689ffff640 1 --2- 172.21.15.37:0/1185439150 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68a8135220 0x7f68a81cf5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:41:33.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f68a4ff9640 1 --2- 172.21.15.37:0/1185439150 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6888035ed0 0x7f6888038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:33.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f68a4ff9640 1 -- 172.21.15.37:0/1185439150 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(860..860 src has 705..860) ==== 3931+0+0 (secure 0 0 0) 0x7f6898048830 con 0x7f68a809fc50 2024-04-26T00:41:33.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f68a7fff640 1 --2- 172.21.15.37:0/1185439150 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6888035ed0 0x7f6888038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:33.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.765+0000 7f68a7fff640 10 monclient: get_auth_request con 0x7f6888035ed0 auth_method 0 2024-04-26T00:41:33.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.769+0000 7f68a7fff640 1 --2- 172.21.15.37:0/1185439150 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6888035ed0 0x7f6888038390 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f68a00ac000 tx=0x7f68a00a7070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:33.781 INFO:tasks.rados.rados.0.smithi045.stdout:3831: writing smithi04526579-50 from 1627754 to 2052556 tid 2 2024-04-26T00:41:33.788 INFO:tasks.rados.rados.0.smithi045.stdout:3831: writing smithi04526579-50 from 2733276 to 3042397 tid 3 2024-04-26T00:41:33.788 INFO:tasks.rados.rados.0.smithi045.stdout:3832: snap_remove snap 420 2024-04-26T00:41:33.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.969+0000 7f689ffff640 1 --2- 172.21.15.37:0/1185439150 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68a8135220 0x7f68a81cf5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:33.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.969+0000 7f689ffff640 10 monclient: get_auth_request con 0x7f68a8135220 auth_method 0 2024-04-26T00:41:33.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.969+0000 7f689ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:33.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.969+0000 7f689ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:33.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.969+0000 7f689ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:41:33.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.969+0000 7f689ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:33.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.969+0000 7f689ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:33.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.969+0000 7f689ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:33.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.969+0000 7f689ffff640 10 monclient(hunting): handle_auth_done global_id 254216 payload 824 2024-04-26T00:41:33.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.969+0000 7f689ffff640 1 -- 172.21.15.37:0/1185439150 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f68a804f510 con 0x7f68a8135220 2024-04-26T00:41:33.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.969+0000 7f689ffff640 1 -- 172.21.15.37:0/1185439150 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68a8135220 msgr2=0x7f68a81cf5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:33.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.969+0000 7f689ffff640 1 --2- 172.21.15.37:0/1185439150 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68a8135220 0x7f68a81cf5d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f68a813a440 tx=0x7f6890011c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:33.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.969+0000 7f68a4ff9640 1 -- 172.21.15.37:0/1185439150 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6890019070 con 0x7f68a8135220 2024-04-26T00:41:33.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.969+0000 7f68a4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:33.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.969+0000 7f68a4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:41:33.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.969+0000 7f68a4ff9640 10 monclient: dump: 2024-04-26T00:41:33.973 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:33.973 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:33.973 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:33.973 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:33.973 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:33.973 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:33.973 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:33.973 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:33.973 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:33.973 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:33.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.973+0000 7f68a4ff9640 1 -- 172.21.15.37:0/1185439150 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6890003b60 con 0x7f68a8135220 2024-04-26T00:41:33.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.973+0000 7f68a4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:33.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.973+0000 7f68a4ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:41:33.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.973+0000 7f68a4ff9640 1 -- 172.21.15.37:0/1185439150 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68a8135220 msgr2=0x7f68a81cf5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:33.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.973+0000 7f68a4ff9640 1 --2- 172.21.15.37:0/1185439150 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68a8135220 0x7f68a81cf5d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f68a813a440 tx=0x7f6890011c80 comp rx=0 tx=0).stop 2024-04-26T00:41:33.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.973+0000 7f68a4ff9640 1 -- 172.21.15.37:0/1185439150 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68a8135220 msgr2=0x7f68a81cf5d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:41:33.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.973+0000 7f68a4ff9640 1 --2- 172.21.15.37:0/1185439150 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68a8135220 0x7f68a81cf5d0 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f68a813a440 tx=0x7f6890011c80 comp rx=0 tx=0).stop 2024-04-26T00:41:33.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.977+0000 7f68ae0ac640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:41:33.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.977+0000 7f68ae0ac640 1 --2- 172.21.15.37:0/1185439150 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68a81cfb10 0x7f68a8135c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:33.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.981+0000 7f68ae0ac640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:33.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.981+0000 7f68ae0ac640 1 -- 172.21.15.37:0/1185439150 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f68a81cff20 con 0x7f68a81cfb10 2024-04-26T00:41:33.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.981+0000 7f689ffff640 1 --2- 172.21.15.37:0/1185439150 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68a81cfb10 0x7f68a8135c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:33.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.981+0000 7f689ffff640 10 monclient: get_auth_request con 0x7f68a81cfb10 auth_method 0 2024-04-26T00:41:33.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.981+0000 7f689ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:33.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.981+0000 7f689ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:33.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.981+0000 7f689ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:33.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.981+0000 7f689ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:33.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.981+0000 7f689ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:33.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.981+0000 7f689ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:33.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.981+0000 7f689ffff640 10 monclient(hunting): handle_auth_done global_id 254219 payload 824 2024-04-26T00:41:33.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.981+0000 7f689ffff640 1 -- 172.21.15.37:0/1185439150 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f68a809eb60 con 0x7f68a81cfb10 2024-04-26T00:41:33.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.981+0000 7f689ffff640 1 -- 172.21.15.37:0/1185439150 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68a81cfb10 msgr2=0x7f68a8135c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:33.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.981+0000 7f689ffff640 1 --2- 172.21.15.37:0/1185439150 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68a81cfb10 0x7f68a8135c70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f68a813a440 tx=0x7f6890003770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:33.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.985+0000 7f68a4ff9640 1 -- 172.21.15.37:0/1185439150 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68900047f0 con 0x7f68a81cfb10 2024-04-26T00:41:33.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.985+0000 7f68a4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:33.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.985+0000 7f68a4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:41:33.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.985+0000 7f68a4ff9640 10 monclient: dump: 2024-04-26T00:41:33.986 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:33.986 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:33.986 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:33.986 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:33.986 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:33.986 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:33.986 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:33.986 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:33.986 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:33.986 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:33.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.985+0000 7f68a4ff9640 1 -- 172.21.15.37:0/1185439150 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4719 (secure 0 0 0) 0x7f68900047f0 con 0x7f68a81cfb10 2024-04-26T00:41:33.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.985+0000 7f68a4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:41:33.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.985+0000 7f68a4ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:41:33.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.985+0000 7f68a4ff9640 1 -- 172.21.15.37:0/1185439150 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68a81cfb10 msgr2=0x7f68a8135c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:33.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.985+0000 7f68a4ff9640 1 --2- 172.21.15.37:0/1185439150 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68a81cfb10 0x7f68a8135c70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f68a813a440 tx=0x7f6890003770 comp rx=0 tx=0).stop 2024-04-26T00:41:33.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.985+0000 7f68a4ff9640 1 -- 172.21.15.37:0/1185439150 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68a81cfb10 msgr2=0x7f68a8135c70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:41:33.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.985+0000 7f68a4ff9640 1 --2- 172.21.15.37:0/1185439150 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68a81cfb10 0x7f68a8135c70 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f68a813a440 tx=0x7f6890003770 comp rx=0 tx=0).stop 2024-04-26T00:41:33.988 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:41:33.988 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:41:33.988 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:41:33.988 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:41:33.988 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 156, 2024-04-26T00:41:33.988 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 62755, 2024-04-26T00:41:33.988 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:41:33.988 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:41:33.988 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:41:33.988 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:41:33.988 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:33.988 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 3, 2024-04-26T00:41:33.988 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:41:33.989 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:41:33.989 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:41:33.989 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:41:33.989 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:41:33.989 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:41:33.989 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:41:33.989 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:41:33.989 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:41:33.989 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:41:33.989 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:41:33.989 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:41:33.989 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:41:33.989 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:41:33.989 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:33.989 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:41:33.990 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:41:33.990 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:41:33.990 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:41:33.990 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:41:33.990 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:41:33.990 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:41:33.990 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:41:33.990 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:41:33.990 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:41:33.990 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:41:33.990 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:41:33.990 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:41:33.990 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:33.990 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:33.991 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:41:33.991 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:41:33.991 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:41:33.991 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:41:33.991 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:41:33.991 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:41:33.991 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:41:33.991 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:41:33.991 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:41:33.991 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:41:33.991 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:41:33.991 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:41:33.991 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:41:33.991 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:41:33.991 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:41:33.992 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:41:33.992 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:41:33.992 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:41:33.992 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:41:33.992 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:41:33.992 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:41:33.992 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:41:33.992 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:41:33.992 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:41:33.992 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:41:33.992 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:41:33.992 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:41:33.992 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:41:33.992 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:33.993 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:41:33.993 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:33.993 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:41:33.993 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:33.993 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:41:33.993 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:41:33.993 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:41:33.993 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:41:33.993 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:33.993 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:41:33.993 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:41:33.993 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:33.993 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:33.993 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:33.993 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:41:33.994 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:41:33.994 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:33.994 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:33.994 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:33.994 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:33.994 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:41:33.994 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:41:33.994 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:41:33.994 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:41:33.994 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:41:33.994 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:33.994 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:33.994 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:41:33.994 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:41:33.995 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:41:33.995 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:41:33.995 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:33.995 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:41:33.995 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:41:33.995 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:33.995 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:33.995 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:33.995 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:41:33.995 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:41:33.995 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:33.995 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:33.995 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:33.995 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:33.995 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:41:33.996 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:41:33.996 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:41:33.996 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:41:33.996 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:41:33.996 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:33.996 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:33.996 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:41:33.996 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:41:33.996 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:41:33.996 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:41:33.996 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:33.996 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:41:33.996 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:41:33.996 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:33.996 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:33.997 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:33.997 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:41:33.997 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:41:33.997 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:33.997 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:33.997 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:33.997 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:33.997 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:41:33.997 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:41:33.997 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:41:33.997 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:41:33.997 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:41:33.997 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:33.997 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:33.998 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:33.998 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:41:33.998 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:41:33.998 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:33.998 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:41:33.998 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:41:33.998 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:41:33.998 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:33.998 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:33.998 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:41:33.998 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:33.998 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:41:33.998 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:41:33.998 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:41:33.999 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:33.999 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:33.999 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:41:33.999 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:33.999 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:41:33.999 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:41:33.999 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:41:33.999 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:33.999 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:33.999 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:33.999 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:41:33.999 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:41:34.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.985+0000 7f68ae0ac640 1 -- 172.21.15.37:0/1185439150 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6888035ed0 msgr2=0x7f6888038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:34.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.985+0000 7f68ae0ac640 1 --2- 172.21.15.37:0/1185439150 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6888035ed0 0x7f6888038390 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f68a00ac000 tx=0x7f68a00a7070 comp rx=0 tx=0).stop 2024-04-26T00:41:34.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.985+0000 7f68ae0ac640 10 monclient: shutdown 2024-04-26T00:41:34.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.985+0000 7f68ae0ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:41:34.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.985+0000 7f68ae0ac640 1 -- 172.21.15.37:0/1185439150 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68a809fc50 msgr2=0x7f68a8139980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:34.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.985+0000 7f68ae0ac640 1 --2- 172.21.15.37:0/1185439150 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f68a809fc50 0x7f68a8139980 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f689800c470 tx=0x7f689800c930 comp rx=0 tx=0).stop 2024-04-26T00:41:34.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.985+0000 7f68a7fff640 1 -- 172.21.15.37:0/1185439150 reap_dead start 2024-04-26T00:41:34.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.985+0000 7f68a7fff640 1 -- 172.21.15.37:0/1185439150 reap_dead start 2024-04-26T00:41:34.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.985+0000 7f68ae0ac640 1 -- 172.21.15.37:0/1185439150 shutdown_connections 2024-04-26T00:41:34.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.985+0000 7f68ae0ac640 1 -- 172.21.15.37:0/1185439150 >> 172.21.15.37:0/1185439150 conn(0x7f68a80a1d50 msgr2=0x7f68a80a3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:41:34.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.985+0000 7f68ae0ac640 1 -- 172.21.15.37:0/1185439150 shutdown_connections 2024-04-26T00:41:34.001 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:33.985+0000 7f68ae0ac640 1 -- 172.21.15.37:0/1185439150 wait complete. 2024-04-26T00:41:34.009 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:41:34.009 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:41:34.009 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:41:34.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb9a72640 10 monclient: get_monmap_and_config 2024-04-26T00:41:34.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb9a72640 10 monclient: build_initial_monmap 2024-04-26T00:41:34.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb9a72640 10 monclient: monmap: 2024-04-26T00:41:34.139 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:41:34.139 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:34.139 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:41:34.140687+0000 2024-04-26T00:41:34.139 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:41:34.140687+0000 2024-04-26T00:41:34.139 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:41:34.139 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:34.139 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:41:34.139 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:41:34.139 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:41:34.139 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:41:34.139 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:41:34.139 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:34.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb9a72640 1 Processor -- start 2024-04-26T00:41:34.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb9a72640 1 -- start start 2024-04-26T00:41:34.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb9a72640 10 monclient: init 2024-04-26T00:41:34.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb9a72640 10 monclient: _reopen_session rank -1 2024-04-26T00:41:34.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb9a72640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-04-26T00:41:34.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb9a72640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7febb40a0300 0x7febb409fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:34.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb9a72640 10 monclient(hunting): picked mon.noname-c con 0x7febb40a0300 addr v2:172.21.15.45:3300/0 2024-04-26T00:41:34.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb9a72640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7febb40a06e0 0x7febb4134ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:34.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb9a72640 10 monclient(hunting): picked mon.noname-b con 0x7febb40a06e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:41:34.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb9a72640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7febb41350e0 addr v1:172.21.15.45:6789/0 2024-04-26T00:41:34.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb9a72640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febb409c190 con 0x7febb41350e0 2024-04-26T00:41:34.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb9a72640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:34.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb9a72640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7febb40a13f0 con 0x7febb40a0300 2024-04-26T00:41:34.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb9a72640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:34.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb9a72640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7febb40a1110 con 0x7febb40a06e0 2024-04-26T00:41:34.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb9a72640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:34.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb9a72640 10 monclient(hunting): authenticate will time out at 2275.446335s 2024-04-26T00:41:34.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb2ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7febb40a06e0 0x7febb4134ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:34.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb2ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7febb40a06e0 0x7febb4134ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:45236/0 (socket says 172.21.15.37:45236) 2024-04-26T00:41:34.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb2ffd640 1 -- 172.21.15.37:0/2918392080 learned_addr learned my addr 172.21.15.37:0/2918392080 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:41:34.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb2ffd640 10 monclient(hunting): get_auth_request con 0x7febb40a06e0 auth_method 0 2024-04-26T00:41:34.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:34.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:34.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:34.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:34.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:34.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:34.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb37fe640 1 --2- 172.21.15.37:0/2918392080 >> v2:172.21.15.45:3300/0 conn(0x7febb40a0300 0x7febb409fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:34.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb37fe640 10 monclient(hunting): get_auth_request con 0x7febb40a0300 auth_method 0 2024-04-26T00:41:34.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:34.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:34.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:34.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb2ffd640 10 monclient(hunting): handle_auth_done global_id 254222 payload 274 2024-04-26T00:41:34.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb2ffd640 1 -- 172.21.15.37:0/2918392080 >> v2:172.21.15.45:3300/0 conn(0x7febb40a0300 msgr2=0x7febb409fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:41:34.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb2ffd640 1 --2- 172.21.15.37:0/2918392080 >> v2:172.21.15.45:3300/0 conn(0x7febb40a0300 0x7febb409fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:34.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb2ffd640 1 -- 172.21.15.37:0/2918392080 >> v1:172.21.15.45:6789/0 conn(0x7febb41350e0 legacy=0x7febb413bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:41:34.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb2ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:41:34.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb2ffd640 1 monclient: found mon.noname-b 2024-04-26T00:41:34.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:34.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:41:34.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.137+0000 7febb2ffd640 1 -- 172.21.15.37:0/2918392080 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febb40a16d0 con 0x7febb40a06e0 2024-04-26T00:41:34.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.141+0000 7febb2ffd640 1 -- 172.21.15.37:0/2918392080 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7febb40a06e0 msgr2=0x7febb4134ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:34.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.141+0000 7febb2ffd640 1 --2- 172.21.15.37:0/2918392080 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7febb40a06e0 0x7febb4134ba0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7feb9c009870 tx=0x7feb9c01c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ef6c4dbfa123f7a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:34.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.141+0000 7febb27fc640 1 -- 172.21.15.37:0/2918392080 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb9c0b0070 con 0x7febb40a06e0 2024-04-26T00:41:34.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.141+0000 7febb27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:34.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.141+0000 7febb27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:41:34.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.141+0000 7febb27fc640 10 monclient: dump: 2024-04-26T00:41:34.146 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:34.146 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:34.146 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:34.146 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:34.146 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:34.146 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:34.146 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:34.146 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:34.146 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:34.147 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:34.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.141+0000 7febb27fc640 10 monclient: _finish_auth 0 2024-04-26T00:41:34.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.141+0000 7febb27fc640 10 monclient: _check_auth_tickets 2024-04-26T00:41:34.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.141+0000 7febb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:41:34.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.141+0000 7febb27fc640 1 -- 172.21.15.37:0/2918392080 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb9c0ab600 con 0x7febb40a06e0 2024-04-26T00:41:34.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.141+0000 7febb9a72640 5 monclient: authenticate success, global_id 254222 2024-04-26T00:41:34.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.141+0000 7febb9a72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:41:34.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.141+0000 7febb27fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:41:34.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.141+0000 7febb9a72640 10 monclient: get_monmap_and_config success 2024-04-26T00:41:34.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.141+0000 7febb9a72640 4 set_mon_vals no callback set 2024-04-26T00:41:34.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.141+0000 7febb27fc640 1 -- 172.21.15.37:0/2918392080 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb9c0ab9d0 con 0x7febb40a06e0 2024-04-26T00:41:34.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.141+0000 7febb27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:34.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.141+0000 7febb27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:41:34.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.141+0000 7febb27fc640 10 monclient: dump: 2024-04-26T00:41:34.147 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:34.148 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:34.148 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:34.148 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:34.148 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:34.148 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:34.148 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:34.148 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:34.148 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:34.148 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:34.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.141+0000 7febb9a72640 10 monclient: shutdown 2024-04-26T00:41:34.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.141+0000 7febb9a72640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:41:34.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.141+0000 7febb9a72640 1 -- 172.21.15.37:0/2918392080 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7febb40a06e0 msgr2=0x7febb4134ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:34.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.141+0000 7febb9a72640 1 --2- 172.21.15.37:0/2918392080 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7febb40a06e0 0x7febb4134ba0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7feb9c009870 tx=0x7feb9c01c410 comp rx=0 tx=0).stop 2024-04-26T00:41:34.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb9a72640 1 -- 172.21.15.37:0/2918392080 shutdown_connections 2024-04-26T00:41:34.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb9a72640 1 --2- 172.21.15.37:0/2918392080 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7febb40a06e0 0x7febb4134ba0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:34.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb9a72640 1 --2- 172.21.15.37:0/2918392080 >> v2:172.21.15.45:3300/0 conn(0x7febb40a0300 0x7febb409fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:34.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb9a72640 1 -- 172.21.15.37:0/2918392080 >> 172.21.15.37:0/2918392080 conn(0x7febb40a1d50 msgr2=0x7febb40a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:41:34.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb9a72640 1 -- 172.21.15.37:0/2918392080 shutdown_connections 2024-04-26T00:41:34.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb9a72640 1 -- 172.21.15.37:0/2918392080 wait complete. 2024-04-26T00:41:34.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb9a72640 10 monclient: build_initial_monmap 2024-04-26T00:41:34.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb9a72640 10 monclient: monmap: 2024-04-26T00:41:34.154 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:41:34.154 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:41:34.154 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:41:34.154 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:41:34.154 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:41:34.154 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:41:34.154 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:41:34.154 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:41:34.154 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:41:34.154 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:34.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb9a72640 1 Processor -- start 2024-04-26T00:41:34.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb9a72640 1 -- start start 2024-04-26T00:41:34.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb9a72640 10 monclient: init 2024-04-26T00:41:34.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb9a72640 10 monclient: _reopen_session rank -1 2024-04-26T00:41:34.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb9a72640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:41:34.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb9a72640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb40a0300 0x7febb41ca950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:34.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb9a72640 10 monclient(hunting): picked mon.noname-b con 0x7febb40a0300 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:41:34.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb9a72640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7febb40a06e0 0x7febb413c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:34.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb9a72640 10 monclient(hunting): picked mon.noname-c con 0x7febb40a06e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:41:34.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb9a72640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7febb41350e0 0x7febb413c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:34.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb9a72640 10 monclient(hunting): picked mon.noname-a con 0x7febb41350e0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:41:34.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb9a72640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:34.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb9a72640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7febb40a0c40 con 0x7febb41350e0 2024-04-26T00:41:34.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb9a72640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:34.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb9a72640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7febb40a0ac0 con 0x7febb40a0300 2024-04-26T00:41:34.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb9a72640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:34.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb9a72640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7febb413fe10 con 0x7febb40a06e0 2024-04-26T00:41:34.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb9a72640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:34.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb9a72640 10 monclient(hunting): authenticate will time out at 2275.453970s 2024-04-26T00:41:34.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb2ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7febb40a06e0 0x7febb413c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:34.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb3fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7febb41350e0 0x7febb413c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:34.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb2ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7febb40a06e0 0x7febb413c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:45248/0 (socket says 172.21.15.37:45248) 2024-04-26T00:41:34.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb2ffd640 1 -- 172.21.15.37:0/2322760998 learned_addr learned my addr 172.21.15.37:0/2322760998 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:41:34.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb2ffd640 10 monclient(hunting): get_auth_request con 0x7febb40a06e0 auth_method 0 2024-04-26T00:41:34.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:34.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:34.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:34.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb3fff640 10 monclient(hunting): get_auth_request con 0x7febb41350e0 auth_method 0 2024-04-26T00:41:34.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:34.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:34.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:34.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb37fe640 1 --2- 172.21.15.37:0/2322760998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb40a0300 0x7febb41ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:34.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:34.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:34.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:34.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb37fe640 10 monclient(hunting): get_auth_request con 0x7febb40a0300 auth_method 0 2024-04-26T00:41:34.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:34.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:34.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:34.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:34.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:34.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:34.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:34.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:34.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:34.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb2ffd640 10 monclient(hunting): handle_auth_done global_id 254225 payload 824 2024-04-26T00:41:34.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb2ffd640 1 -- 172.21.15.37:0/2322760998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb40a0300 msgr2=0x7febb41ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:34.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb2ffd640 1 --2- 172.21.15.37:0/2322760998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb40a0300 0x7febb41ca950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:34.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb2ffd640 1 -- 172.21.15.37:0/2322760998 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7febb41350e0 msgr2=0x7febb413c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:34.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb2ffd640 1 --2- 172.21.15.37:0/2322760998 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7febb41350e0 0x7febb413c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:34.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb2ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:41:34.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb2ffd640 1 monclient: found mon.noname-c 2024-04-26T00:41:34.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:34.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb2ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:41:34.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb2ffd640 1 -- 172.21.15.37:0/2322760998 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febb413ce40 con 0x7febb40a06e0 2024-04-26T00:41:34.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.145+0000 7febb3fff640 1 --2- 172.21.15.37:0/2322760998 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7febb41350e0 0x7febb413c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:41:34.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febb2ffd640 1 -- 172.21.15.37:0/2322760998 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7febb40a06e0 msgr2=0x7febb413c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:34.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febb2ffd640 1 --2- 172.21.15.37:0/2322760998 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7febb40a06e0 0x7febb413c3c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7feb9c01ab20 tx=0x7feb9c019210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:34.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febabfff640 1 -- 172.21.15.37:0/2322760998 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb9c0b0040 con 0x7febb40a06e0 2024-04-26T00:41:34.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febabfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:34.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febabfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:41:34.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febabfff640 10 monclient: dump: 2024-04-26T00:41:34.159 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:34.159 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:34.159 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:34.159 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:34.159 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:34.159 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:34.159 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:34.159 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:34.159 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:34.159 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:34.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febabfff640 10 monclient: _finish_auth 0 2024-04-26T00:41:34.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febabfff640 10 monclient: _check_auth_tickets 2024-04-26T00:41:34.159 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febabfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:41:34.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febabfff640 1 -- 172.21.15.37:0/2322760998 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb9c0aa5b0 con 0x7febb40a06e0 2024-04-26T00:41:34.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febabfff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:41:34.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febabfff640 1 -- 172.21.15.37:0/2322760998 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb9c0aa980 con 0x7febb40a06e0 2024-04-26T00:41:34.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febb9a72640 5 monclient: authenticate success, global_id 254225 2024-04-26T00:41:34.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febb9a72640 10 monclient: _renew_subs 2024-04-26T00:41:34.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febb9a72640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:41:34.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febb9a72640 1 -- 172.21.15.37:0/2322760998 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febb406e580 con 0x7febb40a06e0 2024-04-26T00:41:34.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febb1ffb640 4 set_mon_vals no callback set 2024-04-26T00:41:34.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febabfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:34.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febabfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:41:34.160 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febabfff640 10 monclient: dump: 2024-04-26T00:41:34.160 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:34.160 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:34.160 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:34.161 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:34.161 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:34.161 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:34.161 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:34.161 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:34.161 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:34.161 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:34.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febb9a72640 10 monclient: _renew_subs 2024-04-26T00:41:34.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febb9a72640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:41:34.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febb9a72640 1 -- 172.21.15.37:0/2322760998 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7febb41395e0 con 0x7febb40a06e0 2024-04-26T00:41:34.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febb9a72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:34.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febb9a72640 1 --2- 172.21.15.37:0/2322760998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb41cd430 0x7febb41cd810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:34.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febb9a72640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:34.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febb9a72640 1 -- 172.21.15.37:0/2322760998 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7febb4139de0 con 0x7febb41cd430 2024-04-26T00:41:34.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febabfff640 1 -- 172.21.15.37:0/2322760998 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7feb9c0aab60 con 0x7febb40a06e0 2024-04-26T00:41:34.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febb37fe640 1 --2- 172.21.15.37:0/2322760998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb41cd430 0x7febb41cd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:34.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febb37fe640 10 monclient: get_auth_request con 0x7febb41cd430 auth_method 0 2024-04-26T00:41:34.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febb37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:34.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febb37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:34.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:34.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febabfff640 1 --2- 172.21.15.37:0/2322760998 >> v2:172.21.15.45:6812/2379918623 conn(0x7feb94035e80 0x7feb94038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:34.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febabfff640 1 -- 172.21.15.37:0/2322760998 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(860..860 src has 705..860) ==== 3931+0+0 (secure 0 0 0) 0x7feb9c0e26f0 con 0x7febb40a06e0 2024-04-26T00:41:34.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febb37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:34.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febb37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:34.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febb37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:34.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febb3fff640 1 --2- 172.21.15.37:0/2322760998 >> v2:172.21.15.45:6812/2379918623 conn(0x7feb94035e80 0x7feb94038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:34.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febb3fff640 10 monclient: get_auth_request con 0x7feb94035e80 auth_method 0 2024-04-26T00:41:34.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febb37fe640 10 monclient(hunting): handle_auth_done global_id 224455 payload 824 2024-04-26T00:41:34.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.149+0000 7febb37fe640 1 -- 172.21.15.37:0/2322760998 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7febb404f510 con 0x7febb41cd430 2024-04-26T00:41:34.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.153+0000 7febb37fe640 1 -- 172.21.15.37:0/2322760998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb41cd430 msgr2=0x7febb41cd810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:34.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.153+0000 7febb37fe640 1 --2- 172.21.15.37:0/2322760998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb41cd430 0x7febb41cd810 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7febac004650 tx=0x7febac007700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:34.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.153+0000 7febb3fff640 1 --2- 172.21.15.37:0/2322760998 >> v2:172.21.15.45:6812/2379918623 conn(0x7feb94035e80 0x7feb94038340 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7febb4139c50 tx=0x7feba4007dc0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:34.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.153+0000 7febabfff640 1 -- 172.21.15.37:0/2322760998 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febac014590 con 0x7febb41cd430 2024-04-26T00:41:34.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.153+0000 7febabfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:34.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.153+0000 7febabfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:41:34.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.153+0000 7febabfff640 10 monclient: dump: 2024-04-26T00:41:34.165 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:34.165 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:34.165 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:34.165 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:34.165 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:34.165 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:34.165 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:34.165 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:34.165 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:34.165 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:34.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.153+0000 7febabfff640 1 -- 172.21.15.37:0/2322760998 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7febac01a070 con 0x7febb41cd430 2024-04-26T00:41:34.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.153+0000 7febabfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:34.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.153+0000 7febabfff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:41:34.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.153+0000 7febabfff640 1 -- 172.21.15.37:0/2322760998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb41cd430 msgr2=0x7febb41cd810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:34.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.153+0000 7febabfff640 1 --2- 172.21.15.37:0/2322760998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb41cd430 0x7febb41cd810 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7febac004650 tx=0x7febac007700 comp rx=0 tx=0).stop 2024-04-26T00:41:34.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.153+0000 7febabfff640 1 -- 172.21.15.37:0/2322760998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb41cd430 msgr2=0x7febb41cd810 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:41:34.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.153+0000 7febabfff640 1 --2- 172.21.15.37:0/2322760998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb41cd430 0x7febb41cd810 secure :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7febac004650 tx=0x7febac007700 comp rx=0 tx=0).stop 2024-04-26T00:41:34.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.161+0000 7febb9a72640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:41:34.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.161+0000 7febb9a72640 1 --2- 172.21.15.37:0/2322760998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb41cdd50 0x7febb4070010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:34.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.161+0000 7febb9a72640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:34.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.161+0000 7febb9a72640 1 -- 172.21.15.37:0/2322760998 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7febb409bf70 con 0x7febb41cdd50 2024-04-26T00:41:34.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.161+0000 7febb37fe640 1 --2- 172.21.15.37:0/2322760998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb41cdd50 0x7febb4070010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:34.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.161+0000 7febb37fe640 10 monclient: get_auth_request con 0x7febb41cdd50 auth_method 0 2024-04-26T00:41:34.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.161+0000 7febb37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:34.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.161+0000 7febb37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:34.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.161+0000 7febb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:34.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.161+0000 7febb37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:34.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.161+0000 7febb37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:34.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.161+0000 7febb37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:34.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.161+0000 7febb37fe640 10 monclient(hunting): handle_auth_done global_id 224458 payload 824 2024-04-26T00:41:34.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.161+0000 7febb37fe640 1 -- 172.21.15.37:0/2322760998 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7febb406f0d0 con 0x7febb41cdd50 2024-04-26T00:41:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.165+0000 7febb37fe640 1 -- 172.21.15.37:0/2322760998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb41cdd50 msgr2=0x7febb4070010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.165+0000 7febb37fe640 1 --2- 172.21.15.37:0/2322760998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb41cdd50 0x7febb4070010 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7febac004650 tx=0x7febac004a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.165+0000 7febabfff640 1 -- 172.21.15.37:0/2322760998 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febac0149b0 con 0x7febb41cdd50 2024-04-26T00:41:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.165+0000 7febabfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.165+0000 7febabfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:41:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.165+0000 7febabfff640 10 monclient: dump: 2024-04-26T00:41:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:34.168 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:34.169 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:34.169 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:34.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.165+0000 7febabfff640 1 -- 172.21.15.37:0/2322760998 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4888 (secure 0 0 0) 0x7febac0149b0 con 0x7febb41cdd50 2024-04-26T00:41:34.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.165+0000 7febabfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:41:34.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.165+0000 7febabfff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:41:34.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.165+0000 7febabfff640 1 -- 172.21.15.37:0/2322760998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb41cdd50 msgr2=0x7febb4070010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:34.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.165+0000 7febabfff640 1 --2- 172.21.15.37:0/2322760998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb41cdd50 0x7febb4070010 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7febac004650 tx=0x7febac004a50 comp rx=0 tx=0).stop 2024-04-26T00:41:34.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.165+0000 7febabfff640 1 -- 172.21.15.37:0/2322760998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb41cdd50 msgr2=0x7febb4070010 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:41:34.169 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.165+0000 7febabfff640 1 --2- 172.21.15.37:0/2322760998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febb41cdd50 0x7febb4070010 secure :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7febac004650 tx=0x7febac004a50 comp rx=0 tx=0).stop 2024-04-26T00:41:34.169 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:41:34.169 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:41:34.170 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:41:34.170 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:41:34.170 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 156, 2024-04-26T00:41:34.170 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 207824, 2024-04-26T00:41:34.170 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:41:34.170 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:41:34.170 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:41:34.170 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:41:34.170 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:34.170 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 3, 2024-04-26T00:41:34.170 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:41:34.170 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:41:34.170 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:41:34.170 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:41:34.171 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:41:34.171 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:41:34.171 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:41:34.171 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:41:34.171 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:41:34.171 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:41:34.171 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:41:34.171 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:41:34.171 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:41:34.171 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:41:34.171 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:34.171 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:41:34.171 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:41:34.171 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:41:34.171 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:41:34.172 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:41:34.172 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:41:34.172 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:41:34.172 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:41:34.172 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:41:34.172 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:41:34.172 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:41:34.172 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:41:34.172 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:41:34.172 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:34.172 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:34.172 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:41:34.172 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:41:34.172 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:41:34.172 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:41:34.173 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:41:34.173 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:41:34.173 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:41:34.173 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:41:34.173 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:41:34.173 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:41:34.173 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:41:34.173 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:41:34.173 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:41:34.173 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:41:34.173 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:41:34.173 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:41:34.173 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:41:34.173 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:41:34.174 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:41:34.174 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:41:34.174 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:41:34.174 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:41:34.174 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:41:34.174 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:41:34.174 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:41:34.174 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:41:34.174 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:41:34.174 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:41:34.174 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:34.174 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:41:34.174 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:34.174 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:41:34.175 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:34.175 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:41:34.175 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:41:34.175 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:41:34.175 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:41:34.175 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:34.175 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:41:34.175 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:41:34.175 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:34.175 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:34.175 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:34.175 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:41:34.175 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:41:34.175 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:34.175 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:34.176 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:34.176 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:34.176 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:41:34.176 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:41:34.176 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:41:34.176 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:41:34.176 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:41:34.176 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:34.176 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:34.176 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:41:34.176 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:41:34.176 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:41:34.176 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:41:34.176 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:34.177 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:41:34.177 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:41:34.177 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:34.177 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:34.177 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:34.177 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:41:34.177 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:41:34.177 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:34.177 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:34.177 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:34.177 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:34.177 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:41:34.177 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:41:34.177 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:41:34.177 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:41:34.178 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:41:34.178 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:34.178 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:34.178 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:41:34.178 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:41:34.178 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:41:34.178 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:41:34.178 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:34.178 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:41:34.178 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:41:34.178 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:34.178 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:34.178 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:34.178 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:41:34.178 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:41:34.179 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:34.179 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:34.179 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:34.179 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:34.179 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:41:34.179 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:41:34.179 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:41:34.179 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:41:34.179 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:41:34.179 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:34.179 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:34.179 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:34.179 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:41:34.179 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:41:34.179 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:34.179 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:41:34.180 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:41:34.180 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:41:34.180 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:34.180 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:34.180 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:41:34.180 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:34.180 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:41:34.180 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:41:34.180 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:41:34.180 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:34.180 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:34.180 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:41:34.180 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:34.180 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:41:34.180 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:41:34.181 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:41:34.181 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:34.181 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:34.181 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:41:34.181 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:34.181 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:41:34.181 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:41:34.181 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:41:34.181 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:34.181 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:34.181 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:34.181 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:41:34.181 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:41:34.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.165+0000 7febb9a72640 1 -- 172.21.15.37:0/2322760998 >> v2:172.21.15.45:6812/2379918623 conn(0x7feb94035e80 msgr2=0x7feb94038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:34.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.165+0000 7febb9a72640 1 --2- 172.21.15.37:0/2322760998 >> v2:172.21.15.45:6812/2379918623 conn(0x7feb94035e80 0x7feb94038340 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7febb4139c50 tx=0x7feba4007dc0 comp rx=0 tx=0).stop 2024-04-26T00:41:34.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.165+0000 7febb9a72640 10 monclient: shutdown 2024-04-26T00:41:34.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.165+0000 7febb9a72640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:41:34.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.165+0000 7febb9a72640 1 -- 172.21.15.37:0/2322760998 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7febb40a06e0 msgr2=0x7febb413c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:34.182 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.165+0000 7febb9a72640 1 --2- 172.21.15.37:0/2322760998 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7febb40a06e0 0x7febb413c3c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7feb9c01ab20 tx=0x7feb9c019210 comp rx=0 tx=0).stop 2024-04-26T00:41:34.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.165+0000 7febb3fff640 1 -- 172.21.15.37:0/2322760998 reap_dead start 2024-04-26T00:41:34.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.165+0000 7febb3fff640 1 -- 172.21.15.37:0/2322760998 reap_dead start 2024-04-26T00:41:34.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.165+0000 7febb9a72640 1 -- 172.21.15.37:0/2322760998 shutdown_connections 2024-04-26T00:41:34.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.165+0000 7febb9a72640 1 -- 172.21.15.37:0/2322760998 >> 172.21.15.37:0/2322760998 conn(0x7febb40a1d50 msgr2=0x7febb40a3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:41:34.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.165+0000 7febb9a72640 1 -- 172.21.15.37:0/2322760998 shutdown_connections 2024-04-26T00:41:34.183 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.165+0000 7febb9a72640 1 -- 172.21.15.37:0/2322760998 wait complete. 2024-04-26T00:41:34.184 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:41:34.184 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-26T00:41:34.184 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-26T00:41:34.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.293+0000 7f647225a640 10 monclient: get_monmap_and_config 2024-04-26T00:41:34.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.293+0000 7f647225a640 10 monclient: build_initial_monmap 2024-04-26T00:41:34.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.293+0000 7f647225a640 10 monclient: monmap: 2024-04-26T00:41:34.297 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:41:34.297 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:34.297 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:41:34.299084+0000 2024-04-26T00:41:34.297 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:41:34.299084+0000 2024-04-26T00:41:34.297 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:41:34.297 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:34.297 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:41:34.298 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:41:34.298 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:41:34.298 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:41:34.298 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:41:34.298 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:34.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f647225a640 1 Processor -- start 2024-04-26T00:41:34.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f647225a640 1 -- start start 2024-04-26T00:41:34.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f647225a640 10 monclient: init 2024-04-26T00:41:34.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f647225a640 10 monclient: _reopen_session rank -1 2024-04-26T00:41:34.300 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f647225a640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-04-26T00:41:34.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f647225a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f646c135170 0x7f646c135550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:34.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f647225a640 10 monclient(hunting): picked mon.noname-b con 0x7f646c135170 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:41:34.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f647225a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f646c072b00 addr v1:172.21.15.37:6789/0 2024-04-26T00:41:34.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f647225a640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f646c0a1bc0 0x7f646c13cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:34.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f647225a640 10 monclient(hunting): picked mon.noname-a con 0x7f646c0a1bc0 addr v2:172.21.15.37:3300/0 2024-04-26T00:41:34.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f647225a640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f646c135a90 con 0x7f646c072b00 2024-04-26T00:41:34.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f647225a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:34.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f647225a640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f646c071de0 con 0x7f646c0a1bc0 2024-04-26T00:41:34.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f647225a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:34.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f647225a640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f646c06f310 con 0x7f646c135170 2024-04-26T00:41:34.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f647225a640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:34.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f647225a640 10 monclient(hunting): authenticate will time out at 2275.605173s 2024-04-26T00:41:34.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f646b7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f646c135170 0x7f646c135550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:34.301 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f646bfff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f646c0a1bc0 0x7f646c13cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:34.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f646bfff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f646c0a1bc0 0x7f646c13cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:45946/0 (socket says 172.21.15.37:45946) 2024-04-26T00:41:34.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f646affd640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f646c072b00 0x7f646c0a14b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:44392/0 (socket says 172.21.15.37:44392) 2024-04-26T00:41:34.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f646b7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f646c135170 0x7f646c135550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:45252/0 (socket says 172.21.15.37:45252) 2024-04-26T00:41:34.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f646bfff640 1 -- 172.21.15.37:0/3298065070 learned_addr learned my addr 172.21.15.37:0/3298065070 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:41:34.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f646bfff640 10 monclient(hunting): get_auth_request con 0x7f646c0a1bc0 auth_method 0 2024-04-26T00:41:34.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f646bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:34.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f646bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:34.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f646bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:34.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f646b7fe640 10 monclient(hunting): get_auth_request con 0x7f646c135170 auth_method 0 2024-04-26T00:41:34.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f646b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:34.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f646b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:34.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f646b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:34.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f646bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:34.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f646bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:34.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f646bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:34.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f646b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:34.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f646b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:34.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f646b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:34.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f646b7fe640 10 monclient(hunting): handle_auth_done global_id 254228 payload 274 2024-04-26T00:41:34.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f646b7fe640 1 -- 172.21.15.37:0/3298065070 >> v2:172.21.15.37:3300/0 conn(0x7f646c0a1bc0 msgr2=0x7f646c13cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:41:34.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f646b7fe640 1 --2- 172.21.15.37:0/3298065070 >> v2:172.21.15.37:3300/0 conn(0x7f646c0a1bc0 0x7f646c13cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:34.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f646b7fe640 1 -- 172.21.15.37:0/3298065070 >> v1:172.21.15.37:6789/0 conn(0x7f646c072b00 legacy=0x7f646c0a14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:41:34.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f646b7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:41:34.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f646b7fe640 1 monclient: found mon.noname-b 2024-04-26T00:41:34.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f646b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:34.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f646b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:41:34.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f646b7fe640 1 -- 172.21.15.37:0/3298065070 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f646c0720c0 con 0x7f646c135170 2024-04-26T00:41:34.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.297+0000 7f646bfff640 1 --2- 172.21.15.37:0/3298065070 >> v2:172.21.15.37:3300/0 conn(0x7f646c0a1bc0 0x7f646c13cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:41:34.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.301+0000 7f646b7fe640 1 -- 172.21.15.37:0/3298065070 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f646c135170 msgr2=0x7f646c135550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:34.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.301+0000 7f646b7fe640 1 --2- 172.21.15.37:0/3298065070 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f646c135170 0x7f646c135550 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f646c076680 tx=0x7f645c01c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=297232e3eb0e0fb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:34.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.301+0000 7f646a7fc640 1 -- 172.21.15.37:0/3298065070 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f645c0ab600 con 0x7f646c135170 2024-04-26T00:41:34.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.301+0000 7f646a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:34.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.301+0000 7f646a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:41:34.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.301+0000 7f646a7fc640 10 monclient: dump: 2024-04-26T00:41:34.308 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:34.308 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:34.308 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:34.308 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:34.308 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:34.308 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:34.308 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:34.309 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:34.309 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:34.309 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:34.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.301+0000 7f646a7fc640 10 monclient: _finish_auth 0 2024-04-26T00:41:34.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.301+0000 7f646a7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:41:34.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.301+0000 7f646a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:41:34.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.301+0000 7f646a7fc640 1 -- 172.21.15.37:0/3298065070 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f645c0b0070 con 0x7f646c135170 2024-04-26T00:41:34.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.301+0000 7f647225a640 5 monclient: authenticate success, global_id 254228 2024-04-26T00:41:34.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.301+0000 7f647225a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:41:34.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.301+0000 7f646a7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:41:34.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.301+0000 7f647225a640 10 monclient: get_monmap_and_config success 2024-04-26T00:41:34.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.301+0000 7f646a7fc640 1 -- 172.21.15.37:0/3298065070 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f645c0ab9c0 con 0x7f646c135170 2024-04-26T00:41:34.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.301+0000 7f647225a640 4 set_mon_vals no callback set 2024-04-26T00:41:34.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.301+0000 7f646a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:34.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.301+0000 7f646a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:41:34.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.301+0000 7f646a7fc640 10 monclient: dump: 2024-04-26T00:41:34.310 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:34.310 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:34.310 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:34.310 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:34.310 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:34.310 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:34.310 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:34.310 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:34.310 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:34.310 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:34.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.301+0000 7f647225a640 10 monclient: shutdown 2024-04-26T00:41:34.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.301+0000 7f647225a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:41:34.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.301+0000 7f647225a640 1 -- 172.21.15.37:0/3298065070 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f646c135170 msgr2=0x7f646c135550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:34.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.301+0000 7f647225a640 1 --2- 172.21.15.37:0/3298065070 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f646c135170 0x7f646c135550 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f646c076680 tx=0x7f645c01c420 comp rx=0 tx=0).stop 2024-04-26T00:41:34.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.301+0000 7f647225a640 1 -- 172.21.15.37:0/3298065070 shutdown_connections 2024-04-26T00:41:34.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.301+0000 7f647225a640 1 --2- 172.21.15.37:0/3298065070 >> v2:172.21.15.37:3300/0 conn(0x7f646c0a1bc0 0x7f646c13cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:34.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.301+0000 7f647225a640 1 --2- 172.21.15.37:0/3298065070 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f646c135170 0x7f646c135550 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:34.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.301+0000 7f647225a640 1 -- 172.21.15.37:0/3298065070 >> 172.21.15.37:0/3298065070 conn(0x7f646c0a6300 msgr2=0x7f646c0a2610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:41:34.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.301+0000 7f647225a640 1 -- 172.21.15.37:0/3298065070 shutdown_connections 2024-04-26T00:41:34.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.301+0000 7f647225a640 1 -- 172.21.15.37:0/3298065070 wait complete. 2024-04-26T00:41:34.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.301+0000 7f647225a640 10 monclient: build_initial_monmap 2024-04-26T00:41:34.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.301+0000 7f647225a640 10 monclient: monmap: 2024-04-26T00:41:34.311 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:41:34.311 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:41:34.311 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:41:34.311 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:41:34.312 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:41:34.312 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:41:34.312 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:41:34.312 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:41:34.312 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:41:34.312 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:34.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.301+0000 7f647225a640 1 Processor -- start 2024-04-26T00:41:34.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f647225a640 1 -- start start 2024-04-26T00:41:34.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f647225a640 10 monclient: init 2024-04-26T00:41:34.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f647225a640 10 monclient: _reopen_session rank -1 2024-04-26T00:41:34.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f647225a640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:41:34.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f647225a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f646c072b00 0x7f646c1cf280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:34.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f647225a640 10 monclient(hunting): picked mon.noname-b con 0x7f646c072b00 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:41:34.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f647225a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f646c0a1bc0 0x7f646c1cf7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:34.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f647225a640 10 monclient(hunting): picked mon.noname-a con 0x7f646c0a1bc0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:41:34.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f647225a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f646c135170 0x7f646c1cfd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:34.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f647225a640 10 monclient(hunting): picked mon.noname-c con 0x7f646c135170 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:41:34.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f647225a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:34.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f647225a640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f646c13fdc0 con 0x7f646c0a1bc0 2024-04-26T00:41:34.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f647225a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:34.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f647225a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f646c13fc40 con 0x7f646c072b00 2024-04-26T00:41:34.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f647225a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:34.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f647225a640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f646c13ff40 con 0x7f646c135170 2024-04-26T00:41:34.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f647225a640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:34.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f647225a640 10 monclient(hunting): authenticate will time out at 2275.612546s 2024-04-26T00:41:34.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f646bfff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f646c135170 0x7f646c1cfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:34.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f646affd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f646c0a1bc0 0x7f646c1cf7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:34.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f646bfff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f646c135170 0x7f646c1cfd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:45264/0 (socket says 172.21.15.37:45264) 2024-04-26T00:41:34.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f646affd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f646c0a1bc0 0x7f646c1cf7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:45950/0 (socket says 172.21.15.37:45950) 2024-04-26T00:41:34.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f646bfff640 1 -- 172.21.15.37:0/2703811423 learned_addr learned my addr 172.21.15.37:0/2703811423 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:41:34.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f646bfff640 10 monclient(hunting): get_auth_request con 0x7f646c135170 auth_method 0 2024-04-26T00:41:34.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f646bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:34.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f646bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:34.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f646bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:34.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f646affd640 10 monclient(hunting): get_auth_request con 0x7f646c0a1bc0 auth_method 0 2024-04-26T00:41:34.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f646affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:34.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f646affd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:34.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f646affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:34.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f646bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:34.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f646bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:34.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f646bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:34.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f646affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:34.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f646affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:34.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f646affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:34.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f646b7fe640 1 --2- 172.21.15.37:0/2703811423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f646c072b00 0x7f646c1cf280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:34.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f646b7fe640 10 monclient(hunting): get_auth_request con 0x7f646c072b00 auth_method 0 2024-04-26T00:41:34.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f646b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:34.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f646b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:34.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f646b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:34.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f646bfff640 10 monclient(hunting): handle_auth_done global_id 254231 payload 824 2024-04-26T00:41:34.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f646bfff640 1 -- 172.21.15.37:0/2703811423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f646c072b00 msgr2=0x7f646c1cf280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:34.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f646bfff640 1 --2- 172.21.15.37:0/2703811423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f646c072b00 0x7f646c1cf280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:34.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f646bfff640 1 -- 172.21.15.37:0/2703811423 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f646c0a1bc0 msgr2=0x7f646c1cf7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:34.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f646bfff640 1 --2- 172.21.15.37:0/2703811423 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f646c0a1bc0 0x7f646c1cf7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:34.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f646bfff640 10 monclient: _finish_hunting 0 2024-04-26T00:41:34.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f646bfff640 1 monclient: found mon.noname-c 2024-04-26T00:41:34.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f646bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:34.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f646bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:41:34.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f646bfff640 1 -- 172.21.15.37:0/2703811423 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f646c13b5d0 con 0x7f646c135170 2024-04-26T00:41:34.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f646bfff640 1 -- 172.21.15.37:0/2703811423 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f646c135170 msgr2=0x7f646c1cfd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:34.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f646bfff640 1 --2- 172.21.15.37:0/2703811423 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f646c135170 0x7f646c1cfd00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6464009f10 tx=0x7f646400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:34.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f6453fff640 1 -- 172.21.15.37:0/2703811423 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f646400ce50 con 0x7f646c135170 2024-04-26T00:41:34.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f6453fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:34.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f6453fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:41:34.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f6453fff640 10 monclient: dump: 2024-04-26T00:41:34.318 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:34.318 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:34.318 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:34.318 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:34.318 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:34.318 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:34.318 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:34.318 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:34.318 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:34.318 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:34.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f6453fff640 10 monclient: _finish_auth 0 2024-04-26T00:41:34.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f6453fff640 10 monclient: _check_auth_tickets 2024-04-26T00:41:34.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f6453fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:41:34.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f6453fff640 1 -- 172.21.15.37:0/2703811423 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64640039a0 con 0x7f646c135170 2024-04-26T00:41:34.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f6453fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:41:34.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f6453fff640 1 -- 172.21.15.37:0/2703811423 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6464003d70 con 0x7f646c135170 2024-04-26T00:41:34.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f6453fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:34.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f6453fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:41:34.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f6453fff640 10 monclient: dump: 2024-04-26T00:41:34.319 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:34.319 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:34.319 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:34.319 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:34.319 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:34.320 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:34.320 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:34.320 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:34.320 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:34.320 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:34.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f647225a640 5 monclient: authenticate success, global_id 254231 2024-04-26T00:41:34.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f647225a640 10 monclient: _renew_subs 2024-04-26T00:41:34.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f647225a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:41:34.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f647225a640 1 -- 172.21.15.37:0/2703811423 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f646c13b8b0 con 0x7f646c135170 2024-04-26T00:41:34.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f6469ffb640 4 set_mon_vals no callback set 2024-04-26T00:41:34.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f647225a640 10 monclient: _renew_subs 2024-04-26T00:41:34.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f647225a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:41:34.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.305+0000 7f647225a640 1 -- 172.21.15.37:0/2703811423 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f646c06efe0 con 0x7f646c135170 2024-04-26T00:41:34.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.309+0000 7f647225a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:34.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.309+0000 7f647225a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:34.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.309+0000 7f647225a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:41:34.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.309+0000 7f647225a640 1 -- 172.21.15.37:0/2703811423 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f646c132df0 con 0x7f646c135170 2024-04-26T00:41:34.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.309+0000 7f6453fff640 1 -- 172.21.15.37:0/2703811423 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f646401b030 con 0x7f646c135170 2024-04-26T00:41:34.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.309+0000 7f6453fff640 1 --2- 172.21.15.37:0/2703811423 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6458035ed0 0x7f6458038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:34.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.309+0000 7f6453fff640 1 -- 172.21.15.37:0/2703811423 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(860..860 src has 705..860) ==== 3931+0+0 (secure 0 0 0) 0x7f6464048eb0 con 0x7f646c135170 2024-04-26T00:41:34.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.309+0000 7f646b7fe640 1 --2- 172.21.15.37:0/2703811423 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6458035ed0 0x7f6458038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:34.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.309+0000 7f646b7fe640 10 monclient: get_auth_request con 0x7f6458035ed0 auth_method 0 2024-04-26T00:41:34.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.313+0000 7f646b7fe640 1 --2- 172.21.15.37:0/2703811423 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6458035ed0 0x7f6458038390 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f645c0ac000 tx=0x7f645c0a7000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:34.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.313+0000 7f6453fff640 1 -- 172.21.15.37:0/2703811423 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6464022020 con 0x7f646c135170 2024-04-26T00:41:34.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.313+0000 7f6453fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:34.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.313+0000 7f6453fff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:41:34.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.533+0000 7f647225a640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-26T00:41:34.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.537+0000 7f647225a640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-26T00:41:34.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.537+0000 7f647225a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:41:34.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.537+0000 7f647225a640 1 -- 172.21.15.37:0/2703811423 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f646c066f40 con 0x7f646c135170 2024-04-26T00:41:34.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.537+0000 7f6453fff640 1 -- 172.21.15.37:0/2703811423 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f6464010040 con 0x7f646c135170 2024-04-26T00:41:34.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.537+0000 7f6453fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-26T00:41:34.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.537+0000 7f6453fff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:41:34.540 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.537+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:34.540 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.537+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=27,config=2,health=10,kv=10,logm=51} crc {auth=3498245890,config=692619927,health=300290769,kv=1034996044,logm=764422474}) 2024-04-26T00:41:34.540 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.537+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=12,kv=10,logm=52} crc {auth=1726208403,config=692619927,health=1497466813,kv=1034996044,logm=3988150293}) 2024-04-26T00:41:34.540 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.537+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:34.540 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.537+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=27,config=2,health=10,kv=10,logm=51} crc {auth=3498245890,config=692619927,health=300290769,kv=1034996044,logm=764422474}) 2024-04-26T00:41:34.540 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.537+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=27,config=2,health=11,kv=10,logm=50} crc {auth=3498245890,config=692619927,health=1140756147,kv=1034996044,logm=4286379263}) 2024-04-26T00:41:34.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.541+0000 7f647225a640 1 -- 172.21.15.37:0/2703811423 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6458035ed0 msgr2=0x7f6458038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:34.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.541+0000 7f647225a640 1 --2- 172.21.15.37:0/2703811423 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6458035ed0 0x7f6458038390 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f645c0ac000 tx=0x7f645c0a7000 comp rx=0 tx=0).stop 2024-04-26T00:41:34.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.541+0000 7f647225a640 10 monclient: shutdown 2024-04-26T00:41:34.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.541+0000 7f647225a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:41:34.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.541+0000 7f647225a640 1 -- 172.21.15.37:0/2703811423 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f646c135170 msgr2=0x7f646c1cfd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:34.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.541+0000 7f647225a640 1 --2- 172.21.15.37:0/2703811423 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f646c135170 0x7f646c1cfd00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6464009f10 tx=0x7f646400c490 comp rx=0 tx=0).stop 2024-04-26T00:41:34.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.541+0000 7f647225a640 1 -- 172.21.15.37:0/2703811423 shutdown_connections 2024-04-26T00:41:34.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.541+0000 7f647225a640 1 --2- 172.21.15.37:0/2703811423 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6458035ed0 0x7f6458038390 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:34.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.541+0000 7f647225a640 1 --2- 172.21.15.37:0/2703811423 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f646c135170 0x7f646c1cfd00 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:34.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.541+0000 7f647225a640 1 --2- 172.21.15.37:0/2703811423 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f646c0a1bc0 0x7f646c1cf7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:34.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.541+0000 7f647225a640 1 --2- 172.21.15.37:0/2703811423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f646c072b00 0x7f646c1cf280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:34.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.541+0000 7f647225a640 1 -- 172.21.15.37:0/2703811423 >> 172.21.15.37:0/2703811423 conn(0x7f646c0a6300 msgr2=0x7f646c09f030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:41:34.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.541+0000 7f647225a640 1 -- 172.21.15.37:0/2703811423 shutdown_connections 2024-04-26T00:41:34.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:34.541+0000 7f647225a640 1 -- 172.21.15.37:0/2703811423 wait complete. 2024-04-26T00:41:34.544 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.541+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:34.544 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.541+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3297569407}) 2024-04-26T00:41:34.544 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.541+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3773393456}) 2024-04-26T00:41:34.544 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.541+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:34.544 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.541+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3297569407}) 2024-04-26T00:41:34.544 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.541+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1743311118}) 2024-04-26T00:41:34.546 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.545+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:34.546 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.545+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1902879303}) 2024-04-26T00:41:34.546 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.545+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3888508775}) 2024-04-26T00:41:34.546 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.545+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:34.546 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.545+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1902879303}) 2024-04-26T00:41:34.546 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.545+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2503925017}) 2024-04-26T00:41:34.549 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.545+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:34.549 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.545+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=767300526}) 2024-04-26T00:41:34.549 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.545+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1633864451}) 2024-04-26T00:41:34.549 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.545+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:34.549 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.545+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=767300526}) 2024-04-26T00:41:34.549 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.545+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1131203921}) 2024-04-26T00:41:34.552 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.549+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:34.552 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.549+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1072073423}) 2024-04-26T00:41:34.552 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.549+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1025769666}) 2024-04-26T00:41:34.552 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.549+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:34.552 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.549+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1072073423}) 2024-04-26T00:41:34.552 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.549+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1300617073}) 2024-04-26T00:41:34.555 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.553+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:34.555 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.553+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3339598884}) 2024-04-26T00:41:34.555 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.553+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2265408811}) 2024-04-26T00:41:34.555 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.553+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:34.555 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.553+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3339598884}) 2024-04-26T00:41:34.555 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.553+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3919267772}) 2024-04-26T00:41:34.557 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.553+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:34.557 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.553+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1326903345}) 2024-04-26T00:41:34.557 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.553+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3969359428}) 2024-04-26T00:41:34.557 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.553+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:34.557 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.553+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1326903345}) 2024-04-26T00:41:34.557 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.553+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=602565316}) 2024-04-26T00:41:34.559 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.557+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:34.559 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.557+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2638117869}) 2024-04-26T00:41:34.560 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.557+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3421785062}) 2024-04-26T00:41:34.560 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.557+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:34.560 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.557+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2638117869}) 2024-04-26T00:41:34.560 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.557+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=562922042}) 2024-04-26T00:41:34.561 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-26T00:41:34.563 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.561+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:34.563 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.561+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2047433672}) 2024-04-26T00:41:34.563 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.561+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=670546055}) 2024-04-26T00:41:34.563 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.561+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:34.563 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.561+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2047433672}) 2024-04-26T00:41:34.563 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.561+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3989577917}) 2024-04-26T00:41:34.566 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.565+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:34.567 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.565+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2542110074}) 2024-04-26T00:41:34.567 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.565+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3357129169}) 2024-04-26T00:41:34.567 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.565+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:34.567 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.565+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2542110074}) 2024-04-26T00:41:34.567 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.565+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3688293275}) 2024-04-26T00:41:34.570 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.569+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:34.570 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.569+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1170012233}) 2024-04-26T00:41:34.570 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.569+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3159250038}) 2024-04-26T00:41:34.570 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.569+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:34.570 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.569+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1170012233}) 2024-04-26T00:41:34.570 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.569+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3381416931}) 2024-04-26T00:41:34.573 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.569+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:34.573 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.569+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=307760757}) 2024-04-26T00:41:34.573 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.569+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=21954312}) 2024-04-26T00:41:34.573 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.569+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:34.573 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.569+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=307760757}) 2024-04-26T00:41:34.573 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.569+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=108798798}) 2024-04-26T00:41:34.576 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.573+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:34.577 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.573+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=128943125}) 2024-04-26T00:41:34.577 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.573+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=95425135}) 2024-04-26T00:41:34.577 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.573+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:34.577 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.573+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=128943125}) 2024-04-26T00:41:34.577 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.573+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=840691438}) 2024-04-26T00:41:34.579 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.577+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:34.579 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.577+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3026473695}) 2024-04-26T00:41:34.580 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.577+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=399608675}) 2024-04-26T00:41:34.580 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.577+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:34.580 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.577+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3026473695}) 2024-04-26T00:41:34.580 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.577+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2286470828}) 2024-04-26T00:41:34.583 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.581+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:34.583 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.581+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1492868877}) 2024-04-26T00:41:34.583 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.581+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1669489866}) 2024-04-26T00:41:34.583 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.581+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:34.583 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.581+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1492868877}) 2024-04-26T00:41:34.583 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.581+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2178259350}) 2024-04-26T00:41:34.586 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.585+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:34.586 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.585+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2322273883}) 2024-04-26T00:41:34.586 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.585+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4214329308}) 2024-04-26T00:41:34.586 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.585+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:34.586 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.585+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2322273883}) 2024-04-26T00:41:34.586 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.585+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=686997294}) 2024-04-26T00:41:34.728 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.725+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:34.728 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.725+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=492055589}) 2024-04-26T00:41:34.728 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.725+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1464849515}) 2024-04-26T00:41:34.728 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.725+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:34.728 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.725+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=492055589}) 2024-04-26T00:41:34.728 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.725+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=342285121}) 2024-04-26T00:41:34.738 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.737+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:34.738 INFO:tasks.rados.rados.0.smithi045.stdout:3832: done (2 left) 2024-04-26T00:41:34.739 INFO:tasks.rados.rados.0.smithi045.stdout:3833: snap_remove snap 404 2024-04-26T00:41:34.739 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.737+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3859865216}) 2024-04-26T00:41:34.739 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.737+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1693988034}) 2024-04-26T00:41:34.739 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.737+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:34.739 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.737+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3859865216}) 2024-04-26T00:41:34.739 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.737+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=370256730}) 2024-04-26T00:41:34.742 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.741+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:34.742 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.741+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1839217368}) 2024-04-26T00:41:34.742 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.741+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1597099396}) 2024-04-26T00:41:34.742 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.741+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:34.742 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.741+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1839217368}) 2024-04-26T00:41:34.743 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:34.741+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=500332161}) 2024-04-26T00:41:35.087 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.085+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.087 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.085+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4110539644}) 2024-04-26T00:41:35.087 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.085+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3720640515}) 2024-04-26T00:41:35.087 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.085+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.087 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.085+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4110539644}) 2024-04-26T00:41:35.087 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.085+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3469027610}) 2024-04-26T00:41:35.089 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.085+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.089 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.085+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4209496948}) 2024-04-26T00:41:35.089 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.085+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3229698386}) 2024-04-26T00:41:35.089 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.085+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.089 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.085+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4209496948}) 2024-04-26T00:41:35.089 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.085+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1374848785}) 2024-04-26T00:41:35.091 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.089+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.091 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.089+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=802034545}) 2024-04-26T00:41:35.091 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.089+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=682614012}) 2024-04-26T00:41:35.091 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.089+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.091 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.089+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=802034545}) 2024-04-26T00:41:35.091 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.089+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=467985551}) 2024-04-26T00:41:35.093 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.089+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.093 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.089+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3947472651}) 2024-04-26T00:41:35.093 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.089+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1243358607}) 2024-04-26T00:41:35.093 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.089+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.094 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.093+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3947472651}) 2024-04-26T00:41:35.094 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.093+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=163185377}) 2024-04-26T00:41:35.095 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.093+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.096 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.093+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2568534316}) 2024-04-26T00:41:35.096 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.093+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3141268795}) 2024-04-26T00:41:35.096 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.093+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.096 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.093+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2568534316}) 2024-04-26T00:41:35.096 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.093+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3230718563}) 2024-04-26T00:41:35.098 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.097+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.098 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.097+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3478291395}) 2024-04-26T00:41:35.098 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.097+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3906653640}) 2024-04-26T00:41:35.098 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.097+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.098 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.097+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3478291395}) 2024-04-26T00:41:35.098 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.097+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=420721827}) 2024-04-26T00:41:35.100 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.097+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.100 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.097+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1002906697}) 2024-04-26T00:41:35.100 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.097+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2671385173}) 2024-04-26T00:41:35.100 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.097+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.100 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.097+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1002906697}) 2024-04-26T00:41:35.100 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.097+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2543338159}) 2024-04-26T00:41:35.102 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.101+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.102 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.101+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1476442292}) 2024-04-26T00:41:35.102 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.101+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2233189622}) 2024-04-26T00:41:35.102 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.101+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.102 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.101+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1476442292}) 2024-04-26T00:41:35.103 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.101+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3897122746}) 2024-04-26T00:41:35.104 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.101+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.105 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.101+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=912668801}) 2024-04-26T00:41:35.105 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.101+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2825046961}) 2024-04-26T00:41:35.105 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.101+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.105 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.101+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=912668801}) 2024-04-26T00:41:35.105 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.101+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1628357528}) 2024-04-26T00:41:35.107 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.105+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.107 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.105+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2667481919}) 2024-04-26T00:41:35.107 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.105+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2464689186}) 2024-04-26T00:41:35.107 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.105+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.107 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.105+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2667481919}) 2024-04-26T00:41:35.107 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.105+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2167770774}) 2024-04-26T00:41:35.109 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.105+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.109 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.105+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1894273475}) 2024-04-26T00:41:35.109 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.105+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=948610489}) 2024-04-26T00:41:35.109 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.105+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.109 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.105+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1894273475}) 2024-04-26T00:41:35.109 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.105+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1686342050}) 2024-04-26T00:41:35.111 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.109+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.111 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.109+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1475789130}) 2024-04-26T00:41:35.111 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.109+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=724896943}) 2024-04-26T00:41:35.112 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.109+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.112 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.109+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1475789130}) 2024-04-26T00:41:35.112 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.109+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4208056626}) 2024-04-26T00:41:35.114 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.113+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.114 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.113+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1835897568}) 2024-04-26T00:41:35.114 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.113+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3339967879}) 2024-04-26T00:41:35.114 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.113+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.114 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.113+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1835897568}) 2024-04-26T00:41:35.114 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.113+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4179779405}) 2024-04-26T00:41:35.116 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.113+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.116 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.113+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2541710592}) 2024-04-26T00:41:35.116 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.113+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2397480054}) 2024-04-26T00:41:35.116 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.113+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.116 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.113+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2541710592}) 2024-04-26T00:41:35.117 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.113+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1133243874}) 2024-04-26T00:41:35.118 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.117+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.118 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.117+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3508283463}) 2024-04-26T00:41:35.118 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.117+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=725966013}) 2024-04-26T00:41:35.118 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.117+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.119 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.117+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3508283463}) 2024-04-26T00:41:35.119 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.117+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2544541281}) 2024-04-26T00:41:35.120 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.117+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.121 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.117+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1407484215}) 2024-04-26T00:41:35.121 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.117+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3987219464}) 2024-04-26T00:41:35.121 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.117+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.121 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.117+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1407484215}) 2024-04-26T00:41:35.121 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.117+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2500295814}) 2024-04-26T00:41:35.123 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.121+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.123 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.121+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=548543848}) 2024-04-26T00:41:35.123 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.121+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3902919648}) 2024-04-26T00:41:35.123 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.121+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.123 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.121+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=548543848}) 2024-04-26T00:41:35.124 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.121+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1488544423}) 2024-04-26T00:41:35.125 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.121+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.125 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.121+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1743964920}) 2024-04-26T00:41:35.125 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.121+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=898931958}) 2024-04-26T00:41:35.125 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.121+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.125 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.121+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1743964920}) 2024-04-26T00:41:35.126 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.121+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=860475247}) 2024-04-26T00:41:35.127 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.125+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.128 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.125+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1324770563}) 2024-04-26T00:41:35.128 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.125+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3799812579}) 2024-04-26T00:41:35.128 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.125+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.128 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.125+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1324770563}) 2024-04-26T00:41:35.128 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.125+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2898572261}) 2024-04-26T00:41:35.130 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.129+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.130 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.129+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=352772410}) 2024-04-26T00:41:35.130 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.129+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1845266960}) 2024-04-26T00:41:35.130 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.129+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.130 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.129+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=352772410}) 2024-04-26T00:41:35.130 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.129+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1326456543}) 2024-04-26T00:41:35.134 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.129+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.135 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.129+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3038366524}) 2024-04-26T00:41:35.135 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.129+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1783332745}) 2024-04-26T00:41:35.135 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.129+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.135 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.129+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3038366524}) 2024-04-26T00:41:35.135 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.129+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=644419294}) 2024-04-26T00:41:35.135 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.133+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.135 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.133+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1496147304}) 2024-04-26T00:41:35.135 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.133+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2649752240}) 2024-04-26T00:41:35.135 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.133+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.135 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.133+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1496147304}) 2024-04-26T00:41:35.135 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.133+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3638998866}) 2024-04-26T00:41:35.136 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.133+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.137 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.133+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2784142015}) 2024-04-26T00:41:35.137 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.133+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3993467545}) 2024-04-26T00:41:35.137 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.133+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.137 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.133+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2784142015}) 2024-04-26T00:41:35.137 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.133+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3499903693}) 2024-04-26T00:41:35.139 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.137+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.139 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.137+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=152220643}) 2024-04-26T00:41:35.139 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.137+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1066534721}) 2024-04-26T00:41:35.139 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.137+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.139 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.137+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=152220643}) 2024-04-26T00:41:35.139 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.137+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3399370655}) 2024-04-26T00:41:35.141 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.137+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.141 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.137+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=552847325}) 2024-04-26T00:41:35.141 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.141+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=333932534}) 2024-04-26T00:41:35.142 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.141+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.142 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.141+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=552847325}) 2024-04-26T00:41:35.142 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.141+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3045667396}) 2024-04-26T00:41:35.144 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.141+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.144 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.141+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=866980012}) 2024-04-26T00:41:35.144 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.141+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4246455003}) 2024-04-26T00:41:35.144 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.141+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.144 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.141+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=866980012}) 2024-04-26T00:41:35.144 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.141+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3060317139}) 2024-04-26T00:41:35.146 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.145+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.146 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.145+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3835588539}) 2024-04-26T00:41:35.146 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.145+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3491968058}) 2024-04-26T00:41:35.146 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.145+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.146 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.145+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3835588539}) 2024-04-26T00:41:35.146 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.145+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3478204677}) 2024-04-26T00:41:35.148 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.145+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.148 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.145+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=921325220}) 2024-04-26T00:41:35.148 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.145+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1975629376}) 2024-04-26T00:41:35.149 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.145+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.149 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.145+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=921325220}) 2024-04-26T00:41:35.149 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.145+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=908407996}) 2024-04-26T00:41:35.151 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.149+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.151 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.149+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3936095066}) 2024-04-26T00:41:35.151 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.149+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1832679621}) 2024-04-26T00:41:35.151 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.149+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.151 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.149+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3936095066}) 2024-04-26T00:41:35.151 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.149+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1805885206}) 2024-04-26T00:41:35.153 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.149+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.153 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.149+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1632904533}) 2024-04-26T00:41:35.153 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.149+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2007542333}) 2024-04-26T00:41:35.153 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.149+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.153 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.149+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1632904533}) 2024-04-26T00:41:35.153 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.149+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3415635564}) 2024-04-26T00:41:35.155 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.153+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.156 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.153+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1418813694}) 2024-04-26T00:41:35.156 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.153+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3088779894}) 2024-04-26T00:41:35.156 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.153+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.156 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.153+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1418813694}) 2024-04-26T00:41:35.156 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.153+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2758064035}) 2024-04-26T00:41:35.158 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.157+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.158 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.157+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2084045211}) 2024-04-26T00:41:35.158 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.157+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2186502288}) 2024-04-26T00:41:35.158 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.157+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.158 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.157+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2084045211}) 2024-04-26T00:41:35.158 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.157+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4201815933}) 2024-04-26T00:41:35.160 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.157+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.160 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.157+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3159319044}) 2024-04-26T00:41:35.160 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.157+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2184123538}) 2024-04-26T00:41:35.160 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.157+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.160 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.157+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3159319044}) 2024-04-26T00:41:35.160 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.157+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3321584112}) 2024-04-26T00:41:35.162 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.161+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.163 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.161+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1718588047}) 2024-04-26T00:41:35.163 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.161+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2229624803}) 2024-04-26T00:41:35.163 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.161+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.163 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.161+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1718588047}) 2024-04-26T00:41:35.163 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.161+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1503423313}) 2024-04-26T00:41:35.165 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.161+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.165 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.161+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1282637201}) 2024-04-26T00:41:35.165 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.161+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2920674244}) 2024-04-26T00:41:35.166 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.161+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.166 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.161+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1282637201}) 2024-04-26T00:41:35.166 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.161+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2805720140}) 2024-04-26T00:41:35.176 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.173+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.176 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.173+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2318333205}) 2024-04-26T00:41:35.176 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.173+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2258916917}) 2024-04-26T00:41:35.176 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.173+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.176 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.173+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2318333205}) 2024-04-26T00:41:35.176 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.173+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3191184599}) 2024-04-26T00:41:35.178 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.177+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.178 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.177+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=98138937}) 2024-04-26T00:41:35.178 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.177+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3052644742}) 2024-04-26T00:41:35.178 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.177+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.178 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.177+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=98138937}) 2024-04-26T00:41:35.179 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.177+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3855754402}) 2024-04-26T00:41:35.180 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.177+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.180 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.177+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1171685830}) 2024-04-26T00:41:35.180 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.177+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2912666154}) 2024-04-26T00:41:35.180 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.177+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.181 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.177+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1171685830}) 2024-04-26T00:41:35.181 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.177+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=476219897}) 2024-04-26T00:41:35.183 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.181+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.183 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.181+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=192961594}) 2024-04-26T00:41:35.183 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.181+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=955960567}) 2024-04-26T00:41:35.183 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.181+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.183 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.181+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=192961594}) 2024-04-26T00:41:35.183 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.181+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3918127699}) 2024-04-26T00:41:35.185 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.181+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.185 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.181+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3349346227}) 2024-04-26T00:41:35.185 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.181+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=482975713}) 2024-04-26T00:41:35.185 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.181+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.186 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.185+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3349346227}) 2024-04-26T00:41:35.186 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.185+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=603350860}) 2024-04-26T00:41:35.188 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.185+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.188 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.185+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4186664425}) 2024-04-26T00:41:35.188 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.185+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4037570740}) 2024-04-26T00:41:35.188 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.185+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.188 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.185+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4186664425}) 2024-04-26T00:41:35.189 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.185+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2057035747}) 2024-04-26T00:41:35.190 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.189+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.190 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.189+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=333904350}) 2024-04-26T00:41:35.190 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.189+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2260679721}) 2024-04-26T00:41:35.190 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.189+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.191 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.189+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=333904350}) 2024-04-26T00:41:35.191 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.189+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1271079657}) 2024-04-26T00:41:35.192 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.189+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.192 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.189+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2703735261}) 2024-04-26T00:41:35.192 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.189+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4200813724}) 2024-04-26T00:41:35.192 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.189+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.193 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.189+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2703735261}) 2024-04-26T00:41:35.193 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.189+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2968987775}) 2024-04-26T00:41:35.194 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.193+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.194 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.193+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3794529679}) 2024-04-26T00:41:35.195 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.193+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3806845666}) 2024-04-26T00:41:35.195 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.193+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.195 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.193+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3794529679}) 2024-04-26T00:41:35.195 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.193+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3710108544}) 2024-04-26T00:41:35.197 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.193+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.197 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.193+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=360392571}) 2024-04-26T00:41:35.197 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.193+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1652653287}) 2024-04-26T00:41:35.197 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.193+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.197 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.193+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=360392571}) 2024-04-26T00:41:35.197 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.193+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=181838409}) 2024-04-26T00:41:35.199 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.197+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.199 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.197+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3584082926}) 2024-04-26T00:41:35.199 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.197+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2622280738}) 2024-04-26T00:41:35.199 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.197+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.199 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.197+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3584082926}) 2024-04-26T00:41:35.199 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.197+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3296248740}) 2024-04-26T00:41:35.201 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.197+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.201 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.197+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2898296974}) 2024-04-26T00:41:35.201 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.197+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2760075980}) 2024-04-26T00:41:35.201 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.197+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.201 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.197+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2898296974}) 2024-04-26T00:41:35.201 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.197+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3634468420}) 2024-04-26T00:41:35.203 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.201+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.203 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.201+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=179395344}) 2024-04-26T00:41:35.203 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.201+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3557070373}) 2024-04-26T00:41:35.203 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.201+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.203 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.201+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=179395344}) 2024-04-26T00:41:35.203 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.201+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1181987759}) 2024-04-26T00:41:35.205 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.201+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.205 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.201+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=449566320}) 2024-04-26T00:41:35.205 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.201+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4001085198}) 2024-04-26T00:41:35.205 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.201+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.206 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.201+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=449566320}) 2024-04-26T00:41:35.206 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.205+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1644629654}) 2024-04-26T00:41:35.207 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.205+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.207 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.205+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4183449527}) 2024-04-26T00:41:35.207 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.205+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2237318150}) 2024-04-26T00:41:35.208 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.205+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.208 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.205+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4183449527}) 2024-04-26T00:41:35.208 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.205+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3668434611}) 2024-04-26T00:41:35.209 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.209+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.210 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.209+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1134357844}) 2024-04-26T00:41:35.210 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.209+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1906317460}) 2024-04-26T00:41:35.210 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.209+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.210 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.209+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1134357844}) 2024-04-26T00:41:35.210 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.209+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=649497885}) 2024-04-26T00:41:35.212 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.209+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.212 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.209+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2695592353}) 2024-04-26T00:41:35.212 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.209+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1648440552}) 2024-04-26T00:41:35.212 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.209+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.212 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.209+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2695592353}) 2024-04-26T00:41:35.212 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.209+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2647630013}) 2024-04-26T00:41:35.214 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.213+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.214 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.213+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2222865038}) 2024-04-26T00:41:35.214 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.213+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3008878687}) 2024-04-26T00:41:35.214 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.213+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.214 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.213+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2222865038}) 2024-04-26T00:41:35.214 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.213+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3335457982}) 2024-04-26T00:41:35.216 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.213+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.216 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.213+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1288712274}) 2024-04-26T00:41:35.216 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.213+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3010558037}) 2024-04-26T00:41:35.216 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.213+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.216 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.213+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1288712274}) 2024-04-26T00:41:35.216 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.213+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2890362962}) 2024-04-26T00:41:35.218 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.217+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.218 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.217+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2281718265}) 2024-04-26T00:41:35.218 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.217+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4049414477}) 2024-04-26T00:41:35.218 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.217+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.218 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.217+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2281718265}) 2024-04-26T00:41:35.219 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.217+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3391985976}) 2024-04-26T00:41:35.220 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.217+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.220 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.217+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1652691494}) 2024-04-26T00:41:35.220 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.217+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2946562026}) 2024-04-26T00:41:35.220 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.217+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.220 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.217+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1652691494}) 2024-04-26T00:41:35.220 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.217+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2996435667}) 2024-04-26T00:41:35.222 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.221+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.222 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.221+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2725328995}) 2024-04-26T00:41:35.222 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.221+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3617593959}) 2024-04-26T00:41:35.222 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.221+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.222 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.221+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2725328995}) 2024-04-26T00:41:35.222 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.221+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1753165268}) 2024-04-26T00:41:35.224 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.221+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.224 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.221+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2191908304}) 2024-04-26T00:41:35.224 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.221+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1977202133}) 2024-04-26T00:41:35.224 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.221+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.225 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.221+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2191908304}) 2024-04-26T00:41:35.225 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.221+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3469976849}) 2024-04-26T00:41:35.226 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.225+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.226 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.225+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2909260170}) 2024-04-26T00:41:35.226 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.225+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=151508939}) 2024-04-26T00:41:35.227 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.225+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.227 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.225+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2909260170}) 2024-04-26T00:41:35.227 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.225+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2690036159}) 2024-04-26T00:41:35.228 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.225+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.229 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.225+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3195369812}) 2024-04-26T00:41:35.229 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.225+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1667590485}) 2024-04-26T00:41:35.229 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.225+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.229 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.225+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3195369812}) 2024-04-26T00:41:35.229 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.225+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=123637680}) 2024-04-26T00:41:35.230 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.229+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.231 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.229+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2141386380}) 2024-04-26T00:41:35.231 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.229+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=814083509}) 2024-04-26T00:41:35.231 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.229+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.231 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.229+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2141386380}) 2024-04-26T00:41:35.231 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.229+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1305618090}) 2024-04-26T00:41:35.233 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.229+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.233 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.229+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2545731586}) 2024-04-26T00:41:35.233 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.229+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4256143897}) 2024-04-26T00:41:35.233 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.229+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.233 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.229+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2545731586}) 2024-04-26T00:41:35.233 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.229+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=80317664}) 2024-04-26T00:41:35.235 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.233+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.235 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.233+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=383098643}) 2024-04-26T00:41:35.235 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.233+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4147639116}) 2024-04-26T00:41:35.235 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.233+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.235 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.233+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=383098643}) 2024-04-26T00:41:35.235 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.233+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1291501005}) 2024-04-26T00:41:35.237 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.233+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.237 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.233+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1587417189}) 2024-04-26T00:41:35.237 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.233+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=492964835}) 2024-04-26T00:41:35.237 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.233+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.237 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.233+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1587417189}) 2024-04-26T00:41:35.237 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.233+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2433260052}) 2024-04-26T00:41:35.239 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.237+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.239 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.237+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=575957800}) 2024-04-26T00:41:35.239 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.237+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2674794555}) 2024-04-26T00:41:35.239 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.237+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.239 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.237+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=575957800}) 2024-04-26T00:41:35.239 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.237+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3885805282}) 2024-04-26T00:41:35.251 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.249+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.252 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.249+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=497614214}) 2024-04-26T00:41:35.252 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.249+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1215453837}) 2024-04-26T00:41:35.252 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.249+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.252 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.249+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=497614214}) 2024-04-26T00:41:35.252 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.249+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1482477139}) 2024-04-26T00:41:35.254 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.253+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.254 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.253+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1200361063}) 2024-04-26T00:41:35.254 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.253+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1499952731}) 2024-04-26T00:41:35.254 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.253+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.254 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.253+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1200361063}) 2024-04-26T00:41:35.254 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.253+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3737977673}) 2024-04-26T00:41:35.256 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.253+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.256 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.253+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3491362223}) 2024-04-26T00:41:35.256 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.253+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=841607512}) 2024-04-26T00:41:35.256 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.253+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.256 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.253+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3491362223}) 2024-04-26T00:41:35.256 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.253+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2668359470}) 2024-04-26T00:41:35.258 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.257+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.258 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.257+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1599199170}) 2024-04-26T00:41:35.258 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.257+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=862942038}) 2024-04-26T00:41:35.258 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.257+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.258 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.257+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1599199170}) 2024-04-26T00:41:35.258 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.257+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=394110459}) 2024-04-26T00:41:35.365 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.361+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.365 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.361+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3594457418}) 2024-04-26T00:41:35.365 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.361+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3044587509}) 2024-04-26T00:41:35.365 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.361+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.365 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.361+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3594457418}) 2024-04-26T00:41:35.365 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.361+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3258307955}) 2024-04-26T00:41:35.368 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.365+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.369 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.365+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1909532475}) 2024-04-26T00:41:35.369 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.365+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2019821744}) 2024-04-26T00:41:35.369 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.365+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.369 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.365+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1909532475}) 2024-04-26T00:41:35.369 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.365+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=446946443}) 2024-04-26T00:41:35.452 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.449+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.453 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.449+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2887957314}) 2024-04-26T00:41:35.453 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.449+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=22068405}) 2024-04-26T00:41:35.453 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.449+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.453 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.449+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2887957314}) 2024-04-26T00:41:35.453 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.449+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1880840391}) 2024-04-26T00:41:35.483 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.481+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.483 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.481+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3821060488}) 2024-04-26T00:41:35.483 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.481+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2734531317}) 2024-04-26T00:41:35.483 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.481+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.483 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.481+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3821060488}) 2024-04-26T00:41:35.483 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.481+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1223762354}) 2024-04-26T00:41:35.561 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:41:35.561 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:41:35.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.689+0000 7f5b9a8f5640 10 monclient: get_monmap_and_config 2024-04-26T00:41:35.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.689+0000 7f5b9a8f5640 10 monclient: build_initial_monmap 2024-04-26T00:41:35.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.689+0000 7f5b9a8f5640 10 monclient: monmap: 2024-04-26T00:41:35.692 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:41:35.692 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:35.692 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:41:35.693675+0000 2024-04-26T00:41:35.692 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:41:35.693675+0000 2024-04-26T00:41:35.692 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:41:35.692 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:35.692 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:41:35.692 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:41:35.692 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:41:35.692 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:41:35.692 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:41:35.692 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:35.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.689+0000 7f5b9a8f5640 1 Processor -- start 2024-04-26T00:41:35.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.689+0000 7f5b9a8f5640 1 -- start start 2024-04-26T00:41:35.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.689+0000 7f5b9a8f5640 10 monclient: init 2024-04-26T00:41:35.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.689+0000 7f5b9a8f5640 10 monclient: _reopen_session rank -1 2024-04-26T00:41:35.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.689+0000 7f5b9a8f5640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-04-26T00:41:35.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.689+0000 7f5b9a8f5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5b94136da0 addr v1:172.21.15.45:6789/0 2024-04-26T00:41:35.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.689+0000 7f5b9a8f5640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5b94137180 0x7f5b941375c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:35.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.689+0000 7f5b9a8f5640 10 monclient(hunting): picked mon.noname-c con 0x7f5b94137180 addr v2:172.21.15.45:3300/0 2024-04-26T00:41:35.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.689+0000 7f5b9a8f5640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b9413c2d0 0x7f5b9413e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:35.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.689+0000 7f5b9a8f5640 10 monclient(hunting): picked mon.noname-b con 0x7f5b9413c2d0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:41:35.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.689+0000 7f5b9a8f5640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b9409b4f0 con 0x7f5b94136da0 2024-04-26T00:41:35.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.689+0000 7f5b9a8f5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:35.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.689+0000 7f5b9a8f5640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f5b9409b210 con 0x7f5b94137180 2024-04-26T00:41:35.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.689+0000 7f5b9a8f5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:35.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.689+0000 7f5b9a8f5640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5b9409b7d0 con 0x7f5b9413c2d0 2024-04-26T00:41:35.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.689+0000 7f5b9a8f5640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:35.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.689+0000 7f5b9a8f5640 10 monclient(hunting): authenticate will time out at 2276.999414s 2024-04-26T00:41:35.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.689+0000 7f5b98e92640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b9413c2d0 0x7f5b9413e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:35.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.689+0000 7f5b98e92640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b9413c2d0 0x7f5b9413e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:45282/0 (socket says 172.21.15.37:45282) 2024-04-26T00:41:35.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.689+0000 7f5b98e92640 1 -- 172.21.15.37:0/2008321162 learned_addr learned my addr 172.21.15.37:0/2008321162 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:41:35.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.689+0000 7f5b98e92640 10 monclient(hunting): get_auth_request con 0x7f5b9413c2d0 auth_method 0 2024-04-26T00:41:35.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.689+0000 7f5b98e92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:35.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.689+0000 7f5b98e92640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:35.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.689+0000 7f5b98e92640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:35.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.689+0000 7f5b937fe640 1 --2- 172.21.15.37:0/2008321162 >> v2:172.21.15.45:3300/0 conn(0x7f5b94137180 0x7f5b941375c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:35.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.689+0000 7f5b98e92640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:35.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.689+0000 7f5b98e92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:35.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.689+0000 7f5b98e92640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:35.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.689+0000 7f5b937fe640 10 monclient(hunting): get_auth_request con 0x7f5b94137180 auth_method 0 2024-04-26T00:41:35.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.689+0000 7f5b937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:35.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.689+0000 7f5b937fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:35.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.689+0000 7f5b937fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:35.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.693+0000 7f5b98e92640 10 monclient(hunting): handle_auth_done global_id 254234 payload 274 2024-04-26T00:41:35.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.693+0000 7f5b98e92640 1 -- 172.21.15.37:0/2008321162 >> v2:172.21.15.45:3300/0 conn(0x7f5b94137180 msgr2=0x7f5b941375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:41:35.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.693+0000 7f5b98e92640 1 --2- 172.21.15.37:0/2008321162 >> v2:172.21.15.45:3300/0 conn(0x7f5b94137180 0x7f5b941375c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:35.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.693+0000 7f5b98e92640 1 -- 172.21.15.37:0/2008321162 >> v1:172.21.15.45:6789/0 conn(0x7f5b94136da0 legacy=0x7f5b9409bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:41:35.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.693+0000 7f5b98e92640 10 monclient: _finish_hunting 0 2024-04-26T00:41:35.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.693+0000 7f5b98e92640 1 monclient: found mon.noname-b 2024-04-26T00:41:35.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.693+0000 7f5b98e92640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:35.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.693+0000 7f5b98e92640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:41:35.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.693+0000 7f5b98e92640 1 -- 172.21.15.37:0/2008321162 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b94137b00 con 0x7f5b9413c2d0 2024-04-26T00:41:35.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b98e92640 1 -- 172.21.15.37:0/2008321162 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b9413c2d0 msgr2=0x7f5b9413e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:35.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b98e92640 1 --2- 172.21.15.37:0/2008321162 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b9413c2d0 0x7f5b9413e6c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c009e50 tx=0x7f5b8c01c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=588e3c4ae1f9301d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:35.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b92ffd640 1 -- 172.21.15.37:0/2008321162 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c0a5750 con 0x7f5b9413c2d0 2024-04-26T00:41:35.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b92ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:35.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b92ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:41:35.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b92ffd640 10 monclient: dump: 2024-04-26T00:41:35.700 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:35.700 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:35.700 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:35.700 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:35.700 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:35.700 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:35.700 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:35.700 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:35.700 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:35.700 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:35.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b92ffd640 10 monclient: _finish_auth 0 2024-04-26T00:41:35.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b92ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:41:35.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:41:35.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b9a8f5640 5 monclient: authenticate success, global_id 254234 2024-04-26T00:41:35.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b9a8f5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:41:35.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b92ffd640 1 -- 172.21.15.37:0/2008321162 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b8c0a58f0 con 0x7f5b9413c2d0 2024-04-26T00:41:35.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b92ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:41:35.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b9a8f5640 10 monclient: get_monmap_and_config success 2024-04-26T00:41:35.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b9a8f5640 4 set_mon_vals no callback set 2024-04-26T00:41:35.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b92ffd640 1 -- 172.21.15.37:0/2008321162 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c0a5cc0 con 0x7f5b9413c2d0 2024-04-26T00:41:35.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b92ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:35.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b92ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:41:35.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b92ffd640 10 monclient: dump: 2024-04-26T00:41:35.701 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:35.701 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:35.701 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:35.702 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:35.702 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:35.702 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:35.702 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:35.702 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:35.702 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:35.702 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:35.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b9a8f5640 10 monclient: shutdown 2024-04-26T00:41:35.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b9a8f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:41:35.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b9a8f5640 1 -- 172.21.15.37:0/2008321162 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b9413c2d0 msgr2=0x7f5b9413e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:35.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b9a8f5640 1 --2- 172.21.15.37:0/2008321162 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b9413c2d0 0x7f5b9413e6c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c009e50 tx=0x7f5b8c01c8b0 comp rx=0 tx=0).stop 2024-04-26T00:41:35.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b9a8f5640 1 -- 172.21.15.37:0/2008321162 shutdown_connections 2024-04-26T00:41:35.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b9a8f5640 1 --2- 172.21.15.37:0/2008321162 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b9413c2d0 0x7f5b9413e6c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:35.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b9a8f5640 1 --2- 172.21.15.37:0/2008321162 >> v2:172.21.15.45:3300/0 conn(0x7f5b94137180 0x7f5b941375c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:35.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b9a8f5640 1 -- 172.21.15.37:0/2008321162 >> 172.21.15.37:0/2008321162 conn(0x7f5b940a5f00 msgr2=0x7f5b940a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:41:35.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b9a8f5640 1 -- 172.21.15.37:0/2008321162 shutdown_connections 2024-04-26T00:41:35.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b9a8f5640 1 -- 172.21.15.37:0/2008321162 wait complete. 2024-04-26T00:41:35.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b9a8f5640 10 monclient: build_initial_monmap 2024-04-26T00:41:35.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b9a8f5640 10 monclient: monmap: 2024-04-26T00:41:35.703 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:41:35.703 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:41:35.703 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:41:35.703 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:41:35.703 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:41:35.703 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:41:35.703 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:41:35.703 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:41:35.703 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:41:35.704 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:35.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b9a8f5640 1 Processor -- start 2024-04-26T00:41:35.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b9a8f5640 1 -- start start 2024-04-26T00:41:35.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b9a8f5640 10 monclient: init 2024-04-26T00:41:35.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b9a8f5640 10 monclient: _reopen_session rank -1 2024-04-26T00:41:35.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b9a8f5640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:41:35.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b9a8f5640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b94136da0 0x7f5b9413b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:35.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b9a8f5640 10 monclient(hunting): picked mon.noname-c con 0x7f5b94136da0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:41:35.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b9a8f5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b94137180 0x7f5b9413ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:35.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b9a8f5640 10 monclient(hunting): picked mon.noname-b con 0x7f5b94137180 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:41:35.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b9a8f5640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b9413c2d0 0x7f5b941cbb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:35.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b9a8f5640 10 monclient(hunting): picked mon.noname-a con 0x7f5b9413c2d0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:41:35.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b9a8f5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:35.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b9a8f5640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f5b941417b0 con 0x7f5b9413c2d0 2024-04-26T00:41:35.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b9a8f5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:35.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b9a8f5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5b94141630 con 0x7f5b94137180 2024-04-26T00:41:35.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b9a8f5640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:35.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b9a8f5640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5b94141930 con 0x7f5b94136da0 2024-04-26T00:41:35.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b9a8f5640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:35.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b9a8f5640 10 monclient(hunting): authenticate will time out at 2277.006962s 2024-04-26T00:41:35.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b93fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b94136da0 0x7f5b9413b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:35.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b98e92640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b9413c2d0 0x7f5b941cbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:35.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b93fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b94136da0 0x7f5b9413b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:45296/0 (socket says 172.21.15.37:45296) 2024-04-26T00:41:35.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b93fff640 1 -- 172.21.15.37:0/2422491048 learned_addr learned my addr 172.21.15.37:0/2422491048 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:41:35.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b98e92640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b9413c2d0 0x7f5b941cbb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:45964/0 (socket says 172.21.15.37:45964) 2024-04-26T00:41:35.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b93fff640 10 monclient(hunting): get_auth_request con 0x7f5b94136da0 auth_method 0 2024-04-26T00:41:35.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:35.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b93fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:35.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b93fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:35.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b98e92640 10 monclient(hunting): get_auth_request con 0x7f5b9413c2d0 auth_method 0 2024-04-26T00:41:35.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b98e92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:35.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b98e92640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:35.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b98e92640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:35.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b93fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:35.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b93fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:35.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b93fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:35.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b98e92640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:35.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b98e92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:35.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b98e92640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:35.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b937fe640 1 --2- 172.21.15.37:0/2422491048 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b94137180 0x7f5b9413ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:35.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b937fe640 10 monclient(hunting): get_auth_request con 0x7f5b94137180 auth_method 0 2024-04-26T00:41:35.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:35.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b937fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:35.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b937fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:35.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b98e92640 10 monclient(hunting): handle_auth_done global_id 264117 payload 824 2024-04-26T00:41:35.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b98e92640 1 -- 172.21.15.37:0/2422491048 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b94136da0 msgr2=0x7f5b9413b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:35.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b98e92640 1 --2- 172.21.15.37:0/2422491048 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b94136da0 0x7f5b9413b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:35.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b98e92640 1 -- 172.21.15.37:0/2422491048 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b94137180 msgr2=0x7f5b9413ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:35.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b98e92640 1 --2- 172.21.15.37:0/2422491048 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b94137180 0x7f5b9413ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:35.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b98e92640 10 monclient: _finish_hunting 0 2024-04-26T00:41:35.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b98e92640 1 monclient: found mon.noname-a 2024-04-26T00:41:35.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b98e92640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:35.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b98e92640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:41:35.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b98e92640 1 -- 172.21.15.37:0/2422491048 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b9409aca0 con 0x7f5b9413c2d0 2024-04-26T00:41:35.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.697+0000 7f5b93fff640 1 --2- 172.21.15.37:0/2422491048 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b94136da0 0x7f5b9413b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:41:35.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.701+0000 7f5b98e92640 1 -- 172.21.15.37:0/2422491048 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b9413c2d0 msgr2=0x7f5b941cbb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:35.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.701+0000 7f5b98e92640 1 --2- 172.21.15.37:0/2422491048 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b9413c2d0 0x7f5b941cbb30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c0ab000 tx=0x7f5b8c019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:35.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.701+0000 7f5b90ff9640 1 -- 172.21.15.37:0/2422491048 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c0bc070 con 0x7f5b9413c2d0 2024-04-26T00:41:35.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.701+0000 7f5b90ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:35.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.701+0000 7f5b90ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:41:35.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.701+0000 7f5b90ff9640 10 monclient: dump: 2024-04-26T00:41:35.712 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:35.712 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:35.712 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:35.712 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:35.712 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:35.712 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:35.712 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:35.713 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:35.713 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:35.713 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:35.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.701+0000 7f5b90ff9640 10 monclient: _finish_auth 0 2024-04-26T00:41:35.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.701+0000 7f5b90ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:41:35.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.701+0000 7f5b90ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:41:35.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.701+0000 7f5b9a8f5640 5 monclient: authenticate success, global_id 264117 2024-04-26T00:41:35.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.701+0000 7f5b90ff9640 1 -- 172.21.15.37:0/2422491048 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b8c0b8850 con 0x7f5b9413c2d0 2024-04-26T00:41:35.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.701+0000 7f5b90ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:41:35.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.701+0000 7f5b90ff9640 1 -- 172.21.15.37:0/2422491048 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c0b8c00 con 0x7f5b9413c2d0 2024-04-26T00:41:35.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.701+0000 7f5b90ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:35.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.701+0000 7f5b90ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:41:35.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.701+0000 7f5b90ff9640 10 monclient: dump: 2024-04-26T00:41:35.713 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:35.713 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:35.713 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:35.714 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:35.714 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:35.714 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:35.714 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:35.714 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:35.714 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:35.714 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:35.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.701+0000 7f5b927fc640 4 set_mon_vals no callback set 2024-04-26T00:41:35.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.701+0000 7f5b9a8f5640 10 monclient: _renew_subs 2024-04-26T00:41:35.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.701+0000 7f5b9a8f5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:41:35.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.701+0000 7f5b9a8f5640 1 -- 172.21.15.37:0/2422491048 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b940a8510 con 0x7f5b9413c2d0 2024-04-26T00:41:35.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.701+0000 7f5b9a8f5640 10 monclient: _renew_subs 2024-04-26T00:41:35.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.701+0000 7f5b9a8f5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:41:35.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.701+0000 7f5b9a8f5640 1 -- 172.21.15.37:0/2422491048 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b941385c0 con 0x7f5b9413c2d0 2024-04-26T00:41:35.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.701+0000 7f5b9a8f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:35.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.701+0000 7f5b9a8f5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:35.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.701+0000 7f5b9a8f5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:41:35.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.701+0000 7f5b9a8f5640 1 -- 172.21.15.37:0/2422491048 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b94067ad0 con 0x7f5b9413c2d0 2024-04-26T00:41:35.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.701+0000 7f5b90ff9640 1 -- 172.21.15.37:0/2422491048 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f5b8c0b6410 con 0x7f5b9413c2d0 2024-04-26T00:41:35.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.701+0000 7f5b90ff9640 1 --2- 172.21.15.37:0/2422491048 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5b80035e80 0x7f5b80038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:35.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.705+0000 7f5b90ff9640 1 -- 172.21.15.37:0/2422491048 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(861..861 src has 705..861) ==== 3975+0+0 (secure 0 0 0) 0x7f5b8c0e3ab0 con 0x7f5b9413c2d0 2024-04-26T00:41:35.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.705+0000 7f5b93fff640 1 --2- 172.21.15.37:0/2422491048 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5b80035e80 0x7f5b80038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:35.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.705+0000 7f5b93fff640 10 monclient: get_auth_request con 0x7f5b80035e80 auth_method 0 2024-04-26T00:41:35.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.705+0000 7f5b93fff640 1 --2- 172.21.15.37:0/2422491048 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5b80035e80 0x7f5b80038340 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f5b84004470 tx=0x7f5b84003700 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:35.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.709+0000 7f5b90ff9640 1 -- 172.21.15.37:0/2422491048 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5b8c0165c0 con 0x7f5b9413c2d0 2024-04-26T00:41:35.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.709+0000 7f5b90ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:35.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.709+0000 7f5b90ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:41:35.733 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.729+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.733 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.729+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2601585406}) 2024-04-26T00:41:35.733 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.729+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3028657236}) 2024-04-26T00:41:35.733 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.729+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.733 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.729+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2601585406}) 2024-04-26T00:41:35.733 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.729+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1433368679}) 2024-04-26T00:41:35.739 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.737+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.739 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.737+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1653263215}) 2024-04-26T00:41:35.739 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.737+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=867109859}) 2024-04-26T00:41:35.739 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.737+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.740 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.737+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1653263215}) 2024-04-26T00:41:35.740 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.737+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2854464991}) 2024-04-26T00:41:35.742 INFO:tasks.rados.rados.0.smithi045.stdout:3833: done (2 left) 2024-04-26T00:41:35.742 INFO:tasks.rados.rados.0.smithi045.stdout:3834: delete oid 43 current snap is 423 2024-04-26T00:41:35.743 INFO:tasks.rados.rados.0.smithi045.stdout:3829: expect (ObjNum 1333 snap 411 seq_num 1333) 2024-04-26T00:41:35.902 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.901+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.902 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.901+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3934166735}) 2024-04-26T00:41:35.902 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.901+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=828076150}) 2024-04-26T00:41:35.902 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.901+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.902 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.901+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3934166735}) 2024-04-26T00:41:35.902 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.901+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4026967499}) 2024-04-26T00:41:35.903 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.901+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.903 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.901+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1127528356}) 2024-04-26T00:41:35.903 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.901+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2864858217}) 2024-04-26T00:41:35.903 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.901+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.904 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.901+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1127528356}) 2024-04-26T00:41:35.904 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.901+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3695752402}) 2024-04-26T00:41:35.905 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.901+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.905 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.901+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=595213156}) 2024-04-26T00:41:35.905 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.901+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1825472160}) 2024-04-26T00:41:35.905 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.901+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.905 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.901+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=595213156}) 2024-04-26T00:41:35.906 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.901+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2241450447}) 2024-04-26T00:41:35.906 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.905+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.906 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.905+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=790714608}) 2024-04-26T00:41:35.907 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.905+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1008801696}) 2024-04-26T00:41:35.907 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.905+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.907 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.905+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=790714608}) 2024-04-26T00:41:35.907 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.905+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1679198245}) 2024-04-26T00:41:35.909 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.905+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.909 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.905+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=28,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29} crc {logm=1221632333,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=3312321056,mon_config_key=1720404874,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=359205829,osd_snap=1327299007}) 2024-04-26T00:41:35.909 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.905+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=27,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29} crc {logm=2468657732,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=3312321056,mon_config_key=216630890,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=359205829,osd_snap=4053278068}) 2024-04-26T00:41:35.909 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.905+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.909 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.905+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=28,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29} crc {logm=1221632333,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=3312321056,mon_config_key=1720404874,monmap=3876866194,osd_metadata=3306121363,osd_pg_creating=359205829,osd_snap=1327299007}) 2024-04-26T00:41:35.909 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.905+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=26,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=31} crc {logm=3416647459,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=3312321056,mon_config_key=4061063117,monmap=3876866194,osd_metadata=124267790,osd_pg_creating=359205829,osd_snap=457368586}) 2024-04-26T00:41:35.910 INFO:tasks.rados.rados.0.smithi045.stdout:3831: finishing write tid 1 to smithi04526579-50 2024-04-26T00:41:35.910 INFO:tasks.rados.rados.0.smithi045.stdout:3831: finishing write tid 2 to smithi04526579-50 2024-04-26T00:41:35.910 INFO:tasks.rados.rados.0.smithi045.stdout:3831: finishing write tid 3 to smithi04526579-50 2024-04-26T00:41:35.910 INFO:tasks.rados.rados.0.smithi045.stdout:3831: finishing write tid 4 to smithi04526579-50 2024-04-26T00:41:35.910 INFO:tasks.rados.rados.0.smithi045.stdout:3831: finishing write tid 5 to smithi04526579-50 2024-04-26T00:41:35.910 INFO:tasks.rados.rados.0.smithi045.stdout:3831: oid 50 updating version 0 to 2640 2024-04-26T00:41:35.911 INFO:tasks.rados.rados.0.smithi045.stdout:3831: oid 50 updating version 2640 to 2642 2024-04-26T00:41:35.911 INFO:tasks.rados.rados.0.smithi045.stdout:3831: oid 50 version 2642 is already newer than 2639 2024-04-26T00:41:35.911 INFO:tasks.rados.rados.0.smithi045.stdout:3831: oid 50 version 2642 is already newer than 2641 2024-04-26T00:41:35.911 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 50 v 2642 (ObjNum 1371 snap 423 seq_num 1371) dirty exists 2024-04-26T00:41:35.911 INFO:tasks.rados.rados.0.smithi045.stdout:3831: left oid 50 (ObjNum 1371 snap 423 seq_num 1371) 2024-04-26T00:41:35.911 INFO:tasks.rados.rados.0.smithi045.stdout:3829: done (2 left) 2024-04-26T00:41:35.911 INFO:tasks.rados.rados.0.smithi045.stdout:3831: done (1 left) 2024-04-26T00:41:35.911 INFO:tasks.rados.rados.0.smithi045.stdout:3834: done (0 left) 2024-04-26T00:41:35.911 INFO:tasks.rados.rados.0.smithi045.stdout:3835: read oid 3 snap -1 2024-04-26T00:41:35.911 INFO:tasks.rados.rados.0.smithi045.stdout:3835: expect (ObjNum 1355 snap 419 seq_num 1355) 2024-04-26T00:41:35.911 INFO:tasks.rados.rados.0.smithi045.stdout:3836: snap_create 2024-04-26T00:41:35.912 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.909+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.912 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.909+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=100} crc {osd_snap=4294465396}) 2024-04-26T00:41:35.912 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.909+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=100} crc {osd_snap=2668656203}) 2024-04-26T00:41:35.912 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.909+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:35.912 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.909+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=100} crc {osd_snap=4294465396}) 2024-04-26T00:41:35.912 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:35.909+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=100} crc {osd_snap=2021336642}) 2024-04-26T00:41:35.923 INFO:tasks.rados.rados.0.smithi045.stdout:3835: expect (ObjNum 1355 snap 419 seq_num 1355) 2024-04-26T00:41:35.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.949+0000 7f5b9a8f5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:41:35.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.949+0000 7f5b9a8f5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:41:35.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.949+0000 7f5b9a8f5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:41:35.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.949+0000 7f5b9a8f5640 1 -- 172.21.15.37:0/2422491048 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5b94066f40 con 0x7f5b9413c2d0 2024-04-26T00:41:35.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.949+0000 7f5b90ff9640 1 -- 172.21.15.37:0/2422491048 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f5b8c0c0dc0 con 0x7f5b9413c2d0 2024-04-26T00:41:35.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.949+0000 7f5b90ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:41:35.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.949+0000 7f5b90ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:41:35.951 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":156,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:41:35.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.949+0000 7f5b9a8f5640 1 -- 172.21.15.37:0/2422491048 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5b80035e80 msgr2=0x7f5b80038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:35.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.949+0000 7f5b9a8f5640 1 --2- 172.21.15.37:0/2422491048 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5b80035e80 0x7f5b80038340 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f5b84004470 tx=0x7f5b84003700 comp rx=0 tx=0).stop 2024-04-26T00:41:35.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.949+0000 7f5b9a8f5640 10 monclient: shutdown 2024-04-26T00:41:35.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.949+0000 7f5b9a8f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:41:35.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.949+0000 7f5b9a8f5640 1 -- 172.21.15.37:0/2422491048 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b9413c2d0 msgr2=0x7f5b941cbb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:35.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.949+0000 7f5b9a8f5640 1 --2- 172.21.15.37:0/2422491048 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b9413c2d0 0x7f5b941cbb30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c0ab000 tx=0x7f5b8c019650 comp rx=0 tx=0).stop 2024-04-26T00:41:35.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.949+0000 7f5b9a8f5640 1 -- 172.21.15.37:0/2422491048 shutdown_connections 2024-04-26T00:41:35.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.949+0000 7f5b9a8f5640 1 --2- 172.21.15.37:0/2422491048 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5b80035e80 0x7f5b80038340 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:35.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.949+0000 7f5b9a8f5640 1 --2- 172.21.15.37:0/2422491048 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b9413c2d0 0x7f5b941cbb30 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:35.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.949+0000 7f5b9a8f5640 1 --2- 172.21.15.37:0/2422491048 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b94137180 0x7f5b9413ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:35.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.949+0000 7f5b9a8f5640 1 --2- 172.21.15.37:0/2422491048 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b94136da0 0x7f5b9413b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:35.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.949+0000 7f5b9a8f5640 1 -- 172.21.15.37:0/2422491048 >> 172.21.15.37:0/2422491048 conn(0x7f5b940a5f00 msgr2=0x7f5b9413d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:41:35.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.949+0000 7f5b9a8f5640 1 -- 172.21.15.37:0/2422491048 shutdown_connections 2024-04-26T00:41:35.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:35.949+0000 7f5b9a8f5640 1 -- 172.21.15.37:0/2422491048 wait complete. 2024-04-26T00:41:35.970 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:41:35.970 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-26T00:41:35.970 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:41:35.970 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:41:36.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.085+0000 7f6706c94640 10 monclient: get_monmap_and_config 2024-04-26T00:41:36.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.085+0000 7f6706c94640 10 monclient: build_initial_monmap 2024-04-26T00:41:36.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.085+0000 7f6706c94640 10 monclient: monmap: 2024-04-26T00:41:36.089 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:41:36.090 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:36.090 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:41:36.091550+0000 2024-04-26T00:41:36.090 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:41:36.091550+0000 2024-04-26T00:41:36.090 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:41:36.090 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:36.090 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:41:36.090 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:41:36.090 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:41:36.090 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:41:36.090 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:41:36.090 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:36.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6706c94640 1 Processor -- start 2024-04-26T00:41:36.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6706c94640 1 -- start start 2024-04-26T00:41:36.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6706c94640 10 monclient: init 2024-04-26T00:41:36.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6706c94640 10 monclient: _reopen_session rank -1 2024-04-26T00:41:36.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6706c94640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-04-26T00:41:36.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6706c94640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f67000a3d90 0x7f670009bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:36.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6706c94640 10 monclient(hunting): picked mon.noname-b con 0x7f67000a3d90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:41:36.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6706c94640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f67000a4170 addr v1:172.21.15.45:6789/0 2024-04-26T00:41:36.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6706c94640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f670013c3e0 0x7f670013e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:36.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6706c94640 10 monclient(hunting): picked mon.noname-c con 0x7f670013c3e0 addr v2:172.21.15.45:3300/0 2024-04-26T00:41:36.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6706c94640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67000a5150 con 0x7f67000a4170 2024-04-26T00:41:36.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6706c94640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:36.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6706c94640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f670009b280 con 0x7f670013c3e0 2024-04-26T00:41:36.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6706c94640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:36.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6706c94640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f670009b560 con 0x7f67000a3d90 2024-04-26T00:41:36.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6706c94640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:36.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6706c94640 10 monclient(hunting): authenticate will time out at 2277.397583s 2024-04-26T00:41:36.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6704a30640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f67000a3d90 0x7f670009bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:36.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6704a30640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f67000a3d90 0x7f670009bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:45310/0 (socket says 172.21.15.37:45310) 2024-04-26T00:41:36.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6704a30640 1 -- 172.21.15.37:0/2537331419 learned_addr learned my addr 172.21.15.37:0/2537331419 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:41:36.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6704a30640 10 monclient(hunting): get_auth_request con 0x7f67000a3d90 auth_method 0 2024-04-26T00:41:36.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6704a30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:36.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6704a30640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:36.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6704a30640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:36.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6705231640 1 --2- 172.21.15.37:0/2537331419 >> v2:172.21.15.45:3300/0 conn(0x7f670013c3e0 0x7f670013e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:36.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6705231640 10 monclient(hunting): get_auth_request con 0x7f670013c3e0 auth_method 0 2024-04-26T00:41:36.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6705231640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:36.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6705231640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:36.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6705231640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:36.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6704a30640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:36.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6704a30640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:36.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6704a30640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:36.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6705231640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:36.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6705231640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:36.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6705231640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:36.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6705231640 10 monclient(hunting): handle_auth_done global_id 224470 payload 274 2024-04-26T00:41:36.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6705231640 1 -- 172.21.15.37:0/2537331419 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f67000a3d90 msgr2=0x7f670009bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:41:36.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6705231640 1 --2- 172.21.15.37:0/2537331419 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f67000a3d90 0x7f670009bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:36.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6705231640 1 -- 172.21.15.37:0/2537331419 >> v1:172.21.15.45:6789/0 conn(0x7f67000a4170 legacy=0x7f67000a45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:41:36.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6705231640 10 monclient: _finish_hunting 0 2024-04-26T00:41:36.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6705231640 1 monclient: found mon.noname-c 2024-04-26T00:41:36.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6705231640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:36.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6705231640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:41:36.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6705231640 1 -- 172.21.15.37:0/2537331419 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f670006dc60 con 0x7f670013c3e0 2024-04-26T00:41:36.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.089+0000 7f6704a30640 1 --2- 172.21.15.37:0/2537331419 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f67000a3d90 0x7f670009bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:41:36.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.093+0000 7f6705231640 1 -- 172.21.15.37:0/2537331419 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f670013c3e0 msgr2=0x7f670013e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:36.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.093+0000 7f6705231640 1 --2- 172.21.15.37:0/2537331419 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f670013c3e0 0x7f670013e7d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f66f8009d90 tx=0x7f66f801c7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3d815ecefb217b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:36.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.093+0000 7f66ff7fe640 1 -- 172.21.15.37:0/2537331419 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f80ac070 con 0x7f670013c3e0 2024-04-26T00:41:36.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.093+0000 7f66ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:36.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.093+0000 7f66ff7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:41:36.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.093+0000 7f66ff7fe640 10 monclient: dump: 2024-04-26T00:41:36.101 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:36.101 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:36.101 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.101 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.102 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:36.102 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:36.102 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:36.102 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:36.102 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:36.102 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:36.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.093+0000 7f66ff7fe640 10 monclient: _finish_auth 0 2024-04-26T00:41:36.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.093+0000 7f66ff7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:41:36.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.093+0000 7f66ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:41:36.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.093+0000 7f6706c94640 5 monclient: authenticate success, global_id 224470 2024-04-26T00:41:36.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.093+0000 7f6706c94640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:41:36.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.093+0000 7f66ff7fe640 1 -- 172.21.15.37:0/2537331419 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66f80a6680 con 0x7f670013c3e0 2024-04-26T00:41:36.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.093+0000 7f66ff7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:41:36.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.093+0000 7f66ff7fe640 1 -- 172.21.15.37:0/2537331419 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f80a6a50 con 0x7f670013c3e0 2024-04-26T00:41:36.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.093+0000 7f66ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:36.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.093+0000 7f66ff7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:41:36.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.093+0000 7f66ff7fe640 10 monclient: dump: 2024-04-26T00:41:36.103 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:36.103 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:36.103 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.103 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.103 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:36.103 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:36.103 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:36.103 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:36.104 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:36.104 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:36.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.093+0000 7f6706c94640 10 monclient: get_monmap_and_config success 2024-04-26T00:41:36.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.093+0000 7f6706c94640 4 set_mon_vals no callback set 2024-04-26T00:41:36.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6706c94640 10 monclient: shutdown 2024-04-26T00:41:36.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6706c94640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:41:36.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6706c94640 1 -- 172.21.15.37:0/2537331419 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f670013c3e0 msgr2=0x7f670013e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:36.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6706c94640 1 --2- 172.21.15.37:0/2537331419 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f670013c3e0 0x7f670013e7d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f66f8009d90 tx=0x7f66f801c7f0 comp rx=0 tx=0).stop 2024-04-26T00:41:36.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6706c94640 1 -- 172.21.15.37:0/2537331419 shutdown_connections 2024-04-26T00:41:36.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6706c94640 1 --2- 172.21.15.37:0/2537331419 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f670013c3e0 0x7f670013e7d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:36.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6706c94640 1 --2- 172.21.15.37:0/2537331419 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f67000a3d90 0x7f670009bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:36.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6706c94640 1 -- 172.21.15.37:0/2537331419 >> 172.21.15.37:0/2537331419 conn(0x7f67000aa0a0 msgr2=0x7f67000a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:41:36.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6706c94640 1 -- 172.21.15.37:0/2537331419 shutdown_connections 2024-04-26T00:41:36.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6706c94640 1 -- 172.21.15.37:0/2537331419 wait complete. 2024-04-26T00:41:36.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6706c94640 10 monclient: build_initial_monmap 2024-04-26T00:41:36.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6706c94640 10 monclient: monmap: 2024-04-26T00:41:36.105 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:41:36.105 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:41:36.105 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:41:36.105 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:41:36.105 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:41:36.105 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:41:36.105 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:41:36.105 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:41:36.105 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:41:36.105 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:36.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6706c94640 1 Processor -- start 2024-04-26T00:41:36.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6706c94640 1 -- start start 2024-04-26T00:41:36.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6706c94640 10 monclient: init 2024-04-26T00:41:36.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6706c94640 10 monclient: _reopen_session rank -1 2024-04-26T00:41:36.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6706c94640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:41:36.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6706c94640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f67000a3d90 0x7f67001cee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:36.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6706c94640 10 monclient(hunting): picked mon.noname-a con 0x7f67000a3d90 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:41:36.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6706c94640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f67000a4170 0x7f6700137030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:36.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6706c94640 10 monclient(hunting): picked mon.noname-c con 0x7f67000a4170 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:41:36.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6706c94640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f670013c3e0 0x7f670013a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:36.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6706c94640 10 monclient(hunting): picked mon.noname-b con 0x7f670013c3e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:41:36.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6706c94640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:36.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6706c94640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f67001418c0 con 0x7f67000a3d90 2024-04-26T00:41:36.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6706c94640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:36.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6706c94640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6700141740 con 0x7f670013c3e0 2024-04-26T00:41:36.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6706c94640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:36.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6706c94640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f6700141a40 con 0x7f67000a4170 2024-04-26T00:41:36.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6706c94640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:36.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6706c94640 10 monclient(hunting): authenticate will time out at 2277.405717s 2024-04-26T00:41:36.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f66fffff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f67000a4170 0x7f6700137030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:36.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f66fffff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f67000a4170 0x7f6700137030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:45324/0 (socket says 172.21.15.37:45324) 2024-04-26T00:41:36.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f66fffff640 1 -- 172.21.15.37:0/219601436 learned_addr learned my addr 172.21.15.37:0/219601436 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:41:36.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f66fffff640 10 monclient(hunting): get_auth_request con 0x7f67000a4170 auth_method 0 2024-04-26T00:41:36.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f66fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:36.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f66fffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:36.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f66fffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:36.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f66fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:36.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f66fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:36.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6704a30640 1 --2- 172.21.15.37:0/219601436 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f67000a3d90 0x7f67001cee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:36.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f66fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:36.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6704a30640 10 monclient(hunting): get_auth_request con 0x7f67000a3d90 auth_method 0 2024-04-26T00:41:36.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6704a30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:36.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6704a30640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:36.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6704a30640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:36.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6705231640 1 --2- 172.21.15.37:0/219601436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f670013c3e0 0x7f670013a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:36.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6705231640 10 monclient(hunting): get_auth_request con 0x7f670013c3e0 auth_method 0 2024-04-26T00:41:36.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6705231640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:36.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6705231640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:36.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6705231640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:36.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f66fffff640 10 monclient(hunting): handle_auth_done global_id 254243 payload 824 2024-04-26T00:41:36.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f66fffff640 1 -- 172.21.15.37:0/219601436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f670013c3e0 msgr2=0x7f670013a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:36.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f66fffff640 1 --2- 172.21.15.37:0/219601436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f670013c3e0 0x7f670013a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:36.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f66fffff640 1 -- 172.21.15.37:0/219601436 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f67000a3d90 msgr2=0x7f67001cee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:36.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f66fffff640 1 --2- 172.21.15.37:0/219601436 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f67000a3d90 0x7f67001cee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:36.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f66fffff640 10 monclient: _finish_hunting 0 2024-04-26T00:41:36.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f66fffff640 1 monclient: found mon.noname-c 2024-04-26T00:41:36.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f66fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:36.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f66fffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:41:36.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f66fffff640 1 -- 172.21.15.37:0/219601436 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f670009ada0 con 0x7f67000a4170 2024-04-26T00:41:36.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.097+0000 7f6704a30640 1 --2- 172.21.15.37:0/219601436 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f67000a3d90 0x7f67001cee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:41:36.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f66fffff640 1 -- 172.21.15.37:0/219601436 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f67000a4170 msgr2=0x7f6700137030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:36.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f66fffff640 1 --2- 172.21.15.37:0/219601436 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f67000a4170 0x7f6700137030 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f66f000c480 tx=0x7f66f000c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:36.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f66fd7fa640 1 -- 172.21.15.37:0/219601436 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f0012a40 con 0x7f67000a4170 2024-04-26T00:41:36.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f66fd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:36.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f66fd7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:41:36.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f66fd7fa640 10 monclient: dump: 2024-04-26T00:41:36.112 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:36.112 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:36.112 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.113 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.113 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:36.113 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:36.113 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:36.113 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:36.113 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:36.113 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:36.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f66fd7fa640 10 monclient: _finish_auth 0 2024-04-26T00:41:36.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f66fd7fa640 10 monclient: _check_auth_tickets 2024-04-26T00:41:36.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f66fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:41:36.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f66fd7fa640 1 -- 172.21.15.37:0/219601436 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66f0017070 con 0x7f67000a4170 2024-04-26T00:41:36.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f6706c94640 5 monclient: authenticate success, global_id 254243 2024-04-26T00:41:36.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f6706c94640 10 monclient: _renew_subs 2024-04-26T00:41:36.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f6706c94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:41:36.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f6706c94640 1 -- 172.21.15.37:0/219601436 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67001216e0 con 0x7f67000a4170 2024-04-26T00:41:36.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f66fd7fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:41:36.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f66fd7fa640 1 -- 172.21.15.37:0/219601436 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f0012e40 con 0x7f67000a4170 2024-04-26T00:41:36.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f66fd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:36.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f66fd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:41:36.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f66fd7fa640 10 monclient: dump: 2024-04-26T00:41:36.114 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:36.114 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:36.114 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.114 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.114 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:36.114 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:36.114 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:36.114 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:36.115 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:36.115 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:36.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f6706c94640 10 monclient: _renew_subs 2024-04-26T00:41:36.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f6706c94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:41:36.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f6706c94640 1 -- 172.21.15.37:0/219601436 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f670013bc30 con 0x7f67000a4170 2024-04-26T00:41:36.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f66fe7fc640 4 set_mon_vals no callback set 2024-04-26T00:41:36.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f6706c94640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:36.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f6706c94640 1 --2- 172.21.15.37:0/219601436 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6700137570 0x7f670013a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:36.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f6706c94640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:36.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f6706c94640 1 -- 172.21.15.37:0/219601436 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f670004f510 con 0x7f6700137570 2024-04-26T00:41:36.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f6704a30640 1 --2- 172.21.15.37:0/219601436 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6700137570 0x7f670013a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:36.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f6704a30640 10 monclient: get_auth_request con 0x7f6700137570 auth_method 0 2024-04-26T00:41:36.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f6704a30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:36.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f6704a30640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:36.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f6704a30640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:36.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f66fd7fa640 1 -- 172.21.15.37:0/219601436 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f66f001c460 con 0x7f67000a4170 2024-04-26T00:41:36.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f6704a30640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:36.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f6704a30640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:36.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f6704a30640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:36.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f6704a30640 10 monclient(hunting): handle_auth_done global_id 264123 payload 824 2024-04-26T00:41:36.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f6704a30640 1 -- 172.21.15.37:0/219601436 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f67000008e0 con 0x7f6700137570 2024-04-26T00:41:36.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f66fd7fa640 1 --2- 172.21.15.37:0/219601436 >> v2:172.21.15.45:6812/2379918623 conn(0x7f66e0035e80 0x7f66e0038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:36.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f66fd7fa640 1 -- 172.21.15.37:0/219601436 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(862..862 src has 705..862) ==== 3991+0+0 (secure 0 0 0) 0x7f66f0048980 con 0x7f67000a4170 2024-04-26T00:41:36.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f6705231640 1 --2- 172.21.15.37:0/219601436 >> v2:172.21.15.45:6812/2379918623 conn(0x7f66e0035e80 0x7f66e0038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:36.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.101+0000 7f6705231640 10 monclient: get_auth_request con 0x7f66e0035e80 auth_method 0 2024-04-26T00:41:36.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.105+0000 7f6704a30640 1 -- 172.21.15.37:0/219601436 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6700137570 msgr2=0x7f670013a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:36.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.105+0000 7f6704a30640 1 --2- 172.21.15.37:0/219601436 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6700137570 0x7f670013a570 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f66e8011830 tx=0x7f66e8011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:36.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.105+0000 7f66fd7fa640 1 -- 172.21.15.37:0/219601436 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e800fe50 con 0x7f6700137570 2024-04-26T00:41:36.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.105+0000 7f66fd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:36.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.105+0000 7f66fd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:41:36.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.105+0000 7f66fd7fa640 10 monclient: dump: 2024-04-26T00:41:36.117 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:36.117 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:36.117 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.117 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.117 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:36.117 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:36.117 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:36.118 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:36.118 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:36.118 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:36.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.105+0000 7f66fd7fa640 1 -- 172.21.15.37:0/219601436 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f66e800fe50 con 0x7f6700137570 2024-04-26T00:41:36.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.105+0000 7f66fd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:36.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.105+0000 7f66fd7fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:41:36.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.105+0000 7f66fd7fa640 1 -- 172.21.15.37:0/219601436 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6700137570 msgr2=0x7f670013a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:36.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.105+0000 7f66fd7fa640 1 --2- 172.21.15.37:0/219601436 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6700137570 0x7f670013a570 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f66e8011830 tx=0x7f66e8011cf0 comp rx=0 tx=0).stop 2024-04-26T00:41:36.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.105+0000 7f66fd7fa640 1 -- 172.21.15.37:0/219601436 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6700137570 msgr2=0x7f670013a570 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:41:36.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.105+0000 7f66fd7fa640 1 --2- 172.21.15.37:0/219601436 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6700137570 0x7f670013a570 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f66e8011830 tx=0x7f66e8011cf0 comp rx=0 tx=0).stop 2024-04-26T00:41:36.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.105+0000 7f6705231640 1 --2- 172.21.15.37:0/219601436 >> v2:172.21.15.45:6812/2379918623 conn(0x7f66e0035e80 0x7f66e0038340 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f67000a6d60 tx=0x7f66f80a7000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:36.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.109+0000 7f6706c94640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:41:36.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.109+0000 7f6706c94640 1 --2- 172.21.15.37:0/219601436 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6700121120 0x7f670013aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:36.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.109+0000 7f6706c94640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:36.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.109+0000 7f6706c94640 1 -- 172.21.15.37:0/219601436 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f67001da1c0 con 0x7f6700121120 2024-04-26T00:41:36.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.109+0000 7f6704a30640 1 --2- 172.21.15.37:0/219601436 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6700121120 0x7f670013aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:36.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.109+0000 7f6704a30640 10 monclient: get_auth_request con 0x7f6700121120 auth_method 0 2024-04-26T00:41:36.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.109+0000 7f6704a30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:36.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.109+0000 7f6704a30640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:36.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.109+0000 7f6704a30640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:36.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.109+0000 7f6704a30640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:36.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.109+0000 7f6704a30640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:36.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.109+0000 7f6704a30640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:36.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.109+0000 7f6704a30640 10 monclient(hunting): handle_auth_done global_id 264126 payload 824 2024-04-26T00:41:36.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.109+0000 7f6704a30640 1 -- 172.21.15.37:0/219601436 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f67001da370 con 0x7f6700121120 2024-04-26T00:41:36.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.113+0000 7f6704a30640 1 -- 172.21.15.37:0/219601436 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6700121120 msgr2=0x7f670013aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:36.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.113+0000 7f6704a30640 1 --2- 172.21.15.37:0/219601436 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6700121120 0x7f670013aab0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f66e8008010 tx=0x7f66e80046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:36.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.113+0000 7f66fd7fa640 1 -- 172.21.15.37:0/219601436 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e8003d50 con 0x7f6700121120 2024-04-26T00:41:36.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.113+0000 7f66fd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:36.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.113+0000 7f66fd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:41:36.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.113+0000 7f66fd7fa640 10 monclient: dump: 2024-04-26T00:41:36.122 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:36.122 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:36.122 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.122 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.122 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:36.122 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:36.122 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:36.122 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:36.122 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:36.122 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:36.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.113+0000 7f66fd7fa640 1 -- 172.21.15.37:0/219601436 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4547 (secure 0 0 0) 0x7f66e8003d50 con 0x7f6700121120 2024-04-26T00:41:36.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.113+0000 7f66fd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:41:36.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.113+0000 7f66fd7fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:41:36.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.113+0000 7f66fd7fa640 1 -- 172.21.15.37:0/219601436 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6700121120 msgr2=0x7f670013aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:36.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.113+0000 7f66fd7fa640 1 --2- 172.21.15.37:0/219601436 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6700121120 0x7f670013aab0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f66e8008010 tx=0x7f66e80046d0 comp rx=0 tx=0).stop 2024-04-26T00:41:36.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.113+0000 7f66fd7fa640 1 -- 172.21.15.37:0/219601436 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6700121120 msgr2=0x7f670013aab0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:41:36.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.113+0000 7f66fd7fa640 1 --2- 172.21.15.37:0/219601436 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6700121120 0x7f670013aab0 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f66e8008010 tx=0x7f66e80046d0 comp rx=0 tx=0).stop 2024-04-26T00:41:36.123 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:41:36.123 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:41:36.123 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:41:36.123 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:41:36.123 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 156, 2024-04-26T00:41:36.123 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 18403, 2024-04-26T00:41:36.123 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:41:36.124 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:41:36.124 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:41:36.124 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:41:36.124 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:36.124 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 5, 2024-04-26T00:41:36.124 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:41:36.124 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:41:36.124 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:41:36.124 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:41:36.124 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:41:36.124 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:41:36.124 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:41:36.124 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:41:36.125 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:41:36.125 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:41:36.125 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:41:36.125 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:41:36.125 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:41:36.125 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:41:36.125 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:36.125 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:41:36.125 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:41:36.125 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:41:36.125 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:41:36.125 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:41:36.125 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:41:36.125 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:41:36.126 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:41:36.126 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:41:36.126 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:41:36.126 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:41:36.126 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:41:36.126 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:41:36.126 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:36.126 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:36.126 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:41:36.126 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:41:36.126 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:41:36.126 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:41:36.126 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:41:36.126 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:41:36.127 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:41:36.127 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:41:36.127 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:41:36.127 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:41:36.127 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:41:36.127 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:41:36.127 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:41:36.127 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:41:36.127 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:41:36.127 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:41:36.127 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:41:36.127 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:41:36.127 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:41:36.127 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:41:36.128 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:41:36.128 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:41:36.128 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:41:36.128 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:41:36.128 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:41:36.128 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:41:36.128 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:41:36.128 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:41:36.128 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:36.128 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:41:36.128 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:36.128 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:41:36.128 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:36.128 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:41:36.129 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:41:36.129 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:41:36.129 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:41:36.129 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:36.129 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:41:36.129 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:41:36.129 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:36.129 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:36.129 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:36.129 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:41:36.129 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:41:36.129 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:36.129 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:36.129 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:36.129 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:36.130 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:41:36.130 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:41:36.130 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:41:36.130 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:41:36.130 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:41:36.130 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:36.130 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:36.130 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:41:36.130 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:41:36.130 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:41:36.130 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:41:36.130 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:36.130 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:41:36.131 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:41:36.131 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:36.131 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:36.131 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:36.131 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:41:36.131 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:41:36.131 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:36.131 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:36.131 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:36.131 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:36.131 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:41:36.131 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:41:36.131 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:41:36.132 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:41:36.132 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:41:36.132 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:36.132 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:36.132 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:41:36.132 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:41:36.132 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:41:36.132 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:41:36.132 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:36.132 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:41:36.132 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:41:36.132 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:36.132 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:36.132 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:36.133 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:41:36.133 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:41:36.133 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:36.133 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:36.133 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:36.133 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:36.133 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:41:36.133 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:41:36.133 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:41:36.133 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:41:36.133 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:41:36.133 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:36.133 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:36.133 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:36.134 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:41:36.134 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:41:36.134 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:36.134 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:41:36.134 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:41:36.134 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:41:36.134 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:36.134 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:36.134 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:41:36.134 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:36.134 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:41:36.134 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:41:36.134 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:41:36.134 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:36.135 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:36.135 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:36.135 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:41:36.135 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:41:36.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.113+0000 7f6706c94640 1 -- 172.21.15.37:0/219601436 >> v2:172.21.15.45:6812/2379918623 conn(0x7f66e0035e80 msgr2=0x7f66e0038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:36.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.113+0000 7f6706c94640 1 --2- 172.21.15.37:0/219601436 >> v2:172.21.15.45:6812/2379918623 conn(0x7f66e0035e80 0x7f66e0038340 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f67000a6d60 tx=0x7f66f80a7000 comp rx=0 tx=0).stop 2024-04-26T00:41:36.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.113+0000 7f6706c94640 10 monclient: shutdown 2024-04-26T00:41:36.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.113+0000 7f6706c94640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:41:36.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.113+0000 7f6706c94640 1 -- 172.21.15.37:0/219601436 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f67000a4170 msgr2=0x7f6700137030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:36.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.113+0000 7f6706c94640 1 --2- 172.21.15.37:0/219601436 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f67000a4170 0x7f6700137030 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f66f000c480 tx=0x7f66f000c940 comp rx=0 tx=0).stop 2024-04-26T00:41:36.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.113+0000 7f6705231640 1 -- 172.21.15.37:0/219601436 reap_dead start 2024-04-26T00:41:36.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.113+0000 7f6705231640 1 -- 172.21.15.37:0/219601436 reap_dead start 2024-04-26T00:41:36.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.113+0000 7f6706c94640 1 -- 172.21.15.37:0/219601436 shutdown_connections 2024-04-26T00:41:36.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.113+0000 7f6706c94640 1 -- 172.21.15.37:0/219601436 >> 172.21.15.37:0/219601436 conn(0x7f67000aa0a0 msgr2=0x7f67000a8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:41:36.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.113+0000 7f6706c94640 1 -- 172.21.15.37:0/219601436 shutdown_connections 2024-04-26T00:41:36.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.113+0000 7f6706c94640 1 -- 172.21.15.37:0/219601436 wait complete. 2024-04-26T00:41:36.137 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:41:36.138 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:41:36.138 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:41:36.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa293cda640 10 monclient: get_monmap_and_config 2024-04-26T00:41:36.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa293cda640 10 monclient: build_initial_monmap 2024-04-26T00:41:36.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa293cda640 10 monclient: monmap: 2024-04-26T00:41:36.254 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:41:36.254 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:36.254 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:41:36.256215+0000 2024-04-26T00:41:36.254 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:41:36.256215+0000 2024-04-26T00:41:36.254 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:41:36.254 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:36.255 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:41:36.255 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:41:36.255 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:41:36.255 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:41:36.255 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:41:36.255 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:36.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa293cda640 1 Processor -- start 2024-04-26T00:41:36.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa293cda640 1 -- start start 2024-04-26T00:41:36.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa293cda640 10 monclient: init 2024-04-26T00:41:36.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa293cda640 10 monclient: _reopen_session rank -1 2024-04-26T00:41:36.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa293cda640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-04-26T00:41:36.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa293cda640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa28c0a0300 0x7fa28c09fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:36.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa293cda640 10 monclient(hunting): picked mon.noname-b con 0x7fa28c0a0300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:41:36.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa293cda640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa28c0a06e0 0x7fa28c134ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:36.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa293cda640 10 monclient(hunting): picked mon.noname-a con 0x7fa28c0a06e0 addr v2:172.21.15.37:3300/0 2024-04-26T00:41:36.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa293cda640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa28c1350e0 addr v1:172.21.15.37:6789/0 2024-04-26T00:41:36.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa293cda640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa28c09c190 con 0x7fa28c1350e0 2024-04-26T00:41:36.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa293cda640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:36.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa293cda640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fa28c0a13f0 con 0x7fa28c0a06e0 2024-04-26T00:41:36.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa293cda640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:36.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa293cda640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa28c0a1110 con 0x7fa28c0a0300 2024-04-26T00:41:36.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa293cda640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:36.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa293cda640 10 monclient(hunting): authenticate will time out at 2277.561878s 2024-04-26T00:41:36.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa291275640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa28c0a06e0 0x7fa28c134ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:36.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa292277640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fa28c1350e0 0x7fa28c13bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:44404/0 (socket says 172.21.15.37:44404) 2024-04-26T00:41:36.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa291a76640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa28c0a0300 0x7fa28c09fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:36.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa292277640 1 -- 172.21.15.37:0/2489361636 learned_addr learned my addr 172.21.15.37:0/2489361636 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:41:36.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa291a76640 10 monclient(hunting): get_auth_request con 0x7fa28c0a0300 auth_method 0 2024-04-26T00:41:36.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa291a76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:36.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa291a76640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:36.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa291a76640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:36.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa291275640 10 monclient(hunting): get_auth_request con 0x7fa28c0a06e0 auth_method 0 2024-04-26T00:41:36.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa291275640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:36.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa291275640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:36.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa291275640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:36.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa291a76640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:36.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa291a76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:36.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa291a76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:36.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa291275640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:36.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa291275640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:36.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa291275640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:36.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa291275640 10 monclient(hunting): handle_auth_done global_id 264129 payload 274 2024-04-26T00:41:36.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa291275640 1 -- 172.21.15.37:0/2489361636 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa28c0a0300 msgr2=0x7fa28c09fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:41:36.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa291275640 1 --2- 172.21.15.37:0/2489361636 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa28c0a0300 0x7fa28c09fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:36.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa291275640 1 -- 172.21.15.37:0/2489361636 >> v1:172.21.15.37:6789/0 conn(0x7fa28c1350e0 legacy=0x7fa28c13bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:41:36.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa291275640 10 monclient: _finish_hunting 0 2024-04-26T00:41:36.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa291275640 1 monclient: found mon.noname-a 2024-04-26T00:41:36.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa291275640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:36.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa291275640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:41:36.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa291275640 1 -- 172.21.15.37:0/2489361636 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa28c0a16d0 con 0x7fa28c0a06e0 2024-04-26T00:41:36.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fa291a76640 1 --2- 172.21.15.37:0/2489361636 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa28c0a0300 0x7fa28c09fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:41:36.262 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:36.262 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=37,osdmap=63} crc {osd_snap=1295733285,osdmap=142450560}) 2024-04-26T00:41:36.262 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=34,osdmap=66} crc {osd_snap=546988371,osdmap=2969785869}) 2024-04-26T00:41:36.262 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:36.262 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=37,osdmap=63} crc {osd_snap=1295733285,osdmap=142450560}) 2024-04-26T00:41:36.262 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:36.253+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=36,osdmap=64} crc {osd_snap=1094513415,osdmap=243624973}) 2024-04-26T00:41:36.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.257+0000 7fa291275640 1 -- 172.21.15.37:0/2489361636 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa28c0a06e0 msgr2=0x7fa28c134ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:36.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.257+0000 7fa291275640 1 --2- 172.21.15.37:0/2489361636 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa28c0a06e0 0x7fa28c134ba0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa2880097c0 tx=0x7fa28801c350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d9f99b63bcb324b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:36.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.257+0000 7fa290a74640 1 -- 172.21.15.37:0/2489361636 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2880b0070 con 0x7fa28c0a06e0 2024-04-26T00:41:36.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.257+0000 7fa290a74640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:36.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.257+0000 7fa290a74640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:41:36.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.257+0000 7fa290a74640 10 monclient: dump: 2024-04-26T00:41:36.263 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:36.263 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:36.263 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.263 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.263 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:36.263 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:36.263 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:36.263 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:36.264 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:36.264 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:36.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.257+0000 7fa290a74640 10 monclient: _finish_auth 0 2024-04-26T00:41:36.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.257+0000 7fa290a74640 10 monclient: _check_auth_tickets 2024-04-26T00:41:36.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.257+0000 7fa290a74640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:41:36.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.257+0000 7fa293cda640 5 monclient: authenticate success, global_id 264129 2024-04-26T00:41:36.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.257+0000 7fa293cda640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:41:36.265 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:36.261+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:36.265 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:36.261+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3958661836}) 2024-04-26T00:41:36.265 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:36.261+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3454369816}) 2024-04-26T00:41:36.265 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:36.261+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:36.265 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:36.261+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3958661836}) 2024-04-26T00:41:36.265 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:36.261+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1278310705}) 2024-04-26T00:41:36.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.261+0000 7fa290a74640 1 -- 172.21.15.37:0/2489361636 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2880ab600 con 0x7fa28c0a06e0 2024-04-26T00:41:36.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.261+0000 7fa290a74640 10 monclient: handle_config config(0 keys) 2024-04-26T00:41:36.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.261+0000 7fa290a74640 1 -- 172.21.15.37:0/2489361636 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2880ab9d0 con 0x7fa28c0a06e0 2024-04-26T00:41:36.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.261+0000 7fa290a74640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:36.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.261+0000 7fa290a74640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:41:36.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.261+0000 7fa290a74640 10 monclient: dump: 2024-04-26T00:41:36.268 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:36.269 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:36.269 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.269 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.269 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:36.269 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:36.269 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:36.269 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:36.269 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:36.269 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:36.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.261+0000 7fa293cda640 10 monclient: get_monmap_and_config success 2024-04-26T00:41:36.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.261+0000 7fa293cda640 4 set_mon_vals no callback set 2024-04-26T00:41:36.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.261+0000 7fa293cda640 10 monclient: shutdown 2024-04-26T00:41:36.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.261+0000 7fa293cda640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:41:36.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.261+0000 7fa293cda640 1 -- 172.21.15.37:0/2489361636 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa28c0a06e0 msgr2=0x7fa28c134ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:36.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.261+0000 7fa293cda640 1 --2- 172.21.15.37:0/2489361636 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa28c0a06e0 0x7fa28c134ba0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa2880097c0 tx=0x7fa28801c350 comp rx=0 tx=0).stop 2024-04-26T00:41:36.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.261+0000 7fa293cda640 1 -- 172.21.15.37:0/2489361636 shutdown_connections 2024-04-26T00:41:36.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.261+0000 7fa293cda640 1 --2- 172.21.15.37:0/2489361636 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa28c0a06e0 0x7fa28c134ba0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:36.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.261+0000 7fa293cda640 1 --2- 172.21.15.37:0/2489361636 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa28c0a0300 0x7fa28c09fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:36.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.261+0000 7fa293cda640 1 -- 172.21.15.37:0/2489361636 >> 172.21.15.37:0/2489361636 conn(0x7fa28c0a1d50 msgr2=0x7fa28c0a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:41:36.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.261+0000 7fa293cda640 1 -- 172.21.15.37:0/2489361636 shutdown_connections 2024-04-26T00:41:36.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.261+0000 7fa293cda640 1 -- 172.21.15.37:0/2489361636 wait complete. 2024-04-26T00:41:36.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.261+0000 7fa293cda640 10 monclient: build_initial_monmap 2024-04-26T00:41:36.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.261+0000 7fa293cda640 10 monclient: monmap: 2024-04-26T00:41:36.270 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:41:36.270 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:41:36.270 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:41:36.270 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:41:36.270 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:41:36.270 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:41:36.271 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:41:36.271 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:41:36.271 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:41:36.271 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:36.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa293cda640 1 Processor -- start 2024-04-26T00:41:36.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa293cda640 1 -- start start 2024-04-26T00:41:36.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa293cda640 10 monclient: init 2024-04-26T00:41:36.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa293cda640 10 monclient: _reopen_session rank -1 2024-04-26T00:41:36.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa293cda640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:41:36.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa293cda640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa28c0a0300 0x7fa28c11cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:36.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa293cda640 10 monclient(hunting): picked mon.noname-c con 0x7fa28c0a0300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:41:36.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa293cda640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa28c0a06e0 0x7fa28c133d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:36.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa293cda640 10 monclient(hunting): picked mon.noname-b con 0x7fa28c0a06e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:41:36.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa293cda640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa28c1350e0 0x7fa28c134290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:36.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa293cda640 10 monclient(hunting): picked mon.noname-a con 0x7fa28c1350e0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:41:36.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa293cda640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:36.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa293cda640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa28c0a0c40 con 0x7fa28c1350e0 2024-04-26T00:41:36.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa293cda640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:36.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa293cda640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa28c0a0ac0 con 0x7fa28c0a06e0 2024-04-26T00:41:36.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa293cda640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:36.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa293cda640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa28c13fe10 con 0x7fa28c0a0300 2024-04-26T00:41:36.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa293cda640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:36.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa293cda640 10 monclient(hunting): authenticate will time out at 2277.572705s 2024-04-26T00:41:36.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa291a76640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa28c0a0300 0x7fa28c11cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:36.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa292277640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa28c1350e0 0x7fa28c134290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:36.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa291a76640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa28c0a0300 0x7fa28c11cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:45328/0 (socket says 172.21.15.37:45328) 2024-04-26T00:41:36.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa292277640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa28c1350e0 0x7fa28c134290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:46006/0 (socket says 172.21.15.37:46006) 2024-04-26T00:41:36.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa291a76640 1 -- 172.21.15.37:0/3737690511 learned_addr learned my addr 172.21.15.37:0/3737690511 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:41:36.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa291a76640 10 monclient(hunting): get_auth_request con 0x7fa28c0a0300 auth_method 0 2024-04-26T00:41:36.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa291a76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:36.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa291a76640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:36.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa291a76640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:36.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa292277640 10 monclient(hunting): get_auth_request con 0x7fa28c1350e0 auth_method 0 2024-04-26T00:41:36.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa292277640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:36.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa292277640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:36.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa292277640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:36.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa291a76640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:36.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa291a76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:36.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa291a76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:36.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa292277640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:36.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa292277640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:36.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa292277640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:36.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa291275640 1 --2- 172.21.15.37:0/3737690511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa28c0a06e0 0x7fa28c133d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:36.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa291275640 10 monclient(hunting): get_auth_request con 0x7fa28c0a06e0 auth_method 0 2024-04-26T00:41:36.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa291275640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:36.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa291275640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:36.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa291275640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:36.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa291a76640 10 monclient(hunting): handle_auth_done global_id 254249 payload 824 2024-04-26T00:41:36.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa291a76640 1 -- 172.21.15.37:0/3737690511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa28c0a06e0 msgr2=0x7fa28c133d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:36.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa291a76640 1 --2- 172.21.15.37:0/3737690511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa28c0a06e0 0x7fa28c133d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:36.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa291a76640 1 -- 172.21.15.37:0/3737690511 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa28c1350e0 msgr2=0x7fa28c134290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:36.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa291a76640 1 --2- 172.21.15.37:0/3737690511 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa28c1350e0 0x7fa28c134290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:36.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa291a76640 10 monclient: _finish_hunting 0 2024-04-26T00:41:36.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa291a76640 1 monclient: found mon.noname-c 2024-04-26T00:41:36.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa291a76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:36.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa291a76640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:41:36.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa291a76640 1 -- 172.21.15.37:0/3737690511 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa28c120420 con 0x7fa28c0a0300 2024-04-26T00:41:36.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa292277640 1 --2- 172.21.15.37:0/3737690511 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa28c1350e0 0x7fa28c134290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:41:36.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa291a76640 1 -- 172.21.15.37:0/3737690511 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa28c0a0300 msgr2=0x7fa28c11cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:36.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.265+0000 7fa291a76640 1 --2- 172.21.15.37:0/3737690511 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa28c0a0300 0x7fa28c11cd00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa28000ebe0 tx=0x7fa28000c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:36.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.269+0000 7fa27e7fc640 1 -- 172.21.15.37:0/3737690511 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa280017070 con 0x7fa28c0a0300 2024-04-26T00:41:36.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.269+0000 7fa27e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:36.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.269+0000 7fa27e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:41:36.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.269+0000 7fa27e7fc640 10 monclient: dump: 2024-04-26T00:41:36.278 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:36.278 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:36.278 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.278 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.278 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:36.278 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:36.278 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:36.279 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:36.279 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:36.279 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:36.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.269+0000 7fa27e7fc640 10 monclient: _finish_auth 0 2024-04-26T00:41:36.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.269+0000 7fa27e7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:41:36.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.269+0000 7fa27e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:41:36.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.269+0000 7fa27e7fc640 1 -- 172.21.15.37:0/3737690511 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa28000ce40 con 0x7fa28c0a0300 2024-04-26T00:41:36.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.269+0000 7fa27e7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:41:36.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.269+0000 7fa27e7fc640 1 -- 172.21.15.37:0/3737690511 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa280013690 con 0x7fa28c0a0300 2024-04-26T00:41:36.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.269+0000 7fa293cda640 5 monclient: authenticate success, global_id 254249 2024-04-26T00:41:36.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.269+0000 7fa293cda640 10 monclient: _renew_subs 2024-04-26T00:41:36.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.269+0000 7fa293cda640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:41:36.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.269+0000 7fa293cda640 1 -- 172.21.15.37:0/3737690511 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa28c11fe60 con 0x7fa28c0a0300 2024-04-26T00:41:36.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.269+0000 7fa27e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:36.279 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.269+0000 7fa27ffff640 4 set_mon_vals no callback set 2024-04-26T00:41:36.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.269+0000 7fa27e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:41:36.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.269+0000 7fa27e7fc640 10 monclient: dump: 2024-04-26T00:41:36.280 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:36.280 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:36.280 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.280 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.280 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:36.280 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:36.280 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:36.280 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:36.280 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:36.280 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:36.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.269+0000 7fa293cda640 10 monclient: _renew_subs 2024-04-26T00:41:36.280 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.269+0000 7fa293cda640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:41:36.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.269+0000 7fa293cda640 1 -- 172.21.15.37:0/3737690511 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa28c11d240 con 0x7fa28c0a0300 2024-04-26T00:41:36.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.269+0000 7fa293cda640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:36.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.269+0000 7fa293cda640 1 --2- 172.21.15.37:0/3737690511 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa28c1cf330 0x7fa28c1cf710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:36.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.269+0000 7fa293cda640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:36.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.269+0000 7fa293cda640 1 -- 172.21.15.37:0/3737690511 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa28c11e430 con 0x7fa28c1cf330 2024-04-26T00:41:36.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.269+0000 7fa27e7fc640 1 -- 172.21.15.37:0/3737690511 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa28001a070 con 0x7fa28c0a0300 2024-04-26T00:41:36.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.269+0000 7fa291275640 1 --2- 172.21.15.37:0/3737690511 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa28c1cf330 0x7fa28c1cf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:36.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.269+0000 7fa291275640 10 monclient: get_auth_request con 0x7fa28c1cf330 auth_method 0 2024-04-26T00:41:36.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.269+0000 7fa291275640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:36.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.269+0000 7fa291275640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:36.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.269+0000 7fa291275640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:36.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.269+0000 7fa291275640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:36.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.269+0000 7fa291275640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:36.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.269+0000 7fa291275640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:36.281 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.269+0000 7fa27e7fc640 1 --2- 172.21.15.37:0/3737690511 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa270035ed0 0x7fa270038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:36.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.269+0000 7fa27e7fc640 1 -- 172.21.15.37:0/3737690511 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(863..863 src has 705..863) ==== 3975+0+0 (secure 0 0 0) 0x7fa280048b00 con 0x7fa28c0a0300 2024-04-26T00:41:36.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.269+0000 7fa291275640 10 monclient(hunting): handle_auth_done global_id 254252 payload 824 2024-04-26T00:41:36.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.269+0000 7fa291275640 1 -- 172.21.15.37:0/3737690511 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa28c04f510 con 0x7fa28c1cf330 2024-04-26T00:41:36.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.269+0000 7fa292277640 1 --2- 172.21.15.37:0/3737690511 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa270035ed0 0x7fa270038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:36.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.269+0000 7fa292277640 10 monclient: get_auth_request con 0x7fa270035ed0 auth_method 0 2024-04-26T00:41:36.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.273+0000 7fa291275640 1 -- 172.21.15.37:0/3737690511 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa28c1cf330 msgr2=0x7fa28c1cf710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:36.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.273+0000 7fa291275640 1 --2- 172.21.15.37:0/3737690511 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa28c1cf330 0x7fa28c1cf710 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa2880ad000 tx=0x7fa28801abf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:36.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.273+0000 7fa27e7fc640 1 -- 172.21.15.37:0/3737690511 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2880b0050 con 0x7fa28c1cf330 2024-04-26T00:41:36.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.273+0000 7fa27e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:36.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.273+0000 7fa27e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:41:36.282 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.273+0000 7fa27e7fc640 10 monclient: dump: 2024-04-26T00:41:36.282 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:36.282 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:36.282 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.283 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.283 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:36.283 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:36.283 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:36.283 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:36.283 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:36.283 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:36.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.273+0000 7fa27e7fc640 1 -- 172.21.15.37:0/3737690511 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa2880b1040 con 0x7fa28c1cf330 2024-04-26T00:41:36.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.273+0000 7fa27e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:36.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.273+0000 7fa27e7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:41:36.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.273+0000 7fa27e7fc640 1 -- 172.21.15.37:0/3737690511 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa28c1cf330 msgr2=0x7fa28c1cf710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:36.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.273+0000 7fa27e7fc640 1 --2- 172.21.15.37:0/3737690511 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa28c1cf330 0x7fa28c1cf710 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa2880ad000 tx=0x7fa28801abf0 comp rx=0 tx=0).stop 2024-04-26T00:41:36.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.273+0000 7fa27e7fc640 1 -- 172.21.15.37:0/3737690511 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa28c1cf330 msgr2=0x7fa28c1cf710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:41:36.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.273+0000 7fa27e7fc640 1 --2- 172.21.15.37:0/3737690511 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa28c1cf330 0x7fa28c1cf710 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa2880ad000 tx=0x7fa28801abf0 comp rx=0 tx=0).stop 2024-04-26T00:41:36.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.273+0000 7fa292277640 1 --2- 172.21.15.37:0/3737690511 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa270035ed0 0x7fa270038390 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fa284005c10 tx=0x7fa284009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:36.284 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:36.273+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:36.284 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:36.273+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=80} crc {osdmap=1217027724}) 2024-04-26T00:41:36.284 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:36.273+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=79} crc {osdmap=2307550138}) 2024-04-26T00:41:36.284 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:36.273+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:41:36.284 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:36.273+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=80} crc {osdmap=1217027724}) 2024-04-26T00:41:36.285 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:41:36.273+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=78} crc {osdmap=2172229699}) 2024-04-26T00:41:36.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.277+0000 7fa293cda640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:41:36.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.277+0000 7fa293cda640 1 --2- 172.21.15.37:0/3737690511 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa28c1cfc50 0x7fa28c13a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:36.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.277+0000 7fa293cda640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:36.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.277+0000 7fa293cda640 1 -- 172.21.15.37:0/3737690511 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa28c11dd30 con 0x7fa28c1cfc50 2024-04-26T00:41:36.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.277+0000 7fa291275640 1 --2- 172.21.15.37:0/3737690511 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa28c1cfc50 0x7fa28c13a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:36.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.277+0000 7fa291275640 10 monclient: get_auth_request con 0x7fa28c1cfc50 auth_method 0 2024-04-26T00:41:36.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.277+0000 7fa291275640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:36.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.277+0000 7fa291275640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:36.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.277+0000 7fa291275640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:36.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.281+0000 7fa291275640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:36.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.281+0000 7fa291275640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:36.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.281+0000 7fa291275640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:36.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.281+0000 7fa291275640 10 monclient(hunting): handle_auth_done global_id 254255 payload 824 2024-04-26T00:41:36.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.281+0000 7fa291275640 1 -- 172.21.15.37:0/3737690511 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa28c09eb20 con 0x7fa28c1cfc50 2024-04-26T00:41:36.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.281+0000 7fa291275640 1 -- 172.21.15.37:0/3737690511 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa28c1cfc50 msgr2=0x7fa28c13a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:36.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.281+0000 7fa291275640 1 --2- 172.21.15.37:0/3737690511 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa28c1cfc50 0x7fa28c13a070 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa2880ae040 tx=0x7fa2880a8000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:36.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.281+0000 7fa27e7fc640 1 -- 172.21.15.37:0/3737690511 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2880bfb30 con 0x7fa28c1cfc50 2024-04-26T00:41:36.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.281+0000 7fa27e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:36.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.281+0000 7fa27e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:41:36.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.281+0000 7fa27e7fc640 10 monclient: dump: 2024-04-26T00:41:36.287 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:36.288 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:36.288 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.288 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.288 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:36.288 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:36.288 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:36.288 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:36.288 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:36.288 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:36.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.281+0000 7fa27e7fc640 1 -- 172.21.15.37:0/3737690511 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4719 (secure 0 0 0) 0x7fa2880bfb30 con 0x7fa28c1cfc50 2024-04-26T00:41:36.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.281+0000 7fa27e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:41:36.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.281+0000 7fa27e7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:41:36.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.281+0000 7fa27e7fc640 1 -- 172.21.15.37:0/3737690511 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa28c1cfc50 msgr2=0x7fa28c13a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:36.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.281+0000 7fa27e7fc640 1 --2- 172.21.15.37:0/3737690511 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa28c1cfc50 0x7fa28c13a070 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa2880ae040 tx=0x7fa2880a8000 comp rx=0 tx=0).stop 2024-04-26T00:41:36.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.281+0000 7fa27e7fc640 1 -- 172.21.15.37:0/3737690511 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa28c1cfc50 msgr2=0x7fa28c13a070 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:41:36.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.281+0000 7fa27e7fc640 1 --2- 172.21.15.37:0/3737690511 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa28c1cfc50 0x7fa28c13a070 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:36.289 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:41:36.289 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:41:36.289 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:41:36.289 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:41:36.289 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 156, 2024-04-26T00:41:36.289 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 65051, 2024-04-26T00:41:36.289 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:41:36.289 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:41:36.290 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:41:36.290 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:41:36.290 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:36.290 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 5, 2024-04-26T00:41:36.290 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:41:36.290 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:41:36.290 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:41:36.290 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:41:36.290 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:41:36.290 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:41:36.290 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:41:36.290 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:41:36.290 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:41:36.291 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:41:36.291 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:41:36.291 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:41:36.291 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:41:36.291 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:41:36.291 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:36.291 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:41:36.291 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:41:36.291 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:41:36.291 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:41:36.291 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:41:36.291 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:41:36.291 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:41:36.291 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:41:36.291 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:41:36.292 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:41:36.292 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:41:36.292 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:41:36.292 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:41:36.292 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:36.292 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:36.292 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:41:36.292 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:41:36.292 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:41:36.292 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:41:36.292 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:41:36.292 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:41:36.292 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:41:36.292 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:41:36.293 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:41:36.293 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:41:36.293 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:41:36.293 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:41:36.293 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:41:36.293 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:41:36.293 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:41:36.293 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:41:36.293 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:41:36.293 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:41:36.293 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:41:36.293 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:41:36.294 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:41:36.294 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:41:36.294 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:41:36.294 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:41:36.294 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:41:36.294 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:41:36.294 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:41:36.294 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:41:36.294 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:36.294 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:41:36.294 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:36.294 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:41:36.294 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:36.294 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:41:36.295 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:41:36.295 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:41:36.295 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:41:36.295 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:36.295 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:41:36.295 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:41:36.295 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:36.295 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:36.295 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:36.295 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:41:36.295 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:41:36.295 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:36.295 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:36.295 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:36.296 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:36.296 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:41:36.296 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:41:36.296 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:41:36.296 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:41:36.296 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:41:36.296 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:36.296 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:36.296 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:41:36.297 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:41:36.297 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:41:36.297 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:41:36.297 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:36.297 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:41:36.297 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:41:36.297 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:36.297 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:36.297 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:36.297 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:41:36.297 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:41:36.298 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:36.298 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:36.298 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:36.298 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:36.298 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:41:36.298 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:41:36.298 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:41:36.298 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:41:36.298 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:41:36.298 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:36.298 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:36.298 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:41:36.298 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:41:36.299 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:41:36.299 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:41:36.299 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:36.299 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:41:36.299 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:41:36.299 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:36.299 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:36.299 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:36.299 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:41:36.299 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:41:36.299 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:36.299 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:36.299 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:36.299 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:36.300 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:41:36.300 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:41:36.300 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:41:36.300 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:41:36.300 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:41:36.300 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:36.300 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:36.300 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:36.300 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:41:36.300 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:41:36.300 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:36.300 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:41:36.300 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:41:36.300 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:41:36.301 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:36.301 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:36.301 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:41:36.301 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:36.301 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:41:36.301 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:41:36.301 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:41:36.301 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:36.301 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:36.301 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:41:36.301 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:36.301 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:41:36.301 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:41:36.301 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:41:36.302 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:36.302 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:36.302 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:36.302 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:41:36.302 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:41:36.303 INFO:tasks.rados.rados.0.smithi045.stdout:3835: done (1 left) 2024-04-26T00:41:36.303 INFO:tasks.rados.rados.0.smithi045.stdout:3836: done (0 left) 2024-04-26T00:41:36.303 INFO:tasks.rados.rados.0.smithi045.stdout:3837: snap_remove snap 422 2024-04-26T00:41:36.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.281+0000 7fa293cda640 1 -- 172.21.15.37:0/3737690511 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa270035ed0 msgr2=0x7fa270038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:36.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.281+0000 7fa293cda640 1 --2- 172.21.15.37:0/3737690511 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa270035ed0 0x7fa270038390 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fa284005c10 tx=0x7fa284009040 comp rx=0 tx=0).stop 2024-04-26T00:41:36.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.281+0000 7fa293cda640 10 monclient: shutdown 2024-04-26T00:41:36.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.281+0000 7fa293cda640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:41:36.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.281+0000 7fa293cda640 1 -- 172.21.15.37:0/3737690511 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa28c0a0300 msgr2=0x7fa28c11cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:36.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.281+0000 7fa293cda640 1 --2- 172.21.15.37:0/3737690511 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa28c0a0300 0x7fa28c11cd00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa28000ebe0 tx=0x7fa28000c520 comp rx=0 tx=0).stop 2024-04-26T00:41:36.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.285+0000 7fa292277640 1 -- 172.21.15.37:0/3737690511 reap_dead start 2024-04-26T00:41:36.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.285+0000 7fa292277640 1 -- 172.21.15.37:0/3737690511 reap_dead start 2024-04-26T00:41:36.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.285+0000 7fa293cda640 1 -- 172.21.15.37:0/3737690511 shutdown_connections 2024-04-26T00:41:36.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.285+0000 7fa293cda640 1 -- 172.21.15.37:0/3737690511 >> 172.21.15.37:0/3737690511 conn(0x7fa28c0a1d50 msgr2=0x7fa28c09c750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:41:36.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.285+0000 7fa293cda640 1 -- 172.21.15.37:0/3737690511 shutdown_connections 2024-04-26T00:41:36.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.285+0000 7fa293cda640 1 -- 172.21.15.37:0/3737690511 wait complete. 2024-04-26T00:41:36.305 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:41:36.305 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:41:36.306 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:41:36.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa928cf7640 10 monclient: get_monmap_and_config 2024-04-26T00:41:36.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa928cf7640 10 monclient: build_initial_monmap 2024-04-26T00:41:36.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa928cf7640 10 monclient: monmap: 2024-04-26T00:41:36.422 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:41:36.422 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:36.422 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:41:36.424158+0000 2024-04-26T00:41:36.422 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:41:36.424158+0000 2024-04-26T00:41:36.422 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:41:36.422 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:36.422 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:41:36.423 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:41:36.423 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:41:36.423 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:41:36.423 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:41:36.423 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:36.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa928cf7640 1 Processor -- start 2024-04-26T00:41:36.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa928cf7640 1 -- start start 2024-04-26T00:41:36.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa928cf7640 10 monclient: init 2024-04-26T00:41:36.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa928cf7640 10 monclient: _reopen_session rank -1 2024-04-26T00:41:36.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa928cf7640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-04-26T00:41:36.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa928cf7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa924136f20 addr v1:172.21.15.45:6789/0 2024-04-26T00:41:36.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa928cf7640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa924137a10 0x7fa92409bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:36.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa928cf7640 10 monclient(hunting): picked mon.noname-c con 0x7fa924137a10 addr v2:172.21.15.45:3300/0 2024-04-26T00:41:36.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa928cf7640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa92413c450 0x7fa92413e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:36.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa928cf7640 10 monclient(hunting): picked mon.noname-a con 0x7fa92413c450 addr v2:172.21.15.37:3300/0 2024-04-26T00:41:36.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa928cf7640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9240a50f0 con 0x7fa924136f20 2024-04-26T00:41:36.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa928cf7640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:36.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa928cf7640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fa92409b330 con 0x7fa92413c450 2024-04-26T00:41:36.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa928cf7640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:36.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa928cf7640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fa92409b610 con 0x7fa924137a10 2024-04-26T00:41:36.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa928cf7640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:36.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa928cf7640 10 monclient(hunting): authenticate will time out at 2277.729803s 2024-04-26T00:41:36.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa922d9d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa92413c450 0x7fa92413e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:36.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa922d9d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa92413c450 0x7fa92413e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:46012/0 (socket says 172.21.15.37:46012) 2024-04-26T00:41:36.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa922d9d640 1 -- 172.21.15.37:0/437949691 learned_addr learned my addr 172.21.15.37:0/437949691 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:41:36.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa922d9d640 10 monclient(hunting): get_auth_request con 0x7fa92413c450 auth_method 0 2024-04-26T00:41:36.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa922d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:36.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa922d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:36.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa922d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:36.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa922d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:36.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa922d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:36.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa921d9b640 1 --2- 172.21.15.37:0/437949691 >> v2:172.21.15.45:3300/0 conn(0x7fa924137a10 0x7fa92409bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:36.426 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa922d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:36.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa921d9b640 10 monclient(hunting): get_auth_request con 0x7fa924137a10 auth_method 0 2024-04-26T00:41:36.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa921d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:36.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa921d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:36.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa921d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:36.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa922d9d640 10 monclient(hunting): handle_auth_done global_id 264135 payload 274 2024-04-26T00:41:36.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa922d9d640 1 -- 172.21.15.37:0/437949691 >> v2:172.21.15.45:3300/0 conn(0x7fa924137a10 msgr2=0x7fa92409bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:41:36.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa922d9d640 1 --2- 172.21.15.37:0/437949691 >> v2:172.21.15.45:3300/0 conn(0x7fa924137a10 0x7fa92409bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:36.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa922d9d640 1 -- 172.21.15.37:0/437949691 >> v1:172.21.15.45:6789/0 conn(0x7fa924136f20 legacy=0x7fa924137300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:41:36.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa922d9d640 10 monclient: _finish_hunting 0 2024-04-26T00:41:36.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa922d9d640 1 monclient: found mon.noname-a 2024-04-26T00:41:36.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa922d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:36.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa922d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:41:36.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa922d9d640 1 -- 172.21.15.37:0/437949691 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa92409b8f0 con 0x7fa92413c450 2024-04-26T00:41:36.427 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.421+0000 7fa921d9b640 1 --2- 172.21.15.37:0/437949691 >> v2:172.21.15.45:3300/0 conn(0x7fa924137a10 0x7fa92409bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:41:36.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.425+0000 7fa922d9d640 1 -- 172.21.15.37:0/437949691 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa92413c450 msgr2=0x7fa92413e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:36.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.425+0000 7fa922d9d640 1 --2- 172.21.15.37:0/437949691 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa92413c450 0x7fa92413e840 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa924137ec0 tx=0x7fa91401c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdee77b9ee36a05f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:36.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.425+0000 7fa92159a640 1 -- 172.21.15.37:0/437949691 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9140ab650 con 0x7fa92413c450 2024-04-26T00:41:36.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.425+0000 7fa92159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:36.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.425+0000 7fa92159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:41:36.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.425+0000 7fa92159a640 10 monclient: dump: 2024-04-26T00:41:36.429 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:36.429 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:36.429 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.429 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.429 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:36.429 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:36.429 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:36.429 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:36.429 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:36.429 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:36.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.425+0000 7fa92159a640 10 monclient: _finish_auth 0 2024-04-26T00:41:36.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.425+0000 7fa92159a640 10 monclient: _check_auth_tickets 2024-04-26T00:41:36.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.425+0000 7fa92159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:41:36.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.425+0000 7fa92159a640 1 -- 172.21.15.37:0/437949691 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9140b0070 con 0x7fa92413c450 2024-04-26T00:41:36.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.425+0000 7fa92159a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:41:36.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.425+0000 7fa92159a640 1 -- 172.21.15.37:0/437949691 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9140aba10 con 0x7fa92413c450 2024-04-26T00:41:36.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.425+0000 7fa928cf7640 5 monclient: authenticate success, global_id 264135 2024-04-26T00:41:36.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.425+0000 7fa928cf7640 10 monclient: get_monmap_and_config success 2024-04-26T00:41:36.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.425+0000 7fa928cf7640 4 set_mon_vals no callback set 2024-04-26T00:41:36.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.425+0000 7fa92159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:36.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.425+0000 7fa92159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:41:36.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.425+0000 7fa92159a640 10 monclient: dump: 2024-04-26T00:41:36.430 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:36.431 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:36.431 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.431 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.431 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:36.431 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:36.431 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:36.431 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:36.431 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:36.431 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:36.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.425+0000 7fa928cf7640 10 monclient: shutdown 2024-04-26T00:41:36.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.425+0000 7fa928cf7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:41:36.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.425+0000 7fa928cf7640 1 -- 172.21.15.37:0/437949691 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa92413c450 msgr2=0x7fa92413e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:36.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.425+0000 7fa928cf7640 1 --2- 172.21.15.37:0/437949691 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa92413c450 0x7fa92413e840 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa924137ec0 tx=0x7fa91401c430 comp rx=0 tx=0).stop 2024-04-26T00:41:36.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.425+0000 7fa928cf7640 1 -- 172.21.15.37:0/437949691 shutdown_connections 2024-04-26T00:41:36.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.425+0000 7fa928cf7640 1 --2- 172.21.15.37:0/437949691 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa92413c450 0x7fa92413e840 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:36.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.425+0000 7fa928cf7640 1 --2- 172.21.15.37:0/437949691 >> v2:172.21.15.45:3300/0 conn(0x7fa924137a10 0x7fa92409bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:36.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.425+0000 7fa928cf7640 1 -- 172.21.15.37:0/437949691 >> 172.21.15.37:0/437949691 conn(0x7fa9240a61b0 msgr2=0x7fa9240a65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:41:36.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.425+0000 7fa928cf7640 1 -- 172.21.15.37:0/437949691 shutdown_connections 2024-04-26T00:41:36.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.425+0000 7fa928cf7640 1 -- 172.21.15.37:0/437949691 wait complete. 2024-04-26T00:41:36.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa928cf7640 10 monclient: build_initial_monmap 2024-04-26T00:41:36.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa928cf7640 10 monclient: monmap: 2024-04-26T00:41:36.437 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:41:36.437 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:41:36.437 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:41:36.437 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:41:36.437 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:41:36.437 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:41:36.437 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:41:36.437 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:41:36.437 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:41:36.437 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:36.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa928cf7640 1 Processor -- start 2024-04-26T00:41:36.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa928cf7640 1 -- start start 2024-04-26T00:41:36.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa928cf7640 10 monclient: init 2024-04-26T00:41:36.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa928cf7640 10 monclient: _reopen_session rank -1 2024-04-26T00:41:36.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa928cf7640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:41:36.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa928cf7640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa924136f20 0x7fa9241cebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:36.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa928cf7640 10 monclient(hunting): picked mon.noname-c con 0x7fa924136f20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:41:36.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa928cf7640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa924137a10 0x7fa924134da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:36.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa928cf7640 10 monclient(hunting): picked mon.noname-a con 0x7fa924137a10 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:41:36.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa928cf7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa92413c450 0x7fa924137f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:36.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa928cf7640 10 monclient(hunting): picked mon.noname-b con 0x7fa92413c450 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:41:36.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa928cf7640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:36.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa928cf7640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa924141930 con 0x7fa924137a10 2024-04-26T00:41:36.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa928cf7640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:36.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa928cf7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa9241417b0 con 0x7fa92413c450 2024-04-26T00:41:36.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa928cf7640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:36.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa928cf7640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa924141ab0 con 0x7fa924136f20 2024-04-26T00:41:36.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa928cf7640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:36.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa928cf7640 10 monclient(hunting): authenticate will time out at 2277.737084s 2024-04-26T00:41:36.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa921d9b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa924137a10 0x7fa924134da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:36.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa92259c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa924136f20 0x7fa9241cebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:36.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa921d9b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa924137a10 0x7fa924134da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:46028/0 (socket says 172.21.15.37:46028) 2024-04-26T00:41:36.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa921d9b640 1 -- 172.21.15.37:0/3998742547 learned_addr learned my addr 172.21.15.37:0/3998742547 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:41:36.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa921d9b640 10 monclient(hunting): get_auth_request con 0x7fa924137a10 auth_method 0 2024-04-26T00:41:36.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa921d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:36.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa921d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:36.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa921d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:36.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa92259c640 10 monclient(hunting): get_auth_request con 0x7fa924136f20 auth_method 0 2024-04-26T00:41:36.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa92259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:36.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa92259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:36.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa92259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:36.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa921d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:36.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa921d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:36.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa921d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:36.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa92259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:36.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa92259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:36.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa92259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:36.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa922d9d640 1 --2- 172.21.15.37:0/3998742547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa92413c450 0x7fa924137f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:36.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa922d9d640 10 monclient(hunting): get_auth_request con 0x7fa92413c450 auth_method 0 2024-04-26T00:41:36.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa922d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:36.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa922d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:36.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa922d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:36.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa921d9b640 10 monclient(hunting): handle_auth_done global_id 264138 payload 824 2024-04-26T00:41:36.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa921d9b640 1 -- 172.21.15.37:0/3998742547 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa924136f20 msgr2=0x7fa9241cebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:36.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa921d9b640 1 --2- 172.21.15.37:0/3998742547 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa924136f20 0x7fa9241cebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:36.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa921d9b640 1 -- 172.21.15.37:0/3998742547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa92413c450 msgr2=0x7fa924137f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:36.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa921d9b640 1 --2- 172.21.15.37:0/3998742547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa92413c450 0x7fa924137f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:36.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa921d9b640 10 monclient: _finish_hunting 0 2024-04-26T00:41:36.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa921d9b640 1 monclient: found mon.noname-a 2024-04-26T00:41:36.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa921d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:36.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa921d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:41:36.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa921d9b640 1 -- 172.21.15.37:0/3998742547 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa92409ae30 con 0x7fa924137a10 2024-04-26T00:41:36.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa92259c640 1 --2- 172.21.15.37:0/3998742547 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa924136f20 0x7fa9241cebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:41:36.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa922d9d640 1 --2- 172.21.15.37:0/3998742547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa92413c450 0x7fa924137f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:41:36.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa921d9b640 1 -- 172.21.15.37:0/3998742547 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa924137a10 msgr2=0x7fa924134da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:36.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.429+0000 7fa921d9b640 1 --2- 172.21.15.37:0/3998742547 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa924137a10 0x7fa924134da0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa90c00e810 tx=0x7fa90c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:36.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.433+0000 7fa91affd640 1 -- 172.21.15.37:0/3998742547 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90c017070 con 0x7fa924137a10 2024-04-26T00:41:36.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.433+0000 7fa91affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:36.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.433+0000 7fa91affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:41:36.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.433+0000 7fa91affd640 10 monclient: dump: 2024-04-26T00:41:36.442 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:36.442 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:36.442 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.442 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.442 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:36.442 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:36.442 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:36.442 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:36.443 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:36.443 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:36.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.433+0000 7fa91affd640 10 monclient: _finish_auth 0 2024-04-26T00:41:36.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.433+0000 7fa91affd640 10 monclient: _check_auth_tickets 2024-04-26T00:41:36.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.433+0000 7fa91affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:41:36.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.433+0000 7fa91affd640 1 -- 172.21.15.37:0/3998742547 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa90c00cae0 con 0x7fa924137a10 2024-04-26T00:41:36.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.433+0000 7fa928cf7640 5 monclient: authenticate success, global_id 264138 2024-04-26T00:41:36.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.433+0000 7fa928cf7640 10 monclient: _renew_subs 2024-04-26T00:41:36.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.433+0000 7fa928cf7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:41:36.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.433+0000 7fa928cf7640 1 -- 172.21.15.37:0/3998742547 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa924135f30 con 0x7fa924137a10 2024-04-26T00:41:36.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.433+0000 7fa91affd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:41:36.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.433+0000 7fa91affd640 1 -- 172.21.15.37:0/3998742547 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90c00f040 con 0x7fa924137a10 2024-04-26T00:41:36.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.433+0000 7fa91affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:36.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.433+0000 7fa91affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:41:36.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.433+0000 7fa91affd640 10 monclient: dump: 2024-04-26T00:41:36.444 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:36.444 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:36.444 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.444 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.444 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:36.444 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:36.444 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:36.444 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:36.444 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:36.444 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:36.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.433+0000 7fa928cf7640 10 monclient: _renew_subs 2024-04-26T00:41:36.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.433+0000 7fa928cf7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:41:36.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.433+0000 7fa928cf7640 1 -- 172.21.15.37:0/3998742547 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa92413b890 con 0x7fa924137a10 2024-04-26T00:41:36.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.433+0000 7fa920d99640 4 set_mon_vals no callback set 2024-04-26T00:41:36.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.433+0000 7fa928cf7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:36.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.433+0000 7fa928cf7640 1 --2- 172.21.15.37:0/3998742547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa924138c30 0x7fa92402c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:36.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.433+0000 7fa928cf7640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:36.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.433+0000 7fa928cf7640 1 -- 172.21.15.37:0/3998742547 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa92404f510 con 0x7fa924138c30 2024-04-26T00:41:36.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.433+0000 7fa91affd640 1 -- 172.21.15.37:0/3998742547 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa90c01a070 con 0x7fa924137a10 2024-04-26T00:41:36.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.433+0000 7fa91affd640 1 --2- 172.21.15.37:0/3998742547 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa908035e80 0x7fa908038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:36.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.433+0000 7fa92259c640 1 --2- 172.21.15.37:0/3998742547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa924138c30 0x7fa92402c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:36.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.433+0000 7fa91affd640 1 -- 172.21.15.37:0/3998742547 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(863..863 src has 705..863) ==== 3975+0+0 (secure 0 0 0) 0x7fa90c0487b0 con 0x7fa924137a10 2024-04-26T00:41:36.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.433+0000 7fa92259c640 10 monclient: get_auth_request con 0x7fa924138c30 auth_method 0 2024-04-26T00:41:36.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.433+0000 7fa92259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:36.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.433+0000 7fa92259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:36.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.433+0000 7fa92259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:36.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.433+0000 7fa92259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:36.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.433+0000 7fa92259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:36.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.433+0000 7fa92259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:36.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.433+0000 7fa922d9d640 1 --2- 172.21.15.37:0/3998742547 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa908035e80 0x7fa908038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:36.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.433+0000 7fa922d9d640 10 monclient: get_auth_request con 0x7fa908035e80 auth_method 0 2024-04-26T00:41:36.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.433+0000 7fa92259c640 10 monclient(hunting): handle_auth_done global_id 224485 payload 824 2024-04-26T00:41:36.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.433+0000 7fa92259c640 1 -- 172.21.15.37:0/3998742547 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa9240a3000 con 0x7fa924138c30 2024-04-26T00:41:36.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.437+0000 7fa92259c640 1 -- 172.21.15.37:0/3998742547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa924138c30 msgr2=0x7fa92402c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:36.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.437+0000 7fa92259c640 1 --2- 172.21.15.37:0/3998742547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa924138c30 0x7fa92402c000 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa91c0048d0 tx=0x7fa91c00d520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:36.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.437+0000 7fa922d9d640 1 --2- 172.21.15.37:0/3998742547 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa908035e80 0x7fa908038340 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fa924068940 tx=0x7fa9140a7000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:36.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.437+0000 7fa91affd640 1 -- 172.21.15.37:0/3998742547 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa91c004960 con 0x7fa924138c30 2024-04-26T00:41:36.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.437+0000 7fa91affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:36.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.437+0000 7fa91affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:41:36.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.437+0000 7fa91affd640 10 monclient: dump: 2024-04-26T00:41:36.448 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:36.448 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:36.448 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.448 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.448 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:36.448 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:36.449 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:36.449 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:36.449 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:36.449 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:36.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.437+0000 7fa91affd640 1 -- 172.21.15.37:0/3998742547 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa91c004960 con 0x7fa924138c30 2024-04-26T00:41:36.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.437+0000 7fa91affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:36.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.437+0000 7fa91affd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:41:36.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.437+0000 7fa91affd640 1 -- 172.21.15.37:0/3998742547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa924138c30 msgr2=0x7fa92402c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:36.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.437+0000 7fa91affd640 1 --2- 172.21.15.37:0/3998742547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa924138c30 0x7fa92402c000 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa91c0048d0 tx=0x7fa91c00d520 comp rx=0 tx=0).stop 2024-04-26T00:41:36.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.437+0000 7fa91affd640 1 -- 172.21.15.37:0/3998742547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa924138c30 msgr2=0x7fa92402c000 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:41:36.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.437+0000 7fa91affd640 1 --2- 172.21.15.37:0/3998742547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa924138c30 0x7fa92402c000 secure :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa91c0048d0 tx=0x7fa91c00d520 comp rx=0 tx=0).stop 2024-04-26T00:41:36.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.441+0000 7fa928cf7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:41:36.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.441+0000 7fa928cf7640 1 --2- 172.21.15.37:0/3998742547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa92402c540 0x7fa92402c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:36.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.445+0000 7fa928cf7640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:36.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.445+0000 7fa928cf7640 1 -- 172.21.15.37:0/3998742547 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa924075f10 con 0x7fa92402c540 2024-04-26T00:41:36.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.445+0000 7fa92259c640 1 --2- 172.21.15.37:0/3998742547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa92402c540 0x7fa92402c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:36.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.445+0000 7fa92259c640 10 monclient: get_auth_request con 0x7fa92402c540 auth_method 0 2024-04-26T00:41:36.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.445+0000 7fa92259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:36.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.445+0000 7fa92259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:36.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.445+0000 7fa92259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:36.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.445+0000 7fa92259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:36.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.445+0000 7fa92259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:36.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.445+0000 7fa92259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:36.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.445+0000 7fa92259c640 10 monclient(hunting): handle_auth_done global_id 224488 payload 824 2024-04-26T00:41:36.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.445+0000 7fa92259c640 1 -- 172.21.15.37:0/3998742547 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa924139010 con 0x7fa92402c540 2024-04-26T00:41:36.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.445+0000 7fa92259c640 1 -- 172.21.15.37:0/3998742547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa92402c540 msgr2=0x7fa92402c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:36.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.445+0000 7fa92259c640 1 --2- 172.21.15.37:0/3998742547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa92402c540 0x7fa92402c920 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa91c0048d0 tx=0x7fa91c003af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:36.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.449+0000 7fa91affd640 1 -- 172.21.15.37:0/3998742547 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa91c006d70 con 0x7fa92402c540 2024-04-26T00:41:36.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.449+0000 7fa91affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:36.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.449+0000 7fa91affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:41:36.452 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.449+0000 7fa91affd640 10 monclient: dump: 2024-04-26T00:41:36.452 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:36.452 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:36.452 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.452 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.452 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:36.452 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:36.452 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:36.452 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:36.452 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:36.453 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:36.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.449+0000 7fa91affd640 1 -- 172.21.15.37:0/3998742547 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4888 (secure 0 0 0) 0x7fa91c006d70 con 0x7fa92402c540 2024-04-26T00:41:36.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.449+0000 7fa91affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:41:36.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.449+0000 7fa91affd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:41:36.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.449+0000 7fa91affd640 1 -- 172.21.15.37:0/3998742547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa92402c540 msgr2=0x7fa92402c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:36.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.449+0000 7fa91affd640 1 --2- 172.21.15.37:0/3998742547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa92402c540 0x7fa92402c920 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa91c0048d0 tx=0x7fa91c003af0 comp rx=0 tx=0).stop 2024-04-26T00:41:36.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.449+0000 7fa91affd640 1 -- 172.21.15.37:0/3998742547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa92402c540 msgr2=0x7fa92402c920 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:41:36.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.449+0000 7fa91affd640 1 --2- 172.21.15.37:0/3998742547 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa92402c540 0x7fa92402c920 secure :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa91c0048d0 tx=0x7fa91c003af0 comp rx=0 tx=0).stop 2024-04-26T00:41:36.453 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:41:36.453 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:41:36.453 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:41:36.454 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:41:36.454 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 156, 2024-04-26T00:41:36.454 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 210108, 2024-04-26T00:41:36.454 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:41:36.454 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:41:36.454 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:41:36.454 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:41:36.454 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:36.454 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 6, 2024-04-26T00:41:36.454 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:41:36.454 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:41:36.454 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:41:36.454 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:41:36.455 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:41:36.455 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:41:36.455 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:41:36.455 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:41:36.455 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:41:36.455 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:41:36.455 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:41:36.455 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:41:36.455 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:41:36.455 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:41:36.455 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:36.455 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:41:36.455 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:41:36.455 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:41:36.456 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:41:36.456 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:41:36.456 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:41:36.456 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:41:36.456 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:41:36.456 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:41:36.456 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:41:36.456 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:41:36.456 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:41:36.456 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:41:36.456 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:36.456 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:36.456 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:41:36.456 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:41:36.456 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:41:36.457 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:41:36.457 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:41:36.457 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:41:36.457 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:41:36.457 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:41:36.457 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:41:36.457 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:41:36.457 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:41:36.457 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:41:36.457 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:41:36.457 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:41:36.457 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:41:36.457 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:41:36.458 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:41:36.458 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:41:36.458 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:41:36.458 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:41:36.458 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:41:36.458 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:41:36.458 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:41:36.458 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:41:36.458 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:41:36.458 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:41:36.458 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:41:36.458 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:41:36.458 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:36.459 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:41:36.459 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:36.459 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:41:36.459 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:36.459 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:41:36.459 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:41:36.459 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:41:36.459 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:41:36.459 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:36.459 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:41:36.459 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:41:36.459 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:36.459 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:36.459 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:36.460 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:41:36.460 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:41:36.460 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:36.460 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:36.460 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:36.460 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:36.460 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:41:36.460 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:41:36.460 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:41:36.460 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:41:36.460 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:41:36.460 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:36.460 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:36.460 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:41:36.461 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:41:36.461 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:41:36.461 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:41:36.461 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:36.461 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:41:36.461 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:41:36.461 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:36.461 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:36.461 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:36.461 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:41:36.461 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:41:36.461 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:36.461 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:36.462 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:36.462 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:36.462 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:41:36.462 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:41:36.462 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:41:36.462 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:41:36.462 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:41:36.462 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:36.462 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:36.462 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:41:36.462 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:41:36.462 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:41:36.462 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:41:36.462 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:36.463 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:41:36.463 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:41:36.463 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:36.463 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:36.463 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:36.463 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:41:36.463 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:41:36.463 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:36.463 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:36.463 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:36.463 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:36.463 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:41:36.463 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:41:36.463 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:41:36.464 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:41:36.464 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:41:36.464 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:36.464 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:36.464 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:36.464 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:41:36.464 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:41:36.464 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:36.464 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:41:36.464 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:41:36.464 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:41:36.464 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:36.464 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:36.464 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:41:36.464 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:36.465 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:41:36.465 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:41:36.465 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:41:36.465 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:36.465 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:36.465 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:41:36.465 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:36.465 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:41:36.465 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:41:36.465 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:41:36.465 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:36.465 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:36.465 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:41:36.465 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:36.466 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:41:36.466 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:41:36.466 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:41:36.466 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:36.466 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:36.466 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:36.466 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:41:36.466 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:41:36.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.449+0000 7fa928cf7640 1 -- 172.21.15.37:0/3998742547 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa908035e80 msgr2=0x7fa908038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:36.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.449+0000 7fa928cf7640 1 --2- 172.21.15.37:0/3998742547 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa908035e80 0x7fa908038340 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fa924068940 tx=0x7fa9140a7000 comp rx=0 tx=0).stop 2024-04-26T00:41:36.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.449+0000 7fa928cf7640 10 monclient: shutdown 2024-04-26T00:41:36.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.449+0000 7fa928cf7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:41:36.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.449+0000 7fa928cf7640 1 -- 172.21.15.37:0/3998742547 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa924137a10 msgr2=0x7fa924134da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:36.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.449+0000 7fa928cf7640 1 --2- 172.21.15.37:0/3998742547 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa924137a10 0x7fa924134da0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa90c00e810 tx=0x7fa90c00ecd0 comp rx=0 tx=0).stop 2024-04-26T00:41:36.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.449+0000 7fa922d9d640 1 -- 172.21.15.37:0/3998742547 reap_dead start 2024-04-26T00:41:36.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.449+0000 7fa922d9d640 1 -- 172.21.15.37:0/3998742547 reap_dead start 2024-04-26T00:41:36.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.449+0000 7fa928cf7640 1 -- 172.21.15.37:0/3998742547 shutdown_connections 2024-04-26T00:41:36.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.449+0000 7fa928cf7640 1 -- 172.21.15.37:0/3998742547 >> 172.21.15.37:0/3998742547 conn(0x7fa9240a61b0 msgr2=0x7fa9240a8120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:41:36.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.449+0000 7fa928cf7640 1 -- 172.21.15.37:0/3998742547 shutdown_connections 2024-04-26T00:41:36.468 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.449+0000 7fa928cf7640 1 -- 172.21.15.37:0/3998742547 wait complete. 2024-04-26T00:41:36.469 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:41:36.469 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-04-26T00:41:36.469 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-26T00:41:36.469 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-04-26T00:41:36.469 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-04-26T00:41:36.469 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-26T00:41:36.469 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T00:41:36.492 INFO:tasks.ceph.mon.b:Stopped 2024-04-26T00:41:36.492 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-26T00:41:36.492 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:41:36.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.605+0000 7fa84359f640 10 monclient: get_monmap_and_config 2024-04-26T00:41:36.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.605+0000 7fa84359f640 10 monclient: build_initial_monmap 2024-04-26T00:41:36.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.605+0000 7fa84359f640 10 monclient: monmap: 2024-04-26T00:41:36.609 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:41:36.609 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:36.609 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:41:36.610578+0000 2024-04-26T00:41:36.609 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:41:36.610578+0000 2024-04-26T00:41:36.609 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:41:36.609 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:36.609 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:41:36.609 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:41:36.609 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:41:36.609 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:41:36.609 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:41:36.609 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:36.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.605+0000 7fa84359f640 1 Processor -- start 2024-04-26T00:41:36.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.605+0000 7fa84359f640 1 -- start start 2024-04-26T00:41:36.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.605+0000 7fa84359f640 10 monclient: init 2024-04-26T00:41:36.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa84359f640 10 monclient: _reopen_session rank -1 2024-04-26T00:41:36.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa84359f640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-04-26T00:41:36.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa84359f640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa83c071fc0 0x7fa83c137240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:36.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa84359f640 10 monclient(hunting): picked mon.noname-c con 0x7fa83c071fc0 addr v2:172.21.15.45:3300/0 2024-04-26T00:41:36.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa84359f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa83c137780 0x7fa83c0723a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:36.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa84359f640 10 monclient(hunting): picked mon.noname-b con 0x7fa83c137780 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:41:36.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa84359f640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa83c0728e0 0x7fa83c130770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:36.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa84359f640 10 monclient(hunting): picked mon.noname-a con 0x7fa83c0728e0 addr v2:172.21.15.37:3300/0 2024-04-26T00:41:36.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa84359f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:36.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa84359f640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fa83c073200 con 0x7fa83c0728e0 2024-04-26T00:41:36.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa84359f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:36.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa84359f640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fa83c0a0c20 con 0x7fa83c071fc0 2024-04-26T00:41:36.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa84359f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:36.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa84359f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa83c0a0f00 con 0x7fa83c137780 2024-04-26T00:41:36.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa84359f640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:36.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa84359f640 10 monclient(hunting): authenticate will time out at 2277.916226s 2024-04-26T00:41:36.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa840b3a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa83c137780 0x7fa83c0723a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:36.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa841b3c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa83c0728e0 0x7fa83c130770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:36.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa84133b640 1 -- >> v2:172.21.15.45:3300/0 conn(0x7fa83c071fc0 msgr2=0x7fa83c137240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:41:36.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa840b3a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa83c137780 0x7fa83c0723a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:45374/0 (socket says 172.21.15.37:45374) 2024-04-26T00:41:36.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa841b3c640 10 monclient(hunting): get_auth_request con 0x7fa83c0728e0 auth_method 0 2024-04-26T00:41:36.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa840b3a640 1 -- 172.21.15.37:0/3094146368 learned_addr learned my addr 172.21.15.37:0/3094146368 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:41:36.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa841b3c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:36.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa841b3c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:36.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa841b3c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:36.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa84133b640 1 --2- 172.21.15.37:0/3094146368 >> v2:172.21.15.45:3300/0 conn(0x7fa83c071fc0 0x7fa83c137240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:41:36.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa840b3a640 10 monclient(hunting): get_auth_request con 0x7fa83c137780 auth_method 0 2024-04-26T00:41:36.613 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa840b3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:36.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa840b3a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:36.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa840b3a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:36.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa840b3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:36.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa840b3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:36.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa840b3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:36.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa841b3c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:36.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa841b3c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:36.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa841b3c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:36.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa840b3a640 10 monclient(hunting): handle_auth_done global_id 254261 payload 274 2024-04-26T00:41:36.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa840b3a640 1 -- 172.21.15.37:0/3094146368 >> v2:172.21.15.45:3300/0 conn(0x7fa83c071fc0 msgr2=0x7fa83c137240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:41:36.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa840b3a640 1 --2- 172.21.15.37:0/3094146368 >> v2:172.21.15.45:3300/0 conn(0x7fa83c071fc0 0x7fa83c137240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:36.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa840b3a640 1 -- 172.21.15.37:0/3094146368 >> v2:172.21.15.37:3300/0 conn(0x7fa83c0728e0 msgr2=0x7fa83c130770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:41:36.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa840b3a640 1 --2- 172.21.15.37:0/3094146368 >> v2:172.21.15.37:3300/0 conn(0x7fa83c0728e0 0x7fa83c130770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:36.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa840b3a640 10 monclient: _finish_hunting 0 2024-04-26T00:41:36.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa840b3a640 1 monclient: found mon.noname-b 2024-04-26T00:41:36.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa840b3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:36.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa840b3a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:41:36.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa840b3a640 1 -- 172.21.15.37:0/3094146368 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa83c072d70 con 0x7fa83c137780 2024-04-26T00:41:36.615 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.609+0000 7fa841b3c640 1 --2- 172.21.15.37:0/3094146368 >> v2:172.21.15.37:3300/0 conn(0x7fa83c0728e0 0x7fa83c130770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:41:36.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa840b3a640 1 -- 172.21.15.37:0/3094146368 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa83c137780 msgr2=0x7fa83c0723a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:36.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa840b3a640 1 --2- 172.21.15.37:0/3094146368 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa83c137780 0x7fa83c0723a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa83c0a2ad0 tx=0x7fa83801c610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59972f79c30089c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:36.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa82ffff640 1 -- 172.21.15.37:0/3094146368 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8380a6040 con 0x7fa83c137780 2024-04-26T00:41:36.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa82ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:36.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa82ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:41:36.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa82ffff640 10 monclient: dump: 2024-04-26T00:41:36.618 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:36.618 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:36.618 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.619 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.619 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:36.619 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:36.619 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:36.619 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:36.619 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:36.619 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:36.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa82ffff640 10 monclient: _finish_auth 0 2024-04-26T00:41:36.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa82ffff640 10 monclient: _check_auth_tickets 2024-04-26T00:41:36.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa82ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:41:36.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84359f640 5 monclient: authenticate success, global_id 254261 2024-04-26T00:41:36.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84359f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:41:36.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa82ffff640 1 -- 172.21.15.37:0/3094146368 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8380af070 con 0x7fa83c137780 2024-04-26T00:41:36.619 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa82ffff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:41:36.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84359f640 10 monclient: get_monmap_and_config success 2024-04-26T00:41:36.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84359f640 4 set_mon_vals no callback set 2024-04-26T00:41:36.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa82ffff640 1 -- 172.21.15.37:0/3094146368 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8380aa660 con 0x7fa83c137780 2024-04-26T00:41:36.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa82ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:36.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa82ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:41:36.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa82ffff640 10 monclient: dump: 2024-04-26T00:41:36.620 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:36.620 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:36.620 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.620 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.620 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:36.620 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:36.620 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:36.620 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:36.621 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:36.621 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:36.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84359f640 10 monclient: shutdown 2024-04-26T00:41:36.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84359f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:41:36.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84359f640 1 -- 172.21.15.37:0/3094146368 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa83c137780 msgr2=0x7fa83c0723a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:36.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84359f640 1 --2- 172.21.15.37:0/3094146368 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa83c137780 0x7fa83c0723a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa83c0a2ad0 tx=0x7fa83801c610 comp rx=0 tx=0).stop 2024-04-26T00:41:36.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84359f640 1 -- 172.21.15.37:0/3094146368 shutdown_connections 2024-04-26T00:41:36.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84359f640 1 --2- 172.21.15.37:0/3094146368 >> v2:172.21.15.37:3300/0 conn(0x7fa83c0728e0 0x7fa83c130770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:36.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84359f640 1 --2- 172.21.15.37:0/3094146368 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa83c137780 0x7fa83c0723a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:36.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84359f640 1 --2- 172.21.15.37:0/3094146368 >> v2:172.21.15.45:3300/0 conn(0x7fa83c071fc0 0x7fa83c137240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:36.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84359f640 1 -- 172.21.15.37:0/3094146368 >> 172.21.15.37:0/3094146368 conn(0x7fa83c0a1ad0 msgr2=0x7fa83c0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:41:36.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84359f640 1 -- 172.21.15.37:0/3094146368 shutdown_connections 2024-04-26T00:41:36.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84359f640 1 -- 172.21.15.37:0/3094146368 wait complete. 2024-04-26T00:41:36.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84359f640 10 monclient: build_initial_monmap 2024-04-26T00:41:36.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84359f640 10 monclient: monmap: 2024-04-26T00:41:36.622 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:41:36.622 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:41:36.622 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:41:36.622 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:41:36.622 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:41:36.622 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:41:36.622 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:41:36.622 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:41:36.622 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:41:36.622 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:36.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84359f640 1 Processor -- start 2024-04-26T00:41:36.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84359f640 1 -- start start 2024-04-26T00:41:36.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84359f640 10 monclient: init 2024-04-26T00:41:36.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84359f640 10 monclient: _reopen_session rank -1 2024-04-26T00:41:36.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84359f640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:41:36.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84359f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa83c071fc0 0x7fa83c1cf700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:36.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84359f640 10 monclient(hunting): picked mon.noname-c con 0x7fa83c071fc0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:41:36.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84359f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa83c0728e0 0x7fa83c13d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:36.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84359f640 10 monclient(hunting): picked mon.noname-b con 0x7fa83c0728e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:41:36.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84359f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa83c1cc200 0x7fa83c1cc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:36.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84359f640 10 monclient(hunting): picked mon.noname-a con 0x7fa83c1cc200 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:41:36.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84359f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:36.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84359f640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa83c140f00 con 0x7fa83c1cc200 2024-04-26T00:41:36.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84359f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:36.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84359f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa83c140d80 con 0x7fa83c0728e0 2024-04-26T00:41:36.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84359f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:36.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84359f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa83c141080 con 0x7fa83c071fc0 2024-04-26T00:41:36.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84359f640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:36.623 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84359f640 10 monclient(hunting): authenticate will time out at 2277.923309s 2024-04-26T00:41:36.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84133b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa83c071fc0 0x7fa83c1cf700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:36.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa841b3c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa83c1cc200 0x7fa83c1cc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:36.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa840b3a640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa83c0728e0 msgr2=0x7fa83c13d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:41:36.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84133b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa83c071fc0 0x7fa83c1cf700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:45384/0 (socket says 172.21.15.37:45384) 2024-04-26T00:41:36.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84133b640 1 -- 172.21.15.37:0/3817366754 learned_addr learned my addr 172.21.15.37:0/3817366754 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:41:36.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84133b640 10 monclient(hunting): get_auth_request con 0x7fa83c071fc0 auth_method 0 2024-04-26T00:41:36.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84133b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:36.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84133b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:36.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84133b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:36.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa840b3a640 1 --2- 172.21.15.37:0/3817366754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa83c0728e0 0x7fa83c13d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:41:36.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa841b3c640 10 monclient(hunting): get_auth_request con 0x7fa83c1cc200 auth_method 0 2024-04-26T00:41:36.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa841b3c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:36.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa841b3c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:36.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa841b3c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:36.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84133b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:36.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84133b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:36.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84133b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:36.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa841b3c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:36.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa841b3c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:36.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa841b3c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:36.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84133b640 10 monclient(hunting): handle_auth_done global_id 254264 payload 824 2024-04-26T00:41:36.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84133b640 1 -- 172.21.15.37:0/3817366754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa83c0728e0 msgr2=0x7fa83c13d650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:41:36.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84133b640 1 --2- 172.21.15.37:0/3817366754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa83c0728e0 0x7fa83c13d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:36.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84133b640 1 -- 172.21.15.37:0/3817366754 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa83c1cc200 msgr2=0x7fa83c1cc6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:36.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84133b640 1 --2- 172.21.15.37:0/3817366754 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa83c1cc200 0x7fa83c1cc6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:36.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84133b640 10 monclient: _finish_hunting 0 2024-04-26T00:41:36.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84133b640 1 monclient: found mon.noname-c 2024-04-26T00:41:36.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84133b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:36.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84133b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:41:36.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa84133b640 1 -- 172.21.15.37:0/3817366754 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa83c138ee0 con 0x7fa83c071fc0 2024-04-26T00:41:36.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.613+0000 7fa841b3c640 1 --2- 172.21.15.37:0/3817366754 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa83c1cc200 0x7fa83c1cc6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:41:36.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.617+0000 7fa84133b640 1 -- 172.21.15.37:0/3817366754 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa83c071fc0 msgr2=0x7fa83c1cf700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:36.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.617+0000 7fa84133b640 1 --2- 172.21.15.37:0/3817366754 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa83c071fc0 0x7fa83c1cf700 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa83000b590 tx=0x7fa83000b990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:36.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.617+0000 7fa82dffb640 1 -- 172.21.15.37:0/3817366754 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa830005070 con 0x7fa83c071fc0 2024-04-26T00:41:36.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.617+0000 7fa82dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:36.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.617+0000 7fa82dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:41:36.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.617+0000 7fa82dffb640 10 monclient: dump: 2024-04-26T00:41:36.628 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:36.628 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:36.628 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.628 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.628 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:36.629 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:36.629 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:36.629 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:36.629 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:36.629 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:36.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.617+0000 7fa82dffb640 10 monclient: _finish_auth 0 2024-04-26T00:41:36.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.617+0000 7fa82dffb640 10 monclient: _check_auth_tickets 2024-04-26T00:41:36.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.617+0000 7fa82dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:41:36.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.617+0000 7fa84359f640 5 monclient: authenticate success, global_id 254264 2024-04-26T00:41:36.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.617+0000 7fa82dffb640 1 -- 172.21.15.37:0/3817366754 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa830015070 con 0x7fa83c071fc0 2024-04-26T00:41:36.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.617+0000 7fa82dffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:41:36.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.617+0000 7fa82dffb640 1 -- 172.21.15.37:0/3817366754 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa830005450 con 0x7fa83c071fc0 2024-04-26T00:41:36.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.617+0000 7fa82dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:36.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.617+0000 7fa82dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:41:36.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.617+0000 7fa82dffb640 10 monclient: dump: 2024-04-26T00:41:36.630 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:36.630 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:36.630 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.630 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:36.630 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:36.630 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:36.630 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:36.630 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:36.630 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:36.630 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:36.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.617+0000 7fa82f7fe640 4 set_mon_vals no callback set 2024-04-26T00:41:36.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.617+0000 7fa84359f640 10 monclient: _renew_subs 2024-04-26T00:41:36.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.617+0000 7fa84359f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:41:36.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.617+0000 7fa84359f640 1 -- 172.21.15.37:0/3817366754 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa83c0a0160 con 0x7fa83c071fc0 2024-04-26T00:41:36.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.617+0000 7fa84359f640 10 monclient: _renew_subs 2024-04-26T00:41:36.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.617+0000 7fa84359f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:41:36.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.617+0000 7fa84359f640 1 -- 172.21.15.37:0/3817366754 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa83c1d0330 con 0x7fa83c071fc0 2024-04-26T00:41:36.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.617+0000 7fa84359f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:36.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.617+0000 7fa84359f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:36.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.617+0000 7fa84359f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:41:36.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.617+0000 7fa84359f640 1 -- 172.21.15.37:0/3817366754 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa83c066f40 con 0x7fa83c071fc0 2024-04-26T00:41:36.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.617+0000 7fa82dffb640 1 -- 172.21.15.37:0/3817366754 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa8300055f0 con 0x7fa83c071fc0 2024-04-26T00:41:36.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.621+0000 7fa82dffb640 1 --2- 172.21.15.37:0/3817366754 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa820035e60 0x7fa820038320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:36.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.621+0000 7fa82dffb640 1 -- 172.21.15.37:0/3817366754 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(863..863 src has 705..863) ==== 3975+0+0 (secure 0 0 0) 0x7fa830048950 con 0x7fa83c071fc0 2024-04-26T00:41:36.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.621+0000 7fa840b3a640 1 --2- 172.21.15.37:0/3817366754 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa820035e60 0x7fa820038320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:36.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.621+0000 7fa840b3a640 10 monclient: get_auth_request con 0x7fa820035e60 auth_method 0 2024-04-26T00:41:36.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.621+0000 7fa840b3a640 1 --2- 172.21.15.37:0/3817366754 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa820035e60 0x7fa820038320 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fa83c06d050 tx=0x7fa8380a7000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:36.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.625+0000 7fa82dffb640 1 -- 172.21.15.37:0/3817366754 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa83000e340 con 0x7fa83c071fc0 2024-04-26T00:41:36.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.625+0000 7fa82dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:36.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.625+0000 7fa82dffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:41:36.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.845+0000 7fa84359f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:41:36.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.845+0000 7fa84359f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:41:36.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.845+0000 7fa84359f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:41:36.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.845+0000 7fa84359f640 1 -- 172.21.15.37:0/3817366754 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa83c09e260 con 0x7fa83c071fc0 2024-04-26T00:41:36.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.849+0000 7fa82dffb640 1 -- 172.21.15.37:0/3817366754 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fa830010900 con 0x7fa83c071fc0 2024-04-26T00:41:36.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.849+0000 7fa82dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:41:36.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.849+0000 7fa82dffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:41:36.850 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":156,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:41:36.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.849+0000 7fa84359f640 1 -- 172.21.15.37:0/3817366754 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa820035e60 msgr2=0x7fa820038320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:36.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.849+0000 7fa84359f640 1 --2- 172.21.15.37:0/3817366754 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa820035e60 0x7fa820038320 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fa83c06d050 tx=0x7fa8380a7000 comp rx=0 tx=0).stop 2024-04-26T00:41:36.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.849+0000 7fa84359f640 10 monclient: shutdown 2024-04-26T00:41:36.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.849+0000 7fa84359f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:41:36.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.849+0000 7fa84359f640 1 -- 172.21.15.37:0/3817366754 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa83c071fc0 msgr2=0x7fa83c1cf700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:36.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.849+0000 7fa84359f640 1 --2- 172.21.15.37:0/3817366754 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa83c071fc0 0x7fa83c1cf700 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa83000b590 tx=0x7fa83000b990 comp rx=0 tx=0).stop 2024-04-26T00:41:36.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.849+0000 7fa84359f640 1 -- 172.21.15.37:0/3817366754 shutdown_connections 2024-04-26T00:41:36.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.849+0000 7fa84359f640 1 --2- 172.21.15.37:0/3817366754 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa820035e60 0x7fa820038320 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:36.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.849+0000 7fa84359f640 1 --2- 172.21.15.37:0/3817366754 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa83c1cc200 0x7fa83c1cc6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:36.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.849+0000 7fa84359f640 1 --2- 172.21.15.37:0/3817366754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa83c0728e0 0x7fa83c13d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:36.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.849+0000 7fa84359f640 1 --2- 172.21.15.37:0/3817366754 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa83c071fc0 0x7fa83c1cf700 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:36.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.849+0000 7fa84359f640 1 -- 172.21.15.37:0/3817366754 >> 172.21.15.37:0/3817366754 conn(0x7fa83c0a1ad0 msgr2=0x7fa83c133da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:41:36.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.849+0000 7fa84359f640 1 -- 172.21.15.37:0/3817366754 shutdown_connections 2024-04-26T00:41:36.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:36.849+0000 7fa84359f640 1 -- 172.21.15.37:0/3817366754 wait complete. 2024-04-26T00:41:39.868 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:41:39.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:39.997+0000 7fe2eb70d640 10 monclient: get_monmap_and_config 2024-04-26T00:41:39.999 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:39.997+0000 7fe2eb70d640 10 monclient: build_initial_monmap 2024-04-26T00:41:40.000 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:39.997+0000 7fe2eb70d640 10 monclient: monmap: 2024-04-26T00:41:40.000 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:41:40.000 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:40.000 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:41:40.001727+0000 2024-04-26T00:41:40.000 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:41:40.001727+0000 2024-04-26T00:41:40.000 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:41:40.000 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:40.000 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:41:40.000 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:41:40.000 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:41:40.000 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:41:40.000 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:41:40.000 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:40.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:39.997+0000 7fe2eb70d640 1 Processor -- start 2024-04-26T00:41:40.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:39.997+0000 7fe2eb70d640 1 -- start start 2024-04-26T00:41:40.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:39.997+0000 7fe2eb70d640 10 monclient: init 2024-04-26T00:41:40.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:39.997+0000 7fe2eb70d640 10 monclient: _reopen_session rank -1 2024-04-26T00:41:40.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:39.997+0000 7fe2eb70d640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-04-26T00:41:40.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:39.997+0000 7fe2eb70d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe2e4071f00 addr v1:172.21.15.37:6789/0 2024-04-26T00:41:40.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:39.997+0000 7fe2eb70d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2e40713a0 0x7fe2e4138580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:40.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:39.997+0000 7fe2eb70d640 10 monclient(hunting): picked mon.noname-b con 0x7fe2e40713a0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:41:40.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:39.997+0000 7fe2eb70d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe2e40717c0 addr v1:172.21.15.45:6789/0 2024-04-26T00:41:40.003 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:39.997+0000 7fe2eb70d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2e406ee70 con 0x7fe2e4071f00 2024-04-26T00:41:40.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:39.997+0000 7fe2eb70d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2e406ea30 con 0x7fe2e40717c0 2024-04-26T00:41:40.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:39.997+0000 7fe2eb70d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:40.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:39.997+0000 7fe2eb70d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe2e40729f0 con 0x7fe2e40713a0 2024-04-26T00:41:40.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:39.997+0000 7fe2eb70d640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:40.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:39.997+0000 7fe2eb70d640 10 monclient(hunting): authenticate will time out at 2281.307773s 2024-04-26T00:41:40.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:39.997+0000 7fe2e9caa640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7fe2e40717c0 legacy=0x7fe2e413aad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:41:40.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:39.997+0000 7fe2e94a9640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fe2e4071f00 0x7fe2e40722e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:58222/0 (socket says 172.21.15.37:58222) 2024-04-26T00:41:40.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:39.997+0000 7fe2e94a9640 1 -- 172.21.15.37:0/2623218945 learned_addr learned my addr 172.21.15.37:0/2623218945 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:41:40.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:39.997+0000 7fe2e8ca8640 1 --2- 172.21.15.37:0/2623218945 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2e40713a0 0x7fe2e4138580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:40.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.001+0000 7fe2e8ca8640 10 monclient(hunting): get_auth_request con 0x7fe2e40713a0 auth_method 0 2024-04-26T00:41:40.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.001+0000 7fe2e8ca8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:40.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.001+0000 7fe2e8ca8640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:40.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.001+0000 7fe2e8ca8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:40.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.001+0000 7fe2e8ca8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:40.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.001+0000 7fe2e8ca8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:40.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.001+0000 7fe2e8ca8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:40.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.001+0000 7fe2e8ca8640 10 monclient(hunting): handle_auth_done global_id 254267 payload 274 2024-04-26T00:41:40.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.001+0000 7fe2e8ca8640 1 -- 172.21.15.37:0/2623218945 >> v1:172.21.15.45:6789/0 conn(0x7fe2e40717c0 legacy=0x7fe2e413aad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:41:40.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.001+0000 7fe2e8ca8640 1 -- 172.21.15.37:0/2623218945 >> v1:172.21.15.37:6789/0 conn(0x7fe2e4071f00 legacy=0x7fe2e40722e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:41:40.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.001+0000 7fe2e8ca8640 10 monclient: _finish_hunting 0 2024-04-26T00:41:40.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.001+0000 7fe2e8ca8640 1 monclient: found mon.noname-b 2024-04-26T00:41:40.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.001+0000 7fe2e8ca8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:40.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.001+0000 7fe2e8ca8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:41:40.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.001+0000 7fe2e8ca8640 1 -- 172.21.15.37:0/2623218945 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2e4070120 con 0x7fe2e40713a0 2024-04-26T00:41:40.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.005+0000 7fe2e8ca8640 1 -- 172.21.15.37:0/2623218945 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2e40713a0 msgr2=0x7fe2e4138580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:40.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.005+0000 7fe2e8ca8640 1 --2- 172.21.15.37:0/2623218945 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2e40713a0 0x7fe2e4138580 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe2d40097f0 tx=0x7fe2d401c2f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6034114673e57b28 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:40.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.005+0000 7fe2dbfff640 1 -- 172.21.15.37:0/2623218945 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d40b0070 con 0x7fe2e40713a0 2024-04-26T00:41:40.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.005+0000 7fe2dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:40.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.005+0000 7fe2dbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:41:40.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.005+0000 7fe2dbfff640 10 monclient: dump: 2024-04-26T00:41:40.009 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:40.009 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:40.009 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:40.009 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:40.009 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:40.009 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:40.009 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:40.009 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:40.009 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:40.009 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:40.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.005+0000 7fe2dbfff640 10 monclient: _finish_auth 0 2024-04-26T00:41:40.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.005+0000 7fe2dbfff640 10 monclient: _check_auth_tickets 2024-04-26T00:41:40.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.005+0000 7fe2dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:41:40.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.005+0000 7fe2dbfff640 1 -- 172.21.15.37:0/2623218945 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2d40ab600 con 0x7fe2e40713a0 2024-04-26T00:41:40.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.005+0000 7fe2dbfff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:41:40.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.005+0000 7fe2dbfff640 1 -- 172.21.15.37:0/2623218945 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e40a31e0 con 0x7fe2e40713a0 2024-04-26T00:41:40.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.005+0000 7fe2eb70d640 5 monclient: authenticate success, global_id 254267 2024-04-26T00:41:40.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.005+0000 7fe2eb70d640 10 monclient: get_monmap_and_config success 2024-04-26T00:41:40.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.005+0000 7fe2eb70d640 4 set_mon_vals no callback set 2024-04-26T00:41:40.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.005+0000 7fe2dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:40.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.005+0000 7fe2dbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:41:40.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.005+0000 7fe2dbfff640 10 monclient: dump: 2024-04-26T00:41:40.010 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:40.011 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:40.011 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:40.011 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:40.011 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:40.011 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:40.011 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:40.011 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:40.011 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:40.011 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:40.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.005+0000 7fe2eb70d640 10 monclient: shutdown 2024-04-26T00:41:40.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.005+0000 7fe2eb70d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:41:40.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.005+0000 7fe2eb70d640 1 -- 172.21.15.37:0/2623218945 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2e40713a0 msgr2=0x7fe2e4138580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:40.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.005+0000 7fe2eb70d640 1 --2- 172.21.15.37:0/2623218945 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2e40713a0 0x7fe2e4138580 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe2d40097f0 tx=0x7fe2d401c2f0 comp rx=0 tx=0).stop 2024-04-26T00:41:40.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.005+0000 7fe2eb70d640 1 -- 172.21.15.37:0/2623218945 shutdown_connections 2024-04-26T00:41:40.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.005+0000 7fe2eb70d640 1 --2- 172.21.15.37:0/2623218945 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2e40713a0 0x7fe2e4138580 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:40.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.005+0000 7fe2eb70d640 1 -- 172.21.15.37:0/2623218945 >> 172.21.15.37:0/2623218945 conn(0x7fe2e40a6250 msgr2=0x7fe2e40a2610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:41:40.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2eb70d640 1 -- 172.21.15.37:0/2623218945 shutdown_connections 2024-04-26T00:41:40.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2eb70d640 1 -- 172.21.15.37:0/2623218945 wait complete. 2024-04-26T00:41:40.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2eb70d640 10 monclient: build_initial_monmap 2024-04-26T00:41:40.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2eb70d640 10 monclient: monmap: 2024-04-26T00:41:40.015 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:41:40.015 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:41:40.015 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:41:40.015 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:41:40.015 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:41:40.015 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:41:40.015 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:41:40.015 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:41:40.016 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:41:40.016 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:40.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2eb70d640 1 Processor -- start 2024-04-26T00:41:40.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2eb70d640 1 -- start start 2024-04-26T00:41:40.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2eb70d640 10 monclient: init 2024-04-26T00:41:40.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2eb70d640 10 monclient: _reopen_session rank -1 2024-04-26T00:41:40.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2eb70d640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:41:40.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2eb70d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2e40713a0 0x7fe2e41cee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:40.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2eb70d640 10 monclient(hunting): picked mon.noname-a con 0x7fe2e40713a0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:41:40.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2eb70d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2e40717c0 0x7fe2e413a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:40.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2eb70d640 10 monclient(hunting): picked mon.noname-c con 0x7fe2e40717c0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:41:40.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2eb70d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2e4071f00 0x7fe2e413a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:40.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2eb70d640 10 monclient(hunting): picked mon.noname-b con 0x7fe2e4071f00 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:41:40.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2eb70d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:40.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2eb70d640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e4075a20 con 0x7fe2e40713a0 2024-04-26T00:41:40.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2eb70d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:40.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2eb70d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e40758a0 con 0x7fe2e4071f00 2024-04-26T00:41:40.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2eb70d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:40.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2eb70d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe2e4075ba0 con 0x7fe2e40717c0 2024-04-26T00:41:40.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2eb70d640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:40.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2eb70d640 10 monclient(hunting): authenticate will time out at 2281.317445s 2024-04-26T00:41:40.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2e94a9640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2e40713a0 0x7fe2e41cee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:40.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2e8ca8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2e40717c0 0x7fe2e413a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:40.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2e94a9640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2e40713a0 0x7fe2e41cee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:34062/0 (socket says 172.21.15.37:34062) 2024-04-26T00:41:40.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2e94a9640 1 -- 172.21.15.37:0/1842592905 learned_addr learned my addr 172.21.15.37:0/1842592905 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:41:40.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2e94a9640 10 monclient(hunting): get_auth_request con 0x7fe2e40713a0 auth_method 0 2024-04-26T00:41:40.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2e94a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:40.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2e94a9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:40.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2e94a9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:40.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2e8ca8640 10 monclient(hunting): get_auth_request con 0x7fe2e40717c0 auth_method 0 2024-04-26T00:41:40.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2e8ca8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:40.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2e8ca8640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:40.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2e8ca8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:40.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2e9caa640 1 -- 172.21.15.37:0/1842592905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2e4071f00 msgr2=0x7fe2e413a960 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:41:40.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2e9caa640 1 --2- 172.21.15.37:0/1842592905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2e4071f00 0x7fe2e413a960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:41:40.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2e8ca8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:40.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2e8ca8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:40.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2e8ca8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:40.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2e94a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:40.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2e94a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:40.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2e94a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:40.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2e94a9640 10 monclient(hunting): handle_auth_done global_id 264147 payload 824 2024-04-26T00:41:40.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2e94a9640 1 -- 172.21.15.37:0/1842592905 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2e40717c0 msgr2=0x7fe2e413a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:40.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2e94a9640 1 --2- 172.21.15.37:0/1842592905 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2e40717c0 0x7fe2e413a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:40.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2e94a9640 1 -- 172.21.15.37:0/1842592905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2e4071f00 msgr2=0x7fe2e413a960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:41:40.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2e94a9640 1 --2- 172.21.15.37:0/1842592905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2e4071f00 0x7fe2e413a960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:40.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2e94a9640 10 monclient: _finish_hunting 0 2024-04-26T00:41:40.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2e94a9640 1 monclient: found mon.noname-a 2024-04-26T00:41:40.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2e94a9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:40.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2e94a9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:41:40.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2e94a9640 1 -- 172.21.15.37:0/1842592905 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2e413b880 con 0x7fe2e40713a0 2024-04-26T00:41:40.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.009+0000 7fe2e8ca8640 1 --2- 172.21.15.37:0/1842592905 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2e40717c0 0x7fe2e413a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:41:40.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.013+0000 7fe2e94a9640 1 -- 172.21.15.37:0/1842592905 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2e40713a0 msgr2=0x7fe2e41cee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:40.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.013+0000 7fe2e94a9640 1 --2- 172.21.15.37:0/1842592905 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2e40713a0 0x7fe2e41cee10 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe2e000c4d0 tx=0x7fe2e000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:40.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.013+0000 7fe2d9ffb640 1 -- 172.21.15.37:0/1842592905 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e0007070 con 0x7fe2e40713a0 2024-04-26T00:41:40.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.013+0000 7fe2d9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:40.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.013+0000 7fe2d9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:41:40.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.013+0000 7fe2d9ffb640 10 monclient: dump: 2024-04-26T00:41:40.020 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:40.020 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:40.020 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:40.020 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:40.020 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:40.020 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:40.020 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:40.020 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:40.020 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:40.020 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:40.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.013+0000 7fe2d9ffb640 10 monclient: _finish_auth 0 2024-04-26T00:41:40.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.013+0000 7fe2d9ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:41:40.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.013+0000 7fe2d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:41:40.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.013+0000 7fe2d9ffb640 1 -- 172.21.15.37:0/1842592905 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2e0007230 con 0x7fe2e40713a0 2024-04-26T00:41:40.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.013+0000 7fe2eb70d640 5 monclient: authenticate success, global_id 264147 2024-04-26T00:41:40.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.013+0000 7fe2eb70d640 10 monclient: _renew_subs 2024-04-26T00:41:40.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.013+0000 7fe2eb70d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:41:40.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.013+0000 7fe2eb70d640 1 -- 172.21.15.37:0/1842592905 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2e409c660 con 0x7fe2e40713a0 2024-04-26T00:41:40.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.013+0000 7fe2d9ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:41:40.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.013+0000 7fe2d9ffb640 1 -- 172.21.15.37:0/1842592905 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e0007620 con 0x7fe2e40713a0 2024-04-26T00:41:40.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.013+0000 7fe2d9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:40.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.013+0000 7fe2d9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:41:40.021 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.013+0000 7fe2d9ffb640 10 monclient: dump: 2024-04-26T00:41:40.021 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:40.021 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:40.021 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:40.021 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:40.021 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:40.021 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:40.022 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:40.022 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:40.022 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:40.022 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:40.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.013+0000 7fe2eb70d640 10 monclient: _renew_subs 2024-04-26T00:41:40.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.013+0000 7fe2db7fe640 4 set_mon_vals no callback set 2024-04-26T00:41:40.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.013+0000 7fe2eb70d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:41:40.022 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.013+0000 7fe2eb70d640 1 -- 172.21.15.37:0/1842592905 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2e41260a0 con 0x7fe2e40713a0 2024-04-26T00:41:40.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.013+0000 7fe2eb70d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:40.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.013+0000 7fe2eb70d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:40.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.013+0000 7fe2eb70d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:41:40.023 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.013+0000 7fe2eb70d640 1 -- 172.21.15.37:0/1842592905 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2e4066f40 con 0x7fe2e40713a0 2024-04-26T00:41:40.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.013+0000 7fe2d9ffb640 1 -- 172.21.15.37:0/1842592905 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe2e001b400 con 0x7fe2e40713a0 2024-04-26T00:41:40.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.013+0000 7fe2d9ffb640 1 --2- 172.21.15.37:0/1842592905 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe2c8035e30 0x7fe2c80382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:40.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.013+0000 7fe2d9ffb640 1 -- 172.21.15.37:0/1842592905 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(863..863 src has 705..863) ==== 3975+0+0 (secure 0 0 0) 0x7fe2e0048890 con 0x7fe2e40713a0 2024-04-26T00:41:40.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.013+0000 7fe2e8ca8640 1 --2- 172.21.15.37:0/1842592905 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe2c8035e30 0x7fe2c80382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:40.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.013+0000 7fe2e8ca8640 10 monclient: get_auth_request con 0x7fe2c8035e30 auth_method 0 2024-04-26T00:41:40.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.017+0000 7fe2e8ca8640 1 --2- 172.21.15.37:0/1842592905 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe2c8035e30 0x7fe2c80382f0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fe2e406ee40 tx=0x7fe2d40a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:40.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.017+0000 7fe2d9ffb640 1 -- 172.21.15.37:0/1842592905 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe2e001a760 con 0x7fe2e40713a0 2024-04-26T00:41:40.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.017+0000 7fe2d9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:40.024 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.017+0000 7fe2d9ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:41:40.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.241+0000 7fe2eb70d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:41:40.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.241+0000 7fe2eb70d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:41:40.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.241+0000 7fe2eb70d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:41:40.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.241+0000 7fe2eb70d640 1 -- 172.21.15.37:0/1842592905 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe2e406c1e0 con 0x7fe2e40713a0 2024-04-26T00:41:40.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.241+0000 7fe2d9ffb640 1 -- 172.21.15.37:0/1842592905 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fe2e004ace0 con 0x7fe2e40713a0 2024-04-26T00:41:40.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.241+0000 7fe2d9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:41:40.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.241+0000 7fe2d9ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:41:40.243 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":156,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:41:40.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.241+0000 7fe2eb70d640 1 -- 172.21.15.37:0/1842592905 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe2c8035e30 msgr2=0x7fe2c80382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:40.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.241+0000 7fe2eb70d640 1 --2- 172.21.15.37:0/1842592905 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe2c8035e30 0x7fe2c80382f0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fe2e406ee40 tx=0x7fe2d40a6000 comp rx=0 tx=0).stop 2024-04-26T00:41:40.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.241+0000 7fe2eb70d640 10 monclient: shutdown 2024-04-26T00:41:40.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.241+0000 7fe2eb70d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:41:40.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.241+0000 7fe2eb70d640 1 -- 172.21.15.37:0/1842592905 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2e40713a0 msgr2=0x7fe2e41cee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:40.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.241+0000 7fe2eb70d640 1 --2- 172.21.15.37:0/1842592905 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2e40713a0 0x7fe2e41cee10 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe2e000c4d0 tx=0x7fe2e000c990 comp rx=0 tx=0).stop 2024-04-26T00:41:40.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.241+0000 7fe2eb70d640 1 -- 172.21.15.37:0/1842592905 shutdown_connections 2024-04-26T00:41:40.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.241+0000 7fe2eb70d640 1 --2- 172.21.15.37:0/1842592905 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe2c8035e30 0x7fe2c80382f0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:40.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.241+0000 7fe2eb70d640 1 --2- 172.21.15.37:0/1842592905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2e4071f00 0x7fe2e413a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:40.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.241+0000 7fe2eb70d640 1 --2- 172.21.15.37:0/1842592905 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2e40717c0 0x7fe2e413a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:40.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.241+0000 7fe2eb70d640 1 --2- 172.21.15.37:0/1842592905 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2e40713a0 0x7fe2e41cee10 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:40.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.241+0000 7fe2eb70d640 1 -- 172.21.15.37:0/1842592905 >> 172.21.15.37:0/1842592905 conn(0x7fe2e40a6250 msgr2=0x7fe2e413ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:41:40.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.241+0000 7fe2eb70d640 1 -- 172.21.15.37:0/1842592905 shutdown_connections 2024-04-26T00:41:40.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:40.241+0000 7fe2eb70d640 1 -- 172.21.15.37:0/1842592905 wait complete. 2024-04-26T00:41:43.261 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:41:43.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f909ad640 10 monclient: get_monmap_and_config 2024-04-26T00:41:43.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f909ad640 10 monclient: build_initial_monmap 2024-04-26T00:41:43.395 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f909ad640 10 monclient: monmap: 2024-04-26T00:41:43.395 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:41:43.395 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:43.395 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:41:43.397173+0000 2024-04-26T00:41:43.395 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:41:43.397173+0000 2024-04-26T00:41:43.395 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:41:43.395 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:43.395 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:41:43.396 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:41:43.396 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:41:43.396 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:41:43.396 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:41:43.396 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:43.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f909ad640 1 Processor -- start 2024-04-26T00:41:43.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f909ad640 1 -- start start 2024-04-26T00:41:43.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f909ad640 10 monclient: init 2024-04-26T00:41:43.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f909ad640 10 monclient: _reopen_session rank -1 2024-04-26T00:41:43.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f909ad640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-04-26T00:41:43.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f909ad640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f4f8c0a0f50 0x7f4f8c0a1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:43.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f909ad640 10 monclient(hunting): picked mon.noname-a con 0x7f4f8c0a0f50 addr v2:172.21.15.37:3300/0 2024-04-26T00:41:43.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f909ad640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f8c09f850 0x7f4f8c06c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:43.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f909ad640 10 monclient(hunting): picked mon.noname-b con 0x7f4f8c09f850 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:41:43.398 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f909ad640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4f8c06c820 0x7f4f8c06cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:43.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f909ad640 10 monclient(hunting): picked mon.noname-c con 0x7f4f8c06c820 addr v2:172.21.15.45:3300/0 2024-04-26T00:41:43.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f909ad640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:43.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f909ad640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f4f8c0a36f0 con 0x7f4f8c0a0f50 2024-04-26T00:41:43.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f909ad640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:43.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f909ad640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f4f8c09df60 con 0x7f4f8c06c820 2024-04-26T00:41:43.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f909ad640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:43.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f909ad640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4f8c09ee60 con 0x7f4f8c09f850 2024-04-26T00:41:43.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f909ad640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:43.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f909ad640 10 monclient(hunting): authenticate will time out at 2284.702837s 2024-04-26T00:41:43.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f89d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f8c09f850 0x7f4f8c06c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:43.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f8a59c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f4f8c0a0f50 0x7f4f8c0a1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:43.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f8ad9d640 1 -- >> v2:172.21.15.45:3300/0 conn(0x7f4f8c06c820 msgr2=0x7f4f8c06cc50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:41:43.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f8ad9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4f8c06c820 0x7f4f8c06cc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:41:43.399 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f89d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f8c09f850 0x7f4f8c06c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:37408/0 (socket says 172.21.15.37:37408) 2024-04-26T00:41:43.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f8a59c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f4f8c0a0f50 0x7f4f8c0a1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:34072/0 (socket says 172.21.15.37:34072) 2024-04-26T00:41:43.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f89d9b640 1 -- 172.21.15.37:0/2252963992 learned_addr learned my addr 172.21.15.37:0/2252963992 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:41:43.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f89d9b640 10 monclient(hunting): get_auth_request con 0x7f4f8c09f850 auth_method 0 2024-04-26T00:41:43.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f89d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:43.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f89d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:43.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f89d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:43.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f8a59c640 10 monclient(hunting): get_auth_request con 0x7f4f8c0a0f50 auth_method 0 2024-04-26T00:41:43.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f8a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:43.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f8a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:43.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f8a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:43.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f8a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:43.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f8a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:43.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f8a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:43.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f89d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:43.400 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f89d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:43.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f89d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:43.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f8a59c640 10 monclient(hunting): handle_auth_done global_id 264150 payload 274 2024-04-26T00:41:43.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f8a59c640 1 -- 172.21.15.37:0/2252963992 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f8c09f850 msgr2=0x7f4f8c06c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:41:43.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f8a59c640 1 --2- 172.21.15.37:0/2252963992 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f8c09f850 0x7f4f8c06c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:43.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f8a59c640 1 -- 172.21.15.37:0/2252963992 >> v2:172.21.15.45:3300/0 conn(0x7f4f8c06c820 msgr2=0x7f4f8c06cc50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:41:43.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f8a59c640 1 --2- 172.21.15.37:0/2252963992 >> v2:172.21.15.45:3300/0 conn(0x7f4f8c06c820 0x7f4f8c06cc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:43.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f8a59c640 10 monclient: _finish_hunting 0 2024-04-26T00:41:43.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f8a59c640 1 monclient: found mon.noname-a 2024-04-26T00:41:43.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f8a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:43.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f8a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:41:43.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f8a59c640 1 -- 172.21.15.37:0/2252963992 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f8c09fc30 con 0x7f4f8c0a0f50 2024-04-26T00:41:43.401 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.393+0000 7f4f89d9b640 1 --2- 172.21.15.37:0/2252963992 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f8c09f850 0x7f4f8c06c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:41:43.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f8a59c640 1 -- 172.21.15.37:0/2252963992 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f8c0a0f50 msgr2=0x7f4f8c0a1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:43.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f8a59c640 1 --2- 172.21.15.37:0/2252963992 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f8c0a0f50 0x7f4f8c0a1330 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c0a5120 tx=0x7f4f7c01c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17f93809f5b7ab22 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:43.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f8959a640 1 -- 172.21.15.37:0/2252963992 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f7c0b2070 con 0x7f4f8c0a0f50 2024-04-26T00:41:43.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f8959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:43.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f8959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:41:43.402 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f8959a640 10 monclient: dump: 2024-04-26T00:41:43.402 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:43.402 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:43.403 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:43.403 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:43.403 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:43.403 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:43.403 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:43.403 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:43.403 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:43.403 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:43.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f8959a640 10 monclient: _finish_auth 0 2024-04-26T00:41:43.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f8959a640 10 monclient: _check_auth_tickets 2024-04-26T00:41:43.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f8959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:41:43.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f8959a640 1 -- 172.21.15.37:0/2252963992 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f7c0a58b0 con 0x7f4f8c0a0f50 2024-04-26T00:41:43.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f8959a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:41:43.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f8959a640 1 -- 172.21.15.37:0/2252963992 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f7c0a5c80 con 0x7f4f8c0a0f50 2024-04-26T00:41:43.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f8959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:43.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f8959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:41:43.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f8959a640 10 monclient: dump: 2024-04-26T00:41:43.404 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:43.404 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:43.404 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:43.404 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:43.404 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:43.404 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:43.404 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:43.404 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:43.404 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:43.404 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:43.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f909ad640 5 monclient: authenticate success, global_id 264150 2024-04-26T00:41:43.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f909ad640 10 monclient: get_monmap_and_config success 2024-04-26T00:41:43.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f909ad640 4 set_mon_vals no callback set 2024-04-26T00:41:43.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f909ad640 10 monclient: shutdown 2024-04-26T00:41:43.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f909ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:41:43.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f909ad640 1 -- 172.21.15.37:0/2252963992 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f8c0a0f50 msgr2=0x7f4f8c0a1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:43.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f909ad640 1 --2- 172.21.15.37:0/2252963992 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f8c0a0f50 0x7f4f8c0a1330 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c0a5120 tx=0x7f4f7c01c810 comp rx=0 tx=0).stop 2024-04-26T00:41:43.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f909ad640 1 -- 172.21.15.37:0/2252963992 shutdown_connections 2024-04-26T00:41:43.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f909ad640 1 --2- 172.21.15.37:0/2252963992 >> v2:172.21.15.45:3300/0 conn(0x7f4f8c06c820 0x7f4f8c06cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:43.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f909ad640 1 --2- 172.21.15.37:0/2252963992 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f8c09f850 0x7f4f8c06c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:43.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f909ad640 1 --2- 172.21.15.37:0/2252963992 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f8c0a0f50 0x7f4f8c0a1330 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:43.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f909ad640 1 -- 172.21.15.37:0/2252963992 >> 172.21.15.37:0/2252963992 conn(0x7f4f8c0a4170 msgr2=0x7f4f8c0a4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:41:43.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f909ad640 1 -- 172.21.15.37:0/2252963992 shutdown_connections 2024-04-26T00:41:43.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f909ad640 1 -- 172.21.15.37:0/2252963992 wait complete. 2024-04-26T00:41:43.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f909ad640 10 monclient: build_initial_monmap 2024-04-26T00:41:43.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f909ad640 10 monclient: monmap: 2024-04-26T00:41:43.409 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:41:43.409 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:41:43.410 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:41:43.410 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:41:43.410 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:41:43.410 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:41:43.410 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:41:43.410 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:41:43.410 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:41:43.410 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:43.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f909ad640 1 Processor -- start 2024-04-26T00:41:43.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f909ad640 1 -- start start 2024-04-26T00:41:43.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f909ad640 10 monclient: init 2024-04-26T00:41:43.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f909ad640 10 monclient: _reopen_session rank -1 2024-04-26T00:41:43.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f909ad640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:41:43.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f909ad640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f8c06c820 0x7f4f8c1ced30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:43.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f909ad640 10 monclient(hunting): picked mon.noname-b con 0x7f4f8c06c820 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:41:43.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f909ad640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f8c09f850 0x7f4f8c13cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:43.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f909ad640 10 monclient(hunting): picked mon.noname-c con 0x7f4f8c09f850 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:41:43.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f909ad640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f8c0a0f50 0x7f4f8c124960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:43.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f909ad640 10 monclient(hunting): picked mon.noname-a con 0x7f4f8c0a0f50 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:41:43.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f909ad640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:43.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f909ad640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f4f8c140a10 con 0x7f4f8c0a0f50 2024-04-26T00:41:43.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f909ad640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:43.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f909ad640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4f8c140890 con 0x7f4f8c06c820 2024-04-26T00:41:43.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f909ad640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:43.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f909ad640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4f8c140b90 con 0x7f4f8c09f850 2024-04-26T00:41:43.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f909ad640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:43.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f909ad640 10 monclient(hunting): authenticate will time out at 2284.711003s 2024-04-26T00:41:43.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f8a59c640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f8c06c820 msgr2=0x7f4f8c1ced30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:41:43.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f8a59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f8c06c820 0x7f4f8c1ced30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:41:43.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f89d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f8c09f850 0x7f4f8c13cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:43.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f89d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f8c09f850 0x7f4f8c13cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:37422/0 (socket says 172.21.15.37:37422) 2024-04-26T00:41:43.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f8ad9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f8c0a0f50 0x7f4f8c124960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:43.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f89d9b640 1 -- 172.21.15.37:0/3773793894 learned_addr learned my addr 172.21.15.37:0/3773793894 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:41:43.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f89d9b640 10 monclient(hunting): get_auth_request con 0x7f4f8c09f850 auth_method 0 2024-04-26T00:41:43.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f89d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:43.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f89d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:43.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f89d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:43.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f8ad9d640 10 monclient(hunting): get_auth_request con 0x7f4f8c0a0f50 auth_method 0 2024-04-26T00:41:43.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f8ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:43.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f8ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:43.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f8ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:43.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f89d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:43.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f89d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:43.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f89d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:43.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f8ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:43.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f8ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:43.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f8ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:43.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f89d9b640 10 monclient(hunting): handle_auth_done global_id 254276 payload 824 2024-04-26T00:41:43.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f89d9b640 1 -- 172.21.15.37:0/3773793894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f8c06c820 msgr2=0x7f4f8c1ced30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:41:43.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f89d9b640 1 --2- 172.21.15.37:0/3773793894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f8c06c820 0x7f4f8c1ced30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:43.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f89d9b640 1 -- 172.21.15.37:0/3773793894 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f8c0a0f50 msgr2=0x7f4f8c124960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:43.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f89d9b640 1 --2- 172.21.15.37:0/3773793894 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f8c0a0f50 0x7f4f8c124960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:43.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f89d9b640 10 monclient: _finish_hunting 0 2024-04-26T00:41:43.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f89d9b640 1 monclient: found mon.noname-c 2024-04-26T00:41:43.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f89d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:43.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f89d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:41:43.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f89d9b640 1 -- 172.21.15.37:0/3773793894 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f8c1288a0 con 0x7f4f8c09f850 2024-04-26T00:41:43.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.401+0000 7f4f8ad9d640 1 --2- 172.21.15.37:0/3773793894 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f8c0a0f50 0x7f4f8c124960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:41:43.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.405+0000 7f4f89d9b640 1 -- 172.21.15.37:0/3773793894 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f8c09f850 msgr2=0x7f4f8c13cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:43.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.405+0000 7f4f89d9b640 1 --2- 172.21.15.37:0/3773793894 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f8c09f850 0x7f4f8c13cf00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4f8000e9d0 tx=0x7f4f8000ee90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:43.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.405+0000 7f4f7affd640 1 -- 172.21.15.37:0/3773793894 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f80017070 con 0x7f4f8c09f850 2024-04-26T00:41:43.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.405+0000 7f4f7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:43.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.405+0000 7f4f7affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:41:43.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.405+0000 7f4f7affd640 10 monclient: dump: 2024-04-26T00:41:43.414 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:43.414 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:43.414 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:43.414 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:43.414 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:43.414 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:43.414 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:43.414 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:43.414 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:43.415 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:43.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.405+0000 7f4f7affd640 10 monclient: _finish_auth 0 2024-04-26T00:41:43.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.405+0000 7f4f7affd640 10 monclient: _check_auth_tickets 2024-04-26T00:41:43.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.405+0000 7f4f7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:41:43.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.405+0000 7f4f7affd640 1 -- 172.21.15.37:0/3773793894 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f8000cc70 con 0x7f4f8c09f850 2024-04-26T00:41:43.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.405+0000 7f4f909ad640 5 monclient: authenticate success, global_id 254276 2024-04-26T00:41:43.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.405+0000 7f4f909ad640 10 monclient: _renew_subs 2024-04-26T00:41:43.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.405+0000 7f4f909ad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:41:43.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.405+0000 7f4f909ad640 1 -- 172.21.15.37:0/3773793894 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f8c1265b0 con 0x7f4f8c09f850 2024-04-26T00:41:43.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.405+0000 7f4f7affd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:41:43.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.405+0000 7f4f7affd640 1 -- 172.21.15.37:0/3773793894 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f80013670 con 0x7f4f8c09f850 2024-04-26T00:41:43.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.405+0000 7f4f7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:43.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.405+0000 7f4f7affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:41:43.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.405+0000 7f4f7affd640 10 monclient: dump: 2024-04-26T00:41:43.415 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:43.416 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:43.416 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:43.416 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:43.416 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:43.416 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:43.416 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:43.416 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:43.416 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:43.416 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:43.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.405+0000 7f4f88d99640 4 set_mon_vals no callback set 2024-04-26T00:41:43.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.405+0000 7f4f909ad640 10 monclient: _renew_subs 2024-04-26T00:41:43.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.405+0000 7f4f909ad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:41:43.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.405+0000 7f4f909ad640 1 -- 172.21.15.37:0/3773793894 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f8c127330 con 0x7f4f8c09f850 2024-04-26T00:41:43.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.405+0000 7f4f909ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:43.416 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.405+0000 7f4f909ad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:43.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.405+0000 7f4f909ad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:41:43.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.405+0000 7f4f909ad640 1 -- 172.21.15.37:0/3773793894 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f8c066f40 con 0x7f4f8c09f850 2024-04-26T00:41:43.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.405+0000 7f4f7affd640 1 -- 172.21.15.37:0/3773793894 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f4f8001a070 con 0x7f4f8c09f850 2024-04-26T00:41:43.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.405+0000 7f4f7affd640 1 --2- 172.21.15.37:0/3773793894 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4f6c035e30 0x7f4f6c0382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:43.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.405+0000 7f4f7affd640 1 -- 172.21.15.37:0/3773793894 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(863..863 src has 705..863) ==== 3975+0+0 (secure 0 0 0) 0x7f4f80048ab0 con 0x7f4f8c09f850 2024-04-26T00:41:43.417 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.405+0000 7f4f8a59c640 1 --2- 172.21.15.37:0/3773793894 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4f6c035e30 0x7f4f6c0382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:43.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.405+0000 7f4f8a59c640 10 monclient: get_auth_request con 0x7f4f6c035e30 auth_method 0 2024-04-26T00:41:43.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.409+0000 7f4f8a59c640 1 --2- 172.21.15.37:0/3773793894 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4f6c035e30 0x7f4f6c0382f0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c0ac000 tx=0x7f4f7c0a7070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:43.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.409+0000 7f4f7affd640 1 -- 172.21.15.37:0/3773793894 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f4f8000ce20 con 0x7f4f8c09f850 2024-04-26T00:41:43.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.409+0000 7f4f7affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:43.418 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.409+0000 7f4f7affd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:41:43.624 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.621+0000 7f4f909ad640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:41:43.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.621+0000 7f4f909ad640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:41:43.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.621+0000 7f4f909ad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:41:43.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.621+0000 7f4f909ad640 1 -- 172.21.15.37:0/3773793894 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4f8c09d110 con 0x7f4f8c09f850 2024-04-26T00:41:43.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.621+0000 7f4f7affd640 1 -- 172.21.15.37:0/3773793894 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1558 (secure 0 0 0) 0x7f4f8001a520 con 0x7f4f8c09f850 2024-04-26T00:41:43.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.621+0000 7f4f7affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:41:43.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.621+0000 7f4f7affd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:41:43.626 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":156,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:41:43.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.625+0000 7f4f909ad640 1 -- 172.21.15.37:0/3773793894 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4f6c035e30 msgr2=0x7f4f6c0382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:43.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.625+0000 7f4f909ad640 1 --2- 172.21.15.37:0/3773793894 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4f6c035e30 0x7f4f6c0382f0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c0ac000 tx=0x7f4f7c0a7070 comp rx=0 tx=0).stop 2024-04-26T00:41:43.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.625+0000 7f4f909ad640 10 monclient: shutdown 2024-04-26T00:41:43.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.625+0000 7f4f909ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:41:43.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.625+0000 7f4f909ad640 1 -- 172.21.15.37:0/3773793894 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f8c09f850 msgr2=0x7f4f8c13cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:43.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.625+0000 7f4f909ad640 1 --2- 172.21.15.37:0/3773793894 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f8c09f850 0x7f4f8c13cf00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4f8000e9d0 tx=0x7f4f8000ee90 comp rx=0 tx=0).stop 2024-04-26T00:41:43.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.625+0000 7f4f909ad640 1 -- 172.21.15.37:0/3773793894 shutdown_connections 2024-04-26T00:41:43.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.625+0000 7f4f909ad640 1 --2- 172.21.15.37:0/3773793894 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4f6c035e30 0x7f4f6c0382f0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:43.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.625+0000 7f4f909ad640 1 --2- 172.21.15.37:0/3773793894 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f8c0a0f50 0x7f4f8c124960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:43.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.625+0000 7f4f909ad640 1 --2- 172.21.15.37:0/3773793894 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f8c09f850 0x7f4f8c13cf00 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:43.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.625+0000 7f4f909ad640 1 --2- 172.21.15.37:0/3773793894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f8c06c820 0x7f4f8c1ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:43.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.625+0000 7f4f909ad640 1 -- 172.21.15.37:0/3773793894 >> 172.21.15.37:0/3773793894 conn(0x7f4f8c0a4170 msgr2=0x7f4f8c0a6b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:41:43.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.625+0000 7f4f909ad640 1 -- 172.21.15.37:0/3773793894 shutdown_connections 2024-04-26T00:41:43.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:43.625+0000 7f4f909ad640 1 -- 172.21.15.37:0/3773793894 wait complete. 2024-04-26T00:41:46.643 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:41:46.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.772+0000 7fa424730640 10 monclient: get_monmap_and_config 2024-04-26T00:41:46.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.772+0000 7fa424730640 10 monclient: build_initial_monmap 2024-04-26T00:41:46.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.772+0000 7fa424730640 10 monclient: monmap: 2024-04-26T00:41:46.776 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:41:46.776 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:46.776 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:41:46.777470+0000 2024-04-26T00:41:46.776 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:41:46.777470+0000 2024-04-26T00:41:46.776 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:41:46.776 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:46.776 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:41:46.776 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:41:46.776 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:41:46.776 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:41:46.776 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:41:46.776 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:46.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.772+0000 7fa424730640 1 Processor -- start 2024-04-26T00:41:46.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.772+0000 7fa424730640 1 -- start start 2024-04-26T00:41:46.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.772+0000 7fa424730640 10 monclient: init 2024-04-26T00:41:46.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.772+0000 7fa424730640 10 monclient: _reopen_session rank -1 2024-04-26T00:41:46.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.772+0000 7fa424730640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-04-26T00:41:46.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.772+0000 7fa424730640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa41c0a0260 0x7fa41c0a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:46.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.772+0000 7fa424730640 10 monclient(hunting): picked mon.noname-c con 0x7fa41c0a0260 addr v2:172.21.15.45:3300/0 2024-04-26T00:41:46.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.772+0000 7fa424730640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa41c134b30 0x7fa41c134f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:46.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.772+0000 7fa424730640 10 monclient(hunting): picked mon.noname-a con 0x7fa41c134b30 addr v2:172.21.15.37:3300/0 2024-04-26T00:41:46.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.772+0000 7fa424730640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa41c135490 addr v1:172.21.15.45:6789/0 2024-04-26T00:41:46.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.772+0000 7fa424730640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa41c0a1130 con 0x7fa41c135490 2024-04-26T00:41:46.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.772+0000 7fa424730640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:46.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.772+0000 7fa424730640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fa41c09ec10 con 0x7fa41c134b30 2024-04-26T00:41:46.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.772+0000 7fa424730640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:46.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.772+0000 7fa424730640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fa41c09f0f0 con 0x7fa41c0a0260 2024-04-26T00:41:46.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.772+0000 7fa424730640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:46.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.772+0000 7fa424730640 10 monclient(hunting): authenticate will time out at 2288.083163s 2024-04-26T00:41:46.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.772+0000 7fa4224cc640 1 -- >> v2:172.21.15.45:3300/0 conn(0x7fa41c0a0260 msgr2=0x7fa41c0a0640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:41:46.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.772+0000 7fa422ccd640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7fa41c135490 legacy=0x7fa41c13be60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:41:46.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.772+0000 7fa4224cc640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa41c0a0260 0x7fa41c0a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:41:46.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.772+0000 7fa421ccb640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa41c134b30 0x7fa41c134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:46.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.772+0000 7fa421ccb640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa41c134b30 0x7fa41c134f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:34086/0 (socket says 172.21.15.37:34086) 2024-04-26T00:41:46.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.772+0000 7fa421ccb640 1 -- 172.21.15.37:0/3899998930 learned_addr learned my addr 172.21.15.37:0/3899998930 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:41:46.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.772+0000 7fa421ccb640 10 monclient(hunting): get_auth_request con 0x7fa41c134b30 auth_method 0 2024-04-26T00:41:46.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.772+0000 7fa421ccb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:46.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.772+0000 7fa421ccb640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:46.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.772+0000 7fa421ccb640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:46.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.772+0000 7fa421ccb640 1 -- 172.21.15.37:0/3899998930 >> v2:172.21.15.37:3300/0 conn(0x7fa41c134b30 msgr2=0x7fa41c134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:41:46.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.772+0000 7fa421ccb640 1 -- 172.21.15.37:0/3899998930 >> v2:172.21.15.37:3300/0 conn(0x7fa41c134b30 msgr2=0x7fa41c134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:41:46.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.772+0000 7fa421ccb640 1 --2- 172.21.15.37:0/3899998930 >> v2:172.21.15.37:3300/0 conn(0x7fa41c134b30 0x7fa41c134f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:41:46.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.772+0000 7fa421ccb640 1 --2- 172.21.15.37:0/3899998930 >> v2:172.21.15.37:3300/0 conn(0x7fa41c134b30 0x7fa41c134f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:41:46.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.976+0000 7fa4224cc640 1 -- 172.21.15.37:0/3899998930 >> v2:172.21.15.45:3300/0 conn(0x7fa41c0a0260 msgr2=0x7fa41c0a0640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:41:46.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.976+0000 7fa422ccd640 1 -- 172.21.15.37:0/3899998930 >> v1:172.21.15.45:6789/0 conn(0x7fa41c135490 legacy=0x7fa41c13be60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:41:46.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.976+0000 7fa4224cc640 1 --2- 172.21.15.37:0/3899998930 >> v2:172.21.15.45:3300/0 conn(0x7fa41c0a0260 0x7fa41c0a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:41:46.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.976+0000 7fa421ccb640 1 --2- 172.21.15.37:0/3899998930 >> v2:172.21.15.37:3300/0 conn(0x7fa41c134b30 0x7fa41c134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:46.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.976+0000 7fa421ccb640 10 monclient(hunting): get_auth_request con 0x7fa41c134b30 auth_method 0 2024-04-26T00:41:46.978 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.976+0000 7fa421ccb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:46.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.976+0000 7fa421ccb640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:46.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.976+0000 7fa421ccb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:41:46.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.976+0000 7fa421ccb640 1 -- 172.21.15.37:0/3899998930 >> v2:172.21.15.37:3300/0 conn(0x7fa41c134b30 msgr2=0x7fa41c134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:41:46.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.976+0000 7fa421ccb640 1 -- 172.21.15.37:0/3899998930 >> v2:172.21.15.37:3300/0 conn(0x7fa41c134b30 msgr2=0x7fa41c134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:41:46.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.976+0000 7fa421ccb640 1 --2- 172.21.15.37:0/3899998930 >> v2:172.21.15.37:3300/0 conn(0x7fa41c134b30 0x7fa41c134f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:41:46.979 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:46.976+0000 7fa421ccb640 1 --2- 172.21.15.37:0/3899998930 >> v2:172.21.15.37:3300/0 conn(0x7fa41c134b30 0x7fa41c134f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:41:47.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:47.380+0000 7fa4224cc640 1 -- 172.21.15.37:0/3899998930 >> v2:172.21.15.45:3300/0 conn(0x7fa41c0a0260 msgr2=0x7fa41c0a0640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:41:47.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:47.380+0000 7fa422ccd640 1 -- 172.21.15.37:0/3899998930 >> v1:172.21.15.45:6789/0 conn(0x7fa41c135490 legacy=0x7fa41c13be60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:41:47.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:47.380+0000 7fa4224cc640 1 --2- 172.21.15.37:0/3899998930 >> v2:172.21.15.45:3300/0 conn(0x7fa41c0a0260 0x7fa41c0a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:41:47.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:47.380+0000 7fa421ccb640 1 --2- 172.21.15.37:0/3899998930 >> v2:172.21.15.37:3300/0 conn(0x7fa41c134b30 0x7fa41c134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:47.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:47.380+0000 7fa421ccb640 10 monclient(hunting): get_auth_request con 0x7fa41c134b30 auth_method 0 2024-04-26T00:41:47.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:47.380+0000 7fa421ccb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:47.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:47.380+0000 7fa421ccb640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:47.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:47.380+0000 7fa421ccb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:41:47.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:47.380+0000 7fa421ccb640 1 -- 172.21.15.37:0/3899998930 >> v2:172.21.15.37:3300/0 conn(0x7fa41c134b30 msgr2=0x7fa41c134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:41:47.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:47.380+0000 7fa421ccb640 1 -- 172.21.15.37:0/3899998930 >> v2:172.21.15.37:3300/0 conn(0x7fa41c134b30 msgr2=0x7fa41c134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:41:47.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:47.380+0000 7fa421ccb640 1 --2- 172.21.15.37:0/3899998930 >> v2:172.21.15.37:3300/0 conn(0x7fa41c134b30 0x7fa41c134f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:41:47.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:47.380+0000 7fa421ccb640 1 --2- 172.21.15.37:0/3899998930 >> v2:172.21.15.37:3300/0 conn(0x7fa41c134b30 0x7fa41c134f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:41:48.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:48.184+0000 7fa4224cc640 1 -- 172.21.15.37:0/3899998930 >> v2:172.21.15.45:3300/0 conn(0x7fa41c0a0260 msgr2=0x7fa41c0a0640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:41:48.186 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:48.184+0000 7fa422ccd640 1 -- 172.21.15.37:0/3899998930 >> v1:172.21.15.45:6789/0 conn(0x7fa41c135490 legacy=0x7fa41c13be60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:41:48.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:48.184+0000 7fa4224cc640 1 --2- 172.21.15.37:0/3899998930 >> v2:172.21.15.45:3300/0 conn(0x7fa41c0a0260 0x7fa41c0a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:41:48.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:48.184+0000 7fa421ccb640 1 --2- 172.21.15.37:0/3899998930 >> v2:172.21.15.37:3300/0 conn(0x7fa41c134b30 0x7fa41c134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:48.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:48.184+0000 7fa421ccb640 10 monclient(hunting): get_auth_request con 0x7fa41c134b30 auth_method 0 2024-04-26T00:41:48.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:48.184+0000 7fa421ccb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:48.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:48.184+0000 7fa421ccb640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:48.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:48.184+0000 7fa421ccb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:41:48.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:48.184+0000 7fa421ccb640 1 -- 172.21.15.37:0/3899998930 >> v2:172.21.15.37:3300/0 conn(0x7fa41c134b30 msgr2=0x7fa41c134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:41:48.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:48.184+0000 7fa421ccb640 1 -- 172.21.15.37:0/3899998930 >> v2:172.21.15.37:3300/0 conn(0x7fa41c134b30 msgr2=0x7fa41c134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:41:48.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:48.184+0000 7fa421ccb640 1 --2- 172.21.15.37:0/3899998930 >> v2:172.21.15.37:3300/0 conn(0x7fa41c134b30 0x7fa41c134f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:41:48.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:48.184+0000 7fa421ccb640 1 --2- 172.21.15.37:0/3899998930 >> v2:172.21.15.37:3300/0 conn(0x7fa41c134b30 0x7fa41c134f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:41:49.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:49.776+0000 7fa413fff640 10 monclient(hunting): tick 2024-04-26T00:41:49.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:49.776+0000 7fa413fff640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:41:49.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:49.776+0000 7fa413fff640 1 monclient(hunting): continuing hunt 2024-04-26T00:41:49.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:49.776+0000 7fa413fff640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:41:49.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:49.776+0000 7fa413fff640 1 -- 172.21.15.37:0/3899998930 >> v2:172.21.15.45:3300/0 conn(0x7fa41c0a0260 msgr2=0x7fa41c0a0640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:41:49.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:49.776+0000 7fa413fff640 1 --2- 172.21.15.37:0/3899998930 >> v2:172.21.15.45:3300/0 conn(0x7fa41c0a0260 0x7fa41c0a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:49.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:49.776+0000 7fa413fff640 1 -- 172.21.15.37:0/3899998930 >> v2:172.21.15.37:3300/0 conn(0x7fa41c134b30 msgr2=0x7fa41c134f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:41:49.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:49.776+0000 7fa413fff640 1 --2- 172.21.15.37:0/3899998930 >> v2:172.21.15.37:3300/0 conn(0x7fa41c134b30 0x7fa41c134f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:49.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:49.776+0000 7fa413fff640 1 -- 172.21.15.37:0/3899998930 >> v1:172.21.15.45:6789/0 conn(0x7fa41c135490 legacy=0x7fa41c13be60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:41:49.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:49.776+0000 7fa413fff640 10 monclient: _add_conns ranks=[2,0] 2024-04-26T00:41:49.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:49.776+0000 7fa413fff640 1 --2- 172.21.15.37:0/3899998930 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa400003680 0x7fa400005ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:49.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:49.776+0000 7fa413fff640 10 monclient(hunting): picked mon.noname-b con 0x7fa400003680 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:41:49.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:49.776+0000 7fa413fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa4000061b0 addr v1:172.21.15.37:6789/0 2024-04-26T00:41:49.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:49.776+0000 7fa413fff640 1 -- 172.21.15.37:0/3899998930 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa41c06cff0 con 0x7fa4000061b0 2024-04-26T00:41:49.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:49.776+0000 7fa413fff640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:49.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:49.776+0000 7fa413fff640 1 -- 172.21.15.37:0/3899998930 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa41c09f0f0 con 0x7fa400003680 2024-04-26T00:41:49.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:49.776+0000 7fa413fff640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:49.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:49.776+0000 7fa4224cc640 1 --2- 172.21.15.37:0/3899998930 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa400003680 0x7fa400005ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:49.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:49.776+0000 7fa4224cc640 10 monclient(hunting): get_auth_request con 0x7fa400003680 auth_method 0 2024-04-26T00:41:49.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:49.776+0000 7fa4224cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:49.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:49.776+0000 7fa4224cc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:49.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:49.776+0000 7fa4224cc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:49.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:49.776+0000 7fa4224cc640 1 -- 172.21.15.37:0/3899998930 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa400003680 msgr2=0x7fa400005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:41:49.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:49.776+0000 7fa4224cc640 1 -- 172.21.15.37:0/3899998930 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa400003680 msgr2=0x7fa400005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:41:49.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:49.776+0000 7fa4224cc640 1 --2- 172.21.15.37:0/3899998930 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa400003680 0x7fa400005ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:41:49.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:49.776+0000 7fa4224cc640 1 --2- 172.21.15.37:0/3899998930 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa400003680 0x7fa400005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:41:49.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:49.776+0000 7fa421ccb640 1 -- 172.21.15.37:0/3899998930 >> v1:172.21.15.37:6789/0 conn(0x7fa4000061b0 legacy=0x7fa400008670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:49.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:49.980+0000 7fa4224cc640 1 --2- 172.21.15.37:0/3899998930 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa400003680 0x7fa400005ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:49.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:49.980+0000 7fa4224cc640 10 monclient(hunting): get_auth_request con 0x7fa400003680 auth_method 0 2024-04-26T00:41:49.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:49.980+0000 7fa4224cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:49.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:49.980+0000 7fa4224cc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:49.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:49.980+0000 7fa4224cc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:41:49.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:49.980+0000 7fa4224cc640 1 -- 172.21.15.37:0/3899998930 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa400003680 msgr2=0x7fa400005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:41:49.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:49.980+0000 7fa4224cc640 1 -- 172.21.15.37:0/3899998930 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa400003680 msgr2=0x7fa400005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:41:49.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:49.980+0000 7fa4224cc640 1 --2- 172.21.15.37:0/3899998930 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa400003680 0x7fa400005ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:41:49.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:49.980+0000 7fa4224cc640 1 --2- 172.21.15.37:0/3899998930 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa400003680 0x7fa400005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:41:50.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:50.384+0000 7fa4224cc640 1 --2- 172.21.15.37:0/3899998930 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa400003680 0x7fa400005ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:50.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:50.384+0000 7fa4224cc640 10 monclient(hunting): get_auth_request con 0x7fa400003680 auth_method 0 2024-04-26T00:41:50.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:50.384+0000 7fa4224cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:50.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:50.384+0000 7fa4224cc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:50.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:50.384+0000 7fa4224cc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:41:50.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:50.384+0000 7fa4224cc640 1 -- 172.21.15.37:0/3899998930 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa400003680 msgr2=0x7fa400005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:41:50.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:50.384+0000 7fa4224cc640 1 -- 172.21.15.37:0/3899998930 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa400003680 msgr2=0x7fa400005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:41:50.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:50.384+0000 7fa4224cc640 1 --2- 172.21.15.37:0/3899998930 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa400003680 0x7fa400005ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:41:50.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:50.384+0000 7fa4224cc640 1 --2- 172.21.15.37:0/3899998930 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa400003680 0x7fa400005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:41:51.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.188+0000 7fa4224cc640 1 --2- 172.21.15.37:0/3899998930 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa400003680 0x7fa400005ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:51.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.188+0000 7fa4224cc640 10 monclient(hunting): get_auth_request con 0x7fa400003680 auth_method 0 2024-04-26T00:41:51.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.188+0000 7fa4224cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:51.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.188+0000 7fa4224cc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:51.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.188+0000 7fa4224cc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:41:51.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.188+0000 7fa4224cc640 1 -- 172.21.15.37:0/3899998930 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa400003680 msgr2=0x7fa400005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:41:51.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.188+0000 7fa4224cc640 1 -- 172.21.15.37:0/3899998930 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa400003680 msgr2=0x7fa400005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:41:51.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.188+0000 7fa4224cc640 1 --2- 172.21.15.37:0/3899998930 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa400003680 0x7fa400005ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:41:51.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.188+0000 7fa4224cc640 1 --2- 172.21.15.37:0/3899998930 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa400003680 0x7fa400005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:41:51.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.528+0000 7fa4214ca640 1 -- 172.21.15.37:0/3899998930 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2486179435 0 0) 0x7fa41c06cff0 con 0x7fa4000061b0 2024-04-26T00:41:51.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.528+0000 7fa4214ca640 10 monclient(hunting): _negotiate 2024-04-26T00:41:51.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.528+0000 7fa4214ca640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:51.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.528+0000 7fa4214ca640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:51.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.528+0000 7fa4214ca640 10 monclient(hunting): my global_id is 264156 2024-04-26T00:41:51.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.528+0000 7fa4214ca640 1 -- 172.21.15.37:0/3899998930 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa408003530 con 0x7fa4000061b0 2024-04-26T00:41:51.536 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.528+0000 7fa4214ca640 1 -- 172.21.15.37:0/3899998930 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 82567219 0 0) 0x7fa408003530 con 0x7fa4000061b0 2024-04-26T00:41:51.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.528+0000 7fa4214ca640 1 -- 172.21.15.37:0/3899998930 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa400003680 msgr2=0x7fa400005ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:41:51.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.528+0000 7fa4214ca640 1 --2- 172.21.15.37:0/3899998930 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa400003680 0x7fa400005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:51.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.528+0000 7fa4214ca640 10 monclient: _finish_hunting 0 2024-04-26T00:41:51.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.528+0000 7fa4214ca640 1 monclient: found mon.noname-a-legacy 2024-04-26T00:41:51.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.528+0000 7fa4214ca640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:51.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.528+0000 7fa4214ca640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.37:6789/0 2024-04-26T00:41:51.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.528+0000 7fa4214ca640 1 -- 172.21.15.37:0/3899998930 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa40000af70 con 0x7fa4000061b0 2024-04-26T00:41:51.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.528+0000 7fa4214ca640 10 monclient: _finish_auth 0 2024-04-26T00:41:51.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.528+0000 7fa4214ca640 10 monclient: _check_auth_tickets 2024-04-26T00:41:51.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.528+0000 7fa4214ca640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:41:51.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa4214ca640 1 -- 172.21.15.37:0/3899998930 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 866326386 0 0) 0x7fa40c00f040 con 0x7fa4000061b0 2024-04-26T00:41:51.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa4214ca640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:51.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa4214ca640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-26T00:41:51.538 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa4214ca640 10 monclient: dump: 2024-04-26T00:41:51.538 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:51.538 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:51.538 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:51.538 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:51.538 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:51.538 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:51.538 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:51.538 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:51.539 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:51.539 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:51.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa4214ca640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] but i'm connected to v1:172.21.15.37:6789/0, reconnecting 2024-04-26T00:41:51.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa4214ca640 10 monclient: _reopen_session rank -1 2024-04-26T00:41:51.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa4214ca640 1 -- 172.21.15.37:0/3899998930 >> v1:172.21.15.37:6789/0 conn(0x7fa4000061b0 legacy=0x7fa400008670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:51.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa422ccd640 1 -- 172.21.15.37:0/3899998930 reap_dead start 2024-04-26T00:41:51.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa4214ca640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:41:51.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa4214ca640 1 --2- 172.21.15.37:0/3899998930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4080039c0 0x7fa408003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:51.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa4214ca640 10 monclient(hunting): picked mon.b con 0x7fa4080039c0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:41:51.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa4214ca640 1 --2- 172.21.15.37:0/3899998930 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa408004810 0x7fa408004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:51.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa4214ca640 10 monclient(hunting): picked mon.c con 0x7fa408004810 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:41:51.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa4214ca640 1 --2- 172.21.15.37:0/3899998930 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa408005220 0x7fa40800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:51.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa4214ca640 10 monclient(hunting): picked mon.a con 0x7fa408005220 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:41:51.539 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa4214ca640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:51.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa4214ca640 1 -- 172.21.15.37:0/3899998930 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa41c09f0f0 con 0x7fa408005220 2024-04-26T00:41:51.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa4214ca640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:51.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa4214ca640 1 -- 172.21.15.37:0/3899998930 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa40800c150 con 0x7fa4080039c0 2024-04-26T00:41:51.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa4214ca640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:51.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa4214ca640 1 -- 172.21.15.37:0/3899998930 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa40800c350 con 0x7fa408004810 2024-04-26T00:41:51.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa4214ca640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:51.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa4214ca640 10 monclient(hunting): _finish_auth 0 2024-04-26T00:41:51.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa4224cc640 1 -- 172.21.15.37:0/3899998930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4080039c0 msgr2=0x7fa408003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:41:51.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa4224cc640 1 --2- 172.21.15.37:0/3899998930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4080039c0 0x7fa408003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:41:51.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa422ccd640 1 --2- 172.21.15.37:0/3899998930 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa408005220 0x7fa40800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:51.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa422ccd640 10 monclient(hunting): get_auth_request con 0x7fa408005220 auth_method 0 2024-04-26T00:41:51.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa422ccd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:51.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa422ccd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:51.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa422ccd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:41:51.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa422ccd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:51.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa422ccd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:51.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa422ccd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:41:51.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa422ccd640 10 monclient(hunting): handle_auth_done global_id 264156 payload 293 2024-04-26T00:41:51.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa422ccd640 1 -- 172.21.15.37:0/3899998930 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa408004810 msgr2=0x7fa408004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:41:51.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa422ccd640 1 --2- 172.21.15.37:0/3899998930 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa408004810 0x7fa408004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:51.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa422ccd640 1 -- 172.21.15.37:0/3899998930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4080039c0 msgr2=0x7fa408003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:41:51.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa422ccd640 1 --2- 172.21.15.37:0/3899998930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4080039c0 0x7fa408003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:51.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa422ccd640 10 monclient: _finish_hunting 0 2024-04-26T00:41:51.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa422ccd640 1 monclient: found mon.a 2024-04-26T00:41:51.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa422ccd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:51.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa422ccd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:41:51.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa422ccd640 1 -- 172.21.15.37:0/3899998930 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa408003530 con 0x7fa408005220 2024-04-26T00:41:51.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa422ccd640 10 monclient: _finish_auth 0 2024-04-26T00:41:51.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa422ccd640 10 monclient: _check_auth_tickets 2024-04-26T00:41:51.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa422ccd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:41:51.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa424730640 5 monclient: authenticate success, global_id 264156 2024-04-26T00:41:51.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.532+0000 7fa424730640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:41:51.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.536+0000 7fa422ccd640 1 -- 172.21.15.37:0/3899998930 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa408005220 msgr2=0x7fa40800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:51.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.536+0000 7fa422ccd640 1 --2- 172.21.15.37:0/3899998930 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa408005220 0x7fa40800b9d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa41c0a15a0 tx=0x7fa41401c5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a08c4b09b2f63f1e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:51.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.536+0000 7fa4214ca640 1 -- 172.21.15.37:0/3899998930 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa414006e60 con 0x7fa408005220 2024-04-26T00:41:51.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.536+0000 7fa4214ca640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:51.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.536+0000 7fa4214ca640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:41:51.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.536+0000 7fa4214ca640 10 monclient: dump: 2024-04-26T00:41:51.544 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:51.544 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:51.544 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:51.544 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:51.544 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:51.544 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:51.545 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:51.545 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:51.545 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:51.545 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:51.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.536+0000 7fa4214ca640 1 -- 172.21.15.37:0/3899998930 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4140ad070 con 0x7fa408005220 2024-04-26T00:41:51.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.536+0000 7fa4214ca640 10 monclient: handle_config config(0 keys) 2024-04-26T00:41:51.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.536+0000 7fa4214ca640 1 -- 172.21.15.37:0/3899998930 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4140a8850 con 0x7fa408005220 2024-04-26T00:41:51.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.536+0000 7fa424730640 10 monclient: get_monmap_and_config success 2024-04-26T00:41:51.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.536+0000 7fa424730640 4 set_mon_vals no callback set 2024-04-26T00:41:51.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.536+0000 7fa4214ca640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:51.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.536+0000 7fa4214ca640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:41:51.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.536+0000 7fa4214ca640 10 monclient: dump: 2024-04-26T00:41:51.546 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:51.546 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:51.546 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:51.546 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:51.546 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:51.546 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:51.546 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:51.546 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:51.546 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:51.546 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:51.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa424730640 10 monclient: shutdown 2024-04-26T00:41:51.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa424730640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:41:51.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa424730640 1 -- 172.21.15.37:0/3899998930 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa408005220 msgr2=0x7fa40800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:51.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa424730640 1 --2- 172.21.15.37:0/3899998930 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa408005220 0x7fa40800b9d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa41c0a15a0 tx=0x7fa41401c5c0 comp rx=0 tx=0).stop 2024-04-26T00:41:51.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa424730640 1 -- 172.21.15.37:0/3899998930 shutdown_connections 2024-04-26T00:41:51.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa424730640 1 --2- 172.21.15.37:0/3899998930 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa408005220 0x7fa40800b9d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:51.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa424730640 1 --2- 172.21.15.37:0/3899998930 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa408004810 0x7fa408004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:51.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa424730640 1 --2- 172.21.15.37:0/3899998930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4080039c0 0x7fa408003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:51.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa424730640 1 -- 172.21.15.37:0/3899998930 >> 172.21.15.37:0/3899998930 conn(0x7fa41c0a1ad0 msgr2=0x7fa41c0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:41:51.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa424730640 1 -- 172.21.15.37:0/3899998930 shutdown_connections 2024-04-26T00:41:51.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa424730640 1 -- 172.21.15.37:0/3899998930 wait complete. 2024-04-26T00:41:51.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa424730640 10 monclient: build_initial_monmap 2024-04-26T00:41:51.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa424730640 10 monclient: monmap: 2024-04-26T00:41:51.548 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:41:51.548 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:41:51.548 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:41:51.548 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:41:51.549 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:41:51.549 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:41:51.549 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:41:51.549 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:41:51.549 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:41:51.550 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:51.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa424730640 1 Processor -- start 2024-04-26T00:41:51.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa424730640 1 -- start start 2024-04-26T00:41:51.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa424730640 10 monclient: init 2024-04-26T00:41:51.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa424730640 10 monclient: _reopen_session rank -1 2024-04-26T00:41:51.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa424730640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:41:51.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa424730640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa4080039c0 0x7fa41c1c9720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:51.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa424730640 10 monclient(hunting): picked mon.noname-a con 0x7fa4080039c0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:41:51.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa424730640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa408004810 0x7fa41c131d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:51.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa424730640 10 monclient(hunting): picked mon.noname-b con 0x7fa408004810 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:41:51.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa424730640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa408005220 0x7fa41c132270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:51.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa424730640 10 monclient(hunting): picked mon.noname-c con 0x7fa408005220 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:41:51.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa424730640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:51.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa424730640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa41c09bdd0 con 0x7fa4080039c0 2024-04-26T00:41:51.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa424730640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:51.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa424730640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa41c09bc50 con 0x7fa408004810 2024-04-26T00:41:51.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa424730640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:51.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa424730640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa41c09f2b0 con 0x7fa408005220 2024-04-26T00:41:51.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa424730640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:51.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa424730640 10 monclient(hunting): authenticate will time out at 2292.851704s 2024-04-26T00:41:51.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa4224cc640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa4080039c0 0x7fa41c1c9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:51.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa421ccb640 1 -- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa408004810 msgr2=0x7fa41c131d30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:41:51.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa421ccb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa408004810 0x7fa41c131d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:41:51.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa4224cc640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa4080039c0 0x7fa41c1c9720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:35584/0 (socket says 172.21.15.37:35584) 2024-04-26T00:41:51.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa4224cc640 1 -- 172.21.15.37:0/3332302340 learned_addr learned my addr 172.21.15.37:0/3332302340 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:41:51.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa4224cc640 10 monclient(hunting): get_auth_request con 0x7fa4080039c0 auth_method 0 2024-04-26T00:41:51.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa4224cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:51.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa4224cc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:51.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.540+0000 7fa4224cc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:51.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.544+0000 7fa4224cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:51.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.544+0000 7fa4224cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:51.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.544+0000 7fa422ccd640 1 --2- 172.21.15.37:0/3332302340 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa408005220 0x7fa41c132270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:51.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.544+0000 7fa4224cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:51.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.544+0000 7fa422ccd640 10 monclient(hunting): get_auth_request con 0x7fa408005220 auth_method 0 2024-04-26T00:41:51.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.544+0000 7fa422ccd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:51.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.544+0000 7fa422ccd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:51.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.544+0000 7fa422ccd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:51.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.544+0000 7fa422ccd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:51.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.544+0000 7fa422ccd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:51.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.544+0000 7fa422ccd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:51.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.544+0000 7fa4224cc640 10 monclient(hunting): handle_auth_done global_id 264159 payload 824 2024-04-26T00:41:51.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.544+0000 7fa4224cc640 1 -- 172.21.15.37:0/3332302340 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa408005220 msgr2=0x7fa41c132270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:51.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.544+0000 7fa4224cc640 1 --2- 172.21.15.37:0/3332302340 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa408005220 0x7fa41c132270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:51.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.544+0000 7fa4224cc640 1 -- 172.21.15.37:0/3332302340 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa408004810 msgr2=0x7fa41c131d30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:41:51.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.544+0000 7fa4224cc640 1 --2- 172.21.15.37:0/3332302340 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa408004810 0x7fa41c131d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:51.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.544+0000 7fa4224cc640 10 monclient: _finish_hunting 0 2024-04-26T00:41:51.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.544+0000 7fa4224cc640 1 monclient: found mon.noname-a 2024-04-26T00:41:51.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.544+0000 7fa4224cc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:51.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.544+0000 7fa4224cc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:41:51.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.544+0000 7fa4224cc640 1 -- 172.21.15.37:0/3332302340 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa41c11d400 con 0x7fa4080039c0 2024-04-26T00:41:51.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.544+0000 7fa4224cc640 1 -- 172.21.15.37:0/3332302340 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa4080039c0 msgr2=0x7fa41c1c9720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:51.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.544+0000 7fa4224cc640 1 --2- 172.21.15.37:0/3332302340 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa4080039c0 0x7fa41c1c9720 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa41800bb50 tx=0x7fa418005480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:51.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.548+0000 7fa412ffd640 1 -- 172.21.15.37:0/3332302340 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa418017600 con 0x7fa4080039c0 2024-04-26T00:41:51.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.548+0000 7fa412ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:51.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.548+0000 7fa412ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:41:51.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.548+0000 7fa412ffd640 10 monclient: dump: 2024-04-26T00:41:51.559 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:51.559 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:51.559 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:51.560 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:51.560 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:51.560 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:51.560 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:51.560 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:51.560 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:51.560 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:51.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.548+0000 7fa412ffd640 10 monclient: _finish_auth 0 2024-04-26T00:41:51.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.548+0000 7fa412ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:41:51.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.548+0000 7fa412ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:41:51.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.548+0000 7fa412ffd640 1 -- 172.21.15.37:0/3332302340 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa41800be50 con 0x7fa4080039c0 2024-04-26T00:41:51.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.548+0000 7fa412ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:41:51.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.548+0000 7fa412ffd640 1 -- 172.21.15.37:0/3332302340 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4180179e0 con 0x7fa4080039c0 2024-04-26T00:41:51.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.548+0000 7fa424730640 5 monclient: authenticate success, global_id 264159 2024-04-26T00:41:51.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.548+0000 7fa424730640 10 monclient: _renew_subs 2024-04-26T00:41:51.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.548+0000 7fa424730640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:41:51.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.548+0000 7fa424730640 1 -- 172.21.15.37:0/3332302340 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa41c11f6f0 con 0x7fa4080039c0 2024-04-26T00:41:51.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.548+0000 7fa412ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:51.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.548+0000 7fa412ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:41:51.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.548+0000 7fa412ffd640 10 monclient: dump: 2024-04-26T00:41:51.561 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:51.561 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:51.561 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:51.561 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:51.561 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:51.561 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:51.561 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:51.561 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:51.561 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:51.562 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:51.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.548+0000 7fa424730640 10 monclient: _renew_subs 2024-04-26T00:41:51.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.548+0000 7fa424730640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:41:51.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.548+0000 7fa424730640 1 -- 172.21.15.37:0/3332302340 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa41c11f410 con 0x7fa4080039c0 2024-04-26T00:41:51.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.548+0000 7fa413fff640 4 set_mon_vals no callback set 2024-04-26T00:41:51.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.548+0000 7fa424730640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:51.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.548+0000 7fa424730640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:51.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.548+0000 7fa424730640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:41:51.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.548+0000 7fa424730640 1 -- 172.21.15.37:0/3332302340 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa404005090 con 0x7fa4080039c0 2024-04-26T00:41:51.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.552+0000 7fa412ffd640 1 -- 172.21.15.37:0/3332302340 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa418017b80 con 0x7fa4080039c0 2024-04-26T00:41:51.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.552+0000 7fa412ffd640 1 --2- 172.21.15.37:0/3332302340 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa400036820 0x7fa400038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:51.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.552+0000 7fa412ffd640 1 -- 172.21.15.37:0/3332302340 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(863..863 src has 705..863) ==== 3975+0+0 (secure 0 0 0) 0x7fa41804a180 con 0x7fa4080039c0 2024-04-26T00:41:51.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.552+0000 7fa421ccb640 1 --2- 172.21.15.37:0/3332302340 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa400036820 0x7fa400038ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:51.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.552+0000 7fa421ccb640 10 monclient: get_auth_request con 0x7fa400036820 auth_method 0 2024-04-26T00:41:51.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.556+0000 7fa421ccb640 1 --2- 172.21.15.37:0/3332302340 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa400036820 0x7fa400038ce0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fa40c00c850 tx=0x7fa40c004670 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:51.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.556+0000 7fa412ffd640 1 -- 172.21.15.37:0/3332302340 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa418016350 con 0x7fa4080039c0 2024-04-26T00:41:51.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.556+0000 7fa412ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:51.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.556+0000 7fa412ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:41:51.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.796+0000 7fa424730640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:41:51.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.796+0000 7fa424730640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:41:51.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.796+0000 7fa424730640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:41:51.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.796+0000 7fa424730640 1 -- 172.21.15.37:0/3332302340 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa404005650 con 0x7fa4080039c0 2024-04-26T00:41:51.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.800+0000 7fa412ffd640 1 -- 172.21.15.37:0/3332302340 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7fa418047ef0 con 0x7fa4080039c0 2024-04-26T00:41:51.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.800+0000 7fa412ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:41:51.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.800+0000 7fa412ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:41:51.802 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":158,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:41:51.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.800+0000 7fa424730640 1 -- 172.21.15.37:0/3332302340 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa400036820 msgr2=0x7fa400038ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:51.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.800+0000 7fa424730640 1 --2- 172.21.15.37:0/3332302340 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa400036820 0x7fa400038ce0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fa40c00c850 tx=0x7fa40c004670 comp rx=0 tx=0).stop 2024-04-26T00:41:51.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.800+0000 7fa424730640 10 monclient: shutdown 2024-04-26T00:41:51.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.800+0000 7fa424730640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:41:51.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.800+0000 7fa424730640 1 -- 172.21.15.37:0/3332302340 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa4080039c0 msgr2=0x7fa41c1c9720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:51.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.800+0000 7fa424730640 1 --2- 172.21.15.37:0/3332302340 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa4080039c0 0x7fa41c1c9720 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa41800bb50 tx=0x7fa418005480 comp rx=0 tx=0).stop 2024-04-26T00:41:51.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.800+0000 7fa424730640 1 -- 172.21.15.37:0/3332302340 shutdown_connections 2024-04-26T00:41:51.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.800+0000 7fa424730640 1 --2- 172.21.15.37:0/3332302340 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa400036820 0x7fa400038ce0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:51.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.800+0000 7fa424730640 1 --2- 172.21.15.37:0/3332302340 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa408005220 0x7fa41c132270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:51.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.800+0000 7fa424730640 1 --2- 172.21.15.37:0/3332302340 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa408004810 0x7fa41c131d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:51.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.800+0000 7fa424730640 1 --2- 172.21.15.37:0/3332302340 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa4080039c0 0x7fa41c1c9720 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:51.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.800+0000 7fa424730640 1 -- 172.21.15.37:0/3332302340 >> 172.21.15.37:0/3332302340 conn(0x7fa41c0a1ad0 msgr2=0x7fa41c1bccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:41:51.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.800+0000 7fa424730640 1 -- 172.21.15.37:0/3332302340 shutdown_connections 2024-04-26T00:41:51.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.800+0000 7fa424730640 1 -- 172.21.15.37:0/3332302340 wait complete. 2024-04-26T00:41:51.822 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-26T00:41:51.822 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-04-26T00:41:51.822 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:41:51.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa990a07640 10 monclient: get_monmap_and_config 2024-04-26T00:41:51.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa990a07640 10 monclient: build_initial_monmap 2024-04-26T00:41:51.942 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa990a07640 10 monclient: monmap: 2024-04-26T00:41:51.942 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:41:51.942 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:51.942 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:41:51.944305+0000 2024-04-26T00:41:51.943 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:41:51.944305+0000 2024-04-26T00:41:51.943 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:41:51.943 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:51.943 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:41:51.943 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:41:51.943 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:41:51.943 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:41:51.943 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:41:51.943 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:51.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa990a07640 1 Processor -- start 2024-04-26T00:41:51.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa990a07640 1 -- start start 2024-04-26T00:41:51.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa990a07640 10 monclient: init 2024-04-26T00:41:51.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa990a07640 10 monclient: _reopen_session rank -1 2024-04-26T00:41:51.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa990a07640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-04-26T00:41:51.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa990a07640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa98c0a0300 addr v1:172.21.15.45:6789/0 2024-04-26T00:41:51.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa990a07640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa98c09fc50 0x7fa98c139ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:51.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa990a07640 10 monclient(hunting): picked mon.noname-b con 0x7fa98c09fc50 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:41:51.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa990a07640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa98c13a400 0x7fa98c13c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:51.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa990a07640 10 monclient(hunting): picked mon.noname-c con 0x7fa98c13a400 addr v2:172.21.15.45:3300/0 2024-04-26T00:41:51.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa990a07640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa98c09c190 con 0x7fa98c0a0300 2024-04-26T00:41:51.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa990a07640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:51.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa990a07640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fa98c0a1110 con 0x7fa98c13a400 2024-04-26T00:41:51.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa990a07640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:51.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa990a07640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa98c0a16d0 con 0x7fa98c09fc50 2024-04-26T00:41:51.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa990a07640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:51.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa990a07640 10 monclient(hunting): authenticate will time out at 2293.249959s 2024-04-26T00:41:51.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa981d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa98c09fc50 0x7fa98c139ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:51.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa98a59c640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7fa98c0a0300 legacy=0x7fa98c0a06e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:41:51.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa98ad9d640 1 -- >> v2:172.21.15.45:3300/0 conn(0x7fa98c13a400 msgr2=0x7fa98c13c7f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:41:51.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa98ad9d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa98c13a400 0x7fa98c13c7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:41:51.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa981d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa98c09fc50 0x7fa98c139ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:40518/0 (socket says 172.21.15.37:40518) 2024-04-26T00:41:51.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa981d9b640 1 -- 172.21.15.37:0/879948156 learned_addr learned my addr 172.21.15.37:0/879948156 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:41:51.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa981d9b640 10 monclient(hunting): get_auth_request con 0x7fa98c09fc50 auth_method 0 2024-04-26T00:41:51.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa981d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:51.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa981d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:51.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa981d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:51.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa981d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:51.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa981d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:51.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa981d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:51.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa981d9b640 10 monclient(hunting): handle_auth_done global_id 254282 payload 274 2024-04-26T00:41:51.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa981d9b640 1 -- 172.21.15.37:0/879948156 >> v2:172.21.15.45:3300/0 conn(0x7fa98c13a400 msgr2=0x7fa98c13c7f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:41:51.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa981d9b640 1 --2- 172.21.15.37:0/879948156 >> v2:172.21.15.45:3300/0 conn(0x7fa98c13a400 0x7fa98c13c7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:51.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa981d9b640 1 -- 172.21.15.37:0/879948156 >> v1:172.21.15.45:6789/0 conn(0x7fa98c0a0300 legacy=0x7fa98c0a06e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:41:51.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa981d9b640 10 monclient: _finish_hunting 0 2024-04-26T00:41:51.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa981d9b640 1 monclient: found mon.noname-b 2024-04-26T00:41:51.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa981d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:51.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa981d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:41:51.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.940+0000 7fa981d9b640 1 -- 172.21.15.37:0/879948156 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa98c09f2a0 con 0x7fa98c09fc50 2024-04-26T00:41:51.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.944+0000 7fa981d9b640 1 -- 172.21.15.37:0/879948156 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa98c09fc50 msgr2=0x7fa98c139ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:51.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.944+0000 7fa981d9b640 1 --2- 172.21.15.37:0/879948156 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa98c09fc50 0x7fa98c139ec0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa98c09f4d0 tx=0x7fa97401c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c30ba52cd6e6dceb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:51.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.944+0000 7fa989d9b640 1 -- 172.21.15.37:0/879948156 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9740b0070 con 0x7fa98c09fc50 2024-04-26T00:41:51.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.944+0000 7fa989d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:51.949 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.944+0000 7fa989d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:41:51.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.944+0000 7fa989d9b640 10 monclient: dump: 2024-04-26T00:41:51.950 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:51.950 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:51.950 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:51.950 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:51.950 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:51.950 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:51.950 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:51.950 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:51.950 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:51.950 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:51.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.944+0000 7fa989d9b640 10 monclient: _finish_auth 0 2024-04-26T00:41:51.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.944+0000 7fa989d9b640 10 monclient: _check_auth_tickets 2024-04-26T00:41:51.950 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.944+0000 7fa989d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:41:51.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.944+0000 7fa989d9b640 1 -- 172.21.15.37:0/879948156 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9740ab600 con 0x7fa98c09fc50 2024-04-26T00:41:51.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.944+0000 7fa990a07640 5 monclient: authenticate success, global_id 254282 2024-04-26T00:41:51.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.944+0000 7fa990a07640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:41:51.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.944+0000 7fa989d9b640 10 monclient: handle_config config(0 keys) 2024-04-26T00:41:51.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.944+0000 7fa990a07640 10 monclient: get_monmap_and_config success 2024-04-26T00:41:51.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.944+0000 7fa989d9b640 1 -- 172.21.15.37:0/879948156 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9740ab9d0 con 0x7fa98c09fc50 2024-04-26T00:41:51.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.944+0000 7fa990a07640 4 set_mon_vals no callback set 2024-04-26T00:41:51.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.944+0000 7fa989d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:51.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.944+0000 7fa989d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:41:51.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.944+0000 7fa989d9b640 10 monclient: dump: 2024-04-26T00:41:51.952 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:51.952 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:51.952 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:51.952 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:51.952 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:51.952 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:51.952 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:51.953 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:51.953 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:51.953 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:51.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.944+0000 7fa990a07640 10 monclient: shutdown 2024-04-26T00:41:51.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.944+0000 7fa990a07640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:41:51.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.944+0000 7fa990a07640 1 -- 172.21.15.37:0/879948156 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa98c09fc50 msgr2=0x7fa98c139ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:51.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.944+0000 7fa990a07640 1 --2- 172.21.15.37:0/879948156 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa98c09fc50 0x7fa98c139ec0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa98c09f4d0 tx=0x7fa97401c3e0 comp rx=0 tx=0).stop 2024-04-26T00:41:51.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.944+0000 7fa990a07640 1 -- 172.21.15.37:0/879948156 shutdown_connections 2024-04-26T00:41:51.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.944+0000 7fa990a07640 1 --2- 172.21.15.37:0/879948156 >> v2:172.21.15.45:3300/0 conn(0x7fa98c13a400 0x7fa98c13c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:51.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.944+0000 7fa990a07640 1 --2- 172.21.15.37:0/879948156 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa98c09fc50 0x7fa98c139ec0 secure :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa98c09f4d0 tx=0x7fa97401c3e0 comp rx=0 tx=0).stop 2024-04-26T00:41:51.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.944+0000 7fa990a07640 1 -- 172.21.15.37:0/879948156 >> 172.21.15.37:0/879948156 conn(0x7fa98c0a1d50 msgr2=0x7fa98c0a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:41:51.953 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.944+0000 7fa990a07640 1 -- 172.21.15.37:0/879948156 shutdown_connections 2024-04-26T00:41:51.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.944+0000 7fa990a07640 1 -- 172.21.15.37:0/879948156 wait complete. 2024-04-26T00:41:51.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa990a07640 10 monclient: build_initial_monmap 2024-04-26T00:41:51.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa990a07640 10 monclient: monmap: 2024-04-26T00:41:51.959 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:41:51.960 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:41:51.960 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:41:51.960 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:41:51.960 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:41:51.960 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:41:51.960 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:41:51.960 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:41:51.960 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:41:51.960 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:51.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa990a07640 1 Processor -- start 2024-04-26T00:41:51.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa990a07640 1 -- start start 2024-04-26T00:41:51.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa990a07640 10 monclient: init 2024-04-26T00:41:51.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa990a07640 10 monclient: _reopen_session rank -1 2024-04-26T00:41:51.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa990a07640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:41:51.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa990a07640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa98c0a0300 0x7fa98c11b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:51.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa990a07640 10 monclient(hunting): picked mon.noname-c con 0x7fa98c0a0300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:41:51.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa990a07640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa98c13a400 0x7fa98c1cd650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:51.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa990a07640 10 monclient(hunting): picked mon.noname-b con 0x7fa98c13a400 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:41:51.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa990a07640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa98c131e30 0x7fa98c1c9c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:51.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa990a07640 10 monclient(hunting): picked mon.noname-a con 0x7fa98c131e30 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:41:51.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa990a07640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:51.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa990a07640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa98c13f8e0 con 0x7fa98c131e30 2024-04-26T00:41:51.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa990a07640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:51.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa990a07640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa98c13f760 con 0x7fa98c13a400 2024-04-26T00:41:51.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa990a07640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:51.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa990a07640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa98c13fa60 con 0x7fa98c0a0300 2024-04-26T00:41:51.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa990a07640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:51.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa990a07640 10 monclient(hunting): authenticate will time out at 2293.257169s 2024-04-26T00:41:51.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa98a59c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa98c0a0300 0x7fa98c11b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:51.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa98a59c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa98c0a0300 0x7fa98c11b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:40534/0 (socket says 172.21.15.37:40534) 2024-04-26T00:41:51.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa98a59c640 1 -- 172.21.15.37:0/2376053602 learned_addr learned my addr 172.21.15.37:0/2376053602 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:41:51.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa98a59c640 10 monclient(hunting): get_auth_request con 0x7fa98c0a0300 auth_method 0 2024-04-26T00:41:51.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa98a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:51.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa98a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:51.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa98a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:51.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa98ad9d640 1 --2- 172.21.15.37:0/2376053602 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa98c131e30 0x7fa98c1c9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:51.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa981d9b640 1 -- 172.21.15.37:0/2376053602 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa98c13a400 msgr2=0x7fa98c1cd650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:41:51.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa981d9b640 1 --2- 172.21.15.37:0/2376053602 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa98c13a400 0x7fa98c1cd650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:41:51.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa98ad9d640 10 monclient(hunting): get_auth_request con 0x7fa98c131e30 auth_method 0 2024-04-26T00:41:51.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa98ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:51.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa98ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:51.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa98ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:51.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa98a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:51.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa98a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:51.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa98a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:51.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa98ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:51.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa98ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:51.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa98ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:51.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa98a59c640 10 monclient(hunting): handle_auth_done global_id 254285 payload 824 2024-04-26T00:41:51.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa98a59c640 1 -- 172.21.15.37:0/2376053602 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa98c13a400 msgr2=0x7fa98c1cd650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:41:51.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa98a59c640 1 --2- 172.21.15.37:0/2376053602 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa98c13a400 0x7fa98c1cd650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:51.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa98a59c640 1 -- 172.21.15.37:0/2376053602 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa98c131e30 msgr2=0x7fa98c1c9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:51.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa98a59c640 1 --2- 172.21.15.37:0/2376053602 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa98c131e30 0x7fa98c1c9c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:51.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa98a59c640 10 monclient: _finish_hunting 0 2024-04-26T00:41:51.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa98a59c640 1 monclient: found mon.noname-c 2024-04-26T00:41:51.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa98a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:51.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa98a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:41:51.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa98a59c640 1 -- 172.21.15.37:0/2376053602 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa98c13ac30 con 0x7fa98c0a0300 2024-04-26T00:41:51.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa98ad9d640 1 --2- 172.21.15.37:0/2376053602 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa98c131e30 0x7fa98c1c9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:41:51.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa98a59c640 1 -- 172.21.15.37:0/2376053602 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa98c0a0300 msgr2=0x7fa98c11b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:51.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.948+0000 7fa98a59c640 1 --2- 172.21.15.37:0/2376053602 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa98c0a0300 0x7fa98c11b8a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa97c00b510 tx=0x7fa97c00b9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:51.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa9837fe640 1 -- 172.21.15.37:0/2376053602 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa97c003d90 con 0x7fa98c0a0300 2024-04-26T00:41:51.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa9837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:51.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa9837fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:41:51.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa9837fe640 10 monclient: dump: 2024-04-26T00:41:51.964 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:51.965 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:51.965 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:51.965 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:51.965 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:51.965 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:51.965 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:51.965 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:51.965 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:51.965 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:51.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa9837fe640 10 monclient: _finish_auth 0 2024-04-26T00:41:51.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa9837fe640 10 monclient: _check_auth_tickets 2024-04-26T00:41:51.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa9837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:41:51.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa9837fe640 1 -- 172.21.15.37:0/2376053602 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa97c015070 con 0x7fa98c0a0300 2024-04-26T00:41:51.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa9837fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:41:51.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa9837fe640 1 -- 172.21.15.37:0/2376053602 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa97c0052b0 con 0x7fa98c0a0300 2024-04-26T00:41:51.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa9837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:51.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa9837fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:41:51.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa9837fe640 10 monclient: dump: 2024-04-26T00:41:51.966 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:51.966 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:51.966 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:51.966 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:51.966 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:51.966 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:51.966 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:51.966 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:51.966 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:51.967 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:51.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa990a07640 5 monclient: authenticate success, global_id 254285 2024-04-26T00:41:51.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa990a07640 10 monclient: _renew_subs 2024-04-26T00:41:51.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa990a07640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:41:51.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa98959a640 4 set_mon_vals no callback set 2024-04-26T00:41:51.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa990a07640 1 -- 172.21.15.37:0/2376053602 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa98c11c7c0 con 0x7fa98c0a0300 2024-04-26T00:41:51.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa990a07640 10 monclient: _renew_subs 2024-04-26T00:41:51.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa990a07640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:41:51.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa990a07640 1 -- 172.21.15.37:0/2376053602 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa98c11bf20 con 0x7fa98c0a0300 2024-04-26T00:41:51.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa990a07640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:51.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa990a07640 1 --2- 172.21.15.37:0/2376053602 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa98c132240 0x7fa98c1ca1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:51.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa990a07640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:51.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa990a07640 1 -- 172.21.15.37:0/2376053602 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa98c09f5e0 con 0x7fa98c132240 2024-04-26T00:41:51.967 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa981d9b640 1 --2- 172.21.15.37:0/2376053602 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa98c132240 0x7fa98c1ca1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:51.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa981d9b640 10 monclient: get_auth_request con 0x7fa98c132240 auth_method 0 2024-04-26T00:41:51.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa981d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:51.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa981d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:51.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa981d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:51.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa981d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:51.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa981d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:51.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa981d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:51.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa9837fe640 1 -- 172.21.15.37:0/2376053602 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa97c005490 con 0x7fa98c0a0300 2024-04-26T00:41:51.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa981d9b640 10 monclient(hunting): handle_auth_done global_id 264165 payload 824 2024-04-26T00:41:51.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa981d9b640 1 -- 172.21.15.37:0/2376053602 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa98c09e760 con 0x7fa98c132240 2024-04-26T00:41:51.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa9837fe640 1 --2- 172.21.15.37:0/2376053602 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa970035e80 0x7fa970038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:51.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa9837fe640 1 -- 172.21.15.37:0/2376053602 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(863..863 src has 705..863) ==== 3975+0+0 (secure 0 0 0) 0x7fa97c048540 con 0x7fa98c0a0300 2024-04-26T00:41:51.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa98ad9d640 1 --2- 172.21.15.37:0/2376053602 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa970035e80 0x7fa970038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:51.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa98ad9d640 10 monclient: get_auth_request con 0x7fa970035e80 auth_method 0 2024-04-26T00:41:51.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa981d9b640 1 -- 172.21.15.37:0/2376053602 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa98c132240 msgr2=0x7fa98c1ca1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:51.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa981d9b640 1 --2- 172.21.15.37:0/2376053602 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa98c132240 0x7fa98c1ca1b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa9740ad000 tx=0x7fa97401ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:51.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa9837fe640 1 -- 172.21.15.37:0/2376053602 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9740ba070 con 0x7fa98c132240 2024-04-26T00:41:51.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa9837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:51.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa9837fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:41:51.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.952+0000 7fa9837fe640 10 monclient: dump: 2024-04-26T00:41:51.970 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:51.971 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:51.971 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:51.971 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:51.971 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:51.971 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:51.971 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:51.971 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:51.971 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:51.971 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:51.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.956+0000 7fa9837fe640 1 -- 172.21.15.37:0/2376053602 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa9740ab8f0 con 0x7fa98c132240 2024-04-26T00:41:51.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.956+0000 7fa9837fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:51.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.956+0000 7fa9837fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:41:51.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.956+0000 7fa9837fe640 1 -- 172.21.15.37:0/2376053602 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa98c132240 msgr2=0x7fa98c1ca1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:51.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.956+0000 7fa9837fe640 1 --2- 172.21.15.37:0/2376053602 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa98c132240 0x7fa98c1ca1b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa9740ad000 tx=0x7fa97401ac20 comp rx=0 tx=0).stop 2024-04-26T00:41:51.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.956+0000 7fa9837fe640 1 -- 172.21.15.37:0/2376053602 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa98c132240 msgr2=0x7fa98c1ca1b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:41:51.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.956+0000 7fa9837fe640 1 --2- 172.21.15.37:0/2376053602 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa98c132240 0x7fa98c1ca1b0 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa9740ad000 tx=0x7fa97401ac20 comp rx=0 tx=0).stop 2024-04-26T00:41:51.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.956+0000 7fa98ad9d640 1 --2- 172.21.15.37:0/2376053602 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa970035e80 0x7fa970038340 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fa98c0a3120 tx=0x7fa98400d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:51.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.960+0000 7fa990a07640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:41:51.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.960+0000 7fa990a07640 1 --2- 172.21.15.37:0/2376053602 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa98c1ca6f0 0x7fa98c06de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:51.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.960+0000 7fa990a07640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:51.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.960+0000 7fa990a07640 1 -- 172.21.15.37:0/2376053602 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa98c1cab00 con 0x7fa98c1ca6f0 2024-04-26T00:41:51.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.960+0000 7fa981d9b640 1 --2- 172.21.15.37:0/2376053602 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa98c1ca6f0 0x7fa98c06de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:51.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.960+0000 7fa981d9b640 10 monclient: get_auth_request con 0x7fa98c1ca6f0 auth_method 0 2024-04-26T00:41:51.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.960+0000 7fa981d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:51.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.960+0000 7fa981d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:51.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.960+0000 7fa981d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:51.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.960+0000 7fa981d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:51.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.960+0000 7fa981d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:51.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.960+0000 7fa981d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:51.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.960+0000 7fa981d9b640 10 monclient(hunting): handle_auth_done global_id 264168 payload 824 2024-04-26T00:41:51.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.960+0000 7fa981d9b640 1 -- 172.21.15.37:0/2376053602 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa98c04f510 con 0x7fa98c1ca6f0 2024-04-26T00:41:51.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.964+0000 7fa981d9b640 1 -- 172.21.15.37:0/2376053602 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa98c1ca6f0 msgr2=0x7fa98c06de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:51.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.964+0000 7fa981d9b640 1 --2- 172.21.15.37:0/2376053602 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa98c1ca6f0 0x7fa98c06de80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa97401cc80 tx=0x7fa974019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:51.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.964+0000 7fa9837fe640 1 -- 172.21.15.37:0/2376053602 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9740ba040 con 0x7fa98c1ca6f0 2024-04-26T00:41:51.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.964+0000 7fa9837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:51.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.964+0000 7fa9837fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:41:51.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.964+0000 7fa9837fe640 10 monclient: dump: 2024-04-26T00:41:51.975 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:51.975 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:51.975 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:51.975 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:51.975 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:51.975 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:51.975 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:51.975 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:51.976 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:51.976 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:51.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.964+0000 7fa9837fe640 1 -- 172.21.15.37:0/2376053602 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7fa974003d40 con 0x7fa98c1ca6f0 2024-04-26T00:41:51.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.964+0000 7fa9837fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:41:51.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.964+0000 7fa9837fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:41:51.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.964+0000 7fa9837fe640 1 -- 172.21.15.37:0/2376053602 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa98c1ca6f0 msgr2=0x7fa98c06de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:51.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.964+0000 7fa9837fe640 1 --2- 172.21.15.37:0/2376053602 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa98c1ca6f0 0x7fa98c06de80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa97401cc80 tx=0x7fa974019180 comp rx=0 tx=0).stop 2024-04-26T00:41:51.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.964+0000 7fa9837fe640 1 -- 172.21.15.37:0/2376053602 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa98c1ca6f0 msgr2=0x7fa98c06de80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:41:51.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.964+0000 7fa9837fe640 1 --2- 172.21.15.37:0/2376053602 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa98c1ca6f0 0x7fa98c06de80 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa97401cc80 tx=0x7fa974019180 comp rx=0 tx=0).stop 2024-04-26T00:41:51.976 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:41:51.976 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:41:51.977 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:41:51.977 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:41:51.977 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 158, 2024-04-26T00:41:51.977 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 34255, 2024-04-26T00:41:51.977 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:41:51.977 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:41:51.977 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:41:51.977 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:51.977 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:41:51.977 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:41:51.977 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:41:51.977 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:41:51.977 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:41:51.977 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:41:51.978 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:41:51.978 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:41:51.978 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:41:51.978 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:41:51.978 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:41:51.978 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:41:51.978 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:41:51.978 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:41:51.978 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:41:51.978 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:51.978 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:41:51.978 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:41:51.978 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:41:51.979 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:41:51.979 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:41:51.979 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:41:51.979 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:41:51.979 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:41:51.979 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:41:51.979 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:41:51.979 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:41:51.979 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:41:51.979 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:41:51.979 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:51.979 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:51.979 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:41:51.979 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:41:51.980 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:41:51.980 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:41:51.980 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:41:51.980 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:41:51.980 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:41:51.980 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:41:51.980 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:41:51.980 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:41:51.980 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:41:51.980 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:41:51.980 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:41:51.980 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:41:51.980 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:41:51.981 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:41:51.981 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:41:51.981 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:41:51.981 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:41:51.981 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:41:51.981 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:41:51.981 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:41:51.981 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:41:51.981 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:41:51.981 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:41:51.981 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:41:51.981 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:41:51.981 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:41:51.981 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:51.981 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:41:51.982 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:51.982 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:41:51.982 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:51.982 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:41:51.982 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:41:51.982 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:41:51.982 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:41:51.982 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:51.982 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:41:51.982 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:41:51.982 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:51.982 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:51.982 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:51.983 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:41:51.983 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:41:51.983 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:51.983 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:51.983 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:51.983 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:51.983 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:41:51.983 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:41:51.983 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:41:51.983 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:41:51.983 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:41:51.983 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:51.983 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:51.983 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:41:51.984 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:41:51.984 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:41:51.984 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:41:51.984 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:51.984 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:41:51.984 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:41:51.984 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:51.984 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:51.984 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:51.984 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:41:51.984 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:41:51.984 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:51.984 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:51.984 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:51.984 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:51.985 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:41:51.985 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:41:51.985 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:41:51.985 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:41:51.985 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:41:51.985 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:51.985 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:51.985 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:41:51.985 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:41:51.985 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:41:51.985 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:41:51.985 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:51.985 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:41:51.985 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:41:51.986 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:51.986 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:51.986 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:51.986 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:41:51.986 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:41:51.986 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:51.986 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:51.986 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:51.986 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:51.986 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:41:51.986 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:41:51.986 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:41:51.986 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:41:51.987 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:41:51.987 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:51.987 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:51.987 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:51.987 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:41:51.987 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:41:51.987 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:51.987 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:41:51.987 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:41:51.987 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:41:51.987 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:51.987 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:51.987 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:41:51.988 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:51.988 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:41:51.988 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:41:51.988 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:41:51.988 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:51.988 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:51.988 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:41:51.988 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:51.988 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:41:51.988 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:41:51.988 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:41:51.988 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:51.989 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:51.989 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:41:51.989 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:51.989 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:41:51.989 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:41:51.989 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:41:51.989 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:51.989 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:51.989 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:51.989 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:41:51.989 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:41:51.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.964+0000 7fa990a07640 1 -- 172.21.15.37:0/2376053602 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa970035e80 msgr2=0x7fa970038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:51.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.964+0000 7fa990a07640 1 --2- 172.21.15.37:0/2376053602 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa970035e80 0x7fa970038340 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fa98c0a3120 tx=0x7fa98400d040 comp rx=0 tx=0).stop 2024-04-26T00:41:51.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.964+0000 7fa990a07640 10 monclient: shutdown 2024-04-26T00:41:51.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.964+0000 7fa990a07640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:41:51.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.964+0000 7fa990a07640 1 -- 172.21.15.37:0/2376053602 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa98c0a0300 msgr2=0x7fa98c11b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:51.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.964+0000 7fa990a07640 1 --2- 172.21.15.37:0/2376053602 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa98c0a0300 0x7fa98c11b8a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa97c00b510 tx=0x7fa97c00b9d0 comp rx=0 tx=0).stop 2024-04-26T00:41:51.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.964+0000 7fa98ad9d640 1 -- 172.21.15.37:0/2376053602 reap_dead start 2024-04-26T00:41:51.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.964+0000 7fa98ad9d640 1 -- 172.21.15.37:0/2376053602 reap_dead start 2024-04-26T00:41:51.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.964+0000 7fa990a07640 1 -- 172.21.15.37:0/2376053602 shutdown_connections 2024-04-26T00:41:51.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.964+0000 7fa990a07640 1 -- 172.21.15.37:0/2376053602 >> 172.21.15.37:0/2376053602 conn(0x7fa98c0a1d50 msgr2=0x7fa98c134c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:41:51.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.964+0000 7fa990a07640 1 -- 172.21.15.37:0/2376053602 shutdown_connections 2024-04-26T00:41:51.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:51.964+0000 7fa990a07640 1 -- 172.21.15.37:0/2376053602 wait complete. 2024-04-26T00:41:51.992 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:41:51.992 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-04-26T00:41:51.992 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:41:52.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.104+0000 7f4a2707f640 10 monclient: get_monmap_and_config 2024-04-26T00:41:52.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.104+0000 7f4a2707f640 10 monclient: build_initial_monmap 2024-04-26T00:41:52.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.104+0000 7f4a2707f640 10 monclient: monmap: 2024-04-26T00:41:52.109 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:41:52.110 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:52.110 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:41:52.111658+0000 2024-04-26T00:41:52.110 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:41:52.111658+0000 2024-04-26T00:41:52.110 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:41:52.110 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:52.110 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:41:52.110 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:41:52.110 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:41:52.110 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:41:52.110 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:41:52.110 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:52.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.108+0000 7f4a2707f640 1 Processor -- start 2024-04-26T00:41:52.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.108+0000 7f4a2707f640 1 -- start start 2024-04-26T00:41:52.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.108+0000 7f4a2707f640 10 monclient: init 2024-04-26T00:41:52.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.108+0000 7f4a2707f640 10 monclient: _reopen_session rank -1 2024-04-26T00:41:52.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.108+0000 7f4a2707f640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-04-26T00:41:52.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.108+0000 7f4a2707f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4a20136f20 addr v1:172.21.15.37:6789/0 2024-04-26T00:41:52.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.108+0000 7f4a2707f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4a20137a10 addr v1:172.21.15.45:6789/0 2024-04-26T00:41:52.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.108+0000 7f4a2707f640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4a2009bff0 0x7f4a2013fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:52.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.108+0000 7f4a2707f640 10 monclient(hunting): picked mon.noname-c con 0x7f4a2009bff0 addr v2:172.21.15.45:3300/0 2024-04-26T00:41:52.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.108+0000 7f4a2707f640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a200a50f0 con 0x7f4a20136f20 2024-04-26T00:41:52.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.108+0000 7f4a2707f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a2009c4a0 con 0x7f4a20137a10 2024-04-26T00:41:52.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.108+0000 7f4a2707f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:52.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.108+0000 7f4a2707f640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f4a2009b330 con 0x7f4a2009bff0 2024-04-26T00:41:52.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.108+0000 7f4a2707f640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:52.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.108+0000 7f4a2707f640 10 monclient(hunting): authenticate will time out at 2293.417157s 2024-04-26T00:41:52.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.108+0000 7f4a1ffff640 1 -- >> v1:172.21.15.45:6789/0 conn(0x7f4a20137a10 legacy=0x7f4a2013c450 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.45:6789/0 2024-04-26T00:41:52.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.108+0000 7f4a2561c640 1 -- >> v2:172.21.15.45:3300/0 conn(0x7f4a2009bff0 msgr2=0x7f4a2013fb80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:41:52.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.108+0000 7f4a2561c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f4a2009bff0 0x7f4a2013fb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:41:52.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.108+0000 7f4a24e1b640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f4a20136f20 0x7f4a20137300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:53880/0 (socket says 172.21.15.37:53880) 2024-04-26T00:41:52.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.108+0000 7f4a24e1b640 1 -- 172.21.15.37:0/3649442863 learned_addr learned my addr 172.21.15.37:0/3649442863 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:41:52.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.108+0000 7f4a24e1b640 1 -- 172.21.15.37:0/3649442863 >> v1:172.21.15.37:6789/0 conn(0x7f4a20136f20 legacy=0x7f4a20137300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:52.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.108+0000 7f4a1f7fe640 1 -- 172.21.15.37:0/3649442863 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1947895866 0 0) 0x7f4a200a50f0 con 0x7f4a20136f20 2024-04-26T00:41:52.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.108+0000 7f4a1f7fe640 10 monclient(hunting): _negotiate 2024-04-26T00:41:52.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.108+0000 7f4a1f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:52.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.108+0000 7f4a1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:52.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.108+0000 7f4a1f7fe640 10 monclient(hunting): my global_id is 264171 2024-04-26T00:41:52.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.108+0000 7f4a1f7fe640 1 -- 172.21.15.37:0/3649442863 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a0c003530 con 0x7f4a20136f20 2024-04-26T00:41:52.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a1f7fe640 1 -- 172.21.15.37:0/3649442863 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2845253630 0 0) 0x7f4a0c003530 con 0x7f4a20136f20 2024-04-26T00:41:52.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a1f7fe640 1 -- 172.21.15.37:0/3649442863 >> v2:172.21.15.45:3300/0 conn(0x7f4a2009bff0 msgr2=0x7f4a2013fb80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:41:52.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a1f7fe640 1 --2- 172.21.15.37:0/3649442863 >> v2:172.21.15.45:3300/0 conn(0x7f4a2009bff0 0x7f4a2013fb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:52.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a1f7fe640 1 -- 172.21.15.37:0/3649442863 >> v1:172.21.15.45:6789/0 conn(0x7f4a20137a10 legacy=0x7f4a2013c450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:41:52.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a1f7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:41:52.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a1f7fe640 1 monclient: found mon.noname-a-legacy 2024-04-26T00:41:52.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a1f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:52.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a1f7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.37:6789/0 2024-04-26T00:41:52.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a1f7fe640 1 -- 172.21.15.37:0/3649442863 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a2009b610 con 0x7f4a20136f20 2024-04-26T00:41:52.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a1f7fe640 10 monclient: _finish_auth 0 2024-04-26T00:41:52.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a1f7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:41:52.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:41:52.117 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a1f7fe640 1 -- 172.21.15.37:0/3649442863 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 866326386 0 0) 0x7f4a18001d70 con 0x7f4a20136f20 2024-04-26T00:41:52.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a1f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:52.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a1f7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-26T00:41:52.118 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a1f7fe640 10 monclient: dump: 2024-04-26T00:41:52.118 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:52.118 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:52.118 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:52.118 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:52.118 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:52.118 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:52.118 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:52.119 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:52.119 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:52.119 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:52.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a1f7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] but i'm connected to v1:172.21.15.37:6789/0, reconnecting 2024-04-26T00:41:52.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a1f7fe640 10 monclient: _reopen_session rank -1 2024-04-26T00:41:52.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a1f7fe640 1 -- 172.21.15.37:0/3649442863 >> v1:172.21.15.37:6789/0 conn(0x7f4a20136f20 legacy=0x7f4a20137300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:52.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a1f7fe640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:41:52.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a1f7fe640 1 --2- 172.21.15.37:0/3649442863 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a0c0039c0 0x7f4a0c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:52.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a1f7fe640 10 monclient(hunting): picked mon.c con 0x7f4a0c0039c0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:41:52.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a1f7fe640 1 --2- 172.21.15.37:0/3649442863 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a0c004810 0x7f4a0c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:52.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a1f7fe640 10 monclient(hunting): picked mon.b con 0x7f4a0c004810 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:41:52.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a1f7fe640 1 --2- 172.21.15.37:0/3649442863 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4a0c0052f0 0x7f4a0c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:52.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a1f7fe640 10 monclient(hunting): picked mon.a con 0x7f4a0c0052f0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:41:52.119 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a1f7fe640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:52.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a1f7fe640 1 -- 172.21.15.37:0/3649442863 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f4a2009b330 con 0x7f4a0c0052f0 2024-04-26T00:41:52.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a1f7fe640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:52.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a1f7fe640 1 -- 172.21.15.37:0/3649442863 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0c00c240 con 0x7f4a0c004810 2024-04-26T00:41:52.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a1f7fe640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:52.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a1f7fe640 1 -- 172.21.15.37:0/3649442863 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4a0c00c440 con 0x7f4a0c0039c0 2024-04-26T00:41:52.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a1f7fe640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:52.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a1f7fe640 10 monclient(hunting): _finish_auth 0 2024-04-26T00:41:52.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a24e1b640 1 --2- 172.21.15.37:0/3649442863 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a0c0039c0 0x7f4a0c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:52.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a2561c640 1 --2- 172.21.15.37:0/3649442863 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4a0c0052f0 0x7f4a0c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:52.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a1ffff640 1 -- 172.21.15.37:0/3649442863 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a0c004810 msgr2=0x7f4a0c004c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:41:52.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a2561c640 10 monclient(hunting): get_auth_request con 0x7f4a0c0052f0 auth_method 0 2024-04-26T00:41:52.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a2561c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:52.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a2561c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:52.120 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a2561c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:41:52.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a1ffff640 1 --2- 172.21.15.37:0/3649442863 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a0c004810 0x7f4a0c004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:41:52.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a24e1b640 10 monclient(hunting): get_auth_request con 0x7f4a0c0039c0 auth_method 0 2024-04-26T00:41:52.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a24e1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:52.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a24e1b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:52.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a24e1b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:41:52.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a2561c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:52.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a2561c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:52.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a2561c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:41:52.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a24e1b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:52.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a24e1b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:52.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a24e1b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:41:52.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a2561c640 10 monclient(hunting): handle_auth_done global_id 264171 payload 293 2024-04-26T00:41:52.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a2561c640 1 -- 172.21.15.37:0/3649442863 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a0c0039c0 msgr2=0x7f4a0c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:41:52.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a2561c640 1 --2- 172.21.15.37:0/3649442863 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a0c0039c0 0x7f4a0c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:52.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a2561c640 1 -- 172.21.15.37:0/3649442863 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a0c004810 msgr2=0x7f4a0c004c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:41:52.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a2561c640 1 --2- 172.21.15.37:0/3649442863 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a0c004810 0x7f4a0c004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:52.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a2561c640 10 monclient: _finish_hunting 0 2024-04-26T00:41:52.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a2561c640 1 monclient: found mon.a 2024-04-26T00:41:52.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a2561c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:52.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a2561c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:41:52.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a2561c640 1 -- 172.21.15.37:0/3649442863 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a0c003530 con 0x7f4a0c0052f0 2024-04-26T00:41:52.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a2561c640 10 monclient: _finish_auth 0 2024-04-26T00:41:52.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a2561c640 10 monclient: _check_auth_tickets 2024-04-26T00:41:52.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a2561c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:41:52.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a2707f640 5 monclient: authenticate success, global_id 264171 2024-04-26T00:41:52.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a2707f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:41:52.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a24e1b640 1 --2- 172.21.15.37:0/3649442863 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a0c0039c0 0x7f4a0c003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:41:52.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.112+0000 7f4a2561c640 1 -- 172.21.15.37:0/3649442863 reap_dead start 2024-04-26T00:41:52.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a2561c640 1 -- 172.21.15.37:0/3649442863 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4a0c0052f0 msgr2=0x7f4a0c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:52.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a2561c640 1 --2- 172.21.15.37:0/3649442863 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4a0c0052f0 0x7f4a0c00b9d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4a10006fd0 tx=0x7f4a1001c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c46e891c58205012 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:52.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a2561c640 1 --2- 172.21.15.37:0/3649442863 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4a0c0052f0 0x7f4a0c00b9d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4a10006fd0 tx=0x7f4a1001c320 comp rx=0 tx=0).queue_received will delay after 1.71409e+09 on 0x7f4a10009470 mon_map magic: 0 2024-04-26T00:41:52.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a1f7fe640 1 -- 172.21.15.37:0/3649442863 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a1800a840 con 0x7f4a0c0052f0 2024-04-26T00:41:52.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a1f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:52.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a1f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:41:52.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a1f7fe640 10 monclient: dump: 2024-04-26T00:41:52.130 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:52.130 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:52.130 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:52.130 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:52.130 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:52.130 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:52.130 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:52.131 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:52.131 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:52.131 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:52.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a2707f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:41:52.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a1f7fe640 1 -- 172.21.15.37:0/3649442863 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a10003af0 con 0x7f4a0c0052f0 2024-04-26T00:41:52.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a1f7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:41:52.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a2707f640 10 monclient: get_monmap_and_config success 2024-04-26T00:41:52.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a2707f640 4 set_mon_vals no callback set 2024-04-26T00:41:52.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a2707f640 10 monclient: shutdown 2024-04-26T00:41:52.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a2707f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:41:52.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a2707f640 1 -- 172.21.15.37:0/3649442863 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4a0c0052f0 msgr2=0x7f4a0c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:52.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a2707f640 1 --2- 172.21.15.37:0/3649442863 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4a0c0052f0 0x7f4a0c00b9d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4a10006fd0 tx=0x7f4a1001c320 comp rx=0 tx=0).stop 2024-04-26T00:41:52.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a1f7fe640 1 -- 172.21.15.37:0/3649442863 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 0 0 0) 0x7f4a10009470 con 0x7f4a0c0052f0 2024-04-26T00:41:52.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a1f7fe640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-04-26T00:41:52.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a2707f640 1 -- 172.21.15.37:0/3649442863 shutdown_connections 2024-04-26T00:41:52.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a2707f640 1 --2- 172.21.15.37:0/3649442863 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4a0c0052f0 0x7f4a0c00b9d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:52.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a2707f640 1 -- 172.21.15.37:0/3649442863 >> 172.21.15.37:0/3649442863 conn(0x7f4a200a61b0 msgr2=0x7f4a200a65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:41:52.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a2707f640 1 -- 172.21.15.37:0/3649442863 shutdown_connections 2024-04-26T00:41:52.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a2707f640 1 -- 172.21.15.37:0/3649442863 wait complete. 2024-04-26T00:41:52.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a2707f640 10 monclient: build_initial_monmap 2024-04-26T00:41:52.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a2707f640 10 monclient: monmap: 2024-04-26T00:41:52.132 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:41:52.132 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:41:52.132 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:41:52.132 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:41:52.132 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:41:52.132 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:41:52.133 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:41:52.133 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:41:52.133 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:41:52.133 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:52.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a2707f640 1 Processor -- start 2024-04-26T00:41:52.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a2707f640 1 -- start start 2024-04-26T00:41:52.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a2707f640 10 monclient: init 2024-04-26T00:41:52.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a2707f640 10 monclient: _reopen_session rank -1 2024-04-26T00:41:52.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a2707f640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:41:52.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a2707f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4a0c0052f0 0x7f4a201cc070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:52.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a2707f640 10 monclient(hunting): picked mon.noname-a con 0x7f4a0c0052f0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:41:52.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a2707f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a20122570 0x7f4a201cc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:52.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a2707f640 10 monclient(hunting): picked mon.noname-c con 0x7f4a20122570 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:41:52.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a2707f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a200a2240 0x7f4a200a26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:52.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a2707f640 10 monclient(hunting): picked mon.noname-b con 0x7f4a200a2240 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:41:52.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a2707f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:52.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a2707f640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f4a2009aae0 con 0x7f4a0c0052f0 2024-04-26T00:41:52.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a2707f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:52.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a2707f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4a2009b950 con 0x7f4a200a2240 2024-04-26T00:41:52.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a2707f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:52.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a2707f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4a2009ac60 con 0x7f4a20122570 2024-04-26T00:41:52.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a2707f640 10 monclient(hunting): _renew_subs 2024-04-26T00:41:52.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a2707f640 10 monclient(hunting): authenticate will time out at 2293.427443s 2024-04-26T00:41:52.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a24e1b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4a0c0052f0 0x7f4a201cc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:52.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a1ffff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a20122570 0x7f4a201cc5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:52.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a24e1b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4a0c0052f0 0x7f4a201cc070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:35644/0 (socket says 172.21.15.37:35644) 2024-04-26T00:41:52.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a24e1b640 1 -- 172.21.15.37:0/1612770951 learned_addr learned my addr 172.21.15.37:0/1612770951 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:41:52.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a1ffff640 10 monclient(hunting): get_auth_request con 0x7f4a20122570 auth_method 0 2024-04-26T00:41:52.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:52.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a1ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:52.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:52.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a24e1b640 10 monclient(hunting): get_auth_request con 0x7f4a0c0052f0 auth_method 0 2024-04-26T00:41:52.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a24e1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:52.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a24e1b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:52.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a24e1b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:52.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a2561c640 1 -- 172.21.15.37:0/1612770951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a200a2240 msgr2=0x7f4a200a26b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:3300/0 2024-04-26T00:41:52.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a2561c640 1 --2- 172.21.15.37:0/1612770951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a200a2240 0x7f4a200a26b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:41:52.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a1ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:52.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a1ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:52.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a1ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:52.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a24e1b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:52.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a24e1b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:52.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a24e1b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:52.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a1ffff640 10 monclient(hunting): handle_auth_done global_id 254288 payload 824 2024-04-26T00:41:52.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a1ffff640 1 -- 172.21.15.37:0/1612770951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a200a2240 msgr2=0x7f4a200a26b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:41:52.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a1ffff640 1 --2- 172.21.15.37:0/1612770951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4a200a2240 0x7f4a200a26b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:52.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a1ffff640 1 -- 172.21.15.37:0/1612770951 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4a0c0052f0 msgr2=0x7f4a201cc070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:52.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a1ffff640 1 --2- 172.21.15.37:0/1612770951 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4a0c0052f0 0x7f4a201cc070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:41:52.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a1ffff640 10 monclient: _finish_hunting 0 2024-04-26T00:41:52.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a1ffff640 1 monclient: found mon.noname-c 2024-04-26T00:41:52.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:41:52.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a1ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:41:52.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a1ffff640 1 -- 172.21.15.37:0/1612770951 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a20122150 con 0x7f4a20122570 2024-04-26T00:41:52.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.116+0000 7f4a24e1b640 1 --2- 172.21.15.37:0/1612770951 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4a0c0052f0 0x7f4a201cc070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:41:52.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.120+0000 7f4a1ffff640 1 -- 172.21.15.37:0/1612770951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a20122570 msgr2=0x7f4a201cc5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:52.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.120+0000 7f4a1ffff640 1 --2- 172.21.15.37:0/1612770951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a20122570 0x7f4a201cc5b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4a0800b420 tx=0x7f4a0800b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:52.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.120+0000 7f4a1d7fa640 1 -- 172.21.15.37:0/1612770951 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a08005070 con 0x7f4a20122570 2024-04-26T00:41:52.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.120+0000 7f4a1d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:52.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.120+0000 7f4a1d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:41:52.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.120+0000 7f4a1d7fa640 10 monclient: dump: 2024-04-26T00:41:52.137 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:52.137 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:52.138 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:52.138 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:52.138 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:52.138 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:52.138 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:52.138 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:52.138 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:52.138 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:52.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.120+0000 7f4a1d7fa640 10 monclient: _finish_auth 0 2024-04-26T00:41:52.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.120+0000 7f4a1d7fa640 10 monclient: _check_auth_tickets 2024-04-26T00:41:52.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.120+0000 7f4a1d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:41:52.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.120+0000 7f4a1d7fa640 1 -- 172.21.15.37:0/1612770951 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a08005210 con 0x7f4a20122570 2024-04-26T00:41:52.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.120+0000 7f4a1d7fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:41:52.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.120+0000 7f4a1d7fa640 1 -- 172.21.15.37:0/1612770951 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a08005600 con 0x7f4a20122570 2024-04-26T00:41:52.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.120+0000 7f4a2707f640 5 monclient: authenticate success, global_id 254288 2024-04-26T00:41:52.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.120+0000 7f4a2707f640 10 monclient: _renew_subs 2024-04-26T00:41:52.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.120+0000 7f4a2707f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:41:52.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.120+0000 7f4a2707f640 1 -- 172.21.15.37:0/1612770951 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a2009ae30 con 0x7f4a20122570 2024-04-26T00:41:52.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.120+0000 7f4a1effd640 4 set_mon_vals no callback set 2024-04-26T00:41:52.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.120+0000 7f4a1d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:52.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.120+0000 7f4a1d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:41:52.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.120+0000 7f4a1d7fa640 10 monclient: dump: 2024-04-26T00:41:52.139 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:52.139 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:52.139 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:52.139 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:52.139 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:52.139 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:52.140 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:52.140 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:52.140 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:52.140 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:52.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.120+0000 7f4a2707f640 10 monclient: _renew_subs 2024-04-26T00:41:52.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.120+0000 7f4a2707f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:41:52.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.120+0000 7f4a2707f640 1 -- 172.21.15.37:0/1612770951 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a2011fe60 con 0x7f4a20122570 2024-04-26T00:41:52.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.120+0000 7f4a2707f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:52.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.120+0000 7f4a2707f640 1 --2- 172.21.15.37:0/1612770951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a200a2bf0 0x7f4a200a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:52.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.120+0000 7f4a2707f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:52.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.120+0000 7f4a2707f640 1 -- 172.21.15.37:0/1612770951 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4a2004f510 con 0x7f4a200a2bf0 2024-04-26T00:41:52.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.120+0000 7f4a24e1b640 1 --2- 172.21.15.37:0/1612770951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a200a2bf0 0x7f4a200a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:52.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.120+0000 7f4a1d7fa640 1 -- 172.21.15.37:0/1612770951 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f4a0801a400 con 0x7f4a20122570 2024-04-26T00:41:52.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.120+0000 7f4a24e1b640 10 monclient: get_auth_request con 0x7f4a200a2bf0 auth_method 0 2024-04-26T00:41:52.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.120+0000 7f4a24e1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:52.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.120+0000 7f4a24e1b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:52.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.120+0000 7f4a24e1b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:52.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.120+0000 7f4a24e1b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:52.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.120+0000 7f4a24e1b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:52.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.120+0000 7f4a24e1b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:52.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.120+0000 7f4a1d7fa640 1 --2- 172.21.15.37:0/1612770951 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4a00035e80 0x7f4a00038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:52.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.120+0000 7f4a24e1b640 10 monclient(hunting): handle_auth_done global_id 254291 payload 824 2024-04-26T00:41:52.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.120+0000 7f4a24e1b640 1 -- 172.21.15.37:0/1612770951 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4a20121f30 con 0x7f4a200a2bf0 2024-04-26T00:41:52.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.120+0000 7f4a1d7fa640 1 -- 172.21.15.37:0/1612770951 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(863..863 src has 705..863) ==== 3975+0+0 (secure 0 0 0) 0x7f4a08048660 con 0x7f4a20122570 2024-04-26T00:41:52.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.124+0000 7f4a2561c640 1 --2- 172.21.15.37:0/1612770951 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4a00035e80 0x7f4a00038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:52.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.124+0000 7f4a2561c640 10 monclient: get_auth_request con 0x7f4a00035e80 auth_method 0 2024-04-26T00:41:52.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.124+0000 7f4a24e1b640 1 -- 172.21.15.37:0/1612770951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a200a2bf0 msgr2=0x7f4a200a2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:52.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.124+0000 7f4a24e1b640 1 --2- 172.21.15.37:0/1612770951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a200a2bf0 0x7f4a200a2fd0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4a18001470 tx=0x7f4a18015db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:52.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.124+0000 7f4a1d7fa640 1 -- 172.21.15.37:0/1612770951 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a1801e3f0 con 0x7f4a200a2bf0 2024-04-26T00:41:52.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.124+0000 7f4a1d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:52.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.124+0000 7f4a1d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:41:52.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.124+0000 7f4a1d7fa640 10 monclient: dump: 2024-04-26T00:41:52.142 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:52.142 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:52.142 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:52.142 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:52.142 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:52.142 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:52.142 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:52.142 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:52.142 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:52.142 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:52.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.124+0000 7f4a1d7fa640 1 -- 172.21.15.37:0/1612770951 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4a1801e3f0 con 0x7f4a200a2bf0 2024-04-26T00:41:52.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.124+0000 7f4a1d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:41:52.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.124+0000 7f4a1d7fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:41:52.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.124+0000 7f4a1d7fa640 1 -- 172.21.15.37:0/1612770951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a200a2bf0 msgr2=0x7f4a200a2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:52.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.124+0000 7f4a1d7fa640 1 --2- 172.21.15.37:0/1612770951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a200a2bf0 0x7f4a200a2fd0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4a18001470 tx=0x7f4a18015db0 comp rx=0 tx=0).stop 2024-04-26T00:41:52.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.124+0000 7f4a1d7fa640 1 -- 172.21.15.37:0/1612770951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a200a2bf0 msgr2=0x7f4a200a2fd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:41:52.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.124+0000 7f4a1d7fa640 1 --2- 172.21.15.37:0/1612770951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a200a2bf0 0x7f4a200a2fd0 secure :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4a18001470 tx=0x7f4a18015db0 comp rx=0 tx=0).stop 2024-04-26T00:41:52.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.124+0000 7f4a2561c640 1 --2- 172.21.15.37:0/1612770951 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4a00035e80 0x7f4a00038340 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f4a2006c710 tx=0x7f4a100059c0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:52.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.132+0000 7f4a2707f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:41:52.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.132+0000 7f4a2707f640 1 --2- 172.21.15.37:0/1612770951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a2009d340 0x7f4a2009d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:41:52.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.132+0000 7f4a2707f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:41:52.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.132+0000 7f4a2707f640 1 -- 172.21.15.37:0/1612770951 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4a20075f10 con 0x7f4a2009d340 2024-04-26T00:41:52.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.132+0000 7f4a24e1b640 1 --2- 172.21.15.37:0/1612770951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a2009d340 0x7f4a2009d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:41:52.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.132+0000 7f4a24e1b640 10 monclient: get_auth_request con 0x7f4a2009d340 auth_method 0 2024-04-26T00:41:52.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.132+0000 7f4a24e1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:41:52.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.132+0000 7f4a24e1b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:41:52.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.132+0000 7f4a24e1b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:41:52.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.132+0000 7f4a24e1b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:41:52.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.132+0000 7f4a24e1b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:41:52.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.132+0000 7f4a24e1b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:41:52.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.132+0000 7f4a24e1b640 10 monclient(hunting): handle_auth_done global_id 254294 payload 824 2024-04-26T00:41:52.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.132+0000 7f4a24e1b640 1 -- 172.21.15.37:0/1612770951 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4a200a3510 con 0x7f4a2009d340 2024-04-26T00:41:52.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.132+0000 7f4a24e1b640 1 -- 172.21.15.37:0/1612770951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a2009d340 msgr2=0x7f4a2009d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:41:52.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.132+0000 7f4a24e1b640 1 --2- 172.21.15.37:0/1612770951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a2009d340 0x7f4a2009d720 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4a18005c30 tx=0x7f4a1800a350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:41:52.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.132+0000 7f4a1d7fa640 1 -- 172.21.15.37:0/1612770951 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a1801e9b0 con 0x7f4a2009d340 2024-04-26T00:41:52.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.132+0000 7f4a1d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:41:52.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.132+0000 7f4a1d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:41:52.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.132+0000 7f4a1d7fa640 10 monclient: dump: 2024-04-26T00:41:52.146 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:41:52.146 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:41:52.146 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:52.146 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:41:52.146 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:41:52.146 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:41:52.146 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:41:52.147 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:41:52.147 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:41:52.147 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:41:52.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.132+0000 7f4a1d7fa640 1 -- 172.21.15.37:0/1612770951 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4708 (secure 0 0 0) 0x7f4a18020400 con 0x7f4a2009d340 2024-04-26T00:41:52.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.132+0000 7f4a1d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:41:52.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.132+0000 7f4a1d7fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:41:52.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.132+0000 7f4a1d7fa640 1 -- 172.21.15.37:0/1612770951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a2009d340 msgr2=0x7f4a2009d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:52.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.132+0000 7f4a1d7fa640 1 --2- 172.21.15.37:0/1612770951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a2009d340 0x7f4a2009d720 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4a18005c30 tx=0x7f4a1800a350 comp rx=0 tx=0).stop 2024-04-26T00:41:52.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.132+0000 7f4a1d7fa640 1 -- 172.21.15.37:0/1612770951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a2009d340 msgr2=0x7f4a2009d720 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:41:52.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.132+0000 7f4a1d7fa640 1 --2- 172.21.15.37:0/1612770951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a2009d340 0x7f4a2009d720 secure :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4a18005c30 tx=0x7f4a1800a350 comp rx=0 tx=0).stop 2024-04-26T00:41:52.148 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:41:52.148 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:41:52.148 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:41:52.148 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:41:52.148 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 158, 2024-04-26T00:41:52.148 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 80903, 2024-04-26T00:41:52.148 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:41:52.148 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:41:52.148 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:41:52.148 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:52.148 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:41:52.148 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:41:52.149 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:41:52.149 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:41:52.149 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:41:52.149 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:41:52.149 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:41:52.149 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:41:52.149 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:41:52.149 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:41:52.149 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:41:52.149 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:41:52.149 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:41:52.150 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:41:52.150 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:41:52.150 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:52.150 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:41:52.150 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:41:52.150 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:41:52.150 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:41:52.150 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:41:52.150 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:41:52.150 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:41:52.150 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:41:52.150 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:41:52.150 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:41:52.151 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:41:52.151 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:41:52.151 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:41:52.151 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:52.151 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:52.151 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:41:52.151 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:41:52.151 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:41:52.151 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:41:52.151 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:41:52.151 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:41:52.151 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:41:52.151 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:41:52.151 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:41:52.151 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:41:52.152 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:41:52.152 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:41:52.152 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:41:52.152 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:41:52.152 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:41:52.152 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:41:52.152 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:41:52.152 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:41:52.152 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:41:52.152 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:41:52.152 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:41:52.152 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:41:52.152 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:41:52.152 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:41:52.153 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:41:52.153 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:41:52.153 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:41:52.153 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:41:52.153 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:52.153 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:41:52.153 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:52.153 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:41:52.153 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:52.153 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:41:52.153 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:41:52.153 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:41:52.153 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:41:52.153 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:52.154 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:41:52.154 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:41:52.154 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:52.154 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:52.154 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:52.154 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:41:52.154 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:41:52.154 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:52.154 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:52.154 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:52.154 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:52.154 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:41:52.154 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:41:52.154 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:41:52.155 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:41:52.155 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:41:52.155 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:52.155 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:52.155 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:41:52.155 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:41:52.155 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:41:52.155 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:41:52.155 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:52.155 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:41:52.155 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:41:52.155 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:52.155 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:52.156 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:52.156 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:41:52.156 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:41:52.156 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:52.156 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:52.156 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:52.156 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:52.156 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:41:52.156 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:41:52.156 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:41:52.156 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:41:52.156 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:41:52.156 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:52.157 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:52.157 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:41:52.157 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:41:52.157 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:41:52.157 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:41:52.157 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:52.157 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:41:52.157 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:41:52.157 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:52.157 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:52.157 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:52.157 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:41:52.157 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:41:52.157 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:41:52.158 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:52.158 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:52.158 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:52.158 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:41:52.158 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:41:52.158 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:41:52.158 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:41:52.158 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:41:52.158 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:52.158 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:52.158 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:52.158 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:41:52.158 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:41:52.158 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:52.159 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:41:52.159 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:41:52.159 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:41:52.159 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:52.159 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:52.159 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:41:52.159 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:52.159 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:41:52.159 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:41:52.159 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:41:52.159 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:52.159 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:41:52.159 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:41:52.159 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:41:52.160 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:41:52.160 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:41:52.160 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:41:52.160 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:41:52.160 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:41:52.160 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:41:52.160 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:41:52.160 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:41:52.161 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.136+0000 7f4a2707f640 1 -- 172.21.15.37:0/1612770951 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4a00035e80 msgr2=0x7f4a00038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:52.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.136+0000 7f4a2707f640 1 --2- 172.21.15.37:0/1612770951 >> v2:172.21.15.45:6812/2379918623 conn(0x7f4a00035e80 0x7f4a00038340 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f4a2006c710 tx=0x7f4a100059c0 comp rx=0 tx=0).stop 2024-04-26T00:41:52.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.136+0000 7f4a2707f640 10 monclient: shutdown 2024-04-26T00:41:52.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.136+0000 7f4a2707f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:41:52.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.136+0000 7f4a2707f640 1 -- 172.21.15.37:0/1612770951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a20122570 msgr2=0x7f4a201cc5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:41:52.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.136+0000 7f4a2707f640 1 --2- 172.21.15.37:0/1612770951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a20122570 0x7f4a201cc5b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4a0800b420 tx=0x7f4a0800b820 comp rx=0 tx=0).stop 2024-04-26T00:41:52.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.136+0000 7f4a2561c640 1 -- 172.21.15.37:0/1612770951 reap_dead start 2024-04-26T00:41:52.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.136+0000 7f4a2561c640 1 -- 172.21.15.37:0/1612770951 reap_dead start 2024-04-26T00:41:52.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.136+0000 7f4a2707f640 1 -- 172.21.15.37:0/1612770951 shutdown_connections 2024-04-26T00:41:52.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.136+0000 7f4a2707f640 1 -- 172.21.15.37:0/1612770951 >> 172.21.15.37:0/1612770951 conn(0x7f4a200a61b0 msgr2=0x7f4a200a8c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:41:52.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.136+0000 7f4a2707f640 1 -- 172.21.15.37:0/1612770951 shutdown_connections 2024-04-26T00:41:52.162 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:41:52.136+0000 7f4a2707f640 1 -- 172.21.15.37:0/1612770951 wait complete. 2024-04-26T00:41:52.163 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:41:52.163 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-04-26T00:41:52.535 INFO:tasks.rados.rados.0.smithi045.stdout:3837: done (0 left) 2024-04-26T00:41:52.535 INFO:tasks.rados.rados.0.smithi045.stdout:3838: read oid 3 snap -1 2024-04-26T00:41:52.535 INFO:tasks.rados.rados.0.smithi045.stdout:3838: expect (ObjNum 1355 snap 419 seq_num 1355) 2024-04-26T00:41:52.535 INFO:tasks.rados.rados.0.smithi045.stdout:3839: read oid 27 snap -1 2024-04-26T00:41:52.535 INFO:tasks.rados.rados.0.smithi045.stdout:3839: expect deleted 2024-04-26T00:41:52.535 INFO:tasks.rados.rados.0.smithi045.stdout:3840: read oid 41 snap -1 2024-04-26T00:41:52.535 INFO:tasks.rados.rados.0.smithi045.stdout:3840: expect (ObjNum 1321 snap 407 seq_num 1321) 2024-04-26T00:41:52.535 INFO:tasks.rados.rados.0.smithi045.stdout:3841: read oid 13 snap -1 2024-04-26T00:41:52.535 INFO:tasks.rados.rados.0.smithi045.stdout:3841: expect (ObjNum 1341 snap 415 seq_num 1341) 2024-04-26T00:41:52.536 INFO:tasks.rados.rados.0.smithi045.stdout:3842: read oid 23 snap 419 2024-04-26T00:41:52.536 INFO:tasks.rados.rados.0.smithi045.stdout:3842: expect deleted 2024-04-26T00:41:52.536 INFO:tasks.rados.rados.0.smithi045.stdout:3843: rollback oid 40 current snap is 424 2024-04-26T00:41:52.536 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 40 to 401 2024-04-26T00:41:52.536 INFO:tasks.rados.rados.0.smithi045.stdout:3844: rollback oid 26 current snap is 424 2024-04-26T00:41:52.536 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 26 to 403 2024-04-26T00:41:52.536 INFO:tasks.rados.rados.0.smithi045.stdout:3845: write oid 5 current snap is 424 2024-04-26T00:41:52.536 INFO:tasks.rados.rados.0.smithi045.stdout:3845: seq_num 1373 ranges {676011=660417,1744718=428471,2788122=287889} 2024-04-26T00:41:52.558 INFO:tasks.rados.rados.0.smithi045.stdout:3845: writing smithi04526579-5 from 676011 to 1336428 tid 1 2024-04-26T00:41:52.571 INFO:tasks.rados.rados.0.smithi045.stdout:3845: writing smithi04526579-5 from 1744718 to 2173189 tid 2 2024-04-26T00:41:52.586 INFO:tasks.rados.rados.0.smithi045.stdout:3845: writing smithi04526579-5 from 2788122 to 3076011 tid 3 2024-04-26T00:41:52.586 INFO:tasks.rados.rados.0.smithi045.stdout:3846: write oid 28 current snap is 424 2024-04-26T00:41:52.586 INFO:tasks.rados.rados.0.smithi045.stdout:3846: seq_num 1374 ranges {692818=735666,2203200=688482} 2024-04-26T00:41:52.603 INFO:tasks.rados.rados.0.smithi045.stdout:3846: writing smithi04526579-28 from 692818 to 1428484 tid 1 2024-04-26T00:41:52.618 INFO:tasks.rados.rados.0.smithi045.stdout:3846: writing smithi04526579-28 from 2203200 to 2891682 tid 2 2024-04-26T00:41:52.618 INFO:tasks.rados.rados.0.smithi045.stdout:3847: delete oid 30 current snap is 424 2024-04-26T00:41:52.618 INFO:tasks.rados.rados.0.smithi045.stdout:3840: expect (ObjNum 1321 snap 407 seq_num 1321) 2024-04-26T00:41:52.789 INFO:tasks.rados.rados.0.smithi045.stdout:3841: expect (ObjNum 1341 snap 415 seq_num 1341) 2024-04-26T00:41:52.869 INFO:tasks.rados.rados.0.smithi045.stdout:3843: finishing rollback tid 1 to smithi04526579-40 2024-04-26T00:41:52.870 INFO:tasks.rados.rados.0.smithi045.stdout:3843: finishing rollback tid 2 to smithi04526579-40 2024-04-26T00:41:52.870 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 40 v 2490 (ObjNum 1178 snap 366 seq_num 1178) dirty exists 2024-04-26T00:41:52.870 INFO:tasks.rados.rados.0.smithi045.stdout:3844: finishing rollback tid 0 to smithi04526579-26 2024-04-26T00:41:52.870 INFO:tasks.rados.rados.0.smithi045.stdout:3838: expect (ObjNum 1355 snap 419 seq_num 1355) 2024-04-26T00:41:53.001 INFO:tasks.rados.rados.0.smithi045.stdout:3844: finishing rollback tid 1 to smithi04526579-26 2024-04-26T00:41:53.001 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 26 v 2535 (ObjNum 49 snap 0 seq_num 2242494192) dirty dne 2024-04-26T00:41:53.001 INFO:tasks.rados.rados.0.smithi045.stdout:3845: finishing write tid 1 to smithi04526579-5 2024-04-26T00:41:53.001 INFO:tasks.rados.rados.0.smithi045.stdout:3845: finishing write tid 2 to smithi04526579-5 2024-04-26T00:41:53.001 INFO:tasks.rados.rados.0.smithi045.stdout:3845: finishing write tid 3 to smithi04526579-5 2024-04-26T00:41:53.001 INFO:tasks.rados.rados.0.smithi045.stdout:3845: finishing write tid 4 to smithi04526579-5 2024-04-26T00:41:53.001 INFO:tasks.rados.rados.0.smithi045.stdout:3845: finishing write tid 5 to smithi04526579-5 2024-04-26T00:41:53.001 INFO:tasks.rados.rados.0.smithi045.stdout:3845: oid 5 updating version 0 to 2907 2024-04-26T00:41:53.001 INFO:tasks.rados.rados.0.smithi045.stdout:3845: oid 5 version 2907 is already newer than 2906 2024-04-26T00:41:53.001 INFO:tasks.rados.rados.0.smithi045.stdout:3845: oid 5 version 2907 is already newer than 2904 2024-04-26T00:41:53.002 INFO:tasks.rados.rados.0.smithi045.stdout:3845: oid 5 version 2907 is already newer than 2905 2024-04-26T00:41:53.002 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 2907 (ObjNum 1373 snap 424 seq_num 1373) dirty exists 2024-04-26T00:41:53.002 INFO:tasks.rados.rados.0.smithi045.stdout:3845: left oid 5 (ObjNum 1373 snap 424 seq_num 1373) 2024-04-26T00:41:53.002 INFO:tasks.rados.rados.0.smithi045.stdout:3846: finishing write tid 1 to smithi04526579-28 2024-04-26T00:41:53.002 INFO:tasks.rados.rados.0.smithi045.stdout:3846: finishing write tid 2 to smithi04526579-28 2024-04-26T00:41:53.002 INFO:tasks.rados.rados.0.smithi045.stdout:3846: finishing write tid 3 to smithi04526579-28 2024-04-26T00:41:53.002 INFO:tasks.rados.rados.0.smithi045.stdout:3846: finishing write tid 4 to smithi04526579-28 2024-04-26T00:41:53.002 INFO:tasks.rados.rados.0.smithi045.stdout:3846: oid 28 updating version 0 to 2538 2024-04-26T00:41:53.002 INFO:tasks.rados.rados.0.smithi045.stdout:3846: oid 28 version 2538 is already newer than 2537 2024-04-26T00:41:53.002 INFO:tasks.rados.rados.0.smithi045.stdout:3846: oid 28 version 2538 is already newer than 2536 2024-04-26T00:41:53.002 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 28 v 2538 (ObjNum 1374 snap 424 seq_num 1374) dirty exists 2024-04-26T00:41:53.002 INFO:tasks.rados.rados.0.smithi045.stdout:3846: left oid 28 (ObjNum 1374 snap 424 seq_num 1374) 2024-04-26T00:41:53.002 INFO:tasks.rados.rados.0.smithi045.stdout:3838: done (9 left) 2024-04-26T00:41:53.002 INFO:tasks.rados.rados.0.smithi045.stdout:3839: done (8 left) 2024-04-26T00:41:53.003 INFO:tasks.rados.rados.0.smithi045.stdout:3840: done (7 left) 2024-04-26T00:41:53.003 INFO:tasks.rados.rados.0.smithi045.stdout:3841: done (6 left) 2024-04-26T00:41:53.003 INFO:tasks.rados.rados.0.smithi045.stdout:3842: done (5 left) 2024-04-26T00:41:53.003 INFO:tasks.rados.rados.0.smithi045.stdout:3843: done (4 left) 2024-04-26T00:41:53.003 INFO:tasks.rados.rados.0.smithi045.stdout:3844: done (3 left) 2024-04-26T00:41:53.003 INFO:tasks.rados.rados.0.smithi045.stdout:3845: done (2 left) 2024-04-26T00:41:53.003 INFO:tasks.rados.rados.0.smithi045.stdout:3846: done (1 left) 2024-04-26T00:41:53.003 INFO:tasks.rados.rados.0.smithi045.stdout:3847: done (0 left) 2024-04-26T00:41:53.003 INFO:tasks.rados.rados.0.smithi045.stdout:3848: rollback oid 35 current snap is 424 2024-04-26T00:41:53.003 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 35 to 409 2024-04-26T00:41:53.003 INFO:tasks.rados.rados.0.smithi045.stdout:3849: write (excl) oid 47 current snap is 424 2024-04-26T00:41:53.003 INFO:tasks.rados.rados.0.smithi045.stdout:3849: seq_num 1376 ranges {726432=486164,1803811=692151} 2024-04-26T00:41:53.020 INFO:tasks.rados.rados.0.smithi045.stdout:3849: writing smithi04526579-47 from 726432 to 1212596 tid 1 2024-04-26T00:41:53.041 INFO:tasks.rados.rados.0.smithi045.stdout:3849: writing smithi04526579-47 from 1803811 to 2495962 tid 2 2024-04-26T00:41:53.042 INFO:tasks.rados.rados.0.smithi045.stdout:3850: read oid 14 snap -1 2024-04-26T00:41:53.042 INFO:tasks.rados.rados.0.smithi045.stdout:3850: expect deleted 2024-04-26T00:41:53.042 INFO:tasks.rados.rados.0.smithi045.stdout:3851: snap_create 2024-04-26T00:41:53.042 INFO:tasks.rados.rados.0.smithi045.stdout:3848: finishing rollback tid 0 to smithi04526579-35 2024-04-26T00:41:53.042 INFO:tasks.rados.rados.0.smithi045.stdout:3848: finishing rollback tid 1 to smithi04526579-35 2024-04-26T00:41:53.042 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 2876 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-04-26T00:41:53.042 INFO:tasks.rados.rados.0.smithi045.stdout:3849: finishing write tid 1 to smithi04526579-47 2024-04-26T00:41:53.050 INFO:tasks.rados.rados.0.smithi045.stdout:3849: finishing write tid 2 to smithi04526579-47 2024-04-26T00:41:53.051 INFO:tasks.rados.rados.0.smithi045.stdout:3849: finishing write tid 3 to smithi04526579-47 2024-04-26T00:41:53.052 INFO:tasks.rados.rados.0.smithi045.stdout:3849: finishing write tid 4 to smithi04526579-47 2024-04-26T00:41:53.052 INFO:tasks.rados.rados.0.smithi045.stdout:3849: oid 47 updating version 0 to 2825 2024-04-26T00:41:53.052 INFO:tasks.rados.rados.0.smithi045.stdout:3849: oid 47 updating version 2825 to 2827 2024-04-26T00:41:53.052 INFO:tasks.rados.rados.0.smithi045.stdout:3849: oid 47 version 2827 is already newer than 2826 2024-04-26T00:41:53.052 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 2827 (ObjNum 1376 snap 424 seq_num 1376) dirty exists 2024-04-26T00:41:53.052 INFO:tasks.rados.rados.0.smithi045.stdout:3849: left oid 47 (ObjNum 1376 snap 424 seq_num 1376) 2024-04-26T00:41:54.110 INFO:tasks.rados.rados.0.smithi045.stdout:3848: done (3 left) 2024-04-26T00:41:54.110 INFO:tasks.rados.rados.0.smithi045.stdout:3849: done (2 left) 2024-04-26T00:41:54.110 INFO:tasks.rados.rados.0.smithi045.stdout:3850: done (1 left) 2024-04-26T00:41:54.110 INFO:tasks.rados.rados.0.smithi045.stdout:3851: done (0 left) 2024-04-26T00:41:54.111 INFO:tasks.rados.rados.0.smithi045.stdout:3852: snap_create 2024-04-26T00:41:55.121 INFO:tasks.rados.rados.0.smithi045.stdout:3852: done (0 left) 2024-04-26T00:41:55.121 INFO:tasks.rados.rados.0.smithi045.stdout:3853: read oid 21 snap -1 2024-04-26T00:41:55.121 INFO:tasks.rados.rados.0.smithi045.stdout:3853: expect (ObjNum 1360 snap 420 seq_num 1360) 2024-04-26T00:41:55.122 INFO:tasks.rados.rados.0.smithi045.stdout:3854: write oid 44 current snap is 426 2024-04-26T00:41:55.122 INFO:tasks.rados.rados.0.smithi045.stdout:3854: seq_num 1377 ranges {743239=677766,1978646=435809,2897814=245425} 2024-04-26T00:41:55.144 INFO:tasks.rados.rados.0.smithi045.stdout:3854: writing smithi04526579-44 from 743239 to 1421005 tid 1 2024-04-26T00:41:55.160 INFO:tasks.rados.rados.0.smithi045.stdout:3854: writing smithi04526579-44 from 1978646 to 2414455 tid 2 2024-04-26T00:41:55.172 INFO:tasks.rados.rados.0.smithi045.stdout:3854: writing smithi04526579-44 from 2897814 to 3143239 tid 3 2024-04-26T00:41:55.172 INFO:tasks.rados.rados.0.smithi045.stdout:3855: write (excl) oid 29 current snap is 426 2024-04-26T00:41:55.172 INFO:tasks.rados.rados.0.smithi045.stdout:3855: seq_num 1378 ranges {760046=753015,2037128=695820} 2024-04-26T00:41:55.191 INFO:tasks.rados.rados.0.smithi045.stdout:3855: writing smithi04526579-29 from 760046 to 1513061 tid 1 2024-04-26T00:41:55.204 INFO:tasks.rados.rados.0.smithi045.stdout:3855: writing smithi04526579-29 from 2037128 to 2732948 tid 2 2024-04-26T00:41:55.205 INFO:tasks.rados.rados.0.smithi045.stdout:3856: snap_create 2024-04-26T00:41:55.205 INFO:tasks.rados.rados.0.smithi045.stdout:3853: expect (ObjNum 1360 snap 420 seq_num 1360) 2024-04-26T00:41:55.434 INFO:tasks.rados.rados.0.smithi045.stdout:3854: finishing write tid 1 to smithi04526579-44 2024-04-26T00:41:55.434 INFO:tasks.rados.rados.0.smithi045.stdout:3854: finishing write tid 2 to smithi04526579-44 2024-04-26T00:41:55.434 INFO:tasks.rados.rados.0.smithi045.stdout:3854: finishing write tid 3 to smithi04526579-44 2024-04-26T00:41:55.434 INFO:tasks.rados.rados.0.smithi045.stdout:3854: finishing write tid 4 to smithi04526579-44 2024-04-26T00:41:55.434 INFO:tasks.rados.rados.0.smithi045.stdout:3854: finishing write tid 5 to smithi04526579-44 2024-04-26T00:41:55.434 INFO:tasks.rados.rados.0.smithi045.stdout:3854: oid 44 updating version 0 to 2415 2024-04-26T00:41:55.434 INFO:tasks.rados.rados.0.smithi045.stdout:3854: oid 44 updating version 2415 to 2418 2024-04-26T00:41:55.434 INFO:tasks.rados.rados.0.smithi045.stdout:3854: oid 44 version 2418 is already newer than 2416 2024-04-26T00:41:55.434 INFO:tasks.rados.rados.0.smithi045.stdout:3854: oid 44 version 2418 is already newer than 2417 2024-04-26T00:41:55.435 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 44 v 2418 (ObjNum 1377 snap 426 seq_num 1377) dirty exists 2024-04-26T00:41:55.435 INFO:tasks.rados.rados.0.smithi045.stdout:3854: left oid 44 (ObjNum 1377 snap 426 seq_num 1377) 2024-04-26T00:41:55.435 INFO:tasks.rados.rados.0.smithi045.stdout:3855: finishing write tid 1 to smithi04526579-29 2024-04-26T00:41:55.435 INFO:tasks.rados.rados.0.smithi045.stdout:3855: finishing write tid 2 to smithi04526579-29 2024-04-26T00:41:55.435 INFO:tasks.rados.rados.0.smithi045.stdout:3855: finishing write tid 3 to smithi04526579-29 2024-04-26T00:41:55.435 INFO:tasks.rados.rados.0.smithi045.stdout:3855: finishing write tid 4 to smithi04526579-29 2024-04-26T00:41:55.435 INFO:tasks.rados.rados.0.smithi045.stdout:3855: oid 29 updating version 0 to 2246 2024-04-26T00:41:55.435 INFO:tasks.rados.rados.0.smithi045.stdout:3855: oid 29 version 2246 is already newer than 2245 2024-04-26T00:41:55.435 INFO:tasks.rados.rados.0.smithi045.stdout:3855: oid 29 version 2246 is already newer than 2244 2024-04-26T00:41:55.435 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 2246 (ObjNum 1378 snap 426 seq_num 1378) dirty exists 2024-04-26T00:41:55.435 INFO:tasks.rados.rados.0.smithi045.stdout:3855: left oid 29 (ObjNum 1378 snap 426 seq_num 1378) 2024-04-26T00:41:56.127 INFO:tasks.rados.rados.0.smithi045.stdout:3853: done (3 left) 2024-04-26T00:41:56.127 INFO:tasks.rados.rados.0.smithi045.stdout:3854: done (2 left) 2024-04-26T00:41:56.127 INFO:tasks.rados.rados.0.smithi045.stdout:3855: done (1 left) 2024-04-26T00:41:56.127 INFO:tasks.rados.rados.0.smithi045.stdout:3856: done (0 left) 2024-04-26T00:41:56.127 INFO:tasks.rados.rados.0.smithi045.stdout:3857: snap_create 2024-04-26T00:41:56.267 INFO:tasks.rados.rados.0.smithi045.stdout:3857: done (0 left) 2024-04-26T00:41:56.268 INFO:tasks.rados.rados.0.smithi045.stdout:3858: copy_from oid 2 from oid 3 current snap is 428 2024-04-26T00:41:56.268 INFO:tasks.rados.rados.0.smithi045.stdout:3859: rollback oid 37 current snap is 428 2024-04-26T00:41:56.268 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 37 to 416 2024-04-26T00:41:56.268 INFO:tasks.rados.rados.0.smithi045.stdout:3860: delete oid 35 current snap is 428 2024-04-26T00:41:56.296 INFO:tasks.rados.rados.0.smithi045.stdout:3860: done (2 left) 2024-04-26T00:41:56.296 INFO:tasks.rados.rados.0.smithi045.stdout:3861: rollback oid 13 current snap is 428 2024-04-26T00:41:56.297 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 13 to 384 2024-04-26T00:41:56.297 INFO:tasks.rados.rados.0.smithi045.stdout:3862: write (excl) oid 8 current snap is 428 2024-04-26T00:41:56.297 INFO:tasks.rados.rados.0.smithi045.stdout:3862: seq_num 1380 ranges {793660=503513,2037739=583136} 2024-04-26T00:41:56.317 INFO:tasks.rados.rados.0.smithi045.stdout:3862: writing smithi04526579-8 from 793660 to 1297173 tid 1 2024-04-26T00:41:56.337 INFO:tasks.rados.rados.0.smithi045.stdout:3862: writing smithi04526579-8 from 2037739 to 2620875 tid 2 2024-04-26T00:41:56.337 INFO:tasks.rados.rados.0.smithi045.stdout:3863: delete oid 30 current snap is 428 2024-04-26T00:41:56.338 INFO:tasks.rados.rados.0.smithi045.stdout:3859: finishing rollback tid 0 to smithi04526579-37 2024-04-26T00:41:56.338 INFO:tasks.rados.rados.0.smithi045.stdout:3859: finishing rollback tid 1 to smithi04526579-37 2024-04-26T00:41:56.338 INFO:tasks.rados.rados.0.smithi045.stdout:3858: finishing copy_from racing read to smithi04526579-2 2024-04-26T00:41:56.338 INFO:tasks.rados.rados.0.smithi045.stdout:3861: finishing rollback tid 0 to smithi04526579-13 2024-04-26T00:41:56.338 INFO:tasks.rados.rados.0.smithi045.stdout:3861: finishing rollback tid 1 to smithi04526579-13 2024-04-26T00:41:56.338 INFO:tasks.rados.rados.0.smithi045.stdout:3859: finishing rollback tid 2 to smithi04526579-37 2024-04-26T00:41:56.338 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 37 v 2911 (ObjNum 1075 snap 322 seq_num 1075) dirty exists 2024-04-26T00:41:56.338 INFO:tasks.rados.rados.0.smithi045.stdout:3861: finishing rollback tid 2 to smithi04526579-13 2024-04-26T00:41:56.338 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 13 v 2249 (ObjNum 1237 snap 384 seq_num 1237) dirty exists 2024-04-26T00:41:56.338 INFO:tasks.rados.rados.0.smithi045.stdout:3858: finishing copy_from to smithi04526579-2 2024-04-26T00:41:56.338 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 2 v 2523 (ObjNum 1355 snap 419 seq_num 1355) dirty exists 2024-04-26T00:41:56.338 INFO:tasks.rados.rados.0.smithi045.stdout:3862: finishing write tid 1 to smithi04526579-8 2024-04-26T00:41:56.341 INFO:tasks.rados.rados.0.smithi045.stdout:3858: done (4 left) 2024-04-26T00:41:56.342 INFO:tasks.rados.rados.0.smithi045.stdout:3859: done (3 left) 2024-04-26T00:41:56.342 INFO:tasks.rados.rados.0.smithi045.stdout:3861: done (2 left) 2024-04-26T00:41:56.342 INFO:tasks.rados.rados.0.smithi045.stdout:3863: done (1 left) 2024-04-26T00:41:56.342 INFO:tasks.rados.rados.0.smithi045.stdout:3864: rollback oid 4 current snap is 428 2024-04-26T00:41:56.342 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 4 to 410 2024-04-26T00:41:56.342 INFO:tasks.rados.rados.0.smithi045.stdout:3865: read oid 42 snap -1 2024-04-26T00:41:56.342 INFO:tasks.rados.rados.0.smithi045.stdout:3865: expect (ObjNum 1363 snap 421 seq_num 1363) 2024-04-26T00:41:56.342 INFO:tasks.rados.rados.0.smithi045.stdout:3866: rollback oid 16 current snap is 428 2024-04-26T00:41:56.342 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 16 to 416 2024-04-26T00:41:56.342 INFO:tasks.rados.rados.0.smithi045.stdout:3867: delete oid 14 current snap is 428 2024-04-26T00:41:56.345 INFO:tasks.rados.rados.0.smithi045.stdout:3862: finishing write tid 2 to smithi04526579-8 2024-04-26T00:41:56.345 INFO:tasks.rados.rados.0.smithi045.stdout:3862: finishing write tid 3 to smithi04526579-8 2024-04-26T00:41:56.345 INFO:tasks.rados.rados.0.smithi045.stdout:3862: finishing write tid 4 to smithi04526579-8 2024-04-26T00:41:56.345 INFO:tasks.rados.rados.0.smithi045.stdout:3862: oid 8 updating version 0 to 2251 2024-04-26T00:41:56.345 INFO:tasks.rados.rados.0.smithi045.stdout:3862: oid 8 updating version 2251 to 2252 2024-04-26T00:41:56.345 INFO:tasks.rados.rados.0.smithi045.stdout:3862: oid 8 version 2252 is already newer than 2250 2024-04-26T00:41:56.345 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 2252 (ObjNum 1380 snap 428 seq_num 1380) dirty exists 2024-04-26T00:41:56.345 INFO:tasks.rados.rados.0.smithi045.stdout:3862: left oid 8 (ObjNum 1380 snap 428 seq_num 1380) 2024-04-26T00:41:56.350 INFO:tasks.rados.rados.0.smithi045.stdout:3866: finishing rollback tid 0 to smithi04526579-16 2024-04-26T00:41:56.351 INFO:tasks.rados.rados.0.smithi045.stdout:3866: finishing rollback tid 1 to smithi04526579-16 2024-04-26T00:41:56.351 INFO:tasks.rados.rados.0.smithi045.stdout:3866: finishing rollback tid 2 to smithi04526579-16 2024-04-26T00:41:56.352 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 2500 (ObjNum 1284 snap 398 seq_num 1284) dirty exists 2024-04-26T00:41:56.352 INFO:tasks.rados.rados.0.smithi045.stdout:3862: done (4 left) 2024-04-26T00:41:56.352 INFO:tasks.rados.rados.0.smithi045.stdout:3866: done (3 left) 2024-04-26T00:41:56.352 INFO:tasks.rados.rados.0.smithi045.stdout:3867: done (2 left) 2024-04-26T00:41:56.352 INFO:tasks.rados.rados.0.smithi045.stdout:3868: rollback oid 11 current snap is 428 2024-04-26T00:41:56.352 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 11 to 346 2024-04-26T00:41:56.352 INFO:tasks.rados.rados.0.smithi045.stdout:3869: write (excl) oid 33 current snap is 428 2024-04-26T00:41:56.352 INFO:tasks.rados.rados.0.smithi045.stdout:3869: seq_num 1383 ranges {444081=729260,1813185=446816,2945999=298082} 2024-04-26T00:41:56.370 INFO:tasks.rados.rados.0.smithi045.stdout:3869: writing smithi04526579-33 from 444081 to 1173341 tid 1 2024-04-26T00:41:56.383 INFO:tasks.rados.rados.0.smithi045.stdout:3869: writing smithi04526579-33 from 1813185 to 2260001 tid 2 2024-04-26T00:41:56.394 INFO:tasks.rados.rados.0.smithi045.stdout:3869: writing smithi04526579-33 from 2945999 to 3244081 tid 3 2024-04-26T00:41:56.394 INFO:tasks.rados.rados.0.smithi045.stdout:3870: copy_from oid 30 from oid 18 current snap is 428 2024-04-26T00:41:56.394 INFO:tasks.rados.rados.0.smithi045.stdout:3865: expect (ObjNum 1363 snap 421 seq_num 1363) 2024-04-26T00:41:56.616 INFO:tasks.rados.rados.0.smithi045.stdout:3864: finishing rollback tid 0 to smithi04526579-4 2024-04-26T00:41:56.616 INFO:tasks.rados.rados.0.smithi045.stdout:3864: finishing rollback tid 1 to smithi04526579-4 2024-04-26T00:41:56.616 INFO:tasks.rados.rados.0.smithi045.stdout:3864: finishing rollback tid 2 to smithi04526579-4 2024-04-26T00:41:56.616 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 4 v 2546 (ObjNum 1282 snap 398 seq_num 1282) dirty exists 2024-04-26T00:41:56.616 INFO:tasks.rados.rados.0.smithi045.stdout:3864: done (4 left) 2024-04-26T00:41:56.616 INFO:tasks.rados.rados.0.smithi045.stdout:3865: done (3 left) 2024-04-26T00:41:56.616 INFO:tasks.rados.rados.0.smithi045.stdout:3871: snap_remove snap 418 2024-04-26T00:41:57.276 INFO:tasks.rados.rados.0.smithi045.stdout:3871: done (3 left) 2024-04-26T00:41:57.276 INFO:tasks.rados.rados.0.smithi045.stdout:3872: copy_from oid 6 from oid 17 current snap is 428 2024-04-26T00:41:57.276 INFO:tasks.rados.rados.0.smithi045.stdout:3873: read oid 13 snap -1 2024-04-26T00:41:57.276 INFO:tasks.rados.rados.0.smithi045.stdout:3873: expect (ObjNum 1237 snap 384 seq_num 1237) 2024-04-26T00:41:57.277 INFO:tasks.rados.rados.0.smithi045.stdout:3874: delete oid 2 current snap is 428 2024-04-26T00:41:57.277 INFO:tasks.rados.rados.0.smithi045.stdout:3868: finishing rollback tid 0 to smithi04526579-11 2024-04-26T00:41:57.277 INFO:tasks.rados.rados.0.smithi045.stdout:3868: finishing rollback tid 1 to smithi04526579-11 2024-04-26T00:41:57.277 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 11 v 2459 (ObjNum 257 snap 0 seq_num 2242554560) dirty dne 2024-04-26T00:41:57.277 INFO:tasks.rados.rados.0.smithi045.stdout:3869: finishing write tid 1 to smithi04526579-33 2024-04-26T00:41:57.277 INFO:tasks.rados.rados.0.smithi045.stdout:3869: finishing write tid 2 to smithi04526579-33 2024-04-26T00:41:57.277 INFO:tasks.rados.rados.0.smithi045.stdout:3869: finishing write tid 3 to smithi04526579-33 2024-04-26T00:41:57.277 INFO:tasks.rados.rados.0.smithi045.stdout:3869: finishing write tid 4 to smithi04526579-33 2024-04-26T00:41:57.277 INFO:tasks.rados.rados.0.smithi045.stdout:3869: finishing write tid 5 to smithi04526579-33 2024-04-26T00:41:57.277 INFO:tasks.rados.rados.0.smithi045.stdout:3869: oid 33 updating version 0 to 2914 2024-04-26T00:41:57.277 INFO:tasks.rados.rados.0.smithi045.stdout:3869: oid 33 version 2914 is already newer than 2913 2024-04-26T00:41:57.277 INFO:tasks.rados.rados.0.smithi045.stdout:3869: oid 33 updating version 2914 to 2915 2024-04-26T00:41:57.277 INFO:tasks.rados.rados.0.smithi045.stdout:3869: oid 33 version 2915 is already newer than 2912 2024-04-26T00:41:57.277 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 2915 (ObjNum 1383 snap 428 seq_num 1383) dirty exists 2024-04-26T00:41:57.277 INFO:tasks.rados.rados.0.smithi045.stdout:3869: left oid 33 (ObjNum 1383 snap 428 seq_num 1383) 2024-04-26T00:41:57.278 INFO:tasks.rados.rados.0.smithi045.stdout:3870: finishing copy_from racing read to smithi04526579-30 2024-04-26T00:41:57.278 INFO:tasks.rados.rados.0.smithi045.stdout:3870: finishing copy_from to smithi04526579-30 2024-04-26T00:41:57.278 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 2874 (ObjNum 1363 snap 421 seq_num 1363) dirty exists 2024-04-26T00:41:57.282 INFO:tasks.rados.rados.0.smithi045.stdout:3872: finishing copy_from to smithi04526579-6 2024-04-26T00:41:57.282 INFO:tasks.rados.rados.0.smithi045.stdout:3872: got expected ENOENT (src dne) 2024-04-26T00:41:57.282 INFO:tasks.rados.rados.0.smithi045.stdout:3872: finishing copy_from racing read to smithi04526579-6 2024-04-26T00:41:57.298 INFO:tasks.rados.rados.0.smithi045.stdout:3873: expect (ObjNum 1237 snap 384 seq_num 1237) 2024-04-26T00:41:57.517 INFO:tasks.rados.rados.0.smithi045.stdout:3868: done (5 left) 2024-04-26T00:41:57.518 INFO:tasks.rados.rados.0.smithi045.stdout:3869: done (4 left) 2024-04-26T00:41:57.518 INFO:tasks.rados.rados.0.smithi045.stdout:3870: done (3 left) 2024-04-26T00:41:57.518 INFO:tasks.rados.rados.0.smithi045.stdout:3872: done (2 left) 2024-04-26T00:41:57.518 INFO:tasks.rados.rados.0.smithi045.stdout:3873: done (1 left) 2024-04-26T00:41:57.518 INFO:tasks.rados.rados.0.smithi045.stdout:3874: done (0 left) 2024-04-26T00:41:57.518 INFO:tasks.rados.rados.0.smithi045.stdout:3875: write (excl) oid 34 current snap is 428 2024-04-26T00:41:57.518 INFO:tasks.rados.rados.0.smithi045.stdout:3875: seq_num 1385 ranges {477695=596111,1530149=450485,2600845=666679} 2024-04-26T00:41:57.535 INFO:tasks.rados.rados.0.smithi045.stdout:3875: writing smithi04526579-34 from 477695 to 1073806 tid 1 2024-04-26T00:41:57.550 INFO:tasks.rados.rados.0.smithi045.stdout:3875: writing smithi04526579-34 from 1530149 to 1980634 tid 2 2024-04-26T00:41:57.570 INFO:tasks.rados.rados.0.smithi045.stdout:3875: writing smithi04526579-34 from 2600845 to 3267524 tid 3 2024-04-26T00:41:57.570 INFO:tasks.rados.rados.0.smithi045.stdout:3876: delete oid 28 current snap is 428 2024-04-26T00:41:57.570 INFO:tasks.rados.rados.0.smithi045.stdout:3875: finishing write tid 1 to smithi04526579-34 2024-04-26T00:41:57.571 INFO:tasks.rados.rados.0.smithi045.stdout:3875: finishing write tid 2 to smithi04526579-34 2024-04-26T00:41:57.578 INFO:tasks.rados.rados.0.smithi045.stdout:3876: done (1 left) 2024-04-26T00:41:57.578 INFO:tasks.rados.rados.0.smithi045.stdout:3877: rollback oid 31 current snap is 428 2024-04-26T00:41:57.578 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 31 to 405 2024-04-26T00:41:57.579 INFO:tasks.rados.rados.0.smithi045.stdout:3878: write (excl) oid 49 current snap is 428 2024-04-26T00:41:57.579 INFO:tasks.rados.rados.0.smithi045.stdout:3878: seq_num 1387 ranges {511309=746609,2047113=454154,3055691=255618} 2024-04-26T00:41:57.597 INFO:tasks.rados.rados.0.smithi045.stdout:3878: writing smithi04526579-49 from 511309 to 1257918 tid 1 2024-04-26T00:41:57.613 INFO:tasks.rados.rados.0.smithi045.stdout:3878: writing smithi04526579-49 from 2047113 to 2501267 tid 2 2024-04-26T00:41:57.623 INFO:tasks.rados.rados.0.smithi045.stdout:3878: writing smithi04526579-49 from 3055691 to 3311309 tid 3 2024-04-26T00:41:57.623 INFO:tasks.rados.rados.0.smithi045.stdout:3879: snap_remove snap 413 2024-04-26T00:41:58.305 INFO:tasks.rados.rados.0.smithi045.stdout:3879: done (3 left) 2024-04-26T00:41:58.305 INFO:tasks.rados.rados.0.smithi045.stdout:3880: snap_remove snap 412 2024-04-26T00:41:59.291 INFO:tasks.rados.rados.0.smithi045.stdout:3880: done (3 left) 2024-04-26T00:41:59.291 INFO:tasks.rados.rados.0.smithi045.stdout:3881: write oid 27 current snap is 428 2024-04-26T00:41:59.292 INFO:tasks.rados.rados.0.smithi045.stdout:3881: seq_num 1388 ranges {528116=421858,1589242=597812,2966761=361355} 2024-04-26T00:41:59.301 INFO:tasks.rados.rados.0.smithi045.stdout:3881: writing smithi04526579-27 from 528116 to 949974 tid 1 2024-04-26T00:41:59.316 INFO:tasks.rados.rados.0.smithi045.stdout:3881: writing smithi04526579-27 from 1589242 to 2187054 tid 2 2024-04-26T00:41:59.329 INFO:tasks.rados.rados.0.smithi045.stdout:3881: writing smithi04526579-27 from 2966761 to 3328116 tid 3 2024-04-26T00:41:59.329 INFO:tasks.rados.rados.0.smithi045.stdout:3882: snap_remove snap 410 2024-04-26T00:42:00.303 INFO:tasks.rados.rados.0.smithi045.stdout:3875: finishing write tid 3 to smithi04526579-34 2024-04-26T00:42:00.303 INFO:tasks.rados.rados.0.smithi045.stdout:3875: finishing write tid 4 to smithi04526579-34 2024-04-26T00:42:00.303 INFO:tasks.rados.rados.0.smithi045.stdout:3875: finishing write tid 5 to smithi04526579-34 2024-04-26T00:42:00.303 INFO:tasks.rados.rados.0.smithi045.stdout:3875: oid 34 updating version 0 to 2831 2024-04-26T00:42:00.303 INFO:tasks.rados.rados.0.smithi045.stdout:3875: oid 34 version 2831 is already newer than 2828 2024-04-26T00:42:00.304 INFO:tasks.rados.rados.0.smithi045.stdout:3875: oid 34 version 2831 is already newer than 2829 2024-04-26T00:42:00.304 INFO:tasks.rados.rados.0.smithi045.stdout:3875: oid 34 version 2831 is already newer than 2830 2024-04-26T00:42:00.304 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 2831 (ObjNum 1385 snap 428 seq_num 1385) dirty exists 2024-04-26T00:42:00.304 INFO:tasks.rados.rados.0.smithi045.stdout:3875: left oid 34 (ObjNum 1385 snap 428 seq_num 1385) 2024-04-26T00:42:00.304 INFO:tasks.rados.rados.0.smithi045.stdout:3877: finishing rollback tid 0 to smithi04526579-31 2024-04-26T00:42:00.304 INFO:tasks.rados.rados.0.smithi045.stdout:3877: finishing rollback tid 1 to smithi04526579-31 2024-04-26T00:42:00.304 INFO:tasks.rados.rados.0.smithi045.stdout:3877: finishing rollback tid 2 to smithi04526579-31 2024-04-26T00:42:00.304 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 31 v 2550 (ObjNum 1293 snap 402 seq_num 1293) dirty exists 2024-04-26T00:42:00.304 INFO:tasks.rados.rados.0.smithi045.stdout:3878: finishing write tid 1 to smithi04526579-49 2024-04-26T00:42:00.304 INFO:tasks.rados.rados.0.smithi045.stdout:3878: finishing write tid 2 to smithi04526579-49 2024-04-26T00:42:00.304 INFO:tasks.rados.rados.0.smithi045.stdout:3878: finishing write tid 3 to smithi04526579-49 2024-04-26T00:42:00.304 INFO:tasks.rados.rados.0.smithi045.stdout:3878: finishing write tid 4 to smithi04526579-49 2024-04-26T00:42:00.304 INFO:tasks.rados.rados.0.smithi045.stdout:3878: finishing write tid 5 to smithi04526579-49 2024-04-26T00:42:00.304 INFO:tasks.rados.rados.0.smithi045.stdout:3878: oid 49 updating version 0 to 2832 2024-04-26T00:42:00.304 INFO:tasks.rados.rados.0.smithi045.stdout:3878: oid 49 updating version 2832 to 2833 2024-04-26T00:42:00.305 INFO:tasks.rados.rados.0.smithi045.stdout:3878: oid 49 updating version 2833 to 2835 2024-04-26T00:42:00.305 INFO:tasks.rados.rados.0.smithi045.stdout:3878: oid 49 version 2835 is already newer than 2834 2024-04-26T00:42:00.305 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 2835 (ObjNum 1387 snap 428 seq_num 1387) dirty exists 2024-04-26T00:42:00.305 INFO:tasks.rados.rados.0.smithi045.stdout:3878: left oid 49 (ObjNum 1387 snap 428 seq_num 1387) 2024-04-26T00:42:00.305 INFO:tasks.rados.rados.0.smithi045.stdout:3881: finishing write tid 1 to smithi04526579-27 2024-04-26T00:42:00.305 INFO:tasks.rados.rados.0.smithi045.stdout:3881: finishing write tid 2 to smithi04526579-27 2024-04-26T00:42:00.305 INFO:tasks.rados.rados.0.smithi045.stdout:3881: finishing write tid 3 to smithi04526579-27 2024-04-26T00:42:00.305 INFO:tasks.rados.rados.0.smithi045.stdout:3881: finishing write tid 4 to smithi04526579-27 2024-04-26T00:42:00.305 INFO:tasks.rados.rados.0.smithi045.stdout:3881: finishing write tid 5 to smithi04526579-27 2024-04-26T00:42:00.305 INFO:tasks.rados.rados.0.smithi045.stdout:3881: oid 27 updating version 0 to 2880 2024-04-26T00:42:00.305 INFO:tasks.rados.rados.0.smithi045.stdout:3881: oid 27 version 2880 is already newer than 2879 2024-04-26T00:42:00.305 INFO:tasks.rados.rados.0.smithi045.stdout:3881: oid 27 version 2880 is already newer than 2877 2024-04-26T00:42:00.305 INFO:tasks.rados.rados.0.smithi045.stdout:3881: oid 27 version 2880 is already newer than 2878 2024-04-26T00:42:00.305 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 2880 (ObjNum 1388 snap 428 seq_num 1388) dirty exists 2024-04-26T00:42:00.305 INFO:tasks.rados.rados.0.smithi045.stdout:3881: left oid 27 (ObjNum 1388 snap 428 seq_num 1388) 2024-04-26T00:42:00.305 INFO:tasks.rados.rados.0.smithi045.stdout:3875: done (4 left) 2024-04-26T00:42:00.306 INFO:tasks.rados.rados.0.smithi045.stdout:3877: done (3 left) 2024-04-26T00:42:00.306 INFO:tasks.rados.rados.0.smithi045.stdout:3878: done (2 left) 2024-04-26T00:42:00.306 INFO:tasks.rados.rados.0.smithi045.stdout:3881: done (1 left) 2024-04-26T00:42:00.306 INFO:tasks.rados.rados.0.smithi045.stdout:3882: done (0 left) 2024-04-26T00:42:00.306 INFO:tasks.rados.rados.0.smithi045.stdout:3883: copy_from oid 10 from oid 15 current snap is 428 2024-04-26T00:42:00.306 INFO:tasks.rados.rados.0.smithi045.stdout:3884: snap_remove snap 384 2024-04-26T00:42:01.272 INFO:tasks.rados.rados.0.smithi045.stdout:3883: finishing copy_from to smithi04526579-10 2024-04-26T00:42:01.272 INFO:tasks.rados.rados.0.smithi045.stdout:3883: got expected ENOENT (src dne) 2024-04-26T00:42:01.272 INFO:tasks.rados.rados.0.smithi045.stdout:3883: finishing copy_from racing read to smithi04526579-10 2024-04-26T00:42:01.272 INFO:tasks.rados.rados.0.smithi045.stdout:3883: done (1 left) 2024-04-26T00:42:01.272 INFO:tasks.rados.rados.0.smithi045.stdout:3884: done (0 left) 2024-04-26T00:42:01.272 INFO:tasks.rados.rados.0.smithi045.stdout:3885: snap_remove snap 426 2024-04-26T00:42:02.276 INFO:tasks.rados.rados.0.smithi045.stdout:3885: done (0 left) 2024-04-26T00:42:02.276 INFO:tasks.rados.rados.0.smithi045.stdout:3886: copy_from oid 1 from oid 47 current snap is 428 2024-04-26T00:42:02.277 INFO:tasks.rados.rados.0.smithi045.stdout:3887: copy_from oid 18 from oid 44 current snap is 428 2024-04-26T00:42:02.277 INFO:tasks.rados.rados.0.smithi045.stdout:3888: snap_create 2024-04-26T00:42:02.316 INFO:tasks.rados.rados.0.smithi045.stdout:3887: finishing copy_from racing read to smithi04526579-18 2024-04-26T00:42:02.317 INFO:tasks.rados.rados.0.smithi045.stdout:3886: finishing copy_from racing read to smithi04526579-1 2024-04-26T00:42:02.322 INFO:tasks.rados.rados.0.smithi045.stdout:3886: finishing copy_from to smithi04526579-1 2024-04-26T00:42:02.322 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 2827 (ObjNum 1376 snap 424 seq_num 1376) dirty exists 2024-04-26T00:42:02.322 INFO:tasks.rados.rados.0.smithi045.stdout:3887: finishing copy_from to smithi04526579-18 2024-04-26T00:42:02.322 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 2418 (ObjNum 1377 snap 426 seq_num 1377) dirty exists 2024-04-26T00:42:03.280 INFO:tasks.rados.rados.0.smithi045.stdout:3886: done (2 left) 2024-04-26T00:42:03.280 INFO:tasks.rados.rados.0.smithi045.stdout:3887: done (1 left) 2024-04-26T00:42:03.280 INFO:tasks.rados.rados.0.smithi045.stdout:3888: done (0 left) 2024-04-26T00:42:03.280 INFO:tasks.rados.rados.0.smithi045.stdout:3889: copy_from oid 3 from oid 42 current snap is 429 2024-04-26T00:42:03.280 INFO:tasks.rados.rados.0.smithi045.stdout:3890: snap_remove snap 425 2024-04-26T00:42:04.307 INFO:tasks.rados.rados.0.smithi045.stdout:3890: done (1 left) 2024-04-26T00:42:04.307 INFO:tasks.rados.rados.0.smithi045.stdout:3891: copy_from oid 49 from oid 21 current snap is 429 2024-04-26T00:42:04.307 INFO:tasks.rados.rados.0.smithi045.stdout:3889: finishing copy_from racing read to smithi04526579-3 2024-04-26T00:42:04.307 INFO:tasks.rados.rados.0.smithi045.stdout:3889: finishing copy_from to smithi04526579-3 2024-04-26T00:42:04.307 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 3 v 2874 (ObjNum 1363 snap 421 seq_num 1363) dirty exists 2024-04-26T00:42:04.308 INFO:tasks.rados.rados.0.smithi045.stdout:3889: done (1 left) 2024-04-26T00:42:04.308 INFO:tasks.rados.rados.0.smithi045.stdout:3892: read oid 22 snap -1 2024-04-26T00:42:04.308 INFO:tasks.rados.rados.0.smithi045.stdout:3892: expect (ObjNum 1135 snap 343 seq_num 1135) 2024-04-26T00:42:04.308 INFO:tasks.rados.rados.0.smithi045.stdout:3893: copy_from oid 2 from oid 24 current snap is 429 2024-04-26T00:42:04.308 INFO:tasks.rados.rados.0.smithi045.stdout:3894: rollback oid 29 current snap is 429 2024-04-26T00:42:04.308 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 29 to 414 2024-04-26T00:42:04.308 INFO:tasks.rados.rados.0.smithi045.stdout:3895: read oid 41 snap 417 2024-04-26T00:42:04.308 INFO:tasks.rados.rados.0.smithi045.stdout:3895: expect (ObjNum 1321 snap 407 seq_num 1321) 2024-04-26T00:42:04.309 INFO:tasks.rados.rados.0.smithi045.stdout:3896: write (excl) oid 20 current snap is 429 2024-04-26T00:42:04.309 INFO:tasks.rados.rados.0.smithi045.stdout:3896: seq_num 1389 ranges {544923=497107,1647724=457823,2710537=428120} 2024-04-26T00:42:04.328 INFO:tasks.rados.rados.0.smithi045.stdout:3896: writing smithi04526579-20 from 544923 to 1042030 tid 1 2024-04-26T00:42:04.346 INFO:tasks.rados.rados.0.smithi045.stdout:3896: writing smithi04526579-20 from 1647724 to 2105547 tid 2 2024-04-26T00:42:04.363 INFO:tasks.rados.rados.0.smithi045.stdout:3896: writing smithi04526579-20 from 2710537 to 3138657 tid 3 2024-04-26T00:42:04.363 INFO:tasks.rados.rados.0.smithi045.stdout:3897: delete oid 31 current snap is 429 2024-04-26T00:42:04.364 INFO:tasks.rados.rados.0.smithi045.stdout:3893: finishing copy_from to smithi04526579-2 2024-04-26T00:42:04.364 INFO:tasks.rados.rados.0.smithi045.stdout:3893: got expected ENOENT (src dne) 2024-04-26T00:42:04.364 INFO:tasks.rados.rados.0.smithi045.stdout:3893: finishing copy_from racing read to smithi04526579-2 2024-04-26T00:42:04.364 INFO:tasks.rados.rados.0.smithi045.stdout:3895: expect (ObjNum 1321 snap 407 seq_num 1321) 2024-04-26T00:42:04.550 INFO:tasks.rados.rados.0.smithi045.stdout:3894: finishing rollback tid 0 to smithi04526579-29 2024-04-26T00:42:04.551 INFO:tasks.rados.rados.0.smithi045.stdout:3894: finishing rollback tid 1 to smithi04526579-29 2024-04-26T00:42:04.551 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 29 v 2254 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-04-26T00:42:04.551 INFO:tasks.rados.rados.0.smithi045.stdout:3892: expect (ObjNum 1135 snap 343 seq_num 1135) 2024-04-26T00:42:04.691 INFO:tasks.rados.rados.0.smithi045.stdout:3896: finishing write tid 1 to smithi04526579-20 2024-04-26T00:42:04.691 INFO:tasks.rados.rados.0.smithi045.stdout:3891: finishing copy_from racing read to smithi04526579-49 2024-04-26T00:42:04.691 INFO:tasks.rados.rados.0.smithi045.stdout:3891: finishing copy_from to smithi04526579-49 2024-04-26T00:42:04.691 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 2488 (ObjNum 1360 snap 420 seq_num 1360) dirty exists 2024-04-26T00:42:04.691 INFO:tasks.rados.rados.0.smithi045.stdout:3896: finishing write tid 2 to smithi04526579-20 2024-04-26T00:42:04.691 INFO:tasks.rados.rados.0.smithi045.stdout:3891: done (6 left) 2024-04-26T00:42:04.691 INFO:tasks.rados.rados.0.smithi045.stdout:3892: done (5 left) 2024-04-26T00:42:04.692 INFO:tasks.rados.rados.0.smithi045.stdout:3893: done (4 left) 2024-04-26T00:42:04.692 INFO:tasks.rados.rados.0.smithi045.stdout:3894: done (3 left) 2024-04-26T00:42:04.692 INFO:tasks.rados.rados.0.smithi045.stdout:3895: done (2 left) 2024-04-26T00:42:04.692 INFO:tasks.rados.rados.0.smithi045.stdout:3897: done (1 left) 2024-04-26T00:42:04.692 INFO:tasks.rados.rados.0.smithi045.stdout:3898: copy_from oid 36 from oid 1 current snap is 429 2024-04-26T00:42:04.692 INFO:tasks.rados.rados.0.smithi045.stdout:3896: finishing write tid 3 to smithi04526579-20 2024-04-26T00:42:04.692 INFO:tasks.rados.rados.0.smithi045.stdout:3896: finishing write tid 4 to smithi04526579-20 2024-04-26T00:42:04.692 INFO:tasks.rados.rados.0.smithi045.stdout:3896: finishing write tid 5 to smithi04526579-20 2024-04-26T00:42:04.692 INFO:tasks.rados.rados.0.smithi045.stdout:3896: oid 20 updating version 0 to 2422 2024-04-26T00:42:04.692 INFO:tasks.rados.rados.0.smithi045.stdout:3896: oid 20 version 2422 is already newer than 2420 2024-04-26T00:42:04.692 INFO:tasks.rados.rados.0.smithi045.stdout:3896: oid 20 version 2422 is already newer than 2419 2024-04-26T00:42:04.692 INFO:tasks.rados.rados.0.smithi045.stdout:3896: oid 20 version 2422 is already newer than 2421 2024-04-26T00:42:04.692 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 20 v 2422 (ObjNum 1389 snap 429 seq_num 1389) dirty exists 2024-04-26T00:42:04.692 INFO:tasks.rados.rados.0.smithi045.stdout:3896: left oid 20 (ObjNum 1389 snap 429 seq_num 1389) 2024-04-26T00:42:04.693 INFO:tasks.rados.rados.0.smithi045.stdout:3896: done (1 left) 2024-04-26T00:42:04.693 INFO:tasks.rados.rados.0.smithi045.stdout:3899: delete oid 10 current snap is 429 2024-04-26T00:42:04.697 INFO:tasks.rados.rados.0.smithi045.stdout:3899: done (1 left) 2024-04-26T00:42:04.697 INFO:tasks.rados.rados.0.smithi045.stdout:3900: snap_create 2024-04-26T00:42:04.708 INFO:tasks.rados.rados.0.smithi045.stdout:3898: finishing copy_from racing read to smithi04526579-36 2024-04-26T00:42:04.709 INFO:tasks.rados.rados.0.smithi045.stdout:3898: finishing copy_from to smithi04526579-36 2024-04-26T00:42:04.709 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 2827 (ObjNum 1376 snap 424 seq_num 1376) dirty exists 2024-04-26T00:42:05.306 INFO:tasks.rados.rados.0.smithi045.stdout:3898: done (1 left) 2024-04-26T00:42:05.306 INFO:tasks.rados.rados.0.smithi045.stdout:3900: done (0 left) 2024-04-26T00:42:05.307 INFO:tasks.rados.rados.0.smithi045.stdout:3901: write oid 9 current snap is 430 2024-04-26T00:42:05.307 INFO:tasks.rados.rados.0.smithi045.stdout:3901: seq_num 1392 ranges {595344=439207,1823170=605150,3076453=318891} 2024-04-26T00:42:05.323 INFO:tasks.rados.rados.0.smithi045.stdout:3901: writing smithi04526579-9 from 595344 to 1034551 tid 1 2024-04-26T00:42:05.346 INFO:tasks.rados.rados.0.smithi045.stdout:3901: writing smithi04526579-9 from 1823170 to 2428320 tid 2 2024-04-26T00:42:05.360 INFO:tasks.rados.rados.0.smithi045.stdout:3901: writing smithi04526579-9 from 3076453 to 3395344 tid 3 2024-04-26T00:42:05.360 INFO:tasks.rados.rados.0.smithi045.stdout:3902: read oid 22 snap -1 2024-04-26T00:42:05.361 INFO:tasks.rados.rados.0.smithi045.stdout:3902: expect (ObjNum 1135 snap 343 seq_num 1135) 2024-04-26T00:42:05.361 INFO:tasks.rados.rados.0.smithi045.stdout:3903: snap_create 2024-04-26T00:42:05.361 INFO:tasks.rados.rados.0.smithi045.stdout:3901: finishing write tid 1 to smithi04526579-9 2024-04-26T00:42:05.361 INFO:tasks.rados.rados.0.smithi045.stdout:3901: finishing write tid 2 to smithi04526579-9 2024-04-26T00:42:05.368 INFO:tasks.rados.rados.0.smithi045.stdout:3901: finishing write tid 3 to smithi04526579-9 2024-04-26T00:42:05.368 INFO:tasks.rados.rados.0.smithi045.stdout:3901: finishing write tid 4 to smithi04526579-9 2024-04-26T00:42:05.369 INFO:tasks.rados.rados.0.smithi045.stdout:3901: finishing write tid 5 to smithi04526579-9 2024-04-26T00:42:05.369 INFO:tasks.rados.rados.0.smithi045.stdout:3901: oid 9 updating version 0 to 2552 2024-04-26T00:42:05.369 INFO:tasks.rados.rados.0.smithi045.stdout:3901: oid 9 updating version 2552 to 2554 2024-04-26T00:42:05.370 INFO:tasks.rados.rados.0.smithi045.stdout:3901: oid 9 version 2554 is already newer than 2553 2024-04-26T00:42:05.370 INFO:tasks.rados.rados.0.smithi045.stdout:3901: oid 9 updating version 2554 to 2555 2024-04-26T00:42:05.370 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 2555 (ObjNum 1392 snap 430 seq_num 1392) dirty exists 2024-04-26T00:42:05.370 INFO:tasks.rados.rados.0.smithi045.stdout:3901: left oid 9 (ObjNum 1392 snap 430 seq_num 1392) 2024-04-26T00:42:05.373 INFO:tasks.rados.rados.0.smithi045.stdout:3902: expect (ObjNum 1135 snap 343 seq_num 1135) 2024-04-26T00:42:06.275 INFO:tasks.rados.rados.0.smithi045.stdout:3901: done (2 left) 2024-04-26T00:42:06.275 INFO:tasks.rados.rados.0.smithi045.stdout:3902: done (1 left) 2024-04-26T00:42:06.275 INFO:tasks.rados.rados.0.smithi045.stdout:3903: done (0 left) 2024-04-26T00:42:06.275 INFO:tasks.rados.rados.0.smithi045.stdout:3904: read oid 37 snap -1 2024-04-26T00:42:06.275 INFO:tasks.rados.rados.0.smithi045.stdout:3904: expect (ObjNum 1075 snap 322 seq_num 1075) 2024-04-26T00:42:06.275 INFO:tasks.rados.rados.0.smithi045.stdout:3905: snap_create 2024-04-26T00:42:06.292 INFO:tasks.rados.rados.0.smithi045.stdout:3904: expect (ObjNum 1075 snap 322 seq_num 1075) 2024-04-26T00:42:07.287 INFO:tasks.rados.rados.0.smithi045.stdout:3904: done (1 left) 2024-04-26T00:42:07.288 INFO:tasks.rados.rados.0.smithi045.stdout:3905: done (0 left) 2024-04-26T00:42:07.288 INFO:tasks.rados.rados.0.smithi045.stdout:3906: delete oid 49 current snap is 432 2024-04-26T00:42:07.302 INFO:tasks.rados.rados.0.smithi045.stdout:3906: done (0 left) 2024-04-26T00:42:07.302 INFO:tasks.rados.rados.0.smithi045.stdout:3907: read oid 24 snap -1 2024-04-26T00:42:07.303 INFO:tasks.rados.rados.0.smithi045.stdout:3907: expect deleted 2024-04-26T00:42:07.303 INFO:tasks.rados.rados.0.smithi045.stdout:3908: read oid 40 snap -1 2024-04-26T00:42:07.303 INFO:tasks.rados.rados.0.smithi045.stdout:3908: expect (ObjNum 1178 snap 366 seq_num 1178) 2024-04-26T00:42:07.303 INFO:tasks.rados.rados.0.smithi045.stdout:3909: copy_from oid 38 from oid 17 current snap is 432 2024-04-26T00:42:07.303 INFO:tasks.rados.rados.0.smithi045.stdout:3910: snap_remove snap 414 2024-04-26T00:42:08.299 INFO:tasks.rados.rados.0.smithi045.stdout:3910: done (3 left) 2024-04-26T00:42:08.300 INFO:tasks.rados.rados.0.smithi045.stdout:3911: read oid 6 snap -1 2024-04-26T00:42:08.300 INFO:tasks.rados.rados.0.smithi045.stdout:3911: expect (ObjNum 1367 snap 422 seq_num 1367) 2024-04-26T00:42:08.300 INFO:tasks.rados.rados.0.smithi045.stdout:3912: read oid 28 snap -1 2024-04-26T00:42:08.300 INFO:tasks.rados.rados.0.smithi045.stdout:3912: expect deleted 2024-04-26T00:42:08.300 INFO:tasks.rados.rados.0.smithi045.stdout:3913: delete oid 5 current snap is 432 2024-04-26T00:42:08.300 INFO:tasks.rados.rados.0.smithi045.stdout:3909: finishing copy_from to smithi04526579-38 2024-04-26T00:42:08.301 INFO:tasks.rados.rados.0.smithi045.stdout:3909: got expected ENOENT (src dne) 2024-04-26T00:42:08.301 INFO:tasks.rados.rados.0.smithi045.stdout:3909: finishing copy_from racing read to smithi04526579-38 2024-04-26T00:42:08.301 INFO:tasks.rados.rados.0.smithi045.stdout:3908: expect (ObjNum 1178 snap 366 seq_num 1178) 2024-04-26T00:42:08.612 INFO:tasks.rados.rados.0.smithi045.stdout:3911: expect (ObjNum 1367 snap 422 seq_num 1367) 2024-04-26T00:42:08.779 INFO:tasks.rados.rados.0.smithi045.stdout:3907: done (5 left) 2024-04-26T00:42:08.779 INFO:tasks.rados.rados.0.smithi045.stdout:3908: done (4 left) 2024-04-26T00:42:08.780 INFO:tasks.rados.rados.0.smithi045.stdout:3909: done (3 left) 2024-04-26T00:42:08.780 INFO:tasks.rados.rados.0.smithi045.stdout:3911: done (2 left) 2024-04-26T00:42:08.780 INFO:tasks.rados.rados.0.smithi045.stdout:3912: done (1 left) 2024-04-26T00:42:08.780 INFO:tasks.rados.rados.0.smithi045.stdout:3913: done (0 left) 2024-04-26T00:42:08.780 INFO:tasks.rados.rados.0.smithi045.stdout:3914: write oid 8 current snap is 432 2024-04-26T00:42:08.780 INFO:tasks.rados.rados.0.smithi045.stdout:3914: seq_num 1395 ranges {645765=664954,1998616=752477,3158722=287043} 2024-04-26T00:42:08.787 INFO:tasks.rados.rados.0.smithi045.stdout:3914: writing smithi04526579-8 from 645765 to 1310719 tid 1 2024-04-26T00:42:08.795 INFO:tasks.rados.rados.0.smithi045.stdout:3914: writing smithi04526579-8 from 1998616 to 2751093 tid 2 2024-04-26T00:42:08.799 INFO:tasks.rados.rados.0.smithi045.stdout:3914: writing smithi04526579-8 from 3158722 to 3445765 tid 3 2024-04-26T00:42:08.799 INFO:tasks.rados.rados.0.smithi045.stdout:3915: snap_remove snap 409 2024-04-26T00:42:09.296 INFO:tasks.rados.rados.0.smithi045.stdout:3915: done (1 left) 2024-04-26T00:42:09.297 INFO:tasks.rados.rados.0.smithi045.stdout:3916: rollback oid 18 current snap is 432 2024-04-26T00:42:09.297 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 18 to 401 2024-04-26T00:42:09.297 INFO:tasks.rados.rados.0.smithi045.stdout:3914: finishing write tid 1 to smithi04526579-8 2024-04-26T00:42:09.297 INFO:tasks.rados.rados.0.smithi045.stdout:3914: finishing write tid 2 to smithi04526579-8 2024-04-26T00:42:09.297 INFO:tasks.rados.rados.0.smithi045.stdout:3914: finishing write tid 3 to smithi04526579-8 2024-04-26T00:42:09.297 INFO:tasks.rados.rados.0.smithi045.stdout:3914: finishing write tid 4 to smithi04526579-8 2024-04-26T00:42:09.297 INFO:tasks.rados.rados.0.smithi045.stdout:3914: finishing write tid 5 to smithi04526579-8 2024-04-26T00:42:09.297 INFO:tasks.rados.rados.0.smithi045.stdout:3914: oid 8 updating version 0 to 2258 2024-04-26T00:42:09.297 INFO:tasks.rados.rados.0.smithi045.stdout:3914: oid 8 version 2258 is already newer than 2257 2024-04-26T00:42:09.297 INFO:tasks.rados.rados.0.smithi045.stdout:3914: oid 8 version 2258 is already newer than 2255 2024-04-26T00:42:09.297 INFO:tasks.rados.rados.0.smithi045.stdout:3914: oid 8 version 2258 is already newer than 2256 2024-04-26T00:42:09.297 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 8 v 2258 (ObjNum 1395 snap 432 seq_num 1395) dirty exists 2024-04-26T00:42:09.297 INFO:tasks.rados.rados.0.smithi045.stdout:3914: left oid 8 (ObjNum 1395 snap 432 seq_num 1395) 2024-04-26T00:42:09.298 INFO:tasks.rados.rados.0.smithi045.stdout:3914: done (1 left) 2024-04-26T00:42:09.298 INFO:tasks.rados.rados.0.smithi045.stdout:3917: write oid 1 current snap is 432 2024-04-26T00:42:09.298 INFO:tasks.rados.rados.0.smithi045.stdout:3917: seq_num 1396 ranges {662572=740203,1940745=612488,3069792=392780} 2024-04-26T00:42:09.318 INFO:tasks.rados.rados.0.smithi045.stdout:3917: writing smithi04526579-1 from 662572 to 1402775 tid 1 2024-04-26T00:42:09.332 INFO:tasks.rados.rados.0.smithi045.stdout:3917: writing smithi04526579-1 from 1940745 to 2553233 tid 2 2024-04-26T00:42:09.337 INFO:tasks.rados.rados.0.smithi045.stdout:3917: writing smithi04526579-1 from 3069792 to 3462572 tid 3 2024-04-26T00:42:09.337 INFO:tasks.rados.rados.0.smithi045.stdout:3918: snap_create 2024-04-26T00:42:09.337 INFO:tasks.rados.rados.0.smithi045.stdout:3917: finishing write tid 1 to smithi04526579-1 2024-04-26T00:42:09.341 INFO:tasks.rados.rados.0.smithi045.stdout:3916: finishing rollback tid 0 to smithi04526579-18 2024-04-26T00:42:09.341 INFO:tasks.rados.rados.0.smithi045.stdout:3916: finishing rollback tid 1 to smithi04526579-18 2024-04-26T00:42:09.342 INFO:tasks.rados.rados.0.smithi045.stdout:3916: finishing rollback tid 2 to smithi04526579-18 2024-04-26T00:42:09.342 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 2999 (ObjNum 1096 snap 325 seq_num 1096) dirty exists 2024-04-26T00:42:09.351 INFO:tasks.rados.rados.0.smithi045.stdout:3917: finishing write tid 2 to smithi04526579-1 2024-04-26T00:42:09.351 INFO:tasks.rados.rados.0.smithi045.stdout:3917: finishing write tid 3 to smithi04526579-1 2024-04-26T00:42:09.352 INFO:tasks.rados.rados.0.smithi045.stdout:3917: finishing write tid 4 to smithi04526579-1 2024-04-26T00:42:09.352 INFO:tasks.rados.rados.0.smithi045.stdout:3917: finishing write tid 5 to smithi04526579-1 2024-04-26T00:42:09.352 INFO:tasks.rados.rados.0.smithi045.stdout:3917: oid 1 updating version 0 to 2831 2024-04-26T00:42:09.352 INFO:tasks.rados.rados.0.smithi045.stdout:3917: oid 1 version 2831 is already newer than 2830 2024-04-26T00:42:09.352 INFO:tasks.rados.rados.0.smithi045.stdout:3917: oid 1 version 2831 is already newer than 2829 2024-04-26T00:42:09.352 INFO:tasks.rados.rados.0.smithi045.stdout:3917: oid 1 version 2831 is already newer than 2828 2024-04-26T00:42:09.352 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 2831 (ObjNum 1396 snap 432 seq_num 1396) dirty exists 2024-04-26T00:42:09.352 INFO:tasks.rados.rados.0.smithi045.stdout:3917: left oid 1 (ObjNum 1396 snap 432 seq_num 1396) 2024-04-26T00:42:10.314 INFO:tasks.rados.rados.0.smithi045.stdout:3916: done (2 left) 2024-04-26T00:42:10.314 INFO:tasks.rados.rados.0.smithi045.stdout:3917: done (1 left) 2024-04-26T00:42:10.314 INFO:tasks.rados.rados.0.smithi045.stdout:3918: done (0 left) 2024-04-26T00:42:10.315 INFO:tasks.rados.rados.0.smithi045.stdout:3919: snap_create 2024-04-26T00:42:11.278 INFO:tasks.rados.rados.0.smithi045.stdout:3919: done (0 left) 2024-04-26T00:42:11.279 INFO:tasks.rados.rados.0.smithi045.stdout:3920: read oid 13 snap 428 2024-04-26T00:42:11.279 INFO:tasks.rados.rados.0.smithi045.stdout:3920: expect (ObjNum 1237 snap 384 seq_num 1237) 2024-04-26T00:42:11.279 INFO:tasks.rados.rados.0.smithi045.stdout:3921: snap_remove snap 431 2024-04-26T00:42:12.164 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-04-26T00:42:12.164 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-04-26T00:42:12.164 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-26T00:42:12.164 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-26T00:42:12.168 INFO:tasks.ceph.mon.b:Started 2024-04-26T00:42:12.168 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:42:12.168 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:42:12.292 INFO:tasks.rados.rados.0.smithi045.stdout:3921: done (1 left) 2024-04-26T00:42:12.292 INFO:tasks.rados.rados.0.smithi045.stdout:3922: rollback oid 32 current snap is 434 2024-04-26T00:42:12.292 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 32 to 419 2024-04-26T00:42:12.293 INFO:tasks.rados.rados.0.smithi045.stdout:3920: expect (ObjNum 1237 snap 384 seq_num 1237) 2024-04-26T00:42:12.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.300+0000 7f9f12f02640 10 monclient: get_monmap_and_config 2024-04-26T00:42:12.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.300+0000 7f9f12f02640 10 monclient: build_initial_monmap 2024-04-26T00:42:12.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.300+0000 7f9f12f02640 10 monclient: monmap: 2024-04-26T00:42:12.303 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:42:12.304 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:12.304 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:42:12.305611+0000 2024-04-26T00:42:12.304 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:42:12.305611+0000 2024-04-26T00:42:12.304 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:42:12.304 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:12.304 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:42:12.304 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:42:12.304 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:42:12.304 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:42:12.304 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:42:12.304 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:12.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.300+0000 7f9f12f02640 1 Processor -- start 2024-04-26T00:42:12.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.300+0000 7f9f12f02640 1 -- start start 2024-04-26T00:42:12.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.300+0000 7f9f12f02640 10 monclient: init 2024-04-26T00:42:12.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.300+0000 7f9f12f02640 10 monclient: _reopen_session rank -1 2024-04-26T00:42:12.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.300+0000 7f9f12f02640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-04-26T00:42:12.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.300+0000 7f9f12f02640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9f0c136e10 0x7f9f0c09c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:12.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.300+0000 7f9f12f02640 10 monclient(hunting): picked mon.noname-c con 0x7f9f0c136e10 addr v2:172.21.15.45:3300/0 2024-04-26T00:42:12.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.300+0000 7f9f12f02640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9f0c1371f0 addr v1:172.21.15.45:6789/0 2024-04-26T00:42:12.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.300+0000 7f9f12f02640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9f0c13c340 addr v1:172.21.15.37:6789/0 2024-04-26T00:42:12.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.300+0000 7f9f12f02640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f0c09b680 con 0x7f9f0c13c340 2024-04-26T00:42:12.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.300+0000 7f9f12f02640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f0c09bbd0 con 0x7f9f0c1371f0 2024-04-26T00:42:12.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.300+0000 7f9f12f02640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:12.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.300+0000 7f9f12f02640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f9f0c09b8f0 con 0x7f9f0c136e10 2024-04-26T00:42:12.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.300+0000 7f9f12f02640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:12.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.300+0000 7f9f12f02640 10 monclient(hunting): authenticate will time out at 2313.611339s 2024-04-26T00:42:12.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.300+0000 7f9f1149f640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f9f0c13c340 0x7f9f0c13e730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:48232/0 (socket says 172.21.15.37:48232) 2024-04-26T00:42:12.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.300+0000 7f9f1149f640 1 -- 172.21.15.37:0/2752213374 learned_addr learned my addr 172.21.15.37:0/2752213374 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:42:12.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.300+0000 7f9f10c9e640 1 --2- 172.21.15.37:0/2752213374 >> v2:172.21.15.45:3300/0 conn(0x7f9f0c136e10 0x7f9f0c09c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:12.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.300+0000 7f9f10c9e640 10 monclient(hunting): get_auth_request con 0x7f9f0c136e10 auth_method 0 2024-04-26T00:42:12.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.300+0000 7f9f10c9e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:12.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.300+0000 7f9f10c9e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:12.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.300+0000 7f9f10c9e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:12.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.300+0000 7f9f10c9e640 1 -- 172.21.15.37:0/2752213374 >> v2:172.21.15.45:3300/0 conn(0x7f9f0c136e10 msgr2=0x7f9f0c09c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:42:12.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.300+0000 7f9f10c9e640 1 -- 172.21.15.37:0/2752213374 >> v2:172.21.15.45:3300/0 conn(0x7f9f0c136e10 msgr2=0x7f9f0c09c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:42:12.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.300+0000 7f9f10c9e640 1 --2- 172.21.15.37:0/2752213374 >> v2:172.21.15.45:3300/0 conn(0x7f9f0c136e10 0x7f9f0c09c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:42:12.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.300+0000 7f9f10c9e640 1 --2- 172.21.15.37:0/2752213374 >> v2:172.21.15.45:3300/0 conn(0x7f9f0c136e10 0x7f9f0c09c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:42:12.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f1149f640 1 -- 172.21.15.37:0/2752213374 >> v1:172.21.15.37:6789/0 conn(0x7f9f0c13c340 legacy=0x7f9f0c13e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:12.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0b7fe640 1 -- 172.21.15.37:0/2752213374 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3267740540 0 0) 0x7f9f0c09b680 con 0x7f9f0c13c340 2024-04-26T00:42:12.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0b7fe640 10 monclient(hunting): _negotiate 2024-04-26T00:42:12.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:12.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:12.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0b7fe640 10 monclient(hunting): my global_id is 264177 2024-04-26T00:42:12.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0b7fe640 1 -- 172.21.15.37:0/2752213374 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ef4003530 con 0x7f9f0c13c340 2024-04-26T00:42:12.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0bfff640 1 -- 172.21.15.37:0/2752213374 >> v1:172.21.15.45:6789/0 conn(0x7f9f0c1371f0 legacy=0x7f9f0c137600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:12.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0b7fe640 1 -- 172.21.15.37:0/2752213374 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1189192649 0 0) 0x7f9ef4003530 con 0x7f9f0c13c340 2024-04-26T00:42:12.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0b7fe640 1 -- 172.21.15.37:0/2752213374 >> v2:172.21.15.45:3300/0 conn(0x7f9f0c136e10 msgr2=0x7f9f0c09c5b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:42:12.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0b7fe640 1 --2- 172.21.15.37:0/2752213374 >> v2:172.21.15.45:3300/0 conn(0x7f9f0c136e10 0x7f9f0c09c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:12.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0b7fe640 1 -- 172.21.15.37:0/2752213374 >> v1:172.21.15.45:6789/0 conn(0x7f9f0c1371f0 legacy=0x7f9f0c137600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:12.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0b7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:42:12.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0b7fe640 1 monclient: found mon.noname-a-legacy 2024-04-26T00:42:12.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:42:12.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0b7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.37:6789/0 2024-04-26T00:42:12.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0b7fe640 1 -- 172.21.15.37:0/2752213374 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f0c0a5d90 con 0x7f9f0c13c340 2024-04-26T00:42:12.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0b7fe640 10 monclient: _finish_auth 0 2024-04-26T00:42:12.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0b7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:42:12.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:42:12.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0b7fe640 1 -- 172.21.15.37:0/2752213374 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 866326386 0 0) 0x7f9f04001cb0 con 0x7f9f0c13c340 2024-04-26T00:42:12.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:12.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0b7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-26T00:42:12.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0b7fe640 10 monclient: dump: 2024-04-26T00:42:12.314 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:12.314 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:12.314 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:12.314 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:12.314 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:12.315 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:12.315 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:12.315 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:12.315 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:12.315 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:12.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0b7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] but i'm connected to v1:172.21.15.37:6789/0, reconnecting 2024-04-26T00:42:12.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0b7fe640 10 monclient: _reopen_session rank -1 2024-04-26T00:42:12.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0b7fe640 1 -- 172.21.15.37:0/2752213374 >> v1:172.21.15.37:6789/0 conn(0x7f9f0c13c340 legacy=0x7f9f0c13e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:12.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0b7fe640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:42:12.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0b7fe640 1 --2- 172.21.15.37:0/2752213374 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ef4003a90 0x7f9ef4003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:12.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0b7fe640 10 monclient(hunting): picked mon.b con 0x7f9ef4003a90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:42:12.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0b7fe640 1 --2- 172.21.15.37:0/2752213374 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ef4004810 0x7f9ef4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:12.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0b7fe640 10 monclient(hunting): picked mon.a con 0x7f9ef4004810 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:42:12.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0b7fe640 1 --2- 172.21.15.37:0/2752213374 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ef4005420 0x7f9ef400ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:12.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0b7fe640 10 monclient(hunting): picked mon.c con 0x7f9ef4005420 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:42:12.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0b7fe640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:12.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0b7fe640 1 -- 172.21.15.37:0/2752213374 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0c09b8f0 con 0x7f9ef4004810 2024-04-26T00:42:12.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0b7fe640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:12.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0b7fe640 1 -- 172.21.15.37:0/2752213374 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef400c330 con 0x7f9ef4003a90 2024-04-26T00:42:12.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0b7fe640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:12.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0b7fe640 1 -- 172.21.15.37:0/2752213374 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9ef400c530 con 0x7f9ef4005420 2024-04-26T00:42:12.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0b7fe640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:12.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0b7fe640 10 monclient(hunting): _finish_auth 0 2024-04-26T00:42:12.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0bfff640 1 --2- 172.21.15.37:0/2752213374 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ef4004810 0x7f9ef4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:12.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f1149f640 1 --2- 172.21.15.37:0/2752213374 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ef4005420 0x7f9ef400ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:12.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0bfff640 10 monclient(hunting): get_auth_request con 0x7f9ef4004810 auth_method 0 2024-04-26T00:42:12.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:12.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:12.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:42:12.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f10c9e640 1 --2- 172.21.15.37:0/2752213374 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ef4003a90 0x7f9ef4003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:12.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f10c9e640 10 monclient(hunting): get_auth_request con 0x7f9ef4003a90 auth_method 0 2024-04-26T00:42:12.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f10c9e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:12.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f10c9e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:12.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f10c9e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:42:12.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f1149f640 10 monclient(hunting): get_auth_request con 0x7f9ef4005420 auth_method 0 2024-04-26T00:42:12.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f1149f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:12.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f1149f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:12.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f1149f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:42:12.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:12.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:12.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:42:12.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f10c9e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:12.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f10c9e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:12.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f10c9e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:42:12.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f1149f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:12.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f1149f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:12.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f1149f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:42:12.318 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0bfff640 10 monclient(hunting): handle_auth_done global_id 264177 payload 293 2024-04-26T00:42:12.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0bfff640 1 -- 172.21.15.37:0/2752213374 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ef4005420 msgr2=0x7f9ef400ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:42:12.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0bfff640 1 --2- 172.21.15.37:0/2752213374 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ef4005420 0x7f9ef400ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:12.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0bfff640 1 -- 172.21.15.37:0/2752213374 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ef4003a90 msgr2=0x7f9ef4003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:42:12.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0bfff640 1 --2- 172.21.15.37:0/2752213374 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ef4003a90 0x7f9ef4003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:12.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0bfff640 10 monclient: _finish_hunting 0 2024-04-26T00:42:12.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0bfff640 1 monclient: found mon.a 2024-04-26T00:42:12.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:42:12.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f1149f640 1 -- 172.21.15.37:0/2752213374 reap_dead start 2024-04-26T00:42:12.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0bfff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:42:12.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0bfff640 1 -- 172.21.15.37:0/2752213374 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ef4003530 con 0x7f9ef4004810 2024-04-26T00:42:12.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0bfff640 10 monclient: _finish_auth 0 2024-04-26T00:42:12.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0bfff640 10 monclient: _check_auth_tickets 2024-04-26T00:42:12.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:42:12.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f10c9e640 1 --2- 172.21.15.37:0/2752213374 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ef4003a90 0x7f9ef4003eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:42:12.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f12f02640 5 monclient: authenticate success, global_id 264177 2024-04-26T00:42:12.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.304+0000 7f9f12f02640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:42:12.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.308+0000 7f9f0bfff640 1 -- 172.21.15.37:0/2752213374 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ef4004810 msgr2=0x7f9ef4004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:12.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.308+0000 7f9f0bfff640 1 --2- 172.21.15.37:0/2752213374 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ef4004810 0x7f9ef4004c90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9f00001980 tx=0x7f9f0001c590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5796259aa6a6903f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:12.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.308+0000 7f9f0b7fe640 1 -- 172.21.15.37:0/2752213374 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f00001680 con 0x7f9ef4004810 2024-04-26T00:42:12.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.308+0000 7f9f0b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:12.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.308+0000 7f9f0b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:42:12.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.308+0000 7f9f0b7fe640 10 monclient: dump: 2024-04-26T00:42:12.327 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:12.327 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:12.327 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:12.327 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:12.327 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:12.327 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:12.327 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:12.327 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:12.327 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:12.327 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:12.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.308+0000 7f9f12f02640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:42:12.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.308+0000 7f9f0b7fe640 1 -- 172.21.15.37:0/2752213374 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f000a8070 con 0x7f9ef4004810 2024-04-26T00:42:12.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.308+0000 7f9f0b7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:42:12.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.308+0000 7f9f12f02640 10 monclient: get_monmap_and_config success 2024-04-26T00:42:12.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.308+0000 7f9f12f02640 4 set_mon_vals no callback set 2024-04-26T00:42:12.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.308+0000 7f9f0b7fe640 1 -- 172.21.15.37:0/2752213374 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f00007570 con 0x7f9ef4004810 2024-04-26T00:42:12.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.308+0000 7f9f0b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:12.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.308+0000 7f9f0b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:42:12.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.308+0000 7f9f0b7fe640 10 monclient: dump: 2024-04-26T00:42:12.328 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:12.328 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:12.328 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:12.329 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:12.329 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:12.329 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:12.329 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:12.329 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:12.329 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:12.329 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:12.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f12f02640 10 monclient: shutdown 2024-04-26T00:42:12.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f12f02640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:42:12.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f12f02640 1 -- 172.21.15.37:0/2752213374 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ef4004810 msgr2=0x7f9ef4004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:12.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f12f02640 1 --2- 172.21.15.37:0/2752213374 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ef4004810 0x7f9ef4004c90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9f00001980 tx=0x7f9f0001c590 comp rx=0 tx=0).stop 2024-04-26T00:42:12.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f12f02640 1 -- 172.21.15.37:0/2752213374 shutdown_connections 2024-04-26T00:42:12.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f12f02640 1 --2- 172.21.15.37:0/2752213374 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ef4004810 0x7f9ef4004c90 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:12.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f12f02640 1 -- 172.21.15.37:0/2752213374 >> 172.21.15.37:0/2752213374 conn(0x7f9f0c0a64e0 msgr2=0x7f9f0c0a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:42:12.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f12f02640 1 -- 172.21.15.37:0/2752213374 shutdown_connections 2024-04-26T00:42:12.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f12f02640 1 -- 172.21.15.37:0/2752213374 wait complete. 2024-04-26T00:42:12.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f12f02640 10 monclient: build_initial_monmap 2024-04-26T00:42:12.330 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f12f02640 10 monclient: monmap: 2024-04-26T00:42:12.330 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:42:12.330 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:42:12.330 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:42:12.330 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:42:12.330 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:42:12.331 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:42:12.331 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:42:12.331 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:42:12.331 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:42:12.331 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:12.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f12f02640 1 Processor -- start 2024-04-26T00:42:12.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f12f02640 1 -- start start 2024-04-26T00:42:12.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f12f02640 10 monclient: init 2024-04-26T00:42:12.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f12f02640 10 monclient: _reopen_session rank -1 2024-04-26T00:42:12.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f12f02640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:42:12.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f12f02640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ef4004810 0x7f9f0c1cbc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:12.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f12f02640 10 monclient(hunting): picked mon.noname-c con 0x7f9ef4004810 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:42:12.331 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f12f02640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f0c121580 0x7f9f0c135350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:12.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f12f02640 10 monclient(hunting): picked mon.noname-a con 0x7f9f0c121580 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:42:12.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f12f02640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f0c135890 0x7f9f0c11fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:12.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f12f02640 10 monclient(hunting): picked mon.noname-b con 0x7f9f0c135890 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:42:12.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f12f02640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:12.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f12f02640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0c136120 con 0x7f9f0c121580 2024-04-26T00:42:12.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f12f02640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:12.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f12f02640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0c135fa0 con 0x7f9f0c135890 2024-04-26T00:42:12.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f12f02640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:12.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f12f02640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9f0c1362a0 con 0x7f9ef4004810 2024-04-26T00:42:12.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f12f02640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:12.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f12f02640 10 monclient(hunting): authenticate will time out at 2313.622125s 2024-04-26T00:42:12.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f10c9e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ef4004810 0x7f9f0c1cbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:12.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f0bfff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f0c121580 0x7f9f0c135350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:12.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f10c9e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ef4004810 0x7f9f0c1cbc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:35324/0 (socket says 172.21.15.37:35324) 2024-04-26T00:42:12.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f10c9e640 1 -- 172.21.15.37:0/4030117390 learned_addr learned my addr 172.21.15.37:0/4030117390 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:42:12.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f10c9e640 10 monclient(hunting): get_auth_request con 0x7f9ef4004810 auth_method 0 2024-04-26T00:42:12.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f10c9e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:12.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f10c9e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:12.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f10c9e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:12.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f0bfff640 10 monclient(hunting): get_auth_request con 0x7f9f0c121580 auth_method 0 2024-04-26T00:42:12.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:12.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f0bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:12.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:12.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f1149f640 1 --2- 172.21.15.37:0/4030117390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f0c135890 0x7f9f0c11fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:12.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f10c9e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:12.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f10c9e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:12.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f10c9e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:12.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f1149f640 10 monclient(hunting): get_auth_request con 0x7f9f0c135890 auth_method 0 2024-04-26T00:42:12.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f1149f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:12.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f1149f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:12.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f1149f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:12.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:12.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:12.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:12.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f1149f640 1 -- 172.21.15.37:0/4030117390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f0c135890 msgr2=0x7f9f0c11fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:42:12.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f1149f640 1 -- 172.21.15.37:0/4030117390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f0c135890 msgr2=0x7f9f0c11fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:42:12.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f1149f640 1 --2- 172.21.15.37:0/4030117390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f0c135890 0x7f9f0c11fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:42:12.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f1149f640 1 --2- 172.21.15.37:0/4030117390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f0c135890 0x7f9f0c11fc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:42:12.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f10c9e640 10 monclient(hunting): handle_auth_done global_id 254297 payload 824 2024-04-26T00:42:12.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f10c9e640 1 -- 172.21.15.37:0/4030117390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f0c135890 msgr2=0x7f9f0c11fc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:42:12.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f10c9e640 1 --2- 172.21.15.37:0/4030117390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f0c135890 0x7f9f0c11fc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:12.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f10c9e640 1 -- 172.21.15.37:0/4030117390 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f0c121580 msgr2=0x7f9f0c135350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:12.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f10c9e640 1 --2- 172.21.15.37:0/4030117390 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f0c121580 0x7f9f0c135350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:12.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f10c9e640 10 monclient: _finish_hunting 0 2024-04-26T00:42:12.335 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f10c9e640 1 monclient: found mon.noname-c 2024-04-26T00:42:12.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f10c9e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:42:12.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f10c9e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:42:12.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f10c9e640 1 -- 172.21.15.37:0/4030117390 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f0c1c2440 con 0x7f9ef4004810 2024-04-26T00:42:12.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.312+0000 7f9f0bfff640 1 --2- 172.21.15.37:0/4030117390 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f0c121580 0x7f9f0c135350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:42:12.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.316+0000 7f9f10c9e640 1 -- 172.21.15.37:0/4030117390 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ef4004810 msgr2=0x7f9f0c1cbc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:12.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.316+0000 7f9f10c9e640 1 --2- 172.21.15.37:0/4030117390 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ef4004810 0x7f9f0c1cbc50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9efc011550 tx=0x7f9efc011a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:12.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.316+0000 7f9f097fa640 1 -- 172.21.15.37:0/4030117390 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9efc019070 con 0x7f9ef4004810 2024-04-26T00:42:12.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.316+0000 7f9f097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:12.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.316+0000 7f9f097fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:42:12.336 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.316+0000 7f9f097fa640 10 monclient: dump: 2024-04-26T00:42:12.336 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:12.336 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:12.337 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:12.337 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:12.337 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:12.337 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:12.337 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:12.337 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:12.337 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:12.337 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:12.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.316+0000 7f9f097fa640 10 monclient: _finish_auth 0 2024-04-26T00:42:12.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.316+0000 7f9f097fa640 10 monclient: _check_auth_tickets 2024-04-26T00:42:12.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.316+0000 7f9f097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:42:12.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.316+0000 7f9f097fa640 1 -- 172.21.15.37:0/4030117390 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9efc015aa0 con 0x7f9ef4004810 2024-04-26T00:42:12.337 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.316+0000 7f9f097fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:42:12.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.316+0000 7f9f097fa640 1 -- 172.21.15.37:0/4030117390 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9efc00f040 con 0x7f9ef4004810 2024-04-26T00:42:12.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.316+0000 7f9f097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:12.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.316+0000 7f9f097fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:42:12.338 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.316+0000 7f9f097fa640 10 monclient: dump: 2024-04-26T00:42:12.338 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:12.338 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:12.338 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:12.338 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:12.338 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:12.338 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:12.338 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:12.338 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:12.339 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:12.339 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:12.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.316+0000 7f9f12f02640 5 monclient: authenticate success, global_id 254297 2024-04-26T00:42:12.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.316+0000 7f9f0affd640 4 set_mon_vals no callback set 2024-04-26T00:42:12.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.316+0000 7f9f12f02640 10 monclient: _renew_subs 2024-04-26T00:42:12.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.316+0000 7f9f12f02640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:12.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.316+0000 7f9f12f02640 1 -- 172.21.15.37:0/4030117390 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f0c1ccb50 con 0x7f9ef4004810 2024-04-26T00:42:12.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.316+0000 7f9f12f02640 10 monclient: _renew_subs 2024-04-26T00:42:12.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.316+0000 7f9f12f02640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:12.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.316+0000 7f9f12f02640 1 -- 172.21.15.37:0/4030117390 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f0c122d10 con 0x7f9ef4004810 2024-04-26T00:42:12.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.316+0000 7f9f097fa640 1 -- 172.21.15.37:0/4030117390 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9efc01c070 con 0x7f9ef4004810 2024-04-26T00:42:12.339 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.316+0000 7f9f12f02640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:12.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.316+0000 7f9f12f02640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:12.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.316+0000 7f9f12f02640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:12.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.316+0000 7f9f12f02640 1 -- 172.21.15.37:0/4030117390 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f0c075910 con 0x7f9ef4004810 2024-04-26T00:42:12.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.316+0000 7f9f097fa640 1 --2- 172.21.15.37:0/4030117390 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9ef8035ed0 0x7f9ef8038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:12.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.316+0000 7f9f097fa640 1 -- 172.21.15.37:0/4030117390 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(884..884 src has 705..884) ==== 3975+0+0 (secure 0 0 0) 0x7f9efc048d10 con 0x7f9ef4004810 2024-04-26T00:42:12.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.316+0000 7f9f0bfff640 1 --2- 172.21.15.37:0/4030117390 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9ef8035ed0 0x7f9ef8038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:12.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.316+0000 7f9f0bfff640 10 monclient: get_auth_request con 0x7f9ef8035ed0 auth_method 0 2024-04-26T00:42:12.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.320+0000 7f9f0bfff640 1 --2- 172.21.15.37:0/4030117390 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9ef8035ed0 0x7f9ef8038390 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f9ef400c300 tx=0x7f9f00001530 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:12.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.320+0000 7f9f097fa640 1 -- 172.21.15.37:0/4030117390 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9f0c1c2440 con 0x7f9ef4004810 2024-04-26T00:42:12.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.320+0000 7f9f097fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:12.340 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.320+0000 7f9f097fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:42:12.473 INFO:tasks.rados.rados.0.smithi045.stdout:3922: finishing rollback tid 1 to smithi04526579-32 2024-04-26T00:42:12.473 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 2504 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-04-26T00:42:12.473 INFO:tasks.rados.rados.0.smithi045.stdout:3920: done (1 left) 2024-04-26T00:42:12.474 INFO:tasks.rados.rados.0.smithi045.stdout:3922: done (0 left) 2024-04-26T00:42:12.474 INFO:tasks.rados.rados.0.smithi045.stdout:3923: write oid 1 current snap is 434 2024-04-26T00:42:12.474 INFO:tasks.rados.rados.0.smithi045.stdout:3923: seq_num 1397 ranges {679379=415452,1599227=756146,3097215=382164} 2024-04-26T00:42:12.491 INFO:tasks.rados.rados.0.smithi045.stdout:3923: writing smithi04526579-1 from 679379 to 1094831 tid 1 2024-04-26T00:42:12.512 INFO:tasks.rados.rados.0.smithi045.stdout:3923: writing smithi04526579-1 from 1599227 to 2355373 tid 2 2024-04-26T00:42:12.529 INFO:tasks.rados.rados.0.smithi045.stdout:3923: writing smithi04526579-1 from 3097215 to 3479379 tid 3 2024-04-26T00:42:12.529 INFO:tasks.rados.rados.0.smithi045.stdout:3924: delete oid 29 current snap is 434 2024-04-26T00:42:12.529 INFO:tasks.rados.rados.0.smithi045.stdout:3923: finishing write tid 1 to smithi04526579-1 2024-04-26T00:42:12.529 INFO:tasks.rados.rados.0.smithi045.stdout:3923: finishing write tid 2 to smithi04526579-1 2024-04-26T00:42:12.533 INFO:tasks.rados.rados.0.smithi045.stdout:3924: done (1 left) 2024-04-26T00:42:12.534 INFO:tasks.rados.rados.0.smithi045.stdout:3925: rollback oid 17 current snap is 434 2024-04-26T00:42:12.534 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 17 to 421 2024-04-26T00:42:12.534 INFO:tasks.rados.rados.0.smithi045.stdout:3926: write (excl) oid 27 current snap is 434 2024-04-26T00:42:12.534 INFO:tasks.rados.rados.0.smithi045.stdout:3926: seq_num 1399 ranges {712993=682303,1832544=759815,3268414=244579} 2024-04-26T00:42:12.551 INFO:tasks.rados.rados.0.smithi045.stdout:3926: writing smithi04526579-27 from 712993 to 1395296 tid 1 2024-04-26T00:42:12.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.548+0000 7f9f12f02640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:42:12.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.548+0000 7f9f12f02640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:42:12.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.548+0000 7f9f12f02640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:12.552 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.548+0000 7f9f12f02640 1 -- 172.21.15.37:0/4030117390 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9f0c066f40 con 0x7f9ef4004810 2024-04-26T00:42:12.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.548+0000 7f9f097fa640 1 -- 172.21.15.37:0/4030117390 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7f9efc026350 con 0x7f9ef4004810 2024-04-26T00:42:12.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.548+0000 7f9f097fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:42:12.553 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.548+0000 7f9f097fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:42:12.554 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":158,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"c","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:42:12.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.552+0000 7f9f12f02640 1 -- 172.21.15.37:0/4030117390 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9ef8035ed0 msgr2=0x7f9ef8038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:12.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.552+0000 7f9f12f02640 1 --2- 172.21.15.37:0/4030117390 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9ef8035ed0 0x7f9ef8038390 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f9ef400c300 tx=0x7f9f00001530 comp rx=0 tx=0).stop 2024-04-26T00:42:12.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.552+0000 7f9f12f02640 10 monclient: shutdown 2024-04-26T00:42:12.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.552+0000 7f9f12f02640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:42:12.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.552+0000 7f9f12f02640 1 -- 172.21.15.37:0/4030117390 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ef4004810 msgr2=0x7f9f0c1cbc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:12.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.552+0000 7f9f12f02640 1 --2- 172.21.15.37:0/4030117390 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ef4004810 0x7f9f0c1cbc50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9efc011550 tx=0x7f9efc011a10 comp rx=0 tx=0).stop 2024-04-26T00:42:12.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.552+0000 7f9f12f02640 1 -- 172.21.15.37:0/4030117390 shutdown_connections 2024-04-26T00:42:12.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.552+0000 7f9f12f02640 1 --2- 172.21.15.37:0/4030117390 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9ef8035ed0 0x7f9ef8038390 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:12.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.552+0000 7f9f12f02640 1 --2- 172.21.15.37:0/4030117390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9f0c135890 0x7f9f0c11fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:12.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.552+0000 7f9f12f02640 1 --2- 172.21.15.37:0/4030117390 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f0c121580 0x7f9f0c135350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:12.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.552+0000 7f9f12f02640 1 --2- 172.21.15.37:0/4030117390 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ef4004810 0x7f9f0c1cbc50 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:12.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.552+0000 7f9f12f02640 1 -- 172.21.15.37:0/4030117390 >> 172.21.15.37:0/4030117390 conn(0x7f9f0c0a64e0 msgr2=0x7f9f0c1c1430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:42:12.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.552+0000 7f9f12f02640 1 -- 172.21.15.37:0/4030117390 shutdown_connections 2024-04-26T00:42:12.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:12.552+0000 7f9f12f02640 1 -- 172.21.15.37:0/4030117390 wait complete. 2024-04-26T00:42:12.565 INFO:tasks.rados.rados.0.smithi045.stdout:3926: writing smithi04526579-27 from 1832544 to 2592359 tid 2 2024-04-26T00:42:12.576 INFO:tasks.rados.rados.0.smithi045.stdout:3926: writing smithi04526579-27 from 3268414 to 3512993 tid 3 2024-04-26T00:42:12.577 INFO:tasks.rados.rados.0.smithi045.stdout:3927: delete oid 24 current snap is 434 2024-04-26T00:42:12.577 INFO:tasks.rados.rados.0.smithi045.stdout:3923: finishing write tid 3 to smithi04526579-1 2024-04-26T00:42:12.577 INFO:tasks.rados.rados.0.smithi045.stdout:3923: finishing write tid 4 to smithi04526579-1 2024-04-26T00:42:12.577 INFO:tasks.rados.rados.0.smithi045.stdout:3925: finishing rollback tid 1 to smithi04526579-17 2024-04-26T00:42:12.577 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 2634 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-04-26T00:42:12.577 INFO:tasks.rados.rados.0.smithi045.stdout:3923: finishing write tid 5 to smithi04526579-1 2024-04-26T00:42:12.577 INFO:tasks.rados.rados.0.smithi045.stdout:3923: oid 1 updating version 0 to 2834 2024-04-26T00:42:12.577 INFO:tasks.rados.rados.0.smithi045.stdout:3923: oid 1 updating version 2834 to 2835 2024-04-26T00:42:12.577 INFO:tasks.rados.rados.0.smithi045.stdout:3923: oid 1 version 2835 is already newer than 2832 2024-04-26T00:42:12.577 INFO:tasks.rados.rados.0.smithi045.stdout:3923: oid 1 version 2835 is already newer than 2833 2024-04-26T00:42:12.577 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 2835 (ObjNum 1397 snap 434 seq_num 1397) dirty exists 2024-04-26T00:42:12.577 INFO:tasks.rados.rados.0.smithi045.stdout:3923: left oid 1 (ObjNum 1397 snap 434 seq_num 1397) 2024-04-26T00:42:12.577 INFO:tasks.rados.rados.0.smithi045.stdout:3926: finishing write tid 1 to smithi04526579-27 2024-04-26T00:42:12.577 INFO:tasks.rados.rados.0.smithi045.stdout:3926: finishing write tid 2 to smithi04526579-27 2024-04-26T00:42:12.582 INFO:tasks.rados.rados.0.smithi045.stdout:3926: finishing write tid 3 to smithi04526579-27 2024-04-26T00:42:12.582 INFO:tasks.rados.rados.0.smithi045.stdout:3926: finishing write tid 4 to smithi04526579-27 2024-04-26T00:42:12.582 INFO:tasks.rados.rados.0.smithi045.stdout:3923: done (3 left) 2024-04-26T00:42:12.582 INFO:tasks.rados.rados.0.smithi045.stdout:3925: done (2 left) 2024-04-26T00:42:12.582 INFO:tasks.rados.rados.0.smithi045.stdout:3927: done (1 left) 2024-04-26T00:42:12.582 INFO:tasks.rados.rados.0.smithi045.stdout:3928: write oid 48 current snap is 434 2024-04-26T00:42:12.583 INFO:tasks.rados.rados.0.smithi045.stdout:3928: seq_num 1401 ranges {746607=432801,1833155=763484,3206907=339700} 2024-04-26T00:42:12.594 INFO:tasks.rados.rados.0.smithi045.stdout:3928: writing smithi04526579-48 from 746607 to 1179408 tid 1 2024-04-26T00:42:12.610 INFO:tasks.rados.rados.0.smithi045.stdout:3928: writing smithi04526579-48 from 1833155 to 2596639 tid 2 2024-04-26T00:42:12.619 INFO:tasks.rados.rados.0.smithi045.stdout:3928: writing smithi04526579-48 from 3206907 to 3546607 tid 3 2024-04-26T00:42:12.619 INFO:tasks.rados.rados.0.smithi045.stdout:3929: snap_create 2024-04-26T00:42:12.620 INFO:tasks.rados.rados.0.smithi045.stdout:3926: finishing write tid 5 to smithi04526579-27 2024-04-26T00:42:12.620 INFO:tasks.rados.rados.0.smithi045.stdout:3926: oid 27 updating version 0 to 2881 2024-04-26T00:42:12.620 INFO:tasks.rados.rados.0.smithi045.stdout:3926: oid 27 updating version 2881 to 2883 2024-04-26T00:42:12.620 INFO:tasks.rados.rados.0.smithi045.stdout:3926: oid 27 updating version 2883 to 2884 2024-04-26T00:42:12.620 INFO:tasks.rados.rados.0.smithi045.stdout:3926: oid 27 version 2884 is already newer than 2882 2024-04-26T00:42:12.620 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 2884 (ObjNum 1399 snap 434 seq_num 1399) dirty exists 2024-04-26T00:42:12.620 INFO:tasks.rados.rados.0.smithi045.stdout:3926: left oid 27 (ObjNum 1399 snap 434 seq_num 1399) 2024-04-26T00:42:12.620 INFO:tasks.rados.rados.0.smithi045.stdout:3928: finishing write tid 1 to smithi04526579-48 2024-04-26T00:42:12.620 INFO:tasks.rados.rados.0.smithi045.stdout:3928: finishing write tid 2 to smithi04526579-48 2024-04-26T00:42:12.624 INFO:tasks.rados.rados.0.smithi045.stdout:3928: finishing write tid 3 to smithi04526579-48 2024-04-26T00:42:12.625 INFO:tasks.rados.rados.0.smithi045.stdout:3928: finishing write tid 4 to smithi04526579-48 2024-04-26T00:42:12.625 INFO:tasks.rados.rados.0.smithi045.stdout:3928: finishing write tid 5 to smithi04526579-48 2024-04-26T00:42:12.625 INFO:tasks.rados.rados.0.smithi045.stdout:3928: oid 48 updating version 0 to 2877 2024-04-26T00:42:12.625 INFO:tasks.rados.rados.0.smithi045.stdout:3928: oid 48 updating version 2877 to 2878 2024-04-26T00:42:12.625 INFO:tasks.rados.rados.0.smithi045.stdout:3928: oid 48 updating version 2878 to 2879 2024-04-26T00:42:12.626 INFO:tasks.rados.rados.0.smithi045.stdout:3928: oid 48 version 2879 is already newer than 2876 2024-04-26T00:42:12.626 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 2879 (ObjNum 1401 snap 434 seq_num 1401) dirty exists 2024-04-26T00:42:12.626 INFO:tasks.rados.rados.0.smithi045.stdout:3928: left oid 48 (ObjNum 1401 snap 434 seq_num 1401) 2024-04-26T00:42:13.318 INFO:tasks.rados.rados.0.smithi045.stdout:3926: done (2 left) 2024-04-26T00:42:13.319 INFO:tasks.rados.rados.0.smithi045.stdout:3928: done (1 left) 2024-04-26T00:42:13.319 INFO:tasks.rados.rados.0.smithi045.stdout:3929: done (0 left) 2024-04-26T00:42:13.319 INFO:tasks.rados.rados.0.smithi045.stdout:3930: write (excl) oid 12 current snap is 435 2024-04-26T00:42:13.319 INFO:tasks.rados.rados.0.smithi045.stdout:3930: seq_num 1402 ranges {763414=508050,1891637=623495,3234330=329084} 2024-04-26T00:42:13.329 INFO:tasks.rados.rados.0.smithi045.stdout:3930: writing smithi04526579-12 from 763414 to 1271464 tid 1 2024-04-26T00:42:13.342 INFO:tasks.rados.rados.0.smithi045.stdout:3930: writing smithi04526579-12 from 1891637 to 2515132 tid 2 2024-04-26T00:42:13.351 INFO:tasks.rados.rados.0.smithi045.stdout:3930: writing smithi04526579-12 from 3234330 to 3563414 tid 3 2024-04-26T00:42:13.351 INFO:tasks.rados.rados.0.smithi045.stdout:3930: finishing write tid 1 to smithi04526579-12 2024-04-26T00:42:13.352 INFO:tasks.rados.rados.0.smithi045.stdout:3931: read oid 14 snap 429 2024-04-26T00:42:13.352 INFO:tasks.rados.rados.0.smithi045.stdout:3931: expect deleted 2024-04-26T00:42:13.352 INFO:tasks.rados.rados.0.smithi045.stdout:3932: delete oid 29 current snap is 435 2024-04-26T00:42:13.352 INFO:tasks.rados.rados.0.smithi045.stdout:3930: finishing write tid 2 to smithi04526579-12 2024-04-26T00:42:13.358 INFO:tasks.rados.rados.0.smithi045.stdout:3931: done (2 left) 2024-04-26T00:42:13.358 INFO:tasks.rados.rados.0.smithi045.stdout:3932: done (1 left) 2024-04-26T00:42:13.358 INFO:tasks.rados.rados.0.smithi045.stdout:3933: read oid 49 snap -1 2024-04-26T00:42:13.358 INFO:tasks.rados.rados.0.smithi045.stdout:3933: expect deleted 2024-04-26T00:42:13.358 INFO:tasks.rados.rados.0.smithi045.stdout:3934: rollback oid 30 current snap is 435 2024-04-26T00:42:13.358 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 30 to 432 2024-04-26T00:42:13.358 INFO:tasks.rados.rados.0.smithi045.stdout:3935: read oid 46 snap 401 2024-04-26T00:42:13.358 INFO:tasks.rados.rados.0.smithi045.stdout:3935: expect deleted 2024-04-26T00:42:13.358 INFO:tasks.rados.rados.0.smithi045.stdout:3936: copy_from oid 39 from oid 28 current snap is 435 2024-04-26T00:42:13.359 INFO:tasks.rados.rados.0.smithi045.stdout:3937: write (excl) oid 16 current snap is 435 2024-04-26T00:42:13.359 INFO:tasks.rados.rados.0.smithi045.stdout:3937: seq_num 1404 ranges {797028=658548,1892248=510811,3172823=424205} 2024-04-26T00:42:13.375 INFO:tasks.rados.rados.0.smithi045.stdout:3937: writing smithi04526579-16 from 797028 to 1455576 tid 1 2024-04-26T00:42:13.385 INFO:tasks.rados.rados.0.smithi045.stdout:3937: writing smithi04526579-16 from 1892248 to 2403059 tid 2 2024-04-26T00:42:13.396 INFO:tasks.rados.rados.0.smithi045.stdout:3937: writing smithi04526579-16 from 3172823 to 3597028 tid 3 2024-04-26T00:42:13.396 INFO:tasks.rados.rados.0.smithi045.stdout:3938: snap_remove snap 393 2024-04-26T00:42:14.330 INFO:tasks.rados.rados.0.smithi045.stdout:3938: done (6 left) 2024-04-26T00:42:14.331 INFO:tasks.rados.rados.0.smithi045.stdout:3939: snap_create 2024-04-26T00:42:14.331 INFO:tasks.rados.rados.0.smithi045.stdout:3930: finishing write tid 3 to smithi04526579-12 2024-04-26T00:42:14.331 INFO:tasks.rados.rados.0.smithi045.stdout:3930: finishing write tid 4 to smithi04526579-12 2024-04-26T00:42:14.331 INFO:tasks.rados.rados.0.smithi045.stdout:3930: finishing write tid 5 to smithi04526579-12 2024-04-26T00:42:14.331 INFO:tasks.rados.rados.0.smithi045.stdout:3930: oid 12 updating version 0 to 2492 2024-04-26T00:42:14.331 INFO:tasks.rados.rados.0.smithi045.stdout:3930: oid 12 version 2492 is already newer than 2491 2024-04-26T00:42:14.331 INFO:tasks.rados.rados.0.smithi045.stdout:3930: oid 12 updating version 2492 to 2493 2024-04-26T00:42:14.331 INFO:tasks.rados.rados.0.smithi045.stdout:3930: oid 12 updating version 2493 to 2494 2024-04-26T00:42:14.331 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 2494 (ObjNum 1402 snap 435 seq_num 1402) dirty exists 2024-04-26T00:42:14.331 INFO:tasks.rados.rados.0.smithi045.stdout:3930: left oid 12 (ObjNum 1402 snap 435 seq_num 1402) 2024-04-26T00:42:14.331 INFO:tasks.rados.rados.0.smithi045.stdout:3936: finishing copy_from to smithi04526579-39 2024-04-26T00:42:14.331 INFO:tasks.rados.rados.0.smithi045.stdout:3936: got expected ENOENT (src dne) 2024-04-26T00:42:14.331 INFO:tasks.rados.rados.0.smithi045.stdout:3936: finishing copy_from racing read to smithi04526579-39 2024-04-26T00:42:14.331 INFO:tasks.rados.rados.0.smithi045.stdout:3934: finishing rollback tid 0 to smithi04526579-30 2024-04-26T00:42:14.332 INFO:tasks.rados.rados.0.smithi045.stdout:3934: finishing rollback tid 1 to smithi04526579-30 2024-04-26T00:42:14.332 INFO:tasks.rados.rados.0.smithi045.stdout:3934: finishing rollback tid 2 to smithi04526579-30 2024-04-26T00:42:14.332 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 2877 (ObjNum 1363 snap 421 seq_num 1363) dirty exists 2024-04-26T00:42:14.332 INFO:tasks.rados.rados.0.smithi045.stdout:3937: finishing write tid 1 to smithi04526579-16 2024-04-26T00:42:14.332 INFO:tasks.rados.rados.0.smithi045.stdout:3937: finishing write tid 2 to smithi04526579-16 2024-04-26T00:42:14.332 INFO:tasks.rados.rados.0.smithi045.stdout:3937: finishing write tid 3 to smithi04526579-16 2024-04-26T00:42:14.332 INFO:tasks.rados.rados.0.smithi045.stdout:3937: finishing write tid 4 to smithi04526579-16 2024-04-26T00:42:14.332 INFO:tasks.rados.rados.0.smithi045.stdout:3937: finishing write tid 5 to smithi04526579-16 2024-04-26T00:42:14.332 INFO:tasks.rados.rados.0.smithi045.stdout:3937: oid 16 updating version 0 to 2502 2024-04-26T00:42:14.332 INFO:tasks.rados.rados.0.smithi045.stdout:3937: oid 16 version 2502 is already newer than 2501 2024-04-26T00:42:14.332 INFO:tasks.rados.rados.0.smithi045.stdout:3937: oid 16 updating version 2502 to 2503 2024-04-26T00:42:14.332 INFO:tasks.rados.rados.0.smithi045.stdout:3937: oid 16 updating version 2503 to 2504 2024-04-26T00:42:14.332 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 16 v 2504 (ObjNum 1404 snap 435 seq_num 1404) dirty exists 2024-04-26T00:42:14.332 INFO:tasks.rados.rados.0.smithi045.stdout:3937: left oid 16 (ObjNum 1404 snap 435 seq_num 1404) 2024-04-26T00:42:15.334 INFO:tasks.rados.rados.0.smithi045.stdout:3930: done (6 left) 2024-04-26T00:42:15.334 INFO:tasks.rados.rados.0.smithi045.stdout:3933: done (5 left) 2024-04-26T00:42:15.334 INFO:tasks.rados.rados.0.smithi045.stdout:3934: done (4 left) 2024-04-26T00:42:15.334 INFO:tasks.rados.rados.0.smithi045.stdout:3935: done (3 left) 2024-04-26T00:42:15.334 INFO:tasks.rados.rados.0.smithi045.stdout:3936: done (2 left) 2024-04-26T00:42:15.334 INFO:tasks.rados.rados.0.smithi045.stdout:3937: done (1 left) 2024-04-26T00:42:15.334 INFO:tasks.rados.rados.0.smithi045.stdout:3939: done (0 left) 2024-04-26T00:42:15.334 INFO:tasks.rados.rados.0.smithi045.stdout:3940: snap_remove snap 432 2024-04-26T00:42:15.577 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:42:15.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.704+0000 7fe2a02ef640 10 monclient: get_monmap_and_config 2024-04-26T00:42:15.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.704+0000 7fe2a02ef640 10 monclient: build_initial_monmap 2024-04-26T00:42:15.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.704+0000 7fe2a02ef640 10 monclient: monmap: 2024-04-26T00:42:15.708 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:42:15.708 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:15.708 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:42:15.709803+0000 2024-04-26T00:42:15.708 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:42:15.709803+0000 2024-04-26T00:42:15.708 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:42:15.708 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:15.708 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:42:15.708 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:42:15.709 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:42:15.709 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:42:15.709 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:42:15.709 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:15.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.704+0000 7fe2a02ef640 1 Processor -- start 2024-04-26T00:42:15.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.704+0000 7fe2a02ef640 1 -- start start 2024-04-26T00:42:15.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.704+0000 7fe2a02ef640 10 monclient: init 2024-04-26T00:42:15.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.704+0000 7fe2a02ef640 10 monclient: _reopen_session rank -1 2024-04-26T00:42:15.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.704+0000 7fe2a02ef640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-04-26T00:42:15.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.704+0000 7fe2a02ef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe298133b20 addr v1:172.21.15.37:6789/0 2024-04-26T00:42:15.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.704+0000 7fe2a02ef640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe298133f00 0x7fe298134350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:15.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.704+0000 7fe2a02ef640 10 monclient(hunting): picked mon.noname-c con 0x7fe298133f00 addr v2:172.21.15.45:3300/0 2024-04-26T00:42:15.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.704+0000 7fe2a02ef640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe298139050 0x7fe29813b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:15.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.704+0000 7fe2a02ef640 10 monclient(hunting): picked mon.noname-b con 0x7fe298139050 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:42:15.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.704+0000 7fe2a02ef640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe29809fe20 con 0x7fe298133b20 2024-04-26T00:42:15.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.704+0000 7fe2a02ef640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:15.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.704+0000 7fe2a02ef640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fe29809e620 con 0x7fe298133f00 2024-04-26T00:42:15.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.704+0000 7fe2a02ef640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:15.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.704+0000 7fe2a02ef640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe298072d00 con 0x7fe298139050 2024-04-26T00:42:15.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.704+0000 7fe2a02ef640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:15.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.704+0000 7fe2a02ef640 10 monclient(hunting): authenticate will time out at 2317.015942s 2024-04-26T00:42:15.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.708+0000 7fe29e08b640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fe298133b20 0x7fe29809e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:48246/0 (socket says 172.21.15.37:48246) 2024-04-26T00:42:15.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.708+0000 7fe29e08b640 1 -- 172.21.15.37:0/1021046544 learned_addr learned my addr 172.21.15.37:0/1021046544 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:42:15.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.708+0000 7fe29e88c640 1 --2- 172.21.15.37:0/1021046544 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe298139050 0x7fe29813b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:15.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.708+0000 7fe29e88c640 10 monclient(hunting): get_auth_request con 0x7fe298139050 auth_method 0 2024-04-26T00:42:15.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.708+0000 7fe29e88c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:15.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.708+0000 7fe29e88c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:15.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.708+0000 7fe29e88c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:15.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.708+0000 7fe29d88a640 1 --2- 172.21.15.37:0/1021046544 >> v2:172.21.15.45:3300/0 conn(0x7fe298133f00 0x7fe298134350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:15.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.708+0000 7fe29e88c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:15.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.708+0000 7fe29e88c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:15.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.708+0000 7fe29e88c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:15.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.708+0000 7fe29d88a640 10 monclient(hunting): get_auth_request con 0x7fe298133f00 auth_method 0 2024-04-26T00:42:15.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.708+0000 7fe29d88a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:15.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.708+0000 7fe29d88a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:15.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.708+0000 7fe29d88a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:15.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.708+0000 7fe29e88c640 10 monclient(hunting): handle_auth_done global_id 254300 payload 274 2024-04-26T00:42:15.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.708+0000 7fe29e88c640 1 -- 172.21.15.37:0/1021046544 >> v2:172.21.15.45:3300/0 conn(0x7fe298133f00 msgr2=0x7fe298134350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:42:15.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.708+0000 7fe29e88c640 1 --2- 172.21.15.37:0/1021046544 >> v2:172.21.15.45:3300/0 conn(0x7fe298133f00 0x7fe298134350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:15.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.708+0000 7fe29e88c640 1 -- 172.21.15.37:0/1021046544 >> v1:172.21.15.37:6789/0 conn(0x7fe298133b20 legacy=0x7fe29809e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:42:15.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.708+0000 7fe29e88c640 10 monclient: _finish_hunting 0 2024-04-26T00:42:15.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.708+0000 7fe29e88c640 1 monclient: found mon.noname-b 2024-04-26T00:42:15.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.708+0000 7fe29e88c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:42:15.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.708+0000 7fe29e88c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:42:15.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.708+0000 7fe29e88c640 1 -- 172.21.15.37:0/1021046544 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe29809ba60 con 0x7fe298139050 2024-04-26T00:42:15.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.712+0000 7fe29e88c640 1 -- 172.21.15.37:0/1021046544 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe298139050 msgr2=0x7fe29813b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:15.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.712+0000 7fe29e88c640 1 --2- 172.21.15.37:0/1021046544 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe298139050 0x7fe29813b440 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe290009d90 tx=0x7fe29001c610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b722a763abd02b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:15.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.712+0000 7fe29d089640 1 -- 172.21.15.37:0/1021046544 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2900ad070 con 0x7fe298139050 2024-04-26T00:42:15.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.712+0000 7fe29d089640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:15.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.712+0000 7fe29d089640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:42:15.715 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.712+0000 7fe29d089640 10 monclient: dump: 2024-04-26T00:42:15.715 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:15.715 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:15.715 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:15.715 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:15.715 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:15.716 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:15.716 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:15.716 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:15.716 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:15.716 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:15.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.712+0000 7fe29d089640 10 monclient: _finish_auth 0 2024-04-26T00:42:15.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.712+0000 7fe29d089640 10 monclient: _check_auth_tickets 2024-04-26T00:42:15.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.712+0000 7fe29d089640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:42:15.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.712+0000 7fe29d089640 1 -- 172.21.15.37:0/1021046544 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2900a9620 con 0x7fe298139050 2024-04-26T00:42:15.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.712+0000 7fe29d089640 10 monclient: handle_config config(0 keys) 2024-04-26T00:42:15.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.712+0000 7fe29d089640 1 -- 172.21.15.37:0/1021046544 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2900a99f0 con 0x7fe298139050 2024-04-26T00:42:15.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.712+0000 7fe29d089640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:15.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.712+0000 7fe29d089640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:42:15.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.712+0000 7fe29d089640 10 monclient: dump: 2024-04-26T00:42:15.717 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:15.717 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:15.717 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:15.717 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:15.717 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:15.717 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:15.717 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:15.717 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:15.717 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:15.717 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:15.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.712+0000 7fe2a02ef640 5 monclient: authenticate success, global_id 254300 2024-04-26T00:42:15.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.712+0000 7fe2a02ef640 10 monclient: get_monmap_and_config success 2024-04-26T00:42:15.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.712+0000 7fe2a02ef640 4 set_mon_vals no callback set 2024-04-26T00:42:15.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.712+0000 7fe2a02ef640 10 monclient: shutdown 2024-04-26T00:42:15.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.712+0000 7fe2a02ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:42:15.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.712+0000 7fe2a02ef640 1 -- 172.21.15.37:0/1021046544 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe298139050 msgr2=0x7fe29813b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:15.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.712+0000 7fe2a02ef640 1 --2- 172.21.15.37:0/1021046544 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe298139050 0x7fe29813b440 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe290009d90 tx=0x7fe29001c610 comp rx=0 tx=0).stop 2024-04-26T00:42:15.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.712+0000 7fe2a02ef640 1 -- 172.21.15.37:0/1021046544 shutdown_connections 2024-04-26T00:42:15.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.712+0000 7fe2a02ef640 1 --2- 172.21.15.37:0/1021046544 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe298139050 0x7fe29813b440 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:15.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.712+0000 7fe2a02ef640 1 --2- 172.21.15.37:0/1021046544 >> v2:172.21.15.45:3300/0 conn(0x7fe298133f00 0x7fe298134350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:15.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.712+0000 7fe2a02ef640 1 -- 172.21.15.37:0/1021046544 >> 172.21.15.37:0/1021046544 conn(0x7fe2980a1ad0 msgr2=0x7fe2980a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:42:15.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.712+0000 7fe2a02ef640 1 -- 172.21.15.37:0/1021046544 shutdown_connections 2024-04-26T00:42:15.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.712+0000 7fe2a02ef640 1 -- 172.21.15.37:0/1021046544 wait complete. 2024-04-26T00:42:15.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.712+0000 7fe2a02ef640 10 monclient: build_initial_monmap 2024-04-26T00:42:15.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.712+0000 7fe2a02ef640 10 monclient: monmap: 2024-04-26T00:42:15.724 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:42:15.724 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:42:15.724 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:42:15.724 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:42:15.724 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:42:15.724 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:42:15.724 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:42:15.724 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:42:15.724 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:42:15.724 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:15.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.712+0000 7fe2a02ef640 1 Processor -- start 2024-04-26T00:42:15.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.712+0000 7fe2a02ef640 1 -- start start 2024-04-26T00:42:15.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.712+0000 7fe2a02ef640 10 monclient: init 2024-04-26T00:42:15.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.712+0000 7fe2a02ef640 10 monclient: _reopen_session rank -1 2024-04-26T00:42:15.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.712+0000 7fe2a02ef640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:42:15.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.712+0000 7fe2a02ef640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe298133b20 0x7fe2981cef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:15.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe2a02ef640 10 monclient(hunting): picked mon.noname-a con 0x7fe298133b20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:42:15.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe2a02ef640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe298133f00 0x7fe2981cf4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:15.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe2a02ef640 10 monclient(hunting): picked mon.noname-b con 0x7fe298133f00 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:42:15.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe2a02ef640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe298139050 0x7fe2981cf9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:15.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe2a02ef640 10 monclient(hunting): picked mon.noname-c con 0x7fe298139050 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:42:15.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe2a02ef640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:15.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe2a02ef640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe2980a03e0 con 0x7fe298133b20 2024-04-26T00:42:15.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe2a02ef640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:15.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe2a02ef640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe2980a0260 con 0x7fe298133f00 2024-04-26T00:42:15.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe2a02ef640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:15.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe2a02ef640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe2980a0560 con 0x7fe298139050 2024-04-26T00:42:15.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe2a02ef640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:15.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe2a02ef640 10 monclient(hunting): authenticate will time out at 2317.024091s 2024-04-26T00:42:15.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe29e08b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe298133b20 0x7fe2981cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:15.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe29e88c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe298139050 0x7fe2981cf9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:15.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe29e08b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe298133b20 0x7fe2981cef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:57302/0 (socket says 172.21.15.37:57302) 2024-04-26T00:42:15.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe29e08b640 1 -- 172.21.15.37:0/3266677697 learned_addr learned my addr 172.21.15.37:0/3266677697 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:42:15.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe29e08b640 10 monclient(hunting): get_auth_request con 0x7fe298133b20 auth_method 0 2024-04-26T00:42:15.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe29e08b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:15.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe29e08b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:15.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe29e08b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:15.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe29e88c640 10 monclient(hunting): get_auth_request con 0x7fe298139050 auth_method 0 2024-04-26T00:42:15.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe29e88c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:15.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe29e88c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:15.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe29e88c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:15.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe29e08b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:15.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe29e08b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:15.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe29e08b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:15.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe29e88c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:15.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe29e88c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:15.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe29e88c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:15.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe29d88a640 1 --2- 172.21.15.37:0/3266677697 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe298133f00 0x7fe2981cf4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:15.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe29d88a640 10 monclient(hunting): get_auth_request con 0x7fe298133f00 auth_method 0 2024-04-26T00:42:15.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe29d88a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:15.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe29d88a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:15.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe29d88a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:15.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe29e88c640 10 monclient(hunting): handle_auth_done global_id 254303 payload 824 2024-04-26T00:42:15.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe29e88c640 1 -- 172.21.15.37:0/3266677697 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe298133f00 msgr2=0x7fe2981cf4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:15.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe29e88c640 1 --2- 172.21.15.37:0/3266677697 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe298133f00 0x7fe2981cf4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:15.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe29e88c640 1 -- 172.21.15.37:0/3266677697 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe298133b20 msgr2=0x7fe2981cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:15.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe29e88c640 1 --2- 172.21.15.37:0/3266677697 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe298133b20 0x7fe2981cef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:15.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe29e88c640 10 monclient: _finish_hunting 0 2024-04-26T00:42:15.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe29e88c640 1 monclient: found mon.noname-c 2024-04-26T00:42:15.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe29e88c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:42:15.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe29e88c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:42:15.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe29e88c640 1 -- 172.21.15.37:0/3266677697 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2980a0000 con 0x7fe298139050 2024-04-26T00:42:15.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe29e08b640 1 --2- 172.21.15.37:0/3266677697 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe298133b20 0x7fe2981cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:42:15.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe29e88c640 1 -- 172.21.15.37:0/3266677697 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe298139050 msgr2=0x7fe2981cf9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:15.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe29e88c640 1 --2- 172.21.15.37:0/3266677697 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe298139050 0x7fe2981cf9f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe298072e80 tx=0x7fe2900193b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:15.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe28effd640 1 -- 172.21.15.37:0/3266677697 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2900ad040 con 0x7fe298139050 2024-04-26T00:42:15.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe28effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:15.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe28effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:42:15.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe28effd640 10 monclient: dump: 2024-04-26T00:42:15.728 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:15.728 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:15.729 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:15.729 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:15.729 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:15.729 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:15.729 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:15.729 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:15.729 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:15.729 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:15.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe28effd640 10 monclient: _finish_auth 0 2024-04-26T00:42:15.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe28effd640 10 monclient: _check_auth_tickets 2024-04-26T00:42:15.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe28effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:42:15.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe28effd640 1 -- 172.21.15.37:0/3266677697 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2900b2650 con 0x7fe298139050 2024-04-26T00:42:15.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe28effd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:42:15.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe28effd640 1 -- 172.21.15.37:0/3266677697 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2900b2a40 con 0x7fe298139050 2024-04-26T00:42:15.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe28effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:15.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe28effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:42:15.730 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe28effd640 10 monclient: dump: 2024-04-26T00:42:15.730 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:15.730 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:15.730 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:15.730 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:15.730 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:15.730 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:15.730 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:15.730 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:15.730 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:15.731 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:15.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe2a02ef640 5 monclient: authenticate success, global_id 254303 2024-04-26T00:42:15.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe2a02ef640 10 monclient: _renew_subs 2024-04-26T00:42:15.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe2a02ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:15.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe2a02ef640 1 -- 172.21.15.37:0/3266677697 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2980a36e0 con 0x7fe298139050 2024-04-26T00:42:15.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe29c888640 4 set_mon_vals no callback set 2024-04-26T00:42:15.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe2a02ef640 10 monclient: _renew_subs 2024-04-26T00:42:15.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe2a02ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:15.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe2a02ef640 1 -- 172.21.15.37:0/3266677697 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe298138890 con 0x7fe298139050 2024-04-26T00:42:15.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe2a02ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:15.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe2a02ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:15.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe2a02ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:15.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.716+0000 7fe2a02ef640 1 -- 172.21.15.37:0/3266677697 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe29812f270 con 0x7fe298139050 2024-04-26T00:42:15.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.720+0000 7fe28effd640 1 -- 172.21.15.37:0/3266677697 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe2900b0070 con 0x7fe298139050 2024-04-26T00:42:15.731 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.720+0000 7fe28effd640 1 --2- 172.21.15.37:0/3266677697 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe27c035ed0 0x7fe27c038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:15.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.720+0000 7fe28effd640 1 -- 172.21.15.37:0/3266677697 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(887..887 src has 705..887) ==== 3963+0+0 (secure 0 0 0) 0x7fe2900e3260 con 0x7fe298139050 2024-04-26T00:42:15.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.720+0000 7fe29e08b640 1 --2- 172.21.15.37:0/3266677697 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe27c035ed0 0x7fe27c038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:15.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.720+0000 7fe29e08b640 10 monclient: get_auth_request con 0x7fe27c035ed0 auth_method 0 2024-04-26T00:42:15.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.720+0000 7fe29e08b640 1 --2- 172.21.15.37:0/3266677697 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe27c035ed0 0x7fe27c038390 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fe288007d60 tx=0x7fe288007cb0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:15.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.724+0000 7fe28effd640 1 -- 172.21.15.37:0/3266677697 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe29809fe20 con 0x7fe298139050 2024-04-26T00:42:15.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.724+0000 7fe28effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:15.732 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.724+0000 7fe28effd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:42:15.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.952+0000 7fe2a02ef640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:42:15.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.952+0000 7fe2a02ef640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:42:15.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.952+0000 7fe2a02ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:15.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.952+0000 7fe2a02ef640 1 -- 172.21.15.37:0/3266677697 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe298066f40 con 0x7fe298139050 2024-04-26T00:42:15.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.952+0000 7fe28effd640 1 -- 172.21.15.37:0/3266677697 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7fe2900c57a0 con 0x7fe298139050 2024-04-26T00:42:15.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.952+0000 7fe28effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:42:15.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.952+0000 7fe28effd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:42:15.957 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":158,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"c","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:42:15.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.956+0000 7fe2a02ef640 1 -- 172.21.15.37:0/3266677697 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe27c035ed0 msgr2=0x7fe27c038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:15.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.956+0000 7fe2a02ef640 1 --2- 172.21.15.37:0/3266677697 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe27c035ed0 0x7fe27c038390 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fe288007d60 tx=0x7fe288007cb0 comp rx=0 tx=0).stop 2024-04-26T00:42:15.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.956+0000 7fe2a02ef640 10 monclient: shutdown 2024-04-26T00:42:15.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.956+0000 7fe2a02ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:42:15.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.956+0000 7fe2a02ef640 1 -- 172.21.15.37:0/3266677697 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe298139050 msgr2=0x7fe2981cf9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:15.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.956+0000 7fe2a02ef640 1 --2- 172.21.15.37:0/3266677697 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe298139050 0x7fe2981cf9f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe298072e80 tx=0x7fe2900193b0 comp rx=0 tx=0).stop 2024-04-26T00:42:15.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.956+0000 7fe2a02ef640 1 -- 172.21.15.37:0/3266677697 shutdown_connections 2024-04-26T00:42:15.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.956+0000 7fe2a02ef640 1 --2- 172.21.15.37:0/3266677697 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe27c035ed0 0x7fe27c038390 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:15.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.956+0000 7fe2a02ef640 1 --2- 172.21.15.37:0/3266677697 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe298139050 0x7fe2981cf9f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:15.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.956+0000 7fe2a02ef640 1 --2- 172.21.15.37:0/3266677697 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe298133f00 0x7fe2981cf4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:15.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.956+0000 7fe2a02ef640 1 --2- 172.21.15.37:0/3266677697 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe298133b20 0x7fe2981cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:15.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.956+0000 7fe2a02ef640 1 -- 172.21.15.37:0/3266677697 >> 172.21.15.37:0/3266677697 conn(0x7fe2980a1ad0 msgr2=0x7fe29809c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:42:15.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.956+0000 7fe2a02ef640 1 -- 172.21.15.37:0/3266677697 shutdown_connections 2024-04-26T00:42:15.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:15.956+0000 7fe2a02ef640 1 -- 172.21.15.37:0/3266677697 wait complete. 2024-04-26T00:42:16.355 INFO:tasks.rados.rados.0.smithi045.stdout:3940: done (0 left) 2024-04-26T00:42:16.356 INFO:tasks.rados.rados.0.smithi045.stdout:3941: write (excl) oid 35 current snap is 436 2024-04-26T00:42:16.356 INFO:tasks.rados.rados.0.smithi045.stdout:3941: seq_num 1405 ranges {413835=733797,1550730=770822,2800246=557531} 2024-04-26T00:42:16.372 INFO:tasks.rados.rados.0.smithi045.stdout:3941: writing smithi04526579-35 from 413835 to 1147632 tid 1 2024-04-26T00:42:16.388 INFO:tasks.rados.rados.0.smithi045.stdout:3941: writing smithi04526579-35 from 1550730 to 2321552 tid 2 2024-04-26T00:42:16.403 INFO:tasks.rados.rados.0.smithi045.stdout:3941: writing smithi04526579-35 from 2800246 to 3357777 tid 3 2024-04-26T00:42:16.403 INFO:tasks.rados.rados.0.smithi045.stdout:3942: read oid 21 snap -1 2024-04-26T00:42:16.403 INFO:tasks.rados.rados.0.smithi045.stdout:3942: expect (ObjNum 1360 snap 420 seq_num 1360) 2024-04-26T00:42:16.403 INFO:tasks.rados.rados.0.smithi045.stdout:3943: snap_remove snap 435 2024-04-26T00:42:17.354 INFO:tasks.rados.rados.0.smithi045.stdout:3943: done (2 left) 2024-04-26T00:42:17.354 INFO:tasks.rados.rados.0.smithi045.stdout:3944: read oid 26 snap -1 2024-04-26T00:42:17.354 INFO:tasks.rados.rados.0.smithi045.stdout:3944: expect deleted 2024-04-26T00:42:17.354 INFO:tasks.rados.rados.0.smithi045.stdout:3945: read oid 3 snap -1 2024-04-26T00:42:17.354 INFO:tasks.rados.rados.0.smithi045.stdout:3945: expect (ObjNum 1363 snap 421 seq_num 1363) 2024-04-26T00:42:17.354 INFO:tasks.rados.rados.0.smithi045.stdout:3946: snap_create 2024-04-26T00:42:17.354 INFO:tasks.rados.rados.0.smithi045.stdout:3941: finishing write tid 1 to smithi04526579-35 2024-04-26T00:42:17.354 INFO:tasks.rados.rados.0.smithi045.stdout:3941: finishing write tid 2 to smithi04526579-35 2024-04-26T00:42:17.354 INFO:tasks.rados.rados.0.smithi045.stdout:3941: finishing write tid 3 to smithi04526579-35 2024-04-26T00:42:17.354 INFO:tasks.rados.rados.0.smithi045.stdout:3941: finishing write tid 4 to smithi04526579-35 2024-04-26T00:42:17.354 INFO:tasks.rados.rados.0.smithi045.stdout:3941: finishing write tid 5 to smithi04526579-35 2024-04-26T00:42:17.355 INFO:tasks.rados.rados.0.smithi045.stdout:3941: oid 35 updating version 0 to 2887 2024-04-26T00:42:17.355 INFO:tasks.rados.rados.0.smithi045.stdout:3941: oid 35 version 2887 is already newer than 2885 2024-04-26T00:42:17.355 INFO:tasks.rados.rados.0.smithi045.stdout:3941: oid 35 updating version 2887 to 2888 2024-04-26T00:42:17.355 INFO:tasks.rados.rados.0.smithi045.stdout:3941: oid 35 version 2888 is already newer than 2886 2024-04-26T00:42:17.355 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 35 v 2888 (ObjNum 1405 snap 436 seq_num 1405) dirty exists 2024-04-26T00:42:17.355 INFO:tasks.rados.rados.0.smithi045.stdout:3941: left oid 35 (ObjNum 1405 snap 436 seq_num 1405) 2024-04-26T00:42:17.355 INFO:tasks.rados.rados.0.smithi045.stdout:3942: expect (ObjNum 1360 snap 420 seq_num 1360) 2024-04-26T00:42:17.597 INFO:tasks.rados.rados.0.smithi045.stdout:3945: expect (ObjNum 1363 snap 421 seq_num 1363) 2024-04-26T00:42:18.367 INFO:tasks.rados.rados.0.smithi045.stdout:3941: done (4 left) 2024-04-26T00:42:18.367 INFO:tasks.rados.rados.0.smithi045.stdout:3942: done (3 left) 2024-04-26T00:42:18.367 INFO:tasks.rados.rados.0.smithi045.stdout:3944: done (2 left) 2024-04-26T00:42:18.367 INFO:tasks.rados.rados.0.smithi045.stdout:3945: done (1 left) 2024-04-26T00:42:18.367 INFO:tasks.rados.rados.0.smithi045.stdout:3946: done (0 left) 2024-04-26T00:42:18.368 INFO:tasks.rados.rados.0.smithi045.stdout:3947: write (excl) oid 36 current snap is 437 2024-04-26T00:42:18.368 INFO:tasks.rados.rados.0.smithi045.stdout:3947: seq_num 1406 ranges {430642=409046,1609212=514480,2827669=768803} 2024-04-26T00:42:18.381 INFO:tasks.rados.rados.0.smithi045.stdout:3947: writing smithi04526579-36 from 430642 to 839688 tid 1 2024-04-26T00:42:18.400 INFO:tasks.rados.rados.0.smithi045.stdout:3947: writing smithi04526579-36 from 1609212 to 2123692 tid 2 2024-04-26T00:42:18.419 INFO:tasks.rados.rados.0.smithi045.stdout:3947: writing smithi04526579-36 from 2827669 to 3596472 tid 3 2024-04-26T00:42:18.419 INFO:tasks.rados.rados.0.smithi045.stdout:3948: write (excl) oid 23 current snap is 437 2024-04-26T00:42:18.419 INFO:tasks.rados.rados.0.smithi045.stdout:3947: finishing write tid 1 to smithi04526579-36 2024-04-26T00:42:18.419 INFO:tasks.rados.rados.0.smithi045.stdout:3947: finishing write tid 2 to smithi04526579-36 2024-04-26T00:42:18.419 INFO:tasks.rados.rados.0.smithi045.stdout:3948: seq_num 1407 ranges {447449=600648,1784047=774491,2971445=676004} 2024-04-26T00:42:18.431 INFO:tasks.rados.rados.0.smithi045.stdout:3948: writing smithi04526579-23 from 447449 to 1048097 tid 1 2024-04-26T00:42:18.450 INFO:tasks.rados.rados.0.smithi045.stdout:3948: writing smithi04526579-23 from 1784047 to 2558538 tid 2 2024-04-26T00:42:18.462 INFO:tasks.rados.rados.0.smithi045.stdout:3948: writing smithi04526579-23 from 2971445 to 3647449 tid 3 2024-04-26T00:42:18.462 INFO:tasks.rados.rados.0.smithi045.stdout:3949: snap_create 2024-04-26T00:42:18.462 INFO:tasks.rados.rados.0.smithi045.stdout:3947: finishing write tid 3 to smithi04526579-36 2024-04-26T00:42:18.462 INFO:tasks.rados.rados.0.smithi045.stdout:3947: finishing write tid 4 to smithi04526579-36 2024-04-26T00:42:18.462 INFO:tasks.rados.rados.0.smithi045.stdout:3947: finishing write tid 5 to smithi04526579-36 2024-04-26T00:42:18.462 INFO:tasks.rados.rados.0.smithi045.stdout:3947: oid 36 updating version 0 to 2831 2024-04-26T00:42:18.462 INFO:tasks.rados.rados.0.smithi045.stdout:3947: oid 36 version 2831 is already newer than 2830 2024-04-26T00:42:18.462 INFO:tasks.rados.rados.0.smithi045.stdout:3947: oid 36 version 2831 is already newer than 2829 2024-04-26T00:42:18.462 INFO:tasks.rados.rados.0.smithi045.stdout:3947: oid 36 version 2831 is already newer than 2828 2024-04-26T00:42:18.462 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 36 v 2831 (ObjNum 1406 snap 437 seq_num 1406) dirty exists 2024-04-26T00:42:18.463 INFO:tasks.rados.rados.0.smithi045.stdout:3947: left oid 36 (ObjNum 1406 snap 437 seq_num 1406) 2024-04-26T00:42:18.463 INFO:tasks.rados.rados.0.smithi045.stdout:3948: finishing write tid 1 to smithi04526579-23 2024-04-26T00:42:18.463 INFO:tasks.rados.rados.0.smithi045.stdout:3948: finishing write tid 2 to smithi04526579-23 2024-04-26T00:42:18.468 INFO:tasks.rados.rados.0.smithi045.stdout:3948: finishing write tid 3 to smithi04526579-23 2024-04-26T00:42:18.470 INFO:tasks.rados.rados.0.smithi045.stdout:3948: finishing write tid 4 to smithi04526579-23 2024-04-26T00:42:18.471 INFO:tasks.rados.rados.0.smithi045.stdout:3948: finishing write tid 5 to smithi04526579-23 2024-04-26T00:42:18.471 INFO:tasks.rados.rados.0.smithi045.stdout:3948: oid 23 updating version 0 to 2498 2024-04-26T00:42:18.471 INFO:tasks.rados.rados.0.smithi045.stdout:3948: oid 23 version 2498 is already newer than 2497 2024-04-26T00:42:18.471 INFO:tasks.rados.rados.0.smithi045.stdout:3948: oid 23 version 2498 is already newer than 2495 2024-04-26T00:42:18.471 INFO:tasks.rados.rados.0.smithi045.stdout:3948: oid 23 version 2498 is already newer than 2496 2024-04-26T00:42:18.471 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 2498 (ObjNum 1407 snap 437 seq_num 1407) dirty exists 2024-04-26T00:42:18.471 INFO:tasks.rados.rados.0.smithi045.stdout:3948: left oid 23 (ObjNum 1407 snap 437 seq_num 1407) 2024-04-26T00:42:18.977 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:42:19.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.104+0000 7f909ee1d640 10 monclient: get_monmap_and_config 2024-04-26T00:42:19.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.104+0000 7f909ee1d640 10 monclient: build_initial_monmap 2024-04-26T00:42:19.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.104+0000 7f909ee1d640 10 monclient: monmap: 2024-04-26T00:42:19.108 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:42:19.109 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:19.109 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:42:19.110006+0000 2024-04-26T00:42:19.109 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:42:19.110006+0000 2024-04-26T00:42:19.109 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:42:19.109 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:19.109 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:42:19.109 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:42:19.109 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:42:19.109 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:42:19.109 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:42:19.109 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:19.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.104+0000 7f909ee1d640 1 Processor -- start 2024-04-26T00:42:19.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.104+0000 7f909ee1d640 1 -- start start 2024-04-26T00:42:19.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.104+0000 7f909ee1d640 10 monclient: init 2024-04-26T00:42:19.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.104+0000 7f909ee1d640 10 monclient: _reopen_session rank -1 2024-04-26T00:42:19.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.104+0000 7f909ee1d640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-04-26T00:42:19.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.104+0000 7f909ee1d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9098133b20 0x7f909809e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:19.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.104+0000 7f909ee1d640 10 monclient(hunting): picked mon.noname-a con 0x7f9098133b20 addr v2:172.21.15.37:3300/0 2024-04-26T00:42:19.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.104+0000 7f909ee1d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9098133f00 0x7f9098134350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:19.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.104+0000 7f909ee1d640 10 monclient(hunting): picked mon.noname-c con 0x7f9098133f00 addr v2:172.21.15.45:3300/0 2024-04-26T00:42:19.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.104+0000 7f909ee1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9098130770 addr v1:172.21.15.45:6789/0 2024-04-26T00:42:19.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.104+0000 7f909ee1d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f909809fe20 con 0x7f9098130770 2024-04-26T00:42:19.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.104+0000 7f909ee1d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:19.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.104+0000 7f909ee1d640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f9098072d40 con 0x7f9098133b20 2024-04-26T00:42:19.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.104+0000 7f909ee1d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:19.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.104+0000 7f909ee1d640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f909809ba60 con 0x7f9098133f00 2024-04-26T00:42:19.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.104+0000 7f909ee1d640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:19.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.104+0000 7f909ee1d640 10 monclient(hunting): authenticate will time out at 2320.415851s 2024-04-26T00:42:19.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.108+0000 7f909cbb9640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9098133b20 0x7f909809e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:19.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.108+0000 7f909cbb9640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9098133b20 0x7f909809e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:57308/0 (socket says 172.21.15.37:57308) 2024-04-26T00:42:19.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.108+0000 7f909cbb9640 1 -- 172.21.15.37:0/3690784891 learned_addr learned my addr 172.21.15.37:0/3690784891 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:42:19.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.108+0000 7f909cbb9640 10 monclient(hunting): get_auth_request con 0x7f9098133b20 auth_method 0 2024-04-26T00:42:19.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.108+0000 7f909cbb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:19.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.108+0000 7f909cbb9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:19.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.108+0000 7f909cbb9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:19.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.108+0000 7f909cbb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:19.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.108+0000 7f909cbb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:19.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.108+0000 7f909cbb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:19.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.108+0000 7f9097fff640 1 --2- 172.21.15.37:0/3690784891 >> v2:172.21.15.45:3300/0 conn(0x7f9098133f00 0x7f9098134350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:19.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.108+0000 7f9097fff640 10 monclient(hunting): get_auth_request con 0x7f9098133f00 auth_method 0 2024-04-26T00:42:19.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.108+0000 7f9097fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:19.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.108+0000 7f9097fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:19.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.108+0000 7f9097fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:19.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.108+0000 7f909cbb9640 10 monclient(hunting): handle_auth_done global_id 264186 payload 274 2024-04-26T00:42:19.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.108+0000 7f909cbb9640 1 -- 172.21.15.37:0/3690784891 >> v2:172.21.15.45:3300/0 conn(0x7f9098133f00 msgr2=0x7f9098134350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:42:19.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.108+0000 7f909cbb9640 1 --2- 172.21.15.37:0/3690784891 >> v2:172.21.15.45:3300/0 conn(0x7f9098133f00 0x7f9098134350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:19.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.108+0000 7f909cbb9640 1 -- 172.21.15.37:0/3690784891 >> v1:172.21.15.45:6789/0 conn(0x7f9098130770 legacy=0x7f9098130b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:42:19.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.108+0000 7f909cbb9640 10 monclient: _finish_hunting 0 2024-04-26T00:42:19.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.108+0000 7f909cbb9640 1 monclient: found mon.noname-a 2024-04-26T00:42:19.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.108+0000 7f909cbb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:42:19.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.108+0000 7f909cbb9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:42:19.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.108+0000 7f909cbb9640 1 -- 172.21.15.37:0/3690784891 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f909809dd10 con 0x7f9098133b20 2024-04-26T00:42:19.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909cbb9640 1 -- 172.21.15.37:0/3690784891 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9098133b20 msgr2=0x7f909809e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:19.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909cbb9640 1 --2- 172.21.15.37:0/3690784891 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9098133b20 0x7f909809e9d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f909809f670 tx=0x7f908801c250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1167d413a9899ba4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:19.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f90977fe640 1 -- 172.21.15.37:0/3690784891 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90880b0070 con 0x7f9098133b20 2024-04-26T00:42:19.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f90977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:19.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f90977fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:42:19.123 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f90977fe640 10 monclient: dump: 2024-04-26T00:42:19.123 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:19.123 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:19.123 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:19.123 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:19.123 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:19.123 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:19.123 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:19.123 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:19.124 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:19.124 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:19.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f90977fe640 10 monclient: _finish_auth 0 2024-04-26T00:42:19.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f90977fe640 10 monclient: _check_auth_tickets 2024-04-26T00:42:19.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f90977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:42:19.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f90977fe640 1 -- 172.21.15.37:0/3690784891 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90880ab600 con 0x7f9098133b20 2024-04-26T00:42:19.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f90977fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:42:19.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f90977fe640 1 -- 172.21.15.37:0/3690784891 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90880ab9d0 con 0x7f9098133b20 2024-04-26T00:42:19.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909ee1d640 5 monclient: authenticate success, global_id 264186 2024-04-26T00:42:19.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909ee1d640 10 monclient: get_monmap_and_config success 2024-04-26T00:42:19.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f90977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:19.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909ee1d640 4 set_mon_vals no callback set 2024-04-26T00:42:19.124 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f90977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:42:19.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f90977fe640 10 monclient: dump: 2024-04-26T00:42:19.125 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:19.125 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:19.125 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:19.125 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:19.125 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:19.125 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:19.125 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:19.125 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:19.125 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:19.126 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:19.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909ee1d640 10 monclient: shutdown 2024-04-26T00:42:19.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909ee1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:42:19.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909ee1d640 1 -- 172.21.15.37:0/3690784891 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9098133b20 msgr2=0x7f909809e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:19.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909ee1d640 1 --2- 172.21.15.37:0/3690784891 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9098133b20 0x7f909809e9d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f909809f670 tx=0x7f908801c250 comp rx=0 tx=0).stop 2024-04-26T00:42:19.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909ee1d640 1 -- 172.21.15.37:0/3690784891 shutdown_connections 2024-04-26T00:42:19.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909ee1d640 1 --2- 172.21.15.37:0/3690784891 >> v2:172.21.15.45:3300/0 conn(0x7f9098133f00 0x7f9098134350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:19.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909ee1d640 1 --2- 172.21.15.37:0/3690784891 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9098133b20 0x7f909809e9d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:19.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909ee1d640 1 -- 172.21.15.37:0/3690784891 >> 172.21.15.37:0/3690784891 conn(0x7f90980a1ad0 msgr2=0x7f90980a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:42:19.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909ee1d640 1 -- 172.21.15.37:0/3690784891 shutdown_connections 2024-04-26T00:42:19.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909ee1d640 1 -- 172.21.15.37:0/3690784891 wait complete. 2024-04-26T00:42:19.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909ee1d640 10 monclient: build_initial_monmap 2024-04-26T00:42:19.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909ee1d640 10 monclient: monmap: 2024-04-26T00:42:19.127 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:42:19.127 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:42:19.127 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:42:19.127 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:42:19.127 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:42:19.127 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:42:19.127 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:42:19.127 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:42:19.127 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:42:19.127 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:19.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909ee1d640 1 Processor -- start 2024-04-26T00:42:19.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909ee1d640 1 -- start start 2024-04-26T00:42:19.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909ee1d640 10 monclient: init 2024-04-26T00:42:19.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909ee1d640 10 monclient: _reopen_session rank -1 2024-04-26T00:42:19.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909ee1d640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:42:19.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909ee1d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9098130770 0x7f9098130060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:19.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909ee1d640 10 monclient(hunting): picked mon.noname-c con 0x7f9098130770 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:42:19.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909ee1d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9098133b20 0x7f909806dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:19.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909ee1d640 10 monclient(hunting): picked mon.noname-a con 0x7f9098133b20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:42:19.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909ee1d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9098133f00 0x7f909806e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:19.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909ee1d640 10 monclient(hunting): picked mon.noname-b con 0x7f9098133f00 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:42:19.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909ee1d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:19.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909ee1d640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f909813ea10 con 0x7f9098133b20 2024-04-26T00:42:19.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909ee1d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:19.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909ee1d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f909813e890 con 0x7f9098133f00 2024-04-26T00:42:19.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909ee1d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:19.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909ee1d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f909813eb90 con 0x7f9098130770 2024-04-26T00:42:19.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909ee1d640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:19.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909ee1d640 10 monclient(hunting): authenticate will time out at 2320.423329s 2024-04-26T00:42:19.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909cbb9640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9098130770 0x7f9098130060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:19.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909cbb9640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9098130770 0x7f9098130060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:35338/0 (socket says 172.21.15.37:35338) 2024-04-26T00:42:19.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909cbb9640 1 -- 172.21.15.37:0/3638254573 learned_addr learned my addr 172.21.15.37:0/3638254573 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:42:19.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f9097fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9098133b20 0x7f909806dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:19.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909cbb9640 10 monclient(hunting): get_auth_request con 0x7f9098130770 auth_method 0 2024-04-26T00:42:19.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909cbb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:19.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909cbb9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:19.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909cbb9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:19.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f9097fff640 10 monclient(hunting): get_auth_request con 0x7f9098133b20 auth_method 0 2024-04-26T00:42:19.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f9097fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:19.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f9097fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:19.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f9097fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:19.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909cbb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:19.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909cbb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:19.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909cbb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:19.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f9097fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:19.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f9097fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:19.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f9097fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:19.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909d3ba640 1 --2- 172.21.15.37:0/3638254573 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9098133f00 0x7f909806e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:19.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909d3ba640 10 monclient(hunting): get_auth_request con 0x7f9098133f00 auth_method 0 2024-04-26T00:42:19.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909d3ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:19.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909d3ba640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:19.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909d3ba640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:19.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f9097fff640 10 monclient(hunting): handle_auth_done global_id 264189 payload 824 2024-04-26T00:42:19.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f9097fff640 1 -- 172.21.15.37:0/3638254573 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9098130770 msgr2=0x7f9098130060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:19.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f9097fff640 1 --2- 172.21.15.37:0/3638254573 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9098130770 0x7f9098130060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:19.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f9097fff640 1 -- 172.21.15.37:0/3638254573 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9098133f00 msgr2=0x7f909806e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:19.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f9097fff640 1 --2- 172.21.15.37:0/3638254573 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9098133f00 0x7f909806e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:19.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f9097fff640 10 monclient: _finish_hunting 0 2024-04-26T00:42:19.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f9097fff640 1 monclient: found mon.noname-a 2024-04-26T00:42:19.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f9097fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:42:19.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f9097fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:42:19.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f9097fff640 1 -- 172.21.15.37:0/3638254573 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90980a0000 con 0x7f9098133b20 2024-04-26T00:42:19.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.112+0000 7f909cbb9640 1 --2- 172.21.15.37:0/3638254573 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9098130770 0x7f9098130060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:42:19.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.116+0000 7f9097fff640 1 -- 172.21.15.37:0/3638254573 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9098133b20 msgr2=0x7f909806dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:19.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.116+0000 7f9097fff640 1 --2- 172.21.15.37:0/3638254573 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9098133b20 0x7f909806dae0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f908c007e90 tx=0x7f908c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:19.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.116+0000 7f90957fa640 1 -- 172.21.15.37:0/3638254573 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f908c013790 con 0x7f9098133b20 2024-04-26T00:42:19.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.116+0000 7f90957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:19.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.116+0000 7f90957fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:42:19.132 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.116+0000 7f90957fa640 10 monclient: dump: 2024-04-26T00:42:19.132 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:19.132 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:19.132 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:19.132 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:19.133 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:19.133 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:19.133 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:19.133 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:19.133 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:19.133 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:19.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.116+0000 7f90957fa640 10 monclient: _finish_auth 0 2024-04-26T00:42:19.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.116+0000 7f90957fa640 10 monclient: _check_auth_tickets 2024-04-26T00:42:19.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.116+0000 7f90957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:42:19.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.116+0000 7f90957fa640 1 -- 172.21.15.37:0/3638254573 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f908c019070 con 0x7f9098133b20 2024-04-26T00:42:19.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.116+0000 7f90957fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:42:19.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.116+0000 7f90957fa640 1 -- 172.21.15.37:0/3638254573 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f908c013b50 con 0x7f9098133b20 2024-04-26T00:42:19.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.116+0000 7f90957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:19.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.116+0000 7f90957fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:42:19.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.116+0000 7f90957fa640 10 monclient: dump: 2024-04-26T00:42:19.134 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:19.134 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:19.134 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:19.134 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:19.134 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:19.134 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:19.134 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:19.134 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:19.134 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:19.135 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:19.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.116+0000 7f909ee1d640 5 monclient: authenticate success, global_id 264189 2024-04-26T00:42:19.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.116+0000 7f90967fc640 4 set_mon_vals no callback set 2024-04-26T00:42:19.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.116+0000 7f909ee1d640 10 monclient: _renew_subs 2024-04-26T00:42:19.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.116+0000 7f909ee1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:42:19.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.116+0000 7f909ee1d640 1 -- 172.21.15.37:0/3638254573 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90980a4550 con 0x7f9098133b20 2024-04-26T00:42:19.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.116+0000 7f909ee1d640 10 monclient: _renew_subs 2024-04-26T00:42:19.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.116+0000 7f909ee1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:42:19.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.116+0000 7f909ee1d640 1 -- 172.21.15.37:0/3638254573 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90981394a0 con 0x7f9098133b20 2024-04-26T00:42:19.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.116+0000 7f909ee1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:19.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.116+0000 7f909ee1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:19.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.116+0000 7f909ee1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:42:19.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.116+0000 7f90957fa640 1 -- 172.21.15.37:0/3638254573 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f908c013cf0 con 0x7f9098133b20 2024-04-26T00:42:19.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.116+0000 7f909ee1d640 1 -- 172.21.15.37:0/3638254573 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9098139050 con 0x7f9098133b20 2024-04-26T00:42:19.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.116+0000 7f90957fa640 1 --2- 172.21.15.37:0/3638254573 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9084035f00 0x7f90840383c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:19.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.116+0000 7f90957fa640 1 -- 172.21.15.37:0/3638254573 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(890..890 src has 822..890) ==== 3979+0+0 (secure 0 0 0) 0x7f908c0492b0 con 0x7f9098133b20 2024-04-26T00:42:19.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.120+0000 7f909cbb9640 1 --2- 172.21.15.37:0/3638254573 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9084035f00 0x7f90840383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:19.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.120+0000 7f909cbb9640 10 monclient: get_auth_request con 0x7f9084035f00 auth_method 0 2024-04-26T00:42:19.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.120+0000 7f909cbb9640 1 --2- 172.21.15.37:0/3638254573 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9084035f00 0x7f90840383c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f908801c290 tx=0x7f90880a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:19.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.124+0000 7f90957fa640 1 -- 172.21.15.37:0/3638254573 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9098139050 con 0x7f9098133b20 2024-04-26T00:42:19.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.124+0000 7f90957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:19.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.124+0000 7f90957fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:42:19.347 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.344+0000 7f909ee1d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:42:19.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.344+0000 7f909ee1d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:42:19.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.344+0000 7f909ee1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:42:19.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.344+0000 7f909ee1d640 1 -- 172.21.15.37:0/3638254573 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9098066f40 con 0x7f9098133b20 2024-04-26T00:42:19.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.344+0000 7f90957fa640 1 -- 172.21.15.37:0/3638254573 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f908c047fd0 con 0x7f9098133b20 2024-04-26T00:42:19.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.344+0000 7f90957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:42:19.348 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.344+0000 7f90957fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:42:19.349 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":160,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:42:19.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.348+0000 7f909ee1d640 1 -- 172.21.15.37:0/3638254573 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9084035f00 msgr2=0x7f90840383c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:19.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.348+0000 7f909ee1d640 1 --2- 172.21.15.37:0/3638254573 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9084035f00 0x7f90840383c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f908801c290 tx=0x7f90880a6000 comp rx=0 tx=0).stop 2024-04-26T00:42:19.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.348+0000 7f909ee1d640 10 monclient: shutdown 2024-04-26T00:42:19.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.348+0000 7f909ee1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:42:19.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.348+0000 7f909ee1d640 1 -- 172.21.15.37:0/3638254573 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9098133b20 msgr2=0x7f909806dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:19.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.348+0000 7f909ee1d640 1 --2- 172.21.15.37:0/3638254573 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9098133b20 0x7f909806dae0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f908c007e90 tx=0x7f908c00e4b0 comp rx=0 tx=0).stop 2024-04-26T00:42:19.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.348+0000 7f909ee1d640 1 -- 172.21.15.37:0/3638254573 shutdown_connections 2024-04-26T00:42:19.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.348+0000 7f909ee1d640 1 --2- 172.21.15.37:0/3638254573 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9084035f00 0x7f90840383c0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:19.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.348+0000 7f909ee1d640 1 --2- 172.21.15.37:0/3638254573 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9098133f00 0x7f909806e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:19.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.348+0000 7f909ee1d640 1 --2- 172.21.15.37:0/3638254573 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9098133b20 0x7f909806dae0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:19.351 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.348+0000 7f909ee1d640 1 --2- 172.21.15.37:0/3638254573 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9098130770 0x7f9098130060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:19.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.348+0000 7f909ee1d640 1 -- 172.21.15.37:0/3638254573 >> 172.21.15.37:0/3638254573 conn(0x7f90980a1ad0 msgr2=0x7f9098071cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:42:19.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.348+0000 7f909ee1d640 1 -- 172.21.15.37:0/3638254573 shutdown_connections 2024-04-26T00:42:19.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.348+0000 7f909ee1d640 1 -- 172.21.15.37:0/3638254573 wait complete. 2024-04-26T00:42:19.368 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:42:19.368 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:42:19.368 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:42:19.369 INFO:tasks.rados.rados.0.smithi045.stdout:3947: done (2 left) 2024-04-26T00:42:19.369 INFO:tasks.rados.rados.0.smithi045.stdout:3948: done (1 left) 2024-04-26T00:42:19.370 INFO:tasks.rados.rados.0.smithi045.stdout:3949: done (0 left) 2024-04-26T00:42:19.370 INFO:tasks.rados.rados.0.smithi045.stdout:3950: read oid 8 snap -1 2024-04-26T00:42:19.370 INFO:tasks.rados.rados.0.smithi045.stdout:3950: expect (ObjNum 1395 snap 432 seq_num 1395) 2024-04-26T00:42:19.370 INFO:tasks.rados.rados.0.smithi045.stdout:3951: rollback oid 12 current snap is 438 2024-04-26T00:42:19.370 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 12 to 424 2024-04-26T00:42:19.370 INFO:tasks.rados.rados.0.smithi045.stdout:3952: write oid 27 current snap is 438 2024-04-26T00:42:19.371 INFO:tasks.rados.rados.0.smithi045.stdout:3952: seq_num 1408 ranges {464256=675897,1726176=518149,2882515=507700} 2024-04-26T00:42:19.385 INFO:tasks.rados.rados.0.smithi045.stdout:3952: writing smithi04526579-27 from 464256 to 1140153 tid 1 2024-04-26T00:42:19.399 INFO:tasks.rados.rados.0.smithi045.stdout:3952: writing smithi04526579-27 from 1726176 to 2244325 tid 2 2024-04-26T00:42:19.414 INFO:tasks.rados.rados.0.smithi045.stdout:3952: writing smithi04526579-27 from 2882515 to 3390215 tid 3 2024-04-26T00:42:19.414 INFO:tasks.rados.rados.0.smithi045.stdout:3953: read oid 14 snap -1 2024-04-26T00:42:19.414 INFO:tasks.rados.rados.0.smithi045.stdout:3953: expect deleted 2024-04-26T00:42:19.414 INFO:tasks.rados.rados.0.smithi045.stdout:3954: write (excl) oid 33 current snap is 438 2024-04-26T00:42:19.414 INFO:tasks.rados.rados.0.smithi045.stdout:3954: seq_num 1409 ranges {481063=751146,1784658=778160,3309938=371125} 2024-04-26T00:42:19.428 INFO:tasks.rados.rados.0.smithi045.stdout:3954: writing smithi04526579-33 from 481063 to 1232209 tid 1 2024-04-26T00:42:19.441 INFO:tasks.rados.rados.0.smithi045.stdout:3954: writing smithi04526579-33 from 1784658 to 2562818 tid 2 2024-04-26T00:42:19.450 INFO:tasks.rados.rados.0.smithi045.stdout:3954: writing smithi04526579-33 from 3309938 to 3681063 tid 3 2024-04-26T00:42:19.451 INFO:tasks.rados.rados.0.smithi045.stdout:3955: snap_create 2024-04-26T00:42:19.451 INFO:tasks.rados.rados.0.smithi045.stdout:3950: expect (ObjNum 1395 snap 432 seq_num 1395) 2024-04-26T00:42:19.500 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.496+0000 7f84e20a0640 10 monclient: get_monmap_and_config 2024-04-26T00:42:19.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.496+0000 7f84e20a0640 10 monclient: build_initial_monmap 2024-04-26T00:42:19.501 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.496+0000 7f84e20a0640 10 monclient: monmap: 2024-04-26T00:42:19.501 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:42:19.501 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:19.501 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:42:19.502046+0000 2024-04-26T00:42:19.501 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:42:19.502046+0000 2024-04-26T00:42:19.501 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:42:19.501 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:19.501 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:42:19.501 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:42:19.501 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:42:19.501 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:42:19.501 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:42:19.502 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:19.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.496+0000 7f84e20a0640 1 Processor -- start 2024-04-26T00:42:19.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.496+0000 7f84e20a0640 1 -- start start 2024-04-26T00:42:19.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.496+0000 7f84e20a0640 10 monclient: init 2024-04-26T00:42:19.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.500+0000 7f84e20a0640 10 monclient: _reopen_session rank -1 2024-04-26T00:42:19.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.500+0000 7f84e20a0640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-04-26T00:42:19.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.500+0000 7f84e20a0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f84dc0a3d90 addr v1:172.21.15.37:6789/0 2024-04-26T00:42:19.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.500+0000 7f84e20a0640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f84dc0a4170 0x7f84dc0a45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:19.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.500+0000 7f84e20a0640 10 monclient(hunting): picked mon.noname-c con 0x7f84dc0a4170 addr v2:172.21.15.45:3300/0 2024-04-26T00:42:19.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.500+0000 7f84e20a0640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f84dc13c3e0 0x7f84dc13e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:19.504 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.500+0000 7f84e20a0640 10 monclient(hunting): picked mon.noname-a con 0x7f84dc13c3e0 addr v2:172.21.15.37:3300/0 2024-04-26T00:42:19.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.500+0000 7f84e20a0640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84dc0a5150 con 0x7f84dc0a3d90 2024-04-26T00:42:19.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.500+0000 7f84e20a0640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:19.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.500+0000 7f84e20a0640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f84dc09c650 con 0x7f84dc13c3e0 2024-04-26T00:42:19.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.500+0000 7f84e20a0640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:19.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.500+0000 7f84e20a0640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f84dc09b280 con 0x7f84dc0a4170 2024-04-26T00:42:19.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.500+0000 7f84e20a0640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:19.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.500+0000 7f84e20a0640 10 monclient(hunting): authenticate will time out at 2320.808393s 2024-04-26T00:42:19.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.500+0000 7f84db7fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f84dc0a3d90 0x7f84dc09bf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:48248/0 (socket says 172.21.15.37:48248) 2024-04-26T00:42:19.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.500+0000 7f84dbfff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f84dc13c3e0 0x7f84dc13e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:19.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.500+0000 7f84db7fe640 1 -- 172.21.15.37:0/4051833320 learned_addr learned my addr 172.21.15.37:0/4051833320 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:42:19.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.500+0000 7f84dbfff640 10 monclient(hunting): get_auth_request con 0x7f84dc13c3e0 auth_method 0 2024-04-26T00:42:19.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.500+0000 7f84dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:19.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.500+0000 7f84dbfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:19.505 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.500+0000 7f84dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:19.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.500+0000 7f84dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:19.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.500+0000 7f84dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:19.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.500+0000 7f84dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:19.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.500+0000 7f84daffd640 1 --2- 172.21.15.37:0/4051833320 >> v2:172.21.15.45:3300/0 conn(0x7f84dc0a4170 0x7f84dc0a45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:19.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.500+0000 7f84dbfff640 10 monclient(hunting): handle_auth_done global_id 264192 payload 274 2024-04-26T00:42:19.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.500+0000 7f84dbfff640 1 -- 172.21.15.37:0/4051833320 >> v2:172.21.15.45:3300/0 conn(0x7f84dc0a4170 msgr2=0x7f84dc0a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:42:19.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.500+0000 7f84dbfff640 1 --2- 172.21.15.37:0/4051833320 >> v2:172.21.15.45:3300/0 conn(0x7f84dc0a4170 0x7f84dc0a45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:19.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.500+0000 7f84dbfff640 1 -- 172.21.15.37:0/4051833320 >> v1:172.21.15.37:6789/0 conn(0x7f84dc0a3d90 legacy=0x7f84dc09bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:42:19.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.500+0000 7f84dbfff640 10 monclient: _finish_hunting 0 2024-04-26T00:42:19.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.500+0000 7f84dbfff640 1 monclient: found mon.noname-a 2024-04-26T00:42:19.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.500+0000 7f84dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:42:19.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.500+0000 7f84dbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:42:19.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.500+0000 7f84dbfff640 1 -- 172.21.15.37:0/4051833320 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84dc09b560 con 0x7f84dc13c3e0 2024-04-26T00:42:19.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.500+0000 7f84daffd640 10 monclient: get_auth_request con 0x7f84dc0a4170 auth_method 0 2024-04-26T00:42:19.507 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.500+0000 7f84daffd640 1 --2- 172.21.15.37:0/4051833320 >> v2:172.21.15.45:3300/0 conn(0x7f84dc0a4170 0x7f84dc0a45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:42:19.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84dbfff640 1 -- 172.21.15.37:0/4051833320 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84dc13c3e0 msgr2=0x7f84dc13e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:19.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84dbfff640 1 --2- 172.21.15.37:0/4051833320 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84dc13c3e0 0x7f84dc13e7d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f84dc09d140 tx=0x7f84d401c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e921642179ed7bc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:19.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84da7fc640 1 -- 172.21.15.37:0/4051833320 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d40b2070 con 0x7f84dc13c3e0 2024-04-26T00:42:19.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:19.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84da7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:42:19.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84da7fc640 10 monclient: dump: 2024-04-26T00:42:19.509 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:19.509 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:19.509 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:19.509 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:19.509 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:19.510 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:19.510 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:19.510 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:19.510 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:19.510 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:19.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84da7fc640 10 monclient: _finish_auth 0 2024-04-26T00:42:19.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84da7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:42:19.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:42:19.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84da7fc640 1 -- 172.21.15.37:0/4051833320 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84d40a5860 con 0x7f84dc13c3e0 2024-04-26T00:42:19.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84da7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:42:19.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84da7fc640 1 -- 172.21.15.37:0/4051833320 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d40a5c30 con 0x7f84dc13c3e0 2024-04-26T00:42:19.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84e20a0640 5 monclient: authenticate success, global_id 264192 2024-04-26T00:42:19.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84e20a0640 10 monclient: get_monmap_and_config success 2024-04-26T00:42:19.510 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84e20a0640 4 set_mon_vals no callback set 2024-04-26T00:42:19.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:19.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84da7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:42:19.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84da7fc640 10 monclient: dump: 2024-04-26T00:42:19.511 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:19.511 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:19.511 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:19.511 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:19.511 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:19.511 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:19.511 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:19.511 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:19.511 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:19.511 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:19.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84e20a0640 10 monclient: shutdown 2024-04-26T00:42:19.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84e20a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:42:19.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84e20a0640 1 -- 172.21.15.37:0/4051833320 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84dc13c3e0 msgr2=0x7f84dc13e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:19.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84e20a0640 1 --2- 172.21.15.37:0/4051833320 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84dc13c3e0 0x7f84dc13e7d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f84dc09d140 tx=0x7f84d401c7b0 comp rx=0 tx=0).stop 2024-04-26T00:42:19.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84e20a0640 1 -- 172.21.15.37:0/4051833320 shutdown_connections 2024-04-26T00:42:19.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84e20a0640 1 --2- 172.21.15.37:0/4051833320 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84dc13c3e0 0x7f84dc13e7d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:19.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84e20a0640 1 --2- 172.21.15.37:0/4051833320 >> v2:172.21.15.45:3300/0 conn(0x7f84dc0a4170 0x7f84dc0a45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:19.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84e20a0640 1 -- 172.21.15.37:0/4051833320 >> 172.21.15.37:0/4051833320 conn(0x7f84dc0aa0a0 msgr2=0x7f84dc0a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:42:19.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84e20a0640 1 -- 172.21.15.37:0/4051833320 shutdown_connections 2024-04-26T00:42:19.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84e20a0640 1 -- 172.21.15.37:0/4051833320 wait complete. 2024-04-26T00:42:19.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84e20a0640 10 monclient: build_initial_monmap 2024-04-26T00:42:19.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84e20a0640 10 monclient: monmap: 2024-04-26T00:42:19.512 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:42:19.512 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:42:19.512 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:42:19.513 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:42:19.513 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:42:19.513 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:42:19.513 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:42:19.513 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:42:19.513 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:42:19.513 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:19.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84e20a0640 1 Processor -- start 2024-04-26T00:42:19.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84e20a0640 1 -- start start 2024-04-26T00:42:19.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84e20a0640 10 monclient: init 2024-04-26T00:42:19.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84e20a0640 10 monclient: _reopen_session rank -1 2024-04-26T00:42:19.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84e20a0640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:42:19.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84e20a0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84dc0a3d90 0x7f84dc120500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:19.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84e20a0640 10 monclient(hunting): picked mon.noname-b con 0x7f84dc0a3d90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:42:19.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84e20a0640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f84dc0a4170 0x7f84dc13a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:19.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84e20a0640 10 monclient(hunting): picked mon.noname-c con 0x7f84dc0a4170 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:42:19.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84e20a0640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84dc13c3e0 0x7f84dc1370e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:19.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84e20a0640 10 monclient(hunting): picked mon.noname-a con 0x7f84dc13c3e0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:42:19.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84e20a0640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:19.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84e20a0640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f84dc1418c0 con 0x7f84dc13c3e0 2024-04-26T00:42:19.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84e20a0640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:19.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84e20a0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f84dc141740 con 0x7f84dc0a3d90 2024-04-26T00:42:19.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84e20a0640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:19.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84e20a0640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f84dc141a40 con 0x7f84dc0a4170 2024-04-26T00:42:19.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84e20a0640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:19.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84e20a0640 10 monclient(hunting): authenticate will time out at 2320.815429s 2024-04-26T00:42:19.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84daffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f84dc0a4170 0x7f84dc13a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:19.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84dbfff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84dc13c3e0 0x7f84dc1370e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:19.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84daffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f84dc0a4170 0x7f84dc13a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:35342/0 (socket says 172.21.15.37:35342) 2024-04-26T00:42:19.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84daffd640 1 -- 172.21.15.37:0/1943668845 learned_addr learned my addr 172.21.15.37:0/1943668845 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:42:19.520 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84daffd640 10 monclient(hunting): get_auth_request con 0x7f84dc0a4170 auth_method 0 2024-04-26T00:42:19.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84daffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:19.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84daffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:19.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84daffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:19.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84dbfff640 10 monclient(hunting): get_auth_request con 0x7f84dc13c3e0 auth_method 0 2024-04-26T00:42:19.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:19.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84dbfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:19.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:19.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84daffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:19.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84daffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:19.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84daffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:19.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:19.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:19.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:19.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84db7fe640 1 --2- 172.21.15.37:0/1943668845 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84dc0a3d90 0x7f84dc120500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:19.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84db7fe640 10 monclient(hunting): get_auth_request con 0x7f84dc0a3d90 auth_method 0 2024-04-26T00:42:19.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:19.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84db7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:19.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:19.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84dbfff640 10 monclient(hunting): handle_auth_done global_id 264195 payload 824 2024-04-26T00:42:19.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84dbfff640 1 -- 172.21.15.37:0/1943668845 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f84dc0a4170 msgr2=0x7f84dc13a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:19.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84dbfff640 1 --2- 172.21.15.37:0/1943668845 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f84dc0a4170 0x7f84dc13a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:19.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84dbfff640 1 -- 172.21.15.37:0/1943668845 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84dc0a3d90 msgr2=0x7f84dc120500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:19.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84dbfff640 1 --2- 172.21.15.37:0/1943668845 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84dc0a3d90 0x7f84dc120500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:19.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84dbfff640 10 monclient: _finish_hunting 0 2024-04-26T00:42:19.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84dbfff640 1 monclient: found mon.noname-a 2024-04-26T00:42:19.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:42:19.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84dbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:42:19.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84dbfff640 1 -- 172.21.15.37:0/1943668845 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84dc09ada0 con 0x7f84dc13c3e0 2024-04-26T00:42:19.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.504+0000 7f84daffd640 1 --2- 172.21.15.37:0/1943668845 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f84dc0a4170 0x7f84dc13a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:42:19.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.508+0000 7f84dbfff640 1 -- 172.21.15.37:0/1943668845 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84dc13c3e0 msgr2=0x7f84dc1370e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:19.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.508+0000 7f84dbfff640 1 --2- 172.21.15.37:0/1943668845 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84dc13c3e0 0x7f84dc1370e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f84dc0a6e50 tx=0x7f84d4019550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:19.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.508+0000 7f84c3fff640 1 -- 172.21.15.37:0/1943668845 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d40bd020 con 0x7f84dc13c3e0 2024-04-26T00:42:19.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.508+0000 7f84c3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:19.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.508+0000 7f84c3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:42:19.523 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.508+0000 7f84c3fff640 10 monclient: dump: 2024-04-26T00:42:19.523 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:19.523 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:19.523 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:19.524 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:19.524 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:19.524 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:19.524 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:19.524 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:19.524 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:19.524 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:19.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.508+0000 7f84c3fff640 10 monclient: _finish_auth 0 2024-04-26T00:42:19.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.508+0000 7f84c3fff640 10 monclient: _check_auth_tickets 2024-04-26T00:42:19.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.508+0000 7f84c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:42:19.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.508+0000 7f84e20a0640 5 monclient: authenticate success, global_id 264195 2024-04-26T00:42:19.524 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.508+0000 7f84c3fff640 1 -- 172.21.15.37:0/1943668845 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84d40b3040 con 0x7f84dc13c3e0 2024-04-26T00:42:19.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.508+0000 7f84c3fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:42:19.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.508+0000 7f84c3fff640 1 -- 172.21.15.37:0/1943668845 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d40b2070 con 0x7f84dc13c3e0 2024-04-26T00:42:19.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.508+0000 7f84c3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:19.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.508+0000 7f84c3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:42:19.525 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.508+0000 7f84c3fff640 10 monclient: dump: 2024-04-26T00:42:19.525 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:19.525 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:19.525 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:19.525 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:19.525 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:19.525 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:19.526 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:19.526 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:19.526 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:19.526 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:19.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.508+0000 7f84d97fa640 4 set_mon_vals no callback set 2024-04-26T00:42:19.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.508+0000 7f84e20a0640 10 monclient: _renew_subs 2024-04-26T00:42:19.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.508+0000 7f84e20a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:42:19.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.508+0000 7f84e20a0640 1 -- 172.21.15.37:0/1943668845 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84dc13cb30 con 0x7f84dc13c3e0 2024-04-26T00:42:19.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.508+0000 7f84e20a0640 10 monclient: _renew_subs 2024-04-26T00:42:19.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.508+0000 7f84e20a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:42:19.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.508+0000 7f84e20a0640 1 -- 172.21.15.37:0/1943668845 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84dc13be50 con 0x7f84dc13c3e0 2024-04-26T00:42:19.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.508+0000 7f84e20a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:19.526 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.508+0000 7f84e20a0640 1 --2- 172.21.15.37:0/1943668845 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84dc137620 0x7f84dc1ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:19.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.508+0000 7f84e20a0640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:19.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.508+0000 7f84e20a0640 1 -- 172.21.15.37:0/1943668845 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f84dc1216f0 con 0x7f84dc137620 2024-04-26T00:42:19.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.508+0000 7f84db7fe640 1 --2- 172.21.15.37:0/1943668845 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84dc137620 0x7f84dc1ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:19.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.508+0000 7f84db7fe640 10 monclient: get_auth_request con 0x7f84dc137620 auth_method 0 2024-04-26T00:42:19.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.508+0000 7f84db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:19.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.508+0000 7f84db7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:19.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.508+0000 7f84db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:19.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.508+0000 7f84c3fff640 1 -- 172.21.15.37:0/1943668845 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f84d40c0020 con 0x7f84dc13c3e0 2024-04-26T00:42:19.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.508+0000 7f84db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:19.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.508+0000 7f84db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:19.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.508+0000 7f84db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:19.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.508+0000 7f84c3fff640 1 --2- 172.21.15.37:0/1943668845 >> v2:172.21.15.45:6812/2379918623 conn(0x7f84c8035ed0 0x7f84c8038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:19.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.508+0000 7f84db7fe640 10 monclient(hunting): handle_auth_done global_id 264198 payload 824 2024-04-26T00:42:19.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.508+0000 7f84db7fe640 1 -- 172.21.15.37:0/1943668845 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f84dc04f510 con 0x7f84dc137620 2024-04-26T00:42:19.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.508+0000 7f84c3fff640 1 -- 172.21.15.37:0/1943668845 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(891..891 src has 822..891) ==== 3979+0+0 (secure 0 0 0) 0x7f84d40e3580 con 0x7f84dc13c3e0 2024-04-26T00:42:19.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.512+0000 7f84daffd640 1 --2- 172.21.15.37:0/1943668845 >> v2:172.21.15.45:6812/2379918623 conn(0x7f84c8035ed0 0x7f84c8038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:19.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.512+0000 7f84daffd640 10 monclient: get_auth_request con 0x7f84c8035ed0 auth_method 0 2024-04-26T00:42:19.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.512+0000 7f84db7fe640 1 -- 172.21.15.37:0/1943668845 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84dc137620 msgr2=0x7f84dc1ceef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:19.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.512+0000 7f84db7fe640 1 --2- 172.21.15.37:0/1943668845 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84dc137620 0x7f84dc1ceef0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f84cc00d500 tx=0x7f84cc00d9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:19.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.512+0000 7f84c3fff640 1 -- 172.21.15.37:0/1943668845 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84cc018030 con 0x7f84dc137620 2024-04-26T00:42:19.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.512+0000 7f84c3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:19.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.512+0000 7f84c3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:42:19.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.512+0000 7f84c3fff640 10 monclient: dump: 2024-04-26T00:42:19.528 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:19.528 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:19.529 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:19.529 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:19.529 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:19.529 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:19.529 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:19.529 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:19.529 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:19.529 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:19.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.512+0000 7f84c3fff640 1 -- 172.21.15.37:0/1943668845 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f84cc014910 con 0x7f84dc137620 2024-04-26T00:42:19.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.512+0000 7f84c3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:19.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.512+0000 7f84c3fff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:42:19.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.512+0000 7f84c3fff640 1 -- 172.21.15.37:0/1943668845 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84dc137620 msgr2=0x7f84dc1ceef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:19.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.512+0000 7f84c3fff640 1 --2- 172.21.15.37:0/1943668845 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84dc137620 0x7f84dc1ceef0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f84cc00d500 tx=0x7f84cc00d9c0 comp rx=0 tx=0).stop 2024-04-26T00:42:19.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.512+0000 7f84c3fff640 1 -- 172.21.15.37:0/1943668845 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84dc137620 msgr2=0x7f84dc1ceef0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:42:19.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.512+0000 7f84c3fff640 1 --2- 172.21.15.37:0/1943668845 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84dc137620 0x7f84dc1ceef0 secure :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f84cc00d500 tx=0x7f84cc00d9c0 comp rx=0 tx=0).stop 2024-04-26T00:42:19.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.512+0000 7f84daffd640 1 --2- 172.21.15.37:0/1943668845 >> v2:172.21.15.45:6812/2379918623 conn(0x7f84c8035ed0 0x7f84c8038390 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f84dc121100 tx=0x7f84d0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:19.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.520+0000 7f84e20a0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:42:19.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.520+0000 7f84e20a0640 1 --2- 172.21.15.37:0/1943668845 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84dc1cf430 0x7f84dc1cf840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:19.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.520+0000 7f84e20a0640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:19.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.520+0000 7f84e20a0640 1 -- 172.21.15.37:0/1943668845 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f84dc1cfd80 con 0x7f84dc1cf430 2024-04-26T00:42:19.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.520+0000 7f84db7fe640 1 --2- 172.21.15.37:0/1943668845 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84dc1cf430 0x7f84dc1cf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:19.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.520+0000 7f84db7fe640 10 monclient: get_auth_request con 0x7f84dc1cf430 auth_method 0 2024-04-26T00:42:19.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.520+0000 7f84db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:19.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.520+0000 7f84db7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:19.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.520+0000 7f84db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:19.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.520+0000 7f84db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:19.532 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.520+0000 7f84db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:19.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.520+0000 7f84db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:19.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.520+0000 7f84db7fe640 10 monclient(hunting): handle_auth_done global_id 264201 payload 824 2024-04-26T00:42:19.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.520+0000 7f84db7fe640 1 -- 172.21.15.37:0/1943668845 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f84dc13db80 con 0x7f84dc1cf430 2024-04-26T00:42:19.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.520+0000 7f84db7fe640 1 -- 172.21.15.37:0/1943668845 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84dc1cf430 msgr2=0x7f84dc1cf840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:19.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.520+0000 7f84db7fe640 1 --2- 172.21.15.37:0/1943668845 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84dc1cf430 0x7f84dc1cf840 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f84cc00d4d0 tx=0x7f84cc0058e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:19.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.520+0000 7f84c3fff640 1 -- 172.21.15.37:0/1943668845 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84cc018030 con 0x7f84dc1cf430 2024-04-26T00:42:19.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.520+0000 7f84c3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:19.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.520+0000 7f84c3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:42:19.533 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.520+0000 7f84c3fff640 10 monclient: dump: 2024-04-26T00:42:19.533 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:19.533 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:19.533 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:19.533 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:19.533 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:19.534 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:19.534 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:19.534 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:19.534 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:19.534 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:19.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.520+0000 7f84c3fff640 1 -- 172.21.15.37:0/1943668845 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f84cc0047f0 con 0x7f84dc1cf430 2024-04-26T00:42:19.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.520+0000 7f84c3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:42:19.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.520+0000 7f84c3fff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:42:19.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.520+0000 7f84c3fff640 1 -- 172.21.15.37:0/1943668845 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84dc1cf430 msgr2=0x7f84dc1cf840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:19.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.520+0000 7f84c3fff640 1 --2- 172.21.15.37:0/1943668845 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84dc1cf430 0x7f84dc1cf840 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f84cc00d4d0 tx=0x7f84cc0058e0 comp rx=0 tx=0).stop 2024-04-26T00:42:19.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.520+0000 7f84c3fff640 1 -- 172.21.15.37:0/1943668845 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84dc1cf430 msgr2=0x7f84dc1cf840 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:42:19.534 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.520+0000 7f84c3fff640 1 --2- 172.21.15.37:0/1943668845 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84dc1cf430 0x7f84dc1cf840 secure :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f84cc00d4d0 tx=0x7f84cc0058e0 comp rx=0 tx=0).stop 2024-04-26T00:42:19.535 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:42:19.535 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:42:19.535 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:42:19.535 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:42:19.535 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 160, 2024-04-26T00:42:19.535 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 61811, 2024-04-26T00:42:19.535 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:42:19.535 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:42:19.535 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:42:19.535 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:42:19.535 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:42:19.536 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:42:19.536 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:42:19.536 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:42:19.536 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:42:19.536 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:42:19.536 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:42:19.536 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:42:19.536 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:42:19.536 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:42:19.536 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:42:19.536 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:42:19.536 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:42:19.536 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:42:19.536 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:42:19.537 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:42:19.537 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:42:19.537 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:42:19.537 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:42:19.537 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:42:19.537 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:42:19.537 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:42:19.537 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:42:19.537 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:42:19.537 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:42:19.537 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:42:19.537 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:42:19.537 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:42:19.537 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:42:19.537 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:42:19.537 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:19.538 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:19.538 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:42:19.538 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:42:19.538 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:42:19.538 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:42:19.538 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:42:19.538 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:42:19.538 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:42:19.538 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:42:19.538 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:42:19.538 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:42:19.538 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:42:19.538 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:42:19.538 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:42:19.539 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:42:19.539 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:42:19.539 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:42:19.539 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:42:19.539 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:42:19.539 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:42:19.539 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:42:19.539 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:42:19.539 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:42:19.539 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:42:19.539 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:42:19.539 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:42:19.539 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:42:19.539 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:42:19.540 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:42:19.540 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:42:19.540 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:42:19.540 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:19.540 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:42:19.540 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:19.540 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:42:19.540 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:42:19.540 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:42:19.540 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:42:19.540 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:19.540 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:42:19.540 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:42:19.540 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:19.541 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:19.541 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:19.541 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:42:19.541 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:42:19.541 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:19.541 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:19.541 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:19.541 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:19.541 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:42:19.541 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:42:19.541 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:42:19.541 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:42:19.541 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:42:19.542 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:19.542 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:19.542 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:42:19.542 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:42:19.542 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:42:19.542 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:42:19.542 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:19.542 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:42:19.542 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:42:19.542 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:19.542 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:19.542 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:19.542 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:42:19.543 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:42:19.543 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:19.543 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:19.543 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:19.543 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:19.543 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:42:19.543 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:42:19.543 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:42:19.543 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:42:19.543 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:42:19.543 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:19.543 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:19.543 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:42:19.543 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:42:19.544 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:42:19.544 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:42:19.544 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:19.544 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:42:19.544 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:42:19.544 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:19.544 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:19.544 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:19.544 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:42:19.544 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:42:19.544 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:19.544 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:19.544 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:19.544 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:19.544 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:42:19.544 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:42:19.545 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:42:19.545 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:42:19.545 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:42:19.545 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:19.545 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:19.545 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:19.545 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:42:19.545 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:42:19.545 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:19.545 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:42:19.545 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:42:19.545 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:42:19.545 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:19.545 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:42:19.546 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:42:19.546 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:19.546 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:42:19.546 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:42:19.546 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:42:19.546 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:19.546 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:42:19.546 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:42:19.546 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:19.547 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:42:19.547 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:42:19.547 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:42:19.547 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:19.547 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:42:19.547 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:42:19.547 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:19.547 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:42:19.547 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:42:19.547 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:42:19.547 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:19.548 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:19.548 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:19.548 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:42:19.548 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:42:19.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.524+0000 7f84e20a0640 1 -- 172.21.15.37:0/1943668845 >> v2:172.21.15.45:6812/2379918623 conn(0x7f84c8035ed0 msgr2=0x7f84c8038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:19.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.524+0000 7f84e20a0640 1 --2- 172.21.15.37:0/1943668845 >> v2:172.21.15.45:6812/2379918623 conn(0x7f84c8035ed0 0x7f84c8038390 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f84dc121100 tx=0x7f84d0008040 comp rx=0 tx=0).stop 2024-04-26T00:42:19.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.524+0000 7f84e20a0640 10 monclient: shutdown 2024-04-26T00:42:19.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.524+0000 7f84e20a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:42:19.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.524+0000 7f84e20a0640 1 -- 172.21.15.37:0/1943668845 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84dc13c3e0 msgr2=0x7f84dc1370e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:19.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.524+0000 7f84dbfff640 1 -- 172.21.15.37:0/1943668845 reap_dead start 2024-04-26T00:42:19.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.524+0000 7f84e20a0640 1 --2- 172.21.15.37:0/1943668845 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84dc13c3e0 0x7f84dc1370e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f84dc0a6e50 tx=0x7f84d4019550 comp rx=0 tx=0).stop 2024-04-26T00:42:19.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.524+0000 7f84e20a0640 1 -- 172.21.15.37:0/1943668845 shutdown_connections 2024-04-26T00:42:19.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.524+0000 7f84e20a0640 1 --2- 172.21.15.37:0/1943668845 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84dc13c3e0 0x7f84dc1370e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:19.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.524+0000 7f84e20a0640 1 -- 172.21.15.37:0/1943668845 >> 172.21.15.37:0/1943668845 conn(0x7f84dc0aa0a0 msgr2=0x7f84dc0a8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:42:19.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.524+0000 7f84e20a0640 1 -- 172.21.15.37:0/1943668845 shutdown_connections 2024-04-26T00:42:19.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.524+0000 7f84e20a0640 1 -- 172.21.15.37:0/1943668845 wait complete. 2024-04-26T00:42:19.550 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:42:19.551 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:42:19.551 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:42:19.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d4533640 10 monclient: get_monmap_and_config 2024-04-26T00:42:19.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d4533640 10 monclient: build_initial_monmap 2024-04-26T00:42:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d4533640 10 monclient: monmap: 2024-04-26T00:42:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:42:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:42:19.665455+0000 2024-04-26T00:42:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:42:19.665455+0000 2024-04-26T00:42:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:42:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:42:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:42:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:42:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:42:19.664 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:42:19.664 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:19.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d4533640 1 Processor -- start 2024-04-26T00:42:19.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d4533640 1 -- start start 2024-04-26T00:42:19.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d4533640 10 monclient: init 2024-04-26T00:42:19.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d4533640 10 monclient: _reopen_session rank -1 2024-04-26T00:42:19.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d4533640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-04-26T00:42:19.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d4533640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd2cc0a0300 0x7fd2cc09fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:19.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d4533640 10 monclient(hunting): picked mon.noname-c con 0x7fd2cc0a0300 addr v2:172.21.15.45:3300/0 2024-04-26T00:42:19.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d4533640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd2cc0a06e0 0x7fd2cc134ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:19.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d4533640 10 monclient(hunting): picked mon.noname-a con 0x7fd2cc0a06e0 addr v2:172.21.15.37:3300/0 2024-04-26T00:42:19.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d4533640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2cc1350e0 0x7fd2cc135550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:19.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d4533640 10 monclient(hunting): picked mon.noname-b con 0x7fd2cc1350e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:42:19.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d4533640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:19.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d4533640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fd2cc0a13f0 con 0x7fd2cc0a06e0 2024-04-26T00:42:19.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d4533640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:19.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d4533640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fd2cc09c190 con 0x7fd2cc0a0300 2024-04-26T00:42:19.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d4533640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:19.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d4533640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd2cc0a1110 con 0x7fd2cc1350e0 2024-04-26T00:42:19.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d4533640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:19.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d4533640 10 monclient(hunting): authenticate will time out at 2320.971098s 2024-04-26T00:42:19.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d1ace640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd2cc0a06e0 0x7fd2cc134ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:19.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d2ad0640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2cc1350e0 0x7fd2cc135550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:19.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d1ace640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd2cc0a06e0 0x7fd2cc134ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:57366/0 (socket says 172.21.15.37:57366) 2024-04-26T00:42:19.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d1ace640 1 -- 172.21.15.37:0/2756292002 learned_addr learned my addr 172.21.15.37:0/2756292002 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:42:19.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d1ace640 10 monclient(hunting): get_auth_request con 0x7fd2cc0a06e0 auth_method 0 2024-04-26T00:42:19.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d1ace640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:19.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d1ace640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:19.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d1ace640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:19.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d2ad0640 10 monclient(hunting): get_auth_request con 0x7fd2cc1350e0 auth_method 0 2024-04-26T00:42:19.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d2ad0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:19.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d2ad0640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:19.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d2ad0640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:19.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d22cf640 1 --2- 172.21.15.37:0/2756292002 >> v2:172.21.15.45:3300/0 conn(0x7fd2cc0a0300 0x7fd2cc09fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:19.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d1ace640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:19.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d1ace640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:19.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d1ace640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:19.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d22cf640 10 monclient(hunting): get_auth_request con 0x7fd2cc0a0300 auth_method 0 2024-04-26T00:42:19.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d22cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:19.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d22cf640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:19.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d22cf640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:19.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d2ad0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:19.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d2ad0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:19.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d2ad0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:19.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d22cf640 1 -- 172.21.15.37:0/2756292002 >> v2:172.21.15.45:3300/0 conn(0x7fd2cc0a0300 msgr2=0x7fd2cc09fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:42:19.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d1ace640 10 monclient(hunting): handle_auth_done global_id 264204 payload 274 2024-04-26T00:42:19.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d22cf640 1 -- 172.21.15.37:0/2756292002 >> v2:172.21.15.45:3300/0 conn(0x7fd2cc0a0300 msgr2=0x7fd2cc09fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:42:19.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d22cf640 1 --2- 172.21.15.37:0/2756292002 >> v2:172.21.15.45:3300/0 conn(0x7fd2cc0a0300 0x7fd2cc09fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:42:19.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d1ace640 1 -- 172.21.15.37:0/2756292002 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2cc1350e0 msgr2=0x7fd2cc135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:42:19.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d1ace640 1 --2- 172.21.15.37:0/2756292002 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2cc1350e0 0x7fd2cc135550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:19.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d22cf640 1 --2- 172.21.15.37:0/2756292002 >> v2:172.21.15.45:3300/0 conn(0x7fd2cc0a0300 0x7fd2cc09fc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:42:19.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d1ace640 1 -- 172.21.15.37:0/2756292002 >> v2:172.21.15.45:3300/0 conn(0x7fd2cc0a0300 msgr2=0x7fd2cc09fc50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:42:19.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d1ace640 1 --2- 172.21.15.37:0/2756292002 >> v2:172.21.15.45:3300/0 conn(0x7fd2cc0a0300 0x7fd2cc09fc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:19.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d1ace640 10 monclient: _finish_hunting 0 2024-04-26T00:42:19.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d1ace640 1 monclient: found mon.noname-a 2024-04-26T00:42:19.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d1ace640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:42:19.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d1ace640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:42:19.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.660+0000 7fd2d1ace640 1 -- 172.21.15.37:0/2756292002 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2cc0a16d0 con 0x7fd2cc0a06e0 2024-04-26T00:42:19.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.664+0000 7fd2d1ace640 1 -- 172.21.15.37:0/2756292002 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd2cc0a06e0 msgr2=0x7fd2cc134ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:19.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.664+0000 7fd2d1ace640 1 --2- 172.21.15.37:0/2756292002 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd2cc0a06e0 0x7fd2cc134ba0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc0764d0 tx=0x7fd2c801c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e83d9ac8a81366f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:19.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d12cd640 1 -- 172.21.15.37:0/2756292002 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c80a5640 con 0x7fd2cc0a06e0 2024-04-26T00:42:19.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d12cd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:19.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d12cd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:42:19.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d12cd640 10 monclient: dump: 2024-04-26T00:42:19.675 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:19.675 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:19.675 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:19.675 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:19.675 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:19.675 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:19.676 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:19.676 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:19.676 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:19.676 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:19.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d12cd640 10 monclient: _finish_auth 0 2024-04-26T00:42:19.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d12cd640 10 monclient: _check_auth_tickets 2024-04-26T00:42:19.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d12cd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:42:19.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d12cd640 1 -- 172.21.15.37:0/2756292002 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2c80a57e0 con 0x7fd2cc0a06e0 2024-04-26T00:42:19.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d12cd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:42:19.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d12cd640 1 -- 172.21.15.37:0/2756292002 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c80a5bb0 con 0x7fd2cc0a06e0 2024-04-26T00:42:19.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d4533640 5 monclient: authenticate success, global_id 264204 2024-04-26T00:42:19.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d4533640 10 monclient: get_monmap_and_config success 2024-04-26T00:42:19.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d4533640 4 set_mon_vals no callback set 2024-04-26T00:42:19.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d12cd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:19.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d12cd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:42:19.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d12cd640 10 monclient: dump: 2024-04-26T00:42:19.677 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:19.677 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:19.677 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:19.677 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:19.677 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:19.677 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:19.677 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:19.677 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:19.678 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:19.678 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:19.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d4533640 10 monclient: shutdown 2024-04-26T00:42:19.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d4533640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:42:19.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d4533640 1 -- 172.21.15.37:0/2756292002 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd2cc0a06e0 msgr2=0x7fd2cc134ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:19.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d4533640 1 --2- 172.21.15.37:0/2756292002 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd2cc0a06e0 0x7fd2cc134ba0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc0764d0 tx=0x7fd2c801c8b0 comp rx=0 tx=0).stop 2024-04-26T00:42:19.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d4533640 1 -- 172.21.15.37:0/2756292002 shutdown_connections 2024-04-26T00:42:19.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d4533640 1 --2- 172.21.15.37:0/2756292002 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2cc1350e0 0x7fd2cc135550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:19.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d4533640 1 --2- 172.21.15.37:0/2756292002 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd2cc0a06e0 0x7fd2cc134ba0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:19.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d4533640 1 --2- 172.21.15.37:0/2756292002 >> v2:172.21.15.45:3300/0 conn(0x7fd2cc0a0300 0x7fd2cc09fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:19.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d4533640 1 -- 172.21.15.37:0/2756292002 >> 172.21.15.37:0/2756292002 conn(0x7fd2cc0a1d50 msgr2=0x7fd2cc0a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:42:19.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d4533640 1 -- 172.21.15.37:0/2756292002 shutdown_connections 2024-04-26T00:42:19.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d4533640 1 -- 172.21.15.37:0/2756292002 wait complete. 2024-04-26T00:42:19.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d4533640 10 monclient: build_initial_monmap 2024-04-26T00:42:19.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d4533640 10 monclient: monmap: 2024-04-26T00:42:19.679 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:42:19.679 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:42:19.679 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:42:19.679 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:42:19.679 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:42:19.679 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:42:19.679 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:42:19.680 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:42:19.680 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:42:19.680 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:19.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d4533640 1 Processor -- start 2024-04-26T00:42:19.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d4533640 1 -- start start 2024-04-26T00:42:19.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d4533640 10 monclient: init 2024-04-26T00:42:19.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d4533640 10 monclient: _reopen_session rank -1 2024-04-26T00:42:19.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d4533640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:42:19.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d4533640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2cc0a0300 0x7fd2cc1ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:19.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d4533640 10 monclient(hunting): picked mon.noname-b con 0x7fd2cc0a0300 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:42:19.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d4533640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2cc0a06e0 0x7fd2cc1cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:19.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d4533640 10 monclient(hunting): picked mon.noname-c con 0x7fd2cc0a06e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:42:19.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d4533640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd2cc1350e0 0x7fd2cc1cd830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:19.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d4533640 10 monclient(hunting): picked mon.noname-a con 0x7fd2cc1350e0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:42:19.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d4533640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:19.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d4533640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd2cc13e6f0 con 0x7fd2cc1350e0 2024-04-26T00:42:19.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d4533640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:19.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d4533640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd2cc13e870 con 0x7fd2cc0a0300 2024-04-26T00:42:19.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d4533640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:19.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d4533640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd2cc13e9f0 con 0x7fd2cc0a06e0 2024-04-26T00:42:19.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d4533640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:19.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d4533640 10 monclient(hunting): authenticate will time out at 2320.978809s 2024-04-26T00:42:19.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d1ace640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2cc0a06e0 0x7fd2cc1cd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:19.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d2ad0640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd2cc1350e0 0x7fd2cc1cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:19.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d1ace640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2cc0a06e0 0x7fd2cc1cd2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:35358/0 (socket says 172.21.15.37:35358) 2024-04-26T00:42:19.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d1ace640 1 -- 172.21.15.37:0/1454762427 learned_addr learned my addr 172.21.15.37:0/1454762427 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:42:19.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d1ace640 10 monclient(hunting): get_auth_request con 0x7fd2cc0a06e0 auth_method 0 2024-04-26T00:42:19.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d1ace640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:19.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d1ace640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:19.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d1ace640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:19.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d2ad0640 10 monclient(hunting): get_auth_request con 0x7fd2cc1350e0 auth_method 0 2024-04-26T00:42:19.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d2ad0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:19.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d2ad0640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:19.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d2ad0640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:19.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d1ace640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:19.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d1ace640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:19.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d1ace640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:19.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d2ad0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:19.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d2ad0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:19.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d22cf640 1 --2- 172.21.15.37:0/1454762427 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2cc0a0300 0x7fd2cc1ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:19.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d2ad0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:19.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d22cf640 10 monclient(hunting): get_auth_request con 0x7fd2cc0a0300 auth_method 0 2024-04-26T00:42:19.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d22cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:19.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d22cf640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:19.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d22cf640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:19.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d1ace640 10 monclient(hunting): handle_auth_done global_id 254315 payload 824 2024-04-26T00:42:19.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d1ace640 1 -- 172.21.15.37:0/1454762427 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2cc0a0300 msgr2=0x7fd2cc1ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:19.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d1ace640 1 --2- 172.21.15.37:0/1454762427 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd2cc0a0300 0x7fd2cc1ccdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:19.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d1ace640 1 -- 172.21.15.37:0/1454762427 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd2cc1350e0 msgr2=0x7fd2cc1cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:19.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d1ace640 1 --2- 172.21.15.37:0/1454762427 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd2cc1350e0 0x7fd2cc1cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:19.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d1ace640 10 monclient: _finish_hunting 0 2024-04-26T00:42:19.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d1ace640 1 monclient: found mon.noname-c 2024-04-26T00:42:19.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d1ace640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:42:19.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d1ace640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:42:19.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d1ace640 1 -- 172.21.15.37:0/1454762427 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2cc13a740 con 0x7fd2cc0a06e0 2024-04-26T00:42:19.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.668+0000 7fd2d2ad0640 1 --2- 172.21.15.37:0/1454762427 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd2cc1350e0 0x7fd2cc1cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:42:19.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2d1ace640 1 -- 172.21.15.37:0/1454762427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2cc0a06e0 msgr2=0x7fd2cc1cd2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:19.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2d1ace640 1 --2- 172.21.15.37:0/1454762427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2cc0a06e0 0x7fd2cc1cd2f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd2c80af040 tx=0x7fd2c8019650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:19.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2beffd640 1 -- 172.21.15.37:0/1454762427 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c80a5640 con 0x7fd2cc0a06e0 2024-04-26T00:42:19.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:19.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2beffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:42:19.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2beffd640 10 monclient: dump: 2024-04-26T00:42:19.687 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:19.687 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:19.687 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:19.687 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:19.688 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:19.688 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:19.688 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:19.688 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:19.688 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:19.688 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:19.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2beffd640 10 monclient: _finish_auth 0 2024-04-26T00:42:19.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2beffd640 10 monclient: _check_auth_tickets 2024-04-26T00:42:19.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:42:19.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2d4533640 5 monclient: authenticate success, global_id 254315 2024-04-26T00:42:19.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2beffd640 1 -- 172.21.15.37:0/1454762427 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2c80bc070 con 0x7fd2cc0a06e0 2024-04-26T00:42:19.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2beffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:42:19.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2beffd640 1 -- 172.21.15.37:0/1454762427 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c80b6af0 con 0x7fd2cc0a06e0 2024-04-26T00:42:19.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2d4533640 10 monclient: _renew_subs 2024-04-26T00:42:19.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2d4533640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:19.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2d4533640 1 -- 172.21.15.37:0/1454762427 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2cc13aa20 con 0x7fd2cc0a06e0 2024-04-26T00:42:19.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:19.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2d0acc640 4 set_mon_vals no callback set 2024-04-26T00:42:19.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2beffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:42:19.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2beffd640 10 monclient: dump: 2024-04-26T00:42:19.689 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:19.689 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:19.689 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:19.689 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:19.689 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:19.689 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:19.689 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:19.689 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:19.689 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:19.690 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:19.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2d4533640 10 monclient: _renew_subs 2024-04-26T00:42:19.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2d4533640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:19.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2d4533640 1 -- 172.21.15.37:0/1454762427 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2cc09f550 con 0x7fd2cc0a06e0 2024-04-26T00:42:19.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2d4533640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:19.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2d4533640 1 --2- 172.21.15.37:0/1454762427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2cc13b930 0x7fd2cc12ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:19.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2d4533640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:19.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2beffd640 1 -- 172.21.15.37:0/1454762427 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd2c80b6c90 con 0x7fd2cc0a06e0 2024-04-26T00:42:19.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2d4533640 1 -- 172.21.15.37:0/1454762427 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd2cc0a0cd0 con 0x7fd2cc13b930 2024-04-26T00:42:19.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2d22cf640 1 --2- 172.21.15.37:0/1454762427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2cc13b930 0x7fd2cc12ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:19.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2d22cf640 10 monclient: get_auth_request con 0x7fd2cc13b930 auth_method 0 2024-04-26T00:42:19.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2d22cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:19.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2d22cf640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:19.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2d22cf640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:19.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2d22cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:19.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2d22cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:19.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2d22cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:19.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2beffd640 1 --2- 172.21.15.37:0/1454762427 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd2b0035e80 0x7fd2b0038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:19.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2beffd640 1 -- 172.21.15.37:0/1454762427 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(891..891 src has 822..891) ==== 3979+0+0 (secure 0 0 0) 0x7fd2c80e3690 con 0x7fd2cc0a06e0 2024-04-26T00:42:19.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2d22cf640 10 monclient(hunting): handle_auth_done global_id 254318 payload 824 2024-04-26T00:42:19.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2d22cf640 1 -- 172.21.15.37:0/1454762427 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd2cc13a1f0 con 0x7fd2cc13b930 2024-04-26T00:42:19.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2d2ad0640 1 --2- 172.21.15.37:0/1454762427 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd2b0035e80 0x7fd2b0038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:19.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.672+0000 7fd2d2ad0640 10 monclient: get_auth_request con 0x7fd2b0035e80 auth_method 0 2024-04-26T00:42:19.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.676+0000 7fd2d22cf640 1 -- 172.21.15.37:0/1454762427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2cc13b930 msgr2=0x7fd2cc12ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:19.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.676+0000 7fd2d22cf640 1 --2- 172.21.15.37:0/1454762427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2cc13b930 0x7fd2cc12ead0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd2c0010790 tx=0x7fd2c0010c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:19.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.676+0000 7fd2beffd640 1 -- 172.21.15.37:0/1454762427 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c0019020 con 0x7fd2cc13b930 2024-04-26T00:42:19.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.676+0000 7fd2beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:19.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.676+0000 7fd2beffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:42:19.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.676+0000 7fd2beffd640 10 monclient: dump: 2024-04-26T00:42:19.692 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:19.692 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:19.692 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:19.692 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:19.692 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:19.692 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:19.692 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:19.692 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:19.692 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:19.692 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:19.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.676+0000 7fd2beffd640 1 -- 172.21.15.37:0/1454762427 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd2c000ede0 con 0x7fd2cc13b930 2024-04-26T00:42:19.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.676+0000 7fd2beffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:19.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.676+0000 7fd2beffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:42:19.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.676+0000 7fd2beffd640 1 -- 172.21.15.37:0/1454762427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2cc13b930 msgr2=0x7fd2cc12ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:19.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.676+0000 7fd2beffd640 1 --2- 172.21.15.37:0/1454762427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2cc13b930 0x7fd2cc12ead0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd2c0010790 tx=0x7fd2c0010c50 comp rx=0 tx=0).stop 2024-04-26T00:42:19.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.676+0000 7fd2beffd640 1 -- 172.21.15.37:0/1454762427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2cc13b930 msgr2=0x7fd2cc12ead0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:42:19.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.676+0000 7fd2beffd640 1 --2- 172.21.15.37:0/1454762427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2cc13b930 0x7fd2cc12ead0 secure :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd2c0010790 tx=0x7fd2c0010c50 comp rx=0 tx=0).stop 2024-04-26T00:42:19.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.676+0000 7fd2d2ad0640 1 --2- 172.21.15.37:0/1454762427 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd2b0035e80 0x7fd2b0038340 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc13a1c0 tx=0x7fd2c4009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:19.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.684+0000 7fd2d4533640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:42:19.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.684+0000 7fd2d4533640 1 --2- 172.21.15.37:0/1454762427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2cc12f010 0x7fd2cc12f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:19.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.684+0000 7fd2d4533640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:19.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.684+0000 7fd2d4533640 1 -- 172.21.15.37:0/1454762427 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd2cc12f960 con 0x7fd2cc12f010 2024-04-26T00:42:19.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.684+0000 7fd2d22cf640 1 --2- 172.21.15.37:0/1454762427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2cc12f010 0x7fd2cc12f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:19.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.684+0000 7fd2d22cf640 10 monclient: get_auth_request con 0x7fd2cc12f010 auth_method 0 2024-04-26T00:42:19.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.684+0000 7fd2d22cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:19.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.684+0000 7fd2d22cf640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:19.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.684+0000 7fd2d22cf640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:19.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.684+0000 7fd2d22cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:19.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.684+0000 7fd2d22cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:19.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.684+0000 7fd2d22cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:19.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.684+0000 7fd2d22cf640 10 monclient(hunting): handle_auth_done global_id 254321 payload 824 2024-04-26T00:42:19.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.684+0000 7fd2d22cf640 1 -- 172.21.15.37:0/1454762427 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd2cc138730 con 0x7fd2cc12f010 2024-04-26T00:42:19.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.684+0000 7fd2d22cf640 1 -- 172.21.15.37:0/1454762427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2cc12f010 msgr2=0x7fd2cc12f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:19.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.684+0000 7fd2d22cf640 1 --2- 172.21.15.37:0/1454762427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2cc12f010 0x7fd2cc12f420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd2c001a000 tx=0x7fd2c0004740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:19.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.684+0000 7fd2beffd640 1 -- 172.21.15.37:0/1454762427 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c0019030 con 0x7fd2cc12f010 2024-04-26T00:42:19.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.684+0000 7fd2beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:19.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.684+0000 7fd2beffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:42:19.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.684+0000 7fd2beffd640 10 monclient: dump: 2024-04-26T00:42:19.695 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:19.695 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:19.695 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:19.695 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:19.695 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:19.696 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:19.696 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:19.696 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:19.696 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:19.696 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:19.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.684+0000 7fd2beffd640 1 -- 172.21.15.37:0/1454762427 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7fd2c0015b80 con 0x7fd2cc12f010 2024-04-26T00:42:19.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.684+0000 7fd2beffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:42:19.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.684+0000 7fd2beffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:42:19.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.684+0000 7fd2beffd640 1 -- 172.21.15.37:0/1454762427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2cc12f010 msgr2=0x7fd2cc12f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:19.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.684+0000 7fd2beffd640 1 --2- 172.21.15.37:0/1454762427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2cc12f010 0x7fd2cc12f420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd2c001a000 tx=0x7fd2c0004740 comp rx=0 tx=0).stop 2024-04-26T00:42:19.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.684+0000 7fd2beffd640 1 -- 172.21.15.37:0/1454762427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2cc12f010 msgr2=0x7fd2cc12f420 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:42:19.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.684+0000 7fd2beffd640 1 --2- 172.21.15.37:0/1454762427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2cc12f010 0x7fd2cc12f420 secure :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd2c001a000 tx=0x7fd2c0004740 comp rx=0 tx=0).stop 2024-04-26T00:42:19.697 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:42:19.697 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:42:19.697 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:42:19.697 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:42:19.697 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 160, 2024-04-26T00:42:19.697 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 108454, 2024-04-26T00:42:19.697 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:42:19.697 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:42:19.697 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:42:19.697 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:42:19.697 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:42:19.697 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 1, 2024-04-26T00:42:19.697 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:42:19.697 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:42:19.697 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:42:19.697 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:42:19.698 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:42:19.698 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:42:19.698 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:42:19.698 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:42:19.698 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:42:19.698 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:42:19.698 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:42:19.698 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:42:19.698 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:42:19.698 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:42:19.698 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:42:19.698 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:42:19.699 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:42:19.699 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:42:19.699 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:42:19.699 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:42:19.699 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:42:19.699 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:42:19.699 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:42:19.699 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:42:19.699 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:42:19.699 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:42:19.699 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:42:19.699 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:42:19.699 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:19.700 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:19.700 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:42:19.700 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:42:19.700 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:42:19.700 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:42:19.700 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:42:19.700 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:42:19.700 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:42:19.700 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:42:19.700 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:42:19.700 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:42:19.700 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:42:19.700 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:42:19.700 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:42:19.701 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:42:19.701 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:42:19.701 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:42:19.701 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:42:19.701 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:42:19.701 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:42:19.701 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:42:19.701 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:42:19.701 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:42:19.701 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:42:19.701 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:42:19.701 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:42:19.701 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:42:19.701 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:42:19.702 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:42:19.702 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:42:19.702 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:42:19.702 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:19.702 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:42:19.702 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:19.702 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:42:19.702 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:42:19.702 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:42:19.702 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:42:19.702 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:19.703 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:42:19.703 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:42:19.703 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:19.703 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:19.703 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:19.703 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:42:19.703 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:42:19.703 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:19.703 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:19.703 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:19.703 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:19.703 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:42:19.703 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:42:19.703 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:42:19.703 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:42:19.703 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:42:19.704 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:19.704 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:19.704 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:42:19.704 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:42:19.704 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:42:19.704 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:42:19.704 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:19.704 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:42:19.704 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:42:19.704 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:19.704 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:19.704 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:19.704 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:42:19.704 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:42:19.704 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:19.704 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:19.705 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:19.705 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:19.705 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:42:19.705 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:42:19.705 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:42:19.705 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:42:19.705 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:42:19.705 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:19.705 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:19.705 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:42:19.705 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:42:19.705 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:42:19.705 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:42:19.705 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:19.705 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:42:19.706 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:42:19.706 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:19.706 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:19.706 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:19.706 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:42:19.706 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:42:19.706 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:19.706 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:19.706 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:19.706 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:19.706 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:42:19.706 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:42:19.706 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:42:19.706 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:42:19.706 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:42:19.706 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:19.707 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:19.707 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:19.707 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:42:19.707 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:42:19.707 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:19.707 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:42:19.707 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:42:19.707 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:42:19.707 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:19.707 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:42:19.707 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:42:19.707 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:19.707 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:42:19.707 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:42:19.707 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:42:19.707 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:19.708 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:42:19.708 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:42:19.708 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:19.708 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:42:19.708 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:42:19.708 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:42:19.708 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:19.708 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:19.708 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:19.708 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:42:19.708 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:42:19.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.688+0000 7fd2d4533640 1 -- 172.21.15.37:0/1454762427 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd2b0035e80 msgr2=0x7fd2b0038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:19.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.688+0000 7fd2d4533640 1 --2- 172.21.15.37:0/1454762427 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd2b0035e80 0x7fd2b0038340 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc13a1c0 tx=0x7fd2c4009190 comp rx=0 tx=0).stop 2024-04-26T00:42:19.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.688+0000 7fd2d4533640 10 monclient: shutdown 2024-04-26T00:42:19.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.688+0000 7fd2d4533640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:42:19.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.688+0000 7fd2d4533640 1 -- 172.21.15.37:0/1454762427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2cc0a06e0 msgr2=0x7fd2cc1cd2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:19.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.688+0000 7fd2d4533640 1 --2- 172.21.15.37:0/1454762427 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2cc0a06e0 0x7fd2cc1cd2f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd2c80af040 tx=0x7fd2c8019650 comp rx=0 tx=0).stop 2024-04-26T00:42:19.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.688+0000 7fd2d2ad0640 1 -- 172.21.15.37:0/1454762427 reap_dead start 2024-04-26T00:42:19.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.688+0000 7fd2d2ad0640 1 -- 172.21.15.37:0/1454762427 reap_dead start 2024-04-26T00:42:19.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.688+0000 7fd2d4533640 1 -- 172.21.15.37:0/1454762427 shutdown_connections 2024-04-26T00:42:19.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.688+0000 7fd2d4533640 1 -- 172.21.15.37:0/1454762427 >> 172.21.15.37:0/1454762427 conn(0x7fd2cc0a1d50 msgr2=0x7fd2cc131be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:42:19.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.688+0000 7fd2d4533640 1 -- 172.21.15.37:0/1454762427 shutdown_connections 2024-04-26T00:42:19.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.688+0000 7fd2d4533640 1 -- 172.21.15.37:0/1454762427 wait complete. 2024-04-26T00:42:19.710 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:42:19.711 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:42:19.711 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:42:19.717 INFO:tasks.rados.rados.0.smithi045.stdout:3951: finishing rollback tid 0 to smithi04526579-12 2024-04-26T00:42:19.717 INFO:tasks.rados.rados.0.smithi045.stdout:3951: finishing rollback tid 1 to smithi04526579-12 2024-04-26T00:42:19.717 INFO:tasks.rados.rados.0.smithi045.stdout:3951: finishing rollback tid 2 to smithi04526579-12 2024-04-26T00:42:19.717 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 12 v 2501 (ObjNum 1290 snap 401 seq_num 1290) dirty exists 2024-04-26T00:42:19.717 INFO:tasks.rados.rados.0.smithi045.stdout:3952: finishing write tid 1 to smithi04526579-27 2024-04-26T00:42:19.717 INFO:tasks.rados.rados.0.smithi045.stdout:3952: finishing write tid 2 to smithi04526579-27 2024-04-26T00:42:19.717 INFO:tasks.rados.rados.0.smithi045.stdout:3952: finishing write tid 3 to smithi04526579-27 2024-04-26T00:42:19.717 INFO:tasks.rados.rados.0.smithi045.stdout:3952: finishing write tid 4 to smithi04526579-27 2024-04-26T00:42:19.717 INFO:tasks.rados.rados.0.smithi045.stdout:3952: finishing write tid 5 to smithi04526579-27 2024-04-26T00:42:19.717 INFO:tasks.rados.rados.0.smithi045.stdout:3952: oid 27 updating version 0 to 2889 2024-04-26T00:42:19.717 INFO:tasks.rados.rados.0.smithi045.stdout:3952: oid 27 updating version 2889 to 2892 2024-04-26T00:42:19.718 INFO:tasks.rados.rados.0.smithi045.stdout:3952: oid 27 version 2892 is already newer than 2891 2024-04-26T00:42:19.718 INFO:tasks.rados.rados.0.smithi045.stdout:3952: oid 27 version 2892 is already newer than 2890 2024-04-26T00:42:19.718 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 2892 (ObjNum 1408 snap 438 seq_num 1408) dirty exists 2024-04-26T00:42:19.718 INFO:tasks.rados.rados.0.smithi045.stdout:3952: left oid 27 (ObjNum 1408 snap 438 seq_num 1408) 2024-04-26T00:42:19.718 INFO:tasks.rados.rados.0.smithi045.stdout:3954: finishing write tid 1 to smithi04526579-33 2024-04-26T00:42:19.718 INFO:tasks.rados.rados.0.smithi045.stdout:3954: finishing write tid 2 to smithi04526579-33 2024-04-26T00:42:19.718 INFO:tasks.rados.rados.0.smithi045.stdout:3954: finishing write tid 3 to smithi04526579-33 2024-04-26T00:42:19.718 INFO:tasks.rados.rados.0.smithi045.stdout:3954: finishing write tid 4 to smithi04526579-33 2024-04-26T00:42:19.718 INFO:tasks.rados.rados.0.smithi045.stdout:3954: finishing write tid 5 to smithi04526579-33 2024-04-26T00:42:19.718 INFO:tasks.rados.rados.0.smithi045.stdout:3954: oid 33 updating version 0 to 3024 2024-04-26T00:42:19.718 INFO:tasks.rados.rados.0.smithi045.stdout:3954: oid 33 updating version 3024 to 3027 2024-04-26T00:42:19.718 INFO:tasks.rados.rados.0.smithi045.stdout:3954: oid 33 version 3027 is already newer than 3025 2024-04-26T00:42:19.718 INFO:tasks.rados.rados.0.smithi045.stdout:3954: oid 33 version 3027 is already newer than 3026 2024-04-26T00:42:19.718 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 3027 (ObjNum 1409 snap 438 seq_num 1409) dirty exists 2024-04-26T00:42:19.719 INFO:tasks.rados.rados.0.smithi045.stdout:3954: left oid 33 (ObjNum 1409 snap 438 seq_num 1409) 2024-04-26T00:42:19.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.824+0000 7f7674568640 10 monclient: get_monmap_and_config 2024-04-26T00:42:19.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.824+0000 7f7674568640 10 monclient: build_initial_monmap 2024-04-26T00:42:19.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.824+0000 7f7674568640 10 monclient: monmap: 2024-04-26T00:42:19.828 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:42:19.828 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:19.828 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:42:19.830096+0000 2024-04-26T00:42:19.828 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:42:19.830096+0000 2024-04-26T00:42:19.828 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:42:19.828 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:19.829 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:42:19.829 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:42:19.829 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:42:19.829 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:42:19.829 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:42:19.829 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:19.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.824+0000 7f7674568640 1 Processor -- start 2024-04-26T00:42:19.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.824+0000 7f7674568640 1 -- start start 2024-04-26T00:42:19.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.824+0000 7f7674568640 10 monclient: init 2024-04-26T00:42:19.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7674568640 10 monclient: _reopen_session rank -1 2024-04-26T00:42:19.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7674568640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-04-26T00:42:19.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7674568640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f766c130eb0 addr v1:172.21.15.37:6789/0 2024-04-26T00:42:19.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7674568640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f766c09ef20 0x7f766c138eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:19.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7674568640 10 monclient(hunting): picked mon.noname-a con 0x7f766c09ef20 addr v2:172.21.15.37:3300/0 2024-04-26T00:42:19.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7674568640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f766c1393f0 0x7f766c13b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:19.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7674568640 10 monclient(hunting): picked mon.noname-b con 0x7f766c1393f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:42:19.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7674568640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f766c09c190 con 0x7f766c130eb0 2024-04-26T00:42:19.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7674568640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:19.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7674568640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f766c09df60 con 0x7f766c09ef20 2024-04-26T00:42:19.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7674568640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:19.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7674568640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f766c0a11c0 con 0x7f766c1393f0 2024-04-26T00:42:19.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7674568640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:19.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7674568640 10 monclient(hunting): authenticate will time out at 2321.136244s 2024-04-26T00:42:19.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7671b03640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f766c09ef20 0x7f766c138eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:19.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7672b05640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f766c1393f0 0x7f766c13b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:19.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7672304640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f766c130eb0 0x7f766c131290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:44892/0 (socket says 172.21.15.37:44892) 2024-04-26T00:42:19.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7671b03640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f766c09ef20 0x7f766c138eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40220/0 (socket says 172.21.15.37:40220) 2024-04-26T00:42:19.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7672b05640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f766c1393f0 0x7f766c13b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51044/0 (socket says 172.21.15.37:51044) 2024-04-26T00:42:19.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7672304640 1 -- 172.21.15.37:0/2592769897 learned_addr learned my addr 172.21.15.37:0/2592769897 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:42:19.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7671b03640 10 monclient(hunting): get_auth_request con 0x7f766c09ef20 auth_method 0 2024-04-26T00:42:19.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7671b03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:19.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7671b03640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:19.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7671b03640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:19.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7672b05640 10 monclient(hunting): get_auth_request con 0x7f766c1393f0 auth_method 0 2024-04-26T00:42:19.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7672b05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:19.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7672b05640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:19.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7672b05640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:19.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7672b05640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:19.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7672b05640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:19.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7672b05640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:19.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7671b03640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:19.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7671b03640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:19.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7671b03640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:19.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7672b05640 10 monclient(hunting): handle_auth_done global_id 254324 payload 274 2024-04-26T00:42:19.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7672b05640 1 -- 172.21.15.37:0/2592769897 >> v2:172.21.15.37:3300/0 conn(0x7f766c09ef20 msgr2=0x7f766c138eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:42:19.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7672b05640 1 --2- 172.21.15.37:0/2592769897 >> v2:172.21.15.37:3300/0 conn(0x7f766c09ef20 0x7f766c138eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:19.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7672b05640 1 -- 172.21.15.37:0/2592769897 >> v1:172.21.15.37:6789/0 conn(0x7f766c130eb0 legacy=0x7f766c131290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:42:19.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7672b05640 10 monclient: _finish_hunting 0 2024-04-26T00:42:19.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7672b05640 1 monclient: found mon.noname-b 2024-04-26T00:42:19.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7672b05640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:42:19.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7672b05640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:42:19.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.828+0000 7f7672b05640 1 -- 172.21.15.37:0/2592769897 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f766c09dc80 con 0x7f766c1393f0 2024-04-26T00:42:19.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7672b05640 1 -- 172.21.15.37:0/2592769897 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f766c1393f0 msgr2=0x7f766c13b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:19.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7672b05640 1 --2- 172.21.15.37:0/2592769897 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f766c1393f0 0x7f766c13b7e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7664009d90 tx=0x7f766401c7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d8f77c755c50c53 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:19.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7671302640 1 -- 172.21.15.37:0/2592769897 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76640b2070 con 0x7f766c1393f0 2024-04-26T00:42:19.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7671302640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:19.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7671302640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:42:19.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7671302640 10 monclient: dump: 2024-04-26T00:42:19.839 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:19.839 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:19.839 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:19.839 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:19.839 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:19.840 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:19.840 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:19.840 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:19.840 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:19.840 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:19.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7671302640 10 monclient: _finish_auth 0 2024-04-26T00:42:19.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7671302640 10 monclient: _check_auth_tickets 2024-04-26T00:42:19.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7671302640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:42:19.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7671302640 1 -- 172.21.15.37:0/2592769897 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76640a5860 con 0x7f766c1393f0 2024-04-26T00:42:19.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7671302640 10 monclient: handle_config config(0 keys) 2024-04-26T00:42:19.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7671302640 1 -- 172.21.15.37:0/2592769897 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76640a5c30 con 0x7f766c1393f0 2024-04-26T00:42:19.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7674568640 5 monclient: authenticate success, global_id 254324 2024-04-26T00:42:19.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7674568640 10 monclient: get_monmap_and_config success 2024-04-26T00:42:19.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7674568640 4 set_mon_vals no callback set 2024-04-26T00:42:19.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7671302640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:19.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7671302640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:42:19.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7671302640 10 monclient: dump: 2024-04-26T00:42:19.841 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:19.841 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:19.841 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:19.841 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:19.841 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:19.841 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:19.841 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:19.841 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:19.841 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:19.842 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:19.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7674568640 10 monclient: shutdown 2024-04-26T00:42:19.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7674568640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:42:19.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7674568640 1 -- 172.21.15.37:0/2592769897 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f766c1393f0 msgr2=0x7f766c13b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:19.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7674568640 1 --2- 172.21.15.37:0/2592769897 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f766c1393f0 0x7f766c13b7e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7664009d90 tx=0x7f766401c7c0 comp rx=0 tx=0).stop 2024-04-26T00:42:19.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7674568640 1 -- 172.21.15.37:0/2592769897 shutdown_connections 2024-04-26T00:42:19.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7674568640 1 --2- 172.21.15.37:0/2592769897 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f766c1393f0 0x7f766c13b7e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:19.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7674568640 1 --2- 172.21.15.37:0/2592769897 >> v2:172.21.15.37:3300/0 conn(0x7f766c09ef20 0x7f766c138eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:19.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7674568640 1 -- 172.21.15.37:0/2592769897 >> 172.21.15.37:0/2592769897 conn(0x7f766c0a1d50 msgr2=0x7f766c0a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:42:19.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7674568640 1 -- 172.21.15.37:0/2592769897 shutdown_connections 2024-04-26T00:42:19.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7674568640 1 -- 172.21.15.37:0/2592769897 wait complete. 2024-04-26T00:42:19.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7674568640 10 monclient: build_initial_monmap 2024-04-26T00:42:19.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7674568640 10 monclient: monmap: 2024-04-26T00:42:19.843 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:42:19.843 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:42:19.843 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:42:19.843 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:42:19.843 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:42:19.843 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:42:19.843 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:42:19.843 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:42:19.843 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:42:19.843 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:19.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7674568640 1 Processor -- start 2024-04-26T00:42:19.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7674568640 1 -- start start 2024-04-26T00:42:19.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7674568640 10 monclient: init 2024-04-26T00:42:19.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7674568640 10 monclient: _reopen_session rank -1 2024-04-26T00:42:19.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7674568640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:42:19.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7674568640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c09ef20 0x7f766c1322b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:19.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7674568640 10 monclient(hunting): picked mon.noname-b con 0x7f766c09ef20 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:42:19.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7674568640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f766c130eb0 0x7f766c1327f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:19.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7674568640 10 monclient(hunting): picked mon.noname-c con 0x7f766c130eb0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:42:19.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7674568640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f766c1393f0 0x7f766c132d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:19.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7674568640 10 monclient(hunting): picked mon.noname-a con 0x7f766c1393f0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:42:19.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7674568640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:19.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7674568640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f766c09fdd0 con 0x7f766c1393f0 2024-04-26T00:42:19.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7674568640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:19.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7674568640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f766c09fc50 con 0x7f766c09ef20 2024-04-26T00:42:19.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7674568640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:19.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7674568640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f766c09ff50 con 0x7f766c130eb0 2024-04-26T00:42:19.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7674568640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:19.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7674568640 10 monclient(hunting): authenticate will time out at 2321.143383s 2024-04-26T00:42:19.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7672b05640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f766c1393f0 0x7f766c132d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:19.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7671b03640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f766c130eb0 0x7f766c1327f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:19.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7672b05640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f766c1393f0 0x7f766c132d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40226/0 (socket says 172.21.15.37:40226) 2024-04-26T00:42:19.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7672b05640 1 -- 172.21.15.37:0/2910754178 learned_addr learned my addr 172.21.15.37:0/2910754178 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:42:19.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7672b05640 10 monclient(hunting): get_auth_request con 0x7f766c1393f0 auth_method 0 2024-04-26T00:42:19.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7672b05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:19.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7672b05640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:19.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7672b05640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:19.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7671b03640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f766c130eb0 0x7f766c1327f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51050/0 (socket says 172.21.15.37:51050) 2024-04-26T00:42:19.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7671b03640 10 monclient(hunting): get_auth_request con 0x7f766c130eb0 auth_method 0 2024-04-26T00:42:19.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7671b03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:19.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7671b03640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:19.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7671b03640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:19.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7672b05640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:19.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7672b05640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:19.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7672b05640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:19.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7671b03640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:19.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7671b03640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:19.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7671b03640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:19.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7672b05640 10 monclient(hunting): handle_auth_done global_id 264213 payload 824 2024-04-26T00:42:19.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7672304640 1 --2- 172.21.15.37:0/2910754178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c09ef20 0x7f766c1322b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:19.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7672b05640 1 -- 172.21.15.37:0/2910754178 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f766c130eb0 msgr2=0x7f766c1327f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:19.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7672b05640 1 --2- 172.21.15.37:0/2910754178 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f766c130eb0 0x7f766c1327f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:19.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7672b05640 1 -- 172.21.15.37:0/2910754178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c09ef20 msgr2=0x7f766c1322b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:19.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7672b05640 1 --2- 172.21.15.37:0/2910754178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c09ef20 0x7f766c1322b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:19.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7672b05640 10 monclient: _finish_hunting 0 2024-04-26T00:42:19.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7672b05640 1 monclient: found mon.noname-a 2024-04-26T00:42:19.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7672b05640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:42:19.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7672b05640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:42:19.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7672b05640 1 -- 172.21.15.37:0/2910754178 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f766c0a1470 con 0x7f766c1393f0 2024-04-26T00:42:19.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7672304640 10 monclient: get_auth_request con 0x7f766c09ef20 auth_method 0 2024-04-26T00:42:19.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.832+0000 7f7672304640 1 --2- 172.21.15.37:0/2910754178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c09ef20 0x7f766c1322b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:42:19.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7672b05640 1 -- 172.21.15.37:0/2910754178 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f766c1393f0 msgr2=0x7f766c132d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:19.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7672b05640 1 --2- 172.21.15.37:0/2910754178 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f766c1393f0 0x7f766c132d30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f76640b0040 tx=0x7f7664019560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:19.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7662ffd640 1 -- 172.21.15.37:0/2910754178 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76640b2050 con 0x7f766c1393f0 2024-04-26T00:42:19.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7662ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:19.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7662ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:42:19.850 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7662ffd640 10 monclient: dump: 2024-04-26T00:42:19.850 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:19.850 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:19.850 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:19.850 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:19.850 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:19.850 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:19.850 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:19.850 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:19.851 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:19.851 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:19.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7662ffd640 10 monclient: _finish_auth 0 2024-04-26T00:42:19.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7662ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:42:19.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7662ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:42:19.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7662ffd640 1 -- 172.21.15.37:0/2910754178 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76640be070 con 0x7f766c1393f0 2024-04-26T00:42:19.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7674568640 5 monclient: authenticate success, global_id 264213 2024-04-26T00:42:19.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7674568640 10 monclient: _renew_subs 2024-04-26T00:42:19.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7674568640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:42:19.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7674568640 1 -- 172.21.15.37:0/2910754178 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f766c135ca0 con 0x7f766c1393f0 2024-04-26T00:42:19.851 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7662ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:42:19.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7662ffd640 1 -- 172.21.15.37:0/2910754178 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76640ada60 con 0x7f766c1393f0 2024-04-26T00:42:19.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7662ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:19.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7662ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:42:19.852 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7662ffd640 10 monclient: dump: 2024-04-26T00:42:19.852 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:19.852 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:19.852 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:19.852 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:19.852 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:19.852 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:19.852 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:19.852 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:19.852 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:19.852 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:19.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7670b01640 4 set_mon_vals no callback set 2024-04-26T00:42:19.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7674568640 10 monclient: _renew_subs 2024-04-26T00:42:19.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7674568640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:42:19.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7674568640 1 -- 172.21.15.37:0/2910754178 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f766c1356e0 con 0x7f766c1393f0 2024-04-26T00:42:19.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7674568640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:19.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7674568640 1 --2- 172.21.15.37:0/2910754178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c1347b0 0x7f766c12bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:19.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7674568640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:19.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7674568640 1 -- 172.21.15.37:0/2910754178 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f766c135920 con 0x7f766c1347b0 2024-04-26T00:42:19.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7662ffd640 1 -- 172.21.15.37:0/2910754178 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f76640adc40 con 0x7f766c1393f0 2024-04-26T00:42:19.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7672304640 1 --2- 172.21.15.37:0/2910754178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c1347b0 0x7f766c12bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:19.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7662ffd640 1 --2- 172.21.15.37:0/2910754178 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7650035ed0 0x7f7650038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:19.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7672304640 10 monclient: get_auth_request con 0x7f766c1347b0 auth_method 0 2024-04-26T00:42:19.853 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7672304640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:19.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7672304640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:19.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7672304640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:19.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7662ffd640 1 -- 172.21.15.37:0/2910754178 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(891..891 src has 822..891) ==== 3979+0+0 (secure 0 0 0) 0x7f76640e36e0 con 0x7f766c1393f0 2024-04-26T00:42:19.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7672304640 1 -- 172.21.15.37:0/2910754178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c1347b0 msgr2=0x7f766c12bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:42:19.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7672304640 1 -- 172.21.15.37:0/2910754178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c1347b0 msgr2=0x7f766c12bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:42:19.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7672304640 1 --2- 172.21.15.37:0/2910754178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c1347b0 0x7f766c12bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:42:19.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7672304640 1 --2- 172.21.15.37:0/2910754178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c1347b0 0x7f766c12bcd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:42:19.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7671b03640 1 --2- 172.21.15.37:0/2910754178 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7650035ed0 0x7f7650038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:19.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.836+0000 7f7671b03640 10 monclient: get_auth_request con 0x7f7650035ed0 auth_method 0 2024-04-26T00:42:19.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:19.840+0000 7f7671b03640 1 --2- 172.21.15.37:0/2910754178 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7650035ed0 0x7f7650038390 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f766c133940 tx=0x7f765c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:20.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:20.040+0000 7f7672304640 1 --2- 172.21.15.37:0/2910754178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c1347b0 0x7f766c12bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:20.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:20.040+0000 7f7672304640 10 monclient: get_auth_request con 0x7f766c1347b0 auth_method 0 2024-04-26T00:42:20.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:20.040+0000 7f7672304640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:20.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:20.040+0000 7f7672304640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:20.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:20.040+0000 7f7672304640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:42:20.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:20.040+0000 7f7672304640 1 -- 172.21.15.37:0/2910754178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c1347b0 msgr2=0x7f766c12bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:42:20.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:20.040+0000 7f7672304640 1 -- 172.21.15.37:0/2910754178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c1347b0 msgr2=0x7f766c12bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:42:20.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:20.040+0000 7f7672304640 1 --2- 172.21.15.37:0/2910754178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c1347b0 0x7f766c12bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:42:20.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:20.040+0000 7f7672304640 1 --2- 172.21.15.37:0/2910754178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c1347b0 0x7f766c12bcd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:42:20.397 INFO:tasks.rados.rados.0.smithi045.stdout:3950: done (5 left) 2024-04-26T00:42:20.398 INFO:tasks.rados.rados.0.smithi045.stdout:3951: done (4 left) 2024-04-26T00:42:20.398 INFO:tasks.rados.rados.0.smithi045.stdout:3952: done (3 left) 2024-04-26T00:42:20.398 INFO:tasks.rados.rados.0.smithi045.stdout:3953: done (2 left) 2024-04-26T00:42:20.398 INFO:tasks.rados.rados.0.smithi045.stdout:3954: done (1 left) 2024-04-26T00:42:20.398 INFO:tasks.rados.rados.0.smithi045.stdout:3955: done (0 left) 2024-04-26T00:42:20.398 INFO:tasks.rados.rados.0.smithi045.stdout:3956: rollback oid 42 current snap is 439 2024-04-26T00:42:20.399 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 42 to 436 2024-04-26T00:42:20.399 INFO:tasks.rados.rados.0.smithi045.stdout:3957: delete oid 50 current snap is 439 2024-04-26T00:42:20.410 INFO:tasks.rados.rados.0.smithi045.stdout:3957: done (1 left) 2024-04-26T00:42:20.410 INFO:tasks.rados.rados.0.smithi045.stdout:3958: copy_from oid 23 from oid 11 current snap is 439 2024-04-26T00:42:20.410 INFO:tasks.rados.rados.0.smithi045.stdout:3959: write (excl) oid 34 current snap is 439 2024-04-26T00:42:20.410 INFO:tasks.rados.rados.0.smithi045.stdout:3959: seq_num 1411 ranges {514677=501644,1501622=781829,2964784=741516} 2024-04-26T00:42:20.420 INFO:tasks.rados.rados.0.smithi045.stdout:3959: writing smithi04526579-34 from 514677 to 1016321 tid 1 2024-04-26T00:42:20.433 INFO:tasks.rados.rados.0.smithi045.stdout:3959: writing smithi04526579-34 from 1501622 to 2283451 tid 2 2024-04-26T00:42:20.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:20.444+0000 7f7672304640 1 --2- 172.21.15.37:0/2910754178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c1347b0 0x7f766c12bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:20.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:20.444+0000 7f7672304640 10 monclient: get_auth_request con 0x7f766c1347b0 auth_method 0 2024-04-26T00:42:20.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:20.444+0000 7f7672304640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:20.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:20.444+0000 7f7672304640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:20.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:20.444+0000 7f7672304640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:42:20.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:20.444+0000 7f7672304640 1 -- 172.21.15.37:0/2910754178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c1347b0 msgr2=0x7f766c12bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:42:20.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:20.444+0000 7f7672304640 1 -- 172.21.15.37:0/2910754178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c1347b0 msgr2=0x7f766c12bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:42:20.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:20.444+0000 7f7672304640 1 --2- 172.21.15.37:0/2910754178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c1347b0 0x7f766c12bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:42:20.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:20.444+0000 7f7672304640 1 --2- 172.21.15.37:0/2910754178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c1347b0 0x7f766c12bcd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:42:20.449 INFO:tasks.rados.rados.0.smithi045.stdout:3959: writing smithi04526579-34 from 2964784 to 3706300 tid 3 2024-04-26T00:42:20.449 INFO:tasks.rados.rados.0.smithi045.stdout:3960: write oid 10 current snap is 439 2024-04-26T00:42:20.449 INFO:tasks.rados.rados.0.smithi045.stdout:3956: finishing rollback tid 0 to smithi04526579-42 2024-04-26T00:42:20.449 INFO:tasks.rados.rados.0.smithi045.stdout:3956: finishing rollback tid 1 to smithi04526579-42 2024-04-26T00:42:20.449 INFO:tasks.rados.rados.0.smithi045.stdout:3960: seq_num 1412 ranges {531484=576893,1843751=525487,2875854=669141} 2024-04-26T00:42:20.460 INFO:tasks.rados.rados.0.smithi045.stdout:3960: writing smithi04526579-10 from 531484 to 1108377 tid 1 2024-04-26T00:42:20.474 INFO:tasks.rados.rados.0.smithi045.stdout:3960: writing smithi04526579-10 from 1843751 to 2369238 tid 2 2024-04-26T00:42:20.485 INFO:tasks.rados.rados.0.smithi045.stdout:3960: writing smithi04526579-10 from 2875854 to 3544995 tid 3 2024-04-26T00:42:20.486 INFO:tasks.rados.rados.0.smithi045.stdout:3961: delete oid 35 current snap is 439 2024-04-26T00:42:20.486 INFO:tasks.rados.rados.0.smithi045.stdout:3956: finishing rollback tid 2 to smithi04526579-42 2024-04-26T00:42:20.486 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 42 v 2895 (ObjNum 1363 snap 421 seq_num 1363) dirty exists 2024-04-26T00:42:20.486 INFO:tasks.rados.rados.0.smithi045.stdout:3958: finishing copy_from to smithi04526579-23 2024-04-26T00:42:20.486 INFO:tasks.rados.rados.0.smithi045.stdout:3958: got expected ENOENT (src dne) 2024-04-26T00:42:20.486 INFO:tasks.rados.rados.0.smithi045.stdout:3958: finishing copy_from racing read to smithi04526579-23 2024-04-26T00:42:20.486 INFO:tasks.rados.rados.0.smithi045.stdout:3959: finishing write tid 1 to smithi04526579-34 2024-04-26T00:42:20.486 INFO:tasks.rados.rados.0.smithi045.stdout:3959: finishing write tid 2 to smithi04526579-34 2024-04-26T00:42:20.486 INFO:tasks.rados.rados.0.smithi045.stdout:3959: finishing write tid 3 to smithi04526579-34 2024-04-26T00:42:20.486 INFO:tasks.rados.rados.0.smithi045.stdout:3959: finishing write tid 4 to smithi04526579-34 2024-04-26T00:42:20.486 INFO:tasks.rados.rados.0.smithi045.stdout:3959: finishing write tid 5 to smithi04526579-34 2024-04-26T00:42:20.486 INFO:tasks.rados.rados.0.smithi045.stdout:3959: oid 34 updating version 0 to 2881 2024-04-26T00:42:20.487 INFO:tasks.rados.rados.0.smithi045.stdout:3959: oid 34 version 2881 is already newer than 2880 2024-04-26T00:42:20.487 INFO:tasks.rados.rados.0.smithi045.stdout:3959: oid 34 updating version 2881 to 2882 2024-04-26T00:42:20.487 INFO:tasks.rados.rados.0.smithi045.stdout:3959: oid 34 updating version 2882 to 2883 2024-04-26T00:42:20.487 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 34 v 2883 (ObjNum 1411 snap 439 seq_num 1411) dirty exists 2024-04-26T00:42:20.487 INFO:tasks.rados.rados.0.smithi045.stdout:3959: left oid 34 (ObjNum 1411 snap 439 seq_num 1411) 2024-04-26T00:42:20.487 INFO:tasks.rados.rados.0.smithi045.stdout:3960: finishing write tid 1 to smithi04526579-10 2024-04-26T00:42:20.487 INFO:tasks.rados.rados.0.smithi045.stdout:3960: finishing write tid 2 to smithi04526579-10 2024-04-26T00:42:20.494 INFO:tasks.rados.rados.0.smithi045.stdout:3960: finishing write tid 3 to smithi04526579-10 2024-04-26T00:42:20.495 INFO:tasks.rados.rados.0.smithi045.stdout:3960: finishing write tid 4 to smithi04526579-10 2024-04-26T00:42:20.496 INFO:tasks.rados.rados.0.smithi045.stdout:3956: done (4 left) 2024-04-26T00:42:20.496 INFO:tasks.rados.rados.0.smithi045.stdout:3958: done (3 left) 2024-04-26T00:42:20.496 INFO:tasks.rados.rados.0.smithi045.stdout:3959: done (2 left) 2024-04-26T00:42:20.496 INFO:tasks.rados.rados.0.smithi045.stdout:3961: done (1 left) 2024-04-26T00:42:20.496 INFO:tasks.rados.rados.0.smithi045.stdout:3962: rollback oid 33 current snap is 439 2024-04-26T00:42:20.497 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 33 to 430 2024-04-26T00:42:20.497 INFO:tasks.rados.rados.0.smithi045.stdout:3960: finishing write tid 5 to smithi04526579-10 2024-04-26T00:42:20.497 INFO:tasks.rados.rados.0.smithi045.stdout:3960: oid 10 updating version 0 to 2645 2024-04-26T00:42:20.497 INFO:tasks.rados.rados.0.smithi045.stdout:3960: oid 10 updating version 2645 to 2648 2024-04-26T00:42:20.497 INFO:tasks.rados.rados.0.smithi045.stdout:3960: oid 10 version 2648 is already newer than 2647 2024-04-26T00:42:20.497 INFO:tasks.rados.rados.0.smithi045.stdout:3960: oid 10 version 2648 is already newer than 2646 2024-04-26T00:42:20.497 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 2648 (ObjNum 1412 snap 439 seq_num 1412) dirty exists 2024-04-26T00:42:20.497 INFO:tasks.rados.rados.0.smithi045.stdout:3960: left oid 10 (ObjNum 1412 snap 439 seq_num 1412) 2024-04-26T00:42:20.497 INFO:tasks.rados.rados.0.smithi045.stdout:3960: done (1 left) 2024-04-26T00:42:20.497 INFO:tasks.rados.rados.0.smithi045.stdout:3963: snap_remove snap 436 2024-04-26T00:42:21.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:21.248+0000 7f7672304640 1 --2- 172.21.15.37:0/2910754178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c1347b0 0x7f766c12bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:21.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:21.248+0000 7f7672304640 10 monclient: get_auth_request con 0x7f766c1347b0 auth_method 0 2024-04-26T00:42:21.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:21.248+0000 7f7672304640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:21.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:21.248+0000 7f7672304640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:21.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:21.248+0000 7f7672304640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:42:21.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:21.248+0000 7f7672304640 1 -- 172.21.15.37:0/2910754178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c1347b0 msgr2=0x7f766c12bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:42:21.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:21.248+0000 7f7672304640 1 -- 172.21.15.37:0/2910754178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c1347b0 msgr2=0x7f766c12bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:42:21.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:21.248+0000 7f7672304640 1 --2- 172.21.15.37:0/2910754178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c1347b0 0x7f766c12bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:42:21.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:21.248+0000 7f7672304640 1 --2- 172.21.15.37:0/2910754178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c1347b0 0x7f766c12bcd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:42:21.285 INFO:tasks.rados.rados.0.smithi045.stdout:3963: done (1 left) 2024-04-26T00:42:21.286 INFO:tasks.rados.rados.0.smithi045.stdout:3964: copy_from oid 6 from oid 29 current snap is 439 2024-04-26T00:42:21.286 INFO:tasks.rados.rados.0.smithi045.stdout:3962: finishing rollback tid 0 to smithi04526579-33 2024-04-26T00:42:21.286 INFO:tasks.rados.rados.0.smithi045.stdout:3962: finishing rollback tid 1 to smithi04526579-33 2024-04-26T00:42:21.286 INFO:tasks.rados.rados.0.smithi045.stdout:3962: finishing rollback tid 2 to smithi04526579-33 2024-04-26T00:42:21.286 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 33 v 3031 (ObjNum 1383 snap 428 seq_num 1383) dirty exists 2024-04-26T00:42:21.286 INFO:tasks.rados.rados.0.smithi045.stdout:3962: done (1 left) 2024-04-26T00:42:21.286 INFO:tasks.rados.rados.0.smithi045.stdout:3965: delete oid 14 current snap is 439 2024-04-26T00:42:21.299 INFO:tasks.rados.rados.0.smithi045.stdout:3964: finishing copy_from to smithi04526579-6 2024-04-26T00:42:21.299 INFO:tasks.rados.rados.0.smithi045.stdout:3964: got expected ENOENT (src dne) 2024-04-26T00:42:21.299 INFO:tasks.rados.rados.0.smithi045.stdout:3965: done (1 left) 2024-04-26T00:42:21.299 INFO:tasks.rados.rados.0.smithi045.stdout:3966: snap_remove snap 416 2024-04-26T00:42:22.308 INFO:tasks.rados.rados.0.smithi045.stdout:3966: done (1 left) 2024-04-26T00:42:22.308 INFO:tasks.rados.rados.0.smithi045.stdout:3967: snap_create 2024-04-26T00:42:22.308 INFO:tasks.rados.rados.0.smithi045.stdout:3964: finishing copy_from racing read to smithi04526579-6 2024-04-26T00:42:22.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.832+0000 7f7661ffb640 10 monclient: tick 2024-04-26T00:42:22.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.832+0000 7f7661ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:42:22.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.832+0000 7f7661ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:42:22.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.832+0000 7f7661ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:42:22.854 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.852+0000 7f7672304640 1 --2- 172.21.15.37:0/2910754178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c1347b0 0x7f766c12bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:22.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.852+0000 7f7672304640 10 monclient: get_auth_request con 0x7f766c1347b0 auth_method 0 2024-04-26T00:42:22.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.852+0000 7f7672304640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:22.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.852+0000 7f7672304640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:22.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.852+0000 7f7672304640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:42:22.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.852+0000 7f7672304640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:22.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.852+0000 7f7672304640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:22.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.852+0000 7f7672304640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:22.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.852+0000 7f7672304640 10 monclient(hunting): handle_auth_done global_id 274099 payload 824 2024-04-26T00:42:22.855 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.852+0000 7f7672304640 1 -- 172.21.15.37:0/2910754178 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f766c135440 con 0x7f766c1347b0 2024-04-26T00:42:22.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.856+0000 7f7672304640 1 -- 172.21.15.37:0/2910754178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c1347b0 msgr2=0x7f766c12bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:22.858 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.856+0000 7f7672304640 1 --2- 172.21.15.37:0/2910754178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c1347b0 0x7f766c12bcd0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7668007e50 tx=0x7f7668012d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:22.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.856+0000 7f7662ffd640 1 -- 172.21.15.37:0/2910754178 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7668014400 con 0x7f766c1347b0 2024-04-26T00:42:22.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.856+0000 7f7662ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:22.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.856+0000 7f7662ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:42:22.859 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.856+0000 7f7662ffd640 10 monclient: dump: 2024-04-26T00:42:22.859 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:22.859 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:22.859 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:22.859 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:22.859 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:22.859 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:22.860 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:22.860 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:22.860 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:22.860 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:22.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.856+0000 7f7662ffd640 1 -- 172.21.15.37:0/2910754178 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7668014400 con 0x7f766c1347b0 2024-04-26T00:42:22.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.856+0000 7f7662ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:22.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.856+0000 7f7662ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:42:22.860 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.856+0000 7f7662ffd640 1 -- 172.21.15.37:0/2910754178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c1347b0 msgr2=0x7f766c12bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:22.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.856+0000 7f7662ffd640 1 --2- 172.21.15.37:0/2910754178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c1347b0 0x7f766c12bcd0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7668007e50 tx=0x7f7668012d50 comp rx=0 tx=0).stop 2024-04-26T00:42:22.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.856+0000 7f7662ffd640 1 -- 172.21.15.37:0/2910754178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c1347b0 msgr2=0x7f766c12bcd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:42:22.861 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.856+0000 7f7662ffd640 1 --2- 172.21.15.37:0/2910754178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c1347b0 0x7f766c12bcd0 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7668007e50 tx=0x7f7668012d50 comp rx=0 tx=0).stop 2024-04-26T00:42:22.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.864+0000 7f7674568640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:42:22.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.864+0000 7f7674568640 1 --2- 172.21.15.37:0/2910754178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c12c210 0x7f766c12c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:22.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.864+0000 7f7674568640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:22.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.864+0000 7f7674568640 1 -- 172.21.15.37:0/2910754178 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f766c04f510 con 0x7f766c12c210 2024-04-26T00:42:22.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.864+0000 7f7672304640 1 --2- 172.21.15.37:0/2910754178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c12c210 0x7f766c12c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:22.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.864+0000 7f7672304640 10 monclient: get_auth_request con 0x7f766c12c210 auth_method 0 2024-04-26T00:42:22.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.864+0000 7f7672304640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:22.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.864+0000 7f7672304640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:22.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.864+0000 7f7672304640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:22.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.864+0000 7f7672304640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:22.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.864+0000 7f7672304640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:22.869 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.864+0000 7f7672304640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:22.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.864+0000 7f7672304640 10 monclient(hunting): handle_auth_done global_id 274102 payload 824 2024-04-26T00:42:22.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.864+0000 7f7672304640 1 -- 172.21.15.37:0/2910754178 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f766c0a0f60 con 0x7f766c12c210 2024-04-26T00:42:22.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.868+0000 7f7672304640 1 -- 172.21.15.37:0/2910754178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c12c210 msgr2=0x7f766c12c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:22.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.868+0000 7f7672304640 1 --2- 172.21.15.37:0/2910754178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c12c210 0x7f766c12c5f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7668007e50 tx=0x7f7668004950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:22.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.868+0000 7f7662ffd640 1 -- 172.21.15.37:0/2910754178 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f766801caf0 con 0x7f766c12c210 2024-04-26T00:42:22.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.868+0000 7f7662ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:22.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.868+0000 7f7662ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:42:22.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.868+0000 7f7662ffd640 10 monclient: dump: 2024-04-26T00:42:22.873 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:22.873 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:22.873 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:22.873 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:22.873 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:22.873 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:22.873 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:22.874 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:22.874 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:22.874 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:22.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.868+0000 7f7662ffd640 1 -- 172.21.15.37:0/2910754178 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4547 (secure 0 0 0) 0x7f7668013390 con 0x7f766c12c210 2024-04-26T00:42:22.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.868+0000 7f7662ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:42:22.874 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.868+0000 7f7662ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:42:22.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.868+0000 7f7662ffd640 1 -- 172.21.15.37:0/2910754178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c12c210 msgr2=0x7f766c12c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:22.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.868+0000 7f7662ffd640 1 --2- 172.21.15.37:0/2910754178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c12c210 0x7f766c12c5f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7668007e50 tx=0x7f7668004950 comp rx=0 tx=0).stop 2024-04-26T00:42:22.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.868+0000 7f7662ffd640 1 -- 172.21.15.37:0/2910754178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c12c210 msgr2=0x7f766c12c5f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:42:22.875 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.868+0000 7f7662ffd640 1 --2- 172.21.15.37:0/2910754178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f766c12c210 0x7f766c12c5f0 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7668007e50 tx=0x7f7668004950 comp rx=0 tx=0).stop 2024-04-26T00:42:22.875 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:42:22.875 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:42:22.875 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:42:22.875 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:42:22.875 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 160, 2024-04-26T00:42:22.876 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 10623, 2024-04-26T00:42:22.876 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:42:22.876 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:42:22.876 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:42:22.876 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:42:22.876 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:42:22.876 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 4, 2024-04-26T00:42:22.876 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:42:22.876 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:42:22.876 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:42:22.876 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:42:22.876 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:42:22.876 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:42:22.876 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:42:22.876 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:42:22.876 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:42:22.877 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:42:22.877 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:42:22.877 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:42:22.877 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:42:22.877 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:42:22.877 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:42:22.877 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:42:22.877 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:42:22.877 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:42:22.877 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:42:22.877 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:42:22.877 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:42:22.877 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:42:22.877 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:42:22.877 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:42:22.877 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:42:22.878 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:42:22.878 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:42:22.878 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:42:22.878 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:22.878 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:22.878 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:42:22.878 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:42:22.878 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:42:22.878 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:42:22.878 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:42:22.878 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:42:22.878 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:42:22.878 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:42:22.879 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:42:22.879 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:42:22.879 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:42:22.879 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:42:22.879 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:42:22.879 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:42:22.879 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:42:22.879 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:42:22.879 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:42:22.879 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:42:22.879 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:42:22.879 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:42:22.879 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:42:22.879 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:42:22.879 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:42:22.879 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:42:22.880 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:42:22.880 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:42:22.880 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:42:22.880 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:42:22.880 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:42:22.880 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:42:22.880 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:22.880 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:42:22.880 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:22.880 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:42:22.880 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:42:22.880 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:42:22.880 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:42:22.880 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:22.880 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:42:22.880 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:42:22.881 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:22.881 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:22.881 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:22.881 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:42:22.881 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:42:22.881 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:22.882 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:22.882 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:22.882 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:22.882 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:42:22.882 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:42:22.882 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:42:22.882 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:42:22.882 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:42:22.883 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:22.883 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:22.883 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:42:22.883 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:42:22.883 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:42:22.883 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:42:22.883 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:22.883 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:42:22.883 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:42:22.883 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:22.883 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:22.883 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:22.883 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:42:22.883 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:42:22.884 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:22.884 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:22.884 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:22.884 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:22.884 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:42:22.884 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:42:22.884 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:42:22.884 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:42:22.884 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:42:22.884 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:22.884 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:22.884 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:42:22.884 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:42:22.884 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:42:22.884 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:42:22.885 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:22.885 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:42:22.885 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:42:22.885 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:22.885 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:22.885 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:22.885 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:42:22.885 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:42:22.885 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:22.885 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:22.885 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:22.885 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:22.885 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:42:22.885 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:42:22.886 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:42:22.886 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:42:22.886 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:42:22.886 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:22.886 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:22.886 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:22.886 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:42:22.886 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:42:22.886 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:22.886 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:42:22.886 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:42:22.886 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:42:22.886 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:22.887 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:42:22.887 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:42:22.887 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:22.887 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:42:22.887 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:42:22.887 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:42:22.887 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:22.887 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:22.887 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:22.887 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:42:22.887 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:42:22.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.872+0000 7f7674568640 1 -- 172.21.15.37:0/2910754178 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7650035ed0 msgr2=0x7f7650038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:22.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.872+0000 7f7674568640 1 --2- 172.21.15.37:0/2910754178 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7650035ed0 0x7f7650038390 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f766c133940 tx=0x7f765c009190 comp rx=0 tx=0).stop 2024-04-26T00:42:22.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.872+0000 7f7674568640 10 monclient: shutdown 2024-04-26T00:42:22.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.872+0000 7f7674568640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:42:22.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.872+0000 7f7674568640 1 -- 172.21.15.37:0/2910754178 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f766c1393f0 msgr2=0x7f766c132d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:22.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.872+0000 7f7674568640 1 --2- 172.21.15.37:0/2910754178 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f766c1393f0 0x7f766c132d30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f76640b0040 tx=0x7f7664019560 comp rx=0 tx=0).stop 2024-04-26T00:42:22.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.872+0000 7f7672b05640 1 -- 172.21.15.37:0/2910754178 reap_dead start 2024-04-26T00:42:22.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.872+0000 7f7672b05640 1 -- 172.21.15.37:0/2910754178 reap_dead start 2024-04-26T00:42:22.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.872+0000 7f7674568640 1 -- 172.21.15.37:0/2910754178 shutdown_connections 2024-04-26T00:42:22.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.872+0000 7f7674568640 1 -- 172.21.15.37:0/2910754178 >> 172.21.15.37:0/2910754178 conn(0x7f766c0a1d50 msgr2=0x7f766c12fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:42:22.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.872+0000 7f7674568640 1 -- 172.21.15.37:0/2910754178 shutdown_connections 2024-04-26T00:42:22.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:22.872+0000 7f7674568640 1 -- 172.21.15.37:0/2910754178 wait complete. 2024-04-26T00:42:22.907 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:42:22.907 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-26T00:42:22.907 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-26T00:42:23.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.036+0000 7fa19131d640 10 monclient: get_monmap_and_config 2024-04-26T00:42:23.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.036+0000 7fa19131d640 10 monclient: build_initial_monmap 2024-04-26T00:42:23.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.036+0000 7fa19131d640 10 monclient: monmap: 2024-04-26T00:42:23.041 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:42:23.041 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:23.041 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:42:23.042597+0000 2024-04-26T00:42:23.041 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:42:23.042597+0000 2024-04-26T00:42:23.041 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:42:23.041 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:23.041 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:42:23.041 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:42:23.041 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:42:23.041 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:42:23.041 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:42:23.042 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:23.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.036+0000 7fa19131d640 1 Processor -- start 2024-04-26T00:42:23.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.036+0000 7fa19131d640 1 -- start start 2024-04-26T00:42:23.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.036+0000 7fa19131d640 10 monclient: init 2024-04-26T00:42:23.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.040+0000 7fa19131d640 10 monclient: _reopen_session rank -1 2024-04-26T00:42:23.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.040+0000 7fa19131d640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-04-26T00:42:23.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.040+0000 7fa19131d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa18c136da0 addr v1:172.21.15.37:6789/0 2024-04-26T00:42:23.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.040+0000 7fa19131d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa18c137180 0x7fa18c1375c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:23.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.040+0000 7fa19131d640 10 monclient(hunting): picked mon.noname-b con 0x7fa18c137180 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:42:23.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.040+0000 7fa19131d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa18c13c2d0 addr v1:172.21.15.45:6789/0 2024-04-26T00:42:23.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.040+0000 7fa19131d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa18c09b4f0 con 0x7fa18c136da0 2024-04-26T00:42:23.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.040+0000 7fa19131d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa18c09b7d0 con 0x7fa18c13c2d0 2024-04-26T00:42:23.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.040+0000 7fa19131d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:23.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.040+0000 7fa19131d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa18c09b210 con 0x7fa18c137180 2024-04-26T00:42:23.044 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.040+0000 7fa19131d640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:23.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.040+0000 7fa19131d640 10 monclient(hunting): authenticate will time out at 2324.348504s 2024-04-26T00:42:23.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.040+0000 7fa18affd640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fa18c136da0 0x7fa18c09bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:44904/0 (socket says 172.21.15.37:44904) 2024-04-26T00:42:23.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.040+0000 7fa18affd640 1 -- 172.21.15.37:0/355457564 learned_addr learned my addr 172.21.15.37:0/355457564 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:42:23.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.040+0000 7fa18a7fc640 1 --2- 172.21.15.37:0/355457564 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa18c137180 0x7fa18c1375c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:23.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.040+0000 7fa18a7fc640 10 monclient(hunting): get_auth_request con 0x7fa18c137180 auth_method 0 2024-04-26T00:42:23.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.040+0000 7fa18a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:23.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.040+0000 7fa18a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:23.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.040+0000 7fa18a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:23.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.040+0000 7fa18a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:23.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.040+0000 7fa18a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:23.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.040+0000 7fa18a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:23.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.040+0000 7fa18a7fc640 10 monclient(hunting): handle_auth_done global_id 254330 payload 274 2024-04-26T00:42:23.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.040+0000 7fa18a7fc640 1 -- 172.21.15.37:0/355457564 >> v1:172.21.15.45:6789/0 conn(0x7fa18c13c2d0 legacy=0x7fa18c13e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:42:23.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.040+0000 7fa18a7fc640 1 -- 172.21.15.37:0/355457564 >> v1:172.21.15.37:6789/0 conn(0x7fa18c136da0 legacy=0x7fa18c09bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:42:23.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.040+0000 7fa18a7fc640 10 monclient: _finish_hunting 0 2024-04-26T00:42:23.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.040+0000 7fa18a7fc640 1 monclient: found mon.noname-b 2024-04-26T00:42:23.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.040+0000 7fa18a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:42:23.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.040+0000 7fa18a7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:42:23.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.040+0000 7fa18a7fc640 1 -- 172.21.15.37:0/355457564 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa18c137b00 con 0x7fa18c137180 2024-04-26T00:42:23.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa18a7fc640 1 -- 172.21.15.37:0/355457564 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa18c137180 msgr2=0x7fa18c1375c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:23.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa18a7fc640 1 --2- 172.21.15.37:0/355457564 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa18c137180 0x7fa18c1375c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa1800097f0 tx=0x7fa18001c260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ccbfaf66462d47fe server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:23.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa189ffb640 1 -- 172.21.15.37:0/355457564 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1800b0070 con 0x7fa18c137180 2024-04-26T00:42:23.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa189ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:23.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa189ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:42:23.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa189ffb640 10 monclient: dump: 2024-04-26T00:42:23.049 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:23.049 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:23.049 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:23.049 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:23.049 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:23.049 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:23.049 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:23.049 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:23.050 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:23.050 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:23.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa189ffb640 10 monclient: _finish_auth 0 2024-04-26T00:42:23.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa189ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:42:23.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa189ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:42:23.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa19131d640 5 monclient: authenticate success, global_id 254330 2024-04-26T00:42:23.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa189ffb640 1 -- 172.21.15.37:0/355457564 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1800ab600 con 0x7fa18c137180 2024-04-26T00:42:23.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa189ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:42:23.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa19131d640 10 monclient: get_monmap_and_config success 2024-04-26T00:42:23.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa19131d640 4 set_mon_vals no callback set 2024-04-26T00:42:23.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa189ffb640 1 -- 172.21.15.37:0/355457564 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1800ab9d0 con 0x7fa18c137180 2024-04-26T00:42:23.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa189ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:23.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa189ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:42:23.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa189ffb640 10 monclient: dump: 2024-04-26T00:42:23.051 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:23.051 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:23.051 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:23.051 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:23.051 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:23.051 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:23.051 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:23.051 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:23.051 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:23.051 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:23.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa19131d640 10 monclient: shutdown 2024-04-26T00:42:23.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa19131d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:42:23.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa19131d640 1 -- 172.21.15.37:0/355457564 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa18c137180 msgr2=0x7fa18c1375c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:23.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa19131d640 1 --2- 172.21.15.37:0/355457564 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa18c137180 0x7fa18c1375c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa1800097f0 tx=0x7fa18001c260 comp rx=0 tx=0).stop 2024-04-26T00:42:23.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa19131d640 1 -- 172.21.15.37:0/355457564 shutdown_connections 2024-04-26T00:42:23.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa19131d640 1 --2- 172.21.15.37:0/355457564 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa18c137180 0x7fa18c1375c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:23.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa19131d640 1 -- 172.21.15.37:0/355457564 >> 172.21.15.37:0/355457564 conn(0x7fa18c0a5f00 msgr2=0x7fa18c0a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:42:23.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa19131d640 1 -- 172.21.15.37:0/355457564 shutdown_connections 2024-04-26T00:42:23.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa19131d640 1 -- 172.21.15.37:0/355457564 wait complete. 2024-04-26T00:42:23.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa19131d640 10 monclient: build_initial_monmap 2024-04-26T00:42:23.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa19131d640 10 monclient: monmap: 2024-04-26T00:42:23.052 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:42:23.052 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:42:23.052 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:42:23.052 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:42:23.052 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:42:23.053 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:42:23.053 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:42:23.053 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:42:23.053 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:42:23.053 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:23.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa19131d640 1 Processor -- start 2024-04-26T00:42:23.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa19131d640 1 -- start start 2024-04-26T00:42:23.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa19131d640 10 monclient: init 2024-04-26T00:42:23.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa19131d640 10 monclient: _reopen_session rank -1 2024-04-26T00:42:23.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa19131d640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:42:23.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa19131d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa18c136da0 0x7fa18c135c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:23.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa19131d640 10 monclient(hunting): picked mon.noname-c con 0x7fa18c136da0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:42:23.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa19131d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa18c137180 0x7fa18c137db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:23.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa19131d640 10 monclient(hunting): picked mon.noname-a con 0x7fa18c137180 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:42:23.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa19131d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa18c13c2d0 0x7fa18c1382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:23.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa19131d640 10 monclient(hunting): picked mon.noname-b con 0x7fa18c13c2d0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:42:23.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa19131d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:23.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa19131d640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa18c142990 con 0x7fa18c137180 2024-04-26T00:42:23.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa19131d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:23.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa19131d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa18c142810 con 0x7fa18c13c2d0 2024-04-26T00:42:23.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa19131d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:23.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa19131d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa18c142b10 con 0x7fa18c136da0 2024-04-26T00:42:23.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa19131d640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:23.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.044+0000 7fa19131d640 10 monclient(hunting): authenticate will time out at 2324.355924s 2024-04-26T00:42:23.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa18affd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa18c136da0 0x7fa18c135c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:23.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa18affd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa18c136da0 0x7fa18c135c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51062/0 (socket says 172.21.15.37:51062) 2024-04-26T00:42:23.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa18a7fc640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa18c137180 0x7fa18c137db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:23.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa18affd640 1 -- 172.21.15.37:0/1846134769 learned_addr learned my addr 172.21.15.37:0/1846134769 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:42:23.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa18affd640 10 monclient(hunting): get_auth_request con 0x7fa18c136da0 auth_method 0 2024-04-26T00:42:23.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa18affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:23.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa18affd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:23.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa18affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:23.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa18a7fc640 10 monclient(hunting): get_auth_request con 0x7fa18c137180 auth_method 0 2024-04-26T00:42:23.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa18a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:23.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa18a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:23.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa18a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:23.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa18affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:23.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa18affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:23.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa18affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:23.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa18a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:23.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa18a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:23.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa18a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:23.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa18b7fe640 1 --2- 172.21.15.37:0/1846134769 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa18c13c2d0 0x7fa18c1382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:23.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa18affd640 10 monclient(hunting): handle_auth_done global_id 254333 payload 824 2024-04-26T00:42:23.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa18affd640 1 -- 172.21.15.37:0/1846134769 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa18c13c2d0 msgr2=0x7fa18c1382f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:23.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa18affd640 1 --2- 172.21.15.37:0/1846134769 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa18c13c2d0 0x7fa18c1382f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:23.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa18affd640 1 -- 172.21.15.37:0/1846134769 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa18c137180 msgr2=0x7fa18c137db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:23.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa18affd640 1 --2- 172.21.15.37:0/1846134769 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa18c137180 0x7fa18c137db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:23.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa18affd640 10 monclient: _finish_hunting 0 2024-04-26T00:42:23.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa18affd640 1 monclient: found mon.noname-c 2024-04-26T00:42:23.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa18affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:42:23.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa18affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:42:23.060 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa18affd640 1 -- 172.21.15.37:0/1846134769 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa18c13b910 con 0x7fa18c136da0 2024-04-26T00:42:23.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa18b7fe640 10 monclient: get_auth_request con 0x7fa18c13c2d0 auth_method 0 2024-04-26T00:42:23.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa18b7fe640 1 --2- 172.21.15.37:0/1846134769 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa18c13c2d0 0x7fa18c1382f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:42:23.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa18affd640 1 -- 172.21.15.37:0/1846134769 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa18c136da0 msgr2=0x7fa18c135c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:23.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa18affd640 1 --2- 172.21.15.37:0/1846134769 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa18c136da0 0x7fa18c135c40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa17c00cc10 tx=0x7fa17c00a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:23.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa1737fe640 1 -- 172.21.15.37:0/1846134769 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa17c012400 con 0x7fa18c136da0 2024-04-26T00:42:23.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa1737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:23.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa1737fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:42:23.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa1737fe640 10 monclient: dump: 2024-04-26T00:42:23.061 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:23.061 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:23.061 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:23.061 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:23.061 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:23.062 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:23.062 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:23.062 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:23.062 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:23.062 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:23.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa1737fe640 10 monclient: _finish_auth 0 2024-04-26T00:42:23.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa1737fe640 10 monclient: _check_auth_tickets 2024-04-26T00:42:23.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa1737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:42:23.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa1737fe640 1 -- 172.21.15.37:0/1846134769 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa17c0125e0 con 0x7fa18c136da0 2024-04-26T00:42:23.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa1737fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:42:23.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa1737fe640 1 -- 172.21.15.37:0/1846134769 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa17c0129d0 con 0x7fa18c136da0 2024-04-26T00:42:23.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa1737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:23.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa1737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:42:23.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa1737fe640 10 monclient: dump: 2024-04-26T00:42:23.063 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:23.063 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:23.063 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:23.063 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:23.063 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:23.063 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:23.063 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:23.063 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:23.063 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:23.063 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:23.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa1897fa640 4 set_mon_vals no callback set 2024-04-26T00:42:23.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa19131d640 5 monclient: authenticate success, global_id 254333 2024-04-26T00:42:23.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa19131d640 10 monclient: _renew_subs 2024-04-26T00:42:23.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa19131d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:23.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa19131d640 1 -- 172.21.15.37:0/1846134769 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa18c13b630 con 0x7fa18c136da0 2024-04-26T00:42:23.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa19131d640 10 monclient: _renew_subs 2024-04-26T00:42:23.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa19131d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:23.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.048+0000 7fa19131d640 1 -- 172.21.15.37:0/1846134769 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa18c136180 con 0x7fa18c136da0 2024-04-26T00:42:23.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.052+0000 7fa19131d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:23.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.052+0000 7fa19131d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:23.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.052+0000 7fa1737fe640 1 -- 172.21.15.37:0/1846134769 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa17c012bf0 con 0x7fa18c136da0 2024-04-26T00:42:23.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.052+0000 7fa19131d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:23.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.052+0000 7fa19131d640 1 -- 172.21.15.37:0/1846134769 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa18c066f40 con 0x7fa18c136da0 2024-04-26T00:42:23.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.052+0000 7fa1737fe640 1 --2- 172.21.15.37:0/1846134769 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa178035e80 0x7fa178038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:23.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.052+0000 7fa1737fe640 1 -- 172.21.15.37:0/1846134769 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(894..894 src has 822..894) ==== 3959+0+0 (secure 0 0 0) 0x7fa17c049230 con 0x7fa18c136da0 2024-04-26T00:42:23.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.052+0000 7fa18a7fc640 1 --2- 172.21.15.37:0/1846134769 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa178035e80 0x7fa178038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:23.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.052+0000 7fa18a7fc640 10 monclient: get_auth_request con 0x7fa178035e80 auth_method 0 2024-04-26T00:42:23.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.056+0000 7fa18a7fc640 1 --2- 172.21.15.37:0/1846134769 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa178035e80 0x7fa178038340 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fa18c0a58a0 tx=0x7fa1800a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:23.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.056+0000 7fa1737fe640 1 -- 172.21.15.37:0/1846134769 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa17c01b4b0 con 0x7fa18c136da0 2024-04-26T00:42:23.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.056+0000 7fa1737fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:23.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.056+0000 7fa1737fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:42:23.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.280+0000 7fa19131d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-26T00:42:23.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.280+0000 7fa19131d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-26T00:42:23.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.280+0000 7fa19131d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:23.285 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.280+0000 7fa19131d640 1 -- 172.21.15.37:0/1846134769 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fa18c0a84a0 con 0x7fa18c136da0 2024-04-26T00:42:23.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.284+0000 7fa1737fe640 1 -- 172.21.15.37:0/1846134769 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fa17c01ae10 con 0x7fa18c136da0 2024-04-26T00:42:23.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.284+0000 7fa1737fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-26T00:42:23.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.284+0000 7fa1737fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:42:23.288 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.284+0000 7fa19131d640 1 -- 172.21.15.37:0/1846134769 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa178035e80 msgr2=0x7fa178038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:23.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.284+0000 7fa19131d640 1 --2- 172.21.15.37:0/1846134769 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa178035e80 0x7fa178038340 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fa18c0a58a0 tx=0x7fa1800a6000 comp rx=0 tx=0).stop 2024-04-26T00:42:23.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.284+0000 7fa19131d640 10 monclient: shutdown 2024-04-26T00:42:23.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.284+0000 7fa19131d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:42:23.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.284+0000 7fa19131d640 1 -- 172.21.15.37:0/1846134769 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa18c136da0 msgr2=0x7fa18c135c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:23.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.284+0000 7fa19131d640 1 --2- 172.21.15.37:0/1846134769 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa18c136da0 0x7fa18c135c40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa17c00cc10 tx=0x7fa17c00a430 comp rx=0 tx=0).stop 2024-04-26T00:42:23.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.284+0000 7fa19131d640 1 -- 172.21.15.37:0/1846134769 shutdown_connections 2024-04-26T00:42:23.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.284+0000 7fa19131d640 1 --2- 172.21.15.37:0/1846134769 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa178035e80 0x7fa178038340 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:23.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.284+0000 7fa19131d640 1 --2- 172.21.15.37:0/1846134769 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa18c13c2d0 0x7fa18c1382f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:23.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.284+0000 7fa19131d640 1 --2- 172.21.15.37:0/1846134769 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa18c137180 0x7fa18c137db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:23.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.284+0000 7fa19131d640 1 --2- 172.21.15.37:0/1846134769 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa18c136da0 0x7fa18c135c40 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:23.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.284+0000 7fa19131d640 1 -- 172.21.15.37:0/1846134769 >> 172.21.15.37:0/1846134769 conn(0x7fa18c0a5f00 msgr2=0x7fa18c0a7200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:42:23.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.284+0000 7fa19131d640 1 -- 172.21.15.37:0/1846134769 shutdown_connections 2024-04-26T00:42:23.289 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:23.284+0000 7fa19131d640 1 -- 172.21.15.37:0/1846134769 wait complete. 2024-04-26T00:42:23.304 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.300+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.304 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.300+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=27,config=2,health=12,kv=9,logm=50} crc {auth=4270874671,config=692619927,health=113860567,kv=2885089190,logm=286629447}) 2024-04-26T00:42:23.304 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.300+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=11,kv=7,logm=53} crc {auth=4238810625,config=692619927,health=1603646365,kv=4280154574,logm=3209663673}) 2024-04-26T00:42:23.305 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.300+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.305 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.300+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=27,config=2,health=12,kv=9,logm=50} crc {auth=4270874671,config=692619927,health=113860567,kv=2885089190,logm=286629447}) 2024-04-26T00:42:23.305 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.300+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=29,config=2,health=12,kv=10,logm=47} crc {auth=1426865638,config=692619927,health=2765935843,kv=1034996044,logm=1492797906}) 2024-04-26T00:42:23.306 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-26T00:42:23.309 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.304+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.309 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.304+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1121558363}) 2024-04-26T00:42:23.309 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.304+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1982261956}) 2024-04-26T00:42:23.309 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.304+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.309 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.304+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1121558363}) 2024-04-26T00:42:23.309 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.304+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1063253347}) 2024-04-26T00:42:23.316 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.312+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.316 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.312+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2398614525}) 2024-04-26T00:42:23.316 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.312+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1431025709}) 2024-04-26T00:42:23.316 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.312+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.316 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.312+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2398614525}) 2024-04-26T00:42:23.316 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.312+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1420890726}) 2024-04-26T00:42:23.318 INFO:tasks.rados.rados.0.smithi045.stdout:3964: done (1 left) 2024-04-26T00:42:23.318 INFO:tasks.rados.rados.0.smithi045.stdout:3967: done (0 left) 2024-04-26T00:42:23.318 INFO:tasks.rados.rados.0.smithi045.stdout:3968: rollback oid 15 current snap is 440 2024-04-26T00:42:23.318 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 15 to 437 2024-04-26T00:42:23.318 INFO:tasks.rados.rados.0.smithi045.stdout:3969: delete oid 35 current snap is 440 2024-04-26T00:42:23.322 INFO:tasks.rados.rados.0.smithi045.stdout:3969: done (1 left) 2024-04-26T00:42:23.322 INFO:tasks.rados.rados.0.smithi045.stdout:3970: write oid 46 current snap is 440 2024-04-26T00:42:23.322 INFO:tasks.rados.rados.0.smithi045.stdout:3970: seq_num 1416 ranges {598712=594242,1677679=532825,2985546=430582} 2024-04-26T00:42:23.336 INFO:tasks.rados.rados.0.smithi045.stdout:3970: writing smithi04526579-46 from 598712 to 1192954 tid 1 2024-04-26T00:42:23.347 INFO:tasks.rados.rados.0.smithi045.stdout:3970: writing smithi04526579-46 from 1677679 to 2210504 tid 2 2024-04-26T00:42:23.350 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.348+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.350 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.348+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4118104476}) 2024-04-26T00:42:23.350 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.348+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1164606840}) 2024-04-26T00:42:23.350 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.348+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.350 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.348+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4118104476}) 2024-04-26T00:42:23.350 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.348+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=70168309}) 2024-04-26T00:42:23.354 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.352+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.354 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.352+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1071323506}) 2024-04-26T00:42:23.354 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.352+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1368521418}) 2024-04-26T00:42:23.354 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.352+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.354 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.352+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1071323506}) 2024-04-26T00:42:23.354 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.352+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2253010723}) 2024-04-26T00:42:23.356 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.352+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.356 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.352+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=945291857}) 2024-04-26T00:42:23.356 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.352+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=836846244}) 2024-04-26T00:42:23.356 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.352+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.357 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.352+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=945291857}) 2024-04-26T00:42:23.357 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.352+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2157108502}) 2024-04-26T00:42:23.358 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.356+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.358 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.356+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3795776621}) 2024-04-26T00:42:23.358 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.356+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1274986611}) 2024-04-26T00:42:23.358 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.356+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.358 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.356+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3795776621}) 2024-04-26T00:42:23.359 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.356+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3193984474}) 2024-04-26T00:42:23.360 INFO:tasks.rados.rados.0.smithi045.stdout:3970: writing smithi04526579-46 from 2985546 to 3416128 tid 3 2024-04-26T00:42:23.360 INFO:tasks.rados.rados.0.smithi045.stdout:3971: read oid 21 snap 437 2024-04-26T00:42:23.360 INFO:tasks.rados.rados.0.smithi045.stdout:3971: expect (ObjNum 1360 snap 420 seq_num 1360) 2024-04-26T00:42:23.360 INFO:tasks.rados.rados.0.smithi045.stdout:3972: read oid 28 snap 429 2024-04-26T00:42:23.360 INFO:tasks.rados.rados.0.smithi045.stdout:3972: expect deleted 2024-04-26T00:42:23.360 INFO:tasks.rados.rados.0.smithi045.stdout:3973: delete oid 40 current snap is 440 2024-04-26T00:42:23.360 INFO:tasks.rados.rados.0.smithi045.stdout:3968: finishing rollback tid 1 to smithi04526579-15 2024-04-26T00:42:23.360 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 15 v 2518 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-04-26T00:42:23.360 INFO:tasks.rados.rados.0.smithi045.stdout:3970: finishing write tid 1 to smithi04526579-46 2024-04-26T00:42:23.360 INFO:tasks.rados.rados.0.smithi045.stdout:3970: finishing write tid 2 to smithi04526579-46 2024-04-26T00:42:23.361 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.356+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.361 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.356+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4118405304}) 2024-04-26T00:42:23.361 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.356+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2841432954}) 2024-04-26T00:42:23.362 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.356+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.362 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.356+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4118405304}) 2024-04-26T00:42:23.362 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.356+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3097088479}) 2024-04-26T00:42:23.363 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.360+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.363 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.360+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3448059306}) 2024-04-26T00:42:23.363 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.360+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2571436407}) 2024-04-26T00:42:23.363 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.360+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.363 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.360+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3448059306}) 2024-04-26T00:42:23.364 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.360+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1261449254}) 2024-04-26T00:42:23.367 INFO:tasks.rados.rados.0.smithi045.stdout:3970: finishing write tid 3 to smithi04526579-46 2024-04-26T00:42:23.367 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.364+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.367 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.364+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3960209547}) 2024-04-26T00:42:23.367 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.364+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=798299569}) 2024-04-26T00:42:23.368 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.364+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.368 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.364+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3960209547}) 2024-04-26T00:42:23.368 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.364+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3730344935}) 2024-04-26T00:42:23.368 INFO:tasks.rados.rados.0.smithi045.stdout:3970: finishing write tid 4 to smithi04526579-46 2024-04-26T00:42:23.369 INFO:tasks.rados.rados.0.smithi045.stdout:3970: finishing write tid 5 to smithi04526579-46 2024-04-26T00:42:23.369 INFO:tasks.rados.rados.0.smithi045.stdout:3970: oid 46 updating version 0 to 2833 2024-04-26T00:42:23.369 INFO:tasks.rados.rados.0.smithi045.stdout:3970: oid 46 version 2833 is already newer than 2832 2024-04-26T00:42:23.369 INFO:tasks.rados.rados.0.smithi045.stdout:3970: oid 46 updating version 2833 to 2835 2024-04-26T00:42:23.369 INFO:tasks.rados.rados.0.smithi045.stdout:3970: oid 46 version 2835 is already newer than 2834 2024-04-26T00:42:23.369 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 46 v 2835 (ObjNum 1416 snap 440 seq_num 1416) dirty exists 2024-04-26T00:42:23.369 INFO:tasks.rados.rados.0.smithi045.stdout:3970: left oid 46 (ObjNum 1416 snap 440 seq_num 1416) 2024-04-26T00:42:23.370 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.364+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.370 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.364+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1446140952}) 2024-04-26T00:42:23.370 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.364+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2119036727}) 2024-04-26T00:42:23.370 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.364+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.370 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.364+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1446140952}) 2024-04-26T00:42:23.370 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.364+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2766419168}) 2024-04-26T00:42:23.371 INFO:tasks.rados.rados.0.smithi045.stdout:3968: done (4 left) 2024-04-26T00:42:23.371 INFO:tasks.rados.rados.0.smithi045.stdout:3970: done (3 left) 2024-04-26T00:42:23.371 INFO:tasks.rados.rados.0.smithi045.stdout:3972: done (2 left) 2024-04-26T00:42:23.371 INFO:tasks.rados.rados.0.smithi045.stdout:3973: done (1 left) 2024-04-26T00:42:23.371 INFO:tasks.rados.rados.0.smithi045.stdout:3974: read oid 24 snap -1 2024-04-26T00:42:23.371 INFO:tasks.rados.rados.0.smithi045.stdout:3974: expect deleted 2024-04-26T00:42:23.371 INFO:tasks.rados.rados.0.smithi045.stdout:3975: rollback oid 9 current snap is 440 2024-04-26T00:42:23.371 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 9 to 430 2024-04-26T00:42:23.372 INFO:tasks.rados.rados.0.smithi045.stdout:3976: snap_remove snap 405 2024-04-26T00:42:23.373 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.368+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.373 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.368+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1922252134}) 2024-04-26T00:42:23.373 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.368+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3611891434}) 2024-04-26T00:42:23.373 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.368+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.373 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.368+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1922252134}) 2024-04-26T00:42:23.373 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.368+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4011007661}) 2024-04-26T00:42:23.376 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.372+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.376 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.372+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1464156773}) 2024-04-26T00:42:23.376 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.372+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1195587973}) 2024-04-26T00:42:23.376 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.372+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.376 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.372+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1464156773}) 2024-04-26T00:42:23.376 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.372+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=938779521}) 2024-04-26T00:42:23.382 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.380+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.382 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.380+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=149567959}) 2024-04-26T00:42:23.383 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.380+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3749811350}) 2024-04-26T00:42:23.383 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.380+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.383 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.380+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=149567959}) 2024-04-26T00:42:23.383 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.380+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=984560836}) 2024-04-26T00:42:23.386 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.380+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.386 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.380+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4167982422}) 2024-04-26T00:42:23.386 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.380+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4137621983}) 2024-04-26T00:42:23.386 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.380+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.386 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.380+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4167982422}) 2024-04-26T00:42:23.386 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.380+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3940004856}) 2024-04-26T00:42:23.389 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.384+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.389 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.384+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2647362847}) 2024-04-26T00:42:23.389 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.384+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4124631081}) 2024-04-26T00:42:23.389 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.384+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.389 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.384+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2647362847}) 2024-04-26T00:42:23.389 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.384+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1649744389}) 2024-04-26T00:42:23.392 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.388+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.392 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.388+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3374418846}) 2024-04-26T00:42:23.392 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.388+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=473141991}) 2024-04-26T00:42:23.392 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.388+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.392 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.388+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3374418846}) 2024-04-26T00:42:23.392 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.388+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3529878431}) 2024-04-26T00:42:23.395 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.392+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.395 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.392+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4081331826}) 2024-04-26T00:42:23.395 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.392+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3798986551}) 2024-04-26T00:42:23.395 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.392+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.395 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.392+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4081331826}) 2024-04-26T00:42:23.395 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.392+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1574515414}) 2024-04-26T00:42:23.398 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.392+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.398 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.392+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2456174467}) 2024-04-26T00:42:23.398 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.392+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1064335701}) 2024-04-26T00:42:23.398 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.392+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.398 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.392+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2456174467}) 2024-04-26T00:42:23.398 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.392+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=37065265}) 2024-04-26T00:42:23.400 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.396+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.401 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.396+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3775586897}) 2024-04-26T00:42:23.401 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.396+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3157709655}) 2024-04-26T00:42:23.401 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.396+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.401 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.396+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3775586897}) 2024-04-26T00:42:23.401 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.396+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2313699923}) 2024-04-26T00:42:23.403 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.400+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.403 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.400+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3776030462}) 2024-04-26T00:42:23.403 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.400+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3639939921}) 2024-04-26T00:42:23.403 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.400+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.404 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.400+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3776030462}) 2024-04-26T00:42:23.404 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.400+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4265209333}) 2024-04-26T00:42:23.406 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.404+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.406 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.404+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=413983320}) 2024-04-26T00:42:23.406 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.404+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2019777098}) 2024-04-26T00:42:23.406 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.404+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.406 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.404+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=413983320}) 2024-04-26T00:42:23.406 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.404+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1713207301}) 2024-04-26T00:42:23.409 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.404+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.409 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.404+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=619790283}) 2024-04-26T00:42:23.409 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.404+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1974853606}) 2024-04-26T00:42:23.409 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.404+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.409 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.404+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=619790283}) 2024-04-26T00:42:23.409 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.404+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1790564647}) 2024-04-26T00:42:23.411 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.408+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.411 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.408+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3287715979}) 2024-04-26T00:42:23.411 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.408+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=585716846}) 2024-04-26T00:42:23.411 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.408+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.412 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.408+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3287715979}) 2024-04-26T00:42:23.412 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.408+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2247393046}) 2024-04-26T00:42:23.414 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.408+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.414 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.408+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2658761648}) 2024-04-26T00:42:23.414 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.408+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=257921076}) 2024-04-26T00:42:23.414 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.408+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.414 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.408+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2658761648}) 2024-04-26T00:42:23.414 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.408+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3605860587}) 2024-04-26T00:42:23.547 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.544+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.547 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.544+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3152439803}) 2024-04-26T00:42:23.547 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.544+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1338676487}) 2024-04-26T00:42:23.547 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.544+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.547 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.544+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3152439803}) 2024-04-26T00:42:23.547 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.544+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4194228415}) 2024-04-26T00:42:23.549 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.544+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.549 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.544+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3682029962}) 2024-04-26T00:42:23.549 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.544+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2541775719}) 2024-04-26T00:42:23.549 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.544+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.549 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.544+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3682029962}) 2024-04-26T00:42:23.549 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.544+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1969441272}) 2024-04-26T00:42:23.551 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.548+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.551 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.548+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3519734999}) 2024-04-26T00:42:23.551 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.548+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3091375299}) 2024-04-26T00:42:23.552 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.548+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.552 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.548+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3519734999}) 2024-04-26T00:42:23.552 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.548+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3553169321}) 2024-04-26T00:42:23.554 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.548+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.554 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.548+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4092906361}) 2024-04-26T00:42:23.554 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.548+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3728426086}) 2024-04-26T00:42:23.554 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.548+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.554 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.548+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4092906361}) 2024-04-26T00:42:23.554 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.548+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2227919654}) 2024-04-26T00:42:23.556 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.552+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.556 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.552+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=149816077}) 2024-04-26T00:42:23.556 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.552+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2583107660}) 2024-04-26T00:42:23.556 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.552+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.556 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.552+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=149816077}) 2024-04-26T00:42:23.556 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.552+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2555725087}) 2024-04-26T00:42:23.558 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.556+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.558 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.556+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3164713771}) 2024-04-26T00:42:23.558 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.556+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2559472027}) 2024-04-26T00:42:23.558 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.556+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.558 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.556+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3164713771}) 2024-04-26T00:42:23.558 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.556+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4086071688}) 2024-04-26T00:42:23.560 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.556+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.560 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.556+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2312620857}) 2024-04-26T00:42:23.561 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.556+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1959656529}) 2024-04-26T00:42:23.561 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.556+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.561 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.556+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2312620857}) 2024-04-26T00:42:23.561 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.556+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3239882641}) 2024-04-26T00:42:23.563 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.560+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.563 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.560+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=204489332}) 2024-04-26T00:42:23.563 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.560+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2142748453}) 2024-04-26T00:42:23.563 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.560+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.563 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.560+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=204489332}) 2024-04-26T00:42:23.563 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.560+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3287864279}) 2024-04-26T00:42:23.565 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.560+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.565 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.560+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2852176188}) 2024-04-26T00:42:23.565 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.560+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2121218043}) 2024-04-26T00:42:23.565 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.560+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.565 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.560+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2852176188}) 2024-04-26T00:42:23.566 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.560+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1910885479}) 2024-04-26T00:42:23.568 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.564+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.568 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.564+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3638402243}) 2024-04-26T00:42:23.568 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.564+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1653697661}) 2024-04-26T00:42:23.568 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.564+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.568 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.564+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3638402243}) 2024-04-26T00:42:23.568 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.564+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3309681103}) 2024-04-26T00:42:23.570 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.568+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.570 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.568+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2873869130}) 2024-04-26T00:42:23.570 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.568+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=271123908}) 2024-04-26T00:42:23.570 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.568+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.570 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.568+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2873869130}) 2024-04-26T00:42:23.570 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.568+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=390170494}) 2024-04-26T00:42:23.573 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.568+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.573 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.568+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4093490046}) 2024-04-26T00:42:23.573 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.568+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=820283105}) 2024-04-26T00:42:23.573 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.568+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.573 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.568+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4093490046}) 2024-04-26T00:42:23.573 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.568+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3283142334}) 2024-04-26T00:42:23.575 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.572+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.575 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.572+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=232281066}) 2024-04-26T00:42:23.575 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.572+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3684496585}) 2024-04-26T00:42:23.575 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.572+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.575 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.572+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=232281066}) 2024-04-26T00:42:23.575 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.572+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4084471}) 2024-04-26T00:42:23.577 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.572+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.577 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.572+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1878823872}) 2024-04-26T00:42:23.577 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.572+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2651774371}) 2024-04-26T00:42:23.577 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.572+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.578 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.572+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1878823872}) 2024-04-26T00:42:23.578 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.572+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3749174874}) 2024-04-26T00:42:23.581 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.576+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.581 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.576+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2504617700}) 2024-04-26T00:42:23.581 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.576+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1598484474}) 2024-04-26T00:42:23.581 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.576+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.581 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.576+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2504617700}) 2024-04-26T00:42:23.581 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.576+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2114773060}) 2024-04-26T00:42:23.585 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.580+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.585 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.580+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3291772681}) 2024-04-26T00:42:23.585 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.580+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2056675969}) 2024-04-26T00:42:23.585 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.580+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.585 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.580+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3291772681}) 2024-04-26T00:42:23.585 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.580+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2276810904}) 2024-04-26T00:42:23.587 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.584+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.587 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.584+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=375057143}) 2024-04-26T00:42:23.587 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.584+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3987880126}) 2024-04-26T00:42:23.587 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.584+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.587 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.584+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=375057143}) 2024-04-26T00:42:23.587 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.584+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2904404636}) 2024-04-26T00:42:23.590 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.584+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.590 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.584+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2567964722}) 2024-04-26T00:42:23.590 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.584+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1522886935}) 2024-04-26T00:42:23.590 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.584+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.590 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.584+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2567964722}) 2024-04-26T00:42:23.590 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.584+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=10494623}) 2024-04-26T00:42:23.592 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.588+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.592 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.588+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2982999073}) 2024-04-26T00:42:23.592 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.588+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=844294010}) 2024-04-26T00:42:23.592 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.588+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.592 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.588+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2982999073}) 2024-04-26T00:42:23.592 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.588+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1395290668}) 2024-04-26T00:42:23.593 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.588+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.594 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.588+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2161117672}) 2024-04-26T00:42:23.594 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.588+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2252694385}) 2024-04-26T00:42:23.594 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.588+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.594 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.588+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2161117672}) 2024-04-26T00:42:23.594 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.588+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1907791820}) 2024-04-26T00:42:23.596 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.592+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.596 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.592+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4251532747}) 2024-04-26T00:42:23.596 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.592+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1847933984}) 2024-04-26T00:42:23.596 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.592+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.596 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.592+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4251532747}) 2024-04-26T00:42:23.596 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.592+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2889327228}) 2024-04-26T00:42:23.598 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.596+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.598 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.596+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=167393890}) 2024-04-26T00:42:23.598 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.596+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2106523894}) 2024-04-26T00:42:23.598 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.596+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.598 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.596+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=167393890}) 2024-04-26T00:42:23.599 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.596+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3109646850}) 2024-04-26T00:42:23.604 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.596+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.604 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.596+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3820199672}) 2024-04-26T00:42:23.605 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.596+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1713747140}) 2024-04-26T00:42:23.605 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.596+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.605 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.596+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3820199672}) 2024-04-26T00:42:23.605 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.596+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=826062897}) 2024-04-26T00:42:23.605 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.600+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.605 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.600+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=450467871}) 2024-04-26T00:42:23.605 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.600+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=635867612}) 2024-04-26T00:42:23.605 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.600+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.605 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.600+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=450467871}) 2024-04-26T00:42:23.605 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.600+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=853758954}) 2024-04-26T00:42:23.605 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.600+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.605 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.600+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4087407575}) 2024-04-26T00:42:23.605 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.600+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4033767230}) 2024-04-26T00:42:23.605 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.600+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.606 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.600+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4087407575}) 2024-04-26T00:42:23.606 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.600+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=744743976}) 2024-04-26T00:42:23.607 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.604+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.607 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.604+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4158433225}) 2024-04-26T00:42:23.607 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.604+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3022551207}) 2024-04-26T00:42:23.607 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.604+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.607 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.604+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4158433225}) 2024-04-26T00:42:23.607 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.604+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=255599990}) 2024-04-26T00:42:23.609 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.604+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.609 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.604+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=797545751}) 2024-04-26T00:42:23.609 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.604+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1107271235}) 2024-04-26T00:42:23.609 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.604+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.609 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.604+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=797545751}) 2024-04-26T00:42:23.609 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.604+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2057571973}) 2024-04-26T00:42:23.611 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.608+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.611 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.608+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=633059589}) 2024-04-26T00:42:23.611 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.608+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1192401150}) 2024-04-26T00:42:23.611 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.608+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.611 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.608+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=633059589}) 2024-04-26T00:42:23.611 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.608+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1421132478}) 2024-04-26T00:42:23.613 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.608+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.613 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.608+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3523902614}) 2024-04-26T00:42:23.613 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.608+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2754461688}) 2024-04-26T00:42:23.613 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.608+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.613 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.608+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3523902614}) 2024-04-26T00:42:23.613 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.608+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3809802505}) 2024-04-26T00:42:23.615 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.612+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.615 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.612+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1790740672}) 2024-04-26T00:42:23.615 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.612+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=60937575}) 2024-04-26T00:42:23.615 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.612+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:23.615 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.612+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1790740672}) 2024-04-26T00:42:23.615 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:23.612+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2226855621}) 2024-04-26T00:42:24.294 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.292+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.294 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.292+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2575061095}) 2024-04-26T00:42:24.295 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.292+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2862715316}) 2024-04-26T00:42:24.295 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.292+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.295 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.292+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2575061095}) 2024-04-26T00:42:24.295 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.292+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=514925465}) 2024-04-26T00:42:24.296 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.292+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.296 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.292+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3289416671}) 2024-04-26T00:42:24.296 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.292+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1864856158}) 2024-04-26T00:42:24.296 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.292+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.297 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.292+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3289416671}) 2024-04-26T00:42:24.297 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.292+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2432908823}) 2024-04-26T00:42:24.298 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.296+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.298 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.296+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1084084474}) 2024-04-26T00:42:24.299 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.296+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1478820419}) 2024-04-26T00:42:24.299 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.296+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.299 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.296+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1084084474}) 2024-04-26T00:42:24.299 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.296+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1812906347}) 2024-04-26T00:42:24.301 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.296+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.301 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.296+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=467450818}) 2024-04-26T00:42:24.301 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.296+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4254901990}) 2024-04-26T00:42:24.301 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.296+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.301 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.296+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=467450818}) 2024-04-26T00:42:24.301 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.296+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=330936698}) 2024-04-26T00:42:24.303 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.300+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.303 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.300+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1244967833}) 2024-04-26T00:42:24.303 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.300+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1596395751}) 2024-04-26T00:42:24.303 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.300+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.303 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.300+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1244967833}) 2024-04-26T00:42:24.303 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.300+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2507685627}) 2024-04-26T00:42:24.305 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.300+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.305 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.300+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=744903006}) 2024-04-26T00:42:24.305 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.300+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=878645362}) 2024-04-26T00:42:24.305 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.300+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.305 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.300+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=744903006}) 2024-04-26T00:42:24.305 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.300+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3380124846}) 2024-04-26T00:42:24.307 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:42:24.307 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:42:24.312 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.308+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.312 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.308+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4227375017}) 2024-04-26T00:42:24.313 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.308+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3640844487}) 2024-04-26T00:42:24.313 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.308+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.313 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.308+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4227375017}) 2024-04-26T00:42:24.313 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.308+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3286937846}) 2024-04-26T00:42:24.316 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.312+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.317 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.312+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4126871668}) 2024-04-26T00:42:24.317 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.312+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3044665613}) 2024-04-26T00:42:24.317 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.312+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.317 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.312+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4126871668}) 2024-04-26T00:42:24.317 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.312+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3624750146}) 2024-04-26T00:42:24.336 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.332+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.336 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.332+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2582927910}) 2024-04-26T00:42:24.336 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.332+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=872973339}) 2024-04-26T00:42:24.336 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.332+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.336 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.332+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2582927910}) 2024-04-26T00:42:24.337 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.332+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1062543562}) 2024-04-26T00:42:24.339 INFO:tasks.rados.rados.0.smithi045.stdout:3976: done (3 left) 2024-04-26T00:42:24.339 INFO:tasks.rados.rados.0.smithi045.stdout:3977: copy_from oid 45 from oid 44 current snap is 440 2024-04-26T00:42:24.340 INFO:tasks.rados.rados.0.smithi045.stdout:3971: expect (ObjNum 1360 snap 420 seq_num 1360) 2024-04-26T00:42:24.341 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.336+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.341 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.336+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4293668301}) 2024-04-26T00:42:24.342 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.336+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=562379704}) 2024-04-26T00:42:24.342 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.336+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.342 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.336+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4293668301}) 2024-04-26T00:42:24.342 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.336+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=649184890}) 2024-04-26T00:42:24.346 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.340+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.346 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.340+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3006598845}) 2024-04-26T00:42:24.346 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.340+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4232543983}) 2024-04-26T00:42:24.346 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.344+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.346 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.344+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3006598845}) 2024-04-26T00:42:24.346 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.344+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=472494841}) 2024-04-26T00:42:24.356 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.352+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.356 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.352+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3166589687}) 2024-04-26T00:42:24.356 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.352+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3710185647}) 2024-04-26T00:42:24.356 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.352+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.356 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.352+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3166589687}) 2024-04-26T00:42:24.356 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.352+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1956884768}) 2024-04-26T00:42:24.359 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.356+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.359 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.356+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3938766551}) 2024-04-26T00:42:24.359 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.356+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2188580670}) 2024-04-26T00:42:24.359 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.356+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.360 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.356+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3938766551}) 2024-04-26T00:42:24.360 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.356+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2870628775}) 2024-04-26T00:42:24.364 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.360+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.365 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.360+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1035356645}) 2024-04-26T00:42:24.365 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.360+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3787347844}) 2024-04-26T00:42:24.365 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.360+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.365 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.360+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1035356645}) 2024-04-26T00:42:24.365 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.360+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4044846250}) 2024-04-26T00:42:24.368 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.364+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.368 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.364+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4204743851}) 2024-04-26T00:42:24.368 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.364+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=522064946}) 2024-04-26T00:42:24.368 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.364+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.368 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.364+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4204743851}) 2024-04-26T00:42:24.368 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.364+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3285978139}) 2024-04-26T00:42:24.370 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.368+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.373 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.368+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1781441978}) 2024-04-26T00:42:24.373 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.368+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=787572052}) 2024-04-26T00:42:24.373 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.368+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.373 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.368+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1781441978}) 2024-04-26T00:42:24.374 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.368+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4081122781}) 2024-04-26T00:42:24.374 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.372+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.375 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.372+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3170624702}) 2024-04-26T00:42:24.375 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.372+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=220589783}) 2024-04-26T00:42:24.375 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.372+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.375 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.372+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3170624702}) 2024-04-26T00:42:24.375 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.372+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1614110793}) 2024-04-26T00:42:24.377 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.372+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.377 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.372+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3585470892}) 2024-04-26T00:42:24.378 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.372+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=498927702}) 2024-04-26T00:42:24.378 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.372+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.378 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.372+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3585470892}) 2024-04-26T00:42:24.378 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.372+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1250703804}) 2024-04-26T00:42:24.380 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.376+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.380 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.376+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=254090932}) 2024-04-26T00:42:24.380 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.376+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1249106735}) 2024-04-26T00:42:24.380 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.376+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.380 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.376+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=254090932}) 2024-04-26T00:42:24.380 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.376+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4215617035}) 2024-04-26T00:42:24.382 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.380+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.383 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.380+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3446482032}) 2024-04-26T00:42:24.383 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.380+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3118926480}) 2024-04-26T00:42:24.383 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.380+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.383 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.380+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3446482032}) 2024-04-26T00:42:24.383 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.380+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2624482960}) 2024-04-26T00:42:24.384 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.380+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.384 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.380+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2486507785}) 2024-04-26T00:42:24.384 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.380+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2646008188}) 2024-04-26T00:42:24.384 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.380+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.384 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.380+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2486507785}) 2024-04-26T00:42:24.384 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.380+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2804947008}) 2024-04-26T00:42:24.385 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.380+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.386 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.380+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4256699630}) 2024-04-26T00:42:24.386 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.380+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1084901229}) 2024-04-26T00:42:24.386 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.380+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.386 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.380+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4256699630}) 2024-04-26T00:42:24.386 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.380+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2449718707}) 2024-04-26T00:42:24.387 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.384+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.387 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.384+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2742768918}) 2024-04-26T00:42:24.387 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.384+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1000856719}) 2024-04-26T00:42:24.387 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.384+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.387 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.384+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2742768918}) 2024-04-26T00:42:24.387 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.384+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1058783791}) 2024-04-26T00:42:24.389 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.384+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.389 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.384+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1222763740}) 2024-04-26T00:42:24.389 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.384+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1745883506}) 2024-04-26T00:42:24.389 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.384+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.389 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.384+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1222763740}) 2024-04-26T00:42:24.390 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.384+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4144576577}) 2024-04-26T00:42:24.390 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.388+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.390 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.388+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2783113198}) 2024-04-26T00:42:24.391 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.388+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3639518700}) 2024-04-26T00:42:24.391 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.388+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.391 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.388+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2783113198}) 2024-04-26T00:42:24.391 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.388+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2726982033}) 2024-04-26T00:42:24.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f271abd4640 10 monclient: get_monmap_and_config 2024-04-26T00:42:24.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f271abd4640 10 monclient: build_initial_monmap 2024-04-26T00:42:24.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f271abd4640 10 monclient: monmap: 2024-04-26T00:42:24.457 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:42:24.457 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:24.457 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:42:24.457690+0000 2024-04-26T00:42:24.457 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:42:24.457690+0000 2024-04-26T00:42:24.457 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:42:24.457 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:24.457 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:42:24.458 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:42:24.458 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:42:24.458 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:42:24.458 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:42:24.458 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:24.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f271abd4640 1 Processor -- start 2024-04-26T00:42:24.472 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f271abd4640 1 -- start start 2024-04-26T00:42:24.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f271abd4640 10 monclient: init 2024-04-26T00:42:24.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f271abd4640 10 monclient: _reopen_session rank -1 2024-04-26T00:42:24.473 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f271abd4640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-04-26T00:42:24.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f271abd4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2714135170 addr v1:172.21.15.37:6789/0 2024-04-26T00:42:24.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f271abd4640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f2714072b00 0x7f2714139ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:24.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f271abd4640 10 monclient(hunting): picked mon.noname-a con 0x7f2714072b00 addr v2:172.21.15.37:3300/0 2024-04-26T00:42:24.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f271abd4640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f271413a400 0x7f271413c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:24.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f271abd4640 10 monclient(hunting): picked mon.noname-b con 0x7f271413a400 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:42:24.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f271abd4640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f271406f310 con 0x7f2714135170 2024-04-26T00:42:24.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f271abd4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:24.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f271abd4640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f2714071de0 con 0x7f2714072b00 2024-04-26T00:42:24.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f271abd4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:24.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f271abd4640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f27140720c0 con 0x7f271413a400 2024-04-26T00:42:24.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f271abd4640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:24.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f271abd4640 10 monclient(hunting): authenticate will time out at 2325.763429s 2024-04-26T00:42:24.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f2719171640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f271413a400 0x7f271413c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:24.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f2713fff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f2714072b00 0x7f2714139ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:24.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f2719171640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f271413a400 0x7f271413c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51078/0 (socket says 172.21.15.37:51078) 2024-04-26T00:42:24.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f2719171640 1 -- 172.21.15.37:0/468950690 learned_addr learned my addr 172.21.15.37:0/468950690 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:42:24.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f2719171640 10 monclient(hunting): get_auth_request con 0x7f271413a400 auth_method 0 2024-04-26T00:42:24.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f2719171640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:24.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f2719171640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:24.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f2719171640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:24.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f2713fff640 10 monclient(hunting): get_auth_request con 0x7f2714072b00 auth_method 0 2024-04-26T00:42:24.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f2713fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:24.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f2713fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:24.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f2713fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:24.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f2713fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:24.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f2713fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:24.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f2713fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:24.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f2719171640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:24.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f2719171640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:24.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f2719171640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:24.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f2713fff640 10 monclient(hunting): handle_auth_done global_id 264219 payload 274 2024-04-26T00:42:24.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f2713fff640 1 -- 172.21.15.37:0/468950690 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f271413a400 msgr2=0x7f271413c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:42:24.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f2713fff640 1 --2- 172.21.15.37:0/468950690 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f271413a400 0x7f271413c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:24.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f2713fff640 1 -- 172.21.15.37:0/468950690 >> v1:172.21.15.37:6789/0 conn(0x7f2714135170 legacy=0x7f2714135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:42:24.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f2713fff640 10 monclient: _finish_hunting 0 2024-04-26T00:42:24.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f2713fff640 1 monclient: found mon.noname-a 2024-04-26T00:42:24.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f2713fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:42:24.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f2713fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:42:24.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.452+0000 7f2713fff640 1 -- 172.21.15.37:0/468950690 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27140723a0 con 0x7f2714072b00 2024-04-26T00:42:24.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f2713fff640 1 -- 172.21.15.37:0/468950690 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2714072b00 msgr2=0x7f2714139ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:24.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f2713fff640 1 --2- 172.21.15.37:0/468950690 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2714072b00 0x7f2714139ec0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f27080097c0 tx=0x7f270801c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f6a77108fb58663 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:24.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f27137fe640 1 -- 172.21.15.37:0/468950690 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27080b1070 con 0x7f2714072b00 2024-04-26T00:42:24.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f27137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:24.477 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f27137fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:42:24.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f27137fe640 10 monclient: dump: 2024-04-26T00:42:24.478 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:24.478 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:24.478 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:24.478 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:24.478 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:24.478 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:24.478 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:24.478 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:24.479 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:24.479 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:24.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f27137fe640 10 monclient: _finish_auth 0 2024-04-26T00:42:24.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f27137fe640 10 monclient: _check_auth_tickets 2024-04-26T00:42:24.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f27137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:42:24.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f271abd4640 5 monclient: authenticate success, global_id 264219 2024-04-26T00:42:24.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f271abd4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:42:24.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f27137fe640 1 -- 172.21.15.37:0/468950690 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27080ac640 con 0x7f2714072b00 2024-04-26T00:42:24.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f27137fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:42:24.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f271abd4640 10 monclient: get_monmap_and_config success 2024-04-26T00:42:24.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f271abd4640 4 set_mon_vals no callback set 2024-04-26T00:42:24.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f27137fe640 1 -- 172.21.15.37:0/468950690 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27080aca10 con 0x7f2714072b00 2024-04-26T00:42:24.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f27137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:24.479 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f27137fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:42:24.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f27137fe640 10 monclient: dump: 2024-04-26T00:42:24.480 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:24.480 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:24.480 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:24.480 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:24.480 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:24.480 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:24.480 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:24.480 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:24.480 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:24.480 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:24.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f271abd4640 10 monclient: shutdown 2024-04-26T00:42:24.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f271abd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:42:24.480 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f271abd4640 1 -- 172.21.15.37:0/468950690 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2714072b00 msgr2=0x7f2714139ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:24.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f271abd4640 1 --2- 172.21.15.37:0/468950690 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2714072b00 0x7f2714139ec0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f27080097c0 tx=0x7f270801c440 comp rx=0 tx=0).stop 2024-04-26T00:42:24.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f271abd4640 1 -- 172.21.15.37:0/468950690 shutdown_connections 2024-04-26T00:42:24.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f271abd4640 1 --2- 172.21.15.37:0/468950690 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f271413a400 0x7f271413c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:24.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f271abd4640 1 --2- 172.21.15.37:0/468950690 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2714072b00 0x7f2714139ec0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:24.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f271abd4640 1 -- 172.21.15.37:0/468950690 >> 172.21.15.37:0/468950690 conn(0x7f27140a6300 msgr2=0x7f27140a2610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:42:24.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f271abd4640 1 -- 172.21.15.37:0/468950690 shutdown_connections 2024-04-26T00:42:24.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f271abd4640 1 -- 172.21.15.37:0/468950690 wait complete. 2024-04-26T00:42:24.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f271abd4640 10 monclient: build_initial_monmap 2024-04-26T00:42:24.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f271abd4640 10 monclient: monmap: 2024-04-26T00:42:24.481 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:42:24.481 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:42:24.481 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:42:24.481 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:42:24.481 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:42:24.481 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:42:24.482 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:42:24.482 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:42:24.482 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:42:24.482 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:24.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f271abd4640 1 Processor -- start 2024-04-26T00:42:24.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f271abd4640 1 -- start start 2024-04-26T00:42:24.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f271abd4640 10 monclient: init 2024-04-26T00:42:24.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f271abd4640 10 monclient: _reopen_session rank -1 2024-04-26T00:42:24.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f271abd4640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:42:24.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f271abd4640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2714072b00 0x7f271412e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:24.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f271abd4640 10 monclient(hunting): picked mon.noname-a con 0x7f2714072b00 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:42:24.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f271abd4640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2714135170 0x7f271412edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:24.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f271abd4640 10 monclient(hunting): picked mon.noname-c con 0x7f2714135170 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:42:24.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f271abd4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f271413a400 0x7f27141cef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:24.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f271abd4640 10 monclient(hunting): picked mon.noname-b con 0x7f271413a400 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:42:24.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f271abd4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:24.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f271abd4640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f271413f760 con 0x7f2714072b00 2024-04-26T00:42:24.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f271abd4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:24.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f271abd4640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f271413f8e0 con 0x7f271413a400 2024-04-26T00:42:24.483 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f271abd4640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:24.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f271abd4640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f271413fa60 con 0x7f2714135170 2024-04-26T00:42:24.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f271abd4640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:24.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f271abd4640 10 monclient(hunting): authenticate will time out at 2325.770518s 2024-04-26T00:42:24.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f2713fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2714135170 0x7f271412edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:24.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f2718970640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2714072b00 0x7f271412e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:24.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f2713fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2714135170 0x7f271412edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51092/0 (socket says 172.21.15.37:51092) 2024-04-26T00:42:24.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f2713fff640 1 -- 172.21.15.37:0/3029208765 learned_addr learned my addr 172.21.15.37:0/3029208765 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:42:24.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f2713fff640 10 monclient(hunting): get_auth_request con 0x7f2714135170 auth_method 0 2024-04-26T00:42:24.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f2718970640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2714072b00 0x7f271412e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40270/0 (socket says 172.21.15.37:40270) 2024-04-26T00:42:24.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f2713fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:24.484 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f2713fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:24.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f2713fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:24.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f2718970640 10 monclient(hunting): get_auth_request con 0x7f2714072b00 auth_method 0 2024-04-26T00:42:24.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f2718970640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:24.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f2718970640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:24.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f2718970640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:24.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f2713fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:24.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f2713fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:24.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f2713fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:24.485 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f2718970640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:24.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f2718970640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:24.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f2718970640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:24.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f2719171640 1 --2- 172.21.15.37:0/3029208765 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f271413a400 0x7f27141cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:24.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f2719171640 10 monclient(hunting): get_auth_request con 0x7f271413a400 auth_method 0 2024-04-26T00:42:24.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f2719171640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:24.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f2719171640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:24.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f2719171640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:24.486 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f2713fff640 10 monclient(hunting): handle_auth_done global_id 254339 payload 824 2024-04-26T00:42:24.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f2713fff640 1 -- 172.21.15.37:0/3029208765 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f271413a400 msgr2=0x7f27141cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:24.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f2713fff640 1 --2- 172.21.15.37:0/3029208765 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f271413a400 0x7f27141cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:24.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f2713fff640 1 -- 172.21.15.37:0/3029208765 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2714072b00 msgr2=0x7f271412e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:24.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f2713fff640 1 --2- 172.21.15.37:0/3029208765 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2714072b00 0x7f271412e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:24.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f2713fff640 10 monclient: _finish_hunting 0 2024-04-26T00:42:24.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f2713fff640 1 monclient: found mon.noname-c 2024-04-26T00:42:24.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f2713fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:42:24.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f2713fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:42:24.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f2713fff640 1 -- 172.21.15.37:0/3029208765 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f271409c6c0 con 0x7f2714135170 2024-04-26T00:42:24.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.460+0000 7f2718970640 1 --2- 172.21.15.37:0/3029208765 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2714072b00 0x7f271412e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:42:24.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.464+0000 7f2713fff640 1 -- 172.21.15.37:0/3029208765 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2714135170 msgr2=0x7f271412edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:24.487 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.464+0000 7f2713fff640 1 --2- 172.21.15.37:0/3029208765 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2714135170 0x7f271412edb0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f27080af040 tx=0x7f27080191e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:24.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.464+0000 7f27117fa640 1 -- 172.21.15.37:0/3029208765 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27080b1040 con 0x7f2714135170 2024-04-26T00:42:24.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.464+0000 7f27117fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:24.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.464+0000 7f27117fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:42:24.488 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.464+0000 7f27117fa640 10 monclient: dump: 2024-04-26T00:42:24.488 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:24.488 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:24.488 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:24.488 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:24.488 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:24.488 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:24.488 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:24.489 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:24.489 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:24.489 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:24.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.464+0000 7f27117fa640 10 monclient: _finish_auth 0 2024-04-26T00:42:24.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.464+0000 7f27117fa640 10 monclient: _check_auth_tickets 2024-04-26T00:42:24.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.464+0000 7f27117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:42:24.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.464+0000 7f271abd4640 5 monclient: authenticate success, global_id 254339 2024-04-26T00:42:24.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.464+0000 7f27117fa640 1 -- 172.21.15.37:0/3029208765 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27080ba070 con 0x7f2714135170 2024-04-26T00:42:24.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.464+0000 7f27117fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:42:24.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.464+0000 7f27117fa640 1 -- 172.21.15.37:0/3029208765 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27080b4a00 con 0x7f2714135170 2024-04-26T00:42:24.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.464+0000 7f27117fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:24.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.464+0000 7f27117fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:42:24.489 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.464+0000 7f27117fa640 10 monclient: dump: 2024-04-26T00:42:24.490 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:24.490 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:24.490 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:24.490 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:24.490 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:24.490 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:24.490 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:24.490 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:24.490 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:24.490 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:24.490 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.464+0000 7f2712ffd640 4 set_mon_vals no callback set 2024-04-26T00:42:24.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.464+0000 7f271abd4640 10 monclient: _renew_subs 2024-04-26T00:42:24.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.464+0000 7f271abd4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:24.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.464+0000 7f271abd4640 1 -- 172.21.15.37:0/3029208765 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f271409e9b0 con 0x7f2714135170 2024-04-26T00:42:24.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.464+0000 7f271abd4640 10 monclient: _renew_subs 2024-04-26T00:42:24.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.464+0000 7f271abd4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:24.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.464+0000 7f271abd4640 1 -- 172.21.15.37:0/3029208765 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f271409d220 con 0x7f2714135170 2024-04-26T00:42:24.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.464+0000 7f27117fa640 1 -- 172.21.15.37:0/3029208765 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f27080ac500 con 0x7f2714135170 2024-04-26T00:42:24.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.464+0000 7f271abd4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:24.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.464+0000 7f271abd4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:24.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.464+0000 7f271abd4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:24.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.464+0000 7f271abd4640 1 -- 172.21.15.37:0/3029208765 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2714066f40 con 0x7f2714135170 2024-04-26T00:42:24.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.464+0000 7f27117fa640 1 --2- 172.21.15.37:0/3029208765 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2700035ed0 0x7f2700038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:24.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.464+0000 7f27117fa640 1 -- 172.21.15.37:0/3029208765 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(896..896 src has 822..896) ==== 3975+0+0 (secure 0 0 0) 0x7f27080e26b0 con 0x7f2714135170 2024-04-26T00:42:24.491 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.464+0000 7f2718970640 1 --2- 172.21.15.37:0/3029208765 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2700035ed0 0x7f2700038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:24.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.464+0000 7f2718970640 10 monclient: get_auth_request con 0x7f2700035ed0 auth_method 0 2024-04-26T00:42:24.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.468+0000 7f2718970640 1 --2- 172.21.15.37:0/3029208765 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2700035ed0 0x7f2700038390 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f2704007fd0 tx=0x7f2704007e40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:24.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.472+0000 7f27117fa640 1 -- 172.21.15.37:0/3029208765 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f2708016030 con 0x7f2714135170 2024-04-26T00:42:24.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.472+0000 7f27117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:24.492 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.472+0000 7f27117fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:42:24.504 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.500+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.504 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.500+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1199071414}) 2024-04-26T00:42:24.504 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.500+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=20130771}) 2024-04-26T00:42:24.504 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.500+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:24.504 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.500+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1199071414}) 2024-04-26T00:42:24.504 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:24.500+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1853640712}) 2024-04-26T00:42:24.578 INFO:tasks.rados.rados.0.smithi045.stdout:3975: finishing rollback tid 0 to smithi04526579-9 2024-04-26T00:42:24.578 INFO:tasks.rados.rados.0.smithi045.stdout:3975: finishing rollback tid 1 to smithi04526579-9 2024-04-26T00:42:24.578 INFO:tasks.rados.rados.0.smithi045.stdout:3975: finishing rollback tid 2 to smithi04526579-9 2024-04-26T00:42:24.578 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 9 v 2558 (ObjNum 1392 snap 430 seq_num 1392) dirty exists 2024-04-26T00:42:24.578 INFO:tasks.rados.rados.0.smithi045.stdout:3977: finishing copy_from racing read to smithi04526579-45 2024-04-26T00:42:24.578 INFO:tasks.rados.rados.0.smithi045.stdout:3977: finishing copy_from to smithi04526579-45 2024-04-26T00:42:24.578 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 2418 (ObjNum 1377 snap 426 seq_num 1377) dirty exists 2024-04-26T00:42:24.578 INFO:tasks.rados.rados.0.smithi045.stdout:3971: done (3 left) 2024-04-26T00:42:24.578 INFO:tasks.rados.rados.0.smithi045.stdout:3974: done (2 left) 2024-04-26T00:42:24.578 INFO:tasks.rados.rados.0.smithi045.stdout:3975: done (1 left) 2024-04-26T00:42:24.579 INFO:tasks.rados.rados.0.smithi045.stdout:3977: done (0 left) 2024-04-26T00:42:24.579 INFO:tasks.rados.rados.0.smithi045.stdout:3978: read oid 18 snap -1 2024-04-26T00:42:24.579 INFO:tasks.rados.rados.0.smithi045.stdout:3978: expect (ObjNum 1096 snap 325 seq_num 1096) 2024-04-26T00:42:24.579 INFO:tasks.rados.rados.0.smithi045.stdout:3979: write (excl) oid 30 current snap is 440 2024-04-26T00:42:24.579 INFO:tasks.rados.rados.0.smithi045.stdout:3979: seq_num 1418 ranges {632326=744740,1794643=536494,3040392=453126} 2024-04-26T00:42:24.598 INFO:tasks.rados.rados.0.smithi045.stdout:3979: writing smithi04526579-30 from 632326 to 1377066 tid 1 2024-04-26T00:42:24.612 INFO:tasks.rados.rados.0.smithi045.stdout:3979: writing smithi04526579-30 from 1794643 to 2331137 tid 2 2024-04-26T00:42:24.629 INFO:tasks.rados.rados.0.smithi045.stdout:3979: writing smithi04526579-30 from 3040392 to 3493518 tid 3 2024-04-26T00:42:24.629 INFO:tasks.rados.rados.0.smithi045.stdout:3980: write (excl) oid 1 current snap is 440 2024-04-26T00:42:24.629 INFO:tasks.rados.rados.0.smithi045.stdout:3980: seq_num 1419 ranges {649133=419989,1853125=680152,2951462=664398} 2024-04-26T00:42:24.641 INFO:tasks.rados.rados.0.smithi045.stdout:3980: writing smithi04526579-1 from 649133 to 1069122 tid 1 2024-04-26T00:42:24.655 INFO:tasks.rados.rados.0.smithi045.stdout:3980: writing smithi04526579-1 from 1853125 to 2533277 tid 2 2024-04-26T00:42:24.665 INFO:tasks.rados.rados.0.smithi045.stdout:3980: writing smithi04526579-1 from 2951462 to 3615860 tid 3 2024-04-26T00:42:24.665 INFO:tasks.rados.rados.0.smithi045.stdout:3981: rollback oid 49 current snap is 440 2024-04-26T00:42:24.665 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 49 to 434 2024-04-26T00:42:24.665 INFO:tasks.rados.rados.0.smithi045.stdout:3978: expect (ObjNum 1096 snap 325 seq_num 1096) 2024-04-26T00:42:24.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.696+0000 7f271abd4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:42:24.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.696+0000 7f271abd4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:42:24.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.696+0000 7f271abd4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:24.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.696+0000 7f271abd4640 1 -- 172.21.15.37:0/3029208765 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f271406c1e0 con 0x7f2714135170 2024-04-26T00:42:24.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.696+0000 7f27117fa640 1 -- 172.21.15.37:0/3029208765 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f27080aac40 con 0x7f2714135170 2024-04-26T00:42:24.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.696+0000 7f27117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:42:24.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.696+0000 7f27117fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:42:24.700 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":160,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:42:24.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.696+0000 7f271abd4640 1 -- 172.21.15.37:0/3029208765 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2700035ed0 msgr2=0x7f2700038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:24.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.696+0000 7f271abd4640 1 --2- 172.21.15.37:0/3029208765 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2700035ed0 0x7f2700038390 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f2704007fd0 tx=0x7f2704007e40 comp rx=0 tx=0).stop 2024-04-26T00:42:24.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.696+0000 7f271abd4640 10 monclient: shutdown 2024-04-26T00:42:24.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.696+0000 7f271abd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:42:24.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.696+0000 7f271abd4640 1 -- 172.21.15.37:0/3029208765 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2714135170 msgr2=0x7f271412edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:24.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.696+0000 7f271abd4640 1 --2- 172.21.15.37:0/3029208765 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2714135170 0x7f271412edb0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f27080af040 tx=0x7f27080191e0 comp rx=0 tx=0).stop 2024-04-26T00:42:24.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.700+0000 7f271abd4640 1 -- 172.21.15.37:0/3029208765 shutdown_connections 2024-04-26T00:42:24.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.700+0000 7f271abd4640 1 --2- 172.21.15.37:0/3029208765 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2700035ed0 0x7f2700038390 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:24.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.700+0000 7f271abd4640 1 --2- 172.21.15.37:0/3029208765 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f271413a400 0x7f27141cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:24.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.700+0000 7f271abd4640 1 --2- 172.21.15.37:0/3029208765 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2714135170 0x7f271412edb0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:24.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.700+0000 7f271abd4640 1 --2- 172.21.15.37:0/3029208765 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2714072b00 0x7f271412e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:24.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.700+0000 7f271abd4640 1 -- 172.21.15.37:0/3029208765 >> 172.21.15.37:0/3029208765 conn(0x7f27140a6300 msgr2=0x7f2714133730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:42:24.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.700+0000 7f271abd4640 1 -- 172.21.15.37:0/3029208765 shutdown_connections 2024-04-26T00:42:24.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.700+0000 7f271abd4640 1 -- 172.21.15.37:0/3029208765 wait complete. 2024-04-26T00:42:24.719 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:42:24.719 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-26T00:42:24.719 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:42:24.719 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:42:24.855 INFO:tasks.rados.rados.0.smithi045.stdout:3979: finishing write tid 1 to smithi04526579-30 2024-04-26T00:42:24.855 INFO:tasks.rados.rados.0.smithi045.stdout:3979: finishing write tid 2 to smithi04526579-30 2024-04-26T00:42:24.855 INFO:tasks.rados.rados.0.smithi045.stdout:3978: done (3 left) 2024-04-26T00:42:24.855 INFO:tasks.rados.rados.0.smithi045.stdout:3982: write oid 48 current snap is 440 2024-04-26T00:42:24.855 INFO:tasks.rados.rados.0.smithi045.stdout:3982: seq_num 1420 ranges {665940=495238,1795254=540163,2978885=475670} 2024-04-26T00:42:24.861 INFO:tasks.rados.rados.0.smithi045.stdout:3982: writing smithi04526579-48 from 665940 to 1161178 tid 1 2024-04-26T00:42:24.868 INFO:tasks.rados.rados.0.smithi045.stdout:3982: writing smithi04526579-48 from 1795254 to 2335417 tid 2 2024-04-26T00:42:24.874 INFO:tasks.rados.rados.0.smithi045.stdout:3982: writing smithi04526579-48 from 2978885 to 3454555 tid 3 2024-04-26T00:42:24.874 INFO:tasks.rados.rados.0.smithi045.stdout:3983: write oid 22 current snap is 440 2024-04-26T00:42:24.875 INFO:tasks.rados.rados.0.smithi045.stdout:3979: finishing write tid 3 to smithi04526579-30 2024-04-26T00:42:24.875 INFO:tasks.rados.rados.0.smithi045.stdout:3979: finishing write tid 4 to smithi04526579-30 2024-04-26T00:42:24.875 INFO:tasks.rados.rados.0.smithi045.stdout:3979: finishing write tid 5 to smithi04526579-30 2024-04-26T00:42:24.875 INFO:tasks.rados.rados.0.smithi045.stdout:3979: oid 30 updating version 0 to 2881 2024-04-26T00:42:24.875 INFO:tasks.rados.rados.0.smithi045.stdout:3979: oid 30 version 2881 is already newer than 2879 2024-04-26T00:42:24.875 INFO:tasks.rados.rados.0.smithi045.stdout:3979: oid 30 version 2881 is already newer than 2878 2024-04-26T00:42:24.875 INFO:tasks.rados.rados.0.smithi045.stdout:3979: oid 30 version 2881 is already newer than 2880 2024-04-26T00:42:24.875 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 30 v 2881 (ObjNum 1418 snap 440 seq_num 1418) dirty exists 2024-04-26T00:42:24.875 INFO:tasks.rados.rados.0.smithi045.stdout:3979: left oid 30 (ObjNum 1418 snap 440 seq_num 1418) 2024-04-26T00:42:24.875 INFO:tasks.rados.rados.0.smithi045.stdout:3980: finishing write tid 1 to smithi04526579-1 2024-04-26T00:42:24.875 INFO:tasks.rados.rados.0.smithi045.stdout:3983: seq_num 1421 ranges {682747=570487,1853736=683821,3006308=403295} 2024-04-26T00:42:24.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f94769ae640 10 monclient: get_monmap_and_config 2024-04-26T00:42:24.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f94769ae640 10 monclient: build_initial_monmap 2024-04-26T00:42:24.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f94769ae640 10 monclient: monmap: 2024-04-26T00:42:24.878 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:42:24.878 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:24.878 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:42:24.880255+0000 2024-04-26T00:42:24.879 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:42:24.880255+0000 2024-04-26T00:42:24.879 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:42:24.879 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:24.879 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:42:24.879 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:42:24.879 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:42:24.879 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:42:24.879 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:42:24.879 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:24.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f94769ae640 1 Processor -- start 2024-04-26T00:42:24.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f94769ae640 1 -- start start 2024-04-26T00:42:24.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f94769ae640 10 monclient: init 2024-04-26T00:42:24.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f94769ae640 10 monclient: _reopen_session rank -1 2024-04-26T00:42:24.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f94769ae640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-04-26T00:42:24.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f94769ae640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9470131cd0 0x7f94701320b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:24.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f94769ae640 10 monclient(hunting): picked mon.noname-c con 0x7f9470131cd0 addr v2:172.21.15.45:3300/0 2024-04-26T00:42:24.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f94769ae640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9470073080 0x7f9470134c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:24.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f94769ae640 10 monclient(hunting): picked mon.noname-b con 0x7f9470073080 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:42:24.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f94769ae640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f94700a1b60 addr v1:172.21.15.37:6789/0 2024-04-26T00:42:24.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f94769ae640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9470071660 con 0x7f94700a1b60 2024-04-26T00:42:24.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f94769ae640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:24.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f94769ae640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f9470071940 con 0x7f9470131cd0 2024-04-26T00:42:24.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f94769ae640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:24.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f94769ae640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9470071c20 con 0x7f9470073080 2024-04-26T00:42:24.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f94769ae640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:24.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f94769ae640 10 monclient(hunting): authenticate will time out at 2326.186060s 2024-04-26T00:42:24.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f9474f4b640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f94700a1b60 0x7f9470135150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:44934/0 (socket says 172.21.15.37:44934) 2024-04-26T00:42:24.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f9474f4b640 1 -- 172.21.15.37:0/2827727065 learned_addr learned my addr 172.21.15.37:0/2827727065 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:42:24.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f946ffff640 1 --2- 172.21.15.37:0/2827727065 >> v2:172.21.15.45:3300/0 conn(0x7f9470131cd0 0x7f94701320b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:24.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f946ffff640 10 monclient(hunting): get_auth_request con 0x7f9470131cd0 auth_method 0 2024-04-26T00:42:24.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f9467fff640 1 --2- 172.21.15.37:0/2827727065 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9470073080 0x7f9470134c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:24.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f946ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:24.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f946ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:24.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f946ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:24.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f9467fff640 10 monclient(hunting): get_auth_request con 0x7f9470073080 auth_method 0 2024-04-26T00:42:24.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f9467fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:24.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f9467fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:24.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f9467fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:24.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f946ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:24.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f946ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:24.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f946ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:24.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f9467fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:24.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f9467fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:24.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f9467fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:24.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f946ffff640 10 monclient(hunting): handle_auth_done global_id 274108 payload 274 2024-04-26T00:42:24.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f946ffff640 1 -- 172.21.15.37:0/2827727065 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9470073080 msgr2=0x7f9470134c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:42:24.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f946ffff640 1 --2- 172.21.15.37:0/2827727065 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9470073080 0x7f9470134c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:24.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f946ffff640 1 -- 172.21.15.37:0/2827727065 >> v1:172.21.15.37:6789/0 conn(0x7f94700a1b60 legacy=0x7f9470135150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:42:24.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f946ffff640 10 monclient: _finish_hunting 0 2024-04-26T00:42:24.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f946ffff640 1 monclient: found mon.noname-c 2024-04-26T00:42:24.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f946ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:42:24.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f946ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:42:24.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.876+0000 7f946ffff640 1 -- 172.21.15.37:0/2827727065 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94701325f0 con 0x7f9470131cd0 2024-04-26T00:42:24.885 INFO:tasks.rados.rados.0.smithi045.stdout:3983: writing smithi04526579-22 from 682747 to 1253234 tid 1 2024-04-26T00:42:24.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f946ffff640 1 -- 172.21.15.37:0/2827727065 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9470131cd0 msgr2=0x7f94701320b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:24.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f946ffff640 1 --2- 172.21.15.37:0/2827727065 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9470131cd0 0x7f94701320b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f94600097c0 tx=0x7f946001c3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5da29e534845b29 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:24.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f946f7fe640 1 -- 172.21.15.37:0/2827727065 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94600b2070 con 0x7f9470131cd0 2024-04-26T00:42:24.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f946f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:24.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f946f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:42:24.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f946f7fe640 10 monclient: dump: 2024-04-26T00:42:24.887 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:24.887 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:24.887 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:24.887 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:24.887 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:24.887 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:24.887 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:24.887 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:24.887 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:24.887 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:24.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f946f7fe640 10 monclient: _finish_auth 0 2024-04-26T00:42:24.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f946f7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:42:24.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f946f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:42:24.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f946f7fe640 1 -- 172.21.15.37:0/2827727065 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9470135a70 con 0x7f9470131cd0 2024-04-26T00:42:24.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f94769ae640 5 monclient: authenticate success, global_id 274108 2024-04-26T00:42:24.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f94769ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:42:24.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f946f7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:42:24.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f94769ae640 10 monclient: get_monmap_and_config success 2024-04-26T00:42:24.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f946f7fe640 1 -- 172.21.15.37:0/2827727065 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94600ad860 con 0x7f9470131cd0 2024-04-26T00:42:24.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f946f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:24.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f94769ae640 4 set_mon_vals no callback set 2024-04-26T00:42:24.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f946f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:42:24.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f946f7fe640 10 monclient: dump: 2024-04-26T00:42:24.888 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:24.888 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:24.888 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:24.889 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:24.889 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:24.889 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:24.889 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:24.889 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:24.889 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:24.889 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:24.895 INFO:tasks.rados.rados.0.smithi045.stdout:3983: writing smithi04526579-22 from 1853736 to 2537557 tid 2 2024-04-26T00:42:24.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f94769ae640 10 monclient: shutdown 2024-04-26T00:42:24.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f94769ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:42:24.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f94769ae640 1 -- 172.21.15.37:0/2827727065 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9470131cd0 msgr2=0x7f94701320b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:24.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f94769ae640 1 --2- 172.21.15.37:0/2827727065 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9470131cd0 0x7f94701320b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f94600097c0 tx=0x7f946001c3b0 comp rx=0 tx=0).stop 2024-04-26T00:42:24.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f94769ae640 1 -- 172.21.15.37:0/2827727065 shutdown_connections 2024-04-26T00:42:24.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f94769ae640 1 --2- 172.21.15.37:0/2827727065 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9470073080 0x7f9470134c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:24.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f94769ae640 1 --2- 172.21.15.37:0/2827727065 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9470131cd0 0x7f94701320b0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:24.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f94769ae640 1 -- 172.21.15.37:0/2827727065 >> 172.21.15.37:0/2827727065 conn(0x7f94700a7160 msgr2=0x7f94700a7540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:42:24.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f94769ae640 1 -- 172.21.15.37:0/2827727065 shutdown_connections 2024-04-26T00:42:24.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f94769ae640 1 -- 172.21.15.37:0/2827727065 wait complete. 2024-04-26T00:42:24.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f94769ae640 10 monclient: build_initial_monmap 2024-04-26T00:42:24.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f94769ae640 10 monclient: monmap: 2024-04-26T00:42:24.897 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:42:24.897 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:42:24.897 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:42:24.897 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:42:24.898 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:42:24.898 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:42:24.898 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:42:24.898 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:42:24.898 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:42:24.898 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:24.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f94769ae640 1 Processor -- start 2024-04-26T00:42:24.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f94769ae640 1 -- start start 2024-04-26T00:42:24.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f94769ae640 10 monclient: init 2024-04-26T00:42:24.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f94769ae640 10 monclient: _reopen_session rank -1 2024-04-26T00:42:24.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f94769ae640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:42:24.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f94769ae640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9470073080 0x7f94701cd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:24.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f94769ae640 10 monclient(hunting): picked mon.noname-c con 0x7f9470073080 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:42:24.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f94769ae640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94700a1b60 0x7f947013bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:24.898 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f94769ae640 10 monclient(hunting): picked mon.noname-b con 0x7f94700a1b60 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:42:24.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f94769ae640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9470131cd0 0x7f947013c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:24.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f94769ae640 10 monclient(hunting): picked mon.noname-a con 0x7f9470131cd0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:42:24.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f94769ae640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:24.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f94769ae640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f947013fb00 con 0x7f9470131cd0 2024-04-26T00:42:24.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f94769ae640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:24.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f94769ae640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f947013f980 con 0x7f94700a1b60 2024-04-26T00:42:24.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f94769ae640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:24.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f94769ae640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f947013fc80 con 0x7f9470073080 2024-04-26T00:42:24.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f94769ae640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:24.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f94769ae640 10 monclient(hunting): authenticate will time out at 2326.195353s 2024-04-26T00:42:24.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f946ffff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9470073080 0x7f94701cd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:24.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f946ffff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9470073080 0x7f94701cd0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51118/0 (socket says 172.21.15.37:51118) 2024-04-26T00:42:24.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f946ffff640 1 -- 172.21.15.37:0/4164519502 learned_addr learned my addr 172.21.15.37:0/4164519502 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:42:24.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f946ffff640 10 monclient(hunting): get_auth_request con 0x7f9470073080 auth_method 0 2024-04-26T00:42:24.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f946ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:24.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f946ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:24.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f946ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:24.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f9474f4b640 1 --2- 172.21.15.37:0/4164519502 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9470131cd0 0x7f947013c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:24.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f9474f4b640 10 monclient(hunting): get_auth_request con 0x7f9470131cd0 auth_method 0 2024-04-26T00:42:24.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f9474f4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:24.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f9474f4b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:24.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f9474f4b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:24.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f946ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:24.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f946ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:24.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f946ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:24.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f9474f4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:24.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f9474f4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:24.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f9467fff640 1 --2- 172.21.15.37:0/4164519502 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94700a1b60 0x7f947013bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:24.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f9474f4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:24.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f9467fff640 10 monclient(hunting): get_auth_request con 0x7f94700a1b60 auth_method 0 2024-04-26T00:42:24.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f9467fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:24.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f9467fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:24.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f9467fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:24.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f9467fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:24.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f9467fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:24.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f9467fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:24.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f946ffff640 10 monclient(hunting): handle_auth_done global_id 254345 payload 824 2024-04-26T00:42:24.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f946ffff640 1 -- 172.21.15.37:0/4164519502 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94700a1b60 msgr2=0x7f947013bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:24.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f946ffff640 1 --2- 172.21.15.37:0/4164519502 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94700a1b60 0x7f947013bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:24.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f946ffff640 1 -- 172.21.15.37:0/4164519502 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9470131cd0 msgr2=0x7f947013c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:24.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f946ffff640 1 --2- 172.21.15.37:0/4164519502 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9470131cd0 0x7f947013c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:24.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f946ffff640 10 monclient: _finish_hunting 0 2024-04-26T00:42:24.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f946ffff640 1 monclient: found mon.noname-c 2024-04-26T00:42:24.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f946ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:42:24.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f946ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:42:24.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f946ffff640 1 -- 172.21.15.37:0/4164519502 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9470134230 con 0x7f9470073080 2024-04-26T00:42:24.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.884+0000 7f9474f4b640 1 --2- 172.21.15.37:0/4164519502 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9470131cd0 0x7f947013c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:42:24.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.888+0000 7f946ffff640 1 -- 172.21.15.37:0/4164519502 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9470073080 msgr2=0x7f94701cd0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:24.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.888+0000 7f946ffff640 1 --2- 172.21.15.37:0/4164519502 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9470073080 0x7f94701cd0f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f946001aa60 tx=0x7f9460019150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:24.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.888+0000 7f946d7fa640 1 -- 172.21.15.37:0/4164519502 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94600b2040 con 0x7f9470073080 2024-04-26T00:42:24.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.888+0000 7f946d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:24.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.888+0000 7f946d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:42:24.902 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.888+0000 7f946d7fa640 10 monclient: dump: 2024-04-26T00:42:24.902 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:24.902 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:24.902 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:24.902 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:24.902 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:24.902 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:24.903 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:24.903 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:24.903 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:24.903 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:24.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.888+0000 7f946d7fa640 10 monclient: _finish_auth 0 2024-04-26T00:42:24.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.888+0000 7f946d7fa640 10 monclient: _check_auth_tickets 2024-04-26T00:42:24.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.888+0000 7f946d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:42:24.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.888+0000 7f946d7fa640 1 -- 172.21.15.37:0/4164519502 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94600b6610 con 0x7f9470073080 2024-04-26T00:42:24.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.888+0000 7f94769ae640 5 monclient: authenticate success, global_id 254345 2024-04-26T00:42:24.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.888+0000 7f94769ae640 10 monclient: _renew_subs 2024-04-26T00:42:24.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.888+0000 7f94769ae640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:24.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.888+0000 7f94769ae640 1 -- 172.21.15.37:0/4164519502 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94701387c0 con 0x7f9470073080 2024-04-26T00:42:24.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.888+0000 7f946d7fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:42:24.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.888+0000 7f946d7fa640 1 -- 172.21.15.37:0/4164519502 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94600ade60 con 0x7f9470073080 2024-04-26T00:42:24.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.888+0000 7f946d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:24.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.888+0000 7f946d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:42:24.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.888+0000 7f946d7fa640 10 monclient: dump: 2024-04-26T00:42:24.904 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:24.904 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:24.904 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:24.904 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:24.904 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:24.904 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:24.904 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:24.904 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:24.904 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:24.904 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:24.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.888+0000 7f946effd640 4 set_mon_vals no callback set 2024-04-26T00:42:24.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.888+0000 7f94769ae640 10 monclient: _renew_subs 2024-04-26T00:42:24.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.888+0000 7f94769ae640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:24.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.888+0000 7f94769ae640 1 -- 172.21.15.37:0/4164519502 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f947013a210 con 0x7f9470073080 2024-04-26T00:42:24.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.888+0000 7f94769ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:24.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.888+0000 7f94769ae640 1 --2- 172.21.15.37:0/4164519502 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9470136b20 0x7f9470136f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:24.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.888+0000 7f94769ae640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:24.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.888+0000 7f94769ae640 1 -- 172.21.15.37:0/4164519502 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f94700708e0 con 0x7f9470136b20 2024-04-26T00:42:24.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.888+0000 7f946d7fa640 1 -- 172.21.15.37:0/4164519502 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f94600b6a40 con 0x7f9470073080 2024-04-26T00:42:24.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.888+0000 7f9467fff640 1 --2- 172.21.15.37:0/4164519502 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9470136b20 0x7f9470136f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:24.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.888+0000 7f9467fff640 10 monclient: get_auth_request con 0x7f9470136b20 auth_method 0 2024-04-26T00:42:24.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.888+0000 7f9467fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:24.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.888+0000 7f9467fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:24.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.888+0000 7f9467fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:24.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.888+0000 7f9467fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:24.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.888+0000 7f9467fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:24.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.888+0000 7f9467fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:24.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.888+0000 7f946d7fa640 1 --2- 172.21.15.37:0/4164519502 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9450035ed0 0x7f9450038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:24.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.888+0000 7f946d7fa640 1 -- 172.21.15.37:0/4164519502 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(896..896 src has 822..896) ==== 3975+0+0 (secure 0 0 0) 0x7f94600e28f0 con 0x7f9470073080 2024-04-26T00:42:24.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.888+0000 7f9467fff640 10 monclient(hunting): handle_auth_done global_id 264228 payload 824 2024-04-26T00:42:24.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.888+0000 7f9467fff640 1 -- 172.21.15.37:0/4164519502 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f94700a6280 con 0x7f9470136b20 2024-04-26T00:42:24.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.892+0000 7f9474f4b640 1 --2- 172.21.15.37:0/4164519502 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9450035ed0 0x7f9450038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:24.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.892+0000 7f9474f4b640 10 monclient: get_auth_request con 0x7f9450035ed0 auth_method 0 2024-04-26T00:42:24.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.892+0000 7f9467fff640 1 -- 172.21.15.37:0/4164519502 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9470136b20 msgr2=0x7f9470136f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:24.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.892+0000 7f9467fff640 1 --2- 172.21.15.37:0/4164519502 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9470136b20 0x7f9470136f00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f94701383c0 tx=0x7f9458011e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:24.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.892+0000 7f946d7fa640 1 -- 172.21.15.37:0/4164519502 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9458019070 con 0x7f9470136b20 2024-04-26T00:42:24.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.892+0000 7f946d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:24.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.892+0000 7f946d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:42:24.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.892+0000 7f946d7fa640 10 monclient: dump: 2024-04-26T00:42:24.906 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:24.906 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:24.906 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:24.906 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:24.907 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:24.907 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:24.907 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:24.907 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:24.907 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:24.907 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:24.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.892+0000 7f946d7fa640 1 -- 172.21.15.37:0/4164519502 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f945800f920 con 0x7f9470136b20 2024-04-26T00:42:24.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.892+0000 7f946d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:24.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.892+0000 7f946d7fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:42:24.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.892+0000 7f946d7fa640 1 -- 172.21.15.37:0/4164519502 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9470136b20 msgr2=0x7f9470136f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:24.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.892+0000 7f946d7fa640 1 --2- 172.21.15.37:0/4164519502 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9470136b20 0x7f9470136f00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f94701383c0 tx=0x7f9458011e30 comp rx=0 tx=0).stop 2024-04-26T00:42:24.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.892+0000 7f946d7fa640 1 -- 172.21.15.37:0/4164519502 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9470136b20 msgr2=0x7f9470136f00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:42:24.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.892+0000 7f946d7fa640 1 --2- 172.21.15.37:0/4164519502 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9470136b20 0x7f9470136f00 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:24.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.892+0000 7f9474f4b640 1 --2- 172.21.15.37:0/4164519502 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9450035ed0 0x7f9450038390 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f9468005c10 tx=0x7f9468009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:24.910 INFO:tasks.rados.rados.0.smithi045.stdout:3983: writing smithi04526579-22 from 3006308 to 3409603 tid 3 2024-04-26T00:42:24.910 INFO:tasks.rados.rados.0.smithi045.stdout:3979: done (4 left) 2024-04-26T00:42:24.910 INFO:tasks.rados.rados.0.smithi045.stdout:3984: copy_from oid 27 from oid 16 current snap is 440 2024-04-26T00:42:24.910 INFO:tasks.rados.rados.0.smithi045.stdout:3980: finishing write tid 2 to smithi04526579-1 2024-04-26T00:42:24.910 INFO:tasks.rados.rados.0.smithi045.stdout:3981: finishing rollback tid 1 to smithi04526579-49 2024-04-26T00:42:24.910 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 49 v 2875 (ObjNum 129 snap 0 seq_num 2241898608) dirty dne 2024-04-26T00:42:24.910 INFO:tasks.rados.rados.0.smithi045.stdout:3980: finishing write tid 3 to smithi04526579-1 2024-04-26T00:42:24.911 INFO:tasks.rados.rados.0.smithi045.stdout:3980: finishing write tid 4 to smithi04526579-1 2024-04-26T00:42:24.911 INFO:tasks.rados.rados.0.smithi045.stdout:3980: finishing write tid 5 to smithi04526579-1 2024-04-26T00:42:24.911 INFO:tasks.rados.rados.0.smithi045.stdout:3980: oid 1 updating version 0 to 2836 2024-04-26T00:42:24.911 INFO:tasks.rados.rados.0.smithi045.stdout:3980: oid 1 updating version 2836 to 2838 2024-04-26T00:42:24.911 INFO:tasks.rados.rados.0.smithi045.stdout:3980: oid 1 version 2838 is already newer than 2837 2024-04-26T00:42:24.911 INFO:tasks.rados.rados.0.smithi045.stdout:3980: oid 1 updating version 2838 to 2839 2024-04-26T00:42:24.911 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 1 v 2839 (ObjNum 1419 snap 440 seq_num 1419) dirty exists 2024-04-26T00:42:24.911 INFO:tasks.rados.rados.0.smithi045.stdout:3980: left oid 1 (ObjNum 1419 snap 440 seq_num 1419) 2024-04-26T00:42:24.911 INFO:tasks.rados.rados.0.smithi045.stdout:3982: finishing write tid 1 to smithi04526579-48 2024-04-26T00:42:24.911 INFO:tasks.rados.rados.0.smithi045.stdout:3982: finishing write tid 2 to smithi04526579-48 2024-04-26T00:42:24.911 INFO:tasks.rados.rados.0.smithi045.stdout:3980: done (4 left) 2024-04-26T00:42:24.911 INFO:tasks.rados.rados.0.smithi045.stdout:3981: done (3 left) 2024-04-26T00:42:24.911 INFO:tasks.rados.rados.0.smithi045.stdout:3985: delete oid 31 current snap is 440 2024-04-26T00:42:24.911 INFO:tasks.rados.rados.0.smithi045.stdout:3982: finishing write tid 3 to smithi04526579-48 2024-04-26T00:42:24.911 INFO:tasks.rados.rados.0.smithi045.stdout:3982: finishing write tid 4 to smithi04526579-48 2024-04-26T00:42:24.912 INFO:tasks.rados.rados.0.smithi045.stdout:3982: finishing write tid 5 to smithi04526579-48 2024-04-26T00:42:24.912 INFO:tasks.rados.rados.0.smithi045.stdout:3982: oid 48 updating version 0 to 2886 2024-04-26T00:42:24.912 INFO:tasks.rados.rados.0.smithi045.stdout:3982: oid 48 updating version 2886 to 2887 2024-04-26T00:42:24.912 INFO:tasks.rados.rados.0.smithi045.stdout:3982: oid 48 version 2887 is already newer than 2885 2024-04-26T00:42:24.912 INFO:tasks.rados.rados.0.smithi045.stdout:3982: oid 48 version 2887 is already newer than 2884 2024-04-26T00:42:24.912 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 2887 (ObjNum 1420 snap 440 seq_num 1420) dirty exists 2024-04-26T00:42:24.912 INFO:tasks.rados.rados.0.smithi045.stdout:3982: left oid 48 (ObjNum 1420 snap 440 seq_num 1420) 2024-04-26T00:42:24.912 INFO:tasks.rados.rados.0.smithi045.stdout:3983: finishing write tid 1 to smithi04526579-22 2024-04-26T00:42:24.912 INFO:tasks.rados.rados.0.smithi045.stdout:3983: finishing write tid 2 to smithi04526579-22 2024-04-26T00:42:24.912 INFO:tasks.rados.rados.0.smithi045.stdout:3982: done (3 left) 2024-04-26T00:42:24.912 INFO:tasks.rados.rados.0.smithi045.stdout:3985: done (2 left) 2024-04-26T00:42:24.912 INFO:tasks.rados.rados.0.smithi045.stdout:3986: rollback oid 48 current snap is 440 2024-04-26T00:42:24.912 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 48 to 421 2024-04-26T00:42:24.912 INFO:tasks.rados.rados.0.smithi045.stdout:3987: snap_remove snap 419 2024-04-26T00:42:24.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.900+0000 7f94769ae640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:42:24.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.900+0000 7f94769ae640 1 --2- 172.21.15.37:0/4164519502 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9470137440 0x7f947009b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:24.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.900+0000 7f94769ae640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:24.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.900+0000 7f94769ae640 1 -- 172.21.15.37:0/4164519502 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9470135a70 con 0x7f9470137440 2024-04-26T00:42:24.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.900+0000 7f9467fff640 1 --2- 172.21.15.37:0/4164519502 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9470137440 0x7f947009b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:24.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.900+0000 7f9467fff640 10 monclient: get_auth_request con 0x7f9470137440 auth_method 0 2024-04-26T00:42:24.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.900+0000 7f9467fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:24.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.900+0000 7f9467fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:24.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.900+0000 7f9467fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:24.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.900+0000 7f9467fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:24.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.900+0000 7f9467fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:24.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.900+0000 7f9467fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:24.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.904+0000 7f9467fff640 10 monclient(hunting): handle_auth_done global_id 264231 payload 824 2024-04-26T00:42:24.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.904+0000 7f9467fff640 1 -- 172.21.15.37:0/4164519502 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f947004f510 con 0x7f9470137440 2024-04-26T00:42:24.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.904+0000 7f9467fff640 1 -- 172.21.15.37:0/4164519502 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9470137440 msgr2=0x7f947009b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:24.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.904+0000 7f9467fff640 1 --2- 172.21.15.37:0/4164519502 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9470137440 0x7f947009b060 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f94701394f0 tx=0x7f94580046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:24.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.904+0000 7f946d7fa640 1 -- 172.21.15.37:0/4164519502 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9458019040 con 0x7f9470137440 2024-04-26T00:42:24.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.904+0000 7f946d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:24.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.904+0000 7f946d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:42:24.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.904+0000 7f946d7fa640 10 monclient: dump: 2024-04-26T00:42:24.914 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:24.914 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:24.914 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:24.915 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:24.915 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:24.915 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:24.915 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:24.915 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:24.915 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:24.915 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:24.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.904+0000 7f946d7fa640 1 -- 172.21.15.37:0/4164519502 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f945801e450 con 0x7f9470137440 2024-04-26T00:42:24.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.904+0000 7f946d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:42:24.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.904+0000 7f946d7fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:42:24.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.904+0000 7f946d7fa640 1 -- 172.21.15.37:0/4164519502 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9470137440 msgr2=0x7f947009b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:24.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.904+0000 7f946d7fa640 1 --2- 172.21.15.37:0/4164519502 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9470137440 0x7f947009b060 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f94701394f0 tx=0x7f94580046d0 comp rx=0 tx=0).stop 2024-04-26T00:42:24.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.904+0000 7f946d7fa640 1 -- 172.21.15.37:0/4164519502 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9470137440 msgr2=0x7f947009b060 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:42:24.915 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.904+0000 7f946d7fa640 1 --2- 172.21.15.37:0/4164519502 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9470137440 0x7f947009b060 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f94701394f0 tx=0x7f94580046d0 comp rx=0 tx=0).stop 2024-04-26T00:42:24.916 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:42:24.916 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:42:24.916 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:42:24.916 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:42:24.916 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 160, 2024-04-26T00:42:24.916 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 67195, 2024-04-26T00:42:24.916 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:42:24.916 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:42:24.916 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:42:24.916 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:42:24.916 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:42:24.917 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 6, 2024-04-26T00:42:24.917 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:42:24.917 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:42:24.917 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:42:24.917 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:42:24.917 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:42:24.917 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:42:24.917 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:42:24.917 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:42:24.917 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:42:24.917 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:42:24.917 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:42:24.917 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:42:24.917 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:42:24.918 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:42:24.918 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:42:24.918 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:42:24.918 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:42:24.918 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:42:24.918 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:42:24.918 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:42:24.918 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:42:24.918 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:42:24.918 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:42:24.918 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:42:24.918 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:42:24.918 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:42:24.918 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:42:24.918 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:42:24.919 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:24.919 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:24.919 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:42:24.919 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:42:24.919 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:42:24.919 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:42:24.919 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:42:24.919 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:42:24.919 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:42:24.919 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:42:24.919 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:42:24.919 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:42:24.919 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:42:24.919 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:42:24.919 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:42:24.920 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:42:24.920 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:42:24.920 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:42:24.920 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:42:24.920 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:42:24.920 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:42:24.920 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:42:24.920 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:42:24.920 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:42:24.920 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:42:24.920 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:42:24.920 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:42:24.921 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:42:24.921 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:42:24.921 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:42:24.921 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:42:24.921 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:42:24.921 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:24.921 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:42:24.921 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:24.921 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:42:24.921 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:42:24.921 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:42:24.921 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:42:24.921 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:24.921 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:42:24.922 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:42:24.922 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:24.922 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:24.922 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:24.922 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:42:24.922 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:42:24.922 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:24.922 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:24.922 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:24.922 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:24.922 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:42:24.922 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:42:24.922 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:42:24.922 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:42:24.922 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:42:24.923 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:24.923 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:24.923 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:42:24.923 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:42:24.923 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:42:24.923 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:42:24.923 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:24.923 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:42:24.923 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:42:24.923 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:24.923 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:24.923 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:24.923 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:42:24.923 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:42:24.923 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:24.924 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:24.924 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:24.924 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:24.924 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:42:24.924 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:42:24.924 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:42:24.924 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:42:24.924 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:42:24.924 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:24.924 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:24.924 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:42:24.924 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:42:24.924 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:42:24.924 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:42:24.924 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:24.925 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:42:24.925 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:42:24.925 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:24.925 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:24.925 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:24.925 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:42:24.925 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:42:24.925 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:24.925 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:24.925 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:24.925 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:24.925 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:42:24.925 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:42:24.925 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:42:24.925 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:42:24.925 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:42:24.926 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:24.926 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:24.926 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:24.926 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:42:24.926 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:42:24.926 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:24.926 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:42:24.926 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:42:24.926 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:42:24.926 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:24.926 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:42:24.926 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:42:24.926 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:24.926 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:42:24.927 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:42:24.927 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:42:24.927 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:24.927 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:42:24.927 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:42:24.927 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:24.927 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:42:24.927 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:42:24.927 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:42:24.927 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:24.927 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:42:24.927 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:42:24.927 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:24.927 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:42:24.927 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:42:24.928 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:42:24.928 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:24.928 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:24.928 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:24.928 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:42:24.928 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:42:24.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.904+0000 7f94769ae640 1 -- 172.21.15.37:0/4164519502 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9450035ed0 msgr2=0x7f9450038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:24.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.904+0000 7f94769ae640 1 --2- 172.21.15.37:0/4164519502 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9450035ed0 0x7f9450038390 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f9468005c10 tx=0x7f9468009040 comp rx=0 tx=0).stop 2024-04-26T00:42:24.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.904+0000 7f94769ae640 10 monclient: shutdown 2024-04-26T00:42:24.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.904+0000 7f94769ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:42:24.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.904+0000 7f94769ae640 1 -- 172.21.15.37:0/4164519502 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9470073080 msgr2=0x7f94701cd0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:24.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.904+0000 7f94769ae640 1 --2- 172.21.15.37:0/4164519502 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9470073080 0x7f94701cd0f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f946001aa60 tx=0x7f9460019150 comp rx=0 tx=0).stop 2024-04-26T00:42:24.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.904+0000 7f9474f4b640 1 -- 172.21.15.37:0/4164519502 reap_dead start 2024-04-26T00:42:24.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.908+0000 7f94769ae640 1 -- 172.21.15.37:0/4164519502 shutdown_connections 2024-04-26T00:42:24.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.908+0000 7f94769ae640 1 --2- 172.21.15.37:0/4164519502 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9470073080 0x7f94701cd0f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:24.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.908+0000 7f94769ae640 1 -- 172.21.15.37:0/4164519502 >> 172.21.15.37:0/4164519502 conn(0x7f94700a7160 msgr2=0x7f9470133b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:42:24.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.908+0000 7f94769ae640 1 -- 172.21.15.37:0/4164519502 shutdown_connections 2024-04-26T00:42:24.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:24.908+0000 7f94769ae640 1 -- 172.21.15.37:0/4164519502 wait complete. 2024-04-26T00:42:24.930 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:42:24.930 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:42:24.931 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:42:25.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.040+0000 7f2675288640 10 monclient: get_monmap_and_config 2024-04-26T00:42:25.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.040+0000 7f2675288640 10 monclient: build_initial_monmap 2024-04-26T00:42:25.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.040+0000 7f2675288640 10 monclient: monmap: 2024-04-26T00:42:25.046 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:42:25.046 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:25.046 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:42:25.047266+0000 2024-04-26T00:42:25.046 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:42:25.047266+0000 2024-04-26T00:42:25.046 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:42:25.046 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:25.046 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:42:25.046 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:42:25.046 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:42:25.046 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:42:25.046 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:42:25.046 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:25.048 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f2675288640 1 Processor -- start 2024-04-26T00:42:25.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f2675288640 1 -- start start 2024-04-26T00:42:25.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f2675288640 10 monclient: init 2024-04-26T00:42:25.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f2675288640 10 monclient: _reopen_session rank -1 2024-04-26T00:42:25.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f2675288640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-04-26T00:42:25.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f2675288640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f26700a0300 addr v1:172.21.15.45:6789/0 2024-04-26T00:42:25.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f2675288640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f267009fc50 0x7f2670139ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:25.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f2675288640 10 monclient(hunting): picked mon.noname-b con 0x7f267009fc50 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:42:25.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f2675288640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f267013a400 0x7f267013c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:25.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f2675288640 10 monclient(hunting): picked mon.noname-c con 0x7f267013a400 addr v2:172.21.15.45:3300/0 2024-04-26T00:42:25.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f2675288640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f267009c190 con 0x7f26700a0300 2024-04-26T00:42:25.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f2675288640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:25.049 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f2675288640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f26700a1110 con 0x7f267013a400 2024-04-26T00:42:25.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f2675288640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:25.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f2675288640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f26700a16d0 con 0x7f267009fc50 2024-04-26T00:42:25.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f2675288640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:25.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f2675288640 10 monclient(hunting): authenticate will time out at 2326.353185s 2024-04-26T00:42:25.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f266e7fc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f267009fc50 0x7f2670139ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:25.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f266e7fc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f267009fc50 0x7f2670139ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51134/0 (socket says 172.21.15.37:51134) 2024-04-26T00:42:25.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f266e7fc640 1 -- 172.21.15.37:0/2043333730 learned_addr learned my addr 172.21.15.37:0/2043333730 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:42:25.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f266e7fc640 10 monclient(hunting): get_auth_request con 0x7f267009fc50 auth_method 0 2024-04-26T00:42:25.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f266e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:25.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f266e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:25.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f266e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:25.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f266e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:25.050 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f266e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:25.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f266e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:25.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f266f7fe640 1 --2- 172.21.15.37:0/2043333730 >> v2:172.21.15.45:3300/0 conn(0x7f267013a400 0x7f267013c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:25.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f266f7fe640 10 monclient(hunting): get_auth_request con 0x7f267013a400 auth_method 0 2024-04-26T00:42:25.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f266f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:25.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f266f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:25.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f266f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:25.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f266e7fc640 10 monclient(hunting): handle_auth_done global_id 254348 payload 274 2024-04-26T00:42:25.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f266e7fc640 1 -- 172.21.15.37:0/2043333730 >> v2:172.21.15.45:3300/0 conn(0x7f267013a400 msgr2=0x7f267013c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:42:25.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f266e7fc640 1 --2- 172.21.15.37:0/2043333730 >> v2:172.21.15.45:3300/0 conn(0x7f267013a400 0x7f267013c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:25.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f266e7fc640 1 -- 172.21.15.37:0/2043333730 >> v1:172.21.15.45:6789/0 conn(0x7f26700a0300 legacy=0x7f26700a06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:42:25.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f266e7fc640 10 monclient: _finish_hunting 0 2024-04-26T00:42:25.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f266e7fc640 1 monclient: found mon.noname-b 2024-04-26T00:42:25.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f266e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:42:25.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f266e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:42:25.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.044+0000 7f266e7fc640 1 -- 172.21.15.37:0/2043333730 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f267009f2a0 con 0x7f267009fc50 2024-04-26T00:42:25.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.048+0000 7f266e7fc640 1 -- 172.21.15.37:0/2043333730 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f267009fc50 msgr2=0x7f2670139ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:25.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.048+0000 7f266e7fc640 1 --2- 172.21.15.37:0/2043333730 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f267009fc50 0x7f2670139ec0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f267009f4d0 tx=0x7f266401c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=384a079e274d0c5b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:25.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.048+0000 7f266dffb640 1 -- 172.21.15.37:0/2043333730 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26640b0070 con 0x7f267009fc50 2024-04-26T00:42:25.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.048+0000 7f266dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:25.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.048+0000 7f266dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:42:25.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.048+0000 7f266dffb640 10 monclient: dump: 2024-04-26T00:42:25.054 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:25.054 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:25.054 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:25.054 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:25.054 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:25.054 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:25.054 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:25.054 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:25.054 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:25.054 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:25.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.048+0000 7f266dffb640 10 monclient: _finish_auth 0 2024-04-26T00:42:25.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.048+0000 7f266dffb640 10 monclient: _check_auth_tickets 2024-04-26T00:42:25.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.048+0000 7f266dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:42:25.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.048+0000 7f266dffb640 1 -- 172.21.15.37:0/2043333730 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26640ab600 con 0x7f267009fc50 2024-04-26T00:42:25.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.048+0000 7f266dffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:42:25.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.048+0000 7f266dffb640 1 -- 172.21.15.37:0/2043333730 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26640ab9d0 con 0x7f267009fc50 2024-04-26T00:42:25.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.048+0000 7f2675288640 5 monclient: authenticate success, global_id 254348 2024-04-26T00:42:25.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.048+0000 7f2675288640 10 monclient: get_monmap_and_config success 2024-04-26T00:42:25.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.048+0000 7f2675288640 4 set_mon_vals no callback set 2024-04-26T00:42:25.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.048+0000 7f266dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:25.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.048+0000 7f266dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:42:25.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.048+0000 7f266dffb640 10 monclient: dump: 2024-04-26T00:42:25.055 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:25.055 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:25.055 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:25.055 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:25.055 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:25.056 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:25.056 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:25.056 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:25.056 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:25.056 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:25.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.048+0000 7f2675288640 10 monclient: shutdown 2024-04-26T00:42:25.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.048+0000 7f2675288640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:42:25.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.048+0000 7f2675288640 1 -- 172.21.15.37:0/2043333730 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f267009fc50 msgr2=0x7f2670139ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:25.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.048+0000 7f2675288640 1 --2- 172.21.15.37:0/2043333730 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f267009fc50 0x7f2670139ec0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f267009f4d0 tx=0x7f266401c3c0 comp rx=0 tx=0).stop 2024-04-26T00:42:25.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.048+0000 7f2675288640 1 -- 172.21.15.37:0/2043333730 shutdown_connections 2024-04-26T00:42:25.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.048+0000 7f2675288640 1 --2- 172.21.15.37:0/2043333730 >> v2:172.21.15.45:3300/0 conn(0x7f267013a400 0x7f267013c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:25.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.048+0000 7f2675288640 1 --2- 172.21.15.37:0/2043333730 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f267009fc50 0x7f2670139ec0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:25.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.048+0000 7f2675288640 1 -- 172.21.15.37:0/2043333730 >> 172.21.15.37:0/2043333730 conn(0x7f26700a1d50 msgr2=0x7f26700a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:42:25.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.048+0000 7f2675288640 1 -- 172.21.15.37:0/2043333730 shutdown_connections 2024-04-26T00:42:25.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.048+0000 7f2675288640 1 -- 172.21.15.37:0/2043333730 wait complete. 2024-04-26T00:42:25.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.048+0000 7f2675288640 10 monclient: build_initial_monmap 2024-04-26T00:42:25.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.048+0000 7f2675288640 10 monclient: monmap: 2024-04-26T00:42:25.057 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:42:25.057 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:42:25.057 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:42:25.057 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:42:25.057 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:42:25.057 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:42:25.057 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:42:25.057 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:42:25.057 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:42:25.057 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:25.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f2675288640 1 Processor -- start 2024-04-26T00:42:25.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f2675288640 1 -- start start 2024-04-26T00:42:25.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f2675288640 10 monclient: init 2024-04-26T00:42:25.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f2675288640 10 monclient: _reopen_session rank -1 2024-04-26T00:42:25.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f2675288640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:42:25.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f2675288640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f267009fc50 0x7f2670132200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:25.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f2675288640 10 monclient(hunting): picked mon.noname-b con 0x7f267009fc50 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:42:25.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f2675288640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26700a0300 0x7f267009c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:25.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f2675288640 10 monclient(hunting): picked mon.noname-a con 0x7f26700a0300 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:42:25.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f2675288640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f267013a400 0x7f267009ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:25.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f2675288640 10 monclient(hunting): picked mon.noname-c con 0x7f267013a400 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:42:25.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f2675288640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:25.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f2675288640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f267013f8e0 con 0x7f26700a0300 2024-04-26T00:42:25.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f2675288640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:25.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f2675288640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f267013f760 con 0x7f267009fc50 2024-04-26T00:42:25.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f2675288640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:25.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f2675288640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f267013fa60 con 0x7f267013a400 2024-04-26T00:42:25.063 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f2675288640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:25.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f2675288640 10 monclient(hunting): authenticate will time out at 2326.360665s 2024-04-26T00:42:25.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f266f7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f267013a400 0x7f267009ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:25.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f266e7fc640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26700a0300 0x7f267009c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:25.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f266f7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f267013a400 0x7f267009ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51138/0 (socket says 172.21.15.37:51138) 2024-04-26T00:42:25.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f266f7fe640 1 -- 172.21.15.37:0/3978376271 learned_addr learned my addr 172.21.15.37:0/3978376271 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:42:25.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f266f7fe640 10 monclient(hunting): get_auth_request con 0x7f267013a400 auth_method 0 2024-04-26T00:42:25.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f266f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:25.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f266f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:25.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f266f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:25.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f266e7fc640 10 monclient(hunting): get_auth_request con 0x7f26700a0300 auth_method 0 2024-04-26T00:42:25.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f266e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:25.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f266e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:25.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f266e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:25.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f266f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:25.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f266f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:25.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f266f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:25.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f266e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:25.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f266e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:25.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f266e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:25.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f266effd640 1 --2- 172.21.15.37:0/3978376271 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f267009fc50 0x7f2670132200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:25.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f266effd640 10 monclient(hunting): get_auth_request con 0x7f267009fc50 auth_method 0 2024-04-26T00:42:25.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f266effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:25.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f266effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:25.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f266effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:25.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f266f7fe640 10 monclient(hunting): handle_auth_done global_id 254351 payload 824 2024-04-26T00:42:25.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f266f7fe640 1 -- 172.21.15.37:0/3978376271 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f267009fc50 msgr2=0x7f2670132200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:25.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f266f7fe640 1 --2- 172.21.15.37:0/3978376271 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f267009fc50 0x7f2670132200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:25.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f266f7fe640 1 -- 172.21.15.37:0/3978376271 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26700a0300 msgr2=0x7f267009c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:25.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f266f7fe640 1 --2- 172.21.15.37:0/3978376271 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26700a0300 0x7f267009c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:25.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f266f7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:42:25.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f266f7fe640 1 monclient: found mon.noname-c 2024-04-26T00:42:25.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f266f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:42:25.066 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f266f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:42:25.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f266f7fe640 1 -- 172.21.15.37:0/3978376271 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2670070de0 con 0x7f267013a400 2024-04-26T00:42:25.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f266f7fe640 1 -- 172.21.15.37:0/3978376271 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f267013a400 msgr2=0x7f267009ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:25.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f266f7fe640 1 --2- 172.21.15.37:0/3978376271 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f267013a400 0x7f267009ccc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2668007de0 tx=0x7f266800e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:25.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f26577fe640 1 -- 172.21.15.37:0/3978376271 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2668013790 con 0x7f267013a400 2024-04-26T00:42:25.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f26577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:25.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f26577fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:42:25.067 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f26577fe640 10 monclient: dump: 2024-04-26T00:42:25.067 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:25.067 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:25.067 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:25.067 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:25.067 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:25.067 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:25.067 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:25.067 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:25.068 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:25.068 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:25.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f26577fe640 10 monclient: _finish_auth 0 2024-04-26T00:42:25.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f26577fe640 10 monclient: _check_auth_tickets 2024-04-26T00:42:25.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f26577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:42:25.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f26577fe640 1 -- 172.21.15.37:0/3978376271 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2668019070 con 0x7f267013a400 2024-04-26T00:42:25.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f2675288640 5 monclient: authenticate success, global_id 254351 2024-04-26T00:42:25.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f2675288640 10 monclient: _renew_subs 2024-04-26T00:42:25.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f2675288640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:25.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f2675288640 1 -- 172.21.15.37:0/3978376271 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f267006eb50 con 0x7f267013a400 2024-04-26T00:42:25.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f26577fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:42:25.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f26577fe640 1 -- 172.21.15.37:0/3978376271 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2668013b90 con 0x7f267013a400 2024-04-26T00:42:25.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f26577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:25.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f26577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:42:25.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f26577fe640 10 monclient: dump: 2024-04-26T00:42:25.069 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:25.069 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:25.069 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:25.069 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:25.069 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:25.069 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:25.069 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:25.069 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:25.069 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:25.069 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:25.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f266d7fa640 4 set_mon_vals no callback set 2024-04-26T00:42:25.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f2675288640 10 monclient: _renew_subs 2024-04-26T00:42:25.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f2675288640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:25.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.052+0000 7f2675288640 1 -- 172.21.15.37:0/3978376271 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2670071930 con 0x7f267013a400 2024-04-26T00:42:25.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.056+0000 7f2675288640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:25.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.056+0000 7f2675288640 1 --2- 172.21.15.37:0/3978376271 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f267006f230 0x7f267009d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:25.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.056+0000 7f2675288640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:25.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.056+0000 7f2675288640 1 -- 172.21.15.37:0/3978376271 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f26700705b0 con 0x7f267006f230 2024-04-26T00:42:25.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.056+0000 7f266effd640 1 --2- 172.21.15.37:0/3978376271 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f267006f230 0x7f267009d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:25.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.056+0000 7f266effd640 10 monclient: get_auth_request con 0x7f267006f230 auth_method 0 2024-04-26T00:42:25.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.056+0000 7f266effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:25.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.056+0000 7f266effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:25.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.056+0000 7f266effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:25.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.056+0000 7f26577fe640 1 -- 172.21.15.37:0/3978376271 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f266801e460 con 0x7f267013a400 2024-04-26T00:42:25.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.056+0000 7f266effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:25.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.056+0000 7f266effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:25.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.056+0000 7f266effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:25.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.056+0000 7f266effd640 10 monclient(hunting): handle_auth_done global_id 254354 payload 824 2024-04-26T00:42:25.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.056+0000 7f26577fe640 1 --2- 172.21.15.37:0/3978376271 >> v2:172.21.15.45:6812/2379918623 conn(0x7f265c035f00 0x7f265c0383c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:25.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.056+0000 7f266effd640 1 -- 172.21.15.37:0/3978376271 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f267004f510 con 0x7f267006f230 2024-04-26T00:42:25.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.056+0000 7f26577fe640 1 -- 172.21.15.37:0/3978376271 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(896..896 src has 822..896) ==== 3975+0+0 (secure 0 0 0) 0x7f26680493f0 con 0x7f267013a400 2024-04-26T00:42:25.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.056+0000 7f266e7fc640 1 --2- 172.21.15.37:0/3978376271 >> v2:172.21.15.45:6812/2379918623 conn(0x7f265c035f00 0x7f265c0383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:25.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.056+0000 7f266e7fc640 10 monclient: get_auth_request con 0x7f265c035f00 auth_method 0 2024-04-26T00:42:25.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.056+0000 7f266effd640 1 -- 172.21.15.37:0/3978376271 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f267006f230 msgr2=0x7f267009d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:25.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.056+0000 7f266effd640 1 --2- 172.21.15.37:0/3978376271 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f267006f230 0x7f267009d200 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f266000e450 tx=0x7f266000e910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:25.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.056+0000 7f26577fe640 1 -- 172.21.15.37:0/3978376271 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2660007450 con 0x7f267006f230 2024-04-26T00:42:25.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.056+0000 7f26577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:25.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.056+0000 7f26577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:42:25.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.056+0000 7f26577fe640 10 monclient: dump: 2024-04-26T00:42:25.071 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:25.071 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:25.071 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:25.071 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:25.072 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:25.072 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:25.072 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:25.072 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:25.072 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:25.072 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:25.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.056+0000 7f26577fe640 1 -- 172.21.15.37:0/3978376271 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2660007450 con 0x7f267006f230 2024-04-26T00:42:25.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.056+0000 7f26577fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:25.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.056+0000 7f26577fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:42:25.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.056+0000 7f26577fe640 1 -- 172.21.15.37:0/3978376271 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f267006f230 msgr2=0x7f267009d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:25.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.056+0000 7f26577fe640 1 --2- 172.21.15.37:0/3978376271 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f267006f230 0x7f267009d200 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f266000e450 tx=0x7f266000e910 comp rx=0 tx=0).stop 2024-04-26T00:42:25.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.056+0000 7f26577fe640 1 -- 172.21.15.37:0/3978376271 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f267006f230 msgr2=0x7f267009d200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:42:25.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.056+0000 7f26577fe640 1 --2- 172.21.15.37:0/3978376271 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f267006f230 0x7f267009d200 secure :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f266000e450 tx=0x7f266000e910 comp rx=0 tx=0).stop 2024-04-26T00:42:25.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.060+0000 7f266e7fc640 1 --2- 172.21.15.37:0/3978376271 >> v2:172.21.15.45:6812/2379918623 conn(0x7f265c035f00 0x7f265c0383c0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f26640ab000 tx=0x7f26640a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:25.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.064+0000 7f2675288640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:42:25.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.064+0000 7f2675288640 1 --2- 172.21.15.37:0/3978376271 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2670135c90 0x7f26701360a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:25.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.064+0000 7f2675288640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:25.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.064+0000 7f2675288640 1 -- 172.21.15.37:0/3978376271 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f267009eb60 con 0x7f2670135c90 2024-04-26T00:42:25.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.064+0000 7f266effd640 1 --2- 172.21.15.37:0/3978376271 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2670135c90 0x7f26701360a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:25.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.064+0000 7f266effd640 10 monclient: get_auth_request con 0x7f2670135c90 auth_method 0 2024-04-26T00:42:25.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.064+0000 7f266effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:25.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.064+0000 7f266effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:25.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.064+0000 7f266effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:25.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.064+0000 7f266effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:25.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.064+0000 7f266effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:25.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.064+0000 7f266effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:25.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.064+0000 7f266effd640 10 monclient(hunting): handle_auth_done global_id 254357 payload 824 2024-04-26T00:42:25.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.064+0000 7f266effd640 1 -- 172.21.15.37:0/3978376271 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f267006ba50 con 0x7f2670135c90 2024-04-26T00:42:25.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.068+0000 7f266effd640 1 -- 172.21.15.37:0/3978376271 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2670135c90 msgr2=0x7f26701360a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:25.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.068+0000 7f266effd640 1 --2- 172.21.15.37:0/3978376271 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2670135c90 0x7f26701360a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f266000e450 tx=0x7f2660003770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:25.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.068+0000 7f26577fe640 1 -- 172.21.15.37:0/3978376271 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f266001e960 con 0x7f2670135c90 2024-04-26T00:42:25.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.068+0000 7f26577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:25.075 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.068+0000 7f26577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:42:25.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.068+0000 7f26577fe640 10 monclient: dump: 2024-04-26T00:42:25.076 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:25.076 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:25.076 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:25.076 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:25.076 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:25.076 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:25.076 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:25.076 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:25.076 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:25.076 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:25.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.068+0000 7f26577fe640 1 -- 172.21.15.37:0/3978376271 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7f2660028460 con 0x7f2670135c90 2024-04-26T00:42:25.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.068+0000 7f26577fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:42:25.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.068+0000 7f26577fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:42:25.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.068+0000 7f26577fe640 1 -- 172.21.15.37:0/3978376271 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2670135c90 msgr2=0x7f26701360a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:25.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.068+0000 7f26577fe640 1 --2- 172.21.15.37:0/3978376271 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2670135c90 0x7f26701360a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f266000e450 tx=0x7f2660003770 comp rx=0 tx=0).stop 2024-04-26T00:42:25.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.068+0000 7f26577fe640 1 -- 172.21.15.37:0/3978376271 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2670135c90 msgr2=0x7f26701360a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:42:25.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.068+0000 7f26577fe640 1 --2- 172.21.15.37:0/3978376271 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2670135c90 0x7f26701360a0 secure :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f266000e450 tx=0x7f2660003770 comp rx=0 tx=0).stop 2024-04-26T00:42:25.077 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:42:25.077 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:42:25.077 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:42:25.077 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:42:25.077 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 160, 2024-04-26T00:42:25.077 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 113838, 2024-04-26T00:42:25.077 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:42:25.078 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:42:25.078 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:42:25.078 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:42:25.078 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:42:25.078 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 6, 2024-04-26T00:42:25.078 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:42:25.078 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:42:25.078 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:42:25.078 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:42:25.078 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:42:25.078 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:42:25.078 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:42:25.078 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:42:25.079 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:42:25.079 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:42:25.079 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:42:25.079 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:42:25.079 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:42:25.079 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:42:25.079 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:42:25.079 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:42:25.079 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:42:25.079 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:42:25.079 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:42:25.079 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:42:25.079 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:42:25.079 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:42:25.079 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:42:25.080 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:42:25.080 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:42:25.080 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:42:25.080 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:42:25.080 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:42:25.080 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:25.080 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:25.080 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:42:25.080 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:42:25.080 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:42:25.080 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:42:25.080 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:42:25.080 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:42:25.080 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:42:25.080 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:42:25.081 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:42:25.081 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:42:25.081 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:42:25.081 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:42:25.081 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:42:25.081 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:42:25.081 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:42:25.081 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:42:25.081 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:42:25.081 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:42:25.081 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:42:25.081 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:42:25.081 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:42:25.081 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:42:25.082 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:42:25.082 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:42:25.082 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:42:25.082 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:42:25.082 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:42:25.082 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:42:25.082 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:42:25.082 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:42:25.082 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:25.082 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:42:25.082 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:25.082 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:42:25.082 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:42:25.082 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:42:25.083 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:42:25.083 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:25.083 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:42:25.083 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:42:25.083 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:25.083 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:25.083 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:25.083 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:42:25.083 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:42:25.083 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:25.083 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:25.083 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:25.083 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:25.083 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:42:25.083 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:42:25.084 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:42:25.084 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:42:25.084 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:42:25.084 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:25.084 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:25.084 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:42:25.084 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:42:25.084 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:42:25.084 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:42:25.084 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:25.084 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:42:25.084 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:42:25.084 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:25.084 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:25.084 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:25.085 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:42:25.085 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:42:25.085 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:25.085 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:25.085 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:25.085 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:25.085 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:42:25.085 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:42:25.085 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:42:25.085 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:42:25.085 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:42:25.085 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:25.085 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:25.085 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:42:25.086 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:42:25.086 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:42:25.086 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:42:25.086 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:25.086 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:42:25.086 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:42:25.086 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:25.086 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:25.086 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:25.086 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:42:25.086 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:42:25.086 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:25.086 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:25.086 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:25.087 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:25.087 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:42:25.087 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:42:25.087 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:42:25.087 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:42:25.087 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:42:25.087 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:25.087 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:25.087 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:25.087 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:42:25.087 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:42:25.087 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:25.087 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:42:25.087 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:42:25.087 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:42:25.088 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:25.088 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:42:25.088 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:42:25.088 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:25.088 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:42:25.088 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:42:25.088 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:42:25.088 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:25.088 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:42:25.088 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:42:25.088 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:25.088 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:42:25.088 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:42:25.088 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 4 2024-04-26T00:42:25.089 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:25.089 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:25.089 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:25.089 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:42:25.089 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:42:25.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.068+0000 7f2675288640 1 -- 172.21.15.37:0/3978376271 >> v2:172.21.15.45:6812/2379918623 conn(0x7f265c035f00 msgr2=0x7f265c0383c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:25.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.068+0000 7f2675288640 1 --2- 172.21.15.37:0/3978376271 >> v2:172.21.15.45:6812/2379918623 conn(0x7f265c035f00 0x7f265c0383c0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f26640ab000 tx=0x7f26640a6000 comp rx=0 tx=0).stop 2024-04-26T00:42:25.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.068+0000 7f2675288640 10 monclient: shutdown 2024-04-26T00:42:25.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.068+0000 7f2675288640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:42:25.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.068+0000 7f2675288640 1 -- 172.21.15.37:0/3978376271 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f267013a400 msgr2=0x7f267009ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:25.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.068+0000 7f2675288640 1 --2- 172.21.15.37:0/3978376271 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f267013a400 0x7f267009ccc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2668007de0 tx=0x7f266800e4b0 comp rx=0 tx=0).stop 2024-04-26T00:42:25.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.068+0000 7f266f7fe640 1 -- 172.21.15.37:0/3978376271 reap_dead start 2024-04-26T00:42:25.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.068+0000 7f266f7fe640 1 -- 172.21.15.37:0/3978376271 reap_dead start 2024-04-26T00:42:25.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.068+0000 7f2675288640 1 -- 172.21.15.37:0/3978376271 shutdown_connections 2024-04-26T00:42:25.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.068+0000 7f2675288640 1 -- 172.21.15.37:0/3978376271 >> 172.21.15.37:0/3978376271 conn(0x7f26700a1d50 msgr2=0x7f2670134c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:42:25.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.068+0000 7f2675288640 1 -- 172.21.15.37:0/3978376271 shutdown_connections 2024-04-26T00:42:25.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.068+0000 7f2675288640 1 -- 172.21.15.37:0/3978376271 wait complete. 2024-04-26T00:42:25.091 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:42:25.092 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:42:25.092 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:42:25.182 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.180+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.182 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.180+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2619628291}) 2024-04-26T00:42:25.182 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.180+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=762947543}) 2024-04-26T00:42:25.183 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.180+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.183 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.180+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2619628291}) 2024-04-26T00:42:25.183 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.180+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3978869364}) 2024-04-26T00:42:25.185 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.180+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.185 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.180+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=338198788}) 2024-04-26T00:42:25.185 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.180+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2170295061}) 2024-04-26T00:42:25.185 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.180+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.185 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.180+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=338198788}) 2024-04-26T00:42:25.185 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.180+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=148089355}) 2024-04-26T00:42:25.187 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.184+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.187 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.184+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=823328857}) 2024-04-26T00:42:25.187 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.184+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2997149611}) 2024-04-26T00:42:25.187 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.184+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.187 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.184+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=823328857}) 2024-04-26T00:42:25.187 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.184+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4115401412}) 2024-04-26T00:42:25.189 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.184+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.189 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.184+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2236018961}) 2024-04-26T00:42:25.189 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.184+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1438149554}) 2024-04-26T00:42:25.189 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.184+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.189 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.184+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2236018961}) 2024-04-26T00:42:25.189 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.184+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3422020383}) 2024-04-26T00:42:25.191 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.188+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.191 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.188+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4242384466}) 2024-04-26T00:42:25.191 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.188+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4266715901}) 2024-04-26T00:42:25.191 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.188+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.191 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.188+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4242384466}) 2024-04-26T00:42:25.191 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.188+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2878003009}) 2024-04-26T00:42:25.193 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.188+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.193 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.188+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2437656577}) 2024-04-26T00:42:25.193 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.188+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=922914196}) 2024-04-26T00:42:25.194 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.188+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.194 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.188+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2437656577}) 2024-04-26T00:42:25.194 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.188+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=30046948}) 2024-04-26T00:42:25.195 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.192+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.195 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.192+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1822777409}) 2024-04-26T00:42:25.195 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.192+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4145298678}) 2024-04-26T00:42:25.196 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.192+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.196 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.192+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1822777409}) 2024-04-26T00:42:25.196 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.192+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3980794026}) 2024-04-26T00:42:25.198 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.192+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.198 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.192+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1177522353}) 2024-04-26T00:42:25.198 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.192+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4190651445}) 2024-04-26T00:42:25.198 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.192+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.198 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.192+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1177522353}) 2024-04-26T00:42:25.198 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.196+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1737525084}) 2024-04-26T00:42:25.200 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.196+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.200 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.196+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=756710150}) 2024-04-26T00:42:25.200 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.196+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=188948145}) 2024-04-26T00:42:25.200 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.196+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.200 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.196+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=756710150}) 2024-04-26T00:42:25.200 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.196+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3570874418}) 2024-04-26T00:42:25.202 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.200+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.202 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.200+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1838043457}) 2024-04-26T00:42:25.203 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.200+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=524412498}) 2024-04-26T00:42:25.203 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.200+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.203 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.200+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1838043457}) 2024-04-26T00:42:25.203 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.200+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2856294905}) 2024-04-26T00:42:25.204 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.200+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.205 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.200+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3001266566}) 2024-04-26T00:42:25.205 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.200+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2304560803}) 2024-04-26T00:42:25.205 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.200+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.205 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.200+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3001266566}) 2024-04-26T00:42:25.205 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.200+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1542388628}) 2024-04-26T00:42:25.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fa9c37ee640 10 monclient: get_monmap_and_config 2024-04-26T00:42:25.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fa9c37ee640 10 monclient: build_initial_monmap 2024-04-26T00:42:25.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fa9c37ee640 10 monclient: monmap: 2024-04-26T00:42:25.206 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:42:25.206 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:25.206 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:42:25.208138+0000 2024-04-26T00:42:25.206 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:42:25.208138+0000 2024-04-26T00:42:25.206 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:42:25.206 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:25.206 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:42:25.207 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:42:25.207 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:42:25.207 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:42:25.207 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:42:25.207 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:25.208 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.208 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2769175659}) 2024-04-26T00:42:25.208 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1900662231}) 2024-04-26T00:42:25.208 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.208 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2769175659}) 2024-04-26T00:42:25.208 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3369940501}) 2024-04-26T00:42:25.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fa9c37ee640 1 Processor -- start 2024-04-26T00:42:25.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fa9c37ee640 1 -- start start 2024-04-26T00:42:25.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fa9c37ee640 10 monclient: init 2024-04-26T00:42:25.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fa9c37ee640 10 monclient: _reopen_session rank -1 2024-04-26T00:42:25.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fa9c37ee640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-04-26T00:42:25.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fa9c37ee640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa9bc0a2fe0 addr v1:172.21.15.45:6789/0 2024-04-26T00:42:25.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fa9c37ee640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa9bc071da0 addr v1:172.21.15.37:6789/0 2024-04-26T00:42:25.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fa9c37ee640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa9bc13a1d0 0x7fa9bc13c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:25.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fa9c37ee640 10 monclient(hunting): picked mon.noname-a con 0x7fa9bc13a1d0 addr v2:172.21.15.37:3300/0 2024-04-26T00:42:25.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fa9c37ee640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9bc0a21f0 con 0x7fa9bc071da0 2024-04-26T00:42:25.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fa9c37ee640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9bc072d70 con 0x7fa9bc0a2fe0 2024-04-26T00:42:25.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fa9c37ee640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:25.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fa9c37ee640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fa9bc071100 con 0x7fa9bc13a1d0 2024-04-26T00:42:25.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fa9c37ee640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:25.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fa9c37ee640 10 monclient(hunting): authenticate will time out at 2326.513968s 2024-04-26T00:42:25.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fa9c1d8b640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa9bc13a1d0 0x7fa9bc13c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:25.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fa9c0d89640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fa9bc071da0 0x7fa9bc0a1ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:44944/0 (socket says 172.21.15.37:44944) 2024-04-26T00:42:25.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fa9c0d89640 1 -- 172.21.15.37:0/4218158339 learned_addr learned my addr 172.21.15.37:0/4218158339 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:42:25.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fa9c1d8b640 10 monclient(hunting): get_auth_request con 0x7fa9bc13a1d0 auth_method 0 2024-04-26T00:42:25.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fa9c1d8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:25.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fa9c1d8b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:25.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fa9c1d8b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:25.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fa9c1d8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:25.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fa9c1d8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:25.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fa9c1d8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:25.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fa9c1d8b640 10 monclient(hunting): handle_auth_done global_id 264237 payload 274 2024-04-26T00:42:25.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fa9c1d8b640 1 -- 172.21.15.37:0/4218158339 >> v1:172.21.15.45:6789/0 conn(0x7fa9bc0a2fe0 legacy=0x7fa9bc0a13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:42:25.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fa9c1d8b640 1 -- 172.21.15.37:0/4218158339 >> v1:172.21.15.37:6789/0 conn(0x7fa9bc071da0 legacy=0x7fa9bc0a1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:42:25.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fa9c1d8b640 10 monclient: _finish_hunting 0 2024-04-26T00:42:25.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fa9c1d8b640 1 monclient: found mon.noname-a 2024-04-26T00:42:25.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fa9c1d8b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:42:25.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fa9c1d8b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:42:25.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fa9c1d8b640 1 -- 172.21.15.37:0/4218158339 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9bc070b20 con 0x7fa9bc13a1d0 2024-04-26T00:42:25.214 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.214 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3009242578}) 2024-04-26T00:42:25.214 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1287488571}) 2024-04-26T00:42:25.214 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.215 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3009242578}) 2024-04-26T00:42:25.215 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.204+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2816035637}) 2024-04-26T00:42:25.215 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.208+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.215 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.208+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2259942309}) 2024-04-26T00:42:25.215 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.208+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3966393742}) 2024-04-26T00:42:25.215 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.208+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.215 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.208+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2259942309}) 2024-04-26T00:42:25.215 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.208+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2623365867}) 2024-04-26T00:42:25.215 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.208+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.215 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.208+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=639318201}) 2024-04-26T00:42:25.215 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.208+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1593757107}) 2024-04-26T00:42:25.215 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.208+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.215 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.208+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=639318201}) 2024-04-26T00:42:25.215 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.208+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=500824969}) 2024-04-26T00:42:25.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c1d8b640 1 -- 172.21.15.37:0/4218158339 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa9bc13a1d0 msgr2=0x7fa9bc13c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:25.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c1d8b640 1 --2- 172.21.15.37:0/4218158339 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa9bc13a1d0 0x7fa9bc13c5c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc073040 tx=0x7fa9b401c2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4223ad25602226f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:25.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9bbfff640 1 -- 172.21.15.37:0/4218158339 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b40b0070 con 0x7fa9bc13a1d0 2024-04-26T00:42:25.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:25.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9bbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:42:25.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9bbfff640 10 monclient: dump: 2024-04-26T00:42:25.217 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:25.217 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:25.217 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:25.217 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:25.217 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:25.217 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:25.217 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:25.217 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:25.217 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:25.217 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:25.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9bbfff640 10 monclient: _finish_auth 0 2024-04-26T00:42:25.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9bbfff640 10 monclient: _check_auth_tickets 2024-04-26T00:42:25.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:42:25.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c37ee640 5 monclient: authenticate success, global_id 264237 2024-04-26T00:42:25.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c37ee640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:42:25.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9bbfff640 1 -- 172.21.15.37:0/4218158339 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9b40ab600 con 0x7fa9bc13a1d0 2024-04-26T00:42:25.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9bbfff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:42:25.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c37ee640 10 monclient: get_monmap_and_config success 2024-04-26T00:42:25.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9bbfff640 1 -- 172.21.15.37:0/4218158339 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b40ab9d0 con 0x7fa9bc13a1d0 2024-04-26T00:42:25.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:25.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c37ee640 4 set_mon_vals no callback set 2024-04-26T00:42:25.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9bbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:42:25.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9bbfff640 10 monclient: dump: 2024-04-26T00:42:25.218 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:25.219 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:25.219 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:25.219 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:25.219 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:25.219 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:25.219 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:25.219 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:25.219 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:25.219 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:25.220 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.220 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=875070101}) 2024-04-26T00:42:25.220 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1776653646}) 2024-04-26T00:42:25.220 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.220 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=875070101}) 2024-04-26T00:42:25.220 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=442712203}) 2024-04-26T00:42:25.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c37ee640 10 monclient: shutdown 2024-04-26T00:42:25.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c37ee640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:42:25.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c37ee640 1 -- 172.21.15.37:0/4218158339 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa9bc13a1d0 msgr2=0x7fa9bc13c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:25.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c37ee640 1 --2- 172.21.15.37:0/4218158339 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa9bc13a1d0 0x7fa9bc13c5c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc073040 tx=0x7fa9b401c2c0 comp rx=0 tx=0).stop 2024-04-26T00:42:25.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c37ee640 1 -- 172.21.15.37:0/4218158339 shutdown_connections 2024-04-26T00:42:25.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c37ee640 1 --2- 172.21.15.37:0/4218158339 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa9bc13a1d0 0x7fa9bc13c5c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:25.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c37ee640 1 -- 172.21.15.37:0/4218158339 >> 172.21.15.37:0/4218158339 conn(0x7fa9bc0a24c0 msgr2=0x7fa9bc0a28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:42:25.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c37ee640 1 -- 172.21.15.37:0/4218158339 shutdown_connections 2024-04-26T00:42:25.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c37ee640 1 -- 172.21.15.37:0/4218158339 wait complete. 2024-04-26T00:42:25.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c37ee640 10 monclient: build_initial_monmap 2024-04-26T00:42:25.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c37ee640 10 monclient: monmap: 2024-04-26T00:42:25.222 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:42:25.222 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:42:25.222 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:42:25.222 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:42:25.223 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:42:25.223 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:42:25.223 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:42:25.223 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:42:25.223 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:42:25.223 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:25.223 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c37ee640 1 Processor -- start 2024-04-26T00:42:25.223 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c37ee640 1 -- start start 2024-04-26T00:42:25.224 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c37ee640 10 monclient: init 2024-04-26T00:42:25.224 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1652190984}) 2024-04-26T00:42:25.224 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1154672101}) 2024-04-26T00:42:25.224 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.224 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1652190984}) 2024-04-26T00:42:25.225 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3212197124}) 2024-04-26T00:42:25.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c37ee640 10 monclient: _reopen_session rank -1 2024-04-26T00:42:25.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c37ee640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:42:25.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c37ee640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9bc071da0 0x7fa9bc1cf750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:25.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c37ee640 10 monclient(hunting): picked mon.noname-b con 0x7fa9bc071da0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:42:25.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c37ee640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa9bc0a2fe0 0x7fa9bc1cfc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:25.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c37ee640 10 monclient(hunting): picked mon.noname-a con 0x7fa9bc0a2fe0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:42:25.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c37ee640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa9bc13a1d0 0x7fa9bc11ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:25.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c37ee640 10 monclient(hunting): picked mon.noname-c con 0x7fa9bc13a1d0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:42:25.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c37ee640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:25.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c37ee640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc06e710 con 0x7fa9bc0a2fe0 2024-04-26T00:42:25.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c37ee640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:25.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c37ee640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc06e590 con 0x7fa9bc071da0 2024-04-26T00:42:25.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c37ee640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:25.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c37ee640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa9bc06e890 con 0x7fa9bc13a1d0 2024-04-26T00:42:25.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c37ee640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:25.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c37ee640 10 monclient(hunting): authenticate will time out at 2326.523727s 2024-04-26T00:42:25.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c0d89640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa9bc0a2fe0 0x7fa9bc1cfc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:25.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c1d8b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa9bc13a1d0 0x7fa9bc11ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:25.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c0d89640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa9bc0a2fe0 0x7fa9bc1cfc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40328/0 (socket says 172.21.15.37:40328) 2024-04-26T00:42:25.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c0d89640 1 -- 172.21.15.37:0/3741984527 learned_addr learned my addr 172.21.15.37:0/3741984527 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:42:25.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c1d8b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa9bc13a1d0 0x7fa9bc11ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51172/0 (socket says 172.21.15.37:51172) 2024-04-26T00:42:25.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c0d89640 10 monclient(hunting): get_auth_request con 0x7fa9bc0a2fe0 auth_method 0 2024-04-26T00:42:25.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c0d89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:25.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c0d89640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:25.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c0d89640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:25.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c1d8b640 10 monclient(hunting): get_auth_request con 0x7fa9bc13a1d0 auth_method 0 2024-04-26T00:42:25.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c1d8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:25.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c1d8b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:25.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.212+0000 7fa9c1d8b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:25.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9c1d8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:25.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9c1d8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:25.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9c158a640 1 --2- 172.21.15.37:0/3741984527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9bc071da0 0x7fa9bc1cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:25.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9c1d8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:25.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9c0d89640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:25.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9c0d89640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:25.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9c0d89640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:25.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9c158a640 10 monclient(hunting): get_auth_request con 0x7fa9bc071da0 auth_method 0 2024-04-26T00:42:25.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9c158a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:25.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9c158a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:25.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9c158a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:25.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9c0d89640 10 monclient(hunting): handle_auth_done global_id 264240 payload 824 2024-04-26T00:42:25.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9c0d89640 1 -- 172.21.15.37:0/3741984527 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa9bc13a1d0 msgr2=0x7fa9bc11ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:25.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9c0d89640 1 --2- 172.21.15.37:0/3741984527 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa9bc13a1d0 0x7fa9bc11ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:25.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9c0d89640 1 -- 172.21.15.37:0/3741984527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9bc071da0 msgr2=0x7fa9bc1cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:25.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9c0d89640 1 --2- 172.21.15.37:0/3741984527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9bc071da0 0x7fa9bc1cf750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:25.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9c0d89640 10 monclient: _finish_hunting 0 2024-04-26T00:42:25.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9c0d89640 1 monclient: found mon.noname-a 2024-04-26T00:42:25.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9c0d89640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:42:25.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9c0d89640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:42:25.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9c0d89640 1 -- 172.21.15.37:0/3741984527 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9bc11e630 con 0x7fa9bc0a2fe0 2024-04-26T00:42:25.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9c158a640 1 --2- 172.21.15.37:0/3741984527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9bc071da0 0x7fa9bc1cf750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:42:25.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9c1d8b640 1 --2- 172.21.15.37:0/3741984527 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa9bc13a1d0 0x7fa9bc11ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:42:25.231 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.231 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3686288033}) 2024-04-26T00:42:25.231 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=598344193}) 2024-04-26T00:42:25.231 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.231 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3686288033}) 2024-04-26T00:42:25.231 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2506379844}) 2024-04-26T00:42:25.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9c0d89640 1 -- 172.21.15.37:0/3741984527 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa9bc0a2fe0 msgr2=0x7fa9bc1cfc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:25.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9c0d89640 1 --2- 172.21.15.37:0/3741984527 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa9bc0a2fe0 0x7fa9bc1cfc90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa9b000b460 tx=0x7fa9b000b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:25.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9b9ffb640 1 -- 172.21.15.37:0/3741984527 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b0015020 con 0x7fa9bc0a2fe0 2024-04-26T00:42:25.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:25.232 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9b9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:42:25.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9b9ffb640 10 monclient: dump: 2024-04-26T00:42:25.233 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:25.233 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:25.233 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:25.233 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:25.233 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:25.233 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:25.233 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:25.233 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:25.233 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:25.233 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:25.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9b9ffb640 10 monclient: _finish_auth 0 2024-04-26T00:42:25.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9b9ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:42:25.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:42:25.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9c37ee640 5 monclient: authenticate success, global_id 264240 2024-04-26T00:42:25.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9b9ffb640 1 -- 172.21.15.37:0/3741984527 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9b0011a60 con 0x7fa9bc0a2fe0 2024-04-26T00:42:25.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9c37ee640 10 monclient: _renew_subs 2024-04-26T00:42:25.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9c37ee640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:42:25.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9c37ee640 1 -- 172.21.15.37:0/3741984527 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9bc120480 con 0x7fa9bc0a2fe0 2024-04-26T00:42:25.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9b9ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:42:25.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9b9ffb640 1 -- 172.21.15.37:0/3741984527 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b0011e50 con 0x7fa9bc0a2fe0 2024-04-26T00:42:25.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:25.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9b9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:42:25.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9b9ffb640 10 monclient: dump: 2024-04-26T00:42:25.234 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:25.234 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:25.234 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:25.234 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:25.235 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:25.235 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:25.235 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:25.235 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:25.235 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:25.235 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:25.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9c37ee640 10 monclient: _renew_subs 2024-04-26T00:42:25.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9c37ee640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:42:25.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9c37ee640 1 -- 172.21.15.37:0/3741984527 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9bc1d01d0 con 0x7fa9bc0a2fe0 2024-04-26T00:42:25.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9bb7fe640 4 set_mon_vals no callback set 2024-04-26T00:42:25.236 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.236 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4166228712}) 2024-04-26T00:42:25.236 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4288460655}) 2024-04-26T00:42:25.236 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.236 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4166228712}) 2024-04-26T00:42:25.237 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1823126021}) 2024-04-26T00:42:25.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.216+0000 7fa9c37ee640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:25.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.220+0000 7fa9c37ee640 1 --2- 172.21.15.37:0/3741984527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9bc11e8e0 0x7fa9bc11cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:25.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.220+0000 7fa9c37ee640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:25.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.220+0000 7fa9c37ee640 1 -- 172.21.15.37:0/3741984527 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc04f510 con 0x7fa9bc11e8e0 2024-04-26T00:42:25.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.220+0000 7fa9b9ffb640 1 -- 172.21.15.37:0/3741984527 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa9b0018020 con 0x7fa9bc0a2fe0 2024-04-26T00:42:25.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.220+0000 7fa9b9ffb640 1 --2- 172.21.15.37:0/3741984527 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa9a8036020 0x7fa9a80384e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:25.238 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.220+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.238 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.220+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2821352790}) 2024-04-26T00:42:25.238 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.220+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2586643575}) 2024-04-26T00:42:25.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.220+0000 7fa9b9ffb640 1 -- 172.21.15.37:0/3741984527 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(896..896 src has 822..896) ==== 3975+0+0 (secure 0 0 0) 0x7fa9b0048ae0 con 0x7fa9bc0a2fe0 2024-04-26T00:42:25.239 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.220+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.239 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.220+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2821352790}) 2024-04-26T00:42:25.239 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.220+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3454303231}) 2024-04-26T00:42:25.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.220+0000 7fa9c158a640 1 --2- 172.21.15.37:0/3741984527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9bc11e8e0 0x7fa9bc11cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:25.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.220+0000 7fa9c158a640 10 monclient: get_auth_request con 0x7fa9bc11e8e0 auth_method 0 2024-04-26T00:42:25.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.220+0000 7fa9c158a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:25.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.220+0000 7fa9c158a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:25.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.220+0000 7fa9c158a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:25.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.220+0000 7fa9c1d8b640 1 --2- 172.21.15.37:0/3741984527 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa9a8036020 0x7fa9a80384e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:25.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.220+0000 7fa9c158a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:25.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.220+0000 7fa9c158a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:25.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.220+0000 7fa9c158a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:25.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.220+0000 7fa9c1d8b640 10 monclient: get_auth_request con 0x7fa9a8036020 auth_method 0 2024-04-26T00:42:25.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.220+0000 7fa9c158a640 10 monclient(hunting): handle_auth_done global_id 274123 payload 824 2024-04-26T00:42:25.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.220+0000 7fa9c158a640 1 -- 172.21.15.37:0/3741984527 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa9bc0008e0 con 0x7fa9bc11e8e0 2024-04-26T00:42:25.242 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.220+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.242 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.220+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1720616488}) 2024-04-26T00:42:25.242 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.220+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=383619694}) 2024-04-26T00:42:25.243 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.220+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.243 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.220+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1720616488}) 2024-04-26T00:42:25.243 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.220+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1891294301}) 2024-04-26T00:42:25.243 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.220+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.243 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.220+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4069122897}) 2024-04-26T00:42:25.243 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.220+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=419385270}) 2024-04-26T00:42:25.243 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.220+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.243 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.220+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4069122897}) 2024-04-26T00:42:25.243 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.220+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=600936280}) 2024-04-26T00:42:25.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.220+0000 7fa9c158a640 1 -- 172.21.15.37:0/3741984527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9bc11e8e0 msgr2=0x7fa9bc11cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:25.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.220+0000 7fa9c158a640 1 --2- 172.21.15.37:0/3741984527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9bc11e8e0 0x7fa9bc11cfa0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac00e890 tx=0x7fa9ac00ed50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:25.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.220+0000 7fa9c1d8b640 1 --2- 172.21.15.37:0/3741984527 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa9a8036020 0x7fa9a80384e0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc071570 tx=0x7fa9b40a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:25.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.224+0000 7fa9b9ffb640 1 -- 172.21.15.37:0/3741984527 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ac00cb50 con 0x7fa9bc11e8e0 2024-04-26T00:42:25.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.224+0000 7fa9b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:25.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.224+0000 7fa9b9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:42:25.244 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.224+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.224+0000 7fa9b9ffb640 10 monclient: dump: 2024-04-26T00:42:25.245 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:25.245 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:25.245 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:25.245 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:25.245 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:25.245 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:25.245 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:25.245 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:25.245 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:25.245 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:25.246 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.224+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=63,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=1} crc {logm=1023999505,mdsmap=2840952586,mgr=4207260343,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=1388705204,mon_config_key=4192339162,monmap=3876866194,osd_metadata=1727858850}) 2024-04-26T00:42:25.246 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.224+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=64,mdsmap=3,mgr=5,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=10,monmap=3,osd_metadata=3} crc {logm=3364029469,mdsmap=2840952586,mgr=2055017934,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=1388705204,mon_config_key=3794053260,monmap=3876866194,osd_metadata=1654752143}) 2024-04-26T00:42:25.246 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.224+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.246 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.224+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=63,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=1} crc {logm=1023999505,mdsmap=2840952586,mgr=4207260343,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=1388705204,mon_config_key=4192339162,monmap=3876866194,osd_metadata=1727858850}) 2024-04-26T00:42:25.246 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.224+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=62,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=10,monmap=2,osd_metadata=4} crc {logm=1442340961,mdsmap=2840952586,mgr=3887433460,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=1388705204,mon_config_key=3247883965,monmap=4194790699,osd_metadata=3204546874}) 2024-04-26T00:42:25.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.224+0000 7fa9b9ffb640 1 -- 172.21.15.37:0/3741984527 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa9ac01a070 con 0x7fa9bc11e8e0 2024-04-26T00:42:25.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.224+0000 7fa9b9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:25.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.224+0000 7fa9b9ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:42:25.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.224+0000 7fa9b9ffb640 1 -- 172.21.15.37:0/3741984527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9bc11e8e0 msgr2=0x7fa9bc11cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:25.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.224+0000 7fa9b9ffb640 1 --2- 172.21.15.37:0/3741984527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9bc11e8e0 0x7fa9bc11cfa0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac00e890 tx=0x7fa9ac00ed50 comp rx=0 tx=0).stop 2024-04-26T00:42:25.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.224+0000 7fa9b9ffb640 1 -- 172.21.15.37:0/3741984527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9bc11e8e0 msgr2=0x7fa9bc11cfa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:42:25.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.224+0000 7fa9b9ffb640 1 --2- 172.21.15.37:0/3741984527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9bc11e8e0 0x7fa9bc11cfa0 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac00e890 tx=0x7fa9ac00ed50 comp rx=0 tx=0).stop 2024-04-26T00:42:25.248 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.224+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.248 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.224+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_metadata=5,osd_pg_creating=1,osd_snap=94} crc {osd_metadata=2387245979,osd_pg_creating=1821880714,osd_snap=3582027413}) 2024-04-26T00:42:25.248 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.224+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=5,osd_pg_creating=1,osd_snap=94} crc {osd_metadata=2387245979,osd_pg_creating=1821880713,osd_snap=1447674778}) 2024-04-26T00:42:25.249 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.224+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.249 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.224+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_metadata=5,osd_pg_creating=1,osd_snap=94} crc {osd_metadata=2387245979,osd_pg_creating=1821880714,osd_snap=3582027413}) 2024-04-26T00:42:25.249 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.224+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_metadata=5,osd_pg_creating=1,osd_snap=94} crc {osd_metadata=2387245979,osd_pg_creating=1821880713,osd_snap=4259911423}) 2024-04-26T00:42:25.249 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.228+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.249 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.228+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=77,osdmap=23} crc {osd_snap=1302538171,osdmap=2951311375}) 2024-04-26T00:42:25.249 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.228+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=78,osdmap=22} crc {osd_snap=2731765520,osdmap=2354705472}) 2024-04-26T00:42:25.249 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.228+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.249 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.228+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=77,osdmap=23} crc {osd_snap=1302538171,osdmap=2951311375}) 2024-04-26T00:42:25.249 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.228+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=71,osdmap=29} crc {osd_snap=706612597,osdmap=3922971852}) 2024-04-26T00:42:25.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.232+0000 7fa9c37ee640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:42:25.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.232+0000 7fa9c37ee640 1 --2- 172.21.15.37:0/3741984527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9bc11d4e0 0x7fa9bc11d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:25.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.232+0000 7fa9c37ee640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:25.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.232+0000 7fa9c37ee640 1 -- 172.21.15.37:0/3741984527 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc11de00 con 0x7fa9bc11d4e0 2024-04-26T00:42:25.251 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.232+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.251 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.232+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2822241303}) 2024-04-26T00:42:25.251 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.232+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3203552547}) 2024-04-26T00:42:25.251 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.232+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:42:25.251 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.232+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2822241303}) 2024-04-26T00:42:25.251 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:42:25.232+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=975049002}) 2024-04-26T00:42:25.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.232+0000 7fa9c158a640 1 --2- 172.21.15.37:0/3741984527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9bc11d4e0 0x7fa9bc11d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:25.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.232+0000 7fa9c158a640 10 monclient: get_auth_request con 0x7fa9bc11d4e0 auth_method 0 2024-04-26T00:42:25.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.232+0000 7fa9c158a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:25.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.232+0000 7fa9c158a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:25.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.232+0000 7fa9c158a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:25.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.232+0000 7fa9c158a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:25.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.232+0000 7fa9c158a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:25.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.232+0000 7fa9c158a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:25.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.232+0000 7fa9c158a640 10 monclient(hunting): handle_auth_done global_id 274126 payload 824 2024-04-26T00:42:25.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.232+0000 7fa9c158a640 1 -- 172.21.15.37:0/3741984527 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa9bc120b80 con 0x7fa9bc11d4e0 2024-04-26T00:42:25.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.232+0000 7fa9c158a640 1 -- 172.21.15.37:0/3741984527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9bc11d4e0 msgr2=0x7fa9bc11d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:25.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.232+0000 7fa9c158a640 1 --2- 172.21.15.37:0/3741984527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9bc11d4e0 0x7fa9bc11d8c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac00e890 tx=0x7fa9ac003af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:25.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.240+0000 7fa9b9ffb640 1 -- 172.21.15.37:0/3741984527 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ac01fa90 con 0x7fa9bc11d4e0 2024-04-26T00:42:25.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.240+0000 7fa9b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:25.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.240+0000 7fa9b9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:42:25.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.240+0000 7fa9b9ffb640 10 monclient: dump: 2024-04-26T00:42:25.254 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:25.254 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:25.254 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:25.254 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:25.254 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:25.254 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:25.254 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:25.254 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:25.254 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:25.254 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:25.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.240+0000 7fa9b9ffb640 1 -- 172.21.15.37:0/3741984527 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4547 (secure 0 0 0) 0x7fa9ac015430 con 0x7fa9bc11d4e0 2024-04-26T00:42:25.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.240+0000 7fa9b9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:42:25.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.240+0000 7fa9b9ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:42:25.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.240+0000 7fa9b9ffb640 1 -- 172.21.15.37:0/3741984527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9bc11d4e0 msgr2=0x7fa9bc11d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:25.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.240+0000 7fa9b9ffb640 1 --2- 172.21.15.37:0/3741984527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9bc11d4e0 0x7fa9bc11d8c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac00e890 tx=0x7fa9ac003af0 comp rx=0 tx=0).stop 2024-04-26T00:42:25.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.240+0000 7fa9b9ffb640 1 -- 172.21.15.37:0/3741984527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9bc11d4e0 msgr2=0x7fa9bc11d8c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:42:25.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.240+0000 7fa9b9ffb640 1 --2- 172.21.15.37:0/3741984527 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa9bc11d4e0 0x7fa9bc11d8c0 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac00e890 tx=0x7fa9ac003af0 comp rx=0 tx=0).stop 2024-04-26T00:42:25.255 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:42:25.255 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:42:25.255 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:42:25.255 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:42:25.255 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 160, 2024-04-26T00:42:25.255 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 12987, 2024-04-26T00:42:25.255 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:42:25.256 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:42:25.256 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:42:25.256 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:42:25.256 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:42:25.256 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 6, 2024-04-26T00:42:25.256 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:42:25.256 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:42:25.256 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:42:25.256 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:42:25.256 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:42:25.256 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:42:25.256 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:42:25.257 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:42:25.257 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:42:25.257 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:42:25.257 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:42:25.257 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:42:25.257 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:42:25.257 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:42:25.257 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:42:25.257 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:42:25.257 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:42:25.257 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:42:25.257 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:42:25.257 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:42:25.257 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:42:25.258 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:42:25.258 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:42:25.258 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:42:25.258 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:42:25.258 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:42:25.258 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:42:25.258 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:42:25.258 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:25.258 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:25.258 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:42:25.258 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:42:25.258 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:42:25.259 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:42:25.259 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:42:25.259 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:42:25.259 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:42:25.259 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:42:25.259 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:42:25.259 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:42:25.259 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:42:25.259 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:42:25.259 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:42:25.259 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:42:25.259 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:42:25.259 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:42:25.260 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:42:25.260 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:42:25.260 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:42:25.260 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:42:25.260 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:42:25.260 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:42:25.260 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:42:25.260 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:42:25.260 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:42:25.260 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:42:25.260 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:42:25.260 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:42:25.260 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:42:25.261 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:42:25.261 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:25.261 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:42:25.261 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:25.261 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:42:25.261 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:42:25.261 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:42:25.261 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:42:25.261 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:25.261 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:42:25.261 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:42:25.262 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:25.262 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:25.262 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:25.262 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:42:25.262 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:42:25.262 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:25.262 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:25.262 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:25.262 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:25.262 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:42:25.262 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:42:25.263 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:42:25.263 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:42:25.263 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:42:25.263 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:25.263 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:25.263 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:42:25.263 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:42:25.263 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:42:25.263 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:42:25.263 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:25.263 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:42:25.263 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:42:25.263 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:25.264 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:25.264 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:25.264 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:42:25.264 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:42:25.264 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:25.264 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:25.264 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:25.264 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:25.264 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:42:25.265 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:42:25.265 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:42:25.265 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:42:25.265 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:42:25.265 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:25.265 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:25.265 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:42:25.265 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:42:25.265 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:42:25.265 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:42:25.265 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:25.265 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:42:25.265 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:42:25.265 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:25.265 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:25.266 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:25.266 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:42:25.266 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:42:25.266 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:25.266 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:25.266 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:25.266 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:25.266 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:42:25.266 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:42:25.266 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:42:25.266 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:42:25.266 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:42:25.267 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:25.267 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:25.267 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:25.267 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:42:25.267 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:42:25.267 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:25.267 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:42:25.267 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:42:25.267 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:42:25.267 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:25.267 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:42:25.267 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:42:25.267 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:25.268 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:42:25.268 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:42:25.268 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:42:25.268 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:25.268 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:25.268 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:25.268 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:42:25.268 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:42:25.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.240+0000 7fa9c37ee640 1 -- 172.21.15.37:0/3741984527 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa9a8036020 msgr2=0x7fa9a80384e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:25.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.240+0000 7fa9c37ee640 1 --2- 172.21.15.37:0/3741984527 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa9a8036020 0x7fa9a80384e0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc071570 tx=0x7fa9b40a6000 comp rx=0 tx=0).stop 2024-04-26T00:42:25.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.240+0000 7fa9c37ee640 10 monclient: shutdown 2024-04-26T00:42:25.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.240+0000 7fa9c37ee640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:42:25.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.240+0000 7fa9c37ee640 1 -- 172.21.15.37:0/3741984527 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa9bc0a2fe0 msgr2=0x7fa9bc1cfc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:25.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.240+0000 7fa9c37ee640 1 --2- 172.21.15.37:0/3741984527 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa9bc0a2fe0 0x7fa9bc1cfc90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa9b000b460 tx=0x7fa9b000b920 comp rx=0 tx=0).stop 2024-04-26T00:42:25.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.240+0000 7fa9c1d8b640 1 -- 172.21.15.37:0/3741984527 reap_dead start 2024-04-26T00:42:25.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.240+0000 7fa9c1d8b640 1 -- 172.21.15.37:0/3741984527 reap_dead start 2024-04-26T00:42:25.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.240+0000 7fa9c37ee640 1 -- 172.21.15.37:0/3741984527 shutdown_connections 2024-04-26T00:42:25.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.240+0000 7fa9c37ee640 1 -- 172.21.15.37:0/3741984527 >> 172.21.15.37:0/3741984527 conn(0x7fa9bc0a24c0 msgr2=0x7fa9bc137a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:42:25.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.240+0000 7fa9c37ee640 1 -- 172.21.15.37:0/3741984527 shutdown_connections 2024-04-26T00:42:25.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.240+0000 7fa9c37ee640 1 -- 172.21.15.37:0/3741984527 wait complete. 2024-04-26T00:42:25.271 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:42:25.271 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-04-26T00:42:25.271 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-26T00:42:25.271 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-04-26T00:42:25.271 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-04-26T00:42:25.271 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-26T00:42:25.271 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T00:42:25.284 INFO:tasks.ceph.mon.a:Stopped 2024-04-26T00:42:25.284 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-26T00:42:25.284 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:42:25.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7faceeb640 10 monclient: get_monmap_and_config 2024-04-26T00:42:25.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7faceeb640 10 monclient: build_initial_monmap 2024-04-26T00:42:25.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7faceeb640 10 monclient: monmap: 2024-04-26T00:42:25.440 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:42:25.440 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:25.440 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:42:25.441195+0000 2024-04-26T00:42:25.440 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:42:25.441195+0000 2024-04-26T00:42:25.440 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:42:25.440 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:25.440 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:42:25.440 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:42:25.440 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:42:25.440 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:42:25.440 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:42:25.440 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:25.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7faceeb640 1 Processor -- start 2024-04-26T00:42:25.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7faceeb640 1 -- start start 2024-04-26T00:42:25.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7faceeb640 10 monclient: init 2024-04-26T00:42:25.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7faceeb640 10 monclient: _reopen_session rank -1 2024-04-26T00:42:25.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7faceeb640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-04-26T00:42:25.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7faceeb640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f7fa8136e10 0x7f7fa809c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:25.443 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7faceeb640 10 monclient(hunting): picked mon.noname-a con 0x7f7fa8136e10 addr v2:172.21.15.37:3300/0 2024-04-26T00:42:25.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7faceeb640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7fa81371f0 0x7f7fa8137600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:25.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7faceeb640 10 monclient(hunting): picked mon.noname-c con 0x7f7fa81371f0 addr v2:172.21.15.45:3300/0 2024-04-26T00:42:25.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7faceeb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7fa8135d90 addr v1:172.21.15.37:6789/0 2024-04-26T00:42:25.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7faceeb640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fa809b680 con 0x7f7fa8135d90 2024-04-26T00:42:25.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7faceeb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:25.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7faceeb640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f7fa809b8f0 con 0x7f7fa8136e10 2024-04-26T00:42:25.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7faceeb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:25.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7faceeb640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f7fa809bbd0 con 0x7f7fa81371f0 2024-04-26T00:42:25.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7faceeb640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:25.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7faceeb640 10 monclient(hunting): authenticate will time out at 2326.746972s 2024-04-26T00:42:25.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7fa6d9d640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f7fa8135d90 legacy=0x7f7fa8136220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:42:25.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7fa659c640 1 -- >> v2:172.21.15.37:3300/0 conn(0x7f7fa8136e10 msgr2=0x7f7fa809c5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:42:25.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7fa659c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f7fa8136e10 0x7f7fa809c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:42:25.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7fa5d9b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7fa81371f0 0x7f7fa8137600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:25.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7fa5d9b640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f7fa81371f0 0x7f7fa8137600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.37:60184/0 (socket says 172.21.15.37:60184) 2024-04-26T00:42:25.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7fa5d9b640 1 -- 172.21.15.37:0/4057609536 learned_addr learned my addr 172.21.15.37:0/4057609536 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:42:25.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7fa5d9b640 10 monclient(hunting): get_auth_request con 0x7f7fa81371f0 auth_method 0 2024-04-26T00:42:25.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7fa5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:25.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7fa5d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:25.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7fa5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:25.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7fa5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:25.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7fa5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:25.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7fa5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:25.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7fa5d9b640 10 monclient(hunting): handle_auth_done global_id 274129 payload 274 2024-04-26T00:42:25.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7fa5d9b640 1 -- 172.21.15.37:0/4057609536 >> v2:172.21.15.37:3300/0 conn(0x7f7fa8136e10 msgr2=0x7f7fa809c5b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:42:25.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7fa5d9b640 1 --2- 172.21.15.37:0/4057609536 >> v2:172.21.15.37:3300/0 conn(0x7f7fa8136e10 0x7f7fa809c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:25.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7fa5d9b640 1 -- 172.21.15.37:0/4057609536 >> v1:172.21.15.37:6789/0 conn(0x7f7fa8135d90 legacy=0x7f7fa8136220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:42:25.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7fa5d9b640 10 monclient: _finish_hunting 0 2024-04-26T00:42:25.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7fa5d9b640 1 monclient: found mon.noname-c 2024-04-26T00:42:25.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7fa5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:42:25.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7fa5d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:42:25.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.436+0000 7f7fa5d9b640 1 -- 172.21.15.37:0/4057609536 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fa8137b40 con 0x7f7fa81371f0 2024-04-26T00:42:25.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa5d9b640 1 -- 172.21.15.37:0/4057609536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fa81371f0 msgr2=0x7f7fa8137600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:25.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa5d9b640 1 --2- 172.21.15.37:0/4057609536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fa81371f0 0x7f7fa8137600 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8068e60 tx=0x7f7f9c01c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=502e42d259926131 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:25.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa559a640 1 -- 172.21.15.37:0/4057609536 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f9c0b1070 con 0x7f7fa81371f0 2024-04-26T00:42:25.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:25.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:42:25.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa559a640 10 monclient: dump: 2024-04-26T00:42:25.448 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:25.448 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:25.448 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:25.448 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:25.448 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:25.448 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:25.448 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:25.448 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:25.448 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:25.449 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:25.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa559a640 10 monclient: _finish_auth 0 2024-04-26T00:42:25.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa559a640 10 monclient: _check_auth_tickets 2024-04-26T00:42:25.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:42:25.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa559a640 1 -- 172.21.15.37:0/4057609536 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f9c0a8050 con 0x7f7fa81371f0 2024-04-26T00:42:25.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7faceeb640 5 monclient: authenticate success, global_id 274129 2024-04-26T00:42:25.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7faceeb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:42:25.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa559a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:42:25.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7faceeb640 10 monclient: get_monmap_and_config success 2024-04-26T00:42:25.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7faceeb640 4 set_mon_vals no callback set 2024-04-26T00:42:25.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa559a640 1 -- 172.21.15.37:0/4057609536 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f9c0ac860 con 0x7f7fa81371f0 2024-04-26T00:42:25.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:25.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:42:25.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa559a640 10 monclient: dump: 2024-04-26T00:42:25.450 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:25.450 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:25.450 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:25.450 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:25.450 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:25.450 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:25.450 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:25.450 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:25.450 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:25.450 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:25.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7faceeb640 10 monclient: shutdown 2024-04-26T00:42:25.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7faceeb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:42:25.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7faceeb640 1 -- 172.21.15.37:0/4057609536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fa81371f0 msgr2=0x7f7fa8137600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:25.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7faceeb640 1 --2- 172.21.15.37:0/4057609536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fa81371f0 0x7f7fa8137600 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8068e60 tx=0x7f7f9c01c3c0 comp rx=0 tx=0).stop 2024-04-26T00:42:25.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7faceeb640 1 -- 172.21.15.37:0/4057609536 shutdown_connections 2024-04-26T00:42:25.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7faceeb640 1 --2- 172.21.15.37:0/4057609536 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fa81371f0 0x7f7fa8137600 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:25.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7faceeb640 1 --2- 172.21.15.37:0/4057609536 >> v2:172.21.15.37:3300/0 conn(0x7f7fa8136e10 0x7f7fa809c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:25.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7faceeb640 1 -- 172.21.15.37:0/4057609536 >> 172.21.15.37:0/4057609536 conn(0x7f7fa80a64e0 msgr2=0x7f7fa80a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:42:25.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7faceeb640 1 -- 172.21.15.37:0/4057609536 shutdown_connections 2024-04-26T00:42:25.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7faceeb640 1 -- 172.21.15.37:0/4057609536 wait complete. 2024-04-26T00:42:25.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7faceeb640 10 monclient: build_initial_monmap 2024-04-26T00:42:25.455 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7faceeb640 10 monclient: monmap: 2024-04-26T00:42:25.455 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:42:25.455 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:42:25.455 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:42:25.456 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:42:25.456 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:42:25.456 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:42:25.456 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:42:25.456 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:42:25.456 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:42:25.456 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:25.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7faceeb640 1 Processor -- start 2024-04-26T00:42:25.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7faceeb640 1 -- start start 2024-04-26T00:42:25.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7faceeb640 10 monclient: init 2024-04-26T00:42:25.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7faceeb640 10 monclient: _reopen_session rank -1 2024-04-26T00:42:25.456 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7faceeb640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:42:25.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7faceeb640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fa8135d90 0x7f7fa81cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:25.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7faceeb640 10 monclient(hunting): picked mon.noname-b con 0x7f7fa8135d90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:42:25.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7faceeb640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7fa8136e10 0x7f7fa8134e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:25.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7faceeb640 10 monclient(hunting): picked mon.noname-c con 0x7f7fa8136e10 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:42:25.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7faceeb640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7fa81371f0 0x7f7fa813aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:25.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7faceeb640 10 monclient(hunting): picked mon.noname-a con 0x7f7fa81371f0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:42:25.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7faceeb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:25.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7faceeb640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa8141f10 con 0x7f7fa81371f0 2024-04-26T00:42:25.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7faceeb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:25.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7faceeb640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa8141d90 con 0x7f7fa8135d90 2024-04-26T00:42:25.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7faceeb640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:25.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7faceeb640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f7fa8142090 con 0x7f7fa8136e10 2024-04-26T00:42:25.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7faceeb640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:25.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7faceeb640 10 monclient(hunting): authenticate will time out at 2326.755112s 2024-04-26T00:42:25.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa6d9d640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7fa81371f0 msgr2=0x7f7fa813aa90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:42:25.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa6d9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7fa81371f0 0x7f7fa813aa90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:42:25.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa5d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7fa8136e10 0x7f7fa8134e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:25.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa5d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7fa8136e10 0x7f7fa8134e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51200/0 (socket says 172.21.15.37:51200) 2024-04-26T00:42:25.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa5d9b640 1 -- 172.21.15.37:0/3996305670 learned_addr learned my addr 172.21.15.37:0/3996305670 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:42:25.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa5d9b640 10 monclient(hunting): get_auth_request con 0x7f7fa8136e10 auth_method 0 2024-04-26T00:42:25.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:25.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa5d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:25.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:25.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:25.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:25.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:25.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa659c640 1 --2- 172.21.15.37:0/3996305670 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fa8135d90 0x7f7fa81cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:25.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa659c640 10 monclient(hunting): get_auth_request con 0x7f7fa8135d90 auth_method 0 2024-04-26T00:42:25.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:25.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:25.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:25.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:25.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:25.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:25.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa5d9b640 10 monclient(hunting): handle_auth_done global_id 254363 payload 824 2024-04-26T00:42:25.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa5d9b640 1 -- 172.21.15.37:0/3996305670 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fa8135d90 msgr2=0x7f7fa81cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:25.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa5d9b640 1 --2- 172.21.15.37:0/3996305670 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fa8135d90 0x7f7fa81cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:25.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa5d9b640 1 -- 172.21.15.37:0/3996305670 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7fa81371f0 msgr2=0x7f7fa813aa90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:42:25.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa5d9b640 1 --2- 172.21.15.37:0/3996305670 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7fa81371f0 0x7f7fa813aa90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:25.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa5d9b640 10 monclient: _finish_hunting 0 2024-04-26T00:42:25.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa5d9b640 1 monclient: found mon.noname-c 2024-04-26T00:42:25.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:42:25.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa5d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:42:25.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.444+0000 7f7fa5d9b640 1 -- 172.21.15.37:0/3996305670 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fa813ea20 con 0x7f7fa8136e10 2024-04-26T00:42:25.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.448+0000 7f7fa5d9b640 1 -- 172.21.15.37:0/3996305670 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7fa8136e10 msgr2=0x7f7fa8134e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:25.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.448+0000 7f7fa5d9b640 1 --2- 172.21.15.37:0/3996305670 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7fa8136e10 0x7f7fa8134e80 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c01aa70 tx=0x7f7f9c019160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:25.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.448+0000 7f7f96ffd640 1 -- 172.21.15.37:0/3996305670 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f9c0bc020 con 0x7f7fa8136e10 2024-04-26T00:42:25.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.448+0000 7f7f96ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:25.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.448+0000 7f7f96ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:42:25.460 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.448+0000 7f7f96ffd640 10 monclient: dump: 2024-04-26T00:42:25.460 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:25.460 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:25.460 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:25.460 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:25.460 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:25.460 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:25.460 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:25.461 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:25.461 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:25.461 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:25.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.448+0000 7f7f96ffd640 10 monclient: _finish_auth 0 2024-04-26T00:42:25.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.448+0000 7f7f96ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:42:25.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.448+0000 7f7f96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:42:25.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.448+0000 7f7faceeb640 5 monclient: authenticate success, global_id 254363 2024-04-26T00:42:25.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.448+0000 7f7faceeb640 10 monclient: _renew_subs 2024-04-26T00:42:25.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.448+0000 7f7f96ffd640 1 -- 172.21.15.37:0/3996305670 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f9c0b2040 con 0x7f7fa8136e10 2024-04-26T00:42:25.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.448+0000 7f7faceeb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:25.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.448+0000 7f7faceeb640 1 -- 172.21.15.37:0/3996305670 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fa809aca0 con 0x7f7fa8136e10 2024-04-26T00:42:25.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.448+0000 7f7f96ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:42:25.461 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.448+0000 7f7f96ffd640 1 -- 172.21.15.37:0/3996305670 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f9c0b1070 con 0x7f7fa8136e10 2024-04-26T00:42:25.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.448+0000 7f7f96ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:25.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.448+0000 7f7f96ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:42:25.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.448+0000 7f7f96ffd640 10 monclient: dump: 2024-04-26T00:42:25.462 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:25.462 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:25.462 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:25.462 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:25.462 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:25.462 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:25.462 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:25.462 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:25.462 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:25.462 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:25.462 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.448+0000 7f7fa4d99640 4 set_mon_vals no callback set 2024-04-26T00:42:25.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.448+0000 7f7faceeb640 10 monclient: _renew_subs 2024-04-26T00:42:25.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.448+0000 7f7faceeb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:25.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.448+0000 7f7faceeb640 1 -- 172.21.15.37:0/3996305670 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fa813e740 con 0x7f7fa8136e10 2024-04-26T00:42:25.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.448+0000 7f7faceeb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:25.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.448+0000 7f7faceeb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:25.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.448+0000 7f7faceeb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:25.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.448+0000 7f7faceeb640 1 -- 172.21.15.37:0/3996305670 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fa8075910 con 0x7f7fa8136e10 2024-04-26T00:42:25.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.448+0000 7f7f96ffd640 1 -- 172.21.15.37:0/3996305670 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f7f9c0bf020 con 0x7f7fa8136e10 2024-04-26T00:42:25.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.448+0000 7f7f96ffd640 1 --2- 172.21.15.37:0/3996305670 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7f88035e30 0x7f7f880382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:25.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.448+0000 7f7f96ffd640 1 -- 172.21.15.37:0/3996305670 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(896..896 src has 822..896) ==== 3975+0+0 (secure 0 0 0) 0x7f7f9c0e30b0 con 0x7f7fa8136e10 2024-04-26T00:42:25.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.452+0000 7f7fa659c640 1 --2- 172.21.15.37:0/3996305670 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7f88035e30 0x7f7f880382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:25.463 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.452+0000 7f7fa659c640 10 monclient: get_auth_request con 0x7f7f88035e30 auth_method 0 2024-04-26T00:42:25.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.452+0000 7f7fa659c640 1 --2- 172.21.15.37:0/3996305670 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7f88035e30 0x7f7f880382f0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f7f980038c0 tx=0x7f7f98009110 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:25.464 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:25.456+0000 7f7fa5d9b640 1 --2- 172.21.15.37:0/3996305670 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7fa8136e10 0x7f7fa8134e80 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c01aa70 tx=0x7f7f9c019160 comp rx=0 tx=0).queue_received will delay after 1.71409e+09 on 0x7f7f9c0c6070 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) 2024-04-26T00:42:26.286 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:26.284+0000 7f7f96ffd640 1 -- 172.21.15.37:0/3996305670 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7f9c0c6070 con 0x7f7fa8136e10 2024-04-26T00:42:26.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:26.284+0000 7f7f96ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:26.287 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:26.284+0000 7f7f96ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:42:26.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:26.516+0000 7f7faceeb640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:42:26.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:26.516+0000 7f7faceeb640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:42:26.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:26.516+0000 7f7faceeb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:26.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:26.516+0000 7f7faceeb640 1 -- 172.21.15.37:0/3996305670 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7fa8066f40 con 0x7f7fa8136e10 2024-04-26T00:42:26.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:26.516+0000 7f7f96ffd640 1 -- 172.21.15.37:0/3996305670 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f7f9c0160d0 con 0x7f7fa8136e10 2024-04-26T00:42:26.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:26.516+0000 7f7f96ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:42:26.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:26.516+0000 7f7f96ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:42:26.519 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":160,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:42:26.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:26.516+0000 7f7faceeb640 1 -- 172.21.15.37:0/3996305670 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7f88035e30 msgr2=0x7f7f880382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:26.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:26.516+0000 7f7faceeb640 1 --2- 172.21.15.37:0/3996305670 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7f88035e30 0x7f7f880382f0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f7f980038c0 tx=0x7f7f98009110 comp rx=0 tx=0).stop 2024-04-26T00:42:26.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:26.516+0000 7f7faceeb640 10 monclient: shutdown 2024-04-26T00:42:26.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:26.516+0000 7f7faceeb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:42:26.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:26.516+0000 7f7faceeb640 1 -- 172.21.15.37:0/3996305670 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7fa8136e10 msgr2=0x7f7fa8134e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:26.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:26.516+0000 7f7faceeb640 1 --2- 172.21.15.37:0/3996305670 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7fa8136e10 0x7f7fa8134e80 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c01aa70 tx=0x7f7f9c019160 comp rx=0 tx=0).stop 2024-04-26T00:42:26.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:26.516+0000 7f7faceeb640 1 -- 172.21.15.37:0/3996305670 shutdown_connections 2024-04-26T00:42:26.521 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:26.516+0000 7f7faceeb640 1 --2- 172.21.15.37:0/3996305670 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7f88035e30 0x7f7f880382f0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:26.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:26.516+0000 7f7faceeb640 1 --2- 172.21.15.37:0/3996305670 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7fa81371f0 0x7f7fa813aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:26.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:26.516+0000 7f7faceeb640 1 --2- 172.21.15.37:0/3996305670 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7fa8136e10 0x7f7fa8134e80 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:26.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:26.516+0000 7f7faceeb640 1 --2- 172.21.15.37:0/3996305670 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7fa8135d90 0x7f7fa81cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:26.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:26.516+0000 7f7faceeb640 1 -- 172.21.15.37:0/3996305670 >> 172.21.15.37:0/3996305670 conn(0x7f7fa80a64e0 msgr2=0x7f7fa809d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:42:26.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:26.516+0000 7f7faceeb640 1 -- 172.21.15.37:0/3996305670 shutdown_connections 2024-04-26T00:42:26.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:26.516+0000 7f7faceeb640 1 -- 172.21.15.37:0/3996305670 wait complete. 2024-04-26T00:42:29.540 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:42:29.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a1096e640 10 monclient: get_monmap_and_config 2024-04-26T00:42:29.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a1096e640 10 monclient: build_initial_monmap 2024-04-26T00:42:29.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a1096e640 10 monclient: monmap: 2024-04-26T00:42:29.672 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:42:29.672 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:29.672 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:42:29.673533+0000 2024-04-26T00:42:29.672 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:42:29.673533+0000 2024-04-26T00:42:29.672 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:42:29.672 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:29.672 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:42:29.673 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:42:29.673 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:42:29.673 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:42:29.673 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:42:29.673 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:29.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a1096e640 1 Processor -- start 2024-04-26T00:42:29.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a1096e640 1 -- start start 2024-04-26T00:42:29.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a1096e640 10 monclient: init 2024-04-26T00:42:29.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a1096e640 10 monclient: _reopen_session rank -1 2024-04-26T00:42:29.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a1096e640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-04-26T00:42:29.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a1096e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2a0c136e10 addr v1:172.21.15.45:6789/0 2024-04-26T00:42:29.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a1096e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a0c137900 0x7f2a0c09c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:29.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a1096e640 10 monclient(hunting): picked mon.noname-b con 0x7f2a0c137900 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:42:29.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a1096e640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f2a0c13c340 0x7f2a0c13e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:29.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a1096e640 10 monclient(hunting): picked mon.noname-a con 0x7f2a0c13c340 addr v2:172.21.15.37:3300/0 2024-04-26T00:42:29.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a1096e640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a0c09b680 con 0x7f2a0c136e10 2024-04-26T00:42:29.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a1096e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:29.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a1096e640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f2a0c09b8f0 con 0x7f2a0c13c340 2024-04-26T00:42:29.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a1096e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:29.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a1096e640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2a0c09bbd0 con 0x7f2a0c137900 2024-04-26T00:42:29.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a1096e640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:29.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a1096e640 10 monclient(hunting): authenticate will time out at 2330.979286s 2024-04-26T00:42:29.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a0ad9d640 1 -- >> v2:172.21.15.37:3300/0 conn(0x7f2a0c13c340 msgr2=0x7f2a0c13e730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:42:29.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a0ad9d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f2a0c13c340 0x7f2a0c13e730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:42:29.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a09d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a0c137900 0x7f2a0c09c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:29.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a09d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a0c137900 0x7f2a0c09c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51216/0 (socket says 172.21.15.37:51216) 2024-04-26T00:42:29.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a09d9b640 1 -- 172.21.15.37:0/3249838142 learned_addr learned my addr 172.21.15.37:0/3249838142 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:42:29.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a09d9b640 10 monclient(hunting): get_auth_request con 0x7f2a0c137900 auth_method 0 2024-04-26T00:42:29.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a09d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:29.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a09d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:29.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a09d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:29.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a09d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:29.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a09d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:29.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a09d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:29.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a09d9b640 10 monclient(hunting): handle_auth_done global_id 254366 payload 274 2024-04-26T00:42:29.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a09d9b640 1 -- 172.21.15.37:0/3249838142 >> v2:172.21.15.37:3300/0 conn(0x7f2a0c13c340 msgr2=0x7f2a0c13e730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:42:29.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a09d9b640 1 --2- 172.21.15.37:0/3249838142 >> v2:172.21.15.37:3300/0 conn(0x7f2a0c13c340 0x7f2a0c13e730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:29.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a09d9b640 1 -- 172.21.15.37:0/3249838142 >> v1:172.21.15.45:6789/0 conn(0x7f2a0c136e10 legacy=0x7f2a0c1371f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:42:29.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a09d9b640 10 monclient: _finish_hunting 0 2024-04-26T00:42:29.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a09d9b640 1 monclient: found mon.noname-b 2024-04-26T00:42:29.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a09d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:42:29.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a09d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:42:29.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.668+0000 7f2a09d9b640 1 -- 172.21.15.37:0/3249838142 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a0c0a5d90 con 0x7f2a0c137900 2024-04-26T00:42:29.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a09d9b640 1 -- 172.21.15.37:0/3249838142 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a0c137900 msgr2=0x7f2a0c09c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:29.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a09d9b640 1 --2- 172.21.15.37:0/3249838142 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a0c137900 0x7f2a0c09c5b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f29fc0098a0 tx=0x7f29fc01c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9510b5e623ebaaee server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:29.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a0959a640 1 -- 172.21.15.37:0/3249838142 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29fc0b0070 con 0x7f2a0c137900 2024-04-26T00:42:29.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a0959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:29.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a0959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:42:29.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a0959a640 10 monclient: dump: 2024-04-26T00:42:29.682 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:29.682 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:29.682 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:29.682 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:29.682 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:29.682 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:29.682 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:29.683 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:29.683 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:29.683 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:29.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a0959a640 10 monclient: _finish_auth 0 2024-04-26T00:42:29.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a0959a640 10 monclient: _check_auth_tickets 2024-04-26T00:42:29.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a0959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:42:29.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a0959a640 1 -- 172.21.15.37:0/3249838142 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29fc0ab600 con 0x7f2a0c137900 2024-04-26T00:42:29.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a0959a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:42:29.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a0959a640 1 -- 172.21.15.37:0/3249838142 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29fc0ab9d0 con 0x7f2a0c137900 2024-04-26T00:42:29.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a1096e640 5 monclient: authenticate success, global_id 254366 2024-04-26T00:42:29.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a1096e640 10 monclient: get_monmap_and_config success 2024-04-26T00:42:29.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a1096e640 4 set_mon_vals no callback set 2024-04-26T00:42:29.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a0959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:29.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a0959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:42:29.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a0959a640 10 monclient: dump: 2024-04-26T00:42:29.684 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:29.684 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:29.684 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:29.684 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:29.684 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:29.684 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:29.684 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:29.684 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:29.684 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:29.684 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:29.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a1096e640 10 monclient: shutdown 2024-04-26T00:42:29.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a1096e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:42:29.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a1096e640 1 -- 172.21.15.37:0/3249838142 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a0c137900 msgr2=0x7f2a0c09c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:29.684 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a1096e640 1 --2- 172.21.15.37:0/3249838142 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a0c137900 0x7f2a0c09c5b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f29fc0098a0 tx=0x7f29fc01c3e0 comp rx=0 tx=0).stop 2024-04-26T00:42:29.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a1096e640 1 -- 172.21.15.37:0/3249838142 shutdown_connections 2024-04-26T00:42:29.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a1096e640 1 --2- 172.21.15.37:0/3249838142 >> v2:172.21.15.37:3300/0 conn(0x7f2a0c13c340 0x7f2a0c13e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:29.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a1096e640 1 --2- 172.21.15.37:0/3249838142 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a0c137900 0x7f2a0c09c5b0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:29.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a1096e640 1 -- 172.21.15.37:0/3249838142 >> 172.21.15.37:0/3249838142 conn(0x7f2a0c0a64e0 msgr2=0x7f2a0c0a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:42:29.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a1096e640 1 -- 172.21.15.37:0/3249838142 shutdown_connections 2024-04-26T00:42:29.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a1096e640 1 -- 172.21.15.37:0/3249838142 wait complete. 2024-04-26T00:42:29.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a1096e640 10 monclient: build_initial_monmap 2024-04-26T00:42:29.685 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a1096e640 10 monclient: monmap: 2024-04-26T00:42:29.685 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:42:29.685 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:42:29.685 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:42:29.685 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:42:29.685 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:42:29.685 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:42:29.685 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:42:29.686 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:42:29.686 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:42:29.686 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:29.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a1096e640 1 Processor -- start 2024-04-26T00:42:29.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a1096e640 1 -- start start 2024-04-26T00:42:29.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a1096e640 10 monclient: init 2024-04-26T00:42:29.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a1096e640 10 monclient: _reopen_session rank -1 2024-04-26T00:42:29.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a1096e640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:42:29.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a1096e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a0c136e10 0x7f2a0c135f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:29.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a1096e640 10 monclient(hunting): picked mon.noname-b con 0x7f2a0c136e10 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:42:29.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a1096e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a0c137900 0x7f2a0c138c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:29.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a1096e640 10 monclient(hunting): picked mon.noname-c con 0x7f2a0c137900 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:42:29.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a1096e640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a0c13c340 0x7f2a0c139170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:29.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a1096e640 10 monclient(hunting): picked mon.noname-a con 0x7f2a0c13c340 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:42:29.686 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a1096e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:29.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a1096e640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f2a0c141820 con 0x7f2a0c13c340 2024-04-26T00:42:29.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a1096e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:29.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a1096e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2a0c1416a0 con 0x7f2a0c136e10 2024-04-26T00:42:29.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a1096e640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:29.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a1096e640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2a0c1419a0 con 0x7f2a0c137900 2024-04-26T00:42:29.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a1096e640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:29.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a1096e640 10 monclient(hunting): authenticate will time out at 2330.986417s 2024-04-26T00:42:29.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a0ad9d640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a0c13c340 msgr2=0x7f2a0c139170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:42:29.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a0ad9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a0c13c340 0x7f2a0c139170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:42:29.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a09d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a0c137900 0x7f2a0c138c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:29.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a09d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a0c137900 0x7f2a0c138c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51226/0 (socket says 172.21.15.37:51226) 2024-04-26T00:42:29.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a09d9b640 1 -- 172.21.15.37:0/3890306059 learned_addr learned my addr 172.21.15.37:0/3890306059 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:42:29.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a09d9b640 10 monclient(hunting): get_auth_request con 0x7f2a0c137900 auth_method 0 2024-04-26T00:42:29.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a09d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:29.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a09d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:29.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a09d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:29.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a09d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:29.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a09d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:29.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a09d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:29.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a0a59c640 1 --2- 172.21.15.37:0/3890306059 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a0c136e10 0x7f2a0c135f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:29.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a0a59c640 10 monclient(hunting): get_auth_request con 0x7f2a0c136e10 auth_method 0 2024-04-26T00:42:29.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a0a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:29.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a0a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:29.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a0a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:29.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a09d9b640 10 monclient(hunting): handle_auth_done global_id 254369 payload 824 2024-04-26T00:42:29.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a09d9b640 1 -- 172.21.15.37:0/3890306059 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a0c136e10 msgr2=0x7f2a0c135f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:29.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a09d9b640 1 --2- 172.21.15.37:0/3890306059 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a0c136e10 0x7f2a0c135f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:29.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a09d9b640 1 -- 172.21.15.37:0/3890306059 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a0c13c340 msgr2=0x7f2a0c139170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:42:29.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a09d9b640 1 --2- 172.21.15.37:0/3890306059 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a0c13c340 0x7f2a0c139170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:29.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a09d9b640 10 monclient: _finish_hunting 0 2024-04-26T00:42:29.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a09d9b640 1 monclient: found mon.noname-c 2024-04-26T00:42:29.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a09d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:42:29.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a09d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:42:29.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.676+0000 7f2a09d9b640 1 -- 172.21.15.37:0/3890306059 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a0c120990 con 0x7f2a0c137900 2024-04-26T00:42:29.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.680+0000 7f2a09d9b640 1 -- 172.21.15.37:0/3890306059 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a0c137900 msgr2=0x7f2a0c138c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:29.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.680+0000 7f2a09d9b640 1 --2- 172.21.15.37:0/3890306059 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a0c137900 0x7f2a0c138c30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f29fc0ae040 tx=0x7f29fc01ac20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:29.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.680+0000 7f2a02ffd640 1 -- 172.21.15.37:0/3890306059 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29fc0ba020 con 0x7f2a0c137900 2024-04-26T00:42:29.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.680+0000 7f2a02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:29.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.680+0000 7f2a02ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:42:29.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.680+0000 7f2a02ffd640 10 monclient: dump: 2024-04-26T00:42:29.692 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:29.692 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:29.692 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:29.692 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:29.692 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:29.692 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:29.692 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:29.692 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:29.692 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:29.692 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:29.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.680+0000 7f2a02ffd640 10 monclient: _finish_auth 0 2024-04-26T00:42:29.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.680+0000 7f2a02ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:42:29.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.680+0000 7f2a02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:42:29.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.680+0000 7f2a02ffd640 1 -- 172.21.15.37:0/3890306059 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29fc0b6850 con 0x7f2a0c137900 2024-04-26T00:42:29.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.680+0000 7f2a1096e640 5 monclient: authenticate success, global_id 254369 2024-04-26T00:42:29.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.680+0000 7f2a1096e640 10 monclient: _renew_subs 2024-04-26T00:42:29.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.680+0000 7f2a1096e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:29.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.680+0000 7f2a1096e640 1 -- 172.21.15.37:0/3890306059 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a0c09aca0 con 0x7f2a0c137900 2024-04-26T00:42:29.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.680+0000 7f2a02ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:42:29.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.680+0000 7f2a02ffd640 1 -- 172.21.15.37:0/3890306059 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29fc0b0040 con 0x7f2a0c137900 2024-04-26T00:42:29.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.680+0000 7f2a02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:29.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.680+0000 7f2a02ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:42:29.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.680+0000 7f2a02ffd640 10 monclient: dump: 2024-04-26T00:42:29.693 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:29.693 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:29.693 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:29.693 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:29.693 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:29.694 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:29.694 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:29.694 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:29.694 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:29.694 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:29.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.680+0000 7f2a03fff640 4 set_mon_vals no callback set 2024-04-26T00:42:29.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.680+0000 7f2a1096e640 10 monclient: _renew_subs 2024-04-26T00:42:29.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.680+0000 7f2a1096e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:29.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.680+0000 7f2a1096e640 1 -- 172.21.15.37:0/3890306059 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a0c13b600 con 0x7f2a0c137900 2024-04-26T00:42:29.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.680+0000 7f2a02ffd640 1 -- 172.21.15.37:0/3890306059 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f29fc0bd020 con 0x7f2a0c137900 2024-04-26T00:42:29.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.680+0000 7f2a1096e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:29.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.680+0000 7f2a1096e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:29.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.680+0000 7f2a1096e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:29.694 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.680+0000 7f2a1096e640 1 -- 172.21.15.37:0/3890306059 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a0c075910 con 0x7f2a0c137900 2024-04-26T00:42:29.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.680+0000 7f2a02ffd640 1 --2- 172.21.15.37:0/3890306059 >> v2:172.21.15.45:6812/2379918623 conn(0x7f29ec035eb0 0x7f29ec038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:29.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.680+0000 7f2a02ffd640 1 -- 172.21.15.37:0/3890306059 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(896..896 src has 822..896) ==== 3975+0+0 (secure 0 0 0) 0x7f29fc0e2fe0 con 0x7f2a0c137900 2024-04-26T00:42:29.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.680+0000 7f2a0a59c640 1 --2- 172.21.15.37:0/3890306059 >> v2:172.21.15.45:6812/2379918623 conn(0x7f29ec035eb0 0x7f29ec038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:29.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.680+0000 7f2a0a59c640 10 monclient: get_auth_request con 0x7f29ec035eb0 auth_method 0 2024-04-26T00:42:29.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.684+0000 7f2a0a59c640 1 --2- 172.21.15.37:0/3890306059 >> v2:172.21.15.45:6812/2379918623 conn(0x7f29ec035eb0 0x7f29ec038370 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f29f4004e30 tx=0x7f29f4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:29.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.684+0000 7f2a02ffd640 1 -- 172.21.15.37:0/3890306059 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f29fc0c4030 con 0x7f2a0c137900 2024-04-26T00:42:29.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.684+0000 7f2a02ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:29.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.684+0000 7f2a02ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:42:29.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.908+0000 7f2a1096e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:42:29.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.908+0000 7f2a1096e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:42:29.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.908+0000 7f2a1096e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:29.913 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.908+0000 7f2a1096e640 1 -- 172.21.15.37:0/3890306059 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2a0c066f40 con 0x7f2a0c137900 2024-04-26T00:42:29.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.908+0000 7f2a02ffd640 1 -- 172.21.15.37:0/3890306059 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1558 (secure 0 0 0) 0x7f29fc0160f0 con 0x7f2a0c137900 2024-04-26T00:42:29.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.908+0000 7f2a02ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:42:29.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.908+0000 7f2a02ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:42:29.914 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":160,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:42:29.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.912+0000 7f2a1096e640 1 -- 172.21.15.37:0/3890306059 >> v2:172.21.15.45:6812/2379918623 conn(0x7f29ec035eb0 msgr2=0x7f29ec038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:29.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.912+0000 7f2a1096e640 1 --2- 172.21.15.37:0/3890306059 >> v2:172.21.15.45:6812/2379918623 conn(0x7f29ec035eb0 0x7f29ec038370 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f29f4004e30 tx=0x7f29f4009040 comp rx=0 tx=0).stop 2024-04-26T00:42:29.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.912+0000 7f2a1096e640 10 monclient: shutdown 2024-04-26T00:42:29.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.912+0000 7f2a1096e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:42:29.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.912+0000 7f2a1096e640 1 -- 172.21.15.37:0/3890306059 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a0c137900 msgr2=0x7f2a0c138c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:29.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.912+0000 7f2a1096e640 1 --2- 172.21.15.37:0/3890306059 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a0c137900 0x7f2a0c138c30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f29fc0ae040 tx=0x7f29fc01ac20 comp rx=0 tx=0).stop 2024-04-26T00:42:29.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.912+0000 7f2a1096e640 1 -- 172.21.15.37:0/3890306059 shutdown_connections 2024-04-26T00:42:29.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.912+0000 7f2a1096e640 1 --2- 172.21.15.37:0/3890306059 >> v2:172.21.15.45:6812/2379918623 conn(0x7f29ec035eb0 0x7f29ec038370 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:29.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.912+0000 7f2a1096e640 1 --2- 172.21.15.37:0/3890306059 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a0c13c340 0x7f2a0c139170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:29.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.912+0000 7f2a1096e640 1 --2- 172.21.15.37:0/3890306059 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a0c137900 0x7f2a0c138c30 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:29.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.912+0000 7f2a1096e640 1 --2- 172.21.15.37:0/3890306059 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a0c136e10 0x7f2a0c135f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:29.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.912+0000 7f2a1096e640 1 -- 172.21.15.37:0/3890306059 >> 172.21.15.37:0/3890306059 conn(0x7f2a0c0a64e0 msgr2=0x7f2a0c13d360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:42:29.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.912+0000 7f2a1096e640 1 -- 172.21.15.37:0/3890306059 shutdown_connections 2024-04-26T00:42:29.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:29.912+0000 7f2a1096e640 1 -- 172.21.15.37:0/3890306059 wait complete. 2024-04-26T00:42:32.935 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:42:33.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:33.064+0000 7f0585f6c640 10 monclient: get_monmap_and_config 2024-04-26T00:42:33.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:33.064+0000 7f0585f6c640 10 monclient: build_initial_monmap 2024-04-26T00:42:33.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:33.064+0000 7f0585f6c640 10 monclient: monmap: 2024-04-26T00:42:33.068 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:42:33.068 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:33.068 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:42:33.069384+0000 2024-04-26T00:42:33.068 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:42:33.069384+0000 2024-04-26T00:42:33.068 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:42:33.068 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:33.068 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:42:33.068 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:42:33.068 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:42:33.069 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:42:33.069 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:42:33.069 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:33.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:33.064+0000 7f0585f6c640 1 Processor -- start 2024-04-26T00:42:33.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:33.064+0000 7f0585f6c640 1 -- start start 2024-04-26T00:42:33.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:33.064+0000 7f0585f6c640 10 monclient: init 2024-04-26T00:42:33.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:33.064+0000 7f0585f6c640 10 monclient: _reopen_session rank -1 2024-04-26T00:42:33.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:33.064+0000 7f0585f6c640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-04-26T00:42:33.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:33.064+0000 7f0585f6c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f0580136e10 0x7f058009c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:33.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:33.064+0000 7f0585f6c640 10 monclient(hunting): picked mon.noname-a con 0x7f0580136e10 addr v2:172.21.15.37:3300/0 2024-04-26T00:42:33.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:33.064+0000 7f0585f6c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f05801371f0 addr v1:172.21.15.45:6789/0 2024-04-26T00:42:33.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:33.064+0000 7f0585f6c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f058013c340 addr v1:172.21.15.37:6789/0 2024-04-26T00:42:33.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:33.064+0000 7f0585f6c640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f058009b680 con 0x7f058013c340 2024-04-26T00:42:33.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:33.064+0000 7f0585f6c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f058009bbd0 con 0x7f05801371f0 2024-04-26T00:42:33.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:33.064+0000 7f0585f6c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:33.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:33.064+0000 7f0585f6c640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f058009b8f0 con 0x7f0580136e10 2024-04-26T00:42:33.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:33.064+0000 7f0585f6c640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:33.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:33.064+0000 7f0585f6c640 10 monclient(hunting): authenticate will time out at 2334.375076s 2024-04-26T00:42:33.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:33.064+0000 7f057f7fe640 1 -- >> v2:172.21.15.37:3300/0 conn(0x7f0580136e10 msgr2=0x7f058009c5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:42:33.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:33.064+0000 7f057ffff640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f058013c340 legacy=0x7f058013e730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:42:33.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:33.064+0000 7f057f7fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f0580136e10 0x7f058009c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:42:33.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:33.064+0000 7f057effd640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7f05801371f0 0x7f0580137600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.37:36384/0 (socket says 172.21.15.37:36384) 2024-04-26T00:42:33.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:33.064+0000 7f057effd640 1 -- 172.21.15.37:0/2813615126 learned_addr learned my addr 172.21.15.37:0/2813615126 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:42:33.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:33.068+0000 7f057effd640 1 -- 172.21.15.37:0/2813615126 >> v1:172.21.15.45:6789/0 conn(0x7f05801371f0 legacy=0x7f0580137600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:33.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:33.268+0000 7f057ffff640 1 -- 172.21.15.37:0/2813615126 >> v1:172.21.15.37:6789/0 conn(0x7f058013c340 legacy=0x7f058013e730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:42:33.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:33.268+0000 7f057f7fe640 1 -- 172.21.15.37:0/2813615126 >> v2:172.21.15.37:3300/0 conn(0x7f0580136e10 msgr2=0x7f058009c5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:42:33.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:33.268+0000 7f057f7fe640 1 --2- 172.21.15.37:0/2813615126 >> v2:172.21.15.37:3300/0 conn(0x7f0580136e10 0x7f058009c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:42:33.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:33.672+0000 7f057ffff640 1 -- 172.21.15.37:0/2813615126 >> v1:172.21.15.37:6789/0 conn(0x7f058013c340 legacy=0x7f058013e730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:42:33.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:33.672+0000 7f057f7fe640 1 -- 172.21.15.37:0/2813615126 >> v2:172.21.15.37:3300/0 conn(0x7f0580136e10 msgr2=0x7f058009c5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:42:33.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:33.672+0000 7f057f7fe640 1 --2- 172.21.15.37:0/2813615126 >> v2:172.21.15.37:3300/0 conn(0x7f0580136e10 0x7f058009c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:42:34.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:34.476+0000 7f057ffff640 1 -- 172.21.15.37:0/2813615126 >> v1:172.21.15.37:6789/0 conn(0x7f058013c340 legacy=0x7f058013e730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:42:34.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:34.476+0000 7f057f7fe640 1 -- 172.21.15.37:0/2813615126 >> v2:172.21.15.37:3300/0 conn(0x7f0580136e10 msgr2=0x7f058009c5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:42:34.478 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:34.476+0000 7f057f7fe640 1 --2- 172.21.15.37:0/2813615126 >> v2:172.21.15.37:3300/0 conn(0x7f0580136e10 0x7f058009c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:42:36.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.068+0000 7f057d7fa640 10 monclient(hunting): tick 2024-04-26T00:42:36.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.068+0000 7f057d7fa640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:42:36.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.068+0000 7f057d7fa640 1 monclient(hunting): continuing hunt 2024-04-26T00:42:36.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.068+0000 7f057d7fa640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:42:36.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.068+0000 7f057d7fa640 1 -- 172.21.15.37:0/2813615126 >> v2:172.21.15.37:3300/0 conn(0x7f0580136e10 msgr2=0x7f058009c5b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:42:36.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.068+0000 7f057d7fa640 1 --2- 172.21.15.37:0/2813615126 >> v2:172.21.15.37:3300/0 conn(0x7f0580136e10 0x7f058009c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:36.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.068+0000 7f057d7fa640 1 -- 172.21.15.37:0/2813615126 >> v1:172.21.15.45:6789/0 conn(0x7f05801371f0 legacy=0x7f0580137600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:36.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.068+0000 7f057d7fa640 1 -- 172.21.15.37:0/2813615126 >> v1:172.21.15.37:6789/0 conn(0x7f058013c340 legacy=0x7f058013e730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:42:36.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.068+0000 7f057d7fa640 10 monclient: _add_conns ranks=[2,4] 2024-04-26T00:42:36.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.068+0000 7f057d7fa640 1 --2- 172.21.15.37:0/2813615126 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0560003750 0x7f0560005ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:36.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.068+0000 7f057d7fa640 10 monclient(hunting): picked mon.noname-b con 0x7f0560003750 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:42:36.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.068+0000 7f057d7fa640 1 --2- 172.21.15.37:0/2813615126 >> v2:172.21.15.45:3300/0 conn(0x7f05600062d0 0x7f0560008790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:36.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.068+0000 7f057d7fa640 10 monclient(hunting): picked mon.noname-c con 0x7f05600062d0 addr v2:172.21.15.45:3300/0 2024-04-26T00:42:36.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.068+0000 7f057d7fa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:36.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.068+0000 7f057d7fa640 1 -- 172.21.15.37:0/2813615126 --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f058009b8f0 con 0x7f05600062d0 2024-04-26T00:42:36.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.068+0000 7f057d7fa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:36.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.068+0000 7f057d7fa640 1 -- 172.21.15.37:0/2813615126 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0560008e30 con 0x7f0560003750 2024-04-26T00:42:36.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.068+0000 7f057d7fa640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:36.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.068+0000 7f057f7fe640 1 --2- 172.21.15.37:0/2813615126 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0560003750 0x7f0560005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:36.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.068+0000 7f057f7fe640 10 monclient(hunting): get_auth_request con 0x7f0560003750 auth_method 0 2024-04-26T00:42:36.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.068+0000 7f057f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:36.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.068+0000 7f057f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:36.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.068+0000 7f057f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:36.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.068+0000 7f057effd640 1 --2- 172.21.15.37:0/2813615126 >> v2:172.21.15.45:3300/0 conn(0x7f05600062d0 0x7f0560008790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:36.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.068+0000 7f057effd640 10 monclient(hunting): get_auth_request con 0x7f05600062d0 auth_method 0 2024-04-26T00:42:36.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.068+0000 7f057effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:36.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.068+0000 7f057effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:36.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.068+0000 7f057effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:36.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.068+0000 7f057f7fe640 1 -- 172.21.15.37:0/2813615126 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0560003750 msgr2=0x7f0560005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:42:36.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.068+0000 7f057f7fe640 1 -- 172.21.15.37:0/2813615126 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0560003750 msgr2=0x7f0560005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:42:36.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.068+0000 7f057f7fe640 1 --2- 172.21.15.37:0/2813615126 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0560003750 0x7f0560005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:42:36.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.068+0000 7f057effd640 1 -- 172.21.15.37:0/2813615126 >> v2:172.21.15.45:3300/0 conn(0x7f05600062d0 msgr2=0x7f0560008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:42:36.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.068+0000 7f057effd640 1 -- 172.21.15.37:0/2813615126 >> v2:172.21.15.45:3300/0 conn(0x7f05600062d0 msgr2=0x7f0560008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:42:36.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.068+0000 7f057effd640 1 --2- 172.21.15.37:0/2813615126 >> v2:172.21.15.45:3300/0 conn(0x7f05600062d0 0x7f0560008790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:42:36.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.068+0000 7f057effd640 1 --2- 172.21.15.37:0/2813615126 >> v2:172.21.15.45:3300/0 conn(0x7f05600062d0 0x7f0560008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:42:36.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.068+0000 7f057f7fe640 1 --2- 172.21.15.37:0/2813615126 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0560003750 0x7f0560005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:42:36.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.272+0000 7f057f7fe640 1 --2- 172.21.15.37:0/2813615126 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0560003750 0x7f0560005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:36.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.272+0000 7f057f7fe640 10 monclient(hunting): get_auth_request con 0x7f0560003750 auth_method 0 2024-04-26T00:42:36.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.272+0000 7f057f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:36.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.272+0000 7f057f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:36.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.272+0000 7f057f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:42:36.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.272+0000 7f057effd640 1 --2- 172.21.15.37:0/2813615126 >> v2:172.21.15.45:3300/0 conn(0x7f05600062d0 0x7f0560008790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:36.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.272+0000 7f057effd640 10 monclient(hunting): get_auth_request con 0x7f05600062d0 auth_method 0 2024-04-26T00:42:36.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.272+0000 7f057effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:36.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.272+0000 7f057effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:36.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.272+0000 7f057effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:42:36.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.272+0000 7f057f7fe640 1 -- 172.21.15.37:0/2813615126 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0560003750 msgr2=0x7f0560005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:42:36.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.272+0000 7f057f7fe640 1 -- 172.21.15.37:0/2813615126 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0560003750 msgr2=0x7f0560005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:42:36.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.272+0000 7f057f7fe640 1 --2- 172.21.15.37:0/2813615126 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0560003750 0x7f0560005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:42:36.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.272+0000 7f057effd640 1 -- 172.21.15.37:0/2813615126 >> v2:172.21.15.45:3300/0 conn(0x7f05600062d0 msgr2=0x7f0560008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:42:36.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.272+0000 7f057effd640 1 -- 172.21.15.37:0/2813615126 >> v2:172.21.15.45:3300/0 conn(0x7f05600062d0 msgr2=0x7f0560008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:42:36.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.272+0000 7f057effd640 1 --2- 172.21.15.37:0/2813615126 >> v2:172.21.15.45:3300/0 conn(0x7f05600062d0 0x7f0560008790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:42:36.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.272+0000 7f057effd640 1 --2- 172.21.15.37:0/2813615126 >> v2:172.21.15.45:3300/0 conn(0x7f05600062d0 0x7f0560008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:42:36.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.272+0000 7f057f7fe640 1 --2- 172.21.15.37:0/2813615126 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0560003750 0x7f0560005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:42:36.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.676+0000 7f057f7fe640 1 --2- 172.21.15.37:0/2813615126 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0560003750 0x7f0560005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:36.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.676+0000 7f057f7fe640 10 monclient(hunting): get_auth_request con 0x7f0560003750 auth_method 0 2024-04-26T00:42:36.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.676+0000 7f057f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:36.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.676+0000 7f057f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:36.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.676+0000 7f057f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:42:36.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.676+0000 7f057effd640 1 --2- 172.21.15.37:0/2813615126 >> v2:172.21.15.45:3300/0 conn(0x7f05600062d0 0x7f0560008790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:36.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.676+0000 7f057effd640 10 monclient(hunting): get_auth_request con 0x7f05600062d0 auth_method 0 2024-04-26T00:42:36.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.676+0000 7f057effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:36.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.676+0000 7f057effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:36.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.676+0000 7f057effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:42:36.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.676+0000 7f057f7fe640 1 -- 172.21.15.37:0/2813615126 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0560003750 msgr2=0x7f0560005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:42:36.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.676+0000 7f057f7fe640 1 -- 172.21.15.37:0/2813615126 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0560003750 msgr2=0x7f0560005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:42:36.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.676+0000 7f057f7fe640 1 --2- 172.21.15.37:0/2813615126 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0560003750 0x7f0560005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:42:36.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.676+0000 7f057f7fe640 1 --2- 172.21.15.37:0/2813615126 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0560003750 0x7f0560005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:42:36.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.676+0000 7f057effd640 1 -- 172.21.15.37:0/2813615126 >> v2:172.21.15.45:3300/0 conn(0x7f05600062d0 msgr2=0x7f0560008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:42:36.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.676+0000 7f057effd640 1 -- 172.21.15.37:0/2813615126 >> v2:172.21.15.45:3300/0 conn(0x7f05600062d0 msgr2=0x7f0560008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:42:36.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.676+0000 7f057effd640 1 --2- 172.21.15.37:0/2813615126 >> v2:172.21.15.45:3300/0 conn(0x7f05600062d0 0x7f0560008790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:42:36.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:36.676+0000 7f057effd640 1 --2- 172.21.15.37:0/2813615126 >> v2:172.21.15.45:3300/0 conn(0x7f05600062d0 0x7f0560008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:42:37.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:37.480+0000 7f057f7fe640 1 --2- 172.21.15.37:0/2813615126 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0560003750 0x7f0560005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:37.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:37.480+0000 7f057f7fe640 10 monclient(hunting): get_auth_request con 0x7f0560003750 auth_method 0 2024-04-26T00:42:37.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:37.480+0000 7f057f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:37.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:37.480+0000 7f057f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:37.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:37.480+0000 7f057f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:42:37.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:37.480+0000 7f057effd640 1 --2- 172.21.15.37:0/2813615126 >> v2:172.21.15.45:3300/0 conn(0x7f05600062d0 0x7f0560008790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:37.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:37.480+0000 7f057f7fe640 1 -- 172.21.15.37:0/2813615126 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0560003750 msgr2=0x7f0560005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:42:37.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:37.480+0000 7f057f7fe640 1 -- 172.21.15.37:0/2813615126 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0560003750 msgr2=0x7f0560005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:42:37.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:37.480+0000 7f057f7fe640 1 --2- 172.21.15.37:0/2813615126 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0560003750 0x7f0560005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:42:37.481 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:37.480+0000 7f057effd640 10 monclient(hunting): get_auth_request con 0x7f05600062d0 auth_method 0 2024-04-26T00:42:37.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:37.480+0000 7f057effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:37.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:37.480+0000 7f057effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:37.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:37.480+0000 7f057effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:42:37.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:37.480+0000 7f057f7fe640 1 --2- 172.21.15.37:0/2813615126 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0560003750 0x7f0560005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:42:37.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:37.480+0000 7f057effd640 1 -- 172.21.15.37:0/2813615126 >> v2:172.21.15.45:3300/0 conn(0x7f05600062d0 msgr2=0x7f0560008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:42:37.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:37.480+0000 7f057effd640 1 -- 172.21.15.37:0/2813615126 >> v2:172.21.15.45:3300/0 conn(0x7f05600062d0 msgr2=0x7f0560008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:42:37.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:37.480+0000 7f057effd640 1 --2- 172.21.15.37:0/2813615126 >> v2:172.21.15.45:3300/0 conn(0x7f05600062d0 0x7f0560008790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:42:37.482 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:37.480+0000 7f057effd640 1 --2- 172.21.15.37:0/2813615126 >> v2:172.21.15.45:3300/0 conn(0x7f05600062d0 0x7f0560008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:42:39.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.064+0000 7f057d7fa640 10 monclient(hunting): tick 2024-04-26T00:42:39.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.064+0000 7f057d7fa640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:42:39.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.064+0000 7f057d7fa640 1 monclient(hunting): continuing hunt 2024-04-26T00:42:39.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.064+0000 7f057d7fa640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:42:39.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.064+0000 7f057d7fa640 1 -- 172.21.15.37:0/2813615126 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0560003750 msgr2=0x7f0560005ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:42:39.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.064+0000 7f057d7fa640 1 --2- 172.21.15.37:0/2813615126 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0560003750 0x7f0560005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:39.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.064+0000 7f057d7fa640 1 -- 172.21.15.37:0/2813615126 >> v2:172.21.15.45:3300/0 conn(0x7f05600062d0 msgr2=0x7f0560008790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:42:39.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.064+0000 7f057d7fa640 1 --2- 172.21.15.37:0/2813615126 >> v2:172.21.15.45:3300/0 conn(0x7f05600062d0 0x7f0560008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:39.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.064+0000 7f057ffff640 1 -- 172.21.15.37:0/2813615126 reap_dead start 2024-04-26T00:42:39.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.064+0000 7f057d7fa640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-04-26T00:42:39.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.064+0000 7f057d7fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0560003b90 addr v1:172.21.15.45:6789/0 2024-04-26T00:42:39.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.064+0000 7f057d7fa640 1 --2- 172.21.15.37:0/2813615126 >> v2:172.21.15.37:3300/0 conn(0x7f0560007ea0 0x7f0560008280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:39.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.064+0000 7f057d7fa640 10 monclient(hunting): picked mon.noname-a con 0x7f0560007ea0 addr v2:172.21.15.37:3300/0 2024-04-26T00:42:39.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.064+0000 7f057d7fa640 1 --2- 172.21.15.37:0/2813615126 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05600087f0 0x7f056000d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:39.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.064+0000 7f057d7fa640 10 monclient(hunting): picked mon.noname-b con 0x7f05600087f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:42:39.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.064+0000 7f057d7fa640 1 -- 172.21.15.37:0/2813615126 --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05800a5d90 con 0x7f0560003b90 2024-04-26T00:42:39.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.064+0000 7f057d7fa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:39.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.064+0000 7f057d7fa640 1 -- 172.21.15.37:0/2813615126 --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f058009b8f0 con 0x7f0560007ea0 2024-04-26T00:42:39.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.064+0000 7f057d7fa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:39.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.064+0000 7f057d7fa640 1 -- 172.21.15.37:0/2813615126 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0560008e30 con 0x7f05600087f0 2024-04-26T00:42:39.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.064+0000 7f057d7fa640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:39.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.064+0000 7f057effd640 1 -- 172.21.15.37:0/2813615126 >> v2:172.21.15.37:3300/0 conn(0x7f0560007ea0 msgr2=0x7f0560008280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:42:39.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.064+0000 7f057effd640 1 --2- 172.21.15.37:0/2813615126 >> v2:172.21.15.37:3300/0 conn(0x7f0560007ea0 0x7f0560008280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:42:39.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.068+0000 7f057ffff640 1 --2- 172.21.15.37:0/2813615126 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05600087f0 0x7f056000d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:39.072 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.068+0000 7f057ffff640 10 monclient(hunting): get_auth_request con 0x7f05600087f0 auth_method 0 2024-04-26T00:42:39.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.068+0000 7f057ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:39.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.068+0000 7f057ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:39.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.068+0000 7f057ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:39.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.068+0000 7f057ffff640 1 -- 172.21.15.37:0/2813615126 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05600087f0 msgr2=0x7f056000d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:42:39.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.068+0000 7f057ffff640 1 -- 172.21.15.37:0/2813615126 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05600087f0 msgr2=0x7f056000d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:42:39.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.068+0000 7f057ffff640 1 --2- 172.21.15.37:0/2813615126 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05600087f0 0x7f056000d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:42:39.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.068+0000 7f057ffff640 1 --2- 172.21.15.37:0/2813615126 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05600087f0 0x7f056000d250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:42:39.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.068+0000 7f057f7fe640 1 -- 172.21.15.37:0/2813615126 >> v1:172.21.15.45:6789/0 conn(0x7f0560003b90 legacy=0x7f0560006780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:39.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.268+0000 7f057effd640 1 -- 172.21.15.37:0/2813615126 >> v2:172.21.15.37:3300/0 conn(0x7f0560007ea0 msgr2=0x7f0560008280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:42:39.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.268+0000 7f057effd640 1 --2- 172.21.15.37:0/2813615126 >> v2:172.21.15.37:3300/0 conn(0x7f0560007ea0 0x7f0560008280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:42:39.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.272+0000 7f057ffff640 1 --2- 172.21.15.37:0/2813615126 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05600087f0 0x7f056000d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:39.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.272+0000 7f057ffff640 10 monclient(hunting): get_auth_request con 0x7f05600087f0 auth_method 0 2024-04-26T00:42:39.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.272+0000 7f057ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:39.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.272+0000 7f057ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:39.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.272+0000 7f057ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:42:39.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.272+0000 7f057ffff640 1 -- 172.21.15.37:0/2813615126 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05600087f0 msgr2=0x7f056000d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:42:39.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.272+0000 7f057ffff640 1 -- 172.21.15.37:0/2813615126 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05600087f0 msgr2=0x7f056000d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:42:39.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.272+0000 7f057ffff640 1 --2- 172.21.15.37:0/2813615126 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05600087f0 0x7f056000d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:42:39.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.272+0000 7f057ffff640 1 --2- 172.21.15.37:0/2813615126 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05600087f0 0x7f056000d250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:42:39.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.536+0000 7f057e7fc640 1 -- 172.21.15.37:0/2813615126 <== mon.1 v1:172.21.15.45:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 888360632 0 0) 0x7f05800a5d90 con 0x7f0560003b90 2024-04-26T00:42:39.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.536+0000 7f057e7fc640 10 monclient(hunting): _negotiate 2024-04-26T00:42:39.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.536+0000 7f057e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:39.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.536+0000 7f057e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:39.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.536+0000 7f057e7fc640 10 monclient(hunting): my global_id is 274138 2024-04-26T00:42:39.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.536+0000 7f057e7fc640 1 -- 172.21.15.37:0/2813615126 --> v1:172.21.15.45:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f056c003530 con 0x7f0560003b90 2024-04-26T00:42:39.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.672+0000 7f057effd640 1 -- 172.21.15.37:0/2813615126 >> v2:172.21.15.37:3300/0 conn(0x7f0560007ea0 msgr2=0x7f0560008280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:42:39.671 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.672+0000 7f057effd640 1 --2- 172.21.15.37:0/2813615126 >> v2:172.21.15.37:3300/0 conn(0x7f0560007ea0 0x7f0560008280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:42:39.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.676+0000 7f057ffff640 1 --2- 172.21.15.37:0/2813615126 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05600087f0 0x7f056000d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:39.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.676+0000 7f057ffff640 10 monclient(hunting): get_auth_request con 0x7f05600087f0 auth_method 0 2024-04-26T00:42:39.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.676+0000 7f057ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:39.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.676+0000 7f057ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:39.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.676+0000 7f057ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:42:39.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.676+0000 7f057ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:39.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.676+0000 7f057ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:39.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.676+0000 7f057ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:39.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.676+0000 7f057ffff640 10 monclient(hunting): handle_auth_done global_id 254372 payload 274 2024-04-26T00:42:39.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.676+0000 7f057ffff640 1 -- 172.21.15.37:0/2813615126 >> v2:172.21.15.37:3300/0 conn(0x7f0560007ea0 msgr2=0x7f0560008280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:42:39.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.676+0000 7f057ffff640 1 --2- 172.21.15.37:0/2813615126 >> v2:172.21.15.37:3300/0 conn(0x7f0560007ea0 0x7f0560008280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:39.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.676+0000 7f057ffff640 1 -- 172.21.15.37:0/2813615126 >> v1:172.21.15.45:6789/0 conn(0x7f0560003b90 legacy=0x7f0560006780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:39.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.676+0000 7f057ffff640 10 monclient: _finish_hunting 0 2024-04-26T00:42:39.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.676+0000 7f057ffff640 1 monclient: found mon.noname-b 2024-04-26T00:42:39.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.676+0000 7f057ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:42:39.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.676+0000 7f057ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:42:39.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.676+0000 7f057ffff640 1 -- 172.21.15.37:0/2813615126 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f058009b680 con 0x7f05600087f0 2024-04-26T00:42:39.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.684+0000 7f057ffff640 1 -- 172.21.15.37:0/2813615126 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05600087f0 msgr2=0x7f056000d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:39.687 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.684+0000 7f057ffff640 1 --2- 172.21.15.37:0/2813615126 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05600087f0 0x7f056000d250 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f058006e430 tx=0x7f057801d300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9808b5306771e2c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:39.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.684+0000 7f057e7fc640 1 -- 172.21.15.37:0/2813615126 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f057800e0d0 con 0x7f05600087f0 2024-04-26T00:42:39.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.684+0000 7f057e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:39.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.684+0000 7f057e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:42:39.688 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.684+0000 7f057e7fc640 10 monclient: dump: 2024-04-26T00:42:39.688 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:39.688 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:39.688 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:39.688 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:39.688 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:39.689 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:39.689 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:39.689 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:39.689 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:39.689 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:39.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.684+0000 7f057e7fc640 10 monclient: _finish_auth 0 2024-04-26T00:42:39.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.684+0000 7f057e7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:42:39.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.684+0000 7f057e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:42:39.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.684+0000 7f0585f6c640 5 monclient: authenticate success, global_id 254372 2024-04-26T00:42:39.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.684+0000 7f057e7fc640 1 -- 172.21.15.37:0/2813615126 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f057800e270 con 0x7f05600087f0 2024-04-26T00:42:39.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.684+0000 7f0585f6c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:42:39.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.684+0000 7f057e7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:42:39.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.684+0000 7f0585f6c640 10 monclient: get_monmap_and_config success 2024-04-26T00:42:39.689 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.684+0000 7f0585f6c640 4 set_mon_vals no callback set 2024-04-26T00:42:39.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.684+0000 7f057e7fc640 1 -- 172.21.15.37:0/2813615126 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05780a5dd0 con 0x7f05600087f0 2024-04-26T00:42:39.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.684+0000 7f057e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:39.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.684+0000 7f057e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:42:39.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.684+0000 7f057e7fc640 10 monclient: dump: 2024-04-26T00:42:39.690 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:39.690 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:39.690 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:39.690 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:39.690 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:39.690 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:39.690 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:39.690 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:39.690 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:39.690 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:39.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.684+0000 7f0585f6c640 10 monclient: shutdown 2024-04-26T00:42:39.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.684+0000 7f0585f6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:42:39.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.684+0000 7f0585f6c640 1 -- 172.21.15.37:0/2813615126 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05600087f0 msgr2=0x7f056000d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:39.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.684+0000 7f0585f6c640 1 --2- 172.21.15.37:0/2813615126 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05600087f0 0x7f056000d250 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f058006e430 tx=0x7f057801d300 comp rx=0 tx=0).stop 2024-04-26T00:42:39.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.684+0000 7f0585f6c640 1 -- 172.21.15.37:0/2813615126 shutdown_connections 2024-04-26T00:42:39.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.684+0000 7f0585f6c640 1 --2- 172.21.15.37:0/2813615126 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05600087f0 0x7f056000d250 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:39.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.684+0000 7f0585f6c640 1 --2- 172.21.15.37:0/2813615126 >> v2:172.21.15.37:3300/0 conn(0x7f0560007ea0 0x7f0560008280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:39.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.684+0000 7f0585f6c640 1 -- 172.21.15.37:0/2813615126 >> 172.21.15.37:0/2813615126 conn(0x7f05800a64e0 msgr2=0x7f05800a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:42:39.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.684+0000 7f0585f6c640 1 -- 172.21.15.37:0/2813615126 shutdown_connections 2024-04-26T00:42:39.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.684+0000 7f0585f6c640 1 -- 172.21.15.37:0/2813615126 wait complete. 2024-04-26T00:42:39.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.688+0000 7f0585f6c640 10 monclient: build_initial_monmap 2024-04-26T00:42:39.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.688+0000 7f0585f6c640 10 monclient: monmap: 2024-04-26T00:42:39.693 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:42:39.693 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:42:39.693 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:42:39.693 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:42:39.693 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:42:39.693 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:42:39.693 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:42:39.693 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:42:39.693 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:42:39.693 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:39.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.688+0000 7f0585f6c640 1 Processor -- start 2024-04-26T00:42:39.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.688+0000 7f0585f6c640 1 -- start start 2024-04-26T00:42:39.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.688+0000 7f0585f6c640 10 monclient: init 2024-04-26T00:42:39.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.688+0000 7f0585f6c640 10 monclient: _reopen_session rank -1 2024-04-26T00:42:39.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f0585f6c640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:42:39.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f0585f6c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0560003b90 0x7f05801350f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:39.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f0585f6c640 10 monclient(hunting): picked mon.noname-a con 0x7f0560003b90 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:42:39.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f0585f6c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0560007ea0 0x7f0580135630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:39.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f0585f6c640 10 monclient(hunting): picked mon.noname-b con 0x7f0560007ea0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:42:39.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f057f7fe640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0560003b90 msgr2=0x7f05801350f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:42:39.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f0585f6c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05600087f0 0x7f058011fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:39.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f0585f6c640 10 monclient(hunting): picked mon.noname-c con 0x7f05600087f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:42:39.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f057f7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0560003b90 0x7f05801350f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:42:39.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f0585f6c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:39.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f0585f6c640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f0580136120 con 0x7f0560003b90 2024-04-26T00:42:39.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f0585f6c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:39.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f0585f6c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0580135fa0 con 0x7f0560007ea0 2024-04-26T00:42:39.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f0585f6c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:39.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f0585f6c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f05801362a0 con 0x7f05600087f0 2024-04-26T00:42:39.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f0585f6c640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:39.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f0585f6c640 10 monclient(hunting): authenticate will time out at 2341.000283s 2024-04-26T00:42:39.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f057ffff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05600087f0 0x7f058011fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:39.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f057ffff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05600087f0 0x7f058011fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:46408/0 (socket says 172.21.15.37:46408) 2024-04-26T00:42:39.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f057ffff640 1 -- 172.21.15.37:0/505892390 learned_addr learned my addr 172.21.15.37:0/505892390 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:42:39.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f057ffff640 10 monclient(hunting): get_auth_request con 0x7f05600087f0 auth_method 0 2024-04-26T00:42:39.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f057ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:39.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f057ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:39.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f057ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:39.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f057ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:39.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f057ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:39.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f057ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:39.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f057effd640 1 --2- 172.21.15.37:0/505892390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0560007ea0 0x7f0580135630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:39.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f057effd640 10 monclient(hunting): get_auth_request con 0x7f0560007ea0 auth_method 0 2024-04-26T00:42:39.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f057effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:39.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f057effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:39.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f057effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:39.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f057ffff640 10 monclient(hunting): handle_auth_done global_id 254375 payload 824 2024-04-26T00:42:39.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f057ffff640 1 -- 172.21.15.37:0/505892390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0560007ea0 msgr2=0x7f0580135630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:39.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f057ffff640 1 --2- 172.21.15.37:0/505892390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0560007ea0 0x7f0580135630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:39.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f057ffff640 1 -- 172.21.15.37:0/505892390 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0560003b90 msgr2=0x7f05801350f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:42:39.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f057ffff640 1 --2- 172.21.15.37:0/505892390 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0560003b90 0x7f05801350f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:39.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f057ffff640 10 monclient: _finish_hunting 0 2024-04-26T00:42:39.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f057ffff640 1 monclient: found mon.noname-c 2024-04-26T00:42:39.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f057ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:42:39.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f057ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:42:39.699 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f057ffff640 1 -- 172.21.15.37:0/505892390 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0580121640 con 0x7f05600087f0 2024-04-26T00:42:39.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f057ffff640 1 -- 172.21.15.37:0/505892390 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05600087f0 msgr2=0x7f058011fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:39.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f057ffff640 1 --2- 172.21.15.37:0/505892390 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05600087f0 0x7f058011fc30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f057801af30 tx=0x7f057801a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:39.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f0577fff640 1 -- 172.21.15.37:0/505892390 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05780b68a0 con 0x7f05600087f0 2024-04-26T00:42:39.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f0577fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:39.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f0577fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:42:39.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f0577fff640 10 monclient: dump: 2024-04-26T00:42:39.701 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:39.701 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:39.701 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:39.701 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:39.701 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:39.701 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:39.701 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:39.702 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:39.702 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:39.702 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:39.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f0577fff640 10 monclient: _finish_auth 0 2024-04-26T00:42:39.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f0577fff640 10 monclient: _check_auth_tickets 2024-04-26T00:42:39.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f0577fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:42:39.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f0577fff640 1 -- 172.21.15.37:0/505892390 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05780b6a40 con 0x7f05600087f0 2024-04-26T00:42:39.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f0577fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:42:39.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f0577fff640 1 -- 172.21.15.37:0/505892390 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05780b6e30 con 0x7f05600087f0 2024-04-26T00:42:39.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f0585f6c640 5 monclient: authenticate success, global_id 254375 2024-04-26T00:42:39.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f0585f6c640 10 monclient: _renew_subs 2024-04-26T00:42:39.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f0585f6c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:39.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f0585f6c640 1 -- 172.21.15.37:0/505892390 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05801c2440 con 0x7f05600087f0 2024-04-26T00:42:39.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f057dffb640 4 set_mon_vals no callback set 2024-04-26T00:42:39.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f0577fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:39.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f0577fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:42:39.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f0577fff640 10 monclient: dump: 2024-04-26T00:42:39.703 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:39.703 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:39.703 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:39.703 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:39.703 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:39.703 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:39.703 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:39.703 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:39.703 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:39.704 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:39.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f0585f6c640 10 monclient: _renew_subs 2024-04-26T00:42:39.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f0585f6c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:39.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.692+0000 7f0585f6c640 1 -- 172.21.15.37:0/505892390 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0580135b70 con 0x7f05600087f0 2024-04-26T00:42:39.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.696+0000 7f0577fff640 1 -- 172.21.15.37:0/505892390 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f05780be020 con 0x7f05600087f0 2024-04-26T00:42:39.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.696+0000 7f0577fff640 1 --2- 172.21.15.37:0/505892390 >> v2:172.21.15.45:6812/2379918623 conn(0x7f05600379f0 0x7f0560039eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:39.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.696+0000 7f0585f6c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:39.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.696+0000 7f0585f6c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:39.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.696+0000 7f0585f6c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:39.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.696+0000 7f0577fff640 1 -- 172.21.15.37:0/505892390 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(896..896 src has 822..896) ==== 3975+0+0 (secure 0 0 0) 0x7f05780e3320 con 0x7f05600087f0 2024-04-26T00:42:39.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.696+0000 7f0585f6c640 1 -- 172.21.15.37:0/505892390 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0580075910 con 0x7f05600087f0 2024-04-26T00:42:39.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.696+0000 7f057f7fe640 1 --2- 172.21.15.37:0/505892390 >> v2:172.21.15.45:6812/2379918623 conn(0x7f05600379f0 0x7f0560039eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:39.704 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.696+0000 7f057f7fe640 10 monclient: get_auth_request con 0x7f05600379f0 auth_method 0 2024-04-26T00:42:39.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.700+0000 7f057f7fe640 1 --2- 172.21.15.37:0/505892390 >> v2:172.21.15.45:6812/2379918623 conn(0x7f05600379f0 0x7f0560039eb0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f05700038a0 tx=0x7f057000d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:39.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.700+0000 7f0577fff640 1 -- 172.21.15.37:0/505892390 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f0578017080 con 0x7f05600087f0 2024-04-26T00:42:39.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.700+0000 7f0577fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:39.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.700+0000 7f0577fff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:42:39.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.928+0000 7f0585f6c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:42:39.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.928+0000 7f0585f6c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:42:39.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.928+0000 7f0585f6c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:39.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.928+0000 7f0585f6c640 1 -- 172.21.15.37:0/505892390 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f058013c040 con 0x7f05600087f0 2024-04-26T00:42:39.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.928+0000 7f0577fff640 1 -- 172.21.15.37:0/505892390 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1551 (secure 0 0 0) 0x7f05780e30e0 con 0x7f05600087f0 2024-04-26T00:42:39.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.928+0000 7f0577fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:42:39.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.928+0000 7f0577fff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:42:39.934 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":164,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:42:39.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.932+0000 7f0585f6c640 1 -- 172.21.15.37:0/505892390 >> v2:172.21.15.45:6812/2379918623 conn(0x7f05600379f0 msgr2=0x7f0560039eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:39.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.932+0000 7f0585f6c640 1 --2- 172.21.15.37:0/505892390 >> v2:172.21.15.45:6812/2379918623 conn(0x7f05600379f0 0x7f0560039eb0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f05700038a0 tx=0x7f057000d040 comp rx=0 tx=0).stop 2024-04-26T00:42:39.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.932+0000 7f0585f6c640 10 monclient: shutdown 2024-04-26T00:42:39.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.932+0000 7f0585f6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:42:39.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.932+0000 7f0585f6c640 1 -- 172.21.15.37:0/505892390 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05600087f0 msgr2=0x7f058011fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:39.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.932+0000 7f0585f6c640 1 --2- 172.21.15.37:0/505892390 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05600087f0 0x7f058011fc30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f057801af30 tx=0x7f057801a0a0 comp rx=0 tx=0).stop 2024-04-26T00:42:39.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.932+0000 7f0585f6c640 1 -- 172.21.15.37:0/505892390 shutdown_connections 2024-04-26T00:42:39.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.932+0000 7f0585f6c640 1 --2- 172.21.15.37:0/505892390 >> v2:172.21.15.45:6812/2379918623 conn(0x7f05600379f0 0x7f0560039eb0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:39.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.932+0000 7f0585f6c640 1 --2- 172.21.15.37:0/505892390 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05600087f0 0x7f058011fc30 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:39.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.932+0000 7f0585f6c640 1 --2- 172.21.15.37:0/505892390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0560007ea0 0x7f0580135630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:39.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.932+0000 7f0585f6c640 1 --2- 172.21.15.37:0/505892390 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0560003b90 0x7f05801350f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:39.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.932+0000 7f0585f6c640 1 -- 172.21.15.37:0/505892390 >> 172.21.15.37:0/505892390 conn(0x7f05800a64e0 msgr2=0x7f058013abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:42:39.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.932+0000 7f0585f6c640 1 -- 172.21.15.37:0/505892390 shutdown_connections 2024-04-26T00:42:39.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:39.932+0000 7f0585f6c640 1 -- 172.21.15.37:0/505892390 wait complete. 2024-04-26T00:42:39.954 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-26T00:42:39.954 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-04-26T00:42:39.954 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:42:40.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.076+0000 7f848f83c640 10 monclient: get_monmap_and_config 2024-04-26T00:42:40.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848f83c640 10 monclient: build_initial_monmap 2024-04-26T00:42:40.082 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848f83c640 10 monclient: monmap: 2024-04-26T00:42:40.083 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:42:40.083 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:40.083 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:42:40.083964+0000 2024-04-26T00:42:40.083 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:42:40.083964+0000 2024-04-26T00:42:40.083 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:42:40.083 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:40.083 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:42:40.083 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:42:40.083 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:42:40.083 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:42:40.083 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:42:40.083 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:40.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848f83c640 1 Processor -- start 2024-04-26T00:42:40.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848f83c640 1 -- start start 2024-04-26T00:42:40.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848f83c640 10 monclient: init 2024-04-26T00:42:40.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848f83c640 10 monclient: _reopen_session rank -1 2024-04-26T00:42:40.085 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848f83c640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-04-26T00:42:40.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848f83c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f84880a3d90 addr v1:172.21.15.45:6789/0 2024-04-26T00:42:40.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848f83c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f84880a4170 0x7f84880a45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:40.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848f83c640 10 monclient(hunting): picked mon.noname-c con 0x7f84880a4170 addr v2:172.21.15.45:3300/0 2024-04-26T00:42:40.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848f83c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f848813c3e0 0x7f848813e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:40.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848f83c640 10 monclient(hunting): picked mon.noname-a con 0x7f848813c3e0 addr v2:172.21.15.37:3300/0 2024-04-26T00:42:40.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848f83c640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84880a5150 con 0x7f84880a3d90 2024-04-26T00:42:40.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848f83c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:40.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848f83c640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f848809c650 con 0x7f848813c3e0 2024-04-26T00:42:40.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848f83c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:40.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848f83c640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f848809b280 con 0x7f84880a4170 2024-04-26T00:42:40.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848f83c640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:40.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848f83c640 10 monclient(hunting): authenticate will time out at 2341.389697s 2024-04-26T00:42:40.086 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848ddd9640 1 -- >> v2:172.21.15.37:3300/0 conn(0x7f848813c3e0 msgr2=0x7f848813e7d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:42:40.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848ddd9640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f848813c3e0 0x7f848813e7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:42:40.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848cdd7640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f84880a4170 0x7f84880a45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:40.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848cdd7640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f84880a4170 0x7f84880a45e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.37:53776/0 (socket says 172.21.15.37:53776) 2024-04-26T00:42:40.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848cdd7640 1 -- 172.21.15.37:0/3725025457 learned_addr learned my addr 172.21.15.37:0/3725025457 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:42:40.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848cdd7640 10 monclient(hunting): get_auth_request con 0x7f84880a4170 auth_method 0 2024-04-26T00:42:40.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848cdd7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:40.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848cdd7640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:40.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848cdd7640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:40.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848d5d8640 1 --1- 172.21.15.37:0/3725025457 >> v1:172.21.15.45:6789/0 conn(0x7f84880a3d90 0x7f848809bf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.37:44858/0 (socket says 172.21.15.37:44858) 2024-04-26T00:42:40.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848cdd7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:40.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848cdd7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:40.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848cdd7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:40.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848cdd7640 10 monclient(hunting): handle_auth_done global_id 274144 payload 274 2024-04-26T00:42:40.087 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848cdd7640 1 -- 172.21.15.37:0/3725025457 >> v2:172.21.15.37:3300/0 conn(0x7f848813c3e0 msgr2=0x7f848813e7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:42:40.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848cdd7640 1 --2- 172.21.15.37:0/3725025457 >> v2:172.21.15.37:3300/0 conn(0x7f848813c3e0 0x7f848813e7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:40.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848cdd7640 1 -- 172.21.15.37:0/3725025457 >> v1:172.21.15.45:6789/0 conn(0x7f84880a3d90 legacy=0x7f848809bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:42:40.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848cdd7640 10 monclient: _finish_hunting 0 2024-04-26T00:42:40.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848cdd7640 1 monclient: found mon.noname-c 2024-04-26T00:42:40.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848cdd7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:42:40.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848cdd7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:42:40.088 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.080+0000 7f848cdd7640 1 -- 172.21.15.37:0/3725025457 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f848809b560 con 0x7f84880a4170 2024-04-26T00:42:40.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.084+0000 7f848cdd7640 1 -- 172.21.15.37:0/3725025457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84880a4170 msgr2=0x7f84880a45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:40.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.084+0000 7f848cdd7640 1 --2- 172.21.15.37:0/3725025457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84880a4170 0x7f84880a45e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f848809d140 tx=0x7f847c01c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=316ecedb7de2d3f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:40.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.084+0000 7f847bfff640 1 -- 172.21.15.37:0/3725025457 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f847c0b0070 con 0x7f84880a4170 2024-04-26T00:42:40.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.084+0000 7f847bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:40.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.084+0000 7f847bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:42:40.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.084+0000 7f847bfff640 10 monclient: dump: 2024-04-26T00:42:40.091 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:40.091 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:40.091 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:40.091 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:40.091 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:40.091 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:40.091 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:40.091 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:40.091 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:40.091 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:40.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.084+0000 7f847bfff640 10 monclient: _finish_auth 0 2024-04-26T00:42:40.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.084+0000 7f847bfff640 10 monclient: _check_auth_tickets 2024-04-26T00:42:40.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.084+0000 7f847bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:42:40.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.084+0000 7f847bfff640 1 -- 172.21.15.37:0/3725025457 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f847c0ab600 con 0x7f84880a4170 2024-04-26T00:42:40.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.084+0000 7f847bfff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:42:40.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.084+0000 7f847bfff640 1 -- 172.21.15.37:0/3725025457 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f847c0ab9d0 con 0x7f84880a4170 2024-04-26T00:42:40.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.084+0000 7f847bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:40.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.084+0000 7f847bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:42:40.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.084+0000 7f847bfff640 10 monclient: dump: 2024-04-26T00:42:40.092 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:40.092 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:40.092 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:40.092 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:40.093 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:40.093 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:40.093 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:40.093 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:40.093 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:40.093 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:40.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.084+0000 7f848f83c640 5 monclient: authenticate success, global_id 274144 2024-04-26T00:42:40.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.084+0000 7f848f83c640 10 monclient: get_monmap_and_config success 2024-04-26T00:42:40.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.084+0000 7f848f83c640 4 set_mon_vals no callback set 2024-04-26T00:42:40.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848f83c640 10 monclient: shutdown 2024-04-26T00:42:40.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848f83c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:42:40.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848f83c640 1 -- 172.21.15.37:0/3725025457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84880a4170 msgr2=0x7f84880a45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:40.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848f83c640 1 --2- 172.21.15.37:0/3725025457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84880a4170 0x7f84880a45e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f848809d140 tx=0x7f847c01c3c0 comp rx=0 tx=0).stop 2024-04-26T00:42:40.093 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848f83c640 1 -- 172.21.15.37:0/3725025457 shutdown_connections 2024-04-26T00:42:40.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848f83c640 1 --2- 172.21.15.37:0/3725025457 >> v2:172.21.15.37:3300/0 conn(0x7f848813c3e0 0x7f848813e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:40.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848f83c640 1 --2- 172.21.15.37:0/3725025457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84880a4170 0x7f84880a45e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:40.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848f83c640 1 -- 172.21.15.37:0/3725025457 >> 172.21.15.37:0/3725025457 conn(0x7f84880aa0a0 msgr2=0x7f84880a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:42:40.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848f83c640 1 -- 172.21.15.37:0/3725025457 shutdown_connections 2024-04-26T00:42:40.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848f83c640 1 -- 172.21.15.37:0/3725025457 wait complete. 2024-04-26T00:42:40.099 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848f83c640 10 monclient: build_initial_monmap 2024-04-26T00:42:40.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848f83c640 10 monclient: monmap: 2024-04-26T00:42:40.100 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:42:40.100 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:42:40.100 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:42:40.100 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:42:40.100 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:42:40.100 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:42:40.100 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:42:40.100 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:42:40.100 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:42:40.100 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:40.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848f83c640 1 Processor -- start 2024-04-26T00:42:40.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848f83c640 1 -- start start 2024-04-26T00:42:40.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848f83c640 10 monclient: init 2024-04-26T00:42:40.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848f83c640 10 monclient: _reopen_session rank -1 2024-04-26T00:42:40.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848f83c640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:42:40.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848f83c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84880a3d90 0x7f8488139ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:40.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848f83c640 10 monclient(hunting): picked mon.noname-b con 0x7f84880a3d90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:42:40.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848f83c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84880a4170 0x7f848813a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:40.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848f83c640 10 monclient(hunting): picked mon.noname-a con 0x7f84880a4170 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:42:40.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848f83c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f848813c3e0 0x7f848813a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:40.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848f83c640 10 monclient(hunting): picked mon.noname-c con 0x7f848813c3e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:42:40.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848f83c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:40.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848f83c640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f84881418c0 con 0x7f84880a4170 2024-04-26T00:42:40.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848f83c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:40.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848f83c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8488141740 con 0x7f84880a3d90 2024-04-26T00:42:40.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848f83c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:40.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848f83c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8488141a40 con 0x7f848813c3e0 2024-04-26T00:42:40.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848f83c640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:40.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848f83c640 10 monclient(hunting): authenticate will time out at 2341.398067s 2024-04-26T00:42:40.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848cdd7640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84880a4170 msgr2=0x7f848813a3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:42:40.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848cdd7640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84880a4170 0x7f848813a3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:42:40.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848ddd9640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f848813c3e0 0x7f848813a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:40.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848ddd9640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f848813c3e0 0x7f848813a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:48482/0 (socket says 172.21.15.37:48482) 2024-04-26T00:42:40.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848ddd9640 1 -- 172.21.15.37:0/2764039216 learned_addr learned my addr 172.21.15.37:0/2764039216 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:42:40.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848ddd9640 10 monclient(hunting): get_auth_request con 0x7f848813c3e0 auth_method 0 2024-04-26T00:42:40.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848ddd9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:40.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848ddd9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:40.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848ddd9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:40.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848ddd9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:40.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848ddd9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:40.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848ddd9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:40.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848d5d8640 1 --2- 172.21.15.37:0/2764039216 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84880a3d90 0x7f8488139ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:40.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848d5d8640 10 monclient(hunting): get_auth_request con 0x7f84880a3d90 auth_method 0 2024-04-26T00:42:40.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848d5d8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:40.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848d5d8640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:40.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848d5d8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:40.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848ddd9640 10 monclient(hunting): handle_auth_done global_id 254378 payload 824 2024-04-26T00:42:40.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848ddd9640 1 -- 172.21.15.37:0/2764039216 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84880a3d90 msgr2=0x7f8488139ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:40.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848ddd9640 1 --2- 172.21.15.37:0/2764039216 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f84880a3d90 0x7f8488139ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:40.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848ddd9640 1 -- 172.21.15.37:0/2764039216 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84880a4170 msgr2=0x7f848813a3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:42:40.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848ddd9640 1 --2- 172.21.15.37:0/2764039216 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84880a4170 0x7f848813a3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:40.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848ddd9640 10 monclient: _finish_hunting 0 2024-04-26T00:42:40.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848ddd9640 1 monclient: found mon.noname-c 2024-04-26T00:42:40.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848ddd9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:42:40.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848ddd9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:42:40.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.088+0000 7f848ddd9640 1 -- 172.21.15.37:0/2764039216 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f848809ada0 con 0x7f848813c3e0 2024-04-26T00:42:40.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f848ddd9640 1 -- 172.21.15.37:0/2764039216 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f848813c3e0 msgr2=0x7f848813a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:40.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f848ddd9640 1 --2- 172.21.15.37:0/2764039216 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f848813c3e0 0x7f848813a920 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f848400b440 tx=0x7f848400b900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:40.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f8479ffb640 1 -- 172.21.15.37:0/2764039216 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8484017070 con 0x7f848813c3e0 2024-04-26T00:42:40.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f8479ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:40.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f8479ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:42:40.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f8479ffb640 10 monclient: dump: 2024-04-26T00:42:40.104 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:40.105 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:40.105 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:40.105 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:40.105 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:40.105 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:40.105 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:40.105 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:40.105 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:40.105 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:40.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f8479ffb640 10 monclient: _finish_auth 0 2024-04-26T00:42:40.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f8479ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:42:40.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f8479ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:42:40.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f8479ffb640 1 -- 172.21.15.37:0/2764039216 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84840060d0 con 0x7f848813c3e0 2024-04-26T00:42:40.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f8479ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:42:40.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f8479ffb640 1 -- 172.21.15.37:0/2764039216 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84840064c0 con 0x7f848813c3e0 2024-04-26T00:42:40.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f8479ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:40.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f8479ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:42:40.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f8479ffb640 10 monclient: dump: 2024-04-26T00:42:40.106 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:40.106 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:40.106 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:40.106 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:40.106 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:40.106 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:40.106 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:40.106 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:40.107 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:40.107 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:40.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f848f83c640 5 monclient: authenticate success, global_id 254378 2024-04-26T00:42:40.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f848f83c640 10 monclient: _renew_subs 2024-04-26T00:42:40.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f847b7fe640 4 set_mon_vals no callback set 2024-04-26T00:42:40.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f848f83c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:40.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f848f83c640 1 -- 172.21.15.37:0/2764039216 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f848813cd10 con 0x7f848813c3e0 2024-04-26T00:42:40.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f848f83c640 10 monclient: _renew_subs 2024-04-26T00:42:40.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f848f83c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:40.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f848f83c640 1 -- 172.21.15.37:0/2764039216 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8488133870 con 0x7f848813c3e0 2024-04-26T00:42:40.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f848f83c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:40.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f848f83c640 1 --2- 172.21.15.37:0/2764039216 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f84881cee80 0x7f84881cf260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:40.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f848f83c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:40.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f848f83c640 1 -- 172.21.15.37:0/2764039216 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f848804f510 con 0x7f84881cee80 2024-04-26T00:42:40.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f848d5d8640 1 --2- 172.21.15.37:0/2764039216 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f84881cee80 0x7f84881cf260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:40.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f848d5d8640 10 monclient: get_auth_request con 0x7f84881cee80 auth_method 0 2024-04-26T00:42:40.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f848d5d8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:40.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f848d5d8640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:40.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f848d5d8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:40.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f848d5d8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:40.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f848d5d8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:40.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f848d5d8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:40.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f8479ffb640 1 -- 172.21.15.37:0/2764039216 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f84840066e0 con 0x7f848813c3e0 2024-04-26T00:42:40.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f848d5d8640 10 monclient(hunting): handle_auth_done global_id 254381 payload 824 2024-04-26T00:42:40.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f848d5d8640 1 -- 172.21.15.37:0/2764039216 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f84881320d0 con 0x7f84881cee80 2024-04-26T00:42:40.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f8479ffb640 1 --2- 172.21.15.37:0/2764039216 >> v2:172.21.15.45:6812/2379918623 conn(0x7f846c035e30 0x7f846c0382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:40.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f8479ffb640 1 -- 172.21.15.37:0/2764039216 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(896..896 src has 822..896) ==== 3975+0+0 (secure 0 0 0) 0x7f8484047de0 con 0x7f848813c3e0 2024-04-26T00:42:40.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f848cdd7640 1 --2- 172.21.15.37:0/2764039216 >> v2:172.21.15.45:6812/2379918623 conn(0x7f846c035e30 0x7f846c0382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:40.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.092+0000 7f848cdd7640 10 monclient: get_auth_request con 0x7f846c035e30 auth_method 0 2024-04-26T00:42:40.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.096+0000 7f848d5d8640 1 -- 172.21.15.37:0/2764039216 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f84881cee80 msgr2=0x7f84881cf260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:40.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.096+0000 7f848d5d8640 1 --2- 172.21.15.37:0/2764039216 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f84881cee80 0x7f84881cf260 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f848000e840 tx=0x7f848000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:40.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.096+0000 7f8479ffb640 1 -- 172.21.15.37:0/2764039216 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f848000c6d0 con 0x7f84881cee80 2024-04-26T00:42:40.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.096+0000 7f8479ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:40.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.096+0000 7f8479ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:42:40.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.096+0000 7f8479ffb640 10 monclient: dump: 2024-04-26T00:42:40.109 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:40.109 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:40.109 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:40.109 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:40.109 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:40.109 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:40.109 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:40.109 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:40.109 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:40.110 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:40.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.096+0000 7f8479ffb640 1 -- 172.21.15.37:0/2764039216 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8480018070 con 0x7f84881cee80 2024-04-26T00:42:40.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.096+0000 7f8479ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:40.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.096+0000 7f8479ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:42:40.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.096+0000 7f8479ffb640 1 -- 172.21.15.37:0/2764039216 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f84881cee80 msgr2=0x7f84881cf260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:40.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.096+0000 7f8479ffb640 1 --2- 172.21.15.37:0/2764039216 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f84881cee80 0x7f84881cf260 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f848000e840 tx=0x7f848000ed00 comp rx=0 tx=0).stop 2024-04-26T00:42:40.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.096+0000 7f8479ffb640 1 -- 172.21.15.37:0/2764039216 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f84881cee80 msgr2=0x7f84881cf260 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:42:40.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.096+0000 7f8479ffb640 1 --2- 172.21.15.37:0/2764039216 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f84881cee80 0x7f84881cf260 secure :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f848000e840 tx=0x7f848000ed00 comp rx=0 tx=0).stop 2024-04-26T00:42:40.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.096+0000 7f848cdd7640 1 --2- 172.21.15.37:0/2764039216 >> v2:172.21.15.45:6812/2379918623 conn(0x7f846c035e30 0x7f846c0382f0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f84880a57e0 tx=0x7f847c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:40.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.100+0000 7f848f83c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:42:40.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.100+0000 7f848f83c640 1 --2- 172.21.15.37:0/2764039216 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f84881cf7a0 0x7f84880a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:40.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.100+0000 7f848f83c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:40.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.100+0000 7f848f83c640 1 -- 172.21.15.37:0/2764039216 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8488075dc0 con 0x7f84881cf7a0 2024-04-26T00:42:40.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.100+0000 7f848d5d8640 1 --2- 172.21.15.37:0/2764039216 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f84881cf7a0 0x7f84880a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:40.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.100+0000 7f848d5d8640 10 monclient: get_auth_request con 0x7f84881cf7a0 auth_method 0 2024-04-26T00:42:40.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.100+0000 7f848d5d8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:40.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.100+0000 7f848d5d8640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:40.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.100+0000 7f848d5d8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:40.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.100+0000 7f848d5d8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:40.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.100+0000 7f848d5d8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:40.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.100+0000 7f848d5d8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:40.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.100+0000 7f848d5d8640 10 monclient(hunting): handle_auth_done global_id 254384 payload 824 2024-04-26T00:42:40.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.100+0000 7f848d5d8640 1 -- 172.21.15.37:0/2764039216 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8488132350 con 0x7f84881cf7a0 2024-04-26T00:42:40.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.104+0000 7f848d5d8640 1 -- 172.21.15.37:0/2764039216 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f84881cf7a0 msgr2=0x7f84880a17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:40.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.104+0000 7f848d5d8640 1 --2- 172.21.15.37:0/2764039216 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f84881cf7a0 0x7f84880a17a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8480017000 tx=0x7f8480004cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:40.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.104+0000 7f8479ffb640 1 -- 172.21.15.37:0/2764039216 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f848000ccf0 con 0x7f84881cf7a0 2024-04-26T00:42:40.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.104+0000 7f8479ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:40.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.104+0000 7f8479ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:42:40.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.104+0000 7f8479ffb640 10 monclient: dump: 2024-04-26T00:42:40.113 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:40.113 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:40.114 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:40.114 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:40.114 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:40.114 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:40.114 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:40.114 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:40.114 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:40.114 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:40.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.104+0000 7f8479ffb640 1 -- 172.21.15.37:0/2764039216 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7f848000ccf0 con 0x7f84881cf7a0 2024-04-26T00:42:40.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.104+0000 7f8479ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:42:40.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.104+0000 7f8479ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:42:40.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.104+0000 7f8479ffb640 1 -- 172.21.15.37:0/2764039216 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f84881cf7a0 msgr2=0x7f84880a17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:40.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.104+0000 7f8479ffb640 1 --2- 172.21.15.37:0/2764039216 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f84881cf7a0 0x7f84880a17a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8480017000 tx=0x7f8480004cc0 comp rx=0 tx=0).stop 2024-04-26T00:42:40.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.104+0000 7f8479ffb640 1 -- 172.21.15.37:0/2764039216 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f84881cf7a0 msgr2=0x7f84880a17a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:42:40.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.104+0000 7f8479ffb640 1 --2- 172.21.15.37:0/2764039216 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f84881cf7a0 0x7f84880a17a0 secure :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8480017000 tx=0x7f8480004cc0 comp rx=0 tx=0).stop 2024-04-26T00:42:40.115 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:42:40.115 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:42:40.115 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:42:40.115 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:42:40.115 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 164, 2024-04-26T00:42:40.115 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 128874, 2024-04-26T00:42:40.115 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:42:40.115 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:42:40.115 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:42:40.116 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:42:40.116 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:42:40.116 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:42:40.116 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:42:40.116 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:42:40.116 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:42:40.116 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:42:40.116 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:42:40.116 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:42:40.116 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:42:40.116 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:42:40.116 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:42:40.116 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:42:40.116 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:42:40.117 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:42:40.117 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:42:40.117 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:42:40.117 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:42:40.117 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:42:40.117 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:42:40.117 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:42:40.117 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:42:40.117 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:42:40.117 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:42:40.117 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:42:40.117 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:42:40.117 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:42:40.117 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:42:40.118 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:42:40.118 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:42:40.118 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:40.118 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:40.118 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:42:40.118 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:42:40.118 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:42:40.118 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:42:40.118 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:42:40.118 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:42:40.118 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:42:40.118 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:42:40.118 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:42:40.118 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:42:40.119 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:42:40.119 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:42:40.119 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:42:40.119 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:42:40.119 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:42:40.119 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:42:40.119 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:42:40.119 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:42:40.119 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:42:40.119 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:42:40.119 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:42:40.119 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:42:40.119 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:42:40.120 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:42:40.120 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:42:40.120 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:42:40.120 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:42:40.120 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:42:40.120 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:42:40.120 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:42:40.120 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:40.120 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:42:40.120 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:40.120 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:42:40.120 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:42:40.120 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:42:40.120 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:42:40.121 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:40.121 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:42:40.121 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:42:40.121 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:40.121 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:40.121 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:40.121 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:42:40.121 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:42:40.121 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:40.121 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:40.121 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:40.121 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:40.121 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:42:40.121 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:42:40.121 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:42:40.122 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:42:40.122 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:42:40.122 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:40.122 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:40.122 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:42:40.122 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:42:40.122 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:42:40.122 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:42:40.122 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:40.122 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:42:40.122 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:42:40.122 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:40.122 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:40.122 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:40.123 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:42:40.123 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:42:40.123 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:40.123 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:40.123 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:40.123 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:40.123 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:42:40.123 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:42:40.123 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:42:40.123 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:42:40.123 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:42:40.123 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:40.123 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:40.123 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:42:40.123 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:42:40.124 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:42:40.124 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:42:40.124 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:40.124 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:42:40.124 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:42:40.124 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:40.124 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:40.124 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:40.124 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:42:40.124 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:42:40.124 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:40.124 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:40.124 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:40.124 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:40.124 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:42:40.124 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:42:40.125 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:42:40.125 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:42:40.125 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:42:40.125 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:40.125 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:40.125 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:40.125 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:42:40.125 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:42:40.125 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:40.125 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:42:40.125 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:42:40.125 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:42:40.125 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:40.125 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:42:40.125 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:42:40.126 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:40.126 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:42:40.126 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:42:40.126 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:42:40.126 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:40.126 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:42:40.126 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:42:40.126 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:40.126 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:42:40.126 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:42:40.126 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 6 2024-04-26T00:42:40.126 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:40.126 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:42:40.126 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:42:40.126 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:40.127 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:42:40.127 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:42:40.127 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:42:40.127 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:40.127 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:40.127 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:40.127 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:42:40.127 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:42:40.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.104+0000 7f848f83c640 1 -- 172.21.15.37:0/2764039216 >> v2:172.21.15.45:6812/2379918623 conn(0x7f846c035e30 msgr2=0x7f846c0382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:40.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.104+0000 7f848f83c640 1 --2- 172.21.15.37:0/2764039216 >> v2:172.21.15.45:6812/2379918623 conn(0x7f846c035e30 0x7f846c0382f0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f84880a57e0 tx=0x7f847c008040 comp rx=0 tx=0).stop 2024-04-26T00:42:40.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.104+0000 7f848f83c640 10 monclient: shutdown 2024-04-26T00:42:40.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.104+0000 7f848f83c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:42:40.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.104+0000 7f848f83c640 1 -- 172.21.15.37:0/2764039216 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f848813c3e0 msgr2=0x7f848813a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:40.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.104+0000 7f848f83c640 1 --2- 172.21.15.37:0/2764039216 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f848813c3e0 0x7f848813a920 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f848400b440 tx=0x7f848400b900 comp rx=0 tx=0).stop 2024-04-26T00:42:40.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.104+0000 7f848ddd9640 1 -- 172.21.15.37:0/2764039216 reap_dead start 2024-04-26T00:42:40.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.104+0000 7f848ddd9640 1 -- 172.21.15.37:0/2764039216 reap_dead start 2024-04-26T00:42:40.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.104+0000 7f848f83c640 1 -- 172.21.15.37:0/2764039216 shutdown_connections 2024-04-26T00:42:40.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.104+0000 7f848f83c640 1 -- 172.21.15.37:0/2764039216 >> 172.21.15.37:0/2764039216 conn(0x7f84880aa0a0 msgr2=0x7f84880a8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:42:40.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.104+0000 7f848f83c640 1 -- 172.21.15.37:0/2764039216 shutdown_connections 2024-04-26T00:42:40.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.104+0000 7f848f83c640 1 -- 172.21.15.37:0/2764039216 wait complete. 2024-04-26T00:42:40.130 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:42:40.130 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-04-26T00:42:40.130 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:42:40.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.240+0000 7f58cc6bf640 10 monclient: get_monmap_and_config 2024-04-26T00:42:40.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.240+0000 7f58cc6bf640 10 monclient: build_initial_monmap 2024-04-26T00:42:40.246 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.240+0000 7f58cc6bf640 10 monclient: monmap: 2024-04-26T00:42:40.246 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:42:40.246 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:40.246 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:42:40.247655+0000 2024-04-26T00:42:40.246 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:42:40.247655+0000 2024-04-26T00:42:40.247 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:42:40.247 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:40.247 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:42:40.247 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:42:40.247 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:42:40.247 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:42:40.247 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:42:40.247 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:40.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.244+0000 7f58cc6bf640 1 Processor -- start 2024-04-26T00:42:40.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.244+0000 7f58cc6bf640 1 -- start start 2024-04-26T00:42:40.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.244+0000 7f58cc6bf640 10 monclient: init 2024-04-26T00:42:40.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.244+0000 7f58cc6bf640 10 monclient: _reopen_session rank -1 2024-04-26T00:42:40.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.244+0000 7f58cc6bf640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-04-26T00:42:40.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.244+0000 7f58cc6bf640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f58c409e220 0x7f58c409fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:40.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.244+0000 7f58cc6bf640 10 monclient(hunting): picked mon.noname-a con 0x7f58c409e220 addr v2:172.21.15.37:3300/0 2024-04-26T00:42:40.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.244+0000 7f58cc6bf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f58c40a14f0 addr v1:172.21.15.37:6789/0 2024-04-26T00:42:40.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.244+0000 7f58cc6bf640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f58c409e600 0x7f58c413aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:40.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.244+0000 7f58cc6bf640 10 monclient(hunting): picked mon.noname-c con 0x7f58c409e600 addr v2:172.21.15.45:3300/0 2024-04-26T00:42:40.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.244+0000 7f58cc6bf640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58c40a12b0 con 0x7f58c40a14f0 2024-04-26T00:42:40.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.244+0000 7f58cc6bf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:40.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.244+0000 7f58cc6bf640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f58c409da20 con 0x7f58c409e220 2024-04-26T00:42:40.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.244+0000 7f58cc6bf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:40.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.244+0000 7f58cc6bf640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f58c409dd00 con 0x7f58c409e600 2024-04-26T00:42:40.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.244+0000 7f58cc6bf640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:40.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.244+0000 7f58ca45b640 1 -- >> v2:172.21.15.37:3300/0 conn(0x7f58c409e220 msgr2=0x7f58c409fc50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:42:40.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.244+0000 7f58cc6bf640 10 monclient(hunting): authenticate will time out at 2341.553444s 2024-04-26T00:42:40.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.244+0000 7f58c9c5a640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f58c40a14f0 legacy=0x7f58c40a0190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:42:40.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.244+0000 7f58ca45b640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f58c409e220 0x7f58c409fc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:42:40.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.244+0000 7f58cac5c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f58c409e600 0x7f58c413aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:40.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.244+0000 7f58cac5c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f58c409e600 0x7f58c413aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.37:53798/0 (socket says 172.21.15.37:53798) 2024-04-26T00:42:40.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.244+0000 7f58cac5c640 1 -- 172.21.15.37:0/146275194 learned_addr learned my addr 172.21.15.37:0/146275194 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:42:40.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.244+0000 7f58cac5c640 10 monclient(hunting): get_auth_request con 0x7f58c409e600 auth_method 0 2024-04-26T00:42:40.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.244+0000 7f58cac5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:40.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.244+0000 7f58cac5c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:40.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.244+0000 7f58cac5c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:40.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.244+0000 7f58cac5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:40.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.244+0000 7f58cac5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:40.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.244+0000 7f58cac5c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:40.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.244+0000 7f58cac5c640 10 monclient(hunting): handle_auth_done global_id 274150 payload 274 2024-04-26T00:42:40.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.244+0000 7f58cac5c640 1 -- 172.21.15.37:0/146275194 >> v2:172.21.15.37:3300/0 conn(0x7f58c409e220 msgr2=0x7f58c409fc50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:42:40.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.244+0000 7f58cac5c640 1 --2- 172.21.15.37:0/146275194 >> v2:172.21.15.37:3300/0 conn(0x7f58c409e220 0x7f58c409fc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:40.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.244+0000 7f58cac5c640 1 -- 172.21.15.37:0/146275194 >> v1:172.21.15.37:6789/0 conn(0x7f58c40a14f0 legacy=0x7f58c40a0190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:42:40.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.244+0000 7f58cac5c640 10 monclient: _finish_hunting 0 2024-04-26T00:42:40.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.244+0000 7f58cac5c640 1 monclient: found mon.noname-c 2024-04-26T00:42:40.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.244+0000 7f58cac5c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:42:40.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.244+0000 7f58cac5c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:42:40.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.244+0000 7f58cac5c640 1 -- 172.21.15.37:0/146275194 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58c4073310 con 0x7f58c409e600 2024-04-26T00:42:40.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.248+0000 7f58cac5c640 1 -- 172.21.15.37:0/146275194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58c409e600 msgr2=0x7f58c413aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:40.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.248+0000 7f58cac5c640 1 --2- 172.21.15.37:0/146275194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58c409e600 0x7f58c413aa60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f58c409e1f0 tx=0x7f58bc01c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=114c54660f40eae1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:40.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.248+0000 7f58c9459640 1 -- 172.21.15.37:0/146275194 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58bc0a55b0 con 0x7f58c409e600 2024-04-26T00:42:40.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.248+0000 7f58c9459640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:40.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.248+0000 7f58c9459640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:42:40.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.248+0000 7f58c9459640 10 monclient: dump: 2024-04-26T00:42:40.254 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:40.254 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:40.254 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:40.254 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:40.254 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:40.254 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:40.254 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:40.254 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:40.254 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:40.255 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:40.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.248+0000 7f58c9459640 10 monclient: _finish_auth 0 2024-04-26T00:42:40.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.248+0000 7f58c9459640 10 monclient: _check_auth_tickets 2024-04-26T00:42:40.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.248+0000 7f58c9459640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:42:40.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.248+0000 7f58c9459640 1 -- 172.21.15.37:0/146275194 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58c40a0ab0 con 0x7f58c409e600 2024-04-26T00:42:40.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.248+0000 7f58c9459640 10 monclient: handle_config config(0 keys) 2024-04-26T00:42:40.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.248+0000 7f58c9459640 1 -- 172.21.15.37:0/146275194 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58bc0a5970 con 0x7f58c409e600 2024-04-26T00:42:40.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.248+0000 7f58c9459640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:40.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.248+0000 7f58c9459640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:42:40.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.248+0000 7f58c9459640 10 monclient: dump: 2024-04-26T00:42:40.255 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:40.255 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:40.255 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:40.255 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:40.256 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:40.256 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:40.256 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:40.256 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:40.256 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:40.256 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:40.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.248+0000 7f58cc6bf640 5 monclient: authenticate success, global_id 274150 2024-04-26T00:42:40.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.248+0000 7f58cc6bf640 10 monclient: get_monmap_and_config success 2024-04-26T00:42:40.256 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.248+0000 7f58cc6bf640 4 set_mon_vals no callback set 2024-04-26T00:42:40.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58cc6bf640 10 monclient: shutdown 2024-04-26T00:42:40.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58cc6bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:42:40.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58cc6bf640 1 -- 172.21.15.37:0/146275194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58c409e600 msgr2=0x7f58c413aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:40.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58cc6bf640 1 --2- 172.21.15.37:0/146275194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58c409e600 0x7f58c413aa60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f58c409e1f0 tx=0x7f58bc01c860 comp rx=0 tx=0).stop 2024-04-26T00:42:40.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58cc6bf640 1 -- 172.21.15.37:0/146275194 shutdown_connections 2024-04-26T00:42:40.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58cc6bf640 1 --2- 172.21.15.37:0/146275194 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58c409e600 0x7f58c413aa60 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:40.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58cc6bf640 1 --2- 172.21.15.37:0/146275194 >> v2:172.21.15.37:3300/0 conn(0x7f58c409e220 0x7f58c409fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:40.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58cc6bf640 1 -- 172.21.15.37:0/146275194 >> 172.21.15.37:0/146275194 conn(0x7f58c40a1d50 msgr2=0x7f58c40a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:42:40.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58cc6bf640 1 -- 172.21.15.37:0/146275194 shutdown_connections 2024-04-26T00:42:40.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58cc6bf640 1 -- 172.21.15.37:0/146275194 wait complete. 2024-04-26T00:42:40.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58cc6bf640 10 monclient: build_initial_monmap 2024-04-26T00:42:40.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58cc6bf640 10 monclient: monmap: 2024-04-26T00:42:40.261 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:42:40.261 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:42:40.261 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:42:40.261 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:42:40.261 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:42:40.262 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:42:40.262 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:42:40.262 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:42:40.262 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:42:40.262 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:40.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58cc6bf640 1 Processor -- start 2024-04-26T00:42:40.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58cc6bf640 1 -- start start 2024-04-26T00:42:40.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58cc6bf640 10 monclient: init 2024-04-26T00:42:40.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58cc6bf640 10 monclient: _reopen_session rank -1 2024-04-26T00:42:40.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58cc6bf640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:42:40.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58cc6bf640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f58c409e220 0x7f58c41200d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:40.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58cc6bf640 10 monclient(hunting): picked mon.noname-c con 0x7f58c409e220 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:42:40.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58cc6bf640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f58c409e600 0x7f58c411dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:40.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58cc6bf640 10 monclient(hunting): picked mon.noname-a con 0x7f58c409e600 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:42:40.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58cc6bf640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58c40a14f0 0x7f58c4136cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:40.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58cc6bf640 10 monclient(hunting): picked mon.noname-b con 0x7f58c40a14f0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:42:40.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58cc6bf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:40.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58cc6bf640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f58c413f6f0 con 0x7f58c409e600 2024-04-26T00:42:40.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58cc6bf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:40.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58cc6bf640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f58c413f570 con 0x7f58c40a14f0 2024-04-26T00:42:40.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58cc6bf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:40.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58cc6bf640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f58c413f870 con 0x7f58c409e220 2024-04-26T00:42:40.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58cc6bf640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:40.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58cc6bf640 10 monclient(hunting): authenticate will time out at 2341.562662s 2024-04-26T00:42:40.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58c9c5a640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f58c409e600 msgr2=0x7f58c411dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:42:40.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58ca45b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f58c409e220 0x7f58c41200d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:40.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58c9c5a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f58c409e600 0x7f58c411dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:42:40.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58ca45b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f58c409e220 0x7f58c41200d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:48502/0 (socket says 172.21.15.37:48502) 2024-04-26T00:42:40.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58ca45b640 1 -- 172.21.15.37:0/3393669162 learned_addr learned my addr 172.21.15.37:0/3393669162 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:42:40.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58ca45b640 10 monclient(hunting): get_auth_request con 0x7f58c409e220 auth_method 0 2024-04-26T00:42:40.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58ca45b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:40.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58ca45b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:40.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58ca45b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:40.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58ca45b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:40.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58ca45b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:40.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58ca45b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:40.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58cac5c640 1 --2- 172.21.15.37:0/3393669162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58c40a14f0 0x7f58c4136cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:40.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58cac5c640 10 monclient(hunting): get_auth_request con 0x7f58c40a14f0 auth_method 0 2024-04-26T00:42:40.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58cac5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:40.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58cac5c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:40.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58cac5c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:40.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58cac5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:40.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58cac5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:40.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58cac5c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:40.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58ca45b640 10 monclient(hunting): handle_auth_done global_id 254387 payload 824 2024-04-26T00:42:40.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58ca45b640 1 -- 172.21.15.37:0/3393669162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58c40a14f0 msgr2=0x7f58c4136cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:40.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58ca45b640 1 --2- 172.21.15.37:0/3393669162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58c40a14f0 0x7f58c4136cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:40.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58ca45b640 1 -- 172.21.15.37:0/3393669162 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f58c409e600 msgr2=0x7f58c411dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:42:40.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58ca45b640 1 --2- 172.21.15.37:0/3393669162 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f58c409e600 0x7f58c411dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:40.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58ca45b640 10 monclient: _finish_hunting 0 2024-04-26T00:42:40.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58ca45b640 1 monclient: found mon.noname-c 2024-04-26T00:42:40.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58ca45b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:42:40.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58ca45b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:42:40.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.252+0000 7f58ca45b640 1 -- 172.21.15.37:0/3393669162 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58c411f350 con 0x7f58c409e220 2024-04-26T00:42:40.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.256+0000 7f58ca45b640 1 -- 172.21.15.37:0/3393669162 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f58c409e220 msgr2=0x7f58c41200d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:40.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.256+0000 7f58ca45b640 1 --2- 172.21.15.37:0/3393669162 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f58c409e220 0x7f58c41200d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f58b400b530 tx=0x7f58b400b9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:40.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.256+0000 7f58ca45b640 1 --2- 172.21.15.37:0/3393669162 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f58c409e220 0x7f58c41200d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f58b400b530 tx=0x7f58b400b9f0 comp rx=0 tx=0).queue_received will delay after 1.71409e+09 on 0x7f58b4013020 mon_map magic: 0 2024-04-26T00:42:40.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.256+0000 7f58baffd640 1 -- 172.21.15.37:0/3393669162 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b4013020 con 0x7f58c409e220 2024-04-26T00:42:40.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.256+0000 7f58baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:40.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.256+0000 7f58baffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:42:40.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.256+0000 7f58baffd640 10 monclient: dump: 2024-04-26T00:42:40.266 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:40.266 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:40.266 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:40.266 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:40.266 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:40.266 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:40.266 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:40.267 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:40.267 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:40.267 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:40.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.256+0000 7f58baffd640 10 monclient: _finish_auth 0 2024-04-26T00:42:40.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.256+0000 7f58baffd640 10 monclient: _check_auth_tickets 2024-04-26T00:42:40.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.256+0000 7f58baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:42:40.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.256+0000 7f58baffd640 1 -- 172.21.15.37:0/3393669162 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58b4005070 con 0x7f58c409e220 2024-04-26T00:42:40.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.256+0000 7f58cc6bf640 5 monclient: authenticate success, global_id 254387 2024-04-26T00:42:40.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.256+0000 7f58cc6bf640 10 monclient: _renew_subs 2024-04-26T00:42:40.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.256+0000 7f58cc6bf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:40.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.256+0000 7f58cc6bf640 1 -- 172.21.15.37:0/3393669162 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58c4072a40 con 0x7f58c409e220 2024-04-26T00:42:40.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.256+0000 7f58baffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:42:40.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.256+0000 7f58bbfff640 4 set_mon_vals no callback set 2024-04-26T00:42:40.267 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.256+0000 7f58cc6bf640 10 monclient: _renew_subs 2024-04-26T00:42:40.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.256+0000 7f58cc6bf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:40.268 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.256+0000 7f58cc6bf640 1 -- 172.21.15.37:0/3393669162 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f58c4120da0 con 0x7f58c409e220 2024-04-26T00:42:40.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.256+0000 7f58baffd640 1 -- 172.21.15.37:0/3393669162 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b4005460 con 0x7f58c409e220 2024-04-26T00:42:40.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.256+0000 7f58baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:40.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.256+0000 7f58baffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:42:40.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.256+0000 7f58baffd640 10 monclient: dump: 2024-04-26T00:42:40.270 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:40.270 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:40.270 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:40.270 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:40.270 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:40.270 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:40.270 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:40.271 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:40.271 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:40.271 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:40.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.256+0000 7f58baffd640 1 -- 172.21.15.37:0/3393669162 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f58b4016020 con 0x7f58c409e220 2024-04-26T00:42:40.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.256+0000 7f58cc6bf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:40.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.256+0000 7f58cc6bf640 1 --2- 172.21.15.37:0/3393669162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58c41371f0 0x7f58c41cd600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:40.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.256+0000 7f58cc6bf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:40.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.256+0000 7f58cc6bf640 1 -- 172.21.15.37:0/3393669162 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f58c40a0ab0 con 0x7f58c41371f0 2024-04-26T00:42:40.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.256+0000 7f58baffd640 1 --2- 172.21.15.37:0/3393669162 >> v2:172.21.15.45:6812/2379918623 conn(0x7f58a8035e30 0x7f58a80382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:40.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.256+0000 7f58c9c5a640 1 --2- 172.21.15.37:0/3393669162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58c41371f0 0x7f58c41cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:40.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.256+0000 7f58c9c5a640 10 monclient: get_auth_request con 0x7f58c41371f0 auth_method 0 2024-04-26T00:42:40.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.256+0000 7f58c9c5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:40.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.256+0000 7f58c9c5a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:40.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.256+0000 7f58c9c5a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:40.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.256+0000 7f58cac5c640 1 --2- 172.21.15.37:0/3393669162 >> v2:172.21.15.45:6812/2379918623 conn(0x7f58a8035e30 0x7f58a80382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:40.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.256+0000 7f58c9c5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:40.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.256+0000 7f58c9c5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:40.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.256+0000 7f58c9c5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:40.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.256+0000 7f58cac5c640 10 monclient: get_auth_request con 0x7f58a8035e30 auth_method 0 2024-04-26T00:42:40.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.260+0000 7f58c9c5a640 10 monclient(hunting): handle_auth_done global_id 274156 payload 824 2024-04-26T00:42:40.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.260+0000 7f58c9c5a640 1 -- 172.21.15.37:0/3393669162 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f58c404f510 con 0x7f58c41371f0 2024-04-26T00:42:40.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.260+0000 7f58c9c5a640 1 -- 172.21.15.37:0/3393669162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58c41371f0 msgr2=0x7f58c41cd600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:40.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.260+0000 7f58c9c5a640 1 --2- 172.21.15.37:0/3393669162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58c41371f0 0x7f58c41cd600 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f58c40a1900 tx=0x7f58c000b8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:40.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.260+0000 7f58cac5c640 1 --2- 172.21.15.37:0/3393669162 >> v2:172.21.15.45:6812/2379918623 conn(0x7f58a8035e30 0x7f58a80382f0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f58c406d170 tx=0x7f58bc0a7070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:40.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.260+0000 7f58baffd640 1 -- 172.21.15.37:0/3393669162 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c0016070 con 0x7f58c41371f0 2024-04-26T00:42:40.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.260+0000 7f58baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:40.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.260+0000 7f58baffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:42:40.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.260+0000 7f58baffd640 10 monclient: dump: 2024-04-26T00:42:40.273 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:40.273 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:40.273 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:40.273 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:40.273 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:40.273 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:40.273 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:40.273 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:40.273 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:40.273 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:40.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.260+0000 7f58baffd640 1 -- 172.21.15.37:0/3393669162 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f58c0005270 con 0x7f58c41371f0 2024-04-26T00:42:40.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.260+0000 7f58baffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:40.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.260+0000 7f58baffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:42:40.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.260+0000 7f58baffd640 1 -- 172.21.15.37:0/3393669162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58c41371f0 msgr2=0x7f58c41cd600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:40.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.260+0000 7f58baffd640 1 --2- 172.21.15.37:0/3393669162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58c41371f0 0x7f58c41cd600 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f58c40a1900 tx=0x7f58c000b8e0 comp rx=0 tx=0).stop 2024-04-26T00:42:40.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.260+0000 7f58baffd640 1 -- 172.21.15.37:0/3393669162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58c41371f0 msgr2=0x7f58c41cd600 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:42:40.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.260+0000 7f58baffd640 1 --2- 172.21.15.37:0/3393669162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58c41371f0 0x7f58c41cd600 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f58c40a1900 tx=0x7f58c000b8e0 comp rx=0 tx=0).stop 2024-04-26T00:42:40.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.268+0000 7f58cc6bf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:42:40.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.268+0000 7f58cc6bf640 1 --2- 172.21.15.37:0/3393669162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58c402c000 0x7f58c402c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:40.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.268+0000 7f58cc6bf640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:40.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.268+0000 7f58cc6bf640 1 -- 172.21.15.37:0/3393669162 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f58c4139380 con 0x7f58c402c000 2024-04-26T00:42:40.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.268+0000 7f58c9c5a640 1 --2- 172.21.15.37:0/3393669162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58c402c000 0x7f58c402c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:40.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.268+0000 7f58c9c5a640 10 monclient: get_auth_request con 0x7f58c402c000 auth_method 0 2024-04-26T00:42:40.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.268+0000 7f58c9c5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:40.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.268+0000 7f58c9c5a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:40.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.268+0000 7f58c9c5a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:40.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.268+0000 7f58c9c5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:40.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.268+0000 7f58c9c5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:40.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.268+0000 7f58c9c5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:40.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.268+0000 7f58c9c5a640 10 monclient(hunting): handle_auth_done global_id 274159 payload 824 2024-04-26T00:42:40.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.268+0000 7f58c9c5a640 1 -- 172.21.15.37:0/3393669162 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f58c406ba50 con 0x7f58c402c000 2024-04-26T00:42:40.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.272+0000 7f58c9c5a640 1 -- 172.21.15.37:0/3393669162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58c402c000 msgr2=0x7f58c402c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:40.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.272+0000 7f58c9c5a640 1 --2- 172.21.15.37:0/3393669162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58c402c000 0x7f58c402c3e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f58c0013000 tx=0x7f58c00059e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:40.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.272+0000 7f58baffd640 1 -- 172.21.15.37:0/3393669162 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c0016040 con 0x7f58c402c000 2024-04-26T00:42:40.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.272+0000 7f58baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:40.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.272+0000 7f58baffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:42:40.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.272+0000 7f58baffd640 10 monclient: dump: 2024-04-26T00:42:40.277 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:40.277 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:40.277 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:40.277 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:40.278 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:40.278 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:40.278 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:40.278 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:40.278 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:40.278 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:40.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.272+0000 7f58baffd640 1 -- 172.21.15.37:0/3393669162 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4536 (secure 0 0 0) 0x7f58c0021450 con 0x7f58c402c000 2024-04-26T00:42:40.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.272+0000 7f58baffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:42:40.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.272+0000 7f58baffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:42:40.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.272+0000 7f58baffd640 1 -- 172.21.15.37:0/3393669162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58c402c000 msgr2=0x7f58c402c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:40.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.272+0000 7f58baffd640 1 --2- 172.21.15.37:0/3393669162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58c402c000 0x7f58c402c3e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f58c0013000 tx=0x7f58c00059e0 comp rx=0 tx=0).stop 2024-04-26T00:42:40.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.272+0000 7f58baffd640 1 -- 172.21.15.37:0/3393669162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58c402c000 msgr2=0x7f58c402c3e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:42:40.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.272+0000 7f58baffd640 1 --2- 172.21.15.37:0/3393669162 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58c402c000 0x7f58c402c3e0 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f58c0013000 tx=0x7f58c00059e0 comp rx=0 tx=0).stop 2024-04-26T00:42:40.279 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:42:40.279 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:42:40.279 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:42:40.279 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:42:40.279 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 164, 2024-04-26T00:42:40.279 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 28027, 2024-04-26T00:42:40.279 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:42:40.279 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:42:40.279 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:42:40.279 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:42:40.279 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 0, 2024-04-26T00:42:40.279 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:42:40.280 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:42:40.280 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:42:40.280 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:42:40.280 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:42:40.280 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:42:40.280 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:42:40.280 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:42:40.280 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:42:40.280 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:42:40.280 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:42:40.280 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:42:40.280 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:42:40.280 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:42:40.280 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:42:40.280 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:42:40.281 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:42:40.281 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:42:40.281 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:42:40.281 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:42:40.281 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:42:40.281 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:42:40.281 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:42:40.281 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:42:40.281 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:42:40.281 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:42:40.281 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:42:40.281 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:42:40.281 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:40.281 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:40.281 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:42:40.282 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:42:40.282 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:42:40.282 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:42:40.282 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:42:40.282 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:42:40.282 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:42:40.282 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:42:40.282 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:42:40.282 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:42:40.282 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:42:40.282 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:42:40.282 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:42:40.282 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:42:40.282 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:42:40.283 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:42:40.283 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:42:40.283 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:42:40.283 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:42:40.283 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:42:40.283 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:42:40.283 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:42:40.283 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:42:40.283 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:42:40.283 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:42:40.283 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:42:40.283 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:42:40.283 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:42:40.283 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:42:40.283 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:42:40.284 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:40.284 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:42:40.284 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:40.284 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:42:40.284 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:42:40.284 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:42:40.284 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:42:40.284 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:40.284 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:42:40.284 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:42:40.284 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:40.284 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:40.284 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:40.285 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:42:40.285 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:42:40.285 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:40.285 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:40.285 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:40.285 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:40.285 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:42:40.285 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:42:40.285 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:42:40.285 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:42:40.285 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:42:40.285 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:40.285 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:40.285 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:42:40.285 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:42:40.286 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:42:40.286 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:42:40.286 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:40.286 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:42:40.286 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:42:40.286 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:40.286 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:40.286 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:40.286 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:42:40.286 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:42:40.286 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:40.286 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:40.286 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:40.287 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:40.287 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:42:40.287 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:42:40.287 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:42:40.287 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:42:40.287 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:42:40.287 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:40.287 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:40.287 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:42:40.287 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:42:40.287 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:42:40.287 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:42:40.287 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:40.287 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:42:40.287 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:42:40.288 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:40.288 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:40.288 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:40.288 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:42:40.288 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:42:40.288 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:42:40.288 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:40.288 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:40.288 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:40.288 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:42:40.288 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:42:40.288 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:42:40.288 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:42:40.288 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:42:40.288 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:40.289 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:40.289 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:40.289 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:42:40.289 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:42:40.289 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:40.289 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:42:40.289 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:42:40.289 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:42:40.289 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:40.289 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:42:40.289 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:42:40.289 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:42:40.289 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:42:40.289 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:42:40.289 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 2 2024-04-26T00:42:40.290 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:42:40.290 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:42:40.290 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:42:40.290 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:42:40.290 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:42:40.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.272+0000 7f58cc6bf640 1 -- 172.21.15.37:0/3393669162 >> v2:172.21.15.45:6812/2379918623 conn(0x7f58a8035e30 msgr2=0x7f58a80382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:40.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.272+0000 7f58cc6bf640 1 --2- 172.21.15.37:0/3393669162 >> v2:172.21.15.45:6812/2379918623 conn(0x7f58a8035e30 0x7f58a80382f0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f58c406d170 tx=0x7f58bc0a7070 comp rx=0 tx=0).stop 2024-04-26T00:42:40.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.272+0000 7f58cc6bf640 10 monclient: shutdown 2024-04-26T00:42:40.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.272+0000 7f58cc6bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:42:40.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.272+0000 7f58cc6bf640 1 -- 172.21.15.37:0/3393669162 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f58c409e220 msgr2=0x7f58c41200d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:40.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.272+0000 7f58cc6bf640 1 --2- 172.21.15.37:0/3393669162 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f58c409e220 0x7f58c41200d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f58b400b530 tx=0x7f58b400b9f0 comp rx=0 tx=0).stop 2024-04-26T00:42:40.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.272+0000 7f58cac5c640 1 -- 172.21.15.37:0/3393669162 reap_dead start 2024-04-26T00:42:40.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.272+0000 7f58baffd640 1 -- 172.21.15.37:0/3393669162 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(896..896 src has 822..896) ==== 3975+0+0 (unknown 0 0 0) 0x7f58b4048710 con 0x7f58c409e220 2024-04-26T00:42:40.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.272+0000 7f58cac5c640 1 -- 172.21.15.37:0/3393669162 reap_dead start 2024-04-26T00:42:40.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.272+0000 7f58cc6bf640 1 -- 172.21.15.37:0/3393669162 shutdown_connections 2024-04-26T00:42:40.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.272+0000 7f58cc6bf640 1 -- 172.21.15.37:0/3393669162 >> 172.21.15.37:0/3393669162 conn(0x7f58c40a1d50 msgr2=0x7f58c40a3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:42:40.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.272+0000 7f58cc6bf640 1 -- 172.21.15.37:0/3393669162 shutdown_connections 2024-04-26T00:42:40.291 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:40.272+0000 7f58cc6bf640 1 -- 172.21.15.37:0/3393669162 wait complete. 2024-04-26T00:42:40.294 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T00:42:40.294 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-04-26T00:42:40.526 INFO:tasks.rados.rados.0.smithi045.stdout:3987: done (3 left) 2024-04-26T00:42:40.526 INFO:tasks.rados.rados.0.smithi045.stdout:3988: copy_from oid 21 from oid 34 current snap is 440 2024-04-26T00:42:40.526 INFO:tasks.rados.rados.0.smithi045.stdout:3983: finishing write tid 3 to smithi04526579-22 2024-04-26T00:42:40.526 INFO:tasks.rados.rados.0.smithi045.stdout:3983: finishing write tid 4 to smithi04526579-22 2024-04-26T00:42:40.526 INFO:tasks.rados.rados.0.smithi045.stdout:3983: finishing write tid 5 to smithi04526579-22 2024-04-26T00:42:40.526 INFO:tasks.rados.rados.0.smithi045.stdout:3983: oid 22 updating version 0 to 2836 2024-04-26T00:42:40.526 INFO:tasks.rados.rados.0.smithi045.stdout:3983: oid 22 updating version 2836 to 2837 2024-04-26T00:42:40.526 INFO:tasks.rados.rados.0.smithi045.stdout:3983: oid 22 updating version 2837 to 2838 2024-04-26T00:42:40.526 INFO:tasks.rados.rados.0.smithi045.stdout:3983: oid 22 updating version 2838 to 2839 2024-04-26T00:42:40.526 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 22 v 2839 (ObjNum 1421 snap 440 seq_num 1421) dirty exists 2024-04-26T00:42:40.526 INFO:tasks.rados.rados.0.smithi045.stdout:3983: left oid 22 (ObjNum 1421 snap 440 seq_num 1421) 2024-04-26T00:42:40.527 INFO:tasks.rados.rados.0.smithi045.stdout:3984: finishing copy_from racing read to smithi04526579-27 2024-04-26T00:42:40.527 INFO:tasks.rados.rados.0.smithi045.stdout:3986: finishing rollback tid 0 to smithi04526579-48 2024-04-26T00:42:40.527 INFO:tasks.rados.rados.0.smithi045.stdout:3986: finishing rollback tid 1 to smithi04526579-48 2024-04-26T00:42:40.527 INFO:tasks.rados.rados.0.smithi045.stdout:3986: finishing rollback tid 2 to smithi04526579-48 2024-04-26T00:42:40.527 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 48 v 2890 (ObjNum 1327 snap 410 seq_num 1327) dirty exists 2024-04-26T00:42:40.527 INFO:tasks.rados.rados.0.smithi045.stdout:3984: finishing copy_from to smithi04526579-27 2024-04-26T00:42:40.527 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 27 v 2504 (ObjNum 1404 snap 435 seq_num 1404) dirty exists 2024-04-26T00:42:40.527 INFO:tasks.rados.rados.0.smithi045.stdout:3983: done (3 left) 2024-04-26T00:42:40.527 INFO:tasks.rados.rados.0.smithi045.stdout:3984: done (2 left) 2024-04-26T00:42:40.527 INFO:tasks.rados.rados.0.smithi045.stdout:3986: done (1 left) 2024-04-26T00:42:40.527 INFO:tasks.rados.rados.0.smithi045.stdout:3989: write oid 23 current snap is 440 2024-04-26T00:42:40.527 INFO:tasks.rados.rados.0.smithi045.stdout:3989: seq_num 1423 ranges {716361=437338,1687053=687490,2777507=425839,3877756=38605} 2024-04-26T00:42:40.536 INFO:tasks.rados.rados.0.smithi045.stdout:3989: writing smithi04526579-23 from 716361 to 1153699 tid 1 2024-04-26T00:42:40.546 INFO:tasks.rados.rados.0.smithi045.stdout:3989: writing smithi04526579-23 from 1687053 to 2374543 tid 2 2024-04-26T00:42:40.554 INFO:tasks.rados.rados.0.smithi045.stdout:3989: writing smithi04526579-23 from 2777507 to 3203346 tid 3 2024-04-26T00:42:40.561 INFO:tasks.rados.rados.0.smithi045.stdout:3989: writing smithi04526579-23 from 3877756 to 3916361 tid 4 2024-04-26T00:42:40.561 INFO:tasks.rados.rados.0.smithi045.stdout:3990: copy_from oid 32 from oid 13 current snap is 440 2024-04-26T00:42:40.561 INFO:tasks.rados.rados.0.smithi045.stdout:3991: copy_from oid 5 from oid 38 current snap is 440 2024-04-26T00:42:40.561 INFO:tasks.rados.rados.0.smithi045.stdout:3992: write oid 18 current snap is 440 2024-04-26T00:42:40.561 INFO:tasks.rados.rados.0.smithi045.stdout:3992: seq_num 1424 ranges {733168=512587,2029182=547501,3088577=637111} 2024-04-26T00:42:40.570 INFO:tasks.rados.rados.0.smithi045.stdout:3992: writing smithi04526579-18 from 733168 to 1245755 tid 1 2024-04-26T00:42:40.581 INFO:tasks.rados.rados.0.smithi045.stdout:3992: writing smithi04526579-18 from 2029182 to 2576683 tid 2 2024-04-26T00:42:40.590 INFO:tasks.rados.rados.0.smithi045.stdout:3992: writing smithi04526579-18 from 3088577 to 3725688 tid 3 2024-04-26T00:42:40.590 INFO:tasks.rados.rados.0.smithi045.stdout:3993: write oid 47 current snap is 440 2024-04-26T00:42:40.590 INFO:tasks.rados.rados.0.smithi045.stdout:3993: seq_num 1425 ranges {749975=587836,2087664=691159,3516000=433975} 2024-04-26T00:42:40.599 INFO:tasks.rados.rados.0.smithi045.stdout:3993: writing smithi04526579-47 from 749975 to 1337811 tid 1 2024-04-26T00:42:40.610 INFO:tasks.rados.rados.0.smithi045.stdout:3993: writing smithi04526579-47 from 2087664 to 2778823 tid 2 2024-04-26T00:42:40.617 INFO:tasks.rados.rados.0.smithi045.stdout:3993: writing smithi04526579-47 from 3516000 to 3949975 tid 3 2024-04-26T00:42:40.617 INFO:tasks.rados.rados.0.smithi045.stdout:3994: rollback oid 17 current snap is 440 2024-04-26T00:42:40.617 INFO:tasks.rados.rados.0.smithi045.stdout:rollback oid 17 to 428 2024-04-26T00:42:40.617 INFO:tasks.rados.rados.0.smithi045.stdout:3989: finishing write tid 1 to smithi04526579-23 2024-04-26T00:42:40.617 INFO:tasks.rados.rados.0.smithi045.stdout:3995: write (excl) oid 45 current snap is 440 2024-04-26T00:42:40.617 INFO:tasks.rados.rados.0.smithi045.stdout:3995: seq_num 1426 ranges {766782=663085,2146146=551170,3143423=776008} 2024-04-26T00:42:40.625 INFO:tasks.rados.rados.0.smithi045.stdout:3995: writing smithi04526579-45 from 766782 to 1429867 tid 1 2024-04-26T00:42:40.633 INFO:tasks.rados.rados.0.smithi045.stdout:3995: writing smithi04526579-45 from 2146146 to 2697316 tid 2 2024-04-26T00:42:40.641 INFO:tasks.rados.rados.0.smithi045.stdout:3995: writing smithi04526579-45 from 3143423 to 3919431 tid 3 2024-04-26T00:42:40.641 INFO:tasks.rados.rados.0.smithi045.stdout:3996: read oid 35 snap -1 2024-04-26T00:42:40.641 INFO:tasks.rados.rados.0.smithi045.stdout:3996: expect deleted 2024-04-26T00:42:40.641 INFO:tasks.rados.rados.0.smithi045.stdout:3997: copy_from oid 10 from oid 9 current snap is 440 2024-04-26T00:42:40.641 INFO:tasks.rados.rados.0.smithi045.stdout:3989: finishing write tid 2 to smithi04526579-23 2024-04-26T00:42:40.641 INFO:tasks.rados.rados.0.smithi045.stdout:3989: finishing write tid 3 to smithi04526579-23 2024-04-26T00:42:40.641 INFO:tasks.rados.rados.0.smithi045.stdout:3989: finishing write tid 4 to smithi04526579-23 2024-04-26T00:42:40.641 INFO:tasks.rados.rados.0.smithi045.stdout:3989: finishing write tid 5 to smithi04526579-23 2024-04-26T00:42:40.641 INFO:tasks.rados.rados.0.smithi045.stdout:3989: finishing write tid 6 to smithi04526579-23 2024-04-26T00:42:40.641 INFO:tasks.rados.rados.0.smithi045.stdout:3989: oid 23 updating version 0 to 2504 2024-04-26T00:42:40.641 INFO:tasks.rados.rados.0.smithi045.stdout:3989: oid 23 updating version 2504 to 2505 2024-04-26T00:42:40.642 INFO:tasks.rados.rados.0.smithi045.stdout:3989: oid 23 updating version 2505 to 2507 2024-04-26T00:42:40.642 INFO:tasks.rados.rados.0.smithi045.stdout:3989: oid 23 version 2507 is already newer than 2506 2024-04-26T00:42:40.642 INFO:tasks.rados.rados.0.smithi045.stdout:3989: oid 23 version 2507 is already newer than 2503 2024-04-26T00:42:40.642 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 23 v 2507 (ObjNum 1423 snap 440 seq_num 1423) dirty exists 2024-04-26T00:42:40.642 INFO:tasks.rados.rados.0.smithi045.stdout:3989: left oid 23 (ObjNum 1423 snap 440 seq_num 1423) 2024-04-26T00:42:40.642 INFO:tasks.rados.rados.0.smithi045.stdout:3989: done (9 left) 2024-04-26T00:42:40.642 INFO:tasks.rados.rados.0.smithi045.stdout:3998: read oid 41 snap -1 2024-04-26T00:42:40.642 INFO:tasks.rados.rados.0.smithi045.stdout:3998: expect (ObjNum 1321 snap 407 seq_num 1321) 2024-04-26T00:42:40.642 INFO:tasks.rados.rados.0.smithi045.stdout:3999: snap_remove snap 439 2024-04-26T00:42:41.289 INFO:tasks.rados.rados.0.smithi045.stdout:3999: done (10 left) 2024-04-26T00:42:41.289 INFO:tasks.rados.rados.0.smithi045.stdout: waiting on 10 2024-04-26T00:42:41.289 INFO:tasks.rados.rados.0.smithi045.stdout:3988: finishing copy_from racing read to smithi04526579-21 2024-04-26T00:42:41.289 INFO:tasks.rados.rados.0.smithi045.stdout:3990: finishing copy_from racing read to smithi04526579-32 2024-04-26T00:42:41.289 INFO:tasks.rados.rados.0.smithi045.stdout:3990: finishing copy_from to smithi04526579-32 2024-04-26T00:42:41.290 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 32 v 2249 (ObjNum 1237 snap 384 seq_num 1237) dirty exists 2024-04-26T00:42:41.290 INFO:tasks.rados.rados.0.smithi045.stdout:3991: finishing copy_from racing read to smithi04526579-5 2024-04-26T00:42:41.290 INFO:tasks.rados.rados.0.smithi045.stdout:3992: finishing write tid 1 to smithi04526579-18 2024-04-26T00:42:41.290 INFO:tasks.rados.rados.0.smithi045.stdout:3988: finishing copy_from to smithi04526579-21 2024-04-26T00:42:41.290 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 21 v 2883 (ObjNum 1411 snap 439 seq_num 1411) dirty exists 2024-04-26T00:42:41.290 INFO:tasks.rados.rados.0.smithi045.stdout:3991: finishing copy_from to smithi04526579-5 2024-04-26T00:42:41.290 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 5 v 2413 (ObjNum 1354 snap 419 seq_num 1354) dirty exists 2024-04-26T00:42:41.290 INFO:tasks.rados.rados.0.smithi045.stdout:3992: finishing write tid 2 to smithi04526579-18 2024-04-26T00:42:41.290 INFO:tasks.rados.rados.0.smithi045.stdout:3988: done (9 left) 2024-04-26T00:42:41.291 INFO:tasks.rados.rados.0.smithi045.stdout:3990: done (8 left) 2024-04-26T00:42:41.291 INFO:tasks.rados.rados.0.smithi045.stdout:3991: done (7 left) 2024-04-26T00:42:41.291 INFO:tasks.rados.rados.0.smithi045.stdout: waiting on 7 2024-04-26T00:42:41.291 INFO:tasks.rados.rados.0.smithi045.stdout:3992: finishing write tid 3 to smithi04526579-18 2024-04-26T00:42:41.291 INFO:tasks.rados.rados.0.smithi045.stdout:3992: finishing write tid 4 to smithi04526579-18 2024-04-26T00:42:41.291 INFO:tasks.rados.rados.0.smithi045.stdout:3992: finishing write tid 5 to smithi04526579-18 2024-04-26T00:42:41.291 INFO:tasks.rados.rados.0.smithi045.stdout:3992: oid 18 updating version 0 to 3070 2024-04-26T00:42:41.291 INFO:tasks.rados.rados.0.smithi045.stdout:3992: oid 18 updating version 3070 to 3071 2024-04-26T00:42:41.291 INFO:tasks.rados.rados.0.smithi045.stdout:3992: oid 18 version 3071 is already newer than 3067 2024-04-26T00:42:41.291 INFO:tasks.rados.rados.0.smithi045.stdout:3992: oid 18 version 3071 is already newer than 3069 2024-04-26T00:42:41.291 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 18 v 3071 (ObjNum 1424 snap 440 seq_num 1424) dirty exists 2024-04-26T00:42:41.291 INFO:tasks.rados.rados.0.smithi045.stdout:3992: left oid 18 (ObjNum 1424 snap 440 seq_num 1424) 2024-04-26T00:42:41.291 INFO:tasks.rados.rados.0.smithi045.stdout:3993: finishing write tid 1 to smithi04526579-47 2024-04-26T00:42:41.291 INFO:tasks.rados.rados.0.smithi045.stdout:3993: finishing write tid 2 to smithi04526579-47 2024-04-26T00:42:41.291 INFO:tasks.rados.rados.0.smithi045.stdout:3992: done (6 left) 2024-04-26T00:42:41.292 INFO:tasks.rados.rados.0.smithi045.stdout: waiting on 6 2024-04-26T00:42:41.292 INFO:tasks.rados.rados.0.smithi045.stdout:3994: finishing rollback tid 1 to smithi04526579-17 2024-04-26T00:42:41.292 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 17 v 2634 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-04-26T00:42:41.292 INFO:tasks.rados.rados.0.smithi045.stdout:3993: finishing write tid 3 to smithi04526579-47 2024-04-26T00:42:41.292 INFO:tasks.rados.rados.0.smithi045.stdout:3993: finishing write tid 4 to smithi04526579-47 2024-04-26T00:42:41.292 INFO:tasks.rados.rados.0.smithi045.stdout:3993: finishing write tid 5 to smithi04526579-47 2024-04-26T00:42:41.292 INFO:tasks.rados.rados.0.smithi045.stdout:3993: oid 47 updating version 0 to 2893 2024-04-26T00:42:41.292 INFO:tasks.rados.rados.0.smithi045.stdout:3993: oid 47 updating version 2893 to 2894 2024-04-26T00:42:41.292 INFO:tasks.rados.rados.0.smithi045.stdout:3993: oid 47 version 2894 is already newer than 2891 2024-04-26T00:42:41.292 INFO:tasks.rados.rados.0.smithi045.stdout:3993: oid 47 version 2894 is already newer than 2892 2024-04-26T00:42:41.292 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 47 v 2894 (ObjNum 1425 snap 440 seq_num 1425) dirty exists 2024-04-26T00:42:41.292 INFO:tasks.rados.rados.0.smithi045.stdout:3993: left oid 47 (ObjNum 1425 snap 440 seq_num 1425) 2024-04-26T00:42:41.292 INFO:tasks.rados.rados.0.smithi045.stdout:3995: finishing write tid 1 to smithi04526579-45 2024-04-26T00:42:41.292 INFO:tasks.rados.rados.0.smithi045.stdout:3995: finishing write tid 2 to smithi04526579-45 2024-04-26T00:42:41.292 INFO:tasks.rados.rados.0.smithi045.stdout:3993: done (5 left) 2024-04-26T00:42:41.293 INFO:tasks.rados.rados.0.smithi045.stdout:3994: done (4 left) 2024-04-26T00:42:41.293 INFO:tasks.rados.rados.0.smithi045.stdout: waiting on 4 2024-04-26T00:42:41.293 INFO:tasks.rados.rados.0.smithi045.stdout:3996: done (3 left) 2024-04-26T00:42:41.293 INFO:tasks.rados.rados.0.smithi045.stdout: waiting on 3 2024-04-26T00:42:41.293 INFO:tasks.rados.rados.0.smithi045.stdout:3998: expect (ObjNum 1321 snap 407 seq_num 1321) 2024-04-26T00:42:41.491 INFO:tasks.rados.rados.0.smithi045.stdout:3995: finishing write tid 3 to smithi04526579-45 2024-04-26T00:42:41.491 INFO:tasks.rados.rados.0.smithi045.stdout:3995: finishing write tid 4 to smithi04526579-45 2024-04-26T00:42:41.492 INFO:tasks.rados.rados.0.smithi045.stdout:3995: finishing write tid 5 to smithi04526579-45 2024-04-26T00:42:41.492 INFO:tasks.rados.rados.0.smithi045.stdout:3995: oid 45 updating version 0 to 2460 2024-04-26T00:42:41.492 INFO:tasks.rados.rados.0.smithi045.stdout:3995: oid 45 updating version 2460 to 2461 2024-04-26T00:42:41.492 INFO:tasks.rados.rados.0.smithi045.stdout:3995: oid 45 updating version 2461 to 2463 2024-04-26T00:42:41.492 INFO:tasks.rados.rados.0.smithi045.stdout:3995: oid 45 version 2463 is already newer than 2462 2024-04-26T00:42:41.492 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 45 v 2463 (ObjNum 1426 snap 440 seq_num 1426) dirty exists 2024-04-26T00:42:41.492 INFO:tasks.rados.rados.0.smithi045.stdout:3995: left oid 45 (ObjNum 1426 snap 440 seq_num 1426) 2024-04-26T00:42:41.492 INFO:tasks.rados.rados.0.smithi045.stdout:3997: finishing copy_from racing read to smithi04526579-10 2024-04-26T00:42:41.492 INFO:tasks.rados.rados.0.smithi045.stdout:3997: finishing copy_from to smithi04526579-10 2024-04-26T00:42:41.492 INFO:tasks.rados.rados.0.smithi045.stdout:update_object_version oid 10 v 2558 (ObjNum 1392 snap 430 seq_num 1392) dirty exists 2024-04-26T00:42:41.493 INFO:tasks.rados.rados.0.smithi045.stdout:3995: done (2 left) 2024-04-26T00:42:41.493 INFO:tasks.rados.rados.0.smithi045.stdout:3997: done (1 left) 2024-04-26T00:42:41.493 INFO:tasks.rados.rados.0.smithi045.stdout:3998: done (0 left) 2024-04-26T00:42:41.498 INFO:tasks.rados.rados.0.smithi045.stderr:0 errors. 2024-04-26T00:42:41.498 INFO:tasks.rados.rados.0.smithi045.stderr: 2024-04-26T00:42:41.512 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T00:42:41.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.640+0000 7fa12da8f640 10 monclient: get_monmap_and_config 2024-04-26T00:42:41.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.640+0000 7fa12da8f640 10 monclient: build_initial_monmap 2024-04-26T00:42:41.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.640+0000 7fa12da8f640 10 monclient: monmap: 2024-04-26T00:42:41.646 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:42:41.646 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:41.646 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:42:41.647654+0000 2024-04-26T00:42:41.646 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:42:41.647654+0000 2024-04-26T00:42:41.646 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:42:41.647 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:41.647 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:42:41.647 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:42:41.647 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:42:41.647 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:42:41.647 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:42:41.647 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:41.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.644+0000 7fa12da8f640 1 Processor -- start 2024-04-26T00:42:41.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.644+0000 7fa12da8f640 1 -- start start 2024-04-26T00:42:41.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.644+0000 7fa12da8f640 10 monclient: init 2024-04-26T00:42:41.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.644+0000 7fa12da8f640 10 monclient: _reopen_session rank -1 2024-04-26T00:42:41.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.644+0000 7fa12da8f640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-04-26T00:42:41.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.644+0000 7fa12da8f640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa128136da0 0x7fa12809bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:41.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.644+0000 7fa12da8f640 10 monclient(hunting): picked mon.noname-a con 0x7fa128136da0 addr v2:172.21.15.37:3300/0 2024-04-26T00:42:41.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.644+0000 7fa12da8f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa12809c410 0x7fa128137180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:41.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.644+0000 7fa12da8f640 10 monclient(hunting): picked mon.noname-b con 0x7fa12809c410 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:42:41.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.644+0000 7fa12da8f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa1281376c0 addr v1:172.21.15.37:6789/0 2024-04-26T00:42:41.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.644+0000 7fa12da8f640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa12809b4f0 con 0x7fa1281376c0 2024-04-26T00:42:41.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.644+0000 7fa12da8f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:41.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.644+0000 7fa12da8f640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fa12809b210 con 0x7fa128136da0 2024-04-26T00:42:41.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.644+0000 7fa12da8f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:41.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.644+0000 7fa12da8f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa12809b7d0 con 0x7fa12809c410 2024-04-26T00:42:41.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.644+0000 7fa12da8f640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:41.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.644+0000 7fa12da8f640 10 monclient(hunting): authenticate will time out at 2342.953338s 2024-04-26T00:42:41.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.644+0000 7fa127fff640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7fa1281376c0 legacy=0x7fa128133b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:42:41.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.644+0000 7fa1277fe640 1 -- >> v2:172.21.15.37:3300/0 conn(0x7fa128136da0 msgr2=0x7fa12809bed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:42:41.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.644+0000 7fa1277fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa128136da0 0x7fa12809bed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:42:41.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.644+0000 7fa126ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa12809c410 0x7fa128137180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:41.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.644+0000 7fa126ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa12809c410 0x7fa128137180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:48504/0 (socket says 172.21.15.37:48504) 2024-04-26T00:42:41.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.644+0000 7fa126ffd640 1 -- 172.21.15.37:0/2646709039 learned_addr learned my addr 172.21.15.37:0/2646709039 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:42:41.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.644+0000 7fa126ffd640 10 monclient(hunting): get_auth_request con 0x7fa12809c410 auth_method 0 2024-04-26T00:42:41.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.644+0000 7fa126ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:41.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.644+0000 7fa126ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:41.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.644+0000 7fa126ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:41.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.644+0000 7fa126ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:41.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.644+0000 7fa126ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:41.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.644+0000 7fa126ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:41.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.644+0000 7fa126ffd640 10 monclient(hunting): handle_auth_done global_id 254390 payload 274 2024-04-26T00:42:41.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.644+0000 7fa126ffd640 1 -- 172.21.15.37:0/2646709039 >> v2:172.21.15.37:3300/0 conn(0x7fa128136da0 msgr2=0x7fa12809bed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:42:41.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.644+0000 7fa126ffd640 1 --2- 172.21.15.37:0/2646709039 >> v2:172.21.15.37:3300/0 conn(0x7fa128136da0 0x7fa12809bed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:41.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.644+0000 7fa126ffd640 1 -- 172.21.15.37:0/2646709039 >> v1:172.21.15.37:6789/0 conn(0x7fa1281376c0 legacy=0x7fa128133b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:42:41.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.644+0000 7fa126ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:42:41.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.644+0000 7fa126ffd640 1 monclient: found mon.noname-b 2024-04-26T00:42:41.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.644+0000 7fa126ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:42:41.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.644+0000 7fa126ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:42:41.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.644+0000 7fa126ffd640 1 -- 172.21.15.37:0/2646709039 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa128137b30 con 0x7fa12809c410 2024-04-26T00:42:41.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.648+0000 7fa126ffd640 1 -- 172.21.15.37:0/2646709039 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa12809c410 msgr2=0x7fa128137180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:41.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.648+0000 7fa126ffd640 1 --2- 172.21.15.37:0/2646709039 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa12809c410 0x7fa128137180 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa11c009870 tx=0x7fa11c01c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c43cf661ebb67676 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:41.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.648+0000 7fa1267fc640 1 -- 172.21.15.37:0/2646709039 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11c0b0070 con 0x7fa12809c410 2024-04-26T00:42:41.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.648+0000 7fa1267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:41.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.648+0000 7fa1267fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:42:41.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.648+0000 7fa1267fc640 10 monclient: dump: 2024-04-26T00:42:41.656 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:41.657 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:41.657 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:41.657 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:41.657 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:41.657 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:41.657 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:41.657 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:41.657 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:41.657 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:41.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.648+0000 7fa1267fc640 10 monclient: _finish_auth 0 2024-04-26T00:42:41.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.648+0000 7fa1267fc640 10 monclient: _check_auth_tickets 2024-04-26T00:42:41.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.648+0000 7fa1267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:42:41.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.648+0000 7fa12da8f640 5 monclient: authenticate success, global_id 254390 2024-04-26T00:42:41.657 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.648+0000 7fa1267fc640 1 -- 172.21.15.37:0/2646709039 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa11c0ab600 con 0x7fa12809c410 2024-04-26T00:42:41.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.648+0000 7fa1267fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:42:41.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.648+0000 7fa12da8f640 10 monclient: get_monmap_and_config success 2024-04-26T00:42:41.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.648+0000 7fa12da8f640 4 set_mon_vals no callback set 2024-04-26T00:42:41.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.648+0000 7fa1267fc640 1 -- 172.21.15.37:0/2646709039 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11c0ab9d0 con 0x7fa12809c410 2024-04-26T00:42:41.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.648+0000 7fa1267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:41.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.648+0000 7fa1267fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:42:41.658 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.648+0000 7fa1267fc640 10 monclient: dump: 2024-04-26T00:42:41.658 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:41.658 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:41.658 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:41.658 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:41.658 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:41.658 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:41.659 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:41.659 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:41.659 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:41.659 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:41.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.648+0000 7fa12da8f640 10 monclient: shutdown 2024-04-26T00:42:41.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.648+0000 7fa12da8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:42:41.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.648+0000 7fa12da8f640 1 -- 172.21.15.37:0/2646709039 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa12809c410 msgr2=0x7fa128137180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:41.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.648+0000 7fa12da8f640 1 --2- 172.21.15.37:0/2646709039 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa12809c410 0x7fa128137180 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa11c009870 tx=0x7fa11c01c400 comp rx=0 tx=0).stop 2024-04-26T00:42:41.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.648+0000 7fa12da8f640 1 -- 172.21.15.37:0/2646709039 shutdown_connections 2024-04-26T00:42:41.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.648+0000 7fa12da8f640 1 --2- 172.21.15.37:0/2646709039 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa12809c410 0x7fa128137180 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:41.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.648+0000 7fa12da8f640 1 --2- 172.21.15.37:0/2646709039 >> v2:172.21.15.37:3300/0 conn(0x7fa128136da0 0x7fa12809bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:41.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.648+0000 7fa12da8f640 1 -- 172.21.15.37:0/2646709039 >> 172.21.15.37:0/2646709039 conn(0x7fa1280a5f00 msgr2=0x7fa1280a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:42:41.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.648+0000 7fa12da8f640 1 -- 172.21.15.37:0/2646709039 shutdown_connections 2024-04-26T00:42:41.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.648+0000 7fa12da8f640 1 -- 172.21.15.37:0/2646709039 wait complete. 2024-04-26T00:42:41.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.648+0000 7fa12da8f640 10 monclient: build_initial_monmap 2024-04-26T00:42:41.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.648+0000 7fa12da8f640 10 monclient: monmap: 2024-04-26T00:42:41.660 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:42:41.660 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:42:41.660 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:42:41.660 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:42:41.660 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:42:41.660 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:42:41.660 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:42:41.660 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:42:41.660 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:42:41.660 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:41.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.648+0000 7fa12da8f640 1 Processor -- start 2024-04-26T00:42:41.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa12da8f640 1 -- start start 2024-04-26T00:42:41.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa12da8f640 10 monclient: init 2024-04-26T00:42:41.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa12da8f640 10 monclient: _reopen_session rank -1 2024-04-26T00:42:41.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa12da8f640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:42:41.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa12da8f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa12809c410 0x7fa1281310d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:41.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa12da8f640 10 monclient(hunting): picked mon.noname-a con 0x7fa12809c410 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:42:41.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa12da8f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa128136da0 0x7fa128131610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:41.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa12da8f640 10 monclient(hunting): picked mon.noname-b con 0x7fa128136da0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:42:41.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa12da8f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1281376c0 0x7fa1281cf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:41.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa12da8f640 10 monclient(hunting): picked mon.noname-c con 0x7fa1281376c0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:42:41.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa12da8f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:41.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa12da8f640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa128141ea0 con 0x7fa12809c410 2024-04-26T00:42:41.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa12da8f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:41.661 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa12da8f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa128141d20 con 0x7fa128136da0 2024-04-26T00:42:41.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa12da8f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:41.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa12da8f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa128142020 con 0x7fa1281376c0 2024-04-26T00:42:41.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa12da8f640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:41.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa12da8f640 10 monclient(hunting): authenticate will time out at 2342.960537s 2024-04-26T00:42:41.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa1277fe640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa12809c410 msgr2=0x7fa1281310d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:42:41.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa1277fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa12809c410 0x7fa1281310d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:42:41.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa127fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1281376c0 0x7fa1281cf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:41.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa127fff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1281376c0 0x7fa1281cf1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:48520/0 (socket says 172.21.15.37:48520) 2024-04-26T00:42:41.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa127fff640 1 -- 172.21.15.37:0/1037462075 learned_addr learned my addr 172.21.15.37:0/1037462075 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:42:41.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa127fff640 10 monclient(hunting): get_auth_request con 0x7fa1281376c0 auth_method 0 2024-04-26T00:42:41.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa127fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:41.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa127fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:41.662 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa127fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:41.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa127fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:41.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa127fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:41.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa127fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:41.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa127fff640 10 monclient(hunting): handle_auth_done global_id 254393 payload 824 2024-04-26T00:42:41.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa126ffd640 1 --2- 172.21.15.37:0/1037462075 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa128136da0 0x7fa128131610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:41.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa127fff640 1 -- 172.21.15.37:0/1037462075 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa128136da0 msgr2=0x7fa128131610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:41.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa127fff640 1 --2- 172.21.15.37:0/1037462075 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa128136da0 0x7fa128131610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:41.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa127fff640 1 -- 172.21.15.37:0/1037462075 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa12809c410 msgr2=0x7fa1281310d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:42:41.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa127fff640 1 --2- 172.21.15.37:0/1037462075 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa12809c410 0x7fa1281310d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:41.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa127fff640 10 monclient: _finish_hunting 0 2024-04-26T00:42:41.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa127fff640 1 monclient: found mon.noname-c 2024-04-26T00:42:41.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa127fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:42:41.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa127fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:42:41.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa127fff640 1 -- 172.21.15.37:0/1037462075 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa128137f20 con 0x7fa1281376c0 2024-04-26T00:42:41.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa126ffd640 10 monclient: get_auth_request con 0x7fa128136da0 auth_method 0 2024-04-26T00:42:41.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa126ffd640 1 --2- 172.21.15.37:0/1037462075 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa128136da0 0x7fa128131610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:42:41.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa127fff640 1 -- 172.21.15.37:0/1037462075 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1281376c0 msgr2=0x7fa1281cf1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:41.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa127fff640 1 --2- 172.21.15.37:0/1037462075 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1281376c0 0x7fa1281cf1f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa12000be90 tx=0x7fa12000bf80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:41.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa10ffff640 1 -- 172.21.15.37:0/1037462075 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa120016630 con 0x7fa1281376c0 2024-04-26T00:42:41.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa10ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:41.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa10ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:42:41.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa10ffff640 10 monclient: dump: 2024-04-26T00:42:41.667 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:41.667 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:41.667 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:41.667 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:41.667 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:41.667 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:41.667 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:41.667 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:41.667 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:41.667 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:41.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa10ffff640 10 monclient: _finish_auth 0 2024-04-26T00:42:41.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa10ffff640 10 monclient: _check_auth_tickets 2024-04-26T00:42:41.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa10ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:42:41.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa10ffff640 1 -- 172.21.15.37:0/1037462075 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa120010030 con 0x7fa1281376c0 2024-04-26T00:42:41.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa10ffff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:42:41.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa10ffff640 1 -- 172.21.15.37:0/1037462075 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa120016a10 con 0x7fa1281376c0 2024-04-26T00:42:41.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa10ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:41.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa10ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:42:41.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa10ffff640 10 monclient: dump: 2024-04-26T00:42:41.668 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:41.668 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:41.668 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:41.668 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:41.668 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:41.668 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:41.668 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:41.669 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:41.669 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:41.669 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:41.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa125ffb640 4 set_mon_vals no callback set 2024-04-26T00:42:41.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa12da8f640 5 monclient: authenticate success, global_id 254393 2024-04-26T00:42:41.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa12da8f640 10 monclient: _renew_subs 2024-04-26T00:42:41.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa12da8f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:41.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.652+0000 7fa12da8f640 1 -- 172.21.15.37:0/1037462075 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1281329c0 con 0x7fa1281376c0 2024-04-26T00:42:41.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.656+0000 7fa12da8f640 10 monclient: _renew_subs 2024-04-26T00:42:41.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.656+0000 7fa12da8f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:41.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.656+0000 7fa12da8f640 1 -- 172.21.15.37:0/1037462075 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1281336f0 con 0x7fa1281376c0 2024-04-26T00:42:41.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.656+0000 7fa12da8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:41.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.656+0000 7fa10ffff640 1 -- 172.21.15.37:0/1037462075 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa1200153e0 con 0x7fa1281376c0 2024-04-26T00:42:41.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.656+0000 7fa12da8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:41.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.656+0000 7fa12da8f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:41.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.656+0000 7fa12da8f640 1 -- 172.21.15.37:0/1037462075 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa128066f40 con 0x7fa1281376c0 2024-04-26T00:42:41.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.656+0000 7fa10ffff640 1 --2- 172.21.15.37:0/1037462075 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa114035e80 0x7fa114038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:41.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.656+0000 7fa10ffff640 1 -- 172.21.15.37:0/1037462075 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(898..898 src has 822..898) ==== 4019+0+0 (secure 0 0 0) 0x7fa120049820 con 0x7fa1281376c0 2024-04-26T00:42:41.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.656+0000 7fa1277fe640 1 --2- 172.21.15.37:0/1037462075 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa114035e80 0x7fa114038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:41.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.656+0000 7fa1277fe640 10 monclient: get_auth_request con 0x7fa114035e80 auth_method 0 2024-04-26T00:42:41.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.660+0000 7fa1277fe640 1 --2- 172.21.15.37:0/1037462075 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa114035e80 0x7fa114038340 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fa118006fd0 tx=0x7fa118006d40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:41.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.660+0000 7fa10ffff640 1 -- 172.21.15.37:0/1037462075 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa12001b020 con 0x7fa1281376c0 2024-04-26T00:42:41.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.660+0000 7fa10ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:41.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.660+0000 7fa10ffff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:42:41.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.820+0000 7fa12da8f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-26T00:42:41.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.820+0000 7fa12da8f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-26T00:42:41.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.820+0000 7fa12da8f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:41.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.820+0000 7fa12da8f640 1 -- 172.21.15.37:0/1037462075 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa128075d70 con 0x7fa1281376c0 2024-04-26T00:42:41.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.824+0000 7fa10ffff640 1 -- 172.21.15.37:0/1037462075 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v898) ==== 74+0+12807 (secure 0 0 0) 0x7fa120048510 con 0x7fa1281376c0 2024-04-26T00:42:41.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.824+0000 7fa10ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-26T00:42:41.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.824+0000 7fa10ffff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:42:41.827 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:42:41.827 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":898,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","created":"2024-04-26T00:19:50.025527+0000","modified":"2024-04-26T00:42:41.279262+0000","last_up_change":"2024-04-26T00:19:59.222603+0000","last_in_change":"2024-04-26T00:19:52.138726+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-26T00:20:00.477727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-04-26T00:20:01.233750+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2024-04-26T00:20:08.931001+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"898","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":449,"snap_epoch":898,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a5d4b9d3-5499-491f-a123-85d73de452af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":3204000569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":3204000569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6811","nonce":3204000569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":3204000569}]},"public_addr":"172.21.15.37:6806/3204000569","cluster_addr":"172.21.15.37:6808/3204000569","heartbeat_back_addr":"172.21.15.37:6811/3204000569","heartbeat_front_addr":"172.21.15.37:6810/3204000569","state":["exists","up"]},{"osd":1,"uuid":"cb4583fd-a0fa-4559-9f99-5a3e02b5818c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":1084527868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6805","nonce":1084527868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6809","nonce":1084527868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6807","nonce":1084527868}]},"public_addr":"172.21.15.37:6804/1084527868","cluster_addr":"172.21.15.37:6805/1084527868","heartbeat_back_addr":"172.21.15.37:6809/1084527868","heartbeat_front_addr":"172.21.15.37:6807/1084527868","state":["exists","up"]},{"osd":2,"uuid":"78fc9fe6-2163-4cb3-9944-3773bd488df8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":603175474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6801","nonce":603175474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6803","nonce":603175474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":603175474}]},"public_addr":"172.21.15.37:6800/603175474","cluster_addr":"172.21.15.37:6801/603175474","heartbeat_back_addr":"172.21.15.37:6803/603175474","heartbeat_front_addr":"172.21.15.37:6802/603175474","state":["exists","up"]},{"osd":3,"uuid":"338bb632-eca1-422b-b196-fd6cadb8307c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":2173737584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6801","nonce":2173737584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6803","nonce":2173737584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":2173737584}]},"public_addr":"172.21.15.45:6800/2173737584","cluster_addr":"172.21.15.45:6801/2173737584","heartbeat_back_addr":"172.21.15.45:6803/2173737584","heartbeat_front_addr":"172.21.15.45:6802/2173737584","state":["exists","up"]},{"osd":4,"uuid":"ce2e0db7-d486-4cb1-9913-6a4a011e5c8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":152605921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6805","nonce":152605921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6807","nonce":152605921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":152605921}]},"public_addr":"172.21.15.45:6804/152605921","cluster_addr":"172.21.15.45:6805/152605921","heartbeat_back_addr":"172.21.15.45:6807/152605921","heartbeat_front_addr":"172.21.15.45:6806/152605921","state":["exists","up"]},{"osd":5,"uuid":"a61739cc-41d3-48b1-a471-4092b2edbf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3915746722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6809","nonce":3915746722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6811","nonce":3915746722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3915746722}]},"public_addr":"172.21.15.45:6808/3915746722","cluster_addr":"172.21.15.45:6809/3915746722","heartbeat_back_addr":"172.21.15.45:6811/3915746722","heartbeat_front_addr":"172.21.15.45:6810/3915746722","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T00:19:57.580195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T00:19:57.630479+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T00:19:57.716834+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T00:19:57.665953+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T00:19:57.633161+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T00:19:57.663690+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":429,"length":1},{"begin":449,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":449,"length":1}]}],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":415,"length":1},{"begin":426,"length":1},{"begin":442,"length":1},{"begin":445,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T00:42:41.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.824+0000 7fa12da8f640 1 -- 172.21.15.37:0/1037462075 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa114035e80 msgr2=0x7fa114038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:41.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.824+0000 7fa12da8f640 1 --2- 172.21.15.37:0/1037462075 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa114035e80 0x7fa114038340 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fa118006fd0 tx=0x7fa118006d40 comp rx=0 tx=0).stop 2024-04-26T00:42:41.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.824+0000 7fa12da8f640 10 monclient: shutdown 2024-04-26T00:42:41.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.824+0000 7fa12da8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:42:41.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.824+0000 7fa12da8f640 1 -- 172.21.15.37:0/1037462075 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1281376c0 msgr2=0x7fa1281cf1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:41.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.824+0000 7fa12da8f640 1 --2- 172.21.15.37:0/1037462075 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1281376c0 0x7fa1281cf1f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa12000be90 tx=0x7fa12000bf80 comp rx=0 tx=0).stop 2024-04-26T00:42:41.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.824+0000 7fa12da8f640 1 -- 172.21.15.37:0/1037462075 shutdown_connections 2024-04-26T00:42:41.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.824+0000 7fa12da8f640 1 --2- 172.21.15.37:0/1037462075 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa114035e80 0x7fa114038340 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:41.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.824+0000 7fa12da8f640 1 --2- 172.21.15.37:0/1037462075 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1281376c0 0x7fa1281cf1f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:41.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.824+0000 7fa12da8f640 1 --2- 172.21.15.37:0/1037462075 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa128136da0 0x7fa128131610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:41.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.824+0000 7fa12da8f640 1 --2- 172.21.15.37:0/1037462075 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa12809c410 0x7fa1281310d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:41.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.824+0000 7fa12da8f640 1 -- 172.21.15.37:0/1037462075 >> 172.21.15.37:0/1037462075 conn(0x7fa1280a5f00 msgr2=0x7fa128136040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:42:41.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.824+0000 7fa12da8f640 1 -- 172.21.15.37:0/1037462075 shutdown_connections 2024-04-26T00:42:41.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.824+0000 7fa12da8f640 1 -- 172.21.15.37:0/1037462075 wait complete. 2024-04-26T00:42:41.847 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T00:42:41.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.960+0000 7fc4ee76a640 10 monclient: get_monmap_and_config 2024-04-26T00:42:41.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.960+0000 7fc4ee76a640 10 monclient: build_initial_monmap 2024-04-26T00:42:41.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.960+0000 7fc4ee76a640 10 monclient: monmap: 2024-04-26T00:42:41.965 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:42:41.965 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:41.965 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:42:41.966603+0000 2024-04-26T00:42:41.965 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:42:41.966603+0000 2024-04-26T00:42:41.966 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:42:41.966 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:41.966 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:42:41.966 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:42:41.966 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:42:41.966 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:42:41.966 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:42:41.966 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:41.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.960+0000 7fc4ee76a640 1 Processor -- start 2024-04-26T00:42:41.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.960+0000 7fc4ee76a640 1 -- start start 2024-04-26T00:42:41.968 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.960+0000 7fc4ee76a640 10 monclient: init 2024-04-26T00:42:41.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.964+0000 7fc4ee76a640 10 monclient: _reopen_session rank -1 2024-04-26T00:42:41.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.964+0000 7fc4ee76a640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-04-26T00:42:41.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.964+0000 7fc4ee76a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc4e80a0260 addr v1:172.21.15.45:6789/0 2024-04-26T00:42:41.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.964+0000 7fc4ee76a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc4e8137e40 addr v1:172.21.15.37:6789/0 2024-04-26T00:42:41.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.964+0000 7fc4ee76a640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc4e813b950 0x7fc4e813dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:41.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.964+0000 7fc4ee76a640 10 monclient(hunting): picked mon.noname-c con 0x7fc4e813b950 addr v2:172.21.15.45:3300/0 2024-04-26T00:42:41.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.964+0000 7fc4ee76a640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4e80a1130 con 0x7fc4e8137e40 2024-04-26T00:42:41.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.964+0000 7fc4ee76a640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4e806cff0 con 0x7fc4e80a0260 2024-04-26T00:42:41.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.964+0000 7fc4ee76a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:41.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.964+0000 7fc4ee76a640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc4e80a0d50 con 0x7fc4e813b950 2024-04-26T00:42:41.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.964+0000 7fc4ee76a640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:41.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.964+0000 7fc4ee76a640 10 monclient(hunting): authenticate will time out at 2343.272375s 2024-04-26T00:42:41.969 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.964+0000 7fc4e77fe640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7fc4e8137e40 legacy=0x7fc4e813a230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:42:41.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.964+0000 7fc4ecd07640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc4e813b950 0x7fc4e813dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:41.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.964+0000 7fc4e7fff640 1 --1- >> v1:172.21.15.45:6789/0 conn(0x7fc4e80a0260 0x7fc4e80a0640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.45:6789/0 says I am v1:172.21.15.37:44872/0 (socket says 172.21.15.37:44872) 2024-04-26T00:42:41.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.964+0000 7fc4e7fff640 1 -- 172.21.15.37:0/2506435791 learned_addr learned my addr 172.21.15.37:0/2506435791 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:42:41.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.964+0000 7fc4ecd07640 10 monclient(hunting): get_auth_request con 0x7fc4e813b950 auth_method 0 2024-04-26T00:42:41.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.964+0000 7fc4ecd07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:41.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.964+0000 7fc4ecd07640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:41.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.964+0000 7fc4ecd07640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:41.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.964+0000 7fc4ecd07640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:41.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.964+0000 7fc4ecd07640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:41.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.964+0000 7fc4ecd07640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:41.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.964+0000 7fc4ecd07640 10 monclient(hunting): handle_auth_done global_id 274162 payload 274 2024-04-26T00:42:41.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.964+0000 7fc4ecd07640 1 -- 172.21.15.37:0/2506435791 >> v1:172.21.15.45:6789/0 conn(0x7fc4e80a0260 legacy=0x7fc4e80a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:42:41.970 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.964+0000 7fc4ecd07640 1 -- 172.21.15.37:0/2506435791 >> v1:172.21.15.37:6789/0 conn(0x7fc4e8137e40 legacy=0x7fc4e813a230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:42:41.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.964+0000 7fc4ecd07640 10 monclient: _finish_hunting 0 2024-04-26T00:42:41.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.964+0000 7fc4ecd07640 1 monclient: found mon.noname-c 2024-04-26T00:42:41.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.964+0000 7fc4ecd07640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:42:41.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.964+0000 7fc4ecd07640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:42:41.971 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.964+0000 7fc4ecd07640 1 -- 172.21.15.37:0/2506435791 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4e809bc50 con 0x7fc4e813b950 2024-04-26T00:42:41.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.968+0000 7fc4ecd07640 1 -- 172.21.15.37:0/2506435791 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4e813b950 msgr2=0x7fc4e813dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:41.972 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.968+0000 7fc4ecd07640 1 --2- 172.21.15.37:0/2506435791 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4e813b950 0x7fc4e813dd40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc4d00097c0 tx=0x7fc4d001c100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=258e015c05bf226a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:41.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.968+0000 7fc4e6ffd640 1 -- 172.21.15.37:0/2506435791 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d00b1070 con 0x7fc4e813b950 2024-04-26T00:42:41.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.968+0000 7fc4e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:41.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.968+0000 7fc4e6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:42:41.973 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.968+0000 7fc4e6ffd640 10 monclient: dump: 2024-04-26T00:42:41.973 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:41.974 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:41.974 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:41.974 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:41.974 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:41.974 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:41.974 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:41.974 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:41.974 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:41.974 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:41.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.968+0000 7fc4e6ffd640 10 monclient: _finish_auth 0 2024-04-26T00:42:41.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.968+0000 7fc4e6ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:42:41.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.968+0000 7fc4e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:42:41.974 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.968+0000 7fc4e6ffd640 1 -- 172.21.15.37:0/2506435791 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4d00ac440 con 0x7fc4e813b950 2024-04-26T00:42:41.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.968+0000 7fc4e6ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:42:41.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.968+0000 7fc4e6ffd640 1 -- 172.21.15.37:0/2506435791 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d00ac810 con 0x7fc4e813b950 2024-04-26T00:42:41.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.968+0000 7fc4e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:41.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.968+0000 7fc4e6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:42:41.975 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.968+0000 7fc4e6ffd640 10 monclient: dump: 2024-04-26T00:42:41.975 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:41.975 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:41.975 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:41.975 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:41.975 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:41.975 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:41.975 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:41.975 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:41.975 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:41.976 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:41.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.968+0000 7fc4ee76a640 5 monclient: authenticate success, global_id 274162 2024-04-26T00:42:41.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.968+0000 7fc4ee76a640 10 monclient: get_monmap_and_config success 2024-04-26T00:42:41.976 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.968+0000 7fc4ee76a640 4 set_mon_vals no callback set 2024-04-26T00:42:41.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.968+0000 7fc4ee76a640 10 monclient: shutdown 2024-04-26T00:42:41.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.968+0000 7fc4ee76a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:42:41.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.968+0000 7fc4ee76a640 1 -- 172.21.15.37:0/2506435791 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4e813b950 msgr2=0x7fc4e813dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:41.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.968+0000 7fc4ee76a640 1 --2- 172.21.15.37:0/2506435791 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4e813b950 0x7fc4e813dd40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc4d00097c0 tx=0x7fc4d001c100 comp rx=0 tx=0).stop 2024-04-26T00:42:41.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4ee76a640 1 -- 172.21.15.37:0/2506435791 shutdown_connections 2024-04-26T00:42:41.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4ee76a640 1 --2- 172.21.15.37:0/2506435791 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4e813b950 0x7fc4e813dd40 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:41.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4ee76a640 1 -- 172.21.15.37:0/2506435791 >> 172.21.15.37:0/2506435791 conn(0x7fc4e80a1ad0 msgr2=0x7fc4e80a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:42:41.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4ee76a640 1 -- 172.21.15.37:0/2506435791 shutdown_connections 2024-04-26T00:42:41.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4ee76a640 1 -- 172.21.15.37:0/2506435791 wait complete. 2024-04-26T00:42:41.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4ee76a640 10 monclient: build_initial_monmap 2024-04-26T00:42:41.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4ee76a640 10 monclient: monmap: 2024-04-26T00:42:41.981 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:42:41.982 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:42:41.982 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:42:41.982 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:42:41.982 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:42:41.982 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:42:41.982 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:42:41.982 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:42:41.982 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:42:41.982 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:41.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4ee76a640 1 Processor -- start 2024-04-26T00:42:41.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4ee76a640 1 -- start start 2024-04-26T00:42:41.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4ee76a640 10 monclient: init 2024-04-26T00:42:41.982 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4ee76a640 10 monclient: _reopen_session rank -1 2024-04-26T00:42:41.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4ee76a640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:42:41.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4ee76a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc4e80a0260 0x7fc4e81367f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:41.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4ee76a640 10 monclient(hunting): picked mon.noname-a con 0x7fc4e80a0260 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:42:41.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4ee76a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc4e8137e40 0x7fc4e811c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:41.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4ee76a640 10 monclient(hunting): picked mon.noname-c con 0x7fc4e8137e40 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:42:41.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4ee76a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4e813b950 0x7fc4e811c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:41.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4ee76a640 10 monclient(hunting): picked mon.noname-b con 0x7fc4e813b950 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:42:41.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4ee76a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:41.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4ee76a640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fc4e809f0f0 con 0x7fc4e80a0260 2024-04-26T00:42:41.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4ee76a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:41.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4ee76a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc4e809f4c0 con 0x7fc4e813b950 2024-04-26T00:42:41.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4ee76a640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:41.983 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4ee76a640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc4e80a00a0 con 0x7fc4e8137e40 2024-04-26T00:42:41.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4ee76a640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:41.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4ee76a640 10 monclient(hunting): authenticate will time out at 2343.281959s 2024-04-26T00:42:41.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4e7fff640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc4e80a0260 msgr2=0x7fc4e81367f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:42:41.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4e7fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc4e80a0260 0x7fc4e81367f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:42:41.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4e77fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc4e8137e40 0x7fc4e811c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:41.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4e77fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc4e8137e40 0x7fc4e811c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:48532/0 (socket says 172.21.15.37:48532) 2024-04-26T00:42:41.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4e77fe640 1 -- 172.21.15.37:0/183906490 learned_addr learned my addr 172.21.15.37:0/183906490 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:42:41.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4e77fe640 10 monclient(hunting): get_auth_request con 0x7fc4e8137e40 auth_method 0 2024-04-26T00:42:41.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:41.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4e77fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:41.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:41.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:41.984 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:41.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:41.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4ecd07640 1 --2- 172.21.15.37:0/183906490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4e813b950 0x7fc4e811c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:41.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4ecd07640 10 monclient(hunting): get_auth_request con 0x7fc4e813b950 auth_method 0 2024-04-26T00:42:41.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4ecd07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:41.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4ecd07640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:41.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4ecd07640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:41.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4e77fe640 10 monclient(hunting): handle_auth_done global_id 254396 payload 824 2024-04-26T00:42:41.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4e77fe640 1 -- 172.21.15.37:0/183906490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4e813b950 msgr2=0x7fc4e811c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:41.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4e77fe640 1 --2- 172.21.15.37:0/183906490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4e813b950 0x7fc4e811c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:41.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4e77fe640 1 -- 172.21.15.37:0/183906490 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc4e80a0260 msgr2=0x7fc4e81367f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:42:41.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4e77fe640 1 --2- 172.21.15.37:0/183906490 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc4e80a0260 0x7fc4e81367f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:41.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4e77fe640 10 monclient: _finish_hunting 0 2024-04-26T00:42:41.985 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4e77fe640 1 monclient: found mon.noname-c 2024-04-26T00:42:41.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:42:41.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4e77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:42:41.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.972+0000 7fc4e77fe640 1 -- 172.21.15.37:0/183906490 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4e81cda70 con 0x7fc4e8137e40 2024-04-26T00:42:41.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.976+0000 7fc4e77fe640 1 -- 172.21.15.37:0/183906490 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc4e8137e40 msgr2=0x7fc4e811c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:41.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.976+0000 7fc4e77fe640 1 --2- 172.21.15.37:0/183906490 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc4e8137e40 0x7fc4e811c230 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc4d800b9e0 tx=0x7fc4d800bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:41.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.976+0000 7fc4e4ff9640 1 -- 172.21.15.37:0/183906490 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d800cdb0 con 0x7fc4e8137e40 2024-04-26T00:42:41.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.976+0000 7fc4e4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:41.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.976+0000 7fc4e4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:42:41.986 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.976+0000 7fc4e4ff9640 10 monclient: dump: 2024-04-26T00:42:41.986 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:41.986 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:41.986 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:41.986 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:41.986 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:41.987 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:41.987 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:41.987 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:41.987 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:41.987 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:41.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.976+0000 7fc4e4ff9640 10 monclient: _finish_auth 0 2024-04-26T00:42:41.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.976+0000 7fc4e4ff9640 10 monclient: _check_auth_tickets 2024-04-26T00:42:41.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.976+0000 7fc4e4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:42:41.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.976+0000 7fc4ee76a640 5 monclient: authenticate success, global_id 254396 2024-04-26T00:42:41.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.976+0000 7fc4e4ff9640 1 -- 172.21.15.37:0/183906490 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4d8010070 con 0x7fc4e8137e40 2024-04-26T00:42:41.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.976+0000 7fc4ee76a640 10 monclient: _renew_subs 2024-04-26T00:42:41.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.976+0000 7fc4ee76a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:41.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.976+0000 7fc4ee76a640 1 -- 172.21.15.37:0/183906490 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4e80729d0 con 0x7fc4e8137e40 2024-04-26T00:42:41.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.976+0000 7fc4e4ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T00:42:41.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.976+0000 7fc4e4ff9640 1 -- 172.21.15.37:0/183906490 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d80156b0 con 0x7fc4e8137e40 2024-04-26T00:42:41.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.976+0000 7fc4e4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:41.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.976+0000 7fc4e4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:42:41.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.976+0000 7fc4e4ff9640 10 monclient: dump: 2024-04-26T00:42:41.988 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:41.988 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:41.988 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:41.988 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:41.988 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:41.988 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:41.988 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:41.988 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:41.988 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:41.988 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:41.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.976+0000 7fc4ee76a640 10 monclient: _renew_subs 2024-04-26T00:42:41.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.976+0000 7fc4ee76a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:41.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.976+0000 7fc4e67fc640 4 set_mon_vals no callback set 2024-04-26T00:42:41.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.976+0000 7fc4ee76a640 1 -- 172.21.15.37:0/183906490 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4e8120170 con 0x7fc4e8137e40 2024-04-26T00:42:41.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.976+0000 7fc4ee76a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:41.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.976+0000 7fc4ee76a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:41.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.976+0000 7fc4ee76a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:41.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.976+0000 7fc4ee76a640 1 -- 172.21.15.37:0/183906490 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4e8066f40 con 0x7fc4e8137e40 2024-04-26T00:42:41.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.976+0000 7fc4e4ff9640 1 -- 172.21.15.37:0/183906490 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc4d801d390 con 0x7fc4e8137e40 2024-04-26T00:42:41.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.976+0000 7fc4e4ff9640 1 --2- 172.21.15.37:0/183906490 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc4c8035e30 0x7fc4c80382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:41.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.976+0000 7fc4e4ff9640 1 -- 172.21.15.37:0/183906490 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(898..898 src has 822..898) ==== 4019+0+0 (secure 0 0 0) 0x7fc4d8048ef0 con 0x7fc4e8137e40 2024-04-26T00:42:41.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.976+0000 7fc4e7fff640 1 --2- 172.21.15.37:0/183906490 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc4c8035e30 0x7fc4c80382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:41.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.976+0000 7fc4e7fff640 10 monclient: get_auth_request con 0x7fc4c8035e30 auth_method 0 2024-04-26T00:42:41.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.980+0000 7fc4e7fff640 1 --2- 172.21.15.37:0/183906490 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc4c8035e30 0x7fc4c80382f0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fc4e0004820 tx=0x7fc4e0007b30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:41.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.980+0000 7fc4e4ff9640 1 -- 172.21.15.37:0/183906490 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc4d801a060 con 0x7fc4e8137e40 2024-04-26T00:42:41.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.980+0000 7fc4e4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:41.990 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:41.980+0000 7fc4e4ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:42:42.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.140+0000 7fc4ee76a640 1 -- 172.21.15.37:0/183906490 --> v2:172.21.15.45:6812/2379918623 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc4e811e760 con 0x7fc4c8035e30 2024-04-26T00:42:42.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.148+0000 7fc4e4ff9640 1 -- 172.21.15.37:0/183906490 <== mgr.4107 v2:172.21.15.45:6812/2379918623 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+78146 (secure 0 0 0) 0x7fc4e811e760 con 0x7fc4c8035e30 2024-04-26T00:42:42.154 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:42:42.155 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-04-26T00:42:42.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.152+0000 7fc4ee76a640 1 -- 172.21.15.37:0/183906490 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc4c8035e30 msgr2=0x7fc4c80382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:42.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.152+0000 7fc4ee76a640 1 --2- 172.21.15.37:0/183906490 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc4c8035e30 0x7fc4c80382f0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fc4e0004820 tx=0x7fc4e0007b30 comp rx=0 tx=0).stop 2024-04-26T00:42:42.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.152+0000 7fc4ee76a640 10 monclient: shutdown 2024-04-26T00:42:42.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.152+0000 7fc4ee76a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:42:42.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.152+0000 7fc4ee76a640 1 -- 172.21.15.37:0/183906490 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc4e8137e40 msgr2=0x7fc4e811c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:42.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.152+0000 7fc4ee76a640 1 --2- 172.21.15.37:0/183906490 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc4e8137e40 0x7fc4e811c230 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc4d800b9e0 tx=0x7fc4d800bea0 comp rx=0 tx=0).stop 2024-04-26T00:42:42.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.152+0000 7fc4ee76a640 1 -- 172.21.15.37:0/183906490 shutdown_connections 2024-04-26T00:42:42.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.152+0000 7fc4ee76a640 1 --2- 172.21.15.37:0/183906490 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc4c8035e30 0x7fc4c80382f0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:42.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.152+0000 7fc4ee76a640 1 --2- 172.21.15.37:0/183906490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4e813b950 0x7fc4e811c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:42.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.152+0000 7fc4ee76a640 1 --2- 172.21.15.37:0/183906490 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc4e8137e40 0x7fc4e811c230 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:42.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.152+0000 7fc4ee76a640 1 --2- 172.21.15.37:0/183906490 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc4e80a0260 0x7fc4e81367f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:42.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.152+0000 7fc4ee76a640 1 -- 172.21.15.37:0/183906490 >> 172.21.15.37:0/183906490 conn(0x7fc4e80a1ad0 msgr2=0x7fc4e80a3d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:42:42.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.152+0000 7fc4ee76a640 1 -- 172.21.15.37:0/183906490 shutdown_connections 2024-04-26T00:42:42.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.152+0000 7fc4ee76a640 1 -- 172.21.15.37:0/183906490 wait complete. 2024-04-26T00:42:42.173 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":1563,"stamp":"2024-04-26T00:42:41.540375+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":421177490,"num_objects":192,"num_object_clones":142,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":192,"num_whiteouts":17,"num_read":7917,"num_read_kb":7750911,"num_write":5814,"num_write_kb":2187069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23633,"ondisk_log_size":23633,"up":50,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":50,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":819544,"kb_used_data":491280,"kb_used_omap":9,"kb_used_meta":324342,"kb_avail":565411496,"statfs":{"total":579820584960,"available":578981371904,"internally_reserved":0,"allocated":503070720,"data_stored":653974814,"data_compressed":12018810,"data_compressed_allocated":9297920,"data_compressed_original":31731610,"omap_allocated":9568,"internal_metadata":332126880},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002394"},"pg_stats":[{"pgid":"3.f","version":"0'0","reported_seq":4260,"reported_epoch":896,"state":"active+clean","last_fresh":"2024-04-26T00:42:24.364291+0000","last_change":"2024-04-26T00:42:24.364291+0000","last_active":"2024-04-26T00:42:24.364291+0000","last_peered":"2024-04-26T00:42:24.364291+0000","last_clean":"2024-04-26T00:42:24.364291+0000","last_became_active":"2024-04-26T00:20:25.845857+0000","last_became_peered":"2024-04-26T00:20:25.845857+0000","last_unstale":"2024-04-26T00:42:24.364291+0000","last_undegraded":"2024-04-26T00:42:24.364291+0000","last_fullsized":"2024-04-26T00:42:24.364291+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:09.300089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:09.300089+0000","last_clean_scrub_stamp":"2024-04-26T00:20:09.300089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:21:42.849528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012294388999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"19f","length":"1"},{"start":"1aa","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"}]},{"pgid":"3.e","version":"896'2442","reported_seq":6597,"reported_epoch":896,"state":"active+clean","last_fresh":"2024-04-26T00:42:24.678749+0000","last_change":"2024-04-26T00:42:24.382838+0000","last_active":"2024-04-26T00:42:24.678749+0000","last_peered":"2024-04-26T00:42:24.678749+0000","last_clean":"2024-04-26T00:42:24.678749+0000","last_became_active":"2024-04-26T00:20:25.843544+0000","last_became_peered":"2024-04-26T00:20:25.843544+0000","last_unstale":"2024-04-26T00:42:24.678749+0000","last_undegraded":"2024-04-26T00:42:24.678749+0000","last_fullsized":"2024-04-26T00:42:24.678749+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:09.300089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:09.300089+0000","last_clean_scrub_stamp":"2024-04-26T00:20:09.300089+0000","objects_scrubbed":0,"log_size":2442,"log_dups_size":0,"ondisk_log_size":2442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:17:35.697279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.030131631999999998,"stat_sum":{"num_bytes":44000460,"num_objects":21,"num_object_clones":16,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":3,"num_read":740,"num_read_kb":719343,"num_write":656,"num_write_kb":220054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"19f","length":"1"},{"start":"1aa","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"}]},{"pgid":"3.d","version":"0'0","reported_seq":4269,"reported_epoch":896,"state":"active+clean","last_fresh":"2024-04-26T00:42:24.605026+0000","last_change":"2024-04-26T00:42:24.605026+0000","last_active":"2024-04-26T00:42:24.605026+0000","last_peered":"2024-04-26T00:42:24.605026+0000","last_clean":"2024-04-26T00:42:24.605026+0000","last_became_active":"2024-04-26T00:20:25.839328+0000","last_became_peered":"2024-04-26T00:20:25.839328+0000","last_unstale":"2024-04-26T00:42:24.605026+0000","last_undegraded":"2024-04-26T00:42:24.605026+0000","last_fullsized":"2024-04-26T00:42:24.605026+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:09.300089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:09.300089+0000","last_clean_scrub_stamp":"2024-04-26T00:20:09.300089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:33:04.033689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001002055,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"19f","length":"1"},{"start":"1aa","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"}]},{"pgid":"3.c","version":"896'1316","reported_seq":5553,"reported_epoch":896,"state":"active+clean","last_fresh":"2024-04-26T00:42:24.363364+0000","last_change":"2024-04-26T00:42:24.363364+0000","last_active":"2024-04-26T00:42:24.363364+0000","last_peered":"2024-04-26T00:42:24.363364+0000","last_clean":"2024-04-26T00:42:24.363364+0000","last_became_active":"2024-04-26T00:20:25.843602+0000","last_became_peered":"2024-04-26T00:20:25.843602+0000","last_unstale":"2024-04-26T00:42:24.363364+0000","last_undegraded":"2024-04-26T00:42:24.363364+0000","last_fullsized":"2024-04-26T00:42:24.363364+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:09.300089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:09.300089+0000","last_clean_scrub_stamp":"2024-04-26T00:20:09.300089+0000","objects_scrubbed":0,"log_size":1316,"log_dups_size":0,"ondisk_log_size":1316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:47:32.059158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011387936,"stat_sum":{"num_bytes":31818984,"num_objects":12,"num_object_clones":9,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":473,"num_read_kb":563421,"num_write":305,"num_write_kb":130452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"19f","length":"1"},{"start":"1aa","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"}]},{"pgid":"3.b","version":"896'370","reported_seq":4693,"reported_epoch":896,"state":"active+clean","last_fresh":"2024-04-26T00:42:24.375546+0000","last_change":"2024-04-26T00:42:24.375546+0000","last_active":"2024-04-26T00:42:24.375546+0000","last_peered":"2024-04-26T00:42:24.375546+0000","last_clean":"2024-04-26T00:42:24.375546+0000","last_became_active":"2024-04-26T00:20:25.835412+0000","last_became_peered":"2024-04-26T00:20:25.835412+0000","last_unstale":"2024-04-26T00:42:24.375546+0000","last_undegraded":"2024-04-26T00:42:24.375546+0000","last_fullsized":"2024-04-26T00:42:24.375546+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:09.300089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:09.300089+0000","last_clean_scrub_stamp":"2024-04-26T00:20:09.300089+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:18:16.096951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.028852332000000001,"stat_sum":{"num_bytes":8084193,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":135,"num_read_kb":122322,"num_write":73,"num_write_kb":38576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"19f","length":"1"},{"start":"1aa","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"}]},{"pgid":"3.a","version":"896'950","reported_seq":5209,"reported_epoch":896,"state":"active+clean","last_fresh":"2024-04-26T00:42:24.638722+0000","last_change":"2024-04-26T00:42:24.619658+0000","last_active":"2024-04-26T00:42:24.638722+0000","last_peered":"2024-04-26T00:42:24.638722+0000","last_clean":"2024-04-26T00:42:24.638722+0000","last_became_active":"2024-04-26T00:20:25.841624+0000","last_became_peered":"2024-04-26T00:20:25.841624+0000","last_unstale":"2024-04-26T00:42:24.638722+0000","last_undegraded":"2024-04-26T00:42:24.638722+0000","last_fullsized":"2024-04-26T00:42:24.638722+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:09.300089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:09.300089+0000","last_clean_scrub_stamp":"2024-04-26T00:20:09.300089+0000","objects_scrubbed":0,"log_size":950,"log_dups_size":0,"ondisk_log_size":950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:27.690313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015715309,"stat_sum":{"num_bytes":23412534,"num_objects":10,"num_object_clones":8,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":347,"num_read_kb":350530,"num_write":230,"num_write_kb":88186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"19f","length":"1"},{"start":"1aa","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"}]},{"pgid":"3.9","version":"896'919","reported_seq":5205,"reported_epoch":896,"state":"active+clean","last_fresh":"2024-04-26T00:42:24.377366+0000","last_change":"2024-04-26T00:42:24.346409+0000","last_active":"2024-04-26T00:42:24.377366+0000","last_peered":"2024-04-26T00:42:24.377366+0000","last_clean":"2024-04-26T00:42:24.377366+0000","last_became_active":"2024-04-26T00:20:25.839823+0000","last_became_peered":"2024-04-26T00:20:25.839823+0000","last_unstale":"2024-04-26T00:42:24.377366+0000","last_undegraded":"2024-04-26T00:42:24.377366+0000","last_fullsized":"2024-04-26T00:42:24.377366+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:09.300089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:09.300089+0000","last_clean_scrub_stamp":"2024-04-26T00:20:09.300089+0000","objects_scrubbed":0,"log_size":919,"log_dups_size":0,"ondisk_log_size":919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:00:34.362483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0078471319999999997,"stat_sum":{"num_bytes":13077001,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":319,"num_read_kb":319417,"num_write":249,"num_write_kb":103414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"19f","length":"1"},{"start":"1aa","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"}]},{"pgid":"3.8","version":"896'2625","reported_seq":6761,"reported_epoch":896,"state":"active+clean","last_fresh":"2024-04-26T00:42:24.915200+0000","last_change":"2024-04-26T00:42:24.368539+0000","last_active":"2024-04-26T00:42:24.915200+0000","last_peered":"2024-04-26T00:42:24.915200+0000","last_clean":"2024-04-26T00:42:24.915200+0000","last_became_active":"2024-04-26T00:20:25.830241+0000","last_became_peered":"2024-04-26T00:20:25.830241+0000","last_unstale":"2024-04-26T00:42:24.915200+0000","last_undegraded":"2024-04-26T00:42:24.915200+0000","last_fullsized":"2024-04-26T00:42:24.915200+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:09.300089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:09.300089+0000","last_clean_scrub_stamp":"2024-04-26T00:20:09.300089+0000","objects_scrubbed":0,"log_size":2625,"log_dups_size":0,"ondisk_log_size":2625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:36:08.765892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.029758117000000001,"stat_sum":{"num_bytes":49803992,"num_objects":21,"num_object_clones":16,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":2,"num_read":770,"num_read_kb":747580,"num_write":583,"num_write_kb":252326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"19f","length":"1"},{"start":"1aa","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"}]},{"pgid":"1.0","version":"0'0","reported_seq":1749,"reported_epoch":896,"state":"active+clean","last_fresh":"2024-04-26T00:42:24.603713+0000","last_change":"2024-04-26T00:20:02.279764+0000","last_active":"2024-04-26T00:42:24.603713+0000","last_peered":"2024-04-26T00:42:24.603713+0000","last_clean":"2024-04-26T00:42:24.603713+0000","last_became_active":"2024-04-26T00:20:02.278959+0000","last_became_peered":"2024-04-26T00:20:02.278959+0000","last_unstale":"2024-04-26T00:42:24.603713+0000","last_undegraded":"2024-04-26T00:42:24.603713+0000","last_fullsized":"2024-04-26T00:42:24.603713+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:23:43.610174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"896'1330","reported_seq":5591,"reported_epoch":896,"state":"active+clean","last_fresh":"2024-04-26T00:42:24.353578+0000","last_change":"2024-04-26T00:42:24.353948+0000","last_active":"2024-04-26T00:42:24.353578+0000","last_peered":"2024-04-26T00:42:24.353578+0000","last_clean":"2024-04-26T00:42:24.353578+0000","last_became_active":"2024-04-26T00:20:25.836732+0000","last_became_peered":"2024-04-26T00:20:25.836732+0000","last_unstale":"2024-04-26T00:42:24.353578+0000","last_undegraded":"2024-04-26T00:42:24.353578+0000","last_fullsized":"2024-04-26T00:42:24.353578+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:09.300089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:09.300089+0000","last_clean_scrub_stamp":"2024-04-26T00:20:09.300089+0000","objects_scrubbed":0,"log_size":1330,"log_dups_size":0,"ondisk_log_size":1330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:58:33.612291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074697110000000004,"stat_sum":{"num_bytes":22674229,"num_objects":11,"num_object_clones":8,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":410,"num_read_kb":409782,"num_write":308,"num_write_kb":113401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"19f","length":"1"},{"start":"1aa","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"}]},{"pgid":"1.1","version":"0'0","reported_seq":1743,"reported_epoch":896,"state":"active+clean","last_fresh":"2024-04-26T00:42:24.351583+0000","last_change":"2024-04-26T00:20:02.279805+0000","last_active":"2024-04-26T00:42:24.351583+0000","last_peered":"2024-04-26T00:42:24.351583+0000","last_clean":"2024-04-26T00:42:24.351583+0000","last_became_active":"2024-04-26T00:20:02.279053+0000","last_became_peered":"2024-04-26T00:20:02.279053+0000","last_unstale":"2024-04-26T00:42:24.351583+0000","last_undegraded":"2024-04-26T00:42:24.351583+0000","last_fullsized":"2024-04-26T00:42:24.351583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:11:26.730719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"896'1510","reported_seq":5722,"reported_epoch":896,"state":"active+clean","last_fresh":"2024-04-26T00:42:24.912621+0000","last_change":"2024-04-26T00:42:24.363147+0000","last_active":"2024-04-26T00:42:24.912621+0000","last_peered":"2024-04-26T00:42:24.912621+0000","last_clean":"2024-04-26T00:42:24.912621+0000","last_became_active":"2024-04-26T00:20:25.835784+0000","last_became_peered":"2024-04-26T00:20:25.835784+0000","last_unstale":"2024-04-26T00:42:24.912621+0000","last_undegraded":"2024-04-26T00:42:24.912621+0000","last_fullsized":"2024-04-26T00:42:24.912621+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:09.300089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:09.300089+0000","last_clean_scrub_stamp":"2024-04-26T00:20:09.300089+0000","objects_scrubbed":0,"log_size":1510,"log_dups_size":0,"ondisk_log_size":1510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:27:33.204035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.024437186,"stat_sum":{"num_bytes":28633074,"num_objects":10,"num_object_clones":7,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":463,"num_read_kb":509889,"num_write":339,"num_write_kb":119978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"19f","length":"1"},{"start":"1aa","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"}]},{"pgid":"1.2","version":"0'0","reported_seq":1754,"reported_epoch":896,"state":"active+clean","last_fresh":"2024-04-26T00:42:24.338399+0000","last_change":"2024-04-26T00:20:02.277311+0000","last_active":"2024-04-26T00:42:24.338399+0000","last_peered":"2024-04-26T00:42:24.338399+0000","last_clean":"2024-04-26T00:42:24.338399+0000","last_became_active":"2024-04-26T00:20:02.276409+0000","last_became_peered":"2024-04-26T00:20:02.276409+0000","last_unstale":"2024-04-26T00:42:24.338399+0000","last_undegraded":"2024-04-26T00:42:24.338399+0000","last_fullsized":"2024-04-26T00:42:24.338399+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:14:50.584720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"896'2108","reported_seq":6285,"reported_epoch":896,"state":"active+clean","last_fresh":"2024-04-26T00:42:24.379623+0000","last_change":"2024-04-26T00:42:24.379623+0000","last_active":"2024-04-26T00:42:24.379623+0000","last_peered":"2024-04-26T00:42:24.379623+0000","last_clean":"2024-04-26T00:42:24.379623+0000","last_became_active":"2024-04-26T00:20:25.832063+0000","last_became_peered":"2024-04-26T00:20:25.832063+0000","last_unstale":"2024-04-26T00:42:24.379623+0000","last_undegraded":"2024-04-26T00:42:24.379623+0000","last_fullsized":"2024-04-26T00:42:24.379623+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:09.300089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:09.300089+0000","last_clean_scrub_stamp":"2024-04-26T00:20:09.300089+0000","objects_scrubbed":0,"log_size":2108,"log_dups_size":0,"ondisk_log_size":2108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:13:46.689148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033009959999999998,"stat_sum":{"num_bytes":33632876,"num_objects":17,"num_object_clones":13,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":2,"num_read":661,"num_read_kb":647302,"num_write":545,"num_write_kb":223444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"19f","length":"1"},{"start":"1aa","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"}]},{"pgid":"1.3","version":"0'0","reported_seq":1756,"reported_epoch":896,"state":"active+clean","last_fresh":"2024-04-26T00:42:24.346394+0000","last_change":"2024-04-26T00:20:02.273965+0000","last_active":"2024-04-26T00:42:24.346394+0000","last_peered":"2024-04-26T00:42:24.346394+0000","last_clean":"2024-04-26T00:42:24.346394+0000","last_became_active":"2024-04-26T00:20:02.273213+0000","last_became_peered":"2024-04-26T00:20:02.273213+0000","last_unstale":"2024-04-26T00:42:24.346394+0000","last_undegraded":"2024-04-26T00:42:24.346394+0000","last_fullsized":"2024-04-26T00:42:24.346394+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:42:14.055403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"15'117","reported_seq":1899,"reported_epoch":896,"state":"active+clean","last_fresh":"2024-04-26T00:42:24.604322+0000","last_change":"2024-04-26T00:20:02.281782+0000","last_active":"2024-04-26T00:42:24.604322+0000","last_peered":"2024-04-26T00:42:24.604322+0000","last_clean":"2024-04-26T00:42:24.604322+0000","last_became_active":"2024-04-26T00:20:02.281000+0000","last_became_peered":"2024-04-26T00:20:02.281000+0000","last_unstale":"2024-04-26T00:42:24.604322+0000","last_undegraded":"2024-04-26T00:42:24.604322+0000","last_fullsized":"2024-04-26T00:42:24.604322+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:40:30.790500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"896'2514","reported_seq":6690,"reported_epoch":896,"state":"active+clean","last_fresh":"2024-04-26T00:42:24.918308+0000","last_change":"2024-04-26T00:42:24.362574+0000","last_active":"2024-04-26T00:42:24.918308+0000","last_peered":"2024-04-26T00:42:24.918308+0000","last_clean":"2024-04-26T00:42:24.918308+0000","last_became_active":"2024-04-26T00:20:25.833513+0000","last_became_peered":"2024-04-26T00:20:25.833513+0000","last_unstale":"2024-04-26T00:42:24.918308+0000","last_undegraded":"2024-04-26T00:42:24.918308+0000","last_fullsized":"2024-04-26T00:42:24.918308+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:09.300089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:09.300089+0000","last_clean_scrub_stamp":"2024-04-26T00:20:09.300089+0000","objects_scrubbed":0,"log_size":2514,"log_dups_size":0,"ondisk_log_size":2514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:36:53.041589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01965683,"stat_sum":{"num_bytes":61053610,"num_objects":26,"num_object_clones":21,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":1,"num_read":900,"num_read_kb":858537,"num_write":612,"num_write_kb":244160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"19f","length":"1"},{"start":"1aa","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"}]},{"pgid":"3.6","version":"896'853","reported_seq":5111,"reported_epoch":896,"state":"active+clean","last_fresh":"2024-04-26T00:42:24.361805+0000","last_change":"2024-04-26T00:42:24.361805+0000","last_active":"2024-04-26T00:42:24.361805+0000","last_peered":"2024-04-26T00:42:24.361805+0000","last_clean":"2024-04-26T00:42:24.361805+0000","last_became_active":"2024-04-26T00:20:25.832507+0000","last_became_peered":"2024-04-26T00:20:25.832507+0000","last_unstale":"2024-04-26T00:42:24.361805+0000","last_undegraded":"2024-04-26T00:42:24.361805+0000","last_fullsized":"2024-04-26T00:42:24.361805+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:09.300089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:09.300089+0000","last_clean_scrub_stamp":"2024-04-26T00:20:09.300089+0000","objects_scrubbed":0,"log_size":853,"log_dups_size":0,"ondisk_log_size":853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:22:42.171113+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.018611791999999999,"stat_sum":{"num_bytes":14711142,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":222,"num_read_kb":165138,"num_write":208,"num_write_kb":76632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"19f","length":"1"},{"start":"1aa","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"}]},{"pgid":"1.4","version":"0'0","reported_seq":1756,"reported_epoch":896,"state":"active+clean","last_fresh":"2024-04-26T00:42:24.346698+0000","last_change":"2024-04-26T00:20:02.274249+0000","last_active":"2024-04-26T00:42:24.346698+0000","last_peered":"2024-04-26T00:42:24.346698+0000","last_clean":"2024-04-26T00:42:24.346698+0000","last_became_active":"2024-04-26T00:20:02.273745+0000","last_became_peered":"2024-04-26T00:20:02.273745+0000","last_unstale":"2024-04-26T00:42:24.346698+0000","last_undegraded":"2024-04-26T00:42:24.346698+0000","last_fullsized":"2024-04-26T00:42:24.346698+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:36:25.948539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"896'1695","reported_seq":6010,"reported_epoch":896,"state":"active+clean","last_fresh":"2024-04-26T00:42:24.355601+0000","last_change":"2024-04-26T00:42:24.356098+0000","last_active":"2024-04-26T00:42:24.355601+0000","last_peered":"2024-04-26T00:42:24.355601+0000","last_clean":"2024-04-26T00:42:24.355601+0000","last_became_active":"2024-04-26T00:20:25.830310+0000","last_became_peered":"2024-04-26T00:20:25.830310+0000","last_unstale":"2024-04-26T00:42:24.355601+0000","last_undegraded":"2024-04-26T00:42:24.355601+0000","last_fullsized":"2024-04-26T00:42:24.355601+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:09.300089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:09.300089+0000","last_clean_scrub_stamp":"2024-04-26T00:20:09.300089+0000","objects_scrubbed":0,"log_size":1695,"log_dups_size":0,"ondisk_log_size":1695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:54:40.762389+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.016666846999999999,"stat_sum":{"num_bytes":19458075,"num_objects":8,"num_object_clones":5,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":749,"num_read_kb":810434,"num_write":372,"num_write_kb":141426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"19f","length":"1"},{"start":"1aa","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"}]},{"pgid":"1.5","version":"0'0","reported_seq":1754,"reported_epoch":896,"state":"active+clean","last_fresh":"2024-04-26T00:42:24.338619+0000","last_change":"2024-04-26T00:20:02.474779+0000","last_active":"2024-04-26T00:42:24.338619+0000","last_peered":"2024-04-26T00:42:24.338619+0000","last_clean":"2024-04-26T00:42:24.338619+0000","last_became_active":"2024-04-26T00:20:02.473888+0000","last_became_peered":"2024-04-26T00:20:02.473888+0000","last_unstale":"2024-04-26T00:42:24.338619+0000","last_undegraded":"2024-04-26T00:42:24.338619+0000","last_fullsized":"2024-04-26T00:42:24.338619+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:33:18.962907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"896'3053","reported_seq":7214,"reported_epoch":896,"state":"active+clean","last_fresh":"2024-04-26T00:42:24.583898+0000","last_change":"2024-04-26T00:42:24.373005+0000","last_active":"2024-04-26T00:42:24.583898+0000","last_peered":"2024-04-26T00:42:24.583898+0000","last_clean":"2024-04-26T00:42:24.583898+0000","last_became_active":"2024-04-26T00:20:25.831343+0000","last_became_peered":"2024-04-26T00:20:25.831343+0000","last_unstale":"2024-04-26T00:42:24.583898+0000","last_undegraded":"2024-04-26T00:42:24.583898+0000","last_fullsized":"2024-04-26T00:42:24.583898+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:09.300089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:09.300089+0000","last_clean_scrub_stamp":"2024-04-26T00:20:09.300089+0000","objects_scrubbed":0,"log_size":3053,"log_dups_size":0,"ondisk_log_size":3053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:16:16.816521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.026431518000000001,"stat_sum":{"num_bytes":48997665,"num_objects":26,"num_object_clones":20,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":2,"num_read":1084,"num_read_kb":1011240,"num_write":777,"num_write_kb":278155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"19f","length":"1"},{"start":"1aa","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"}]},{"pgid":"1.6","version":"0'0","reported_seq":1748,"reported_epoch":896,"state":"active+clean","last_fresh":"2024-04-26T00:42:24.603906+0000","last_change":"2024-04-26T00:20:02.279987+0000","last_active":"2024-04-26T00:42:24.603906+0000","last_peered":"2024-04-26T00:42:24.603906+0000","last_clean":"2024-04-26T00:42:24.603906+0000","last_became_active":"2024-04-26T00:20:02.279183+0000","last_became_peered":"2024-04-26T00:20:02.279183+0000","last_unstale":"2024-04-26T00:42:24.603906+0000","last_undegraded":"2024-04-26T00:42:24.603906+0000","last_fullsized":"2024-04-26T00:42:24.603906+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:45:44.508952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"896'1831","reported_seq":5997,"reported_epoch":896,"state":"active+clean","last_fresh":"2024-04-26T00:42:24.364125+0000","last_change":"2024-04-26T00:42:24.364688+0000","last_active":"2024-04-26T00:42:24.364125+0000","last_peered":"2024-04-26T00:42:24.364125+0000","last_clean":"2024-04-26T00:42:24.364125+0000","last_became_active":"2024-04-26T00:20:25.840458+0000","last_became_peered":"2024-04-26T00:20:25.840458+0000","last_unstale":"2024-04-26T00:42:24.364125+0000","last_undegraded":"2024-04-26T00:42:24.364125+0000","last_fullsized":"2024-04-26T00:42:24.364125+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:09.300089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:09.300089+0000","last_clean_scrub_stamp":"2024-04-26T00:20:09.300089+0000","objects_scrubbed":0,"log_size":1831,"log_dups_size":0,"ondisk_log_size":1831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:58:38.054103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012682809,"stat_sum":{"num_bytes":21229287,"num_objects":12,"num_object_clones":8,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":3,"num_read":528,"num_read_kb":515876,"num_write":398,"num_write_kb":154449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"19f","length":"1"},{"start":"1aa","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"}]},{"pgid":"1.7","version":"0'0","reported_seq":1743,"reported_epoch":896,"state":"active+clean","last_fresh":"2024-04-26T00:42:24.352095+0000","last_change":"2024-04-26T00:20:02.475146+0000","last_active":"2024-04-26T00:42:24.352095+0000","last_peered":"2024-04-26T00:42:24.352095+0000","last_clean":"2024-04-26T00:42:24.352095+0000","last_became_active":"2024-04-26T00:20:02.473928+0000","last_became_peered":"2024-04-26T00:20:02.473928+0000","last_unstale":"2024-04-26T00:42:24.352095+0000","last_undegraded":"2024-04-26T00:42:24.352095+0000","last_fullsized":"2024-04-26T00:42:24.352095+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:25:12.904295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":420587122,"num_objects":190,"num_object_clones":142,"num_object_copies":380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":190,"num_whiteouts":17,"num_read":7801,"num_read_kb":7750811,"num_write":5655,"num_write_kb":2184653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":499212288,"data_stored":650872968,"data_compressed":11957576,"data_compressed_allocated":9216000,"data_compressed_original":30707610,"omap_allocated":0,"internal_metadata":0},"log_size":23516,"ondisk_log_size":23516,"up":32,"acting":32,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":61234,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640531,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":129760,"kb_used_data":71988,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94242080,"statfs":{"total":96636764160,"available":96503889920,"internally_reserved":0,"allocated":73715712,"data_stored":98097243,"data_compressed":1655291,"data_compressed_allocated":1089536,"data_compressed_original":4390912,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640530,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":151796,"kb_used_data":101080,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94220044,"statfs":{"total":96636764160,"available":96481325056,"internally_reserved":0,"allocated":103505920,"data_stored":132191276,"data_compressed":2389657,"data_compressed_allocated":1777664,"data_compressed_original":6741507,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640532,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54572,"kb_used_data":14852,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94317268,"statfs":{"total":96636764160,"available":96580882432,"internally_reserved":0,"allocated":15208448,"data_stored":20220396,"data_compressed":371067,"data_compressed_allocated":319488,"data_compressed_original":1449862,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640532,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":156720,"kb_used_data":103232,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94215120,"statfs":{"total":96636764160,"available":96476282880,"internally_reserved":0,"allocated":105709568,"data_stored":138615207,"data_compressed":2792995,"data_compressed_allocated":2383872,"data_compressed_original":6590370,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640532,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":187016,"kb_used_data":115548,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94184824,"statfs":{"total":96636764160,"available":96445259776,"internally_reserved":0,"allocated":118321152,"data_stored":152670051,"data_compressed":2572134,"data_compressed_allocated":2179072,"data_compressed_original":6502858,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640531,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":139680,"kb_used_data":84580,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94232160,"statfs":{"total":96636764160,"available":96493731840,"internally_reserved":0,"allocated":86609920,"data_stored":112180641,"data_compressed":2237666,"data_compressed_allocated":1548288,"data_compressed_original":6056101,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":85987328,"data_stored":111839976,"data_compressed":2237666,"data_compressed_allocated":1548288,"data_compressed_original":6056101,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":117637120,"data_stored":151800458,"data_compressed":2541517,"data_compressed_allocated":2138112,"data_compressed_original":5990858,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":105086976,"data_stored":138274542,"data_compressed":2792995,"data_compressed_allocated":2383872,"data_compressed_original":6590370,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":14524416,"data_stored":19350803,"data_compressed":340450,"data_compressed_allocated":278528,"data_compressed_original":937862,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":102883328,"data_stored":131850611,"data_compressed":2389657,"data_compressed_allocated":1777664,"data_compressed_original":6741507,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":73093120,"data_stored":97756578,"data_compressed":1655291,"data_compressed_allocated":1089536,"data_compressed_original":4390912,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T00:42:42.175 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2024-04-26T00:42:42.175 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-04-26T00:42:42.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd338e4f640 10 monclient: get_monmap_and_config 2024-04-26T00:42:42.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd338e4f640 10 monclient: build_initial_monmap 2024-04-26T00:42:42.299 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd338e4f640 10 monclient: monmap: 2024-04-26T00:42:42.299 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:42:42.299 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:42.299 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:42:42.300281+0000 2024-04-26T00:42:42.299 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:42:42.300281+0000 2024-04-26T00:42:42.299 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:42:42.299 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:42.299 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:42:42.299 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:42:42.299 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:42:42.299 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:42:42.299 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:42:42.300 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:42.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd338e4f640 1 Processor -- start 2024-04-26T00:42:42.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd338e4f640 1 -- start start 2024-04-26T00:42:42.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd338e4f640 10 monclient: init 2024-04-26T00:42:42.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd338e4f640 10 monclient: _reopen_session rank -1 2024-04-26T00:42:42.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd338e4f640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-04-26T00:42:42.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd338e4f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd33409b5d0 addr v1:172.21.15.37:6789/0 2024-04-26T00:42:42.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd338e4f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd33409bf20 0x7fd3340a14e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:42.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd338e4f640 10 monclient(hunting): picked mon.noname-b con 0x7fd33409bf20 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:42:42.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd338e4f640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fd3340730c0 0x7fd33413c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:42.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd338e4f640 10 monclient(hunting): picked mon.noname-c con 0x7fd3340730c0 addr v2:172.21.15.45:3300/0 2024-04-26T00:42:42.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd338e4f640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd33409e3b0 con 0x7fd33409b5d0 2024-04-26T00:42:42.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd338e4f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:42.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd338e4f640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fd33409cd30 con 0x7fd3340730c0 2024-04-26T00:42:42.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd338e4f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:42.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd338e4f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd33409b9b0 con 0x7fd33409bf20 2024-04-26T00:42:42.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd338e4f640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:42.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd338e4f640 10 monclient(hunting): authenticate will time out at 2343.605911s 2024-04-26T00:42:42.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd33259c640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7fd33409b5d0 legacy=0x7fd3340a0dd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-04-26T00:42:42.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd329d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd33409bf20 0x7fd3340a14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:42.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd329d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd33409bf20 0x7fd3340a14e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:48534/0 (socket says 172.21.15.37:48534) 2024-04-26T00:42:42.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd329d9b640 1 -- 172.21.15.37:0/569385947 learned_addr learned my addr 172.21.15.37:0/569385947 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:42:42.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd329d9b640 10 monclient(hunting): get_auth_request con 0x7fd33409bf20 auth_method 0 2024-04-26T00:42:42.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd329d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:42.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd329d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:42.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd329d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:42.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd332d9d640 1 --2- 172.21.15.37:0/569385947 >> v2:172.21.15.45:3300/0 conn(0x7fd3340730c0 0x7fd33413c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:42.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd329d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:42.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd329d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:42.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd329d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:42.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd332d9d640 10 monclient(hunting): get_auth_request con 0x7fd3340730c0 auth_method 0 2024-04-26T00:42:42.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd332d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:42.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd332d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:42.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd332d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:42.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd329d9b640 10 monclient(hunting): handle_auth_done global_id 254399 payload 274 2024-04-26T00:42:42.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd329d9b640 1 -- 172.21.15.37:0/569385947 >> v2:172.21.15.45:3300/0 conn(0x7fd3340730c0 msgr2=0x7fd33413c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:42:42.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd329d9b640 1 --2- 172.21.15.37:0/569385947 >> v2:172.21.15.45:3300/0 conn(0x7fd3340730c0 0x7fd33413c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:42.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd329d9b640 1 -- 172.21.15.37:0/569385947 >> v1:172.21.15.37:6789/0 conn(0x7fd33409b5d0 legacy=0x7fd3340a0dd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:42:42.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd329d9b640 10 monclient: _finish_hunting 0 2024-04-26T00:42:42.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd329d9b640 1 monclient: found mon.noname-b 2024-04-26T00:42:42.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd329d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:42:42.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd329d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:42:42.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.296+0000 7fd329d9b640 1 -- 172.21.15.37:0/569385947 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd33409df90 con 0x7fd33409bf20 2024-04-26T00:42:42.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.300+0000 7fd329d9b640 1 -- 172.21.15.37:0/569385947 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd33409bf20 msgr2=0x7fd3340a14e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:42.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.300+0000 7fd329d9b640 1 --2- 172.21.15.37:0/569385947 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd33409bf20 0x7fd3340a14e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd31c009870 tx=0x7fd31c01c200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47826a3d68117e3a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:42.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.300+0000 7fd331d9b640 1 -- 172.21.15.37:0/569385947 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd31c0b0070 con 0x7fd33409bf20 2024-04-26T00:42:42.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.300+0000 7fd331d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:42.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.300+0000 7fd331d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:42:42.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.300+0000 7fd331d9b640 10 monclient: dump: 2024-04-26T00:42:42.309 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:42.309 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:42.309 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:42.310 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:42.310 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:42.310 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:42.310 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:42.310 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:42.310 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:42.310 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:42.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.300+0000 7fd331d9b640 10 monclient: _finish_auth 0 2024-04-26T00:42:42.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.300+0000 7fd331d9b640 10 monclient: _check_auth_tickets 2024-04-26T00:42:42.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.300+0000 7fd331d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:42:42.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.300+0000 7fd331d9b640 1 -- 172.21.15.37:0/569385947 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3340a1c30 con 0x7fd33409bf20 2024-04-26T00:42:42.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.300+0000 7fd338e4f640 5 monclient: authenticate success, global_id 254399 2024-04-26T00:42:42.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.300+0000 7fd338e4f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:42:42.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.300+0000 7fd331d9b640 10 monclient: handle_config config(0 keys) 2024-04-26T00:42:42.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.300+0000 7fd338e4f640 10 monclient: get_monmap_and_config success 2024-04-26T00:42:42.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.300+0000 7fd331d9b640 1 -- 172.21.15.37:0/569385947 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd31c0ab820 con 0x7fd33409bf20 2024-04-26T00:42:42.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.300+0000 7fd338e4f640 4 set_mon_vals no callback set 2024-04-26T00:42:42.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.300+0000 7fd331d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:42.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.300+0000 7fd331d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:42:42.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.300+0000 7fd331d9b640 10 monclient: dump: 2024-04-26T00:42:42.311 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:42.311 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:42.311 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:42.311 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:42.311 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:42.312 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:42.312 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:42.312 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:42.312 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:42.312 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:42.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.300+0000 7fd338e4f640 10 monclient: shutdown 2024-04-26T00:42:42.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.300+0000 7fd338e4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:42:42.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.300+0000 7fd338e4f640 1 -- 172.21.15.37:0/569385947 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd33409bf20 msgr2=0x7fd3340a14e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:42.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.300+0000 7fd338e4f640 1 --2- 172.21.15.37:0/569385947 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd33409bf20 0x7fd3340a14e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd31c009870 tx=0x7fd31c01c200 comp rx=0 tx=0).stop 2024-04-26T00:42:42.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.300+0000 7fd338e4f640 1 -- 172.21.15.37:0/569385947 shutdown_connections 2024-04-26T00:42:42.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.300+0000 7fd338e4f640 1 --2- 172.21.15.37:0/569385947 >> v2:172.21.15.45:3300/0 conn(0x7fd3340730c0 0x7fd33413c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:42.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.300+0000 7fd338e4f640 1 --2- 172.21.15.37:0/569385947 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd33409bf20 0x7fd3340a14e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:42.312 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.300+0000 7fd338e4f640 1 -- 172.21.15.37:0/569385947 >> 172.21.15.37:0/569385947 conn(0x7fd3340a6080 msgr2=0x7fd3340a1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:42:42.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.300+0000 7fd338e4f640 1 -- 172.21.15.37:0/569385947 shutdown_connections 2024-04-26T00:42:42.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.300+0000 7fd338e4f640 1 -- 172.21.15.37:0/569385947 wait complete. 2024-04-26T00:42:42.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.300+0000 7fd338e4f640 10 monclient: build_initial_monmap 2024-04-26T00:42:42.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.300+0000 7fd338e4f640 10 monclient: monmap: 2024-04-26T00:42:42.313 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:42:42.313 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:42:42.313 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:42:42.313 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:42:42.313 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:42:42.313 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:42:42.313 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:42:42.313 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:42:42.313 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:42:42.313 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:42.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd338e4f640 1 Processor -- start 2024-04-26T00:42:42.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd338e4f640 1 -- start start 2024-04-26T00:42:42.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd338e4f640 10 monclient: init 2024-04-26T00:42:42.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd338e4f640 10 monclient: _reopen_session rank -1 2024-04-26T00:42:42.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd338e4f640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:42:42.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd338e4f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd3340730c0 0x7fd3340993c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:42.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd338e4f640 10 monclient(hunting): picked mon.noname-a con 0x7fd3340730c0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:42:42.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd338e4f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd33409b5d0 0x7fd334099900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:42.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd338e4f640 10 monclient(hunting): picked mon.noname-c con 0x7fd33409b5d0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:42:42.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd338e4f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd33409bf20 0x7fd334099e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:42.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd338e4f640 10 monclient(hunting): picked mon.noname-b con 0x7fd33409bf20 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:42:42.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd338e4f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:42.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd338e4f640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd33413f6f0 con 0x7fd3340730c0 2024-04-26T00:42:42.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd338e4f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:42.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd338e4f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd33413f570 con 0x7fd33409bf20 2024-04-26T00:42:42.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd338e4f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:42:42.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd338e4f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd33413f870 con 0x7fd33409b5d0 2024-04-26T00:42:42.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd338e4f640 10 monclient(hunting): _renew_subs 2024-04-26T00:42:42.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd338e4f640 10 monclient(hunting): authenticate will time out at 2343.612796s 2024-04-26T00:42:42.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd33259c640 1 -- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd3340730c0 msgr2=0x7fd3340993c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:3300/0 2024-04-26T00:42:42.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd33259c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd3340730c0 0x7fd3340993c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:42:42.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd329d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd33409b5d0 0x7fd334099900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:42.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd329d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd33409b5d0 0x7fd334099900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:48542/0 (socket says 172.21.15.37:48542) 2024-04-26T00:42:42.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd329d9b640 1 -- 172.21.15.37:0/3060268286 learned_addr learned my addr 172.21.15.37:0/3060268286 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:42:42.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd329d9b640 10 monclient(hunting): get_auth_request con 0x7fd33409b5d0 auth_method 0 2024-04-26T00:42:42.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd329d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:42:42.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd329d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:42:42.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd329d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:42:42.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd329d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:42:42.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd329d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:42:42.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd329d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:42:42.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd332d9d640 1 --2- 172.21.15.37:0/3060268286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd33409bf20 0x7fd334099e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:42.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd329d9b640 10 monclient(hunting): handle_auth_done global_id 254402 payload 824 2024-04-26T00:42:42.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd329d9b640 1 -- 172.21.15.37:0/3060268286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd33409bf20 msgr2=0x7fd334099e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:42.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd329d9b640 1 --2- 172.21.15.37:0/3060268286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd33409bf20 0x7fd334099e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:42.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd329d9b640 1 -- 172.21.15.37:0/3060268286 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd3340730c0 msgr2=0x7fd3340993c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:42:42.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd329d9b640 1 --2- 172.21.15.37:0/3060268286 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd3340730c0 0x7fd3340993c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:42.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd329d9b640 10 monclient: _finish_hunting 0 2024-04-26T00:42:42.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd329d9b640 1 monclient: found mon.noname-c 2024-04-26T00:42:42.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd329d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:42:42.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd329d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:42:42.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd329d9b640 1 -- 172.21.15.37:0/3060268286 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3340925f0 con 0x7fd33409b5d0 2024-04-26T00:42:42.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd332d9d640 10 monclient: get_auth_request con 0x7fd33409bf20 auth_method 0 2024-04-26T00:42:42.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd332d9d640 1 --2- 172.21.15.37:0/3060268286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd33409bf20 0x7fd334099e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:42:42.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd329d9b640 1 -- 172.21.15.37:0/3060268286 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd33409b5d0 msgr2=0x7fd334099900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:42:42.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd329d9b640 1 --2- 172.21.15.37:0/3060268286 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd33409b5d0 0x7fd334099900 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd31c019e30 tx=0x7fd31c018fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:42.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd329d9b640 1 --2- 172.21.15.37:0/3060268286 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd33409b5d0 0x7fd334099900 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd31c019e30 tx=0x7fd31c018fa0 comp rx=0 tx=0).queue_received will delay after 1.71409e+09 on 0x7fd31c0b1040 config(0 keys) 2024-04-26T00:42:42.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd32b7fe640 1 -- 172.21.15.37:0/3060268286 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd31c0ba070 con 0x7fd33409b5d0 2024-04-26T00:42:42.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd32b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:42.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd32b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:42:42.319 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd32b7fe640 10 monclient: dump: 2024-04-26T00:42:42.319 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:42.319 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:42.319 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:42.319 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:42.319 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:42.320 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:42.320 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:42.320 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:42.320 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:42.320 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:42.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd32b7fe640 10 monclient: _finish_auth 0 2024-04-26T00:42:42.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd32b7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:42:42.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd32b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:42:42.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd32b7fe640 1 -- 172.21.15.37:0/3060268286 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd31c0b1040 con 0x7fd33409b5d0 2024-04-26T00:42:42.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd32b7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:42:42.320 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd338e4f640 5 monclient: authenticate success, global_id 254402 2024-04-26T00:42:42.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd338e4f640 10 monclient: _renew_subs 2024-04-26T00:42:42.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd338e4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:42.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd338e4f640 1 -- 172.21.15.37:0/3060268286 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd334098cd0 con 0x7fd33409b5d0 2024-04-26T00:42:42.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd33159a640 4 set_mon_vals no callback set 2024-04-26T00:42:42.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd338e4f640 10 monclient: _renew_subs 2024-04-26T00:42:42.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd338e4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:42.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.304+0000 7fd338e4f640 1 -- 172.21.15.37:0/3060268286 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd334093a10 con 0x7fd33409b5d0 2024-04-26T00:42:42.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.308+0000 7fd32b7fe640 1 -- 172.21.15.37:0/3060268286 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd31c0b0070 con 0x7fd33409b5d0 2024-04-26T00:42:42.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.308+0000 7fd32b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:42:42.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.308+0000 7fd32b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:42:42.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.308+0000 7fd32b7fe640 10 monclient: dump: 2024-04-26T00:42:42.321 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:42:42.322 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:42:42.322 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:42.322 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:42:42.322 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:42:42.322 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:42:42.322 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:42:42.322 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:42:42.322 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:42:42.322 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:42:42.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.308+0000 7fd32b7fe640 1 -- 172.21.15.37:0/3060268286 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd31c0bd070 con 0x7fd33409b5d0 2024-04-26T00:42:42.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.308+0000 7fd338e4f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:42.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.308+0000 7fd338e4f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:42.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.308+0000 7fd338e4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:42.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.308+0000 7fd338e4f640 1 -- 172.21.15.37:0/3060268286 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd33404f580 con 0x7fd33409b5d0 2024-04-26T00:42:42.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.308+0000 7fd32b7fe640 1 --2- 172.21.15.37:0/3060268286 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd314035e30 0x7fd3140382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:42:42.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.308+0000 7fd33259c640 1 --2- 172.21.15.37:0/3060268286 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd314035e30 0x7fd3140382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:42:42.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.308+0000 7fd33259c640 10 monclient: get_auth_request con 0x7fd314035e30 auth_method 0 2024-04-26T00:42:42.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.312+0000 7fd33259c640 1 --2- 172.21.15.37:0/3060268286 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd314035e30 0x7fd3140382f0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fd324006fd0 tx=0x7fd324006d40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:42:42.323 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.312+0000 7fd32b7fe640 1 -- 172.21.15.37:0/3060268286 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(898..898 src has 822..898) ==== 4019+0+0 (secure 0 0 0) 0x7fd31c0e2150 con 0x7fd33409b5d0 2024-04-26T00:42:42.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.932+0000 7fd32b7fe640 1 -- 172.21.15.37:0/3060268286 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fd33409e3b0 con 0x7fd33409b5d0 2024-04-26T00:42:42.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.932+0000 7fd32b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:42:42.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:42.932+0000 7fd32b7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:42:43.100 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:43.096+0000 7fd338e4f640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}] 2024-04-26T00:42:43.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:43.096+0000 7fd338e4f640 10 monclient: _send_command 2 [{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}] 2024-04-26T00:42:43.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:43.096+0000 7fd338e4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:42:43.101 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:43.096+0000 7fd338e4f640 1 -- 172.21.15.37:0/3060268286 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd33406bcc0 con 0x7fd33409b5d0 2024-04-26T00:42:43.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:43.744+0000 7fd32b7fe640 1 -- 172.21.15.37:0/3060268286 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]=0 pool 'unique_pool_0' removed v899) ==== 175+0+0 (secure 0 0 0) 0x7fd31c0bce10 con 0x7fd33409b5d0 2024-04-26T00:42:43.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:43.744+0000 7fd32b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}] 2024-04-26T00:42:43.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:43.744+0000 7fd32b7fe640 10 monclient: _finish_command 2 = system:0 pool 'unique_pool_0' removed 2024-04-26T00:42:43.748 INFO:teuthology.orchestra.run.smithi037.stderr:pool 'unique_pool_0' removed 2024-04-26T00:42:43.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:43.748+0000 7fd338e4f640 1 -- 172.21.15.37:0/3060268286 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd314035e30 msgr2=0x7fd3140382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:43.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:43.748+0000 7fd338e4f640 1 --2- 172.21.15.37:0/3060268286 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd314035e30 0x7fd3140382f0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fd324006fd0 tx=0x7fd324006d40 comp rx=0 tx=0).stop 2024-04-26T00:42:43.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:43.748+0000 7fd338e4f640 10 monclient: shutdown 2024-04-26T00:42:43.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:43.748+0000 7fd338e4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:42:43.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:43.748+0000 7fd338e4f640 1 -- 172.21.15.37:0/3060268286 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd33409b5d0 msgr2=0x7fd334099900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:42:43.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:43.748+0000 7fd338e4f640 1 --2- 172.21.15.37:0/3060268286 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd33409b5d0 0x7fd334099900 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd31c019e30 tx=0x7fd31c018fa0 comp rx=0 tx=0).stop 2024-04-26T00:42:43.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:43.748+0000 7fd338e4f640 1 -- 172.21.15.37:0/3060268286 shutdown_connections 2024-04-26T00:42:43.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:43.748+0000 7fd338e4f640 1 --2- 172.21.15.37:0/3060268286 >> v2:172.21.15.45:6812/2379918623 conn(0x7fd314035e30 0x7fd3140382f0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:43.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:43.748+0000 7fd338e4f640 1 --2- 172.21.15.37:0/3060268286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd33409bf20 0x7fd334099e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:43.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:43.748+0000 7fd338e4f640 1 --2- 172.21.15.37:0/3060268286 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd33409b5d0 0x7fd334099900 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:43.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:43.748+0000 7fd338e4f640 1 --2- 172.21.15.37:0/3060268286 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd3340730c0 0x7fd3340993c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:42:43.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:43.748+0000 7fd338e4f640 1 -- 172.21.15.37:0/3060268286 >> 172.21.15.37:0/3060268286 conn(0x7fd3340a6080 msgr2=0x7fd334070c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:42:43.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:43.748+0000 7fd338e4f640 1 -- 172.21.15.37:0/3060268286 shutdown_connections 2024-04-26T00:42:43.754 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:42:43.748+0000 7fd338e4f640 1 -- 172.21.15.37:0/3060268286 wait complete. 2024-04-26T00:42:43.778 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-04-26T00:42:43.789 INFO:tasks.mon_thrash:joining mon_thrasher 2024-04-26T00:43:00.295 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-04-26T00:43:00.295 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-04-26T00:43:00.296 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-26T00:43:00.296 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-26T00:43:00.299 INFO:tasks.ceph.mon.a:Started 2024-04-26T00:43:00.299 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:43:00.299 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:43:00.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f6660364640 10 monclient: get_monmap_and_config 2024-04-26T00:43:00.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f6660364640 10 monclient: build_initial_monmap 2024-04-26T00:43:00.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f6660364640 10 monclient: monmap: 2024-04-26T00:43:00.424 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:00.424 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:00.424 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:43:00.425331+0000 2024-04-26T00:43:00.425 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:43:00.425331+0000 2024-04-26T00:43:00.425 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:00.425 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:00.425 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:43:00.425 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:43:00.425 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:43:00.425 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:43:00.425 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:43:00.425 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:00.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f6660364640 1 Processor -- start 2024-04-26T00:43:00.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f6660364640 1 -- start start 2024-04-26T00:43:00.425 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f6660364640 10 monclient: init 2024-04-26T00:43:00.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f6660364640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:00.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f6660364640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-04-26T00:43:00.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f6660364640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f66580a0260 0x7f66580a0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:00.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f6660364640 10 monclient(hunting): picked mon.noname-a con 0x7f66580a0260 addr v2:172.21.15.37:3300/0 2024-04-26T00:43:00.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f6660364640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6658134b30 addr v1:172.21.15.37:6789/0 2024-04-26T00:43:00.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f6660364640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f6658135660 0x7f665813ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:00.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f6660364640 10 monclient(hunting): picked mon.noname-c con 0x7f6658135660 addr v2:172.21.15.45:3300/0 2024-04-26T00:43:00.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f6660364640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66580a1130 con 0x7f6658134b30 2024-04-26T00:43:00.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f6660364640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:00.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f6660364640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f665809ec10 con 0x7f66580a0260 2024-04-26T00:43:00.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f6660364640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:00.428 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f6660364640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f665809f0f0 con 0x7f6658135660 2024-04-26T00:43:00.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f6660364640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:00.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f6660364640 10 monclient(hunting): authenticate will time out at 2361.730592s 2024-04-26T00:43:00.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f665e100640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f66580a0260 0x7f66580a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:00.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f665e100640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f66580a0260 0x7f66580a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36510/0 (socket says 172.21.15.37:36510) 2024-04-26T00:43:00.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f665e100640 1 -- 172.21.15.37:0/476655932 learned_addr learned my addr 172.21.15.37:0/476655932 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:00.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f665e100640 10 monclient(hunting): get_auth_request con 0x7f66580a0260 auth_method 0 2024-04-26T00:43:00.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f665e100640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:00.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f665e100640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:00.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f665e100640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:00.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f665d8ff640 1 --1- 172.21.15.37:0/476655932 >> v1:172.21.15.37:6789/0 conn(0x7f6658134b30 0x7f6658134f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:42036/0 (socket says 172.21.15.37:42036) 2024-04-26T00:43:00.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f665e100640 1 -- 172.21.15.37:0/476655932 >> v2:172.21.15.37:3300/0 conn(0x7f66580a0260 msgr2=0x7f66580a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:43:00.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f665e100640 1 -- 172.21.15.37:0/476655932 >> v2:172.21.15.37:3300/0 conn(0x7f66580a0260 msgr2=0x7f66580a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:43:00.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f665e100640 1 --2- 172.21.15.37:0/476655932 >> v2:172.21.15.37:3300/0 conn(0x7f66580a0260 0x7f66580a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:43:00.429 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f665e100640 1 --2- 172.21.15.37:0/476655932 >> v2:172.21.15.37:3300/0 conn(0x7f66580a0260 0x7f66580a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:43:00.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f665e901640 1 --2- 172.21.15.37:0/476655932 >> v2:172.21.15.45:3300/0 conn(0x7f6658135660 0x7f665813ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:00.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f665e901640 10 monclient(hunting): get_auth_request con 0x7f6658135660 auth_method 0 2024-04-26T00:43:00.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f665e901640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:00.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f665e901640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:00.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f665e901640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:00.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f665e901640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:00.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f665e901640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:00.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f665e901640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:00.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f665e901640 10 monclient(hunting): handle_auth_done global_id 274171 payload 274 2024-04-26T00:43:00.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f665e901640 1 -- 172.21.15.37:0/476655932 >> v2:172.21.15.37:3300/0 conn(0x7f66580a0260 msgr2=0x7f66580a0640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:43:00.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f665e901640 1 --2- 172.21.15.37:0/476655932 >> v2:172.21.15.37:3300/0 conn(0x7f66580a0260 0x7f66580a0640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:00.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f665e901640 1 -- 172.21.15.37:0/476655932 >> v1:172.21.15.37:6789/0 conn(0x7f6658134b30 legacy=0x7f6658134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:00.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f665e901640 10 monclient: _finish_hunting 0 2024-04-26T00:43:00.430 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f665e901640 1 monclient: found mon.noname-c 2024-04-26T00:43:00.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f665e901640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:00.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f665e901640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:43:00.431 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.423+0000 7f665e901640 1 -- 172.21.15.37:0/476655932 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f665806cff0 con 0x7f6658135660 2024-04-26T00:43:00.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.427+0000 7f665e901640 1 -- 172.21.15.37:0/476655932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6658135660 msgr2=0x7f665813ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:00.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665e901640 1 --2- 172.21.15.37:0/476655932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6658135660 0x7f665813ce70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f66580a1a60 tx=0x7f665401c120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f88f002c067061d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:00.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665d0fe640 1 -- 172.21.15.37:0/476655932 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66540b1070 con 0x7f6658135660 2024-04-26T00:43:00.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665d0fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:00.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665d0fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:43:00.434 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665d0fe640 10 monclient: dump: 2024-04-26T00:43:00.434 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:00.434 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:00.434 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:00.435 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:00.435 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:00.435 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:00.435 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:00.435 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:00.435 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:00.435 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:00.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665d0fe640 10 monclient: _finish_auth 0 2024-04-26T00:43:00.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665d0fe640 10 monclient: _check_auth_tickets 2024-04-26T00:43:00.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665d0fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:00.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665d0fe640 1 -- 172.21.15.37:0/476655932 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66540ac440 con 0x7f6658135660 2024-04-26T00:43:00.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665d0fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:00.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665d0fe640 1 -- 172.21.15.37:0/476655932 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66540ac810 con 0x7f6658135660 2024-04-26T00:43:00.435 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665d0fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:00.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665d0fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:43:00.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665d0fe640 10 monclient: dump: 2024-04-26T00:43:00.436 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:00.436 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:00.436 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:00.436 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:00.436 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:00.436 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:00.436 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:00.436 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:00.436 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:00.436 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:00.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f6660364640 5 monclient: authenticate success, global_id 274171 2024-04-26T00:43:00.436 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f6660364640 10 monclient: get_monmap_and_config success 2024-04-26T00:43:00.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f6660364640 4 set_mon_vals no callback set 2024-04-26T00:43:00.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f6660364640 10 monclient: shutdown 2024-04-26T00:43:00.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f6660364640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:00.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f6660364640 1 -- 172.21.15.37:0/476655932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6658135660 msgr2=0x7f665813ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:00.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f6660364640 1 --2- 172.21.15.37:0/476655932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6658135660 0x7f665813ce70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f66580a1a60 tx=0x7f665401c120 comp rx=0 tx=0).stop 2024-04-26T00:43:00.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f6660364640 1 -- 172.21.15.37:0/476655932 shutdown_connections 2024-04-26T00:43:00.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f6660364640 1 --2- 172.21.15.37:0/476655932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6658135660 0x7f665813ce70 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:00.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f6660364640 1 --2- 172.21.15.37:0/476655932 >> v2:172.21.15.37:3300/0 conn(0x7f66580a0260 0x7f66580a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:00.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f6660364640 1 -- 172.21.15.37:0/476655932 >> 172.21.15.37:0/476655932 conn(0x7f66580a1ad0 msgr2=0x7f66580a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:00.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f6660364640 1 -- 172.21.15.37:0/476655932 shutdown_connections 2024-04-26T00:43:00.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f6660364640 1 -- 172.21.15.37:0/476655932 wait complete. 2024-04-26T00:43:00.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f6660364640 10 monclient: build_initial_monmap 2024-04-26T00:43:00.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f6660364640 10 monclient: monmap: 2024-04-26T00:43:00.437 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:00.437 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:43:00.438 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:43:00.438 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:43:00.438 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:00.438 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:43:00.438 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:43:00.438 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:43:00.438 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:43:00.438 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:00.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f6660364640 1 Processor -- start 2024-04-26T00:43:00.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f6660364640 1 -- start start 2024-04-26T00:43:00.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f6660364640 10 monclient: init 2024-04-26T00:43:00.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f6660364640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:00.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f6660364640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:43:00.438 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f6660364640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66580a0260 0x7f66581cccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:00.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f6660364640 10 monclient(hunting): picked mon.noname-a con 0x7f66580a0260 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:43:00.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f6660364640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6658134b30 0x7f6658131ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:00.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f6660364640 10 monclient(hunting): picked mon.noname-c con 0x7f6658134b30 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:00.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f6660364640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6658135660 0x7f66581399a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:00.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f6660364640 10 monclient(hunting): picked mon.noname-b con 0x7f6658135660 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:43:00.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f6660364640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:00.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f6660364640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f66580a0010 con 0x7f66580a0260 2024-04-26T00:43:00.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f6660364640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:00.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f6660364640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f665809fe90 con 0x7f6658135660 2024-04-26T00:43:00.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f6660364640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:00.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f6660364640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f665813fbd0 con 0x7f6658134b30 2024-04-26T00:43:00.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f6660364640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:00.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f6660364640 10 monclient(hunting): authenticate will time out at 2361.738558s 2024-04-26T00:43:00.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665e100640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66580a0260 0x7f66581cccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:00.439 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665d8ff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6658134b30 0x7f6658131ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:00.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665d8ff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6658134b30 0x7f6658131ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:40586/0 (socket says 172.21.15.37:40586) 2024-04-26T00:43:00.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665d8ff640 1 -- 172.21.15.37:0/1051109667 learned_addr learned my addr 172.21.15.37:0/1051109667 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:00.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665d8ff640 10 monclient(hunting): get_auth_request con 0x7f6658134b30 auth_method 0 2024-04-26T00:43:00.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665d8ff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:00.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665d8ff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:00.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665d8ff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:00.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665e100640 10 monclient(hunting): get_auth_request con 0x7f66580a0260 auth_method 0 2024-04-26T00:43:00.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665e100640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:00.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665e100640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:00.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665e100640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:00.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665d8ff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:00.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665d8ff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:00.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665e100640 1 -- 172.21.15.37:0/1051109667 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66580a0260 msgr2=0x7f66581cccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:43:00.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665d8ff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:00.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665e100640 1 -- 172.21.15.37:0/1051109667 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66580a0260 msgr2=0x7f66581cccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:43:00.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665e100640 1 --2- 172.21.15.37:0/1051109667 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66580a0260 0x7f66581cccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:43:00.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665e100640 1 --2- 172.21.15.37:0/1051109667 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66580a0260 0x7f66581cccc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:43:00.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665e901640 1 --2- 172.21.15.37:0/1051109667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6658135660 0x7f66581399a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:00.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665e901640 10 monclient(hunting): get_auth_request con 0x7f6658135660 auth_method 0 2024-04-26T00:43:00.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665e901640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:00.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665e901640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:00.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665e901640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:00.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665d8ff640 10 monclient(hunting): handle_auth_done global_id 254405 payload 824 2024-04-26T00:43:00.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665d8ff640 1 -- 172.21.15.37:0/1051109667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6658135660 msgr2=0x7f66581399a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:00.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665d8ff640 1 --2- 172.21.15.37:0/1051109667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6658135660 0x7f66581399a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:00.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665d8ff640 1 -- 172.21.15.37:0/1051109667 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66580a0260 msgr2=0x7f66581cccc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:43:00.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665d8ff640 1 --2- 172.21.15.37:0/1051109667 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66580a0260 0x7f66581cccc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:00.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665d8ff640 10 monclient: _finish_hunting 0 2024-04-26T00:43:00.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665d8ff640 1 monclient: found mon.noname-c 2024-04-26T00:43:00.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665d8ff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:00.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665d8ff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:43:00.442 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.431+0000 7f665d8ff640 1 -- 172.21.15.37:0/1051109667 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f665811b7a0 con 0x7f6658134b30 2024-04-26T00:43:00.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.435+0000 7f665d8ff640 1 -- 172.21.15.37:0/1051109667 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6658134b30 msgr2=0x7f6658131ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:00.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.435+0000 7f665d8ff640 1 --2- 172.21.15.37:0/1051109667 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6658134b30 0x7f6658131ae0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f664800c490 tx=0x7f664800c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:00.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.435+0000 7f664effd640 1 -- 172.21.15.37:0/1051109667 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6648017070 con 0x7f6658134b30 2024-04-26T00:43:00.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.435+0000 7f664effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:00.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.435+0000 7f664effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:43:00.444 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.435+0000 7f664effd640 10 monclient: dump: 2024-04-26T00:43:00.445 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:00.445 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:00.445 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:00.445 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:00.445 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:00.445 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:00.445 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:00.445 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:00.445 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:00.445 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:00.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.435+0000 7f664effd640 10 monclient: _finish_auth 0 2024-04-26T00:43:00.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.435+0000 7f664effd640 10 monclient: _check_auth_tickets 2024-04-26T00:43:00.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.435+0000 7f664effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:00.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.435+0000 7f6660364640 5 monclient: authenticate success, global_id 254405 2024-04-26T00:43:00.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.435+0000 7f664effd640 1 -- 172.21.15.37:0/1051109667 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6648005820 con 0x7f6658134b30 2024-04-26T00:43:00.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.435+0000 7f664effd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:00.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.435+0000 7f6660364640 10 monclient: _renew_subs 2024-04-26T00:43:00.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.435+0000 7f664effd640 1 -- 172.21.15.37:0/1051109667 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6648005c10 con 0x7f6658134b30 2024-04-26T00:43:00.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.435+0000 7f6660364640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:00.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.435+0000 7f6660364640 1 -- 172.21.15.37:0/1051109667 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6658134770 con 0x7f6658134b30 2024-04-26T00:43:00.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.435+0000 7f665c8fd640 4 set_mon_vals no callback set 2024-04-26T00:43:00.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.435+0000 7f664effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:00.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.435+0000 7f664effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:43:00.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.435+0000 7f664effd640 10 monclient: dump: 2024-04-26T00:43:00.446 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:00.446 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:00.446 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:00.446 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:00.446 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:00.447 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:00.447 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:00.447 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:00.447 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:00.447 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:00.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.435+0000 7f6660364640 10 monclient: _renew_subs 2024-04-26T00:43:00.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.435+0000 7f6660364640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:00.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.435+0000 7f6660364640 1 -- 172.21.15.37:0/1051109667 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f665813c1a0 con 0x7f6658134b30 2024-04-26T00:43:00.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.435+0000 7f6660364640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:00.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.435+0000 7f6660364640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:00.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.435+0000 7f6660364640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:00.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.435+0000 7f6660364640 1 -- 172.21.15.37:0/1051109667 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6658066f40 con 0x7f6658134b30 2024-04-26T00:43:00.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.435+0000 7f664effd640 1 -- 172.21.15.37:0/1051109667 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f664801a070 con 0x7f6658134b30 2024-04-26T00:43:00.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.435+0000 7f664effd640 1 --2- 172.21.15.37:0/1051109667 >> v2:172.21.15.45:6812/2379918623 conn(0x7f663c035e80 0x7f663c038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:00.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.435+0000 7f664effd640 1 -- 172.21.15.37:0/1051109667 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(899..899 src has 822..899) ==== 3550+0+0 (secure 0 0 0) 0x7f6648047db0 con 0x7f6658134b30 2024-04-26T00:43:00.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.435+0000 7f665e100640 1 --2- 172.21.15.37:0/1051109667 >> v2:172.21.15.45:6812/2379918623 conn(0x7f663c035e80 0x7f663c038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:00.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.435+0000 7f665e100640 10 monclient: get_auth_request con 0x7f663c035e80 auth_method 0 2024-04-26T00:43:00.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.439+0000 7f665e100640 1 --2- 172.21.15.37:0/1051109667 >> v2:172.21.15.45:6812/2379918623 conn(0x7f663c035e80 0x7f663c038340 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f66500096d0 tx=0x7f6650009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:00.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.439+0000 7f664effd640 1 -- 172.21.15.37:0/1051109667 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6648010300 con 0x7f6658134b30 2024-04-26T00:43:00.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.439+0000 7f664effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:00.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.439+0000 7f664effd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:43:00.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.667+0000 7f6660364640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:43:00.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.667+0000 7f6660364640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:43:00.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.667+0000 7f6660364640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:00.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.667+0000 7f6660364640 1 -- 172.21.15.37:0/1051109667 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f665813b640 con 0x7f6658134b30 2024-04-26T00:43:00.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.667+0000 7f664effd640 1 -- 172.21.15.37:0/1051109667 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7f664801b460 con 0x7f6658134b30 2024-04-26T00:43:00.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.667+0000 7f664effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:43:00.670 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.667+0000 7f664effd640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:43:00.670 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":164,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:43:00.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.671+0000 7f6660364640 1 -- 172.21.15.37:0/1051109667 >> v2:172.21.15.45:6812/2379918623 conn(0x7f663c035e80 msgr2=0x7f663c038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:00.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.671+0000 7f6660364640 1 --2- 172.21.15.37:0/1051109667 >> v2:172.21.15.45:6812/2379918623 conn(0x7f663c035e80 0x7f663c038340 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f66500096d0 tx=0x7f6650009040 comp rx=0 tx=0).stop 2024-04-26T00:43:00.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.671+0000 7f6660364640 10 monclient: shutdown 2024-04-26T00:43:00.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.671+0000 7f6660364640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:00.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.671+0000 7f6660364640 1 -- 172.21.15.37:0/1051109667 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6658134b30 msgr2=0x7f6658131ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:00.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.671+0000 7f6660364640 1 --2- 172.21.15.37:0/1051109667 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6658134b30 0x7f6658131ae0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f664800c490 tx=0x7f664800c950 comp rx=0 tx=0).stop 2024-04-26T00:43:00.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.671+0000 7f6660364640 1 -- 172.21.15.37:0/1051109667 shutdown_connections 2024-04-26T00:43:00.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.671+0000 7f6660364640 1 --2- 172.21.15.37:0/1051109667 >> v2:172.21.15.45:6812/2379918623 conn(0x7f663c035e80 0x7f663c038340 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:00.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.671+0000 7f6660364640 1 --2- 172.21.15.37:0/1051109667 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6658135660 0x7f66581399a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:00.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.671+0000 7f6660364640 1 --2- 172.21.15.37:0/1051109667 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6658134b30 0x7f6658131ae0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:00.672 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.671+0000 7f6660364640 1 --2- 172.21.15.37:0/1051109667 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66580a0260 0x7f66581cccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:00.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.671+0000 7f6660364640 1 -- 172.21.15.37:0/1051109667 >> 172.21.15.37:0/1051109667 conn(0x7f66580a1ad0 msgr2=0x7f66580a3d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:00.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.671+0000 7f6660364640 1 -- 172.21.15.37:0/1051109667 shutdown_connections 2024-04-26T00:43:00.673 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:00.671+0000 7f6660364640 1 -- 172.21.15.37:0/1051109667 wait complete. 2024-04-26T00:43:03.689 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:43:03.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.819+0000 7ff190c49640 10 monclient: get_monmap_and_config 2024-04-26T00:43:03.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.819+0000 7ff190c49640 10 monclient: build_initial_monmap 2024-04-26T00:43:03.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.819+0000 7ff190c49640 10 monclient: monmap: 2024-04-26T00:43:03.822 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:03.822 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:03.822 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:43:03.822720+0000 2024-04-26T00:43:03.822 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:43:03.822720+0000 2024-04-26T00:43:03.822 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:03.822 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:03.822 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:43:03.822 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:43:03.823 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:43:03.823 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:43:03.823 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:43:03.823 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:03.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff190c49640 1 Processor -- start 2024-04-26T00:43:03.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff190c49640 1 -- start start 2024-04-26T00:43:03.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff190c49640 10 monclient: init 2024-04-26T00:43:03.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff190c49640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:03.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff190c49640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-04-26T00:43:03.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff190c49640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff18c0a0260 addr v1:172.21.15.45:6789/0 2024-04-26T00:43:03.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff190c49640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff18c0a0d50 0x7ff18c072e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:03.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff190c49640 10 monclient(hunting): picked mon.noname-b con 0x7ff18c0a0d50 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:03.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff190c49640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff18c1389e0 0x7ff18c13add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:03.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff190c49640 10 monclient(hunting): picked mon.noname-a con 0x7ff18c1389e0 addr v2:172.21.15.37:3300/0 2024-04-26T00:43:03.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff190c49640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff18c0a12e0 con 0x7ff18c0a0260 2024-04-26T00:43:03.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff190c49640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:03.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff190c49640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7ff18c06d050 con 0x7ff18c1389e0 2024-04-26T00:43:03.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff190c49640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:03.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff190c49640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff18c09dcb0 con 0x7ff18c0a0d50 2024-04-26T00:43:03.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff190c49640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:03.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff190c49640 10 monclient(hunting): authenticate will time out at 2365.128885s 2024-04-26T00:43:03.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff18ad9d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff18c1389e0 0x7ff18c13add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:03.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff181d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff18c0a0d50 0x7ff18c072e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:03.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff18ad9d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff18c1389e0 0x7ff18c13add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36524/0 (socket says 172.21.15.37:36524) 2024-04-26T00:43:03.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff18ad9d640 1 -- 172.21.15.37:0/1758635870 learned_addr learned my addr 172.21.15.37:0/1758635870 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:03.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff18ad9d640 10 monclient(hunting): get_auth_request con 0x7ff18c1389e0 auth_method 0 2024-04-26T00:43:03.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff18ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:03.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff18ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:03.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff18ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:03.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff181d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff18c0a0d50 0x7ff18c072e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:40622/0 (socket says 172.21.15.37:40622) 2024-04-26T00:43:03.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff181d9b640 10 monclient(hunting): get_auth_request con 0x7ff18c0a0d50 auth_method 0 2024-04-26T00:43:03.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff181d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:03.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff181d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:03.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff181d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:03.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff181d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:03.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff181d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:03.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff18ad9d640 1 -- 172.21.15.37:0/1758635870 >> v2:172.21.15.37:3300/0 conn(0x7ff18c1389e0 msgr2=0x7ff18c13add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T00:43:03.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff181d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:03.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff18ad9d640 1 -- 172.21.15.37:0/1758635870 >> v2:172.21.15.37:3300/0 conn(0x7ff18c1389e0 msgr2=0x7ff18c13add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:43:03.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff18ad9d640 1 --2- 172.21.15.37:0/1758635870 >> v2:172.21.15.37:3300/0 conn(0x7ff18c1389e0 0x7ff18c13add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:43:03.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff18ad9d640 1 --2- 172.21.15.37:0/1758635870 >> v2:172.21.15.37:3300/0 conn(0x7ff18c1389e0 0x7ff18c13add0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:43:03.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff181d9b640 10 monclient(hunting): handle_auth_done global_id 254408 payload 274 2024-04-26T00:43:03.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff181d9b640 1 -- 172.21.15.37:0/1758635870 >> v2:172.21.15.37:3300/0 conn(0x7ff18c1389e0 msgr2=0x7ff18c13add0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:43:03.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff181d9b640 1 --2- 172.21.15.37:0/1758635870 >> v2:172.21.15.37:3300/0 conn(0x7ff18c1389e0 0x7ff18c13add0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:03.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff181d9b640 1 -- 172.21.15.37:0/1758635870 >> v1:172.21.15.45:6789/0 conn(0x7ff18c0a0260 legacy=0x7ff18c0a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:03.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff181d9b640 10 monclient: _finish_hunting 0 2024-04-26T00:43:03.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff181d9b640 1 monclient: found mon.noname-b 2024-04-26T00:43:03.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff181d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:03.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff181d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:43:03.828 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.823+0000 7ff181d9b640 1 -- 172.21.15.37:0/1758635870 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff18c0a16f0 con 0x7ff18c0a0d50 2024-04-26T00:43:03.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.827+0000 7ff181d9b640 1 -- 172.21.15.37:0/1758635870 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff18c0a0d50 msgr2=0x7ff18c072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:03.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.827+0000 7ff181d9b640 1 --2- 172.21.15.37:0/1758635870 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff18c0a0d50 0x7ff18c072e10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff17400ba50 tx=0x7ff17401e860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6ce43f20467d65e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:03.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.827+0000 7ff189d9b640 1 -- 172.21.15.37:0/1758635870 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1740b0020 con 0x7ff18c0a0d50 2024-04-26T00:43:03.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.827+0000 7ff189d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:03.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.827+0000 7ff189d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:43:03.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.827+0000 7ff189d9b640 10 monclient: dump: 2024-04-26T00:43:03.835 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:03.835 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:03.836 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:03.836 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:03.836 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:03.836 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:03.836 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:03.836 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:03.836 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:03.836 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:03.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.827+0000 7ff189d9b640 10 monclient: _finish_auth 0 2024-04-26T00:43:03.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.827+0000 7ff189d9b640 10 monclient: _check_auth_tickets 2024-04-26T00:43:03.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.827+0000 7ff189d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:03.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.827+0000 7ff189d9b640 1 -- 172.21.15.37:0/1758635870 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1740a8680 con 0x7ff18c0a0d50 2024-04-26T00:43:03.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.827+0000 7ff189d9b640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:03.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.827+0000 7ff189d9b640 1 -- 172.21.15.37:0/1758635870 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1740a8a50 con 0x7ff18c0a0d50 2024-04-26T00:43:03.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.827+0000 7ff190c49640 5 monclient: authenticate success, global_id 254408 2024-04-26T00:43:03.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.827+0000 7ff190c49640 10 monclient: get_monmap_and_config success 2024-04-26T00:43:03.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.827+0000 7ff190c49640 4 set_mon_vals no callback set 2024-04-26T00:43:03.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.827+0000 7ff189d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:03.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.827+0000 7ff189d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:43:03.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.827+0000 7ff189d9b640 10 monclient: dump: 2024-04-26T00:43:03.837 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:03.837 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:03.837 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:03.837 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:03.837 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:03.837 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:03.837 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:03.838 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:03.838 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:03.838 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:03.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.827+0000 7ff190c49640 10 monclient: shutdown 2024-04-26T00:43:03.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.827+0000 7ff190c49640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:03.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.827+0000 7ff190c49640 1 -- 172.21.15.37:0/1758635870 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff18c0a0d50 msgr2=0x7ff18c072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:03.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.827+0000 7ff190c49640 1 --2- 172.21.15.37:0/1758635870 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff18c0a0d50 0x7ff18c072e10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff17400ba50 tx=0x7ff17401e860 comp rx=0 tx=0).stop 2024-04-26T00:43:03.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.827+0000 7ff190c49640 1 -- 172.21.15.37:0/1758635870 shutdown_connections 2024-04-26T00:43:03.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.827+0000 7ff190c49640 1 --2- 172.21.15.37:0/1758635870 >> v2:172.21.15.37:3300/0 conn(0x7ff18c1389e0 0x7ff18c13add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:03.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.827+0000 7ff190c49640 1 --2- 172.21.15.37:0/1758635870 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff18c0a0d50 0x7ff18c072e10 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:03.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.827+0000 7ff190c49640 1 -- 172.21.15.37:0/1758635870 >> 172.21.15.37:0/1758635870 conn(0x7ff18c0a1ad0 msgr2=0x7ff18c0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:03.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.827+0000 7ff190c49640 1 -- 172.21.15.37:0/1758635870 shutdown_connections 2024-04-26T00:43:03.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.827+0000 7ff190c49640 1 -- 172.21.15.37:0/1758635870 wait complete. 2024-04-26T00:43:03.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.827+0000 7ff190c49640 10 monclient: build_initial_monmap 2024-04-26T00:43:03.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.827+0000 7ff190c49640 10 monclient: monmap: 2024-04-26T00:43:03.839 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:03.839 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:43:03.839 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:43:03.839 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:43:03.839 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:03.839 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:43:03.839 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:43:03.839 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:43:03.839 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:43:03.839 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:03.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.827+0000 7ff190c49640 1 Processor -- start 2024-04-26T00:43:03.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.827+0000 7ff190c49640 1 -- start start 2024-04-26T00:43:03.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.827+0000 7ff190c49640 10 monclient: init 2024-04-26T00:43:03.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff190c49640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:03.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff190c49640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:43:03.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff190c49640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff18c0a0260 0x7ff18c1cfd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:03.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff190c49640 10 monclient(hunting): picked mon.noname-c con 0x7ff18c0a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:03.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff190c49640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff18c1389e0 0x7ff18c1d0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:03.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff190c49640 10 monclient(hunting): picked mon.noname-b con 0x7ff18c1389e0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:43:03.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff190c49640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff18c11fae0 0x7ff18c11ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:03.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff190c49640 10 monclient(hunting): picked mon.noname-a con 0x7ff18c11fae0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:43:03.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff190c49640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:03.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff190c49640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff18c13d530 con 0x7ff18c11fae0 2024-04-26T00:43:03.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff190c49640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:03.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff190c49640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff18c09be60 con 0x7ff18c1389e0 2024-04-26T00:43:03.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff190c49640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:03.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff190c49640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff18c13d6b0 con 0x7ff18c0a0260 2024-04-26T00:43:03.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff190c49640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:03.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff190c49640 10 monclient(hunting): authenticate will time out at 2365.136164s 2024-04-26T00:43:03.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff18a59c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff18c0a0260 0x7ff18c1cfd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:03.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff18ad9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff18c11fae0 0x7ff18c11ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:03.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff18a59c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff18c0a0260 0x7ff18c1cfd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:40630/0 (socket says 172.21.15.37:40630) 2024-04-26T00:43:03.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff18a59c640 1 -- 172.21.15.37:0/1806061101 learned_addr learned my addr 172.21.15.37:0/1806061101 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:03.841 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff18a59c640 10 monclient(hunting): get_auth_request con 0x7ff18c0a0260 auth_method 0 2024-04-26T00:43:03.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff18a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:03.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff18a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:03.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff18a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:03.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff18ad9d640 10 monclient(hunting): get_auth_request con 0x7ff18c11fae0 auth_method 0 2024-04-26T00:43:03.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff18ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:03.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff18ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:03.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff18ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:03.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff18a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:03.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff18a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:03.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff18a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:03.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff18ad9d640 1 -- 172.21.15.37:0/1806061101 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff18c11fae0 msgr2=0x7ff18c11ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:43:03.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff18ad9d640 1 -- 172.21.15.37:0/1806061101 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff18c11fae0 msgr2=0x7ff18c11ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:43:03.842 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff18ad9d640 1 --2- 172.21.15.37:0/1806061101 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff18c11fae0 0x7ff18c11ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:43:03.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff18ad9d640 1 --2- 172.21.15.37:0/1806061101 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff18c11fae0 0x7ff18c11ff70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:43:03.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff181d9b640 1 --2- 172.21.15.37:0/1806061101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff18c1389e0 0x7ff18c1d0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:03.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff181d9b640 10 monclient(hunting): get_auth_request con 0x7ff18c1389e0 auth_method 0 2024-04-26T00:43:03.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff181d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:03.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff181d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:03.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff181d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:03.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff18a59c640 10 monclient(hunting): handle_auth_done global_id 254411 payload 824 2024-04-26T00:43:03.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff18a59c640 1 -- 172.21.15.37:0/1806061101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff18c1389e0 msgr2=0x7ff18c1d0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:03.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff18a59c640 1 --2- 172.21.15.37:0/1806061101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff18c1389e0 0x7ff18c1d0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:03.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff18a59c640 1 -- 172.21.15.37:0/1806061101 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff18c11fae0 msgr2=0x7ff18c11ff70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:43:03.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff18a59c640 1 --2- 172.21.15.37:0/1806061101 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff18c11fae0 0x7ff18c11ff70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:03.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff18a59c640 10 monclient: _finish_hunting 0 2024-04-26T00:43:03.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff18a59c640 1 monclient: found mon.noname-c 2024-04-26T00:43:03.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff18a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:03.843 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff18a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:43:03.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff18a59c640 1 -- 172.21.15.37:0/1806061101 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff18c09b130 con 0x7ff18c0a0260 2024-04-26T00:43:03.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff18a59c640 1 -- 172.21.15.37:0/1806061101 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff18c0a0260 msgr2=0x7ff18c1cfd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:03.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff18a59c640 1 --2- 172.21.15.37:0/1806061101 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff18c0a0260 0x7ff18c1cfd40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff17c00c480 tx=0x7ff17c00c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:03.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff1837fe640 1 -- 172.21.15.37:0/1806061101 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff17c012a40 con 0x7ff18c0a0260 2024-04-26T00:43:03.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff1837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:03.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff1837fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:43:03.844 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff1837fe640 10 monclient: dump: 2024-04-26T00:43:03.844 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:03.844 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:03.844 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:03.844 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:03.844 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:03.844 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:03.844 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:03.844 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:03.845 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:03.845 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:03.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff1837fe640 10 monclient: _finish_auth 0 2024-04-26T00:43:03.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff1837fe640 10 monclient: _check_auth_tickets 2024-04-26T00:43:03.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff1837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:03.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff1837fe640 1 -- 172.21.15.37:0/1806061101 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff17c017070 con 0x7ff18c0a0260 2024-04-26T00:43:03.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff1837fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:03.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff1837fe640 1 -- 172.21.15.37:0/1806061101 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff17c012e40 con 0x7ff18c0a0260 2024-04-26T00:43:03.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff190c49640 5 monclient: authenticate success, global_id 254411 2024-04-26T00:43:03.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff190c49640 10 monclient: _renew_subs 2024-04-26T00:43:03.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff190c49640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:03.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff190c49640 1 -- 172.21.15.37:0/1806061101 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff18c121780 con 0x7ff18c0a0260 2024-04-26T00:43:03.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff18959a640 4 set_mon_vals no callback set 2024-04-26T00:43:03.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff1837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:03.845 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff1837fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:43:03.846 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff1837fe640 10 monclient: dump: 2024-04-26T00:43:03.846 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:03.846 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:03.846 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:03.846 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:03.846 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:03.846 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:03.846 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:03.846 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:03.846 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:03.847 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:03.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff190c49640 10 monclient: _renew_subs 2024-04-26T00:43:03.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff190c49640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:03.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.831+0000 7ff190c49640 1 -- 172.21.15.37:0/1806061101 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff18c13b080 con 0x7ff18c0a0260 2024-04-26T00:43:03.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.835+0000 7ff1837fe640 1 -- 172.21.15.37:0/1806061101 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff17c01c460 con 0x7ff18c0a0260 2024-04-26T00:43:03.847 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.835+0000 7ff190c49640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:03.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.835+0000 7ff190c49640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:03.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.835+0000 7ff190c49640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:03.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.835+0000 7ff190c49640 1 -- 172.21.15.37:0/1806061101 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff18c066f40 con 0x7ff18c0a0260 2024-04-26T00:43:03.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.835+0000 7ff1837fe640 1 --2- 172.21.15.37:0/1806061101 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff16c035f00 0x7ff16c0383c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:03.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.835+0000 7ff1837fe640 1 -- 172.21.15.37:0/1806061101 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(899..899 src has 822..899) ==== 3550+0+0 (secure 0 0 0) 0x7ff17c048790 con 0x7ff18c0a0260 2024-04-26T00:43:03.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.835+0000 7ff181d9b640 1 --2- 172.21.15.37:0/1806061101 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff16c035f00 0x7ff16c0383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:03.848 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.835+0000 7ff181d9b640 10 monclient: get_auth_request con 0x7ff16c035f00 auth_method 0 2024-04-26T00:43:03.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.835+0000 7ff181d9b640 1 --2- 172.21.15.37:0/1806061101 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff16c035f00 0x7ff16c0383c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7ff1740ae000 tx=0x7ff1740a9000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:03.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.839+0000 7ff1837fe640 1 -- 172.21.15.37:0/1806061101 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff17c047660 con 0x7ff18c0a0260 2024-04-26T00:43:03.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.839+0000 7ff1837fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:03.849 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:03.839+0000 7ff1837fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:43:04.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:04.059+0000 7ff190c49640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:43:04.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:04.059+0000 7ff190c49640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:43:04.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:04.059+0000 7ff190c49640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:04.061 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:04.059+0000 7ff190c49640 1 -- 172.21.15.37:0/1806061101 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff18c13a870 con 0x7ff18c0a0260 2024-04-26T00:43:04.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:04.059+0000 7ff1837fe640 1 -- 172.21.15.37:0/1806061101 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7ff17c01a760 con 0x7ff18c0a0260 2024-04-26T00:43:04.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:04.059+0000 7ff1837fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:43:04.062 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:04.059+0000 7ff1837fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:43:04.062 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":164,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:43:04.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:04.063+0000 7ff190c49640 1 -- 172.21.15.37:0/1806061101 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff16c035f00 msgr2=0x7ff16c0383c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:04.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:04.063+0000 7ff190c49640 1 --2- 172.21.15.37:0/1806061101 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff16c035f00 0x7ff16c0383c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7ff1740ae000 tx=0x7ff1740a9000 comp rx=0 tx=0).stop 2024-04-26T00:43:04.064 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:04.063+0000 7ff190c49640 10 monclient: shutdown 2024-04-26T00:43:04.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:04.063+0000 7ff190c49640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:04.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:04.063+0000 7ff190c49640 1 -- 172.21.15.37:0/1806061101 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff18c0a0260 msgr2=0x7ff18c1cfd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:04.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:04.063+0000 7ff190c49640 1 --2- 172.21.15.37:0/1806061101 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff18c0a0260 0x7ff18c1cfd40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff17c00c480 tx=0x7ff17c00c940 comp rx=0 tx=0).stop 2024-04-26T00:43:04.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:04.063+0000 7ff190c49640 1 -- 172.21.15.37:0/1806061101 shutdown_connections 2024-04-26T00:43:04.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:04.063+0000 7ff190c49640 1 --2- 172.21.15.37:0/1806061101 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff16c035f00 0x7ff16c0383c0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:04.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:04.063+0000 7ff190c49640 1 --2- 172.21.15.37:0/1806061101 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff18c11fae0 0x7ff18c11ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:04.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:04.063+0000 7ff190c49640 1 --2- 172.21.15.37:0/1806061101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff18c1389e0 0x7ff18c1d0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:04.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:04.063+0000 7ff190c49640 1 --2- 172.21.15.37:0/1806061101 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff18c0a0260 0x7ff18c1cfd40 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:04.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:04.063+0000 7ff190c49640 1 -- 172.21.15.37:0/1806061101 >> 172.21.15.37:0/1806061101 conn(0x7ff18c0a1ad0 msgr2=0x7ff18c0a4660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:04.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:04.063+0000 7ff190c49640 1 -- 172.21.15.37:0/1806061101 shutdown_connections 2024-04-26T00:43:04.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:04.063+0000 7ff190c49640 1 -- 172.21.15.37:0/1806061101 wait complete. 2024-04-26T00:43:07.082 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:43:07.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.215+0000 7f2a81f37640 10 monclient: get_monmap_and_config 2024-04-26T00:43:07.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.215+0000 7f2a81f37640 10 monclient: build_initial_monmap 2024-04-26T00:43:07.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.215+0000 7f2a81f37640 10 monclient: monmap: 2024-04-26T00:43:07.216 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:07.216 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:07.216 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:43:07.216918+0000 2024-04-26T00:43:07.216 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:43:07.216918+0000 2024-04-26T00:43:07.216 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:07.216 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:07.216 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:43:07.216 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:43:07.216 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:43:07.216 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:43:07.216 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:43:07.216 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:07.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.215+0000 7f2a81f37640 1 Processor -- start 2024-04-26T00:43:07.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.215+0000 7f2a81f37640 1 -- start start 2024-04-26T00:43:07.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.215+0000 7f2a81f37640 10 monclient: init 2024-04-26T00:43:07.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.215+0000 7f2a81f37640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:07.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.215+0000 7f2a81f37640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-04-26T00:43:07.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.215+0000 7f2a81f37640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f2a7c136da0 0x7f2a7c09bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:07.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.215+0000 7f2a81f37640 10 monclient(hunting): picked mon.noname-a con 0x7f2a7c136da0 addr v2:172.21.15.37:3300/0 2024-04-26T00:43:07.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.215+0000 7f2a81f37640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2a7c09c410 addr v1:172.21.15.45:6789/0 2024-04-26T00:43:07.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.215+0000 7f2a81f37640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2a7c137890 addr v1:172.21.15.37:6789/0 2024-04-26T00:43:07.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.215+0000 7f2a81f37640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a7c09b4f0 con 0x7f2a7c137890 2024-04-26T00:43:07.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.215+0000 7f2a81f37640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a7c09b7d0 con 0x7f2a7c09c410 2024-04-26T00:43:07.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.215+0000 7f2a81f37640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:07.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.215+0000 7f2a81f37640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f2a7c09bb40 con 0x7f2a7c136da0 2024-04-26T00:43:07.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.215+0000 7f2a81f37640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:07.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.215+0000 7f2a81f37640 10 monclient(hunting): authenticate will time out at 2368.522760s 2024-04-26T00:43:07.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.215+0000 7f2a7bfff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f2a7c137890 0x7f2a7c13e2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:42038/0 (socket says 172.21.15.37:42038) 2024-04-26T00:43:07.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.215+0000 7f2a7bfff640 1 -- 172.21.15.37:0/3991884741 learned_addr learned my addr 172.21.15.37:0/3991884741 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:43:07.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.215+0000 7f2a7b7fe640 1 --2- 172.21.15.37:0/3991884741 >> v2:172.21.15.37:3300/0 conn(0x7f2a7c136da0 0x7f2a7c09bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:07.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.215+0000 7f2a7b7fe640 10 monclient(hunting): get_auth_request con 0x7f2a7c136da0 auth_method 0 2024-04-26T00:43:07.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.215+0000 7f2a7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:07.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.215+0000 7f2a7b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:07.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.215+0000 7f2a7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:07.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.215+0000 7f2a7b7fe640 1 -- 172.21.15.37:0/3991884741 >> v2:172.21.15.37:3300/0 conn(0x7f2a7c136da0 msgr2=0x7f2a7c09bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:43:07.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.215+0000 7f2a7b7fe640 1 -- 172.21.15.37:0/3991884741 >> v2:172.21.15.37:3300/0 conn(0x7f2a7c136da0 msgr2=0x7f2a7c09bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:43:07.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.215+0000 7f2a7b7fe640 1 --2- 172.21.15.37:0/3991884741 >> v2:172.21.15.37:3300/0 conn(0x7f2a7c136da0 0x7f2a7c09bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:43:07.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.215+0000 7f2a7b7fe640 1 --2- 172.21.15.37:0/3991884741 >> v2:172.21.15.37:3300/0 conn(0x7f2a7c136da0 0x7f2a7c09bed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:43:07.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.219+0000 7f2a7bfff640 1 -- 172.21.15.37:0/3991884741 >> v1:172.21.15.37:6789/0 conn(0x7f2a7c137890 legacy=0x7f2a7c13e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:07.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.219+0000 7f2a7affd640 1 -- 172.21.15.37:0/3991884741 >> v1:172.21.15.45:6789/0 conn(0x7f2a7c09c410 legacy=0x7f2a7c137180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:07.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.219+0000 7f2a7affd640 1 -- 172.21.15.37:0/3991884741 >> v1:172.21.15.45:6789/0 conn(0x7f2a7c09c410 legacy=0x7f2a7c137180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-26T00:43:07.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.219+0000 7f2a7affd640 1 -- 172.21.15.37:0/3991884741 >> v1:172.21.15.45:6789/0 conn(0x7f2a7c09c410 legacy=0x7f2a7c137180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:43:07.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.219+0000 7f2a7affd640 1 --1- 172.21.15.37:0/3991884741 >> v1:172.21.15.45:6789/0 conn(0x7f2a7c09c410 0x7f2a7c137180 :-1 s=OPENED pgs=56 cs=1 l=1).handle_message read tag failed 2024-04-26T00:43:07.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.219+0000 7f2a7affd640 1 --1- 172.21.15.37:0/3991884741 >> v1:172.21.15.45:6789/0 conn(0x7f2a7c09c410 0x7f2a7c137180 :-1 s=OPENED pgs=56 cs=1 l=1).fault on lossy channel, failing 2024-04-26T00:43:07.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.219+0000 7f2a7a7fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.45:6789/0 2024-04-26T00:43:07.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.419+0000 7f2a7b7fe640 1 --2- 172.21.15.37:0/3991884741 >> v2:172.21.15.37:3300/0 conn(0x7f2a7c136da0 0x7f2a7c09bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:07.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.419+0000 7f2a7b7fe640 10 monclient(hunting): get_auth_request con 0x7f2a7c136da0 auth_method 0 2024-04-26T00:43:07.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.419+0000 7f2a7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:07.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.419+0000 7f2a7b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:07.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.419+0000 7f2a7b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:43:07.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.419+0000 7f2a7b7fe640 1 -- 172.21.15.37:0/3991884741 >> v2:172.21.15.37:3300/0 conn(0x7f2a7c136da0 msgr2=0x7f2a7c09bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:43:07.419 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.419+0000 7f2a7b7fe640 1 -- 172.21.15.37:0/3991884741 >> v2:172.21.15.37:3300/0 conn(0x7f2a7c136da0 msgr2=0x7f2a7c09bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:43:07.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.419+0000 7f2a7b7fe640 1 --2- 172.21.15.37:0/3991884741 >> v2:172.21.15.37:3300/0 conn(0x7f2a7c136da0 0x7f2a7c09bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:43:07.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.419+0000 7f2a7b7fe640 1 --2- 172.21.15.37:0/3991884741 >> v2:172.21.15.37:3300/0 conn(0x7f2a7c136da0 0x7f2a7c09bed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:43:07.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.823+0000 7f2a7b7fe640 1 --2- 172.21.15.37:0/3991884741 >> v2:172.21.15.37:3300/0 conn(0x7f2a7c136da0 0x7f2a7c09bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:07.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.823+0000 7f2a7b7fe640 10 monclient(hunting): get_auth_request con 0x7f2a7c136da0 auth_method 0 2024-04-26T00:43:07.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.823+0000 7f2a7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:07.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.823+0000 7f2a7b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:07.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.823+0000 7f2a7b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:43:07.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.823+0000 7f2a7b7fe640 1 -- 172.21.15.37:0/3991884741 >> v2:172.21.15.37:3300/0 conn(0x7f2a7c136da0 msgr2=0x7f2a7c09bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:43:07.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.823+0000 7f2a7b7fe640 1 -- 172.21.15.37:0/3991884741 >> v2:172.21.15.37:3300/0 conn(0x7f2a7c136da0 msgr2=0x7f2a7c09bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:43:07.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.823+0000 7f2a7b7fe640 1 --2- 172.21.15.37:0/3991884741 >> v2:172.21.15.37:3300/0 conn(0x7f2a7c136da0 0x7f2a7c09bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:43:07.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:07.823+0000 7f2a7b7fe640 1 --2- 172.21.15.37:0/3991884741 >> v2:172.21.15.37:3300/0 conn(0x7f2a7c136da0 0x7f2a7c09bed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:43:08.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:08.627+0000 7f2a7b7fe640 1 --2- 172.21.15.37:0/3991884741 >> v2:172.21.15.37:3300/0 conn(0x7f2a7c136da0 0x7f2a7c09bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:08.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:08.627+0000 7f2a7b7fe640 10 monclient(hunting): get_auth_request con 0x7f2a7c136da0 auth_method 0 2024-04-26T00:43:08.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:08.627+0000 7f2a7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:08.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:08.627+0000 7f2a7b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:08.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:08.627+0000 7f2a7b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:43:08.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:08.627+0000 7f2a7b7fe640 1 -- 172.21.15.37:0/3991884741 >> v2:172.21.15.37:3300/0 conn(0x7f2a7c136da0 msgr2=0x7f2a7c09bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:43:08.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:08.627+0000 7f2a7b7fe640 1 -- 172.21.15.37:0/3991884741 >> v2:172.21.15.37:3300/0 conn(0x7f2a7c136da0 msgr2=0x7f2a7c09bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:43:08.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:08.627+0000 7f2a7b7fe640 1 --2- 172.21.15.37:0/3991884741 >> v2:172.21.15.37:3300/0 conn(0x7f2a7c136da0 0x7f2a7c09bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:43:08.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:08.627+0000 7f2a7b7fe640 1 --2- 172.21.15.37:0/3991884741 >> v2:172.21.15.37:3300/0 conn(0x7f2a7c136da0 0x7f2a7c09bed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:43:10.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.215+0000 7f2a797fa640 10 monclient(hunting): tick 2024-04-26T00:43:10.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.215+0000 7f2a797fa640 10 monclient(hunting): _check_auth_tickets 2024-04-26T00:43:10.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.215+0000 7f2a797fa640 1 monclient(hunting): continuing hunt 2024-04-26T00:43:10.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.215+0000 7f2a797fa640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T00:43:10.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.215+0000 7f2a797fa640 1 -- 172.21.15.37:0/3991884741 >> v2:172.21.15.37:3300/0 conn(0x7f2a7c136da0 msgr2=0x7f2a7c09bed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:43:10.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.215+0000 7f2a797fa640 1 --2- 172.21.15.37:0/3991884741 >> v2:172.21.15.37:3300/0 conn(0x7f2a7c136da0 0x7f2a7c09bed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:10.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.215+0000 7f2a797fa640 1 -- 172.21.15.37:0/3991884741 >> v1:172.21.15.45:6789/0 conn(0x7f2a7c09c410 legacy=0x7f2a7c137180 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:43:10.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.215+0000 7f2a797fa640 1 -- 172.21.15.37:0/3991884741 >> v1:172.21.15.37:6789/0 conn(0x7f2a7c137890 legacy=0x7f2a7c13e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:10.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.215+0000 7f2a797fa640 10 monclient: _add_conns ranks=[4,2] 2024-04-26T00:43:10.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.215+0000 7f2a797fa640 1 --2- 172.21.15.37:0/3991884741 >> v2:172.21.15.45:3300/0 conn(0x7f2a680036a0 0x7f2a68005b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:10.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.215+0000 7f2a797fa640 10 monclient(hunting): picked mon.noname-c con 0x7f2a680036a0 addr v2:172.21.15.45:3300/0 2024-04-26T00:43:10.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.215+0000 7f2a797fa640 1 --2- 172.21.15.37:0/3991884741 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a680061e0 0x7f2a680086a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:10.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.215+0000 7f2a797fa640 10 monclient(hunting): picked mon.noname-b con 0x7f2a680061e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:10.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.215+0000 7f2a797fa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:10.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.215+0000 7f2a797fa640 1 -- 172.21.15.37:0/3991884741 --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f2a7c09bb40 con 0x7f2a680036a0 2024-04-26T00:43:10.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.215+0000 7f2a797fa640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:10.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.215+0000 7f2a797fa640 1 -- 172.21.15.37:0/3991884741 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2a68008dd0 con 0x7f2a680061e0 2024-04-26T00:43:10.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.215+0000 7f2a797fa640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:10.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.215+0000 7f2a7affd640 1 --2- 172.21.15.37:0/3991884741 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a680061e0 0x7f2a680086a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:10.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.215+0000 7f2a7affd640 10 monclient(hunting): get_auth_request con 0x7f2a680061e0 auth_method 0 2024-04-26T00:43:10.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.215+0000 7f2a7affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:10.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.215+0000 7f2a7affd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:10.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.215+0000 7f2a7affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:10.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.215+0000 7f2a7b7fe640 1 --2- 172.21.15.37:0/3991884741 >> v2:172.21.15.45:3300/0 conn(0x7f2a680036a0 0x7f2a68005b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:10.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.215+0000 7f2a7b7fe640 10 monclient(hunting): get_auth_request con 0x7f2a680036a0 auth_method 0 2024-04-26T00:43:10.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.215+0000 7f2a7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:10.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.215+0000 7f2a7b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:10.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.215+0000 7f2a7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:10.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.215+0000 7f2a7affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:10.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.215+0000 7f2a7affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:10.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.215+0000 7f2a7affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:10.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.215+0000 7f2a7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:10.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.215+0000 7f2a7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:10.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.215+0000 7f2a7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:10.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.219+0000 7f2a7affd640 10 monclient(hunting): handle_auth_done global_id 254414 payload 274 2024-04-26T00:43:10.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.219+0000 7f2a7affd640 1 -- 172.21.15.37:0/3991884741 >> v2:172.21.15.45:3300/0 conn(0x7f2a680036a0 msgr2=0x7f2a68005b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:10.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.219+0000 7f2a7affd640 1 --2- 172.21.15.37:0/3991884741 >> v2:172.21.15.45:3300/0 conn(0x7f2a680036a0 0x7f2a68005b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:10.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.219+0000 7f2a7affd640 10 monclient: _finish_hunting 0 2024-04-26T00:43:10.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.219+0000 7f2a7affd640 1 monclient: found mon.noname-b 2024-04-26T00:43:10.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.219+0000 7f2a7affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:10.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.219+0000 7f2a7affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:43:10.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.219+0000 7f2a7affd640 1 -- 172.21.15.37:0/3991884741 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a7c06dbb0 con 0x7f2a680061e0 2024-04-26T00:43:10.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.219+0000 7f2a7b7fe640 1 --2- 172.21.15.37:0/3991884741 >> v2:172.21.15.45:3300/0 conn(0x7f2a680036a0 0x7f2a68005b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:43:10.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.227+0000 7f2a7affd640 1 -- 172.21.15.37:0/3991884741 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a680061e0 msgr2=0x7f2a680086a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:10.226 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.227+0000 7f2a7affd640 1 --2- 172.21.15.37:0/3991884741 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a680061e0 0x7f2a680086a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2a70007b40 tx=0x7f2a7001c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3aba681c5d56d11 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:10.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.227+0000 7f2a7a7fc640 1 -- 172.21.15.37:0/3991884741 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a70007330 con 0x7f2a680061e0 2024-04-26T00:43:10.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.227+0000 7f2a7a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:10.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.227+0000 7f2a7a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:43:10.227 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.227+0000 7f2a7a7fc640 10 monclient: dump: 2024-04-26T00:43:10.227 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:10.227 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:10.227 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:10.228 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:10.228 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:10.228 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:10.228 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:10.228 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:10.228 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:10.228 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:10.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.227+0000 7f2a7a7fc640 10 monclient: _finish_auth 0 2024-04-26T00:43:10.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.227+0000 7f2a7a7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:43:10.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.227+0000 7f2a7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:10.228 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.227+0000 7f2a7a7fc640 1 -- 172.21.15.37:0/3991884741 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a700ad070 con 0x7f2a680061e0 2024-04-26T00:43:10.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.227+0000 7f2a7a7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:10.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.227+0000 7f2a7a7fc640 1 -- 172.21.15.37:0/3991884741 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a700057d0 con 0x7f2a680061e0 2024-04-26T00:43:10.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.227+0000 7f2a7a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:10.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.227+0000 7f2a7a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:43:10.229 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.227+0000 7f2a7a7fc640 10 monclient: dump: 2024-04-26T00:43:10.229 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:10.229 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:10.229 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:10.229 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:10.230 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:10.230 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:10.230 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:10.230 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:10.230 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:10.230 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:10.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.227+0000 7f2a81f37640 5 monclient: authenticate success, global_id 254414 2024-04-26T00:43:10.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.227+0000 7f2a81f37640 10 monclient: get_monmap_and_config success 2024-04-26T00:43:10.230 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.227+0000 7f2a81f37640 4 set_mon_vals no callback set 2024-04-26T00:43:10.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.227+0000 7f2a81f37640 10 monclient: shutdown 2024-04-26T00:43:10.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.227+0000 7f2a81f37640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:10.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.227+0000 7f2a81f37640 1 -- 172.21.15.37:0/3991884741 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a680061e0 msgr2=0x7f2a680086a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:10.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.227+0000 7f2a81f37640 1 --2- 172.21.15.37:0/3991884741 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a680061e0 0x7f2a680086a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2a70007b40 tx=0x7f2a7001c7f0 comp rx=0 tx=0).stop 2024-04-26T00:43:10.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.227+0000 7f2a7bfff640 1 -- 172.21.15.37:0/3991884741 reap_dead start 2024-04-26T00:43:10.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.227+0000 7f2a81f37640 1 -- 172.21.15.37:0/3991884741 shutdown_connections 2024-04-26T00:43:10.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.227+0000 7f2a81f37640 1 -- 172.21.15.37:0/3991884741 >> 172.21.15.37:0/3991884741 conn(0x7f2a7c0a5f00 msgr2=0x7f2a7c0a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:10.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.227+0000 7f2a81f37640 1 -- 172.21.15.37:0/3991884741 shutdown_connections 2024-04-26T00:43:10.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.227+0000 7f2a81f37640 1 -- 172.21.15.37:0/3991884741 wait complete. 2024-04-26T00:43:10.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.227+0000 7f2a81f37640 10 monclient: build_initial_monmap 2024-04-26T00:43:10.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.227+0000 7f2a81f37640 10 monclient: monmap: 2024-04-26T00:43:10.234 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:10.234 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:43:10.234 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:43:10.234 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:43:10.234 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:10.234 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:43:10.234 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:43:10.234 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:43:10.234 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:43:10.234 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:10.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a81f37640 1 Processor -- start 2024-04-26T00:43:10.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a81f37640 1 -- start start 2024-04-26T00:43:10.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a81f37640 10 monclient: init 2024-04-26T00:43:10.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a81f37640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:10.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a81f37640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:43:10.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a81f37640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a7c1ced60 0x7f2a7c135dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:10.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a81f37640 10 monclient(hunting): picked mon.noname-a con 0x7f2a7c1ced60 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:43:10.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a81f37640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a7c13ace0 0x7f2a7c13b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:10.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a81f37640 10 monclient(hunting): picked mon.noname-b con 0x7f2a7c13ace0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:43:10.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a81f37640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a7c13b680 0x7f2a7c13bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:10.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a81f37640 10 monclient(hunting): picked mon.noname-c con 0x7f2a7c13b680 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:10.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a81f37640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:10.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a81f37640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f2a7c1429c0 con 0x7f2a7c1ced60 2024-04-26T00:43:10.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a81f37640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:10.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a81f37640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2a7c142840 con 0x7f2a7c13ace0 2024-04-26T00:43:10.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a81f37640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:10.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a81f37640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2a7c142b40 con 0x7f2a7c13b680 2024-04-26T00:43:10.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a81f37640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:10.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a81f37640 10 monclient(hunting): authenticate will time out at 2371.537127s 2024-04-26T00:43:10.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a7b7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a7c1ced60 0x7f2a7c135dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:10.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a7bfff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a7c13b680 0x7f2a7c13bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:10.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a7b7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a7c1ced60 0x7f2a7c135dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:53964/0 (socket says 172.21.15.37:53964) 2024-04-26T00:43:10.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a7b7fe640 1 -- 172.21.15.37:0/3959743356 learned_addr learned my addr 172.21.15.37:0/3959743356 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:10.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a7b7fe640 10 monclient(hunting): get_auth_request con 0x7f2a7c1ced60 auth_method 0 2024-04-26T00:43:10.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:10.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a7b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:10.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:10.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a7bfff640 10 monclient(hunting): get_auth_request con 0x7f2a7c13b680 auth_method 0 2024-04-26T00:43:10.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:10.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a7bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:10.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:10.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a7affd640 1 --2- 172.21.15.37:0/3959743356 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a7c13ace0 0x7f2a7c13b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:10.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:10.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:10.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a7b7fe640 1 -- 172.21.15.37:0/3959743356 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a7c1ced60 msgr2=0x7f2a7c135dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:43:10.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a7b7fe640 1 -- 172.21.15.37:0/3959743356 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a7c1ced60 msgr2=0x7f2a7c135dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:43:10.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a7b7fe640 1 --2- 172.21.15.37:0/3959743356 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a7c1ced60 0x7f2a7c135dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:43:10.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:10.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a7affd640 10 monclient(hunting): get_auth_request con 0x7f2a7c13ace0 auth_method 0 2024-04-26T00:43:10.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a7affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:10.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a7affd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:10.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a7affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:10.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a7b7fe640 1 --2- 172.21.15.37:0/3959743356 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a7c1ced60 0x7f2a7c135dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:43:10.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a7affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:10.238 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a7affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:10.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a7affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:10.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a7bfff640 10 monclient(hunting): handle_auth_done global_id 254417 payload 824 2024-04-26T00:43:10.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a7bfff640 1 -- 172.21.15.37:0/3959743356 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a7c13ace0 msgr2=0x7f2a7c13b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:10.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a7bfff640 1 --2- 172.21.15.37:0/3959743356 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a7c13ace0 0x7f2a7c13b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:10.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a7bfff640 1 -- 172.21.15.37:0/3959743356 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a7c1ced60 msgr2=0x7f2a7c135dd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:43:10.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a7bfff640 1 --2- 172.21.15.37:0/3959743356 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a7c1ced60 0x7f2a7c135dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:10.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a7bfff640 10 monclient: _finish_hunting 0 2024-04-26T00:43:10.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a7bfff640 1 monclient: found mon.noname-c 2024-04-26T00:43:10.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a7bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:10.239 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a7bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:43:10.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.231+0000 7f2a7bfff640 1 -- 172.21.15.37:0/3959743356 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a7c13e380 con 0x7f2a7c13b680 2024-04-26T00:43:10.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.235+0000 7f2a7bfff640 1 -- 172.21.15.37:0/3959743356 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a7c13b680 msgr2=0x7f2a7c13bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:10.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.235+0000 7f2a7bfff640 1 --2- 172.21.15.37:0/3959743356 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a7c13b680 0x7f2a7c13bb00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c09ac20 tx=0x7f2a7400ce10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:10.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.235+0000 7f2a63fff640 1 -- 172.21.15.37:0/3959743356 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a74017070 con 0x7f2a7c13b680 2024-04-26T00:43:10.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.235+0000 7f2a63fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:10.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.235+0000 7f2a63fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:43:10.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.235+0000 7f2a63fff640 10 monclient: dump: 2024-04-26T00:43:10.241 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:10.242 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:10.242 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:10.242 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:10.242 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:10.242 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:10.242 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:10.242 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:10.242 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:10.242 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:10.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.235+0000 7f2a63fff640 10 monclient: _finish_auth 0 2024-04-26T00:43:10.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.235+0000 7f2a63fff640 10 monclient: _check_auth_tickets 2024-04-26T00:43:10.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.235+0000 7f2a63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:10.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.235+0000 7f2a63fff640 1 -- 172.21.15.37:0/3959743356 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a7400ac90 con 0x7f2a7c13b680 2024-04-26T00:43:10.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.235+0000 7f2a81f37640 5 monclient: authenticate success, global_id 254417 2024-04-26T00:43:10.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.235+0000 7f2a63fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:10.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.235+0000 7f2a81f37640 10 monclient: _renew_subs 2024-04-26T00:43:10.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.235+0000 7f2a63fff640 1 -- 172.21.15.37:0/3959743356 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a740070d0 con 0x7f2a7c13b680 2024-04-26T00:43:10.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.235+0000 7f2a81f37640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:10.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.235+0000 7f2a81f37640 1 -- 172.21.15.37:0/3959743356 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a7c13f140 con 0x7f2a7c13b680 2024-04-26T00:43:10.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.235+0000 7f2a63fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:10.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.235+0000 7f2a63fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:43:10.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.235+0000 7f2a63fff640 10 monclient: dump: 2024-04-26T00:43:10.243 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:10.243 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:10.243 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:10.243 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:10.244 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:10.244 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:10.244 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:10.244 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:10.244 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:10.244 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:10.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.235+0000 7f2a79ffb640 4 set_mon_vals no callback set 2024-04-26T00:43:10.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.235+0000 7f2a81f37640 10 monclient: _renew_subs 2024-04-26T00:43:10.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.235+0000 7f2a81f37640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:10.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.235+0000 7f2a81f37640 1 -- 172.21.15.37:0/3959743356 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a7c13e660 con 0x7f2a7c13b680 2024-04-26T00:43:10.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.235+0000 7f2a81f37640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:10.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.235+0000 7f2a81f37640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:10.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.235+0000 7f2a81f37640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:10.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.235+0000 7f2a81f37640 1 -- 172.21.15.37:0/3959743356 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a7c066f40 con 0x7f2a7c13b680 2024-04-26T00:43:10.244 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.235+0000 7f2a63fff640 1 -- 172.21.15.37:0/3959743356 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2a7401a070 con 0x7f2a7c13b680 2024-04-26T00:43:10.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.235+0000 7f2a63fff640 1 --2- 172.21.15.37:0/3959743356 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2a68036db0 0x7f2a68039270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:10.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.235+0000 7f2a63fff640 1 -- 172.21.15.37:0/3959743356 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(899..899 src has 822..899) ==== 3550+0+0 (secure 0 0 0) 0x7f2a74048350 con 0x7f2a7c13b680 2024-04-26T00:43:10.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.235+0000 7f2a7b7fe640 1 --2- 172.21.15.37:0/3959743356 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2a68036db0 0x7f2a68039270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:10.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.235+0000 7f2a7b7fe640 10 monclient: get_auth_request con 0x7f2a68036db0 auth_method 0 2024-04-26T00:43:10.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.239+0000 7f2a7b7fe640 1 --2- 172.21.15.37:0/3959743356 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2a68036db0 0x7f2a68039270 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c006e50 tx=0x7f2a6c00c3c0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:10.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.243+0000 7f2a63fff640 1 -- 172.21.15.37:0/3959743356 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f2a7c13e380 con 0x7f2a7c13b680 2024-04-26T00:43:10.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.243+0000 7f2a63fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:10.245 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.243+0000 7f2a63fff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:43:10.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.467+0000 7f2a81f37640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:43:10.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.467+0000 7f2a81f37640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:43:10.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.467+0000 7f2a81f37640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:10.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.467+0000 7f2a81f37640 1 -- 172.21.15.37:0/3959743356 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2a7c09fec0 con 0x7f2a7c13b680 2024-04-26T00:43:10.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.467+0000 7f2a63fff640 1 -- 172.21.15.37:0/3959743356 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1552 (secure 0 0 0) 0x7f2a740072b0 con 0x7f2a7c13b680 2024-04-26T00:43:10.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.467+0000 7f2a63fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:43:10.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.467+0000 7f2a63fff640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:43:10.471 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":164,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"c","quorum_age":31,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:43:10.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.471+0000 7f2a81f37640 1 -- 172.21.15.37:0/3959743356 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2a68036db0 msgr2=0x7f2a68039270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:10.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.471+0000 7f2a81f37640 1 --2- 172.21.15.37:0/3959743356 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2a68036db0 0x7f2a68039270 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c006e50 tx=0x7f2a6c00c3c0 comp rx=0 tx=0).stop 2024-04-26T00:43:10.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.471+0000 7f2a81f37640 10 monclient: shutdown 2024-04-26T00:43:10.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.471+0000 7f2a81f37640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:10.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.471+0000 7f2a81f37640 1 -- 172.21.15.37:0/3959743356 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a7c13b680 msgr2=0x7f2a7c13bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:10.474 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.471+0000 7f2a81f37640 1 --2- 172.21.15.37:0/3959743356 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a7c13b680 0x7f2a7c13bb00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c09ac20 tx=0x7f2a7400ce10 comp rx=0 tx=0).stop 2024-04-26T00:43:10.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.471+0000 7f2a81f37640 1 -- 172.21.15.37:0/3959743356 shutdown_connections 2024-04-26T00:43:10.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.471+0000 7f2a81f37640 1 --2- 172.21.15.37:0/3959743356 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2a68036db0 0x7f2a68039270 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:10.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.471+0000 7f2a81f37640 1 --2- 172.21.15.37:0/3959743356 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a7c13b680 0x7f2a7c13bb00 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:10.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.471+0000 7f2a81f37640 1 --2- 172.21.15.37:0/3959743356 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a7c13ace0 0x7f2a7c13b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:10.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.471+0000 7f2a81f37640 1 --2- 172.21.15.37:0/3959743356 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a7c1ced60 0x7f2a7c135dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:10.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.471+0000 7f2a81f37640 1 -- 172.21.15.37:0/3959743356 >> 172.21.15.37:0/3959743356 conn(0x7f2a7c0a5f00 msgr2=0x7f2a7c09d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:10.475 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.471+0000 7f2a81f37640 1 -- 172.21.15.37:0/3959743356 shutdown_connections 2024-04-26T00:43:10.476 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:10.471+0000 7f2a81f37640 1 -- 172.21.15.37:0/3959743356 wait complete. 2024-04-26T00:43:13.489 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:43:13.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.619+0000 7f0c814b9640 10 monclient: get_monmap_and_config 2024-04-26T00:43:13.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.619+0000 7f0c814b9640 10 monclient: build_initial_monmap 2024-04-26T00:43:13.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.619+0000 7f0c814b9640 10 monclient: monmap: 2024-04-26T00:43:13.621 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:13.621 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:13.622 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:43:13.622164+0000 2024-04-26T00:43:13.622 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:43:13.622164+0000 2024-04-26T00:43:13.622 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:13.622 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:13.622 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:43:13.622 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:43:13.622 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:43:13.622 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:43:13.622 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:43:13.622 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:13.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.619+0000 7f0c814b9640 1 Processor -- start 2024-04-26T00:43:13.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.619+0000 7f0c814b9640 1 -- start start 2024-04-26T00:43:13.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.619+0000 7f0c814b9640 10 monclient: init 2024-04-26T00:43:13.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c814b9640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:13.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c814b9640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-04-26T00:43:13.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c814b9640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f0c7c136da0 0x7f0c7c09bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:13.625 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c814b9640 10 monclient(hunting): picked mon.noname-a con 0x7f0c7c136da0 addr v2:172.21.15.37:3300/0 2024-04-26T00:43:13.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c814b9640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f0c7c09c410 0x7f0c7c137180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:13.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c814b9640 10 monclient(hunting): picked mon.noname-c con 0x7f0c7c09c410 addr v2:172.21.15.45:3300/0 2024-04-26T00:43:13.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c814b9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0c7c1376c0 addr v1:172.21.15.45:6789/0 2024-04-26T00:43:13.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c814b9640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c7c09b4f0 con 0x7f0c7c1376c0 2024-04-26T00:43:13.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c814b9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:13.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c814b9640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f0c7c09b210 con 0x7f0c7c136da0 2024-04-26T00:43:13.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c814b9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:13.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c814b9640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f0c7c09b7d0 con 0x7f0c7c09c410 2024-04-26T00:43:13.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c814b9640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:13.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c814b9640 10 monclient(hunting): authenticate will time out at 2374.928608s 2024-04-26T00:43:13.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c7affd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f0c7c136da0 0x7f0c7c09bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:13.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c7affd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f0c7c136da0 0x7f0c7c09bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:53990/0 (socket says 172.21.15.37:53990) 2024-04-26T00:43:13.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c7affd640 1 -- 172.21.15.37:0/4294545597 learned_addr learned my addr 172.21.15.37:0/4294545597 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:13.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c7affd640 10 monclient(hunting): get_auth_request con 0x7f0c7c136da0 auth_method 0 2024-04-26T00:43:13.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c7affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:13.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c7affd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:13.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c7affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:13.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c7affd640 1 -- 172.21.15.37:0/4294545597 >> v2:172.21.15.37:3300/0 conn(0x7f0c7c136da0 msgr2=0x7f0c7c09bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T00:43:13.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c7affd640 1 -- 172.21.15.37:0/4294545597 >> v2:172.21.15.37:3300/0 conn(0x7f0c7c136da0 msgr2=0x7f0c7c09bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:43:13.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c7affd640 1 --2- 172.21.15.37:0/4294545597 >> v2:172.21.15.37:3300/0 conn(0x7f0c7c136da0 0x7f0c7c09bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:43:13.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c7affd640 1 --2- 172.21.15.37:0/4294545597 >> v2:172.21.15.37:3300/0 conn(0x7f0c7c136da0 0x7f0c7c09bed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:43:13.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c7a7fc640 1 --2- 172.21.15.37:0/4294545597 >> v2:172.21.15.45:3300/0 conn(0x7f0c7c09c410 0x7f0c7c137180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:13.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c7a7fc640 10 monclient(hunting): get_auth_request con 0x7f0c7c09c410 auth_method 0 2024-04-26T00:43:13.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c7a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:13.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c7a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:13.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:13.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c7a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:13.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c7a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:13.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c7a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:13.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c7a7fc640 10 monclient(hunting): handle_auth_done global_id 274186 payload 274 2024-04-26T00:43:13.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c7a7fc640 1 -- 172.21.15.37:0/4294545597 >> v2:172.21.15.37:3300/0 conn(0x7f0c7c136da0 msgr2=0x7f0c7c09bed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:43:13.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c7a7fc640 1 --2- 172.21.15.37:0/4294545597 >> v2:172.21.15.37:3300/0 conn(0x7f0c7c136da0 0x7f0c7c09bed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:13.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c7a7fc640 1 -- 172.21.15.37:0/4294545597 >> v1:172.21.15.45:6789/0 conn(0x7f0c7c1376c0 legacy=0x7f0c7c133b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:13.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c7a7fc640 10 monclient: _finish_hunting 0 2024-04-26T00:43:13.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c7a7fc640 1 monclient: found mon.noname-c 2024-04-26T00:43:13.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c7a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:13.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c7a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:43:13.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.623+0000 7f0c7a7fc640 1 -- 172.21.15.37:0/4294545597 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c7c137b30 con 0x7f0c7c09c410 2024-04-26T00:43:13.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.627+0000 7f0c7a7fc640 1 -- 172.21.15.37:0/4294545597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c7c09c410 msgr2=0x7f0c7c137180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:13.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.627+0000 7f0c7a7fc640 1 --2- 172.21.15.37:0/4294545597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c7c09c410 0x7f0c7c137180 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c0a05c0 tx=0x7f0c7001c390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2af8848a17a99d44 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:13.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.627+0000 7f0c79ffb640 1 -- 172.21.15.37:0/4294545597 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c700b0070 con 0x7f0c7c09c410 2024-04-26T00:43:13.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.627+0000 7f0c79ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:13.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.627+0000 7f0c79ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:43:13.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.627+0000 7f0c79ffb640 10 monclient: dump: 2024-04-26T00:43:13.631 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:13.631 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:13.631 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:13.632 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:13.632 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:13.632 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:13.632 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:13.632 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:13.632 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:13.632 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:13.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.627+0000 7f0c79ffb640 10 monclient: _finish_auth 0 2024-04-26T00:43:13.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.627+0000 7f0c79ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:43:13.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.627+0000 7f0c79ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:13.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.627+0000 7f0c814b9640 5 monclient: authenticate success, global_id 274186 2024-04-26T00:43:13.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.627+0000 7f0c814b9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:43:13.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.627+0000 7f0c79ffb640 1 -- 172.21.15.37:0/4294545597 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c700ab600 con 0x7f0c7c09c410 2024-04-26T00:43:13.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.627+0000 7f0c79ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:13.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.627+0000 7f0c79ffb640 1 -- 172.21.15.37:0/4294545597 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c700ab9d0 con 0x7f0c7c09c410 2024-04-26T00:43:13.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.627+0000 7f0c814b9640 10 monclient: get_monmap_and_config success 2024-04-26T00:43:13.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.627+0000 7f0c79ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:13.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.627+0000 7f0c814b9640 4 set_mon_vals no callback set 2024-04-26T00:43:13.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.627+0000 7f0c79ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:43:13.633 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.627+0000 7f0c79ffb640 10 monclient: dump: 2024-04-26T00:43:13.633 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:13.633 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:13.633 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:13.633 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:13.633 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:13.634 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:13.634 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:13.634 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:13.634 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:13.634 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:13.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c814b9640 10 monclient: shutdown 2024-04-26T00:43:13.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c814b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:13.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c814b9640 1 -- 172.21.15.37:0/4294545597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c7c09c410 msgr2=0x7f0c7c137180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:13.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c814b9640 1 --2- 172.21.15.37:0/4294545597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c7c09c410 0x7f0c7c137180 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c0a05c0 tx=0x7f0c7001c390 comp rx=0 tx=0).stop 2024-04-26T00:43:13.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c814b9640 1 -- 172.21.15.37:0/4294545597 shutdown_connections 2024-04-26T00:43:13.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c814b9640 1 --2- 172.21.15.37:0/4294545597 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c7c09c410 0x7f0c7c137180 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:13.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c814b9640 1 --2- 172.21.15.37:0/4294545597 >> v2:172.21.15.37:3300/0 conn(0x7f0c7c136da0 0x7f0c7c09bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:13.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c814b9640 1 -- 172.21.15.37:0/4294545597 >> 172.21.15.37:0/4294545597 conn(0x7f0c7c0a5f00 msgr2=0x7f0c7c0a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:13.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c814b9640 1 -- 172.21.15.37:0/4294545597 shutdown_connections 2024-04-26T00:43:13.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c814b9640 1 -- 172.21.15.37:0/4294545597 wait complete. 2024-04-26T00:43:13.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c814b9640 10 monclient: build_initial_monmap 2024-04-26T00:43:13.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c814b9640 10 monclient: monmap: 2024-04-26T00:43:13.640 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:13.640 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:43:13.640 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:43:13.640 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:43:13.640 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:13.640 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:43:13.640 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:43:13.640 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:43:13.640 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:43:13.640 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:13.640 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c814b9640 1 Processor -- start 2024-04-26T00:43:13.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c814b9640 1 -- start start 2024-04-26T00:43:13.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c814b9640 10 monclient: init 2024-04-26T00:43:13.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c814b9640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:13.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c814b9640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:43:13.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c814b9640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0c7c09c410 0x7f0c7c12cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:13.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c814b9640 10 monclient(hunting): picked mon.noname-a con 0x7f0c7c09c410 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:43:13.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c814b9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c7c136da0 0x7f0c7c12d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:13.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c814b9640 10 monclient(hunting): picked mon.noname-b con 0x7f0c7c136da0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:43:13.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c814b9640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c7c1376c0 0x7f0c7c1cf320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:13.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c814b9640 10 monclient(hunting): picked mon.noname-c con 0x7f0c7c1376c0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:13.641 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c814b9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:13.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c814b9640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f0c7c141ee0 con 0x7f0c7c09c410 2024-04-26T00:43:13.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c814b9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:13.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c814b9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0c7c141d60 con 0x7f0c7c136da0 2024-04-26T00:43:13.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c814b9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:13.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c814b9640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0c7c142060 con 0x7f0c7c1376c0 2024-04-26T00:43:13.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c814b9640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:13.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c814b9640 10 monclient(hunting): authenticate will time out at 2374.938309s 2024-04-26T00:43:13.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c7affd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0c7c09c410 0x7f0c7c12cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:13.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c7affd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0c7c09c410 0x7f0c7c12cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:54004/0 (socket says 172.21.15.37:54004) 2024-04-26T00:43:13.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c7affd640 1 -- 172.21.15.37:0/3178329560 learned_addr learned my addr 172.21.15.37:0/3178329560 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:13.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c7affd640 10 monclient(hunting): get_auth_request con 0x7f0c7c09c410 auth_method 0 2024-04-26T00:43:13.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c7affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:13.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c7affd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:13.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c7affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:13.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c7b7fe640 1 --2- 172.21.15.37:0/3178329560 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c7c1376c0 0x7f0c7c1cf320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:13.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c7b7fe640 10 monclient(hunting): get_auth_request con 0x7f0c7c1376c0 auth_method 0 2024-04-26T00:43:13.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:13.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c7b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:13.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:13.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c7affd640 1 -- 172.21.15.37:0/3178329560 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0c7c09c410 msgr2=0x7f0c7c12cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:43:13.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c7affd640 1 -- 172.21.15.37:0/3178329560 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0c7c09c410 msgr2=0x7f0c7c12cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:43:13.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c7affd640 1 --2- 172.21.15.37:0/3178329560 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0c7c09c410 0x7f0c7c12cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:43:13.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:13.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:13.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:13.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c7affd640 1 --2- 172.21.15.37:0/3178329560 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0c7c09c410 0x7f0c7c12cf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:43:13.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c7a7fc640 1 --2- 172.21.15.37:0/3178329560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c7c136da0 0x7f0c7c12d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:13.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c7a7fc640 10 monclient(hunting): get_auth_request con 0x7f0c7c136da0 auth_method 0 2024-04-26T00:43:13.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c7a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:13.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c7a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:13.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:13.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c7a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:13.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c7a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:13.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c7a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:13.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c7b7fe640 10 monclient(hunting): handle_auth_done global_id 254420 payload 824 2024-04-26T00:43:13.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c7b7fe640 1 -- 172.21.15.37:0/3178329560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c7c136da0 msgr2=0x7f0c7c12d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:13.644 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c7b7fe640 1 --2- 172.21.15.37:0/3178329560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c7c136da0 0x7f0c7c12d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:13.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c7b7fe640 1 -- 172.21.15.37:0/3178329560 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0c7c09c410 msgr2=0x7f0c7c12cf70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:43:13.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c7b7fe640 1 --2- 172.21.15.37:0/3178329560 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0c7c09c410 0x7f0c7c12cf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:13.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c7b7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:43:13.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c7b7fe640 1 monclient: found mon.noname-c 2024-04-26T00:43:13.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:13.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c7b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:43:13.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.631+0000 7f0c7b7fe640 1 -- 172.21.15.37:0/3178329560 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c7c09ec50 con 0x7f0c7c1376c0 2024-04-26T00:43:13.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.635+0000 7f0c7b7fe640 1 -- 172.21.15.37:0/3178329560 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c7c1376c0 msgr2=0x7f0c7c1cf320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:13.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.635+0000 7f0c7b7fe640 1 --2- 172.21.15.37:0/3178329560 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c7c1376c0 0x7f0c7c1cf320 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0c7400c7b0 tx=0x7f0c7400cc70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:13.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.635+0000 7f0c637fe640 1 -- 172.21.15.37:0/3178329560 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c74016020 con 0x7f0c7c1376c0 2024-04-26T00:43:13.645 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.635+0000 7f0c637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:13.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.635+0000 7f0c637fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:43:13.646 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.635+0000 7f0c637fe640 10 monclient: dump: 2024-04-26T00:43:13.646 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:13.646 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:13.646 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:13.646 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:13.646 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:13.646 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:13.646 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:13.646 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:13.646 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:13.647 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:13.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.635+0000 7f0c637fe640 10 monclient: _finish_auth 0 2024-04-26T00:43:13.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.635+0000 7f0c637fe640 10 monclient: _check_auth_tickets 2024-04-26T00:43:13.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.635+0000 7f0c637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:13.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.635+0000 7f0c637fe640 1 -- 172.21.15.37:0/3178329560 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c7400ac60 con 0x7f0c7c1376c0 2024-04-26T00:43:13.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.635+0000 7f0c637fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:13.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.635+0000 7f0c637fe640 1 -- 172.21.15.37:0/3178329560 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c7400ce60 con 0x7f0c7c1376c0 2024-04-26T00:43:13.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.635+0000 7f0c814b9640 5 monclient: authenticate success, global_id 254420 2024-04-26T00:43:13.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.635+0000 7f0c814b9640 10 monclient: _renew_subs 2024-04-26T00:43:13.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.635+0000 7f0c814b9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:13.647 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.635+0000 7f0c814b9640 1 -- 172.21.15.37:0/3178329560 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c7c0a8cf0 con 0x7f0c7c1376c0 2024-04-26T00:43:13.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.635+0000 7f0c637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:13.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.635+0000 7f0c637fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:43:13.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.635+0000 7f0c637fe640 10 monclient: dump: 2024-04-26T00:43:13.648 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:13.648 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:13.648 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:13.648 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:13.648 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:13.648 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:13.648 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:13.648 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:13.649 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:13.649 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:13.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.635+0000 7f0c797fa640 4 set_mon_vals no callback set 2024-04-26T00:43:13.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.635+0000 7f0c814b9640 10 monclient: _renew_subs 2024-04-26T00:43:13.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.635+0000 7f0c814b9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:13.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.635+0000 7f0c814b9640 1 -- 172.21.15.37:0/3178329560 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c7c09ef30 con 0x7f0c7c1376c0 2024-04-26T00:43:13.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.635+0000 7f0c637fe640 1 -- 172.21.15.37:0/3178329560 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f0c74019020 con 0x7f0c7c1376c0 2024-04-26T00:43:13.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.635+0000 7f0c637fe640 1 --2- 172.21.15.37:0/3178329560 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0c68035e80 0x7f0c68038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:13.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.635+0000 7f0c814b9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:13.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.635+0000 7f0c814b9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:13.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.635+0000 7f0c814b9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:13.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.635+0000 7f0c637fe640 1 -- 172.21.15.37:0/3178329560 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(899..899 src has 822..899) ==== 3550+0+0 (secure 0 0 0) 0x7f0c740487f0 con 0x7f0c7c1376c0 2024-04-26T00:43:13.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.635+0000 7f0c814b9640 1 -- 172.21.15.37:0/3178329560 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c7c066f40 con 0x7f0c7c1376c0 2024-04-26T00:43:13.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.635+0000 7f0c7affd640 1 --2- 172.21.15.37:0/3178329560 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0c68035e80 0x7f0c68038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:13.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.635+0000 7f0c7affd640 10 monclient: get_auth_request con 0x7f0c68035e80 auth_method 0 2024-04-26T00:43:13.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.639+0000 7f0c7affd640 1 --2- 172.21.15.37:0/3178329560 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0c68035e80 0x7f0c68038340 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c0091d0 tx=0x7f0c6c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:13.650 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.639+0000 7f0c637fe640 1 -- 172.21.15.37:0/3178329560 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f0c74020030 con 0x7f0c7c1376c0 2024-04-26T00:43:13.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.643+0000 7f0c637fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:13.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.643+0000 7f0c637fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:43:13.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.863+0000 7f0c814b9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:43:13.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.863+0000 7f0c814b9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:43:13.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.863+0000 7f0c814b9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:13.864 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.863+0000 7f0c814b9640 1 -- 172.21.15.37:0/3178329560 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0c7c12d9f0 con 0x7f0c7c1376c0 2024-04-26T00:43:13.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.863+0000 7f0c637fe640 1 -- 172.21.15.37:0/3178329560 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7f0c7400ae10 con 0x7f0c7c1376c0 2024-04-26T00:43:13.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.863+0000 7f0c637fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:43:13.865 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.863+0000 7f0c637fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:43:13.865 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":166,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:43:13.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.867+0000 7f0c814b9640 1 -- 172.21.15.37:0/3178329560 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0c68035e80 msgr2=0x7f0c68038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:13.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.867+0000 7f0c814b9640 1 --2- 172.21.15.37:0/3178329560 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0c68035e80 0x7f0c68038340 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c0091d0 tx=0x7f0c6c008040 comp rx=0 tx=0).stop 2024-04-26T00:43:13.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.867+0000 7f0c814b9640 10 monclient: shutdown 2024-04-26T00:43:13.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.867+0000 7f0c814b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:13.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.867+0000 7f0c814b9640 1 -- 172.21.15.37:0/3178329560 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c7c1376c0 msgr2=0x7f0c7c1cf320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:13.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.867+0000 7f0c814b9640 1 --2- 172.21.15.37:0/3178329560 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c7c1376c0 0x7f0c7c1cf320 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0c7400c7b0 tx=0x7f0c7400cc70 comp rx=0 tx=0).stop 2024-04-26T00:43:13.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.867+0000 7f0c814b9640 1 -- 172.21.15.37:0/3178329560 shutdown_connections 2024-04-26T00:43:13.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.867+0000 7f0c814b9640 1 --2- 172.21.15.37:0/3178329560 >> v2:172.21.15.45:6812/2379918623 conn(0x7f0c68035e80 0x7f0c68038340 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:13.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.867+0000 7f0c814b9640 1 --2- 172.21.15.37:0/3178329560 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0c7c1376c0 0x7f0c7c1cf320 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:13.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.867+0000 7f0c814b9640 1 --2- 172.21.15.37:0/3178329560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0c7c136da0 0x7f0c7c12d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:13.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.867+0000 7f0c814b9640 1 --2- 172.21.15.37:0/3178329560 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0c7c09c410 0x7f0c7c12cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:13.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.867+0000 7f0c814b9640 1 -- 172.21.15.37:0/3178329560 >> 172.21.15.37:0/3178329560 conn(0x7f0c7c0a5f00 msgr2=0x7f0c7c135550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:13.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.867+0000 7f0c814b9640 1 -- 172.21.15.37:0/3178329560 shutdown_connections 2024-04-26T00:43:13.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.867+0000 7f0c814b9640 1 -- 172.21.15.37:0/3178329560 wait complete. 2024-04-26T00:43:13.882 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:43:13.882 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T00:43:13.883 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T00:43:13.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.987+0000 7f23a576b640 10 monclient: get_monmap_and_config 2024-04-26T00:43:13.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.987+0000 7f23a576b640 10 monclient: build_initial_monmap 2024-04-26T00:43:13.989 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.987+0000 7f23a576b640 10 monclient: monmap: 2024-04-26T00:43:13.989 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:13.990 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:13.990 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:43:13.990623+0000 2024-04-26T00:43:13.990 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:43:13.990623+0000 2024-04-26T00:43:13.990 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:13.990 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:13.990 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:43:13.990 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:43:13.990 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:43:13.990 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:43:13.990 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:43:13.990 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:13.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.987+0000 7f23a576b640 1 Processor -- start 2024-04-26T00:43:13.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.987+0000 7f23a576b640 1 -- start start 2024-04-26T00:43:13.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.987+0000 7f23a576b640 10 monclient: init 2024-04-26T00:43:13.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.987+0000 7f23a576b640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:13.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.987+0000 7f23a576b640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-04-26T00:43:13.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.987+0000 7f23a576b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f23a0133d70 0x7f23a0134150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:13.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.987+0000 7f23a576b640 10 monclient(hunting): picked mon.noname-b con 0x7f23a0133d70 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:13.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.987+0000 7f23a576b640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f23a00a0520 0x7f23a0072070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:13.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.987+0000 7f23a576b640 10 monclient(hunting): picked mon.noname-a con 0x7f23a00a0520 addr v2:172.21.15.37:3300/0 2024-04-26T00:43:13.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.987+0000 7f23a576b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f23a00a1890 addr v1:172.21.15.37:6789/0 2024-04-26T00:43:13.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.987+0000 7f23a576b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23a00a0bd0 con 0x7f23a00a1890 2024-04-26T00:43:13.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.987+0000 7f23a576b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:13.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.987+0000 7f23a576b640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f23a009fc50 con 0x7f23a00a0520 2024-04-26T00:43:13.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.987+0000 7f23a576b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:13.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.987+0000 7f23a576b640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f23a009f030 con 0x7f23a0133d70 2024-04-26T00:43:13.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.987+0000 7f23a576b640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:13.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.987+0000 7f23a576b640 10 monclient(hunting): authenticate will time out at 2375.295886s 2024-04-26T00:43:13.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.991+0000 7f239effd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f23a0133d70 0x7f23a0134150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:13.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.991+0000 7f239effd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f23a0133d70 0x7f23a0134150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:35860/0 (socket says 172.21.15.37:35860) 2024-04-26T00:43:13.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.991+0000 7f239effd640 1 -- 172.21.15.37:0/1372579623 learned_addr learned my addr 172.21.15.37:0/1372579623 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:13.993 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.991+0000 7f239effd640 10 monclient(hunting): get_auth_request con 0x7f23a0133d70 auth_method 0 2024-04-26T00:43:13.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.991+0000 7f239effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:13.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.991+0000 7f239effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:13.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.991+0000 7f239effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:13.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.991+0000 7f239e7fc640 1 --2- 172.21.15.37:0/1372579623 >> v2:172.21.15.37:3300/0 conn(0x7f23a00a0520 0x7f23a0072070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:13.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.991+0000 7f239effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:13.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.991+0000 7f239effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:13.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.991+0000 7f239effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:13.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.991+0000 7f239e7fc640 10 monclient(hunting): get_auth_request con 0x7f23a00a0520 auth_method 0 2024-04-26T00:43:13.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.991+0000 7f239e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:13.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.991+0000 7f239e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:13.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.991+0000 7f239e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:13.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.991+0000 7f239e7fc640 1 -- 172.21.15.37:0/1372579623 >> v2:172.21.15.37:3300/0 conn(0x7f23a00a0520 msgr2=0x7f23a0072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T00:43:13.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.991+0000 7f239e7fc640 1 -- 172.21.15.37:0/1372579623 >> v2:172.21.15.37:3300/0 conn(0x7f23a00a0520 msgr2=0x7f23a0072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T00:43:13.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.991+0000 7f239e7fc640 1 --2- 172.21.15.37:0/1372579623 >> v2:172.21.15.37:3300/0 conn(0x7f23a00a0520 0x7f23a0072070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:43:13.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.991+0000 7f239effd640 10 monclient(hunting): handle_auth_done global_id 254423 payload 274 2024-04-26T00:43:13.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.991+0000 7f239e7fc640 1 --2- 172.21.15.37:0/1372579623 >> v2:172.21.15.37:3300/0 conn(0x7f23a00a0520 0x7f23a0072070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:43:13.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.991+0000 7f239effd640 1 -- 172.21.15.37:0/1372579623 >> v2:172.21.15.37:3300/0 conn(0x7f23a00a0520 msgr2=0x7f23a0072070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T00:43:13.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.991+0000 7f239effd640 1 --2- 172.21.15.37:0/1372579623 >> v2:172.21.15.37:3300/0 conn(0x7f23a00a0520 0x7f23a0072070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:13.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.991+0000 7f239effd640 1 -- 172.21.15.37:0/1372579623 >> v1:172.21.15.37:6789/0 conn(0x7f23a00a1890 legacy=0x7f23a00725b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:13.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.991+0000 7f239effd640 10 monclient: _finish_hunting 0 2024-04-26T00:43:13.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.991+0000 7f239effd640 1 monclient: found mon.noname-b 2024-04-26T00:43:13.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.991+0000 7f239effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:13.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.991+0000 7f239effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:43:13.995 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.991+0000 7f239effd640 1 -- 172.21.15.37:0/1372579623 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23a00a2ad0 con 0x7f23a0133d70 2024-04-26T00:43:14.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239effd640 1 -- 172.21.15.37:0/1372579623 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f23a0133d70 msgr2=0x7f23a0134150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:14.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239effd640 1 --2- 172.21.15.37:0/1372579623 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f23a0133d70 0x7f23a0134150 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f23900098a0 tx=0x7f239001c2a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75e576e1802b82dc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:14.004 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239dffb640 1 -- 172.21.15.37:0/1372579623 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23900b0070 con 0x7f23a0133d70 2024-04-26T00:43:14.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:14.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:43:14.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239dffb640 10 monclient: dump: 2024-04-26T00:43:14.005 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:14.005 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:14.005 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:14.005 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:14.005 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:14.005 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:14.005 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:14.005 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:14.005 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:14.005 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:14.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239dffb640 10 monclient: _finish_auth 0 2024-04-26T00:43:14.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239dffb640 10 monclient: _check_auth_tickets 2024-04-26T00:43:14.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:14.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239dffb640 1 -- 172.21.15.37:0/1372579623 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23900ab600 con 0x7f23a0133d70 2024-04-26T00:43:14.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f23a576b640 5 monclient: authenticate success, global_id 254423 2024-04-26T00:43:14.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f23a576b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:43:14.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239dffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:14.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f23a576b640 10 monclient: get_monmap_and_config success 2024-04-26T00:43:14.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f23a576b640 4 set_mon_vals no callback set 2024-04-26T00:43:14.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239dffb640 1 -- 172.21.15.37:0/1372579623 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23900ab9d0 con 0x7f23a0133d70 2024-04-26T00:43:14.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:14.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:43:14.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239dffb640 10 monclient: dump: 2024-04-26T00:43:14.006 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:14.007 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:14.007 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:14.007 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:14.007 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:14.007 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:14.007 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:14.007 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:14.007 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:14.007 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:14.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f23a576b640 10 monclient: shutdown 2024-04-26T00:43:14.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f23a576b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:14.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f23a576b640 1 -- 172.21.15.37:0/1372579623 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f23a0133d70 msgr2=0x7f23a0134150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:14.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f23a576b640 1 --2- 172.21.15.37:0/1372579623 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f23a0133d70 0x7f23a0134150 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f23900098a0 tx=0x7f239001c2a0 comp rx=0 tx=0).stop 2024-04-26T00:43:14.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f23a576b640 1 -- 172.21.15.37:0/1372579623 shutdown_connections 2024-04-26T00:43:14.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f23a576b640 1 --2- 172.21.15.37:0/1372579623 >> v2:172.21.15.37:3300/0 conn(0x7f23a00a0520 0x7f23a0072070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:14.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f23a576b640 1 --2- 172.21.15.37:0/1372579623 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f23a0133d70 0x7f23a0134150 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:14.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f23a576b640 1 -- 172.21.15.37:0/1372579623 >> 172.21.15.37:0/1372579623 conn(0x7f23a00a1d50 msgr2=0x7f23a00a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:14.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f23a576b640 1 -- 172.21.15.37:0/1372579623 shutdown_connections 2024-04-26T00:43:14.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f23a576b640 1 -- 172.21.15.37:0/1372579623 wait complete. 2024-04-26T00:43:14.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f23a576b640 10 monclient: build_initial_monmap 2024-04-26T00:43:14.008 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f23a576b640 10 monclient: monmap: 2024-04-26T00:43:14.008 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:14.008 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:43:14.008 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:43:14.008 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:43:14.008 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:14.008 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:43:14.008 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:43:14.009 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:43:14.009 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:43:14.009 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:14.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f23a576b640 1 Processor -- start 2024-04-26T00:43:14.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f23a576b640 1 -- start start 2024-04-26T00:43:14.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f23a576b640 10 monclient: init 2024-04-26T00:43:14.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f23a576b640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:14.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f23a576b640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:43:14.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f23a576b640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a00a0520 0x7f23a01cee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:14.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f23a576b640 10 monclient(hunting): picked mon.noname-a con 0x7f23a00a0520 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:43:14.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f23a576b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f23a00a1890 0x7f23a0132a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:14.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f23a576b640 10 monclient(hunting): picked mon.noname-c con 0x7f23a00a1890 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:14.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f23a576b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23a0133d70 0x7f23a013cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:14.009 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f23a576b640 10 monclient(hunting): picked mon.noname-b con 0x7f23a0133d70 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:43:14.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f23a576b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:14.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f23a576b640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f23a0072ed0 con 0x7f23a00a0520 2024-04-26T00:43:14.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f23a576b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:14.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f23a576b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f23a0141a40 con 0x7f23a0133d70 2024-04-26T00:43:14.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f23a576b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:14.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f23a576b640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f23a0141bc0 con 0x7f23a00a1890 2024-04-26T00:43:14.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f23a576b640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:14.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f23a576b640 10 monclient(hunting): authenticate will time out at 2375.302771s 2024-04-26T00:43:14.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239effd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a00a0520 0x7f23a01cee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:14.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239e7fc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f23a00a1890 0x7f23a0132a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:14.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239effd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a00a0520 0x7f23a01cee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:54016/0 (socket says 172.21.15.37:54016) 2024-04-26T00:43:14.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239effd640 1 -- 172.21.15.37:0/1544468132 learned_addr learned my addr 172.21.15.37:0/1544468132 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:14.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239effd640 10 monclient(hunting): get_auth_request con 0x7f23a00a0520 auth_method 0 2024-04-26T00:43:14.010 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:14.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:14.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:14.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239e7fc640 10 monclient(hunting): get_auth_request con 0x7f23a00a1890 auth_method 0 2024-04-26T00:43:14.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:14.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:14.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:14.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239effd640 1 -- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a00a0520 msgr2=0x7f23a01cee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:43:14.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239effd640 1 -- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a00a0520 msgr2=0x7f23a01cee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:43:14.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239effd640 1 --2- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a00a0520 0x7f23a01cee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:43:14.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:14.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:14.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239effd640 1 --2- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a00a0520 0x7f23a01cee90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:43:14.011 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:14.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239f7fe640 1 --2- 172.21.15.37:0/1544468132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23a0133d70 0x7f23a013cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:14.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239f7fe640 10 monclient(hunting): get_auth_request con 0x7f23a0133d70 auth_method 0 2024-04-26T00:43:14.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:14.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:14.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:14.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239e7fc640 10 monclient(hunting): handle_auth_done global_id 254426 payload 824 2024-04-26T00:43:14.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239e7fc640 1 -- 172.21.15.37:0/1544468132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23a0133d70 msgr2=0x7f23a013cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:14.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239e7fc640 1 --2- 172.21.15.37:0/1544468132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23a0133d70 0x7f23a013cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:14.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239e7fc640 1 -- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a00a0520 msgr2=0x7f23a01cee90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T00:43:14.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239e7fc640 1 --2- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a00a0520 0x7f23a01cee90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:14.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239e7fc640 10 monclient: _finish_hunting 0 2024-04-26T00:43:14.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239e7fc640 1 monclient: found mon.noname-c 2024-04-26T00:43:14.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:14.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239e7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:43:14.012 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.995+0000 7f239e7fc640 1 -- 172.21.15.37:0/1544468132 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23a01cfda0 con 0x7f23a00a1890 2024-04-26T00:43:14.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f239e7fc640 1 -- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f23a00a1890 msgr2=0x7f23a0132a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:14.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f239e7fc640 1 --2- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f23a00a1890 0x7f23a0132a70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f238800e840 tx=0x7f238800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:14.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f23977fe640 1 -- 172.21.15.37:0/1544468132 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2388017030 con 0x7f23a00a1890 2024-04-26T00:43:14.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f23977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:14.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f23977fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:43:14.013 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f23977fe640 10 monclient: dump: 2024-04-26T00:43:14.013 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:14.013 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:14.013 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:14.013 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:14.013 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:14.013 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:14.013 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:14.014 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:14.014 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:14.014 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:14.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f23977fe640 10 monclient: _finish_auth 0 2024-04-26T00:43:14.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f23977fe640 10 monclient: _check_auth_tickets 2024-04-26T00:43:14.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f23977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:14.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f23977fe640 1 -- 172.21.15.37:0/1544468132 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2388004020 con 0x7f23a00a1890 2024-04-26T00:43:14.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f23977fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:14.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f23977fe640 1 -- 172.21.15.37:0/1544468132 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f238800f040 con 0x7f23a00a1890 2024-04-26T00:43:14.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f23a576b640 5 monclient: authenticate success, global_id 254426 2024-04-26T00:43:14.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f23a576b640 10 monclient: _renew_subs 2024-04-26T00:43:14.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f23a576b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:14.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f23a576b640 1 -- 172.21.15.37:0/1544468132 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23a011b9f0 con 0x7f23a00a1890 2024-04-26T00:43:14.014 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f23977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:14.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f23977fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:43:14.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f239cff9640 4 set_mon_vals no callback set 2024-04-26T00:43:14.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f23977fe640 10 monclient: dump: 2024-04-26T00:43:14.015 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:14.015 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:14.015 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:14.015 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:14.015 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:14.015 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:14.015 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:14.015 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:14.015 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:14.015 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:14.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f23a576b640 10 monclient: _renew_subs 2024-04-26T00:43:14.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f23a576b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:14.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f23a576b640 1 -- 172.21.15.37:0/1544468132 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f23a011bc70 con 0x7f23a00a1890 2024-04-26T00:43:14.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f23a576b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:14.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f23a576b640 1 --2- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a013d0f0 0x7f23a013d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:14.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f23a576b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:14.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f23a576b640 1 -- 172.21.15.37:0/1544468132 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f23a01348a0 con 0x7f23a013d0f0 2024-04-26T00:43:14.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f239effd640 1 --2- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a013d0f0 0x7f23a013d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:14.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f239effd640 10 monclient: get_auth_request con 0x7f23a013d0f0 auth_method 0 2024-04-26T00:43:14.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f239effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:14.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f239effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:14.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f239effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:14.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f239effd640 1 -- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a013d0f0 msgr2=0x7f23a013d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:43:14.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f239effd640 1 -- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a013d0f0 msgr2=0x7f23a013d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:43:14.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f239effd640 1 --2- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a013d0f0 0x7f23a013d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:43:14.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f23977fe640 1 -- 172.21.15.37:0/1544468132 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f238801a070 con 0x7f23a00a1890 2024-04-26T00:43:14.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f239effd640 1 --2- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a013d0f0 0x7f23a013d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T00:43:14.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f23977fe640 1 --2- 172.21.15.37:0/1544468132 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2380035f00 0x7f23800383c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:14.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f23977fe640 1 -- 172.21.15.37:0/1544468132 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(899..899 src has 822..899) ==== 3550+0+0 (secure 0 0 0) 0x7f2388048270 con 0x7f23a00a1890 2024-04-26T00:43:14.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f239f7fe640 1 --2- 172.21.15.37:0/1544468132 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2380035f00 0x7f23800383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:14.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:13.999+0000 7f239f7fe640 10 monclient: get_auth_request con 0x7f2380035f00 auth_method 0 2024-04-26T00:43:14.017 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:14.003+0000 7f239f7fe640 1 --2- 172.21.15.37:0/1544468132 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2380035f00 0x7f23800383c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f23a011cc10 tx=0x7f2398009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:14.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:14.203+0000 7f239effd640 1 --2- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a013d0f0 0x7f23a013d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:14.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:14.203+0000 7f239effd640 10 monclient: get_auth_request con 0x7f23a013d0f0 auth_method 0 2024-04-26T00:43:14.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:14.203+0000 7f239effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:14.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:14.203+0000 7f239effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:14.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:14.203+0000 7f239effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:43:14.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:14.203+0000 7f239effd640 1 -- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a013d0f0 msgr2=0x7f23a013d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:43:14.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:14.203+0000 7f239effd640 1 -- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a013d0f0 msgr2=0x7f23a013d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:43:14.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:14.203+0000 7f239effd640 1 --2- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a013d0f0 0x7f23a013d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:43:14.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:14.203+0000 7f239effd640 1 --2- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a013d0f0 0x7f23a013d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T00:43:14.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:14.607+0000 7f239effd640 1 --2- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a013d0f0 0x7f23a013d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:14.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:14.607+0000 7f239effd640 10 monclient: get_auth_request con 0x7f23a013d0f0 auth_method 0 2024-04-26T00:43:14.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:14.607+0000 7f239effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:14.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:14.607+0000 7f239effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:14.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:14.607+0000 7f239effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:43:14.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:14.607+0000 7f239effd640 1 -- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a013d0f0 msgr2=0x7f23a013d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:43:14.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:14.607+0000 7f239effd640 1 -- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a013d0f0 msgr2=0x7f23a013d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:43:14.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:14.607+0000 7f239effd640 1 --2- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a013d0f0 0x7f23a013d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:43:14.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:14.607+0000 7f239effd640 1 --2- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a013d0f0 0x7f23a013d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T00:43:15.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:15.411+0000 7f239effd640 1 --2- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a013d0f0 0x7f23a013d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:15.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:15.411+0000 7f239effd640 10 monclient: get_auth_request con 0x7f23a013d0f0 auth_method 0 2024-04-26T00:43:15.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:15.411+0000 7f239effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:15.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:15.411+0000 7f239effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:15.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:15.411+0000 7f239effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:43:15.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:15.411+0000 7f239effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:15.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:15.411+0000 7f239effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:15.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:15.411+0000 7f239effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:15.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:15.411+0000 7f239effd640 1 -- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a013d0f0 msgr2=0x7f23a013d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T00:43:15.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:15.411+0000 7f239effd640 1 -- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a013d0f0 msgr2=0x7f23a013d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:43:15.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:15.411+0000 7f239effd640 1 --2- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a013d0f0 0x7f23a013d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:43:15.412 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:15.411+0000 7f239effd640 1 --2- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a013d0f0 0x7f23a013d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T00:43:16.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:16.995+0000 7f23967fc640 10 monclient: tick 2024-04-26T00:43:16.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:16.995+0000 7f23967fc640 10 monclient: _check_auth_tickets 2024-04-26T00:43:16.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:16.995+0000 7f23967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:16.998 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:16.995+0000 7f23967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:17.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.015+0000 7f239effd640 1 --2- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a013d0f0 0x7f23a013d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:17.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.015+0000 7f239effd640 10 monclient: get_auth_request con 0x7f23a013d0f0 auth_method 0 2024-04-26T00:43:17.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.015+0000 7f239effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:17.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.015+0000 7f239effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:17.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.015+0000 7f239effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:43:17.015 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.015+0000 7f239effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:17.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.015+0000 7f239effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:17.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.015+0000 7f239effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:17.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.015+0000 7f239effd640 10 monclient(hunting): handle_auth_done global_id 284100 payload 824 2024-04-26T00:43:17.016 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.015+0000 7f239effd640 1 -- 172.21.15.37:0/1544468132 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f23a013da10 con 0x7f23a013d0f0 2024-04-26T00:43:17.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.015+0000 7f239effd640 1 -- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a013d0f0 msgr2=0x7f23a013d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:17.018 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.015+0000 7f239effd640 1 --2- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a013d0f0 0x7f23a013d4d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f23900098a0 tx=0x7f23900190a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:17.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.019+0000 7f23977fe640 1 -- 172.21.15.37:0/1544468132 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23900b0040 con 0x7f23a013d0f0 2024-04-26T00:43:17.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.019+0000 7f23977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:17.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.019+0000 7f23977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:43:17.019 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.019+0000 7f23977fe640 10 monclient: dump: 2024-04-26T00:43:17.019 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:17.019 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:17.019 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:17.019 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:17.019 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:17.019 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:17.019 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:17.019 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:17.020 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:17.020 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:17.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.019+0000 7f23977fe640 1 -- 172.21.15.37:0/1544468132 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f23900b1040 con 0x7f23a013d0f0 2024-04-26T00:43:17.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.019+0000 7f23977fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:17.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.019+0000 7f23977fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:43:17.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.019+0000 7f23977fe640 1 -- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a013d0f0 msgr2=0x7f23a013d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:17.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.019+0000 7f23977fe640 1 --2- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a013d0f0 0x7f23a013d4d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f23900098a0 tx=0x7f23900190a0 comp rx=0 tx=0).stop 2024-04-26T00:43:17.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.019+0000 7f23977fe640 1 -- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a013d0f0 msgr2=0x7f23a013d4d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:43:17.020 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.019+0000 7f23977fe640 1 --2- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a013d0f0 0x7f23a013d4d0 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f23900098a0 tx=0x7f23900190a0 comp rx=0 tx=0).stop 2024-04-26T00:43:17.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.027+0000 7f23a576b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:43:17.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.027+0000 7f23a576b640 1 --2- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a0138dc0 0x7f23a01391a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:17.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.027+0000 7f23a576b640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:17.027 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.027+0000 7f23a576b640 1 -- 172.21.15.37:0/1544468132 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f23a004f510 con 0x7f23a0138dc0 2024-04-26T00:43:17.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.027+0000 7f239effd640 1 --2- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a0138dc0 0x7f23a01391a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:17.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.027+0000 7f239effd640 10 monclient: get_auth_request con 0x7f23a0138dc0 auth_method 0 2024-04-26T00:43:17.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.027+0000 7f239effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:17.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.027+0000 7f239effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:17.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.027+0000 7f239effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:17.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.027+0000 7f239effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:17.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.027+0000 7f239effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:17.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.027+0000 7f239effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:17.028 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.027+0000 7f239effd640 10 monclient(hunting): handle_auth_done global_id 284103 payload 824 2024-04-26T00:43:17.029 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.027+0000 7f239effd640 1 -- 172.21.15.37:0/1544468132 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f23a00a4880 con 0x7f23a0138dc0 2024-04-26T00:43:17.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.031+0000 7f239effd640 1 -- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a0138dc0 msgr2=0x7f23a01391a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:17.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.031+0000 7f239effd640 1 --2- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a0138dc0 0x7f23a01391a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f23a0136610 tx=0x7f23900a8000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:17.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.031+0000 7f23977fe640 1 -- 172.21.15.37:0/1544468132 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23900b0040 con 0x7f23a0138dc0 2024-04-26T00:43:17.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.031+0000 7f23977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:17.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.031+0000 7f23977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:43:17.031 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.031+0000 7f23977fe640 10 monclient: dump: 2024-04-26T00:43:17.032 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:17.032 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:17.032 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:17.032 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:17.032 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:17.032 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:17.032 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:17.032 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:17.032 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:17.032 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:17.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.031+0000 7f23977fe640 1 -- 172.21.15.37:0/1544468132 <== mon.0 v2:172.21.15.37:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4547 (secure 0 0 0) 0x7f23900b6d50 con 0x7f23a0138dc0 2024-04-26T00:43:17.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.031+0000 7f23977fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:43:17.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.031+0000 7f23977fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:43:17.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.031+0000 7f23977fe640 1 -- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a0138dc0 msgr2=0x7f23a01391a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:17.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.031+0000 7f23977fe640 1 --2- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a0138dc0 0x7f23a01391a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f23a0136610 tx=0x7f23900a8000 comp rx=0 tx=0).stop 2024-04-26T00:43:17.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.031+0000 7f23977fe640 1 -- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a0138dc0 msgr2=0x7f23a01391a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:43:17.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.031+0000 7f23977fe640 1 --2- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f23a0138dc0 0x7f23a01391a0 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f23a0136610 tx=0x7f23900a8000 comp rx=0 tx=0).stop 2024-04-26T00:43:17.034 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:43:17.034 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:43:17.034 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:43:17.034 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:43:17.034 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 166, 2024-04-26T00:43:17.034 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 16651, 2024-04-26T00:43:17.034 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:43:17.034 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:43:17.034 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:43:17.034 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:43:17.034 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:43:17.034 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 3, 2024-04-26T00:43:17.035 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:43:17.035 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:43:17.035 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:43:17.035 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:43:17.035 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:43:17.035 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:43:17.035 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:43:17.035 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:43:17.035 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:43:17.035 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:43:17.035 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:43:17.035 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:43:17.035 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:43:17.035 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:43:17.035 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:43:17.036 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:43:17.036 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:43:17.036 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:43:17.036 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:43:17.036 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:43:17.036 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:43:17.036 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:43:17.036 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:43:17.036 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:43:17.036 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:43:17.036 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:43:17.036 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:43:17.036 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:43:17.036 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:43:17.037 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:43:17.037 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:43:17.037 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:43:17.037 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:43:17.037 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:43:17.037 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:43:17.037 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:43:17.037 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:43:17.037 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:43:17.037 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:43:17.037 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:43:17.037 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:43:17.037 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:43:17.037 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:43:17.037 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:43:17.037 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:43:17.037 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:43:17.038 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:43:17.038 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:43:17.038 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:43:17.038 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:43:17.038 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:43:17.038 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:43:17.038 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:43:17.038 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:43:17.038 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:43:17.038 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:43:17.038 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:43:17.038 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:43:17.038 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:43:17.038 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:43:17.039 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:43:17.039 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:43:17.039 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:43:17.039 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:43:17.039 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:43:17.039 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:43:17.039 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:43:17.039 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:43:17.039 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:43:17.039 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:43:17.039 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:43:17.039 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:43:17.039 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:43:17.039 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:43:17.040 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:43:17.040 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:43:17.040 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:43:17.040 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:43:17.040 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:43:17.040 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:43:17.040 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:43:17.040 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:43:17.040 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:43:17.040 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:43:17.040 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:43:17.040 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:43:17.040 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:43:17.040 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:43:17.041 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:43:17.041 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:43:17.041 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:43:17.041 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:43:17.041 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:43:17.041 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:43:17.041 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:43:17.041 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:43:17.041 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:43:17.041 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:43:17.041 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:43:17.041 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:43:17.041 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:43:17.041 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:43:17.042 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:43:17.042 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:43:17.042 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:43:17.042 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:43:17.042 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:43:17.042 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:43:17.042 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:43:17.042 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:43:17.042 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:43:17.042 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:43:17.042 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:43:17.042 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:43:17.042 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:43:17.042 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:43:17.042 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:43:17.042 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:43:17.043 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:43:17.043 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:43:17.043 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:43:17.043 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:43:17.043 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:43:17.043 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:43:17.043 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:43:17.043 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:43:17.043 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:43:17.043 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:43:17.043 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:43:17.043 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:43:17.043 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:43:17.043 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:43:17.043 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:43:17.044 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:43:17.044 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:43:17.044 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:43:17.044 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:43:17.044 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:43:17.044 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:43:17.044 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:43:17.044 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:43:17.044 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:43:17.044 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:43:17.044 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:43:17.044 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:43:17.044 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:43:17.044 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:43:17.044 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:43:17.045 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:43:17.045 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:43:17.045 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:43:17.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.031+0000 7f23a576b640 1 -- 172.21.15.37:0/1544468132 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2380035f00 msgr2=0x7f23800383c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:17.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.031+0000 7f23a576b640 1 --2- 172.21.15.37:0/1544468132 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2380035f00 0x7f23800383c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f23a011cc10 tx=0x7f2398009040 comp rx=0 tx=0).stop 2024-04-26T00:43:17.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.031+0000 7f23a576b640 10 monclient: shutdown 2024-04-26T00:43:17.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.031+0000 7f23a576b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:17.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.031+0000 7f23a576b640 1 -- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f23a00a1890 msgr2=0x7f23a0132a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:17.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.031+0000 7f23a576b640 1 --2- 172.21.15.37:0/1544468132 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f23a00a1890 0x7f23a0132a70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f238800e840 tx=0x7f238800ed00 comp rx=0 tx=0).stop 2024-04-26T00:43:17.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.031+0000 7f239f7fe640 1 -- 172.21.15.37:0/1544468132 reap_dead start 2024-04-26T00:43:17.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.031+0000 7f239f7fe640 1 -- 172.21.15.37:0/1544468132 reap_dead start 2024-04-26T00:43:17.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.031+0000 7f23a576b640 1 -- 172.21.15.37:0/1544468132 shutdown_connections 2024-04-26T00:43:17.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.031+0000 7f23a576b640 1 -- 172.21.15.37:0/1544468132 >> 172.21.15.37:0/1544468132 conn(0x7f23a00a1d50 msgr2=0x7f23a00a4a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:17.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.031+0000 7f23a576b640 1 -- 172.21.15.37:0/1544468132 shutdown_connections 2024-04-26T00:43:17.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.031+0000 7f23a576b640 1 -- 172.21.15.37:0/1544468132 wait complete. 2024-04-26T00:43:17.058 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:43:17.058 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T00:43:17.058 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T00:43:17.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cfe69640 10 monclient: get_monmap_and_config 2024-04-26T00:43:17.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cfe69640 10 monclient: build_initial_monmap 2024-04-26T00:43:17.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cfe69640 10 monclient: monmap: 2024-04-26T00:43:17.188 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:17.188 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:17.188 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:43:17.188762+0000 2024-04-26T00:43:17.188 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:43:17.188762+0000 2024-04-26T00:43:17.188 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:17.188 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:17.188 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:43:17.188 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:43:17.188 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:43:17.188 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:43:17.188 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:43:17.189 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:17.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cfe69640 1 Processor -- start 2024-04-26T00:43:17.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cfe69640 1 -- start start 2024-04-26T00:43:17.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cfe69640 10 monclient: init 2024-04-26T00:43:17.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cfe69640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:17.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cfe69640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-04-26T00:43:17.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cfe69640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fe2c8130eb0 0x7fe2c809ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:17.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cfe69640 10 monclient(hunting): picked mon.noname-c con 0x7fe2c8130eb0 addr v2:172.21.15.45:3300/0 2024-04-26T00:43:17.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cfe69640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2c8131290 0x7fe2c8133b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:17.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cfe69640 10 monclient(hunting): picked mon.noname-b con 0x7fe2c8131290 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:17.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cfe69640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe2c81340d0 addr v1:172.21.15.37:6789/0 2024-04-26T00:43:17.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cfe69640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2c809c190 con 0x7fe2c81340d0 2024-04-26T00:43:17.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cfe69640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:17.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cfe69640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fe2c809df60 con 0x7fe2c8130eb0 2024-04-26T00:43:17.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cfe69640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:17.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cfe69640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe2c80a11c0 con 0x7fe2c8131290 2024-04-26T00:43:17.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cfe69640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:17.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cfe69640 10 monclient(hunting): authenticate will time out at 2378.494902s 2024-04-26T00:43:17.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cd404640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2c8131290 0x7fe2c8133b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:17.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2ce406640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fe2c81340d0 0x7fe2c813aec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:43836/0 (socket says 172.21.15.37:43836) 2024-04-26T00:43:17.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2ce406640 1 -- 172.21.15.37:0/171308288 learned_addr learned my addr 172.21.15.37:0/171308288 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:43:17.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cd404640 10 monclient(hunting): get_auth_request con 0x7fe2c8131290 auth_method 0 2024-04-26T00:43:17.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cd404640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:17.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cd404640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:17.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cd404640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:17.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cd404640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:17.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cd404640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:17.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cd404640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:17.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cdc05640 1 --2- 172.21.15.37:0/171308288 >> v2:172.21.15.45:3300/0 conn(0x7fe2c8130eb0 0x7fe2c809ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:17.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cdc05640 10 monclient(hunting): get_auth_request con 0x7fe2c8130eb0 auth_method 0 2024-04-26T00:43:17.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cdc05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:17.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cdc05640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:17.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cdc05640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:17.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cd404640 10 monclient(hunting): handle_auth_done global_id 254429 payload 274 2024-04-26T00:43:17.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cd404640 1 -- 172.21.15.37:0/171308288 >> v2:172.21.15.45:3300/0 conn(0x7fe2c8130eb0 msgr2=0x7fe2c809ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:17.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cd404640 1 --2- 172.21.15.37:0/171308288 >> v2:172.21.15.45:3300/0 conn(0x7fe2c8130eb0 0x7fe2c809ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:17.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cd404640 1 -- 172.21.15.37:0/171308288 >> v1:172.21.15.37:6789/0 conn(0x7fe2c81340d0 legacy=0x7fe2c813aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:17.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cd404640 10 monclient: _finish_hunting 0 2024-04-26T00:43:17.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cd404640 1 monclient: found mon.noname-b 2024-04-26T00:43:17.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cd404640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:17.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cd404640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:43:17.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.187+0000 7fe2cd404640 1 -- 172.21.15.37:0/171308288 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2c809dc80 con 0x7fe2c8131290 2024-04-26T00:43:17.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.191+0000 7fe2cd404640 1 -- 172.21.15.37:0/171308288 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2c8131290 msgr2=0x7fe2c8133b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:17.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cd404640 1 --2- 172.21.15.37:0/171308288 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2c8131290 0x7fe2c8133b90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe2b80097c0 tx=0x7fe2b801c320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9e3f932c46ea225 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:17.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2ccc03640 1 -- 172.21.15.37:0/171308288 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b80b0070 con 0x7fe2c8131290 2024-04-26T00:43:17.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2ccc03640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:17.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2ccc03640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:43:17.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2ccc03640 10 monclient: dump: 2024-04-26T00:43:17.196 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:17.196 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:17.196 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:17.196 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:17.196 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:17.196 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:17.197 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:17.197 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:17.197 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:17.197 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:17.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2ccc03640 10 monclient: _finish_auth 0 2024-04-26T00:43:17.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2ccc03640 10 monclient: _check_auth_tickets 2024-04-26T00:43:17.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2ccc03640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:17.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cfe69640 5 monclient: authenticate success, global_id 254429 2024-04-26T00:43:17.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cfe69640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:43:17.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2ccc03640 1 -- 172.21.15.37:0/171308288 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2b80ab600 con 0x7fe2c8131290 2024-04-26T00:43:17.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2ccc03640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:17.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cfe69640 10 monclient: get_monmap_and_config success 2024-04-26T00:43:17.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2ccc03640 1 -- 172.21.15.37:0/171308288 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b80ab9d0 con 0x7fe2c8131290 2024-04-26T00:43:17.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2ccc03640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:17.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cfe69640 4 set_mon_vals no callback set 2024-04-26T00:43:17.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2ccc03640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:43:17.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2ccc03640 10 monclient: dump: 2024-04-26T00:43:17.198 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:17.198 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:17.198 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:17.198 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:17.198 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:17.198 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:17.198 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:17.198 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:17.198 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:17.198 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:17.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cfe69640 10 monclient: shutdown 2024-04-26T00:43:17.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cfe69640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:17.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cfe69640 1 -- 172.21.15.37:0/171308288 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2c8131290 msgr2=0x7fe2c8133b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:17.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cfe69640 1 --2- 172.21.15.37:0/171308288 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2c8131290 0x7fe2c8133b90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe2b80097c0 tx=0x7fe2b801c320 comp rx=0 tx=0).stop 2024-04-26T00:43:17.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cfe69640 1 -- 172.21.15.37:0/171308288 shutdown_connections 2024-04-26T00:43:17.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cfe69640 1 --2- 172.21.15.37:0/171308288 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2c8131290 0x7fe2c8133b90 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:17.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cfe69640 1 --2- 172.21.15.37:0/171308288 >> v2:172.21.15.45:3300/0 conn(0x7fe2c8130eb0 0x7fe2c809ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:17.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cfe69640 1 -- 172.21.15.37:0/171308288 >> 172.21.15.37:0/171308288 conn(0x7fe2c80a1d50 msgr2=0x7fe2c80a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:17.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cfe69640 1 -- 172.21.15.37:0/171308288 shutdown_connections 2024-04-26T00:43:17.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cfe69640 1 -- 172.21.15.37:0/171308288 wait complete. 2024-04-26T00:43:17.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cfe69640 10 monclient: build_initial_monmap 2024-04-26T00:43:17.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cfe69640 10 monclient: monmap: 2024-04-26T00:43:17.199 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:17.199 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:43:17.199 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:43:17.199 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:43:17.199 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:17.199 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:43:17.200 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:43:17.200 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:43:17.200 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:43:17.200 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:17.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cfe69640 1 Processor -- start 2024-04-26T00:43:17.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cfe69640 1 -- start start 2024-04-26T00:43:17.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cfe69640 10 monclient: init 2024-04-26T00:43:17.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cfe69640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:17.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cfe69640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:43:17.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cfe69640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2c8130eb0 0x7fe2c81cf810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:17.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cfe69640 10 monclient(hunting): picked mon.noname-b con 0x7fe2c8130eb0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:43:17.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cfe69640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2c8131290 0x7fe2c81cfd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:17.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cfe69640 10 monclient(hunting): picked mon.noname-c con 0x7fe2c8131290 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:17.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cfe69640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2c81340d0 0x7fe2c80713f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:17.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cfe69640 10 monclient(hunting): picked mon.noname-a con 0x7fe2c81340d0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:43:17.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cfe69640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:17.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cfe69640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe2c81317f0 con 0x7fe2c81340d0 2024-04-26T00:43:17.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cfe69640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:17.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cfe69640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe2c8131670 con 0x7fe2c8130eb0 2024-04-26T00:43:17.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cfe69640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:17.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cfe69640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe2c813cdf0 con 0x7fe2c8131290 2024-04-26T00:43:17.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cfe69640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:17.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cfe69640 10 monclient(hunting): authenticate will time out at 2378.502508s 2024-04-26T00:43:17.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2ce406640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2c81340d0 0x7fe2c80713f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:17.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cd404640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2c8131290 0x7fe2c81cfd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:17.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2ce406640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2c81340d0 0x7fe2c80713f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:54076/0 (socket says 172.21.15.37:54076) 2024-04-26T00:43:17.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cd404640 10 monclient(hunting): get_auth_request con 0x7fe2c8131290 auth_method 0 2024-04-26T00:43:17.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2ce406640 1 -- 172.21.15.37:0/748337172 learned_addr learned my addr 172.21.15.37:0/748337172 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:17.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cd404640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:17.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cd404640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:17.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cd404640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:17.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2ce406640 10 monclient(hunting): get_auth_request con 0x7fe2c81340d0 auth_method 0 2024-04-26T00:43:17.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2ce406640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:17.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2ce406640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:17.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2ce406640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:17.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2ce406640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:17.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2ce406640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:17.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2ce406640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:17.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cd404640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:17.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cd404640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:17.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cd404640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:17.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cdc05640 1 --2- 172.21.15.37:0/748337172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2c8130eb0 0x7fe2c81cf810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:17.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cdc05640 10 monclient(hunting): get_auth_request con 0x7fe2c8130eb0 auth_method 0 2024-04-26T00:43:17.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cdc05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:17.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cdc05640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:17.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2cdc05640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:17.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2ce406640 10 monclient(hunting): handle_auth_done global_id 284106 payload 824 2024-04-26T00:43:17.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2ce406640 1 -- 172.21.15.37:0/748337172 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2c8131290 msgr2=0x7fe2c81cfd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:17.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2ce406640 1 --2- 172.21.15.37:0/748337172 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2c8131290 0x7fe2c81cfd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:17.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2ce406640 1 -- 172.21.15.37:0/748337172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2c8130eb0 msgr2=0x7fe2c81cf810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:17.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2ce406640 1 --2- 172.21.15.37:0/748337172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2c8130eb0 0x7fe2c81cf810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:17.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2ce406640 10 monclient: _finish_hunting 0 2024-04-26T00:43:17.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2ce406640 1 monclient: found mon.noname-a 2024-04-26T00:43:17.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2ce406640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:17.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2ce406640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:43:17.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.195+0000 7fe2ce406640 1 -- 172.21.15.37:0/748337172 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2c8133a20 con 0x7fe2c81340d0 2024-04-26T00:43:17.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2ce406640 1 -- 172.21.15.37:0/748337172 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2c81340d0 msgr2=0x7fe2c80713f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:17.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2ce406640 1 --2- 172.21.15.37:0/748337172 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2c81340d0 0x7fe2c80713f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe2c000caf0 tx=0x7fe2c000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:17.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2be7fc640 1 -- 172.21.15.37:0/748337172 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2c0012400 con 0x7fe2c81340d0 2024-04-26T00:43:17.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2be7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:17.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2be7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:43:17.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2be7fc640 10 monclient: dump: 2024-04-26T00:43:17.209 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:17.209 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:17.209 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:17.209 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:17.210 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:17.210 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:17.210 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:17.210 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:17.210 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:17.210 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:17.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2be7fc640 10 monclient: _finish_auth 0 2024-04-26T00:43:17.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2be7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:43:17.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:17.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2be7fc640 1 -- 172.21.15.37:0/748337172 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2c00125c0 con 0x7fe2c81340d0 2024-04-26T00:43:17.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2cfe69640 5 monclient: authenticate success, global_id 284106 2024-04-26T00:43:17.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2cfe69640 10 monclient: _renew_subs 2024-04-26T00:43:17.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2cfe69640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:43:17.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2cfe69640 1 -- 172.21.15.37:0/748337172 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2c80a1470 con 0x7fe2c81340d0 2024-04-26T00:43:17.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2be7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:17.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2be7fc640 1 -- 172.21.15.37:0/748337172 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2c00129b0 con 0x7fe2c81340d0 2024-04-26T00:43:17.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2be7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:17.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2be7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:43:17.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2be7fc640 10 monclient: dump: 2024-04-26T00:43:17.211 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:17.211 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:17.211 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:17.211 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:17.211 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:17.211 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:17.211 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:17.211 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:17.211 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:17.211 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:17.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2bffff640 4 set_mon_vals no callback set 2024-04-26T00:43:17.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2cfe69640 10 monclient: _renew_subs 2024-04-26T00:43:17.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2cfe69640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:43:17.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2cfe69640 1 -- 172.21.15.37:0/748337172 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2c813e430 con 0x7fe2c81340d0 2024-04-26T00:43:17.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2cfe69640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:17.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2cfe69640 1 --2- 172.21.15.37:0/748337172 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2c8071930 0x7fe2c8071d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:17.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2cfe69640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:17.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2cfe69640 1 -- 172.21.15.37:0/748337172 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe2c804f510 con 0x7fe2c8071930 2024-04-26T00:43:17.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2be7fc640 1 -- 172.21.15.37:0/748337172 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe2c0012bd0 con 0x7fe2c81340d0 2024-04-26T00:43:17.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2cdc05640 1 --2- 172.21.15.37:0/748337172 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2c8071930 0x7fe2c8071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:17.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2cdc05640 10 monclient: get_auth_request con 0x7fe2c8071930 auth_method 0 2024-04-26T00:43:17.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2cdc05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:17.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2cdc05640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:17.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2cdc05640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:17.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2be7fc640 1 --2- 172.21.15.37:0/748337172 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe2ac035e80 0x7fe2ac038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:17.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2cdc05640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:17.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2cdc05640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:17.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2cdc05640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:17.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2be7fc640 1 -- 172.21.15.37:0/748337172 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(899..899 src has 822..899) ==== 3550+0+0 (secure 0 0 0) 0x7fe2c0048fd0 con 0x7fe2c81340d0 2024-04-26T00:43:17.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2cdc05640 10 monclient(hunting): handle_auth_done global_id 254435 payload 824 2024-04-26T00:43:17.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2cdc05640 1 -- 172.21.15.37:0/748337172 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe2c80a0f60 con 0x7fe2c8071930 2024-04-26T00:43:17.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2cd404640 1 --2- 172.21.15.37:0/748337172 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe2ac035e80 0x7fe2ac038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:17.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.199+0000 7fe2cd404640 10 monclient: get_auth_request con 0x7fe2ac035e80 auth_method 0 2024-04-26T00:43:17.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.203+0000 7fe2cdc05640 1 -- 172.21.15.37:0/748337172 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2c8071930 msgr2=0x7fe2c8071d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:17.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.203+0000 7fe2cdc05640 1 --2- 172.21.15.37:0/748337172 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2c8071930 0x7fe2c8071d10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe2c400f4b0 tx=0x7fe2c400f970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:17.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.203+0000 7fe2be7fc640 1 -- 172.21.15.37:0/748337172 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2c4019020 con 0x7fe2c8071930 2024-04-26T00:43:17.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.203+0000 7fe2be7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:17.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.203+0000 7fe2be7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:43:17.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.203+0000 7fe2be7fc640 10 monclient: dump: 2024-04-26T00:43:17.213 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:17.213 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:17.214 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:17.214 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:17.214 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:17.214 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:17.215 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:17.215 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:17.215 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:17.215 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:17.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.203+0000 7fe2be7fc640 1 -- 172.21.15.37:0/748337172 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe2c4014a80 con 0x7fe2c8071930 2024-04-26T00:43:17.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.203+0000 7fe2be7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:17.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.203+0000 7fe2be7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:43:17.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.203+0000 7fe2be7fc640 1 -- 172.21.15.37:0/748337172 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2c8071930 msgr2=0x7fe2c8071d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:17.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.203+0000 7fe2be7fc640 1 --2- 172.21.15.37:0/748337172 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2c8071930 0x7fe2c8071d10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe2c400f4b0 tx=0x7fe2c400f970 comp rx=0 tx=0).stop 2024-04-26T00:43:17.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.203+0000 7fe2be7fc640 1 -- 172.21.15.37:0/748337172 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2c8071930 msgr2=0x7fe2c8071d10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:43:17.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.203+0000 7fe2be7fc640 1 --2- 172.21.15.37:0/748337172 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2c8071930 0x7fe2c8071d10 secure :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe2c400f4b0 tx=0x7fe2c400f970 comp rx=0 tx=0).stop 2024-04-26T00:43:17.215 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.203+0000 7fe2cd404640 1 --2- 172.21.15.37:0/748337172 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe2ac035e80 0x7fe2ac038340 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fe2b80097c0 tx=0x7fe2b8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:17.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.207+0000 7fe2cfe69640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:43:17.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.207+0000 7fe2cfe69640 1 --2- 172.21.15.37:0/748337172 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2c8072250 0x7fe2c806dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:17.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.207+0000 7fe2cfe69640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:17.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.207+0000 7fe2cfe69640 1 -- 172.21.15.37:0/748337172 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe2c806ba50 con 0x7fe2c8072250 2024-04-26T00:43:17.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.211+0000 7fe2cdc05640 1 --2- 172.21.15.37:0/748337172 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2c8072250 0x7fe2c806dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:17.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.211+0000 7fe2cdc05640 10 monclient: get_auth_request con 0x7fe2c8072250 auth_method 0 2024-04-26T00:43:17.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.211+0000 7fe2cdc05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:17.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.211+0000 7fe2cdc05640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:17.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.211+0000 7fe2cdc05640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:17.217 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.211+0000 7fe2cdc05640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:17.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.211+0000 7fe2cdc05640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:17.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.211+0000 7fe2cdc05640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:17.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.211+0000 7fe2cdc05640 10 monclient(hunting): handle_auth_done global_id 254438 payload 824 2024-04-26T00:43:17.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.211+0000 7fe2cdc05640 1 -- 172.21.15.37:0/748337172 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe2c8139ea0 con 0x7fe2c8072250 2024-04-26T00:43:17.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.211+0000 7fe2cdc05640 1 -- 172.21.15.37:0/748337172 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2c8072250 msgr2=0x7fe2c806dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:17.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.211+0000 7fe2cdc05640 1 --2- 172.21.15.37:0/748337172 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2c8072250 0x7fe2c806dc40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe2c401a000 tx=0x7fe2c40046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:17.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.211+0000 7fe2be7fc640 1 -- 172.21.15.37:0/748337172 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2c4019070 con 0x7fe2c8072250 2024-04-26T00:43:17.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.211+0000 7fe2be7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:17.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.211+0000 7fe2be7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:43:17.218 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.211+0000 7fe2be7fc640 10 monclient: dump: 2024-04-26T00:43:17.218 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:17.218 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:17.218 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:17.218 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:17.219 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:17.219 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:17.219 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:17.219 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:17.219 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:17.219 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:17.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.211+0000 7fe2be7fc640 1 -- 172.21.15.37:0/748337172 <== mon.2 v2:172.21.15.37:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4890 (secure 0 0 0) 0x7fe2c4004230 con 0x7fe2c8072250 2024-04-26T00:43:17.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.211+0000 7fe2be7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:43:17.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.211+0000 7fe2be7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:43:17.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.211+0000 7fe2be7fc640 1 -- 172.21.15.37:0/748337172 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2c8072250 msgr2=0x7fe2c806dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:17.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.211+0000 7fe2be7fc640 1 --2- 172.21.15.37:0/748337172 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2c8072250 0x7fe2c806dc40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe2c401a000 tx=0x7fe2c40046d0 comp rx=0 tx=0).stop 2024-04-26T00:43:17.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.211+0000 7fe2be7fc640 1 -- 172.21.15.37:0/748337172 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2c8072250 msgr2=0x7fe2c806dc40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:43:17.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.211+0000 7fe2be7fc640 1 --2- 172.21.15.37:0/748337172 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe2c8072250 0x7fe2c806dc40 secure :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe2c401a000 tx=0x7fe2c40046d0 comp rx=0 tx=0).stop 2024-04-26T00:43:17.220 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:43:17.220 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:43:17.220 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:43:17.220 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "leader", 2024-04-26T00:43:17.220 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 166, 2024-04-26T00:43:17.220 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 165981, 2024-04-26T00:43:17.220 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:43:17.220 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:43:17.220 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:43:17.220 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:43:17.220 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:43:17.220 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 4, 2024-04-26T00:43:17.220 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:43:17.221 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:43:17.221 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:43:17.221 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:43:17.221 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:43:17.221 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:43:17.221 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:43:17.221 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:43:17.221 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:43:17.221 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:43:17.221 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:43:17.221 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:43:17.221 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:43:17.221 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:43:17.221 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:43:17.221 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:43:17.222 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:43:17.222 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:43:17.222 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:43:17.222 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:43:17.222 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:43:17.222 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:43:17.222 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:43:17.222 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:43:17.222 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:43:17.222 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:43:17.222 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:43:17.222 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:43:17.222 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:43:17.222 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:43:17.223 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:43:17.223 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:43:17.223 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:43:17.223 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:43:17.223 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:43:17.223 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:43:17.223 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:43:17.223 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:43:17.223 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:43:17.223 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:43:17.224 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:43:17.224 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:43:17.224 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:43:17.224 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:43:17.224 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:43:17.224 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:43:17.224 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:43:17.224 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:43:17.224 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:43:17.224 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:43:17.224 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:43:17.224 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:43:17.224 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:43:17.224 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:43:17.224 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:43:17.225 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:43:17.225 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:43:17.225 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:43:17.225 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:43:17.225 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:43:17.225 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:43:17.225 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:43:17.225 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:43:17.225 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:43:17.225 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:43:17.225 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:43:17.225 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:43:17.225 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:43:17.225 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:43:17.225 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:43:17.226 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:43:17.226 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:43:17.226 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:43:17.226 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:43:17.226 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:43:17.226 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:43:17.226 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:43:17.226 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:43:17.226 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:43:17.226 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:43:17.226 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:43:17.226 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:43:17.226 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:43:17.226 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:43:17.226 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:43:17.227 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:43:17.227 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:43:17.227 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:43:17.227 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:43:17.227 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:43:17.227 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:43:17.227 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:43:17.227 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:43:17.227 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:43:17.227 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:43:17.227 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:43:17.227 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:43:17.227 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:43:17.227 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:43:17.227 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:43:17.228 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:43:17.228 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:43:17.228 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:43:17.228 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:43:17.228 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:43:17.228 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:43:17.228 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:43:17.228 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:43:17.228 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:43:17.228 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:43:17.228 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:43:17.228 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:43:17.228 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:43:17.228 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:43:17.228 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:43:17.229 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:43:17.229 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:43:17.229 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:43:17.229 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:43:17.229 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:43:17.229 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:43:17.229 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:43:17.229 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:43:17.229 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:43:17.229 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:43:17.229 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:43:17.229 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:43:17.229 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:43:17.229 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:43:17.229 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:43:17.230 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:43:17.230 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:43:17.230 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:43:17.230 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:43:17.230 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:43:17.230 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:43:17.230 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:43:17.230 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:43:17.230 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:43:17.230 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:43:17.230 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:43:17.230 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:43:17.230 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:43:17.230 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:43:17.231 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:43:17.231 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:43:17.231 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:43:17.231 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:43:17.231 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:43:17.231 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:43:17.231 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:43:17.231 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:43:17.231 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 5 2024-04-26T00:43:17.231 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:43:17.231 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:43:17.231 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr": [ 2024-04-26T00:43:17.231 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:43:17.231 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:43:17.231 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:43:17.232 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:43:17.232 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:43:17.232 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:43:17.232 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:43:17.232 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:43:17.232 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:43:17.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.211+0000 7fe2cfe69640 1 -- 172.21.15.37:0/748337172 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe2ac035e80 msgr2=0x7fe2ac038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:17.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.211+0000 7fe2cfe69640 1 --2- 172.21.15.37:0/748337172 >> v2:172.21.15.45:6812/2379918623 conn(0x7fe2ac035e80 0x7fe2ac038340 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fe2b80097c0 tx=0x7fe2b8008040 comp rx=0 tx=0).stop 2024-04-26T00:43:17.233 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.211+0000 7fe2cfe69640 10 monclient: shutdown 2024-04-26T00:43:17.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.211+0000 7fe2cfe69640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:17.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.211+0000 7fe2cfe69640 1 -- 172.21.15.37:0/748337172 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2c81340d0 msgr2=0x7fe2c80713f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:17.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.211+0000 7fe2cfe69640 1 --2- 172.21.15.37:0/748337172 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2c81340d0 0x7fe2c80713f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe2c000caf0 tx=0x7fe2c000a060 comp rx=0 tx=0).stop 2024-04-26T00:43:17.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.211+0000 7fe2ce406640 1 -- 172.21.15.37:0/748337172 reap_dead start 2024-04-26T00:43:17.234 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.211+0000 7fe2ce406640 1 -- 172.21.15.37:0/748337172 reap_dead start 2024-04-26T00:43:17.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.211+0000 7fe2cfe69640 1 -- 172.21.15.37:0/748337172 shutdown_connections 2024-04-26T00:43:17.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.211+0000 7fe2cfe69640 1 -- 172.21.15.37:0/748337172 >> 172.21.15.37:0/748337172 conn(0x7fe2c80a1d50 msgr2=0x7fe2c8133260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:17.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.211+0000 7fe2cfe69640 1 -- 172.21.15.37:0/748337172 shutdown_connections 2024-04-26T00:43:17.235 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.211+0000 7fe2cfe69640 1 -- 172.21.15.37:0/748337172 wait complete. 2024-04-26T00:43:17.235 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:43:17.235 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T00:43:17.235 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T00:43:17.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f689eea6640 10 monclient: get_monmap_and_config 2024-04-26T00:43:17.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f689eea6640 10 monclient: build_initial_monmap 2024-04-26T00:43:17.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f689eea6640 10 monclient: monmap: 2024-04-26T00:43:17.356 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:17.356 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:17.356 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:43:17.356751+0000 2024-04-26T00:43:17.356 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:43:17.356751+0000 2024-04-26T00:43:17.356 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:17.356 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:17.356 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:43:17.356 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:43:17.356 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:43:17.356 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:43:17.357 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:43:17.357 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:17.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f689eea6640 1 Processor -- start 2024-04-26T00:43:17.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f689eea6640 1 -- start start 2024-04-26T00:43:17.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f689eea6640 10 monclient: init 2024-04-26T00:43:17.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f689eea6640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:17.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f689eea6640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-04-26T00:43:17.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f689eea6640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f68980726b0 0x7f6898072a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:17.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f689eea6640 10 monclient(hunting): picked mon.noname-c con 0x7f68980726b0 addr v2:172.21.15.45:3300/0 2024-04-26T00:43:17.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f689eea6640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f68980a0110 0x7f6898136dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:17.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f689eea6640 10 monclient(hunting): picked mon.noname-a con 0x7f68980a0110 addr v2:172.21.15.37:3300/0 2024-04-26T00:43:17.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f689eea6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6898137310 addr v1:172.21.15.45:6789/0 2024-04-26T00:43:17.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f689eea6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68980a1470 con 0x7f6898137310 2024-04-26T00:43:17.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f689eea6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:17.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f689eea6640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f68980a1bd0 con 0x7f68980a0110 2024-04-26T00:43:17.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f689eea6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:17.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f689eea6640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f68980a0eb0 con 0x7f68980726b0 2024-04-26T00:43:17.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f689eea6640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:17.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f689eea6640 10 monclient(hunting): authenticate will time out at 2378.662465s 2024-04-26T00:43:17.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f6897fff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f68980a0110 0x7f6898136dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:17.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f6897fff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f68980a0110 0x7f6898136dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:54082/0 (socket says 172.21.15.37:54082) 2024-04-26T00:43:17.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f6897fff640 1 -- 172.21.15.37:0/3053512790 learned_addr learned my addr 172.21.15.37:0/3053512790 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:17.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f6897fff640 10 monclient(hunting): get_auth_request con 0x7f68980a0110 auth_method 0 2024-04-26T00:43:17.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f6897fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:17.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f6897fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:17.360 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f6897fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:17.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f6897fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:17.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f6897fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:17.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f6897fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:17.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f689cc42640 1 --2- 172.21.15.37:0/3053512790 >> v2:172.21.15.45:3300/0 conn(0x7f68980726b0 0x7f6898072a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:17.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f689cc42640 10 monclient(hunting): get_auth_request con 0x7f68980726b0 auth_method 0 2024-04-26T00:43:17.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f689cc42640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:17.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f689cc42640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:17.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f689cc42640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:17.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f6897fff640 10 monclient(hunting): handle_auth_done global_id 284109 payload 274 2024-04-26T00:43:17.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f6897fff640 1 -- 172.21.15.37:0/3053512790 >> v2:172.21.15.45:3300/0 conn(0x7f68980726b0 msgr2=0x7f6898072a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:17.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f6897fff640 1 --2- 172.21.15.37:0/3053512790 >> v2:172.21.15.45:3300/0 conn(0x7f68980726b0 0x7f6898072a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:17.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f6897fff640 1 -- 172.21.15.37:0/3053512790 >> v1:172.21.15.45:6789/0 conn(0x7f6898137310 legacy=0x7f689813e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:17.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f6897fff640 10 monclient: _finish_hunting 0 2024-04-26T00:43:17.361 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f6897fff640 1 monclient: found mon.noname-a 2024-04-26T00:43:17.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f6897fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:17.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f6897fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:43:17.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.355+0000 7f6897fff640 1 -- 172.21.15.37:0/3053512790 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68980a0bd0 con 0x7f68980a0110 2024-04-26T00:43:17.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.359+0000 7f6897fff640 1 -- 172.21.15.37:0/3053512790 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f68980a0110 msgr2=0x7f6898136dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:17.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.359+0000 7f6897fff640 1 --2- 172.21.15.37:0/3053512790 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f68980a0110 0x7f6898136dd0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f68980764d0 tx=0x7f688001cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=762438fd2fa3f335 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:17.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.359+0000 7f68977fe640 1 -- 172.21.15.37:0/3053512790 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68800b0070 con 0x7f68980a0110 2024-04-26T00:43:17.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.359+0000 7f68977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:17.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.359+0000 7f68977fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:43:17.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.359+0000 7f68977fe640 10 monclient: dump: 2024-04-26T00:43:17.366 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:17.366 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:17.366 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:17.366 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:17.366 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:17.366 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:17.366 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:17.366 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:17.367 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:17.367 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:17.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.359+0000 7f68977fe640 10 monclient: _finish_auth 0 2024-04-26T00:43:17.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.359+0000 7f68977fe640 10 monclient: _check_auth_tickets 2024-04-26T00:43:17.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.359+0000 7f68977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:17.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.359+0000 7f68977fe640 1 -- 172.21.15.37:0/3053512790 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68800ab600 con 0x7f68980a0110 2024-04-26T00:43:17.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.359+0000 7f689eea6640 5 monclient: authenticate success, global_id 284109 2024-04-26T00:43:17.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.359+0000 7f689eea6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:43:17.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.359+0000 7f68977fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:17.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.359+0000 7f689eea6640 10 monclient: get_monmap_and_config success 2024-04-26T00:43:17.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.359+0000 7f689eea6640 4 set_mon_vals no callback set 2024-04-26T00:43:17.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.359+0000 7f68977fe640 1 -- 172.21.15.37:0/3053512790 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68800ab9d0 con 0x7f68980a0110 2024-04-26T00:43:17.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.359+0000 7f68977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:17.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.359+0000 7f68977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:43:17.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.359+0000 7f68977fe640 10 monclient: dump: 2024-04-26T00:43:17.368 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:17.368 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:17.368 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:17.368 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:17.368 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:17.368 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:17.368 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:17.368 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:17.368 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:17.368 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:17.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.359+0000 7f689eea6640 10 monclient: shutdown 2024-04-26T00:43:17.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.359+0000 7f689eea6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:17.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.359+0000 7f689eea6640 1 -- 172.21.15.37:0/3053512790 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f68980a0110 msgr2=0x7f6898136dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:17.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.359+0000 7f689eea6640 1 --2- 172.21.15.37:0/3053512790 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f68980a0110 0x7f6898136dd0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f68980764d0 tx=0x7f688001cbf0 comp rx=0 tx=0).stop 2024-04-26T00:43:17.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689eea6640 1 -- 172.21.15.37:0/3053512790 shutdown_connections 2024-04-26T00:43:17.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689eea6640 1 --2- 172.21.15.37:0/3053512790 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f68980a0110 0x7f6898136dd0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:17.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689eea6640 1 --2- 172.21.15.37:0/3053512790 >> v2:172.21.15.45:3300/0 conn(0x7f68980726b0 0x7f6898072a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:17.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689eea6640 1 -- 172.21.15.37:0/3053512790 >> 172.21.15.37:0/3053512790 conn(0x7f68980a1d50 msgr2=0x7f68980a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:17.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689eea6640 1 -- 172.21.15.37:0/3053512790 shutdown_connections 2024-04-26T00:43:17.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689eea6640 1 -- 172.21.15.37:0/3053512790 wait complete. 2024-04-26T00:43:17.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689eea6640 10 monclient: build_initial_monmap 2024-04-26T00:43:17.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689eea6640 10 monclient: monmap: 2024-04-26T00:43:17.370 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:17.370 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:43:17.370 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:43:17.370 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:43:17.370 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:17.370 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:43:17.370 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:43:17.370 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:43:17.370 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:43:17.370 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:17.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689eea6640 1 Processor -- start 2024-04-26T00:43:17.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689eea6640 1 -- start start 2024-04-26T00:43:17.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689eea6640 10 monclient: init 2024-04-26T00:43:17.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689eea6640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:17.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689eea6640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:43:17.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689eea6640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f68980726b0 0x7f68981cf200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:17.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689eea6640 10 monclient(hunting): picked mon.noname-a con 0x7f68980726b0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:43:17.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689eea6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68980a0110 0x7f6898131a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:17.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689eea6640 10 monclient(hunting): picked mon.noname-b con 0x7f68980a0110 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:43:17.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689eea6640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6898137310 0x7f689813b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:17.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689eea6640 10 monclient(hunting): picked mon.noname-c con 0x7f6898137310 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:17.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689eea6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:17.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689eea6640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f689809bc50 con 0x7f68980726b0 2024-04-26T00:43:17.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689eea6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:17.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689eea6640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f68980730c0 con 0x7f68980a0110 2024-04-26T00:43:17.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689eea6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:17.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689eea6640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f689809bdd0 con 0x7f6898137310 2024-04-26T00:43:17.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689eea6640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:17.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689eea6640 10 monclient(hunting): authenticate will time out at 2378.669533s 2024-04-26T00:43:17.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689cc42640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f68980726b0 0x7f68981cf200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:17.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689d443640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6898137310 0x7f689813b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:17.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689cc42640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f68980726b0 0x7f68981cf200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:54092/0 (socket says 172.21.15.37:54092) 2024-04-26T00:43:17.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689cc42640 1 -- 172.21.15.37:0/532122551 learned_addr learned my addr 172.21.15.37:0/532122551 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:17.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689cc42640 10 monclient(hunting): get_auth_request con 0x7f68980726b0 auth_method 0 2024-04-26T00:43:17.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689cc42640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:17.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689cc42640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:17.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689cc42640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:17.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689d443640 10 monclient(hunting): get_auth_request con 0x7f6898137310 auth_method 0 2024-04-26T00:43:17.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689d443640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:17.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689d443640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:17.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689d443640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:17.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689cc42640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:17.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689cc42640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:17.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689cc42640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:17.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689d443640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:17.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689d443640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:17.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f6897fff640 1 --2- 172.21.15.37:0/532122551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68980a0110 0x7f6898131a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:17.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689d443640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:17.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f6897fff640 10 monclient(hunting): get_auth_request con 0x7f68980a0110 auth_method 0 2024-04-26T00:43:17.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f6897fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:17.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f6897fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:17.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f6897fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:17.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689cc42640 10 monclient(hunting): handle_auth_done global_id 284112 payload 824 2024-04-26T00:43:17.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689cc42640 1 -- 172.21.15.37:0/532122551 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6898137310 msgr2=0x7f689813b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:17.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689cc42640 1 --2- 172.21.15.37:0/532122551 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6898137310 0x7f689813b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:17.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689cc42640 1 -- 172.21.15.37:0/532122551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68980a0110 msgr2=0x7f6898131a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:17.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689cc42640 1 --2- 172.21.15.37:0/532122551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68980a0110 0x7f6898131a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:17.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689cc42640 10 monclient: _finish_hunting 0 2024-04-26T00:43:17.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689cc42640 1 monclient: found mon.noname-a 2024-04-26T00:43:17.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689cc42640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:17.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689cc42640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:43:17.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.363+0000 7f689cc42640 1 -- 172.21.15.37:0/532122551 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f689813d660 con 0x7f68980726b0 2024-04-26T00:43:17.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f689cc42640 1 -- 172.21.15.37:0/532122551 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f68980726b0 msgr2=0x7f68981cf200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:17.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f689cc42640 1 --2- 172.21.15.37:0/532122551 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f68980726b0 0x7f68981cf200 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f688800d830 tx=0x7f688800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:17.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f68957fa640 1 -- 172.21.15.37:0/532122551 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f688800b8d0 con 0x7f68980726b0 2024-04-26T00:43:17.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f68957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:17.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f68957fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:43:17.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f68957fa640 10 monclient: dump: 2024-04-26T00:43:17.378 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:17.378 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:17.378 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:17.379 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:17.379 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:17.379 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:17.379 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:17.379 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:17.379 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:17.379 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:17.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f68957fa640 10 monclient: _finish_auth 0 2024-04-26T00:43:17.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f68957fa640 10 monclient: _check_auth_tickets 2024-04-26T00:43:17.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f68957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:17.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f689eea6640 5 monclient: authenticate success, global_id 284112 2024-04-26T00:43:17.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f68957fa640 1 -- 172.21.15.37:0/532122551 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6888019070 con 0x7f68980726b0 2024-04-26T00:43:17.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f68957fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:17.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f68957fa640 1 -- 172.21.15.37:0/532122551 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f688800bc70 con 0x7f68980726b0 2024-04-26T00:43:17.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f689eea6640 10 monclient: _renew_subs 2024-04-26T00:43:17.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f689eea6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:43:17.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f689eea6640 1 -- 172.21.15.37:0/532122551 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f689811bb60 con 0x7f68980726b0 2024-04-26T00:43:17.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f6896ffd640 4 set_mon_vals no callback set 2024-04-26T00:43:17.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f68957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:17.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f68957fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:43:17.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f68957fa640 10 monclient: dump: 2024-04-26T00:43:17.382 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:17.382 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:17.382 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:17.382 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:17.382 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:17.383 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:17.383 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:17.383 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:17.383 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:17.383 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:17.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f689eea6640 10 monclient: _renew_subs 2024-04-26T00:43:17.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f689eea6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:43:17.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f689eea6640 1 -- 172.21.15.37:0/532122551 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f689813ebf0 con 0x7f68980726b0 2024-04-26T00:43:17.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f689eea6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:17.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f689eea6640 1 --2- 172.21.15.37:0/532122551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f689813bed0 0x7f689813c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:17.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f689eea6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:17.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f689eea6640 1 -- 172.21.15.37:0/532122551 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f689811bea0 con 0x7f689813bed0 2024-04-26T00:43:17.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f68957fa640 1 -- 172.21.15.37:0/532122551 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f688801e460 con 0x7f68980726b0 2024-04-26T00:43:17.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f68957fa640 1 --2- 172.21.15.37:0/532122551 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6878035ed0 0x7f6878038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:17.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f6897fff640 1 --2- 172.21.15.37:0/532122551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f689813bed0 0x7f689813c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:17.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f6897fff640 10 monclient: get_auth_request con 0x7f689813bed0 auth_method 0 2024-04-26T00:43:17.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f6897fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:17.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f6897fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:17.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f6897fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:17.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f68957fa640 1 -- 172.21.15.37:0/532122551 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(899..899 src has 822..899) ==== 3550+0+0 (secure 0 0 0) 0x7f6888048de0 con 0x7f68980726b0 2024-04-26T00:43:17.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f6897fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:17.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f6897fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:17.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f6897fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:17.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f689d443640 1 --2- 172.21.15.37:0/532122551 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6878035ed0 0x7f6878038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:17.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f689d443640 10 monclient: get_auth_request con 0x7f6878035ed0 auth_method 0 2024-04-26T00:43:17.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f6897fff640 10 monclient(hunting): handle_auth_done global_id 274207 payload 824 2024-04-26T00:43:17.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.367+0000 7f6897fff640 1 -- 172.21.15.37:0/532122551 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f689813c7f0 con 0x7f689813bed0 2024-04-26T00:43:17.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.371+0000 7f6897fff640 1 -- 172.21.15.37:0/532122551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f689813bed0 msgr2=0x7f689813c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:17.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.371+0000 7f6897fff640 1 --2- 172.21.15.37:0/532122551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f689813bed0 0x7f689813c2b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f68800ad000 tx=0x7f688001a9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:17.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.371+0000 7f68957fa640 1 -- 172.21.15.37:0/532122551 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68800b0040 con 0x7f689813bed0 2024-04-26T00:43:17.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.371+0000 7f68957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:17.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.371+0000 7f68957fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:43:17.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.371+0000 7f68957fa640 10 monclient: dump: 2024-04-26T00:43:17.387 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:17.388 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:17.388 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:17.388 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:17.388 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:17.388 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:17.388 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:17.388 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:17.388 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:17.388 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:17.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.371+0000 7f689d443640 1 --2- 172.21.15.37:0/532122551 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6878035ed0 0x7f6878038390 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f6890005600 tx=0x7f6890003d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:17.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.371+0000 7f68957fa640 1 -- 172.21.15.37:0/532122551 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f68800b5660 con 0x7f689813bed0 2024-04-26T00:43:17.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.371+0000 7f68957fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:17.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.371+0000 7f68957fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:43:17.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.371+0000 7f68957fa640 1 -- 172.21.15.37:0/532122551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f689813bed0 msgr2=0x7f689813c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:17.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.371+0000 7f68957fa640 1 --2- 172.21.15.37:0/532122551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f689813bed0 0x7f689813c2b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f68800ad000 tx=0x7f688001a9d0 comp rx=0 tx=0).stop 2024-04-26T00:43:17.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.371+0000 7f68957fa640 1 -- 172.21.15.37:0/532122551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f689813bed0 msgr2=0x7f689813c2b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:43:17.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.371+0000 7f68957fa640 1 --2- 172.21.15.37:0/532122551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f689813bed0 0x7f689813c2b0 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f68800ad000 tx=0x7f688001a9d0 comp rx=0 tx=0).stop 2024-04-26T00:43:17.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.379+0000 7f689eea6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T00:43:17.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.379+0000 7f689eea6640 1 --2- 172.21.15.37:0/532122551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6898137de0 0x7f68981381c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:17.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.379+0000 7f689eea6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:17.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.379+0000 7f689eea6640 1 -- 172.21.15.37:0/532122551 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f689804f510 con 0x7f6898137de0 2024-04-26T00:43:17.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.379+0000 7f6897fff640 1 --2- 172.21.15.37:0/532122551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6898137de0 0x7f68981381c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:17.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.379+0000 7f6897fff640 10 monclient: get_auth_request con 0x7f6898137de0 auth_method 0 2024-04-26T00:43:17.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.379+0000 7f6897fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:17.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.379+0000 7f6897fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:17.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.379+0000 7f6897fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:17.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.379+0000 7f6897fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:17.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.379+0000 7f6897fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:17.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.379+0000 7f6897fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:17.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.379+0000 7f6897fff640 10 monclient(hunting): handle_auth_done global_id 274210 payload 824 2024-04-26T00:43:17.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.379+0000 7f6897fff640 1 -- 172.21.15.37:0/532122551 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6898073280 con 0x7f6898137de0 2024-04-26T00:43:17.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.379+0000 7f6897fff640 1 -- 172.21.15.37:0/532122551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6898137de0 msgr2=0x7f68981381c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:17.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.379+0000 7f6897fff640 1 --2- 172.21.15.37:0/532122551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6898137de0 0x7f68981381c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f688001cc30 tx=0x7f6880018f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:17.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.383+0000 7f68957fa640 1 -- 172.21.15.37:0/532122551 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68800b0070 con 0x7f6898137de0 2024-04-26T00:43:17.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.383+0000 7f68957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:17.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.383+0000 7f68957fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:43:17.392 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.383+0000 7f68957fa640 10 monclient: dump: 2024-04-26T00:43:17.392 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:17.392 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:17.392 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:17.392 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:17.392 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:17.393 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:17.393 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:17.393 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:17.393 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:17.393 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:17.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.383+0000 7f68957fa640 1 -- 172.21.15.37:0/532122551 <== mon.1 v2:172.21.15.45:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4717 (secure 0 0 0) 0x7f6880003820 con 0x7f6898137de0 2024-04-26T00:43:17.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.383+0000 7f68957fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T00:43:17.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.383+0000 7f68957fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:43:17.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.383+0000 7f68957fa640 1 -- 172.21.15.37:0/532122551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6898137de0 msgr2=0x7f68981381c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:17.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.383+0000 7f68957fa640 1 --2- 172.21.15.37:0/532122551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6898137de0 0x7f68981381c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f688001cc30 tx=0x7f6880018f30 comp rx=0 tx=0).stop 2024-04-26T00:43:17.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.383+0000 7f68957fa640 1 -- 172.21.15.37:0/532122551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6898137de0 msgr2=0x7f68981381c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:43:17.393 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.383+0000 7f68957fa640 1 --2- 172.21.15.37:0/532122551 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6898137de0 0x7f68981381c0 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f688001cc30 tx=0x7f6880018f30 comp rx=0 tx=0).stop 2024-04-26T00:43:17.394 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:43:17.394 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:43:17.394 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:43:17.394 INFO:teuthology.orchestra.run.smithi037.stdout: "state": "peon", 2024-04-26T00:43:17.394 INFO:teuthology.orchestra.run.smithi037.stdout: "election_epoch": 166, 2024-04-26T00:43:17.394 INFO:teuthology.orchestra.run.smithi037.stdout: "uptime": 65130, 2024-04-26T00:43:17.394 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum": [ 2024-04-26T00:43:17.394 INFO:teuthology.orchestra.run.smithi037.stdout: 0, 2024-04-26T00:43:17.394 INFO:teuthology.orchestra.run.smithi037.stdout: 1, 2024-04-26T00:43:17.394 INFO:teuthology.orchestra.run.smithi037.stdout: 2 2024-04-26T00:43:17.395 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:43:17.395 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_age": 4, 2024-04-26T00:43:17.395 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:43:17.395 INFO:teuthology.orchestra.run.smithi037.stdout: "required_con": "2450521708007866372", 2024-04-26T00:43:17.395 INFO:teuthology.orchestra.run.smithi037.stdout: "required_mon": [ 2024-04-26T00:43:17.395 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:43:17.395 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:43:17.395 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:43:17.395 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:43:17.395 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:43:17.395 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:43:17.395 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:43:17.395 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:43:17.395 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:43:17.396 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:43:17.396 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:43:17.396 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:43:17.396 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_con": "4540701547738038271", 2024-04-26T00:43:17.396 INFO:teuthology.orchestra.run.smithi037.stdout: "quorum_mon": [ 2024-04-26T00:43:17.396 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:43:17.396 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:43:17.396 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:43:17.396 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:43:17.396 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:43:17.396 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:43:17.396 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:43:17.396 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:43:17.397 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:43:17.397 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:43:17.397 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:43:17.397 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:43:17.397 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:43:17.397 INFO:teuthology.orchestra.run.smithi037.stdout: "outside_quorum": [], 2024-04-26T00:43:17.397 INFO:teuthology.orchestra.run.smithi037.stdout: "extra_probe_peers": [], 2024-04-26T00:43:17.397 INFO:teuthology.orchestra.run.smithi037.stdout: "sync_provider": [], 2024-04-26T00:43:17.397 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": { 2024-04-26T00:43:17.397 INFO:teuthology.orchestra.run.smithi037.stdout: "epoch": 1, 2024-04-26T00:43:17.397 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "3cf2f905-fe97-4d94-8eba-4a21c519e869", 2024-04-26T00:43:17.397 INFO:teuthology.orchestra.run.smithi037.stdout: "modified": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:43:17.397 INFO:teuthology.orchestra.run.smithi037.stdout: "created": "2024-04-26T00:19:19.195498Z", 2024-04-26T00:43:17.397 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release": 19, 2024-04-26T00:43:17.398 INFO:teuthology.orchestra.run.smithi037.stdout: "min_mon_release_name": "squid", 2024-04-26T00:43:17.398 INFO:teuthology.orchestra.run.smithi037.stdout: "election_strategy": 3, 2024-04-26T00:43:17.398 INFO:teuthology.orchestra.run.smithi037.stdout: "disallowed_leaders: ": "", 2024-04-26T00:43:17.398 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false, 2024-04-26T00:43:17.398 INFO:teuthology.orchestra.run.smithi037.stdout: "tiebreaker_mon": "", 2024-04-26T00:43:17.398 INFO:teuthology.orchestra.run.smithi037.stdout: "removed_ranks: ": "", 2024-04-26T00:43:17.398 INFO:teuthology.orchestra.run.smithi037.stdout: "features": { 2024-04-26T00:43:17.398 INFO:teuthology.orchestra.run.smithi037.stdout: "persistent": [ 2024-04-26T00:43:17.398 INFO:teuthology.orchestra.run.smithi037.stdout: "kraken", 2024-04-26T00:43:17.398 INFO:teuthology.orchestra.run.smithi037.stdout: "luminous", 2024-04-26T00:43:17.398 INFO:teuthology.orchestra.run.smithi037.stdout: "mimic", 2024-04-26T00:43:17.398 INFO:teuthology.orchestra.run.smithi037.stdout: "osdmap-prune", 2024-04-26T00:43:17.399 INFO:teuthology.orchestra.run.smithi037.stdout: "nautilus", 2024-04-26T00:43:17.399 INFO:teuthology.orchestra.run.smithi037.stdout: "octopus", 2024-04-26T00:43:17.399 INFO:teuthology.orchestra.run.smithi037.stdout: "pacific", 2024-04-26T00:43:17.399 INFO:teuthology.orchestra.run.smithi037.stdout: "elector-pinging", 2024-04-26T00:43:17.399 INFO:teuthology.orchestra.run.smithi037.stdout: "quincy", 2024-04-26T00:43:17.399 INFO:teuthology.orchestra.run.smithi037.stdout: "reef", 2024-04-26T00:43:17.399 INFO:teuthology.orchestra.run.smithi037.stdout: "squid" 2024-04-26T00:43:17.399 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:43:17.399 INFO:teuthology.orchestra.run.smithi037.stdout: "optional": [] 2024-04-26T00:43:17.399 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:43:17.399 INFO:teuthology.orchestra.run.smithi037.stdout: "mons": [ 2024-04-26T00:43:17.399 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:43:17.399 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 0, 2024-04-26T00:43:17.399 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "a", 2024-04-26T00:43:17.400 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:43:17.400 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:43:17.400 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:43:17.400 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:43:17.400 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3300", 2024-04-26T00:43:17.400 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:43:17.400 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:43:17.401 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:43:17.401 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:43:17.401 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789", 2024-04-26T00:43:17.401 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:43:17.401 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:43:17.401 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:43:17.401 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:43:17.401 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6789/0", 2024-04-26T00:43:17.401 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6789/0", 2024-04-26T00:43:17.402 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:43:17.402 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:43:17.402 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:43:17.402 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:43:17.402 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:43:17.402 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 1, 2024-04-26T00:43:17.402 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "b", 2024-04-26T00:43:17.402 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:43:17.403 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:43:17.403 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:43:17.403 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:43:17.403 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:3300", 2024-04-26T00:43:17.403 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:43:17.403 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:43:17.403 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:43:17.403 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:43:17.404 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789", 2024-04-26T00:43:17.404 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:43:17.404 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:43:17.404 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:43:17.404 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:43:17.404 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.45:6789/0", 2024-04-26T00:43:17.404 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.45:6789/0", 2024-04-26T00:43:17.404 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:43:17.404 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:43:17.405 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:43:17.405 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:43:17.405 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:43:17.405 INFO:teuthology.orchestra.run.smithi037.stdout: "rank": 2, 2024-04-26T00:43:17.405 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "c", 2024-04-26T00:43:17.405 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrs": { 2024-04-26T00:43:17.405 INFO:teuthology.orchestra.run.smithi037.stdout: "addrvec": [ 2024-04-26T00:43:17.405 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:43:17.406 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v2", 2024-04-26T00:43:17.406 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:3301", 2024-04-26T00:43:17.406 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:43:17.406 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:43:17.406 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:43:17.406 INFO:teuthology.orchestra.run.smithi037.stdout: "type": "v1", 2024-04-26T00:43:17.406 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790", 2024-04-26T00:43:17.406 INFO:teuthology.orchestra.run.smithi037.stdout: "nonce": 0 2024-04-26T00:43:17.407 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:43:17.407 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:43:17.407 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:43:17.407 INFO:teuthology.orchestra.run.smithi037.stdout: "addr": "172.21.15.37:6790/0", 2024-04-26T00:43:17.407 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "172.21.15.37:6790/0", 2024-04-26T00:43:17.407 INFO:teuthology.orchestra.run.smithi037.stdout: "priority": 0, 2024-04-26T00:43:17.407 INFO:teuthology.orchestra.run.smithi037.stdout: "weight": 0, 2024-04-26T00:43:17.407 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "{}" 2024-04-26T00:43:17.407 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:43:17.407 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:43:17.408 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:43:17.408 INFO:teuthology.orchestra.run.smithi037.stdout: "feature_map": { 2024-04-26T00:43:17.408 INFO:teuthology.orchestra.run.smithi037.stdout: "mon": [ 2024-04-26T00:43:17.408 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:43:17.408 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:43:17.408 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:43:17.408 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:43:17.408 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:43:17.408 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:43:17.409 INFO:teuthology.orchestra.run.smithi037.stdout: "osd": [ 2024-04-26T00:43:17.409 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:43:17.409 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:43:17.409 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:43:17.409 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:43:17.409 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:43:17.409 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T00:43:17.409 INFO:teuthology.orchestra.run.smithi037.stdout: "client": [ 2024-04-26T00:43:17.410 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T00:43:17.410 INFO:teuthology.orchestra.run.smithi037.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T00:43:17.410 INFO:teuthology.orchestra.run.smithi037.stdout: "release": "luminous", 2024-04-26T00:43:17.410 INFO:teuthology.orchestra.run.smithi037.stdout: "num": 1 2024-04-26T00:43:17.410 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T00:43:17.410 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T00:43:17.410 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T00:43:17.410 INFO:teuthology.orchestra.run.smithi037.stdout: "stretch_mode": false 2024-04-26T00:43:17.411 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:43:17.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.383+0000 7f689eea6640 1 -- 172.21.15.37:0/532122551 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6878035ed0 msgr2=0x7f6878038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:17.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.383+0000 7f689eea6640 1 --2- 172.21.15.37:0/532122551 >> v2:172.21.15.45:6812/2379918623 conn(0x7f6878035ed0 0x7f6878038390 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f6890005600 tx=0x7f6890003d50 comp rx=0 tx=0).stop 2024-04-26T00:43:17.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.383+0000 7f689eea6640 10 monclient: shutdown 2024-04-26T00:43:17.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.383+0000 7f689eea6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:17.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.383+0000 7f689eea6640 1 -- 172.21.15.37:0/532122551 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f68980726b0 msgr2=0x7f68981cf200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:17.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.383+0000 7f689eea6640 1 --2- 172.21.15.37:0/532122551 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f68980726b0 0x7f68981cf200 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f688800d830 tx=0x7f688800dcf0 comp rx=0 tx=0).stop 2024-04-26T00:43:17.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.383+0000 7f689d443640 1 -- 172.21.15.37:0/532122551 reap_dead start 2024-04-26T00:43:17.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.383+0000 7f689d443640 1 -- 172.21.15.37:0/532122551 reap_dead start 2024-04-26T00:43:17.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.383+0000 7f689eea6640 1 -- 172.21.15.37:0/532122551 shutdown_connections 2024-04-26T00:43:17.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.383+0000 7f689eea6640 1 -- 172.21.15.37:0/532122551 >> 172.21.15.37:0/532122551 conn(0x7f68980a1d50 msgr2=0x7f689809d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:17.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.383+0000 7f689eea6640 1 -- 172.21.15.37:0/532122551 shutdown_connections 2024-04-26T00:43:17.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.383+0000 7f689eea6640 1 -- 172.21.15.37:0/532122551 wait complete. 2024-04-26T00:43:17.414 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T00:43:17.414 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-26T00:43:17.414 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-26T00:43:17.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.535+0000 7ff6cd644640 10 monclient: get_monmap_and_config 2024-04-26T00:43:17.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.535+0000 7ff6cd644640 10 monclient: build_initial_monmap 2024-04-26T00:43:17.537 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.535+0000 7ff6cd644640 10 monclient: monmap: 2024-04-26T00:43:17.537 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:17.537 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:17.537 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:43:17.537734+0000 2024-04-26T00:43:17.537 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:43:17.537734+0000 2024-04-26T00:43:17.537 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:17.537 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:17.537 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:43:17.538 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:43:17.538 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:43:17.538 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:43:17.538 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:43:17.538 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:17.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.535+0000 7ff6cd644640 1 Processor -- start 2024-04-26T00:43:17.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.535+0000 7ff6cd644640 1 -- start start 2024-04-26T00:43:17.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.535+0000 7ff6cd644640 10 monclient: init 2024-04-26T00:43:17.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.535+0000 7ff6cd644640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:17.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.535+0000 7ff6cd644640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-04-26T00:43:17.540 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.535+0000 7ff6cd644640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff6c8136da0 0x7ff6c809bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:17.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.535+0000 7ff6cd644640 10 monclient(hunting): picked mon.noname-a con 0x7ff6c8136da0 addr v2:172.21.15.37:3300/0 2024-04-26T00:43:17.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.535+0000 7ff6cd644640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff6c809c410 addr v1:172.21.15.37:6789/0 2024-04-26T00:43:17.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.535+0000 7ff6cd644640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6c8137890 0x7ff6c813e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:17.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.535+0000 7ff6cd644640 10 monclient(hunting): picked mon.noname-b con 0x7ff6c8137890 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:17.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.535+0000 7ff6cd644640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6c809b4f0 con 0x7ff6c809c410 2024-04-26T00:43:17.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.535+0000 7ff6cd644640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:17.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.535+0000 7ff6cd644640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7ff6c809bb40 con 0x7ff6c8136da0 2024-04-26T00:43:17.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.535+0000 7ff6cd644640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:17.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.535+0000 7ff6cd644640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff6c809b210 con 0x7ff6c8137890 2024-04-26T00:43:17.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.535+0000 7ff6cd644640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:17.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.535+0000 7ff6cd644640 10 monclient(hunting): authenticate will time out at 2378.843638s 2024-04-26T00:43:17.541 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.535+0000 7ff6c6ffd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff6c8136da0 0x7ff6c809bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:17.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.535+0000 7ff6c77fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6c8137890 0x7ff6c813e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:17.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.535+0000 7ff6c67fc640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7ff6c809c410 0x7ff6c8137180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:43850/0 (socket says 172.21.15.37:43850) 2024-04-26T00:43:17.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.535+0000 7ff6c67fc640 1 -- 172.21.15.37:0/1844620925 learned_addr learned my addr 172.21.15.37:0/1844620925 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:43:17.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.535+0000 7ff6c6ffd640 10 monclient(hunting): get_auth_request con 0x7ff6c8136da0 auth_method 0 2024-04-26T00:43:17.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.535+0000 7ff6c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:17.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.535+0000 7ff6c6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:17.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.535+0000 7ff6c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:17.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.535+0000 7ff6c77fe640 10 monclient(hunting): get_auth_request con 0x7ff6c8137890 auth_method 0 2024-04-26T00:43:17.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.535+0000 7ff6c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:17.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.535+0000 7ff6c77fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:17.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.535+0000 7ff6c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:17.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.535+0000 7ff6c6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:17.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.535+0000 7ff6c6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:17.542 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.535+0000 7ff6c6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:17.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.535+0000 7ff6c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:17.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.535+0000 7ff6c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:17.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.539+0000 7ff6c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:17.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.539+0000 7ff6c77fe640 10 monclient(hunting): handle_auth_done global_id 254444 payload 274 2024-04-26T00:43:17.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.539+0000 7ff6c77fe640 1 -- 172.21.15.37:0/1844620925 >> v2:172.21.15.37:3300/0 conn(0x7ff6c8136da0 msgr2=0x7ff6c809bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:17.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.539+0000 7ff6c77fe640 1 --2- 172.21.15.37:0/1844620925 >> v2:172.21.15.37:3300/0 conn(0x7ff6c8136da0 0x7ff6c809bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:17.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.539+0000 7ff6c77fe640 1 -- 172.21.15.37:0/1844620925 >> v1:172.21.15.37:6789/0 conn(0x7ff6c809c410 legacy=0x7ff6c8137180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:17.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.539+0000 7ff6c77fe640 10 monclient: _finish_hunting 0 2024-04-26T00:43:17.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.539+0000 7ff6c77fe640 1 monclient: found mon.noname-b 2024-04-26T00:43:17.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.539+0000 7ff6c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:17.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.539+0000 7ff6c77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:43:17.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.539+0000 7ff6c77fe640 1 -- 172.21.15.37:0/1844620925 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6c809b7d0 con 0x7ff6c8137890 2024-04-26T00:43:17.543 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.539+0000 7ff6c6ffd640 1 --2- 172.21.15.37:0/1844620925 >> v2:172.21.15.37:3300/0 conn(0x7ff6c8136da0 0x7ff6c809bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:43:17.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6c77fe640 1 -- 172.21.15.37:0/1844620925 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6c8137890 msgr2=0x7ff6c813e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:17.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6c77fe640 1 --2- 172.21.15.37:0/1844620925 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6c8137890 0x7ff6c813e2e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff6c0009d90 tx=0x7ff6c001c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=baf63eaa0d42eec server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:17.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6c5ffb640 1 -- 172.21.15.37:0/1844620925 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c00b2070 con 0x7ff6c8137890 2024-04-26T00:43:17.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6c5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:17.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6c5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:43:17.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6c5ffb640 10 monclient: dump: 2024-04-26T00:43:17.545 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:17.545 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:17.545 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:17.545 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:17.545 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:17.546 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:17.546 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:17.546 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:17.546 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:17.546 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:17.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6c5ffb640 10 monclient: _finish_auth 0 2024-04-26T00:43:17.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6c5ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:43:17.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:17.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6cd644640 5 monclient: authenticate success, global_id 254444 2024-04-26T00:43:17.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6cd644640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:43:17.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6c5ffb640 1 -- 172.21.15.37:0/1844620925 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6c00a5890 con 0x7ff6c8137890 2024-04-26T00:43:17.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6c5ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:17.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6c5ffb640 1 -- 172.21.15.37:0/1844620925 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c00a5c60 con 0x7ff6c8137890 2024-04-26T00:43:17.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6c5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:17.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6c5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:43:17.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6c5ffb640 10 monclient: dump: 2024-04-26T00:43:17.547 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:17.547 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:17.547 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:17.547 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:17.547 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:17.547 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:17.547 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:17.547 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:17.547 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:17.547 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:17.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6cd644640 10 monclient: get_monmap_and_config success 2024-04-26T00:43:17.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6cd644640 4 set_mon_vals no callback set 2024-04-26T00:43:17.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6cd644640 10 monclient: shutdown 2024-04-26T00:43:17.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6cd644640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:17.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6cd644640 1 -- 172.21.15.37:0/1844620925 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6c8137890 msgr2=0x7ff6c813e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:17.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6cd644640 1 --2- 172.21.15.37:0/1844620925 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6c8137890 0x7ff6c813e2e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff6c0009d90 tx=0x7ff6c001c7f0 comp rx=0 tx=0).stop 2024-04-26T00:43:17.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6cd644640 1 -- 172.21.15.37:0/1844620925 shutdown_connections 2024-04-26T00:43:17.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6cd644640 1 --2- 172.21.15.37:0/1844620925 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6c8137890 0x7ff6c813e2e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:17.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6cd644640 1 --2- 172.21.15.37:0/1844620925 >> v2:172.21.15.37:3300/0 conn(0x7ff6c8136da0 0x7ff6c809bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:17.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6cd644640 1 -- 172.21.15.37:0/1844620925 >> 172.21.15.37:0/1844620925 conn(0x7ff6c80a5f00 msgr2=0x7ff6c80a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:17.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6cd644640 1 -- 172.21.15.37:0/1844620925 shutdown_connections 2024-04-26T00:43:17.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6cd644640 1 -- 172.21.15.37:0/1844620925 wait complete. 2024-04-26T00:43:17.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6cd644640 10 monclient: build_initial_monmap 2024-04-26T00:43:17.549 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6cd644640 10 monclient: monmap: 2024-04-26T00:43:17.549 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:17.549 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:43:17.549 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:43:17.549 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:43:17.549 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:17.549 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:43:17.549 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:43:17.549 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:43:17.549 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:43:17.549 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:17.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6cd644640 1 Processor -- start 2024-04-26T00:43:17.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6cd644640 1 -- start start 2024-04-26T00:43:17.555 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6cd644640 10 monclient: init 2024-04-26T00:43:17.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6cd644640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:17.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6cd644640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:43:17.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6cd644640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6c809c410 0x7ff6c81cea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:17.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6cd644640 10 monclient(hunting): picked mon.noname-c con 0x7ff6c809c410 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:17.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6cd644640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff6c8136da0 0x7ff6c813d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:17.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6cd644640 10 monclient(hunting): picked mon.noname-a con 0x7ff6c8136da0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:43:17.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6cd644640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6c8137890 0x7ff6c813d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:17.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6cd644640 10 monclient(hunting): picked mon.noname-b con 0x7ff6c8137890 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:43:17.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6cd644640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:17.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6cd644640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c81414b0 con 0x7ff6c8136da0 2024-04-26T00:43:17.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6cd644640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:17.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6cd644640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c8141330 con 0x7ff6c8137890 2024-04-26T00:43:17.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6cd644640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:17.556 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6cd644640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff6c8141630 con 0x7ff6c809c410 2024-04-26T00:43:17.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6cd644640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:17.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6cd644640 10 monclient(hunting): authenticate will time out at 2378.851705s 2024-04-26T00:43:17.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6c6ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6c809c410 0x7ff6c81cea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:17.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6c6ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6c809c410 0x7ff6c81cea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:35948/0 (socket says 172.21.15.37:35948) 2024-04-26T00:43:17.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6c6ffd640 1 -- 172.21.15.37:0/1683306594 learned_addr learned my addr 172.21.15.37:0/1683306594 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:17.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6c6ffd640 10 monclient(hunting): get_auth_request con 0x7ff6c809c410 auth_method 0 2024-04-26T00:43:17.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:17.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6c6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:17.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:17.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6c67fc640 1 --2- 172.21.15.37:0/1683306594 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff6c8136da0 0x7ff6c813d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:17.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6c67fc640 10 monclient(hunting): get_auth_request con 0x7ff6c8136da0 auth_method 0 2024-04-26T00:43:17.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6c67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:17.557 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6c67fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:17.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.543+0000 7ff6c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:17.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:17.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:17.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:17.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c77fe640 1 --2- 172.21.15.37:0/1683306594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6c8137890 0x7ff6c813d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:17.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c6ffd640 10 monclient(hunting): handle_auth_done global_id 254447 payload 824 2024-04-26T00:43:17.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c6ffd640 1 -- 172.21.15.37:0/1683306594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6c8137890 msgr2=0x7ff6c813d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:17.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c6ffd640 1 --2- 172.21.15.37:0/1683306594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6c8137890 0x7ff6c813d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:17.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c6ffd640 1 -- 172.21.15.37:0/1683306594 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff6c8136da0 msgr2=0x7ff6c813d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:17.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c6ffd640 1 --2- 172.21.15.37:0/1683306594 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff6c8136da0 0x7ff6c813d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:17.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c6ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:43:17.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c6ffd640 1 monclient: found mon.noname-c 2024-04-26T00:43:17.558 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:17.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:43:17.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c6ffd640 1 -- 172.21.15.37:0/1683306594 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6c813ba20 con 0x7ff6c809c410 2024-04-26T00:43:17.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c77fe640 10 monclient: get_auth_request con 0x7ff6c8137890 auth_method 0 2024-04-26T00:43:17.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c77fe640 1 --2- 172.21.15.37:0/1683306594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6c8137890 0x7ff6c813d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:43:17.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c6ffd640 1 -- 172.21.15.37:0/1683306594 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6c809c410 msgr2=0x7ff6c81cea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:17.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c6ffd640 1 --2- 172.21.15.37:0/1683306594 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6c809c410 0x7ff6c81cea00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff6b000ec10 tx=0x7ff6b000c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:17.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c6ffd640 1 -- 172.21.15.37:0/1683306594 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6c809c410 msgr2=0x7ff6c81cea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T00:43:17.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c6ffd640 1 -- 172.21.15.37:0/1683306594 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6c809c410 msgr2=0x7ff6c81cea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:43:17.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c6ffd640 1 --2- 172.21.15.37:0/1683306594 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6c809c410 0x7ff6c81cea00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff6b000ec10 tx=0x7ff6b000c520 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:43:17.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c6ffd640 1 --2- 172.21.15.37:0/1683306594 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6c809c410 0x7ff6c81cea00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff6b000ec10 tx=0x7ff6b000c520 comp rx=0 tx=0).stop 2024-04-26T00:43:17.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6bf7fe640 10 monclient: ms_handle_reset current mon [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:17.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6bf7fe640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:17.559 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6bf7fe640 1 -- 172.21.15.37:0/1683306594 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6c809c410 msgr2=0x7ff6c81cea00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:43:17.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6bf7fe640 1 --2- 172.21.15.37:0/1683306594 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6c809c410 0x7ff6c81cea00 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:17.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6bf7fe640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:43:17.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6bf7fe640 1 --2- 172.21.15.37:0/1683306594 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6a80034b0 0x7ff6a8005900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:17.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6bf7fe640 10 monclient(hunting): picked mon.noname-c con 0x7ff6a80034b0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:17.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6bf7fe640 1 --2- 172.21.15.37:0/1683306594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6c8137890 0x7ff6a8008430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:17.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6bf7fe640 10 monclient(hunting): picked mon.noname-b con 0x7ff6c8137890 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:43:17.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6bf7fe640 1 --2- 172.21.15.37:0/1683306594 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff6c8136da0 0x7ff6a800ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:17.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6bf7fe640 10 monclient(hunting): picked mon.noname-a con 0x7ff6c8136da0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:43:17.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6bf7fe640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:17.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6bf7fe640 1 -- 172.21.15.37:0/1683306594 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a800b880 con 0x7ff6c8136da0 2024-04-26T00:43:17.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6bf7fe640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:17.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6bf7fe640 1 -- 172.21.15.37:0/1683306594 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a800ba00 con 0x7ff6c8137890 2024-04-26T00:43:17.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6bf7fe640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:17.560 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6bf7fe640 1 -- 172.21.15.37:0/1683306594 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff6a800bc00 con 0x7ff6a80034b0 2024-04-26T00:43:17.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6bf7fe640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:17.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c6ffd640 1 --2- 172.21.15.37:0/1683306594 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6a80034b0 0x7ff6a8005900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:17.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c6ffd640 10 monclient(hunting): get_auth_request con 0x7ff6a80034b0 auth_method 0 2024-04-26T00:43:17.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:17.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:17.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:43:17.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c77fe640 1 --2- 172.21.15.37:0/1683306594 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff6c8136da0 0x7ff6a800ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:17.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c77fe640 10 monclient(hunting): get_auth_request con 0x7ff6c8136da0 auth_method 0 2024-04-26T00:43:17.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:17.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c77fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:17.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:43:17.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:17.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:17.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:43:17.561 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:17.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:17.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:43:17.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c67fc640 1 --2- 172.21.15.37:0/1683306594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6c8137890 0x7ff6a8008430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:17.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c67fc640 10 monclient(hunting): get_auth_request con 0x7ff6c8137890 auth_method 0 2024-04-26T00:43:17.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:17.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c67fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:17.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c67fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:43:17.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c6ffd640 10 monclient(hunting): handle_auth_done global_id 254447 payload 293 2024-04-26T00:43:17.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c6ffd640 1 -- 172.21.15.37:0/1683306594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6c8137890 msgr2=0x7ff6a8008430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:17.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c6ffd640 1 --2- 172.21.15.37:0/1683306594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6c8137890 0x7ff6a8008430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:17.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c6ffd640 1 -- 172.21.15.37:0/1683306594 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff6c8136da0 msgr2=0x7ff6a800ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:17.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c6ffd640 1 --2- 172.21.15.37:0/1683306594 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff6c8136da0 0x7ff6a800ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:17.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c6ffd640 10 monclient: _finish_hunting 0 2024-04-26T00:43:17.562 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c6ffd640 1 monclient: found mon.noname-c 2024-04-26T00:43:17.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:17.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:43:17.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c6ffd640 1 -- 172.21.15.37:0/1683306594 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6a800bec0 con 0x7ff6a80034b0 2024-04-26T00:43:17.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.547+0000 7ff6c77fe640 1 --2- 172.21.15.37:0/1683306594 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff6c8136da0 0x7ff6a800ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:43:17.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.551+0000 7ff6c6ffd640 1 -- 172.21.15.37:0/1683306594 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6a80034b0 msgr2=0x7ff6a8005900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:17.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.551+0000 7ff6c6ffd640 1 --2- 172.21.15.37:0/1683306594 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6a80034b0 0x7ff6a8005900 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff6b0019000 tx=0x7ff6b0009190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:17.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.551+0000 7ff6bf7fe640 1 -- 172.21.15.37:0/1683306594 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b0017070 con 0x7ff6a80034b0 2024-04-26T00:43:17.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.551+0000 7ff6bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:17.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.551+0000 7ff6bf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:43:17.563 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.551+0000 7ff6bf7fe640 10 monclient: dump: 2024-04-26T00:43:17.563 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:17.563 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:17.563 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:17.563 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:17.564 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:17.564 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:17.564 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:17.564 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:17.564 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:17.564 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:17.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.551+0000 7ff6bf7fe640 10 monclient: _finish_auth 0 2024-04-26T00:43:17.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.551+0000 7ff6bf7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:43:17.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.551+0000 7ff6bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:17.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.551+0000 7ff6cd644640 5 monclient: authenticate success, global_id 254447 2024-04-26T00:43:17.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.551+0000 7ff6bf7fe640 1 -- 172.21.15.37:0/1683306594 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6b0004030 con 0x7ff6a80034b0 2024-04-26T00:43:17.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.551+0000 7ff6cd644640 10 monclient: _renew_subs 2024-04-26T00:43:17.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.551+0000 7ff6cd644640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:17.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.551+0000 7ff6cd644640 1 -- 172.21.15.37:0/1683306594 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6c809aca0 con 0x7ff6a80034b0 2024-04-26T00:43:17.564 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.551+0000 7ff6bf7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:17.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.551+0000 7ff6bf7fe640 1 -- 172.21.15.37:0/1683306594 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b0004400 con 0x7ff6a80034b0 2024-04-26T00:43:17.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.551+0000 7ff6bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:17.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.551+0000 7ff6bf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:43:17.565 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.551+0000 7ff6bf7fe640 10 monclient: dump: 2024-04-26T00:43:17.565 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:17.565 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:17.565 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:17.565 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:17.565 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:17.565 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:17.565 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:17.566 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:17.566 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:17.566 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:17.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.551+0000 7ff6c57fa640 4 set_mon_vals no callback set 2024-04-26T00:43:17.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.551+0000 7ff6cd644640 10 monclient: _renew_subs 2024-04-26T00:43:17.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.551+0000 7ff6cd644640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:17.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.551+0000 7ff6cd644640 1 -- 172.21.15.37:0/1683306594 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6c81cf120 con 0x7ff6a80034b0 2024-04-26T00:43:17.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.551+0000 7ff6bf7fe640 1 -- 172.21.15.37:0/1683306594 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff6b001a070 con 0x7ff6a80034b0 2024-04-26T00:43:17.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.551+0000 7ff6cd644640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:17.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.551+0000 7ff6cd644640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:17.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.551+0000 7ff6cd644640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:17.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.551+0000 7ff6bf7fe640 1 --2- 172.21.15.37:0/1683306594 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff6c809c410 0x7ff6a8040450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:17.566 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.551+0000 7ff6cd644640 1 -- 172.21.15.37:0/1683306594 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6c8066f40 con 0x7ff6a80034b0 2024-04-26T00:43:17.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.551+0000 7ff6bf7fe640 1 -- 172.21.15.37:0/1683306594 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(899..899 src has 822..899) ==== 3550+0+0 (secure 0 0 0) 0x7ff6b004a100 con 0x7ff6a80034b0 2024-04-26T00:43:17.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.551+0000 7ff6c67fc640 1 --2- 172.21.15.37:0/1683306594 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff6c809c410 0x7ff6a8040450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:17.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.551+0000 7ff6c67fc640 10 monclient: get_auth_request con 0x7ff6c809c410 auth_method 0 2024-04-26T00:43:17.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.555+0000 7ff6c67fc640 1 --2- 172.21.15.37:0/1683306594 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff6c809c410 0x7ff6a8040450 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ff6b8007fd0 tx=0x7ff6b8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:17.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.559+0000 7ff6bf7fe640 1 -- 172.21.15.37:0/1683306594 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff6b0023070 con 0x7ff6a80034b0 2024-04-26T00:43:17.567 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.559+0000 7ff6bf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:17.568 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.559+0000 7ff6bf7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:43:17.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.779+0000 7ff6cd644640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-26T00:43:17.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.779+0000 7ff6cd644640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-26T00:43:17.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.779+0000 7ff6cd644640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:17.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.779+0000 7ff6cd644640 1 -- 172.21.15.37:0/1683306594 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7ff6c813b280 con 0x7ff6a80034b0 2024-04-26T00:43:17.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.783+0000 7ff6bf7fe640 1 -- 172.21.15.37:0/1683306594 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7ff6b000d1c0 con 0x7ff6a80034b0 2024-04-26T00:43:17.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.783+0000 7ff6bf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-26T00:43:17.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.783+0000 7ff6bf7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:43:17.785 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.783+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.786 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.783+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=30,config=2,health=11,kv=10,logm=47} crc {auth=1487714734,config=692619927,health=3508314031,kv=1034996044,logm=3067151566}) 2024-04-26T00:43:17.786 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.783+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=2,health=12,kv=10,logm=46} crc {auth=3147525344,config=692619927,health=3364491971,kv=1034996044,logm=2811341949}) 2024-04-26T00:43:17.786 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.783+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.786 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.783+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=30,config=2,health=11,kv=10,logm=47} crc {auth=1487714734,config=692619927,health=3508314031,kv=1034996044,logm=3067151566}) 2024-04-26T00:43:17.786 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.783+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=29,config=2,health=11,kv=8,logm=50} crc {auth=3962370067,config=692619927,health=1713110318,kv=2821313474,logm=828733809}) 2024-04-26T00:43:17.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.783+0000 7ff6cd644640 1 -- 172.21.15.37:0/1683306594 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff6c809c410 msgr2=0x7ff6a8040450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:17.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.783+0000 7ff6cd644640 1 --2- 172.21.15.37:0/1683306594 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff6c809c410 0x7ff6a8040450 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ff6b8007fd0 tx=0x7ff6b8009040 comp rx=0 tx=0).stop 2024-04-26T00:43:17.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.783+0000 7ff6cd644640 10 monclient: shutdown 2024-04-26T00:43:17.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.783+0000 7ff6cd644640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:17.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.783+0000 7ff6cd644640 1 -- 172.21.15.37:0/1683306594 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6a80034b0 msgr2=0x7ff6a8005900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:17.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.783+0000 7ff6cd644640 1 --2- 172.21.15.37:0/1683306594 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6a80034b0 0x7ff6a8005900 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff6b0019000 tx=0x7ff6b0009190 comp rx=0 tx=0).stop 2024-04-26T00:43:17.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.783+0000 7ff6cd644640 1 -- 172.21.15.37:0/1683306594 shutdown_connections 2024-04-26T00:43:17.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.783+0000 7ff6cd644640 1 --2- 172.21.15.37:0/1683306594 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff6c809c410 0x7ff6a8040450 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:17.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.783+0000 7ff6cd644640 1 --2- 172.21.15.37:0/1683306594 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff6c8136da0 0x7ff6a800ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:17.787 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.783+0000 7ff6cd644640 1 --2- 172.21.15.37:0/1683306594 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6c8137890 0x7ff6a8008430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:17.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.783+0000 7ff6cd644640 1 --2- 172.21.15.37:0/1683306594 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6a80034b0 0x7ff6a8005900 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:17.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.783+0000 7ff6cd644640 1 -- 172.21.15.37:0/1683306594 >> 172.21.15.37:0/1683306594 conn(0x7ff6c80a5f00 msgr2=0x7ff6c8135db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:17.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.787+0000 7ff6cd644640 1 -- 172.21.15.37:0/1683306594 shutdown_connections 2024-04-26T00:43:17.788 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:17.787+0000 7ff6cd644640 1 -- 172.21.15.37:0/1683306594 wait complete. 2024-04-26T00:43:17.788 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.787+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.789 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.787+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3403278773}) 2024-04-26T00:43:17.789 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.787+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1818528028}) 2024-04-26T00:43:17.789 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.787+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.789 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.787+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3403278773}) 2024-04-26T00:43:17.789 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.787+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4134923973}) 2024-04-26T00:43:17.791 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.791+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.791 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.791+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1858467542}) 2024-04-26T00:43:17.792 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.791+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2055852215}) 2024-04-26T00:43:17.792 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.791+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.792 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.791+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1858467542}) 2024-04-26T00:43:17.792 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.791+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1596180248}) 2024-04-26T00:43:17.794 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.791+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.795 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.791+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1297040816}) 2024-04-26T00:43:17.795 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.791+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4001459029}) 2024-04-26T00:43:17.795 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.791+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.795 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.795+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1297040816}) 2024-04-26T00:43:17.795 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.795+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=239524864}) 2024-04-26T00:43:17.798 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.795+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.798 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.795+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1016323439}) 2024-04-26T00:43:17.798 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.795+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2677922932}) 2024-04-26T00:43:17.798 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.795+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.798 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.795+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1016323439}) 2024-04-26T00:43:17.798 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.795+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2222937965}) 2024-04-26T00:43:17.800 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.799+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.801 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.799+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1542408464}) 2024-04-26T00:43:17.801 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.799+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3822886683}) 2024-04-26T00:43:17.801 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.799+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.801 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.799+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1542408464}) 2024-04-26T00:43:17.801 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.799+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3645756008}) 2024-04-26T00:43:17.803 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.799+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.803 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.803+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2552986471}) 2024-04-26T00:43:17.803 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.803+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1240924868}) 2024-04-26T00:43:17.803 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.803+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.803 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.803+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2552986471}) 2024-04-26T00:43:17.803 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.803+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2812035657}) 2024-04-26T00:43:17.805 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.803+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.805 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.803+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=701099278}) 2024-04-26T00:43:17.805 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.803+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3278787803}) 2024-04-26T00:43:17.805 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.803+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.805 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.803+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=701099278}) 2024-04-26T00:43:17.805 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.803+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2945380593}) 2024-04-26T00:43:17.807 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.807+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.807 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.807+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1757646284}) 2024-04-26T00:43:17.807 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.807+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=380853236}) 2024-04-26T00:43:17.807 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.807+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.807 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.807+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1757646284}) 2024-04-26T00:43:17.807 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.807+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1330874780}) 2024-04-26T00:43:17.808 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-26T00:43:17.809 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.807+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.809 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.807+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2188708533}) 2024-04-26T00:43:17.809 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.807+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3875157908}) 2024-04-26T00:43:17.809 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.807+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.809 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.807+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2188708533}) 2024-04-26T00:43:17.810 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.807+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1677258060}) 2024-04-26T00:43:17.812 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.811+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.812 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.811+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2781011585}) 2024-04-26T00:43:17.812 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.811+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1295996831}) 2024-04-26T00:43:17.812 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.811+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.812 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.811+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2781011585}) 2024-04-26T00:43:17.812 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.811+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=939352289}) 2024-04-26T00:43:17.814 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.811+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.814 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.811+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4189244744}) 2024-04-26T00:43:17.814 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.811+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3096286104}) 2024-04-26T00:43:17.814 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.811+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.814 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.811+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4189244744}) 2024-04-26T00:43:17.815 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.811+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=298284769}) 2024-04-26T00:43:17.816 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.815+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.817 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.815+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2426265811}) 2024-04-26T00:43:17.817 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.815+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1614717651}) 2024-04-26T00:43:17.817 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.815+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.817 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.815+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2426265811}) 2024-04-26T00:43:17.817 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.815+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2063512290}) 2024-04-26T00:43:17.818 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.815+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.818 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.815+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4216292583}) 2024-04-26T00:43:17.818 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.815+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1537036602}) 2024-04-26T00:43:17.818 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.815+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.819 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.815+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4216292583}) 2024-04-26T00:43:17.819 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.815+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1499426296}) 2024-04-26T00:43:17.821 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.819+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.821 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.819+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3374825333}) 2024-04-26T00:43:17.821 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.819+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1410149332}) 2024-04-26T00:43:17.821 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.819+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.821 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.819+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3374825333}) 2024-04-26T00:43:17.821 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.819+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3077388160}) 2024-04-26T00:43:17.823 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.823+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.823 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.823+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2006484408}) 2024-04-26T00:43:17.823 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.823+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=547557477}) 2024-04-26T00:43:17.823 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.823+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.823 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.823+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2006484408}) 2024-04-26T00:43:17.823 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.823+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1513731286}) 2024-04-26T00:43:17.825 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.823+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.825 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.823+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3578410338}) 2024-04-26T00:43:17.825 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.823+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=400710034}) 2024-04-26T00:43:17.825 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.823+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.826 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.823+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3578410338}) 2024-04-26T00:43:17.826 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.823+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3506938498}) 2024-04-26T00:43:17.828 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.827+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.828 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.827+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=693957032}) 2024-04-26T00:43:17.828 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.827+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4151348673}) 2024-04-26T00:43:17.828 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.827+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.828 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.827+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=693957032}) 2024-04-26T00:43:17.828 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.827+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1967694514}) 2024-04-26T00:43:17.830 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.827+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.830 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.827+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3863056833}) 2024-04-26T00:43:17.830 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.827+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1526428920}) 2024-04-26T00:43:17.830 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.827+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.830 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.827+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3863056833}) 2024-04-26T00:43:17.830 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.827+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3213509635}) 2024-04-26T00:43:17.833 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.831+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.833 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.831+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4044137629}) 2024-04-26T00:43:17.833 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.831+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1631458821}) 2024-04-26T00:43:17.833 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.831+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.833 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.831+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4044137629}) 2024-04-26T00:43:17.833 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.831+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=169357474}) 2024-04-26T00:43:17.835 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.835+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.835 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.835+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=246986233}) 2024-04-26T00:43:17.835 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.835+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2676446118}) 2024-04-26T00:43:17.835 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.835+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.835 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.835+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=246986233}) 2024-04-26T00:43:17.835 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.835+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3508959875}) 2024-04-26T00:43:17.837 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.835+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.837 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.835+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=984217134}) 2024-04-26T00:43:17.837 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.835+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1708979756}) 2024-04-26T00:43:17.837 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.835+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.837 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.835+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=984217134}) 2024-04-26T00:43:17.837 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.835+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2549757994}) 2024-04-26T00:43:17.839 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.839+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.839 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.839+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2696080489}) 2024-04-26T00:43:17.840 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.839+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2976289072}) 2024-04-26T00:43:17.840 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.839+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.840 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.839+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2696080489}) 2024-04-26T00:43:17.840 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.839+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2502781048}) 2024-04-26T00:43:17.842 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.839+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.842 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.839+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3419223118}) 2024-04-26T00:43:17.842 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.839+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1693236054}) 2024-04-26T00:43:17.842 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.839+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.842 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.839+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3419223118}) 2024-04-26T00:43:17.843 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.839+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4000961908}) 2024-04-26T00:43:17.844 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.843+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.844 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.843+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1583003158}) 2024-04-26T00:43:17.844 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.843+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2236830367}) 2024-04-26T00:43:17.845 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.843+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.845 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.843+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1583003158}) 2024-04-26T00:43:17.845 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.843+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3763753536}) 2024-04-26T00:43:17.847 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.847+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.847 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.847+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=736666162}) 2024-04-26T00:43:17.847 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.847+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1107425165}) 2024-04-26T00:43:17.847 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.847+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.847 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.847+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=736666162}) 2024-04-26T00:43:17.847 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.847+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=510615876}) 2024-04-26T00:43:17.849 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.847+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.849 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.847+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3974998573}) 2024-04-26T00:43:17.849 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.847+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1414166942}) 2024-04-26T00:43:17.849 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.847+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.850 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.847+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3974998573}) 2024-04-26T00:43:17.850 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.847+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=349923607}) 2024-04-26T00:43:17.852 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.851+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.852 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.851+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2936974265}) 2024-04-26T00:43:17.852 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.851+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1578929632}) 2024-04-26T00:43:17.852 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.851+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.852 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.851+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2936974265}) 2024-04-26T00:43:17.852 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.851+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3545260659}) 2024-04-26T00:43:17.854 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.851+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.854 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.851+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3976998850}) 2024-04-26T00:43:17.854 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.851+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2083677978}) 2024-04-26T00:43:17.854 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.851+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.855 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.851+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3976998850}) 2024-04-26T00:43:17.855 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.851+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1179036233}) 2024-04-26T00:43:17.856 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.855+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.856 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.855+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2152128653}) 2024-04-26T00:43:17.857 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.855+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=780702697}) 2024-04-26T00:43:17.857 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.855+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.857 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.855+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2152128653}) 2024-04-26T00:43:17.857 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.855+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1833531279}) 2024-04-26T00:43:17.859 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.855+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.859 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.855+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=386622055}) 2024-04-26T00:43:17.859 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.855+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2635056664}) 2024-04-26T00:43:17.859 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.855+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.859 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.859+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=386622055}) 2024-04-26T00:43:17.859 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.859+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1134292649}) 2024-04-26T00:43:17.861 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.859+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.861 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.859+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2254413552}) 2024-04-26T00:43:17.861 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.859+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2363635305}) 2024-04-26T00:43:17.861 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.859+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.861 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.859+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2254413552}) 2024-04-26T00:43:17.861 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.859+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2416729744}) 2024-04-26T00:43:17.863 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.863+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.863 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.863+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3232219725}) 2024-04-26T00:43:17.864 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.863+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4092054801}) 2024-04-26T00:43:17.864 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.863+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.864 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.863+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3232219725}) 2024-04-26T00:43:17.864 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.863+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2714426533}) 2024-04-26T00:43:17.866 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.863+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.866 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.863+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2864532409}) 2024-04-26T00:43:17.866 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.863+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1194375310}) 2024-04-26T00:43:17.866 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.863+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.866 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.863+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2864532409}) 2024-04-26T00:43:17.866 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.863+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3521975691}) 2024-04-26T00:43:17.868 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.867+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.868 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.867+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2680471294}) 2024-04-26T00:43:17.868 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.867+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=738456916}) 2024-04-26T00:43:17.869 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.867+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.869 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.867+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2680471294}) 2024-04-26T00:43:17.869 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.867+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3338227377}) 2024-04-26T00:43:17.871 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.871+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.871 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.871+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3183118389}) 2024-04-26T00:43:17.871 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.871+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3056188152}) 2024-04-26T00:43:17.871 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.871+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.871 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.871+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3183118389}) 2024-04-26T00:43:17.871 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.871+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2280856601}) 2024-04-26T00:43:17.873 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.871+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.873 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.871+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1116397603}) 2024-04-26T00:43:17.873 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.871+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=575192147}) 2024-04-26T00:43:17.873 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.871+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.873 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.871+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1116397603}) 2024-04-26T00:43:17.874 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.871+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3104627320}) 2024-04-26T00:43:17.876 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.875+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.876 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.875+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2156699387}) 2024-04-26T00:43:17.876 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.875+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=769112782}) 2024-04-26T00:43:17.876 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.875+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.876 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.875+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2156699387}) 2024-04-26T00:43:17.876 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.875+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=885347953}) 2024-04-26T00:43:17.879 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.875+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.879 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.875+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4250657397}) 2024-04-26T00:43:17.879 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.879+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2327401538}) 2024-04-26T00:43:17.879 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.879+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.879 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.879+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4250657397}) 2024-04-26T00:43:17.879 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.879+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4118249968}) 2024-04-26T00:43:17.881 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.879+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.881 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.879+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1686444630}) 2024-04-26T00:43:17.881 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.879+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2105848065}) 2024-04-26T00:43:17.881 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.879+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.881 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.879+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1686444630}) 2024-04-26T00:43:17.881 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.879+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1884978140}) 2024-04-26T00:43:17.884 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.883+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.884 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.883+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3652607844}) 2024-04-26T00:43:17.884 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.883+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3414378823}) 2024-04-26T00:43:17.884 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.883+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.884 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.883+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3652607844}) 2024-04-26T00:43:17.884 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.883+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3824937474}) 2024-04-26T00:43:17.886 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.883+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.886 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.883+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2605901918}) 2024-04-26T00:43:17.887 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.883+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2809591292}) 2024-04-26T00:43:17.887 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.883+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.887 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.883+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2605901918}) 2024-04-26T00:43:17.887 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.883+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2772361375}) 2024-04-26T00:43:17.889 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.887+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.889 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.887+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2780250709}) 2024-04-26T00:43:17.889 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.887+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2142558219}) 2024-04-26T00:43:17.889 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.887+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.889 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.887+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2780250709}) 2024-04-26T00:43:17.889 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.887+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=824979773}) 2024-04-26T00:43:17.891 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.891+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.891 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.891+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1884869685}) 2024-04-26T00:43:17.892 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.891+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=42528120}) 2024-04-26T00:43:17.892 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.891+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:17.892 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.891+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1884869685}) 2024-04-26T00:43:17.892 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:17.891+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2967753073}) 2024-04-26T00:43:18.325 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.323+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:18.326 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.323+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=556332355}) 2024-04-26T00:43:18.326 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.323+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3461185055}) 2024-04-26T00:43:18.326 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.323+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:18.326 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.323+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=556332355}) 2024-04-26T00:43:18.326 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.323+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2787210818}) 2024-04-26T00:43:18.328 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.327+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:18.328 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.327+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=497294516}) 2024-04-26T00:43:18.328 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.327+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2999830459}) 2024-04-26T00:43:18.328 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.327+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:18.328 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.327+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=497294516}) 2024-04-26T00:43:18.328 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.327+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=843786676}) 2024-04-26T00:43:18.331 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.331+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:18.331 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.331+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2861343992}) 2024-04-26T00:43:18.331 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.331+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3009510978}) 2024-04-26T00:43:18.331 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.331+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:18.331 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.331+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2861343992}) 2024-04-26T00:43:18.331 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.331+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3313784343}) 2024-04-26T00:43:18.333 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.331+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:18.333 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.331+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1208037069}) 2024-04-26T00:43:18.334 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.331+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1703965857}) 2024-04-26T00:43:18.334 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.331+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:18.334 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.331+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1208037069}) 2024-04-26T00:43:18.334 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.331+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=615112204}) 2024-04-26T00:43:18.336 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.335+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:18.336 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.335+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3989412969}) 2024-04-26T00:43:18.336 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.335+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3079344243}) 2024-04-26T00:43:18.336 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.335+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:18.336 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.335+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3989412969}) 2024-04-26T00:43:18.336 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.335+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=223592796}) 2024-04-26T00:43:18.338 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.335+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:18.339 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.335+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1932389279}) 2024-04-26T00:43:18.339 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.339+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=990734677}) 2024-04-26T00:43:18.339 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.339+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:18.339 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.339+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1932389279}) 2024-04-26T00:43:18.339 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.339+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1963729330}) 2024-04-26T00:43:18.341 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.339+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:18.341 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.339+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3629050729}) 2024-04-26T00:43:18.341 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.339+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=75480276}) 2024-04-26T00:43:18.341 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.339+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:18.342 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.339+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3629050729}) 2024-04-26T00:43:18.342 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.339+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=581254332}) 2024-04-26T00:43:18.344 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.343+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:18.344 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.343+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1003366738}) 2024-04-26T00:43:18.344 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.343+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3074445555}) 2024-04-26T00:43:18.344 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.343+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:18.344 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.343+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1003366738}) 2024-04-26T00:43:18.344 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.343+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1926272283}) 2024-04-26T00:43:18.346 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.343+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:18.346 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.343+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2272951563}) 2024-04-26T00:43:18.346 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.343+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3274180018}) 2024-04-26T00:43:18.346 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.343+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:18.346 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.343+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2272951563}) 2024-04-26T00:43:18.346 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.343+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1128279066}) 2024-04-26T00:43:18.349 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.347+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:18.349 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.347+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3235601578}) 2024-04-26T00:43:18.349 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.347+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=287923101}) 2024-04-26T00:43:18.349 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.347+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:18.349 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.347+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3235601578}) 2024-04-26T00:43:18.349 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.347+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4098829212}) 2024-04-26T00:43:18.351 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.351+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:18.351 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.351+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1137605691}) 2024-04-26T00:43:18.351 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.351+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=319917860}) 2024-04-26T00:43:18.351 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.351+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:18.352 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.351+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1137605691}) 2024-04-26T00:43:18.352 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.351+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=609993948}) 2024-04-26T00:43:18.354 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.351+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:18.354 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.351+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1256643497}) 2024-04-26T00:43:18.354 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.351+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3340392362}) 2024-04-26T00:43:18.354 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.351+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:18.354 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.351+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1256643497}) 2024-04-26T00:43:18.354 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.351+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=146343401}) 2024-04-26T00:43:18.356 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.355+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:18.356 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.355+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3454345736}) 2024-04-26T00:43:18.356 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.355+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2992455246}) 2024-04-26T00:43:18.357 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.355+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:18.357 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.355+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3454345736}) 2024-04-26T00:43:18.357 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.355+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2860620714}) 2024-04-26T00:43:18.359 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.359+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:18.359 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.359+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3808764776}) 2024-04-26T00:43:18.359 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.359+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1562650465}) 2024-04-26T00:43:18.360 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.359+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:18.360 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.359+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3808764776}) 2024-04-26T00:43:18.360 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.359+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4236338185}) 2024-04-26T00:43:18.361 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.359+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:18.361 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.359+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1949300310}) 2024-04-26T00:43:18.362 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.359+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4116345560}) 2024-04-26T00:43:18.362 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.359+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:18.362 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.359+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1949300310}) 2024-04-26T00:43:18.362 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.359+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2121490099}) 2024-04-26T00:43:18.364 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.363+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:18.364 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.363+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3095200331}) 2024-04-26T00:43:18.364 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.363+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1279104146}) 2024-04-26T00:43:18.364 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.363+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:18.364 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.363+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3095200331}) 2024-04-26T00:43:18.364 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.363+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3121771564}) 2024-04-26T00:43:18.366 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.363+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:18.367 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.363+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3696518658}) 2024-04-26T00:43:18.367 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.363+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3531955978}) 2024-04-26T00:43:18.367 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.363+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:18.367 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.363+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3696518658}) 2024-04-26T00:43:18.367 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.363+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=500428765}) 2024-04-26T00:43:18.369 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.367+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:18.369 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.367+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3234376771}) 2024-04-26T00:43:18.369 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.367+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4079191904}) 2024-04-26T00:43:18.370 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.367+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:18.370 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.367+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3234376771}) 2024-04-26T00:43:18.370 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.367+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1099489257}) 2024-04-26T00:43:18.372 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.371+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:18.372 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.371+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1495048711}) 2024-04-26T00:43:18.372 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.371+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=851519942}) 2024-04-26T00:43:18.372 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.371+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:18.372 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.371+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1495048711}) 2024-04-26T00:43:18.372 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.371+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2174959947}) 2024-04-26T00:43:18.374 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.371+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:18.374 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.371+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3459580639}) 2024-04-26T00:43:18.374 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.371+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3269528264}) 2024-04-26T00:43:18.374 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.371+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:18.374 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.371+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3459580639}) 2024-04-26T00:43:18.375 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:18.371+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=811047976}) 2024-04-26T00:43:18.809 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T00:43:18.809 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T00:43:18.940 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.939+0000 7fdae79a6640 10 monclient: get_monmap_and_config 2024-04-26T00:43:18.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.939+0000 7fdae79a6640 10 monclient: build_initial_monmap 2024-04-26T00:43:18.941 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.939+0000 7fdae79a6640 10 monclient: monmap: 2024-04-26T00:43:18.941 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:18.941 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:18.941 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:43:18.941703+0000 2024-04-26T00:43:18.941 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:43:18.941703+0000 2024-04-26T00:43:18.941 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:18.941 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:18.941 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:43:18.941 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:43:18.941 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:43:18.942 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:43:18.942 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:43:18.942 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:18.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.939+0000 7fdae79a6640 1 Processor -- start 2024-04-26T00:43:18.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.939+0000 7fdae79a6640 1 -- start start 2024-04-26T00:43:18.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.939+0000 7fdae79a6640 10 monclient: init 2024-04-26T00:43:18.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.939+0000 7fdae79a6640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:18.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.939+0000 7fdae79a6640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-04-26T00:43:18.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.939+0000 7fdae79a6640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdae0136e10 0x7fdae009c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:18.944 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.939+0000 7fdae79a6640 10 monclient(hunting): picked mon.noname-b con 0x7fdae0136e10 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:18.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.939+0000 7fdae79a6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdae01371f0 addr v1:172.21.15.45:6789/0 2024-04-26T00:43:18.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.939+0000 7fdae79a6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdae013c340 addr v1:172.21.15.37:6789/0 2024-04-26T00:43:18.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.939+0000 7fdae79a6640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdae009b680 con 0x7fdae013c340 2024-04-26T00:43:18.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.939+0000 7fdae79a6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdae009bbd0 con 0x7fdae01371f0 2024-04-26T00:43:18.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.939+0000 7fdae79a6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:18.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.939+0000 7fdae79a6640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fdae009b8f0 con 0x7fdae0136e10 2024-04-26T00:43:18.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.939+0000 7fdae79a6640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:18.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.939+0000 7fdae79a6640 10 monclient(hunting): authenticate will time out at 2380.247363s 2024-04-26T00:43:18.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.939+0000 7fdae5f43640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fdae013c340 0x7fdae013e730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:43852/0 (socket says 172.21.15.37:43852) 2024-04-26T00:43:18.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.939+0000 7fdae5f43640 1 -- 172.21.15.37:0/1796735259 learned_addr learned my addr 172.21.15.37:0/1796735259 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:43:18.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.939+0000 7fdae5742640 1 --2- 172.21.15.37:0/1796735259 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdae0136e10 0x7fdae009c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:18.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.939+0000 7fdae5742640 10 monclient(hunting): get_auth_request con 0x7fdae0136e10 auth_method 0 2024-04-26T00:43:18.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.939+0000 7fdae5742640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:18.945 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.939+0000 7fdae5742640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:18.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.939+0000 7fdae5742640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:18.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.939+0000 7fdae5742640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:18.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.939+0000 7fdae5742640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:18.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.939+0000 7fdae5742640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:18.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.939+0000 7fdae5742640 10 monclient(hunting): handle_auth_done global_id 254450 payload 274 2024-04-26T00:43:18.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.939+0000 7fdae5742640 1 -- 172.21.15.37:0/1796735259 >> v1:172.21.15.45:6789/0 conn(0x7fdae01371f0 legacy=0x7fdae01375d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:18.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.943+0000 7fdae5742640 1 -- 172.21.15.37:0/1796735259 >> v1:172.21.15.37:6789/0 conn(0x7fdae013c340 legacy=0x7fdae013e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:18.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.943+0000 7fdae5742640 10 monclient: _finish_hunting 0 2024-04-26T00:43:18.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.943+0000 7fdae5742640 1 monclient: found mon.noname-b 2024-04-26T00:43:18.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.943+0000 7fdae5742640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:18.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.943+0000 7fdae5742640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:43:18.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.943+0000 7fdae5742640 1 -- 172.21.15.37:0/1796735259 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdae00a5d90 con 0x7fdae0136e10 2024-04-26T00:43:18.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae5742640 1 -- 172.21.15.37:0/1796735259 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdae0136e10 msgr2=0x7fdae009c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:18.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae5742640 1 --2- 172.21.15.37:0/1796735259 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdae0136e10 0x7fdae009c5b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdadc009820 tx=0x7fdadc01c300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93f4e77a1740ba93 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:18.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdad3fff640 1 -- 172.21.15.37:0/1796735259 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdadc0b0070 con 0x7fdae0136e10 2024-04-26T00:43:18.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdad3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:18.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdad3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:43:18.954 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdad3fff640 10 monclient: dump: 2024-04-26T00:43:18.954 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:18.955 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:18.955 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:18.955 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:18.955 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:18.955 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:18.955 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:18.955 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:18.955 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:18.955 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:18.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdad3fff640 10 monclient: _finish_auth 0 2024-04-26T00:43:18.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdad3fff640 10 monclient: _check_auth_tickets 2024-04-26T00:43:18.955 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdad3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:18.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdad3fff640 1 -- 172.21.15.37:0/1796735259 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdadc0ab600 con 0x7fdae0136e10 2024-04-26T00:43:18.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae79a6640 5 monclient: authenticate success, global_id 254450 2024-04-26T00:43:18.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae79a6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:43:18.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdad3fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:18.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae79a6640 10 monclient: get_monmap_and_config success 2024-04-26T00:43:18.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae79a6640 4 set_mon_vals no callback set 2024-04-26T00:43:18.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdad3fff640 1 -- 172.21.15.37:0/1796735259 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdadc0ab9d0 con 0x7fdae0136e10 2024-04-26T00:43:18.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdad3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:18.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdad3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:43:18.956 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdad3fff640 10 monclient: dump: 2024-04-26T00:43:18.956 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:18.956 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:18.956 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:18.956 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:18.956 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:18.957 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:18.957 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:18.957 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:18.957 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:18.957 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:18.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae79a6640 10 monclient: shutdown 2024-04-26T00:43:18.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae79a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:18.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae79a6640 1 -- 172.21.15.37:0/1796735259 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdae0136e10 msgr2=0x7fdae009c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:18.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae79a6640 1 --2- 172.21.15.37:0/1796735259 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdae0136e10 0x7fdae009c5b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdadc009820 tx=0x7fdadc01c300 comp rx=0 tx=0).stop 2024-04-26T00:43:18.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae79a6640 1 -- 172.21.15.37:0/1796735259 shutdown_connections 2024-04-26T00:43:18.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae79a6640 1 --2- 172.21.15.37:0/1796735259 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdae0136e10 0x7fdae009c5b0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:18.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae79a6640 1 -- 172.21.15.37:0/1796735259 >> 172.21.15.37:0/1796735259 conn(0x7fdae00a64e0 msgr2=0x7fdae00a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:18.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae79a6640 1 -- 172.21.15.37:0/1796735259 shutdown_connections 2024-04-26T00:43:18.957 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae79a6640 1 -- 172.21.15.37:0/1796735259 wait complete. 2024-04-26T00:43:18.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae79a6640 10 monclient: build_initial_monmap 2024-04-26T00:43:18.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae79a6640 10 monclient: monmap: 2024-04-26T00:43:18.958 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:18.958 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:43:18.958 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:43:18.958 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:43:18.958 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:18.958 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:43:18.958 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:43:18.958 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:43:18.958 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:43:18.958 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:18.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae79a6640 1 Processor -- start 2024-04-26T00:43:18.958 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae79a6640 1 -- start start 2024-04-26T00:43:18.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae79a6640 10 monclient: init 2024-04-26T00:43:18.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae79a6640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:18.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae79a6640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:43:18.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae79a6640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdae0136e10 0x7fdae01cf450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:18.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae79a6640 10 monclient(hunting): picked mon.noname-a con 0x7fdae0136e10 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:43:18.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae79a6640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdae01371f0 0x7fdae0137e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:18.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae79a6640 10 monclient(hunting): picked mon.noname-c con 0x7fdae01371f0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:18.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae79a6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdae013c340 0x7fdae0138360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:18.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae79a6640 10 monclient(hunting): picked mon.noname-b con 0x7fdae013c340 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:43:18.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae79a6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:18.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae79a6640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fdae0142a00 con 0x7fdae0136e10 2024-04-26T00:43:18.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae79a6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:18.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae79a6640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdae0142880 con 0x7fdae013c340 2024-04-26T00:43:18.959 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae79a6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:18.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae79a6640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fdae0142b80 con 0x7fdae01371f0 2024-04-26T00:43:18.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae79a6640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:18.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae79a6640 10 monclient(hunting): authenticate will time out at 2380.254787s 2024-04-26T00:43:18.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae5742640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdae0136e10 0x7fdae01cf450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:18.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae4f41640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdae01371f0 0x7fdae0137e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:18.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae5742640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdae0136e10 0x7fdae01cf450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:54132/0 (socket says 172.21.15.37:54132) 2024-04-26T00:43:18.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae5742640 1 -- 172.21.15.37:0/2647705351 learned_addr learned my addr 172.21.15.37:0/2647705351 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:18.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae5742640 10 monclient(hunting): get_auth_request con 0x7fdae0136e10 auth_method 0 2024-04-26T00:43:18.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae5742640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:18.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae5742640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:18.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae5742640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:18.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae4f41640 10 monclient(hunting): get_auth_request con 0x7fdae01371f0 auth_method 0 2024-04-26T00:43:18.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae4f41640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:18.960 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae4f41640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:18.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae4f41640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:18.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae5742640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:18.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae5742640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:18.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae5742640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:18.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae4f41640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:18.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae4f41640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:18.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae4f41640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:18.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae5f43640 1 --2- 172.21.15.37:0/2647705351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdae013c340 0x7fdae0138360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:18.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae5f43640 10 monclient(hunting): get_auth_request con 0x7fdae013c340 auth_method 0 2024-04-26T00:43:18.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae5f43640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:18.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae5f43640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:18.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae5f43640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:18.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae5742640 10 monclient(hunting): handle_auth_done global_id 284121 payload 824 2024-04-26T00:43:18.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae5742640 1 -- 172.21.15.37:0/2647705351 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdae01371f0 msgr2=0x7fdae0137e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:18.961 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae5742640 1 --2- 172.21.15.37:0/2647705351 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdae01371f0 0x7fdae0137e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:18.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae5742640 1 -- 172.21.15.37:0/2647705351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdae013c340 msgr2=0x7fdae0138360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:18.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae5742640 1 --2- 172.21.15.37:0/2647705351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdae013c340 0x7fdae0138360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:18.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae5742640 10 monclient: _finish_hunting 0 2024-04-26T00:43:18.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae5742640 1 monclient: found mon.noname-a 2024-04-26T00:43:18.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae5742640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:18.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae5742640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:43:18.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.947+0000 7fdae5742640 1 -- 172.21.15.37:0/2647705351 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdae013a0f0 con 0x7fdae0136e10 2024-04-26T00:43:18.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.951+0000 7fdae5742640 1 -- 172.21.15.37:0/2647705351 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdae0136e10 msgr2=0x7fdae01cf450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:18.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.951+0000 7fdae5742640 1 --2- 172.21.15.37:0/2647705351 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdae0136e10 0x7fdae01cf450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdadc01a9b0 tx=0x7fdadc01ab40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:18.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.951+0000 7fdad1ffb640 1 -- 172.21.15.37:0/2647705351 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdadc0b0050 con 0x7fdae0136e10 2024-04-26T00:43:18.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.951+0000 7fdad1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:18.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.951+0000 7fdad1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:43:18.962 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.951+0000 7fdad1ffb640 10 monclient: dump: 2024-04-26T00:43:18.962 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:18.963 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:18.963 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:18.963 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:18.963 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:18.963 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:18.963 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:18.963 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:18.963 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:18.963 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:18.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.951+0000 7fdad1ffb640 10 monclient: _finish_auth 0 2024-04-26T00:43:18.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.951+0000 7fdad1ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:43:18.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.951+0000 7fdad1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:18.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.951+0000 7fdad1ffb640 1 -- 172.21.15.37:0/2647705351 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdadc0bc070 con 0x7fdae0136e10 2024-04-26T00:43:18.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.951+0000 7fdae79a6640 5 monclient: authenticate success, global_id 284121 2024-04-26T00:43:18.963 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.951+0000 7fdae79a6640 10 monclient: _renew_subs 2024-04-26T00:43:18.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.951+0000 7fdae79a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:43:18.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.951+0000 7fdae79a6640 1 -- 172.21.15.37:0/2647705351 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdae00a84a0 con 0x7fdae0136e10 2024-04-26T00:43:18.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.951+0000 7fdad1ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:18.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.951+0000 7fdad1ffb640 1 -- 172.21.15.37:0/2647705351 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdadc0aadd0 con 0x7fdae0136e10 2024-04-26T00:43:18.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.951+0000 7fdad1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:18.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.951+0000 7fdad1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:43:18.964 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.951+0000 7fdad1ffb640 10 monclient: dump: 2024-04-26T00:43:18.964 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:18.964 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:18.964 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:18.964 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:18.964 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:18.964 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:18.964 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:18.965 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:18.965 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:18.965 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:18.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.951+0000 7fdad2ffd640 4 set_mon_vals no callback set 2024-04-26T00:43:18.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.951+0000 7fdae79a6640 10 monclient: _renew_subs 2024-04-26T00:43:18.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.951+0000 7fdae79a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:43:18.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.951+0000 7fdae79a6640 1 -- 172.21.15.37:0/2647705351 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdae013b5a0 con 0x7fdae0136e10 2024-04-26T00:43:18.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.951+0000 7fdad1ffb640 1 -- 172.21.15.37:0/2647705351 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fdadc0c1460 con 0x7fdae0136e10 2024-04-26T00:43:18.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.951+0000 7fdae79a6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:18.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.951+0000 7fdae79a6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:18.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.951+0000 7fdae79a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:43:18.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.951+0000 7fdae79a6640 1 -- 172.21.15.37:0/2647705351 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdae0075910 con 0x7fdae0136e10 2024-04-26T00:43:18.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.951+0000 7fdad1ffb640 1 --2- 172.21.15.37:0/2647705351 >> v2:172.21.15.45:6812/2379918623 conn(0x7fdac4035e80 0x7fdac4038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:18.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.951+0000 7fdad1ffb640 1 -- 172.21.15.37:0/2647705351 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(899..899 src has 822..899) ==== 3550+0+0 (secure 0 0 0) 0x7fdadc0e3490 con 0x7fdae0136e10 2024-04-26T00:43:18.965 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.951+0000 7fdae4f41640 1 --2- 172.21.15.37:0/2647705351 >> v2:172.21.15.45:6812/2379918623 conn(0x7fdac4035e80 0x7fdac4038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:18.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.951+0000 7fdae4f41640 10 monclient: get_auth_request con 0x7fdac4035e80 auth_method 0 2024-04-26T00:43:18.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.955+0000 7fdae4f41640 1 --2- 172.21.15.37:0/2647705351 >> v2:172.21.15.45:6812/2379918623 conn(0x7fdac4035e80 0x7fdac4038340 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fdae00a0590 tx=0x7fdad4007e40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:18.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.959+0000 7fdad1ffb640 1 -- 172.21.15.37:0/2647705351 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fdadc016010 con 0x7fdae0136e10 2024-04-26T00:43:18.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.959+0000 7fdad1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:18.966 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:18.959+0000 7fdad1ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:43:19.008 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.007+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.008 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.007+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4029335418}) 2024-04-26T00:43:19.008 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.007+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=740637180}) 2024-04-26T00:43:19.008 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.007+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.008 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.007+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4029335418}) 2024-04-26T00:43:19.008 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.007+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1591272885}) 2024-04-26T00:43:19.014 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.011+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.014 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.011+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1162676285}) 2024-04-26T00:43:19.014 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.011+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2093838672}) 2024-04-26T00:43:19.014 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.011+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.014 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.011+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1162676285}) 2024-04-26T00:43:19.014 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.011+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1535994923}) 2024-04-26T00:43:19.043 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.043+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.043 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.043+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2109816399}) 2024-04-26T00:43:19.043 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.043+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1709657519}) 2024-04-26T00:43:19.043 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.043+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.043 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.043+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2109816399}) 2024-04-26T00:43:19.043 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.043+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3074640843}) 2024-04-26T00:43:19.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.187+0000 7fdae79a6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T00:43:19.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.187+0000 7fdae79a6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T00:43:19.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.187+0000 7fdae79a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:43:19.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.187+0000 7fdae79a6640 1 -- 172.21.15.37:0/2647705351 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fdae0066f40 con 0x7fdae0136e10 2024-04-26T00:43:19.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.191+0000 7fdad1ffb640 1 -- 172.21.15.37:0/2647705351 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1557 (secure 0 0 0) 0x7fdadc0e2360 con 0x7fdae0136e10 2024-04-26T00:43:19.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.191+0000 7fdad1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T00:43:19.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.191+0000 7fdad1ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:43:19.192 INFO:teuthology.orchestra.run.smithi037.stdout:{"election_epoch":166,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"c","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","modified":"2024-04-26T00:19:19.195498Z","created":"2024-04-26T00:19:19.195498Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3301","nonce":0},{"type":"v1","addr":"172.21.15.37:6790","nonce":0}]},"addr":"172.21.15.37:6790/0","public_addr":"172.21.15.37:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T00:43:19.192 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.191+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.192 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.191+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3903129499}) 2024-04-26T00:43:19.192 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.191+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1252148618}) 2024-04-26T00:43:19.193 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.191+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.193 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.191+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3903129499}) 2024-04-26T00:43:19.193 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.191+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=650147296}) 2024-04-26T00:43:19.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.191+0000 7fdae79a6640 1 -- 172.21.15.37:0/2647705351 >> v2:172.21.15.45:6812/2379918623 conn(0x7fdac4035e80 msgr2=0x7fdac4038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:19.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.191+0000 7fdae79a6640 1 --2- 172.21.15.37:0/2647705351 >> v2:172.21.15.45:6812/2379918623 conn(0x7fdac4035e80 0x7fdac4038340 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fdae00a0590 tx=0x7fdad4007e40 comp rx=0 tx=0).stop 2024-04-26T00:43:19.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.191+0000 7fdae79a6640 10 monclient: shutdown 2024-04-26T00:43:19.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.191+0000 7fdae79a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:19.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.191+0000 7fdae79a6640 1 -- 172.21.15.37:0/2647705351 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdae0136e10 msgr2=0x7fdae01cf450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:19.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.191+0000 7fdae79a6640 1 --2- 172.21.15.37:0/2647705351 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdae0136e10 0x7fdae01cf450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdadc01a9b0 tx=0x7fdadc01ab40 comp rx=0 tx=0).stop 2024-04-26T00:43:19.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.191+0000 7fdae79a6640 1 -- 172.21.15.37:0/2647705351 shutdown_connections 2024-04-26T00:43:19.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.191+0000 7fdae79a6640 1 --2- 172.21.15.37:0/2647705351 >> v2:172.21.15.45:6812/2379918623 conn(0x7fdac4035e80 0x7fdac4038340 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:19.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.191+0000 7fdae79a6640 1 --2- 172.21.15.37:0/2647705351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdae013c340 0x7fdae0138360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:19.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.191+0000 7fdae79a6640 1 --2- 172.21.15.37:0/2647705351 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdae01371f0 0x7fdae0137e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:19.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.191+0000 7fdae79a6640 1 --2- 172.21.15.37:0/2647705351 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdae0136e10 0x7fdae01cf450 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:19.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.191+0000 7fdae79a6640 1 -- 172.21.15.37:0/2647705351 >> 172.21.15.37:0/2647705351 conn(0x7fdae00a64e0 msgr2=0x7fdae00a7490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:19.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.191+0000 7fdae79a6640 1 -- 172.21.15.37:0/2647705351 shutdown_connections 2024-04-26T00:43:19.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.191+0000 7fdae79a6640 1 -- 172.21.15.37:0/2647705351 wait complete. 2024-04-26T00:43:19.210 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T00:43:19.210 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-26T00:43:19.233 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-26T00:43:19.233 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T00:43:19.235 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.227+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.235 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.227+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=136102685}) 2024-04-26T00:43:19.235 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.227+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3094731598}) 2024-04-26T00:43:19.235 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.227+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.235 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.227+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=136102685}) 2024-04-26T00:43:19.235 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.227+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2962121200}) 2024-04-26T00:43:19.235 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.227+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.235 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.227+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=189301076}) 2024-04-26T00:43:19.235 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.227+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2052549174}) 2024-04-26T00:43:19.235 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.227+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.236 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.227+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=189301076}) 2024-04-26T00:43:19.236 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.227+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=867170183}) 2024-04-26T00:43:19.236 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.231+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.236 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.231+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3723208877}) 2024-04-26T00:43:19.236 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.231+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=554932194}) 2024-04-26T00:43:19.236 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.231+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.236 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.231+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3723208877}) 2024-04-26T00:43:19.236 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.231+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=303529921}) 2024-04-26T00:43:19.236 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.231+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.236 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.231+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1996032910}) 2024-04-26T00:43:19.236 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.231+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2996973300}) 2024-04-26T00:43:19.236 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.231+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.236 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.231+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1996032910}) 2024-04-26T00:43:19.237 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.231+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2030605882}) 2024-04-26T00:43:19.238 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.235+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.238 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.235+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3264547201}) 2024-04-26T00:43:19.238 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.235+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=60753538}) 2024-04-26T00:43:19.238 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.235+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.238 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.235+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3264547201}) 2024-04-26T00:43:19.238 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.235+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2671975924}) 2024-04-26T00:43:19.239 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.235+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.239 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.235+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1588117671}) 2024-04-26T00:43:19.239 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.235+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2487571337}) 2024-04-26T00:43:19.239 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.235+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.239 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.235+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1588117671}) 2024-04-26T00:43:19.239 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.235+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2982947185}) 2024-04-26T00:43:19.240 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.239+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.240 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.239+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=294780385}) 2024-04-26T00:43:19.240 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.239+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2896771788}) 2024-04-26T00:43:19.240 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.239+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.240 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.239+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=294780385}) 2024-04-26T00:43:19.240 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.239+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2711824615}) 2024-04-26T00:43:19.242 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.239+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.242 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.239+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3757032159}) 2024-04-26T00:43:19.242 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.239+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3290645031}) 2024-04-26T00:43:19.242 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.239+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.242 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.239+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3757032159}) 2024-04-26T00:43:19.242 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.239+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1776857337}) 2024-04-26T00:43:19.244 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.243+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.244 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.243+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2214735157}) 2024-04-26T00:43:19.244 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.243+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4004724115}) 2024-04-26T00:43:19.244 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.243+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.244 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.243+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2214735157}) 2024-04-26T00:43:19.244 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.243+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2905011586}) 2024-04-26T00:43:19.246 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.243+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.246 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.243+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4079426737}) 2024-04-26T00:43:19.246 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.243+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3682068708}) 2024-04-26T00:43:19.246 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.243+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.246 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.243+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4079426737}) 2024-04-26T00:43:19.246 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.243+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1413987098}) 2024-04-26T00:43:19.248 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.247+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.248 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.247+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=894707386}) 2024-04-26T00:43:19.248 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.247+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2261878074}) 2024-04-26T00:43:19.248 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.247+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.248 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.247+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=894707386}) 2024-04-26T00:43:19.249 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.247+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2940192516}) 2024-04-26T00:43:19.250 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.247+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.250 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.247+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2375820018}) 2024-04-26T00:43:19.250 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.247+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3612728312}) 2024-04-26T00:43:19.250 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.247+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.251 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.247+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2375820018}) 2024-04-26T00:43:19.251 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.247+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3613297275}) 2024-04-26T00:43:19.252 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.251+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.252 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.251+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1771103522}) 2024-04-26T00:43:19.252 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.251+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1307880333}) 2024-04-26T00:43:19.253 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.251+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.253 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.251+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1771103522}) 2024-04-26T00:43:19.253 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.251+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1851981718}) 2024-04-26T00:43:19.254 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.251+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.255 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.251+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=370816879}) 2024-04-26T00:43:19.255 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.251+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2028413632}) 2024-04-26T00:43:19.255 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.251+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.255 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.251+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=370816879}) 2024-04-26T00:43:19.255 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.251+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2204387366}) 2024-04-26T00:43:19.256 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.255+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.256 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.255+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=655421440}) 2024-04-26T00:43:19.257 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.255+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2432922220}) 2024-04-26T00:43:19.257 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.255+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.257 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.255+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=655421440}) 2024-04-26T00:43:19.257 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.255+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1444020685}) 2024-04-26T00:43:19.258 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.255+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.259 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.255+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4139960054}) 2024-04-26T00:43:19.259 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.255+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3329865922}) 2024-04-26T00:43:19.259 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.255+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.259 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.255+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4139960054}) 2024-04-26T00:43:19.259 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.255+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3704185146}) 2024-04-26T00:43:19.260 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.259+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.261 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.259+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3014937406}) 2024-04-26T00:43:19.261 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.259+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1626762940}) 2024-04-26T00:43:19.261 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.259+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.261 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.259+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3014937406}) 2024-04-26T00:43:19.261 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.259+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2974809963}) 2024-04-26T00:43:19.263 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.259+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.263 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.259+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3849182803}) 2024-04-26T00:43:19.263 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.259+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=952798565}) 2024-04-26T00:43:19.263 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.259+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.263 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.259+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3849182803}) 2024-04-26T00:43:19.263 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.259+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3731361337}) 2024-04-26T00:43:19.265 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.263+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.265 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.263+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2583605729}) 2024-04-26T00:43:19.265 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.263+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2388005624}) 2024-04-26T00:43:19.265 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.263+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.265 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.263+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2583605729}) 2024-04-26T00:43:19.265 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.263+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1226004250}) 2024-04-26T00:43:19.267 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.267+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.267 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.267+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4276049888}) 2024-04-26T00:43:19.267 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.267+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2788457790}) 2024-04-26T00:43:19.267 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.267+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.267 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.267+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4276049888}) 2024-04-26T00:43:19.267 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.267+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3822748685}) 2024-04-26T00:43:19.269 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.267+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.269 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.267+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=581617155}) 2024-04-26T00:43:19.269 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.267+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3031660184}) 2024-04-26T00:43:19.269 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.267+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.269 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.267+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=581617155}) 2024-04-26T00:43:19.269 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.267+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=63992331}) 2024-04-26T00:43:19.271 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.271+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.271 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.271+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3479019171}) 2024-04-26T00:43:19.271 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.271+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1776438179}) 2024-04-26T00:43:19.271 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.271+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.271 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.271+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3479019171}) 2024-04-26T00:43:19.271 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.271+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2686810820}) 2024-04-26T00:43:19.273 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.271+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.273 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.271+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=869360309}) 2024-04-26T00:43:19.273 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.271+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1228789372}) 2024-04-26T00:43:19.273 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.271+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.273 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.271+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=869360309}) 2024-04-26T00:43:19.273 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.271+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2118978554}) 2024-04-26T00:43:19.275 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.275+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.275 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.275+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=775998441}) 2024-04-26T00:43:19.275 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.275+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1939537193}) 2024-04-26T00:43:19.275 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.275+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.275 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.275+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=775998441}) 2024-04-26T00:43:19.275 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.275+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3309133960}) 2024-04-26T00:43:19.277 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.275+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.277 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.275+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1534763399}) 2024-04-26T00:43:19.277 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.275+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2516833955}) 2024-04-26T00:43:19.277 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.275+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.278 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.275+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1534763399}) 2024-04-26T00:43:19.278 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.275+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3257884291}) 2024-04-26T00:43:19.279 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.279+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.279 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.279+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1343148788}) 2024-04-26T00:43:19.279 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.279+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1557401316}) 2024-04-26T00:43:19.279 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.279+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.280 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.279+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1343148788}) 2024-04-26T00:43:19.280 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.279+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2531459815}) 2024-04-26T00:43:19.281 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.279+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.282 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.279+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2486299274}) 2024-04-26T00:43:19.282 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.279+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4151152018}) 2024-04-26T00:43:19.282 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.279+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.282 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.279+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2486299274}) 2024-04-26T00:43:19.282 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.279+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2329781407}) 2024-04-26T00:43:19.284 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.283+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.284 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.283+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2680018035}) 2024-04-26T00:43:19.284 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.283+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3503637899}) 2024-04-26T00:43:19.284 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.283+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.284 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.283+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2680018035}) 2024-04-26T00:43:19.284 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.283+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=578219552}) 2024-04-26T00:43:19.285 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.283+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.286 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.283+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=915339201}) 2024-04-26T00:43:19.286 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.283+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1196328098}) 2024-04-26T00:43:19.286 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.283+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.286 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.283+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=915339201}) 2024-04-26T00:43:19.286 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.283+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=861515486}) 2024-04-26T00:43:19.288 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.287+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.288 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.287+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3637990166}) 2024-04-26T00:43:19.288 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.287+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3014953551}) 2024-04-26T00:43:19.288 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.287+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.288 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.287+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3637990166}) 2024-04-26T00:43:19.288 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.287+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=28315945}) 2024-04-26T00:43:19.290 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.287+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.290 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.287+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3184727934}) 2024-04-26T00:43:19.290 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.287+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1527942783}) 2024-04-26T00:43:19.290 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.287+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.290 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.287+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3184727934}) 2024-04-26T00:43:19.290 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.287+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3919551975}) 2024-04-26T00:43:19.291 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.291+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.292 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.291+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3125068052}) 2024-04-26T00:43:19.292 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.291+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2786928566}) 2024-04-26T00:43:19.292 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.291+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.292 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.291+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3125068052}) 2024-04-26T00:43:19.292 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.291+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1816669885}) 2024-04-26T00:43:19.293 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.291+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.294 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.291+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2203067660}) 2024-04-26T00:43:19.294 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.291+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1286540505}) 2024-04-26T00:43:19.294 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.291+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.294 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.291+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2203067660}) 2024-04-26T00:43:19.294 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.291+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2395473761}) 2024-04-26T00:43:19.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.367+0000 7f5090ae3640 10 monclient: get_monmap_and_config 2024-04-26T00:43:19.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.367+0000 7f5090ae3640 10 monclient: build_initial_monmap 2024-04-26T00:43:19.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.367+0000 7f5090ae3640 10 monclient: monmap: 2024-04-26T00:43:19.368 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:19.368 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:19.368 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:43:19.368740+0000 2024-04-26T00:43:19.368 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:43:19.368740+0000 2024-04-26T00:43:19.368 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:19.368 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:19.368 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:43:19.368 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:43:19.368 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:43:19.368 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:43:19.369 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:43:19.369 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:19.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.367+0000 7f5090ae3640 1 Processor -- start 2024-04-26T00:43:19.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.367+0000 7f5090ae3640 1 -- start start 2024-04-26T00:43:19.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.367+0000 7f5090ae3640 10 monclient: init 2024-04-26T00:43:19.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.367+0000 7f5090ae3640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:19.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.367+0000 7f5090ae3640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-04-26T00:43:19.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.367+0000 7f5090ae3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f508c0a0260 0x7f508c072e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:19.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.367+0000 7f5090ae3640 10 monclient(hunting): picked mon.noname-b con 0x7f508c0a0260 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:19.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.367+0000 7f5090ae3640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f508c0a0640 0x7f508c0a0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:19.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.367+0000 7f5090ae3640 10 monclient(hunting): picked mon.noname-c con 0x7f508c0a0640 addr v2:172.21.15.45:3300/0 2024-04-26T00:43:19.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.367+0000 7f5090ae3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f508c1337a0 addr v1:172.21.15.45:6789/0 2024-04-26T00:43:19.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.367+0000 7f5090ae3640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f508c0a12e0 con 0x7f508c1337a0 2024-04-26T00:43:19.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.367+0000 7f5090ae3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:19.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.367+0000 7f5090ae3640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f508c06d050 con 0x7f508c0a0640 2024-04-26T00:43:19.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.367+0000 7f5090ae3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:19.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.367+0000 7f5090ae3640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f508c09dcb0 con 0x7f508c0a0260 2024-04-26T00:43:19.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.367+0000 7f5090ae3640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:19.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.367+0000 7f5090ae3640 10 monclient(hunting): authenticate will time out at 2380.674364s 2024-04-26T00:43:19.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.367+0000 7f508a59c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f508c0a0260 0x7f508c072e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:19.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.367+0000 7f508a59c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f508c0a0260 0x7f508c072e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:35976/0 (socket says 172.21.15.37:35976) 2024-04-26T00:43:19.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.367+0000 7f508a59c640 1 -- 172.21.15.37:0/1753568103 learned_addr learned my addr 172.21.15.37:0/1753568103 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:19.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.367+0000 7f508a59c640 10 monclient(hunting): get_auth_request con 0x7f508c0a0260 auth_method 0 2024-04-26T00:43:19.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.367+0000 7f508a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:19.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.367+0000 7f508a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:19.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.367+0000 7f508a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:19.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.367+0000 7f508a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:19.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.367+0000 7f508a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:19.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.367+0000 7f508a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:19.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.367+0000 7f5089d9b640 1 --2- 172.21.15.37:0/1753568103 >> v2:172.21.15.45:3300/0 conn(0x7f508c0a0640 0x7f508c0a0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:19.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.367+0000 7f508a59c640 10 monclient(hunting): handle_auth_done global_id 254456 payload 274 2024-04-26T00:43:19.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.367+0000 7f508a59c640 1 -- 172.21.15.37:0/1753568103 >> v2:172.21.15.45:3300/0 conn(0x7f508c0a0640 msgr2=0x7f508c0a0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:19.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.367+0000 7f508a59c640 1 --2- 172.21.15.37:0/1753568103 >> v2:172.21.15.45:3300/0 conn(0x7f508c0a0640 0x7f508c0a0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:19.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.367+0000 7f508a59c640 1 -- 172.21.15.37:0/1753568103 >> v1:172.21.15.45:6789/0 conn(0x7f508c1337a0 legacy=0x7f508c133c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:19.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.367+0000 7f508a59c640 10 monclient: _finish_hunting 0 2024-04-26T00:43:19.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.367+0000 7f508a59c640 1 monclient: found mon.noname-b 2024-04-26T00:43:19.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.367+0000 7f508a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:19.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.367+0000 7f508a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:43:19.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.367+0000 7f508a59c640 1 -- 172.21.15.37:0/1753568103 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f508c0a16f0 con 0x7f508c0a0260 2024-04-26T00:43:19.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.371+0000 7f508a59c640 1 -- 172.21.15.37:0/1753568103 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f508c0a0260 msgr2=0x7f508c072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:19.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.371+0000 7f508a59c640 1 --2- 172.21.15.37:0/1753568103 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f508c0a0260 0x7f508c072e10 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f507c009870 tx=0x7f507c01c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65b55d1ab51329bb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:19.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.371+0000 7f508959a640 1 -- 172.21.15.37:0/1753568103 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507c0b0070 con 0x7f508c0a0260 2024-04-26T00:43:19.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.371+0000 7f508959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:19.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.371+0000 7f508959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:43:19.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.371+0000 7f508959a640 10 monclient: dump: 2024-04-26T00:43:19.375 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:19.375 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:19.375 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:19.375 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:19.375 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:19.375 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:19.375 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:19.376 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:19.376 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:19.376 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:19.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.371+0000 7f508959a640 10 monclient: _finish_auth 0 2024-04-26T00:43:19.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.371+0000 7f508959a640 10 monclient: _check_auth_tickets 2024-04-26T00:43:19.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.371+0000 7f508959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:19.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.371+0000 7f508959a640 1 -- 172.21.15.37:0/1753568103 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f507c0ab600 con 0x7f508c0a0260 2024-04-26T00:43:19.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.371+0000 7f508959a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:19.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.371+0000 7f508959a640 1 -- 172.21.15.37:0/1753568103 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507c0ab9d0 con 0x7f508c0a0260 2024-04-26T00:43:19.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.371+0000 7f5090ae3640 5 monclient: authenticate success, global_id 254456 2024-04-26T00:43:19.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.371+0000 7f5090ae3640 10 monclient: get_monmap_and_config success 2024-04-26T00:43:19.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.371+0000 7f5090ae3640 4 set_mon_vals no callback set 2024-04-26T00:43:19.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.371+0000 7f508959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:19.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.371+0000 7f508959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:43:19.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.371+0000 7f508959a640 10 monclient: dump: 2024-04-26T00:43:19.377 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:19.377 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:19.377 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:19.377 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:19.377 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:19.377 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:19.377 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:19.377 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:19.377 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:19.377 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:19.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.371+0000 7f5090ae3640 10 monclient: shutdown 2024-04-26T00:43:19.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.371+0000 7f5090ae3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:19.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.371+0000 7f5090ae3640 1 -- 172.21.15.37:0/1753568103 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f508c0a0260 msgr2=0x7f508c072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:19.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.371+0000 7f5090ae3640 1 --2- 172.21.15.37:0/1753568103 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f508c0a0260 0x7f508c072e10 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f507c009870 tx=0x7f507c01c470 comp rx=0 tx=0).stop 2024-04-26T00:43:19.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.371+0000 7f5090ae3640 1 -- 172.21.15.37:0/1753568103 shutdown_connections 2024-04-26T00:43:19.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.371+0000 7f5090ae3640 1 --2- 172.21.15.37:0/1753568103 >> v2:172.21.15.45:3300/0 conn(0x7f508c0a0640 0x7f508c0a0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:19.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.371+0000 7f5090ae3640 1 --2- 172.21.15.37:0/1753568103 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f508c0a0260 0x7f508c072e10 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:19.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.371+0000 7f5090ae3640 1 -- 172.21.15.37:0/1753568103 >> 172.21.15.37:0/1753568103 conn(0x7f508c0a1ad0 msgr2=0x7f508c0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:19.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5090ae3640 1 -- 172.21.15.37:0/1753568103 shutdown_connections 2024-04-26T00:43:19.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5090ae3640 1 -- 172.21.15.37:0/1753568103 wait complete. 2024-04-26T00:43:19.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5090ae3640 10 monclient: build_initial_monmap 2024-04-26T00:43:19.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5090ae3640 10 monclient: monmap: 2024-04-26T00:43:19.382 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:19.382 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:43:19.382 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:43:19.382 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:43:19.383 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:19.383 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:43:19.383 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:43:19.383 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:43:19.383 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:43:19.383 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:19.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5090ae3640 1 Processor -- start 2024-04-26T00:43:19.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5090ae3640 1 -- start start 2024-04-26T00:43:19.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5090ae3640 10 monclient: init 2024-04-26T00:43:19.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5090ae3640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:19.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5090ae3640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:43:19.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5090ae3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f508c0a0260 0x7f508c1cd020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:19.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5090ae3640 10 monclient(hunting): picked mon.noname-b con 0x7f508c0a0260 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:43:19.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5090ae3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f508c0a0640 0x7f508c1cd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:19.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5090ae3640 10 monclient(hunting): picked mon.noname-c con 0x7f508c0a0640 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:19.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5090ae3640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f508c1337a0 0x7f508c139480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:19.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5090ae3640 10 monclient(hunting): picked mon.noname-a con 0x7f508c1337a0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:43:19.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5090ae3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:19.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5090ae3640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f508c13feb0 con 0x7f508c1337a0 2024-04-26T00:43:19.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5090ae3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:19.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5090ae3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f508c13fd30 con 0x7f508c0a0260 2024-04-26T00:43:19.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5090ae3640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:19.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5090ae3640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f508c140030 con 0x7f508c0a0640 2024-04-26T00:43:19.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5090ae3640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:19.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5090ae3640 10 monclient(hunting): authenticate will time out at 2380.681412s 2024-04-26T00:43:19.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5089d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f508c0a0640 0x7f508c1cd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:19.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5089d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f508c0a0640 0x7f508c1cd7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:35978/0 (socket says 172.21.15.37:35978) 2024-04-26T00:43:19.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f508ad9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f508c1337a0 0x7f508c139480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:19.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5089d9b640 1 -- 172.21.15.37:0/4181231341 learned_addr learned my addr 172.21.15.37:0/4181231341 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:19.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5089d9b640 10 monclient(hunting): get_auth_request con 0x7f508c0a0640 auth_method 0 2024-04-26T00:43:19.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5089d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:19.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5089d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:19.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5089d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:19.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f508ad9d640 10 monclient(hunting): get_auth_request con 0x7f508c1337a0 auth_method 0 2024-04-26T00:43:19.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f508ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:19.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f508ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:19.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f508ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:19.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5089d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:19.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5089d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:19.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5089d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:19.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f508ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:19.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f508ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:19.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f508ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:19.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f508a59c640 1 --2- 172.21.15.37:0/4181231341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f508c0a0260 0x7f508c1cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:19.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f508a59c640 10 monclient(hunting): get_auth_request con 0x7f508c0a0260 auth_method 0 2024-04-26T00:43:19.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f508a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:19.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f508a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:19.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f508a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:19.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5089d9b640 10 monclient(hunting): handle_auth_done global_id 254459 payload 824 2024-04-26T00:43:19.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5089d9b640 1 -- 172.21.15.37:0/4181231341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f508c0a0260 msgr2=0x7f508c1cd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:19.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5089d9b640 1 --2- 172.21.15.37:0/4181231341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f508c0a0260 0x7f508c1cd020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:19.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5089d9b640 1 -- 172.21.15.37:0/4181231341 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f508c1337a0 msgr2=0x7f508c139480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:19.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5089d9b640 1 --2- 172.21.15.37:0/4181231341 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f508c1337a0 0x7f508c139480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:19.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5089d9b640 10 monclient: _finish_hunting 0 2024-04-26T00:43:19.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5089d9b640 1 monclient: found mon.noname-c 2024-04-26T00:43:19.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5089d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:19.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5089d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:43:19.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f5089d9b640 1 -- 172.21.15.37:0/4181231341 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f508c13b040 con 0x7f508c0a0640 2024-04-26T00:43:19.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.375+0000 7f508ad9d640 1 --2- 172.21.15.37:0/4181231341 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f508c1337a0 0x7f508c139480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:43:19.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.379+0000 7f5089d9b640 1 -- 172.21.15.37:0/4181231341 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f508c0a0640 msgr2=0x7f508c1cd7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:19.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.379+0000 7f5089d9b640 1 --2- 172.21.15.37:0/4181231341 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f508c0a0640 0x7f508c1cd7f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f507400c7f0 tx=0x7f507400ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:19.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.379+0000 7f5082ffd640 1 -- 172.21.15.37:0/4181231341 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5074015070 con 0x7f508c0a0640 2024-04-26T00:43:19.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.379+0000 7f5082ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:19.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.379+0000 7f5082ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:43:19.387 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.379+0000 7f5082ffd640 10 monclient: dump: 2024-04-26T00:43:19.387 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:19.387 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:19.387 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:19.387 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:19.387 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:19.387 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:19.388 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:19.388 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:19.388 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:19.388 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:19.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.379+0000 7f5082ffd640 10 monclient: _finish_auth 0 2024-04-26T00:43:19.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.379+0000 7f5082ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:43:19.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.379+0000 7f5082ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:19.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.379+0000 7f5082ffd640 1 -- 172.21.15.37:0/4181231341 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5074004240 con 0x7f508c0a0640 2024-04-26T00:43:19.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.379+0000 7f5082ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:19.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.379+0000 7f5082ffd640 1 -- 172.21.15.37:0/4181231341 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5074007830 con 0x7f508c0a0640 2024-04-26T00:43:19.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.379+0000 7f5082ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:19.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.379+0000 7f5082ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:43:19.388 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.379+0000 7f5082ffd640 10 monclient: dump: 2024-04-26T00:43:19.388 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:19.389 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:19.389 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:19.389 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:19.389 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:19.389 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:19.389 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:19.389 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:19.389 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:19.389 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:19.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.379+0000 7f5090ae3640 5 monclient: authenticate success, global_id 254459 2024-04-26T00:43:19.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.379+0000 7f5090ae3640 10 monclient: _renew_subs 2024-04-26T00:43:19.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.379+0000 7f5090ae3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:19.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.379+0000 7f5088d99640 4 set_mon_vals no callback set 2024-04-26T00:43:19.389 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.379+0000 7f5090ae3640 1 -- 172.21.15.37:0/4181231341 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f508c11c710 con 0x7f508c0a0640 2024-04-26T00:43:19.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.379+0000 7f5090ae3640 10 monclient: _renew_subs 2024-04-26T00:43:19.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.379+0000 7f5090ae3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:19.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.379+0000 7f5090ae3640 1 -- 172.21.15.37:0/4181231341 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f508c11c370 con 0x7f508c0a0640 2024-04-26T00:43:19.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.379+0000 7f5090ae3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:19.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.379+0000 7f5090ae3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:19.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.379+0000 7f5090ae3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:19.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.379+0000 7f5090ae3640 1 -- 172.21.15.37:0/4181231341 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f508c066f40 con 0x7f508c0a0640 2024-04-26T00:43:19.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.379+0000 7f5082ffd640 1 -- 172.21.15.37:0/4181231341 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f5074007a10 con 0x7f508c0a0640 2024-04-26T00:43:19.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.379+0000 7f5082ffd640 1 --2- 172.21.15.37:0/4181231341 >> v2:172.21.15.45:6812/2379918623 conn(0x7f506c035f00 0x7f506c0383c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:19.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.379+0000 7f5082ffd640 1 -- 172.21.15.37:0/4181231341 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(899..899 src has 822..899) ==== 3550+0+0 (secure 0 0 0) 0x7f5074048830 con 0x7f508c0a0640 2024-04-26T00:43:19.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.379+0000 7f508a59c640 1 --2- 172.21.15.37:0/4181231341 >> v2:172.21.15.45:6812/2379918623 conn(0x7f506c035f00 0x7f506c0383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:19.390 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.379+0000 7f508a59c640 10 monclient: get_auth_request con 0x7f506c035f00 auth_method 0 2024-04-26T00:43:19.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.383+0000 7f508a59c640 1 --2- 172.21.15.37:0/4181231341 >> v2:172.21.15.45:6812/2379918623 conn(0x7f506c035f00 0x7f506c0383c0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f507c0ab000 tx=0x7f507c0a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:19.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.383+0000 7f5082ffd640 1 -- 172.21.15.37:0/4181231341 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5074010350 con 0x7f508c0a0640 2024-04-26T00:43:19.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.383+0000 7f5082ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:19.391 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.383+0000 7f5082ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:43:19.515 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.515+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.516 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.515+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3089130988}) 2024-04-26T00:43:19.516 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.515+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1254368016}) 2024-04-26T00:43:19.516 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.515+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:19.516 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.515+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3089130988}) 2024-04-26T00:43:19.516 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:19.515+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2466572169}) 2024-04-26T00:43:19.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.543+0000 7f5090ae3640 1 -- 172.21.15.37:0/4181231341 --> v2:172.21.15.45:6812/2379918623 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f508c072330 con 0x7f506c035f00 2024-04-26T00:43:19.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.547+0000 7f5082ffd640 1 -- 172.21.15.37:0/4181231341 <== mgr.4107 v2:172.21.15.45:6812/2379918623 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33844 (secure 0 0 0) 0x7f508c072330 con 0x7f506c035f00 2024-04-26T00:43:19.548 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:43:19.548 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-04-26T00:43:19.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.547+0000 7f5090ae3640 1 -- 172.21.15.37:0/4181231341 >> v2:172.21.15.45:6812/2379918623 conn(0x7f506c035f00 msgr2=0x7f506c0383c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:19.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.547+0000 7f5090ae3640 1 --2- 172.21.15.37:0/4181231341 >> v2:172.21.15.45:6812/2379918623 conn(0x7f506c035f00 0x7f506c0383c0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f507c0ab000 tx=0x7f507c0a6000 comp rx=0 tx=0).stop 2024-04-26T00:43:19.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.547+0000 7f5090ae3640 10 monclient: shutdown 2024-04-26T00:43:19.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.547+0000 7f5090ae3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:19.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.547+0000 7f5090ae3640 1 -- 172.21.15.37:0/4181231341 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f508c0a0640 msgr2=0x7f508c1cd7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:19.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.547+0000 7f5090ae3640 1 --2- 172.21.15.37:0/4181231341 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f508c0a0640 0x7f508c1cd7f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f507400c7f0 tx=0x7f507400ccb0 comp rx=0 tx=0).stop 2024-04-26T00:43:19.550 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.547+0000 7f5090ae3640 1 -- 172.21.15.37:0/4181231341 shutdown_connections 2024-04-26T00:43:19.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.547+0000 7f5090ae3640 1 --2- 172.21.15.37:0/4181231341 >> v2:172.21.15.45:6812/2379918623 conn(0x7f506c035f00 0x7f506c0383c0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:19.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.547+0000 7f5090ae3640 1 --2- 172.21.15.37:0/4181231341 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f508c1337a0 0x7f508c139480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:19.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.547+0000 7f5090ae3640 1 --2- 172.21.15.37:0/4181231341 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f508c0a0640 0x7f508c1cd7f0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:19.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.547+0000 7f5090ae3640 1 --2- 172.21.15.37:0/4181231341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f508c0a0260 0x7f508c1cd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:19.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.547+0000 7f5090ae3640 1 -- 172.21.15.37:0/4181231341 >> 172.21.15.37:0/4181231341 conn(0x7f508c0a1ad0 msgr2=0x7f508c0708b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:19.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.547+0000 7f5090ae3640 1 -- 172.21.15.37:0/4181231341 shutdown_connections 2024-04-26T00:43:19.551 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.547+0000 7f5090ae3640 1 -- 172.21.15.37:0/4181231341 wait complete. 2024-04-26T00:43:19.567 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":1582,"stamp":"2024-04-26T00:43:17.551947+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":346348,"kb_used_data":3936,"kb_used_omap":9,"kb_used_meta":342326,"kb_avail":565884692,"statfs":{"total":579820584960,"available":579465924608,"internally_reserved":0,"allocated":4030464,"data_stored":3181916,"data_compressed":61234,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":350542496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003877"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":1755,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:42:43.917048+0000","last_change":"2024-04-26T00:20:02.279764+0000","last_active":"2024-04-26T00:42:43.917048+0000","last_peered":"2024-04-26T00:42:43.917048+0000","last_clean":"2024-04-26T00:42:43.917048+0000","last_became_active":"2024-04-26T00:20:02.278959+0000","last_became_peered":"2024-04-26T00:20:02.278959+0000","last_unstale":"2024-04-26T00:42:43.917048+0000","last_undegraded":"2024-04-26T00:42:43.917048+0000","last_fullsized":"2024-04-26T00:42:43.917048+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:23:43.610174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":1750,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:42:43.753251+0000","last_change":"2024-04-26T00:20:02.279805+0000","last_active":"2024-04-26T00:42:43.753251+0000","last_peered":"2024-04-26T00:42:43.753251+0000","last_clean":"2024-04-26T00:42:43.753251+0000","last_became_active":"2024-04-26T00:20:02.279053+0000","last_became_peered":"2024-04-26T00:20:02.279053+0000","last_unstale":"2024-04-26T00:42:43.753251+0000","last_undegraded":"2024-04-26T00:42:43.753251+0000","last_fullsized":"2024-04-26T00:42:43.753251+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:11:26.730719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":1761,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:42:43.918127+0000","last_change":"2024-04-26T00:20:02.277311+0000","last_active":"2024-04-26T00:42:43.918127+0000","last_peered":"2024-04-26T00:42:43.918127+0000","last_clean":"2024-04-26T00:42:43.918127+0000","last_became_active":"2024-04-26T00:20:02.276409+0000","last_became_peered":"2024-04-26T00:20:02.276409+0000","last_unstale":"2024-04-26T00:42:43.918127+0000","last_undegraded":"2024-04-26T00:42:43.918127+0000","last_fullsized":"2024-04-26T00:42:43.918127+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:14:50.584720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":1763,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:42:43.918487+0000","last_change":"2024-04-26T00:20:02.273965+0000","last_active":"2024-04-26T00:42:43.918487+0000","last_peered":"2024-04-26T00:42:43.918487+0000","last_clean":"2024-04-26T00:42:43.918487+0000","last_became_active":"2024-04-26T00:20:02.273213+0000","last_became_peered":"2024-04-26T00:20:02.273213+0000","last_unstale":"2024-04-26T00:42:43.918487+0000","last_undegraded":"2024-04-26T00:42:43.918487+0000","last_fullsized":"2024-04-26T00:42:43.918487+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:42:14.055403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"15'117","reported_seq":1905,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:42:43.916767+0000","last_change":"2024-04-26T00:20:02.281782+0000","last_active":"2024-04-26T00:42:43.916767+0000","last_peered":"2024-04-26T00:42:43.916767+0000","last_clean":"2024-04-26T00:42:43.916767+0000","last_became_active":"2024-04-26T00:20:02.281000+0000","last_became_peered":"2024-04-26T00:20:02.281000+0000","last_unstale":"2024-04-26T00:42:43.916767+0000","last_undegraded":"2024-04-26T00:42:43.916767+0000","last_fullsized":"2024-04-26T00:42:43.916767+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:40:30.790500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":1763,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:42:43.920813+0000","last_change":"2024-04-26T00:20:02.274249+0000","last_active":"2024-04-26T00:42:43.920813+0000","last_peered":"2024-04-26T00:42:43.920813+0000","last_clean":"2024-04-26T00:42:43.920813+0000","last_became_active":"2024-04-26T00:20:02.273745+0000","last_became_peered":"2024-04-26T00:20:02.273745+0000","last_unstale":"2024-04-26T00:42:43.920813+0000","last_undegraded":"2024-04-26T00:42:43.920813+0000","last_fullsized":"2024-04-26T00:42:43.920813+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:36:25.948539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":1761,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:42:43.918336+0000","last_change":"2024-04-26T00:20:02.474779+0000","last_active":"2024-04-26T00:42:43.918336+0000","last_peered":"2024-04-26T00:42:43.918336+0000","last_clean":"2024-04-26T00:42:43.918336+0000","last_became_active":"2024-04-26T00:20:02.473888+0000","last_became_peered":"2024-04-26T00:20:02.473888+0000","last_unstale":"2024-04-26T00:42:43.918336+0000","last_undegraded":"2024-04-26T00:42:43.918336+0000","last_fullsized":"2024-04-26T00:42:43.918336+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:33:18.962907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":1754,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:42:43.918328+0000","last_change":"2024-04-26T00:20:02.279987+0000","last_active":"2024-04-26T00:42:43.918328+0000","last_peered":"2024-04-26T00:42:43.918328+0000","last_clean":"2024-04-26T00:42:43.918328+0000","last_became_active":"2024-04-26T00:20:02.279183+0000","last_became_peered":"2024-04-26T00:20:02.279183+0000","last_unstale":"2024-04-26T00:42:43.918328+0000","last_undegraded":"2024-04-26T00:42:43.918328+0000","last_fullsized":"2024-04-26T00:42:43.918328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:45:44.508952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":1749,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:42:43.754128+0000","last_change":"2024-04-26T00:20:02.475146+0000","last_active":"2024-04-26T00:42:43.754128+0000","last_peered":"2024-04-26T00:42:43.754128+0000","last_clean":"2024-04-26T00:42:43.754128+0000","last_became_active":"2024-04-26T00:20:02.473928+0000","last_became_peered":"2024-04-26T00:20:02.473928+0000","last_unstale":"2024-04-26T00:42:43.754128+0000","last_undegraded":"2024-04-26T00:42:43.754128+0000","last_fullsized":"2024-04-26T00:42:43.754128+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:25:12.904295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":61234,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640538,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58184,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313656,"statfs":{"total":96636764160,"available":96577183744,"internally_reserved":0,"allocated":651264,"data_stored":354010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640537,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49864,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321976,"statfs":{"total":96636764160,"available":96585703424,"internally_reserved":0,"allocated":651264,"data_stored":354010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640539,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40200,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331640,"statfs":{"total":96636764160,"available":96595599360,"internally_reserved":0,"allocated":712704,"data_stored":882938,"data_compressed":30617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640539,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72008,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299832,"statfs":{"total":96636764160,"available":96563027968,"internally_reserved":0,"allocated":651264,"data_stored":354010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640539,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72068,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299772,"statfs":{"total":96636764160,"available":96562966528,"internally_reserved":0,"allocated":712704,"data_stored":882938,"data_compressed":30617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640538,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54024,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317816,"statfs":{"total":96636764160,"available":96581443584,"internally_reserved":0,"allocated":651264,"data_stored":354010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T00:43:19.568 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T00:43:19.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.727+0000 7ff760428640 10 monclient: get_monmap_and_config 2024-04-26T00:43:19.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.727+0000 7ff760428640 10 monclient: build_initial_monmap 2024-04-26T00:43:19.729 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.727+0000 7ff760428640 10 monclient: monmap: 2024-04-26T00:43:19.729 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:19.729 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:19.729 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:43:19.729820+0000 2024-04-26T00:43:19.729 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:43:19.729820+0000 2024-04-26T00:43:19.729 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:19.729 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:19.730 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:43:19.730 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:43:19.730 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:43:19.730 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:43:19.730 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:43:19.730 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:19.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.727+0000 7ff760428640 1 Processor -- start 2024-04-26T00:43:19.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.727+0000 7ff760428640 1 -- start start 2024-04-26T00:43:19.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.727+0000 7ff760428640 10 monclient: init 2024-04-26T00:43:19.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.731+0000 7ff760428640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:19.733 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.731+0000 7ff760428640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-04-26T00:43:19.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.731+0000 7ff760428640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff758136e10 addr v1:172.21.15.45:6789/0 2024-04-26T00:43:19.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.731+0000 7ff760428640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff758137900 addr v1:172.21.15.37:6789/0 2024-04-26T00:43:19.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.731+0000 7ff760428640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff75809c5b0 0x7ff75813fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:19.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.731+0000 7ff760428640 10 monclient(hunting): picked mon.noname-c con 0x7ff75809c5b0 addr v2:172.21.15.45:3300/0 2024-04-26T00:43:19.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.731+0000 7ff760428640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff75809b680 con 0x7ff758137900 2024-04-26T00:43:19.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.731+0000 7ff760428640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff75809bbd0 con 0x7ff758136e10 2024-04-26T00:43:19.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.731+0000 7ff760428640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:19.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.731+0000 7ff760428640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7ff75809b8f0 con 0x7ff75809c5b0 2024-04-26T00:43:19.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.731+0000 7ff760428640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:19.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.731+0000 7ff760428640 10 monclient(hunting): authenticate will time out at 2381.036166s 2024-04-26T00:43:19.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.731+0000 7ff75d9c3640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7ff758137900 0x7ff75813c340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:49496/0 (socket says 172.21.15.37:49496) 2024-04-26T00:43:19.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.731+0000 7ff75d9c3640 1 -- 172.21.15.37:0/2244044089 learned_addr learned my addr 172.21.15.37:0/2244044089 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:43:19.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.731+0000 7ff75e9c5640 1 --2- 172.21.15.37:0/2244044089 >> v2:172.21.15.45:3300/0 conn(0x7ff75809c5b0 0x7ff75813fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:19.734 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.731+0000 7ff75e9c5640 10 monclient(hunting): get_auth_request con 0x7ff75809c5b0 auth_method 0 2024-04-26T00:43:19.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.731+0000 7ff75e9c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:19.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.731+0000 7ff75e9c5640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:19.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.731+0000 7ff75e9c5640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:19.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.731+0000 7ff75e9c5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:19.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.731+0000 7ff75e9c5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:19.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.731+0000 7ff75e9c5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:19.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.731+0000 7ff75e9c5640 10 monclient(hunting): handle_auth_done global_id 274219 payload 274 2024-04-26T00:43:19.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.731+0000 7ff75e9c5640 1 -- 172.21.15.37:0/2244044089 >> v1:172.21.15.45:6789/0 conn(0x7ff758136e10 legacy=0x7ff7581371f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:19.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.731+0000 7ff75e9c5640 1 -- 172.21.15.37:0/2244044089 >> v1:172.21.15.37:6789/0 conn(0x7ff758137900 legacy=0x7ff75813c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:19.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.731+0000 7ff75e9c5640 10 monclient: _finish_hunting 0 2024-04-26T00:43:19.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.731+0000 7ff75e9c5640 1 monclient: found mon.noname-c 2024-04-26T00:43:19.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.731+0000 7ff75e9c5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:19.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.731+0000 7ff75e9c5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.45:3300/0 2024-04-26T00:43:19.736 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.731+0000 7ff75e9c5640 1 -- 172.21.15.37:0/2244044089 --> v2:172.21.15.45:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7580a5d90 con 0x7ff75809c5b0 2024-04-26T00:43:19.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.735+0000 7ff75e9c5640 1 -- 172.21.15.37:0/2244044089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff75809c5b0 msgr2=0x7ff75813fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:19.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.735+0000 7ff75e9c5640 1 --2- 172.21.15.37:0/2244044089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff75809c5b0 0x7ff75813fa70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff750009d90 tx=0x7ff75001c7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c16cc68b130ddc13 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:19.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.735+0000 7ff75d1c2640 1 -- 172.21.15.37:0/2244044089 <== mon.1 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7500b2070 con 0x7ff75809c5b0 2024-04-26T00:43:19.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.735+0000 7ff75d1c2640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:19.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.735+0000 7ff75d1c2640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:43:19.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.735+0000 7ff75d1c2640 10 monclient: dump: 2024-04-26T00:43:19.737 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:19.737 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:19.737 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:19.737 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:19.737 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:19.737 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:19.737 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:19.737 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:19.738 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:19.738 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:19.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.735+0000 7ff75d1c2640 10 monclient: _finish_auth 0 2024-04-26T00:43:19.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.735+0000 7ff75d1c2640 10 monclient: _check_auth_tickets 2024-04-26T00:43:19.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.735+0000 7ff75d1c2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:19.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.735+0000 7ff75d1c2640 1 -- 172.21.15.37:0/2244044089 <== mon.1 v2:172.21.15.45:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7500a5860 con 0x7ff75809c5b0 2024-04-26T00:43:19.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.735+0000 7ff760428640 5 monclient: authenticate success, global_id 274219 2024-04-26T00:43:19.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.735+0000 7ff760428640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:43:19.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.735+0000 7ff75d1c2640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:19.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.735+0000 7ff760428640 10 monclient: get_monmap_and_config success 2024-04-26T00:43:19.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.735+0000 7ff75d1c2640 1 -- 172.21.15.37:0/2244044089 <== mon.1 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7500a5c30 con 0x7ff75809c5b0 2024-04-26T00:43:19.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.735+0000 7ff75d1c2640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:19.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.735+0000 7ff760428640 4 set_mon_vals no callback set 2024-04-26T00:43:19.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.735+0000 7ff75d1c2640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T00:43:19.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.735+0000 7ff75d1c2640 10 monclient: dump: 2024-04-26T00:43:19.739 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:19.739 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:19.739 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:19.739 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:19.739 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:19.739 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:19.739 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:19.739 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:19.739 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:19.740 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:19.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.735+0000 7ff760428640 10 monclient: shutdown 2024-04-26T00:43:19.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.735+0000 7ff760428640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:19.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.735+0000 7ff760428640 1 -- 172.21.15.37:0/2244044089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff75809c5b0 msgr2=0x7ff75813fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:19.746 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.735+0000 7ff760428640 1 --2- 172.21.15.37:0/2244044089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff75809c5b0 0x7ff75813fa70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff750009d90 tx=0x7ff75001c7b0 comp rx=0 tx=0).stop 2024-04-26T00:43:19.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.735+0000 7ff760428640 1 -- 172.21.15.37:0/2244044089 shutdown_connections 2024-04-26T00:43:19.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.735+0000 7ff760428640 1 --2- 172.21.15.37:0/2244044089 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff75809c5b0 0x7ff75813fa70 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:19.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.735+0000 7ff760428640 1 -- 172.21.15.37:0/2244044089 >> 172.21.15.37:0/2244044089 conn(0x7ff7580a64e0 msgr2=0x7ff7580a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:19.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.735+0000 7ff760428640 1 -- 172.21.15.37:0/2244044089 shutdown_connections 2024-04-26T00:43:19.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.735+0000 7ff760428640 1 -- 172.21.15.37:0/2244044089 wait complete. 2024-04-26T00:43:19.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.735+0000 7ff760428640 10 monclient: build_initial_monmap 2024-04-26T00:43:19.747 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.735+0000 7ff760428640 10 monclient: monmap: 2024-04-26T00:43:19.747 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:19.747 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:43:19.747 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:43:19.747 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:43:19.747 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:19.747 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:43:19.747 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:43:19.747 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:43:19.748 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:43:19.748 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:19.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.735+0000 7ff760428640 1 Processor -- start 2024-04-26T00:43:19.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.735+0000 7ff760428640 1 -- start start 2024-04-26T00:43:19.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.735+0000 7ff760428640 10 monclient: init 2024-04-26T00:43:19.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff760428640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:19.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff760428640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:43:19.748 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff760428640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff75809c5b0 0x7ff7581cee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:19.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff760428640 10 monclient(hunting): picked mon.noname-c con 0x7ff75809c5b0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:19.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff760428640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff758136e10 0x7ff758135e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:19.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff760428640 10 monclient(hunting): picked mon.noname-a con 0x7ff758136e10 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:43:19.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff760428640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff758137900 0x7ff75813bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:19.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff760428640 10 monclient(hunting): picked mon.noname-b con 0x7ff758137900 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:43:19.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff760428640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:19.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff760428640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff758142b60 con 0x7ff758136e10 2024-04-26T00:43:19.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff760428640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:19.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff760428640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff7581429e0 con 0x7ff758137900 2024-04-26T00:43:19.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff760428640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:19.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff760428640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff758142ce0 con 0x7ff75809c5b0 2024-04-26T00:43:19.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff760428640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:19.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff760428640 10 monclient(hunting): authenticate will time out at 2381.044185s 2024-04-26T00:43:19.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff75d9c3640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff758136e10 0x7ff758135e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:19.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff75e1c4640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff75809c5b0 0x7ff7581cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:19.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff75d9c3640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff758136e10 0x7ff758135e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58070/0 (socket says 172.21.15.37:58070) 2024-04-26T00:43:19.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff75d9c3640 1 -- 172.21.15.37:0/2861847514 learned_addr learned my addr 172.21.15.37:0/2861847514 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:19.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff75e1c4640 10 monclient(hunting): get_auth_request con 0x7ff75809c5b0 auth_method 0 2024-04-26T00:43:19.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff75e1c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:19.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff75e1c4640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:19.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff75e1c4640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:19.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff75d9c3640 10 monclient(hunting): get_auth_request con 0x7ff758136e10 auth_method 0 2024-04-26T00:43:19.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff75d9c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:19.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff75d9c3640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:19.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff75d9c3640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:19.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff75e1c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:19.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff75e1c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:19.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff75e1c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:19.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff75d9c3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:19.751 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff75d9c3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:19.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff75d9c3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:19.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff75e9c5640 1 --2- 172.21.15.37:0/2861847514 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff758137900 0x7ff75813bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:19.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff75e9c5640 10 monclient(hunting): get_auth_request con 0x7ff758137900 auth_method 0 2024-04-26T00:43:19.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff75e9c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:19.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff75e9c5640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:19.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff75e9c5640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:19.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff75e1c4640 10 monclient(hunting): handle_auth_done global_id 254462 payload 824 2024-04-26T00:43:19.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff75e1c4640 1 -- 172.21.15.37:0/2861847514 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff758137900 msgr2=0x7ff75813bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:19.752 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff75e1c4640 1 --2- 172.21.15.37:0/2861847514 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff758137900 0x7ff75813bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:19.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff75e1c4640 1 -- 172.21.15.37:0/2861847514 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff758136e10 msgr2=0x7ff758135e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:19.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff75e1c4640 1 --2- 172.21.15.37:0/2861847514 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff758136e10 0x7ff758135e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:19.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff75e1c4640 10 monclient: _finish_hunting 0 2024-04-26T00:43:19.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff75e1c4640 1 monclient: found mon.noname-c 2024-04-26T00:43:19.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff75e1c4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:19.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff75e1c4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:43:19.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff75e1c4640 1 -- 172.21.15.37:0/2861847514 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7580a3030 con 0x7ff75809c5b0 2024-04-26T00:43:19.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff75d9c3640 1 --2- 172.21.15.37:0/2861847514 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff758136e10 0x7ff758135e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:43:19.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff75e1c4640 1 -- 172.21.15.37:0/2861847514 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff75809c5b0 msgr2=0x7ff7581cee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:19.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff75e1c4640 1 --2- 172.21.15.37:0/2861847514 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff75809c5b0 0x7ff7581cee70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff75400c3e0 tx=0x7ff75400c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:19.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff74effd640 1 -- 172.21.15.37:0/2861847514 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7540119a0 con 0x7ff75809c5b0 2024-04-26T00:43:19.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff74effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:19.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff74effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:43:19.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff74effd640 10 monclient: dump: 2024-04-26T00:43:19.754 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:19.754 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:19.754 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:19.754 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:19.754 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:19.754 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:19.755 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:19.755 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:19.755 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:19.755 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:19.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff74effd640 10 monclient: _finish_auth 0 2024-04-26T00:43:19.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff74effd640 10 monclient: _check_auth_tickets 2024-04-26T00:43:19.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff74effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:19.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff74effd640 1 -- 172.21.15.37:0/2861847514 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff754017070 con 0x7ff75809c5b0 2024-04-26T00:43:19.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff74effd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:19.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff74effd640 1 -- 172.21.15.37:0/2861847514 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff754011d80 con 0x7ff75809c5b0 2024-04-26T00:43:19.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff74effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:19.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff74effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:43:19.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff74effd640 10 monclient: dump: 2024-04-26T00:43:19.756 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:19.756 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:19.756 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:19.756 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:19.756 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:19.756 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:19.756 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:19.756 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:19.756 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:19.756 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:19.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff760428640 5 monclient: authenticate success, global_id 254462 2024-04-26T00:43:19.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff760428640 10 monclient: _renew_subs 2024-04-26T00:43:19.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff760428640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:19.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff760428640 1 -- 172.21.15.37:0/2861847514 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff75813fae0 con 0x7ff75809c5b0 2024-04-26T00:43:19.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff74ffff640 4 set_mon_vals no callback set 2024-04-26T00:43:19.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff760428640 10 monclient: _renew_subs 2024-04-26T00:43:19.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff760428640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:19.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.739+0000 7ff760428640 1 -- 172.21.15.37:0/2861847514 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff75809aca0 con 0x7ff75809c5b0 2024-04-26T00:43:19.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.743+0000 7ff74effd640 1 -- 172.21.15.37:0/2861847514 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff75401c460 con 0x7ff75809c5b0 2024-04-26T00:43:19.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.743+0000 7ff760428640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:19.757 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.743+0000 7ff760428640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:19.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.743+0000 7ff760428640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:19.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.743+0000 7ff760428640 1 -- 172.21.15.37:0/2861847514 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff758075910 con 0x7ff75809c5b0 2024-04-26T00:43:19.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.743+0000 7ff74effd640 1 --2- 172.21.15.37:0/2861847514 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff73c035e80 0x7ff73c038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:19.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.743+0000 7ff74effd640 1 -- 172.21.15.37:0/2861847514 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(899..899 src has 822..899) ==== 3550+0+0 (secure 0 0 0) 0x7ff7540487d0 con 0x7ff75809c5b0 2024-04-26T00:43:19.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.743+0000 7ff75d9c3640 1 --2- 172.21.15.37:0/2861847514 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff73c035e80 0x7ff73c038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:19.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.743+0000 7ff75d9c3640 10 monclient: get_auth_request con 0x7ff73c035e80 auth_method 0 2024-04-26T00:43:19.758 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.743+0000 7ff75d9c3640 1 --2- 172.21.15.37:0/2861847514 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff73c035e80 0x7ff73c038340 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ff748007d60 tx=0x7ff748007cb0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:19.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.747+0000 7ff74effd640 1 -- 172.21.15.37:0/2861847514 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff7580a3030 con 0x7ff75809c5b0 2024-04-26T00:43:19.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.747+0000 7ff74effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:19.759 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.747+0000 7ff74effd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:43:19.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.895+0000 7ff760428640 1 -- 172.21.15.37:0/2861847514 --> v2:172.21.15.45:6812/2379918623 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff758066f40 con 0x7ff73c035e80 2024-04-26T00:43:19.901 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.899+0000 7ff74effd640 1 -- 172.21.15.37:0/2861847514 <== mgr.4107 v2:172.21.15.45:6812/2379918623 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33844 (secure 0 0 0) 0x7ff758066f40 con 0x7ff73c035e80 2024-04-26T00:43:19.902 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:43:19.903 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-04-26T00:43:19.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.903+0000 7ff760428640 1 -- 172.21.15.37:0/2861847514 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff73c035e80 msgr2=0x7ff73c038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:19.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.903+0000 7ff760428640 1 --2- 172.21.15.37:0/2861847514 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff73c035e80 0x7ff73c038340 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ff748007d60 tx=0x7ff748007cb0 comp rx=0 tx=0).stop 2024-04-26T00:43:19.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.903+0000 7ff760428640 10 monclient: shutdown 2024-04-26T00:43:19.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.903+0000 7ff760428640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:19.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.903+0000 7ff760428640 1 -- 172.21.15.37:0/2861847514 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff75809c5b0 msgr2=0x7ff7581cee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:19.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.903+0000 7ff760428640 1 --2- 172.21.15.37:0/2861847514 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff75809c5b0 0x7ff7581cee70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff75400c3e0 tx=0x7ff75400c8a0 comp rx=0 tx=0).stop 2024-04-26T00:43:19.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.903+0000 7ff760428640 1 -- 172.21.15.37:0/2861847514 shutdown_connections 2024-04-26T00:43:19.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.903+0000 7ff760428640 1 --2- 172.21.15.37:0/2861847514 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff73c035e80 0x7ff73c038340 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:19.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.903+0000 7ff760428640 1 --2- 172.21.15.37:0/2861847514 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff758137900 0x7ff75813bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:19.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.903+0000 7ff760428640 1 --2- 172.21.15.37:0/2861847514 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff758136e10 0x7ff758135e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:19.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.903+0000 7ff760428640 1 --2- 172.21.15.37:0/2861847514 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff75809c5b0 0x7ff7581cee70 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:19.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.903+0000 7ff760428640 1 -- 172.21.15.37:0/2861847514 >> 172.21.15.37:0/2861847514 conn(0x7ff7580a64e0 msgr2=0x7ff7580a1940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:19.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.903+0000 7ff760428640 1 -- 172.21.15.37:0/2861847514 shutdown_connections 2024-04-26T00:43:19.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:19.903+0000 7ff760428640 1 -- 172.21.15.37:0/2861847514 wait complete. 2024-04-26T00:43:19.919 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":1583,"stamp":"2024-04-26T00:43:19.552564+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":346348,"kb_used_data":3936,"kb_used_omap":9,"kb_used_meta":342326,"kb_avail":565884692,"statfs":{"total":579820584960,"available":579465924608,"internally_reserved":0,"allocated":4030464,"data_stored":3181916,"data_compressed":61234,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":350542496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003799"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":1755,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:42:43.917048+0000","last_change":"2024-04-26T00:20:02.279764+0000","last_active":"2024-04-26T00:42:43.917048+0000","last_peered":"2024-04-26T00:42:43.917048+0000","last_clean":"2024-04-26T00:42:43.917048+0000","last_became_active":"2024-04-26T00:20:02.278959+0000","last_became_peered":"2024-04-26T00:20:02.278959+0000","last_unstale":"2024-04-26T00:42:43.917048+0000","last_undegraded":"2024-04-26T00:42:43.917048+0000","last_fullsized":"2024-04-26T00:42:43.917048+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:23:43.610174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":1750,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:42:43.753251+0000","last_change":"2024-04-26T00:20:02.279805+0000","last_active":"2024-04-26T00:42:43.753251+0000","last_peered":"2024-04-26T00:42:43.753251+0000","last_clean":"2024-04-26T00:42:43.753251+0000","last_became_active":"2024-04-26T00:20:02.279053+0000","last_became_peered":"2024-04-26T00:20:02.279053+0000","last_unstale":"2024-04-26T00:42:43.753251+0000","last_undegraded":"2024-04-26T00:42:43.753251+0000","last_fullsized":"2024-04-26T00:42:43.753251+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:11:26.730719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":1761,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:42:43.918127+0000","last_change":"2024-04-26T00:20:02.277311+0000","last_active":"2024-04-26T00:42:43.918127+0000","last_peered":"2024-04-26T00:42:43.918127+0000","last_clean":"2024-04-26T00:42:43.918127+0000","last_became_active":"2024-04-26T00:20:02.276409+0000","last_became_peered":"2024-04-26T00:20:02.276409+0000","last_unstale":"2024-04-26T00:42:43.918127+0000","last_undegraded":"2024-04-26T00:42:43.918127+0000","last_fullsized":"2024-04-26T00:42:43.918127+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:14:50.584720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":1763,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:42:43.918487+0000","last_change":"2024-04-26T00:20:02.273965+0000","last_active":"2024-04-26T00:42:43.918487+0000","last_peered":"2024-04-26T00:42:43.918487+0000","last_clean":"2024-04-26T00:42:43.918487+0000","last_became_active":"2024-04-26T00:20:02.273213+0000","last_became_peered":"2024-04-26T00:20:02.273213+0000","last_unstale":"2024-04-26T00:42:43.918487+0000","last_undegraded":"2024-04-26T00:42:43.918487+0000","last_fullsized":"2024-04-26T00:42:43.918487+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:42:14.055403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"15'117","reported_seq":1905,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:42:43.916767+0000","last_change":"2024-04-26T00:20:02.281782+0000","last_active":"2024-04-26T00:42:43.916767+0000","last_peered":"2024-04-26T00:42:43.916767+0000","last_clean":"2024-04-26T00:42:43.916767+0000","last_became_active":"2024-04-26T00:20:02.281000+0000","last_became_peered":"2024-04-26T00:20:02.281000+0000","last_unstale":"2024-04-26T00:42:43.916767+0000","last_undegraded":"2024-04-26T00:42:43.916767+0000","last_fullsized":"2024-04-26T00:42:43.916767+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:40:30.790500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":1763,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:42:43.920813+0000","last_change":"2024-04-26T00:20:02.274249+0000","last_active":"2024-04-26T00:42:43.920813+0000","last_peered":"2024-04-26T00:42:43.920813+0000","last_clean":"2024-04-26T00:42:43.920813+0000","last_became_active":"2024-04-26T00:20:02.273745+0000","last_became_peered":"2024-04-26T00:20:02.273745+0000","last_unstale":"2024-04-26T00:42:43.920813+0000","last_undegraded":"2024-04-26T00:42:43.920813+0000","last_fullsized":"2024-04-26T00:42:43.920813+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:36:25.948539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":1761,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:42:43.918336+0000","last_change":"2024-04-26T00:20:02.474779+0000","last_active":"2024-04-26T00:42:43.918336+0000","last_peered":"2024-04-26T00:42:43.918336+0000","last_clean":"2024-04-26T00:42:43.918336+0000","last_became_active":"2024-04-26T00:20:02.473888+0000","last_became_peered":"2024-04-26T00:20:02.473888+0000","last_unstale":"2024-04-26T00:42:43.918336+0000","last_undegraded":"2024-04-26T00:42:43.918336+0000","last_fullsized":"2024-04-26T00:42:43.918336+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:33:18.962907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":1754,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:42:43.918328+0000","last_change":"2024-04-26T00:20:02.279987+0000","last_active":"2024-04-26T00:42:43.918328+0000","last_peered":"2024-04-26T00:42:43.918328+0000","last_clean":"2024-04-26T00:42:43.918328+0000","last_became_active":"2024-04-26T00:20:02.279183+0000","last_became_peered":"2024-04-26T00:20:02.279183+0000","last_unstale":"2024-04-26T00:42:43.918328+0000","last_undegraded":"2024-04-26T00:42:43.918328+0000","last_fullsized":"2024-04-26T00:42:43.918328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:45:44.508952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":1749,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:42:43.754128+0000","last_change":"2024-04-26T00:20:02.475146+0000","last_active":"2024-04-26T00:42:43.754128+0000","last_peered":"2024-04-26T00:42:43.754128+0000","last_clean":"2024-04-26T00:42:43.754128+0000","last_became_active":"2024-04-26T00:20:02.473928+0000","last_became_peered":"2024-04-26T00:20:02.473928+0000","last_unstale":"2024-04-26T00:42:43.754128+0000","last_undegraded":"2024-04-26T00:42:43.754128+0000","last_fullsized":"2024-04-26T00:42:43.754128+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:25:12.904295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":61234,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640538,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58184,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313656,"statfs":{"total":96636764160,"available":96577183744,"internally_reserved":0,"allocated":651264,"data_stored":354010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640537,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49864,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321976,"statfs":{"total":96636764160,"available":96585703424,"internally_reserved":0,"allocated":651264,"data_stored":354010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640539,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40200,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331640,"statfs":{"total":96636764160,"available":96595599360,"internally_reserved":0,"allocated":712704,"data_stored":882938,"data_compressed":30617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640539,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72008,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299832,"statfs":{"total":96636764160,"available":96563027968,"internally_reserved":0,"allocated":651264,"data_stored":354010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640539,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72068,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299772,"statfs":{"total":96636764160,"available":96562966528,"internally_reserved":0,"allocated":712704,"data_stored":882938,"data_compressed":30617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640538,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54024,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317816,"statfs":{"total":96636764160,"available":96581443584,"internally_reserved":0,"allocated":651264,"data_stored":354010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T00:43:19.919 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-26T00:43:19.920 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T00:43:20.021 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.019+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:20.022 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.019+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3953400173}) 2024-04-26T00:43:20.022 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.019+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2560725858}) 2024-04-26T00:43:20.022 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.019+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:20.022 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.019+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3953400173}) 2024-04-26T00:43:20.022 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.019+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=128454378}) 2024-04-26T00:43:20.028 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.027+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:20.028 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.027+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3111941556}) 2024-04-26T00:43:20.028 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.027+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=241708516}) 2024-04-26T00:43:20.028 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.027+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:20.028 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.027+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3111941556}) 2024-04-26T00:43:20.028 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.027+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=654364137}) 2024-04-26T00:43:20.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.031+0000 7f8abb50d640 10 monclient: get_monmap_and_config 2024-04-26T00:43:20.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.031+0000 7f8abb50d640 10 monclient: build_initial_monmap 2024-04-26T00:43:20.033 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.031+0000 7f8abb50d640 10 monclient: monmap: 2024-04-26T00:43:20.034 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:20.034 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:20.034 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:43:20.034781+0000 2024-04-26T00:43:20.034 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:43:20.034781+0000 2024-04-26T00:43:20.034 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:20.034 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:20.034 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:43:20.034 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:43:20.034 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:43:20.034 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:43:20.034 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:43:20.034 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:20.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.031+0000 7f8abb50d640 1 Processor -- start 2024-04-26T00:43:20.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.031+0000 7f8abb50d640 1 -- start start 2024-04-26T00:43:20.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.031+0000 7f8abb50d640 10 monclient: init 2024-04-26T00:43:20.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8abb50d640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:20.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8abb50d640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-04-26T00:43:20.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8abb50d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f8ab4136eb0 0x7f8ab4137290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:20.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8abb50d640 10 monclient(hunting): picked mon.noname-c con 0x7f8ab4136eb0 addr v2:172.21.15.45:3300/0 2024-04-26T00:43:20.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8abb50d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f8ab41377d0 0x7f8ab4133b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:20.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8abb50d640 10 monclient(hunting): picked mon.noname-a con 0x7f8ab41377d0 addr v2:172.21.15.37:3300/0 2024-04-26T00:43:20.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8abb50d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8ab41340d0 0x7f8ab4134560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:20.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8abb50d640 10 monclient(hunting): picked mon.noname-b con 0x7f8ab41340d0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:20.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8abb50d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:20.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8abb50d640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f8ab409c7d0 con 0x7f8ab41377d0 2024-04-26T00:43:20.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8abb50d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:20.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8abb50d640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f8ab409b8c0 con 0x7f8ab4136eb0 2024-04-26T00:43:20.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8abb50d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:20.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8abb50d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8ab409b300 con 0x7f8ab41340d0 2024-04-26T00:43:20.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8abb50d640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:20.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8abb50d640 10 monclient(hunting): authenticate will time out at 2381.340863s 2024-04-26T00:43:20.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8ab9aaa640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8ab41340d0 0x7f8ab4134560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:20.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8ab8aa8640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f8ab41377d0 0x7f8ab4133b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:20.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8ab9aaa640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8ab41340d0 0x7f8ab4134560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51258/0 (socket says 172.21.15.37:51258) 2024-04-26T00:43:20.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8ab8aa8640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f8ab41377d0 0x7f8ab4133b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58084/0 (socket says 172.21.15.37:58084) 2024-04-26T00:43:20.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8ab9aaa640 1 -- 172.21.15.37:0/3975868630 learned_addr learned my addr 172.21.15.37:0/3975868630 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:20.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8ab9aaa640 10 monclient(hunting): get_auth_request con 0x7f8ab41340d0 auth_method 0 2024-04-26T00:43:20.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8ab9aaa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:20.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8ab9aaa640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:20.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8ab9aaa640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:20.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8ab8aa8640 10 monclient(hunting): get_auth_request con 0x7f8ab41377d0 auth_method 0 2024-04-26T00:43:20.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8ab8aa8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:20.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8ab8aa8640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:20.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8ab8aa8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:20.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8ab9aaa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:20.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8ab9aaa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:20.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8ab9aaa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:20.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8ab8aa8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:20.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8ab8aa8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:20.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8ab8aa8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:20.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8ab92a9640 1 --2- 172.21.15.37:0/3975868630 >> v2:172.21.15.45:3300/0 conn(0x7f8ab4136eb0 0x7f8ab4137290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:20.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8ab92a9640 10 monclient(hunting): get_auth_request con 0x7f8ab4136eb0 auth_method 0 2024-04-26T00:43:20.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8ab92a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:20.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8ab92a9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:20.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8ab92a9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:20.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8ab9aaa640 10 monclient(hunting): handle_auth_done global_id 254465 payload 274 2024-04-26T00:43:20.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8ab9aaa640 1 -- 172.21.15.37:0/3975868630 >> v2:172.21.15.45:3300/0 conn(0x7f8ab4136eb0 msgr2=0x7f8ab4137290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:20.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8ab9aaa640 1 --2- 172.21.15.37:0/3975868630 >> v2:172.21.15.45:3300/0 conn(0x7f8ab4136eb0 0x7f8ab4137290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:20.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8ab9aaa640 1 -- 172.21.15.37:0/3975868630 >> v2:172.21.15.37:3300/0 conn(0x7f8ab41377d0 msgr2=0x7f8ab4133b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:20.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8ab9aaa640 1 --2- 172.21.15.37:0/3975868630 >> v2:172.21.15.37:3300/0 conn(0x7f8ab41377d0 0x7f8ab4133b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:20.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8ab9aaa640 10 monclient: _finish_hunting 0 2024-04-26T00:43:20.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8ab9aaa640 1 monclient: found mon.noname-b 2024-04-26T00:43:20.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8ab9aaa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:20.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8ab9aaa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:43:20.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8ab9aaa640 1 -- 172.21.15.37:0/3975868630 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ab409b5e0 con 0x7f8ab41340d0 2024-04-26T00:43:20.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.035+0000 7f8ab8aa8640 1 --2- 172.21.15.37:0/3975868630 >> v2:172.21.15.37:3300/0 conn(0x7f8ab41377d0 0x7f8ab4133b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:43:20.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.039+0000 7f8ab9aaa640 1 -- 172.21.15.37:0/3975868630 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8ab41340d0 msgr2=0x7f8ab4134560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:20.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.039+0000 7f8ab9aaa640 1 --2- 172.21.15.37:0/3975868630 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8ab41340d0 0x7f8ab4134560 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8ab40a5990 tx=0x7f8aac01c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bae6795ff6d38e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:20.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.039+0000 7f8aabfff640 1 -- 172.21.15.37:0/3975868630 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aac0a5640 con 0x7f8ab41340d0 2024-04-26T00:43:20.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.039+0000 7f8aabfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:20.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.039+0000 7f8aabfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:43:20.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.039+0000 7f8aabfff640 10 monclient: dump: 2024-04-26T00:43:20.042 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:20.042 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:20.042 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:20.042 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:20.042 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:20.042 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:20.042 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:20.042 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:20.043 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:20.043 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:20.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.039+0000 7f8aabfff640 10 monclient: _finish_auth 0 2024-04-26T00:43:20.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.039+0000 7f8aabfff640 10 monclient: _check_auth_tickets 2024-04-26T00:43:20.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.039+0000 7f8aabfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:20.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.039+0000 7f8aabfff640 1 -- 172.21.15.37:0/3975868630 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8aac0a57e0 con 0x7f8ab41340d0 2024-04-26T00:43:20.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.039+0000 7f8aabfff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:20.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.039+0000 7f8aabfff640 1 -- 172.21.15.37:0/3975868630 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aac0a5bb0 con 0x7f8ab41340d0 2024-04-26T00:43:20.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.039+0000 7f8abb50d640 5 monclient: authenticate success, global_id 254465 2024-04-26T00:43:20.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.039+0000 7f8abb50d640 10 monclient: get_monmap_and_config success 2024-04-26T00:43:20.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.039+0000 7f8abb50d640 4 set_mon_vals no callback set 2024-04-26T00:43:20.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.039+0000 7f8aabfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:20.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.039+0000 7f8aabfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:43:20.043 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.039+0000 7f8aabfff640 10 monclient: dump: 2024-04-26T00:43:20.043 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:20.044 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:20.044 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:20.044 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:20.044 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:20.044 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:20.044 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:20.044 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:20.044 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:20.044 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:20.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.039+0000 7f8abb50d640 10 monclient: shutdown 2024-04-26T00:43:20.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.039+0000 7f8abb50d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:20.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.039+0000 7f8abb50d640 1 -- 172.21.15.37:0/3975868630 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8ab41340d0 msgr2=0x7f8ab4134560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:20.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.039+0000 7f8abb50d640 1 --2- 172.21.15.37:0/3975868630 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8ab41340d0 0x7f8ab4134560 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8ab40a5990 tx=0x7f8aac01c8b0 comp rx=0 tx=0).stop 2024-04-26T00:43:20.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.039+0000 7f8abb50d640 1 -- 172.21.15.37:0/3975868630 shutdown_connections 2024-04-26T00:43:20.045 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.039+0000 7f8abb50d640 1 --2- 172.21.15.37:0/3975868630 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8ab41340d0 0x7f8ab4134560 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:20.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.039+0000 7f8abb50d640 1 --2- 172.21.15.37:0/3975868630 >> v2:172.21.15.37:3300/0 conn(0x7f8ab41377d0 0x7f8ab4133b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:20.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.039+0000 7f8abb50d640 1 --2- 172.21.15.37:0/3975868630 >> v2:172.21.15.45:3300/0 conn(0x7f8ab4136eb0 0x7f8ab4137290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:20.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.039+0000 7f8abb50d640 1 -- 172.21.15.37:0/3975868630 >> 172.21.15.37:0/3975868630 conn(0x7f8ab40a6300 msgr2=0x7f8ab40a6710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:20.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.039+0000 7f8abb50d640 1 -- 172.21.15.37:0/3975868630 shutdown_connections 2024-04-26T00:43:20.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.039+0000 7f8abb50d640 1 -- 172.21.15.37:0/3975868630 wait complete. 2024-04-26T00:43:20.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8abb50d640 10 monclient: build_initial_monmap 2024-04-26T00:43:20.046 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8abb50d640 10 monclient: monmap: 2024-04-26T00:43:20.046 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:20.046 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:43:20.046 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:43:20.046 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:43:20.046 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:20.046 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:43:20.046 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:43:20.047 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:43:20.047 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:43:20.047 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:20.047 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.043+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:20.047 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.043+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4021564310}) 2024-04-26T00:43:20.047 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.043+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1552778341}) 2024-04-26T00:43:20.047 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.043+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:20.047 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.043+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4021564310}) 2024-04-26T00:43:20.047 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.043+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2192535485}) 2024-04-26T00:43:20.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8abb50d640 1 Processor -- start 2024-04-26T00:43:20.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8abb50d640 1 -- start start 2024-04-26T00:43:20.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8abb50d640 10 monclient: init 2024-04-26T00:43:20.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8abb50d640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:20.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8abb50d640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:43:20.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8abb50d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ab41340d0 0x7f8ab4130eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:20.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8abb50d640 10 monclient(hunting): picked mon.noname-b con 0x7f8ab41340d0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:43:20.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8abb50d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8ab4136eb0 0x7f8ab41313f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:20.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8abb50d640 10 monclient(hunting): picked mon.noname-c con 0x7f8ab4136eb0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:20.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8abb50d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8ab41377d0 0x7f8ab4137fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:20.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8abb50d640 10 monclient(hunting): picked mon.noname-a con 0x7f8ab41377d0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:43:20.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8abb50d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:20.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8abb50d640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f8ab409a9e0 con 0x7f8ab41377d0 2024-04-26T00:43:20.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8abb50d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:20.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8abb50d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8ab409ab60 con 0x7f8ab41340d0 2024-04-26T00:43:20.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8abb50d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:20.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8abb50d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8ab4140c80 con 0x7f8ab4136eb0 2024-04-26T00:43:20.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8abb50d640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:20.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8abb50d640 10 monclient(hunting): authenticate will time out at 2381.349730s 2024-04-26T00:43:20.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8ab8aa8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8ab4136eb0 0x7f8ab41313f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:20.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8ab9aaa640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8ab41377d0 0x7f8ab4137fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:20.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8ab8aa8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8ab4136eb0 0x7f8ab41313f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51270/0 (socket says 172.21.15.37:51270) 2024-04-26T00:43:20.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8ab8aa8640 1 -- 172.21.15.37:0/2381584027 learned_addr learned my addr 172.21.15.37:0/2381584027 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:20.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8ab8aa8640 10 monclient(hunting): get_auth_request con 0x7f8ab4136eb0 auth_method 0 2024-04-26T00:43:20.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8ab8aa8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:20.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8ab8aa8640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:20.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8ab8aa8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:20.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8ab9aaa640 10 monclient(hunting): get_auth_request con 0x7f8ab41377d0 auth_method 0 2024-04-26T00:43:20.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8ab9aaa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:20.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8ab9aaa640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:20.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8ab9aaa640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:20.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8ab8aa8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:20.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8ab8aa8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:20.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8ab8aa8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:20.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8ab9aaa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:20.053 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8ab9aaa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:20.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8ab9aaa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:20.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8ab92a9640 1 --2- 172.21.15.37:0/2381584027 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ab41340d0 0x7f8ab4130eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:20.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8ab92a9640 10 monclient(hunting): get_auth_request con 0x7f8ab41340d0 auth_method 0 2024-04-26T00:43:20.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8ab92a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:20.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8ab92a9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:20.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8ab92a9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:20.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8ab8aa8640 10 monclient(hunting): handle_auth_done global_id 254468 payload 824 2024-04-26T00:43:20.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8ab8aa8640 1 -- 172.21.15.37:0/2381584027 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ab41340d0 msgr2=0x7f8ab4130eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:20.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8ab8aa8640 1 --2- 172.21.15.37:0/2381584027 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ab41340d0 0x7f8ab4130eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:20.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8ab8aa8640 1 -- 172.21.15.37:0/2381584027 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8ab41377d0 msgr2=0x7f8ab4137fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:20.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8ab8aa8640 1 --2- 172.21.15.37:0/2381584027 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8ab41377d0 0x7f8ab4137fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:20.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8ab8aa8640 10 monclient: _finish_hunting 0 2024-04-26T00:43:20.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8ab8aa8640 1 monclient: found mon.noname-c 2024-04-26T00:43:20.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8ab8aa8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:20.054 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8ab8aa8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:43:20.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.043+0000 7f8ab8aa8640 1 -- 172.21.15.37:0/2381584027 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ab409ae10 con 0x7f8ab4136eb0 2024-04-26T00:43:20.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.047+0000 7f8ab8aa8640 1 -- 172.21.15.37:0/2381584027 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8ab4136eb0 msgr2=0x7f8ab41313f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:20.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.047+0000 7f8ab8aa8640 1 --2- 172.21.15.37:0/2381584027 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8ab4136eb0 0x7f8ab41313f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8aa400ebe0 tx=0x7f8aa400c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:20.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.047+0000 7f8aa9ffb640 1 -- 172.21.15.37:0/2381584027 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa4017070 con 0x7f8ab4136eb0 2024-04-26T00:43:20.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.047+0000 7f8aa9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:20.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.047+0000 7f8aa9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:43:20.055 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.047+0000 7f8aa9ffb640 10 monclient: dump: 2024-04-26T00:43:20.055 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:20.055 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:20.055 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:20.055 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:20.055 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:20.055 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:20.055 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:20.056 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:20.056 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:20.056 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:20.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.047+0000 7f8aa9ffb640 10 monclient: _finish_auth 0 2024-04-26T00:43:20.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.047+0000 7f8aa9ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:43:20.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.047+0000 7f8aa9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:20.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.047+0000 7f8aa9ffb640 1 -- 172.21.15.37:0/2381584027 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8aa400ce40 con 0x7f8ab4136eb0 2024-04-26T00:43:20.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.047+0000 7f8aa9ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:20.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.047+0000 7f8aa9ffb640 1 -- 172.21.15.37:0/2381584027 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa4013690 con 0x7f8ab4136eb0 2024-04-26T00:43:20.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.047+0000 7f8aa9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:20.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.047+0000 7f8aa9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:43:20.056 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.047+0000 7f8aa9ffb640 10 monclient: dump: 2024-04-26T00:43:20.056 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:20.056 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:20.056 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:20.057 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:20.057 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:20.057 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:20.057 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:20.057 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:20.057 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:20.057 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:20.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.047+0000 7f8abb50d640 5 monclient: authenticate success, global_id 254468 2024-04-26T00:43:20.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.047+0000 7f8abb50d640 10 monclient: _renew_subs 2024-04-26T00:43:20.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.047+0000 7f8aab7fe640 4 set_mon_vals no callback set 2024-04-26T00:43:20.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.047+0000 7f8abb50d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:20.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.047+0000 7f8abb50d640 1 -- 172.21.15.37:0/2381584027 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ab40a8840 con 0x7f8ab4136eb0 2024-04-26T00:43:20.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.047+0000 7f8abb50d640 10 monclient: _renew_subs 2024-04-26T00:43:20.057 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.047+0000 7f8abb50d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:20.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.047+0000 7f8abb50d640 1 -- 172.21.15.37:0/2381584027 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ab413a990 con 0x7f8ab4136eb0 2024-04-26T00:43:20.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.047+0000 7f8aa9ffb640 1 -- 172.21.15.37:0/2381584027 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f8aa401a070 con 0x7f8ab4136eb0 2024-04-26T00:43:20.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.047+0000 7f8abb50d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:20.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.047+0000 7f8abb50d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:20.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.047+0000 7f8abb50d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:20.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.047+0000 7f8abb50d640 1 -- 172.21.15.37:0/2381584027 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ab4066f40 con 0x7f8ab4136eb0 2024-04-26T00:43:20.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.047+0000 7f8aa9ffb640 1 --2- 172.21.15.37:0/2381584027 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8a98035ed0 0x7f8a98038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:20.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.047+0000 7f8aa9ffb640 1 -- 172.21.15.37:0/2381584027 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(899..899 src has 822..899) ==== 3550+0+0 (secure 0 0 0) 0x7f8aa40488e0 con 0x7f8ab4136eb0 2024-04-26T00:43:20.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.047+0000 7f8ab92a9640 1 --2- 172.21.15.37:0/2381584027 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8a98035ed0 0x7f8a98038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:20.058 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.047+0000 7f8ab92a9640 10 monclient: get_auth_request con 0x7f8a98035ed0 auth_method 0 2024-04-26T00:43:20.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.051+0000 7f8ab92a9640 1 --2- 172.21.15.37:0/2381584027 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8a98035ed0 0x7f8a98038390 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f8ab4139c00 tx=0x7f8ab0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:20.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.051+0000 7f8aa9ffb640 1 -- 172.21.15.37:0/2381584027 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f8aa40477b0 con 0x7f8ab4136eb0 2024-04-26T00:43:20.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.051+0000 7f8aa9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:20.059 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.051+0000 7f8aa9ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:43:20.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.211+0000 7f8abb50d640 1 -- 172.21.15.37:0/2381584027 --> v2:172.21.15.45:6812/2379918623 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8ab4131930 con 0x7f8a98035ed0 2024-04-26T00:43:20.216 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.215+0000 7f8aa9ffb640 1 -- 172.21.15.37:0/2381584027 <== mgr.4107 v2:172.21.15.45:6812/2379918623 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33844 (secure 0 0 0) 0x7f8ab4131930 con 0x7f8a98035ed0 2024-04-26T00:43:20.217 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:43:20.217 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-04-26T00:43:20.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.215+0000 7f8abb50d640 1 -- 172.21.15.37:0/2381584027 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8a98035ed0 msgr2=0x7f8a98038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:20.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.215+0000 7f8abb50d640 1 --2- 172.21.15.37:0/2381584027 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8a98035ed0 0x7f8a98038390 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f8ab4139c00 tx=0x7f8ab0008040 comp rx=0 tx=0).stop 2024-04-26T00:43:20.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.215+0000 7f8abb50d640 10 monclient: shutdown 2024-04-26T00:43:20.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.215+0000 7f8abb50d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:20.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.215+0000 7f8abb50d640 1 -- 172.21.15.37:0/2381584027 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8ab4136eb0 msgr2=0x7f8ab41313f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:20.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.215+0000 7f8abb50d640 1 --2- 172.21.15.37:0/2381584027 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8ab4136eb0 0x7f8ab41313f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8aa400ebe0 tx=0x7f8aa400c520 comp rx=0 tx=0).stop 2024-04-26T00:43:20.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.219+0000 7f8abb50d640 1 -- 172.21.15.37:0/2381584027 shutdown_connections 2024-04-26T00:43:20.219 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.219+0000 7f8abb50d640 1 --2- 172.21.15.37:0/2381584027 >> v2:172.21.15.45:6812/2379918623 conn(0x7f8a98035ed0 0x7f8a98038390 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:20.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.219+0000 7f8abb50d640 1 --2- 172.21.15.37:0/2381584027 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8ab41377d0 0x7f8ab4137fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:20.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.219+0000 7f8abb50d640 1 --2- 172.21.15.37:0/2381584027 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8ab4136eb0 0x7f8ab41313f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:20.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.219+0000 7f8abb50d640 1 --2- 172.21.15.37:0/2381584027 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ab41340d0 0x7f8ab4130eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:20.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.219+0000 7f8abb50d640 1 -- 172.21.15.37:0/2381584027 >> 172.21.15.37:0/2381584027 conn(0x7f8ab40a6300 msgr2=0x7f8ab4135e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:20.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.219+0000 7f8abb50d640 1 -- 172.21.15.37:0/2381584027 shutdown_connections 2024-04-26T00:43:20.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.219+0000 7f8abb50d640 1 -- 172.21.15.37:0/2381584027 wait complete. 2024-04-26T00:43:20.235 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":1583,"stamp":"2024-04-26T00:43:19.552564+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":346348,"kb_used_data":3936,"kb_used_omap":9,"kb_used_meta":342326,"kb_avail":565884692,"statfs":{"total":579820584960,"available":579465924608,"internally_reserved":0,"allocated":4030464,"data_stored":3181916,"data_compressed":61234,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":350542496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003799"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":1755,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:42:43.917048+0000","last_change":"2024-04-26T00:20:02.279764+0000","last_active":"2024-04-26T00:42:43.917048+0000","last_peered":"2024-04-26T00:42:43.917048+0000","last_clean":"2024-04-26T00:42:43.917048+0000","last_became_active":"2024-04-26T00:20:02.278959+0000","last_became_peered":"2024-04-26T00:20:02.278959+0000","last_unstale":"2024-04-26T00:42:43.917048+0000","last_undegraded":"2024-04-26T00:42:43.917048+0000","last_fullsized":"2024-04-26T00:42:43.917048+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:23:43.610174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":1750,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:42:43.753251+0000","last_change":"2024-04-26T00:20:02.279805+0000","last_active":"2024-04-26T00:42:43.753251+0000","last_peered":"2024-04-26T00:42:43.753251+0000","last_clean":"2024-04-26T00:42:43.753251+0000","last_became_active":"2024-04-26T00:20:02.279053+0000","last_became_peered":"2024-04-26T00:20:02.279053+0000","last_unstale":"2024-04-26T00:42:43.753251+0000","last_undegraded":"2024-04-26T00:42:43.753251+0000","last_fullsized":"2024-04-26T00:42:43.753251+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:11:26.730719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":1761,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:42:43.918127+0000","last_change":"2024-04-26T00:20:02.277311+0000","last_active":"2024-04-26T00:42:43.918127+0000","last_peered":"2024-04-26T00:42:43.918127+0000","last_clean":"2024-04-26T00:42:43.918127+0000","last_became_active":"2024-04-26T00:20:02.276409+0000","last_became_peered":"2024-04-26T00:20:02.276409+0000","last_unstale":"2024-04-26T00:42:43.918127+0000","last_undegraded":"2024-04-26T00:42:43.918127+0000","last_fullsized":"2024-04-26T00:42:43.918127+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:14:50.584720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":1763,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:42:43.918487+0000","last_change":"2024-04-26T00:20:02.273965+0000","last_active":"2024-04-26T00:42:43.918487+0000","last_peered":"2024-04-26T00:42:43.918487+0000","last_clean":"2024-04-26T00:42:43.918487+0000","last_became_active":"2024-04-26T00:20:02.273213+0000","last_became_peered":"2024-04-26T00:20:02.273213+0000","last_unstale":"2024-04-26T00:42:43.918487+0000","last_undegraded":"2024-04-26T00:42:43.918487+0000","last_fullsized":"2024-04-26T00:42:43.918487+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:42:14.055403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"15'117","reported_seq":1905,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:42:43.916767+0000","last_change":"2024-04-26T00:20:02.281782+0000","last_active":"2024-04-26T00:42:43.916767+0000","last_peered":"2024-04-26T00:42:43.916767+0000","last_clean":"2024-04-26T00:42:43.916767+0000","last_became_active":"2024-04-26T00:20:02.281000+0000","last_became_peered":"2024-04-26T00:20:02.281000+0000","last_unstale":"2024-04-26T00:42:43.916767+0000","last_undegraded":"2024-04-26T00:42:43.916767+0000","last_fullsized":"2024-04-26T00:42:43.916767+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:40:30.790500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":1763,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:42:43.920813+0000","last_change":"2024-04-26T00:20:02.274249+0000","last_active":"2024-04-26T00:42:43.920813+0000","last_peered":"2024-04-26T00:42:43.920813+0000","last_clean":"2024-04-26T00:42:43.920813+0000","last_became_active":"2024-04-26T00:20:02.273745+0000","last_became_peered":"2024-04-26T00:20:02.273745+0000","last_unstale":"2024-04-26T00:42:43.920813+0000","last_undegraded":"2024-04-26T00:42:43.920813+0000","last_fullsized":"2024-04-26T00:42:43.920813+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:36:25.948539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":1761,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:42:43.918336+0000","last_change":"2024-04-26T00:20:02.474779+0000","last_active":"2024-04-26T00:42:43.918336+0000","last_peered":"2024-04-26T00:42:43.918336+0000","last_clean":"2024-04-26T00:42:43.918336+0000","last_became_active":"2024-04-26T00:20:02.473888+0000","last_became_peered":"2024-04-26T00:20:02.473888+0000","last_unstale":"2024-04-26T00:42:43.918336+0000","last_undegraded":"2024-04-26T00:42:43.918336+0000","last_fullsized":"2024-04-26T00:42:43.918336+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:33:18.962907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":1754,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:42:43.918328+0000","last_change":"2024-04-26T00:20:02.279987+0000","last_active":"2024-04-26T00:42:43.918328+0000","last_peered":"2024-04-26T00:42:43.918328+0000","last_clean":"2024-04-26T00:42:43.918328+0000","last_became_active":"2024-04-26T00:20:02.279183+0000","last_became_peered":"2024-04-26T00:20:02.279183+0000","last_unstale":"2024-04-26T00:42:43.918328+0000","last_undegraded":"2024-04-26T00:42:43.918328+0000","last_fullsized":"2024-04-26T00:42:43.918328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:45:44.508952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":1749,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:42:43.754128+0000","last_change":"2024-04-26T00:20:02.475146+0000","last_active":"2024-04-26T00:42:43.754128+0000","last_peered":"2024-04-26T00:42:43.754128+0000","last_clean":"2024-04-26T00:42:43.754128+0000","last_became_active":"2024-04-26T00:20:02.473928+0000","last_became_peered":"2024-04-26T00:20:02.473928+0000","last_unstale":"2024-04-26T00:42:43.754128+0000","last_undegraded":"2024-04-26T00:42:43.754128+0000","last_fullsized":"2024-04-26T00:42:43.754128+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:25:12.904295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":61234,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640538,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58184,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313656,"statfs":{"total":96636764160,"available":96577183744,"internally_reserved":0,"allocated":651264,"data_stored":354010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640537,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49864,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321976,"statfs":{"total":96636764160,"available":96585703424,"internally_reserved":0,"allocated":651264,"data_stored":354010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640539,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40200,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331640,"statfs":{"total":96636764160,"available":96595599360,"internally_reserved":0,"allocated":712704,"data_stored":882938,"data_compressed":30617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640539,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72008,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299832,"statfs":{"total":96636764160,"available":96563027968,"internally_reserved":0,"allocated":651264,"data_stored":354010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640539,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72068,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299772,"statfs":{"total":96636764160,"available":96562966528,"internally_reserved":0,"allocated":712704,"data_stored":882938,"data_compressed":30617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640538,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54024,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317816,"statfs":{"total":96636764160,"available":96581443584,"internally_reserved":0,"allocated":651264,"data_stored":354010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T00:43:20.236 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T00:43:20.253 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.251+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:20.253 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.251+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=647471778}) 2024-04-26T00:43:20.253 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.251+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3227203155}) 2024-04-26T00:43:20.253 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.251+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:20.253 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.251+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=647471778}) 2024-04-26T00:43:20.254 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.251+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4250148869}) 2024-04-26T00:43:20.255 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.255+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:20.255 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.255+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3744560226}) 2024-04-26T00:43:20.255 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.255+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3811048159}) 2024-04-26T00:43:20.255 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.255+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:20.255 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.255+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3744560226}) 2024-04-26T00:43:20.255 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.255+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=792770529}) 2024-04-26T00:43:20.256 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.255+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:20.256 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.255+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1541834166}) 2024-04-26T00:43:20.256 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.255+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4119871207}) 2024-04-26T00:43:20.257 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.255+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:20.257 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.255+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1541834166}) 2024-04-26T00:43:20.257 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.255+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=989722972}) 2024-04-26T00:43:20.257 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.255+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:20.257 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.255+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=243784429}) 2024-04-26T00:43:20.258 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.255+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1179340372}) 2024-04-26T00:43:20.258 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.255+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:20.258 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.255+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=243784429}) 2024-04-26T00:43:20.258 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.255+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2819537346}) 2024-04-26T00:43:20.259 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.259+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:20.259 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.259+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3285873364}) 2024-04-26T00:43:20.259 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.259+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1175335531}) 2024-04-26T00:43:20.259 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.259+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:20.259 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.259+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3285873364}) 2024-04-26T00:43:20.259 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.259+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3796194651}) 2024-04-26T00:43:20.260 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.259+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:20.260 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.259+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2158269416}) 2024-04-26T00:43:20.260 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.259+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1934906584}) 2024-04-26T00:43:20.261 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.259+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:20.261 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.259+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2158269416}) 2024-04-26T00:43:20.261 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.259+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=58490775}) 2024-04-26T00:43:20.262 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.259+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:20.262 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.259+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3836186870}) 2024-04-26T00:43:20.262 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.259+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1592757268}) 2024-04-26T00:43:20.262 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.259+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:20.262 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.259+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3836186870}) 2024-04-26T00:43:20.262 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.259+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=539079982}) 2024-04-26T00:43:20.264 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.263+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:20.264 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.263+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1059562662}) 2024-04-26T00:43:20.264 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.263+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=98,mdsmap=2} crc {logm=2099928782,mdsmap=1401145748}) 2024-04-26T00:43:20.264 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.263+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:20.264 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.263+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1059562662}) 2024-04-26T00:43:20.264 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.263+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3711982220}) 2024-04-26T00:43:20.267 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.267+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:20.267 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.267+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=3,mdsmap=2,mgr=7,mgr_command_descs=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=54} crc {logm=1041355198,mdsmap=1401145748,mgr=3887433460,mgr_command_descs=3057339438,mgrstat=2093372921,mon_config_key=4061063117,monmap=3876866195,osd_metadata=3306121363,osd_pg_creating=920254315,osd_snap=1896859014}) 2024-04-26T00:43:20.267 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.267+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=2,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=53} crc {logm=2831636251,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=3065954992,mon_config_key=4061063117,monmap=3876866194,osd_metadata=1540339580,osd_pg_creating=920254315,osd_snap=466922687}) 2024-04-26T00:43:20.267 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.267+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:20.267 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.267+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=3,mdsmap=2,mgr=7,mgr_command_descs=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=54} crc {logm=1041355198,mdsmap=1401145748,mgr=3887433460,mgr_command_descs=3057339438,mgrstat=2093372921,mon_config_key=4061063117,monmap=3876866195,osd_metadata=3306121363,osd_pg_creating=920254315,osd_snap=1896859014}) 2024-04-26T00:43:20.267 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.267+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=2,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=11,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=52} crc {logm=2831636251,mdsmap=2840952586,mgr=1189067752,mgr_command_descs=3057339438,mgr_metadata=3686404438,mgrstat=2554643428,mon_config_key=4061063117,monmap=4194790699,osd_metadata=3306121363,osd_pg_creating=920254315,osd_snap=3702893290}) 2024-04-26T00:43:20.269 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.267+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:20.269 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.267+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=100} crc {osd_snap=3048281629}) 2024-04-26T00:43:20.269 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.267+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=100} crc {osd_snap=3852678379}) 2024-04-26T00:43:20.269 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.267+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:20.269 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.267+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=100} crc {osd_snap=3048281629}) 2024-04-26T00:43:20.269 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.267+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=100} crc {osd_snap=3007041023}) 2024-04-26T00:43:20.272 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.271+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:20.272 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.271+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=15,osdmap=85} crc {osd_snap=4286289313,osdmap=113632959}) 2024-04-26T00:43:20.272 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.271+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=14,osdmap=86} crc {osd_snap=1652268848,osdmap=3706200051}) 2024-04-26T00:43:20.272 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.271+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:20.272 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.271+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=15,osdmap=85} crc {osd_snap=4286289313,osdmap=113632959}) 2024-04-26T00:43:20.272 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.271+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=13,osdmap=87} crc {osd_snap=3827023670,osdmap=110170086}) 2024-04-26T00:43:20.287 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.287+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:20.287 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.287+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=43} crc {osdmap=364558783}) 2024-04-26T00:43:20.287 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.287+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=41} crc {osdmap=3049357359}) 2024-04-26T00:43:20.287 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.287+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T00:43:20.287 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.287+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=43} crc {osdmap=364558783}) 2024-04-26T00:43:20.287 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-04-26T00:43:20.287+0000 7fee08ceb640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=39} crc {osdmap=3631468875}) 2024-04-26T00:43:20.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.359+0000 7faa3175d640 10 monclient: get_monmap_and_config 2024-04-26T00:43:20.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.359+0000 7faa3175d640 10 monclient: build_initial_monmap 2024-04-26T00:43:20.362 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.359+0000 7faa3175d640 10 monclient: monmap: 2024-04-26T00:43:20.362 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:20.363 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:20.363 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:43:20.363300+0000 2024-04-26T00:43:20.363 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:43:20.363300+0000 2024-04-26T00:43:20.363 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:20.363 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:20.363 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:43:20.363 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:43:20.363 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:43:20.363 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:43:20.363 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:43:20.363 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:20.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.363+0000 7faa3175d640 1 Processor -- start 2024-04-26T00:43:20.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.363+0000 7faa3175d640 1 -- start start 2024-04-26T00:43:20.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.363+0000 7faa3175d640 10 monclient: init 2024-04-26T00:43:20.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.363+0000 7faa3175d640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:20.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.363+0000 7faa3175d640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-04-26T00:43:20.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.363+0000 7faa3175d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7faa2c133b20 0x7faa2c09e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:20.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.363+0000 7faa3175d640 10 monclient(hunting): picked mon.noname-a con 0x7faa2c133b20 addr v2:172.21.15.37:3300/0 2024-04-26T00:43:20.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.363+0000 7faa3175d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faa2c133f00 addr v1:172.21.15.37:6789/0 2024-04-26T00:43:20.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.363+0000 7faa3175d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faa2c139050 addr v1:172.21.15.45:6789/0 2024-04-26T00:43:20.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.363+0000 7faa3175d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa2c09fe20 con 0x7faa2c133f00 2024-04-26T00:43:20.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.363+0000 7faa3175d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa2c072d00 con 0x7faa2c139050 2024-04-26T00:43:20.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.363+0000 7faa3175d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:20.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.363+0000 7faa3175d640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7faa2c09ba60 con 0x7faa2c133b20 2024-04-26T00:43:20.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.363+0000 7faa3175d640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:20.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.363+0000 7faa3175d640 10 monclient(hunting): authenticate will time out at 2381.669110s 2024-04-26T00:43:20.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.363+0000 7faa2affd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7faa2c133b20 0x7faa2c09e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:20.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.363+0000 7faa2affd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7faa2c133b20 0x7faa2c09e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58106/0 (socket says 172.21.15.37:58106) 2024-04-26T00:43:20.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.363+0000 7faa2affd640 1 -- 172.21.15.37:0/4215339073 learned_addr learned my addr 172.21.15.37:0/4215339073 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:20.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.363+0000 7faa2affd640 10 monclient(hunting): get_auth_request con 0x7faa2c133b20 auth_method 0 2024-04-26T00:43:20.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.363+0000 7faa2affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:20.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.363+0000 7faa2affd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:20.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.363+0000 7faa2affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:20.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.363+0000 7faa2affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:20.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.363+0000 7faa2affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:20.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.363+0000 7faa2affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:20.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.363+0000 7faa2affd640 10 monclient(hunting): handle_auth_done global_id 284136 payload 274 2024-04-26T00:43:20.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.363+0000 7faa2affd640 1 -- 172.21.15.37:0/4215339073 >> v1:172.21.15.45:6789/0 conn(0x7faa2c139050 legacy=0x7faa2c13b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:20.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.363+0000 7faa2affd640 1 -- 172.21.15.37:0/4215339073 >> v1:172.21.15.37:6789/0 conn(0x7faa2c133f00 legacy=0x7faa2c134350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:20.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.363+0000 7faa2affd640 10 monclient: _finish_hunting 0 2024-04-26T00:43:20.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.363+0000 7faa2affd640 1 monclient: found mon.noname-a 2024-04-26T00:43:20.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.363+0000 7faa2affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:20.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.363+0000 7faa2affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:43:20.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.363+0000 7faa2affd640 1 -- 172.21.15.37:0/4215339073 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa2c09dd10 con 0x7faa2c133b20 2024-04-26T00:43:20.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.367+0000 7faa2affd640 1 -- 172.21.15.37:0/4215339073 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faa2c133b20 msgr2=0x7faa2c09e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:20.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.367+0000 7faa2affd640 1 --2- 172.21.15.37:0/4215339073 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faa2c133b20 0x7faa2c09e9d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7faa14009870 tx=0x7faa1401c190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fb4da7116105005 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:20.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.367+0000 7faa29ffb640 1 -- 172.21.15.37:0/4215339073 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa140b0070 con 0x7faa2c133b20 2024-04-26T00:43:20.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.367+0000 7faa29ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:20.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.367+0000 7faa29ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:43:20.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.367+0000 7faa29ffb640 10 monclient: dump: 2024-04-26T00:43:20.371 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:20.371 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:20.371 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:20.371 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:20.371 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:20.371 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:20.371 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:20.371 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:20.371 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:20.371 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:20.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.367+0000 7faa29ffb640 10 monclient: _finish_auth 0 2024-04-26T00:43:20.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.367+0000 7faa29ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:43:20.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.367+0000 7faa29ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:20.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.367+0000 7faa29ffb640 1 -- 172.21.15.37:0/4215339073 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa140ab600 con 0x7faa2c133b20 2024-04-26T00:43:20.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.367+0000 7faa3175d640 5 monclient: authenticate success, global_id 284136 2024-04-26T00:43:20.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.367+0000 7faa3175d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:43:20.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.367+0000 7faa29ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:20.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.367+0000 7faa3175d640 10 monclient: get_monmap_and_config success 2024-04-26T00:43:20.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.367+0000 7faa3175d640 4 set_mon_vals no callback set 2024-04-26T00:43:20.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.367+0000 7faa29ffb640 1 -- 172.21.15.37:0/4215339073 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa140ab9d0 con 0x7faa2c133b20 2024-04-26T00:43:20.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.367+0000 7faa29ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:20.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.367+0000 7faa29ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:43:20.372 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.367+0000 7faa29ffb640 10 monclient: dump: 2024-04-26T00:43:20.372 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:20.372 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:20.372 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:20.373 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:20.373 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:20.373 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:20.373 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:20.373 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:20.373 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:20.373 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:20.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.367+0000 7faa3175d640 10 monclient: shutdown 2024-04-26T00:43:20.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.367+0000 7faa3175d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:20.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.367+0000 7faa3175d640 1 -- 172.21.15.37:0/4215339073 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faa2c133b20 msgr2=0x7faa2c09e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:20.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.367+0000 7faa3175d640 1 --2- 172.21.15.37:0/4215339073 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faa2c133b20 0x7faa2c09e9d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7faa14009870 tx=0x7faa1401c190 comp rx=0 tx=0).stop 2024-04-26T00:43:20.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.367+0000 7faa3175d640 1 -- 172.21.15.37:0/4215339073 shutdown_connections 2024-04-26T00:43:20.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.367+0000 7faa3175d640 1 --2- 172.21.15.37:0/4215339073 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faa2c133b20 0x7faa2c09e9d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:20.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.367+0000 7faa3175d640 1 -- 172.21.15.37:0/4215339073 >> 172.21.15.37:0/4215339073 conn(0x7faa2c0a1ad0 msgr2=0x7faa2c0a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:20.373 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa3175d640 1 -- 172.21.15.37:0/4215339073 shutdown_connections 2024-04-26T00:43:20.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa3175d640 1 -- 172.21.15.37:0/4215339073 wait complete. 2024-04-26T00:43:20.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa3175d640 10 monclient: build_initial_monmap 2024-04-26T00:43:20.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa3175d640 10 monclient: monmap: 2024-04-26T00:43:20.378 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:20.378 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:43:20.378 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:43:20.378 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:43:20.378 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:20.378 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:43:20.378 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:43:20.378 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:43:20.378 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:43:20.378 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:20.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa3175d640 1 Processor -- start 2024-04-26T00:43:20.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa3175d640 1 -- start start 2024-04-26T00:43:20.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa3175d640 10 monclient: init 2024-04-26T00:43:20.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa3175d640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:20.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa3175d640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:43:20.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa3175d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faa2c133b20 0x7faa2c1cf120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:20.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa3175d640 10 monclient(hunting): picked mon.noname-a con 0x7faa2c133b20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:43:20.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa3175d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faa2c133f00 0x7faa2c132da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:20.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa3175d640 10 monclient(hunting): picked mon.noname-c con 0x7faa2c133f00 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:20.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa3175d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa2c139050 0x7faa2c1332e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:20.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa3175d640 10 monclient(hunting): picked mon.noname-b con 0x7faa2c139050 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:43:20.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa3175d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:20.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa3175d640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7faa2c13f910 con 0x7faa2c133b20 2024-04-26T00:43:20.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa3175d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:20.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa3175d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7faa2c13f790 con 0x7faa2c139050 2024-04-26T00:43:20.379 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa3175d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:20.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa3175d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7faa2c13fa90 con 0x7faa2c133f00 2024-04-26T00:43:20.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa3175d640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:20.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa3175d640 10 monclient(hunting): authenticate will time out at 2381.677486s 2024-04-26T00:43:20.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa2a7fc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faa2c133f00 0x7faa2c132da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:20.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa2affd640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faa2c133b20 0x7faa2c1cf120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:20.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa2a7fc640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faa2c133f00 0x7faa2c132da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51298/0 (socket says 172.21.15.37:51298) 2024-04-26T00:43:20.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa2a7fc640 1 -- 172.21.15.37:0/1636551918 learned_addr learned my addr 172.21.15.37:0/1636551918 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:20.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa2a7fc640 10 monclient(hunting): get_auth_request con 0x7faa2c133f00 auth_method 0 2024-04-26T00:43:20.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa2a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:20.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa2a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:20.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:20.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa2affd640 10 monclient(hunting): get_auth_request con 0x7faa2c133b20 auth_method 0 2024-04-26T00:43:20.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa2affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:20.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa2affd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:20.380 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa2affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:20.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa2a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:20.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa2a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:20.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa2a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:20.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa2affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:20.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa2affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:20.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa2affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:20.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa2b7fe640 1 --2- 172.21.15.37:0/1636551918 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa2c139050 0x7faa2c1332e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:20.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa2a7fc640 10 monclient(hunting): handle_auth_done global_id 254471 payload 824 2024-04-26T00:43:20.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa2a7fc640 1 -- 172.21.15.37:0/1636551918 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa2c139050 msgr2=0x7faa2c1332e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:20.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa2a7fc640 1 --2- 172.21.15.37:0/1636551918 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa2c139050 0x7faa2c1332e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:20.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa2a7fc640 1 -- 172.21.15.37:0/1636551918 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faa2c133b20 msgr2=0x7faa2c1cf120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:20.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa2a7fc640 1 --2- 172.21.15.37:0/1636551918 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faa2c133b20 0x7faa2c1cf120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:20.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa2a7fc640 10 monclient: _finish_hunting 0 2024-04-26T00:43:20.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa2a7fc640 1 monclient: found mon.noname-c 2024-04-26T00:43:20.381 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa2a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:20.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa2a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:43:20.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa2a7fc640 1 -- 172.21.15.37:0/1636551918 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa2c139ee0 con 0x7faa2c133f00 2024-04-26T00:43:20.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa2b7fe640 10 monclient: get_auth_request con 0x7faa2c139050 auth_method 0 2024-04-26T00:43:20.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa2b7fe640 1 --2- 172.21.15.37:0/1636551918 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa2c139050 0x7faa2c1332e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:43:20.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.371+0000 7faa2affd640 1 --2- 172.21.15.37:0/1636551918 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faa2c133b20 0x7faa2c1cf120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:43:20.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.375+0000 7faa2a7fc640 1 -- 172.21.15.37:0/1636551918 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faa2c133f00 msgr2=0x7faa2c132da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:20.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.375+0000 7faa2a7fc640 1 --2- 172.21.15.37:0/1636551918 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faa2c133f00 0x7faa2c132da0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7faa1c00c7c0 tx=0x7faa1c00cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:20.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.375+0000 7faa237fe640 1 -- 172.21.15.37:0/1636551918 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c016020 con 0x7faa2c133f00 2024-04-26T00:43:20.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.375+0000 7faa237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:20.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.375+0000 7faa237fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:43:20.382 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.375+0000 7faa237fe640 10 monclient: dump: 2024-04-26T00:43:20.382 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:20.382 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:20.382 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:20.383 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:20.383 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:20.383 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:20.383 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:20.383 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:20.383 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:20.383 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:20.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.375+0000 7faa237fe640 10 monclient: _finish_auth 0 2024-04-26T00:43:20.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.375+0000 7faa237fe640 10 monclient: _check_auth_tickets 2024-04-26T00:43:20.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.375+0000 7faa237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:20.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.375+0000 7faa237fe640 1 -- 172.21.15.37:0/1636551918 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa1c00ac60 con 0x7faa2c133f00 2024-04-26T00:43:20.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.375+0000 7faa237fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:20.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.375+0000 7faa237fe640 1 -- 172.21.15.37:0/1636551918 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c012660 con 0x7faa2c133f00 2024-04-26T00:43:20.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.375+0000 7faa3175d640 5 monclient: authenticate success, global_id 254471 2024-04-26T00:43:20.383 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.375+0000 7faa3175d640 10 monclient: _renew_subs 2024-04-26T00:43:20.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.375+0000 7faa3175d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:20.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.375+0000 7faa3175d640 1 -- 172.21.15.37:0/1636551918 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa2c13bb90 con 0x7faa2c133f00 2024-04-26T00:43:20.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.375+0000 7faa297fa640 4 set_mon_vals no callback set 2024-04-26T00:43:20.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.375+0000 7faa237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:20.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.375+0000 7faa237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:43:20.384 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.375+0000 7faa237fe640 10 monclient: dump: 2024-04-26T00:43:20.384 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:20.384 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:20.384 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:20.384 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:20.384 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:20.384 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:20.384 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:20.384 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:20.384 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:20.385 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:20.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.375+0000 7faa3175d640 10 monclient: _renew_subs 2024-04-26T00:43:20.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.375+0000 7faa3175d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:20.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.375+0000 7faa3175d640 1 -- 172.21.15.37:0/1636551918 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7faa2c139c40 con 0x7faa2c133f00 2024-04-26T00:43:20.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.375+0000 7faa237fe640 1 -- 172.21.15.37:0/1636551918 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7faa1c019020 con 0x7faa2c133f00 2024-04-26T00:43:20.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.375+0000 7faa3175d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:20.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.375+0000 7faa3175d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:20.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.375+0000 7faa3175d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:20.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.375+0000 7faa3175d640 1 -- 172.21.15.37:0/1636551918 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa2c066f40 con 0x7faa2c133f00 2024-04-26T00:43:20.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.375+0000 7faa237fe640 1 --2- 172.21.15.37:0/1636551918 >> v2:172.21.15.45:6812/2379918623 conn(0x7faa0c035ed0 0x7faa0c038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:20.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.375+0000 7faa237fe640 1 -- 172.21.15.37:0/1636551918 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(899..899 src has 822..899) ==== 3550+0+0 (secure 0 0 0) 0x7faa1c048930 con 0x7faa2c133f00 2024-04-26T00:43:20.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.375+0000 7faa2affd640 1 --2- 172.21.15.37:0/1636551918 >> v2:172.21.15.45:6812/2379918623 conn(0x7faa0c035ed0 0x7faa0c038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:20.385 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.375+0000 7faa2affd640 10 monclient: get_auth_request con 0x7faa0c035ed0 auth_method 0 2024-04-26T00:43:20.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.379+0000 7faa2affd640 1 --2- 172.21.15.37:0/1636551918 >> v2:172.21.15.45:6812/2379918623 conn(0x7faa0c035ed0 0x7faa0c038390 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7faa140ab000 tx=0x7faa140a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:20.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.379+0000 7faa237fe640 1 -- 172.21.15.37:0/1636551918 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7faa1c020030 con 0x7faa2c133f00 2024-04-26T00:43:20.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.379+0000 7faa237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:20.386 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.379+0000 7faa237fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:43:20.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.543+0000 7faa3175d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-26T00:43:20.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.543+0000 7faa3175d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-26T00:43:20.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.543+0000 7faa3175d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:20.544 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.543+0000 7faa3175d640 1 -- 172.21.15.37:0/1636551918 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faa2c13b390 con 0x7faa2c133f00 2024-04-26T00:43:20.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.543+0000 7faa237fe640 1 -- 172.21.15.37:0/1636551918 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v899) ==== 74+0+10834 (secure 0 0 0) 0x7faa1c00ae10 con 0x7faa2c133f00 2024-04-26T00:43:20.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.543+0000 7faa237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-26T00:43:20.545 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.543+0000 7faa237fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:43:20.545 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:43:20.545 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":899,"fsid":"3cf2f905-fe97-4d94-8eba-4a21c519e869","created":"2024-04-26T00:19:50.025527+0000","modified":"2024-04-26T00:42:43.742102+0000","last_up_change":"2024-04-26T00:19:59.222603+0000","last_in_change":"2024-04-26T00:19:52.138726+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-26T00:20:00.477727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-04-26T00:20:01.233750+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a5d4b9d3-5499-491f-a123-85d73de452af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":3204000569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":3204000569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6811","nonce":3204000569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":3204000569}]},"public_addr":"172.21.15.37:6806/3204000569","cluster_addr":"172.21.15.37:6808/3204000569","heartbeat_back_addr":"172.21.15.37:6811/3204000569","heartbeat_front_addr":"172.21.15.37:6810/3204000569","state":["exists","up"]},{"osd":1,"uuid":"cb4583fd-a0fa-4559-9f99-5a3e02b5818c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":1084527868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6805","nonce":1084527868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6809","nonce":1084527868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6807","nonce":1084527868}]},"public_addr":"172.21.15.37:6804/1084527868","cluster_addr":"172.21.15.37:6805/1084527868","heartbeat_back_addr":"172.21.15.37:6809/1084527868","heartbeat_front_addr":"172.21.15.37:6807/1084527868","state":["exists","up"]},{"osd":2,"uuid":"78fc9fe6-2163-4cb3-9944-3773bd488df8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":603175474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6801","nonce":603175474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6803","nonce":603175474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":603175474}]},"public_addr":"172.21.15.37:6800/603175474","cluster_addr":"172.21.15.37:6801/603175474","heartbeat_back_addr":"172.21.15.37:6803/603175474","heartbeat_front_addr":"172.21.15.37:6802/603175474","state":["exists","up"]},{"osd":3,"uuid":"338bb632-eca1-422b-b196-fd6cadb8307c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":2173737584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6801","nonce":2173737584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6803","nonce":2173737584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":2173737584}]},"public_addr":"172.21.15.45:6800/2173737584","cluster_addr":"172.21.15.45:6801/2173737584","heartbeat_back_addr":"172.21.15.45:6803/2173737584","heartbeat_front_addr":"172.21.15.45:6802/2173737584","state":["exists","up"]},{"osd":4,"uuid":"ce2e0db7-d486-4cb1-9913-6a4a011e5c8c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":152605921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6805","nonce":152605921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6807","nonce":152605921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":152605921}]},"public_addr":"172.21.15.45:6804/152605921","cluster_addr":"172.21.15.45:6805/152605921","heartbeat_back_addr":"172.21.15.45:6807/152605921","heartbeat_front_addr":"172.21.15.45:6806/152605921","state":["exists","up"]},{"osd":5,"uuid":"a61739cc-41d3-48b1-a471-4092b2edbf1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3915746722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6809","nonce":3915746722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6811","nonce":3915746722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3915746722}]},"public_addr":"172.21.15.45:6808/3915746722","cluster_addr":"172.21.15.45:6809/3915746722","heartbeat_back_addr":"172.21.15.45:6811/3915746722","heartbeat_front_addr":"172.21.15.45:6810/3915746722","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T00:19:57.580195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T00:19:57.630479+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T00:19:57.716834+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T00:19:57.665953+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T00:19:57.633161+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T00:19:57.663690+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":429,"length":1},{"begin":449,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T00:43:20.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.547+0000 7faa3175d640 1 -- 172.21.15.37:0/1636551918 >> v2:172.21.15.45:6812/2379918623 conn(0x7faa0c035ed0 msgr2=0x7faa0c038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:20.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.547+0000 7faa3175d640 1 --2- 172.21.15.37:0/1636551918 >> v2:172.21.15.45:6812/2379918623 conn(0x7faa0c035ed0 0x7faa0c038390 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7faa140ab000 tx=0x7faa140a6000 comp rx=0 tx=0).stop 2024-04-26T00:43:20.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.547+0000 7faa3175d640 10 monclient: shutdown 2024-04-26T00:43:20.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.547+0000 7faa3175d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:20.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.547+0000 7faa3175d640 1 -- 172.21.15.37:0/1636551918 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faa2c133f00 msgr2=0x7faa2c132da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:20.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.547+0000 7faa3175d640 1 --2- 172.21.15.37:0/1636551918 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faa2c133f00 0x7faa2c132da0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7faa1c00c7c0 tx=0x7faa1c00cc80 comp rx=0 tx=0).stop 2024-04-26T00:43:20.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.547+0000 7faa3175d640 1 -- 172.21.15.37:0/1636551918 shutdown_connections 2024-04-26T00:43:20.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.547+0000 7faa3175d640 1 --2- 172.21.15.37:0/1636551918 >> v2:172.21.15.45:6812/2379918623 conn(0x7faa0c035ed0 0x7faa0c038390 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:20.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.547+0000 7faa3175d640 1 --2- 172.21.15.37:0/1636551918 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faa2c139050 0x7faa2c1332e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:20.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.547+0000 7faa3175d640 1 --2- 172.21.15.37:0/1636551918 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faa2c133f00 0x7faa2c132da0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:20.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.547+0000 7faa3175d640 1 --2- 172.21.15.37:0/1636551918 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faa2c133b20 0x7faa2c1cf120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:20.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.547+0000 7faa3175d640 1 -- 172.21.15.37:0/1636551918 >> 172.21.15.37:0/1636551918 conn(0x7faa2c0a1ad0 msgr2=0x7faa2c0a0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:20.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.547+0000 7faa3175d640 1 -- 172.21.15.37:0/1636551918 shutdown_connections 2024-04-26T00:43:20.548 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:20.547+0000 7faa3175d640 1 -- 172.21.15.37:0/1636551918 wait complete. 2024-04-26T00:43:21.564 INFO:tasks.ceph:Scrubbing osd.0 2024-04-26T00:43:21.564 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-26T00:43:21.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff341ed0640 10 monclient: get_monmap_and_config 2024-04-26T00:43:21.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff341ed0640 10 monclient: build_initial_monmap 2024-04-26T00:43:21.703 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff341ed0640 10 monclient: monmap: 2024-04-26T00:43:21.703 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:21.703 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:21.703 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:43:21.704204+0000 2024-04-26T00:43:21.703 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:43:21.704204+0000 2024-04-26T00:43:21.703 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:21.704 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:21.704 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:43:21.704 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:43:21.704 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:43:21.704 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:43:21.704 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:43:21.704 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:21.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff341ed0640 1 Processor -- start 2024-04-26T00:43:21.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff341ed0640 1 -- start start 2024-04-26T00:43:21.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff341ed0640 10 monclient: init 2024-04-26T00:43:21.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff341ed0640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:21.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff341ed0640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-04-26T00:43:21.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff341ed0640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff33c131ec0 0x7ff33c1322a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:21.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff341ed0640 10 monclient(hunting): picked mon.noname-a con 0x7ff33c131ec0 addr v2:172.21.15.37:3300/0 2024-04-26T00:43:21.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff341ed0640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff33c134d80 0x7ff33c1351c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:21.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff341ed0640 10 monclient(hunting): picked mon.noname-c con 0x7ff33c134d80 addr v2:172.21.15.45:3300/0 2024-04-26T00:43:21.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff341ed0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff33c135700 addr v1:172.21.15.37:6789/0 2024-04-26T00:43:21.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff341ed0640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff33c0a3680 con 0x7ff33c135700 2024-04-26T00:43:21.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff341ed0640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:21.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff341ed0640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7ff33c071ef0 con 0x7ff33c131ec0 2024-04-26T00:43:21.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff341ed0640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:21.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff341ed0640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7ff33c0a38c0 con 0x7ff33c134d80 2024-04-26T00:43:21.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff341ed0640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:21.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff341ed0640 10 monclient(hunting): authenticate will time out at 2383.009930s 2024-04-26T00:43:21.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff33b7fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff33c131ec0 0x7ff33c1322a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:21.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff33bfff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7ff33c135700 0x7ff33c13c0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:49514/0 (socket says 172.21.15.37:49514) 2024-04-26T00:43:21.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff33bfff640 1 -- 172.21.15.37:0/209965684 learned_addr learned my addr 172.21.15.37:0/209965684 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:43:21.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff33b7fe640 10 monclient(hunting): get_auth_request con 0x7ff33c131ec0 auth_method 0 2024-04-26T00:43:21.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff33b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:21.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff33b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:21.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff33b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:21.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff33b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:21.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff33b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:21.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff33b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:21.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff33affd640 1 --2- 172.21.15.37:0/209965684 >> v2:172.21.15.45:3300/0 conn(0x7ff33c134d80 0x7ff33c1351c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:21.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff33b7fe640 10 monclient(hunting): handle_auth_done global_id 284142 payload 274 2024-04-26T00:43:21.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff33b7fe640 1 -- 172.21.15.37:0/209965684 >> v2:172.21.15.45:3300/0 conn(0x7ff33c134d80 msgr2=0x7ff33c1351c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:21.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff33b7fe640 1 --2- 172.21.15.37:0/209965684 >> v2:172.21.15.45:3300/0 conn(0x7ff33c134d80 0x7ff33c1351c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:21.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff33b7fe640 1 -- 172.21.15.37:0/209965684 >> v1:172.21.15.37:6789/0 conn(0x7ff33c135700 legacy=0x7ff33c13c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:21.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff33b7fe640 10 monclient: _finish_hunting 0 2024-04-26T00:43:21.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff33b7fe640 1 monclient: found mon.noname-a 2024-04-26T00:43:21.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff33b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:21.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff33b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:43:21.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff33b7fe640 1 -- 172.21.15.37:0/209965684 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff33c071650 con 0x7ff33c131ec0 2024-04-26T00:43:21.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff33affd640 10 monclient: get_auth_request con 0x7ff33c134d80 auth_method 0 2024-04-26T00:43:21.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.703+0000 7ff33affd640 1 --2- 172.21.15.37:0/209965684 >> v2:172.21.15.45:3300/0 conn(0x7ff33c134d80 0x7ff33c1351c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:43:21.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.707+0000 7ff33b7fe640 1 -- 172.21.15.37:0/209965684 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff33c131ec0 msgr2=0x7ff33c1322a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:21.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.707+0000 7ff33b7fe640 1 --2- 172.21.15.37:0/209965684 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff33c131ec0 0x7ff33c1322a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff33c0a84a0 tx=0x7ff32401c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11172dbd4d1b8476 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:21.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.707+0000 7ff33a7fc640 1 -- 172.21.15.37:0/209965684 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3240a67f0 con 0x7ff33c131ec0 2024-04-26T00:43:21.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.707+0000 7ff33a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:21.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.707+0000 7ff33a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:43:21.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.707+0000 7ff33a7fc640 10 monclient: dump: 2024-04-26T00:43:21.710 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:21.710 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:21.710 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:21.711 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:21.711 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:21.711 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:21.711 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:21.711 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:21.711 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:21.711 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:21.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.707+0000 7ff33a7fc640 10 monclient: _finish_auth 0 2024-04-26T00:43:21.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.707+0000 7ff33a7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:43:21.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.707+0000 7ff33a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:21.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.707+0000 7ff33a7fc640 1 -- 172.21.15.37:0/209965684 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3240a6990 con 0x7ff33c131ec0 2024-04-26T00:43:21.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.707+0000 7ff341ed0640 5 monclient: authenticate success, global_id 284142 2024-04-26T00:43:21.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.707+0000 7ff341ed0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:43:21.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.707+0000 7ff33a7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:21.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.707+0000 7ff33a7fc640 1 -- 172.21.15.37:0/209965684 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3240a6d60 con 0x7ff33c131ec0 2024-04-26T00:43:21.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.707+0000 7ff341ed0640 10 monclient: get_monmap_and_config success 2024-04-26T00:43:21.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.707+0000 7ff341ed0640 4 set_mon_vals no callback set 2024-04-26T00:43:21.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.707+0000 7ff33a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:21.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.707+0000 7ff33a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:43:21.712 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.707+0000 7ff33a7fc640 10 monclient: dump: 2024-04-26T00:43:21.712 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:21.712 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:21.712 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:21.712 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:21.712 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:21.712 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:21.712 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:21.712 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:21.712 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:21.713 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:21.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.707+0000 7ff341ed0640 10 monclient: shutdown 2024-04-26T00:43:21.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.707+0000 7ff341ed0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:21.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.707+0000 7ff341ed0640 1 -- 172.21.15.37:0/209965684 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff33c131ec0 msgr2=0x7ff33c1322a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:21.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.707+0000 7ff341ed0640 1 --2- 172.21.15.37:0/209965684 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff33c131ec0 0x7ff33c1322a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff33c0a84a0 tx=0x7ff32401c840 comp rx=0 tx=0).stop 2024-04-26T00:43:21.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.707+0000 7ff341ed0640 1 -- 172.21.15.37:0/209965684 shutdown_connections 2024-04-26T00:43:21.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.707+0000 7ff341ed0640 1 --2- 172.21.15.37:0/209965684 >> v2:172.21.15.45:3300/0 conn(0x7ff33c134d80 0x7ff33c1351c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:21.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.707+0000 7ff341ed0640 1 --2- 172.21.15.37:0/209965684 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff33c131ec0 0x7ff33c1322a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:21.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.707+0000 7ff341ed0640 1 -- 172.21.15.37:0/209965684 >> 172.21.15.37:0/209965684 conn(0x7ff33c09abb0 msgr2=0x7ff33c0a73e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:21.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.707+0000 7ff341ed0640 1 -- 172.21.15.37:0/209965684 shutdown_connections 2024-04-26T00:43:21.713 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.707+0000 7ff341ed0640 1 -- 172.21.15.37:0/209965684 wait complete. 2024-04-26T00:43:21.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff341ed0640 10 monclient: build_initial_monmap 2024-04-26T00:43:21.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff341ed0640 10 monclient: monmap: 2024-04-26T00:43:21.717 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:21.718 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:43:21.718 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:43:21.718 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:43:21.718 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:21.718 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:43:21.718 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:43:21.718 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:43:21.718 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:43:21.718 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:21.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff341ed0640 1 Processor -- start 2024-04-26T00:43:21.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff341ed0640 1 -- start start 2024-04-26T00:43:21.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff341ed0640 10 monclient: init 2024-04-26T00:43:21.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff341ed0640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:21.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff341ed0640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:43:21.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff341ed0640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff33c131ec0 0x7ff33c130dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:21.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff341ed0640 10 monclient(hunting): picked mon.noname-a con 0x7ff33c131ec0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:43:21.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff341ed0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff33c134d80 0x7ff33c131300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:21.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff341ed0640 10 monclient(hunting): picked mon.noname-b con 0x7ff33c134d80 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:43:21.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff341ed0640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff33c135700 0x7ff33c1366e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:21.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff341ed0640 10 monclient(hunting): picked mon.noname-c con 0x7ff33c135700 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:21.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff341ed0640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:21.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff341ed0640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff33c072a50 con 0x7ff33c131ec0 2024-04-26T00:43:21.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff341ed0640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:21.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff341ed0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff33c0728d0 con 0x7ff33c134d80 2024-04-26T00:43:21.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff341ed0640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:21.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff341ed0640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff33c1327e0 con 0x7ff33c135700 2024-04-26T00:43:21.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff341ed0640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:21.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff341ed0640 10 monclient(hunting): authenticate will time out at 2383.016994s 2024-04-26T00:43:21.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff33bfff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff33c135700 0x7ff33c1366e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:21.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff33bfff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff33c135700 0x7ff33c1366e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51314/0 (socket says 172.21.15.37:51314) 2024-04-26T00:43:21.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff33b7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff33c131ec0 0x7ff33c130dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:21.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff33bfff640 1 -- 172.21.15.37:0/1187805872 learned_addr learned my addr 172.21.15.37:0/1187805872 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:21.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff33bfff640 10 monclient(hunting): get_auth_request con 0x7ff33c135700 auth_method 0 2024-04-26T00:43:21.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff33bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:21.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff33bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:21.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff33bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:21.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff33b7fe640 10 monclient(hunting): get_auth_request con 0x7ff33c131ec0 auth_method 0 2024-04-26T00:43:21.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff33b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:21.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff33b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:21.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff33b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:21.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff33bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:21.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff33bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:21.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff33bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:21.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff33b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:21.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff33b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:21.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff33affd640 1 --2- 172.21.15.37:0/1187805872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff33c134d80 0x7ff33c131300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:21.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff33b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:21.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff33affd640 10 monclient(hunting): get_auth_request con 0x7ff33c134d80 auth_method 0 2024-04-26T00:43:21.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff33affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:21.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff33affd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:21.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff33affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:21.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff33affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:21.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff33affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:21.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff33affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:21.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff33bfff640 10 monclient(hunting): handle_auth_done global_id 254474 payload 824 2024-04-26T00:43:21.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff33bfff640 1 -- 172.21.15.37:0/1187805872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff33c134d80 msgr2=0x7ff33c131300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:21.721 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff33bfff640 1 --2- 172.21.15.37:0/1187805872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff33c134d80 0x7ff33c131300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:21.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff33bfff640 1 -- 172.21.15.37:0/1187805872 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff33c131ec0 msgr2=0x7ff33c130dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:21.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff33bfff640 1 --2- 172.21.15.37:0/1187805872 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff33c131ec0 0x7ff33c130dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:21.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff33bfff640 10 monclient: _finish_hunting 0 2024-04-26T00:43:21.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff33bfff640 1 monclient: found mon.noname-c 2024-04-26T00:43:21.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff33bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:21.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff33bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:43:21.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff33bfff640 1 -- 172.21.15.37:0/1187805872 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff33c134b50 con 0x7ff33c135700 2024-04-26T00:43:21.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff33b7fe640 1 --2- 172.21.15.37:0/1187805872 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff33c131ec0 0x7ff33c130dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:43:21.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff33bfff640 1 -- 172.21.15.37:0/1187805872 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff33c135700 msgr2=0x7ff33c1366e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:21.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.711+0000 7ff33bfff640 1 --2- 172.21.15.37:0/1187805872 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff33c135700 0x7ff33c1366e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff33400c3f0 tx=0x7ff33400c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:21.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.715+0000 7ff333fff640 1 -- 172.21.15.37:0/1187805872 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff334017070 con 0x7ff33c135700 2024-04-26T00:43:21.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.715+0000 7ff333fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:21.722 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.715+0000 7ff333fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:43:21.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.715+0000 7ff333fff640 10 monclient: dump: 2024-04-26T00:43:21.723 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:21.723 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:21.723 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:21.723 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:21.723 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:21.723 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:21.723 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:21.723 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:21.723 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:21.723 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:21.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.715+0000 7ff333fff640 10 monclient: _finish_auth 0 2024-04-26T00:43:21.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.715+0000 7ff333fff640 10 monclient: _check_auth_tickets 2024-04-26T00:43:21.723 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.715+0000 7ff333fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:21.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.715+0000 7ff333fff640 1 -- 172.21.15.37:0/1187805872 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3340070d0 con 0x7ff33c135700 2024-04-26T00:43:21.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.715+0000 7ff333fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:21.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.715+0000 7ff333fff640 1 -- 172.21.15.37:0/1187805872 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3340074c0 con 0x7ff33c135700 2024-04-26T00:43:21.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.715+0000 7ff333fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:21.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.715+0000 7ff333fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:43:21.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.715+0000 7ff333fff640 10 monclient: dump: 2024-04-26T00:43:21.724 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:21.724 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:21.724 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:21.724 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:21.724 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:21.724 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:21.724 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:21.724 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:21.724 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:21.724 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:21.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.715+0000 7ff339ffb640 4 set_mon_vals no callback set 2024-04-26T00:43:21.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.715+0000 7ff341ed0640 5 monclient: authenticate success, global_id 254474 2024-04-26T00:43:21.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.715+0000 7ff341ed0640 10 monclient: _renew_subs 2024-04-26T00:43:21.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.715+0000 7ff341ed0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:21.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.715+0000 7ff341ed0640 1 -- 172.21.15.37:0/1187805872 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff33c138380 con 0x7ff33c135700 2024-04-26T00:43:21.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.715+0000 7ff341ed0640 10 monclient: _renew_subs 2024-04-26T00:43:21.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.715+0000 7ff341ed0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:21.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.715+0000 7ff341ed0640 1 -- 172.21.15.37:0/1187805872 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff33c139dd0 con 0x7ff33c135700 2024-04-26T00:43:21.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.715+0000 7ff341ed0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:21.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.715+0000 7ff341ed0640 1 -- 172.21.15.37:0/1187805872 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff33c138f40 con 0x7ff33c135700 2024-04-26T00:43:21.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.715+0000 7ff333fff640 1 -- 172.21.15.37:0/1187805872 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff33401a070 con 0x7ff33c135700 2024-04-26T00:43:21.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.715+0000 7ff333fff640 1 --2- 172.21.15.37:0/1187805872 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff31c035ed0 0x7ff31c038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:21.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.715+0000 7ff333fff640 1 -- 172.21.15.37:0/1187805872 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(899..899 src has 822..899) ==== 3550+0+0 (secure 0 0 0) 0x7ff334047de0 con 0x7ff33c135700 2024-04-26T00:43:21.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.715+0000 7ff333fff640 1 --2- 172.21.15.37:0/1187805872 >> v2:172.21.15.37:6806/3204000569 conn(0x7ff31c03ba20 0x7ff31c03de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:21.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.715+0000 7ff333fff640 1 -- 172.21.15.37:0/1187805872 --> v2:172.21.15.37:6806/3204000569 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff334011e50 con 0x7ff31c03ba20 2024-04-26T00:43:21.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.715+0000 7ff333fff640 1 -- 172.21.15.37:0/1187805872 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_get_version_reply(handle=1 version=899) ==== 24+0+0 (secure 0 0 0) 0x7ff33404a3b0 con 0x7ff33c135700 2024-04-26T00:43:21.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.715+0000 7ff333fff640 10 monclient: handle_get_version_reply finishing 1 version 899 2024-04-26T00:43:21.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.715+0000 7ff33affd640 1 --2- 172.21.15.37:0/1187805872 >> v2:172.21.15.37:6806/3204000569 conn(0x7ff31c03ba20 0x7ff31c03de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:21.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.715+0000 7ff33b7fe640 1 --2- 172.21.15.37:0/1187805872 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff31c035ed0 0x7ff31c038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:21.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.715+0000 7ff33affd640 10 monclient: get_auth_request con 0x7ff31c03ba20 auth_method 0 2024-04-26T00:43:21.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.715+0000 7ff33b7fe640 10 monclient: get_auth_request con 0x7ff31c035ed0 auth_method 0 2024-04-26T00:43:21.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.719+0000 7ff33b7fe640 1 --2- 172.21.15.37:0/1187805872 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff31c035ed0 0x7ff31c038390 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7ff33c070780 tx=0x7ff3240a7000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:21.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.719+0000 7ff33affd640 1 --2- 172.21.15.37:0/1187805872 >> v2:172.21.15.37:6806/3204000569 conn(0x7ff31c03ba20 0x7ff31c03de40 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:21.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.719+0000 7ff333fff640 1 -- 172.21.15.37:0/1187805872 <== osd.0 v2:172.21.15.37:6806/3204000569 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff334011e50 con 0x7ff31c03ba20 2024-04-26T00:43:21.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.735+0000 7ff341ed0640 1 -- 172.21.15.37:0/1187805872 --> v2:172.21.15.37:6806/3204000569 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ff33c04f580 con 0x7ff31c03ba20 2024-04-26T00:43:21.735 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.735+0000 7ff333fff640 1 -- 172.21.15.37:0/1187805872 <== osd.0 v2:172.21.15.37:6806/3204000569 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ff33c04f580 con 0x7ff31c03ba20 2024-04-26T00:43:21.736 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:43:21.736 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T00:43:21.736 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:43:21.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.735+0000 7ff341ed0640 1 -- 172.21.15.37:0/1187805872 >> v2:172.21.15.37:6806/3204000569 conn(0x7ff31c03ba20 msgr2=0x7ff31c03de40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:21.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.735+0000 7ff341ed0640 1 --2- 172.21.15.37:0/1187805872 >> v2:172.21.15.37:6806/3204000569 conn(0x7ff31c03ba20 0x7ff31c03de40 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:21.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.735+0000 7ff341ed0640 1 -- 172.21.15.37:0/1187805872 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff31c035ed0 msgr2=0x7ff31c038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:21.737 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.735+0000 7ff341ed0640 1 --2- 172.21.15.37:0/1187805872 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff31c035ed0 0x7ff31c038390 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7ff33c070780 tx=0x7ff3240a7000 comp rx=0 tx=0).stop 2024-04-26T00:43:21.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.735+0000 7ff341ed0640 10 monclient: shutdown 2024-04-26T00:43:21.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.735+0000 7ff341ed0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:21.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.735+0000 7ff341ed0640 1 -- 172.21.15.37:0/1187805872 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff33c135700 msgr2=0x7ff33c1366e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:21.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.735+0000 7ff341ed0640 1 --2- 172.21.15.37:0/1187805872 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff33c135700 0x7ff33c1366e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff33400c3f0 tx=0x7ff33400c8b0 comp rx=0 tx=0).stop 2024-04-26T00:43:21.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.735+0000 7ff33bfff640 1 -- 172.21.15.37:0/1187805872 reap_dead start 2024-04-26T00:43:21.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.735+0000 7ff341ed0640 1 -- 172.21.15.37:0/1187805872 shutdown_connections 2024-04-26T00:43:21.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.735+0000 7ff341ed0640 1 -- 172.21.15.37:0/1187805872 >> 172.21.15.37:0/1187805872 conn(0x7ff33c09abb0 msgr2=0x7ff33c0a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:21.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.735+0000 7ff341ed0640 1 -- 172.21.15.37:0/1187805872 shutdown_connections 2024-04-26T00:43:21.738 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.735+0000 7ff341ed0640 1 -- 172.21.15.37:0/1187805872 wait complete. 2024-04-26T00:43:21.752 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-26T00:43:21.866 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.863+0000 7f3b23d49640 10 monclient: get_monmap_and_config 2024-04-26T00:43:21.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.863+0000 7f3b23d49640 10 monclient: build_initial_monmap 2024-04-26T00:43:21.867 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.863+0000 7f3b23d49640 10 monclient: monmap: 2024-04-26T00:43:21.867 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:21.867 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:21.867 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:43:21.867278+0000 2024-04-26T00:43:21.867 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:43:21.867278+0000 2024-04-26T00:43:21.867 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:21.867 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:21.867 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:43:21.867 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:43:21.867 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:43:21.867 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:43:21.867 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:43:21.867 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:21.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b23d49640 1 Processor -- start 2024-04-26T00:43:21.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b23d49640 1 -- start start 2024-04-26T00:43:21.868 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b23d49640 10 monclient: init 2024-04-26T00:43:21.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b23d49640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:21.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b23d49640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-04-26T00:43:21.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b23d49640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3b1c0a0300 0x7f3b1c09fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:21.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b23d49640 10 monclient(hunting): picked mon.noname-b con 0x7f3b1c0a0300 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:21.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b23d49640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f3b1c0a06e0 0x7f3b1c134ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:21.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b23d49640 10 monclient(hunting): picked mon.noname-a con 0x7f3b1c0a06e0 addr v2:172.21.15.37:3300/0 2024-04-26T00:43:21.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b23d49640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f3b1c1350e0 0x7f3b1c135570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:21.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b23d49640 10 monclient(hunting): picked mon.noname-c con 0x7f3b1c1350e0 addr v2:172.21.15.45:3300/0 2024-04-26T00:43:21.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b23d49640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:21.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b23d49640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f3b1c0a13f0 con 0x7f3b1c0a06e0 2024-04-26T00:43:21.870 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b23d49640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:21.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b23d49640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f3b1c09c190 con 0x7f3b1c1350e0 2024-04-26T00:43:21.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b23d49640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:21.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b23d49640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f3b1c0a1110 con 0x7f3b1c0a0300 2024-04-26T00:43:21.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b23d49640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:21.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b23d49640 10 monclient(hunting): authenticate will time out at 2383.172846s 2024-04-26T00:43:21.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b21ae5640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3b1c0a0300 0x7f3b1c09fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:21.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b212e4640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f3b1c0a06e0 0x7f3b1c134ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:21.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b212e4640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f3b1c0a06e0 0x7f3b1c134ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58158/0 (socket says 172.21.15.37:58158) 2024-04-26T00:43:21.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b212e4640 1 -- 172.21.15.37:0/38183609 learned_addr learned my addr 172.21.15.37:0/38183609 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:21.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b212e4640 10 monclient(hunting): get_auth_request con 0x7f3b1c0a06e0 auth_method 0 2024-04-26T00:43:21.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b212e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:21.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b212e4640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:21.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b212e4640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:21.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b21ae5640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3b1c0a0300 0x7f3b1c09fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51320/0 (socket says 172.21.15.37:51320) 2024-04-26T00:43:21.871 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b21ae5640 10 monclient(hunting): get_auth_request con 0x7f3b1c0a0300 auth_method 0 2024-04-26T00:43:21.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b21ae5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:21.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b21ae5640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:21.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b21ae5640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:21.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b212e4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:21.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b212e4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:21.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b212e4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:21.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b21ae5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:21.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b21ae5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:21.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b21ae5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:21.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b222e6640 1 --2- 172.21.15.37:0/38183609 >> v2:172.21.15.45:3300/0 conn(0x7f3b1c1350e0 0x7f3b1c135570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:21.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b222e6640 10 monclient(hunting): get_auth_request con 0x7f3b1c1350e0 auth_method 0 2024-04-26T00:43:21.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b222e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:21.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b222e6640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:21.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b222e6640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:21.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b212e4640 10 monclient(hunting): handle_auth_done global_id 284148 payload 274 2024-04-26T00:43:21.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b212e4640 1 -- 172.21.15.37:0/38183609 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3b1c0a0300 msgr2=0x7f3b1c09fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:21.872 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b212e4640 1 --2- 172.21.15.37:0/38183609 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3b1c0a0300 0x7f3b1c09fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:21.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b212e4640 1 -- 172.21.15.37:0/38183609 >> v2:172.21.15.45:3300/0 conn(0x7f3b1c1350e0 msgr2=0x7f3b1c135570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:21.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b212e4640 1 --2- 172.21.15.37:0/38183609 >> v2:172.21.15.45:3300/0 conn(0x7f3b1c1350e0 0x7f3b1c135570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:21.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b212e4640 10 monclient: _finish_hunting 0 2024-04-26T00:43:21.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b212e4640 1 monclient: found mon.noname-a 2024-04-26T00:43:21.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b212e4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:21.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b212e4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:43:21.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b212e4640 1 -- 172.21.15.37:0/38183609 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b1c0a16d0 con 0x7f3b1c0a06e0 2024-04-26T00:43:21.873 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.867+0000 7f3b21ae5640 1 --2- 172.21.15.37:0/38183609 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3b1c0a0300 0x7f3b1c09fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:43:21.877 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b212e4640 1 -- 172.21.15.37:0/38183609 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b1c0a06e0 msgr2=0x7f3b1c134ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:21.877 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b212e4640 1 --2- 172.21.15.37:0/38183609 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b1c0a06e0 0x7f3b1c134ba0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c0a2da0 tx=0x7f3b1001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4df2dd7e4f8d189f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:21.877 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b20ae3640 1 -- 172.21.15.37:0/38183609 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b100b0070 con 0x7f3b1c0a06e0 2024-04-26T00:43:21.877 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b20ae3640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:21.877 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b20ae3640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:43:21.877 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b20ae3640 10 monclient: dump: 2024-04-26T00:43:21.877 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:21.877 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:21.877 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:21.877 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:21.877 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:21.877 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:21.877 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:21.878 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:21.878 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:21.878 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:21.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b20ae3640 10 monclient: _finish_auth 0 2024-04-26T00:43:21.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b20ae3640 10 monclient: _check_auth_tickets 2024-04-26T00:43:21.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b20ae3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:21.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b20ae3640 1 -- 172.21.15.37:0/38183609 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b100ab600 con 0x7f3b1c0a06e0 2024-04-26T00:43:21.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b20ae3640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:21.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b20ae3640 1 -- 172.21.15.37:0/38183609 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b100ab9d0 con 0x7f3b1c0a06e0 2024-04-26T00:43:21.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b23d49640 5 monclient: authenticate success, global_id 284148 2024-04-26T00:43:21.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b23d49640 10 monclient: get_monmap_and_config success 2024-04-26T00:43:21.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b23d49640 4 set_mon_vals no callback set 2024-04-26T00:43:21.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b20ae3640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:21.878 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b20ae3640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:43:21.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b20ae3640 10 monclient: dump: 2024-04-26T00:43:21.879 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:21.879 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:21.879 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:21.879 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:21.879 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:21.879 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:21.879 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:21.879 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:21.879 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:21.879 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:21.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b23d49640 10 monclient: shutdown 2024-04-26T00:43:21.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b23d49640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:21.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b23d49640 1 -- 172.21.15.37:0/38183609 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b1c0a06e0 msgr2=0x7f3b1c134ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:21.879 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b23d49640 1 --2- 172.21.15.37:0/38183609 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b1c0a06e0 0x7f3b1c134ba0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c0a2da0 tx=0x7f3b1001c3e0 comp rx=0 tx=0).stop 2024-04-26T00:43:21.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b23d49640 1 -- 172.21.15.37:0/38183609 shutdown_connections 2024-04-26T00:43:21.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b23d49640 1 --2- 172.21.15.37:0/38183609 >> v2:172.21.15.45:3300/0 conn(0x7f3b1c1350e0 0x7f3b1c135570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:21.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b23d49640 1 --2- 172.21.15.37:0/38183609 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b1c0a06e0 0x7f3b1c134ba0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:21.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b23d49640 1 --2- 172.21.15.37:0/38183609 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3b1c0a0300 0x7f3b1c09fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:21.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b23d49640 1 -- 172.21.15.37:0/38183609 >> 172.21.15.37:0/38183609 conn(0x7f3b1c0a1d50 msgr2=0x7f3b1c0a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:21.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b23d49640 1 -- 172.21.15.37:0/38183609 shutdown_connections 2024-04-26T00:43:21.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b23d49640 1 -- 172.21.15.37:0/38183609 wait complete. 2024-04-26T00:43:21.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b23d49640 10 monclient: build_initial_monmap 2024-04-26T00:43:21.880 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b23d49640 10 monclient: monmap: 2024-04-26T00:43:21.880 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:21.880 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:43:21.880 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:43:21.880 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:43:21.880 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:21.881 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:43:21.881 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:43:21.881 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:43:21.881 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:43:21.881 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:21.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b23d49640 1 Processor -- start 2024-04-26T00:43:21.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b23d49640 1 -- start start 2024-04-26T00:43:21.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b23d49640 10 monclient: init 2024-04-26T00:43:21.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b23d49640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:21.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b23d49640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:43:21.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b23d49640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b1c0a0300 0x7f3b1c130b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:21.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b23d49640 10 monclient(hunting): picked mon.noname-b con 0x7f3b1c0a0300 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:43:21.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b23d49640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b1c0a06e0 0x7f3b1c131080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:21.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b23d49640 10 monclient(hunting): picked mon.noname-a con 0x7f3b1c0a06e0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:43:21.881 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.871+0000 7f3b23d49640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3b1c1350e0 0x7f3b1c11b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:21.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b23d49640 10 monclient(hunting): picked mon.noname-c con 0x7f3b1c1350e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:21.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b23d49640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:21.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b23d49640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f3b1c13e6f0 con 0x7f3b1c0a06e0 2024-04-26T00:43:21.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b23d49640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:21.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b23d49640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3b1c13e870 con 0x7f3b1c0a0300 2024-04-26T00:43:21.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b23d49640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:21.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b23d49640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f3b1c13e9f0 con 0x7f3b1c1350e0 2024-04-26T00:43:21.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b23d49640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:21.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b23d49640 10 monclient(hunting): authenticate will time out at 2383.180040s 2024-04-26T00:43:21.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b212e4640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b1c0a06e0 0x7f3b1c131080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:21.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b212e4640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b1c0a06e0 0x7f3b1c131080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58162/0 (socket says 172.21.15.37:58162) 2024-04-26T00:43:21.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b212e4640 1 -- 172.21.15.37:0/1390242774 learned_addr learned my addr 172.21.15.37:0/1390242774 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:21.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b212e4640 10 monclient(hunting): get_auth_request con 0x7f3b1c0a06e0 auth_method 0 2024-04-26T00:43:21.882 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b212e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:21.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b212e4640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:21.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b212e4640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:21.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b222e6640 1 --2- 172.21.15.37:0/1390242774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3b1c1350e0 0x7f3b1c11b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:21.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b222e6640 10 monclient(hunting): get_auth_request con 0x7f3b1c1350e0 auth_method 0 2024-04-26T00:43:21.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b222e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:21.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b222e6640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:21.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b222e6640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:21.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b212e4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:21.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b212e4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:21.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b212e4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:21.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b222e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:21.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b222e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:21.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b222e6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:21.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b21ae5640 1 --2- 172.21.15.37:0/1390242774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b1c0a0300 0x7f3b1c130b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:21.883 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b212e4640 10 monclient(hunting): handle_auth_done global_id 284151 payload 824 2024-04-26T00:43:21.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b212e4640 1 -- 172.21.15.37:0/1390242774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3b1c1350e0 msgr2=0x7f3b1c11b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:21.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b212e4640 1 --2- 172.21.15.37:0/1390242774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3b1c1350e0 0x7f3b1c11b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:21.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b212e4640 1 -- 172.21.15.37:0/1390242774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b1c0a0300 msgr2=0x7f3b1c130b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:21.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b212e4640 1 --2- 172.21.15.37:0/1390242774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b1c0a0300 0x7f3b1c130b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:21.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b212e4640 10 monclient: _finish_hunting 0 2024-04-26T00:43:21.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b212e4640 1 monclient: found mon.noname-a 2024-04-26T00:43:21.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b212e4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:21.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b212e4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:43:21.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b212e4640 1 -- 172.21.15.37:0/1390242774 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b1c11e500 con 0x7f3b1c0a06e0 2024-04-26T00:43:21.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b21ae5640 10 monclient: get_auth_request con 0x7f3b1c0a0300 auth_method 0 2024-04-26T00:43:21.884 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b21ae5640 1 --2- 172.21.15.37:0/1390242774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b1c0a0300 0x7f3b1c130b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:43:21.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b212e4640 1 -- 172.21.15.37:0/1390242774 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b1c0a06e0 msgr2=0x7f3b1c131080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:21.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b212e4640 1 --2- 172.21.15.37:0/1390242774 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b1c0a06e0 0x7f3b1c131080 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3b1001aa90 tx=0x7f3b10019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:21.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b0e7fc640 1 -- 172.21.15.37:0/1390242774 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b100b0040 con 0x7f3b1c0a06e0 2024-04-26T00:43:21.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b0e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:21.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b0e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:43:21.886 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b0e7fc640 10 monclient: dump: 2024-04-26T00:43:21.886 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:21.886 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:21.887 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:21.887 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:21.887 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:21.887 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:21.887 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:21.887 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:21.887 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:21.887 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:21.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b0e7fc640 10 monclient: _finish_auth 0 2024-04-26T00:43:21.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b0e7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:43:21.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b0e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:21.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b0e7fc640 1 -- 172.21.15.37:0/1390242774 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b100aa5b0 con 0x7f3b1c0a06e0 2024-04-26T00:43:21.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b0e7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:21.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b0e7fc640 1 -- 172.21.15.37:0/1390242774 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b100aa9a0 con 0x7f3b1c0a06e0 2024-04-26T00:43:21.887 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b23d49640 5 monclient: authenticate success, global_id 284151 2024-04-26T00:43:21.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b23d49640 10 monclient: _renew_subs 2024-04-26T00:43:21.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b23d49640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:43:21.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b23d49640 1 -- 172.21.15.37:0/1390242774 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b1c09f550 con 0x7f3b1c0a06e0 2024-04-26T00:43:21.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b0ffff640 4 set_mon_vals no callback set 2024-04-26T00:43:21.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b0e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:21.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b0e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:43:21.888 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b0e7fc640 10 monclient: dump: 2024-04-26T00:43:21.888 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:21.888 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:21.888 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:21.888 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:21.888 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:21.888 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:21.888 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:21.888 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:21.888 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:21.889 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:21.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b23d49640 10 monclient: _renew_subs 2024-04-26T00:43:21.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b23d49640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:43:21.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.875+0000 7f3b23d49640 1 -- 172.21.15.37:0/1390242774 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b1c071890 con 0x7f3b1c0a06e0 2024-04-26T00:43:21.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.879+0000 7f3b23d49640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:21.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.879+0000 7f3b23d49640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:21.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.879+0000 7f3b23d49640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:43:21.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.879+0000 7f3b0e7fc640 1 -- 172.21.15.37:0/1390242774 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f3b100aab40 con 0x7f3b1c0a06e0 2024-04-26T00:43:21.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.879+0000 7f3b23d49640 1 -- 172.21.15.37:0/1390242774 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b1c04f510 con 0x7f3b1c0a06e0 2024-04-26T00:43:21.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.879+0000 7f3b0e7fc640 1 --2- 172.21.15.37:0/1390242774 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3b00035ed0 0x7f3b00038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:21.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.879+0000 7f3b0e7fc640 1 -- 172.21.15.37:0/1390242774 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(899..899 src has 822..899) ==== 3550+0+0 (secure 0 0 0) 0x7f3b100e2480 con 0x7f3b1c0a06e0 2024-04-26T00:43:21.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.879+0000 7f3b21ae5640 1 --2- 172.21.15.37:0/1390242774 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3b00035ed0 0x7f3b00038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:21.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.879+0000 7f3b21ae5640 10 monclient: get_auth_request con 0x7f3b00035ed0 auth_method 0 2024-04-26T00:43:21.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.879+0000 7f3b21ae5640 1 --2- 172.21.15.37:0/1390242774 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3b00035ed0 0x7f3b00038390 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f3b18009ae0 tx=0x7f3b18008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:21.889 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.883+0000 7f3b0e7fc640 1 -- 172.21.15.37:0/1390242774 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f3b100160f0 con 0x7f3b1c0a06e0 2024-04-26T00:43:21.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.883+0000 7f3b0e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:21.890 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:21.883+0000 7f3b0e7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:43:22.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.047+0000 7f3b23d49640 1 -- 172.21.15.37:0/1390242774 --> v2:172.21.15.45:6812/2379918623 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f3b1c09eb60 con 0x7f3b00035ed0 2024-04-26T00:43:22.047 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.047+0000 7f3b0e7fc640 1 -- 172.21.15.37:0/1390242774 <== mgr.4107 v2:172.21.15.45:6812/2379918623 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f3b1c09eb60 con 0x7f3b00035ed0 2024-04-26T00:43:22.048 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 0 to deep-scrub 2024-04-26T00:43:22.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.047+0000 7f3b23d49640 1 -- 172.21.15.37:0/1390242774 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3b00035ed0 msgr2=0x7f3b00038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:22.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.047+0000 7f3b23d49640 1 --2- 172.21.15.37:0/1390242774 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3b00035ed0 0x7f3b00038390 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f3b18009ae0 tx=0x7f3b18008040 comp rx=0 tx=0).stop 2024-04-26T00:43:22.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.047+0000 7f3b23d49640 10 monclient: shutdown 2024-04-26T00:43:22.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.047+0000 7f3b23d49640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:22.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.047+0000 7f3b23d49640 1 -- 172.21.15.37:0/1390242774 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b1c0a06e0 msgr2=0x7f3b1c131080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:22.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.047+0000 7f3b23d49640 1 --2- 172.21.15.37:0/1390242774 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b1c0a06e0 0x7f3b1c131080 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3b1001aa90 tx=0x7f3b10019180 comp rx=0 tx=0).stop 2024-04-26T00:43:22.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.051+0000 7f3b23d49640 1 -- 172.21.15.37:0/1390242774 shutdown_connections 2024-04-26T00:43:22.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.051+0000 7f3b23d49640 1 --2- 172.21.15.37:0/1390242774 >> v2:172.21.15.45:6812/2379918623 conn(0x7f3b00035ed0 0x7f3b00038390 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:22.051 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.051+0000 7f3b23d49640 1 --2- 172.21.15.37:0/1390242774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3b1c1350e0 0x7f3b1c11b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:22.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.051+0000 7f3b23d49640 1 --2- 172.21.15.37:0/1390242774 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b1c0a06e0 0x7f3b1c131080 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:22.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.051+0000 7f3b23d49640 1 --2- 172.21.15.37:0/1390242774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b1c0a0300 0x7f3b1c130b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:22.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.051+0000 7f3b23d49640 1 -- 172.21.15.37:0/1390242774 >> 172.21.15.37:0/1390242774 conn(0x7f3b1c0a1d50 msgr2=0x7f3b1c0a3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:22.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.051+0000 7f3b23d49640 1 -- 172.21.15.37:0/1390242774 shutdown_connections 2024-04-26T00:43:22.052 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.051+0000 7f3b23d49640 1 -- 172.21.15.37:0/1390242774 wait complete. 2024-04-26T00:43:22.067 INFO:tasks.ceph:Scrubbing osd.1 2024-04-26T00:43:22.067 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-26T00:43:22.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f94835ff640 10 monclient: get_monmap_and_config 2024-04-26T00:43:22.187 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f94835ff640 10 monclient: build_initial_monmap 2024-04-26T00:43:22.188 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f94835ff640 10 monclient: monmap: 2024-04-26T00:43:22.188 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:22.188 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:22.188 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:43:22.188441+0000 2024-04-26T00:43:22.188 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:43:22.188441+0000 2024-04-26T00:43:22.188 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:22.188 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:22.188 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:43:22.188 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:43:22.188 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:43:22.188 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:43:22.188 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:43:22.188 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:22.190 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f94835ff640 1 Processor -- start 2024-04-26T00:43:22.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f94835ff640 1 -- start start 2024-04-26T00:43:22.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f94835ff640 10 monclient: init 2024-04-26T00:43:22.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f94835ff640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:22.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f94835ff640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-04-26T00:43:22.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f94835ff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f947c09bf90 0x7f947c0a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:22.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f94835ff640 10 monclient(hunting): picked mon.noname-b con 0x7f947c09bf90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:22.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f94835ff640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f947c09d9d0 0x7f947c09ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:22.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f94835ff640 10 monclient(hunting): picked mon.noname-a con 0x7f947c09d9d0 addr v2:172.21.15.37:3300/0 2024-04-26T00:43:22.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f94835ff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f947c09e330 addr v1:172.21.15.37:6789/0 2024-04-26T00:43:22.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f94835ff640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f947c09edc0 con 0x7f947c09e330 2024-04-26T00:43:22.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f94835ff640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:22.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f94835ff640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f947c0a17b0 con 0x7f947c09d9d0 2024-04-26T00:43:22.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f94835ff640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:22.191 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f94835ff640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f947c09c370 con 0x7f947c09bf90 2024-04-26T00:43:22.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f94835ff640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:22.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f94835ff640 10 monclient(hunting): authenticate will time out at 2383.494029s 2024-04-26T00:43:22.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f9481b9c640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f947c09e330 0x7f947c13a550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:49524/0 (socket says 172.21.15.37:49524) 2024-04-26T00:43:22.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f9480b9a640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f947c09d9d0 0x7f947c09ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:22.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f9481b9c640 1 -- 172.21.15.37:0/3480230859 learned_addr learned my addr 172.21.15.37:0/3480230859 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:43:22.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f948139b640 1 --2- 172.21.15.37:0/3480230859 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f947c09bf90 0x7f947c0a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:22.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f9480b9a640 10 monclient(hunting): get_auth_request con 0x7f947c09d9d0 auth_method 0 2024-04-26T00:43:22.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f9480b9a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:22.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f9480b9a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:22.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f9480b9a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:22.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f948139b640 10 monclient(hunting): get_auth_request con 0x7f947c09bf90 auth_method 0 2024-04-26T00:43:22.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f948139b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:22.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f948139b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:22.192 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f948139b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:22.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f948139b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:22.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f948139b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:22.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f948139b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:22.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f9480b9a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:22.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f9480b9a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:22.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f9480b9a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:22.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f948139b640 10 monclient(hunting): handle_auth_done global_id 254483 payload 274 2024-04-26T00:43:22.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f948139b640 1 -- 172.21.15.37:0/3480230859 >> v2:172.21.15.37:3300/0 conn(0x7f947c09d9d0 msgr2=0x7f947c09ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:22.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f948139b640 1 --2- 172.21.15.37:0/3480230859 >> v2:172.21.15.37:3300/0 conn(0x7f947c09d9d0 0x7f947c09ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:22.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f948139b640 1 -- 172.21.15.37:0/3480230859 >> v1:172.21.15.37:6789/0 conn(0x7f947c09e330 legacy=0x7f947c13a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:22.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f948139b640 10 monclient: _finish_hunting 0 2024-04-26T00:43:22.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f948139b640 1 monclient: found mon.noname-b 2024-04-26T00:43:22.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f948139b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:22.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f948139b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:43:22.193 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f948139b640 1 -- 172.21.15.37:0/3480230859 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f947c09e7c0 con 0x7f947c09bf90 2024-04-26T00:43:22.194 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.187+0000 7f9480b9a640 1 --2- 172.21.15.37:0/3480230859 >> v2:172.21.15.37:3300/0 conn(0x7f947c09d9d0 0x7f947c09ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:43:22.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.191+0000 7f948139b640 1 -- 172.21.15.37:0/3480230859 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f947c09bf90 msgr2=0x7f947c0a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:22.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.191+0000 7f948139b640 1 --2- 172.21.15.37:0/3480230859 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f947c09bf90 0x7f947c0a0dd0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9478009d90 tx=0x7f947801c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f003cb1952102e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:22.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.191+0000 7f9473fff640 1 -- 172.21.15.37:0/3480230859 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94780ac070 con 0x7f947c09bf90 2024-04-26T00:43:22.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.191+0000 7f9473fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:22.195 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.191+0000 7f9473fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:43:22.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.191+0000 7f9473fff640 10 monclient: dump: 2024-04-26T00:43:22.196 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:22.196 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:22.196 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:22.196 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:22.196 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:22.196 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:22.196 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:22.196 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:22.197 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:22.197 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:22.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.191+0000 7f9473fff640 10 monclient: _finish_auth 0 2024-04-26T00:43:22.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.191+0000 7f9473fff640 10 monclient: _check_auth_tickets 2024-04-26T00:43:22.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.191+0000 7f9473fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:22.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.191+0000 7f9473fff640 1 -- 172.21.15.37:0/3480230859 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94780a6680 con 0x7f947c09bf90 2024-04-26T00:43:22.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.191+0000 7f94835ff640 5 monclient: authenticate success, global_id 254483 2024-04-26T00:43:22.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.191+0000 7f94835ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:43:22.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.191+0000 7f9473fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:22.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.191+0000 7f94835ff640 10 monclient: get_monmap_and_config success 2024-04-26T00:43:22.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.191+0000 7f9473fff640 1 -- 172.21.15.37:0/3480230859 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94780a6a50 con 0x7f947c09bf90 2024-04-26T00:43:22.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.191+0000 7f94835ff640 4 set_mon_vals no callback set 2024-04-26T00:43:22.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.191+0000 7f9473fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:22.197 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.191+0000 7f9473fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:43:22.198 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.191+0000 7f9473fff640 10 monclient: dump: 2024-04-26T00:43:22.198 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:22.198 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:22.198 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:22.198 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:22.198 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:22.198 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:22.198 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:22.198 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:22.198 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:22.199 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:22.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.191+0000 7f94835ff640 10 monclient: shutdown 2024-04-26T00:43:22.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.191+0000 7f94835ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:22.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.191+0000 7f94835ff640 1 -- 172.21.15.37:0/3480230859 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f947c09bf90 msgr2=0x7f947c0a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:22.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.191+0000 7f94835ff640 1 --2- 172.21.15.37:0/3480230859 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f947c09bf90 0x7f947c0a0dd0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9478009d90 tx=0x7f947801c7f0 comp rx=0 tx=0).stop 2024-04-26T00:43:22.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.191+0000 7f94835ff640 1 -- 172.21.15.37:0/3480230859 shutdown_connections 2024-04-26T00:43:22.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.191+0000 7f94835ff640 1 --2- 172.21.15.37:0/3480230859 >> v2:172.21.15.37:3300/0 conn(0x7f947c09d9d0 0x7f947c09ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:22.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.191+0000 7f94835ff640 1 --2- 172.21.15.37:0/3480230859 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f947c09bf90 0x7f947c0a0dd0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:22.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.191+0000 7f94835ff640 1 -- 172.21.15.37:0/3480230859 >> 172.21.15.37:0/3480230859 conn(0x7f947c0a6080 msgr2=0x7f947c0a1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:22.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f94835ff640 1 -- 172.21.15.37:0/3480230859 shutdown_connections 2024-04-26T00:43:22.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f94835ff640 1 -- 172.21.15.37:0/3480230859 wait complete. 2024-04-26T00:43:22.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f94835ff640 10 monclient: build_initial_monmap 2024-04-26T00:43:22.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f94835ff640 10 monclient: monmap: 2024-04-26T00:43:22.200 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:22.200 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:43:22.200 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:43:22.200 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:43:22.200 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:22.200 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:43:22.200 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:43:22.200 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:43:22.200 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:43:22.200 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:22.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f94835ff640 1 Processor -- start 2024-04-26T00:43:22.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f94835ff640 1 -- start start 2024-04-26T00:43:22.201 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f94835ff640 10 monclient: init 2024-04-26T00:43:22.203 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f94835ff640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:22.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f94835ff640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:43:22.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f94835ff640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f947c09bf90 0x7f947c139a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:22.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f94835ff640 10 monclient(hunting): picked mon.noname-c con 0x7f947c09bf90 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:22.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f94835ff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f947c09d9d0 0x7f947c139f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:22.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f94835ff640 10 monclient(hunting): picked mon.noname-a con 0x7f947c09d9d0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:43:22.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f94835ff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f947c09e330 0x7f947c13a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:22.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f94835ff640 10 monclient(hunting): picked mon.noname-b con 0x7f947c09e330 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:43:22.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f94835ff640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:22.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f94835ff640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f947c13e6a0 con 0x7f947c09d9d0 2024-04-26T00:43:22.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f94835ff640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:22.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f94835ff640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f947c0a1cc0 con 0x7f947c09e330 2024-04-26T00:43:22.204 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f94835ff640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:22.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f94835ff640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f947c13e820 con 0x7f947c09bf90 2024-04-26T00:43:22.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f94835ff640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:22.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f94835ff640 10 monclient(hunting): authenticate will time out at 2383.501545s 2024-04-26T00:43:22.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f9480b9a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f947c09d9d0 0x7f947c139f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:22.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f948139b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f947c09bf90 0x7f947c139a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:22.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f9480b9a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f947c09d9d0 0x7f947c139f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58178/0 (socket says 172.21.15.37:58178) 2024-04-26T00:43:22.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f9480b9a640 1 -- 172.21.15.37:0/3136460842 learned_addr learned my addr 172.21.15.37:0/3136460842 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:22.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f948139b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f947c09bf90 0x7f947c139a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51332/0 (socket says 172.21.15.37:51332) 2024-04-26T00:43:22.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f9480b9a640 10 monclient(hunting): get_auth_request con 0x7f947c09d9d0 auth_method 0 2024-04-26T00:43:22.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f9480b9a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:22.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f9480b9a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:22.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f9480b9a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:22.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f948139b640 10 monclient(hunting): get_auth_request con 0x7f947c09bf90 auth_method 0 2024-04-26T00:43:22.205 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f948139b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:22.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f948139b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:22.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f948139b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:22.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f9480b9a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:22.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f9480b9a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:22.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f9480b9a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:22.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f948139b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:22.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f948139b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:22.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f948139b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:22.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f9480b9a640 10 monclient(hunting): handle_auth_done global_id 284157 payload 824 2024-04-26T00:43:22.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f9481b9c640 1 --2- 172.21.15.37:0/3136460842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f947c09e330 0x7f947c13a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:22.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f9480b9a640 1 -- 172.21.15.37:0/3136460842 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f947c09bf90 msgr2=0x7f947c139a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:22.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f9480b9a640 1 --2- 172.21.15.37:0/3136460842 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f947c09bf90 0x7f947c139a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:22.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f9480b9a640 1 -- 172.21.15.37:0/3136460842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f947c09e330 msgr2=0x7f947c13a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:22.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f9480b9a640 1 --2- 172.21.15.37:0/3136460842 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f947c09e330 0x7f947c13a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:22.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f9480b9a640 10 monclient: _finish_hunting 0 2024-04-26T00:43:22.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f9480b9a640 1 monclient: found mon.noname-a 2024-04-26T00:43:22.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f9480b9a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:22.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f9480b9a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:43:22.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.195+0000 7f9480b9a640 1 -- 172.21.15.37:0/3136460842 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f947c13ff00 con 0x7f947c09d9d0 2024-04-26T00:43:22.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.199+0000 7f9480b9a640 1 -- 172.21.15.37:0/3136460842 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f947c09d9d0 msgr2=0x7f947c139f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:22.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.199+0000 7f9480b9a640 1 --2- 172.21.15.37:0/3136460842 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f947c09d9d0 0x7f947c139f90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f946c00ebe0 tx=0x7f946c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:22.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.199+0000 7f9471ffb640 1 -- 172.21.15.37:0/3136460842 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f946c017070 con 0x7f947c09d9d0 2024-04-26T00:43:22.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.199+0000 7f9471ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:22.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.199+0000 7f9471ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:43:22.207 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.199+0000 7f9471ffb640 10 monclient: dump: 2024-04-26T00:43:22.207 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:22.207 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:22.207 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:22.208 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:22.208 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:22.208 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:22.208 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:22.208 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:22.208 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:22.208 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:22.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.199+0000 7f9471ffb640 10 monclient: _finish_auth 0 2024-04-26T00:43:22.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.199+0000 7f9471ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:43:22.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.199+0000 7f9471ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:22.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.199+0000 7f9471ffb640 1 -- 172.21.15.37:0/3136460842 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f946c00ce40 con 0x7f947c09d9d0 2024-04-26T00:43:22.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.199+0000 7f9471ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:22.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.199+0000 7f9471ffb640 1 -- 172.21.15.37:0/3136460842 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f946c013690 con 0x7f947c09d9d0 2024-04-26T00:43:22.208 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.199+0000 7f9471ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:22.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.199+0000 7f9471ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:43:22.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.199+0000 7f9471ffb640 10 monclient: dump: 2024-04-26T00:43:22.209 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:22.209 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:22.209 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:22.209 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:22.209 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:22.209 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:22.209 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:22.209 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:22.209 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:22.209 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:22.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.199+0000 7f94737fe640 4 set_mon_vals no callback set 2024-04-26T00:43:22.209 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.199+0000 7f94835ff640 5 monclient: authenticate success, global_id 284157 2024-04-26T00:43:22.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.199+0000 7f94835ff640 10 monclient: _renew_subs 2024-04-26T00:43:22.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.199+0000 7f94835ff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:43:22.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.199+0000 7f94835ff640 1 -- 172.21.15.37:0/3136460842 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f947c1401e0 con 0x7f947c09d9d0 2024-04-26T00:43:22.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.199+0000 7f94835ff640 10 monclient: _renew_subs 2024-04-26T00:43:22.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.199+0000 7f94835ff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:43:22.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.199+0000 7f94835ff640 1 -- 172.21.15.37:0/3136460842 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f947c13f010 con 0x7f947c09d9d0 2024-04-26T00:43:22.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.199+0000 7f94835ff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:43:22.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.199+0000 7f94835ff640 1 -- 172.21.15.37:0/3136460842 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f947c04f580 con 0x7f947c09d9d0 2024-04-26T00:43:22.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.199+0000 7f9471ffb640 1 -- 172.21.15.37:0/3136460842 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f946c01a070 con 0x7f947c09d9d0 2024-04-26T00:43:22.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.199+0000 7f9471ffb640 1 --2- 172.21.15.37:0/3136460842 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9460035e80 0x7f9460038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:22.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.199+0000 7f9471ffb640 1 -- 172.21.15.37:0/3136460842 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(899..899 src has 822..899) ==== 3550+0+0 (secure 0 0 0) 0x7f946c048d60 con 0x7f947c09d9d0 2024-04-26T00:43:22.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.199+0000 7f9471ffb640 1 --2- 172.21.15.37:0/3136460842 >> v2:172.21.15.37:6804/1084527868 conn(0x7f946003b950 0x7f946003dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:22.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.199+0000 7f9471ffb640 1 -- 172.21.15.37:0/3136460842 --> v2:172.21.15.37:6804/1084527868 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f946003e3e0 con 0x7f946003b950 2024-04-26T00:43:22.210 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.199+0000 7f9471ffb640 1 -- 172.21.15.37:0/3136460842 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_get_version_reply(handle=1 version=899) ==== 24+0+0 (secure 0 0 0) 0x7f946c04b330 con 0x7f947c09d9d0 2024-04-26T00:43:22.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.199+0000 7f9471ffb640 10 monclient: handle_get_version_reply finishing 1 version 899 2024-04-26T00:43:22.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.199+0000 7f9481b9c640 1 --2- 172.21.15.37:0/3136460842 >> v2:172.21.15.37:6804/1084527868 conn(0x7f946003b950 0x7f946003dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:22.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.199+0000 7f9481b9c640 10 monclient: get_auth_request con 0x7f946003b950 auth_method 0 2024-04-26T00:43:22.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.199+0000 7f948139b640 1 --2- 172.21.15.37:0/3136460842 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9460035e80 0x7f9460038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:22.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.199+0000 7f948139b640 10 monclient: get_auth_request con 0x7f9460035e80 auth_method 0 2024-04-26T00:43:22.211 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.203+0000 7f948139b640 1 --2- 172.21.15.37:0/3136460842 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9460035e80 0x7f9460038340 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f947c09f410 tx=0x7f94780a7000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:22.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.203+0000 7f9481b9c640 1 --2- 172.21.15.37:0/3136460842 >> v2:172.21.15.37:6804/1084527868 conn(0x7f946003b950 0x7f946003dd70 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:22.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.203+0000 7f9471ffb640 1 -- 172.21.15.37:0/3136460842 <== osd.1 v2:172.21.15.37:6804/1084527868 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f946003e3e0 con 0x7f946003b950 2024-04-26T00:43:22.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.219+0000 7f94835ff640 1 -- 172.21.15.37:0/3136460842 --> v2:172.21.15.37:6804/1084527868 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f947c09ced0 con 0x7f946003b950 2024-04-26T00:43:22.220 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.219+0000 7f9471ffb640 1 -- 172.21.15.37:0/3136460842 <== osd.1 v2:172.21.15.37:6804/1084527868 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f947c09ced0 con 0x7f946003b950 2024-04-26T00:43:22.220 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:43:22.220 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T00:43:22.220 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:43:22.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.219+0000 7f94835ff640 1 -- 172.21.15.37:0/3136460842 >> v2:172.21.15.37:6804/1084527868 conn(0x7f946003b950 msgr2=0x7f946003dd70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:22.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.219+0000 7f94835ff640 1 --2- 172.21.15.37:0/3136460842 >> v2:172.21.15.37:6804/1084527868 conn(0x7f946003b950 0x7f946003dd70 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:22.221 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.219+0000 7f94835ff640 1 -- 172.21.15.37:0/3136460842 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9460035e80 msgr2=0x7f9460038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:22.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.219+0000 7f94835ff640 1 --2- 172.21.15.37:0/3136460842 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9460035e80 0x7f9460038340 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f947c09f410 tx=0x7f94780a7000 comp rx=0 tx=0).stop 2024-04-26T00:43:22.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.219+0000 7f94835ff640 10 monclient: shutdown 2024-04-26T00:43:22.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.219+0000 7f94835ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:22.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.219+0000 7f94835ff640 1 -- 172.21.15.37:0/3136460842 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f947c09d9d0 msgr2=0x7f947c139f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:22.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.219+0000 7f94835ff640 1 --2- 172.21.15.37:0/3136460842 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f947c09d9d0 0x7f947c139f90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f946c00ebe0 tx=0x7f946c00c520 comp rx=0 tx=0).stop 2024-04-26T00:43:22.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.219+0000 7f9481b9c640 1 -- 172.21.15.37:0/3136460842 reap_dead start 2024-04-26T00:43:22.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.219+0000 7f94835ff640 1 -- 172.21.15.37:0/3136460842 shutdown_connections 2024-04-26T00:43:22.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.219+0000 7f94835ff640 1 -- 172.21.15.37:0/3136460842 >> 172.21.15.37:0/3136460842 conn(0x7f947c0a6080 msgr2=0x7f947c09c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:22.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.219+0000 7f94835ff640 1 -- 172.21.15.37:0/3136460842 shutdown_connections 2024-04-26T00:43:22.222 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.219+0000 7f94835ff640 1 -- 172.21.15.37:0/3136460842 wait complete. 2024-04-26T00:43:22.237 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-26T00:43:22.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefb0449640 10 monclient: get_monmap_and_config 2024-04-26T00:43:22.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefb0449640 10 monclient: build_initial_monmap 2024-04-26T00:43:22.352 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefb0449640 10 monclient: monmap: 2024-04-26T00:43:22.352 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:22.352 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:22.352 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:43:22.352858+0000 2024-04-26T00:43:22.352 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:43:22.352858+0000 2024-04-26T00:43:22.352 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:22.353 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:22.353 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:43:22.353 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:43:22.353 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:43:22.353 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:43:22.353 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:43:22.353 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:22.355 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefb0449640 1 Processor -- start 2024-04-26T00:43:22.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefb0449640 1 -- start start 2024-04-26T00:43:22.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefb0449640 10 monclient: init 2024-04-26T00:43:22.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefb0449640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:22.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefb0449640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-04-26T00:43:22.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefb0449640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fefa8072500 0x7fefa80728e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:22.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefb0449640 10 monclient(hunting): picked mon.noname-b con 0x7fefa8072500 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:22.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefb0449640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fefa8133b80 0x7fefa80a1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:22.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefb0449640 10 monclient(hunting): picked mon.noname-a con 0x7fefa8133b80 addr v2:172.21.15.37:3300/0 2024-04-26T00:43:22.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefb0449640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fefa8133fe0 addr v1:172.21.15.45:6789/0 2024-04-26T00:43:22.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefb0449640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefa80a33f0 con 0x7fefa8133fe0 2024-04-26T00:43:22.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefb0449640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:22.356 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefb0449640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fefa806e4e0 con 0x7fefa8133b80 2024-04-26T00:43:22.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefb0449640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:22.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefb0449640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fefa806e7c0 con 0x7fefa8072500 2024-04-26T00:43:22.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefb0449640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:22.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefb0449640 10 monclient(hunting): authenticate will time out at 2383.659260s 2024-04-26T00:43:22.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefae1e5640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fefa8072500 0x7fefa80728e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:22.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefad9e4640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fefa8133b80 0x7fefa80a1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:22.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefad9e4640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fefa8133b80 0x7fefa80a1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58190/0 (socket says 172.21.15.37:58190) 2024-04-26T00:43:22.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefae1e5640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fefa8072500 0x7fefa80728e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51344/0 (socket says 172.21.15.37:51344) 2024-04-26T00:43:22.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefad9e4640 1 -- 172.21.15.37:0/4198234921 learned_addr learned my addr 172.21.15.37:0/4198234921 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:22.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefad9e4640 10 monclient(hunting): get_auth_request con 0x7fefa8133b80 auth_method 0 2024-04-26T00:43:22.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefad9e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:22.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefad9e4640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:22.357 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefad9e4640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:22.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefae1e5640 10 monclient(hunting): get_auth_request con 0x7fefa8072500 auth_method 0 2024-04-26T00:43:22.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefae1e5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:22.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefae1e5640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:22.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefae1e5640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:22.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefad9e4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:22.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefad9e4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:22.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefad9e4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:22.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefae1e5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:22.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefae1e5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:22.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefae1e5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:22.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefad9e4640 10 monclient(hunting): handle_auth_done global_id 284160 payload 274 2024-04-26T00:43:22.358 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefad9e4640 1 -- 172.21.15.37:0/4198234921 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fefa8072500 msgr2=0x7fefa80728e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:22.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefad9e4640 1 --2- 172.21.15.37:0/4198234921 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fefa8072500 0x7fefa80728e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:22.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefad9e4640 1 -- 172.21.15.37:0/4198234921 >> v1:172.21.15.45:6789/0 conn(0x7fefa8133fe0 legacy=0x7fefa80a1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:22.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefad9e4640 10 monclient: _finish_hunting 0 2024-04-26T00:43:22.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefad9e4640 1 monclient: found mon.noname-a 2024-04-26T00:43:22.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefad9e4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:22.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefad9e4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:43:22.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefad9e4640 1 -- 172.21.15.37:0/4198234921 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefa806eaa0 con 0x7fefa8133b80 2024-04-26T00:43:22.359 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.351+0000 7fefae1e5640 1 --2- 172.21.15.37:0/4198234921 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fefa8072500 0x7fefa80728e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:43:22.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefad9e4640 1 -- 172.21.15.37:0/4198234921 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fefa8133b80 msgr2=0x7fefa80a1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:22.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefad9e4640 1 --2- 172.21.15.37:0/4198234921 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fefa8133b80 0x7fefa80a1520 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fef98009870 tx=0x7fef9801c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffbf9391f5e836ac server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:22.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefad1e3640 1 -- 172.21.15.37:0/4198234921 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef980b2070 con 0x7fefa8133b80 2024-04-26T00:43:22.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefad1e3640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:22.363 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefad1e3640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:43:22.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefad1e3640 10 monclient: dump: 2024-04-26T00:43:22.364 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:22.364 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:22.364 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:22.364 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:22.364 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:22.364 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:22.364 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:22.364 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:22.364 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:22.364 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:22.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefad1e3640 10 monclient: _finish_auth 0 2024-04-26T00:43:22.364 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefad1e3640 10 monclient: _check_auth_tickets 2024-04-26T00:43:22.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefad1e3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:22.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefad1e3640 1 -- 172.21.15.37:0/4198234921 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef980a9070 con 0x7fefa8133b80 2024-04-26T00:43:22.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefb0449640 5 monclient: authenticate success, global_id 284160 2024-04-26T00:43:22.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefb0449640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:43:22.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefad1e3640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:22.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefb0449640 10 monclient: get_monmap_and_config success 2024-04-26T00:43:22.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefad1e3640 1 -- 172.21.15.37:0/4198234921 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef980ad880 con 0x7fefa8133b80 2024-04-26T00:43:22.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefb0449640 4 set_mon_vals no callback set 2024-04-26T00:43:22.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefad1e3640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:22.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefad1e3640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:43:22.365 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefad1e3640 10 monclient: dump: 2024-04-26T00:43:22.365 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:22.365 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:22.365 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:22.366 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:22.366 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:22.366 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:22.366 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:22.366 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:22.366 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:22.366 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:22.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefb0449640 10 monclient: shutdown 2024-04-26T00:43:22.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefb0449640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:22.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefb0449640 1 -- 172.21.15.37:0/4198234921 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fefa8133b80 msgr2=0x7fefa80a1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:22.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefb0449640 1 --2- 172.21.15.37:0/4198234921 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fefa8133b80 0x7fefa80a1520 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fef98009870 tx=0x7fef9801c380 comp rx=0 tx=0).stop 2024-04-26T00:43:22.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefb0449640 1 -- 172.21.15.37:0/4198234921 shutdown_connections 2024-04-26T00:43:22.366 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefb0449640 1 --2- 172.21.15.37:0/4198234921 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fefa8133b80 0x7fefa80a1520 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:22.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefb0449640 1 --2- 172.21.15.37:0/4198234921 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fefa8072500 0x7fefa80728e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:22.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefb0449640 1 -- 172.21.15.37:0/4198234921 >> 172.21.15.37:0/4198234921 conn(0x7fefa80a6200 msgr2=0x7fefa80a2610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:22.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefb0449640 1 -- 172.21.15.37:0/4198234921 shutdown_connections 2024-04-26T00:43:22.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefb0449640 1 -- 172.21.15.37:0/4198234921 wait complete. 2024-04-26T00:43:22.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefb0449640 10 monclient: build_initial_monmap 2024-04-26T00:43:22.367 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefb0449640 10 monclient: monmap: 2024-04-26T00:43:22.367 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:22.367 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:43:22.367 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:43:22.367 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:43:22.367 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:22.367 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:43:22.367 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:43:22.367 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:43:22.367 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:43:22.368 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:22.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefb0449640 1 Processor -- start 2024-04-26T00:43:22.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefb0449640 1 -- start start 2024-04-26T00:43:22.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefb0449640 10 monclient: init 2024-04-26T00:43:22.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefb0449640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:22.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefb0449640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:43:22.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefb0449640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fefa8072500 0x7fefa813e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:22.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefb0449640 10 monclient(hunting): picked mon.noname-a con 0x7fefa8072500 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:43:22.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefb0449640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fefa8133b80 0x7fefa81307f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:22.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefb0449640 10 monclient(hunting): picked mon.noname-c con 0x7fefa8133b80 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:22.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefb0449640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefa8133fe0 0x7fefa8130d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:22.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefb0449640 10 monclient(hunting): picked mon.noname-b con 0x7fefa8133fe0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:43:22.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefb0449640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:22.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefb0449640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fefa8141810 con 0x7fefa8072500 2024-04-26T00:43:22.368 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefb0449640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:22.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefb0449640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fefa80a2380 con 0x7fefa8133fe0 2024-04-26T00:43:22.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefb0449640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:22.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefb0449640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fefa8141990 con 0x7fefa8133b80 2024-04-26T00:43:22.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefb0449640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:22.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefb0449640 10 monclient(hunting): authenticate will time out at 2383.666908s 2024-04-26T00:43:22.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefad9e4640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fefa8133b80 0x7fefa81307f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:22.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefae1e5640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fefa8072500 0x7fefa813e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:22.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefad9e4640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fefa8133b80 0x7fefa81307f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51348/0 (socket says 172.21.15.37:51348) 2024-04-26T00:43:22.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefad9e4640 1 -- 172.21.15.37:0/2384186994 learned_addr learned my addr 172.21.15.37:0/2384186994 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:22.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefad9e4640 10 monclient(hunting): get_auth_request con 0x7fefa8133b80 auth_method 0 2024-04-26T00:43:22.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefad9e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:22.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefad9e4640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:22.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefad9e4640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:22.369 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefae1e5640 10 monclient(hunting): get_auth_request con 0x7fefa8072500 auth_method 0 2024-04-26T00:43:22.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefae1e5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:22.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefae1e5640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:22.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefae1e5640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:22.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefad9e4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:22.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefad9e4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:22.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefad9e4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:22.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefae1e5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:22.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefae1e5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:22.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefae1e5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:22.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefae9e6640 1 --2- 172.21.15.37:0/2384186994 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefa8133fe0 0x7fefa8130d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:22.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefae1e5640 10 monclient(hunting): handle_auth_done global_id 284163 payload 824 2024-04-26T00:43:22.370 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefae1e5640 1 -- 172.21.15.37:0/2384186994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fefa8133b80 msgr2=0x7fefa81307f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:22.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefae1e5640 1 --2- 172.21.15.37:0/2384186994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fefa8133b80 0x7fefa81307f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:22.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefae1e5640 1 -- 172.21.15.37:0/2384186994 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefa8133fe0 msgr2=0x7fefa8130d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:22.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefae1e5640 1 --2- 172.21.15.37:0/2384186994 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefa8133fe0 0x7fefa8130d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:22.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefae1e5640 10 monclient: _finish_hunting 0 2024-04-26T00:43:22.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefae1e5640 1 monclient: found mon.noname-a 2024-04-26T00:43:22.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefae1e5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:22.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefae1e5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:43:22.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefae1e5640 1 -- 172.21.15.37:0/2384186994 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefa8133890 con 0x7fefa8072500 2024-04-26T00:43:22.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefae9e6640 10 monclient: get_auth_request con 0x7fefa8133fe0 auth_method 0 2024-04-26T00:43:22.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefae9e6640 1 --2- 172.21.15.37:0/2384186994 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefa8133fe0 0x7fefa8130d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:43:22.371 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.359+0000 7fefad9e4640 1 --2- 172.21.15.37:0/2384186994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fefa8133b80 0x7fefa81307f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:43:22.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.363+0000 7fefae1e5640 1 -- 172.21.15.37:0/2384186994 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fefa8072500 msgr2=0x7fefa813e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:22.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.363+0000 7fefae1e5640 1 --2- 172.21.15.37:0/2384186994 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fefa8072500 0x7fefa813e830 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fefa400e8e0 tx=0x7fefa400eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:22.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.363+0000 7fef9effd640 1 -- 172.21.15.37:0/2384186994 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa4017070 con 0x7fefa8072500 2024-04-26T00:43:22.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.363+0000 7fef9effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:22.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.363+0000 7fef9effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:43:22.374 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.363+0000 7fef9effd640 10 monclient: dump: 2024-04-26T00:43:22.374 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:22.375 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:22.375 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:22.375 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:22.375 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:22.375 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:22.375 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:22.375 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:22.375 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:22.375 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:22.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.363+0000 7fef9effd640 10 monclient: _finish_auth 0 2024-04-26T00:43:22.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.363+0000 7fef9effd640 10 monclient: _check_auth_tickets 2024-04-26T00:43:22.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.363+0000 7fef9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:22.375 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.363+0000 7fef9effd640 1 -- 172.21.15.37:0/2384186994 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefa400caf0 con 0x7fefa8072500 2024-04-26T00:43:22.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.363+0000 7fefb0449640 5 monclient: authenticate success, global_id 284163 2024-04-26T00:43:22.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.363+0000 7fefb0449640 10 monclient: _renew_subs 2024-04-26T00:43:22.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.363+0000 7fefb0449640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:43:22.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.363+0000 7fefb0449640 1 -- 172.21.15.37:0/2384186994 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefa813e120 con 0x7fefa8072500 2024-04-26T00:43:22.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.363+0000 7fef9effd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:22.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.363+0000 7fef9effd640 1 -- 172.21.15.37:0/2384186994 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa4013470 con 0x7fefa8072500 2024-04-26T00:43:22.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.363+0000 7fef9effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:22.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.363+0000 7fef9effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:43:22.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.363+0000 7fef9effd640 10 monclient: dump: 2024-04-26T00:43:22.376 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:22.376 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:22.376 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:22.376 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:22.376 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:22.376 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:22.377 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:22.377 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:22.377 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:22.377 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:22.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.363+0000 7fefb0449640 10 monclient: _renew_subs 2024-04-26T00:43:22.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.363+0000 7fefb0449640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:43:22.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.363+0000 7fefb0449640 1 -- 172.21.15.37:0/2384186994 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefa813de40 con 0x7fefa8072500 2024-04-26T00:43:22.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.363+0000 7fef9ffff640 4 set_mon_vals no callback set 2024-04-26T00:43:22.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.363+0000 7fefb0449640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:22.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.363+0000 7fefb0449640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:22.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.363+0000 7fefb0449640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:43:22.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.363+0000 7fefb0449640 1 -- 172.21.15.37:0/2384186994 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefa804f510 con 0x7fefa8072500 2024-04-26T00:43:22.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.363+0000 7fef9effd640 1 -- 172.21.15.37:0/2384186994 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fefa401a070 con 0x7fefa8072500 2024-04-26T00:43:22.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.363+0000 7fef9effd640 1 --2- 172.21.15.37:0/2384186994 >> v2:172.21.15.45:6812/2379918623 conn(0x7fef8c035ed0 0x7fef8c038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:22.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.363+0000 7fef9effd640 1 -- 172.21.15.37:0/2384186994 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(899..899 src has 822..899) ==== 3550+0+0 (secure 0 0 0) 0x7fefa40488e0 con 0x7fefa8072500 2024-04-26T00:43:22.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.363+0000 7fefad9e4640 1 --2- 172.21.15.37:0/2384186994 >> v2:172.21.15.45:6812/2379918623 conn(0x7fef8c035ed0 0x7fef8c038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:22.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.363+0000 7fefad9e4640 10 monclient: get_auth_request con 0x7fef8c035ed0 auth_method 0 2024-04-26T00:43:22.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.367+0000 7fefad9e4640 1 --2- 172.21.15.37:0/2384186994 >> v2:172.21.15.45:6812/2379918623 conn(0x7fef8c035ed0 0x7fef8c038390 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fef9801c3c0 tx=0x7fef98008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:22.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.367+0000 7fef9effd640 1 -- 172.21.15.37:0/2384186994 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fefa4013610 con 0x7fefa8072500 2024-04-26T00:43:22.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.367+0000 7fef9effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:22.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.367+0000 7fef9effd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:43:22.527 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.527+0000 7fefb0449640 1 -- 172.21.15.37:0/2384186994 --> v2:172.21.15.45:6812/2379918623 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7fefa813da10 con 0x7fef8c035ed0 2024-04-26T00:43:22.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.527+0000 7fef9effd640 1 -- 172.21.15.37:0/2384186994 <== mgr.4107 v2:172.21.15.45:6812/2379918623 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fef980b3040 con 0x7fef8c035ed0 2024-04-26T00:43:22.528 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 1 to deep-scrub 2024-04-26T00:43:22.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.527+0000 7fefb0449640 1 -- 172.21.15.37:0/2384186994 >> v2:172.21.15.45:6812/2379918623 conn(0x7fef8c035ed0 msgr2=0x7fef8c038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:22.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.527+0000 7fefb0449640 1 --2- 172.21.15.37:0/2384186994 >> v2:172.21.15.45:6812/2379918623 conn(0x7fef8c035ed0 0x7fef8c038390 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fef9801c3c0 tx=0x7fef98008040 comp rx=0 tx=0).stop 2024-04-26T00:43:22.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.527+0000 7fefb0449640 10 monclient: shutdown 2024-04-26T00:43:22.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.527+0000 7fefb0449640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:22.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.527+0000 7fefb0449640 1 -- 172.21.15.37:0/2384186994 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fefa8072500 msgr2=0x7fefa813e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:22.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.527+0000 7fefb0449640 1 --2- 172.21.15.37:0/2384186994 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fefa8072500 0x7fefa813e830 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fefa400e8e0 tx=0x7fefa400eda0 comp rx=0 tx=0).stop 2024-04-26T00:43:22.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.527+0000 7fefb0449640 1 -- 172.21.15.37:0/2384186994 shutdown_connections 2024-04-26T00:43:22.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.527+0000 7fefb0449640 1 --2- 172.21.15.37:0/2384186994 >> v2:172.21.15.45:6812/2379918623 conn(0x7fef8c035ed0 0x7fef8c038390 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:22.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.527+0000 7fefb0449640 1 --2- 172.21.15.37:0/2384186994 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefa8133fe0 0x7fefa8130d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:22.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.527+0000 7fefb0449640 1 --2- 172.21.15.37:0/2384186994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fefa8133b80 0x7fefa81307f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:22.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.527+0000 7fefb0449640 1 --2- 172.21.15.37:0/2384186994 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fefa8072500 0x7fefa813e830 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:22.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.527+0000 7fefb0449640 1 -- 172.21.15.37:0/2384186994 >> 172.21.15.37:0/2384186994 conn(0x7fefa80a6200 msgr2=0x7fefa80712b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:22.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.527+0000 7fefb0449640 1 -- 172.21.15.37:0/2384186994 shutdown_connections 2024-04-26T00:43:22.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.527+0000 7fefb0449640 1 -- 172.21.15.37:0/2384186994 wait complete. 2024-04-26T00:43:22.545 INFO:tasks.ceph:Scrubbing osd.2 2024-04-26T00:43:22.545 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-26T00:43:22.659 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7fc3a6640 10 monclient: get_monmap_and_config 2024-04-26T00:43:22.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7fc3a6640 10 monclient: build_initial_monmap 2024-04-26T00:43:22.660 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7fc3a6640 10 monclient: monmap: 2024-04-26T00:43:22.660 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:22.660 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:22.660 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:43:22.660707+0000 2024-04-26T00:43:22.660 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:43:22.660707+0000 2024-04-26T00:43:22.660 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:22.660 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:22.660 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:43:22.660 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:43:22.660 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:43:22.660 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:43:22.660 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:43:22.660 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:22.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7fc3a6640 1 Processor -- start 2024-04-26T00:43:22.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7fc3a6640 1 -- start start 2024-04-26T00:43:22.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7fc3a6640 10 monclient: init 2024-04-26T00:43:22.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7fc3a6640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:22.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7fc3a6640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-04-26T00:43:22.663 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7fc3a6640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc7f4075590 0x7fc7f4072320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:22.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7fc3a6640 10 monclient(hunting): picked mon.noname-a con 0x7fc7f4075590 addr v2:172.21.15.37:3300/0 2024-04-26T00:43:22.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7fc3a6640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fc7f4072860 0x7fc7f40a1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:22.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7fc3a6640 10 monclient(hunting): picked mon.noname-c con 0x7fc7f4072860 addr v2:172.21.15.45:3300/0 2024-04-26T00:43:22.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7fc3a6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc7f40a1ad0 addr v1:172.21.15.45:6789/0 2024-04-26T00:43:22.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7fc3a6640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7f406e880 con 0x7fc7f40a1ad0 2024-04-26T00:43:22.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7fc3a6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:22.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7fc3a6640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fc7f4075970 con 0x7fc7f4075590 2024-04-26T00:43:22.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7fc3a6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:22.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7fc3a6640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fc7f406edd0 con 0x7fc7f4072860 2024-04-26T00:43:22.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7fc3a6640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:22.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7fc3a6640 10 monclient(hunting): authenticate will time out at 2383.966977s 2024-04-26T00:43:22.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7fa142640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc7f4075590 0x7fc7f4072320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:22.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7fa142640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc7f4075590 0x7fc7f4072320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58206/0 (socket says 172.21.15.37:58206) 2024-04-26T00:43:22.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7fa142640 1 -- 172.21.15.37:0/3983499975 learned_addr learned my addr 172.21.15.37:0/3983499975 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:22.664 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7fa142640 10 monclient(hunting): get_auth_request con 0x7fc7f4075590 auth_method 0 2024-04-26T00:43:22.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7fa142640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:22.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7fa142640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:22.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7fa142640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:22.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7fa142640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:22.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7fa142640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:22.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7fa142640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:22.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7f9941640 1 --2- 172.21.15.37:0/3983499975 >> v2:172.21.15.45:3300/0 conn(0x7fc7f4072860 0x7fc7f40a1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:22.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7f9941640 10 monclient(hunting): get_auth_request con 0x7fc7f4072860 auth_method 0 2024-04-26T00:43:22.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7f9941640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:22.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7f9941640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:22.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7f9941640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:22.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7fa142640 10 monclient(hunting): handle_auth_done global_id 284166 payload 274 2024-04-26T00:43:22.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7fa142640 1 -- 172.21.15.37:0/3983499975 >> v2:172.21.15.45:3300/0 conn(0x7fc7f4072860 msgr2=0x7fc7f40a1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:22.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7fa142640 1 --2- 172.21.15.37:0/3983499975 >> v2:172.21.15.45:3300/0 conn(0x7fc7f4072860 0x7fc7f40a1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:22.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7fa142640 1 -- 172.21.15.37:0/3983499975 >> v1:172.21.15.45:6789/0 conn(0x7fc7f40a1ad0 legacy=0x7fc7f413e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:22.665 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7fa142640 10 monclient: _finish_hunting 0 2024-04-26T00:43:22.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7fa142640 1 monclient: found mon.noname-a 2024-04-26T00:43:22.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7fa142640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:22.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7fa142640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:43:22.666 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.659+0000 7fc7fa142640 1 -- 172.21.15.37:0/3983499975 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7f4137710 con 0x7fc7f4075590 2024-04-26T00:43:22.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fa142640 1 -- 172.21.15.37:0/3983499975 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc7f4075590 msgr2=0x7fc7f4072320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:22.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fa142640 1 --2- 172.21.15.37:0/3983499975 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc7f4075590 0x7fc7f4072320 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc7f41315b0 tx=0x7fc7f001c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9a15be13aabbf49 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:22.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7f9140640 1 -- 172.21.15.37:0/3983499975 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f00b0070 con 0x7fc7f4075590 2024-04-26T00:43:22.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7f9140640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:22.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7f9140640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:43:22.667 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7f9140640 10 monclient: dump: 2024-04-26T00:43:22.667 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:22.667 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:22.667 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:22.667 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:22.668 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:22.668 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:22.668 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:22.668 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:22.668 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:22.668 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:22.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7f9140640 10 monclient: _finish_auth 0 2024-04-26T00:43:22.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7f9140640 10 monclient: _check_auth_tickets 2024-04-26T00:43:22.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7f9140640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:22.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7f9140640 1 -- 172.21.15.37:0/3983499975 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7f00ab600 con 0x7fc7f4075590 2024-04-26T00:43:22.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fc3a6640 5 monclient: authenticate success, global_id 284166 2024-04-26T00:43:22.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fc3a6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:43:22.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7f9140640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:22.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fc3a6640 10 monclient: get_monmap_and_config success 2024-04-26T00:43:22.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fc3a6640 4 set_mon_vals no callback set 2024-04-26T00:43:22.668 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7f9140640 1 -- 172.21.15.37:0/3983499975 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f00ab9d0 con 0x7fc7f4075590 2024-04-26T00:43:22.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7f9140640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:22.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7f9140640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:43:22.669 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7f9140640 10 monclient: dump: 2024-04-26T00:43:22.669 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:22.669 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:22.669 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:22.669 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:22.669 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:22.669 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:22.669 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:22.669 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:22.669 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:22.669 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:22.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fc3a6640 10 monclient: shutdown 2024-04-26T00:43:22.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fc3a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:22.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fc3a6640 1 -- 172.21.15.37:0/3983499975 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc7f4075590 msgr2=0x7fc7f4072320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:22.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fc3a6640 1 --2- 172.21.15.37:0/3983499975 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc7f4075590 0x7fc7f4072320 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc7f41315b0 tx=0x7fc7f001c3a0 comp rx=0 tx=0).stop 2024-04-26T00:43:22.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fc3a6640 1 -- 172.21.15.37:0/3983499975 shutdown_connections 2024-04-26T00:43:22.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fc3a6640 1 --2- 172.21.15.37:0/3983499975 >> v2:172.21.15.45:3300/0 conn(0x7fc7f4072860 0x7fc7f40a1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:22.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fc3a6640 1 --2- 172.21.15.37:0/3983499975 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc7f4075590 0x7fc7f4072320 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:22.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fc3a6640 1 -- 172.21.15.37:0/3983499975 >> 172.21.15.37:0/3983499975 conn(0x7fc7f40a6180 msgr2=0x7fc7f40a2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:22.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fc3a6640 1 -- 172.21.15.37:0/3983499975 shutdown_connections 2024-04-26T00:43:22.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fc3a6640 1 -- 172.21.15.37:0/3983499975 wait complete. 2024-04-26T00:43:22.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fc3a6640 10 monclient: build_initial_monmap 2024-04-26T00:43:22.674 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fc3a6640 10 monclient: monmap: 2024-04-26T00:43:22.674 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:22.675 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:43:22.675 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:43:22.675 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:43:22.675 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:22.675 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:43:22.675 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:43:22.675 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:43:22.675 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:43:22.675 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:22.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fc3a6640 1 Processor -- start 2024-04-26T00:43:22.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fc3a6640 1 -- start start 2024-04-26T00:43:22.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fc3a6640 10 monclient: init 2024-04-26T00:43:22.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fc3a6640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:22.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fc3a6640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:43:22.675 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fc3a6640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc7f4072860 0x7fc7f41306d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:22.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fc3a6640 10 monclient(hunting): picked mon.noname-a con 0x7fc7f4072860 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:43:22.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fc3a6640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc7f4075590 0x7fc7f4130c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:22.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fc3a6640 10 monclient(hunting): picked mon.noname-c con 0x7fc7f4075590 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:22.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fc3a6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7f40a1ad0 0x7fc7f41cf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:22.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fc3a6640 10 monclient(hunting): picked mon.noname-b con 0x7fc7f40a1ad0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:43:22.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fc3a6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:22.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fc3a6640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f409f470 con 0x7fc7f4072860 2024-04-26T00:43:22.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fc3a6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:22.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fc3a6640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f4072c40 con 0x7fc7f40a1ad0 2024-04-26T00:43:22.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fc3a6640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:22.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fc3a6640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc7f4135010 con 0x7fc7f4075590 2024-04-26T00:43:22.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fc3a6640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:22.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fc3a6640 10 monclient(hunting): authenticate will time out at 2383.974646s 2024-04-26T00:43:22.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fa142640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc7f4072860 0x7fc7f41306d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:22.676 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fa142640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc7f4072860 0x7fc7f41306d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58216/0 (socket says 172.21.15.37:58216) 2024-04-26T00:43:22.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fa142640 1 -- 172.21.15.37:0/3832280246 learned_addr learned my addr 172.21.15.37:0/3832280246 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:22.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fa142640 10 monclient(hunting): get_auth_request con 0x7fc7f4072860 auth_method 0 2024-04-26T00:43:22.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fa142640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:22.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fa142640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:22.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fa142640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:22.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7f9941640 1 --2- 172.21.15.37:0/3832280246 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc7f4075590 0x7fc7f4130c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:22.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fa142640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:22.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fa142640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:22.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fa142640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:22.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7f9941640 10 monclient(hunting): get_auth_request con 0x7fc7f4075590 auth_method 0 2024-04-26T00:43:22.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7f9941640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:22.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7f9941640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:22.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7f9941640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:22.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7f9941640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:22.677 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7f9941640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:22.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7f9941640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:22.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fa943640 1 --2- 172.21.15.37:0/3832280246 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7f40a1ad0 0x7fc7f41cf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:22.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fa142640 10 monclient(hunting): handle_auth_done global_id 284169 payload 824 2024-04-26T00:43:22.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fa142640 1 -- 172.21.15.37:0/3832280246 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc7f4075590 msgr2=0x7fc7f4130c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:22.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fa142640 1 --2- 172.21.15.37:0/3832280246 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc7f4075590 0x7fc7f4130c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:22.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fa142640 1 -- 172.21.15.37:0/3832280246 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7f40a1ad0 msgr2=0x7fc7f41cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:22.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fa142640 1 --2- 172.21.15.37:0/3832280246 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7f40a1ad0 0x7fc7f41cf1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:22.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fa142640 10 monclient: _finish_hunting 0 2024-04-26T00:43:22.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fa142640 1 monclient: found mon.noname-a 2024-04-26T00:43:22.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fa142640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:22.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fa142640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:43:22.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fa142640 1 -- 172.21.15.37:0/3832280246 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7f4131150 con 0x7fc7f4072860 2024-04-26T00:43:22.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fa943640 10 monclient: get_auth_request con 0x7fc7f40a1ad0 auth_method 0 2024-04-26T00:43:22.678 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.667+0000 7fc7fa943640 1 --2- 172.21.15.37:0/3832280246 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7f40a1ad0 0x7fc7f41cf1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:43:22.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.671+0000 7fc7fa142640 1 -- 172.21.15.37:0/3832280246 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc7f4072860 msgr2=0x7fc7f41306d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:22.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.671+0000 7fc7fa142640 1 --2- 172.21.15.37:0/3832280246 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc7f4072860 0x7fc7f41306d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc7f00ae040 tx=0x7fc7f0019140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:22.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.671+0000 7fc7eaffd640 1 -- 172.21.15.37:0/3832280246 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f00b0040 con 0x7fc7f4072860 2024-04-26T00:43:22.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.671+0000 7fc7eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:22.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.671+0000 7fc7eaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:43:22.679 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.671+0000 7fc7eaffd640 10 monclient: dump: 2024-04-26T00:43:22.679 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:22.679 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:22.679 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:22.679 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:22.679 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:22.679 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:22.679 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:22.679 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:22.679 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:22.680 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:22.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.671+0000 7fc7eaffd640 10 monclient: _finish_auth 0 2024-04-26T00:43:22.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.671+0000 7fc7eaffd640 10 monclient: _check_auth_tickets 2024-04-26T00:43:22.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.671+0000 7fc7eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:22.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.671+0000 7fc7eaffd640 1 -- 172.21.15.37:0/3832280246 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7f00b9070 con 0x7fc7f4072860 2024-04-26T00:43:22.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.671+0000 7fc7fc3a6640 5 monclient: authenticate success, global_id 284169 2024-04-26T00:43:22.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.671+0000 7fc7fc3a6640 10 monclient: _renew_subs 2024-04-26T00:43:22.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.671+0000 7fc7fc3a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:43:22.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.671+0000 7fc7fc3a6640 1 -- 172.21.15.37:0/3832280246 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7f409fca0 con 0x7fc7f4072860 2024-04-26T00:43:22.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.671+0000 7fc7eaffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:22.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.671+0000 7fc7eaffd640 1 -- 172.21.15.37:0/3832280246 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f00b39f0 con 0x7fc7f4072860 2024-04-26T00:43:22.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.671+0000 7fc7eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:22.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.671+0000 7fc7eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:43:22.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.671+0000 7fc7eaffd640 10 monclient: dump: 2024-04-26T00:43:22.680 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:22.681 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:22.681 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:22.681 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:22.681 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:22.681 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:22.681 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:22.681 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:22.681 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:22.681 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:22.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.671+0000 7fc7fc3a6640 10 monclient: _renew_subs 2024-04-26T00:43:22.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.671+0000 7fc7fc3a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:43:22.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.671+0000 7fc7f893f640 4 set_mon_vals no callback set 2024-04-26T00:43:22.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.671+0000 7fc7fc3a6640 1 -- 172.21.15.37:0/3832280246 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7f413ddd0 con 0x7fc7f4072860 2024-04-26T00:43:22.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.671+0000 7fc7fc3a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:43:22.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.671+0000 7fc7fc3a6640 1 -- 172.21.15.37:0/3832280246 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc7f404f580 con 0x7fc7f4072860 2024-04-26T00:43:22.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.671+0000 7fc7eaffd640 1 -- 172.21.15.37:0/3832280246 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc7f00b3b90 con 0x7fc7f4072860 2024-04-26T00:43:22.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.671+0000 7fc7eaffd640 1 --2- 172.21.15.37:0/3832280246 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc7d8035eb0 0x7fc7d8038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:22.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.671+0000 7fc7eaffd640 1 -- 172.21.15.37:0/3832280246 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(899..899 src has 822..899) ==== 3550+0+0 (secure 0 0 0) 0x7fc7f00e2730 con 0x7fc7f4072860 2024-04-26T00:43:22.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.671+0000 7fc7eaffd640 1 --2- 172.21.15.37:0/3832280246 >> v2:172.21.15.37:6800/603175474 conn(0x7fc7d803b9f0 0x7fc7d803de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:22.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.671+0000 7fc7eaffd640 1 -- 172.21.15.37:0/3832280246 --> v2:172.21.15.37:6800/603175474 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc7f00b7e80 con 0x7fc7d803b9f0 2024-04-26T00:43:22.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.671+0000 7fc7eaffd640 1 -- 172.21.15.37:0/3832280246 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_get_version_reply(handle=1 version=899) ==== 24+0+0 (secure 0 0 0) 0x7fc7f00e9050 con 0x7fc7f4072860 2024-04-26T00:43:22.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.671+0000 7fc7eaffd640 10 monclient: handle_get_version_reply finishing 1 version 899 2024-04-26T00:43:22.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.671+0000 7fc7fa943640 1 --2- 172.21.15.37:0/3832280246 >> v2:172.21.15.37:6800/603175474 conn(0x7fc7d803b9f0 0x7fc7d803de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:22.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.671+0000 7fc7fa943640 10 monclient: get_auth_request con 0x7fc7d803b9f0 auth_method 0 2024-04-26T00:43:22.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.671+0000 7fc7f9941640 1 --2- 172.21.15.37:0/3832280246 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc7d8035eb0 0x7fc7d8038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:22.682 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.671+0000 7fc7f9941640 10 monclient: get_auth_request con 0x7fc7d8035eb0 auth_method 0 2024-04-26T00:43:22.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.675+0000 7fc7f9941640 1 --2- 172.21.15.37:0/3832280246 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc7d8035eb0 0x7fc7d8038370 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fc7f40a4f90 tx=0x7fc7e4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:22.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.675+0000 7fc7fa943640 1 --2- 172.21.15.37:0/3832280246 >> v2:172.21.15.37:6800/603175474 conn(0x7fc7d803b9f0 0x7fc7d803de10 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:22.683 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.675+0000 7fc7eaffd640 1 -- 172.21.15.37:0/3832280246 <== osd.2 v2:172.21.15.37:6800/603175474 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc7f00b7e80 con 0x7fc7d803b9f0 2024-04-26T00:43:22.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.691+0000 7fc7fc3a6640 1 -- 172.21.15.37:0/3832280246 --> v2:172.21.15.37:6800/603175474 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fc7f409f5f0 con 0x7fc7d803b9f0 2024-04-26T00:43:22.691 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.691+0000 7fc7eaffd640 1 -- 172.21.15.37:0/3832280246 <== osd.2 v2:172.21.15.37:6800/603175474 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fc7f409f5f0 con 0x7fc7d803b9f0 2024-04-26T00:43:22.691 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:43:22.692 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T00:43:22.692 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:43:22.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.691+0000 7fc7fc3a6640 1 -- 172.21.15.37:0/3832280246 >> v2:172.21.15.37:6800/603175474 conn(0x7fc7d803b9f0 msgr2=0x7fc7d803de10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:22.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.691+0000 7fc7fc3a6640 1 --2- 172.21.15.37:0/3832280246 >> v2:172.21.15.37:6800/603175474 conn(0x7fc7d803b9f0 0x7fc7d803de10 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:22.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.691+0000 7fc7fc3a6640 1 -- 172.21.15.37:0/3832280246 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc7d8035eb0 msgr2=0x7fc7d8038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:22.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.691+0000 7fc7fc3a6640 1 --2- 172.21.15.37:0/3832280246 >> v2:172.21.15.45:6812/2379918623 conn(0x7fc7d8035eb0 0x7fc7d8038370 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fc7f40a4f90 tx=0x7fc7e4008040 comp rx=0 tx=0).stop 2024-04-26T00:43:22.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.691+0000 7fc7fc3a6640 10 monclient: shutdown 2024-04-26T00:43:22.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.691+0000 7fc7fc3a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:22.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.691+0000 7fc7fc3a6640 1 -- 172.21.15.37:0/3832280246 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc7f4072860 msgr2=0x7fc7f41306d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:22.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.691+0000 7fc7fc3a6640 1 --2- 172.21.15.37:0/3832280246 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc7f4072860 0x7fc7f41306d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc7f00ae040 tx=0x7fc7f0019140 comp rx=0 tx=0).stop 2024-04-26T00:43:22.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.691+0000 7fc7fa943640 1 -- 172.21.15.37:0/3832280246 reap_dead start 2024-04-26T00:43:22.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.691+0000 7fc7fc3a6640 1 -- 172.21.15.37:0/3832280246 shutdown_connections 2024-04-26T00:43:22.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.691+0000 7fc7fc3a6640 1 -- 172.21.15.37:0/3832280246 >> 172.21.15.37:0/3832280246 conn(0x7fc7f40a6180 msgr2=0x7fc7f40a3980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:22.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.691+0000 7fc7fc3a6640 1 -- 172.21.15.37:0/3832280246 shutdown_connections 2024-04-26T00:43:22.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.691+0000 7fc7fc3a6640 1 -- 172.21.15.37:0/3832280246 wait complete. 2024-04-26T00:43:22.706 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-26T00:43:22.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.811+0000 7f5d2310f640 10 monclient: get_monmap_and_config 2024-04-26T00:43:22.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.811+0000 7f5d2310f640 10 monclient: build_initial_monmap 2024-04-26T00:43:22.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.815+0000 7f5d2310f640 10 monclient: monmap: 2024-04-26T00:43:22.815 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:22.815 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:22.815 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:43:22.815912+0000 2024-04-26T00:43:22.815 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:43:22.815912+0000 2024-04-26T00:43:22.815 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:22.815 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:22.815 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:43:22.815 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:43:22.815 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:43:22.815 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:43:22.815 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:43:22.816 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:22.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.815+0000 7f5d2310f640 1 Processor -- start 2024-04-26T00:43:22.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.815+0000 7f5d2310f640 1 -- start start 2024-04-26T00:43:22.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.815+0000 7f5d2310f640 10 monclient: init 2024-04-26T00:43:22.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.815+0000 7f5d2310f640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:22.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.815+0000 7f5d2310f640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-04-26T00:43:22.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.815+0000 7f5d2310f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d1c0a2fe0 0x7f5d1c1351b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:22.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.815+0000 7f5d2310f640 10 monclient(hunting): picked mon.noname-b con 0x7f5d1c0a2fe0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:22.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.815+0000 7f5d2310f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5d1c1356f0 addr v1:172.21.15.45:6789/0 2024-04-26T00:43:22.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.815+0000 7f5d2310f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5d1c0a1ae0 addr v1:172.21.15.37:6789/0 2024-04-26T00:43:22.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.815+0000 7f5d2310f640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d1c072ec0 con 0x7f5d1c0a1ae0 2024-04-26T00:43:22.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.815+0000 7f5d2310f640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d1c134ac0 con 0x7f5d1c1356f0 2024-04-26T00:43:22.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.815+0000 7f5d2310f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:22.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.815+0000 7f5d2310f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5d1c0a33c0 con 0x7f5d1c0a2fe0 2024-04-26T00:43:22.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.815+0000 7f5d2310f640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:22.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.815+0000 7f5d2310f640 10 monclient(hunting): authenticate will time out at 2384.121701s 2024-04-26T00:43:22.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.815+0000 7f5d20eab640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d1c0a2fe0 0x7f5d1c1351b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:22.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.815+0000 7f5d216ac640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f5d1c0a1ae0 0x7f5d1c13ce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:49540/0 (socket says 172.21.15.37:49540) 2024-04-26T00:43:22.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.815+0000 7f5d20eab640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d1c0a2fe0 0x7f5d1c1351b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51360/0 (socket says 172.21.15.37:51360) 2024-04-26T00:43:22.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.815+0000 7f5d216ac640 1 -- 172.21.15.37:0/3053603631 learned_addr learned my addr 172.21.15.37:0/3053603631 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:43:22.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.815+0000 7f5d20eab640 10 monclient(hunting): get_auth_request con 0x7f5d1c0a2fe0 auth_method 0 2024-04-26T00:43:22.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.815+0000 7f5d20eab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:22.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.815+0000 7f5d20eab640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:22.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.815+0000 7f5d20eab640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:22.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.815+0000 7f5d20eab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:22.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.815+0000 7f5d20eab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:22.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.815+0000 7f5d20eab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:22.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.815+0000 7f5d20eab640 10 monclient(hunting): handle_auth_done global_id 254498 payload 274 2024-04-26T00:43:22.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.815+0000 7f5d20eab640 1 -- 172.21.15.37:0/3053603631 >> v1:172.21.15.45:6789/0 conn(0x7f5d1c1356f0 legacy=0x7f5d1c0a13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:22.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.815+0000 7f5d20eab640 1 -- 172.21.15.37:0/3053603631 >> v1:172.21.15.37:6789/0 conn(0x7f5d1c0a1ae0 legacy=0x7f5d1c13ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:22.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.815+0000 7f5d20eab640 10 monclient: _finish_hunting 0 2024-04-26T00:43:22.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.815+0000 7f5d20eab640 1 monclient: found mon.noname-b 2024-04-26T00:43:22.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.815+0000 7f5d20eab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:22.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.815+0000 7f5d20eab640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:43:22.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.815+0000 7f5d20eab640 1 -- 172.21.15.37:0/3053603631 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d1c134d60 con 0x7f5d1c0a2fe0 2024-04-26T00:43:22.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d20eab640 1 -- 172.21.15.37:0/3053603631 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d1c0a2fe0 msgr2=0x7f5d1c1351b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:22.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d20eab640 1 --2- 172.21.15.37:0/3053603631 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d1c0a2fe0 0x7f5d1c1351b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c134c90 tx=0x7f5d0c01c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52d1b25a3ee33936 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:22.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d1b7fe640 1 -- 172.21.15.37:0/3053603631 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d0c0ac070 con 0x7f5d1c0a2fe0 2024-04-26T00:43:22.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:22.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d1b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:43:22.823 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d1b7fe640 10 monclient: dump: 2024-04-26T00:43:22.823 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:22.823 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:22.823 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:22.823 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:22.823 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:22.824 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:22.824 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:22.824 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:22.824 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:22.824 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:22.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d1b7fe640 10 monclient: _finish_auth 0 2024-04-26T00:43:22.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d1b7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:43:22.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:22.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d1b7fe640 1 -- 172.21.15.37:0/3053603631 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d0c0a6700 con 0x7f5d1c0a2fe0 2024-04-26T00:43:22.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d1b7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:22.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d1b7fe640 1 -- 172.21.15.37:0/3053603631 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d0c0a6ad0 con 0x7f5d1c0a2fe0 2024-04-26T00:43:22.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d2310f640 5 monclient: authenticate success, global_id 254498 2024-04-26T00:43:22.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d2310f640 10 monclient: get_monmap_and_config success 2024-04-26T00:43:22.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d2310f640 4 set_mon_vals no callback set 2024-04-26T00:43:22.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:22.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d1b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:43:22.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d1b7fe640 10 monclient: dump: 2024-04-26T00:43:22.825 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:22.825 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:22.825 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:22.825 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:22.825 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:22.825 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:22.825 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:22.825 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:22.825 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:22.825 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:22.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d2310f640 10 monclient: shutdown 2024-04-26T00:43:22.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d2310f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:22.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d2310f640 1 -- 172.21.15.37:0/3053603631 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d1c0a2fe0 msgr2=0x7f5d1c1351b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:22.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d2310f640 1 --2- 172.21.15.37:0/3053603631 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d1c0a2fe0 0x7f5d1c1351b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c134c90 tx=0x7f5d0c01c680 comp rx=0 tx=0).stop 2024-04-26T00:43:22.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d2310f640 1 -- 172.21.15.37:0/3053603631 shutdown_connections 2024-04-26T00:43:22.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d2310f640 1 --2- 172.21.15.37:0/3053603631 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d1c0a2fe0 0x7f5d1c1351b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:22.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d2310f640 1 -- 172.21.15.37:0/3053603631 >> 172.21.15.37:0/3053603631 conn(0x7f5d1c0a24c0 msgr2=0x7f5d1c0a28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:22.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d2310f640 1 -- 172.21.15.37:0/3053603631 shutdown_connections 2024-04-26T00:43:22.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d2310f640 1 -- 172.21.15.37:0/3053603631 wait complete. 2024-04-26T00:43:22.830 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d2310f640 10 monclient: build_initial_monmap 2024-04-26T00:43:22.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d2310f640 10 monclient: monmap: 2024-04-26T00:43:22.831 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:22.831 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:43:22.831 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:43:22.831 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:43:22.831 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:22.831 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:43:22.831 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:43:22.831 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:43:22.831 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:43:22.831 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:22.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d2310f640 1 Processor -- start 2024-04-26T00:43:22.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d2310f640 1 -- start start 2024-04-26T00:43:22.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d2310f640 10 monclient: init 2024-04-26T00:43:22.831 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d2310f640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:22.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d2310f640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:43:22.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d2310f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d1c0a1ae0 0x7f5d1c1cd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:22.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d2310f640 10 monclient(hunting): picked mon.noname-c con 0x7f5d1c0a1ae0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:22.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d2310f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d1c0a2fe0 0x7f5d1c13d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:22.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d2310f640 10 monclient(hunting): picked mon.noname-b con 0x7f5d1c0a2fe0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:43:22.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d2310f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5d1c1356f0 0x7f5d1c135bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:22.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d2310f640 10 monclient(hunting): picked mon.noname-a con 0x7f5d1c1356f0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:43:22.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d2310f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:22.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d2310f640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f5d1c140d40 con 0x7f5d1c1356f0 2024-04-26T00:43:22.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d2310f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:22.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d2310f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5d1c071f00 con 0x7f5d1c0a2fe0 2024-04-26T00:43:22.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d2310f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:22.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d2310f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5d1c140ec0 con 0x7f5d1c0a1ae0 2024-04-26T00:43:22.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d2310f640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:22.832 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d2310f640 10 monclient(hunting): authenticate will time out at 2384.130849s 2024-04-26T00:43:22.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d20eab640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d1c0a1ae0 0x7f5d1c1cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:22.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d216ac640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5d1c1356f0 0x7f5d1c135bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:22.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d20eab640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d1c0a1ae0 0x7f5d1c1cd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51374/0 (socket says 172.21.15.37:51374) 2024-04-26T00:43:22.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d20eab640 1 -- 172.21.15.37:0/361724507 learned_addr learned my addr 172.21.15.37:0/361724507 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:22.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d216ac640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5d1c1356f0 0x7f5d1c135bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58222/0 (socket says 172.21.15.37:58222) 2024-04-26T00:43:22.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d20eab640 10 monclient(hunting): get_auth_request con 0x7f5d1c0a1ae0 auth_method 0 2024-04-26T00:43:22.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d20eab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:22.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d20eab640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:22.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d20eab640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:22.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d216ac640 10 monclient(hunting): get_auth_request con 0x7f5d1c1356f0 auth_method 0 2024-04-26T00:43:22.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d216ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:22.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d216ac640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:22.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d216ac640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:22.833 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d20eab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:22.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d20eab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:22.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d20eab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:22.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d216ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:22.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d216ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:22.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d216ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:22.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d1bfff640 1 --2- 172.21.15.37:0/361724507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d1c0a2fe0 0x7f5d1c13d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:22.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d1bfff640 10 monclient(hunting): get_auth_request con 0x7f5d1c0a2fe0 auth_method 0 2024-04-26T00:43:22.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:22.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d1bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:22.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:22.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:22.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:22.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:22.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d216ac640 10 monclient(hunting): handle_auth_done global_id 284172 payload 824 2024-04-26T00:43:22.834 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d216ac640 1 -- 172.21.15.37:0/361724507 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d1c0a1ae0 msgr2=0x7f5d1c1cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:22.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d216ac640 1 --2- 172.21.15.37:0/361724507 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d1c0a1ae0 0x7f5d1c1cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:22.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d216ac640 1 -- 172.21.15.37:0/361724507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d1c0a2fe0 msgr2=0x7f5d1c13d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:22.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d216ac640 1 --2- 172.21.15.37:0/361724507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d1c0a2fe0 0x7f5d1c13d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:22.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d216ac640 10 monclient: _finish_hunting 0 2024-04-26T00:43:22.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d216ac640 1 monclient: found mon.noname-a 2024-04-26T00:43:22.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d216ac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:22.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d216ac640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:43:22.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d216ac640 1 -- 172.21.15.37:0/361724507 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d1c06ef50 con 0x7f5d1c1356f0 2024-04-26T00:43:22.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.823+0000 7f5d20eab640 1 --2- 172.21.15.37:0/361724507 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d1c0a1ae0 0x7f5d1c1cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:43:22.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.827+0000 7f5d216ac640 1 -- 172.21.15.37:0/361724507 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5d1c1356f0 msgr2=0x7f5d1c135bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:22.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.827+0000 7f5d216ac640 1 --2- 172.21.15.37:0/361724507 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5d1c1356f0 0x7f5d1c135bb0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5d1400b410 tx=0x7f5d1400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:22.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.827+0000 7f5d197fa640 1 -- 172.21.15.37:0/361724507 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d14015020 con 0x7f5d1c1356f0 2024-04-26T00:43:22.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.827+0000 7f5d197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:22.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.827+0000 7f5d197fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:43:22.835 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.827+0000 7f5d197fa640 10 monclient: dump: 2024-04-26T00:43:22.836 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:22.836 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:22.836 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:22.836 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:22.836 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:22.836 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:22.836 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:22.836 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:22.836 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:22.836 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:22.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.827+0000 7f5d197fa640 10 monclient: _finish_auth 0 2024-04-26T00:43:22.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.827+0000 7f5d197fa640 10 monclient: _check_auth_tickets 2024-04-26T00:43:22.836 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.827+0000 7f5d197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:22.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.827+0000 7f5d197fa640 1 -- 172.21.15.37:0/361724507 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d14011a40 con 0x7f5d1c1356f0 2024-04-26T00:43:22.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.827+0000 7f5d197fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:22.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.827+0000 7f5d197fa640 1 -- 172.21.15.37:0/361724507 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d14011e30 con 0x7f5d1c1356f0 2024-04-26T00:43:22.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.827+0000 7f5d197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:22.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.827+0000 7f5d197fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:43:22.837 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.827+0000 7f5d197fa640 10 monclient: dump: 2024-04-26T00:43:22.837 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:22.837 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:22.837 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:22.837 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:22.837 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:22.837 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:22.837 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:22.837 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:22.837 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:22.838 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:22.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.827+0000 7f5d1affd640 4 set_mon_vals no callback set 2024-04-26T00:43:22.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.827+0000 7f5d2310f640 5 monclient: authenticate success, global_id 284172 2024-04-26T00:43:22.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.827+0000 7f5d2310f640 10 monclient: _renew_subs 2024-04-26T00:43:22.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.827+0000 7f5d2310f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:43:22.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.827+0000 7f5d2310f640 1 -- 172.21.15.37:0/361724507 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d1c138010 con 0x7f5d1c1356f0 2024-04-26T00:43:22.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.827+0000 7f5d2310f640 10 monclient: _renew_subs 2024-04-26T00:43:22.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.827+0000 7f5d2310f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:43:22.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.827+0000 7f5d2310f640 1 -- 172.21.15.37:0/361724507 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d1c137730 con 0x7f5d1c1356f0 2024-04-26T00:43:22.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.827+0000 7f5d197fa640 1 -- 172.21.15.37:0/361724507 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f5d14018020 con 0x7f5d1c1356f0 2024-04-26T00:43:22.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.827+0000 7f5d2310f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:22.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.827+0000 7f5d2310f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:22.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.827+0000 7f5d2310f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:43:22.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.827+0000 7f5d2310f640 1 -- 172.21.15.37:0/361724507 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d1c04f510 con 0x7f5d1c1356f0 2024-04-26T00:43:22.838 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.827+0000 7f5d197fa640 1 --2- 172.21.15.37:0/361724507 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5d08035e80 0x7f5d08038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:22.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.827+0000 7f5d197fa640 1 -- 172.21.15.37:0/361724507 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(899..899 src has 822..899) ==== 3550+0+0 (secure 0 0 0) 0x7f5d140486a0 con 0x7f5d1c1356f0 2024-04-26T00:43:22.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.827+0000 7f5d20eab640 1 --2- 172.21.15.37:0/361724507 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5d08035e80 0x7f5d08038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:22.839 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.827+0000 7f5d20eab640 10 monclient: get_auth_request con 0x7f5d08035e80 auth_method 0 2024-04-26T00:43:22.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.831+0000 7f5d20eab640 1 --2- 172.21.15.37:0/361724507 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5d08035e80 0x7f5d08038340 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c000f90 tx=0x7f5d0c0a7000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:22.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.831+0000 7f5d197fa640 1 -- 172.21.15.37:0/361724507 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5d1401f020 con 0x7f5d1c1356f0 2024-04-26T00:43:22.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.831+0000 7f5d197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:22.840 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.831+0000 7f5d197fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:43:22.987 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.987+0000 7f5d2310f640 1 -- 172.21.15.37:0/361724507 --> v2:172.21.15.45:6812/2379918623 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f5d1c06c1e0 con 0x7f5d08035e80 2024-04-26T00:43:22.988 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.987+0000 7f5d197fa640 1 -- 172.21.15.37:0/361724507 <== mgr.4107 v2:172.21.15.45:6812/2379918623 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f5d1c06c1e0 con 0x7f5d08035e80 2024-04-26T00:43:22.988 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 2 to deep-scrub 2024-04-26T00:43:22.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.987+0000 7f5d2310f640 1 -- 172.21.15.37:0/361724507 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5d08035e80 msgr2=0x7f5d08038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:22.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.987+0000 7f5d2310f640 1 --2- 172.21.15.37:0/361724507 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5d08035e80 0x7f5d08038340 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c000f90 tx=0x7f5d0c0a7000 comp rx=0 tx=0).stop 2024-04-26T00:43:22.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.987+0000 7f5d2310f640 10 monclient: shutdown 2024-04-26T00:43:22.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.987+0000 7f5d2310f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:22.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.987+0000 7f5d2310f640 1 -- 172.21.15.37:0/361724507 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5d1c1356f0 msgr2=0x7f5d1c135bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:22.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.987+0000 7f5d2310f640 1 --2- 172.21.15.37:0/361724507 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5d1c1356f0 0x7f5d1c135bb0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5d1400b410 tx=0x7f5d1400b8d0 comp rx=0 tx=0).stop 2024-04-26T00:43:22.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.987+0000 7f5d2310f640 1 -- 172.21.15.37:0/361724507 shutdown_connections 2024-04-26T00:43:22.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.987+0000 7f5d2310f640 1 --2- 172.21.15.37:0/361724507 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5d08035e80 0x7f5d08038340 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:22.991 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.987+0000 7f5d2310f640 1 --2- 172.21.15.37:0/361724507 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5d1c1356f0 0x7f5d1c135bb0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:22.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.987+0000 7f5d2310f640 1 --2- 172.21.15.37:0/361724507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5d1c0a2fe0 0x7f5d1c13d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:22.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.987+0000 7f5d2310f640 1 --2- 172.21.15.37:0/361724507 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d1c0a1ae0 0x7f5d1c1cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:22.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.987+0000 7f5d2310f640 1 -- 172.21.15.37:0/361724507 >> 172.21.15.37:0/361724507 conn(0x7f5d1c0a24c0 msgr2=0x7f5d1c131880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:22.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.991+0000 7f5d2310f640 1 -- 172.21.15.37:0/361724507 shutdown_connections 2024-04-26T00:43:22.992 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:22.991+0000 7f5d2310f640 1 -- 172.21.15.37:0/361724507 wait complete. 2024-04-26T00:43:23.006 INFO:tasks.ceph:Scrubbing osd.3 2024-04-26T00:43:23.006 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-26T00:43:23.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b18c0c640 10 monclient: get_monmap_and_config 2024-04-26T00:43:23.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b18c0c640 10 monclient: build_initial_monmap 2024-04-26T00:43:23.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b18c0c640 10 monclient: monmap: 2024-04-26T00:43:23.131 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:23.131 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:23.131 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:43:23.131918+0000 2024-04-26T00:43:23.131 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:43:23.131918+0000 2024-04-26T00:43:23.131 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:23.132 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:23.132 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:43:23.132 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:43:23.132 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:43:23.132 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:43:23.132 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:43:23.132 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:23.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b18c0c640 1 Processor -- start 2024-04-26T00:43:23.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b18c0c640 1 -- start start 2024-04-26T00:43:23.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b18c0c640 10 monclient: init 2024-04-26T00:43:23.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b18c0c640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:23.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b18c0c640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-04-26T00:43:23.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b18c0c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f5b1409bae0 0x7f5b1409bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:23.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b18c0c640 10 monclient(hunting): picked mon.noname-a con 0x7f5b1409bae0 addr v2:172.21.15.37:3300/0 2024-04-26T00:43:23.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b18c0c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b14137150 0x7f5b14072270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:23.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b18c0c640 10 monclient(hunting): picked mon.noname-b con 0x7f5b14137150 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:23.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b18c0c640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5b14137550 0x7f5b140727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:23.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b18c0c640 10 monclient(hunting): picked mon.noname-c con 0x7f5b14137550 addr v2:172.21.15.45:3300/0 2024-04-26T00:43:23.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b18c0c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:23.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b18c0c640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f5b140a1c30 con 0x7f5b1409bae0 2024-04-26T00:43:23.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b18c0c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:23.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b18c0c640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f5b1409c400 con 0x7f5b14137550 2024-04-26T00:43:23.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b18c0c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:23.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b18c0c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5b140a14c0 con 0x7f5b14137150 2024-04-26T00:43:23.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b18c0c640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:23.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b18c0c640 10 monclient(hunting): authenticate will time out at 2384.437490s 2024-04-26T00:43:23.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b1259c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f5b1409bae0 0x7f5b1409bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:23.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b11d9b640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b14137150 0x7f5b14072270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:23.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b1259c640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f5b1409bae0 0x7f5b1409bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58230/0 (socket says 172.21.15.37:58230) 2024-04-26T00:43:23.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b1259c640 1 -- 172.21.15.37:0/2242105074 learned_addr learned my addr 172.21.15.37:0/2242105074 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:23.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b1259c640 10 monclient(hunting): get_auth_request con 0x7f5b1409bae0 auth_method 0 2024-04-26T00:43:23.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b1259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:23.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b1259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:23.136 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b1259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:23.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b11d9b640 10 monclient(hunting): get_auth_request con 0x7f5b14137150 auth_method 0 2024-04-26T00:43:23.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b11d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:23.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b11d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:23.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b11d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:23.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b1259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:23.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b1259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:23.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b1259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:23.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b11d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:23.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b11d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:23.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b11d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:23.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b12d9d640 1 --2- 172.21.15.37:0/2242105074 >> v2:172.21.15.45:3300/0 conn(0x7f5b14137550 0x7f5b140727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:23.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b12d9d640 10 monclient(hunting): get_auth_request con 0x7f5b14137550 auth_method 0 2024-04-26T00:43:23.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b12d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:23.137 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b12d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:23.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b12d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:23.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b1259c640 10 monclient(hunting): handle_auth_done global_id 284175 payload 274 2024-04-26T00:43:23.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b1259c640 1 -- 172.21.15.37:0/2242105074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b14137150 msgr2=0x7f5b14072270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:23.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b1259c640 1 --2- 172.21.15.37:0/2242105074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b14137150 0x7f5b14072270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:23.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b1259c640 1 -- 172.21.15.37:0/2242105074 >> v2:172.21.15.45:3300/0 conn(0x7f5b14137550 msgr2=0x7f5b140727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:23.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b1259c640 1 --2- 172.21.15.37:0/2242105074 >> v2:172.21.15.45:3300/0 conn(0x7f5b14137550 0x7f5b140727b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:23.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b1259c640 10 monclient: _finish_hunting 0 2024-04-26T00:43:23.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b1259c640 1 monclient: found mon.noname-a 2024-04-26T00:43:23.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b1259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:23.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b1259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:43:23.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b1259c640 1 -- 172.21.15.37:0/2242105074 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b14099d10 con 0x7f5b1409bae0 2024-04-26T00:43:23.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.131+0000 7f5b11d9b640 1 --2- 172.21.15.37:0/2242105074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b14137150 0x7f5b14072270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:43:23.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.135+0000 7f5b1259c640 1 -- 172.21.15.37:0/2242105074 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b1409bae0 msgr2=0x7f5b1409bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:23.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.135+0000 7f5b1259c640 1 --2- 172.21.15.37:0/2242105074 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b1409bae0 0x7f5b1409bec0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5b14099290 tx=0x7f5b0401c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a1317548e75bacb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:23.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.135+0000 7f5b1159a640 1 -- 172.21.15.37:0/2242105074 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b040b0070 con 0x7f5b1409bae0 2024-04-26T00:43:23.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.135+0000 7f5b1159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:23.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.135+0000 7f5b1159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:43:23.142 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.135+0000 7f5b1159a640 10 monclient: dump: 2024-04-26T00:43:23.143 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:23.143 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:23.143 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:23.143 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:23.143 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:23.143 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:23.143 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:23.143 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:23.143 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:23.143 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:23.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.135+0000 7f5b1159a640 10 monclient: _finish_auth 0 2024-04-26T00:43:23.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.135+0000 7f5b1159a640 10 monclient: _check_auth_tickets 2024-04-26T00:43:23.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.135+0000 7f5b1159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:23.143 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.135+0000 7f5b1159a640 1 -- 172.21.15.37:0/2242105074 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b040ab600 con 0x7f5b1409bae0 2024-04-26T00:43:23.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.135+0000 7f5b1159a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:23.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.135+0000 7f5b18c0c640 5 monclient: authenticate success, global_id 284175 2024-04-26T00:43:23.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.135+0000 7f5b18c0c640 10 monclient: get_monmap_and_config success 2024-04-26T00:43:23.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.135+0000 7f5b18c0c640 4 set_mon_vals no callback set 2024-04-26T00:43:23.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.135+0000 7f5b1159a640 1 -- 172.21.15.37:0/2242105074 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b040ab9d0 con 0x7f5b1409bae0 2024-04-26T00:43:23.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.135+0000 7f5b1159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:23.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.135+0000 7f5b1159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:43:23.144 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.135+0000 7f5b1159a640 10 monclient: dump: 2024-04-26T00:43:23.144 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:23.144 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:23.144 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:23.144 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:23.144 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:23.144 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:23.145 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:23.145 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:23.145 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:23.145 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:23.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.135+0000 7f5b18c0c640 10 monclient: shutdown 2024-04-26T00:43:23.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.135+0000 7f5b18c0c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:23.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.135+0000 7f5b18c0c640 1 -- 172.21.15.37:0/2242105074 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b1409bae0 msgr2=0x7f5b1409bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:23.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.135+0000 7f5b18c0c640 1 --2- 172.21.15.37:0/2242105074 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b1409bae0 0x7f5b1409bec0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5b14099290 tx=0x7f5b0401c3b0 comp rx=0 tx=0).stop 2024-04-26T00:43:23.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.135+0000 7f5b18c0c640 1 -- 172.21.15.37:0/2242105074 shutdown_connections 2024-04-26T00:43:23.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.135+0000 7f5b18c0c640 1 --2- 172.21.15.37:0/2242105074 >> v2:172.21.15.45:3300/0 conn(0x7f5b14137550 0x7f5b140727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:23.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.135+0000 7f5b18c0c640 1 --2- 172.21.15.37:0/2242105074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b14137150 0x7f5b14072270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:23.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.135+0000 7f5b18c0c640 1 --2- 172.21.15.37:0/2242105074 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b1409bae0 0x7f5b1409bec0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:23.145 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.135+0000 7f5b18c0c640 1 -- 172.21.15.37:0/2242105074 >> 172.21.15.37:0/2242105074 conn(0x7f5b140a6080 msgr2=0x7f5b140a1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:23.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.135+0000 7f5b18c0c640 1 -- 172.21.15.37:0/2242105074 shutdown_connections 2024-04-26T00:43:23.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.135+0000 7f5b18c0c640 1 -- 172.21.15.37:0/2242105074 wait complete. 2024-04-26T00:43:23.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b18c0c640 10 monclient: build_initial_monmap 2024-04-26T00:43:23.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b18c0c640 10 monclient: monmap: 2024-04-26T00:43:23.146 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:23.146 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:43:23.146 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:43:23.146 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:43:23.146 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:23.146 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:43:23.146 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:43:23.146 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:43:23.146 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:43:23.146 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:23.146 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b18c0c640 1 Processor -- start 2024-04-26T00:43:23.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b18c0c640 1 -- start start 2024-04-26T00:43:23.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b18c0c640 10 monclient: init 2024-04-26T00:43:23.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b18c0c640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:23.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b18c0c640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:43:23.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b18c0c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b1409bae0 0x7f5b141cf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:23.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b18c0c640 10 monclient(hunting): picked mon.noname-c con 0x7f5b1409bae0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:23.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b18c0c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b14137150 0x7f5b141cf730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:23.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b18c0c640 10 monclient(hunting): picked mon.noname-a con 0x7f5b14137150 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:43:23.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b18c0c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b14137550 0x7f5b141cfc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:23.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b18c0c640 10 monclient(hunting): picked mon.noname-b con 0x7f5b14137550 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:43:23.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b18c0c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:23.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b18c0c640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f5b14140a80 con 0x7f5b14137150 2024-04-26T00:43:23.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b18c0c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:23.147 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b18c0c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5b14140900 con 0x7f5b14137550 2024-04-26T00:43:23.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b18c0c640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:23.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b18c0c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5b14140c00 con 0x7f5b1409bae0 2024-04-26T00:43:23.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b18c0c640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:23.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b18c0c640 10 monclient(hunting): authenticate will time out at 2384.445182s 2024-04-26T00:43:23.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b1259c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b1409bae0 0x7f5b141cf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:23.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b1259c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b1409bae0 0x7f5b141cf1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51406/0 (socket says 172.21.15.37:51406) 2024-04-26T00:43:23.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b1259c640 1 -- 172.21.15.37:0/3271112102 learned_addr learned my addr 172.21.15.37:0/3271112102 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:23.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b1259c640 10 monclient(hunting): get_auth_request con 0x7f5b1409bae0 auth_method 0 2024-04-26T00:43:23.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b1259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:23.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b1259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:23.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b1259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:23.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b11d9b640 1 --2- 172.21.15.37:0/3271112102 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b14137150 0x7f5b141cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:23.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b11d9b640 10 monclient(hunting): get_auth_request con 0x7f5b14137150 auth_method 0 2024-04-26T00:43:23.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b11d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:23.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b11d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:23.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b11d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:23.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b1259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:23.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b1259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:23.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b1259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:23.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b11d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:23.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b11d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:23.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b11d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:23.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b1259c640 10 monclient(hunting): handle_auth_done global_id 254507 payload 824 2024-04-26T00:43:23.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b12d9d640 1 --2- 172.21.15.37:0/3271112102 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b14137550 0x7f5b141cfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:23.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b1259c640 1 -- 172.21.15.37:0/3271112102 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b14137550 msgr2=0x7f5b141cfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:23.149 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b1259c640 1 --2- 172.21.15.37:0/3271112102 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b14137550 0x7f5b141cfc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:23.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b1259c640 1 -- 172.21.15.37:0/3271112102 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b14137150 msgr2=0x7f5b141cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:23.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b1259c640 1 --2- 172.21.15.37:0/3271112102 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b14137150 0x7f5b141cf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:23.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b1259c640 10 monclient: _finish_hunting 0 2024-04-26T00:43:23.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b1259c640 1 monclient: found mon.noname-c 2024-04-26T00:43:23.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b1259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:23.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b1259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:43:23.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b1259c640 1 -- 172.21.15.37:0/3271112102 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b1413acf0 con 0x7f5b1409bae0 2024-04-26T00:43:23.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b12d9d640 10 monclient: get_auth_request con 0x7f5b14137550 auth_method 0 2024-04-26T00:43:23.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b12d9d640 1 --2- 172.21.15.37:0/3271112102 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b14137550 0x7f5b141cfc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:43:23.150 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b11d9b640 1 --2- 172.21.15.37:0/3271112102 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b14137150 0x7f5b141cf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:43:23.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b1259c640 1 -- 172.21.15.37:0/3271112102 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b1409bae0 msgr2=0x7f5b141cf1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:23.151 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.139+0000 7f5b1259c640 1 --2- 172.21.15.37:0/3271112102 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b1409bae0 0x7f5b141cf1f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5b040ae040 tx=0x7f5b04019150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:23.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.143+0000 7f5b02ffd640 1 -- 172.21.15.37:0/3271112102 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b040b0040 con 0x7f5b1409bae0 2024-04-26T00:43:23.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.143+0000 7f5b02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:23.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.143+0000 7f5b02ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:43:23.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.143+0000 7f5b02ffd640 10 monclient: dump: 2024-04-26T00:43:23.153 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:23.153 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:23.154 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:23.154 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:23.154 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:23.154 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:23.154 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:23.154 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:23.154 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:23.154 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:23.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.143+0000 7f5b02ffd640 10 monclient: _finish_auth 0 2024-04-26T00:43:23.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.143+0000 7f5b02ffd640 10 monclient: _check_auth_tickets 2024-04-26T00:43:23.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.143+0000 7f5b02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:23.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.143+0000 7f5b02ffd640 1 -- 172.21.15.37:0/3271112102 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b040aa5b0 con 0x7f5b1409bae0 2024-04-26T00:43:23.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.143+0000 7f5b02ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:23.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.143+0000 7f5b02ffd640 1 -- 172.21.15.37:0/3271112102 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b040aa9a0 con 0x7f5b1409bae0 2024-04-26T00:43:23.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.143+0000 7f5b18c0c640 5 monclient: authenticate success, global_id 254507 2024-04-26T00:43:23.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.143+0000 7f5b02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:23.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.143+0000 7f5b02ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:43:23.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.143+0000 7f5b02ffd640 10 monclient: dump: 2024-04-26T00:43:23.155 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:23.155 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:23.155 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:23.155 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:23.155 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:23.156 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:23.156 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:23.156 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:23.156 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:23.156 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:23.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.143+0000 7f5b10d99640 4 set_mon_vals no callback set 2024-04-26T00:43:23.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.143+0000 7f5b18c0c640 10 monclient: _renew_subs 2024-04-26T00:43:23.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.143+0000 7f5b18c0c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:23.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.143+0000 7f5b18c0c640 1 -- 172.21.15.37:0/3271112102 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b1413c7b0 con 0x7f5b1409bae0 2024-04-26T00:43:23.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.143+0000 7f5b18c0c640 10 monclient: _renew_subs 2024-04-26T00:43:23.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.143+0000 7f5b18c0c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:23.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.143+0000 7f5b18c0c640 1 -- 172.21.15.37:0/3271112102 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b141392f0 con 0x7f5b1409bae0 2024-04-26T00:43:23.156 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.143+0000 7f5b18c0c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:23.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.143+0000 7f5b18c0c640 1 -- 172.21.15.37:0/3271112102 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5b1404f580 con 0x7f5b1409bae0 2024-04-26T00:43:23.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.143+0000 7f5b02ffd640 1 -- 172.21.15.37:0/3271112102 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f5b040aab40 con 0x7f5b1409bae0 2024-04-26T00:43:23.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.143+0000 7f5b02ffd640 1 --2- 172.21.15.37:0/3271112102 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5af4035ed0 0x7f5af4038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:23.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.143+0000 7f5b02ffd640 1 -- 172.21.15.37:0/3271112102 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(899..899 src has 822..899) ==== 3550+0+0 (secure 0 0 0) 0x7f5b040e2590 con 0x7f5b1409bae0 2024-04-26T00:43:23.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.143+0000 7f5b02ffd640 1 --2- 172.21.15.37:0/3271112102 >> v2:172.21.15.45:6800/2173737584 conn(0x7f5af403ba20 0x7f5af403de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:23.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.143+0000 7f5b02ffd640 1 -- 172.21.15.37:0/3271112102 --> v2:172.21.15.45:6800/2173737584 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5b04007e50 con 0x7f5af403ba20 2024-04-26T00:43:23.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.143+0000 7f5b02ffd640 1 -- 172.21.15.37:0/3271112102 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_get_version_reply(handle=1 version=899) ==== 24+0+0 (secure 0 0 0) 0x7f5b040e9050 con 0x7f5b1409bae0 2024-04-26T00:43:23.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.143+0000 7f5b02ffd640 10 monclient: handle_get_version_reply finishing 1 version 899 2024-04-26T00:43:23.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.143+0000 7f5b11d9b640 1 --2- 172.21.15.37:0/3271112102 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5af4035ed0 0x7f5af4038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:23.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.143+0000 7f5b11d9b640 10 monclient: get_auth_request con 0x7f5af4035ed0 auth_method 0 2024-04-26T00:43:23.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.143+0000 7f5b12d9d640 1 --2- 172.21.15.37:0/3271112102 >> v2:172.21.15.45:6800/2173737584 conn(0x7f5af403ba20 0x7f5af403de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:23.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.143+0000 7f5b12d9d640 10 monclient: get_auth_request con 0x7f5af403ba20 auth_method 0 2024-04-26T00:43:23.157 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.147+0000 7f5b11d9b640 1 --2- 172.21.15.37:0/3271112102 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5af4035ed0 0x7f5af4038390 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f5b140a3bb0 tx=0x7f5b08009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:23.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.147+0000 7f5b12d9d640 1 --2- 172.21.15.37:0/3271112102 >> v2:172.21.15.45:6800/2173737584 conn(0x7f5af403ba20 0x7f5af403de40 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:23.158 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.151+0000 7f5b02ffd640 1 -- 172.21.15.37:0/3271112102 <== osd.3 v2:172.21.15.45:6800/2173737584 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5b04007e50 con 0x7f5af403ba20 2024-04-26T00:43:23.164 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.163+0000 7f5b18c0c640 1 -- 172.21.15.37:0/3271112102 --> v2:172.21.15.45:6800/2173737584 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f5b1413baa0 con 0x7f5af403ba20 2024-04-26T00:43:23.165 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.163+0000 7f5b02ffd640 1 -- 172.21.15.37:0/3271112102 <== osd.3 v2:172.21.15.45:6800/2173737584 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f5b1413baa0 con 0x7f5af403ba20 2024-04-26T00:43:23.165 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:43:23.165 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T00:43:23.166 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:43:23.166 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.163+0000 7f5b18c0c640 1 -- 172.21.15.37:0/3271112102 >> v2:172.21.15.45:6800/2173737584 conn(0x7f5af403ba20 msgr2=0x7f5af403de40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:23.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.163+0000 7f5b18c0c640 1 --2- 172.21.15.37:0/3271112102 >> v2:172.21.15.45:6800/2173737584 conn(0x7f5af403ba20 0x7f5af403de40 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:23.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.163+0000 7f5b18c0c640 1 -- 172.21.15.37:0/3271112102 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5af4035ed0 msgr2=0x7f5af4038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:23.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.163+0000 7f5b18c0c640 1 --2- 172.21.15.37:0/3271112102 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5af4035ed0 0x7f5af4038390 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f5b140a3bb0 tx=0x7f5b08009190 comp rx=0 tx=0).stop 2024-04-26T00:43:23.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.163+0000 7f5b18c0c640 10 monclient: shutdown 2024-04-26T00:43:23.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.163+0000 7f5b18c0c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:23.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.163+0000 7f5b18c0c640 1 -- 172.21.15.37:0/3271112102 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b1409bae0 msgr2=0x7f5b141cf1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:23.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.163+0000 7f5b18c0c640 1 --2- 172.21.15.37:0/3271112102 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b1409bae0 0x7f5b141cf1f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5b040ae040 tx=0x7f5b04019150 comp rx=0 tx=0).stop 2024-04-26T00:43:23.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.163+0000 7f5b12d9d640 1 -- 172.21.15.37:0/3271112102 reap_dead start 2024-04-26T00:43:23.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.163+0000 7f5b18c0c640 1 -- 172.21.15.37:0/3271112102 shutdown_connections 2024-04-26T00:43:23.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.163+0000 7f5b18c0c640 1 -- 172.21.15.37:0/3271112102 >> 172.21.15.37:0/3271112102 conn(0x7f5b140a6080 msgr2=0x7f5b1409aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:23.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.163+0000 7f5b18c0c640 1 -- 172.21.15.37:0/3271112102 shutdown_connections 2024-04-26T00:43:23.167 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.163+0000 7f5b18c0c640 1 -- 172.21.15.37:0/3271112102 wait complete. 2024-04-26T00:43:23.180 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-26T00:43:23.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b44da9640 10 monclient: get_monmap_and_config 2024-04-26T00:43:23.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b44da9640 10 monclient: build_initial_monmap 2024-04-26T00:43:23.292 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b44da9640 10 monclient: monmap: 2024-04-26T00:43:23.292 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:23.292 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:23.292 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:43:23.292973+0000 2024-04-26T00:43:23.292 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:43:23.292973+0000 2024-04-26T00:43:23.292 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:23.292 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:23.293 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:43:23.293 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:43:23.293 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:43:23.293 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:43:23.293 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:43:23.293 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:23.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b44da9640 1 Processor -- start 2024-04-26T00:43:23.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b44da9640 1 -- start start 2024-04-26T00:43:23.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b44da9640 10 monclient: init 2024-04-26T00:43:23.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b44da9640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:23.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b44da9640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-04-26T00:43:23.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b44da9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5b400a0300 addr v1:172.21.15.45:6789/0 2024-04-26T00:43:23.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b44da9640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f5b4009fc50 0x7f5b40139ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:23.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b44da9640 10 monclient(hunting): picked mon.noname-c con 0x7f5b4009fc50 addr v2:172.21.15.45:3300/0 2024-04-26T00:43:23.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b44da9640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f5b4013a400 0x7f5b4013c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:23.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b44da9640 10 monclient(hunting): picked mon.noname-a con 0x7f5b4013a400 addr v2:172.21.15.37:3300/0 2024-04-26T00:43:23.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b44da9640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b4009c190 con 0x7f5b400a0300 2024-04-26T00:43:23.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b44da9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:23.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b44da9640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f5b400a1110 con 0x7f5b4013a400 2024-04-26T00:43:23.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b44da9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:23.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b44da9640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f5b400a16d0 con 0x7f5b4009fc50 2024-04-26T00:43:23.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b44da9640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:23.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b44da9640 10 monclient(hunting): authenticate will time out at 2384.598341s 2024-04-26T00:43:23.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b3ed9d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f5b4013a400 0x7f5b4013c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:23.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b3ed9d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f5b4013a400 0x7f5b4013c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58260/0 (socket says 172.21.15.37:58260) 2024-04-26T00:43:23.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b3ed9d640 1 -- 172.21.15.37:0/2438463407 learned_addr learned my addr 172.21.15.37:0/2438463407 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:23.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b3ed9d640 10 monclient(hunting): get_auth_request con 0x7f5b4013a400 auth_method 0 2024-04-26T00:43:23.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b3ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:23.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b3ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:23.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b3ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:23.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b3ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:23.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b3ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:23.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b3dd9b640 1 --2- 172.21.15.37:0/2438463407 >> v2:172.21.15.45:3300/0 conn(0x7f5b4009fc50 0x7f5b40139ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:23.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b3ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:23.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b3dd9b640 10 monclient(hunting): get_auth_request con 0x7f5b4009fc50 auth_method 0 2024-04-26T00:43:23.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b3dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:23.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b3dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:23.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b3dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:23.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b3ed9d640 10 monclient(hunting): handle_auth_done global_id 284181 payload 274 2024-04-26T00:43:23.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b3ed9d640 1 -- 172.21.15.37:0/2438463407 >> v2:172.21.15.45:3300/0 conn(0x7f5b4009fc50 msgr2=0x7f5b40139ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:23.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b3ed9d640 1 --2- 172.21.15.37:0/2438463407 >> v2:172.21.15.45:3300/0 conn(0x7f5b4009fc50 0x7f5b40139ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:23.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b3ed9d640 1 -- 172.21.15.37:0/2438463407 >> v1:172.21.15.45:6789/0 conn(0x7f5b400a0300 legacy=0x7f5b400a06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:23.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b3ed9d640 10 monclient: _finish_hunting 0 2024-04-26T00:43:23.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b3ed9d640 1 monclient: found mon.noname-a 2024-04-26T00:43:23.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b3ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:23.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b3ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:43:23.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b3ed9d640 1 -- 172.21.15.37:0/2438463407 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b4009f2a0 con 0x7f5b4013a400 2024-04-26T00:43:23.298 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.291+0000 7f5b3dd9b640 1 --2- 172.21.15.37:0/2438463407 >> v2:172.21.15.45:3300/0 conn(0x7f5b4009fc50 0x7f5b40139ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:43:23.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.295+0000 7f5b3ed9d640 1 -- 172.21.15.37:0/2438463407 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b4013a400 msgr2=0x7f5b4013c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:23.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.295+0000 7f5b3ed9d640 1 --2- 172.21.15.37:0/2438463407 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b4013a400 0x7f5b4013c7f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5b4006caa0 tx=0x7f5b3801c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21cfef0b2d6e6029 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:23.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3d59a640 1 -- 172.21.15.37:0/2438463407 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b380a5640 con 0x7f5b4013a400 2024-04-26T00:43:23.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:23.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:43:23.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3d59a640 10 monclient: dump: 2024-04-26T00:43:23.303 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:23.303 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:23.303 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:23.304 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:23.304 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:23.304 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:23.304 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:23.304 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:23.304 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:23.304 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:23.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3d59a640 10 monclient: _finish_auth 0 2024-04-26T00:43:23.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3d59a640 10 monclient: _check_auth_tickets 2024-04-26T00:43:23.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:23.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3d59a640 1 -- 172.21.15.37:0/2438463407 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b380a57e0 con 0x7f5b4013a400 2024-04-26T00:43:23.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3d59a640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:23.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3d59a640 1 -- 172.21.15.37:0/2438463407 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b380a5bb0 con 0x7f5b4013a400 2024-04-26T00:43:23.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:23.304 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:43:23.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3d59a640 10 monclient: dump: 2024-04-26T00:43:23.305 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:23.305 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:23.305 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:23.305 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:23.305 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:23.305 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:23.305 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:23.305 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:23.305 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:23.305 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:23.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b44da9640 5 monclient: authenticate success, global_id 284181 2024-04-26T00:43:23.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b44da9640 10 monclient: get_monmap_and_config success 2024-04-26T00:43:23.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b44da9640 4 set_mon_vals no callback set 2024-04-26T00:43:23.305 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b44da9640 10 monclient: shutdown 2024-04-26T00:43:23.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b44da9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:23.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b44da9640 1 -- 172.21.15.37:0/2438463407 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b4013a400 msgr2=0x7f5b4013c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:23.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b44da9640 1 --2- 172.21.15.37:0/2438463407 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b4013a400 0x7f5b4013c7f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5b4006caa0 tx=0x7f5b3801c870 comp rx=0 tx=0).stop 2024-04-26T00:43:23.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b44da9640 1 -- 172.21.15.37:0/2438463407 shutdown_connections 2024-04-26T00:43:23.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b44da9640 1 --2- 172.21.15.37:0/2438463407 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b4013a400 0x7f5b4013c7f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:23.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b44da9640 1 --2- 172.21.15.37:0/2438463407 >> v2:172.21.15.45:3300/0 conn(0x7f5b4009fc50 0x7f5b40139ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:23.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b44da9640 1 -- 172.21.15.37:0/2438463407 >> 172.21.15.37:0/2438463407 conn(0x7f5b400a1d50 msgr2=0x7f5b400a2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:23.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b44da9640 1 -- 172.21.15.37:0/2438463407 shutdown_connections 2024-04-26T00:43:23.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b44da9640 1 -- 172.21.15.37:0/2438463407 wait complete. 2024-04-26T00:43:23.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b44da9640 10 monclient: build_initial_monmap 2024-04-26T00:43:23.306 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b44da9640 10 monclient: monmap: 2024-04-26T00:43:23.306 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:23.306 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:43:23.306 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:43:23.307 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:43:23.307 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:23.307 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:43:23.307 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:43:23.307 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:43:23.307 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:43:23.307 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:23.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b44da9640 1 Processor -- start 2024-04-26T00:43:23.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b44da9640 1 -- start start 2024-04-26T00:43:23.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b44da9640 10 monclient: init 2024-04-26T00:43:23.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b44da9640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:23.307 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b44da9640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T00:43:23.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b44da9640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b4009fc50 0x7f5b401cee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:23.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b44da9640 10 monclient(hunting): picked mon.noname-a con 0x7f5b4009fc50 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:43:23.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b44da9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b400a0300 0x7f5b40133bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:23.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b44da9640 10 monclient(hunting): picked mon.noname-b con 0x7f5b400a0300 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:43:23.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b44da9640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b4013a400 0x7f5b401340f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:23.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b44da9640 10 monclient(hunting): picked mon.noname-c con 0x7f5b4013a400 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:23.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b44da9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:23.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b44da9640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f5b4013f8e0 con 0x7f5b4009fc50 2024-04-26T00:43:23.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b44da9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:23.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b44da9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f5b4013f760 con 0x7f5b400a0300 2024-04-26T00:43:23.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b44da9640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:23.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b44da9640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5b4013fa60 con 0x7f5b4013a400 2024-04-26T00:43:23.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b44da9640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:23.308 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b44da9640 10 monclient(hunting): authenticate will time out at 2384.606598s 2024-04-26T00:43:23.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3e59c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b4009fc50 0x7f5b401cee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:23.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3e59c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b4009fc50 0x7f5b401cee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58270/0 (socket says 172.21.15.37:58270) 2024-04-26T00:43:23.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3e59c640 1 -- 172.21.15.37:0/319299390 learned_addr learned my addr 172.21.15.37:0/319299390 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:23.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3e59c640 10 monclient(hunting): get_auth_request con 0x7f5b4009fc50 auth_method 0 2024-04-26T00:43:23.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:23.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:23.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:23.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3ed9d640 1 --2- 172.21.15.37:0/319299390 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b4013a400 0x7f5b401340f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:23.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3ed9d640 10 monclient(hunting): get_auth_request con 0x7f5b4013a400 auth_method 0 2024-04-26T00:43:23.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:23.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:23.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:23.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:23.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:23.309 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:23.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3dd9b640 1 --2- 172.21.15.37:0/319299390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b400a0300 0x7f5b40133bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:23.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:23.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:23.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:23.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3dd9b640 10 monclient(hunting): get_auth_request con 0x7f5b400a0300 auth_method 0 2024-04-26T00:43:23.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:23.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:23.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:23.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3e59c640 10 monclient(hunting): handle_auth_done global_id 284184 payload 824 2024-04-26T00:43:23.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3e59c640 1 -- 172.21.15.37:0/319299390 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b4013a400 msgr2=0x7f5b401340f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:23.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3e59c640 1 --2- 172.21.15.37:0/319299390 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b4013a400 0x7f5b401340f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:23.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3e59c640 1 -- 172.21.15.37:0/319299390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b400a0300 msgr2=0x7f5b40133bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:23.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3e59c640 1 --2- 172.21.15.37:0/319299390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b400a0300 0x7f5b40133bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:23.310 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3e59c640 10 monclient: _finish_hunting 0 2024-04-26T00:43:23.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3e59c640 1 monclient: found mon.noname-a 2024-04-26T00:43:23.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:23.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:43:23.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3e59c640 1 -- 172.21.15.37:0/319299390 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b4009f550 con 0x7f5b4009fc50 2024-04-26T00:43:23.311 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.299+0000 7f5b3dd9b640 1 --2- 172.21.15.37:0/319299390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b400a0300 0x7f5b40133bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:43:23.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.303+0000 7f5b3e59c640 1 -- 172.21.15.37:0/319299390 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b4009fc50 msgr2=0x7f5b401cee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:23.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.303+0000 7f5b3e59c640 1 --2- 172.21.15.37:0/319299390 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b4009fc50 0x7f5b401cee30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5b3000c450 tx=0x7f5b3000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:23.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.307+0000 7f5b2effd640 1 -- 172.21.15.37:0/319299390 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b300119a0 con 0x7f5b4009fc50 2024-04-26T00:43:23.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.307+0000 7f5b2effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:23.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.307+0000 7f5b2effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:43:23.313 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.307+0000 7f5b2effd640 10 monclient: dump: 2024-04-26T00:43:23.313 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:23.313 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:23.313 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:23.313 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:23.314 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:23.314 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:23.314 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:23.314 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:23.314 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:23.314 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:23.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.307+0000 7f5b2effd640 10 monclient: _finish_auth 0 2024-04-26T00:43:23.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.307+0000 7f5b2effd640 10 monclient: _check_auth_tickets 2024-04-26T00:43:23.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.307+0000 7f5b2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:23.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.307+0000 7f5b2effd640 1 -- 172.21.15.37:0/319299390 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b30017070 con 0x7f5b4009fc50 2024-04-26T00:43:23.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.307+0000 7f5b44da9640 5 monclient: authenticate success, global_id 284184 2024-04-26T00:43:23.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.307+0000 7f5b44da9640 10 monclient: _renew_subs 2024-04-26T00:43:23.314 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.307+0000 7f5b44da9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:43:23.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.307+0000 7f5b44da9640 1 -- 172.21.15.37:0/319299390 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b4009e390 con 0x7f5b4009fc50 2024-04-26T00:43:23.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.307+0000 7f5b2effd640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:23.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.307+0000 7f5b2effd640 1 -- 172.21.15.37:0/319299390 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b30011da0 con 0x7f5b4009fc50 2024-04-26T00:43:23.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.307+0000 7f5b2effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:23.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.307+0000 7f5b2effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:43:23.315 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.307+0000 7f5b2effd640 10 monclient: dump: 2024-04-26T00:43:23.315 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:23.315 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:23.315 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:23.315 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:23.315 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:23.315 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:23.315 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:23.316 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:23.316 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:23.316 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:23.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.307+0000 7f5b3cd99640 4 set_mon_vals no callback set 2024-04-26T00:43:23.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.307+0000 7f5b44da9640 10 monclient: _renew_subs 2024-04-26T00:43:23.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.307+0000 7f5b44da9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:43:23.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.307+0000 7f5b44da9640 1 -- 172.21.15.37:0/319299390 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b4011fcc0 con 0x7f5b4009fc50 2024-04-26T00:43:23.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.307+0000 7f5b2effd640 1 -- 172.21.15.37:0/319299390 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f5b3001c460 con 0x7f5b4009fc50 2024-04-26T00:43:23.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.307+0000 7f5b44da9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:23.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.307+0000 7f5b44da9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:23.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.307+0000 7f5b44da9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:43:23.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.307+0000 7f5b44da9640 1 -- 172.21.15.37:0/319299390 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b4004f510 con 0x7f5b4009fc50 2024-04-26T00:43:23.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.307+0000 7f5b2effd640 1 --2- 172.21.15.37:0/319299390 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5b20035ed0 0x7f5b20038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:23.316 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.307+0000 7f5b2effd640 1 -- 172.21.15.37:0/319299390 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(899..899 src has 822..899) ==== 3550+0+0 (secure 0 0 0) 0x7f5b30048760 con 0x7f5b4009fc50 2024-04-26T00:43:23.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.307+0000 7f5b3dd9b640 1 --2- 172.21.15.37:0/319299390 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5b20035ed0 0x7f5b20038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:23.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.307+0000 7f5b3dd9b640 10 monclient: get_auth_request con 0x7f5b20035ed0 auth_method 0 2024-04-26T00:43:23.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.311+0000 7f5b3dd9b640 1 --2- 172.21.15.37:0/319299390 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5b20035ed0 0x7f5b20038390 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f5b4011de70 tx=0x7f5b34008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:23.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.311+0000 7f5b2effd640 1 -- 172.21.15.37:0/319299390 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5b30047630 con 0x7f5b4009fc50 2024-04-26T00:43:23.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.311+0000 7f5b2effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:23.317 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.311+0000 7f5b2effd640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:43:23.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.467+0000 7f5b44da9640 1 -- 172.21.15.37:0/319299390 --> v2:172.21.15.45:6812/2379918623 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f5b40120620 con 0x7f5b20035ed0 2024-04-26T00:43:23.467 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.467+0000 7f5b2effd640 1 -- 172.21.15.37:0/319299390 <== mgr.4107 v2:172.21.15.45:6812/2379918623 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f5b40120620 con 0x7f5b20035ed0 2024-04-26T00:43:23.467 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 3 to deep-scrub 2024-04-26T00:43:23.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.467+0000 7f5b44da9640 1 -- 172.21.15.37:0/319299390 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5b20035ed0 msgr2=0x7f5b20038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:23.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.467+0000 7f5b44da9640 1 --2- 172.21.15.37:0/319299390 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5b20035ed0 0x7f5b20038390 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f5b4011de70 tx=0x7f5b34008040 comp rx=0 tx=0).stop 2024-04-26T00:43:23.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.467+0000 7f5b44da9640 10 monclient: shutdown 2024-04-26T00:43:23.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.467+0000 7f5b44da9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:23.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.467+0000 7f5b44da9640 1 -- 172.21.15.37:0/319299390 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b4009fc50 msgr2=0x7f5b401cee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:23.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.467+0000 7f5b44da9640 1 --2- 172.21.15.37:0/319299390 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b4009fc50 0x7f5b401cee30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5b3000c450 tx=0x7f5b3000c910 comp rx=0 tx=0).stop 2024-04-26T00:43:23.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.467+0000 7f5b44da9640 1 -- 172.21.15.37:0/319299390 shutdown_connections 2024-04-26T00:43:23.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.467+0000 7f5b44da9640 1 --2- 172.21.15.37:0/319299390 >> v2:172.21.15.45:6812/2379918623 conn(0x7f5b20035ed0 0x7f5b20038390 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:23.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.467+0000 7f5b44da9640 1 --2- 172.21.15.37:0/319299390 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5b4013a400 0x7f5b401340f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:23.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.467+0000 7f5b44da9640 1 --2- 172.21.15.37:0/319299390 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5b400a0300 0x7f5b40133bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:23.470 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.467+0000 7f5b44da9640 1 --2- 172.21.15.37:0/319299390 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5b4009fc50 0x7f5b401cee30 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:23.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.467+0000 7f5b44da9640 1 -- 172.21.15.37:0/319299390 >> 172.21.15.37:0/319299390 conn(0x7f5b400a1d50 msgr2=0x7f5b40134c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:23.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.467+0000 7f5b44da9640 1 -- 172.21.15.37:0/319299390 shutdown_connections 2024-04-26T00:43:23.471 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.467+0000 7f5b44da9640 1 -- 172.21.15.37:0/319299390 wait complete. 2024-04-26T00:43:23.485 INFO:tasks.ceph:Scrubbing osd.4 2024-04-26T00:43:23.485 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-26T00:43:23.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.611+0000 7fa27bf1d640 10 monclient: get_monmap_and_config 2024-04-26T00:43:23.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.611+0000 7fa27bf1d640 10 monclient: build_initial_monmap 2024-04-26T00:43:23.612 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.611+0000 7fa27bf1d640 10 monclient: monmap: 2024-04-26T00:43:23.612 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:23.613 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:23.613 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:43:23.613413+0000 2024-04-26T00:43:23.613 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:43:23.613413+0000 2024-04-26T00:43:23.613 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:23.613 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:23.613 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:43:23.613 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:43:23.613 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:43:23.613 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:43:23.613 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:43:23.613 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:23.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.611+0000 7fa27bf1d640 1 Processor -- start 2024-04-26T00:43:23.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.611+0000 7fa27bf1d640 1 -- start start 2024-04-26T00:43:23.614 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.611+0000 7fa27bf1d640 10 monclient: init 2024-04-26T00:43:23.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.611+0000 7fa27bf1d640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:23.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.611+0000 7fa27bf1d640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-04-26T00:43:23.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.611+0000 7fa27bf1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa2740a0dd0 addr v1:172.21.15.45:6789/0 2024-04-26T00:43:23.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.611+0000 7fa27bf1d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fa274136530 0x7fa274138920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:23.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.611+0000 7fa27bf1d640 10 monclient(hunting): picked mon.noname-c con 0x7fa274136530 addr v2:172.21.15.45:3300/0 2024-04-26T00:43:23.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.611+0000 7fa27bf1d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa274138e60 0x7fa27413b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:23.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.611+0000 7fa27bf1d640 10 monclient(hunting): picked mon.noname-b con 0x7fa274138e60 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:23.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.611+0000 7fa27bf1d640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa274099740 con 0x7fa2740a0dd0 2024-04-26T00:43:23.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.611+0000 7fa27bf1d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:23.616 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.611+0000 7fa27bf1d640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fa2740a11b0 con 0x7fa274136530 2024-04-26T00:43:23.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.611+0000 7fa27bf1d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:23.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.611+0000 7fa27bf1d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa27409eac0 con 0x7fa274138e60 2024-04-26T00:43:23.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.611+0000 7fa27bf1d640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:23.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.611+0000 7fa27bf1d640 10 monclient(hunting): authenticate will time out at 2384.919779s 2024-04-26T00:43:23.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.611+0000 7fa27a4ba640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa274138e60 0x7fa27413b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:23.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.615+0000 7fa27a4ba640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa274138e60 0x7fa27413b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51432/0 (socket says 172.21.15.37:51432) 2024-04-26T00:43:23.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.615+0000 7fa27a4ba640 1 -- 172.21.15.37:0/1010882914 learned_addr learned my addr 172.21.15.37:0/1010882914 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:23.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.615+0000 7fa27a4ba640 10 monclient(hunting): get_auth_request con 0x7fa274138e60 auth_method 0 2024-04-26T00:43:23.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.615+0000 7fa27a4ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:23.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.615+0000 7fa27a4ba640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:23.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.615+0000 7fa27a4ba640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:23.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.615+0000 7fa2794b8640 1 --2- 172.21.15.37:0/1010882914 >> v2:172.21.15.45:3300/0 conn(0x7fa274136530 0x7fa274138920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:23.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.615+0000 7fa27a4ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:23.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.615+0000 7fa27a4ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:23.617 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.615+0000 7fa27a4ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:23.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.615+0000 7fa2794b8640 10 monclient(hunting): get_auth_request con 0x7fa274136530 auth_method 0 2024-04-26T00:43:23.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.615+0000 7fa2794b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:23.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.615+0000 7fa2794b8640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:23.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.615+0000 7fa2794b8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:23.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.615+0000 7fa27a4ba640 10 monclient(hunting): handle_auth_done global_id 254513 payload 274 2024-04-26T00:43:23.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.615+0000 7fa27a4ba640 1 -- 172.21.15.37:0/1010882914 >> v2:172.21.15.45:3300/0 conn(0x7fa274136530 msgr2=0x7fa274138920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:23.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.615+0000 7fa27a4ba640 1 --2- 172.21.15.37:0/1010882914 >> v2:172.21.15.45:3300/0 conn(0x7fa274136530 0x7fa274138920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:23.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.615+0000 7fa27a4ba640 1 -- 172.21.15.37:0/1010882914 >> v1:172.21.15.45:6789/0 conn(0x7fa2740a0dd0 legacy=0x7fa274099e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:23.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.615+0000 7fa27a4ba640 10 monclient: _finish_hunting 0 2024-04-26T00:43:23.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.615+0000 7fa27a4ba640 1 monclient: found mon.noname-b 2024-04-26T00:43:23.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.615+0000 7fa27a4ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:23.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.615+0000 7fa27a4ba640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:43:23.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.615+0000 7fa27a4ba640 1 -- 172.21.15.37:0/1010882914 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa274099a20 con 0x7fa274138e60 2024-04-26T00:43:23.618 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.615+0000 7fa2794b8640 1 --2- 172.21.15.37:0/1010882914 >> v2:172.21.15.45:3300/0 conn(0x7fa274136530 0x7fa274138920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T00:43:23.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa27a4ba640 1 -- 172.21.15.37:0/1010882914 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa274138e60 msgr2=0x7fa27413b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:23.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa27a4ba640 1 --2- 172.21.15.37:0/1010882914 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa274138e60 0x7fa27413b250 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa26c00d150 tx=0x7fa26c01eb40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bc06d7bdda76e25 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:23.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa278cb7640 1 -- 172.21.15.37:0/1010882914 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa26c0a77e0 con 0x7fa274138e60 2024-04-26T00:43:23.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa278cb7640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:23.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa278cb7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:43:23.620 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa278cb7640 10 monclient: dump: 2024-04-26T00:43:23.620 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:23.621 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:23.621 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:23.621 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:23.621 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:23.621 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:23.621 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:23.621 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:23.621 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:23.621 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:23.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa278cb7640 10 monclient: _finish_auth 0 2024-04-26T00:43:23.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa278cb7640 10 monclient: _check_auth_tickets 2024-04-26T00:43:23.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa278cb7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:23.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa278cb7640 1 -- 172.21.15.37:0/1010882914 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa26c0a7980 con 0x7fa274138e60 2024-04-26T00:43:23.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa27bf1d640 5 monclient: authenticate success, global_id 254513 2024-04-26T00:43:23.621 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa27bf1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:43:23.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa278cb7640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:23.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa27bf1d640 10 monclient: get_monmap_and_config success 2024-04-26T00:43:23.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa27bf1d640 4 set_mon_vals no callback set 2024-04-26T00:43:23.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa278cb7640 1 -- 172.21.15.37:0/1010882914 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa26c0a7d50 con 0x7fa274138e60 2024-04-26T00:43:23.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa278cb7640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:23.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa278cb7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:43:23.622 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa278cb7640 10 monclient: dump: 2024-04-26T00:43:23.622 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:23.622 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:23.622 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:23.622 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:23.622 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:23.622 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:23.622 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:23.623 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:23.623 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:23.623 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:23.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa27bf1d640 10 monclient: shutdown 2024-04-26T00:43:23.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa27bf1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:23.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa27bf1d640 1 -- 172.21.15.37:0/1010882914 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa274138e60 msgr2=0x7fa27413b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:23.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa27bf1d640 1 --2- 172.21.15.37:0/1010882914 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa274138e60 0x7fa27413b250 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa26c00d150 tx=0x7fa26c01eb40 comp rx=0 tx=0).stop 2024-04-26T00:43:23.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa27bf1d640 1 -- 172.21.15.37:0/1010882914 shutdown_connections 2024-04-26T00:43:23.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa27bf1d640 1 --2- 172.21.15.37:0/1010882914 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa274138e60 0x7fa27413b250 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:23.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa27bf1d640 1 --2- 172.21.15.37:0/1010882914 >> v2:172.21.15.45:3300/0 conn(0x7fa274136530 0x7fa274138920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:23.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa27bf1d640 1 -- 172.21.15.37:0/1010882914 >> 172.21.15.37:0/1010882914 conn(0x7fa2740a6080 msgr2=0x7fa2740a1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:23.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa27bf1d640 1 -- 172.21.15.37:0/1010882914 shutdown_connections 2024-04-26T00:43:23.626 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa27bf1d640 1 -- 172.21.15.37:0/1010882914 wait complete. 2024-04-26T00:43:23.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa27bf1d640 10 monclient: build_initial_monmap 2024-04-26T00:43:23.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa27bf1d640 10 monclient: monmap: 2024-04-26T00:43:23.627 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:23.627 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:43:23.627 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:43:23.627 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:43:23.627 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:23.627 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:43:23.627 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:43:23.627 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:43:23.627 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:43:23.627 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:23.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa27bf1d640 1 Processor -- start 2024-04-26T00:43:23.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa27bf1d640 1 -- start start 2024-04-26T00:43:23.627 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa27bf1d640 10 monclient: init 2024-04-26T00:43:23.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa27bf1d640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:23.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa27bf1d640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:43:23.628 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa27bf1d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2740a0dd0 0x7fa274071ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:23.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa27bf1d640 10 monclient(hunting): picked mon.noname-a con 0x7fa2740a0dd0 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:43:23.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa27bf1d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa274136530 0x7fa2741358c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:23.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa27bf1d640 10 monclient(hunting): picked mon.noname-c con 0x7fa274136530 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:23.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa27bf1d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa274138e60 0x7fa274135e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:23.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa27bf1d640 10 monclient(hunting): picked mon.noname-b con 0x7fa274138e60 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:43:23.629 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa27bf1d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:23.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa27bf1d640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa27409d4a0 con 0x7fa2740a0dd0 2024-04-26T00:43:23.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa27bf1d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:23.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa27bf1d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa27409d320 con 0x7fa274138e60 2024-04-26T00:43:23.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa27bf1d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:23.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa27bf1d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa27409d620 con 0x7fa274136530 2024-04-26T00:43:23.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa27bf1d640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:23.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa27bf1d640 10 monclient(hunting): authenticate will time out at 2384.927751s 2024-04-26T00:43:23.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa2794b8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa274136530 0x7fa2741358c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:23.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa2794b8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa274136530 0x7fa2741358c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51442/0 (socket says 172.21.15.37:51442) 2024-04-26T00:43:23.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa2794b8640 1 -- 172.21.15.37:0/3671279593 learned_addr learned my addr 172.21.15.37:0/3671279593 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:23.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa2794b8640 10 monclient(hunting): get_auth_request con 0x7fa274136530 auth_method 0 2024-04-26T00:43:23.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa2794b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:23.630 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa2794b8640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:23.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa2794b8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:23.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.623+0000 7fa2794b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:23.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.623+0000 7fa2794b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:23.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.619+0000 7fa279cb9640 1 --2- 172.21.15.37:0/3671279593 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2740a0dd0 0x7fa274071ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:23.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.623+0000 7fa2794b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:23.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.623+0000 7fa279cb9640 10 monclient(hunting): get_auth_request con 0x7fa2740a0dd0 auth_method 0 2024-04-26T00:43:23.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.623+0000 7fa279cb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:23.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.623+0000 7fa279cb9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:23.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.623+0000 7fa279cb9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:23.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.623+0000 7fa279cb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:23.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.623+0000 7fa279cb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:23.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.623+0000 7fa27a4ba640 1 --2- 172.21.15.37:0/3671279593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa274138e60 0x7fa274135e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:23.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.623+0000 7fa279cb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:23.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.623+0000 7fa27a4ba640 10 monclient(hunting): get_auth_request con 0x7fa274138e60 auth_method 0 2024-04-26T00:43:23.631 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.623+0000 7fa27a4ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:23.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.623+0000 7fa27a4ba640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:23.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.623+0000 7fa27a4ba640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:23.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.623+0000 7fa2794b8640 10 monclient(hunting): handle_auth_done global_id 254516 payload 824 2024-04-26T00:43:23.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.623+0000 7fa2794b8640 1 -- 172.21.15.37:0/3671279593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa274138e60 msgr2=0x7fa274135e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:23.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.623+0000 7fa2794b8640 1 --2- 172.21.15.37:0/3671279593 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa274138e60 0x7fa274135e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:23.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.623+0000 7fa2794b8640 1 -- 172.21.15.37:0/3671279593 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2740a0dd0 msgr2=0x7fa274071ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:23.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.623+0000 7fa2794b8640 1 --2- 172.21.15.37:0/3671279593 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2740a0dd0 0x7fa274071ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:23.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.623+0000 7fa2794b8640 10 monclient: _finish_hunting 0 2024-04-26T00:43:23.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.623+0000 7fa2794b8640 1 monclient: found mon.noname-c 2024-04-26T00:43:23.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.623+0000 7fa2794b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:23.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.623+0000 7fa2794b8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:43:23.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.623+0000 7fa2794b8640 1 -- 172.21.15.37:0/3671279593 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa27412d320 con 0x7fa274136530 2024-04-26T00:43:23.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.623+0000 7fa2794b8640 1 -- 172.21.15.37:0/3671279593 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa274136530 msgr2=0x7fa2741358c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:23.632 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.623+0000 7fa2794b8640 1 --2- 172.21.15.37:0/3671279593 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa274136530 0x7fa2741358c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa26400e810 tx=0x7fa26400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:23.634 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.627+0000 7fa26a7fc640 1 -- 172.21.15.37:0/3671279593 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa264017070 con 0x7fa274136530 2024-04-26T00:43:23.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.627+0000 7fa26a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:23.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.627+0000 7fa26a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:43:23.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.627+0000 7fa26a7fc640 10 monclient: dump: 2024-04-26T00:43:23.635 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:23.635 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:23.635 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:23.635 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:23.635 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:23.635 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:23.635 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:23.635 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:23.635 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:23.635 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:23.635 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.627+0000 7fa26a7fc640 10 monclient: _finish_auth 0 2024-04-26T00:43:23.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.627+0000 7fa26a7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:43:23.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.627+0000 7fa26a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:23.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.627+0000 7fa27bf1d640 5 monclient: authenticate success, global_id 254516 2024-04-26T00:43:23.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.627+0000 7fa26a7fc640 1 -- 172.21.15.37:0/3671279593 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa26400cae0 con 0x7fa274136530 2024-04-26T00:43:23.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.627+0000 7fa27bf1d640 10 monclient: _renew_subs 2024-04-26T00:43:23.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.627+0000 7fa27bf1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:23.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.627+0000 7fa27bf1d640 1 -- 172.21.15.37:0/3671279593 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa274136340 con 0x7fa274136530 2024-04-26T00:43:23.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.627+0000 7fa26a7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:23.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.627+0000 7fa26a7fc640 1 -- 172.21.15.37:0/3671279593 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa26400f040 con 0x7fa274136530 2024-04-26T00:43:23.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.627+0000 7fa26a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:23.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.627+0000 7fa26a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:43:23.636 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.627+0000 7fa26a7fc640 10 monclient: dump: 2024-04-26T00:43:23.636 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:23.636 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:23.637 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:23.637 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:23.637 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:23.637 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:23.637 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:23.637 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:23.637 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:23.637 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:23.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.627+0000 7fa26bfff640 4 set_mon_vals no callback set 2024-04-26T00:43:23.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.627+0000 7fa27bf1d640 10 monclient: _renew_subs 2024-04-26T00:43:23.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.627+0000 7fa27bf1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:23.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.627+0000 7fa27bf1d640 1 -- 172.21.15.37:0/3671279593 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2740a31c0 con 0x7fa274136530 2024-04-26T00:43:23.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.627+0000 7fa27bf1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:23.637 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.627+0000 7fa27bf1d640 1 -- 172.21.15.37:0/3671279593 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa27409ece0 con 0x7fa274136530 2024-04-26T00:43:23.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.627+0000 7fa26a7fc640 1 -- 172.21.15.37:0/3671279593 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa26401a070 con 0x7fa274136530 2024-04-26T00:43:23.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.627+0000 7fa26a7fc640 1 --2- 172.21.15.37:0/3671279593 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa258035ed0 0x7fa258038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:23.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.627+0000 7fa26a7fc640 1 -- 172.21.15.37:0/3671279593 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(899..899 src has 822..899) ==== 3550+0+0 (secure 0 0 0) 0x7fa264048590 con 0x7fa274136530 2024-04-26T00:43:23.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.627+0000 7fa26a7fc640 1 --2- 172.21.15.37:0/3671279593 >> v2:172.21.15.45:6804/152605921 conn(0x7fa25803ba20 0x7fa25803de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:23.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.627+0000 7fa26a7fc640 1 -- 172.21.15.37:0/3671279593 --> v2:172.21.15.45:6804/152605921 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa25803e4b0 con 0x7fa25803ba20 2024-04-26T00:43:23.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.627+0000 7fa26a7fc640 1 -- 172.21.15.37:0/3671279593 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_get_version_reply(handle=1 version=899) ==== 24+0+0 (secure 0 0 0) 0x7fa26404f050 con 0x7fa274136530 2024-04-26T00:43:23.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.627+0000 7fa26a7fc640 10 monclient: handle_get_version_reply finishing 1 version 899 2024-04-26T00:43:23.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.627+0000 7fa279cb9640 1 --2- 172.21.15.37:0/3671279593 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa258035ed0 0x7fa258038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:23.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.627+0000 7fa279cb9640 10 monclient: get_auth_request con 0x7fa258035ed0 auth_method 0 2024-04-26T00:43:23.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.627+0000 7fa27a4ba640 1 --2- 172.21.15.37:0/3671279593 >> v2:172.21.15.45:6804/152605921 conn(0x7fa25803ba20 0x7fa25803de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:23.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.627+0000 7fa27a4ba640 10 monclient: get_auth_request con 0x7fa25803ba20 auth_method 0 2024-04-26T00:43:23.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.631+0000 7fa279cb9640 1 --2- 172.21.15.37:0/3671279593 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa258035ed0 0x7fa258038390 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fa2700098e0 tx=0x7fa270009060 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:23.638 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.631+0000 7fa27a4ba640 1 --2- 172.21.15.37:0/3671279593 >> v2:172.21.15.45:6804/152605921 conn(0x7fa25803ba20 0x7fa25803de40 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:23.639 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.635+0000 7fa26a7fc640 1 -- 172.21.15.37:0/3671279593 <== osd.4 v2:172.21.15.45:6804/152605921 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa25803e4b0 con 0x7fa25803ba20 2024-04-26T00:43:23.648 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.647+0000 7fa27bf1d640 1 -- 172.21.15.37:0/3671279593 --> v2:172.21.15.45:6804/152605921 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fa27404f580 con 0x7fa25803ba20 2024-04-26T00:43:23.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.647+0000 7fa26a7fc640 1 -- 172.21.15.37:0/3671279593 <== osd.4 v2:172.21.15.45:6804/152605921 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fa27404f580 con 0x7fa25803ba20 2024-04-26T00:43:23.650 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:43:23.650 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T00:43:23.650 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:43:23.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.647+0000 7fa27bf1d640 1 -- 172.21.15.37:0/3671279593 >> v2:172.21.15.45:6804/152605921 conn(0x7fa25803ba20 msgr2=0x7fa25803de40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:23.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.647+0000 7fa27bf1d640 1 --2- 172.21.15.37:0/3671279593 >> v2:172.21.15.45:6804/152605921 conn(0x7fa25803ba20 0x7fa25803de40 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:23.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.647+0000 7fa27bf1d640 1 -- 172.21.15.37:0/3671279593 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa258035ed0 msgr2=0x7fa258038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:23.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.647+0000 7fa27bf1d640 1 --2- 172.21.15.37:0/3671279593 >> v2:172.21.15.45:6812/2379918623 conn(0x7fa258035ed0 0x7fa258038390 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fa2700098e0 tx=0x7fa270009060 comp rx=0 tx=0).stop 2024-04-26T00:43:23.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.647+0000 7fa27bf1d640 10 monclient: shutdown 2024-04-26T00:43:23.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.647+0000 7fa27bf1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:23.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.647+0000 7fa27bf1d640 1 -- 172.21.15.37:0/3671279593 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa274136530 msgr2=0x7fa2741358c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:23.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.647+0000 7fa27bf1d640 1 --2- 172.21.15.37:0/3671279593 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa274136530 0x7fa2741358c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa26400e810 tx=0x7fa26400ecd0 comp rx=0 tx=0).stop 2024-04-26T00:43:23.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.647+0000 7fa27a4ba640 1 -- 172.21.15.37:0/3671279593 reap_dead start 2024-04-26T00:43:23.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.647+0000 7fa27bf1d640 1 -- 172.21.15.37:0/3671279593 shutdown_connections 2024-04-26T00:43:23.651 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.647+0000 7fa27bf1d640 1 -- 172.21.15.37:0/3671279593 >> 172.21.15.37:0/3671279593 conn(0x7fa2740a6080 msgr2=0x7fa27409a630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:23.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.647+0000 7fa27bf1d640 1 -- 172.21.15.37:0/3671279593 shutdown_connections 2024-04-26T00:43:23.652 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.647+0000 7fa27bf1d640 1 -- 172.21.15.37:0/3671279593 wait complete. 2024-04-26T00:43:23.665 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-26T00:43:23.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.771+0000 7f9d0d188640 10 monclient: get_monmap_and_config 2024-04-26T00:43:23.774 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.771+0000 7f9d0d188640 10 monclient: build_initial_monmap 2024-04-26T00:43:23.775 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.771+0000 7f9d0d188640 10 monclient: monmap: 2024-04-26T00:43:23.775 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:23.775 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:23.775 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:43:23.775314+0000 2024-04-26T00:43:23.775 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:43:23.775314+0000 2024-04-26T00:43:23.775 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:23.775 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:23.775 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:43:23.776 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:43:23.776 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:43:23.776 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:43:23.776 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:43:23.776 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:23.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d0d188640 1 Processor -- start 2024-04-26T00:43:23.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d0d188640 1 -- start start 2024-04-26T00:43:23.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d0d188640 10 monclient: init 2024-04-26T00:43:23.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d0d188640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:23.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d0d188640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-04-26T00:43:23.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d0d188640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f9d080a3d90 0x7f9d0809bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:23.778 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d0d188640 10 monclient(hunting): picked mon.noname-c con 0x7f9d080a3d90 addr v2:172.21.15.45:3300/0 2024-04-26T00:43:23.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d0d188640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9d080a4170 addr v1:172.21.15.37:6789/0 2024-04-26T00:43:23.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d0d188640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9d0813c3e0 0x7f9d0813e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:23.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d0d188640 10 monclient(hunting): picked mon.noname-a con 0x7f9d0813c3e0 addr v2:172.21.15.37:3300/0 2024-04-26T00:43:23.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d0d188640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d080a5150 con 0x7f9d080a4170 2024-04-26T00:43:23.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d0d188640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:23.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d0d188640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f9d0809b280 con 0x7f9d0813c3e0 2024-04-26T00:43:23.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d0d188640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:23.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d0d188640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f9d0809b560 con 0x7f9d080a3d90 2024-04-26T00:43:23.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d0d188640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:23.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d0d188640 10 monclient(hunting): authenticate will time out at 2385.081461s 2024-04-26T00:43:23.779 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d0659c640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f9d080a4170 0x7f9d080a45e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:49556/0 (socket says 172.21.15.37:49556) 2024-04-26T00:43:23.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d0659c640 1 -- 172.21.15.37:0/454652855 learned_addr learned my addr 172.21.15.37:0/454652855 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:43:23.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d0759e640 1 --2- 172.21.15.37:0/454652855 >> v2:172.21.15.37:3300/0 conn(0x7f9d0813c3e0 0x7f9d0813e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:23.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d0759e640 10 monclient(hunting): get_auth_request con 0x7f9d0813c3e0 auth_method 0 2024-04-26T00:43:23.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d0759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:23.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d0759e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:23.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d0759e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:23.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d0759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:23.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d0759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:23.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d0759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:23.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d06d9d640 1 --2- 172.21.15.37:0/454652855 >> v2:172.21.15.45:3300/0 conn(0x7f9d080a3d90 0x7f9d0809bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:23.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d06d9d640 10 monclient(hunting): get_auth_request con 0x7f9d080a3d90 auth_method 0 2024-04-26T00:43:23.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d06d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:23.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d06d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:23.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d06d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:23.780 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d0759e640 10 monclient(hunting): handle_auth_done global_id 284190 payload 274 2024-04-26T00:43:23.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d0759e640 1 -- 172.21.15.37:0/454652855 >> v2:172.21.15.45:3300/0 conn(0x7f9d080a3d90 msgr2=0x7f9d0809bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:23.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d0759e640 1 --2- 172.21.15.37:0/454652855 >> v2:172.21.15.45:3300/0 conn(0x7f9d080a3d90 0x7f9d0809bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:23.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d0759e640 1 -- 172.21.15.37:0/454652855 >> v1:172.21.15.37:6789/0 conn(0x7f9d080a4170 legacy=0x7f9d080a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:23.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d0759e640 10 monclient: _finish_hunting 0 2024-04-26T00:43:23.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d0759e640 1 monclient: found mon.noname-a 2024-04-26T00:43:23.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d0759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:23.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d0759e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:43:23.781 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.775+0000 7f9d0759e640 1 -- 172.21.15.37:0/454652855 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d0806dc60 con 0x7f9d0813c3e0 2024-04-26T00:43:23.782 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.779+0000 7f9d0759e640 1 -- 172.21.15.37:0/454652855 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d0813c3e0 msgr2=0x7f9d0813e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:23.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.779+0000 7f9d0759e640 1 --2- 172.21.15.37:0/454652855 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d0813c3e0 0x7f9d0813e7d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9d080a57e0 tx=0x7f9cf801c300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5c80634e702b891 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:23.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.779+0000 7f9d05d9b640 1 -- 172.21.15.37:0/454652855 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf80b0070 con 0x7f9d0813c3e0 2024-04-26T00:43:23.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.779+0000 7f9d05d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:23.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.779+0000 7f9d05d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:43:23.783 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.779+0000 7f9d05d9b640 10 monclient: dump: 2024-04-26T00:43:23.783 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:23.783 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:23.783 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:23.783 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:23.783 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:23.783 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:23.783 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:23.783 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:23.784 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:23.784 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:23.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.779+0000 7f9d05d9b640 10 monclient: _finish_auth 0 2024-04-26T00:43:23.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.779+0000 7f9d05d9b640 10 monclient: _check_auth_tickets 2024-04-26T00:43:23.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.779+0000 7f9d05d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:23.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.779+0000 7f9d05d9b640 1 -- 172.21.15.37:0/454652855 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cf80ab600 con 0x7f9d0813c3e0 2024-04-26T00:43:23.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.779+0000 7f9d0d188640 5 monclient: authenticate success, global_id 284190 2024-04-26T00:43:23.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.779+0000 7f9d0d188640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:43:23.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.779+0000 7f9d05d9b640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:23.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.779+0000 7f9d0d188640 10 monclient: get_monmap_and_config success 2024-04-26T00:43:23.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.779+0000 7f9d05d9b640 1 -- 172.21.15.37:0/454652855 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf80ab9d0 con 0x7f9d0813c3e0 2024-04-26T00:43:23.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.779+0000 7f9d0d188640 4 set_mon_vals no callback set 2024-04-26T00:43:23.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.779+0000 7f9d05d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:23.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.779+0000 7f9d05d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:43:23.784 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.779+0000 7f9d05d9b640 10 monclient: dump: 2024-04-26T00:43:23.785 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:23.785 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:23.785 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:23.785 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:23.785 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:23.785 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:23.785 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:23.785 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:23.785 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:23.785 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:23.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.779+0000 7f9d0d188640 10 monclient: shutdown 2024-04-26T00:43:23.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.779+0000 7f9d0d188640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:23.785 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.779+0000 7f9d0d188640 1 -- 172.21.15.37:0/454652855 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d0813c3e0 msgr2=0x7f9d0813e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:23.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.779+0000 7f9d0d188640 1 --2- 172.21.15.37:0/454652855 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d0813c3e0 0x7f9d0813e7d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9d080a57e0 tx=0x7f9cf801c300 comp rx=0 tx=0).stop 2024-04-26T00:43:23.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.779+0000 7f9d0d188640 1 -- 172.21.15.37:0/454652855 shutdown_connections 2024-04-26T00:43:23.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0d188640 1 --2- 172.21.15.37:0/454652855 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d0813c3e0 0x7f9d0813e7d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:23.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0d188640 1 --2- 172.21.15.37:0/454652855 >> v2:172.21.15.45:3300/0 conn(0x7f9d080a3d90 0x7f9d0809bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:23.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0d188640 1 -- 172.21.15.37:0/454652855 >> 172.21.15.37:0/454652855 conn(0x7f9d080aa0a0 msgr2=0x7f9d080a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:23.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0d188640 1 -- 172.21.15.37:0/454652855 shutdown_connections 2024-04-26T00:43:23.786 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0d188640 1 -- 172.21.15.37:0/454652855 wait complete. 2024-04-26T00:43:23.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0d188640 10 monclient: build_initial_monmap 2024-04-26T00:43:23.792 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0d188640 10 monclient: monmap: 2024-04-26T00:43:23.792 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:23.792 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:43:23.793 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:43:23.793 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:43:23.793 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:23.793 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:43:23.793 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:43:23.793 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:43:23.793 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:43:23.793 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:23.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0d188640 1 Processor -- start 2024-04-26T00:43:23.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0d188640 1 -- start start 2024-04-26T00:43:23.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0d188640 10 monclient: init 2024-04-26T00:43:23.793 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0d188640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:23.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0d188640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:43:23.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0d188640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d080a3d90 0x7f9d081cf200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:23.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0d188640 10 monclient(hunting): picked mon.noname-b con 0x7f9d080a3d90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:43:23.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0d188640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d080a4170 0x7f9d081371b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:23.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0d188640 10 monclient(hunting): picked mon.noname-a con 0x7f9d080a4170 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:43:23.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0d188640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d0813c3e0 0x7f9d081376f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:23.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0d188640 10 monclient(hunting): picked mon.noname-c con 0x7f9d0813c3e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:23.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0d188640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:23.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0d188640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9d081418c0 con 0x7f9d080a4170 2024-04-26T00:43:23.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0d188640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:23.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0d188640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9d08141740 con 0x7f9d080a3d90 2024-04-26T00:43:23.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0d188640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:23.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0d188640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9d08141a40 con 0x7f9d0813c3e0 2024-04-26T00:43:23.794 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0d188640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:23.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0d188640 10 monclient(hunting): authenticate will time out at 2385.089503s 2024-04-26T00:43:23.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0659c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d080a4170 0x7f9d081371b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:23.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0659c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d080a4170 0x7f9d081371b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58292/0 (socket says 172.21.15.37:58292) 2024-04-26T00:43:23.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0659c640 1 -- 172.21.15.37:0/3698720096 learned_addr learned my addr 172.21.15.37:0/3698720096 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:23.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0659c640 10 monclient(hunting): get_auth_request con 0x7f9d080a4170 auth_method 0 2024-04-26T00:43:23.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:23.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:23.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:23.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0759e640 1 --2- 172.21.15.37:0/3698720096 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d0813c3e0 0x7f9d081376f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:23.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0759e640 10 monclient(hunting): get_auth_request con 0x7f9d0813c3e0 auth_method 0 2024-04-26T00:43:23.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:23.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0759e640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:23.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0759e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:23.795 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:23.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:23.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:23.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:23.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:23.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:23.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d06d9d640 1 --2- 172.21.15.37:0/3698720096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d080a3d90 0x7f9d081cf200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:23.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0659c640 10 monclient(hunting): handle_auth_done global_id 284193 payload 824 2024-04-26T00:43:23.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0659c640 1 -- 172.21.15.37:0/3698720096 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d0813c3e0 msgr2=0x7f9d081376f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:23.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0659c640 1 --2- 172.21.15.37:0/3698720096 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d0813c3e0 0x7f9d081376f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:23.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0659c640 1 -- 172.21.15.37:0/3698720096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d080a3d90 msgr2=0x7f9d081cf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:23.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0659c640 1 --2- 172.21.15.37:0/3698720096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d080a3d90 0x7f9d081cf200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:23.796 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0659c640 10 monclient: _finish_hunting 0 2024-04-26T00:43:23.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0659c640 1 monclient: found mon.noname-a 2024-04-26T00:43:23.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:23.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:43:23.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d0659c640 1 -- 172.21.15.37:0/3698720096 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d0813c950 con 0x7f9d080a4170 2024-04-26T00:43:23.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d06d9d640 10 monclient: get_auth_request con 0x7f9d080a3d90 auth_method 0 2024-04-26T00:43:23.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.783+0000 7f9d06d9d640 1 --2- 172.21.15.37:0/3698720096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d080a3d90 0x7f9d081cf200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:43:23.797 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d0659c640 1 -- 172.21.15.37:0/3698720096 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d080a4170 msgr2=0x7f9d081371b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:23.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d0659c640 1 --2- 172.21.15.37:0/3698720096 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d080a4170 0x7f9d081371b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc00c470 tx=0x7f9cfc00c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:23.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d0659c640 1 -- 172.21.15.37:0/3698720096 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d080a4170 msgr2=0x7f9d081371b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T00:43:23.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d0659c640 1 -- 172.21.15.37:0/3698720096 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d080a4170 msgr2=0x7f9d081371b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T00:43:23.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d0659c640 1 --2- 172.21.15.37:0/3698720096 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d080a4170 0x7f9d081371b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc00c470 tx=0x7f9cfc00c930 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T00:43:23.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d0659c640 1 --2- 172.21.15.37:0/3698720096 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d080a4170 0x7f9d081371b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc00c470 tx=0x7f9cfc00c930 comp rx=0 tx=0).stop 2024-04-26T00:43:23.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9cef7fe640 10 monclient: ms_handle_reset current mon [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:43:23.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9cef7fe640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:23.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9cef7fe640 1 -- 172.21.15.37:0/3698720096 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d080a4170 msgr2=0x7f9d081371b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T00:43:23.798 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9cef7fe640 1 --2- 172.21.15.37:0/3698720096 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d080a4170 0x7f9d081371b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:23.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9cef7fe640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T00:43:23.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9cef7fe640 1 --2- 172.21.15.37:0/3698720096 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d0813c3e0 0x7f9cf4005510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:23.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9cef7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f9d0813c3e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:23.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9cef7fe640 1 --2- 172.21.15.37:0/3698720096 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9cf4005f20 0x7f9cf40083b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:23.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9cef7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f9cf4005f20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:43:23.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9cef7fe640 1 --2- 172.21.15.37:0/3698720096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d080a3d90 0x7f9cf400af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:23.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9cef7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f9d080a3d90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:43:23.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9cef7fe640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:23.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9cef7fe640 1 -- 172.21.15.37:0/3698720096 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9cf400b8b0 con 0x7f9cf4005f20 2024-04-26T00:43:23.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9cef7fe640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:23.799 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9cef7fe640 1 -- 172.21.15.37:0/3698720096 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9cf400ba60 con 0x7f9d080a3d90 2024-04-26T00:43:23.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9cef7fe640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:23.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9cef7fe640 1 -- 172.21.15.37:0/3698720096 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9cf400bc60 con 0x7f9d0813c3e0 2024-04-26T00:43:23.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9cef7fe640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:23.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d0659c640 1 --2- 172.21.15.37:0/3698720096 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9cf4005f20 0x7f9cf40083b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:23.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d06d9d640 1 --2- 172.21.15.37:0/3698720096 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d0813c3e0 0x7f9cf4005510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:23.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d0659c640 10 monclient(hunting): get_auth_request con 0x7f9cf4005f20 auth_method 0 2024-04-26T00:43:23.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d0659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:23.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d0659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:23.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d0659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:43:23.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d06d9d640 10 monclient(hunting): get_auth_request con 0x7f9d0813c3e0 auth_method 0 2024-04-26T00:43:23.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d06d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:23.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d06d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:23.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d06d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T00:43:23.800 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d0659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:23.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d0659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:23.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d0659c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:43:23.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d06d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:23.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d06d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:23.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d06d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T00:43:23.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d0659c640 10 monclient(hunting): handle_auth_done global_id 284193 payload 293 2024-04-26T00:43:23.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d0659c640 1 -- 172.21.15.37:0/3698720096 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d0813c3e0 msgr2=0x7f9cf4005510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:23.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d0659c640 1 --2- 172.21.15.37:0/3698720096 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d0813c3e0 0x7f9cf4005510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:23.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d0659c640 1 -- 172.21.15.37:0/3698720096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d080a3d90 msgr2=0x7f9cf400af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:23.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d0659c640 1 --2- 172.21.15.37:0/3698720096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d080a3d90 0x7f9cf400af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:23.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d0659c640 10 monclient: _finish_hunting 0 2024-04-26T00:43:23.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d0659c640 1 monclient: found mon.noname-a 2024-04-26T00:43:23.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d0659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:23.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d0659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:43:23.801 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d0659c640 1 -- 172.21.15.37:0/3698720096 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cf400bf20 con 0x7f9cf4005f20 2024-04-26T00:43:23.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d0659c640 1 -- 172.21.15.37:0/3698720096 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9cf4005f20 msgr2=0x7f9cf40083b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:23.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d0659c640 1 --2- 172.21.15.37:0/3698720096 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9cf4005f20 0x7f9cf40083b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc014000 tx=0x7f9cfc004a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:23.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9cef7fe640 1 -- 172.21.15.37:0/3698720096 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc017070 con 0x7f9cf4005f20 2024-04-26T00:43:23.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9cef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:23.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9cef7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:43:23.802 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9cef7fe640 10 monclient: dump: 2024-04-26T00:43:23.802 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:23.802 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:23.802 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:23.802 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:23.803 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:23.803 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:23.803 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:23.803 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:23.803 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:23.804 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:23.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9cef7fe640 10 monclient: _finish_auth 0 2024-04-26T00:43:23.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9cef7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:43:23.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9cef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:23.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9cef7fe640 1 -- 172.21.15.37:0/3698720096 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cfc0072d0 con 0x7f9cf4005f20 2024-04-26T00:43:23.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9cef7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:23.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9cef7fe640 1 -- 172.21.15.37:0/3698720096 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc0076a0 con 0x7f9cf4005f20 2024-04-26T00:43:23.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d0d188640 5 monclient: authenticate success, global_id 284193 2024-04-26T00:43:23.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d0d188640 10 monclient: _renew_subs 2024-04-26T00:43:23.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d0d188640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:43:23.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d0d188640 1 -- 172.21.15.37:0/3698720096 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d0813bc40 con 0x7f9cf4005f20 2024-04-26T00:43:23.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9cef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:23.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9cef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:43:23.804 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9cef7fe640 10 monclient: dump: 2024-04-26T00:43:23.805 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:23.805 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:23.805 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:23.805 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:23.805 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:23.805 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:23.805 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:23.805 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:23.805 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:23.805 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:23.805 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d0559a640 4 set_mon_vals no callback set 2024-04-26T00:43:23.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d0d188640 10 monclient: _renew_subs 2024-04-26T00:43:23.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d0d188640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:43:23.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d0d188640 1 -- 172.21.15.37:0/3698720096 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d08133c80 con 0x7f9cf4005f20 2024-04-26T00:43:23.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9cef7fe640 1 -- 172.21.15.37:0/3698720096 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9cfc029430 con 0x7f9cf4005f20 2024-04-26T00:43:23.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d0d188640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:23.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d0d188640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:23.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d0d188640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:43:23.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.787+0000 7f9d0d188640 1 -- 172.21.15.37:0/3698720096 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d0804f510 con 0x7f9cf4005f20 2024-04-26T00:43:23.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.791+0000 7f9cef7fe640 1 --2- 172.21.15.37:0/3698720096 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9d080a4170 0x7f9cf4040480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:23.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.791+0000 7f9cef7fe640 1 -- 172.21.15.37:0/3698720096 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(899..899 src has 822..899) ==== 3550+0+0 (secure 0 0 0) 0x7f9cfc0494a0 con 0x7f9cf4005f20 2024-04-26T00:43:23.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.791+0000 7f9d06d9d640 1 --2- 172.21.15.37:0/3698720096 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9d080a4170 0x7f9cf4040480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:23.806 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.791+0000 7f9d06d9d640 10 monclient: get_auth_request con 0x7f9d080a4170 auth_method 0 2024-04-26T00:43:23.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.791+0000 7f9d06d9d640 1 --2- 172.21.15.37:0/3698720096 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9d080a4170 0x7f9cf4040480 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f9d00009fd0 tx=0x7f9d0000b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:23.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.795+0000 7f9cef7fe640 1 -- 172.21.15.37:0/3698720096 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9cfc048370 con 0x7f9cf4005f20 2024-04-26T00:43:23.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.795+0000 7f9cef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:23.807 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.795+0000 7f9cef7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:43:23.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.947+0000 7f9d0d188640 1 -- 172.21.15.37:0/3698720096 --> v2:172.21.15.45:6812/2379918623 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f9d08132270 con 0x7f9d080a4170 2024-04-26T00:43:23.948 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.947+0000 7f9cef7fe640 1 -- 172.21.15.37:0/3698720096 <== mgr.4107 v2:172.21.15.45:6812/2379918623 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f9d08132270 con 0x7f9d080a4170 2024-04-26T00:43:23.949 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 4 to deep-scrub 2024-04-26T00:43:23.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.947+0000 7f9d0d188640 1 -- 172.21.15.37:0/3698720096 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9d080a4170 msgr2=0x7f9cf4040480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:23.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.947+0000 7f9d0d188640 1 --2- 172.21.15.37:0/3698720096 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9d080a4170 0x7f9cf4040480 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f9d00009fd0 tx=0x7f9d0000b040 comp rx=0 tx=0).stop 2024-04-26T00:43:23.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.947+0000 7f9d0d188640 10 monclient: shutdown 2024-04-26T00:43:23.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.947+0000 7f9d0d188640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:23.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.947+0000 7f9d0d188640 1 -- 172.21.15.37:0/3698720096 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9cf4005f20 msgr2=0x7f9cf40083b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:23.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.947+0000 7f9d0d188640 1 --2- 172.21.15.37:0/3698720096 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9cf4005f20 0x7f9cf40083b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc014000 tx=0x7f9cfc004a20 comp rx=0 tx=0).stop 2024-04-26T00:43:23.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.947+0000 7f9d0d188640 1 -- 172.21.15.37:0/3698720096 shutdown_connections 2024-04-26T00:43:23.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.947+0000 7f9d0d188640 1 --2- 172.21.15.37:0/3698720096 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9d080a4170 0x7f9cf4040480 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:23.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.947+0000 7f9d0d188640 1 --2- 172.21.15.37:0/3698720096 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9d080a3d90 0x7f9cf400af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:23.951 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.947+0000 7f9d0d188640 1 --2- 172.21.15.37:0/3698720096 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9cf4005f20 0x7f9cf40083b0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:23.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.947+0000 7f9d0d188640 1 --2- 172.21.15.37:0/3698720096 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d0813c3e0 0x7f9cf4005510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:23.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.947+0000 7f9d0d188640 1 -- 172.21.15.37:0/3698720096 >> 172.21.15.37:0/3698720096 conn(0x7f9d080aa0a0 msgr2=0x7f9d080a8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:23.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.947+0000 7f9d0d188640 1 -- 172.21.15.37:0/3698720096 shutdown_connections 2024-04-26T00:43:23.952 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:23.951+0000 7f9d0d188640 1 -- 172.21.15.37:0/3698720096 wait complete. 2024-04-26T00:43:23.966 INFO:tasks.ceph:Scrubbing osd.5 2024-04-26T00:43:23.966 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-26T00:43:24.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20fb4d640 10 monclient: get_monmap_and_config 2024-04-26T00:43:24.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20fb4d640 10 monclient: build_initial_monmap 2024-04-26T00:43:24.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20fb4d640 10 monclient: monmap: 2024-04-26T00:43:24.092 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:24.092 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:24.092 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:43:24.092606+0000 2024-04-26T00:43:24.092 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:43:24.092606+0000 2024-04-26T00:43:24.092 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:24.092 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:24.092 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:43:24.092 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:43:24.092 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:43:24.092 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:43:24.092 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:43:24.092 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:24.094 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20fb4d640 1 Processor -- start 2024-04-26T00:43:24.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20fb4d640 1 -- start start 2024-04-26T00:43:24.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20fb4d640 10 monclient: init 2024-04-26T00:43:24.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20fb4d640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:24.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20fb4d640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-04-26T00:43:24.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20fb4d640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff2080a3ee0 0x7ff20809bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:24.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20fb4d640 10 monclient(hunting): picked mon.noname-a con 0x7ff2080a3ee0 addr v2:172.21.15.37:3300/0 2024-04-26T00:43:24.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20fb4d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff20809c4d0 0x7ff2080a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:24.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20fb4d640 10 monclient(hunting): picked mon.noname-b con 0x7ff20809c4d0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:24.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20fb4d640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7ff2080a4800 0x7ff208133c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:24.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20fb4d640 10 monclient(hunting): picked mon.noname-c con 0x7ff2080a4800 addr v2:172.21.15.45:3300/0 2024-04-26T00:43:24.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20fb4d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:24.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20fb4d640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7ff20809b2d0 con 0x7ff2080a3ee0 2024-04-26T00:43:24.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20fb4d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:24.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20fb4d640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7ff20809b5b0 con 0x7ff2080a4800 2024-04-26T00:43:24.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20fb4d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:24.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20fb4d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff20809b890 con 0x7ff20809c4d0 2024-04-26T00:43:24.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20fb4d640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:24.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20fb4d640 10 monclient(hunting): authenticate will time out at 2385.398278s 2024-04-26T00:43:24.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20d0e8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff20809c4d0 0x7ff2080a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:24.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20d0e8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff20809c4d0 0x7ff2080a42c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51458/0 (socket says 172.21.15.37:51458) 2024-04-26T00:43:24.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20d0e8640 1 -- 172.21.15.37:0/3571152495 learned_addr learned my addr 172.21.15.37:0/3571152495 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:24.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20d8e9640 1 --2- 172.21.15.37:0/3571152495 >> v2:172.21.15.37:3300/0 conn(0x7ff2080a3ee0 0x7ff20809bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:24.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20d0e8640 10 monclient(hunting): get_auth_request con 0x7ff20809c4d0 auth_method 0 2024-04-26T00:43:24.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20d0e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:24.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20d0e8640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:24.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20d0e8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:24.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20d8e9640 10 monclient(hunting): get_auth_request con 0x7ff2080a3ee0 auth_method 0 2024-04-26T00:43:24.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20d8e9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:24.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20d8e9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:24.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20d8e9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:24.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20d8e9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:24.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20d8e9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:24.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20d8e9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:24.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20d0e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:24.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20d0e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:24.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20e0ea640 1 --2- 172.21.15.37:0/3571152495 >> v2:172.21.15.45:3300/0 conn(0x7ff2080a4800 0x7ff208133c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:24.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20d0e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:24.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20e0ea640 10 monclient(hunting): get_auth_request con 0x7ff2080a4800 auth_method 0 2024-04-26T00:43:24.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20e0ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:24.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20e0ea640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:24.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20e0ea640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:24.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20e0ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:24.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20e0ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:24.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20e0ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:24.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20d0e8640 10 monclient(hunting): handle_auth_done global_id 254522 payload 274 2024-04-26T00:43:24.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20d0e8640 1 -- 172.21.15.37:0/3571152495 >> v2:172.21.15.45:3300/0 conn(0x7ff2080a4800 msgr2=0x7ff208133c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:24.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20d0e8640 1 --2- 172.21.15.37:0/3571152495 >> v2:172.21.15.45:3300/0 conn(0x7ff2080a4800 0x7ff208133c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:24.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20d0e8640 1 -- 172.21.15.37:0/3571152495 >> v2:172.21.15.37:3300/0 conn(0x7ff2080a3ee0 msgr2=0x7ff20809bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:24.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20d0e8640 1 --2- 172.21.15.37:0/3571152495 >> v2:172.21.15.37:3300/0 conn(0x7ff2080a3ee0 0x7ff20809bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:24.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20d0e8640 10 monclient: _finish_hunting 0 2024-04-26T00:43:24.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20d0e8640 1 monclient: found mon.noname-b 2024-04-26T00:43:24.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20d0e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:24.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20d0e8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:43:24.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20d0e8640 1 -- 172.21.15.37:0/3571152495 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2080a4cb0 con 0x7ff20809c4d0 2024-04-26T00:43:24.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.091+0000 7ff20d8e9640 1 --2- 172.21.15.37:0/3571152495 >> v2:172.21.15.37:3300/0 conn(0x7ff2080a3ee0 0x7ff20809bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:43:24.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.095+0000 7ff20d0e8640 1 -- 172.21.15.37:0/3571152495 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff20809c4d0 msgr2=0x7ff2080a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:24.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.095+0000 7ff20d0e8640 1 --2- 172.21.15.37:0/3571152495 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff20809c4d0 0x7ff2080a42c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff2080a5550 tx=0x7ff1f801c120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88af564bbbd9ec12 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:24.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.095+0000 7ff20c8e7640 1 -- 172.21.15.37:0/3571152495 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1f80b0070 con 0x7ff20809c4d0 2024-04-26T00:43:24.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.095+0000 7ff20c8e7640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:24.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.095+0000 7ff20c8e7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:43:24.102 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.095+0000 7ff20c8e7640 10 monclient: dump: 2024-04-26T00:43:24.102 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:24.102 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:24.103 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:24.103 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:24.103 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:24.103 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:24.103 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:24.103 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:24.103 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:24.103 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:24.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.095+0000 7ff20c8e7640 10 monclient: _finish_auth 0 2024-04-26T00:43:24.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.095+0000 7ff20c8e7640 10 monclient: _check_auth_tickets 2024-04-26T00:43:24.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.095+0000 7ff20c8e7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:24.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.095+0000 7ff20fb4d640 5 monclient: authenticate success, global_id 254522 2024-04-26T00:43:24.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.095+0000 7ff20fb4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:43:24.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.095+0000 7ff20c8e7640 1 -- 172.21.15.37:0/3571152495 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1f80ab600 con 0x7ff20809c4d0 2024-04-26T00:43:24.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.095+0000 7ff20c8e7640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:24.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.095+0000 7ff20c8e7640 1 -- 172.21.15.37:0/3571152495 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1f80ab9d0 con 0x7ff20809c4d0 2024-04-26T00:43:24.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.095+0000 7ff20c8e7640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:24.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.095+0000 7ff20c8e7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:43:24.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.095+0000 7ff20c8e7640 10 monclient: dump: 2024-04-26T00:43:24.104 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:24.104 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:24.104 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:24.104 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:24.104 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:24.104 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:24.104 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:24.104 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:24.104 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:24.105 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:24.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.095+0000 7ff20fb4d640 10 monclient: get_monmap_and_config success 2024-04-26T00:43:24.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.095+0000 7ff20fb4d640 4 set_mon_vals no callback set 2024-04-26T00:43:24.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.095+0000 7ff20fb4d640 10 monclient: shutdown 2024-04-26T00:43:24.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.095+0000 7ff20fb4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:24.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.095+0000 7ff20fb4d640 1 -- 172.21.15.37:0/3571152495 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff20809c4d0 msgr2=0x7ff2080a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:24.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.095+0000 7ff20fb4d640 1 --2- 172.21.15.37:0/3571152495 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff20809c4d0 0x7ff2080a42c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff2080a5550 tx=0x7ff1f801c120 comp rx=0 tx=0).stop 2024-04-26T00:43:24.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20fb4d640 1 -- 172.21.15.37:0/3571152495 shutdown_connections 2024-04-26T00:43:24.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20fb4d640 1 --2- 172.21.15.37:0/3571152495 >> v2:172.21.15.45:3300/0 conn(0x7ff2080a4800 0x7ff208133c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:24.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20fb4d640 1 --2- 172.21.15.37:0/3571152495 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff20809c4d0 0x7ff2080a42c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:24.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20fb4d640 1 --2- 172.21.15.37:0/3571152495 >> v2:172.21.15.37:3300/0 conn(0x7ff2080a3ee0 0x7ff20809bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:24.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20fb4d640 1 -- 172.21.15.37:0/3571152495 >> 172.21.15.37:0/3571152495 conn(0x7ff2080a5fe0 msgr2=0x7ff2080a63f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:24.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20fb4d640 1 -- 172.21.15.37:0/3571152495 shutdown_connections 2024-04-26T00:43:24.105 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20fb4d640 1 -- 172.21.15.37:0/3571152495 wait complete. 2024-04-26T00:43:24.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20fb4d640 10 monclient: build_initial_monmap 2024-04-26T00:43:24.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20fb4d640 10 monclient: monmap: 2024-04-26T00:43:24.106 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:24.106 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:43:24.106 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:43:24.106 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:43:24.106 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:24.106 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:43:24.106 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:43:24.106 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:43:24.106 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:43:24.106 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:24.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20fb4d640 1 Processor -- start 2024-04-26T00:43:24.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20fb4d640 1 -- start start 2024-04-26T00:43:24.106 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20fb4d640 10 monclient: init 2024-04-26T00:43:24.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20fb4d640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:24.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20fb4d640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:43:24.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20fb4d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff20809c4d0 0x7ff2081cc820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:24.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20fb4d640 10 monclient(hunting): picked mon.noname-b con 0x7ff20809c4d0 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:43:24.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20fb4d640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff2080a3ee0 0x7ff2081ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:24.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20fb4d640 10 monclient(hunting): picked mon.noname-c con 0x7ff2080a3ee0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:24.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20fb4d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff2080a4800 0x7ff2081cf810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:24.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20fb4d640 10 monclient(hunting): picked mon.noname-a con 0x7ff2080a4800 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:43:24.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20fb4d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:24.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20fb4d640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff208140fe0 con 0x7ff2080a4800 2024-04-26T00:43:24.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20fb4d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:24.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20fb4d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff208140e60 con 0x7ff20809c4d0 2024-04-26T00:43:24.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20fb4d640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:24.107 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20fb4d640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff208141160 con 0x7ff2080a3ee0 2024-04-26T00:43:24.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20fb4d640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:24.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20fb4d640 10 monclient(hunting): authenticate will time out at 2385.405663s 2024-04-26T00:43:24.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20e0ea640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff2080a4800 0x7ff2081cf810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:24.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20d0e8640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff2080a3ee0 0x7ff2081ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:24.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20e0ea640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff2080a4800 0x7ff2081cf810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58320/0 (socket says 172.21.15.37:58320) 2024-04-26T00:43:24.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20e0ea640 1 -- 172.21.15.37:0/3607738461 learned_addr learned my addr 172.21.15.37:0/3607738461 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:24.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20e0ea640 10 monclient(hunting): get_auth_request con 0x7ff2080a4800 auth_method 0 2024-04-26T00:43:24.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20e0ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:24.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20e0ea640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:24.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20e0ea640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:24.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20d0e8640 10 monclient(hunting): get_auth_request con 0x7ff2080a3ee0 auth_method 0 2024-04-26T00:43:24.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20d0e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:24.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20d0e8640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:24.108 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20d0e8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:24.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20e0ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:24.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20e0ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:24.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20e0ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:24.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20d0e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:24.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20d0e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:24.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20d0e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:24.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20d8e9640 1 --2- 172.21.15.37:0/3607738461 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff20809c4d0 0x7ff2081cc820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:24.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20d8e9640 10 monclient(hunting): get_auth_request con 0x7ff20809c4d0 auth_method 0 2024-04-26T00:43:24.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20d8e9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:24.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20d8e9640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:24.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20d8e9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:24.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20e0ea640 10 monclient(hunting): handle_auth_done global_id 284199 payload 824 2024-04-26T00:43:24.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20e0ea640 1 -- 172.21.15.37:0/3607738461 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff2080a3ee0 msgr2=0x7ff2081ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:24.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20e0ea640 1 --2- 172.21.15.37:0/3607738461 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff2080a3ee0 0x7ff2081ccd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:24.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20e0ea640 1 -- 172.21.15.37:0/3607738461 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff20809c4d0 msgr2=0x7ff2081cc820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:24.109 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20e0ea640 1 --2- 172.21.15.37:0/3607738461 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff20809c4d0 0x7ff2081cc820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:24.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20e0ea640 10 monclient: _finish_hunting 0 2024-04-26T00:43:24.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20e0ea640 1 monclient: found mon.noname-a 2024-04-26T00:43:24.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20e0ea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:24.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20e0ea640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:43:24.110 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.099+0000 7ff20e0ea640 1 -- 172.21.15.37:0/3607738461 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff20813d380 con 0x7ff2080a4800 2024-04-26T00:43:24.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.103+0000 7ff20e0ea640 1 -- 172.21.15.37:0/3607738461 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff2080a4800 msgr2=0x7ff2081cf810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:24.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.103+0000 7ff20e0ea640 1 --2- 172.21.15.37:0/3607738461 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff2080a4800 0x7ff2081cf810 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff20000efc0 tx=0x7ff20000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:24.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.103+0000 7ff1fe7fc640 1 -- 172.21.15.37:0/3607738461 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff200018020 con 0x7ff2080a4800 2024-04-26T00:43:24.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.103+0000 7ff1fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:24.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.103+0000 7ff1fe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:43:24.112 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.103+0000 7ff1fe7fc640 10 monclient: dump: 2024-04-26T00:43:24.112 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:24.113 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:24.113 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:24.113 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:24.113 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:24.113 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:24.113 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:24.113 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:24.113 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:24.113 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:24.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.103+0000 7ff1fe7fc640 10 monclient: _finish_auth 0 2024-04-26T00:43:24.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.103+0000 7ff1fe7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:43:24.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.103+0000 7ff1fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:24.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.103+0000 7ff1fe7fc640 1 -- 172.21.15.37:0/3607738461 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff200014680 con 0x7ff2080a4800 2024-04-26T00:43:24.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.103+0000 7ff20fb4d640 5 monclient: authenticate success, global_id 284199 2024-04-26T00:43:24.113 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.103+0000 7ff20fb4d640 10 monclient: _renew_subs 2024-04-26T00:43:24.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.103+0000 7ff20fb4d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:43:24.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.103+0000 7ff20fb4d640 1 -- 172.21.15.37:0/3607738461 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff20813b970 con 0x7ff2080a4800 2024-04-26T00:43:24.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.103+0000 7ff1fe7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:24.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.103+0000 7ff1fe7fc640 1 -- 172.21.15.37:0/3607738461 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff200014a70 con 0x7ff2080a4800 2024-04-26T00:43:24.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.103+0000 7ff1fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:24.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.103+0000 7ff1fe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:43:24.114 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.103+0000 7ff1fe7fc640 10 monclient: dump: 2024-04-26T00:43:24.114 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:24.114 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:24.114 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:24.114 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:24.114 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:24.114 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:24.114 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:24.115 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:24.115 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:24.115 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:24.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.103+0000 7ff20fb4d640 10 monclient: _renew_subs 2024-04-26T00:43:24.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.103+0000 7ff1fffff640 4 set_mon_vals no callback set 2024-04-26T00:43:24.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.103+0000 7ff20fb4d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:43:24.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.103+0000 7ff20fb4d640 1 -- 172.21.15.37:0/3607738461 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff20813b090 con 0x7ff2080a4800 2024-04-26T00:43:24.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.103+0000 7ff20fb4d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:43:24.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.103+0000 7ff20fb4d640 1 -- 172.21.15.37:0/3607738461 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff20804f580 con 0x7ff2080a4800 2024-04-26T00:43:24.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.103+0000 7ff1fe7fc640 1 -- 172.21.15.37:0/3607738461 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff20001b020 con 0x7ff2080a4800 2024-04-26T00:43:24.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.103+0000 7ff1fe7fc640 1 --2- 172.21.15.37:0/3607738461 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff1ec035ed0 0x7ff1ec038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:24.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.103+0000 7ff1fe7fc640 1 -- 172.21.15.37:0/3607738461 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(899..899 src has 822..899) ==== 3550+0+0 (secure 0 0 0) 0x7ff200049340 con 0x7ff2080a4800 2024-04-26T00:43:24.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.103+0000 7ff1fe7fc640 1 --2- 172.21.15.37:0/3607738461 >> v2:172.21.15.45:6808/3915746722 conn(0x7ff1ec03ba90 0x7ff1ec03deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:24.115 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.103+0000 7ff1fe7fc640 1 -- 172.21.15.37:0/3607738461 --> v2:172.21.15.45:6808/3915746722 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff1ec03e520 con 0x7ff1ec03ba90 2024-04-26T00:43:24.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.103+0000 7ff1fe7fc640 1 -- 172.21.15.37:0/3607738461 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_get_version_reply(handle=1 version=899) ==== 24+0+0 (secure 0 0 0) 0x7ff200050050 con 0x7ff2080a4800 2024-04-26T00:43:24.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.103+0000 7ff1fe7fc640 10 monclient: handle_get_version_reply finishing 1 version 899 2024-04-26T00:43:24.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.103+0000 7ff20d0e8640 1 --2- 172.21.15.37:0/3607738461 >> v2:172.21.15.45:6808/3915746722 conn(0x7ff1ec03ba90 0x7ff1ec03deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:24.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.103+0000 7ff20d8e9640 1 --2- 172.21.15.37:0/3607738461 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff1ec035ed0 0x7ff1ec038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:24.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.103+0000 7ff20d8e9640 10 monclient: get_auth_request con 0x7ff1ec035ed0 auth_method 0 2024-04-26T00:43:24.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.103+0000 7ff20d0e8640 10 monclient: get_auth_request con 0x7ff1ec03ba90 auth_method 0 2024-04-26T00:43:24.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.107+0000 7ff20d8e9640 1 --2- 172.21.15.37:0/3607738461 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff1ec035ed0 0x7ff1ec038390 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7ff204009580 tx=0x7ff204008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:24.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.107+0000 7ff20d0e8640 1 --2- 172.21.15.37:0/3607738461 >> v2:172.21.15.45:6808/3915746722 conn(0x7ff1ec03ba90 0x7ff1ec03deb0 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:24.116 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.111+0000 7ff1fe7fc640 1 -- 172.21.15.37:0/3607738461 <== osd.5 v2:172.21.15.45:6808/3915746722 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff1ec03e520 con 0x7ff1ec03ba90 2024-04-26T00:43:24.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.123+0000 7ff20fb4d640 1 -- 172.21.15.37:0/3607738461 --> v2:172.21.15.45:6808/3915746722 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ff20813cce0 con 0x7ff1ec03ba90 2024-04-26T00:43:24.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.123+0000 7ff1fe7fc640 1 -- 172.21.15.37:0/3607738461 <== osd.5 v2:172.21.15.45:6808/3915746722 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ff1f80ab400 con 0x7ff1ec03ba90 2024-04-26T00:43:24.126 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T00:43:24.126 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T00:43:24.126 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T00:43:24.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.123+0000 7ff20fb4d640 1 -- 172.21.15.37:0/3607738461 >> v2:172.21.15.45:6808/3915746722 conn(0x7ff1ec03ba90 msgr2=0x7ff1ec03deb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:24.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.123+0000 7ff20fb4d640 1 --2- 172.21.15.37:0/3607738461 >> v2:172.21.15.45:6808/3915746722 conn(0x7ff1ec03ba90 0x7ff1ec03deb0 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:24.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.123+0000 7ff20fb4d640 1 -- 172.21.15.37:0/3607738461 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff1ec035ed0 msgr2=0x7ff1ec038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:24.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.123+0000 7ff20fb4d640 1 --2- 172.21.15.37:0/3607738461 >> v2:172.21.15.45:6812/2379918623 conn(0x7ff1ec035ed0 0x7ff1ec038390 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7ff204009580 tx=0x7ff204008040 comp rx=0 tx=0).stop 2024-04-26T00:43:24.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.123+0000 7ff20fb4d640 10 monclient: shutdown 2024-04-26T00:43:24.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.123+0000 7ff20fb4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:24.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.123+0000 7ff20fb4d640 1 -- 172.21.15.37:0/3607738461 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff2080a4800 msgr2=0x7ff2081cf810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:24.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.123+0000 7ff20fb4d640 1 --2- 172.21.15.37:0/3607738461 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff2080a4800 0x7ff2081cf810 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff20000efc0 tx=0x7ff20000c490 comp rx=0 tx=0).stop 2024-04-26T00:43:24.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.123+0000 7ff20e0ea640 1 -- 172.21.15.37:0/3607738461 reap_dead start 2024-04-26T00:43:24.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.123+0000 7ff20fb4d640 1 -- 172.21.15.37:0/3607738461 shutdown_connections 2024-04-26T00:43:24.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.123+0000 7ff20fb4d640 1 -- 172.21.15.37:0/3607738461 >> 172.21.15.37:0/3607738461 conn(0x7ff2080a5fe0 msgr2=0x7ff2080a2850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:24.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.127+0000 7ff20fb4d640 1 -- 172.21.15.37:0/3607738461 shutdown_connections 2024-04-26T00:43:24.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.127+0000 7ff20fb4d640 1 -- 172.21.15.37:0/3607738461 wait complete. 2024-04-26T00:43:24.141 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-26T00:43:24.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed46b61640 10 monclient: get_monmap_and_config 2024-04-26T00:43:24.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed46b61640 10 monclient: build_initial_monmap 2024-04-26T00:43:24.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed46b61640 10 monclient: monmap: 2024-04-26T00:43:24.255 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:24.255 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:24.255 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:43:24.256146+0000 2024-04-26T00:43:24.255 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:43:24.256146+0000 2024-04-26T00:43:24.255 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:24.255 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:24.256 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:43:24.256 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:43:24.256 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:43:24.256 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:43:24.256 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:43:24.256 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:24.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed46b61640 1 Processor -- start 2024-04-26T00:43:24.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed46b61640 1 -- start start 2024-04-26T00:43:24.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed46b61640 10 monclient: init 2024-04-26T00:43:24.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed46b61640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:24.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed46b61640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-04-26T00:43:24.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed46b61640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7fed40133f70 0x7fed4009c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:24.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed46b61640 10 monclient(hunting): picked mon.noname-c con 0x7fed40133f70 addr v2:172.21.15.45:3300/0 2024-04-26T00:43:24.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed46b61640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fed40134350 addr v1:172.21.15.37:6789/0 2024-04-26T00:43:24.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed46b61640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fed40134790 0x7fed401374b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:24.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed46b61640 10 monclient(hunting): picked mon.noname-a con 0x7fed40134790 addr v2:172.21.15.37:3300/0 2024-04-26T00:43:24.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed46b61640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed400a4fa0 con 0x7fed40134350 2024-04-26T00:43:24.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed46b61640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:24.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed46b61640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fed400a51e0 con 0x7fed40134790 2024-04-26T00:43:24.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed46b61640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:24.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed46b61640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7fed4009b280 con 0x7fed40133f70 2024-04-26T00:43:24.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed46b61640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:24.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed46b61640 10 monclient(hunting): authenticate will time out at 2385.561834s 2024-04-26T00:43:24.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed450fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fed40134790 0x7fed401374b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:24.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed3ffff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fed40134350 0x7fed40136da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:49568/0 (socket says 172.21.15.37:49568) 2024-04-26T00:43:24.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed450fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fed40134790 0x7fed401374b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58328/0 (socket says 172.21.15.37:58328) 2024-04-26T00:43:24.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed3ffff640 1 -- 172.21.15.37:0/2665833723 learned_addr learned my addr 172.21.15.37:0/2665833723 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:43:24.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed450fe640 10 monclient(hunting): get_auth_request con 0x7fed40134790 auth_method 0 2024-04-26T00:43:24.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed450fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:24.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed450fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:24.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed450fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:24.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed450fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:24.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed450fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:24.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed450fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:24.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed448fd640 1 --2- 172.21.15.37:0/2665833723 >> v2:172.21.15.45:3300/0 conn(0x7fed40133f70 0x7fed4009c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:24.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed448fd640 10 monclient(hunting): get_auth_request con 0x7fed40133f70 auth_method 0 2024-04-26T00:43:24.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed448fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:24.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed448fd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:24.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed448fd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:24.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed450fe640 10 monclient(hunting): handle_auth_done global_id 284202 payload 274 2024-04-26T00:43:24.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed450fe640 1 -- 172.21.15.37:0/2665833723 >> v2:172.21.15.45:3300/0 conn(0x7fed40133f70 msgr2=0x7fed4009c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:24.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed450fe640 1 --2- 172.21.15.37:0/2665833723 >> v2:172.21.15.45:3300/0 conn(0x7fed40133f70 0x7fed4009c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:24.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed450fe640 1 -- 172.21.15.37:0/2665833723 >> v1:172.21.15.37:6789/0 conn(0x7fed40134350 legacy=0x7fed40136da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:24.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed450fe640 10 monclient: _finish_hunting 0 2024-04-26T00:43:24.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed450fe640 1 monclient: found mon.noname-a 2024-04-26T00:43:24.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed450fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:24.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed450fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:43:24.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.255+0000 7fed450fe640 1 -- 172.21.15.37:0/2665833723 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed4009c680 con 0x7fed40134790 2024-04-26T00:43:24.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.259+0000 7fed450fe640 1 -- 172.21.15.37:0/2665833723 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed40134790 msgr2=0x7fed401374b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:24.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.259+0000 7fed450fe640 1 --2- 172.21.15.37:0/2665833723 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed40134790 0x7fed401374b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fed400a5870 tx=0x7fed3801c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2f30804f75d94c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:24.262 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.259+0000 7fed3f7fe640 1 -- 172.21.15.37:0/2665833723 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed380ac070 con 0x7fed40134790 2024-04-26T00:43:24.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.259+0000 7fed3f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:24.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.259+0000 7fed3f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:43:24.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.259+0000 7fed3f7fe640 10 monclient: dump: 2024-04-26T00:43:24.263 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:24.263 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:24.263 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:24.263 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:24.263 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:24.263 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:24.263 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:24.263 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:24.263 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:24.263 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:24.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.259+0000 7fed3f7fe640 10 monclient: _finish_auth 0 2024-04-26T00:43:24.263 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.259+0000 7fed3f7fe640 10 monclient: _check_auth_tickets 2024-04-26T00:43:24.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.259+0000 7fed3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:24.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.259+0000 7fed3f7fe640 1 -- 172.21.15.37:0/2665833723 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed380a6680 con 0x7fed40134790 2024-04-26T00:43:24.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.259+0000 7fed46b61640 5 monclient: authenticate success, global_id 284202 2024-04-26T00:43:24.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.259+0000 7fed46b61640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:43:24.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.259+0000 7fed3f7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:24.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.259+0000 7fed46b61640 10 monclient: get_monmap_and_config success 2024-04-26T00:43:24.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.259+0000 7fed46b61640 4 set_mon_vals no callback set 2024-04-26T00:43:24.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.259+0000 7fed3f7fe640 1 -- 172.21.15.37:0/2665833723 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed380a6a50 con 0x7fed40134790 2024-04-26T00:43:24.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.259+0000 7fed3f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:24.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.259+0000 7fed3f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:43:24.264 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.259+0000 7fed3f7fe640 10 monclient: dump: 2024-04-26T00:43:24.264 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:24.264 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:24.264 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:24.264 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:24.265 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:24.265 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:24.265 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:24.265 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:24.265 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:24.265 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:24.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.259+0000 7fed46b61640 10 monclient: shutdown 2024-04-26T00:43:24.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.259+0000 7fed46b61640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:24.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.259+0000 7fed46b61640 1 -- 172.21.15.37:0/2665833723 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed40134790 msgr2=0x7fed401374b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:24.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.259+0000 7fed46b61640 1 --2- 172.21.15.37:0/2665833723 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed40134790 0x7fed401374b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fed400a5870 tx=0x7fed3801c7b0 comp rx=0 tx=0).stop 2024-04-26T00:43:24.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.259+0000 7fed46b61640 1 -- 172.21.15.37:0/2665833723 shutdown_connections 2024-04-26T00:43:24.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.259+0000 7fed46b61640 1 --2- 172.21.15.37:0/2665833723 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed40134790 0x7fed401374b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:24.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.259+0000 7fed46b61640 1 --2- 172.21.15.37:0/2665833723 >> v2:172.21.15.45:3300/0 conn(0x7fed40133f70 0x7fed4009c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:24.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.259+0000 7fed46b61640 1 -- 172.21.15.37:0/2665833723 >> 172.21.15.37:0/2665833723 conn(0x7fed400a5f00 msgr2=0x7fed400a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:24.265 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.259+0000 7fed46b61640 1 -- 172.21.15.37:0/2665833723 shutdown_connections 2024-04-26T00:43:24.266 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.259+0000 7fed46b61640 1 -- 172.21.15.37:0/2665833723 wait complete. 2024-04-26T00:43:24.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed46b61640 10 monclient: build_initial_monmap 2024-04-26T00:43:24.269 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed46b61640 10 monclient: monmap: 2024-04-26T00:43:24.269 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:24.269 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:43:24.269 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:43:24.269 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:43:24.270 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:24.270 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:43:24.270 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:43:24.270 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:43:24.270 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:43:24.270 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:24.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed46b61640 1 Processor -- start 2024-04-26T00:43:24.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed46b61640 1 -- start start 2024-04-26T00:43:24.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed46b61640 10 monclient: init 2024-04-26T00:43:24.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed46b61640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:24.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed46b61640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T00:43:24.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed46b61640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed40133f70 0x7fed401cf380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:24.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed46b61640 10 monclient(hunting): picked mon.noname-c con 0x7fed40133f70 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:24.270 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed46b61640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed40134350 0x7fed4013e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:24.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed46b61640 10 monclient(hunting): picked mon.noname-b con 0x7fed40134350 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:43:24.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed46b61640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed40134790 0x7fed4013e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:24.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed46b61640 10 monclient(hunting): picked mon.noname-a con 0x7fed40134790 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:43:24.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed46b61640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:24.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed46b61640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fed40141b10 con 0x7fed40134790 2024-04-26T00:43:24.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed46b61640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:24.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed46b61640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fed40137c00 con 0x7fed40134350 2024-04-26T00:43:24.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed46b61640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:24.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed46b61640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fed40141c90 con 0x7fed40133f70 2024-04-26T00:43:24.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed46b61640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:24.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed46b61640 10 monclient(hunting): authenticate will time out at 2385.569143s 2024-04-26T00:43:24.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed448fd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed40133f70 0x7fed401cf380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:24.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed450fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed40134790 0x7fed4013e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:24.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed450fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed40134790 0x7fed4013e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:58334/0 (socket says 172.21.15.37:58334) 2024-04-26T00:43:24.271 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed450fe640 1 -- 172.21.15.37:0/4120986752 learned_addr learned my addr 172.21.15.37:0/4120986752 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:24.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed450fe640 10 monclient(hunting): get_auth_request con 0x7fed40134790 auth_method 0 2024-04-26T00:43:24.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed450fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:24.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed450fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:24.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed450fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:24.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed448fd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed40133f70 0x7fed401cf380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51468/0 (socket says 172.21.15.37:51468) 2024-04-26T00:43:24.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed448fd640 10 monclient(hunting): get_auth_request con 0x7fed40133f70 auth_method 0 2024-04-26T00:43:24.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed448fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:24.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed448fd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:24.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed448fd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:24.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed450fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:24.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed450fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:24.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed450fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:24.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed448fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:24.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed448fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:24.272 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed448fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:24.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed3ffff640 1 --2- 172.21.15.37:0/4120986752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed40134350 0x7fed4013e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:24.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed3ffff640 10 monclient(hunting): get_auth_request con 0x7fed40134350 auth_method 0 2024-04-26T00:43:24.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:24.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed3ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:24.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:24.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed450fe640 10 monclient(hunting): handle_auth_done global_id 284205 payload 824 2024-04-26T00:43:24.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed450fe640 1 -- 172.21.15.37:0/4120986752 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed40133f70 msgr2=0x7fed401cf380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:24.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed450fe640 1 --2- 172.21.15.37:0/4120986752 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed40133f70 0x7fed401cf380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:24.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed450fe640 1 -- 172.21.15.37:0/4120986752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed40134350 msgr2=0x7fed4013e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:24.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed450fe640 1 --2- 172.21.15.37:0/4120986752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed40134350 0x7fed4013e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:24.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed450fe640 10 monclient: _finish_hunting 0 2024-04-26T00:43:24.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed450fe640 1 monclient: found mon.noname-a 2024-04-26T00:43:24.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed450fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:24.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed450fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:43:24.273 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.263+0000 7fed450fe640 1 -- 172.21.15.37:0/4120986752 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed4013caf0 con 0x7fed40134790 2024-04-26T00:43:24.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.267+0000 7fed450fe640 1 -- 172.21.15.37:0/4120986752 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed40134790 msgr2=0x7fed4013e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:24.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.267+0000 7fed450fe640 1 --2- 172.21.15.37:0/4120986752 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed40134790 0x7fed4013e840 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fed3801a3e0 tx=0x7fed38019550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:24.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.267+0000 7fed3d7fa640 1 -- 172.21.15.37:0/4120986752 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed380c0070 con 0x7fed40134790 2024-04-26T00:43:24.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.267+0000 7fed3d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:24.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.267+0000 7fed3d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:43:24.274 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.267+0000 7fed3d7fa640 10 monclient: dump: 2024-04-26T00:43:24.274 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:24.274 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:24.274 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:24.274 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:24.274 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:24.274 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:24.274 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:24.274 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:24.274 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:24.275 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:24.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.267+0000 7fed3d7fa640 10 monclient: _finish_auth 0 2024-04-26T00:43:24.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.267+0000 7fed3d7fa640 10 monclient: _check_auth_tickets 2024-04-26T00:43:24.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.267+0000 7fed3d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:24.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.267+0000 7fed3d7fa640 1 -- 172.21.15.37:0/4120986752 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed380a6480 con 0x7fed40134790 2024-04-26T00:43:24.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.267+0000 7fed3d7fa640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:24.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.267+0000 7fed3d7fa640 1 -- 172.21.15.37:0/4120986752 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed380ac050 con 0x7fed40134790 2024-04-26T00:43:24.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.267+0000 7fed46b61640 5 monclient: authenticate success, global_id 284205 2024-04-26T00:43:24.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.267+0000 7fed46b61640 10 monclient: _renew_subs 2024-04-26T00:43:24.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.267+0000 7fed46b61640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:43:24.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.267+0000 7fed46b61640 1 -- 172.21.15.37:0/4120986752 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed4013b640 con 0x7fed40134790 2024-04-26T00:43:24.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.267+0000 7fed3d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:24.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.267+0000 7fed3d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:43:24.275 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.267+0000 7fed3d7fa640 10 monclient: dump: 2024-04-26T00:43:24.275 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:24.276 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:24.276 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:24.276 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:24.276 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:24.276 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:24.276 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:24.276 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:24.276 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:24.276 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:24.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.267+0000 7fed3e7fc640 4 set_mon_vals no callback set 2024-04-26T00:43:24.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.267+0000 7fed46b61640 10 monclient: _renew_subs 2024-04-26T00:43:24.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.267+0000 7fed46b61640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:43:24.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.267+0000 7fed46b61640 1 -- 172.21.15.37:0/4120986752 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed4013ed80 con 0x7fed40134790 2024-04-26T00:43:24.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.267+0000 7fed3d7fa640 1 -- 172.21.15.37:0/4120986752 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fed380af070 con 0x7fed40134790 2024-04-26T00:43:24.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.267+0000 7fed46b61640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:24.276 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.267+0000 7fed46b61640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:24.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.267+0000 7fed46b61640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:43:24.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.267+0000 7fed46b61640 1 -- 172.21.15.37:0/4120986752 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed4004f510 con 0x7fed40134790 2024-04-26T00:43:24.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.267+0000 7fed3d7fa640 1 --2- 172.21.15.37:0/4120986752 >> v2:172.21.15.45:6812/2379918623 conn(0x7fed20035ed0 0x7fed20038390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:24.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.267+0000 7fed3d7fa640 1 -- 172.21.15.37:0/4120986752 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(899..899 src has 822..899) ==== 3550+0+0 (secure 0 0 0) 0x7fed380e2d50 con 0x7fed40134790 2024-04-26T00:43:24.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.267+0000 7fed448fd640 1 --2- 172.21.15.37:0/4120986752 >> v2:172.21.15.45:6812/2379918623 conn(0x7fed20035ed0 0x7fed20038390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:24.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.267+0000 7fed448fd640 10 monclient: get_auth_request con 0x7fed20035ed0 auth_method 0 2024-04-26T00:43:24.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.271+0000 7fed448fd640 1 --2- 172.21.15.37:0/4120986752 >> v2:172.21.15.45:6812/2379918623 conn(0x7fed20035ed0 0x7fed20038390 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fed300079c0 tx=0x7fed30008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:24.277 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.271+0000 7fed3d7fa640 1 -- 172.21.15.37:0/4120986752 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fed380164c0 con 0x7fed40134790 2024-04-26T00:43:24.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.271+0000 7fed3d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:24.278 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.271+0000 7fed3d7fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:43:24.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.435+0000 7fed46b61640 1 -- 172.21.15.37:0/4120986752 --> v2:172.21.15.45:6812/2379918623 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7fed4009d0e0 con 0x7fed20035ed0 2024-04-26T00:43:24.437 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.435+0000 7fed3d7fa640 1 -- 172.21.15.37:0/4120986752 <== mgr.4107 v2:172.21.15.45:6812/2379918623 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fed4009d0e0 con 0x7fed20035ed0 2024-04-26T00:43:24.437 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 5 to deep-scrub 2024-04-26T00:43:24.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.439+0000 7fed46b61640 1 -- 172.21.15.37:0/4120986752 >> v2:172.21.15.45:6812/2379918623 conn(0x7fed20035ed0 msgr2=0x7fed20038390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:24.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.439+0000 7fed46b61640 1 --2- 172.21.15.37:0/4120986752 >> v2:172.21.15.45:6812/2379918623 conn(0x7fed20035ed0 0x7fed20038390 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fed300079c0 tx=0x7fed30008040 comp rx=0 tx=0).stop 2024-04-26T00:43:24.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.439+0000 7fed46b61640 10 monclient: shutdown 2024-04-26T00:43:24.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.439+0000 7fed46b61640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:24.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.439+0000 7fed46b61640 1 -- 172.21.15.37:0/4120986752 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed40134790 msgr2=0x7fed4013e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:24.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.439+0000 7fed46b61640 1 --2- 172.21.15.37:0/4120986752 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed40134790 0x7fed4013e840 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fed3801a3e0 tx=0x7fed38019550 comp rx=0 tx=0).stop 2024-04-26T00:43:24.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.439+0000 7fed46b61640 1 -- 172.21.15.37:0/4120986752 shutdown_connections 2024-04-26T00:43:24.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.439+0000 7fed46b61640 1 --2- 172.21.15.37:0/4120986752 >> v2:172.21.15.45:6812/2379918623 conn(0x7fed20035ed0 0x7fed20038390 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:24.440 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.439+0000 7fed46b61640 1 --2- 172.21.15.37:0/4120986752 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fed40134790 0x7fed4013e840 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:24.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.439+0000 7fed46b61640 1 --2- 172.21.15.37:0/4120986752 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fed40134350 0x7fed4013e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:24.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.439+0000 7fed46b61640 1 --2- 172.21.15.37:0/4120986752 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fed40133f70 0x7fed401cf380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:24.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.439+0000 7fed46b61640 1 -- 172.21.15.37:0/4120986752 >> 172.21.15.37:0/4120986752 conn(0x7fed400a5f00 msgr2=0x7fed40135cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:24.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.439+0000 7fed46b61640 1 -- 172.21.15.37:0/4120986752 shutdown_connections 2024-04-26T00:43:24.441 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.439+0000 7fed46b61640 1 -- 172.21.15.37:0/4120986752 wait complete. 2024-04-26T00:43:24.456 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T00:43:24.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.571+0000 7f94b5f0f640 10 monclient: get_monmap_and_config 2024-04-26T00:43:24.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.571+0000 7f94b5f0f640 10 monclient: build_initial_monmap 2024-04-26T00:43:24.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.571+0000 7f94b5f0f640 10 monclient: monmap: 2024-04-26T00:43:24.573 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:24.574 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:24.574 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:43:24.574419+0000 2024-04-26T00:43:24.574 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:43:24.574419+0000 2024-04-26T00:43:24.574 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:24.574 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:24.574 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:43:24.574 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:43:24.574 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:43:24.574 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:43:24.574 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:43:24.574 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:24.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.571+0000 7f94b5f0f640 1 Processor -- start 2024-04-26T00:43:24.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.571+0000 7f94b5f0f640 1 -- start start 2024-04-26T00:43:24.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.571+0000 7f94b5f0f640 10 monclient: init 2024-04-26T00:43:24.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.575+0000 7f94b5f0f640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:24.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.575+0000 7f94b5f0f640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-04-26T00:43:24.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.575+0000 7f94b5f0f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f94b0136e10 addr v1:172.21.15.37:6789/0 2024-04-26T00:43:24.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.575+0000 7f94b5f0f640 1 --2- >> v2:172.21.15.45:3300/0 conn(0x7f94b0137900 0x7f94b009c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:24.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.575+0000 7f94b5f0f640 10 monclient(hunting): picked mon.noname-c con 0x7f94b0137900 addr v2:172.21.15.45:3300/0 2024-04-26T00:43:24.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.575+0000 7f94b5f0f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f94b013c340 0x7f94b013e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:24.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.575+0000 7f94b5f0f640 10 monclient(hunting): picked mon.noname-b con 0x7f94b013c340 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:24.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.575+0000 7f94b5f0f640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94b009b680 con 0x7f94b0136e10 2024-04-26T00:43:24.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.575+0000 7f94b5f0f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:24.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.575+0000 7f94b5f0f640 1 -- --> v2:172.21.15.45:3300/0 -- mon_getmap magic: 0 -- 0x7f94b009b8f0 con 0x7f94b0137900 2024-04-26T00:43:24.578 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.575+0000 7f94b5f0f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:24.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.575+0000 7f94b5f0f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f94b009bbd0 con 0x7f94b013c340 2024-04-26T00:43:24.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.575+0000 7f94b5f0f640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:24.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.575+0000 7f94b5f0f640 10 monclient(hunting): authenticate will time out at 2385.880222s 2024-04-26T00:43:24.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.575+0000 7f94af7fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f94b0136e10 0x7f94b01371f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:49572/0 (socket says 172.21.15.37:49572) 2024-04-26T00:43:24.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.575+0000 7f94af7fe640 1 -- 172.21.15.37:0/2646346070 learned_addr learned my addr 172.21.15.37:0/2646346070 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:43:24.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.575+0000 7f94affff640 1 --2- 172.21.15.37:0/2646346070 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f94b013c340 0x7f94b013e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:24.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.575+0000 7f94affff640 10 monclient(hunting): get_auth_request con 0x7f94b013c340 auth_method 0 2024-04-26T00:43:24.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.575+0000 7f94affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:24.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.575+0000 7f94affff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:24.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.575+0000 7f94affff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:24.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.575+0000 7f94affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:24.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.575+0000 7f94affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:24.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.575+0000 7f94affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:24.579 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.575+0000 7f94affff640 10 monclient(hunting): handle_auth_done global_id 254531 payload 274 2024-04-26T00:43:24.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.575+0000 7f94aeffd640 1 --2- 172.21.15.37:0/2646346070 >> v2:172.21.15.45:3300/0 conn(0x7f94b0137900 0x7f94b009c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:24.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.575+0000 7f94affff640 1 -- 172.21.15.37:0/2646346070 >> v2:172.21.15.45:3300/0 conn(0x7f94b0137900 msgr2=0x7f94b009c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:24.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.575+0000 7f94affff640 1 --2- 172.21.15.37:0/2646346070 >> v2:172.21.15.45:3300/0 conn(0x7f94b0137900 0x7f94b009c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:24.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.575+0000 7f94affff640 1 -- 172.21.15.37:0/2646346070 >> v1:172.21.15.37:6789/0 conn(0x7f94b0136e10 legacy=0x7f94b01371f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:24.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.575+0000 7f94affff640 10 monclient: _finish_hunting 0 2024-04-26T00:43:24.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.575+0000 7f94affff640 1 monclient: found mon.noname-b 2024-04-26T00:43:24.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.575+0000 7f94affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:24.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.575+0000 7f94affff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:43:24.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.575+0000 7f94affff640 1 -- 172.21.15.37:0/2646346070 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94b00a5d90 con 0x7f94b013c340 2024-04-26T00:43:24.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.575+0000 7f94aeffd640 10 monclient: get_auth_request con 0x7f94b0137900 auth_method 0 2024-04-26T00:43:24.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.575+0000 7f94aeffd640 1 --2- 172.21.15.37:0/2646346070 >> v2:172.21.15.45:3300/0 conn(0x7f94b0137900 0x7f94b009c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:43:24.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94affff640 1 -- 172.21.15.37:0/2646346070 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f94b013c340 msgr2=0x7f94b013e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:24.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94affff640 1 --2- 172.21.15.37:0/2646346070 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f94b013c340 0x7f94b013e730 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f94a8009d90 tx=0x7f94a801c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cc35ca008df67e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:24.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94ae7fc640 1 -- 172.21.15.37:0/2646346070 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a80ac070 con 0x7f94b013c340 2024-04-26T00:43:24.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:24.584 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94ae7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:43:24.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94ae7fc640 10 monclient: dump: 2024-04-26T00:43:24.585 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:24.585 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:24.585 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:24.585 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:24.585 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:24.585 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:24.585 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:24.585 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:24.585 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:24.585 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:24.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94ae7fc640 10 monclient: _finish_auth 0 2024-04-26T00:43:24.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94ae7fc640 10 monclient: _check_auth_tickets 2024-04-26T00:43:24.585 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:24.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94ae7fc640 1 -- 172.21.15.37:0/2646346070 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94a80a6760 con 0x7f94b013c340 2024-04-26T00:43:24.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94ae7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:24.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94ae7fc640 1 -- 172.21.15.37:0/2646346070 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a80a6b30 con 0x7f94b013c340 2024-04-26T00:43:24.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94b5f0f640 5 monclient: authenticate success, global_id 254531 2024-04-26T00:43:24.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94b5f0f640 10 monclient: get_monmap_and_config success 2024-04-26T00:43:24.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94b5f0f640 4 set_mon_vals no callback set 2024-04-26T00:43:24.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:24.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94ae7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:43:24.586 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94ae7fc640 10 monclient: dump: 2024-04-26T00:43:24.586 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:24.586 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:24.586 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:24.587 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:24.587 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:24.587 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:24.587 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:24.587 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:24.587 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:24.587 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:24.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94b5f0f640 10 monclient: shutdown 2024-04-26T00:43:24.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94b5f0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:24.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94b5f0f640 1 -- 172.21.15.37:0/2646346070 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f94b013c340 msgr2=0x7f94b013e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:24.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94b5f0f640 1 --2- 172.21.15.37:0/2646346070 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f94b013c340 0x7f94b013e730 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f94a8009d90 tx=0x7f94a801c7f0 comp rx=0 tx=0).stop 2024-04-26T00:43:24.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94b5f0f640 1 -- 172.21.15.37:0/2646346070 shutdown_connections 2024-04-26T00:43:24.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94b5f0f640 1 --2- 172.21.15.37:0/2646346070 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f94b013c340 0x7f94b013e730 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:24.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94b5f0f640 1 --2- 172.21.15.37:0/2646346070 >> v2:172.21.15.45:3300/0 conn(0x7f94b0137900 0x7f94b009c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:24.587 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94b5f0f640 1 -- 172.21.15.37:0/2646346070 >> 172.21.15.37:0/2646346070 conn(0x7f94b00a64e0 msgr2=0x7f94b00a68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:24.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94b5f0f640 1 -- 172.21.15.37:0/2646346070 shutdown_connections 2024-04-26T00:43:24.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94b5f0f640 1 -- 172.21.15.37:0/2646346070 wait complete. 2024-04-26T00:43:24.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94b5f0f640 10 monclient: build_initial_monmap 2024-04-26T00:43:24.588 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94b5f0f640 10 monclient: monmap: 2024-04-26T00:43:24.588 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:24.588 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:43:24.588 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:43:24.588 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:43:24.588 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:24.588 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:43:24.588 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:43:24.588 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:43:24.588 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:43:24.588 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:24.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94b5f0f640 1 Processor -- start 2024-04-26T00:43:24.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94b5f0f640 1 -- start start 2024-04-26T00:43:24.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94b5f0f640 10 monclient: init 2024-04-26T00:43:24.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94b5f0f640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:24.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94b5f0f640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T00:43:24.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94b5f0f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94b0136e10 0x7f94b01cef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:24.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94b5f0f640 10 monclient(hunting): picked mon.noname-b con 0x7f94b0136e10 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:43:24.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94b5f0f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f94b0137900 0x7f94b013b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:24.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94b5f0f640 10 monclient(hunting): picked mon.noname-c con 0x7f94b0137900 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:24.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94b5f0f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f94b013c340 0x7f94b013b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:24.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94b5f0f640 10 monclient(hunting): picked mon.noname-a con 0x7f94b013c340 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:43:24.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94b5f0f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:24.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94b5f0f640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f94b0141820 con 0x7f94b013c340 2024-04-26T00:43:24.589 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94b5f0f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:24.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94b5f0f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f94b01416a0 con 0x7f94b0136e10 2024-04-26T00:43:24.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94b5f0f640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:24.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94b5f0f640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f94b01419a0 con 0x7f94b0137900 2024-04-26T00:43:24.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94b5f0f640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:24.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94b5f0f640 10 monclient(hunting): authenticate will time out at 2385.887565s 2024-04-26T00:43:24.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94aeffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f94b0137900 0x7f94b013b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:24.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94aeffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f94b0137900 0x7f94b013b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:51496/0 (socket says 172.21.15.37:51496) 2024-04-26T00:43:24.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94aeffd640 1 -- 172.21.15.37:0/3329139567 learned_addr learned my addr 172.21.15.37:0/3329139567 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:24.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94aeffd640 10 monclient(hunting): get_auth_request con 0x7f94b0137900 auth_method 0 2024-04-26T00:43:24.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:24.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94aeffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:24.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94affff640 1 --2- 172.21.15.37:0/3329139567 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f94b013c340 0x7f94b013b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:24.590 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:24.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94affff640 10 monclient(hunting): get_auth_request con 0x7f94b013c340 auth_method 0 2024-04-26T00:43:24.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:24.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94affff640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:24.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94affff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:24.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:24.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:24.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:24.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:24.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:24.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:24.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94af7fe640 1 --2- 172.21.15.37:0/3329139567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94b0136e10 0x7f94b01cef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:24.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94af7fe640 10 monclient(hunting): get_auth_request con 0x7f94b0136e10 auth_method 0 2024-04-26T00:43:24.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:24.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94af7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:24.591 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.579+0000 7f94af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:24.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.583+0000 7f94aeffd640 10 monclient(hunting): handle_auth_done global_id 254534 payload 824 2024-04-26T00:43:24.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.583+0000 7f94aeffd640 1 -- 172.21.15.37:0/3329139567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94b0136e10 msgr2=0x7f94b01cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:24.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.583+0000 7f94aeffd640 1 --2- 172.21.15.37:0/3329139567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94b0136e10 0x7f94b01cef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:24.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.583+0000 7f94aeffd640 1 -- 172.21.15.37:0/3329139567 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f94b013c340 msgr2=0x7f94b013b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:24.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.583+0000 7f94aeffd640 1 --2- 172.21.15.37:0/3329139567 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f94b013c340 0x7f94b013b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:24.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.583+0000 7f94aeffd640 10 monclient: _finish_hunting 0 2024-04-26T00:43:24.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.583+0000 7f94aeffd640 1 monclient: found mon.noname-c 2024-04-26T00:43:24.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.583+0000 7f94aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:24.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.583+0000 7f94aeffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:43:24.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.583+0000 7f94aeffd640 1 -- 172.21.15.37:0/3329139567 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94b00a84a0 con 0x7f94b0137900 2024-04-26T00:43:24.592 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.583+0000 7f94affff640 1 --2- 172.21.15.37:0/3329139567 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f94b013c340 0x7f94b013b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:43:24.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.583+0000 7f94aeffd640 1 -- 172.21.15.37:0/3329139567 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f94b0137900 msgr2=0x7f94b013b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:24.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.583+0000 7f94aeffd640 1 --2- 172.21.15.37:0/3329139567 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f94b0137900 0x7f94b013b130 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f949800c7c0 tx=0x7f949800cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:24.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.583+0000 7f94a7fff640 1 -- 172.21.15.37:0/3329139567 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9498007bc0 con 0x7f94b0137900 2024-04-26T00:43:24.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.583+0000 7f94a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:24.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.583+0000 7f94a7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:43:24.594 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.583+0000 7f94a7fff640 10 monclient: dump: 2024-04-26T00:43:24.594 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:24.594 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:24.594 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:24.595 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:24.595 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:24.595 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:24.595 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:24.595 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:24.595 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:24.595 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:24.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.583+0000 7f94a7fff640 10 monclient: _finish_auth 0 2024-04-26T00:43:24.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.583+0000 7f94a7fff640 10 monclient: _check_auth_tickets 2024-04-26T00:43:24.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.583+0000 7f94a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:24.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.583+0000 7f94b5f0f640 5 monclient: authenticate success, global_id 254534 2024-04-26T00:43:24.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.583+0000 7f94a7fff640 1 -- 172.21.15.37:0/3329139567 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9498007d60 con 0x7f94b0137900 2024-04-26T00:43:24.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.583+0000 7f94b5f0f640 10 monclient: _renew_subs 2024-04-26T00:43:24.595 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.583+0000 7f94b5f0f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:24.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.583+0000 7f94b5f0f640 1 -- 172.21.15.37:0/3329139567 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94b01399d0 con 0x7f94b0137900 2024-04-26T00:43:24.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.583+0000 7f94a7fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:24.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.583+0000 7f94a7fff640 1 -- 172.21.15.37:0/3329139567 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9498011630 con 0x7f94b0137900 2024-04-26T00:43:24.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.583+0000 7f94a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:24.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.583+0000 7f94a7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:43:24.596 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.583+0000 7f94a7fff640 10 monclient: dump: 2024-04-26T00:43:24.596 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:24.596 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:24.596 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:24.596 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:24.596 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:24.597 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:24.597 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:24.597 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:24.597 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:24.597 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:24.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.583+0000 7f94adffb640 4 set_mon_vals no callback set 2024-04-26T00:43:24.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.583+0000 7f94b5f0f640 10 monclient: _renew_subs 2024-04-26T00:43:24.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.583+0000 7f94b5f0f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:24.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.583+0000 7f94b5f0f640 1 -- 172.21.15.37:0/3329139567 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f94b0139130 con 0x7f94b0137900 2024-04-26T00:43:24.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.583+0000 7f94b5f0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:24.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.583+0000 7f94b5f0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:24.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.583+0000 7f94b5f0f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:24.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.583+0000 7f94b5f0f640 1 -- 172.21.15.37:0/3329139567 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94b0075910 con 0x7f94b0137900 2024-04-26T00:43:24.597 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.583+0000 7f94a7fff640 1 -- 172.21.15.37:0/3329139567 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f94980117d0 con 0x7f94b0137900 2024-04-26T00:43:24.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.583+0000 7f94a7fff640 1 --2- 172.21.15.37:0/3329139567 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9490035e80 0x7f9490038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:24.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.587+0000 7f94a7fff640 1 -- 172.21.15.37:0/3329139567 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(899..899 src has 822..899) ==== 3550+0+0 (secure 0 0 0) 0x7f9498048b50 con 0x7f94b0137900 2024-04-26T00:43:24.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.587+0000 7f94af7fe640 1 --2- 172.21.15.37:0/3329139567 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9490035e80 0x7f9490038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:24.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.587+0000 7f94af7fe640 10 monclient: get_auth_request con 0x7f9490035e80 auth_method 0 2024-04-26T00:43:24.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.587+0000 7f94af7fe640 1 --2- 172.21.15.37:0/3329139567 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9490035e80 0x7f9490038340 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f94b00a3270 tx=0x7f94a0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:24.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.591+0000 7f94a7fff640 1 -- 172.21.15.37:0/3329139567 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f94b00a84a0 con 0x7f94b0137900 2024-04-26T00:43:24.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.591+0000 7f94a7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:24.598 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.591+0000 7f94a7fff640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:43:24.750 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.747+0000 7f94b5f0f640 1 -- 172.21.15.37:0/3329139567 --> v2:172.21.15.45:6812/2379918623 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f94b0066f40 con 0x7f9490035e80 2024-04-26T00:43:24.753 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.751+0000 7f94a7fff640 1 -- 172.21.15.37:0/3329139567 <== mgr.4107 v2:172.21.15.45:6812/2379918623 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33844 (secure 0 0 0) 0x7f94b0066f40 con 0x7f9490035e80 2024-04-26T00:43:24.754 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:43:24.754 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-04-26T00:43:24.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.755+0000 7f94b5f0f640 1 -- 172.21.15.37:0/3329139567 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9490035e80 msgr2=0x7f9490038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:24.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.755+0000 7f94b5f0f640 1 --2- 172.21.15.37:0/3329139567 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9490035e80 0x7f9490038340 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f94b00a3270 tx=0x7f94a0009040 comp rx=0 tx=0).stop 2024-04-26T00:43:24.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.755+0000 7f94b5f0f640 10 monclient: shutdown 2024-04-26T00:43:24.755 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.755+0000 7f94b5f0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:24.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.755+0000 7f94b5f0f640 1 -- 172.21.15.37:0/3329139567 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f94b0137900 msgr2=0x7f94b013b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:24.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.755+0000 7f94b5f0f640 1 --2- 172.21.15.37:0/3329139567 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f94b0137900 0x7f94b013b130 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f949800c7c0 tx=0x7f949800cc80 comp rx=0 tx=0).stop 2024-04-26T00:43:24.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.755+0000 7f94b5f0f640 1 -- 172.21.15.37:0/3329139567 shutdown_connections 2024-04-26T00:43:24.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.755+0000 7f94b5f0f640 1 --2- 172.21.15.37:0/3329139567 >> v2:172.21.15.45:6812/2379918623 conn(0x7f9490035e80 0x7f9490038340 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:24.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.755+0000 7f94b5f0f640 1 --2- 172.21.15.37:0/3329139567 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f94b013c340 0x7f94b013b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:24.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.755+0000 7f94b5f0f640 1 --2- 172.21.15.37:0/3329139567 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f94b0137900 0x7f94b013b130 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:24.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.755+0000 7f94b5f0f640 1 --2- 172.21.15.37:0/3329139567 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94b0136e10 0x7f94b01cef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:24.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.755+0000 7f94b5f0f640 1 -- 172.21.15.37:0/3329139567 >> 172.21.15.37:0/3329139567 conn(0x7f94b00a64e0 msgr2=0x7f94b013d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:24.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.755+0000 7f94b5f0f640 1 -- 172.21.15.37:0/3329139567 shutdown_connections 2024-04-26T00:43:24.756 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:24.755+0000 7f94b5f0f640 1 -- 172.21.15.37:0/3329139567 wait complete. 2024-04-26T00:43:24.772 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":1585,"stamp":"2024-04-26T00:43:23.553871+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":346348,"kb_used_data":3936,"kb_used_omap":9,"kb_used_meta":342326,"kb_avail":565884692,"statfs":{"total":579820584960,"available":579465924608,"internally_reserved":0,"allocated":4030464,"data_stored":3181916,"data_compressed":61234,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":350542496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003895"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":1755,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:42:43.917048+0000","last_change":"2024-04-26T00:20:02.279764+0000","last_active":"2024-04-26T00:42:43.917048+0000","last_peered":"2024-04-26T00:42:43.917048+0000","last_clean":"2024-04-26T00:42:43.917048+0000","last_became_active":"2024-04-26T00:20:02.278959+0000","last_became_peered":"2024-04-26T00:20:02.278959+0000","last_unstale":"2024-04-26T00:42:43.917048+0000","last_undegraded":"2024-04-26T00:42:43.917048+0000","last_fullsized":"2024-04-26T00:42:43.917048+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:23:43.610174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":1750,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:42:43.753251+0000","last_change":"2024-04-26T00:20:02.279805+0000","last_active":"2024-04-26T00:42:43.753251+0000","last_peered":"2024-04-26T00:42:43.753251+0000","last_clean":"2024-04-26T00:42:43.753251+0000","last_became_active":"2024-04-26T00:20:02.279053+0000","last_became_peered":"2024-04-26T00:20:02.279053+0000","last_unstale":"2024-04-26T00:42:43.753251+0000","last_undegraded":"2024-04-26T00:42:43.753251+0000","last_fullsized":"2024-04-26T00:42:43.753251+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:11:26.730719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":1761,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:42:43.918127+0000","last_change":"2024-04-26T00:20:02.277311+0000","last_active":"2024-04-26T00:42:43.918127+0000","last_peered":"2024-04-26T00:42:43.918127+0000","last_clean":"2024-04-26T00:42:43.918127+0000","last_became_active":"2024-04-26T00:20:02.276409+0000","last_became_peered":"2024-04-26T00:20:02.276409+0000","last_unstale":"2024-04-26T00:42:43.918127+0000","last_undegraded":"2024-04-26T00:42:43.918127+0000","last_fullsized":"2024-04-26T00:42:43.918127+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:14:50.584720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":1763,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:42:43.918487+0000","last_change":"2024-04-26T00:20:02.273965+0000","last_active":"2024-04-26T00:42:43.918487+0000","last_peered":"2024-04-26T00:42:43.918487+0000","last_clean":"2024-04-26T00:42:43.918487+0000","last_became_active":"2024-04-26T00:20:02.273213+0000","last_became_peered":"2024-04-26T00:20:02.273213+0000","last_unstale":"2024-04-26T00:42:43.918487+0000","last_undegraded":"2024-04-26T00:42:43.918487+0000","last_fullsized":"2024-04-26T00:42:43.918487+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:42:14.055403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"15'117","reported_seq":1905,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:42:43.916767+0000","last_change":"2024-04-26T00:20:02.281782+0000","last_active":"2024-04-26T00:42:43.916767+0000","last_peered":"2024-04-26T00:42:43.916767+0000","last_clean":"2024-04-26T00:42:43.916767+0000","last_became_active":"2024-04-26T00:20:02.281000+0000","last_became_peered":"2024-04-26T00:20:02.281000+0000","last_unstale":"2024-04-26T00:42:43.916767+0000","last_undegraded":"2024-04-26T00:42:43.916767+0000","last_fullsized":"2024-04-26T00:42:43.916767+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:40:30.790500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":1763,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:42:43.920813+0000","last_change":"2024-04-26T00:20:02.274249+0000","last_active":"2024-04-26T00:42:43.920813+0000","last_peered":"2024-04-26T00:42:43.920813+0000","last_clean":"2024-04-26T00:42:43.920813+0000","last_became_active":"2024-04-26T00:20:02.273745+0000","last_became_peered":"2024-04-26T00:20:02.273745+0000","last_unstale":"2024-04-26T00:42:43.920813+0000","last_undegraded":"2024-04-26T00:42:43.920813+0000","last_fullsized":"2024-04-26T00:42:43.920813+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:36:25.948539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":1761,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:42:43.918336+0000","last_change":"2024-04-26T00:20:02.474779+0000","last_active":"2024-04-26T00:42:43.918336+0000","last_peered":"2024-04-26T00:42:43.918336+0000","last_clean":"2024-04-26T00:42:43.918336+0000","last_became_active":"2024-04-26T00:20:02.473888+0000","last_became_peered":"2024-04-26T00:20:02.473888+0000","last_unstale":"2024-04-26T00:42:43.918336+0000","last_undegraded":"2024-04-26T00:42:43.918336+0000","last_fullsized":"2024-04-26T00:42:43.918336+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:33:18.962907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":1754,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:42:43.918328+0000","last_change":"2024-04-26T00:20:02.279987+0000","last_active":"2024-04-26T00:42:43.918328+0000","last_peered":"2024-04-26T00:42:43.918328+0000","last_clean":"2024-04-26T00:42:43.918328+0000","last_became_active":"2024-04-26T00:20:02.279183+0000","last_became_peered":"2024-04-26T00:20:02.279183+0000","last_unstale":"2024-04-26T00:42:43.918328+0000","last_undegraded":"2024-04-26T00:42:43.918328+0000","last_fullsized":"2024-04-26T00:42:43.918328+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:45:44.508952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":1749,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:42:43.754128+0000","last_change":"2024-04-26T00:20:02.475146+0000","last_active":"2024-04-26T00:42:43.754128+0000","last_peered":"2024-04-26T00:42:43.754128+0000","last_clean":"2024-04-26T00:42:43.754128+0000","last_became_active":"2024-04-26T00:20:02.473928+0000","last_became_peered":"2024-04-26T00:20:02.473928+0000","last_unstale":"2024-04-26T00:42:43.754128+0000","last_undegraded":"2024-04-26T00:42:43.754128+0000","last_fullsized":"2024-04-26T00:42:43.754128+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:20:01.249484+0000","last_clean_scrub_stamp":"2024-04-26T00:20:01.249484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:25:12.904295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":61234,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640539,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58184,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313656,"statfs":{"total":96636764160,"available":96577183744,"internally_reserved":0,"allocated":651264,"data_stored":354010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640538,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49864,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321976,"statfs":{"total":96636764160,"available":96585703424,"internally_reserved":0,"allocated":651264,"data_stored":354010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640540,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40200,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331640,"statfs":{"total":96636764160,"available":96595599360,"internally_reserved":0,"allocated":712704,"data_stored":882938,"data_compressed":30617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640540,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72008,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299832,"statfs":{"total":96636764160,"available":96563027968,"internally_reserved":0,"allocated":651264,"data_stored":354010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640540,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72068,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299772,"statfs":{"total":96636764160,"available":96562966528,"internally_reserved":0,"allocated":712704,"data_stored":882938,"data_compressed":30617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640539,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54024,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317816,"statfs":{"total":96636764160,"available":96581443584,"internally_reserved":0,"allocated":651264,"data_stored":354010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T00:43:24.776 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-26T00:20:01.249484+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=0, tm_min=20, tm_sec=1, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=0, tm_min=43, tm_sec=20, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T00:43:24.776 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2024-04-26T00:20:01.249484+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=0, tm_min=20, tm_sec=1, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=0, tm_min=43, tm_sec=20, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T00:43:24.776 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2024-04-26T00:20:01.249484+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=0, tm_min=20, tm_sec=1, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=0, tm_min=43, tm_sec=20, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T00:43:24.777 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2024-04-26T00:20:01.249484+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=0, tm_min=20, tm_sec=1, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=0, tm_min=43, tm_sec=20, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T00:43:24.777 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-26T00:20:01.249484+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=0, tm_min=20, tm_sec=1, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=0, tm_min=43, tm_sec=20, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T00:43:24.777 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2024-04-26T00:20:01.249484+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=0, tm_min=20, tm_sec=1, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=0, tm_min=43, tm_sec=20, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T00:43:24.777 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2024-04-26T00:20:01.249484+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=0, tm_min=20, tm_sec=1, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=0, tm_min=43, tm_sec=20, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T00:43:24.777 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2024-04-26T00:20:01.249484+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=0, tm_min=20, tm_sec=1, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=0, tm_min=43, tm_sec=20, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T00:43:24.777 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2024-04-26T00:20:01.249484+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=0, tm_min=20, tm_sec=1, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=0, tm_min=43, tm_sec=20, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T00:43:24.777 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-26T00:43:44.779 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T00:43:44.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.911+0000 7f7e277fe640 10 monclient: get_monmap_and_config 2024-04-26T00:43:44.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.911+0000 7f7e277fe640 10 monclient: build_initial_monmap 2024-04-26T00:43:44.914 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.911+0000 7f7e277fe640 10 monclient: monmap: 2024-04-26T00:43:44.914 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:44.914 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:44.914 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:43:44.914755+0000 2024-04-26T00:43:44.914 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:43:44.914755+0000 2024-04-26T00:43:44.914 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:44.914 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:44.915 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:43:44.915 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:43:44.915 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:43:44.915 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:43:44.915 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:43:44.915 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:44.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.911+0000 7f7e277fe640 1 Processor -- start 2024-04-26T00:43:44.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.911+0000 7f7e277fe640 1 -- start start 2024-04-26T00:43:44.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.911+0000 7f7e277fe640 10 monclient: init 2024-04-26T00:43:44.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.915+0000 7f7e277fe640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:44.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.915+0000 7f7e277fe640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-04-26T00:43:44.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.915+0000 7f7e277fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7e20133b20 addr v1:172.21.15.37:6789/0 2024-04-26T00:43:44.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.915+0000 7f7e277fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7e20133f00 0x7f7e20134350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:44.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.915+0000 7f7e277fe640 10 monclient(hunting): picked mon.noname-b con 0x7f7e20133f00 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:44.917 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.915+0000 7f7e277fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7e20139050 addr v1:172.21.15.45:6789/0 2024-04-26T00:43:44.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.915+0000 7f7e277fe640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e2009fe20 con 0x7f7e20133b20 2024-04-26T00:43:44.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.915+0000 7f7e277fe640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e20072d00 con 0x7f7e20139050 2024-04-26T00:43:44.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.915+0000 7f7e277fe640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:44.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.915+0000 7f7e277fe640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f7e2009e620 con 0x7f7e20133f00 2024-04-26T00:43:44.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.915+0000 7f7e277fe640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:44.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.915+0000 7f7e277fe640 10 monclient(hunting): authenticate will time out at 2406.220453s 2024-04-26T00:43:44.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.915+0000 7f7e2559a640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f7e20133b20 0x7f7e2009e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:48904/0 (socket says 172.21.15.37:48904) 2024-04-26T00:43:44.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.915+0000 7f7e2559a640 1 -- 172.21.15.37:0/2867057108 learned_addr learned my addr 172.21.15.37:0/2867057108 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-26T00:43:44.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.915+0000 7f7e24d99640 1 --2- 172.21.15.37:0/2867057108 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7e20133f00 0x7f7e20134350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:44.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.915+0000 7f7e24d99640 10 monclient(hunting): get_auth_request con 0x7f7e20133f00 auth_method 0 2024-04-26T00:43:44.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.915+0000 7f7e24d99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:44.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.915+0000 7f7e24d99640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:44.918 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.915+0000 7f7e24d99640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:44.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.915+0000 7f7e24d99640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:44.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.915+0000 7f7e24d99640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:44.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.915+0000 7f7e24d99640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:44.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.915+0000 7f7e24d99640 10 monclient(hunting): handle_auth_done global_id 254537 payload 274 2024-04-26T00:43:44.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.915+0000 7f7e24d99640 1 -- 172.21.15.37:0/2867057108 >> v1:172.21.15.45:6789/0 conn(0x7f7e20139050 legacy=0x7f7e2013b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:44.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.915+0000 7f7e24d99640 1 -- 172.21.15.37:0/2867057108 >> v1:172.21.15.37:6789/0 conn(0x7f7e20133b20 legacy=0x7f7e2009e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:44.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.915+0000 7f7e24d99640 10 monclient: _finish_hunting 0 2024-04-26T00:43:44.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.915+0000 7f7e24d99640 1 monclient: found mon.noname-b 2024-04-26T00:43:44.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.915+0000 7f7e24d99640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:44.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.915+0000 7f7e24d99640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.37:3301/0 2024-04-26T00:43:44.919 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.915+0000 7f7e24d99640 1 -- 172.21.15.37:0/2867057108 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e2009ba60 con 0x7f7e20133f00 2024-04-26T00:43:44.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.919+0000 7f7e24d99640 1 -- 172.21.15.37:0/2867057108 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7e20133f00 msgr2=0x7f7e20134350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:44.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.919+0000 7f7e24d99640 1 --2- 172.21.15.37:0/2867057108 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7e20133f00 0x7f7e20134350 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7e100097f0 tx=0x7f7e1001c140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f81bc3806d151547 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:44.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.919+0000 7f7e17fff640 1 -- 172.21.15.37:0/2867057108 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e100b0070 con 0x7f7e20133f00 2024-04-26T00:43:44.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.919+0000 7f7e17fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:44.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.919+0000 7f7e17fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T00:43:44.920 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.919+0000 7f7e17fff640 10 monclient: dump: 2024-04-26T00:43:44.921 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:44.921 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:44.921 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:44.921 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:44.921 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:44.921 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:44.921 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:44.921 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:44.921 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:44.921 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:44.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.919+0000 7f7e17fff640 10 monclient: _finish_auth 0 2024-04-26T00:43:44.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.919+0000 7f7e17fff640 10 monclient: _check_auth_tickets 2024-04-26T00:43:44.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.919+0000 7f7e17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:44.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.919+0000 7f7e17fff640 1 -- 172.21.15.37:0/2867057108 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e100ab600 con 0x7f7e20133f00 2024-04-26T00:43:44.921 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.919+0000 7f7e17fff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:44.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.919+0000 7f7e17fff640 1 -- 172.21.15.37:0/2867057108 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e100ab9d0 con 0x7f7e20133f00 2024-04-26T00:43:44.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.919+0000 7f7e17fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:44.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.919+0000 7f7e17fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:43:44.922 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.919+0000 7f7e17fff640 10 monclient: dump: 2024-04-26T00:43:44.922 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:44.922 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:44.922 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:44.922 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:44.922 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:44.922 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:44.922 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:44.922 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:44.922 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:44.922 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:44.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.919+0000 7f7e277fe640 5 monclient: authenticate success, global_id 254537 2024-04-26T00:43:44.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.919+0000 7f7e277fe640 10 monclient: get_monmap_and_config success 2024-04-26T00:43:44.923 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.919+0000 7f7e277fe640 4 set_mon_vals no callback set 2024-04-26T00:43:44.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.919+0000 7f7e277fe640 10 monclient: shutdown 2024-04-26T00:43:44.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.919+0000 7f7e277fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:44.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.919+0000 7f7e277fe640 1 -- 172.21.15.37:0/2867057108 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7e20133f00 msgr2=0x7f7e20134350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:44.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.919+0000 7f7e277fe640 1 --2- 172.21.15.37:0/2867057108 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7e20133f00 0x7f7e20134350 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7e100097f0 tx=0x7f7e1001c140 comp rx=0 tx=0).stop 2024-04-26T00:43:44.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.919+0000 7f7e277fe640 1 -- 172.21.15.37:0/2867057108 shutdown_connections 2024-04-26T00:43:44.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.919+0000 7f7e277fe640 1 --2- 172.21.15.37:0/2867057108 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7e20133f00 0x7f7e20134350 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:44.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.919+0000 7f7e277fe640 1 -- 172.21.15.37:0/2867057108 >> 172.21.15.37:0/2867057108 conn(0x7f7e200a1ad0 msgr2=0x7f7e200a1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:44.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.919+0000 7f7e277fe640 1 -- 172.21.15.37:0/2867057108 shutdown_connections 2024-04-26T00:43:44.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.919+0000 7f7e277fe640 1 -- 172.21.15.37:0/2867057108 wait complete. 2024-04-26T00:43:44.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.919+0000 7f7e277fe640 10 monclient: build_initial_monmap 2024-04-26T00:43:44.928 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.919+0000 7f7e277fe640 10 monclient: monmap: 2024-04-26T00:43:44.929 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:44.929 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:43:44.929 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:43:44.929 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:43:44.929 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:44.929 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:43:44.929 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:43:44.929 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:43:44.929 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:43:44.929 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:44.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.919+0000 7f7e277fe640 1 Processor -- start 2024-04-26T00:43:44.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.919+0000 7f7e277fe640 1 -- start start 2024-04-26T00:43:44.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.919+0000 7f7e277fe640 10 monclient: init 2024-04-26T00:43:44.929 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.919+0000 7f7e277fe640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:44.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e277fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T00:43:44.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e277fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e20133b20 0x7f7e201cd160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:44.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e277fe640 10 monclient(hunting): picked mon.noname-a con 0x7f7e20133b20 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:43:44.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e277fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7e20133f00 0x7f7e20130e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:44.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e277fe640 10 monclient(hunting): picked mon.noname-c con 0x7f7e20133f00 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:44.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e277fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e20139050 0x7f7e2006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:44.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e277fe640 10 monclient(hunting): picked mon.noname-b con 0x7f7e20139050 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:43:44.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e277fe640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:44.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e277fe640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f7e2013f950 con 0x7f7e20133b20 2024-04-26T00:43:44.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e277fe640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:44.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e277fe640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7e2013f7d0 con 0x7f7e20139050 2024-04-26T00:43:44.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e277fe640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:44.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e277fe640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f7e2013fad0 con 0x7f7e20133f00 2024-04-26T00:43:44.930 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e277fe640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:44.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e277fe640 10 monclient(hunting): authenticate will time out at 2406.228168s 2024-04-26T00:43:44.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e2559a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e20133b20 0x7f7e201cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:44.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e2559a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e20133b20 0x7f7e201cd160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:32886/0 (socket says 172.21.15.37:32886) 2024-04-26T00:43:44.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e2559a640 1 -- 172.21.15.37:0/2713185963 learned_addr learned my addr 172.21.15.37:0/2713185963 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:44.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e2559a640 10 monclient(hunting): get_auth_request con 0x7f7e20133b20 auth_method 0 2024-04-26T00:43:44.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e24d99640 1 --2- 172.21.15.37:0/2713185963 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7e20133f00 0x7f7e20130e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:44.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e2559a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:44.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e2559a640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:44.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e2559a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:44.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e2559a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:44.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e2559a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:44.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e2559a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:44.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e24d99640 10 monclient(hunting): get_auth_request con 0x7f7e20133f00 auth_method 0 2024-04-26T00:43:44.931 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e24d99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:44.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e24d99640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:44.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e24d99640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:44.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e24d99640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:44.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e24d99640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:44.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e24d99640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:44.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e25d9b640 1 --2- 172.21.15.37:0/2713185963 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e20139050 0x7f7e2006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:44.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e2559a640 10 monclient(hunting): handle_auth_done global_id 284211 payload 824 2024-04-26T00:43:44.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e2559a640 1 -- 172.21.15.37:0/2713185963 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7e20133f00 msgr2=0x7f7e20130e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:44.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e2559a640 1 --2- 172.21.15.37:0/2713185963 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7e20133f00 0x7f7e20130e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:44.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e2559a640 1 -- 172.21.15.37:0/2713185963 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e20139050 msgr2=0x7f7e2006ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:44.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e2559a640 1 --2- 172.21.15.37:0/2713185963 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e20139050 0x7f7e2006ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:44.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e2559a640 10 monclient: _finish_hunting 0 2024-04-26T00:43:44.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e2559a640 1 monclient: found mon.noname-a 2024-04-26T00:43:44.932 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e2559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:44.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e2559a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:43:44.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e2559a640 1 -- 172.21.15.37:0/2713185963 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e201cd6a0 con 0x7f7e20133b20 2024-04-26T00:43:44.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e25d9b640 10 monclient: get_auth_request con 0x7f7e20139050 auth_method 0 2024-04-26T00:43:44.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e25d9b640 1 --2- 172.21.15.37:0/2713185963 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e20139050 0x7f7e2006ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:43:44.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e2559a640 1 -- 172.21.15.37:0/2713185963 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e20133b20 msgr2=0x7f7e201cd160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:44.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e2559a640 1 --2- 172.21.15.37:0/2713185963 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e20133b20 0x7f7e201cd160 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c00c470 tx=0x7f7e1c00c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:44.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e15ffb640 1 -- 172.21.15.37:0/2713185963 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c007070 con 0x7f7e20133b20 2024-04-26T00:43:44.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e15ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:44.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e15ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:43:44.933 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e15ffb640 10 monclient: dump: 2024-04-26T00:43:44.933 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:44.933 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:44.933 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:44.933 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:44.933 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:44.934 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:44.934 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:44.934 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:44.934 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:44.934 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:44.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e15ffb640 10 monclient: _finish_auth 0 2024-04-26T00:43:44.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e15ffb640 10 monclient: _check_auth_tickets 2024-04-26T00:43:44.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e15ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:44.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e15ffb640 1 -- 172.21.15.37:0/2713185963 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e1c007210 con 0x7f7e20133b20 2024-04-26T00:43:44.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e15ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:44.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e15ffb640 1 -- 172.21.15.37:0/2713185963 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c007600 con 0x7f7e20133b20 2024-04-26T00:43:44.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e15ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:44.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e15ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:43:44.934 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e15ffb640 10 monclient: dump: 2024-04-26T00:43:44.935 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:44.935 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:44.935 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:44.935 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:44.935 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:44.935 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:44.935 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:44.935 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:44.935 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:44.935 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:44.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e177fe640 4 set_mon_vals no callback set 2024-04-26T00:43:44.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e277fe640 5 monclient: authenticate success, global_id 284211 2024-04-26T00:43:44.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e277fe640 10 monclient: _renew_subs 2024-04-26T00:43:44.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e277fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:43:44.935 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e277fe640 1 -- 172.21.15.37:0/2713185963 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e20070a80 con 0x7f7e20133b20 2024-04-26T00:43:44.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e277fe640 10 monclient: _renew_subs 2024-04-26T00:43:44.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e277fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:43:44.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.923+0000 7f7e277fe640 1 -- 172.21.15.37:0/2713185963 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e200727b0 con 0x7f7e20133b20 2024-04-26T00:43:44.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.927+0000 7f7e277fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:44.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.927+0000 7f7e277fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:44.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.927+0000 7f7e277fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.37:3300/0 2024-04-26T00:43:44.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.927+0000 7f7e277fe640 1 -- 172.21.15.37:0/2713185963 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e20066f40 con 0x7f7e20133b20 2024-04-26T00:43:44.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.927+0000 7f7e15ffb640 1 -- 172.21.15.37:0/2713185963 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f7e1c01b400 con 0x7f7e20133b20 2024-04-26T00:43:44.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.927+0000 7f7e15ffb640 1 --2- 172.21.15.37:0/2713185963 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7e04035e80 0x7f7e04038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:44.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.927+0000 7f7e15ffb640 1 -- 172.21.15.37:0/2713185963 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(899..899 src has 822..899) ==== 3550+0+0 (secure 0 0 0) 0x7f7e1c0486c0 con 0x7f7e20133b20 2024-04-26T00:43:44.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.927+0000 7f7e24d99640 1 --2- 172.21.15.37:0/2713185963 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7e04035e80 0x7f7e04038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:44.936 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.927+0000 7f7e24d99640 10 monclient: get_auth_request con 0x7f7e04035e80 auth_method 0 2024-04-26T00:43:44.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.931+0000 7f7e24d99640 1 --2- 172.21.15.37:0/2713185963 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7e04035e80 0x7f7e04038340 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f7e100ab000 tx=0x7f7e10008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:44.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.931+0000 7f7e15ffb640 1 -- 172.21.15.37:0/2713185963 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7e1c01b980 con 0x7f7e20133b20 2024-04-26T00:43:44.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.931+0000 7f7e15ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:44.937 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:44.931+0000 7f7e15ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:43:45.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.091+0000 7f7e277fe640 1 -- 172.21.15.37:0/2713185963 --> v2:172.21.15.45:6812/2379918623 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7e2011b820 con 0x7f7e04035e80 2024-04-26T00:43:45.095 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.095+0000 7f7e15ffb640 1 -- 172.21.15.37:0/2713185963 <== mgr.4107 v2:172.21.15.45:6812/2379918623 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33858 (secure 0 0 0) 0x7f7e2011b820 con 0x7f7e04035e80 2024-04-26T00:43:45.095 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T00:43:45.096 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-04-26T00:43:45.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.095+0000 7f7e277fe640 1 -- 172.21.15.37:0/2713185963 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7e04035e80 msgr2=0x7f7e04038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:45.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.095+0000 7f7e277fe640 1 --2- 172.21.15.37:0/2713185963 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7e04035e80 0x7f7e04038340 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f7e100ab000 tx=0x7f7e10008040 comp rx=0 tx=0).stop 2024-04-26T00:43:45.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.095+0000 7f7e277fe640 10 monclient: shutdown 2024-04-26T00:43:45.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.095+0000 7f7e277fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:45.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.095+0000 7f7e277fe640 1 -- 172.21.15.37:0/2713185963 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e20133b20 msgr2=0x7f7e201cd160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:45.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.095+0000 7f7e277fe640 1 --2- 172.21.15.37:0/2713185963 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e20133b20 0x7f7e201cd160 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c00c470 tx=0x7f7e1c00c930 comp rx=0 tx=0).stop 2024-04-26T00:43:45.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.095+0000 7f7e277fe640 1 -- 172.21.15.37:0/2713185963 shutdown_connections 2024-04-26T00:43:45.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.095+0000 7f7e277fe640 1 --2- 172.21.15.37:0/2713185963 >> v2:172.21.15.45:6812/2379918623 conn(0x7f7e04035e80 0x7f7e04038340 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:45.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.095+0000 7f7e277fe640 1 --2- 172.21.15.37:0/2713185963 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7e20139050 0x7f7e2006ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:45.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.095+0000 7f7e277fe640 1 --2- 172.21.15.37:0/2713185963 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7e20133f00 0x7f7e20130e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:45.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.095+0000 7f7e277fe640 1 --2- 172.21.15.37:0/2713185963 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e20133b20 0x7f7e201cd160 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:45.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.095+0000 7f7e277fe640 1 -- 172.21.15.37:0/2713185963 >> 172.21.15.37:0/2713185963 conn(0x7f7e200a1ad0 msgr2=0x7f7e20131eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:45.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.095+0000 7f7e277fe640 1 -- 172.21.15.37:0/2713185963 shutdown_connections 2024-04-26T00:43:45.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.095+0000 7f7e277fe640 1 -- 172.21.15.37:0/2713185963 wait complete. 2024-04-26T00:43:45.114 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":1595,"stamp":"2024-04-26T00:43:43.559030+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":346320,"kb_used_data":3936,"kb_used_omap":9,"kb_used_meta":342326,"kb_avail":565884720,"statfs":{"total":579820584960,"available":579465953280,"internally_reserved":0,"allocated":4030464,"data_stored":3181916,"data_compressed":61234,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":350542496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003172"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":1764,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:43:24.090052+0000","last_change":"2024-04-26T00:43:24.090109+0000","last_active":"2024-04-26T00:43:24.090052+0000","last_peered":"2024-04-26T00:43:24.090052+0000","last_clean":"2024-04-26T00:43:24.090052+0000","last_became_active":"2024-04-26T00:20:02.278959+0000","last_became_peered":"2024-04-26T00:20:02.278959+0000","last_unstale":"2024-04-26T00:43:24.090052+0000","last_undegraded":"2024-04-26T00:43:24.090052+0000","last_fullsized":"2024-04-26T00:43:24.090052+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:43:24.089846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:43:24.089846+0000","last_clean_scrub_stamp":"2024-04-26T00:43:24.089846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:03:54.989096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":1759,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:43:24.577896+0000","last_change":"2024-04-26T00:43:24.577951+0000","last_active":"2024-04-26T00:43:24.577896+0000","last_peered":"2024-04-26T00:43:24.577896+0000","last_clean":"2024-04-26T00:43:24.577896+0000","last_became_active":"2024-04-26T00:20:02.279053+0000","last_became_peered":"2024-04-26T00:20:02.279053+0000","last_unstale":"2024-04-26T00:43:24.577896+0000","last_undegraded":"2024-04-26T00:43:24.577896+0000","last_fullsized":"2024-04-26T00:43:24.577896+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:43:24.577717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:43:24.577717+0000","last_clean_scrub_stamp":"2024-04-26T00:43:24.577717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:45:31.789615+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":1770,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:43:24.406895+0000","last_change":"2024-04-26T00:43:24.406951+0000","last_active":"2024-04-26T00:43:24.406895+0000","last_peered":"2024-04-26T00:43:24.406895+0000","last_clean":"2024-04-26T00:43:24.406895+0000","last_became_active":"2024-04-26T00:20:02.276409+0000","last_became_peered":"2024-04-26T00:20:02.276409+0000","last_unstale":"2024-04-26T00:43:24.406895+0000","last_undegraded":"2024-04-26T00:43:24.406895+0000","last_fullsized":"2024-04-26T00:43:24.406895+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:43:24.406707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:43:24.406707+0000","last_clean_scrub_stamp":"2024-04-26T00:43:24.406707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:16:11.732945+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":1772,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:43:23.302015+0000","last_change":"2024-04-26T00:43:23.302035+0000","last_active":"2024-04-26T00:43:23.302015+0000","last_peered":"2024-04-26T00:43:23.302015+0000","last_clean":"2024-04-26T00:43:23.302015+0000","last_became_active":"2024-04-26T00:20:02.273213+0000","last_became_peered":"2024-04-26T00:20:02.273213+0000","last_unstale":"2024-04-26T00:43:23.302015+0000","last_undegraded":"2024-04-26T00:43:23.302015+0000","last_fullsized":"2024-04-26T00:43:23.302015+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:43:23.301933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:43:23.301933+0000","last_clean_scrub_stamp":"2024-04-26T00:43:23.301933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:00:48.504491+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"899'119","reported_seq":1916,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:43:26.128516+0000","last_change":"2024-04-26T00:43:26.128588+0000","last_active":"2024-04-26T00:43:26.128516+0000","last_peered":"2024-04-26T00:43:26.128516+0000","last_clean":"2024-04-26T00:43:26.128516+0000","last_became_active":"2024-04-26T00:20:02.281000+0000","last_became_peered":"2024-04-26T00:20:02.281000+0000","last_unstale":"2024-04-26T00:43:26.128516+0000","last_undegraded":"2024-04-26T00:43:26.128516+0000","last_fullsized":"2024-04-26T00:43:26.128516+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'119","last_scrub_stamp":"2024-04-26T00:43:26.128347+0000","last_deep_scrub":"899'119","last_deep_scrub_stamp":"2024-04-26T00:43:26.128347+0000","last_clean_scrub_stamp":"2024-04-26T00:43:26.128347+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:37:10.653400+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":1772,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:43:24.314408+0000","last_change":"2024-04-26T00:43:24.314444+0000","last_active":"2024-04-26T00:43:24.314408+0000","last_peered":"2024-04-26T00:43:24.314408+0000","last_clean":"2024-04-26T00:43:24.314408+0000","last_became_active":"2024-04-26T00:20:02.273745+0000","last_became_peered":"2024-04-26T00:20:02.273745+0000","last_unstale":"2024-04-26T00:43:24.314408+0000","last_undegraded":"2024-04-26T00:43:24.314408+0000","last_fullsized":"2024-04-26T00:43:24.314408+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:43:24.314289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:43:24.314289+0000","last_clean_scrub_stamp":"2024-04-26T00:43:24.314289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:02:48.655580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":1770,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:43:25.440343+0000","last_change":"2024-04-26T00:43:25.440462+0000","last_active":"2024-04-26T00:43:25.440343+0000","last_peered":"2024-04-26T00:43:25.440343+0000","last_clean":"2024-04-26T00:43:25.440343+0000","last_became_active":"2024-04-26T00:20:02.473888+0000","last_became_peered":"2024-04-26T00:20:02.473888+0000","last_unstale":"2024-04-26T00:43:25.440343+0000","last_undegraded":"2024-04-26T00:43:25.440343+0000","last_fullsized":"2024-04-26T00:43:25.440343+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:43:25.440131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:43:25.440131+0000","last_clean_scrub_stamp":"2024-04-26T00:43:25.440131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:45:21.858734+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":1763,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:43:25.141665+0000","last_change":"2024-04-26T00:43:25.141717+0000","last_active":"2024-04-26T00:43:25.141665+0000","last_peered":"2024-04-26T00:43:25.141665+0000","last_clean":"2024-04-26T00:43:25.141665+0000","last_became_active":"2024-04-26T00:20:02.279183+0000","last_became_peered":"2024-04-26T00:20:02.279183+0000","last_unstale":"2024-04-26T00:43:25.141665+0000","last_undegraded":"2024-04-26T00:43:25.141665+0000","last_fullsized":"2024-04-26T00:43:25.141665+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:43:25.141493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:43:25.141493+0000","last_clean_scrub_stamp":"2024-04-26T00:43:25.141493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:34:22.024208+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":1758,"reported_epoch":899,"state":"active+clean","last_fresh":"2024-04-26T00:43:25.632436+0000","last_change":"2024-04-26T00:43:25.632491+0000","last_active":"2024-04-26T00:43:25.632436+0000","last_peered":"2024-04-26T00:43:25.632436+0000","last_clean":"2024-04-26T00:43:25.632436+0000","last_became_active":"2024-04-26T00:20:02.473928+0000","last_became_peered":"2024-04-26T00:20:02.473928+0000","last_unstale":"2024-04-26T00:43:25.632436+0000","last_undegraded":"2024-04-26T00:43:25.632436+0000","last_fullsized":"2024-04-26T00:43:25.632436+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T00:43:25.632242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T00:43:25.632242+0000","last_clean_scrub_stamp":"2024-04-26T00:43:25.632242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T08:26:49.704679+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":61234,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640543,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58180,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313660,"statfs":{"total":96636764160,"available":96577187840,"internally_reserved":0,"allocated":651264,"data_stored":354010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640542,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49860,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321980,"statfs":{"total":96636764160,"available":96585707520,"internally_reserved":0,"allocated":651264,"data_stored":354010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640544,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40192,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331648,"statfs":{"total":96636764160,"available":96595607552,"internally_reserved":0,"allocated":712704,"data_stored":882938,"data_compressed":30617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640544,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72004,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299836,"statfs":{"total":96636764160,"available":96563032064,"internally_reserved":0,"allocated":651264,"data_stored":354010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640544,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72064,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299776,"statfs":{"total":96636764160,"available":96562970624,"internally_reserved":0,"allocated":712704,"data_stored":882938,"data_compressed":30617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640543,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54020,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317820,"statfs":{"total":96636764160,"available":96581447680,"internally_reserved":0,"allocated":651264,"data_stored":354010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T00:43:45.115 DEBUG:teuthology.orchestra.run.smithi037:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-26T00:43:45.236 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a73604640 10 monclient: get_monmap_and_config 2024-04-26T00:43:45.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a73604640 10 monclient: build_initial_monmap 2024-04-26T00:43:45.237 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a73604640 10 monclient: monmap: 2024-04-26T00:43:45.237 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:45.237 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:45.237 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:43:45.237431+0000 2024-04-26T00:43:45.237 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:43:45.237431+0000 2024-04-26T00:43:45.237 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:45.237 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:45.237 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a-legacy 2024-04-26T00:43:45.237 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.45:6789/0 mon.noname-c-legacy 2024-04-26T00:43:45.237 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-b 2024-04-26T00:43:45.238 INFO:teuthology.orchestra.run.smithi037.stderr:3: v2:172.21.15.37:3300/0 mon.noname-a 2024-04-26T00:43:45.238 INFO:teuthology.orchestra.run.smithi037.stderr:4: v2:172.21.15.45:3300/0 mon.noname-c 2024-04-26T00:43:45.238 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:45.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a73604640 1 Processor -- start 2024-04-26T00:43:45.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a73604640 1 -- start start 2024-04-26T00:43:45.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a73604640 10 monclient: init 2024-04-26T00:43:45.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a73604640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:45.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a73604640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-04-26T00:43:45.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a73604640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f2a6c0a3d90 0x7f2a6c09bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:45.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a73604640 10 monclient(hunting): picked mon.noname-a con 0x7f2a6c0a3d90 addr v2:172.21.15.37:3300/0 2024-04-26T00:43:45.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a73604640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2a6c0a4170 addr v1:172.21.15.45:6789/0 2024-04-26T00:43:45.240 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a73604640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a6c13c3e0 0x7f2a6c13e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:45.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a73604640 10 monclient(hunting): picked mon.noname-b con 0x7f2a6c13c3e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:45.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a73604640 1 -- --> v1:172.21.15.45:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a6c0a5150 con 0x7f2a6c0a4170 2024-04-26T00:43:45.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a73604640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:45.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a73604640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f2a6c09b280 con 0x7f2a6c0a3d90 2024-04-26T00:43:45.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a73604640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:45.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a73604640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2a6c09b560 con 0x7f2a6c13c3e0 2024-04-26T00:43:45.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a73604640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:45.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a73604640 10 monclient(hunting): authenticate will time out at 2406.543130s 2024-04-26T00:43:45.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a71ba1640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a6c13c3e0 0x7f2a6c13e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:45.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a713a0640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f2a6c0a3d90 0x7f2a6c09bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:45.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a71ba1640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a6c13c3e0 0x7f2a6c13e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:49362/0 (socket says 172.21.15.37:49362) 2024-04-26T00:43:45.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a71ba1640 1 -- 172.21.15.37:0/3297673729 learned_addr learned my addr 172.21.15.37:0/3297673729 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:45.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a713a0640 10 monclient(hunting): get_auth_request con 0x7f2a6c0a3d90 auth_method 0 2024-04-26T00:43:45.241 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a713a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:45.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a713a0640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:45.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a713a0640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:45.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a71ba1640 10 monclient(hunting): get_auth_request con 0x7f2a6c13c3e0 auth_method 0 2024-04-26T00:43:45.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a71ba1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:45.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a71ba1640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:45.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a71ba1640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:45.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a713a0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:45.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a713a0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:45.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a713a0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:45.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a71ba1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:45.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a71ba1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:45.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a71ba1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:45.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a713a0640 10 monclient(hunting): handle_auth_done global_id 284214 payload 274 2024-04-26T00:43:45.242 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a713a0640 1 -- 172.21.15.37:0/3297673729 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a6c13c3e0 msgr2=0x7f2a6c13e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:45.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a713a0640 1 --2- 172.21.15.37:0/3297673729 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a6c13c3e0 0x7f2a6c13e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:45.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a713a0640 1 -- 172.21.15.37:0/3297673729 >> v1:172.21.15.45:6789/0 conn(0x7f2a6c0a4170 legacy=0x7f2a6c0a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T00:43:45.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a713a0640 10 monclient: _finish_hunting 0 2024-04-26T00:43:45.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a713a0640 1 monclient: found mon.noname-a 2024-04-26T00:43:45.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a713a0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:45.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a713a0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.37:3300/0 2024-04-26T00:43:45.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a713a0640 1 -- 172.21.15.37:0/3297673729 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a6c06dc60 con 0x7f2a6c0a3d90 2024-04-26T00:43:45.243 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.235+0000 7f2a71ba1640 1 --2- 172.21.15.37:0/3297673729 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a6c13c3e0 0x7f2a6c13e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T00:43:45.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.239+0000 7f2a713a0640 1 -- 172.21.15.37:0/3297673729 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a6c0a3d90 msgr2=0x7f2a6c09bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:45.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.239+0000 7f2a713a0640 1 --2- 172.21.15.37:0/3297673729 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a6c0a3d90 0x7f2a6c09bf40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2a600097c0 tx=0x7f2a6001c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90f714d0d71557a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:45.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.239+0000 7f2a5ffff640 1 -- 172.21.15.37:0/3297673729 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a600b0070 con 0x7f2a6c0a3d90 2024-04-26T00:43:45.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.239+0000 7f2a5ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:45.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.239+0000 7f2a5ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T00:43:45.247 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.239+0000 7f2a5ffff640 10 monclient: dump: 2024-04-26T00:43:45.247 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:45.247 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:45.247 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:45.247 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:45.248 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:45.248 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:45.248 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:45.248 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:45.248 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:45.248 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:45.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a5ffff640 10 monclient: _finish_auth 0 2024-04-26T00:43:45.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a5ffff640 10 monclient: _check_auth_tickets 2024-04-26T00:43:45.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a5ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:45.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a5ffff640 1 -- 172.21.15.37:0/3297673729 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a600ab600 con 0x7f2a6c0a3d90 2024-04-26T00:43:45.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a73604640 5 monclient: authenticate success, global_id 284214 2024-04-26T00:43:45.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a73604640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T00:43:45.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a5ffff640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:45.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a73604640 10 monclient: get_monmap_and_config success 2024-04-26T00:43:45.248 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a5ffff640 1 -- 172.21.15.37:0/3297673729 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a600ab9d0 con 0x7f2a6c0a3d90 2024-04-26T00:43:45.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a73604640 4 set_mon_vals no callback set 2024-04-26T00:43:45.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a5ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:45.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a5ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T00:43:45.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a5ffff640 10 monclient: dump: 2024-04-26T00:43:45.249 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:45.249 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:45.249 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:45.249 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:45.249 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:45.249 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:45.249 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:45.249 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:45.249 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:45.249 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:45.249 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a73604640 10 monclient: shutdown 2024-04-26T00:43:45.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a73604640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:45.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a73604640 1 -- 172.21.15.37:0/3297673729 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a6c0a3d90 msgr2=0x7f2a6c09bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:45.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a73604640 1 --2- 172.21.15.37:0/3297673729 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a6c0a3d90 0x7f2a6c09bf40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2a600097c0 tx=0x7f2a6001c320 comp rx=0 tx=0).stop 2024-04-26T00:43:45.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a73604640 1 -- 172.21.15.37:0/3297673729 shutdown_connections 2024-04-26T00:43:45.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a73604640 1 --2- 172.21.15.37:0/3297673729 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a6c13c3e0 0x7f2a6c13e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:45.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a73604640 1 --2- 172.21.15.37:0/3297673729 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a6c0a3d90 0x7f2a6c09bf40 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:45.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a73604640 1 -- 172.21.15.37:0/3297673729 >> 172.21.15.37:0/3297673729 conn(0x7f2a6c0aa0a0 msgr2=0x7f2a6c0a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:45.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a73604640 1 -- 172.21.15.37:0/3297673729 shutdown_connections 2024-04-26T00:43:45.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a73604640 1 -- 172.21.15.37:0/3297673729 wait complete. 2024-04-26T00:43:45.250 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a73604640 10 monclient: build_initial_monmap 2024-04-26T00:43:45.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a73604640 10 monclient: monmap: 2024-04-26T00:43:45.251 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-04-26T00:43:45.251 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T00:43:45.251 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-04-26T00:43:45.251 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-04-26T00:43:45.251 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-04-26T00:43:45.251 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-04-26T00:43:45.251 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.noname-a 2024-04-26T00:43:45.251 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.noname-b 2024-04-26T00:43:45.251 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.noname-c 2024-04-26T00:43:45.251 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:45.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a73604640 1 Processor -- start 2024-04-26T00:43:45.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a73604640 1 -- start start 2024-04-26T00:43:45.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a73604640 10 monclient: init 2024-04-26T00:43:45.251 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a73604640 10 monclient: _reopen_session rank -1 2024-04-26T00:43:45.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a73604640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T00:43:45.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a73604640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a6c0a3d90 0x7f2a6c139f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:45.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a73604640 10 monclient(hunting): picked mon.noname-b con 0x7f2a6c0a3d90 addr [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] 2024-04-26T00:43:45.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a73604640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a6c0a4170 0x7f2a6c13a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:45.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a73604640 10 monclient(hunting): picked mon.noname-a con 0x7f2a6c0a4170 addr [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] 2024-04-26T00:43:45.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a73604640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a6c13c3e0 0x7f2a6c13a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:45.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a73604640 10 monclient(hunting): picked mon.noname-c con 0x7f2a6c13c3e0 addr [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] 2024-04-26T00:43:45.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a73604640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:45.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a73604640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f2a6c141740 con 0x7f2a6c0a4170 2024-04-26T00:43:45.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a73604640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:45.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a73604640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2a6c09c690 con 0x7f2a6c0a3d90 2024-04-26T00:43:45.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a73604640 10 monclient(hunting): start opening mon connection 2024-04-26T00:43:45.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a73604640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2a6c1418c0 con 0x7f2a6c13c3e0 2024-04-26T00:43:45.252 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a73604640 10 monclient(hunting): _renew_subs 2024-04-26T00:43:45.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a73604640 10 monclient(hunting): authenticate will time out at 2406.550129s 2024-04-26T00:43:45.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a70b9f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a6c0a4170 0x7f2a6c13a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:45.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a71ba1640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a6c13c3e0 0x7f2a6c13a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:45.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a70b9f640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a6c0a4170 0x7f2a6c13a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:32904/0 (socket says 172.21.15.37:32904) 2024-04-26T00:43:45.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a70b9f640 1 -- 172.21.15.37:0/3018194923 learned_addr learned my addr 172.21.15.37:0/3018194923 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-26T00:43:45.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a71ba1640 10 monclient(hunting): get_auth_request con 0x7f2a6c13c3e0 auth_method 0 2024-04-26T00:43:45.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a71ba1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:45.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a71ba1640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:45.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a71ba1640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:45.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a70b9f640 10 monclient(hunting): get_auth_request con 0x7f2a6c0a4170 auth_method 0 2024-04-26T00:43:45.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a70b9f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T00:43:45.253 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a70b9f640 10 monclient(hunting): _init_auth method 2 2024-04-26T00:43:45.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a70b9f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T00:43:45.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a71ba1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:45.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a71ba1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:45.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a71ba1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:45.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a70b9f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T00:43:45.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a70b9f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T00:43:45.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a70b9f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T00:43:45.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a713a0640 1 --2- 172.21.15.37:0/3018194923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a6c0a3d90 0x7f2a6c139f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:45.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a71ba1640 10 monclient(hunting): handle_auth_done global_id 254546 payload 824 2024-04-26T00:43:45.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a71ba1640 1 -- 172.21.15.37:0/3018194923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a6c0a3d90 msgr2=0x7f2a6c139f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:45.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a71ba1640 1 --2- 172.21.15.37:0/3018194923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a6c0a3d90 0x7f2a6c139f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:45.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a71ba1640 1 -- 172.21.15.37:0/3018194923 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a6c0a4170 msgr2=0x7f2a6c13a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:45.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a71ba1640 1 --2- 172.21.15.37:0/3018194923 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a6c0a4170 0x7f2a6c13a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:45.254 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a71ba1640 10 monclient: _finish_hunting 0 2024-04-26T00:43:45.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a71ba1640 1 monclient: found mon.noname-c 2024-04-26T00:43:45.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a71ba1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T00:43:45.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a71ba1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.37:3301/0 2024-04-26T00:43:45.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a71ba1640 1 -- 172.21.15.37:0/3018194923 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a6c09ada0 con 0x7f2a6c13c3e0 2024-04-26T00:43:45.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a713a0640 10 monclient: get_auth_request con 0x7f2a6c0a3d90 auth_method 0 2024-04-26T00:43:45.255 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.243+0000 7f2a713a0640 1 --2- 172.21.15.37:0/3018194923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a6c0a3d90 0x7f2a6c139f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T00:43:45.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.247+0000 7f2a71ba1640 1 -- 172.21.15.37:0/3018194923 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a6c13c3e0 msgr2=0x7f2a6c13a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T00:43:45.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.247+0000 7f2a71ba1640 1 --2- 172.21.15.37:0/3018194923 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a6c13c3e0 0x7f2a6c13a9f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2a64009e50 tx=0x7f2a6400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:45.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.247+0000 7f2a5dffb640 1 -- 172.21.15.37:0/3018194923 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a6400cd90 con 0x7f2a6c13c3e0 2024-04-26T00:43:45.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.247+0000 7f2a5dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:45.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.247+0000 7f2a5dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T00:43:45.257 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.247+0000 7f2a5dffb640 10 monclient: dump: 2024-04-26T00:43:45.257 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:45.257 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:45.257 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:45.257 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:45.257 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:45.258 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:45.258 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:45.258 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:45.258 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:45.258 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:45.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.247+0000 7f2a5dffb640 10 monclient: _finish_auth 0 2024-04-26T00:43:45.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.247+0000 7f2a5dffb640 10 monclient: _check_auth_tickets 2024-04-26T00:43:45.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.247+0000 7f2a5dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T00:43:45.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.247+0000 7f2a5dffb640 1 -- 172.21.15.37:0/3018194923 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a640039a0 con 0x7f2a6c13c3e0 2024-04-26T00:43:45.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.247+0000 7f2a73604640 5 monclient: authenticate success, global_id 254546 2024-04-26T00:43:45.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.247+0000 7f2a73604640 10 monclient: _renew_subs 2024-04-26T00:43:45.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.247+0000 7f2a73604640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:45.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.247+0000 7f2a73604640 1 -- 172.21.15.37:0/3018194923 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a6c13bbc0 con 0x7f2a6c13c3e0 2024-04-26T00:43:45.258 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.247+0000 7f2a5dffb640 10 monclient: handle_config config(0 keys) 2024-04-26T00:43:45.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.247+0000 7f2a5dffb640 1 -- 172.21.15.37:0/3018194923 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a64003d90 con 0x7f2a6c13c3e0 2024-04-26T00:43:45.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.247+0000 7f2a5dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T00:43:45.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.247+0000 7f2a5dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T00:43:45.259 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.247+0000 7f2a5dffb640 10 monclient: dump: 2024-04-26T00:43:45.259 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-04-26T00:43:45.259 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 3cf2f905-fe97-4d94-8eba-4a21c519e869 2024-04-26T00:43:45.259 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:45.259 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-04-26T00:19:19.195498+0000 2024-04-26T00:43:45.259 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-04-26T00:43:45.259 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-04-26T00:43:45.259 INFO:teuthology.orchestra.run.smithi037.stderr:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T00:43:45.259 INFO:teuthology.orchestra.run.smithi037.stderr:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T00:43:45.260 INFO:teuthology.orchestra.run.smithi037.stderr:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-26T00:43:45.260 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:43:45.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.247+0000 7f2a5f7fe640 4 set_mon_vals no callback set 2024-04-26T00:43:45.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.247+0000 7f2a73604640 10 monclient: _renew_subs 2024-04-26T00:43:45.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.247+0000 7f2a73604640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:45.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.247+0000 7f2a73604640 1 -- 172.21.15.37:0/3018194923 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a6c137ad0 con 0x7f2a6c13c3e0 2024-04-26T00:43:45.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.247+0000 7f2a73604640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:45.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.247+0000 7f2a73604640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:45.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.247+0000 7f2a73604640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:45.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.247+0000 7f2a73604640 1 -- 172.21.15.37:0/3018194923 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a6c04f510 con 0x7f2a6c13c3e0 2024-04-26T00:43:45.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.247+0000 7f2a5dffb640 1 -- 172.21.15.37:0/3018194923 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2a6401b030 con 0x7f2a6c13c3e0 2024-04-26T00:43:45.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.247+0000 7f2a5dffb640 1 --2- 172.21.15.37:0/3018194923 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2a50035e80 0x7f2a50038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T00:43:45.260 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.247+0000 7f2a5dffb640 1 -- 172.21.15.37:0/3018194923 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(899..899 src has 822..899) ==== 3550+0+0 (secure 0 0 0) 0x7f2a64048cd0 con 0x7f2a6c13c3e0 2024-04-26T00:43:45.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.247+0000 7f2a713a0640 1 --2- 172.21.15.37:0/3018194923 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2a50035e80 0x7f2a50038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T00:43:45.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.247+0000 7f2a713a0640 10 monclient: get_auth_request con 0x7f2a50035e80 auth_method 0 2024-04-26T00:43:45.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.251+0000 7f2a713a0640 1 --2- 172.21.15.37:0/3018194923 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2a50035e80 0x7f2a50038340 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c09d140 tx=0x7f2a60008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T00:43:45.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.251+0000 7f2a5dffb640 1 -- 172.21.15.37:0/3018194923 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f2a64022020 con 0x7f2a6c13c3e0 2024-04-26T00:43:45.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.251+0000 7f2a5dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T00:43:45.261 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.251+0000 7f2a5dffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T00:43:45.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.415+0000 7f2a73604640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-04-26T00:43:45.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.415+0000 7f2a73604640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-04-26T00:43:45.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.415+0000 7f2a73604640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.37:3301/0 2024-04-26T00:43:45.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.415+0000 7f2a73604640 1 -- 172.21.15.37:0/3018194923 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f2a6c075dc0 con 0x7f2a6c13c3e0 2024-04-26T00:43:45.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.419+0000 7f2a5dffb640 1 -- 172.21.15.37:0/3018194923 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v4)=0 v4) ==== 123+0+0 (secure 0 0 0) 0x7f2a64010040 con 0x7f2a6c13c3e0 2024-04-26T00:43:45.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.419+0000 7f2a5dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-04-26T00:43:45.420 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.419+0000 7f2a5dffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T00:43:45.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.419+0000 7f2a5dffb640 1 -- 172.21.15.37:0/3018194923 <== mon.2 v2:172.21.15.37:3301/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f2a6401c780 con 0x7f2a6c13c3e0 2024-04-26T00:43:45.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.419+0000 7f2a5dffb640 10 monclient: handle_config config(1 keys) 2024-04-26T00:43:45.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.419+0000 7f2a5effd640 4 set_mon_vals no callback set 2024-04-26T00:43:45.422 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.419+0000 7f2a5effd640 10 set_mon_vals mon_health_to_clog = false 2024-04-26T00:43:45.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.423+0000 7f2a73604640 1 -- 172.21.15.37:0/3018194923 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2a50035e80 msgr2=0x7f2a50038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:45.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.423+0000 7f2a73604640 1 --2- 172.21.15.37:0/3018194923 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2a50035e80 0x7f2a50038340 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c09d140 tx=0x7f2a60008040 comp rx=0 tx=0).stop 2024-04-26T00:43:45.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.423+0000 7f2a73604640 10 monclient: shutdown 2024-04-26T00:43:45.423 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.423+0000 7f2a73604640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T00:43:45.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.423+0000 7f2a73604640 1 -- 172.21.15.37:0/3018194923 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a6c13c3e0 msgr2=0x7f2a6c13a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T00:43:45.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.423+0000 7f2a73604640 1 --2- 172.21.15.37:0/3018194923 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a6c13c3e0 0x7f2a6c13a9f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2a64009e50 tx=0x7f2a6400c490 comp rx=0 tx=0).stop 2024-04-26T00:43:45.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.423+0000 7f2a73604640 1 -- 172.21.15.37:0/3018194923 shutdown_connections 2024-04-26T00:43:45.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.423+0000 7f2a73604640 1 --2- 172.21.15.37:0/3018194923 >> v2:172.21.15.45:6812/2379918623 conn(0x7f2a50035e80 0x7f2a50038340 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:45.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.423+0000 7f2a73604640 1 --2- 172.21.15.37:0/3018194923 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a6c13c3e0 0x7f2a6c13a9f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:45.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.423+0000 7f2a73604640 1 --2- 172.21.15.37:0/3018194923 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a6c0a4170 0x7f2a6c13a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:45.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.423+0000 7f2a73604640 1 --2- 172.21.15.37:0/3018194923 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2a6c0a3d90 0x7f2a6c139f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T00:43:45.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.423+0000 7f2a73604640 1 -- 172.21.15.37:0/3018194923 >> 172.21.15.37:0/3018194923 conn(0x7f2a6c0aa0a0 msgr2=0x7f2a6c0a8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T00:43:45.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.423+0000 7f2a73604640 1 -- 172.21.15.37:0/3018194923 shutdown_connections 2024-04-26T00:43:45.424 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T00:43:45.423+0000 7f2a73604640 1 -- 172.21.15.37:0/3018194923 wait complete. 2024-04-26T00:43:45.439 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-26T00:43:45.439 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-26T00:43:45.439 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-26T00:43:45.439 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T00:43:45.534 INFO:tasks.ceph.osd.0:Stopped 2024-04-26T00:43:45.534 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-26T00:43:45.534 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T00:43:45.619 INFO:tasks.ceph.osd.1:Stopped 2024-04-26T00:43:45.619 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-26T00:43:45.619 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T00:43:48.011 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-04-26T00:43:51.720 INFO:tasks.ceph.osd.2:Stopped 2024-04-26T00:43:51.721 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-26T00:43:51.721 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T00:43:51.792 INFO:tasks.ceph.osd.3:Stopped 2024-04-26T00:43:51.793 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-26T00:43:51.793 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T00:43:51.890 INFO:tasks.ceph.osd.4:Stopped 2024-04-26T00:43:51.891 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-26T00:43:51.891 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T00:43:53.414 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-04-26T00:43:53.415 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-04-26T00:43:57.993 INFO:tasks.ceph.osd.5:Stopped 2024-04-26T00:43:57.993 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-26T00:43:57.993 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-26T00:43:57.994 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T00:43:58.025 INFO:tasks.ceph.mgr.x:Stopped 2024-04-26T00:43:58.025 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-26T00:43:58.025 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-26T00:43:58.025 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T00:43:58.042 INFO:tasks.ceph.mon.a:Stopped 2024-04-26T00:43:58.042 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-26T00:43:58.043 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T00:43:58.107 INFO:tasks.ceph.mon.c:Stopped 2024-04-26T00:43:58.107 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-26T00:43:58.107 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T00:43:58.128 INFO:tasks.ceph.mon.b:Stopped 2024-04-26T00:43:58.128 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-26T00:43:58.128 DEBUG:teuthology.orchestra.run.smithi037:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-04-26T00:43:58.168 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi037.front.sepia.ceph.com 2024-04-26T00:43:58.168 DEBUG:teuthology.orchestra.run.smithi037:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-26T00:43:58.416 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-04-26T00:43:58.556 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi037.front.sepia.ceph.com 2024-04-26T00:43:58.556 DEBUG:teuthology.orchestra.run.smithi037:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-26T00:43:58.787 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi037.front.sepia.ceph.com 2024-04-26T00:43:58.787 DEBUG:teuthology.orchestra.run.smithi037:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-26T00:43:58.979 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi045.front.sepia.ceph.com 2024-04-26T00:43:58.979 DEBUG:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-26T00:43:59.114 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi045.front.sepia.ceph.com 2024-04-26T00:43:59.115 DEBUG:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-26T00:43:59.284 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi045.front.sepia.ceph.com 2024-04-26T00:43:59.284 DEBUG:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-26T00:43:59.469 INFO:tasks.ceph:Archiving mon data... 2024-04-26T00:43:59.470 DEBUG:teuthology.misc:Transferring archived files from smithi037:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673875/data/mon.a.tgz 2024-04-26T00:43:59.470 DEBUG:teuthology.orchestra.run.smithi037:> mktemp 2024-04-26T00:43:59.476 INFO:teuthology.orchestra.run.smithi037.stdout:/tmp/tmp.A21RpA9uCe 2024-04-26T00:43:59.476 DEBUG:teuthology.orchestra.run.smithi037:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.A21RpA9uCe 2024-04-26T00:43:59.671 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 0666 /tmp/tmp.A21RpA9uCe 2024-04-26T00:43:59.736 DEBUG:teuthology.orchestra.remote:smithi037:/tmp/tmp.A21RpA9uCe is 830KB 2024-04-26T00:43:59.801 DEBUG:teuthology.orchestra.run.smithi037:> rm -fr /tmp/tmp.A21RpA9uCe 2024-04-26T00:43:59.806 DEBUG:teuthology.misc:Transferring archived files from smithi037:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673875/data/mon.c.tgz 2024-04-26T00:43:59.806 DEBUG:teuthology.orchestra.run.smithi037:> mktemp 2024-04-26T00:43:59.852 INFO:teuthology.orchestra.run.smithi037.stdout:/tmp/tmp.F34pm76TMx 2024-04-26T00:43:59.852 DEBUG:teuthology.orchestra.run.smithi037:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.F34pm76TMx 2024-04-26T00:44:00.049 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 0666 /tmp/tmp.F34pm76TMx 2024-04-26T00:44:00.120 DEBUG:teuthology.orchestra.remote:smithi037:/tmp/tmp.F34pm76TMx is 833KB 2024-04-26T00:44:00.189 DEBUG:teuthology.orchestra.run.smithi037:> rm -fr /tmp/tmp.F34pm76TMx 2024-04-26T00:44:00.195 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673875/data/mon.b.tgz 2024-04-26T00:44:00.195 DEBUG:teuthology.orchestra.run.smithi045:> mktemp 2024-04-26T00:44:00.201 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/tmp.97mKjMZ6C5 2024-04-26T00:44:00.202 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.97mKjMZ6C5 2024-04-26T00:44:00.393 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0666 /tmp/tmp.97mKjMZ6C5 2024-04-26T00:44:00.461 DEBUG:teuthology.orchestra.remote:smithi045:/tmp/tmp.97mKjMZ6C5 is 830KB 2024-04-26T00:44:00.528 DEBUG:teuthology.orchestra.run.smithi045:> rm -fr /tmp/tmp.97mKjMZ6C5 2024-04-26T00:44:00.533 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-26T00:44:00.533 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-26T00:44:00.536 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-26T00:44:00.600 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-26T00:44:00.634 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-26T00:44:00.635 INFO:tasks.ceph:Archiving crash dumps... 2024-04-26T00:44:00.636 DEBUG:teuthology.misc:Transferring archived files from smithi037:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673875/remote/smithi037/crash 2024-04-26T00:44:00.637 DEBUG:teuthology.orchestra.run.smithi037:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-26T00:44:00.656 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673875/remote/smithi045/crash 2024-04-26T00:44:00.657 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-26T00:44:00.688 INFO:tasks.ceph:Compressing logs... 2024-04-26T00:44:00.688 DEBUG:teuthology.orchestra.run.smithi037:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T00:44:00.701 DEBUG:teuthology.orchestra.run.smithi045:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T00:44:00.727 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35457.log 2024-04-26T00:44:00.727 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39725.log 2024-04-26T00:44:00.727 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27646.log 2024-04-26T00:44:00.728 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32008.log 2024-04-26T00:44:00.728 INFO:teuthology.orchestra.run.smithi037.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.35457.log.gz 2024-04-26T00:44:00.730 INFO:teuthology.orchestra.run.smithi037.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39725.log.gz 2024-04-26T00:44:00.730 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.27646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31482.log 2024-04-26T00:44:00.730 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27199.log 2024-04-26T00:44:00.730 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31482.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.27646.log.gz 2024-04-26T00:44:00.730 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38644.log 2024-04-26T00:44:00.730 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.27199.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.32008.log.gz 2024-04-26T00:44:00.730 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38774.log 2024-04-26T00:44:00.730 INFO:teuthology.orchestra.run.smithi037.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.31482.log.gz 2024-04-26T00:44:00.730 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38644.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.27199.log.gz 2024-04-26T00:44:00.730 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31352.log 2024-04-26T00:44:00.734 INFO:teuthology.orchestra.run.smithi037.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38644.log.gz 2024-04-26T00:44:00.734 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.38774.log: /var/log/ceph/ceph-client.admin.35202.log 2024-04-26T00:44:00.734 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27069.log 2024-04-26T00:44:00.734 INFO:teuthology.orchestra.run.smithi037.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38774.log.gz 2024-04-26T00:44:00.734 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28322.log 2024-04-26T00:44:00.734 INFO:teuthology.orchestra.run.smithi037.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.31352.log.gz 2024-04-26T00:44:00.734 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.27069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39313.log 2024-04-26T00:44:00.734 INFO:teuthology.orchestra.run.smithi037.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.35202.log.gz 2024-04-26T00:44:00.734 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28322.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.27069.log.gz 2024-04-26T00:44:00.734 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26891.log 2024-04-26T00:44:00.735 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39803.log 2024-04-26T00:44:00.735 INFO:teuthology.orchestra.run.smithi037.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.28322.log.gz 2024-04-26T00:44:00.735 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28348.log 2024-04-26T00:44:00.735 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.26891.log: /var/log/ceph/ceph-client.admin.39803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31956.log 2024-04-26T00:44:00.735 INFO:teuthology.orchestra.run.smithi037.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.39313.log.gz 2024-04-26T00:44:00.735 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28348.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.26891.log.gz 2024-04-26T00:44:00.735 INFO:teuthology.orchestra.run.smithi037.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.39803.log.gz 2024-04-26T00:44:00.735 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37056.log 2024-04-26T00:44:00.780 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23883.log 2024-04-26T00:44:00.781 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-26T00:44:00.781 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.23883.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-26T00:44:00.781 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23859.log 2024-04-26T00:44:00.781 INFO:teuthology.orchestra.run.smithi045.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.23883.log.gz 2024-04-26T00:44:00.781 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23835.log 2024-04-26T00:44:00.781 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.23859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23811.log 2024-04-26T00:44:00.782 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.23835.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.23859.log.gz 2024-04-26T00:44:00.782 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.26579.log 2024-04-26T00:44:00.782 INFO:teuthology.orchestra.run.smithi045.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.23835.log.gz 2024-04-26T00:44:00.782 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-26T00:44:00.782 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.23811.log: /var/log/ceph/ceph-client.0.26579.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-26T00:44:00.782 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23931.log 2024-04-26T00:44:00.783 INFO:teuthology.orchestra.run.smithi045.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.23811.log.gz 2024-04-26T00:44:00.783 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-26T00:44:00.783 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.23931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23907.log 2024-04-26T00:44:00.783 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph.audit.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.23931.log.gz 2024-04-26T00:44:00.783 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-26T00:44:00.783 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-26T00:44:00.784 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.admin.23907.log: /var/log/ceph/ceph-mon.b.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.23907.log.gz 2024-04-26T00:44:00.784 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-osd.4.log: 89.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-26T00:44:00.791 INFO:teuthology.orchestra.run.smithi045.stderr: 87.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-26T00:44:00.851 INFO:teuthology.orchestra.run.smithi045.stderr: 91.8% -- replaced with /var/log/ceph/ceph-client.0.26579.log.gz 2024-04-26T00:44:00.868 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31956.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.28348.log.gz 2024-04-26T00:44:00.868 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33595.log 2024-04-26T00:44:00.868 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39459.log 2024-04-26T00:44:00.868 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37056.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.31956.log.gz 2024-04-26T00:44:00.868 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31148.log 2024-04-26T00:44:00.868 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33595.log: /var/log/ceph/ceph-client.admin.39459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30282.log 2024-04-26T00:44:00.868 INFO:teuthology.orchestra.run.smithi037.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.37056.log.gz 2024-04-26T00:44:00.869 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31148.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.33595.log.gz 2024-04-26T00:44:00.869 INFO:teuthology.orchestra.run.smithi037.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39459.log.gz 2024-04-26T00:44:00.869 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36405.log 2024-04-26T00:44:00.869 INFO:teuthology.orchestra.run.smithi037.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.31148.log.gz 2024-04-26T00:44:00.869 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39157.log 2024-04-26T00:44:00.869 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.36405.log: /var/log/ceph/ceph-client.admin.35992.log 2024-04-26T00:44:00.869 INFO:teuthology.orchestra.run.smithi037.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.30282.log.gz 2024-04-26T00:44:00.869 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37879.log 2024-04-26T00:44:00.869 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38262.log 2024-04-26T00:44:00.869 INFO:teuthology.orchestra.run.smithi037.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.39157.log.gz 2024-04-26T00:44:00.869 INFO:teuthology.orchestra.run.smithi037.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.36405.log.gz 2024-04-26T00:44:00.869 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40288.log 2024-04-26T00:44:00.869 INFO:teuthology.orchestra.run.smithi037.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35992.log.gz 2024-04-26T00:44:00.869 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34496.log 2024-04-26T00:44:00.870 INFO:teuthology.orchestra.run.smithi037.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.37879.log.gz 2024-04-26T00:44:00.870 INFO:teuthology.orchestra.run.smithi037.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38262.log.gz 2024-04-26T00:44:00.870 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29889.log 2024-04-26T00:44:00.870 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40288.log: /var/log/ceph/ceph-client.admin.34496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28088.log 2024-04-26T00:44:00.870 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27984.log 2024-04-26T00:44:00.870 INFO:teuthology.orchestra.run.smithi037.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40288.log.gz 2024-04-26T00:44:00.870 INFO:teuthology.orchestra.run.smithi037.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34496.log.gz 2024-04-26T00:44:00.870 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36848.log 2024-04-26T00:44:00.870 INFO:teuthology.orchestra.run.smithi037.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.29889.log.gz 2024-04-26T00:44:00.870 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.27984.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.28088.log.gz 2024-04-26T00:44:00.870 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39537.log 2024-04-26T00:44:00.870 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39209.log 2024-04-26T00:44:00.870 INFO:teuthology.orchestra.run.smithi037.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.27984.log.gz 2024-04-26T00:44:00.870 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26945.log 2024-04-26T00:44:00.871 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40496.log 2024-04-26T00:44:00.871 INFO:teuthology.orchestra.run.smithi037.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36848.log.gz 2024-04-26T00:44:00.871 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.26945.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.39537.log.gz 2024-04-26T00:44:00.871 INFO:teuthology.orchestra.run.smithi037.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.39209.log.gz 2024-04-26T00:44:00.871 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38800.log 2024-04-26T00:44:00.871 INFO:teuthology.orchestra.run.smithi037.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.26945.log.gz 2024-04-26T00:44:00.871 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33569.log 2024-04-26T00:44:00.871 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40496.log: /var/log/ceph/ceph-client.admin.38800.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-26T00:44:00.871 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30227.log 2024-04-26T00:44:00.871 INFO:teuthology.orchestra.run.smithi037.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40496.log.gz 2024-04-26T00:44:00.871 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38289.log 2024-04-26T00:44:00.871 INFO:teuthology.orchestra.run.smithi037.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.33569.log.gz 2024-04-26T00:44:00.871 INFO:teuthology.orchestra.run.smithi037.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.38800.log.gz 2024-04-26T00:44:00.871 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30966.log 2024-04-26T00:44:00.872 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39053.log 2024-04-26T00:44:00.872 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30966.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.30227.log.gz 2024-04-26T00:44:00.872 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28296.log 2024-04-26T00:44:00.872 INFO:teuthology.orchestra.run.smithi037.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38289.log.gz 2024-04-26T00:44:00.872 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35694.log 2024-04-26T00:44:00.872 INFO:teuthology.orchestra.run.smithi037.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.30966.log.gz 2024-04-26T00:44:00.872 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40080.log 2024-04-26T00:44:00.872 INFO:teuthology.orchestra.run.smithi037.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.39053.log.gz 2024-04-26T00:44:00.872 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30123.log 2024-04-26T00:44:00.872 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28140.log 2024-04-26T00:44:00.872 INFO:teuthology.orchestra.run.smithi037.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28296.log.gz 2024-04-26T00:44:00.872 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40080.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35694.log.gz 2024-04-26T00:44:00.872 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31721.log 2024-04-26T00:44:00.872 INFO:teuthology.orchestra.run.smithi037.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.40080.log.gz 2024-04-26T00:44:00.872 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27173.log 2024-04-26T00:44:00.873 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28140.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.30123.log.gz 2024-04-26T00:44:00.873 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29786.log 2024-04-26T00:44:00.873 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31721.log: /var/log/ceph/ceph-client.admin.27173.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.28140.log.gz 2024-04-26T00:44:00.873 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29010.log 2024-04-26T00:44:00.873 INFO:teuthology.orchestra.run.smithi037.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.31721.log.gz 2024-04-26T00:44:00.873 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32821.log 2024-04-26T00:44:00.873 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29786.log: /var/log/ceph/ceph-client.admin.29010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32717.log 2024-04-26T00:44:00.873 INFO:teuthology.orchestra.run.smithi037.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.27173.log.gz 2024-04-26T00:44:00.873 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33725.log 2024-04-26T00:44:00.873 INFO:teuthology.orchestra.run.smithi037.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.29786.log.gz 2024-04-26T00:44:00.873 INFO:teuthology.orchestra.run.smithi037.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.29010.log.gz 2024-04-26T00:44:00.873 INFO:teuthology.orchestra.run.smithi037.stderr:gzip/var/log/ceph/ceph-client.admin.32717.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.31586.log 2024-04-26T00:44:00.873 INFO:teuthology.orchestra.run.smithi037.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.32821.log.gz 2024-04-26T00:44:00.873 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29116.log 2024-04-26T00:44:00.874 INFO:teuthology.orchestra.run.smithi037.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.32717.log.gz 2024-04-26T00:44:00.874 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40106.log 2024-04-26T00:44:00.874 INFO:teuthology.orchestra.run.smithi037.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.31586.log.gz 2024-04-26T00:44:00.874 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose 89.5% -- /var/log/ceph/ceph-client.admin.27225.log -- replaced with /var/log/ceph/ceph-client.admin.33725.log.gz 2024-04-26T00:44:00.874 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:44:00.874 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29116.log: /var/log/ceph/ceph-client.admin.40106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33292.log 2024-04-26T00:44:00.874 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.27225.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.29116.log.gz 2024-04-26T00:44:00.874 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37108.log 2024-04-26T00:44:00.874 INFO:teuthology.orchestra.run.smithi037.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.40106.log.gz 2024-04-26T00:44:00.874 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31326.log 2024-04-26T00:44:00.874 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33292.log: /var/log/ceph/ceph-client.admin.37108.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.27225.log.gz 2024-04-26T00:44:00.874 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39563.log 2024-04-26T00:44:00.874 INFO:teuthology.orchestra.run.smithi037.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33292.log.gz 2024-04-26T00:44:00.874 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39699.log 2024-04-26T00:44:00.875 INFO:teuthology.orchestra.run.smithi037.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.37108.log.gz 2024-04-26T00:44:00.875 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36483.log 2024-04-26T00:44:00.875 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39563.log: /var/log/ceph/ceph-client.admin.39699.log: gzip -5 90.6% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.31326.log.gz /var/log/ceph/ceph-client.admin.29168.log 2024-04-26T00:44:00.875 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:44:00.875 INFO:teuthology.orchestra.run.smithi037.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39563.log.gz 2024-04-26T00:44:00.875 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38054.log 2024-04-26T00:44:00.875 INFO:teuthology.orchestra.run.smithi037.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39699.log.gz 2024-04-26T00:44:00.875 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34964.log 2024-04-26T00:44:00.875 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38054.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.29168.log.gz 2024-04-26T00:44:00.875 INFO:teuthology.orchestra.run.smithi037.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36483.log.gz 2024-04-26T00:44:00.875 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36535.log 2024-04-26T00:44:00.875 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30992.log 2024-04-26T00:44:00.875 INFO:teuthology.orchestra.run.smithi037.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.38054.log.gz 2024-04-26T00:44:00.875 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34132.log 2024-04-26T00:44:00.876 INFO:teuthology.orchestra.run.smithi037.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.34964.log.gz 2024-04-26T00:44:00.876 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35147.log 2024-04-26T00:44:00.876 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39339.log 2024-04-26T00:44:00.876 INFO:teuthology.orchestra.run.smithi037.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.36535.log.gz 2024-04-26T00:44:00.876 INFO:teuthology.orchestra.run.smithi037.stderr: 89.5%/var/log/ceph/ceph-client.admin.35147.log: -- replaced with /var/log/ceph/ceph-client.admin.30992.log.gz 2024-04-26T00:44:00.876 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36822.log 2024-04-26T00:44:00.876 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36193.log 2024-04-26T00:44:00.876 INFO:teuthology.orchestra.run.smithi037.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.34132.log.gz 2024-04-26T00:44:00.876 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39339.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.35147.log.gz 2024-04-26T00:44:00.876 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29142.log 2024-04-26T00:44:00.876 INFO:teuthology.orchestra.run.smithi037.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.39339.log.gz 2024-04-26T00:44:00.876 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36193.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.36822.log.gz 2024-04-26T00:44:00.876 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31878.log 2024-04-26T00:44:00.876 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33884.log 2024-04-26T00:44:00.876 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29142.log: /var/log/ceph/ceph-client.admin.31878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26388.log 2024-04-26T00:44:00.877 INFO:teuthology.orchestra.run.smithi037.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36193.log.gz 2024-04-26T00:44:00.877 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33884.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.29142.log.gz 2024-04-26T00:44:00.877 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27828.log 2024-04-26T00:44:00.877 INFO:teuthology.orchestra.run.smithi037.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.31878.log.gz 2024-04-26T00:44:00.877 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34522.log 2024-04-26T00:44:00.877 INFO:teuthology.orchestra.run.smithi037.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.33884.log.gz 2024-04-26T00:44:00.877 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.26388.log: /var/log/ceph/ceph-client.admin.27828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37238.log 2024-04-26T00:44:00.877 INFO:teuthology.orchestra.run.smithi037.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.26388.log.gz 2024-04-26T00:44:00.877 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28575.log 2024-04-26T00:44:00.877 INFO:teuthology.orchestra.run.smithi037.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.27828.log.gz 2024-04-26T00:44:00.877 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37238.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34522.log.gz 2024-04-26T00:44:00.877 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30686.log 2024-04-26T00:44:00.877 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38881.log 2024-04-26T00:44:00.877 INFO:teuthology.orchestra.run.smithi037.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37238.log.gz 2024-04-26T00:44:00.878 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37264.log 2024-04-26T00:44:00.878 INFO:teuthology.orchestra.run.smithi037.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.28575.log.gz 2024-04-26T00:44:00.878 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38881.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.30686.log.gz 2024-04-26T00:44:00.878 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28192.log 2024-04-26T00:44:00.878 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39105.log 2024-04-26T00:44:00.878 INFO:teuthology.orchestra.run.smithi037.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38881.log.gz 2024-04-26T00:44:00.878 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37082.log 2024-04-26T00:44:00.878 INFO:teuthology.orchestra.run.smithi037.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37264.log.gz 2024-04-26T00:44:00.878 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40028.log 2024-04-26T00:44:00.878 INFO:teuthology.orchestra.run.smithi037.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28192.log.gz 2024-04-26T00:44:00.878 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37082.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39105.log.gz 2024-04-26T00:44:00.878 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38722.log 2024-04-26T00:44:00.878 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37488.log 2024-04-26T00:44:00.878 INFO:teuthology.orchestra.run.smithi037.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.37082.log.gz 2024-04-26T00:44:00.879 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37134.log 2024-04-26T00:44:00.881 INFO:teuthology.orchestra.run.smithi037.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40028.log.gz 2024-04-26T00:44:00.881 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32086.log 2024-04-26T00:44:00.882 INFO:teuthology.orchestra.run.smithi037.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.38722.log.gz 2024-04-26T00:44:00.882 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36099.log 2024-04-26T00:44:00.882 INFO:teuthology.orchestra.run.smithi037.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37488.log.gz 2024-04-26T00:44:00.882 INFO:teuthology.orchestra.run.smithi037.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37134.log.gz 2024-04-26T00:44:00.882 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32847.log 2024-04-26T00:44:00.882 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38422.log 2024-04-26T00:44:00.882 INFO:teuthology.orchestra.run.smithi037.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32086.log.gz 2024-04-26T00:44:00.882 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34392.log 2024-04-26T00:44:00.882 INFO:teuthology.orchestra.run.smithi037.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.36099.log.gz 2024-04-26T00:44:00.882 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38422.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.32847.log.gz 2024-04-26T00:44:00.882 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33214.log 2024-04-26T00:44:00.882 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29037.log 2024-04-26T00:44:00.882 INFO:teuthology.orchestra.run.smithi037.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38422.log.gz 2024-04-26T00:44:00.882 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33214.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-26T00:44:00.883 INFO:teuthology.orchestra.run.smithi037.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.34392.log.gz 2024-04-26T00:44:00.883 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36326.log 2024-04-26T00:44:00.883 INFO:teuthology.orchestra.run.smithi037.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.33214.log.gz 2024-04-26T00:44:00.883 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31904.log 2024-04-26T00:44:00.883 INFO:teuthology.orchestra.run.smithi037.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.29037.log.gz 2024-04-26T00:44:00.883 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35016.log 2024-04-26T00:44:00.883 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31904.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36326.log.gz 2024-04-26T00:44:00.883 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.20149.log 2024-04-26T00:44:00.883 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39621.log 2024-04-26T00:44:00.883 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35016.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.31904.log.gz 2024-04-26T00:44:00.883 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph.tmp-client.admin.20149.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.20149.log.gz 2024-04-26T00:44:00.883 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36952.log 2024-04-26T00:44:00.883 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38184.log 2024-04-26T00:44:00.883 INFO:teuthology.orchestra.run.smithi037.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.35016.log.gz 2024-04-26T00:44:00.883 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27724.log 2024-04-26T00:44:00.884 INFO:teuthology.orchestra.run.smithi037.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.39621.log.gz 2024-04-26T00:44:00.884 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27254.log 2024-04-26T00:44:00.884 INFO:teuthology.orchestra.run.smithi037.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36952.log.gz 2024-04-26T00:44:00.884 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33949.log 2024-04-26T00:44:00.884 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.27724.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.38184.log.gz 2024-04-26T00:44:00.884 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.27254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30888.log 2024-04-26T00:44:00.884 INFO:teuthology.orchestra.run.smithi037.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.27724.log.gz 2024-04-26T00:44:00.884 INFO:teuthology.orchestra.run.smithi037.stderr:gzip/var/log/ceph/ceph-client.admin.33949.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.28114.log 2024-04-26T00:44:00.884 INFO:teuthology.orchestra.run.smithi037.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.27254.log.gz 2024-04-26T00:44:00.884 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35173.log 2024-04-26T00:44:00.884 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34314.log 2024-04-26T00:44:00.884 INFO:teuthology.orchestra.run.smithi037.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.33949.log.gz 2024-04-26T00:44:00.884 INFO:teuthology.orchestra.run.smithi037.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.30888.log.gz 2024-04-26T00:44:00.884 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35862.log 2024-04-26T00:44:00.885 INFO:teuthology.orchestra.run.smithi037.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.28114.log.gz 2024-04-26T00:44:00.885 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33855.log 2024-04-26T00:44:00.885 INFO:teuthology.orchestra.run.smithi037.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.35173.log.gz 2024-04-26T00:44:00.885 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29246.log 2024-04-26T00:44:00.885 INFO:teuthology.orchestra.run.smithi037.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.34314.log.gz 2024-04-26T00:44:00.885 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23928.log 2024-04-26T00:44:00.885 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29246.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35862.log.gz 2024-04-26T00:44:00.885 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35587.log 2024-04-26T00:44:00.885 INFO:teuthology.orchestra.run.smithi037.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.33855.log.gz 2024-04-26T00:44:00.885 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.23928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28679.log 2024-04-26T00:44:00.885 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35587.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.29246.log.gz 2024-04-26T00:44:00.885 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36431.log 2024-04-26T00:44:00.885 INFO:teuthology.orchestra.run.smithi037.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.23928.log.gz 2024-04-26T00:44:00.885 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28374.log 2024-04-26T00:44:00.886 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28679.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35587.log.gz 2024-04-26T00:44:00.886 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29402.log 2024-04-26T00:44:00.886 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28374.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.28679.log.gz 2024-04-26T00:44:00.886 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39131.log 2024-04-26T00:44:00.886 INFO:teuthology.orchestra.run.smithi037.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.36431.log.gz 2024-04-26T00:44:00.886 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29432.log 2024-04-26T00:44:00.886 INFO:teuthology.orchestra.run.smithi037.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.28374.log.gz 2024-04-26T00:44:00.886 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28549.log 2024-04-26T00:44:00.886 INFO:teuthology.orchestra.run.smithi037.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.29402.log.gz 2024-04-26T00:44:00.886 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-26T00:44:00.886 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29432.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39131.log.gz 2024-04-26T00:44:00.886 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32060.log 2024-04-26T00:44:00.886 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30862.log 2024-04-26T00:44:00.886 INFO:teuthology.orchestra.run.smithi037.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.28549.log.gz 2024-04-26T00:44:00.886 INFO:teuthology.orchestra.run.smithi037.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.29432.log.gz 2024-04-26T00:44:00.887 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31430.log 2024-04-26T00:44:00.887 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32925.log 2024-04-26T00:44:00.887 INFO:teuthology.orchestra.run.smithi037.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32060.log.gz 2024-04-26T00:44:00.887 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26802.log 2024-04-26T00:44:00.887 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31430.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.30862.log.gz 2024-04-26T00:44:00.887 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26584.log 2024-04-26T00:44:00.887 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.26802.log: 89.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31612.log 2024-04-26T00:44:00.887 INFO:teuthology.orchestra.run.smithi037.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31430.log.gz 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32925.log.gz 2024-04-26T00:44:00.887 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:44:00.887 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.26584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34730.log 2024-04-26T00:44:00.887 INFO:teuthology.orchestra.run.smithi037.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.26802.log.gz 2024-04-26T00:44:00.887 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34574.log 2024-04-26T00:44:00.887 INFO:teuthology.orchestra.run.smithi037.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.26584.log.gz/var/log/ceph/ceph-client.admin.34730.log: 2024-04-26T00:44:00.887 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30530.log 2024-04-26T00:44:00.888 INFO:teuthology.orchestra.run.smithi037.stderr: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.31612.log.gz 2024-04-26T00:44:00.888 INFO:teuthology.orchestra.run.smithi037.stderr:gzip/var/log/ceph/ceph-client.admin.34574.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.35405.log 2024-04-26T00:44:00.888 INFO:teuthology.orchestra.run.smithi037.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.34730.log.gz 2024-04-26T00:44:00.888 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34938.log 2024-04-26T00:44:00.888 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40522.log 2024-04-26T00:44:00.888 INFO:teuthology.orchestra.run.smithi037.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.34574.log.gz 2024-04-26T00:44:00.888 INFO:teuthology.orchestra.run.smithi037.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.30530.log.gz 2024-04-26T00:44:00.888 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34938.log: gzip 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35405.log.gz 2024-04-26T00:44:00.888 INFO:teuthology.orchestra.run.smithi037.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.33188.log 2024-04-26T00:44:00.888 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5/var/log/ceph/ceph-client.admin.40522.log: --verbose -- /var/log/ceph/ceph-client.admin.30149.log 2024-04-26T00:44:00.888 INFO:teuthology.orchestra.run.smithi037.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34938.log.gz 2024-04-26T00:44:00.888 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38080.log 2024-04-26T00:44:00.888 INFO:teuthology.orchestra.run.smithi037.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40522.log.gz 2024-04-26T00:44:00.888 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 89.6% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.33188.log.gz 2024-04-26T00:44:00.888 INFO:teuthology.orchestra.run.smithi037.stderr: /var/log/ceph/ceph-client.admin.34652.log 2024-04-26T00:44:00.889 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30149.log: /var/log/ceph/ceph-client.admin.38080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37827.log 2024-04-26T00:44:00.889 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34652.log: gzip 88.4% -5 -- replaced with /var/log/ceph/ceph-client.admin.30149.log.gz --verbose 2024-04-26T00:44:00.889 INFO:teuthology.orchestra.run.smithi037.stderr: -- /var/log/ceph/ceph-client.admin.29706.log 2024-04-26T00:44:00.889 INFO:teuthology.orchestra.run.smithi037.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.38080.log.gz 2024-04-26T00:44:00.889 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26500.log 2024-04-26T00:44:00.889 INFO:teuthology.orchestra.run.smithi037.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34652.log.gz 2024-04-26T00:44:00.889 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32978.log 2024-04-26T00:44:00.889 INFO:teuthology.orchestra.run.smithi037.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37827.log.gz 2024-04-26T00:44:00.889 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.26500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31456.log 2024-04-26T00:44:00.889 INFO:teuthology.orchestra.run.smithi037.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29706.log.gz 2024-04-26T00:44:00.889 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40574.log 2024-04-26T00:44:00.889 INFO:teuthology.orchestra.run.smithi037.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.26500.log.gz 2024-04-26T00:44:00.889 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31378.log 2024-04-26T00:44:00.889 INFO:teuthology.orchestra.run.smithi037.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.32978.log.gz 2024-04-26T00:44:00.889 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38592.log 2024-04-26T00:44:00.890 INFO:teuthology.orchestra.run.smithi037.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.31456.log.gz 2024-04-26T00:44:00.890 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37936.log 2024-04-26T00:44:00.890 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31378.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40574.log.gz 2024-04-26T00:44:00.890 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31122.log 2024-04-26T00:44:00.890 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37394.log 2024-04-26T00:44:00.890 INFO:teuthology.orchestra.run.smithi037.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.31378.log.gz 2024-04-26T00:44:00.890 INFO:teuthology.orchestra.run.smithi037.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.38592.log.gz 2024-04-26T00:44:00.890 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27095.log 2024-04-26T00:44:00.890 INFO:teuthology.orchestra.run.smithi037.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37936.log.gz 2024-04-26T00:44:00.890 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40548.log 2024-04-26T00:44:00.890 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37394.log: /var/log/ceph/ceph-client.admin.27095.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.31122.log.gz 2024-04-26T00:44:00.890 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29063.log 2024-04-26T00:44:00.890 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32346.log 2024-04-26T00:44:00.890 INFO:teuthology.orchestra.run.smithi037.stderr: 90.2%/var/log/ceph/ceph-client.admin.40548.log: -- replaced with /var/log/ceph/ceph-client.admin.37394.log.gz 2024-04-26T00:44:00.891 INFO:teuthology.orchestra.run.smithi037.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.27095.log.gz 2024-04-26T00:44:00.891 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35120.log 2024-04-26T00:44:00.891 INFO:teuthology.orchestra.run.smithi037.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40548.log.gz 2024-04-26T00:44:00.891 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28244.log 2024-04-26T00:44:00.891 INFO:teuthology.orchestra.run.smithi037.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.29063.log.gz 2024-04-26T00:44:00.891 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27620.log 2024-04-26T00:44:00.891 INFO:teuthology.orchestra.run.smithi037.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.32346.log.gz 2024-04-26T00:44:00.891 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35535.log 2024-04-26T00:44:00.891 INFO:teuthology.orchestra.run.smithi037.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.35120.log.gz 2024-04-26T00:44:00.891 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.27620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37697.log 2024-04-26T00:44:00.891 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26919.log 2024-04-26T00:44:00.891 INFO:teuthology.orchestra.run.smithi037.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.28244.log.gz 2024-04-26T00:44:00.891 INFO:teuthology.orchestra.run.smithi037.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.27620.log.gz 2024-04-26T00:44:00.891 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39673.log 2024-04-26T00:44:00.892 INFO:teuthology.orchestra.run.smithi037.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35535.log.gz 2024-04-26T00:44:00.892 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.26919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39000.log 2024-04-26T00:44:00.892 INFO:teuthology.orchestra.run.smithi037.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37697.log.gz 2024-04-26T00:44:00.892 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38696.log 2024-04-26T00:44:00.892 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39673.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.26919.log.gz 2024-04-26T00:44:00.892 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32510.log 2024-04-26T00:44:00.892 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31695.log 2024-04-26T00:44:00.892 INFO:teuthology.orchestra.run.smithi037.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.39673.log.gz 2024-04-26T00:44:00.892 INFO:teuthology.orchestra.run.smithi037.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.39000.log.gz 2024-04-26T00:44:00.892 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27304.log 2024-04-26T00:44:00.892 INFO:teuthology.orchestra.run.smithi037.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.38696.log.gz 2024-04-26T00:44:00.892 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28521.log 2024-04-26T00:44:00.892 INFO:teuthology.orchestra.run.smithi037.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31695.log.gz 2024-04-26T00:44:00.892 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36248.log 2024-04-26T00:44:00.892 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.27304.log: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.32510.log.gz 2024-04-26T00:44:00.893 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28601.log 2024-04-26T00:44:00.893 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34912.log 2024-04-26T00:44:00.893 INFO:teuthology.orchestra.run.smithi037.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.27304.log.gz 2024-04-26T00:44:00.893 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28601.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28521.log.gz 2024-04-26T00:44:00.893 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26580.log 2024-04-26T00:44:00.893 INFO:teuthology.orchestra.run.smithi037.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36248.log.gz 2024-04-26T00:44:00.893 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29812.log 2024-04-26T00:44:00.893 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34912.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.28601.log.gz 2024-04-26T00:44:00.893 INFO:teuthology.orchestra.run.smithi037.stderr:gzip/var/log/ceph/ceph-client.admin.26580.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30348.log 2024-04-26T00:44:00.893 INFO:teuthology.orchestra.run.smithi037.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34912.log.gz 2024-04-26T00:44:00.893 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39751.log 2024-04-26T00:44:00.893 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29550.log 2024-04-26T00:44:00.893 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30348.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.26580.log.gz 2024-04-26T00:44:00.893 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39751.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.29812.log.gz 2024-04-26T00:44:00.893 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27880.log 2024-04-26T00:44:00.894 INFO:teuthology.orchestra.run.smithi037.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.30348.log.gz 2024-04-26T00:44:00.894 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26586.log 2024-04-26T00:44:00.894 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27147.log 2024-04-26T00:44:00.894 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.27880.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.39751.log.gz 2024-04-26T00:44:00.894 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.26586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35274.log 2024-04-26T00:44:00.894 INFO:teuthology.orchestra.run.smithi037.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.29550.log.gz 2024-04-26T00:44:00.894 INFO:teuthology.orchestra.run.smithi037.stderr: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35431.log 2024-04-26T00:44:00.894 INFO:teuthology.orchestra.run.smithi037.stderr: -- replaced with /var/log/ceph/ceph-client.admin.27880.log.gz 2024-04-26T00:44:00.894 INFO:teuthology.orchestra.run.smithi037.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.26586.log.gz 2024-04-26T00:44:00.894 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.27147.log: /var/log/ceph/ceph-client.admin.35274.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35274.log.gz 2024-04-26T00:44:00.894 INFO:teuthology.orchestra.run.smithi037.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.27147.log.gz 2024-04-26T00:44:00.894 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28218.log 2024-04-26T00:44:00.894 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32665.log 2024-04-26T00:44:00.894 INFO:teuthology.orchestra.run.smithi037.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35431.log.gz 2024-04-26T00:44:00.895 INFO:teuthology.orchestra.run.smithi037.stderr:gzip/var/log/ceph/ceph-client.admin.28218.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.31404.log 2024-04-26T00:44:00.895 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37342.log 2024-04-26T00:44:00.895 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37801.log 2024-04-26T00:44:00.895 INFO:teuthology.orchestra.run.smithi037.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.28218.log.gz 2024-04-26T00:44:00.895 INFO:teuthology.orchestra.run.smithi037.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.32665.log.gz 2024-04-26T00:44:00.895 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35725.log 2024-04-26T00:44:00.895 INFO:teuthology.orchestra.run.smithi037.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.31404.log.gz 2024-04-26T00:44:00.895 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39079.log 2024-04-26T00:44:00.895 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35725.log: gzip 90.5% -5 -- replaced with /var/log/ceph/ceph-client.admin.37342.log.gz 2024-04-26T00:44:00.895 INFO:teuthology.orchestra.run.smithi037.stderr: --verbose -- /var/log/ceph/ceph-client.admin.33829.log 2024-04-26T00:44:00.895 INFO:teuthology.orchestra.run.smithi037.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37801.log.gz 2024-04-26T00:44:00.895 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38393.log 2024-04-26T00:44:00.895 INFO:teuthology.orchestra.run.smithi037.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35725.log.gz 2024-04-26T00:44:00.895 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32743.log 2024-04-26T00:44:00.896 INFO:teuthology.orchestra.run.smithi037.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.39079.log.gz 2024-04-26T00:44:00.896 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40392.log 2024-04-26T00:44:00.896 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32743.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.33829.log.gz 2024-04-26T00:44:00.896 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27776.log 2024-04-26T00:44:00.896 INFO:teuthology.orchestra.run.smithi037.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38393.log.gz 2024-04-26T00:44:00.896 INFO:teuthology.orchestra.run.smithi037.stderr:gzip/var/log/ceph/ceph-client.admin.40392.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.33084.log 2024-04-26T00:44:00.896 INFO:teuthology.orchestra.run.smithi037.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.32743.log.gz 2024-04-26T00:44:00.896 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.27776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27043.log 2024-04-26T00:44:00.896 INFO:teuthology.orchestra.run.smithi037.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40392.log.gz 2024-04-26T00:44:00.896 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35613.log 2024-04-26T00:44:00.896 INFO:teuthology.orchestra.run.smithi037.stderr: 89.8%/var/log/ceph/ceph-client.admin.27043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36274.log 2024-04-26T00:44:00.896 INFO:teuthology.orchestra.run.smithi037.stderr: -- replaced with /var/log/ceph/ceph-client.admin.27776.log.gz 2024-04-26T00:44:00.896 INFO:teuthology.orchestra.run.smithi037.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.33084.log.gz 2024-04-26T00:44:00.896 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36744.log 2024-04-26T00:44:00.897 INFO:teuthology.orchestra.run.smithi037.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.27043.log.gz 2024-04-26T00:44:00.897 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26741.log 2024-04-26T00:44:00.897 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39235.log 2024-04-26T00:44:00.897 INFO:teuthology.orchestra.run.smithi037.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.35613.log.gz 2024-04-26T00:44:00.897 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.26741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23854.log 2024-04-26T00:44:00.897 INFO:teuthology.orchestra.run.smithi037.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.36274.log.gz 2024-04-26T00:44:00.897 INFO:teuthology.orchestra.run.smithi037.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36744.log.gz 2024-04-26T00:44:00.897 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34002.log 2024-04-26T00:44:00.897 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39235.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.26741.log.gz 2024-04-26T00:44:00.897 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35665.log 2024-04-26T00:44:00.897 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.23854.log: /var/log/ceph/ceph-client.admin.34002.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.39235.log.gz 2024-04-26T00:44:00.897 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30608.log 2024-04-26T00:44:00.897 INFO:teuthology.orchestra.run.smithi037.stderr: 88.8%/var/log/ceph/ceph-client.admin.35665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33621.log 2024-04-26T00:44:00.897 INFO:teuthology.orchestra.run.smithi037.stderr: -- replaced with /var/log/ceph/ceph-client.admin.23854.log.gz 2024-04-26T00:44:00.897 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35966.log 2024-04-26T00:44:00.898 INFO:teuthology.orchestra.run.smithi037.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.34002.log.gz 2024-04-26T00:44:00.898 INFO:teuthology.orchestra.run.smithi037.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.35665.log.gz 2024-04-26T00:44:00.898 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26577.log 2024-04-26T00:44:00.898 INFO:teuthology.orchestra.run.smithi037.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.30608.log.gz 2024-04-26T00:44:00.898 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33621.log: /var/log/ceph/ceph-client.admin.35966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38132.log 2024-04-26T00:44:00.898 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.26577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37905.log 2024-04-26T00:44:00.898 INFO:teuthology.orchestra.run.smithi037.stderr: 88.0% 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33621.log.gz 2024-04-26T00:44:00.898 INFO:teuthology.orchestra.run.smithi037.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35966.log.gz 2024-04-26T00:44:00.898 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30914.log 2024-04-26T00:44:00.898 INFO:teuthology.orchestra.run.smithi037.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.26577.log.gz 2024-04-26T00:44:00.898 INFO:teuthology.orchestra.run.smithi037.stderr:gzip/var/log/ceph/ceph-client.admin.37905.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.32456.log 2024-04-26T00:44:00.898 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30914.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38132.log.gz 2024-04-26T00:44:00.898 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30374.log 2024-04-26T00:44:00.898 INFO:teuthology.orchestra.run.smithi037.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.37905.log.gz 2024-04-26T00:44:00.899 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30400.log 2024-04-26T00:44:00.899 INFO:teuthology.orchestra.run.smithi037.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.30914.log.gz 2024-04-26T00:44:00.899 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28401.log 2024-04-26T00:44:00.899 INFO:teuthology.orchestra.run.smithi037.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.32456.log.gz 2024-04-26T00:44:00.899 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28705.log 2024-04-26T00:44:00.899 INFO:teuthology.orchestra.run.smithi037.stderr: 89.3%/var/log/ceph/ceph-client.admin.28401.log: -- replaced with /var/log/ceph/ceph-client.admin.30374.log.gz 2024-04-26T00:44:00.899 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30478.log 2024-04-26T00:44:00.899 INFO:teuthology.orchestra.run.smithi037.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.30400.log.gz 2024-04-26T00:44:00.899 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33110.log 2024-04-26T00:44:00.899 INFO:teuthology.orchestra.run.smithi037.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.28401.log.gz 2024-04-26T00:44:00.899 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34444.log 2024-04-26T00:44:00.899 INFO:teuthology.orchestra.run.smithi037.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.28705.log.gz 2024-04-26T00:44:00.899 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29654.log 2024-04-26T00:44:00.899 INFO:teuthology.orchestra.run.smithi037.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.30478.log.gz 2024-04-26T00:44:00.899 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40626.log 2024-04-26T00:44:00.900 INFO:teuthology.orchestra.run.smithi037.stderr: 91.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28010.log 2024-04-26T00:44:00.900 INFO:teuthology.orchestra.run.smithi037.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33110.log.gz 2024-04-26T00:44:00.900 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29654.log: /var/log/ceph/ceph-client.admin.40626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40210.log 2024-04-26T00:44:00.900 INFO:teuthology.orchestra.run.smithi037.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.34444.log.gz 2024-04-26T00:44:00.900 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-26T00:44:00.900 INFO:teuthology.orchestra.run.smithi037.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.29654.log.gz 2024-04-26T00:44:00.900 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28010.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.40626.log.gz 2024-04-26T00:44:00.900 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39647.log 2024-04-26T00:44:00.900 INFO:teuthology.orchestra.run.smithi037.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.28010.log.gz 2024-04-26T00:44:00.900 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38974.log 2024-04-26T00:44:00.900 INFO:teuthology.orchestra.run.smithi037.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40210.log.gz 2024-04-26T00:44:00.900 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33396.log 2024-04-26T00:44:00.900 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27435.log 2024-04-26T00:44:00.900 INFO:teuthology.orchestra.run.smithi037.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.39647.log.gz 2024-04-26T00:44:00.901 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37212.log 2024-04-26T00:44:00.901 INFO:teuthology.orchestra.run.smithi037.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38974.log.gz 2024-04-26T00:44:00.901 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28270.log 2024-04-26T00:44:00.901 INFO:teuthology.orchestra.run.smithi037.stderr: 89.6%/var/log/ceph/ceph-client.admin.27435.log: -- replaced with /var/log/ceph/ceph-client.admin.33396.log.gz 2024-04-26T00:44:00.901 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32562.log 2024-04-26T00:44:00.901 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32691.log 2024-04-26T00:44:00.901 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32562.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.37212.log.gz 89.0% -- replaced with /var/log/ceph/ceph-client.admin.27435.log.gz 2024-04-26T00:44:00.901 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:44:00.901 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40184.log 2024-04-26T00:44:00.901 INFO:teuthology.orchestra.run.smithi037.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.28270.log.gz 2024-04-26T00:44:00.901 INFO:teuthology.orchestra.run.smithi037.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.32562.log.gz 2024-04-26T00:44:00.901 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32952.log 2024-04-26T00:44:00.901 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35888.log 2024-04-26T00:44:00.901 INFO:teuthology.orchestra.run.smithi037.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.32691.log.gz 2024-04-26T00:44:00.901 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31096.log 2024-04-26T00:44:00.902 INFO:teuthology.orchestra.run.smithi037.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40184.log.gz 2024-04-26T00:44:00.902 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35888.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.32952.log.gz 2024-04-26T00:44:00.902 INFO:teuthology.orchestra.run.smithi037.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.35888.log.gz 2024-04-26T00:44:00.902 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28427.log 2024-04-26T00:44:00.902 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35068.log 2024-04-26T00:44:00.902 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28036.log 2024-04-26T00:44:00.902 INFO:teuthology.orchestra.run.smithi037.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.31096.log.gz 2024-04-26T00:44:00.902 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35068.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.28427.log.gz 2024-04-26T00:44:00.902 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29732.log 2024-04-26T00:44:00.902 INFO:teuthology.orchestra.run.smithi037.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35068.log.gz 2024-04-26T00:44:00.902 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39881.log/var/log/ceph/ceph-client.admin.28036.log: 2024-04-26T00:44:00.902 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39485.log 2024-04-26T00:44:00.902 INFO:teuthology.orchestra.run.smithi037.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28036.log.gz/var/log/ceph/ceph-client.admin.39881.log: 2024-04-26T00:44:00.902 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35639.log 2024-04-26T00:44:00.903 INFO:teuthology.orchestra.run.smithi037.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.29732.log.gz 2024-04-26T00:44:00.903 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39485.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39881.log.gz 2024-04-26T00:44:00.903 INFO:teuthology.orchestra.run.smithi037.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.39485.log.gz 2024-04-26T00:44:00.903 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31204.log 2024-04-26T00:44:00.903 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35301.log 2024-04-26T00:44:00.903 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34028.log 2024-04-26T00:44:00.903 INFO:teuthology.orchestra.run.smithi037.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.35639.log.gz 2024-04-26T00:44:00.903 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32372.log 2024-04-26T00:44:00.903 INFO:teuthology.orchestra.run.smithi037.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.31204.log.gz 2024-04-26T00:44:00.903 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27514.log 2024-04-26T00:44:00.903 INFO:teuthology.orchestra.run.smithi037.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35301.log.gz 2024-04-26T00:44:00.903 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34028.log: gzip/var/log/ceph/ceph-client.admin.32372.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.33318.log 2024-04-26T00:44:00.903 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31044.log 2024-04-26T00:44:00.903 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.27514.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.32372.log.gz 2024-04-26T00:44:00.904 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33318.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.34028.log.gz 2024-04-26T00:44:00.904 INFO:teuthology.orchestra.run.smithi037.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33318.log.gz 2024-04-26T00:44:00.904 INFO:teuthology.orchestra.run.smithi037.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.27514.log.gz 2024-04-26T00:44:00.904 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38210.log 2024-04-26T00:44:00.904 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33490.log 2024-04-26T00:44:00.904 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36587.log 2024-04-26T00:44:00.904 INFO:teuthology.orchestra.run.smithi037.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31044.log.gz 2024-04-26T00:44:00.904 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38341.log 2024-04-26T00:44:00.904 INFO:teuthology.orchestra.run.smithi037.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.38210.log.gz 2024-04-26T00:44:00.904 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33004.log 2024-04-26T00:44:00.904 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36587.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.33490.log.gz 2024-04-26T00:44:00.904 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29680.log 2024-04-26T00:44:00.904 INFO:teuthology.orchestra.run.smithi037.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.36587.log.gz 2024-04-26T00:44:00.904 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36222.log 2024-04-26T00:44:00.904 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33004.log: /var/log/ceph/ceph-client.admin.29680.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.38341.log.gz 2024-04-26T00:44:00.905 INFO:teuthology.orchestra.run.smithi037.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.33004.log.gz 2024-04-26T00:44:00.905 INFO:teuthology.orchestra.run.smithi037.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.29680.log.gz 2024-04-26T00:44:00.905 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39829.log 2024-04-26T00:44:00.905 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30253.log 2024-04-26T00:44:00.905 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36614.log 2024-04-26T00:44:00.905 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34600.log 2024-04-26T00:44:00.905 INFO:teuthology.orchestra.run.smithi037.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36222.log.gz 2024-04-26T00:44:00.905 INFO:teuthology.orchestra.run.smithi037.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39829.log.gz 2024-04-26T00:44:00.905 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34262.log 2024-04-26T00:44:00.905 INFO:teuthology.orchestra.run.smithi037.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.30253.log.gz 2024-04-26T00:44:00.905 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32242.log 2024-04-26T00:44:00.905 INFO:teuthology.orchestra.run.smithi037.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36614.log.gz 2024-04-26T00:44:00.905 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36018.log 2024-04-26T00:44:00.905 INFO:teuthology.orchestra.run.smithi037.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.34600.log.gz 2024-04-26T00:44:00.905 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29993.log 2024-04-26T00:44:00.906 INFO:teuthology.orchestra.run.smithi037.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34262.log.gz 2024-04-26T00:44:00.906 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32242.log: /var/log/ceph/ceph-client.admin.36018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34886.log 2024-04-26T00:44:00.906 INFO:teuthology.orchestra.run.smithi037.stderr: 89.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30452.log 2024-04-26T00:44:00.906 INFO:teuthology.orchestra.run.smithi037.stderr: -- replaced with /var/log/ceph/ceph-client.admin.32242.log.gz 2024-04-26T00:44:00.906 INFO:teuthology.orchestra.run.smithi037.stderr: 87.6%/var/log/ceph/ceph-client.admin.29993.log: -- replaced with /var/log/ceph/ceph-client.admin.36018.log.gz 2024-04-26T00:44:00.906 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34886.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29993.log.gz 2024-04-26T00:44:00.906 INFO:teuthology.orchestra.run.smithi037.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.34886.log.gz 2024-04-26T00:44:00.906 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34236.log 2024-04-26T00:44:00.906 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27854.log 2024-04-26T00:44:00.906 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33162.log 2024-04-26T00:44:00.906 INFO:teuthology.orchestra.run.smithi037.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.30452.log.gz 2024-04-26T00:44:00.906 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39396.log 2024-04-26T00:44:00.906 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.27854.log: /var/log/ceph/ceph-client.admin.33162.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34236.log.gz 2024-04-26T00:44:00.906 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30810.log 2024-04-26T00:44:00.907 INFO:teuthology.orchestra.run.smithi037.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.27854.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30556.log 2024-04-26T00:44:00.907 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:44:00.907 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39396.log: /var/log/ceph/ceph-client.admin.30810.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33162.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27356.log 2024-04-26T00:44:00.907 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:44:00.907 INFO:teuthology.orchestra.run.smithi037.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.39396.log.gz 2024-04-26T00:44:00.907 INFO:teuthology.orchestra.run.smithi037.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.30810.log.gz 2024-04-26T00:44:00.907 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30556.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.30556.log.gz 2024-04-26T00:44:00.907 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30045.log 2024-04-26T00:44:00.907 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.27356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34626.log 2024-04-26T00:44:00.907 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28456.log 2024-04-26T00:44:00.907 INFO:teuthology.orchestra.run.smithi037.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.27356.log.gz/var/log/ceph/ceph-client.admin.34626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33543.log 2024-04-26T00:44:00.907 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:44:00.907 INFO:teuthology.orchestra.run.smithi037.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.30045.log.gz 2024-04-26T00:44:00.907 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28456.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34626.log.gz 2024-04-26T00:44:00.908 INFO:teuthology.orchestra.run.smithi037.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.28456.log.gz 2024-04-26T00:44:00.908 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36073.log 2024-04-26T00:44:00.908 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35509.log 2024-04-26T00:44:00.908 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40470.log 2024-04-26T00:44:00.908 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28062.log 2024-04-26T00:44:00.908 INFO:teuthology.orchestra.run.smithi037.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.33543.log.gz 2024-04-26T00:44:00.908 INFO:teuthology.orchestra.run.smithi037.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.36073.log.gz/var/log/ceph/ceph-client.admin.40470.log: 2024-04-26T00:44:00.908 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39511.log 2024-04-26T00:44:00.908 INFO:teuthology.orchestra.run.smithi037.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.35509.log.gz 2024-04-26T00:44:00.908 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.28062.log: -- /var/log/ceph/ceph-client.admin.37567.log 2024-04-26T00:44:00.908 INFO:teuthology.orchestra.run.smithi037.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40470.log.gz/var/log/ceph/ceph-client.admin.39511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36300.log 2024-04-26T00:44:00.908 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:44:00.908 INFO:teuthology.orchestra.run.smithi037.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28062.log.gz 2024-04-26T00:44:00.908 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37567.log: gzip 90.1% -5 -- replaced with /var/log/ceph/ceph-client.admin.39511.log.gz --verbose 2024-04-26T00:44:00.908 INFO:teuthology.orchestra.run.smithi037.stderr: -- /var/log/ceph/ceph-client.admin.31799.log 2024-04-26T00:44:00.909 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36300.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37567.log.gz 2024-04-26T00:44:00.909 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37853.log 2024-04-26T00:44:00.909 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-26T00:44:00.909 INFO:teuthology.orchestra.run.smithi037.stderr: 90.2%/var/log/ceph/ceph-client.admin.31799.log: -- replaced with /var/log/ceph/ceph-client.admin.36300.log.gz 2024-04-26T00:44:00.909 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29576.log 2024-04-26T00:44:00.909 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32112.log 2024-04-26T00:44:00.909 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29576.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37853.log.gz 2024-04-26T00:44:00.909 INFO:teuthology.orchestra.run.smithi037.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.31799.log.gz 2024-04-26T00:44:00.909 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31508.log 2024-04-26T00:44:00.909 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32112.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.29576.log.gz 2024-04-26T00:44:00.909 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36926.log 2024-04-26T00:44:00.909 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31508.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32112.log.gz 2024-04-26T00:44:00.909 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37004.log 2024-04-26T00:44:00.909 INFO:teuthology.orchestra.run.smithi037.stderr: 89.5%/var/log/ceph/ceph-client.admin.36926.log: -- replaced with /var/log/ceph/ceph-client.admin.31508.log.gzgzip 2024-04-26T00:44:00.909 INFO:teuthology.orchestra.run.smithi037.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.36692.log 2024-04-26T00:44:00.910 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37004.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37004.log.gz 2024-04-26T00:44:00.910 INFO:teuthology.orchestra.run.smithi037.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.36926.log.gz 2024-04-26T00:44:00.910 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28731.log 2024-04-26T00:44:00.910 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26740.log 2024-04-26T00:44:00.910 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38826.log 2024-04-26T00:44:00.910 INFO:teuthology.orchestra.run.smithi037.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.36692.log.gz 2024-04-26T00:44:00.910 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.26740.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.28731.log.gz 2024-04-26T00:44:00.910 INFO:teuthology.orchestra.run.smithi037.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.26740.log.gz 2024-04-26T00:44:00.910 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27672.log 2024-04-26T00:44:00.910 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38236.log 2024-04-26T00:44:00.910 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.27672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34340.log 2024-04-26T00:44:00.910 INFO:teuthology.orchestra.run.smithi037.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.38826.log.gz 2024-04-26T00:44:00.910 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38236.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.27672.log.gz 2024-04-26T00:44:00.911 INFO:teuthology.orchestra.run.smithi037.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38236.log.gz 2024-04-26T00:44:00.911 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37316.log 2024-04-26T00:44:00.911 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26745.log 2024-04-26T00:44:00.911 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40158.log 2024-04-26T00:44:00.911 INFO:teuthology.orchestra.run.smithi037.stderr: 89.5%/var/log/ceph/ceph-client.admin.26745.log: -- replaced with /var/log/ceph/ceph-client.admin.34340.log.gz 2024-04-26T00:44:00.911 INFO:teuthology.orchestra.run.smithi037.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37316.log.gz 2024-04-26T00:44:00.911 INFO:teuthology.orchestra.run.smithi037.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.26745.log.gz 2024-04-26T00:44:00.911 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32482.log 2024-04-26T00:44:00.911 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32639.log 2024-04-26T00:44:00.911 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29272.log 2024-04-26T00:44:00.911 INFO:teuthology.orchestra.run.smithi037.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40158.log.gz 2024-04-26T00:44:00.911 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32639.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.32482.log.gz 2024-04-26T00:44:00.911 INFO:teuthology.orchestra.run.smithi037.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.32639.log.gz 2024-04-26T00:44:00.911 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34834.log 2024-04-26T00:44:00.911 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39261.log 2024-04-26T00:44:00.912 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31560.log 2024-04-26T00:44:00.912 INFO:teuthology.orchestra.run.smithi037.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.29272.log.gz 2024-04-26T00:44:00.912 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39261.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.34834.log.gz 2024-04-26T00:44:00.912 INFO:teuthology.orchestra.run.smithi037.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39261.log.gz 2024-04-26T00:44:00.912 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31826.log 2024-04-26T00:44:00.912 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34860.log 2024-04-26T00:44:00.912 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38947.log 2024-04-26T00:44:00.912 INFO:teuthology.orchestra.run.smithi037.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.31560.log.gz 2024-04-26T00:44:00.912 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34860.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31826.log.gz 2024-04-26T00:44:00.912 INFO:teuthology.orchestra.run.smithi037.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.34860.log.gz 2024-04-26T00:44:00.912 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29497.log 2024-04-26T00:44:00.912 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33803.log 2024-04-26T00:44:00.912 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29602.log 2024-04-26T00:44:00.912 INFO:teuthology.orchestra.run.smithi037.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38947.log.gz 2024-04-26T00:44:00.912 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33803.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.29497.log.gz 2024-04-26T00:44:00.913 INFO:teuthology.orchestra.run.smithi037.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.33803.log.gz 2024-04-26T00:44:00.913 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35042.log 2024-04-26T00:44:00.958 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37186.log 2024-04-26T00:44:00.958 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27390.log 2024-04-26T00:44:00.958 INFO:teuthology.orchestra.run.smithi037.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.29602.log.gz 2024-04-26T00:44:00.958 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37186.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35042.log.gz 2024-04-26T00:44:00.958 INFO:teuthology.orchestra.run.smithi037.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.37186.log.gz 2024-04-26T00:44:00.958 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40236.log 2024-04-26T00:44:00.958 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.27390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37593.log 2024-04-26T00:44:00.958 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34158.log 2024-04-26T00:44:00.958 INFO:teuthology.orchestra.run.smithi037.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.27390.log.gz 2024-04-26T00:44:00.959 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37593.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40236.log.gz 2024-04-26T00:44:00.959 INFO:teuthology.orchestra.run.smithi037.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.37593.log.gz 2024-04-26T00:44:00.959 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35914.log 2024-04-26T00:44:00.959 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29220.log 2024-04-26T00:44:00.959 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37540.log 2024-04-26T00:44:00.959 INFO:teuthology.orchestra.run.smithi037.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.34158.log.gz 2024-04-26T00:44:00.959 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29220.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.35914.log.gz 2024-04-26T00:44:00.959 INFO:teuthology.orchestra.run.smithi037.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.29220.log.gz 2024-04-26T00:44:00.959 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32398.log 2024-04-26T00:44:00.959 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28166.log 2024-04-26T00:44:00.959 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26467.log 2024-04-26T00:44:00.959 INFO:teuthology.orchestra.run.smithi037.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37540.log.gz 2024-04-26T00:44:00.959 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28166.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.32398.log.gz 2024-04-26T00:44:00.959 INFO:teuthology.orchestra.run.smithi037.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.28166.log.gz 2024-04-26T00:44:00.960 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36666.log 2024-04-26T00:44:00.960 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.26467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33344.log 2024-04-26T00:44:00.960 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36666.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-26T00:44:00.960 INFO:teuthology.orchestra.run.smithi037.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.26467.log.gz 2024-04-26T00:44:00.960 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33344.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.36666.log.gz 2024-04-26T00:44:00.960 INFO:teuthology.orchestra.run.smithi037.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.33344.log.gz 2024-04-26T00:44:00.960 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30940.log 2024-04-26T00:44:00.960 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29194.log 2024-04-26T00:44:00.960 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33751.log 2024-04-26T00:44:00.960 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29194.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.30940.log.gz 2024-04-26T00:44:00.960 INFO:teuthology.orchestra.run.smithi037.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.29194.log.gz 2024-04-26T00:44:00.960 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34704.log 2024-04-26T00:44:00.960 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31534.log 2024-04-26T00:44:00.960 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33751.log: /var/log/ceph/ceph-client.admin.34704.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.33751.log.gz 2024-04-26T00:44:00.960 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34990.log 2024-04-26T00:44:00.961 INFO:teuthology.orchestra.run.smithi037.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.34704.log.gz 2024-04-26T00:44:00.961 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31534.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31534.log.gz 2024-04-26T00:44:00.961 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31018.log 2024-04-26T00:44:00.961 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29941.log 2024-04-26T00:44:00.961 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34990.log: /var/log/ceph/ceph-client.admin.31018.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.34990.log.gz 2024-04-26T00:44:00.961 INFO:teuthology.orchestra.run.smithi037.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.31018.log.gz 2024-04-26T00:44:00.961 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38315.log 2024-04-26T00:44:00.961 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29941.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.29941.log.gz 2024-04-26T00:44:00.961 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36128.log 2024-04-26T00:44:00.961 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27568.log 2024-04-26T00:44:00.961 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38315.log: /var/log/ceph/ceph-client.admin.36128.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38315.log.gz 2024-04-26T00:44:00.961 INFO:teuthology.orchestra.run.smithi037.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36128.log.gz 2024-04-26T00:44:00.961 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37290.log 2024-04-26T00:44:00.961 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.27568.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.27568.log.gz 2024-04-26T00:44:00.962 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40054.log 2024-04-26T00:44:00.962 INFO:teuthology.orchestra.run.smithi037.stderr: 91.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-26T00:44:00.962 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35483.log 2024-04-26T00:44:00.962 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37290.log: /var/log/ceph/ceph-client.admin.40054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27750.log 2024-04-26T00:44:00.962 INFO:teuthology.orchestra.run.smithi037.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37290.log.gz 2024-04-26T00:44:00.962 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35483.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.40054.log.gz 2024-04-26T00:44:00.962 INFO:teuthology.orchestra.run.smithi037.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.35483.log.gz 2024-04-26T00:44:00.962 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38514.log 2024-04-26T00:44:00.962 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.27750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31638.log 2024-04-26T00:44:00.962 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36640.log 2024-04-26T00:44:00.962 INFO:teuthology.orchestra.run.smithi037.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.27750.log.gz 2024-04-26T00:44:00.962 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31638.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38514.log.gz 2024-04-26T00:44:00.962 INFO:teuthology.orchestra.run.smithi037.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.31638.log.gz 2024-04-26T00:44:00.962 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36457.log 2024-04-26T00:44:00.962 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35094.log 2024-04-26T00:44:00.963 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27487.log 2024-04-26T00:44:00.963 INFO:teuthology.orchestra.run.smithi037.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36640.log.gz 2024-04-26T00:44:00.963 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35094.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36457.log.gz 2024-04-26T00:44:00.963 INFO:teuthology.orchestra.run.smithi037.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35094.log.gz 2024-04-26T00:44:00.963 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26729.log 2024-04-26T00:44:00.963 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.27487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32430.log 2024-04-26T00:44:00.963 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.26729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28945.log 2024-04-26T00:44:00.963 INFO:teuthology.orchestra.run.smithi037.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.27487.log.gz 2024-04-26T00:44:00.963 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32430.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.26729.log.gz 2024-04-26T00:44:00.963 INFO:teuthology.orchestra.run.smithi037.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.32430.log.gz 2024-04-26T00:44:00.963 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34418.log 2024-04-26T00:44:00.963 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32190.log 2024-04-26T00:44:00.963 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32899.log 2024-04-26T00:44:00.963 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32190.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.28945.log.gz 2024-04-26T00:44:00.964 INFO:teuthology.orchestra.run.smithi037.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.34418.log.gz 2024-04-26T00:44:00.964 INFO:teuthology.orchestra.run.smithi037.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.32190.log.gz 2024-04-26T00:44:00.964 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37514.log 2024-04-26T00:44:00.964 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39365.log 2024-04-26T00:44:00.964 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32899.log: /var/log/ceph/ceph-client.admin.37514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34106.log 2024-04-26T00:44:00.964 INFO:teuthology.orchestra.run.smithi037.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.32899.log.gz 2024-04-26T00:44:00.964 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39365.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37514.log.gz 2024-04-26T00:44:00.964 INFO:teuthology.orchestra.run.smithi037.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.39365.log.gz 2024-04-26T00:44:00.964 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30634.log 2024-04-26T00:44:00.964 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34756.log 2024-04-26T00:44:00.964 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34808.log 2024-04-26T00:44:00.964 INFO:teuthology.orchestra.run.smithi037.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34106.log.gz 2024-04-26T00:44:00.964 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34756.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.30634.log.gz 2024-04-26T00:44:00.964 INFO:teuthology.orchestra.run.smithi037.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.34756.log.gz 2024-04-26T00:44:00.964 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29090.log 2024-04-26T00:44:00.965 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39183.log 2024-04-26T00:44:00.965 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34210.log 2024-04-26T00:44:00.965 INFO:teuthology.orchestra.run.smithi037.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34808.log.gz 2024-04-26T00:44:00.965 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39183.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.29090.log.gz 2024-04-26T00:44:00.965 INFO:teuthology.orchestra.run.smithi037.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.39183.log.gz 2024-04-26T00:44:00.965 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32034.log 2024-04-26T00:44:00.965 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34210.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-26T00:44:00.965 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36796.log 2024-04-26T00:44:00.965 INFO:teuthology.orchestra.run.smithi037.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34210.log.gz 2024-04-26T00:44:00.965 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph.audit.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.32034.log.gz 2024-04-26T00:44:00.965 INFO:teuthology.orchestra.run.smithi037.stderr: 94.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-26T00:44:00.965 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34782.log 2024-04-26T00:44:00.965 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36900.log 2024-04-26T00:44:00.965 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28653.log 2024-04-26T00:44:00.966 INFO:teuthology.orchestra.run.smithi037.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.36796.log.gz 2024-04-26T00:44:00.966 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36900.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34782.log.gz 2024-04-26T00:44:00.966 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32320.log 2024-04-26T00:44:00.966 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40600.log 2024-04-26T00:44:00.966 INFO:teuthology.orchestra.run.smithi037.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36900.log.gz 2024-04-26T00:44:00.966 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37030.log 2024-04-26T00:44:00.966 INFO:teuthology.orchestra.run.smithi037.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.28653.log.gz 2024-04-26T00:44:00.966 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40600.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.32320.log.gz 2024-04-26T00:44:00.966 INFO:teuthology.orchestra.run.smithi037.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40600.log.gz 2024-04-26T00:44:00.966 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38540.log 2024-04-26T00:44:00.966 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30744.log 2024-04-26T00:44:00.966 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31773.log 2024-04-26T00:44:00.966 INFO:teuthology.orchestra.run.smithi037.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37030.log.gz 2024-04-26T00:44:00.966 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30744.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.38540.log.gz 2024-04-26T00:44:00.967 INFO:teuthology.orchestra.run.smithi037.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.30744.log.gz 2024-04-26T00:44:00.967 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37423.log 2024-04-26T00:44:00.967 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30175.log 2024-04-26T00:44:00.967 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31773.log: /var/log/ceph/ceph-client.admin.37423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33425.log 2024-04-26T00:44:00.967 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30175.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.31773.log.gz 2024-04-26T00:44:00.967 INFO:teuthology.orchestra.run.smithi037.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37423.log.gz 2024-04-26T00:44:00.967 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39027.log 2024-04-26T00:44:00.967 INFO:teuthology.orchestra.run.smithi037.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.30175.log.gz 2024-04-26T00:44:00.967 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38748.log 2024-04-26T00:44:00.967 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29628.log 2024-04-26T00:44:00.967 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39027.log: /var/log/ceph/ceph-client.admin.38748.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.33425.log.gz 2024-04-26T00:44:00.967 INFO:teuthology.orchestra.run.smithi037.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.39027.log.gz 2024-04-26T00:44:00.967 INFO:teuthology.orchestra.run.smithi037.stderr: 87.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.38748.log.gz 2024-04-26T00:44:00.967 INFO:teuthology.orchestra.run.smithi037.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.32873.log 2024-04-26T00:44:00.967 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30097.log 2024-04-26T00:44:00.968 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29628.log: /var/log/ceph/ceph-client.admin.32873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27932.log 2024-04-26T00:44:00.968 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30097.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.32873.log.gz 2024-04-26T00:44:00.968 INFO:teuthology.orchestra.run.smithi037.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.29628.log.gz 2024-04-26T00:44:00.968 INFO:teuthology.orchestra.run.smithi037.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.30097.log.gz 2024-04-26T00:44:00.968 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31297.log 2024-04-26T00:44:00.968 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28835.log 2024-04-26T00:44:00.968 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.27932.log: gzip -5 --verbose/var/log/ceph/ceph-client.admin.31297.log: -- /var/log/ceph/ceph-client.admin.35353.log 2024-04-26T00:44:00.968 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28835.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.31297.log.gz 2024-04-26T00:44:00.968 INFO:teuthology.orchestra.run.smithi037.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.27932.log.gz 2024-04-26T00:44:00.968 INFO:teuthology.orchestra.run.smithi037.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.28835.log.gz 2024-04-26T00:44:00.968 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31982.log 2024-04-26T00:44:00.968 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28809.log 2024-04-26T00:44:00.968 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34054.log 2024-04-26T00:44:00.968 INFO:teuthology.orchestra.run.smithi037.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.35353.log.gz 2024-04-26T00:44:00.968 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28809.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.31982.log.gz 2024-04-26T00:44:00.969 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34366.log 2024-04-26T00:44:00.969 INFO:teuthology.orchestra.run.smithi037.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.28809.log.gz 2024-04-26T00:44:00.969 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34054.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.34054.log.gz 2024-04-26T00:44:00.969 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32294.log 2024-04-26T00:44:00.969 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27958.log 2024-04-26T00:44:00.969 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26895.log 2024-04-26T00:44:00.969 INFO:teuthology.orchestra.run.smithi037.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34366.log.gz 2024-04-26T00:44:00.969 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.27958.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.32294.log.gz 2024-04-26T00:44:00.969 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40262.log 2024-04-26T00:44:00.969 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.26895.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.27958.log.gz 2024-04-26T00:44:00.969 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37160.log 2024-04-26T00:44:00.969 INFO:teuthology.orchestra.run.smithi037.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.26895.log.gz 2024-04-26T00:44:00.969 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33032.log 2024-04-26T00:44:00.969 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30836.log 2024-04-26T00:44:00.970 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37160.log: /var/log/ceph/ceph-client.admin.33032.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40262.log.gz 2024-04-26T00:44:00.970 INFO:teuthology.orchestra.run.smithi037.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37160.log.gz 2024-04-26T00:44:00.970 INFO:teuthology.orchestra.run.smithi037.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.33032.log.gzgzip 2024-04-26T00:44:00.970 INFO:teuthology.orchestra.run.smithi037.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.36874.log 2024-04-26T00:44:00.970 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31174.log 2024-04-26T00:44:00.970 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30836.log: /var/log/ceph/ceph-client.admin.36874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33058.log 2024-04-26T00:44:00.970 INFO:teuthology.orchestra.run.smithi037.stderr: 90.4%/var/log/ceph/ceph-client.admin.31174.log: -- replaced with /var/log/ceph/ceph-client.admin.30836.log.gz 2024-04-26T00:44:00.970 INFO:teuthology.orchestra.run.smithi037.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.36874.log.gz 2024-04-26T00:44:00.970 INFO:teuthology.orchestra.run.smithi037.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.31174.log.gz 2024-04-26T00:44:00.970 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30019.log 2024-04-26T00:44:00.970 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32795.log 2024-04-26T00:44:00.970 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38852.log 2024-04-26T00:44:00.970 INFO:teuthology.orchestra.run.smithi037.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.33058.log.gz 2024-04-26T00:44:00.970 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32795.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.30019.log.gz 2024-04-26T00:44:00.970 INFO:teuthology.orchestra.run.smithi037.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.32795.log.gz 2024-04-26T00:44:00.971 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35940.log 2024-04-26T00:44:00.971 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39777.log 2024-04-26T00:44:00.971 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31270.log 2024-04-26T00:44:00.971 INFO:teuthology.orchestra.run.smithi037.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.38852.log.gz 2024-04-26T00:44:00.971 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39777.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.35940.log.gz 2024-04-26T00:44:00.971 INFO:teuthology.orchestra.run.smithi037.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.39777.log.gz 2024-04-26T00:44:00.971 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29350.log 2024-04-26T00:44:00.971 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27255.log 2024-04-26T00:44:00.971 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39422.log 2024-04-26T00:44:00.971 INFO:teuthology.orchestra.run.smithi037.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.31270.log.gz 2024-04-26T00:44:00.971 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.27255.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.29350.log.gz 2024-04-26T00:44:00.971 INFO:teuthology.orchestra.run.smithi037.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.27255.log.gz 2024-04-26T00:44:00.971 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38001.log 2024-04-26T00:44:00.971 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35327.log 2024-04-26T00:44:00.972 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30504.log 2024-04-26T00:44:00.972 INFO:teuthology.orchestra.run.smithi037.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39422.log.gz 2024-04-26T00:44:00.972 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35327.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38001.log.gz 2024-04-26T00:44:00.972 INFO:teuthology.orchestra.run.smithi037.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.35327.log.gz 2024-04-26T00:44:00.972 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40366.log 2024-04-26T00:44:00.972 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30582.log 2024-04-26T00:44:00.972 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38487.log 2024-04-26T00:44:00.972 INFO:teuthology.orchestra.run.smithi037.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.30504.log.gz 2024-04-26T00:44:00.972 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30582.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40366.log.gz 2024-04-26T00:44:00.972 INFO:teuthology.orchestra.run.smithi037.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.30582.log.gz 2024-04-26T00:44:00.972 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35829.log 2024-04-26T00:44:00.972 INFO:teuthology.orchestra.run.smithi037.stderr:gzip/var/log/ceph/ceph-client.admin.38487.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.31747.log 2024-04-26T00:44:00.972 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30201.log 2024-04-26T00:44:00.972 INFO:teuthology.orchestra.run.smithi037.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38487.log.gz 2024-04-26T00:44:00.972 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31747.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35829.log.gz 2024-04-26T00:44:00.973 INFO:teuthology.orchestra.run.smithi037.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.31747.log.gz 2024-04-26T00:44:00.973 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35777.log 2024-04-26T00:44:00.973 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28861.log 2024-04-26T00:44:00.973 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30201.log: gzip/var/log/ceph/ceph-client.admin.35777.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.29967.log 2024-04-26T00:44:00.973 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28861.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.35777.log.gz 2024-04-26T00:44:00.973 INFO:teuthology.orchestra.run.smithi037.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.30201.log.gz 2024-04-26T00:44:00.973 INFO:teuthology.orchestra.run.smithi037.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.28861.log.gz 2024-04-26T00:44:00.973 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27372.log 2024-04-26T00:44:00.973 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38106.log 2024-04-26T00:44:00.973 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30660.log 2024-04-26T00:44:00.973 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.27372.log: /var/log/ceph/ceph-client.admin.38106.log: 88.7% 89.7% -- replaced with /var/log/ceph/ceph-client.admin.29967.log.gz 2024-04-26T00:44:00.973 INFO:teuthology.orchestra.run.smithi037.stderr: -- replaced with /var/log/ceph/ceph-client.admin.27372.log.gz 2024-04-26T00:44:00.973 INFO:teuthology.orchestra.run.smithi037.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.38106.log.gz 2024-04-26T00:44:00.973 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34288.log 2024-04-26T00:44:00.974 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36353.log 2024-04-26T00:44:00.974 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33673.log 2024-04-26T00:44:00.974 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36353.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.30660.log.gz 2024-04-26T00:44:00.974 INFO:teuthology.orchestra.run.smithi037.stderr: 91.1% 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36353.log.gz 2024-04-26T00:44:00.974 INFO:teuthology.orchestra.run.smithi037.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34288.log.gz 2024-04-26T00:44:00.974 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37775.log 2024-04-26T00:44:00.974 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33976.log 2024-04-26T00:44:00.974 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39589.log 2024-04-26T00:44:00.974 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37775.log: /var/log/ceph/ceph-client.admin.33976.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.33673.log.gz 2024-04-26T00:44:00.974 INFO:teuthology.orchestra.run.smithi037.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.37775.log.gz 2024-04-26T00:44:00.974 INFO:teuthology.orchestra.run.smithi037.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33976.log.gz 2024-04-26T00:44:00.974 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26441.log 2024-04-26T00:44:00.974 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38028.log 2024-04-26T00:44:00.974 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39589.log: /var/log/ceph/ceph-client.admin.26441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32138.log 2024-04-26T00:44:00.974 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38028.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39589.log.gz 2024-04-26T00:44:00.975 INFO:teuthology.orchestra.run.smithi037.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.26441.log.gz 2024-04-26T00:44:00.975 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39975.log 2024-04-26T00:44:00.975 INFO:teuthology.orchestra.run.smithi037.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38028.log.gz 2024-04-26T00:44:00.975 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29915.log 2024-04-26T00:44:00.975 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32138.log: /var/log/ceph/ceph-client.admin.39975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26742.log 2024-04-26T00:44:00.975 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29915.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39975.log.gz 2024-04-26T00:44:00.975 INFO:teuthology.orchestra.run.smithi037.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.32138.log.gz 2024-04-26T00:44:00.975 INFO:teuthology.orchestra.run.smithi037.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.29915.log.gz 2024-04-26T00:44:00.975 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37671.log 2024-04-26T00:44:00.975 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33777.log 2024-04-26T00:44:00.975 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.26742.log: /var/log/ceph/ceph-client.admin.37671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26537.log 2024-04-26T00:44:00.975 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33777.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.26742.log.gz 2024-04-26T00:44:00.975 INFO:teuthology.orchestra.run.smithi037.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37671.log.gz 2024-04-26T00:44:00.975 INFO:teuthology.orchestra.run.smithi037.stderr:gzip 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33777.log.gz -5 2024-04-26T00:44:00.976 INFO:teuthology.orchestra.run.smithi037.stderr: --verbose -- /var/log/ceph/ceph-client.admin.37749.log 2024-04-26T00:44:00.976 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29324.log 2024-04-26T00:44:00.995 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.26537.log: /var/log/ceph/ceph-client.admin.37749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40650.log 2024-04-26T00:44:00.995 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29324.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37749.log.gz 2024-04-26T00:44:00.996 INFO:teuthology.orchestra.run.smithi037.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.26537.log.gz 2024-04-26T00:44:00.996 INFO:teuthology.orchestra.run.smithi037.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.29324.log.gz 2024-04-26T00:44:00.996 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33136.log 2024-04-26T00:44:00.996 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26583.log 2024-04-26T00:44:00.996 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40650.log: /var/log/ceph/ceph-client.admin.33136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33699.log 2024-04-26T00:44:00.996 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.26583.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40650.log.gz 2024-04-26T00:44:00.996 INFO:teuthology.orchestra.run.smithi037.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33136.log.gz 2024-04-26T00:44:00.996 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31664.log 2024-04-26T00:44:00.996 INFO:teuthology.orchestra.run.smithi037.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.26583.log.gz 2024-04-26T00:44:00.996 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36718.log 2024-04-26T00:44:00.996 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33699.log: /var/log/ceph/ceph-client.admin.31664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26413.log 2024-04-26T00:44:00.996 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36718.log: 89.3% 90.3% -- replaced with /var/log/ceph/ceph-client.admin.33699.log.gz 2024-04-26T00:44:00.996 INFO:teuthology.orchestra.run.smithi037.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31664.log.gz 2024-04-26T00:44:00.996 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33517.log 2024-04-26T00:44:00.997 INFO:teuthology.orchestra.run.smithi037.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.36718.log.gz 2024-04-26T00:44:00.997 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.26413.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.26413.log.gz 2024-04-26T00:44:00.997 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27906.log 2024-04-26T00:44:00.997 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28627.log 2024-04-26T00:44:00.997 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33517.log: /var/log/ceph/ceph-client.admin.27906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33647.log 2024-04-26T00:44:00.997 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28627.log: 88.4% 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33517.log.gz 2024-04-26T00:44:00.997 INFO:teuthology.orchestra.run.smithi037.stderr: -- replaced with /var/log/ceph/ceph-client.admin.27906.log.gz 2024-04-26T00:44:00.997 INFO:teuthology.orchestra.run.smithi037.stderr:gzip 89.8% -- replaced with /var/log/ceph/ceph-client.admin.28627.log.gz -5 2024-04-26T00:44:00.997 INFO:teuthology.orchestra.run.smithi037.stderr: --verbose -- /var/log/ceph/ceph-client.admin.31852.log 2024-04-26T00:44:00.997 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27540.log 2024-04-26T00:44:00.997 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33647.log: /var/log/ceph/ceph-client.admin.31852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37368.log 2024-04-26T00:44:00.997 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.27540.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33647.log.gz 2024-04-26T00:44:00.997 INFO:teuthology.orchestra.run.smithi037.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.31852.log.gz 2024-04-26T00:44:00.997 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33370.log 2024-04-26T00:44:00.998 INFO:teuthology.orchestra.run.smithi037.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.27540.log.gz 2024-04-26T00:44:00.998 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37368.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.37368.log.gz 2024-04-26T00:44:00.998 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34080.log 2024-04-26T00:44:00.998 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32536.log 2024-04-26T00:44:00.998 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33370.log: gzip/var/log/ceph/ceph-client.admin.34080.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.34470.log 2024-04-26T00:44:00.998 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32536.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34080.log.gz 2024-04-26T00:44:00.998 INFO:teuthology.orchestra.run.smithi037.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.33370.log.gz 2024-04-26T00:44:00.998 INFO:teuthology.orchestra.run.smithi037.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.32536.log.gz 2024-04-26T00:44:00.998 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27802.log 2024-04-26T00:44:00.998 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38618.log 2024-04-26T00:44:00.998 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34470.log: /var/log/ceph/ceph-client.admin.27802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36978.log 2024-04-26T00:44:00.998 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38618.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.34470.log.gz 89.6% -- replaced with /var/log/ceph/ceph-client.admin.27802.log.gz 2024-04-26T00:44:00.998 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:44:00.998 INFO:teuthology.orchestra.run.smithi037.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38618.log.gz 2024-04-26T00:44:00.999 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40001.log 2024-04-26T00:44:00.999 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27330.log 2024-04-26T00:44:00.999 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27121.log 2024-04-26T00:44:00.999 INFO:teuthology.orchestra.run.smithi037.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36978.log.gz 2024-04-26T00:44:00.999 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.27330.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.40001.log.gz 2024-04-26T00:44:00.999 INFO:teuthology.orchestra.run.smithi037.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.27330.log.gz 2024-04-26T00:44:00.999 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29376.log 2024-04-26T00:44:00.999 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28757.log 2024-04-26T00:44:00.999 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.27121.log: gzip -5/var/log/ceph/ceph-client.admin.29376.log: --verbose -- /var/log/ceph/ceph-client.admin.35751.log 2024-04-26T00:44:00.999 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28757.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.27121.log.gz 2024-04-26T00:44:00.999 INFO:teuthology.orchestra.run.smithi037.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.29376.log.gz 2024-04-26T00:44:00.999 INFO:teuthology.orchestra.run.smithi037.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.28757.log.gz 2024-04-26T00:44:00.999 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30426.log 2024-04-26T00:44:00.999 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30712.log 2024-04-26T00:44:00.999 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40340.log 2024-04-26T00:44:01.000 INFO:teuthology.orchestra.run.smithi037.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.35751.log.gz 2024-04-26T00:44:01.000 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30712.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.30426.log.gz 2024-04-26T00:44:01.000 INFO:teuthology.orchestra.run.smithi037.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.30712.log.gz 2024-04-26T00:44:01.000 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32216.log 2024-04-26T00:44:01.000 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36044.log 2024-04-26T00:44:01.000 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40340.log: /var/log/ceph/ceph-client.admin.32216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38367.log 2024-04-26T00:44:01.000 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36044.log: 88.5% 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40340.log.gz 2024-04-26T00:44:01.000 INFO:teuthology.orchestra.run.smithi037.stderr: -- replaced with /var/log/ceph/ceph-client.admin.32216.log.gz 2024-04-26T00:44:01.000 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36379.log 2024-04-26T00:44:01.000 INFO:teuthology.orchestra.run.smithi037.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.36044.log.gzgzip 2024-04-26T00:44:01.000 INFO:teuthology.orchestra.run.smithi037.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.32164.log 2024-04-26T00:44:01.000 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38367.log: /var/log/ceph/ceph-client.admin.36379.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.38367.log.gz 2024-04-26T00:44:01.000 INFO:teuthology.orchestra.run.smithi037.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.36379.log.gz 2024-04-26T00:44:01.000 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39287.log 2024-04-26T00:44:01.000 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31930.log 2024-04-26T00:44:01.001 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32164.log: /var/log/ceph/ceph-client.admin.39287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40444.log 2024-04-26T00:44:01.001 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31930.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.39287.log.gz 89.4% -- replaced with /var/log/ceph/ceph-client.admin.32164.log.gz 2024-04-26T00:44:01.001 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:44:01.001 INFO:teuthology.orchestra.run.smithi037.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.31930.log.gz 2024-04-26T00:44:01.001 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29298.log 2024-04-26T00:44:01.001 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39855.log 2024-04-26T00:44:01.001 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23737.log 2024-04-26T00:44:01.001 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29298.log: /var/log/ceph/ceph-client.admin.39855.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.29298.log.gz 89.1% -- replaced with /var/log/ceph/ceph-client.admin.40444.log.gz 2024-04-26T00:44:01.001 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:44:01.001 INFO:teuthology.orchestra.run.smithi037.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39855.log.gz 2024-04-26T00:44:01.001 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38566.log 2024-04-26T00:44:01.001 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26887.log 2024-04-26T00:44:01.001 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.23737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30071.log 2024-04-26T00:44:01.001 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38566.log: /var/log/ceph/ceph-client.admin.26887.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.38566.log.gz 2024-04-26T00:44:01.002 INFO:teuthology.orchestra.run.smithi037.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.26887.log.gz 2024-04-26T00:44:01.002 INFO:teuthology.orchestra.run.smithi037.stderr: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.23737.log.gz 2024-04-26T00:44:01.002 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35561.log 2024-04-26T00:44:01.002 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33240.log 2024-04-26T00:44:01.002 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32268.log 2024-04-26T00:44:01.002 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33240.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.30071.log.gz 2024-04-26T00:44:01.002 INFO:teuthology.orchestra.run.smithi037.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35561.log.gz 2024-04-26T00:44:01.002 INFO:teuthology.orchestra.run.smithi037.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.33240.log.gz 2024-04-26T00:44:01.002 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36770.log 2024-04-26T00:44:01.002 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36509.log 2024-04-26T00:44:01.002 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32268.log: /var/log/ceph/ceph-client.admin.36770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35803.log 2024-04-26T00:44:01.002 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36509.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32268.log.gz 2024-04-26T00:44:01.002 INFO:teuthology.orchestra.run.smithi037.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36770.log.gz 2024-04-26T00:44:01.002 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27461.log 2024-04-26T00:44:01.003 INFO:teuthology.orchestra.run.smithi037.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36509.log.gz 2024-04-26T00:44:01.003 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28913.log 2024-04-26T00:44:01.003 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35803.log: /var/log/ceph/ceph-client.admin.27461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27698.log 2024-04-26T00:44:01.003 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28913.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.27461.log.gz 2024-04-26T00:44:01.003 INFO:teuthology.orchestra.run.smithi037.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.35803.log.gz 2024-04-26T00:44:01.003 INFO:teuthology.orchestra.run.smithi037.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.28913.log.gz 2024-04-26T00:44:01.003 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38670.log 2024-04-26T00:44:01.003 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27016.log 2024-04-26T00:44:01.003 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.27698.log: /var/log/ceph/ceph-client.admin.38670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31070.log 2024-04-26T00:44:01.003 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.27016.log: 89.4% 89.9% -- replaced with /var/log/ceph/ceph-client.admin.38670.log.gz -- replaced with /var/log/ceph/ceph-client.admin.27698.log.gz 2024-04-26T00:44:01.003 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:44:01.003 INFO:teuthology.orchestra.run.smithi037.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.27016.log.gz 2024-04-26T00:44:01.003 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34184.log 2024-04-26T00:44:01.003 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33266.log 2024-04-26T00:44:01.010 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31070.log: gzip -5 --verbose --/var/log/ceph/ceph-client.admin.34184.log: /var/log/ceph/ceph-client.admin.39910.log 2024-04-26T00:44:01.010 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33266.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.31070.log.gz 2024-04-26T00:44:01.010 INFO:teuthology.orchestra.run.smithi037.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.34184.log.gz 2024-04-26T00:44:01.010 INFO:teuthology.orchestra.run.smithi037.stderr:gzip 88.1% -5 -- replaced with /var/log/ceph/ceph-client.admin.33266.log.gz --verbose 2024-04-26T00:44:01.011 INFO:teuthology.orchestra.run.smithi037.stderr: -- /var/log/ceph/ceph-client.admin.40132.log 2024-04-26T00:44:01.011 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35379.log 2024-04-26T00:44:01.011 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37645.log/var/log/ceph/ceph-client.admin.40132.log: 2024-04-26T00:44:01.011 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35379.log: 88.7% 89.9% -- replaced with /var/log/ceph/ceph-client.admin.39910.log.gz -- replaced with /var/log/ceph/ceph-client.admin.40132.log.gz 2024-04-26T00:44:01.011 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:44:01.011 INFO:teuthology.orchestra.run.smithi037.stderr: 89.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34678.log 2024-04-26T00:44:01.011 INFO:teuthology.orchestra.run.smithi037.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35379.log.gz 2024-04-26T00:44:01.011 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28887.log 2024-04-26T00:44:01.011 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37645.log: /var/log/ceph/ceph-client.admin.34678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27594.log 2024-04-26T00:44:01.011 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28887.log: 88.5% 89.8% -- replaced with /var/log/ceph/ceph-client.admin.37645.log.gz 2024-04-26T00:44:01.011 INFO:teuthology.orchestra.run.smithi037.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34678.log.gz 2024-04-26T00:44:01.011 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36561.log 2024-04-26T00:44:01.011 INFO:teuthology.orchestra.run.smithi037.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.28887.log.gz 2024-04-26T00:44:01.011 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38158.log 2024-04-26T00:44:01.011 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.27594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37723.log 2024-04-26T00:44:01.012 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36561.log: /var/log/ceph/ceph-client.admin.38158.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.27594.log.gz 2024-04-26T00:44:01.012 INFO:teuthology.orchestra.run.smithi037.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36561.log.gz 2024-04-26T00:44:01.012 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32769.log 2024-04-26T00:44:01.012 INFO:teuthology.orchestra.run.smithi037.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.38158.log.gz 2024-04-26T00:44:01.012 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34548.log 2024-04-26T00:44:01.012 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37723.log: /var/log/ceph/ceph-client.admin.32769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29760.log 2024-04-26T00:44:01.012 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34548.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.32769.log.gz 2024-04-26T00:44:01.012 INFO:teuthology.orchestra.run.smithi037.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37723.log.gz 2024-04-26T00:44:01.012 INFO:teuthology.orchestra.run.smithi037.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.34548.log.gz 2024-04-26T00:44:01.012 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37619.log 2024-04-26T00:44:01.012 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40314.log 2024-04-26T00:44:01.012 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29760.log: /var/log/ceph/ceph-client.admin.37619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40418.log 2024-04-26T00:44:01.012 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40314.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.37619.log.gz 2024-04-26T00:44:01.012 INFO:teuthology.orchestra.run.smithi037.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.40314.log.gz 2024-04-26T00:44:01.013 INFO:teuthology.orchestra.run.smithi037.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.29760.log.gz 2024-04-26T00:44:01.013 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29524.log 2024-04-26T00:44:01.013 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26587.log 2024-04-26T00:44:01.013 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28783.log 2024-04-26T00:44:01.013 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29524.log: /var/log/ceph/ceph-client.admin.26587.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40418.log.gz 2024-04-26T00:44:01.013 INFO:teuthology.orchestra.run.smithi037.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.29524.log.gz 2024-04-26T00:44:01.013 INFO:teuthology.orchestra.run.smithi037.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.26587.log.gz 2024-04-26T00:44:01.013 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28783.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28783.log.gz 2024-04-26T00:44:01.094 INFO:teuthology.orchestra.run.smithi045.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-26T00:44:01.375 INFO:teuthology.orchestra.run.smithi045.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-26T00:44:01.618 INFO:teuthology.orchestra.run.smithi037.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-26T00:44:01.877 INFO:teuthology.orchestra.run.smithi037.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-26T00:44:02.554 INFO:teuthology.orchestra.run.smithi045.stderr: 91.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-26T00:44:04.508 INFO:teuthology.orchestra.run.smithi037.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-26T00:44:05.068 INFO:teuthology.orchestra.run.smithi045.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-26T00:44:05.292 INFO:teuthology.orchestra.run.smithi045.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-26T00:44:05.293 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-26T00:44:05.293 INFO:teuthology.orchestra.run.smithi045.stderr:real 0m4.554s 2024-04-26T00:44:05.293 INFO:teuthology.orchestra.run.smithi045.stderr:user 0m11.379s 2024-04-26T00:44:05.293 INFO:teuthology.orchestra.run.smithi045.stderr:sys 0m0.338s 2024-04-26T00:44:06.295 INFO:teuthology.orchestra.run.smithi037.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-26T00:44:07.518 INFO:teuthology.orchestra.run.smithi037.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-26T00:44:07.520 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T00:44:07.520 INFO:teuthology.orchestra.run.smithi037.stderr:real 0m6.817s 2024-04-26T00:44:07.520 INFO:teuthology.orchestra.run.smithi037.stderr:user 0m17.963s 2024-04-26T00:44:07.520 INFO:teuthology.orchestra.run.smithi037.stderr:sys 0m0.686s 2024-04-26T00:44:07.521 INFO:tasks.ceph:Archiving logs... 2024-04-26T00:44:07.521 DEBUG:teuthology.misc:Transferring archived files from smithi037:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673875/remote/smithi037/log 2024-04-26T00:44:07.522 DEBUG:teuthology.orchestra.run.smithi037:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T00:44:10.334 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673875/remote/smithi045/log 2024-04-26T00:44:10.334 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T00:44:11.900 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-26T00:44:11.910 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-26T00:44:11.910 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T00:44:11.913 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T00:44:11.987 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-26T00:44:11.996 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-26T00:44:11.996 DEBUG:teuthology.orchestra.run.smithi037:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T00:44:11.998 DEBUG:teuthology.orchestra.run.smithi045:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T00:44:12.010 INFO:teuthology.orchestra.run.smithi045.stdout: remote refid st t when poll reach delay offset jitter 2024-04-26T00:44:12.010 INFO:teuthology.orchestra.run.smithi045.stdout:============================================================================== 2024-04-26T00:44:12.010 INFO:teuthology.orchestra.run.smithi045.stdout:+hv01.front.sepi 67.205.162.81 3 u 50 64 377 0.091 -4.411 0.433 2024-04-26T00:44:12.010 INFO:teuthology.orchestra.run.smithi045.stdout:*hv02.front.sepi 63.231.80.2 3 u 55 64 377 0.099 -2.425 0.506 2024-04-26T00:44:12.010 INFO:teuthology.orchestra.run.smithi045.stdout:+hv03.front.sepi 74.6.168.72 3 u 55 64 377 0.087 -1.785 0.447 2024-04-26T00:44:12.011 INFO:teuthology.orchestra.run.smithi045.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-04-26T00:44:12.012 INFO:teuthology.orchestra.run.smithi037.stdout: remote refid st t when poll reach delay offset jitter 2024-04-26T00:44:12.012 INFO:teuthology.orchestra.run.smithi037.stdout:============================================================================== 2024-04-26T00:44:12.012 INFO:teuthology.orchestra.run.smithi037.stdout:+hv01.front.sepi 67.205.162.81 3 u 66 64 377 0.101 -3.604 0.294 2024-04-26T00:44:12.012 INFO:teuthology.orchestra.run.smithi037.stdout:*hv02.front.sepi 63.231.80.2 3 u 51 64 377 0.096 -1.732 0.451 2024-04-26T00:44:12.012 INFO:teuthology.orchestra.run.smithi037.stdout:+hv03.front.sepi 74.6.168.72 3 u 64 64 377 0.077 -1.455 0.197 2024-04-26T00:44:12.012 INFO:teuthology.orchestra.run.smithi037.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-04-26T00:44:12.013 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-26T00:44:12.022 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-26T00:44:12.022 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-26T00:44:12.051 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-26T00:44:12.084 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-26T00:44:12.118 INFO:teuthology.task.internal:Duration was 2065.648260 seconds 2024-04-26T00:44:12.121 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-26T00:44:12.151 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-26T00:44:12.151 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T00:44:12.154 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T00:44:12.191 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-26T00:44:12.192 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi037.front.sepia.ceph.com 2024-04-26T00:44:12.192 DEBUG:teuthology.orchestra.run.smithi037:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T00:44:12.245 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi045.front.sepia.ceph.com 2024-04-26T00:44:12.245 DEBUG:teuthology.orchestra.run.smithi045:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T00:44:12.255 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-26T00:44:12.255 DEBUG:teuthology.orchestra.run.smithi037:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T00:44:12.292 DEBUG:teuthology.orchestra.run.smithi045:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T00:44:12.326 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-26T00:44:12.327 DEBUG:teuthology.orchestra.run.smithi037:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T00:44:12.364 DEBUG:teuthology.orchestra.run.smithi045:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T00:44:12.429 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-26T00:44:12.440 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-26T00:44:12.441 DEBUG:teuthology.orchestra.run.smithi037:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T00:44:12.451 DEBUG:teuthology.orchestra.run.smithi045:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T00:44:12.465 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-26T00:44:12.487 DEBUG:teuthology.orchestra.run.smithi037:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T00:44:12.496 DEBUG:teuthology.orchestra.run.smithi045:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T00:44:12.505 INFO:teuthology.orchestra.run.smithi037.stdout:kernel.core_pattern = core 2024-04-26T00:44:12.521 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern = core 2024-04-26T00:44:12.547 DEBUG:teuthology.orchestra.run.smithi037:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T00:44:12.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T00:44:12.579 DEBUG:teuthology.orchestra.run.smithi045:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T00:44:12.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T00:44:12.593 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-26T00:44:12.607 INFO:teuthology.task.internal:Transferring archived files... 2024-04-26T00:44:12.608 DEBUG:teuthology.misc:Transferring archived files from smithi037:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673875/remote/smithi037 2024-04-26T00:44:12.609 DEBUG:teuthology.orchestra.run.smithi037:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T00:44:12.646 DEBUG:teuthology.misc:Transferring archived files from smithi045:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7673875/remote/smithi045 2024-04-26T00:44:12.646 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T00:44:12.667 INFO:teuthology.task.internal:Removing archive directory... 2024-04-26T00:44:12.667 DEBUG:teuthology.orchestra.run.smithi037:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T00:44:12.685 DEBUG:teuthology.orchestra.run.smithi045:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T00:44:12.713 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-26T00:44:12.724 INFO:teuthology.task.internal:Not uploading archives. 2024-04-26T00:44:12.724 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-26T00:44:12.752 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-26T00:44:12.752 DEBUG:teuthology.orchestra.run.smithi037:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T00:44:12.755 DEBUG:teuthology.orchestra.run.smithi045:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T00:44:12.758 INFO:teuthology.orchestra.run.smithi037.stdout: 658645 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 00:44 /home/ubuntu/cephtest 2024-04-26T00:44:12.762 INFO:teuthology.orchestra.run.smithi045.stdout: 658465 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 00:44 /home/ubuntu/cephtest 2024-04-26T00:44:12.763 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-26T00:44:12.785 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-26T00:44:12.854 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/snaps-few-objects} duration: 2065.648260116577 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-04-26T00:44:12.854 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T00:44:12.942 INFO:teuthology.run:pass